2024-04-06T11:58:36.378 INFO:root:teuthology version: 0.0.1.dev265+gd7d6e60 2024-04-06T11:58:36.379 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} --archive /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7638946 --name teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7638946/orig.config.yaml 2024-04-06T11:58:36.424 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T11:58:36.518 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7638946 branch: reef description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} email: ceph-qa@ceph.com first_in_suite: false job_id: '7638946' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: reef ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 052adcadd0534eddb0d73c5afbc083a84cd9dc46 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 052adcadd0534eddb0d73c5afbc083a84cd9dc46 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: 800ec6d64a7fa0648ad53edfc6e50b41fc4cd674 owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 3144 sha1: 052adcadd0534eddb0d73c5afbc083a84cd9dc46 sleep_before_teardown: 0 subset: 362/512 suite: fs suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 800ec6d64a7fa0648ad53edfc6e50b41fc4cd674 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi189.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOXPuN01E+uZRPA3yUcIGKyxrMNnqr19RUH/qeLjk0BP0SOkrpy/5Y/DlQM9mxutjjN3CWjMmARBPcUTj2zBRV0= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNuSqPRtJ6bQFn5hEQRh6snW6a/7ZpRBA3dX92nc3WrdVqhFxOKLgJ9IREoVgDeyxeuIF4P2zNIcRk9MH9o9b2I= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: d7d6e60867cadf19fbfd7655a551ae7d58060757 timestamp: 2024-04-03_22:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-06T11:58:36.518 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa; will attempt to use it 2024-04-06T11:58:36.519 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa/tasks 2024-04-06T11:58:36.519 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-06T11:58:36.521 INFO:teuthology.task.internal:Checking packages... 2024-04-06T11:58:36.545 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '052adcadd0534eddb0d73c5afbc083a84cd9dc46' 2024-04-06T11:58:36.545 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-06T11:58:36.545 INFO:teuthology.packaging:ref: None 2024-04-06T11:58:36.545 INFO:teuthology.packaging:tag: None 2024-04-06T11:58:36.545 INFO:teuthology.packaging:branch: reef 2024-04-06T11:58:36.545 INFO:teuthology.packaging:sha1: 052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T11:58:36.545 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=reef 2024-04-06T11:58:36.832 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1169.g37f5b3f8 2024-04-06T11:58:36.834 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-06T11:58:36.840 INFO:teuthology.task.internal:no buildpackages task found 2024-04-06T11:58:36.840 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-06T11:58:36.896 INFO:teuthology.task.internal:Saving configuration 2024-04-06T11:58:36.913 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-06T11:58:36.979 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T11:58:37.003 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7638946', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 11:53:19.488950', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOXPuN01E+uZRPA3yUcIGKyxrMNnqr19RUH/qeLjk0BP0SOkrpy/5Y/DlQM9mxutjjN3CWjMmARBPcUTj2zBRV0='} 2024-04-06T11:58:37.026 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7638946', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 11:53:19.490898', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNuSqPRtJ6bQFn5hEQRh6snW6a/7ZpRBA3dX92nc3WrdVqhFxOKLgJ9IREoVgDeyxeuIF4P2zNIcRk9MH9o9b2I='} 2024-04-06T11:58:37.026 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-06T11:58:37.037 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-06T11:58:37.037 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-06T11:58:37.037 INFO:teuthology.run_tasks:Running task console_log... 2024-04-06T11:58:37.146 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f01cc73f0d0>, signals=[15]) 2024-04-06T11:58:37.146 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-06T11:58:37.153 INFO:teuthology.task.internal:Opening connections... 2024-04-06T11:58:37.153 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T11:58:37.155 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T11:58:37.234 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T11:58:37.236 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T11:58:37.310 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-06T11:58:37.316 DEBUG:teuthology.orchestra.run.smithi189:> uname -m 2024-04-06T11:58:37.337 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2024-04-06T11:58:37.338 DEBUG:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2024-04-06T11:58:37.398 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="Red Hat Enterprise Linux" 2024-04-06T11:58:37.398 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="8.6 (Ootpa)" 2024-04-06T11:58:37.398 INFO:teuthology.orchestra.run.smithi189.stdout:ID="rhel" 2024-04-06T11:58:37.398 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE="fedora" 2024-04-06T11:58:37.398 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="8.6" 2024-04-06T11:58:37.398 INFO:teuthology.orchestra.run.smithi189.stdout:PLATFORM_ID="platform:el8" 2024-04-06T11:58:37.398 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-06T11:58:37.398 INFO:teuthology.orchestra.run.smithi189.stdout:ANSI_COLOR="0;31" 2024-04-06T11:58:37.398 INFO:teuthology.orchestra.run.smithi189.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-06T11:58:37.398 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://www.redhat.com/" 2024-04-06T11:58:37.398 INFO:teuthology.orchestra.run.smithi189.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-06T11:58:37.398 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-06T11:58:37.399 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T11:58:37.399 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-06T11:58:37.399 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-06T11:58:37.399 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-06T11:58:37.399 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-06T11:58:37.400 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2024-04-06T11:58:37.424 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2024-04-06T11:58:37.444 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2024-04-06T11:58:37.445 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2024-04-06T11:58:37.505 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="Red Hat Enterprise Linux" 2024-04-06T11:58:37.505 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="8.6 (Ootpa)" 2024-04-06T11:58:37.505 INFO:teuthology.orchestra.run.smithi192.stdout:ID="rhel" 2024-04-06T11:58:37.505 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="fedora" 2024-04-06T11:58:37.505 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="8.6" 2024-04-06T11:58:37.505 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el8" 2024-04-06T11:58:37.505 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-06T11:58:37.505 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2024-04-06T11:58:37.505 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-06T11:58:37.505 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://www.redhat.com/" 2024-04-06T11:58:37.506 INFO:teuthology.orchestra.run.smithi192.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-06T11:58:37.506 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-06T11:58:37.506 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T11:58:37.506 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-06T11:58:37.506 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-06T11:58:37.506 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-06T11:58:37.506 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-06T11:58:37.506 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2024-04-06T11:58:37.523 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-06T11:58:37.533 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-06T11:58:37.612 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-06T11:58:37.613 DEBUG:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2024-04-06T11:58:37.615 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2024-04-06T11:58:37.634 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-06T11:58:37.662 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-06T11:58:37.663 DEBUG:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2024-04-06T11:58:37.675 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2024-04-06T11:58:37.720 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-06T11:58:37.771 INFO:teuthology.run_tasks:Running task kernel... 2024-04-06T11:58:37.804 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-06T11:58:37.804 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2024-04-06T11:58:37.804 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2024-04-06T11:58:37.805 DEBUG:teuthology.orchestra.run.smithi189:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-06T11:58:37.805 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-06T11:58:37.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T11:58:37.823 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-04-06T11:58:37.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T11:58:37.824 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-04-06T11:58:37.883 INFO:teuthology.orchestra.run.smithi189.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-06T11:58:37.884 INFO:teuthology.task.kernel:Running kernel on smithi189: 4.18.0-513.9.1.el8_9.x86_64 2024-04-06T11:58:37.884 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum install -y kernel 2024-04-06T11:58:37.886 INFO:teuthology.orchestra.run.smithi192.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-06T11:58:37.886 INFO:teuthology.task.kernel:Running kernel on smithi192: 4.18.0-513.9.1.el8_9.x86_64 2024-04-06T11:58:37.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum install -y kernel 2024-04-06T11:58:39.388 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2024-04-06T11:58:39.388 INFO:teuthology.orchestra.run.smithi192.stdout:Unable to read consumer identity 2024-04-06T11:58:39.522 INFO:teuthology.orchestra.run.smithi189.stdout:Updating Subscription Management repositories. 2024-04-06T11:58:39.522 INFO:teuthology.orchestra.run.smithi189.stdout:Unable to read consumer identity 2024-04-06T11:58:39.702 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T11:58:39.702 INFO:teuthology.orchestra.run.smithi192.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-06T11:58:39.702 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T11:58:39.793 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T11:58:39.794 INFO:teuthology.orchestra.run.smithi189.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-06T11:58:39.794 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T11:58:41.473 INFO:teuthology.orchestra.run.smithi192.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-04-06T11:58:41.501 INFO:teuthology.orchestra.run.smithi189.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-04-06T11:58:42.660 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-04-06T11:58:43.351 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 8.8 MB/s | 16 MB 00:01 2024-04-06T11:58:46.311 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 450 kB/s | 24 kB 00:00 2024-04-06T11:58:47.015 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 479 kB/s | 24 kB 00:00 2024-04-06T11:58:47.063 INFO:teuthology.orchestra.run.smithi189.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-06T11:58:47.063 INFO:teuthology.orchestra.run.smithi189.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-06T11:58:47.141 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-06T11:58:47.142 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-06T11:58:47.142 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-06T11:58:47.294 DEBUG:teuthology.orchestra.run.smithi189:> echo no | sudo yum reinstall kernel || true 2024-04-06T11:58:47.631 INFO:teuthology.orchestra.run.smithi189.stdout:Updating Subscription Management repositories. 2024-04-06T11:58:47.632 INFO:teuthology.orchestra.run.smithi189.stdout:Unable to read consumer identity 2024-04-06T11:58:47.776 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-06T11:58:47.777 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-06T11:58:47.852 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-06T11:58:47.853 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-06T11:58:47.853 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-06T11:58:47.888 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T11:58:47.888 INFO:teuthology.orchestra.run.smithi189.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-06T11:58:47.889 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T11:58:47.988 DEBUG:teuthology.orchestra.run.smithi192:> echo no | sudo yum reinstall kernel || true 2024-04-06T11:58:48.203 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:02 ago on Sat 06 Apr 2024 11:58:46 AM UTC. 2024-04-06T11:58:48.291 INFO:teuthology.orchestra.run.smithi189.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-06T11:58:48.298 INFO:teuthology.orchestra.run.smithi189.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-06T11:58:48.327 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2024-04-06T11:58:48.327 INFO:teuthology.orchestra.run.smithi192.stdout:Unable to read consumer identity 2024-04-06T11:58:48.336 INFO:teuthology.orchestra.run.smithi189.stderr:Error: No packages marked for reinstall. 2024-04-06T11:58:48.384 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum reinstall -y kernel || true 2024-04-06T11:58:48.547 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T11:58:48.548 INFO:teuthology.orchestra.run.smithi192.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-06T11:58:48.548 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T11:58:48.719 INFO:teuthology.orchestra.run.smithi189.stdout:Updating Subscription Management repositories. 2024-04-06T11:58:48.719 INFO:teuthology.orchestra.run.smithi189.stdout:Unable to read consumer identity 2024-04-06T11:58:48.859 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:02 ago on Sat 06 Apr 2024 11:58:46 AM UTC. 2024-04-06T11:58:48.950 INFO:teuthology.orchestra.run.smithi192.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-06T11:58:48.950 INFO:teuthology.orchestra.run.smithi192.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-06T11:58:48.950 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T11:58:48.951 INFO:teuthology.orchestra.run.smithi189.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-06T11:58:48.951 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T11:58:48.988 INFO:teuthology.orchestra.run.smithi192.stderr:Error: No packages marked for reinstall. 2024-04-06T11:58:49.035 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum reinstall -y kernel || true 2024-04-06T11:58:49.262 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:03 ago on Sat 06 Apr 2024 11:58:46 AM UTC. 2024-04-06T11:58:49.328 INFO:teuthology.orchestra.run.smithi189.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-06T11:58:49.328 INFO:teuthology.orchestra.run.smithi189.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-06T11:58:49.358 INFO:teuthology.orchestra.run.smithi189.stderr:Error: No packages marked for reinstall. 2024-04-06T11:58:49.369 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2024-04-06T11:58:49.369 INFO:teuthology.orchestra.run.smithi192.stdout:Unable to read consumer identity 2024-04-06T11:58:49.404 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q kernel | sort -rV | head -n 1 2024-04-06T11:58:49.570 INFO:teuthology.orchestra.run.smithi189.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-06T11:58:49.570 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-06T11:58:49.570 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-06T11:58:49.570 DEBUG:teuthology.parallel:result is None 2024-04-06T11:58:49.573 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T11:58:49.574 INFO:teuthology.orchestra.run.smithi192.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-06T11:58:49.574 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T11:58:49.890 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:03 ago on Sat 06 Apr 2024 11:58:46 AM UTC. 2024-04-06T11:58:49.961 INFO:teuthology.orchestra.run.smithi192.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-06T11:58:49.962 INFO:teuthology.orchestra.run.smithi192.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-06T11:58:49.993 INFO:teuthology.orchestra.run.smithi192.stderr:Error: No packages marked for reinstall. 2024-04-06T11:58:50.046 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2024-04-06T11:58:50.200 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-06T11:58:50.200 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-06T11:58:50.201 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-06T11:58:50.201 DEBUG:teuthology.parallel:result is None 2024-04-06T11:58:50.201 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-06T11:58:50.208 INFO:teuthology.task.internal:Creating test directory... 2024-04-06T11:58:50.208 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-06T11:58:50.211 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-06T11:58:50.227 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-06T11:58:50.245 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-06T11:58:50.287 INFO:teuthology.task.internal:Creating archive directory... 2024-04-06T11:58:50.287 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-06T11:58:50.289 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-06T11:58:50.322 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-06T11:58:50.328 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-06T11:58:50.328 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-06T11:58:50.357 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-06T11:58:50.385 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T11:58:50.392 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T11:58:50.404 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T11:58:50.423 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T11:58:50.425 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-06T11:58:50.433 INFO:teuthology.task.internal:Configuring sudo... 2024-04-06T11:58:50.433 DEBUG:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-06T11:58:50.449 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-06T11:58:50.492 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-06T11:58:50.501 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-06T11:58:50.501 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-06T11:58:50.517 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-06T11:58:50.549 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T11:58:50.614 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T11:58:50.699 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T11:58:50.743 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T11:58:50.784 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T11:58:50.784 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-06T11:58:50.851 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T11:58:50.896 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T11:58:50.941 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T11:58:51.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T11:58:51.067 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T11:58:51.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-06T11:58:51.136 DEBUG:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2024-04-06T11:58:51.138 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2024-04-06T11:58:51.185 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-06T11:58:51.227 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-06T11:58:51.643 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-06T11:58:51.650 INFO:teuthology.task.internal:Starting timer... 2024-04-06T11:58:51.650 INFO:teuthology.run_tasks:Running task pcp... 2024-04-06T11:58:51.715 INFO:teuthology.run_tasks:Running task selinux... 2024-04-06T11:58:51.772 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2024-04-06T11:58:51.772 DEBUG:teuthology.orchestra.run.smithi189:> sudo service auditd rotate 2024-04-06T11:58:51.820 INFO:teuthology.orchestra.run.smithi189.stdout:Rotating logs: 2024-04-06T11:58:51.822 DEBUG:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2024-04-06T11:58:51.879 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: 2024-04-06T11:58:51.881 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-06T11:58:51.882 DEBUG:teuthology.orchestra.run.smithi189:> /usr/sbin/getenforce 2024-04-06T11:58:51.903 INFO:teuthology.orchestra.run.smithi189.stdout:Permissive 2024-04-06T11:58:51.903 DEBUG:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2024-04-06T11:58:51.926 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2024-04-06T11:58:51.926 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi189.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi192.front.sepia.ceph.com': 'permissive'} 2024-04-06T11:58:51.926 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-06T11:58:51.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T11:58:51.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-06T11:58:51.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T11:58:51.998 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-06T11:58:51.998 DEBUG:teuthology.orchestra.run.smithi189:> sudo /usr/sbin/setenforce permissive 2024-04-06T11:58:52.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2024-04-06T11:58:52.065 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-06T11:58:52.077 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-06T11:58:52.083 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-06T11:58:52.211 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-06T11:58:52.226 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-06T11:58:52.227 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi189.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-06T12:08:20.413 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi189.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2024-04-06T12:08:20.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-06T12:08:20.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T12:08:20.494 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-04-06T12:08:20.566 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-06T12:08:20.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-04-06T12:08:20.567 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T12:08:20.643 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-04-06T12:08:20.669 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-04-06T12:08:20.669 INFO:teuthology.run_tasks:Running task clock... 2024-04-06T12:08:20.679 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-06T12:08:20.679 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-06T12:08:20.680 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T12:08:20.682 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-06T12:08:20.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T12:08:20.713 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-06T12:08:20.728 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-06T12:08:20.742 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-06T12:08:20.757 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-06T12:08:20.797 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: ntpd: command not found 2024-04-06T12:08:20.809 INFO:teuthology.orchestra.run.smithi189.stdout:506 Cannot talk to daemon 2024-04-06T12:08:20.824 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-06T12:08:20.827 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: ntpd: command not found 2024-04-06T12:08:20.839 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-06T12:08:20.839 INFO:teuthology.orchestra.run.smithi192.stdout:506 Cannot talk to daemon 2024-04-06T12:08:20.855 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-06T12:08:20.869 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-06T12:08:20.917 INFO:teuthology.orchestra.run.smithi189.stderr:bash: ntpq: command not found 2024-04-06T12:08:20.921 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-06T12:08:20.921 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-04-06T12:08:20.921 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T12:08:20.921 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T12:08:20.921 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T12:08:20.921 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T12:08:20.944 INFO:teuthology.orchestra.run.smithi192.stderr:bash: ntpq: command not found 2024-04-06T12:08:20.948 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-06T12:08:20.948 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-04-06T12:08:20.948 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T12:08:20.948 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T12:08:20.948 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T12:08:20.948 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T12:08:20.949 INFO:teuthology.run_tasks:Running task install... 2024-04-06T12:08:20.957 DEBUG:teuthology.task.install:project ceph 2024-04-06T12:08:20.957 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46'}} 2024-04-06T12:08:20.957 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46'} 2024-04-06T12:08:20.957 INFO:teuthology.task.install:Using flavor: default 2024-04-06T12:08:20.963 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-06T12:08:20.963 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-04-06T12:08:20.964 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46', 'tag': None, 'wait_for_package': False} 2024-04-06T12:08:20.964 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T12:08:20.966 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46', 'tag': None, 'wait_for_package': False} 2024-04-06T12:08:20.967 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T12:08:21.199 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/reef/052adcadd0534eddb0d73c5afbc083a84cd9dc46/centos/8/flavors/default/ 2024-04-06T12:08:21.199 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1086.g052adcad 2024-04-06T12:08:21.203 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/reef/052adcadd0534eddb0d73c5afbc083a84cd9dc46/centos/8/flavors/default/ 2024-04-06T12:08:21.203 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1086.g052adcad 2024-04-06T12:08:21.311 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/reef/052adcadd0534eddb0d73c5afbc083a84cd9dc46/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/052adcadd0534eddb0d73c5afbc083a84cd9dc46/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/052adcadd0534eddb0d73c5afbc083a84cd9dc46/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-06T12:08:21.312 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:08:21.312 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-06T12:08:21.317 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/reef/052adcadd0534eddb0d73c5afbc083a84cd9dc46/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/052adcadd0534eddb0d73c5afbc083a84cd9dc46/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/052adcadd0534eddb0d73c5afbc083a84cd9dc46/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-06T12:08:21.317 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:08:21.317 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-06T12:08:21.341 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-06T12:08:21.341 DEBUG:teuthology.orchestra.run.smithi189:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/052adcadd0534eddb0d73c5afbc083a84cd9dc46/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-06T12:08:21.345 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-06T12:08:21.346 DEBUG:teuthology.orchestra.run.smithi192:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/052adcadd0534eddb0d73c5afbc083a84cd9dc46/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-06T12:08:21.410 DEBUG:teuthology.orchestra.run.smithi189:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-06T12:08:21.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-06T12:08:21.490 DEBUG:teuthology.orchestra.run.smithi189:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-06T12:08:21.492 DEBUG:teuthology.orchestra.run.smithi192:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-06T12:08:21.558 INFO:teuthology.orchestra.run.smithi192.stdout:check_obsoletes = 1 2024-04-06T12:08:21.558 INFO:teuthology.orchestra.run.smithi189.stdout:check_obsoletes = 1 2024-04-06T12:08:21.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum clean all 2024-04-06T12:08:21.560 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum clean all 2024-04-06T12:08:21.934 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2024-04-06T12:08:21.941 INFO:teuthology.orchestra.run.smithi189.stdout:Updating Subscription Management repositories. 2024-04-06T12:08:23.473 INFO:teuthology.orchestra.run.smithi189.stdout:61 files removed 2024-04-06T12:08:23.507 INFO:teuthology.orchestra.run.smithi192.stdout:61 files removed 2024-04-06T12:08:23.526 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-06T12:08:23.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-06T12:08:23.862 INFO:teuthology.orchestra.run.smithi189.stdout:Updating Subscription Management repositories. 2024-04-06T12:08:23.899 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2024-04-06T12:08:25.677 INFO:teuthology.orchestra.run.smithi192.stdout:ceph packages for x86_64 241 kB/s | 80 kB 00:00 2024-04-06T12:08:26.027 INFO:teuthology.orchestra.run.smithi192.stdout:ceph noarch packages 36 kB/s | 11 kB 00:00 2024-04-06T12:08:26.653 INFO:teuthology.orchestra.run.smithi192.stdout:ceph source packages 2.5 kB/s | 1.5 kB 00:00 2024-04-06T12:08:26.838 INFO:teuthology.orchestra.run.smithi189.stdout:ceph packages for x86_64 56 kB/s | 80 kB 00:01 2024-04-06T12:08:26.846 INFO:teuthology.orchestra.run.smithi192.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-04-06T12:08:27.463 INFO:teuthology.orchestra.run.smithi189.stdout:ceph noarch packages 19 kB/s | 11 kB 00:00 2024-04-06T12:08:27.789 INFO:teuthology.orchestra.run.smithi189.stdout:ceph source packages 5.0 kB/s | 1.5 kB 00:00 2024-04-06T12:08:27.956 INFO:teuthology.orchestra.run.smithi189.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-04-06T12:08:28.647 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 9.2 MB/s | 16 MB 00:01 2024-04-06T12:08:29.234 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-04-06T12:08:32.238 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 422 kB/s | 24 kB 00:00 2024-04-06T12:08:32.857 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 347 kB/s | 24 kB 00:00 2024-04-06T12:08:33.015 INFO:teuthology.orchestra.run.smithi192.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 13 MB/s | 8.8 MB 00:00 2024-04-06T12:08:34.209 INFO:teuthology.orchestra.run.smithi189.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 52 MB/s | 66 MB 00:01 2024-04-06T12:08:35.801 INFO:teuthology.orchestra.run.smithi192.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 52 MB/s | 66 MB 00:01 2024-04-06T12:08:44.234 INFO:teuthology.orchestra.run.smithi189.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 15 MB/s | 8.8 MB 00:00 2024-04-06T12:08:46.468 INFO:teuthology.orchestra.run.smithi192.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 48 MB/s | 60 MB 00:01 2024-04-06T12:08:47.177 INFO:teuthology.orchestra.run.smithi189.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 49 MB/s | 60 MB 00:01 2024-04-06T12:09:01.865 INFO:teuthology.orchestra.run.smithi192.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-06T12:09:01.866 INFO:teuthology.orchestra.run.smithi192.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-06T12:09:01.885 INFO:teuthology.orchestra.run.smithi192.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-04-06T12:09:01.887 INFO:teuthology.orchestra.run.smithi192.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-04-06T12:09:01.889 INFO:teuthology.orchestra.run.smithi192.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-04-06T12:09:01.890 INFO:teuthology.orchestra.run.smithi192.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-04-06T12:09:01.893 INFO:teuthology.orchestra.run.smithi192.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-04-06T12:09:01.897 INFO:teuthology.orchestra.run.smithi192.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-04-06T12:09:01.899 INFO:teuthology.orchestra.run.smithi192.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-04-06T12:09:02.111 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-06T12:09:02.125 INFO:teuthology.orchestra.run.smithi192.stdout:=================================================================================================================================== 2024-04-06T12:09:02.125 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-04-06T12:09:02.125 INFO:teuthology.orchestra.run.smithi192.stdout:=================================================================================================================================== 2024-04-06T12:09:02.126 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-06T12:09:02.126 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-04-06T12:09:02.126 INFO:teuthology.orchestra.run.smithi192.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-04-06T12:09:02.126 INFO:teuthology.orchestra.run.smithi192.stdout: ceph x86_64 2:18.2.2-1086.g052adcad.el8 ceph 6.4 k 2024-04-06T12:09:02.126 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base x86_64 2:18.2.2-1086.g052adcad.el8 ceph 5.5 M 2024-04-06T12:09:02.126 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse x86_64 2:18.2.2-1086.g052adcad.el8 ceph 860 k 2024-04-06T12:09:02.126 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1086.g052adcad.el8 ceph 165 k 2024-04-06T12:09:02.126 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr x86_64 2:18.2.2-1086.g052adcad.el8 ceph 1.6 M 2024-04-06T12:09:02.126 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 138 k 2024-04-06T12:09:02.126 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 3.5 M 2024-04-06T12:09:02.127 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 7.4 M 2024-04-06T12:09:02.127 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 49 k 2024-04-06T12:09:02.127 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw x86_64 2:18.2.2-1086.g052adcad.el8 ceph 13 M 2024-04-06T12:09:02.127 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test x86_64 2:18.2.2-1086.g052adcad.el8 ceph 47 M 2024-04-06T12:09:02.127 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 263 k 2024-04-06T12:09:02.127 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 173 k 2024-04-06T12:09:02.127 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-mirror x86_64 2:18.2.2-1086.g052adcad.el8 ceph 261 k 2024-04-06T12:09:02.127 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-top noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 18 k 2024-04-06T12:09:02.127 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-04-06T12:09:02.127 INFO:teuthology.orchestra.run.smithi192.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-04-06T12:09:02.127 INFO:teuthology.orchestra.run.smithi192.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-04-06T12:09:02.127 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel x86_64 2:18.2.2-1086.g052adcad.el8 ceph 31 k 2024-04-06T12:09:02.127 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2 x86_64 2:18.2.2-1086.g052adcad.el8 ceph 781 k 2024-04-06T12:09:02.128 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel x86_64 2:18.2.2-1086.g052adcad.el8 ceph 139 k 2024-04-06T12:09:02.128 INFO:teuthology.orchestra.run.smithi192.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-04-06T12:09:02.128 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-04-06T12:09:02.128 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-04-06T12:09:02.128 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-04-06T12:09:02.128 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-contrib x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 812 k 2024-04-06T12:09:02.128 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-server x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-04-06T12:09:02.128 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs x86_64 2:18.2.2-1086.g052adcad.el8 ceph 196 k 2024-04-06T12:09:02.128 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados x86_64 2:18.2.2-1086.g052adcad.el8 ceph 370 k 2024-04-06T12:09:02.128 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd x86_64 2:18.2.2-1086.g052adcad.el8 ceph 365 k 2024-04-06T12:09:02.128 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw x86_64 2:18.2.2-1086.g052adcad.el8 ceph 107 k 2024-04-06T12:09:02.128 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse x86_64 2:18.2.2-1086.g052adcad.el8 ceph 95 k 2024-04-06T12:09:02.129 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror x86_64 2:18.2.2-1086.g052adcad.el8 ceph 3.5 M 2024-04-06T12:09:02.129 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd x86_64 2:18.2.2-1086.g052adcad.el8 ceph 182 k 2024-04-06T12:09:02.129 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-04-06T12:09:02.129 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrading: 2024-04-06T12:09:02.129 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-04-06T12:09:02.129 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-04-06T12:09:02.129 INFO:teuthology.orchestra.run.smithi192.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-04-06T12:09:02.129 INFO:teuthology.orchestra.run.smithi192.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-04-06T12:09:02.129 INFO:teuthology.orchestra.run.smithi192.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-04-06T12:09:02.129 INFO:teuthology.orchestra.run.smithi192.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-04-06T12:09:02.129 INFO:teuthology.orchestra.run.smithi192.stdout: librados2 x86_64 2:18.2.2-1086.g052adcad.el8 ceph 3.7 M 2024-04-06T12:09:02.130 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1 x86_64 2:18.2.2-1086.g052adcad.el8 ceph 3.6 M 2024-04-06T12:09:02.130 INFO:teuthology.orchestra.run.smithi192.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-04-06T12:09:02.130 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-04-06T12:09:02.130 INFO:teuthology.orchestra.run.smithi192.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-04-06T12:09:02.130 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-04-06T12:09:02.130 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-04-06T12:09:02.130 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-04-06T12:09:02.130 INFO:teuthology.orchestra.run.smithi192.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-04-06T12:09:02.130 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-04-06T12:09:02.130 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-06T12:09:02.130 INFO:teuthology.orchestra.run.smithi192.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-04-06T12:09:02.130 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common x86_64 2:18.2.2-1086.g052adcad.el8 ceph 22 M 2024-04-06T12:09:02.130 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 23 k 2024-04-06T12:09:02.131 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds x86_64 2:18.2.2-1086.g052adcad.el8 ceph 2.3 M 2024-04-06T12:09:02.131 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 245 k 2024-04-06T12:09:02.131 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon x86_64 2:18.2.2-1086.g052adcad.el8 ceph 4.8 M 2024-04-06T12:09:02.131 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd x86_64 2:18.2.2-1086.g052adcad.el8 ceph 18 M 2024-04-06T12:09:02.131 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 15 k 2024-04-06T12:09:02.131 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux x86_64 2:18.2.2-1086.g052adcad.el8 ceph 24 k 2024-04-06T12:09:02.131 INFO:teuthology.orchestra.run.smithi192.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-06T12:09:02.131 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-06T12:09:02.131 INFO:teuthology.orchestra.run.smithi192.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-06T12:09:02.131 INFO:teuthology.orchestra.run.smithi192.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-04-06T12:09:02.131 INFO:teuthology.orchestra.run.smithi192.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-04-06T12:09:02.131 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite x86_64 2:18.2.2-1086.g052adcad.el8 ceph 142 k 2024-04-06T12:09:02.131 INFO:teuthology.orchestra.run.smithi192.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-04-06T12:09:02.131 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-06T12:09:02.132 INFO:teuthology.orchestra.run.smithi192.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-06T12:09:02.132 INFO:teuthology.orchestra.run.smithi192.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-04-06T12:09:02.132 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-06T12:09:02.132 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-06T12:09:02.132 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1 x86_64 2:18.2.2-1086.g052adcad.el8 ceph 475 k 2024-04-06T12:09:02.132 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-06T12:09:02.132 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2 x86_64 2:18.2.2-1086.g052adcad.el8 ceph 6.5 M 2024-04-06T12:09:02.132 INFO:teuthology.orchestra.run.smithi192.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-04-06T12:09:02.132 INFO:teuthology.orchestra.run.smithi192.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-04-06T12:09:02.132 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-06T12:09:02.132 INFO:teuthology.orchestra.run.smithi192.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-04-06T12:09:02.132 INFO:teuthology.orchestra.run.smithi192.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-04-06T12:09:02.132 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-06T12:09:02.133 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-06T12:09:02.133 INFO:teuthology.orchestra.run.smithi192.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-06T12:09:02.133 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-06T12:09:02.133 INFO:teuthology.orchestra.run.smithi192.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-04-06T12:09:02.133 INFO:teuthology.orchestra.run.smithi192.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-04-06T12:09:02.133 INFO:teuthology.orchestra.run.smithi192.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-04-06T12:09:02.133 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-06T12:09:02.133 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-06T12:09:02.133 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-06T12:09:02.133 INFO:teuthology.orchestra.run.smithi192.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-06T12:09:02.133 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-06T12:09:02.133 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse x86_64 2:18.2.2-1086.g052adcad.el8 ceph 45 k 2024-04-06T12:09:02.133 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common x86_64 2:18.2.2-1086.g052adcad.el8 ceph 129 k 2024-04-06T12:09:02.133 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-06T12:09:02.133 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-06T12:09:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-06T12:09:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-06T12:09:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-06T12:09:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-06T12:09:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-06T12:09:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-06T12:09:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-06T12:09:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-06T12:09:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-06T12:09:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-06T12:09:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-06T12:09:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-06T12:09:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-06T12:09:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-06T12:09:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-06T12:09:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-06T12:09:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-06T12:09:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-06T12:09:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-06T12:09:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-04-06T12:09:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-06T12:09:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-06T12:09:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-06T12:09:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-06T12:09:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-06T12:09:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-06T12:09:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-06T12:09:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-06T12:09:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-06T12:09:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-06T12:09:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-06T12:09:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-06T12:09:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-06T12:09:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-06T12:09:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-06T12:09:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-06T12:09:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-06T12:09:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-06T12:09:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-06T12:09:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-06T12:09:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-06T12:09:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-06T12:09:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-06T12:09:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-06T12:09:02.137 INFO:teuthology.orchestra.run.smithi192.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-06T12:09:02.137 INFO:teuthology.orchestra.run.smithi192.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-04-06T12:09:02.137 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-06T12:09:02.137 INFO:teuthology.orchestra.run.smithi192.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-04-06T12:09:02.137 INFO:teuthology.orchestra.run.smithi192.stdout:Enabling module streams: 2024-04-06T12:09:02.137 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql 10 2024-04-06T12:09:02.137 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:09:02.137 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-06T12:09:02.137 INFO:teuthology.orchestra.run.smithi192.stdout:=================================================================================================================================== 2024-04-06T12:09:02.137 INFO:teuthology.orchestra.run.smithi192.stdout:Install 130 Packages 2024-04-06T12:09:02.138 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrade 16 Packages 2024-04-06T12:09:02.138 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:09:02.138 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 215 M 2024-04-06T12:09:02.138 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-06T12:09:02.450 INFO:teuthology.orchestra.run.smithi192.stdout:(1/146): ceph-18.2.2-1086.g052adcad.el8.x86_64. 29 kB/s | 6.4 kB 00:00 2024-04-06T12:09:02.745 INFO:teuthology.orchestra.run.smithi189.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-06T12:09:02.747 INFO:teuthology.orchestra.run.smithi189.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-06T12:09:02.765 INFO:teuthology.orchestra.run.smithi189.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-04-06T12:09:02.767 INFO:teuthology.orchestra.run.smithi189.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-04-06T12:09:02.768 INFO:teuthology.orchestra.run.smithi189.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-04-06T12:09:02.769 INFO:teuthology.orchestra.run.smithi189.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-04-06T12:09:02.772 INFO:teuthology.orchestra.run.smithi189.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-04-06T12:09:02.776 INFO:teuthology.orchestra.run.smithi189.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-04-06T12:09:02.778 INFO:teuthology.orchestra.run.smithi189.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-04-06T12:09:02.991 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-06T12:09:03.005 INFO:teuthology.orchestra.run.smithi189.stdout:=================================================================================================================================== 2024-04-06T12:09:03.005 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-06T12:09:03.005 INFO:teuthology.orchestra.run.smithi189.stdout:=================================================================================================================================== 2024-04-06T12:09:03.005 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-06T12:09:03.005 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-04-06T12:09:03.005 INFO:teuthology.orchestra.run.smithi189.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-04-06T12:09:03.005 INFO:teuthology.orchestra.run.smithi189.stdout: ceph x86_64 2:18.2.2-1086.g052adcad.el8 ceph 6.4 k 2024-04-06T12:09:03.005 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base x86_64 2:18.2.2-1086.g052adcad.el8 ceph 5.5 M 2024-04-06T12:09:03.005 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse x86_64 2:18.2.2-1086.g052adcad.el8 ceph 860 k 2024-04-06T12:09:03.005 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1086.g052adcad.el8 ceph 165 k 2024-04-06T12:09:03.005 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr x86_64 2:18.2.2-1086.g052adcad.el8 ceph 1.6 M 2024-04-06T12:09:03.006 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 138 k 2024-04-06T12:09:03.006 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 3.5 M 2024-04-06T12:09:03.006 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 7.4 M 2024-04-06T12:09:03.006 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 49 k 2024-04-06T12:09:03.006 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw x86_64 2:18.2.2-1086.g052adcad.el8 ceph 13 M 2024-04-06T12:09:03.006 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test x86_64 2:18.2.2-1086.g052adcad.el8 ceph 47 M 2024-04-06T12:09:03.006 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 263 k 2024-04-06T12:09:03.006 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 173 k 2024-04-06T12:09:03.006 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-mirror x86_64 2:18.2.2-1086.g052adcad.el8 ceph 261 k 2024-04-06T12:09:03.006 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-top noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 18 k 2024-04-06T12:09:03.006 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-04-06T12:09:03.006 INFO:teuthology.orchestra.run.smithi189.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-04-06T12:09:03.006 INFO:teuthology.orchestra.run.smithi189.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-04-06T12:09:03.006 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel x86_64 2:18.2.2-1086.g052adcad.el8 ceph 31 k 2024-04-06T12:09:03.007 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2 x86_64 2:18.2.2-1086.g052adcad.el8 ceph 781 k 2024-04-06T12:09:03.007 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel x86_64 2:18.2.2-1086.g052adcad.el8 ceph 139 k 2024-04-06T12:09:03.007 INFO:teuthology.orchestra.run.smithi189.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-04-06T12:09:03.007 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-04-06T12:09:03.007 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-04-06T12:09:03.007 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-04-06T12:09:03.007 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-contrib x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 812 k 2024-04-06T12:09:03.007 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-server x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-04-06T12:09:03.007 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs x86_64 2:18.2.2-1086.g052adcad.el8 ceph 196 k 2024-04-06T12:09:03.007 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados x86_64 2:18.2.2-1086.g052adcad.el8 ceph 370 k 2024-04-06T12:09:03.007 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd x86_64 2:18.2.2-1086.g052adcad.el8 ceph 365 k 2024-04-06T12:09:03.007 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw x86_64 2:18.2.2-1086.g052adcad.el8 ceph 107 k 2024-04-06T12:09:03.007 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse x86_64 2:18.2.2-1086.g052adcad.el8 ceph 95 k 2024-04-06T12:09:03.007 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror x86_64 2:18.2.2-1086.g052adcad.el8 ceph 3.5 M 2024-04-06T12:09:03.008 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd x86_64 2:18.2.2-1086.g052adcad.el8 ceph 182 k 2024-04-06T12:09:03.008 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-04-06T12:09:03.008 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrading: 2024-04-06T12:09:03.008 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-04-06T12:09:03.008 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-04-06T12:09:03.008 INFO:teuthology.orchestra.run.smithi189.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-04-06T12:09:03.008 INFO:teuthology.orchestra.run.smithi189.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-04-06T12:09:03.008 INFO:teuthology.orchestra.run.smithi189.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-04-06T12:09:03.008 INFO:teuthology.orchestra.run.smithi189.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-04-06T12:09:03.008 INFO:teuthology.orchestra.run.smithi189.stdout: librados2 x86_64 2:18.2.2-1086.g052adcad.el8 ceph 3.7 M 2024-04-06T12:09:03.008 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1 x86_64 2:18.2.2-1086.g052adcad.el8 ceph 3.6 M 2024-04-06T12:09:03.008 INFO:teuthology.orchestra.run.smithi189.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-04-06T12:09:03.008 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-04-06T12:09:03.008 INFO:teuthology.orchestra.run.smithi189.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-04-06T12:09:03.009 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-04-06T12:09:03.009 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-04-06T12:09:03.009 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-04-06T12:09:03.009 INFO:teuthology.orchestra.run.smithi189.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-04-06T12:09:03.009 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-04-06T12:09:03.009 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-06T12:09:03.009 INFO:teuthology.orchestra.run.smithi189.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-04-06T12:09:03.009 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common x86_64 2:18.2.2-1086.g052adcad.el8 ceph 22 M 2024-04-06T12:09:03.009 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 23 k 2024-04-06T12:09:03.009 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds x86_64 2:18.2.2-1086.g052adcad.el8 ceph 2.3 M 2024-04-06T12:09:03.009 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 245 k 2024-04-06T12:09:03.009 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon x86_64 2:18.2.2-1086.g052adcad.el8 ceph 4.8 M 2024-04-06T12:09:03.009 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd x86_64 2:18.2.2-1086.g052adcad.el8 ceph 18 M 2024-04-06T12:09:03.009 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1086.g052adcad.el8 ceph-noarch 15 k 2024-04-06T12:09:03.010 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux x86_64 2:18.2.2-1086.g052adcad.el8 ceph 24 k 2024-04-06T12:09:03.010 INFO:teuthology.orchestra.run.smithi189.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-06T12:09:03.010 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-06T12:09:03.010 INFO:teuthology.orchestra.run.smithi189.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-06T12:09:03.010 INFO:teuthology.orchestra.run.smithi189.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-04-06T12:09:03.010 INFO:teuthology.orchestra.run.smithi189.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-04-06T12:09:03.010 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite x86_64 2:18.2.2-1086.g052adcad.el8 ceph 142 k 2024-04-06T12:09:03.010 INFO:teuthology.orchestra.run.smithi189.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-04-06T12:09:03.010 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-06T12:09:03.010 INFO:teuthology.orchestra.run.smithi189.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-06T12:09:03.010 INFO:teuthology.orchestra.run.smithi189.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-04-06T12:09:03.010 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-06T12:09:03.010 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-06T12:09:03.010 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1 x86_64 2:18.2.2-1086.g052adcad.el8 ceph 475 k 2024-04-06T12:09:03.011 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-06T12:09:03.011 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2 x86_64 2:18.2.2-1086.g052adcad.el8 ceph 6.5 M 2024-04-06T12:09:03.011 INFO:teuthology.orchestra.run.smithi189.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-04-06T12:09:03.011 INFO:teuthology.orchestra.run.smithi189.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-04-06T12:09:03.011 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-06T12:09:03.011 INFO:teuthology.orchestra.run.smithi189.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-04-06T12:09:03.011 INFO:teuthology.orchestra.run.smithi189.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-04-06T12:09:03.011 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-06T12:09:03.011 INFO:teuthology.orchestra.run.smithi189.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-06T12:09:03.011 INFO:teuthology.orchestra.run.smithi189.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-06T12:09:03.011 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-06T12:09:03.011 INFO:teuthology.orchestra.run.smithi189.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-04-06T12:09:03.011 INFO:teuthology.orchestra.run.smithi189.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-04-06T12:09:03.011 INFO:teuthology.orchestra.run.smithi189.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-04-06T12:09:03.011 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-06T12:09:03.012 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-06T12:09:03.012 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-06T12:09:03.012 INFO:teuthology.orchestra.run.smithi189.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-06T12:09:03.012 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-06T12:09:03.012 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse x86_64 2:18.2.2-1086.g052adcad.el8 ceph 45 k 2024-04-06T12:09:03.012 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common x86_64 2:18.2.2-1086.g052adcad.el8 ceph 129 k 2024-04-06T12:09:03.012 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-06T12:09:03.012 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-06T12:09:03.012 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-06T12:09:03.012 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-06T12:09:03.012 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-06T12:09:03.012 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-06T12:09:03.012 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-06T12:09:03.012 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-06T12:09:03.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-06T12:09:03.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-06T12:09:03.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-06T12:09:03.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-06T12:09:03.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-06T12:09:03.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-06T12:09:03.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-06T12:09:03.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-06T12:09:03.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-06T12:09:03.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-06T12:09:03.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-06T12:09:03.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-06T12:09:03.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-06T12:09:03.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-04-06T12:09:03.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-06T12:09:03.014 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-06T12:09:03.014 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-06T12:09:03.014 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-06T12:09:03.014 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-06T12:09:03.014 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-06T12:09:03.014 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-06T12:09:03.014 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-06T12:09:03.014 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-06T12:09:03.014 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-06T12:09:03.014 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-06T12:09:03.014 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-06T12:09:03.014 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-06T12:09:03.014 INFO:teuthology.orchestra.run.smithi189.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-06T12:09:03.015 INFO:teuthology.orchestra.run.smithi189.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-06T12:09:03.015 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-06T12:09:03.015 INFO:teuthology.orchestra.run.smithi189.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-06T12:09:03.015 INFO:teuthology.orchestra.run.smithi189.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-06T12:09:03.015 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-06T12:09:03.015 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-06T12:09:03.015 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-06T12:09:03.015 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-06T12:09:03.015 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-06T12:09:03.015 INFO:teuthology.orchestra.run.smithi189.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-06T12:09:03.015 INFO:teuthology.orchestra.run.smithi189.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-06T12:09:03.015 INFO:teuthology.orchestra.run.smithi189.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-04-06T12:09:03.015 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-06T12:09:03.015 INFO:teuthology.orchestra.run.smithi189.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-04-06T12:09:03.015 INFO:teuthology.orchestra.run.smithi189.stdout:Enabling module streams: 2024-04-06T12:09:03.016 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql 10 2024-04-06T12:09:03.016 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:09:03.016 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-06T12:09:03.016 INFO:teuthology.orchestra.run.smithi189.stdout:=================================================================================================================================== 2024-04-06T12:09:03.016 INFO:teuthology.orchestra.run.smithi189.stdout:Install 130 Packages 2024-04-06T12:09:03.016 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrade 16 Packages 2024-04-06T12:09:03.016 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:09:03.017 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 215 M 2024-04-06T12:09:03.017 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-06T12:09:03.051 INFO:teuthology.orchestra.run.smithi192.stdout:(2/146): ceph-fuse-18.2.2-1086.g052adcad.el8.x8 1.4 MB/s | 860 kB 00:00 2024-04-06T12:09:03.184 INFO:teuthology.orchestra.run.smithi192.stdout:(3/146): ceph-base-18.2.2-1086.g052adcad.el8.x8 5.8 MB/s | 5.5 MB 00:00 2024-04-06T12:09:03.226 INFO:teuthology.orchestra.run.smithi192.stdout:(4/146): ceph-immutable-object-cache-18.2.2-108 946 kB/s | 165 kB 00:00 2024-04-06T12:09:03.326 INFO:teuthology.orchestra.run.smithi192.stdout:(5/146): ceph-mds-18.2.2-1086.g052adcad.el8.x86 16 MB/s | 2.3 MB 00:00 2024-04-06T12:09:03.609 INFO:teuthology.orchestra.run.smithi189.stdout:(1/146): ceph-18.2.2-1086.g052adcad.el8.x86_64. 22 kB/s | 6.4 kB 00:00 2024-04-06T12:09:03.618 INFO:teuthology.orchestra.run.smithi192.stdout:(6/146): ceph-mon-18.2.2-1086.g052adcad.el8.x86 17 MB/s | 4.8 MB 00:00 2024-04-06T12:09:04.110 INFO:teuthology.orchestra.run.smithi192.stdout:(7/146): ceph-mgr-18.2.2-1086.g052adcad.el8.x86 1.8 MB/s | 1.6 MB 00:00 2024-04-06T12:09:04.576 INFO:teuthology.orchestra.run.smithi189.stdout:(2/146): ceph-fuse-18.2.2-1086.g052adcad.el8.x8 889 kB/s | 860 kB 00:00 2024-04-06T12:09:04.785 INFO:teuthology.orchestra.run.smithi189.stdout:(3/146): ceph-immutable-object-cache-18.2.2-108 795 kB/s | 165 kB 00:00 2024-04-06T12:09:05.002 INFO:teuthology.orchestra.run.smithi192.stdout:(8/146): ceph-osd-18.2.2-1086.g052adcad.el8.x86 13 MB/s | 18 MB 00:01 2024-04-06T12:09:05.178 INFO:teuthology.orchestra.run.smithi192.stdout:(9/146): ceph-selinux-18.2.2-1086.g052adcad.el8 139 kB/s | 24 kB 00:00 2024-04-06T12:09:05.760 INFO:teuthology.orchestra.run.smithi189.stdout:(4/146): ceph-mds-18.2.2-1086.g052adcad.el8.x86 2.3 MB/s | 2.3 MB 00:00 2024-04-06T12:09:05.835 INFO:teuthology.orchestra.run.smithi189.stdout:(5/146): ceph-base-18.2.2-1086.g052adcad.el8.x8 2.2 MB/s | 5.5 MB 00:02 2024-04-06T12:09:06.502 INFO:teuthology.orchestra.run.smithi189.stdout:(6/146): ceph-mgr-18.2.2-1086.g052adcad.el8.x86 2.1 MB/s | 1.6 MB 00:00 2024-04-06T12:09:06.835 INFO:teuthology.orchestra.run.smithi189.stdout:(7/146): ceph-mon-18.2.2-1086.g052adcad.el8.x86 4.8 MB/s | 4.8 MB 00:01 2024-04-06T12:09:06.870 INFO:teuthology.orchestra.run.smithi192.stdout:(10/146): ceph-common-18.2.2-1086.g052adcad.el8 4.7 MB/s | 22 MB 00:04 2024-04-06T12:09:07.045 INFO:teuthology.orchestra.run.smithi192.stdout:(11/146): cephfs-mirror-18.2.2-1086.g052adcad.e 1.5 MB/s | 261 kB 00:00 2024-04-06T12:09:07.120 INFO:teuthology.orchestra.run.smithi192.stdout:(12/146): libcephfs-devel-18.2.2-1086.g052adcad 413 kB/s | 31 kB 00:00 2024-04-06T12:09:07.236 INFO:teuthology.orchestra.run.smithi189.stdout:(8/146): ceph-common-18.2.2-1086.g052adcad.el8. 5.5 MB/s | 22 MB 00:03 2024-04-06T12:09:07.278 INFO:teuthology.orchestra.run.smithi192.stdout:(13/146): libcephfs2-18.2.2-1086.g052adcad.el8. 4.8 MB/s | 781 kB 00:00 2024-04-06T12:09:07.321 INFO:teuthology.orchestra.run.smithi189.stdout:(9/146): ceph-selinux-18.2.2-1086.g052adcad.el8 285 kB/s | 24 kB 00:00 2024-04-06T12:09:07.379 INFO:teuthology.orchestra.run.smithi192.stdout:(14/146): libcephsqlite-18.2.2-1086.g052adcad.e 1.4 MB/s | 142 kB 00:00 2024-04-06T12:09:07.479 INFO:teuthology.orchestra.run.smithi192.stdout:(15/146): librados-devel-18.2.2-1086.g052adcad. 1.4 MB/s | 139 kB 00:00 2024-04-06T12:09:07.688 INFO:teuthology.orchestra.run.smithi192.stdout:(16/146): libradosstriper1-18.2.2-1086.g052adca 2.2 MB/s | 475 kB 00:00 2024-04-06T12:09:08.280 INFO:teuthology.orchestra.run.smithi192.stdout:(17/146): ceph-radosgw-18.2.2-1086.g052adcad.el 3.1 MB/s | 13 MB 00:04 2024-04-06T12:09:08.372 INFO:teuthology.orchestra.run.smithi192.stdout:(18/146): python3-ceph-argparse-18.2.2-1086.g05 489 kB/s | 45 kB 00:00 2024-04-06T12:09:08.382 INFO:teuthology.orchestra.run.smithi189.stdout:(10/146): ceph-radosgw-18.2.2-1086.g052adcad.el 8.5 MB/s | 13 MB 00:01 2024-04-06T12:09:08.464 INFO:teuthology.orchestra.run.smithi192.stdout:(19/146): python3-ceph-common-18.2.2-1086.g052a 1.4 MB/s | 129 kB 00:00 2024-04-06T12:09:08.480 INFO:teuthology.orchestra.run.smithi189.stdout:(11/146): cephfs-mirror-18.2.2-1086.g052adcad.e 2.6 MB/s | 261 kB 00:00 2024-04-06T12:09:08.606 INFO:teuthology.orchestra.run.smithi192.stdout:(20/146): python3-cephfs-18.2.2-1086.g052adcad. 1.4 MB/s | 196 kB 00:00 2024-04-06T12:09:08.622 INFO:teuthology.orchestra.run.smithi189.stdout:(12/146): libcephfs-devel-18.2.2-1086.g052adcad 218 kB/s | 31 kB 00:00 2024-04-06T12:09:08.798 INFO:teuthology.orchestra.run.smithi192.stdout:(21/146): librgw2-18.2.2-1086.g052adcad.el8.x86 5.9 MB/s | 6.5 MB 00:01 2024-04-06T12:09:08.820 INFO:teuthology.orchestra.run.smithi189.stdout:(13/146): libcephfs2-18.2.2-1086.g052adcad.el8. 3.9 MB/s | 781 kB 00:00 2024-04-06T12:09:08.823 INFO:teuthology.orchestra.run.smithi192.stdout:(22/146): python3-rados-18.2.2-1086.g052adcad.e 1.7 MB/s | 370 kB 00:00 2024-04-06T12:09:08.865 INFO:teuthology.orchestra.run.smithi192.stdout:(23/146): python3-rbd-18.2.2-1086.g052adcad.el8 5.3 MB/s | 365 kB 00:00 2024-04-06T12:09:08.906 INFO:teuthology.orchestra.run.smithi189.stdout:(14/146): libcephsqlite-18.2.2-1086.g052adcad.e 1.6 MB/s | 142 kB 00:00 2024-04-06T12:09:08.907 INFO:teuthology.orchestra.run.smithi192.stdout:(24/146): python3-rgw-18.2.2-1086.g052adcad.el8 1.2 MB/s | 107 kB 00:00 2024-04-06T12:09:08.949 INFO:teuthology.orchestra.run.smithi192.stdout:(25/146): rbd-fuse-18.2.2-1086.g052adcad.el8.x8 1.1 MB/s | 95 kB 00:00 2024-04-06T12:09:08.989 INFO:teuthology.orchestra.run.smithi189.stdout:(15/146): librados-devel-18.2.2-1086.g052adcad. 1.6 MB/s | 139 kB 00:00 2024-04-06T12:09:09.066 INFO:teuthology.orchestra.run.smithi192.stdout:(26/146): rbd-nbd-18.2.2-1086.g052adcad.el8.x86 1.5 MB/s | 182 kB 00:00 2024-04-06T12:09:09.098 INFO:teuthology.orchestra.run.smithi189.stdout:(16/146): libradosstriper1-18.2.2-1086.g052adca 4.3 MB/s | 475 kB 00:00 2024-04-06T12:09:09.141 INFO:teuthology.orchestra.run.smithi192.stdout:(27/146): ceph-grafana-dashboards-18.2.2-1086.g 311 kB/s | 23 kB 00:00 2024-04-06T12:09:09.250 INFO:teuthology.orchestra.run.smithi192.stdout:(28/146): ceph-mgr-cephadm-18.2.2-1086.g052adca 1.2 MB/s | 138 kB 00:00 2024-04-06T12:09:09.981 INFO:teuthology.orchestra.run.smithi189.stdout:(17/146): librgw2-18.2.2-1086.g052adcad.el8.x86 7.4 MB/s | 6.5 MB 00:00 2024-04-06T12:09:10.017 INFO:teuthology.orchestra.run.smithi192.stdout:(29/146): ceph-mgr-dashboard-18.2.2-1086.g052ad 4.6 MB/s | 3.5 MB 00:00 2024-04-06T12:09:10.057 INFO:teuthology.orchestra.run.smithi189.stdout:(18/146): python3-ceph-argparse-18.2.2-1086.g05 596 kB/s | 45 kB 00:00 2024-04-06T12:09:10.157 INFO:teuthology.orchestra.run.smithi189.stdout:(19/146): python3-ceph-common-18.2.2-1086.g052a 1.3 MB/s | 129 kB 00:00 2024-04-06T12:09:10.257 INFO:teuthology.orchestra.run.smithi189.stdout:(20/146): python3-cephfs-18.2.2-1086.g052adcad. 1.9 MB/s | 196 kB 00:00 2024-04-06T12:09:10.366 INFO:teuthology.orchestra.run.smithi189.stdout:(21/146): python3-rados-18.2.2-1086.g052adcad.e 3.3 MB/s | 370 kB 00:00 2024-04-06T12:09:10.367 INFO:teuthology.orchestra.run.smithi192.stdout:(30/146): rbd-mirror-18.2.2-1086.g052adcad.el8. 2.4 MB/s | 3.5 MB 00:01 2024-04-06T12:09:10.474 INFO:teuthology.orchestra.run.smithi189.stdout:(22/146): python3-rbd-18.2.2-1086.g052adcad.el8 3.3 MB/s | 365 kB 00:00 2024-04-06T12:09:10.534 INFO:teuthology.orchestra.run.smithi192.stdout:(31/146): ceph-mgr-modules-core-18.2.2-1086.g05 1.4 MB/s | 245 kB 00:00 2024-04-06T12:09:10.566 INFO:teuthology.orchestra.run.smithi189.stdout:(23/146): python3-rgw-18.2.2-1086.g052adcad.el8 1.1 MB/s | 107 kB 00:00 2024-04-06T12:09:10.634 INFO:teuthology.orchestra.run.smithi192.stdout:(32/146): ceph-mgr-rook-18.2.2-1086.g052adcad.e 493 kB/s | 49 kB 00:00 2024-04-06T12:09:10.658 INFO:teuthology.orchestra.run.smithi189.stdout:(24/146): rbd-fuse-18.2.2-1086.g052adcad.el8.x8 1.0 MB/s | 95 kB 00:00 2024-04-06T12:09:10.726 INFO:teuthology.orchestra.run.smithi192.stdout:(33/146): ceph-prometheus-alerts-18.2.2-1086.g0 161 kB/s | 15 kB 00:00 2024-04-06T12:09:11.150 INFO:teuthology.orchestra.run.smithi189.stdout:(25/146): rbd-mirror-18.2.2-1086.g052adcad.el8. 7.0 MB/s | 3.5 MB 00:00 2024-04-06T12:09:11.193 INFO:teuthology.orchestra.run.smithi192.stdout:(34/146): ceph-volume-18.2.2-1086.g052adcad.el8 563 kB/s | 263 kB 00:00 2024-04-06T12:09:11.250 INFO:teuthology.orchestra.run.smithi189.stdout:(26/146): rbd-nbd-18.2.2-1086.g052adcad.el8.x86 1.8 MB/s | 182 kB 00:00 2024-04-06T12:09:11.277 INFO:teuthology.orchestra.run.smithi192.stdout:(35/146): ceph-mgr-diskprediction-local-18.2.2- 5.9 MB/s | 7.4 MB 00:01 2024-04-06T12:09:11.294 INFO:teuthology.orchestra.run.smithi192.stdout:(36/146): cephadm-18.2.2-1086.g052adcad.el8.noa 1.7 MB/s | 173 kB 00:00 2024-04-06T12:09:11.317 INFO:teuthology.orchestra.run.smithi189.stdout:(27/146): ceph-grafana-dashboards-18.2.2-1086.g 350 kB/s | 23 kB 00:00 2024-04-06T12:09:11.361 INFO:teuthology.orchestra.run.smithi192.stdout:(37/146): cephfs-top-18.2.2-1086.g052adcad.el8. 217 kB/s | 18 kB 00:00 2024-04-06T12:09:11.392 INFO:teuthology.orchestra.run.smithi189.stdout:(28/146): ceph-mgr-cephadm-18.2.2-1086.g052adca 1.8 MB/s | 138 kB 00:00 2024-04-06T12:09:11.478 INFO:teuthology.orchestra.run.smithi192.stdout:(38/146): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2024-04-06T12:09:11.503 INFO:teuthology.orchestra.run.smithi192.stdout:(39/146): liboath-2.6.2-3.el8.x86_64.rpm 415 kB/s | 59 kB 00:00 2024-04-06T12:09:11.537 INFO:teuthology.orchestra.run.smithi192.stdout:(40/146): libunwind-1.3.1-3.el8.x86_64.rpm 2.2 MB/s | 75 kB 00:00 2024-04-06T12:09:11.578 INFO:teuthology.orchestra.run.smithi192.stdout:(41/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.0 MB/s | 44 kB 00:00 2024-04-06T12:09:11.629 INFO:teuthology.orchestra.run.smithi192.stdout:(42/146): python3-asyncssh-2.7.0-2.el8.noarch.r 3.7 MB/s | 476 kB 00:00 2024-04-06T12:09:11.654 INFO:teuthology.orchestra.run.smithi192.stdout:(43/146): python3-beautifulsoup4-4.6.3-2.el8.1. 2.4 MB/s | 185 kB 00:00 2024-04-06T12:09:11.680 INFO:teuthology.orchestra.run.smithi192.stdout:(44/146): python3-cachetools-3.1.1-4.el8.noarch 654 kB/s | 33 kB 00:00 2024-04-06T12:09:11.705 INFO:teuthology.orchestra.run.smithi192.stdout:(45/146): python3-certifi-2018.10.15-7.el8.noar 325 kB/s | 16 kB 00:00 2024-04-06T12:09:11.738 INFO:teuthology.orchestra.run.smithi192.stdout:(46/146): python3-cheroot-8.5.2-1.el8.noarch.rp 2.9 MB/s | 173 kB 00:00 2024-04-06T12:09:11.801 INFO:teuthology.orchestra.run.smithi189.stdout:(29/146): ceph-mgr-dashboard-18.2.2-1086.g052ad 8.6 MB/s | 3.5 MB 00:00 2024-04-06T12:09:11.812 INFO:teuthology.orchestra.run.smithi192.stdout:(47/146): python3-google-auth-1.1.1-10.el8.noar 1.1 MB/s | 81 kB 00:00 2024-04-06T12:09:11.850 INFO:teuthology.orchestra.run.smithi192.stdout:(48/146): python3-cherrypy-18.4.0-1.el8.noarch. 2.6 MB/s | 384 kB 00:00 2024-04-06T12:09:11.864 INFO:teuthology.orchestra.run.smithi192.stdout:(49/146): python3-jaraco-6.2-6.el8.noarch.rpm 215 kB/s | 11 kB 00:00 2024-04-06T12:09:11.881 INFO:teuthology.orchestra.run.smithi192.stdout:(50/146): python3-jaraco-functools-2.0-4.el8.no 526 kB/s | 18 kB 00:00 2024-04-06T12:09:11.915 INFO:teuthology.orchestra.run.smithi192.stdout:(51/146): python3-logutils-0.3.5-11.el8.noarch. 1.4 MB/s | 49 kB 00:00 2024-04-06T12:09:11.949 INFO:teuthology.orchestra.run.smithi192.stdout:(52/146): python3-more-itertools-7.2.0-3.el8.no 1.7 MB/s | 59 kB 00:00 2024-04-06T12:09:11.982 INFO:teuthology.orchestra.run.smithi192.stdout:(53/146): python3-natsort-7.1.1-2.el8.noarch.rp 1.8 MB/s | 60 kB 00:00 2024-04-06T12:09:12.074 INFO:teuthology.orchestra.run.smithi192.stdout:(54/146): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2024-04-06T12:09:12.108 INFO:teuthology.orchestra.run.smithi192.stdout:(55/146): python3-portend-2.6-1.el8.noarch.rpm 470 kB/s | 16 kB 00:00 2024-04-06T12:09:12.167 INFO:teuthology.orchestra.run.smithi192.stdout:(56/146): python3-repoze-lru-0.7-6.el8.noarch.r 577 kB/s | 34 kB 00:00 2024-04-06T12:09:12.200 INFO:teuthology.orchestra.run.smithi192.stdout:(57/146): python3-kubernetes-11.0.0-6.el8.noarc 3.5 MB/s | 1.2 MB 00:00 2024-04-06T12:09:12.251 INFO:teuthology.orchestra.run.smithi192.stdout:(58/146): python3-routes-2.4.1-12.el8.noarch.rp 2.3 MB/s | 196 kB 00:00 2024-04-06T12:09:12.276 INFO:teuthology.orchestra.run.smithi192.stdout:(59/146): python3-rsa-4.9-2.el8.noarch.rpm 856 kB/s | 64 kB 00:00 2024-04-06T12:09:12.318 INFO:teuthology.orchestra.run.smithi192.stdout:(60/146): python3-simplegeneric-0.8.1-17.el8.no 451 kB/s | 19 kB 00:00 2024-04-06T12:09:12.343 INFO:teuthology.orchestra.run.smithi192.stdout:(61/146): python3-singledispatch-3.4.0.3-18.el8 359 kB/s | 24 kB 00:00 2024-04-06T12:09:12.368 INFO:teuthology.orchestra.run.smithi192.stdout:(62/146): python3-tempora-1.14.1-5.el8.noarch.r 581 kB/s | 29 kB 00:00 2024-04-06T12:09:12.394 INFO:teuthology.orchestra.run.smithi192.stdout:(63/146): python3-trustme-0.6.0-4.el8.noarch.rp 548 kB/s | 27 kB 00:00 2024-04-06T12:09:12.426 INFO:teuthology.orchestra.run.smithi189.stdout:(30/146): ceph-test-18.2.2-1086.g052adcad.el8.x 9.2 MB/s | 47 MB 00:05 2024-04-06T12:09:12.444 INFO:teuthology.orchestra.run.smithi192.stdout:(64/146): python3-waitress-1.4.3-1.el8.noarch.r 3.3 MB/s | 255 kB 00:00 2024-04-06T12:09:12.462 INFO:teuthology.orchestra.run.smithi192.stdout:(65/146): python3-webob-1.8.5-1.el8.1.noarch.rp 3.7 MB/s | 251 kB 00:00 2024-04-06T12:09:12.493 INFO:teuthology.orchestra.run.smithi189.stdout:(31/146): ceph-mgr-modules-core-18.2.2-1086.g05 3.6 MB/s | 245 kB 00:00 2024-04-06T12:09:12.519 INFO:teuthology.orchestra.run.smithi192.stdout:(66/146): python3-websocket-client-0.56.0-5.el8 815 kB/s | 61 kB 00:00 2024-04-06T12:09:12.545 INFO:teuthology.orchestra.run.smithi192.stdout:(67/146): python3-webtest-2.0.33-1.el8.noarch.r 1.0 MB/s | 85 kB 00:00 2024-04-06T12:09:12.568 INFO:teuthology.orchestra.run.smithi189.stdout:(32/146): ceph-mgr-rook-18.2.2-1086.g052adcad.e 659 kB/s | 49 kB 00:00 2024-04-06T12:09:12.570 INFO:teuthology.orchestra.run.smithi192.stdout:(68/146): python3-zc-lockfile-2.0-2.el8.noarch. 456 kB/s | 23 kB 00:00 2024-04-06T12:09:12.587 INFO:teuthology.orchestra.run.smithi192.stdout:(69/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 4.2 MB/s | 70 kB 00:00 2024-04-06T12:09:12.669 INFO:teuthology.orchestra.run.smithi189.stdout:(33/146): ceph-mgr-diskprediction-local-18.2.2- 8.5 MB/s | 7.4 MB 00:00 2024-04-06T12:09:12.671 INFO:teuthology.orchestra.run.smithi192.stdout:(70/146): python3-packaging-16.8-9.el8.noarch.r 576 kB/s | 48 kB 00:00 2024-04-06T12:09:12.694 INFO:teuthology.orchestra.run.smithi189.stdout:(34/146): ceph-prometheus-alerts-18.2.2-1086.g0 118 kB/s | 15 kB 00:00 2024-04-06T12:09:12.736 INFO:teuthology.orchestra.run.smithi189.stdout:(35/146): ceph-volume-18.2.2-1086.g052adcad.el8 3.9 MB/s | 263 kB 00:00 2024-04-06T12:09:12.746 INFO:teuthology.orchestra.run.smithi192.stdout:(71/146): libverto-devel-0.3.0-5.el8.x86_64.rpm 239 kB/s | 18 kB 00:00 2024-04-06T12:09:12.761 INFO:teuthology.orchestra.run.smithi189.stdout:(36/146): cephadm-18.2.2-1086.g052adcad.el8.noa 2.5 MB/s | 173 kB 00:00 2024-04-06T12:09:12.820 INFO:teuthology.orchestra.run.smithi189.stdout:(37/146): cephfs-top-18.2.2-1086.g052adcad.el8. 217 kB/s | 18 kB 00:00 2024-04-06T12:09:12.897 INFO:teuthology.orchestra.run.smithi192.stdout:(72/146): python3-jwt-1.6.1-2.el8.noarch.rpm 284 kB/s | 43 kB 00:00 2024-04-06T12:09:12.955 INFO:teuthology.orchestra.run.smithi192.stdout:(73/146): thrift-0.13.0-2.el8.x86_64.rpm 4.2 MB/s | 1.7 MB 00:00 2024-04-06T12:09:12.970 INFO:teuthology.orchestra.run.smithi189.stdout:(38/146): liboath-2.6.2-3.el8.x86_64.rpm 392 kB/s | 59 kB 00:00 2024-04-06T12:09:13.022 INFO:teuthology.orchestra.run.smithi192.stdout:(74/146): python3-oauthlib-2.1.0-1.el8.noarch.r 1.2 MB/s | 155 kB 00:00 2024-04-06T12:09:13.037 INFO:teuthology.orchestra.run.smithi189.stdout:(39/146): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2024-04-06T12:09:13.062 INFO:teuthology.orchestra.run.smithi189.stdout:(40/146): libunwind-1.3.1-3.el8.x86_64.rpm 816 kB/s | 75 kB 00:00 2024-04-06T12:09:13.089 INFO:teuthology.orchestra.run.smithi192.stdout:(75/146): python3-pycparser-2.14-14.el8.noarch. 817 kB/s | 109 kB 00:00 2024-04-06T12:09:13.121 INFO:teuthology.orchestra.run.smithi189.stdout:(41/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 748 kB/s | 44 kB 00:00 2024-04-06T12:09:13.139 INFO:teuthology.orchestra.run.smithi192.stdout:(76/146): python3-requests-oauthlib-1.0.0-1.el8 365 kB/s | 43 kB 00:00 2024-04-06T12:09:13.146 INFO:teuthology.orchestra.run.smithi189.stdout:(42/146): python3-asyncssh-2.7.0-2.el8.noarch.r 4.3 MB/s | 476 kB 00:00 2024-04-06T12:09:13.222 INFO:teuthology.orchestra.run.smithi189.stdout:(43/146): python3-cachetools-3.1.1-4.el8.noarch 437 kB/s | 33 kB 00:00 2024-04-06T12:09:13.255 INFO:teuthology.orchestra.run.smithi189.stdout:(44/146): python3-beautifulsoup4-4.6.3-2.el8.1. 1.4 MB/s | 185 kB 00:00 2024-04-06T12:09:13.281 INFO:teuthology.orchestra.run.smithi189.stdout:(45/146): python3-certifi-2018.10.15-7.el8.noar 281 kB/s | 16 kB 00:00 2024-04-06T12:09:13.323 INFO:teuthology.orchestra.run.smithi189.stdout:(46/146): python3-cheroot-8.5.2-1.el8.noarch.rp 2.5 MB/s | 173 kB 00:00 2024-04-06T12:09:13.348 INFO:teuthology.orchestra.run.smithi189.stdout:(47/146): python3-cherrypy-18.4.0-1.el8.noarch. 5.6 MB/s | 384 kB 00:00 2024-04-06T12:09:13.398 INFO:teuthology.orchestra.run.smithi189.stdout:(48/146): python3-google-auth-1.1.1-10.el8.noar 1.1 MB/s | 81 kB 00:00 2024-04-06T12:09:13.415 INFO:teuthology.orchestra.run.smithi189.stdout:(49/146): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2024-04-06T12:09:13.449 INFO:teuthology.orchestra.run.smithi189.stdout:(50/146): python3-jaraco-functools-2.0-4.el8.no 350 kB/s | 18 kB 00:00 2024-04-06T12:09:13.499 INFO:teuthology.orchestra.run.smithi189.stdout:(51/146): python3-logutils-0.3.5-11.el8.noarch. 980 kB/s | 49 kB 00:00 2024-04-06T12:09:13.541 INFO:teuthology.orchestra.run.smithi189.stdout:(52/146): python3-kubernetes-11.0.0-6.el8.noarc 9.3 MB/s | 1.2 MB 00:00 2024-04-06T12:09:13.566 INFO:teuthology.orchestra.run.smithi189.stdout:(53/146): python3-more-itertools-7.2.0-3.el8.no 877 kB/s | 59 kB 00:00 2024-04-06T12:09:13.570 INFO:teuthology.orchestra.run.smithi192.stdout:(77/146): ceph-test-18.2.2-1086.g052adcad.el8.x 5.6 MB/s | 47 MB 00:08 2024-04-06T12:09:13.598 INFO:teuthology.orchestra.run.smithi192.stdout:(78/146): libaio-devel-0.3.112-1.el8.x86_64.rpm 37 kB/s | 19 kB 00:00 2024-04-06T12:09:13.624 INFO:teuthology.orchestra.run.smithi192.stdout:(79/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 91 kB/s | 44 kB 00:00 2024-04-06T12:09:13.625 INFO:teuthology.orchestra.run.smithi189.stdout:(54/146): python3-natsort-7.1.1-2.el8.noarch.rp 721 kB/s | 60 kB 00:00 2024-04-06T12:09:13.724 INFO:teuthology.orchestra.run.smithi192.stdout:(80/146): pcre2-devel-10.32-2.el8.x86_64.rpm 4.7 MB/s | 605 kB 00:00 2024-04-06T12:09:13.758 INFO:teuthology.orchestra.run.smithi192.stdout:(81/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.7 MB/s | 220 kB 00:00 2024-04-06T12:09:13.774 INFO:teuthology.orchestra.run.smithi192.stdout:(82/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.3 MB/s | 229 kB 00:00 2024-04-06T12:09:13.867 INFO:teuthology.orchestra.run.smithi192.stdout:(83/146): keyutils-libs-devel-1.5.10-9.el8.x86_ 444 kB/s | 48 kB 00:00 2024-04-06T12:09:13.892 INFO:teuthology.orchestra.run.smithi192.stdout:(84/146): libselinux-devel-2.9-5.el8.x86_64.rpm 1.2 MB/s | 200 kB 00:00 2024-04-06T12:09:13.951 INFO:teuthology.orchestra.run.smithi192.stdout:(85/146): libsepol-devel-2.9-3.el8.x86_64.rpm 495 kB/s | 87 kB 00:00 2024-04-06T12:09:14.026 INFO:teuthology.orchestra.run.smithi192.stdout:(86/146): libcom_err-devel-1.45.6-4.el8.x86_64. 289 kB/s | 39 kB 00:00 2024-04-06T12:09:14.059 INFO:teuthology.orchestra.run.smithi189.stdout:(55/146): ceph-osd-18.2.2-1086.g052adcad.el8.x8 2.4 MB/s | 18 MB 00:07 2024-04-06T12:09:14.059 INFO:teuthology.orchestra.run.smithi192.stdout:(87/146): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 299 kB/s | 58 kB 00:00 2024-04-06T12:09:14.084 INFO:teuthology.orchestra.run.smithi189.stdout:(56/146): python3-portend-2.6-1.el8.noarch.rpm 34 kB/s | 16 kB 00:00 2024-04-06T12:09:15.216 INFO:teuthology.orchestra.run.smithi192.stdout:(88/146): fuse-2.9.7-17.el8.x86_64.rpm 621 kB/s | 83 kB 00:00 2024-04-06T12:09:15.216 INFO:teuthology.orchestra.run.smithi192.stdout:(89/146): libgfortran-8.5.0-20.el8.x86_64.rpm 4.7 MB/s | 645 kB 00:00 2024-04-06T12:09:15.216 INFO:teuthology.orchestra.run.smithi192.stdout:(90/146): libuuid-devel-2.32.1-43.el8.x86_64.rp 996 kB/s | 100 kB 00:00 2024-04-06T12:09:15.216 INFO:teuthology.orchestra.run.smithi192.stdout:(91/146): libquadmath-8.5.0-20.el8.x86_64.rpm 1.1 MB/s | 172 kB 00:00 2024-04-06T12:09:15.216 INFO:teuthology.orchestra.run.smithi192.stdout:(92/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.7 MB/s | 238 kB 00:00 2024-04-06T12:09:15.216 INFO:teuthology.orchestra.run.smithi192.stdout:(93/146): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 543 kB/s | 73 kB 00:00 2024-04-06T12:09:15.216 INFO:teuthology.orchestra.run.smithi192.stdout:(94/146): NetworkManager-initscripts-updown-1.4 862 kB/s | 144 kB 00:00 2024-04-06T12:09:15.216 INFO:teuthology.orchestra.run.smithi192.stdout:(95/146): elfutils-libelf-devel-0.189-3.el8.x86 567 kB/s | 62 kB 00:00 2024-04-06T12:09:15.216 INFO:teuthology.orchestra.run.smithi192.stdout:(96/146): krb5-devel-1.18.2-26.el8_9.x86_64.rpm 4.4 MB/s | 562 kB 00:00 2024-04-06T12:09:15.216 INFO:teuthology.orchestra.run.smithi192.stdout:(97/146): openssl-devel-1.1.1k-12.el8_9.x86_64. 14 MB/s | 2.3 MB 00:00 2024-04-06T12:09:15.216 INFO:teuthology.orchestra.run.smithi192.stdout:(98/146): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 268 kB/s | 47 kB 00:00 2024-04-06T12:09:15.216 INFO:teuthology.orchestra.run.smithi192.stdout:(99/146): python3-cryptography-3.2.1-7.el8_9.x8 2.8 MB/s | 559 kB 00:00 2024-04-06T12:09:15.217 INFO:teuthology.orchestra.run.smithi192.stdout:(100/146): python3-prettytable-0.7.2-14.el8.noa 405 kB/s | 44 kB 00:00 2024-04-06T12:09:15.217 INFO:teuthology.orchestra.run.smithi192.stdout:(101/146): python3-jsonpointer-1.10-11.el8.noar 119 kB/s | 20 kB 00:00 2024-04-06T12:09:15.217 INFO:teuthology.orchestra.run.smithi192.stdout:(102/146): python3-pyasn1-modules-0.3.7-6.el8.n 1.1 MB/s | 110 kB 00:00 2024-04-06T12:09:15.217 INFO:teuthology.orchestra.run.smithi192.stdout:(103/146): python3-jsonpatch-1.21-2.el8.noarch. 227 kB/s | 27 kB 00:00 2024-04-06T12:09:15.217 INFO:teuthology.orchestra.run.smithi192.stdout:(104/146): python3-pyasn1-0.3.7-6.el8.noarch.rp 1.1 MB/s | 126 kB 00:00 2024-04-06T12:09:15.217 INFO:teuthology.orchestra.run.smithi192.stdout:(105/146): python3-werkzeug-0.12.2-4.el8.noarch 2.5 MB/s | 457 kB 00:00 2024-04-06T12:09:15.217 INFO:teuthology.orchestra.run.smithi192.stdout:(106/146): bison-3.0.4-10.el8.x86_64.rpm 3.8 MB/s | 688 kB 00:00 2024-04-06T12:09:15.217 INFO:teuthology.orchestra.run.smithi192.stdout:(107/146): flex-2.6.1-9.el8.x86_64.rpm 1.7 MB/s | 320 kB 00:00 2024-04-06T12:09:15.217 INFO:teuthology.orchestra.run.smithi192.stdout:(108/146): python3-markupsafe-0.23-19.el8.x86_6 423 kB/s | 39 kB 00:00 2024-04-06T12:09:15.217 INFO:teuthology.orchestra.run.smithi192.stdout:(109/146): libtool-2.4.6-25.el8.x86_64.rpm 3.5 MB/s | 709 kB 00:00 2024-04-06T12:09:15.217 INFO:teuthology.orchestra.run.smithi192.stdout:(110/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.0 MB/s | 259 kB 00:00 2024-04-06T12:09:15.217 INFO:teuthology.orchestra.run.smithi192.stdout:(111/146): uuid-1.6.2-43.el8.x86_64.rpm 637 kB/s | 64 kB 00:00 2024-04-06T12:09:15.218 INFO:teuthology.orchestra.run.smithi189.stdout:(57/146): python3-pecan-1.3.2-9.el8.noarch.rpm 521 kB/s | 283 kB 00:00 2024-04-06T12:09:15.218 INFO:teuthology.orchestra.run.smithi189.stdout:(58/146): python3-repoze-lru-0.7-6.el8.noarch.r 447 kB/s | 34 kB 00:00 2024-04-06T12:09:15.218 INFO:teuthology.orchestra.run.smithi189.stdout:(59/146): python3-rsa-4.9-2.el8.noarch.rpm 1.3 MB/s | 64 kB 00:00 2024-04-06T12:09:15.218 INFO:teuthology.orchestra.run.smithi189.stdout:(60/146): python3-simplegeneric-0.8.1-17.el8.no 374 kB/s | 19 kB 00:00 2024-04-06T12:09:15.218 INFO:teuthology.orchestra.run.smithi189.stdout:(61/146): python3-singledispatch-3.4.0.3-18.el8 476 kB/s | 24 kB 00:00 2024-04-06T12:09:15.218 INFO:teuthology.orchestra.run.smithi189.stdout:(62/146): python3-tempora-1.14.1-5.el8.noarch.r 583 kB/s | 29 kB 00:00 2024-04-06T12:09:15.218 INFO:teuthology.orchestra.run.smithi189.stdout:(63/146): python3-trustme-0.6.0-4.el8.noarch.rp 548 kB/s | 27 kB 00:00 2024-04-06T12:09:15.219 INFO:teuthology.orchestra.run.smithi189.stdout:(64/146): python3-waitress-1.4.3-1.el8.noarch.r 4.3 MB/s | 255 kB 00:00 2024-04-06T12:09:15.219 INFO:teuthology.orchestra.run.smithi189.stdout:(65/146): python3-routes-2.4.1-12.el8.noarch.rp 835 kB/s | 196 kB 00:00 2024-04-06T12:09:15.219 INFO:teuthology.orchestra.run.smithi189.stdout:(66/146): python3-webob-1.8.5-1.el8.1.noarch.rp 2.9 MB/s | 251 kB 00:00 2024-04-06T12:09:15.219 INFO:teuthology.orchestra.run.smithi189.stdout:(67/146): python3-websocket-client-0.56.0-5.el8 916 kB/s | 61 kB 00:00 2024-04-06T12:09:15.219 INFO:teuthology.orchestra.run.smithi189.stdout:(68/146): python3-webtest-2.0.33-1.el8.noarch.r 1.2 MB/s | 85 kB 00:00 2024-04-06T12:09:15.219 INFO:teuthology.orchestra.run.smithi189.stdout:(69/146): python3-zc-lockfile-2.0-2.el8.noarch. 249 kB/s | 23 kB 00:00 2024-04-06T12:09:15.219 INFO:teuthology.orchestra.run.smithi189.stdout:(70/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 831 kB/s | 70 kB 00:00 2024-04-06T12:09:15.219 INFO:teuthology.orchestra.run.smithi189.stdout:(71/146): thrift-0.13.0-2.el8.x86_64.rpm 11 MB/s | 1.7 MB 00:00 2024-04-06T12:09:15.219 INFO:teuthology.orchestra.run.smithi189.stdout:(72/146): libverto-devel-0.3.0-5.el8.x86_64.rpm 97 kB/s | 18 kB 00:00 2024-04-06T12:09:15.219 INFO:teuthology.orchestra.run.smithi189.stdout:(73/146): python3-jwt-1.6.1-2.el8.noarch.rpm 209 kB/s | 43 kB 00:00 2024-04-06T12:09:15.219 INFO:teuthology.orchestra.run.smithi189.stdout:(74/146): python3-pycparser-2.14-14.el8.noarch. 1.1 MB/s | 109 kB 00:00 2024-04-06T12:09:15.219 INFO:teuthology.orchestra.run.smithi189.stdout:(75/146): python3-requests-oauthlib-1.0.0-1.el8 639 kB/s | 43 kB 00:00 2024-04-06T12:09:15.219 INFO:teuthology.orchestra.run.smithi189.stdout:(76/146): python3-oauthlib-2.1.0-1.el8.noarch.r 610 kB/s | 155 kB 00:00 2024-04-06T12:09:15.219 INFO:teuthology.orchestra.run.smithi189.stdout:(77/146): libaio-devel-0.3.112-1.el8.x86_64.rpm 149 kB/s | 19 kB 00:00 2024-04-06T12:09:15.220 INFO:teuthology.orchestra.run.smithi189.stdout:(78/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 328 kB/s | 44 kB 00:00 2024-04-06T12:09:15.220 INFO:teuthology.orchestra.run.smithi189.stdout:(79/146): pcre2-devel-10.32-2.el8.x86_64.rpm 3.9 MB/s | 605 kB 00:00 2024-04-06T12:09:15.220 INFO:teuthology.orchestra.run.smithi189.stdout:(80/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.5 MB/s | 229 kB 00:00 2024-04-06T12:09:15.220 INFO:teuthology.orchestra.run.smithi189.stdout:(81/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.4 MB/s | 220 kB 00:00 2024-04-06T12:09:15.220 INFO:teuthology.orchestra.run.smithi189.stdout:(82/146): libselinux-devel-2.9-5.el8.x86_64.rpm 1.4 MB/s | 200 kB 00:00 2024-04-06T12:09:15.220 INFO:teuthology.orchestra.run.smithi189.stdout:(83/146): keyutils-libs-devel-1.5.10-9.el8.x86_ 384 kB/s | 48 kB 00:00 2024-04-06T12:09:15.220 INFO:teuthology.orchestra.run.smithi189.stdout:(84/146): libsepol-devel-2.9-3.el8.x86_64.rpm 799 kB/s | 87 kB 00:00 2024-04-06T12:09:15.220 INFO:teuthology.orchestra.run.smithi189.stdout:(85/146): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 460 kB/s | 58 kB 00:00 2024-04-06T12:09:15.242 INFO:teuthology.orchestra.run.smithi192.stdout:(112/146): python3-pyOpenSSL-19.0.0-1.el8.noarc 824 kB/s | 103 kB 00:00 2024-04-06T12:09:15.267 INFO:teuthology.orchestra.run.smithi189.stdout:(86/146): libcom_err-devel-1.45.6-4.el8.x86_64. 272 kB/s | 39 kB 00:00 2024-04-06T12:09:15.292 INFO:teuthology.orchestra.run.smithi189.stdout:(87/146): fuse-2.9.7-17.el8.x86_64.rpm 552 kB/s | 83 kB 00:00 2024-04-06T12:09:15.326 INFO:teuthology.orchestra.run.smithi189.stdout:(88/146): libgfortran-8.5.0-20.el8.x86_64.rpm 4.7 MB/s | 645 kB 00:00 2024-04-06T12:09:15.334 INFO:teuthology.orchestra.run.smithi192.stdout:(113/146): python3-jinja2-2.10.1-3.el8.noarch.r 3.3 MB/s | 538 kB 00:00 2024-04-06T12:09:15.401 INFO:teuthology.orchestra.run.smithi189.stdout:(89/146): libuuid-devel-2.32.1-43.el8.x86_64.rp 920 kB/s | 100 kB 00:00 2024-04-06T12:09:15.409 INFO:teuthology.orchestra.run.smithi192.stdout:(114/146): python3-babel-2.5.1-7.el8.noarch.rpm 18 MB/s | 4.8 MB 00:00 2024-04-06T12:09:15.418 INFO:teuthology.orchestra.run.smithi189.stdout:(90/146): libquadmath-8.5.0-20.el8.x86_64.rpm 1.1 MB/s | 172 kB 00:00 2024-04-06T12:09:15.443 INFO:teuthology.orchestra.run.smithi189.stdout:(91/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.0 MB/s | 238 kB 00:00 2024-04-06T12:09:15.493 INFO:teuthology.orchestra.run.smithi192.stdout:(115/146): python3-numpy-f2py-1.14.3-10.el8.x86 2.6 MB/s | 225 kB 00:00 2024-04-06T12:09:15.526 INFO:teuthology.orchestra.run.smithi189.stdout:(92/146): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 579 kB/s | 73 kB 00:00 2024-04-06T12:09:15.527 INFO:teuthology.orchestra.run.smithi192.stdout:(116/146): socat-1.7.4.1-1.el8.x86_64.rpm 1.6 MB/s | 323 kB 00:00 2024-04-06T12:09:15.553 INFO:teuthology.orchestra.run.smithi189.stdout:(93/146): NetworkManager-initscripts-updown-1.4 1.1 MB/s | 144 kB 00:00 2024-04-06T12:09:15.640 INFO:teuthology.orchestra.run.smithi189.stdout:(94/146): elfutils-libelf-devel-0.189-3.el8.x86 314 kB/s | 62 kB 00:00 2024-04-06T12:09:15.686 INFO:teuthology.orchestra.run.smithi189.stdout:(95/146): krb5-devel-1.18.2-26.el8_9.x86_64.rpm 3.5 MB/s | 562 kB 00:00 2024-04-06T12:09:15.727 INFO:teuthology.orchestra.run.smithi192.stdout:(117/146): python3-numpy-1.14.3-10.el8.x86_64.r 7.7 MB/s | 3.7 MB 00:00 2024-04-06T12:09:15.761 INFO:teuthology.orchestra.run.smithi189.stdout:(96/146): openssl-devel-1.1.1k-12.el8_9.x86_64. 11 MB/s | 2.3 MB 00:00 2024-04-06T12:09:15.787 INFO:teuthology.orchestra.run.smithi189.stdout:(97/146): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 321 kB/s | 47 kB 00:00 2024-04-06T12:09:15.821 INFO:teuthology.orchestra.run.smithi189.stdout:(98/146): python3-cryptography-3.2.1-7.el8_9.x8 4.1 MB/s | 559 kB 00:00 2024-04-06T12:09:15.854 INFO:teuthology.orchestra.run.smithi189.stdout:(99/146): python3-packaging-16.8-9.el8.noarch.r 524 kB/s | 48 kB 00:00 2024-04-06T12:09:15.860 INFO:teuthology.orchestra.run.smithi192.stdout:(118/146): librdkafka-0.11.4-3.el8.x86_64.rpm 964 kB/s | 354 kB 00:00 2024-04-06T12:09:15.886 INFO:teuthology.orchestra.run.smithi192.stdout:(119/146): python3-mako-1.0.6-14.el8.noarch.rpm 991 kB/s | 157 kB 00:00 2024-04-06T12:09:15.946 INFO:teuthology.orchestra.run.smithi189.stdout:(100/146): python3-prettytable-0.7.2-14.el8.noa 350 kB/s | 44 kB 00:00 2024-04-06T12:09:15.972 INFO:teuthology.orchestra.run.smithi189.stdout:(101/146): python3-jsonpointer-1.10-11.el8.noar 108 kB/s | 20 kB 00:00 2024-04-06T12:09:15.996 INFO:teuthology.orchestra.run.smithi189.stdout:(102/146): python3-pyasn1-modules-0.3.7-6.el8.n 772 kB/s | 110 kB 00:00 2024-04-06T12:09:16.036 INFO:teuthology.orchestra.run.smithi192.stdout:(120/146): openblas-threads-0.3.15-6.el8.x86_64 10 MB/s | 5.2 MB 00:00 2024-04-06T12:09:16.063 INFO:teuthology.orchestra.run.smithi189.stdout:(103/146): python3-jsonpatch-1.21-2.el8.noarch. 227 kB/s | 27 kB 00:00 2024-04-06T12:09:16.078 INFO:teuthology.orchestra.run.smithi192.stdout:(121/146): automake-1.16.1-8.el8.noarch.rpm 3.6 MB/s | 714 kB 00:00 2024-04-06T12:09:16.097 INFO:teuthology.orchestra.run.smithi189.stdout:(104/146): python3-werkzeug-0.12.2-4.el8.noarch 4.5 MB/s | 457 kB 00:00 2024-04-06T12:09:16.122 INFO:teuthology.orchestra.run.smithi189.stdout:(105/146): python3-pyasn1-0.3.7-6.el8.noarch.rp 839 kB/s | 126 kB 00:00 2024-04-06T12:09:16.173 INFO:teuthology.orchestra.run.smithi189.stdout:(106/146): bison-3.0.4-10.el8.x86_64.rpm 6.2 MB/s | 688 kB 00:00 2024-04-06T12:09:16.179 INFO:teuthology.orchestra.run.smithi192.stdout:(122/146): openblas-0.3.15-6.el8.x86_64.rpm 16 MB/s | 5.0 MB 00:00 2024-04-06T12:09:16.195 INFO:teuthology.orchestra.run.smithi192.stdout:(123/146): libpq-13.11-1.el8.x86_64.rpm 1.2 MB/s | 198 kB 00:00 2024-04-06T12:09:16.237 INFO:teuthology.orchestra.run.smithi192.stdout:(124/146): jq-1.6-7.el8.x86_64.rpm 1.2 MB/s | 202 kB 00:00 2024-04-06T12:09:16.239 INFO:teuthology.orchestra.run.smithi189.stdout:(107/146): flex-2.6.1-9.el8.x86_64.rpm 2.2 MB/s | 320 kB 00:00 2024-04-06T12:09:16.315 INFO:teuthology.orchestra.run.smithi189.stdout:(108/146): python3-markupsafe-0.23-19.el8.x86_6 274 kB/s | 39 kB 00:00 2024-04-06T12:09:16.337 INFO:teuthology.orchestra.run.smithi192.stdout:(125/146): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-04-06T12:09:16.356 INFO:teuthology.orchestra.run.smithi189.stdout:(109/146): libtool-2.4.6-25.el8.x86_64.rpm 3.0 MB/s | 709 kB 00:00 2024-04-06T12:09:16.382 INFO:teuthology.orchestra.run.smithi189.stdout:(110/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.8 MB/s | 259 kB 00:00 2024-04-06T12:09:16.397 INFO:teuthology.orchestra.run.smithi192.stdout:(126/146): python3-pytz-2017.2-11.el8.noarch.rp 249 kB/s | 54 kB 00:00 2024-04-06T12:09:16.438 INFO:teuthology.orchestra.run.smithi189.stdout:(111/146): uuid-1.6.2-43.el8.x86_64.rpm 520 kB/s | 64 kB 00:00 2024-04-06T12:09:16.455 INFO:teuthology.orchestra.run.smithi192.stdout:(127/146): postgresql-10.23-4.module+el8.9.0+21 13 MB/s | 1.5 MB 00:00 2024-04-06T12:09:16.457 INFO:teuthology.orchestra.run.smithi189.stdout:(112/146): python3-pyOpenSSL-19.0.0-1.el8.noarc 1.0 MB/s | 103 kB 00:00 2024-04-06T12:09:16.505 INFO:teuthology.orchestra.run.smithi192.stdout:(128/146): postgresql-contrib-10.23-4.module+el 7.3 MB/s | 812 kB 00:00 2024-04-06T12:09:16.541 INFO:teuthology.orchestra.run.smithi189.stdout:(113/146): python3-jinja2-2.10.1-3.el8.noarch.r 5.1 MB/s | 538 kB 00:00 2024-04-06T12:09:16.642 INFO:teuthology.orchestra.run.smithi189.stdout:(114/146): python3-babel-2.5.1-7.el8.noarch.rpm 18 MB/s | 4.8 MB 00:00 2024-04-06T12:09:16.747 INFO:teuthology.orchestra.run.smithi192.stdout:(129/146): python3-scipy-1.0.0-21.module+el8.9. 25 MB/s | 14 MB 00:00 2024-04-06T12:09:16.766 INFO:teuthology.orchestra.run.smithi189.stdout:(115/146): python3-numpy-1.14.3-10.el8.x86_64.r 12 MB/s | 3.7 MB 00:00 2024-04-06T12:09:16.791 INFO:teuthology.orchestra.run.smithi189.stdout:(116/146): python3-numpy-f2py-1.14.3-10.el8.x86 1.5 MB/s | 225 kB 00:00 2024-04-06T12:09:16.849 INFO:teuthology.orchestra.run.smithi189.stdout:(117/146): socat-1.7.4.1-1.el8.x86_64.rpm 1.0 MB/s | 323 kB 00:00 2024-04-06T12:09:16.856 INFO:teuthology.orchestra.run.smithi192.stdout:(130/146): postgresql-server-10.23-4.module+el8 13 MB/s | 5.1 MB 00:00 2024-04-06T12:09:16.925 INFO:teuthology.orchestra.run.smithi189.stdout:(118/146): librdkafka-0.11.4-3.el8.x86_64.rpm 2.2 MB/s | 354 kB 00:00 2024-04-06T12:09:17.009 INFO:teuthology.orchestra.run.smithi189.stdout:(119/146): openblas-threads-0.3.15-6.el8.x86_64 24 MB/s | 5.2 MB 00:00 2024-04-06T12:09:17.034 INFO:teuthology.orchestra.run.smithi189.stdout:(120/146): python3-mako-1.0.6-14.el8.noarch.rpm 851 kB/s | 157 kB 00:00 2024-04-06T12:09:17.086 INFO:teuthology.orchestra.run.smithi192.stdout:(131/146): elfutils-libelf-0.189-3.el8.x86_64.r 1.0 MB/s | 232 kB 00:00 2024-04-06T12:09:17.109 INFO:teuthology.orchestra.run.smithi189.stdout:(121/146): automake-1.16.1-8.el8.noarch.rpm 7.0 MB/s | 714 kB 00:00 2024-04-06T12:09:17.168 INFO:teuthology.orchestra.run.smithi189.stdout:(122/146): openblas-0.3.15-6.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2024-04-06T12:09:17.181 INFO:teuthology.orchestra.run.smithi192.stdout:(132/146): librbd1-18.2.2-1086.g052adcad.el8.x8 8.2 MB/s | 3.6 MB 00:00 2024-04-06T12:09:17.193 INFO:teuthology.orchestra.run.smithi189.stdout:(123/146): libpq-13.11-1.el8.x86_64.rpm 1.2 MB/s | 198 kB 00:00 2024-04-06T12:09:17.206 INFO:teuthology.orchestra.run.smithi192.stdout:(133/146): elfutils-libs-0.189-3.el8.x86_64.rpm 2.5 MB/s | 303 kB 00:00 2024-04-06T12:09:17.260 INFO:teuthology.orchestra.run.smithi189.stdout:(124/146): jq-1.6-7.el8.x86_64.rpm 1.3 MB/s | 202 kB 00:00 2024-04-06T12:09:17.327 INFO:teuthology.orchestra.run.smithi189.stdout:(125/146): python3-pytz-2017.2-11.el8.noarch.rp 341 kB/s | 54 kB 00:00 2024-04-06T12:09:17.340 INFO:teuthology.orchestra.run.smithi192.stdout:(134/146): libmount-2.32.1-43.el8.x86_64.rpm 1.5 MB/s | 237 kB 00:00 2024-04-06T12:09:17.377 INFO:teuthology.orchestra.run.smithi189.stdout:(126/146): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-04-06T12:09:17.465 INFO:teuthology.orchestra.run.smithi192.stdout:(135/146): libsmartcols-2.32.1-43.el8.x86_64.rp 694 kB/s | 179 kB 00:00 2024-04-06T12:09:17.469 INFO:teuthology.orchestra.run.smithi189.stdout:(127/146): postgresql-10.23-4.module+el8.9.0+21 11 MB/s | 1.5 MB 00:00 2024-04-06T12:09:17.503 INFO:teuthology.orchestra.run.smithi189.stdout:(128/146): postgresql-contrib-10.23-4.module+el 6.3 MB/s | 812 kB 00:00 2024-04-06T12:09:17.541 INFO:teuthology.orchestra.run.smithi192.stdout:(136/146): librados2-18.2.2-1086.g052adcad.el8. 3.6 MB/s | 3.7 MB 00:01 2024-04-06T12:09:17.566 INFO:teuthology.orchestra.run.smithi192.stdout:(137/146): libuuid-2.32.1-43.el8.x86_64.rpm 439 kB/s | 99 kB 00:00 2024-04-06T12:09:17.683 INFO:teuthology.orchestra.run.smithi192.stdout:(138/146): util-linux-2.32.1-43.el8.x86_64.rpm 11 MB/s | 2.5 MB 00:00 2024-04-06T12:09:17.703 INFO:teuthology.orchestra.run.smithi189.stdout:(129/146): python3-scipy-1.0.0-21.module+el8.9. 27 MB/s | 14 MB 00:00 2024-04-06T12:09:17.708 INFO:teuthology.orchestra.run.smithi192.stdout:(139/146): util-linux-user-2.32.1-43.el8.x86_64 614 kB/s | 103 kB 00:00 2024-04-06T12:09:17.783 INFO:teuthology.orchestra.run.smithi192.stdout:(140/146): iproute-6.2.0-5.el8_9.x86_64.rpm 3.8 MB/s | 854 kB 00:00 2024-04-06T12:09:17.817 INFO:teuthology.orchestra.run.smithi192.stdout:(141/146): libblkid-2.32.1-43.el8.x86_64.rpm 1.6 MB/s | 220 kB 00:00 2024-04-06T12:09:17.820 INFO:teuthology.orchestra.run.smithi189.stdout:(130/146): postgresql-server-10.23-4.module+el8 14 MB/s | 5.1 MB 00:00 2024-04-06T12:09:17.842 INFO:teuthology.orchestra.run.smithi192.stdout:(142/146): libfdisk-2.32.1-43.el8.x86_64.rpm 1.8 MB/s | 254 kB 00:00 2024-04-06T12:09:17.934 INFO:teuthology.orchestra.run.smithi192.stdout:(143/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 4.6 MB/s | 711 kB 00:00 2024-04-06T12:09:17.962 INFO:teuthology.orchestra.run.smithi189.stdout:(131/146): elfutils-libelf-0.189-3.el8.x86_64.r 1.6 MB/s | 232 kB 00:00 2024-04-06T12:09:17.976 INFO:teuthology.orchestra.run.smithi192.stdout:(144/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 2.4 MB/s | 333 kB 00:00 2024-04-06T12:09:18.026 INFO:teuthology.orchestra.run.smithi192.stdout:(145/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 7.0 MB/s | 1.5 MB 00:00 2024-04-06T12:09:18.068 INFO:teuthology.orchestra.run.smithi192.stdout:(146/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 8.0 MB/s | 1.1 MB 00:00 2024-04-06T12:09:18.070 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-06T12:09:18.071 INFO:teuthology.orchestra.run.smithi192.stdout:Total 13 MB/s | 215 MB 00:15 2024-04-06T12:09:18.096 INFO:teuthology.orchestra.run.smithi189.stdout:(132/146): elfutils-libs-0.189-3.el8.x86_64.rpm 2.2 MB/s | 303 kB 00:00 2024-04-06T12:09:18.163 INFO:teuthology.orchestra.run.smithi189.stdout:(133/146): librbd1-18.2.2-1086.g052adcad.el8.x8 7.8 MB/s | 3.6 MB 00:00 2024-04-06T12:09:18.238 INFO:teuthology.orchestra.run.smithi189.stdout:(134/146): libmount-2.32.1-43.el8.x86_64.rpm 1.6 MB/s | 237 kB 00:00 2024-04-06T12:09:18.271 INFO:teuthology.orchestra.run.smithi189.stdout:(135/146): libsmartcols-2.32.1-43.el8.x86_64.rp 1.6 MB/s | 179 kB 00:00 2024-04-06T12:09:18.330 INFO:teuthology.orchestra.run.smithi189.stdout:(136/146): libuuid-2.32.1-43.el8.x86_64.rpm 1.1 MB/s | 99 kB 00:00 2024-04-06T12:09:18.422 INFO:teuthology.orchestra.run.smithi189.stdout:(137/146): util-linux-2.32.1-43.el8.x86_64.rpm 16 MB/s | 2.5 MB 00:00 2024-04-06T12:09:18.489 INFO:teuthology.orchestra.run.smithi189.stdout:(138/146): util-linux-user-2.32.1-43.el8.x86_64 647 kB/s | 103 kB 00:00 2024-04-06T12:09:18.539 INFO:teuthology.orchestra.run.smithi189.stdout:(139/146): librados2-18.2.2-1086.g052adcad.el8. 3.6 MB/s | 3.7 MB 00:01 2024-04-06T12:09:18.573 INFO:teuthology.orchestra.run.smithi189.stdout:(140/146): iproute-6.2.0-5.el8_9.x86_64.rpm 5.5 MB/s | 854 kB 00:00 2024-04-06T12:09:18.598 INFO:teuthology.orchestra.run.smithi189.stdout:(141/146): libblkid-2.32.1-43.el8.x86_64.rpm 2.0 MB/s | 220 kB 00:00 2024-04-06T12:09:18.681 INFO:teuthology.orchestra.run.smithi189.stdout:(142/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 6.4 MB/s | 711 kB 00:00 2024-04-06T12:09:18.698 INFO:teuthology.orchestra.run.smithi189.stdout:(143/146): libfdisk-2.32.1-43.el8.x86_64.rpm 1.6 MB/s | 254 kB 00:00 2024-04-06T12:09:18.794 INFO:teuthology.orchestra.run.smithi189.stdout:(144/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 7.7 MB/s | 1.5 MB 00:00 2024-04-06T12:09:18.815 INFO:teuthology.orchestra.run.smithi189.stdout:(145/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 2.4 MB/s | 333 kB 00:00 2024-04-06T12:09:18.957 INFO:teuthology.orchestra.run.smithi189.stdout:(146/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 4.1 MB/s | 1.1 MB 00:00 2024-04-06T12:09:18.960 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-06T12:09:18.960 INFO:teuthology.orchestra.run.smithi189.stdout:Total 13 MB/s | 215 MB 00:15 2024-04-06T12:09:19.157 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-06T12:09:19.552 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-06T12:09:19.552 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-06T12:09:19.998 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-06T12:09:20.398 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-06T12:09:20.399 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-06T12:09:20.767 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-06T12:09:20.777 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-06T12:09:21.602 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-06T12:09:21.612 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-06T12:09:26.474 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-06T12:09:26.651 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-04-06T12:09:26.930 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-04-06T12:09:27.152 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-04-06T12:09:27.244 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-06T12:09:27.360 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-04-06T12:09:27.379 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-04-06T12:09:27.484 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-04-06T12:09:27.561 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-04-06T12:09:27.661 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-04-06T12:09:27.743 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-04-06T12:09:27.798 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-04-06T12:09:27.864 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-04-06T12:09:27.997 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-04-06T12:09:28.021 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-04-06T12:09:28.166 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 6/162 2024-04-06T12:09:28.195 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-04-06T12:09:28.250 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-04-06T12:09:28.360 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 7/162 2024-04-06T12:09:28.428 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-04-06T12:09:28.446 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 7/162 2024-04-06T12:09:28.519 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-04-06T12:09:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-04-06T12:09:28.716 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-04-06T12:09:28.935 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 6/162 2024-04-06T12:09:29.052 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-04-06T12:09:29.095 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 7/162 2024-04-06T12:09:29.192 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 7/162 2024-04-06T12:09:29.210 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librados2-2:18.2.2-1086.g052adcad.el8.x86_64 9/162 2024-04-06T12:09:29.436 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-04-06T12:09:29.602 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:18.2.2-1086.g052adcad.el8.x86_64 9/162 2024-04-06T12:09:29.822 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librbd1-2:18.2.2-1086.g052adcad.el8.x86_64 10/162 2024-04-06T12:09:29.844 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-04-06T12:09:29.948 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:18.2.2-1086.g052adcad.el8.x86_64 10/162 2024-04-06T12:09:30.118 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs2-2:18.2.2-1086.g052adcad.el8.x86_64 11/162 2024-04-06T12:09:30.145 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librados2-2:18.2.2-1086.g052adcad.el8.x86_64 9/162 2024-04-06T12:09:30.201 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs2-2:18.2.2-1086.g052adcad.el8.x86_64 11/162 2024-04-06T12:09:30.453 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rados-2:18.2.2-1086.g052adcad.el8.x86_64 12/162 2024-04-06T12:09:30.525 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:18.2.2-1086.g052adcad.el8.x86_64 9/162 2024-04-06T12:09:30.684 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-04-06T12:09:30.743 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librbd1-2:18.2.2-1086.g052adcad.el8.x86_64 10/162 2024-04-06T12:09:30.800 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-04-06T12:09:30.852 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:18.2.2-1086.g052adcad.el8.x86_64 10/162 2024-04-06T12:09:30.982 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-04-06T12:09:31.028 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs2-2:18.2.2-1086.g052adcad.el8.x86_64 11/162 2024-04-06T12:09:31.106 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephfs2-2:18.2.2-1086.g052adcad.el8.x86_64 11/162 2024-04-06T12:09:31.196 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-04-06T12:09:31.371 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-04-06T12:09:31.372 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rados-2:18.2.2-1086.g052adcad.el8.x86_64 12/162 2024-04-06T12:09:31.611 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-04-06T12:09:31.696 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-04-06T12:09:31.756 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-04-06T12:09:31.884 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-04-06T12:09:32.071 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 18/162 2024-04-06T12:09:32.075 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-04-06T12:09:32.217 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephsqlite-2:18.2.2-1086.g052adcad.el8.x86_64 19/162 2024-04-06T12:09:32.275 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-04-06T12:09:32.309 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1086.g052adcad.el8.x86_64 19/162 2024-04-06T12:09:32.434 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libradosstriper1-2:18.2.2-1086.g052adcad.el8.x86 20/162 2024-04-06T12:09:32.498 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1086.g052adcad.el8.x86 20/162 2024-04-06T12:09:32.616 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-04-06T12:09:32.688 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 21/162 2024-04-06T12:09:32.823 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 18/162 2024-04-06T12:09:32.943 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 22/162 2024-04-06T12:09:32.986 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephsqlite-2:18.2.2-1086.g052adcad.el8.x86_64 19/162 2024-04-06T12:09:33.071 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1086.g052adcad.el8.x86_64 19/162 2024-04-06T12:09:33.164 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/162 2024-04-06T12:09:33.203 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libradosstriper1-2:18.2.2-1086.g052adcad.el8.x86 20/162 2024-04-06T12:09:33.260 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1086.g052adcad.el8.x86 20/162 2024-04-06T12:09:33.367 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 24/162 2024-04-06T12:09:33.448 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 21/162 2024-04-06T12:09:33.505 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 24/162 2024-04-06T12:09:33.720 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 22/162 2024-04-06T12:09:33.721 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 25/162 2024-04-06T12:09:33.933 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/162 2024-04-06T12:09:34.144 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 24/162 2024-04-06T12:09:34.273 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 24/162 2024-04-06T12:09:34.344 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 25/162 2024-04-06T12:09:34.440 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 25/162 2024-04-06T12:09:34.519 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-04-06T12:09:35.092 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 25/162 2024-04-06T12:09:35.110 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-04-06T12:09:35.301 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-0.3.15-6.el8.x86_64 27/162 2024-04-06T12:09:35.594 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-04-06T12:09:35.991 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 27/162 2024-04-06T12:09:36.205 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-04-06T12:09:36.265 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 28/162 2024-04-06T12:09:36.388 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-0.3.15-6.el8.x86_64 27/162 2024-04-06T12:09:36.497 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 29/162 2024-04-06T12:09:36.751 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-04-06T12:09:36.919 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-common-2:18.2.2-1086.g052adcad.el8. 31/162 2024-04-06T12:09:37.044 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-argparse-2:18.2.2-1086.g052adcad.el 32/162 2024-04-06T12:09:37.072 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 27/162 2024-04-06T12:09:37.383 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 28/162 2024-04-06T12:09:37.591 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 29/162 2024-04-06T12:09:37.623 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cephfs-2:18.2.2-1086.g052adcad.el8.x86_6 33/162 2024-04-06T12:09:37.833 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-04-06T12:09:37.896 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-04-06T12:09:38.088 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-common-2:18.2.2-1086.g052adcad.el8. 31/162 2024-04-06T12:09:38.255 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-argparse-2:18.2.2-1086.g052adcad.el 32/162 2024-04-06T12:09:38.523 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cephfs-2:18.2.2-1086.g052adcad.el8.x86_6 33/162 2024-04-06T12:09:38.777 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-04-06T12:09:39.580 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-04-06T12:09:39.985 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 35/162 2024-04-06T12:09:40.525 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-04-06T12:09:40.912 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 35/162 2024-04-06T12:09:40.941 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 36/162 2024-04-06T12:09:41.255 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 37/162 2024-04-06T12:09:41.500 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 38/162 2024-04-06T12:09:41.711 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-04-06T12:09:41.894 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-04-06T12:09:41.906 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 36/162 2024-04-06T12:09:42.097 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rbd-2:18.2.2-1086.g052adcad.el8.x86_64 41/162 2024-04-06T12:09:42.269 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 37/162 2024-04-06T12:09:42.297 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados-devel-2:18.2.2-1086.g052adcad.el8.x86_6 42/162 2024-04-06T12:09:42.502 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 38/162 2024-04-06T12:09:42.541 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-04-06T12:09:42.619 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-04-06T12:09:42.762 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-04-06T12:09:42.841 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-04-06T12:09:42.920 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-04-06T12:09:42.930 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-04-06T12:09:43.168 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rbd-2:18.2.2-1086.g052adcad.el8.x86_64 41/162 2024-04-06T12:09:43.188 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-04-06T12:09:43.401 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librados-devel-2:18.2.2-1086.g052adcad.el8.x86_6 42/162 2024-04-06T12:09:43.410 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-04-06T12:09:43.584 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-04-06T12:09:43.644 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-04-06T12:09:43.647 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-04-06T12:09:43.859 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-04-06T12:09:43.912 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-04-06T12:09:44.075 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-04-06T12:09:44.126 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-04-06T12:09:44.370 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-04-06T12:09:44.408 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-04-06T12:09:44.436 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-04-06T12:09:44.622 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-04-06T12:09:44.633 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 49/162 2024-04-06T12:09:44.717 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 49/162 2024-04-06T12:09:44.994 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : jq-1.6-7.el8.x86_64 50/162 2024-04-06T12:09:45.074 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-04-06T12:09:45.260 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : automake-1.16.1-8.el8.noarch 51/162 2024-04-06T12:09:45.424 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : uuid-1.6.2-43.el8.x86_64 52/162 2024-04-06T12:09:45.444 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-04-06T12:09:45.471 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-04-06T12:09:45.495 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 52/162 2024-04-06T12:09:45.640 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 49/162 2024-04-06T12:09:45.703 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 53/162 2024-04-06T12:09:45.708 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 49/162 2024-04-06T12:09:45.890 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 54/162 2024-04-06T12:09:46.021 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : jq-1.6-7.el8.x86_64 50/162 2024-04-06T12:09:46.084 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 55/162 2024-04-06T12:09:46.283 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : automake-1.16.1-8.el8.noarch 51/162 2024-04-06T12:09:46.304 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : fuse-2.9.7-17.el8.x86_64 56/162 2024-04-06T12:09:46.459 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : uuid-1.6.2-43.el8.x86_64 52/162 2024-04-06T12:09:46.490 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 57/162 2024-04-06T12:09:46.527 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 52/162 2024-04-06T12:09:46.680 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 58/162 2024-04-06T12:09:46.702 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 53/162 2024-04-06T12:09:46.870 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 59/162 2024-04-06T12:09:46.892 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 54/162 2024-04-06T12:09:47.094 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 60/162 2024-04-06T12:09:47.106 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 55/162 2024-04-06T12:09:47.295 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-packaging-16.8-9.el8.noarch 56/162 2024-04-06T12:09:47.403 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 61/162 2024-04-06T12:09:47.522 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : fuse-2.9.7-17.el8.x86_64 57/162 2024-04-06T12:09:47.688 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 62/162 2024-04-06T12:09:47.710 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 58/162 2024-04-06T12:09:47.887 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 63/162 2024-04-06T12:09:47.931 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 59/162 2024-04-06T12:09:48.061 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 64/162 2024-04-06T12:09:48.098 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 60/162 2024-04-06T12:09:48.251 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 65/162 2024-04-06T12:09:48.323 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 61/162 2024-04-06T12:09:48.614 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 66/162 2024-04-06T12:09:48.638 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 62/162 2024-04-06T12:09:48.873 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 63/162 2024-04-06T12:09:48.951 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 67/162 2024-04-06T12:09:49.141 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 64/162 2024-04-06T12:09:49.284 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 68/162 2024-04-06T12:09:49.305 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 65/162 2024-04-06T12:09:49.498 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 69/162 2024-04-06T12:09:49.546 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 66/162 2024-04-06T12:09:49.683 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 70/162 2024-04-06T12:09:49.917 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 67/162 2024-04-06T12:09:49.940 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 71/162 2024-04-06T12:09:50.169 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 72/162 2024-04-06T12:09:50.254 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 68/162 2024-04-06T12:09:50.340 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 73/162 2024-04-06T12:09:50.525 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 69/162 2024-04-06T12:09:50.584 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 74/162 2024-04-06T12:09:50.770 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 75/162 2024-04-06T12:09:50.783 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 70/162 2024-04-06T12:09:50.951 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 71/162 2024-04-06T12:09:51.066 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 76/162 2024-04-06T12:09:51.241 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-packaging-16.8-9.el8.noarch 77/162 2024-04-06T12:09:51.264 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 72/162 2024-04-06T12:09:51.414 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-04-06T12:09:51.554 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 73/162 2024-04-06T12:09:51.585 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-04-06T12:09:51.751 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 74/162 2024-04-06T12:09:51.841 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-04-06T12:09:51.996 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 75/162 2024-04-06T12:09:52.063 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-04-06T12:09:52.202 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 76/162 2024-04-06T12:09:52.252 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-04-06T12:09:52.425 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-04-06T12:09:52.479 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 77/162 2024-04-06T12:09:52.607 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-04-06T12:09:52.675 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-04-06T12:09:52.819 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-04-06T12:09:52.850 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-04-06T12:09:53.123 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-04-06T12:09:53.126 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-04-06T12:09:53.301 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-04-06T12:09:53.314 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-04-06T12:09:53.487 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-04-06T12:09:53.529 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-04-06T12:09:53.680 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-04-06T12:09:53.727 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-04-06T12:09:53.917 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-04-06T12:09:53.952 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-04-06T12:09:54.146 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-04-06T12:09:54.199 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-04-06T12:09:54.386 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-04-06T12:09:54.443 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-04-06T12:09:54.555 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-04-06T12:09:54.663 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-04-06T12:09:54.730 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-04-06T12:09:54.872 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-04-06T12:09:54.956 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-04-06T12:09:55.072 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-04-06T12:09:55.195 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-04-06T12:09:55.395 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-04-06T12:09:55.678 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-04-06T12:09:55.940 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-04-06T12:09:56.114 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-04-06T12:09:56.165 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-04-06T12:09:56.349 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-04-06T12:09:56.484 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-04-06T12:09:56.730 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-04-06T12:09:57.040 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-04-06T12:09:57.331 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-04-06T12:09:57.413 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-04-06T12:09:57.513 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-04-06T12:09:57.739 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-04-06T12:09:57.999 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-04-06T12:09:58.329 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-04-06T12:09:58.377 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-04-06T12:09:58.472 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librgw2-2:18.2.2-1086.g052adcad.el8.x86_64 103/162 2024-04-06T12:09:58.547 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librgw2-2:18.2.2-1086.g052adcad.el8.x86_64 103/162 2024-04-06T12:09:58.668 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-04-06T12:09:58.789 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rgw-2:18.2.2-1086.g052adcad.el8.x86_64 104/162 2024-04-06T12:09:58.882 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-04-06T12:09:59.697 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-04-06T12:09:59.865 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librgw2-2:18.2.2-1086.g052adcad.el8.x86_64 103/162 2024-04-06T12:09:59.940 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librgw2-2:18.2.2-1086.g052adcad.el8.x86_64 103/162 2024-04-06T12:10:00.149 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rgw-2:18.2.2-1086.g052adcad.el8.x86_64 104/162 2024-04-06T12:10:02.466 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:18.2.2-1086.g052adcad.el8.x86_64 105/162 2024-04-06T12:10:02.730 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-common-2:18.2.2-1086.g052adcad.el8.x86_64 105/162 2024-04-06T12:10:03.321 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:18.2.2-1086.g052adcad.el8.x86_64 105/162 2024-04-06T12:10:04.374 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-base-2:18.2.2-1086.g052adcad.el8.x86_64 106/162 2024-04-06T12:10:04.424 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:18.2.2-1086.g052adcad.el8.x86_64 105/162 2024-04-06T12:10:04.451 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-base-2:18.2.2-1086.g052adcad.el8.x86_64 106/162 2024-04-06T12:10:04.605 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-selinux-2:18.2.2-1086.g052adcad.el8.x86_64 107/162 2024-04-06T12:10:04.689 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-common-2:18.2.2-1086.g052adcad.el8.x86_64 105/162 2024-04-06T12:10:05.283 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:18.2.2-1086.g052adcad.el8.x86_64 105/162 2024-04-06T12:10:05.602 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-base-2:18.2.2-1086.g052adcad.el8.x86_64 106/162 2024-04-06T12:10:05.684 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-base-2:18.2.2-1086.g052adcad.el8.x86_64 106/162 2024-04-06T12:10:05.839 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-selinux-2:18.2.2-1086.g052adcad.el8.x86_64 107/162 2024-04-06T12:10:24.111 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1086.g052adcad.el8.x86_64 107/162 2024-04-06T12:10:24.111 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /sys 2024-04-06T12:10:24.111 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /proc 2024-04-06T12:10:24.111 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /dev 2024-04-06T12:10:24.111 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /run 2024-04-06T12:10:24.111 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /mnt 2024-04-06T12:10:24.111 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /var/tmp 2024-04-06T12:10:24.112 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /home 2024-04-06T12:10:24.112 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /tmp 2024-04-06T12:10:24.112 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /dev 2024-04-06T12:10:24.112 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:10:24.444 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-volume-2:18.2.2-1086.g052adcad.el8.noarch 108/162 2024-04-06T12:10:25.323 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1086.g052adcad.el8.x86_64 107/162 2024-04-06T12:10:25.323 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /sys 2024-04-06T12:10:25.323 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /proc 2024-04-06T12:10:25.323 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /dev 2024-04-06T12:10:25.323 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /run 2024-04-06T12:10:25.323 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /mnt 2024-04-06T12:10:25.323 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /var/tmp 2024-04-06T12:10:25.324 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /home 2024-04-06T12:10:25.324 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /tmp 2024-04-06T12:10:25.324 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /dev 2024-04-06T12:10:25.324 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:10:25.689 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-volume-2:18.2.2-1086.g052adcad.el8.noarch 108/162 2024-04-06T12:10:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-volume-2:18.2.2-1086.g052adcad.el8.noarch 108/162 2024-04-06T12:10:26.296 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-osd-2:18.2.2-1086.g052adcad.el8.x86_64 109/162 2024-04-06T12:10:26.517 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-osd-2:18.2.2-1086.g052adcad.el8.x86_64 109/162 2024-04-06T12:10:26.693 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mds-2:18.2.2-1086.g052adcad.el8.x86_64 110/162 2024-04-06T12:10:27.172 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mds-2:18.2.2-1086.g052adcad.el8.x86_64 110/162 2024-04-06T12:10:27.331 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-volume-2:18.2.2-1086.g052adcad.el8.noarch 108/162 2024-04-06T12:10:27.351 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mon-2:18.2.2-1086.g052adcad.el8.x86_64 111/162 2024-04-06T12:10:27.369 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mon-2:18.2.2-1086.g052adcad.el8.x86_64 111/162 2024-04-06T12:10:27.532 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-osd-2:18.2.2-1086.g052adcad.el8.x86_64 109/162 2024-04-06T12:10:27.750 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-osd-2:18.2.2-1086.g052adcad.el8.x86_64 109/162 2024-04-06T12:10:27.944 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mds-2:18.2.2-1086.g052adcad.el8.x86_64 110/162 2024-04-06T12:10:28.423 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mds-2:18.2.2-1086.g052adcad.el8.x86_64 110/162 2024-04-06T12:10:28.569 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mon-2:18.2.2-1086.g052adcad.el8.x86_64 111/162 2024-04-06T12:10:28.587 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mon-2:18.2.2-1086.g052adcad.el8.x86_64 111/162 2024-04-06T12:10:29.206 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:18.2.2-1086.g052adcad.el8.noarch 112/162 2024-04-06T12:10:29.415 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephadm-2:18.2.2-1086.g052adcad.el8.noarch 112/162 2024-04-06T12:10:29.553 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1086.g052adcad.e 113/162 2024-04-06T12:10:29.753 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1086.g052adcad. 114/162 2024-04-06T12:10:29.951 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1086.g052adcad.el8.noa 115/162 2024-04-06T12:10:30.408 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephadm-2:18.2.2-1086.g052adcad.el8.noarch 112/162 2024-04-06T12:10:30.576 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephadm-2:18.2.2-1086.g052adcad.el8.noarch 112/162 2024-04-06T12:10:30.739 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1086.g052adcad.e 113/162 2024-04-06T12:10:30.906 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1086.g052adcad. 114/162 2024-04-06T12:10:30.984 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1086.g052adcad.el8.noa 115/162 2024-04-06T12:10:31.146 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1086.g052adcad.el8.noa 115/162 2024-04-06T12:10:31.236 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1086.g052adcad.el8.n 116/162 2024-04-06T12:10:32.183 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1086.g052adcad.el8.noa 115/162 2024-04-06T12:10:32.208 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1086.g052adcad.el8.n 116/162 2024-04-06T12:10:32.375 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1086.g052 117/162 2024-04-06T12:10:32.456 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1086.g052adcad.el8.n 116/162 2024-04-06T12:10:32.469 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1086.g052 117/162 2024-04-06T12:10:32.857 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1086.g052adcad.el 118/162 2024-04-06T12:10:33.012 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-2:18.2.2-1086.g052adcad.el8.x86_64 119/162 2024-04-06T12:10:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1086.g052adcad.el8.x86_64 119/162 2024-04-06T12:10:33.225 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-rook-2:18.2.2-1086.g052adcad.el8.noarch 120/162 2024-04-06T12:10:33.237 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1086.g052adcad.el8.noarch 120/162 2024-04-06T12:10:33.428 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1086.g052adcad.el8.n 116/162 2024-04-06T12:10:33.626 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1086.g052 117/162 2024-04-06T12:10:33.722 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1086.g052 117/162 2024-04-06T12:10:34.126 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1086.g052adcad.el 118/162 2024-04-06T12:10:34.289 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-2:18.2.2-1086.g052adcad.el8.x86_64 119/162 2024-04-06T12:10:34.328 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1086.g052adcad.el8.x86_64 119/162 2024-04-06T12:10:34.494 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-2:18.2.2-1086.g052adcad.el8.x86_64 121/162 2024-04-06T12:10:34.495 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-rook-2:18.2.2-1086.g052adcad.el8.noarch 120/162 2024-04-06T12:10:34.507 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1086.g052adcad.el8.noarch 120/162 2024-04-06T12:10:34.686 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-radosgw-2:18.2.2-1086.g052adcad.el8.x86_64 122/162 2024-04-06T12:10:34.721 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1086.g052adcad.el8.x86_64 122/162 2024-04-06T12:10:34.858 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1086.g052ad 123/162 2024-04-06T12:10:34.902 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1086.g052ad 123/162 2024-04-06T12:10:35.059 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephfs-mirror-2:18.2.2-1086.g052adcad.el8.x86_64 124/162 2024-04-06T12:10:35.407 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-1086.g052adcad.el8.x86_64 124/162 2024-04-06T12:10:35.550 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-mirror-2:18.2.2-1086.g052adcad.el8.x86_64 125/162 2024-04-06T12:10:35.775 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-2:18.2.2-1086.g052adcad.el8.x86_64 121/162 2024-04-06T12:10:35.964 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-radosgw-2:18.2.2-1086.g052adcad.el8.x86_64 122/162 2024-04-06T12:10:35.999 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1086.g052adcad.el8.x86_64 122/162 2024-04-06T12:10:36.143 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1086.g052ad 123/162 2024-04-06T12:10:36.186 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1086.g052ad 123/162 2024-04-06T12:10:36.345 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-mirror-2:18.2.2-1086.g052adcad.el8.x86_64 124/162 2024-04-06T12:10:36.694 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-1086.g052adcad.el8.x86_64 124/162 2024-04-06T12:10:36.885 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-mirror-2:18.2.2-1086.g052adcad.el8.x86_64 125/162 2024-04-06T12:10:40.015 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1086.g052adcad.el8.x86_64 125/162 2024-04-06T12:10:40.970 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-test-2:18.2.2-1086.g052adcad.el8.x86_64 126/162 2024-04-06T12:10:41.312 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 127/162 2024-04-06T12:10:41.360 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1086.g052adcad.el8.x86_64 125/162 2024-04-06T12:10:41.582 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 128/162 2024-04-06T12:10:41.934 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-fuse-2:18.2.2-1086.g052adcad.el8.x86_64 129/162 2024-04-06T12:10:42.251 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 130/162 2024-04-06T12:10:42.349 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-test-2:18.2.2-1086.g052adcad.el8.x86_64 126/162 2024-04-06T12:10:42.457 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libtool-2.4.6-25.el8.x86_64 131/162 2024-04-06T12:10:42.470 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 131/162 2024-04-06T12:10:42.665 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 127/162 2024-04-06T12:10:42.701 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-04-06T12:10:42.895 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-04-06T12:10:42.968 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 128/162 2024-04-06T12:10:43.075 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs-devel-2:18.2.2-1086.g052adcad.el8.x86_ 134/162 2024-04-06T12:10:43.317 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-fuse-2:18.2.2-1086.g052adcad.el8.x86_64 129/162 2024-04-06T12:10:43.596 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 130/162 2024-04-06T12:10:43.808 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libtool-2.4.6-25.el8.x86_64 131/162 2024-04-06T12:10:43.821 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 131/162 2024-04-06T12:10:44.054 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-04-06T12:10:44.229 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-04-06T12:10:44.445 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs-devel-2:18.2.2-1086.g052adcad.el8.x86_ 134/162 2024-04-06T12:10:45.480 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-04-06T12:10:45.799 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-04-06T12:10:45.839 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-04-06T12:10:46.256 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-04-06T12:10:46.631 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-04-06T12:10:46.693 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-04-06T12:10:46.852 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-04-06T12:10:46.994 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-04-06T12:10:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephfs-top-2:18.2.2-1086.g052adcad.el8.noarch 139/162 2024-04-06T12:10:47.033 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-04-06T12:10:47.240 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-fuse-2:18.2.2-1086.g052adcad.el8.x86_64 140/162 2024-04-06T12:10:47.453 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-04-06T12:10:47.534 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-nbd-2:18.2.2-1086.g052adcad.el8.x86_64 141/162 2024-04-06T12:10:47.817 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-04-06T12:10:47.825 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-04-06T12:10:48.047 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flex-2.6.1-9.el8.x86_64 143/162 2024-04-06T12:10:48.071 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-04-06T12:10:48.127 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 143/162 2024-04-06T12:10:48.266 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-top-2:18.2.2-1086.g052adcad.el8.noarch 139/162 2024-04-06T12:10:48.356 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : bison-3.0.4-10.el8.x86_64 144/162 2024-04-06T12:10:48.370 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 144/162 2024-04-06T12:10:48.486 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-fuse-2:18.2.2-1086.g052adcad.el8.x86_64 140/162 2024-04-06T12:10:48.590 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-04-06T12:10:48.599 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-04-06T12:10:48.742 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 146/162 2024-04-06T12:10:48.749 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-nbd-2:18.2.2-1086.g052adcad.el8.x86_64 141/162 2024-04-06T12:10:48.937 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-04-06T12:10:49.079 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-04-06T12:10:49.125 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-04-06T12:10:49.293 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-04-06T12:10:49.293 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-04-06T12:10:49.316 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flex-2.6.1-9.el8.x86_64 143/162 2024-04-06T12:10:49.399 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 143/162 2024-04-06T12:10:49.526 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-04-06T12:10:49.526 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-04-06T12:10:49.593 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : bison-3.0.4-10.el8.x86_64 144/162 2024-04-06T12:10:49.608 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 144/162 2024-04-06T12:10:49.735 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-04-06T12:10:49.735 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-04-06T12:10:49.801 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-04-06T12:10:49.811 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-04-06T12:10:49.937 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 146/162 2024-04-06T12:10:49.996 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-04-06T12:10:50.149 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-04-06T12:10:50.208 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-04-06T12:10:50.342 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-04-06T12:10:50.391 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-04-06T12:10:50.469 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-04-06T12:10:50.469 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-04-06T12:10:50.633 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-04-06T12:10:50.633 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-04-06T12:10:50.760 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-04-06T12:10:50.760 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-04-06T12:10:50.884 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-04-06T12:10:50.884 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-04-06T12:10:50.988 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-04-06T12:10:50.988 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-04-06T12:10:51.133 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-04-06T12:10:51.134 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-04-06T12:10:51.249 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-04-06T12:10:51.358 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-04-06T12:10:51.359 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-04-06T12:10:51.484 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-04-06T12:10:51.591 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-04-06T12:10:51.591 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-04-06T12:10:51.653 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-04-06T12:10:51.841 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-04-06T12:10:51.911 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-04-06T12:10:51.911 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-04-06T12:10:52.017 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-04-06T12:10:52.017 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-04-06T12:10:52.145 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-04-06T12:10:52.145 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-04-06T12:10:52.345 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-04-06T12:10:52.345 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-04-06T12:10:52.603 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-04-06T12:10:52.604 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-04-06T12:10:52.834 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-04-06T12:10:52.835 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-04-06T12:10:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-04-06T12:10:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-2:18.2.2-1086.g052adcad.el8.x86_64 1/162 2024-04-06T12:10:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-base-2:18.2.2-1086.g052adcad.el8.x86_64 2/162 2024-04-06T12:10:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-common-2:18.2.2-1086.g052adcad.el8.x86_64 3/162 2024-04-06T12:10:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-fuse-2:18.2.2-1086.g052adcad.el8.x86_64 4/162 2024-04-06T12:10:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1086.g052ad 5/162 2024-04-06T12:10:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mds-2:18.2.2-1086.g052adcad.el8.x86_64 6/162 2024-04-06T12:10:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-2:18.2.2-1086.g052adcad.el8.x86_64 7/162 2024-04-06T12:10:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mon-2:18.2.2-1086.g052adcad.el8.x86_64 8/162 2024-04-06T12:10:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-osd-2:18.2.2-1086.g052adcad.el8.x86_64 9/162 2024-04-06T12:10:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-radosgw-2:18.2.2-1086.g052adcad.el8.x86_64 10/162 2024-04-06T12:10:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-selinux-2:18.2.2-1086.g052adcad.el8.x86_64 11/162 2024-04-06T12:10:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-test-2:18.2.2-1086.g052adcad.el8.x86_64 12/162 2024-04-06T12:10:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephfs-mirror-2:18.2.2-1086.g052adcad.el8.x86_64 13/162 2024-04-06T12:10:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs-devel-2:18.2.2-1086.g052adcad.el8.x86_ 14/162 2024-04-06T12:10:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs2-2:18.2.2-1086.g052adcad.el8.x86_64 15/162 2024-04-06T12:10:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephsqlite-2:18.2.2-1086.g052adcad.el8.x86_64 16/162 2024-04-06T12:10:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados-devel-2:18.2.2-1086.g052adcad.el8.x86_6 17/162 2024-04-06T12:10:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libradosstriper1-2:18.2.2-1086.g052adcad.el8.x86 18/162 2024-04-06T12:10:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librgw2-2:18.2.2-1086.g052adcad.el8.x86_64 19/162 2024-04-06T12:10:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1086.g052adcad.el 20/162 2024-04-06T12:10:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-common-2:18.2.2-1086.g052adcad.el8. 21/162 2024-04-06T12:10:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cephfs-2:18.2.2-1086.g052adcad.el8.x86_6 22/162 2024-04-06T12:10:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rados-2:18.2.2-1086.g052adcad.el8.x86_64 23/162 2024-04-06T12:10:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rbd-2:18.2.2-1086.g052adcad.el8.x86_64 24/162 2024-04-06T12:10:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rgw-2:18.2.2-1086.g052adcad.el8.x86_64 25/162 2024-04-06T12:10:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-fuse-2:18.2.2-1086.g052adcad.el8.x86_64 26/162 2024-04-06T12:10:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-mirror-2:18.2.2-1086.g052adcad.el8.x86_64 27/162 2024-04-06T12:10:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-nbd-2:18.2.2-1086.g052adcad.el8.x86_64 28/162 2024-04-06T12:10:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1086.g052adcad. 29/162 2024-04-06T12:10:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1086.g052adcad.el8.noa 30/162 2024-04-06T12:10:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1086.g052adcad.el8.n 31/162 2024-04-06T12:10:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1086.g052 32/162 2024-04-06T12:10:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1086.g052adcad.el 33/162 2024-04-06T12:10:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1086.g052adcad.el8.noarch 34/162 2024-04-06T12:10:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1086.g052adcad.e 35/162 2024-04-06T12:10:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-volume-2:18.2.2-1086.g052adcad.el8.noarch 36/162 2024-04-06T12:10:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephadm-2:18.2.2-1086.g052adcad.el8.noarch 37/162 2024-04-06T12:10:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephfs-top-2:18.2.2-1086.g052adcad.el8.noarch 38/162 2024-04-06T12:10:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-04-06T12:10:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-04-06T12:10:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-04-06T12:10:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-04-06T12:10:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-04-06T12:10:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-04-06T12:10:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-04-06T12:10:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-04-06T12:10:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-04-06T12:10:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-04-06T12:10:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-04-06T12:10:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-04-06T12:10:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-04-06T12:10:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-04-06T12:10:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-04-06T12:10:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-04-06T12:10:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-04-06T12:10:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-04-06T12:10:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-04-06T12:10:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-04-06T12:10:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-04-06T12:10:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-04-06T12:10:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-04-06T12:10:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-04-06T12:10:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-04-06T12:10:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-04-06T12:10:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-04-06T12:10:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-04-06T12:10:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-04-06T12:10:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-04-06T12:10:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-04-06T12:10:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-04-06T12:10:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-04-06T12:10:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 72/162 2024-04-06T12:10:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 73/162 2024-04-06T12:10:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/162 2024-04-06T12:10:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 75/162 2024-04-06T12:10:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 76/162 2024-04-06T12:10:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 77/162 2024-04-06T12:10:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 78/162 2024-04-06T12:10:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 79/162 2024-04-06T12:10:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 80/162 2024-04-06T12:10:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 81/162 2024-04-06T12:10:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 82/162 2024-04-06T12:10:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 83/162 2024-04-06T12:10:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 84/162 2024-04-06T12:10:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 85/162 2024-04-06T12:10:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 86/162 2024-04-06T12:10:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 87/162 2024-04-06T12:10:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 88/162 2024-04-06T12:10:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 89/162 2024-04-06T12:10:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 90/162 2024-04-06T12:10:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 91/162 2024-04-06T12:10:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 92/162 2024-04-06T12:10:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 93/162 2024-04-06T12:10:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 94/162 2024-04-06T12:10:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 95/162 2024-04-06T12:10:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 96/162 2024-04-06T12:10:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 97/162 2024-04-06T12:10:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 98/162 2024-04-06T12:10:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 99/162 2024-04-06T12:10:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 100/162 2024-04-06T12:10:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 101/162 2024-04-06T12:10:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 102/162 2024-04-06T12:10:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 103/162 2024-04-06T12:10:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 104/162 2024-04-06T12:10:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 105/162 2024-04-06T12:10:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : bison-3.0.4-10.el8.x86_64 106/162 2024-04-06T12:10:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flex-2.6.1-9.el8.x86_64 107/162 2024-04-06T12:10:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 108/162 2024-04-06T12:10:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 109/162 2024-04-06T12:10:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 110/162 2024-04-06T12:10:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 111/162 2024-04-06T12:10:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 112/162 2024-04-06T12:10:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 113/162 2024-04-06T12:10:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 114/162 2024-04-06T12:10:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 115/162 2024-04-06T12:10:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 116/162 2024-04-06T12:10:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 117/162 2024-04-06T12:10:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 118/162 2024-04-06T12:10:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 119/162 2024-04-06T12:10:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 120/162 2024-04-06T12:10:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 121/162 2024-04-06T12:10:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : automake-1.16.1-8.el8.noarch 122/162 2024-04-06T12:10:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libpq-13.11-1.el8.x86_64 123/162 2024-04-06T12:10:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : jq-1.6-7.el8.x86_64 124/162 2024-04-06T12:10:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 125/162 2024-04-06T12:10:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 126/162 2024-04-06T12:10:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 127/162 2024-04-06T12:10:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 128/162 2024-04-06T12:10:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 129/162 2024-04-06T12:10:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-server-10.23-4.module+el8.9.0+21317+d 130/162 2024-04-06T12:10:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:18.2.2-1086.g052adcad.el8.x86_64 131/162 2024-04-06T12:10:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-04-06T12:10:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:18.2.2-1086.g052adcad.el8.x86_64 133/162 2024-04-06T12:10:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-04-06T12:10:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-04-06T12:10:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-04-06T12:10:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-04-06T12:10:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-04-06T12:10:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-04-06T12:10:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-04-06T12:10:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-04-06T12:10:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-04-06T12:10:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-04-06T12:10:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-04-06T12:10:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-04-06T12:10:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-04-06T12:10:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-04-06T12:10:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-04-06T12:10:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-04-06T12:10:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-04-06T12:10:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-04-06T12:10:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-04-06T12:10:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-04-06T12:10:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-04-06T12:10:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-04-06T12:10:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-04-06T12:10:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-04-06T12:10:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-04-06T12:10:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-04-06T12:10:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-04-06T12:10:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-04-06T12:10:53.084 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-04-06T12:10:53.243 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-04-06T12:10:53.243 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-04-06T12:10:54.203 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-04-06T12:10:54.203 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-2:18.2.2-1086.g052adcad.el8.x86_64 1/162 2024-04-06T12:10:54.203 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-base-2:18.2.2-1086.g052adcad.el8.x86_64 2/162 2024-04-06T12:10:54.203 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-common-2:18.2.2-1086.g052adcad.el8.x86_64 3/162 2024-04-06T12:10:54.204 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-fuse-2:18.2.2-1086.g052adcad.el8.x86_64 4/162 2024-04-06T12:10:54.204 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1086.g052ad 5/162 2024-04-06T12:10:54.204 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mds-2:18.2.2-1086.g052adcad.el8.x86_64 6/162 2024-04-06T12:10:54.204 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-2:18.2.2-1086.g052adcad.el8.x86_64 7/162 2024-04-06T12:10:54.204 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mon-2:18.2.2-1086.g052adcad.el8.x86_64 8/162 2024-04-06T12:10:54.204 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-osd-2:18.2.2-1086.g052adcad.el8.x86_64 9/162 2024-04-06T12:10:54.204 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-radosgw-2:18.2.2-1086.g052adcad.el8.x86_64 10/162 2024-04-06T12:10:54.204 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-selinux-2:18.2.2-1086.g052adcad.el8.x86_64 11/162 2024-04-06T12:10:54.204 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-test-2:18.2.2-1086.g052adcad.el8.x86_64 12/162 2024-04-06T12:10:54.204 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-mirror-2:18.2.2-1086.g052adcad.el8.x86_64 13/162 2024-04-06T12:10:54.204 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs-devel-2:18.2.2-1086.g052adcad.el8.x86_ 14/162 2024-04-06T12:10:54.204 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs2-2:18.2.2-1086.g052adcad.el8.x86_64 15/162 2024-04-06T12:10:54.204 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephsqlite-2:18.2.2-1086.g052adcad.el8.x86_64 16/162 2024-04-06T12:10:54.204 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados-devel-2:18.2.2-1086.g052adcad.el8.x86_6 17/162 2024-04-06T12:10:54.205 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libradosstriper1-2:18.2.2-1086.g052adcad.el8.x86 18/162 2024-04-06T12:10:54.205 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librgw2-2:18.2.2-1086.g052adcad.el8.x86_64 19/162 2024-04-06T12:10:54.205 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1086.g052adcad.el 20/162 2024-04-06T12:10:54.205 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-common-2:18.2.2-1086.g052adcad.el8. 21/162 2024-04-06T12:10:54.205 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cephfs-2:18.2.2-1086.g052adcad.el8.x86_6 22/162 2024-04-06T12:10:54.205 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rados-2:18.2.2-1086.g052adcad.el8.x86_64 23/162 2024-04-06T12:10:54.205 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rbd-2:18.2.2-1086.g052adcad.el8.x86_64 24/162 2024-04-06T12:10:54.205 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rgw-2:18.2.2-1086.g052adcad.el8.x86_64 25/162 2024-04-06T12:10:54.205 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-fuse-2:18.2.2-1086.g052adcad.el8.x86_64 26/162 2024-04-06T12:10:54.205 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-mirror-2:18.2.2-1086.g052adcad.el8.x86_64 27/162 2024-04-06T12:10:54.205 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-nbd-2:18.2.2-1086.g052adcad.el8.x86_64 28/162 2024-04-06T12:10:54.205 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1086.g052adcad. 29/162 2024-04-06T12:10:54.205 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1086.g052adcad.el8.noa 30/162 2024-04-06T12:10:54.205 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1086.g052adcad.el8.n 31/162 2024-04-06T12:10:54.205 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1086.g052 32/162 2024-04-06T12:10:54.206 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1086.g052adcad.el 33/162 2024-04-06T12:10:54.206 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1086.g052adcad.el8.noarch 34/162 2024-04-06T12:10:54.206 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1086.g052adcad.e 35/162 2024-04-06T12:10:54.206 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-volume-2:18.2.2-1086.g052adcad.el8.noarch 36/162 2024-04-06T12:10:54.206 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephadm-2:18.2.2-1086.g052adcad.el8.noarch 37/162 2024-04-06T12:10:54.206 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-top-2:18.2.2-1086.g052adcad.el8.noarch 38/162 2024-04-06T12:10:54.206 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-04-06T12:10:54.206 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-04-06T12:10:54.206 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-04-06T12:10:54.206 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-04-06T12:10:54.206 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-04-06T12:10:54.206 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-04-06T12:10:54.207 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-04-06T12:10:54.207 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-04-06T12:10:54.207 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-04-06T12:10:54.207 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-04-06T12:10:54.207 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-04-06T12:10:54.207 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-04-06T12:10:54.207 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-04-06T12:10:54.207 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-04-06T12:10:54.207 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-04-06T12:10:54.207 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-04-06T12:10:54.207 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-04-06T12:10:54.207 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-04-06T12:10:54.207 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-04-06T12:10:54.207 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-04-06T12:10:54.208 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-04-06T12:10:54.208 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-04-06T12:10:54.208 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-04-06T12:10:54.208 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-04-06T12:10:54.208 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-04-06T12:10:54.208 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-04-06T12:10:54.208 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-04-06T12:10:54.208 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-04-06T12:10:54.208 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-04-06T12:10:54.208 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-04-06T12:10:54.208 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-04-06T12:10:54.208 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-04-06T12:10:54.208 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-04-06T12:10:54.208 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 72/162 2024-04-06T12:10:54.209 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/162 2024-04-06T12:10:54.209 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/162 2024-04-06T12:10:54.209 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 75/162 2024-04-06T12:10:54.209 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 76/162 2024-04-06T12:10:54.209 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 77/162 2024-04-06T12:10:54.209 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 78/162 2024-04-06T12:10:54.209 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 79/162 2024-04-06T12:10:54.209 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/162 2024-04-06T12:10:54.209 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 81/162 2024-04-06T12:10:54.209 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 82/162 2024-04-06T12:10:54.209 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 83/162 2024-04-06T12:10:54.209 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 84/162 2024-04-06T12:10:54.209 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 85/162 2024-04-06T12:10:54.209 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 86/162 2024-04-06T12:10:54.209 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 87/162 2024-04-06T12:10:54.210 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 88/162 2024-04-06T12:10:54.210 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 89/162 2024-04-06T12:10:54.210 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 90/162 2024-04-06T12:10:54.210 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 91/162 2024-04-06T12:10:54.210 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 92/162 2024-04-06T12:10:54.210 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 93/162 2024-04-06T12:10:54.210 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 94/162 2024-04-06T12:10:54.210 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 95/162 2024-04-06T12:10:54.210 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 96/162 2024-04-06T12:10:54.210 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 97/162 2024-04-06T12:10:54.210 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 98/162 2024-04-06T12:10:54.210 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 99/162 2024-04-06T12:10:54.210 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 100/162 2024-04-06T12:10:54.210 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 101/162 2024-04-06T12:10:54.211 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 102/162 2024-04-06T12:10:54.211 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 103/162 2024-04-06T12:10:54.211 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 104/162 2024-04-06T12:10:54.211 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 105/162 2024-04-06T12:10:54.211 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : bison-3.0.4-10.el8.x86_64 106/162 2024-04-06T12:10:54.211 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flex-2.6.1-9.el8.x86_64 107/162 2024-04-06T12:10:54.211 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 108/162 2024-04-06T12:10:54.211 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 109/162 2024-04-06T12:10:54.211 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 110/162 2024-04-06T12:10:54.211 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 111/162 2024-04-06T12:10:54.211 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 112/162 2024-04-06T12:10:54.211 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 113/162 2024-04-06T12:10:54.211 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 114/162 2024-04-06T12:10:54.211 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 115/162 2024-04-06T12:10:54.212 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 116/162 2024-04-06T12:10:54.212 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 117/162 2024-04-06T12:10:54.212 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 118/162 2024-04-06T12:10:54.212 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 119/162 2024-04-06T12:10:54.212 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 120/162 2024-04-06T12:10:54.212 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 121/162 2024-04-06T12:10:54.212 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : automake-1.16.1-8.el8.noarch 122/162 2024-04-06T12:10:54.212 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libpq-13.11-1.el8.x86_64 123/162 2024-04-06T12:10:54.212 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : jq-1.6-7.el8.x86_64 124/162 2024-04-06T12:10:54.212 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 125/162 2024-04-06T12:10:54.212 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 126/162 2024-04-06T12:10:54.212 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 127/162 2024-04-06T12:10:54.212 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 128/162 2024-04-06T12:10:54.212 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 129/162 2024-04-06T12:10:54.213 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-server-10.23-4.module+el8.9.0+21317+d 130/162 2024-04-06T12:10:54.213 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:18.2.2-1086.g052adcad.el8.x86_64 131/162 2024-04-06T12:10:54.213 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-04-06T12:10:54.213 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:18.2.2-1086.g052adcad.el8.x86_64 133/162 2024-04-06T12:10:54.213 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-04-06T12:10:54.213 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-04-06T12:10:54.213 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-04-06T12:10:54.213 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-04-06T12:10:54.213 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-04-06T12:10:54.213 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-04-06T12:10:54.213 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-04-06T12:10:54.213 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-04-06T12:10:54.213 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-04-06T12:10:54.213 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-04-06T12:10:54.213 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-04-06T12:10:54.214 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-04-06T12:10:54.214 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-04-06T12:10:54.214 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-04-06T12:10:54.214 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-04-06T12:10:54.214 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-04-06T12:10:54.214 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-04-06T12:10:54.214 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-04-06T12:10:54.214 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-04-06T12:10:54.214 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-04-06T12:10:54.214 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-04-06T12:10:54.214 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-04-06T12:10:54.214 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-04-06T12:10:54.214 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-04-06T12:10:54.214 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-04-06T12:10:54.215 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-04-06T12:10:54.215 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-04-06T12:10:54.215 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-04-06T12:11:06.784 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-04-06T12:11:06.784 INFO:teuthology.orchestra.run.smithi189.stdout:Installed products updated. 2024-04-06T12:11:07.455 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-04-06T12:11:07.455 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2024-04-06T12:11:08.052 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:08.052 INFO:teuthology.orchestra.run.smithi189.stdout:Upgraded: 2024-04-06T12:11:08.052 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-04-06T12:11:08.052 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-04-06T12:11:08.052 INFO:teuthology.orchestra.run.smithi189.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-04-06T12:11:08.052 INFO:teuthology.orchestra.run.smithi189.stdout: libblkid-2.32.1-43.el8.x86_64 2024-04-06T12:11:08.052 INFO:teuthology.orchestra.run.smithi189.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-04-06T12:11:08.052 INFO:teuthology.orchestra.run.smithi189.stdout: libmount-2.32.1-43.el8.x86_64 2024-04-06T12:11:08.052 INFO:teuthology.orchestra.run.smithi189.stdout: librados2-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.052 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.053 INFO:teuthology.orchestra.run.smithi189.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-04-06T12:11:08.053 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-2.32.1-43.el8.x86_64 2024-04-06T12:11:08.053 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-04-06T12:11:08.053 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-04-06T12:11:08.053 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux-2.32.1-43.el8.x86_64 2024-04-06T12:11:08.053 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-04-06T12:11:08.053 INFO:teuthology.orchestra.run.smithi189.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-04-06T12:11:08.053 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-04-06T12:11:08.053 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-06T12:11:08.053 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-04-06T12:11:08.053 INFO:teuthology.orchestra.run.smithi189.stdout: automake-1.16.1-8.el8.noarch 2024-04-06T12:11:08.053 INFO:teuthology.orchestra.run.smithi189.stdout: bison-3.0.4-10.el8.x86_64 2024-04-06T12:11:08.053 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.053 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.054 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.054 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.054 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.054 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.054 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.054 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.054 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.054 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.054 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.054 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.054 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.054 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.054 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.055 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.055 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.055 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.055 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.055 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.055 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.055 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-mirror-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.055 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-top-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.055 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-04-06T12:11:08.055 INFO:teuthology.orchestra.run.smithi189.stdout: flex-2.6.1-9.el8.x86_64 2024-04-06T12:11:08.055 INFO:teuthology.orchestra.run.smithi189.stdout: fuse-2.9.7-17.el8.x86_64 2024-04-06T12:11:08.055 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-06T12:11:08.055 INFO:teuthology.orchestra.run.smithi189.stdout: jq-1.6-7.el8.x86_64 2024-04-06T12:11:08.055 INFO:teuthology.orchestra.run.smithi189.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-04-06T12:11:08.056 INFO:teuthology.orchestra.run.smithi189.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-04-06T12:11:08.056 INFO:teuthology.orchestra.run.smithi189.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-04-06T12:11:08.056 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.056 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.056 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.056 INFO:teuthology.orchestra.run.smithi189.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-04-06T12:11:08.056 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-06T12:11:08.056 INFO:teuthology.orchestra.run.smithi189.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-06T12:11:08.056 INFO:teuthology.orchestra.run.smithi189.stdout: libpq-13.11-1.el8.x86_64 2024-04-06T12:11:08.056 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-06T12:11:08.056 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-06T12:11:08.056 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.056 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.057 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-06T12:11:08.057 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.057 INFO:teuthology.orchestra.run.smithi189.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-04-06T12:11:08.057 INFO:teuthology.orchestra.run.smithi189.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-04-06T12:11:08.057 INFO:teuthology.orchestra.run.smithi189.stdout: libtool-2.4.6-25.el8.x86_64 2024-04-06T12:11:08.057 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-06T12:11:08.057 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-04-06T12:11:08.057 INFO:teuthology.orchestra.run.smithi189.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-04-06T12:11:08.057 INFO:teuthology.orchestra.run.smithi189.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-04-06T12:11:08.057 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-06T12:11:08.057 INFO:teuthology.orchestra.run.smithi189.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-06T12:11:08.057 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-06T12:11:08.057 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-06T12:11:08.058 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-04-06T12:11:08.058 INFO:teuthology.orchestra.run.smithi189.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-04-06T12:11:08.058 INFO:teuthology.orchestra.run.smithi189.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-04-06T12:11:08.058 INFO:teuthology.orchestra.run.smithi189.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-04-06T12:11:08.058 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-04-06T12:11:08.058 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-contrib-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-04-06T12:11:08.058 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-server-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-04-06T12:11:08.058 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-06T12:11:08.058 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-06T12:11:08.058 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-06T12:11:08.058 INFO:teuthology.orchestra.run.smithi189.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-06T12:11:08.058 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-06T12:11:08.058 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-06T12:11:08.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-06T12:11:08.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-06T12:11:08.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-06T12:11:08.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-06T12:11:08.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-06T12:11:08.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-06T12:11:08.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-06T12:11:08.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-06T12:11:08.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-06T12:11:08.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-06T12:11:08.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-06T12:11:08.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-06T12:11:08.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-06T12:11:08.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-06T12:11:08.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-06T12:11:08.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-06T12:11:08.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-06T12:11:08.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-06T12:11:08.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-06T12:11:08.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-06T12:11:08.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-06T12:11:08.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-06T12:11:08.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend-2.6-1.el8.noarch 2024-04-06T12:11:08.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-06T12:11:08.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-06T12:11:08.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-06T12:11:08.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-06T12:11:08.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-06T12:11:08.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-06T12:11:08.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-06T12:11:08.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-06T12:11:08.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-06T12:11:08.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-06T12:11:08.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-06T12:11:08.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-06T12:11:08.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-06T12:11:08.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-06T12:11:08.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-06T12:11:08.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-06T12:11:08.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-06T12:11:08.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-06T12:11:08.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-06T12:11:08.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-06T12:11:08.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-06T12:11:08.062 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.062 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.062 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.063 INFO:teuthology.orchestra.run.smithi189.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-06T12:11:08.063 INFO:teuthology.orchestra.run.smithi189.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-06T12:11:08.063 INFO:teuthology.orchestra.run.smithi189.stdout: uuid-1.6.2-43.el8.x86_64 2024-04-06T12:11:08.063 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-04-06T12:11:08.063 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-06T12:11:08.063 INFO:teuthology.orchestra.run.smithi189.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-04-06T12:11:08.063 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:08.063 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-06T12:11:08.073 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:08.073 INFO:teuthology.orchestra.run.smithi192.stdout:Upgraded: 2024-04-06T12:11:08.073 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-04-06T12:11:08.073 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-04-06T12:11:08.073 INFO:teuthology.orchestra.run.smithi192.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-04-06T12:11:08.073 INFO:teuthology.orchestra.run.smithi192.stdout: libblkid-2.32.1-43.el8.x86_64 2024-04-06T12:11:08.073 INFO:teuthology.orchestra.run.smithi192.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-04-06T12:11:08.073 INFO:teuthology.orchestra.run.smithi192.stdout: libmount-2.32.1-43.el8.x86_64 2024-04-06T12:11:08.073 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.073 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.073 INFO:teuthology.orchestra.run.smithi192.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-04-06T12:11:08.074 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid-2.32.1-43.el8.x86_64 2024-04-06T12:11:08.074 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-04-06T12:11:08.074 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-04-06T12:11:08.074 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux-2.32.1-43.el8.x86_64 2024-04-06T12:11:08.074 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-04-06T12:11:08.074 INFO:teuthology.orchestra.run.smithi192.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-04-06T12:11:08.074 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-04-06T12:11:08.074 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-06T12:11:08.074 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-04-06T12:11:08.074 INFO:teuthology.orchestra.run.smithi192.stdout: automake-1.16.1-8.el8.noarch 2024-04-06T12:11:08.074 INFO:teuthology.orchestra.run.smithi192.stdout: bison-3.0.4-10.el8.x86_64 2024-04-06T12:11:08.074 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.074 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.074 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.075 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.075 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.075 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.075 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.075 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.075 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.075 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.075 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.075 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.075 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.075 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.075 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.075 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.076 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.076 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.076 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.076 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.076 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.076 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-mirror-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.076 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-top-2:18.2.2-1086.g052adcad.el8.noarch 2024-04-06T12:11:08.076 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-04-06T12:11:08.076 INFO:teuthology.orchestra.run.smithi192.stdout: flex-2.6.1-9.el8.x86_64 2024-04-06T12:11:08.076 INFO:teuthology.orchestra.run.smithi192.stdout: fuse-2.9.7-17.el8.x86_64 2024-04-06T12:11:08.076 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-06T12:11:08.076 INFO:teuthology.orchestra.run.smithi192.stdout: jq-1.6-7.el8.x86_64 2024-04-06T12:11:08.076 INFO:teuthology.orchestra.run.smithi192.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-04-06T12:11:08.076 INFO:teuthology.orchestra.run.smithi192.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-04-06T12:11:08.077 INFO:teuthology.orchestra.run.smithi192.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-04-06T12:11:08.077 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.077 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.077 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.077 INFO:teuthology.orchestra.run.smithi192.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-04-06T12:11:08.077 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-06T12:11:08.077 INFO:teuthology.orchestra.run.smithi192.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-06T12:11:08.077 INFO:teuthology.orchestra.run.smithi192.stdout: libpq-13.11-1.el8.x86_64 2024-04-06T12:11:08.077 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-06T12:11:08.077 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-06T12:11:08.077 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.077 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.077 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-06T12:11:08.077 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.078 INFO:teuthology.orchestra.run.smithi192.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-04-06T12:11:08.078 INFO:teuthology.orchestra.run.smithi192.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-04-06T12:11:08.078 INFO:teuthology.orchestra.run.smithi192.stdout: libtool-2.4.6-25.el8.x86_64 2024-04-06T12:11:08.078 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-06T12:11:08.078 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-04-06T12:11:08.078 INFO:teuthology.orchestra.run.smithi192.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-04-06T12:11:08.078 INFO:teuthology.orchestra.run.smithi192.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-04-06T12:11:08.078 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-06T12:11:08.078 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-06T12:11:08.078 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-06T12:11:08.078 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-06T12:11:08.078 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-04-06T12:11:08.078 INFO:teuthology.orchestra.run.smithi192.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-04-06T12:11:08.079 INFO:teuthology.orchestra.run.smithi192.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-04-06T12:11:08.079 INFO:teuthology.orchestra.run.smithi192.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-04-06T12:11:08.079 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-04-06T12:11:08.079 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-contrib-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-04-06T12:11:08.079 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-server-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-04-06T12:11:08.079 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-06T12:11:08.079 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-06T12:11:08.079 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-06T12:11:08.079 INFO:teuthology.orchestra.run.smithi192.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-06T12:11:08.079 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-06T12:11:08.079 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.079 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.079 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.079 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-06T12:11:08.079 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-06T12:11:08.080 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-06T12:11:08.080 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-06T12:11:08.080 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-06T12:11:08.080 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-06T12:11:08.080 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-06T12:11:08.080 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-06T12:11:08.080 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-06T12:11:08.080 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-06T12:11:08.080 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-06T12:11:08.080 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-06T12:11:08.080 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-06T12:11:08.080 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-06T12:11:08.080 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-06T12:11:08.080 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-06T12:11:08.081 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-06T12:11:08.081 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-06T12:11:08.081 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-06T12:11:08.081 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-06T12:11:08.081 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-06T12:11:08.081 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-06T12:11:08.081 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-06T12:11:08.081 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend-2.6-1.el8.noarch 2024-04-06T12:11:08.081 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-06T12:11:08.081 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-06T12:11:08.081 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-06T12:11:08.081 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-06T12:11:08.081 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-06T12:11:08.081 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-06T12:11:08.082 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.082 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.082 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-06T12:11:08.082 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-06T12:11:08.082 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.082 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-06T12:11:08.082 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-06T12:11:08.082 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-06T12:11:08.082 INFO:teuthology.orchestra.run.smithi192.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-06T12:11:08.082 INFO:teuthology.orchestra.run.smithi192.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-06T12:11:08.082 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-06T12:11:08.082 INFO:teuthology.orchestra.run.smithi192.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-06T12:11:08.082 INFO:teuthology.orchestra.run.smithi192.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-06T12:11:08.083 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-06T12:11:08.083 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-06T12:11:08.083 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-06T12:11:08.083 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-06T12:11:08.083 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-06T12:11:08.083 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.083 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.083 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-2:18.2.2-1086.g052adcad.el8.x86_64 2024-04-06T12:11:08.083 INFO:teuthology.orchestra.run.smithi192.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-06T12:11:08.083 INFO:teuthology.orchestra.run.smithi192.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-06T12:11:08.083 INFO:teuthology.orchestra.run.smithi192.stdout: uuid-1.6.2-43.el8.x86_64 2024-04-06T12:11:08.083 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-04-06T12:11:08.083 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-06T12:11:08.084 INFO:teuthology.orchestra.run.smithi192.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-04-06T12:11:08.084 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:08.084 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-06T12:11:08.289 DEBUG:teuthology.parallel:result is None 2024-04-06T12:11:08.332 DEBUG:teuthology.parallel:result is None 2024-04-06T12:11:08.333 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T12:11:08.486 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-06T12:11:08.509 INFO:teuthology.orchestra.run.smithi189.stdout:18.2.2-1086.g052adcad.el8 2024-04-06T12:11:08.509 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1086.g052adcad.el8 2024-04-06T12:11:08.510 INFO:teuthology.task.install:The correct ceph version 18.2.2-1086.g052adcad is installed. 2024-04-06T12:11:08.512 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T12:11:08.728 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-06T12:11:08.752 INFO:teuthology.orchestra.run.smithi192.stdout:18.2.2-1086.g052adcad.el8 2024-04-06T12:11:08.752 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1086.g052adcad.el8 2024-04-06T12:11:08.752 INFO:teuthology.task.install:The correct ceph version 18.2.2-1086.g052adcad is installed. 2024-04-06T12:11:08.754 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-06T12:11:08.754 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:08.754 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-06T12:11:08.798 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:08.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-06T12:11:08.839 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-06T12:11:08.840 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:08.840 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/daemon-helper 2024-04-06T12:11:08.867 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-06T12:11:08.935 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:08.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2024-04-06T12:11:08.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-06T12:11:09.031 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-06T12:11:09.031 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:09.031 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-06T12:11:09.059 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-06T12:11:09.126 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:09.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-06T12:11:09.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-06T12:11:09.221 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-06T12:11:09.222 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:09.222 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/stdin-killer 2024-04-06T12:11:09.249 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-06T12:11:09.315 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:09.315 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2024-04-06T12:11:09.341 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-06T12:11:09.409 INFO:teuthology.run_tasks:Running task ceph... 2024-04-06T12:11:09.485 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-06T12:11:09.485 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 777 /var/log/ceph 2024-04-06T12:11:09.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /var/log/ceph 2024-04-06T12:11:09.513 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-06T12:11:09.514 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-06T12:11:09.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-06T12:11:09.582 INFO:tasks.ceph:Creating extra log directories... 2024-04-06T12:11:09.582 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-06T12:11:09.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-06T12:11:09.658 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-06T12:11:09.659 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-06T12:11:09.659 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-06T12:11:09.659 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:09.659 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-06T12:11:09.703 DEBUG:teuthology.orchestra.run.smithi189:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-06T12:11:09.818 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-06T12:11:09.845 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:09.845 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-06T12:11:09.862 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-06T12:11:09.975 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-06T12:11:10.007 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-06T12:11:10.007 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-06T12:11:10.008 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7638946', 'branch': 'reef', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7638946', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': '800ec6d64a7fa0648ad53edfc6e50b41fc4cd674'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 3144, 'sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46', 'sleep_before_teardown': 0, 'subset': '362/512', 'suite': 'fs', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '800ec6d64a7fa0648ad53edfc6e50b41fc4cd674', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi189.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOXPuN01E+uZRPA3yUcIGKyxrMNnqr19RUH/qeLjk0BP0SOkrpy/5Y/DlQM9mxutjjN3CWjMmARBPcUTj2zBRV0=', 'smithi192.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNuSqPRtJ6bQFn5hEQRh6snW6a/7ZpRBA3dX92nc3WrdVqhFxOKLgJ9IREoVgDeyxeuIF4P2zNIcRk9MH9o9b2I='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'd7d6e60867cadf19fbfd7655a551ae7d58060757', 'timestamp': '2024-04-03_22:24:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-06T12:11:10.008 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-06T12:11:10.010 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-06T12:11:10.058 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m0777 -- /var/run/ceph 2024-04-06T12:11:10.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/run/ceph 2024-04-06T12:11:10.126 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:10.126 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-04-06T12:11:10.150 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-06T12:11:10.150 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_1 2024-04-06T12:11:10.207 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-06T12:11:10.207 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T12:11:10.207 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 177738 Links: 1 2024-04-06T12:11:10.207 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T12:11:10.207 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T12:11:10.207 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-06 12:10:53.807336045 +0000 2024-04-06T12:11:10.207 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-06 12:07:31.206094459 +0000 2024-04-06T12:11:10.207 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-06 12:07:31.206094459 +0000 2024-04-06T12:11:10.207 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: - 2024-04-06T12:11:10.208 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-06T12:11:10.272 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-06T12:11:10.272 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-06T12:11:10.273 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000275218 s, 1.9 MB/s 2024-04-06T12:11:10.274 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-06T12:11:10.332 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_2 2024-04-06T12:11:10.389 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-06T12:11:10.389 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T12:11:10.389 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 183376 Links: 1 2024-04-06T12:11:10.389 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T12:11:10.389 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T12:11:10.389 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-06 12:10:53.807336045 +0000 2024-04-06T12:11:10.389 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-06 12:07:31.560087502 +0000 2024-04-06T12:11:10.389 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-06 12:07:31.560087502 +0000 2024-04-06T12:11:10.389 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: - 2024-04-06T12:11:10.390 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-06T12:11:10.454 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-06T12:11:10.454 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-06T12:11:10.454 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000317843 s, 1.6 MB/s 2024-04-06T12:11:10.456 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-06T12:11:10.514 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_3 2024-04-06T12:11:10.571 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-06T12:11:10.571 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T12:11:10.571 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 181458 Links: 1 2024-04-06T12:11:10.571 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T12:11:10.571 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T12:11:10.571 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-06 12:10:53.807336045 +0000 2024-04-06T12:11:10.571 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-06 12:07:31.892080978 +0000 2024-04-06T12:11:10.571 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-06 12:07:31.892080978 +0000 2024-04-06T12:11:10.571 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: - 2024-04-06T12:11:10.572 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-06T12:11:10.636 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-06T12:11:10.637 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-06T12:11:10.637 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000192999 s, 2.7 MB/s 2024-04-06T12:11:10.638 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-06T12:11:10.695 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_4 2024-04-06T12:11:10.753 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-06T12:11:10.753 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T12:11:10.753 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 177838 Links: 1 2024-04-06T12:11:10.753 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T12:11:10.753 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T12:11:10.753 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-06 12:10:53.807336045 +0000 2024-04-06T12:11:10.753 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-06 12:07:32.224074453 +0000 2024-04-06T12:11:10.753 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-06 12:07:32.224074453 +0000 2024-04-06T12:11:10.753 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: - 2024-04-06T12:11:10.754 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-06T12:11:10.818 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-06T12:11:10.819 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-06T12:11:10.819 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.0002291 s, 2.2 MB/s 2024-04-06T12:11:10.819 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-06T12:11:10.877 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-06T12:11:10.878 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:10.878 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2024-04-06T12:11:10.896 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-06T12:11:10.897 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2024-04-06T12:11:10.955 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-06T12:11:10.955 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T12:11:10.955 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 172935 Links: 1 2024-04-06T12:11:10.955 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T12:11:10.956 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T12:11:10.956 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-06 12:10:52.552573676 +0000 2024-04-06T12:11:10.956 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-06 12:07:31.625741541 +0000 2024-04-06T12:11:10.956 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-06 12:07:31.625741541 +0000 2024-04-06T12:11:10.956 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2024-04-06T12:11:10.956 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-06T12:11:11.022 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-06T12:11:11.023 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-06T12:11:11.023 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000201477 s, 2.5 MB/s 2024-04-06T12:11:11.024 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-06T12:11:11.083 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2024-04-06T12:11:11.140 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-06T12:11:11.141 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T12:11:11.141 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 174953 Links: 1 2024-04-06T12:11:11.141 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T12:11:11.141 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T12:11:11.141 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-06 12:10:52.552573676 +0000 2024-04-06T12:11:11.141 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-06 12:07:31.978734124 +0000 2024-04-06T12:11:11.141 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-06 12:07:31.978734124 +0000 2024-04-06T12:11:11.141 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2024-04-06T12:11:11.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-06T12:11:11.207 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-06T12:11:11.207 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-06T12:11:11.207 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000227316 s, 2.3 MB/s 2024-04-06T12:11:11.208 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-06T12:11:11.266 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2024-04-06T12:11:11.323 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-06T12:11:11.323 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T12:11:11.323 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 181630 Links: 1 2024-04-06T12:11:11.323 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T12:11:11.323 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T12:11:11.323 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-06 12:10:52.552573676 +0000 2024-04-06T12:11:11.324 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-06 12:07:32.344726435 +0000 2024-04-06T12:11:11.324 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-06 12:07:32.344726435 +0000 2024-04-06T12:11:11.324 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2024-04-06T12:11:11.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-06T12:11:11.390 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-06T12:11:11.390 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-06T12:11:11.391 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000276069 s, 1.9 MB/s 2024-04-06T12:11:11.392 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-06T12:11:11.449 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2024-04-06T12:11:11.506 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-06T12:11:11.507 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T12:11:11.507 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 183371 Links: 1 2024-04-06T12:11:11.507 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T12:11:11.507 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T12:11:11.507 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-06 12:10:52.552573676 +0000 2024-04-06T12:11:11.507 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-06 12:07:32.693719102 +0000 2024-04-06T12:11:11.507 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-06 12:07:32.693719102 +0000 2024-04-06T12:11:11.507 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2024-04-06T12:11:11.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-06T12:11:11.571 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-06T12:11:11.572 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-06T12:11:11.572 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000209979 s, 2.4 MB/s 2024-04-06T12:11:11.574 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-06T12:11:11.631 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-06T12:11:11.632 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi189.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi192.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-06T12:11:11.632 INFO:tasks.ceph:Generating config... 2024-04-06T12:11:11.633 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-06T12:11:11.633 INFO:tasks.ceph:[client] debug client = 20 2024-04-06T12:11:11.633 INFO:tasks.ceph:[client] debug ms = 1 2024-04-06T12:11:11.633 INFO:tasks.ceph:[client] pid file = 2024-04-06T12:11:11.633 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-06T12:11:11.634 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-06T12:11:11.634 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-04-06T12:11:11.634 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-06T12:11:11.634 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-06T12:11:11.634 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-06T12:11:11.634 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-06T12:11:11.634 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-06T12:11:11.634 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-06T12:11:11.634 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-06T12:11:11.634 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-06T12:11:11.635 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-06T12:11:11.635 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-06T12:11:11.635 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-06T12:11:11.635 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-04-06T12:11:11.635 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-06T12:11:11.635 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-06T12:11:11.635 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-06T12:11:11.635 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-06T12:11:11.635 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-06T12:11:11.635 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-06T12:11:11.635 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-06T12:11:11.635 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-06T12:11:11.635 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-06T12:11:11.635 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-06T12:11:11.636 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-06T12:11:11.636 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-06T12:11:11.636 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-06T12:11:11.636 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-06T12:11:11.636 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-06T12:11:11.636 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-06T12:11:11.636 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-06T12:11:11.636 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-06T12:11:11.636 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-06T12:11:11.636 INFO:tasks.ceph:Setting up mon.a... 2024-04-06T12:11:11.636 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-06T12:11:11.694 INFO:teuthology.orchestra.run.smithi189.stdout:creating /etc/ceph/ceph.keyring 2024-04-06T12:11:11.697 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-06T12:11:11.745 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-06T12:11:11.773 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.189'), ('mon.b', '172.21.15.192'), ('mon.c', '[v2:172.21.15.192:3301,v1:172.21.15.192:6790]')] 2024-04-06T12:11:11.773 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.189,172.21.15.192,[v2:172.21.15.192:3301,v1:172.21.15.192:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-06T12:11:11.774 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:11.774 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-06T12:11:11.830 DEBUG:teuthology.orchestra.run.smithi189:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.189 --add b 172.21.15.192 --addv c '[v2:172.21.15.192:3301,v1:172.21.15.192:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-06T12:11:11.920 INFO:teuthology.orchestra.run.smithi189.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-06T12:11:11.920 INFO:teuthology.orchestra.run.smithi189.stdout:monmaptool: generated fsid e30041de-e9a3-4351-8ca7-29d7f3e3728c 2024-04-06T12:11:11.920 INFO:teuthology.orchestra.run.smithi189.stdout:setting min_mon_release = pacific 2024-04-06T12:11:11.920 INFO:teuthology.orchestra.run.smithi189.stdout:epoch 0 2024-04-06T12:11:11.920 INFO:teuthology.orchestra.run.smithi189.stdout:fsid e30041de-e9a3-4351-8ca7-29d7f3e3728c 2024-04-06T12:11:11.920 INFO:teuthology.orchestra.run.smithi189.stdout:last_changed 2024-04-06T12:11:11.922290+0000 2024-04-06T12:11:11.920 INFO:teuthology.orchestra.run.smithi189.stdout:created 2024-04-06T12:11:11.922290+0000 2024-04-06T12:11:11.920 INFO:teuthology.orchestra.run.smithi189.stdout:min_mon_release 16 (pacific) 2024-04-06T12:11:11.920 INFO:teuthology.orchestra.run.smithi189.stdout:election_strategy: 1 2024-04-06T12:11:11.921 INFO:teuthology.orchestra.run.smithi189.stdout:0: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.a 2024-04-06T12:11:11.921 INFO:teuthology.orchestra.run.smithi189.stdout:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2024-04-06T12:11:11.921 INFO:teuthology.orchestra.run.smithi189.stdout:2: [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] mon.c 2024-04-06T12:11:11.921 INFO:teuthology.orchestra.run.smithi189.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-06T12:11:11.922 DEBUG:teuthology.orchestra.run.smithi189:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-06T12:11:11.938 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e30041de-e9a3-4351-8ca7-29d7f3e3728c... 2024-04-06T12:11:11.939 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-06T12:11:11.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-06T12:11:12.024 INFO:teuthology.orchestra.run.smithi189.stdout:[global] 2024-04-06T12:11:12.024 INFO:teuthology.orchestra.run.smithi189.stdout: chdir = "" 2024-04-06T12:11:12.025 INFO:teuthology.orchestra.run.smithi189.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-06T12:11:12.025 INFO:teuthology.orchestra.run.smithi189.stdout: auth supported = cephx 2024-04-06T12:11:12.025 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.025 INFO:teuthology.orchestra.run.smithi189.stdout: filestore xattr use omap = true 2024-04-06T12:11:12.025 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.025 INFO:teuthology.orchestra.run.smithi189.stdout: mon clock drift allowed = 1.000 2024-04-06T12:11:12.025 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.025 INFO:teuthology.orchestra.run.smithi189.stdout: osd crush chooseleaf type = 0 2024-04-06T12:11:12.025 INFO:teuthology.orchestra.run.smithi189.stdout: auth debug = true 2024-04-06T12:11:12.025 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.025 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on old message = true 2024-04-06T12:11:12.025 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on bug = true 2024-04-06T12:11:12.025 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.025 INFO:teuthology.orchestra.run.smithi189.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-06T12:11:12.025 INFO:teuthology.orchestra.run.smithi189.stdout: mon pg warn max object skew = 0 2024-04-06T12:11:12.026 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.026 INFO:teuthology.orchestra.run.smithi189.stdout: # disable pg_autoscaler by default for new pools 2024-04-06T12:11:12.026 INFO:teuthology.orchestra.run.smithi189.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-06T12:11:12.026 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.026 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default size = 2 2024-04-06T12:11:12.026 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.026 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow primary affinity = true 2024-04-06T12:11:12.026 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow pg remap = true 2024-04-06T12:11:12.026 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on legacy crush tunables = false 2024-04-06T12:11:12.026 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on crush straw calc version zero = false 2024-04-06T12:11:12.026 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on no sortbitwise = false 2024-04-06T12:11:12.026 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on osd down out interval zero = false 2024-04-06T12:11:12.026 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on too few osds = false 2024-04-06T12:11:12.027 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-06T12:11:12.027 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-06T12:11:12.027 INFO:teuthology.orchestra.run.smithi189.stdout: mon_allow_pool_size_one = true 2024-04-06T12:11:12.027 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.027 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-06T12:11:12.027 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.027 INFO:teuthology.orchestra.run.smithi189.stdout: osd default data pool replay window = 5 2024-04-06T12:11:12.027 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.027 INFO:teuthology.orchestra.run.smithi189.stdout: mon allow pool delete = true 2024-04-06T12:11:12.027 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.027 INFO:teuthology.orchestra.run.smithi189.stdout: mon cluster log file level = debug 2024-04-06T12:11:12.027 INFO:teuthology.orchestra.run.smithi189.stdout: debug asserts on shutdown = true 2024-04-06T12:11:12.027 INFO:teuthology.orchestra.run.smithi189.stdout: mon health detail to clog = false 2024-04-06T12:11:12.027 INFO:teuthology.orchestra.run.smithi189.stdout: mon host = "172.21.15.189,172.21.15.192,[v2:172.21.15.192:3301,v1:172.21.15.192:6790]" 2024-04-06T12:11:12.028 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on skipped message = False 2024-04-06T12:11:12.028 INFO:teuthology.orchestra.run.smithi189.stdout: fsid = e30041de-e9a3-4351-8ca7-29d7f3e3728c 2024-04-06T12:11:12.028 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.028 INFO:teuthology.orchestra.run.smithi189.stdout:[osd] 2024-04-06T12:11:12.028 INFO:teuthology.orchestra.run.smithi189.stdout: osd journal size = 100 2024-04-06T12:11:12.028 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.028 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub load threshold = 5.0 2024-04-06T12:11:12.028 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub max interval = 600 2024-04-06T12:11:12.028 INFO:teuthology.orchestra.run.smithi189.stdout: osd mclock profile = high_recovery_ops 2024-04-06T12:11:12.028 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.028 INFO:teuthology.orchestra.run.smithi189.stdout: osd recover clone overlap = true 2024-04-06T12:11:12.028 INFO:teuthology.orchestra.run.smithi189.stdout: osd recovery max chunk = 1048576 2024-04-06T12:11:12.028 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.029 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug shutdown = true 2024-04-06T12:11:12.029 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug op order = true 2024-04-06T12:11:12.029 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug verify stray on activate = true 2024-04-06T12:11:12.029 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.029 INFO:teuthology.orchestra.run.smithi189.stdout: osd open classes on start = true 2024-04-06T12:11:12.029 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug pg log writeout = true 2024-04-06T12:11:12.029 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.029 INFO:teuthology.orchestra.run.smithi189.stdout: osd deep scrub update digest min age = 30 2024-04-06T12:11:12.029 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.029 INFO:teuthology.orchestra.run.smithi189.stdout: osd map max advance = 10 2024-04-06T12:11:12.029 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.029 INFO:teuthology.orchestra.run.smithi189.stdout: journal zero on create = true 2024-04-06T12:11:12.029 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.029 INFO:teuthology.orchestra.run.smithi189.stdout: filestore ondisk finisher threads = 3 2024-04-06T12:11:12.029 INFO:teuthology.orchestra.run.smithi189.stdout: filestore apply finisher threads = 3 2024-04-06T12:11:12.030 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.030 INFO:teuthology.orchestra.run.smithi189.stdout: bdev debug aio = true 2024-04-06T12:11:12.030 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug misdirected ops = true 2024-04-06T12:11:12.030 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore block size = 96636764160 2024-04-06T12:11:12.030 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore fsck on mount = True 2024-04-06T12:11:12.030 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluefs = 20 2024-04-06T12:11:12.030 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluestore = 20 2024-04-06T12:11:12.030 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-04-06T12:11:12.030 INFO:teuthology.orchestra.run.smithi189.stdout: debug osd = 20 2024-04-06T12:11:12.030 INFO:teuthology.orchestra.run.smithi189.stdout: debug rocksdb = 10 2024-04-06T12:11:12.030 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd backfillfull_ratio = 0.85 2024-04-06T12:11:12.030 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd full ratio = 0.9 2024-04-06T12:11:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd nearfull ratio = 0.8 2024-04-06T12:11:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: osd failsafe full ratio = 0.95 2024-04-06T12:11:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: osd objectstore = bluestore 2024-04-06T12:11:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: osd op complaint time = 180 2024-04-06T12:11:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.031 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr] 2024-04-06T12:11:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-04-06T12:11:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: debug mgr = 20 2024-04-06T12:11:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-04-06T12:11:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-04-06T12:11:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-04-06T12:11:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-04-06T12:11:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: mgr/telemetry/nag = false 2024-04-06T12:11:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: debug client = 20 2024-04-06T12:11:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: debug finisher = 20 2024-04-06T12:11:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.032 INFO:teuthology.orchestra.run.smithi189.stdout:[mon] 2024-04-06T12:11:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-04-06T12:11:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-04-06T12:11:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: debug paxos = 20 2024-04-06T12:11:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-04-06T12:11:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: mon data avail warn = 5 2024-04-06T12:11:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: mon mgr mkfs grace = 240 2024-04-06T12:11:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-04-06T12:11:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd reporter subtree level = osd 2024-04-06T12:11:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd prime pg temp = true 2024-04-06T12:11:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-04-06T12:11:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.033 INFO:teuthology.orchestra.run.smithi189.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-06T12:11:12.033 INFO:teuthology.orchestra.run.smithi189.stdout: auth mon ticket ttl = 660 # 11m 2024-04-06T12:11:12.033 INFO:teuthology.orchestra.run.smithi189.stdout: auth service ticket ttl = 240 # 4m 2024-04-06T12:11:12.033 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.033 INFO:teuthology.orchestra.run.smithi189.stdout: # don't complain about insecure global_id in the test suite 2024-04-06T12:11:12.033 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-06T12:11:12.033 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-06T12:11:12.033 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.033 INFO:teuthology.orchestra.run.smithi189.stdout: # 1m isn't quite enough 2024-04-06T12:11:12.033 INFO:teuthology.orchestra.run.smithi189.stdout: mon_down_mkfs_grace = 2m 2024-04-06T12:11:12.033 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.033 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_filestore_osds = false 2024-04-06T12:11:12.033 INFO:teuthology.orchestra.run.smithi189.stdout: mon op complaint time = 120 2024-04-06T12:11:12.034 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:12.034 INFO:teuthology.orchestra.run.smithi189.stdout:[client] 2024-04-06T12:11:12.034 INFO:teuthology.orchestra.run.smithi189.stdout: rgw cache enabled = true 2024-04-06T12:11:12.034 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable ops log = true 2024-04-06T12:11:12.034 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable usage log = true 2024-04-06T12:11:12.034 INFO:teuthology.orchestra.run.smithi189.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-06T12:11:12.034 INFO:teuthology.orchestra.run.smithi189.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-06T12:11:12.034 INFO:teuthology.orchestra.run.smithi189.stdout: client mount timeout = 600 2024-04-06T12:11:12.034 INFO:teuthology.orchestra.run.smithi189.stdout: debug client = 20 2024-04-06T12:11:12.034 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-04-06T12:11:12.034 INFO:teuthology.orchestra.run.smithi189.stdout: pid file = "" 2024-04-06T12:11:12.034 INFO:teuthology.orchestra.run.smithi189.stdout: rados mon op timeout = 900 2024-04-06T12:11:12.034 INFO:teuthology.orchestra.run.smithi189.stdout: rados osd op timeout = 900 2024-04-06T12:11:12.035 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.a] 2024-04-06T12:11:12.035 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.b] 2024-04-06T12:11:12.035 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.c] 2024-04-06T12:11:12.035 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.a] 2024-04-06T12:11:12.035 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.c] 2024-04-06T12:11:12.035 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.b] 2024-04-06T12:11:12.035 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.d] 2024-04-06T12:11:12.035 INFO:teuthology.orchestra.run.smithi189.stdout:[mds] 2024-04-06T12:11:12.035 INFO:teuthology.orchestra.run.smithi189.stdout: debug mds = 20 2024-04-06T12:11:12.035 INFO:teuthology.orchestra.run.smithi189.stdout: debug mds balancer = 20 2024-04-06T12:11:12.035 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-04-06T12:11:12.035 INFO:teuthology.orchestra.run.smithi189.stdout: mds debug frag = True 2024-04-06T12:11:12.035 INFO:teuthology.orchestra.run.smithi189.stdout: mds debug scatterstat = True 2024-04-06T12:11:12.035 INFO:teuthology.orchestra.run.smithi189.stdout: mds op complaint time = 180 2024-04-06T12:11:12.035 INFO:teuthology.orchestra.run.smithi189.stdout: mds verify scatter = True 2024-04-06T12:11:12.036 INFO:teuthology.orchestra.run.smithi189.stdout: osd op complaint time = 180 2024-04-06T12:11:12.036 INFO:teuthology.orchestra.run.smithi189.stdout: rados mon op timeout = 900 2024-04-06T12:11:12.036 INFO:teuthology.orchestra.run.smithi189.stdout: rados osd op timeout = 900 2024-04-06T12:11:12.036 INFO:teuthology.orchestra.run.smithi192.stdout:[global] 2024-04-06T12:11:12.036 INFO:teuthology.orchestra.run.smithi192.stdout: chdir = "" 2024-04-06T12:11:12.036 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-06T12:11:12.036 INFO:teuthology.orchestra.run.smithi192.stdout: auth supported = cephx 2024-04-06T12:11:12.036 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.037 INFO:teuthology.orchestra.run.smithi192.stdout: filestore xattr use omap = true 2024-04-06T12:11:12.037 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.037 INFO:teuthology.orchestra.run.smithi192.stdout: mon clock drift allowed = 1.000 2024-04-06T12:11:12.037 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.037 INFO:teuthology.orchestra.run.smithi192.stdout: osd crush chooseleaf type = 0 2024-04-06T12:11:12.037 INFO:teuthology.orchestra.run.smithi192.stdout: auth debug = true 2024-04-06T12:11:12.037 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.037 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on old message = true 2024-04-06T12:11:12.037 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on bug = true 2024-04-06T12:11:12.037 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.037 INFO:teuthology.orchestra.run.smithi192.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-06T12:11:12.037 INFO:teuthology.orchestra.run.smithi192.stdout: mon pg warn max object skew = 0 2024-04-06T12:11:12.037 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.037 INFO:teuthology.orchestra.run.smithi192.stdout: # disable pg_autoscaler by default for new pools 2024-04-06T12:11:12.037 INFO:teuthology.orchestra.run.smithi192.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-06T12:11:12.038 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.038 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default size = 2 2024-04-06T12:11:12.038 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.038 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow primary affinity = true 2024-04-06T12:11:12.038 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow pg remap = true 2024-04-06T12:11:12.038 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on legacy crush tunables = false 2024-04-06T12:11:12.038 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on crush straw calc version zero = false 2024-04-06T12:11:12.038 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on no sortbitwise = false 2024-04-06T12:11:12.038 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on osd down out interval zero = false 2024-04-06T12:11:12.038 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on too few osds = false 2024-04-06T12:11:12.038 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-06T12:11:12.038 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-06T12:11:12.038 INFO:teuthology.orchestra.run.smithi192.stdout: mon_allow_pool_size_one = true 2024-04-06T12:11:12.038 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.038 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-06T12:11:12.039 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.039 INFO:teuthology.orchestra.run.smithi192.stdout: osd default data pool replay window = 5 2024-04-06T12:11:12.039 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.039 INFO:teuthology.orchestra.run.smithi192.stdout: mon allow pool delete = true 2024-04-06T12:11:12.039 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.039 INFO:teuthology.orchestra.run.smithi192.stdout: mon cluster log file level = debug 2024-04-06T12:11:12.039 INFO:teuthology.orchestra.run.smithi192.stdout: debug asserts on shutdown = true 2024-04-06T12:11:12.039 INFO:teuthology.orchestra.run.smithi192.stdout: mon health detail to clog = false 2024-04-06T12:11:12.039 INFO:teuthology.orchestra.run.smithi192.stdout: mon host = "172.21.15.189,172.21.15.192,[v2:172.21.15.192:3301,v1:172.21.15.192:6790]" 2024-04-06T12:11:12.039 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on skipped message = False 2024-04-06T12:11:12.039 INFO:teuthology.orchestra.run.smithi192.stdout: fsid = e30041de-e9a3-4351-8ca7-29d7f3e3728c 2024-04-06T12:11:12.039 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.039 INFO:teuthology.orchestra.run.smithi192.stdout:[osd] 2024-04-06T12:11:12.040 INFO:teuthology.orchestra.run.smithi192.stdout: osd journal size = 100 2024-04-06T12:11:12.040 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.040 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub load threshold = 5.0 2024-04-06T12:11:12.040 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub max interval = 600 2024-04-06T12:11:12.040 INFO:teuthology.orchestra.run.smithi192.stdout: osd mclock profile = high_recovery_ops 2024-04-06T12:11:12.040 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.040 INFO:teuthology.orchestra.run.smithi192.stdout: osd recover clone overlap = true 2024-04-06T12:11:12.040 INFO:teuthology.orchestra.run.smithi192.stdout: osd recovery max chunk = 1048576 2024-04-06T12:11:12.040 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.040 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug shutdown = true 2024-04-06T12:11:12.040 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug op order = true 2024-04-06T12:11:12.041 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug verify stray on activate = true 2024-04-06T12:11:12.041 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.041 INFO:teuthology.orchestra.run.smithi192.stdout: osd open classes on start = true 2024-04-06T12:11:12.041 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug pg log writeout = true 2024-04-06T12:11:12.041 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.041 INFO:teuthology.orchestra.run.smithi192.stdout: osd deep scrub update digest min age = 30 2024-04-06T12:11:12.041 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.041 INFO:teuthology.orchestra.run.smithi192.stdout: osd map max advance = 10 2024-04-06T12:11:12.041 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.041 INFO:teuthology.orchestra.run.smithi192.stdout: journal zero on create = true 2024-04-06T12:11:12.041 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.041 INFO:teuthology.orchestra.run.smithi192.stdout: filestore ondisk finisher threads = 3 2024-04-06T12:11:12.041 INFO:teuthology.orchestra.run.smithi192.stdout: filestore apply finisher threads = 3 2024-04-06T12:11:12.041 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.042 INFO:teuthology.orchestra.run.smithi192.stdout: bdev debug aio = true 2024-04-06T12:11:12.042 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug misdirected ops = true 2024-04-06T12:11:12.042 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore block size = 96636764160 2024-04-06T12:11:12.042 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore fsck on mount = True 2024-04-06T12:11:12.042 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluefs = 20 2024-04-06T12:11:12.042 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluestore = 20 2024-04-06T12:11:12.042 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-04-06T12:11:12.042 INFO:teuthology.orchestra.run.smithi192.stdout: debug osd = 20 2024-04-06T12:11:12.042 INFO:teuthology.orchestra.run.smithi192.stdout: debug rocksdb = 10 2024-04-06T12:11:12.042 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd backfillfull_ratio = 0.85 2024-04-06T12:11:12.042 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd full ratio = 0.9 2024-04-06T12:11:12.042 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd nearfull ratio = 0.8 2024-04-06T12:11:12.042 INFO:teuthology.orchestra.run.smithi192.stdout: osd failsafe full ratio = 0.95 2024-04-06T12:11:12.042 INFO:teuthology.orchestra.run.smithi192.stdout: osd objectstore = bluestore 2024-04-06T12:11:12.042 INFO:teuthology.orchestra.run.smithi192.stdout: osd op complaint time = 180 2024-04-06T12:11:12.043 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.043 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr] 2024-04-06T12:11:12.043 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-04-06T12:11:12.043 INFO:teuthology.orchestra.run.smithi192.stdout: debug mgr = 20 2024-04-06T12:11:12.043 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2024-04-06T12:11:12.043 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2024-04-06T12:11:12.043 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2024-04-06T12:11:12.043 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2024-04-06T12:11:12.043 INFO:teuthology.orchestra.run.smithi192.stdout: mgr/telemetry/nag = false 2024-04-06T12:11:12.043 INFO:teuthology.orchestra.run.smithi192.stdout: debug client = 20 2024-04-06T12:11:12.043 INFO:teuthology.orchestra.run.smithi192.stdout: debug finisher = 20 2024-04-06T12:11:12.043 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.043 INFO:teuthology.orchestra.run.smithi192.stdout:[mon] 2024-04-06T12:11:12.043 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-04-06T12:11:12.043 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2024-04-06T12:11:12.044 INFO:teuthology.orchestra.run.smithi192.stdout: debug paxos = 20 2024-04-06T12:11:12.044 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2024-04-06T12:11:12.044 INFO:teuthology.orchestra.run.smithi192.stdout: mon data avail warn = 5 2024-04-06T12:11:12.044 INFO:teuthology.orchestra.run.smithi192.stdout: mon mgr mkfs grace = 240 2024-04-06T12:11:12.044 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2024-04-06T12:11:12.044 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd reporter subtree level = osd 2024-04-06T12:11:12.044 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd prime pg temp = true 2024-04-06T12:11:12.044 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2024-04-06T12:11:12.044 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.044 INFO:teuthology.orchestra.run.smithi192.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-06T12:11:12.044 INFO:teuthology.orchestra.run.smithi192.stdout: auth mon ticket ttl = 660 # 11m 2024-04-06T12:11:12.044 INFO:teuthology.orchestra.run.smithi192.stdout: auth service ticket ttl = 240 # 4m 2024-04-06T12:11:12.044 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.044 INFO:teuthology.orchestra.run.smithi192.stdout: # don't complain about insecure global_id in the test suite 2024-04-06T12:11:12.044 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-06T12:11:12.044 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-06T12:11:12.044 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.044 INFO:teuthology.orchestra.run.smithi192.stdout: # 1m isn't quite enough 2024-04-06T12:11:12.045 INFO:teuthology.orchestra.run.smithi192.stdout: mon_down_mkfs_grace = 2m 2024-04-06T12:11:12.045 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.045 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_filestore_osds = false 2024-04-06T12:11:12.045 INFO:teuthology.orchestra.run.smithi192.stdout: mon op complaint time = 120 2024-04-06T12:11:12.045 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T12:11:12.045 INFO:teuthology.orchestra.run.smithi192.stdout:[client] 2024-04-06T12:11:12.045 INFO:teuthology.orchestra.run.smithi192.stdout: rgw cache enabled = true 2024-04-06T12:11:12.045 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable ops log = true 2024-04-06T12:11:12.045 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable usage log = true 2024-04-06T12:11:12.045 INFO:teuthology.orchestra.run.smithi192.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-06T12:11:12.045 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-06T12:11:12.045 INFO:teuthology.orchestra.run.smithi192.stdout: client mount timeout = 600 2024-04-06T12:11:12.045 INFO:teuthology.orchestra.run.smithi192.stdout: debug client = 20 2024-04-06T12:11:12.045 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-04-06T12:11:12.045 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = "" 2024-04-06T12:11:12.045 INFO:teuthology.orchestra.run.smithi192.stdout: rados mon op timeout = 900 2024-04-06T12:11:12.045 INFO:teuthology.orchestra.run.smithi192.stdout: rados osd op timeout = 900 2024-04-06T12:11:12.045 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.a] 2024-04-06T12:11:12.046 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.b] 2024-04-06T12:11:12.046 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.c] 2024-04-06T12:11:12.046 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.a] 2024-04-06T12:11:12.046 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.c] 2024-04-06T12:11:12.046 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.b] 2024-04-06T12:11:12.046 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.d] 2024-04-06T12:11:12.046 INFO:teuthology.orchestra.run.smithi192.stdout:[mds] 2024-04-06T12:11:12.046 INFO:teuthology.orchestra.run.smithi192.stdout: debug mds = 20 2024-04-06T12:11:12.046 INFO:teuthology.orchestra.run.smithi192.stdout: debug mds balancer = 20 2024-04-06T12:11:12.046 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-04-06T12:11:12.046 INFO:teuthology.orchestra.run.smithi192.stdout: mds debug frag = True 2024-04-06T12:11:12.046 INFO:teuthology.orchestra.run.smithi192.stdout: mds debug scatterstat = True 2024-04-06T12:11:12.046 INFO:teuthology.orchestra.run.smithi192.stdout: mds op complaint time = 180 2024-04-06T12:11:12.046 INFO:teuthology.orchestra.run.smithi192.stdout: mds verify scatter = True 2024-04-06T12:11:12.046 INFO:teuthology.orchestra.run.smithi192.stdout: osd op complaint time = 180 2024-04-06T12:11:12.047 INFO:teuthology.orchestra.run.smithi192.stdout: rados mon op timeout = 900 2024-04-06T12:11:12.047 INFO:teuthology.orchestra.run.smithi192.stdout: rados osd op timeout = 900 2024-04-06T12:11:12.048 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-06T12:11:12.049 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-06T12:11:12.145 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-06T12:11:12.146 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:12.146 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-06T12:11:12.163 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:12.163 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-06T12:11:12.219 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T12:11:12.219 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:12.219 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-06T12:11:12.220 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-06T12:11:12.296 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:12.296 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-06T12:11:12.353 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T12:11:12.354 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:12.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-06T12:11:12.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-06T12:11:12.399 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:12.399 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-06T12:11:12.459 INFO:tasks.ceph:Setting up mon nodes... 2024-04-06T12:11:12.459 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-06T12:11:12.459 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-06T12:11:12.513 INFO:teuthology.orchestra.run.smithi189.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-06T12:11:12.516 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-06T12:11:12.811 INFO:teuthology.orchestra.run.smithi192.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-06T12:11:12.814 INFO:tasks.ceph:Setting up mds nodes... 2024-04-06T12:11:12.814 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-06T12:11:12.870 INFO:teuthology.orchestra.run.smithi189.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-06T12:11:12.873 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-06T12:11:12.900 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-06T12:11:12.993 INFO:teuthology.orchestra.run.smithi189.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-06T12:11:12.996 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-06T12:11:13.022 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-06T12:11:13.086 INFO:teuthology.orchestra.run.smithi192.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-06T12:11:13.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-06T12:11:13.119 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-06T12:11:13.217 INFO:teuthology.orchestra.run.smithi192.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-06T12:11:13.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-06T12:11:13.247 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-06T12:11:13.247 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-06T12:11:13.291 INFO:teuthology.orchestra.run.smithi189.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-06T12:11:13.305 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-06T12:11:13.349 INFO:teuthology.orchestra.run.smithi189.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-06T12:11:13.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-06T12:11:13.406 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-06T12:11:13.419 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-06T12:11:13.462 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-06T12:11:13.476 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-06T12:11:13.476 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi189.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi192.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-06T12:11:13.476 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-06T12:11:13.504 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-06T12:11:13.504 INFO:tasks.ceph:role: osd.0 2024-04-06T12:11:13.504 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T12:11:13.504 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-06T12:11:13.578 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T12:11:13.578 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T12:11:13.578 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T12:11:13.578 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T12:11:13.578 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T12:11:13.578 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-04-06T12:11:13.578 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T12:11:13.578 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T12:11:13.578 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T12:11:13.578 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T12:11:14.004 INFO:teuthology.orchestra.run.smithi189.stdout:Discarding blocks...Done. 2024-04-06T12:11:14.006 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-04-06T12:11:14.006 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-06T12:11:14.193 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-06T12:11:14.232 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-06T12:11:14.302 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-06T12:11:14.302 INFO:tasks.ceph:role: osd.1 2024-04-06T12:11:14.303 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T12:11:14.303 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-06T12:11:14.382 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T12:11:14.382 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T12:11:14.383 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T12:11:14.383 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T12:11:14.383 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T12:11:14.383 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-04-06T12:11:14.383 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T12:11:14.383 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T12:11:14.383 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T12:11:14.383 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T12:11:14.793 INFO:teuthology.orchestra.run.smithi189.stdout:Discarding blocks...Done. 2024-04-06T12:11:14.795 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-04-06T12:11:14.795 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-06T12:11:14.905 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-06T12:11:14.939 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-06T12:11:15.009 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-06T12:11:15.009 INFO:tasks.ceph:role: osd.2 2024-04-06T12:11:15.009 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T12:11:15.009 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-06T12:11:15.085 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T12:11:15.085 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T12:11:15.085 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T12:11:15.085 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T12:11:15.085 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T12:11:15.085 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-04-06T12:11:15.085 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T12:11:15.085 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T12:11:15.085 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T12:11:15.085 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T12:11:15.537 INFO:teuthology.orchestra.run.smithi189.stdout:Discarding blocks...Done. 2024-04-06T12:11:15.538 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-04-06T12:11:15.539 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-06T12:11:15.723 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-06T12:11:15.761 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-06T12:11:15.833 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-06T12:11:15.833 INFO:tasks.ceph:role: osd.3 2024-04-06T12:11:15.833 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T12:11:15.834 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-06T12:11:15.911 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T12:11:15.911 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T12:11:15.911 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T12:11:15.911 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T12:11:15.911 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T12:11:15.911 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-04-06T12:11:15.911 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T12:11:15.911 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T12:11:15.911 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T12:11:15.911 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T12:11:16.345 INFO:teuthology.orchestra.run.smithi189.stdout:Discarding blocks...Done. 2024-04-06T12:11:16.347 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-04-06T12:11:16.347 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-06T12:11:16.516 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-06T12:11:16.556 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-06T12:11:16.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:16.650+0000 7fbe22b48640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-06T12:11:16.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:16.650+0000 7fbe22b48640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-06T12:11:16.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:16.650+0000 7fbe22b48640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-06T12:11:16.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:16.650+0000 7fbe22b48640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-06T12:11:16.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:16.650+0000 7fbe22b48640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-06T12:11:16.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:16.650+0000 7fbe22b48640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-06T12:11:19.501 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-06T12:11:19.534 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-06T12:11:19.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:19.629+0000 7f8c93a44640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-06T12:11:19.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:19.629+0000 7f8c93a44640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-06T12:11:19.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:19.629+0000 7f8c93a44640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-06T12:11:19.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:19.629+0000 7f8c93a44640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-06T12:11:19.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:19.629+0000 7f8c93a44640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-06T12:11:19.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:19.629+0000 7f8c93a44640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-06T12:11:22.480 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-06T12:11:22.512 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-06T12:11:22.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:22.605+0000 7f196028f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-06T12:11:22.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:22.605+0000 7f196028f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-06T12:11:22.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:22.605+0000 7f196028f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-06T12:11:22.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:22.605+0000 7f196028f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-06T12:11:22.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:22.605+0000 7f196028f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-06T12:11:22.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:22.606+0000 7f196028f640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-06T12:11:25.477 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-06T12:11:25.508 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-06T12:11:25.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:25.600+0000 7f5c25f61640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-06T12:11:25.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:25.600+0000 7f5c25f61640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-06T12:11:25.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:25.600+0000 7f5c25f61640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-06T12:11:25.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:25.600+0000 7f5c25f61640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-06T12:11:25.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:25.600+0000 7f5c25f61640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-06T12:11:25.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:25.601+0000 7f5c25f61640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-06T12:11:28.450 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-06T12:11:28.482 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-06T12:11:28.516 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-06T12:11:28.516 INFO:tasks.ceph:role: osd.4 2024-04-06T12:11:28.516 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T12:11:28.516 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-06T12:11:28.593 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T12:11:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T12:11:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T12:11:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T12:11:28.594 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T12:11:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-04-06T12:11:28.594 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T12:11:28.594 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T12:11:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T12:11:28.594 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T12:11:29.027 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-04-06T12:11:29.029 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-04-06T12:11:29.029 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-06T12:11:29.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-06T12:11:29.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-06T12:11:29.325 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-06T12:11:29.325 INFO:tasks.ceph:role: osd.5 2024-04-06T12:11:29.325 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T12:11:29.326 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-06T12:11:29.402 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T12:11:29.403 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T12:11:29.403 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T12:11:29.403 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T12:11:29.403 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T12:11:29.403 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-04-06T12:11:29.403 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T12:11:29.403 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T12:11:29.403 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T12:11:29.403 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T12:11:29.812 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-04-06T12:11:29.814 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-04-06T12:11:29.814 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-06T12:11:29.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-06T12:11:29.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-06T12:11:30.042 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-06T12:11:30.042 INFO:tasks.ceph:role: osd.6 2024-04-06T12:11:30.042 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T12:11:30.043 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-06T12:11:30.118 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T12:11:30.118 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T12:11:30.118 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T12:11:30.119 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T12:11:30.119 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T12:11:30.119 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-04-06T12:11:30.119 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T12:11:30.119 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T12:11:30.119 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T12:11:30.119 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T12:11:30.568 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-04-06T12:11:30.570 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-04-06T12:11:30.571 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-06T12:11:30.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-06T12:11:30.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-06T12:11:30.832 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-06T12:11:30.832 INFO:tasks.ceph:role: osd.7 2024-04-06T12:11:30.832 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T12:11:30.832 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-06T12:11:30.908 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T12:11:30.908 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T12:11:30.908 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T12:11:30.908 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T12:11:30.908 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T12:11:30.909 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-04-06T12:11:30.909 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T12:11:30.909 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T12:11:30.909 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T12:11:30.909 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T12:11:31.353 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-04-06T12:11:31.355 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-04-06T12:11:31.355 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-06T12:11:31.529 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-06T12:11:31.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-06T12:11:31.656 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:31.657+0000 7f0b435e5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-06T12:11:31.656 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:31.657+0000 7f0b435e5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-06T12:11:31.656 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:31.657+0000 7f0b435e5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-06T12:11:31.656 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:31.657+0000 7f0b435e5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-06T12:11:31.657 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:31.657+0000 7f0b435e5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-06T12:11:31.657 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:31.657+0000 7f0b435e5640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-06T12:11:34.510 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-06T12:11:34.542 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-06T12:11:34.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:34.634+0000 7f4f4b51b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-06T12:11:34.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:34.635+0000 7f4f4b51b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-06T12:11:34.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:34.635+0000 7f4f4b51b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-06T12:11:34.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:34.635+0000 7f4f4b51b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-06T12:11:34.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:34.635+0000 7f4f4b51b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-06T12:11:34.634 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:34.635+0000 7f4f4b51b640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-06T12:11:37.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-06T12:11:37.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-06T12:11:37.613 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:37.613+0000 7f1ca68d5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-06T12:11:37.613 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:37.614+0000 7f1ca68d5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-06T12:11:37.613 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:37.614+0000 7f1ca68d5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-06T12:11:37.613 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:37.614+0000 7f1ca68d5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-06T12:11:37.613 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:37.614+0000 7f1ca68d5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-06T12:11:37.614 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:37.614+0000 7f1ca68d5640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-06T12:11:40.010 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:11:40.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:11:40.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-06T12:11:40.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-06T12:11:40.591 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:40.592+0000 7fef56f49640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-06T12:11:40.591 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:40.593+0000 7fef56f49640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-06T12:11:40.591 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:40.593+0000 7fef56f49640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-06T12:11:40.591 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:40.593+0000 7fef56f49640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-06T12:11:40.591 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:40.593+0000 7fef56f49640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-06T12:11:40.592 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:40.593+0000 7fef56f49640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-06T12:11:43.456 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-06T12:11:43.487 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-06T12:11:43.487 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:43.487 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-06T12:11:43.519 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:43.520 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-06T12:11:43.591 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:43.591 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-06T12:11:43.662 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:43.662 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-06T12:11:43.730 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:43.731 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-06T12:11:43.799 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:43.799 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-06T12:11:43.867 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:43.868 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-06T12:11:43.934 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:43.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-06T12:11:43.967 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:43.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-06T12:11:44.037 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:44.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-06T12:11:44.107 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:44.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-06T12:11:44.175 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:44.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-06T12:11:44.242 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:44.242 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-06T12:11:44.309 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:44.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-06T12:11:44.375 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:44.375 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-06T12:11:44.394 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:44.394 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-06T12:11:44.453 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:44.453 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-06T12:11:44.470 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:44.470 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-06T12:11:44.527 INFO:tasks.ceph:Adding keys to all mons... 2024-04-06T12:11:44.527 DEBUG:teuthology.orchestra.run.smithi189:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-06T12:11:44.529 DEBUG:teuthology.orchestra.run.smithi192:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-06T12:11:44.570 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr.y] 2024-04-06T12:11:44.570 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDgOxFm1nHJHhAAQWRe8cU8duWuUJiYaDeTaQ== 2024-04-06T12:11:44.570 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.a] 2024-04-06T12:11:44.570 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDgOxFmTjYMNBAA5voWGV4Pa7pE27VogrdipQ== 2024-04-06T12:11:44.570 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.c] 2024-04-06T12:11:44.570 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDgOxFmz0hhOxAAD3/GINNmGoFk+oPyiHHJXg== 2024-04-06T12:11:44.570 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.0] 2024-04-06T12:11:44.570 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDkOxFmkxLWJhAAFXewCNOFX7dhAZVgUmewpQ== 2024-04-06T12:11:44.570 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.1] 2024-04-06T12:11:44.570 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDnOxFmHYqWJRAABlsVCiv3e7YJMbfodJTWag== 2024-04-06T12:11:44.571 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.2] 2024-04-06T12:11:44.571 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDqOxFmkhcvJBAA8NMEpHVGbGALjV9RISKP8g== 2024-04-06T12:11:44.571 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.3] 2024-04-06T12:11:44.571 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDtOxFmDGzhIxAAamBoAbE8+lAel4os67meyQ== 2024-04-06T12:11:44.571 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr.x] 2024-04-06T12:11:44.571 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDgOxFmPLyBMBAAYfhkI9Z1CPlf1Tijm+L7xA== 2024-04-06T12:11:44.571 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.b] 2024-04-06T12:11:44.571 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDhOxFm9A1OBRAAnHzubCvRUX8UdW8hmDzKbQ== 2024-04-06T12:11:44.571 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.d] 2024-04-06T12:11:44.571 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDhOxFmhoAeDRAAoQC0vicWumBRTwpi3PtYaw== 2024-04-06T12:11:44.571 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.4] 2024-04-06T12:11:44.572 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDzOxFmobZDJxAA9l8wdS81Eyw6+17I5ac6bw== 2024-04-06T12:11:44.572 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.5] 2024-04-06T12:11:44.572 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQD2OxFmi+ruJRAASlQ0o9jX/wA7U5AosnL+Wg== 2024-04-06T12:11:44.572 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.6] 2024-04-06T12:11:44.572 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQD5OxFmbwuuJBAA0wqJRmnJyskQPZJ7/LnpvA== 2024-04-06T12:11:44.572 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.7] 2024-04-06T12:11:44.572 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQD8OxFmDmRgIxAAzyW9AGAbmb67ZvB5qunwLQ== 2024-04-06T12:11:44.572 INFO:teuthology.orchestra.run.smithi189.stdout:[client.0] 2024-04-06T12:11:44.572 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDhOxFm0ByJERAAoEV8Yfc09+Go6u95XafElQ== 2024-04-06T12:11:44.572 INFO:teuthology.orchestra.run.smithi189.stdout:[client.2] 2024-04-06T12:11:44.572 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDhOxFm/Gz5FBAA3xX/rsAyWZ1JoUVlQlBIvQ== 2024-04-06T12:11:44.572 INFO:teuthology.orchestra.run.smithi189.stdout:[client.1] 2024-04-06T12:11:44.572 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDhOxFmSV5lGBAAavDEo2LGy0IHpWSAuitHXg== 2024-04-06T12:11:44.572 INFO:teuthology.orchestra.run.smithi189.stdout:[client.3] 2024-04-06T12:11:44.573 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDhOxFmzra7GxAADbnzIvWnRTo65LvPwtxWrg== 2024-04-06T12:11:44.592 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr.y] 2024-04-06T12:11:44.592 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDgOxFm1nHJHhAAQWRe8cU8duWuUJiYaDeTaQ== 2024-04-06T12:11:44.593 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.a] 2024-04-06T12:11:44.593 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDgOxFmTjYMNBAA5voWGV4Pa7pE27VogrdipQ== 2024-04-06T12:11:44.593 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.c] 2024-04-06T12:11:44.593 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDgOxFmz0hhOxAAD3/GINNmGoFk+oPyiHHJXg== 2024-04-06T12:11:44.593 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.0] 2024-04-06T12:11:44.593 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDkOxFmkxLWJhAAFXewCNOFX7dhAZVgUmewpQ== 2024-04-06T12:11:44.593 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.1] 2024-04-06T12:11:44.593 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDnOxFmHYqWJRAABlsVCiv3e7YJMbfodJTWag== 2024-04-06T12:11:44.593 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.2] 2024-04-06T12:11:44.593 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDqOxFmkhcvJBAA8NMEpHVGbGALjV9RISKP8g== 2024-04-06T12:11:44.593 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.3] 2024-04-06T12:11:44.593 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDtOxFmDGzhIxAAamBoAbE8+lAel4os67meyQ== 2024-04-06T12:11:44.594 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr.x] 2024-04-06T12:11:44.594 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDgOxFmPLyBMBAAYfhkI9Z1CPlf1Tijm+L7xA== 2024-04-06T12:11:44.594 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.b] 2024-04-06T12:11:44.594 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDhOxFm9A1OBRAAnHzubCvRUX8UdW8hmDzKbQ== 2024-04-06T12:11:44.594 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.d] 2024-04-06T12:11:44.594 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDhOxFmhoAeDRAAoQC0vicWumBRTwpi3PtYaw== 2024-04-06T12:11:44.594 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.4] 2024-04-06T12:11:44.594 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDzOxFmobZDJxAA9l8wdS81Eyw6+17I5ac6bw== 2024-04-06T12:11:44.594 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.5] 2024-04-06T12:11:44.594 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQD2OxFmi+ruJRAASlQ0o9jX/wA7U5AosnL+Wg== 2024-04-06T12:11:44.594 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.6] 2024-04-06T12:11:44.594 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQD5OxFmbwuuJBAA0wqJRmnJyskQPZJ7/LnpvA== 2024-04-06T12:11:44.594 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.7] 2024-04-06T12:11:44.595 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQD8OxFmDmRgIxAAzyW9AGAbmb67ZvB5qunwLQ== 2024-04-06T12:11:44.595 INFO:teuthology.orchestra.run.smithi192.stdout:[client.0] 2024-04-06T12:11:44.595 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDhOxFm0ByJERAAoEV8Yfc09+Go6u95XafElQ== 2024-04-06T12:11:44.595 INFO:teuthology.orchestra.run.smithi192.stdout:[client.2] 2024-04-06T12:11:44.595 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDhOxFm/Gz5FBAA3xX/rsAyWZ1JoUVlQlBIvQ== 2024-04-06T12:11:44.595 INFO:teuthology.orchestra.run.smithi192.stdout:[client.1] 2024-04-06T12:11:44.595 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDhOxFmSV5lGBAAavDEo2LGy0IHpWSAuitHXg== 2024-04-06T12:11:44.595 INFO:teuthology.orchestra.run.smithi192.stdout:[client.3] 2024-04-06T12:11:44.595 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDhOxFmzra7GxAADbnzIvWnRTo65LvPwtxWrg== 2024-04-06T12:11:44.596 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-06T12:11:44.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-06T12:11:44.690 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-06T12:11:44.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-06T12:11:44.748 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-06T12:11:44.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-06T12:11:44.807 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T12:11:44.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T12:11:44.862 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T12:11:44.864 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T12:11:44.918 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T12:11:44.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T12:11:44.973 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T12:11:44.975 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T12:11:45.028 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-06T12:11:45.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-06T12:11:45.086 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-06T12:11:45.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-06T12:11:45.146 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-06T12:11:45.148 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-06T12:11:45.205 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T12:11:45.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T12:11:45.261 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T12:11:45.263 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T12:11:45.317 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T12:11:45.320 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T12:11:45.373 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T12:11:45.375 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T12:11:45.428 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T12:11:45.430 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T12:11:45.487 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T12:11:45.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T12:11:45.545 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T12:11:45.548 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T12:11:45.604 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T12:11:45.606 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T12:11:45.663 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-06T12:11:45.663 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-06T12:11:45.690 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-06T12:11:45.789 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-06T12:11:45.816 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-06T12:11:45.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-06T12:11:45.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-06T12:11:45.968 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-06T12:11:46.036 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-06T12:11:46.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-06T12:11:46.161 DEBUG:teuthology.orchestra.run.smithi189:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-06T12:11:46.163 DEBUG:teuthology.orchestra.run.smithi192:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-06T12:11:46.217 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-06T12:11:46.217 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-06T12:11:46.218 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-06T12:11:46.220 INFO:tasks.ceph.mon.a:Started 2024-04-06T12:11:46.220 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-06T12:11:46.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-06T12:11:46.260 INFO:tasks.ceph.mon.b:Started 2024-04-06T12:11:46.260 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-06T12:11:46.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-06T12:11:46.262 INFO:tasks.ceph.mon.c:Started 2024-04-06T12:11:46.262 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-06T12:11:46.263 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-06T12:11:46.263 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-06T12:11:46.265 INFO:tasks.ceph.mgr.y:Started 2024-04-06T12:11:46.265 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-06T12:11:46.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-06T12:11:46.267 INFO:tasks.ceph.mgr.x:Started 2024-04-06T12:11:46.267 INFO:tasks.ceph:Setting crush tunables to default 2024-04-06T12:11:46.267 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd crush tunables default 2024-04-06T12:11:46.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.393+0000 7ffacb9bd700 1 Processor -- start 2024-04-06T12:11:46.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.393+0000 7ffacb9bd700 1 -- start start 2024-04-06T12:11:46.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.394+0000 7ffacb9bd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ffac415d7b0 0x7ffac40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:46.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.394+0000 7ffacb9bd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffac415e500 0x7ffac406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:46.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.394+0000 7ffacb9bd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac40b3990 con 0x7ffac406a730 2024-04-06T12:11:46.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.394+0000 7ffacb9bd700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac415e950 con 0x7ffac415e500 2024-04-06T12:11:46.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.394+0000 7ffacb9bd700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac406b0f0 con 0x7ffac415d7b0 2024-04-06T12:11:46.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.394+0000 7ffac9f5a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffac415e500 0x7ffac406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:46.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.394+0000 7ffac8f58700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ffac406a730 0x7ffac415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38788/0 (socket says 172.21.15.189:38788) 2024-04-06T12:11:46.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.395+0000 7ffac8f58700 1 -- 172.21.15.189:0/4007322263 learned_addr learned my addr 172.21.15.189:0/4007322263 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:11:46.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.395+0000 7ffac9f5a700 1 -- 172.21.15.189:0/4007322263 >> v2:172.21.15.189:3300/0 conn(0x7ffac415e500 msgr2=0x7ffac406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-06T12:11:46.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.395+0000 7ffac9f5a700 1 -- 172.21.15.189:0/4007322263 >> v2:172.21.15.189:3300/0 conn(0x7ffac415e500 msgr2=0x7ffac406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-06T12:11:46.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.395+0000 7ffac9f5a700 1 --2- 172.21.15.189:0/4007322263 >> v2:172.21.15.189:3300/0 conn(0x7ffac415e500 0x7ffac406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-06T12:11:46.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.395+0000 7ffac9759700 1 --2- 172.21.15.189:0/4007322263 >> v2:172.21.15.192:3300/0 conn(0x7ffac415d7b0 0x7ffac40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:46.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.395+0000 7ffac9f5a700 1 --2- 172.21.15.189:0/4007322263 >> v2:172.21.15.189:3300/0 conn(0x7ffac415e500 0x7ffac406abb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:11:46.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.395+0000 7ffac9759700 1 -- 172.21.15.189:0/4007322263 >> v2:172.21.15.192:3300/0 conn(0x7ffac415d7b0 msgr2=0x7ffac40b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-06T12:11:46.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.395+0000 7ffac9759700 1 -- 172.21.15.189:0/4007322263 >> v2:172.21.15.192:3300/0 conn(0x7ffac415d7b0 msgr2=0x7ffac40b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-06T12:11:46.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.395+0000 7ffac9759700 1 --2- 172.21.15.189:0/4007322263 >> v2:172.21.15.192:3300/0 conn(0x7ffac415d7b0 0x7ffac40b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-06T12:11:46.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.395+0000 7ffac9759700 1 --2- 172.21.15.189:0/4007322263 >> v2:172.21.15.192:3300/0 conn(0x7ffac415d7b0 0x7ffac40b3e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:11:46.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.590+0000 7ffabbfff700 1 -- 172.21.15.189:0/4007322263 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282391979 0 0) 0x7ffac40b3990 con 0x7ffac406a730 2024-04-06T12:11:46.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.590+0000 7ffabbfff700 1 -- 172.21.15.189:0/4007322263 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffab00035b0 con 0x7ffac406a730 2024-04-06T12:11:46.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffabbfff700 1 -- 172.21.15.189:0/4007322263 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3289904791 0 0) 0x7ffab00035b0 con 0x7ffac406a730 2024-04-06T12:11:46.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffabbfff700 1 -- 172.21.15.189:0/4007322263 >> v2:172.21.15.192:3300/0 conn(0x7ffac415d7b0 msgr2=0x7ffac40b3e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-06T12:11:46.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffabbfff700 1 --2- 172.21.15.189:0/4007322263 >> v2:172.21.15.192:3300/0 conn(0x7ffac415d7b0 0x7ffac40b3e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:46.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffabbfff700 1 -- 172.21.15.189:0/4007322263 >> v2:172.21.15.189:3300/0 conn(0x7ffac415e500 msgr2=0x7ffac406abb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-06T12:11:46.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffabbfff700 1 --2- 172.21.15.189:0/4007322263 >> v2:172.21.15.189:3300/0 conn(0x7ffac415e500 0x7ffac406abb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:46.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffabbfff700 1 -- 172.21.15.189:0/4007322263 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac40b3060 con 0x7ffac406a730 2024-04-06T12:11:46.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffabbfff700 1 -- 172.21.15.189:0/4007322263 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7ffac0001670 con 0x7ffac406a730 2024-04-06T12:11:46.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffabbfff700 1 -- 172.21.15.189:0/4007322263 >> v1:172.21.15.189:6789/0 conn(0x7ffac406a730 legacy=0x7ffac415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:46.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffabbfff700 1 --2- 172.21.15.189:0/4007322263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffab0003a50 0x7ffab0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:46.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffabbfff700 1 --2- 172.21.15.189:0/4007322263 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffab0004850 0x7ffab0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:46.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffabbfff700 1 --2- 172.21.15.189:0/4007322263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab0005260 0x7ffab000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:46.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffabbfff700 1 -- 172.21.15.189:0/4007322263 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac406b0f0 con 0x7ffab0003a50 2024-04-06T12:11:46.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffabbfff700 1 -- 172.21.15.189:0/4007322263 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab000c190 con 0x7ffab0005260 2024-04-06T12:11:46.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffabbfff700 1 -- 172.21.15.189:0/4007322263 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab000c350 con 0x7ffab0004850 2024-04-06T12:11:46.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffac9759700 1 --2- 172.21.15.189:0/4007322263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffab0003a50 0x7ffab0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:46.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffac8f58700 1 --2- 172.21.15.189:0/4007322263 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffab0004850 0x7ffab0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:46.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffac9f5a700 1 --2- 172.21.15.189:0/4007322263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab0005260 0x7ffab000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:46.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffac9759700 1 -- 172.21.15.189:0/4007322263 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffab0004850 msgr2=0x7ffab0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:11:46.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffac9759700 1 --2- 172.21.15.189:0/4007322263 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffab0004850 0x7ffab0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:46.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffac9759700 1 -- 172.21.15.189:0/4007322263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab0005260 msgr2=0x7ffab000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:11:46.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffac9759700 1 --2- 172.21.15.189:0/4007322263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab0005260 0x7ffab000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:46.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffac9759700 1 -- 172.21.15.189:0/4007322263 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab00035b0 con 0x7ffab0003a50 2024-04-06T12:11:46.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffac9f5a700 1 -- 172.21.15.189:0/4007322263 reap_dead start 2024-04-06T12:11:46.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.591+0000 7ffac9759700 1 --2- 172.21.15.189:0/4007322263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffab0003a50 0x7ffab0003e70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ffab4009890 tx=0x7ffab0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f36fceec9552469 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:46.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.592+0000 7ffabbfff700 1 -- 172.21.15.189:0/4007322263 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab4020070 con 0x7ffab0003a50 2024-04-06T12:11:46.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.592+0000 7ffabbfff700 1 -- 172.21.15.189:0/4007322263 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab4004020 con 0x7ffab0003a50 2024-04-06T12:11:46.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.592+0000 7ffabbfff700 1 -- 172.21.15.189:0/4007322263 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab400c070 con 0x7ffab0003a50 2024-04-06T12:11:46.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.592+0000 7ffacb9bd700 1 -- 172.21.15.189:0/4007322263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffab0003a50 msgr2=0x7ffab0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:46.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.592+0000 7ffacb9bd700 1 --2- 172.21.15.189:0/4007322263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffab0003a50 0x7ffab0003e70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ffab4009890 tx=0x7ffab0005880 comp rx=0 tx=0).stop 2024-04-06T12:11:46.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.592+0000 7ffacb9bd700 1 -- 172.21.15.189:0/4007322263 shutdown_connections 2024-04-06T12:11:46.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.592+0000 7ffacb9bd700 1 --2- 172.21.15.189:0/4007322263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffab0003a50 0x7ffab0003e70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:46.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.593+0000 7ffacb9bd700 1 -- 172.21.15.189:0/4007322263 >> 172.21.15.189:0/4007322263 conn(0x7ffac40d00b0 msgr2=0x7ffac40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:46.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.593+0000 7ffacb9bd700 1 -- 172.21.15.189:0/4007322263 shutdown_connections 2024-04-06T12:11:46.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.593+0000 7ffacb9bd700 1 -- 172.21.15.189:0/4007322263 wait complete. 2024-04-06T12:11:46.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.594+0000 7ffacb9bd700 1 Processor -- start 2024-04-06T12:11:46.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.594+0000 7ffacb9bd700 1 -- start start 2024-04-06T12:11:46.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.595+0000 7ffacb9bd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffab0003a50 0x7ffac414b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:46.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.595+0000 7ffacb9bd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffac4153b40 0x7ffac406eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:46.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.595+0000 7ffacb9bd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac4153fc0 0x7ffac406f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:46.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.595+0000 7ffacb9bd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac40ce6c0 con 0x7ffac4153fc0 2024-04-06T12:11:46.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.595+0000 7ffacb9bd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac40cda40 con 0x7ffac4153b40 2024-04-06T12:11:46.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.595+0000 7ffacb9bd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac40cdd20 con 0x7ffab0003a50 2024-04-06T12:11:46.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.595+0000 7ffac9f5a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac4153fc0 0x7ffac406f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:46.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.595+0000 7ffac8f58700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffac4153b40 0x7ffac406eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:46.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.595+0000 7ffac9f5a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac4153fc0 0x7ffac406f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44746/0 (socket says 172.21.15.189:44746) 2024-04-06T12:11:46.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.595+0000 7ffac9f5a700 1 -- 172.21.15.189:0/2972562819 learned_addr learned my addr 172.21.15.189:0/2972562819 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:11:46.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.595+0000 7ffac9759700 1 --2- 172.21.15.189:0/2972562819 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffab0003a50 0x7ffac414b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:46.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.595+0000 7ffac9f5a700 1 -- 172.21.15.189:0/2972562819 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffab0003a50 msgr2=0x7ffac414b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:46.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.595+0000 7ffac9f5a700 1 --2- 172.21.15.189:0/2972562819 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffab0003a50 0x7ffac414b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:46.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.595+0000 7ffac9f5a700 1 -- 172.21.15.189:0/2972562819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffac4153b40 msgr2=0x7ffac406eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:46.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.595+0000 7ffac9f5a700 1 --2- 172.21.15.189:0/2972562819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffac4153b40 0x7ffac406eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:46.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.595+0000 7ffac9f5a700 1 -- 172.21.15.189:0/2972562819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac40ce000 con 0x7ffac4153fc0 2024-04-06T12:11:46.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.595+0000 7ffac8f58700 1 --2- 172.21.15.189:0/2972562819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffac4153b40 0x7ffac406eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:11:46.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.595+0000 7ffac9f5a700 1 --2- 172.21.15.189:0/2972562819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac4153fc0 0x7ffac406f090 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffac40b4800 tx=0x7ffabc00f430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:46.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.596+0000 7ffab9ffb700 1 -- 172.21.15.189:0/2972562819 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc019070 con 0x7ffac4153fc0 2024-04-06T12:11:46.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.596+0000 7ffab9ffb700 1 -- 172.21.15.189:0/2972562819 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffabc009a90 con 0x7ffac4153fc0 2024-04-06T12:11:46.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.596+0000 7ffacb9bd700 1 -- 172.21.15.189:0/2972562819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac40ce2e0 con 0x7ffac4153fc0 2024-04-06T12:11:46.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.596+0000 7ffab9ffb700 1 -- 172.21.15.189:0/2972562819 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc009e00 con 0x7ffac4153fc0 2024-04-06T12:11:46.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.596+0000 7ffacb9bd700 1 -- 172.21.15.189:0/2972562819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac40ceae0 con 0x7ffac4153fc0 2024-04-06T12:11:46.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.596+0000 7ffab9ffb700 1 -- 172.21.15.189:0/2972562819 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7ffabc00fa90 con 0x7ffac4153fc0 2024-04-06T12:11:46.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.596+0000 7ffab9ffb700 1 -- 172.21.15.189:0/2972562819 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ffabc00a730 con 0x7ffac4153fc0 2024-04-06T12:11:46.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.597+0000 7ffacb9bd700 1 -- 172.21.15.189:0/2972562819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac4067bb0 con 0x7ffac4153fc0 2024-04-06T12:11:46.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.599+0000 7ffab9ffb700 1 -- 172.21.15.189:0/2972562819 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7ffabc00a8f0 con 0x7ffac4153fc0 2024-04-06T12:11:46.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.674+0000 7ffacb9bd700 1 -- 172.21.15.189:0/2972562819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ffac4074990 con 0x7ffac4153fc0 2024-04-06T12:11:46.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.677+0000 7ffab9ffb700 1 -- 172.21.15.189:0/2972562819 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7ffabc009f60 con 0x7ffac4153fc0 2024-04-06T12:11:46.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.678+0000 7ffacb9bd700 1 -- 172.21.15.189:0/2972562819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac4153fc0 msgr2=0x7ffac406f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:46.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.678+0000 7ffacb9bd700 1 --2- 172.21.15.189:0/2972562819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac4153fc0 0x7ffac406f090 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffac40b4800 tx=0x7ffabc00f430 comp rx=0 tx=0).stop 2024-04-06T12:11:46.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.678+0000 7ffacb9bd700 1 -- 172.21.15.189:0/2972562819 shutdown_connections 2024-04-06T12:11:46.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.678+0000 7ffacb9bd700 1 --2- 172.21.15.189:0/2972562819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac4153fc0 0x7ffac406f090 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:46.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.678+0000 7ffacb9bd700 1 --2- 172.21.15.189:0/2972562819 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffab0003a50 0x7ffac414b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:46.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.678+0000 7ffacb9bd700 1 --2- 172.21.15.189:0/2972562819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffac4153b40 0x7ffac406eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:46.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.678+0000 7ffacb9bd700 1 -- 172.21.15.189:0/2972562819 >> 172.21.15.189:0/2972562819 conn(0x7ffac40d00b0 msgr2=0x7ffac4154e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:46.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.678+0000 7ffacb9bd700 1 -- 172.21.15.189:0/2972562819 shutdown_connections 2024-04-06T12:11:46.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:46.678+0000 7ffacb9bd700 1 -- 172.21.15.189:0/2972562819 wait complete. 2024-04-06T12:11:46.677 INFO:teuthology.orchestra.run.smithi189.stderr:adjusted tunables profile to default 2024-04-06T12:11:46.688 INFO:tasks.ceph:check_enable_crimson: False 2024-04-06T12:11:46.688 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-06T12:11:46.688 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:46.688 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-06T12:11:46.715 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:46.715 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-06T12:11:46.780 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:46.781 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-06T12:11:46.850 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:11:46.850 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-06T12:11:46.917 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:46.917 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-06T12:11:46.945 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:46.945 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-06T12:11:47.005 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:47.008+0000 7f3e06cc8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:11:47.011 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:47.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-06T12:11:47.076 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:47.079+0000 7f70d0691200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:11:47.079 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:11:47.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-06T12:11:47.089 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:47.091+0000 7f3e06cc8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:11:47.114 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c 0 2024-04-06T12:11:47.158 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:47.161+0000 7f70d0691200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:11:47.166 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:47.168+0000 7f3e06cc8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:11:47.231 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:47.234+0000 7f70d0691200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:11:47.240 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:47.242+0000 7f3e06cc8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:11:47.278 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.280+0000 7f6f8a6cf700 1 Processor -- start 2024-04-06T12:11:47.279 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.280+0000 7f6f8a6cf700 1 -- start start 2024-04-06T12:11:47.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.281+0000 7f6f8a6cf700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6f840d11b0 0x7f6f840d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:47.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.281+0000 7f6f8a6cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f840cfa00 0x7f6f840cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:47.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.281+0000 7f6f8a6cf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f840d1ad0 con 0x7f6f840cb870 2024-04-06T12:11:47.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.281+0000 7f6f8a6cf700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f840ccdc0 con 0x7f6f840d11b0 2024-04-06T12:11:47.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.281+0000 7f6f8a6cf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f840d0b50 con 0x7f6f840cfa00 2024-04-06T12:11:47.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.281+0000 7f6f83fff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6f840d11b0 0x7f6f840d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:47.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.281+0000 7f6f88c6c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f840cfa00 0x7f6f840cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:47.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.281+0000 7f6f88c6c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f840cfa00 0x7f6f840cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:45014/0 (socket says 172.21.15.192:45014) 2024-04-06T12:11:47.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.281+0000 7f6f88c6c700 1 -- 172.21.15.192:0/794384248 learned_addr learned my addr 172.21.15.192:0/794384248 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-06T12:11:47.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.282+0000 7f6f88c6c700 1 -- 172.21.15.192:0/794384248 >> v2:172.21.15.192:3300/0 conn(0x7f6f840d11b0 msgr2=0x7f6f840d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:11:47.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.282+0000 7f6f88c6c700 1 --2- 172.21.15.192:0/794384248 >> v2:172.21.15.192:3300/0 conn(0x7f6f840d11b0 0x7f6f840d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.282+0000 7f6f88c6c700 1 -- 172.21.15.192:0/794384248 >> v1:172.21.15.189:6789/0 conn(0x7f6f840cb870 legacy=0x7f6f8415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:47.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.282+0000 7f6f88c6c700 1 -- 172.21.15.192:0/794384248 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f840cbd80 con 0x7f6f840cfa00 2024-04-06T12:11:47.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.282+0000 7f6f83fff700 1 --2- 172.21.15.192:0/794384248 >> v2:172.21.15.192:3300/0 conn(0x7f6f840d11b0 0x7f6f840d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:11:47.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.282+0000 7f6f88c6c700 1 --2- 172.21.15.192:0/794384248 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f840cfa00 0x7f6f840cd560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c009e50 tx=0x7f6f7c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f05b4da133219c38 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:47.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.282+0000 7f6f82ffd700 1 -- 172.21.15.192:0/794384248 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c00e070 con 0x7f6f840cfa00 2024-04-06T12:11:47.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.282+0000 7f6f82ffd700 1 -- 172.21.15.192:0/794384248 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7c00b910 con 0x7f6f840cfa00 2024-04-06T12:11:47.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.282+0000 7f6f82ffd700 1 -- 172.21.15.192:0/794384248 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c00bcc0 con 0x7f6f840cfa00 2024-04-06T12:11:47.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.282+0000 7f6f8a6cf700 1 -- 172.21.15.192:0/794384248 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f840cfa00 msgr2=0x7f6f840cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:47.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.282+0000 7f6f8a6cf700 1 --2- 172.21.15.192:0/794384248 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f840cfa00 0x7f6f840cd560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c009e50 tx=0x7f6f7c00b330 comp rx=0 tx=0).stop 2024-04-06T12:11:47.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.283+0000 7f6f8a6cf700 1 -- 172.21.15.192:0/794384248 shutdown_connections 2024-04-06T12:11:47.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.283+0000 7f6f8a6cf700 1 --2- 172.21.15.192:0/794384248 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f840cfa00 0x7f6f840cd560 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.283+0000 7f6f8a6cf700 1 --2- 172.21.15.192:0/794384248 >> v2:172.21.15.192:3300/0 conn(0x7f6f840d11b0 0x7f6f840d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.283+0000 7f6f8a6cf700 1 -- 172.21.15.192:0/794384248 >> 172.21.15.192:0/794384248 conn(0x7f6f840b2db0 msgr2=0x7f6f840b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:47.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.283+0000 7f6f8a6cf700 1 -- 172.21.15.192:0/794384248 shutdown_connections 2024-04-06T12:11:47.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.283+0000 7f6f8a6cf700 1 -- 172.21.15.192:0/794384248 wait complete. 2024-04-06T12:11:47.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.284+0000 7f6f8a6cf700 1 Processor -- start 2024-04-06T12:11:47.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.284+0000 7f6f8a6cf700 1 -- start start 2024-04-06T12:11:47.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f8a6cf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f840cb870 0x7f6f8415ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:47.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f8a6cf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f840cfa00 0x7f6f84154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:47.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f8a6cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f840d11b0 0x7f6f841e68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:47.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f8a6cf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f84154dc0 con 0x7f6f840cfa00 2024-04-06T12:11:47.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f8a6cf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f841557b0 con 0x7f6f840cb870 2024-04-06T12:11:47.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f8a6cf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f84158360 con 0x7f6f840d11b0 2024-04-06T12:11:47.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f83fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f840cb870 0x7f6f8415ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:47.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f88c6c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f840d11b0 0x7f6f841e68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:47.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f83fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f840cb870 0x7f6f8415ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:58330/0 (socket says 172.21.15.192:58330) 2024-04-06T12:11:47.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f88c6c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f840d11b0 0x7f6f841e68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:45026/0 (socket says 172.21.15.192:45026) 2024-04-06T12:11:47.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f83fff700 1 -- 172.21.15.192:0/4129677938 learned_addr learned my addr 172.21.15.192:0/4129677938 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-06T12:11:47.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f837fe700 1 --2- 172.21.15.192:0/4129677938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f840cfa00 0x7f6f84154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:47.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f88c6c700 1 -- 172.21.15.192:0/4129677938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f840cb870 msgr2=0x7f6f8415ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:47.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f88c6c700 1 --2- 172.21.15.192:0/4129677938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f840cb870 0x7f6f8415ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f88c6c700 1 -- 172.21.15.192:0/4129677938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f840cfa00 msgr2=0x7f6f84154880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:47.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f88c6c700 1 --2- 172.21.15.192:0/4129677938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f840cfa00 0x7f6f84154880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f88c6c700 1 -- 172.21.15.192:0/4129677938 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f740094a0 con 0x7f6f840d11b0 2024-04-06T12:11:47.287 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f83fff700 1 --2- 172.21.15.192:0/4129677938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f840cb870 0x7f6f8415ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:11:47.288 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f88c6c700 1 --2- 172.21.15.192:0/4129677938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f840d11b0 0x7f6f841e68a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6f8404fb40 tx=0x7f6f7c004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:47.288 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f80ff9700 1 -- 172.21.15.192:0/4129677938 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c004800 con 0x7f6f840d11b0 2024-04-06T12:11:47.288 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f80ff9700 1 -- 172.21.15.192:0/4129677938 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7c004960 con 0x7f6f840d11b0 2024-04-06T12:11:47.288 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f80ff9700 1 -- 172.21.15.192:0/4129677938 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c01c680 con 0x7f6f840d11b0 2024-04-06T12:11:47.288 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.285+0000 7f6f8a6cf700 1 -- 172.21.15.192:0/4129677938 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7c009b30 con 0x7f6f840d11b0 2024-04-06T12:11:47.288 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.286+0000 7f6f8a6cf700 1 -- 172.21.15.192:0/4129677938 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f841588e0 con 0x7f6f840d11b0 2024-04-06T12:11:47.288 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.286+0000 7f6f80ff9700 1 -- 172.21.15.192:0/4129677938 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f6f7c01cae0 con 0x7f6f840d11b0 2024-04-06T12:11:47.288 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.286+0000 7f6f80ff9700 1 -- 172.21.15.192:0/4129677938 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6f7c025be0 con 0x7f6f840d11b0 2024-04-06T12:11:47.288 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.287+0000 7f6f8a6cf700 1 -- 172.21.15.192:0/4129677938 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f84074990 con 0x7f6f840d11b0 2024-04-06T12:11:47.289 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.289+0000 7f6f80ff9700 1 -- 172.21.15.192:0/4129677938 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7f6f7c020070 con 0x7f6f840d11b0 2024-04-06T12:11:47.302 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:47.304+0000 7f70d0691200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:11:47.370 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.372+0000 7f6f8a6cf700 1 -- 172.21.15.192:0/4129677938 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c", "id": 0} v 0) v1 -- 0x7f6f840d1e20 con 0x7f6f840d11b0 2024-04-06T12:11:47.374 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.376+0000 7f6f80ff9700 1 -- 172.21.15.192:0/4129677938 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6f7c02e430 con 0x7f6f840d11b0 2024-04-06T12:11:47.374 INFO:teuthology.orchestra.run.smithi192.stdout:0 2024-04-06T12:11:47.375 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.377+0000 7f6f8a6cf700 1 -- 172.21.15.192:0/4129677938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f840d11b0 msgr2=0x7f6f841e68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:47.375 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.377+0000 7f6f8a6cf700 1 --2- 172.21.15.192:0/4129677938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f840d11b0 0x7f6f841e68a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6f8404fb40 tx=0x7f6f7c004ad0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.375 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.377+0000 7f6f8a6cf700 1 -- 172.21.15.192:0/4129677938 shutdown_connections 2024-04-06T12:11:47.375 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.377+0000 7f6f8a6cf700 1 --2- 172.21.15.192:0/4129677938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f840d11b0 0x7f6f841e68a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.375 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.377+0000 7f6f8a6cf700 1 --2- 172.21.15.192:0/4129677938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f840cb870 0x7f6f8415ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.377+0000 7f6f8a6cf700 1 --2- 172.21.15.192:0/4129677938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f840cfa00 0x7f6f84154880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.377+0000 7f6f8a6cf700 1 -- 172.21.15.192:0/4129677938 >> 172.21.15.192:0/4129677938 conn(0x7f6f840b2db0 msgr2=0x7f6f840b6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:47.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.377+0000 7f6f8a6cf700 1 -- 172.21.15.192:0/4129677938 shutdown_connections 2024-04-06T12:11:47.376 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.377+0000 7f6f8a6cf700 1 -- 172.21.15.192:0/4129677938 wait complete. 2024-04-06T12:11:47.388 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 2d7c167b-3876-470d-9cff-e889bca278c3 1 2024-04-06T12:11:47.426 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:47.428+0000 7f3e06cc8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:11:47.476 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:47.477+0000 7f70d0691200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:11:47.547 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.549+0000 7f8b13850700 1 Processor -- start 2024-04-06T12:11:47.547 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.549+0000 7f8b13850700 1 -- start start 2024-04-06T12:11:47.549 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.550+0000 7f8b13850700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8b0c15d7b0 0x7f8b0c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:47.549 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.550+0000 7f8b13850700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8b0c15e500 0x7f8b0c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:47.550 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.550+0000 7f8b13850700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b0c0b3990 con 0x7f8b0c06a730 2024-04-06T12:11:47.550 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.550+0000 7f8b13850700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b0c06b0f0 con 0x7f8b0c15d7b0 2024-04-06T12:11:47.550 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.550+0000 7f8b13850700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c0b3060 con 0x7f8b0c15e500 2024-04-06T12:11:47.550 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.550+0000 7f8b115ec700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8b0c15d7b0 0x7f8b0c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:47.550 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.550+0000 7f8b11ded700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8b0c15e500 0x7f8b0c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:47.550 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.550+0000 7f8b115ec700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8b0c15d7b0 0x7f8b0c0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:58346/0 (socket says 172.21.15.192:58346) 2024-04-06T12:11:47.550 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.550+0000 7f8b115ec700 1 -- 172.21.15.192:0/2219725527 learned_addr learned my addr 172.21.15.192:0/2219725527 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-06T12:11:47.550 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.550+0000 7f8b11ded700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8b0c15e500 0x7f8b0c06abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:45040/0 (socket says 172.21.15.192:45040) 2024-04-06T12:11:47.550 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.550+0000 7f8b11ded700 1 -- 172.21.15.192:0/2219725527 >> v2:172.21.15.192:3300/0 conn(0x7f8b0c15d7b0 msgr2=0x7f8b0c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:11:47.550 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.550+0000 7f8b11ded700 1 --2- 172.21.15.192:0/2219725527 >> v2:172.21.15.192:3300/0 conn(0x7f8b0c15d7b0 0x7f8b0c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.550 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.550+0000 7f8b11ded700 1 -- 172.21.15.192:0/2219725527 >> v1:172.21.15.192:6789/0 conn(0x7f8b0c06a730 legacy=0x7f8b0c15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:47.551 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.550+0000 7f8b11ded700 1 -- 172.21.15.192:0/2219725527 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0c0b3340 con 0x7f8b0c15e500 2024-04-06T12:11:47.551 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.550+0000 7f8b115ec700 1 --2- 172.21.15.192:0/2219725527 >> v2:172.21.15.192:3300/0 conn(0x7f8b0c15d7b0 0x7f8b0c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:11:47.551 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.550+0000 7f8b11ded700 1 --2- 172.21.15.192:0/2219725527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8b0c15e500 0x7f8b0c06abb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8b04009d90 tx=0x7f8b0400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=694cfbe0d52664a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:47.551 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.551+0000 7f8b03fff700 1 -- 172.21.15.192:0/2219725527 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0400e070 con 0x7f8b0c15e500 2024-04-06T12:11:47.551 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.551+0000 7f8b03fff700 1 -- 172.21.15.192:0/2219725527 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0400bb20 con 0x7f8b0c15e500 2024-04-06T12:11:47.551 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.551+0000 7f8b03fff700 1 -- 172.21.15.192:0/2219725527 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b04012430 con 0x7f8b0c15e500 2024-04-06T12:11:47.551 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.551+0000 7f8b13850700 1 -- 172.21.15.192:0/2219725527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8b0c15e500 msgr2=0x7f8b0c06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:47.551 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.551+0000 7f8b13850700 1 --2- 172.21.15.192:0/2219725527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8b0c15e500 0x7f8b0c06abb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8b04009d90 tx=0x7f8b0400b330 comp rx=0 tx=0).stop 2024-04-06T12:11:47.551 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.551+0000 7f8b13850700 1 -- 172.21.15.192:0/2219725527 shutdown_connections 2024-04-06T12:11:47.551 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.551+0000 7f8b13850700 1 --2- 172.21.15.192:0/2219725527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8b0c15e500 0x7f8b0c06abb0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.551 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.551+0000 7f8b13850700 1 --2- 172.21.15.192:0/2219725527 >> v2:172.21.15.192:3300/0 conn(0x7f8b0c15d7b0 0x7f8b0c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.551 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.551+0000 7f8b13850700 1 -- 172.21.15.192:0/2219725527 >> 172.21.15.192:0/2219725527 conn(0x7f8b0c0d00b0 msgr2=0x7f8b0c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:47.551 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.551+0000 7f8b13850700 1 -- 172.21.15.192:0/2219725527 shutdown_connections 2024-04-06T12:11:47.551 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.551+0000 7f8b13850700 1 -- 172.21.15.192:0/2219725527 wait complete. 2024-04-06T12:11:47.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.553+0000 7f8b13850700 1 Processor -- start 2024-04-06T12:11:47.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.553+0000 7f8b13850700 1 -- start start 2024-04-06T12:11:47.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.554+0000 7f8b13850700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b0c06a730 0x7f8b0c150fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:47.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.554+0000 7f8b13850700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b0c15d7b0 0x7f8b0c14f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:47.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.554+0000 7f8b13850700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8b0c15e500 0x7f8b0c1517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:47.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.554+0000 7f8b13850700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c15ac00 con 0x7f8b0c15d7b0 2024-04-06T12:11:47.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.554+0000 7f8b13850700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c15a8b0 con 0x7f8b0c06a730 2024-04-06T12:11:47.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.554+0000 7f8b13850700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c153bc0 con 0x7f8b0c15e500 2024-04-06T12:11:47.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.554+0000 7f8b11ded700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8b0c15e500 0x7f8b0c1517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:47.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.554+0000 7f8b115ec700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b0c06a730 0x7f8b0c150fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:47.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.554+0000 7f8b11ded700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8b0c15e500 0x7f8b0c1517c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:45052/0 (socket says 172.21.15.192:45052) 2024-04-06T12:11:47.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.554+0000 7f8b115ec700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b0c06a730 0x7f8b0c150fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:58350/0 (socket says 172.21.15.192:58350) 2024-04-06T12:11:47.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.554+0000 7f8b11ded700 1 -- 172.21.15.192:0/933941542 learned_addr learned my addr 172.21.15.192:0/933941542 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-06T12:11:47.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.554+0000 7f8b10deb700 1 --2- 172.21.15.192:0/933941542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b0c15d7b0 0x7f8b0c14f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:47.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.554+0000 7f8b115ec700 1 -- 172.21.15.192:0/933941542 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8b0c15e500 msgr2=0x7f8b0c1517c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:47.555 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.554+0000 7f8b115ec700 1 --2- 172.21.15.192:0/933941542 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8b0c15e500 0x7f8b0c1517c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.554+0000 7f8b115ec700 1 -- 172.21.15.192:0/933941542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b0c15d7b0 msgr2=0x7f8b0c14f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:47.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.554+0000 7f8b115ec700 1 --2- 172.21.15.192:0/933941542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b0c15d7b0 0x7f8b0c14f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.554+0000 7f8b115ec700 1 -- 172.21.15.192:0/933941542 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8afc009550 con 0x7f8b0c06a730 2024-04-06T12:11:47.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.554+0000 7f8b10deb700 1 --2- 172.21.15.192:0/933941542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b0c15d7b0 0x7f8b0c14f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:11:47.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.554+0000 7f8b115ec700 1 --2- 172.21.15.192:0/933941542 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b0c06a730 0x7f8b0c150fb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00e9a0 tx=0x7f8afc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:47.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.555+0000 7f8b01ffb700 1 -- 172.21.15.192:0/933941542 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc004030 con 0x7f8b0c06a730 2024-04-06T12:11:47.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.555+0000 7f8b01ffb700 1 -- 172.21.15.192:0/933941542 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8afc004190 con 0x7f8b0c06a730 2024-04-06T12:11:47.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.555+0000 7f8b13850700 1 -- 172.21.15.192:0/933941542 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b04009a70 con 0x7f8b0c06a730 2024-04-06T12:11:47.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.555+0000 7f8b01ffb700 1 -- 172.21.15.192:0/933941542 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc004520 con 0x7f8b0c06a730 2024-04-06T12:11:47.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.555+0000 7f8b01ffb700 1 -- 172.21.15.192:0/933941542 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f8afc00cb50 con 0x7f8b0c06a730 2024-04-06T12:11:47.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.555+0000 7f8b13850700 1 -- 172.21.15.192:0/933941542 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b0c154180 con 0x7f8b0c06a730 2024-04-06T12:11:47.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.555+0000 7f8b01ffb700 1 -- 172.21.15.192:0/933941542 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f8afc00ce00 con 0x7f8b0c06a730 2024-04-06T12:11:47.556 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.556+0000 7f8b13850700 1 -- 172.21.15.192:0/933941542 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b0c15ae70 con 0x7f8b0c06a730 2024-04-06T12:11:47.557 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.558+0000 7f8b01ffb700 1 -- 172.21.15.192:0/933941542 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7f8afc03ac20 con 0x7f8b0c06a730 2024-04-06T12:11:47.601 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:47.603+0000 7f3e06cc8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:11:47.641 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:47.643+0000 7f70d0691200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:11:47.642 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.644+0000 7f8b13850700 1 -- 172.21.15.192:0/933941542 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2d7c167b-3876-470d-9cff-e889bca278c3", "id": 1} v 0) v1 -- 0x7f8b0c067bb0 con 0x7f8b0c06a730 2024-04-06T12:11:47.648 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.650+0000 7f8b01ffb700 1 -- 172.21.15.192:0/933941542 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2d7c167b-3876-470d-9cff-e889bca278c3", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8afc029070 con 0x7f8b0c06a730 2024-04-06T12:11:47.648 INFO:teuthology.orchestra.run.smithi192.stdout:1 2024-04-06T12:11:47.649 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.651+0000 7f8b13850700 1 -- 172.21.15.192:0/933941542 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b0c06a730 msgr2=0x7f8b0c150fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:47.649 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.651+0000 7f8b13850700 1 --2- 172.21.15.192:0/933941542 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b0c06a730 0x7f8b0c150fb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00e9a0 tx=0x7f8afc00ecb0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.649 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.651+0000 7f8b13850700 1 -- 172.21.15.192:0/933941542 shutdown_connections 2024-04-06T12:11:47.650 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.651+0000 7f8b13850700 1 --2- 172.21.15.192:0/933941542 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8b0c15e500 0x7f8b0c1517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.650 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.651+0000 7f8b13850700 1 --2- 172.21.15.192:0/933941542 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b0c06a730 0x7f8b0c150fb0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.650 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.651+0000 7f8b13850700 1 --2- 172.21.15.192:0/933941542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b0c15d7b0 0x7f8b0c14f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.650 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.651+0000 7f8b13850700 1 -- 172.21.15.192:0/933941542 >> 172.21.15.192:0/933941542 conn(0x7f8b0c0d00b0 msgr2=0x7f8b0c0aeb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:47.650 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.651+0000 7f8b13850700 1 -- 172.21.15.192:0/933941542 shutdown_connections 2024-04-06T12:11:47.650 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.651+0000 7f8b13850700 1 -- 172.21.15.192:0/933941542 wait complete. 2024-04-06T12:11:47.661 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new fc510b86-46ed-4c2c-8a50-211c375a1d40 2 2024-04-06T12:11:47.722 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:47.724+0000 7f3e06cc8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:11:47.752 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:47.754+0000 7f70d0691200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:11:47.783 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.785+0000 7ff600be9700 1 Processor -- start 2024-04-06T12:11:47.783 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.785+0000 7ff600be9700 1 -- start start 2024-04-06T12:11:47.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.786+0000 7ff600be9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff5fc158fe0 0x7ff5fc06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:47.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.786+0000 7ff600be9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc06ac70 0x7ff5fc15bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:47.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.786+0000 7ff600be9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5fc15f9e0 con 0x7ff5fc15c100 2024-04-06T12:11:47.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.786+0000 7ff600be9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5fc06b0f0 con 0x7ff5fc158fe0 2024-04-06T12:11:47.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.786+0000 7ff600be9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc15f0d0 con 0x7ff5fc06ac70 2024-04-06T12:11:47.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.786+0000 7ff5f9d9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc06ac70 0x7ff5fc15bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:47.786 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.786+0000 7ff5fa59c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff5fc158fe0 0x7ff5fc06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:47.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.786+0000 7ff5f9d9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc06ac70 0x7ff5fc15bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:45066/0 (socket says 172.21.15.192:45066) 2024-04-06T12:11:47.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.786+0000 7ff5fa59c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff5fc158fe0 0x7ff5fc06a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:58352/0 (socket says 172.21.15.192:58352) 2024-04-06T12:11:47.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.786+0000 7ff5f9d9b700 1 -- 172.21.15.192:0/3013030539 learned_addr learned my addr 172.21.15.192:0/3013030539 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-06T12:11:47.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.787+0000 7ff5f9d9b700 1 -- 172.21.15.192:0/3013030539 >> v2:172.21.15.192:3300/0 conn(0x7ff5fc158fe0 msgr2=0x7ff5fc06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:11:47.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.787+0000 7ff5f9d9b700 1 --2- 172.21.15.192:0/3013030539 >> v2:172.21.15.192:3300/0 conn(0x7ff5fc158fe0 0x7ff5fc06a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.787+0000 7ff5f9d9b700 1 -- 172.21.15.192:0/3013030539 >> v1:172.21.15.192:6789/0 conn(0x7ff5fc15c100 legacy=0x7ff5fc0717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:47.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.787+0000 7ff5f9d9b700 1 -- 172.21.15.192:0/3013030539 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5fc15c8f0 con 0x7ff5fc06ac70 2024-04-06T12:11:47.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.787+0000 7ff5fa59c700 1 --2- 172.21.15.192:0/3013030539 >> v2:172.21.15.192:3300/0 conn(0x7ff5fc158fe0 0x7ff5fc06a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:11:47.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.787+0000 7ff5f9d9b700 1 --2- 172.21.15.192:0/3013030539 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc06ac70 0x7ff5fc15bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0098f0 tx=0x7ff5ec009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99010617406c31c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:47.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.787+0000 7ff5f959a700 1 -- 172.21.15.192:0/3013030539 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec011830 con 0x7ff5fc06ac70 2024-04-06T12:11:47.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.787+0000 7ff5f959a700 1 -- 172.21.15.192:0/3013030539 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ec011990 con 0x7ff5fc06ac70 2024-04-06T12:11:47.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.787+0000 7ff5f959a700 1 -- 172.21.15.192:0/3013030539 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec011d20 con 0x7ff5fc06ac70 2024-04-06T12:11:47.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.787+0000 7ff600be9700 1 -- 172.21.15.192:0/3013030539 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc06ac70 msgr2=0x7ff5fc15bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:47.787 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.787+0000 7ff600be9700 1 --2- 172.21.15.192:0/3013030539 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc06ac70 0x7ff5fc15bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0098f0 tx=0x7ff5ec009c00 comp rx=0 tx=0).stop 2024-04-06T12:11:47.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.788+0000 7ff600be9700 1 -- 172.21.15.192:0/3013030539 shutdown_connections 2024-04-06T12:11:47.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.788+0000 7ff600be9700 1 --2- 172.21.15.192:0/3013030539 >> v2:172.21.15.192:3300/0 conn(0x7ff5fc158fe0 0x7ff5fc06a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.788+0000 7ff600be9700 1 --2- 172.21.15.192:0/3013030539 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc06ac70 0x7ff5fc15bbc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.788+0000 7ff600be9700 1 -- 172.21.15.192:0/3013030539 >> 172.21.15.192:0/3013030539 conn(0x7ff5fc0d00b0 msgr2=0x7ff5fc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:47.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.788+0000 7ff600be9700 1 -- 172.21.15.192:0/3013030539 shutdown_connections 2024-04-06T12:11:47.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.788+0000 7ff600be9700 1 -- 172.21.15.192:0/3013030539 wait complete. 2024-04-06T12:11:47.788 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.790+0000 7ff600be9700 1 Processor -- start 2024-04-06T12:11:47.789 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.790+0000 7ff600be9700 1 -- start start 2024-04-06T12:11:47.791 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.791+0000 7ff600be9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5fc06ac70 0x7ff5fc158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:47.791 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.791+0000 7ff600be9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc158fe0 0x7ff5fc0b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:47.791 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.791+0000 7ff600be9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc15c100 0x7ff5fc0b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:47.791 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.791+0000 7ff600be9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc13fdb0 con 0x7ff5fc06ac70 2024-04-06T12:11:47.792 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.791+0000 7ff600be9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc13ed10 con 0x7ff5fc158fe0 2024-04-06T12:11:47.792 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.791+0000 7ff600be9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc13eff0 con 0x7ff5fc15c100 2024-04-06T12:11:47.792 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.791+0000 7ff5f9d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc158fe0 0x7ff5fc0b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:47.792 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.791+0000 7ff5fad9d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc15c100 0x7ff5fc0b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:47.792 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.791+0000 7ff5f9d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc158fe0 0x7ff5fc0b0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:58366/0 (socket says 172.21.15.192:58366) 2024-04-06T12:11:47.792 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.791+0000 7ff5f9d9b700 1 -- 172.21.15.192:0/1658197823 learned_addr learned my addr 172.21.15.192:0/1658197823 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-06T12:11:47.792 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.791+0000 7ff5fa59c700 1 --2- 172.21.15.192:0/1658197823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5fc06ac70 0x7ff5fc158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:47.792 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.791+0000 7ff5fad9d700 1 -- 172.21.15.192:0/1658197823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc158fe0 msgr2=0x7ff5fc0b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:47.792 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.791+0000 7ff5fad9d700 1 --2- 172.21.15.192:0/1658197823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc158fe0 0x7ff5fc0b0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.792 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.791+0000 7ff5fad9d700 1 -- 172.21.15.192:0/1658197823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5fc06ac70 msgr2=0x7ff5fc158210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:47.792 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.791+0000 7ff5fad9d700 1 --2- 172.21.15.192:0/1658197823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5fc06ac70 0x7ff5fc158210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.792 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.791+0000 7ff5fad9d700 1 -- 172.21.15.192:0/1658197823 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f00094a0 con 0x7ff5fc15c100 2024-04-06T12:11:47.792 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.791+0000 7ff5f9d9b700 1 --2- 172.21.15.192:0/1658197823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc158fe0 0x7ff5fc0b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:11:47.792 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.792+0000 7ff5fad9d700 1 --2- 172.21.15.192:0/1658197823 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc15c100 0x7ff5fc0b12f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400cb90 tx=0x7ff5f400cea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:47.792 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.792+0000 7ff5eaffd700 1 -- 172.21.15.192:0/1658197823 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4003c00 con 0x7ff5fc15c100 2024-04-06T12:11:47.793 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.792+0000 7ff5eaffd700 1 -- 172.21.15.192:0/1658197823 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f4003d60 con 0x7ff5fc15c100 2024-04-06T12:11:47.793 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.792+0000 7ff600be9700 1 -- 172.21.15.192:0/1658197823 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5ec009550 con 0x7ff5fc15c100 2024-04-06T12:11:47.793 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.792+0000 7ff5eaffd700 1 -- 172.21.15.192:0/1658197823 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f400aa80 con 0x7ff5fc15c100 2024-04-06T12:11:47.793 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.792+0000 7ff600be9700 1 -- 172.21.15.192:0/1658197823 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5fc13f5b0 con 0x7ff5fc15c100 2024-04-06T12:11:47.793 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.792+0000 7ff5eaffd700 1 -- 172.21.15.192:0/1658197823 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7ff5f40050d0 con 0x7ff5fc15c100 2024-04-06T12:11:47.793 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.792+0000 7ff5eaffd700 1 -- 172.21.15.192:0/1658197823 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7ff5f40057a0 con 0x7ff5fc15c100 2024-04-06T12:11:47.793 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.793+0000 7ff600be9700 1 -- 172.21.15.192:0/1658197823 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5fc067bb0 con 0x7ff5fc15c100 2024-04-06T12:11:47.794 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.797+0000 7ff5eaffd700 1 -- 172.21.15.192:0/1658197823 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7ff5f4005380 con 0x7ff5fc15c100 2024-04-06T12:11:47.825 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:47.828+0000 7f3e06cc8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:11:47.847 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:47.849+0000 7f70d0691200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:11:47.878 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.880+0000 7ff600be9700 1 -- 172.21.15.192:0/1658197823 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd new", "uuid": "fc510b86-46ed-4c2c-8a50-211c375a1d40", "id": 2} v 0) v1 -- 0x7ff5fc0cfcb0 con 0x7ff5fc15c100 2024-04-06T12:11:47.886 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.888+0000 7ff5eaffd700 1 -- 172.21.15.192:0/1658197823 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fc510b86-46ed-4c2c-8a50-211c375a1d40", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff5f4005600 con 0x7ff5fc15c100 2024-04-06T12:11:47.887 INFO:teuthology.orchestra.run.smithi192.stdout:2 2024-04-06T12:11:47.888 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.889+0000 7ff600be9700 1 -- 172.21.15.192:0/1658197823 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc15c100 msgr2=0x7ff5fc0b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:47.888 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.889+0000 7ff600be9700 1 --2- 172.21.15.192:0/1658197823 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc15c100 0x7ff5fc0b12f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400cb90 tx=0x7ff5f400cea0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.888 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.889+0000 7ff600be9700 1 -- 172.21.15.192:0/1658197823 shutdown_connections 2024-04-06T12:11:47.888 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.889+0000 7ff600be9700 1 --2- 172.21.15.192:0/1658197823 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc15c100 0x7ff5fc0b12f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.888 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.889+0000 7ff600be9700 1 --2- 172.21.15.192:0/1658197823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5fc06ac70 0x7ff5fc158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.888 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.889+0000 7ff600be9700 1 --2- 172.21.15.192:0/1658197823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc158fe0 0x7ff5fc0b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:47.888 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.889+0000 7ff600be9700 1 -- 172.21.15.192:0/1658197823 >> 172.21.15.192:0/1658197823 conn(0x7ff5fc0d00b0 msgr2=0x7ff5fc15b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:47.888 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.889+0000 7ff600be9700 1 -- 172.21.15.192:0/1658197823 shutdown_connections 2024-04-06T12:11:47.888 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:47.889+0000 7ff600be9700 1 -- 172.21.15.192:0/1658197823 wait complete. 2024-04-06T12:11:47.899 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 56f5a991-198d-4087-90d4-72338925912c 3 2024-04-06T12:11:47.906 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:47.908+0000 7f3e06cc8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:11:47.924 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:47.926+0000 7f70d0691200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:11:48.024 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.025+0000 7f32de25e700 1 Processor -- start 2024-04-06T12:11:48.024 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.025+0000 7f32de25e700 1 -- start start 2024-04-06T12:11:48.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.026+0000 7f32de25e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f32d80d0a60 0x7f32d80d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:48.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.026+0000 7f32de25e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32d80d1af0 0x7f32d80b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:48.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.026+0000 7f32de25e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d80cdf80 con 0x7f32d815d7e0 2024-04-06T12:11:48.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.026+0000 7f32de25e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d80d1f80 con 0x7f32d80d0a60 2024-04-06T12:11:48.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.026+0000 7f32de25e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d80d3230 con 0x7f32d80d1af0 2024-04-06T12:11:48.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.026+0000 7f32d6ffd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f32d80d0a60 0x7f32d80d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:48.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.026+0000 7f32d7fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32d80d1af0 0x7f32d80b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:48.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.026+0000 7f32d6ffd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f32d80d0a60 0x7f32d80d15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:58372/0 (socket says 172.21.15.192:58372) 2024-04-06T12:11:48.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.026+0000 7f32d7fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32d80d1af0 0x7f32d80b66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:45072/0 (socket says 172.21.15.192:45072) 2024-04-06T12:11:48.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.026+0000 7f32d77fe700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f32d815d7e0 0x7f32d80d0350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:55508/0 (socket says 172.21.15.192:55508) 2024-04-06T12:11:48.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.026+0000 7f32d6ffd700 1 -- 172.21.15.192:0/3633217057 learned_addr learned my addr 172.21.15.192:0/3633217057 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-06T12:11:48.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.027+0000 7f32d6ffd700 1 -- 172.21.15.192:0/3633217057 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32d80d1af0 msgr2=0x7f32d80b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:11:48.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.027+0000 7f32d6ffd700 1 --2- 172.21.15.192:0/3633217057 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32d80d1af0 0x7f32d80b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.027+0000 7f32d6ffd700 1 -- 172.21.15.192:0/3633217057 >> v1:172.21.15.192:6789/0 conn(0x7f32d815d7e0 legacy=0x7f32d80d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.027+0000 7f32d6ffd700 1 -- 172.21.15.192:0/3633217057 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d80d2da0 con 0x7f32d80d0a60 2024-04-06T12:11:48.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.027+0000 7f32d67fc700 1 -- 172.21.15.192:0/3633217057 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954396208 0 0) 0x7f32d80cdf80 con 0x7f32d815d7e0 2024-04-06T12:11:48.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.027+0000 7f32d7fff700 1 --2- 172.21.15.192:0/3633217057 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32d80d1af0 0x7f32d80b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:11:48.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.027+0000 7f32d6ffd700 1 --2- 172.21.15.192:0/3633217057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32d80d0a60 0x7f32d80d15b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f32cc0098f0 tx=0x7f32cc009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f95a8a0168cefe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:48.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.027+0000 7f32d67fc700 1 -- 172.21.15.192:0/3633217057 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32cc011830 con 0x7f32d80d0a60 2024-04-06T12:11:48.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.027+0000 7f32d67fc700 1 -- 172.21.15.192:0/3633217057 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32cc011990 con 0x7f32d80d0a60 2024-04-06T12:11:48.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.027+0000 7f32d67fc700 1 -- 172.21.15.192:0/3633217057 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32cc011d20 con 0x7f32d80d0a60 2024-04-06T12:11:48.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.028+0000 7f32de25e700 1 -- 172.21.15.192:0/3633217057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32d80d0a60 msgr2=0x7f32d80d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.028+0000 7f32de25e700 1 --2- 172.21.15.192:0/3633217057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32d80d0a60 0x7f32d80d15b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f32cc0098f0 tx=0x7f32cc009c00 comp rx=0 tx=0).stop 2024-04-06T12:11:48.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.028+0000 7f32de25e700 1 -- 172.21.15.192:0/3633217057 shutdown_connections 2024-04-06T12:11:48.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.028+0000 7f32de25e700 1 --2- 172.21.15.192:0/3633217057 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32d80d1af0 0x7f32d80b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.028+0000 7f32de25e700 1 --2- 172.21.15.192:0/3633217057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32d80d0a60 0x7f32d80d15b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.029 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.028+0000 7f32de25e700 1 -- 172.21.15.192:0/3633217057 >> 172.21.15.192:0/3633217057 conn(0x7f32d8072250 msgr2=0x7f32d8072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:48.029 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.028+0000 7f32de25e700 1 -- 172.21.15.192:0/3633217057 shutdown_connections 2024-04-06T12:11:48.029 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.028+0000 7f32de25e700 1 -- 172.21.15.192:0/3633217057 wait complete. 2024-04-06T12:11:48.031 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.029+0000 7f32de25e700 1 Processor -- start 2024-04-06T12:11:48.031 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.030+0000 7f32de25e700 1 -- start start 2024-04-06T12:11:48.031 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.030+0000 7f32de25e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32d80d0a60 0x7f32d80c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:48.031 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.030+0000 7f32de25e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32d80d1af0 0x7f32d80c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:48.031 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.030+0000 7f32de25e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32d815d7e0 0x7f32d81f2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:48.032 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.030+0000 7f32de25e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d80b4db0 con 0x7f32d80d0a60 2024-04-06T12:11:48.032 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.030+0000 7f32de25e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d80c70f0 con 0x7f32d815d7e0 2024-04-06T12:11:48.032 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.030+0000 7f32de25e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d80b4480 con 0x7f32d80d1af0 2024-04-06T12:11:48.032 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32d6ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32d80d1af0 0x7f32d80c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:48.032 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32d6ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32d80d1af0 0x7f32d80c7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:45078/0 (socket says 172.21.15.192:45078) 2024-04-06T12:11:48.032 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32d6ffd700 1 -- 172.21.15.192:0/1671515466 learned_addr learned my addr 172.21.15.192:0/1671515466 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-06T12:11:48.032 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32d7fff700 1 --2- 172.21.15.192:0/1671515466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32d815d7e0 0x7f32d81f2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:48.032 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32d77fe700 1 --2- 172.21.15.192:0/1671515466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32d80d0a60 0x7f32d80c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:48.032 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32d7fff700 1 -- 172.21.15.192:0/1671515466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32d80d1af0 msgr2=0x7f32d80c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.032 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32d7fff700 1 --2- 172.21.15.192:0/1671515466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32d80d1af0 0x7f32d80c7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.032 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32d7fff700 1 -- 172.21.15.192:0/1671515466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32d80d0a60 msgr2=0x7f32d80c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.032 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32d7fff700 1 --2- 172.21.15.192:0/1671515466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32d80d0a60 0x7f32d80c6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.032 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32d7fff700 1 -- 172.21.15.192:0/1671515466 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d00094a0 con 0x7f32d815d7e0 2024-04-06T12:11:48.033 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32d6ffd700 1 --2- 172.21.15.192:0/1671515466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32d80d1af0 0x7f32d80c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:11:48.033 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32d77fe700 1 --2- 172.21.15.192:0/1671515466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32d80d0a60 0x7f32d80c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:11:48.033 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32d7fff700 1 --2- 172.21.15.192:0/1671515466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32d815d7e0 0x7f32d81f2370 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f32d000e910 tx=0x7f32d000ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:48.033 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32bffff700 1 -- 172.21.15.192:0/1671515466 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d000ee60 con 0x7f32d815d7e0 2024-04-06T12:11:48.033 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32bffff700 1 -- 172.21.15.192:0/1671515466 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d0004020 con 0x7f32d815d7e0 2024-04-06T12:11:48.033 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32de25e700 1 -- 172.21.15.192:0/1671515466 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32cc009550 con 0x7f32d815d7e0 2024-04-06T12:11:48.033 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32bffff700 1 -- 172.21.15.192:0/1671515466 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d00043d0 con 0x7f32d815d7e0 2024-04-06T12:11:48.033 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32de25e700 1 -- 172.21.15.192:0/1671515466 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d80b4a40 con 0x7f32d815d7e0 2024-04-06T12:11:48.033 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32bffff700 1 -- 172.21.15.192:0/1671515466 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f32d0005d20 con 0x7f32d815d7e0 2024-04-06T12:11:48.033 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.031+0000 7f32bffff700 1 -- 172.21.15.192:0/1671515466 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f32d0003c70 con 0x7f32d815d7e0 2024-04-06T12:11:48.033 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.032+0000 7f32de25e700 1 -- 172.21.15.192:0/1671515466 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d8074990 con 0x7f32d815d7e0 2024-04-06T12:11:48.034 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.034+0000 7f32bffff700 1 -- 172.21.15.192:0/1671515466 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7f32d0020060 con 0x7f32d815d7e0 2024-04-06T12:11:48.144 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:48.145+0000 7f3e06cc8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:11:48.149 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:48.151+0000 7f70d0691200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:11:48.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.153+0000 7f32de25e700 1 -- 172.21.15.192:0/1671515466 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "56f5a991-198d-4087-90d4-72338925912c", "id": 3} v 0) v1 -- 0x7f32d8073a70 con 0x7f32d815d7e0 2024-04-06T12:11:48.159 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.161+0000 7f32bffff700 1 -- 172.21.15.192:0/1671515466 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "56f5a991-198d-4087-90d4-72338925912c", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f32d0017070 con 0x7f32d815d7e0 2024-04-06T12:11:48.159 INFO:teuthology.orchestra.run.smithi192.stdout:3 2024-04-06T12:11:48.161 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.162+0000 7f32de25e700 1 -- 172.21.15.192:0/1671515466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32d815d7e0 msgr2=0x7f32d81f2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.161 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.162+0000 7f32de25e700 1 --2- 172.21.15.192:0/1671515466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32d815d7e0 0x7f32d81f2370 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f32d000e910 tx=0x7f32d000ec20 comp rx=0 tx=0).stop 2024-04-06T12:11:48.161 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.162+0000 7f32de25e700 1 -- 172.21.15.192:0/1671515466 shutdown_connections 2024-04-06T12:11:48.161 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.162+0000 7f32de25e700 1 --2- 172.21.15.192:0/1671515466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32d815d7e0 0x7f32d81f2370 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.161 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.162+0000 7f32de25e700 1 --2- 172.21.15.192:0/1671515466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32d80d0a60 0x7f32d80c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.161 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.162+0000 7f32de25e700 1 --2- 172.21.15.192:0/1671515466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32d80d1af0 0x7f32d80c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.161 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.162+0000 7f32de25e700 1 -- 172.21.15.192:0/1671515466 >> 172.21.15.192:0/1671515466 conn(0x7f32d8072250 msgr2=0x7f32d80cd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:48.161 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.163+0000 7f32de25e700 1 -- 172.21.15.192:0/1671515466 shutdown_connections 2024-04-06T12:11:48.161 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.163+0000 7f32de25e700 1 -- 172.21.15.192:0/1671515466 wait complete. 2024-04-06T12:11:48.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new c27c81cb-146f-4950-8f79-fd82d2ba6c17 4 2024-04-06T12:11:48.218 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:48.220+0000 7f3e06cc8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:11:48.221 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:48.223+0000 7f70d0691200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:11:48.336 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.337+0000 7fb7a7bea700 1 Processor -- start 2024-04-06T12:11:48.336 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.338+0000 7fb7a7bea700 1 -- start start 2024-04-06T12:11:48.339 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.339+0000 7fb7a7bea700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb7a00b4660 0x7fb7a00b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:48.339 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.339+0000 7fb7a7bea700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb7a00b2fc0 0x7fb7a00b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:48.339 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.339+0000 7fb7a7bea700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a00b5030 con 0x7fb7a015a550 2024-04-06T12:11:48.339 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.339+0000 7fb7a7bea700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a00cdbe0 con 0x7fb7a00b2fc0 2024-04-06T12:11:48.339 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.339+0000 7fb7a7bea700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a015a940 con 0x7fb7a00b4660 2024-04-06T12:11:48.340 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.339+0000 7fb7a5185700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb7a00b4660 0x7fb7a00b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:48.340 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.339+0000 7fb7a5986700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fb7a015a550 0x7fb7a00cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:55522/0 (socket says 172.21.15.192:55522) 2024-04-06T12:11:48.340 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.339+0000 7fb7a5185700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb7a00b4660 0x7fb7a00b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:58392/0 (socket says 172.21.15.192:58392) 2024-04-06T12:11:48.340 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.339+0000 7fb7a5986700 1 -- 172.21.15.192:0/3295188325 learned_addr learned my addr 172.21.15.192:0/3295188325 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-04-06T12:11:48.340 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.339+0000 7fb7a6187700 1 --2- 172.21.15.192:0/3295188325 >> v2:172.21.15.189:3300/0 conn(0x7fb7a00b2fc0 0x7fb7a00b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:48.340 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.340+0000 7fb7a5185700 1 -- 172.21.15.192:0/3295188325 >> v2:172.21.15.189:3300/0 conn(0x7fb7a00b2fc0 msgr2=0x7fb7a00b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:11:48.340 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.340+0000 7fb7a5185700 1 --2- 172.21.15.192:0/3295188325 >> v2:172.21.15.189:3300/0 conn(0x7fb7a00b2fc0 0x7fb7a00b33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.340 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.340+0000 7fb7a5185700 1 -- 172.21.15.192:0/3295188325 >> v1:172.21.15.192:6789/0 conn(0x7fb7a015a550 legacy=0x7fb7a00cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.340 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.340+0000 7fb7a5185700 1 -- 172.21.15.192:0/3295188325 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a015ac00 con 0x7fb7a00b4660 2024-04-06T12:11:48.340 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.340+0000 7fb7a4984700 1 -- 172.21.15.192:0/3295188325 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902262064 0 0) 0x7fb7a00b5030 con 0x7fb7a015a550 2024-04-06T12:11:48.340 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.340+0000 7fb7a5185700 1 --2- 172.21.15.192:0/3295188325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7a00b4660 0x7fb7a00b4af0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb7a00b2d40 tx=0x7fb790009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eff65b1bedd9725f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:48.340 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.340+0000 7fb7a4984700 1 -- 172.21.15.192:0/3295188325 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb790011830 con 0x7fb7a00b4660 2024-04-06T12:11:48.341 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.340+0000 7fb7a4984700 1 -- 172.21.15.192:0/3295188325 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb790011990 con 0x7fb7a00b4660 2024-04-06T12:11:48.341 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.340+0000 7fb7a4984700 1 -- 172.21.15.192:0/3295188325 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb790011d20 con 0x7fb7a00b4660 2024-04-06T12:11:48.341 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.341+0000 7fb7a7bea700 1 -- 172.21.15.192:0/3295188325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7a00b4660 msgr2=0x7fb7a00b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.341 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.341+0000 7fb7a7bea700 1 --2- 172.21.15.192:0/3295188325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7a00b4660 0x7fb7a00b4af0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb7a00b2d40 tx=0x7fb790009b20 comp rx=0 tx=0).stop 2024-04-06T12:11:48.341 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.341+0000 7fb7a7bea700 1 -- 172.21.15.192:0/3295188325 shutdown_connections 2024-04-06T12:11:48.341 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.341+0000 7fb7a7bea700 1 --2- 172.21.15.192:0/3295188325 >> v2:172.21.15.189:3300/0 conn(0x7fb7a00b2fc0 0x7fb7a00b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.341 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.341+0000 7fb7a7bea700 1 --2- 172.21.15.192:0/3295188325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7a00b4660 0x7fb7a00b4af0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.341 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.341+0000 7fb7a7bea700 1 -- 172.21.15.192:0/3295188325 >> 172.21.15.192:0/3295188325 conn(0x7fb7a00d00b0 msgr2=0x7fb7a00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:48.341 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.341+0000 7fb7a7bea700 1 -- 172.21.15.192:0/3295188325 shutdown_connections 2024-04-06T12:11:48.341 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.341+0000 7fb7a7bea700 1 -- 172.21.15.192:0/3295188325 wait complete. 2024-04-06T12:11:48.344 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.342+0000 7fb7a7bea700 1 Processor -- start 2024-04-06T12:11:48.344 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.342+0000 7fb7a7bea700 1 -- start start 2024-04-06T12:11:48.344 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.343+0000 7fb7a7bea700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7a00b2fc0 0x7fb7a014ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:48.344 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.343+0000 7fb7a7bea700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7a00b4660 0x7fb7a014e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:48.344 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.343+0000 7fb7a7bea700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7a01ef6f0 0x7fb7a01f2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:48.344 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.343+0000 7fb7a7bea700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a014ee10 con 0x7fb7a00b4660 2024-04-06T12:11:48.344 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.343+0000 7fb7a7bea700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a014c110 con 0x7fb7a00b2fc0 2024-04-06T12:11:48.345 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.343+0000 7fb7a7bea700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a014c3f0 con 0x7fb7a01ef6f0 2024-04-06T12:11:48.345 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.343+0000 7fb7a5986700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7a00b2fc0 0x7fb7a014ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:48.345 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.343+0000 7fb7a6187700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7a01ef6f0 0x7fb7a01f2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:48.345 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.343+0000 7fb7a5986700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7a00b2fc0 0x7fb7a014ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:58406/0 (socket says 172.21.15.192:58406) 2024-04-06T12:11:48.345 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.343+0000 7fb7a5986700 1 -- 172.21.15.192:0/963970289 learned_addr learned my addr 172.21.15.192:0/963970289 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-06T12:11:48.345 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.344+0000 7fb7a5185700 1 --2- 172.21.15.192:0/963970289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7a00b4660 0x7fb7a014e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:48.345 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.344+0000 7fb7a5986700 1 -- 172.21.15.192:0/963970289 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7a01ef6f0 msgr2=0x7fb7a01f2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.345 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.344+0000 7fb7a5986700 1 --2- 172.21.15.192:0/963970289 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7a01ef6f0 0x7fb7a01f2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.345 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.344+0000 7fb7a5986700 1 -- 172.21.15.192:0/963970289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7a00b4660 msgr2=0x7fb7a014e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.345 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.344+0000 7fb7a5986700 1 --2- 172.21.15.192:0/963970289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7a00b4660 0x7fb7a014e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.345 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.344+0000 7fb7a5986700 1 -- 172.21.15.192:0/963970289 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7900094a0 con 0x7fb7a00b2fc0 2024-04-06T12:11:48.345 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.344+0000 7fb7a6187700 1 --2- 172.21.15.192:0/963970289 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7a01ef6f0 0x7fb7a01f2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:11:48.345 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.344+0000 7fb7a5986700 1 --2- 172.21.15.192:0/963970289 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7a00b2fc0 0x7fb7a014ba10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb79c0011e0 tx=0x7fb79c00ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:48.345 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.344+0000 7fb7967fc700 1 -- 172.21.15.192:0/963970289 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c004c80 con 0x7fb7a00b2fc0 2024-04-06T12:11:48.346 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.344+0000 7fb7967fc700 1 -- 172.21.15.192:0/963970289 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79c004e00 con 0x7fb7a00b2fc0 2024-04-06T12:11:48.346 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.344+0000 7fb7a7bea700 1 -- 172.21.15.192:0/963970289 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a015b960 con 0x7fb7a00b2fc0 2024-04-06T12:11:48.346 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.344+0000 7fb7967fc700 1 -- 172.21.15.192:0/963970289 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c007310 con 0x7fb7a00b2fc0 2024-04-06T12:11:48.346 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.344+0000 7fb7a7bea700 1 -- 172.21.15.192:0/963970289 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a01efb70 con 0x7fb7a00b2fc0 2024-04-06T12:11:48.346 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.344+0000 7fb7967fc700 1 -- 172.21.15.192:0/963970289 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fb79c01d3f0 con 0x7fb7a00b2fc0 2024-04-06T12:11:48.346 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.344+0000 7fb7967fc700 1 -- 172.21.15.192:0/963970289 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fb79c0138d0 con 0x7fb7a00b2fc0 2024-04-06T12:11:48.346 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.345+0000 7fb7a7bea700 1 -- 172.21.15.192:0/963970289 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a0067bb0 con 0x7fb7a00b2fc0 2024-04-06T12:11:48.346 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.349+0000 7fb7967fc700 1 -- 172.21.15.192:0/963970289 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7fb79c013b30 con 0x7fb7a00b2fc0 2024-04-06T12:11:48.432 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.434+0000 7fb7a7bea700 1 -- 172.21.15.192:0/963970289 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c27c81cb-146f-4950-8f79-fd82d2ba6c17", "id": 4} v 0) v1 -- 0x7fb7a0074990 con 0x7fb7a00b2fc0 2024-04-06T12:11:48.439 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.440+0000 7fb7967fc700 1 -- 172.21.15.192:0/963970289 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c27c81cb-146f-4950-8f79-fd82d2ba6c17", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb79c013e00 con 0x7fb7a00b2fc0 2024-04-06T12:11:48.439 INFO:teuthology.orchestra.run.smithi192.stdout:4 2024-04-06T12:11:48.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.441+0000 7fb7a7bea700 1 -- 172.21.15.192:0/963970289 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7a00b2fc0 msgr2=0x7fb7a014ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.441+0000 7fb7a7bea700 1 --2- 172.21.15.192:0/963970289 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7a00b2fc0 0x7fb7a014ba10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb79c0011e0 tx=0x7fb79c00ca30 comp rx=0 tx=0).stop 2024-04-06T12:11:48.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.441+0000 7fb7a7bea700 1 -- 172.21.15.192:0/963970289 shutdown_connections 2024-04-06T12:11:48.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.441+0000 7fb7a7bea700 1 --2- 172.21.15.192:0/963970289 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7a01ef6f0 0x7fb7a01f2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.441+0000 7fb7a7bea700 1 --2- 172.21.15.192:0/963970289 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7a00b2fc0 0x7fb7a014ba10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.441+0000 7fb7a7bea700 1 --2- 172.21.15.192:0/963970289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7a00b4660 0x7fb7a014e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.441+0000 7fb7a7bea700 1 -- 172.21.15.192:0/963970289 >> 172.21.15.192:0/963970289 conn(0x7fb7a00d00b0 msgr2=0x7fb7a01579d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:48.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.441+0000 7fb7a7bea700 1 -- 172.21.15.192:0/963970289 shutdown_connections 2024-04-06T12:11:48.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.441+0000 7fb7a7bea700 1 -- 172.21.15.192:0/963970289 wait complete. 2024-04-06T12:11:48.451 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 8dd707b1-5813-478c-9579-7072ee3070c1 5 2024-04-06T12:11:48.602 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:48.604+0000 7f70d0691200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:11:48.618 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:48.620+0000 7f3e06cc8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:11:48.624 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.620+0000 7f542c5af700 1 Processor -- start 2024-04-06T12:11:48.625 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.620+0000 7f542c5af700 1 -- start start 2024-04-06T12:11:48.625 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.621+0000 7f542c5af700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f542406ae40 0x7f54240b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:48.625 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.621+0000 7f542c5af700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f542415ddf0 0x7f542415e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:48.625 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.621+0000 7f542c5af700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54240b3990 con 0x7f542415d7b0 2024-04-06T12:11:48.625 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.621+0000 7f542c5af700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54240b3060 con 0x7f542415ddf0 2024-04-06T12:11:48.625 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.621+0000 7f542c5af700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54240b3340 con 0x7f542406ae40 2024-04-06T12:11:48.625 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.621+0000 7f5429b4a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f542406ae40 0x7f54240b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:48.625 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.621+0000 7f5429b4a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f542406ae40 0x7f54240b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:45106/0 (socket says 172.21.15.192:45106) 2024-04-06T12:11:48.625 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.621+0000 7f5429b4a700 1 -- 172.21.15.192:0/2608642527 learned_addr learned my addr 172.21.15.192:0/2608642527 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-06T12:11:48.625 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.621+0000 7f542ab4c700 1 --2- 172.21.15.192:0/2608642527 >> v2:172.21.15.189:3300/0 conn(0x7f542415ddf0 0x7f542415e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:48.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.621+0000 7f5429b4a700 1 -- 172.21.15.192:0/2608642527 >> v2:172.21.15.189:3300/0 conn(0x7f542415ddf0 msgr2=0x7f542415e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:11:48.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.621+0000 7f5429b4a700 1 --2- 172.21.15.192:0/2608642527 >> v2:172.21.15.189:3300/0 conn(0x7f542415ddf0 0x7f542415e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.621+0000 7f5429b4a700 1 -- 172.21.15.192:0/2608642527 >> v1:172.21.15.189:6789/0 conn(0x7f542415d7b0 legacy=0x7f542406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.621+0000 7f5429b4a700 1 -- 172.21.15.192:0/2608642527 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54240b3620 con 0x7f542406ae40 2024-04-06T12:11:48.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.621+0000 7f5429b4a700 1 --2- 172.21.15.192:0/2608642527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f542406ae40 0x7f54240b3e20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f54140098f0 tx=0x7f5414009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0f7b1fcc7b16162 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:48.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.621+0000 7f5429349700 1 -- 172.21.15.192:0/2608642527 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5414011830 con 0x7f542406ae40 2024-04-06T12:11:48.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.622+0000 7f5429349700 1 -- 172.21.15.192:0/2608642527 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5414011990 con 0x7f542406ae40 2024-04-06T12:11:48.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.622+0000 7f5429349700 1 -- 172.21.15.192:0/2608642527 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5414011d20 con 0x7f542406ae40 2024-04-06T12:11:48.626 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.622+0000 7f542c5af700 1 -- 172.21.15.192:0/2608642527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f542406ae40 msgr2=0x7f54240b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.622+0000 7f542c5af700 1 --2- 172.21.15.192:0/2608642527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f542406ae40 0x7f54240b3e20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f54140098f0 tx=0x7f5414009c00 comp rx=0 tx=0).stop 2024-04-06T12:11:48.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.622+0000 7f542c5af700 1 -- 172.21.15.192:0/2608642527 shutdown_connections 2024-04-06T12:11:48.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.622+0000 7f542c5af700 1 --2- 172.21.15.192:0/2608642527 >> v2:172.21.15.189:3300/0 conn(0x7f542415ddf0 0x7f542415e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.622+0000 7f542c5af700 1 --2- 172.21.15.192:0/2608642527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f542406ae40 0x7f54240b3e20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.622+0000 7f542c5af700 1 -- 172.21.15.192:0/2608642527 >> 172.21.15.192:0/2608642527 conn(0x7f54240d00b0 msgr2=0x7f54240d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:48.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.622+0000 7f542c5af700 1 -- 172.21.15.192:0/2608642527 shutdown_connections 2024-04-06T12:11:48.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.622+0000 7f542c5af700 1 -- 172.21.15.192:0/2608642527 wait complete. 2024-04-06T12:11:48.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.623+0000 7f542c5af700 1 Processor -- start 2024-04-06T12:11:48.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.623+0000 7f542c5af700 1 -- start start 2024-04-06T12:11:48.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.624+0000 7f542c5af700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f542406ae40 0x7f542415c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:48.627 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.624+0000 7f542c5af700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f542415d7b0 0x7f54241f2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:48.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.624+0000 7f542c5af700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f542415ddf0 0x7f542415b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:48.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.624+0000 7f542c5af700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54241f2c80 con 0x7f542406ae40 2024-04-06T12:11:48.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.624+0000 7f542c5af700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5424149cf0 con 0x7f542415d7b0 2024-04-06T12:11:48.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.624+0000 7f542c5af700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5424143900 con 0x7f542415ddf0 2024-04-06T12:11:48.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.624+0000 7f542ab4c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f542415ddf0 0x7f542415b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:48.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.624+0000 7f542ab4c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f542415ddf0 0x7f542415b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:45118/0 (socket says 172.21.15.192:45118) 2024-04-06T12:11:48.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.624+0000 7f5429b4a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f542415d7b0 0x7f54241f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:48.628 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.624+0000 7f542ab4c700 1 -- 172.21.15.192:0/1276228114 learned_addr learned my addr 172.21.15.192:0/1276228114 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-06T12:11:48.629 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.624+0000 7f542a34b700 1 --2- 172.21.15.192:0/1276228114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f542406ae40 0x7f542415c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:48.629 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.625+0000 7f5429b4a700 1 -- 172.21.15.192:0/1276228114 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f542415ddf0 msgr2=0x7f542415b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.629 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.625+0000 7f5429b4a700 1 --2- 172.21.15.192:0/1276228114 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f542415ddf0 0x7f542415b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.629 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.625+0000 7f5429b4a700 1 -- 172.21.15.192:0/1276228114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f542406ae40 msgr2=0x7f542415c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.629 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.625+0000 7f5429b4a700 1 --2- 172.21.15.192:0/1276228114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f542406ae40 0x7f542415c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.629 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.625+0000 7f5429b4a700 1 -- 172.21.15.192:0/1276228114 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5414009550 con 0x7f542415d7b0 2024-04-06T12:11:48.629 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.625+0000 7f542ab4c700 1 --2- 172.21.15.192:0/1276228114 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f542415ddf0 0x7f542415b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:11:48.629 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.625+0000 7f542a34b700 1 --2- 172.21.15.192:0/1276228114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f542406ae40 0x7f542415c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:11:48.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.625+0000 7f5429b4a700 1 --2- 172.21.15.192:0/1276228114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f542415d7b0 0x7f54241f2740 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5414016000 tx=0x7f5414004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:48.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.625+0000 7f541affd700 1 -- 172.21.15.192:0/1276228114 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5414004480 con 0x7f542415d7b0 2024-04-06T12:11:48.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.625+0000 7f541affd700 1 -- 172.21.15.192:0/1276228114 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541401f070 con 0x7f542415d7b0 2024-04-06T12:11:48.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.625+0000 7f541affd700 1 -- 172.21.15.192:0/1276228114 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54140038e0 con 0x7f542415d7b0 2024-04-06T12:11:48.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.625+0000 7f542c5af700 1 -- 172.21.15.192:0/1276228114 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5424145f00 con 0x7f542415d7b0 2024-04-06T12:11:48.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.625+0000 7f542c5af700 1 -- 172.21.15.192:0/1276228114 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54241464c0 con 0x7f542415d7b0 2024-04-06T12:11:48.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.625+0000 7f541affd700 1 -- 172.21.15.192:0/1276228114 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f5414021030 con 0x7f542415d7b0 2024-04-06T12:11:48.630 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.625+0000 7f541affd700 1 -- 172.21.15.192:0/1276228114 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f541402d440 con 0x7f542415d7b0 2024-04-06T12:11:48.631 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.626+0000 7f542c5af700 1 -- 172.21.15.192:0/1276228114 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5424067bb0 con 0x7f542415d7b0 2024-04-06T12:11:48.631 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.628+0000 7f541affd700 1 -- 172.21.15.192:0/1276228114 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7f5414015020 con 0x7f542415d7b0 2024-04-06T12:11:48.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.712+0000 7f542c5af700 1 -- 172.21.15.192:0/1276228114 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8dd707b1-5813-478c-9579-7072ee3070c1", "id": 5} v 0) v1 -- 0x7f5424074990 con 0x7f542415d7b0 2024-04-06T12:11:48.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.717+0000 7f541affd700 1 -- 172.21.15.192:0/1276228114 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8dd707b1-5813-478c-9579-7072ee3070c1", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f541401a450 con 0x7f542415d7b0 2024-04-06T12:11:48.715 INFO:teuthology.orchestra.run.smithi192.stdout:5 2024-04-06T12:11:48.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.718+0000 7f542c5af700 1 -- 172.21.15.192:0/1276228114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f542415d7b0 msgr2=0x7f54241f2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.718+0000 7f542c5af700 1 --2- 172.21.15.192:0/1276228114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f542415d7b0 0x7f54241f2740 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5414016000 tx=0x7f5414004750 comp rx=0 tx=0).stop 2024-04-06T12:11:48.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.718+0000 7f542c5af700 1 -- 172.21.15.192:0/1276228114 shutdown_connections 2024-04-06T12:11:48.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.718+0000 7f542c5af700 1 --2- 172.21.15.192:0/1276228114 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f542415ddf0 0x7f542415b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.718+0000 7f542c5af700 1 --2- 172.21.15.192:0/1276228114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f542406ae40 0x7f542415c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.718+0000 7f542c5af700 1 --2- 172.21.15.192:0/1276228114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f542415d7b0 0x7f54241f2740 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.718+0000 7f542c5af700 1 -- 172.21.15.192:0/1276228114 >> 172.21.15.192:0/1276228114 conn(0x7f54240d00b0 msgr2=0x7f54240adc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:48.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.718+0000 7f542c5af700 1 -- 172.21.15.192:0/1276228114 shutdown_connections 2024-04-06T12:11:48.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.718+0000 7f542c5af700 1 -- 172.21.15.192:0/1276228114 wait complete. 2024-04-06T12:11:48.728 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 8af1f576-d0e4-4f37-b782-6b2b084c89d3 6 2024-04-06T12:11:48.851 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.853+0000 7fb849ca5700 1 Processor -- start 2024-04-06T12:11:48.852 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.853+0000 7fb849ca5700 1 -- start start 2024-04-06T12:11:48.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.854+0000 7fb849ca5700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb8440cb870 0x7fb84415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:48.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.854+0000 7fb849ca5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8440cbd80 con 0x7fb8440d11b0 2024-04-06T12:11:48.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.854+0000 7fb849ca5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8440cef20 con 0x7fb8440cfa00 2024-04-06T12:11:48.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.854+0000 7fb849ca5700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8440ccdc0 con 0x7fb8440cb870 2024-04-06T12:11:48.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.854+0000 7fb842ffd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb8440cb870 0x7fb84415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:48.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.854+0000 7fb843fff700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fb8440cfa00 0x7fb8440cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:55526/0 (socket says 172.21.15.192:55526) 2024-04-06T12:11:48.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.854+0000 7fb842ffd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb8440cb870 0x7fb84415c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:58426/0 (socket says 172.21.15.192:58426) 2024-04-06T12:11:48.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.854+0000 7fb843fff700 1 -- 172.21.15.192:0/2396153272 learned_addr learned my addr 172.21.15.192:0/2396153272 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-04-06T12:11:48.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.855+0000 7fb842ffd700 1 -- 172.21.15.192:0/2396153272 >> v1:172.21.15.192:6789/0 conn(0x7fb8440cfa00 legacy=0x7fb8440cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.855+0000 7fb842ffd700 1 -- 172.21.15.192:0/2396153272 >> v1:172.21.15.189:6789/0 conn(0x7fb8440d11b0 legacy=0x7fb8440d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.855+0000 7fb8427fc700 1 -- 172.21.15.192:0/2396153272 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453609827 0 0) 0x7fb8440cef20 con 0x7fb8440cfa00 2024-04-06T12:11:48.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.855+0000 7fb842ffd700 1 -- 172.21.15.192:0/2396153272 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8440cf200 con 0x7fb8440cb870 2024-04-06T12:11:48.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.855+0000 7fb842ffd700 1 --2- 172.21.15.192:0/2396153272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8440cb870 0x7fb84415c280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb82c009840 tx=0x7fb82c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cf2230305e969ae server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:48.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.855+0000 7fb8427fc700 1 -- 172.21.15.192:0/2396153272 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c011830 con 0x7fb8440cb870 2024-04-06T12:11:48.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.855+0000 7fb8427fc700 1 -- 172.21.15.192:0/2396153272 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82c011990 con 0x7fb8440cb870 2024-04-06T12:11:48.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.855+0000 7fb8427fc700 1 -- 172.21.15.192:0/2396153272 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c011d20 con 0x7fb8440cb870 2024-04-06T12:11:48.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.856+0000 7fb849ca5700 1 -- 172.21.15.192:0/2396153272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8440cb870 msgr2=0x7fb84415c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.856+0000 7fb849ca5700 1 --2- 172.21.15.192:0/2396153272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8440cb870 0x7fb84415c280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb82c009840 tx=0x7fb82c009b50 comp rx=0 tx=0).stop 2024-04-06T12:11:48.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.856+0000 7fb849ca5700 1 -- 172.21.15.192:0/2396153272 shutdown_connections 2024-04-06T12:11:48.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.856+0000 7fb849ca5700 1 --2- 172.21.15.192:0/2396153272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8440cb870 0x7fb84415c280 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.856+0000 7fb849ca5700 1 -- 172.21.15.192:0/2396153272 >> 172.21.15.192:0/2396153272 conn(0x7fb8440b2db0 msgr2=0x7fb8440b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:48.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.856+0000 7fb849ca5700 1 -- 172.21.15.192:0/2396153272 shutdown_connections 2024-04-06T12:11:48.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.856+0000 7fb849ca5700 1 -- 172.21.15.192:0/2396153272 wait complete. 2024-04-06T12:11:48.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.857+0000 7fb849ca5700 1 Processor -- start 2024-04-06T12:11:48.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.857+0000 7fb849ca5700 1 -- start start 2024-04-06T12:11:48.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.858+0000 7fb849ca5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8440cb870 0x7fb844151270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:48.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.858+0000 7fb849ca5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8440cfa00 0x7fb8440b5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:48.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.858+0000 7fb849ca5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8440d11b0 0x7fb8440c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:48.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.858+0000 7fb849ca5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8441569c0 con 0x7fb8440d11b0 2024-04-06T12:11:48.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.858+0000 7fb849ca5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8440b67f0 con 0x7fb8440cfa00 2024-04-06T12:11:48.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.858+0000 7fb849ca5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb844156020 con 0x7fb8440cb870 2024-04-06T12:11:48.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.858+0000 7fb842ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8440cfa00 0x7fb8440b5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:48.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.858+0000 7fb8437fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8440cb870 0x7fb844151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:48.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.858+0000 7fb842ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8440cfa00 0x7fb8440b5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:58430/0 (socket says 172.21.15.192:58430) 2024-04-06T12:11:48.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.858+0000 7fb842ffd700 1 -- 172.21.15.192:0/2353259404 learned_addr learned my addr 172.21.15.192:0/2353259404 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-06T12:11:48.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.858+0000 7fb843fff700 1 --2- 172.21.15.192:0/2353259404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8440d11b0 0x7fb8440c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:48.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.858+0000 7fb842ffd700 1 -- 172.21.15.192:0/2353259404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8440cb870 msgr2=0x7fb844151270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.858+0000 7fb842ffd700 1 --2- 172.21.15.192:0/2353259404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8440cb870 0x7fb844151270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.858+0000 7fb842ffd700 1 -- 172.21.15.192:0/2353259404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8440d11b0 msgr2=0x7fb8440c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.858+0000 7fb842ffd700 1 --2- 172.21.15.192:0/2353259404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8440d11b0 0x7fb8440c82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.858+0000 7fb842ffd700 1 -- 172.21.15.192:0/2353259404 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c0094a0 con 0x7fb8440cfa00 2024-04-06T12:11:48.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.858+0000 7fb8437fe700 1 --2- 172.21.15.192:0/2353259404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8440cb870 0x7fb844151270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:11:48.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.859+0000 7fb842ffd700 1 --2- 172.21.15.192:0/2353259404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8440cfa00 0x7fb8440b5de0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0090b0 tx=0x7fb82c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:48.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.859+0000 7fb83bfff700 1 -- 172.21.15.192:0/2353259404 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c01a7e0 con 0x7fb8440cfa00 2024-04-06T12:11:48.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.859+0000 7fb83bfff700 1 -- 172.21.15.192:0/2353259404 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82c01a940 con 0x7fb8440cfa00 2024-04-06T12:11:48.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.859+0000 7fb83bfff700 1 -- 172.21.15.192:0/2353259404 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c01acb0 con 0x7fb8440cfa00 2024-04-06T12:11:48.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.859+0000 7fb849ca5700 1 -- 172.21.15.192:0/2353259404 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb844156300 con 0x7fb8440cfa00 2024-04-06T12:11:48.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.859+0000 7fb849ca5700 1 -- 172.21.15.192:0/2353259404 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8441517b0 con 0x7fb8440cfa00 2024-04-06T12:11:48.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.859+0000 7fb83bfff700 1 -- 172.21.15.192:0/2353259404 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fb82c022070 con 0x7fb8440cfa00 2024-04-06T12:11:48.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.859+0000 7fb83bfff700 1 -- 172.21.15.192:0/2353259404 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fb82c024d10 con 0x7fb8440cfa00 2024-04-06T12:11:48.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.860+0000 7fb849ca5700 1 -- 172.21.15.192:0/2353259404 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb844156b00 con 0x7fb8440cfa00 2024-04-06T12:11:48.861 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.862+0000 7fb83bfff700 1 -- 172.21.15.192:0/2353259404 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7fb844156b00 con 0x7fb8440cfa00 2024-04-06T12:11:48.925 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:48.927+0000 7f70d0691200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:11:48.945 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.947+0000 7fb849ca5700 1 -- 172.21.15.192:0/2353259404 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8af1f576-d0e4-4f37-b782-6b2b084c89d3", "id": 6} v 0) v1 -- 0x7fb84415c790 con 0x7fb8440cfa00 2024-04-06T12:11:48.952 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.954+0000 7fb83bfff700 1 -- 172.21.15.192:0/2353259404 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8af1f576-d0e4-4f37-b782-6b2b084c89d3", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb82c032060 con 0x7fb8440cfa00 2024-04-06T12:11:48.953 INFO:teuthology.orchestra.run.smithi192.stdout:6 2024-04-06T12:11:48.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.955+0000 7fb849ca5700 1 -- 172.21.15.192:0/2353259404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8440cfa00 msgr2=0x7fb8440b5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:48.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.955+0000 7fb849ca5700 1 --2- 172.21.15.192:0/2353259404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8440cfa00 0x7fb8440b5de0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0090b0 tx=0x7fb82c004750 comp rx=0 tx=0).stop 2024-04-06T12:11:48.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.955+0000 7fb849ca5700 1 -- 172.21.15.192:0/2353259404 shutdown_connections 2024-04-06T12:11:48.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.955+0000 7fb849ca5700 1 --2- 172.21.15.192:0/2353259404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8440d11b0 0x7fb8440c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.955+0000 7fb849ca5700 1 --2- 172.21.15.192:0/2353259404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8440cb870 0x7fb844151270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.955+0000 7fb849ca5700 1 --2- 172.21.15.192:0/2353259404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8440cfa00 0x7fb8440b5de0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:48.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.955+0000 7fb849ca5700 1 -- 172.21.15.192:0/2353259404 >> 172.21.15.192:0/2353259404 conn(0x7fb8440b2db0 msgr2=0x7fb844154d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:48.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.956+0000 7fb849ca5700 1 -- 172.21.15.192:0/2353259404 shutdown_connections 2024-04-06T12:11:48.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:48.956+0000 7fb849ca5700 1 -- 172.21.15.192:0/2353259404 wait complete. 2024-04-06T12:11:48.955 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:48.957+0000 7f3e06cc8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:11:48.966 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new b24dafce-ea55-4073-b8e6-23c2c08cbb23 7 2024-04-06T12:11:49.085 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:49.087+0000 7f70d0691200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:11:49.088 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.090+0000 7f1a8ce9c700 1 Processor -- start 2024-04-06T12:11:49.088 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.090+0000 7f1a8ce9c700 1 -- start start 2024-04-06T12:11:49.091 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.091+0000 7f1a8ce9c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a880aec70 0x7f1a880cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:49.091 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.091+0000 7f1a8ce9c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1a880cdae0 0x7f1a880ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:49.091 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.091+0000 7f1a8ce9c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8815a830 con 0x7f1a880ce8b0 2024-04-06T12:11:49.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.091+0000 7f1a8ce9c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a880cd920 con 0x7f1a880cdae0 2024-04-06T12:11:49.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.091+0000 7f1a8ce9c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a880ced20 con 0x7f1a880aec70 2024-04-06T12:11:49.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.091+0000 7f1a8659c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a880aec70 0x7f1a880cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:49.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.091+0000 7f1a85d9b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1a880cdae0 0x7f1a880ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:49.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.091+0000 7f1a8659c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a880aec70 0x7f1a880cd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:45134/0 (socket says 172.21.15.192:45134) 2024-04-06T12:11:49.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.091+0000 7f1a85d9b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1a880cdae0 0x7f1a880ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:58444/0 (socket says 172.21.15.192:58444) 2024-04-06T12:11:49.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.091+0000 7f1a8659c700 1 -- 172.21.15.192:0/1996275594 learned_addr learned my addr 172.21.15.192:0/1996275594 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-06T12:11:49.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.092+0000 7f1a8659c700 1 -- 172.21.15.192:0/1996275594 >> v2:172.21.15.192:3300/0 conn(0x7f1a880cdae0 msgr2=0x7f1a880ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:11:49.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.092+0000 7f1a8659c700 1 --2- 172.21.15.192:0/1996275594 >> v2:172.21.15.192:3300/0 conn(0x7f1a880cdae0 0x7f1a880ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:49.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.092+0000 7f1a8659c700 1 -- 172.21.15.192:0/1996275594 >> v1:172.21.15.189:6789/0 conn(0x7f1a880ce8b0 legacy=0x7f1a8815df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:49.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.092+0000 7f1a8659c700 1 -- 172.21.15.192:0/1996275594 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a880cc050 con 0x7f1a880aec70 2024-04-06T12:11:49.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.092+0000 7f1a8659c700 1 --2- 172.21.15.192:0/1996275594 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a880aec70 0x7f1a880cd3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1a7800bb90 tx=0x7f1a7800bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc84d96719a9ef0c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:49.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.092+0000 7f1a8559a700 1 -- 172.21.15.192:0/1996275594 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7800dd30 con 0x7f1a880aec70 2024-04-06T12:11:49.093 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.092+0000 7f1a8559a700 1 -- 172.21.15.192:0/1996275594 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a7800de90 con 0x7f1a880aec70 2024-04-06T12:11:49.093 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.092+0000 7f1a8559a700 1 -- 172.21.15.192:0/1996275594 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a78010690 con 0x7f1a880aec70 2024-04-06T12:11:49.093 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.093+0000 7f1a8ce9c700 1 -- 172.21.15.192:0/1996275594 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a880aec70 msgr2=0x7f1a880cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:49.093 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.093+0000 7f1a8ce9c700 1 --2- 172.21.15.192:0/1996275594 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a880aec70 0x7f1a880cd3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1a7800bb90 tx=0x7f1a7800bea0 comp rx=0 tx=0).stop 2024-04-06T12:11:49.093 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.093+0000 7f1a8ce9c700 1 -- 172.21.15.192:0/1996275594 shutdown_connections 2024-04-06T12:11:49.093 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.093+0000 7f1a8ce9c700 1 --2- 172.21.15.192:0/1996275594 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a880aec70 0x7f1a880cd3e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:49.093 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.093+0000 7f1a8ce9c700 1 --2- 172.21.15.192:0/1996275594 >> v2:172.21.15.192:3300/0 conn(0x7f1a880cdae0 0x7f1a880ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:49.093 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.093+0000 7f1a8ce9c700 1 -- 172.21.15.192:0/1996275594 >> 172.21.15.192:0/1996275594 conn(0x7f1a880cc290 msgr2=0x7f1a880cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:49.093 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.093+0000 7f1a8ce9c700 1 -- 172.21.15.192:0/1996275594 shutdown_connections 2024-04-06T12:11:49.093 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.093+0000 7f1a8ce9c700 1 -- 172.21.15.192:0/1996275594 wait complete. 2024-04-06T12:11:49.096 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.094+0000 7f1a8ce9c700 1 Processor -- start 2024-04-06T12:11:49.096 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.094+0000 7f1a8ce9c700 1 -- start start 2024-04-06T12:11:49.096 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.095+0000 7f1a8ce9c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a880aec70 0x7f1a880b2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:49.096 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.095+0000 7f1a8ce9c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a880cdae0 0x7f1a881f2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:49.097 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.095+0000 7f1a8ce9c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a880ce8b0 0x7f1a880b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:49.097 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.095+0000 7f1a8ce9c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a880ac960 con 0x7f1a880ce8b0 2024-04-06T12:11:49.097 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.095+0000 7f1a8ce9c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a880b2c60 con 0x7f1a880aec70 2024-04-06T12:11:49.097 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.095+0000 7f1a8ce9c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a880b2f40 con 0x7f1a880cdae0 2024-04-06T12:11:49.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.095+0000 7f1a8659c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a880aec70 0x7f1a880b2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:49.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.095+0000 7f1a8659c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a880aec70 0x7f1a880b2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:58452/0 (socket says 172.21.15.192:58452) 2024-04-06T12:11:49.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.095+0000 7f1a8659c700 1 -- 172.21.15.192:0/177796184 learned_addr learned my addr 172.21.15.192:0/177796184 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-06T12:11:49.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.095+0000 7f1a85d9b700 1 --2- 172.21.15.192:0/177796184 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a880cdae0 0x7f1a881f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:49.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.095+0000 7f1a86d9d700 1 --2- 172.21.15.192:0/177796184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a880ce8b0 0x7f1a880b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:49.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.095+0000 7f1a8659c700 1 -- 172.21.15.192:0/177796184 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a880cdae0 msgr2=0x7f1a881f2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:49.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.095+0000 7f1a8659c700 1 --2- 172.21.15.192:0/177796184 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a880cdae0 0x7f1a881f2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:49.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.095+0000 7f1a8659c700 1 -- 172.21.15.192:0/177796184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a880ce8b0 msgr2=0x7f1a880b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:49.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.095+0000 7f1a8659c700 1 --2- 172.21.15.192:0/177796184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a880ce8b0 0x7f1a880b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:49.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.095+0000 7f1a8659c700 1 -- 172.21.15.192:0/177796184 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7800b7f0 con 0x7f1a880aec70 2024-04-06T12:11:49.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.095+0000 7f1a86d9d700 1 --2- 172.21.15.192:0/177796184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a880ce8b0 0x7f1a880b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:11:49.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.095+0000 7f1a8659c700 1 --2- 172.21.15.192:0/177796184 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a880aec70 0x7f1a880b2410 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1a78000c00 tx=0x7f1a78004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:49.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.096+0000 7f1a76ffd700 1 -- 172.21.15.192:0/177796184 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a78003bf0 con 0x7f1a880aec70 2024-04-06T12:11:49.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.096+0000 7f1a76ffd700 1 -- 172.21.15.192:0/177796184 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a78003d50 con 0x7f1a880aec70 2024-04-06T12:11:49.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.096+0000 7f1a8ce9c700 1 -- 172.21.15.192:0/177796184 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a880b3220 con 0x7f1a880aec70 2024-04-06T12:11:49.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.096+0000 7f1a76ffd700 1 -- 172.21.15.192:0/177796184 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a780106b0 con 0x7f1a880aec70 2024-04-06T12:11:49.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.096+0000 7f1a8ce9c700 1 -- 172.21.15.192:0/177796184 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a880b2950 con 0x7f1a880aec70 2024-04-06T12:11:49.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.096+0000 7f1a76ffd700 1 -- 172.21.15.192:0/177796184 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f1a78010b10 con 0x7f1a880aec70 2024-04-06T12:11:49.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.096+0000 7f1a76ffd700 1 -- 172.21.15.192:0/177796184 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f1a78019a40 con 0x7f1a880aec70 2024-04-06T12:11:49.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.097+0000 7f1a8ce9c700 1 -- 172.21.15.192:0/177796184 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a88074990 con 0x7f1a880aec70 2024-04-06T12:11:49.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.099+0000 7f1a76ffd700 1 -- 172.21.15.192:0/177796184 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7f1a78019ca0 con 0x7f1a880aec70 2024-04-06T12:11:49.127 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:49.130+0000 7f3e06cc8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:11:49.164 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:49.167+0000 7f70d0691200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:11:49.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.183+0000 7f1a8ce9c700 1 -- 172.21.15.192:0/177796184 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b24dafce-ea55-4073-b8e6-23c2c08cbb23", "id": 7} v 0) v1 -- 0x7f1a880b5790 con 0x7f1a880aec70 2024-04-06T12:11:49.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.191+0000 7f1a76ffd700 1 -- 172.21.15.192:0/177796184 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b24dafce-ea55-4073-b8e6-23c2c08cbb23", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1a78019e40 con 0x7f1a880aec70 2024-04-06T12:11:49.189 INFO:teuthology.orchestra.run.smithi192.stdout:7 2024-04-06T12:11:49.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.192+0000 7f1a8ce9c700 1 -- 172.21.15.192:0/177796184 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a880aec70 msgr2=0x7f1a880b2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:49.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.192+0000 7f1a8ce9c700 1 --2- 172.21.15.192:0/177796184 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a880aec70 0x7f1a880b2410 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1a78000c00 tx=0x7f1a78004750 comp rx=0 tx=0).stop 2024-04-06T12:11:49.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.192+0000 7f1a8ce9c700 1 -- 172.21.15.192:0/177796184 shutdown_connections 2024-04-06T12:11:49.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.192+0000 7f1a8ce9c700 1 --2- 172.21.15.192:0/177796184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a880ce8b0 0x7f1a880b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:49.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.192+0000 7f1a8ce9c700 1 --2- 172.21.15.192:0/177796184 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a880aec70 0x7f1a880b2410 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:49.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.192+0000 7f1a8ce9c700 1 --2- 172.21.15.192:0/177796184 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a880cdae0 0x7f1a881f2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:49.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.192+0000 7f1a8ce9c700 1 -- 172.21.15.192:0/177796184 >> 172.21.15.192:0/177796184 conn(0x7f1a880cc290 msgr2=0x7f1a880afea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:49.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.192+0000 7f1a8ce9c700 1 -- 172.21.15.192:0/177796184 shutdown_connections 2024-04-06T12:11:49.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T12:11:49.192+0000 7f1a8ce9c700 1 -- 172.21.15.192:0/177796184 wait complete. 2024-04-06T12:11:49.202 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-06T12:11:49.202 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-06T12:11:49.204 INFO:tasks.ceph.osd.0:Started 2024-04-06T12:11:49.204 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-06T12:11:49.205 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-06T12:11:49.206 INFO:tasks.ceph.osd.1:Started 2024-04-06T12:11:49.207 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-06T12:11:49.207 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-06T12:11:49.209 INFO:tasks.ceph.osd.2:Started 2024-04-06T12:11:49.209 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-06T12:11:49.209 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-06T12:11:49.210 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:49.212+0000 7f3e06cc8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:11:49.211 INFO:tasks.ceph.osd.3:Started 2024-04-06T12:11:49.211 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-06T12:11:49.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-06T12:11:49.214 INFO:tasks.ceph.osd.4:Started 2024-04-06T12:11:49.214 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-06T12:11:49.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-06T12:11:49.216 INFO:tasks.ceph.osd.5:Started 2024-04-06T12:11:49.216 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-06T12:11:49.216 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-06T12:11:49.218 INFO:tasks.ceph.osd.6:Started 2024-04-06T12:11:49.218 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-06T12:11:49.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-06T12:11:49.220 INFO:tasks.ceph.osd.7:Started 2024-04-06T12:11:49.220 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:11:49.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.386+0000 7fae08ba0700 1 Processor -- start 2024-04-06T12:11:49.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.386+0000 7fae08ba0700 1 -- start start 2024-04-06T12:11:49.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.387+0000 7fae08ba0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fae0406ae40 0x7fae040b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:49.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.387+0000 7fae08ba0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae0415ddf0 0x7fae0415e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:49.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.387+0000 7fae08ba0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae040b3990 con 0x7fae0415d7b0 2024-04-06T12:11:49.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.387+0000 7fae08ba0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae040b3060 con 0x7fae0406ae40 2024-04-06T12:11:49.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.387+0000 7fae08ba0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae040b3340 con 0x7fae0415ddf0 2024-04-06T12:11:49.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.387+0000 7fae01d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fae0406ae40 0x7fae040b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:49.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.387+0000 7fae01d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fae0406ae40 0x7fae040b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59902/0 (socket says 172.21.15.189:59902) 2024-04-06T12:11:49.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.387+0000 7fae01d9b700 1 -- 172.21.15.189:0/3069865805 learned_addr learned my addr 172.21.15.189:0/3069865805 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:11:49.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.387+0000 7fae02d9d700 1 --2- 172.21.15.189:0/3069865805 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae0415ddf0 0x7fae0415e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:49.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.387+0000 7fae01d9b700 1 -- 172.21.15.189:0/3069865805 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae0415ddf0 msgr2=0x7fae0415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:11:49.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.387+0000 7fae01d9b700 1 --2- 172.21.15.189:0/3069865805 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae0415ddf0 0x7fae0415e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:49.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.387+0000 7fae01d9b700 1 -- 172.21.15.189:0/3069865805 >> v1:172.21.15.192:6789/0 conn(0x7fae0415d7b0 legacy=0x7fae0406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:49.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.387+0000 7fae01d9b700 1 -- 172.21.15.189:0/3069865805 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae040b3620 con 0x7fae0406ae40 2024-04-06T12:11:49.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.387+0000 7fae01d9b700 1 --2- 172.21.15.189:0/3069865805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae0406ae40 0x7fae040b3e20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fadf80098f0 tx=0x7fadf8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=835804ee81f9cab6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:49.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.388+0000 7fae0159a700 1 -- 172.21.15.189:0/3069865805 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf8011830 con 0x7fae0406ae40 2024-04-06T12:11:49.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.388+0000 7fae0159a700 1 -- 172.21.15.189:0/3069865805 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf8011990 con 0x7fae0406ae40 2024-04-06T12:11:49.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.388+0000 7fae0159a700 1 -- 172.21.15.189:0/3069865805 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf8011d20 con 0x7fae0406ae40 2024-04-06T12:11:49.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.388+0000 7fae08ba0700 1 -- 172.21.15.189:0/3069865805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae0406ae40 msgr2=0x7fae040b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:49.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.388+0000 7fae08ba0700 1 --2- 172.21.15.189:0/3069865805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae0406ae40 0x7fae040b3e20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fadf80098f0 tx=0x7fadf8009c00 comp rx=0 tx=0).stop 2024-04-06T12:11:49.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.388+0000 7fae08ba0700 1 -- 172.21.15.189:0/3069865805 shutdown_connections 2024-04-06T12:11:49.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.388+0000 7fae08ba0700 1 --2- 172.21.15.189:0/3069865805 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae0415ddf0 0x7fae0415e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:49.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.388+0000 7fae08ba0700 1 --2- 172.21.15.189:0/3069865805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae0406ae40 0x7fae040b3e20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:49.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.388+0000 7fae08ba0700 1 -- 172.21.15.189:0/3069865805 >> 172.21.15.189:0/3069865805 conn(0x7fae040d00b0 msgr2=0x7fae040d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:49.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.388+0000 7fae08ba0700 1 -- 172.21.15.189:0/3069865805 shutdown_connections 2024-04-06T12:11:49.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.388+0000 7fae08ba0700 1 -- 172.21.15.189:0/3069865805 wait complete. 2024-04-06T12:11:49.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.390+0000 7fae08ba0700 1 Processor -- start 2024-04-06T12:11:49.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.390+0000 7fae08ba0700 1 -- start start 2024-04-06T12:11:49.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.391+0000 7fae08ba0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae0406ae40 0x7fae0414bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:49.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.391+0000 7fae08ba0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae0415d7b0 0x7fae040c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:49.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.391+0000 7fae08ba0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae0415ddf0 0x7fae041f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:49.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.391+0000 7fae08ba0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae040c61d0 con 0x7fae0415d7b0 2024-04-06T12:11:49.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.391+0000 7fae08ba0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0414c930 con 0x7fae0406ae40 2024-04-06T12:11:49.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.391+0000 7fae08ba0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae040cf580 con 0x7fae0415ddf0 2024-04-06T12:11:49.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.391+0000 7fae01d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae0415d7b0 0x7fae040c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:49.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.391+0000 7fae01d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae0415d7b0 0x7fae040c4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59904/0 (socket says 172.21.15.189:59904) 2024-04-06T12:11:49.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.391+0000 7fae0259c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae0406ae40 0x7fae0414bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:49.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.391+0000 7fae01d9b700 1 -- 172.21.15.189:0/3503961415 learned_addr learned my addr 172.21.15.189:0/3503961415 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:11:49.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.391+0000 7fae02d9d700 1 --2- 172.21.15.189:0/3503961415 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae0415ddf0 0x7fae041f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:49.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.391+0000 7fae01d9b700 1 -- 172.21.15.189:0/3503961415 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae0415ddf0 msgr2=0x7fae041f5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:49.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.391+0000 7fae01d9b700 1 --2- 172.21.15.189:0/3503961415 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae0415ddf0 0x7fae041f5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:49.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.391+0000 7fae01d9b700 1 -- 172.21.15.189:0/3503961415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae0406ae40 msgr2=0x7fae0414bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:49.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.391+0000 7fae01d9b700 1 --2- 172.21.15.189:0/3503961415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae0406ae40 0x7fae0414bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:49.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.391+0000 7fae01d9b700 1 -- 172.21.15.189:0/3503961415 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf8009550 con 0x7fae0415d7b0 2024-04-06T12:11:49.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.392+0000 7fae01d9b700 1 --2- 172.21.15.189:0/3503961415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae0415d7b0 0x7fae040c4f30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fadf8000c00 tx=0x7fadf8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:49.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.392+0000 7fadf2ffd700 1 -- 172.21.15.189:0/3503961415 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf8004400 con 0x7fae0415d7b0 2024-04-06T12:11:49.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.392+0000 7fadf2ffd700 1 -- 172.21.15.189:0/3503961415 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf801f070 con 0x7fae0415d7b0 2024-04-06T12:11:49.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.392+0000 7fae08ba0700 1 -- 172.21.15.189:0/3503961415 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae040ccc90 con 0x7fae0415d7b0 2024-04-06T12:11:49.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.392+0000 7fadf2ffd700 1 -- 172.21.15.189:0/3503961415 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf80038e0 con 0x7fae0415d7b0 2024-04-06T12:11:49.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.392+0000 7fae08ba0700 1 -- 172.21.15.189:0/3503961415 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae040cd020 con 0x7fae0415d7b0 2024-04-06T12:11:49.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.392+0000 7fadf2ffd700 1 -- 172.21.15.189:0/3503961415 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fadf8021030 con 0x7fae0415d7b0 2024-04-06T12:11:49.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.392+0000 7fadf2ffd700 1 -- 172.21.15.189:0/3503961415 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fadf8024b00 con 0x7fae0415d7b0 2024-04-06T12:11:49.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.393+0000 7fae08ba0700 1 -- 172.21.15.189:0/3503961415 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae04067bb0 con 0x7fae0415d7b0 2024-04-06T12:11:49.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.395+0000 7fadf2ffd700 1 -- 172.21.15.189:0/3503961415 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7fadf8015020 con 0x7fae0415d7b0 2024-04-06T12:11:49.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.466+0000 7fae08ba0700 1 -- 172.21.15.189:0/3503961415 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae04074990 con 0x7fae0415d7b0 2024-04-06T12:11:49.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.466+0000 7fadf2ffd700 1 -- 172.21.15.189:0/3503961415 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fadf801a450 con 0x7fae0415d7b0 2024-04-06T12:11:49.464 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:49.465 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":10,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:11:49.184725+0000","last_up_change":"0.000000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:11:49.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.467+0000 7fae08ba0700 1 -- 172.21.15.189:0/3503961415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae0415d7b0 msgr2=0x7fae040c4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:49.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.467+0000 7fae08ba0700 1 --2- 172.21.15.189:0/3503961415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae0415d7b0 0x7fae040c4f30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fadf8000c00 tx=0x7fadf8004750 comp rx=0 tx=0).stop 2024-04-06T12:11:49.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.467+0000 7fae08ba0700 1 -- 172.21.15.189:0/3503961415 shutdown_connections 2024-04-06T12:11:49.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.467+0000 7fae08ba0700 1 --2- 172.21.15.189:0/3503961415 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae0415ddf0 0x7fae041f5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:49.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.467+0000 7fae08ba0700 1 --2- 172.21.15.189:0/3503961415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae0406ae40 0x7fae0414bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:49.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.467+0000 7fae08ba0700 1 --2- 172.21.15.189:0/3503961415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae0415d7b0 0x7fae040c4f30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:49.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.467+0000 7fae08ba0700 1 -- 172.21.15.189:0/3503961415 >> 172.21.15.189:0/3503961415 conn(0x7fae040d00b0 msgr2=0x7fae040ae130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:49.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.467+0000 7fae08ba0700 1 -- 172.21.15.189:0/3503961415 shutdown_connections 2024-04-06T12:11:49.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:49.467+0000 7fae08ba0700 1 -- 172.21.15.189:0/3503961415 wait complete. 2024-04-06T12:11:49.479 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-06T12:11:49.479 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-06T12:11:49.817 INFO:tasks.ceph.osd.1.smithi189.stderr:2024-04-06T12:11:49.819+0000 7f7a56211640 -1 Falling back to public interface 2024-04-06T12:11:49.825 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-06T12:11:49.827+0000 7fe3243db640 -1 Falling back to public interface 2024-04-06T12:11:49.826 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-06T12:11:49.827+0000 7fab15291640 -1 Falling back to public interface 2024-04-06T12:11:49.826 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-06T12:11:49.827+0000 7fe31b796700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2024-04-06T12:11:49.826 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-06T12:11:49.827+0000 7fe31b796700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6801/0: (98) Address already in use 2024-04-06T12:11:49.834 INFO:tasks.ceph.osd.0.smithi189.stderr:2024-04-06T12:11:49.836+0000 7f2957437640 -1 Falling back to public interface 2024-04-06T12:11:49.842 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-06T12:11:49.844+0000 7ffb8faa4640 -1 Falling back to public interface 2024-04-06T12:11:49.843 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-06T12:11:49.845+0000 7fd466f66640 -1 Falling back to public interface 2024-04-06T12:11:49.844 INFO:tasks.ceph.osd.2.smithi189.stderr:2024-04-06T12:11:49.846+0000 7ff7f030c640 -1 Falling back to public interface 2024-04-06T12:11:49.856 INFO:tasks.ceph.osd.3.smithi189.stderr:2024-04-06T12:11:49.858+0000 7f9c4a8dc640 -1 Falling back to public interface 2024-04-06T12:11:49.907 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:49.909+0000 7f70d0691200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:11:49.936 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:49.937+0000 7f3e06cc8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:11:49.990 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:49.992+0000 7f70d0691200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:11:50.019 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:50.021+0000 7f3e06cc8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:11:50.125 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:50.127+0000 7f70d0691200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:11:50.152 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:50.153+0000 7f3e06cc8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:11:50.284 DEBUG:teuthology.orchestra.run.smithi189:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-06T12:11:50.399 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.401+0000 7fe1669c7700 1 Processor -- start 2024-04-06T12:11:50.399 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.401+0000 7fe1669c7700 1 -- start start 2024-04-06T12:11:50.402 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.402+0000 7fe1669c7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe1600b4280 0x7fe16006dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:50.402 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.402+0000 7fe1669c7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe16015a6b0 0x7fe1600ade10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:50.402 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.402+0000 7fe1669c7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1600b5e20 con 0x7fe16015a2d0 2024-04-06T12:11:50.402 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.402+0000 7fe1669c7700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1600ad9c0 con 0x7fe1600b4280 2024-04-06T12:11:50.402 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.402+0000 7fe1669c7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1600d0a90 con 0x7fe16015a6b0 2024-04-06T12:11:50.402 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.402+0000 7fe15f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe1600b4280 0x7fe16006dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:50.402 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.402+0000 7fe15f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe1600b4280 0x7fe16006dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59916/0 (socket says 172.21.15.189:59916) 2024-04-06T12:11:50.402 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.402+0000 7fe15ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe16015a2d0 0x7fe1600b3b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35688/0 (socket says 172.21.15.189:35688) 2024-04-06T12:11:50.402 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.402+0000 7fe15f7fe700 1 -- 172.21.15.189:0/192982395 learned_addr learned my addr 172.21.15.189:0/192982395 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:11:50.402 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.402+0000 7fe164f64700 1 --2- 172.21.15.189:0/192982395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe16015a6b0 0x7fe1600ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:50.402 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.402+0000 7fe15f7fe700 1 -- 172.21.15.189:0/192982395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe16015a6b0 msgr2=0x7fe1600ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:11:50.403 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.402+0000 7fe15f7fe700 1 --2- 172.21.15.189:0/192982395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe16015a6b0 0x7fe1600ade10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:50.403 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.402+0000 7fe15f7fe700 1 -- 172.21.15.189:0/192982395 >> v1:172.21.15.189:6789/0 conn(0x7fe16015a2d0 legacy=0x7fe1600b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:50.403 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.402+0000 7fe15f7fe700 1 -- 172.21.15.189:0/192982395 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1600b33a0 con 0x7fe1600b4280 2024-04-06T12:11:50.403 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.402+0000 7fe15effd700 1 -- 172.21.15.189:0/192982395 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121343964 0 0) 0x7fe1600b5e20 con 0x7fe16015a2d0 2024-04-06T12:11:50.403 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.403+0000 7fe15f7fe700 1 --2- 172.21.15.189:0/192982395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1600b4280 0x7fe16006dac0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe15400bb90 tx=0x7fe15400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac5dd6f5d825d6f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:50.403 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.403+0000 7fe15effd700 1 -- 172.21.15.189:0/192982395 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15400dd30 con 0x7fe1600b4280 2024-04-06T12:11:50.403 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.403+0000 7fe15effd700 1 -- 172.21.15.189:0/192982395 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15400de90 con 0x7fe1600b4280 2024-04-06T12:11:50.403 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.403+0000 7fe15effd700 1 -- 172.21.15.189:0/192982395 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154010690 con 0x7fe1600b4280 2024-04-06T12:11:50.403 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.403+0000 7fe1669c7700 1 -- 172.21.15.189:0/192982395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1600b4280 msgr2=0x7fe16006dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:50.403 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.403+0000 7fe1669c7700 1 --2- 172.21.15.189:0/192982395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1600b4280 0x7fe16006dac0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe15400bb90 tx=0x7fe15400bea0 comp rx=0 tx=0).stop 2024-04-06T12:11:50.403 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.403+0000 7fe1669c7700 1 -- 172.21.15.189:0/192982395 shutdown_connections 2024-04-06T12:11:50.403 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.403+0000 7fe1669c7700 1 --2- 172.21.15.189:0/192982395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe16015a6b0 0x7fe1600ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:50.403 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.403+0000 7fe1669c7700 1 --2- 172.21.15.189:0/192982395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1600b4280 0x7fe16006dac0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:50.404 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.403+0000 7fe1669c7700 1 -- 172.21.15.189:0/192982395 >> 172.21.15.189:0/192982395 conn(0x7fe160099a50 msgr2=0x7fe1600b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:50.404 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.403+0000 7fe1669c7700 1 -- 172.21.15.189:0/192982395 shutdown_connections 2024-04-06T12:11:50.404 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.403+0000 7fe1669c7700 1 -- 172.21.15.189:0/192982395 wait complete. 2024-04-06T12:11:50.406 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.405+0000 7fe1669c7700 1 Processor -- start 2024-04-06T12:11:50.406 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.405+0000 7fe1669c7700 1 -- start start 2024-04-06T12:11:50.406 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.405+0000 7fe1669c7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1600b4280 0x7fe1601f21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:50.406 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.405+0000 7fe1669c7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe16015a2d0 0x7fe160156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:50.406 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe1669c7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe16015a6b0 0x7fe1601ef370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:50.407 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe1669c7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1601eebd0 con 0x7fe16015a2d0 2024-04-06T12:11:50.407 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe1669c7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1601f2700 con 0x7fe16015a6b0 2024-04-06T12:11:50.407 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe1669c7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1601f29a0 con 0x7fe1600b4280 2024-04-06T12:11:50.407 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe15f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe16015a2d0 0x7fe160156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:50.407 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe15f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe16015a2d0 0x7fe160156480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59922/0 (socket says 172.21.15.189:59922) 2024-04-06T12:11:50.407 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe15f7fe700 1 -- 172.21.15.189:0/4238955684 learned_addr learned my addr 172.21.15.189:0/4238955684 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:11:50.407 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe15ffff700 1 --2- 172.21.15.189:0/4238955684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1600b4280 0x7fe1601f21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:50.407 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe164f64700 1 --2- 172.21.15.189:0/4238955684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe16015a6b0 0x7fe1601ef370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:50.407 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe15f7fe700 1 -- 172.21.15.189:0/4238955684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1600b4280 msgr2=0x7fe1601f21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:50.407 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe15f7fe700 1 --2- 172.21.15.189:0/4238955684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1600b4280 0x7fe1601f21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:50.407 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe15f7fe700 1 -- 172.21.15.189:0/4238955684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe16015a6b0 msgr2=0x7fe1601ef370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:50.408 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe15f7fe700 1 --2- 172.21.15.189:0/4238955684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe16015a6b0 0x7fe1601ef370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:50.408 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe15f7fe700 1 -- 172.21.15.189:0/4238955684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15400b7f0 con 0x7fe16015a2d0 2024-04-06T12:11:50.408 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe15f7fe700 1 --2- 172.21.15.189:0/4238955684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe16015a2d0 0x7fe160156480 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe154000c00 tx=0x7fe154004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:50.408 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe15cff9700 1 -- 172.21.15.189:0/4238955684 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15401f070 con 0x7fe16015a2d0 2024-04-06T12:11:50.408 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe15cff9700 1 -- 172.21.15.189:0/4238955684 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe154003970 con 0x7fe16015a2d0 2024-04-06T12:11:50.408 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe1669c7700 1 -- 172.21.15.189:0/4238955684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1601f2c80 con 0x7fe16015a2d0 2024-04-06T12:11:50.408 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe15cff9700 1 -- 172.21.15.189:0/4238955684 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154003d20 con 0x7fe16015a2d0 2024-04-06T12:11:50.408 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe1669c7700 1 -- 172.21.15.189:0/4238955684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1601eee50 con 0x7fe16015a2d0 2024-04-06T12:11:50.408 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.406+0000 7fe15cff9700 1 -- 172.21.15.189:0/4238955684 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fe154024470 con 0x7fe16015a2d0 2024-04-06T12:11:50.408 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.407+0000 7fe15cff9700 1 -- 172.21.15.189:0/4238955684 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fe154010b40 con 0x7fe16015a2d0 2024-04-06T12:11:50.408 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.407+0000 7fe1669c7700 1 -- 172.21.15.189:0/4238955684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe160067bb0 con 0x7fe16015a2d0 2024-04-06T12:11:50.409 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.409+0000 7fe15cff9700 1 -- 172.21.15.189:0/4238955684 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7fe154029070 con 0x7fe16015a2d0 2024-04-06T12:11:50.477 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.479+0000 7fe1669c7700 1 -- 172.21.15.189:0/4238955684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe160074930 con 0x7fe16015a2d0 2024-04-06T12:11:50.478 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.480+0000 7fe15cff9700 1 -- 172.21.15.189:0/4238955684 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fe154010470 con 0x7fe16015a2d0 2024-04-06T12:11:50.478 INFO:teuthology.misc.health.smithi189.stdout: 2024-04-06T12:11:50.478 INFO:teuthology.misc.health.smithi189.stdout:{"epoch":10,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:11:49.184725+0000","last_up_change":"0.000000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:11:50.479 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.481+0000 7fe1669c7700 1 -- 172.21.15.189:0/4238955684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe16015a2d0 msgr2=0x7fe160156480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:50.479 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.481+0000 7fe1669c7700 1 --2- 172.21.15.189:0/4238955684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe16015a2d0 0x7fe160156480 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe154000c00 tx=0x7fe154004750 comp rx=0 tx=0).stop 2024-04-06T12:11:50.480 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.481+0000 7fe1669c7700 1 -- 172.21.15.189:0/4238955684 shutdown_connections 2024-04-06T12:11:50.480 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.481+0000 7fe1669c7700 1 --2- 172.21.15.189:0/4238955684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe16015a6b0 0x7fe1601ef370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:50.480 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.481+0000 7fe1669c7700 1 --2- 172.21.15.189:0/4238955684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1600b4280 0x7fe1601f21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:50.480 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.481+0000 7fe1669c7700 1 --2- 172.21.15.189:0/4238955684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe16015a2d0 0x7fe160156480 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:50.480 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.481+0000 7fe1669c7700 1 -- 172.21.15.189:0/4238955684 >> 172.21.15.189:0/4238955684 conn(0x7fe160099a50 msgr2=0x7fe1600b4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:50.480 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.481+0000 7fe1669c7700 1 -- 172.21.15.189:0/4238955684 shutdown_connections 2024-04-06T12:11:50.480 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:50.481+0000 7fe1669c7700 1 -- 172.21.15.189:0/4238955684 wait complete. 2024-04-06T12:11:50.490 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-06T12:11:50.690 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:50.692+0000 7f70d0691200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:11:50.705 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:50.708+0000 7f3e06cc8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:11:50.844 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:50.846+0000 7f70d0691200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:11:50.900 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:50.902+0000 7f3e06cc8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:11:51.010 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:51.012+0000 7f70d0691200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:11:51.069 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:51.070+0000 7f3e06cc8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:11:51.833 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:51.836+0000 7f70d0691200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:11:51.888 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:51.890+0000 7f3e06cc8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:11:52.073 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:52.075+0000 7f70d0691200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:11:52.130 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:52.132+0000 7f3e06cc8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:11:52.146 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:52.148+0000 7f70d0691200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:11:52.204 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:52.206+0000 7f3e06cc8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:11:52.538 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:52.540+0000 7f70d0691200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:11:52.615 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:11:52.617+0000 7f3e06cc8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:11:53.331 INFO:tasks.ceph.osd.1.smithi189.stderr:2024-04-06T12:11:53.333+0000 7f7a56211640 -1 osd.1 0 log_to_monitors true 2024-04-06T12:11:53.350 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-06T12:11:53.352+0000 7fe3243db640 -1 osd.7 0 log_to_monitors true 2024-04-06T12:11:53.357 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-06T12:11:53.359+0000 7fab15291640 -1 osd.5 0 log_to_monitors true 2024-04-06T12:11:53.358 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-06T12:11:53.360+0000 7ffb8faa4640 -1 osd.4 0 log_to_monitors true 2024-04-06T12:11:53.361 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-06T12:11:53.363+0000 7fd466f66640 -1 osd.6 0 log_to_monitors true 2024-04-06T12:11:53.365 INFO:tasks.ceph.osd.0.smithi189.stderr:2024-04-06T12:11:53.367+0000 7f2957437640 -1 osd.0 0 log_to_monitors true 2024-04-06T12:11:53.396 INFO:tasks.ceph.osd.2.smithi189.stderr:2024-04-06T12:11:53.398+0000 7ff7f030c640 -1 osd.2 0 log_to_monitors true 2024-04-06T12:11:53.405 INFO:tasks.ceph.osd.3.smithi189.stderr:2024-04-06T12:11:53.407+0000 7f9c4a8dc640 -1 osd.3 0 log_to_monitors true 2024-04-06T12:11:53.564 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:53.566+0000 7f708b90a700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-06T12:11:53.567 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:11:53.569+0000 7f708b90a700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-06T12:11:56.604 INFO:tasks.ceph.osd.2.smithi189.stderr:2024-04-06T12:11:56.606+0000 7ff7e46c1700 -1 osd.2 0 waiting for initial osdmap 2024-04-06T12:11:56.621 INFO:tasks.ceph.osd.2.smithi189.stderr:2024-04-06T12:11:56.623+0000 7ff7dc29b700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T12:11:56.631 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-06T12:11:56.631+0000 7fe318790700 -1 osd.7 0 waiting for initial osdmap 2024-04-06T12:11:56.643 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-06T12:11:56.645+0000 7fe31036a700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T12:11:56.648 INFO:tasks.ceph.osd.1.smithi189.stderr:2024-04-06T12:11:56.651+0000 7f7a4a5c6700 -1 osd.1 0 waiting for initial osdmap 2024-04-06T12:11:56.653 INFO:tasks.ceph.osd.0.smithi189.stderr:2024-04-06T12:11:56.654+0000 7f294b7ec700 -1 osd.0 0 waiting for initial osdmap 2024-04-06T12:11:56.653 INFO:tasks.ceph.osd.1.smithi189.stderr:2024-04-06T12:11:56.655+0000 7f7a421a0700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T12:11:56.660 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-06T12:11:56.659+0000 7ffb83e59700 -1 osd.4 0 waiting for initial osdmap 2024-04-06T12:11:56.667 INFO:tasks.ceph.osd.0.smithi189.stderr:2024-04-06T12:11:56.670+0000 7f29433c6700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T12:11:56.670 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-06T12:11:56.672+0000 7fd45b31b700 -1 osd.6 0 waiting for initial osdmap 2024-04-06T12:11:56.674 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-06T12:11:56.676+0000 7ffb7ba33700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T12:11:56.684 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-06T12:11:56.686+0000 7fd452ef5700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T12:11:56.698 INFO:tasks.ceph.osd.3.smithi189.stderr:2024-04-06T12:11:56.700+0000 7f9c3ec91700 -1 osd.3 0 waiting for initial osdmap 2024-04-06T12:11:56.708 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-06T12:11:56.710+0000 7fab09646700 -1 osd.5 0 waiting for initial osdmap 2024-04-06T12:11:56.709 INFO:tasks.ceph.osd.3.smithi189.stderr:2024-04-06T12:11:56.712+0000 7f9c3686b700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T12:11:56.724 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-06T12:11:56.726+0000 7fab01220700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T12:11:57.296 DEBUG:teuthology.orchestra.run.smithi189:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-06T12:11:57.423 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.425+0000 7efbfb6b9700 1 Processor -- start 2024-04-06T12:11:57.423 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.425+0000 7efbfb6b9700 1 -- start start 2024-04-06T12:11:57.427 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.426+0000 7efbfb6b9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efbf406e1d0 0x7efbf40b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:57.427 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.426+0000 7efbfb6b9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efbf4099a50 0x7efbf40ad160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:57.427 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.426+0000 7efbfb6b9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf406b170 con 0x7efbf415d890 2024-04-06T12:11:57.428 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.426+0000 7efbfb6b9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbf415dc70 con 0x7efbf4099a50 2024-04-06T12:11:57.428 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.426+0000 7efbfb6b9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf406b600 con 0x7efbf406e1d0 2024-04-06T12:11:57.428 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.426+0000 7efbf9455700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efbf415d890 0x7efbf406dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35690/0 (socket says 172.21.15.189:35690) 2024-04-06T12:11:57.428 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.426+0000 7efbf9455700 1 -- 172.21.15.189:0/1427137066 learned_addr learned my addr 172.21.15.189:0/1427137066 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:11:57.428 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.426+0000 7efbf8c54700 1 --2- 172.21.15.189:0/1427137066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efbf406e1d0 0x7efbf40b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:57.428 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.426+0000 7efbf9c56700 1 --2- 172.21.15.189:0/1427137066 >> v2:172.21.15.192:3300/0 conn(0x7efbf4099a50 0x7efbf40ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:57.428 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.426+0000 7efbebfff700 1 -- 172.21.15.189:0/1427137066 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864955632 0 0) 0x7efbf406b170 con 0x7efbf415d890 2024-04-06T12:11:57.429 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.426+0000 7efbebfff700 1 -- 172.21.15.189:0/1427137066 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbe00035b0 con 0x7efbf415d890 2024-04-06T12:11:57.429 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbebfff700 1 -- 172.21.15.189:0/1427137066 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3926237555 0 0) 0x7efbe00035b0 con 0x7efbf415d890 2024-04-06T12:11:57.429 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbebfff700 1 -- 172.21.15.189:0/1427137066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efbf406e1d0 msgr2=0x7efbf40b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:11:57.429 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbebfff700 1 --2- 172.21.15.189:0/1427137066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efbf406e1d0 0x7efbf40b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:57.429 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbebfff700 1 -- 172.21.15.189:0/1427137066 >> v2:172.21.15.192:3300/0 conn(0x7efbf4099a50 msgr2=0x7efbf40ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:11:57.429 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbebfff700 1 --2- 172.21.15.189:0/1427137066 >> v2:172.21.15.192:3300/0 conn(0x7efbf4099a50 0x7efbf40ad160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:57.429 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbebfff700 1 -- 172.21.15.189:0/1427137066 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf406a840 con 0x7efbf415d890 2024-04-06T12:11:57.429 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbebfff700 1 -- 172.21.15.189:0/1427137066 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7efbf0001670 con 0x7efbf415d890 2024-04-06T12:11:57.429 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbebfff700 1 -- 172.21.15.189:0/1427137066 >> v1:172.21.15.189:6789/0 conn(0x7efbf415d890 legacy=0x7efbf406dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:57.429 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbebfff700 1 --2- 172.21.15.189:0/1427137066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efbe0003a00 0x7efbe0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:57.429 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbebfff700 1 --2- 172.21.15.189:0/1427137066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efbf406e1d0 0x7efbe0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:57.429 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbebfff700 1 --2- 172.21.15.189:0/1427137066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efbe0004d90 0x7efbe0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:57.430 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbebfff700 1 -- 172.21.15.189:0/1427137066 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf415dc70 con 0x7efbe0004d90 2024-04-06T12:11:57.430 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbebfff700 1 -- 172.21.15.189:0/1427137066 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf406b600 con 0x7efbe0003a00 2024-04-06T12:11:57.430 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbebfff700 1 -- 172.21.15.189:0/1427137066 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbe000ba10 con 0x7efbf406e1d0 2024-04-06T12:11:57.430 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbf9455700 1 --2- 172.21.15.189:0/1427137066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efbe0003a00 0x7efbe0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:57.430 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbf9c56700 1 --2- 172.21.15.189:0/1427137066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efbe0004d90 0x7efbe0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:57.430 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbf8c54700 1 --2- 172.21.15.189:0/1427137066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efbf406e1d0 0x7efbe0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:57.431 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbf9c56700 1 -- 172.21.15.189:0/1427137066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efbf406e1d0 msgr2=0x7efbe0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:11:57.431 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbf9c56700 1 --2- 172.21.15.189:0/1427137066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efbf406e1d0 0x7efbe0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:57.431 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbf9c56700 1 -- 172.21.15.189:0/1427137066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efbe0003a00 msgr2=0x7efbe0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:11:57.431 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbf9c56700 1 --2- 172.21.15.189:0/1427137066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efbe0003a00 0x7efbe0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:57.431 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbf9c56700 1 -- 172.21.15.189:0/1427137066 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbe00035b0 con 0x7efbe0004d90 2024-04-06T12:11:57.431 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbf9c56700 1 --2- 172.21.15.189:0/1427137066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efbe0004d90 0x7efbe0005210 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efbec009fc0 tx=0x7efbe0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99a5701a125f827 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:57.431 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbebfff700 1 -- 172.21.15.189:0/1427137066 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbec00ec30 con 0x7efbe0004d90 2024-04-06T12:11:57.431 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbebfff700 1 -- 172.21.15.189:0/1427137066 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbec00f040 con 0x7efbe0004d90 2024-04-06T12:11:57.431 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.427+0000 7efbebfff700 1 -- 172.21.15.189:0/1427137066 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbec014480 con 0x7efbe0004d90 2024-04-06T12:11:57.431 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.428+0000 7efbfb6b9700 1 -- 172.21.15.189:0/1427137066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efbe0004d90 msgr2=0x7efbe0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:57.431 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.428+0000 7efbfb6b9700 1 --2- 172.21.15.189:0/1427137066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efbe0004d90 0x7efbe0005210 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efbec009fc0 tx=0x7efbe0005880 comp rx=0 tx=0).stop 2024-04-06T12:11:57.431 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.428+0000 7efbf9c56700 1 -- 172.21.15.189:0/1427137066 reap_dead start 2024-04-06T12:11:57.431 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.428+0000 7efbfb6b9700 1 -- 172.21.15.189:0/1427137066 shutdown_connections 2024-04-06T12:11:57.432 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.428+0000 7efbfb6b9700 1 -- 172.21.15.189:0/1427137066 >> 172.21.15.189:0/1427137066 conn(0x7efbf4071f60 msgr2=0x7efbf40b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:57.432 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.428+0000 7efbfb6b9700 1 -- 172.21.15.189:0/1427137066 shutdown_connections 2024-04-06T12:11:57.432 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.428+0000 7efbfb6b9700 1 -- 172.21.15.189:0/1427137066 wait complete. 2024-04-06T12:11:57.432 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.429+0000 7efbfb6b9700 1 Processor -- start 2024-04-06T12:11:57.432 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.429+0000 7efbfb6b9700 1 -- start start 2024-04-06T12:11:57.435 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.430+0000 7efbfb6b9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efbf4144c70 0x7efbf40ae800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:57.435 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.430+0000 7efbfb6b9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efbf40b0b60 0x7efbf40b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:57.435 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.430+0000 7efbfb6b9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efbf41f5e40 0x7efbf41f62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:57.435 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.430+0000 7efbfb6b9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf40b1520 con 0x7efbf40b0b60 2024-04-06T12:11:57.435 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.430+0000 7efbfb6b9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf40adf10 con 0x7efbf41f5e40 2024-04-06T12:11:57.435 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.430+0000 7efbfb6b9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf4145050 con 0x7efbf4144c70 2024-04-06T12:11:57.435 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.430+0000 7efbf8c54700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efbf40b0b60 0x7efbf40b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:57.435 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.430+0000 7efbf8c54700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efbf40b0b60 0x7efbf40b0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60014/0 (socket says 172.21.15.189:60014) 2024-04-06T12:11:57.435 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.430+0000 7efbf8c54700 1 -- 172.21.15.189:0/2276748766 learned_addr learned my addr 172.21.15.189:0/2276748766 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:11:57.435 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.430+0000 7efbf9455700 1 --2- 172.21.15.189:0/2276748766 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efbf4144c70 0x7efbf40ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:57.435 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.430+0000 7efbf9c56700 1 --2- 172.21.15.189:0/2276748766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efbf41f5e40 0x7efbf41f62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:57.435 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.430+0000 7efbf8c54700 1 -- 172.21.15.189:0/2276748766 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efbf4144c70 msgr2=0x7efbf40ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:57.436 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.430+0000 7efbf8c54700 1 --2- 172.21.15.189:0/2276748766 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efbf4144c70 0x7efbf40ae800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:57.436 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.430+0000 7efbf8c54700 1 -- 172.21.15.189:0/2276748766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efbf41f5e40 msgr2=0x7efbf41f62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:57.436 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.430+0000 7efbf8c54700 1 --2- 172.21.15.189:0/2276748766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efbf41f5e40 0x7efbf41f62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:57.436 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.430+0000 7efbf8c54700 1 -- 172.21.15.189:0/2276748766 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf40ad300 con 0x7efbf40b0b60 2024-04-06T12:11:57.436 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.431+0000 7efbf8c54700 1 --2- 172.21.15.189:0/2276748766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efbf40b0b60 0x7efbf40b0fe0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efbe4011aa0 tx=0x7efbe4011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:57.436 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.431+0000 7efbe9ffb700 1 -- 172.21.15.189:0/2276748766 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe4011ee0 con 0x7efbf40b0b60 2024-04-06T12:11:57.436 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.431+0000 7efbe9ffb700 1 -- 172.21.15.189:0/2276748766 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbe400f940 con 0x7efbf40b0b60 2024-04-06T12:11:57.436 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.431+0000 7efbe9ffb700 1 -- 172.21.15.189:0/2276748766 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe400fcf0 con 0x7efbf40b0b60 2024-04-06T12:11:57.436 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.431+0000 7efbfb6b9700 1 -- 172.21.15.189:0/2276748766 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf40ad5e0 con 0x7efbf40b0b60 2024-04-06T12:11:57.436 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.431+0000 7efbfb6b9700 1 -- 172.21.15.189:0/2276748766 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf40adba0 con 0x7efbf40b0b60 2024-04-06T12:11:57.436 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.432+0000 7efbfb6b9700 1 -- 172.21.15.189:0/2276748766 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf4074930 con 0x7efbf40b0b60 2024-04-06T12:11:57.436 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.432+0000 7efbe9ffb700 1 -- 172.21.15.189:0/2276748766 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7efbe4014440 con 0x7efbf40b0b60 2024-04-06T12:11:57.436 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.433+0000 7efbe9ffb700 1 --2- 172.21.15.189:0/2276748766 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efbd806ac00 0x7efbd806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:57.436 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.433+0000 7efbe9ffb700 1 -- 172.21.15.189:0/2276748766 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7efbe4088ab0 con 0x7efbf40b0b60 2024-04-06T12:11:57.436 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.433+0000 7efbf9455700 1 --2- 172.21.15.189:0/2276748766 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efbd806ac00 0x7efbd806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:57.437 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.433+0000 7efbf9455700 1 --2- 172.21.15.189:0/2276748766 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efbd806ac00 0x7efbd806d0c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efbf0000c00 tx=0x7efbf0007980 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:57.437 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.437+0000 7efbe9ffb700 1 -- 172.21.15.189:0/2276748766 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efbe4054bc0 con 0x7efbf40b0b60 2024-04-06T12:11:57.639 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.641+0000 7efbfb6b9700 1 -- 172.21.15.189:0/2276748766 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efbf406c9e0 con 0x7efbf40b0b60 2024-04-06T12:11:57.639 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.642+0000 7efbe9ffb700 1 -- 172.21.15.189:0/2276748766 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10488 (secure 0 0 0) 0x7efbe4022070 con 0x7efbf40b0b60 2024-04-06T12:11:57.640 INFO:teuthology.misc.health.smithi189.stdout: 2024-04-06T12:11:57.640 INFO:teuthology.misc.health.smithi189.stdout:{"epoch":13,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:11:57.614001+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:11:57.642 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.644+0000 7efbfb6b9700 1 -- 172.21.15.189:0/2276748766 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efbd806ac00 msgr2=0x7efbd806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:57.642 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.644+0000 7efbfb6b9700 1 --2- 172.21.15.189:0/2276748766 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efbd806ac00 0x7efbd806d0c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efbf0000c00 tx=0x7efbf0007980 comp rx=0 tx=0).stop 2024-04-06T12:11:57.642 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.644+0000 7efbfb6b9700 1 -- 172.21.15.189:0/2276748766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efbf40b0b60 msgr2=0x7efbf40b0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:57.642 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.644+0000 7efbfb6b9700 1 --2- 172.21.15.189:0/2276748766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efbf40b0b60 0x7efbf40b0fe0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efbe4011aa0 tx=0x7efbe4011db0 comp rx=0 tx=0).stop 2024-04-06T12:11:57.643 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.644+0000 7efbfb6b9700 1 -- 172.21.15.189:0/2276748766 shutdown_connections 2024-04-06T12:11:57.643 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.644+0000 7efbfb6b9700 1 --2- 172.21.15.189:0/2276748766 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efbd806ac00 0x7efbd806d0c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:57.643 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.644+0000 7efbfb6b9700 1 --2- 172.21.15.189:0/2276748766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efbf41f5e40 0x7efbf41f62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:57.643 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.644+0000 7efbfb6b9700 1 --2- 172.21.15.189:0/2276748766 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efbf4144c70 0x7efbf40ae800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:57.643 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.644+0000 7efbfb6b9700 1 --2- 172.21.15.189:0/2276748766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efbf40b0b60 0x7efbf40b0fe0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:57.643 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.644+0000 7efbfb6b9700 1 -- 172.21.15.189:0/2276748766 >> 172.21.15.189:0/2276748766 conn(0x7efbf4071f60 msgr2=0x7efbf40b1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:57.643 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.644+0000 7efbfb6b9700 1 -- 172.21.15.189:0/2276748766 shutdown_connections 2024-04-06T12:11:57.643 INFO:teuthology.misc.health.smithi189.stderr:2024-04-06T12:11:57.644+0000 7efbfb6b9700 1 -- 172.21.15.189:0/2276748766 wait complete. 2024-04-06T12:11:57.654 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-06T12:11:57.654 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-06T12:11:57.654 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-06T12:11:57.655 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-06T12:11:57.657 INFO:tasks.ceph.mds.a:Started 2024-04-06T12:11:57.657 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-06T12:11:57.657 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-06T12:11:57.659 INFO:tasks.ceph.mds.c:Started 2024-04-06T12:11:57.659 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-06T12:11:57.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-06T12:11:57.661 INFO:tasks.ceph.mds.b:Started 2024-04-06T12:11:57.661 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-06T12:11:57.662 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-06T12:11:57.664 INFO:tasks.ceph.mds.d:Started 2024-04-06T12:11:57.664 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-06T12:11:57.664 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:11:57.737 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2024-04-06T12:11:57.738 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T12:11:57.747 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2024-04-06T12:11:57.749 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T12:11:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.796+0000 7f95b3cdd700 1 Processor -- start 2024-04-06T12:11:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.797+0000 7f95b3cdd700 1 -- start start 2024-04-06T12:11:57.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.797+0000 7f95b3cdd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f95ac15d7b0 0x7f95ac0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:57.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.797+0000 7f95b3cdd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ac0b3990 con 0x7f95ac15e500 2024-04-06T12:11:57.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.797+0000 7f95b3cdd700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ac0b3060 con 0x7f95ac06a730 2024-04-06T12:11:57.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.797+0000 7f95b3cdd700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ac15e950 con 0x7f95ac15d7b0 2024-04-06T12:11:57.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b227a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f95ac15e500 0x7f95ac0717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35700/0 (socket says 172.21.15.189:35700) 2024-04-06T12:11:57.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b227a700 1 -- 172.21.15.189:0/1320508046 learned_addr learned my addr 172.21.15.189:0/1320508046 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:11:57.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b1a79700 1 --2- 172.21.15.189:0/1320508046 >> v2:172.21.15.192:3300/0 conn(0x7f95ac15d7b0 0x7f95ac0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:57.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b0a77700 1 -- 172.21.15.189:0/1320508046 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36593030 0 0) 0x7f95ac0b3990 con 0x7f95ac15e500 2024-04-06T12:11:57.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b0a77700 1 -- 172.21.15.189:0/1320508046 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95980035b0 con 0x7f95ac15e500 2024-04-06T12:11:57.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b0a77700 1 -- 172.21.15.189:0/1320508046 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 310792249 0 0) 0x7f95980035b0 con 0x7f95ac15e500 2024-04-06T12:11:57.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b0a77700 1 -- 172.21.15.189:0/1320508046 >> v2:172.21.15.192:3300/0 conn(0x7f95ac15d7b0 msgr2=0x7f95ac0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:11:57.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b0a77700 1 --2- 172.21.15.189:0/1320508046 >> v2:172.21.15.192:3300/0 conn(0x7f95ac15d7b0 0x7f95ac0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:57.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b0a77700 1 -- 172.21.15.189:0/1320508046 >> v1:172.21.15.192:6789/0 conn(0x7f95ac06a730 legacy=0x7f95ac15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:57.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b0a77700 1 -- 172.21.15.189:0/1320508046 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ac0b3340 con 0x7f95ac15e500 2024-04-06T12:11:57.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b0a77700 1 -- 172.21.15.189:0/1320508046 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f95a40019f0 con 0x7f95ac15e500 2024-04-06T12:11:57.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b0a77700 1 -- 172.21.15.189:0/1320508046 >> v1:172.21.15.189:6789/0 conn(0x7f95ac15e500 legacy=0x7f95ac0717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:57.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b0a77700 1 --2- 172.21.15.189:0/1320508046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9598003a00 0x7f9598003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:57.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b0a77700 1 --2- 172.21.15.189:0/1320508046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9598004850 0x7f9598004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:57.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b0a77700 1 --2- 172.21.15.189:0/1320508046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9598005210 0x7f959800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:57.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b0a77700 1 -- 172.21.15.189:0/1320508046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959800c180 con 0x7f9598005210 2024-04-06T12:11:57.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b0a77700 1 -- 172.21.15.189:0/1320508046 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959800c2f0 con 0x7f9598004850 2024-04-06T12:11:57.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b0a77700 1 -- 172.21.15.189:0/1320508046 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f959800c4b0 con 0x7f9598003a00 2024-04-06T12:11:57.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b227a700 1 --2- 172.21.15.189:0/1320508046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9598005210 0x7f959800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:57.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b1a79700 1 --2- 172.21.15.189:0/1320508046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9598003a00 0x7f9598003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:57.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b1278700 1 --2- 172.21.15.189:0/1320508046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9598004850 0x7f9598004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:57.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b227a700 1 -- 172.21.15.189:0/1320508046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9598003a00 msgr2=0x7f9598003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:11:57.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b227a700 1 --2- 172.21.15.189:0/1320508046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9598003a00 0x7f9598003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:57.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b227a700 1 -- 172.21.15.189:0/1320508046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9598004850 msgr2=0x7f9598004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:11:57.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.798+0000 7f95b227a700 1 --2- 172.21.15.189:0/1320508046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9598004850 0x7f9598004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:57.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.799+0000 7f95b227a700 1 -- 172.21.15.189:0/1320508046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95980035b0 con 0x7f9598005210 2024-04-06T12:11:57.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.799+0000 7f95b1278700 1 --2- 172.21.15.189:0/1320508046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9598004850 0x7f9598004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:11:57.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.799+0000 7f95b227a700 1 -- 172.21.15.189:0/1320508046 reap_dead start 2024-04-06T12:11:57.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.799+0000 7f95b227a700 1 --2- 172.21.15.189:0/1320508046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9598005210 0x7f959800ba10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f95a400a830 tx=0x7f9598005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e7bffe5a3fcfc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:57.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.799+0000 7f95b0a77700 1 -- 172.21.15.189:0/1320508046 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a400a2f0 con 0x7f9598005210 2024-04-06T12:11:57.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.799+0000 7f95b0a77700 1 -- 172.21.15.189:0/1320508046 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a4012620 con 0x7f9598005210 2024-04-06T12:11:57.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.799+0000 7f95b0a77700 1 -- 172.21.15.189:0/1320508046 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a40129d0 con 0x7f9598005210 2024-04-06T12:11:57.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.799+0000 7f95b3cdd700 1 -- 172.21.15.189:0/1320508046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9598005210 msgr2=0x7f959800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:57.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.799+0000 7f95b3cdd700 1 --2- 172.21.15.189:0/1320508046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9598005210 0x7f959800ba10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f95a400a830 tx=0x7f9598005880 comp rx=0 tx=0).stop 2024-04-06T12:11:57.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.799+0000 7f95b3cdd700 1 -- 172.21.15.189:0/1320508046 shutdown_connections 2024-04-06T12:11:57.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.799+0000 7f95b3cdd700 1 --2- 172.21.15.189:0/1320508046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9598005210 0x7f959800ba10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:57.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.799+0000 7f95b3cdd700 1 -- 172.21.15.189:0/1320508046 >> 172.21.15.189:0/1320508046 conn(0x7f95ac0d00b0 msgr2=0x7f95ac0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:57.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.799+0000 7f95b3cdd700 1 -- 172.21.15.189:0/1320508046 shutdown_connections 2024-04-06T12:11:57.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.799+0000 7f95b3cdd700 1 -- 172.21.15.189:0/1320508046 wait complete. 2024-04-06T12:11:57.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.801+0000 7f95b3cdd700 1 Processor -- start 2024-04-06T12:11:57.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.801+0000 7f95b3cdd700 1 -- start start 2024-04-06T12:11:57.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.801+0000 7f95b3cdd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9598005210 0x7f95ac0afa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:57.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.801+0000 7f95b3cdd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95ac0ce390 0x7f95ac154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:57.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.801+0000 7f95b3cdd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95ac0ce810 0x7f95ac155110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:57.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.801+0000 7f95b3cdd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac0affd0 con 0x7f95ac0ce810 2024-04-06T12:11:57.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.801+0000 7f95b3cdd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac0cd490 con 0x7f95ac0ce390 2024-04-06T12:11:57.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.801+0000 7f95b3cdd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac15a270 con 0x7f9598005210 2024-04-06T12:11:57.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.801+0000 7f95b227a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95ac0ce810 0x7f95ac155110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:57.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.801+0000 7f95b227a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95ac0ce810 0x7f95ac155110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60058/0 (socket says 172.21.15.189:60058) 2024-04-06T12:11:57.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.801+0000 7f95b227a700 1 -- 172.21.15.189:0/1677049761 learned_addr learned my addr 172.21.15.189:0/1677049761 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:11:57.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.802+0000 7f95b1278700 1 --2- 172.21.15.189:0/1677049761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95ac0ce390 0x7f95ac154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:57.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.802+0000 7f95b1a79700 1 --2- 172.21.15.189:0/1677049761 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9598005210 0x7f95ac0afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:57.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.802+0000 7f95b227a700 1 -- 172.21.15.189:0/1677049761 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9598005210 msgr2=0x7f95ac0afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:57.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.802+0000 7f95b227a700 1 --2- 172.21.15.189:0/1677049761 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9598005210 0x7f95ac0afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:57.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.802+0000 7f95b227a700 1 -- 172.21.15.189:0/1677049761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95ac0ce390 msgr2=0x7f95ac154bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:57.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.802+0000 7f95b227a700 1 --2- 172.21.15.189:0/1677049761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95ac0ce390 0x7f95ac154bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:57.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.802+0000 7f95b227a700 1 -- 172.21.15.189:0/1677049761 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ac15a550 con 0x7f95ac0ce810 2024-04-06T12:11:57.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.802+0000 7f95b227a700 1 --2- 172.21.15.189:0/1677049761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95ac0ce810 0x7f95ac155110 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f95ac0d0060 tx=0x7f95a4001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:57.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.802+0000 7f959e7fc700 1 -- 172.21.15.189:0/1677049761 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a4024070 con 0x7f95ac0ce810 2024-04-06T12:11:57.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.802+0000 7f959e7fc700 1 -- 172.21.15.189:0/1677049761 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a400a210 con 0x7f95ac0ce810 2024-04-06T12:11:57.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.802+0000 7f95b3cdd700 1 -- 172.21.15.189:0/1677049761 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ac15a830 con 0x7f95ac0ce810 2024-04-06T12:11:57.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.802+0000 7f959e7fc700 1 -- 172.21.15.189:0/1677049761 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a40019f0 con 0x7f95ac0ce810 2024-04-06T12:11:57.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.802+0000 7f95b3cdd700 1 -- 172.21.15.189:0/1677049761 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ac0cc740 con 0x7f95ac0ce810 2024-04-06T12:11:57.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.803+0000 7f95b3cdd700 1 -- 172.21.15.189:0/1677049761 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ac15ad50 con 0x7f95ac0ce810 2024-04-06T12:11:57.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.803+0000 7f959e7fc700 1 -- 172.21.15.189:0/1677049761 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7f95a401d630 con 0x7f95ac0ce810 2024-04-06T12:11:57.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.804+0000 7f959e7fc700 1 --2- 172.21.15.189:0/1677049761 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f959006ac00 0x7f959006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:57.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.804+0000 7f95b1a79700 1 --2- 172.21.15.189:0/1677049761 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f959006ac00 0x7f959006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:57.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.804+0000 7f959e7fc700 1 -- 172.21.15.189:0/1677049761 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f95a408a0f0 con 0x7f95ac0ce810 2024-04-06T12:11:57.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.804+0000 7f95b1a79700 1 --2- 172.21.15.189:0/1677049761 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f959006ac00 0x7f959006d0c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f95a0009be0 tx=0x7f95a0009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:57.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:57.808+0000 7f959e7fc700 1 -- 172.21.15.189:0/1677049761 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f95ac15ad50 con 0x7f95ac0ce810 2024-04-06T12:11:58.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.005+0000 7f95b3cdd700 1 -- 172.21.15.189:0/1677049761 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95ac15ad50 con 0x7f95ac0ce810 2024-04-06T12:11:58.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.006+0000 7f959e7fc700 1 -- 172.21.15.189:0/1677049761 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10488 (secure 0 0 0) 0x7f95ac15ad50 con 0x7f95ac0ce810 2024-04-06T12:11:58.004 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:58.004 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":13,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:11:57.614001+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:11:58.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.008+0000 7f95b3cdd700 1 -- 172.21.15.189:0/1677049761 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f959006ac00 msgr2=0x7f959006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:58.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.008+0000 7f95b3cdd700 1 --2- 172.21.15.189:0/1677049761 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f959006ac00 0x7f959006d0c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f95a0009be0 tx=0x7f95a0009210 comp rx=0 tx=0).stop 2024-04-06T12:11:58.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.008+0000 7f95b3cdd700 1 -- 172.21.15.189:0/1677049761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95ac0ce810 msgr2=0x7f95ac155110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:58.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.008+0000 7f95b3cdd700 1 --2- 172.21.15.189:0/1677049761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95ac0ce810 0x7f95ac155110 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f95ac0d0060 tx=0x7f95a4001d80 comp rx=0 tx=0).stop 2024-04-06T12:11:58.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.008+0000 7f95b3cdd700 1 -- 172.21.15.189:0/1677049761 shutdown_connections 2024-04-06T12:11:58.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.008+0000 7f95b3cdd700 1 --2- 172.21.15.189:0/1677049761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95ac0ce810 0x7f95ac155110 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:58.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.008+0000 7f95b3cdd700 1 --2- 172.21.15.189:0/1677049761 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9598005210 0x7f95ac0afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:58.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.008+0000 7f95b3cdd700 1 --2- 172.21.15.189:0/1677049761 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f959006ac00 0x7f959006d0c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:58.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.008+0000 7f95b3cdd700 1 --2- 172.21.15.189:0/1677049761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95ac0ce390 0x7f95ac154bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:58.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.008+0000 7f95b3cdd700 1 -- 172.21.15.189:0/1677049761 >> 172.21.15.189:0/1677049761 conn(0x7f95ac0d00b0 msgr2=0x7f95ac06ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:58.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.008+0000 7f95b3cdd700 1 -- 172.21.15.189:0/1677049761 shutdown_connections 2024-04-06T12:11:58.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.008+0000 7f95b3cdd700 1 -- 172.21.15.189:0/1677049761 wait complete. 2024-04-06T12:11:58.019 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-04-06T12:11:58.019 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:11:58.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.142+0000 7f8bcfcc0700 1 Processor -- start 2024-04-06T12:11:58.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.143+0000 7f8bcfcc0700 1 -- start start 2024-04-06T12:11:58.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.143+0000 7f8bcfcc0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8bc8146dd0 0x7f8bc806d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:58.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.143+0000 7f8bcfcc0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc806c0a0 con 0x7f8bc80d17a0 2024-04-06T12:11:58.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.143+0000 7f8bcfcc0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc806c380 con 0x7f8bc80cedf0 2024-04-06T12:11:58.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.143+0000 7f8bcfcc0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc806beb0 con 0x7f8bc8146dd0 2024-04-06T12:11:58.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.143+0000 7f8bcd25b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8bc8146dd0 0x7f8bc806d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:58.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.144+0000 7f8bcda5c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8bc80d17a0 0x7f8bc806b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35702/0 (socket says 172.21.15.189:35702) 2024-04-06T12:11:58.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.144+0000 7f8bcda5c700 1 -- 172.21.15.189:0/946056880 learned_addr learned my addr 172.21.15.189:0/946056880 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:11:58.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.144+0000 7f8bcd25b700 1 -- 172.21.15.189:0/946056880 >> v1:172.21.15.192:6789/0 conn(0x7f8bc80cedf0 legacy=0x7f8bc80cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:58.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.144+0000 7f8bcd25b700 1 -- 172.21.15.189:0/946056880 >> v1:172.21.15.189:6789/0 conn(0x7f8bc80d17a0 legacy=0x7f8bc806b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:58.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.144+0000 7f8bcd25b700 1 -- 172.21.15.189:0/946056880 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc806c660 con 0x7f8bc8146dd0 2024-04-06T12:11:58.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.144+0000 7f8bcca5a700 1 -- 172.21.15.189:0/946056880 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296454851 0 0) 0x7f8bc806c0a0 con 0x7f8bc80d17a0 2024-04-06T12:11:58.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.144+0000 7f8bcd25b700 1 --2- 172.21.15.189:0/946056880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc8146dd0 0x7f8bc806d140 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009840 tx=0x7f8bb8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5d8f3b875830ae server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:58.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.144+0000 7f8bcca5a700 1 -- 172.21.15.189:0/946056880 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8011830 con 0x7f8bc8146dd0 2024-04-06T12:11:58.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.144+0000 7f8bcca5a700 1 -- 172.21.15.189:0/946056880 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb8011990 con 0x7f8bc8146dd0 2024-04-06T12:11:58.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.144+0000 7f8bcca5a700 1 -- 172.21.15.189:0/946056880 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8011d20 con 0x7f8bc8146dd0 2024-04-06T12:11:58.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.145+0000 7f8bcfcc0700 1 -- 172.21.15.189:0/946056880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc8146dd0 msgr2=0x7f8bc806d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:58.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.145+0000 7f8bcfcc0700 1 --2- 172.21.15.189:0/946056880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc8146dd0 0x7f8bc806d140 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009840 tx=0x7f8bb8009b50 comp rx=0 tx=0).stop 2024-04-06T12:11:58.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.145+0000 7f8bcfcc0700 1 -- 172.21.15.189:0/946056880 shutdown_connections 2024-04-06T12:11:58.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.145+0000 7f8bcfcc0700 1 --2- 172.21.15.189:0/946056880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc8146dd0 0x7f8bc806d140 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:58.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.145+0000 7f8bcfcc0700 1 -- 172.21.15.189:0/946056880 >> 172.21.15.189:0/946056880 conn(0x7f8bc80b2db0 msgr2=0x7f8bc80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:58.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.145+0000 7f8bcfcc0700 1 -- 172.21.15.189:0/946056880 shutdown_connections 2024-04-06T12:11:58.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.145+0000 7f8bcfcc0700 1 -- 172.21.15.189:0/946056880 wait complete. 2024-04-06T12:11:58.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.146+0000 7f8bcfcc0700 1 Processor -- start 2024-04-06T12:11:58.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.146+0000 7f8bcfcc0700 1 -- start start 2024-04-06T12:11:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bcfcc0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8bc80cedf0 0x7f8bc80b5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bcfcc0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc80d17a0 0x7f8bc8155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bcfcc0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bc8146dd0 0x7f8bc8157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bcfcc0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc80c69a0 con 0x7f8bc80d17a0 2024-04-06T12:11:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bcfcc0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc80c6dc0 con 0x7f8bc80cedf0 2024-04-06T12:11:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bcfcc0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc8156480 con 0x7f8bc8146dd0 2024-04-06T12:11:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bcd25b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc80d17a0 0x7f8bc8155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bcd25b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc80d17a0 0x7f8bc8155f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60090/0 (socket says 172.21.15.189:60090) 2024-04-06T12:11:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bcd25b700 1 -- 172.21.15.189:0/2630869643 learned_addr learned my addr 172.21.15.189:0/2630869643 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:11:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bcda5c700 1 --2- 172.21.15.189:0/2630869643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8bc80cedf0 0x7f8bc80b5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bce25d700 1 --2- 172.21.15.189:0/2630869643 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bc8146dd0 0x7f8bc8157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bcd25b700 1 -- 172.21.15.189:0/2630869643 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bc8146dd0 msgr2=0x7f8bc8157c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:58.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bcd25b700 1 --2- 172.21.15.189:0/2630869643 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bc8146dd0 0x7f8bc8157c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:58.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bcd25b700 1 -- 172.21.15.189:0/2630869643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8bc80cedf0 msgr2=0x7f8bc80b5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:58.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bcd25b700 1 --2- 172.21.15.189:0/2630869643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8bc80cedf0 0x7f8bc80b5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:58.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bcd25b700 1 -- 172.21.15.189:0/2630869643 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb80094a0 con 0x7f8bc80d17a0 2024-04-06T12:11:58.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bce25d700 1 --2- 172.21.15.189:0/2630869643 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bc8146dd0 0x7f8bc8157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:11:58.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bcd25b700 1 --2- 172.21.15.189:0/2630869643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc80d17a0 0x7f8bc8155f40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8bb80090b0 tx=0x7f8bb8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:58.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bbe7fc700 1 -- 172.21.15.189:0/2630869643 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb801a900 con 0x7f8bc80d17a0 2024-04-06T12:11:58.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bbe7fc700 1 -- 172.21.15.189:0/2630869643 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb801aa60 con 0x7f8bc80d17a0 2024-04-06T12:11:58.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bcfcc0700 1 -- 172.21.15.189:0/2630869643 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc80c6000 con 0x7f8bc80d17a0 2024-04-06T12:11:58.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bbe7fc700 1 -- 172.21.15.189:0/2630869643 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb801add0 con 0x7f8bc80d17a0 2024-04-06T12:11:58.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.147+0000 7f8bcfcc0700 1 -- 172.21.15.189:0/2630869643 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc80c65c0 con 0x7f8bc80d17a0 2024-04-06T12:11:58.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.148+0000 7f8bcfcc0700 1 -- 172.21.15.189:0/2630869643 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc8074990 con 0x7f8bc80d17a0 2024-04-06T12:11:58.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.149+0000 7f8bbe7fc700 1 -- 172.21.15.189:0/2630869643 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7f8bb8022070 con 0x7f8bc80d17a0 2024-04-06T12:11:58.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.149+0000 7f8bbe7fc700 1 --2- 172.21.15.189:0/2630869643 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8bac06ac00 0x7f8bac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:11:58.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.149+0000 7f8bbe7fc700 1 -- 172.21.15.189:0/2630869643 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f8bb808a120 con 0x7f8bc80d17a0 2024-04-06T12:11:58.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.149+0000 7f8bcda5c700 1 --2- 172.21.15.189:0/2630869643 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8bac06ac00 0x7f8bac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:11:58.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.149+0000 7f8bcda5c700 1 --2- 172.21.15.189:0/2630869643 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8bac06ac00 0x7f8bac06d0c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4007fd0 tx=0x7f8bc4007c20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:11:58.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.153+0000 7f8bbe7fc700 1 -- 172.21.15.189:0/2630869643 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8bb8055a60 con 0x7f8bc80d17a0 2024-04-06T12:11:58.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.392+0000 7f8bcfcc0700 1 -- 172.21.15.189:0/2630869643 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8bc80d1e20 con 0x7f8bc80d17a0 2024-04-06T12:11:58.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.393+0000 7f8bbe7fc700 1 -- 172.21.15.189:0/2630869643 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1259 (secure 0 0 0) 0x7f8bb8059080 con 0x7f8bc80d17a0 2024-04-06T12:11:58.391 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:11:58.391 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4233,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3550531296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3550531296},{"type":"v1","addr":"172.21.15.189:6837","nonce":3550531296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2024-04-06T12:11:58.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.395+0000 7f8bcfcc0700 1 -- 172.21.15.189:0/2630869643 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8bac06ac00 msgr2=0x7f8bac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:58.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.395+0000 7f8bcfcc0700 1 --2- 172.21.15.189:0/2630869643 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8bac06ac00 0x7f8bac06d0c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4007fd0 tx=0x7f8bc4007c20 comp rx=0 tx=0).stop 2024-04-06T12:11:58.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.395+0000 7f8bcfcc0700 1 -- 172.21.15.189:0/2630869643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc80d17a0 msgr2=0x7f8bc8155f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:11:58.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.395+0000 7f8bcfcc0700 1 --2- 172.21.15.189:0/2630869643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc80d17a0 0x7f8bc8155f40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8bb80090b0 tx=0x7f8bb8004750 comp rx=0 tx=0).stop 2024-04-06T12:11:58.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.395+0000 7f8bcfcc0700 1 -- 172.21.15.189:0/2630869643 shutdown_connections 2024-04-06T12:11:58.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.395+0000 7f8bcfcc0700 1 --2- 172.21.15.189:0/2630869643 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bc8146dd0 0x7f8bc8157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:58.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.395+0000 7f8bcfcc0700 1 --2- 172.21.15.189:0/2630869643 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8bac06ac00 0x7f8bac06d0c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:58.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.395+0000 7f8bcfcc0700 1 --2- 172.21.15.189:0/2630869643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8bc80cedf0 0x7f8bc80b5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:58.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.395+0000 7f8bcfcc0700 1 --2- 172.21.15.189:0/2630869643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc80d17a0 0x7f8bc8155f40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:11:58.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.395+0000 7f8bcfcc0700 1 -- 172.21.15.189:0/2630869643 >> 172.21.15.189:0/2630869643 conn(0x7f8bc80b2db0 msgr2=0x7f8bc80b4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:11:58.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.395+0000 7f8bcfcc0700 1 -- 172.21.15.189:0/2630869643 shutdown_connections 2024-04-06T12:11:58.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:11:58.395+0000 7f8bcfcc0700 1 -- 172.21.15.189:0/2630869643 wait complete. 2024-04-06T12:11:58.394 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 2 2024-04-06T12:12:00.406 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:12:00.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.544+0000 7fac3761f700 1 Processor -- start 2024-04-06T12:12:00.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.544+0000 7fac3761f700 1 -- start start 2024-04-06T12:12:00.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.545+0000 7fac3761f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac3006ae40 0x7fac3015ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:00.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.545+0000 7fac3761f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fac3015fcd0 0x7fac3015bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:00.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.545+0000 7fac3761f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3015c4c0 con 0x7fac30144f40 2024-04-06T12:12:00.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.545+0000 7fac3761f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac300cfcb0 con 0x7fac3015fcd0 2024-04-06T12:12:00.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.545+0000 7fac3761f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac3015ca20 con 0x7fac3006ae40 2024-04-06T12:12:00.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.545+0000 7fac35bbc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fac3015fcd0 0x7fac3015bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:00.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.545+0000 7fac35bbc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fac3015fcd0 0x7fac3015bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44398/0 (socket says 172.21.15.189:44398) 2024-04-06T12:12:00.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.545+0000 7fac35bbc700 1 -- 172.21.15.189:0/451094974 learned_addr learned my addr 172.21.15.189:0/451094974 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:00.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.545+0000 7fac34bba700 1 --2- 172.21.15.189:0/451094974 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac3006ae40 0x7fac3015ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:00.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.545+0000 7fac35bbc700 1 -- 172.21.15.189:0/451094974 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac3006ae40 msgr2=0x7fac3015ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:00.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.545+0000 7fac35bbc700 1 --2- 172.21.15.189:0/451094974 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac3006ae40 0x7fac3015ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:00.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.545+0000 7fac35bbc700 1 -- 172.21.15.189:0/451094974 >> v1:172.21.15.192:6789/0 conn(0x7fac30144f40 legacy=0x7fac3006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:00.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.545+0000 7fac35bbc700 1 -- 172.21.15.189:0/451094974 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3015c740 con 0x7fac3015fcd0 2024-04-06T12:12:00.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.545+0000 7fac34bba700 1 --2- 172.21.15.189:0/451094974 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac3006ae40 0x7fac3015ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:00.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.545+0000 7fac35bbc700 1 --2- 172.21.15.189:0/451094974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac3015fcd0 0x7fac3015bf80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fac28009e50 tx=0x7fac2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6a36be7cb8705 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:00.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.546+0000 7fac27fff700 1 -- 172.21.15.189:0/451094974 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2800e070 con 0x7fac3015fcd0 2024-04-06T12:12:00.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.546+0000 7fac27fff700 1 -- 172.21.15.189:0/451094974 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2800b9c0 con 0x7fac3015fcd0 2024-04-06T12:12:00.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.546+0000 7fac27fff700 1 -- 172.21.15.189:0/451094974 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2800bd70 con 0x7fac3015fcd0 2024-04-06T12:12:00.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.546+0000 7fac3761f700 1 -- 172.21.15.189:0/451094974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac3015fcd0 msgr2=0x7fac3015bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:00.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.546+0000 7fac3761f700 1 --2- 172.21.15.189:0/451094974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac3015fcd0 0x7fac3015bf80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fac28009e50 tx=0x7fac2800b330 comp rx=0 tx=0).stop 2024-04-06T12:12:00.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.546+0000 7fac3761f700 1 -- 172.21.15.189:0/451094974 shutdown_connections 2024-04-06T12:12:00.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.546+0000 7fac3761f700 1 --2- 172.21.15.189:0/451094974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac3015fcd0 0x7fac3015bf80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:00.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.546+0000 7fac3761f700 1 --2- 172.21.15.189:0/451094974 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac3006ae40 0x7fac3015ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:00.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.546+0000 7fac3761f700 1 -- 172.21.15.189:0/451094974 >> 172.21.15.189:0/451094974 conn(0x7fac300d00b0 msgr2=0x7fac300d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:00.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.546+0000 7fac3761f700 1 -- 172.21.15.189:0/451094974 shutdown_connections 2024-04-06T12:12:00.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.546+0000 7fac3761f700 1 -- 172.21.15.189:0/451094974 wait complete. 2024-04-06T12:12:00.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.547+0000 7fac3761f700 1 Processor -- start 2024-04-06T12:12:00.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.547+0000 7fac3761f700 1 -- start start 2024-04-06T12:12:00.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.548+0000 7fac3761f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac3006ae40 0x7fac301314c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:00.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.548+0000 7fac3761f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac30144f40 0x7fac300cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:00.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.548+0000 7fac3761f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac3015fcd0 0x7fac3015ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:00.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.548+0000 7fac3761f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac300cbfd0 con 0x7fac3006ae40 2024-04-06T12:12:00.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.548+0000 7fac3761f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac30131b50 con 0x7fac3015fcd0 2024-04-06T12:12:00.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.548+0000 7fac3761f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac30131e30 con 0x7fac30144f40 2024-04-06T12:12:00.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.548+0000 7fac353bb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac3006ae40 0x7fac301314c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:00.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.548+0000 7fac353bb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac3006ae40 0x7fac301314c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44400/0 (socket says 172.21.15.189:44400) 2024-04-06T12:12:00.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.548+0000 7fac353bb700 1 -- 172.21.15.189:0/2892576620 learned_addr learned my addr 172.21.15.189:0/2892576620 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:00.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.548+0000 7fac34bba700 1 --2- 172.21.15.189:0/2892576620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac30144f40 0x7fac300cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:00.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.548+0000 7fac35bbc700 1 --2- 172.21.15.189:0/2892576620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac3015fcd0 0x7fac3015ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:00.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.549+0000 7fac353bb700 1 -- 172.21.15.189:0/2892576620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac30144f40 msgr2=0x7fac300cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:00.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.549+0000 7fac353bb700 1 --2- 172.21.15.189:0/2892576620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac30144f40 0x7fac300cba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:00.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.549+0000 7fac353bb700 1 -- 172.21.15.189:0/2892576620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac3015fcd0 msgr2=0x7fac3015ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:00.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.549+0000 7fac353bb700 1 --2- 172.21.15.189:0/2892576620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac3015fcd0 0x7fac3015ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:00.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.549+0000 7fac353bb700 1 -- 172.21.15.189:0/2892576620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac28009b30 con 0x7fac3006ae40 2024-04-06T12:12:00.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.549+0000 7fac353bb700 1 --2- 172.21.15.189:0/2892576620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac3006ae40 0x7fac301314c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00c4e0 tx=0x7fac2c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:00.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.549+0000 7fac25ffb700 1 -- 172.21.15.189:0/2892576620 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0052d0 con 0x7fac3006ae40 2024-04-06T12:12:00.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.549+0000 7fac25ffb700 1 -- 172.21.15.189:0/2892576620 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2c005430 con 0x7fac3006ae40 2024-04-06T12:12:00.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.549+0000 7fac3761f700 1 -- 172.21.15.189:0/2892576620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac30132110 con 0x7fac3006ae40 2024-04-06T12:12:00.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.549+0000 7fac25ffb700 1 -- 172.21.15.189:0/2892576620 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0038c0 con 0x7fac3006ae40 2024-04-06T12:12:00.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.549+0000 7fac3761f700 1 -- 172.21.15.189:0/2892576620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac301326d0 con 0x7fac3006ae40 2024-04-06T12:12:00.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.550+0000 7fac3761f700 1 -- 172.21.15.189:0/2892576620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac30067bb0 con 0x7fac3006ae40 2024-04-06T12:12:00.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.551+0000 7fac25ffb700 1 -- 172.21.15.189:0/2892576620 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7fac2c003a20 con 0x7fac3006ae40 2024-04-06T12:12:00.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.551+0000 7fac25ffb700 1 --2- 172.21.15.189:0/2892576620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fac1406ac00 0x7fac1406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:00.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.551+0000 7fac25ffb700 1 -- 172.21.15.189:0/2892576620 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fac2c088260 con 0x7fac3006ae40 2024-04-06T12:12:00.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.551+0000 7fac34bba700 1 --2- 172.21.15.189:0/2892576620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fac1406ac00 0x7fac1406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:00.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.551+0000 7fac34bba700 1 --2- 172.21.15.189:0/2892576620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fac1406ac00 0x7fac1406d0c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fac3013f060 tx=0x7fac20008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:00.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.556+0000 7fac25ffb700 1 -- 172.21.15.189:0/2892576620 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fac2c053870 con 0x7fac3006ae40 2024-04-06T12:12:00.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.818+0000 7fac3761f700 1 -- 172.21.15.189:0/2892576620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac30074990 con 0x7fac3006ae40 2024-04-06T12:12:00.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.819+0000 7fac25ffb700 1 -- 172.21.15.189:0/2892576620 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3533 (secure 0 0 0) 0x7fac2c017de0 con 0x7fac3006ae40 2024-04-06T12:12:00.817 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:00.817 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2769778155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2769778155},{"type":"v1","addr":"172.21.15.189:6835","nonce":2769778155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4233,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3550531296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3550531296},{"type":"v1","addr":"172.21.15.189:6837","nonce":3550531296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3198235148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3198235148},{"type":"v1","addr":"172.21.15.192:6833","nonce":3198235148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/516212261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":516212261},{"type":"v1","addr":"172.21.15.192:6835","nonce":516212261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-04-06T12:12:00.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.821+0000 7fac3761f700 1 -- 172.21.15.189:0/2892576620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fac1406ac00 msgr2=0x7fac1406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:00.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.821+0000 7fac3761f700 1 --2- 172.21.15.189:0/2892576620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fac1406ac00 0x7fac1406d0c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fac3013f060 tx=0x7fac20008040 comp rx=0 tx=0).stop 2024-04-06T12:12:00.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.821+0000 7fac3761f700 1 -- 172.21.15.189:0/2892576620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac3006ae40 msgr2=0x7fac301314c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:00.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.821+0000 7fac3761f700 1 --2- 172.21.15.189:0/2892576620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac3006ae40 0x7fac301314c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00c4e0 tx=0x7fac2c00c7f0 comp rx=0 tx=0).stop 2024-04-06T12:12:00.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.821+0000 7fac3761f700 1 -- 172.21.15.189:0/2892576620 shutdown_connections 2024-04-06T12:12:00.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.821+0000 7fac3761f700 1 --2- 172.21.15.189:0/2892576620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fac1406ac00 0x7fac1406d0c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:00.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.821+0000 7fac3761f700 1 --2- 172.21.15.189:0/2892576620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac3015fcd0 0x7fac3015ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:00.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.821+0000 7fac3761f700 1 --2- 172.21.15.189:0/2892576620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac3006ae40 0x7fac301314c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:00.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.821+0000 7fac3761f700 1 --2- 172.21.15.189:0/2892576620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac30144f40 0x7fac300cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:00.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.821+0000 7fac3761f700 1 -- 172.21.15.189:0/2892576620 >> 172.21.15.189:0/2892576620 conn(0x7fac300d00b0 msgr2=0x7fac3015deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:00.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.821+0000 7fac3761f700 1 -- 172.21.15.189:0/2892576620 shutdown_connections 2024-04-06T12:12:00.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.821+0000 7fac3761f700 1 -- 172.21.15.189:0/2892576620 wait complete. 2024-04-06T12:12:00.820 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 3 2024-04-06T12:12:00.831 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:00.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.959+0000 7f7e56f26700 1 Processor -- start 2024-04-06T12:12:00.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.959+0000 7f7e56f26700 1 -- start start 2024-04-06T12:12:00.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.960+0000 7f7e56f26700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e5015d7b0 0x7f7e500b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:00.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.960+0000 7f7e56f26700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7e5006a730 0x7f7e5006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:00.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.960+0000 7f7e56f26700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7e5015ddf0 0x7f7e5015e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:00.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.960+0000 7f7e56f26700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5006b0f0 con 0x7f7e5006a730 2024-04-06T12:12:00.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.960+0000 7f7e56f26700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e500b3990 con 0x7f7e5015ddf0 2024-04-06T12:12:00.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.960+0000 7f7e56f26700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e500b3060 con 0x7f7e5015d7b0 2024-04-06T12:12:00.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.960+0000 7f7e4ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7e5006a730 0x7f7e5006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:00.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.960+0000 7f7e4ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7e5006a730 0x7f7e5006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44414/0 (socket says 172.21.15.189:44414) 2024-04-06T12:12:00.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.960+0000 7f7e4ffff700 1 -- 172.21.15.189:0/1824863008 learned_addr learned my addr 172.21.15.189:0/1824863008 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.960+0000 7f7e554c3700 1 --2- 172.21.15.189:0/1824863008 >> v2:172.21.15.192:3300/0 conn(0x7f7e5015ddf0 0x7f7e5015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.960+0000 7f7e54cc2700 1 --2- 172.21.15.189:0/1824863008 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e5015d7b0 0x7f7e500b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.961+0000 7f7e4ffff700 1 -- 172.21.15.189:0/1824863008 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e5015d7b0 msgr2=0x7f7e500b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.961+0000 7f7e4ffff700 1 --2- 172.21.15.189:0/1824863008 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e5015d7b0 0x7f7e500b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.961+0000 7f7e4ffff700 1 -- 172.21.15.189:0/1824863008 >> v2:172.21.15.192:3300/0 conn(0x7f7e5015ddf0 msgr2=0x7f7e5015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.961+0000 7f7e4ffff700 1 --2- 172.21.15.189:0/1824863008 >> v2:172.21.15.192:3300/0 conn(0x7f7e5015ddf0 0x7f7e5015e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.961+0000 7f7e4ffff700 1 -- 172.21.15.189:0/1824863008 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e500b3340 con 0x7f7e5006a730 2024-04-06T12:12:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.961+0000 7f7e4ffff700 1 --2- 172.21.15.189:0/1824863008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e5006a730 0x7f7e5006abb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7e500b4800 tx=0x7f7e3c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88eaccd43ab20944 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.961+0000 7f7e4f7fe700 1 -- 172.21.15.189:0/1824863008 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c011830 con 0x7f7e5006a730 2024-04-06T12:12:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.961+0000 7f7e4f7fe700 1 -- 172.21.15.189:0/1824863008 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3c011990 con 0x7f7e5006a730 2024-04-06T12:12:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.961+0000 7f7e4f7fe700 1 -- 172.21.15.189:0/1824863008 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c011d20 con 0x7f7e5006a730 2024-04-06T12:12:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.961+0000 7f7e56f26700 1 -- 172.21.15.189:0/1824863008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e5006a730 msgr2=0x7f7e5006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.961+0000 7f7e56f26700 1 --2- 172.21.15.189:0/1824863008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e5006a730 0x7f7e5006abb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7e500b4800 tx=0x7f7e3c009bd0 comp rx=0 tx=0).stop 2024-04-06T12:12:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.962+0000 7f7e56f26700 1 -- 172.21.15.189:0/1824863008 shutdown_connections 2024-04-06T12:12:00.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.962+0000 7f7e56f26700 1 --2- 172.21.15.189:0/1824863008 >> v2:172.21.15.192:3300/0 conn(0x7f7e5015ddf0 0x7f7e5015e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:00.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.962+0000 7f7e56f26700 1 --2- 172.21.15.189:0/1824863008 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e5015d7b0 0x7f7e500b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:00.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.962+0000 7f7e56f26700 1 --2- 172.21.15.189:0/1824863008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e5006a730 0x7f7e5006abb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:00.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.962+0000 7f7e56f26700 1 -- 172.21.15.189:0/1824863008 >> 172.21.15.189:0/1824863008 conn(0x7f7e500d00b0 msgr2=0x7f7e500d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:00.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.962+0000 7f7e56f26700 1 -- 172.21.15.189:0/1824863008 shutdown_connections 2024-04-06T12:12:00.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.962+0000 7f7e56f26700 1 -- 172.21.15.189:0/1824863008 wait complete. 2024-04-06T12:12:00.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.963+0000 7f7e56f26700 1 Processor -- start 2024-04-06T12:12:00.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.963+0000 7f7e56f26700 1 -- start start 2024-04-06T12:12:00.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e56f26700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e5006a730 0x7f7e50149140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:00.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e56f26700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e5015d7b0 0x7f7e5014c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:00.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e56f26700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e5015ddf0 0x7f7e501498f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:00.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e56f26700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5014fd50 con 0x7f7e5015ddf0 2024-04-06T12:12:00.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e56f26700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e50149680 con 0x7f7e5015d7b0 2024-04-06T12:12:00.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e56f26700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5014c8d0 con 0x7f7e5006a730 2024-04-06T12:12:00.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e554c3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e5015ddf0 0x7f7e501498f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:00.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e554c3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e5015ddf0 0x7f7e501498f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44416/0 (socket says 172.21.15.189:44416) 2024-04-06T12:12:00.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e554c3700 1 -- 172.21.15.189:0/3838750857 learned_addr learned my addr 172.21.15.189:0/3838750857 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:00.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e54cc2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e5006a730 0x7f7e50149140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:00.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e4ffff700 1 --2- 172.21.15.189:0/3838750857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e5015d7b0 0x7f7e5014c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:00.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e554c3700 1 -- 172.21.15.189:0/3838750857 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e5006a730 msgr2=0x7f7e50149140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:00.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e554c3700 1 --2- 172.21.15.189:0/3838750857 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e5006a730 0x7f7e50149140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:00.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e554c3700 1 -- 172.21.15.189:0/3838750857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e5015d7b0 msgr2=0x7f7e5014c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:00.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e554c3700 1 --2- 172.21.15.189:0/3838750857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e5015d7b0 0x7f7e5014c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:00.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e554c3700 1 -- 172.21.15.189:0/3838750857 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c009550 con 0x7f7e5015ddf0 2024-04-06T12:12:00.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e54cc2700 1 --2- 172.21.15.189:0/3838750857 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e5006a730 0x7f7e50149140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:00.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e4ffff700 1 --2- 172.21.15.189:0/3838750857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e5015d7b0 0x7f7e5014c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:00.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e554c3700 1 --2- 172.21.15.189:0/3838750857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e5015ddf0 0x7f7e501498f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7e4800efc0 tx=0x7f7e4800c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e4d7fa700 1 -- 172.21.15.189:0/3838750857 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4800cad0 con 0x7f7e5015ddf0 2024-04-06T12:12:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e4d7fa700 1 -- 172.21.15.189:0/3838750857 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4800f040 con 0x7f7e5015ddf0 2024-04-06T12:12:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e56f26700 1 -- 172.21.15.189:0/3838750857 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e5015b460 con 0x7f7e5015ddf0 2024-04-06T12:12:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e4d7fa700 1 -- 172.21.15.189:0/3838750857 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4800ce70 con 0x7f7e5015ddf0 2024-04-06T12:12:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.964+0000 7f7e56f26700 1 -- 172.21.15.189:0/3838750857 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e5015ba20 con 0x7f7e5015ddf0 2024-04-06T12:12:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.965+0000 7f7e56f26700 1 -- 172.21.15.189:0/3838750857 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e50067bb0 con 0x7f7e5015ddf0 2024-04-06T12:12:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.966+0000 7f7e4d7fa700 1 -- 172.21.15.189:0/3838750857 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7f7e48014480 con 0x7f7e5015ddf0 2024-04-06T12:12:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.966+0000 7f7e4d7fa700 1 --2- 172.21.15.189:0/3838750857 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e3406ac00 0x7f7e3406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.966+0000 7f7e4d7fa700 1 -- 172.21.15.189:0/3838750857 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7e48089860 con 0x7f7e5015ddf0 2024-04-06T12:12:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.966+0000 7f7e54cc2700 1 --2- 172.21.15.189:0/3838750857 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e3406ac00 0x7f7e3406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.966+0000 7f7e54cc2700 1 --2- 172.21.15.189:0/3838750857 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e3406ac00 0x7f7e3406d0c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7e500cd060 tx=0x7f7e44009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:00.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:00.971+0000 7f7e4d7fa700 1 -- 172.21.15.189:0/3838750857 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7e48054e70 con 0x7f7e5015ddf0 2024-04-06T12:12:01.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.170+0000 7f7e56f26700 1 -- 172.21.15.189:0/3838750857 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e50074990 con 0x7f7e5015ddf0 2024-04-06T12:12:01.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.221+0000 7f7e4d7fa700 1 -- 172.21.15.189:0/3838750857 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12430 (secure 0 0 0) 0x7f7e48018090 con 0x7f7e5015ddf0 2024-04-06T12:12:01.219 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:01.219 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":16,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:00.761416+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:01.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.223+0000 7f7e56f26700 1 -- 172.21.15.189:0/3838750857 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e3406ac00 msgr2=0x7f7e3406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:01.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.223+0000 7f7e56f26700 1 --2- 172.21.15.189:0/3838750857 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e3406ac00 0x7f7e3406d0c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7e500cd060 tx=0x7f7e44009210 comp rx=0 tx=0).stop 2024-04-06T12:12:01.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.223+0000 7f7e56f26700 1 -- 172.21.15.189:0/3838750857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e5015ddf0 msgr2=0x7f7e501498f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:01.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.223+0000 7f7e56f26700 1 --2- 172.21.15.189:0/3838750857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e5015ddf0 0x7f7e501498f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7e4800efc0 tx=0x7f7e4800c0c0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.223+0000 7f7e56f26700 1 -- 172.21.15.189:0/3838750857 shutdown_connections 2024-04-06T12:12:01.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.223+0000 7f7e56f26700 1 --2- 172.21.15.189:0/3838750857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e5015ddf0 0x7f7e501498f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.223+0000 7f7e56f26700 1 --2- 172.21.15.189:0/3838750857 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e5006a730 0x7f7e50149140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.223+0000 7f7e56f26700 1 --2- 172.21.15.189:0/3838750857 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e3406ac00 0x7f7e3406d0c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.223+0000 7f7e56f26700 1 --2- 172.21.15.189:0/3838750857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e5015d7b0 0x7f7e5014c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.223+0000 7f7e56f26700 1 -- 172.21.15.189:0/3838750857 >> 172.21.15.189:0/3838750857 conn(0x7f7e500d00b0 msgr2=0x7f7e500ad470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:01.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.223+0000 7f7e56f26700 1 -- 172.21.15.189:0/3838750857 shutdown_connections 2024-04-06T12:12:01.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.223+0000 7f7e56f26700 1 -- 172.21.15.189:0/3838750857 wait complete. 2024-04-06T12:12:01.234 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:12:01.234 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:12:01.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.362+0000 7f83539c6700 1 Processor -- start 2024-04-06T12:12:01.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.362+0000 7f83539c6700 1 -- start start 2024-04-06T12:12:01.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.363+0000 7f83539c6700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f834c0cd3e0 0x7f834c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:01.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.363+0000 7f83539c6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834c06bcd0 con 0x7f834c06b8c0 2024-04-06T12:12:01.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.363+0000 7f83539c6700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834c06ceb0 con 0x7f834c15d7c0 2024-04-06T12:12:01.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.363+0000 7f83539c6700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f834c0cdfa0 con 0x7f834c0cd3e0 2024-04-06T12:12:01.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.363+0000 7f8351f63700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f834c06b8c0 0x7f834c06dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58892/0 (socket says 172.21.15.189:58892) 2024-04-06T12:12:01.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.363+0000 7f8351f63700 1 -- 172.21.15.189:0/2525379084 learned_addr learned my addr 172.21.15.189:0/2525379084 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:01.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.363+0000 7f8351762700 1 --2- 172.21.15.189:0/2525379084 >> v2:172.21.15.192:3300/0 conn(0x7f834c0cd3e0 0x7f834c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:01.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.363+0000 7f8351762700 1 -- 172.21.15.189:0/2525379084 >> v1:172.21.15.192:6789/0 conn(0x7f834c15d7c0 legacy=0x7f834c06b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:01.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.363+0000 7f8351762700 1 -- 172.21.15.189:0/2525379084 >> v1:172.21.15.189:6789/0 conn(0x7f834c06b8c0 legacy=0x7f834c06dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:01.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.363+0000 7f8351762700 1 -- 172.21.15.189:0/2525379084 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834c06d750 con 0x7f834c0cd3e0 2024-04-06T12:12:01.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.363+0000 7f8351762700 1 --2- 172.21.15.189:0/2525379084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f834c0cd3e0 0x7f834c0cd7c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8340009840 tx=0x7f8340009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6079c432b29a394d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:01.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.365+0000 7f833ffff700 1 -- 172.21.15.189:0/2525379084 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8340011830 con 0x7f834c0cd3e0 2024-04-06T12:12:01.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.365+0000 7f833ffff700 1 -- 172.21.15.189:0/2525379084 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8340011990 con 0x7f834c0cd3e0 2024-04-06T12:12:01.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.365+0000 7f833ffff700 1 -- 172.21.15.189:0/2525379084 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8340011d20 con 0x7f834c0cd3e0 2024-04-06T12:12:01.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.366+0000 7f83539c6700 1 -- 172.21.15.189:0/2525379084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f834c0cd3e0 msgr2=0x7f834c0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:01.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.366+0000 7f83539c6700 1 --2- 172.21.15.189:0/2525379084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f834c0cd3e0 0x7f834c0cd7c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8340009840 tx=0x7f8340009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:01.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.366+0000 7f83539c6700 1 -- 172.21.15.189:0/2525379084 shutdown_connections 2024-04-06T12:12:01.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.366+0000 7f83539c6700 1 --2- 172.21.15.189:0/2525379084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f834c0cd3e0 0x7f834c0cd7c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.366+0000 7f83539c6700 1 -- 172.21.15.189:0/2525379084 >> 172.21.15.189:0/2525379084 conn(0x7f834c0cc290 msgr2=0x7f834c0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:01.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.366+0000 7f83539c6700 1 -- 172.21.15.189:0/2525379084 shutdown_connections 2024-04-06T12:12:01.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.366+0000 7f83539c6700 1 -- 172.21.15.189:0/2525379084 wait complete. 2024-04-06T12:12:01.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.367+0000 7f83539c6700 1 Processor -- start 2024-04-06T12:12:01.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.367+0000 7f83539c6700 1 -- start start 2024-04-06T12:12:01.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.368+0000 7f83539c6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f834c06b8c0 0x7f834c0b3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:01.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.368+0000 7f83539c6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f834c0cd3e0 0x7f834c146cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:01.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.368+0000 7f83539c6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f834c15d7c0 0x7f834c15faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:01.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.368+0000 7f83539c6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834c0c4a60 con 0x7f834c15d7c0 2024-04-06T12:12:01.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.368+0000 7f83539c6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834c1471f0 con 0x7f834c0cd3e0 2024-04-06T12:12:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.368+0000 7f83539c6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f834c15e990 con 0x7f834c06b8c0 2024-04-06T12:12:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.368+0000 7f8351f63700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f834c15d7c0 0x7f834c15faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.368+0000 7f8351762700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f834c06b8c0 0x7f834c0b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.368+0000 7f8350f61700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f834c0cd3e0 0x7f834c146cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.368+0000 7f8351762700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f834c06b8c0 0x7f834c0b3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:47266/0 (socket says 172.21.15.189:47266) 2024-04-06T12:12:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.368+0000 7f8350f61700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f834c0cd3e0 0x7f834c146cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:52872/0 (socket says 172.21.15.189:52872) 2024-04-06T12:12:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.368+0000 7f8351762700 1 -- 172.21.15.189:0/303183085 learned_addr learned my addr 172.21.15.189:0/303183085 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.368+0000 7f8350f61700 1 -- 172.21.15.189:0/303183085 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f834c06b8c0 msgr2=0x7f834c0b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.368+0000 7f8350f61700 1 --2- 172.21.15.189:0/303183085 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f834c06b8c0 0x7f834c0b3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.368+0000 7f8350f61700 1 -- 172.21.15.189:0/303183085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f834c15d7c0 msgr2=0x7f834c15faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.368+0000 7f8350f61700 1 --2- 172.21.15.189:0/303183085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f834c15d7c0 0x7f834c15faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.368+0000 7f8350f61700 1 -- 172.21.15.189:0/303183085 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83400094a0 con 0x7f834c0cd3e0 2024-04-06T12:12:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.368+0000 7f8351f63700 1 --2- 172.21.15.189:0/303183085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f834c15d7c0 0x7f834c15faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.369+0000 7f8351762700 1 --2- 172.21.15.189:0/303183085 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f834c06b8c0 0x7f834c0b3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:01.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.369+0000 7f8350f61700 1 --2- 172.21.15.189:0/303183085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f834c0cd3e0 0x7f834c146cb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8348000f60 tx=0x7f834800c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:01.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.369+0000 7f833dffb700 1 -- 172.21.15.189:0/303183085 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83480052d0 con 0x7f834c0cd3e0 2024-04-06T12:12:01.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.369+0000 7f833dffb700 1 -- 172.21.15.189:0/303183085 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8348005430 con 0x7f834c0cd3e0 2024-04-06T12:12:01.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.369+0000 7f833dffb700 1 -- 172.21.15.189:0/303183085 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83480038c0 con 0x7f834c0cd3e0 2024-04-06T12:12:01.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.369+0000 7f83539c6700 1 -- 172.21.15.189:0/303183085 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f834c15ec70 con 0x7f834c0cd3e0 2024-04-06T12:12:01.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.369+0000 7f83539c6700 1 -- 172.21.15.189:0/303183085 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f834c0c46f0 con 0x7f834c0cd3e0 2024-04-06T12:12:01.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.370+0000 7f83539c6700 1 -- 172.21.15.189:0/303183085 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f834c0c48b0 con 0x7f834c0cd3e0 2024-04-06T12:12:01.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.370+0000 7f833dffb700 1 -- 172.21.15.189:0/303183085 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7f8348003c30 con 0x7f834c0cd3e0 2024-04-06T12:12:01.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.371+0000 7f833dffb700 1 --2- 172.21.15.189:0/303183085 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f833006ac00 0x7f833006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:01.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.371+0000 7f833dffb700 1 -- 172.21.15.189:0/303183085 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8348088420 con 0x7f834c0cd3e0 2024-04-06T12:12:01.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.371+0000 7f8351762700 1 --2- 172.21.15.189:0/303183085 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f833006ac00 0x7f833006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:01.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.371+0000 7f8351762700 1 --2- 172.21.15.189:0/303183085 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f833006ac00 0x7f833006d0c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8340009b90 tx=0x7f8340008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:01.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.377+0000 7f833dffb700 1 -- 172.21.15.189:0/303183085 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8348053910 con 0x7f834c0cd3e0 2024-04-06T12:12:01.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.570+0000 7f83539c6700 1 -- 172.21.15.189:0/303183085 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f834c0b0610 con 0x7f834c0cd3e0 2024-04-06T12:12:01.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.571+0000 7f833dffb700 1 -- 172.21.15.189:0/303183085 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8348056f30 con 0x7f834c0cd3e0 2024-04-06T12:12:01.569 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:12:01.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.573+0000 7f83539c6700 1 -- 172.21.15.189:0/303183085 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f833006ac00 msgr2=0x7f833006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:01.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.573+0000 7f83539c6700 1 --2- 172.21.15.189:0/303183085 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f833006ac00 0x7f833006d0c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8340009b90 tx=0x7f8340008030 comp rx=0 tx=0).stop 2024-04-06T12:12:01.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.573+0000 7f83539c6700 1 -- 172.21.15.189:0/303183085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f834c0cd3e0 msgr2=0x7f834c146cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:01.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.573+0000 7f83539c6700 1 --2- 172.21.15.189:0/303183085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f834c0cd3e0 0x7f834c146cb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8348000f60 tx=0x7f834800c6e0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.573+0000 7f83539c6700 1 -- 172.21.15.189:0/303183085 shutdown_connections 2024-04-06T12:12:01.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.573+0000 7f83539c6700 1 --2- 172.21.15.189:0/303183085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f834c15d7c0 0x7f834c15faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.573+0000 7f83539c6700 1 --2- 172.21.15.189:0/303183085 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f834c06b8c0 0x7f834c0b3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.573+0000 7f83539c6700 1 --2- 172.21.15.189:0/303183085 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f833006ac00 0x7f833006d0c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.573+0000 7f83539c6700 1 --2- 172.21.15.189:0/303183085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f834c0cd3e0 0x7f834c146cb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.573+0000 7f83539c6700 1 -- 172.21.15.189:0/303183085 >> 172.21.15.189:0/303183085 conn(0x7f834c0cc290 msgr2=0x7f834c0b0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:01.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.573+0000 7f83539c6700 1 -- 172.21.15.189:0/303183085 shutdown_connections 2024-04-06T12:12:01.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.573+0000 7f83539c6700 1 -- 172.21.15.189:0/303183085 wait complete. 2024-04-06T12:12:01.584 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-06T12:12:01.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.709+0000 7fe47051a700 1 Processor -- start 2024-04-06T12:12:01.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.709+0000 7fe47051a700 1 -- start start 2024-04-06T12:12:01.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.710+0000 7fe47051a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe4680b4af0 0x7fe4680b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:01.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.710+0000 7fe47051a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4680cdbe0 con 0x7fe46815a550 2024-04-06T12:12:01.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.710+0000 7fe47051a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe46815a940 con 0x7fe4680b4660 2024-04-06T12:12:01.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.710+0000 7fe47051a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe46815ac00 con 0x7fe4680b4af0 2024-04-06T12:12:01.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.710+0000 7fe46eab7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe4680b4af0 0x7fe4680b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:01.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.710+0000 7fe46e2b6700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe46815a550 0x7fe4680cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58896/0 (socket says 172.21.15.189:58896) 2024-04-06T12:12:01.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.710+0000 7fe46e2b6700 1 -- 172.21.15.189:0/3196738228 learned_addr learned my addr 172.21.15.189:0/3196738228 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:01.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.711+0000 7fe46eab7700 1 -- 172.21.15.189:0/3196738228 >> v1:172.21.15.192:6789/0 conn(0x7fe4680b4660 legacy=0x7fe4680b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:01.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.711+0000 7fe46eab7700 1 -- 172.21.15.189:0/3196738228 >> v1:172.21.15.189:6789/0 conn(0x7fe46815a550 legacy=0x7fe4680cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:01.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.711+0000 7fe46eab7700 1 -- 172.21.15.189:0/3196738228 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe46815aee0 con 0x7fe4680b4af0 2024-04-06T12:12:01.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.711+0000 7fe46d2b4700 1 -- 172.21.15.189:0/3196738228 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848078183 0 0) 0x7fe4680cdbe0 con 0x7fe46815a550 2024-04-06T12:12:01.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.711+0000 7fe46eab7700 1 --2- 172.21.15.189:0/3196738228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4680b4af0 0x7fe4680b36d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe460009d90 tx=0x7fe46000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f33bfa1a01a40d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:01.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.711+0000 7fe46d2b4700 1 -- 172.21.15.189:0/3196738228 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46000e070 con 0x7fe4680b4af0 2024-04-06T12:12:01.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.711+0000 7fe46d2b4700 1 -- 172.21.15.189:0/3196738228 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46000bbd0 con 0x7fe4680b4af0 2024-04-06T12:12:01.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.711+0000 7fe46d2b4700 1 -- 172.21.15.189:0/3196738228 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe460012430 con 0x7fe4680b4af0 2024-04-06T12:12:01.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.711+0000 7fe47051a700 1 -- 172.21.15.189:0/3196738228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4680b4af0 msgr2=0x7fe4680b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:01.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.711+0000 7fe47051a700 1 --2- 172.21.15.189:0/3196738228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4680b4af0 0x7fe4680b36d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe460009d90 tx=0x7fe46000b330 comp rx=0 tx=0).stop 2024-04-06T12:12:01.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.712+0000 7fe47051a700 1 -- 172.21.15.189:0/3196738228 shutdown_connections 2024-04-06T12:12:01.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.712+0000 7fe47051a700 1 --2- 172.21.15.189:0/3196738228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4680b4af0 0x7fe4680b36d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.712+0000 7fe47051a700 1 -- 172.21.15.189:0/3196738228 >> 172.21.15.189:0/3196738228 conn(0x7fe4680d00b0 msgr2=0x7fe4680d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:01.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.712+0000 7fe47051a700 1 -- 172.21.15.189:0/3196738228 shutdown_connections 2024-04-06T12:12:01.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.712+0000 7fe47051a700 1 -- 172.21.15.189:0/3196738228 wait complete. 2024-04-06T12:12:01.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.713+0000 7fe47051a700 1 Processor -- start 2024-04-06T12:12:01.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.713+0000 7fe47051a700 1 -- start start 2024-04-06T12:12:01.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.713+0000 7fe47051a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4680b4660 0x7fe46806a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:01.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.713+0000 7fe47051a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe4680b4af0 0x7fe468155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:01.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.713+0000 7fe47051a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe468155fa0 0x7fe46814fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:01.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.713+0000 7fe47051a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4681570c0 con 0x7fe4680b4660 2024-04-06T12:12:01.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.713+0000 7fe47051a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe468156440 con 0x7fe468155fa0 2024-04-06T12:12:01.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.713+0000 7fe47051a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe468156720 con 0x7fe4680b4af0 2024-04-06T12:12:01.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.714+0000 7fe46e2b6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4680b4660 0x7fe46806a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:01.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.714+0000 7fe46e2b6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4680b4660 0x7fe46806a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44464/0 (socket says 172.21.15.189:44464) 2024-04-06T12:12:01.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.714+0000 7fe46e2b6700 1 -- 172.21.15.189:0/970409928 learned_addr learned my addr 172.21.15.189:0/970409928 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:01.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.714+0000 7fe46eab7700 1 --2- 172.21.15.189:0/970409928 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe468155fa0 0x7fe46814fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:01.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.714+0000 7fe46dab5700 1 --2- 172.21.15.189:0/970409928 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe4680b4af0 0x7fe468155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:01.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.714+0000 7fe46e2b6700 1 -- 172.21.15.189:0/970409928 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe4680b4af0 msgr2=0x7fe468155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:01.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.714+0000 7fe46e2b6700 1 --2- 172.21.15.189:0/970409928 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe4680b4af0 0x7fe468155a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.714+0000 7fe46e2b6700 1 -- 172.21.15.189:0/970409928 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe468155fa0 msgr2=0x7fe46814fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:01.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.714+0000 7fe46e2b6700 1 --2- 172.21.15.189:0/970409928 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe468155fa0 0x7fe46814fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.714+0000 7fe46e2b6700 1 -- 172.21.15.189:0/970409928 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe460009a70 con 0x7fe4680b4660 2024-04-06T12:12:01.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.714+0000 7fe46e2b6700 1 --2- 172.21.15.189:0/970409928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4680b4660 0x7fe46806a9a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe46400c520 tx=0x7fe46400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:01.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.714+0000 7fe45effd700 1 -- 172.21.15.189:0/970409928 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe464004c80 con 0x7fe4680b4660 2024-04-06T12:12:01.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.714+0000 7fe45effd700 1 -- 172.21.15.189:0/970409928 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe464004de0 con 0x7fe4680b4660 2024-04-06T12:12:01.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.714+0000 7fe47051a700 1 -- 172.21.15.189:0/970409928 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe468156a00 con 0x7fe4680b4660 2024-04-06T12:12:01.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.714+0000 7fe45effd700 1 -- 172.21.15.189:0/970409928 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe464007310 con 0x7fe4680b4660 2024-04-06T12:12:01.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.714+0000 7fe47051a700 1 -- 172.21.15.189:0/970409928 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4680b1750 con 0x7fe4680b4660 2024-04-06T12:12:01.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.715+0000 7fe47051a700 1 -- 172.21.15.189:0/970409928 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4680b1940 con 0x7fe4680b4660 2024-04-06T12:12:01.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.715+0000 7fe45effd700 1 -- 172.21.15.189:0/970409928 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe464007470 con 0x7fe4680b4660 2024-04-06T12:12:01.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.716+0000 7fe45effd700 1 --2- 172.21.15.189:0/970409928 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe44c06ac90 0x7fe44c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:01.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.716+0000 7fe45effd700 1 -- 172.21.15.189:0/970409928 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe464088ae0 con 0x7fe4680b4660 2024-04-06T12:12:01.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.716+0000 7fe46dab5700 1 --2- 172.21.15.189:0/970409928 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe44c06ac90 0x7fe44c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:01.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.716+0000 7fe46dab5700 1 --2- 172.21.15.189:0/970409928 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe44c06ac90 0x7fe44c06d150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe458007fd0 tx=0x7fe458009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:01.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.721+0000 7fe45effd700 1 -- 172.21.15.189:0/970409928 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe4640540a0 con 0x7fe4680b4660 2024-04-06T12:12:01.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.917+0000 7fe47051a700 1 -- 172.21.15.189:0/970409928 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe468067bb0 con 0x7fe4680b4660 2024-04-06T12:12:01.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.918+0000 7fe45effd700 1 -- 172.21.15.189:0/970409928 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe4640576c0 con 0x7fe4680b4660 2024-04-06T12:12:01.916 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:01.916 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T12:12:01.916 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:12:01.916 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 1, 2024-04-06T12:12:01.916 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": ".mgr" 2024-04-06T12:12:01.916 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:12:01.916 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T12:12:01.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.920+0000 7fe47051a700 1 -- 172.21.15.189:0/970409928 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe44c06ac90 msgr2=0x7fe44c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:01.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.920+0000 7fe47051a700 1 --2- 172.21.15.189:0/970409928 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe44c06ac90 0x7fe44c06d150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe458007fd0 tx=0x7fe458009040 comp rx=0 tx=0).stop 2024-04-06T12:12:01.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.920+0000 7fe47051a700 1 -- 172.21.15.189:0/970409928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4680b4660 msgr2=0x7fe46806a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:01.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.920+0000 7fe47051a700 1 --2- 172.21.15.189:0/970409928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4680b4660 0x7fe46806a9a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe46400c520 tx=0x7fe46400c830 comp rx=0 tx=0).stop 2024-04-06T12:12:01.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.920+0000 7fe47051a700 1 -- 172.21.15.189:0/970409928 shutdown_connections 2024-04-06T12:12:01.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.920+0000 7fe47051a700 1 --2- 172.21.15.189:0/970409928 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe44c06ac90 0x7fe44c06d150 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.920+0000 7fe47051a700 1 --2- 172.21.15.189:0/970409928 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe468155fa0 0x7fe46814fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.920+0000 7fe47051a700 1 --2- 172.21.15.189:0/970409928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4680b4660 0x7fe46806a9a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.920+0000 7fe47051a700 1 --2- 172.21.15.189:0/970409928 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe4680b4af0 0x7fe468155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:01.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.920+0000 7fe47051a700 1 -- 172.21.15.189:0/970409928 >> 172.21.15.189:0/970409928 conn(0x7fe4680d00b0 msgr2=0x7fe4680ce630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:01.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.920+0000 7fe47051a700 1 -- 172.21.15.189:0/970409928 shutdown_connections 2024-04-06T12:12:01.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:01.920+0000 7fe47051a700 1 -- 172.21.15.189:0/970409928 wait complete. 2024-04-06T12:12:01.930 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-06T12:12:01.930 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-06T12:12:02.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.058+0000 7f409f9d7700 1 Processor -- start 2024-04-06T12:12:02.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.058+0000 7f409f9d7700 1 -- start start 2024-04-06T12:12:02.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.059+0000 7f409f9d7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f409815e8a0 0x7f40980cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:02.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.059+0000 7f409f9d7700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f409815eee0 0x7f409815f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:02.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.059+0000 7f409f9d7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f409815fcd0 con 0x7f409815f8a0 2024-04-06T12:12:02.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.059+0000 7f409f9d7700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40980669d0 con 0x7f409815e8a0 2024-04-06T12:12:02.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.059+0000 7f409f9d7700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f409806ede0 con 0x7f409815eee0 2024-04-06T12:12:02.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.059+0000 7f409d773700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f409815e8a0 0x7f40980cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:02.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.059+0000 7f409df74700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f409815f8a0 0x7f409806ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58912/0 (socket says 172.21.15.189:58912) 2024-04-06T12:12:02.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.059+0000 7f409df74700 1 -- 172.21.15.189:0/1222774897 learned_addr learned my addr 172.21.15.189:0/1222774897 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:02.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.059+0000 7f409cf72700 1 --2- 172.21.15.189:0/1222774897 >> v2:172.21.15.192:3300/0 conn(0x7f409815eee0 0x7f409815f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:02.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.059+0000 7f409d773700 1 -- 172.21.15.189:0/1222774897 >> v2:172.21.15.192:3300/0 conn(0x7f409815eee0 msgr2=0x7f409815f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:02.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.059+0000 7f409d773700 1 --2- 172.21.15.189:0/1222774897 >> v2:172.21.15.192:3300/0 conn(0x7f409815eee0 0x7f409815f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:02.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.059+0000 7f409d773700 1 -- 172.21.15.189:0/1222774897 >> v1:172.21.15.189:6789/0 conn(0x7f409815f8a0 legacy=0x7f409806ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:02.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.059+0000 7f408ffff700 1 -- 172.21.15.189:0/1222774897 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204635756 0 0) 0x7f4090001490 con 0x7f409815f8a0 2024-04-06T12:12:02.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.059+0000 7f409d773700 1 -- 172.21.15.189:0/1222774897 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409815ec80 con 0x7f409815e8a0 2024-04-06T12:12:02.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.059+0000 7f409d773700 1 --2- 172.21.15.189:0/1222774897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f409815e8a0 0x7f40980cbc20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f40980723a0 tx=0x7f4094009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b840848fc67ecbd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:02.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.059+0000 7f408ffff700 1 -- 172.21.15.189:0/1222774897 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4094011830 con 0x7f409815e8a0 2024-04-06T12:12:02.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.059+0000 7f408ffff700 1 -- 172.21.15.189:0/1222774897 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4094011990 con 0x7f409815e8a0 2024-04-06T12:12:02.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.059+0000 7f408ffff700 1 -- 172.21.15.189:0/1222774897 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4094011d20 con 0x7f409815e8a0 2024-04-06T12:12:02.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.060+0000 7f409f9d7700 1 -- 172.21.15.189:0/1222774897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f409815e8a0 msgr2=0x7f40980cbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:02.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.060+0000 7f409f9d7700 1 --2- 172.21.15.189:0/1222774897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f409815e8a0 0x7f40980cbc20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f40980723a0 tx=0x7f4094009b20 comp rx=0 tx=0).stop 2024-04-06T12:12:02.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.060+0000 7f409f9d7700 1 -- 172.21.15.189:0/1222774897 shutdown_connections 2024-04-06T12:12:02.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.060+0000 7f409f9d7700 1 --2- 172.21.15.189:0/1222774897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f409815e8a0 0x7f40980cbc20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:02.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.060+0000 7f409f9d7700 1 --2- 172.21.15.189:0/1222774897 >> v2:172.21.15.192:3300/0 conn(0x7f409815eee0 0x7f409815f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:02.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.060+0000 7f409f9d7700 1 -- 172.21.15.189:0/1222774897 >> 172.21.15.189:0/1222774897 conn(0x7f40980cc290 msgr2=0x7f40980cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:02.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.060+0000 7f409f9d7700 1 -- 172.21.15.189:0/1222774897 shutdown_connections 2024-04-06T12:12:02.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.060+0000 7f409f9d7700 1 -- 172.21.15.189:0/1222774897 wait complete. 2024-04-06T12:12:02.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.062+0000 7f409f9d7700 1 Processor -- start 2024-04-06T12:12:02.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.062+0000 7f409f9d7700 1 -- start start 2024-04-06T12:12:02.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.062+0000 7f409f9d7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f409815e8a0 0x7f40981f1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:02.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.062+0000 7f409f9d7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f409815eee0 0x7f40980af810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:02.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.062+0000 7f409f9d7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f409815f8a0 0x7f40980afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:02.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.062+0000 7f409f9d7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40981565c0 con 0x7f409815f8a0 2024-04-06T12:12:02.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.062+0000 7f409f9d7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40981f0ca0 con 0x7f409815eee0 2024-04-06T12:12:02.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.063+0000 7f409f9d7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40980c5810 con 0x7f409815e8a0 2024-04-06T12:12:02.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.063+0000 7f409df74700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f409815f8a0 0x7f40980afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:02.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.063+0000 7f409df74700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f409815f8a0 0x7f40980afd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44482/0 (socket says 172.21.15.189:44482) 2024-04-06T12:12:02.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.063+0000 7f409df74700 1 -- 172.21.15.189:0/995638588 learned_addr learned my addr 172.21.15.189:0/995638588 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:02.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.063+0000 7f409d773700 1 --2- 172.21.15.189:0/995638588 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f409815e8a0 0x7f40981f1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:02.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.063+0000 7f409cf72700 1 --2- 172.21.15.189:0/995638588 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f409815eee0 0x7f40980af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:02.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.063+0000 7f409df74700 1 -- 172.21.15.189:0/995638588 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f409815e8a0 msgr2=0x7f40981f1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:02.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.063+0000 7f409df74700 1 --2- 172.21.15.189:0/995638588 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f409815e8a0 0x7f40981f1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:02.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.063+0000 7f409df74700 1 -- 172.21.15.189:0/995638588 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f409815eee0 msgr2=0x7f40980af810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:02.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.063+0000 7f409df74700 1 --2- 172.21.15.189:0/995638588 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f409815eee0 0x7f40980af810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:02.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.063+0000 7f409df74700 1 -- 172.21.15.189:0/995638588 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40940094a0 con 0x7f409815f8a0 2024-04-06T12:12:02.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.063+0000 7f409df74700 1 --2- 172.21.15.189:0/995638588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f409815f8a0 0x7f40980afd50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f409000cba0 tx=0x7f409000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:02.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.063+0000 7f408dffb700 1 -- 172.21.15.189:0/995638588 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409000ab80 con 0x7f409815f8a0 2024-04-06T12:12:02.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.063+0000 7f408dffb700 1 -- 172.21.15.189:0/995638588 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409000ad00 con 0x7f409815f8a0 2024-04-06T12:12:02.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.063+0000 7f409f9d7700 1 -- 172.21.15.189:0/995638588 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40980c5af0 con 0x7f409815f8a0 2024-04-06T12:12:02.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.063+0000 7f408dffb700 1 -- 172.21.15.189:0/995638588 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4090007310 con 0x7f409815f8a0 2024-04-06T12:12:02.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.063+0000 7f409f9d7700 1 -- 172.21.15.189:0/995638588 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40980c60b0 con 0x7f409815f8a0 2024-04-06T12:12:02.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.064+0000 7f409f9d7700 1 -- 172.21.15.189:0/995638588 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4098072140 con 0x7f409815f8a0 2024-04-06T12:12:02.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.065+0000 7f408dffb700 1 -- 172.21.15.189:0/995638588 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4090007470 con 0x7f409815f8a0 2024-04-06T12:12:02.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.065+0000 7f408dffb700 1 --2- 172.21.15.189:0/995638588 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f407c06ac90 0x7f407c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:02.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.065+0000 7f408dffb700 1 -- 172.21.15.189:0/995638588 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4090088bf0 con 0x7f409815f8a0 2024-04-06T12:12:02.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.065+0000 7f409d773700 1 --2- 172.21.15.189:0/995638588 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f407c06ac90 0x7f407c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:02.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.066+0000 7f409d773700 1 --2- 172.21.15.189:0/995638588 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f407c06ac90 0x7f407c06d150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f40940090b0 tx=0x7f4094008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:02.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.070+0000 7f408dffb700 1 -- 172.21.15.189:0/995638588 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f40900541b0 con 0x7f409815f8a0 2024-04-06T12:12:02.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.262+0000 7f409f9d7700 1 -- 172.21.15.189:0/995638588 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f409815fcd0 con 0x7f409815f8a0 2024-04-06T12:12:02.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.782+0000 7f408dffb700 1 -- 172.21.15.189:0/995638588 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f409815fcd0 con 0x7f409815f8a0 2024-04-06T12:12:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.786+0000 7f409f9d7700 1 -- 172.21.15.189:0/995638588 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f407c06ac90 msgr2=0x7f407c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.786+0000 7f409f9d7700 1 --2- 172.21.15.189:0/995638588 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f407c06ac90 0x7f407c06d150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f40940090b0 tx=0x7f4094008030 comp rx=0 tx=0).stop 2024-04-06T12:12:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.786+0000 7f409f9d7700 1 -- 172.21.15.189:0/995638588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f409815f8a0 msgr2=0x7f40980afd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.786+0000 7f409f9d7700 1 --2- 172.21.15.189:0/995638588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f409815f8a0 0x7f40980afd50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f409000cba0 tx=0x7f409000cf60 comp rx=0 tx=0).stop 2024-04-06T12:12:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.786+0000 7f409f9d7700 1 -- 172.21.15.189:0/995638588 shutdown_connections 2024-04-06T12:12:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.786+0000 7f409f9d7700 1 --2- 172.21.15.189:0/995638588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f409815f8a0 0x7f40980afd50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.786+0000 7f409f9d7700 1 --2- 172.21.15.189:0/995638588 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f409815e8a0 0x7f40981f1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.786+0000 7f409f9d7700 1 --2- 172.21.15.189:0/995638588 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f407c06ac90 0x7f407c06d150 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:02.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.786+0000 7f409f9d7700 1 --2- 172.21.15.189:0/995638588 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f409815eee0 0x7f40980af810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:02.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.786+0000 7f409f9d7700 1 -- 172.21.15.189:0/995638588 >> 172.21.15.189:0/995638588 conn(0x7f40980cc290 msgr2=0x7f409815c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:02.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.786+0000 7f409f9d7700 1 -- 172.21.15.189:0/995638588 shutdown_connections 2024-04-06T12:12:02.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.786+0000 7f409f9d7700 1 -- 172.21.15.189:0/995638588 wait complete. 2024-04-06T12:12:02.788 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' created 2024-04-06T12:12:02.799 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-06T12:12:02.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.930+0000 7fd356f81700 1 Processor -- start 2024-04-06T12:12:02.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.930+0000 7fd356f81700 1 -- start start 2024-04-06T12:12:02.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.931+0000 7fd356f81700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd3500cd3e0 0x7fd3500cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:02.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.931+0000 7fd356f81700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd35006b8c0 0x7fd35006dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:02.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.931+0000 7fd356f81700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35006bcd0 con 0x7fd35015d7c0 2024-04-06T12:12:02.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.931+0000 7fd356f81700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3500cdfa0 con 0x7fd35006b8c0 2024-04-06T12:12:02.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.931+0000 7fd356f81700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd35006ceb0 con 0x7fd3500cd3e0 2024-04-06T12:12:02.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.931+0000 7fd35551e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd35006b8c0 0x7fd35006dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:02.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.931+0000 7fd34ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd35015d7c0 0x7fd35006b1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58926/0 (socket says 172.21.15.189:58926) 2024-04-06T12:12:02.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.931+0000 7fd34ffff700 1 -- 172.21.15.189:0/1081197541 learned_addr learned my addr 172.21.15.189:0/1081197541 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:02.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.932+0000 7fd354d1d700 1 --2- 172.21.15.189:0/1081197541 >> v2:172.21.15.192:3300/0 conn(0x7fd3500cd3e0 0x7fd3500cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:02.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.932+0000 7fd35551e700 1 -- 172.21.15.189:0/1081197541 >> v2:172.21.15.192:3300/0 conn(0x7fd3500cd3e0 msgr2=0x7fd3500cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:02.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.932+0000 7fd35551e700 1 --2- 172.21.15.189:0/1081197541 >> v2:172.21.15.192:3300/0 conn(0x7fd3500cd3e0 0x7fd3500cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:02.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.932+0000 7fd35551e700 1 -- 172.21.15.189:0/1081197541 >> v1:172.21.15.189:6789/0 conn(0x7fd35015d7c0 legacy=0x7fd35006b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:02.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.932+0000 7fd35551e700 1 -- 172.21.15.189:0/1081197541 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35006d750 con 0x7fd35006b8c0 2024-04-06T12:12:02.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.932+0000 7fd34f7fe700 1 -- 172.21.15.189:0/1081197541 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832362760 0 0) 0x7fd35006bcd0 con 0x7fd35015d7c0 2024-04-06T12:12:02.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.932+0000 7fd35551e700 1 --2- 172.21.15.189:0/1081197541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd35006b8c0 0x7fd35006dbe0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd3500723a0 tx=0x7fd34800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9465e71f0ae5dc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:02.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.932+0000 7fd34f7fe700 1 -- 172.21.15.189:0/1081197541 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34800e070 con 0x7fd35006b8c0 2024-04-06T12:12:02.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.932+0000 7fd34f7fe700 1 -- 172.21.15.189:0/1081197541 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34800bbd0 con 0x7fd35006b8c0 2024-04-06T12:12:02.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.932+0000 7fd34f7fe700 1 -- 172.21.15.189:0/1081197541 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348012430 con 0x7fd35006b8c0 2024-04-06T12:12:02.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.932+0000 7fd356f81700 1 -- 172.21.15.189:0/1081197541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd35006b8c0 msgr2=0x7fd35006dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:02.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.932+0000 7fd356f81700 1 --2- 172.21.15.189:0/1081197541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd35006b8c0 0x7fd35006dbe0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd3500723a0 tx=0x7fd34800b330 comp rx=0 tx=0).stop 2024-04-06T12:12:02.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.932+0000 7fd356f81700 1 -- 172.21.15.189:0/1081197541 shutdown_connections 2024-04-06T12:12:02.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.932+0000 7fd356f81700 1 --2- 172.21.15.189:0/1081197541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd35006b8c0 0x7fd35006dbe0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:02.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.932+0000 7fd356f81700 1 --2- 172.21.15.189:0/1081197541 >> v2:172.21.15.192:3300/0 conn(0x7fd3500cd3e0 0x7fd3500cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:02.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.932+0000 7fd356f81700 1 -- 172.21.15.189:0/1081197541 >> 172.21.15.189:0/1081197541 conn(0x7fd3500cc290 msgr2=0x7fd3500cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:02.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.933+0000 7fd356f81700 1 -- 172.21.15.189:0/1081197541 shutdown_connections 2024-04-06T12:12:02.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.933+0000 7fd356f81700 1 -- 172.21.15.189:0/1081197541 wait complete. 2024-04-06T12:12:02.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.934+0000 7fd356f81700 1 Processor -- start 2024-04-06T12:12:02.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.934+0000 7fd356f81700 1 -- start start 2024-04-06T12:12:02.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd356f81700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd35006b8c0 0x7fd3500b4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:02.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd356f81700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3500cd3e0 0x7fd350160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:02.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd356f81700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd35015d7c0 0x7fd350156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:02.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd356f81700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3500b4930 con 0x7fd35006b8c0 2024-04-06T12:12:02.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd356f81700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3500b4c10 con 0x7fd35015d7c0 2024-04-06T12:12:02.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd356f81700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3500b4ef0 con 0x7fd3500cd3e0 2024-04-06T12:12:02.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd354d1d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd35006b8c0 0x7fd3500b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:02.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd354d1d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd35006b8c0 0x7fd3500b4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44508/0 (socket says 172.21.15.189:44508) 2024-04-06T12:12:02.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd354d1d700 1 -- 172.21.15.189:0/3342900279 learned_addr learned my addr 172.21.15.189:0/3342900279 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:02.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd35551e700 1 --2- 172.21.15.189:0/3342900279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd35015d7c0 0x7fd350156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:02.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd34ffff700 1 --2- 172.21.15.189:0/3342900279 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3500cd3e0 0x7fd350160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:02.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd354d1d700 1 -- 172.21.15.189:0/3342900279 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3500cd3e0 msgr2=0x7fd350160560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:02.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd354d1d700 1 --2- 172.21.15.189:0/3342900279 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3500cd3e0 0x7fd350160560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:02.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd354d1d700 1 -- 172.21.15.189:0/3342900279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd35015d7c0 msgr2=0x7fd350156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:02.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd354d1d700 1 --2- 172.21.15.189:0/3342900279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd35015d7c0 0x7fd350156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:02.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd354d1d700 1 -- 172.21.15.189:0/3342900279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd348009a70 con 0x7fd35006b8c0 2024-04-06T12:12:02.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd34ffff700 1 --2- 172.21.15.189:0/3342900279 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3500cd3e0 0x7fd350160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:02.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd35551e700 1 --2- 172.21.15.189:0/3342900279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd35015d7c0 0x7fd350156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:02.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd354d1d700 1 --2- 172.21.15.189:0/3342900279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd35006b8c0 0x7fd3500b4310 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd34400c7f0 tx=0x7fd34400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:02.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd34d7fa700 1 -- 172.21.15.189:0/3342900279 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd344011800 con 0x7fd35006b8c0 2024-04-06T12:12:02.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd34d7fa700 1 -- 172.21.15.189:0/3342900279 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd344011960 con 0x7fd35006b8c0 2024-04-06T12:12:02.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd34d7fa700 1 -- 172.21.15.189:0/3342900279 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd344011cf0 con 0x7fd35006b8c0 2024-04-06T12:12:02.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd356f81700 1 -- 172.21.15.189:0/3342900279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35015dc40 con 0x7fd35006b8c0 2024-04-06T12:12:02.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.935+0000 7fd356f81700 1 -- 172.21.15.189:0/3342900279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3500b51d0 con 0x7fd35006b8c0 2024-04-06T12:12:02.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.936+0000 7fd356f81700 1 -- 172.21.15.189:0/3342900279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd350074990 con 0x7fd35006b8c0 2024-04-06T12:12:02.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.938+0000 7fd34d7fa700 1 -- 172.21.15.189:0/3342900279 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd344024480 con 0x7fd35006b8c0 2024-04-06T12:12:02.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.938+0000 7fd34d7fa700 1 --2- 172.21.15.189:0/3342900279 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd33406ac90 0x7fd33406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:02.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.938+0000 7fd34d7fa700 1 -- 172.21.15.189:0/3342900279 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fd344089510 con 0x7fd35006b8c0 2024-04-06T12:12:02.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.938+0000 7fd34ffff700 1 --2- 172.21.15.189:0/3342900279 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd33406ac90 0x7fd33406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:02.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.938+0000 7fd34ffff700 1 --2- 172.21.15.189:0/3342900279 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd33406ac90 0x7fd33406d150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd33c000f60 tx=0x7fd33c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:02.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:02.942+0000 7fd34d7fa700 1 -- 172.21.15.189:0/3342900279 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd344054960 con 0x7fd35006b8c0 2024-04-06T12:12:03.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.138+0000 7fd356f81700 1 -- 172.21.15.189:0/3342900279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7fd3500b5790 con 0x7fd35006b8c0 2024-04-06T12:12:03.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.785+0000 7fd34d7fa700 1 -- 172.21.15.189:0/3342900279 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) v1 ==== 171+0+0 (secure 0 0 0) 0x7fd34401f0d0 con 0x7fd35006b8c0 2024-04-06T12:12:03.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.791+0000 7fd356f81700 1 -- 172.21.15.189:0/3342900279 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd33406ac90 msgr2=0x7fd33406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:03.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.791+0000 7fd356f81700 1 --2- 172.21.15.189:0/3342900279 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd33406ac90 0x7fd33406d150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd33c000f60 tx=0x7fd33c009040 comp rx=0 tx=0).stop 2024-04-06T12:12:03.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.791+0000 7fd356f81700 1 -- 172.21.15.189:0/3342900279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd35006b8c0 msgr2=0x7fd3500b4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:03.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.791+0000 7fd356f81700 1 --2- 172.21.15.189:0/3342900279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd35006b8c0 0x7fd3500b4310 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd34400c7f0 tx=0x7fd34400cbb0 comp rx=0 tx=0).stop 2024-04-06T12:12:03.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.792+0000 7fd356f81700 1 -- 172.21.15.189:0/3342900279 shutdown_connections 2024-04-06T12:12:03.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.792+0000 7fd356f81700 1 --2- 172.21.15.189:0/3342900279 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd33406ac90 0x7fd33406d150 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:03.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.792+0000 7fd356f81700 1 --2- 172.21.15.189:0/3342900279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd35015d7c0 0x7fd350156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:03.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.792+0000 7fd356f81700 1 --2- 172.21.15.189:0/3342900279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd35006b8c0 0x7fd3500b4310 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:03.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.792+0000 7fd356f81700 1 --2- 172.21.15.189:0/3342900279 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3500cd3e0 0x7fd350160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:03.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.792+0000 7fd356f81700 1 -- 172.21.15.189:0/3342900279 >> 172.21.15.189:0/3342900279 conn(0x7fd3500cc290 msgr2=0x7fd35015c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:03.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.793+0000 7fd356f81700 1 -- 172.21.15.189:0/3342900279 shutdown_connections 2024-04-06T12:12:03.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.793+0000 7fd356f81700 1 -- 172.21.15.189:0/3342900279 wait complete. 2024-04-06T12:12:03.798 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' created 2024-04-06T12:12:03.813 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-06T12:12:03.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.941+0000 7f72c8694700 1 Processor -- start 2024-04-06T12:12:03.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.941+0000 7f72c8694700 1 -- start start 2024-04-06T12:12:03.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.942+0000 7f72c8694700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72c00d11b0 0x7f72c00d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:03.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.942+0000 7f72c8694700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72c00d1ad0 con 0x7f72c00cfa00 2024-04-06T12:12:03.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.942+0000 7f72c8694700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72c00cbd80 con 0x7f72c00cb870 2024-04-06T12:12:03.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.942+0000 7f72c8694700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c00ccdc0 con 0x7f72c00d11b0 2024-04-06T12:12:03.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.943+0000 7f72c6c31700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f72c00cfa00 0x7f72c00cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58940/0 (socket says 172.21.15.189:58940) 2024-04-06T12:12:03.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.943+0000 7f72c6c31700 1 -- 172.21.15.189:0/3353408642 learned_addr learned my addr 172.21.15.189:0/3353408642 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:03.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.943+0000 7f72c6430700 1 --2- 172.21.15.189:0/3353408642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72c00d11b0 0x7f72c00d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:03.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.943+0000 7f72c542e700 1 -- 172.21.15.189:0/3353408642 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733899327 0 0) 0x7f72c00d1ad0 con 0x7f72c00cfa00 2024-04-06T12:12:03.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.943+0000 7f72c542e700 1 -- 172.21.15.189:0/3353408642 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ac0035b0 con 0x7f72c00cfa00 2024-04-06T12:12:03.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.943+0000 7f72c542e700 1 -- 172.21.15.189:0/3353408642 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934218741 0 0) 0x7f72c00cbd80 con 0x7f72c00cb870 2024-04-06T12:12:03.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.943+0000 7f72c542e700 1 -- 172.21.15.189:0/3353408642 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72c00d1ad0 con 0x7f72c00cb870 2024-04-06T12:12:03.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.943+0000 7f72c6430700 1 -- 172.21.15.189:0/3353408642 >> v1:172.21.15.192:6789/0 conn(0x7f72c00cb870 legacy=0x7f72c015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:03.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.943+0000 7f72c6430700 1 -- 172.21.15.189:0/3353408642 >> v1:172.21.15.189:6789/0 conn(0x7f72c00cfa00 legacy=0x7f72c00cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:03.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.943+0000 7f72c6430700 1 -- 172.21.15.189:0/3353408642 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c00cef20 con 0x7f72c00d11b0 2024-04-06T12:12:03.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.943+0000 7f72c6430700 1 --2- 172.21.15.189:0/3353408642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72c00d11b0 0x7f72c00d1590 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009870 tx=0x7f72b0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d2bdf05a1d05260 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:03.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.943+0000 7f72c542e700 1 -- 172.21.15.189:0/3353408642 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b0011830 con 0x7f72c00d11b0 2024-04-06T12:12:03.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.943+0000 7f72c542e700 1 -- 172.21.15.189:0/3353408642 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b0011990 con 0x7f72c00d11b0 2024-04-06T12:12:03.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.943+0000 7f72c542e700 1 -- 172.21.15.189:0/3353408642 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b0011d20 con 0x7f72c00d11b0 2024-04-06T12:12:03.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.944+0000 7f72c8694700 1 -- 172.21.15.189:0/3353408642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72c00d11b0 msgr2=0x7f72c00d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:03.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.944+0000 7f72c8694700 1 --2- 172.21.15.189:0/3353408642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72c00d11b0 0x7f72c00d1590 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009870 tx=0x7f72b0009b80 comp rx=0 tx=0).stop 2024-04-06T12:12:03.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.944+0000 7f72c8694700 1 -- 172.21.15.189:0/3353408642 shutdown_connections 2024-04-06T12:12:03.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.944+0000 7f72c8694700 1 --2- 172.21.15.189:0/3353408642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72c00d11b0 0x7f72c00d1590 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:03.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.944+0000 7f72c8694700 1 -- 172.21.15.189:0/3353408642 >> 172.21.15.189:0/3353408642 conn(0x7f72c00b2db0 msgr2=0x7f72c00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:03.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.944+0000 7f72c8694700 1 -- 172.21.15.189:0/3353408642 shutdown_connections 2024-04-06T12:12:03.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.944+0000 7f72c8694700 1 -- 172.21.15.189:0/3353408642 wait complete. 2024-04-06T12:12:03.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.946+0000 7f72c8694700 1 Processor -- start 2024-04-06T12:12:03.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.946+0000 7f72c8694700 1 -- start start 2024-04-06T12:12:03.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.946+0000 7f72c8694700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72c00cb870 0x7f72c01f1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:03.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.946+0000 7f72c8694700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72c00cfa00 0x7f72c01f0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:03.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.946+0000 7f72c8694700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72c00d11b0 0x7f72c00cd520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:03.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72c8694700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c01f0ad0 con 0x7f72c00cb870 2024-04-06T12:12:03.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72c8694700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c00c6440 con 0x7f72c00d11b0 2024-04-06T12:12:03.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72c8694700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c00b58a0 con 0x7f72c00cfa00 2024-04-06T12:12:03.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72c6430700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72c00cb870 0x7f72c01f1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:03.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72c6430700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72c00cb870 0x7f72c01f1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44522/0 (socket says 172.21.15.189:44522) 2024-04-06T12:12:03.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72c6430700 1 -- 172.21.15.189:0/2363910444 learned_addr learned my addr 172.21.15.189:0/2363910444 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:03.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72c6c31700 1 --2- 172.21.15.189:0/2363910444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72c00d11b0 0x7f72c00cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:03.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72c5c2f700 1 --2- 172.21.15.189:0/2363910444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72c00cfa00 0x7f72c01f0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:03.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72c6430700 1 -- 172.21.15.189:0/2363910444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72c00cfa00 msgr2=0x7f72c01f0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:03.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72c6430700 1 --2- 172.21.15.189:0/2363910444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72c00cfa00 0x7f72c01f0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:03.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72c6430700 1 -- 172.21.15.189:0/2363910444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72c00d11b0 msgr2=0x7f72c00cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:03.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72c6430700 1 --2- 172.21.15.189:0/2363910444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72c00d11b0 0x7f72c00cd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:03.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72c6430700 1 -- 172.21.15.189:0/2363910444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b00094a0 con 0x7f72c00cb870 2024-04-06T12:12:03.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72c6c31700 1 --2- 172.21.15.189:0/2363910444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72c00d11b0 0x7f72c00cd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:03.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72c6430700 1 --2- 172.21.15.189:0/2363910444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72c00cb870 0x7f72c01f1df0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f72b00097c0 tx=0x7f72b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:03.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72b6ffd700 1 -- 172.21.15.189:0/2363910444 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b001b870 con 0x7f72c00cb870 2024-04-06T12:12:03.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72b6ffd700 1 -- 172.21.15.189:0/2363910444 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b001b9d0 con 0x7f72c00cb870 2024-04-06T12:12:03.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72c8694700 1 -- 172.21.15.189:0/2363910444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c00b55c0 con 0x7f72c00cb870 2024-04-06T12:12:03.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72b6ffd700 1 -- 172.21.15.189:0/2363910444 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b001bd80 con 0x7f72c00cb870 2024-04-06T12:12:03.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.947+0000 7f72c8694700 1 -- 172.21.15.189:0/2363910444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c00c69c0 con 0x7f72c00cb870 2024-04-06T12:12:03.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.948+0000 7f72c8694700 1 -- 172.21.15.189:0/2363910444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72c00c62a0 con 0x7f72c00cb870 2024-04-06T12:12:03.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.949+0000 7f72b6ffd700 1 -- 172.21.15.189:0/2363910444 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f72b0023070 con 0x7f72c00cb870 2024-04-06T12:12:03.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.949+0000 7f72b6ffd700 1 --2- 172.21.15.189:0/2363910444 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f72a406ac90 0x7f72a406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:03.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.949+0000 7f72b6ffd700 1 -- 172.21.15.189:0/2363910444 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f72b008a350 con 0x7f72c00cb870 2024-04-06T12:12:03.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.949+0000 7f72c5c2f700 1 --2- 172.21.15.189:0/2363910444 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f72a406ac90 0x7f72a406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:03.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.949+0000 7f72c5c2f700 1 --2- 172.21.15.189:0/2363910444 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f72a406ac90 0x7f72a406d150 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f72bc007fd0 tx=0x7f72bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:03.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:03.954+0000 7f72b6ffd700 1 -- 172.21.15.189:0/2363910444 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f72c00c62a0 con 0x7f72c00cb870 2024-04-06T12:12:04.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.193+0000 7f72c8694700 1 -- 172.21.15.189:0/2363910444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f72c00c62a0 con 0x7f72c00cb870 2024-04-06T12:12:04.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.199+0000 7f72b6ffd700 1 -- 172.21.15.189:0/2363910444 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f72c00c62a0 con 0x7f72c00cb870 2024-04-06T12:12:04.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.204+0000 7f72c8694700 1 -- 172.21.15.189:0/2363910444 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f72a406ac90 msgr2=0x7f72a406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:04.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.204+0000 7f72c8694700 1 --2- 172.21.15.189:0/2363910444 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f72a406ac90 0x7f72a406d150 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f72bc007fd0 tx=0x7f72bc009040 comp rx=0 tx=0).stop 2024-04-06T12:12:04.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.204+0000 7f72c8694700 1 -- 172.21.15.189:0/2363910444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72c00cb870 msgr2=0x7f72c01f1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:04.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.204+0000 7f72c8694700 1 --2- 172.21.15.189:0/2363910444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72c00cb870 0x7f72c01f1df0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f72b00097c0 tx=0x7f72b0004750 comp rx=0 tx=0).stop 2024-04-06T12:12:04.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.205+0000 7f72c8694700 1 -- 172.21.15.189:0/2363910444 shutdown_connections 2024-04-06T12:12:04.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.205+0000 7f72c8694700 1 --2- 172.21.15.189:0/2363910444 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f72a406ac90 0x7f72a406d150 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f72bc007fd0 tx=0x7f72bc009040 comp rx=0 tx=0).stop 2024-04-06T12:12:04.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.205+0000 7f72c8694700 1 --2- 172.21.15.189:0/2363910444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72c00d11b0 0x7f72c00cd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:04.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.205+0000 7f72c8694700 1 --2- 172.21.15.189:0/2363910444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72c00cb870 0x7f72c01f1df0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:04.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.205+0000 7f72c8694700 1 --2- 172.21.15.189:0/2363910444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72c00cfa00 0x7f72c01f0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:04.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.205+0000 7f72c8694700 1 -- 172.21.15.189:0/2363910444 >> 172.21.15.189:0/2363910444 conn(0x7f72c00b2db0 msgr2=0x7f72c015aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:04.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.208+0000 7f72c8694700 1 -- 172.21.15.189:0/2363910444 shutdown_connections 2024-04-06T12:12:04.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.208+0000 7f72c8694700 1 -- 172.21.15.189:0/2363910444 wait complete. 2024-04-06T12:12:04.213 INFO:teuthology.orchestra.run.smithi189.stderr:new fs with metadata pool 2 and data pool 3 2024-04-06T12:12:04.227 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-06T12:12:04.227 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-06T12:12:04.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.354+0000 7faed4ca4700 1 Processor -- start 2024-04-06T12:12:04.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.354+0000 7faed4ca4700 1 -- start start 2024-04-06T12:12:04.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.355+0000 7faed4ca4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faed00d17a0 0x7faed006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:04.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.355+0000 7faed4ca4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7faed0146dd0 0x7faed006d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:04.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.355+0000 7faed4ca4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faed006c0a0 con 0x7faed00cedf0 2024-04-06T12:12:04.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.355+0000 7faed4ca4700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed006c380 con 0x7faed00d17a0 2024-04-06T12:12:04.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.355+0000 7faed4ca4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed006c660 con 0x7faed0146dd0 2024-04-06T12:12:04.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.356+0000 7faece59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faed00d17a0 0x7faed006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:04.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.356+0000 7faeced9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7faed00cedf0 0x7faed00cf1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58942/0 (socket says 172.21.15.189:58942) 2024-04-06T12:12:04.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.356+0000 7faeced9d700 1 -- 172.21.15.189:0/508051027 learned_addr learned my addr 172.21.15.189:0/508051027 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:04.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.356+0000 7faecdd9b700 1 --2- 172.21.15.189:0/508051027 >> v2:172.21.15.192:3300/0 conn(0x7faed0146dd0 0x7faed006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:04.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.356+0000 7faece59c700 1 -- 172.21.15.189:0/508051027 >> v2:172.21.15.192:3300/0 conn(0x7faed0146dd0 msgr2=0x7faed006d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:04.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.356+0000 7faece59c700 1 --2- 172.21.15.189:0/508051027 >> v2:172.21.15.192:3300/0 conn(0x7faed0146dd0 0x7faed006d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:04.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.356+0000 7faece59c700 1 -- 172.21.15.189:0/508051027 >> v1:172.21.15.189:6789/0 conn(0x7faed00cedf0 legacy=0x7faed00cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:04.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.356+0000 7faecd59a700 1 -- 172.21.15.189:0/508051027 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410413155 0 0) 0x7faed006c0a0 con 0x7faed00cedf0 2024-04-06T12:12:04.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.356+0000 7faece59c700 1 -- 172.21.15.189:0/508051027 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed006c940 con 0x7faed00d17a0 2024-04-06T12:12:04.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.356+0000 7faece59c700 1 --2- 172.21.15.189:0/508051027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faed00d17a0 0x7faed006b7a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faed00d2300 tx=0x7faeb8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2726ce562d3256 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:04.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.356+0000 7faecd59a700 1 -- 172.21.15.189:0/508051027 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb8011830 con 0x7faed00d17a0 2024-04-06T12:12:04.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.356+0000 7faecd59a700 1 -- 172.21.15.189:0/508051027 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb8011990 con 0x7faed00d17a0 2024-04-06T12:12:04.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.356+0000 7faecd59a700 1 -- 172.21.15.189:0/508051027 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb8011d20 con 0x7faed00d17a0 2024-04-06T12:12:04.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.357+0000 7faed4ca4700 1 -- 172.21.15.189:0/508051027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faed00d17a0 msgr2=0x7faed006b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:04.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.357+0000 7faed4ca4700 1 --2- 172.21.15.189:0/508051027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faed00d17a0 0x7faed006b7a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faed00d2300 tx=0x7faeb8009b20 comp rx=0 tx=0).stop 2024-04-06T12:12:04.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.357+0000 7faed4ca4700 1 -- 172.21.15.189:0/508051027 shutdown_connections 2024-04-06T12:12:04.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.357+0000 7faed4ca4700 1 --2- 172.21.15.189:0/508051027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faed00d17a0 0x7faed006b7a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:04.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.357+0000 7faed4ca4700 1 --2- 172.21.15.189:0/508051027 >> v2:172.21.15.192:3300/0 conn(0x7faed0146dd0 0x7faed006d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:04.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.357+0000 7faed4ca4700 1 -- 172.21.15.189:0/508051027 >> 172.21.15.189:0/508051027 conn(0x7faed00b2db0 msgr2=0x7faed00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:04.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.357+0000 7faed4ca4700 1 -- 172.21.15.189:0/508051027 shutdown_connections 2024-04-06T12:12:04.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.357+0000 7faed4ca4700 1 -- 172.21.15.189:0/508051027 wait complete. 2024-04-06T12:12:04.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.358+0000 7faed4ca4700 1 Processor -- start 2024-04-06T12:12:04.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.358+0000 7faed4ca4700 1 -- start start 2024-04-06T12:12:04.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.359+0000 7faed4ca4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faed00cedf0 0x7faed00ae600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:04.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.359+0000 7faed4ca4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faed00d17a0 0x7faed00ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:04.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.359+0000 7faed4ca4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faed0146dd0 0x7faed00ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:04.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.359+0000 7faed4ca4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed00b42b0 con 0x7faed00d17a0 2024-04-06T12:12:04.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.359+0000 7faed4ca4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed01f2b20 con 0x7faed00cedf0 2024-04-06T12:12:04.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.359+0000 7faed4ca4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed00ada00 con 0x7faed0146dd0 2024-04-06T12:12:04.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.359+0000 7faecdd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faed00d17a0 0x7faed00ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:04.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.359+0000 7faecdd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faed00d17a0 0x7faed00ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44538/0 (socket says 172.21.15.189:44538) 2024-04-06T12:12:04.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.359+0000 7faecdd9b700 1 -- 172.21.15.189:0/937681300 learned_addr learned my addr 172.21.15.189:0/937681300 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:04.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.359+0000 7faece59c700 1 --2- 172.21.15.189:0/937681300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faed00cedf0 0x7faed00ae600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:04.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.359+0000 7faecdd9b700 1 -- 172.21.15.189:0/937681300 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faed0146dd0 msgr2=0x7faed00ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:04.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.359+0000 7faecdd9b700 1 --2- 172.21.15.189:0/937681300 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faed0146dd0 0x7faed00ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:04.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.359+0000 7faecdd9b700 1 -- 172.21.15.189:0/937681300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faed00cedf0 msgr2=0x7faed00ae600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:04.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.359+0000 7faecdd9b700 1 --2- 172.21.15.189:0/937681300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faed00cedf0 0x7faed00ae600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:04.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.359+0000 7faecdd9b700 1 -- 172.21.15.189:0/937681300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb80094a0 con 0x7faed00d17a0 2024-04-06T12:12:04.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.359+0000 7faecdd9b700 1 --2- 172.21.15.189:0/937681300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faed00d17a0 0x7faed00ac8d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faec000e980 tx=0x7faec000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:04.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.360+0000 7faec6ffd700 1 -- 172.21.15.189:0/937681300 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec0009710 con 0x7faed00d17a0 2024-04-06T12:12:04.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.360+0000 7faec6ffd700 1 -- 172.21.15.189:0/937681300 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec000c850 con 0x7faed00d17a0 2024-04-06T12:12:04.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.360+0000 7faed4ca4700 1 -- 172.21.15.189:0/937681300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed00ad440 con 0x7faed00d17a0 2024-04-06T12:12:04.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.360+0000 7faec6ffd700 1 -- 172.21.15.189:0/937681300 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec000cc00 con 0x7faed00d17a0 2024-04-06T12:12:04.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.360+0000 7faed4ca4700 1 -- 172.21.15.189:0/937681300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed00b3f40 con 0x7faed00d17a0 2024-04-06T12:12:04.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.361+0000 7faed4ca4700 1 -- 172.21.15.189:0/937681300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed00b4460 con 0x7faed00d17a0 2024-04-06T12:12:04.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.361+0000 7faec6ffd700 1 -- 172.21.15.189:0/937681300 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7faec0013470 con 0x7faed00d17a0 2024-04-06T12:12:04.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.361+0000 7faec6ffd700 1 --2- 172.21.15.189:0/937681300 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faeb006ac90 0x7faeb006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:04.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.361+0000 7faec6ffd700 1 -- 172.21.15.189:0/937681300 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faec0089450 con 0x7faed00d17a0 2024-04-06T12:12:04.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.362+0000 7faece59c700 1 --2- 172.21.15.189:0/937681300 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faeb006ac90 0x7faeb006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:04.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.362+0000 7faece59c700 1 --2- 172.21.15.189:0/937681300 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faeb006ac90 0x7faeb006d150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faed00b01a0 tx=0x7faeb8008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:04.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.366+0000 7faec6ffd700 1 -- 172.21.15.189:0/937681300 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7faed00b4460 con 0x7faed00d17a0 2024-04-06T12:12:04.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:04.571+0000 7faed4ca4700 1 -- 172.21.15.189:0/937681300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7faed0074990 con 0x7faed00d17a0 2024-04-06T12:12:05.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.205+0000 7faec6ffd700 1 -- 172.21.15.189:0/937681300 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) v1 ==== 157+0+0 (secure 0 0 0) 0x7faec00546f0 con 0x7faed00d17a0 2024-04-06T12:12:05.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.207+0000 7faed4ca4700 1 -- 172.21.15.189:0/937681300 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faeb006ac90 msgr2=0x7faeb006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:05.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.207+0000 7faed4ca4700 1 --2- 172.21.15.189:0/937681300 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faeb006ac90 0x7faeb006d150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faed00b01a0 tx=0x7faeb8008030 comp rx=0 tx=0).stop 2024-04-06T12:12:05.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.207+0000 7faed4ca4700 1 -- 172.21.15.189:0/937681300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faed00d17a0 msgr2=0x7faed00ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:05.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.207+0000 7faed4ca4700 1 --2- 172.21.15.189:0/937681300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faed00d17a0 0x7faed00ac8d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faec000e980 tx=0x7faec000ed40 comp rx=0 tx=0).stop 2024-04-06T12:12:05.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.207+0000 7faed4ca4700 1 -- 172.21.15.189:0/937681300 shutdown_connections 2024-04-06T12:12:05.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.207+0000 7faed4ca4700 1 --2- 172.21.15.189:0/937681300 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faed0146dd0 0x7faed00ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:05.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.207+0000 7faed4ca4700 1 --2- 172.21.15.189:0/937681300 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faeb006ac90 0x7faeb006d150 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:05.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.207+0000 7faed4ca4700 1 --2- 172.21.15.189:0/937681300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faed00cedf0 0x7faed00ae600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:05.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.207+0000 7faed4ca4700 1 --2- 172.21.15.189:0/937681300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faed00d17a0 0x7faed00ac8d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:05.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.207+0000 7faed4ca4700 1 -- 172.21.15.189:0/937681300 >> 172.21.15.189:0/937681300 conn(0x7faed00b2db0 msgr2=0x7faed015db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:05.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.208+0000 7faed4ca4700 1 -- 172.21.15.189:0/937681300 shutdown_connections 2024-04-06T12:12:05.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.208+0000 7faed4ca4700 1 -- 172.21.15.189:0/937681300 wait complete. 2024-04-06T12:12:05.222 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-06T12:12:05.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.359+0000 7f4faad5a700 1 Processor -- start 2024-04-06T12:12:05.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.359+0000 7f4faad5a700 1 -- start start 2024-04-06T12:12:05.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.360+0000 7f4faad5a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4fa4153f60 0x7f4fa415ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:05.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.360+0000 7f4faad5a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4fa406eab0 0x7f4fa406b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:05.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.360+0000 7f4faad5a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fa4070e90 0x7f4fa4071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:05.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.360+0000 7f4faad5a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fa406ac40 con 0x7f4fa4153f60 2024-04-06T12:12:05.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.360+0000 7f4faad5a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fa4071840 con 0x7f4fa406eab0 2024-04-06T12:12:05.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.360+0000 7f4faad5a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa415b060 con 0x7f4fa4070e90 2024-04-06T12:12:05.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.360+0000 7f4fa8af6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4fa4153f60 0x7f4fa415ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:05.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.360+0000 7f4fa8af6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4fa4153f60 0x7f4fa415ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44542/0 (socket says 172.21.15.189:44542) 2024-04-06T12:12:05.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.360+0000 7f4fa8af6700 1 -- 172.21.15.189:0/659038272 learned_addr learned my addr 172.21.15.189:0/659038272 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:05.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.360+0000 7f4fa3fff700 1 --2- 172.21.15.189:0/659038272 >> v2:172.21.15.192:3300/0 conn(0x7f4fa406eab0 0x7f4fa406b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:05.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.360+0000 7f4fa92f7700 1 --2- 172.21.15.189:0/659038272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fa4070e90 0x7f4fa4071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:05.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.361+0000 7f4fa8af6700 1 -- 172.21.15.189:0/659038272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fa4070e90 msgr2=0x7f4fa4071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:05.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.361+0000 7f4fa8af6700 1 --2- 172.21.15.189:0/659038272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fa4070e90 0x7f4fa4071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:05.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.361+0000 7f4fa8af6700 1 -- 172.21.15.189:0/659038272 >> v2:172.21.15.192:3300/0 conn(0x7f4fa406eab0 msgr2=0x7f4fa406b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:05.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.361+0000 7f4fa8af6700 1 --2- 172.21.15.189:0/659038272 >> v2:172.21.15.192:3300/0 conn(0x7f4fa406eab0 0x7f4fa406b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:05.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.361+0000 7f4fa8af6700 1 -- 172.21.15.189:0/659038272 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa406cd90 con 0x7f4fa4153f60 2024-04-06T12:12:05.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.361+0000 7f4fa8af6700 1 --2- 172.21.15.189:0/659038272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa4153f60 0x7f4fa415ab20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4fa40722b0 tx=0x7f4f9800ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e12c927ce5dfe10 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:05.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.361+0000 7f4fa37fe700 1 -- 172.21.15.189:0/659038272 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9800cb70 con 0x7f4fa4153f60 2024-04-06T12:12:05.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.361+0000 7f4fa37fe700 1 -- 172.21.15.189:0/659038272 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9800ccd0 con 0x7f4fa4153f60 2024-04-06T12:12:05.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.361+0000 7f4fa37fe700 1 -- 172.21.15.189:0/659038272 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f98018620 con 0x7f4fa4153f60 2024-04-06T12:12:05.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.361+0000 7f4faad5a700 1 -- 172.21.15.189:0/659038272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa4153f60 msgr2=0x7f4fa415ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:05.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.361+0000 7f4faad5a700 1 --2- 172.21.15.189:0/659038272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa4153f60 0x7f4fa415ab20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4fa40722b0 tx=0x7f4f9800ad70 comp rx=0 tx=0).stop 2024-04-06T12:12:05.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.361+0000 7f4faad5a700 1 -- 172.21.15.189:0/659038272 shutdown_connections 2024-04-06T12:12:05.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.361+0000 7f4faad5a700 1 --2- 172.21.15.189:0/659038272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa4153f60 0x7f4fa415ab20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:05.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.361+0000 7f4faad5a700 1 --2- 172.21.15.189:0/659038272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fa4070e90 0x7f4fa4071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:05.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.361+0000 7f4faad5a700 1 --2- 172.21.15.189:0/659038272 >> v2:172.21.15.192:3300/0 conn(0x7f4fa406eab0 0x7f4fa406b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:05.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.361+0000 7f4faad5a700 1 -- 172.21.15.189:0/659038272 >> 172.21.15.189:0/659038272 conn(0x7f4fa40b1da0 msgr2=0x7f4fa40cf840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:05.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.361+0000 7f4faad5a700 1 -- 172.21.15.189:0/659038272 shutdown_connections 2024-04-06T12:12:05.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.361+0000 7f4faad5a700 1 -- 172.21.15.189:0/659038272 wait complete. 2024-04-06T12:12:05.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.363+0000 7f4faad5a700 1 Processor -- start 2024-04-06T12:12:05.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.363+0000 7f4faad5a700 1 -- start start 2024-04-06T12:12:05.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.363+0000 7f4faad5a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa406eab0 0x7f4fa40c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:05.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.363+0000 7f4faad5a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fa4070e90 0x7f4fa4148b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:05.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.363+0000 7f4faad5a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fa4153f60 0x7f4fa41490c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:05.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.363+0000 7f4faad5a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa4159070 con 0x7f4fa406eab0 2024-04-06T12:12:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.363+0000 7f4faad5a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa40c9500 con 0x7f4fa4153f60 2024-04-06T12:12:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.363+0000 7f4faad5a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa4158d90 con 0x7f4fa4070e90 2024-04-06T12:12:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.364+0000 7f4fa8af6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa406eab0 0x7f4fa40c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.364+0000 7f4fa8af6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa406eab0 0x7f4fa40c7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44544/0 (socket says 172.21.15.189:44544) 2024-04-06T12:12:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.364+0000 7f4fa8af6700 1 -- 172.21.15.189:0/638585894 learned_addr learned my addr 172.21.15.189:0/638585894 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.364+0000 7f4fa3fff700 1 --2- 172.21.15.189:0/638585894 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fa4070e90 0x7f4fa4148b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.364+0000 7f4fa92f7700 1 --2- 172.21.15.189:0/638585894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fa4153f60 0x7f4fa41490c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.364+0000 7f4fa8af6700 1 -- 172.21.15.189:0/638585894 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fa4070e90 msgr2=0x7f4fa4148b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.364+0000 7f4fa8af6700 1 --2- 172.21.15.189:0/638585894 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fa4070e90 0x7f4fa4148b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.364+0000 7f4fa8af6700 1 -- 172.21.15.189:0/638585894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fa4153f60 msgr2=0x7f4fa41490c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.364+0000 7f4fa8af6700 1 --2- 172.21.15.189:0/638585894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fa4153f60 0x7f4fa41490c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.364+0000 7f4fa8af6700 1 -- 172.21.15.189:0/638585894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f9800a6f0 con 0x7f4fa406eab0 2024-04-06T12:12:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.364+0000 7f4fa92f7700 1 --2- 172.21.15.189:0/638585894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fa4153f60 0x7f4fa41490c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.364+0000 7f4fa8af6700 1 --2- 172.21.15.189:0/638585894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa406eab0 0x7f4fa40c7780 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4f98000c00 tx=0x7f4f980047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:05.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.364+0000 7f4fa17fa700 1 -- 172.21.15.189:0/638585894 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f98003ab0 con 0x7f4fa406eab0 2024-04-06T12:12:05.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.364+0000 7f4fa17fa700 1 -- 172.21.15.189:0/638585894 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f98003c30 con 0x7f4fa406eab0 2024-04-06T12:12:05.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.364+0000 7f4fa17fa700 1 -- 172.21.15.189:0/638585894 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f98020400 con 0x7f4fa406eab0 2024-04-06T12:12:05.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.364+0000 7f4faad5a700 1 -- 172.21.15.189:0/638585894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa4143780 con 0x7f4fa406eab0 2024-04-06T12:12:05.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.364+0000 7f4faad5a700 1 -- 172.21.15.189:0/638585894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa4143d40 con 0x7f4fa406eab0 2024-04-06T12:12:05.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.365+0000 7f4faad5a700 1 -- 172.21.15.189:0/638585894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa40cd090 con 0x7f4fa406eab0 2024-04-06T12:12:05.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.365+0000 7f4fa17fa700 1 -- 172.21.15.189:0/638585894 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4f98018400 con 0x7f4fa406eab0 2024-04-06T12:12:05.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.366+0000 7f4fa17fa700 1 --2- 172.21.15.189:0/638585894 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f8806ac90 0x7f4f8806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:05.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.366+0000 7f4fa17fa700 1 -- 172.21.15.189:0/638585894 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5935+0+0 (secure 0 0 0) 0x7f4f9808a580 con 0x7f4fa406eab0 2024-04-06T12:12:05.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.366+0000 7f4fa3fff700 1 --2- 172.21.15.189:0/638585894 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f8806ac90 0x7f4f8806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:05.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.366+0000 7f4fa3fff700 1 --2- 172.21.15.189:0/638585894 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f8806ac90 0x7f4f8806d150 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4151ae0 tx=0x7f4f90008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:05.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.370+0000 7f4fa17fa700 1 -- 172.21.15.189:0/638585894 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4f98055710 con 0x7f4fa406eab0 2024-04-06T12:12:05.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:05.564+0000 7f4faad5a700 1 -- 172.21.15.189:0/638585894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4fa40748f0 con 0x7f4fa406eab0 2024-04-06T12:12:07.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.219+0000 7f4fa17fa700 1 -- 172.21.15.189:0/638585894 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) v1 ==== 228+0+0 (secure 0 0 0) 0x7f4f98058d30 con 0x7f4fa406eab0 2024-04-06T12:12:07.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.225+0000 7f4faad5a700 1 -- 172.21.15.189:0/638585894 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f8806ac90 msgr2=0x7f4f8806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:07.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.225+0000 7f4faad5a700 1 --2- 172.21.15.189:0/638585894 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f8806ac90 0x7f4f8806d150 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4151ae0 tx=0x7f4f90008040 comp rx=0 tx=0).stop 2024-04-06T12:12:07.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.225+0000 7f4faad5a700 1 -- 172.21.15.189:0/638585894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa406eab0 msgr2=0x7f4fa40c7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:07.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.225+0000 7f4faad5a700 1 --2- 172.21.15.189:0/638585894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa406eab0 0x7f4fa40c7780 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4f98000c00 tx=0x7f4f980047c0 comp rx=0 tx=0).stop 2024-04-06T12:12:07.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.225+0000 7f4faad5a700 1 -- 172.21.15.189:0/638585894 shutdown_connections 2024-04-06T12:12:07.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.225+0000 7f4faad5a700 1 --2- 172.21.15.189:0/638585894 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f8806ac90 0x7f4f8806d150 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:07.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.225+0000 7f4faad5a700 1 --2- 172.21.15.189:0/638585894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fa4153f60 0x7f4fa41490c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:07.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.225+0000 7f4faad5a700 1 --2- 172.21.15.189:0/638585894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa406eab0 0x7f4fa40c7780 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:07.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.225+0000 7f4faad5a700 1 --2- 172.21.15.189:0/638585894 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fa4070e90 0x7f4fa4148b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:07.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.225+0000 7f4faad5a700 1 -- 172.21.15.189:0/638585894 >> 172.21.15.189:0/638585894 conn(0x7f4fa40b1da0 msgr2=0x7f4fa4154340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:07.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.225+0000 7f4faad5a700 1 -- 172.21.15.189:0/638585894 shutdown_connections 2024-04-06T12:12:07.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.226+0000 7f4faad5a700 1 -- 172.21.15.189:0/638585894 wait complete. 2024-04-06T12:12:07.228 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data_ec' created 2024-04-06T12:12:07.244 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-06T12:12:07.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.375+0000 7f4bb286b700 1 Processor -- start 2024-04-06T12:12:07.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.375+0000 7f4bb286b700 1 -- start start 2024-04-06T12:12:07.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.376+0000 7f4bb286b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4bac15d7b0 0x7f4bac0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:07.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.376+0000 7f4bb286b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4bac15e500 0x7f4bac06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:07.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.376+0000 7f4bb286b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bac0b3990 con 0x7f4bac06a730 2024-04-06T12:12:07.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.376+0000 7f4bb286b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bac06b0f0 con 0x7f4bac15d7b0 2024-04-06T12:12:07.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.376+0000 7f4bb286b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac0b3060 con 0x7f4bac15e500 2024-04-06T12:12:07.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.376+0000 7f4babfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4bac15d7b0 0x7f4bac0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:07.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.376+0000 7f4bab7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4bac06a730 0x7f4bac15ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58944/0 (socket says 172.21.15.189:58944) 2024-04-06T12:12:07.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.376+0000 7f4bab7fe700 1 -- 172.21.15.189:0/3385438117 learned_addr learned my addr 172.21.15.189:0/3385438117 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:07.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.376+0000 7f4bb0e08700 1 --2- 172.21.15.189:0/3385438117 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4bac15e500 0x7f4bac06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:07.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.376+0000 7f4babfff700 1 -- 172.21.15.189:0/3385438117 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4bac15e500 msgr2=0x7f4bac06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:07.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.376+0000 7f4babfff700 1 --2- 172.21.15.189:0/3385438117 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4bac15e500 0x7f4bac06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:07.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.376+0000 7f4babfff700 1 -- 172.21.15.189:0/3385438117 >> v1:172.21.15.189:6789/0 conn(0x7f4bac06a730 legacy=0x7f4bac15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:07.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.376+0000 7f4babfff700 1 -- 172.21.15.189:0/3385438117 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bac0b3340 con 0x7f4bac15d7b0 2024-04-06T12:12:07.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.376+0000 7f4baaffd700 1 -- 172.21.15.189:0/3385438117 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275578016 0 0) 0x7f4bac0b3990 con 0x7f4bac06a730 2024-04-06T12:12:07.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.376+0000 7f4babfff700 1 --2- 172.21.15.189:0/3385438117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bac15d7b0 0x7f4bac0b3e20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0009840 tx=0x7f4ba0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78042c0653bdd0bb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:07.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.377+0000 7f4baaffd700 1 -- 172.21.15.189:0/3385438117 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0011830 con 0x7f4bac15d7b0 2024-04-06T12:12:07.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.377+0000 7f4baaffd700 1 -- 172.21.15.189:0/3385438117 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba0011990 con 0x7f4bac15d7b0 2024-04-06T12:12:07.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.377+0000 7f4baaffd700 1 -- 172.21.15.189:0/3385438117 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0011d20 con 0x7f4bac15d7b0 2024-04-06T12:12:07.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.377+0000 7f4bb286b700 1 -- 172.21.15.189:0/3385438117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bac15d7b0 msgr2=0x7f4bac0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:07.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.377+0000 7f4bb286b700 1 --2- 172.21.15.189:0/3385438117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bac15d7b0 0x7f4bac0b3e20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0009840 tx=0x7f4ba0009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:07.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.377+0000 7f4bb286b700 1 -- 172.21.15.189:0/3385438117 shutdown_connections 2024-04-06T12:12:07.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.377+0000 7f4bb286b700 1 --2- 172.21.15.189:0/3385438117 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4bac15e500 0x7f4bac06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:07.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.377+0000 7f4bb286b700 1 --2- 172.21.15.189:0/3385438117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bac15d7b0 0x7f4bac0b3e20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:07.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.377+0000 7f4bb286b700 1 -- 172.21.15.189:0/3385438117 >> 172.21.15.189:0/3385438117 conn(0x7f4bac0d00b0 msgr2=0x7f4bac0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:07.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.377+0000 7f4bb286b700 1 -- 172.21.15.189:0/3385438117 shutdown_connections 2024-04-06T12:12:07.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.377+0000 7f4bb286b700 1 -- 172.21.15.189:0/3385438117 wait complete. 2024-04-06T12:12:07.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.379+0000 7f4bb286b700 1 Processor -- start 2024-04-06T12:12:07.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.379+0000 7f4bb286b700 1 -- start start 2024-04-06T12:12:07.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.380+0000 7f4bb286b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4bac06a730 0x7f4bac0c6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:07.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.380+0000 7f4bb286b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4bac15d7b0 0x7f4bac15fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:07.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.380+0000 7f4bb286b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bac15e500 0x7f4bac160250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.380+0000 7f4bb286b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac149f70 con 0x7f4bac15e500 2024-04-06T12:12:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.380+0000 7f4bb286b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac1492f0 con 0x7f4bac15d7b0 2024-04-06T12:12:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.380+0000 7f4bb286b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac1495d0 con 0x7f4bac06a730 2024-04-06T12:12:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.380+0000 7f4bb0e08700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bac15e500 0x7f4bac160250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.380+0000 7f4bb0e08700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bac15e500 0x7f4bac160250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44564/0 (socket says 172.21.15.189:44564) 2024-04-06T12:12:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.380+0000 7f4bb0e08700 1 -- 172.21.15.189:0/166314969 learned_addr learned my addr 172.21.15.189:0/166314969 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.380+0000 7f4bab7fe700 1 --2- 172.21.15.189:0/166314969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4bac15d7b0 0x7f4bac15fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.380+0000 7f4babfff700 1 --2- 172.21.15.189:0/166314969 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4bac06a730 0x7f4bac0c6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.380+0000 7f4bb0e08700 1 -- 172.21.15.189:0/166314969 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4bac06a730 msgr2=0x7f4bac0c6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.380+0000 7f4bb0e08700 1 --2- 172.21.15.189:0/166314969 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4bac06a730 0x7f4bac0c6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.380+0000 7f4bb0e08700 1 -- 172.21.15.189:0/166314969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4bac15d7b0 msgr2=0x7f4bac15fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.380+0000 7f4bb0e08700 1 --2- 172.21.15.189:0/166314969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4bac15d7b0 0x7f4bac15fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.380+0000 7f4bb0e08700 1 -- 172.21.15.189:0/166314969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba00094a0 con 0x7f4bac15e500 2024-04-06T12:12:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.380+0000 7f4bb0e08700 1 --2- 172.21.15.189:0/166314969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bac15e500 0x7f4bac160250 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400efc0 tx=0x7f4ba400c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:07.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.380+0000 7f4ba8ff9700 1 -- 172.21.15.189:0/166314969 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba400cad0 con 0x7f4bac15e500 2024-04-06T12:12:07.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.381+0000 7f4ba8ff9700 1 -- 172.21.15.189:0/166314969 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba400f040 con 0x7f4bac15e500 2024-04-06T12:12:07.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.381+0000 7f4ba8ff9700 1 -- 172.21.15.189:0/166314969 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba400ce70 con 0x7f4bac15e500 2024-04-06T12:12:07.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.381+0000 7f4bb286b700 1 -- 172.21.15.189:0/166314969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bac1498b0 con 0x7f4bac15e500 2024-04-06T12:12:07.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.381+0000 7f4bb286b700 1 -- 172.21.15.189:0/166314969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bac0c69c0 con 0x7f4bac15e500 2024-04-06T12:12:07.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.381+0000 7f4bb286b700 1 -- 172.21.15.189:0/166314969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bac14a0b0 con 0x7f4bac15e500 2024-04-06T12:12:07.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.382+0000 7f4ba8ff9700 1 -- 172.21.15.189:0/166314969 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4ba4014480 con 0x7f4bac15e500 2024-04-06T12:12:07.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.382+0000 7f4ba8ff9700 1 --2- 172.21.15.189:0/166314969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b9806ac90 0x7f4b9806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:07.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.382+0000 7f4ba8ff9700 1 -- 172.21.15.189:0/166314969 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6428+0+0 (secure 0 0 0) 0x7f4ba4089ca0 con 0x7f4bac15e500 2024-04-06T12:12:07.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.382+0000 7f4babfff700 1 --2- 172.21.15.189:0/166314969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b9806ac90 0x7f4b9806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:07.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.383+0000 7f4babfff700 1 --2- 172.21.15.189:0/166314969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b9806ac90 0x7f4b9806d150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4ba00090b0 tx=0x7f4ba0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:07.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.387+0000 7f4ba8ff9700 1 -- 172.21.15.189:0/166314969 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4bac14a0b0 con 0x7f4bac15e500 2024-04-06T12:12:07.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:07.581+0000 7f4bb286b700 1 -- 172.21.15.189:0/166314969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f4bac14a0b0 con 0x7f4bac15e500 2024-04-06T12:12:08.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.223+0000 7f4ba8ff9700 1 -- 172.21.15.189:0/166314969 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4bac14a0b0 con 0x7f4bac15e500 2024-04-06T12:12:08.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.233+0000 7f4bb286b700 1 -- 172.21.15.189:0/166314969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b9806ac90 msgr2=0x7f4b9806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:08.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.233+0000 7f4bb286b700 1 --2- 172.21.15.189:0/166314969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b9806ac90 0x7f4b9806d150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4ba00090b0 tx=0x7f4ba0008030 comp rx=0 tx=0).stop 2024-04-06T12:12:08.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.233+0000 7f4bb286b700 1 -- 172.21.15.189:0/166314969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bac15e500 msgr2=0x7f4bac160250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:08.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.233+0000 7f4bb286b700 1 --2- 172.21.15.189:0/166314969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bac15e500 0x7f4bac160250 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400efc0 tx=0x7f4ba400c0c0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.237+0000 7f4bb286b700 1 -- 172.21.15.189:0/166314969 shutdown_connections 2024-04-06T12:12:08.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.237+0000 7f4bb286b700 1 --2- 172.21.15.189:0/166314969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bac15e500 0x7f4bac160250 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.237+0000 7f4bb286b700 1 --2- 172.21.15.189:0/166314969 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4bac06a730 0x7f4bac0c6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.237+0000 7f4bb286b700 1 --2- 172.21.15.189:0/166314969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b9806ac90 0x7f4b9806d150 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.237+0000 7f4bb286b700 1 --2- 172.21.15.189:0/166314969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4bac15d7b0 0x7f4bac15fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.237+0000 7f4bb286b700 1 -- 172.21.15.189:0/166314969 >> 172.21.15.189:0/166314969 conn(0x7f4bac0d00b0 msgr2=0x7f4bac0b1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:08.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.238+0000 7f4bb286b700 1 -- 172.21.15.189:0/166314969 shutdown_connections 2024-04-06T12:12:08.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.238+0000 7f4bb286b700 1 -- 172.21.15.189:0/166314969 wait complete. 2024-04-06T12:12:08.241 INFO:teuthology.orchestra.run.smithi189.stderr:set pool 4 allow_ec_overwrites to true 2024-04-06T12:12:08.260 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-06T12:12:08.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.389+0000 7fbc538bb700 1 Processor -- start 2024-04-06T12:12:08.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.390+0000 7fbc538bb700 1 -- start start 2024-04-06T12:12:08.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.390+0000 7fbc538bb700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbc4c0cfa00 0x7fbc4c0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:08.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.390+0000 7fbc538bb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4c0cbd80 con 0x7fbc4c0cb870 2024-04-06T12:12:08.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.390+0000 7fbc538bb700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4c0cef20 con 0x7fbc4c0d11b0 2024-04-06T12:12:08.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.390+0000 7fbc538bb700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc4c0ccdc0 con 0x7fbc4c0cfa00 2024-04-06T12:12:08.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc50e56700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbc4c0cb870 0x7fbc4c15c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58958/0 (socket says 172.21.15.189:58958) 2024-04-06T12:12:08.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc50e56700 1 -- 172.21.15.189:0/1334269203 learned_addr learned my addr 172.21.15.189:0/1334269203 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:08.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc43fff700 1 -- 172.21.15.189:0/1334269203 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169464222 0 0) 0x7fbc4c0cbd80 con 0x7fbc4c0cb870 2024-04-06T12:12:08.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc51e58700 1 --2- 172.21.15.189:0/1334269203 >> v2:172.21.15.192:3300/0 conn(0x7fbc4c0cfa00 0x7fbc4c0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:08.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc43fff700 1 -- 172.21.15.189:0/1334269203 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc380035b0 con 0x7fbc4c0cb870 2024-04-06T12:12:08.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc43fff700 1 -- 172.21.15.189:0/1334269203 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2431009512 0 0) 0x7fbc380035b0 con 0x7fbc4c0cb870 2024-04-06T12:12:08.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc43fff700 1 -- 172.21.15.189:0/1334269203 >> v2:172.21.15.192:3300/0 conn(0x7fbc4c0cfa00 msgr2=0x7fbc4c0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:08.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc43fff700 1 --2- 172.21.15.189:0/1334269203 >> v2:172.21.15.192:3300/0 conn(0x7fbc4c0cfa00 0x7fbc4c0cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc43fff700 1 -- 172.21.15.189:0/1334269203 >> v1:172.21.15.192:6789/0 conn(0x7fbc4c0d11b0 legacy=0x7fbc4c0d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:08.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc43fff700 1 -- 172.21.15.189:0/1334269203 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4c0cf200 con 0x7fbc4c0cb870 2024-04-06T12:12:08.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc43fff700 1 -- 172.21.15.189:0/1334269203 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fbc48001670 con 0x7fbc4c0cb870 2024-04-06T12:12:08.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc43fff700 1 -- 172.21.15.189:0/1334269203 >> v1:172.21.15.189:6789/0 conn(0x7fbc4c0cb870 legacy=0x7fbc4c15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:08.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc43fff700 1 --2- 172.21.15.189:0/1334269203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc38003a00 0x7fbc38003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:08.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc43fff700 1 --2- 172.21.15.189:0/1334269203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc38004850 0x7fbc38004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:08.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc43fff700 1 --2- 172.21.15.189:0/1334269203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc38005210 0x7fbc3800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:08.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc43fff700 1 -- 172.21.15.189:0/1334269203 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3800c180 con 0x7fbc38005210 2024-04-06T12:12:08.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc43fff700 1 -- 172.21.15.189:0/1334269203 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3800c2f0 con 0x7fbc38003a00 2024-04-06T12:12:08.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc43fff700 1 -- 172.21.15.189:0/1334269203 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3800c4b0 con 0x7fbc38004850 2024-04-06T12:12:08.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc51e58700 1 --2- 172.21.15.189:0/1334269203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc38005210 0x7fbc3800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:08.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc51657700 1 --2- 172.21.15.189:0/1334269203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc38003a00 0x7fbc38003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:08.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc50e56700 1 --2- 172.21.15.189:0/1334269203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc38004850 0x7fbc38004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:08.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc51e58700 1 -- 172.21.15.189:0/1334269203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc38004850 msgr2=0x7fbc38004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:08.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc51e58700 1 --2- 172.21.15.189:0/1334269203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc38004850 0x7fbc38004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc51e58700 1 -- 172.21.15.189:0/1334269203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc38003a00 msgr2=0x7fbc38003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:08.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc51e58700 1 --2- 172.21.15.189:0/1334269203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc38003a00 0x7fbc38003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.391+0000 7fbc51e58700 1 -- 172.21.15.189:0/1334269203 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc380035b0 con 0x7fbc38005210 2024-04-06T12:12:08.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.392+0000 7fbc51e58700 1 -- 172.21.15.189:0/1334269203 reap_dead start 2024-04-06T12:12:08.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.392+0000 7fbc51e58700 1 --2- 172.21.15.189:0/1334269203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc38005210 0x7fbc3800ba10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbc44009fc0 tx=0x7fbc38005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46be22033e5ead07 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:08.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.392+0000 7fbc43fff700 1 -- 172.21.15.189:0/1334269203 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc44003a20 con 0x7fbc38005210 2024-04-06T12:12:08.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.392+0000 7fbc43fff700 1 -- 172.21.15.189:0/1334269203 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc4400f040 con 0x7fbc38005210 2024-04-06T12:12:08.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.392+0000 7fbc43fff700 1 -- 172.21.15.189:0/1334269203 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc44003de0 con 0x7fbc38005210 2024-04-06T12:12:08.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.392+0000 7fbc538bb700 1 -- 172.21.15.189:0/1334269203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc38005210 msgr2=0x7fbc3800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:08.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.392+0000 7fbc538bb700 1 --2- 172.21.15.189:0/1334269203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc38005210 0x7fbc3800ba10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbc44009fc0 tx=0x7fbc38005880 comp rx=0 tx=0).stop 2024-04-06T12:12:08.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.392+0000 7fbc538bb700 1 -- 172.21.15.189:0/1334269203 shutdown_connections 2024-04-06T12:12:08.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.392+0000 7fbc538bb700 1 --2- 172.21.15.189:0/1334269203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc38005210 0x7fbc3800ba10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.392+0000 7fbc538bb700 1 -- 172.21.15.189:0/1334269203 >> 172.21.15.189:0/1334269203 conn(0x7fbc4c0b2db0 msgr2=0x7fbc4c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:08.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.392+0000 7fbc538bb700 1 -- 172.21.15.189:0/1334269203 shutdown_connections 2024-04-06T12:12:08.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.392+0000 7fbc538bb700 1 -- 172.21.15.189:0/1334269203 wait complete. 2024-04-06T12:12:08.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.394+0000 7fbc538bb700 1 Processor -- start 2024-04-06T12:12:08.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.394+0000 7fbc538bb700 1 -- start start 2024-04-06T12:12:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.394+0000 7fbc538bb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc38005210 0x7fbc4c153230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.394+0000 7fbc538bb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c0ca660 0x7fbc4c0caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.394+0000 7fbc538bb700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc4c0c4d50 0x7fbc4c0c51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.394+0000 7fbc538bb700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c0ca240 con 0x7fbc4c0ca660 2024-04-06T12:12:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.394+0000 7fbc538bb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c0c56f0 con 0x7fbc38005210 2024-04-06T12:12:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.394+0000 7fbc538bb700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c0c95c0 con 0x7fbc4c0c4d50 2024-04-06T12:12:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.394+0000 7fbc50e56700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c0ca660 0x7fbc4c0caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.394+0000 7fbc50e56700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c0ca660 0x7fbc4c0caac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44572/0 (socket says 172.21.15.189:44572) 2024-04-06T12:12:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.394+0000 7fbc50e56700 1 -- 172.21.15.189:0/1634040134 learned_addr learned my addr 172.21.15.189:0/1634040134 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.394+0000 7fbc51e58700 1 --2- 172.21.15.189:0/1634040134 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc4c0c4d50 0x7fbc4c0c51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.394+0000 7fbc51657700 1 --2- 172.21.15.189:0/1634040134 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc38005210 0x7fbc4c153230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.395+0000 7fbc50e56700 1 -- 172.21.15.189:0/1634040134 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc4c0c4d50 msgr2=0x7fbc4c0c51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:08.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.395+0000 7fbc50e56700 1 --2- 172.21.15.189:0/1634040134 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc4c0c4d50 0x7fbc4c0c51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.395+0000 7fbc50e56700 1 -- 172.21.15.189:0/1634040134 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc38005210 msgr2=0x7fbc4c153230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:08.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.395+0000 7fbc50e56700 1 --2- 172.21.15.189:0/1634040134 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc38005210 0x7fbc4c153230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.395+0000 7fbc50e56700 1 -- 172.21.15.189:0/1634040134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4c0c98a0 con 0x7fbc4c0ca660 2024-04-06T12:12:08.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.395+0000 7fbc51657700 1 --2- 172.21.15.189:0/1634040134 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc38005210 0x7fbc4c153230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:08.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.395+0000 7fbc50e56700 1 --2- 172.21.15.189:0/1634040134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c0ca660 0x7fbc4c0caac0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbc48007290 tx=0x7fbc480075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:08.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.395+0000 7fbc41ffb700 1 -- 172.21.15.189:0/1634040134 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc48011580 con 0x7fbc4c0ca660 2024-04-06T12:12:08.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.395+0000 7fbc41ffb700 1 -- 172.21.15.189:0/1634040134 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc480116e0 con 0x7fbc4c0ca660 2024-04-06T12:12:08.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.395+0000 7fbc538bb700 1 -- 172.21.15.189:0/1634040134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4c0c9b80 con 0x7fbc4c0ca660 2024-04-06T12:12:08.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.395+0000 7fbc41ffb700 1 -- 172.21.15.189:0/1634040134 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc48011a90 con 0x7fbc4c0ca660 2024-04-06T12:12:08.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.395+0000 7fbc538bb700 1 -- 172.21.15.189:0/1634040134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc4c153770 con 0x7fbc4c0ca660 2024-04-06T12:12:08.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.396+0000 7fbc538bb700 1 -- 172.21.15.189:0/1634040134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc4c0ca380 con 0x7fbc4c0ca660 2024-04-06T12:12:08.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.397+0000 7fbc41ffb700 1 -- 172.21.15.189:0/1634040134 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbc48011bf0 con 0x7fbc4c0ca660 2024-04-06T12:12:08.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.397+0000 7fbc41ffb700 1 --2- 172.21.15.189:0/1634040134 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc3006ac90 0x7fbc3006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:08.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.397+0000 7fbc41ffb700 1 -- 172.21.15.189:0/1634040134 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6428+0+0 (secure 0 0 0) 0x7fbc48089530 con 0x7fbc4c0ca660 2024-04-06T12:12:08.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.397+0000 7fbc51657700 1 --2- 172.21.15.189:0/1634040134 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc3006ac90 0x7fbc3006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:08.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.397+0000 7fbc51657700 1 --2- 172.21.15.189:0/1634040134 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc3006ac90 0x7fbc3006d150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c000c00 tx=0x7fbc3c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:08.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.401+0000 7fbc41ffb700 1 -- 172.21.15.189:0/1634040134 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbc4c0ca380 con 0x7fbc4c0ca660 2024-04-06T12:12:08.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.647+0000 7fbc538bb700 1 -- 172.21.15.189:0/1634040134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fbc4c0ca380 con 0x7fbc4c0ca660 2024-04-06T12:12:08.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.651+0000 7fbc41ffb700 1 -- 172.21.15.189:0/1634040134 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) v1 ==== 137+0+0 (secure 0 0 0) 0x7fbc4c0ca380 con 0x7fbc4c0ca660 2024-04-06T12:12:08.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.653+0000 7fbc538bb700 1 -- 172.21.15.189:0/1634040134 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc3006ac90 msgr2=0x7fbc3006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:08.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.653+0000 7fbc538bb700 1 --2- 172.21.15.189:0/1634040134 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc3006ac90 0x7fbc3006d150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c000c00 tx=0x7fbc3c009040 comp rx=0 tx=0).stop 2024-04-06T12:12:08.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.653+0000 7fbc538bb700 1 -- 172.21.15.189:0/1634040134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c0ca660 msgr2=0x7fbc4c0caac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:08.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.653+0000 7fbc538bb700 1 --2- 172.21.15.189:0/1634040134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c0ca660 0x7fbc4c0caac0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbc48007290 tx=0x7fbc480075a0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.653+0000 7fbc538bb700 1 -- 172.21.15.189:0/1634040134 shutdown_connections 2024-04-06T12:12:08.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.653+0000 7fbc538bb700 1 --2- 172.21.15.189:0/1634040134 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc4c0c4d50 0x7fbc4c0c51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.653+0000 7fbc538bb700 1 --2- 172.21.15.189:0/1634040134 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc3006ac90 0x7fbc3006d150 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.653+0000 7fbc538bb700 1 --2- 172.21.15.189:0/1634040134 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc38005210 0x7fbc4c153230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.653+0000 7fbc538bb700 1 --2- 172.21.15.189:0/1634040134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c0ca660 0x7fbc4c0caac0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.653+0000 7fbc538bb700 1 -- 172.21.15.189:0/1634040134 >> 172.21.15.189:0/1634040134 conn(0x7fbc4c0b2db0 msgr2=0x7fbc4c0b52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:08.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.653+0000 7fbc538bb700 1 -- 172.21.15.189:0/1634040134 shutdown_connections 2024-04-06T12:12:08.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.653+0000 7fbc538bb700 1 -- 172.21.15.189:0/1634040134 wait complete. 2024-04-06T12:12:08.654 INFO:teuthology.orchestra.run.smithi189.stderr:added data pool 4 to fsmap 2024-04-06T12:12:08.665 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:12:08.760 INFO:tasks.ceph.mds.b.smithi192.stderr:2024-04-06T12:12:08.761+0000 7fdcdd88c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T12:12:08.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.788+0000 7ff478a5d700 1 Processor -- start 2024-04-06T12:12:08.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.788+0000 7ff478a5d700 1 -- start start 2024-04-06T12:12:08.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.789+0000 7ff478a5d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff4740d11b0 0x7ff4740d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:08.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.789+0000 7ff478a5d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff4740cb870 0x7ff47415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:08.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.789+0000 7ff478a5d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4740d1ad0 con 0x7ff4740cfa00 2024-04-06T12:12:08.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.789+0000 7ff478a5d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4740ccdc0 con 0x7ff4740cb870 2024-04-06T12:12:08.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.789+0000 7ff478a5d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4740d0b50 con 0x7ff4740d11b0 2024-04-06T12:12:08.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.789+0000 7ff471d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff4740cb870 0x7ff47415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:08.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.789+0000 7ff471d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff4740cb870 0x7ff47415c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44582/0 (socket says 172.21.15.189:44582) 2024-04-06T12:12:08.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.789+0000 7ff471d9b700 1 -- 172.21.15.189:0/1813000924 learned_addr learned my addr 172.21.15.189:0/1813000924 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:08.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.789+0000 7ff47259c700 1 --2- 172.21.15.189:0/1813000924 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff4740d11b0 0x7ff4740d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:08.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.789+0000 7ff471d9b700 1 -- 172.21.15.189:0/1813000924 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff4740d11b0 msgr2=0x7ff4740d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:08.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.789+0000 7ff471d9b700 1 --2- 172.21.15.189:0/1813000924 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff4740d11b0 0x7ff4740d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.789+0000 7ff471d9b700 1 -- 172.21.15.189:0/1813000924 >> v1:172.21.15.192:6789/0 conn(0x7ff4740cfa00 legacy=0x7ff4740cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:08.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.789+0000 7ff471d9b700 1 -- 172.21.15.189:0/1813000924 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4740cbd80 con 0x7ff4740cb870 2024-04-06T12:12:08.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.789+0000 7ff471d9b700 1 --2- 172.21.15.189:0/1813000924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4740cb870 0x7ff47415c280 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff4680098f0 tx=0x7ff468009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442eed5e390903b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:08.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.789+0000 7ff47159a700 1 -- 172.21.15.189:0/1813000924 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46800b8a0 con 0x7ff4740cb870 2024-04-06T12:12:08.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.789+0000 7ff47159a700 1 -- 172.21.15.189:0/1813000924 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46800ba00 con 0x7ff4740cb870 2024-04-06T12:12:08.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.789+0000 7ff47159a700 1 -- 172.21.15.189:0/1813000924 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46800bd90 con 0x7ff4740cb870 2024-04-06T12:12:08.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.790+0000 7ff478a5d700 1 -- 172.21.15.189:0/1813000924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4740cb870 msgr2=0x7ff47415c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:08.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.790+0000 7ff478a5d700 1 --2- 172.21.15.189:0/1813000924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4740cb870 0x7ff47415c280 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff4680098f0 tx=0x7ff468009c00 comp rx=0 tx=0).stop 2024-04-06T12:12:08.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.790+0000 7ff478a5d700 1 -- 172.21.15.189:0/1813000924 shutdown_connections 2024-04-06T12:12:08.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.790+0000 7ff478a5d700 1 --2- 172.21.15.189:0/1813000924 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff4740d11b0 0x7ff4740d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.790+0000 7ff478a5d700 1 --2- 172.21.15.189:0/1813000924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4740cb870 0x7ff47415c280 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.790+0000 7ff478a5d700 1 -- 172.21.15.189:0/1813000924 >> 172.21.15.189:0/1813000924 conn(0x7ff4740b2db0 msgr2=0x7ff4740b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:08.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.790+0000 7ff478a5d700 1 -- 172.21.15.189:0/1813000924 shutdown_connections 2024-04-06T12:12:08.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.790+0000 7ff478a5d700 1 -- 172.21.15.189:0/1813000924 wait complete. 2024-04-06T12:12:08.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.792+0000 7ff478a5d700 1 Processor -- start 2024-04-06T12:12:08.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.792+0000 7ff478a5d700 1 -- start start 2024-04-06T12:12:08.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff478a5d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4740cb870 0x7ff474155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:08.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff478a5d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4740cfa00 0x7ff474158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:08.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff478a5d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff4740d11b0 0x7ff4740c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:08.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff478a5d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4740c5d40 con 0x7ff4740cfa00 2024-04-06T12:12:08.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff478a5d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4740c5130 con 0x7ff4740cb870 2024-04-06T12:12:08.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff478a5d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4740c5410 con 0x7ff4740d11b0 2024-04-06T12:12:08.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff471d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4740cfa00 0x7ff474158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:08.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff471d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4740cfa00 0x7ff474158710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44592/0 (socket says 172.21.15.189:44592) 2024-04-06T12:12:08.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff471d9b700 1 -- 172.21.15.189:0/3680103237 learned_addr learned my addr 172.21.15.189:0/3680103237 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:08.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff47259c700 1 --2- 172.21.15.189:0/3680103237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4740cb870 0x7ff474155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:08.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff472d9d700 1 --2- 172.21.15.189:0/3680103237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff4740d11b0 0x7ff4740c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:08.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff471d9b700 1 -- 172.21.15.189:0/3680103237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff4740d11b0 msgr2=0x7ff4740c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:08.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff471d9b700 1 --2- 172.21.15.189:0/3680103237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff4740d11b0 0x7ff4740c61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff471d9b700 1 -- 172.21.15.189:0/3680103237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4740cb870 msgr2=0x7ff474155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:08.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff471d9b700 1 --2- 172.21.15.189:0/3680103237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4740cb870 0x7ff474155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:08.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff471d9b700 1 -- 172.21.15.189:0/3680103237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff468009550 con 0x7ff4740cfa00 2024-04-06T12:12:08.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff471d9b700 1 --2- 172.21.15.189:0/3680103237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4740cfa00 0x7ff474158710 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff468000c00 tx=0x7ff468004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:08.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff462ffd700 1 -- 172.21.15.189:0/3680103237 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468004300 con 0x7ff4740cfa00 2024-04-06T12:12:08.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff462ffd700 1 -- 172.21.15.189:0/3680103237 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46801f070 con 0x7ff4740cfa00 2024-04-06T12:12:08.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff478a5d700 1 -- 172.21.15.189:0/3680103237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4740c56f0 con 0x7ff4740cfa00 2024-04-06T12:12:08.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff462ffd700 1 -- 172.21.15.189:0/3680103237 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4680036a0 con 0x7ff4740cfa00 2024-04-06T12:12:08.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.793+0000 7ff478a5d700 1 -- 172.21.15.189:0/3680103237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff474155bd0 con 0x7ff4740cfa00 2024-04-06T12:12:08.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.794+0000 7ff478a5d700 1 -- 172.21.15.189:0/3680103237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff474074990 con 0x7ff4740cfa00 2024-04-06T12:12:08.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.795+0000 7ff462ffd700 1 -- 172.21.15.189:0/3680103237 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff468021030 con 0x7ff4740cfa00 2024-04-06T12:12:08.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.795+0000 7ff462ffd700 1 --2- 172.21.15.189:0/3680103237 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff45406acc0 0x7ff45406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:08.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.795+0000 7ff462ffd700 1 -- 172.21.15.189:0/3680103237 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff46808a7b0 con 0x7ff4740cfa00 2024-04-06T12:12:08.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.795+0000 7ff47259c700 1 --2- 172.21.15.189:0/3680103237 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff45406acc0 0x7ff45406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:08.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.795+0000 7ff47259c700 1 --2- 172.21.15.189:0/3680103237 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff45406acc0 0x7ff45406d180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff464007a30 tx=0x7ff464008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:08.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:08.799+0000 7ff462ffd700 1 -- 172.21.15.189:0/3680103237 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff468055730 con 0x7ff4740cfa00 2024-04-06T12:12:09.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.040+0000 7ff478a5d700 1 -- 172.21.15.189:0/3680103237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4740b6ec0 con 0x7ff4740cfa00 2024-04-06T12:12:09.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.040+0000 7ff462ffd700 1 -- 172.21.15.189:0/3680103237 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4739 (secure 0 0 0) 0x7ff468058d50 con 0x7ff4740cfa00 2024-04-06T12:12:09.040 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:09.040 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2769778155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2769778155},{"type":"v1","addr":"172.21.15.189:6835","nonce":2769778155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4233,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3550531296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3550531296},{"type":"v1","addr":"172.21.15.189:6837","nonce":3550531296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3198235148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3198235148},{"type":"v1","addr":"172.21.15.192:6833","nonce":3198235148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:12:04.195581+0000","modified":"2024-04-06T12:12:08.649444+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/516212261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":516212261},{"type":"v1","addr":"172.21.15.192:6835","nonce":516212261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T12:12:09.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.043+0000 7ff478a5d700 1 -- 172.21.15.189:0/3680103237 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff45406acc0 msgr2=0x7ff45406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.043+0000 7ff478a5d700 1 --2- 172.21.15.189:0/3680103237 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff45406acc0 0x7ff45406d180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff464007a30 tx=0x7ff464008040 comp rx=0 tx=0).stop 2024-04-06T12:12:09.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.043+0000 7ff478a5d700 1 -- 172.21.15.189:0/3680103237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4740cfa00 msgr2=0x7ff474158710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.043+0000 7ff478a5d700 1 --2- 172.21.15.189:0/3680103237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4740cfa00 0x7ff474158710 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff468000c00 tx=0x7ff468004750 comp rx=0 tx=0).stop 2024-04-06T12:12:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.043+0000 7ff478a5d700 1 -- 172.21.15.189:0/3680103237 shutdown_connections 2024-04-06T12:12:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.043+0000 7ff478a5d700 1 --2- 172.21.15.189:0/3680103237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff4740d11b0 0x7ff4740c61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.043+0000 7ff478a5d700 1 --2- 172.21.15.189:0/3680103237 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff45406acc0 0x7ff45406d180 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.043+0000 7ff478a5d700 1 --2- 172.21.15.189:0/3680103237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4740cb870 0x7ff474155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.043+0000 7ff478a5d700 1 --2- 172.21.15.189:0/3680103237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4740cfa00 0x7ff474158710 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.043+0000 7ff478a5d700 1 -- 172.21.15.189:0/3680103237 >> 172.21.15.189:0/3680103237 conn(0x7ff4740b2db0 msgr2=0x7ff4740cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.043+0000 7ff478a5d700 1 -- 172.21.15.189:0/3680103237 shutdown_connections 2024-04-06T12:12:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.043+0000 7ff478a5d700 1 -- 172.21.15.189:0/3680103237 wait complete. 2024-04-06T12:12:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 7 2024-04-06T12:12:09.054 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:09.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.177+0000 7f17c8ed5700 1 Processor -- start 2024-04-06T12:12:09.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.177+0000 7f17c8ed5700 1 -- start start 2024-04-06T12:12:09.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.178+0000 7f17c8ed5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f17c40cedf0 0x7f17c40cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:09.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.178+0000 7f17c8ed5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c40cf730 con 0x7f17c40d17a0 2024-04-06T12:12:09.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.178+0000 7f17c8ed5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c406d850 con 0x7f17c4146dd0 2024-04-06T12:12:09.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.178+0000 7f17c8ed5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c406beb0 con 0x7f17c40cedf0 2024-04-06T12:12:09.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.178+0000 7f17c2d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f17c40cedf0 0x7f17c40cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:09.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.178+0000 7f17c259c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f17c40d17a0 0x7f17c406b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58964/0 (socket says 172.21.15.189:58964) 2024-04-06T12:12:09.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.178+0000 7f17c259c700 1 -- 172.21.15.189:0/2277883141 learned_addr learned my addr 172.21.15.189:0/2277883141 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:09.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.178+0000 7f17c2d9d700 1 -- 172.21.15.189:0/2277883141 >> v1:172.21.15.192:6789/0 conn(0x7f17c4146dd0 legacy=0x7f17c406d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.178+0000 7f17c2d9d700 1 -- 172.21.15.189:0/2277883141 >> v1:172.21.15.189:6789/0 conn(0x7f17c40d17a0 legacy=0x7f17c406b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.178+0000 7f17c2d9d700 1 -- 172.21.15.189:0/2277883141 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c406c0a0 con 0x7f17c40cedf0 2024-04-06T12:12:09.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.178+0000 7f17c2d9d700 1 --2- 172.21.15.189:0/2277883141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17c40cedf0 0x7f17c40cf1f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009d90 tx=0x7f17bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=644a0144dc0c1a6a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:09.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.178+0000 7f17c159a700 1 -- 172.21.15.189:0/2277883141 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc00e070 con 0x7f17c40cedf0 2024-04-06T12:12:09.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.178+0000 7f17c159a700 1 -- 172.21.15.189:0/2277883141 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17bc00bbd0 con 0x7f17c40cedf0 2024-04-06T12:12:09.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.178+0000 7f17c159a700 1 -- 172.21.15.189:0/2277883141 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc012430 con 0x7f17c40cedf0 2024-04-06T12:12:09.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.179+0000 7f17c8ed5700 1 -- 172.21.15.189:0/2277883141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17c40cedf0 msgr2=0x7f17c40cf1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.179+0000 7f17c8ed5700 1 --2- 172.21.15.189:0/2277883141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17c40cedf0 0x7f17c40cf1f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009d90 tx=0x7f17bc00b330 comp rx=0 tx=0).stop 2024-04-06T12:12:09.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.179+0000 7f17c8ed5700 1 -- 172.21.15.189:0/2277883141 shutdown_connections 2024-04-06T12:12:09.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.179+0000 7f17c8ed5700 1 --2- 172.21.15.189:0/2277883141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17c40cedf0 0x7f17c40cf1f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.179+0000 7f17c8ed5700 1 -- 172.21.15.189:0/2277883141 >> 172.21.15.189:0/2277883141 conn(0x7f17c40b2db0 msgr2=0x7f17c40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:09.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.179+0000 7f17c8ed5700 1 -- 172.21.15.189:0/2277883141 shutdown_connections 2024-04-06T12:12:09.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.179+0000 7f17c8ed5700 1 -- 172.21.15.189:0/2277883141 wait complete. 2024-04-06T12:12:09.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.180+0000 7f17c8ed5700 1 Processor -- start 2024-04-06T12:12:09.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.180+0000 7f17c8ed5700 1 -- start start 2024-04-06T12:12:09.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.181+0000 7f17c8ed5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17c40cedf0 0x7f17c41553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:09.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.181+0000 7f17c8ed5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17c40d17a0 0x7f17c415edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:09.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.181+0000 7f17c8ed5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17c4146dd0 0x7f17c40c7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:09.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.181+0000 7f17c8ed5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c40ada30 con 0x7f17c4146dd0 2024-04-06T12:12:09.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.181+0000 7f17c8ed5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c415f300 con 0x7f17c40d17a0 2024-04-06T12:12:09.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.181+0000 7f17c8ed5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c40af820 con 0x7f17c40cedf0 2024-04-06T12:12:09.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.181+0000 7f17c2d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17c4146dd0 0x7f17c40c7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:09.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.181+0000 7f17c2d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17c4146dd0 0x7f17c40c7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44612/0 (socket says 172.21.15.189:44612) 2024-04-06T12:12:09.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.181+0000 7f17c2d9d700 1 -- 172.21.15.189:0/2457381377 learned_addr learned my addr 172.21.15.189:0/2457381377 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:09.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.181+0000 7f17c259c700 1 --2- 172.21.15.189:0/2457381377 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17c40cedf0 0x7f17c41553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:09.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.181+0000 7f17c1d9b700 1 --2- 172.21.15.189:0/2457381377 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17c40d17a0 0x7f17c415edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:09.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.182+0000 7f17c2d9d700 1 -- 172.21.15.189:0/2457381377 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17c40cedf0 msgr2=0x7f17c41553f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.182+0000 7f17c2d9d700 1 --2- 172.21.15.189:0/2457381377 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17c40cedf0 0x7f17c41553f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.182+0000 7f17c2d9d700 1 -- 172.21.15.189:0/2457381377 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17c40d17a0 msgr2=0x7f17c415edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.182+0000 7f17c2d9d700 1 --2- 172.21.15.189:0/2457381377 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17c40d17a0 0x7f17c415edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.182+0000 7f17c2d9d700 1 -- 172.21.15.189:0/2457381377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17bc009a70 con 0x7f17c4146dd0 2024-04-06T12:12:09.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.182+0000 7f17c2d9d700 1 --2- 172.21.15.189:0/2457381377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17c4146dd0 0x7f17c40c7b00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f17c40677e0 tx=0x7f17bc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:09.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.182+0000 7f17b2ffd700 1 -- 172.21.15.189:0/2457381377 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc012d20 con 0x7f17c4146dd0 2024-04-06T12:12:09.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.182+0000 7f17b2ffd700 1 -- 172.21.15.189:0/2457381377 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17bc020070 con 0x7f17c4146dd0 2024-04-06T12:12:09.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.182+0000 7f17c8ed5700 1 -- 172.21.15.189:0/2457381377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c40acdb0 con 0x7f17c4146dd0 2024-04-06T12:12:09.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.182+0000 7f17b2ffd700 1 -- 172.21.15.189:0/2457381377 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc012ea0 con 0x7f17c4146dd0 2024-04-06T12:12:09.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.182+0000 7f17c8ed5700 1 -- 172.21.15.189:0/2457381377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c40ad370 con 0x7f17c4146dd0 2024-04-06T12:12:09.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.183+0000 7f17c8ed5700 1 -- 172.21.15.189:0/2457381377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c40d1610 con 0x7f17c4146dd0 2024-04-06T12:12:09.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.183+0000 7f17b2ffd700 1 -- 172.21.15.189:0/2457381377 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f17bc004280 con 0x7f17c4146dd0 2024-04-06T12:12:09.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.184+0000 7f17b2ffd700 1 --2- 172.21.15.189:0/2457381377 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f17a406ac90 0x7f17a406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:09.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.184+0000 7f17b2ffd700 1 -- 172.21.15.189:0/2457381377 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f17bc022070 con 0x7f17c4146dd0 2024-04-06T12:12:09.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.184+0000 7f17c259c700 1 --2- 172.21.15.189:0/2457381377 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f17a406ac90 0x7f17a406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:09.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.184+0000 7f17c259c700 1 --2- 172.21.15.189:0/2457381377 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f17a406ac90 0x7f17a406d150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f17b4005910 tx=0x7f17b4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:09.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.188+0000 7f17b2ffd700 1 -- 172.21.15.189:0/2457381377 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f17bc056280 con 0x7f17c4146dd0 2024-04-06T12:12:09.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.390+0000 7f17c8ed5700 1 -- 172.21.15.189:0/2457381377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17c40adb70 con 0x7f17c4146dd0 2024-04-06T12:12:09.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.392+0000 7f17b2ffd700 1 -- 172.21.15.189:0/2457381377 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f17c40adb70 con 0x7f17c4146dd0 2024-04-06T12:12:09.391 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:09.391 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:08.649170+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:09.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.394+0000 7f17c8ed5700 1 -- 172.21.15.189:0/2457381377 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f17a406ac90 msgr2=0x7f17a406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.394+0000 7f17c8ed5700 1 --2- 172.21.15.189:0/2457381377 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f17a406ac90 0x7f17a406d150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f17b4005910 tx=0x7f17b4009040 comp rx=0 tx=0).stop 2024-04-06T12:12:09.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.394+0000 7f17c8ed5700 1 -- 172.21.15.189:0/2457381377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17c4146dd0 msgr2=0x7f17c40c7b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.394+0000 7f17c8ed5700 1 --2- 172.21.15.189:0/2457381377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17c4146dd0 0x7f17c40c7b00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f17c40677e0 tx=0x7f17bc004ad0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.394+0000 7f17c8ed5700 1 -- 172.21.15.189:0/2457381377 shutdown_connections 2024-04-06T12:12:09.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.394+0000 7f17c8ed5700 1 --2- 172.21.15.189:0/2457381377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17c4146dd0 0x7f17c40c7b00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.394+0000 7f17c8ed5700 1 --2- 172.21.15.189:0/2457381377 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17c40cedf0 0x7f17c41553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.394+0000 7f17c8ed5700 1 --2- 172.21.15.189:0/2457381377 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f17a406ac90 0x7f17a406d150 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.394+0000 7f17c8ed5700 1 --2- 172.21.15.189:0/2457381377 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17c40d17a0 0x7f17c415edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.394+0000 7f17c8ed5700 1 -- 172.21.15.189:0/2457381377 >> 172.21.15.189:0/2457381377 conn(0x7f17c40b2db0 msgr2=0x7f17c40b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:09.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.394+0000 7f17c8ed5700 1 -- 172.21.15.189:0/2457381377 shutdown_connections 2024-04-06T12:12:09.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.394+0000 7f17c8ed5700 1 -- 172.21.15.189:0/2457381377 wait complete. 2024-04-06T12:12:09.406 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:09.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.528+0000 7f3f4413b700 1 Processor -- start 2024-04-06T12:12:09.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.529+0000 7f3f4413b700 1 -- start start 2024-04-06T12:12:09.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.529+0000 7f3f4413b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f3c15d7b0 0x7f3f3c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:09.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.529+0000 7f3f4413b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3f3c15e500 0x7f3f3c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:09.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.529+0000 7f3f4413b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c0b3990 con 0x7f3f3c06a730 2024-04-06T12:12:09.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.529+0000 7f3f4413b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f3c06b0f0 con 0x7f3f3c15e500 2024-04-06T12:12:09.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.529+0000 7f3f4413b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c0b3060 con 0x7f3f3c15d7b0 2024-04-06T12:12:09.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f416d6700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3f3c06a730 0x7f3f3c15ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55444/0 (socket says 172.21.15.189:55444) 2024-04-06T12:12:09.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f416d6700 1 -- 172.21.15.189:0/4138610788 learned_addr learned my addr 172.21.15.189:0/4138610788 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:09.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f40ed5700 1 -- 172.21.15.189:0/4138610788 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3479626623 0 0) 0x7f3f3c0b3990 con 0x7f3f3c06a730 2024-04-06T12:12:09.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f40ed5700 1 -- 172.21.15.189:0/4138610788 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f280035b0 con 0x7f3f3c06a730 2024-04-06T12:12:09.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f426d8700 1 --2- 172.21.15.189:0/4138610788 >> v2:172.21.15.192:3300/0 conn(0x7f3f3c15e500 0x7f3f3c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:09.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f41ed7700 1 --2- 172.21.15.189:0/4138610788 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f3c15d7b0 0x7f3f3c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:09.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f40ed5700 1 -- 172.21.15.189:0/4138610788 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 219797196 0 0) 0x7f3f280035b0 con 0x7f3f3c06a730 2024-04-06T12:12:09.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f40ed5700 1 -- 172.21.15.189:0/4138610788 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f3c15d7b0 msgr2=0x7f3f3c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:09.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f40ed5700 1 --2- 172.21.15.189:0/4138610788 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f3c15d7b0 0x7f3f3c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f40ed5700 1 -- 172.21.15.189:0/4138610788 >> v2:172.21.15.192:3300/0 conn(0x7f3f3c15e500 msgr2=0x7f3f3c06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:09.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f40ed5700 1 --2- 172.21.15.189:0/4138610788 >> v2:172.21.15.192:3300/0 conn(0x7f3f3c15e500 0x7f3f3c06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f40ed5700 1 -- 172.21.15.189:0/4138610788 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c0b3340 con 0x7f3f3c06a730 2024-04-06T12:12:09.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f40ed5700 1 -- 172.21.15.189:0/4138610788 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f3f2c001670 con 0x7f3f3c06a730 2024-04-06T12:12:09.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f40ed5700 1 -- 172.21.15.189:0/4138610788 >> v1:172.21.15.189:6789/0 conn(0x7f3f3c06a730 legacy=0x7f3f3c15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f40ed5700 1 --2- 172.21.15.189:0/4138610788 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f3c15d7b0 0x7f3f28003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:09.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f40ed5700 1 --2- 172.21.15.189:0/4138610788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f28003f80 0x7f3f28004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:09.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f40ed5700 1 --2- 172.21.15.189:0/4138610788 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f28004d90 0x7f3f28005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:09.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f40ed5700 1 -- 172.21.15.189:0/4138610788 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c06b0f0 con 0x7f3f28003f80 2024-04-06T12:12:09.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f40ed5700 1 -- 172.21.15.189:0/4138610788 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c0b3060 con 0x7f3f28004d90 2024-04-06T12:12:09.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f40ed5700 1 -- 172.21.15.189:0/4138610788 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2800ba60 con 0x7f3f3c15d7b0 2024-04-06T12:12:09.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f416d6700 1 --2- 172.21.15.189:0/4138610788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f28003f80 0x7f3f28004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:09.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f426d8700 1 --2- 172.21.15.189:0/4138610788 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f28004d90 0x7f3f28005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:09.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.530+0000 7f3f41ed7700 1 --2- 172.21.15.189:0/4138610788 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f3c15d7b0 0x7f3f28003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:09.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.531+0000 7f3f416d6700 1 -- 172.21.15.189:0/4138610788 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f3c15d7b0 msgr2=0x7f3f28003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:09.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.531+0000 7f3f416d6700 1 --2- 172.21.15.189:0/4138610788 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f3c15d7b0 0x7f3f28003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.531+0000 7f3f416d6700 1 -- 172.21.15.189:0/4138610788 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f28004d90 msgr2=0x7f3f28005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:09.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.531+0000 7f3f416d6700 1 --2- 172.21.15.189:0/4138610788 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f28004d90 0x7f3f28005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.531+0000 7f3f416d6700 1 -- 172.21.15.189:0/4138610788 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f280035b0 con 0x7f3f28003f80 2024-04-06T12:12:09.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.531+0000 7f3f416d6700 1 --2- 172.21.15.189:0/4138610788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f28003f80 0x7f3f28004850 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c00c3a0 tx=0x7f3f28005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a2c967841e53ff6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:09.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.531+0000 7f3f40ed5700 1 -- 172.21.15.189:0/4138610788 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c007c60 con 0x7f3f28003f80 2024-04-06T12:12:09.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.531+0000 7f3f40ed5700 1 -- 172.21.15.189:0/4138610788 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2c005960 con 0x7f3f28003f80 2024-04-06T12:12:09.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.531+0000 7f3f40ed5700 1 -- 172.21.15.189:0/4138610788 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c00cea0 con 0x7f3f28003f80 2024-04-06T12:12:09.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.531+0000 7f3f4413b700 1 -- 172.21.15.189:0/4138610788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f28003f80 msgr2=0x7f3f28004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.531+0000 7f3f4413b700 1 --2- 172.21.15.189:0/4138610788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f28003f80 0x7f3f28004850 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c00c3a0 tx=0x7f3f28005880 comp rx=0 tx=0).stop 2024-04-06T12:12:09.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.531+0000 7f3f426d8700 1 -- 172.21.15.189:0/4138610788 reap_dead start 2024-04-06T12:12:09.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.531+0000 7f3f4413b700 1 -- 172.21.15.189:0/4138610788 shutdown_connections 2024-04-06T12:12:09.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.531+0000 7f3f4413b700 1 -- 172.21.15.189:0/4138610788 >> 172.21.15.189:0/4138610788 conn(0x7f3f3c0d00b0 msgr2=0x7f3f3c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:09.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.531+0000 7f3f4413b700 1 -- 172.21.15.189:0/4138610788 shutdown_connections 2024-04-06T12:12:09.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.531+0000 7f3f4413b700 1 -- 172.21.15.189:0/4138610788 wait complete. 2024-04-06T12:12:09.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.533+0000 7f3f4413b700 1 Processor -- start 2024-04-06T12:12:09.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.533+0000 7f3f4413b700 1 -- start start 2024-04-06T12:12:09.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.533+0000 7f3f4413b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f28003f80 0x7f3f3c153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:09.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.533+0000 7f3f4413b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f3c150e20 0x7f3f3c14ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:09.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.533+0000 7f3f4413b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f3c0ccfb0 0x7f3f3c15bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:09.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.533+0000 7f3f4413b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c1541d0 con 0x7f3f3c0ccfb0 2024-04-06T12:12:09.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.533+0000 7f3f4413b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c1504a0 con 0x7f3f3c150e20 2024-04-06T12:12:09.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.533+0000 7f3f4413b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c15a9c0 con 0x7f3f28003f80 2024-04-06T12:12:09.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.533+0000 7f3f426d8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f3c0ccfb0 0x7f3f3c15bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:09.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.534+0000 7f3f426d8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f3c0ccfb0 0x7f3f3c15bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55960/0 (socket says 172.21.15.189:55960) 2024-04-06T12:12:09.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.534+0000 7f3f426d8700 1 -- 172.21.15.189:0/2114890945 learned_addr learned my addr 172.21.15.189:0/2114890945 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:09.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.534+0000 7f3f416d6700 1 --2- 172.21.15.189:0/2114890945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f3c150e20 0x7f3f3c14ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:09.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.534+0000 7f3f41ed7700 1 --2- 172.21.15.189:0/2114890945 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f28003f80 0x7f3f3c153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:09.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.534+0000 7f3f426d8700 1 -- 172.21.15.189:0/2114890945 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f28003f80 msgr2=0x7f3f3c153c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.534+0000 7f3f426d8700 1 --2- 172.21.15.189:0/2114890945 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f28003f80 0x7f3f3c153c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.534+0000 7f3f426d8700 1 -- 172.21.15.189:0/2114890945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f3c150e20 msgr2=0x7f3f3c14ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.534+0000 7f3f426d8700 1 --2- 172.21.15.189:0/2114890945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f3c150e20 0x7f3f3c14ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.534+0000 7f3f426d8700 1 -- 172.21.15.189:0/2114890945 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c15aca0 con 0x7f3f3c0ccfb0 2024-04-06T12:12:09.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.534+0000 7f3f416d6700 1 --2- 172.21.15.189:0/2114890945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f3c150e20 0x7f3f3c14ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:09.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.534+0000 7f3f426d8700 1 --2- 172.21.15.189:0/2114890945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f3c0ccfb0 0x7f3f3c15bd40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c06d2c0 tx=0x7f3f34011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:09.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.534+0000 7f3f327fc700 1 -- 172.21.15.189:0/2114890945 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3400e040 con 0x7f3f3c0ccfb0 2024-04-06T12:12:09.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.534+0000 7f3f327fc700 1 -- 172.21.15.189:0/2114890945 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3400f040 con 0x7f3f3c0ccfb0 2024-04-06T12:12:09.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.534+0000 7f3f4413b700 1 -- 172.21.15.189:0/2114890945 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3c15af80 con 0x7f3f3c0ccfb0 2024-04-06T12:12:09.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.534+0000 7f3f327fc700 1 -- 172.21.15.189:0/2114890945 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3400fdb0 con 0x7f3f3c0ccfb0 2024-04-06T12:12:09.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.534+0000 7f3f4413b700 1 -- 172.21.15.189:0/2114890945 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3c15b540 con 0x7f3f3c0ccfb0 2024-04-06T12:12:09.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.535+0000 7f3f4413b700 1 -- 172.21.15.189:0/2114890945 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f3c067bb0 con 0x7f3f3c0ccfb0 2024-04-06T12:12:09.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.535+0000 7f3f327fc700 1 -- 172.21.15.189:0/2114890945 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3f34015430 con 0x7f3f3c0ccfb0 2024-04-06T12:12:09.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.536+0000 7f3f327fc700 1 --2- 172.21.15.189:0/2114890945 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3f2006ac90 0x7f3f2006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:09.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.536+0000 7f3f327fc700 1 -- 172.21.15.189:0/2114890945 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3f34089b50 con 0x7f3f3c0ccfb0 2024-04-06T12:12:09.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.536+0000 7f3f41ed7700 1 --2- 172.21.15.189:0/2114890945 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3f2006ac90 0x7f3f2006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:09.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.536+0000 7f3f41ed7700 1 --2- 172.21.15.189:0/2114890945 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3f2006ac90 0x7f3f2006d150 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009a60 tx=0x7f3f38008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:09.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.540+0000 7f3f327fc700 1 -- 172.21.15.189:0/2114890945 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3f34054ad0 con 0x7f3f3c0ccfb0 2024-04-06T12:12:09.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.736+0000 7f3f4413b700 1 -- 172.21.15.189:0/2114890945 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f3c074990 con 0x7f3f3c0ccfb0 2024-04-06T12:12:09.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.737+0000 7f3f327fc700 1 -- 172.21.15.189:0/2114890945 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f3f340580f0 con 0x7f3f3c0ccfb0 2024-04-06T12:12:09.736 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:09.736 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:08.649170+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:09.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.739+0000 7f3f4413b700 1 -- 172.21.15.189:0/2114890945 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3f2006ac90 msgr2=0x7f3f2006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.739+0000 7f3f4413b700 1 --2- 172.21.15.189:0/2114890945 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3f2006ac90 0x7f3f2006d150 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009a60 tx=0x7f3f38008040 comp rx=0 tx=0).stop 2024-04-06T12:12:09.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.739+0000 7f3f4413b700 1 -- 172.21.15.189:0/2114890945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f3c0ccfb0 msgr2=0x7f3f3c15bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.739+0000 7f3f4413b700 1 --2- 172.21.15.189:0/2114890945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f3c0ccfb0 0x7f3f3c15bd40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c06d2c0 tx=0x7f3f34011f80 comp rx=0 tx=0).stop 2024-04-06T12:12:09.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.739+0000 7f3f4413b700 1 -- 172.21.15.189:0/2114890945 shutdown_connections 2024-04-06T12:12:09.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.739+0000 7f3f4413b700 1 --2- 172.21.15.189:0/2114890945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f3c0ccfb0 0x7f3f3c15bd40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.739+0000 7f3f4413b700 1 --2- 172.21.15.189:0/2114890945 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f28003f80 0x7f3f3c153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.739+0000 7f3f4413b700 1 --2- 172.21.15.189:0/2114890945 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3f2006ac90 0x7f3f2006d150 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.739+0000 7f3f4413b700 1 --2- 172.21.15.189:0/2114890945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f3c150e20 0x7f3f3c14ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.739+0000 7f3f4413b700 1 -- 172.21.15.189:0/2114890945 >> 172.21.15.189:0/2114890945 conn(0x7f3f3c0d00b0 msgr2=0x7f3f3c06fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:09.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.739+0000 7f3f4413b700 1 -- 172.21.15.189:0/2114890945 shutdown_connections 2024-04-06T12:12:09.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.739+0000 7f3f4413b700 1 -- 172.21.15.189:0/2114890945 wait complete. 2024-04-06T12:12:09.765 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:12:09.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.887+0000 7f9826ded700 1 Processor -- start 2024-04-06T12:12:09.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.887+0000 7f9826ded700 1 -- start start 2024-04-06T12:12:09.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.888+0000 7f9826ded700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f98200d11b0 0x7f98200d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:09.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.888+0000 7f9826ded700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98200d1ad0 con 0x7f98200cfa00 2024-04-06T12:12:09.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.888+0000 7f9826ded700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98200cbd80 con 0x7f98200cb870 2024-04-06T12:12:09.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.888+0000 7f9826ded700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98200ccdc0 con 0x7f98200d11b0 2024-04-06T12:12:09.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.888+0000 7f982538a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f98200cfa00 0x7f98200cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55446/0 (socket says 172.21.15.189:55446) 2024-04-06T12:12:09.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.888+0000 7f982538a700 1 -- 172.21.15.189:0/2894227664 learned_addr learned my addr 172.21.15.189:0/2894227664 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:09.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.888+0000 7f9824b89700 1 --2- 172.21.15.189:0/2894227664 >> v2:172.21.15.192:3300/0 conn(0x7f98200d11b0 0x7f98200d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:09.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.888+0000 7f981f7fe700 1 -- 172.21.15.189:0/2894227664 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3519050573 0 0) 0x7f98200d1ad0 con 0x7f98200cfa00 2024-04-06T12:12:09.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.888+0000 7f981f7fe700 1 -- 172.21.15.189:0/2894227664 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98100035b0 con 0x7f98200cfa00 2024-04-06T12:12:09.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.888+0000 7f981f7fe700 1 -- 172.21.15.189:0/2894227664 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3281991243 0 0) 0x7f98100035b0 con 0x7f98200cfa00 2024-04-06T12:12:09.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.888+0000 7f9824b89700 1 -- 172.21.15.189:0/2894227664 >> v1:172.21.15.192:6789/0 conn(0x7f98200cb870 legacy=0x7f982015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.889+0000 7f9824b89700 1 -- 172.21.15.189:0/2894227664 >> v1:172.21.15.189:6789/0 conn(0x7f98200cfa00 legacy=0x7f98200cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.889+0000 7f9824b89700 1 -- 172.21.15.189:0/2894227664 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98200cef20 con 0x7f98200d11b0 2024-04-06T12:12:09.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.889+0000 7f9824b89700 1 --2- 172.21.15.189:0/2894227664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200d11b0 0x7f98200d1590 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9814009840 tx=0x7f9814009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53f0f1bd8f5b35fa server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:09.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.889+0000 7f981f7fe700 1 -- 172.21.15.189:0/2894227664 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98180019f0 con 0x7f98200d11b0 2024-04-06T12:12:09.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.889+0000 7f981f7fe700 1 -- 172.21.15.189:0/2894227664 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9814004030 con 0x7f98200d11b0 2024-04-06T12:12:09.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.889+0000 7f981f7fe700 1 -- 172.21.15.189:0/2894227664 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98140043a0 con 0x7f98200d11b0 2024-04-06T12:12:09.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.889+0000 7f9826ded700 1 -- 172.21.15.189:0/2894227664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200d11b0 msgr2=0x7f98200d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.889+0000 7f9826ded700 1 --2- 172.21.15.189:0/2894227664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200d11b0 0x7f98200d1590 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9814009840 tx=0x7f9814009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:09.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.889+0000 7f9826ded700 1 -- 172.21.15.189:0/2894227664 shutdown_connections 2024-04-06T12:12:09.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.889+0000 7f9826ded700 1 --2- 172.21.15.189:0/2894227664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200d11b0 0x7f98200d1590 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.889+0000 7f9826ded700 1 -- 172.21.15.189:0/2894227664 >> 172.21.15.189:0/2894227664 conn(0x7f98200b2db0 msgr2=0x7f98200b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:09.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.889+0000 7f9826ded700 1 -- 172.21.15.189:0/2894227664 shutdown_connections 2024-04-06T12:12:09.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.889+0000 7f9826ded700 1 -- 172.21.15.189:0/2894227664 wait complete. 2024-04-06T12:12:09.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.891+0000 7f9826ded700 1 Processor -- start 2024-04-06T12:12:09.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.891+0000 7f9826ded700 1 -- start start 2024-04-06T12:12:09.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.891+0000 7f9826ded700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200cb870 0x7f9820152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:09.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.891+0000 7f9826ded700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200cfa00 0x7f982015a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:09.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.891+0000 7f9826ded700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98200d11b0 0x7f982015a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:09.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.891+0000 7f9826ded700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9820153020 con 0x7f98200cfa00 2024-04-06T12:12:09.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.891+0000 7f9826ded700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982006ba70 con 0x7f98200cb870 2024-04-06T12:12:09.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.891+0000 7f9826ded700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982006bd10 con 0x7f98200d11b0 2024-04-06T12:12:09.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.892+0000 7f981ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200cfa00 0x7f982015a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:09.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.892+0000 7f981ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200cfa00 0x7f982015a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55972/0 (socket says 172.21.15.189:55972) 2024-04-06T12:12:09.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.892+0000 7f981ffff700 1 -- 172.21.15.189:0/1015358656 learned_addr learned my addr 172.21.15.189:0/1015358656 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:09.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.892+0000 7f9824b89700 1 --2- 172.21.15.189:0/1015358656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200cb870 0x7f9820152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:09.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.892+0000 7f982538a700 1 --2- 172.21.15.189:0/1015358656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98200d11b0 0x7f982015a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:09.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.892+0000 7f981ffff700 1 -- 172.21.15.189:0/1015358656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98200d11b0 msgr2=0x7f982015a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.892+0000 7f981ffff700 1 --2- 172.21.15.189:0/1015358656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98200d11b0 0x7f982015a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.892+0000 7f981ffff700 1 -- 172.21.15.189:0/1015358656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200cb870 msgr2=0x7f9820152ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:09.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.892+0000 7f981ffff700 1 --2- 172.21.15.189:0/1015358656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200cb870 0x7f9820152ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:09.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.892+0000 7f981ffff700 1 -- 172.21.15.189:0/1015358656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98140094a0 con 0x7f98200cfa00 2024-04-06T12:12:09.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.892+0000 7f981ffff700 1 --2- 172.21.15.189:0/1015358656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200cfa00 0x7f982015a390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f980c00c500 tx=0x7f980c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:09.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.892+0000 7f981d7fa700 1 -- 172.21.15.189:0/1015358656 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c0052d0 con 0x7f98200cfa00 2024-04-06T12:12:09.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.892+0000 7f981d7fa700 1 -- 172.21.15.189:0/1015358656 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980c005430 con 0x7f98200cfa00 2024-04-06T12:12:09.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.892+0000 7f9826ded700 1 -- 172.21.15.189:0/1015358656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f982006bff0 con 0x7f98200cfa00 2024-04-06T12:12:09.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.892+0000 7f981d7fa700 1 -- 172.21.15.189:0/1015358656 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c006aa0 con 0x7f98200cfa00 2024-04-06T12:12:09.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.892+0000 7f9826ded700 1 -- 172.21.15.189:0/1015358656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982015c840 con 0x7f98200cfa00 2024-04-06T12:12:09.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.893+0000 7f9826ded700 1 -- 172.21.15.189:0/1015358656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9820074990 con 0x7f98200cfa00 2024-04-06T12:12:09.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.893+0000 7f981d7fa700 1 -- 172.21.15.189:0/1015358656 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f980c003680 con 0x7f98200cfa00 2024-04-06T12:12:09.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.894+0000 7f981d7fa700 1 --2- 172.21.15.189:0/1015358656 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f980406ac90 0x7f980406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:09.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.894+0000 7f981d7fa700 1 -- 172.21.15.189:0/1015358656 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f980c088850 con 0x7f98200cfa00 2024-04-06T12:12:09.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.894+0000 7f9824b89700 1 --2- 172.21.15.189:0/1015358656 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f980406ac90 0x7f980406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:09.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.894+0000 7f9824b89700 1 --2- 172.21.15.189:0/1015358656 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f980406ac90 0x7f980406d150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9814000c00 tx=0x7f9814008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:09.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:09.898+0000 7f981d7fa700 1 -- 172.21.15.189:0/1015358656 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f980c053730 con 0x7f98200cfa00 2024-04-06T12:12:10.094 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:12:10.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:12:10.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.137+0000 7f9826ded700 1 -- 172.21.15.189:0/1015358656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9820155c70 con 0x7f98200cfa00 2024-04-06T12:12:10.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.138+0000 7f981d7fa700 1 -- 172.21.15.189:0/1015358656 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4739 (secure 0 0 0) 0x7f980c056d50 con 0x7f98200cfa00 2024-04-06T12:12:10.137 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:10.137 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2769778155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2769778155},{"type":"v1","addr":"172.21.15.189:6835","nonce":2769778155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4233,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3550531296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3550531296},{"type":"v1","addr":"172.21.15.189:6837","nonce":3550531296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3198235148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3198235148},{"type":"v1","addr":"172.21.15.192:6833","nonce":3198235148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:12:04.195581+0000","modified":"2024-04-06T12:12:08.649444+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/516212261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":516212261},{"type":"v1","addr":"172.21.15.192:6835","nonce":516212261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T12:12:10.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.140+0000 7f9826ded700 1 -- 172.21.15.189:0/1015358656 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f980406ac90 msgr2=0x7f980406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:10.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.140+0000 7f9826ded700 1 --2- 172.21.15.189:0/1015358656 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f980406ac90 0x7f980406d150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9814000c00 tx=0x7f9814008030 comp rx=0 tx=0).stop 2024-04-06T12:12:10.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.140+0000 7f9826ded700 1 -- 172.21.15.189:0/1015358656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200cfa00 msgr2=0x7f982015a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:10.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.140+0000 7f9826ded700 1 --2- 172.21.15.189:0/1015358656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200cfa00 0x7f982015a390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f980c00c500 tx=0x7f980c00c810 comp rx=0 tx=0).stop 2024-04-06T12:12:10.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.140+0000 7f9826ded700 1 -- 172.21.15.189:0/1015358656 shutdown_connections 2024-04-06T12:12:10.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.140+0000 7f9826ded700 1 --2- 172.21.15.189:0/1015358656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98200d11b0 0x7f982015a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:10.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.140+0000 7f9826ded700 1 --2- 172.21.15.189:0/1015358656 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f980406ac90 0x7f980406d150 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:10.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.140+0000 7f9826ded700 1 --2- 172.21.15.189:0/1015358656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200cb870 0x7f9820152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:10.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.140+0000 7f9826ded700 1 --2- 172.21.15.189:0/1015358656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200cfa00 0x7f982015a390 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:10.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.140+0000 7f9826ded700 1 -- 172.21.15.189:0/1015358656 >> 172.21.15.189:0/1015358656 conn(0x7f98200b2db0 msgr2=0x7f98200ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:10.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.140+0000 7f9826ded700 1 -- 172.21.15.189:0/1015358656 shutdown_connections 2024-04-06T12:12:10.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.140+0000 7f9826ded700 1 -- 172.21.15.189:0/1015358656 wait complete. 2024-04-06T12:12:10.141 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 7 2024-04-06T12:12:10.151 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:10.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.315+0000 7f9aeb434700 1 Processor -- start 2024-04-06T12:12:10.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.315+0000 7f9aeb434700 1 -- start start 2024-04-06T12:12:10.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.316+0000 7f9aeb434700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9ae4144f40 0x7f9ae406a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:10.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.316+0000 7f9aeb434700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9ae415c150 0x7f9ae406ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:10.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.316+0000 7f9aeb434700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae415c740 con 0x7f9ae415fcd0 2024-04-06T12:12:10.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.316+0000 7f9aeb434700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae40cfcb0 con 0x7f9ae4144f40 2024-04-06T12:12:10.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.316+0000 7f9aeb434700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae415ca20 con 0x7f9ae415c150 2024-04-06T12:12:10.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.316+0000 7f9ae91d0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9ae4144f40 0x7f9ae406a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:10.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.316+0000 7f9ae89cf700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9ae415fcd0 0x7f9ae415ba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55452/0 (socket says 172.21.15.189:55452) 2024-04-06T12:12:10.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.316+0000 7f9ae89cf700 1 -- 172.21.15.189:0/3346479353 learned_addr learned my addr 172.21.15.189:0/3346479353 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:10.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.316+0000 7f9ae99d1700 1 --2- 172.21.15.189:0/3346479353 >> v2:172.21.15.192:3300/0 conn(0x7f9ae415c150 0x7f9ae406ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:10.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.316+0000 7f9ae91d0700 1 -- 172.21.15.189:0/3346479353 >> v2:172.21.15.192:3300/0 conn(0x7f9ae415c150 msgr2=0x7f9ae406ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:10.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.316+0000 7f9ae91d0700 1 --2- 172.21.15.189:0/3346479353 >> v2:172.21.15.192:3300/0 conn(0x7f9ae415c150 0x7f9ae406ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:10.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.316+0000 7f9ae91d0700 1 -- 172.21.15.189:0/3346479353 >> v1:172.21.15.189:6789/0 conn(0x7f9ae415fcd0 legacy=0x7f9ae415ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:10.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.316+0000 7f9ae91d0700 1 -- 172.21.15.189:0/3346479353 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae415f1f0 con 0x7f9ae4144f40 2024-04-06T12:12:10.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.316+0000 7f9adbfff700 1 -- 172.21.15.189:0/3346479353 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578170707 0 0) 0x7f9ae415c740 con 0x7f9ae415fcd0 2024-04-06T12:12:10.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.316+0000 7f9ae99d1700 1 --2- 172.21.15.189:0/3346479353 >> v2:172.21.15.192:3300/0 conn(0x7f9ae415c150 0x7f9ae406ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:10.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.317+0000 7f9ae91d0700 1 --2- 172.21.15.189:0/3346479353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ae4144f40 0x7f9ae406a730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40b5600 tx=0x7f9ad4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7777f4eaeb02cfb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:10.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.317+0000 7f9adbfff700 1 -- 172.21.15.189:0/3346479353 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4011830 con 0x7f9ae4144f40 2024-04-06T12:12:10.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.317+0000 7f9adbfff700 1 -- 172.21.15.189:0/3346479353 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad4011990 con 0x7f9ae4144f40 2024-04-06T12:12:10.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.317+0000 7f9adbfff700 1 -- 172.21.15.189:0/3346479353 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4011d20 con 0x7f9ae4144f40 2024-04-06T12:12:10.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.317+0000 7f9aeb434700 1 -- 172.21.15.189:0/3346479353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ae4144f40 msgr2=0x7f9ae406a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:10.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.317+0000 7f9aeb434700 1 --2- 172.21.15.189:0/3346479353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ae4144f40 0x7f9ae406a730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40b5600 tx=0x7f9ad4009b20 comp rx=0 tx=0).stop 2024-04-06T12:12:10.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.317+0000 7f9aeb434700 1 -- 172.21.15.189:0/3346479353 shutdown_connections 2024-04-06T12:12:10.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.317+0000 7f9aeb434700 1 --2- 172.21.15.189:0/3346479353 >> v2:172.21.15.192:3300/0 conn(0x7f9ae415c150 0x7f9ae406ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:10.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.317+0000 7f9aeb434700 1 --2- 172.21.15.189:0/3346479353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ae4144f40 0x7f9ae406a730 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:10.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.317+0000 7f9aeb434700 1 -- 172.21.15.189:0/3346479353 >> 172.21.15.189:0/3346479353 conn(0x7f9ae40d00b0 msgr2=0x7f9ae40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:10.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.317+0000 7f9aeb434700 1 -- 172.21.15.189:0/3346479353 shutdown_connections 2024-04-06T12:12:10.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.317+0000 7f9aeb434700 1 -- 172.21.15.189:0/3346479353 wait complete. 2024-04-06T12:12:10.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.318+0000 7f9aeb434700 1 Processor -- start 2024-04-06T12:12:10.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.319+0000 7f9aeb434700 1 -- start start 2024-04-06T12:12:10.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.319+0000 7f9aeb434700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ae4144f40 0x7f9ae40c8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:10.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.319+0000 7f9aeb434700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae415c150 0x7f9ae40ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:10.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.319+0000 7f9aeb434700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae415fcd0 0x7f9ae415dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:10.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.319+0000 7f9aeb434700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae415cd10 con 0x7f9ae4144f40 2024-04-06T12:12:10.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.319+0000 7f9aeb434700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae415cf20 con 0x7f9ae415c150 2024-04-06T12:12:10.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.319+0000 7f9aeb434700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae415d200 con 0x7f9ae415fcd0 2024-04-06T12:12:10.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.319+0000 7f9ae91d0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ae4144f40 0x7f9ae40c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:10.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.319+0000 7f9ae91d0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ae4144f40 0x7f9ae40c8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55994/0 (socket says 172.21.15.189:55994) 2024-04-06T12:12:10.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.319+0000 7f9ae91d0700 1 -- 172.21.15.189:0/400587558 learned_addr learned my addr 172.21.15.189:0/400587558 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:10.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.320+0000 7f9ae89cf700 1 --2- 172.21.15.189:0/400587558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae415c150 0x7f9ae40ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:10.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.320+0000 7f9ae99d1700 1 --2- 172.21.15.189:0/400587558 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae415fcd0 0x7f9ae415dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:10.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.320+0000 7f9ae91d0700 1 -- 172.21.15.189:0/400587558 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae415fcd0 msgr2=0x7f9ae415dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:10.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.320+0000 7f9ae91d0700 1 --2- 172.21.15.189:0/400587558 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae415fcd0 0x7f9ae415dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:10.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.320+0000 7f9ae91d0700 1 -- 172.21.15.189:0/400587558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae415c150 msgr2=0x7f9ae40ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:10.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.320+0000 7f9ae91d0700 1 --2- 172.21.15.189:0/400587558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae415c150 0x7f9ae40ca3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:10.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.320+0000 7f9ae91d0700 1 -- 172.21.15.189:0/400587558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad40094a0 con 0x7f9ae4144f40 2024-04-06T12:12:10.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.320+0000 7f9ae91d0700 1 --2- 172.21.15.189:0/400587558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ae4144f40 0x7f9ae40c8670 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4000c00 tx=0x7f9ad4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:10.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.320+0000 7f9ad9ffb700 1 -- 172.21.15.189:0/400587558 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad40038a0 con 0x7f9ae4144f40 2024-04-06T12:12:10.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.320+0000 7f9ad9ffb700 1 -- 172.21.15.189:0/400587558 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad401f070 con 0x7f9ae4144f40 2024-04-06T12:12:10.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.320+0000 7f9ad9ffb700 1 -- 172.21.15.189:0/400587558 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4003c60 con 0x7f9ae4144f40 2024-04-06T12:12:10.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.320+0000 7f9aeb434700 1 -- 172.21.15.189:0/400587558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae415d4e0 con 0x7f9ae4144f40 2024-04-06T12:12:10.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.320+0000 7f9aeb434700 1 -- 172.21.15.189:0/400587558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae40c8bb0 con 0x7f9ae4144f40 2024-04-06T12:12:10.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.321+0000 7f9aeb434700 1 -- 172.21.15.189:0/400587558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae415d060 con 0x7f9ae4144f40 2024-04-06T12:12:10.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.322+0000 7f9ad9ffb700 1 -- 172.21.15.189:0/400587558 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9ad401a430 con 0x7f9ae4144f40 2024-04-06T12:12:10.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.322+0000 7f9ad9ffb700 1 --2- 172.21.15.189:0/400587558 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9ac806ac90 0x7f9ac806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:10.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.322+0000 7f9ad9ffb700 1 -- 172.21.15.189:0/400587558 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9ad4021070 con 0x7f9ae4144f40 2024-04-06T12:12:10.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.322+0000 7f9ae89cf700 1 --2- 172.21.15.189:0/400587558 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9ac806ac90 0x7f9ac806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:10.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.322+0000 7f9ae89cf700 1 --2- 172.21.15.189:0/400587558 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9ac806ac90 0x7f9ac806d150 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40cd910 tx=0x7f9ae0007c90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:10.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.326+0000 7f9ad9ffb700 1 -- 172.21.15.189:0/400587558 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9ae415d060 con 0x7f9ae4144f40 2024-04-06T12:12:10.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.522+0000 7f9aeb434700 1 -- 172.21.15.189:0/400587558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ae415d060 con 0x7f9ae4144f40 2024-04-06T12:12:10.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.523+0000 7f9ad9ffb700 1 -- 172.21.15.189:0/400587558 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f9ae415d060 con 0x7f9ae4144f40 2024-04-06T12:12:10.522 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:10.522 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:08.649170+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:10.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.525+0000 7f9aeb434700 1 -- 172.21.15.189:0/400587558 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9ac806ac90 msgr2=0x7f9ac806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:10.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.525+0000 7f9aeb434700 1 --2- 172.21.15.189:0/400587558 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9ac806ac90 0x7f9ac806d150 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40cd910 tx=0x7f9ae0007c90 comp rx=0 tx=0).stop 2024-04-06T12:12:10.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.525+0000 7f9aeb434700 1 -- 172.21.15.189:0/400587558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ae4144f40 msgr2=0x7f9ae40c8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:10.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.525+0000 7f9aeb434700 1 --2- 172.21.15.189:0/400587558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ae4144f40 0x7f9ae40c8670 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4000c00 tx=0x7f9ad4004750 comp rx=0 tx=0).stop 2024-04-06T12:12:10.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.525+0000 7f9aeb434700 1 -- 172.21.15.189:0/400587558 shutdown_connections 2024-04-06T12:12:10.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.525+0000 7f9aeb434700 1 --2- 172.21.15.189:0/400587558 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae415fcd0 0x7f9ae415dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:10.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.525+0000 7f9aeb434700 1 --2- 172.21.15.189:0/400587558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ae4144f40 0x7f9ae40c8670 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:10.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.525+0000 7f9aeb434700 1 --2- 172.21.15.189:0/400587558 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9ac806ac90 0x7f9ac806d150 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:10.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.525+0000 7f9aeb434700 1 --2- 172.21.15.189:0/400587558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae415c150 0x7f9ae40ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:10.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.525+0000 7f9aeb434700 1 -- 172.21.15.189:0/400587558 >> 172.21.15.189:0/400587558 conn(0x7f9ae40d00b0 msgr2=0x7f9ae415aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:10.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.525+0000 7f9aeb434700 1 -- 172.21.15.189:0/400587558 shutdown_connections 2024-04-06T12:12:10.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:10.525+0000 7f9aeb434700 1 -- 172.21.15.189:0/400587558 wait complete. 2024-04-06T12:12:10.941 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-06T12:12:04.195581+0000', 'modified': '2024-04-06T12:12:08.649444+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4270': {'gid': 4270, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.192:6835/516212261', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6834', 'nonce': 516212261}, {'type': 'v1', 'addr': '172.21.15.192:6835', 'nonce': 516212261}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-04-06T12:12:10.941 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-06T12:12:10.941 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-06T12:12:11.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.068+0000 7f220e60f700 1 Processor -- start 2024-04-06T12:12:11.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.068+0000 7f220e60f700 1 -- start start 2024-04-06T12:12:11.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.068+0000 7f220e60f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f220806ae40 0x7f220815bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.068+0000 7f220e60f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f220815c100 0x7f22080ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.068+0000 7f220e60f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f220815f9e0 con 0x7f2208158fe0 2024-04-06T12:12:11.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.068+0000 7f220e60f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f220815f0d0 con 0x7f220806ae40 2024-04-06T12:12:11.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.068+0000 7f220e60f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f220815c8f0 con 0x7f220815c100 2024-04-06T12:12:11.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.069+0000 7f22077fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f220806ae40 0x7f220815bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.069+0000 7f2207fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2208158fe0 0x7f220806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55460/0 (socket says 172.21.15.189:55460) 2024-04-06T12:12:11.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.069+0000 7f2207fff700 1 -- 172.21.15.189:0/2468972663 learned_addr learned my addr 172.21.15.189:0/2468972663 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:11.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.069+0000 7f220cbac700 1 --2- 172.21.15.189:0/2468972663 >> v2:172.21.15.192:3300/0 conn(0x7f220815c100 0x7f22080ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.069+0000 7f22077fe700 1 -- 172.21.15.189:0/2468972663 >> v2:172.21.15.192:3300/0 conn(0x7f220815c100 msgr2=0x7f22080ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:11.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.069+0000 7f22077fe700 1 --2- 172.21.15.189:0/2468972663 >> v2:172.21.15.192:3300/0 conn(0x7f220815c100 0x7f22080ceca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.069+0000 7f22077fe700 1 -- 172.21.15.189:0/2468972663 >> v1:172.21.15.189:6789/0 conn(0x7f2208158fe0 legacy=0x7f220806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:11.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.069+0000 7f22077fe700 1 -- 172.21.15.189:0/2468972663 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220815f390 con 0x7f220806ae40 2024-04-06T12:12:11.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.069+0000 7f22077fe700 1 --2- 172.21.15.189:0/2468972663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f220806ae40 0x7f220815bbc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2208145340 tx=0x7f21fc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72e0ce31be80ee89 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:11.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.069+0000 7f2206ffd700 1 -- 172.21.15.189:0/2468972663 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc011830 con 0x7f220806ae40 2024-04-06T12:12:11.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.069+0000 7f2206ffd700 1 -- 172.21.15.189:0/2468972663 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21fc011990 con 0x7f220806ae40 2024-04-06T12:12:11.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.069+0000 7f2206ffd700 1 -- 172.21.15.189:0/2468972663 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc011d20 con 0x7f220806ae40 2024-04-06T12:12:11.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.070+0000 7f220e60f700 1 -- 172.21.15.189:0/2468972663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f220806ae40 msgr2=0x7f220815bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:11.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.070+0000 7f220e60f700 1 --2- 172.21.15.189:0/2468972663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f220806ae40 0x7f220815bbc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2208145340 tx=0x7f21fc009b20 comp rx=0 tx=0).stop 2024-04-06T12:12:11.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.070+0000 7f220e60f700 1 -- 172.21.15.189:0/2468972663 shutdown_connections 2024-04-06T12:12:11.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.070+0000 7f220e60f700 1 --2- 172.21.15.189:0/2468972663 >> v2:172.21.15.192:3300/0 conn(0x7f220815c100 0x7f22080ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.070+0000 7f220e60f700 1 --2- 172.21.15.189:0/2468972663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f220806ae40 0x7f220815bbc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.070+0000 7f220e60f700 1 -- 172.21.15.189:0/2468972663 >> 172.21.15.189:0/2468972663 conn(0x7f22080d00b0 msgr2=0x7f22080d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:11.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.070+0000 7f220e60f700 1 -- 172.21.15.189:0/2468972663 shutdown_connections 2024-04-06T12:12:11.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.070+0000 7f220e60f700 1 -- 172.21.15.189:0/2468972663 wait complete. 2024-04-06T12:12:11.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.071+0000 7f220e60f700 1 Processor -- start 2024-04-06T12:12:11.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.071+0000 7f220e60f700 1 -- start start 2024-04-06T12:12:11.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.072+0000 7f220e60f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f220806ae40 0x7f2208157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.072+0000 7f220e60f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2208158fe0 0x7f220813a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.072+0000 7f220e60f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f220815c100 0x7f220813a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.072+0000 7f220e60f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220813b410 con 0x7f220806ae40 2024-04-06T12:12:11.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.072+0000 7f220e60f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2208140db0 con 0x7f2208158fe0 2024-04-06T12:12:11.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.072+0000 7f220e60f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220813b6b0 con 0x7f220815c100 2024-04-06T12:12:11.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.072+0000 7f2207fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f220806ae40 0x7f2208157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.072+0000 7f2207fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f220806ae40 0x7f2208157bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56006/0 (socket says 172.21.15.189:56006) 2024-04-06T12:12:11.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.072+0000 7f2207fff700 1 -- 172.21.15.189:0/2588116487 learned_addr learned my addr 172.21.15.189:0/2588116487 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:11.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.072+0000 7f220cbac700 1 --2- 172.21.15.189:0/2588116487 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f220815c100 0x7f220813a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.072+0000 7f22077fe700 1 --2- 172.21.15.189:0/2588116487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2208158fe0 0x7f220813a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.072+0000 7f2207fff700 1 -- 172.21.15.189:0/2588116487 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f220815c100 msgr2=0x7f220813a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:11.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.072+0000 7f2207fff700 1 --2- 172.21.15.189:0/2588116487 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f220815c100 0x7f220813a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.072+0000 7f2207fff700 1 -- 172.21.15.189:0/2588116487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2208158fe0 msgr2=0x7f220813a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:11.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.072+0000 7f2207fff700 1 --2- 172.21.15.189:0/2588116487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2208158fe0 0x7f220813a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.072+0000 7f2207fff700 1 -- 172.21.15.189:0/2588116487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21fc0094a0 con 0x7f220806ae40 2024-04-06T12:12:11.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.072+0000 7f2207fff700 1 --2- 172.21.15.189:0/2588116487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f220806ae40 0x7f2208157bc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f21f800c520 tx=0x7f21f800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:11.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.073+0000 7f2204ff9700 1 -- 172.21.15.189:0/2588116487 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f8005d70 con 0x7f220806ae40 2024-04-06T12:12:11.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.073+0000 7f2204ff9700 1 -- 172.21.15.189:0/2588116487 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f80070d0 con 0x7f220806ae40 2024-04-06T12:12:11.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.073+0000 7f220e60f700 1 -- 172.21.15.189:0/2588116487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220813b990 con 0x7f220806ae40 2024-04-06T12:12:11.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.073+0000 7f2204ff9700 1 -- 172.21.15.189:0/2588116487 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f8007440 con 0x7f220806ae40 2024-04-06T12:12:11.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.073+0000 7f220e60f700 1 -- 172.21.15.189:0/2588116487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220813f2b0 con 0x7f220806ae40 2024-04-06T12:12:11.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.074+0000 7f2204ff9700 1 -- 172.21.15.189:0/2588116487 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f21f80075a0 con 0x7f220806ae40 2024-04-06T12:12:11.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.074+0000 7f2204ff9700 1 --2- 172.21.15.189:0/2588116487 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f21f406ac90 0x7f21f406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.075+0000 7f22077fe700 1 --2- 172.21.15.189:0/2588116487 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f21f406ac90 0x7f21f406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.075+0000 7f2204ff9700 1 -- 172.21.15.189:0/2588116487 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f21f8088990 con 0x7f220806ae40 2024-04-06T12:12:11.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.075+0000 7f22077fe700 1 --2- 172.21.15.189:0/2588116487 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f21f406ac90 0x7f21f406d150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f21fc0090b0 tx=0x7f21fc008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:11.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.076+0000 7f221a7c3b80 1 Processor -- start 2024-04-06T12:12:11.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.077+0000 7f221a7c3b80 1 -- start start 2024-04-06T12:12:11.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.077+0000 7f221a7c3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55734000bc20 0x55734000fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.077+0000 7f221a7c3b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x557340010fc0 0x55733ffe75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.077+0000 7f221a7c3b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55733ffe6980 0x55733ffe7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.077+0000 7f221a7c3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55733ff40940 con 0x55734000bc20 2024-04-06T12:12:11.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.077+0000 7f221a7c3b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55733ff3e300 con 0x55733ffe6980 2024-04-06T12:12:11.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.077+0000 7f221a7c3b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55733ff39bf0 con 0x557340010fc0 2024-04-06T12:12:11.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.077+0000 7f2207fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55734000bc20 0x55734000fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.077+0000 7f2207fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55734000bc20 0x55734000fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56020/0 (socket says 172.21.15.189:56020) 2024-04-06T12:12:11.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.077+0000 7f2207fff700 1 -- 172.21.15.189:0/4197880920 learned_addr learned my addr 172.21.15.189:0/4197880920 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:11.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f2207fff700 1 -- 172.21.15.189:0/4197880920 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x557340010fc0 msgr2=0x55733ffe75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:11.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f22077fe700 1 --2- 172.21.15.189:0/4197880920 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x557340010fc0 0x55733ffe75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f220cbac700 1 --2- 172.21.15.189:0/4197880920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55733ffe6980 0x55733ffe7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f2207fff700 1 --2- 172.21.15.189:0/4197880920 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x557340010fc0 0x55733ffe75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f2207fff700 1 -- 172.21.15.189:0/4197880920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55733ffe6980 msgr2=0x55733ffe7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:11.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f2207fff700 1 --2- 172.21.15.189:0/4197880920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55733ffe6980 0x55733ffe7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f2207fff700 1 -- 172.21.15.189:0/4197880920 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55733fd67b10 con 0x55734000bc20 2024-04-06T12:12:11.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f220cbac700 1 --2- 172.21.15.189:0/4197880920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55733ffe6980 0x55733ffe7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:11.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f2207fff700 1 --2- 172.21.15.189:0/4197880920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55734000bc20 0x55734000fa60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f21f80070a0 tx=0x7f21f8057160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68aed500398a459f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:11.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f21edffb700 1 -- 172.21.15.189:0/4197880920 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f805b2d0 con 0x55734000bc20 2024-04-06T12:12:11.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f21edffb700 1 -- 172.21.15.189:0/4197880920 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f805f890 con 0x55734000bc20 2024-04-06T12:12:11.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f21edffb700 1 -- 172.21.15.189:0/4197880920 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f8056a40 con 0x55734000bc20 2024-04-06T12:12:11.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f221a7c3b80 1 -- 172.21.15.189:0/4197880920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55734000bc20 msgr2=0x55734000fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:11.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/4197880920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55734000bc20 0x55734000fa60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f21f80070a0 tx=0x7f21f8057160 comp rx=0 tx=0).stop 2024-04-06T12:12:11.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f221a7c3b80 1 -- 172.21.15.189:0/4197880920 shutdown_connections 2024-04-06T12:12:11.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/4197880920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55733ffe6980 0x55733ffe7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/4197880920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55734000bc20 0x55734000fa60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/4197880920 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x557340010fc0 0x55733ffe75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f221a7c3b80 1 -- 172.21.15.189:0/4197880920 >> 172.21.15.189:0/4197880920 conn(0x55733ff47650 msgr2=0x55734002e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:11.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.078+0000 7f221a7c3b80 1 -- 172.21.15.189:0/4197880920 shutdown_connections 2024-04-06T12:12:11.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.079+0000 7f221a7c3b80 1 -- 172.21.15.189:0/4197880920 wait complete. 2024-04-06T12:12:11.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.079+0000 7f221a7c3b80 1 -- start start 2024-04-06T12:12:11.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.079+0000 7f221a7c3b80 1 Processor -- start 2024-04-06T12:12:11.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.080+0000 7f221a7c3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 0x55734000a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.080+0000 7f221a7c3b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55734000bc20 0x5573400123f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.080+0000 7f221a7c3b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x557340010fc0 0x55734000db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.080+0000 7f221a7c3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55733fed5d30 con 0x55733ffe6980 2024-04-06T12:12:11.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.080+0000 7f221a7c3b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55733fed59d0 con 0x55734000bc20 2024-04-06T12:12:11.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.080+0000 7f221a7c3b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55733fdbd650 con 0x557340010fc0 2024-04-06T12:12:11.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.080+0000 7f221a7c3b80 10 client.0 _ll_register_callbacks cb 0x55733fedf420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:12:11.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.080+0000 7f2207fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 0x55734000a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.080+0000 7f2207fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 0x55734000a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56030/0 (socket says 172.21.15.189:56030) 2024-04-06T12:12:11.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.080+0000 7f2207fff700 1 -- 172.21.15.189:0/55230365 learned_addr learned my addr 172.21.15.189:0/55230365 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:11.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.080+0000 7f22077fe700 1 --2- 172.21.15.189:0/55230365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55734000bc20 0x5573400123f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.080+0000 7f2207fff700 1 -- 172.21.15.189:0/55230365 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x557340010fc0 msgr2=0x55734000db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:11.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.080+0000 7f220cbac700 1 --2- 172.21.15.189:0/55230365 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x557340010fc0 0x55734000db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.080+0000 7f2207fff700 1 --2- 172.21.15.189:0/55230365 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x557340010fc0 0x55734000db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.080+0000 7f2207fff700 1 -- 172.21.15.189:0/55230365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55734000bc20 msgr2=0x5573400123f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:11.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.080+0000 7f2207fff700 1 --2- 172.21.15.189:0/55230365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55734000bc20 0x5573400123f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.080+0000 7f2207fff700 1 -- 172.21.15.189:0/55230365 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220813f2b0 con 0x55733ffe6980 2024-04-06T12:12:11.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.080+0000 7f22077fe700 1 --2- 172.21.15.189:0/55230365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55734000bc20 0x5573400123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:11.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.080+0000 7f220cbac700 1 --2- 172.21.15.189:0/55230365 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x557340010fc0 0x55734000db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:11.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.081+0000 7f2207fff700 1 --2- 172.21.15.189:0/55230365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 0x55734000a610 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f21f800cb80 tx=0x7f21f8000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f564ab817322d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:11.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.081+0000 7f21e6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:12:11.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.081+0000 7f21e6ffd700 1 -- 172.21.15.189:0/55230365 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f8086070 con 0x55733ffe6980 2024-04-06T12:12:11.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.081+0000 7f21e6ffd700 1 -- 172.21.15.189:0/55230365 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f21f8056c40 con 0x55733ffe6980 2024-04-06T12:12:11.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.081+0000 7f221a7c3b80 1 -- 172.21.15.189:0/55230365 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55733fefa330 con 0x55733ffe6980 2024-04-06T12:12:11.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.081+0000 7f21e6ffd700 1 -- 172.21.15.189:0/55230365 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21f805f400 con 0x55733ffe6980 2024-04-06T12:12:11.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.081+0000 7f221a7c3b80 10 client.4398 fetch_fsmap learned FSMap version 7 2024-04-06T12:12:11.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.081+0000 7f221a7c3b80 1 -- 172.21.15.189:0/55230365 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x55733feffe50 con 0x55733ffe6980 2024-04-06T12:12:11.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.081+0000 7f21e6ffd700 1 -- 172.21.15.189:0/55230365 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55733fefa330 con 0x55733ffe6980 2024-04-06T12:12:11.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.081+0000 7f21e6ffd700 1 -- 172.21.15.189:0/55230365 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 7) v1 ==== 3122+0+0 (secure 0 0 0) 0x7f21f805d270 con 0x55733ffe6980 2024-04-06T12:12:11.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.081+0000 7f221a7c3b80 10 client.4398 fetch_fsmap finished waiting for FSMap version 7 2024-04-06T12:12:11.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.081+0000 7f221a7c3b80 10 client.4398 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-06T12:12:11.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.081+0000 7f221a7c3b80 20 client.4398 populate_metadata read hostname 'smithi189' 2024-04-06T12:12:11.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.081+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/55230365 >> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] conn(0x557340015920 0x557340015d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.081+0000 7f221a7c3b80 4 client.4398 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-06T12:12:11.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.081+0000 7f221a7c3b80 1 -- 172.21.15.189:0/55230365 --> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55733ff326b0 con 0x557340015920 2024-04-06T12:12:11.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.081+0000 7f22077fe700 1 --2- 172.21.15.189:0/55230365 >> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] conn(0x557340015920 0x557340015d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.082+0000 7f21e6ffd700 10 client.4398 ms_handle_connect on v2:172.21.15.192:6834/516212261 2024-04-06T12:12:11.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.082+0000 7f22077fe700 1 --2- 172.21.15.189:0/55230365 >> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] conn(0x557340015920 0x557340015d00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db9c21fce089bbe3 server_cookie=c63256fc8d13b647 in_seq=0 out_seq=0 2024-04-06T12:12:11.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.083+0000 7f21e6ffd700 1 -- 172.21.15.189:0/55230365 <== mds.0 v2:172.21.15.192:6834/516212261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55733ff326b0 con 0x557340015920 2024-04-06T12:12:11.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.083+0000 7f21e6ffd700 10 client.4398 handle_command_reply: tid=0 2024-04-06T12:12:11.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.083+0000 7f221a7c3b80 1 client.4398 shutdown 2024-04-06T12:12:11.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.083+0000 7f221a7c3b80 1 -- 172.21.15.189:0/55230365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 msgr2=0x55734000a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:11.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.083+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/55230365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 0x55734000a610 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f21f800cb80 tx=0x7f21f8000ed0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.083+0000 7f221a7c3b80 1 -- 172.21.15.189:0/55230365 shutdown_connections 2024-04-06T12:12:11.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.083+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/55230365 >> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] conn(0x557340015920 0x557340015d00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.083+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/55230365 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x557340010fc0 0x55734000db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.083+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/55230365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 0x55734000a610 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.083+0000 7f21e6ffd700 0 client.4398 ms_handle_reset on v2:172.21.15.192:6834/516212261 2024-04-06T12:12:11.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.083+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/55230365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55734000bc20 0x5573400123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.083+0000 7f221a7c3b80 1 -- 172.21.15.189:0/55230365 >> 172.21.15.189:0/55230365 conn(0x55733ff47650 msgr2=0x55733ffe75f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:11.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.083+0000 7f221a7c3b80 1 -- 172.21.15.189:0/55230365 shutdown_connections 2024-04-06T12:12:11.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.083+0000 7f221a7c3b80 1 -- 172.21.15.189:0/55230365 wait complete. 2024-04-06T12:12:11.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.083+0000 7f221a7c3b80 20 client.4398 trim_cache size 0 max 16384 2024-04-06T12:12:11.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.101+0000 7f221a7c3b80 1 Processor -- start 2024-04-06T12:12:11.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.101+0000 7f221a7c3b80 1 -- start start 2024-04-06T12:12:11.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.101+0000 7f221a7c3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 0x55733ffe5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.101+0000 7f221a7c3b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55734000bc20 0x55733ffe6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.101+0000 7f221a7c3b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557340010fc0 0x55734002e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.101+0000 7f221a7c3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55734000ed40 con 0x55733ffe6980 2024-04-06T12:12:11.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.101+0000 7f221a7c3b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55733ffe7330 con 0x557340010fc0 2024-04-06T12:12:11.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.101+0000 7f221a7c3b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x5573400140e0 con 0x55734000bc20 2024-04-06T12:12:11.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.101+0000 7f2207fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 0x55733ffe5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.101+0000 7f2207fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 0x55733ffe5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56040/0 (socket says 172.21.15.189:56040) 2024-04-06T12:12:11.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.101+0000 7f2207fff700 1 -- 172.21.15.189:0/1971186202 learned_addr learned my addr 172.21.15.189:0/1971186202 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:11.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.102+0000 7f2207fff700 1 -- 172.21.15.189:0/1971186202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55734000bc20 msgr2=0x55733ffe6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:11.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.102+0000 7f2207fff700 1 --2- 172.21.15.189:0/1971186202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55734000bc20 0x55733ffe6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.102+0000 7f2207fff700 1 -- 172.21.15.189:0/1971186202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557340010fc0 msgr2=0x55734002e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:11.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.102+0000 7f2207fff700 1 --2- 172.21.15.189:0/1971186202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557340010fc0 0x55734002e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.102+0000 7f2207fff700 1 -- 172.21.15.189:0/1971186202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55733fdbfc80 con 0x55733ffe6980 2024-04-06T12:12:11.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.102+0000 7f2207fff700 1 --2- 172.21.15.189:0/1971186202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 0x55733ffe5c70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f21f80570b0 tx=0x7f21f8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c48f105986bbe07b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:11.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.102+0000 7f21ee7fc700 1 -- 172.21.15.189:0/1971186202 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f8086040 con 0x55733ffe6980 2024-04-06T12:12:11.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.102+0000 7f21ee7fc700 1 -- 172.21.15.189:0/1971186202 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f805f400 con 0x55733ffe6980 2024-04-06T12:12:11.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.102+0000 7f21ee7fc700 1 -- 172.21.15.189:0/1971186202 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f805d270 con 0x55733ffe6980 2024-04-06T12:12:11.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.102+0000 7f221a7c3b80 1 -- 172.21.15.189:0/1971186202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 msgr2=0x55733ffe5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:11.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.102+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/1971186202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 0x55733ffe5c70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f21f80570b0 tx=0x7f21f8013000 comp rx=0 tx=0).stop 2024-04-06T12:12:11.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.102+0000 7f221a7c3b80 1 -- 172.21.15.189:0/1971186202 shutdown_connections 2024-04-06T12:12:11.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.102+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/1971186202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557340010fc0 0x55734002e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.102+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/1971186202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 0x55733ffe5c70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.102+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/1971186202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55734000bc20 0x55733ffe6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.102+0000 7f221a7c3b80 1 -- 172.21.15.189:0/1971186202 >> 172.21.15.189:0/1971186202 conn(0x55733ff47650 msgr2=0x55733ffdf5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:11.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.102+0000 7f221a7c3b80 1 -- 172.21.15.189:0/1971186202 shutdown_connections 2024-04-06T12:12:11.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.102+0000 7f221a7c3b80 1 -- 172.21.15.189:0/1971186202 wait complete. 2024-04-06T12:12:11.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.103+0000 7f221a7c3b80 1 -- start start 2024-04-06T12:12:11.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.103+0000 7f221a7c3b80 1 Processor -- start 2024-04-06T12:12:11.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f221a7c3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 0x55733ffe74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f221a7c3b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55734000bc20 0x55733ffe7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f221a7c3b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x557340010fc0 0x5573400473a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f221a7c3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557340016c40 con 0x55733ffe6980 2024-04-06T12:12:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f221a7c3b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55733ffe2640 con 0x55734000bc20 2024-04-06T12:12:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f221a7c3b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x557340016710 con 0x557340010fc0 2024-04-06T12:12:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f221a7c3b80 10 client.0 _ll_register_callbacks cb 0x55733fedf420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:12:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f2207fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 0x55733ffe74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f2207fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 0x55733ffe74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56046/0 (socket says 172.21.15.189:56046) 2024-04-06T12:12:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f2207fff700 1 -- 172.21.15.189:0/4154380178 learned_addr learned my addr 172.21.15.189:0/4154380178 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f22077fe700 1 --2- 172.21.15.189:0/4154380178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55734000bc20 0x55733ffe7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f220cbac700 1 --2- 172.21.15.189:0/4154380178 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x557340010fc0 0x5573400473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f2207fff700 1 -- 172.21.15.189:0/4154380178 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x557340010fc0 msgr2=0x5573400473a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f2207fff700 1 --2- 172.21.15.189:0/4154380178 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x557340010fc0 0x5573400473a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f2207fff700 1 -- 172.21.15.189:0/4154380178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55734000bc20 msgr2=0x55733ffe7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:11.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f2207fff700 1 --2- 172.21.15.189:0/4154380178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55734000bc20 0x55733ffe7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f2207fff700 1 -- 172.21.15.189:0/4154380178 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55733feffe50 con 0x55733ffe6980 2024-04-06T12:12:11.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f220cbac700 1 --2- 172.21.15.189:0/4154380178 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x557340010fc0 0x5573400473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:11.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f2207fff700 1 --2- 172.21.15.189:0/4154380178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 0x55733ffe74f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f21f80570b0 tx=0x7f21f8086f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=367de68d2d5fc899 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:11.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f21e6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:12:11.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f21e6ffd700 1 -- 172.21.15.189:0/4154380178 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f805f570 con 0x55733ffe6980 2024-04-06T12:12:11.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f21e6ffd700 1 -- 172.21.15.189:0/4154380178 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f21f8072020 con 0x55733ffe6980 2024-04-06T12:12:11.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.104+0000 7f221a7c3b80 1 -- 172.21.15.189:0/4154380178 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55733ffe3ef0 con 0x55733ffe6980 2024-04-06T12:12:11.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.105+0000 7f21e6ffd700 1 -- 172.21.15.189:0/4154380178 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21f805f6d0 con 0x55733ffe6980 2024-04-06T12:12:11.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.105+0000 7f221a7c3b80 10 client.4404 fetch_fsmap learned FSMap version 7 2024-04-06T12:12:11.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.105+0000 7f221a7c3b80 1 -- 172.21.15.189:0/4154380178 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x55733fd65fe0 con 0x55733ffe6980 2024-04-06T12:12:11.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.105+0000 7f21e6ffd700 1 -- 172.21.15.189:0/4154380178 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55733ffe3ef0 con 0x55733ffe6980 2024-04-06T12:12:11.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.105+0000 7f21e6ffd700 1 -- 172.21.15.189:0/4154380178 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 7) v1 ==== 3122+0+0 (secure 0 0 0) 0x7f21f805d270 con 0x55733ffe6980 2024-04-06T12:12:11.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.105+0000 7f221a7c3b80 10 client.4404 fetch_fsmap finished waiting for FSMap version 7 2024-04-06T12:12:11.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.105+0000 7f221a7c3b80 10 client.4404 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-06T12:12:11.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.105+0000 7f221a7c3b80 20 client.4404 populate_metadata read hostname 'smithi189' 2024-04-06T12:12:11.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.105+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/4154380178 >> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] conn(0x557340098cc0 0x5573400478e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.105+0000 7f221a7c3b80 4 client.4404 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-06T12:12:11.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.105+0000 7f221a7c3b80 1 -- 172.21.15.189:0/4154380178 --> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f21f805f890 con 0x557340098cc0 2024-04-06T12:12:11.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.105+0000 7f22077fe700 1 --2- 172.21.15.189:0/4154380178 >> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] conn(0x557340098cc0 0x5573400478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.105+0000 7f21e6ffd700 10 client.4404 ms_handle_connect on v2:172.21.15.192:6834/516212261 2024-04-06T12:12:11.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.105+0000 7f22077fe700 1 --2- 172.21.15.189:0/4154380178 >> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] conn(0x557340098cc0 0x5573400478e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb122fa9e1a89238 server_cookie=eaf56b1c80c44ee2 in_seq=0 out_seq=0 2024-04-06T12:12:11.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.105+0000 7f21e6ffd700 1 -- 172.21.15.189:0/4154380178 <== mds.0 v2:172.21.15.192:6834/516212261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x7f21f805f890 con 0x557340098cc0 2024-04-06T12:12:11.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.105+0000 7f21e6ffd700 10 client.4404 handle_command_reply: tid=0 2024-04-06T12:12:11.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.105+0000 7f221a7c3b80 1 client.4404 shutdown 2024-04-06T12:12:11.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f221a7c3b80 1 -- 172.21.15.189:0/4154380178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 msgr2=0x55733ffe74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:11.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/4154380178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 0x55733ffe74f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f21f80570b0 tx=0x7f21f8086f90 comp rx=0 tx=0).stop 2024-04-06T12:12:11.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f221a7c3b80 1 -- 172.21.15.189:0/4154380178 shutdown_connections 2024-04-06T12:12:11.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/4154380178 >> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] conn(0x557340098cc0 0x5573400478e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/4154380178 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x557340010fc0 0x5573400473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/4154380178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55733ffe6980 0x55733ffe74f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f21e6ffd700 0 client.4404 ms_handle_reset on v2:172.21.15.192:6834/516212261 2024-04-06T12:12:11.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f221a7c3b80 1 --2- 172.21.15.189:0/4154380178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55734000bc20 0x55733ffe7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f221a7c3b80 1 -- 172.21.15.189:0/4154380178 >> 172.21.15.189:0/4154380178 conn(0x55733ff47650 msgr2=0x55734000fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:11.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f221a7c3b80 1 -- 172.21.15.189:0/4154380178 shutdown_connections 2024-04-06T12:12:11.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f221a7c3b80 1 -- 172.21.15.189:0/4154380178 wait complete. 2024-04-06T12:12:11.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f221a7c3b80 20 client.4404 trim_cache size 0 max 16384 2024-04-06T12:12:11.112 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T12:12:11.113 INFO:teuthology.orchestra.run.smithi189.stdout: "cluster_fsid": "e30041de-e9a3-4351-8ca7-29d7f3e3728c", 2024-04-06T12:12:11.113 INFO:teuthology.orchestra.run.smithi189.stdout: "whoami": 0, 2024-04-06T12:12:11.113 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 4270, 2024-04-06T12:12:11.113 INFO:teuthology.orchestra.run.smithi189.stdout: "want_state": "up:active", 2024-04-06T12:12:11.113 INFO:teuthology.orchestra.run.smithi189.stdout: "state": "up:active", 2024-04-06T12:12:11.113 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs", 2024-04-06T12:12:11.113 INFO:teuthology.orchestra.run.smithi189.stdout: "rank_uptime": 6.9093224449999999, 2024-04-06T12:12:11.113 INFO:teuthology.orchestra.run.smithi189.stdout: "mdsmap_epoch": 7, 2024-04-06T12:12:11.113 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch": 24, 2024-04-06T12:12:11.113 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch_barrier": 23, 2024-04-06T12:12:11.113 INFO:teuthology.orchestra.run.smithi189.stdout: "uptime": 13.353655887 2024-04-06T12:12:11.113 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T12:12:11.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f220e60f700 1 -- 172.21.15.189:0/2588116487 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f21f406ac90 msgr2=0x7f21f406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:11.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f220e60f700 1 --2- 172.21.15.189:0/2588116487 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f21f406ac90 0x7f21f406d150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f21fc0090b0 tx=0x7f21fc008030 comp rx=0 tx=0).stop 2024-04-06T12:12:11.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f220e60f700 1 -- 172.21.15.189:0/2588116487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f220806ae40 msgr2=0x7f2208157bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:11.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f220e60f700 1 --2- 172.21.15.189:0/2588116487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f220806ae40 0x7f2208157bc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f21f800c520 tx=0x7f21f800c830 comp rx=0 tx=0).stop 2024-04-06T12:12:11.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f220e60f700 1 -- 172.21.15.189:0/2588116487 shutdown_connections 2024-04-06T12:12:11.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f220e60f700 1 --2- 172.21.15.189:0/2588116487 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f220815c100 0x7f220813a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f220e60f700 1 --2- 172.21.15.189:0/2588116487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f220806ae40 0x7f2208157bc0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f220e60f700 1 --2- 172.21.15.189:0/2588116487 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f21f406ac90 0x7f21f406d150 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f220e60f700 1 --2- 172.21.15.189:0/2588116487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2208158fe0 0x7f220813a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f220e60f700 1 -- 172.21.15.189:0/2588116487 >> 172.21.15.189:0/2588116487 conn(0x7f22080d00b0 msgr2=0x7f22080ccac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:11.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f220e60f700 1 -- 172.21.15.189:0/2588116487 shutdown_connections 2024-04-06T12:12:11.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.106+0000 7f220e60f700 1 -- 172.21.15.189:0/2588116487 wait complete. 2024-04-06T12:12:11.118 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-06T12:12:11.118 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-06T12:12:11.118 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T12:12:11.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.241+0000 7f6cae20f700 1 Processor -- start 2024-04-06T12:12:11.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.241+0000 7f6cae20f700 1 -- start start 2024-04-06T12:12:11.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.242+0000 7f6cae20f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6ca815d7b0 0x7f6ca80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.242+0000 7f6cae20f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca806a730 0x7f6ca806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.242+0000 7f6cae20f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca80b3990 con 0x7f6ca815ddf0 2024-04-06T12:12:11.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.242+0000 7f6cae20f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca806b0f0 con 0x7f6ca815d7b0 2024-04-06T12:12:11.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.242+0000 7f6cae20f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca80b3060 con 0x7f6ca806a730 2024-04-06T12:12:11.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.242+0000 7f6ca6ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca806a730 0x7f6ca806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.242+0000 7f6ca77fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6ca815d7b0 0x7f6ca80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.242+0000 7f6ca6ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca806a730 0x7f6ca806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:49176/0 (socket says 172.21.15.189:49176) 2024-04-06T12:12:11.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.242+0000 7f6ca6ffd700 1 -- 172.21.15.189:0/1842663779 learned_addr learned my addr 172.21.15.189:0/1842663779 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:11.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.242+0000 7f6ca7fff700 1 --1- 172.21.15.189:0/1842663779 >> v1:172.21.15.192:6789/0 conn(0x7f6ca815ddf0 0x7f6ca815e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:44076/0 (socket says 172.21.15.189:44076) 2024-04-06T12:12:11.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.242+0000 7f6ca6ffd700 1 -- 172.21.15.189:0/1842663779 >> v2:172.21.15.192:3300/0 conn(0x7f6ca815d7b0 msgr2=0x7f6ca80b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:11.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.242+0000 7f6ca6ffd700 1 --2- 172.21.15.189:0/1842663779 >> v2:172.21.15.192:3300/0 conn(0x7f6ca815d7b0 0x7f6ca80b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.242+0000 7f6ca6ffd700 1 -- 172.21.15.189:0/1842663779 >> v1:172.21.15.192:6789/0 conn(0x7f6ca815ddf0 legacy=0x7f6ca815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:11.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.242+0000 7f6ca6ffd700 1 -- 172.21.15.189:0/1842663779 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca80b3340 con 0x7f6ca806a730 2024-04-06T12:12:11.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.242+0000 7f6ca6ffd700 1 --2- 172.21.15.189:0/1842663779 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca806a730 0x7f6ca806abb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c0098f0 tx=0x7f6c9c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a446f741cd2385e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:11.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.244+0000 7f6ca67fc700 1 -- 172.21.15.189:0/1842663779 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c011830 con 0x7f6ca806a730 2024-04-06T12:12:11.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.244+0000 7f6ca67fc700 1 -- 172.21.15.189:0/1842663779 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9c011990 con 0x7f6ca806a730 2024-04-06T12:12:11.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.244+0000 7f6ca67fc700 1 -- 172.21.15.189:0/1842663779 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c011d20 con 0x7f6ca806a730 2024-04-06T12:12:11.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.245+0000 7f6cae20f700 1 -- 172.21.15.189:0/1842663779 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca806a730 msgr2=0x7f6ca806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:11.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.245+0000 7f6cae20f700 1 --2- 172.21.15.189:0/1842663779 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca806a730 0x7f6ca806abb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c0098f0 tx=0x7f6c9c009c00 comp rx=0 tx=0).stop 2024-04-06T12:12:11.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.245+0000 7f6cae20f700 1 -- 172.21.15.189:0/1842663779 shutdown_connections 2024-04-06T12:12:11.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.245+0000 7f6cae20f700 1 --2- 172.21.15.189:0/1842663779 >> v2:172.21.15.192:3300/0 conn(0x7f6ca815d7b0 0x7f6ca80b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.245+0000 7f6cae20f700 1 --2- 172.21.15.189:0/1842663779 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca806a730 0x7f6ca806abb0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.245+0000 7f6cae20f700 1 -- 172.21.15.189:0/1842663779 >> 172.21.15.189:0/1842663779 conn(0x7f6ca80d00b0 msgr2=0x7f6ca80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:11.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.245+0000 7f6cae20f700 1 -- 172.21.15.189:0/1842663779 shutdown_connections 2024-04-06T12:12:11.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.245+0000 7f6cae20f700 1 -- 172.21.15.189:0/1842663779 wait complete. 2024-04-06T12:12:11.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.246+0000 7f6cae20f700 1 Processor -- start 2024-04-06T12:12:11.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.247+0000 7f6cae20f700 1 -- start start 2024-04-06T12:12:11.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.247+0000 7f6cae20f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca806a730 0x7f6ca80cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.247+0000 7f6cae20f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca815d7b0 0x7f6ca80cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.247+0000 7f6cae20f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca815ddf0 0x7f6ca81f6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.247+0000 7f6cae20f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca815bf90 con 0x7f6ca815d7b0 2024-04-06T12:12:11.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.247+0000 7f6cae20f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca815c270 con 0x7f6ca815ddf0 2024-04-06T12:12:11.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.247+0000 7f6cae20f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca815c550 con 0x7f6ca806a730 2024-04-06T12:12:11.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.247+0000 7f6ca6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca815d7b0 0x7f6ca80cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.247+0000 7f6ca6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca815d7b0 0x7f6ca80cebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56050/0 (socket says 172.21.15.189:56050) 2024-04-06T12:12:11.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.248+0000 7f6ca6ffd700 1 -- 172.21.15.189:0/1825227943 learned_addr learned my addr 172.21.15.189:0/1825227943 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:11.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.248+0000 7f6ca7fff700 1 --2- 172.21.15.189:0/1825227943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca815ddf0 0x7f6ca81f6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.248+0000 7f6ca77fe700 1 --2- 172.21.15.189:0/1825227943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca806a730 0x7f6ca80cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.248+0000 7f6ca6ffd700 1 -- 172.21.15.189:0/1825227943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca806a730 msgr2=0x7f6ca80cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:11.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.248+0000 7f6ca6ffd700 1 --2- 172.21.15.189:0/1825227943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca806a730 0x7f6ca80cce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.248+0000 7f6ca6ffd700 1 -- 172.21.15.189:0/1825227943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca815ddf0 msgr2=0x7f6ca81f6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:11.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.248+0000 7f6ca6ffd700 1 --2- 172.21.15.189:0/1825227943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca815ddf0 0x7f6ca81f6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.248+0000 7f6ca6ffd700 1 -- 172.21.15.189:0/1825227943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9c009550 con 0x7f6ca815d7b0 2024-04-06T12:12:11.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.248+0000 7f6ca6ffd700 1 --2- 172.21.15.189:0/1825227943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca815d7b0 0x7f6ca80cebd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c000c00 tx=0x7f6c9c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:11.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.248+0000 7f6c8ffff700 1 -- 172.21.15.189:0/1825227943 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c004480 con 0x7f6ca815d7b0 2024-04-06T12:12:11.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.248+0000 7f6c8ffff700 1 -- 172.21.15.189:0/1825227943 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9c01f070 con 0x7f6ca815d7b0 2024-04-06T12:12:11.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.248+0000 7f6cae20f700 1 -- 172.21.15.189:0/1825227943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca815c830 con 0x7f6ca815d7b0 2024-04-06T12:12:11.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.248+0000 7f6c8ffff700 1 -- 172.21.15.189:0/1825227943 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c0038c0 con 0x7f6ca815d7b0 2024-04-06T12:12:11.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.248+0000 7f6cae20f700 1 -- 172.21.15.189:0/1825227943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca80cd360 con 0x7f6ca815d7b0 2024-04-06T12:12:11.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.249+0000 7f6cae20f700 1 -- 172.21.15.189:0/1825227943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca8067bb0 con 0x7f6ca815d7b0 2024-04-06T12:12:11.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.250+0000 7f6c8ffff700 1 -- 172.21.15.189:0/1825227943 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6c9c021030 con 0x7f6ca815d7b0 2024-04-06T12:12:11.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.250+0000 7f6c8ffff700 1 --2- 172.21.15.189:0/1825227943 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c9406ac90 0x7f6c9406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:11.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.250+0000 7f6ca77fe700 1 --2- 172.21.15.189:0/1825227943 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c9406ac90 0x7f6c9406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:11.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.250+0000 7f6c8ffff700 1 -- 172.21.15.189:0/1825227943 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6c9c08aac0 con 0x7f6ca815d7b0 2024-04-06T12:12:11.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.250+0000 7f6ca77fe700 1 --2- 172.21.15.189:0/1825227943 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c9406ac90 0x7f6c9406d150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009be0 tx=0x7f6c98009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:11.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.255+0000 7f6c8ffff700 1 -- 172.21.15.189:0/1825227943 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6c9c055b60 con 0x7f6ca815d7b0 2024-04-06T12:12:11.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.495+0000 7f6cae20f700 1 -- 172.21.15.189:0/1825227943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6ca80cd4f0 con 0x7f6ca815d7b0 2024-04-06T12:12:11.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.495+0000 7f6c8ffff700 1 -- 172.21.15.189:0/1825227943 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7f6c9c059180 con 0x7f6ca815d7b0 2024-04-06T12:12:11.495 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-06T12:12:11.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.497+0000 7f6cae20f700 1 -- 172.21.15.189:0/1825227943 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c9406ac90 msgr2=0x7f6c9406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:11.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.497+0000 7f6cae20f700 1 --2- 172.21.15.189:0/1825227943 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c9406ac90 0x7f6c9406d150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009be0 tx=0x7f6c98009210 comp rx=0 tx=0).stop 2024-04-06T12:12:11.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.497+0000 7f6cae20f700 1 -- 172.21.15.189:0/1825227943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca815d7b0 msgr2=0x7f6ca80cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:11.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.497+0000 7f6cae20f700 1 --2- 172.21.15.189:0/1825227943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca815d7b0 0x7f6ca80cebd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c000c00 tx=0x7f6c9c004750 comp rx=0 tx=0).stop 2024-04-06T12:12:11.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.497+0000 7f6cae20f700 1 -- 172.21.15.189:0/1825227943 shutdown_connections 2024-04-06T12:12:11.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.497+0000 7f6cae20f700 1 --2- 172.21.15.189:0/1825227943 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c9406ac90 0x7f6c9406d150 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.497+0000 7f6cae20f700 1 --2- 172.21.15.189:0/1825227943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca815ddf0 0x7f6ca81f6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.497+0000 7f6cae20f700 1 --2- 172.21.15.189:0/1825227943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca806a730 0x7f6ca80cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.497+0000 7f6cae20f700 1 --2- 172.21.15.189:0/1825227943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca815d7b0 0x7f6ca80cebd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:11.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.498+0000 7f6cae20f700 1 -- 172.21.15.189:0/1825227943 >> 172.21.15.189:0/1825227943 conn(0x7f6ca80d00b0 msgr2=0x7f6ca806eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:11.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.498+0000 7f6cae20f700 1 -- 172.21.15.189:0/1825227943 shutdown_connections 2024-04-06T12:12:11.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:11.498+0000 7f6cae20f700 1 -- 172.21.15.189:0/1825227943 wait complete. 2024-04-06T12:12:11.509 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-06T12:12:11.510 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T12:12:11.510 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-06T12:12:11.510 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-06T12:12:11.510 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-06T12:12:11.510 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-06T12:12:11.510 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-06T12:12:11.510 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T12:12:11.510 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-06T12:12:11.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:11.511 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-06T12:12:11.528 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T12:12:11.528 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T12:12:11.528 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T12:12:11.528 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:11.529 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T12:12:11.529 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:11.529 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:12:11.529 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:11.529 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-06T12:12:11.529 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:12:11.529 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:11.529 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-06T12:12:11.529 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:12:11.529 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:11.529 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T12:12:11.529 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:11.530 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T12:12:11.530 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 42359sec preferred_lft 42359sec 2024-04-06T12:12:11.530 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::8592:df2c:92e2:dace/64 scope link noprefixroute 2024-04-06T12:12:11.530 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:11.530 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-06T12:12:11.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:11.531 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:12:11.531 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:12:11.531 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link add name ceph-brx type bridge 2024-04-06T12:12:11.531 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip addr flush dev ceph-brx 2024-04-06T12:12:11.531 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set ceph-brx up 2024-04-06T12:12:11.531 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-06T12:12:11.531 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:12:11.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:11.699 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T12:12:11.729 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-06T12:12:11.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:11.730 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-06T12:12:11.808 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-06T12:12:11.809 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T12:12:11.809 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T12:12:11.809 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T12:12:11.809 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T12:12:11.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:11.809 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:12:11.809 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:12:11.809 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T12:12:11.809 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T12:12:11.810 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T12:12:11.810 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:12:12.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:12.015 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T12:12:12.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:12.033 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list-id 2024-04-06T12:12:12.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:12.090 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:12:12.090 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:12:12.090 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-06T12:12:12.090 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-06T12:12:12.091 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:12:12.186 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-06T12:12:12.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:12.186 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:12:12.187 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:12:12.187 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-06T12:12:12.187 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-06T12:12:12.187 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-06T12:12:12.187 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-06T12:12:12.187 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-06T12:12:12.187 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:12:12.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:12.382 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:12:12.382 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:12:12.382 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-04-06T12:12:12.382 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set dev brx.0 master ceph-brx 2024-04-06T12:12:12.382 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:12:12.480 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-06T12:12:12.480 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:12:12.480 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-06T12:12:12.497 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-06T12:12:12.497 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:12:12.497 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-06T12:12:12.553 DEBUG:teuthology.orchestra.run.smithi189:> sudo modprobe fuse 2024-04-06T12:12:12.621 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-06T12:12:12.621 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-04-06T12:12:12.687 INFO:teuthology.orchestra.run.smithi189.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-04-06T12:12:12.688 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-04-06T12:12:12.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:12.689 DEBUG:teuthology.orchestra.run.smithi189:> ls /sys/fs/fuse/connections 2024-04-06T12:12:12.746 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-06T12:12:12.746 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-06T12:12:12.788 DEBUG:teuthology.orchestra.run.smithi189:> sudo modprobe fuse 2024-04-06T12:12:12.814 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-06T12:12:12.814 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-04-06T12:12:12.876 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi189.stderr:2024-04-06T12:12:12.878+0000 7f55d55ea5c0 -1 init, newargv = 0x559d4b990910 newargc=15 2024-04-06T12:12:12.877 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi189.stderr:ceph-fuse[54790]: starting ceph client 2024-04-06T12:12:12.881 INFO:teuthology.orchestra.run.smithi189.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-04-06T12:12:12.882 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-04-06T12:12:12.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:12.883 DEBUG:teuthology.orchestra.run.smithi189:> ls /sys/fs/fuse/connections 2024-04-06T12:12:12.884 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi189.stderr:ceph-fuse[54790]: starting fuse 2024-04-06T12:12:12.941 INFO:teuthology.orchestra.run.smithi189.stdout:47 2024-04-06T12:12:12.941 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-04-06T12:12:12.941 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-04-06T12:12:12.942 DEBUG:teuthology.orchestra.run.smithi189:> import glob 2024-04-06T12:12:12.942 DEBUG:teuthology.orchestra.run.smithi189:> import re 2024-04-06T12:12:12.942 DEBUG:teuthology.orchestra.run.smithi189:> import os 2024-04-06T12:12:12.942 DEBUG:teuthology.orchestra.run.smithi189:> import subprocess 2024-04-06T12:12:12.942 DEBUG:teuthology.orchestra.run.smithi189:> 2024-04-06T12:12:12.942 DEBUG:teuthology.orchestra.run.smithi189:> def _find_admin_socket(client_name): 2024-04-06T12:12:12.942 DEBUG:teuthology.orchestra.run.smithi189:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-06T12:12:12.942 DEBUG:teuthology.orchestra.run.smithi189:> files = glob.glob(asok_path) 2024-04-06T12:12:12.942 DEBUG:teuthology.orchestra.run.smithi189:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-06T12:12:12.942 DEBUG:teuthology.orchestra.run.smithi189:> 2024-04-06T12:12:12.942 DEBUG:teuthology.orchestra.run.smithi189:> # Given a non-glob path, it better be there 2024-04-06T12:12:12.942 DEBUG:teuthology.orchestra.run.smithi189:> if "*" not in asok_path: 2024-04-06T12:12:12.942 DEBUG:teuthology.orchestra.run.smithi189:> assert(len(files) == 1) 2024-04-06T12:12:12.942 DEBUG:teuthology.orchestra.run.smithi189:> return files[0] 2024-04-06T12:12:12.942 DEBUG:teuthology.orchestra.run.smithi189:> 2024-04-06T12:12:12.942 DEBUG:teuthology.orchestra.run.smithi189:> for f in files: 2024-04-06T12:12:12.943 DEBUG:teuthology.orchestra.run.smithi189:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-06T12:12:12.943 DEBUG:teuthology.orchestra.run.smithi189:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-06T12:12:12.943 DEBUG:teuthology.orchestra.run.smithi189:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-06T12:12:12.943 DEBUG:teuthology.orchestra.run.smithi189:> contents = proc_f.read() 2024-04-06T12:12:12.943 DEBUG:teuthology.orchestra.run.smithi189:> if mountpoint in contents: 2024-04-06T12:12:12.943 DEBUG:teuthology.orchestra.run.smithi189:> return f 2024-04-06T12:12:12.943 DEBUG:teuthology.orchestra.run.smithi189:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-06T12:12:12.943 DEBUG:teuthology.orchestra.run.smithi189:> 2024-04-06T12:12:12.943 DEBUG:teuthology.orchestra.run.smithi189:> print(_find_admin_socket("client.admin")) 2024-04-06T12:12:12.943 DEBUG:teuthology.orchestra.run.smithi189:> ' 2024-04-06T12:12:13.065 INFO:teuthology.orchestra.run.smithi189.stdout:/var/run/ceph/ceph-client.admin.54790.asok 2024-04-06T12:12:13.248 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.54790.asok 2024-04-06T12:12:13.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:13.249 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.54790.asok status 2024-04-06T12:12:13.407 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T12:12:13.408 INFO:teuthology.orchestra.run.smithi189.stdout: "metadata": { 2024-04-06T12:12:13.408 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_sha1": "052adcadd0534eddb0d73c5afbc083a84cd9dc46", 2024-04-06T12:12:13.408 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_version": "ceph version 18.2.2-1086-g052adcad (052adcadd0534eddb0d73c5afbc083a84cd9dc46) reef (stable)", 2024-04-06T12:12:13.408 INFO:teuthology.orchestra.run.smithi189.stdout: "entity_id": "admin", 2024-04-06T12:12:13.408 INFO:teuthology.orchestra.run.smithi189.stdout: "hostname": "smithi189", 2024-04-06T12:12:13.408 INFO:teuthology.orchestra.run.smithi189.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-06T12:12:13.408 INFO:teuthology.orchestra.run.smithi189.stdout: "pid": "54790", 2024-04-06T12:12:13.408 INFO:teuthology.orchestra.run.smithi189.stdout: "root": "/" 2024-04-06T12:12:13.408 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:12:13.408 INFO:teuthology.orchestra.run.smithi189.stdout: "dentry_count": 0, 2024-04-06T12:12:13.408 INFO:teuthology.orchestra.run.smithi189.stdout: "dentry_pinned_count": 0, 2024-04-06T12:12:13.408 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 4413, 2024-04-06T12:12:13.408 INFO:teuthology.orchestra.run.smithi189.stdout: "inst": { 2024-04-06T12:12:13.408 INFO:teuthology.orchestra.run.smithi189.stdout: "name": { 2024-04-06T12:12:13.408 INFO:teuthology.orchestra.run.smithi189.stdout: "type": "client", 2024-04-06T12:12:13.409 INFO:teuthology.orchestra.run.smithi189.stdout: "num": 4413 2024-04-06T12:12:13.409 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:12:13.409 INFO:teuthology.orchestra.run.smithi189.stdout: "addr": { 2024-04-06T12:12:13.409 INFO:teuthology.orchestra.run.smithi189.stdout: "type": "v1", 2024-04-06T12:12:13.409 INFO:teuthology.orchestra.run.smithi189.stdout: "addr": "192.168.0.1:0", 2024-04-06T12:12:13.409 INFO:teuthology.orchestra.run.smithi189.stdout: "nonce": 1323842670 2024-04-06T12:12:13.409 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:12:13.409 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:12:13.409 INFO:teuthology.orchestra.run.smithi189.stdout: "addr": { 2024-04-06T12:12:13.409 INFO:teuthology.orchestra.run.smithi189.stdout: "type": "v1", 2024-04-06T12:12:13.409 INFO:teuthology.orchestra.run.smithi189.stdout: "addr": "192.168.0.1:0", 2024-04-06T12:12:13.409 INFO:teuthology.orchestra.run.smithi189.stdout: "nonce": 1323842670 2024-04-06T12:12:13.409 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:12:13.409 INFO:teuthology.orchestra.run.smithi189.stdout: "inst_str": "client.4413 192.168.0.1:0/1323842670", 2024-04-06T12:12:13.409 INFO:teuthology.orchestra.run.smithi189.stdout: "addr_str": "192.168.0.1:0/1323842670", 2024-04-06T12:12:13.409 INFO:teuthology.orchestra.run.smithi189.stdout: "inode_count": 1, 2024-04-06T12:12:13.409 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_epoch": 7, 2024-04-06T12:12:13.410 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_epoch": 24, 2024-04-06T12:12:13.410 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_epoch_barrier": 0, 2024-04-06T12:12:13.410 INFO:teuthology.orchestra.run.smithi189.stdout: "blocklisted": false, 2024-04-06T12:12:13.410 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs" 2024-04-06T12:12:13.410 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T12:12:13.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:13.418 DEBUG:teuthology.orchestra.run.smithi189:> stat --file-system '--printf=%T 2024-04-06T12:12:13.418 DEBUG:teuthology.orchestra.run.smithi189:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-06T12:12:13.436 INFO:teuthology.orchestra.run.smithi189.stdout:fuseblk 2024-04-06T12:12:13.436 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-06T12:12:13.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:13.436 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-06T12:12:13.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:13.505 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-06T12:12:13.610 INFO:teuthology.orchestra.run.smithi189.stdout:# file: . 2024-04-06T12:12:13.610 INFO:teuthology.orchestra.run.smithi189.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-06T12:12:13.610 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:13.611 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:12:13.611 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:12:13.628 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:13.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.793+0000 7f0b7e631700 1 Processor -- start 2024-04-06T12:12:13.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.793+0000 7f0b7e631700 1 -- start start 2024-04-06T12:12:13.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.793+0000 7f0b7e631700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b780b3b80 0x7f0b78071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:13.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.793+0000 7f0b7e631700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0b7815b510 0x7f0b78072320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:13.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.794+0000 7f0b7e631700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b780b1fb0 con 0x7f0b7815b130 2024-04-06T12:12:13.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.794+0000 7f0b7e631700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b78072860 con 0x7f0b7815b510 2024-04-06T12:12:13.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.794+0000 7f0b7e631700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b780b2200 con 0x7f0b780b3b80 2024-04-06T12:12:13.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.794+0000 7f0b777fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b780b3b80 0x7f0b78071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:13.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.794+0000 7f0b7cbce700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0b7815b510 0x7f0b78072320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:13.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.794+0000 7f0b777fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b780b3b80 0x7f0b78071de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:49194/0 (socket says 172.21.15.189:49194) 2024-04-06T12:12:13.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.794+0000 7f0b777fe700 1 -- 172.21.15.189:0/1607967399 learned_addr learned my addr 172.21.15.189:0/1607967399 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:13.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.794+0000 7f0b7cbce700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0b7815b510 0x7f0b78072320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:51464/0 (socket says 172.21.15.189:51464) 2024-04-06T12:12:13.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.794+0000 7f0b7cbce700 1 -- 172.21.15.189:0/1607967399 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b780b3b80 msgr2=0x7f0b78071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:13.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.794+0000 7f0b7cbce700 1 --2- 172.21.15.189:0/1607967399 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b780b3b80 0x7f0b78071de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:13.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.794+0000 7f0b7cbce700 1 -- 172.21.15.189:0/1607967399 >> v1:172.21.15.192:6789/0 conn(0x7f0b7815b130 legacy=0x7f0b780b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:13.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.794+0000 7f0b7cbce700 1 -- 172.21.15.189:0/1607967399 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b780b24e0 con 0x7f0b7815b510 2024-04-06T12:12:13.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.794+0000 7f0b777fe700 1 --2- 172.21.15.189:0/1607967399 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b780b3b80 0x7f0b78071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:13.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.794+0000 7f0b7cbce700 1 --2- 172.21.15.189:0/1607967399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b7815b510 0x7f0b78072320 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009d90 tx=0x7f0b7000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86f8ddebdd81fe90 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:13.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.795+0000 7f0b76ffd700 1 -- 172.21.15.189:0/1607967399 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7000e070 con 0x7f0b7815b510 2024-04-06T12:12:13.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.795+0000 7f0b76ffd700 1 -- 172.21.15.189:0/1607967399 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7000bb20 con 0x7f0b7815b510 2024-04-06T12:12:13.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.795+0000 7f0b76ffd700 1 -- 172.21.15.189:0/1607967399 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70012430 con 0x7f0b7815b510 2024-04-06T12:12:13.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.795+0000 7f0b7e631700 1 -- 172.21.15.189:0/1607967399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b7815b510 msgr2=0x7f0b78072320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:13.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.795+0000 7f0b7e631700 1 --2- 172.21.15.189:0/1607967399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b7815b510 0x7f0b78072320 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009d90 tx=0x7f0b7000b330 comp rx=0 tx=0).stop 2024-04-06T12:12:13.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.795+0000 7f0b7e631700 1 -- 172.21.15.189:0/1607967399 shutdown_connections 2024-04-06T12:12:13.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.795+0000 7f0b7e631700 1 --2- 172.21.15.189:0/1607967399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b7815b510 0x7f0b78072320 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:13.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.795+0000 7f0b7e631700 1 --2- 172.21.15.189:0/1607967399 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b780b3b80 0x7f0b78071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:13.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.795+0000 7f0b7e631700 1 -- 172.21.15.189:0/1607967399 >> 172.21.15.189:0/1607967399 conn(0x7f0b7806a730 msgr2=0x7f0b7806ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:13.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.795+0000 7f0b7e631700 1 -- 172.21.15.189:0/1607967399 shutdown_connections 2024-04-06T12:12:13.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.795+0000 7f0b7e631700 1 -- 172.21.15.189:0/1607967399 wait complete. 2024-04-06T12:12:13.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.796+0000 7f0b7e631700 1 Processor -- start 2024-04-06T12:12:13.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.796+0000 7f0b7e631700 1 -- start start 2024-04-06T12:12:13.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.797+0000 7f0b7e631700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b780b3b80 0x7f0b780cf700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:13.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.797+0000 7f0b7e631700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b7815b130 0x7f0b781ef400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:13.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.797+0000 7f0b7e631700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b7815b510 0x7f0b781f31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:13.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.797+0000 7f0b7e631700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b78153330 con 0x7f0b7815b510 2024-04-06T12:12:13.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.797+0000 7f0b7e631700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b78153b10 con 0x7f0b780b3b80 2024-04-06T12:12:13.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.797+0000 7f0b7e631700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7813a2c0 con 0x7f0b7815b130 2024-04-06T12:12:13.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.797+0000 7f0b7cbce700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b7815b510 0x7f0b781f31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:13.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.797+0000 7f0b7cbce700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b7815b510 0x7f0b781f31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56066/0 (socket says 172.21.15.189:56066) 2024-04-06T12:12:13.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.797+0000 7f0b7cbce700 1 -- 172.21.15.189:0/572142735 learned_addr learned my addr 172.21.15.189:0/572142735 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:13.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.797+0000 7f0b77fff700 1 --2- 172.21.15.189:0/572142735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b780b3b80 0x7f0b780cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:13.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.797+0000 7f0b777fe700 1 --2- 172.21.15.189:0/572142735 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b7815b130 0x7f0b781ef400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:13.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.798+0000 7f0b7cbce700 1 -- 172.21.15.189:0/572142735 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b7815b130 msgr2=0x7f0b781ef400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:13.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.798+0000 7f0b7cbce700 1 --2- 172.21.15.189:0/572142735 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b7815b130 0x7f0b781ef400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:13.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.798+0000 7f0b7cbce700 1 -- 172.21.15.189:0/572142735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b780b3b80 msgr2=0x7f0b780cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:13.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.798+0000 7f0b7cbce700 1 --2- 172.21.15.189:0/572142735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b780b3b80 0x7f0b780cf700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:13.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.798+0000 7f0b7cbce700 1 -- 172.21.15.189:0/572142735 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6c009550 con 0x7f0b7815b510 2024-04-06T12:12:13.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.798+0000 7f0b7cbce700 1 --2- 172.21.15.189:0/572142735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b7815b510 0x7f0b781f31b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0b70014040 tx=0x7f0b70004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:13.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.798+0000 7f0b74ff9700 1 -- 172.21.15.189:0/572142735 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70020070 con 0x7f0b7815b510 2024-04-06T12:12:13.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.798+0000 7f0b74ff9700 1 -- 172.21.15.189:0/572142735 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b70012910 con 0x7f0b7815b510 2024-04-06T12:12:13.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.798+0000 7f0b74ff9700 1 -- 172.21.15.189:0/572142735 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70012430 con 0x7f0b7815b510 2024-04-06T12:12:13.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.798+0000 7f0b7e631700 1 -- 172.21.15.189:0/572142735 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b70009a70 con 0x7f0b7815b510 2024-04-06T12:12:13.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.798+0000 7f0b7e631700 1 -- 172.21.15.189:0/572142735 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7813a880 con 0x7f0b7815b510 2024-04-06T12:12:13.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.799+0000 7f0b7e631700 1 -- 172.21.15.189:0/572142735 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b78073270 con 0x7f0b7815b510 2024-04-06T12:12:13.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.799+0000 7f0b74ff9700 1 -- 172.21.15.189:0/572142735 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0b70024070 con 0x7f0b7815b510 2024-04-06T12:12:13.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.800+0000 7f0b74ff9700 1 --2- 172.21.15.189:0/572142735 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b6406ac90 0x7f0b6406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:13.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.800+0000 7f0b74ff9700 1 -- 172.21.15.189:0/572142735 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0b7008c860 con 0x7f0b7815b510 2024-04-06T12:12:13.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.800+0000 7f0b77fff700 1 --2- 172.21.15.189:0/572142735 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b6406ac90 0x7f0b6406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:13.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.800+0000 7f0b77fff700 1 --2- 172.21.15.189:0/572142735 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b6406ac90 0x7f0b6406d150 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0b68007fd0 tx=0x7f0b68009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:13.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:13.804+0000 7f0b74ff9700 1 -- 172.21.15.189:0/572142735 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0b7005b310 con 0x7f0b7815b510 2024-04-06T12:12:14.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.001+0000 7f0b7e631700 1 -- 172.21.15.189:0/572142735 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b780ae720 con 0x7f0b7815b510 2024-04-06T12:12:14.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.002+0000 7f0b74ff9700 1 -- 172.21.15.189:0/572142735 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f0b7005aea0 con 0x7f0b7815b510 2024-04-06T12:12:14.010 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:14.010 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:08.649170+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:14.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.004+0000 7f0b7e631700 1 -- 172.21.15.189:0/572142735 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b6406ac90 msgr2=0x7f0b6406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:14.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.004+0000 7f0b7e631700 1 --2- 172.21.15.189:0/572142735 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b6406ac90 0x7f0b6406d150 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0b68007fd0 tx=0x7f0b68009040 comp rx=0 tx=0).stop 2024-04-06T12:12:14.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.004+0000 7f0b7e631700 1 -- 172.21.15.189:0/572142735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b7815b510 msgr2=0x7f0b781f31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:14.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.004+0000 7f0b7e631700 1 --2- 172.21.15.189:0/572142735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b7815b510 0x7f0b781f31b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0b70014040 tx=0x7f0b70004ad0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.004+0000 7f0b7e631700 1 -- 172.21.15.189:0/572142735 shutdown_connections 2024-04-06T12:12:14.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.004+0000 7f0b7e631700 1 --2- 172.21.15.189:0/572142735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b7815b510 0x7f0b781f31b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.004+0000 7f0b7e631700 1 --2- 172.21.15.189:0/572142735 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b6406ac90 0x7f0b6406d150 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.004+0000 7f0b7e631700 1 --2- 172.21.15.189:0/572142735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b780b3b80 0x7f0b780cf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.004+0000 7f0b7e631700 1 --2- 172.21.15.189:0/572142735 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b7815b130 0x7f0b781ef400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.004+0000 7f0b7e631700 1 -- 172.21.15.189:0/572142735 >> 172.21.15.189:0/572142735 conn(0x7f0b7806a730 msgr2=0x7f0b7813b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:14.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.004+0000 7f0b7e631700 1 -- 172.21.15.189:0/572142735 shutdown_connections 2024-04-06T12:12:14.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.004+0000 7f0b7e631700 1 -- 172.21.15.189:0/572142735 wait complete. 2024-04-06T12:12:14.015 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:12:02.280866+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:12:03.156593+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-06T12:12:06.237184+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-06T12:12:14.016 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:12:14.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.141+0000 7f1ff41e3700 1 Processor -- start 2024-04-06T12:12:14.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.141+0000 7f1ff41e3700 1 -- start start 2024-04-06T12:12:14.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.142+0000 7f1ff41e3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fec0d11b0 0x7f1fec0d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:14.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.142+0000 7f1ff41e3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1fec0cfa00 0x7f1fec0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:14.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.142+0000 7f1ff41e3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fec0d1ad0 con 0x7f1fec0cb870 2024-04-06T12:12:14.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.142+0000 7f1ff41e3700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fec0ccdc0 con 0x7f1fec0cfa00 2024-04-06T12:12:14.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.142+0000 7f1ff41e3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec0d0b50 con 0x7f1fec0d11b0 2024-04-06T12:12:14.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.142+0000 7f1ff177e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1fec0cb870 0x7f1fec15c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55472/0 (socket says 172.21.15.189:55472) 2024-04-06T12:12:14.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.142+0000 7f1ff177e700 1 -- 172.21.15.189:0/2440020666 learned_addr learned my addr 172.21.15.189:0/2440020666 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:14.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.142+0000 7f1ff2780700 1 --2- 172.21.15.189:0/2440020666 >> v2:172.21.15.189:3300/0 conn(0x7f1fec0cfa00 0x7f1fec0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:14.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.142+0000 7f1ff1f7f700 1 --2- 172.21.15.189:0/2440020666 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fec0d11b0 0x7f1fec0d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:14.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.142+0000 7f1ff0f7d700 1 -- 172.21.15.189:0/2440020666 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052427823 0 0) 0x7f1fec0d1ad0 con 0x7f1fec0cb870 2024-04-06T12:12:14.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.142+0000 7f1ff0f7d700 1 -- 172.21.15.189:0/2440020666 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fd80035b0 con 0x7f1fec0cb870 2024-04-06T12:12:14.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.142+0000 7f1ff2780700 1 -- 172.21.15.189:0/2440020666 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fec0d11b0 msgr2=0x7f1fec0d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:14.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.142+0000 7f1ff2780700 1 --2- 172.21.15.189:0/2440020666 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fec0d11b0 0x7f1fec0d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.142+0000 7f1ff2780700 1 -- 172.21.15.189:0/2440020666 >> v1:172.21.15.189:6789/0 conn(0x7f1fec0cb870 legacy=0x7f1fec15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:14.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.142+0000 7f1ff2780700 1 -- 172.21.15.189:0/2440020666 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec0cbd80 con 0x7f1fec0cfa00 2024-04-06T12:12:14.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.142+0000 7f1ff2780700 1 --2- 172.21.15.189:0/2440020666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fec0cfa00 0x7f1fec0cd560 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4009d90 tx=0x7f1fe400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75568b3fb447c11b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:14.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.143+0000 7f1ff0f7d700 1 -- 172.21.15.189:0/2440020666 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe400e070 con 0x7f1fec0cfa00 2024-04-06T12:12:14.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.143+0000 7f1ff0f7d700 1 -- 172.21.15.189:0/2440020666 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe400bb20 con 0x7f1fec0cfa00 2024-04-06T12:12:14.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.143+0000 7f1ff0f7d700 1 -- 172.21.15.189:0/2440020666 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4012430 con 0x7f1fec0cfa00 2024-04-06T12:12:14.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.143+0000 7f1ff41e3700 1 -- 172.21.15.189:0/2440020666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fec0cfa00 msgr2=0x7f1fec0cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:14.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.143+0000 7f1ff41e3700 1 --2- 172.21.15.189:0/2440020666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fec0cfa00 0x7f1fec0cd560 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4009d90 tx=0x7f1fe400b330 comp rx=0 tx=0).stop 2024-04-06T12:12:14.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.143+0000 7f1ff41e3700 1 -- 172.21.15.189:0/2440020666 shutdown_connections 2024-04-06T12:12:14.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.143+0000 7f1ff41e3700 1 --2- 172.21.15.189:0/2440020666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fec0cfa00 0x7f1fec0cd560 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.143+0000 7f1ff41e3700 1 --2- 172.21.15.189:0/2440020666 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fec0d11b0 0x7f1fec0d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.143+0000 7f1ff41e3700 1 -- 172.21.15.189:0/2440020666 >> 172.21.15.189:0/2440020666 conn(0x7f1fec0b2db0 msgr2=0x7f1fec0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:14.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.143+0000 7f1ff41e3700 1 -- 172.21.15.189:0/2440020666 shutdown_connections 2024-04-06T12:12:14.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.143+0000 7f1ff41e3700 1 -- 172.21.15.189:0/2440020666 wait complete. 2024-04-06T12:12:14.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.144+0000 7f1ff41e3700 1 Processor -- start 2024-04-06T12:12:14.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.144+0000 7f1ff41e3700 1 -- start start 2024-04-06T12:12:14.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.145+0000 7f1ff41e3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fec0cb870 0x7f1fec1f0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:14.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.145+0000 7f1ff41e3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fec0cfa00 0x7f1fec0b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:14.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.145+0000 7f1ff41e3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fec0d11b0 0x7f1fec1f62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:14.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.145+0000 7f1ff41e3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec157960 con 0x7f1fec0cb870 2024-04-06T12:12:14.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.145+0000 7f1ff41e3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec157070 con 0x7f1fec0cfa00 2024-04-06T12:12:14.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.145+0000 7f1ff41e3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec157310 con 0x7f1fec0d11b0 2024-04-06T12:12:14.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.145+0000 7f1ff1f7f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fec0cb870 0x7f1fec1f0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:14.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.145+0000 7f1ff1f7f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fec0cb870 0x7f1fec1f0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56084/0 (socket says 172.21.15.189:56084) 2024-04-06T12:12:14.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.145+0000 7f1ff1f7f700 1 -- 172.21.15.189:0/4164864872 learned_addr learned my addr 172.21.15.189:0/4164864872 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:14.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.145+0000 7f1ff177e700 1 --2- 172.21.15.189:0/4164864872 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fec0cfa00 0x7f1fec0b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:14.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.145+0000 7f1ff2780700 1 --2- 172.21.15.189:0/4164864872 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fec0d11b0 0x7f1fec1f62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:14.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.146+0000 7f1ff1f7f700 1 -- 172.21.15.189:0/4164864872 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fec0d11b0 msgr2=0x7f1fec1f62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:14.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.146+0000 7f1ff1f7f700 1 --2- 172.21.15.189:0/4164864872 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fec0d11b0 0x7f1fec1f62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.146+0000 7f1ff1f7f700 1 -- 172.21.15.189:0/4164864872 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fec0cfa00 msgr2=0x7f1fec0b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:14.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.146+0000 7f1ff1f7f700 1 --2- 172.21.15.189:0/4164864872 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fec0cfa00 0x7f1fec0b3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.146+0000 7f1ff1f7f700 1 -- 172.21.15.189:0/4164864872 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe4009a70 con 0x7f1fec0cb870 2024-04-06T12:12:14.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.146+0000 7f1ff177e700 1 --2- 172.21.15.189:0/4164864872 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fec0cfa00 0x7f1fec0b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:14.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.146+0000 7f1ff1f7f700 1 --2- 172.21.15.189:0/4164864872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fec0cb870 0x7f1fec1f0260 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1fe800e870 tx=0x7f1fe800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:14.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.146+0000 7f1fe27fc700 1 -- 172.21.15.189:0/4164864872 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8009590 con 0x7f1fec0cb870 2024-04-06T12:12:14.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.146+0000 7f1fe27fc700 1 -- 172.21.15.189:0/4164864872 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe800ee70 con 0x7f1fec0cb870 2024-04-06T12:12:14.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.146+0000 7f1ff41e3700 1 -- 172.21.15.189:0/4164864872 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fec1575f0 con 0x7f1fec0cb870 2024-04-06T12:12:14.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.146+0000 7f1fe27fc700 1 -- 172.21.15.189:0/4164864872 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe800ca50 con 0x7f1fec0cb870 2024-04-06T12:12:14.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.146+0000 7f1ff41e3700 1 -- 172.21.15.189:0/4164864872 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fec06d770 con 0x7f1fec0cb870 2024-04-06T12:12:14.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.147+0000 7f1ff41e3700 1 -- 172.21.15.189:0/4164864872 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fec074990 con 0x7f1fec0cb870 2024-04-06T12:12:14.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.147+0000 7f1fe27fc700 1 -- 172.21.15.189:0/4164864872 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1fe800cbb0 con 0x7f1fec0cb870 2024-04-06T12:12:14.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.148+0000 7f1fe27fc700 1 --2- 172.21.15.189:0/4164864872 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1fd006ac90 0x7f1fd006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:14.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.148+0000 7f1fe27fc700 1 -- 172.21.15.189:0/4164864872 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1fe80894f0 con 0x7f1fec0cb870 2024-04-06T12:12:14.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.148+0000 7f1ff177e700 1 --2- 172.21.15.189:0/4164864872 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1fd006ac90 0x7f1fd006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:14.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.148+0000 7f1ff177e700 1 --2- 172.21.15.189:0/4164864872 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1fd006ac90 0x7f1fd006d150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0c7120 tx=0x7f1fdc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:14.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.152+0000 7f1fe27fc700 1 -- 172.21.15.189:0/4164864872 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1fe8054470 con 0x7f1fec0cb870 2024-04-06T12:12:14.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.345+0000 7f1ff41e3700 1 -- 172.21.15.189:0/4164864872 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1fec0b0a00 con 0x7f1fec0cb870 2024-04-06T12:12:14.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.345+0000 7f1fe27fc700 1 -- 172.21.15.189:0/4164864872 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1fe800ce60 con 0x7f1fec0cb870 2024-04-06T12:12:14.345 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:12:14.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.347+0000 7f1ff41e3700 1 -- 172.21.15.189:0/4164864872 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1fd006ac90 msgr2=0x7f1fd006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:14.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.347+0000 7f1ff41e3700 1 --2- 172.21.15.189:0/4164864872 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1fd006ac90 0x7f1fd006d150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0c7120 tx=0x7f1fdc009040 comp rx=0 tx=0).stop 2024-04-06T12:12:14.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.347+0000 7f1ff41e3700 1 -- 172.21.15.189:0/4164864872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fec0cb870 msgr2=0x7f1fec1f0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:14.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.347+0000 7f1ff41e3700 1 --2- 172.21.15.189:0/4164864872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fec0cb870 0x7f1fec1f0260 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1fe800e870 tx=0x7f1fe800ec30 comp rx=0 tx=0).stop 2024-04-06T12:12:14.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.348+0000 7f1ff41e3700 1 -- 172.21.15.189:0/4164864872 shutdown_connections 2024-04-06T12:12:14.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.348+0000 7f1ff41e3700 1 --2- 172.21.15.189:0/4164864872 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fec0d11b0 0x7f1fec1f62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.348+0000 7f1ff41e3700 1 --2- 172.21.15.189:0/4164864872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fec0cb870 0x7f1fec1f0260 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.348+0000 7f1ff41e3700 1 --2- 172.21.15.189:0/4164864872 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1fd006ac90 0x7f1fd006d150 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.348+0000 7f1ff41e3700 1 --2- 172.21.15.189:0/4164864872 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fec0cfa00 0x7f1fec0b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.348+0000 7f1ff41e3700 1 -- 172.21.15.189:0/4164864872 >> 172.21.15.189:0/4164864872 conn(0x7f1fec0b2db0 msgr2=0x7f1fec1543e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:14.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.348+0000 7f1ff41e3700 1 -- 172.21.15.189:0/4164864872 shutdown_connections 2024-04-06T12:12:14.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.348+0000 7f1ff41e3700 1 -- 172.21.15.189:0/4164864872 wait complete. 2024-04-06T12:12:14.359 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:12:14.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.483+0000 7f06beb0d700 1 Processor -- start 2024-04-06T12:12:14.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.484+0000 7f06beb0d700 1 -- start start 2024-04-06T12:12:14.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.484+0000 7f06beb0d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06b815e8a0 0x7f06b80cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:14.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.484+0000 7f06beb0d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f06b815fa70 0x7f06b806aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:14.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.484+0000 7f06beb0d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b80669d0 con 0x7f06b815eee0 2024-04-06T12:12:14.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.484+0000 7f06beb0d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b806ede0 con 0x7f06b815e8a0 2024-04-06T12:12:14.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.484+0000 7f06beb0d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b815ec80 con 0x7f06b815fa70 2024-04-06T12:12:14.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.484+0000 7f06bc8a9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06b815e8a0 0x7f06b80cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:14.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.484+0000 7f06bc8a9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06b815e8a0 0x7f06b80cbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56092/0 (socket says 172.21.15.189:56092) 2024-04-06T12:12:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.484+0000 7f06bc8a9700 1 -- 172.21.15.189:0/526694177 learned_addr learned my addr 172.21.15.189:0/526694177 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.485+0000 7f06bd0aa700 1 --2- 172.21.15.189:0/526694177 >> v2:172.21.15.192:3300/0 conn(0x7f06b815fa70 0x7f06b806aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.485+0000 7f06bc8a9700 1 -- 172.21.15.189:0/526694177 >> v2:172.21.15.192:3300/0 conn(0x7f06b815fa70 msgr2=0x7f06b806aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.485+0000 7f06bc8a9700 1 --2- 172.21.15.189:0/526694177 >> v2:172.21.15.192:3300/0 conn(0x7f06b815fa70 0x7f06b806aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.485+0000 7f06bc8a9700 1 -- 172.21.15.189:0/526694177 >> v1:172.21.15.192:6789/0 conn(0x7f06b815eee0 legacy=0x7f06b815f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.485+0000 7f06bc8a9700 1 -- 172.21.15.189:0/526694177 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b80666f0 con 0x7f06b815e8a0 2024-04-06T12:12:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.485+0000 7f06bc8a9700 1 --2- 172.21.15.189:0/526694177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06b815e8a0 0x7f06b80cbc20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f06b80cce40 tx=0x7f06ac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e700999e14d57dc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.485+0000 7f06b77fe700 1 -- 172.21.15.189:0/526694177 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ac004230 con 0x7f06b815e8a0 2024-04-06T12:12:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.485+0000 7f06b77fe700 1 -- 172.21.15.189:0/526694177 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ac004390 con 0x7f06b815e8a0 2024-04-06T12:12:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.485+0000 7f06b77fe700 1 -- 172.21.15.189:0/526694177 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ac0036a0 con 0x7f06b815e8a0 2024-04-06T12:12:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.486+0000 7f06beb0d700 1 -- 172.21.15.189:0/526694177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06b815e8a0 msgr2=0x7f06b80cbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.486+0000 7f06beb0d700 1 --2- 172.21.15.189:0/526694177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06b815e8a0 0x7f06b80cbc20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f06b80cce40 tx=0x7f06ac009bd0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.486+0000 7f06beb0d700 1 -- 172.21.15.189:0/526694177 shutdown_connections 2024-04-06T12:12:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.486+0000 7f06beb0d700 1 --2- 172.21.15.189:0/526694177 >> v2:172.21.15.192:3300/0 conn(0x7f06b815fa70 0x7f06b806aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.486+0000 7f06beb0d700 1 --2- 172.21.15.189:0/526694177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06b815e8a0 0x7f06b80cbc20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.486+0000 7f06beb0d700 1 -- 172.21.15.189:0/526694177 >> 172.21.15.189:0/526694177 conn(0x7f06b80cc290 msgr2=0x7f06b80cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:14.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.486+0000 7f06beb0d700 1 -- 172.21.15.189:0/526694177 shutdown_connections 2024-04-06T12:12:14.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.486+0000 7f06beb0d700 1 -- 172.21.15.189:0/526694177 wait complete. 2024-04-06T12:12:14.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.487+0000 7f06beb0d700 1 Processor -- start 2024-04-06T12:12:14.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.487+0000 7f06beb0d700 1 -- start start 2024-04-06T12:12:14.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.488+0000 7f06beb0d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06b815e8a0 0x7f06b8155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:14.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.488+0000 7f06beb0d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06b815eee0 0x7f06b80c50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:14.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.488+0000 7f06beb0d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06b815fa70 0x7f06b80c3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:14.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.488+0000 7f06beb0d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b80cd3e0 con 0x7f06b815e8a0 2024-04-06T12:12:14.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.488+0000 7f06beb0d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b80cd9a0 con 0x7f06b815eee0 2024-04-06T12:12:14.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.488+0000 7f06beb0d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b80c5ab0 con 0x7f06b815fa70 2024-04-06T12:12:14.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.488+0000 7f06bc8a9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06b815e8a0 0x7f06b8155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:14.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.488+0000 7f06bc8a9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06b815e8a0 0x7f06b8155be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56102/0 (socket says 172.21.15.189:56102) 2024-04-06T12:12:14.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.488+0000 7f06bc8a9700 1 -- 172.21.15.189:0/1191268315 learned_addr learned my addr 172.21.15.189:0/1191268315 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:14.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.488+0000 7f06b7fff700 1 --2- 172.21.15.189:0/1191268315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06b815eee0 0x7f06b80c50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:14.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.489+0000 7f06bd0aa700 1 --2- 172.21.15.189:0/1191268315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06b815fa70 0x7f06b80c3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:14.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.489+0000 7f06bc8a9700 1 -- 172.21.15.189:0/1191268315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06b815fa70 msgr2=0x7f06b80c3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:14.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.489+0000 7f06bc8a9700 1 --2- 172.21.15.189:0/1191268315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06b815fa70 0x7f06b80c3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.489+0000 7f06bc8a9700 1 -- 172.21.15.189:0/1191268315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06b815eee0 msgr2=0x7f06b80c50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:14.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.489+0000 7f06bc8a9700 1 --2- 172.21.15.189:0/1191268315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06b815eee0 0x7f06b80c50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.489+0000 7f06bc8a9700 1 -- 172.21.15.189:0/1191268315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ac009550 con 0x7f06b815e8a0 2024-04-06T12:12:14.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.489+0000 7f06bc8a9700 1 --2- 172.21.15.189:0/1191268315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06b815e8a0 0x7f06b8155be0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f06b8073430 tx=0x7f06ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:14.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.489+0000 7f06b57fa700 1 -- 172.21.15.189:0/1191268315 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ac003c90 con 0x7f06b815e8a0 2024-04-06T12:12:14.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.489+0000 7f06b57fa700 1 -- 172.21.15.189:0/1191268315 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ac01f070 con 0x7f06b815e8a0 2024-04-06T12:12:14.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.489+0000 7f06beb0d700 1 -- 172.21.15.189:0/1191268315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b80c5d90 con 0x7f06b815e8a0 2024-04-06T12:12:14.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.489+0000 7f06b57fa700 1 -- 172.21.15.189:0/1191268315 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ac011870 con 0x7f06b815e8a0 2024-04-06T12:12:14.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.489+0000 7f06beb0d700 1 -- 172.21.15.189:0/1191268315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b8156120 con 0x7f06b815e8a0 2024-04-06T12:12:14.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.490+0000 7f06beb0d700 1 -- 172.21.15.189:0/1191268315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b8072140 con 0x7f06b815e8a0 2024-04-06T12:12:14.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.490+0000 7f06b57fa700 1 -- 172.21.15.189:0/1191268315 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f06ac021030 con 0x7f06b815e8a0 2024-04-06T12:12:14.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.491+0000 7f06b57fa700 1 --2- 172.21.15.189:0/1191268315 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f069c06ac90 0x7f069c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:14.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.491+0000 7f06b57fa700 1 -- 172.21.15.189:0/1191268315 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f06ac08aae0 con 0x7f06b815e8a0 2024-04-06T12:12:14.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.491+0000 7f06b7fff700 1 --2- 172.21.15.189:0/1191268315 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f069c06ac90 0x7f069c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:14.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.491+0000 7f06b7fff700 1 --2- 172.21.15.189:0/1191268315 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f069c06ac90 0x7f069c06d150 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f06b80cf5c0 tx=0x7f06a4007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:14.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.495+0000 7f06b57fa700 1 -- 172.21.15.189:0/1191268315 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f06ac055a60 con 0x7f06b815e8a0 2024-04-06T12:12:14.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.689+0000 7f06beb0d700 1 -- 172.21.15.189:0/1191268315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f06b8074990 con 0x7f06b815e8a0 2024-04-06T12:12:14.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.690+0000 7f06b57fa700 1 -- 172.21.15.189:0/1191268315 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f06ac059080 con 0x7f06b815e8a0 2024-04-06T12:12:14.689 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:12:14.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.692+0000 7f06beb0d700 1 -- 172.21.15.189:0/1191268315 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f069c06ac90 msgr2=0x7f069c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:14.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.692+0000 7f06beb0d700 1 --2- 172.21.15.189:0/1191268315 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f069c06ac90 0x7f069c06d150 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f06b80cf5c0 tx=0x7f06a4007ec0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.692+0000 7f06beb0d700 1 -- 172.21.15.189:0/1191268315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06b815e8a0 msgr2=0x7f06b8155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:14.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.692+0000 7f06beb0d700 1 --2- 172.21.15.189:0/1191268315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06b815e8a0 0x7f06b8155be0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f06b8073430 tx=0x7f06ac004750 comp rx=0 tx=0).stop 2024-04-06T12:12:14.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.692+0000 7f06beb0d700 1 -- 172.21.15.189:0/1191268315 shutdown_connections 2024-04-06T12:12:14.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.692+0000 7f06beb0d700 1 --2- 172.21.15.189:0/1191268315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06b815fa70 0x7f06b80c3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.692+0000 7f06beb0d700 1 --2- 172.21.15.189:0/1191268315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06b815e8a0 0x7f06b8155be0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.692+0000 7f06beb0d700 1 --2- 172.21.15.189:0/1191268315 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f069c06ac90 0x7f069c06d150 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.692+0000 7f06beb0d700 1 --2- 172.21.15.189:0/1191268315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06b815eee0 0x7f06b80c50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.692+0000 7f06beb0d700 1 -- 172.21.15.189:0/1191268315 >> 172.21.15.189:0/1191268315 conn(0x7f06b80cc290 msgr2=0x7f06b815d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:14.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.692+0000 7f06beb0d700 1 -- 172.21.15.189:0/1191268315 shutdown_connections 2024-04-06T12:12:14.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.692+0000 7f06beb0d700 1 -- 172.21.15.189:0/1191268315 wait complete. 2024-04-06T12:12:14.703 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:12:14.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.829+0000 7f1409edc700 1 Processor -- start 2024-04-06T12:12:14.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.829+0000 7f1409edc700 1 -- start start 2024-04-06T12:12:14.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.830+0000 7f1409edc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f140406ae40 0x7f140415bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:14.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.830+0000 7f1409edc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f140415c100 0x7f14040ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:14.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.830+0000 7f1409edc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f140415f9e0 con 0x7f1404158fe0 2024-04-06T12:12:14.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.830+0000 7f1409edc700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f140415f0d0 con 0x7f140406ae40 2024-04-06T12:12:14.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.830+0000 7f1409edc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140415c8f0 con 0x7f140415c100 2024-04-06T12:12:14.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.830+0000 7f1402ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f140406ae40 0x7f140415bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:14.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.830+0000 7f14037fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1404158fe0 0x7f140406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55486/0 (socket says 172.21.15.189:55486) 2024-04-06T12:12:14.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.830+0000 7f14037fe700 1 -- 172.21.15.189:0/2947870204 learned_addr learned my addr 172.21.15.189:0/2947870204 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:14.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.830+0000 7f1403fff700 1 --2- 172.21.15.189:0/2947870204 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f140415c100 0x7f14040ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:14.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.830+0000 7f1402ffd700 1 -- 172.21.15.189:0/2947870204 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f140415c100 msgr2=0x7f14040ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.830+0000 7f1402ffd700 1 --2- 172.21.15.189:0/2947870204 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f140415c100 0x7f14040ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.830+0000 7f1402ffd700 1 -- 172.21.15.189:0/2947870204 >> v1:172.21.15.189:6789/0 conn(0x7f1404158fe0 legacy=0x7f140406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.830+0000 7f14027fc700 1 -- 172.21.15.189:0/2947870204 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931025268 0 0) 0x7f140415f9e0 con 0x7f1404158fe0 2024-04-06T12:12:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.830+0000 7f1402ffd700 1 -- 172.21.15.189:0/2947870204 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f140415f390 con 0x7f140406ae40 2024-04-06T12:12:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.830+0000 7f1402ffd700 1 --2- 172.21.15.189:0/2947870204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f140406ae40 0x7f140415bbc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f13f8009840 tx=0x7f13f8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f48c27126acc32 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.831+0000 7f14027fc700 1 -- 172.21.15.189:0/2947870204 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f8011830 con 0x7f140406ae40 2024-04-06T12:12:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.831+0000 7f14027fc700 1 -- 172.21.15.189:0/2947870204 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f8011990 con 0x7f140406ae40 2024-04-06T12:12:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.831+0000 7f14027fc700 1 -- 172.21.15.189:0/2947870204 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f8011d20 con 0x7f140406ae40 2024-04-06T12:12:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.831+0000 7f1409edc700 1 -- 172.21.15.189:0/2947870204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f140406ae40 msgr2=0x7f140415bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.831+0000 7f1409edc700 1 --2- 172.21.15.189:0/2947870204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f140406ae40 0x7f140415bbc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f13f8009840 tx=0x7f13f8009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.831+0000 7f1409edc700 1 -- 172.21.15.189:0/2947870204 shutdown_connections 2024-04-06T12:12:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.831+0000 7f1409edc700 1 --2- 172.21.15.189:0/2947870204 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f140415c100 0x7f14040ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.831+0000 7f1409edc700 1 --2- 172.21.15.189:0/2947870204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f140406ae40 0x7f140415bbc0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.831+0000 7f1409edc700 1 -- 172.21.15.189:0/2947870204 >> 172.21.15.189:0/2947870204 conn(0x7f14040d00b0 msgr2=0x7f14040d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.831+0000 7f1409edc700 1 -- 172.21.15.189:0/2947870204 shutdown_connections 2024-04-06T12:12:14.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.831+0000 7f1409edc700 1 -- 172.21.15.189:0/2947870204 wait complete. 2024-04-06T12:12:14.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.832+0000 7f1409edc700 1 Processor -- start 2024-04-06T12:12:14.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.832+0000 7f1409edc700 1 -- start start 2024-04-06T12:12:14.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.833+0000 7f1409edc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f140406ae40 0x7f1404159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:14.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.833+0000 7f1409edc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1404158fe0 0x7f140413b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:14.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.833+0000 7f1409edc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f140415c100 0x7f1404140db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:14.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.833+0000 7f1409edc700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140415b640 con 0x7f1404158fe0 2024-04-06T12:12:14.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.833+0000 7f1409edc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140415a5c0 con 0x7f140406ae40 2024-04-06T12:12:14.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.833+0000 7f1409edc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140415a8a0 con 0x7f140415c100 2024-04-06T12:12:14.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.833+0000 7f1402ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1404158fe0 0x7f140413b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:14.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.833+0000 7f1402ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1404158fe0 0x7f140413b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56108/0 (socket says 172.21.15.189:56108) 2024-04-06T12:12:14.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.833+0000 7f1402ffd700 1 -- 172.21.15.189:0/2609824988 learned_addr learned my addr 172.21.15.189:0/2609824988 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:14.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.833+0000 7f1403fff700 1 --2- 172.21.15.189:0/2609824988 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f140415c100 0x7f1404140db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:14.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.834+0000 7f14037fe700 1 --2- 172.21.15.189:0/2609824988 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f140406ae40 0x7f1404159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:14.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.834+0000 7f1402ffd700 1 -- 172.21.15.189:0/2609824988 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f140415c100 msgr2=0x7f1404140db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:14.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.834+0000 7f1402ffd700 1 --2- 172.21.15.189:0/2609824988 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f140415c100 0x7f1404140db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.834+0000 7f1402ffd700 1 -- 172.21.15.189:0/2609824988 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f140406ae40 msgr2=0x7f1404159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:14.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.834+0000 7f1402ffd700 1 --2- 172.21.15.189:0/2609824988 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f140406ae40 0x7f1404159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:14.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.834+0000 7f1402ffd700 1 -- 172.21.15.189:0/2609824988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f80094a0 con 0x7f1404158fe0 2024-04-06T12:12:14.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.834+0000 7f14037fe700 1 --2- 172.21.15.189:0/2609824988 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f140406ae40 0x7f1404159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:14.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.834+0000 7f1402ffd700 1 --2- 172.21.15.189:0/2609824988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1404158fe0 0x7f140413b490 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f13f8009fd0 tx=0x7f13f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:14.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.834+0000 7f13ebfff700 1 -- 172.21.15.189:0/2609824988 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f80045d0 con 0x7f1404158fe0 2024-04-06T12:12:14.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.834+0000 7f13ebfff700 1 -- 172.21.15.189:0/2609824988 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f801f070 con 0x7f1404158fe0 2024-04-06T12:12:14.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.834+0000 7f1409edc700 1 -- 172.21.15.189:0/2609824988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f140415ab80 con 0x7f1404158fe0 2024-04-06T12:12:14.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.834+0000 7f13ebfff700 1 -- 172.21.15.189:0/2609824988 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f80038e0 con 0x7f1404158fe0 2024-04-06T12:12:14.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.834+0000 7f1409edc700 1 -- 172.21.15.189:0/2609824988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f140415b140 con 0x7f1404158fe0 2024-04-06T12:12:14.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.835+0000 7f1409edc700 1 -- 172.21.15.189:0/2609824988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f140415b4b0 con 0x7f1404158fe0 2024-04-06T12:12:14.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.835+0000 7f13ebfff700 1 -- 172.21.15.189:0/2609824988 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f13f8003a40 con 0x7f1404158fe0 2024-04-06T12:12:14.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.836+0000 7f13ebfff700 1 --2- 172.21.15.189:0/2609824988 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13f006ac90 0x7f13f006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:14.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.836+0000 7f13ebfff700 1 -- 172.21.15.189:0/2609824988 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f13f8021070 con 0x7f1404158fe0 2024-04-06T12:12:14.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.836+0000 7f14037fe700 1 --2- 172.21.15.189:0/2609824988 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13f006ac90 0x7f13f006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:14.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.836+0000 7f14037fe700 1 --2- 172.21.15.189:0/2609824988 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13f006ac90 0x7f13f006d150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f13f4000f60 tx=0x7f13f4007ca0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:14.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:14.840+0000 7f13ebfff700 1 -- 172.21.15.189:0/2609824988 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f13f8090020 con 0x7f1404158fe0 2024-04-06T12:12:15.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.035+0000 7f1409edc700 1 -- 172.21.15.189:0/2609824988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f140415b2d0 con 0x7f1404158fe0 2024-04-06T12:12:15.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.035+0000 7f13ebfff700 1 -- 172.21.15.189:0/2609824988 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f13f8015020 con 0x7f1404158fe0 2024-04-06T12:12:15.034 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:12:15.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.037+0000 7f1409edc700 1 -- 172.21.15.189:0/2609824988 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13f006ac90 msgr2=0x7f13f006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:15.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.037+0000 7f1409edc700 1 --2- 172.21.15.189:0/2609824988 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13f006ac90 0x7f13f006d150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f13f4000f60 tx=0x7f13f4007ca0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.037+0000 7f1409edc700 1 -- 172.21.15.189:0/2609824988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1404158fe0 msgr2=0x7f140413b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:15.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.037+0000 7f1409edc700 1 --2- 172.21.15.189:0/2609824988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1404158fe0 0x7f140413b490 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f13f8009fd0 tx=0x7f13f8004750 comp rx=0 tx=0).stop 2024-04-06T12:12:15.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.037+0000 7f1409edc700 1 -- 172.21.15.189:0/2609824988 shutdown_connections 2024-04-06T12:12:15.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.037+0000 7f1409edc700 1 --2- 172.21.15.189:0/2609824988 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f140415c100 0x7f1404140db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.037+0000 7f1409edc700 1 --2- 172.21.15.189:0/2609824988 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13f006ac90 0x7f13f006d150 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.037+0000 7f1409edc700 1 --2- 172.21.15.189:0/2609824988 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f140406ae40 0x7f1404159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.037+0000 7f1409edc700 1 --2- 172.21.15.189:0/2609824988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1404158fe0 0x7f140413b490 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.037+0000 7f1409edc700 1 -- 172.21.15.189:0/2609824988 >> 172.21.15.189:0/2609824988 conn(0x7f14040d00b0 msgr2=0x7f140406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:15.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.038+0000 7f1409edc700 1 -- 172.21.15.189:0/2609824988 shutdown_connections 2024-04-06T12:12:15.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.038+0000 7f1409edc700 1 -- 172.21.15.189:0/2609824988 wait complete. 2024-04-06T12:12:15.048 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-06T12:12:15.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.174+0000 7fdc1292c700 1 Processor -- start 2024-04-06T12:12:15.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.174+0000 7fdc1292c700 1 -- start start 2024-04-06T12:12:15.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.174+0000 7fdc1292c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdc0c0d11b0 0x7fdc0c0d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:15.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.174+0000 7fdc1292c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdc0c0cb870 0x7fdc0c15c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:15.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.174+0000 7fdc1292c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc0c0d1ad0 con 0x7fdc0c0cfa00 2024-04-06T12:12:15.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.174+0000 7fdc1292c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc0c0ccdc0 con 0x7fdc0c0d11b0 2024-04-06T12:12:15.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.174+0000 7fdc1292c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc0c0d0b50 con 0x7fdc0c0cb870 2024-04-06T12:12:15.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.175+0000 7fdc0bfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdc0c0d11b0 0x7fdc0c0d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:15.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.175+0000 7fdc10ec9700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdc0c0cfa00 0x7fdc0c0cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55496/0 (socket says 172.21.15.189:55496) 2024-04-06T12:12:15.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.175+0000 7fdc10ec9700 1 -- 172.21.15.189:0/653854890 learned_addr learned my addr 172.21.15.189:0/653854890 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:15.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.175+0000 7fdc0bfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdc0c0d11b0 0x7fdc0c0d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56110/0 (socket says 172.21.15.189:56110) 2024-04-06T12:12:15.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.175+0000 7fdc0b7fe700 1 --2- 172.21.15.189:0/653854890 >> v2:172.21.15.192:3300/0 conn(0x7fdc0c0cb870 0x7fdc0c15c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:15.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.175+0000 7fdc0bfff700 1 -- 172.21.15.189:0/653854890 >> v2:172.21.15.192:3300/0 conn(0x7fdc0c0cb870 msgr2=0x7fdc0c15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:15.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.175+0000 7fdc0bfff700 1 --2- 172.21.15.189:0/653854890 >> v2:172.21.15.192:3300/0 conn(0x7fdc0c0cb870 0x7fdc0c15c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.175+0000 7fdc0bfff700 1 -- 172.21.15.189:0/653854890 >> v1:172.21.15.189:6789/0 conn(0x7fdc0c0cfa00 legacy=0x7fdc0c0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:15.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.175+0000 7fdc0bfff700 1 -- 172.21.15.189:0/653854890 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0c0cbd80 con 0x7fdc0c0d11b0 2024-04-06T12:12:15.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.175+0000 7fdc0affd700 1 -- 172.21.15.189:0/653854890 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824758069 0 0) 0x7fdc0c0d1ad0 con 0x7fdc0c0cfa00 2024-04-06T12:12:15.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.175+0000 7fdc0b7fe700 1 --2- 172.21.15.189:0/653854890 >> v2:172.21.15.192:3300/0 conn(0x7fdc0c0cb870 0x7fdc0c15c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:15.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.175+0000 7fdc0bfff700 1 --2- 172.21.15.189:0/653854890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c0d11b0 0x7fdc0c0d1590 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c04fb40 tx=0x7fdbfc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d3dff3ead8b720c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:15.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.175+0000 7fdc0affd700 1 -- 172.21.15.189:0/653854890 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc011830 con 0x7fdc0c0d11b0 2024-04-06T12:12:15.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.175+0000 7fdc0affd700 1 -- 172.21.15.189:0/653854890 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbfc011990 con 0x7fdc0c0d11b0 2024-04-06T12:12:15.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.175+0000 7fdc0affd700 1 -- 172.21.15.189:0/653854890 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc011d20 con 0x7fdc0c0d11b0 2024-04-06T12:12:15.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.176+0000 7fdc1292c700 1 -- 172.21.15.189:0/653854890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c0d11b0 msgr2=0x7fdc0c0d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:15.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.176+0000 7fdc1292c700 1 --2- 172.21.15.189:0/653854890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c0d11b0 0x7fdc0c0d1590 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c04fb40 tx=0x7fdbfc009b20 comp rx=0 tx=0).stop 2024-04-06T12:12:15.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.176+0000 7fdc1292c700 1 -- 172.21.15.189:0/653854890 shutdown_connections 2024-04-06T12:12:15.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.176+0000 7fdc1292c700 1 --2- 172.21.15.189:0/653854890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c0d11b0 0x7fdc0c0d1590 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.176+0000 7fdc1292c700 1 --2- 172.21.15.189:0/653854890 >> v2:172.21.15.192:3300/0 conn(0x7fdc0c0cb870 0x7fdc0c15c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.176+0000 7fdc1292c700 1 -- 172.21.15.189:0/653854890 >> 172.21.15.189:0/653854890 conn(0x7fdc0c0b2db0 msgr2=0x7fdc0c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:15.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.176+0000 7fdc1292c700 1 -- 172.21.15.189:0/653854890 shutdown_connections 2024-04-06T12:12:15.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.176+0000 7fdc1292c700 1 -- 172.21.15.189:0/653854890 wait complete. 2024-04-06T12:12:15.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.177+0000 7fdc1292c700 1 Processor -- start 2024-04-06T12:12:15.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.177+0000 7fdc1292c700 1 -- start start 2024-04-06T12:12:15.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.178+0000 7fdc1292c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c0cb870 0x7fdc0c154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:15.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.178+0000 7fdc1292c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc0c0cfa00 0x7fdc0c06c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:15.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.178+0000 7fdc1292c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc0c0d11b0 0x7fdc0c158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:15.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.178+0000 7fdc1292c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c0b3fd0 con 0x7fdc0c0cb870 2024-04-06T12:12:15.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.178+0000 7fdc1292c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c0b42b0 con 0x7fdc0c0cfa00 2024-04-06T12:12:15.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.178+0000 7fdc1292c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c0b4590 con 0x7fdc0c0d11b0 2024-04-06T12:12:15.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.178+0000 7fdc0bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c0cb870 0x7fdc0c154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:15.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.178+0000 7fdc0bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c0cb870 0x7fdc0c154f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56126/0 (socket says 172.21.15.189:56126) 2024-04-06T12:12:15.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.178+0000 7fdc0bfff700 1 -- 172.21.15.189:0/2778019375 learned_addr learned my addr 172.21.15.189:0/2778019375 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:15.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.178+0000 7fdc0b7fe700 1 --2- 172.21.15.189:0/2778019375 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc0c0cfa00 0x7fdc0c06c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:15.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.178+0000 7fdc10ec9700 1 --2- 172.21.15.189:0/2778019375 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc0c0d11b0 0x7fdc0c158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:15.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.178+0000 7fdc0bfff700 1 -- 172.21.15.189:0/2778019375 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc0c0d11b0 msgr2=0x7fdc0c158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:15.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.178+0000 7fdc0bfff700 1 --2- 172.21.15.189:0/2778019375 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc0c0d11b0 0x7fdc0c158150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.178+0000 7fdc0bfff700 1 -- 172.21.15.189:0/2778019375 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc0c0cfa00 msgr2=0x7fdc0c06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:15.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.178+0000 7fdc0bfff700 1 --2- 172.21.15.189:0/2778019375 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc0c0cfa00 0x7fdc0c06c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.178+0000 7fdc0bfff700 1 -- 172.21.15.189:0/2778019375 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc0094a0 con 0x7fdc0c0cb870 2024-04-06T12:12:15.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.178+0000 7fdc0b7fe700 1 --2- 172.21.15.189:0/2778019375 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc0c0cfa00 0x7fdc0c06c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:15.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.178+0000 7fdc0bfff700 1 --2- 172.21.15.189:0/2778019375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c0cb870 0x7fdc0c154f60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0090b0 tx=0x7fdbfc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:15.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.179+0000 7fdc08ff9700 1 -- 172.21.15.189:0/2778019375 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc0038a0 con 0x7fdc0c0cb870 2024-04-06T12:12:15.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.179+0000 7fdc08ff9700 1 -- 172.21.15.189:0/2778019375 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbfc01f070 con 0x7fdc0c0cb870 2024-04-06T12:12:15.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.179+0000 7fdc1292c700 1 -- 172.21.15.189:0/2778019375 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc0c0b4870 con 0x7fdc0c0cb870 2024-04-06T12:12:15.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.179+0000 7fdc08ff9700 1 -- 172.21.15.189:0/2778019375 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc003c00 con 0x7fdc0c0cb870 2024-04-06T12:12:15.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.179+0000 7fdc1292c700 1 -- 172.21.15.189:0/2778019375 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc0c0c50c0 con 0x7fdc0c0cb870 2024-04-06T12:12:15.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.180+0000 7fdc1292c700 1 -- 172.21.15.189:0/2778019375 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc0c074990 con 0x7fdc0c0cb870 2024-04-06T12:12:15.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.180+0000 7fdc08ff9700 1 -- 172.21.15.189:0/2778019375 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdbfc01a430 con 0x7fdc0c0cb870 2024-04-06T12:12:15.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.180+0000 7fdc08ff9700 1 --2- 172.21.15.189:0/2778019375 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdbec06ac90 0x7fdbec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:15.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.180+0000 7fdc08ff9700 1 -- 172.21.15.189:0/2778019375 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fdbfc021070 con 0x7fdc0c0cb870 2024-04-06T12:12:15.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.180+0000 7fdc0b7fe700 1 --2- 172.21.15.189:0/2778019375 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdbec06ac90 0x7fdbec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:15.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.181+0000 7fdc0b7fe700 1 --2- 172.21.15.189:0/2778019375 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdbec06ac90 0x7fdbec06d150 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c068290 tx=0x7fdbf4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:15.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.185+0000 7fdc08ff9700 1 -- 172.21.15.189:0/2778019375 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdbfc055f90 con 0x7fdc0c0cb870 2024-04-06T12:12:15.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.378+0000 7fdc1292c700 1 -- 172.21.15.189:0/2778019375 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fdc0c0b4110 con 0x7fdc0c0cb870 2024-04-06T12:12:15.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.378+0000 7fdc08ff9700 1 -- 172.21.15.189:0/2778019375 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7fdc0c0b4110 con 0x7fdc0c0cb870 2024-04-06T12:12:15.378 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:12:15.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.381+0000 7fdc1292c700 1 -- 172.21.15.189:0/2778019375 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdbec06ac90 msgr2=0x7fdbec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:15.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.381+0000 7fdc1292c700 1 --2- 172.21.15.189:0/2778019375 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdbec06ac90 0x7fdbec06d150 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c068290 tx=0x7fdbf4008040 comp rx=0 tx=0).stop 2024-04-06T12:12:15.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.381+0000 7fdc1292c700 1 -- 172.21.15.189:0/2778019375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c0cb870 msgr2=0x7fdc0c154f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:15.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.381+0000 7fdc1292c700 1 --2- 172.21.15.189:0/2778019375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c0cb870 0x7fdc0c154f60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0090b0 tx=0x7fdbfc004750 comp rx=0 tx=0).stop 2024-04-06T12:12:15.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.381+0000 7fdc1292c700 1 -- 172.21.15.189:0/2778019375 shutdown_connections 2024-04-06T12:12:15.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.381+0000 7fdc1292c700 1 --2- 172.21.15.189:0/2778019375 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc0c0d11b0 0x7fdc0c158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.381+0000 7fdc1292c700 1 --2- 172.21.15.189:0/2778019375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c0cb870 0x7fdc0c154f60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.381+0000 7fdc1292c700 1 --2- 172.21.15.189:0/2778019375 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdbec06ac90 0x7fdbec06d150 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.381+0000 7fdc1292c700 1 --2- 172.21.15.189:0/2778019375 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc0c0cfa00 0x7fdc0c06c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.381+0000 7fdc1292c700 1 -- 172.21.15.189:0/2778019375 >> 172.21.15.189:0/2778019375 conn(0x7fdc0c0b2db0 msgr2=0x7fdc0c15ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:15.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.381+0000 7fdc1292c700 1 -- 172.21.15.189:0/2778019375 shutdown_connections 2024-04-06T12:12:15.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.381+0000 7fdc1292c700 1 -- 172.21.15.189:0/2778019375 wait complete. 2024-04-06T12:12:15.392 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T12:12:15.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.515+0000 7ffad033f700 1 Processor -- start 2024-04-06T12:12:15.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.515+0000 7ffad033f700 1 -- start start 2024-04-06T12:12:15.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.516+0000 7ffad033f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffac815a550 0x7ffac80cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:15.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.516+0000 7ffad033f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ffac80b4d70 0x7ffac80b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:15.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.516+0000 7ffad033f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac80cdbe0 con 0x7ffac80cf900 2024-04-06T12:12:15.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.516+0000 7ffad033f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac815a940 con 0x7ffac80b4d70 2024-04-06T12:12:15.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.516+0000 7ffad033f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac815ac00 con 0x7ffac815a550 2024-04-06T12:12:15.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.516+0000 7ffacd8da700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ffac80cf900 0x7ffac80b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55504/0 (socket says 172.21.15.189:55504) 2024-04-06T12:12:15.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.516+0000 7ffacd8da700 1 -- 172.21.15.189:0/1264389200 learned_addr learned my addr 172.21.15.189:0/1264389200 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:15.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.516+0000 7fface0db700 1 --2- 172.21.15.189:0/1264389200 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffac815a550 0x7ffac80cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:15.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.516+0000 7fface8dc700 1 --2- 172.21.15.189:0/1264389200 >> v2:172.21.15.192:3300/0 conn(0x7ffac80b4d70 0x7ffac80b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:15.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.516+0000 7ffacd0d9700 1 -- 172.21.15.189:0/1264389200 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111730573 0 0) 0x7ffac80cdbe0 con 0x7ffac80cf900 2024-04-06T12:12:15.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.516+0000 7ffacd0d9700 1 -- 172.21.15.189:0/1264389200 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffab40035b0 con 0x7ffac80cf900 2024-04-06T12:12:15.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.517+0000 7fface0db700 1 -- 172.21.15.189:0/1264389200 >> v2:172.21.15.192:3300/0 conn(0x7ffac80b4d70 msgr2=0x7ffac80b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:15.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.517+0000 7fface0db700 1 --2- 172.21.15.189:0/1264389200 >> v2:172.21.15.192:3300/0 conn(0x7ffac80b4d70 0x7ffac80b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.517+0000 7ffacd0d9700 1 -- 172.21.15.189:0/1264389200 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1398141673 0 0) 0x7ffab40035b0 con 0x7ffac80cf900 2024-04-06T12:12:15.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.517+0000 7fface0db700 1 -- 172.21.15.189:0/1264389200 >> v1:172.21.15.189:6789/0 conn(0x7ffac80cf900 legacy=0x7ffac80b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:15.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.517+0000 7fface0db700 1 -- 172.21.15.189:0/1264389200 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac815aee0 con 0x7ffac815a550 2024-04-06T12:12:15.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.517+0000 7fface8dc700 1 --2- 172.21.15.189:0/1264389200 >> v2:172.21.15.192:3300/0 conn(0x7ffac80b4d70 0x7ffac80b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:15.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.517+0000 7fface0db700 1 --2- 172.21.15.189:0/1264389200 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffac815a550 0x7ffac80cf3c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffac4009840 tx=0x7ffac4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d17aa3a84f383e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:15.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.517+0000 7ffacd0d9700 1 -- 172.21.15.189:0/1264389200 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab8001670 con 0x7ffac815a550 2024-04-06T12:12:15.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.517+0000 7ffacd0d9700 1 -- 172.21.15.189:0/1264389200 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac4004030 con 0x7ffac815a550 2024-04-06T12:12:15.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.517+0000 7ffacd0d9700 1 -- 172.21.15.189:0/1264389200 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac40043c0 con 0x7ffac815a550 2024-04-06T12:12:15.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.517+0000 7ffad033f700 1 -- 172.21.15.189:0/1264389200 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffac815a550 msgr2=0x7ffac80cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:15.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.517+0000 7ffad033f700 1 --2- 172.21.15.189:0/1264389200 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffac815a550 0x7ffac80cf3c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffac4009840 tx=0x7ffac4009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:15.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.518+0000 7ffad033f700 1 -- 172.21.15.189:0/1264389200 shutdown_connections 2024-04-06T12:12:15.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.518+0000 7ffad033f700 1 --2- 172.21.15.189:0/1264389200 >> v2:172.21.15.192:3300/0 conn(0x7ffac80b4d70 0x7ffac80b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.518+0000 7ffad033f700 1 --2- 172.21.15.189:0/1264389200 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffac815a550 0x7ffac80cf3c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.518+0000 7ffad033f700 1 -- 172.21.15.189:0/1264389200 >> 172.21.15.189:0/1264389200 conn(0x7ffac80d00b0 msgr2=0x7ffac80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:15.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.518+0000 7ffad033f700 1 -- 172.21.15.189:0/1264389200 shutdown_connections 2024-04-06T12:12:15.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.518+0000 7ffad033f700 1 -- 172.21.15.189:0/1264389200 wait complete. 2024-04-06T12:12:15.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.519+0000 7ffad033f700 1 Processor -- start 2024-04-06T12:12:15.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.519+0000 7ffad033f700 1 -- start start 2024-04-06T12:12:15.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffad033f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffac80b4d70 0x7ffac81466f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:15.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffad033f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac80cf900 0x7ffac80a9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:15.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffad033f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffac80aad80 0x7ffac815cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:15.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffad033f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac81f0860 con 0x7ffac80cf900 2024-04-06T12:12:15.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffad033f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac8147dc0 con 0x7ffac80aad80 2024-04-06T12:12:15.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffad033f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac8146c30 con 0x7ffac80b4d70 2024-04-06T12:12:15.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffacd8da700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac80cf900 0x7ffac80a9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:15.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffacd8da700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac80cf900 0x7ffac80a9f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56128/0 (socket says 172.21.15.189:56128) 2024-04-06T12:12:15.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffacd8da700 1 -- 172.21.15.189:0/2341433604 learned_addr learned my addr 172.21.15.189:0/2341433604 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:15.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7fface0db700 1 --2- 172.21.15.189:0/2341433604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffac80b4d70 0x7ffac81466f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:15.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7fface8dc700 1 --2- 172.21.15.189:0/2341433604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffac80aad80 0x7ffac815cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:15.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffacd8da700 1 -- 172.21.15.189:0/2341433604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffac80b4d70 msgr2=0x7ffac81466f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:15.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffacd8da700 1 --2- 172.21.15.189:0/2341433604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffac80b4d70 0x7ffac81466f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffacd8da700 1 -- 172.21.15.189:0/2341433604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffac80aad80 msgr2=0x7ffac815cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:15.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffacd8da700 1 --2- 172.21.15.189:0/2341433604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffac80aad80 0x7ffac815cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffacd8da700 1 -- 172.21.15.189:0/2341433604 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac0009a70 con 0x7ffac80cf900 2024-04-06T12:12:15.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffacd8da700 1 --2- 172.21.15.189:0/2341433604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac80cf900 0x7ffac80a9f10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffab800c7b0 tx=0x7ffab800cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:15.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffabeffd700 1 -- 172.21.15.189:0/2341433604 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab8004c80 con 0x7ffac80cf900 2024-04-06T12:12:15.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffabeffd700 1 -- 172.21.15.189:0/2341433604 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab8004de0 con 0x7ffac80cf900 2024-04-06T12:12:15.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffad033f700 1 -- 172.21.15.189:0/2341433604 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac40094a0 con 0x7ffac80cf900 2024-04-06T12:12:15.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffabeffd700 1 -- 172.21.15.189:0/2341433604 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab80072f0 con 0x7ffac80cf900 2024-04-06T12:12:15.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.520+0000 7ffad033f700 1 -- 172.21.15.189:0/2341433604 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac81471b0 con 0x7ffac80cf900 2024-04-06T12:12:15.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.521+0000 7ffad033f700 1 -- 172.21.15.189:0/2341433604 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac8147f70 con 0x7ffac80cf900 2024-04-06T12:12:15.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.522+0000 7ffabeffd700 1 -- 172.21.15.189:0/2341433604 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ffab8007450 con 0x7ffac80cf900 2024-04-06T12:12:15.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.522+0000 7ffabeffd700 1 --2- 172.21.15.189:0/2341433604 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffaac06acc0 0x7ffaac06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:15.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.522+0000 7ffabeffd700 1 -- 172.21.15.189:0/2341433604 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ffab8088970 con 0x7ffac80cf900 2024-04-06T12:12:15.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.522+0000 7fface0db700 1 --2- 172.21.15.189:0/2341433604 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffaac06acc0 0x7ffaac06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:15.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.523+0000 7fface0db700 1 --2- 172.21.15.189:0/2341433604 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffaac06acc0 0x7ffaac06d180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffac40090b0 tx=0x7ffac4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:15.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.527+0000 7ffabeffd700 1 -- 172.21.15.189:0/2341433604 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ffac8147f70 con 0x7ffac80cf900 2024-04-06T12:12:15.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.736+0000 7ffad033f700 1 -- 172.21.15.189:0/2341433604 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ffac8147f70 con 0x7ffac80cf900 2024-04-06T12:12:15.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.736+0000 7ffabeffd700 1 -- 172.21.15.189:0/2341433604 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7ffac8147f70 con 0x7ffac80cf900 2024-04-06T12:12:15.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.738+0000 7ffad033f700 1 -- 172.21.15.189:0/2341433604 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffaac06acc0 msgr2=0x7ffaac06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:15.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.738+0000 7ffad033f700 1 --2- 172.21.15.189:0/2341433604 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffaac06acc0 0x7ffaac06d180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffac40090b0 tx=0x7ffac4008030 comp rx=0 tx=0).stop 2024-04-06T12:12:15.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.738+0000 7ffad033f700 1 -- 172.21.15.189:0/2341433604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac80cf900 msgr2=0x7ffac80a9f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:15.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.738+0000 7ffad033f700 1 --2- 172.21.15.189:0/2341433604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac80cf900 0x7ffac80a9f10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffab800c7b0 tx=0x7ffab800cac0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.738+0000 7ffad033f700 1 -- 172.21.15.189:0/2341433604 shutdown_connections 2024-04-06T12:12:15.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.738+0000 7ffad033f700 1 --2- 172.21.15.189:0/2341433604 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffaac06acc0 0x7ffaac06d180 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.738+0000 7ffad033f700 1 --2- 172.21.15.189:0/2341433604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffac80aad80 0x7ffac815cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.738+0000 7ffad033f700 1 --2- 172.21.15.189:0/2341433604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffac80b4d70 0x7ffac81466f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.738+0000 7ffad033f700 1 --2- 172.21.15.189:0/2341433604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac80cf900 0x7ffac80a9f10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.738+0000 7ffad033f700 1 -- 172.21.15.189:0/2341433604 >> 172.21.15.189:0/2341433604 conn(0x7ffac80d00b0 msgr2=0x7ffac8157240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:15.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.738+0000 7ffad033f700 1 -- 172.21.15.189:0/2341433604 shutdown_connections 2024-04-06T12:12:15.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.738+0000 7ffad033f700 1 -- 172.21.15.189:0/2341433604 wait complete. 2024-04-06T12:12:15.739 INFO:teuthology.orchestra.run.smithi189.stderr:listed 0 entries 2024-04-06T12:12:15.749 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:12:15.749 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:12:15.767 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:15.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.928+0000 7fddea4d3700 1 Processor -- start 2024-04-06T12:12:15.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.929+0000 7fddea4d3700 1 -- start start 2024-04-06T12:12:15.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.929+0000 7fddea4d3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdde40cb870 0x7fdde415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:15.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.929+0000 7fddea4d3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdde40cfa00 0x7fdde40cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:15.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.929+0000 7fddea4d3700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde40cbd80 con 0x7fdde40d11b0 2024-04-06T12:12:15.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.929+0000 7fddea4d3700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde40ccdc0 con 0x7fdde40cb870 2024-04-06T12:12:15.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.929+0000 7fddea4d3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde40d0b50 con 0x7fdde40cfa00 2024-04-06T12:12:15.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.929+0000 7fdde37fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdde40cb870 0x7fdde415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:15.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.930+0000 7fdde37fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdde40cb870 0x7fdde415c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56142/0 (socket says 172.21.15.189:56142) 2024-04-06T12:12:15.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.930+0000 7fdde37fe700 1 -- 172.21.15.189:0/1154380183 learned_addr learned my addr 172.21.15.189:0/1154380183 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:15.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.930+0000 7fdde8a70700 1 --2- 172.21.15.189:0/1154380183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdde40cfa00 0x7fdde40cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:15.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.930+0000 7fdde37fe700 1 -- 172.21.15.189:0/1154380183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdde40cfa00 msgr2=0x7fdde40cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:15.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.930+0000 7fdde37fe700 1 --2- 172.21.15.189:0/1154380183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdde40cfa00 0x7fdde40cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.930+0000 7fdde37fe700 1 -- 172.21.15.189:0/1154380183 >> v1:172.21.15.192:6789/0 conn(0x7fdde40d11b0 legacy=0x7fdde40d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:15.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.930+0000 7fdde37fe700 1 -- 172.21.15.189:0/1154380183 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde40cef20 con 0x7fdde40cb870 2024-04-06T12:12:15.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.930+0000 7fdde37fe700 1 --2- 172.21.15.189:0/1154380183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdde40cb870 0x7fdde415c280 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fddd80098f0 tx=0x7fddd8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2e393b0ca67d15 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:15.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.930+0000 7fdde2ffd700 1 -- 172.21.15.189:0/1154380183 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd8011830 con 0x7fdde40cb870 2024-04-06T12:12:15.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.930+0000 7fdde2ffd700 1 -- 172.21.15.189:0/1154380183 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd8011990 con 0x7fdde40cb870 2024-04-06T12:12:15.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.930+0000 7fdde2ffd700 1 -- 172.21.15.189:0/1154380183 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd8011d20 con 0x7fdde40cb870 2024-04-06T12:12:15.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.931+0000 7fddea4d3700 1 -- 172.21.15.189:0/1154380183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdde40cb870 msgr2=0x7fdde415c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:15.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.931+0000 7fddea4d3700 1 --2- 172.21.15.189:0/1154380183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdde40cb870 0x7fdde415c280 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fddd80098f0 tx=0x7fddd8009c00 comp rx=0 tx=0).stop 2024-04-06T12:12:15.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.931+0000 7fddea4d3700 1 -- 172.21.15.189:0/1154380183 shutdown_connections 2024-04-06T12:12:15.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.931+0000 7fddea4d3700 1 --2- 172.21.15.189:0/1154380183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdde40cfa00 0x7fdde40cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.931+0000 7fddea4d3700 1 --2- 172.21.15.189:0/1154380183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdde40cb870 0x7fdde415c280 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.931+0000 7fddea4d3700 1 -- 172.21.15.189:0/1154380183 >> 172.21.15.189:0/1154380183 conn(0x7fdde40b2db0 msgr2=0x7fdde40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:15.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.931+0000 7fddea4d3700 1 -- 172.21.15.189:0/1154380183 shutdown_connections 2024-04-06T12:12:15.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.931+0000 7fddea4d3700 1 -- 172.21.15.189:0/1154380183 wait complete. 2024-04-06T12:12:15.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.932+0000 7fddea4d3700 1 Processor -- start 2024-04-06T12:12:15.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.932+0000 7fddea4d3700 1 -- start start 2024-04-06T12:12:15.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.933+0000 7fddea4d3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdde40cb870 0x7fdde40b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:15.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.933+0000 7fddea4d3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdde40cfa00 0x7fdde4158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:15.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.933+0000 7fddea4d3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdde40d11b0 0x7fdde4151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:15.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.933+0000 7fddea4d3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde4151c70 con 0x7fdde40cfa00 2024-04-06T12:12:15.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.933+0000 7fddea4d3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde40b44f0 con 0x7fdde40cb870 2024-04-06T12:12:15.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.933+0000 7fddea4d3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde40b47d0 con 0x7fdde40d11b0 2024-04-06T12:12:15.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.933+0000 7fdde37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdde40cfa00 0x7fdde4158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:15.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.933+0000 7fdde37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdde40cfa00 0x7fdde4158940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56146/0 (socket says 172.21.15.189:56146) 2024-04-06T12:12:15.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.933+0000 7fdde37fe700 1 -- 172.21.15.189:0/1813465017 learned_addr learned my addr 172.21.15.189:0/1813465017 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:15.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.933+0000 7fdde3fff700 1 --2- 172.21.15.189:0/1813465017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdde40cb870 0x7fdde40b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:15.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.933+0000 7fdde8a70700 1 --2- 172.21.15.189:0/1813465017 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdde40d11b0 0x7fdde4151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:15.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.933+0000 7fdde37fe700 1 -- 172.21.15.189:0/1813465017 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdde40d11b0 msgr2=0x7fdde4151730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:15.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.933+0000 7fdde37fe700 1 --2- 172.21.15.189:0/1813465017 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdde40d11b0 0x7fdde4151730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.933+0000 7fdde37fe700 1 -- 172.21.15.189:0/1813465017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdde40cb870 msgr2=0x7fdde40b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:15.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.933+0000 7fdde37fe700 1 --2- 172.21.15.189:0/1813465017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdde40cb870 0x7fdde40b3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:15.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.933+0000 7fdde37fe700 1 -- 172.21.15.189:0/1813465017 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd8009550 con 0x7fdde40cfa00 2024-04-06T12:12:15.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.933+0000 7fdde37fe700 1 --2- 172.21.15.189:0/1813465017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdde40cfa00 0x7fdde4158940 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fddd8000c00 tx=0x7fddd8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:15.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.933+0000 7fdde0ff9700 1 -- 172.21.15.189:0/1813465017 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd8004380 con 0x7fdde40cfa00 2024-04-06T12:12:15.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.934+0000 7fdde0ff9700 1 -- 172.21.15.189:0/1813465017 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd801f070 con 0x7fdde40cfa00 2024-04-06T12:12:15.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.934+0000 7fdde0ff9700 1 -- 172.21.15.189:0/1813465017 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd80036a0 con 0x7fdde40cfa00 2024-04-06T12:12:15.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.934+0000 7fddea4d3700 1 -- 172.21.15.189:0/1813465017 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde40b4ab0 con 0x7fdde40cfa00 2024-04-06T12:12:15.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.934+0000 7fddea4d3700 1 -- 172.21.15.189:0/1813465017 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde40b5070 con 0x7fdde40cfa00 2024-04-06T12:12:15.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.934+0000 7fddea4d3700 1 -- 172.21.15.189:0/1813465017 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde40b4910 con 0x7fdde40cfa00 2024-04-06T12:12:15.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.935+0000 7fdde0ff9700 1 -- 172.21.15.189:0/1813465017 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fddd8021030 con 0x7fdde40cfa00 2024-04-06T12:12:15.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.935+0000 7fdde0ff9700 1 --2- 172.21.15.189:0/1813465017 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fddd006acc0 0x7fddd006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:15.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.935+0000 7fdde0ff9700 1 -- 172.21.15.189:0/1813465017 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fddd808a9c0 con 0x7fdde40cfa00 2024-04-06T12:12:15.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.935+0000 7fdde3fff700 1 --2- 172.21.15.189:0/1813465017 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fddd006acc0 0x7fddd006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:15.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.936+0000 7fdde3fff700 1 --2- 172.21.15.189:0/1813465017 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fddd006acc0 0x7fddd006d180 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fddd4007fd0 tx=0x7fddd4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:15.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:15.940+0000 7fdde0ff9700 1 -- 172.21.15.189:0/1813465017 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdde40b4910 con 0x7fdde40cfa00 2024-04-06T12:12:16.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.136+0000 7fddea4d3700 1 -- 172.21.15.189:0/1813465017 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdde40b4910 con 0x7fdde40cfa00 2024-04-06T12:12:16.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.138+0000 7fdde0ff9700 1 -- 172.21.15.189:0/1813465017 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17926 (secure 0 0 0) 0x7fdde40b4910 con 0x7fdde40cfa00 2024-04-06T12:12:16.137 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:16.137 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:08.649170+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:16.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.140+0000 7fddea4d3700 1 -- 172.21.15.189:0/1813465017 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fddd006acc0 msgr2=0x7fddd006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:16.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.140+0000 7fddea4d3700 1 --2- 172.21.15.189:0/1813465017 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fddd006acc0 0x7fddd006d180 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fddd4007fd0 tx=0x7fddd4009040 comp rx=0 tx=0).stop 2024-04-06T12:12:16.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.140+0000 7fddea4d3700 1 -- 172.21.15.189:0/1813465017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdde40cfa00 msgr2=0x7fdde4158940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:16.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.140+0000 7fddea4d3700 1 --2- 172.21.15.189:0/1813465017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdde40cfa00 0x7fdde4158940 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fddd8000c00 tx=0x7fddd8004750 comp rx=0 tx=0).stop 2024-04-06T12:12:16.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.140+0000 7fddea4d3700 1 -- 172.21.15.189:0/1813465017 shutdown_connections 2024-04-06T12:12:16.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.140+0000 7fddea4d3700 1 --2- 172.21.15.189:0/1813465017 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdde40d11b0 0x7fdde4151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.140+0000 7fddea4d3700 1 --2- 172.21.15.189:0/1813465017 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fddd006acc0 0x7fddd006d180 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.140+0000 7fddea4d3700 1 --2- 172.21.15.189:0/1813465017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdde40cb870 0x7fdde40b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.140+0000 7fddea4d3700 1 --2- 172.21.15.189:0/1813465017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdde40cfa00 0x7fdde4158940 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.140+0000 7fddea4d3700 1 -- 172.21.15.189:0/1813465017 >> 172.21.15.189:0/1813465017 conn(0x7fdde40b2db0 msgr2=0x7fdde40d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:16.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.140+0000 7fddea4d3700 1 -- 172.21.15.189:0/1813465017 shutdown_connections 2024-04-06T12:12:16.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.140+0000 7fddea4d3700 1 -- 172.21.15.189:0/1813465017 wait complete. 2024-04-06T12:12:16.151 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:12:02.280866+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:12:03.156593+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-06T12:12:06.237184+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-06T12:12:16.151 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:12:16.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.280+0000 7fa316db2700 1 Processor -- start 2024-04-06T12:12:16.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.280+0000 7fa316db2700 1 -- start start 2024-04-06T12:12:16.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.281+0000 7fa316db2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa31015d7b0 0x7fa3100b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:16.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.281+0000 7fa316db2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa31006a730 0x7fa31006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:16.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.281+0000 7fa316db2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3100b3990 con 0x7fa31015ddf0 2024-04-06T12:12:16.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.281+0000 7fa316db2700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa31006b0f0 con 0x7fa31006a730 2024-04-06T12:12:16.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.281+0000 7fa316db2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3100b3060 con 0x7fa31015d7b0 2024-04-06T12:12:16.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.281+0000 7fa30ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa31006a730 0x7fa31006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:16.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.281+0000 7fa31534f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa31015ddf0 0x7fa31015e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55510/0 (socket says 172.21.15.189:55510) 2024-04-06T12:12:16.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.281+0000 7fa31534f700 1 -- 172.21.15.189:0/2661079376 learned_addr learned my addr 172.21.15.189:0/2661079376 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:16.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.281+0000 7fa314b4e700 1 --2- 172.21.15.189:0/2661079376 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa31015d7b0 0x7fa3100b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:16.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.281+0000 7fa30ffff700 1 -- 172.21.15.189:0/2661079376 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa31015d7b0 msgr2=0x7fa3100b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:16.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.281+0000 7fa30ffff700 1 --2- 172.21.15.189:0/2661079376 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa31015d7b0 0x7fa3100b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.281+0000 7fa30ffff700 1 -- 172.21.15.189:0/2661079376 >> v1:172.21.15.189:6789/0 conn(0x7fa31015ddf0 legacy=0x7fa31015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:16.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.281+0000 7fa30ffff700 1 -- 172.21.15.189:0/2661079376 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3100b3340 con 0x7fa31006a730 2024-04-06T12:12:16.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.281+0000 7fa30f7fe700 1 -- 172.21.15.189:0/2661079376 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244229553 0 0) 0x7fa3100b3990 con 0x7fa31015ddf0 2024-04-06T12:12:16.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.281+0000 7fa30ffff700 1 --2- 172.21.15.189:0/2661079376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa31006a730 0x7fa31006abb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc009840 tx=0x7fa2fc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8ec984746540a2b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:16.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.282+0000 7fa30f7fe700 1 -- 172.21.15.189:0/2661079376 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc011830 con 0x7fa31006a730 2024-04-06T12:12:16.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.282+0000 7fa30f7fe700 1 -- 172.21.15.189:0/2661079376 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2fc011990 con 0x7fa31006a730 2024-04-06T12:12:16.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.282+0000 7fa30f7fe700 1 -- 172.21.15.189:0/2661079376 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc011d20 con 0x7fa31006a730 2024-04-06T12:12:16.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.282+0000 7fa316db2700 1 -- 172.21.15.189:0/2661079376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa31006a730 msgr2=0x7fa31006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:16.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.282+0000 7fa316db2700 1 --2- 172.21.15.189:0/2661079376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa31006a730 0x7fa31006abb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc009840 tx=0x7fa2fc009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:16.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.282+0000 7fa316db2700 1 -- 172.21.15.189:0/2661079376 shutdown_connections 2024-04-06T12:12:16.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.282+0000 7fa316db2700 1 --2- 172.21.15.189:0/2661079376 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa31015d7b0 0x7fa3100b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.282+0000 7fa316db2700 1 --2- 172.21.15.189:0/2661079376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa31006a730 0x7fa31006abb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.282+0000 7fa316db2700 1 -- 172.21.15.189:0/2661079376 >> 172.21.15.189:0/2661079376 conn(0x7fa3100d00b0 msgr2=0x7fa3100d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:16.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.282+0000 7fa316db2700 1 -- 172.21.15.189:0/2661079376 shutdown_connections 2024-04-06T12:12:16.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.282+0000 7fa316db2700 1 -- 172.21.15.189:0/2661079376 wait complete. 2024-04-06T12:12:16.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.283+0000 7fa316db2700 1 Processor -- start 2024-04-06T12:12:16.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.283+0000 7fa316db2700 1 -- start start 2024-04-06T12:12:16.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.284+0000 7fa316db2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa31006a730 0x7fa3100c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:16.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.284+0000 7fa316db2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa31015d7b0 0x7fa3101f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:16.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.284+0000 7fa316db2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa31015ddf0 0x7fa31015fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:16.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.284+0000 7fa316db2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31014b8c0 con 0x7fa31015ddf0 2024-04-06T12:12:16.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.284+0000 7fa316db2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31014bba0 con 0x7fa31015d7b0 2024-04-06T12:12:16.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.284+0000 7fa316db2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa31014be80 con 0x7fa31006a730 2024-04-06T12:12:16.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.284+0000 7fa31534f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa31015ddf0 0x7fa31015fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:16.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.284+0000 7fa31534f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa31015ddf0 0x7fa31015fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56152/0 (socket says 172.21.15.189:56152) 2024-04-06T12:12:16.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.284+0000 7fa31534f700 1 -- 172.21.15.189:0/1525923417 learned_addr learned my addr 172.21.15.189:0/1525923417 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:16.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.285+0000 7fa314b4e700 1 --2- 172.21.15.189:0/1525923417 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa31006a730 0x7fa3100c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:16.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.285+0000 7fa30ffff700 1 --2- 172.21.15.189:0/1525923417 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa31015d7b0 0x7fa3101f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:16.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.285+0000 7fa31534f700 1 -- 172.21.15.189:0/1525923417 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa31006a730 msgr2=0x7fa3100c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:16.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.285+0000 7fa31534f700 1 --2- 172.21.15.189:0/1525923417 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa31006a730 0x7fa3100c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.285+0000 7fa31534f700 1 -- 172.21.15.189:0/1525923417 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa31015d7b0 msgr2=0x7fa3101f22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:16.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.285+0000 7fa31534f700 1 --2- 172.21.15.189:0/1525923417 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa31015d7b0 0x7fa3101f22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.285+0000 7fa31534f700 1 -- 172.21.15.189:0/1525923417 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2fc0094a0 con 0x7fa31015ddf0 2024-04-06T12:12:16.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.285+0000 7fa31534f700 1 --2- 172.21.15.189:0/1525923417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa31015ddf0 0x7fa31015fd10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa30800cba0 tx=0x7fa30800ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:16.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.285+0000 7fa30d7fa700 1 -- 172.21.15.189:0/1525923417 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30800aaa0 con 0x7fa31015ddf0 2024-04-06T12:12:16.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.285+0000 7fa30d7fa700 1 -- 172.21.15.189:0/1525923417 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30800ac20 con 0x7fa31015ddf0 2024-04-06T12:12:16.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.285+0000 7fa316db2700 1 -- 172.21.15.189:0/1525923417 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa31014c160 con 0x7fa31015ddf0 2024-04-06T12:12:16.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.285+0000 7fa30d7fa700 1 -- 172.21.15.189:0/1525923417 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3080070d0 con 0x7fa31015ddf0 2024-04-06T12:12:16.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.285+0000 7fa316db2700 1 -- 172.21.15.189:0/1525923417 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa31014b5e0 con 0x7fa31015ddf0 2024-04-06T12:12:16.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.286+0000 7fa316db2700 1 -- 172.21.15.189:0/1525923417 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa310067bb0 con 0x7fa31015ddf0 2024-04-06T12:12:16.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.286+0000 7fa30d7fa700 1 -- 172.21.15.189:0/1525923417 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa308007230 con 0x7fa31015ddf0 2024-04-06T12:12:16.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.287+0000 7fa30d7fa700 1 --2- 172.21.15.189:0/1525923417 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2f406ac90 0x7fa2f406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:16.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.287+0000 7fa30d7fa700 1 -- 172.21.15.189:0/1525923417 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa308089330 con 0x7fa31015ddf0 2024-04-06T12:12:16.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.287+0000 7fa314b4e700 1 --2- 172.21.15.189:0/1525923417 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2f406ac90 0x7fa2f406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:16.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.287+0000 7fa314b4e700 1 --2- 172.21.15.189:0/1525923417 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2f406ac90 0x7fa2f406d150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa304009be0 tx=0x7fa304009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:16.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.291+0000 7fa30d7fa700 1 -- 172.21.15.189:0/1525923417 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa308054220 con 0x7fa31015ddf0 2024-04-06T12:12:16.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.485+0000 7fa316db2700 1 -- 172.21.15.189:0/1525923417 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa310074990 con 0x7fa31015ddf0 2024-04-06T12:12:16.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.485+0000 7fa30d7fa700 1 -- 172.21.15.189:0/1525923417 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa308057840 con 0x7fa31015ddf0 2024-04-06T12:12:16.484 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:12:16.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.487+0000 7fa316db2700 1 -- 172.21.15.189:0/1525923417 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2f406ac90 msgr2=0x7fa2f406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:16.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.487+0000 7fa316db2700 1 --2- 172.21.15.189:0/1525923417 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2f406ac90 0x7fa2f406d150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa304009be0 tx=0x7fa304009210 comp rx=0 tx=0).stop 2024-04-06T12:12:16.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.487+0000 7fa316db2700 1 -- 172.21.15.189:0/1525923417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa31015ddf0 msgr2=0x7fa31015fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:16.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.487+0000 7fa316db2700 1 --2- 172.21.15.189:0/1525923417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa31015ddf0 0x7fa31015fd10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa30800cba0 tx=0x7fa30800ceb0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.487+0000 7fa316db2700 1 -- 172.21.15.189:0/1525923417 shutdown_connections 2024-04-06T12:12:16.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.487+0000 7fa316db2700 1 --2- 172.21.15.189:0/1525923417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa31015ddf0 0x7fa31015fd10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.487+0000 7fa316db2700 1 --2- 172.21.15.189:0/1525923417 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa31006a730 0x7fa3100c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.487+0000 7fa316db2700 1 --2- 172.21.15.189:0/1525923417 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2f406ac90 0x7fa2f406d150 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.487+0000 7fa316db2700 1 --2- 172.21.15.189:0/1525923417 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa31015d7b0 0x7fa3101f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.487+0000 7fa316db2700 1 -- 172.21.15.189:0/1525923417 >> 172.21.15.189:0/1525923417 conn(0x7fa3100d00b0 msgr2=0x7fa3100aeb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:16.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.487+0000 7fa316db2700 1 -- 172.21.15.189:0/1525923417 shutdown_connections 2024-04-06T12:12:16.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.487+0000 7fa316db2700 1 -- 172.21.15.189:0/1525923417 wait complete. 2024-04-06T12:12:16.498 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:12:16.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.624+0000 7f8d93d51700 1 Processor -- start 2024-04-06T12:12:16.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.624+0000 7f8d93d51700 1 -- start start 2024-04-06T12:12:16.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.625+0000 7f8d93d51700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8d8c15a550 0x7f8d8c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:16.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.625+0000 7f8d93d51700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8d8c0b4d70 0x7f8d8c0b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:16.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.625+0000 7f8d93d51700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d8c0cdbe0 con 0x7f8d8c0cf900 2024-04-06T12:12:16.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.625+0000 7f8d93d51700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d8c15a940 con 0x7f8d8c15a550 2024-04-06T12:12:16.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.625+0000 7f8d93d51700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d8c15ac00 con 0x7f8d8c0b4d70 2024-04-06T12:12:16.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.625+0000 7f8d91aed700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8d8c15a550 0x7f8d8c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:16.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.625+0000 7f8d91aed700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8d8c15a550 0x7f8d8c0cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56154/0 (socket says 172.21.15.189:56154) 2024-04-06T12:12:16.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.625+0000 7f8d91aed700 1 -- 172.21.15.189:0/1058544539 learned_addr learned my addr 172.21.15.189:0/1058544539 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:16.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.625+0000 7f8d922ee700 1 --2- 172.21.15.189:0/1058544539 >> v2:172.21.15.192:3300/0 conn(0x7f8d8c0b4d70 0x7f8d8c0b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:16.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.626+0000 7f8d91aed700 1 -- 172.21.15.189:0/1058544539 >> v2:172.21.15.192:3300/0 conn(0x7f8d8c0b4d70 msgr2=0x7f8d8c0b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:16.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.626+0000 7f8d91aed700 1 --2- 172.21.15.189:0/1058544539 >> v2:172.21.15.192:3300/0 conn(0x7f8d8c0b4d70 0x7f8d8c0b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.626+0000 7f8d91aed700 1 -- 172.21.15.189:0/1058544539 >> v1:172.21.15.192:6789/0 conn(0x7f8d8c0cf900 legacy=0x7f8d8c0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:16.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.626+0000 7f8d91aed700 1 -- 172.21.15.189:0/1058544539 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c15aee0 con 0x7f8d8c15a550 2024-04-06T12:12:16.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.626+0000 7f8d91aed700 1 --2- 172.21.15.189:0/1058544539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c15a550 0x7f8d8c0cf3c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c072830 tx=0x7f8d7c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fece223c231eb27e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:16.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.626+0000 7f8d90aeb700 1 -- 172.21.15.189:0/1058544539 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c011830 con 0x7f8d8c15a550 2024-04-06T12:12:16.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.626+0000 7f8d90aeb700 1 -- 172.21.15.189:0/1058544539 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7c011990 con 0x7f8d8c15a550 2024-04-06T12:12:16.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.626+0000 7f8d90aeb700 1 -- 172.21.15.189:0/1058544539 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c011d20 con 0x7f8d8c15a550 2024-04-06T12:12:16.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.626+0000 7f8d93d51700 1 -- 172.21.15.189:0/1058544539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c15a550 msgr2=0x7f8d8c0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:16.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.626+0000 7f8d93d51700 1 --2- 172.21.15.189:0/1058544539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c15a550 0x7f8d8c0cf3c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c072830 tx=0x7f8d7c009bd0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.627+0000 7f8d93d51700 1 -- 172.21.15.189:0/1058544539 shutdown_connections 2024-04-06T12:12:16.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.627+0000 7f8d93d51700 1 --2- 172.21.15.189:0/1058544539 >> v2:172.21.15.192:3300/0 conn(0x7f8d8c0b4d70 0x7f8d8c0b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.627+0000 7f8d93d51700 1 --2- 172.21.15.189:0/1058544539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c15a550 0x7f8d8c0cf3c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.627+0000 7f8d93d51700 1 -- 172.21.15.189:0/1058544539 >> 172.21.15.189:0/1058544539 conn(0x7f8d8c0d00b0 msgr2=0x7f8d8c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:16.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.627+0000 7f8d93d51700 1 -- 172.21.15.189:0/1058544539 shutdown_connections 2024-04-06T12:12:16.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.627+0000 7f8d93d51700 1 -- 172.21.15.189:0/1058544539 wait complete. 2024-04-06T12:12:16.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.628+0000 7f8d93d51700 1 Processor -- start 2024-04-06T12:12:16.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.628+0000 7f8d93d51700 1 -- start start 2024-04-06T12:12:16.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d93d51700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d8c0b4d70 0x7f8d8c0cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:16.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d93d51700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d8c0cf900 0x7f8d8c1595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:16.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d93d51700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c157c70 0x7f8d8c14a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:16.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d93d51700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c0ad100 con 0x7f8d8c157c70 2024-04-06T12:12:16.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d93d51700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c0ad740 con 0x7f8d8c0cf900 2024-04-06T12:12:16.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d93d51700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c14ab00 con 0x7f8d8c0b4d70 2024-04-06T12:12:16.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d922ee700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c157c70 0x7f8d8c14a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:16.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d922ee700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c157c70 0x7f8d8c14a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56170/0 (socket says 172.21.15.189:56170) 2024-04-06T12:12:16.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d922ee700 1 -- 172.21.15.189:0/2013646608 learned_addr learned my addr 172.21.15.189:0/2013646608 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:16.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d912ec700 1 --2- 172.21.15.189:0/2013646608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d8c0cf900 0x7f8d8c1595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:16.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d91aed700 1 --2- 172.21.15.189:0/2013646608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d8c0b4d70 0x7f8d8c0cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:16.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d922ee700 1 -- 172.21.15.189:0/2013646608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d8c0b4d70 msgr2=0x7f8d8c0cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:16.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d922ee700 1 --2- 172.21.15.189:0/2013646608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d8c0b4d70 0x7f8d8c0cba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d922ee700 1 -- 172.21.15.189:0/2013646608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d8c0cf900 msgr2=0x7f8d8c1595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:16.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d922ee700 1 --2- 172.21.15.189:0/2013646608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d8c0cf900 0x7f8d8c1595b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d922ee700 1 -- 172.21.15.189:0/2013646608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c009550 con 0x7f8d8c157c70 2024-04-06T12:12:16.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d922ee700 1 --2- 172.21.15.189:0/2013646608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c157c70 0x7f8d8c14a5c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8d8400efc0 tx=0x7f8d8400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:16.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d827fc700 1 -- 172.21.15.189:0/2013646608 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d84009ea0 con 0x7f8d8c157c70 2024-04-06T12:12:16.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d827fc700 1 -- 172.21.15.189:0/2013646608 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8400f040 con 0x7f8d8c157c70 2024-04-06T12:12:16.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d827fc700 1 -- 172.21.15.189:0/2013646608 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8400ced0 con 0x7f8d8c157c70 2024-04-06T12:12:16.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d93d51700 1 -- 172.21.15.189:0/2013646608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d8c158100 con 0x7f8d8c157c70 2024-04-06T12:12:16.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.629+0000 7f8d93d51700 1 -- 172.21.15.189:0/2013646608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d8c0ad9e0 con 0x7f8d8c157c70 2024-04-06T12:12:16.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.630+0000 7f8d93d51700 1 -- 172.21.15.189:0/2013646608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d8c067bb0 con 0x7f8d8c157c70 2024-04-06T12:12:16.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.631+0000 7f8d827fc700 1 -- 172.21.15.189:0/2013646608 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8d84014480 con 0x7f8d8c157c70 2024-04-06T12:12:16.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.631+0000 7f8d827fc700 1 --2- 172.21.15.189:0/2013646608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8d7006ac90 0x7f8d7006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:16.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.631+0000 7f8d827fc700 1 -- 172.21.15.189:0/2013646608 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f8d8408a100 con 0x7f8d8c157c70 2024-04-06T12:12:16.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.631+0000 7f8d91aed700 1 --2- 172.21.15.189:0/2013646608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8d7006ac90 0x7f8d7006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:16.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.632+0000 7f8d91aed700 1 --2- 172.21.15.189:0/2013646608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8d7006ac90 0x7f8d7006d150 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c014040 tx=0x7f8d7c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:16.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.636+0000 7f8d827fc700 1 -- 172.21.15.189:0/2013646608 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8d84054ff0 con 0x7f8d8c157c70 2024-04-06T12:12:16.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.833+0000 7f8d93d51700 1 -- 172.21.15.189:0/2013646608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8d8c074990 con 0x7f8d8c157c70 2024-04-06T12:12:16.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.834+0000 7f8d827fc700 1 -- 172.21.15.189:0/2013646608 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8d84058610 con 0x7f8d8c157c70 2024-04-06T12:12:16.833 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:12:16.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.836+0000 7f8d93d51700 1 -- 172.21.15.189:0/2013646608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8d7006ac90 msgr2=0x7f8d7006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:16.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.836+0000 7f8d93d51700 1 --2- 172.21.15.189:0/2013646608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8d7006ac90 0x7f8d7006d150 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c014040 tx=0x7f8d7c008030 comp rx=0 tx=0).stop 2024-04-06T12:12:16.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.836+0000 7f8d93d51700 1 -- 172.21.15.189:0/2013646608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c157c70 msgr2=0x7f8d8c14a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:16.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.836+0000 7f8d93d51700 1 --2- 172.21.15.189:0/2013646608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c157c70 0x7f8d8c14a5c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8d8400efc0 tx=0x7f8d8400c3a0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.836+0000 7f8d93d51700 1 -- 172.21.15.189:0/2013646608 shutdown_connections 2024-04-06T12:12:16.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.836+0000 7f8d93d51700 1 --2- 172.21.15.189:0/2013646608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c157c70 0x7f8d8c14a5c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.836+0000 7f8d93d51700 1 --2- 172.21.15.189:0/2013646608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d8c0b4d70 0x7f8d8c0cba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.836+0000 7f8d93d51700 1 --2- 172.21.15.189:0/2013646608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8d7006ac90 0x7f8d7006d150 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.836+0000 7f8d93d51700 1 --2- 172.21.15.189:0/2013646608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d8c0cf900 0x7f8d8c1595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.836+0000 7f8d93d51700 1 -- 172.21.15.189:0/2013646608 >> 172.21.15.189:0/2013646608 conn(0x7f8d8c0d00b0 msgr2=0x7f8d8c06d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:16.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.836+0000 7f8d93d51700 1 -- 172.21.15.189:0/2013646608 shutdown_connections 2024-04-06T12:12:16.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.836+0000 7f8d93d51700 1 -- 172.21.15.189:0/2013646608 wait complete. 2024-04-06T12:12:16.847 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:12:16.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.972+0000 7fd3d6a25700 1 Processor -- start 2024-04-06T12:12:16.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.972+0000 7fd3d6a25700 1 -- start start 2024-04-06T12:12:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.973+0000 7fd3d6a25700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3d00cd3e0 0x7fd3d00cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.973+0000 7fd3d6a25700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d006ceb0 con 0x7fd3d006b8c0 2024-04-06T12:12:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.973+0000 7fd3d6a25700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d006d750 con 0x7fd3d015d7c0 2024-04-06T12:12:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.973+0000 7fd3d6a25700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d00cdfa0 con 0x7fd3d00cd3e0 2024-04-06T12:12:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.973+0000 7fd3d4fc2700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd3d006b8c0 0x7fd3d006dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55512/0 (socket says 172.21.15.189:55512) 2024-04-06T12:12:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.973+0000 7fd3d4fc2700 1 -- 172.21.15.189:0/2730379841 learned_addr learned my addr 172.21.15.189:0/2730379841 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.973+0000 7fd3cffff700 1 --2- 172.21.15.189:0/2730379841 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3d00cd3e0 0x7fd3d00cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.973+0000 7fd3ceffd700 1 -- 172.21.15.189:0/2730379841 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482349819 0 0) 0x7fd3d006ceb0 con 0x7fd3d006b8c0 2024-04-06T12:12:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.973+0000 7fd3ceffd700 1 -- 172.21.15.189:0/2730379841 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3b80035b0 con 0x7fd3d006b8c0 2024-04-06T12:12:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.973+0000 7fd3ceffd700 1 -- 172.21.15.189:0/2730379841 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825979781 0 0) 0x7fd3d006d750 con 0x7fd3d015d7c0 2024-04-06T12:12:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.973+0000 7fd3ceffd700 1 -- 172.21.15.189:0/2730379841 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3d006ceb0 con 0x7fd3d015d7c0 2024-04-06T12:12:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.973+0000 7fd3cffff700 1 -- 172.21.15.189:0/2730379841 >> v1:172.21.15.192:6789/0 conn(0x7fd3d015d7c0 legacy=0x7fd3d006b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.973+0000 7fd3cffff700 1 -- 172.21.15.189:0/2730379841 >> v1:172.21.15.189:6789/0 conn(0x7fd3d006b8c0 legacy=0x7fd3d006dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.973+0000 7fd3cffff700 1 -- 172.21.15.189:0/2730379841 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d00cdd00 con 0x7fd3d00cd3e0 2024-04-06T12:12:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.974+0000 7fd3cffff700 1 --2- 172.21.15.189:0/2730379841 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3d00cd3e0 0x7fd3d00cd7c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4009870 tx=0x7fd3c4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a94aeb60c61d263c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.974+0000 7fd3ceffd700 1 -- 172.21.15.189:0/2730379841 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4011830 con 0x7fd3d00cd3e0 2024-04-06T12:12:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.974+0000 7fd3ceffd700 1 -- 172.21.15.189:0/2730379841 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c4011990 con 0x7fd3d00cd3e0 2024-04-06T12:12:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.974+0000 7fd3ceffd700 1 -- 172.21.15.189:0/2730379841 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4011d20 con 0x7fd3d00cd3e0 2024-04-06T12:12:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.974+0000 7fd3d6a25700 1 -- 172.21.15.189:0/2730379841 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3d00cd3e0 msgr2=0x7fd3d00cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.974+0000 7fd3d6a25700 1 --2- 172.21.15.189:0/2730379841 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3d00cd3e0 0x7fd3d00cd7c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4009870 tx=0x7fd3c4009b80 comp rx=0 tx=0).stop 2024-04-06T12:12:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.974+0000 7fd3d6a25700 1 -- 172.21.15.189:0/2730379841 shutdown_connections 2024-04-06T12:12:16.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.974+0000 7fd3d6a25700 1 --2- 172.21.15.189:0/2730379841 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3d00cd3e0 0x7fd3d00cd7c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.974+0000 7fd3d6a25700 1 -- 172.21.15.189:0/2730379841 >> 172.21.15.189:0/2730379841 conn(0x7fd3d00cc290 msgr2=0x7fd3d00cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:16.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.974+0000 7fd3d6a25700 1 -- 172.21.15.189:0/2730379841 shutdown_connections 2024-04-06T12:12:16.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.974+0000 7fd3d6a25700 1 -- 172.21.15.189:0/2730379841 wait complete. 2024-04-06T12:12:16.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.975+0000 7fd3d6a25700 1 Processor -- start 2024-04-06T12:12:16.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.976+0000 7fd3d6a25700 1 -- start start 2024-04-06T12:12:16.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.976+0000 7fd3d6a25700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3d006b8c0 0x7fd3d0158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:16.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.976+0000 7fd3d6a25700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3d00cd3e0 0x7fd3d00c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:16.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.976+0000 7fd3d6a25700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3d015d7c0 0x7fd3d00b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:16.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.976+0000 7fd3d6a25700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d01596d0 con 0x7fd3d006b8c0 2024-04-06T12:12:16.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.976+0000 7fd3d6a25700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d00b5130 con 0x7fd3d00cd3e0 2024-04-06T12:12:16.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.976+0000 7fd3d6a25700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d0159130 con 0x7fd3d015d7c0 2024-04-06T12:12:16.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.976+0000 7fd3cffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3d006b8c0 0x7fd3d0158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:16.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.977+0000 7fd3cffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3d006b8c0 0x7fd3d0158bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56178/0 (socket says 172.21.15.189:56178) 2024-04-06T12:12:16.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.977+0000 7fd3cffff700 1 -- 172.21.15.189:0/2971995905 learned_addr learned my addr 172.21.15.189:0/2971995905 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:16.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.977+0000 7fd3d4fc2700 1 --2- 172.21.15.189:0/2971995905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3d015d7c0 0x7fd3d00b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:16.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.977+0000 7fd3cf7fe700 1 --2- 172.21.15.189:0/2971995905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3d00cd3e0 0x7fd3d00c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:16.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.977+0000 7fd3cffff700 1 -- 172.21.15.189:0/2971995905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3d015d7c0 msgr2=0x7fd3d00b38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:16.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.977+0000 7fd3cffff700 1 --2- 172.21.15.189:0/2971995905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3d015d7c0 0x7fd3d00b38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.977+0000 7fd3cffff700 1 -- 172.21.15.189:0/2971995905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3d00cd3e0 msgr2=0x7fd3d00c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:16.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.977+0000 7fd3cffff700 1 --2- 172.21.15.189:0/2971995905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3d00cd3e0 0x7fd3d00c72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:16.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.977+0000 7fd3cffff700 1 -- 172.21.15.189:0/2971995905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c40094a0 con 0x7fd3d006b8c0 2024-04-06T12:12:16.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.977+0000 7fd3cffff700 1 --2- 172.21.15.189:0/2971995905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3d006b8c0 0x7fd3d0158bf0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd3c40090b0 tx=0x7fd3c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:16.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.977+0000 7fd3ccff9700 1 -- 172.21.15.189:0/2971995905 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c401a860 con 0x7fd3d006b8c0 2024-04-06T12:12:16.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.977+0000 7fd3ccff9700 1 -- 172.21.15.189:0/2971995905 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c401a9c0 con 0x7fd3d006b8c0 2024-04-06T12:12:16.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.977+0000 7fd3d6a25700 1 -- 172.21.15.189:0/2971995905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d00cad10 con 0x7fd3d006b8c0 2024-04-06T12:12:16.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.977+0000 7fd3ccff9700 1 -- 172.21.15.189:0/2971995905 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c401ad30 con 0x7fd3d006b8c0 2024-04-06T12:12:16.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.977+0000 7fd3d6a25700 1 -- 172.21.15.189:0/2971995905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d00cb2d0 con 0x7fd3d006b8c0 2024-04-06T12:12:16.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.978+0000 7fd3d6a25700 1 -- 172.21.15.189:0/2971995905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d0074990 con 0x7fd3d006b8c0 2024-04-06T12:12:16.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.978+0000 7fd3ccff9700 1 -- 172.21.15.189:0/2971995905 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd3c4022070 con 0x7fd3d006b8c0 2024-04-06T12:12:16.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.979+0000 7fd3ccff9700 1 --2- 172.21.15.189:0/2971995905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3bc06ac90 0x7fd3bc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:16.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.979+0000 7fd3ccff9700 1 -- 172.21.15.189:0/2971995905 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd3c408a680 con 0x7fd3d006b8c0 2024-04-06T12:12:16.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.979+0000 7fd3cf7fe700 1 --2- 172.21.15.189:0/2971995905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3bc06ac90 0x7fd3bc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:16.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.979+0000 7fd3cf7fe700 1 --2- 172.21.15.189:0/2971995905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3bc06ac90 0x7fd3bc06d150 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd3c0007fd0 tx=0x7fd3c0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:16.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:16.983+0000 7fd3ccff9700 1 -- 172.21.15.189:0/2971995905 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd3c4055630 con 0x7fd3d006b8c0 2024-04-06T12:12:17.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.177+0000 7fd3d6a25700 1 -- 172.21.15.189:0/2971995905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd3d00b5790 con 0x7fd3d006b8c0 2024-04-06T12:12:17.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.177+0000 7fd3ccff9700 1 -- 172.21.15.189:0/2971995905 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd3c4058c50 con 0x7fd3d006b8c0 2024-04-06T12:12:17.176 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:12:17.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.179+0000 7fd3d6a25700 1 -- 172.21.15.189:0/2971995905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3bc06ac90 msgr2=0x7fd3bc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:17.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.179+0000 7fd3d6a25700 1 --2- 172.21.15.189:0/2971995905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3bc06ac90 0x7fd3bc06d150 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd3c0007fd0 tx=0x7fd3c0009040 comp rx=0 tx=0).stop 2024-04-06T12:12:17.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.179+0000 7fd3d6a25700 1 -- 172.21.15.189:0/2971995905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3d006b8c0 msgr2=0x7fd3d0158bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:17.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.179+0000 7fd3d6a25700 1 --2- 172.21.15.189:0/2971995905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3d006b8c0 0x7fd3d0158bf0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd3c40090b0 tx=0x7fd3c4004750 comp rx=0 tx=0).stop 2024-04-06T12:12:17.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.180+0000 7fd3d6a25700 1 -- 172.21.15.189:0/2971995905 shutdown_connections 2024-04-06T12:12:17.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.180+0000 7fd3d6a25700 1 --2- 172.21.15.189:0/2971995905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3d015d7c0 0x7fd3d00b38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.180+0000 7fd3d6a25700 1 --2- 172.21.15.189:0/2971995905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3d006b8c0 0x7fd3d0158bf0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.180+0000 7fd3d6a25700 1 --2- 172.21.15.189:0/2971995905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3bc06ac90 0x7fd3bc06d150 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.180+0000 7fd3d6a25700 1 --2- 172.21.15.189:0/2971995905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3d00cd3e0 0x7fd3d00c72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.180+0000 7fd3d6a25700 1 -- 172.21.15.189:0/2971995905 >> 172.21.15.189:0/2971995905 conn(0x7fd3d00cc290 msgr2=0x7fd3d00adf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:17.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.180+0000 7fd3d6a25700 1 -- 172.21.15.189:0/2971995905 shutdown_connections 2024-04-06T12:12:17.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.180+0000 7fd3d6a25700 1 -- 172.21.15.189:0/2971995905 wait complete. 2024-04-06T12:12:17.191 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-06T12:12:17.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.316+0000 7fabbf675700 1 Processor -- start 2024-04-06T12:12:17.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.316+0000 7fabbf675700 1 -- start start 2024-04-06T12:12:17.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.317+0000 7fabbf675700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fabb806ae40 0x7fabb80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:17.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.317+0000 7fabbf675700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fabb815ddf0 0x7fabb815e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:17.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.317+0000 7fabbf675700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb80b3990 con 0x7fabb815d7b0 2024-04-06T12:12:17.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.317+0000 7fabbf675700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb80b3060 con 0x7fabb815ddf0 2024-04-06T12:12:17.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.317+0000 7fabbf675700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb80b3340 con 0x7fabb806ae40 2024-04-06T12:12:17.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.317+0000 7fabbdc12700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fabb815ddf0 0x7fabb815e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:17.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.317+0000 7fabbd411700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fabb815d7b0 0x7fabb806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55522/0 (socket says 172.21.15.189:55522) 2024-04-06T12:12:17.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.317+0000 7fabbd411700 1 -- 172.21.15.189:0/3838856593 learned_addr learned my addr 172.21.15.189:0/3838856593 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:17.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.317+0000 7fabbcc10700 1 --2- 172.21.15.189:0/3838856593 >> v2:172.21.15.192:3300/0 conn(0x7fabb806ae40 0x7fabb80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:17.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.318+0000 7fabbdc12700 1 -- 172.21.15.189:0/3838856593 >> v2:172.21.15.192:3300/0 conn(0x7fabb806ae40 msgr2=0x7fabb80b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:17.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.318+0000 7fabbdc12700 1 --2- 172.21.15.189:0/3838856593 >> v2:172.21.15.192:3300/0 conn(0x7fabb806ae40 0x7fabb80b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.318+0000 7fabbdc12700 1 -- 172.21.15.189:0/3838856593 >> v1:172.21.15.189:6789/0 conn(0x7fabb815d7b0 legacy=0x7fabb806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:17.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.318+0000 7fabbdc12700 1 -- 172.21.15.189:0/3838856593 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb80b3620 con 0x7fabb815ddf0 2024-04-06T12:12:17.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.318+0000 7fabaffff700 1 -- 172.21.15.189:0/3838856593 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756352151 0 0) 0x7fabb80b3990 con 0x7fabb815d7b0 2024-04-06T12:12:17.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.318+0000 7fabbdc12700 1 --2- 172.21.15.189:0/3838856593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb815ddf0 0x7fabb815e240 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fabb80afc00 tx=0x7fabb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbc2e0ae1f515dcb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:17.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.318+0000 7fabaffff700 1 -- 172.21.15.189:0/3838856593 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb000e070 con 0x7fabb815ddf0 2024-04-06T12:12:17.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.318+0000 7fabaffff700 1 -- 172.21.15.189:0/3838856593 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb000bb20 con 0x7fabb815ddf0 2024-04-06T12:12:17.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.318+0000 7fabaffff700 1 -- 172.21.15.189:0/3838856593 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb0012430 con 0x7fabb815ddf0 2024-04-06T12:12:17.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.318+0000 7fabbf675700 1 -- 172.21.15.189:0/3838856593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb815ddf0 msgr2=0x7fabb815e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:17.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.318+0000 7fabbf675700 1 --2- 172.21.15.189:0/3838856593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb815ddf0 0x7fabb815e240 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fabb80afc00 tx=0x7fabb000b330 comp rx=0 tx=0).stop 2024-04-06T12:12:17.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.318+0000 7fabbf675700 1 -- 172.21.15.189:0/3838856593 shutdown_connections 2024-04-06T12:12:17.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.318+0000 7fabbf675700 1 --2- 172.21.15.189:0/3838856593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb815ddf0 0x7fabb815e240 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.318+0000 7fabbf675700 1 --2- 172.21.15.189:0/3838856593 >> v2:172.21.15.192:3300/0 conn(0x7fabb806ae40 0x7fabb80b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.319+0000 7fabbf675700 1 -- 172.21.15.189:0/3838856593 >> 172.21.15.189:0/3838856593 conn(0x7fabb80d00b0 msgr2=0x7fabb80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:17.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.319+0000 7fabbf675700 1 -- 172.21.15.189:0/3838856593 shutdown_connections 2024-04-06T12:12:17.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.319+0000 7fabbf675700 1 -- 172.21.15.189:0/3838856593 wait complete. 2024-04-06T12:12:17.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.320+0000 7fabbf675700 1 Processor -- start 2024-04-06T12:12:17.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.320+0000 7fabbf675700 1 -- start start 2024-04-06T12:12:17.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.320+0000 7fabbf675700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb806ae40 0x7fabb8146b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:17.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.320+0000 7fabbf675700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fabb815d7b0 0x7fabb80cd410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:17.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.320+0000 7fabbf675700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabb815ddf0 0x7fabb80cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:17.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.320+0000 7fabbf675700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb815a480 con 0x7fabb806ae40 2024-04-06T12:12:17.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.320+0000 7fabbf675700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb815a760 con 0x7fabb815ddf0 2024-04-06T12:12:17.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.320+0000 7fabbf675700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb815aa40 con 0x7fabb815d7b0 2024-04-06T12:12:17.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.321+0000 7fabbd411700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb806ae40 0x7fabb8146b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:17.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.321+0000 7fabbd411700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb806ae40 0x7fabb8146b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56196/0 (socket says 172.21.15.189:56196) 2024-04-06T12:12:17.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.321+0000 7fabbd411700 1 -- 172.21.15.189:0/3275031284 learned_addr learned my addr 172.21.15.189:0/3275031284 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:17.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.321+0000 7fabbdc12700 1 --2- 172.21.15.189:0/3275031284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabb815ddf0 0x7fabb80cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:17.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.321+0000 7fabbcc10700 1 --2- 172.21.15.189:0/3275031284 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fabb815d7b0 0x7fabb80cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:17.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.321+0000 7fabbd411700 1 -- 172.21.15.189:0/3275031284 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fabb815d7b0 msgr2=0x7fabb80cd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:17.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.321+0000 7fabbd411700 1 --2- 172.21.15.189:0/3275031284 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fabb815d7b0 0x7fabb80cd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.321+0000 7fabbd411700 1 -- 172.21.15.189:0/3275031284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabb815ddf0 msgr2=0x7fabb80cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:17.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.321+0000 7fabbd411700 1 --2- 172.21.15.189:0/3275031284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabb815ddf0 0x7fabb80cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.321+0000 7fabbd411700 1 -- 172.21.15.189:0/3275031284 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb0009a70 con 0x7fabb806ae40 2024-04-06T12:12:17.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.321+0000 7fabbd411700 1 --2- 172.21.15.189:0/3275031284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb806ae40 0x7fabb8146b90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7faba80011e0 tx=0x7faba800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:17.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.321+0000 7fabadffb700 1 -- 172.21.15.189:0/3275031284 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba8004c80 con 0x7fabb806ae40 2024-04-06T12:12:17.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.321+0000 7fabadffb700 1 -- 172.21.15.189:0/3275031284 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba8004e00 con 0x7fabb806ae40 2024-04-06T12:12:17.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.321+0000 7fabadffb700 1 -- 172.21.15.189:0/3275031284 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba8007310 con 0x7fabb806ae40 2024-04-06T12:12:17.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.321+0000 7fabbf675700 1 -- 172.21.15.189:0/3275031284 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb815ad20 con 0x7fabb806ae40 2024-04-06T12:12:17.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.321+0000 7fabbf675700 1 -- 172.21.15.189:0/3275031284 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb81470d0 con 0x7fabb806ae40 2024-04-06T12:12:17.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.322+0000 7fabbf675700 1 -- 172.21.15.189:0/3275031284 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb815b240 con 0x7fabb806ae40 2024-04-06T12:12:17.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.323+0000 7fabadffb700 1 -- 172.21.15.189:0/3275031284 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7faba8007470 con 0x7fabb806ae40 2024-04-06T12:12:17.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.323+0000 7fabadffb700 1 --2- 172.21.15.189:0/3275031284 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fab9c06ac90 0x7fab9c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:17.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.323+0000 7fabadffb700 1 -- 172.21.15.189:0/3275031284 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7faba8089250 con 0x7fabb806ae40 2024-04-06T12:12:17.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.323+0000 7fabbcc10700 1 --2- 172.21.15.189:0/3275031284 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fab9c06ac90 0x7fab9c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:17.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.324+0000 7fabbcc10700 1 --2- 172.21.15.189:0/3275031284 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fab9c06ac90 0x7fab9c06d150 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fabb8157c40 tx=0x7fabb4009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:17.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.331+0000 7fabadffb700 1 -- 172.21.15.189:0/3275031284 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7faba8054140 con 0x7fabb806ae40 2024-04-06T12:12:17.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.524+0000 7fabbf675700 1 -- 172.21.15.189:0/3275031284 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fabb8067bb0 con 0x7fabb806ae40 2024-04-06T12:12:17.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.525+0000 7fabadffb700 1 -- 172.21.15.189:0/3275031284 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7faba8057760 con 0x7fabb806ae40 2024-04-06T12:12:17.524 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:12:17.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.527+0000 7fabbf675700 1 -- 172.21.15.189:0/3275031284 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fab9c06ac90 msgr2=0x7fab9c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:17.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.527+0000 7fabbf675700 1 --2- 172.21.15.189:0/3275031284 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fab9c06ac90 0x7fab9c06d150 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fabb8157c40 tx=0x7fabb4009210 comp rx=0 tx=0).stop 2024-04-06T12:12:17.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.527+0000 7fabbf675700 1 -- 172.21.15.189:0/3275031284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb806ae40 msgr2=0x7fabb8146b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:17.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.527+0000 7fabbf675700 1 --2- 172.21.15.189:0/3275031284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb806ae40 0x7fabb8146b90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7faba80011e0 tx=0x7faba800ca30 comp rx=0 tx=0).stop 2024-04-06T12:12:17.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.527+0000 7fabbf675700 1 -- 172.21.15.189:0/3275031284 shutdown_connections 2024-04-06T12:12:17.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.527+0000 7fabbf675700 1 --2- 172.21.15.189:0/3275031284 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fab9c06ac90 0x7fab9c06d150 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.527+0000 7fabbf675700 1 --2- 172.21.15.189:0/3275031284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabb815ddf0 0x7fabb80cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.527+0000 7fabbf675700 1 --2- 172.21.15.189:0/3275031284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb806ae40 0x7fabb8146b90 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.527+0000 7fabbf675700 1 --2- 172.21.15.189:0/3275031284 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fabb815d7b0 0x7fabb80cd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.527+0000 7fabbf675700 1 -- 172.21.15.189:0/3275031284 >> 172.21.15.189:0/3275031284 conn(0x7fabb80d00b0 msgr2=0x7fabb806de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:17.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.527+0000 7fabbf675700 1 -- 172.21.15.189:0/3275031284 shutdown_connections 2024-04-06T12:12:17.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.527+0000 7fabbf675700 1 -- 172.21.15.189:0/3275031284 wait complete. 2024-04-06T12:12:17.538 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T12:12:17.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.663+0000 7fa56f9c1700 1 Processor -- start 2024-04-06T12:12:17.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.663+0000 7fa56f9c1700 1 -- start start 2024-04-06T12:12:17.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.664+0000 7fa56f9c1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5680cd3e0 0x7fa5680cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:17.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.664+0000 7fa56f9c1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa56806b8c0 0x7fa56806dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:17.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.664+0000 7fa56f9c1700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56806ceb0 con 0x7fa56815d7c0 2024-04-06T12:12:17.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.664+0000 7fa56f9c1700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5680cdfa0 con 0x7fa56806b8c0 2024-04-06T12:12:17.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.664+0000 7fa56f9c1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56806d750 con 0x7fa5680cd3e0 2024-04-06T12:12:17.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.664+0000 7fa56df5e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa56806b8c0 0x7fa56806dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:17.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.664+0000 7fa56df5e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa56806b8c0 0x7fa56806dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56202/0 (socket says 172.21.15.189:56202) 2024-04-06T12:12:17.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.664+0000 7fa56df5e700 1 -- 172.21.15.189:0/2537469732 learned_addr learned my addr 172.21.15.189:0/2537469732 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:17.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.664+0000 7fa56d75d700 1 --2- 172.21.15.189:0/2537469732 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5680cd3e0 0x7fa5680cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:17.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.664+0000 7fa56df5e700 1 -- 172.21.15.189:0/2537469732 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5680cd3e0 msgr2=0x7fa5680cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:17.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.664+0000 7fa56df5e700 1 --2- 172.21.15.189:0/2537469732 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5680cd3e0 0x7fa5680cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.664+0000 7fa56df5e700 1 -- 172.21.15.189:0/2537469732 >> v1:172.21.15.192:6789/0 conn(0x7fa56815d7c0 legacy=0x7fa56806b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:17.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.664+0000 7fa56df5e700 1 -- 172.21.15.189:0/2537469732 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5680cdd00 con 0x7fa56806b8c0 2024-04-06T12:12:17.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.664+0000 7fa56df5e700 1 --2- 172.21.15.189:0/2537469732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56806b8c0 0x7fa56806dbe0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa560009e50 tx=0x7fa56000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f21089666f1e096 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:17.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.664+0000 7fa55ffff700 1 -- 172.21.15.189:0/2537469732 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56000e070 con 0x7fa56806b8c0 2024-04-06T12:12:17.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.664+0000 7fa55ffff700 1 -- 172.21.15.189:0/2537469732 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56000b910 con 0x7fa56806b8c0 2024-04-06T12:12:17.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.664+0000 7fa55ffff700 1 -- 172.21.15.189:0/2537469732 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56000bcc0 con 0x7fa56806b8c0 2024-04-06T12:12:17.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.665+0000 7fa56f9c1700 1 -- 172.21.15.189:0/2537469732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56806b8c0 msgr2=0x7fa56806dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:17.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.665+0000 7fa56f9c1700 1 --2- 172.21.15.189:0/2537469732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56806b8c0 0x7fa56806dbe0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa560009e50 tx=0x7fa56000b330 comp rx=0 tx=0).stop 2024-04-06T12:12:17.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.665+0000 7fa56f9c1700 1 -- 172.21.15.189:0/2537469732 shutdown_connections 2024-04-06T12:12:17.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.665+0000 7fa56f9c1700 1 --2- 172.21.15.189:0/2537469732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56806b8c0 0x7fa56806dbe0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.665+0000 7fa56f9c1700 1 --2- 172.21.15.189:0/2537469732 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5680cd3e0 0x7fa5680cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.665+0000 7fa56f9c1700 1 -- 172.21.15.189:0/2537469732 >> 172.21.15.189:0/2537469732 conn(0x7fa5680cc290 msgr2=0x7fa5680cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:17.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.665+0000 7fa56f9c1700 1 -- 172.21.15.189:0/2537469732 shutdown_connections 2024-04-06T12:12:17.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.665+0000 7fa56f9c1700 1 -- 172.21.15.189:0/2537469732 wait complete. 2024-04-06T12:12:17.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.666+0000 7fa56f9c1700 1 Processor -- start 2024-04-06T12:12:17.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.666+0000 7fa56f9c1700 1 -- start start 2024-04-06T12:12:17.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.667+0000 7fa56f9c1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa56806b8c0 0x7fa5680c59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:17.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.667+0000 7fa56f9c1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5680cd3e0 0x7fa5680b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:17.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.667+0000 7fa56f9c1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa56815d7c0 0x7fa56815f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:17.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.667+0000 7fa56f9c1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5680b22e0 con 0x7fa5680cd3e0 2024-04-06T12:12:17.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.667+0000 7fa56f9c1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5680b3140 con 0x7fa56806b8c0 2024-04-06T12:12:17.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.667+0000 7fa56f9c1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5680b2530 con 0x7fa56815d7c0 2024-04-06T12:12:17.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.667+0000 7fa56cf5c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5680cd3e0 0x7fa5680b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:17.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.667+0000 7fa56cf5c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5680cd3e0 0x7fa5680b3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56218/0 (socket says 172.21.15.189:56218) 2024-04-06T12:12:17.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.667+0000 7fa56cf5c700 1 -- 172.21.15.189:0/4161896290 learned_addr learned my addr 172.21.15.189:0/4161896290 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:17.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.667+0000 7fa56d75d700 1 --2- 172.21.15.189:0/4161896290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa56806b8c0 0x7fa5680c59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:17.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.667+0000 7fa56df5e700 1 --2- 172.21.15.189:0/4161896290 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa56815d7c0 0x7fa56815f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:17.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.667+0000 7fa56cf5c700 1 -- 172.21.15.189:0/4161896290 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa56815d7c0 msgr2=0x7fa56815f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:17.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.667+0000 7fa56cf5c700 1 --2- 172.21.15.189:0/4161896290 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa56815d7c0 0x7fa56815f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.667+0000 7fa56cf5c700 1 -- 172.21.15.189:0/4161896290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa56806b8c0 msgr2=0x7fa5680c59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:17.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.667+0000 7fa56cf5c700 1 --2- 172.21.15.189:0/4161896290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa56806b8c0 0x7fa5680c59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.667+0000 7fa56cf5c700 1 -- 172.21.15.189:0/4161896290 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa560009b30 con 0x7fa5680cd3e0 2024-04-06T12:12:17.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.667+0000 7fa56df5e700 1 --2- 172.21.15.189:0/4161896290 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa56815d7c0 0x7fa56815f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:17.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.668+0000 7fa56cf5c700 1 --2- 172.21.15.189:0/4161896290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5680cd3e0 0x7fa5680b3650 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa56400c420 tx=0x7fa56400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:17.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.668+0000 7fa55dffb700 1 -- 172.21.15.189:0/4161896290 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5640052d0 con 0x7fa5680cd3e0 2024-04-06T12:12:17.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.668+0000 7fa55dffb700 1 -- 172.21.15.189:0/4161896290 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa564005430 con 0x7fa5680cd3e0 2024-04-06T12:12:17.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.668+0000 7fa56f9c1700 1 -- 172.21.15.189:0/4161896290 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5680b2810 con 0x7fa5680cd3e0 2024-04-06T12:12:17.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.668+0000 7fa55dffb700 1 -- 172.21.15.189:0/4161896290 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5640038c0 con 0x7fa5680cd3e0 2024-04-06T12:12:17.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.668+0000 7fa56f9c1700 1 -- 172.21.15.189:0/4161896290 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5680b2dd0 con 0x7fa5680cd3e0 2024-04-06T12:12:17.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.669+0000 7fa56f9c1700 1 -- 172.21.15.189:0/4161896290 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5680b32b0 con 0x7fa5680cd3e0 2024-04-06T12:12:17.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.669+0000 7fa55dffb700 1 -- 172.21.15.189:0/4161896290 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa564003a20 con 0x7fa5680cd3e0 2024-04-06T12:12:17.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.669+0000 7fa55dffb700 1 --2- 172.21.15.189:0/4161896290 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa54c06acc0 0x7fa54c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:17.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.670+0000 7fa55dffb700 1 -- 172.21.15.189:0/4161896290 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa5640889f0 con 0x7fa5680cd3e0 2024-04-06T12:12:17.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.670+0000 7fa56d75d700 1 --2- 172.21.15.189:0/4161896290 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa54c06acc0 0x7fa54c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:17.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.670+0000 7fa56d75d700 1 --2- 172.21.15.189:0/4161896290 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa54c06acc0 0x7fa54c06d180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa558009b70 tx=0x7fa558008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:17.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.674+0000 7fa55dffb700 1 -- 172.21.15.189:0/4161896290 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa5640538f0 con 0x7fa5680cd3e0 2024-04-06T12:12:17.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.883+0000 7fa56f9c1700 1 -- 172.21.15.189:0/4161896290 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa568074990 con 0x7fa5680cd3e0 2024-04-06T12:12:17.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.883+0000 7fa55dffb700 1 -- 172.21.15.189:0/4161896290 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa564056f10 con 0x7fa5680cd3e0 2024-04-06T12:12:17.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.885+0000 7fa56f9c1700 1 -- 172.21.15.189:0/4161896290 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa54c06acc0 msgr2=0x7fa54c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:17.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.885+0000 7fa56f9c1700 1 --2- 172.21.15.189:0/4161896290 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa54c06acc0 0x7fa54c06d180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa558009b70 tx=0x7fa558008040 comp rx=0 tx=0).stop 2024-04-06T12:12:17.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.885+0000 7fa56f9c1700 1 -- 172.21.15.189:0/4161896290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5680cd3e0 msgr2=0x7fa5680b3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:17.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.885+0000 7fa56f9c1700 1 --2- 172.21.15.189:0/4161896290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5680cd3e0 0x7fa5680b3650 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa56400c420 tx=0x7fa56400c7e0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.885+0000 7fa56f9c1700 1 -- 172.21.15.189:0/4161896290 shutdown_connections 2024-04-06T12:12:17.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.885+0000 7fa56f9c1700 1 --2- 172.21.15.189:0/4161896290 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa56815d7c0 0x7fa56815f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.885+0000 7fa56f9c1700 1 --2- 172.21.15.189:0/4161896290 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa54c06acc0 0x7fa54c06d180 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.885+0000 7fa56f9c1700 1 --2- 172.21.15.189:0/4161896290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa56806b8c0 0x7fa5680c59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.885+0000 7fa56f9c1700 1 --2- 172.21.15.189:0/4161896290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5680cd3e0 0x7fa5680b3650 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:17.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.885+0000 7fa56f9c1700 1 -- 172.21.15.189:0/4161896290 >> 172.21.15.189:0/4161896290 conn(0x7fa5680cc290 msgr2=0x7fa5680af540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:17.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.885+0000 7fa56f9c1700 1 -- 172.21.15.189:0/4161896290 shutdown_connections 2024-04-06T12:12:17.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:17.885+0000 7fa56f9c1700 1 -- 172.21.15.189:0/4161896290 wait complete. 2024-04-06T12:12:17.885 INFO:teuthology.orchestra.run.smithi189.stderr:listed 0 entries 2024-04-06T12:12:17.896 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi189.front.sepia.ceph.com... 2024-04-06T12:12:17.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:17.896 DEBUG:teuthology.orchestra.run.smithi189:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-06T12:12:17.943 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi189.stderr:ceph-fuse[54790]: fuse finished with error 0 and tester_r 0 2024-04-06T12:12:17.944 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:12:24.046 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T12:12:24.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:24.046 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-06T12:12:24.128 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:24.265 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.266+0000 7f215ceb9700 1 Processor -- start 2024-04-06T12:12:24.265 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.266+0000 7f215ceb9700 1 -- start start 2024-04-06T12:12:24.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.267+0000 7f215ceb9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f215815d7b0 0x7f21580b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:24.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.267+0000 7f215ceb9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f215806a730 0x7f215806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:24.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.267+0000 7f215ceb9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21580b3990 con 0x7f215815ddf0 2024-04-06T12:12:24.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.267+0000 7f215ceb9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f215806b0f0 con 0x7f215815d7b0 2024-04-06T12:12:24.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.267+0000 7f215ceb9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21580b3060 con 0x7f215806a730 2024-04-06T12:12:24.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.267+0000 7f215659c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f215815d7b0 0x7f21580b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:24.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.267+0000 7f215659c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f215815d7b0 0x7f21580b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42658/0 (socket says 172.21.15.189:42658) 2024-04-06T12:12:24.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.267+0000 7f215659c700 1 -- 172.21.15.189:0/3593642072 learned_addr learned my addr 172.21.15.189:0/3593642072 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:24.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.267+0000 7f2155d9b700 1 --2- 172.21.15.189:0/3593642072 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f215806a730 0x7f215806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:24.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.268+0000 7f215659c700 1 -- 172.21.15.189:0/3593642072 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f215806a730 msgr2=0x7f215806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:24.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.268+0000 7f215659c700 1 --2- 172.21.15.189:0/3593642072 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f215806a730 0x7f215806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.268+0000 7f215659c700 1 -- 172.21.15.189:0/3593642072 >> v1:172.21.15.192:6789/0 conn(0x7f215815ddf0 legacy=0x7f215815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:24.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.268+0000 7f215659c700 1 -- 172.21.15.189:0/3593642072 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21580b3340 con 0x7f215815d7b0 2024-04-06T12:12:24.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.268+0000 7f2155d9b700 1 --2- 172.21.15.189:0/3593642072 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f215806a730 0x7f215806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:24.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.268+0000 7f215659c700 1 --2- 172.21.15.189:0/3593642072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f215815d7b0 0x7f21580b3e20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f21400098f0 tx=0x7f2140009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b58e90506f11b031 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:24.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.268+0000 7f215559a700 1 -- 172.21.15.189:0/3593642072 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214000b8a0 con 0x7f215815d7b0 2024-04-06T12:12:24.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.268+0000 7f215559a700 1 -- 172.21.15.189:0/3593642072 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f214000ba00 con 0x7f215815d7b0 2024-04-06T12:12:24.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.268+0000 7f215559a700 1 -- 172.21.15.189:0/3593642072 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214000bd90 con 0x7f215815d7b0 2024-04-06T12:12:24.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.268+0000 7f215ceb9700 1 -- 172.21.15.189:0/3593642072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f215815d7b0 msgr2=0x7f21580b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:24.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.268+0000 7f215ceb9700 1 --2- 172.21.15.189:0/3593642072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f215815d7b0 0x7f21580b3e20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f21400098f0 tx=0x7f2140009c00 comp rx=0 tx=0).stop 2024-04-06T12:12:24.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.268+0000 7f215ceb9700 1 -- 172.21.15.189:0/3593642072 shutdown_connections 2024-04-06T12:12:24.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.268+0000 7f215ceb9700 1 --2- 172.21.15.189:0/3593642072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f215815d7b0 0x7f21580b3e20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.268+0000 7f215ceb9700 1 --2- 172.21.15.189:0/3593642072 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f215806a730 0x7f215806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.268+0000 7f215ceb9700 1 -- 172.21.15.189:0/3593642072 >> 172.21.15.189:0/3593642072 conn(0x7f21580d00b0 msgr2=0x7f21580d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:24.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.269+0000 7f215ceb9700 1 -- 172.21.15.189:0/3593642072 shutdown_connections 2024-04-06T12:12:24.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.269+0000 7f215ceb9700 1 -- 172.21.15.189:0/3593642072 wait complete. 2024-04-06T12:12:24.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.270+0000 7f215ceb9700 1 Processor -- start 2024-04-06T12:12:24.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.270+0000 7f215ceb9700 1 -- start start 2024-04-06T12:12:24.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f215ceb9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f215806a730 0x7f2158151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:24.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f215ceb9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f215815d7b0 0x7f21580cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:24.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f215ceb9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f215815ddf0 0x7f215815ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:24.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f215ceb9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215815be00 con 0x7f215806a730 2024-04-06T12:12:24.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f215ceb9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215815b350 con 0x7f215815ddf0 2024-04-06T12:12:24.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f215ceb9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21580b0030 con 0x7f215815d7b0 2024-04-06T12:12:24.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f215659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f215806a730 0x7f2158151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:24.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f215659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f215806a730 0x7f2158151df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42672/0 (socket says 172.21.15.189:42672) 2024-04-06T12:12:24.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f215659c700 1 -- 172.21.15.189:0/3532107037 learned_addr learned my addr 172.21.15.189:0/3532107037 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:24.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f2155d9b700 1 --2- 172.21.15.189:0/3532107037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f215815d7b0 0x7f21580cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:24.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f2156d9d700 1 --2- 172.21.15.189:0/3532107037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f215815ddf0 0x7f215815ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:24.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f215659c700 1 -- 172.21.15.189:0/3532107037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f215815d7b0 msgr2=0x7f21580cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:24.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f215659c700 1 --2- 172.21.15.189:0/3532107037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f215815d7b0 0x7f21580cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f215659c700 1 -- 172.21.15.189:0/3532107037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f215815ddf0 msgr2=0x7f215815ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:24.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f215659c700 1 --2- 172.21.15.189:0/3532107037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f215815ddf0 0x7f215815ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f215659c700 1 -- 172.21.15.189:0/3532107037 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2140009550 con 0x7f215806a730 2024-04-06T12:12:24.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f215659c700 1 --2- 172.21.15.189:0/3532107037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f215806a730 0x7f2158151df0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2140000c00 tx=0x7f2140004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:24.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f214effd700 1 -- 172.21.15.189:0/3532107037 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2140004380 con 0x7f215806a730 2024-04-06T12:12:24.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f214effd700 1 -- 172.21.15.189:0/3532107037 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f214001f070 con 0x7f215806a730 2024-04-06T12:12:24.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.271+0000 7f215ceb9700 1 -- 172.21.15.189:0/3532107037 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f215815b560 con 0x7f215806a730 2024-04-06T12:12:24.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.272+0000 7f214effd700 1 -- 172.21.15.189:0/3532107037 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21400036a0 con 0x7f215806a730 2024-04-06T12:12:24.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.272+0000 7f215ceb9700 1 -- 172.21.15.189:0/3532107037 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f215815bb20 con 0x7f215806a730 2024-04-06T12:12:24.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.272+0000 7f215ceb9700 1 -- 172.21.15.189:0/3532107037 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2158067bb0 con 0x7f215806a730 2024-04-06T12:12:24.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.273+0000 7f214effd700 1 -- 172.21.15.189:0/3532107037 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2140021030 con 0x7f215806a730 2024-04-06T12:12:24.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.273+0000 7f214effd700 1 --2- 172.21.15.189:0/3532107037 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f213806ac90 0x7f213806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:24.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.273+0000 7f214effd700 1 -- 172.21.15.189:0/3532107037 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f214008a970 con 0x7f215806a730 2024-04-06T12:12:24.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.273+0000 7f2155d9b700 1 --2- 172.21.15.189:0/3532107037 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f213806ac90 0x7f213806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:24.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.274+0000 7f2155d9b700 1 --2- 172.21.15.189:0/3532107037 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f213806ac90 0x7f213806d150 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2158160c20 tx=0x7f2148008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:24.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.278+0000 7f214effd700 1 -- 172.21.15.189:0/3532107037 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2140055850 con 0x7f215806a730 2024-04-06T12:12:24.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.475+0000 7f215ceb9700 1 -- 172.21.15.189:0/3532107037 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21580cc320 con 0x7f215806a730 2024-04-06T12:12:24.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.477+0000 7f214effd700 1 -- 172.21.15.189:0/3532107037 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f2140058e70 con 0x7f215806a730 2024-04-06T12:12:24.476 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:24.476 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:08.649170+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:24.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.479+0000 7f215ceb9700 1 -- 172.21.15.189:0/3532107037 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f213806ac90 msgr2=0x7f213806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:24.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.479+0000 7f215ceb9700 1 --2- 172.21.15.189:0/3532107037 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f213806ac90 0x7f213806d150 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2158160c20 tx=0x7f2148008040 comp rx=0 tx=0).stop 2024-04-06T12:12:24.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.479+0000 7f215ceb9700 1 -- 172.21.15.189:0/3532107037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f215806a730 msgr2=0x7f2158151df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:24.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.479+0000 7f215ceb9700 1 --2- 172.21.15.189:0/3532107037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f215806a730 0x7f2158151df0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2140000c00 tx=0x7f2140004750 comp rx=0 tx=0).stop 2024-04-06T12:12:24.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.479+0000 7f215ceb9700 1 -- 172.21.15.189:0/3532107037 shutdown_connections 2024-04-06T12:12:24.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.479+0000 7f215ceb9700 1 --2- 172.21.15.189:0/3532107037 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f213806ac90 0x7f213806d150 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.479+0000 7f215ceb9700 1 --2- 172.21.15.189:0/3532107037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f215815ddf0 0x7f215815ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.479+0000 7f215ceb9700 1 --2- 172.21.15.189:0/3532107037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f215806a730 0x7f2158151df0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.479+0000 7f215ceb9700 1 --2- 172.21.15.189:0/3532107037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f215815d7b0 0x7f21580cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.479+0000 7f215ceb9700 1 -- 172.21.15.189:0/3532107037 >> 172.21.15.189:0/3532107037 conn(0x7f21580d00b0 msgr2=0x7f21580adec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:24.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.479+0000 7f215ceb9700 1 -- 172.21.15.189:0/3532107037 shutdown_connections 2024-04-06T12:12:24.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.479+0000 7f215ceb9700 1 -- 172.21.15.189:0/3532107037 wait complete. 2024-04-06T12:12:24.490 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:24.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.614+0000 7f4381bec700 1 Processor -- start 2024-04-06T12:12:24.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.614+0000 7f4381bec700 1 -- start start 2024-04-06T12:12:24.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.615+0000 7f4381bec700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f437c158fe0 0x7f437c06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:24.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.615+0000 7f4381bec700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437c15f9e0 con 0x7f437c06ac70 2024-04-06T12:12:24.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.615+0000 7f4381bec700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437c15f0d0 con 0x7f437c15c2d0 2024-04-06T12:12:24.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.615+0000 7f4381bec700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f437c06b0f0 con 0x7f437c158fe0 2024-04-06T12:12:24.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.615+0000 7f437b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f437c158fe0 0x7f437c06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:24.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.615+0000 7f437affd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f437c06ac70 0x7f437c15bbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55074/0 (socket says 172.21.15.189:55074) 2024-04-06T12:12:24.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.615+0000 7f437b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f437c158fe0 0x7f437c06a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42682/0 (socket says 172.21.15.189:42682) 2024-04-06T12:12:24.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.615+0000 7f437affd700 1 -- 172.21.15.189:0/3365066973 learned_addr learned my addr 172.21.15.189:0/3365066973 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:24.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.615+0000 7f437b7fe700 1 -- 172.21.15.189:0/3365066973 >> v1:172.21.15.192:6789/0 conn(0x7f437c15c2d0 legacy=0x7f437c0ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:24.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.615+0000 7f437b7fe700 1 -- 172.21.15.189:0/3365066973 >> v1:172.21.15.189:6789/0 conn(0x7f437c06ac70 legacy=0x7f437c15bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:24.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.615+0000 7f437b7fe700 1 -- 172.21.15.189:0/3365066973 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c15c8f0 con 0x7f437c158fe0 2024-04-06T12:12:24.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.615+0000 7f437a7fc700 1 -- 172.21.15.189:0/3365066973 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961869936 0 0) 0x7f437c15f9e0 con 0x7f437c06ac70 2024-04-06T12:12:24.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.615+0000 7f437b7fe700 1 --2- 172.21.15.189:0/3365066973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f437c158fe0 0x7f437c06a730 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4364009840 tx=0x7f4364009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d205a08e16cc7af7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:24.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.616+0000 7f437a7fc700 1 -- 172.21.15.189:0/3365066973 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4364011830 con 0x7f437c158fe0 2024-04-06T12:12:24.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.616+0000 7f437a7fc700 1 -- 172.21.15.189:0/3365066973 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4364011990 con 0x7f437c158fe0 2024-04-06T12:12:24.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.616+0000 7f437a7fc700 1 -- 172.21.15.189:0/3365066973 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4364011d20 con 0x7f437c158fe0 2024-04-06T12:12:24.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.616+0000 7f4381bec700 1 -- 172.21.15.189:0/3365066973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f437c158fe0 msgr2=0x7f437c06a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:24.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.616+0000 7f4381bec700 1 --2- 172.21.15.189:0/3365066973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f437c158fe0 0x7f437c06a730 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4364009840 tx=0x7f4364009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:24.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.616+0000 7f4381bec700 1 -- 172.21.15.189:0/3365066973 shutdown_connections 2024-04-06T12:12:24.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.616+0000 7f4381bec700 1 --2- 172.21.15.189:0/3365066973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f437c158fe0 0x7f437c06a730 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.616+0000 7f4381bec700 1 -- 172.21.15.189:0/3365066973 >> 172.21.15.189:0/3365066973 conn(0x7f437c0d00b0 msgr2=0x7f437c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:24.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.616+0000 7f4381bec700 1 -- 172.21.15.189:0/3365066973 shutdown_connections 2024-04-06T12:12:24.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.616+0000 7f4381bec700 1 -- 172.21.15.189:0/3365066973 wait complete. 2024-04-06T12:12:24.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.617+0000 7f4381bec700 1 Processor -- start 2024-04-06T12:12:24.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.618+0000 7f4381bec700 1 -- start start 2024-04-06T12:12:24.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.618+0000 7f4381bec700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f437c06ac70 0x7f437c1f0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:24.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.618+0000 7f4381bec700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f437c158fe0 0x7f437c15d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:24.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.618+0000 7f4381bec700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f437c15c2d0 0x7f437c15dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:24.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.618+0000 7f4381bec700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c13adc0 con 0x7f437c06ac70 2024-04-06T12:12:24.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.618+0000 7f4381bec700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c13b660 con 0x7f437c15c2d0 2024-04-06T12:12:24.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.618+0000 7f4381bec700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437c15e5a0 con 0x7f437c158fe0 2024-04-06T12:12:24.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.618+0000 7f437b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f437c06ac70 0x7f437c1f0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:24.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.618+0000 7f437b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f437c06ac70 0x7f437c1f0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42690/0 (socket says 172.21.15.189:42690) 2024-04-06T12:12:24.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.618+0000 7f437b7fe700 1 -- 172.21.15.189:0/112298205 learned_addr learned my addr 172.21.15.189:0/112298205 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:24.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.619+0000 7f437bfff700 1 --2- 172.21.15.189:0/112298205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f437c15c2d0 0x7f437c15dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:24.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.619+0000 7f437b7fe700 1 -- 172.21.15.189:0/112298205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f437c158fe0 msgr2=0x7f437c15d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:24.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.619+0000 7f437b7fe700 1 --2- 172.21.15.189:0/112298205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f437c158fe0 0x7f437c15d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.619+0000 7f437b7fe700 1 -- 172.21.15.189:0/112298205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f437c15c2d0 msgr2=0x7f437c15dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:24.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.619+0000 7f437b7fe700 1 --2- 172.21.15.189:0/112298205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f437c15c2d0 0x7f437c15dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.619+0000 7f437b7fe700 1 -- 172.21.15.189:0/112298205 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43640094a0 con 0x7f437c06ac70 2024-04-06T12:12:24.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.619+0000 7f437b7fe700 1 --2- 172.21.15.189:0/112298205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f437c06ac70 0x7f437c1f0420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f43640090b0 tx=0x7f4364004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:24.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.619+0000 7f4373fff700 1 -- 172.21.15.189:0/112298205 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436401a900 con 0x7f437c06ac70 2024-04-06T12:12:24.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.619+0000 7f4373fff700 1 -- 172.21.15.189:0/112298205 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f436401aa60 con 0x7f437c06ac70 2024-04-06T12:12:24.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.619+0000 7f4381bec700 1 -- 172.21.15.189:0/112298205 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f437c13b010 con 0x7f437c06ac70 2024-04-06T12:12:24.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.619+0000 7f4373fff700 1 -- 172.21.15.189:0/112298205 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436401add0 con 0x7f437c06ac70 2024-04-06T12:12:24.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.619+0000 7f4381bec700 1 -- 172.21.15.189:0/112298205 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f437c15e2b0 con 0x7f437c06ac70 2024-04-06T12:12:24.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.620+0000 7f4381bec700 1 -- 172.21.15.189:0/112298205 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f437c067bb0 con 0x7f437c06ac70 2024-04-06T12:12:24.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.620+0000 7f4373fff700 1 -- 172.21.15.189:0/112298205 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4364022070 con 0x7f437c06ac70 2024-04-06T12:12:24.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.621+0000 7f4373fff700 1 --2- 172.21.15.189:0/112298205 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f435c06ac90 0x7f435c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:24.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.621+0000 7f4373fff700 1 -- 172.21.15.189:0/112298205 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f436408a7b0 con 0x7f437c06ac70 2024-04-06T12:12:24.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.621+0000 7f437affd700 1 --2- 172.21.15.189:0/112298205 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f435c06ac90 0x7f435c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:24.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.621+0000 7f437affd700 1 --2- 172.21.15.189:0/112298205 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f435c06ac90 0x7f435c06d150 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f436c000c00 tx=0x7f436c007990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:24.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.625+0000 7f4373fff700 1 -- 172.21.15.189:0/112298205 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f43640557e0 con 0x7f437c06ac70 2024-04-06T12:12:24.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.821+0000 7f4381bec700 1 -- 172.21.15.189:0/112298205 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f437c074990 con 0x7f437c06ac70 2024-04-06T12:12:24.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.822+0000 7f4373fff700 1 -- 172.21.15.189:0/112298205 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f4364058e00 con 0x7f437c06ac70 2024-04-06T12:12:24.821 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:24.821 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:08.649170+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:24.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.824+0000 7f4381bec700 1 -- 172.21.15.189:0/112298205 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f435c06ac90 msgr2=0x7f435c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:24.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.824+0000 7f4381bec700 1 --2- 172.21.15.189:0/112298205 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f435c06ac90 0x7f435c06d150 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f436c000c00 tx=0x7f436c007990 comp rx=0 tx=0).stop 2024-04-06T12:12:24.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.824+0000 7f4381bec700 1 -- 172.21.15.189:0/112298205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f437c06ac70 msgr2=0x7f437c1f0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:24.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.824+0000 7f4381bec700 1 --2- 172.21.15.189:0/112298205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f437c06ac70 0x7f437c1f0420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f43640090b0 tx=0x7f4364004750 comp rx=0 tx=0).stop 2024-04-06T12:12:24.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.824+0000 7f4381bec700 1 -- 172.21.15.189:0/112298205 shutdown_connections 2024-04-06T12:12:24.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.824+0000 7f4381bec700 1 --2- 172.21.15.189:0/112298205 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f435c06ac90 0x7f435c06d150 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.824+0000 7f4381bec700 1 --2- 172.21.15.189:0/112298205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f437c15c2d0 0x7f437c15dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.824+0000 7f4381bec700 1 --2- 172.21.15.189:0/112298205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f437c06ac70 0x7f437c1f0420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.824+0000 7f4381bec700 1 --2- 172.21.15.189:0/112298205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f437c158fe0 0x7f437c15d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.824+0000 7f4381bec700 1 -- 172.21.15.189:0/112298205 >> 172.21.15.189:0/112298205 conn(0x7f437c0d00b0 msgr2=0x7f437c158250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:24.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.824+0000 7f4381bec700 1 -- 172.21.15.189:0/112298205 shutdown_connections 2024-04-06T12:12:24.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.824+0000 7f4381bec700 1 -- 172.21.15.189:0/112298205 wait complete. 2024-04-06T12:12:24.836 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-06T12:12:24.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.961+0000 7f9c2c5d8700 1 Processor -- start 2024-04-06T12:12:24.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.961+0000 7f9c2c5d8700 1 -- start start 2024-04-06T12:12:24.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.962+0000 7f9c2c5d8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c24072b00 0x7f9c240af040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:24.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.962+0000 7f9c2c5d8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c24071950 con 0x7f9c240b39d0 2024-04-06T12:12:24.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.962+0000 7f9c2c5d8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c240b4170 con 0x7f9c2415d7b0 2024-04-06T12:12:24.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.962+0000 7f9c2c5d8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c240b3e90 con 0x7f9c24072b00 2024-04-06T12:12:24.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.962+0000 7f9c29b73700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9c240b39d0 0x7f9c240723f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55078/0 (socket says 172.21.15.189:55078) 2024-04-06T12:12:24.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.962+0000 7f9c29b73700 1 -- 172.21.15.189:0/367131467 learned_addr learned my addr 172.21.15.189:0/367131467 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:24.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c2ab75700 1 --2- 172.21.15.189:0/367131467 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c24072b00 0x7f9c240af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:24.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c29372700 1 -- 172.21.15.189:0/367131467 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217251511 0 0) 0x7f9c24071950 con 0x7f9c240b39d0 2024-04-06T12:12:24.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c29372700 1 -- 172.21.15.189:0/367131467 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c100035b0 con 0x7f9c240b39d0 2024-04-06T12:12:24.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c29372700 1 -- 172.21.15.189:0/367131467 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310680674 0 0) 0x7f9c240b4170 con 0x7f9c2415d7b0 2024-04-06T12:12:24.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c29372700 1 -- 172.21.15.189:0/367131467 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c24071950 con 0x7f9c2415d7b0 2024-04-06T12:12:24.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c29372700 1 -- 172.21.15.189:0/367131467 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3768341282 0 0) 0x7f9c100035b0 con 0x7f9c240b39d0 2024-04-06T12:12:24.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c29372700 1 -- 172.21.15.189:0/367131467 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c24072b00 msgr2=0x7f9c240af040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:24.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c29372700 1 --2- 172.21.15.189:0/367131467 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c24072b00 0x7f9c240af040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c29372700 1 -- 172.21.15.189:0/367131467 >> v1:172.21.15.192:6789/0 conn(0x7f9c2415d7b0 legacy=0x7f9c240ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:24.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c29372700 1 -- 172.21.15.189:0/367131467 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c24071bf0 con 0x7f9c240b39d0 2024-04-06T12:12:24.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c29372700 1 -- 172.21.15.189:0/367131467 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f9c20001670 con 0x7f9c240b39d0 2024-04-06T12:12:24.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c2ab75700 1 --2- 172.21.15.189:0/367131467 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c24072b00 0x7f9c240af040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:24.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c29372700 1 -- 172.21.15.189:0/367131467 >> v1:172.21.15.189:6789/0 conn(0x7f9c240b39d0 legacy=0x7f9c240723f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:24.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c29372700 1 --2- 172.21.15.189:0/367131467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c10003bc0 0x7f9c10003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:24.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c29372700 1 --2- 172.21.15.189:0/367131467 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c24072b00 0x7f9c10004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:24.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c29372700 1 --2- 172.21.15.189:0/367131467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c10004d90 0x7f9c10005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:24.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c29372700 1 -- 172.21.15.189:0/367131467 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c240b3e90 con 0x7f9c10004d90 2024-04-06T12:12:24.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c29372700 1 -- 172.21.15.189:0/367131467 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1000bac0 con 0x7f9c10003bc0 2024-04-06T12:12:24.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c29372700 1 -- 172.21.15.189:0/367131467 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1000bc00 con 0x7f9c24072b00 2024-04-06T12:12:24.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c2ab75700 1 --2- 172.21.15.189:0/367131467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c10004d90 0x7f9c10005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:24.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c2a374700 1 --2- 172.21.15.189:0/367131467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c10003bc0 0x7f9c10003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:24.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c29b73700 1 --2- 172.21.15.189:0/367131467 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c24072b00 0x7f9c10004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:24.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c2ab75700 1 -- 172.21.15.189:0/367131467 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c24072b00 msgr2=0x7f9c10004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:24.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c2ab75700 1 --2- 172.21.15.189:0/367131467 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c24072b00 0x7f9c10004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c2ab75700 1 -- 172.21.15.189:0/367131467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c10003bc0 msgr2=0x7f9c10003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:24.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c2ab75700 1 --2- 172.21.15.189:0/367131467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c10003bc0 0x7f9c10003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.963+0000 7f9c2ab75700 1 -- 172.21.15.189:0/367131467 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1c009a70 con 0x7f9c10004d90 2024-04-06T12:12:24.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.964+0000 7f9c2ab75700 1 --2- 172.21.15.189:0/367131467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c10004d90 0x7f9c10005210 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c00e320 tx=0x7f9c10005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a56a1c9d9d5394e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:24.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.964+0000 7f9c29372700 1 -- 172.21.15.189:0/367131467 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c00ed60 con 0x7f9c10004d90 2024-04-06T12:12:24.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.964+0000 7f9c29372700 1 -- 172.21.15.189:0/367131467 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1c00f040 con 0x7f9c10004d90 2024-04-06T12:12:24.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.964+0000 7f9c29372700 1 -- 172.21.15.189:0/367131467 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c003c40 con 0x7f9c10004d90 2024-04-06T12:12:24.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.964+0000 7f9c2c5d8700 1 -- 172.21.15.189:0/367131467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c10004d90 msgr2=0x7f9c10005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:24.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.964+0000 7f9c2c5d8700 1 --2- 172.21.15.189:0/367131467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c10004d90 0x7f9c10005210 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c00e320 tx=0x7f9c10005880 comp rx=0 tx=0).stop 2024-04-06T12:12:24.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.964+0000 7f9c2ab75700 1 -- 172.21.15.189:0/367131467 reap_dead start 2024-04-06T12:12:24.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.964+0000 7f9c2c5d8700 1 -- 172.21.15.189:0/367131467 shutdown_connections 2024-04-06T12:12:24.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.964+0000 7f9c2c5d8700 1 -- 172.21.15.189:0/367131467 >> 172.21.15.189:0/367131467 conn(0x7f9c240d00b0 msgr2=0x7f9c240d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:24.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.964+0000 7f9c2c5d8700 1 -- 172.21.15.189:0/367131467 shutdown_connections 2024-04-06T12:12:24.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.964+0000 7f9c2c5d8700 1 -- 172.21.15.189:0/367131467 wait complete. 2024-04-06T12:12:24.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.966+0000 7f9c2c5d8700 1 Processor -- start 2024-04-06T12:12:24.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.966+0000 7f9c2c5d8700 1 -- start start 2024-04-06T12:12:24.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.966+0000 7f9c2c5d8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c10004d90 0x7f9c241f2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:24.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.966+0000 7f9c2c5d8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c2415b950 0x7f9c2415bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:24.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.966+0000 7f9c2c5d8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c2415ea00 0x7f9c2415ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:24.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.966+0000 7f9c2c5d8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2415c310 con 0x7f9c2415ea00 2024-04-06T12:12:24.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.966+0000 7f9c2c5d8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2415f3c0 con 0x7f9c10004d90 2024-04-06T12:12:24.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.966+0000 7f9c2c5d8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2414a5d0 con 0x7f9c2415b950 2024-04-06T12:12:24.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.966+0000 7f9c2ab75700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c2415ea00 0x7f9c2415ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:24.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.966+0000 7f9c2ab75700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c2415ea00 0x7f9c2415ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42708/0 (socket says 172.21.15.189:42708) 2024-04-06T12:12:24.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.966+0000 7f9c2ab75700 1 -- 172.21.15.189:0/1882995868 learned_addr learned my addr 172.21.15.189:0/1882995868 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:24.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.967+0000 7f9c29b73700 1 --2- 172.21.15.189:0/1882995868 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c2415b950 0x7f9c2415bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:24.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.967+0000 7f9c2a374700 1 --2- 172.21.15.189:0/1882995868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c10004d90 0x7f9c241f2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:24.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.967+0000 7f9c2ab75700 1 -- 172.21.15.189:0/1882995868 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c2415b950 msgr2=0x7f9c2415bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:24.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.967+0000 7f9c2ab75700 1 --2- 172.21.15.189:0/1882995868 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c2415b950 0x7f9c2415bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.967+0000 7f9c2ab75700 1 -- 172.21.15.189:0/1882995868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c10004d90 msgr2=0x7f9c241f2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:24.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.967+0000 7f9c2ab75700 1 --2- 172.21.15.189:0/1882995868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c10004d90 0x7f9c241f2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:24.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.967+0000 7f9c2ab75700 1 -- 172.21.15.189:0/1882995868 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c24155120 con 0x7f9c2415ea00 2024-04-06T12:12:24.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.967+0000 7f9c2a374700 1 --2- 172.21.15.189:0/1882995868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c10004d90 0x7f9c241f2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:24.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.967+0000 7f9c2ab75700 1 --2- 172.21.15.189:0/1882995868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c2415ea00 0x7f9c2415ee80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c004000 tx=0x7f9c1c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:24.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.967+0000 7f9c1affd700 1 -- 172.21.15.189:0/1882995868 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c004440 con 0x7f9c2415ea00 2024-04-06T12:12:24.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.967+0000 7f9c1affd700 1 -- 172.21.15.189:0/1882995868 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1c0045a0 con 0x7f9c2415ea00 2024-04-06T12:12:24.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.967+0000 7f9c2c5d8700 1 -- 172.21.15.189:0/1882995868 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c2415f5e0 con 0x7f9c2415ea00 2024-04-06T12:12:24.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.967+0000 7f9c1affd700 1 -- 172.21.15.189:0/1882995868 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c004950 con 0x7f9c2415ea00 2024-04-06T12:12:24.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.967+0000 7f9c2c5d8700 1 -- 172.21.15.189:0/1882995868 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c24154ad0 con 0x7f9c2415ea00 2024-04-06T12:12:24.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.968+0000 7f9c2c5d8700 1 -- 172.21.15.189:0/1882995868 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c24067bb0 con 0x7f9c2415ea00 2024-04-06T12:12:24.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.968+0000 7f9c1affd700 1 -- 172.21.15.189:0/1882995868 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9c1c024020 con 0x7f9c2415ea00 2024-04-06T12:12:24.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.969+0000 7f9c1affd700 1 --2- 172.21.15.189:0/1882995868 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c0806ac90 0x7f9c0806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:24.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.969+0000 7f9c1affd700 1 -- 172.21.15.189:0/1882995868 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9c1c08b370 con 0x7f9c2415ea00 2024-04-06T12:12:24.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.969+0000 7f9c2a374700 1 --2- 172.21.15.189:0/1882995868 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c0806ac90 0x7f9c0806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:24.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.969+0000 7f9c2a374700 1 --2- 172.21.15.189:0/1882995868 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c0806ac90 0x7f9c0806d150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9c14007fd0 tx=0x7f9c14009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:24.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:24.973+0000 7f9c1affd700 1 -- 172.21.15.189:0/1882995868 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9c1c0562f0 con 0x7f9c2415ea00 2024-04-06T12:12:25.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.208+0000 7f9c2c5d8700 1 -- 172.21.15.189:0/1882995868 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9c24074990 con 0x7f9c2415ea00 2024-04-06T12:12:25.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.535+0000 7f9c1affd700 1 -- 172.21.15.189:0/1882995868 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9c1c059910 con 0x7f9c2415ea00 2024-04-06T12:12:25.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.537+0000 7f9c2c5d8700 1 -- 172.21.15.189:0/1882995868 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c0806ac90 msgr2=0x7f9c0806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:25.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.537+0000 7f9c2c5d8700 1 --2- 172.21.15.189:0/1882995868 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c0806ac90 0x7f9c0806d150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9c14007fd0 tx=0x7f9c14009040 comp rx=0 tx=0).stop 2024-04-06T12:12:25.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.537+0000 7f9c2c5d8700 1 -- 172.21.15.189:0/1882995868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c2415ea00 msgr2=0x7f9c2415ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:25.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.537+0000 7f9c2c5d8700 1 --2- 172.21.15.189:0/1882995868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c2415ea00 0x7f9c2415ee80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c004000 tx=0x7f9c1c009750 comp rx=0 tx=0).stop 2024-04-06T12:12:25.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.538+0000 7f9c2c5d8700 1 -- 172.21.15.189:0/1882995868 shutdown_connections 2024-04-06T12:12:25.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.538+0000 7f9c2c5d8700 1 --2- 172.21.15.189:0/1882995868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c2415ea00 0x7f9c2415ee80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:25.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.538+0000 7f9c2c5d8700 1 --2- 172.21.15.189:0/1882995868 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c0806ac90 0x7f9c0806d150 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:25.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.538+0000 7f9c2c5d8700 1 --2- 172.21.15.189:0/1882995868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c10004d90 0x7f9c241f2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:25.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.538+0000 7f9c2c5d8700 1 --2- 172.21.15.189:0/1882995868 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c2415b950 0x7f9c2415bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:25.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.538+0000 7f9c2c5d8700 1 -- 172.21.15.189:0/1882995868 >> 172.21.15.189:0/1882995868 conn(0x7f9c240d00b0 msgr2=0x7f9c240ad2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:25.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.538+0000 7f9c2c5d8700 1 -- 172.21.15.189:0/1882995868 shutdown_connections 2024-04-06T12:12:25.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.538+0000 7f9c2c5d8700 1 -- 172.21.15.189:0/1882995868 wait complete. 2024-04-06T12:12:25.550 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-04-06T12:12:25.550 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-06T12:12:25.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.679+0000 7ff79a7ff700 1 Processor -- start 2024-04-06T12:12:25.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.679+0000 7ff79a7ff700 1 -- start start 2024-04-06T12:12:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.680+0000 7ff79a7ff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff79415d7b0 0x7ff7940b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.680+0000 7ff79a7ff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7940b3990 con 0x7ff79406a730 2024-04-06T12:12:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.680+0000 7ff79a7ff700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7940b3060 con 0x7ff79415e500 2024-04-06T12:12:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.680+0000 7ff79a7ff700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff79415e950 con 0x7ff79415d7b0 2024-04-06T12:12:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.680+0000 7ff78bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff79406a730 0x7ff79415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55092/0 (socket says 172.21.15.189:55092) 2024-04-06T12:12:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.680+0000 7ff78bfff700 1 -- 172.21.15.189:0/144241399 learned_addr learned my addr 172.21.15.189:0/144241399 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.680+0000 7ff793fff700 1 --2- 172.21.15.189:0/144241399 >> v2:172.21.15.192:3300/0 conn(0x7ff79415d7b0 0x7ff7940b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.680+0000 7ff7937fe700 1 -- 172.21.15.189:0/144241399 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105220200 0 0) 0x7ff7940b3990 con 0x7ff79406a730 2024-04-06T12:12:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.680+0000 7ff7937fe700 1 -- 172.21.15.189:0/144241399 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7800035b0 con 0x7ff79406a730 2024-04-06T12:12:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.680+0000 7ff793fff700 1 -- 172.21.15.189:0/144241399 >> v1:172.21.15.192:6789/0 conn(0x7ff79415e500 legacy=0x7ff7940717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.680+0000 7ff793fff700 1 -- 172.21.15.189:0/144241399 >> v1:172.21.15.189:6789/0 conn(0x7ff79406a730 legacy=0x7ff79415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.680+0000 7ff793fff700 1 -- 172.21.15.189:0/144241399 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7940b3340 con 0x7ff79415d7b0 2024-04-06T12:12:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.681+0000 7ff793fff700 1 --2- 172.21.15.189:0/144241399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff79415d7b0 0x7ff7940b3e20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff784009840 tx=0x7ff784009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71d60041e9ff9617 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.681+0000 7ff7937fe700 1 -- 172.21.15.189:0/144241399 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff784011830 con 0x7ff79415d7b0 2024-04-06T12:12:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.681+0000 7ff7937fe700 1 -- 172.21.15.189:0/144241399 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff784011990 con 0x7ff79415d7b0 2024-04-06T12:12:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.681+0000 7ff7937fe700 1 -- 172.21.15.189:0/144241399 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff784011d20 con 0x7ff79415d7b0 2024-04-06T12:12:25.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.681+0000 7ff79a7ff700 1 -- 172.21.15.189:0/144241399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff79415d7b0 msgr2=0x7ff7940b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:25.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.681+0000 7ff79a7ff700 1 --2- 172.21.15.189:0/144241399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff79415d7b0 0x7ff7940b3e20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff784009840 tx=0x7ff784009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:25.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.681+0000 7ff79a7ff700 1 -- 172.21.15.189:0/144241399 shutdown_connections 2024-04-06T12:12:25.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.681+0000 7ff79a7ff700 1 --2- 172.21.15.189:0/144241399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff79415d7b0 0x7ff7940b3e20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:25.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.681+0000 7ff79a7ff700 1 -- 172.21.15.189:0/144241399 >> 172.21.15.189:0/144241399 conn(0x7ff7940d00b0 msgr2=0x7ff7940d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:25.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.681+0000 7ff79a7ff700 1 -- 172.21.15.189:0/144241399 shutdown_connections 2024-04-06T12:12:25.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.681+0000 7ff79a7ff700 1 -- 172.21.15.189:0/144241399 wait complete. 2024-04-06T12:12:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.682+0000 7ff79a7ff700 1 Processor -- start 2024-04-06T12:12:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.683+0000 7ff79a7ff700 1 -- start start 2024-04-06T12:12:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.683+0000 7ff79a7ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff79406a730 0x7ff79415ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.683+0000 7ff79a7ff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff79415d7b0 0x7ff7941f2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.683+0000 7ff79a7ff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff79415e500 0x7ff794158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.683+0000 7ff79a7ff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff794152910 con 0x7ff79415e500 2024-04-06T12:12:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.683+0000 7ff79a7ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79414a290 con 0x7ff79406a730 2024-04-06T12:12:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.683+0000 7ff79a7ff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff79415b3c0 con 0x7ff79415d7b0 2024-04-06T12:12:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.683+0000 7ff798d9c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff79415e500 0x7ff794158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.683+0000 7ff798d9c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff79415e500 0x7ff794158a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42712/0 (socket says 172.21.15.189:42712) 2024-04-06T12:12:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.683+0000 7ff798d9c700 1 -- 172.21.15.189:0/2168211220 learned_addr learned my addr 172.21.15.189:0/2168211220 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.684+0000 7ff793fff700 1 --2- 172.21.15.189:0/2168211220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff79406a730 0x7ff79415ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.684+0000 7ff78bfff700 1 --2- 172.21.15.189:0/2168211220 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff79415d7b0 0x7ff7941f2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.684+0000 7ff798d9c700 1 -- 172.21.15.189:0/2168211220 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff79415d7b0 msgr2=0x7ff7941f2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.684+0000 7ff798d9c700 1 --2- 172.21.15.189:0/2168211220 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff79415d7b0 0x7ff7941f2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.684+0000 7ff798d9c700 1 -- 172.21.15.189:0/2168211220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff79406a730 msgr2=0x7ff79415ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.684+0000 7ff798d9c700 1 --2- 172.21.15.189:0/2168211220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff79406a730 0x7ff79415ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.684+0000 7ff798d9c700 1 -- 172.21.15.189:0/2168211220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7840094a0 con 0x7ff79415e500 2024-04-06T12:12:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.684+0000 7ff78bfff700 1 --2- 172.21.15.189:0/2168211220 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff79415d7b0 0x7ff7941f2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.684+0000 7ff798d9c700 1 --2- 172.21.15.189:0/2168211220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff79415e500 0x7ff794158a00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff78c00cba0 tx=0x7ff78c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.684+0000 7ff7917fa700 1 -- 172.21.15.189:0/2168211220 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c003c00 con 0x7ff79415e500 2024-04-06T12:12:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.684+0000 7ff7917fa700 1 -- 172.21.15.189:0/2168211220 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78c003d60 con 0x7ff79415e500 2024-04-06T12:12:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.684+0000 7ff79a7ff700 1 -- 172.21.15.189:0/2168211220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff794151c90 con 0x7ff79415e500 2024-04-06T12:12:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.684+0000 7ff7917fa700 1 -- 172.21.15.189:0/2168211220 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c00aba0 con 0x7ff79415e500 2024-04-06T12:12:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.684+0000 7ff79a7ff700 1 -- 172.21.15.189:0/2168211220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff794152250 con 0x7ff79415e500 2024-04-06T12:12:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.685+0000 7ff79a7ff700 1 -- 172.21.15.189:0/2168211220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff794067bb0 con 0x7ff79415e500 2024-04-06T12:12:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.685+0000 7ff7917fa700 1 -- 172.21.15.189:0/2168211220 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff78c00ad00 con 0x7ff79415e500 2024-04-06T12:12:25.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.686+0000 7ff7917fa700 1 --2- 172.21.15.189:0/2168211220 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff77406ac90 0x7ff77406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:25.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.686+0000 7ff7917fa700 1 -- 172.21.15.189:0/2168211220 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff78c089680 con 0x7ff79415e500 2024-04-06T12:12:25.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.686+0000 7ff793fff700 1 --2- 172.21.15.189:0/2168211220 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff77406ac90 0x7ff77406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:25.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.686+0000 7ff793fff700 1 --2- 172.21.15.189:0/2168211220 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff77406ac90 0x7ff77406d150 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff784000c00 tx=0x7ff784008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:25.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.690+0000 7ff7917fa700 1 -- 172.21.15.189:0/2168211220 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff78c054570 con 0x7ff79415e500 2024-04-06T12:12:25.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:25.928+0000 7ff79a7ff700 1 -- 172.21.15.189:0/2168211220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff794074990 con 0x7ff79415e500 2024-04-06T12:12:26.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.541+0000 7ff7917fa700 1 -- 172.21.15.189:0/2168211220 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff78c057b90 con 0x7ff79415e500 2024-04-06T12:12:26.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.544+0000 7ff79a7ff700 1 -- 172.21.15.189:0/2168211220 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff77406ac90 msgr2=0x7ff77406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:26.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.544+0000 7ff79a7ff700 1 --2- 172.21.15.189:0/2168211220 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff77406ac90 0x7ff77406d150 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff784000c00 tx=0x7ff784008030 comp rx=0 tx=0).stop 2024-04-06T12:12:26.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.544+0000 7ff79a7ff700 1 -- 172.21.15.189:0/2168211220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff79415e500 msgr2=0x7ff794158a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:26.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.544+0000 7ff79a7ff700 1 --2- 172.21.15.189:0/2168211220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff79415e500 0x7ff794158a00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff78c00cba0 tx=0x7ff78c00cf60 comp rx=0 tx=0).stop 2024-04-06T12:12:26.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.544+0000 7ff79a7ff700 1 -- 172.21.15.189:0/2168211220 shutdown_connections 2024-04-06T12:12:26.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.544+0000 7ff79a7ff700 1 --2- 172.21.15.189:0/2168211220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff79415e500 0x7ff794158a00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:26.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.544+0000 7ff79a7ff700 1 --2- 172.21.15.189:0/2168211220 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff77406ac90 0x7ff77406d150 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:26.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.544+0000 7ff79a7ff700 1 --2- 172.21.15.189:0/2168211220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff79406a730 0x7ff79415ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:26.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.544+0000 7ff79a7ff700 1 --2- 172.21.15.189:0/2168211220 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff79415d7b0 0x7ff7941f2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:26.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.544+0000 7ff79a7ff700 1 -- 172.21.15.189:0/2168211220 >> 172.21.15.189:0/2168211220 conn(0x7ff7940d00b0 msgr2=0x7ff7940af4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:26.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.544+0000 7ff79a7ff700 1 -- 172.21.15.189:0/2168211220 shutdown_connections 2024-04-06T12:12:26.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.544+0000 7ff79a7ff700 1 -- 172.21.15.189:0/2168211220 wait complete. 2024-04-06T12:12:26.559 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-06T12:12:26.559 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:12:26.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.688+0000 7efdb593d700 1 Processor -- start 2024-04-06T12:12:26.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.688+0000 7efdb593d700 1 -- start start 2024-04-06T12:12:26.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.689+0000 7efdb593d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efdb015d590 0x7efdb00b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:26.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.689+0000 7efdb593d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efdb00b4ed0 0x7efdb00b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:26.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.689+0000 7efdb593d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb00b5cc0 con 0x7efdb00b6580 2024-04-06T12:12:26.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.689+0000 7efdb593d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb00b58c0 con 0x7efdb015d590 2024-04-06T12:12:26.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.689+0000 7efdb593d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb00d1500 con 0x7efdb00b4ed0 2024-04-06T12:12:26.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.689+0000 7efdaeffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efdb015d590 0x7efdb00b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:26.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.689+0000 7efdaeffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efdb015d590 0x7efdb00b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42720/0 (socket says 172.21.15.189:42720) 2024-04-06T12:12:26.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.689+0000 7efdaeffd700 1 -- 172.21.15.189:0/2130840924 learned_addr learned my addr 172.21.15.189:0/2130840924 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:26.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.689+0000 7efdaeffd700 1 -- 172.21.15.189:0/2130840924 >> v2:172.21.15.192:3300/0 conn(0x7efdb00b4ed0 msgr2=0x7efdb00b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:26.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.689+0000 7efdaeffd700 1 --2- 172.21.15.189:0/2130840924 >> v2:172.21.15.192:3300/0 conn(0x7efdb00b4ed0 0x7efdb00b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:26.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.689+0000 7efdaeffd700 1 -- 172.21.15.189:0/2130840924 >> v1:172.21.15.192:6789/0 conn(0x7efdb00b6580 legacy=0x7efdb00b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:26.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.689+0000 7efdaeffd700 1 -- 172.21.15.189:0/2130840924 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb00d17e0 con 0x7efdb015d590 2024-04-06T12:12:26.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.689+0000 7efdaeffd700 1 --2- 172.21.15.189:0/2130840924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdb015d590 0x7efdb00b8d70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efdb00b6330 tx=0x7efda4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1982379f821b0e6c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:26.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.690+0000 7efdadffb700 1 -- 172.21.15.189:0/2130840924 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda4004230 con 0x7efdb015d590 2024-04-06T12:12:26.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.690+0000 7efdadffb700 1 -- 172.21.15.189:0/2130840924 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda4004390 con 0x7efdb015d590 2024-04-06T12:12:26.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.690+0000 7efdadffb700 1 -- 172.21.15.189:0/2130840924 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda40036a0 con 0x7efdb015d590 2024-04-06T12:12:26.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.690+0000 7efdb593d700 1 -- 172.21.15.189:0/2130840924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdb015d590 msgr2=0x7efdb00b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:26.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.690+0000 7efdb593d700 1 --2- 172.21.15.189:0/2130840924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdb015d590 0x7efdb00b8d70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efdb00b6330 tx=0x7efda4009bd0 comp rx=0 tx=0).stop 2024-04-06T12:12:26.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.690+0000 7efdb593d700 1 -- 172.21.15.189:0/2130840924 shutdown_connections 2024-04-06T12:12:26.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.690+0000 7efdb593d700 1 --2- 172.21.15.189:0/2130840924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdb015d590 0x7efdb00b8d70 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:26.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.690+0000 7efdb593d700 1 --2- 172.21.15.189:0/2130840924 >> v2:172.21.15.192:3300/0 conn(0x7efdb00b4ed0 0x7efdb00b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:26.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.690+0000 7efdb593d700 1 -- 172.21.15.189:0/2130840924 >> 172.21.15.189:0/2130840924 conn(0x7efdb006bbc0 msgr2=0x7efdb006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:26.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.690+0000 7efdb593d700 1 -- 172.21.15.189:0/2130840924 shutdown_connections 2024-04-06T12:12:26.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.690+0000 7efdb593d700 1 -- 172.21.15.189:0/2130840924 wait complete. 2024-04-06T12:12:26.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.691+0000 7efdb593d700 1 Processor -- start 2024-04-06T12:12:26.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.692+0000 7efdb593d700 1 -- start start 2024-04-06T12:12:26.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.692+0000 7efdb593d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdb00b4ed0 0x7efdb01f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:26.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.692+0000 7efdb593d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdb00b6580 0x7efdb015b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:26.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.692+0000 7efdb593d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efdb015d590 0x7efdb015b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:26.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.692+0000 7efdb593d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb00cd390 con 0x7efdb00b6580 2024-04-06T12:12:26.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.692+0000 7efdb593d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb00cd0b0 con 0x7efdb015d590 2024-04-06T12:12:26.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.692+0000 7efdb593d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb01f3d40 con 0x7efdb00b4ed0 2024-04-06T12:12:26.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.692+0000 7efdae7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdb00b6580 0x7efdb015b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:26.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.692+0000 7efdae7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdb00b6580 0x7efdb015b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42726/0 (socket says 172.21.15.189:42726) 2024-04-06T12:12:26.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.692+0000 7efdae7fc700 1 -- 172.21.15.189:0/1414260458 learned_addr learned my addr 172.21.15.189:0/1414260458 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:26.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.693+0000 7efdaf7fe700 1 --2- 172.21.15.189:0/1414260458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efdb015d590 0x7efdb015b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:26.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.693+0000 7efdaeffd700 1 --2- 172.21.15.189:0/1414260458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdb00b4ed0 0x7efdb01f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:26.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.693+0000 7efdae7fc700 1 -- 172.21.15.189:0/1414260458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdb00b4ed0 msgr2=0x7efdb01f26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:26.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.693+0000 7efdae7fc700 1 --2- 172.21.15.189:0/1414260458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdb00b4ed0 0x7efdb01f26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:26.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.693+0000 7efdae7fc700 1 -- 172.21.15.189:0/1414260458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efdb015d590 msgr2=0x7efdb015b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:26.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.693+0000 7efdae7fc700 1 --2- 172.21.15.189:0/1414260458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efdb015d590 0x7efdb015b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:26.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.693+0000 7efdae7fc700 1 -- 172.21.15.189:0/1414260458 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda4009550 con 0x7efdb00b6580 2024-04-06T12:12:26.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.693+0000 7efdae7fc700 1 --2- 172.21.15.189:0/1414260458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdb00b6580 0x7efdb015b480 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efda000b530 tx=0x7efda000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:26.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.693+0000 7efd97fff700 1 -- 172.21.15.189:0/1414260458 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda0011830 con 0x7efdb00b6580 2024-04-06T12:12:26.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.693+0000 7efd97fff700 1 -- 172.21.15.189:0/1414260458 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda00119b0 con 0x7efdb00b6580 2024-04-06T12:12:26.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.693+0000 7efdb593d700 1 -- 172.21.15.189:0/1414260458 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb00ccd40 con 0x7efdb00b6580 2024-04-06T12:12:26.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.693+0000 7efd97fff700 1 -- 172.21.15.189:0/1414260458 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda0011d60 con 0x7efdb00b6580 2024-04-06T12:12:26.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.693+0000 7efdb593d700 1 -- 172.21.15.189:0/1414260458 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb00b4af0 con 0x7efdb00b6580 2024-04-06T12:12:26.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.694+0000 7efdb593d700 1 -- 172.21.15.189:0/1414260458 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb00b4d10 con 0x7efdb00b6580 2024-04-06T12:12:26.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.694+0000 7efd97fff700 1 -- 172.21.15.189:0/1414260458 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efda001a400 con 0x7efdb00b6580 2024-04-06T12:12:26.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.695+0000 7efd97fff700 1 --2- 172.21.15.189:0/1414260458 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efd9c06ac90 0x7efd9c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:26.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.695+0000 7efd97fff700 1 -- 172.21.15.189:0/1414260458 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7efda00896b0 con 0x7efdb00b6580 2024-04-06T12:12:26.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.695+0000 7efdaeffd700 1 --2- 172.21.15.189:0/1414260458 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efd9c06ac90 0x7efd9c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:26.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.695+0000 7efdaeffd700 1 --2- 172.21.15.189:0/1414260458 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efd9c06ac90 0x7efd9c06d150 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efda4000c00 tx=0x7efda4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:26.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.699+0000 7efd97fff700 1 -- 172.21.15.189:0/1414260458 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efda00545a0 con 0x7efdb00b6580 2024-04-06T12:12:26.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.938+0000 7efdb593d700 1 -- 172.21.15.189:0/1414260458 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdb0074990 con 0x7efdb00b6580 2024-04-06T12:12:26.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.938+0000 7efd97fff700 1 -- 172.21.15.189:0/1414260458 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4739 (secure 0 0 0) 0x7efda0057bc0 con 0x7efdb00b6580 2024-04-06T12:12:26.937 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:26.938 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2769778155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2769778155},{"type":"v1","addr":"172.21.15.189:6835","nonce":2769778155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4233,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3550531296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3550531296},{"type":"v1","addr":"172.21.15.189:6837","nonce":3550531296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3198235148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3198235148},{"type":"v1","addr":"172.21.15.192:6833","nonce":3198235148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:12:04.195581+0000","modified":"2024-04-06T12:12:26.535185+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/516212261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":516212261},{"type":"v1","addr":"172.21.15.192:6835","nonce":516212261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-06T12:12:26.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.940+0000 7efdb593d700 1 -- 172.21.15.189:0/1414260458 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efd9c06ac90 msgr2=0x7efd9c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:26.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.940+0000 7efdb593d700 1 --2- 172.21.15.189:0/1414260458 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efd9c06ac90 0x7efd9c06d150 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efda4000c00 tx=0x7efda4008030 comp rx=0 tx=0).stop 2024-04-06T12:12:26.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.940+0000 7efdb593d700 1 -- 172.21.15.189:0/1414260458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdb00b6580 msgr2=0x7efdb015b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:26.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.940+0000 7efdb593d700 1 --2- 172.21.15.189:0/1414260458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdb00b6580 0x7efdb015b480 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efda000b530 tx=0x7efda000b8f0 comp rx=0 tx=0).stop 2024-04-06T12:12:26.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.940+0000 7efdb593d700 1 -- 172.21.15.189:0/1414260458 shutdown_connections 2024-04-06T12:12:26.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.940+0000 7efdb593d700 1 --2- 172.21.15.189:0/1414260458 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efd9c06ac90 0x7efd9c06d150 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:26.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.940+0000 7efdb593d700 1 --2- 172.21.15.189:0/1414260458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efdb015d590 0x7efdb015b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:26.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.940+0000 7efdb593d700 1 --2- 172.21.15.189:0/1414260458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdb00b4ed0 0x7efdb01f26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:26.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.940+0000 7efdb593d700 1 --2- 172.21.15.189:0/1414260458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdb00b6580 0x7efdb015b480 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:26.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.940+0000 7efdb593d700 1 -- 172.21.15.189:0/1414260458 >> 172.21.15.189:0/1414260458 conn(0x7efdb006bbc0 msgr2=0x7efdb00d2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:26.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.941+0000 7efdb593d700 1 -- 172.21.15.189:0/1414260458 shutdown_connections 2024-04-06T12:12:26.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:26.941+0000 7efdb593d700 1 -- 172.21.15.189:0/1414260458 wait complete. 2024-04-06T12:12:26.941 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 9 2024-04-06T12:12:26.952 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:27.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.075+0000 7f3fdddc6700 1 Processor -- start 2024-04-06T12:12:27.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.075+0000 7f3fdddc6700 1 -- start start 2024-04-06T12:12:27.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.076+0000 7f3fdddc6700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3fd815d7b0 0x7f3fd80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:27.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.076+0000 7f3fdddc6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3fd815e500 0x7f3fd806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:27.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.076+0000 7f3fdddc6700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd80b3990 con 0x7f3fd806a730 2024-04-06T12:12:27.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.076+0000 7f3fdddc6700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fd815e950 con 0x7f3fd815e500 2024-04-06T12:12:27.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.076+0000 7f3fdddc6700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fd806b0f0 con 0x7f3fd815d7b0 2024-04-06T12:12:27.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.076+0000 7f3fd7fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3fd815e500 0x7f3fd806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:27.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.076+0000 7f3fd7fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3fd815e500 0x7f3fd806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42732/0 (socket says 172.21.15.189:42732) 2024-04-06T12:12:27.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.076+0000 7f3fd7fff700 1 -- 172.21.15.189:0/1124625276 learned_addr learned my addr 172.21.15.189:0/1124625276 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:27.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.076+0000 7f3fd77fe700 1 --2- 172.21.15.189:0/1124625276 >> v2:172.21.15.192:3300/0 conn(0x7f3fd815d7b0 0x7f3fd80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:27.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.076+0000 7f3fd7fff700 1 -- 172.21.15.189:0/1124625276 >> v2:172.21.15.192:3300/0 conn(0x7f3fd815d7b0 msgr2=0x7f3fd80b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:27.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.076+0000 7f3fd7fff700 1 --2- 172.21.15.189:0/1124625276 >> v2:172.21.15.192:3300/0 conn(0x7f3fd815d7b0 0x7f3fd80b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.076+0000 7f3fd7fff700 1 -- 172.21.15.189:0/1124625276 >> v1:172.21.15.192:6789/0 conn(0x7f3fd806a730 legacy=0x7f3fd815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:27.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.076+0000 7f3fd7fff700 1 -- 172.21.15.189:0/1124625276 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd80b3060 con 0x7f3fd815e500 2024-04-06T12:12:27.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.076+0000 7f3fd7fff700 1 --2- 172.21.15.189:0/1124625276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fd815e500 0x7f3fd806abb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3fd80b6200 tx=0x7f3fd000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12ba0e041ef46e04 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:27.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.077+0000 7f3fd67fc700 1 -- 172.21.15.189:0/1124625276 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd000e070 con 0x7f3fd815e500 2024-04-06T12:12:27.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.077+0000 7f3fd67fc700 1 -- 172.21.15.189:0/1124625276 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd000b910 con 0x7f3fd815e500 2024-04-06T12:12:27.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.077+0000 7f3fd67fc700 1 -- 172.21.15.189:0/1124625276 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd000bcc0 con 0x7f3fd815e500 2024-04-06T12:12:27.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.077+0000 7f3fdddc6700 1 -- 172.21.15.189:0/1124625276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fd815e500 msgr2=0x7f3fd806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:27.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.077+0000 7f3fdddc6700 1 --2- 172.21.15.189:0/1124625276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fd815e500 0x7f3fd806abb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3fd80b6200 tx=0x7f3fd000b330 comp rx=0 tx=0).stop 2024-04-06T12:12:27.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.077+0000 7f3fdddc6700 1 -- 172.21.15.189:0/1124625276 shutdown_connections 2024-04-06T12:12:27.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.077+0000 7f3fdddc6700 1 --2- 172.21.15.189:0/1124625276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fd815e500 0x7f3fd806abb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.077+0000 7f3fdddc6700 1 --2- 172.21.15.189:0/1124625276 >> v2:172.21.15.192:3300/0 conn(0x7f3fd815d7b0 0x7f3fd80b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.077+0000 7f3fdddc6700 1 -- 172.21.15.189:0/1124625276 >> 172.21.15.189:0/1124625276 conn(0x7f3fd80d00b0 msgr2=0x7f3fd80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:27.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.077+0000 7f3fdddc6700 1 -- 172.21.15.189:0/1124625276 shutdown_connections 2024-04-06T12:12:27.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.077+0000 7f3fdddc6700 1 -- 172.21.15.189:0/1124625276 wait complete. 2024-04-06T12:12:27.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.078+0000 7f3fdddc6700 1 Processor -- start 2024-04-06T12:12:27.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.079+0000 7f3fdddc6700 1 -- start start 2024-04-06T12:12:27.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.079+0000 7f3fdddc6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fd806a730 0x7f3fd8153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:27.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.079+0000 7f3fdddc6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fd815d7b0 0x7f3fd815b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:27.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.079+0000 7f3fdddc6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fd815e500 0x7f3fd81597a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:27.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.079+0000 7f3fdddc6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd815b540 con 0x7f3fd815e500 2024-04-06T12:12:27.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.079+0000 7f3fdddc6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd80cd290 con 0x7f3fd815d7b0 2024-04-06T12:12:27.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.079+0000 7f3fdddc6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd80cd570 con 0x7f3fd806a730 2024-04-06T12:12:27.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.079+0000 7f3fd7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fd815e500 0x7f3fd81597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:27.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.079+0000 7f3fd7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fd815e500 0x7f3fd81597a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42746/0 (socket says 172.21.15.189:42746) 2024-04-06T12:12:27.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.079+0000 7f3fd7fff700 1 -- 172.21.15.189:0/3670048751 learned_addr learned my addr 172.21.15.189:0/3670048751 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:27.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.079+0000 7f3fd6ffd700 1 --2- 172.21.15.189:0/3670048751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fd815d7b0 0x7f3fd815b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:27.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.079+0000 7f3fd77fe700 1 --2- 172.21.15.189:0/3670048751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fd806a730 0x7f3fd8153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:27.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.080+0000 7f3fd7fff700 1 -- 172.21.15.189:0/3670048751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fd806a730 msgr2=0x7f3fd8153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:27.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.080+0000 7f3fd7fff700 1 --2- 172.21.15.189:0/3670048751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fd806a730 0x7f3fd8153e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.080+0000 7f3fd7fff700 1 -- 172.21.15.189:0/3670048751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fd815d7b0 msgr2=0x7f3fd815b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:27.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.080+0000 7f3fd7fff700 1 --2- 172.21.15.189:0/3670048751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fd815d7b0 0x7f3fd815b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.080+0000 7f3fd7fff700 1 -- 172.21.15.189:0/3670048751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd0009b30 con 0x7f3fd815e500 2024-04-06T12:12:27.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.080+0000 7f3fd7fff700 1 --2- 172.21.15.189:0/3670048751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fd815e500 0x7f3fd81597a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3fd806c820 tx=0x7f3fd0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:27.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.080+0000 7f3fbffff700 1 -- 172.21.15.189:0/3670048751 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0004880 con 0x7f3fd815e500 2024-04-06T12:12:27.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.080+0000 7f3fbffff700 1 -- 172.21.15.189:0/3670048751 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd00099b0 con 0x7f3fd815e500 2024-04-06T12:12:27.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.080+0000 7f3fdddc6700 1 -- 172.21.15.189:0/3670048751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd815aef0 con 0x7f3fd815e500 2024-04-06T12:12:27.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.080+0000 7f3fbffff700 1 -- 172.21.15.189:0/3670048751 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd001b670 con 0x7f3fd815e500 2024-04-06T12:12:27.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.080+0000 7f3fdddc6700 1 -- 172.21.15.189:0/3670048751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd80cceb0 con 0x7f3fd815e500 2024-04-06T12:12:27.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.081+0000 7f3fdddc6700 1 -- 172.21.15.189:0/3670048751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd815b6f0 con 0x7f3fd815e500 2024-04-06T12:12:27.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.081+0000 7f3fbffff700 1 -- 172.21.15.189:0/3670048751 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3fd001b7d0 con 0x7f3fd815e500 2024-04-06T12:12:27.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.082+0000 7f3fbffff700 1 --2- 172.21.15.189:0/3670048751 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3fc406ac90 0x7f3fc406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:27.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.082+0000 7f3fbffff700 1 -- 172.21.15.189:0/3670048751 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3fd008ad70 con 0x7f3fd815e500 2024-04-06T12:12:27.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.082+0000 7f3fd77fe700 1 --2- 172.21.15.189:0/3670048751 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3fc406ac90 0x7f3fc406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:27.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.082+0000 7f3fd77fe700 1 --2- 172.21.15.189:0/3670048751 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3fc406ac90 0x7f3fc406d150 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc009be0 tx=0x7f3fcc009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:27.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.086+0000 7f3fbffff700 1 -- 172.21.15.189:0/3670048751 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3fd815b6f0 con 0x7f3fd815e500 2024-04-06T12:12:27.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.280+0000 7f3fdddc6700 1 -- 172.21.15.189:0/3670048751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fd8067bb0 con 0x7f3fd815e500 2024-04-06T12:12:27.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.281+0000 7f3fbffff700 1 -- 172.21.15.189:0/3670048751 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f3fd0055cf0 con 0x7f3fd815e500 2024-04-06T12:12:27.280 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:27.280 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:08.649170+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:27.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.283+0000 7f3fdddc6700 1 -- 172.21.15.189:0/3670048751 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3fc406ac90 msgr2=0x7f3fc406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:27.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.283+0000 7f3fdddc6700 1 --2- 172.21.15.189:0/3670048751 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3fc406ac90 0x7f3fc406d150 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc009be0 tx=0x7f3fcc009210 comp rx=0 tx=0).stop 2024-04-06T12:12:27.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.283+0000 7f3fdddc6700 1 -- 172.21.15.189:0/3670048751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fd815e500 msgr2=0x7f3fd81597a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:27.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.283+0000 7f3fdddc6700 1 --2- 172.21.15.189:0/3670048751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fd815e500 0x7f3fd81597a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3fd806c820 tx=0x7f3fd0004ad0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.283+0000 7f3fdddc6700 1 -- 172.21.15.189:0/3670048751 shutdown_connections 2024-04-06T12:12:27.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.283+0000 7f3fdddc6700 1 --2- 172.21.15.189:0/3670048751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fd815e500 0x7f3fd81597a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.283+0000 7f3fdddc6700 1 --2- 172.21.15.189:0/3670048751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fd806a730 0x7f3fd8153e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.283+0000 7f3fdddc6700 1 --2- 172.21.15.189:0/3670048751 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3fc406ac90 0x7f3fc406d150 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.283+0000 7f3fdddc6700 1 --2- 172.21.15.189:0/3670048751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fd815d7b0 0x7f3fd815b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.283+0000 7f3fdddc6700 1 -- 172.21.15.189:0/3670048751 >> 172.21.15.189:0/3670048751 conn(0x7f3fd80d00b0 msgr2=0x7f3fd806eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:27.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.283+0000 7f3fdddc6700 1 -- 172.21.15.189:0/3670048751 shutdown_connections 2024-04-06T12:12:27.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.283+0000 7f3fdddc6700 1 -- 172.21.15.189:0/3670048751 wait complete. 2024-04-06T12:12:27.294 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-06T12:12:27.295 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:12:27.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.418+0000 7f148de93700 1 Processor -- start 2024-04-06T12:12:27.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.418+0000 7f148de93700 1 -- start start 2024-04-06T12:12:27.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.418+0000 7f148de93700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f148815d7b0 0x7f14880b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:27.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.418+0000 7f148de93700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f148815e500 0x7f148806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:27.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.418+0000 7f148de93700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14880b3990 con 0x7f148806a730 2024-04-06T12:12:27.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.418+0000 7f148de93700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f148815e950 con 0x7f148815e500 2024-04-06T12:12:27.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.419+0000 7f148de93700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f148806b0f0 con 0x7f148815d7b0 2024-04-06T12:12:27.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.419+0000 7f1487fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f148815e500 0x7f148806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:27.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.419+0000 7f1486ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f148806a730 0x7f148815ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55104/0 (socket says 172.21.15.189:55104) 2024-04-06T12:12:27.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.419+0000 7f1486ffd700 1 -- 172.21.15.189:0/997947046 learned_addr learned my addr 172.21.15.189:0/997947046 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:27.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.419+0000 7f14877fe700 1 --2- 172.21.15.189:0/997947046 >> v2:172.21.15.192:3300/0 conn(0x7f148815d7b0 0x7f14880b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:27.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.419+0000 7f1487fff700 1 -- 172.21.15.189:0/997947046 >> v2:172.21.15.192:3300/0 conn(0x7f148815d7b0 msgr2=0x7f14880b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:27.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.419+0000 7f1487fff700 1 --2- 172.21.15.189:0/997947046 >> v2:172.21.15.192:3300/0 conn(0x7f148815d7b0 0x7f14880b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.419+0000 7f1487fff700 1 -- 172.21.15.189:0/997947046 >> v1:172.21.15.189:6789/0 conn(0x7f148806a730 legacy=0x7f148815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:27.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.419+0000 7f1487fff700 1 -- 172.21.15.189:0/997947046 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14880b3060 con 0x7f148815e500 2024-04-06T12:12:27.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.419+0000 7f1487fff700 1 --2- 172.21.15.189:0/997947046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f148815e500 0x7f148806abb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f14880b6200 tx=0x7f148000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a73f43cdb81239a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:27.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.419+0000 7f14867fc700 1 -- 172.21.15.189:0/997947046 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148000e070 con 0x7f148815e500 2024-04-06T12:12:27.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.419+0000 7f14867fc700 1 -- 172.21.15.189:0/997947046 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148000bbd0 con 0x7f148815e500 2024-04-06T12:12:27.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.419+0000 7f14867fc700 1 -- 172.21.15.189:0/997947046 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480012430 con 0x7f148815e500 2024-04-06T12:12:27.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.420+0000 7f148de93700 1 -- 172.21.15.189:0/997947046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f148815e500 msgr2=0x7f148806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:27.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.420+0000 7f148de93700 1 --2- 172.21.15.189:0/997947046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f148815e500 0x7f148806abb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f14880b6200 tx=0x7f148000b330 comp rx=0 tx=0).stop 2024-04-06T12:12:27.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.420+0000 7f148de93700 1 -- 172.21.15.189:0/997947046 shutdown_connections 2024-04-06T12:12:27.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.420+0000 7f148de93700 1 --2- 172.21.15.189:0/997947046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f148815e500 0x7f148806abb0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.420+0000 7f148de93700 1 --2- 172.21.15.189:0/997947046 >> v2:172.21.15.192:3300/0 conn(0x7f148815d7b0 0x7f14880b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.420+0000 7f148de93700 1 -- 172.21.15.189:0/997947046 >> 172.21.15.189:0/997947046 conn(0x7f14880d00b0 msgr2=0x7f14880d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:27.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.420+0000 7f148de93700 1 -- 172.21.15.189:0/997947046 shutdown_connections 2024-04-06T12:12:27.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.420+0000 7f148de93700 1 -- 172.21.15.189:0/997947046 wait complete. 2024-04-06T12:12:27.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.421+0000 7f148de93700 1 Processor -- start 2024-04-06T12:12:27.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.421+0000 7f148de93700 1 -- start start 2024-04-06T12:12:27.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.422+0000 7f148de93700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f148806a730 0x7f148815a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:27.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.422+0000 7f148de93700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f148815d7b0 0x7f1488146bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:27.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.422+0000 7f148de93700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f148815e500 0x7f14880cd310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:27.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.422+0000 7f148de93700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14881470f0 con 0x7f148806a730 2024-04-06T12:12:27.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.422+0000 7f148de93700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14880cc7d0 con 0x7f148815e500 2024-04-06T12:12:27.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.422+0000 7f148de93700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148815ab70 con 0x7f148815d7b0 2024-04-06T12:12:27.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.422+0000 7f14877fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f148806a730 0x7f148815a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:27.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.422+0000 7f14877fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f148806a730 0x7f148815a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42762/0 (socket says 172.21.15.189:42762) 2024-04-06T12:12:27.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.422+0000 7f14877fe700 1 -- 172.21.15.189:0/3684052200 learned_addr learned my addr 172.21.15.189:0/3684052200 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:27.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.422+0000 7f1487fff700 1 --2- 172.21.15.189:0/3684052200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f148815e500 0x7f14880cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:27.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.422+0000 7f14877fe700 1 -- 172.21.15.189:0/3684052200 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f148815d7b0 msgr2=0x7f1488146bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:27.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.422+0000 7f1486ffd700 1 --2- 172.21.15.189:0/3684052200 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f148815d7b0 0x7f1488146bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:27.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.422+0000 7f14877fe700 1 --2- 172.21.15.189:0/3684052200 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f148815d7b0 0x7f1488146bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.422+0000 7f14877fe700 1 -- 172.21.15.189:0/3684052200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f148815e500 msgr2=0x7f14880cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:27.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.422+0000 7f14877fe700 1 --2- 172.21.15.189:0/3684052200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f148815e500 0x7f14880cd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.422+0000 7f14877fe700 1 -- 172.21.15.189:0/3684052200 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1480009a70 con 0x7f148806a730 2024-04-06T12:12:27.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.423+0000 7f14877fe700 1 --2- 172.21.15.189:0/3684052200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f148806a730 0x7f148815a4e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f147800e810 tx=0x7f147800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:27.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.423+0000 7f147ffff700 1 -- 172.21.15.189:0/3684052200 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1478013610 con 0x7f148806a730 2024-04-06T12:12:27.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.423+0000 7f147ffff700 1 -- 172.21.15.189:0/3684052200 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1478013770 con 0x7f148806a730 2024-04-06T12:12:27.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.423+0000 7f148de93700 1 -- 172.21.15.189:0/3684052200 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f148815ae50 con 0x7f148806a730 2024-04-06T12:12:27.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.423+0000 7f147ffff700 1 -- 172.21.15.189:0/3684052200 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1478013b20 con 0x7f148806a730 2024-04-06T12:12:27.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.423+0000 7f148de93700 1 -- 172.21.15.189:0/3684052200 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f148815b410 con 0x7f148806a730 2024-04-06T12:12:27.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.424+0000 7f148de93700 1 -- 172.21.15.189:0/3684052200 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1488067bb0 con 0x7f148806a730 2024-04-06T12:12:27.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.424+0000 7f147ffff700 1 -- 172.21.15.189:0/3684052200 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1478013c80 con 0x7f148806a730 2024-04-06T12:12:27.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.424+0000 7f147ffff700 1 --2- 172.21.15.189:0/3684052200 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f146806ac90 0x7f146806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:27.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.424+0000 7f147ffff700 1 -- 172.21.15.189:0/3684052200 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1478089290 con 0x7f148806a730 2024-04-06T12:12:27.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.425+0000 7f1486ffd700 1 --2- 172.21.15.189:0/3684052200 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f146806ac90 0x7f146806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:27.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.425+0000 7f1486ffd700 1 --2- 172.21.15.189:0/3684052200 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f146806ac90 0x7f146806d150 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1470005910 tx=0x7f1470009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:27.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.429+0000 7f147ffff700 1 -- 172.21.15.189:0/3684052200 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1478054190 con 0x7f148806a730 2024-04-06T12:12:27.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.622+0000 7f148de93700 1 -- 172.21.15.189:0/3684052200 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f14880b3620 con 0x7f146806ac90 2024-04-06T12:12:27.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.632+0000 7f147ffff700 1 -- 172.21.15.189:0/3684052200 <== mgr.4100 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345300 (secure 0 0 0) 0x7f14700013f0 con 0x7f146806ac90 2024-04-06T12:12:27.635 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:27.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.635+0000 7f148de93700 1 -- 172.21.15.189:0/3684052200 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f146806ac90 msgr2=0x7f146806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:27.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.635+0000 7f148de93700 1 --2- 172.21.15.189:0/3684052200 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f146806ac90 0x7f146806d150 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1470005910 tx=0x7f1470009040 comp rx=0 tx=0).stop 2024-04-06T12:12:27.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.635+0000 7f148de93700 1 -- 172.21.15.189:0/3684052200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f148806a730 msgr2=0x7f148815a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:27.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.635+0000 7f148de93700 1 --2- 172.21.15.189:0/3684052200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f148806a730 0x7f148815a4e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f147800e810 tx=0x7f147800ebd0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.635+0000 7f148de93700 1 -- 172.21.15.189:0/3684052200 shutdown_connections 2024-04-06T12:12:27.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.635+0000 7f148de93700 1 --2- 172.21.15.189:0/3684052200 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f146806ac90 0x7f146806d150 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.635+0000 7f148de93700 1 --2- 172.21.15.189:0/3684052200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f148815e500 0x7f14880cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.635+0000 7f148de93700 1 --2- 172.21.15.189:0/3684052200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f148806a730 0x7f148815a4e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.635+0000 7f148de93700 1 --2- 172.21.15.189:0/3684052200 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f148815d7b0 0x7f1488146bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.635+0000 7f148de93700 1 -- 172.21.15.189:0/3684052200 >> 172.21.15.189:0/3684052200 conn(0x7f14880d00b0 msgr2=0x7f1488150770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:27.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.635+0000 7f148de93700 1 -- 172.21.15.189:0/3684052200 shutdown_connections 2024-04-06T12:12:27.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.636+0000 7f148de93700 1 -- 172.21.15.189:0/3684052200 wait complete. 2024-04-06T12:12:27.639 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:12:27.647 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-04-06T12:12:26.570505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594962,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219360,"kb_used_data":3188,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755360,"statfs":{"total":773094113280,"available":772869488640,"internally_reserved":0,"allocated":3264512,"data_stored":1983069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2232,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003462"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224930+0000","last_change":"2024-04-06T12:12:04.226505+0000","last_active":"2024-04-06T12:12:08.224930+0000","last_peered":"2024-04-06T12:12:08.224930+0000","last_clean":"2024-04-06T12:12:08.224930+0000","last_became_active":"2024-04-06T12:12:04.226112+0000","last_became_peered":"2024-04-06T12:12:04.226112+0000","last_unstale":"2024-04-06T12:12:08.224930+0000","last_undegraded":"2024-04-06T12:12:08.224930+0000","last_fullsized":"2024-04-06T12:12:08.224930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:09:38.014527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223002+0000","last_change":"2024-04-06T12:12:04.217545+0000","last_active":"2024-04-06T12:12:08.223002+0000","last_peered":"2024-04-06T12:12:08.223002+0000","last_clean":"2024-04-06T12:12:08.223002+0000","last_became_active":"2024-04-06T12:12:04.217304+0000","last_became_peered":"2024-04-06T12:12:04.217304+0000","last_unstale":"2024-04-06T12:12:08.223002+0000","last_undegraded":"2024-04-06T12:12:08.223002+0000","last_fullsized":"2024-04-06T12:12:08.223002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.231018+0000","last_change":"2024-04-06T12:12:04.209328+0000","last_active":"2024-04-06T12:12:08.231018+0000","last_peered":"2024-04-06T12:12:08.231018+0000","last_clean":"2024-04-06T12:12:08.231018+0000","last_became_active":"2024-04-06T12:12:04.209066+0000","last_became_peered":"2024-04-06T12:12:04.209066+0000","last_unstale":"2024-04-06T12:12:08.231018+0000","last_undegraded":"2024-04-06T12:12:08.231018+0000","last_fullsized":"2024-04-06T12:12:08.231018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:02:53.621993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227488+0000","last_change":"2024-04-06T12:12:04.221440+0000","last_active":"2024-04-06T12:12:08.227488+0000","last_peered":"2024-04-06T12:12:08.227488+0000","last_clean":"2024-04-06T12:12:08.227488+0000","last_became_active":"2024-04-06T12:12:04.220738+0000","last_became_peered":"2024-04-06T12:12:04.220738+0000","last_unstale":"2024-04-06T12:12:08.227488+0000","last_undegraded":"2024-04-06T12:12:08.227488+0000","last_fullsized":"2024-04-06T12:12:08.227488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:43:39.601900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223982+0000","last_change":"2024-04-06T12:12:04.223078+0000","last_active":"2024-04-06T12:12:08.223982+0000","last_peered":"2024-04-06T12:12:08.223982+0000","last_clean":"2024-04-06T12:12:08.223982+0000","last_became_active":"2024-04-06T12:12:04.222831+0000","last_became_peered":"2024-04-06T12:12:04.222831+0000","last_unstale":"2024-04-06T12:12:08.223982+0000","last_undegraded":"2024-04-06T12:12:08.223982+0000","last_fullsized":"2024-04-06T12:12:08.223982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:02:00.219999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227898+0000","last_change":"2024-04-06T12:12:04.215126+0000","last_active":"2024-04-06T12:12:08.227898+0000","last_peered":"2024-04-06T12:12:08.227898+0000","last_clean":"2024-04-06T12:12:08.227898+0000","last_became_active":"2024-04-06T12:12:04.214857+0000","last_became_peered":"2024-04-06T12:12:04.214857+0000","last_unstale":"2024-04-06T12:12:08.227898+0000","last_undegraded":"2024-04-06T12:12:08.227898+0000","last_fullsized":"2024-04-06T12:12:08.227898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:02:00.219999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223337+0000","last_change":"2024-04-06T12:12:04.222961+0000","last_active":"2024-04-06T12:12:08.223337+0000","last_peered":"2024-04-06T12:12:08.223337+0000","last_clean":"2024-04-06T12:12:08.223337+0000","last_became_active":"2024-04-06T12:12:04.222659+0000","last_became_peered":"2024-04-06T12:12:04.222659+0000","last_unstale":"2024-04-06T12:12:08.223337+0000","last_undegraded":"2024-04-06T12:12:08.223337+0000","last_fullsized":"2024-04-06T12:12:08.223337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:55.517965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.233016+0000","last_change":"2024-04-06T12:12:04.222214+0000","last_active":"2024-04-06T12:12:08.233016+0000","last_peered":"2024-04-06T12:12:08.233016+0000","last_clean":"2024-04-06T12:12:08.233016+0000","last_became_active":"2024-04-06T12:12:04.221817+0000","last_became_peered":"2024-04-06T12:12:04.221817+0000","last_unstale":"2024-04-06T12:12:08.233016+0000","last_undegraded":"2024-04-06T12:12:08.233016+0000","last_fullsized":"2024-04-06T12:12:08.233016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224843+0000","last_change":"2024-04-06T12:12:04.224106+0000","last_active":"2024-04-06T12:12:08.224843+0000","last_peered":"2024-04-06T12:12:08.224843+0000","last_clean":"2024-04-06T12:12:08.224843+0000","last_became_active":"2024-04-06T12:12:04.223653+0000","last_became_peered":"2024-04-06T12:12:04.223653+0000","last_unstale":"2024-04-06T12:12:08.224843+0000","last_undegraded":"2024-04-06T12:12:08.224843+0000","last_fullsized":"2024-04-06T12:12:08.224843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:50:27.596378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223873+0000","last_change":"2024-04-06T12:12:04.223892+0000","last_active":"2024-04-06T12:12:08.223873+0000","last_peered":"2024-04-06T12:12:08.223873+0000","last_clean":"2024-04-06T12:12:08.223873+0000","last_became_active":"2024-04-06T12:12:04.223615+0000","last_became_peered":"2024-04-06T12:12:04.223615+0000","last_unstale":"2024-04-06T12:12:08.223873+0000","last_undegraded":"2024-04-06T12:12:08.223873+0000","last_fullsized":"2024-04-06T12:12:08.223873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:02:53.621993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.225163+0000","last_change":"2024-04-06T12:12:04.223861+0000","last_active":"2024-04-06T12:12:08.225163+0000","last_peered":"2024-04-06T12:12:08.225163+0000","last_clean":"2024-04-06T12:12:08.225163+0000","last_became_active":"2024-04-06T12:12:04.223284+0000","last_became_peered":"2024-04-06T12:12:04.223284+0000","last_unstale":"2024-04-06T12:12:08.225163+0000","last_undegraded":"2024-04-06T12:12:08.225163+0000","last_fullsized":"2024-04-06T12:12:08.225163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:00.258491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232662+0000","last_change":"2024-04-06T12:12:04.230879+0000","last_active":"2024-04-06T12:12:08.232662+0000","last_peered":"2024-04-06T12:12:08.232662+0000","last_clean":"2024-04-06T12:12:08.232662+0000","last_became_active":"2024-04-06T12:12:04.230592+0000","last_became_peered":"2024-04-06T12:12:04.230592+0000","last_unstale":"2024-04-06T12:12:08.232662+0000","last_undegraded":"2024-04-06T12:12:08.232662+0000","last_fullsized":"2024-04-06T12:12:08.232662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:10.512788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232539+0000","last_change":"2024-04-06T12:12:04.218704+0000","last_active":"2024-04-06T12:12:08.232539+0000","last_peered":"2024-04-06T12:12:08.232539+0000","last_clean":"2024-04-06T12:12:08.232539+0000","last_became_active":"2024-04-06T12:12:04.218455+0000","last_became_peered":"2024-04-06T12:12:04.218455+0000","last_unstale":"2024-04-06T12:12:08.232539+0000","last_undegraded":"2024-04-06T12:12:08.232539+0000","last_fullsized":"2024-04-06T12:12:08.232539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:21:16.167775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.235340+0000","last_change":"2024-04-06T12:12:04.209351+0000","last_active":"2024-04-06T12:12:08.235340+0000","last_peered":"2024-04-06T12:12:08.235340+0000","last_clean":"2024-04-06T12:12:08.235340+0000","last_became_active":"2024-04-06T12:12:04.209092+0000","last_became_peered":"2024-04-06T12:12:04.209092+0000","last_unstale":"2024-04-06T12:12:08.235340+0000","last_undegraded":"2024-04-06T12:12:08.235340+0000","last_fullsized":"2024-04-06T12:12:08.235340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222602+0000","last_change":"2024-04-06T12:12:04.219642+0000","last_active":"2024-04-06T12:12:08.222602+0000","last_peered":"2024-04-06T12:12:08.222602+0000","last_clean":"2024-04-06T12:12:08.222602+0000","last_became_active":"2024-04-06T12:12:04.219398+0000","last_became_peered":"2024-04-06T12:12:04.219398+0000","last_unstale":"2024-04-06T12:12:08.222602+0000","last_undegraded":"2024-04-06T12:12:08.222602+0000","last_fullsized":"2024-04-06T12:12:08.222602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227279+0000","last_change":"2024-04-06T12:12:04.215338+0000","last_active":"2024-04-06T12:12:08.227279+0000","last_peered":"2024-04-06T12:12:08.227279+0000","last_clean":"2024-04-06T12:12:08.227279+0000","last_became_active":"2024-04-06T12:12:04.215021+0000","last_became_peered":"2024-04-06T12:12:04.215021+0000","last_unstale":"2024-04-06T12:12:08.227279+0000","last_undegraded":"2024-04-06T12:12:08.227279+0000","last_fullsized":"2024-04-06T12:12:08.227279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:09:38.014527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.229099+0000","last_change":"2024-04-06T12:12:04.225406+0000","last_active":"2024-04-06T12:12:08.229099+0000","last_peered":"2024-04-06T12:12:08.229099+0000","last_clean":"2024-04-06T12:12:08.229099+0000","last_became_active":"2024-04-06T12:12:04.224950+0000","last_became_peered":"2024-04-06T12:12:04.224950+0000","last_unstale":"2024-04-06T12:12:08.229099+0000","last_undegraded":"2024-04-06T12:12:08.229099+0000","last_fullsized":"2024-04-06T12:12:08.229099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:55.517965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228664+0000","last_change":"2024-04-06T12:12:04.217691+0000","last_active":"2024-04-06T12:12:08.228664+0000","last_peered":"2024-04-06T12:12:08.228664+0000","last_clean":"2024-04-06T12:12:08.228664+0000","last_became_active":"2024-04-06T12:12:04.217425+0000","last_became_peered":"2024-04-06T12:12:04.217425+0000","last_unstale":"2024-04-06T12:12:08.228664+0000","last_undegraded":"2024-04-06T12:12:08.228664+0000","last_fullsized":"2024-04-06T12:12:08.228664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:10.512788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224434+0000","last_change":"2024-04-06T12:12:04.217807+0000","last_active":"2024-04-06T12:12:08.224434+0000","last_peered":"2024-04-06T12:12:08.224434+0000","last_clean":"2024-04-06T12:12:08.224434+0000","last_became_active":"2024-04-06T12:12:04.217533+0000","last_became_peered":"2024-04-06T12:12:04.217533+0000","last_unstale":"2024-04-06T12:12:08.224434+0000","last_undegraded":"2024-04-06T12:12:08.224434+0000","last_fullsized":"2024-04-06T12:12:08.224434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:43:39.601900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247241+0000","last_change":"2024-04-06T12:12:08.242261+0000","last_active":"2024-04-06T12:12:08.247241+0000","last_peered":"2024-04-06T12:12:08.247241+0000","last_clean":"2024-04-06T12:12:08.247241+0000","last_became_active":"2024-04-06T12:12:08.241896+0000","last_became_peered":"2024-04-06T12:12:08.241896+0000","last_unstale":"2024-04-06T12:12:08.247241+0000","last_undegraded":"2024-04-06T12:12:08.247241+0000","last_fullsized":"2024-04-06T12:12:08.247241+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:31:31.349113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232576+0000","last_change":"2024-04-06T12:12:03.799562+0000","last_active":"2024-04-06T12:12:08.232576+0000","last_peered":"2024-04-06T12:12:08.232576+0000","last_clean":"2024-04-06T12:12:08.232576+0000","last_became_active":"2024-04-06T12:12:03.799298+0000","last_became_peered":"2024-04-06T12:12:03.799298+0000","last_unstale":"2024-04-06T12:12:08.232576+0000","last_undegraded":"2024-04-06T12:12:08.232576+0000","last_fullsized":"2024-04-06T12:12:08.232576+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:05:14.778375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232962+0000","last_change":"2024-04-06T12:12:04.217203+0000","last_active":"2024-04-06T12:12:08.232962+0000","last_peered":"2024-04-06T12:12:08.232962+0000","last_clean":"2024-04-06T12:12:08.232962+0000","last_became_active":"2024-04-06T12:12:04.216782+0000","last_became_peered":"2024-04-06T12:12:04.216782+0000","last_unstale":"2024-04-06T12:12:08.232962+0000","last_undegraded":"2024-04-06T12:12:08.232962+0000","last_fullsized":"2024-04-06T12:12:08.232962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:55.517965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.255247+0000","last_change":"2024-04-06T12:12:08.251632+0000","last_active":"2024-04-06T12:12:08.255247+0000","last_peered":"2024-04-06T12:12:08.255247+0000","last_clean":"2024-04-06T12:12:08.255247+0000","last_became_active":"2024-04-06T12:12:08.251238+0000","last_became_peered":"2024-04-06T12:12:08.251238+0000","last_unstale":"2024-04-06T12:12:08.255247+0000","last_undegraded":"2024-04-06T12:12:08.255247+0000","last_fullsized":"2024-04-06T12:12:08.255247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:32:02.566075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.233059+0000","last_change":"2024-04-06T12:12:03.792714+0000","last_active":"2024-04-06T12:12:08.233059+0000","last_peered":"2024-04-06T12:12:08.233059+0000","last_clean":"2024-04-06T12:12:08.233059+0000","last_became_active":"2024-04-06T12:12:03.792445+0000","last_became_peered":"2024-04-06T12:12:03.792445+0000","last_unstale":"2024-04-06T12:12:08.233059+0000","last_undegraded":"2024-04-06T12:12:08.233059+0000","last_fullsized":"2024-04-06T12:12:08.233059+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:40:20.784453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232461+0000","last_change":"2024-04-06T12:12:04.215439+0000","last_active":"2024-04-06T12:12:08.232461+0000","last_peered":"2024-04-06T12:12:08.232461+0000","last_clean":"2024-04-06T12:12:08.232461+0000","last_became_active":"2024-04-06T12:12:04.214846+0000","last_became_peered":"2024-04-06T12:12:04.214846+0000","last_unstale":"2024-04-06T12:12:08.232461+0000","last_undegraded":"2024-04-06T12:12:08.232461+0000","last_fullsized":"2024-04-06T12:12:08.232461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.250137+0000","last_change":"2024-04-06T12:12:08.248890+0000","last_active":"2024-04-06T12:12:08.250137+0000","last_peered":"2024-04-06T12:12:08.250137+0000","last_clean":"2024-04-06T12:12:08.250137+0000","last_became_active":"2024-04-06T12:12:08.248470+0000","last_became_peered":"2024-04-06T12:12:08.248470+0000","last_unstale":"2024-04-06T12:12:08.250137+0000","last_undegraded":"2024-04-06T12:12:08.250137+0000","last_fullsized":"2024-04-06T12:12:08.250137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:06.085793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223703+0000","last_change":"2024-04-06T12:12:03.799557+0000","last_active":"2024-04-06T12:12:08.223703+0000","last_peered":"2024-04-06T12:12:08.223703+0000","last_clean":"2024-04-06T12:12:08.223703+0000","last_became_active":"2024-04-06T12:12:03.799271+0000","last_became_peered":"2024-04-06T12:12:03.799271+0000","last_unstale":"2024-04-06T12:12:08.223703+0000","last_undegraded":"2024-04-06T12:12:08.223703+0000","last_fullsized":"2024-04-06T12:12:08.223703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:26:21.506491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227705+0000","last_change":"2024-04-06T12:12:04.215431+0000","last_active":"2024-04-06T12:12:08.227705+0000","last_peered":"2024-04-06T12:12:08.227705+0000","last_clean":"2024-04-06T12:12:08.227705+0000","last_became_active":"2024-04-06T12:12:04.214459+0000","last_became_peered":"2024-04-06T12:12:04.214459+0000","last_unstale":"2024-04-06T12:12:08.227705+0000","last_undegraded":"2024-04-06T12:12:08.227705+0000","last_fullsized":"2024-04-06T12:12:08.227705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.252843+0000","last_change":"2024-04-06T12:12:08.249210+0000","last_active":"2024-04-06T12:12:08.252843+0000","last_peered":"2024-04-06T12:12:08.252843+0000","last_clean":"2024-04-06T12:12:08.252843+0000","last_became_active":"2024-04-06T12:12:08.248661+0000","last_became_peered":"2024-04-06T12:12:08.248661+0000","last_unstale":"2024-04-06T12:12:08.252843+0000","last_undegraded":"2024-04-06T12:12:08.252843+0000","last_fullsized":"2024-04-06T12:12:08.252843+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:29.953853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224295+0000","last_change":"2024-04-06T12:12:03.800006+0000","last_active":"2024-04-06T12:12:08.224295+0000","last_peered":"2024-04-06T12:12:08.224295+0000","last_clean":"2024-04-06T12:12:08.224295+0000","last_became_active":"2024-04-06T12:12:03.799598+0000","last_became_peered":"2024-04-06T12:12:03.799598+0000","last_unstale":"2024-04-06T12:12:08.224295+0000","last_undegraded":"2024-04-06T12:12:08.224295+0000","last_fullsized":"2024-04-06T12:12:08.224295+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:04.938049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.229419+0000","last_change":"2024-04-06T12:12:04.220105+0000","last_active":"2024-04-06T12:12:08.229419+0000","last_peered":"2024-04-06T12:12:08.229419+0000","last_clean":"2024-04-06T12:12:08.229419+0000","last_became_active":"2024-04-06T12:12:04.219876+0000","last_became_peered":"2024-04-06T12:12:04.219876+0000","last_unstale":"2024-04-06T12:12:08.229419+0000","last_undegraded":"2024-04-06T12:12:08.229419+0000","last_fullsized":"2024-04-06T12:12:08.229419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.249020+0000","last_change":"2024-04-06T12:12:08.246410+0000","last_active":"2024-04-06T12:12:08.249020+0000","last_peered":"2024-04-06T12:12:08.249020+0000","last_clean":"2024-04-06T12:12:08.249020+0000","last_became_active":"2024-04-06T12:12:08.246057+0000","last_became_peered":"2024-04-06T12:12:08.246057+0000","last_unstale":"2024-04-06T12:12:08.249020+0000","last_undegraded":"2024-04-06T12:12:08.249020+0000","last_fullsized":"2024-04-06T12:12:08.249020+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:16.183719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232695+0000","last_change":"2024-04-06T12:12:03.798850+0000","last_active":"2024-04-06T12:12:08.232695+0000","last_peered":"2024-04-06T12:12:08.232695+0000","last_clean":"2024-04-06T12:12:08.232695+0000","last_became_active":"2024-04-06T12:12:03.798550+0000","last_became_peered":"2024-04-06T12:12:03.798550+0000","last_unstale":"2024-04-06T12:12:08.232695+0000","last_undegraded":"2024-04-06T12:12:08.232695+0000","last_fullsized":"2024-04-06T12:12:08.232695+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:05:14.778375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232223+0000","last_change":"2024-04-06T12:12:04.212276+0000","last_active":"2024-04-06T12:12:08.232223+0000","last_peered":"2024-04-06T12:12:08.232223+0000","last_clean":"2024-04-06T12:12:08.232223+0000","last_became_active":"2024-04-06T12:12:04.211681+0000","last_became_peered":"2024-04-06T12:12:04.211681+0000","last_unstale":"2024-04-06T12:12:08.232223+0000","last_undegraded":"2024-04-06T12:12:08.232223+0000","last_fullsized":"2024-04-06T12:12:08.232223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:21:43.141955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.248388+0000","last_change":"2024-04-06T12:12:08.244209+0000","last_active":"2024-04-06T12:12:08.248388+0000","last_peered":"2024-04-06T12:12:08.248388+0000","last_clean":"2024-04-06T12:12:08.248388+0000","last_became_active":"2024-04-06T12:12:08.243274+0000","last_became_peered":"2024-04-06T12:12:08.243274+0000","last_unstale":"2024-04-06T12:12:08.248388+0000","last_undegraded":"2024-04-06T12:12:08.248388+0000","last_fullsized":"2024-04-06T12:12:08.248388+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:03.696838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223468+0000","last_change":"2024-04-06T12:12:03.796799+0000","last_active":"2024-04-06T12:12:08.223468+0000","last_peered":"2024-04-06T12:12:08.223468+0000","last_clean":"2024-04-06T12:12:08.223468+0000","last_became_active":"2024-04-06T12:12:03.796536+0000","last_became_peered":"2024-04-06T12:12:03.796536+0000","last_unstale":"2024-04-06T12:12:08.223468+0000","last_undegraded":"2024-04-06T12:12:08.223468+0000","last_fullsized":"2024-04-06T12:12:08.223468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:15.699305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234439+0000","last_change":"2024-04-06T12:12:04.220607+0000","last_active":"2024-04-06T12:12:08.234439+0000","last_peered":"2024-04-06T12:12:08.234439+0000","last_clean":"2024-04-06T12:12:08.234439+0000","last_became_active":"2024-04-06T12:12:04.220242+0000","last_became_peered":"2024-04-06T12:12:04.220242+0000","last_unstale":"2024-04-06T12:12:08.234439+0000","last_undegraded":"2024-04-06T12:12:08.234439+0000","last_fullsized":"2024-04-06T12:12:08.234439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:00.258491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.243307+0000","last_change":"2024-04-06T12:12:08.237694+0000","last_active":"2024-04-06T12:12:08.243307+0000","last_peered":"2024-04-06T12:12:08.243307+0000","last_clean":"2024-04-06T12:12:08.243307+0000","last_became_active":"2024-04-06T12:12:08.237190+0000","last_became_peered":"2024-04-06T12:12:08.237190+0000","last_unstale":"2024-04-06T12:12:08.243307+0000","last_undegraded":"2024-04-06T12:12:08.243307+0000","last_fullsized":"2024-04-06T12:12:08.243307+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:50:32.228428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232240+0000","last_change":"2024-04-06T12:12:03.800903+0000","last_active":"2024-04-06T12:12:08.232240+0000","last_peered":"2024-04-06T12:12:08.232240+0000","last_clean":"2024-04-06T12:12:08.232240+0000","last_became_active":"2024-04-06T12:12:03.800590+0000","last_became_peered":"2024-04-06T12:12:03.800590+0000","last_unstale":"2024-04-06T12:12:08.232240+0000","last_undegraded":"2024-04-06T12:12:08.232240+0000","last_fullsized":"2024-04-06T12:12:08.232240+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:00:31.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232899+0000","last_change":"2024-04-06T12:12:04.216144+0000","last_active":"2024-04-06T12:12:08.232899+0000","last_peered":"2024-04-06T12:12:08.232899+0000","last_clean":"2024-04-06T12:12:08.232899+0000","last_became_active":"2024-04-06T12:12:04.215885+0000","last_became_peered":"2024-04-06T12:12:04.215885+0000","last_unstale":"2024-04-06T12:12:08.232899+0000","last_undegraded":"2024-04-06T12:12:08.232899+0000","last_fullsized":"2024-04-06T12:12:08.232899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:36:25.380472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256611+0000","last_change":"2024-04-06T12:12:08.252363+0000","last_active":"2024-04-06T12:12:08.256611+0000","last_peered":"2024-04-06T12:12:08.256611+0000","last_clean":"2024-04-06T12:12:08.256611+0000","last_became_active":"2024-04-06T12:12:08.252029+0000","last_became_peered":"2024-04-06T12:12:08.252029+0000","last_unstale":"2024-04-06T12:12:08.256611+0000","last_undegraded":"2024-04-06T12:12:08.256611+0000","last_fullsized":"2024-04-06T12:12:08.256611+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:35:54.255608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227386+0000","last_change":"2024-04-06T12:12:03.800462+0000","last_active":"2024-04-06T12:12:08.227386+0000","last_peered":"2024-04-06T12:12:08.227386+0000","last_clean":"2024-04-06T12:12:08.227386+0000","last_became_active":"2024-04-06T12:12:03.800210+0000","last_became_peered":"2024-04-06T12:12:03.800210+0000","last_unstale":"2024-04-06T12:12:08.227386+0000","last_undegraded":"2024-04-06T12:12:08.227386+0000","last_fullsized":"2024-04-06T12:12:08.227386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:00:31.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222742+0000","last_change":"2024-04-06T12:12:04.216700+0000","last_active":"2024-04-06T12:12:08.222742+0000","last_peered":"2024-04-06T12:12:08.222742+0000","last_clean":"2024-04-06T12:12:08.222742+0000","last_became_active":"2024-04-06T12:12:04.216435+0000","last_became_peered":"2024-04-06T12:12:04.216435+0000","last_unstale":"2024-04-06T12:12:08.222742+0000","last_undegraded":"2024-04-06T12:12:08.222742+0000","last_fullsized":"2024-04-06T12:12:08.222742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:36:25.380472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247221+0000","last_change":"2024-04-06T12:12:08.242079+0000","last_active":"2024-04-06T12:12:08.247221+0000","last_peered":"2024-04-06T12:12:08.247221+0000","last_clean":"2024-04-06T12:12:08.247221+0000","last_became_active":"2024-04-06T12:12:08.241648+0000","last_became_peered":"2024-04-06T12:12:08.241648+0000","last_unstale":"2024-04-06T12:12:08.247221+0000","last_undegraded":"2024-04-06T12:12:08.247221+0000","last_fullsized":"2024-04-06T12:12:08.247221+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:37:57.999646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224055+0000","last_change":"2024-04-06T12:12:03.799875+0000","last_active":"2024-04-06T12:12:08.224055+0000","last_peered":"2024-04-06T12:12:08.224055+0000","last_clean":"2024-04-06T12:12:08.224055+0000","last_became_active":"2024-04-06T12:12:03.799562+0000","last_became_peered":"2024-04-06T12:12:03.799562+0000","last_unstale":"2024-04-06T12:12:08.224055+0000","last_undegraded":"2024-04-06T12:12:08.224055+0000","last_fullsized":"2024-04-06T12:12:08.224055+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:08:55.770454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.229090+0000","last_change":"2024-04-06T12:12:04.225142+0000","last_active":"2024-04-06T12:12:08.229090+0000","last_peered":"2024-04-06T12:12:08.229090+0000","last_clean":"2024-04-06T12:12:08.229090+0000","last_became_active":"2024-04-06T12:12:04.224838+0000","last_became_peered":"2024-04-06T12:12:04.224838+0000","last_unstale":"2024-04-06T12:12:08.229090+0000","last_undegraded":"2024-04-06T12:12:08.229090+0000","last_fullsized":"2024-04-06T12:12:08.229090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:21:16.167775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247222+0000","last_change":"2024-04-06T12:12:08.244037+0000","last_active":"2024-04-06T12:12:08.247222+0000","last_peered":"2024-04-06T12:12:08.247222+0000","last_clean":"2024-04-06T12:12:08.247222+0000","last_became_active":"2024-04-06T12:12:08.243685+0000","last_became_peered":"2024-04-06T12:12:08.243685+0000","last_unstale":"2024-04-06T12:12:08.247222+0000","last_undegraded":"2024-04-06T12:12:08.247222+0000","last_fullsized":"2024-04-06T12:12:08.247222+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:02:51.216167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228999+0000","last_change":"2024-04-06T12:12:03.799591+0000","last_active":"2024-04-06T12:12:08.228999+0000","last_peered":"2024-04-06T12:12:08.228999+0000","last_clean":"2024-04-06T12:12:08.228999+0000","last_became_active":"2024-04-06T12:12:03.799333+0000","last_became_peered":"2024-04-06T12:12:03.799333+0000","last_unstale":"2024-04-06T12:12:08.228999+0000","last_undegraded":"2024-04-06T12:12:08.228999+0000","last_fullsized":"2024-04-06T12:12:08.228999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:11.635313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232670+0000","last_change":"2024-04-06T12:12:04.215620+0000","last_active":"2024-04-06T12:12:08.232670+0000","last_peered":"2024-04-06T12:12:08.232670+0000","last_clean":"2024-04-06T12:12:08.232670+0000","last_became_active":"2024-04-06T12:12:04.215189+0000","last_became_peered":"2024-04-06T12:12:04.215189+0000","last_unstale":"2024-04-06T12:12:08.232670+0000","last_undegraded":"2024-04-06T12:12:08.232670+0000","last_fullsized":"2024-04-06T12:12:08.232670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247852+0000","last_change":"2024-04-06T12:12:08.246657+0000","last_active":"2024-04-06T12:12:08.247852+0000","last_peered":"2024-04-06T12:12:08.247852+0000","last_clean":"2024-04-06T12:12:08.247852+0000","last_became_active":"2024-04-06T12:12:08.246310+0000","last_became_peered":"2024-04-06T12:12:08.246310+0000","last_unstale":"2024-04-06T12:12:08.247852+0000","last_undegraded":"2024-04-06T12:12:08.247852+0000","last_fullsized":"2024-04-06T12:12:08.247852+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:57:02.114694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223115+0000","last_change":"2024-04-06T12:12:03.802440+0000","last_active":"2024-04-06T12:12:08.223115+0000","last_peered":"2024-04-06T12:12:08.223115+0000","last_clean":"2024-04-06T12:12:08.223115+0000","last_became_active":"2024-04-06T12:12:03.802193+0000","last_became_peered":"2024-04-06T12:12:03.802193+0000","last_unstale":"2024-04-06T12:12:08.223115+0000","last_undegraded":"2024-04-06T12:12:08.223115+0000","last_fullsized":"2024-04-06T12:12:08.223115+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:04.938049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232783+0000","last_change":"2024-04-06T12:12:04.225740+0000","last_active":"2024-04-06T12:12:08.232783+0000","last_peered":"2024-04-06T12:12:08.232783+0000","last_clean":"2024-04-06T12:12:08.232783+0000","last_became_active":"2024-04-06T12:12:04.224605+0000","last_became_peered":"2024-04-06T12:12:04.224605+0000","last_unstale":"2024-04-06T12:12:08.232783+0000","last_undegraded":"2024-04-06T12:12:08.232783+0000","last_fullsized":"2024-04-06T12:12:08.232783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:26:22.511847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.245748+0000","last_change":"2024-04-06T12:12:08.242260+0000","last_active":"2024-04-06T12:12:08.245748+0000","last_peered":"2024-04-06T12:12:08.245748+0000","last_clean":"2024-04-06T12:12:08.245748+0000","last_became_active":"2024-04-06T12:12:08.241863+0000","last_became_peered":"2024-04-06T12:12:08.241863+0000","last_unstale":"2024-04-06T12:12:08.245748+0000","last_undegraded":"2024-04-06T12:12:08.245748+0000","last_fullsized":"2024-04-06T12:12:08.245748+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:06.085793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-06T12:12:13.609454+0000","last_change":"2024-04-06T12:12:03.799037+0000","last_active":"2024-04-06T12:12:13.609454+0000","last_peered":"2024-04-06T12:12:13.609454+0000","last_clean":"2024-04-06T12:12:13.609454+0000","last_became_active":"2024-04-06T12:12:03.798723+0000","last_became_peered":"2024-04-06T12:12:03.798723+0000","last_unstale":"2024-04-06T12:12:13.609454+0000","last_undegraded":"2024-04-06T12:12:13.609454+0000","last_fullsized":"2024-04-06T12:12:13.609454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:00:31.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3796,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224345+0000","last_change":"2024-04-06T12:12:04.226914+0000","last_active":"2024-04-06T12:12:08.224345+0000","last_peered":"2024-04-06T12:12:08.224345+0000","last_clean":"2024-04-06T12:12:08.224345+0000","last_became_active":"2024-04-06T12:12:04.226421+0000","last_became_peered":"2024-04-06T12:12:04.226421+0000","last_unstale":"2024-04-06T12:12:08.224345+0000","last_undegraded":"2024-04-06T12:12:08.224345+0000","last_fullsized":"2024-04-06T12:12:08.224345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:45:10.755538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247061+0000","last_change":"2024-04-06T12:12:08.242923+0000","last_active":"2024-04-06T12:12:08.247061+0000","last_peered":"2024-04-06T12:12:08.247061+0000","last_clean":"2024-04-06T12:12:08.247061+0000","last_became_active":"2024-04-06T12:12:08.242491+0000","last_became_peered":"2024-04-06T12:12:08.242491+0000","last_unstale":"2024-04-06T12:12:08.247061+0000","last_undegraded":"2024-04-06T12:12:08.247061+0000","last_fullsized":"2024-04-06T12:12:08.247061+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:57:02.114694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234681+0000","last_change":"2024-04-06T12:12:03.796757+0000","last_active":"2024-04-06T12:12:08.234681+0000","last_peered":"2024-04-06T12:12:08.234681+0000","last_clean":"2024-04-06T12:12:08.234681+0000","last_became_active":"2024-04-06T12:12:03.796519+0000","last_became_peered":"2024-04-06T12:12:03.796519+0000","last_unstale":"2024-04-06T12:12:08.234681+0000","last_undegraded":"2024-04-06T12:12:08.234681+0000","last_fullsized":"2024-04-06T12:12:08.234681+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:40:20.784453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.235485+0000","last_change":"2024-04-06T12:12:04.224445+0000","last_active":"2024-04-06T12:12:08.235485+0000","last_peered":"2024-04-06T12:12:08.235485+0000","last_clean":"2024-04-06T12:12:08.235485+0000","last_became_active":"2024-04-06T12:12:04.224153+0000","last_became_peered":"2024-04-06T12:12:04.224153+0000","last_unstale":"2024-04-06T12:12:08.235485+0000","last_undegraded":"2024-04-06T12:12:08.235485+0000","last_fullsized":"2024-04-06T12:12:08.235485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:15.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256724+0000","last_change":"2024-04-06T12:12:08.251536+0000","last_active":"2024-04-06T12:12:08.256724+0000","last_peered":"2024-04-06T12:12:08.256724+0000","last_clean":"2024-04-06T12:12:08.256724+0000","last_became_active":"2024-04-06T12:12:08.251142+0000","last_became_peered":"2024-04-06T12:12:08.251142+0000","last_unstale":"2024-04-06T12:12:08.256724+0000","last_undegraded":"2024-04-06T12:12:08.256724+0000","last_fullsized":"2024-04-06T12:12:08.256724+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:48:55.155611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228030+0000","last_change":"2024-04-06T12:12:03.797816+0000","last_active":"2024-04-06T12:12:08.228030+0000","last_peered":"2024-04-06T12:12:08.228030+0000","last_clean":"2024-04-06T12:12:08.228030+0000","last_became_active":"2024-04-06T12:12:03.797540+0000","last_became_peered":"2024-04-06T12:12:03.797540+0000","last_unstale":"2024-04-06T12:12:08.228030+0000","last_undegraded":"2024-04-06T12:12:08.228030+0000","last_fullsized":"2024-04-06T12:12:08.228030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:15.699305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224078+0000","last_change":"2024-04-06T12:12:04.223703+0000","last_active":"2024-04-06T12:12:08.224078+0000","last_peered":"2024-04-06T12:12:08.224078+0000","last_clean":"2024-04-06T12:12:08.224078+0000","last_became_active":"2024-04-06T12:12:04.223394+0000","last_became_peered":"2024-04-06T12:12:04.223394+0000","last_unstale":"2024-04-06T12:12:08.224078+0000","last_undegraded":"2024-04-06T12:12:08.224078+0000","last_fullsized":"2024-04-06T12:12:08.224078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:19:38.810657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256643+0000","last_change":"2024-04-06T12:12:08.250454+0000","last_active":"2024-04-06T12:12:08.256643+0000","last_peered":"2024-04-06T12:12:08.256643+0000","last_clean":"2024-04-06T12:12:08.256643+0000","last_became_active":"2024-04-06T12:12:08.250052+0000","last_became_peered":"2024-04-06T12:12:08.250052+0000","last_unstale":"2024-04-06T12:12:08.256643+0000","last_undegraded":"2024-04-06T12:12:08.256643+0000","last_fullsized":"2024-04-06T12:12:08.256643+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:11:04.976956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223382+0000","last_change":"2024-04-06T12:12:03.799361+0000","last_active":"2024-04-06T12:12:08.223382+0000","last_peered":"2024-04-06T12:12:08.223382+0000","last_clean":"2024-04-06T12:12:08.223382+0000","last_became_active":"2024-04-06T12:12:03.798981+0000","last_became_peered":"2024-04-06T12:12:03.798981+0000","last_unstale":"2024-04-06T12:12:08.223382+0000","last_undegraded":"2024-04-06T12:12:08.223382+0000","last_fullsized":"2024-04-06T12:12:08.223382+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:11.635313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224104+0000","last_change":"2024-04-06T12:12:04.223535+0000","last_active":"2024-04-06T12:12:08.224104+0000","last_peered":"2024-04-06T12:12:08.224104+0000","last_clean":"2024-04-06T12:12:08.224104+0000","last_became_active":"2024-04-06T12:12:04.217966+0000","last_became_peered":"2024-04-06T12:12:04.217966+0000","last_unstale":"2024-04-06T12:12:08.224104+0000","last_undegraded":"2024-04-06T12:12:08.224104+0000","last_fullsized":"2024-04-06T12:12:08.224104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:15.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256634+0000","last_change":"2024-04-06T12:12:08.252578+0000","last_active":"2024-04-06T12:12:08.256634+0000","last_peered":"2024-04-06T12:12:08.256634+0000","last_clean":"2024-04-06T12:12:08.256634+0000","last_became_active":"2024-04-06T12:12:08.252149+0000","last_became_peered":"2024-04-06T12:12:08.252149+0000","last_unstale":"2024-04-06T12:12:08.256634+0000","last_undegraded":"2024-04-06T12:12:08.256634+0000","last_fullsized":"2024-04-06T12:12:08.256634+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:06.085793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.226746+0000","last_change":"2024-04-06T12:12:03.799994+0000","last_active":"2024-04-06T12:12:08.226746+0000","last_peered":"2024-04-06T12:12:08.226746+0000","last_clean":"2024-04-06T12:12:08.226746+0000","last_became_active":"2024-04-06T12:12:03.799663+0000","last_became_peered":"2024-04-06T12:12:03.799663+0000","last_unstale":"2024-04-06T12:12:08.226746+0000","last_undegraded":"2024-04-06T12:12:08.226746+0000","last_fullsized":"2024-04-06T12:12:08.226746+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:08:55.770454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223501+0000","last_change":"2024-04-06T12:12:04.223425+0000","last_active":"2024-04-06T12:12:08.223501+0000","last_peered":"2024-04-06T12:12:08.223501+0000","last_clean":"2024-04-06T12:12:08.223501+0000","last_became_active":"2024-04-06T12:12:04.223101+0000","last_became_peered":"2024-04-06T12:12:04.223101+0000","last_unstale":"2024-04-06T12:12:08.223501+0000","last_undegraded":"2024-04-06T12:12:08.223501+0000","last_fullsized":"2024-04-06T12:12:08.223501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.250437+0000","last_change":"2024-04-06T12:12:08.249078+0000","last_active":"2024-04-06T12:12:08.250437+0000","last_peered":"2024-04-06T12:12:08.250437+0000","last_clean":"2024-04-06T12:12:08.250437+0000","last_became_active":"2024-04-06T12:12:08.248512+0000","last_became_peered":"2024-04-06T12:12:08.248512+0000","last_unstale":"2024-04-06T12:12:08.250437+0000","last_undegraded":"2024-04-06T12:12:08.250437+0000","last_fullsized":"2024-04-06T12:12:08.250437+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:24:33.553367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.230408+0000","last_change":"2024-04-06T12:12:03.793832+0000","last_active":"2024-04-06T12:12:08.230408+0000","last_peered":"2024-04-06T12:12:08.230408+0000","last_clean":"2024-04-06T12:12:08.230408+0000","last_became_active":"2024-04-06T12:12:03.793518+0000","last_became_peered":"2024-04-06T12:12:03.793518+0000","last_unstale":"2024-04-06T12:12:08.230408+0000","last_undegraded":"2024-04-06T12:12:08.230408+0000","last_fullsized":"2024-04-06T12:12:08.230408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:37:27.257815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224591+0000","last_change":"2024-04-06T12:12:04.224260+0000","last_active":"2024-04-06T12:12:08.224591+0000","last_peered":"2024-04-06T12:12:08.224591+0000","last_clean":"2024-04-06T12:12:08.224591+0000","last_became_active":"2024-04-06T12:12:04.223968+0000","last_became_peered":"2024-04-06T12:12:04.223968+0000","last_unstale":"2024-04-06T12:12:08.224591+0000","last_undegraded":"2024-04-06T12:12:08.224591+0000","last_fullsized":"2024-04-06T12:12:08.224591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.245727+0000","last_change":"2024-04-06T12:12:08.242094+0000","last_active":"2024-04-06T12:12:08.245727+0000","last_peered":"2024-04-06T12:12:08.245727+0000","last_clean":"2024-04-06T12:12:08.245727+0000","last_became_active":"2024-04-06T12:12:08.241707+0000","last_became_peered":"2024-04-06T12:12:08.241707+0000","last_unstale":"2024-04-06T12:12:08.245727+0000","last_undegraded":"2024-04-06T12:12:08.245727+0000","last_fullsized":"2024-04-06T12:12:08.245727+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:47:51.958478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222772+0000","last_change":"2024-04-06T12:12:03.800106+0000","last_active":"2024-04-06T12:12:08.222772+0000","last_peered":"2024-04-06T12:12:08.222772+0000","last_clean":"2024-04-06T12:12:08.222772+0000","last_became_active":"2024-04-06T12:12:03.799843+0000","last_became_peered":"2024-04-06T12:12:03.799843+0000","last_unstale":"2024-04-06T12:12:08.222772+0000","last_undegraded":"2024-04-06T12:12:08.222772+0000","last_fullsized":"2024-04-06T12:12:08.222772+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:26:21.506491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.229502+0000","last_change":"2024-04-06T12:12:04.214371+0000","last_active":"2024-04-06T12:12:08.229502+0000","last_peered":"2024-04-06T12:12:08.229502+0000","last_clean":"2024-04-06T12:12:08.229502+0000","last_became_active":"2024-04-06T12:12:04.214095+0000","last_became_peered":"2024-04-06T12:12:04.214095+0000","last_unstale":"2024-04-06T12:12:08.229502+0000","last_undegraded":"2024-04-06T12:12:08.229502+0000","last_fullsized":"2024-04-06T12:12:08.229502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:45:10.755538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.258734+0000","last_change":"2024-04-06T12:12:08.255356+0000","last_active":"2024-04-06T12:12:08.258734+0000","last_peered":"2024-04-06T12:12:08.258734+0000","last_clean":"2024-04-06T12:12:08.258734+0000","last_became_active":"2024-04-06T12:12:08.254148+0000","last_became_peered":"2024-04-06T12:12:08.254148+0000","last_unstale":"2024-04-06T12:12:08.258734+0000","last_undegraded":"2024-04-06T12:12:08.258734+0000","last_fullsized":"2024-04-06T12:12:08.258734+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:35:54.255608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227821+0000","last_change":"2024-04-06T12:12:03.801770+0000","last_active":"2024-04-06T12:12:08.227821+0000","last_peered":"2024-04-06T12:12:08.227821+0000","last_clean":"2024-04-06T12:12:08.227821+0000","last_became_active":"2024-04-06T12:12:03.801494+0000","last_became_peered":"2024-04-06T12:12:03.801494+0000","last_unstale":"2024-04-06T12:12:08.227821+0000","last_undegraded":"2024-04-06T12:12:08.227821+0000","last_fullsized":"2024-04-06T12:12:08.227821+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:08:55.770454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232799+0000","last_change":"2024-04-06T12:12:04.217734+0000","last_active":"2024-04-06T12:12:08.232799+0000","last_peered":"2024-04-06T12:12:08.232799+0000","last_clean":"2024-04-06T12:12:08.232799+0000","last_became_active":"2024-04-06T12:12:04.216838+0000","last_became_peered":"2024-04-06T12:12:04.216838+0000","last_unstale":"2024-04-06T12:12:08.232799+0000","last_undegraded":"2024-04-06T12:12:08.232799+0000","last_fullsized":"2024-04-06T12:12:08.232799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:05.943405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.258759+0000","last_change":"2024-04-06T12:12:08.255965+0000","last_active":"2024-04-06T12:12:08.258759+0000","last_peered":"2024-04-06T12:12:08.258759+0000","last_clean":"2024-04-06T12:12:08.258759+0000","last_became_active":"2024-04-06T12:12:08.251143+0000","last_became_peered":"2024-04-06T12:12:08.251143+0000","last_unstale":"2024-04-06T12:12:08.258759+0000","last_undegraded":"2024-04-06T12:12:08.258759+0000","last_fullsized":"2024-04-06T12:12:08.258759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:21:01.506046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222665+0000","last_change":"2024-04-06T12:12:04.208976+0000","last_active":"2024-04-06T12:12:08.222665+0000","last_peered":"2024-04-06T12:12:08.222665+0000","last_clean":"2024-04-06T12:12:08.222665+0000","last_became_active":"2024-04-06T12:12:04.208524+0000","last_became_peered":"2024-04-06T12:12:04.208524+0000","last_unstale":"2024-04-06T12:12:08.222665+0000","last_undegraded":"2024-04-06T12:12:08.222665+0000","last_fullsized":"2024-04-06T12:12:08.222665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.226929+0000","last_change":"2024-04-06T12:12:03.796992+0000","last_active":"2024-04-06T12:12:08.226929+0000","last_peered":"2024-04-06T12:12:08.226929+0000","last_clean":"2024-04-06T12:12:08.226929+0000","last_became_active":"2024-04-06T12:12:03.795824+0000","last_became_peered":"2024-04-06T12:12:03.795824+0000","last_unstale":"2024-04-06T12:12:08.226929+0000","last_undegraded":"2024-04-06T12:12:08.226929+0000","last_fullsized":"2024-04-06T12:12:08.226929+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:11.635313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.246589+0000","last_change":"2024-04-06T12:12:08.244008+0000","last_active":"2024-04-06T12:12:08.246589+0000","last_peered":"2024-04-06T12:12:08.246589+0000","last_clean":"2024-04-06T12:12:08.246589+0000","last_became_active":"2024-04-06T12:12:08.242895+0000","last_became_peered":"2024-04-06T12:12:08.242895+0000","last_unstale":"2024-04-06T12:12:08.246589+0000","last_undegraded":"2024-04-06T12:12:08.246589+0000","last_fullsized":"2024-04-06T12:12:08.246589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:54:54.771467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232422+0000","last_change":"2024-04-06T12:12:04.215137+0000","last_active":"2024-04-06T12:12:08.232422+0000","last_peered":"2024-04-06T12:12:08.232422+0000","last_clean":"2024-04-06T12:12:08.232422+0000","last_became_active":"2024-04-06T12:12:04.214872+0000","last_became_peered":"2024-04-06T12:12:04.214872+0000","last_unstale":"2024-04-06T12:12:08.232422+0000","last_undegraded":"2024-04-06T12:12:08.232422+0000","last_fullsized":"2024-04-06T12:12:08.232422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227366+0000","last_change":"2024-04-06T12:12:03.797439+0000","last_active":"2024-04-06T12:12:08.227366+0000","last_peered":"2024-04-06T12:12:08.227366+0000","last_clean":"2024-04-06T12:12:08.227366+0000","last_became_active":"2024-04-06T12:12:03.797176+0000","last_became_peered":"2024-04-06T12:12:03.797176+0000","last_unstale":"2024-04-06T12:12:08.227366+0000","last_undegraded":"2024-04-06T12:12:08.227366+0000","last_fullsized":"2024-04-06T12:12:08.227366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:26:21.506491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223206+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:12:08.223206+0000","last_peered":"2024-04-06T12:12:08.223206+0000","last_clean":"2024-04-06T12:12:08.223206+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:12:08.223206+0000","last_undegraded":"2024-04-06T12:12:08.223206+0000","last_fullsized":"2024-04-06T12:12:08.223206+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.252451+0000","last_change":"2024-04-06T12:12:08.248645+0000","last_active":"2024-04-06T12:12:08.252451+0000","last_peered":"2024-04-06T12:12:08.252451+0000","last_clean":"2024-04-06T12:12:08.252451+0000","last_became_active":"2024-04-06T12:12:08.247420+0000","last_became_peered":"2024-04-06T12:12:08.247420+0000","last_unstale":"2024-04-06T12:12:08.252451+0000","last_undegraded":"2024-04-06T12:12:08.252451+0000","last_fullsized":"2024-04-06T12:12:08.252451+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:31:21.467656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232818+0000","last_change":"2024-04-06T12:12:04.220328+0000","last_active":"2024-04-06T12:12:08.232818+0000","last_peered":"2024-04-06T12:12:08.232818+0000","last_clean":"2024-04-06T12:12:08.232818+0000","last_became_active":"2024-04-06T12:12:04.219948+0000","last_became_peered":"2024-04-06T12:12:04.219948+0000","last_unstale":"2024-04-06T12:12:08.232818+0000","last_undegraded":"2024-04-06T12:12:08.232818+0000","last_fullsized":"2024-04-06T12:12:08.232818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:36:25.380472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223326+0000","last_change":"2024-04-06T12:12:03.800906+0000","last_active":"2024-04-06T12:12:08.223326+0000","last_peered":"2024-04-06T12:12:08.223326+0000","last_clean":"2024-04-06T12:12:08.223326+0000","last_became_active":"2024-04-06T12:12:03.800460+0000","last_became_peered":"2024-04-06T12:12:03.800460+0000","last_unstale":"2024-04-06T12:12:08.223326+0000","last_undegraded":"2024-04-06T12:12:08.223326+0000","last_fullsized":"2024-04-06T12:12:08.223326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:11.635313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.251977+0000","last_change":"2024-04-06T12:12:08.246821+0000","last_active":"2024-04-06T12:12:08.251977+0000","last_peered":"2024-04-06T12:12:08.251977+0000","last_clean":"2024-04-06T12:12:08.251977+0000","last_became_active":"2024-04-06T12:12:08.246154+0000","last_became_peered":"2024-04-06T12:12:08.246154+0000","last_unstale":"2024-04-06T12:12:08.251977+0000","last_undegraded":"2024-04-06T12:12:08.251977+0000","last_fullsized":"2024-04-06T12:12:08.251977+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:47:51.958478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223565+0000","last_change":"2024-04-06T12:12:03.801675+0000","last_active":"2024-04-06T12:12:08.223565+0000","last_peered":"2024-04-06T12:12:08.223565+0000","last_clean":"2024-04-06T12:12:08.223565+0000","last_became_active":"2024-04-06T12:12:03.800427+0000","last_became_peered":"2024-04-06T12:12:03.800427+0000","last_unstale":"2024-04-06T12:12:08.223565+0000","last_undegraded":"2024-04-06T12:12:08.223565+0000","last_fullsized":"2024-04-06T12:12:08.223565+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:21:42.136599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222894+0000","last_change":"2024-04-06T12:12:04.216377+0000","last_active":"2024-04-06T12:12:08.222894+0000","last_peered":"2024-04-06T12:12:08.222894+0000","last_clean":"2024-04-06T12:12:08.222894+0000","last_became_active":"2024-04-06T12:12:04.216084+0000","last_became_peered":"2024-04-06T12:12:04.216084+0000","last_unstale":"2024-04-06T12:12:08.222894+0000","last_undegraded":"2024-04-06T12:12:08.222894+0000","last_fullsized":"2024-04-06T12:12:08.222894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256642+0000","last_change":"2024-04-06T12:12:08.250308+0000","last_active":"2024-04-06T12:12:08.256642+0000","last_peered":"2024-04-06T12:12:08.256642+0000","last_clean":"2024-04-06T12:12:08.256642+0000","last_became_active":"2024-04-06T12:12:08.249904+0000","last_became_peered":"2024-04-06T12:12:08.249904+0000","last_unstale":"2024-04-06T12:12:08.256642+0000","last_undegraded":"2024-04-06T12:12:08.256642+0000","last_fullsized":"2024-04-06T12:12:08.256642+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:35:23.088849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222894+0000","last_change":"2024-04-06T12:12:03.803297+0000","last_active":"2024-04-06T12:12:08.222894+0000","last_peered":"2024-04-06T12:12:08.222894+0000","last_clean":"2024-04-06T12:12:08.222894+0000","last_became_active":"2024-04-06T12:12:03.803053+0000","last_became_peered":"2024-04-06T12:12:03.803053+0000","last_unstale":"2024-04-06T12:12:08.222894+0000","last_undegraded":"2024-04-06T12:12:08.222894+0000","last_fullsized":"2024-04-06T12:12:08.222894+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:21:42.136599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234446+0000","last_change":"2024-04-06T12:12:04.225056+0000","last_active":"2024-04-06T12:12:08.234446+0000","last_peered":"2024-04-06T12:12:08.234446+0000","last_clean":"2024-04-06T12:12:08.234446+0000","last_became_active":"2024-04-06T12:12:04.224635+0000","last_became_peered":"2024-04-06T12:12:04.224635+0000","last_unstale":"2024-04-06T12:12:08.234446+0000","last_undegraded":"2024-04-06T12:12:08.234446+0000","last_fullsized":"2024-04-06T12:12:08.234446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247224+0000","last_change":"2024-04-06T12:12:08.244844+0000","last_active":"2024-04-06T12:12:08.247224+0000","last_peered":"2024-04-06T12:12:08.247224+0000","last_clean":"2024-04-06T12:12:08.247224+0000","last_became_active":"2024-04-06T12:12:08.244422+0000","last_became_peered":"2024-04-06T12:12:08.244422+0000","last_unstale":"2024-04-06T12:12:08.247224+0000","last_undegraded":"2024-04-06T12:12:08.247224+0000","last_fullsized":"2024-04-06T12:12:08.247224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:54:54.771467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223933+0000","last_change":"2024-04-06T12:12:03.800460+0000","last_active":"2024-04-06T12:12:08.223933+0000","last_peered":"2024-04-06T12:12:08.223933+0000","last_clean":"2024-04-06T12:12:08.223933+0000","last_became_active":"2024-04-06T12:12:03.800183+0000","last_became_peered":"2024-04-06T12:12:03.800183+0000","last_unstale":"2024-04-06T12:12:08.223933+0000","last_undegraded":"2024-04-06T12:12:08.223933+0000","last_fullsized":"2024-04-06T12:12:08.223933+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:40:20.784453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.235166+0000","last_change":"2024-04-06T12:12:04.209360+0000","last_active":"2024-04-06T12:12:08.235166+0000","last_peered":"2024-04-06T12:12:08.235166+0000","last_clean":"2024-04-06T12:12:08.235166+0000","last_became_active":"2024-04-06T12:12:04.209105+0000","last_became_peered":"2024-04-06T12:12:04.209105+0000","last_unstale":"2024-04-06T12:12:08.235166+0000","last_undegraded":"2024-04-06T12:12:08.235166+0000","last_fullsized":"2024-04-06T12:12:08.235166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.252713+0000","last_change":"2024-04-06T12:12:08.249325+0000","last_active":"2024-04-06T12:12:08.252713+0000","last_peered":"2024-04-06T12:12:08.252713+0000","last_clean":"2024-04-06T12:12:08.252713+0000","last_became_active":"2024-04-06T12:12:08.248776+0000","last_became_peered":"2024-04-06T12:12:08.248776+0000","last_unstale":"2024-04-06T12:12:08.252713+0000","last_undegraded":"2024-04-06T12:12:08.252713+0000","last_fullsized":"2024-04-06T12:12:08.252713+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:31:21.467656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223337+0000","last_change":"2024-04-06T12:12:03.797556+0000","last_active":"2024-04-06T12:12:08.223337+0000","last_peered":"2024-04-06T12:12:08.223337+0000","last_clean":"2024-04-06T12:12:08.223337+0000","last_became_active":"2024-04-06T12:12:03.797288+0000","last_became_peered":"2024-04-06T12:12:03.797288+0000","last_unstale":"2024-04-06T12:12:08.223337+0000","last_undegraded":"2024-04-06T12:12:08.223337+0000","last_fullsized":"2024-04-06T12:12:08.223337+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:15.699305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.231368+0000","last_change":"2024-04-06T12:12:04.219859+0000","last_active":"2024-04-06T12:12:08.231368+0000","last_peered":"2024-04-06T12:12:08.231368+0000","last_clean":"2024-04-06T12:12:08.231368+0000","last_became_active":"2024-04-06T12:12:04.219470+0000","last_became_peered":"2024-04-06T12:12:04.219470+0000","last_unstale":"2024-04-06T12:12:08.231368+0000","last_undegraded":"2024-04-06T12:12:08.231368+0000","last_fullsized":"2024-04-06T12:12:08.231368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:16.704661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.258673+0000","last_change":"2024-04-06T12:12:08.254506+0000","last_active":"2024-04-06T12:12:08.258673+0000","last_peered":"2024-04-06T12:12:08.258673+0000","last_clean":"2024-04-06T12:12:08.258673+0000","last_became_active":"2024-04-06T12:12:08.250366+0000","last_became_peered":"2024-04-06T12:12:08.250366+0000","last_unstale":"2024-04-06T12:12:08.258673+0000","last_undegraded":"2024-04-06T12:12:08.258673+0000","last_fullsized":"2024-04-06T12:12:08.258673+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:29.953853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222644+0000","last_change":"2024-04-06T12:12:03.801858+0000","last_active":"2024-04-06T12:12:08.222644+0000","last_peered":"2024-04-06T12:12:08.222644+0000","last_clean":"2024-04-06T12:12:08.222644+0000","last_became_active":"2024-04-06T12:12:03.801643+0000","last_became_peered":"2024-04-06T12:12:03.801643+0000","last_unstale":"2024-04-06T12:12:08.222644+0000","last_undegraded":"2024-04-06T12:12:08.222644+0000","last_fullsized":"2024-04-06T12:12:08.222644+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:08:55.770454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232963+0000","last_change":"2024-04-06T12:12:04.213199+0000","last_active":"2024-04-06T12:12:08.232963+0000","last_peered":"2024-04-06T12:12:08.232963+0000","last_clean":"2024-04-06T12:12:08.232963+0000","last_became_active":"2024-04-06T12:12:04.212685+0000","last_became_peered":"2024-04-06T12:12:04.212685+0000","last_unstale":"2024-04-06T12:12:08.232963+0000","last_undegraded":"2024-04-06T12:12:08.232963+0000","last_fullsized":"2024-04-06T12:12:08.232963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:45:10.755538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.250618+0000","last_change":"2024-04-06T12:12:08.249234+0000","last_active":"2024-04-06T12:12:08.250618+0000","last_peered":"2024-04-06T12:12:08.250618+0000","last_clean":"2024-04-06T12:12:08.250618+0000","last_became_active":"2024-04-06T12:12:08.248705+0000","last_became_peered":"2024-04-06T12:12:08.248705+0000","last_unstale":"2024-04-06T12:12:08.250618+0000","last_undegraded":"2024-04-06T12:12:08.250618+0000","last_fullsized":"2024-04-06T12:12:08.250618+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:58:47.243168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222776+0000","last_change":"2024-04-06T12:12:03.798517+0000","last_active":"2024-04-06T12:12:08.222776+0000","last_peered":"2024-04-06T12:12:08.222776+0000","last_clean":"2024-04-06T12:12:08.222776+0000","last_became_active":"2024-04-06T12:12:03.798118+0000","last_became_peered":"2024-04-06T12:12:03.798118+0000","last_unstale":"2024-04-06T12:12:08.222776+0000","last_undegraded":"2024-04-06T12:12:08.222776+0000","last_fullsized":"2024-04-06T12:12:08.222776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:15.699305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232781+0000","last_change":"2024-04-06T12:12:04.216405+0000","last_active":"2024-04-06T12:12:08.232781+0000","last_peered":"2024-04-06T12:12:08.232781+0000","last_clean":"2024-04-06T12:12:08.232781+0000","last_became_active":"2024-04-06T12:12:04.216059+0000","last_became_peered":"2024-04-06T12:12:04.216059+0000","last_unstale":"2024-04-06T12:12:08.232781+0000","last_undegraded":"2024-04-06T12:12:08.232781+0000","last_fullsized":"2024-04-06T12:12:08.232781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.258081+0000","last_change":"2024-04-06T12:12:08.253652+0000","last_active":"2024-04-06T12:12:08.258081+0000","last_peered":"2024-04-06T12:12:08.258081+0000","last_clean":"2024-04-06T12:12:08.258081+0000","last_became_active":"2024-04-06T12:12:08.253321+0000","last_became_peered":"2024-04-06T12:12:08.253321+0000","last_unstale":"2024-04-06T12:12:08.258081+0000","last_undegraded":"2024-04-06T12:12:08.258081+0000","last_fullsized":"2024-04-06T12:12:08.258081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:31:50.315335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224175+0000","last_change":"2024-04-06T12:12:03.798088+0000","last_active":"2024-04-06T12:12:08.224175+0000","last_peered":"2024-04-06T12:12:08.224175+0000","last_clean":"2024-04-06T12:12:08.224175+0000","last_became_active":"2024-04-06T12:12:03.797844+0000","last_became_peered":"2024-04-06T12:12:03.797844+0000","last_unstale":"2024-04-06T12:12:08.224175+0000","last_undegraded":"2024-04-06T12:12:08.224175+0000","last_fullsized":"2024-04-06T12:12:08.224175+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:09.507432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223640+0000","last_change":"2024-04-06T12:12:04.224049+0000","last_active":"2024-04-06T12:12:08.223640+0000","last_peered":"2024-04-06T12:12:08.223640+0000","last_clean":"2024-04-06T12:12:08.223640+0000","last_became_active":"2024-04-06T12:12:04.223437+0000","last_became_peered":"2024-04-06T12:12:04.223437+0000","last_unstale":"2024-04-06T12:12:08.223640+0000","last_undegraded":"2024-04-06T12:12:08.223640+0000","last_fullsized":"2024-04-06T12:12:08.223640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:36:25.380472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247262+0000","last_change":"2024-04-06T12:12:08.244925+0000","last_active":"2024-04-06T12:12:08.247262+0000","last_peered":"2024-04-06T12:12:08.247262+0000","last_clean":"2024-04-06T12:12:08.247262+0000","last_became_active":"2024-04-06T12:12:08.244571+0000","last_became_peered":"2024-04-06T12:12:08.244571+0000","last_unstale":"2024-04-06T12:12:08.247262+0000","last_undegraded":"2024-04-06T12:12:08.247262+0000","last_fullsized":"2024-04-06T12:12:08.247262+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:31:50.315335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227747+0000","last_change":"2024-04-06T12:12:03.800145+0000","last_active":"2024-04-06T12:12:08.227747+0000","last_peered":"2024-04-06T12:12:08.227747+0000","last_clean":"2024-04-06T12:12:08.227747+0000","last_became_active":"2024-04-06T12:12:03.799406+0000","last_became_peered":"2024-04-06T12:12:03.799406+0000","last_unstale":"2024-04-06T12:12:08.227747+0000","last_undegraded":"2024-04-06T12:12:08.227747+0000","last_fullsized":"2024-04-06T12:12:08.227747+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:40:20.784453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224246+0000","last_change":"2024-04-06T12:12:04.226479+0000","last_active":"2024-04-06T12:12:08.224246+0000","last_peered":"2024-04-06T12:12:08.224246+0000","last_clean":"2024-04-06T12:12:08.224246+0000","last_became_active":"2024-04-06T12:12:04.222457+0000","last_became_peered":"2024-04-06T12:12:04.222457+0000","last_unstale":"2024-04-06T12:12:08.224246+0000","last_undegraded":"2024-04-06T12:12:08.224246+0000","last_fullsized":"2024-04-06T12:12:08.224246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:41:20.002927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.252006+0000","last_change":"2024-04-06T12:12:08.247112+0000","last_active":"2024-04-06T12:12:08.252006+0000","last_peered":"2024-04-06T12:12:08.252006+0000","last_clean":"2024-04-06T12:12:08.252006+0000","last_became_active":"2024-04-06T12:12:08.246705+0000","last_became_peered":"2024-04-06T12:12:08.246705+0000","last_unstale":"2024-04-06T12:12:08.252006+0000","last_undegraded":"2024-04-06T12:12:08.252006+0000","last_fullsized":"2024-04-06T12:12:08.252006+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:29.953853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222999+0000","last_change":"2024-04-06T12:12:03.799395+0000","last_active":"2024-04-06T12:12:08.222999+0000","last_peered":"2024-04-06T12:12:08.222999+0000","last_clean":"2024-04-06T12:12:08.222999+0000","last_became_active":"2024-04-06T12:12:03.799116+0000","last_became_peered":"2024-04-06T12:12:03.799116+0000","last_unstale":"2024-04-06T12:12:08.222999+0000","last_undegraded":"2024-04-06T12:12:08.222999+0000","last_fullsized":"2024-04-06T12:12:08.222999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:09.507432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227629+0000","last_change":"2024-04-06T12:12:04.221238+0000","last_active":"2024-04-06T12:12:08.227629+0000","last_peered":"2024-04-06T12:12:08.227629+0000","last_clean":"2024-04-06T12:12:08.227629+0000","last_became_active":"2024-04-06T12:12:04.220606+0000","last_became_peered":"2024-04-06T12:12:04.220606+0000","last_unstale":"2024-04-06T12:12:08.227629+0000","last_undegraded":"2024-04-06T12:12:08.227629+0000","last_fullsized":"2024-04-06T12:12:08.227629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.246867+0000","last_change":"2024-04-06T12:12:08.241508+0000","last_active":"2024-04-06T12:12:08.246867+0000","last_peered":"2024-04-06T12:12:08.246867+0000","last_clean":"2024-04-06T12:12:08.246867+0000","last_became_active":"2024-04-06T12:12:08.241184+0000","last_became_peered":"2024-04-06T12:12:08.241184+0000","last_unstale":"2024-04-06T12:12:08.246867+0000","last_undegraded":"2024-04-06T12:12:08.246867+0000","last_fullsized":"2024-04-06T12:12:08.246867+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:35:54.255608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228578+0000","last_change":"2024-04-06T12:12:03.795282+0000","last_active":"2024-04-06T12:12:08.228578+0000","last_peered":"2024-04-06T12:12:08.228578+0000","last_clean":"2024-04-06T12:12:08.228578+0000","last_became_active":"2024-04-06T12:12:03.794928+0000","last_became_peered":"2024-04-06T12:12:03.794928+0000","last_unstale":"2024-04-06T12:12:08.228578+0000","last_undegraded":"2024-04-06T12:12:08.228578+0000","last_fullsized":"2024-04-06T12:12:08.228578+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:00:31.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.225091+0000","last_change":"2024-04-06T12:12:04.224265+0000","last_active":"2024-04-06T12:12:08.225091+0000","last_peered":"2024-04-06T12:12:08.225091+0000","last_clean":"2024-04-06T12:12:08.225091+0000","last_became_active":"2024-04-06T12:12:04.223892+0000","last_became_peered":"2024-04-06T12:12:04.223892+0000","last_unstale":"2024-04-06T12:12:08.225091+0000","last_undegraded":"2024-04-06T12:12:08.225091+0000","last_fullsized":"2024-04-06T12:12:08.225091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:21:16.167775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223213+0000","last_change":"2024-04-06T12:12:04.223037+0000","last_active":"2024-04-06T12:12:08.223213+0000","last_peered":"2024-04-06T12:12:08.223213+0000","last_clean":"2024-04-06T12:12:08.223213+0000","last_became_active":"2024-04-06T12:12:04.218051+0000","last_became_peered":"2024-04-06T12:12:04.218051+0000","last_unstale":"2024-04-06T12:12:08.223213+0000","last_undegraded":"2024-04-06T12:12:08.223213+0000","last_fullsized":"2024-04-06T12:12:08.223213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:22:59.062918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227020+0000","last_change":"2024-04-06T12:12:04.214042+0000","last_active":"2024-04-06T12:12:08.227020+0000","last_peered":"2024-04-06T12:12:08.227020+0000","last_clean":"2024-04-06T12:12:08.227020+0000","last_became_active":"2024-04-06T12:12:04.213777+0000","last_became_peered":"2024-04-06T12:12:04.213777+0000","last_unstale":"2024-04-06T12:12:08.227020+0000","last_undegraded":"2024-04-06T12:12:08.227020+0000","last_fullsized":"2024-04-06T12:12:08.227020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:05.943405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228110+0000","last_change":"2024-04-06T12:12:04.216964+0000","last_active":"2024-04-06T12:12:08.228110+0000","last_peered":"2024-04-06T12:12:08.228110+0000","last_clean":"2024-04-06T12:12:08.228110+0000","last_became_active":"2024-04-06T12:12:04.216692+0000","last_became_peered":"2024-04-06T12:12:04.216692+0000","last_unstale":"2024-04-06T12:12:08.228110+0000","last_undegraded":"2024-04-06T12:12:08.228110+0000","last_fullsized":"2024-04-06T12:12:08.228110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234533+0000","last_change":"2024-04-06T12:12:04.219048+0000","last_active":"2024-04-06T12:12:08.234533+0000","last_peered":"2024-04-06T12:12:08.234533+0000","last_clean":"2024-04-06T12:12:08.234533+0000","last_became_active":"2024-04-06T12:12:04.218624+0000","last_became_peered":"2024-04-06T12:12:04.218624+0000","last_unstale":"2024-04-06T12:12:08.234533+0000","last_undegraded":"2024-04-06T12:12:08.234533+0000","last_fullsized":"2024-04-06T12:12:08.234533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:28:15.608394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.233536+0000","last_change":"2024-04-06T12:12:04.218893+0000","last_active":"2024-04-06T12:12:08.233536+0000","last_peered":"2024-04-06T12:12:08.233536+0000","last_clean":"2024-04-06T12:12:08.233536+0000","last_became_active":"2024-04-06T12:12:04.218526+0000","last_became_peered":"2024-04-06T12:12:04.218526+0000","last_unstale":"2024-04-06T12:12:08.233536+0000","last_undegraded":"2024-04-06T12:12:08.233536+0000","last_fullsized":"2024-04-06T12:12:08.233536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:15.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223037+0000","last_change":"2024-04-06T12:12:04.216794+0000","last_active":"2024-04-06T12:12:08.223037+0000","last_peered":"2024-04-06T12:12:08.223037+0000","last_clean":"2024-04-06T12:12:08.223037+0000","last_became_active":"2024-04-06T12:12:04.216532+0000","last_became_peered":"2024-04-06T12:12:04.216532+0000","last_unstale":"2024-04-06T12:12:08.223037+0000","last_undegraded":"2024-04-06T12:12:08.223037+0000","last_fullsized":"2024-04-06T12:12:08.223037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:15.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224560+0000","last_change":"2024-04-06T12:12:04.229182+0000","last_active":"2024-04-06T12:12:08.224560+0000","last_peered":"2024-04-06T12:12:08.224560+0000","last_clean":"2024-04-06T12:12:08.224560+0000","last_became_active":"2024-04-06T12:12:04.228927+0000","last_became_peered":"2024-04-06T12:12:04.228927+0000","last_unstale":"2024-04-06T12:12:08.224560+0000","last_undegraded":"2024-04-06T12:12:08.224560+0000","last_fullsized":"2024-04-06T12:12:08.224560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234558+0000","last_change":"2024-04-06T12:12:04.217003+0000","last_active":"2024-04-06T12:12:08.234558+0000","last_peered":"2024-04-06T12:12:08.234558+0000","last_clean":"2024-04-06T12:12:08.234558+0000","last_became_active":"2024-04-06T12:12:04.216757+0000","last_became_peered":"2024-04-06T12:12:04.216757+0000","last_unstale":"2024-04-06T12:12:08.234558+0000","last_undegraded":"2024-04-06T12:12:08.234558+0000","last_fullsized":"2024-04-06T12:12:08.234558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:00.258491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227358+0000","last_change":"2024-04-06T12:12:04.216442+0000","last_active":"2024-04-06T12:12:08.227358+0000","last_peered":"2024-04-06T12:12:08.227358+0000","last_clean":"2024-04-06T12:12:08.227358+0000","last_became_active":"2024-04-06T12:12:04.215051+0000","last_became_peered":"2024-04-06T12:12:04.215051+0000","last_unstale":"2024-04-06T12:12:08.227358+0000","last_undegraded":"2024-04-06T12:12:08.227358+0000","last_fullsized":"2024-04-06T12:12:08.227358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:19:38.810657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232330+0000","last_change":"2024-04-06T12:12:04.219526+0000","last_active":"2024-04-06T12:12:08.232330+0000","last_peered":"2024-04-06T12:12:08.232330+0000","last_clean":"2024-04-06T12:12:08.232330+0000","last_became_active":"2024-04-06T12:12:04.219229+0000","last_became_peered":"2024-04-06T12:12:04.219229+0000","last_unstale":"2024-04-06T12:12:08.232330+0000","last_undegraded":"2024-04-06T12:12:08.232330+0000","last_fullsized":"2024-04-06T12:12:08.232330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:22:59.062918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228186+0000","last_change":"2024-04-06T12:12:04.226116+0000","last_active":"2024-04-06T12:12:08.228186+0000","last_peered":"2024-04-06T12:12:08.228186+0000","last_clean":"2024-04-06T12:12:08.228186+0000","last_became_active":"2024-04-06T12:12:04.225820+0000","last_became_peered":"2024-04-06T12:12:04.225820+0000","last_unstale":"2024-04-06T12:12:08.228186+0000","last_undegraded":"2024-04-06T12:12:08.228186+0000","last_fullsized":"2024-04-06T12:12:08.228186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:41:20.002927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234612+0000","last_change":"2024-04-06T12:12:04.228504+0000","last_active":"2024-04-06T12:12:08.234612+0000","last_peered":"2024-04-06T12:12:08.234612+0000","last_clean":"2024-04-06T12:12:08.234612+0000","last_became_active":"2024-04-06T12:12:04.228232+0000","last_became_peered":"2024-04-06T12:12:04.228232+0000","last_unstale":"2024-04-06T12:12:08.234612+0000","last_undegraded":"2024-04-06T12:12:08.234612+0000","last_fullsized":"2024-04-06T12:12:08.234612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:22:59.062918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223649+0000","last_change":"2024-04-06T12:12:04.230394+0000","last_active":"2024-04-06T12:12:08.223649+0000","last_peered":"2024-04-06T12:12:08.223649+0000","last_clean":"2024-04-06T12:12:08.223649+0000","last_became_active":"2024-04-06T12:12:04.230065+0000","last_became_peered":"2024-04-06T12:12:04.230065+0000","last_unstale":"2024-04-06T12:12:08.223649+0000","last_undegraded":"2024-04-06T12:12:08.223649+0000","last_fullsized":"2024-04-06T12:12:08.223649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4594,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":851968,"data_stored":690730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":262144,"data_stored":102557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":98921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":856064,"data_stored":689767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":98761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":103576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":99906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":98851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:12:27.654 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:12:27.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.811+0000 7fe7e3948700 1 Processor -- start 2024-04-06T12:12:27.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.811+0000 7fe7e3948700 1 -- start start 2024-04-06T12:12:27.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.812+0000 7fe7e3948700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe7dc15d7b0 0x7fe7dc0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:27.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.812+0000 7fe7e3948700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7dc06a730 0x7fe7dc06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:27.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.812+0000 7fe7e3948700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7dc0b3990 con 0x7fe7dc15ddf0 2024-04-06T12:12:27.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.812+0000 7fe7e3948700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7dc06b0f0 con 0x7fe7dc15d7b0 2024-04-06T12:12:27.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.812+0000 7fe7e3948700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7dc0b3060 con 0x7fe7dc06a730 2024-04-06T12:12:27.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.812+0000 7fe7e1ee5700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe7dc15ddf0 0x7fe7dc15e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55118/0 (socket says 172.21.15.189:55118) 2024-04-06T12:12:27.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.812+0000 7fe7e1ee5700 1 -- 172.21.15.189:0/3319402310 learned_addr learned my addr 172.21.15.189:0/3319402310 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:27.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.812+0000 7fe7e16e4700 1 --2- 172.21.15.189:0/3319402310 >> v2:172.21.15.192:3300/0 conn(0x7fe7dc15d7b0 0x7fe7dc0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:27.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.812+0000 7fe7e0ee3700 1 --2- 172.21.15.189:0/3319402310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7dc06a730 0x7fe7dc06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:27.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.812+0000 7fe7d3fff700 1 -- 172.21.15.189:0/3319402310 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163942447 0 0) 0x7fe7dc0b3990 con 0x7fe7dc15ddf0 2024-04-06T12:12:27.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.812+0000 7fe7d3fff700 1 -- 172.21.15.189:0/3319402310 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c80035b0 con 0x7fe7dc15ddf0 2024-04-06T12:12:27.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.812+0000 7fe7e0ee3700 1 -- 172.21.15.189:0/3319402310 >> v2:172.21.15.192:3300/0 conn(0x7fe7dc15d7b0 msgr2=0x7fe7dc0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:27.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.812+0000 7fe7e0ee3700 1 --2- 172.21.15.189:0/3319402310 >> v2:172.21.15.192:3300/0 conn(0x7fe7dc15d7b0 0x7fe7dc0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.812+0000 7fe7e0ee3700 1 -- 172.21.15.189:0/3319402310 >> v1:172.21.15.189:6789/0 conn(0x7fe7dc15ddf0 legacy=0x7fe7dc15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:27.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.812+0000 7fe7e0ee3700 1 -- 172.21.15.189:0/3319402310 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7dc0b3340 con 0x7fe7dc06a730 2024-04-06T12:12:27.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.812+0000 7fe7e16e4700 1 --2- 172.21.15.189:0/3319402310 >> v2:172.21.15.192:3300/0 conn(0x7fe7dc15d7b0 0x7fe7dc0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:27.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.813+0000 7fe7e0ee3700 1 --2- 172.21.15.189:0/3319402310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7dc06a730 0x7fe7dc06abb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc009840 tx=0x7fe7cc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a43f626469ae5292 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:27.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.813+0000 7fe7d3fff700 1 -- 172.21.15.189:0/3319402310 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc011830 con 0x7fe7dc06a730 2024-04-06T12:12:27.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.813+0000 7fe7d3fff700 1 -- 172.21.15.189:0/3319402310 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc011990 con 0x7fe7dc06a730 2024-04-06T12:12:27.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.813+0000 7fe7d3fff700 1 -- 172.21.15.189:0/3319402310 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc011d00 con 0x7fe7dc06a730 2024-04-06T12:12:27.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.813+0000 7fe7e3948700 1 -- 172.21.15.189:0/3319402310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7dc06a730 msgr2=0x7fe7dc06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:27.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.813+0000 7fe7e3948700 1 --2- 172.21.15.189:0/3319402310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7dc06a730 0x7fe7dc06abb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc009840 tx=0x7fe7cc009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:27.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.813+0000 7fe7e3948700 1 -- 172.21.15.189:0/3319402310 shutdown_connections 2024-04-06T12:12:27.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.813+0000 7fe7e3948700 1 --2- 172.21.15.189:0/3319402310 >> v2:172.21.15.192:3300/0 conn(0x7fe7dc15d7b0 0x7fe7dc0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.813+0000 7fe7e3948700 1 --2- 172.21.15.189:0/3319402310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7dc06a730 0x7fe7dc06abb0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.813+0000 7fe7e3948700 1 -- 172.21.15.189:0/3319402310 >> 172.21.15.189:0/3319402310 conn(0x7fe7dc0d00b0 msgr2=0x7fe7dc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:27.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.814+0000 7fe7e3948700 1 -- 172.21.15.189:0/3319402310 shutdown_connections 2024-04-06T12:12:27.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.814+0000 7fe7e3948700 1 -- 172.21.15.189:0/3319402310 wait complete. 2024-04-06T12:12:27.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.815+0000 7fe7e3948700 1 Processor -- start 2024-04-06T12:12:27.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.815+0000 7fe7e3948700 1 -- start start 2024-04-06T12:12:27.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.815+0000 7fe7e3948700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7dc06a730 0x7fe7dc0ce110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:27.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.815+0000 7fe7e3948700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7dc15d7b0 0x7fe7dc153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:27.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.815+0000 7fe7e3948700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7dc15ddf0 0x7fe7dc0cc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:27.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.815+0000 7fe7e3948700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7dc0cdc80 con 0x7fe7dc15d7b0 2024-04-06T12:12:27.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.815+0000 7fe7e3948700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7dc1544e0 con 0x7fe7dc06a730 2024-04-06T12:12:27.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.815+0000 7fe7e3948700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7dc0cd070 con 0x7fe7dc15ddf0 2024-04-06T12:12:27.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.816+0000 7fe7e0ee3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7dc15d7b0 0x7fe7dc153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:27.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.816+0000 7fe7e0ee3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7dc15d7b0 0x7fe7dc153fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42778/0 (socket says 172.21.15.189:42778) 2024-04-06T12:12:27.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.816+0000 7fe7e0ee3700 1 -- 172.21.15.189:0/878010794 learned_addr learned my addr 172.21.15.189:0/878010794 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:27.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.816+0000 7fe7e16e4700 1 --2- 172.21.15.189:0/878010794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7dc06a730 0x7fe7dc0ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:27.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.816+0000 7fe7e1ee5700 1 --2- 172.21.15.189:0/878010794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7dc15ddf0 0x7fe7dc0cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:27.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.816+0000 7fe7e0ee3700 1 -- 172.21.15.189:0/878010794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7dc15ddf0 msgr2=0x7fe7dc0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:27.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.816+0000 7fe7e0ee3700 1 --2- 172.21.15.189:0/878010794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7dc15ddf0 0x7fe7dc0cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.816+0000 7fe7e0ee3700 1 -- 172.21.15.189:0/878010794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7dc06a730 msgr2=0x7fe7dc0ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:27.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.816+0000 7fe7e0ee3700 1 --2- 172.21.15.189:0/878010794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7dc06a730 0x7fe7dc0ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:27.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.816+0000 7fe7e0ee3700 1 -- 172.21.15.189:0/878010794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d80094a0 con 0x7fe7dc15d7b0 2024-04-06T12:12:27.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.816+0000 7fe7e0ee3700 1 --2- 172.21.15.189:0/878010794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7dc15d7b0 0x7fe7dc153fa0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0090b0 tx=0x7fe7cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:27.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.816+0000 7fe7d1ffb700 1 -- 172.21.15.189:0/878010794 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc004550 con 0x7fe7dc15d7b0 2024-04-06T12:12:27.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.816+0000 7fe7d1ffb700 1 -- 172.21.15.189:0/878010794 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc01f070 con 0x7fe7dc15d7b0 2024-04-06T12:12:27.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.816+0000 7fe7e3948700 1 -- 172.21.15.189:0/878010794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7cc0094a0 con 0x7fe7dc15d7b0 2024-04-06T12:12:27.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.816+0000 7fe7d1ffb700 1 -- 172.21.15.189:0/878010794 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc0038a0 con 0x7fe7dc15d7b0 2024-04-06T12:12:27.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.816+0000 7fe7e3948700 1 -- 172.21.15.189:0/878010794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7dc0cd630 con 0x7fe7dc15d7b0 2024-04-06T12:12:27.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.817+0000 7fe7e3948700 1 -- 172.21.15.189:0/878010794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7dc067bb0 con 0x7fe7dc15d7b0 2024-04-06T12:12:27.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.818+0000 7fe7d1ffb700 1 -- 172.21.15.189:0/878010794 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe7cc003a00 con 0x7fe7dc15d7b0 2024-04-06T12:12:27.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.818+0000 7fe7d1ffb700 1 --2- 172.21.15.189:0/878010794 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe7c006ac90 0x7fe7c006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:27.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.818+0000 7fe7d1ffb700 1 -- 172.21.15.189:0/878010794 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe7cc021070 con 0x7fe7dc15d7b0 2024-04-06T12:12:27.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.818+0000 7fe7e16e4700 1 --2- 172.21.15.189:0/878010794 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe7c006ac90 0x7fe7c006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:27.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.819+0000 7fe7e16e4700 1 --2- 172.21.15.189:0/878010794 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe7c006ac90 0x7fe7c006d150 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8009d70 tx=0x7fe7d8009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:27.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:27.823+0000 7fe7d1ffb700 1 -- 172.21.15.189:0/878010794 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe7cc055dd0 con 0x7fe7dc15d7b0 2024-04-06T12:12:28.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.010+0000 7fe7e3948700 1 -- 172.21.15.189:0/878010794 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe7dc15eb40 con 0x7fe7c006ac90 2024-04-06T12:12:28.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.019+0000 7fe7d1ffb700 1 -- 172.21.15.189:0/878010794 <== mgr.4100 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345300 (secure 0 0 0) 0x7fe7d80099c0 con 0x7fe7c006ac90 2024-04-06T12:12:28.023 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:28.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.023+0000 7fe7e3948700 1 -- 172.21.15.189:0/878010794 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe7c006ac90 msgr2=0x7fe7c006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:28.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.023+0000 7fe7e3948700 1 --2- 172.21.15.189:0/878010794 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe7c006ac90 0x7fe7c006d150 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8009d70 tx=0x7fe7d8009210 comp rx=0 tx=0).stop 2024-04-06T12:12:28.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.023+0000 7fe7e3948700 1 -- 172.21.15.189:0/878010794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7dc15d7b0 msgr2=0x7fe7dc153fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:28.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.023+0000 7fe7e3948700 1 --2- 172.21.15.189:0/878010794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7dc15d7b0 0x7fe7dc153fa0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0090b0 tx=0x7fe7cc004750 comp rx=0 tx=0).stop 2024-04-06T12:12:28.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.023+0000 7fe7e3948700 1 -- 172.21.15.189:0/878010794 shutdown_connections 2024-04-06T12:12:28.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.023+0000 7fe7e3948700 1 --2- 172.21.15.189:0/878010794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7dc15ddf0 0x7fe7dc0cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.023+0000 7fe7e3948700 1 --2- 172.21.15.189:0/878010794 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe7c006ac90 0x7fe7c006d150 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.023+0000 7fe7e3948700 1 --2- 172.21.15.189:0/878010794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7dc06a730 0x7fe7dc0ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.023+0000 7fe7e3948700 1 --2- 172.21.15.189:0/878010794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7dc15d7b0 0x7fe7dc153fa0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.023+0000 7fe7e3948700 1 -- 172.21.15.189:0/878010794 >> 172.21.15.189:0/878010794 conn(0x7fe7dc0d00b0 msgr2=0x7fe7dc0adf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:28.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.023+0000 7fe7e3948700 1 -- 172.21.15.189:0/878010794 shutdown_connections 2024-04-06T12:12:28.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.023+0000 7fe7e3948700 1 -- 172.21.15.189:0/878010794 wait complete. 2024-04-06T12:12:28.027 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:12:28.034 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-04-06T12:12:26.570505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594962,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219360,"kb_used_data":3188,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755360,"statfs":{"total":773094113280,"available":772869488640,"internally_reserved":0,"allocated":3264512,"data_stored":1983069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2232,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003462"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224930+0000","last_change":"2024-04-06T12:12:04.226505+0000","last_active":"2024-04-06T12:12:08.224930+0000","last_peered":"2024-04-06T12:12:08.224930+0000","last_clean":"2024-04-06T12:12:08.224930+0000","last_became_active":"2024-04-06T12:12:04.226112+0000","last_became_peered":"2024-04-06T12:12:04.226112+0000","last_unstale":"2024-04-06T12:12:08.224930+0000","last_undegraded":"2024-04-06T12:12:08.224930+0000","last_fullsized":"2024-04-06T12:12:08.224930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:09:38.014527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223002+0000","last_change":"2024-04-06T12:12:04.217545+0000","last_active":"2024-04-06T12:12:08.223002+0000","last_peered":"2024-04-06T12:12:08.223002+0000","last_clean":"2024-04-06T12:12:08.223002+0000","last_became_active":"2024-04-06T12:12:04.217304+0000","last_became_peered":"2024-04-06T12:12:04.217304+0000","last_unstale":"2024-04-06T12:12:08.223002+0000","last_undegraded":"2024-04-06T12:12:08.223002+0000","last_fullsized":"2024-04-06T12:12:08.223002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.231018+0000","last_change":"2024-04-06T12:12:04.209328+0000","last_active":"2024-04-06T12:12:08.231018+0000","last_peered":"2024-04-06T12:12:08.231018+0000","last_clean":"2024-04-06T12:12:08.231018+0000","last_became_active":"2024-04-06T12:12:04.209066+0000","last_became_peered":"2024-04-06T12:12:04.209066+0000","last_unstale":"2024-04-06T12:12:08.231018+0000","last_undegraded":"2024-04-06T12:12:08.231018+0000","last_fullsized":"2024-04-06T12:12:08.231018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:02:53.621993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227488+0000","last_change":"2024-04-06T12:12:04.221440+0000","last_active":"2024-04-06T12:12:08.227488+0000","last_peered":"2024-04-06T12:12:08.227488+0000","last_clean":"2024-04-06T12:12:08.227488+0000","last_became_active":"2024-04-06T12:12:04.220738+0000","last_became_peered":"2024-04-06T12:12:04.220738+0000","last_unstale":"2024-04-06T12:12:08.227488+0000","last_undegraded":"2024-04-06T12:12:08.227488+0000","last_fullsized":"2024-04-06T12:12:08.227488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:43:39.601900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223982+0000","last_change":"2024-04-06T12:12:04.223078+0000","last_active":"2024-04-06T12:12:08.223982+0000","last_peered":"2024-04-06T12:12:08.223982+0000","last_clean":"2024-04-06T12:12:08.223982+0000","last_became_active":"2024-04-06T12:12:04.222831+0000","last_became_peered":"2024-04-06T12:12:04.222831+0000","last_unstale":"2024-04-06T12:12:08.223982+0000","last_undegraded":"2024-04-06T12:12:08.223982+0000","last_fullsized":"2024-04-06T12:12:08.223982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:02:00.219999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227898+0000","last_change":"2024-04-06T12:12:04.215126+0000","last_active":"2024-04-06T12:12:08.227898+0000","last_peered":"2024-04-06T12:12:08.227898+0000","last_clean":"2024-04-06T12:12:08.227898+0000","last_became_active":"2024-04-06T12:12:04.214857+0000","last_became_peered":"2024-04-06T12:12:04.214857+0000","last_unstale":"2024-04-06T12:12:08.227898+0000","last_undegraded":"2024-04-06T12:12:08.227898+0000","last_fullsized":"2024-04-06T12:12:08.227898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:02:00.219999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223337+0000","last_change":"2024-04-06T12:12:04.222961+0000","last_active":"2024-04-06T12:12:08.223337+0000","last_peered":"2024-04-06T12:12:08.223337+0000","last_clean":"2024-04-06T12:12:08.223337+0000","last_became_active":"2024-04-06T12:12:04.222659+0000","last_became_peered":"2024-04-06T12:12:04.222659+0000","last_unstale":"2024-04-06T12:12:08.223337+0000","last_undegraded":"2024-04-06T12:12:08.223337+0000","last_fullsized":"2024-04-06T12:12:08.223337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:55.517965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.233016+0000","last_change":"2024-04-06T12:12:04.222214+0000","last_active":"2024-04-06T12:12:08.233016+0000","last_peered":"2024-04-06T12:12:08.233016+0000","last_clean":"2024-04-06T12:12:08.233016+0000","last_became_active":"2024-04-06T12:12:04.221817+0000","last_became_peered":"2024-04-06T12:12:04.221817+0000","last_unstale":"2024-04-06T12:12:08.233016+0000","last_undegraded":"2024-04-06T12:12:08.233016+0000","last_fullsized":"2024-04-06T12:12:08.233016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224843+0000","last_change":"2024-04-06T12:12:04.224106+0000","last_active":"2024-04-06T12:12:08.224843+0000","last_peered":"2024-04-06T12:12:08.224843+0000","last_clean":"2024-04-06T12:12:08.224843+0000","last_became_active":"2024-04-06T12:12:04.223653+0000","last_became_peered":"2024-04-06T12:12:04.223653+0000","last_unstale":"2024-04-06T12:12:08.224843+0000","last_undegraded":"2024-04-06T12:12:08.224843+0000","last_fullsized":"2024-04-06T12:12:08.224843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:50:27.596378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223873+0000","last_change":"2024-04-06T12:12:04.223892+0000","last_active":"2024-04-06T12:12:08.223873+0000","last_peered":"2024-04-06T12:12:08.223873+0000","last_clean":"2024-04-06T12:12:08.223873+0000","last_became_active":"2024-04-06T12:12:04.223615+0000","last_became_peered":"2024-04-06T12:12:04.223615+0000","last_unstale":"2024-04-06T12:12:08.223873+0000","last_undegraded":"2024-04-06T12:12:08.223873+0000","last_fullsized":"2024-04-06T12:12:08.223873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:02:53.621993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.225163+0000","last_change":"2024-04-06T12:12:04.223861+0000","last_active":"2024-04-06T12:12:08.225163+0000","last_peered":"2024-04-06T12:12:08.225163+0000","last_clean":"2024-04-06T12:12:08.225163+0000","last_became_active":"2024-04-06T12:12:04.223284+0000","last_became_peered":"2024-04-06T12:12:04.223284+0000","last_unstale":"2024-04-06T12:12:08.225163+0000","last_undegraded":"2024-04-06T12:12:08.225163+0000","last_fullsized":"2024-04-06T12:12:08.225163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:00.258491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232662+0000","last_change":"2024-04-06T12:12:04.230879+0000","last_active":"2024-04-06T12:12:08.232662+0000","last_peered":"2024-04-06T12:12:08.232662+0000","last_clean":"2024-04-06T12:12:08.232662+0000","last_became_active":"2024-04-06T12:12:04.230592+0000","last_became_peered":"2024-04-06T12:12:04.230592+0000","last_unstale":"2024-04-06T12:12:08.232662+0000","last_undegraded":"2024-04-06T12:12:08.232662+0000","last_fullsized":"2024-04-06T12:12:08.232662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:10.512788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232539+0000","last_change":"2024-04-06T12:12:04.218704+0000","last_active":"2024-04-06T12:12:08.232539+0000","last_peered":"2024-04-06T12:12:08.232539+0000","last_clean":"2024-04-06T12:12:08.232539+0000","last_became_active":"2024-04-06T12:12:04.218455+0000","last_became_peered":"2024-04-06T12:12:04.218455+0000","last_unstale":"2024-04-06T12:12:08.232539+0000","last_undegraded":"2024-04-06T12:12:08.232539+0000","last_fullsized":"2024-04-06T12:12:08.232539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:21:16.167775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.235340+0000","last_change":"2024-04-06T12:12:04.209351+0000","last_active":"2024-04-06T12:12:08.235340+0000","last_peered":"2024-04-06T12:12:08.235340+0000","last_clean":"2024-04-06T12:12:08.235340+0000","last_became_active":"2024-04-06T12:12:04.209092+0000","last_became_peered":"2024-04-06T12:12:04.209092+0000","last_unstale":"2024-04-06T12:12:08.235340+0000","last_undegraded":"2024-04-06T12:12:08.235340+0000","last_fullsized":"2024-04-06T12:12:08.235340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222602+0000","last_change":"2024-04-06T12:12:04.219642+0000","last_active":"2024-04-06T12:12:08.222602+0000","last_peered":"2024-04-06T12:12:08.222602+0000","last_clean":"2024-04-06T12:12:08.222602+0000","last_became_active":"2024-04-06T12:12:04.219398+0000","last_became_peered":"2024-04-06T12:12:04.219398+0000","last_unstale":"2024-04-06T12:12:08.222602+0000","last_undegraded":"2024-04-06T12:12:08.222602+0000","last_fullsized":"2024-04-06T12:12:08.222602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227279+0000","last_change":"2024-04-06T12:12:04.215338+0000","last_active":"2024-04-06T12:12:08.227279+0000","last_peered":"2024-04-06T12:12:08.227279+0000","last_clean":"2024-04-06T12:12:08.227279+0000","last_became_active":"2024-04-06T12:12:04.215021+0000","last_became_peered":"2024-04-06T12:12:04.215021+0000","last_unstale":"2024-04-06T12:12:08.227279+0000","last_undegraded":"2024-04-06T12:12:08.227279+0000","last_fullsized":"2024-04-06T12:12:08.227279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:09:38.014527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.229099+0000","last_change":"2024-04-06T12:12:04.225406+0000","last_active":"2024-04-06T12:12:08.229099+0000","last_peered":"2024-04-06T12:12:08.229099+0000","last_clean":"2024-04-06T12:12:08.229099+0000","last_became_active":"2024-04-06T12:12:04.224950+0000","last_became_peered":"2024-04-06T12:12:04.224950+0000","last_unstale":"2024-04-06T12:12:08.229099+0000","last_undegraded":"2024-04-06T12:12:08.229099+0000","last_fullsized":"2024-04-06T12:12:08.229099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:55.517965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228664+0000","last_change":"2024-04-06T12:12:04.217691+0000","last_active":"2024-04-06T12:12:08.228664+0000","last_peered":"2024-04-06T12:12:08.228664+0000","last_clean":"2024-04-06T12:12:08.228664+0000","last_became_active":"2024-04-06T12:12:04.217425+0000","last_became_peered":"2024-04-06T12:12:04.217425+0000","last_unstale":"2024-04-06T12:12:08.228664+0000","last_undegraded":"2024-04-06T12:12:08.228664+0000","last_fullsized":"2024-04-06T12:12:08.228664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:10.512788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224434+0000","last_change":"2024-04-06T12:12:04.217807+0000","last_active":"2024-04-06T12:12:08.224434+0000","last_peered":"2024-04-06T12:12:08.224434+0000","last_clean":"2024-04-06T12:12:08.224434+0000","last_became_active":"2024-04-06T12:12:04.217533+0000","last_became_peered":"2024-04-06T12:12:04.217533+0000","last_unstale":"2024-04-06T12:12:08.224434+0000","last_undegraded":"2024-04-06T12:12:08.224434+0000","last_fullsized":"2024-04-06T12:12:08.224434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:43:39.601900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247241+0000","last_change":"2024-04-06T12:12:08.242261+0000","last_active":"2024-04-06T12:12:08.247241+0000","last_peered":"2024-04-06T12:12:08.247241+0000","last_clean":"2024-04-06T12:12:08.247241+0000","last_became_active":"2024-04-06T12:12:08.241896+0000","last_became_peered":"2024-04-06T12:12:08.241896+0000","last_unstale":"2024-04-06T12:12:08.247241+0000","last_undegraded":"2024-04-06T12:12:08.247241+0000","last_fullsized":"2024-04-06T12:12:08.247241+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:31:31.349113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232576+0000","last_change":"2024-04-06T12:12:03.799562+0000","last_active":"2024-04-06T12:12:08.232576+0000","last_peered":"2024-04-06T12:12:08.232576+0000","last_clean":"2024-04-06T12:12:08.232576+0000","last_became_active":"2024-04-06T12:12:03.799298+0000","last_became_peered":"2024-04-06T12:12:03.799298+0000","last_unstale":"2024-04-06T12:12:08.232576+0000","last_undegraded":"2024-04-06T12:12:08.232576+0000","last_fullsized":"2024-04-06T12:12:08.232576+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:05:14.778375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232962+0000","last_change":"2024-04-06T12:12:04.217203+0000","last_active":"2024-04-06T12:12:08.232962+0000","last_peered":"2024-04-06T12:12:08.232962+0000","last_clean":"2024-04-06T12:12:08.232962+0000","last_became_active":"2024-04-06T12:12:04.216782+0000","last_became_peered":"2024-04-06T12:12:04.216782+0000","last_unstale":"2024-04-06T12:12:08.232962+0000","last_undegraded":"2024-04-06T12:12:08.232962+0000","last_fullsized":"2024-04-06T12:12:08.232962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:55.517965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.255247+0000","last_change":"2024-04-06T12:12:08.251632+0000","last_active":"2024-04-06T12:12:08.255247+0000","last_peered":"2024-04-06T12:12:08.255247+0000","last_clean":"2024-04-06T12:12:08.255247+0000","last_became_active":"2024-04-06T12:12:08.251238+0000","last_became_peered":"2024-04-06T12:12:08.251238+0000","last_unstale":"2024-04-06T12:12:08.255247+0000","last_undegraded":"2024-04-06T12:12:08.255247+0000","last_fullsized":"2024-04-06T12:12:08.255247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:32:02.566075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.233059+0000","last_change":"2024-04-06T12:12:03.792714+0000","last_active":"2024-04-06T12:12:08.233059+0000","last_peered":"2024-04-06T12:12:08.233059+0000","last_clean":"2024-04-06T12:12:08.233059+0000","last_became_active":"2024-04-06T12:12:03.792445+0000","last_became_peered":"2024-04-06T12:12:03.792445+0000","last_unstale":"2024-04-06T12:12:08.233059+0000","last_undegraded":"2024-04-06T12:12:08.233059+0000","last_fullsized":"2024-04-06T12:12:08.233059+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:40:20.784453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232461+0000","last_change":"2024-04-06T12:12:04.215439+0000","last_active":"2024-04-06T12:12:08.232461+0000","last_peered":"2024-04-06T12:12:08.232461+0000","last_clean":"2024-04-06T12:12:08.232461+0000","last_became_active":"2024-04-06T12:12:04.214846+0000","last_became_peered":"2024-04-06T12:12:04.214846+0000","last_unstale":"2024-04-06T12:12:08.232461+0000","last_undegraded":"2024-04-06T12:12:08.232461+0000","last_fullsized":"2024-04-06T12:12:08.232461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.250137+0000","last_change":"2024-04-06T12:12:08.248890+0000","last_active":"2024-04-06T12:12:08.250137+0000","last_peered":"2024-04-06T12:12:08.250137+0000","last_clean":"2024-04-06T12:12:08.250137+0000","last_became_active":"2024-04-06T12:12:08.248470+0000","last_became_peered":"2024-04-06T12:12:08.248470+0000","last_unstale":"2024-04-06T12:12:08.250137+0000","last_undegraded":"2024-04-06T12:12:08.250137+0000","last_fullsized":"2024-04-06T12:12:08.250137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:06.085793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223703+0000","last_change":"2024-04-06T12:12:03.799557+0000","last_active":"2024-04-06T12:12:08.223703+0000","last_peered":"2024-04-06T12:12:08.223703+0000","last_clean":"2024-04-06T12:12:08.223703+0000","last_became_active":"2024-04-06T12:12:03.799271+0000","last_became_peered":"2024-04-06T12:12:03.799271+0000","last_unstale":"2024-04-06T12:12:08.223703+0000","last_undegraded":"2024-04-06T12:12:08.223703+0000","last_fullsized":"2024-04-06T12:12:08.223703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:26:21.506491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227705+0000","last_change":"2024-04-06T12:12:04.215431+0000","last_active":"2024-04-06T12:12:08.227705+0000","last_peered":"2024-04-06T12:12:08.227705+0000","last_clean":"2024-04-06T12:12:08.227705+0000","last_became_active":"2024-04-06T12:12:04.214459+0000","last_became_peered":"2024-04-06T12:12:04.214459+0000","last_unstale":"2024-04-06T12:12:08.227705+0000","last_undegraded":"2024-04-06T12:12:08.227705+0000","last_fullsized":"2024-04-06T12:12:08.227705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.252843+0000","last_change":"2024-04-06T12:12:08.249210+0000","last_active":"2024-04-06T12:12:08.252843+0000","last_peered":"2024-04-06T12:12:08.252843+0000","last_clean":"2024-04-06T12:12:08.252843+0000","last_became_active":"2024-04-06T12:12:08.248661+0000","last_became_peered":"2024-04-06T12:12:08.248661+0000","last_unstale":"2024-04-06T12:12:08.252843+0000","last_undegraded":"2024-04-06T12:12:08.252843+0000","last_fullsized":"2024-04-06T12:12:08.252843+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:29.953853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224295+0000","last_change":"2024-04-06T12:12:03.800006+0000","last_active":"2024-04-06T12:12:08.224295+0000","last_peered":"2024-04-06T12:12:08.224295+0000","last_clean":"2024-04-06T12:12:08.224295+0000","last_became_active":"2024-04-06T12:12:03.799598+0000","last_became_peered":"2024-04-06T12:12:03.799598+0000","last_unstale":"2024-04-06T12:12:08.224295+0000","last_undegraded":"2024-04-06T12:12:08.224295+0000","last_fullsized":"2024-04-06T12:12:08.224295+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:04.938049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.229419+0000","last_change":"2024-04-06T12:12:04.220105+0000","last_active":"2024-04-06T12:12:08.229419+0000","last_peered":"2024-04-06T12:12:08.229419+0000","last_clean":"2024-04-06T12:12:08.229419+0000","last_became_active":"2024-04-06T12:12:04.219876+0000","last_became_peered":"2024-04-06T12:12:04.219876+0000","last_unstale":"2024-04-06T12:12:08.229419+0000","last_undegraded":"2024-04-06T12:12:08.229419+0000","last_fullsized":"2024-04-06T12:12:08.229419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.249020+0000","last_change":"2024-04-06T12:12:08.246410+0000","last_active":"2024-04-06T12:12:08.249020+0000","last_peered":"2024-04-06T12:12:08.249020+0000","last_clean":"2024-04-06T12:12:08.249020+0000","last_became_active":"2024-04-06T12:12:08.246057+0000","last_became_peered":"2024-04-06T12:12:08.246057+0000","last_unstale":"2024-04-06T12:12:08.249020+0000","last_undegraded":"2024-04-06T12:12:08.249020+0000","last_fullsized":"2024-04-06T12:12:08.249020+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:16.183719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232695+0000","last_change":"2024-04-06T12:12:03.798850+0000","last_active":"2024-04-06T12:12:08.232695+0000","last_peered":"2024-04-06T12:12:08.232695+0000","last_clean":"2024-04-06T12:12:08.232695+0000","last_became_active":"2024-04-06T12:12:03.798550+0000","last_became_peered":"2024-04-06T12:12:03.798550+0000","last_unstale":"2024-04-06T12:12:08.232695+0000","last_undegraded":"2024-04-06T12:12:08.232695+0000","last_fullsized":"2024-04-06T12:12:08.232695+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:05:14.778375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232223+0000","last_change":"2024-04-06T12:12:04.212276+0000","last_active":"2024-04-06T12:12:08.232223+0000","last_peered":"2024-04-06T12:12:08.232223+0000","last_clean":"2024-04-06T12:12:08.232223+0000","last_became_active":"2024-04-06T12:12:04.211681+0000","last_became_peered":"2024-04-06T12:12:04.211681+0000","last_unstale":"2024-04-06T12:12:08.232223+0000","last_undegraded":"2024-04-06T12:12:08.232223+0000","last_fullsized":"2024-04-06T12:12:08.232223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:21:43.141955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.248388+0000","last_change":"2024-04-06T12:12:08.244209+0000","last_active":"2024-04-06T12:12:08.248388+0000","last_peered":"2024-04-06T12:12:08.248388+0000","last_clean":"2024-04-06T12:12:08.248388+0000","last_became_active":"2024-04-06T12:12:08.243274+0000","last_became_peered":"2024-04-06T12:12:08.243274+0000","last_unstale":"2024-04-06T12:12:08.248388+0000","last_undegraded":"2024-04-06T12:12:08.248388+0000","last_fullsized":"2024-04-06T12:12:08.248388+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:03.696838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223468+0000","last_change":"2024-04-06T12:12:03.796799+0000","last_active":"2024-04-06T12:12:08.223468+0000","last_peered":"2024-04-06T12:12:08.223468+0000","last_clean":"2024-04-06T12:12:08.223468+0000","last_became_active":"2024-04-06T12:12:03.796536+0000","last_became_peered":"2024-04-06T12:12:03.796536+0000","last_unstale":"2024-04-06T12:12:08.223468+0000","last_undegraded":"2024-04-06T12:12:08.223468+0000","last_fullsized":"2024-04-06T12:12:08.223468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:15.699305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234439+0000","last_change":"2024-04-06T12:12:04.220607+0000","last_active":"2024-04-06T12:12:08.234439+0000","last_peered":"2024-04-06T12:12:08.234439+0000","last_clean":"2024-04-06T12:12:08.234439+0000","last_became_active":"2024-04-06T12:12:04.220242+0000","last_became_peered":"2024-04-06T12:12:04.220242+0000","last_unstale":"2024-04-06T12:12:08.234439+0000","last_undegraded":"2024-04-06T12:12:08.234439+0000","last_fullsized":"2024-04-06T12:12:08.234439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:00.258491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.243307+0000","last_change":"2024-04-06T12:12:08.237694+0000","last_active":"2024-04-06T12:12:08.243307+0000","last_peered":"2024-04-06T12:12:08.243307+0000","last_clean":"2024-04-06T12:12:08.243307+0000","last_became_active":"2024-04-06T12:12:08.237190+0000","last_became_peered":"2024-04-06T12:12:08.237190+0000","last_unstale":"2024-04-06T12:12:08.243307+0000","last_undegraded":"2024-04-06T12:12:08.243307+0000","last_fullsized":"2024-04-06T12:12:08.243307+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:50:32.228428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232240+0000","last_change":"2024-04-06T12:12:03.800903+0000","last_active":"2024-04-06T12:12:08.232240+0000","last_peered":"2024-04-06T12:12:08.232240+0000","last_clean":"2024-04-06T12:12:08.232240+0000","last_became_active":"2024-04-06T12:12:03.800590+0000","last_became_peered":"2024-04-06T12:12:03.800590+0000","last_unstale":"2024-04-06T12:12:08.232240+0000","last_undegraded":"2024-04-06T12:12:08.232240+0000","last_fullsized":"2024-04-06T12:12:08.232240+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:00:31.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232899+0000","last_change":"2024-04-06T12:12:04.216144+0000","last_active":"2024-04-06T12:12:08.232899+0000","last_peered":"2024-04-06T12:12:08.232899+0000","last_clean":"2024-04-06T12:12:08.232899+0000","last_became_active":"2024-04-06T12:12:04.215885+0000","last_became_peered":"2024-04-06T12:12:04.215885+0000","last_unstale":"2024-04-06T12:12:08.232899+0000","last_undegraded":"2024-04-06T12:12:08.232899+0000","last_fullsized":"2024-04-06T12:12:08.232899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:36:25.380472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256611+0000","last_change":"2024-04-06T12:12:08.252363+0000","last_active":"2024-04-06T12:12:08.256611+0000","last_peered":"2024-04-06T12:12:08.256611+0000","last_clean":"2024-04-06T12:12:08.256611+0000","last_became_active":"2024-04-06T12:12:08.252029+0000","last_became_peered":"2024-04-06T12:12:08.252029+0000","last_unstale":"2024-04-06T12:12:08.256611+0000","last_undegraded":"2024-04-06T12:12:08.256611+0000","last_fullsized":"2024-04-06T12:12:08.256611+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:35:54.255608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227386+0000","last_change":"2024-04-06T12:12:03.800462+0000","last_active":"2024-04-06T12:12:08.227386+0000","last_peered":"2024-04-06T12:12:08.227386+0000","last_clean":"2024-04-06T12:12:08.227386+0000","last_became_active":"2024-04-06T12:12:03.800210+0000","last_became_peered":"2024-04-06T12:12:03.800210+0000","last_unstale":"2024-04-06T12:12:08.227386+0000","last_undegraded":"2024-04-06T12:12:08.227386+0000","last_fullsized":"2024-04-06T12:12:08.227386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:00:31.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222742+0000","last_change":"2024-04-06T12:12:04.216700+0000","last_active":"2024-04-06T12:12:08.222742+0000","last_peered":"2024-04-06T12:12:08.222742+0000","last_clean":"2024-04-06T12:12:08.222742+0000","last_became_active":"2024-04-06T12:12:04.216435+0000","last_became_peered":"2024-04-06T12:12:04.216435+0000","last_unstale":"2024-04-06T12:12:08.222742+0000","last_undegraded":"2024-04-06T12:12:08.222742+0000","last_fullsized":"2024-04-06T12:12:08.222742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:36:25.380472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247221+0000","last_change":"2024-04-06T12:12:08.242079+0000","last_active":"2024-04-06T12:12:08.247221+0000","last_peered":"2024-04-06T12:12:08.247221+0000","last_clean":"2024-04-06T12:12:08.247221+0000","last_became_active":"2024-04-06T12:12:08.241648+0000","last_became_peered":"2024-04-06T12:12:08.241648+0000","last_unstale":"2024-04-06T12:12:08.247221+0000","last_undegraded":"2024-04-06T12:12:08.247221+0000","last_fullsized":"2024-04-06T12:12:08.247221+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:37:57.999646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224055+0000","last_change":"2024-04-06T12:12:03.799875+0000","last_active":"2024-04-06T12:12:08.224055+0000","last_peered":"2024-04-06T12:12:08.224055+0000","last_clean":"2024-04-06T12:12:08.224055+0000","last_became_active":"2024-04-06T12:12:03.799562+0000","last_became_peered":"2024-04-06T12:12:03.799562+0000","last_unstale":"2024-04-06T12:12:08.224055+0000","last_undegraded":"2024-04-06T12:12:08.224055+0000","last_fullsized":"2024-04-06T12:12:08.224055+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:08:55.770454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.229090+0000","last_change":"2024-04-06T12:12:04.225142+0000","last_active":"2024-04-06T12:12:08.229090+0000","last_peered":"2024-04-06T12:12:08.229090+0000","last_clean":"2024-04-06T12:12:08.229090+0000","last_became_active":"2024-04-06T12:12:04.224838+0000","last_became_peered":"2024-04-06T12:12:04.224838+0000","last_unstale":"2024-04-06T12:12:08.229090+0000","last_undegraded":"2024-04-06T12:12:08.229090+0000","last_fullsized":"2024-04-06T12:12:08.229090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:21:16.167775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247222+0000","last_change":"2024-04-06T12:12:08.244037+0000","last_active":"2024-04-06T12:12:08.247222+0000","last_peered":"2024-04-06T12:12:08.247222+0000","last_clean":"2024-04-06T12:12:08.247222+0000","last_became_active":"2024-04-06T12:12:08.243685+0000","last_became_peered":"2024-04-06T12:12:08.243685+0000","last_unstale":"2024-04-06T12:12:08.247222+0000","last_undegraded":"2024-04-06T12:12:08.247222+0000","last_fullsized":"2024-04-06T12:12:08.247222+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:02:51.216167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228999+0000","last_change":"2024-04-06T12:12:03.799591+0000","last_active":"2024-04-06T12:12:08.228999+0000","last_peered":"2024-04-06T12:12:08.228999+0000","last_clean":"2024-04-06T12:12:08.228999+0000","last_became_active":"2024-04-06T12:12:03.799333+0000","last_became_peered":"2024-04-06T12:12:03.799333+0000","last_unstale":"2024-04-06T12:12:08.228999+0000","last_undegraded":"2024-04-06T12:12:08.228999+0000","last_fullsized":"2024-04-06T12:12:08.228999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:11.635313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232670+0000","last_change":"2024-04-06T12:12:04.215620+0000","last_active":"2024-04-06T12:12:08.232670+0000","last_peered":"2024-04-06T12:12:08.232670+0000","last_clean":"2024-04-06T12:12:08.232670+0000","last_became_active":"2024-04-06T12:12:04.215189+0000","last_became_peered":"2024-04-06T12:12:04.215189+0000","last_unstale":"2024-04-06T12:12:08.232670+0000","last_undegraded":"2024-04-06T12:12:08.232670+0000","last_fullsized":"2024-04-06T12:12:08.232670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247852+0000","last_change":"2024-04-06T12:12:08.246657+0000","last_active":"2024-04-06T12:12:08.247852+0000","last_peered":"2024-04-06T12:12:08.247852+0000","last_clean":"2024-04-06T12:12:08.247852+0000","last_became_active":"2024-04-06T12:12:08.246310+0000","last_became_peered":"2024-04-06T12:12:08.246310+0000","last_unstale":"2024-04-06T12:12:08.247852+0000","last_undegraded":"2024-04-06T12:12:08.247852+0000","last_fullsized":"2024-04-06T12:12:08.247852+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:57:02.114694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223115+0000","last_change":"2024-04-06T12:12:03.802440+0000","last_active":"2024-04-06T12:12:08.223115+0000","last_peered":"2024-04-06T12:12:08.223115+0000","last_clean":"2024-04-06T12:12:08.223115+0000","last_became_active":"2024-04-06T12:12:03.802193+0000","last_became_peered":"2024-04-06T12:12:03.802193+0000","last_unstale":"2024-04-06T12:12:08.223115+0000","last_undegraded":"2024-04-06T12:12:08.223115+0000","last_fullsized":"2024-04-06T12:12:08.223115+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:04.938049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232783+0000","last_change":"2024-04-06T12:12:04.225740+0000","last_active":"2024-04-06T12:12:08.232783+0000","last_peered":"2024-04-06T12:12:08.232783+0000","last_clean":"2024-04-06T12:12:08.232783+0000","last_became_active":"2024-04-06T12:12:04.224605+0000","last_became_peered":"2024-04-06T12:12:04.224605+0000","last_unstale":"2024-04-06T12:12:08.232783+0000","last_undegraded":"2024-04-06T12:12:08.232783+0000","last_fullsized":"2024-04-06T12:12:08.232783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:26:22.511847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.245748+0000","last_change":"2024-04-06T12:12:08.242260+0000","last_active":"2024-04-06T12:12:08.245748+0000","last_peered":"2024-04-06T12:12:08.245748+0000","last_clean":"2024-04-06T12:12:08.245748+0000","last_became_active":"2024-04-06T12:12:08.241863+0000","last_became_peered":"2024-04-06T12:12:08.241863+0000","last_unstale":"2024-04-06T12:12:08.245748+0000","last_undegraded":"2024-04-06T12:12:08.245748+0000","last_fullsized":"2024-04-06T12:12:08.245748+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:06.085793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-06T12:12:13.609454+0000","last_change":"2024-04-06T12:12:03.799037+0000","last_active":"2024-04-06T12:12:13.609454+0000","last_peered":"2024-04-06T12:12:13.609454+0000","last_clean":"2024-04-06T12:12:13.609454+0000","last_became_active":"2024-04-06T12:12:03.798723+0000","last_became_peered":"2024-04-06T12:12:03.798723+0000","last_unstale":"2024-04-06T12:12:13.609454+0000","last_undegraded":"2024-04-06T12:12:13.609454+0000","last_fullsized":"2024-04-06T12:12:13.609454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:00:31.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3796,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224345+0000","last_change":"2024-04-06T12:12:04.226914+0000","last_active":"2024-04-06T12:12:08.224345+0000","last_peered":"2024-04-06T12:12:08.224345+0000","last_clean":"2024-04-06T12:12:08.224345+0000","last_became_active":"2024-04-06T12:12:04.226421+0000","last_became_peered":"2024-04-06T12:12:04.226421+0000","last_unstale":"2024-04-06T12:12:08.224345+0000","last_undegraded":"2024-04-06T12:12:08.224345+0000","last_fullsized":"2024-04-06T12:12:08.224345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:45:10.755538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247061+0000","last_change":"2024-04-06T12:12:08.242923+0000","last_active":"2024-04-06T12:12:08.247061+0000","last_peered":"2024-04-06T12:12:08.247061+0000","last_clean":"2024-04-06T12:12:08.247061+0000","last_became_active":"2024-04-06T12:12:08.242491+0000","last_became_peered":"2024-04-06T12:12:08.242491+0000","last_unstale":"2024-04-06T12:12:08.247061+0000","last_undegraded":"2024-04-06T12:12:08.247061+0000","last_fullsized":"2024-04-06T12:12:08.247061+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:57:02.114694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234681+0000","last_change":"2024-04-06T12:12:03.796757+0000","last_active":"2024-04-06T12:12:08.234681+0000","last_peered":"2024-04-06T12:12:08.234681+0000","last_clean":"2024-04-06T12:12:08.234681+0000","last_became_active":"2024-04-06T12:12:03.796519+0000","last_became_peered":"2024-04-06T12:12:03.796519+0000","last_unstale":"2024-04-06T12:12:08.234681+0000","last_undegraded":"2024-04-06T12:12:08.234681+0000","last_fullsized":"2024-04-06T12:12:08.234681+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:40:20.784453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.235485+0000","last_change":"2024-04-06T12:12:04.224445+0000","last_active":"2024-04-06T12:12:08.235485+0000","last_peered":"2024-04-06T12:12:08.235485+0000","last_clean":"2024-04-06T12:12:08.235485+0000","last_became_active":"2024-04-06T12:12:04.224153+0000","last_became_peered":"2024-04-06T12:12:04.224153+0000","last_unstale":"2024-04-06T12:12:08.235485+0000","last_undegraded":"2024-04-06T12:12:08.235485+0000","last_fullsized":"2024-04-06T12:12:08.235485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:15.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256724+0000","last_change":"2024-04-06T12:12:08.251536+0000","last_active":"2024-04-06T12:12:08.256724+0000","last_peered":"2024-04-06T12:12:08.256724+0000","last_clean":"2024-04-06T12:12:08.256724+0000","last_became_active":"2024-04-06T12:12:08.251142+0000","last_became_peered":"2024-04-06T12:12:08.251142+0000","last_unstale":"2024-04-06T12:12:08.256724+0000","last_undegraded":"2024-04-06T12:12:08.256724+0000","last_fullsized":"2024-04-06T12:12:08.256724+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:48:55.155611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228030+0000","last_change":"2024-04-06T12:12:03.797816+0000","last_active":"2024-04-06T12:12:08.228030+0000","last_peered":"2024-04-06T12:12:08.228030+0000","last_clean":"2024-04-06T12:12:08.228030+0000","last_became_active":"2024-04-06T12:12:03.797540+0000","last_became_peered":"2024-04-06T12:12:03.797540+0000","last_unstale":"2024-04-06T12:12:08.228030+0000","last_undegraded":"2024-04-06T12:12:08.228030+0000","last_fullsized":"2024-04-06T12:12:08.228030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:15.699305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224078+0000","last_change":"2024-04-06T12:12:04.223703+0000","last_active":"2024-04-06T12:12:08.224078+0000","last_peered":"2024-04-06T12:12:08.224078+0000","last_clean":"2024-04-06T12:12:08.224078+0000","last_became_active":"2024-04-06T12:12:04.223394+0000","last_became_peered":"2024-04-06T12:12:04.223394+0000","last_unstale":"2024-04-06T12:12:08.224078+0000","last_undegraded":"2024-04-06T12:12:08.224078+0000","last_fullsized":"2024-04-06T12:12:08.224078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:19:38.810657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256643+0000","last_change":"2024-04-06T12:12:08.250454+0000","last_active":"2024-04-06T12:12:08.256643+0000","last_peered":"2024-04-06T12:12:08.256643+0000","last_clean":"2024-04-06T12:12:08.256643+0000","last_became_active":"2024-04-06T12:12:08.250052+0000","last_became_peered":"2024-04-06T12:12:08.250052+0000","last_unstale":"2024-04-06T12:12:08.256643+0000","last_undegraded":"2024-04-06T12:12:08.256643+0000","last_fullsized":"2024-04-06T12:12:08.256643+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:11:04.976956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223382+0000","last_change":"2024-04-06T12:12:03.799361+0000","last_active":"2024-04-06T12:12:08.223382+0000","last_peered":"2024-04-06T12:12:08.223382+0000","last_clean":"2024-04-06T12:12:08.223382+0000","last_became_active":"2024-04-06T12:12:03.798981+0000","last_became_peered":"2024-04-06T12:12:03.798981+0000","last_unstale":"2024-04-06T12:12:08.223382+0000","last_undegraded":"2024-04-06T12:12:08.223382+0000","last_fullsized":"2024-04-06T12:12:08.223382+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:11.635313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224104+0000","last_change":"2024-04-06T12:12:04.223535+0000","last_active":"2024-04-06T12:12:08.224104+0000","last_peered":"2024-04-06T12:12:08.224104+0000","last_clean":"2024-04-06T12:12:08.224104+0000","last_became_active":"2024-04-06T12:12:04.217966+0000","last_became_peered":"2024-04-06T12:12:04.217966+0000","last_unstale":"2024-04-06T12:12:08.224104+0000","last_undegraded":"2024-04-06T12:12:08.224104+0000","last_fullsized":"2024-04-06T12:12:08.224104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:15.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256634+0000","last_change":"2024-04-06T12:12:08.252578+0000","last_active":"2024-04-06T12:12:08.256634+0000","last_peered":"2024-04-06T12:12:08.256634+0000","last_clean":"2024-04-06T12:12:08.256634+0000","last_became_active":"2024-04-06T12:12:08.252149+0000","last_became_peered":"2024-04-06T12:12:08.252149+0000","last_unstale":"2024-04-06T12:12:08.256634+0000","last_undegraded":"2024-04-06T12:12:08.256634+0000","last_fullsized":"2024-04-06T12:12:08.256634+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:06.085793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.226746+0000","last_change":"2024-04-06T12:12:03.799994+0000","last_active":"2024-04-06T12:12:08.226746+0000","last_peered":"2024-04-06T12:12:08.226746+0000","last_clean":"2024-04-06T12:12:08.226746+0000","last_became_active":"2024-04-06T12:12:03.799663+0000","last_became_peered":"2024-04-06T12:12:03.799663+0000","last_unstale":"2024-04-06T12:12:08.226746+0000","last_undegraded":"2024-04-06T12:12:08.226746+0000","last_fullsized":"2024-04-06T12:12:08.226746+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:08:55.770454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223501+0000","last_change":"2024-04-06T12:12:04.223425+0000","last_active":"2024-04-06T12:12:08.223501+0000","last_peered":"2024-04-06T12:12:08.223501+0000","last_clean":"2024-04-06T12:12:08.223501+0000","last_became_active":"2024-04-06T12:12:04.223101+0000","last_became_peered":"2024-04-06T12:12:04.223101+0000","last_unstale":"2024-04-06T12:12:08.223501+0000","last_undegraded":"2024-04-06T12:12:08.223501+0000","last_fullsized":"2024-04-06T12:12:08.223501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.250437+0000","last_change":"2024-04-06T12:12:08.249078+0000","last_active":"2024-04-06T12:12:08.250437+0000","last_peered":"2024-04-06T12:12:08.250437+0000","last_clean":"2024-04-06T12:12:08.250437+0000","last_became_active":"2024-04-06T12:12:08.248512+0000","last_became_peered":"2024-04-06T12:12:08.248512+0000","last_unstale":"2024-04-06T12:12:08.250437+0000","last_undegraded":"2024-04-06T12:12:08.250437+0000","last_fullsized":"2024-04-06T12:12:08.250437+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:24:33.553367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.230408+0000","last_change":"2024-04-06T12:12:03.793832+0000","last_active":"2024-04-06T12:12:08.230408+0000","last_peered":"2024-04-06T12:12:08.230408+0000","last_clean":"2024-04-06T12:12:08.230408+0000","last_became_active":"2024-04-06T12:12:03.793518+0000","last_became_peered":"2024-04-06T12:12:03.793518+0000","last_unstale":"2024-04-06T12:12:08.230408+0000","last_undegraded":"2024-04-06T12:12:08.230408+0000","last_fullsized":"2024-04-06T12:12:08.230408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:37:27.257815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224591+0000","last_change":"2024-04-06T12:12:04.224260+0000","last_active":"2024-04-06T12:12:08.224591+0000","last_peered":"2024-04-06T12:12:08.224591+0000","last_clean":"2024-04-06T12:12:08.224591+0000","last_became_active":"2024-04-06T12:12:04.223968+0000","last_became_peered":"2024-04-06T12:12:04.223968+0000","last_unstale":"2024-04-06T12:12:08.224591+0000","last_undegraded":"2024-04-06T12:12:08.224591+0000","last_fullsized":"2024-04-06T12:12:08.224591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.245727+0000","last_change":"2024-04-06T12:12:08.242094+0000","last_active":"2024-04-06T12:12:08.245727+0000","last_peered":"2024-04-06T12:12:08.245727+0000","last_clean":"2024-04-06T12:12:08.245727+0000","last_became_active":"2024-04-06T12:12:08.241707+0000","last_became_peered":"2024-04-06T12:12:08.241707+0000","last_unstale":"2024-04-06T12:12:08.245727+0000","last_undegraded":"2024-04-06T12:12:08.245727+0000","last_fullsized":"2024-04-06T12:12:08.245727+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:47:51.958478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222772+0000","last_change":"2024-04-06T12:12:03.800106+0000","last_active":"2024-04-06T12:12:08.222772+0000","last_peered":"2024-04-06T12:12:08.222772+0000","last_clean":"2024-04-06T12:12:08.222772+0000","last_became_active":"2024-04-06T12:12:03.799843+0000","last_became_peered":"2024-04-06T12:12:03.799843+0000","last_unstale":"2024-04-06T12:12:08.222772+0000","last_undegraded":"2024-04-06T12:12:08.222772+0000","last_fullsized":"2024-04-06T12:12:08.222772+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:26:21.506491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.229502+0000","last_change":"2024-04-06T12:12:04.214371+0000","last_active":"2024-04-06T12:12:08.229502+0000","last_peered":"2024-04-06T12:12:08.229502+0000","last_clean":"2024-04-06T12:12:08.229502+0000","last_became_active":"2024-04-06T12:12:04.214095+0000","last_became_peered":"2024-04-06T12:12:04.214095+0000","last_unstale":"2024-04-06T12:12:08.229502+0000","last_undegraded":"2024-04-06T12:12:08.229502+0000","last_fullsized":"2024-04-06T12:12:08.229502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:45:10.755538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.258734+0000","last_change":"2024-04-06T12:12:08.255356+0000","last_active":"2024-04-06T12:12:08.258734+0000","last_peered":"2024-04-06T12:12:08.258734+0000","last_clean":"2024-04-06T12:12:08.258734+0000","last_became_active":"2024-04-06T12:12:08.254148+0000","last_became_peered":"2024-04-06T12:12:08.254148+0000","last_unstale":"2024-04-06T12:12:08.258734+0000","last_undegraded":"2024-04-06T12:12:08.258734+0000","last_fullsized":"2024-04-06T12:12:08.258734+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:35:54.255608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227821+0000","last_change":"2024-04-06T12:12:03.801770+0000","last_active":"2024-04-06T12:12:08.227821+0000","last_peered":"2024-04-06T12:12:08.227821+0000","last_clean":"2024-04-06T12:12:08.227821+0000","last_became_active":"2024-04-06T12:12:03.801494+0000","last_became_peered":"2024-04-06T12:12:03.801494+0000","last_unstale":"2024-04-06T12:12:08.227821+0000","last_undegraded":"2024-04-06T12:12:08.227821+0000","last_fullsized":"2024-04-06T12:12:08.227821+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:08:55.770454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232799+0000","last_change":"2024-04-06T12:12:04.217734+0000","last_active":"2024-04-06T12:12:08.232799+0000","last_peered":"2024-04-06T12:12:08.232799+0000","last_clean":"2024-04-06T12:12:08.232799+0000","last_became_active":"2024-04-06T12:12:04.216838+0000","last_became_peered":"2024-04-06T12:12:04.216838+0000","last_unstale":"2024-04-06T12:12:08.232799+0000","last_undegraded":"2024-04-06T12:12:08.232799+0000","last_fullsized":"2024-04-06T12:12:08.232799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:05.943405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.258759+0000","last_change":"2024-04-06T12:12:08.255965+0000","last_active":"2024-04-06T12:12:08.258759+0000","last_peered":"2024-04-06T12:12:08.258759+0000","last_clean":"2024-04-06T12:12:08.258759+0000","last_became_active":"2024-04-06T12:12:08.251143+0000","last_became_peered":"2024-04-06T12:12:08.251143+0000","last_unstale":"2024-04-06T12:12:08.258759+0000","last_undegraded":"2024-04-06T12:12:08.258759+0000","last_fullsized":"2024-04-06T12:12:08.258759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:21:01.506046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222665+0000","last_change":"2024-04-06T12:12:04.208976+0000","last_active":"2024-04-06T12:12:08.222665+0000","last_peered":"2024-04-06T12:12:08.222665+0000","last_clean":"2024-04-06T12:12:08.222665+0000","last_became_active":"2024-04-06T12:12:04.208524+0000","last_became_peered":"2024-04-06T12:12:04.208524+0000","last_unstale":"2024-04-06T12:12:08.222665+0000","last_undegraded":"2024-04-06T12:12:08.222665+0000","last_fullsized":"2024-04-06T12:12:08.222665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.226929+0000","last_change":"2024-04-06T12:12:03.796992+0000","last_active":"2024-04-06T12:12:08.226929+0000","last_peered":"2024-04-06T12:12:08.226929+0000","last_clean":"2024-04-06T12:12:08.226929+0000","last_became_active":"2024-04-06T12:12:03.795824+0000","last_became_peered":"2024-04-06T12:12:03.795824+0000","last_unstale":"2024-04-06T12:12:08.226929+0000","last_undegraded":"2024-04-06T12:12:08.226929+0000","last_fullsized":"2024-04-06T12:12:08.226929+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:11.635313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.246589+0000","last_change":"2024-04-06T12:12:08.244008+0000","last_active":"2024-04-06T12:12:08.246589+0000","last_peered":"2024-04-06T12:12:08.246589+0000","last_clean":"2024-04-06T12:12:08.246589+0000","last_became_active":"2024-04-06T12:12:08.242895+0000","last_became_peered":"2024-04-06T12:12:08.242895+0000","last_unstale":"2024-04-06T12:12:08.246589+0000","last_undegraded":"2024-04-06T12:12:08.246589+0000","last_fullsized":"2024-04-06T12:12:08.246589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:54:54.771467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232422+0000","last_change":"2024-04-06T12:12:04.215137+0000","last_active":"2024-04-06T12:12:08.232422+0000","last_peered":"2024-04-06T12:12:08.232422+0000","last_clean":"2024-04-06T12:12:08.232422+0000","last_became_active":"2024-04-06T12:12:04.214872+0000","last_became_peered":"2024-04-06T12:12:04.214872+0000","last_unstale":"2024-04-06T12:12:08.232422+0000","last_undegraded":"2024-04-06T12:12:08.232422+0000","last_fullsized":"2024-04-06T12:12:08.232422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227366+0000","last_change":"2024-04-06T12:12:03.797439+0000","last_active":"2024-04-06T12:12:08.227366+0000","last_peered":"2024-04-06T12:12:08.227366+0000","last_clean":"2024-04-06T12:12:08.227366+0000","last_became_active":"2024-04-06T12:12:03.797176+0000","last_became_peered":"2024-04-06T12:12:03.797176+0000","last_unstale":"2024-04-06T12:12:08.227366+0000","last_undegraded":"2024-04-06T12:12:08.227366+0000","last_fullsized":"2024-04-06T12:12:08.227366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:26:21.506491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223206+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:12:08.223206+0000","last_peered":"2024-04-06T12:12:08.223206+0000","last_clean":"2024-04-06T12:12:08.223206+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:12:08.223206+0000","last_undegraded":"2024-04-06T12:12:08.223206+0000","last_fullsized":"2024-04-06T12:12:08.223206+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.252451+0000","last_change":"2024-04-06T12:12:08.248645+0000","last_active":"2024-04-06T12:12:08.252451+0000","last_peered":"2024-04-06T12:12:08.252451+0000","last_clean":"2024-04-06T12:12:08.252451+0000","last_became_active":"2024-04-06T12:12:08.247420+0000","last_became_peered":"2024-04-06T12:12:08.247420+0000","last_unstale":"2024-04-06T12:12:08.252451+0000","last_undegraded":"2024-04-06T12:12:08.252451+0000","last_fullsized":"2024-04-06T12:12:08.252451+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:31:21.467656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232818+0000","last_change":"2024-04-06T12:12:04.220328+0000","last_active":"2024-04-06T12:12:08.232818+0000","last_peered":"2024-04-06T12:12:08.232818+0000","last_clean":"2024-04-06T12:12:08.232818+0000","last_became_active":"2024-04-06T12:12:04.219948+0000","last_became_peered":"2024-04-06T12:12:04.219948+0000","last_unstale":"2024-04-06T12:12:08.232818+0000","last_undegraded":"2024-04-06T12:12:08.232818+0000","last_fullsized":"2024-04-06T12:12:08.232818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:36:25.380472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223326+0000","last_change":"2024-04-06T12:12:03.800906+0000","last_active":"2024-04-06T12:12:08.223326+0000","last_peered":"2024-04-06T12:12:08.223326+0000","last_clean":"2024-04-06T12:12:08.223326+0000","last_became_active":"2024-04-06T12:12:03.800460+0000","last_became_peered":"2024-04-06T12:12:03.800460+0000","last_unstale":"2024-04-06T12:12:08.223326+0000","last_undegraded":"2024-04-06T12:12:08.223326+0000","last_fullsized":"2024-04-06T12:12:08.223326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:11.635313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.251977+0000","last_change":"2024-04-06T12:12:08.246821+0000","last_active":"2024-04-06T12:12:08.251977+0000","last_peered":"2024-04-06T12:12:08.251977+0000","last_clean":"2024-04-06T12:12:08.251977+0000","last_became_active":"2024-04-06T12:12:08.246154+0000","last_became_peered":"2024-04-06T12:12:08.246154+0000","last_unstale":"2024-04-06T12:12:08.251977+0000","last_undegraded":"2024-04-06T12:12:08.251977+0000","last_fullsized":"2024-04-06T12:12:08.251977+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:47:51.958478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223565+0000","last_change":"2024-04-06T12:12:03.801675+0000","last_active":"2024-04-06T12:12:08.223565+0000","last_peered":"2024-04-06T12:12:08.223565+0000","last_clean":"2024-04-06T12:12:08.223565+0000","last_became_active":"2024-04-06T12:12:03.800427+0000","last_became_peered":"2024-04-06T12:12:03.800427+0000","last_unstale":"2024-04-06T12:12:08.223565+0000","last_undegraded":"2024-04-06T12:12:08.223565+0000","last_fullsized":"2024-04-06T12:12:08.223565+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:21:42.136599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222894+0000","last_change":"2024-04-06T12:12:04.216377+0000","last_active":"2024-04-06T12:12:08.222894+0000","last_peered":"2024-04-06T12:12:08.222894+0000","last_clean":"2024-04-06T12:12:08.222894+0000","last_became_active":"2024-04-06T12:12:04.216084+0000","last_became_peered":"2024-04-06T12:12:04.216084+0000","last_unstale":"2024-04-06T12:12:08.222894+0000","last_undegraded":"2024-04-06T12:12:08.222894+0000","last_fullsized":"2024-04-06T12:12:08.222894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256642+0000","last_change":"2024-04-06T12:12:08.250308+0000","last_active":"2024-04-06T12:12:08.256642+0000","last_peered":"2024-04-06T12:12:08.256642+0000","last_clean":"2024-04-06T12:12:08.256642+0000","last_became_active":"2024-04-06T12:12:08.249904+0000","last_became_peered":"2024-04-06T12:12:08.249904+0000","last_unstale":"2024-04-06T12:12:08.256642+0000","last_undegraded":"2024-04-06T12:12:08.256642+0000","last_fullsized":"2024-04-06T12:12:08.256642+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:35:23.088849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222894+0000","last_change":"2024-04-06T12:12:03.803297+0000","last_active":"2024-04-06T12:12:08.222894+0000","last_peered":"2024-04-06T12:12:08.222894+0000","last_clean":"2024-04-06T12:12:08.222894+0000","last_became_active":"2024-04-06T12:12:03.803053+0000","last_became_peered":"2024-04-06T12:12:03.803053+0000","last_unstale":"2024-04-06T12:12:08.222894+0000","last_undegraded":"2024-04-06T12:12:08.222894+0000","last_fullsized":"2024-04-06T12:12:08.222894+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:21:42.136599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234446+0000","last_change":"2024-04-06T12:12:04.225056+0000","last_active":"2024-04-06T12:12:08.234446+0000","last_peered":"2024-04-06T12:12:08.234446+0000","last_clean":"2024-04-06T12:12:08.234446+0000","last_became_active":"2024-04-06T12:12:04.224635+0000","last_became_peered":"2024-04-06T12:12:04.224635+0000","last_unstale":"2024-04-06T12:12:08.234446+0000","last_undegraded":"2024-04-06T12:12:08.234446+0000","last_fullsized":"2024-04-06T12:12:08.234446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247224+0000","last_change":"2024-04-06T12:12:08.244844+0000","last_active":"2024-04-06T12:12:08.247224+0000","last_peered":"2024-04-06T12:12:08.247224+0000","last_clean":"2024-04-06T12:12:08.247224+0000","last_became_active":"2024-04-06T12:12:08.244422+0000","last_became_peered":"2024-04-06T12:12:08.244422+0000","last_unstale":"2024-04-06T12:12:08.247224+0000","last_undegraded":"2024-04-06T12:12:08.247224+0000","last_fullsized":"2024-04-06T12:12:08.247224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:54:54.771467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223933+0000","last_change":"2024-04-06T12:12:03.800460+0000","last_active":"2024-04-06T12:12:08.223933+0000","last_peered":"2024-04-06T12:12:08.223933+0000","last_clean":"2024-04-06T12:12:08.223933+0000","last_became_active":"2024-04-06T12:12:03.800183+0000","last_became_peered":"2024-04-06T12:12:03.800183+0000","last_unstale":"2024-04-06T12:12:08.223933+0000","last_undegraded":"2024-04-06T12:12:08.223933+0000","last_fullsized":"2024-04-06T12:12:08.223933+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:40:20.784453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.235166+0000","last_change":"2024-04-06T12:12:04.209360+0000","last_active":"2024-04-06T12:12:08.235166+0000","last_peered":"2024-04-06T12:12:08.235166+0000","last_clean":"2024-04-06T12:12:08.235166+0000","last_became_active":"2024-04-06T12:12:04.209105+0000","last_became_peered":"2024-04-06T12:12:04.209105+0000","last_unstale":"2024-04-06T12:12:08.235166+0000","last_undegraded":"2024-04-06T12:12:08.235166+0000","last_fullsized":"2024-04-06T12:12:08.235166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.252713+0000","last_change":"2024-04-06T12:12:08.249325+0000","last_active":"2024-04-06T12:12:08.252713+0000","last_peered":"2024-04-06T12:12:08.252713+0000","last_clean":"2024-04-06T12:12:08.252713+0000","last_became_active":"2024-04-06T12:12:08.248776+0000","last_became_peered":"2024-04-06T12:12:08.248776+0000","last_unstale":"2024-04-06T12:12:08.252713+0000","last_undegraded":"2024-04-06T12:12:08.252713+0000","last_fullsized":"2024-04-06T12:12:08.252713+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:31:21.467656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223337+0000","last_change":"2024-04-06T12:12:03.797556+0000","last_active":"2024-04-06T12:12:08.223337+0000","last_peered":"2024-04-06T12:12:08.223337+0000","last_clean":"2024-04-06T12:12:08.223337+0000","last_became_active":"2024-04-06T12:12:03.797288+0000","last_became_peered":"2024-04-06T12:12:03.797288+0000","last_unstale":"2024-04-06T12:12:08.223337+0000","last_undegraded":"2024-04-06T12:12:08.223337+0000","last_fullsized":"2024-04-06T12:12:08.223337+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:15.699305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.231368+0000","last_change":"2024-04-06T12:12:04.219859+0000","last_active":"2024-04-06T12:12:08.231368+0000","last_peered":"2024-04-06T12:12:08.231368+0000","last_clean":"2024-04-06T12:12:08.231368+0000","last_became_active":"2024-04-06T12:12:04.219470+0000","last_became_peered":"2024-04-06T12:12:04.219470+0000","last_unstale":"2024-04-06T12:12:08.231368+0000","last_undegraded":"2024-04-06T12:12:08.231368+0000","last_fullsized":"2024-04-06T12:12:08.231368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:16.704661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.258673+0000","last_change":"2024-04-06T12:12:08.254506+0000","last_active":"2024-04-06T12:12:08.258673+0000","last_peered":"2024-04-06T12:12:08.258673+0000","last_clean":"2024-04-06T12:12:08.258673+0000","last_became_active":"2024-04-06T12:12:08.250366+0000","last_became_peered":"2024-04-06T12:12:08.250366+0000","last_unstale":"2024-04-06T12:12:08.258673+0000","last_undegraded":"2024-04-06T12:12:08.258673+0000","last_fullsized":"2024-04-06T12:12:08.258673+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:29.953853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222644+0000","last_change":"2024-04-06T12:12:03.801858+0000","last_active":"2024-04-06T12:12:08.222644+0000","last_peered":"2024-04-06T12:12:08.222644+0000","last_clean":"2024-04-06T12:12:08.222644+0000","last_became_active":"2024-04-06T12:12:03.801643+0000","last_became_peered":"2024-04-06T12:12:03.801643+0000","last_unstale":"2024-04-06T12:12:08.222644+0000","last_undegraded":"2024-04-06T12:12:08.222644+0000","last_fullsized":"2024-04-06T12:12:08.222644+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:08:55.770454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232963+0000","last_change":"2024-04-06T12:12:04.213199+0000","last_active":"2024-04-06T12:12:08.232963+0000","last_peered":"2024-04-06T12:12:08.232963+0000","last_clean":"2024-04-06T12:12:08.232963+0000","last_became_active":"2024-04-06T12:12:04.212685+0000","last_became_peered":"2024-04-06T12:12:04.212685+0000","last_unstale":"2024-04-06T12:12:08.232963+0000","last_undegraded":"2024-04-06T12:12:08.232963+0000","last_fullsized":"2024-04-06T12:12:08.232963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:45:10.755538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.250618+0000","last_change":"2024-04-06T12:12:08.249234+0000","last_active":"2024-04-06T12:12:08.250618+0000","last_peered":"2024-04-06T12:12:08.250618+0000","last_clean":"2024-04-06T12:12:08.250618+0000","last_became_active":"2024-04-06T12:12:08.248705+0000","last_became_peered":"2024-04-06T12:12:08.248705+0000","last_unstale":"2024-04-06T12:12:08.250618+0000","last_undegraded":"2024-04-06T12:12:08.250618+0000","last_fullsized":"2024-04-06T12:12:08.250618+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:58:47.243168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222776+0000","last_change":"2024-04-06T12:12:03.798517+0000","last_active":"2024-04-06T12:12:08.222776+0000","last_peered":"2024-04-06T12:12:08.222776+0000","last_clean":"2024-04-06T12:12:08.222776+0000","last_became_active":"2024-04-06T12:12:03.798118+0000","last_became_peered":"2024-04-06T12:12:03.798118+0000","last_unstale":"2024-04-06T12:12:08.222776+0000","last_undegraded":"2024-04-06T12:12:08.222776+0000","last_fullsized":"2024-04-06T12:12:08.222776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:15.699305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232781+0000","last_change":"2024-04-06T12:12:04.216405+0000","last_active":"2024-04-06T12:12:08.232781+0000","last_peered":"2024-04-06T12:12:08.232781+0000","last_clean":"2024-04-06T12:12:08.232781+0000","last_became_active":"2024-04-06T12:12:04.216059+0000","last_became_peered":"2024-04-06T12:12:04.216059+0000","last_unstale":"2024-04-06T12:12:08.232781+0000","last_undegraded":"2024-04-06T12:12:08.232781+0000","last_fullsized":"2024-04-06T12:12:08.232781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.258081+0000","last_change":"2024-04-06T12:12:08.253652+0000","last_active":"2024-04-06T12:12:08.258081+0000","last_peered":"2024-04-06T12:12:08.258081+0000","last_clean":"2024-04-06T12:12:08.258081+0000","last_became_active":"2024-04-06T12:12:08.253321+0000","last_became_peered":"2024-04-06T12:12:08.253321+0000","last_unstale":"2024-04-06T12:12:08.258081+0000","last_undegraded":"2024-04-06T12:12:08.258081+0000","last_fullsized":"2024-04-06T12:12:08.258081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:31:50.315335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224175+0000","last_change":"2024-04-06T12:12:03.798088+0000","last_active":"2024-04-06T12:12:08.224175+0000","last_peered":"2024-04-06T12:12:08.224175+0000","last_clean":"2024-04-06T12:12:08.224175+0000","last_became_active":"2024-04-06T12:12:03.797844+0000","last_became_peered":"2024-04-06T12:12:03.797844+0000","last_unstale":"2024-04-06T12:12:08.224175+0000","last_undegraded":"2024-04-06T12:12:08.224175+0000","last_fullsized":"2024-04-06T12:12:08.224175+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:09.507432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223640+0000","last_change":"2024-04-06T12:12:04.224049+0000","last_active":"2024-04-06T12:12:08.223640+0000","last_peered":"2024-04-06T12:12:08.223640+0000","last_clean":"2024-04-06T12:12:08.223640+0000","last_became_active":"2024-04-06T12:12:04.223437+0000","last_became_peered":"2024-04-06T12:12:04.223437+0000","last_unstale":"2024-04-06T12:12:08.223640+0000","last_undegraded":"2024-04-06T12:12:08.223640+0000","last_fullsized":"2024-04-06T12:12:08.223640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:36:25.380472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247262+0000","last_change":"2024-04-06T12:12:08.244925+0000","last_active":"2024-04-06T12:12:08.247262+0000","last_peered":"2024-04-06T12:12:08.247262+0000","last_clean":"2024-04-06T12:12:08.247262+0000","last_became_active":"2024-04-06T12:12:08.244571+0000","last_became_peered":"2024-04-06T12:12:08.244571+0000","last_unstale":"2024-04-06T12:12:08.247262+0000","last_undegraded":"2024-04-06T12:12:08.247262+0000","last_fullsized":"2024-04-06T12:12:08.247262+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:31:50.315335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227747+0000","last_change":"2024-04-06T12:12:03.800145+0000","last_active":"2024-04-06T12:12:08.227747+0000","last_peered":"2024-04-06T12:12:08.227747+0000","last_clean":"2024-04-06T12:12:08.227747+0000","last_became_active":"2024-04-06T12:12:03.799406+0000","last_became_peered":"2024-04-06T12:12:03.799406+0000","last_unstale":"2024-04-06T12:12:08.227747+0000","last_undegraded":"2024-04-06T12:12:08.227747+0000","last_fullsized":"2024-04-06T12:12:08.227747+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:40:20.784453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224246+0000","last_change":"2024-04-06T12:12:04.226479+0000","last_active":"2024-04-06T12:12:08.224246+0000","last_peered":"2024-04-06T12:12:08.224246+0000","last_clean":"2024-04-06T12:12:08.224246+0000","last_became_active":"2024-04-06T12:12:04.222457+0000","last_became_peered":"2024-04-06T12:12:04.222457+0000","last_unstale":"2024-04-06T12:12:08.224246+0000","last_undegraded":"2024-04-06T12:12:08.224246+0000","last_fullsized":"2024-04-06T12:12:08.224246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:41:20.002927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.252006+0000","last_change":"2024-04-06T12:12:08.247112+0000","last_active":"2024-04-06T12:12:08.252006+0000","last_peered":"2024-04-06T12:12:08.252006+0000","last_clean":"2024-04-06T12:12:08.252006+0000","last_became_active":"2024-04-06T12:12:08.246705+0000","last_became_peered":"2024-04-06T12:12:08.246705+0000","last_unstale":"2024-04-06T12:12:08.252006+0000","last_undegraded":"2024-04-06T12:12:08.252006+0000","last_fullsized":"2024-04-06T12:12:08.252006+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:29.953853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222999+0000","last_change":"2024-04-06T12:12:03.799395+0000","last_active":"2024-04-06T12:12:08.222999+0000","last_peered":"2024-04-06T12:12:08.222999+0000","last_clean":"2024-04-06T12:12:08.222999+0000","last_became_active":"2024-04-06T12:12:03.799116+0000","last_became_peered":"2024-04-06T12:12:03.799116+0000","last_unstale":"2024-04-06T12:12:08.222999+0000","last_undegraded":"2024-04-06T12:12:08.222999+0000","last_fullsized":"2024-04-06T12:12:08.222999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:09.507432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227629+0000","last_change":"2024-04-06T12:12:04.221238+0000","last_active":"2024-04-06T12:12:08.227629+0000","last_peered":"2024-04-06T12:12:08.227629+0000","last_clean":"2024-04-06T12:12:08.227629+0000","last_became_active":"2024-04-06T12:12:04.220606+0000","last_became_peered":"2024-04-06T12:12:04.220606+0000","last_unstale":"2024-04-06T12:12:08.227629+0000","last_undegraded":"2024-04-06T12:12:08.227629+0000","last_fullsized":"2024-04-06T12:12:08.227629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.246867+0000","last_change":"2024-04-06T12:12:08.241508+0000","last_active":"2024-04-06T12:12:08.246867+0000","last_peered":"2024-04-06T12:12:08.246867+0000","last_clean":"2024-04-06T12:12:08.246867+0000","last_became_active":"2024-04-06T12:12:08.241184+0000","last_became_peered":"2024-04-06T12:12:08.241184+0000","last_unstale":"2024-04-06T12:12:08.246867+0000","last_undegraded":"2024-04-06T12:12:08.246867+0000","last_fullsized":"2024-04-06T12:12:08.246867+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:35:54.255608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228578+0000","last_change":"2024-04-06T12:12:03.795282+0000","last_active":"2024-04-06T12:12:08.228578+0000","last_peered":"2024-04-06T12:12:08.228578+0000","last_clean":"2024-04-06T12:12:08.228578+0000","last_became_active":"2024-04-06T12:12:03.794928+0000","last_became_peered":"2024-04-06T12:12:03.794928+0000","last_unstale":"2024-04-06T12:12:08.228578+0000","last_undegraded":"2024-04-06T12:12:08.228578+0000","last_fullsized":"2024-04-06T12:12:08.228578+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:00:31.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.225091+0000","last_change":"2024-04-06T12:12:04.224265+0000","last_active":"2024-04-06T12:12:08.225091+0000","last_peered":"2024-04-06T12:12:08.225091+0000","last_clean":"2024-04-06T12:12:08.225091+0000","last_became_active":"2024-04-06T12:12:04.223892+0000","last_became_peered":"2024-04-06T12:12:04.223892+0000","last_unstale":"2024-04-06T12:12:08.225091+0000","last_undegraded":"2024-04-06T12:12:08.225091+0000","last_fullsized":"2024-04-06T12:12:08.225091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:21:16.167775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223213+0000","last_change":"2024-04-06T12:12:04.223037+0000","last_active":"2024-04-06T12:12:08.223213+0000","last_peered":"2024-04-06T12:12:08.223213+0000","last_clean":"2024-04-06T12:12:08.223213+0000","last_became_active":"2024-04-06T12:12:04.218051+0000","last_became_peered":"2024-04-06T12:12:04.218051+0000","last_unstale":"2024-04-06T12:12:08.223213+0000","last_undegraded":"2024-04-06T12:12:08.223213+0000","last_fullsized":"2024-04-06T12:12:08.223213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:22:59.062918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227020+0000","last_change":"2024-04-06T12:12:04.214042+0000","last_active":"2024-04-06T12:12:08.227020+0000","last_peered":"2024-04-06T12:12:08.227020+0000","last_clean":"2024-04-06T12:12:08.227020+0000","last_became_active":"2024-04-06T12:12:04.213777+0000","last_became_peered":"2024-04-06T12:12:04.213777+0000","last_unstale":"2024-04-06T12:12:08.227020+0000","last_undegraded":"2024-04-06T12:12:08.227020+0000","last_fullsized":"2024-04-06T12:12:08.227020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:05.943405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228110+0000","last_change":"2024-04-06T12:12:04.216964+0000","last_active":"2024-04-06T12:12:08.228110+0000","last_peered":"2024-04-06T12:12:08.228110+0000","last_clean":"2024-04-06T12:12:08.228110+0000","last_became_active":"2024-04-06T12:12:04.216692+0000","last_became_peered":"2024-04-06T12:12:04.216692+0000","last_unstale":"2024-04-06T12:12:08.228110+0000","last_undegraded":"2024-04-06T12:12:08.228110+0000","last_fullsized":"2024-04-06T12:12:08.228110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234533+0000","last_change":"2024-04-06T12:12:04.219048+0000","last_active":"2024-04-06T12:12:08.234533+0000","last_peered":"2024-04-06T12:12:08.234533+0000","last_clean":"2024-04-06T12:12:08.234533+0000","last_became_active":"2024-04-06T12:12:04.218624+0000","last_became_peered":"2024-04-06T12:12:04.218624+0000","last_unstale":"2024-04-06T12:12:08.234533+0000","last_undegraded":"2024-04-06T12:12:08.234533+0000","last_fullsized":"2024-04-06T12:12:08.234533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:28:15.608394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.233536+0000","last_change":"2024-04-06T12:12:04.218893+0000","last_active":"2024-04-06T12:12:08.233536+0000","last_peered":"2024-04-06T12:12:08.233536+0000","last_clean":"2024-04-06T12:12:08.233536+0000","last_became_active":"2024-04-06T12:12:04.218526+0000","last_became_peered":"2024-04-06T12:12:04.218526+0000","last_unstale":"2024-04-06T12:12:08.233536+0000","last_undegraded":"2024-04-06T12:12:08.233536+0000","last_fullsized":"2024-04-06T12:12:08.233536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:15.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223037+0000","last_change":"2024-04-06T12:12:04.216794+0000","last_active":"2024-04-06T12:12:08.223037+0000","last_peered":"2024-04-06T12:12:08.223037+0000","last_clean":"2024-04-06T12:12:08.223037+0000","last_became_active":"2024-04-06T12:12:04.216532+0000","last_became_peered":"2024-04-06T12:12:04.216532+0000","last_unstale":"2024-04-06T12:12:08.223037+0000","last_undegraded":"2024-04-06T12:12:08.223037+0000","last_fullsized":"2024-04-06T12:12:08.223037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:15.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224560+0000","last_change":"2024-04-06T12:12:04.229182+0000","last_active":"2024-04-06T12:12:08.224560+0000","last_peered":"2024-04-06T12:12:08.224560+0000","last_clean":"2024-04-06T12:12:08.224560+0000","last_became_active":"2024-04-06T12:12:04.228927+0000","last_became_peered":"2024-04-06T12:12:04.228927+0000","last_unstale":"2024-04-06T12:12:08.224560+0000","last_undegraded":"2024-04-06T12:12:08.224560+0000","last_fullsized":"2024-04-06T12:12:08.224560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234558+0000","last_change":"2024-04-06T12:12:04.217003+0000","last_active":"2024-04-06T12:12:08.234558+0000","last_peered":"2024-04-06T12:12:08.234558+0000","last_clean":"2024-04-06T12:12:08.234558+0000","last_became_active":"2024-04-06T12:12:04.216757+0000","last_became_peered":"2024-04-06T12:12:04.216757+0000","last_unstale":"2024-04-06T12:12:08.234558+0000","last_undegraded":"2024-04-06T12:12:08.234558+0000","last_fullsized":"2024-04-06T12:12:08.234558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:00.258491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227358+0000","last_change":"2024-04-06T12:12:04.216442+0000","last_active":"2024-04-06T12:12:08.227358+0000","last_peered":"2024-04-06T12:12:08.227358+0000","last_clean":"2024-04-06T12:12:08.227358+0000","last_became_active":"2024-04-06T12:12:04.215051+0000","last_became_peered":"2024-04-06T12:12:04.215051+0000","last_unstale":"2024-04-06T12:12:08.227358+0000","last_undegraded":"2024-04-06T12:12:08.227358+0000","last_fullsized":"2024-04-06T12:12:08.227358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:19:38.810657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232330+0000","last_change":"2024-04-06T12:12:04.219526+0000","last_active":"2024-04-06T12:12:08.232330+0000","last_peered":"2024-04-06T12:12:08.232330+0000","last_clean":"2024-04-06T12:12:08.232330+0000","last_became_active":"2024-04-06T12:12:04.219229+0000","last_became_peered":"2024-04-06T12:12:04.219229+0000","last_unstale":"2024-04-06T12:12:08.232330+0000","last_undegraded":"2024-04-06T12:12:08.232330+0000","last_fullsized":"2024-04-06T12:12:08.232330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:22:59.062918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228186+0000","last_change":"2024-04-06T12:12:04.226116+0000","last_active":"2024-04-06T12:12:08.228186+0000","last_peered":"2024-04-06T12:12:08.228186+0000","last_clean":"2024-04-06T12:12:08.228186+0000","last_became_active":"2024-04-06T12:12:04.225820+0000","last_became_peered":"2024-04-06T12:12:04.225820+0000","last_unstale":"2024-04-06T12:12:08.228186+0000","last_undegraded":"2024-04-06T12:12:08.228186+0000","last_fullsized":"2024-04-06T12:12:08.228186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:41:20.002927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234612+0000","last_change":"2024-04-06T12:12:04.228504+0000","last_active":"2024-04-06T12:12:08.234612+0000","last_peered":"2024-04-06T12:12:08.234612+0000","last_clean":"2024-04-06T12:12:08.234612+0000","last_became_active":"2024-04-06T12:12:04.228232+0000","last_became_peered":"2024-04-06T12:12:04.228232+0000","last_unstale":"2024-04-06T12:12:08.234612+0000","last_undegraded":"2024-04-06T12:12:08.234612+0000","last_fullsized":"2024-04-06T12:12:08.234612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:22:59.062918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223649+0000","last_change":"2024-04-06T12:12:04.230394+0000","last_active":"2024-04-06T12:12:08.223649+0000","last_peered":"2024-04-06T12:12:08.223649+0000","last_clean":"2024-04-06T12:12:08.223649+0000","last_became_active":"2024-04-06T12:12:04.230065+0000","last_became_peered":"2024-04-06T12:12:04.230065+0000","last_unstale":"2024-04-06T12:12:08.223649+0000","last_undegraded":"2024-04-06T12:12:08.223649+0000","last_fullsized":"2024-04-06T12:12:08.223649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4594,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":851968,"data_stored":690730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":262144,"data_stored":102557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":98921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":856064,"data_stored":689767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":98761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":103576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":99906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":98851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:12:28.039 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-06T12:12:28.040 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-06T12:12:28.040 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-06T12:12:28.040 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-06T12:12:28.040 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-06T12:12:28.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.198+0000 7f3fe848a700 1 Processor -- start 2024-04-06T12:12:28.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.198+0000 7f3fe848a700 1 -- start start 2024-04-06T12:12:28.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.199+0000 7f3fe848a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fe00d11b0 0x7f3fe00d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.199+0000 7f3fe848a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3fe00cb870 0x7f3fe015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.199+0000 7f3fe848a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3fe00cfa00 0x7f3fe00cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.199+0000 7f3fe848a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe00ccdc0 con 0x7f3fe00cfa00 2024-04-06T12:12:28.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.199+0000 7f3fe848a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe00d0b50 con 0x7f3fe00cb870 2024-04-06T12:12:28.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.199+0000 7f3fe848a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe00d1ad0 con 0x7f3fe00d11b0 2024-04-06T12:12:28.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.199+0000 7f3fe6a27700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3fe00cfa00 0x7f3fe00cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.199+0000 7f3fe6a27700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3fe00cfa00 0x7f3fe00cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42782/0 (socket says 172.21.15.189:42782) 2024-04-06T12:12:28.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.199+0000 7f3fe6a27700 1 -- 172.21.15.189:0/541001006 learned_addr learned my addr 172.21.15.189:0/541001006 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:28.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.199+0000 7f3fe5a25700 1 --2- 172.21.15.189:0/541001006 >> v2:172.21.15.192:3300/0 conn(0x7f3fe00cb870 0x7f3fe015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.199+0000 7f3fe6226700 1 --2- 172.21.15.189:0/541001006 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fe00d11b0 0x7f3fe00d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.199+0000 7f3fe6a27700 1 -- 172.21.15.189:0/541001006 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fe00d11b0 msgr2=0x7f3fe00d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:28.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.199+0000 7f3fe6a27700 1 --2- 172.21.15.189:0/541001006 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fe00d11b0 0x7f3fe00d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.199+0000 7f3fe6a27700 1 -- 172.21.15.189:0/541001006 >> v2:172.21.15.192:3300/0 conn(0x7f3fe00cb870 msgr2=0x7f3fe015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:28.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.199+0000 7f3fe6a27700 1 --2- 172.21.15.189:0/541001006 >> v2:172.21.15.192:3300/0 conn(0x7f3fe00cb870 0x7f3fe015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.199+0000 7f3fe6a27700 1 -- 172.21.15.189:0/541001006 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe00cbd80 con 0x7f3fe00cfa00 2024-04-06T12:12:28.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.199+0000 7f3fe6a27700 1 --2- 172.21.15.189:0/541001006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe00cfa00 0x7f3fe00cd560 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3fe00ced30 tx=0x7f3fd800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f7f64dfebad77f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:28.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.200+0000 7f3fe5224700 1 -- 172.21.15.189:0/541001006 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd800e070 con 0x7f3fe00cfa00 2024-04-06T12:12:28.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.200+0000 7f3fe5224700 1 -- 172.21.15.189:0/541001006 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd800b910 con 0x7f3fe00cfa00 2024-04-06T12:12:28.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.200+0000 7f3fe5224700 1 -- 172.21.15.189:0/541001006 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd800bcc0 con 0x7f3fe00cfa00 2024-04-06T12:12:28.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.200+0000 7f3fe848a700 1 -- 172.21.15.189:0/541001006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe00cfa00 msgr2=0x7f3fe00cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:28.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.200+0000 7f3fe848a700 1 --2- 172.21.15.189:0/541001006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe00cfa00 0x7f3fe00cd560 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3fe00ced30 tx=0x7f3fd800b330 comp rx=0 tx=0).stop 2024-04-06T12:12:28.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.200+0000 7f3fe848a700 1 -- 172.21.15.189:0/541001006 shutdown_connections 2024-04-06T12:12:28.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.200+0000 7f3fe848a700 1 --2- 172.21.15.189:0/541001006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe00cfa00 0x7f3fe00cd560 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.200+0000 7f3fe848a700 1 --2- 172.21.15.189:0/541001006 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fe00d11b0 0x7f3fe00d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.200+0000 7f3fe848a700 1 --2- 172.21.15.189:0/541001006 >> v2:172.21.15.192:3300/0 conn(0x7f3fe00cb870 0x7f3fe015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.200+0000 7f3fe848a700 1 -- 172.21.15.189:0/541001006 >> 172.21.15.189:0/541001006 conn(0x7f3fe00b2db0 msgr2=0x7f3fe00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:28.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.200+0000 7f3fe848a700 1 -- 172.21.15.189:0/541001006 shutdown_connections 2024-04-06T12:12:28.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.200+0000 7f3fe848a700 1 -- 172.21.15.189:0/541001006 wait complete. 2024-04-06T12:12:28.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.201+0000 7f3fe848a700 1 Processor -- start 2024-04-06T12:12:28.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.201+0000 7f3fe848a700 1 -- start start 2024-04-06T12:12:28.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.202+0000 7f3fe848a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fe00cb870 0x7f3fe00c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.202+0000 7f3fe848a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe00cfa00 0x7f3fe0154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.202+0000 7f3fe848a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fe00d11b0 0x7f3fe01566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.202+0000 7f3fe848a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe0157460 con 0x7f3fe00cfa00 2024-04-06T12:12:28.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.202+0000 7f3fe848a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe015dd40 con 0x7f3fe00cb870 2024-04-06T12:12:28.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.202+0000 7f3fe848a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe015e020 con 0x7f3fe00d11b0 2024-04-06T12:12:28.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.202+0000 7f3fe5a25700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe00cfa00 0x7f3fe0154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.202+0000 7f3fe5a25700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe00cfa00 0x7f3fe0154760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42794/0 (socket says 172.21.15.189:42794) 2024-04-06T12:12:28.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.202+0000 7f3fe5a25700 1 -- 172.21.15.189:0/2873955699 learned_addr learned my addr 172.21.15.189:0/2873955699 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:28.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.202+0000 7f3fe6226700 1 --2- 172.21.15.189:0/2873955699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fe00cb870 0x7f3fe00c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.202+0000 7f3fe6a27700 1 --2- 172.21.15.189:0/2873955699 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fe00d11b0 0x7f3fe01566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.203+0000 7f3fe5a25700 1 -- 172.21.15.189:0/2873955699 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fe00d11b0 msgr2=0x7f3fe01566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:28.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.203+0000 7f3fe5a25700 1 --2- 172.21.15.189:0/2873955699 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fe00d11b0 0x7f3fe01566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.203+0000 7f3fe5a25700 1 -- 172.21.15.189:0/2873955699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fe00cb870 msgr2=0x7f3fe00c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:28.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.203+0000 7f3fe5a25700 1 --2- 172.21.15.189:0/2873955699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fe00cb870 0x7f3fe00c82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.203+0000 7f3fe5a25700 1 -- 172.21.15.189:0/2873955699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd8009b30 con 0x7f3fe00cfa00 2024-04-06T12:12:28.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.203+0000 7f3fe5a25700 1 --2- 172.21.15.189:0/2873955699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe00cfa00 0x7f3fe0154760 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3fd000e840 tx=0x7f3fd000eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:28.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.203+0000 7f3fd6ffd700 1 -- 172.21.15.189:0/2873955699 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0013680 con 0x7f3fe00cfa00 2024-04-06T12:12:28.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.203+0000 7f3fd6ffd700 1 -- 172.21.15.189:0/2873955699 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd00137e0 con 0x7f3fe00cfa00 2024-04-06T12:12:28.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.203+0000 7f3fd6ffd700 1 -- 172.21.15.189:0/2873955699 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0013b70 con 0x7f3fe00cfa00 2024-04-06T12:12:28.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.203+0000 7f3fe848a700 1 -- 172.21.15.189:0/2873955699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe0154ca0 con 0x7f3fe00cfa00 2024-04-06T12:12:28.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.203+0000 7f3fe848a700 1 -- 172.21.15.189:0/2873955699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe01555f0 con 0x7f3fe00cfa00 2024-04-06T12:12:28.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.204+0000 7f3fe848a700 1 -- 172.21.15.189:0/2873955699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe01575a0 con 0x7f3fe00cfa00 2024-04-06T12:12:28.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.204+0000 7f3fd6ffd700 1 -- 172.21.15.189:0/2873955699 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3fd0013cd0 con 0x7f3fe00cfa00 2024-04-06T12:12:28.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.205+0000 7f3fd6ffd700 1 --2- 172.21.15.189:0/2873955699 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3fc406ac90 0x7f3fc406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.205+0000 7f3fd6ffd700 1 -- 172.21.15.189:0/2873955699 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3fd0089320 con 0x7f3fe00cfa00 2024-04-06T12:12:28.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.205+0000 7f3fe6226700 1 --2- 172.21.15.189:0/2873955699 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3fc406ac90 0x7f3fc406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.205+0000 7f3fe6226700 1 --2- 172.21.15.189:0/2873955699 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3fc406ac90 0x7f3fc406d150 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc009be0 tx=0x7f3fdc009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:28.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.209+0000 7f3fd6ffd700 1 -- 172.21.15.189:0/2873955699 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3fe01575a0 con 0x7f3fe00cfa00 2024-04-06T12:12:28.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.463+0000 7f3fe848a700 1 -- 172.21.15.189:0/2873955699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f3fe015e1e0 con 0x7f3fe00cfa00 2024-04-06T12:12:28.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.466+0000 7f3fd6ffd700 1 -- 172.21.15.189:0/2873955699 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+170943 (secure 0 0 0) 0x7f3fd00542a0 con 0x7f3fe00cfa00 2024-04-06T12:12:28.467 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:28.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.468+0000 7f3fe848a700 1 -- 172.21.15.189:0/2873955699 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3fc406ac90 msgr2=0x7f3fc406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:28.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.468+0000 7f3fe848a700 1 --2- 172.21.15.189:0/2873955699 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3fc406ac90 0x7f3fc406d150 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc009be0 tx=0x7f3fdc009210 comp rx=0 tx=0).stop 2024-04-06T12:12:28.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.468+0000 7f3fe848a700 1 -- 172.21.15.189:0/2873955699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe00cfa00 msgr2=0x7f3fe0154760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:28.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.468+0000 7f3fe848a700 1 --2- 172.21.15.189:0/2873955699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe00cfa00 0x7f3fe0154760 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3fd000e840 tx=0x7f3fd000eb50 comp rx=0 tx=0).stop 2024-04-06T12:12:28.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.468+0000 7f3fe848a700 1 -- 172.21.15.189:0/2873955699 shutdown_connections 2024-04-06T12:12:28.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.468+0000 7f3fe848a700 1 --2- 172.21.15.189:0/2873955699 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fe00d11b0 0x7f3fe01566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.468+0000 7f3fe848a700 1 --2- 172.21.15.189:0/2873955699 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3fc406ac90 0x7f3fc406d150 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.468+0000 7f3fe848a700 1 --2- 172.21.15.189:0/2873955699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fe00cb870 0x7f3fe00c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.468+0000 7f3fe848a700 1 --2- 172.21.15.189:0/2873955699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe00cfa00 0x7f3fe0154760 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.468+0000 7f3fe848a700 1 -- 172.21.15.189:0/2873955699 >> 172.21.15.189:0/2873955699 conn(0x7f3fe00b2db0 msgr2=0x7f3fe015a550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:28.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.469+0000 7f3fe848a700 1 -- 172.21.15.189:0/2873955699 shutdown_connections 2024-04-06T12:12:28.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.469+0000 7f3fe848a700 1 -- 172.21.15.189:0/2873955699 wait complete. 2024-04-06T12:12:28.479 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":47128},{"type":"v1","addr":"172.21.15.189:6833","nonce":47128}]},"active_addr":"172.21.15.189:6833/47128","active_change":"2024-04-06T12:11:52.554619+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4105,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":3092563682}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":3015633422}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":1075995033}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":155214328}]}]} 2024-04-06T12:12:28.484 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-06T12:12:28.484 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-06T12:12:28.484 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:28.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.609+0000 7f508a21d700 1 Processor -- start 2024-04-06T12:12:28.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.609+0000 7f508a21d700 1 -- start start 2024-04-06T12:12:28.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.610+0000 7f508a21d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f508415e8a0 0x7f50840cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.610+0000 7f508a21d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f508415fa70 0x7f508406aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.610+0000 7f508a21d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50840669d0 con 0x7f508415eee0 2024-04-06T12:12:28.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.610+0000 7f508a21d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f508406ede0 con 0x7f508415e8a0 2024-04-06T12:12:28.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.610+0000 7f508a21d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f508415ec80 con 0x7f508415fa70 2024-04-06T12:12:28.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.610+0000 7f5083fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f508415fa70 0x7f508406aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.610+0000 7f5083fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f508415fa70 0x7f508406aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:50400/0 (socket says 172.21.15.189:50400) 2024-04-06T12:12:28.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.610+0000 7f50837fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f508415e8a0 0x7f50840cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.610+0000 7f5083fff700 1 -- 172.21.15.189:0/110939643 learned_addr learned my addr 172.21.15.189:0/110939643 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:28.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.610+0000 7f5082ffd700 1 --1- 172.21.15.189:0/110939643 >> v1:172.21.15.192:6789/0 conn(0x7f508415eee0 0x7f508415f360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:36594/0 (socket says 172.21.15.189:36594) 2024-04-06T12:12:28.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.610+0000 7f5083fff700 1 -- 172.21.15.189:0/110939643 >> v2:172.21.15.192:3300/0 conn(0x7f508415e8a0 msgr2=0x7f50840cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:28.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.610+0000 7f5083fff700 1 --2- 172.21.15.189:0/110939643 >> v2:172.21.15.192:3300/0 conn(0x7f508415e8a0 0x7f50840cbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.610+0000 7f5083fff700 1 -- 172.21.15.189:0/110939643 >> v1:172.21.15.192:6789/0 conn(0x7f508415eee0 legacy=0x7f508415f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:28.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.610+0000 7f5083fff700 1 -- 172.21.15.189:0/110939643 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50840666f0 con 0x7f508415fa70 2024-04-06T12:12:28.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.611+0000 7f50837fe700 1 --2- 172.21.15.189:0/110939643 >> v2:172.21.15.192:3300/0 conn(0x7f508415e8a0 0x7f50840cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:28.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.611+0000 7f5083fff700 1 --2- 172.21.15.189:0/110939643 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f508415fa70 0x7f508406aa10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f507c009e50 tx=0x7f507c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=517f27b5a014b528 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:28.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.611+0000 7f50827fc700 1 -- 172.21.15.189:0/110939643 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c00e070 con 0x7f508415fa70 2024-04-06T12:12:28.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.611+0000 7f50827fc700 1 -- 172.21.15.189:0/110939643 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507c00b910 con 0x7f508415fa70 2024-04-06T12:12:28.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.611+0000 7f50827fc700 1 -- 172.21.15.189:0/110939643 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c00bcc0 con 0x7f508415fa70 2024-04-06T12:12:28.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.611+0000 7f508a21d700 1 -- 172.21.15.189:0/110939643 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f508415fa70 msgr2=0x7f508406aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:28.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.611+0000 7f508a21d700 1 --2- 172.21.15.189:0/110939643 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f508415fa70 0x7f508406aa10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f507c009e50 tx=0x7f507c00b330 comp rx=0 tx=0).stop 2024-04-06T12:12:28.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.611+0000 7f508a21d700 1 -- 172.21.15.189:0/110939643 shutdown_connections 2024-04-06T12:12:28.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.611+0000 7f508a21d700 1 --2- 172.21.15.189:0/110939643 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f508415fa70 0x7f508406aa10 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.611+0000 7f508a21d700 1 --2- 172.21.15.189:0/110939643 >> v2:172.21.15.192:3300/0 conn(0x7f508415e8a0 0x7f50840cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.611+0000 7f508a21d700 1 -- 172.21.15.189:0/110939643 >> 172.21.15.189:0/110939643 conn(0x7f50840cc290 msgr2=0x7f50840cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:28.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.611+0000 7f508a21d700 1 -- 172.21.15.189:0/110939643 shutdown_connections 2024-04-06T12:12:28.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.611+0000 7f508a21d700 1 -- 172.21.15.189:0/110939643 wait complete. 2024-04-06T12:12:28.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.613+0000 7f508a21d700 1 Processor -- start 2024-04-06T12:12:28.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.613+0000 7f508a21d700 1 -- start start 2024-04-06T12:12:28.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.613+0000 7f508a21d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f508415e8a0 0x7f50840cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.613+0000 7f508a21d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f508415eee0 0x7f50840c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f508a21d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f508415fa70 0x7f50840c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f508a21d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50840c8cd0 con 0x7f508415eee0 2024-04-06T12:12:28.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f508a21d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50840cda30 con 0x7f508415e8a0 2024-04-06T12:12:28.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f508a21d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50840c79d0 con 0x7f508415fa70 2024-04-06T12:12:28.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f5082ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f508415eee0 0x7f50840c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f5082ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f508415eee0 0x7f50840c93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42810/0 (socket says 172.21.15.189:42810) 2024-04-06T12:12:28.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f50837fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f508415e8a0 0x7f50840cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f5082ffd700 1 -- 172.21.15.189:0/1790610250 learned_addr learned my addr 172.21.15.189:0/1790610250 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:28.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f5083fff700 1 --2- 172.21.15.189:0/1790610250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f508415fa70 0x7f50840c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f5082ffd700 1 -- 172.21.15.189:0/1790610250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f508415fa70 msgr2=0x7f50840c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:28.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f5082ffd700 1 --2- 172.21.15.189:0/1790610250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f508415fa70 0x7f50840c8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f5082ffd700 1 -- 172.21.15.189:0/1790610250 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f508415e8a0 msgr2=0x7f50840cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:28.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f5082ffd700 1 --2- 172.21.15.189:0/1790610250 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f508415e8a0 0x7f50840cd3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f5082ffd700 1 -- 172.21.15.189:0/1790610250 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506c0094a0 con 0x7f508415eee0 2024-04-06T12:12:28.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f50837fe700 1 --2- 172.21.15.189:0/1790610250 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f508415e8a0 0x7f50840cd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:28.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f5082ffd700 1 --2- 172.21.15.189:0/1790610250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f508415eee0 0x7f50840c93e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f507400c520 tx=0x7f507400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:28.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f507bfff700 1 -- 172.21.15.189:0/1790610250 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5074005cb0 con 0x7f508415eee0 2024-04-06T12:12:28.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f507bfff700 1 -- 172.21.15.189:0/1790610250 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5074005e10 con 0x7f508415eee0 2024-04-06T12:12:28.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f507bfff700 1 -- 172.21.15.189:0/1790610250 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50740072d0 con 0x7f508415eee0 2024-04-06T12:12:28.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f508a21d700 1 -- 172.21.15.189:0/1790610250 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507c009b30 con 0x7f508415eee0 2024-04-06T12:12:28.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.614+0000 7f508a21d700 1 -- 172.21.15.189:0/1790610250 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50840c7f90 con 0x7f508415eee0 2024-04-06T12:12:28.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.615+0000 7f508a21d700 1 -- 172.21.15.189:0/1790610250 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5084072140 con 0x7f508415eee0 2024-04-06T12:12:28.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.616+0000 7f507bfff700 1 -- 172.21.15.189:0/1790610250 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5074007430 con 0x7f508415eee0 2024-04-06T12:12:28.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.616+0000 7f507bfff700 1 --2- 172.21.15.189:0/1790610250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f506406acc0 0x7f506406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.616+0000 7f507bfff700 1 -- 172.21.15.189:0/1790610250 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5074088920 con 0x7f508415eee0 2024-04-06T12:12:28.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.616+0000 7f50837fe700 1 --2- 172.21.15.189:0/1790610250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f506406acc0 0x7f506406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.617+0000 7f50837fe700 1 --2- 172.21.15.189:0/1790610250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f506406acc0 0x7f506406d180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f506c0114b0 tx=0x7f506c009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:28.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.621+0000 7f507bfff700 1 -- 172.21.15.189:0/1790610250 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5074053820 con 0x7f508415eee0 2024-04-06T12:12:28.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.816+0000 7f508a21d700 1 -- 172.21.15.189:0/1790610250 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50840ca0b0 con 0x7f508415eee0 2024-04-06T12:12:28.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.818+0000 7f507bfff700 1 -- 172.21.15.189:0/1790610250 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f5074017090 con 0x7f508415eee0 2024-04-06T12:12:28.817 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:28.817 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:08.649170+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:28.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.820+0000 7f508a21d700 1 -- 172.21.15.189:0/1790610250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f506406acc0 msgr2=0x7f506406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:28.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.820+0000 7f508a21d700 1 --2- 172.21.15.189:0/1790610250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f506406acc0 0x7f506406d180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f506c0114b0 tx=0x7f506c009210 comp rx=0 tx=0).stop 2024-04-06T12:12:28.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.820+0000 7f508a21d700 1 -- 172.21.15.189:0/1790610250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f508415eee0 msgr2=0x7f50840c93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:28.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.820+0000 7f508a21d700 1 --2- 172.21.15.189:0/1790610250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f508415eee0 0x7f50840c93e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f507400c520 tx=0x7f507400c830 comp rx=0 tx=0).stop 2024-04-06T12:12:28.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.820+0000 7f508a21d700 1 -- 172.21.15.189:0/1790610250 shutdown_connections 2024-04-06T12:12:28.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.820+0000 7f508a21d700 1 --2- 172.21.15.189:0/1790610250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f508415fa70 0x7f50840c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.820+0000 7f508a21d700 1 --2- 172.21.15.189:0/1790610250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f506406acc0 0x7f506406d180 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.820+0000 7f508a21d700 1 --2- 172.21.15.189:0/1790610250 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f508415e8a0 0x7f50840cd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.820+0000 7f508a21d700 1 --2- 172.21.15.189:0/1790610250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f508415eee0 0x7f50840c93e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.820+0000 7f508a21d700 1 -- 172.21.15.189:0/1790610250 >> 172.21.15.189:0/1790610250 conn(0x7f50840cc290 msgr2=0x7f508406b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:28.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.820+0000 7f508a21d700 1 -- 172.21.15.189:0/1790610250 shutdown_connections 2024-04-06T12:12:28.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.820+0000 7f508a21d700 1 -- 172.21.15.189:0/1790610250 wait complete. 2024-04-06T12:12:28.831 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-06T12:12:28.831 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:28.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.954+0000 7ff8394ec700 1 Processor -- start 2024-04-06T12:12:28.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.954+0000 7ff8394ec700 1 -- start start 2024-04-06T12:12:28.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.954+0000 7ff8394ec700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8340b2cd0 0x7ff8340cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.954+0000 7ff8394ec700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff83415d040 0x7ff83415dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.954+0000 7ff8394ec700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff83415e2a0 0x7ff83406c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.954+0000 7ff8394ec700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8340cba40 con 0x7ff83415e2a0 2024-04-06T12:12:28.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.954+0000 7ff8394ec700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff83415d8d0 con 0x7ff83415d040 2024-04-06T12:12:28.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.954+0000 7ff8394ec700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83415d580 con 0x7ff8340b2cd0 2024-04-06T12:12:28.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.955+0000 7ff8337fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff83415e2a0 0x7ff83406c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.955+0000 7ff8337fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff83415e2a0 0x7ff83406c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42822/0 (socket says 172.21.15.189:42822) 2024-04-06T12:12:28.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.955+0000 7ff8337fe700 1 -- 172.21.15.189:0/3205376080 learned_addr learned my addr 172.21.15.189:0/3205376080 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:28.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.955+0000 7ff8327fc700 1 --2- 172.21.15.189:0/3205376080 >> v2:172.21.15.192:3300/0 conn(0x7ff83415d040 0x7ff83415dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.955+0000 7ff832ffd700 1 --2- 172.21.15.189:0/3205376080 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8340b2cd0 0x7ff8340cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.955+0000 7ff8337fe700 1 -- 172.21.15.189:0/3205376080 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8340b2cd0 msgr2=0x7ff8340cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:28.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.955+0000 7ff8337fe700 1 --2- 172.21.15.189:0/3205376080 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8340b2cd0 0x7ff8340cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.955+0000 7ff8337fe700 1 -- 172.21.15.189:0/3205376080 >> v2:172.21.15.192:3300/0 conn(0x7ff83415d040 msgr2=0x7ff83415dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:28.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.955+0000 7ff8337fe700 1 --2- 172.21.15.189:0/3205376080 >> v2:172.21.15.192:3300/0 conn(0x7ff83415d040 0x7ff83415dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.955+0000 7ff8337fe700 1 -- 172.21.15.189:0/3205376080 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8340b3240 con 0x7ff83415e2a0 2024-04-06T12:12:28.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.955+0000 7ff832ffd700 1 --2- 172.21.15.189:0/3205376080 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8340b2cd0 0x7ff8340cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:28.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.955+0000 7ff8337fe700 1 --2- 172.21.15.189:0/3205376080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff83415e2a0 0x7ff83406c4e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff83406e860 tx=0x7ff82c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=150c996e88283caf server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:28.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.955+0000 7ff831ffb700 1 -- 172.21.15.189:0/3205376080 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82c00e070 con 0x7ff83415e2a0 2024-04-06T12:12:28.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.955+0000 7ff831ffb700 1 -- 172.21.15.189:0/3205376080 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82c00b8d0 con 0x7ff83415e2a0 2024-04-06T12:12:28.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.955+0000 7ff831ffb700 1 -- 172.21.15.189:0/3205376080 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82c00bc80 con 0x7ff83415e2a0 2024-04-06T12:12:28.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.956+0000 7ff8394ec700 1 -- 172.21.15.189:0/3205376080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff83415e2a0 msgr2=0x7ff83406c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:28.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.956+0000 7ff8394ec700 1 --2- 172.21.15.189:0/3205376080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff83415e2a0 0x7ff83406c4e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff83406e860 tx=0x7ff82c00b330 comp rx=0 tx=0).stop 2024-04-06T12:12:28.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.956+0000 7ff8394ec700 1 -- 172.21.15.189:0/3205376080 shutdown_connections 2024-04-06T12:12:28.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.956+0000 7ff8394ec700 1 --2- 172.21.15.189:0/3205376080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff83415e2a0 0x7ff83406c4e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.956+0000 7ff8394ec700 1 --2- 172.21.15.189:0/3205376080 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8340b2cd0 0x7ff8340cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.956+0000 7ff8394ec700 1 --2- 172.21.15.189:0/3205376080 >> v2:172.21.15.192:3300/0 conn(0x7ff83415d040 0x7ff83415dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.956+0000 7ff8394ec700 1 -- 172.21.15.189:0/3205376080 >> 172.21.15.189:0/3205376080 conn(0x7ff83406a730 msgr2=0x7ff83406ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:28.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.956+0000 7ff8394ec700 1 -- 172.21.15.189:0/3205376080 shutdown_connections 2024-04-06T12:12:28.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.956+0000 7ff8394ec700 1 -- 172.21.15.189:0/3205376080 wait complete. 2024-04-06T12:12:28.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.957+0000 7ff8394ec700 1 Processor -- start 2024-04-06T12:12:28.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.957+0000 7ff8394ec700 1 -- start start 2024-04-06T12:12:28.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.958+0000 7ff8394ec700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8340b2cd0 0x7ff8341f0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.958+0000 7ff8394ec700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff83415d040 0x7ff83414ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.958+0000 7ff8394ec700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff83414eef0 0x7ff834149860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.958+0000 7ff8394ec700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff834151300 con 0x7ff8340b2cd0 2024-04-06T12:12:28.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.958+0000 7ff8394ec700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8341f1160 con 0x7ff83414eef0 2024-04-06T12:12:28.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.958+0000 7ff8394ec700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff834151570 con 0x7ff83415d040 2024-04-06T12:12:28.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.958+0000 7ff832ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8340b2cd0 0x7ff8341f0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.958+0000 7ff832ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8340b2cd0 0x7ff8341f0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42830/0 (socket says 172.21.15.189:42830) 2024-04-06T12:12:28.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.958+0000 7ff832ffd700 1 -- 172.21.15.189:0/324314416 learned_addr learned my addr 172.21.15.189:0/324314416 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:28.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.958+0000 7ff8337fe700 1 --2- 172.21.15.189:0/324314416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff83414eef0 0x7ff834149860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.958+0000 7ff8327fc700 1 --2- 172.21.15.189:0/324314416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff83415d040 0x7ff83414ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.958+0000 7ff832ffd700 1 -- 172.21.15.189:0/324314416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff83415d040 msgr2=0x7ff83414ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:28.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.958+0000 7ff832ffd700 1 --2- 172.21.15.189:0/324314416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff83415d040 0x7ff83414ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.958+0000 7ff832ffd700 1 -- 172.21.15.189:0/324314416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff83414eef0 msgr2=0x7ff834149860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:28.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.958+0000 7ff832ffd700 1 --2- 172.21.15.189:0/324314416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff83414eef0 0x7ff834149860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:28.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.958+0000 7ff832ffd700 1 -- 172.21.15.189:0/324314416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c009b30 con 0x7ff8340b2cd0 2024-04-06T12:12:28.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.958+0000 7ff8337fe700 1 --2- 172.21.15.189:0/324314416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff83414eef0 0x7ff834149860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:28.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.958+0000 7ff832ffd700 1 --2- 172.21.15.189:0/324314416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8340b2cd0 0x7ff8341f0c20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff82400d830 tx=0x7ff82400dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:28.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.958+0000 7ff81b7fe700 1 -- 172.21.15.189:0/324314416 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff824004030 con 0x7ff8340b2cd0 2024-04-06T12:12:28.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.959+0000 7ff81b7fe700 1 -- 172.21.15.189:0/324314416 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff824004190 con 0x7ff8340b2cd0 2024-04-06T12:12:28.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.959+0000 7ff8394ec700 1 -- 172.21.15.189:0/324314416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff834151850 con 0x7ff8340b2cd0 2024-04-06T12:12:28.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.959+0000 7ff81b7fe700 1 -- 172.21.15.189:0/324314416 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff824004520 con 0x7ff8340b2cd0 2024-04-06T12:12:28.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.959+0000 7ff8394ec700 1 -- 172.21.15.189:0/324314416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff83414b660 con 0x7ff8340b2cd0 2024-04-06T12:12:28.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.960+0000 7ff8394ec700 1 -- 172.21.15.189:0/324314416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff834073270 con 0x7ff8340b2cd0 2024-04-06T12:12:28.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.960+0000 7ff81b7fe700 1 -- 172.21.15.189:0/324314416 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff8240036a0 con 0x7ff8340b2cd0 2024-04-06T12:12:28.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.960+0000 7ff81b7fe700 1 --2- 172.21.15.189:0/324314416 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff82006ac90 0x7ff82006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:28.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.961+0000 7ff8327fc700 1 --2- 172.21.15.189:0/324314416 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff82006ac90 0x7ff82006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:28.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.961+0000 7ff81b7fe700 1 -- 172.21.15.189:0/324314416 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff824089c00 con 0x7ff8340b2cd0 2024-04-06T12:12:28.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.961+0000 7ff8327fc700 1 --2- 172.21.15.189:0/324314416 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff82006ac90 0x7ff82006d150 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff8340af050 tx=0x7ff828008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:28.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:28.966+0000 7ff81b7fe700 1 -- 172.21.15.189:0/324314416 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff824090080 con 0x7ff8340b2cd0 2024-04-06T12:12:29.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.160+0000 7ff8394ec700 1 -- 172.21.15.189:0/324314416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff834074990 con 0x7ff8340b2cd0 2024-04-06T12:12:29.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.161+0000 7ff81b7fe700 1 -- 172.21.15.189:0/324314416 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17926 (secure 0 0 0) 0x7ff824054af0 con 0x7ff8340b2cd0 2024-04-06T12:12:29.160 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:29.161 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:08.649170+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:29.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.163+0000 7ff8394ec700 1 -- 172.21.15.189:0/324314416 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff82006ac90 msgr2=0x7ff82006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.163+0000 7ff8394ec700 1 --2- 172.21.15.189:0/324314416 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff82006ac90 0x7ff82006d150 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff8340af050 tx=0x7ff828008040 comp rx=0 tx=0).stop 2024-04-06T12:12:29.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.163+0000 7ff8394ec700 1 -- 172.21.15.189:0/324314416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8340b2cd0 msgr2=0x7ff8341f0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.163+0000 7ff8394ec700 1 --2- 172.21.15.189:0/324314416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8340b2cd0 0x7ff8341f0c20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff82400d830 tx=0x7ff82400dbf0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.164+0000 7ff8394ec700 1 -- 172.21.15.189:0/324314416 shutdown_connections 2024-04-06T12:12:29.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.164+0000 7ff8394ec700 1 --2- 172.21.15.189:0/324314416 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff82006ac90 0x7ff82006d150 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.164+0000 7ff8394ec700 1 --2- 172.21.15.189:0/324314416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff83414eef0 0x7ff834149860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.164+0000 7ff8394ec700 1 --2- 172.21.15.189:0/324314416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8340b2cd0 0x7ff8341f0c20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.164+0000 7ff8394ec700 1 --2- 172.21.15.189:0/324314416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff83415d040 0x7ff83414ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.164+0000 7ff8394ec700 1 -- 172.21.15.189:0/324314416 >> 172.21.15.189:0/324314416 conn(0x7ff83406a730 msgr2=0x7ff8340b0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.164+0000 7ff8394ec700 1 -- 172.21.15.189:0/324314416 shutdown_connections 2024-04-06T12:12:29.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.164+0000 7ff8394ec700 1 -- 172.21.15.189:0/324314416 wait complete. 2024-04-06T12:12:29.175 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-06T12:12:29.176 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-06T12:12:29.176 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-06T12:12:29.177 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-06T12:12:29.177 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-06T12:12:29.177 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-06T12:12:29.178 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-06T12:12:29.178 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-06T12:12:29.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.393+0000 7f26b96a4700 1 Processor -- start 2024-04-06T12:12:29.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.393+0000 7f26b96a4700 1 -- start start 2024-04-06T12:12:29.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.394+0000 7f26b96a4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26b4150690 0x7f26b4159b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.394+0000 7f26b96a4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f26b406e130 0x7f26b415b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.394+0000 7f26b96a4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b415a180 con 0x7f26b40b2300 2024-04-06T12:12:29.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.394+0000 7f26b96a4700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b4150a70 con 0x7f26b406e130 2024-04-06T12:12:29.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.394+0000 7f26b96a4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b415a460 con 0x7f26b4150690 2024-04-06T12:12:29.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.394+0000 7f26b2ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26b4150690 0x7f26b4159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.394+0000 7f25d664d700 1 Processor -- start 2024-04-06T12:12:29.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.395+0000 7f25d664d700 1 -- start start 2024-04-06T12:12:29.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.395+0000 7f26b2ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26b4150690 0x7f26b4159b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59320/0 (socket says 172.21.15.189:59320) 2024-04-06T12:12:29.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.395+0000 7f26b2ffd700 1 -- 172.21.15.189:0/2295568361 learned_addr learned my addr 172.21.15.189:0/2295568361 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.395+0000 7f26b37fe700 1 --2- 172.21.15.189:0/2295568361 >> v2:172.21.15.189:3300/0 conn(0x7f26b406e130 0x7f26b415b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.395+0000 7f26b1ffb700 1 -- 172.21.15.189:0/2295568361 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983460423 0 0) 0x7f26b415a180 con 0x7f26b40b2300 2024-04-06T12:12:29.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.395+0000 7f26b1ffb700 1 -- 172.21.15.189:0/2295568361 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f269c0035b0 con 0x7f26b40b2300 2024-04-06T12:12:29.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.395+0000 7f26b37fe700 1 -- 172.21.15.189:0/2295568361 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26b4150690 msgr2=0x7f26b4159b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:29.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.395+0000 7f26b37fe700 1 --2- 172.21.15.189:0/2295568361 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26b4150690 0x7f26b4159b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.395+0000 7f26b37fe700 1 -- 172.21.15.189:0/2295568361 >> v1:172.21.15.192:6789/0 conn(0x7f26b40b2300 legacy=0x7f26b406da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.395+0000 7f26b37fe700 1 -- 172.21.15.189:0/2295568361 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b415a740 con 0x7f26b406e130 2024-04-06T12:12:29.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.395+0000 7f26b37fe700 1 --2- 172.21.15.189:0/2295568361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b406e130 0x7f26b415b220 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f26a8009840 tx=0x7f26a8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e85bebe436bfb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.396+0000 7f26b1ffb700 1 -- 172.21.15.189:0/2295568361 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a8010a90 con 0x7f26b406e130 2024-04-06T12:12:29.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.396+0000 7f26b1ffb700 1 -- 172.21.15.189:0/2295568361 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a8010bf0 con 0x7f26b406e130 2024-04-06T12:12:29.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.396+0000 7f26b1ffb700 1 -- 172.21.15.189:0/2295568361 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a8018440 con 0x7f26b406e130 2024-04-06T12:12:29.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.396+0000 7f25d664d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25d006ae40 0x7f25d015ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.396+0000 7f25d664d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f25d015fcd0 0x7f25d015bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.396+0000 7f25d664d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d015c4c0 con 0x7f25d0144f40 2024-04-06T12:12:29.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.396+0000 7f25d664d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d00cfcb0 con 0x7f25d015fcd0 2024-04-06T12:12:29.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.396+0000 7f25d664d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d015ca20 con 0x7f25d006ae40 2024-04-06T12:12:29.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.396+0000 7f25d4bea700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f25d015fcd0 0x7f25d015bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.396+0000 7f25d4bea700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f25d015fcd0 0x7f25d015bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:58734/0 (socket says 172.21.15.189:58734) 2024-04-06T12:12:29.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.396+0000 7f25cffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f25d0144f40 0x7f25d006a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34208/0 (socket says 172.21.15.189:34208) 2024-04-06T12:12:29.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.396+0000 7f25d4bea700 1 -- 172.21.15.189:0/942217188 learned_addr learned my addr 172.21.15.189:0/942217188 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.396+0000 7f25cf7fe700 1 --2- 172.21.15.189:0/942217188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25d006ae40 0x7f25d015ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.396+0000 7f25cf7fe700 1 -- 172.21.15.189:0/942217188 >> v2:172.21.15.192:3300/0 conn(0x7f25d015fcd0 msgr2=0x7f25d015bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:29.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.396+0000 7f25cf7fe700 1 --2- 172.21.15.189:0/942217188 >> v2:172.21.15.192:3300/0 conn(0x7f25d015fcd0 0x7f25d015bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.396+0000 7f25cf7fe700 1 -- 172.21.15.189:0/942217188 >> v1:172.21.15.189:6789/0 conn(0x7f25d0144f40 legacy=0x7f25d006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.396+0000 7f25cf7fe700 1 -- 172.21.15.189:0/942217188 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d015c740 con 0x7f25d006ae40 2024-04-06T12:12:29.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.397+0000 7f26b96a4700 1 -- 172.21.15.189:0/2295568361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b406e130 msgr2=0x7f26b415b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.397+0000 7f26b96a4700 1 --2- 172.21.15.189:0/2295568361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b406e130 0x7f26b415b220 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f26a8009840 tx=0x7f26a8009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:29.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.397+0000 7f25cf7fe700 1 --2- 172.21.15.189:0/942217188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25d006ae40 0x7f25d015ba40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f25c0009840 tx=0x7f25c0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e87ece6c984cf17b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.398+0000 7f26b96a4700 1 -- 172.21.15.189:0/2295568361 shutdown_connections 2024-04-06T12:12:29.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.398+0000 7f26b96a4700 1 --2- 172.21.15.189:0/2295568361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b406e130 0x7f26b415b220 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.398+0000 7f26b96a4700 1 --2- 172.21.15.189:0/2295568361 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26b4150690 0x7f26b4159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.398+0000 7f26b96a4700 1 -- 172.21.15.189:0/2295568361 >> 172.21.15.189:0/2295568361 conn(0x7f26b40cefa0 msgr2=0x7f26b40cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.398+0000 7fc6bbabd700 1 Processor -- start 2024-04-06T12:12:29.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.399+0000 7fe85c81f700 1 Processor -- start 2024-04-06T12:12:29.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.398+0000 7fc6bbabd700 1 -- start start 2024-04-06T12:12:29.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.399+0000 7fc6bbabd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6b4159b30 0x7fc6b406da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.399+0000 7fc6bbabd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc6b406df60 0x7fc6b415b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.399+0000 7fc6bbabd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b406e3b0 con 0x7fc6b4150690 2024-04-06T12:12:29.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.399+0000 7fc6bbabd700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6b4150a70 con 0x7fc6b406df60 2024-04-06T12:12:29.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.399+0000 7fc6bbabd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b4159f90 con 0x7fc6b4159b30 2024-04-06T12:12:29.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.399+0000 7fc6b9859700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc6b4150690 0x7fc6b40b2300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34222/0 (socket says 172.21.15.189:34222) 2024-04-06T12:12:29.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.399+0000 7fc6b9859700 1 -- 172.21.15.189:0/347425761 learned_addr learned my addr 172.21.15.189:0/347425761 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:29.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.399+0000 7fc6b9058700 1 --2- 172.21.15.189:0/347425761 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6b4159b30 0x7fc6b406da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.399+0000 7fc6ba05a700 1 --2- 172.21.15.189:0/347425761 >> v2:172.21.15.192:3300/0 conn(0x7fc6b406df60 0x7fc6b415b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.399+0000 7fc6b8857700 1 -- 172.21.15.189:0/347425761 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500982659 0 0) 0x7fc6a4004b40 con 0x7fc6b4150690 2024-04-06T12:12:29.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.399+0000 7fc6b8857700 1 -- 172.21.15.189:0/347425761 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6a00035b0 con 0x7fc6b4150690 2024-04-06T12:12:29.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.399+0000 7fc6b9058700 1 -- 172.21.15.189:0/347425761 >> v2:172.21.15.192:3300/0 conn(0x7fc6b406df60 msgr2=0x7fc6b415b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:29.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.399+0000 7fc6b9058700 1 --2- 172.21.15.189:0/347425761 >> v2:172.21.15.192:3300/0 conn(0x7fc6b406df60 0x7fc6b415b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.399+0000 7fc6b9058700 1 -- 172.21.15.189:0/347425761 >> v1:172.21.15.189:6789/0 conn(0x7fc6b4150690 legacy=0x7fc6b40b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.399+0000 7fc6b9058700 1 -- 172.21.15.189:0/347425761 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b415a180 con 0x7fc6b4159b30 2024-04-06T12:12:29.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fc6b9058700 1 --2- 172.21.15.189:0/347425761 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6b4159b30 0x7fc6b406da20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac00be90 tx=0x7fc6ac00d330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec97e5749e56b5ff server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.399+0000 7fe85c81f700 1 -- start start 2024-04-06T12:12:29.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fe85c81f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe858159b20 0x7fe85806b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fe85c81f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8580cdc60 0x7fe8580ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fe85c81f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85806a8d0 con 0x7fe85806e470 2024-04-06T12:12:29.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fe85c81f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8580ce620 con 0x7fe858159b20 2024-04-06T12:12:29.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fe85c81f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe858159f00 con 0x7fe8580cdc60 2024-04-06T12:12:29.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fe85659c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe858159b20 0x7fe85806b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fe85659c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe858159b20 0x7fe85806b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38030/0 (socket says 172.21.15.189:38030) 2024-04-06T12:12:29.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fe85659c700 1 -- 172.21.15.189:0/4131443604 learned_addr learned my addr 172.21.15.189:0/4131443604 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fe855d9b700 1 --2- 172.21.15.189:0/4131443604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8580cdc60 0x7fe8580ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fe85559a700 1 -- 172.21.15.189:0/4131443604 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1586488283 0 0) 0x7fe85806a8d0 con 0x7fe85806e470 2024-04-06T12:12:29.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fe85559a700 1 -- 172.21.15.189:0/4131443604 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8400035b0 con 0x7fe85806e470 2024-04-06T12:12:29.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fe85659c700 1 -- 172.21.15.189:0/4131443604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8580cdc60 msgr2=0x7fe8580ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:29.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fe85659c700 1 --2- 172.21.15.189:0/4131443604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8580cdc60 0x7fe8580ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fe85659c700 1 -- 172.21.15.189:0/4131443604 >> v1:172.21.15.189:6789/0 conn(0x7fe85806e470 legacy=0x7fe85806e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fe85659c700 1 -- 172.21.15.189:0/4131443604 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85806abb0 con 0x7fe858159b20 2024-04-06T12:12:29.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fe855d9b700 1 --2- 172.21.15.189:0/4131443604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8580cdc60 0x7fe8580ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:29.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.401+0000 7fe85659c700 1 --2- 172.21.15.189:0/4131443604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe858159b20 0x7fe85806b970 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe84800aa90 tx=0x7fe84800ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30282fd57c18c8e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.401+0000 7fe85559a700 1 -- 172.21.15.189:0/4131443604 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84800cb70 con 0x7fe858159b20 2024-04-06T12:12:29.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fc6b8857700 1 -- 172.21.15.189:0/347425761 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac010070 con 0x7fc6b4159b30 2024-04-06T12:12:29.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fc6b8857700 1 -- 172.21.15.189:0/347425761 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ac00d910 con 0x7fc6b4159b30 2024-04-06T12:12:29.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.400+0000 7fc6b8857700 1 -- 172.21.15.189:0/347425761 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac00dc80 con 0x7fc6b4159b30 2024-04-06T12:12:29.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.401+0000 7fc6bbabd700 1 -- 172.21.15.189:0/347425761 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6b4159b30 msgr2=0x7fc6b406da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.401+0000 7fc6bbabd700 1 --2- 172.21.15.189:0/347425761 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6b4159b30 0x7fc6b406da20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac00be90 tx=0x7fc6ac00d330 comp rx=0 tx=0).stop 2024-04-06T12:12:29.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.401+0000 7fe85559a700 1 -- 172.21.15.189:0/4131443604 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84800ccd0 con 0x7fe858159b20 2024-04-06T12:12:29.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.401+0000 7fe85559a700 1 -- 172.21.15.189:0/4131443604 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84800f690 con 0x7fe858159b20 2024-04-06T12:12:29.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.401+0000 7fc6bbabd700 1 -- 172.21.15.189:0/347425761 shutdown_connections 2024-04-06T12:12:29.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.401+0000 7fc6bbabd700 1 --2- 172.21.15.189:0/347425761 >> v2:172.21.15.192:3300/0 conn(0x7fc6b406df60 0x7fc6b415b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.401+0000 7fc6bbabd700 1 --2- 172.21.15.189:0/347425761 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6b4159b30 0x7fc6b406da20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.401+0000 7fc6bbabd700 1 -- 172.21.15.189:0/347425761 >> 172.21.15.189:0/347425761 conn(0x7fc6b40cefa0 msgr2=0x7fc6b40cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.401+0000 7fc6bbabd700 1 -- 172.21.15.189:0/347425761 shutdown_connections 2024-04-06T12:12:29.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.401+0000 7f26b96a4700 1 -- 172.21.15.189:0/2295568361 shutdown_connections 2024-04-06T12:12:29.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.401+0000 7fc6bbabd700 1 -- 172.21.15.189:0/347425761 wait complete. 2024-04-06T12:12:29.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.402+0000 7fe85c81f700 1 -- 172.21.15.189:0/4131443604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe858159b20 msgr2=0x7fe85806b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.402+0000 7fe85c81f700 1 --2- 172.21.15.189:0/4131443604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe858159b20 0x7fe85806b970 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe84800aa90 tx=0x7fe84800ada0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.402+0000 7fe85c81f700 1 -- 172.21.15.189:0/4131443604 shutdown_connections 2024-04-06T12:12:29.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.402+0000 7fe85c81f700 1 --2- 172.21.15.189:0/4131443604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe858159b20 0x7fe85806b970 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.402+0000 7fe85c81f700 1 --2- 172.21.15.189:0/4131443604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8580cdc60 0x7fe8580ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.402+0000 7fe85c81f700 1 -- 172.21.15.189:0/4131443604 >> 172.21.15.189:0/4131443604 conn(0x7fe8580cc290 msgr2=0x7fe8580cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.402+0000 7fe85c81f700 1 -- 172.21.15.189:0/4131443604 shutdown_connections 2024-04-06T12:12:29.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.402+0000 7fe85c81f700 1 -- 172.21.15.189:0/4131443604 wait complete. 2024-04-06T12:12:29.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.403+0000 7fe85c81f700 1 Processor -- start 2024-04-06T12:12:29.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.403+0000 7fc6bbabd700 1 Processor -- start 2024-04-06T12:12:29.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.403+0000 7f0229f11700 1 Processor -- start 2024-04-06T12:12:29.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.402+0000 7f26b96a4700 1 -- 172.21.15.189:0/2295568361 wait complete. 2024-04-06T12:12:29.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7f26b96a4700 1 Processor -- start 2024-04-06T12:12:29.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.403+0000 7fe85c81f700 1 -- start start 2024-04-06T12:12:29.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7fe85c81f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe85806e470 0x7fe858158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7fe85c81f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8580cdc60 0x7fe8580c27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7fe85c81f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe858159b20 0x7fe85815c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7fe85c81f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85815b6b0 con 0x7fe8580cdc60 2024-04-06T12:12:29.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7fe85c81f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85815b990 con 0x7fe858159b20 2024-04-06T12:12:29.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7fe85c81f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85815bc70 con 0x7fe85806e470 2024-04-06T12:12:29.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.403+0000 7fc6bbabd700 1 -- start start 2024-04-06T12:12:29.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7fc6bbabd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6b406df60 0x7fc6b414d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7fc6bbabd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6b4150690 0x7fc6b4140620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7fc6bbabd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6b4159b30 0x7fc6b40cd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7fc6bbabd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b41fa4b0 con 0x7fc6b4150690 2024-04-06T12:12:29.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7fc6bbabd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b4153790 con 0x7fc6b4159b30 2024-04-06T12:12:29.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7fc6bbabd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b41fa690 con 0x7fc6b406df60 2024-04-06T12:12:29.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7fc6b9058700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6b4150690 0x7fc6b4140620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7fc6b9058700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6b4150690 0x7fc6b4140620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38032/0 (socket says 172.21.15.189:38032) 2024-04-06T12:12:29.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7fc6b9058700 1 -- 172.21.15.189:0/1123577174 learned_addr learned my addr 172.21.15.189:0/1123577174 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7fc6ba05a700 1 --2- 172.21.15.189:0/1123577174 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6b4159b30 0x7fc6b40cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7f024f17b700 1 Processor -- start 2024-04-06T12:12:29.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7fe85659c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe85806e470 0x7fe858158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7fe85659c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe85806e470 0x7fe858158cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59352/0 (socket says 172.21.15.189:59352) 2024-04-06T12:12:29.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7fe85659c700 1 -- 172.21.15.189:0/636056931 learned_addr learned my addr 172.21.15.189:0/636056931 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7fe856d9d700 1 --2- 172.21.15.189:0/636056931 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe858159b20 0x7fe85815c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7fc6b9058700 1 -- 172.21.15.189:0/1123577174 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6b406df60 msgr2=0x7fc6b414d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7fc6b9058700 1 --2- 172.21.15.189:0/1123577174 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6b406df60 0x7fc6b414d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7fc6b9058700 1 -- 172.21.15.189:0/1123577174 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6b4159b30 msgr2=0x7fc6b40cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7fc6b9058700 1 --2- 172.21.15.189:0/1123577174 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6b4159b30 0x7fc6b40cd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7fc6b9058700 1 -- 172.21.15.189:0/1123577174 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ac00bb70 con 0x7fc6b4150690 2024-04-06T12:12:29.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7fe855d9b700 1 --2- 172.21.15.189:0/636056931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8580cdc60 0x7fe8580c27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7fe85659c700 1 -- 172.21.15.189:0/636056931 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe858159b20 msgr2=0x7fe85815c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7fe85659c700 1 --2- 172.21.15.189:0/636056931 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe858159b20 0x7fe85815c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7fe85659c700 1 -- 172.21.15.189:0/636056931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8580cdc60 msgr2=0x7fe8580c27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7fe85659c700 1 --2- 172.21.15.189:0/636056931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8580cdc60 0x7fe8580c27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7fe85659c700 1 -- 172.21.15.189:0/636056931 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84800a6f0 con 0x7fe85806e470 2024-04-06T12:12:29.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.403+0000 7f0229f11700 1 -- start start 2024-04-06T12:12:29.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7f0229f11700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02240b8a40 0x7f02240b8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7f0229f11700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f022406b7d0 0x7f022406bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7f0229f11700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02240ce7b0 con 0x7f02240b24c0 2024-04-06T12:12:29.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7f0229f11700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f022406c180 con 0x7f022406b7d0 2024-04-06T12:12:29.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7f0229f11700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02240cea90 con 0x7f02240b8a40 2024-04-06T12:12:29.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.404+0000 7f26b96a4700 1 -- start start 2024-04-06T12:12:29.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7fc6b9058700 1 --2- 172.21.15.189:0/1123577174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6b4150690 0x7fc6b4140620 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac007040 tx=0x7fc6ac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7f26b96a4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b406e130 0x7f26b4153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7f26b96a4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26b40b2300 0x7f26b414ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7f26b96a4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26b4150690 0x7f26b414f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7f26b96a4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b4153a40 con 0x7f26b406e130 2024-04-06T12:12:29.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7f26b96a4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b4151ff0 con 0x7f26b40b2300 2024-04-06T12:12:29.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7f26b96a4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b41522d0 con 0x7f26b4150690 2024-04-06T12:12:29.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7f26b27fc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26b40b2300 0x7f26b414ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7f26b2ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b406e130 0x7f26b4153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7f26b27fc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26b40b2300 0x7f26b414ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:58776/0 (socket says 172.21.15.189:58776) 2024-04-06T12:12:29.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7f26b27fc700 1 -- 172.21.15.189:0/4112896556 learned_addr learned my addr 172.21.15.189:0/4112896556 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7fe855d9b700 1 --2- 172.21.15.189:0/636056931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8580cdc60 0x7fe8580c27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:29.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7f024f17b700 1 -- start start 2024-04-06T12:12:29.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7fe85659c700 1 --2- 172.21.15.189:0/636056931 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe85806e470 0x7fe858158cd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe84800ade0 tx=0x7fe848004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7fc6aa7fc700 1 -- 172.21.15.189:0/1123577174 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac004450 con 0x7fc6b4150690 2024-04-06T12:12:29.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7fc6aa7fc700 1 -- 172.21.15.189:0/1123577174 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ac00b340 con 0x7fc6b4150690 2024-04-06T12:12:29.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7fc6aa7fc700 1 -- 172.21.15.189:0/1123577174 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac0047b0 con 0x7fc6b4150690 2024-04-06T12:12:29.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.405+0000 7f26b27fc700 1 -- 172.21.15.189:0/4112896556 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26b4150690 msgr2=0x7f26b414f7c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-06T12:12:29.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.406+0000 7f26b27fc700 1 --2- 172.21.15.189:0/4112896556 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26b4150690 0x7f26b414f7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.406+0000 7f26b27fc700 1 -- 172.21.15.189:0/4112896556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b406e130 msgr2=0x7f26b4153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.406+0000 7f26b27fc700 1 --2- 172.21.15.189:0/4112896556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b406e130 0x7f26b4153500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.406+0000 7f26b27fc700 1 -- 172.21.15.189:0/4112896556 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a80094a0 con 0x7f26b40b2300 2024-04-06T12:12:29.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.406+0000 7f8a607c2700 1 Processor -- start 2024-04-06T12:12:29.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.406+0000 7f26b2ffd700 1 --2- 172.21.15.189:0/4112896556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b406e130 0x7f26b4153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:29.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.406+0000 7f25ceffd700 1 -- 172.21.15.189:0/942217188 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c0004230 con 0x7f25d006ae40 2024-04-06T12:12:29.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.406+0000 7f25ceffd700 1 -- 172.21.15.189:0/942217188 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c0004390 con 0x7f25d006ae40 2024-04-06T12:12:29.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.406+0000 7f25ceffd700 1 -- 172.21.15.189:0/942217188 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c00036a0 con 0x7f25d006ae40 2024-04-06T12:12:29.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.406+0000 7f024f17b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024815d7b0 0x7f02480b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.406+0000 7fc6bbabd700 1 -- 172.21.15.189:0/1123577174 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b41fa970 con 0x7fc6b4150690 2024-04-06T12:12:29.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.406+0000 7fc6bbabd700 1 -- 172.21.15.189:0/1123577174 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b41faf30 con 0x7fc6b4150690 2024-04-06T12:12:29.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.406+0000 7f024f17b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f024815e500 0x7f024806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.406+0000 7f024f17b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02480b3990 con 0x7f024806a730 2024-04-06T12:12:29.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.406+0000 7f024f17b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f024806b0f0 con 0x7f024815e500 2024-04-06T12:12:29.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.406+0000 7f024f17b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02480b3060 con 0x7f024815d7b0 2024-04-06T12:12:29.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.407+0000 7f024cf17700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024815d7b0 0x7f02480b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.407+0000 7f024cf17700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024815d7b0 0x7f02480b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59372/0 (socket says 172.21.15.189:59372) 2024-04-06T12:12:29.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.407+0000 7f024cf17700 1 -- 172.21.15.189:0/617665278 learned_addr learned my addr 172.21.15.189:0/617665278 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.407+0000 7f024d718700 1 --2- 172.21.15.189:0/617665278 >> v2:172.21.15.189:3300/0 conn(0x7f024815e500 0x7f024806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.407+0000 7fc6bbabd700 1 -- 172.21.15.189:0/1123577174 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc6b415aa20 con 0x7fc6b4150690 2024-04-06T12:12:29.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.406+0000 7f8a607c2700 1 -- start start 2024-04-06T12:12:29.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.407+0000 7f8a607c2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a58159b30 0x7f8a5806da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.407+0000 7f8a607c2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8a5806df60 0x7f8a5815b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.407+0000 7f8a607c2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5806e3b0 con 0x7f8a58150690 2024-04-06T12:12:29.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.407+0000 7f8a607c2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a58150a70 con 0x7f8a5806df60 2024-04-06T12:12:29.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.407+0000 7f8a607c2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a58159f90 con 0x7f8a58159b30 2024-04-06T12:12:29.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.407+0000 7f8a5e55e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8a58150690 0x7f8a580b2300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34238/0 (socket says 172.21.15.189:34238) 2024-04-06T12:12:29.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.407+0000 7f8a5e55e700 1 -- 172.21.15.189:0/2687140945 learned_addr learned my addr 172.21.15.189:0/2687140945 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:29.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.407+0000 7f8a5ed5f700 1 --2- 172.21.15.189:0/2687140945 >> v2:172.21.15.192:3300/0 conn(0x7f8a5806df60 0x7f8a5815b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.406+0000 7f26b27fc700 1 --2- 172.21.15.189:0/4112896556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26b40b2300 0x7f26b414ea10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f26a400c550 tx=0x7f26a400c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.406+0000 7f26a37fe700 1 -- 172.21.15.189:0/4112896556 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a40052d0 con 0x7f26b40b2300 2024-04-06T12:12:29.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.407+0000 7f26b96a4700 1 -- 172.21.15.189:0/4112896556 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b41525b0 con 0x7f26b40b2300 2024-04-06T12:12:29.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.407+0000 7f26b96a4700 1 -- 172.21.15.189:0/4112896556 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b4152b70 con 0x7f26b40b2300 2024-04-06T12:12:29.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.407+0000 7f26a37fe700 1 -- 172.21.15.189:0/4112896556 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a4005430 con 0x7f26b40b2300 2024-04-06T12:12:29.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.407+0000 7f26a37fe700 1 -- 172.21.15.189:0/4112896556 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a40038c0 con 0x7f26b40b2300 2024-04-06T12:12:29.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.407+0000 7f25d664d700 1 -- 172.21.15.189:0/942217188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25d006ae40 msgr2=0x7f25d015ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.407+0000 7f25d664d700 1 --2- 172.21.15.189:0/942217188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25d006ae40 0x7f25d015ba40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f25c0009840 tx=0x7f25c0009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:29.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.408+0000 7f024d718700 1 -- 172.21.15.189:0/617665278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024815d7b0 msgr2=0x7f02480b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:29.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.408+0000 7f024d718700 1 --2- 172.21.15.189:0/617665278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024815d7b0 0x7f02480b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.408+0000 7f024d718700 1 -- 172.21.15.189:0/617665278 >> v1:172.21.15.192:6789/0 conn(0x7f024806a730 legacy=0x7f024815ddf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-06T12:12:29.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.408+0000 7f024d718700 1 -- 172.21.15.189:0/617665278 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02480b3340 con 0x7f024815e500 2024-04-06T12:12:29.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.408+0000 7f8a5ed5f700 1 -- 172.21.15.189:0/2687140945 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a58159b30 msgr2=0x7f8a5806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:29.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.408+0000 7f8a5ed5f700 1 --2- 172.21.15.189:0/2687140945 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a58159b30 0x7f8a5806da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.408+0000 7f8a5ed5f700 1 -- 172.21.15.189:0/2687140945 >> v1:172.21.15.189:6789/0 conn(0x7f8a58150690 legacy=0x7f8a580b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.408+0000 7f8a5ed5f700 1 -- 172.21.15.189:0/2687140945 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5815a180 con 0x7f8a5806df60 2024-04-06T12:12:29.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.408+0000 7f024d718700 1 --2- 172.21.15.189:0/617665278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024815e500 0x7f024806abb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0238009d90 tx=0x7f023800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d40e734eb14bd0b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.408+0000 7f02477fe700 1 -- 172.21.15.189:0/617665278 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023800e070 con 0x7f024815e500 2024-04-06T12:12:29.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.408+0000 7f02477fe700 1 -- 172.21.15.189:0/617665278 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023800bae0 con 0x7f024815e500 2024-04-06T12:12:29.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.408+0000 7f02477fe700 1 -- 172.21.15.189:0/617665278 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023800be70 con 0x7f024815e500 2024-04-06T12:12:29.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.409+0000 7f024f17b700 1 -- 172.21.15.189:0/617665278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024815e500 msgr2=0x7f024806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.409+0000 7f024f17b700 1 --2- 172.21.15.189:0/617665278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024815e500 0x7f024806abb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0238009d90 tx=0x7f023800b330 comp rx=0 tx=0).stop 2024-04-06T12:12:29.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.409+0000 7f8a5ed5f700 1 --2- 172.21.15.189:0/2687140945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a5806df60 0x7f8a5815b220 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8a540091b0 tx=0x7f8a540094c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a6101e36402a508 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.409+0000 7f8a5d55c700 1 -- 172.21.15.189:0/2687140945 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5400c900 con 0x7f8a5806df60 2024-04-06T12:12:29.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.409+0000 7f8a5d55c700 1 -- 172.21.15.189:0/2687140945 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a5400ca60 con 0x7f8a5806df60 2024-04-06T12:12:29.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.409+0000 7f8a5d55c700 1 -- 172.21.15.189:0/2687140945 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5400cdf0 con 0x7f8a5806df60 2024-04-06T12:12:29.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.409+0000 7f25d664d700 1 -- 172.21.15.189:0/942217188 shutdown_connections 2024-04-06T12:12:29.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.409+0000 7f25d664d700 1 --2- 172.21.15.189:0/942217188 >> v2:172.21.15.192:3300/0 conn(0x7f25d015fcd0 0x7f25d015bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.409+0000 7f25d664d700 1 --2- 172.21.15.189:0/942217188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25d006ae40 0x7f25d015ba40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.409+0000 7f25d664d700 1 -- 172.21.15.189:0/942217188 >> 172.21.15.189:0/942217188 conn(0x7f25d00d00b0 msgr2=0x7f25d00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.409+0000 7f024f17b700 1 -- 172.21.15.189:0/617665278 shutdown_connections 2024-04-06T12:12:29.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.409+0000 7f024f17b700 1 --2- 172.21.15.189:0/617665278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024815e500 0x7f024806abb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.409+0000 7f024f17b700 1 --2- 172.21.15.189:0/617665278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024815d7b0 0x7f02480b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.409+0000 7f024f17b700 1 -- 172.21.15.189:0/617665278 >> 172.21.15.189:0/617665278 conn(0x7f02480d00b0 msgr2=0x7f02480d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7f024f17b700 1 -- 172.21.15.189:0/617665278 shutdown_connections 2024-04-06T12:12:29.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7f8a607c2700 1 -- 172.21.15.189:0/2687140945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a5806df60 msgr2=0x7f8a5815b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7f8a607c2700 1 --2- 172.21.15.189:0/2687140945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a5806df60 0x7f8a5815b220 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8a540091b0 tx=0x7f8a540094c0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7f0222ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02240b8a40 0x7f02240b8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7f0223fff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f022406b7d0 0x7f022406bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7f25d664d700 1 -- 172.21.15.189:0/942217188 shutdown_connections 2024-04-06T12:12:29.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7f02237fe700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f02240b24c0 0x7f02240b6220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:52462/0 (socket says 172.21.15.189:52462) 2024-04-06T12:12:29.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7f02237fe700 1 -- 172.21.15.189:0/159433323 learned_addr learned my addr 172.21.15.189:0/159433323 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:29.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7fcbbc2e6700 1 Processor -- start 2024-04-06T12:12:29.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7f02227fc700 1 -- 172.21.15.189:0/159433323 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889550747 0 0) 0x7f02240ce7b0 con 0x7f02240b24c0 2024-04-06T12:12:29.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7f02227fc700 1 -- 172.21.15.189:0/159433323 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f020c0035b0 con 0x7f02240b24c0 2024-04-06T12:12:29.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7f26a37fe700 1 -- 172.21.15.189:0/4112896556 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f26a4003a20 con 0x7f26b40b2300 2024-04-06T12:12:29.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7f26a37fe700 1 --2- 172.21.15.189:0/4112896556 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f269806ac90 0x7f269806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7f26a37fe700 1 -- 172.21.15.189:0/4112896556 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f26a4088ae0 con 0x7f26b40b2300 2024-04-06T12:12:29.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.409+0000 7fc6aa7fc700 1 -- 172.21.15.189:0/1123577174 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc6ac020410 con 0x7fc6b4150690 2024-04-06T12:12:29.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7fc6aa7fc700 1 --2- 172.21.15.189:0/1123577174 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc69806ad60 0x7fc69806d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7fc6aa7fc700 1 -- 172.21.15.189:0/1123577174 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc6ac08b360 con 0x7fc6b4150690 2024-04-06T12:12:29.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7fc6aa7fc700 1 --2- 172.21.15.189:0/1123577174 >> [v2:172.21.15.189:6816/47416,v1:172.21.15.189:6817/47416] conn(0x7fc698074320 0x7fc698076760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7fc6aa7fc700 1 -- 172.21.15.189:0/1123577174 --> [v2:172.21.15.189:6816/47416,v1:172.21.15.189:6817/47416] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc698076e30 con 0x7fc698074320 2024-04-06T12:12:29.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.411+0000 7fc6b9859700 1 --2- 172.21.15.189:0/1123577174 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc69806ad60 0x7fc69806d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.411+0000 7fc6ba05a700 1 --2- 172.21.15.189:0/1123577174 >> [v2:172.21.15.189:6816/47416,v1:172.21.15.189:6817/47416] conn(0x7fc698074320 0x7fc698076760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.411+0000 7fc6b9859700 1 --2- 172.21.15.189:0/1123577174 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc69806ad60 0x7fc69806d220 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4004da0 tx=0x7fc6a4004b60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.411+0000 7fc6ba05a700 1 --2- 172.21.15.189:0/1123577174 >> [v2:172.21.15.189:6816/47416,v1:172.21.15.189:6817/47416] conn(0x7fc698074320 0x7fc698076760 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7fc6aa7fc700 1 -- 172.21.15.189:0/1123577174 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6ac0563e0 con 0x7fc6b4150690 2024-04-06T12:12:29.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.411+0000 7f0222ffd700 1 -- 172.21.15.189:0/159433323 >> v2:172.21.15.192:3300/0 conn(0x7f022406b7d0 msgr2=0x7f022406bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:29.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.411+0000 7f0222ffd700 1 --2- 172.21.15.189:0/159433323 >> v2:172.21.15.192:3300/0 conn(0x7f022406b7d0 0x7f022406bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.411+0000 7f0222ffd700 1 -- 172.21.15.189:0/159433323 >> v1:172.21.15.192:6789/0 conn(0x7f02240b24c0 legacy=0x7f02240b6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.411+0000 7f0222ffd700 1 -- 172.21.15.189:0/159433323 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02240b8240 con 0x7f02240b8a40 2024-04-06T12:12:29.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7f0222ffd700 1 --2- 172.21.15.189:0/159433323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02240b8a40 0x7f02240b8ef0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0218009840 tx=0x7f0218009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ac057da08904ce2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7f024f17b700 1 -- 172.21.15.189:0/617665278 wait complete. 2024-04-06T12:12:29.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.411+0000 7f024f17b700 1 Processor -- start 2024-04-06T12:12:29.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7f8a607c2700 1 -- 172.21.15.189:0/2687140945 shutdown_connections 2024-04-06T12:12:29.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7f8a607c2700 1 --2- 172.21.15.189:0/2687140945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a5806df60 0x7f8a5815b220 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7f8a607c2700 1 --2- 172.21.15.189:0/2687140945 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a58159b30 0x7f8a5806da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7f8a607c2700 1 -- 172.21.15.189:0/2687140945 >> 172.21.15.189:0/2687140945 conn(0x7f8a580cefa0 msgr2=0x7f8a580cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.410+0000 7fcbbc2e6700 1 -- start start 2024-04-06T12:12:29.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.411+0000 7fcbbc2e6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbb406a730 0x7fcbb40ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.411+0000 7fcbbc2e6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcbb40b3ad0 0x7fcbb4072050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.411+0000 7fcbbc2e6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb4071bc0 con 0x7fcbb40cf2b0 2024-04-06T12:12:29.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.411+0000 7fcbbc2e6700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbb40cf820 con 0x7fcbb40b3ad0 2024-04-06T12:12:29.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.411+0000 7fcbbc2e6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb40cfb00 con 0x7fcbb406a730 2024-04-06T12:12:29.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7fcbba082700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbb406a730 0x7fcbb40ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7fcbba082700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbb406a730 0x7fcbb40ccb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59402/0 (socket says 172.21.15.189:59402) 2024-04-06T12:12:29.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7fcbba082700 1 -- 172.21.15.189:0/3083263884 learned_addr learned my addr 172.21.15.189:0/3083263884 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7fcbba082700 1 -- 172.21.15.189:0/3083263884 >> v2:172.21.15.189:3300/0 conn(0x7fcbb40b3ad0 msgr2=0x7fcbb4072050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-06T12:12:29.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7fcbba082700 1 --2- 172.21.15.189:0/3083263884 >> v2:172.21.15.189:3300/0 conn(0x7fcbb40b3ad0 0x7fcbb4072050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7fcbba082700 1 -- 172.21.15.189:0/3083263884 >> v1:172.21.15.189:6789/0 conn(0x7fcbb40cf2b0 legacy=0x7fcbb406ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:29.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7fcbba082700 1 -- 172.21.15.189:0/3083263884 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb4071850 con 0x7fcbb406a730 2024-04-06T12:12:29.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.411+0000 7f25d664d700 1 -- 172.21.15.189:0/942217188 wait complete. 2024-04-06T12:12:29.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7f25d664d700 1 Processor -- start 2024-04-06T12:12:29.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.411+0000 7f26b2ffd700 1 --2- 172.21.15.189:0/4112896556 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f269806ac90 0x7f269806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.411+0000 7f26b2ffd700 1 --2- 172.21.15.189:0/4112896556 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f269806ac90 0x7f269806d150 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f26ac003b30 tx=0x7f26ac0091b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7f8a607c2700 1 -- 172.21.15.189:0/2687140945 shutdown_connections 2024-04-06T12:12:29.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7fcbba082700 1 --2- 172.21.15.189:0/3083263884 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbb406a730 0x7fcbb40ccb30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcbb00098f0 tx=0x7fcbb0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=692aefc75f1e1a4a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7f26b96a4700 1 --2- 172.21.15.189:0/4112896556 >> [v2:172.21.15.189:6808/47415,v1:172.21.15.189:6809/47415] conn(0x7f26900015e0 0x7f2690003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.413+0000 7f26b96a4700 1 -- 172.21.15.189:0/4112896556 --> [v2:172.21.15.189:6808/47415,v1:172.21.15.189:6809/47415] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2690006bd0 con 0x7f26900015e0 2024-04-06T12:12:29.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7fe846ffd700 1 -- 172.21.15.189:0/636056931 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84800a570 con 0x7fe85806e470 2024-04-06T12:12:29.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7fe846ffd700 1 -- 172.21.15.189:0/636056931 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84800f840 con 0x7fe85806e470 2024-04-06T12:12:29.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7fe846ffd700 1 -- 172.21.15.189:0/636056931 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84800fbf0 con 0x7fe85806e470 2024-04-06T12:12:29.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7fe85c81f700 1 -- 172.21.15.189:0/636056931 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe85815bf50 con 0x7fe85806e470 2024-04-06T12:12:29.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7fe85c81f700 1 -- 172.21.15.189:0/636056931 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe858159210 con 0x7fe85806e470 2024-04-06T12:12:29.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.413+0000 7f26b37fe700 1 --2- 172.21.15.189:0/4112896556 >> [v2:172.21.15.189:6808/47415,v1:172.21.15.189:6809/47415] conn(0x7f26900015e0 0x7f2690003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.413+0000 7fe85c81f700 1 -- 172.21.15.189:0/636056931 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe85806b450 con 0x7fe85806e470 2024-04-06T12:12:29.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.413+0000 7f02227fc700 1 -- 172.21.15.189:0/159433323 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218011830 con 0x7f02240b8a40 2024-04-06T12:12:29.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.413+0000 7f02227fc700 1 -- 172.21.15.189:0/159433323 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0218011990 con 0x7f02240b8a40 2024-04-06T12:12:29.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.413+0000 7f02227fc700 1 -- 172.21.15.189:0/159433323 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218011d20 con 0x7f02240b8a40 2024-04-06T12:12:29.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.414+0000 7fc6aa7fc700 1 -- 172.21.15.189:0/1123577174 <== osd.2 v2:172.21.15.189:6816/47416 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fc698076e30 con 0x7fc698074320 2024-04-06T12:12:29.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.414+0000 7f26b37fe700 1 --2- 172.21.15.189:0/4112896556 >> [v2:172.21.15.189:6808/47415,v1:172.21.15.189:6809/47415] conn(0x7f26900015e0 0x7f2690003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.413+0000 7f25d664d700 1 -- start start 2024-04-06T12:12:29.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.414+0000 7f25d664d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25d006ae40 0x7f25d013dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.414+0000 7f25d664d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25d0144f40 0x7f25d02038d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.414+0000 7f25d664d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25d015fcd0 0x7f25d0202690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.414+0000 7f25d664d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d013e3d0 con 0x7f25d006ae40 2024-04-06T12:12:29.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.414+0000 7f25d664d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d0202bd0 con 0x7f25d0144f40 2024-04-06T12:12:29.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.414+0000 7f25d664d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d01ff820 con 0x7f25d015fcd0 2024-04-06T12:12:29.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7f8a607c2700 1 -- 172.21.15.189:0/2687140945 wait complete. 2024-04-06T12:12:29.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.414+0000 7f0229f11700 1 -- 172.21.15.189:0/159433323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02240b8a40 msgr2=0x7f02240b8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.414+0000 7f0229f11700 1 --2- 172.21.15.189:0/159433323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02240b8a40 0x7f02240b8ef0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0218009840 tx=0x7f0218009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:29.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.414+0000 7f8a607c2700 1 Processor -- start 2024-04-06T12:12:29.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.411+0000 7f024f17b700 1 -- start start 2024-04-06T12:12:29.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7f024f17b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024815d7b0 0x7f024814fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7f024f17b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024815e500 0x7f02480ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7f024f17b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f024815c3e0 0x7f024815fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7f024f17b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0248150230 con 0x7f024815d7b0 2024-04-06T12:12:29.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7f024f17b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024815b620 con 0x7f024815c3e0 2024-04-06T12:12:29.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.412+0000 7f024f17b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f024815b900 con 0x7f024815e500 2024-04-06T12:12:29.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.413+0000 7f0247fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024815e500 0x7f02480ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.413+0000 7f0247fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024815e500 0x7f02480ceb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59416/0 (socket says 172.21.15.189:59416) 2024-04-06T12:12:29.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.413+0000 7f0247fff700 1 -- 172.21.15.189:0/2271597329 learned_addr learned my addr 172.21.15.189:0/2271597329 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.413+0000 7f0247fff700 1 -- 172.21.15.189:0/2271597329 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f024815c3e0 msgr2=0x7f024815fd10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:12:29.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.413+0000 7f0247fff700 1 --2- 172.21.15.189:0/2271597329 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f024815c3e0 0x7f024815fd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.413+0000 7f0247fff700 1 -- 172.21.15.189:0/2271597329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024815d7b0 msgr2=0x7f024814fcf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:12:29.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.413+0000 7f0247fff700 1 --2- 172.21.15.189:0/2271597329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024815d7b0 0x7f024814fcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.413+0000 7f0247fff700 1 -- 172.21.15.189:0/2271597329 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0238009a70 con 0x7f024815e500 2024-04-06T12:12:29.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.413+0000 7f0247fff700 1 --2- 172.21.15.189:0/2271597329 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024815e500 0x7f02480ceb10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f024000b4a0 tx=0x7f024000b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.413+0000 7f02457fa700 1 -- 172.21.15.189:0/2271597329 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0240011830 con 0x7f024815e500 2024-04-06T12:12:29.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.413+0000 7f02457fa700 1 -- 172.21.15.189:0/2271597329 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02400119b0 con 0x7f024815e500 2024-04-06T12:12:29.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.413+0000 7f02457fa700 1 -- 172.21.15.189:0/2271597329 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0240011d60 con 0x7f024815e500 2024-04-06T12:12:29.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.414+0000 7f024f17b700 1 -- 172.21.15.189:0/2271597329 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f024815bbe0 con 0x7f024815e500 2024-04-06T12:12:29.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.414+0000 7f024f17b700 1 -- 172.21.15.189:0/2271597329 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02480cf460 con 0x7f024815e500 2024-04-06T12:12:29.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.415+0000 7f0229f11700 1 -- 172.21.15.189:0/159433323 shutdown_connections 2024-04-06T12:12:29.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.415+0000 7f0229f11700 1 --2- 172.21.15.189:0/159433323 >> v2:172.21.15.192:3300/0 conn(0x7f022406b7d0 0x7f022406bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.415+0000 7f0229f11700 1 --2- 172.21.15.189:0/159433323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02240b8a40 0x7f02240b8ef0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.415+0000 7f0229f11700 1 -- 172.21.15.189:0/159433323 >> 172.21.15.189:0/159433323 conn(0x7f02240d33d0 msgr2=0x7f02240d37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.415+0000 7f0229f11700 1 -- 172.21.15.189:0/159433323 shutdown_connections 2024-04-06T12:12:29.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.415+0000 7f25cf7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25d0144f40 0x7f25d02038d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.415+0000 7f25cf7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25d0144f40 0x7f25d02038d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:58796/0 (socket says 172.21.15.189:58796) 2024-04-06T12:12:29.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.415+0000 7f25cf7fe700 1 -- 172.21.15.189:0/1395053118 learned_addr learned my addr 172.21.15.189:0/1395053118 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.415+0000 7f26a37fe700 1 -- 172.21.15.189:0/4112896556 <== osd.0 v2:172.21.15.189:6808/47415 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2690006bd0 con 0x7f26900015e0 2024-04-06T12:12:29.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.415+0000 7f25cf7fe700 1 -- 172.21.15.189:0/1395053118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25d015fcd0 msgr2=0x7f25d0202690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-06T12:12:29.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.415+0000 7f25cf7fe700 1 --2- 172.21.15.189:0/1395053118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25d015fcd0 0x7f25d0202690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.415+0000 7f25cf7fe700 1 -- 172.21.15.189:0/1395053118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25d006ae40 msgr2=0x7f25d013dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7f25cf7fe700 1 --2- 172.21.15.189:0/1395053118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25d006ae40 0x7f25d013dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7f25cf7fe700 1 -- 172.21.15.189:0/1395053118 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c00094a0 con 0x7f25d0144f40 2024-04-06T12:12:29.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.414+0000 7f8a607c2700 1 -- start start 2024-04-06T12:12:29.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7f8a607c2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a5806df60 0x7f8a580ce400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7f8a607c2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a58150690 0x7f8a581646c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7f8a607c2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a58159b30 0x7f8a5814f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7f8a607c2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a580ce940 con 0x7f8a58150690 2024-04-06T12:12:29.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7f8a607c2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a58164c00 con 0x7f8a58159b30 2024-04-06T12:12:29.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7f8a607c2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5814fd30 con 0x7f8a5806df60 2024-04-06T12:12:29.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7f25cf7fe700 1 --2- 172.21.15.189:0/1395053118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25d0144f40 0x7f25d02038d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f25c0009b90 tx=0x7f25c0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7fcbb9080700 1 -- 172.21.15.189:0/3083263884 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0004230 con 0x7fcbb406a730 2024-04-06T12:12:29.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7fcbb9080700 1 -- 172.21.15.189:0/3083263884 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb0004390 con 0x7fcbb406a730 2024-04-06T12:12:29.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7fcbb9080700 1 -- 172.21.15.189:0/3083263884 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb00036a0 con 0x7fcbb406a730 2024-04-06T12:12:29.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.415+0000 7f0229f11700 1 -- 172.21.15.189:0/159433323 wait complete. 2024-04-06T12:12:29.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7f0229f11700 1 Processor -- start 2024-04-06T12:12:29.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.415+0000 7f024f17b700 1 -- 172.21.15.189:0/2271597329 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f024815b790 con 0x7f024815e500 2024-04-06T12:12:29.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.415+0000 7fe846ffd700 1 -- 172.21.15.189:0/636056931 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe84800fd50 con 0x7fe85806e470 2024-04-06T12:12:29.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7fe846ffd700 1 --2- 172.21.15.189:0/636056931 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe83806ace0 0x7fe83806d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7fe846ffd700 1 -- 172.21.15.189:0/636056931 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe848013070 con 0x7fe85806e470 2024-04-06T12:12:29.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7fe855d9b700 1 --2- 172.21.15.189:0/636056931 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe83806ace0 0x7fe83806d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7fe846ffd700 1 --2- 172.21.15.189:0/636056931 >> [v2:172.21.15.189:6800/47414,v1:172.21.15.189:6801/47414] conn(0x7fe8380742a0 0x7fe8380766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7fe846ffd700 1 -- 172.21.15.189:0/636056931 --> [v2:172.21.15.189:6800/47414,v1:172.21.15.189:6801/47414] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe838076db0 con 0x7fe8380742a0 2024-04-06T12:12:29.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7f25ccff9700 1 -- 172.21.15.189:0/1395053118 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c0011630 con 0x7f25d0144f40 2024-04-06T12:12:29.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.417+0000 7f25ccff9700 1 -- 172.21.15.189:0/1395053118 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c0011790 con 0x7f25d0144f40 2024-04-06T12:12:29.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.417+0000 7f25ccff9700 1 -- 172.21.15.189:0/1395053118 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c0011b40 con 0x7f25d0144f40 2024-04-06T12:12:29.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.417+0000 7fe856d9d700 1 --2- 172.21.15.189:0/636056931 >> [v2:172.21.15.189:6800/47414,v1:172.21.15.189:6801/47414] conn(0x7fe8380742a0 0x7fe8380766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7f8a5e55e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a5806df60 0x7f8a580ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7f8a5dd5d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a58150690 0x7f8a581646c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7f8a5dd5d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a58150690 0x7f8a581646c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38068/0 (socket says 172.21.15.189:38068) 2024-04-06T12:12:29.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7f8a5dd5d700 1 -- 172.21.15.189:0/945979242 learned_addr learned my addr 172.21.15.189:0/945979242 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7f8a5ed5f700 1 --2- 172.21.15.189:0/945979242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a58159b30 0x7f8a5814f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.417+0000 7f8a5ed5f700 1 -- 172.21.15.189:0/945979242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a5806df60 msgr2=0x7f8a580ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.417+0000 7f8a5ed5f700 1 --2- 172.21.15.189:0/945979242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a5806df60 0x7f8a580ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.417+0000 7f8a5ed5f700 1 -- 172.21.15.189:0/945979242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a58150690 msgr2=0x7f8a581646c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.417+0000 7f8a5ed5f700 1 --2- 172.21.15.189:0/945979242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a58150690 0x7f8a581646c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.417+0000 7f8a5ed5f700 1 -- 172.21.15.189:0/945979242 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a54009d30 con 0x7f8a58159b30 2024-04-06T12:12:29.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.417+0000 7fe856d9d700 1 --2- 172.21.15.189:0/636056931 >> [v2:172.21.15.189:6800/47414,v1:172.21.15.189:6801/47414] conn(0x7fe8380742a0 0x7fe8380766e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.417+0000 7f8a5ed5f700 1 --2- 172.21.15.189:0/945979242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a58159b30 0x7f8a5814f7f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8a54017000 tx=0x7f8a54004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.417+0000 7fe855d9b700 1 --2- 172.21.15.189:0/636056931 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe83806ace0 0x7fe83806d1a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe84c009820 tx=0x7fe84c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.417+0000 7fe846ffd700 1 -- 172.21.15.189:0/636056931 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe848055f50 con 0x7fe85806e470 2024-04-06T12:12:29.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.417+0000 7f8a4effd700 1 -- 172.21.15.189:0/945979242 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a540045f0 con 0x7f8a58159b30 2024-04-06T12:12:29.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.418+0000 7f8a4effd700 1 -- 172.21.15.189:0/945979242 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a54003ab0 con 0x7f8a58159b30 2024-04-06T12:12:29.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.418+0000 7f8a607c2700 1 -- 172.21.15.189:0/945979242 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a580cd360 con 0x7f8a58159b30 2024-04-06T12:12:29.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.418+0000 7f25d664d700 1 -- 172.21.15.189:0/1395053118 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d01ffaa0 con 0x7f25d0144f40 2024-04-06T12:12:29.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.417+0000 7f0229f11700 1 -- start start 2024-04-06T12:12:29.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.418+0000 7f25d664d700 1 -- 172.21.15.189:0/1395053118 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d0200060 con 0x7f25d0144f40 2024-04-06T12:12:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.416+0000 7f02457fa700 1 -- 172.21.15.189:0/2271597329 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f024001a400 con 0x7f024815e500 2024-04-06T12:12:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.417+0000 7f02457fa700 1 --2- 172.21.15.189:0/2271597329 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f022808cd70 0x7f022808f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.417+0000 7f02457fa700 1 -- 172.21.15.189:0/2271597329 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0240089d90 con 0x7f024815e500 2024-04-06T12:12:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.417+0000 7f02457fa700 1 --2- 172.21.15.189:0/2271597329 >> [v2:172.21.15.192:6816/47716,v1:172.21.15.192:6817/47716] conn(0x7f02280963b0 0x7f02280987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.417+0000 7f02457fa700 1 -- 172.21.15.189:0/2271597329 --> [v2:172.21.15.192:6816/47716,v1:172.21.15.192:6817/47716] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0228098ec0 con 0x7f02280963b0 2024-04-06T12:12:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.417+0000 7f02457fa700 1 -- 172.21.15.189:0/2271597329 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f024008a140 con 0x7f024815e500 2024-04-06T12:12:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.418+0000 7f0229f11700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f022406b7d0 0x7f02240cac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.418+0000 7f0229f11700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02240b24c0 0x7f02241613e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.418+0000 7f0229f11700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02240b8a40 0x7f02240ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.418+0000 7f0229f11700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02240cc010 con 0x7f02240b24c0 2024-04-06T12:12:29.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.418+0000 7f0229f11700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0224161920 con 0x7f022406b7d0 2024-04-06T12:12:29.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.418+0000 7f0229f11700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0224152530 con 0x7f02240b8a40 2024-04-06T12:12:29.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.418+0000 7f8a607c2700 1 -- 172.21.15.189:0/945979242 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a580cd920 con 0x7f8a58159b30 2024-04-06T12:12:29.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.418+0000 7f8a4effd700 1 -- 172.21.15.189:0/945979242 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a54003e60 con 0x7f8a58159b30 2024-04-06T12:12:29.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.418+0000 7f024d718700 1 --2- 172.21.15.189:0/2271597329 >> [v2:172.21.15.192:6816/47716,v1:172.21.15.192:6817/47716] conn(0x7f02280963b0 0x7f02280987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.418+0000 7f024cf17700 1 --2- 172.21.15.189:0/2271597329 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f022808cd70 0x7f022808f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.418+0000 7f0222ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02240b24c0 0x7f02241613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.418+0000 7f0223fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02240b8a40 0x7f02240ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.418+0000 7f0222ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02240b24c0 0x7f02241613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38082/0 (socket says 172.21.15.189:38082) 2024-04-06T12:12:29.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.418+0000 7f0222ffd700 1 -- 172.21.15.189:0/3947046127 learned_addr learned my addr 172.21.15.189:0/3947046127 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.419+0000 7f0223fff700 1 -- 172.21.15.189:0/3947046127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f022406b7d0 msgr2=0x7f02240cac80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:12:29.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.419+0000 7f0223fff700 1 --2- 172.21.15.189:0/3947046127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f022406b7d0 0x7f02240cac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.419+0000 7f0223fff700 1 -- 172.21.15.189:0/3947046127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02240b24c0 msgr2=0x7f02241613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.419+0000 7f0223fff700 1 --2- 172.21.15.189:0/3947046127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02240b24c0 0x7f02241613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.419+0000 7f0223fff700 1 -- 172.21.15.189:0/3947046127 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02180094a0 con 0x7f02240b8a40 2024-04-06T12:12:29.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.419+0000 7f0223fff700 1 --2- 172.21.15.189:0/3947046127 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02240b8a40 0x7f02240ca4a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f021c007d90 tx=0x7f021c00e3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.418+0000 7f25d664d700 1 -- 172.21.15.189:0/1395053118 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f25d015f7b0 con 0x7f25d0144f40 2024-04-06T12:12:29.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.419+0000 7f024cf17700 1 --2- 172.21.15.189:0/2271597329 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f022808cd70 0x7f022808f230 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0230009a60 tx=0x7f02300092c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.420+0000 7f25ccff9700 1 -- 172.21.15.189:0/1395053118 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f25c0022070 con 0x7f25d0144f40 2024-04-06T12:12:29.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.421+0000 7f024d718700 1 --2- 172.21.15.189:0/2271597329 >> [v2:172.21.15.192:6816/47716,v1:172.21.15.192:6817/47716] conn(0x7f02280963b0 0x7f02280987f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.421+0000 7f25ccff9700 1 --2- 172.21.15.189:0/1395053118 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f25bc06ace0 0x7f25bc06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.421+0000 7f25ccff9700 1 -- 172.21.15.189:0/1395053118 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f25c008b060 con 0x7f25d0144f40 2024-04-06T12:12:29.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.421+0000 7f25cffff700 1 --2- 172.21.15.189:0/1395053118 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f25bc06ace0 0x7f25bc06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.421+0000 7f25ccff9700 1 --2- 172.21.15.189:0/1395053118 >> [v2:172.21.15.192:6800/47714,v1:172.21.15.192:6801/47714] conn(0x7f25bc074320 0x7f25bc076760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.421+0000 7f25ccff9700 1 -- 172.21.15.189:0/1395053118 --> [v2:172.21.15.192:6800/47714,v1:172.21.15.192:6801/47714] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f25bc076e30 con 0x7f25bc074320 2024-04-06T12:12:29.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.421+0000 7f25ccff9700 1 -- 172.21.15.189:0/1395053118 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f25c008d770 con 0x7f25d0144f40 2024-04-06T12:12:29.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.421+0000 7f25d4bea700 1 --2- 172.21.15.189:0/1395053118 >> [v2:172.21.15.192:6800/47714,v1:172.21.15.192:6801/47714] conn(0x7f25bc074320 0x7f25bc076760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.421+0000 7f25cffff700 1 --2- 172.21.15.189:0/1395053118 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f25bc06ace0 0x7f25bc06d1a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f25c4000f60 tx=0x7f25c4007a70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.422+0000 7f02457fa700 1 -- 172.21.15.189:0/2271597329 <== osd.4 v2:172.21.15.192:6816/47716 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f0228098ec0 con 0x7f02280963b0 2024-04-06T12:12:29.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.422+0000 7f25d4bea700 1 --2- 172.21.15.189:0/1395053118 >> [v2:172.21.15.192:6800/47714,v1:172.21.15.192:6801/47714] conn(0x7f25bc074320 0x7f25bc076760 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.419+0000 7fcbbc2e6700 1 -- 172.21.15.189:0/3083263884 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbb406a730 msgr2=0x7fcbb40ccb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.419+0000 7fcbbc2e6700 1 --2- 172.21.15.189:0/3083263884 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbb406a730 0x7fcbb40ccb30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcbb00098f0 tx=0x7fcbb0009c00 comp rx=0 tx=0).stop 2024-04-06T12:12:29.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.421+0000 7fcbbc2e6700 1 -- 172.21.15.189:0/3083263884 shutdown_connections 2024-04-06T12:12:29.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.421+0000 7fcbbc2e6700 1 --2- 172.21.15.189:0/3083263884 >> v2:172.21.15.189:3300/0 conn(0x7fcbb40b3ad0 0x7fcbb4072050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.421+0000 7fcbbc2e6700 1 --2- 172.21.15.189:0/3083263884 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbb406a730 0x7fcbb40ccb30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.421+0000 7fcbbc2e6700 1 -- 172.21.15.189:0/3083263884 >> 172.21.15.189:0/3083263884 conn(0x7fcbb40d00b0 msgr2=0x7fcbb40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.423+0000 7fe846ffd700 1 -- 172.21.15.189:0/636056931 <== osd.1 v2:172.21.15.189:6800/47414 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe838076db0 con 0x7fe8380742a0 2024-04-06T12:12:29.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.423+0000 7f8a607c2700 1 -- 172.21.15.189:0/945979242 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8a40000fc0 con 0x7f8a58159b30 2024-04-06T12:12:29.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.424+0000 7fcbbc2e6700 1 -- 172.21.15.189:0/3083263884 shutdown_connections 2024-04-06T12:12:29.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.424+0000 7f020bfff700 1 -- 172.21.15.189:0/3947046127 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021c018070 con 0x7f02240b8a40 2024-04-06T12:12:29.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.424+0000 7f020bfff700 1 -- 172.21.15.189:0/3947046127 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f021c00eb80 con 0x7f02240b8a40 2024-04-06T12:12:29.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.424+0000 7f0229f11700 1 -- 172.21.15.189:0/3947046127 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02241530b0 con 0x7f02240b8a40 2024-04-06T12:12:29.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.424+0000 7f0229f11700 1 -- 172.21.15.189:0/3947046127 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0224152780 con 0x7f02240b8a40 2024-04-06T12:12:29.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.425+0000 7f0229f11700 1 -- 172.21.15.189:0/3947046127 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f02240ce430 con 0x7f02240b8a40 2024-04-06T12:12:29.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.424+0000 7fcbbc2e6700 1 -- 172.21.15.189:0/3083263884 wait complete. 2024-04-06T12:12:29.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.426+0000 7fcbbc2e6700 1 Processor -- start 2024-04-06T12:12:29.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.426+0000 7f020bfff700 1 -- 172.21.15.189:0/3947046127 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021c003a20 con 0x7f02240b8a40 2024-04-06T12:12:29.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.426+0000 7f020bfff700 1 -- 172.21.15.189:0/3947046127 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f021c003b80 con 0x7f02240b8a40 2024-04-06T12:12:29.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.426+0000 7f020bfff700 1 --2- 172.21.15.189:0/3947046127 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f021006ade0 0x7f021006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.427+0000 7f020bfff700 1 -- 172.21.15.189:0/3947046127 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f021c089730 con 0x7f02240b8a40 2024-04-06T12:12:29.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.427+0000 7f020bfff700 1 --2- 172.21.15.189:0/3947046127 >> [v2:172.21.15.189:6824/47425,v1:172.21.15.189:6825/47425] conn(0x7f0210074420 0x7f0210076860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.427+0000 7f020bfff700 1 -- 172.21.15.189:0/3947046127 --> [v2:172.21.15.189:6824/47425,v1:172.21.15.189:6825/47425] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0210076f30 con 0x7f0210074420 2024-04-06T12:12:29.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.427+0000 7f020bfff700 1 -- 172.21.15.189:0/3947046127 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f021c08bc80 con 0x7f02240b8a40 2024-04-06T12:12:29.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.427+0000 7f02237fe700 1 --2- 172.21.15.189:0/3947046127 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f021006ade0 0x7f021006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.428+0000 7f02237fe700 1 --2- 172.21.15.189:0/3947046127 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f021006ade0 0x7f021006d2a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0214007fd0 tx=0x7f0214007980 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.429+0000 7f0222ffd700 1 --2- 172.21.15.189:0/3947046127 >> [v2:172.21.15.189:6824/47425,v1:172.21.15.189:6825/47425] conn(0x7f0210074420 0x7f0210076860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.429+0000 7f8a4effd700 1 -- 172.21.15.189:0/945979242 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8a54018400 con 0x7f8a58159b30 2024-04-06T12:12:29.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.429+0000 7f8a4effd700 1 --2- 172.21.15.189:0/945979242 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8a3c06ace0 0x7f8a3c06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.430+0000 7f8a4effd700 1 -- 172.21.15.189:0/945979242 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f8a5408afe0 con 0x7f8a58159b30 2024-04-06T12:12:29.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.430+0000 7f8a4effd700 1 --2- 172.21.15.189:0/945979242 >> [v2:172.21.15.192:6819/47717,v1:172.21.15.192:6821/47717] conn(0x7f8a3c074320 0x7f8a3c076760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.430+0000 7f8a4effd700 1 -- 172.21.15.189:0/945979242 --> [v2:172.21.15.192:6819/47717,v1:172.21.15.192:6821/47717] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a3c076e30 con 0x7f8a3c074320 2024-04-06T12:12:29.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.430+0000 7f8a4effd700 1 -- 172.21.15.189:0/945979242 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a5408b390 con 0x7f8a58159b30 2024-04-06T12:12:29.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.428+0000 7fcbbc2e6700 1 -- start start 2024-04-06T12:12:29.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.429+0000 7fcbbc2e6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb406a730 0x7fcbb415dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.429+0000 7fcbbc2e6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbb40b3ad0 0x7fcbb40adee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.429+0000 7fcbbc2e6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbb40cf2b0 0x7fcbb40ae420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.429+0000 7fcbbc2e6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb40aea30 con 0x7fcbb406a730 2024-04-06T12:12:29.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.429+0000 7fcbbc2e6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb40aed10 con 0x7fcbb40b3ad0 2024-04-06T12:12:29.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.429+0000 7fcbbc2e6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb40aeff0 con 0x7fcbb40cf2b0 2024-04-06T12:12:29.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.430+0000 7fcbba082700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb406a730 0x7fcbb415dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.430+0000 7fcbba082700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb406a730 0x7fcbb415dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38086/0 (socket says 172.21.15.189:38086) 2024-04-06T12:12:29.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.430+0000 7fcbba082700 1 -- 172.21.15.189:0/3650478294 learned_addr learned my addr 172.21.15.189:0/3650478294 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.430+0000 7f25ccff9700 1 -- 172.21.15.189:0/1395053118 <== osd.5 v2:172.21.15.192:6800/47714 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f25bc076e30 con 0x7f25bc074320 2024-04-06T12:12:29.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.430+0000 7f8a5dd5d700 1 --2- 172.21.15.189:0/945979242 >> [v2:172.21.15.192:6819/47717,v1:172.21.15.192:6821/47717] conn(0x7f8a3c074320 0x7f8a3c076760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.430+0000 7fcbba082700 1 -- 172.21.15.189:0/3650478294 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbb40cf2b0 msgr2=0x7fcbb40ae420 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-06T12:12:29.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.430+0000 7fcbba082700 1 --2- 172.21.15.189:0/3650478294 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbb40cf2b0 0x7fcbb40ae420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.430+0000 7fcbba082700 1 -- 172.21.15.189:0/3650478294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbb40b3ad0 msgr2=0x7fcbb40adee0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-06T12:12:29.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.430+0000 7fcbba082700 1 --2- 172.21.15.189:0/3650478294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbb40b3ad0 0x7fcbb40adee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.430+0000 7fcbba082700 1 -- 172.21.15.189:0/3650478294 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb0009550 con 0x7fcbb406a730 2024-04-06T12:12:29.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.431+0000 7f0222ffd700 1 --2- 172.21.15.189:0/3947046127 >> [v2:172.21.15.189:6824/47425,v1:172.21.15.189:6825/47425] conn(0x7f0210074420 0x7f0210076860 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.431+0000 7f8a5dd5d700 1 --2- 172.21.15.189:0/945979242 >> [v2:172.21.15.192:6819/47717,v1:172.21.15.192:6821/47717] conn(0x7f8a3c074320 0x7f8a3c076760 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.431+0000 7fcbba082700 1 --2- 172.21.15.189:0/3650478294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb406a730 0x7fcbb415dd90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0000c00 tx=0x7fcbb0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.431+0000 7fcbaaffd700 1 -- 172.21.15.189:0/3650478294 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb000b660 con 0x7fcbb406a730 2024-04-06T12:12:29.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.431+0000 7fcbaaffd700 1 -- 172.21.15.189:0/3650478294 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb000b7e0 con 0x7fcbb406a730 2024-04-06T12:12:29.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.431+0000 7fcbaaffd700 1 -- 172.21.15.189:0/3650478294 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb000bb90 con 0x7fcbb406a730 2024-04-06T12:12:29.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.432+0000 7fcbbc2e6700 1 -- 172.21.15.189:0/3650478294 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb40af2d0 con 0x7fcbb406a730 2024-04-06T12:12:29.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.432+0000 7fcbbc2e6700 1 -- 172.21.15.189:0/3650478294 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb415e5a0 con 0x7fcbb406a730 2024-04-06T12:12:29.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.432+0000 7f8a4effd700 1 -- 172.21.15.189:0/945979242 <== osd.6 v2:172.21.15.192:6819/47717 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f8a3c076e30 con 0x7f8a3c074320 2024-04-06T12:12:29.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.435+0000 7fcbbc2e6700 1 -- 172.21.15.189:0/3650478294 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcbb40b3f60 con 0x7fcbb406a730 2024-04-06T12:12:29.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.435+0000 7fcbaaffd700 1 -- 172.21.15.189:0/3650478294 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcbb0022070 con 0x7fcbb406a730 2024-04-06T12:12:29.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.435+0000 7f020bfff700 1 -- 172.21.15.189:0/3947046127 <== osd.3 v2:172.21.15.189:6824/47425 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f0210076f30 con 0x7f0210074420 2024-04-06T12:12:29.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.435+0000 7fcbaaffd700 1 --2- 172.21.15.189:0/3650478294 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcb9806ac90 0x7fcb9806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.436+0000 7fcbaaffd700 1 -- 172.21.15.189:0/3650478294 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fcbb008b310 con 0x7fcbb406a730 2024-04-06T12:12:29.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.436+0000 7fcbaaffd700 1 --2- 172.21.15.189:0/3650478294 >> [v2:172.21.15.192:6802/47715,v1:172.21.15.192:6804/47715] conn(0x7fcb98074250 0x7fcb98076690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.436+0000 7fcbaaffd700 1 -- 172.21.15.189:0/3650478294 --> [v2:172.21.15.192:6802/47715,v1:172.21.15.192:6804/47715] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcb98076d60 con 0x7fcb98074250 2024-04-06T12:12:29.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.436+0000 7fcbaaffd700 1 -- 172.21.15.189:0/3650478294 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbb008d8f0 con 0x7fcbb406a730 2024-04-06T12:12:29.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.436+0000 7fcbb9881700 1 --2- 172.21.15.189:0/3650478294 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcb9806ac90 0x7fcb9806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.437+0000 7fcbb9881700 1 --2- 172.21.15.189:0/3650478294 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcb9806ac90 0x7fcb9806d150 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcba4006fd0 tx=0x7fcba4006ba0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.437+0000 7f26b96a4700 1 -- 172.21.15.189:0/4112896556 --> [v2:172.21.15.189:6808/47415,v1:172.21.15.189:6809/47415] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2690005c50 con 0x7f26900015e0 2024-04-06T12:12:29.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.438+0000 7fcbba883700 1 --2- 172.21.15.189:0/3650478294 >> [v2:172.21.15.192:6802/47715,v1:172.21.15.192:6804/47715] conn(0x7fcb98074250 0x7fcb98076690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.438+0000 7f26a37fe700 1 -- 172.21.15.189:0/4112896556 <== osd.0 v2:172.21.15.189:6808/47415 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2690005c50 con 0x7f26900015e0 2024-04-06T12:12:29.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.438+0000 7f26b96a4700 1 -- 172.21.15.189:0/4112896556 >> [v2:172.21.15.189:6808/47415,v1:172.21.15.189:6809/47415] conn(0x7f26900015e0 msgr2=0x7f2690003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.438+0000 7f26b96a4700 1 --2- 172.21.15.189:0/4112896556 >> [v2:172.21.15.189:6808/47415,v1:172.21.15.189:6809/47415] conn(0x7f26900015e0 0x7f2690003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.439+0000 7f8a5e55e700 1 --2- 172.21.15.189:0/945979242 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8a3c06ace0 0x7f8a3c06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.442+0000 7f8a5e55e700 1 --2- 172.21.15.189:0/945979242 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8a3c06ace0 0x7f8a3c06d1a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8a50005c90 tx=0x7f8a50009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.444+0000 7f26b96a4700 1 -- 172.21.15.189:0/4112896556 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f269806ac90 msgr2=0x7f269806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.444+0000 7f26b96a4700 1 --2- 172.21.15.189:0/4112896556 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f269806ac90 0x7f269806d150 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f26ac003b30 tx=0x7f26ac0091b0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.444+0000 7f26b96a4700 1 -- 172.21.15.189:0/4112896556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26b40b2300 msgr2=0x7f26b414ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.444+0000 7f26b96a4700 1 --2- 172.21.15.189:0/4112896556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26b40b2300 0x7f26b414ea10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f26a400c550 tx=0x7f26a400c860 comp rx=0 tx=0).stop 2024-04-06T12:12:29.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.444+0000 7f26b37fe700 1 -- 172.21.15.189:0/4112896556 reap_dead start 2024-04-06T12:12:29.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.441+0000 7f024f17b700 1 -- 172.21.15.189:0/2271597329 --> [v2:172.21.15.192:6816/47716,v1:172.21.15.192:6817/47716] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f024815eb40 con 0x7f02280963b0 2024-04-06T12:12:29.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.446+0000 7f26b96a4700 1 -- 172.21.15.189:0/4112896556 shutdown_connections 2024-04-06T12:12:29.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.446+0000 7f26b96a4700 1 -- 172.21.15.189:0/4112896556 >> 172.21.15.189:0/4112896556 conn(0x7f26b40cefa0 msgr2=0x7f26b406da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.446+0000 7f26b96a4700 1 -- 172.21.15.189:0/4112896556 shutdown_connections 2024-04-06T12:12:29.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.446+0000 7f26b96a4700 1 -- 172.21.15.189:0/4112896556 wait complete. 2024-04-06T12:12:29.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.446+0000 7fcbba883700 1 --2- 172.21.15.189:0/3650478294 >> [v2:172.21.15.192:6802/47715,v1:172.21.15.192:6804/47715] conn(0x7fcb98074250 0x7fcb98076690 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.446+0000 7f02457fa700 1 -- 172.21.15.189:0/2271597329 <== osd.4 v2:172.21.15.192:6816/47716 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0238003b40 con 0x7f02280963b0 2024-04-06T12:12:29.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.447+0000 7f024f17b700 1 -- 172.21.15.189:0/2271597329 >> [v2:172.21.15.192:6816/47716,v1:172.21.15.192:6817/47716] conn(0x7f02280963b0 msgr2=0x7f02280987f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.447+0000 7f024f17b700 1 --2- 172.21.15.189:0/2271597329 >> [v2:172.21.15.192:6816/47716,v1:172.21.15.192:6817/47716] conn(0x7f02280963b0 0x7f02280987f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.447+0000 7f024f17b700 1 -- 172.21.15.189:0/2271597329 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f022808cd70 msgr2=0x7f022808f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.447+0000 7f024f17b700 1 --2- 172.21.15.189:0/2271597329 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f022808cd70 0x7f022808f230 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0230009a60 tx=0x7f02300092c0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.447+0000 7f024f17b700 1 -- 172.21.15.189:0/2271597329 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024815e500 msgr2=0x7f02480ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.447+0000 7f024f17b700 1 --2- 172.21.15.189:0/2271597329 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024815e500 0x7f02480ceb10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f024000b4a0 tx=0x7f024000b860 comp rx=0 tx=0).stop 2024-04-06T12:12:29.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.447+0000 7f024d718700 1 -- 172.21.15.189:0/2271597329 reap_dead start 2024-04-06T12:12:29.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.447+0000 7f024f17b700 1 -- 172.21.15.189:0/2271597329 shutdown_connections 2024-04-06T12:12:29.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.447+0000 7f024f17b700 1 -- 172.21.15.189:0/2271597329 >> 172.21.15.189:0/2271597329 conn(0x7f02480d00b0 msgr2=0x7f0248071c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.447+0000 7fcbaaffd700 1 -- 172.21.15.189:0/3650478294 <== osd.7 v2:172.21.15.192:6802/47715 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fcb98076d60 con 0x7fcb98074250 2024-04-06T12:12:29.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.448+0000 7f024f17b700 1 -- 172.21.15.189:0/2271597329 shutdown_connections 2024-04-06T12:12:29.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.449+0000 7f024f17b700 1 -- 172.21.15.189:0/2271597329 wait complete. 2024-04-06T12:12:29.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.458+0000 7fe85c81f700 1 -- 172.21.15.189:0/636056931 --> [v2:172.21.15.189:6800/47414,v1:172.21.15.189:6801/47414] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe85806b170 con 0x7fe8380742a0 2024-04-06T12:12:29.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.460+0000 7f8a607c2700 1 -- 172.21.15.189:0/945979242 --> [v2:172.21.15.192:6819/47717,v1:172.21.15.192:6821/47717] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8a40002d40 con 0x7f8a3c074320 2024-04-06T12:12:29.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.460+0000 7f0229f11700 1 -- 172.21.15.189:0/3947046127 --> [v2:172.21.15.189:6824/47425,v1:172.21.15.189:6825/47425] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f02240b28a0 con 0x7f0210074420 2024-04-06T12:12:29.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.460+0000 7f25d664d700 1 -- 172.21.15.189:0/1395053118 --> [v2:172.21.15.192:6800/47714,v1:172.21.15.192:6801/47714] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f25d015f4d0 con 0x7f25bc074320 2024-04-06T12:12:29.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.460+0000 7fe846ffd700 1 -- 172.21.15.189:0/636056931 <== osd.1 v2:172.21.15.189:6800/47414 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe850001f40 con 0x7fe8380742a0 2024-04-06T12:12:29.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.461+0000 7fe85c81f700 1 -- 172.21.15.189:0/636056931 >> [v2:172.21.15.189:6800/47414,v1:172.21.15.189:6801/47414] conn(0x7fe8380742a0 msgr2=0x7fe8380766e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.461+0000 7fe85c81f700 1 --2- 172.21.15.189:0/636056931 >> [v2:172.21.15.189:6800/47414,v1:172.21.15.189:6801/47414] conn(0x7fe8380742a0 0x7fe8380766e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.461+0000 7fe85c81f700 1 -- 172.21.15.189:0/636056931 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe83806ace0 msgr2=0x7fe83806d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.461+0000 7fe85c81f700 1 --2- 172.21.15.189:0/636056931 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe83806ace0 0x7fe83806d1a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe84c009820 tx=0x7fe84c008040 comp rx=0 tx=0).stop 2024-04-06T12:12:29.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.461+0000 7fe85c81f700 1 -- 172.21.15.189:0/636056931 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe85806e470 msgr2=0x7fe858158cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.461+0000 7fe85c81f700 1 --2- 172.21.15.189:0/636056931 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe85806e470 0x7fe858158cd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe84800ade0 tx=0x7fe848004750 comp rx=0 tx=0).stop 2024-04-06T12:12:29.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.461+0000 7fe856d9d700 1 -- 172.21.15.189:0/636056931 reap_dead start 2024-04-06T12:12:29.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.462+0000 7f25ccff9700 1 -- 172.21.15.189:0/1395053118 <== osd.5 v2:172.21.15.192:6800/47714 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f25c800f040 con 0x7f25bc074320 2024-04-06T12:12:29.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.463+0000 7f25d664d700 1 -- 172.21.15.189:0/1395053118 >> [v2:172.21.15.192:6800/47714,v1:172.21.15.192:6801/47714] conn(0x7f25bc074320 msgr2=0x7f25bc076760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.463+0000 7f25d664d700 1 --2- 172.21.15.189:0/1395053118 >> [v2:172.21.15.192:6800/47714,v1:172.21.15.192:6801/47714] conn(0x7f25bc074320 0x7f25bc076760 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.463+0000 7f25d664d700 1 -- 172.21.15.189:0/1395053118 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f25bc06ace0 msgr2=0x7f25bc06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.463+0000 7f25d664d700 1 --2- 172.21.15.189:0/1395053118 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f25bc06ace0 0x7f25bc06d1a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f25c4000f60 tx=0x7f25c4007a70 comp rx=0 tx=0).stop 2024-04-06T12:12:29.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.463+0000 7f25d664d700 1 -- 172.21.15.189:0/1395053118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25d0144f40 msgr2=0x7f25d02038d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.463+0000 7f25d664d700 1 --2- 172.21.15.189:0/1395053118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25d0144f40 0x7f25d02038d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f25c0009b90 tx=0x7f25c0004750 comp rx=0 tx=0).stop 2024-04-06T12:12:29.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.463+0000 7f25d4bea700 1 -- 172.21.15.189:0/1395053118 reap_dead start 2024-04-06T12:12:29.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.459+0000 7fc6bbabd700 1 -- 172.21.15.189:0/1123577174 --> [v2:172.21.15.189:6816/47416,v1:172.21.15.189:6817/47416] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc6b415a740 con 0x7fc698074320 2024-04-06T12:12:29.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.463+0000 7f25d664d700 1 -- 172.21.15.189:0/1395053118 shutdown_connections 2024-04-06T12:12:29.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.463+0000 7f25d664d700 1 -- 172.21.15.189:0/1395053118 >> 172.21.15.189:0/1395053118 conn(0x7f25d00d00b0 msgr2=0x7f25d006f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.463+0000 7fe85c81f700 1 -- 172.21.15.189:0/636056931 shutdown_connections 2024-04-06T12:12:29.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.463+0000 7fe85c81f700 1 -- 172.21.15.189:0/636056931 >> 172.21.15.189:0/636056931 conn(0x7fe8580cc290 msgr2=0x7fe8580c9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.463+0000 7f25d664d700 1 -- 172.21.15.189:0/1395053118 shutdown_connections 2024-04-06T12:12:29.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.463+0000 7fe85c81f700 1 -- 172.21.15.189:0/636056931 shutdown_connections 2024-04-06T12:12:29.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.463+0000 7fe85c81f700 1 -- 172.21.15.189:0/636056931 wait complete. 2024-04-06T12:12:29.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.464+0000 7f020bfff700 1 -- 172.21.15.189:0/3947046127 <== osd.3 v2:172.21.15.189:6824/47425 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0218022040 con 0x7f0210074420 2024-04-06T12:12:29.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.464+0000 7fc6aa7fc700 1 -- 172.21.15.189:0/1123577174 <== osd.2 v2:172.21.15.189:6816/47416 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc6b0009910 con 0x7fc698074320 2024-04-06T12:12:29.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.464+0000 7f0229f11700 1 -- 172.21.15.189:0/3947046127 >> [v2:172.21.15.189:6824/47425,v1:172.21.15.189:6825/47425] conn(0x7f0210074420 msgr2=0x7f0210076860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.464+0000 7f0229f11700 1 --2- 172.21.15.189:0/3947046127 >> [v2:172.21.15.189:6824/47425,v1:172.21.15.189:6825/47425] conn(0x7f0210074420 0x7f0210076860 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.464+0000 7fc6bbabd700 1 -- 172.21.15.189:0/1123577174 >> [v2:172.21.15.189:6816/47416,v1:172.21.15.189:6817/47416] conn(0x7fc698074320 msgr2=0x7fc698076760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.464+0000 7fc6bbabd700 1 --2- 172.21.15.189:0/1123577174 >> [v2:172.21.15.189:6816/47416,v1:172.21.15.189:6817/47416] conn(0x7fc698074320 0x7fc698076760 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.465+0000 7fc6bbabd700 1 -- 172.21.15.189:0/1123577174 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc69806ad60 msgr2=0x7fc69806d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.465+0000 7fc6bbabd700 1 --2- 172.21.15.189:0/1123577174 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc69806ad60 0x7fc69806d220 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4004da0 tx=0x7fc6a4004b60 comp rx=0 tx=0).stop 2024-04-06T12:12:29.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.465+0000 7fc6bbabd700 1 -- 172.21.15.189:0/1123577174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6b4150690 msgr2=0x7fc6b4140620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.465+0000 7fc6bbabd700 1 --2- 172.21.15.189:0/1123577174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6b4150690 0x7fc6b4140620 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac007040 tx=0x7fc6ac004ad0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.465+0000 7fc6ba05a700 1 -- 172.21.15.189:0/1123577174 reap_dead start 2024-04-06T12:12:29.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.465+0000 7fc6bbabd700 1 -- 172.21.15.189:0/1123577174 shutdown_connections 2024-04-06T12:12:29.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.465+0000 7fc6bbabd700 1 -- 172.21.15.189:0/1123577174 >> 172.21.15.189:0/1123577174 conn(0x7fc6b40cefa0 msgr2=0x7fc6b41476a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.465+0000 7fc6bbabd700 1 -- 172.21.15.189:0/1123577174 shutdown_connections 2024-04-06T12:12:29.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.465+0000 7fc6bbabd700 1 -- 172.21.15.189:0/1123577174 wait complete. 2024-04-06T12:12:29.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.464+0000 7f0229f11700 1 -- 172.21.15.189:0/3947046127 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f021006ade0 msgr2=0x7f021006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.464+0000 7f0229f11700 1 --2- 172.21.15.189:0/3947046127 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f021006ade0 0x7f021006d2a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0214007fd0 tx=0x7f0214007980 comp rx=0 tx=0).stop 2024-04-06T12:12:29.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.464+0000 7f0229f11700 1 -- 172.21.15.189:0/3947046127 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02240b8a40 msgr2=0x7f02240ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.464+0000 7f0229f11700 1 --2- 172.21.15.189:0/3947046127 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02240b8a40 0x7f02240ca4a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f021c007d90 tx=0x7f021c00e3c0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.464+0000 7f0223fff700 1 -- 172.21.15.189:0/3947046127 reap_dead start 2024-04-06T12:12:29.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.465+0000 7f0229f11700 1 -- 172.21.15.189:0/3947046127 shutdown_connections 2024-04-06T12:12:29.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.465+0000 7f0229f11700 1 -- 172.21.15.189:0/3947046127 >> 172.21.15.189:0/3947046127 conn(0x7f02240d33d0 msgr2=0x7f02240b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.465+0000 7f0229f11700 1 -- 172.21.15.189:0/3947046127 shutdown_connections 2024-04-06T12:12:29.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.465+0000 7f0229f11700 1 -- 172.21.15.189:0/3947046127 wait complete. 2024-04-06T12:12:29.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.466+0000 7f8a4effd700 1 -- 172.21.15.189:0/945979242 <== osd.6 v2:172.21.15.192:6819/47717 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8a40002d40 con 0x7f8a3c074320 2024-04-06T12:12:29.482 INFO:teuthology.orchestra.run.smithi189.stdout:55834574856 2024-04-06T12:12:29.483 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-06T12:12:29.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.467+0000 7f8a607c2700 1 -- 172.21.15.189:0/945979242 >> [v2:172.21.15.192:6819/47717,v1:172.21.15.192:6821/47717] conn(0x7f8a3c074320 msgr2=0x7f8a3c076760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.467+0000 7f8a607c2700 1 --2- 172.21.15.189:0/945979242 >> [v2:172.21.15.192:6819/47717,v1:172.21.15.192:6821/47717] conn(0x7f8a3c074320 0x7f8a3c076760 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.468+0000 7f25d664d700 1 -- 172.21.15.189:0/1395053118 wait complete. 2024-04-06T12:12:29.484 INFO:teuthology.orchestra.run.smithi189.stdout:55834574857 2024-04-06T12:12:29.485 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-06T12:12:29.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.469+0000 7f8a607c2700 1 -- 172.21.15.189:0/945979242 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8a3c06ace0 msgr2=0x7f8a3c06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.469+0000 7f8a607c2700 1 --2- 172.21.15.189:0/945979242 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8a3c06ace0 0x7f8a3c06d1a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8a50005c90 tx=0x7f8a50009040 comp rx=0 tx=0).stop 2024-04-06T12:12:29.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.469+0000 7f8a607c2700 1 -- 172.21.15.189:0/945979242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a58159b30 msgr2=0x7f8a5814f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.469+0000 7f8a607c2700 1 --2- 172.21.15.189:0/945979242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a58159b30 0x7f8a5814f7f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8a54017000 tx=0x7f8a54004750 comp rx=0 tx=0).stop 2024-04-06T12:12:29.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.469+0000 7f8a5ed5f700 1 -- 172.21.15.189:0/945979242 reap_dead start 2024-04-06T12:12:29.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.470+0000 7f8a607c2700 1 -- 172.21.15.189:0/945979242 shutdown_connections 2024-04-06T12:12:29.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.470+0000 7f8a607c2700 1 -- 172.21.15.189:0/945979242 >> 172.21.15.189:0/945979242 conn(0x7f8a580cefa0 msgr2=0x7f8a580739d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.473+0000 7f8a607c2700 1 -- 172.21.15.189:0/945979242 shutdown_connections 2024-04-06T12:12:29.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.475+0000 7f8a607c2700 1 -- 172.21.15.189:0/945979242 wait complete. 2024-04-06T12:12:29.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.478+0000 7fcbbc2e6700 1 -- 172.21.15.189:0/3650478294 --> [v2:172.21.15.192:6802/47715,v1:172.21.15.192:6804/47715] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcbb414b5a0 con 0x7fcb98074250 2024-04-06T12:12:29.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.479+0000 7fcbaaffd700 1 -- 172.21.15.189:0/3650478294 <== osd.7 v2:172.21.15.192:6802/47715 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcbb414b5a0 con 0x7fcb98074250 2024-04-06T12:12:29.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.479+0000 7fcbbc2e6700 1 -- 172.21.15.189:0/3650478294 >> [v2:172.21.15.192:6802/47715,v1:172.21.15.192:6804/47715] conn(0x7fcb98074250 msgr2=0x7fcb98076690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.479+0000 7fcbbc2e6700 1 --2- 172.21.15.189:0/3650478294 >> [v2:172.21.15.192:6802/47715,v1:172.21.15.192:6804/47715] conn(0x7fcb98074250 0x7fcb98076690 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.479+0000 7fcbbc2e6700 1 -- 172.21.15.189:0/3650478294 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcb9806ac90 msgr2=0x7fcb9806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.479+0000 7fcbbc2e6700 1 --2- 172.21.15.189:0/3650478294 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcb9806ac90 0x7fcb9806d150 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcba4006fd0 tx=0x7fcba4006ba0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.479+0000 7fcbbc2e6700 1 -- 172.21.15.189:0/3650478294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb406a730 msgr2=0x7fcbb415dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.479+0000 7fcbbc2e6700 1 --2- 172.21.15.189:0/3650478294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb406a730 0x7fcbb415dd90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0000c00 tx=0x7fcbb0004750 comp rx=0 tx=0).stop 2024-04-06T12:12:29.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.479+0000 7fcbba883700 1 -- 172.21.15.189:0/3650478294 reap_dead start 2024-04-06T12:12:29.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.480+0000 7fcbbc2e6700 1 -- 172.21.15.189:0/3650478294 shutdown_connections 2024-04-06T12:12:29.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.480+0000 7fcbbc2e6700 1 -- 172.21.15.189:0/3650478294 >> 172.21.15.189:0/3650478294 conn(0x7fcbb40d00b0 msgr2=0x7fcbb406eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.481+0000 7fcbbc2e6700 1 -- 172.21.15.189:0/3650478294 shutdown_connections 2024-04-06T12:12:29.488 INFO:teuthology.orchestra.run.smithi189.stdout:55834574857 2024-04-06T12:12:29.489 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-06T12:12:29.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.484+0000 7fcbbc2e6700 1 -- 172.21.15.189:0/3650478294 wait complete. 2024-04-06T12:12:29.490 INFO:teuthology.orchestra.run.smithi189.stdout:55834574858 2024-04-06T12:12:29.490 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-06T12:12:29.491 INFO:teuthology.orchestra.run.smithi189.stdout:55834574857 2024-04-06T12:12:29.491 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-06T12:12:29.492 INFO:teuthology.orchestra.run.smithi189.stdout:55834574856 2024-04-06T12:12:29.492 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-06T12:12:29.494 INFO:teuthology.orchestra.run.smithi189.stdout:55834574857 2024-04-06T12:12:29.495 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-06T12:12:29.498 INFO:teuthology.orchestra.run.smithi189.stdout:55834574858 2024-04-06T12:12:29.498 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-06T12:12:29.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.688+0000 7fd4f58d4700 1 Processor -- start 2024-04-06T12:12:29.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.688+0000 7fd4f58d4700 1 -- start start 2024-04-06T12:12:29.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.689+0000 7fd4f58d4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4f015d7e0 0x7fd4f00d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.689+0000 7fd4f58d4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd4f00d0890 0x7fd4f00d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.689+0000 7fd4f58d4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4f00cdf80 con 0x7fd4f00d1af0 2024-04-06T12:12:29.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.689+0000 7fd4f58d4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4f00d0d10 con 0x7fd4f00d0890 2024-04-06T12:12:29.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.689+0000 7fd4f58d4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f00d1f80 con 0x7fd4f015d7e0 2024-04-06T12:12:29.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.689+0000 7fd4eeffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4f015d7e0 0x7fd4f00d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.690+0000 7fd4eeffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4f015d7e0 0x7fd4f00d0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59458/0 (socket says 172.21.15.189:59458) 2024-04-06T12:12:29.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.690+0000 7fd4eeffd700 1 -- 172.21.15.189:0/1855469496 learned_addr learned my addr 172.21.15.189:0/1855469496 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.690+0000 7fd4eeffd700 1 -- 172.21.15.189:0/1855469496 >> v2:172.21.15.192:3300/0 conn(0x7fd4f00d0890 msgr2=0x7fd4f00d15b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-06T12:12:29.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.690+0000 7fd4eeffd700 1 --2- 172.21.15.189:0/1855469496 >> v2:172.21.15.192:3300/0 conn(0x7fd4f00d0890 0x7fd4f00d15b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.690+0000 7fd4eeffd700 1 -- 172.21.15.189:0/1855469496 >> v1:172.21.15.192:6789/0 conn(0x7fd4f00d1af0 legacy=0x7fd4f00b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:29.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.690+0000 7fd4eeffd700 1 -- 172.21.15.189:0/1855469496 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f00d3230 con 0x7fd4f015d7e0 2024-04-06T12:12:29.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.690+0000 7fd4eeffd700 1 --2- 172.21.15.189:0/1855469496 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4f015d7e0 0x7fd4f00d0350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd4e00098f0 tx=0x7fd4e0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b9a97f936760f98 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.691+0000 7fd4ee7fc700 1 -- 172.21.15.189:0/1855469496 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0014e90 con 0x7fd4f015d7e0 2024-04-06T12:12:29.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.691+0000 7fd4ee7fc700 1 -- 172.21.15.189:0/1855469496 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e0004030 con 0x7fd4f015d7e0 2024-04-06T12:12:29.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.691+0000 7fd4ee7fc700 1 -- 172.21.15.189:0/1855469496 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e00043c0 con 0x7fd4f015d7e0 2024-04-06T12:12:29.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.692+0000 7fd4f58d4700 1 -- 172.21.15.189:0/1855469496 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4f015d7e0 msgr2=0x7fd4f00d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.692+0000 7fd4f58d4700 1 --2- 172.21.15.189:0/1855469496 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4f015d7e0 0x7fd4f00d0350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd4e00098f0 tx=0x7fd4e0009c00 comp rx=0 tx=0).stop 2024-04-06T12:12:29.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.692+0000 7fd4f58d4700 1 -- 172.21.15.189:0/1855469496 shutdown_connections 2024-04-06T12:12:29.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.692+0000 7fd4f58d4700 1 --2- 172.21.15.189:0/1855469496 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4f015d7e0 0x7fd4f00d0350 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.692+0000 7fd4f58d4700 1 --2- 172.21.15.189:0/1855469496 >> v2:172.21.15.192:3300/0 conn(0x7fd4f00d0890 0x7fd4f00d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.692+0000 7fd4f58d4700 1 -- 172.21.15.189:0/1855469496 >> 172.21.15.189:0/1855469496 conn(0x7fd4f0072250 msgr2=0x7fd4f0072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.698+0000 7fd4f58d4700 1 -- 172.21.15.189:0/1855469496 shutdown_connections 2024-04-06T12:12:29.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.698+0000 7fd4f58d4700 1 -- 172.21.15.189:0/1855469496 wait complete. 2024-04-06T12:12:29.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.700+0000 7fd4f58d4700 1 Processor -- start 2024-04-06T12:12:29.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.700+0000 7fd4f58d4700 1 -- start start 2024-04-06T12:12:29.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.701+0000 7fd4f58d4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4f00d0890 0x7fd4f015a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.701+0000 7fd4f58d4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4f00d1af0 0x7fd4f01579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.701+0000 7fd4f58d4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4f015d7e0 0x7fd4f00cbf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.701+0000 7fd4f58d4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f00b3070 con 0x7fd4f00d0890 2024-04-06T12:12:29.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.701+0000 7fd4f58d4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f00cc440 con 0x7fd4f015d7e0 2024-04-06T12:12:29.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.701+0000 7fd4f58d4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f015af10 con 0x7fd4f00d1af0 2024-04-06T12:12:29.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.701+0000 7fd4eeffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4f00d0890 0x7fd4f015a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.701+0000 7fd4eeffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4f00d0890 0x7fd4f015a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38094/0 (socket says 172.21.15.189:38094) 2024-04-06T12:12:29.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.701+0000 7fd4eeffd700 1 -- 172.21.15.189:0/30185214 learned_addr learned my addr 172.21.15.189:0/30185214 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.701+0000 7fd4ef7fe700 1 --2- 172.21.15.189:0/30185214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4f015d7e0 0x7fd4f00cbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.702+0000 7fd4e7fff700 1 --2- 172.21.15.189:0/30185214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4f00d1af0 0x7fd4f01579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.702+0000 7fd4eeffd700 1 -- 172.21.15.189:0/30185214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4f00d1af0 msgr2=0x7fd4f01579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.702+0000 7fd4eeffd700 1 --2- 172.21.15.189:0/30185214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4f00d1af0 0x7fd4f01579b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.702+0000 7fd4eeffd700 1 -- 172.21.15.189:0/30185214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4f015d7e0 msgr2=0x7fd4f00cbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.702+0000 7fd4eeffd700 1 --2- 172.21.15.189:0/30185214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4f015d7e0 0x7fd4f00cbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.702+0000 7fd4eeffd700 1 -- 172.21.15.189:0/30185214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e0009550 con 0x7fd4f00d0890 2024-04-06T12:12:29.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.702+0000 7fd4eeffd700 1 --2- 172.21.15.189:0/30185214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4f00d0890 0x7fd4f015a900 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0009c40 tx=0x7fd4e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.707+0000 7fd4ecff9700 1 -- 172.21.15.189:0/30185214 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0003dc0 con 0x7fd4f00d0890 2024-04-06T12:12:29.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.707+0000 7fd4ecff9700 1 -- 172.21.15.189:0/30185214 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e00143e0 con 0x7fd4f00d0890 2024-04-06T12:12:29.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.707+0000 7fd4ecff9700 1 -- 172.21.15.189:0/30185214 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0014790 con 0x7fd4f00d0890 2024-04-06T12:12:29.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.707+0000 7fd4f58d4700 1 -- 172.21.15.189:0/30185214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f015b1f0 con 0x7fd4f00d0890 2024-04-06T12:12:29.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.707+0000 7fd4f58d4700 1 -- 172.21.15.189:0/30185214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f015b7b0 con 0x7fd4f00d0890 2024-04-06T12:12:29.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.708+0000 7fd4f58d4700 1 -- 172.21.15.189:0/30185214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d40052f0 con 0x7fd4f00d0890 2024-04-06T12:12:29.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.710+0000 7fd4ecff9700 1 -- 172.21.15.189:0/30185214 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd4e0013450 con 0x7fd4f00d0890 2024-04-06T12:12:29.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.711+0000 7fd4ecff9700 1 --2- 172.21.15.189:0/30185214 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd4d006ad60 0x7fd4d006d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.711+0000 7fd4ecff9700 1 -- 172.21.15.189:0/30185214 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd4e008b500 con 0x7fd4f00d0890 2024-04-06T12:12:29.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.711+0000 7fca4ad2c700 1 Processor -- start 2024-04-06T12:12:29.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.711+0000 7fca4ad2c700 1 -- start start 2024-04-06T12:12:29.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.711+0000 7fd4e7fff700 1 --2- 172.21.15.189:0/30185214 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd4d006ad60 0x7fd4d006d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.712+0000 7fd4e7fff700 1 --2- 172.21.15.189:0/30185214 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd4d006ad60 0x7fd4d006d220 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd4f015ae40 tx=0x7fd4d8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.712+0000 7fca4ad2c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fca440d11b0 0x7fca440d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.712+0000 7fca4ad2c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fca440cfa00 0x7fca440cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.712+0000 7fca4ad2c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca440d1ad0 con 0x7fca440cb870 2024-04-06T12:12:29.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.712+0000 7fca4ad2c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca440ccdc0 con 0x7fca440cfa00 2024-04-06T12:12:29.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.712+0000 7fca4ad2c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca440d0b50 con 0x7fca440d11b0 2024-04-06T12:12:29.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.713+0000 7fca48ac8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fca440d11b0 0x7fca440d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.713+0000 7fca48ac8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fca440d11b0 0x7fca440d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:58844/0 (socket says 172.21.15.189:58844) 2024-04-06T12:12:29.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.713+0000 7fca48ac8700 1 -- 172.21.15.189:0/2761421187 learned_addr learned my addr 172.21.15.189:0/2761421187 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.713+0000 7fca48ac8700 1 -- 172.21.15.189:0/2761421187 >> v2:172.21.15.189:3300/0 conn(0x7fca440cfa00 msgr2=0x7fca440cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:29.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.713+0000 7fca48ac8700 1 --2- 172.21.15.189:0/2761421187 >> v2:172.21.15.189:3300/0 conn(0x7fca440cfa00 0x7fca440cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.713+0000 7fca48ac8700 1 -- 172.21.15.189:0/2761421187 >> v1:172.21.15.189:6789/0 conn(0x7fca440cb870 legacy=0x7fca4415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:29.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.713+0000 7fca48ac8700 1 -- 172.21.15.189:0/2761421187 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca440cbd80 con 0x7fca440d11b0 2024-04-06T12:12:29.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.713+0000 7fca48ac8700 1 --2- 172.21.15.189:0/2761421187 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca440d11b0 0x7fca440d1590 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fca440ced00 tx=0x7fca30009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7a6880cedc5580a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.713+0000 7fca437fe700 1 -- 172.21.15.189:0/2761421187 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca30004230 con 0x7fca440d11b0 2024-04-06T12:12:29.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.713+0000 7fca437fe700 1 -- 172.21.15.189:0/2761421187 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca30004390 con 0x7fca440d11b0 2024-04-06T12:12:29.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.713+0000 7fca437fe700 1 -- 172.21.15.189:0/2761421187 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca300036a0 con 0x7fca440d11b0 2024-04-06T12:12:29.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.714+0000 7fca4ad2c700 1 -- 172.21.15.189:0/2761421187 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca440d11b0 msgr2=0x7fca440d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.714+0000 7fca4ad2c700 1 --2- 172.21.15.189:0/2761421187 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca440d11b0 0x7fca440d1590 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fca440ced00 tx=0x7fca30009bd0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.716+0000 7efda2987700 1 Processor -- start 2024-04-06T12:12:29.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.716+0000 7efda2987700 1 -- start start 2024-04-06T12:12:29.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.717+0000 7efda2987700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efd9c0cc0e0 0x7efd9c13e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.717+0000 7efda2987700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efd9c15bc70 0x7efd9c15c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.717+0000 7efda2987700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c0cdeb0 con 0x7efd9c13e340 2024-04-06T12:12:29.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.717+0000 7efda2987700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd9c0ce190 con 0x7efd9c0cc0e0 2024-04-06T12:12:29.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.717+0000 7efda2987700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd9c0ce470 con 0x7efd9c15bc70 2024-04-06T12:12:29.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.717+0000 7efda0f24700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efd9c15bc70 0x7efd9c15c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.717+0000 7efda0f24700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efd9c15bc70 0x7efd9c15c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:58852/0 (socket says 172.21.15.189:58852) 2024-04-06T12:12:29.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.717+0000 7efda0f24700 1 -- 172.21.15.189:0/2117485009 learned_addr learned my addr 172.21.15.189:0/2117485009 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.717+0000 7f0496de6700 1 Processor -- start 2024-04-06T12:12:29.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.717+0000 7fd4ecff9700 1 -- 172.21.15.189:0/30185214 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd4e0056460 con 0x7fd4f00d0890 2024-04-06T12:12:29.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.718+0000 7efda0f24700 1 -- 172.21.15.189:0/2117485009 >> v2:172.21.15.189:3300/0 conn(0x7efd9c0cc0e0 msgr2=0x7efd9c13e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:29.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.718+0000 7efda0f24700 1 --2- 172.21.15.189:0/2117485009 >> v2:172.21.15.189:3300/0 conn(0x7efd9c0cc0e0 0x7efd9c13e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.718+0000 7efda0f24700 1 -- 172.21.15.189:0/2117485009 >> v1:172.21.15.189:6789/0 conn(0x7efd9c13e340 legacy=0x7efd9c0cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:29.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.718+0000 7efda0f24700 1 -- 172.21.15.189:0/2117485009 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9c0ce750 con 0x7efd9c15bc70 2024-04-06T12:12:29.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.718+0000 7fca4ad2c700 1 -- 172.21.15.189:0/2761421187 shutdown_connections 2024-04-06T12:12:29.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.718+0000 7fca4ad2c700 1 --2- 172.21.15.189:0/2761421187 >> v2:172.21.15.189:3300/0 conn(0x7fca440cfa00 0x7fca440cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.718+0000 7fca4ad2c700 1 --2- 172.21.15.189:0/2761421187 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca440d11b0 0x7fca440d1590 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.718+0000 7fca4ad2c700 1 -- 172.21.15.189:0/2761421187 >> 172.21.15.189:0/2761421187 conn(0x7fca440b2db0 msgr2=0x7fca440b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.718+0000 7fca4ad2c700 1 -- 172.21.15.189:0/2761421187 shutdown_connections 2024-04-06T12:12:29.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.718+0000 7efda0f24700 1 --2- 172.21.15.189:0/2117485009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd9c15bc70 0x7efd9c15c100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efd9c06b460 tx=0x7efd9400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8eb7128b6f6b1cc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.718+0000 7efd9affd700 1 -- 172.21.15.189:0/2117485009 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9400e070 con 0x7efd9c15bc70 2024-04-06T12:12:29.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.718+0000 7efd9affd700 1 -- 172.21.15.189:0/2117485009 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd94003a20 con 0x7efd9c15bc70 2024-04-06T12:12:29.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.718+0000 7efd9affd700 1 -- 172.21.15.189:0/2117485009 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd94003dd0 con 0x7efd9c15bc70 2024-04-06T12:12:29.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.719+0000 7efda2987700 1 -- 172.21.15.189:0/2117485009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd9c15bc70 msgr2=0x7efd9c15c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.719+0000 7efda2987700 1 --2- 172.21.15.189:0/2117485009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd9c15bc70 0x7efd9c15c100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efd9c06b460 tx=0x7efd9400b330 comp rx=0 tx=0).stop 2024-04-06T12:12:29.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.718+0000 7fca4ad2c700 1 -- 172.21.15.189:0/2761421187 wait complete. 2024-04-06T12:12:29.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.720+0000 7fca4ad2c700 1 Processor -- start 2024-04-06T12:12:29.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.721+0000 7f829afd7700 1 Processor -- start 2024-04-06T12:12:29.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7efda2987700 1 -- 172.21.15.189:0/2117485009 shutdown_connections 2024-04-06T12:12:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7efda2987700 1 --2- 172.21.15.189:0/2117485009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd9c15bc70 0x7efd9c15c100 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7efda2987700 1 --2- 172.21.15.189:0/2117485009 >> v2:172.21.15.189:3300/0 conn(0x7efd9c0cc0e0 0x7efd9c13e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7efda2987700 1 -- 172.21.15.189:0/2117485009 >> 172.21.15.189:0/2117485009 conn(0x7efd9c06a730 msgr2=0x7efd9c06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.721+0000 7f829afd7700 1 -- start start 2024-04-06T12:12:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7f829afd7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8294159b30 0x7f829406da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7f829afd7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f829406df60 0x7f829415b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7f829afd7700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829406e3b0 con 0x7f8294150690 2024-04-06T12:12:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7f829afd7700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8294150a70 con 0x7f829406df60 2024-04-06T12:12:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7f829afd7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8294159f90 con 0x7f8294159b30 2024-04-06T12:12:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.720+0000 7fca4ad2c700 1 -- start start 2024-04-06T12:12:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.721+0000 7fca4ad2c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fca440cb870 0x7fca44144f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.721+0000 7fca4ad2c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca440cfa00 0x7fca4414cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.721+0000 7fca4ad2c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca440d11b0 0x7fca4414d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.721+0000 7fca4ad2c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca441454c0 con 0x7fca440cfa00 2024-04-06T12:12:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.721+0000 7fca4ad2c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca440c4dd0 con 0x7fca440d11b0 2024-04-06T12:12:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.721+0000 7fca4ad2c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca440c50b0 con 0x7fca440cb870 2024-04-06T12:12:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7fca48ac8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fca440cb870 0x7fca44144f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7fca48ac8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fca440cb870 0x7fca44144f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59480/0 (socket says 172.21.15.189:59480) 2024-04-06T12:12:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7fca48ac8700 1 -- 172.21.15.189:0/4150698729 learned_addr learned my addr 172.21.15.189:0/4150698729 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7fca43fff700 1 --2- 172.21.15.189:0/4150698729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca440cfa00 0x7fca4414cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7fca492c9700 1 --2- 172.21.15.189:0/4150698729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca440d11b0 0x7fca4414d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7f8298d73700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f8294150690 0x7f82940b2300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:52476/0 (socket says 172.21.15.189:52476) 2024-04-06T12:12:29.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7f8298d73700 1 -- 172.21.15.189:0/2718846687 learned_addr learned my addr 172.21.15.189:0/2718846687 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:29.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7fca48ac8700 1 -- 172.21.15.189:0/4150698729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca440d11b0 msgr2=0x7fca4414d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7fca48ac8700 1 --2- 172.21.15.189:0/4150698729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca440d11b0 0x7fca4414d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7fca48ac8700 1 -- 172.21.15.189:0/4150698729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca440cfa00 msgr2=0x7fca4414cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7fca48ac8700 1 --2- 172.21.15.189:0/4150698729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca440cfa00 0x7fca4414cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7fca48ac8700 1 -- 172.21.15.189:0/4150698729 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca30009550 con 0x7fca440cb870 2024-04-06T12:12:29.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7fca492c9700 1 --2- 172.21.15.189:0/4150698729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca440d11b0 0x7fca4414d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:29.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.723+0000 7f8299574700 1 --2- 172.21.15.189:0/2718846687 >> v2:172.21.15.189:3300/0 conn(0x7f829406df60 0x7f829415b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7fca48ac8700 1 --2- 172.21.15.189:0/4150698729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fca440cb870 0x7fca44144f80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fca30009130 tx=0x7fca30004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.723+0000 7fca417fa700 1 -- 172.21.15.189:0/4150698729 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca30003d10 con 0x7fca440cb870 2024-04-06T12:12:29.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.723+0000 7fca417fa700 1 -- 172.21.15.189:0/4150698729 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3001f070 con 0x7fca440cb870 2024-04-06T12:12:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.719+0000 7f0496de6700 1 -- start start 2024-04-06T12:12:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.720+0000 7f0496de6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0490161980 0x7f04900cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.720+0000 7f0496de6700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f049006b050 0x7f04900cd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.720+0000 7f0496de6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0490161d60 con 0x7f04900c2a70 2024-04-06T12:12:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.720+0000 7f0496de6700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f049006c130 con 0x7f049006b050 2024-04-06T12:12:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.720+0000 7f0496de6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049006b5b0 con 0x7f0490161980 2024-04-06T12:12:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.721+0000 7f0494b82700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0490161980 0x7f04900cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.721+0000 7f0494b82700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0490161980 0x7f04900cd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59478/0 (socket says 172.21.15.189:59478) 2024-04-06T12:12:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.721+0000 7f0494b82700 1 -- 172.21.15.189:0/3014047200 learned_addr learned my addr 172.21.15.189:0/3014047200 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.721+0000 7f0494b82700 1 -- 172.21.15.189:0/3014047200 >> v2:172.21.15.192:3300/0 conn(0x7f049006b050 msgr2=0x7f04900cd920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-06T12:12:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.721+0000 7f0494b82700 1 --2- 172.21.15.189:0/3014047200 >> v2:172.21.15.192:3300/0 conn(0x7f049006b050 0x7f04900cd920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.721+0000 7f0494b82700 1 -- 172.21.15.189:0/3014047200 >> v1:172.21.15.189:6789/0 conn(0x7f04900c2a70 legacy=0x7f0490165480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-06T12:12:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.721+0000 7f0494b82700 1 -- 172.21.15.189:0/3014047200 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049006b890 con 0x7f0490161980 2024-04-06T12:12:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.721+0000 7f0494b82700 1 --2- 172.21.15.189:0/3014047200 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0490161980 0x7f04900cd3e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f04840098f0 tx=0x7f0484009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f894ae75f058603 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7f048f7fe700 1 -- 172.21.15.189:0/3014047200 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0484004230 con 0x7f0490161980 2024-04-06T12:12:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7f048f7fe700 1 -- 172.21.15.189:0/3014047200 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0484004390 con 0x7f0490161980 2024-04-06T12:12:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.722+0000 7f048f7fe700 1 -- 172.21.15.189:0/3014047200 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04840036a0 con 0x7f0490161980 2024-04-06T12:12:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.723+0000 7f0496de6700 1 -- 172.21.15.189:0/3014047200 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0490161980 msgr2=0x7f04900cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.723+0000 7f0496de6700 1 --2- 172.21.15.189:0/3014047200 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0490161980 0x7f04900cd3e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f04840098f0 tx=0x7f0484009c00 comp rx=0 tx=0).stop 2024-04-06T12:12:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.723+0000 7f0496de6700 1 -- 172.21.15.189:0/3014047200 shutdown_connections 2024-04-06T12:12:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.723+0000 7f0496de6700 1 --2- 172.21.15.189:0/3014047200 >> v2:172.21.15.192:3300/0 conn(0x7f049006b050 0x7f04900cd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.723+0000 7f0496de6700 1 --2- 172.21.15.189:0/3014047200 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0490161980 0x7f04900cd3e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.723+0000 7f0496de6700 1 -- 172.21.15.189:0/3014047200 >> 172.21.15.189:0/3014047200 conn(0x7f04900cc290 msgr2=0x7f04900cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.723+0000 7f8299574700 1 -- 172.21.15.189:0/2718846687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8294159b30 msgr2=0x7f829406da20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-06T12:12:29.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.723+0000 7f8299574700 1 --2- 172.21.15.189:0/2718846687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8294159b30 0x7f829406da20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.723+0000 7f8299574700 1 -- 172.21.15.189:0/2718846687 >> v1:172.21.15.192:6789/0 conn(0x7f8294150690 legacy=0x7f82940b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.723+0000 7f8299574700 1 -- 172.21.15.189:0/2718846687 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829415a180 con 0x7f829406df60 2024-04-06T12:12:29.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.723+0000 7f8299574700 1 --2- 172.21.15.189:0/2718846687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f829406df60 0x7f829415b220 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f828c009c80 tx=0x7f828c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca0c480030682fbd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.723+0000 7fca4ad2c700 1 -- 172.21.15.189:0/4150698729 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca440c5390 con 0x7fca440cb870 2024-04-06T12:12:29.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.723+0000 7fca4ad2c700 1 -- 172.21.15.189:0/4150698729 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca44146940 con 0x7fca440cb870 2024-04-06T12:12:29.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.723+0000 7fd3039f6700 1 Processor -- start 2024-04-06T12:12:29.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.724+0000 7efda2987700 1 -- 172.21.15.189:0/2117485009 shutdown_connections 2024-04-06T12:12:29.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.724+0000 7efda2987700 1 -- 172.21.15.189:0/2117485009 wait complete. 2024-04-06T12:12:29.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.724+0000 7fca417fa700 1 -- 172.21.15.189:0/4150698729 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca30011850 con 0x7fca440cb870 2024-04-06T12:12:29.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.724+0000 7f82937fe700 1 -- 172.21.15.189:0/2718846687 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c003a20 con 0x7f829406df60 2024-04-06T12:12:29.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.724+0000 7f82937fe700 1 -- 172.21.15.189:0/2718846687 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828c003b80 con 0x7f829406df60 2024-04-06T12:12:29.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.724+0000 7f82937fe700 1 -- 172.21.15.189:0/2718846687 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c00c9b0 con 0x7f829406df60 2024-04-06T12:12:29.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.725+0000 7f829afd7700 1 -- 172.21.15.189:0/2718846687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f829406df60 msgr2=0x7f829415b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.725+0000 7f829afd7700 1 --2- 172.21.15.189:0/2718846687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f829406df60 0x7f829415b220 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f828c009c80 tx=0x7f828c009f90 comp rx=0 tx=0).stop 2024-04-06T12:12:29.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.724+0000 7fd3039f6700 1 -- start start 2024-04-06T12:12:29.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.725+0000 7fd3039f6700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd2fc072650 0x7fd2fc06da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.725+0000 7fd3039f6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2fc06df60 0x7fd2fc16b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.725+0000 7fd3039f6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2fc06e3b0 con 0x7fd2fc161a30 2024-04-06T12:12:29.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.725+0000 7fd3039f6700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2fc0b1fb0 con 0x7fd2fc072650 2024-04-06T12:12:29.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.725+0000 7fd3039f6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc0cba40 con 0x7fd2fc06df60 2024-04-06T12:12:29.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.725+0000 7fd300f91700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd2fc072650 0x7fd2fc06da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.725+0000 7fd300f91700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd2fc072650 0x7fd2fc06da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:58870/0 (socket says 172.21.15.189:58870) 2024-04-06T12:12:29.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.725+0000 7fd300f91700 1 -- 172.21.15.189:0/472913489 learned_addr learned my addr 172.21.15.189:0/472913489 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.726+0000 7f0496de6700 1 -- 172.21.15.189:0/3014047200 shutdown_connections 2024-04-06T12:12:29.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.726+0000 7f829afd7700 1 -- 172.21.15.189:0/2718846687 shutdown_connections 2024-04-06T12:12:29.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.726+0000 7f829afd7700 1 --2- 172.21.15.189:0/2718846687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f829406df60 0x7f829415b220 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.726+0000 7f829afd7700 1 --2- 172.21.15.189:0/2718846687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8294159b30 0x7f829406da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.726+0000 7f829afd7700 1 -- 172.21.15.189:0/2718846687 >> 172.21.15.189:0/2718846687 conn(0x7f82940cefa0 msgr2=0x7f82940cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.726+0000 7f829afd7700 1 -- 172.21.15.189:0/2718846687 shutdown_connections 2024-04-06T12:12:29.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.726+0000 7fca417fa700 1 -- 172.21.15.189:0/4150698729 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fca30021030 con 0x7fca440cb870 2024-04-06T12:12:29.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.726+0000 7fca417fa700 1 --2- 172.21.15.189:0/4150698729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fca2806ace0 0x7fca2806d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.726+0000 7fca417fa700 1 -- 172.21.15.189:0/4150698729 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fca3008ac70 con 0x7fca440cb870 2024-04-06T12:12:29.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.726+0000 7efda2987700 1 Processor -- start 2024-04-06T12:12:29.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.726+0000 7f0496de6700 1 -- 172.21.15.189:0/3014047200 wait complete. 2024-04-06T12:12:29.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.727+0000 7f0496de6700 1 Processor -- start 2024-04-06T12:12:29.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.727+0000 7efda2987700 1 -- start start 2024-04-06T12:12:29.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.728+0000 7efda2987700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd9c0cc0e0 0x7efd9c152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.728+0000 7efda2987700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd9c13e340 0x7efd9c153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.728+0000 7efda2987700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd9c15bc70 0x7efd9c154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.728+0000 7efda2987700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c156330 con 0x7efd9c13e340 2024-04-06T12:12:29.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.728+0000 7efda2987700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c0b0c90 con 0x7efd9c15bc70 2024-04-06T12:12:29.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.728+0000 7efda2987700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c156580 con 0x7efd9c0cc0e0 2024-04-06T12:12:29.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.728+0000 7efd9bfff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd9c0cc0e0 0x7efd9c152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.728+0000 7efd9bfff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd9c0cc0e0 0x7efd9c152c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59492/0 (socket says 172.21.15.189:59492) 2024-04-06T12:12:29.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.728+0000 7efd9bfff700 1 -- 172.21.15.189:0/337024570 learned_addr learned my addr 172.21.15.189:0/337024570 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.728+0000 7efd9bfff700 1 -- 172.21.15.189:0/337024570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd9c15bc70 msgr2=0x7efd9c154350 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-06T12:12:29.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.728+0000 7efd9bfff700 1 --2- 172.21.15.189:0/337024570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd9c15bc70 0x7efd9c154350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.728+0000 7efd9bfff700 1 -- 172.21.15.189:0/337024570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd9c13e340 msgr2=0x7efd9c153e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.728+0000 7efd9bfff700 1 --2- 172.21.15.189:0/337024570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd9c13e340 0x7efd9c153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.728+0000 7efd9bfff700 1 -- 172.21.15.189:0/337024570 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd94009b30 con 0x7efd9c0cc0e0 2024-04-06T12:12:29.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.726+0000 7f829afd7700 1 -- 172.21.15.189:0/2718846687 wait complete. 2024-04-06T12:12:29.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.728+0000 7f829afd7700 1 Processor -- start 2024-04-06T12:12:29.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.728+0000 7fca43fff700 1 --2- 172.21.15.189:0/4150698729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fca2806ace0 0x7fca2806d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.729+0000 7fca43fff700 1 --2- 172.21.15.189:0/4150698729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fca2806ace0 0x7fca2806d1a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fca440d21c0 tx=0x7fca380090e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.729+0000 7efd9bfff700 1 --2- 172.21.15.189:0/337024570 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd9c0cc0e0 0x7efd9c152c40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efd9000c4d0 tx=0x7efd9000c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.728+0000 7f0496de6700 1 -- start start 2024-04-06T12:12:29.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.729+0000 7f0496de6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f049006b050 0x7f049015afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.729+0000 7f0496de6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04900c2a70 0x7f0490162910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.729+0000 7f0496de6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0490161980 0x7f04901594b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.729+0000 7f0496de6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0490163920 con 0x7f0490161980 2024-04-06T12:12:29.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.729+0000 7f0496de6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0490162e50 con 0x7f049006b050 2024-04-06T12:12:29.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.729+0000 7f0496de6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0490163080 con 0x7f04900c2a70 2024-04-06T12:12:29.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.729+0000 7f048ffff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04900c2a70 0x7f0490162910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.729+0000 7f048ffff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04900c2a70 0x7f0490162910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59508/0 (socket says 172.21.15.189:59508) 2024-04-06T12:12:29.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.729+0000 7f048ffff700 1 -- 172.21.15.189:0/2703372104 learned_addr learned my addr 172.21.15.189:0/2703372104 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.729+0000 7f048ffff700 1 -- 172.21.15.189:0/2703372104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f049006b050 msgr2=0x7f049015afb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-06T12:12:29.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.729+0000 7f048ffff700 1 --2- 172.21.15.189:0/2703372104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f049006b050 0x7f049015afb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.729+0000 7f048ffff700 1 -- 172.21.15.189:0/2703372104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0490161980 msgr2=0x7f04901594b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.729+0000 7f048ffff700 1 --2- 172.21.15.189:0/2703372104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0490161980 0x7f04901594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.729+0000 7f048ffff700 1 -- 172.21.15.189:0/2703372104 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0484009550 con 0x7f04900c2a70 2024-04-06T12:12:29.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.730+0000 7f048ffff700 1 --2- 172.21.15.189:0/2703372104 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04900c2a70 0x7f0490162910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f047c00b550 tx=0x7f047c00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.730+0000 7fca4ad2c700 1 -- 172.21.15.189:0/4150698729 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca440c58b0 con 0x7fca440cb870 2024-04-06T12:12:29.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.730+0000 7efd98ff9700 1 -- 172.21.15.189:0/337024570 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd900054e0 con 0x7efd9c0cc0e0 2024-04-06T12:12:29.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.730+0000 7efd98ff9700 1 -- 172.21.15.189:0/337024570 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd90003680 con 0x7efd9c0cc0e0 2024-04-06T12:12:29.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.730+0000 7efd98ff9700 1 -- 172.21.15.189:0/337024570 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90003a30 con 0x7efd9c0cc0e0 2024-04-06T12:12:29.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.729+0000 7f829afd7700 1 -- start start 2024-04-06T12:12:29.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.730+0000 7f829afd7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f829406df60 0x7f82940cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.730+0000 7f829afd7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8294150690 0x7f829414c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.730+0000 7f829afd7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8294159b30 0x7f82940cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.730+0000 7f829afd7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82941f7080 con 0x7f829406df60 2024-04-06T12:12:29.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.730+0000 7f829afd7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82940ce070 con 0x7f8294150690 2024-04-06T12:12:29.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.730+0000 7f829afd7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82940ce350 con 0x7f8294159b30 2024-04-06T12:12:29.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.730+0000 7f8293fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8294150690 0x7f829414c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.730+0000 7f8293fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8294150690 0x7f829414c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:58894/0 (socket says 172.21.15.189:58894) 2024-04-06T12:12:29.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.730+0000 7f8293fff700 1 -- 172.21.15.189:0/1326876708 learned_addr learned my addr 172.21.15.189:0/1326876708 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.730+0000 7f8299574700 1 --2- 172.21.15.189:0/1326876708 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8294159b30 0x7f82940cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.731+0000 7f8298d73700 1 --2- 172.21.15.189:0/1326876708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f829406df60 0x7f82940cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.726+0000 7fd300f91700 1 -- 172.21.15.189:0/472913489 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2fc06df60 msgr2=0x7fd2fc16b670 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-06T12:12:29.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.726+0000 7fd300f91700 1 --2- 172.21.15.189:0/472913489 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2fc06df60 0x7fd2fc16b670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.726+0000 7fd300f91700 1 -- 172.21.15.189:0/472913489 >> v1:172.21.15.189:6789/0 conn(0x7fd2fc161a30 legacy=0x7fd2fc06eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.726+0000 7fd300f91700 1 -- 172.21.15.189:0/472913489 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc0b2c60 con 0x7fd2fc072650 2024-04-06T12:12:29.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.726+0000 7fd300f91700 1 --2- 172.21.15.189:0/472913489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2fc072650 0x7fd2fc06da20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4009e50 tx=0x7fd2f400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d137517e48b4b129 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.731+0000 7efda2987700 1 -- 172.21.15.189:0/337024570 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd9c156860 con 0x7efd9c0cc0e0 2024-04-06T12:12:29.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.731+0000 7efda2987700 1 -- 172.21.15.189:0/337024570 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd9c156e20 con 0x7efd9c0cc0e0 2024-04-06T12:12:29.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.731+0000 7f8293fff700 1 -- 172.21.15.189:0/1326876708 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8294159b30 msgr2=0x7f82940cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.731+0000 7f8293fff700 1 --2- 172.21.15.189:0/1326876708 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8294159b30 0x7f82940cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.731+0000 7f8293fff700 1 -- 172.21.15.189:0/1326876708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f829406df60 msgr2=0x7f82940cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.731+0000 7f8293fff700 1 --2- 172.21.15.189:0/1326876708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f829406df60 0x7f82940cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.731+0000 7f8293fff700 1 -- 172.21.15.189:0/1326876708 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828c00d040 con 0x7f8294150690 2024-04-06T12:12:29.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.731+0000 7f8293fff700 1 --2- 172.21.15.189:0/1326876708 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8294150690 0x7f829414c2b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f828800b450 tx=0x7f828800b760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.732+0000 7f82917fa700 1 -- 172.21.15.189:0/1326876708 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8288011830 con 0x7f8294150690 2024-04-06T12:12:29.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.732+0000 7f82917fa700 1 -- 172.21.15.189:0/1326876708 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8288011990 con 0x7f8294150690 2024-04-06T12:12:29.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.732+0000 7f82917fa700 1 -- 172.21.15.189:0/1326876708 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8288011d40 con 0x7f8294150690 2024-04-06T12:12:29.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.731+0000 7fd2f3fff700 1 -- 172.21.15.189:0/472913489 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f400e070 con 0x7fd2fc072650 2024-04-06T12:12:29.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.731+0000 7fd2f3fff700 1 -- 172.21.15.189:0/472913489 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f400b9c0 con 0x7fd2fc072650 2024-04-06T12:12:29.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.731+0000 7fd2f3fff700 1 -- 172.21.15.189:0/472913489 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f400bd70 con 0x7fd2fc072650 2024-04-06T12:12:29.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.732+0000 7fd3039f6700 1 -- 172.21.15.189:0/472913489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2fc072650 msgr2=0x7fd2fc06da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.732+0000 7fd3039f6700 1 --2- 172.21.15.189:0/472913489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2fc072650 0x7fd2fc06da20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4009e50 tx=0x7fd2f400b330 comp rx=0 tx=0).stop 2024-04-06T12:12:29.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.732+0000 7fd3039f6700 1 -- 172.21.15.189:0/472913489 shutdown_connections 2024-04-06T12:12:29.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.732+0000 7fd3039f6700 1 --2- 172.21.15.189:0/472913489 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2fc06df60 0x7fd2fc16b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.732+0000 7fd3039f6700 1 --2- 172.21.15.189:0/472913489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2fc072650 0x7fd2fc06da20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.732+0000 7fd3039f6700 1 -- 172.21.15.189:0/472913489 >> 172.21.15.189:0/472913489 conn(0x7fd2fc0cefa0 msgr2=0x7fd2fc0cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.731+0000 7f048d7fa700 1 -- 172.21.15.189:0/2703372104 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c011830 con 0x7f04900c2a70 2024-04-06T12:12:29.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.731+0000 7f048d7fa700 1 -- 172.21.15.189:0/2703372104 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047c0119b0 con 0x7f04900c2a70 2024-04-06T12:12:29.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.731+0000 7f048d7fa700 1 -- 172.21.15.189:0/2703372104 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c011d60 con 0x7f04900c2a70 2024-04-06T12:12:29.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.731+0000 7f0496de6700 1 -- 172.21.15.189:0/2703372104 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0490163360 con 0x7f04900c2a70 2024-04-06T12:12:29.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.731+0000 7f0496de6700 1 -- 172.21.15.189:0/2703372104 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0490164360 con 0x7f04900c2a70 2024-04-06T12:12:29.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.733+0000 7f0496de6700 1 -- 172.21.15.189:0/2703372104 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04740052f0 con 0x7f04900c2a70 2024-04-06T12:12:29.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.732+0000 7f829afd7700 1 -- 172.21.15.189:0/1326876708 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82940ce630 con 0x7f8294150690 2024-04-06T12:12:29.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.732+0000 7f829afd7700 1 -- 172.21.15.189:0/1326876708 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82940cebf0 con 0x7f8294150690 2024-04-06T12:12:29.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.733+0000 7f829afd7700 1 -- 172.21.15.189:0/1326876708 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82940cee10 con 0x7f8294150690 2024-04-06T12:12:29.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.734+0000 7f82917fa700 1 -- 172.21.15.189:0/1326876708 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f828801a400 con 0x7f8294150690 2024-04-06T12:12:29.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.734+0000 7f82917fa700 1 --2- 172.21.15.189:0/1326876708 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f827c06aef0 0x7f827c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.734+0000 7f82917fa700 1 -- 172.21.15.189:0/1326876708 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f8288089420 con 0x7f8294150690 2024-04-06T12:12:29.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.735+0000 7efda2987700 1 -- 172.21.15.189:0/337024570 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd9c073270 con 0x7efd9c0cc0e0 2024-04-06T12:12:29.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.732+0000 7fd3039f6700 1 -- 172.21.15.189:0/472913489 shutdown_connections 2024-04-06T12:12:29.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.732+0000 7fd3039f6700 1 -- 172.21.15.189:0/472913489 wait complete. 2024-04-06T12:12:29.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.734+0000 7fd3039f6700 1 Processor -- start 2024-04-06T12:12:29.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.734+0000 7fd3039f6700 1 -- start start 2024-04-06T12:12:29.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.735+0000 7fd3039f6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2fc06df60 0x7fd2fc15a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.735+0000 7fd3039f6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2fc072650 0x7fd2fc15aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.735+0000 7fd3039f6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2fc161a30 0x7fd2fc12e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.735+0000 7fd3039f6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc15b400 con 0x7fd2fc06df60 2024-04-06T12:12:29.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.735+0000 7fd3039f6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc142340 con 0x7fd2fc161a30 2024-04-06T12:12:29.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.735+0000 7fd3039f6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc1586c0 con 0x7fd2fc072650 2024-04-06T12:12:29.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.735+0000 7f8298d73700 1 --2- 172.21.15.189:0/1326876708 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f827c06aef0 0x7f827c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.735+0000 7fd301f93700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2fc161a30 0x7fd2fc12e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.735+0000 7fd301f93700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2fc161a30 0x7fd2fc12e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:58896/0 (socket says 172.21.15.189:58896) 2024-04-06T12:12:29.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.735+0000 7fd301f93700 1 -- 172.21.15.189:0/3018597709 learned_addr learned my addr 172.21.15.189:0/3018597709 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.735+0000 7fd300f91700 1 --2- 172.21.15.189:0/3018597709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2fc072650 0x7fd2fc15aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.735+0000 7fd301792700 1 --2- 172.21.15.189:0/3018597709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2fc06df60 0x7fd2fc15a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.735+0000 7fd301f93700 1 -- 172.21.15.189:0/3018597709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2fc072650 msgr2=0x7fd2fc15aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.735+0000 7fd301f93700 1 --2- 172.21.15.189:0/3018597709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2fc072650 0x7fd2fc15aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.735+0000 7fd301f93700 1 -- 172.21.15.189:0/3018597709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2fc06df60 msgr2=0x7fd2fc15a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.735+0000 7fd301f93700 1 --2- 172.21.15.189:0/3018597709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2fc06df60 0x7fd2fc15a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.736+0000 7fd301f93700 1 -- 172.21.15.189:0/3018597709 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f4009b30 con 0x7fd2fc161a30 2024-04-06T12:12:29.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.736+0000 7fca417fa700 1 -- 172.21.15.189:0/4150698729 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fca440c58b0 con 0x7fca440cb870 2024-04-06T12:12:29.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.736+0000 7fd301f93700 1 --2- 172.21.15.189:0/3018597709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2fc161a30 0x7fd2fc12e2d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00b410 tx=0x7fd2ec00b720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.736+0000 7f8298d73700 1 --2- 172.21.15.189:0/1326876708 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f827c06aef0 0x7f827c06d3b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8284000f60 tx=0x7f8284000ed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.737+0000 7efd98ff9700 1 -- 172.21.15.189:0/337024570 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efd90003b90 con 0x7efd9c0cc0e0 2024-04-06T12:12:29.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.737+0000 7efd98ff9700 1 --2- 172.21.15.189:0/337024570 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efd8806ad60 0x7efd8806d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.737+0000 7efd98ff9700 1 -- 172.21.15.189:0/337024570 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7efd90088ff0 con 0x7efd9c0cc0e0 2024-04-06T12:12:29.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.737+0000 7f048d7fa700 1 -- 172.21.15.189:0/2703372104 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f047c01a400 con 0x7f04900c2a70 2024-04-06T12:12:29.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.737+0000 7f048d7fa700 1 --2- 172.21.15.189:0/2703372104 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f048008ce80 0x7f048008f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.737+0000 7f048d7fa700 1 -- 172.21.15.189:0/2703372104 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f047c089cf0 con 0x7f04900c2a70 2024-04-06T12:12:29.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.738+0000 7efd9b7fe700 1 --2- 172.21.15.189:0/337024570 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efd8806ad60 0x7efd8806d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.738+0000 7efd9b7fe700 1 --2- 172.21.15.189:0/337024570 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efd8806ad60 0x7efd8806d220 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efd8c0097b0 tx=0x7efd8c006d20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.739+0000 7f78dece1700 1 Processor -- start 2024-04-06T12:12:29.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.739+0000 7f82917fa700 1 -- 172.21.15.189:0/1326876708 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8288054380 con 0x7f8294150690 2024-04-06T12:12:29.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.739+0000 7fd2f1ffb700 1 -- 172.21.15.189:0/3018597709 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec011620 con 0x7fd2fc161a30 2024-04-06T12:12:29.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.739+0000 7fd2f1ffb700 1 -- 172.21.15.189:0/3018597709 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ec011780 con 0x7fd2fc161a30 2024-04-06T12:12:29.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.739+0000 7fd2f1ffb700 1 -- 172.21.15.189:0/3018597709 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec011b30 con 0x7fd2fc161a30 2024-04-06T12:12:29.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.739+0000 7fd3039f6700 1 -- 172.21.15.189:0/3018597709 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2fc072390 con 0x7fd2fc161a30 2024-04-06T12:12:29.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.739+0000 7fd3039f6700 1 -- 172.21.15.189:0/3018597709 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2fc158c80 con 0x7fd2fc161a30 2024-04-06T12:12:29.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.739+0000 7f78dece1700 1 -- start start 2024-04-06T12:12:29.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.740+0000 7f78dece1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78d8159b30 0x7f78d806da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.740+0000 7f78dece1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f78d806df60 0x7f78d815b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.740+0000 7f78dece1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d806e3b0 con 0x7f78d8150690 2024-04-06T12:12:29.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.740+0000 7f78dece1700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d8150a70 con 0x7f78d806df60 2024-04-06T12:12:29.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.740+0000 7f78dece1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d8159f90 con 0x7f78d8159b30 2024-04-06T12:12:29.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.740+0000 7f78d7fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78d8159b30 0x7f78d806da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.740+0000 7f78d7fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78d8159b30 0x7f78d806da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59536/0 (socket says 172.21.15.189:59536) 2024-04-06T12:12:29.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.740+0000 7f78d7fff700 1 -- 172.21.15.189:0/2569041751 learned_addr learned my addr 172.21.15.189:0/2569041751 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.741+0000 7f78d7fff700 1 -- 172.21.15.189:0/2569041751 >> v2:172.21.15.189:3300/0 conn(0x7f78d806df60 msgr2=0x7f78d815b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:29.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.741+0000 7f78d7fff700 1 --2- 172.21.15.189:0/2569041751 >> v2:172.21.15.189:3300/0 conn(0x7f78d806df60 0x7f78d815b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.741+0000 7f78d7fff700 1 -- 172.21.15.189:0/2569041751 >> v1:172.21.15.189:6789/0 conn(0x7f78d8150690 legacy=0x7f78d80b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:29.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.741+0000 7f78d7fff700 1 -- 172.21.15.189:0/2569041751 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d815a180 con 0x7f78d8159b30 2024-04-06T12:12:29.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.741+0000 7efd98ff9700 1 -- 172.21.15.189:0/337024570 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efd90053ec0 con 0x7efd9c0cc0e0 2024-04-06T12:12:29.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.742+0000 7fd2f1ffb700 1 -- 172.21.15.189:0/3018597709 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd2ec011c90 con 0x7fd2fc161a30 2024-04-06T12:12:29.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.742+0000 7fd2f1ffb700 1 --2- 172.21.15.189:0/3018597709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd2e006ace0 0x7fd2e006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.743+0000 7fd2f1ffb700 1 -- 172.21.15.189:0/3018597709 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd2ec0893e0 con 0x7fd2fc161a30 2024-04-06T12:12:29.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.743+0000 7fd301792700 1 --2- 172.21.15.189:0/3018597709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd2e006ace0 0x7fd2e006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.743+0000 7fd3039f6700 1 -- 172.21.15.189:0/3018597709 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2dc0052f0 con 0x7fd2fc161a30 2024-04-06T12:12:29.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.743+0000 7fd301792700 1 --2- 172.21.15.189:0/3018597709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd2e006ace0 0x7fd2e006d1a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0cb700 tx=0x7fd2f8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.744+0000 7f78d7fff700 1 --2- 172.21.15.189:0/2569041751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78d8159b30 0x7f78d806da20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f78d0009e50 tx=0x7f78d000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=480326eaedd4d165 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.746+0000 7f0494b82700 1 --2- 172.21.15.189:0/2703372104 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f048008ce80 0x7f048008f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.746+0000 7f048d7fa700 1 -- 172.21.15.189:0/2703372104 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f047c054d00 con 0x7f04900c2a70 2024-04-06T12:12:29.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.746+0000 7f0494b82700 1 --2- 172.21.15.189:0/2703372104 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f048008ce80 0x7f048008f340 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0484009c40 tx=0x7f0484008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.745+0000 7f78d77fe700 1 -- 172.21.15.189:0/2569041751 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d0003c20 con 0x7f78d8159b30 2024-04-06T12:12:29.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.746+0000 7f78d77fe700 1 -- 172.21.15.189:0/2569041751 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d0003d80 con 0x7f78d8159b30 2024-04-06T12:12:29.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.746+0000 7f78d77fe700 1 -- 172.21.15.189:0/2569041751 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d000bbc0 con 0x7f78d8159b30 2024-04-06T12:12:29.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.746+0000 7f78dece1700 1 -- 172.21.15.189:0/2569041751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78d8159b30 msgr2=0x7f78d806da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.746+0000 7f78dece1700 1 --2- 172.21.15.189:0/2569041751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78d8159b30 0x7f78d806da20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f78d0009e50 tx=0x7f78d000b330 comp rx=0 tx=0).stop 2024-04-06T12:12:29.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.757+0000 7fd2f1ffb700 1 -- 172.21.15.189:0/3018597709 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd2ec054340 con 0x7fd2fc161a30 2024-04-06T12:12:29.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.762+0000 7f78dece1700 1 -- 172.21.15.189:0/2569041751 shutdown_connections 2024-04-06T12:12:29.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.762+0000 7f78dece1700 1 --2- 172.21.15.189:0/2569041751 >> v2:172.21.15.189:3300/0 conn(0x7f78d806df60 0x7f78d815b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.762+0000 7f78dece1700 1 --2- 172.21.15.189:0/2569041751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78d8159b30 0x7f78d806da20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.762+0000 7f78dece1700 1 -- 172.21.15.189:0/2569041751 >> 172.21.15.189:0/2569041751 conn(0x7f78d80cefa0 msgr2=0x7f78d80cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.763+0000 7f78dece1700 1 -- 172.21.15.189:0/2569041751 shutdown_connections 2024-04-06T12:12:29.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.763+0000 7f78dece1700 1 -- 172.21.15.189:0/2569041751 wait complete. 2024-04-06T12:12:29.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.765+0000 7f78dece1700 1 Processor -- start 2024-04-06T12:12:29.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.766+0000 7f78dece1700 1 -- start start 2024-04-06T12:12:29.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.767+0000 7f78dece1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78d806df60 0x7f78d814f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.767+0000 7f78dece1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78d8150690 0x7f78d8146c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.767+0000 7f78dece1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78d8159b30 0x7f78d8154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.767+0000 7f78dece1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d80cea30 con 0x7f78d8159b30 2024-04-06T12:12:29.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.767+0000 7f78dece1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d81471c0 con 0x7f78d8150690 2024-04-06T12:12:29.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.767+0000 7f78dece1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d80cde90 con 0x7f78d806df60 2024-04-06T12:12:29.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.768+0000 7f78dd27e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78d8159b30 0x7f78d8154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.768+0000 7f78dd27e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78d8159b30 0x7f78d8154120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38174/0 (socket says 172.21.15.189:38174) 2024-04-06T12:12:29.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.768+0000 7f78dd27e700 1 -- 172.21.15.189:0/3142231133 learned_addr learned my addr 172.21.15.189:0/3142231133 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.769+0000 7f78dd27e700 1 -- 172.21.15.189:0/3142231133 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78d806df60 msgr2=0x7f78d814f4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-06T12:12:29.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.769+0000 7f78dd27e700 1 --2- 172.21.15.189:0/3142231133 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78d806df60 0x7f78d814f4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.769+0000 7f78dd27e700 1 -- 172.21.15.189:0/3142231133 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78d8150690 msgr2=0x7f78d8146c80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-06T12:12:29.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.769+0000 7f78dd27e700 1 --2- 172.21.15.189:0/3142231133 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78d8150690 0x7f78d8146c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.769+0000 7f78dd27e700 1 -- 172.21.15.189:0/3142231133 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d0009b30 con 0x7f78d8159b30 2024-04-06T12:12:29.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.769+0000 7f78dd27e700 1 --2- 172.21.15.189:0/3142231133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78d8159b30 0x7f78d8154120 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f78c800b550 tx=0x7f78c800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.770+0000 7f78d57fa700 1 -- 172.21.15.189:0/3142231133 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c8011830 con 0x7f78d8159b30 2024-04-06T12:12:29.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.770+0000 7f78d57fa700 1 -- 172.21.15.189:0/3142231133 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c80119b0 con 0x7f78d8159b30 2024-04-06T12:12:29.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.770+0000 7f78d57fa700 1 -- 172.21.15.189:0/3142231133 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c8011d60 con 0x7f78d8159b30 2024-04-06T12:12:29.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.770+0000 7f78dece1700 1 -- 172.21.15.189:0/3142231133 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d80ce100 con 0x7f78d8159b30 2024-04-06T12:12:29.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.770+0000 7f78dece1700 1 -- 172.21.15.189:0/3142231133 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d80ce6c0 con 0x7f78d8159b30 2024-04-06T12:12:29.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.773+0000 7f78d57fa700 1 -- 172.21.15.189:0/3142231133 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f78c801a400 con 0x7f78d8159b30 2024-04-06T12:12:29.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.774+0000 7f78d57fa700 1 --2- 172.21.15.189:0/3142231133 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f78c406ad60 0x7f78c406d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.774+0000 7f78d57fa700 1 -- 172.21.15.189:0/3142231133 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f78c8089670 con 0x7f78d8159b30 2024-04-06T12:12:29.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.778+0000 7f78dece1700 1 -- 172.21.15.189:0/3142231133 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d8074990 con 0x7f78d8159b30 2024-04-06T12:12:29.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.783+0000 7f78dca7d700 1 --2- 172.21.15.189:0/3142231133 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f78c406ad60 0x7f78c406d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.786+0000 7f78dca7d700 1 --2- 172.21.15.189:0/3142231133 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f78c406ad60 0x7f78c406d220 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f78d80cee70 tx=0x7f78cc007a40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.787+0000 7f56cb2f5700 1 Processor -- start 2024-04-06T12:12:29.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.788+0000 7f78d57fa700 1 -- 172.21.15.189:0/3142231133 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f78c80545f0 con 0x7f78d8159b30 2024-04-06T12:12:29.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.787+0000 7f56cb2f5700 1 -- start start 2024-04-06T12:12:29.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.789+0000 7f56cb2f5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f56c406e130 0x7f56c4164280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.789+0000 7f56cb2f5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c415a180 con 0x7f56c4150690 2024-04-06T12:12:29.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.789+0000 7f56cb2f5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c415a460 con 0x7f56c4159b30 2024-04-06T12:12:29.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.789+0000 7f56cb2f5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c4150a70 con 0x7f56c406e130 2024-04-06T12:12:29.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.790+0000 7f56c8890700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f56c4159b30 0x7f56c406da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:52478/0 (socket says 172.21.15.189:52478) 2024-04-06T12:12:29.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.790+0000 7f56c8890700 1 -- 172.21.15.189:0/2677294206 learned_addr learned my addr 172.21.15.189:0/2677294206 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:29.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.797+0000 7f56c9892700 1 --2- 172.21.15.189:0/2677294206 >> v2:172.21.15.189:3300/0 conn(0x7f56c406e130 0x7f56c4164280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.797+0000 7f56c9892700 1 -- 172.21.15.189:0/2677294206 >> v1:172.21.15.192:6789/0 conn(0x7f56c4159b30 legacy=0x7f56c406da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.797+0000 7f56c9892700 1 -- 172.21.15.189:0/2677294206 >> v1:172.21.15.189:6789/0 conn(0x7f56c4150690 legacy=0x7f56c40b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.797+0000 7f56c9892700 1 -- 172.21.15.189:0/2677294206 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c415a740 con 0x7f56c406e130 2024-04-06T12:12:29.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.798+0000 7f56c9892700 1 --2- 172.21.15.189:0/2677294206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c406e130 0x7f56c4164280 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f56bc00aea0 tx=0x7f56bc00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f8be075724bc6f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.800+0000 7f56bbfff700 1 -- 172.21.15.189:0/2677294206 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc003c20 con 0x7f56c406e130 2024-04-06T12:12:29.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.801+0000 7f56bbfff700 1 -- 172.21.15.189:0/2677294206 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56bc003d80 con 0x7f56c406e130 2024-04-06T12:12:29.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.801+0000 7f56bbfff700 1 -- 172.21.15.189:0/2677294206 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc00cae0 con 0x7f56c406e130 2024-04-06T12:12:29.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.801+0000 7f56cb2f5700 1 -- 172.21.15.189:0/2677294206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c406e130 msgr2=0x7f56c4164280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.801+0000 7f56cb2f5700 1 --2- 172.21.15.189:0/2677294206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c406e130 0x7f56c4164280 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f56bc00aea0 tx=0x7f56bc00c330 comp rx=0 tx=0).stop 2024-04-06T12:12:29.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.802+0000 7f56cb2f5700 1 -- 172.21.15.189:0/2677294206 shutdown_connections 2024-04-06T12:12:29.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.802+0000 7f56cb2f5700 1 --2- 172.21.15.189:0/2677294206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c406e130 0x7f56c4164280 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.802+0000 7f56cb2f5700 1 -- 172.21.15.189:0/2677294206 >> 172.21.15.189:0/2677294206 conn(0x7f56c40cefa0 msgr2=0x7f56c40cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:29.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.802+0000 7f56cb2f5700 1 -- 172.21.15.189:0/2677294206 shutdown_connections 2024-04-06T12:12:29.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.802+0000 7f56cb2f5700 1 -- 172.21.15.189:0/2677294206 wait complete. 2024-04-06T12:12:29.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.804+0000 7f56cb2f5700 1 Processor -- start 2024-04-06T12:12:29.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.805+0000 7f56cb2f5700 1 -- start start 2024-04-06T12:12:29.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.806+0000 7f56cb2f5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c406e130 0x7f56c414ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.806+0000 7f56cb2f5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56c4150690 0x7f56c414caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.806+0000 7f56cb2f5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56c4159b30 0x7f56c41537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.806+0000 7f56cb2f5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c41501f0 con 0x7f56c406e130 2024-04-06T12:12:29.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.806+0000 7f56cb2f5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c414f460 con 0x7f56c4150690 2024-04-06T12:12:29.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.806+0000 7f56cb2f5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c414cfe0 con 0x7f56c4159b30 2024-04-06T12:12:29.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.806+0000 7f56c9091700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c406e130 0x7f56c414ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.807+0000 7f56c9091700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c406e130 0x7f56c414ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38192/0 (socket says 172.21.15.189:38192) 2024-04-06T12:12:29.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.807+0000 7f56c8890700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56c4150690 0x7f56c414caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.807+0000 7f56c9091700 1 -- 172.21.15.189:0/1769241171 learned_addr learned my addr 172.21.15.189:0/1769241171 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:29.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.807+0000 7f56c9892700 1 --2- 172.21.15.189:0/1769241171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56c4159b30 0x7f56c41537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.807+0000 7f56c9091700 1 -- 172.21.15.189:0/1769241171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56c4159b30 msgr2=0x7f56c41537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.807+0000 7f56c9091700 1 --2- 172.21.15.189:0/1769241171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56c4159b30 0x7f56c41537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.807+0000 7f56c9091700 1 -- 172.21.15.189:0/1769241171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56c4150690 msgr2=0x7f56c414caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:29.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.807+0000 7f56c9091700 1 --2- 172.21.15.189:0/1769241171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56c4150690 0x7f56c414caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:29.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.807+0000 7f56c9091700 1 -- 172.21.15.189:0/1769241171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56bc00ab80 con 0x7f56c406e130 2024-04-06T12:12:29.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.808+0000 7f56c9091700 1 --2- 172.21.15.189:0/1769241171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c406e130 0x7f56c414ef20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f56b400c700 tx=0x7f56b400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:29.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.808+0000 7f56b9ffb700 1 -- 172.21.15.189:0/1769241171 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b40054e0 con 0x7f56c406e130 2024-04-06T12:12:29.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.808+0000 7f56b9ffb700 1 -- 172.21.15.189:0/1769241171 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b4003680 con 0x7f56c406e130 2024-04-06T12:12:29.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.809+0000 7f56b9ffb700 1 -- 172.21.15.189:0/1769241171 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b4003a30 con 0x7f56c406e130 2024-04-06T12:12:29.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.809+0000 7f56cb2f5700 1 -- 172.21.15.189:0/1769241171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c414f670 con 0x7f56c406e130 2024-04-06T12:12:29.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.809+0000 7f56cb2f5700 1 -- 172.21.15.189:0/1769241171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c414fc30 con 0x7f56c406e130 2024-04-06T12:12:29.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.813+0000 7f56b9ffb700 1 -- 172.21.15.189:0/1769241171 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f56b4003b90 con 0x7f56c406e130 2024-04-06T12:12:29.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.813+0000 7f56b9ffb700 1 --2- 172.21.15.189:0/1769241171 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f56b006af00 0x7f56b006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:29.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.814+0000 7f56b9ffb700 1 -- 172.21.15.189:0/1769241171 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f56b4088af0 con 0x7f56c406e130 2024-04-06T12:12:29.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.814+0000 7f56cb2f5700 1 -- 172.21.15.189:0/1769241171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a80052f0 con 0x7f56c406e130 2024-04-06T12:12:29.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.818+0000 7f56c8890700 1 --2- 172.21.15.189:0/1769241171 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f56b006af00 0x7f56b006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:29.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.825+0000 7f56b9ffb700 1 -- 172.21.15.189:0/1769241171 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f56b4053a50 con 0x7f56c406e130 2024-04-06T12:12:29.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:29.825+0000 7f56c8890700 1 --2- 172.21.15.189:0/1769241171 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f56b006af00 0x7f56b006d3c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f56c0001280 tx=0x7f56c0001210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:30.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.125+0000 7fd4f58d4700 1 -- 172.21.15.189:0/30185214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd4d4005870 con 0x7fd4f00d0890 2024-04-06T12:12:30.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.129+0000 7fd4ecff9700 1 -- 172.21.15.189:0/30185214 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd4e0059a80 con 0x7fd4f00d0890 2024-04-06T12:12:30.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.129+0000 7f829afd7700 1 -- 172.21.15.189:0/1326876708 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f8294074990 con 0x7f8294150690 2024-04-06T12:12:30.128 INFO:teuthology.orchestra.run.smithi189.stdout:55834574855 2024-04-06T12:12:30.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.130+0000 7f82917fa700 1 -- 172.21.15.189:0/1326876708 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f828800be60 con 0x7f8294150690 2024-04-06T12:12:30.131 INFO:teuthology.orchestra.run.smithi189.stdout:55834574857 2024-04-06T12:12:30.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.134+0000 7fd4f58d4700 1 -- 172.21.15.189:0/30185214 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd4d006ad60 msgr2=0x7fd4d006d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:30.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.134+0000 7fd4f58d4700 1 --2- 172.21.15.189:0/30185214 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd4d006ad60 0x7fd4d006d220 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd4f015ae40 tx=0x7fd4d8008040 comp rx=0 tx=0).stop 2024-04-06T12:12:30.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.134+0000 7fd4f58d4700 1 -- 172.21.15.189:0/30185214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4f00d0890 msgr2=0x7fd4f015a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:30.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.134+0000 7fd4f58d4700 1 --2- 172.21.15.189:0/30185214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4f00d0890 0x7fd4f015a900 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0009c40 tx=0x7fd4e0004750 comp rx=0 tx=0).stop 2024-04-06T12:12:30.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.134+0000 7f829afd7700 1 -- 172.21.15.189:0/1326876708 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f827c06aef0 msgr2=0x7f827c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:30.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.134+0000 7f829afd7700 1 --2- 172.21.15.189:0/1326876708 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f827c06aef0 0x7f827c06d3b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8284000f60 tx=0x7f8284000ed0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.134+0000 7f829afd7700 1 -- 172.21.15.189:0/1326876708 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8294150690 msgr2=0x7f829414c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.134+0000 7f829afd7700 1 --2- 172.21.15.189:0/1326876708 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8294150690 0x7f829414c2b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f828800b450 tx=0x7f828800b760 comp rx=0 tx=0).stop 2024-04-06T12:12:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.134+0000 7fd4f58d4700 1 -- 172.21.15.189:0/30185214 shutdown_connections 2024-04-06T12:12:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.134+0000 7fd4f58d4700 1 --2- 172.21.15.189:0/30185214 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd4d006ad60 0x7fd4d006d220 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd4f015ae40 tx=0x7fd4d8008040 comp rx=0 tx=0).stop 2024-04-06T12:12:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.134+0000 7fd4f58d4700 1 --2- 172.21.15.189:0/30185214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4f015d7e0 0x7fd4f00cbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.134+0000 7fd4f58d4700 1 --2- 172.21.15.189:0/30185214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4f00d0890 0x7fd4f015a900 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.134+0000 7fd4f58d4700 1 --2- 172.21.15.189:0/30185214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4f00d1af0 0x7fd4f01579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.134+0000 7fd4f58d4700 1 -- 172.21.15.189:0/30185214 >> 172.21.15.189:0/30185214 conn(0x7fd4f0072250 msgr2=0x7fd4f015f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:30.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.135+0000 7fd4f58d4700 1 -- 172.21.15.189:0/30185214 shutdown_connections 2024-04-06T12:12:30.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.135+0000 7fd4f58d4700 1 -- 172.21.15.189:0/30185214 wait complete. 2024-04-06T12:12:30.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.135+0000 7f829afd7700 1 -- 172.21.15.189:0/1326876708 shutdown_connections 2024-04-06T12:12:30.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.135+0000 7f829afd7700 1 --2- 172.21.15.189:0/1326876708 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8294159b30 0x7f82940cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.135+0000 7f829afd7700 1 --2- 172.21.15.189:0/1326876708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f829406df60 0x7f82940cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.135+0000 7f829afd7700 1 --2- 172.21.15.189:0/1326876708 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f827c06aef0 0x7f827c06d3b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.135+0000 7f829afd7700 1 --2- 172.21.15.189:0/1326876708 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8294150690 0x7f829414c2b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.135+0000 7f829afd7700 1 -- 172.21.15.189:0/1326876708 >> 172.21.15.189:0/1326876708 conn(0x7f82940cefa0 msgr2=0x7f8294163a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:30.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.136+0000 7f829afd7700 1 -- 172.21.15.189:0/1326876708 shutdown_connections 2024-04-06T12:12:30.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.136+0000 7f829afd7700 1 -- 172.21.15.189:0/1326876708 wait complete. 2024-04-06T12:12:30.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.146+0000 7fca4ad2c700 1 -- 172.21.15.189:0/4150698729 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fca440c58b0 con 0x7fca440cb870 2024-04-06T12:12:30.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.148+0000 7fca417fa700 1 -- 172.21.15.189:0/4150698729 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fca440c58b0 con 0x7fca440cb870 2024-04-06T12:12:30.149 INFO:teuthology.orchestra.run.smithi189.stdout:55834574856 2024-04-06T12:12:30.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.153+0000 7efda2987700 1 -- 172.21.15.189:0/337024570 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7efd9c13ff70 con 0x7efd9c0cc0e0 2024-04-06T12:12:30.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.153+0000 7f0496de6700 1 -- 172.21.15.189:0/2703372104 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f0474005870 con 0x7f04900c2a70 2024-04-06T12:12:30.153 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.0 2024-04-06T12:12:30.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.155+0000 7fca4ad2c700 1 -- 172.21.15.189:0/4150698729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fca2806ace0 msgr2=0x7fca2806d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:30.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.155+0000 7fca4ad2c700 1 --2- 172.21.15.189:0/4150698729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fca2806ace0 0x7fca2806d1a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fca440d21c0 tx=0x7fca380090e0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.155+0000 7fca4ad2c700 1 -- 172.21.15.189:0/4150698729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fca440cb870 msgr2=0x7fca44144f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:30.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.155+0000 7fca4ad2c700 1 --2- 172.21.15.189:0/4150698729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fca440cb870 0x7fca44144f80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fca30009130 tx=0x7fca30004750 comp rx=0 tx=0).stop 2024-04-06T12:12:30.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.155+0000 7efd98ff9700 1 -- 172.21.15.189:0/337024570 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efd90017090 con 0x7efd9c0cc0e0 2024-04-06T12:12:30.154 INFO:teuthology.orchestra.run.smithi189.stdout:55834574856 2024-04-06T12:12:30.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.156+0000 7fca4ad2c700 1 -- 172.21.15.189:0/4150698729 shutdown_connections 2024-04-06T12:12:30.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.156+0000 7fca4ad2c700 1 --2- 172.21.15.189:0/4150698729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fca2806ace0 0x7fca2806d1a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.156+0000 7fca4ad2c700 1 --2- 172.21.15.189:0/4150698729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca440d11b0 0x7fca4414d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.156+0000 7fca4ad2c700 1 --2- 172.21.15.189:0/4150698729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fca440cb870 0x7fca44144f80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.156+0000 7fca4ad2c700 1 --2- 172.21.15.189:0/4150698729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca440cfa00 0x7fca4414cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.156+0000 7fca4ad2c700 1 -- 172.21.15.189:0/4150698729 >> 172.21.15.189:0/4150698729 conn(0x7fca440b2db0 msgr2=0x7fca4406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:30.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.156+0000 7fca4ad2c700 1 -- 172.21.15.189:0/4150698729 shutdown_connections 2024-04-06T12:12:30.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.156+0000 7fca4ad2c700 1 -- 172.21.15.189:0/4150698729 wait complete. 2024-04-06T12:12:30.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.156+0000 7f048d7fa700 1 -- 172.21.15.189:0/2703372104 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f047c015090 con 0x7f04900c2a70 2024-04-06T12:12:30.156 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.2 2024-04-06T12:12:30.156 INFO:teuthology.orchestra.run.smithi189.stdout:55834574856 2024-04-06T12:12:30.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.161+0000 7efda2987700 1 -- 172.21.15.189:0/337024570 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efd8806ad60 msgr2=0x7efd8806d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:30.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.161+0000 7efda2987700 1 --2- 172.21.15.189:0/337024570 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efd8806ad60 0x7efd8806d220 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efd8c0097b0 tx=0x7efd8c006d20 comp rx=0 tx=0).stop 2024-04-06T12:12:30.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.161+0000 7efda2987700 1 -- 172.21.15.189:0/337024570 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd9c0cc0e0 msgr2=0x7efd9c152c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:30.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.161+0000 7efda2987700 1 --2- 172.21.15.189:0/337024570 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd9c0cc0e0 0x7efd9c152c40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efd9000c4d0 tx=0x7efd9000c890 comp rx=0 tx=0).stop 2024-04-06T12:12:30.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.161+0000 7efda2987700 1 -- 172.21.15.189:0/337024570 shutdown_connections 2024-04-06T12:12:30.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.161+0000 7efda2987700 1 --2- 172.21.15.189:0/337024570 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efd8806ad60 0x7efd8806d220 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.161+0000 7efda2987700 1 --2- 172.21.15.189:0/337024570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd9c15bc70 0x7efd9c154350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.161+0000 7efda2987700 1 --2- 172.21.15.189:0/337024570 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd9c0cc0e0 0x7efd9c152c40 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efd9000c4d0 tx=0x7efd9000c890 comp rx=0 tx=0).stop 2024-04-06T12:12:30.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.161+0000 7efda2987700 1 --2- 172.21.15.189:0/337024570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd9c13e340 0x7efd9c153e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.161+0000 7efda2987700 1 -- 172.21.15.189:0/337024570 >> 172.21.15.189:0/337024570 conn(0x7efd9c06a730 msgr2=0x7efd9c1341a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:30.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.161+0000 7f0496de6700 1 -- 172.21.15.189:0/2703372104 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f048008ce80 msgr2=0x7f048008f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:30.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.161+0000 7f0496de6700 1 --2- 172.21.15.189:0/2703372104 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f048008ce80 0x7f048008f340 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0484009c40 tx=0x7f0484008030 comp rx=0 tx=0).stop 2024-04-06T12:12:30.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.161+0000 7f0496de6700 1 -- 172.21.15.189:0/2703372104 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04900c2a70 msgr2=0x7f0490162910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:30.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.161+0000 7f0496de6700 1 --2- 172.21.15.189:0/2703372104 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04900c2a70 0x7f0490162910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f047c00b550 tx=0x7f047c00b910 comp rx=0 tx=0).stop 2024-04-06T12:12:30.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.162+0000 7f0496de6700 1 -- 172.21.15.189:0/2703372104 shutdown_connections 2024-04-06T12:12:30.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.162+0000 7f0496de6700 1 --2- 172.21.15.189:0/2703372104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0490161980 0x7f04901594b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.162+0000 7f0496de6700 1 --2- 172.21.15.189:0/2703372104 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f048008ce80 0x7f048008f340 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.162+0000 7f0496de6700 1 --2- 172.21.15.189:0/2703372104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f049006b050 0x7f049015afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.162+0000 7f0496de6700 1 --2- 172.21.15.189:0/2703372104 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04900c2a70 0x7f0490162910 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.162+0000 7f0496de6700 1 -- 172.21.15.189:0/2703372104 >> 172.21.15.189:0/2703372104 conn(0x7f04900cc290 msgr2=0x7f0490070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:30.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.162+0000 7f0496de6700 1 -- 172.21.15.189:0/2703372104 shutdown_connections 2024-04-06T12:12:30.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.162+0000 7f0496de6700 1 -- 172.21.15.189:0/2703372104 wait complete. 2024-04-06T12:12:30.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.162+0000 7efda2987700 1 -- 172.21.15.189:0/337024570 shutdown_connections 2024-04-06T12:12:30.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.162+0000 7efda2987700 1 -- 172.21.15.189:0/337024570 wait complete. 2024-04-06T12:12:30.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.164+0000 7fd3039f6700 1 -- 172.21.15.189:0/3018597709 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd2dc0055a0 con 0x7fd2fc161a30 2024-04-06T12:12:30.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.165+0000 7fd2f1ffb700 1 -- 172.21.15.189:0/3018597709 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd2ec015070 con 0x7fd2fc161a30 2024-04-06T12:12:30.163 INFO:teuthology.orchestra.run.smithi189.stdout:55834574855 2024-04-06T12:12:30.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.167+0000 7fd3039f6700 1 -- 172.21.15.189:0/3018597709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd2e006ace0 msgr2=0x7fd2e006d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:30.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.167+0000 7fd3039f6700 1 --2- 172.21.15.189:0/3018597709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd2e006ace0 0x7fd2e006d1a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0cb700 tx=0x7fd2f8009040 comp rx=0 tx=0).stop 2024-04-06T12:12:30.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.167+0000 7fd3039f6700 1 -- 172.21.15.189:0/3018597709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2fc161a30 msgr2=0x7fd2fc12e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:30.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.167+0000 7fd3039f6700 1 --2- 172.21.15.189:0/3018597709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2fc161a30 0x7fd2fc12e2d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00b410 tx=0x7fd2ec00b720 comp rx=0 tx=0).stop 2024-04-06T12:12:30.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.167+0000 7fd3039f6700 1 -- 172.21.15.189:0/3018597709 shutdown_connections 2024-04-06T12:12:30.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.167+0000 7fd3039f6700 1 --2- 172.21.15.189:0/3018597709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd2e006ace0 0x7fd2e006d1a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.167+0000 7fd3039f6700 1 --2- 172.21.15.189:0/3018597709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2fc161a30 0x7fd2fc12e2d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.167+0000 7fd3039f6700 1 --2- 172.21.15.189:0/3018597709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2fc06df60 0x7fd2fc15a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.167+0000 7fd3039f6700 1 --2- 172.21.15.189:0/3018597709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2fc072650 0x7fd2fc15aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.167+0000 7fd3039f6700 1 -- 172.21.15.189:0/3018597709 >> 172.21.15.189:0/3018597709 conn(0x7fd2fc0cefa0 msgr2=0x7fd2fc0b0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:30.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.167+0000 7fd3039f6700 1 -- 172.21.15.189:0/3018597709 shutdown_connections 2024-04-06T12:12:30.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.168+0000 7fd3039f6700 1 -- 172.21.15.189:0/3018597709 wait complete. 2024-04-06T12:12:30.176 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.5 2024-04-06T12:12:30.179 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.4 2024-04-06T12:12:30.183 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.3 2024-04-06T12:12:30.185 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.1 2024-04-06T12:12:30.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.190+0000 7f78dece1700 1 -- 172.21.15.189:0/3142231133 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f78d8148d90 con 0x7f78d8159b30 2024-04-06T12:12:30.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.191+0000 7f78d57fa700 1 -- 172.21.15.189:0/3142231133 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f78c8015070 con 0x7f78d8159b30 2024-04-06T12:12:30.189 INFO:teuthology.orchestra.run.smithi189.stdout:55834574856 2024-04-06T12:12:30.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.193+0000 7f78dece1700 1 -- 172.21.15.189:0/3142231133 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f78c406ad60 msgr2=0x7f78c406d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:30.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.193+0000 7f78dece1700 1 --2- 172.21.15.189:0/3142231133 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f78c406ad60 0x7f78c406d220 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f78d80cee70 tx=0x7f78cc007a40 comp rx=0 tx=0).stop 2024-04-06T12:12:30.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.193+0000 7f78dece1700 1 -- 172.21.15.189:0/3142231133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78d8159b30 msgr2=0x7f78d8154120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:30.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.193+0000 7f78dece1700 1 --2- 172.21.15.189:0/3142231133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78d8159b30 0x7f78d8154120 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f78c800b550 tx=0x7f78c800b910 comp rx=0 tx=0).stop 2024-04-06T12:12:30.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.193+0000 7f78dece1700 1 -- 172.21.15.189:0/3142231133 shutdown_connections 2024-04-06T12:12:30.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.193+0000 7f78dece1700 1 --2- 172.21.15.189:0/3142231133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78d8159b30 0x7f78d8154120 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.193+0000 7f78dece1700 1 --2- 172.21.15.189:0/3142231133 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78d806df60 0x7f78d814f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.193+0000 7f78dece1700 1 --2- 172.21.15.189:0/3142231133 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f78c406ad60 0x7f78c406d220 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.193+0000 7f78dece1700 1 --2- 172.21.15.189:0/3142231133 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78d8150690 0x7f78d8146c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.193+0000 7f78dece1700 1 -- 172.21.15.189:0/3142231133 >> 172.21.15.189:0/3142231133 conn(0x7f78d80cefa0 msgr2=0x7f78d814fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:30.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.193+0000 7f78dece1700 1 -- 172.21.15.189:0/3142231133 shutdown_connections 2024-04-06T12:12:30.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.193+0000 7f78dece1700 1 -- 172.21.15.189:0/3142231133 wait complete. 2024-04-06T12:12:30.203 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.6 2024-04-06T12:12:30.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.207+0000 7f56cb2f5700 1 -- 172.21.15.189:0/1769241171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f56a40051a0 con 0x7f56c406e130 2024-04-06T12:12:30.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.207+0000 7f56b9ffb700 1 -- 172.21.15.189:0/1769241171 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f56b4017070 con 0x7f56c406e130 2024-04-06T12:12:30.205 INFO:teuthology.orchestra.run.smithi189.stdout:55834574857 2024-04-06T12:12:30.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.209+0000 7f56cb2f5700 1 -- 172.21.15.189:0/1769241171 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f56b006af00 msgr2=0x7f56b006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:30.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.209+0000 7f56cb2f5700 1 --2- 172.21.15.189:0/1769241171 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f56b006af00 0x7f56b006d3c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f56c0001280 tx=0x7f56c0001210 comp rx=0 tx=0).stop 2024-04-06T12:12:30.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.209+0000 7f56cb2f5700 1 -- 172.21.15.189:0/1769241171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c406e130 msgr2=0x7f56c414ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:30.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.209+0000 7f56cb2f5700 1 --2- 172.21.15.189:0/1769241171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c406e130 0x7f56c414ef20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f56b400c700 tx=0x7f56b400ca10 comp rx=0 tx=0).stop 2024-04-06T12:12:30.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.209+0000 7f56cb2f5700 1 -- 172.21.15.189:0/1769241171 shutdown_connections 2024-04-06T12:12:30.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.209+0000 7f56cb2f5700 1 --2- 172.21.15.189:0/1769241171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56c4159b30 0x7f56c41537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.209+0000 7f56cb2f5700 1 --2- 172.21.15.189:0/1769241171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c406e130 0x7f56c414ef20 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.209+0000 7f56cb2f5700 1 --2- 172.21.15.189:0/1769241171 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f56b006af00 0x7f56b006d3c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.209+0000 7f56cb2f5700 1 --2- 172.21.15.189:0/1769241171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56c4150690 0x7f56c414caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:30.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.209+0000 7f56cb2f5700 1 -- 172.21.15.189:0/1769241171 >> 172.21.15.189:0/1769241171 conn(0x7f56c40cefa0 msgr2=0x7f56c40739d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:30.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.209+0000 7f56cb2f5700 1 -- 172.21.15.189:0/1769241171 shutdown_connections 2024-04-06T12:12:30.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:30.209+0000 7f56cb2f5700 1 -- 172.21.15.189:0/1769241171 wait complete. 2024-04-06T12:12:30.219 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.7 2024-04-06T12:12:31.154 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-06T12:12:31.157 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-06T12:12:31.177 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-06T12:12:31.180 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-06T12:12:31.184 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-06T12:12:31.186 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-06T12:12:31.203 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-06T12:12:31.220 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-06T12:12:31.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.358+0000 7fed43506700 1 Processor -- start 2024-04-06T12:12:31.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.358+0000 7fed43506700 1 -- start start 2024-04-06T12:12:31.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.359+0000 7fed43506700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fed3c150690 0x7fed3c159b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.359+0000 7fed43506700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed3c0b2300 0x7fed3c06da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.359+0000 7fed43506700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3c06e390 con 0x7fed3c06df60 2024-04-06T12:12:31.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.359+0000 7fed43506700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed3c150a70 con 0x7fed3c150690 2024-04-06T12:12:31.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.359+0000 7fed43506700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c15a180 con 0x7fed3c0b2300 2024-04-06T12:12:31.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.360+0000 7fed40aa1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed3c0b2300 0x7fed3c06da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.360+0000 7fed40aa1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed3c0b2300 0x7fed3c06da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59576/0 (socket says 172.21.15.189:59576) 2024-04-06T12:12:31.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.360+0000 7fed40aa1700 1 -- 172.21.15.189:0/1126015045 learned_addr learned my addr 172.21.15.189:0/1126015045 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:31.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.360+0000 7fed40aa1700 1 -- 172.21.15.189:0/1126015045 >> v2:172.21.15.192:3300/0 conn(0x7fed3c150690 msgr2=0x7fed3c159b30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-06T12:12:31.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.360+0000 7fed40aa1700 1 --2- 172.21.15.189:0/1126015045 >> v2:172.21.15.192:3300/0 conn(0x7fed3c150690 0x7fed3c159b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.360+0000 7fed40aa1700 1 -- 172.21.15.189:0/1126015045 >> v1:172.21.15.192:6789/0 conn(0x7fed3c06df60 legacy=0x7fed3c15d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.360+0000 7fed40aa1700 1 -- 172.21.15.189:0/1126015045 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3c15a460 con 0x7fed3c0b2300 2024-04-06T12:12:31.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.361+0000 7fed40aa1700 1 --2- 172.21.15.189:0/1126015045 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed3c0b2300 0x7fed3c06da20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fed2c0092e0 tx=0x7fed2c0095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d52b3444ba3fd618 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.361+0000 7fed33fff700 1 -- 172.21.15.189:0/1126015045 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2c00c9b0 con 0x7fed3c0b2300 2024-04-06T12:12:31.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.361+0000 7fed33fff700 1 -- 172.21.15.189:0/1126015045 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2c00cb10 con 0x7fed3c0b2300 2024-04-06T12:12:31.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.361+0000 7fed33fff700 1 -- 172.21.15.189:0/1126015045 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2c00cea0 con 0x7fed3c0b2300 2024-04-06T12:12:31.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.362+0000 7fed43506700 1 -- 172.21.15.189:0/1126015045 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed3c0b2300 msgr2=0x7fed3c06da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.362+0000 7fed43506700 1 --2- 172.21.15.189:0/1126015045 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed3c0b2300 0x7fed3c06da20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fed2c0092e0 tx=0x7fed2c0095f0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.363+0000 7fed43506700 1 -- 172.21.15.189:0/1126015045 shutdown_connections 2024-04-06T12:12:31.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.363+0000 7fed43506700 1 --2- 172.21.15.189:0/1126015045 >> v2:172.21.15.192:3300/0 conn(0x7fed3c150690 0x7fed3c159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.363+0000 7fed43506700 1 --2- 172.21.15.189:0/1126015045 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed3c0b2300 0x7fed3c06da20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.363+0000 7fed43506700 1 -- 172.21.15.189:0/1126015045 >> 172.21.15.189:0/1126015045 conn(0x7fed3c0cefa0 msgr2=0x7fed3c0cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:31.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.363+0000 7f5a2935a700 1 Processor -- start 2024-04-06T12:12:31.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.363+0000 7f5a2935a700 1 -- start start 2024-04-06T12:12:31.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.364+0000 7f5a2935a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a24159b30 0x7f5a2406da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.364+0000 7f5a2935a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5a2406df60 0x7f5a2415b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.365+0000 7f5a2935a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a2406e3b0 con 0x7f5a24150690 2024-04-06T12:12:31.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.365+0000 7f5a2935a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a24150a70 con 0x7f5a2406df60 2024-04-06T12:12:31.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.365+0000 7f5a2935a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a24159f90 con 0x7f5a24159b30 2024-04-06T12:12:31.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.365+0000 7f5a227fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a24159b30 0x7f5a2406da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.365+0000 7f5a227fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a24159b30 0x7f5a2406da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59580/0 (socket says 172.21.15.189:59580) 2024-04-06T12:12:31.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.365+0000 7f5a227fc700 1 -- 172.21.15.189:0/3400463687 learned_addr learned my addr 172.21.15.189:0/3400463687 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:31.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.365+0000 7fed43506700 1 -- 172.21.15.189:0/1126015045 shutdown_connections 2024-04-06T12:12:31.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.365+0000 7f5a227fc700 1 -- 172.21.15.189:0/3400463687 >> v2:172.21.15.189:3300/0 conn(0x7f5a2406df60 msgr2=0x7f5a2415b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:31.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.365+0000 7f5a227fc700 1 --2- 172.21.15.189:0/3400463687 >> v2:172.21.15.189:3300/0 conn(0x7f5a2406df60 0x7f5a2415b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.365+0000 7f5a227fc700 1 -- 172.21.15.189:0/3400463687 >> v1:172.21.15.189:6789/0 conn(0x7f5a24150690 legacy=0x7f5a240b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:31.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.365+0000 7f5a227fc700 1 -- 172.21.15.189:0/3400463687 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2415a180 con 0x7f5a24159b30 2024-04-06T12:12:31.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.366+0000 7fed43506700 1 -- 172.21.15.189:0/1126015045 wait complete. 2024-04-06T12:12:31.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.367+0000 7fed43506700 1 Processor -- start 2024-04-06T12:12:31.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.367+0000 7fed43506700 1 -- start start 2024-04-06T12:12:31.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.366+0000 7f5a227fc700 1 --2- 172.21.15.189:0/3400463687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a24159b30 0x7f5a2406da20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c00d370 tx=0x7f5a1c00d680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bbb6d6a2d2be0db server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.366+0000 7f5a21ffb700 1 -- 172.21.15.189:0/3400463687 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1c010070 con 0x7f5a24159b30 2024-04-06T12:12:31.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.366+0000 7f5a21ffb700 1 -- 172.21.15.189:0/3400463687 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1c00dcd0 con 0x7f5a24159b30 2024-04-06T12:12:31.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.366+0000 7f5a21ffb700 1 -- 172.21.15.189:0/3400463687 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1c007d50 con 0x7f5a24159b30 2024-04-06T12:12:31.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.369+0000 7fed43506700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed3c06df60 0x7fed3c1f9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.369+0000 7fed43506700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed3c0b2300 0x7fed3c14b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.369+0000 7fed43506700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed3c150690 0x7fed3c08a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.369+0000 7fed43506700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c14ae60 con 0x7fed3c06df60 2024-04-06T12:12:31.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.369+0000 7fed43506700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c200430 con 0x7fed3c150690 2024-04-06T12:12:31.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.369+0000 7fed43506700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c14cc50 con 0x7fed3c0b2300 2024-04-06T12:12:31.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.369+0000 7fed41aa3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed3c150690 0x7fed3c08a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.369+0000 7fed41aa3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed3c150690 0x7fed3c08a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:58934/0 (socket says 172.21.15.189:58934) 2024-04-06T12:12:31.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.369+0000 7fed41aa3700 1 -- 172.21.15.189:0/2781432276 learned_addr learned my addr 172.21.15.189:0/2781432276 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:31.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.369+0000 7fed40aa1700 1 --2- 172.21.15.189:0/2781432276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed3c0b2300 0x7fed3c14b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.370+0000 7fed40aa1700 1 -- 172.21.15.189:0/2781432276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed3c150690 msgr2=0x7fed3c08a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.370+0000 7fed40aa1700 1 --2- 172.21.15.189:0/2781432276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed3c150690 0x7fed3c08a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.370+0000 7fed40aa1700 1 -- 172.21.15.189:0/2781432276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed3c06df60 msgr2=0x7fed3c1f9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.370+0000 7fed40aa1700 1 --2- 172.21.15.189:0/2781432276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed3c06df60 0x7fed3c1f9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.370+0000 7fed40aa1700 1 -- 172.21.15.189:0/2781432276 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2c009de0 con 0x7fed3c0b2300 2024-04-06T12:12:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.370+0000 7fed40aa1700 1 --2- 172.21.15.189:0/2781432276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed3c0b2300 0x7fed3c14b280 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fed2c017000 tx=0x7fed2c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.367+0000 7f5a2935a700 1 -- 172.21.15.189:0/3400463687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a24159b30 msgr2=0x7f5a2406da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.367+0000 7f5a2935a700 1 --2- 172.21.15.189:0/3400463687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a24159b30 0x7f5a2406da20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c00d370 tx=0x7f5a1c00d680 comp rx=0 tx=0).stop 2024-04-06T12:12:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.368+0000 7f5a2935a700 1 -- 172.21.15.189:0/3400463687 shutdown_connections 2024-04-06T12:12:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.368+0000 7f5a2935a700 1 --2- 172.21.15.189:0/3400463687 >> v2:172.21.15.189:3300/0 conn(0x7f5a2406df60 0x7f5a2415b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.368+0000 7f5a2935a700 1 --2- 172.21.15.189:0/3400463687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a24159b30 0x7f5a2406da20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.368+0000 7f5a2935a700 1 -- 172.21.15.189:0/3400463687 >> 172.21.15.189:0/3400463687 conn(0x7f5a240cefa0 msgr2=0x7f5a240cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:31.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.369+0000 7f5a2935a700 1 -- 172.21.15.189:0/3400463687 shutdown_connections 2024-04-06T12:12:31.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.369+0000 7f5a2935a700 1 -- 172.21.15.189:0/3400463687 wait complete. 2024-04-06T12:12:31.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.371+0000 7fed31ffb700 1 -- 172.21.15.189:0/2781432276 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2c0038a0 con 0x7fed3c0b2300 2024-04-06T12:12:31.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.371+0000 7fed31ffb700 1 -- 172.21.15.189:0/2781432276 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2c01d040 con 0x7fed3c0b2300 2024-04-06T12:12:31.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.371+0000 7fed31ffb700 1 -- 172.21.15.189:0/2781432276 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2c003c00 con 0x7fed3c0b2300 2024-04-06T12:12:31.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.371+0000 7fed43506700 1 -- 172.21.15.189:0/2781432276 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed3c205be0 con 0x7fed3c0b2300 2024-04-06T12:12:31.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.371+0000 7fed43506700 1 -- 172.21.15.189:0/2781432276 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed3c14a7a0 con 0x7fed3c0b2300 2024-04-06T12:12:31.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.371+0000 7f5a2935a700 1 Processor -- start 2024-04-06T12:12:31.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.372+0000 7f5a2935a700 1 -- start start 2024-04-06T12:12:31.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.373+0000 7fed43506700 1 -- 172.21.15.189:0/2781432276 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed200052f0 con 0x7fed3c0b2300 2024-04-06T12:12:31.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.373+0000 7f5a2935a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a2406df60 0x7f5a241515d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.373+0000 7f5a2935a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a24150690 0x7f5a240ce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.373+0000 7f5a2935a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a24159b30 0x7f5a241562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.373+0000 7f5a2935a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a24161c20 con 0x7f5a24150690 2024-04-06T12:12:31.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.373+0000 7f5a2935a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a241f5f30 con 0x7f5a2406df60 2024-04-06T12:12:31.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.373+0000 7f5a2935a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a240cbd90 con 0x7f5a24159b30 2024-04-06T12:12:31.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.373+0000 7f5a22ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a2406df60 0x7f5a241515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.374+0000 7fed31ffb700 1 -- 172.21.15.189:0/2781432276 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fed2c018400 con 0x7fed3c0b2300 2024-04-06T12:12:31.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.375+0000 7fed31ffb700 1 --2- 172.21.15.189:0/2781432276 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fed2806aef0 0x7fed2806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.375+0000 7fed31ffb700 1 -- 172.21.15.189:0/2781432276 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fed2c08af20 con 0x7fed3c0b2300 2024-04-06T12:12:31.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.373+0000 7f5a237fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a24159b30 0x7f5a241562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.373+0000 7f5a22ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a2406df60 0x7f5a241515d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:58942/0 (socket says 172.21.15.189:58942) 2024-04-06T12:12:31.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.373+0000 7f5a22ffd700 1 -- 172.21.15.189:0/3256631209 learned_addr learned my addr 172.21.15.189:0/3256631209 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:31.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.375+0000 7f5a237fe700 1 -- 172.21.15.189:0/3256631209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a2406df60 msgr2=0x7f5a241515d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.375+0000 7f5a237fe700 1 --2- 172.21.15.189:0/3256631209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a2406df60 0x7f5a241515d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.375+0000 7f5a237fe700 1 -- 172.21.15.189:0/3256631209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a24150690 msgr2=0x7f5a240ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.375+0000 7f5a237fe700 1 --2- 172.21.15.189:0/3256631209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a24150690 0x7f5a240ce7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.375+0000 7f5a237fe700 1 -- 172.21.15.189:0/3256631209 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1c00d050 con 0x7f5a24159b30 2024-04-06T12:12:31.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.376+0000 7fed412a2700 1 --2- 172.21.15.189:0/2781432276 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fed2806aef0 0x7fed2806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.377+0000 7fed412a2700 1 --2- 172.21.15.189:0/2781432276 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fed2806aef0 0x7fed2806d3b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fed3400abb0 tx=0x7fed34009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.378+0000 7f5a237fe700 1 --2- 172.21.15.189:0/3256631209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a24159b30 0x7f5a241562f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5a1400b420 tx=0x7f5a1400b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.379+0000 7fed31ffb700 1 -- 172.21.15.189:0/2781432276 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fed2c056050 con 0x7fed3c0b2300 2024-04-06T12:12:31.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.379+0000 7f5a0b7fe700 1 -- 172.21.15.189:0/3256631209 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1400bea0 con 0x7f5a24159b30 2024-04-06T12:12:31.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.379+0000 7f5a0b7fe700 1 -- 172.21.15.189:0/3256631209 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a14011830 con 0x7f5a24159b30 2024-04-06T12:12:31.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.379+0000 7f5a0b7fe700 1 -- 172.21.15.189:0/3256631209 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a14011be0 con 0x7f5a24159b30 2024-04-06T12:12:31.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.379+0000 7f5a2935a700 1 -- 172.21.15.189:0/3256631209 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a240cc070 con 0x7f5a24159b30 2024-04-06T12:12:31.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.379+0000 7f5a2935a700 1 -- 172.21.15.189:0/3256631209 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a240cc630 con 0x7f5a24159b30 2024-04-06T12:12:31.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.381+0000 7f5a2935a700 1 -- 172.21.15.189:0/3256631209 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a24074990 con 0x7f5a24159b30 2024-04-06T12:12:31.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.388+0000 7f5a0b7fe700 1 -- 172.21.15.189:0/3256631209 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5a14011d40 con 0x7f5a24159b30 2024-04-06T12:12:31.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.388+0000 7f5a0b7fe700 1 --2- 172.21.15.189:0/3256631209 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5a1006ad60 0x7f5a1006d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.388+0000 7f5a0b7fe700 1 -- 172.21.15.189:0/3256631209 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5a14089500 con 0x7f5a24159b30 2024-04-06T12:12:31.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.389+0000 7f5a22ffd700 1 --2- 172.21.15.189:0/3256631209 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5a1006ad60 0x7f5a1006d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.390+0000 7f5a22ffd700 1 --2- 172.21.15.189:0/3256631209 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5a1006ad60 0x7f5a1006d220 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5a24202aa0 tx=0x7f5a18009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.395+0000 7f5a0b7fe700 1 -- 172.21.15.189:0/3256631209 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5a140543d0 con 0x7f5a24159b30 2024-04-06T12:12:31.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.411+0000 7f6b08f92700 1 Processor -- start 2024-04-06T12:12:31.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.411+0000 7f6b08f92700 1 -- start start 2024-04-06T12:12:31.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.413+0000 7f6b08f92700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6b0415d7e0 0x7f6b040d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.413+0000 7f6b08f92700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b040d1cc0 0x7f6b040b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.413+0000 7f6b08f92700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b040cdf80 con 0x7f6b040d0890 2024-04-06T12:12:31.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.413+0000 7f6b08f92700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b040d3230 con 0x7f6b0415d7e0 2024-04-06T12:12:31.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.413+0000 7f6b08f92700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b040d2da0 con 0x7f6b040d1cc0 2024-04-06T12:12:31.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.415+0000 7f6b02d9d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b040d1cc0 0x7f6b040b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.415+0000 7f6b02d9d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b040d1cc0 0x7f6b040b66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59596/0 (socket says 172.21.15.189:59596) 2024-04-06T12:12:31.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.415+0000 7f6b02d9d700 1 -- 172.21.15.189:0/1661762451 learned_addr learned my addr 172.21.15.189:0/1661762451 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:31.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.415+0000 7f6b0259c700 1 --2- 172.21.15.189:0/1661762451 >> v2:172.21.15.192:3300/0 conn(0x7f6b0415d7e0 0x7f6b040d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.415+0000 7f6b02d9d700 1 -- 172.21.15.189:0/1661762451 >> v2:172.21.15.192:3300/0 conn(0x7f6b0415d7e0 msgr2=0x7f6b040d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:31.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.415+0000 7f6b02d9d700 1 --2- 172.21.15.189:0/1661762451 >> v2:172.21.15.192:3300/0 conn(0x7f6b0415d7e0 0x7f6b040d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.415+0000 7f6b02d9d700 1 -- 172.21.15.189:0/1661762451 >> v1:172.21.15.192:6789/0 conn(0x7f6b040d0890 legacy=0x7f6b040d15b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-06T12:12:31.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.415+0000 7f6b02d9d700 1 -- 172.21.15.189:0/1661762451 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b040d0cf0 con 0x7f6b040d1cc0 2024-04-06T12:12:31.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.415+0000 7f6b02d9d700 1 --2- 172.21.15.189:0/1661762451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b040d1cc0 0x7f6b040b66b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6afc009e50 tx=0x7f6afc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da467372a574a8a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.416+0000 7f6b0159a700 1 -- 172.21.15.189:0/1661762451 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc00e070 con 0x7f6b040d1cc0 2024-04-06T12:12:31.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.416+0000 7f6b0159a700 1 -- 172.21.15.189:0/1661762451 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6afc003a20 con 0x7f6b040d1cc0 2024-04-06T12:12:31.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.416+0000 7f6b0159a700 1 -- 172.21.15.189:0/1661762451 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc003db0 con 0x7f6b040d1cc0 2024-04-06T12:12:31.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.417+0000 7f6b08f92700 1 -- 172.21.15.189:0/1661762451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b040d1cc0 msgr2=0x7f6b040b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.417+0000 7f6b08f92700 1 --2- 172.21.15.189:0/1661762451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b040d1cc0 0x7f6b040b66b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6afc009e50 tx=0x7f6afc00b330 comp rx=0 tx=0).stop 2024-04-06T12:12:31.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.417+0000 7f6b08f92700 1 -- 172.21.15.189:0/1661762451 shutdown_connections 2024-04-06T12:12:31.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.417+0000 7f6b08f92700 1 --2- 172.21.15.189:0/1661762451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b040d1cc0 0x7f6b040b66b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.417+0000 7f6b08f92700 1 --2- 172.21.15.189:0/1661762451 >> v2:172.21.15.192:3300/0 conn(0x7f6b0415d7e0 0x7f6b040d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.417+0000 7f6b08f92700 1 -- 172.21.15.189:0/1661762451 >> 172.21.15.189:0/1661762451 conn(0x7f6b04072250 msgr2=0x7f6b04072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:31.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.417+0000 7f6b08f92700 1 -- 172.21.15.189:0/1661762451 shutdown_connections 2024-04-06T12:12:31.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.417+0000 7f6b08f92700 1 -- 172.21.15.189:0/1661762451 wait complete. 2024-04-06T12:12:31.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.419+0000 7f6b08f92700 1 Processor -- start 2024-04-06T12:12:31.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.419+0000 7f6b08f92700 1 -- start start 2024-04-06T12:12:31.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.420+0000 7f6b08f92700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b040d0890 0x7f6b040cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.420+0000 7f6b08f92700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b040d1cc0 0x7f6b04156fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.420+0000 7f6b08f92700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b0415d7e0 0x7f6b040c7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.420+0000 7f6b08f92700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b040cac00 con 0x7f6b040d1cc0 2024-04-06T12:12:31.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.420+0000 7f6b08f92700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b04157500 con 0x7f6b0415d7e0 2024-04-06T12:12:31.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.420+0000 7f6b08f92700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b040c7e70 con 0x7f6b040d0890 2024-04-06T12:12:31.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.421+0000 7f6c23e11700 1 Processor -- start 2024-04-06T12:12:31.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.421+0000 7f6b0259c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b040d0890 0x7f6b040cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.421+0000 7f6b0259c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b040d0890 0x7f6b040cb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59606/0 (socket says 172.21.15.189:59606) 2024-04-06T12:12:31.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.421+0000 7f6b0259c700 1 -- 172.21.15.189:0/3100661007 learned_addr learned my addr 172.21.15.189:0/3100661007 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:31.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.421+0000 7f6c23e11700 1 -- start start 2024-04-06T12:12:31.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.422+0000 7f6c23e11700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6c1c150690 0x7f6c1c159b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.422+0000 7f6c23e11700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6c1c0b2300 0x7f6c1c06da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.422+0000 7f6c23e11700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c1c06df60 0x7f6c1c15d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.422+0000 7f6c23e11700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c1c150a70 con 0x7f6c1c0b2300 2024-04-06T12:12:31.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.422+0000 7f6c23e11700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c1c06e390 con 0x7f6c1c150690 2024-04-06T12:12:31.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.422+0000 7f6c23e11700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c15a180 con 0x7f6c1c06df60 2024-04-06T12:12:31.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.423+0000 7f6c213ac700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6c1c0b2300 0x7f6c1c06da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.423+0000 7f6c213ac700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6c1c0b2300 0x7f6c1c06da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38234/0 (socket says 172.21.15.189:38234) 2024-04-06T12:12:31.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.423+0000 7f6c213ac700 1 -- 172.21.15.189:0/3768981096 learned_addr learned my addr 172.21.15.189:0/3768981096 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:31.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.423+0000 7f6c213ac700 1 -- 172.21.15.189:0/3768981096 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c1c06df60 msgr2=0x7f6c1c15d120 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-06T12:12:31.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.423+0000 7f6c213ac700 1 --2- 172.21.15.189:0/3768981096 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c1c06df60 0x7f6c1c15d120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.423+0000 7f6c213ac700 1 -- 172.21.15.189:0/3768981096 >> v2:172.21.15.192:3300/0 conn(0x7f6c1c150690 msgr2=0x7f6c1c159b30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-06T12:12:31.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.423+0000 7f6c213ac700 1 --2- 172.21.15.189:0/3768981096 >> v2:172.21.15.192:3300/0 conn(0x7f6c1c150690 0x7f6c1c159b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.423+0000 7f6c213ac700 1 -- 172.21.15.189:0/3768981096 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1c15a460 con 0x7f6c1c0b2300 2024-04-06T12:12:31.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.423+0000 7f7eb2e3e700 1 Processor -- start 2024-04-06T12:12:31.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.424+0000 7f7eb2e3e700 1 -- start start 2024-04-06T12:12:31.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f7eb2e3e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7eac150690 0x7f7eac159b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f7eb2e3e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7eac0b2300 0x7f7eac06da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f7eb2e3e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eac06e370 con 0x7f7eac06df60 2024-04-06T12:12:31.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f7eb2e3e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eac150a70 con 0x7f7eac150690 2024-04-06T12:12:31.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f7eb2e3e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eac15a180 con 0x7f7eac0b2300 2024-04-06T12:12:31.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.424+0000 7f6c213ac700 1 --2- 172.21.15.189:0/3768981096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c1c0b2300 0x7f6c1c06da20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c0cb9a0 tx=0x7f6c18009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59407094c63e2c38 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.424+0000 7f6c20bab700 1 -- 172.21.15.189:0/3768981096 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1800b870 con 0x7f6c1c0b2300 2024-04-06T12:12:31.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.424+0000 7f6c20bab700 1 -- 172.21.15.189:0/3768981096 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1800b9d0 con 0x7f6c1c0b2300 2024-04-06T12:12:31.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.424+0000 7f6c20bab700 1 -- 172.21.15.189:0/3768981096 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1800bd60 con 0x7f6c1c0b2300 2024-04-06T12:12:31.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f6c23e11700 1 -- 172.21.15.189:0/3768981096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c1c0b2300 msgr2=0x7f6c1c06da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f6c23e11700 1 --2- 172.21.15.189:0/3768981096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c1c0b2300 0x7f6c1c06da20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c0cb9a0 tx=0x7f6c18009bd0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f6c23e11700 1 -- 172.21.15.189:0/3768981096 shutdown_connections 2024-04-06T12:12:31.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f6c23e11700 1 --2- 172.21.15.189:0/3768981096 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c1c06df60 0x7f6c1c15d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f6c23e11700 1 --2- 172.21.15.189:0/3768981096 >> v2:172.21.15.192:3300/0 conn(0x7f6c1c150690 0x7f6c1c159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f6c23e11700 1 --2- 172.21.15.189:0/3768981096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c1c0b2300 0x7f6c1c06da20 secure :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c0cb9a0 tx=0x7f6c18009bd0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f6c23e11700 1 -- 172.21.15.189:0/3768981096 >> 172.21.15.189:0/3768981096 conn(0x7f6c1c0cefa0 msgr2=0x7f6c1c0cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:31.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f6b02d9d700 1 --2- 172.21.15.189:0/3100661007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b0415d7e0 0x7f6b040c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f7eabfff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7eac0b2300 0x7f7eac06da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f7eb13db700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f7eac06df60 0x7f7eac15d120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:52510/0 (socket says 172.21.15.189:52510) 2024-04-06T12:12:31.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f7eb13db700 1 -- 172.21.15.189:0/2638239314 learned_addr learned my addr 172.21.15.189:0/2638239314 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:31.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f1bf7c02700 1 Processor -- start 2024-04-06T12:12:31.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f6b0259c700 1 -- 172.21.15.189:0/3100661007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b0415d7e0 msgr2=0x7f6b040c7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f6b0259c700 1 --2- 172.21.15.189:0/3100661007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b0415d7e0 0x7f6b040c7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f6b0259c700 1 -- 172.21.15.189:0/3100661007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b040d1cc0 msgr2=0x7f6b04156fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f6b0259c700 1 --2- 172.21.15.189:0/3100661007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b040d1cc0 0x7f6b04156fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.425+0000 7f6b0259c700 1 -- 172.21.15.189:0/3100661007 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6afc009b30 con 0x7f6b040d0890 2024-04-06T12:12:31.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.426+0000 7f7eb0bda700 1 --2- 172.21.15.189:0/2638239314 >> v2:172.21.15.189:3300/0 conn(0x7f7eac150690 0x7f7eac159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.426+0000 7f7eabfff700 1 -- 172.21.15.189:0/2638239314 >> v2:172.21.15.189:3300/0 conn(0x7f7eac150690 msgr2=0x7f7eac159b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:31.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.426+0000 7f7eabfff700 1 --2- 172.21.15.189:0/2638239314 >> v2:172.21.15.189:3300/0 conn(0x7f7eac150690 0x7f7eac159b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.426+0000 7f7eabfff700 1 -- 172.21.15.189:0/2638239314 >> v1:172.21.15.192:6789/0 conn(0x7f7eac06df60 legacy=0x7f7eac15d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.426+0000 7f7eabfff700 1 -- 172.21.15.189:0/2638239314 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eac15a460 con 0x7f7eac0b2300 2024-04-06T12:12:31.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.426+0000 7f6c23e11700 1 -- 172.21.15.189:0/3768981096 shutdown_connections 2024-04-06T12:12:31.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.426+0000 7f1bf7c02700 1 -- start start 2024-04-06T12:12:31.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.427+0000 7f1bf7c02700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bf0159b30 0x7f1bf006da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.427+0000 7f1bf7c02700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1bf006df60 0x7f1bf015b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.427+0000 7f1bf7c02700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf006e3b0 con 0x7f1bf0150690 2024-04-06T12:12:31.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.427+0000 7f1bf7c02700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bf0150a70 con 0x7f1bf006df60 2024-04-06T12:12:31.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.427+0000 7f1bf7c02700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf0159f90 con 0x7f1bf0159b30 2024-04-06T12:12:31.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.428+0000 7f1bf599e700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f1bf0150690 0x7f1bf00b2300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:52516/0 (socket says 172.21.15.189:52516) 2024-04-06T12:12:31.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.428+0000 7f1bf599e700 1 -- 172.21.15.189:0/1222214669 learned_addr learned my addr 172.21.15.189:0/1222214669 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:31.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.426+0000 7f6c23e11700 1 -- 172.21.15.189:0/3768981096 wait complete. 2024-04-06T12:12:31.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.428+0000 7f6c23e11700 1 Processor -- start 2024-04-06T12:12:31.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.428+0000 7f1bf619f700 1 --2- 172.21.15.189:0/1222214669 >> v2:172.21.15.192:3300/0 conn(0x7f1bf006df60 0x7f1bf015b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.428+0000 7f1bf499c700 1 -- 172.21.15.189:0/1222214669 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2247067801 0 0) 0x7f1bec001110 con 0x7f1bf0150690 2024-04-06T12:12:31.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.428+0000 7f1bf499c700 1 -- 172.21.15.189:0/1222214669 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bdc0034d0 con 0x7f1bf0150690 2024-04-06T12:12:31.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.428+0000 7f6c23e11700 1 -- start start 2024-04-06T12:12:31.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.428+0000 7f1bf519d700 1 --2- 172.21.15.189:0/1222214669 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bf0159b30 0x7f1bf006da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f6b0259c700 1 --2- 172.21.15.189:0/3100661007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b040d0890 0x7f6b040cb020 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6af400e810 tx=0x7f6af400ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f6af2ffd700 1 -- 172.21.15.189:0/3100661007 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af4013610 con 0x7f6b040d0890 2024-04-06T12:12:31.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f6af2ffd700 1 -- 172.21.15.189:0/3100661007 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af4013770 con 0x7f6b040d0890 2024-04-06T12:12:31.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f6af2ffd700 1 -- 172.21.15.189:0/3100661007 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af4013b20 con 0x7f6b040d0890 2024-04-06T12:12:31.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f1bf499c700 1 -- 172.21.15.189:0/1222214669 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1315310015 0 0) 0x7f1bdc0034d0 con 0x7f1bf0150690 2024-04-06T12:12:31.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f1bf499c700 1 -- 172.21.15.189:0/1222214669 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bf0159b30 msgr2=0x7f1bf006da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:31.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f1bf499c700 1 --2- 172.21.15.189:0/1222214669 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bf0159b30 0x7f1bf006da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f1bf499c700 1 -- 172.21.15.189:0/1222214669 >> v2:172.21.15.192:3300/0 conn(0x7f1bf006df60 msgr2=0x7f1bf015b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:31.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f1bf499c700 1 --2- 172.21.15.189:0/1222214669 >> v2:172.21.15.192:3300/0 conn(0x7f1bf006df60 0x7f1bf015b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f1bf499c700 1 -- 172.21.15.189:0/1222214669 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf015a180 con 0x7f1bf0150690 2024-04-06T12:12:31.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f1bf499c700 1 -- 172.21.15.189:0/1222214669 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f1bec004d60 con 0x7f1bf0150690 2024-04-06T12:12:31.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f1bf499c700 1 -- 172.21.15.189:0/1222214669 >> v1:172.21.15.192:6789/0 conn(0x7f1bf0150690 legacy=0x7f1bf00b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f1bf499c700 1 --2- 172.21.15.189:0/1222214669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bdc003af0 0x7f1bdc003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f1bf499c700 1 --2- 172.21.15.189:0/1222214669 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bdc0088e0 0x7f1bdc00acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f1bf499c700 1 --2- 172.21.15.189:0/1222214669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bdc00b3a0 0x7f1bdc00d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f1bf499c700 1 -- 172.21.15.189:0/1222214669 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc00e0e0 con 0x7f1bdc00b3a0 2024-04-06T12:12:31.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f1bf499c700 1 -- 172.21.15.189:0/1222214669 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc00e250 con 0x7f1bdc003af0 2024-04-06T12:12:31.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f1bf499c700 1 -- 172.21.15.189:0/1222214669 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc00e410 con 0x7f1bdc0088e0 2024-04-06T12:12:31.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f1bf619f700 1 --2- 172.21.15.189:0/1222214669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bdc00b3a0 0x7f1bdc00d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.430+0000 7f1bf519d700 1 --2- 172.21.15.189:0/1222214669 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bdc0088e0 0x7f1bdc00acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.430+0000 7f1bf619f700 1 -- 172.21.15.189:0/1222214669 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bdc0088e0 msgr2=0x7f1bdc00acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:31.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.430+0000 7f1bf619f700 1 --2- 172.21.15.189:0/1222214669 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bdc0088e0 0x7f1bdc00acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.430+0000 7f1bf619f700 1 -- 172.21.15.189:0/1222214669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bdc003af0 msgr2=0x7f1bdc003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:31.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.430+0000 7f1bf619f700 1 --2- 172.21.15.189:0/1222214669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bdc003af0 0x7f1bdc003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.430+0000 7f1bf619f700 1 -- 172.21.15.189:0/1222214669 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf015a180 con 0x7f1bdc00b3a0 2024-04-06T12:12:31.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.430+0000 7f1bf619f700 1 --2- 172.21.15.189:0/1222214669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bdc00b3a0 0x7f1bdc00d830 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009bb0 tx=0x7f1bdc003780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9860d2cd97de7fbf server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.430+0000 7f1bf499c700 1 -- 172.21.15.189:0/1222214669 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be8003c20 con 0x7f1bdc00b3a0 2024-04-06T12:12:31.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.430+0000 7f1bf499c700 1 -- 172.21.15.189:0/1222214669 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be800f040 con 0x7f1bdc00b3a0 2024-04-06T12:12:31.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.430+0000 7f1bf499c700 1 -- 172.21.15.189:0/1222214669 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be800e9d0 con 0x7f1bdc00b3a0 2024-04-06T12:12:31.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f6c23e11700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c1c06df60 0x7f6c1c0cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f6c23e11700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c1c14b610 0x7f6c1c149e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f6c23e11700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c1c148710 0x7f6c1c158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f6c23e11700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c14a360 con 0x7f6c1c14b610 2024-04-06T12:12:31.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f6c23e11700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c14a570 con 0x7f6c1c148710 2024-04-06T12:12:31.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f6c23e11700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c14a850 con 0x7f6c1c06df60 2024-04-06T12:12:31.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f6c213ac700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c1c14b610 0x7f6c1c149e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.430+0000 7f6c213ac700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c1c14b610 0x7f6c1c149e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38272/0 (socket says 172.21.15.189:38272) 2024-04-06T12:12:31.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.430+0000 7f6c213ac700 1 -- 172.21.15.189:0/2412168515 learned_addr learned my addr 172.21.15.189:0/2412168515 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:31.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.430+0000 7f6c213ac700 1 -- 172.21.15.189:0/2412168515 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c1c06df60 msgr2=0x7f6c1c0cdd10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:12:31.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.430+0000 7f6c213ac700 1 --2- 172.21.15.189:0/2412168515 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c1c06df60 0x7f6c1c0cdd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.430+0000 7f6c213ac700 1 -- 172.21.15.189:0/2412168515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c1c148710 msgr2=0x7f6c1c158510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:12:31.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.430+0000 7f6c213ac700 1 --2- 172.21.15.189:0/2412168515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c1c148710 0x7f6c1c158510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.430+0000 7f6c213ac700 1 -- 172.21.15.189:0/2412168515 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c18009550 con 0x7f6c1c14b610 2024-04-06T12:12:31.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.431+0000 7f6c213ac700 1 --2- 172.21.15.189:0/2412168515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c1c14b610 0x7f6c1c149e20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6c18013000 tx=0x7f6c18004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.426+0000 7f7eabfff700 1 --2- 172.21.15.189:0/2638239314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7eac0b2300 0x7f7eac06da20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7eac0703e0 tx=0x7f7ea000bdc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb4bd5da925e01ae server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.427+0000 7f7eab7fe700 1 -- 172.21.15.189:0/2638239314 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea000dc70 con 0x7f7eac0b2300 2024-04-06T12:12:31.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.427+0000 7f7eab7fe700 1 -- 172.21.15.189:0/2638239314 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea000ddd0 con 0x7f7eac0b2300 2024-04-06T12:12:31.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.427+0000 7f7eab7fe700 1 -- 172.21.15.189:0/2638239314 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0010690 con 0x7f7eac0b2300 2024-04-06T12:12:31.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.428+0000 7f7eb2e3e700 1 -- 172.21.15.189:0/2638239314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7eac0b2300 msgr2=0x7f7eac06da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.428+0000 7f7eb2e3e700 1 --2- 172.21.15.189:0/2638239314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7eac0b2300 0x7f7eac06da20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7eac0703e0 tx=0x7f7ea000bdc0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f7eb2e3e700 1 -- 172.21.15.189:0/2638239314 shutdown_connections 2024-04-06T12:12:31.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f7eb2e3e700 1 --2- 172.21.15.189:0/2638239314 >> v2:172.21.15.189:3300/0 conn(0x7f7eac150690 0x7f7eac159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f7eb2e3e700 1 --2- 172.21.15.189:0/2638239314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7eac0b2300 0x7f7eac06da20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f7eb2e3e700 1 -- 172.21.15.189:0/2638239314 >> 172.21.15.189:0/2638239314 conn(0x7f7eac0cefa0 msgr2=0x7f7eac0cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:31.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f6b08f92700 1 -- 172.21.15.189:0/3100661007 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b040ca540 con 0x7f6b040d0890 2024-04-06T12:12:31.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.429+0000 7f6b08f92700 1 -- 172.21.15.189:0/3100661007 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b040c7b60 con 0x7f6b040d0890 2024-04-06T12:12:31.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.431+0000 7f6b08f92700 1 -- 172.21.15.189:0/3100661007 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b04074990 con 0x7f6b040d0890 2024-04-06T12:12:31.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.431+0000 7f1bf7c02700 1 -- 172.21.15.189:0/1222214669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bdc00b3a0 msgr2=0x7f1bdc00d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.431+0000 7f1bf7c02700 1 --2- 172.21.15.189:0/1222214669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bdc00b3a0 0x7f1bdc00d830 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009bb0 tx=0x7f1bdc003780 comp rx=0 tx=0).stop 2024-04-06T12:12:31.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.431+0000 7f1bf619f700 1 -- 172.21.15.189:0/1222214669 reap_dead start 2024-04-06T12:12:31.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.432+0000 7f1bf7c02700 1 -- 172.21.15.189:0/1222214669 shutdown_connections 2024-04-06T12:12:31.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.432+0000 7f1bf7c02700 1 -- 172.21.15.189:0/1222214669 >> 172.21.15.189:0/1222214669 conn(0x7f1bf00cefa0 msgr2=0x7f1bf00cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:31.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.432+0000 7f1bf7c02700 1 -- 172.21.15.189:0/1222214669 shutdown_connections 2024-04-06T12:12:31.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.432+0000 7f1bf7c02700 1 -- 172.21.15.189:0/1222214669 wait complete. 2024-04-06T12:12:31.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.434+0000 7f1bf7c02700 1 Processor -- start 2024-04-06T12:12:31.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.431+0000 7f6c127fc700 1 -- 172.21.15.189:0/2412168515 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c18004550 con 0x7f6c1c14b610 2024-04-06T12:12:31.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.432+0000 7f6c127fc700 1 -- 172.21.15.189:0/2412168515 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c180036a0 con 0x7f6c1c14b610 2024-04-06T12:12:31.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.432+0000 7f6c127fc700 1 -- 172.21.15.189:0/2412168515 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c18003a50 con 0x7f6c1c14b610 2024-04-06T12:12:31.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.432+0000 7f6c23e11700 1 -- 172.21.15.189:0/2412168515 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c1c14ab30 con 0x7f6c1c14b610 2024-04-06T12:12:31.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.432+0000 7f6c23e11700 1 -- 172.21.15.189:0/2412168515 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c1c14b0f0 con 0x7f6c1c14b610 2024-04-06T12:12:31.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.434+0000 7f6c23e11700 1 -- 172.21.15.189:0/2412168515 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c040052f0 con 0x7f6c1c14b610 2024-04-06T12:12:31.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.431+0000 7f7eb2e3e700 1 -- 172.21.15.189:0/2638239314 shutdown_connections 2024-04-06T12:12:31.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.432+0000 7f7eb2e3e700 1 -- 172.21.15.189:0/2638239314 wait complete. 2024-04-06T12:12:31.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.433+0000 7f7eb2e3e700 1 Processor -- start 2024-04-06T12:12:31.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.433+0000 7f7eb2e3e700 1 -- start start 2024-04-06T12:12:31.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.434+0000 7f7eb2e3e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7eac06df60 0x7f7eac14fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.434+0000 7f7eb2e3e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7eac0b2300 0x7f7eac162270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.434+0000 7f7eb2e3e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7eac150690 0x7f7eac1627b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.434+0000 7f7eb2e3e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac14cd00 con 0x7f7eac0b2300 2024-04-06T12:12:31.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.434+0000 7f7eb2e3e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac14c080 con 0x7f7eac150690 2024-04-06T12:12:31.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.434+0000 7f7eb2e3e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac14c360 con 0x7f7eac06df60 2024-04-06T12:12:31.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.431+0000 7f6af2ffd700 1 -- 172.21.15.189:0/3100661007 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6af4013c80 con 0x7f6b040d0890 2024-04-06T12:12:31.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.431+0000 7f6af2ffd700 1 --2- 172.21.15.189:0/3100661007 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ae406ad60 0x7f6ae406d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.431+0000 7f6af2ffd700 1 -- 172.21.15.189:0/3100661007 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6af40892d0 con 0x7f6b040d0890 2024-04-06T12:12:31.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.434+0000 7f7eb0bda700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7eac06df60 0x7f7eac14fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.434+0000 7f7eb0bda700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7eac06df60 0x7f7eac14fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59632/0 (socket says 172.21.15.189:59632) 2024-04-06T12:12:31.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.434+0000 7f7eb0bda700 1 -- 172.21.15.189:0/797495531 learned_addr learned my addr 172.21.15.189:0/797495531 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:31.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.434+0000 7f7eabfff700 1 --2- 172.21.15.189:0/797495531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7eac0b2300 0x7f7eac162270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.434+0000 7f1bf7c02700 1 -- start start 2024-04-06T12:12:31.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.435+0000 7f1bf7c02700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bdc00b3a0 0x7f1bf00cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.435+0000 7f1bf7c02700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bf014d3a0 0x7f1bf0157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.435+0000 7f1bf7c02700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bf0157ff0 0x7f1bf01e3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.435+0000 7f1bf7c02700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf01e3fc0 con 0x7f1bdc00b3a0 2024-04-06T12:12:31.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.435+0000 7f1bf7c02700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf0158470 con 0x7f1bf014d3a0 2024-04-06T12:12:31.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.435+0000 7f1bf7c02700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf00cc860 con 0x7f1bf0157ff0 2024-04-06T12:12:31.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.435+0000 7f1bf619f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bf0157ff0 0x7f1bf01e3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.435+0000 7f1bf619f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bf0157ff0 0x7f1bf01e3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59642/0 (socket says 172.21.15.189:59642) 2024-04-06T12:12:31.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.435+0000 7f1bf619f700 1 -- 172.21.15.189:0/4150771694 learned_addr learned my addr 172.21.15.189:0/4150771694 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:31.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.435+0000 7f7eabfff700 1 -- 172.21.15.189:0/797495531 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7eac06df60 msgr2=0x7f7eac14fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.435+0000 7f7eabfff700 1 --2- 172.21.15.189:0/797495531 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7eac06df60 0x7f7eac14fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.435+0000 7f7eabfff700 1 -- 172.21.15.189:0/797495531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7eac150690 msgr2=0x7f7eac1627b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-06T12:12:31.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.435+0000 7f7eabfff700 1 --2- 172.21.15.189:0/797495531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7eac150690 0x7f7eac1627b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.435+0000 7f7eabfff700 1 -- 172.21.15.189:0/797495531 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea000b740 con 0x7f7eac0b2300 2024-04-06T12:12:31.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.435+0000 7f7eabfff700 1 --2- 172.21.15.189:0/797495531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7eac0b2300 0x7f7eac162270 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0018000 tx=0x7f7ea0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.436+0000 7f1bf619f700 1 -- 172.21.15.189:0/4150771694 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bf014d3a0 msgr2=0x7f1bf0157ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-06T12:12:31.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.436+0000 7f1bf619f700 1 --2- 172.21.15.189:0/4150771694 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bf014d3a0 0x7f1bf0157ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.436+0000 7f1bf619f700 1 -- 172.21.15.189:0/4150771694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bdc00b3a0 msgr2=0x7f1bf00cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.436+0000 7f1bf619f700 1 --2- 172.21.15.189:0/4150771694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bdc00b3a0 0x7f1bf00cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.436+0000 7f1bf619f700 1 -- 172.21.15.189:0/4150771694 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf00cbf30 con 0x7f1bf0157ff0 2024-04-06T12:12:31.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.436+0000 7f6b01d9b700 1 --2- 172.21.15.189:0/3100661007 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ae406ad60 0x7f6ae406d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.436+0000 7f1bf619f700 1 --2- 172.21.15.189:0/4150771694 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bf0157ff0 0x7f1bf01e3200 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1be801d040 tx=0x7f1be8009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.437+0000 7f1be67fc700 1 -- 172.21.15.189:0/4150771694 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be801b850 con 0x7f1bf0157ff0 2024-04-06T12:12:31.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.437+0000 7f1be67fc700 1 -- 172.21.15.189:0/4150771694 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be801b9b0 con 0x7f1bf0157ff0 2024-04-06T12:12:31.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.437+0000 7f1be67fc700 1 -- 172.21.15.189:0/4150771694 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be801bd40 con 0x7f1bf0157ff0 2024-04-06T12:12:31.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.437+0000 7f6b01d9b700 1 --2- 172.21.15.189:0/3100661007 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ae406ad60 0x7f6ae406d220 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6b0415ed20 tx=0x7f6af8006c30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.437+0000 7f1bf7c02700 1 -- 172.21.15.189:0/4150771694 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf00cc210 con 0x7f1bf0157ff0 2024-04-06T12:12:31.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.437+0000 7f1bf7c02700 1 -- 172.21.15.189:0/4150771694 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf014d820 con 0x7f1bf0157ff0 2024-04-06T12:12:31.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.439+0000 7f1bf7c02700 1 -- 172.21.15.189:0/4150771694 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd80052f0 con 0x7f1bf0157ff0 2024-04-06T12:12:31.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.441+0000 7f1be67fc700 1 -- 172.21.15.189:0/4150771694 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1be80233c0 con 0x7f1bf0157ff0 2024-04-06T12:12:31.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.441+0000 7f1be67fc700 1 --2- 172.21.15.189:0/4150771694 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1bd406ad60 0x7f1bd406d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.441+0000 7f1be67fc700 1 -- 172.21.15.189:0/4150771694 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1be808b240 con 0x7f1bf0157ff0 2024-04-06T12:12:31.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.442+0000 7f1bf599e700 1 --2- 172.21.15.189:0/4150771694 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1bd406ad60 0x7f1bd406d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.443+0000 7f1bf599e700 1 --2- 172.21.15.189:0/4150771694 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1bd406ad60 0x7f1bd406d220 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1bec004e90 tx=0x7f1bec004b20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.444+0000 7f7ea97fa700 1 -- 172.21.15.189:0/797495531 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea001d070 con 0x7f7eac0b2300 2024-04-06T12:12:31.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.444+0000 7f7ea97fa700 1 -- 172.21.15.189:0/797495531 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea0003ae0 con 0x7f7eac0b2300 2024-04-06T12:12:31.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.444+0000 7f7ea97fa700 1 -- 172.21.15.189:0/797495531 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0003e50 con 0x7f7eac0b2300 2024-04-06T12:12:31.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.446+0000 7f1be67fc700 1 -- 172.21.15.189:0/4150771694 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1be8056240 con 0x7f1bf0157ff0 2024-04-06T12:12:31.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.444+0000 7f7eb2e3e700 1 -- 172.21.15.189:0/797495531 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eac14c640 con 0x7f7eac0b2300 2024-04-06T12:12:31.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.444+0000 7f7eb2e3e700 1 -- 172.21.15.189:0/797495531 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eac14bd10 con 0x7f7eac0b2300 2024-04-06T12:12:31.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.441+0000 7f6c127fc700 1 -- 172.21.15.189:0/2412168515 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6c18003bb0 con 0x7f6c1c14b610 2024-04-06T12:12:31.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.441+0000 7f6c127fc700 1 --2- 172.21.15.189:0/2412168515 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c0006ad60 0x7f6c0006d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.442+0000 7f6c127fc700 1 -- 172.21.15.189:0/2412168515 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6c1808b110 con 0x7f6c1c14b610 2024-04-06T12:12:31.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.442+0000 7f6c21bad700 1 --2- 172.21.15.189:0/2412168515 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c0006ad60 0x7f6c0006d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.444+0000 7f6c21bad700 1 --2- 172.21.15.189:0/2412168515 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c0006ad60 0x7f6c0006d220 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c0b5250 tx=0x7f6c14009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.444+0000 7f6c127fc700 1 -- 172.21.15.189:0/2412168515 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6c18056070 con 0x7f6c1c14b610 2024-04-06T12:12:31.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.446+0000 7f7eb2e3e700 1 -- 172.21.15.189:0/797495531 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8c0052f0 con 0x7f7eac0b2300 2024-04-06T12:12:31.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.441+0000 7f6af2ffd700 1 -- 172.21.15.189:0/3100661007 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6af40541b0 con 0x7f6b040d0890 2024-04-06T12:12:31.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.457+0000 7f7ea97fa700 1 -- 172.21.15.189:0/797495531 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7ea0019400 con 0x7f7eac0b2300 2024-04-06T12:12:31.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.457+0000 7f7ea97fa700 1 --2- 172.21.15.189:0/797495531 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e9806ad60 0x7f7e9806d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.457+0000 7f7ea97fa700 1 -- 172.21.15.189:0/797495531 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7ea008aae0 con 0x7f7eac0b2300 2024-04-06T12:12:31.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.463+0000 7f7ea97fa700 1 -- 172.21.15.189:0/797495531 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7ea0055b60 con 0x7f7eac0b2300 2024-04-06T12:12:31.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.465+0000 7f7eb0bda700 1 --2- 172.21.15.189:0/797495531 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e9806ad60 0x7f7e9806d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.466+0000 7f7eb0bda700 1 --2- 172.21.15.189:0/797495531 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e9806ad60 0x7f7e9806d220 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7eac0a93e0 tx=0x7f7e9c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.481+0000 7fe73837b700 1 Processor -- start 2024-04-06T12:12:31.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.482+0000 7fe73837b700 1 -- start start 2024-04-06T12:12:31.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.483+0000 7fe73837b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe730150690 0x7fe730159b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.483+0000 7fe73837b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe73006e130 0x7fe73015b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.483+0000 7fe73837b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73015a180 con 0x7fe7300b2300 2024-04-06T12:12:31.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.483+0000 7fe73837b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe730150a70 con 0x7fe730150690 2024-04-06T12:12:31.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.483+0000 7fe73837b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe73015a460 con 0x7fe73006e130 2024-04-06T12:12:31.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe736117700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe730150690 0x7fe730159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe736117700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe730150690 0x7fe730159b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38302/0 (socket says 172.21.15.189:38302) 2024-04-06T12:12:31.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe736117700 1 -- 172.21.15.189:0/3128065604 learned_addr learned my addr 172.21.15.189:0/3128065604 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:31.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe736918700 1 --2- 172.21.15.189:0/3128065604 >> v2:172.21.15.192:3300/0 conn(0x7fe73006e130 0x7fe73015b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe735115700 1 -- 172.21.15.189:0/3128065604 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237039614 0 0) 0x7fe73015a180 con 0x7fe7300b2300 2024-04-06T12:12:31.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe735115700 1 -- 172.21.15.189:0/3128065604 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe71c0035b0 con 0x7fe7300b2300 2024-04-06T12:12:31.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe735115700 1 -- 172.21.15.189:0/3128065604 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2608417027 0 0) 0x7fe71c0035b0 con 0x7fe7300b2300 2024-04-06T12:12:31.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe735115700 1 -- 172.21.15.189:0/3128065604 >> v2:172.21.15.192:3300/0 conn(0x7fe73006e130 msgr2=0x7fe73015b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:31.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe735115700 1 --2- 172.21.15.189:0/3128065604 >> v2:172.21.15.192:3300/0 conn(0x7fe73006e130 0x7fe73015b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe735115700 1 -- 172.21.15.189:0/3128065604 >> v2:172.21.15.189:3300/0 conn(0x7fe730150690 msgr2=0x7fe730159b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:31.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe735115700 1 --2- 172.21.15.189:0/3128065604 >> v2:172.21.15.189:3300/0 conn(0x7fe730150690 0x7fe730159b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe735115700 1 -- 172.21.15.189:0/3128065604 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73015a740 con 0x7fe7300b2300 2024-04-06T12:12:31.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe735115700 1 -- 172.21.15.189:0/3128065604 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fe720001670 con 0x7fe7300b2300 2024-04-06T12:12:31.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe735115700 1 -- 172.21.15.189:0/3128065604 >> v1:172.21.15.192:6789/0 conn(0x7fe7300b2300 legacy=0x7fe73006da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe735115700 1 --2- 172.21.15.189:0/3128065604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe71c003a00 0x7fe71c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe735115700 1 --2- 172.21.15.189:0/3128065604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe71c004850 0x7fe71c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe735115700 1 --2- 172.21.15.189:0/3128065604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe71c005210 0x7fe71c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe735115700 1 -- 172.21.15.189:0/3128065604 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73015a460 con 0x7fe71c004850 2024-04-06T12:12:31.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe735115700 1 -- 172.21.15.189:0/3128065604 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71c00bf50 con 0x7fe71c005210 2024-04-06T12:12:31.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.484+0000 7fe735115700 1 -- 172.21.15.189:0/3128065604 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe71c00c0e0 con 0x7fe71c003a00 2024-04-06T12:12:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.485+0000 7fe736117700 1 --2- 172.21.15.189:0/3128065604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe71c003a00 0x7fe71c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.485+0000 7fe736918700 1 --2- 172.21.15.189:0/3128065604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe71c005210 0x7fe71c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.485+0000 7fe735916700 1 --2- 172.21.15.189:0/3128065604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe71c004850 0x7fe71c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.485+0000 7fe736117700 1 -- 172.21.15.189:0/3128065604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe71c005210 msgr2=0x7fe71c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.485+0000 7fe736117700 1 --2- 172.21.15.189:0/3128065604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe71c005210 0x7fe71c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.485+0000 7fe736117700 1 -- 172.21.15.189:0/3128065604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe71c004850 msgr2=0x7fe71c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.485+0000 7fe736117700 1 --2- 172.21.15.189:0/3128065604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe71c004850 0x7fe71c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.485+0000 7fe736117700 1 -- 172.21.15.189:0/3128065604 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73015a740 con 0x7fe71c003a00 2024-04-06T12:12:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.485+0000 7fe736918700 1 -- 172.21.15.189:0/3128065604 reap_dead start 2024-04-06T12:12:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.485+0000 7fe736117700 1 --2- 172.21.15.189:0/3128065604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe71c003a00 0x7fe71c003e20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe728007bd0 tx=0x7fe71c005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e62cccc375cedf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.486+0000 7fbf9a652700 1 Processor -- start 2024-04-06T12:12:31.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.486+0000 7fe735115700 1 -- 172.21.15.189:0/3128065604 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe728010070 con 0x7fe71c003a00 2024-04-06T12:12:31.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.486+0000 7fe735115700 1 -- 172.21.15.189:0/3128065604 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe728011040 con 0x7fe71c003a00 2024-04-06T12:12:31.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.486+0000 7fe735115700 1 -- 172.21.15.189:0/3128065604 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7280157c0 con 0x7fe71c003a00 2024-04-06T12:12:31.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.487+0000 7fe73837b700 1 -- 172.21.15.189:0/3128065604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe71c003a00 msgr2=0x7fe71c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.487+0000 7fe73837b700 1 --2- 172.21.15.189:0/3128065604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe71c003a00 0x7fe71c003e20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe728007bd0 tx=0x7fe71c005880 comp rx=0 tx=0).stop 2024-04-06T12:12:31.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.486+0000 7fbf9a652700 1 -- start start 2024-04-06T12:12:31.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.487+0000 7fbf9a652700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf940cf640 0x7fbf940b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.487+0000 7fbf9a652700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbf9414b4c0 0x7fbf940b2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.487+0000 7fbf9a652700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf941620e0 con 0x7fbf9414b0e0 2024-04-06T12:12:31.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.487+0000 7fbf9a652700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf941623a0 con 0x7fbf9414b4c0 2024-04-06T12:12:31.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.487+0000 7fbf9a652700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf94162680 con 0x7fbf940cf640 2024-04-06T12:12:31.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.488+0000 7fe73837b700 1 -- 172.21.15.189:0/3128065604 shutdown_connections 2024-04-06T12:12:31.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.488+0000 7fe73837b700 1 --2- 172.21.15.189:0/3128065604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe71c003a00 0x7fe71c003e20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.488+0000 7fe73837b700 1 -- 172.21.15.189:0/3128065604 >> 172.21.15.189:0/3128065604 conn(0x7fe7300cefa0 msgr2=0x7fe7300cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:31.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.488+0000 7fbf937fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf940cf640 0x7fbf940b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.488+0000 7fbf937fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf940cf640 0x7fbf940b23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59660/0 (socket says 172.21.15.189:59660) 2024-04-06T12:12:31.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.488+0000 7fbf937fe700 1 -- 172.21.15.189:0/947500088 learned_addr learned my addr 172.21.15.189:0/947500088 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:31.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.488+0000 7fbf98bef700 1 --2- 172.21.15.189:0/947500088 >> v2:172.21.15.192:3300/0 conn(0x7fbf9414b4c0 0x7fbf940b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.488+0000 7fe73837b700 1 -- 172.21.15.189:0/3128065604 shutdown_connections 2024-04-06T12:12:31.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.488+0000 7fbf937fe700 1 -- 172.21.15.189:0/947500088 >> v2:172.21.15.192:3300/0 conn(0x7fbf9414b4c0 msgr2=0x7fbf940b2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:31.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.488+0000 7fbf937fe700 1 --2- 172.21.15.189:0/947500088 >> v2:172.21.15.192:3300/0 conn(0x7fbf9414b4c0 0x7fbf940b2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.488+0000 7fbf937fe700 1 -- 172.21.15.189:0/947500088 >> v1:172.21.15.192:6789/0 conn(0x7fbf9414b0e0 legacy=0x7fbf940cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.488+0000 7fbf937fe700 1 -- 172.21.15.189:0/947500088 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf94162960 con 0x7fbf940cf640 2024-04-06T12:12:31.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.488+0000 7fbf937fe700 1 --2- 172.21.15.189:0/947500088 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf940cf640 0x7fbf940b23c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbf840098f0 tx=0x7fbf84009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b68af4c7df93c91 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.488+0000 7fe73837b700 1 -- 172.21.15.189:0/3128065604 wait complete. 2024-04-06T12:12:31.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.489+0000 7fbf92ffd700 1 -- 172.21.15.189:0/947500088 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf84011830 con 0x7fbf940cf640 2024-04-06T12:12:31.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.489+0000 7fbf92ffd700 1 -- 172.21.15.189:0/947500088 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf84011990 con 0x7fbf940cf640 2024-04-06T12:12:31.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.489+0000 7fbf92ffd700 1 -- 172.21.15.189:0/947500088 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf84011d20 con 0x7fbf940cf640 2024-04-06T12:12:31.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.490+0000 7fe73837b700 1 Processor -- start 2024-04-06T12:12:31.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.490+0000 7fbf9a652700 1 -- 172.21.15.189:0/947500088 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf940cf640 msgr2=0x7fbf940b23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.490+0000 7fbf9a652700 1 --2- 172.21.15.189:0/947500088 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf940cf640 0x7fbf940b23c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbf840098f0 tx=0x7fbf84009c00 comp rx=0 tx=0).stop 2024-04-06T12:12:31.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.490+0000 7fe73837b700 1 -- start start 2024-04-06T12:12:31.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.491+0000 7fe73837b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe71c003a00 0x7fe73014d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.491+0000 7fe73837b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7300ce2c0 0x7fe7300ce740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.491+0000 7fe73837b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7301fa2e0 0x7fe7301fa790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.491+0000 7fe73837b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7300cec80 con 0x7fe71c003a00 2024-04-06T12:12:31.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.491+0000 7fe73837b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe730154d30 con 0x7fe7301fa2e0 2024-04-06T12:12:31.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.491+0000 7fe73837b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe730154120 con 0x7fe7300ce2c0 2024-04-06T12:12:31.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.492+0000 7fe736918700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7301fa2e0 0x7fe7301fa790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.492+0000 7fe736918700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7301fa2e0 0x7fe7301fa790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:59052/0 (socket says 172.21.15.189:59052) 2024-04-06T12:12:31.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.492+0000 7fe736918700 1 -- 172.21.15.189:0/480051481 learned_addr learned my addr 172.21.15.189:0/480051481 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:31.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.492+0000 7fe735916700 1 --2- 172.21.15.189:0/480051481 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7300ce2c0 0x7fe7300ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.492+0000 7fe736117700 1 --2- 172.21.15.189:0/480051481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe71c003a00 0x7fe73014d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.492+0000 7fe736918700 1 -- 172.21.15.189:0/480051481 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7300ce2c0 msgr2=0x7fe7300ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.492+0000 7fe736918700 1 --2- 172.21.15.189:0/480051481 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7300ce2c0 0x7fe7300ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.492+0000 7fe736918700 1 -- 172.21.15.189:0/480051481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe71c003a00 msgr2=0x7fe73014d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.492+0000 7fe736918700 1 --2- 172.21.15.189:0/480051481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe71c003a00 0x7fe73014d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.492+0000 7fe736918700 1 -- 172.21.15.189:0/480051481 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe730154400 con 0x7fe7301fa2e0 2024-04-06T12:12:31.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.493+0000 7fe736918700 1 --2- 172.21.15.189:0/480051481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7301fa2e0 0x7fe7301fa790 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe7300b4b50 tx=0x7fe72c010ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.494+0000 7fbf9a652700 1 -- 172.21.15.189:0/947500088 shutdown_connections 2024-04-06T12:12:31.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.494+0000 7fbf9a652700 1 --2- 172.21.15.189:0/947500088 >> v2:172.21.15.192:3300/0 conn(0x7fbf9414b4c0 0x7fbf940b2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.494+0000 7fbf9a652700 1 --2- 172.21.15.189:0/947500088 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf940cf640 0x7fbf940b23c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.494+0000 7fbf9a652700 1 -- 172.21.15.189:0/947500088 >> 172.21.15.189:0/947500088 conn(0x7fbf9406a730 msgr2=0x7fbf9406ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:31.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.495+0000 7fbf9a652700 1 -- 172.21.15.189:0/947500088 shutdown_connections 2024-04-06T12:12:31.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.495+0000 7fbf9a652700 1 -- 172.21.15.189:0/947500088 wait complete. 2024-04-06T12:12:31.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.497+0000 7fbf9a652700 1 Processor -- start 2024-04-06T12:12:31.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.493+0000 7fe726ffd700 1 -- 172.21.15.189:0/480051481 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c004300 con 0x7fe7301fa2e0 2024-04-06T12:12:31.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.493+0000 7fe726ffd700 1 -- 172.21.15.189:0/480051481 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72c004460 con 0x7fe7301fa2e0 2024-04-06T12:12:31.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.493+0000 7fe726ffd700 1 -- 172.21.15.189:0/480051481 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c0038e0 con 0x7fe7301fa2e0 2024-04-06T12:12:31.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.493+0000 7fe73837b700 1 -- 172.21.15.189:0/480051481 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7301546e0 con 0x7fe7301fa2e0 2024-04-06T12:12:31.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.493+0000 7fe73837b700 1 -- 172.21.15.189:0/480051481 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe73014d6e0 con 0x7fe7301fa2e0 2024-04-06T12:12:31.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.494+0000 7fe73837b700 1 -- 172.21.15.189:0/480051481 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe730074990 con 0x7fe7301fa2e0 2024-04-06T12:12:31.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.498+0000 7fe726ffd700 1 -- 172.21.15.189:0/480051481 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe72c003a40 con 0x7fe7301fa2e0 2024-04-06T12:12:31.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.498+0000 7fbf9a652700 1 -- start start 2024-04-06T12:12:31.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.499+0000 7fbf9a652700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf940cf640 0x7fbf94138dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.499+0000 7fbf9a652700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf9414b0e0 0x7fbf9414a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.499+0000 7fbf9a652700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf9414b4c0 0x7fbf9413b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.499+0000 7fbf9a652700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9414a8c0 con 0x7fbf9414b0e0 2024-04-06T12:12:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.499+0000 7fbf9a652700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf94136710 con 0x7fbf940cf640 2024-04-06T12:12:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.499+0000 7fbf9a652700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf941376e0 con 0x7fbf9414b4c0 2024-04-06T12:12:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.500+0000 7fbf937fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf9414b0e0 0x7fbf9414a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.500+0000 7fbf937fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf9414b0e0 0x7fbf9414a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38330/0 (socket says 172.21.15.189:38330) 2024-04-06T12:12:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.500+0000 7fbf937fe700 1 -- 172.21.15.189:0/1874336414 learned_addr learned my addr 172.21.15.189:0/1874336414 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.500+0000 7fbf93fff700 1 --2- 172.21.15.189:0/1874336414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf940cf640 0x7fbf94138dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.500+0000 7fbf98bef700 1 --2- 172.21.15.189:0/1874336414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf9414b4c0 0x7fbf9413b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.501+0000 7fbf98bef700 1 -- 172.21.15.189:0/1874336414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf940cf640 msgr2=0x7fbf94138dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.501+0000 7fbf98bef700 1 --2- 172.21.15.189:0/1874336414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf940cf640 0x7fbf94138dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.501+0000 7fbf98bef700 1 -- 172.21.15.189:0/1874336414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf9414b0e0 msgr2=0x7fbf9414a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.501+0000 7fbf98bef700 1 --2- 172.21.15.189:0/1874336414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf9414b0e0 0x7fbf9414a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.501+0000 7fbf98bef700 1 -- 172.21.15.189:0/1874336414 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf84009550 con 0x7fbf9414b4c0 2024-04-06T12:12:31.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.498+0000 7fe726ffd700 1 --2- 172.21.15.189:0/480051481 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe71406ace0 0x7fe71406d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.499+0000 7fe726ffd700 1 -- 172.21.15.189:0/480051481 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe72c08a540 con 0x7fe7301fa2e0 2024-04-06T12:12:31.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.502+0000 7fbf98bef700 1 --2- 172.21.15.189:0/1874336414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf9414b4c0 0x7fbf9413b980 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00eb70 tx=0x7fbf8c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.502+0000 7fe726ffd700 1 -- 172.21.15.189:0/480051481 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe72c055410 con 0x7fe7301fa2e0 2024-04-06T12:12:31.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.504+0000 7fe736117700 1 --2- 172.21.15.189:0/480051481 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe71406ace0 0x7fe71406d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.504+0000 7fe736117700 1 --2- 172.21.15.189:0/480051481 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe71406ace0 0x7fe71406d1a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe728007ba0 tx=0x7fe728010000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.505+0000 7fbf90ff9700 1 -- 172.21.15.189:0/1874336414 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c009a80 con 0x7fbf9414b4c0 2024-04-06T12:12:31.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.505+0000 7fbf90ff9700 1 -- 172.21.15.189:0/1874336414 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8c00f040 con 0x7fbf9414b4c0 2024-04-06T12:12:31.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.505+0000 7fbf90ff9700 1 -- 172.21.15.189:0/1874336414 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c00cc80 con 0x7fbf9414b4c0 2024-04-06T12:12:31.505 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.505+0000 7fbf9a652700 1 -- 172.21.15.189:0/1874336414 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9415df90 con 0x7fbf9414b4c0 2024-04-06T12:12:31.505 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.505+0000 7fbf9a652700 1 -- 172.21.15.189:0/1874336414 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf94136c00 con 0x7fbf9414b4c0 2024-04-06T12:12:31.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.508+0000 7fbf90ff9700 1 -- 172.21.15.189:0/1874336414 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbf8c014480 con 0x7fbf9414b4c0 2024-04-06T12:12:31.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.509+0000 7fbf90ff9700 1 --2- 172.21.15.189:0/1874336414 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf7806ad60 0x7fbf7806d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:31.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.509+0000 7fbf90ff9700 1 -- 172.21.15.189:0/1874336414 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbf8c089f50 con 0x7fbf9414b4c0 2024-04-06T12:12:31.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.510+0000 7fbf93fff700 1 --2- 172.21.15.189:0/1874336414 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf7806ad60 0x7fbf7806d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:31.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.510+0000 7fbf93fff700 1 --2- 172.21.15.189:0/1874336414 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf7806ad60 0x7fbf7806d220 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbf940cc9a0 tx=0x7fbf88009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:31.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.511+0000 7fbf9a652700 1 -- 172.21.15.189:0/1874336414 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf94073270 con 0x7fbf9414b4c0 2024-04-06T12:12:31.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.517+0000 7fbf90ff9700 1 -- 172.21.15.189:0/1874336414 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbf8c054e20 con 0x7fbf9414b4c0 2024-04-06T12:12:31.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.779+0000 7fed43506700 1 -- 172.21.15.189:0/2781432276 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fed20005870 con 0x7fed3c0b2300 2024-04-06T12:12:31.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.780+0000 7fed31ffb700 1 -- 172.21.15.189:0/2781432276 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fed2c030090 con 0x7fed3c0b2300 2024-04-06T12:12:31.781 INFO:teuthology.orchestra.run.smithi189.stdout:55834574856 2024-04-06T12:12:31.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.784+0000 7fed43506700 1 -- 172.21.15.189:0/2781432276 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fed2806aef0 msgr2=0x7fed2806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.784+0000 7fed43506700 1 --2- 172.21.15.189:0/2781432276 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fed2806aef0 0x7fed2806d3b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fed3400abb0 tx=0x7fed34009250 comp rx=0 tx=0).stop 2024-04-06T12:12:31.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.784+0000 7fed43506700 1 -- 172.21.15.189:0/2781432276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed3c0b2300 msgr2=0x7fed3c14b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.784+0000 7fed43506700 1 --2- 172.21.15.189:0/2781432276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed3c0b2300 0x7fed3c14b280 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fed2c017000 tx=0x7fed2c004750 comp rx=0 tx=0).stop 2024-04-06T12:12:31.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.784+0000 7fed43506700 1 -- 172.21.15.189:0/2781432276 shutdown_connections 2024-04-06T12:12:31.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.784+0000 7fed43506700 1 --2- 172.21.15.189:0/2781432276 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fed2806aef0 0x7fed2806d3b0 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fed3400abb0 tx=0x7fed34009250 comp rx=0 tx=0).stop 2024-04-06T12:12:31.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.784+0000 7fed43506700 1 --2- 172.21.15.189:0/2781432276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed3c150690 0x7fed3c08a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.784+0000 7fed43506700 1 --2- 172.21.15.189:0/2781432276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed3c06df60 0x7fed3c1f9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.784+0000 7fed43506700 1 --2- 172.21.15.189:0/2781432276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed3c0b2300 0x7fed3c14b280 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.784+0000 7fed43506700 1 -- 172.21.15.189:0/2781432276 >> 172.21.15.189:0/2781432276 conn(0x7fed3c0cefa0 msgr2=0x7fed3c14fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:31.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.784+0000 7fed43506700 1 -- 172.21.15.189:0/2781432276 shutdown_connections 2024-04-06T12:12:31.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.785+0000 7fed43506700 1 -- 172.21.15.189:0/2781432276 wait complete. 2024-04-06T12:12:31.806 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.0 2024-04-06T12:12:31.806 DEBUG:teuthology.parallel:result is None 2024-04-06T12:12:31.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.810+0000 7f5a2935a700 1 -- 172.21.15.189:0/3256631209 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f5a2406b740 con 0x7f5a24159b30 2024-04-06T12:12:31.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.811+0000 7f5a0b7fe700 1 -- 172.21.15.189:0/3256631209 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5a140579f0 con 0x7f5a24159b30 2024-04-06T12:12:31.811 INFO:teuthology.orchestra.run.smithi189.stdout:55834574858 2024-04-06T12:12:31.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.816+0000 7f5a2935a700 1 -- 172.21.15.189:0/3256631209 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5a1006ad60 msgr2=0x7f5a1006d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.816+0000 7f5a2935a700 1 --2- 172.21.15.189:0/3256631209 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5a1006ad60 0x7f5a1006d220 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5a24202aa0 tx=0x7f5a18009040 comp rx=0 tx=0).stop 2024-04-06T12:12:31.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.816+0000 7f5a2935a700 1 -- 172.21.15.189:0/3256631209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a24159b30 msgr2=0x7f5a241562f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.816+0000 7f5a2935a700 1 --2- 172.21.15.189:0/3256631209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a24159b30 0x7f5a241562f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5a1400b420 tx=0x7f5a1400b730 comp rx=0 tx=0).stop 2024-04-06T12:12:31.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.817+0000 7f5a2935a700 1 -- 172.21.15.189:0/3256631209 shutdown_connections 2024-04-06T12:12:31.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.817+0000 7f5a2935a700 1 --2- 172.21.15.189:0/3256631209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a24159b30 0x7f5a241562f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.817+0000 7f5a2935a700 1 --2- 172.21.15.189:0/3256631209 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5a1006ad60 0x7f5a1006d220 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.817+0000 7f5a2935a700 1 --2- 172.21.15.189:0/3256631209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a2406df60 0x7f5a241515d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.817+0000 7f5a2935a700 1 --2- 172.21.15.189:0/3256631209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a24150690 0x7f5a240ce7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.817+0000 7f5a2935a700 1 -- 172.21.15.189:0/3256631209 >> 172.21.15.189:0/3256631209 conn(0x7f5a240cefa0 msgr2=0x7f5a24071bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:31.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.817+0000 7f5a2935a700 1 -- 172.21.15.189:0/3256631209 shutdown_connections 2024-04-06T12:12:31.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.817+0000 7f5a2935a700 1 -- 172.21.15.189:0/3256631209 wait complete. 2024-04-06T12:12:31.832 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.2 2024-04-06T12:12:31.832 DEBUG:teuthology.parallel:result is None 2024-04-06T12:12:31.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.841+0000 7f6b08f92700 1 -- 172.21.15.189:0/3100661007 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6b040b5a40 con 0x7f6b040d0890 2024-04-06T12:12:31.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.842+0000 7f6af2ffd700 1 -- 172.21.15.189:0/3100661007 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6af40577d0 con 0x7f6b040d0890 2024-04-06T12:12:31.840 INFO:teuthology.orchestra.run.smithi189.stdout:55834574856 2024-04-06T12:12:31.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.845+0000 7f6b08f92700 1 -- 172.21.15.189:0/3100661007 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ae406ad60 msgr2=0x7f6ae406d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.845+0000 7f6b08f92700 1 --2- 172.21.15.189:0/3100661007 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ae406ad60 0x7f6ae406d220 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6b0415ed20 tx=0x7f6af8006c30 comp rx=0 tx=0).stop 2024-04-06T12:12:31.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.845+0000 7f6b08f92700 1 -- 172.21.15.189:0/3100661007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b040d0890 msgr2=0x7f6b040cb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.845+0000 7f6b08f92700 1 --2- 172.21.15.189:0/3100661007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b040d0890 0x7f6b040cb020 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6af400e810 tx=0x7f6af400ebd0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.846+0000 7f6b08f92700 1 -- 172.21.15.189:0/3100661007 shutdown_connections 2024-04-06T12:12:31.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.846+0000 7f6b08f92700 1 --2- 172.21.15.189:0/3100661007 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ae406ad60 0x7f6ae406d220 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.846+0000 7f6b08f92700 1 --2- 172.21.15.189:0/3100661007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b0415d7e0 0x7f6b040c7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.846+0000 7f6b08f92700 1 --2- 172.21.15.189:0/3100661007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b040d0890 0x7f6b040cb020 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.846+0000 7f6b08f92700 1 --2- 172.21.15.189:0/3100661007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b040d1cc0 0x7f6b04156fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.846+0000 7f6b08f92700 1 -- 172.21.15.189:0/3100661007 >> 172.21.15.189:0/3100661007 conn(0x7f6b04072250 msgr2=0x7f6b040b4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:31.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.849+0000 7f6b08f92700 1 -- 172.21.15.189:0/3100661007 shutdown_connections 2024-04-06T12:12:31.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.849+0000 7f6b08f92700 1 -- 172.21.15.189:0/3100661007 wait complete. 2024-04-06T12:12:31.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.849+0000 7f1bf7c02700 1 -- 172.21.15.189:0/4150771694 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f1bd80055a0 con 0x7f1bf0157ff0 2024-04-06T12:12:31.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.850+0000 7f1be67fc700 1 -- 172.21.15.189:0/4150771694 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1be8059860 con 0x7f1bf0157ff0 2024-04-06T12:12:31.848 INFO:teuthology.orchestra.run.smithi189.stdout:55834574857 2024-04-06T12:12:31.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.854+0000 7f1bf7c02700 1 -- 172.21.15.189:0/4150771694 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1bd406ad60 msgr2=0x7f1bd406d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.854+0000 7f1bf7c02700 1 --2- 172.21.15.189:0/4150771694 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1bd406ad60 0x7f1bd406d220 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1bec004e90 tx=0x7f1bec004b20 comp rx=0 tx=0).stop 2024-04-06T12:12:31.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.854+0000 7f1bf7c02700 1 -- 172.21.15.189:0/4150771694 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bf0157ff0 msgr2=0x7f1bf01e3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.854+0000 7f1bf7c02700 1 --2- 172.21.15.189:0/4150771694 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bf0157ff0 0x7f1bf01e3200 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1be801d040 tx=0x7f1be8009750 comp rx=0 tx=0).stop 2024-04-06T12:12:31.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.855+0000 7f1bf7c02700 1 -- 172.21.15.189:0/4150771694 shutdown_connections 2024-04-06T12:12:31.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.855+0000 7f1bf7c02700 1 --2- 172.21.15.189:0/4150771694 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bf0157ff0 0x7f1bf01e3200 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.855+0000 7f1bf7c02700 1 --2- 172.21.15.189:0/4150771694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bdc00b3a0 0x7f1bf00cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.855+0000 7f1bf7c02700 1 --2- 172.21.15.189:0/4150771694 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1bd406ad60 0x7f1bd406d220 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.855+0000 7f1bf7c02700 1 --2- 172.21.15.189:0/4150771694 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bf014d3a0 0x7f1bf0157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.855+0000 7f1bf7c02700 1 -- 172.21.15.189:0/4150771694 >> 172.21.15.189:0/4150771694 conn(0x7f1bf00cefa0 msgr2=0x7f1bf00cb1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:31.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.855+0000 7f7eb2e3e700 1 -- 172.21.15.189:0/797495531 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f7e8c005870 con 0x7f7eac0b2300 2024-04-06T12:12:31.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.855+0000 7f1bf7c02700 1 -- 172.21.15.189:0/4150771694 shutdown_connections 2024-04-06T12:12:31.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.855+0000 7f1bf7c02700 1 -- 172.21.15.189:0/4150771694 wait complete. 2024-04-06T12:12:31.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.856+0000 7f7ea97fa700 1 -- 172.21.15.189:0/797495531 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7ea0059180 con 0x7f7eac0b2300 2024-04-06T12:12:31.855 INFO:teuthology.orchestra.run.smithi189.stdout:55834574857 2024-04-06T12:12:31.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.858+0000 7f7eb2e3e700 1 -- 172.21.15.189:0/797495531 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e9806ad60 msgr2=0x7f7e9806d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.858+0000 7f7eb2e3e700 1 --2- 172.21.15.189:0/797495531 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e9806ad60 0x7f7e9806d220 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7eac0a93e0 tx=0x7f7e9c008040 comp rx=0 tx=0).stop 2024-04-06T12:12:31.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.858+0000 7f7eb2e3e700 1 -- 172.21.15.189:0/797495531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7eac0b2300 msgr2=0x7f7eac162270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.858+0000 7f7eb2e3e700 1 --2- 172.21.15.189:0/797495531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7eac0b2300 0x7f7eac162270 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0018000 tx=0x7f7ea0004750 comp rx=0 tx=0).stop 2024-04-06T12:12:31.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.858+0000 7f7eb2e3e700 1 -- 172.21.15.189:0/797495531 shutdown_connections 2024-04-06T12:12:31.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.858+0000 7f7eb2e3e700 1 --2- 172.21.15.189:0/797495531 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e9806ad60 0x7f7e9806d220 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.858+0000 7f7eb2e3e700 1 --2- 172.21.15.189:0/797495531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7eac150690 0x7f7eac1627b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.858+0000 7f7eb2e3e700 1 --2- 172.21.15.189:0/797495531 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7eac06df60 0x7f7eac14fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.858+0000 7f7eb2e3e700 1 --2- 172.21.15.189:0/797495531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7eac0b2300 0x7f7eac162270 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.858+0000 7f7eb2e3e700 1 -- 172.21.15.189:0/797495531 >> 172.21.15.189:0/797495531 conn(0x7f7eac0cefa0 msgr2=0x7f7eac0709d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:31.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.859+0000 7f7eb2e3e700 1 -- 172.21.15.189:0/797495531 shutdown_connections 2024-04-06T12:12:31.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.859+0000 7f7eb2e3e700 1 -- 172.21.15.189:0/797495531 wait complete. 2024-04-06T12:12:31.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.868+0000 7f6c23e11700 1 -- 172.21.15.189:0/2412168515 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f6c04005870 con 0x7f6c1c14b610 2024-04-06T12:12:31.867 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.1 2024-04-06T12:12:31.867 DEBUG:teuthology.parallel:result is None 2024-04-06T12:12:31.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.872+0000 7f6c127fc700 1 -- 172.21.15.189:0/2412168515 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6c18003e60 con 0x7f6c1c14b610 2024-04-06T12:12:31.870 INFO:teuthology.orchestra.run.smithi189.stdout:55834574857 2024-04-06T12:12:31.873 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.3 2024-04-06T12:12:31.874 DEBUG:teuthology.parallel:result is None 2024-04-06T12:12:31.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.875+0000 7f6c23e11700 1 -- 172.21.15.189:0/2412168515 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c0006ad60 msgr2=0x7f6c0006d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.876+0000 7f6c23e11700 1 --2- 172.21.15.189:0/2412168515 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c0006ad60 0x7f6c0006d220 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c0b5250 tx=0x7f6c14009250 comp rx=0 tx=0).stop 2024-04-06T12:12:31.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.876+0000 7f6c23e11700 1 -- 172.21.15.189:0/2412168515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c1c14b610 msgr2=0x7f6c1c149e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.876+0000 7f6c23e11700 1 --2- 172.21.15.189:0/2412168515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c1c14b610 0x7f6c1c149e20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6c18013000 tx=0x7f6c18004750 comp rx=0 tx=0).stop 2024-04-06T12:12:31.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.876+0000 7f6c23e11700 1 -- 172.21.15.189:0/2412168515 shutdown_connections 2024-04-06T12:12:31.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.876+0000 7f6c23e11700 1 --2- 172.21.15.189:0/2412168515 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c0006ad60 0x7f6c0006d220 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.876+0000 7f6c23e11700 1 --2- 172.21.15.189:0/2412168515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c1c148710 0x7f6c1c158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.876+0000 7f6c23e11700 1 --2- 172.21.15.189:0/2412168515 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c1c06df60 0x7f6c1c0cdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.876+0000 7f6c23e11700 1 --2- 172.21.15.189:0/2412168515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c1c14b610 0x7f6c1c149e20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.876+0000 7f6c23e11700 1 -- 172.21.15.189:0/2412168515 >> 172.21.15.189:0/2412168515 conn(0x7f6c1c0cefa0 msgr2=0x7f6c1c0739d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:31.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.876+0000 7f6c23e11700 1 -- 172.21.15.189:0/2412168515 shutdown_connections 2024-04-06T12:12:31.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.876+0000 7f6c23e11700 1 -- 172.21.15.189:0/2412168515 wait complete. 2024-04-06T12:12:31.876 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.5 2024-04-06T12:12:31.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.878+0000 7fbf9a652700 1 -- 172.21.15.189:0/1874336414 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fbf94138a30 con 0x7fbf9414b4c0 2024-04-06T12:12:31.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.879+0000 7fbf90ff9700 1 -- 172.21.15.189:0/1874336414 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbf8c058440 con 0x7fbf9414b4c0 2024-04-06T12:12:31.877 DEBUG:teuthology.parallel:result is None 2024-04-06T12:12:31.877 INFO:teuthology.orchestra.run.smithi189.stdout:55834574857 2024-04-06T12:12:31.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.881+0000 7fbf9a652700 1 -- 172.21.15.189:0/1874336414 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf7806ad60 msgr2=0x7fbf7806d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.881+0000 7fbf9a652700 1 --2- 172.21.15.189:0/1874336414 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf7806ad60 0x7fbf7806d220 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbf940cc9a0 tx=0x7fbf88009040 comp rx=0 tx=0).stop 2024-04-06T12:12:31.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.881+0000 7fbf9a652700 1 -- 172.21.15.189:0/1874336414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf9414b4c0 msgr2=0x7fbf9413b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.881+0000 7fbf9a652700 1 --2- 172.21.15.189:0/1874336414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf9414b4c0 0x7fbf9413b980 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00eb70 tx=0x7fbf8c00ee80 comp rx=0 tx=0).stop 2024-04-06T12:12:31.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.881+0000 7fbf9a652700 1 -- 172.21.15.189:0/1874336414 shutdown_connections 2024-04-06T12:12:31.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.881+0000 7fbf9a652700 1 --2- 172.21.15.189:0/1874336414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf9414b4c0 0x7fbf9413b980 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.881+0000 7fbf9a652700 1 --2- 172.21.15.189:0/1874336414 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf7806ad60 0x7fbf7806d220 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.881+0000 7fbf9a652700 1 --2- 172.21.15.189:0/1874336414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf940cf640 0x7fbf94138dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.881+0000 7fbf9a652700 1 --2- 172.21.15.189:0/1874336414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf9414b0e0 0x7fbf9414a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.881+0000 7fbf9a652700 1 -- 172.21.15.189:0/1874336414 >> 172.21.15.189:0/1874336414 conn(0x7fbf9406a730 msgr2=0x7fbf940b0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:31.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.881+0000 7fbf9a652700 1 -- 172.21.15.189:0/1874336414 shutdown_connections 2024-04-06T12:12:31.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.881+0000 7fbf9a652700 1 -- 172.21.15.189:0/1874336414 wait complete. 2024-04-06T12:12:31.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.888+0000 7fe73837b700 1 -- 172.21.15.189:0/480051481 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fe7300b1e90 con 0x7fe7301fa2e0 2024-04-06T12:12:31.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.888+0000 7fe726ffd700 1 -- 172.21.15.189:0/480051481 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe72c058a30 con 0x7fe7301fa2e0 2024-04-06T12:12:31.886 INFO:teuthology.orchestra.run.smithi189.stdout:55834574858 2024-04-06T12:12:31.887 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.4 2024-04-06T12:12:31.887 DEBUG:teuthology.parallel:result is None 2024-04-06T12:12:31.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.891+0000 7fe73837b700 1 -- 172.21.15.189:0/480051481 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe71406ace0 msgr2=0x7fe71406d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.891+0000 7fe73837b700 1 --2- 172.21.15.189:0/480051481 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe71406ace0 0x7fe71406d1a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe728007ba0 tx=0x7fe728010000 comp rx=0 tx=0).stop 2024-04-06T12:12:31.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.891+0000 7fe73837b700 1 -- 172.21.15.189:0/480051481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7301fa2e0 msgr2=0x7fe7301fa790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:31.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.891+0000 7fe73837b700 1 --2- 172.21.15.189:0/480051481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7301fa2e0 0x7fe7301fa790 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe7300b4b50 tx=0x7fe72c010ad0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.891+0000 7fe73837b700 1 -- 172.21.15.189:0/480051481 shutdown_connections 2024-04-06T12:12:31.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.891+0000 7fe73837b700 1 --2- 172.21.15.189:0/480051481 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe71406ace0 0x7fe71406d1a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.891+0000 7fe73837b700 1 --2- 172.21.15.189:0/480051481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7301fa2e0 0x7fe7301fa790 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.891+0000 7fe73837b700 1 --2- 172.21.15.189:0/480051481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe71c003a00 0x7fe73014d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.891+0000 7fe73837b700 1 --2- 172.21.15.189:0/480051481 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7300ce2c0 0x7fe7300ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:31.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.891+0000 7fe73837b700 1 -- 172.21.15.189:0/480051481 >> 172.21.15.189:0/480051481 conn(0x7fe7300cefa0 msgr2=0x7fe7300706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:31.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.891+0000 7fe73837b700 1 -- 172.21.15.189:0/480051481 shutdown_connections 2024-04-06T12:12:31.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:31.891+0000 7fe73837b700 1 -- 172.21.15.189:0/480051481 wait complete. 2024-04-06T12:12:31.893 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2024-04-06T12:12:31.893 DEBUG:teuthology.parallel:result is None 2024-04-06T12:12:31.901 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.7 2024-04-06T12:12:31.902 DEBUG:teuthology.parallel:result is None 2024-04-06T12:12:31.902 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-06T12:12:31.902 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:12:32.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.065+0000 7fa063651700 1 Processor -- start 2024-04-06T12:12:32.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.065+0000 7fa063651700 1 -- start start 2024-04-06T12:12:32.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.066+0000 7fa063651700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa05c161980 0x7fa05c0cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:32.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.066+0000 7fa063651700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa05c161d60 con 0x7fa05c06b050 2024-04-06T12:12:32.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.066+0000 7fa063651700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa05c1662a0 con 0x7fa05c0c2a70 2024-04-06T12:12:32.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.066+0000 7fa063651700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa05c06c130 con 0x7fa05c161980 2024-04-06T12:12:32.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.066+0000 7fa0613ed700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa05c161980 0x7fa05c0cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:32.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.066+0000 7fa061bee700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa05c06b050 0x7fa05c165b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34332/0 (socket says 172.21.15.189:34332) 2024-04-06T12:12:32.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.066+0000 7fa061bee700 1 -- 172.21.15.189:0/1387985852 learned_addr learned my addr 172.21.15.189:0/1387985852 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:32.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.067+0000 7fa053fff700 1 -- 172.21.15.189:0/1387985852 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828998758 0 0) 0x7fa054001490 con 0x7fa05c06b050 2024-04-06T12:12:32.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.067+0000 7fa053fff700 1 -- 172.21.15.189:0/1387985852 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0480035b0 con 0x7fa05c06b050 2024-04-06T12:12:32.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.067+0000 7fa0613ed700 1 -- 172.21.15.189:0/1387985852 >> v1:172.21.15.192:6789/0 conn(0x7fa05c0c2a70 legacy=0x7fa05c165480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:32.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.067+0000 7fa0613ed700 1 -- 172.21.15.189:0/1387985852 >> v1:172.21.15.189:6789/0 conn(0x7fa05c06b050 legacy=0x7fa05c165b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:32.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.067+0000 7fa0613ed700 1 -- 172.21.15.189:0/1387985852 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c06b5b0 con 0x7fa05c161980 2024-04-06T12:12:32.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.067+0000 7fa0613ed700 1 --2- 172.21.15.189:0/1387985852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa05c161980 0x7fa05c0cd3e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa04c009840 tx=0x7fa04c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbdc6d8a59c8fb8c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:32.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.067+0000 7fa053fff700 1 -- 172.21.15.189:0/1387985852 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c011830 con 0x7fa05c161980 2024-04-06T12:12:32.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.067+0000 7fa053fff700 1 -- 172.21.15.189:0/1387985852 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04c011990 con 0x7fa05c161980 2024-04-06T12:12:32.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.067+0000 7fa053fff700 1 -- 172.21.15.189:0/1387985852 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c011d20 con 0x7fa05c161980 2024-04-06T12:12:32.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.068+0000 7fa063651700 1 -- 172.21.15.189:0/1387985852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa05c161980 msgr2=0x7fa05c0cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:32.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.068+0000 7fa063651700 1 --2- 172.21.15.189:0/1387985852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa05c161980 0x7fa05c0cd3e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa04c009840 tx=0x7fa04c009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:32.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.068+0000 7fa063651700 1 -- 172.21.15.189:0/1387985852 shutdown_connections 2024-04-06T12:12:32.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.068+0000 7fa063651700 1 --2- 172.21.15.189:0/1387985852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa05c161980 0x7fa05c0cd3e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.068+0000 7fa063651700 1 -- 172.21.15.189:0/1387985852 >> 172.21.15.189:0/1387985852 conn(0x7fa05c0cc290 msgr2=0x7fa05c0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:32.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.068+0000 7fa063651700 1 -- 172.21.15.189:0/1387985852 shutdown_connections 2024-04-06T12:12:32.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.068+0000 7fa063651700 1 -- 172.21.15.189:0/1387985852 wait complete. 2024-04-06T12:12:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.069+0000 7fa063651700 1 Processor -- start 2024-04-06T12:12:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.069+0000 7fa063651700 1 -- start start 2024-04-06T12:12:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.070+0000 7fa063651700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa05c06b050 0x7fa05c156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.070+0000 7fa063651700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa05c0c2a70 0x7fa05c0c58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.070+0000 7fa063651700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa05c161980 0x7fa05c154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.070+0000 7fa063651700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c156420 con 0x7fa05c06b050 2024-04-06T12:12:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.070+0000 7fa063651700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c157380 con 0x7fa05c0c2a70 2024-04-06T12:12:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.070+0000 7fa063651700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c157660 con 0x7fa05c161980 2024-04-06T12:12:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.070+0000 7fa0613ed700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa05c06b050 0x7fa05c156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.070+0000 7fa0613ed700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa05c06b050 0x7fa05c156ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38346/0 (socket says 172.21.15.189:38346) 2024-04-06T12:12:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.070+0000 7fa0613ed700 1 -- 172.21.15.189:0/2939853083 learned_addr learned my addr 172.21.15.189:0/2939853083 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.070+0000 7fa060bec700 1 --2- 172.21.15.189:0/2939853083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa05c0c2a70 0x7fa05c0c58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.071+0000 7fa061bee700 1 --2- 172.21.15.189:0/2939853083 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa05c161980 0x7fa05c154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.071+0000 7fa0613ed700 1 -- 172.21.15.189:0/2939853083 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa05c161980 msgr2=0x7fa05c154f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.071+0000 7fa0613ed700 1 --2- 172.21.15.189:0/2939853083 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa05c161980 0x7fa05c154f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.071+0000 7fa0613ed700 1 -- 172.21.15.189:0/2939853083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa05c0c2a70 msgr2=0x7fa05c0c58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.071+0000 7fa0613ed700 1 --2- 172.21.15.189:0/2939853083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa05c0c2a70 0x7fa05c0c58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.071+0000 7fa0613ed700 1 -- 172.21.15.189:0/2939853083 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04c0094a0 con 0x7fa05c06b050 2024-04-06T12:12:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.071+0000 7fa061bee700 1 --2- 172.21.15.189:0/2939853083 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa05c161980 0x7fa05c154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.071+0000 7fa0613ed700 1 --2- 172.21.15.189:0/2939853083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa05c06b050 0x7fa05c156ca0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa04c009b90 tx=0x7fa04c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.071+0000 7fa051ffb700 1 -- 172.21.15.189:0/2939853083 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c0038a0 con 0x7fa05c06b050 2024-04-06T12:12:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.071+0000 7fa051ffb700 1 -- 172.21.15.189:0/2939853083 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04c01e070 con 0x7fa05c06b050 2024-04-06T12:12:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.071+0000 7fa051ffb700 1 -- 172.21.15.189:0/2939853083 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c003c40 con 0x7fa05c06b050 2024-04-06T12:12:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.071+0000 7fa063651700 1 -- 172.21.15.189:0/2939853083 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa05c157940 con 0x7fa05c06b050 2024-04-06T12:12:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.071+0000 7fa063651700 1 -- 172.21.15.189:0/2939853083 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa05c157f00 con 0x7fa05c06b050 2024-04-06T12:12:32.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.072+0000 7fa063651700 1 -- 172.21.15.189:0/2939853083 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa05c0af040 con 0x7fa05c06b050 2024-04-06T12:12:32.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.072+0000 7fa051ffb700 1 -- 172.21.15.189:0/2939853083 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa04c02d450 con 0x7fa05c06b050 2024-04-06T12:12:32.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.073+0000 7fa051ffb700 1 --2- 172.21.15.189:0/2939853083 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa04006ac90 0x7fa04006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:32.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.073+0000 7fa051ffb700 1 -- 172.21.15.189:0/2939853083 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa04c08a5c0 con 0x7fa05c06b050 2024-04-06T12:12:32.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.073+0000 7fa060bec700 1 --2- 172.21.15.189:0/2939853083 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa04006ac90 0x7fa04006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:32.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.073+0000 7fa060bec700 1 --2- 172.21.15.189:0/2939853083 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa04006ac90 0x7fa04006d150 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa05c0c55c0 tx=0x7fa058009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:32.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.077+0000 7fa051ffb700 1 -- 172.21.15.189:0/2939853083 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa04c055540 con 0x7fa05c06b050 2024-04-06T12:12:32.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.272+0000 7fa063651700 1 -- 172.21.15.189:0/2939853083 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa05c06be50 con 0x7fa04006ac90 2024-04-06T12:12:32.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.282+0000 7fa051ffb700 1 -- 172.21.15.189:0/2939853083 <== mgr.4100 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345214 (secure 0 0 0) 0x7fa058001490 con 0x7fa04006ac90 2024-04-06T12:12:32.284 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:32.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.285+0000 7fa063651700 1 -- 172.21.15.189:0/2939853083 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa04006ac90 msgr2=0x7fa04006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:32.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.285+0000 7fa063651700 1 --2- 172.21.15.189:0/2939853083 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa04006ac90 0x7fa04006d150 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa05c0c55c0 tx=0x7fa058009040 comp rx=0 tx=0).stop 2024-04-06T12:12:32.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.285+0000 7fa063651700 1 -- 172.21.15.189:0/2939853083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa05c06b050 msgr2=0x7fa05c156ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:32.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.285+0000 7fa063651700 1 --2- 172.21.15.189:0/2939853083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa05c06b050 0x7fa05c156ca0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa04c009b90 tx=0x7fa04c004750 comp rx=0 tx=0).stop 2024-04-06T12:12:32.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.285+0000 7fa063651700 1 -- 172.21.15.189:0/2939853083 shutdown_connections 2024-04-06T12:12:32.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.285+0000 7fa063651700 1 --2- 172.21.15.189:0/2939853083 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa05c161980 0x7fa05c154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.285+0000 7fa063651700 1 --2- 172.21.15.189:0/2939853083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa05c06b050 0x7fa05c156ca0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.285+0000 7fa063651700 1 --2- 172.21.15.189:0/2939853083 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa04006ac90 0x7fa04006d150 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.285+0000 7fa063651700 1 --2- 172.21.15.189:0/2939853083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa05c0c2a70 0x7fa05c0c58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.285+0000 7fa063651700 1 -- 172.21.15.189:0/2939853083 >> 172.21.15.189:0/2939853083 conn(0x7fa05c0cc290 msgr2=0x7fa05c0b3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:32.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.285+0000 7fa063651700 1 -- 172.21.15.189:0/2939853083 shutdown_connections 2024-04-06T12:12:32.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.285+0000 7fa063651700 1 -- 172.21.15.189:0/2939853083 wait complete. 2024-04-06T12:12:32.287 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:12:32.295 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-04-06T12:12:30.571354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594962,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219360,"kb_used_data":3188,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755360,"statfs":{"total":773094113280,"available":772869488640,"internally_reserved":0,"allocated":3264512,"data_stored":1983069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003289"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224930+0000","last_change":"2024-04-06T12:12:04.226505+0000","last_active":"2024-04-06T12:12:08.224930+0000","last_peered":"2024-04-06T12:12:08.224930+0000","last_clean":"2024-04-06T12:12:08.224930+0000","last_became_active":"2024-04-06T12:12:04.226112+0000","last_became_peered":"2024-04-06T12:12:04.226112+0000","last_unstale":"2024-04-06T12:12:08.224930+0000","last_undegraded":"2024-04-06T12:12:08.224930+0000","last_fullsized":"2024-04-06T12:12:08.224930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:09:38.014527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223002+0000","last_change":"2024-04-06T12:12:04.217545+0000","last_active":"2024-04-06T12:12:08.223002+0000","last_peered":"2024-04-06T12:12:08.223002+0000","last_clean":"2024-04-06T12:12:08.223002+0000","last_became_active":"2024-04-06T12:12:04.217304+0000","last_became_peered":"2024-04-06T12:12:04.217304+0000","last_unstale":"2024-04-06T12:12:08.223002+0000","last_undegraded":"2024-04-06T12:12:08.223002+0000","last_fullsized":"2024-04-06T12:12:08.223002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.231018+0000","last_change":"2024-04-06T12:12:04.209328+0000","last_active":"2024-04-06T12:12:08.231018+0000","last_peered":"2024-04-06T12:12:08.231018+0000","last_clean":"2024-04-06T12:12:08.231018+0000","last_became_active":"2024-04-06T12:12:04.209066+0000","last_became_peered":"2024-04-06T12:12:04.209066+0000","last_unstale":"2024-04-06T12:12:08.231018+0000","last_undegraded":"2024-04-06T12:12:08.231018+0000","last_fullsized":"2024-04-06T12:12:08.231018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:02:53.621993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227488+0000","last_change":"2024-04-06T12:12:04.221440+0000","last_active":"2024-04-06T12:12:08.227488+0000","last_peered":"2024-04-06T12:12:08.227488+0000","last_clean":"2024-04-06T12:12:08.227488+0000","last_became_active":"2024-04-06T12:12:04.220738+0000","last_became_peered":"2024-04-06T12:12:04.220738+0000","last_unstale":"2024-04-06T12:12:08.227488+0000","last_undegraded":"2024-04-06T12:12:08.227488+0000","last_fullsized":"2024-04-06T12:12:08.227488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:43:39.601900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223982+0000","last_change":"2024-04-06T12:12:04.223078+0000","last_active":"2024-04-06T12:12:08.223982+0000","last_peered":"2024-04-06T12:12:08.223982+0000","last_clean":"2024-04-06T12:12:08.223982+0000","last_became_active":"2024-04-06T12:12:04.222831+0000","last_became_peered":"2024-04-06T12:12:04.222831+0000","last_unstale":"2024-04-06T12:12:08.223982+0000","last_undegraded":"2024-04-06T12:12:08.223982+0000","last_fullsized":"2024-04-06T12:12:08.223982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:02:00.219999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227898+0000","last_change":"2024-04-06T12:12:04.215126+0000","last_active":"2024-04-06T12:12:08.227898+0000","last_peered":"2024-04-06T12:12:08.227898+0000","last_clean":"2024-04-06T12:12:08.227898+0000","last_became_active":"2024-04-06T12:12:04.214857+0000","last_became_peered":"2024-04-06T12:12:04.214857+0000","last_unstale":"2024-04-06T12:12:08.227898+0000","last_undegraded":"2024-04-06T12:12:08.227898+0000","last_fullsized":"2024-04-06T12:12:08.227898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:02:00.219999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223337+0000","last_change":"2024-04-06T12:12:04.222961+0000","last_active":"2024-04-06T12:12:08.223337+0000","last_peered":"2024-04-06T12:12:08.223337+0000","last_clean":"2024-04-06T12:12:08.223337+0000","last_became_active":"2024-04-06T12:12:04.222659+0000","last_became_peered":"2024-04-06T12:12:04.222659+0000","last_unstale":"2024-04-06T12:12:08.223337+0000","last_undegraded":"2024-04-06T12:12:08.223337+0000","last_fullsized":"2024-04-06T12:12:08.223337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:55.517965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.233016+0000","last_change":"2024-04-06T12:12:04.222214+0000","last_active":"2024-04-06T12:12:08.233016+0000","last_peered":"2024-04-06T12:12:08.233016+0000","last_clean":"2024-04-06T12:12:08.233016+0000","last_became_active":"2024-04-06T12:12:04.221817+0000","last_became_peered":"2024-04-06T12:12:04.221817+0000","last_unstale":"2024-04-06T12:12:08.233016+0000","last_undegraded":"2024-04-06T12:12:08.233016+0000","last_fullsized":"2024-04-06T12:12:08.233016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224843+0000","last_change":"2024-04-06T12:12:04.224106+0000","last_active":"2024-04-06T12:12:08.224843+0000","last_peered":"2024-04-06T12:12:08.224843+0000","last_clean":"2024-04-06T12:12:08.224843+0000","last_became_active":"2024-04-06T12:12:04.223653+0000","last_became_peered":"2024-04-06T12:12:04.223653+0000","last_unstale":"2024-04-06T12:12:08.224843+0000","last_undegraded":"2024-04-06T12:12:08.224843+0000","last_fullsized":"2024-04-06T12:12:08.224843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:50:27.596378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223873+0000","last_change":"2024-04-06T12:12:04.223892+0000","last_active":"2024-04-06T12:12:08.223873+0000","last_peered":"2024-04-06T12:12:08.223873+0000","last_clean":"2024-04-06T12:12:08.223873+0000","last_became_active":"2024-04-06T12:12:04.223615+0000","last_became_peered":"2024-04-06T12:12:04.223615+0000","last_unstale":"2024-04-06T12:12:08.223873+0000","last_undegraded":"2024-04-06T12:12:08.223873+0000","last_fullsized":"2024-04-06T12:12:08.223873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:02:53.621993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.225163+0000","last_change":"2024-04-06T12:12:04.223861+0000","last_active":"2024-04-06T12:12:08.225163+0000","last_peered":"2024-04-06T12:12:08.225163+0000","last_clean":"2024-04-06T12:12:08.225163+0000","last_became_active":"2024-04-06T12:12:04.223284+0000","last_became_peered":"2024-04-06T12:12:04.223284+0000","last_unstale":"2024-04-06T12:12:08.225163+0000","last_undegraded":"2024-04-06T12:12:08.225163+0000","last_fullsized":"2024-04-06T12:12:08.225163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:00.258491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232662+0000","last_change":"2024-04-06T12:12:04.230879+0000","last_active":"2024-04-06T12:12:08.232662+0000","last_peered":"2024-04-06T12:12:08.232662+0000","last_clean":"2024-04-06T12:12:08.232662+0000","last_became_active":"2024-04-06T12:12:04.230592+0000","last_became_peered":"2024-04-06T12:12:04.230592+0000","last_unstale":"2024-04-06T12:12:08.232662+0000","last_undegraded":"2024-04-06T12:12:08.232662+0000","last_fullsized":"2024-04-06T12:12:08.232662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:10.512788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232539+0000","last_change":"2024-04-06T12:12:04.218704+0000","last_active":"2024-04-06T12:12:08.232539+0000","last_peered":"2024-04-06T12:12:08.232539+0000","last_clean":"2024-04-06T12:12:08.232539+0000","last_became_active":"2024-04-06T12:12:04.218455+0000","last_became_peered":"2024-04-06T12:12:04.218455+0000","last_unstale":"2024-04-06T12:12:08.232539+0000","last_undegraded":"2024-04-06T12:12:08.232539+0000","last_fullsized":"2024-04-06T12:12:08.232539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:21:16.167775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.235340+0000","last_change":"2024-04-06T12:12:04.209351+0000","last_active":"2024-04-06T12:12:08.235340+0000","last_peered":"2024-04-06T12:12:08.235340+0000","last_clean":"2024-04-06T12:12:08.235340+0000","last_became_active":"2024-04-06T12:12:04.209092+0000","last_became_peered":"2024-04-06T12:12:04.209092+0000","last_unstale":"2024-04-06T12:12:08.235340+0000","last_undegraded":"2024-04-06T12:12:08.235340+0000","last_fullsized":"2024-04-06T12:12:08.235340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222602+0000","last_change":"2024-04-06T12:12:04.219642+0000","last_active":"2024-04-06T12:12:08.222602+0000","last_peered":"2024-04-06T12:12:08.222602+0000","last_clean":"2024-04-06T12:12:08.222602+0000","last_became_active":"2024-04-06T12:12:04.219398+0000","last_became_peered":"2024-04-06T12:12:04.219398+0000","last_unstale":"2024-04-06T12:12:08.222602+0000","last_undegraded":"2024-04-06T12:12:08.222602+0000","last_fullsized":"2024-04-06T12:12:08.222602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227279+0000","last_change":"2024-04-06T12:12:04.215338+0000","last_active":"2024-04-06T12:12:08.227279+0000","last_peered":"2024-04-06T12:12:08.227279+0000","last_clean":"2024-04-06T12:12:08.227279+0000","last_became_active":"2024-04-06T12:12:04.215021+0000","last_became_peered":"2024-04-06T12:12:04.215021+0000","last_unstale":"2024-04-06T12:12:08.227279+0000","last_undegraded":"2024-04-06T12:12:08.227279+0000","last_fullsized":"2024-04-06T12:12:08.227279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:09:38.014527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.229099+0000","last_change":"2024-04-06T12:12:04.225406+0000","last_active":"2024-04-06T12:12:08.229099+0000","last_peered":"2024-04-06T12:12:08.229099+0000","last_clean":"2024-04-06T12:12:08.229099+0000","last_became_active":"2024-04-06T12:12:04.224950+0000","last_became_peered":"2024-04-06T12:12:04.224950+0000","last_unstale":"2024-04-06T12:12:08.229099+0000","last_undegraded":"2024-04-06T12:12:08.229099+0000","last_fullsized":"2024-04-06T12:12:08.229099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:55.517965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228664+0000","last_change":"2024-04-06T12:12:04.217691+0000","last_active":"2024-04-06T12:12:08.228664+0000","last_peered":"2024-04-06T12:12:08.228664+0000","last_clean":"2024-04-06T12:12:08.228664+0000","last_became_active":"2024-04-06T12:12:04.217425+0000","last_became_peered":"2024-04-06T12:12:04.217425+0000","last_unstale":"2024-04-06T12:12:08.228664+0000","last_undegraded":"2024-04-06T12:12:08.228664+0000","last_fullsized":"2024-04-06T12:12:08.228664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:10.512788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224434+0000","last_change":"2024-04-06T12:12:04.217807+0000","last_active":"2024-04-06T12:12:08.224434+0000","last_peered":"2024-04-06T12:12:08.224434+0000","last_clean":"2024-04-06T12:12:08.224434+0000","last_became_active":"2024-04-06T12:12:04.217533+0000","last_became_peered":"2024-04-06T12:12:04.217533+0000","last_unstale":"2024-04-06T12:12:08.224434+0000","last_undegraded":"2024-04-06T12:12:08.224434+0000","last_fullsized":"2024-04-06T12:12:08.224434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:43:39.601900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247241+0000","last_change":"2024-04-06T12:12:08.242261+0000","last_active":"2024-04-06T12:12:08.247241+0000","last_peered":"2024-04-06T12:12:08.247241+0000","last_clean":"2024-04-06T12:12:08.247241+0000","last_became_active":"2024-04-06T12:12:08.241896+0000","last_became_peered":"2024-04-06T12:12:08.241896+0000","last_unstale":"2024-04-06T12:12:08.247241+0000","last_undegraded":"2024-04-06T12:12:08.247241+0000","last_fullsized":"2024-04-06T12:12:08.247241+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:31:31.349113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232576+0000","last_change":"2024-04-06T12:12:03.799562+0000","last_active":"2024-04-06T12:12:08.232576+0000","last_peered":"2024-04-06T12:12:08.232576+0000","last_clean":"2024-04-06T12:12:08.232576+0000","last_became_active":"2024-04-06T12:12:03.799298+0000","last_became_peered":"2024-04-06T12:12:03.799298+0000","last_unstale":"2024-04-06T12:12:08.232576+0000","last_undegraded":"2024-04-06T12:12:08.232576+0000","last_fullsized":"2024-04-06T12:12:08.232576+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:05:14.778375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232962+0000","last_change":"2024-04-06T12:12:04.217203+0000","last_active":"2024-04-06T12:12:08.232962+0000","last_peered":"2024-04-06T12:12:08.232962+0000","last_clean":"2024-04-06T12:12:08.232962+0000","last_became_active":"2024-04-06T12:12:04.216782+0000","last_became_peered":"2024-04-06T12:12:04.216782+0000","last_unstale":"2024-04-06T12:12:08.232962+0000","last_undegraded":"2024-04-06T12:12:08.232962+0000","last_fullsized":"2024-04-06T12:12:08.232962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:55.517965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.255247+0000","last_change":"2024-04-06T12:12:08.251632+0000","last_active":"2024-04-06T12:12:08.255247+0000","last_peered":"2024-04-06T12:12:08.255247+0000","last_clean":"2024-04-06T12:12:08.255247+0000","last_became_active":"2024-04-06T12:12:08.251238+0000","last_became_peered":"2024-04-06T12:12:08.251238+0000","last_unstale":"2024-04-06T12:12:08.255247+0000","last_undegraded":"2024-04-06T12:12:08.255247+0000","last_fullsized":"2024-04-06T12:12:08.255247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:32:02.566075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.233059+0000","last_change":"2024-04-06T12:12:03.792714+0000","last_active":"2024-04-06T12:12:08.233059+0000","last_peered":"2024-04-06T12:12:08.233059+0000","last_clean":"2024-04-06T12:12:08.233059+0000","last_became_active":"2024-04-06T12:12:03.792445+0000","last_became_peered":"2024-04-06T12:12:03.792445+0000","last_unstale":"2024-04-06T12:12:08.233059+0000","last_undegraded":"2024-04-06T12:12:08.233059+0000","last_fullsized":"2024-04-06T12:12:08.233059+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:40:20.784453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232461+0000","last_change":"2024-04-06T12:12:04.215439+0000","last_active":"2024-04-06T12:12:08.232461+0000","last_peered":"2024-04-06T12:12:08.232461+0000","last_clean":"2024-04-06T12:12:08.232461+0000","last_became_active":"2024-04-06T12:12:04.214846+0000","last_became_peered":"2024-04-06T12:12:04.214846+0000","last_unstale":"2024-04-06T12:12:08.232461+0000","last_undegraded":"2024-04-06T12:12:08.232461+0000","last_fullsized":"2024-04-06T12:12:08.232461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.250137+0000","last_change":"2024-04-06T12:12:08.248890+0000","last_active":"2024-04-06T12:12:08.250137+0000","last_peered":"2024-04-06T12:12:08.250137+0000","last_clean":"2024-04-06T12:12:08.250137+0000","last_became_active":"2024-04-06T12:12:08.248470+0000","last_became_peered":"2024-04-06T12:12:08.248470+0000","last_unstale":"2024-04-06T12:12:08.250137+0000","last_undegraded":"2024-04-06T12:12:08.250137+0000","last_fullsized":"2024-04-06T12:12:08.250137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:06.085793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223703+0000","last_change":"2024-04-06T12:12:03.799557+0000","last_active":"2024-04-06T12:12:08.223703+0000","last_peered":"2024-04-06T12:12:08.223703+0000","last_clean":"2024-04-06T12:12:08.223703+0000","last_became_active":"2024-04-06T12:12:03.799271+0000","last_became_peered":"2024-04-06T12:12:03.799271+0000","last_unstale":"2024-04-06T12:12:08.223703+0000","last_undegraded":"2024-04-06T12:12:08.223703+0000","last_fullsized":"2024-04-06T12:12:08.223703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:26:21.506491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227705+0000","last_change":"2024-04-06T12:12:04.215431+0000","last_active":"2024-04-06T12:12:08.227705+0000","last_peered":"2024-04-06T12:12:08.227705+0000","last_clean":"2024-04-06T12:12:08.227705+0000","last_became_active":"2024-04-06T12:12:04.214459+0000","last_became_peered":"2024-04-06T12:12:04.214459+0000","last_unstale":"2024-04-06T12:12:08.227705+0000","last_undegraded":"2024-04-06T12:12:08.227705+0000","last_fullsized":"2024-04-06T12:12:08.227705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.252843+0000","last_change":"2024-04-06T12:12:08.249210+0000","last_active":"2024-04-06T12:12:08.252843+0000","last_peered":"2024-04-06T12:12:08.252843+0000","last_clean":"2024-04-06T12:12:08.252843+0000","last_became_active":"2024-04-06T12:12:08.248661+0000","last_became_peered":"2024-04-06T12:12:08.248661+0000","last_unstale":"2024-04-06T12:12:08.252843+0000","last_undegraded":"2024-04-06T12:12:08.252843+0000","last_fullsized":"2024-04-06T12:12:08.252843+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:29.953853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224295+0000","last_change":"2024-04-06T12:12:03.800006+0000","last_active":"2024-04-06T12:12:08.224295+0000","last_peered":"2024-04-06T12:12:08.224295+0000","last_clean":"2024-04-06T12:12:08.224295+0000","last_became_active":"2024-04-06T12:12:03.799598+0000","last_became_peered":"2024-04-06T12:12:03.799598+0000","last_unstale":"2024-04-06T12:12:08.224295+0000","last_undegraded":"2024-04-06T12:12:08.224295+0000","last_fullsized":"2024-04-06T12:12:08.224295+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:04.938049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.229419+0000","last_change":"2024-04-06T12:12:04.220105+0000","last_active":"2024-04-06T12:12:08.229419+0000","last_peered":"2024-04-06T12:12:08.229419+0000","last_clean":"2024-04-06T12:12:08.229419+0000","last_became_active":"2024-04-06T12:12:04.219876+0000","last_became_peered":"2024-04-06T12:12:04.219876+0000","last_unstale":"2024-04-06T12:12:08.229419+0000","last_undegraded":"2024-04-06T12:12:08.229419+0000","last_fullsized":"2024-04-06T12:12:08.229419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.249020+0000","last_change":"2024-04-06T12:12:08.246410+0000","last_active":"2024-04-06T12:12:08.249020+0000","last_peered":"2024-04-06T12:12:08.249020+0000","last_clean":"2024-04-06T12:12:08.249020+0000","last_became_active":"2024-04-06T12:12:08.246057+0000","last_became_peered":"2024-04-06T12:12:08.246057+0000","last_unstale":"2024-04-06T12:12:08.249020+0000","last_undegraded":"2024-04-06T12:12:08.249020+0000","last_fullsized":"2024-04-06T12:12:08.249020+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:16.183719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232695+0000","last_change":"2024-04-06T12:12:03.798850+0000","last_active":"2024-04-06T12:12:08.232695+0000","last_peered":"2024-04-06T12:12:08.232695+0000","last_clean":"2024-04-06T12:12:08.232695+0000","last_became_active":"2024-04-06T12:12:03.798550+0000","last_became_peered":"2024-04-06T12:12:03.798550+0000","last_unstale":"2024-04-06T12:12:08.232695+0000","last_undegraded":"2024-04-06T12:12:08.232695+0000","last_fullsized":"2024-04-06T12:12:08.232695+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:05:14.778375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232223+0000","last_change":"2024-04-06T12:12:04.212276+0000","last_active":"2024-04-06T12:12:08.232223+0000","last_peered":"2024-04-06T12:12:08.232223+0000","last_clean":"2024-04-06T12:12:08.232223+0000","last_became_active":"2024-04-06T12:12:04.211681+0000","last_became_peered":"2024-04-06T12:12:04.211681+0000","last_unstale":"2024-04-06T12:12:08.232223+0000","last_undegraded":"2024-04-06T12:12:08.232223+0000","last_fullsized":"2024-04-06T12:12:08.232223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:21:43.141955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.248388+0000","last_change":"2024-04-06T12:12:08.244209+0000","last_active":"2024-04-06T12:12:08.248388+0000","last_peered":"2024-04-06T12:12:08.248388+0000","last_clean":"2024-04-06T12:12:08.248388+0000","last_became_active":"2024-04-06T12:12:08.243274+0000","last_became_peered":"2024-04-06T12:12:08.243274+0000","last_unstale":"2024-04-06T12:12:08.248388+0000","last_undegraded":"2024-04-06T12:12:08.248388+0000","last_fullsized":"2024-04-06T12:12:08.248388+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:03.696838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223468+0000","last_change":"2024-04-06T12:12:03.796799+0000","last_active":"2024-04-06T12:12:08.223468+0000","last_peered":"2024-04-06T12:12:08.223468+0000","last_clean":"2024-04-06T12:12:08.223468+0000","last_became_active":"2024-04-06T12:12:03.796536+0000","last_became_peered":"2024-04-06T12:12:03.796536+0000","last_unstale":"2024-04-06T12:12:08.223468+0000","last_undegraded":"2024-04-06T12:12:08.223468+0000","last_fullsized":"2024-04-06T12:12:08.223468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:15.699305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234439+0000","last_change":"2024-04-06T12:12:04.220607+0000","last_active":"2024-04-06T12:12:08.234439+0000","last_peered":"2024-04-06T12:12:08.234439+0000","last_clean":"2024-04-06T12:12:08.234439+0000","last_became_active":"2024-04-06T12:12:04.220242+0000","last_became_peered":"2024-04-06T12:12:04.220242+0000","last_unstale":"2024-04-06T12:12:08.234439+0000","last_undegraded":"2024-04-06T12:12:08.234439+0000","last_fullsized":"2024-04-06T12:12:08.234439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:00.258491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.243307+0000","last_change":"2024-04-06T12:12:08.237694+0000","last_active":"2024-04-06T12:12:08.243307+0000","last_peered":"2024-04-06T12:12:08.243307+0000","last_clean":"2024-04-06T12:12:08.243307+0000","last_became_active":"2024-04-06T12:12:08.237190+0000","last_became_peered":"2024-04-06T12:12:08.237190+0000","last_unstale":"2024-04-06T12:12:08.243307+0000","last_undegraded":"2024-04-06T12:12:08.243307+0000","last_fullsized":"2024-04-06T12:12:08.243307+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:50:32.228428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232240+0000","last_change":"2024-04-06T12:12:03.800903+0000","last_active":"2024-04-06T12:12:08.232240+0000","last_peered":"2024-04-06T12:12:08.232240+0000","last_clean":"2024-04-06T12:12:08.232240+0000","last_became_active":"2024-04-06T12:12:03.800590+0000","last_became_peered":"2024-04-06T12:12:03.800590+0000","last_unstale":"2024-04-06T12:12:08.232240+0000","last_undegraded":"2024-04-06T12:12:08.232240+0000","last_fullsized":"2024-04-06T12:12:08.232240+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:00:31.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232899+0000","last_change":"2024-04-06T12:12:04.216144+0000","last_active":"2024-04-06T12:12:08.232899+0000","last_peered":"2024-04-06T12:12:08.232899+0000","last_clean":"2024-04-06T12:12:08.232899+0000","last_became_active":"2024-04-06T12:12:04.215885+0000","last_became_peered":"2024-04-06T12:12:04.215885+0000","last_unstale":"2024-04-06T12:12:08.232899+0000","last_undegraded":"2024-04-06T12:12:08.232899+0000","last_fullsized":"2024-04-06T12:12:08.232899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:36:25.380472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256611+0000","last_change":"2024-04-06T12:12:08.252363+0000","last_active":"2024-04-06T12:12:08.256611+0000","last_peered":"2024-04-06T12:12:08.256611+0000","last_clean":"2024-04-06T12:12:08.256611+0000","last_became_active":"2024-04-06T12:12:08.252029+0000","last_became_peered":"2024-04-06T12:12:08.252029+0000","last_unstale":"2024-04-06T12:12:08.256611+0000","last_undegraded":"2024-04-06T12:12:08.256611+0000","last_fullsized":"2024-04-06T12:12:08.256611+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:35:54.255608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227386+0000","last_change":"2024-04-06T12:12:03.800462+0000","last_active":"2024-04-06T12:12:08.227386+0000","last_peered":"2024-04-06T12:12:08.227386+0000","last_clean":"2024-04-06T12:12:08.227386+0000","last_became_active":"2024-04-06T12:12:03.800210+0000","last_became_peered":"2024-04-06T12:12:03.800210+0000","last_unstale":"2024-04-06T12:12:08.227386+0000","last_undegraded":"2024-04-06T12:12:08.227386+0000","last_fullsized":"2024-04-06T12:12:08.227386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:00:31.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222742+0000","last_change":"2024-04-06T12:12:04.216700+0000","last_active":"2024-04-06T12:12:08.222742+0000","last_peered":"2024-04-06T12:12:08.222742+0000","last_clean":"2024-04-06T12:12:08.222742+0000","last_became_active":"2024-04-06T12:12:04.216435+0000","last_became_peered":"2024-04-06T12:12:04.216435+0000","last_unstale":"2024-04-06T12:12:08.222742+0000","last_undegraded":"2024-04-06T12:12:08.222742+0000","last_fullsized":"2024-04-06T12:12:08.222742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:36:25.380472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247221+0000","last_change":"2024-04-06T12:12:08.242079+0000","last_active":"2024-04-06T12:12:08.247221+0000","last_peered":"2024-04-06T12:12:08.247221+0000","last_clean":"2024-04-06T12:12:08.247221+0000","last_became_active":"2024-04-06T12:12:08.241648+0000","last_became_peered":"2024-04-06T12:12:08.241648+0000","last_unstale":"2024-04-06T12:12:08.247221+0000","last_undegraded":"2024-04-06T12:12:08.247221+0000","last_fullsized":"2024-04-06T12:12:08.247221+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:37:57.999646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224055+0000","last_change":"2024-04-06T12:12:03.799875+0000","last_active":"2024-04-06T12:12:08.224055+0000","last_peered":"2024-04-06T12:12:08.224055+0000","last_clean":"2024-04-06T12:12:08.224055+0000","last_became_active":"2024-04-06T12:12:03.799562+0000","last_became_peered":"2024-04-06T12:12:03.799562+0000","last_unstale":"2024-04-06T12:12:08.224055+0000","last_undegraded":"2024-04-06T12:12:08.224055+0000","last_fullsized":"2024-04-06T12:12:08.224055+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:08:55.770454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.229090+0000","last_change":"2024-04-06T12:12:04.225142+0000","last_active":"2024-04-06T12:12:08.229090+0000","last_peered":"2024-04-06T12:12:08.229090+0000","last_clean":"2024-04-06T12:12:08.229090+0000","last_became_active":"2024-04-06T12:12:04.224838+0000","last_became_peered":"2024-04-06T12:12:04.224838+0000","last_unstale":"2024-04-06T12:12:08.229090+0000","last_undegraded":"2024-04-06T12:12:08.229090+0000","last_fullsized":"2024-04-06T12:12:08.229090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:21:16.167775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247222+0000","last_change":"2024-04-06T12:12:08.244037+0000","last_active":"2024-04-06T12:12:08.247222+0000","last_peered":"2024-04-06T12:12:08.247222+0000","last_clean":"2024-04-06T12:12:08.247222+0000","last_became_active":"2024-04-06T12:12:08.243685+0000","last_became_peered":"2024-04-06T12:12:08.243685+0000","last_unstale":"2024-04-06T12:12:08.247222+0000","last_undegraded":"2024-04-06T12:12:08.247222+0000","last_fullsized":"2024-04-06T12:12:08.247222+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:02:51.216167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228999+0000","last_change":"2024-04-06T12:12:03.799591+0000","last_active":"2024-04-06T12:12:08.228999+0000","last_peered":"2024-04-06T12:12:08.228999+0000","last_clean":"2024-04-06T12:12:08.228999+0000","last_became_active":"2024-04-06T12:12:03.799333+0000","last_became_peered":"2024-04-06T12:12:03.799333+0000","last_unstale":"2024-04-06T12:12:08.228999+0000","last_undegraded":"2024-04-06T12:12:08.228999+0000","last_fullsized":"2024-04-06T12:12:08.228999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:11.635313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232670+0000","last_change":"2024-04-06T12:12:04.215620+0000","last_active":"2024-04-06T12:12:08.232670+0000","last_peered":"2024-04-06T12:12:08.232670+0000","last_clean":"2024-04-06T12:12:08.232670+0000","last_became_active":"2024-04-06T12:12:04.215189+0000","last_became_peered":"2024-04-06T12:12:04.215189+0000","last_unstale":"2024-04-06T12:12:08.232670+0000","last_undegraded":"2024-04-06T12:12:08.232670+0000","last_fullsized":"2024-04-06T12:12:08.232670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247852+0000","last_change":"2024-04-06T12:12:08.246657+0000","last_active":"2024-04-06T12:12:08.247852+0000","last_peered":"2024-04-06T12:12:08.247852+0000","last_clean":"2024-04-06T12:12:08.247852+0000","last_became_active":"2024-04-06T12:12:08.246310+0000","last_became_peered":"2024-04-06T12:12:08.246310+0000","last_unstale":"2024-04-06T12:12:08.247852+0000","last_undegraded":"2024-04-06T12:12:08.247852+0000","last_fullsized":"2024-04-06T12:12:08.247852+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:57:02.114694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223115+0000","last_change":"2024-04-06T12:12:03.802440+0000","last_active":"2024-04-06T12:12:08.223115+0000","last_peered":"2024-04-06T12:12:08.223115+0000","last_clean":"2024-04-06T12:12:08.223115+0000","last_became_active":"2024-04-06T12:12:03.802193+0000","last_became_peered":"2024-04-06T12:12:03.802193+0000","last_unstale":"2024-04-06T12:12:08.223115+0000","last_undegraded":"2024-04-06T12:12:08.223115+0000","last_fullsized":"2024-04-06T12:12:08.223115+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:04.938049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232783+0000","last_change":"2024-04-06T12:12:04.225740+0000","last_active":"2024-04-06T12:12:08.232783+0000","last_peered":"2024-04-06T12:12:08.232783+0000","last_clean":"2024-04-06T12:12:08.232783+0000","last_became_active":"2024-04-06T12:12:04.224605+0000","last_became_peered":"2024-04-06T12:12:04.224605+0000","last_unstale":"2024-04-06T12:12:08.232783+0000","last_undegraded":"2024-04-06T12:12:08.232783+0000","last_fullsized":"2024-04-06T12:12:08.232783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:26:22.511847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.245748+0000","last_change":"2024-04-06T12:12:08.242260+0000","last_active":"2024-04-06T12:12:08.245748+0000","last_peered":"2024-04-06T12:12:08.245748+0000","last_clean":"2024-04-06T12:12:08.245748+0000","last_became_active":"2024-04-06T12:12:08.241863+0000","last_became_peered":"2024-04-06T12:12:08.241863+0000","last_unstale":"2024-04-06T12:12:08.245748+0000","last_undegraded":"2024-04-06T12:12:08.245748+0000","last_fullsized":"2024-04-06T12:12:08.245748+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:06.085793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-06T12:12:13.609454+0000","last_change":"2024-04-06T12:12:03.799037+0000","last_active":"2024-04-06T12:12:13.609454+0000","last_peered":"2024-04-06T12:12:13.609454+0000","last_clean":"2024-04-06T12:12:13.609454+0000","last_became_active":"2024-04-06T12:12:03.798723+0000","last_became_peered":"2024-04-06T12:12:03.798723+0000","last_unstale":"2024-04-06T12:12:13.609454+0000","last_undegraded":"2024-04-06T12:12:13.609454+0000","last_fullsized":"2024-04-06T12:12:13.609454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:00:31.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3796,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224345+0000","last_change":"2024-04-06T12:12:04.226914+0000","last_active":"2024-04-06T12:12:08.224345+0000","last_peered":"2024-04-06T12:12:08.224345+0000","last_clean":"2024-04-06T12:12:08.224345+0000","last_became_active":"2024-04-06T12:12:04.226421+0000","last_became_peered":"2024-04-06T12:12:04.226421+0000","last_unstale":"2024-04-06T12:12:08.224345+0000","last_undegraded":"2024-04-06T12:12:08.224345+0000","last_fullsized":"2024-04-06T12:12:08.224345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:45:10.755538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247061+0000","last_change":"2024-04-06T12:12:08.242923+0000","last_active":"2024-04-06T12:12:08.247061+0000","last_peered":"2024-04-06T12:12:08.247061+0000","last_clean":"2024-04-06T12:12:08.247061+0000","last_became_active":"2024-04-06T12:12:08.242491+0000","last_became_peered":"2024-04-06T12:12:08.242491+0000","last_unstale":"2024-04-06T12:12:08.247061+0000","last_undegraded":"2024-04-06T12:12:08.247061+0000","last_fullsized":"2024-04-06T12:12:08.247061+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:57:02.114694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234681+0000","last_change":"2024-04-06T12:12:03.796757+0000","last_active":"2024-04-06T12:12:08.234681+0000","last_peered":"2024-04-06T12:12:08.234681+0000","last_clean":"2024-04-06T12:12:08.234681+0000","last_became_active":"2024-04-06T12:12:03.796519+0000","last_became_peered":"2024-04-06T12:12:03.796519+0000","last_unstale":"2024-04-06T12:12:08.234681+0000","last_undegraded":"2024-04-06T12:12:08.234681+0000","last_fullsized":"2024-04-06T12:12:08.234681+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:40:20.784453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.235485+0000","last_change":"2024-04-06T12:12:04.224445+0000","last_active":"2024-04-06T12:12:08.235485+0000","last_peered":"2024-04-06T12:12:08.235485+0000","last_clean":"2024-04-06T12:12:08.235485+0000","last_became_active":"2024-04-06T12:12:04.224153+0000","last_became_peered":"2024-04-06T12:12:04.224153+0000","last_unstale":"2024-04-06T12:12:08.235485+0000","last_undegraded":"2024-04-06T12:12:08.235485+0000","last_fullsized":"2024-04-06T12:12:08.235485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:15.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256724+0000","last_change":"2024-04-06T12:12:08.251536+0000","last_active":"2024-04-06T12:12:08.256724+0000","last_peered":"2024-04-06T12:12:08.256724+0000","last_clean":"2024-04-06T12:12:08.256724+0000","last_became_active":"2024-04-06T12:12:08.251142+0000","last_became_peered":"2024-04-06T12:12:08.251142+0000","last_unstale":"2024-04-06T12:12:08.256724+0000","last_undegraded":"2024-04-06T12:12:08.256724+0000","last_fullsized":"2024-04-06T12:12:08.256724+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:48:55.155611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228030+0000","last_change":"2024-04-06T12:12:03.797816+0000","last_active":"2024-04-06T12:12:08.228030+0000","last_peered":"2024-04-06T12:12:08.228030+0000","last_clean":"2024-04-06T12:12:08.228030+0000","last_became_active":"2024-04-06T12:12:03.797540+0000","last_became_peered":"2024-04-06T12:12:03.797540+0000","last_unstale":"2024-04-06T12:12:08.228030+0000","last_undegraded":"2024-04-06T12:12:08.228030+0000","last_fullsized":"2024-04-06T12:12:08.228030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:15.699305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224078+0000","last_change":"2024-04-06T12:12:04.223703+0000","last_active":"2024-04-06T12:12:08.224078+0000","last_peered":"2024-04-06T12:12:08.224078+0000","last_clean":"2024-04-06T12:12:08.224078+0000","last_became_active":"2024-04-06T12:12:04.223394+0000","last_became_peered":"2024-04-06T12:12:04.223394+0000","last_unstale":"2024-04-06T12:12:08.224078+0000","last_undegraded":"2024-04-06T12:12:08.224078+0000","last_fullsized":"2024-04-06T12:12:08.224078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:19:38.810657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256643+0000","last_change":"2024-04-06T12:12:08.250454+0000","last_active":"2024-04-06T12:12:08.256643+0000","last_peered":"2024-04-06T12:12:08.256643+0000","last_clean":"2024-04-06T12:12:08.256643+0000","last_became_active":"2024-04-06T12:12:08.250052+0000","last_became_peered":"2024-04-06T12:12:08.250052+0000","last_unstale":"2024-04-06T12:12:08.256643+0000","last_undegraded":"2024-04-06T12:12:08.256643+0000","last_fullsized":"2024-04-06T12:12:08.256643+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:11:04.976956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223382+0000","last_change":"2024-04-06T12:12:03.799361+0000","last_active":"2024-04-06T12:12:08.223382+0000","last_peered":"2024-04-06T12:12:08.223382+0000","last_clean":"2024-04-06T12:12:08.223382+0000","last_became_active":"2024-04-06T12:12:03.798981+0000","last_became_peered":"2024-04-06T12:12:03.798981+0000","last_unstale":"2024-04-06T12:12:08.223382+0000","last_undegraded":"2024-04-06T12:12:08.223382+0000","last_fullsized":"2024-04-06T12:12:08.223382+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:11.635313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224104+0000","last_change":"2024-04-06T12:12:04.223535+0000","last_active":"2024-04-06T12:12:08.224104+0000","last_peered":"2024-04-06T12:12:08.224104+0000","last_clean":"2024-04-06T12:12:08.224104+0000","last_became_active":"2024-04-06T12:12:04.217966+0000","last_became_peered":"2024-04-06T12:12:04.217966+0000","last_unstale":"2024-04-06T12:12:08.224104+0000","last_undegraded":"2024-04-06T12:12:08.224104+0000","last_fullsized":"2024-04-06T12:12:08.224104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:15.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256634+0000","last_change":"2024-04-06T12:12:08.252578+0000","last_active":"2024-04-06T12:12:08.256634+0000","last_peered":"2024-04-06T12:12:08.256634+0000","last_clean":"2024-04-06T12:12:08.256634+0000","last_became_active":"2024-04-06T12:12:08.252149+0000","last_became_peered":"2024-04-06T12:12:08.252149+0000","last_unstale":"2024-04-06T12:12:08.256634+0000","last_undegraded":"2024-04-06T12:12:08.256634+0000","last_fullsized":"2024-04-06T12:12:08.256634+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:06.085793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.226746+0000","last_change":"2024-04-06T12:12:03.799994+0000","last_active":"2024-04-06T12:12:08.226746+0000","last_peered":"2024-04-06T12:12:08.226746+0000","last_clean":"2024-04-06T12:12:08.226746+0000","last_became_active":"2024-04-06T12:12:03.799663+0000","last_became_peered":"2024-04-06T12:12:03.799663+0000","last_unstale":"2024-04-06T12:12:08.226746+0000","last_undegraded":"2024-04-06T12:12:08.226746+0000","last_fullsized":"2024-04-06T12:12:08.226746+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:08:55.770454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223501+0000","last_change":"2024-04-06T12:12:04.223425+0000","last_active":"2024-04-06T12:12:08.223501+0000","last_peered":"2024-04-06T12:12:08.223501+0000","last_clean":"2024-04-06T12:12:08.223501+0000","last_became_active":"2024-04-06T12:12:04.223101+0000","last_became_peered":"2024-04-06T12:12:04.223101+0000","last_unstale":"2024-04-06T12:12:08.223501+0000","last_undegraded":"2024-04-06T12:12:08.223501+0000","last_fullsized":"2024-04-06T12:12:08.223501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.250437+0000","last_change":"2024-04-06T12:12:08.249078+0000","last_active":"2024-04-06T12:12:08.250437+0000","last_peered":"2024-04-06T12:12:08.250437+0000","last_clean":"2024-04-06T12:12:08.250437+0000","last_became_active":"2024-04-06T12:12:08.248512+0000","last_became_peered":"2024-04-06T12:12:08.248512+0000","last_unstale":"2024-04-06T12:12:08.250437+0000","last_undegraded":"2024-04-06T12:12:08.250437+0000","last_fullsized":"2024-04-06T12:12:08.250437+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:24:33.553367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.230408+0000","last_change":"2024-04-06T12:12:03.793832+0000","last_active":"2024-04-06T12:12:08.230408+0000","last_peered":"2024-04-06T12:12:08.230408+0000","last_clean":"2024-04-06T12:12:08.230408+0000","last_became_active":"2024-04-06T12:12:03.793518+0000","last_became_peered":"2024-04-06T12:12:03.793518+0000","last_unstale":"2024-04-06T12:12:08.230408+0000","last_undegraded":"2024-04-06T12:12:08.230408+0000","last_fullsized":"2024-04-06T12:12:08.230408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:37:27.257815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224591+0000","last_change":"2024-04-06T12:12:04.224260+0000","last_active":"2024-04-06T12:12:08.224591+0000","last_peered":"2024-04-06T12:12:08.224591+0000","last_clean":"2024-04-06T12:12:08.224591+0000","last_became_active":"2024-04-06T12:12:04.223968+0000","last_became_peered":"2024-04-06T12:12:04.223968+0000","last_unstale":"2024-04-06T12:12:08.224591+0000","last_undegraded":"2024-04-06T12:12:08.224591+0000","last_fullsized":"2024-04-06T12:12:08.224591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.245727+0000","last_change":"2024-04-06T12:12:08.242094+0000","last_active":"2024-04-06T12:12:08.245727+0000","last_peered":"2024-04-06T12:12:08.245727+0000","last_clean":"2024-04-06T12:12:08.245727+0000","last_became_active":"2024-04-06T12:12:08.241707+0000","last_became_peered":"2024-04-06T12:12:08.241707+0000","last_unstale":"2024-04-06T12:12:08.245727+0000","last_undegraded":"2024-04-06T12:12:08.245727+0000","last_fullsized":"2024-04-06T12:12:08.245727+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:47:51.958478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222772+0000","last_change":"2024-04-06T12:12:03.800106+0000","last_active":"2024-04-06T12:12:08.222772+0000","last_peered":"2024-04-06T12:12:08.222772+0000","last_clean":"2024-04-06T12:12:08.222772+0000","last_became_active":"2024-04-06T12:12:03.799843+0000","last_became_peered":"2024-04-06T12:12:03.799843+0000","last_unstale":"2024-04-06T12:12:08.222772+0000","last_undegraded":"2024-04-06T12:12:08.222772+0000","last_fullsized":"2024-04-06T12:12:08.222772+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:26:21.506491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.229502+0000","last_change":"2024-04-06T12:12:04.214371+0000","last_active":"2024-04-06T12:12:08.229502+0000","last_peered":"2024-04-06T12:12:08.229502+0000","last_clean":"2024-04-06T12:12:08.229502+0000","last_became_active":"2024-04-06T12:12:04.214095+0000","last_became_peered":"2024-04-06T12:12:04.214095+0000","last_unstale":"2024-04-06T12:12:08.229502+0000","last_undegraded":"2024-04-06T12:12:08.229502+0000","last_fullsized":"2024-04-06T12:12:08.229502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:45:10.755538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.258734+0000","last_change":"2024-04-06T12:12:08.255356+0000","last_active":"2024-04-06T12:12:08.258734+0000","last_peered":"2024-04-06T12:12:08.258734+0000","last_clean":"2024-04-06T12:12:08.258734+0000","last_became_active":"2024-04-06T12:12:08.254148+0000","last_became_peered":"2024-04-06T12:12:08.254148+0000","last_unstale":"2024-04-06T12:12:08.258734+0000","last_undegraded":"2024-04-06T12:12:08.258734+0000","last_fullsized":"2024-04-06T12:12:08.258734+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:35:54.255608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227821+0000","last_change":"2024-04-06T12:12:03.801770+0000","last_active":"2024-04-06T12:12:08.227821+0000","last_peered":"2024-04-06T12:12:08.227821+0000","last_clean":"2024-04-06T12:12:08.227821+0000","last_became_active":"2024-04-06T12:12:03.801494+0000","last_became_peered":"2024-04-06T12:12:03.801494+0000","last_unstale":"2024-04-06T12:12:08.227821+0000","last_undegraded":"2024-04-06T12:12:08.227821+0000","last_fullsized":"2024-04-06T12:12:08.227821+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:08:55.770454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232799+0000","last_change":"2024-04-06T12:12:04.217734+0000","last_active":"2024-04-06T12:12:08.232799+0000","last_peered":"2024-04-06T12:12:08.232799+0000","last_clean":"2024-04-06T12:12:08.232799+0000","last_became_active":"2024-04-06T12:12:04.216838+0000","last_became_peered":"2024-04-06T12:12:04.216838+0000","last_unstale":"2024-04-06T12:12:08.232799+0000","last_undegraded":"2024-04-06T12:12:08.232799+0000","last_fullsized":"2024-04-06T12:12:08.232799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:05.943405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.258759+0000","last_change":"2024-04-06T12:12:08.255965+0000","last_active":"2024-04-06T12:12:08.258759+0000","last_peered":"2024-04-06T12:12:08.258759+0000","last_clean":"2024-04-06T12:12:08.258759+0000","last_became_active":"2024-04-06T12:12:08.251143+0000","last_became_peered":"2024-04-06T12:12:08.251143+0000","last_unstale":"2024-04-06T12:12:08.258759+0000","last_undegraded":"2024-04-06T12:12:08.258759+0000","last_fullsized":"2024-04-06T12:12:08.258759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:21:01.506046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222665+0000","last_change":"2024-04-06T12:12:04.208976+0000","last_active":"2024-04-06T12:12:08.222665+0000","last_peered":"2024-04-06T12:12:08.222665+0000","last_clean":"2024-04-06T12:12:08.222665+0000","last_became_active":"2024-04-06T12:12:04.208524+0000","last_became_peered":"2024-04-06T12:12:04.208524+0000","last_unstale":"2024-04-06T12:12:08.222665+0000","last_undegraded":"2024-04-06T12:12:08.222665+0000","last_fullsized":"2024-04-06T12:12:08.222665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.226929+0000","last_change":"2024-04-06T12:12:03.796992+0000","last_active":"2024-04-06T12:12:08.226929+0000","last_peered":"2024-04-06T12:12:08.226929+0000","last_clean":"2024-04-06T12:12:08.226929+0000","last_became_active":"2024-04-06T12:12:03.795824+0000","last_became_peered":"2024-04-06T12:12:03.795824+0000","last_unstale":"2024-04-06T12:12:08.226929+0000","last_undegraded":"2024-04-06T12:12:08.226929+0000","last_fullsized":"2024-04-06T12:12:08.226929+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:11.635313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.246589+0000","last_change":"2024-04-06T12:12:08.244008+0000","last_active":"2024-04-06T12:12:08.246589+0000","last_peered":"2024-04-06T12:12:08.246589+0000","last_clean":"2024-04-06T12:12:08.246589+0000","last_became_active":"2024-04-06T12:12:08.242895+0000","last_became_peered":"2024-04-06T12:12:08.242895+0000","last_unstale":"2024-04-06T12:12:08.246589+0000","last_undegraded":"2024-04-06T12:12:08.246589+0000","last_fullsized":"2024-04-06T12:12:08.246589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:54:54.771467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232422+0000","last_change":"2024-04-06T12:12:04.215137+0000","last_active":"2024-04-06T12:12:08.232422+0000","last_peered":"2024-04-06T12:12:08.232422+0000","last_clean":"2024-04-06T12:12:08.232422+0000","last_became_active":"2024-04-06T12:12:04.214872+0000","last_became_peered":"2024-04-06T12:12:04.214872+0000","last_unstale":"2024-04-06T12:12:08.232422+0000","last_undegraded":"2024-04-06T12:12:08.232422+0000","last_fullsized":"2024-04-06T12:12:08.232422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227366+0000","last_change":"2024-04-06T12:12:03.797439+0000","last_active":"2024-04-06T12:12:08.227366+0000","last_peered":"2024-04-06T12:12:08.227366+0000","last_clean":"2024-04-06T12:12:08.227366+0000","last_became_active":"2024-04-06T12:12:03.797176+0000","last_became_peered":"2024-04-06T12:12:03.797176+0000","last_unstale":"2024-04-06T12:12:08.227366+0000","last_undegraded":"2024-04-06T12:12:08.227366+0000","last_fullsized":"2024-04-06T12:12:08.227366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:26:21.506491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223206+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:12:08.223206+0000","last_peered":"2024-04-06T12:12:08.223206+0000","last_clean":"2024-04-06T12:12:08.223206+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:12:08.223206+0000","last_undegraded":"2024-04-06T12:12:08.223206+0000","last_fullsized":"2024-04-06T12:12:08.223206+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.252451+0000","last_change":"2024-04-06T12:12:08.248645+0000","last_active":"2024-04-06T12:12:08.252451+0000","last_peered":"2024-04-06T12:12:08.252451+0000","last_clean":"2024-04-06T12:12:08.252451+0000","last_became_active":"2024-04-06T12:12:08.247420+0000","last_became_peered":"2024-04-06T12:12:08.247420+0000","last_unstale":"2024-04-06T12:12:08.252451+0000","last_undegraded":"2024-04-06T12:12:08.252451+0000","last_fullsized":"2024-04-06T12:12:08.252451+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:31:21.467656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232818+0000","last_change":"2024-04-06T12:12:04.220328+0000","last_active":"2024-04-06T12:12:08.232818+0000","last_peered":"2024-04-06T12:12:08.232818+0000","last_clean":"2024-04-06T12:12:08.232818+0000","last_became_active":"2024-04-06T12:12:04.219948+0000","last_became_peered":"2024-04-06T12:12:04.219948+0000","last_unstale":"2024-04-06T12:12:08.232818+0000","last_undegraded":"2024-04-06T12:12:08.232818+0000","last_fullsized":"2024-04-06T12:12:08.232818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:36:25.380472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223326+0000","last_change":"2024-04-06T12:12:03.800906+0000","last_active":"2024-04-06T12:12:08.223326+0000","last_peered":"2024-04-06T12:12:08.223326+0000","last_clean":"2024-04-06T12:12:08.223326+0000","last_became_active":"2024-04-06T12:12:03.800460+0000","last_became_peered":"2024-04-06T12:12:03.800460+0000","last_unstale":"2024-04-06T12:12:08.223326+0000","last_undegraded":"2024-04-06T12:12:08.223326+0000","last_fullsized":"2024-04-06T12:12:08.223326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:11.635313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.251977+0000","last_change":"2024-04-06T12:12:08.246821+0000","last_active":"2024-04-06T12:12:08.251977+0000","last_peered":"2024-04-06T12:12:08.251977+0000","last_clean":"2024-04-06T12:12:08.251977+0000","last_became_active":"2024-04-06T12:12:08.246154+0000","last_became_peered":"2024-04-06T12:12:08.246154+0000","last_unstale":"2024-04-06T12:12:08.251977+0000","last_undegraded":"2024-04-06T12:12:08.251977+0000","last_fullsized":"2024-04-06T12:12:08.251977+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:47:51.958478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223565+0000","last_change":"2024-04-06T12:12:03.801675+0000","last_active":"2024-04-06T12:12:08.223565+0000","last_peered":"2024-04-06T12:12:08.223565+0000","last_clean":"2024-04-06T12:12:08.223565+0000","last_became_active":"2024-04-06T12:12:03.800427+0000","last_became_peered":"2024-04-06T12:12:03.800427+0000","last_unstale":"2024-04-06T12:12:08.223565+0000","last_undegraded":"2024-04-06T12:12:08.223565+0000","last_fullsized":"2024-04-06T12:12:08.223565+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:21:42.136599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222894+0000","last_change":"2024-04-06T12:12:04.216377+0000","last_active":"2024-04-06T12:12:08.222894+0000","last_peered":"2024-04-06T12:12:08.222894+0000","last_clean":"2024-04-06T12:12:08.222894+0000","last_became_active":"2024-04-06T12:12:04.216084+0000","last_became_peered":"2024-04-06T12:12:04.216084+0000","last_unstale":"2024-04-06T12:12:08.222894+0000","last_undegraded":"2024-04-06T12:12:08.222894+0000","last_fullsized":"2024-04-06T12:12:08.222894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256642+0000","last_change":"2024-04-06T12:12:08.250308+0000","last_active":"2024-04-06T12:12:08.256642+0000","last_peered":"2024-04-06T12:12:08.256642+0000","last_clean":"2024-04-06T12:12:08.256642+0000","last_became_active":"2024-04-06T12:12:08.249904+0000","last_became_peered":"2024-04-06T12:12:08.249904+0000","last_unstale":"2024-04-06T12:12:08.256642+0000","last_undegraded":"2024-04-06T12:12:08.256642+0000","last_fullsized":"2024-04-06T12:12:08.256642+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:35:23.088849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222894+0000","last_change":"2024-04-06T12:12:03.803297+0000","last_active":"2024-04-06T12:12:08.222894+0000","last_peered":"2024-04-06T12:12:08.222894+0000","last_clean":"2024-04-06T12:12:08.222894+0000","last_became_active":"2024-04-06T12:12:03.803053+0000","last_became_peered":"2024-04-06T12:12:03.803053+0000","last_unstale":"2024-04-06T12:12:08.222894+0000","last_undegraded":"2024-04-06T12:12:08.222894+0000","last_fullsized":"2024-04-06T12:12:08.222894+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:21:42.136599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234446+0000","last_change":"2024-04-06T12:12:04.225056+0000","last_active":"2024-04-06T12:12:08.234446+0000","last_peered":"2024-04-06T12:12:08.234446+0000","last_clean":"2024-04-06T12:12:08.234446+0000","last_became_active":"2024-04-06T12:12:04.224635+0000","last_became_peered":"2024-04-06T12:12:04.224635+0000","last_unstale":"2024-04-06T12:12:08.234446+0000","last_undegraded":"2024-04-06T12:12:08.234446+0000","last_fullsized":"2024-04-06T12:12:08.234446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247224+0000","last_change":"2024-04-06T12:12:08.244844+0000","last_active":"2024-04-06T12:12:08.247224+0000","last_peered":"2024-04-06T12:12:08.247224+0000","last_clean":"2024-04-06T12:12:08.247224+0000","last_became_active":"2024-04-06T12:12:08.244422+0000","last_became_peered":"2024-04-06T12:12:08.244422+0000","last_unstale":"2024-04-06T12:12:08.247224+0000","last_undegraded":"2024-04-06T12:12:08.247224+0000","last_fullsized":"2024-04-06T12:12:08.247224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:54:54.771467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223933+0000","last_change":"2024-04-06T12:12:03.800460+0000","last_active":"2024-04-06T12:12:08.223933+0000","last_peered":"2024-04-06T12:12:08.223933+0000","last_clean":"2024-04-06T12:12:08.223933+0000","last_became_active":"2024-04-06T12:12:03.800183+0000","last_became_peered":"2024-04-06T12:12:03.800183+0000","last_unstale":"2024-04-06T12:12:08.223933+0000","last_undegraded":"2024-04-06T12:12:08.223933+0000","last_fullsized":"2024-04-06T12:12:08.223933+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:40:20.784453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.235166+0000","last_change":"2024-04-06T12:12:04.209360+0000","last_active":"2024-04-06T12:12:08.235166+0000","last_peered":"2024-04-06T12:12:08.235166+0000","last_clean":"2024-04-06T12:12:08.235166+0000","last_became_active":"2024-04-06T12:12:04.209105+0000","last_became_peered":"2024-04-06T12:12:04.209105+0000","last_unstale":"2024-04-06T12:12:08.235166+0000","last_undegraded":"2024-04-06T12:12:08.235166+0000","last_fullsized":"2024-04-06T12:12:08.235166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.252713+0000","last_change":"2024-04-06T12:12:08.249325+0000","last_active":"2024-04-06T12:12:08.252713+0000","last_peered":"2024-04-06T12:12:08.252713+0000","last_clean":"2024-04-06T12:12:08.252713+0000","last_became_active":"2024-04-06T12:12:08.248776+0000","last_became_peered":"2024-04-06T12:12:08.248776+0000","last_unstale":"2024-04-06T12:12:08.252713+0000","last_undegraded":"2024-04-06T12:12:08.252713+0000","last_fullsized":"2024-04-06T12:12:08.252713+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:31:21.467656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223337+0000","last_change":"2024-04-06T12:12:03.797556+0000","last_active":"2024-04-06T12:12:08.223337+0000","last_peered":"2024-04-06T12:12:08.223337+0000","last_clean":"2024-04-06T12:12:08.223337+0000","last_became_active":"2024-04-06T12:12:03.797288+0000","last_became_peered":"2024-04-06T12:12:03.797288+0000","last_unstale":"2024-04-06T12:12:08.223337+0000","last_undegraded":"2024-04-06T12:12:08.223337+0000","last_fullsized":"2024-04-06T12:12:08.223337+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:15.699305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.231368+0000","last_change":"2024-04-06T12:12:04.219859+0000","last_active":"2024-04-06T12:12:08.231368+0000","last_peered":"2024-04-06T12:12:08.231368+0000","last_clean":"2024-04-06T12:12:08.231368+0000","last_became_active":"2024-04-06T12:12:04.219470+0000","last_became_peered":"2024-04-06T12:12:04.219470+0000","last_unstale":"2024-04-06T12:12:08.231368+0000","last_undegraded":"2024-04-06T12:12:08.231368+0000","last_fullsized":"2024-04-06T12:12:08.231368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:16.704661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.258673+0000","last_change":"2024-04-06T12:12:08.254506+0000","last_active":"2024-04-06T12:12:08.258673+0000","last_peered":"2024-04-06T12:12:08.258673+0000","last_clean":"2024-04-06T12:12:08.258673+0000","last_became_active":"2024-04-06T12:12:08.250366+0000","last_became_peered":"2024-04-06T12:12:08.250366+0000","last_unstale":"2024-04-06T12:12:08.258673+0000","last_undegraded":"2024-04-06T12:12:08.258673+0000","last_fullsized":"2024-04-06T12:12:08.258673+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:29.953853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222644+0000","last_change":"2024-04-06T12:12:03.801858+0000","last_active":"2024-04-06T12:12:08.222644+0000","last_peered":"2024-04-06T12:12:08.222644+0000","last_clean":"2024-04-06T12:12:08.222644+0000","last_became_active":"2024-04-06T12:12:03.801643+0000","last_became_peered":"2024-04-06T12:12:03.801643+0000","last_unstale":"2024-04-06T12:12:08.222644+0000","last_undegraded":"2024-04-06T12:12:08.222644+0000","last_fullsized":"2024-04-06T12:12:08.222644+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:08:55.770454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232963+0000","last_change":"2024-04-06T12:12:04.213199+0000","last_active":"2024-04-06T12:12:08.232963+0000","last_peered":"2024-04-06T12:12:08.232963+0000","last_clean":"2024-04-06T12:12:08.232963+0000","last_became_active":"2024-04-06T12:12:04.212685+0000","last_became_peered":"2024-04-06T12:12:04.212685+0000","last_unstale":"2024-04-06T12:12:08.232963+0000","last_undegraded":"2024-04-06T12:12:08.232963+0000","last_fullsized":"2024-04-06T12:12:08.232963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:45:10.755538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.250618+0000","last_change":"2024-04-06T12:12:08.249234+0000","last_active":"2024-04-06T12:12:08.250618+0000","last_peered":"2024-04-06T12:12:08.250618+0000","last_clean":"2024-04-06T12:12:08.250618+0000","last_became_active":"2024-04-06T12:12:08.248705+0000","last_became_peered":"2024-04-06T12:12:08.248705+0000","last_unstale":"2024-04-06T12:12:08.250618+0000","last_undegraded":"2024-04-06T12:12:08.250618+0000","last_fullsized":"2024-04-06T12:12:08.250618+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:58:47.243168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222776+0000","last_change":"2024-04-06T12:12:03.798517+0000","last_active":"2024-04-06T12:12:08.222776+0000","last_peered":"2024-04-06T12:12:08.222776+0000","last_clean":"2024-04-06T12:12:08.222776+0000","last_became_active":"2024-04-06T12:12:03.798118+0000","last_became_peered":"2024-04-06T12:12:03.798118+0000","last_unstale":"2024-04-06T12:12:08.222776+0000","last_undegraded":"2024-04-06T12:12:08.222776+0000","last_fullsized":"2024-04-06T12:12:08.222776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:15.699305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232781+0000","last_change":"2024-04-06T12:12:04.216405+0000","last_active":"2024-04-06T12:12:08.232781+0000","last_peered":"2024-04-06T12:12:08.232781+0000","last_clean":"2024-04-06T12:12:08.232781+0000","last_became_active":"2024-04-06T12:12:04.216059+0000","last_became_peered":"2024-04-06T12:12:04.216059+0000","last_unstale":"2024-04-06T12:12:08.232781+0000","last_undegraded":"2024-04-06T12:12:08.232781+0000","last_fullsized":"2024-04-06T12:12:08.232781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.258081+0000","last_change":"2024-04-06T12:12:08.253652+0000","last_active":"2024-04-06T12:12:08.258081+0000","last_peered":"2024-04-06T12:12:08.258081+0000","last_clean":"2024-04-06T12:12:08.258081+0000","last_became_active":"2024-04-06T12:12:08.253321+0000","last_became_peered":"2024-04-06T12:12:08.253321+0000","last_unstale":"2024-04-06T12:12:08.258081+0000","last_undegraded":"2024-04-06T12:12:08.258081+0000","last_fullsized":"2024-04-06T12:12:08.258081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:31:50.315335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224175+0000","last_change":"2024-04-06T12:12:03.798088+0000","last_active":"2024-04-06T12:12:08.224175+0000","last_peered":"2024-04-06T12:12:08.224175+0000","last_clean":"2024-04-06T12:12:08.224175+0000","last_became_active":"2024-04-06T12:12:03.797844+0000","last_became_peered":"2024-04-06T12:12:03.797844+0000","last_unstale":"2024-04-06T12:12:08.224175+0000","last_undegraded":"2024-04-06T12:12:08.224175+0000","last_fullsized":"2024-04-06T12:12:08.224175+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:09.507432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223640+0000","last_change":"2024-04-06T12:12:04.224049+0000","last_active":"2024-04-06T12:12:08.223640+0000","last_peered":"2024-04-06T12:12:08.223640+0000","last_clean":"2024-04-06T12:12:08.223640+0000","last_became_active":"2024-04-06T12:12:04.223437+0000","last_became_peered":"2024-04-06T12:12:04.223437+0000","last_unstale":"2024-04-06T12:12:08.223640+0000","last_undegraded":"2024-04-06T12:12:08.223640+0000","last_fullsized":"2024-04-06T12:12:08.223640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:36:25.380472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247262+0000","last_change":"2024-04-06T12:12:08.244925+0000","last_active":"2024-04-06T12:12:08.247262+0000","last_peered":"2024-04-06T12:12:08.247262+0000","last_clean":"2024-04-06T12:12:08.247262+0000","last_became_active":"2024-04-06T12:12:08.244571+0000","last_became_peered":"2024-04-06T12:12:08.244571+0000","last_unstale":"2024-04-06T12:12:08.247262+0000","last_undegraded":"2024-04-06T12:12:08.247262+0000","last_fullsized":"2024-04-06T12:12:08.247262+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:31:50.315335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227747+0000","last_change":"2024-04-06T12:12:03.800145+0000","last_active":"2024-04-06T12:12:08.227747+0000","last_peered":"2024-04-06T12:12:08.227747+0000","last_clean":"2024-04-06T12:12:08.227747+0000","last_became_active":"2024-04-06T12:12:03.799406+0000","last_became_peered":"2024-04-06T12:12:03.799406+0000","last_unstale":"2024-04-06T12:12:08.227747+0000","last_undegraded":"2024-04-06T12:12:08.227747+0000","last_fullsized":"2024-04-06T12:12:08.227747+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:40:20.784453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224246+0000","last_change":"2024-04-06T12:12:04.226479+0000","last_active":"2024-04-06T12:12:08.224246+0000","last_peered":"2024-04-06T12:12:08.224246+0000","last_clean":"2024-04-06T12:12:08.224246+0000","last_became_active":"2024-04-06T12:12:04.222457+0000","last_became_peered":"2024-04-06T12:12:04.222457+0000","last_unstale":"2024-04-06T12:12:08.224246+0000","last_undegraded":"2024-04-06T12:12:08.224246+0000","last_fullsized":"2024-04-06T12:12:08.224246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:41:20.002927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.252006+0000","last_change":"2024-04-06T12:12:08.247112+0000","last_active":"2024-04-06T12:12:08.252006+0000","last_peered":"2024-04-06T12:12:08.252006+0000","last_clean":"2024-04-06T12:12:08.252006+0000","last_became_active":"2024-04-06T12:12:08.246705+0000","last_became_peered":"2024-04-06T12:12:08.246705+0000","last_unstale":"2024-04-06T12:12:08.252006+0000","last_undegraded":"2024-04-06T12:12:08.252006+0000","last_fullsized":"2024-04-06T12:12:08.252006+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:29.953853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222999+0000","last_change":"2024-04-06T12:12:03.799395+0000","last_active":"2024-04-06T12:12:08.222999+0000","last_peered":"2024-04-06T12:12:08.222999+0000","last_clean":"2024-04-06T12:12:08.222999+0000","last_became_active":"2024-04-06T12:12:03.799116+0000","last_became_peered":"2024-04-06T12:12:03.799116+0000","last_unstale":"2024-04-06T12:12:08.222999+0000","last_undegraded":"2024-04-06T12:12:08.222999+0000","last_fullsized":"2024-04-06T12:12:08.222999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:09.507432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227629+0000","last_change":"2024-04-06T12:12:04.221238+0000","last_active":"2024-04-06T12:12:08.227629+0000","last_peered":"2024-04-06T12:12:08.227629+0000","last_clean":"2024-04-06T12:12:08.227629+0000","last_became_active":"2024-04-06T12:12:04.220606+0000","last_became_peered":"2024-04-06T12:12:04.220606+0000","last_unstale":"2024-04-06T12:12:08.227629+0000","last_undegraded":"2024-04-06T12:12:08.227629+0000","last_fullsized":"2024-04-06T12:12:08.227629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.246867+0000","last_change":"2024-04-06T12:12:08.241508+0000","last_active":"2024-04-06T12:12:08.246867+0000","last_peered":"2024-04-06T12:12:08.246867+0000","last_clean":"2024-04-06T12:12:08.246867+0000","last_became_active":"2024-04-06T12:12:08.241184+0000","last_became_peered":"2024-04-06T12:12:08.241184+0000","last_unstale":"2024-04-06T12:12:08.246867+0000","last_undegraded":"2024-04-06T12:12:08.246867+0000","last_fullsized":"2024-04-06T12:12:08.246867+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:35:54.255608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228578+0000","last_change":"2024-04-06T12:12:03.795282+0000","last_active":"2024-04-06T12:12:08.228578+0000","last_peered":"2024-04-06T12:12:08.228578+0000","last_clean":"2024-04-06T12:12:08.228578+0000","last_became_active":"2024-04-06T12:12:03.794928+0000","last_became_peered":"2024-04-06T12:12:03.794928+0000","last_unstale":"2024-04-06T12:12:08.228578+0000","last_undegraded":"2024-04-06T12:12:08.228578+0000","last_fullsized":"2024-04-06T12:12:08.228578+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:00:31.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.225091+0000","last_change":"2024-04-06T12:12:04.224265+0000","last_active":"2024-04-06T12:12:08.225091+0000","last_peered":"2024-04-06T12:12:08.225091+0000","last_clean":"2024-04-06T12:12:08.225091+0000","last_became_active":"2024-04-06T12:12:04.223892+0000","last_became_peered":"2024-04-06T12:12:04.223892+0000","last_unstale":"2024-04-06T12:12:08.225091+0000","last_undegraded":"2024-04-06T12:12:08.225091+0000","last_fullsized":"2024-04-06T12:12:08.225091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:21:16.167775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223213+0000","last_change":"2024-04-06T12:12:04.223037+0000","last_active":"2024-04-06T12:12:08.223213+0000","last_peered":"2024-04-06T12:12:08.223213+0000","last_clean":"2024-04-06T12:12:08.223213+0000","last_became_active":"2024-04-06T12:12:04.218051+0000","last_became_peered":"2024-04-06T12:12:04.218051+0000","last_unstale":"2024-04-06T12:12:08.223213+0000","last_undegraded":"2024-04-06T12:12:08.223213+0000","last_fullsized":"2024-04-06T12:12:08.223213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:22:59.062918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227020+0000","last_change":"2024-04-06T12:12:04.214042+0000","last_active":"2024-04-06T12:12:08.227020+0000","last_peered":"2024-04-06T12:12:08.227020+0000","last_clean":"2024-04-06T12:12:08.227020+0000","last_became_active":"2024-04-06T12:12:04.213777+0000","last_became_peered":"2024-04-06T12:12:04.213777+0000","last_unstale":"2024-04-06T12:12:08.227020+0000","last_undegraded":"2024-04-06T12:12:08.227020+0000","last_fullsized":"2024-04-06T12:12:08.227020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:05.943405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228110+0000","last_change":"2024-04-06T12:12:04.216964+0000","last_active":"2024-04-06T12:12:08.228110+0000","last_peered":"2024-04-06T12:12:08.228110+0000","last_clean":"2024-04-06T12:12:08.228110+0000","last_became_active":"2024-04-06T12:12:04.216692+0000","last_became_peered":"2024-04-06T12:12:04.216692+0000","last_unstale":"2024-04-06T12:12:08.228110+0000","last_undegraded":"2024-04-06T12:12:08.228110+0000","last_fullsized":"2024-04-06T12:12:08.228110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234533+0000","last_change":"2024-04-06T12:12:04.219048+0000","last_active":"2024-04-06T12:12:08.234533+0000","last_peered":"2024-04-06T12:12:08.234533+0000","last_clean":"2024-04-06T12:12:08.234533+0000","last_became_active":"2024-04-06T12:12:04.218624+0000","last_became_peered":"2024-04-06T12:12:04.218624+0000","last_unstale":"2024-04-06T12:12:08.234533+0000","last_undegraded":"2024-04-06T12:12:08.234533+0000","last_fullsized":"2024-04-06T12:12:08.234533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:28:15.608394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.233536+0000","last_change":"2024-04-06T12:12:04.218893+0000","last_active":"2024-04-06T12:12:08.233536+0000","last_peered":"2024-04-06T12:12:08.233536+0000","last_clean":"2024-04-06T12:12:08.233536+0000","last_became_active":"2024-04-06T12:12:04.218526+0000","last_became_peered":"2024-04-06T12:12:04.218526+0000","last_unstale":"2024-04-06T12:12:08.233536+0000","last_undegraded":"2024-04-06T12:12:08.233536+0000","last_fullsized":"2024-04-06T12:12:08.233536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:15.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223037+0000","last_change":"2024-04-06T12:12:04.216794+0000","last_active":"2024-04-06T12:12:08.223037+0000","last_peered":"2024-04-06T12:12:08.223037+0000","last_clean":"2024-04-06T12:12:08.223037+0000","last_became_active":"2024-04-06T12:12:04.216532+0000","last_became_peered":"2024-04-06T12:12:04.216532+0000","last_unstale":"2024-04-06T12:12:08.223037+0000","last_undegraded":"2024-04-06T12:12:08.223037+0000","last_fullsized":"2024-04-06T12:12:08.223037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:15.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224560+0000","last_change":"2024-04-06T12:12:04.229182+0000","last_active":"2024-04-06T12:12:08.224560+0000","last_peered":"2024-04-06T12:12:08.224560+0000","last_clean":"2024-04-06T12:12:08.224560+0000","last_became_active":"2024-04-06T12:12:04.228927+0000","last_became_peered":"2024-04-06T12:12:04.228927+0000","last_unstale":"2024-04-06T12:12:08.224560+0000","last_undegraded":"2024-04-06T12:12:08.224560+0000","last_fullsized":"2024-04-06T12:12:08.224560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234558+0000","last_change":"2024-04-06T12:12:04.217003+0000","last_active":"2024-04-06T12:12:08.234558+0000","last_peered":"2024-04-06T12:12:08.234558+0000","last_clean":"2024-04-06T12:12:08.234558+0000","last_became_active":"2024-04-06T12:12:04.216757+0000","last_became_peered":"2024-04-06T12:12:04.216757+0000","last_unstale":"2024-04-06T12:12:08.234558+0000","last_undegraded":"2024-04-06T12:12:08.234558+0000","last_fullsized":"2024-04-06T12:12:08.234558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:00.258491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227358+0000","last_change":"2024-04-06T12:12:04.216442+0000","last_active":"2024-04-06T12:12:08.227358+0000","last_peered":"2024-04-06T12:12:08.227358+0000","last_clean":"2024-04-06T12:12:08.227358+0000","last_became_active":"2024-04-06T12:12:04.215051+0000","last_became_peered":"2024-04-06T12:12:04.215051+0000","last_unstale":"2024-04-06T12:12:08.227358+0000","last_undegraded":"2024-04-06T12:12:08.227358+0000","last_fullsized":"2024-04-06T12:12:08.227358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:19:38.810657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232330+0000","last_change":"2024-04-06T12:12:04.219526+0000","last_active":"2024-04-06T12:12:08.232330+0000","last_peered":"2024-04-06T12:12:08.232330+0000","last_clean":"2024-04-06T12:12:08.232330+0000","last_became_active":"2024-04-06T12:12:04.219229+0000","last_became_peered":"2024-04-06T12:12:04.219229+0000","last_unstale":"2024-04-06T12:12:08.232330+0000","last_undegraded":"2024-04-06T12:12:08.232330+0000","last_fullsized":"2024-04-06T12:12:08.232330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:22:59.062918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228186+0000","last_change":"2024-04-06T12:12:04.226116+0000","last_active":"2024-04-06T12:12:08.228186+0000","last_peered":"2024-04-06T12:12:08.228186+0000","last_clean":"2024-04-06T12:12:08.228186+0000","last_became_active":"2024-04-06T12:12:04.225820+0000","last_became_peered":"2024-04-06T12:12:04.225820+0000","last_unstale":"2024-04-06T12:12:08.228186+0000","last_undegraded":"2024-04-06T12:12:08.228186+0000","last_fullsized":"2024-04-06T12:12:08.228186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:41:20.002927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234612+0000","last_change":"2024-04-06T12:12:04.228504+0000","last_active":"2024-04-06T12:12:08.234612+0000","last_peered":"2024-04-06T12:12:08.234612+0000","last_clean":"2024-04-06T12:12:08.234612+0000","last_became_active":"2024-04-06T12:12:04.228232+0000","last_became_peered":"2024-04-06T12:12:04.228232+0000","last_unstale":"2024-04-06T12:12:08.234612+0000","last_undegraded":"2024-04-06T12:12:08.234612+0000","last_fullsized":"2024-04-06T12:12:08.234612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:22:59.062918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223649+0000","last_change":"2024-04-06T12:12:04.230394+0000","last_active":"2024-04-06T12:12:08.223649+0000","last_peered":"2024-04-06T12:12:08.223649+0000","last_clean":"2024-04-06T12:12:08.223649+0000","last_became_active":"2024-04-06T12:12:04.230065+0000","last_became_peered":"2024-04-06T12:12:04.230065+0000","last_unstale":"2024-04-06T12:12:08.223649+0000","last_undegraded":"2024-04-06T12:12:08.223649+0000","last_fullsized":"2024-04-06T12:12:08.223649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4594,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":851968,"data_stored":690730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":262144,"data_stored":102557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":98921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":856064,"data_stored":689767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":98761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":103576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.223}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":99906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":98851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:12:32.301 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:12:32.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.459+0000 7f6ea08c2700 1 Processor -- start 2024-04-06T12:12:32.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.459+0000 7f6ea08c2700 1 -- start start 2024-04-06T12:12:32.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.459+0000 7f6ea08c2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6e9c0d11b0 0x7f6e9c0d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:32.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.459+0000 7f6ea08c2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6e9c0cb870 0x7f6e9c15c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:32.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.459+0000 7f6ea08c2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9c0d1ad0 con 0x7f6e9c0cfa00 2024-04-06T12:12:32.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.460+0000 7f6ea08c2700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e9c0ccdc0 con 0x7f6e9c0cb870 2024-04-06T12:12:32.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.460+0000 7f6ea08c2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e9c0d0b50 con 0x7f6e9c0d11b0 2024-04-06T12:12:32.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.460+0000 7f6e99d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6e9c0cb870 0x7f6e9c15c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:32.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.460+0000 7f6e9ad9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6e9c0cfa00 0x7f6e9c0cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34336/0 (socket says 172.21.15.189:34336) 2024-04-06T12:12:32.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.460+0000 7f6e9ad9d700 1 -- 172.21.15.189:0/3421377984 learned_addr learned my addr 172.21.15.189:0/3421377984 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:32.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.460+0000 7f6e9a59c700 1 --2- 172.21.15.189:0/3421377984 >> v2:172.21.15.192:3300/0 conn(0x7f6e9c0d11b0 0x7f6e9c0d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.460+0000 7f6e99d9b700 1 -- 172.21.15.189:0/3421377984 >> v2:172.21.15.192:3300/0 conn(0x7f6e9c0d11b0 msgr2=0x7f6e9c0d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.460+0000 7f6e99d9b700 1 --2- 172.21.15.189:0/3421377984 >> v2:172.21.15.192:3300/0 conn(0x7f6e9c0d11b0 0x7f6e9c0d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.460+0000 7f6e99d9b700 1 -- 172.21.15.189:0/3421377984 >> v1:172.21.15.189:6789/0 conn(0x7f6e9c0cfa00 legacy=0x7f6e9c0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.460+0000 7f6e9959a700 1 -- 172.21.15.189:0/3421377984 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523163119 0 0) 0x7f6e9c0d1ad0 con 0x7f6e9c0cfa00 2024-04-06T12:12:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.460+0000 7f6e99d9b700 1 -- 172.21.15.189:0/3421377984 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9c0cbd80 con 0x7f6e9c0cb870 2024-04-06T12:12:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.460+0000 7f6e99d9b700 1 --2- 172.21.15.189:0/3421377984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e9c0cb870 0x7f6e9c15c280 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c0ced00 tx=0x7f6e8c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=860fe12e60d3f65a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.460+0000 7f6e9959a700 1 -- 172.21.15.189:0/3421377984 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c011830 con 0x7f6e9c0cb870 2024-04-06T12:12:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.460+0000 7f6e9959a700 1 -- 172.21.15.189:0/3421377984 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e8c011990 con 0x7f6e9c0cb870 2024-04-06T12:12:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.460+0000 7f6e9959a700 1 -- 172.21.15.189:0/3421377984 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c011d20 con 0x7f6e9c0cb870 2024-04-06T12:12:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.461+0000 7f6ea08c2700 1 -- 172.21.15.189:0/3421377984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e9c0cb870 msgr2=0x7f6e9c15c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.461+0000 7f6ea08c2700 1 --2- 172.21.15.189:0/3421377984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e9c0cb870 0x7f6e9c15c280 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c0ced00 tx=0x7f6e8c009b20 comp rx=0 tx=0).stop 2024-04-06T12:12:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.461+0000 7f6ea08c2700 1 -- 172.21.15.189:0/3421377984 shutdown_connections 2024-04-06T12:12:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.461+0000 7f6ea08c2700 1 --2- 172.21.15.189:0/3421377984 >> v2:172.21.15.192:3300/0 conn(0x7f6e9c0d11b0 0x7f6e9c0d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.461+0000 7f6ea08c2700 1 --2- 172.21.15.189:0/3421377984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e9c0cb870 0x7f6e9c15c280 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.461+0000 7f6ea08c2700 1 -- 172.21.15.189:0/3421377984 >> 172.21.15.189:0/3421377984 conn(0x7f6e9c0b2db0 msgr2=0x7f6e9c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:32.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.461+0000 7f6ea08c2700 1 -- 172.21.15.189:0/3421377984 shutdown_connections 2024-04-06T12:12:32.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.461+0000 7f6ea08c2700 1 -- 172.21.15.189:0/3421377984 wait complete. 2024-04-06T12:12:32.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.462+0000 7f6ea08c2700 1 Processor -- start 2024-04-06T12:12:32.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.462+0000 7f6ea08c2700 1 -- start start 2024-04-06T12:12:32.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.463+0000 7f6ea08c2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e9c0cb870 0x7f6e9c1526b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:32.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.463+0000 7f6ea08c2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e9c0cfa00 0x7f6e9c0cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:32.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.463+0000 7f6ea08c2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e9c0d11b0 0x7f6e9c0c8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:32.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.463+0000 7f6ea08c2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c0c51c0 con 0x7f6e9c0d11b0 2024-04-06T12:12:32.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.463+0000 7f6ea08c2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c0c54a0 con 0x7f6e9c0cb870 2024-04-06T12:12:32.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.463+0000 7f6ea08c2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c0c5780 con 0x7f6e9c0cfa00 2024-04-06T12:12:32.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.463+0000 7f6e9ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e9c0d11b0 0x7f6e9c0c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:32.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.463+0000 7f6e9ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e9c0d11b0 0x7f6e9c0c8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38364/0 (socket says 172.21.15.189:38364) 2024-04-06T12:12:32.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.463+0000 7f6e9ad9d700 1 -- 172.21.15.189:0/1684999760 learned_addr learned my addr 172.21.15.189:0/1684999760 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:32.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.463+0000 7f6e9a59c700 1 --2- 172.21.15.189:0/1684999760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e9c0cb870 0x7f6e9c1526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:32.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.463+0000 7f6e99d9b700 1 --2- 172.21.15.189:0/1684999760 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e9c0cfa00 0x7f6e9c0cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:32.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.463+0000 7f6e9ad9d700 1 -- 172.21.15.189:0/1684999760 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e9c0cfa00 msgr2=0x7f6e9c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:32.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.463+0000 7f6e9ad9d700 1 --2- 172.21.15.189:0/1684999760 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e9c0cfa00 0x7f6e9c0cadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.463+0000 7f6e9ad9d700 1 -- 172.21.15.189:0/1684999760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e9c0cb870 msgr2=0x7f6e9c1526b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:32.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.463+0000 7f6e9ad9d700 1 --2- 172.21.15.189:0/1684999760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e9c0cb870 0x7f6e9c1526b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.463+0000 7f6e9ad9d700 1 -- 172.21.15.189:0/1684999760 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8c0094a0 con 0x7f6e9c0d11b0 2024-04-06T12:12:32.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.463+0000 7f6e99d9b700 1 --2- 172.21.15.189:0/1684999760 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e9c0cfa00 0x7f6e9c0cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:32.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.464+0000 7f6e9ad9d700 1 --2- 172.21.15.189:0/1684999760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e9c0d11b0 0x7f6e9c0c8430 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6e9400cba0 tx=0x7f6e9400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:32.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.464+0000 7f6e92ffd700 1 -- 172.21.15.189:0/1684999760 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9400ab80 con 0x7f6e9c0d11b0 2024-04-06T12:12:32.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.464+0000 7f6e92ffd700 1 -- 172.21.15.189:0/1684999760 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9400ad00 con 0x7f6e9c0d11b0 2024-04-06T12:12:32.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.464+0000 7f6ea08c2700 1 -- 172.21.15.189:0/1684999760 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e9c0c5a60 con 0x7f6e9c0d11b0 2024-04-06T12:12:32.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.464+0000 7f6e92ffd700 1 -- 172.21.15.189:0/1684999760 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e94007310 con 0x7f6e9c0d11b0 2024-04-06T12:12:32.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.464+0000 7f6ea08c2700 1 -- 172.21.15.189:0/1684999760 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e9c152bf0 con 0x7f6e9c0d11b0 2024-04-06T12:12:32.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.465+0000 7f6ea08c2700 1 -- 172.21.15.189:0/1684999760 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e9c0c5f80 con 0x7f6e9c0d11b0 2024-04-06T12:12:32.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.465+0000 7f6e92ffd700 1 -- 172.21.15.189:0/1684999760 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6e94007470 con 0x7f6e9c0d11b0 2024-04-06T12:12:32.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.465+0000 7f6e92ffd700 1 --2- 172.21.15.189:0/1684999760 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6e7c06ac90 0x7f6e7c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:32.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.466+0000 7f6e92ffd700 1 -- 172.21.15.189:0/1684999760 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6e94089440 con 0x7f6e9c0d11b0 2024-04-06T12:12:32.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.466+0000 7f6e9a59c700 1 --2- 172.21.15.189:0/1684999760 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6e7c06ac90 0x7f6e7c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:32.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.466+0000 7f6e9a59c700 1 --2- 172.21.15.189:0/1684999760 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6e7c06ac90 0x7f6e7c06d150 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c1f4770 tx=0x7f6e84008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:32.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.470+0000 7f6e92ffd700 1 -- 172.21.15.189:0/1684999760 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6e9c0c5f80 con 0x7f6e9c0d11b0 2024-04-06T12:12:32.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.662+0000 7f6ea08c2700 1 -- 172.21.15.189:0/1684999760 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6e9c0cf4e0 con 0x7f6e7c06ac90 2024-04-06T12:12:32.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.671+0000 7f6e92ffd700 1 -- 172.21.15.189:0/1684999760 <== mgr.4100 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345214 (secure 0 0 0) 0x7f6e840117b0 con 0x7f6e7c06ac90 2024-04-06T12:12:32.673 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:32.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.674+0000 7f6ea08c2700 1 -- 172.21.15.189:0/1684999760 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6e7c06ac90 msgr2=0x7f6e7c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:32.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.674+0000 7f6ea08c2700 1 --2- 172.21.15.189:0/1684999760 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6e7c06ac90 0x7f6e7c06d150 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c1f4770 tx=0x7f6e84008040 comp rx=0 tx=0).stop 2024-04-06T12:12:32.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.675+0000 7f6ea08c2700 1 -- 172.21.15.189:0/1684999760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e9c0d11b0 msgr2=0x7f6e9c0c8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:32.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.675+0000 7f6ea08c2700 1 --2- 172.21.15.189:0/1684999760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e9c0d11b0 0x7f6e9c0c8430 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6e9400cba0 tx=0x7f6e9400cf60 comp rx=0 tx=0).stop 2024-04-06T12:12:32.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.675+0000 7f6ea08c2700 1 -- 172.21.15.189:0/1684999760 shutdown_connections 2024-04-06T12:12:32.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.675+0000 7f6ea08c2700 1 --2- 172.21.15.189:0/1684999760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e9c0d11b0 0x7f6e9c0c8430 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.675+0000 7f6ea08c2700 1 --2- 172.21.15.189:0/1684999760 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6e7c06ac90 0x7f6e7c06d150 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.675+0000 7f6ea08c2700 1 --2- 172.21.15.189:0/1684999760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e9c0cb870 0x7f6e9c1526b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.675+0000 7f6ea08c2700 1 --2- 172.21.15.189:0/1684999760 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e9c0cfa00 0x7f6e9c0cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.675+0000 7f6ea08c2700 1 -- 172.21.15.189:0/1684999760 >> 172.21.15.189:0/1684999760 conn(0x7f6e9c0b2db0 msgr2=0x7f6e9c15a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:32.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.675+0000 7f6ea08c2700 1 -- 172.21.15.189:0/1684999760 shutdown_connections 2024-04-06T12:12:32.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.675+0000 7f6ea08c2700 1 -- 172.21.15.189:0/1684999760 wait complete. 2024-04-06T12:12:32.677 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:12:32.685 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-04-06T12:12:32.571958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594962,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219360,"kb_used_data":3188,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755360,"statfs":{"total":773094113280,"available":772869488640,"internally_reserved":0,"allocated":3264512,"data_stored":1983069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003224"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224930+0000","last_change":"2024-04-06T12:12:04.226505+0000","last_active":"2024-04-06T12:12:08.224930+0000","last_peered":"2024-04-06T12:12:08.224930+0000","last_clean":"2024-04-06T12:12:08.224930+0000","last_became_active":"2024-04-06T12:12:04.226112+0000","last_became_peered":"2024-04-06T12:12:04.226112+0000","last_unstale":"2024-04-06T12:12:08.224930+0000","last_undegraded":"2024-04-06T12:12:08.224930+0000","last_fullsized":"2024-04-06T12:12:08.224930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:09:38.014527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223002+0000","last_change":"2024-04-06T12:12:04.217545+0000","last_active":"2024-04-06T12:12:08.223002+0000","last_peered":"2024-04-06T12:12:08.223002+0000","last_clean":"2024-04-06T12:12:08.223002+0000","last_became_active":"2024-04-06T12:12:04.217304+0000","last_became_peered":"2024-04-06T12:12:04.217304+0000","last_unstale":"2024-04-06T12:12:08.223002+0000","last_undegraded":"2024-04-06T12:12:08.223002+0000","last_fullsized":"2024-04-06T12:12:08.223002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.231018+0000","last_change":"2024-04-06T12:12:04.209328+0000","last_active":"2024-04-06T12:12:08.231018+0000","last_peered":"2024-04-06T12:12:08.231018+0000","last_clean":"2024-04-06T12:12:08.231018+0000","last_became_active":"2024-04-06T12:12:04.209066+0000","last_became_peered":"2024-04-06T12:12:04.209066+0000","last_unstale":"2024-04-06T12:12:08.231018+0000","last_undegraded":"2024-04-06T12:12:08.231018+0000","last_fullsized":"2024-04-06T12:12:08.231018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:02:53.621993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227488+0000","last_change":"2024-04-06T12:12:04.221440+0000","last_active":"2024-04-06T12:12:08.227488+0000","last_peered":"2024-04-06T12:12:08.227488+0000","last_clean":"2024-04-06T12:12:08.227488+0000","last_became_active":"2024-04-06T12:12:04.220738+0000","last_became_peered":"2024-04-06T12:12:04.220738+0000","last_unstale":"2024-04-06T12:12:08.227488+0000","last_undegraded":"2024-04-06T12:12:08.227488+0000","last_fullsized":"2024-04-06T12:12:08.227488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:43:39.601900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223982+0000","last_change":"2024-04-06T12:12:04.223078+0000","last_active":"2024-04-06T12:12:08.223982+0000","last_peered":"2024-04-06T12:12:08.223982+0000","last_clean":"2024-04-06T12:12:08.223982+0000","last_became_active":"2024-04-06T12:12:04.222831+0000","last_became_peered":"2024-04-06T12:12:04.222831+0000","last_unstale":"2024-04-06T12:12:08.223982+0000","last_undegraded":"2024-04-06T12:12:08.223982+0000","last_fullsized":"2024-04-06T12:12:08.223982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:02:00.219999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227898+0000","last_change":"2024-04-06T12:12:04.215126+0000","last_active":"2024-04-06T12:12:08.227898+0000","last_peered":"2024-04-06T12:12:08.227898+0000","last_clean":"2024-04-06T12:12:08.227898+0000","last_became_active":"2024-04-06T12:12:04.214857+0000","last_became_peered":"2024-04-06T12:12:04.214857+0000","last_unstale":"2024-04-06T12:12:08.227898+0000","last_undegraded":"2024-04-06T12:12:08.227898+0000","last_fullsized":"2024-04-06T12:12:08.227898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:02:00.219999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223337+0000","last_change":"2024-04-06T12:12:04.222961+0000","last_active":"2024-04-06T12:12:08.223337+0000","last_peered":"2024-04-06T12:12:08.223337+0000","last_clean":"2024-04-06T12:12:08.223337+0000","last_became_active":"2024-04-06T12:12:04.222659+0000","last_became_peered":"2024-04-06T12:12:04.222659+0000","last_unstale":"2024-04-06T12:12:08.223337+0000","last_undegraded":"2024-04-06T12:12:08.223337+0000","last_fullsized":"2024-04-06T12:12:08.223337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:55.517965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.233016+0000","last_change":"2024-04-06T12:12:04.222214+0000","last_active":"2024-04-06T12:12:08.233016+0000","last_peered":"2024-04-06T12:12:08.233016+0000","last_clean":"2024-04-06T12:12:08.233016+0000","last_became_active":"2024-04-06T12:12:04.221817+0000","last_became_peered":"2024-04-06T12:12:04.221817+0000","last_unstale":"2024-04-06T12:12:08.233016+0000","last_undegraded":"2024-04-06T12:12:08.233016+0000","last_fullsized":"2024-04-06T12:12:08.233016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224843+0000","last_change":"2024-04-06T12:12:04.224106+0000","last_active":"2024-04-06T12:12:08.224843+0000","last_peered":"2024-04-06T12:12:08.224843+0000","last_clean":"2024-04-06T12:12:08.224843+0000","last_became_active":"2024-04-06T12:12:04.223653+0000","last_became_peered":"2024-04-06T12:12:04.223653+0000","last_unstale":"2024-04-06T12:12:08.224843+0000","last_undegraded":"2024-04-06T12:12:08.224843+0000","last_fullsized":"2024-04-06T12:12:08.224843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:50:27.596378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223873+0000","last_change":"2024-04-06T12:12:04.223892+0000","last_active":"2024-04-06T12:12:08.223873+0000","last_peered":"2024-04-06T12:12:08.223873+0000","last_clean":"2024-04-06T12:12:08.223873+0000","last_became_active":"2024-04-06T12:12:04.223615+0000","last_became_peered":"2024-04-06T12:12:04.223615+0000","last_unstale":"2024-04-06T12:12:08.223873+0000","last_undegraded":"2024-04-06T12:12:08.223873+0000","last_fullsized":"2024-04-06T12:12:08.223873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:02:53.621993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.225163+0000","last_change":"2024-04-06T12:12:04.223861+0000","last_active":"2024-04-06T12:12:08.225163+0000","last_peered":"2024-04-06T12:12:08.225163+0000","last_clean":"2024-04-06T12:12:08.225163+0000","last_became_active":"2024-04-06T12:12:04.223284+0000","last_became_peered":"2024-04-06T12:12:04.223284+0000","last_unstale":"2024-04-06T12:12:08.225163+0000","last_undegraded":"2024-04-06T12:12:08.225163+0000","last_fullsized":"2024-04-06T12:12:08.225163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:00.258491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232662+0000","last_change":"2024-04-06T12:12:04.230879+0000","last_active":"2024-04-06T12:12:08.232662+0000","last_peered":"2024-04-06T12:12:08.232662+0000","last_clean":"2024-04-06T12:12:08.232662+0000","last_became_active":"2024-04-06T12:12:04.230592+0000","last_became_peered":"2024-04-06T12:12:04.230592+0000","last_unstale":"2024-04-06T12:12:08.232662+0000","last_undegraded":"2024-04-06T12:12:08.232662+0000","last_fullsized":"2024-04-06T12:12:08.232662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:10.512788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232539+0000","last_change":"2024-04-06T12:12:04.218704+0000","last_active":"2024-04-06T12:12:08.232539+0000","last_peered":"2024-04-06T12:12:08.232539+0000","last_clean":"2024-04-06T12:12:08.232539+0000","last_became_active":"2024-04-06T12:12:04.218455+0000","last_became_peered":"2024-04-06T12:12:04.218455+0000","last_unstale":"2024-04-06T12:12:08.232539+0000","last_undegraded":"2024-04-06T12:12:08.232539+0000","last_fullsized":"2024-04-06T12:12:08.232539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:21:16.167775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.235340+0000","last_change":"2024-04-06T12:12:04.209351+0000","last_active":"2024-04-06T12:12:08.235340+0000","last_peered":"2024-04-06T12:12:08.235340+0000","last_clean":"2024-04-06T12:12:08.235340+0000","last_became_active":"2024-04-06T12:12:04.209092+0000","last_became_peered":"2024-04-06T12:12:04.209092+0000","last_unstale":"2024-04-06T12:12:08.235340+0000","last_undegraded":"2024-04-06T12:12:08.235340+0000","last_fullsized":"2024-04-06T12:12:08.235340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222602+0000","last_change":"2024-04-06T12:12:04.219642+0000","last_active":"2024-04-06T12:12:08.222602+0000","last_peered":"2024-04-06T12:12:08.222602+0000","last_clean":"2024-04-06T12:12:08.222602+0000","last_became_active":"2024-04-06T12:12:04.219398+0000","last_became_peered":"2024-04-06T12:12:04.219398+0000","last_unstale":"2024-04-06T12:12:08.222602+0000","last_undegraded":"2024-04-06T12:12:08.222602+0000","last_fullsized":"2024-04-06T12:12:08.222602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227279+0000","last_change":"2024-04-06T12:12:04.215338+0000","last_active":"2024-04-06T12:12:08.227279+0000","last_peered":"2024-04-06T12:12:08.227279+0000","last_clean":"2024-04-06T12:12:08.227279+0000","last_became_active":"2024-04-06T12:12:04.215021+0000","last_became_peered":"2024-04-06T12:12:04.215021+0000","last_unstale":"2024-04-06T12:12:08.227279+0000","last_undegraded":"2024-04-06T12:12:08.227279+0000","last_fullsized":"2024-04-06T12:12:08.227279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:09:38.014527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.229099+0000","last_change":"2024-04-06T12:12:04.225406+0000","last_active":"2024-04-06T12:12:08.229099+0000","last_peered":"2024-04-06T12:12:08.229099+0000","last_clean":"2024-04-06T12:12:08.229099+0000","last_became_active":"2024-04-06T12:12:04.224950+0000","last_became_peered":"2024-04-06T12:12:04.224950+0000","last_unstale":"2024-04-06T12:12:08.229099+0000","last_undegraded":"2024-04-06T12:12:08.229099+0000","last_fullsized":"2024-04-06T12:12:08.229099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:55.517965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228664+0000","last_change":"2024-04-06T12:12:04.217691+0000","last_active":"2024-04-06T12:12:08.228664+0000","last_peered":"2024-04-06T12:12:08.228664+0000","last_clean":"2024-04-06T12:12:08.228664+0000","last_became_active":"2024-04-06T12:12:04.217425+0000","last_became_peered":"2024-04-06T12:12:04.217425+0000","last_unstale":"2024-04-06T12:12:08.228664+0000","last_undegraded":"2024-04-06T12:12:08.228664+0000","last_fullsized":"2024-04-06T12:12:08.228664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:10.512788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224434+0000","last_change":"2024-04-06T12:12:04.217807+0000","last_active":"2024-04-06T12:12:08.224434+0000","last_peered":"2024-04-06T12:12:08.224434+0000","last_clean":"2024-04-06T12:12:08.224434+0000","last_became_active":"2024-04-06T12:12:04.217533+0000","last_became_peered":"2024-04-06T12:12:04.217533+0000","last_unstale":"2024-04-06T12:12:08.224434+0000","last_undegraded":"2024-04-06T12:12:08.224434+0000","last_fullsized":"2024-04-06T12:12:08.224434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:43:39.601900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247241+0000","last_change":"2024-04-06T12:12:08.242261+0000","last_active":"2024-04-06T12:12:08.247241+0000","last_peered":"2024-04-06T12:12:08.247241+0000","last_clean":"2024-04-06T12:12:08.247241+0000","last_became_active":"2024-04-06T12:12:08.241896+0000","last_became_peered":"2024-04-06T12:12:08.241896+0000","last_unstale":"2024-04-06T12:12:08.247241+0000","last_undegraded":"2024-04-06T12:12:08.247241+0000","last_fullsized":"2024-04-06T12:12:08.247241+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:31:31.349113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232576+0000","last_change":"2024-04-06T12:12:03.799562+0000","last_active":"2024-04-06T12:12:08.232576+0000","last_peered":"2024-04-06T12:12:08.232576+0000","last_clean":"2024-04-06T12:12:08.232576+0000","last_became_active":"2024-04-06T12:12:03.799298+0000","last_became_peered":"2024-04-06T12:12:03.799298+0000","last_unstale":"2024-04-06T12:12:08.232576+0000","last_undegraded":"2024-04-06T12:12:08.232576+0000","last_fullsized":"2024-04-06T12:12:08.232576+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:05:14.778375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232962+0000","last_change":"2024-04-06T12:12:04.217203+0000","last_active":"2024-04-06T12:12:08.232962+0000","last_peered":"2024-04-06T12:12:08.232962+0000","last_clean":"2024-04-06T12:12:08.232962+0000","last_became_active":"2024-04-06T12:12:04.216782+0000","last_became_peered":"2024-04-06T12:12:04.216782+0000","last_unstale":"2024-04-06T12:12:08.232962+0000","last_undegraded":"2024-04-06T12:12:08.232962+0000","last_fullsized":"2024-04-06T12:12:08.232962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:55.517965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.255247+0000","last_change":"2024-04-06T12:12:08.251632+0000","last_active":"2024-04-06T12:12:08.255247+0000","last_peered":"2024-04-06T12:12:08.255247+0000","last_clean":"2024-04-06T12:12:08.255247+0000","last_became_active":"2024-04-06T12:12:08.251238+0000","last_became_peered":"2024-04-06T12:12:08.251238+0000","last_unstale":"2024-04-06T12:12:08.255247+0000","last_undegraded":"2024-04-06T12:12:08.255247+0000","last_fullsized":"2024-04-06T12:12:08.255247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:32:02.566075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.233059+0000","last_change":"2024-04-06T12:12:03.792714+0000","last_active":"2024-04-06T12:12:08.233059+0000","last_peered":"2024-04-06T12:12:08.233059+0000","last_clean":"2024-04-06T12:12:08.233059+0000","last_became_active":"2024-04-06T12:12:03.792445+0000","last_became_peered":"2024-04-06T12:12:03.792445+0000","last_unstale":"2024-04-06T12:12:08.233059+0000","last_undegraded":"2024-04-06T12:12:08.233059+0000","last_fullsized":"2024-04-06T12:12:08.233059+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:40:20.784453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232461+0000","last_change":"2024-04-06T12:12:04.215439+0000","last_active":"2024-04-06T12:12:08.232461+0000","last_peered":"2024-04-06T12:12:08.232461+0000","last_clean":"2024-04-06T12:12:08.232461+0000","last_became_active":"2024-04-06T12:12:04.214846+0000","last_became_peered":"2024-04-06T12:12:04.214846+0000","last_unstale":"2024-04-06T12:12:08.232461+0000","last_undegraded":"2024-04-06T12:12:08.232461+0000","last_fullsized":"2024-04-06T12:12:08.232461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.250137+0000","last_change":"2024-04-06T12:12:08.248890+0000","last_active":"2024-04-06T12:12:08.250137+0000","last_peered":"2024-04-06T12:12:08.250137+0000","last_clean":"2024-04-06T12:12:08.250137+0000","last_became_active":"2024-04-06T12:12:08.248470+0000","last_became_peered":"2024-04-06T12:12:08.248470+0000","last_unstale":"2024-04-06T12:12:08.250137+0000","last_undegraded":"2024-04-06T12:12:08.250137+0000","last_fullsized":"2024-04-06T12:12:08.250137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:06.085793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223703+0000","last_change":"2024-04-06T12:12:03.799557+0000","last_active":"2024-04-06T12:12:08.223703+0000","last_peered":"2024-04-06T12:12:08.223703+0000","last_clean":"2024-04-06T12:12:08.223703+0000","last_became_active":"2024-04-06T12:12:03.799271+0000","last_became_peered":"2024-04-06T12:12:03.799271+0000","last_unstale":"2024-04-06T12:12:08.223703+0000","last_undegraded":"2024-04-06T12:12:08.223703+0000","last_fullsized":"2024-04-06T12:12:08.223703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:26:21.506491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227705+0000","last_change":"2024-04-06T12:12:04.215431+0000","last_active":"2024-04-06T12:12:08.227705+0000","last_peered":"2024-04-06T12:12:08.227705+0000","last_clean":"2024-04-06T12:12:08.227705+0000","last_became_active":"2024-04-06T12:12:04.214459+0000","last_became_peered":"2024-04-06T12:12:04.214459+0000","last_unstale":"2024-04-06T12:12:08.227705+0000","last_undegraded":"2024-04-06T12:12:08.227705+0000","last_fullsized":"2024-04-06T12:12:08.227705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.252843+0000","last_change":"2024-04-06T12:12:08.249210+0000","last_active":"2024-04-06T12:12:08.252843+0000","last_peered":"2024-04-06T12:12:08.252843+0000","last_clean":"2024-04-06T12:12:08.252843+0000","last_became_active":"2024-04-06T12:12:08.248661+0000","last_became_peered":"2024-04-06T12:12:08.248661+0000","last_unstale":"2024-04-06T12:12:08.252843+0000","last_undegraded":"2024-04-06T12:12:08.252843+0000","last_fullsized":"2024-04-06T12:12:08.252843+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:29.953853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224295+0000","last_change":"2024-04-06T12:12:03.800006+0000","last_active":"2024-04-06T12:12:08.224295+0000","last_peered":"2024-04-06T12:12:08.224295+0000","last_clean":"2024-04-06T12:12:08.224295+0000","last_became_active":"2024-04-06T12:12:03.799598+0000","last_became_peered":"2024-04-06T12:12:03.799598+0000","last_unstale":"2024-04-06T12:12:08.224295+0000","last_undegraded":"2024-04-06T12:12:08.224295+0000","last_fullsized":"2024-04-06T12:12:08.224295+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:04.938049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.229419+0000","last_change":"2024-04-06T12:12:04.220105+0000","last_active":"2024-04-06T12:12:08.229419+0000","last_peered":"2024-04-06T12:12:08.229419+0000","last_clean":"2024-04-06T12:12:08.229419+0000","last_became_active":"2024-04-06T12:12:04.219876+0000","last_became_peered":"2024-04-06T12:12:04.219876+0000","last_unstale":"2024-04-06T12:12:08.229419+0000","last_undegraded":"2024-04-06T12:12:08.229419+0000","last_fullsized":"2024-04-06T12:12:08.229419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.249020+0000","last_change":"2024-04-06T12:12:08.246410+0000","last_active":"2024-04-06T12:12:08.249020+0000","last_peered":"2024-04-06T12:12:08.249020+0000","last_clean":"2024-04-06T12:12:08.249020+0000","last_became_active":"2024-04-06T12:12:08.246057+0000","last_became_peered":"2024-04-06T12:12:08.246057+0000","last_unstale":"2024-04-06T12:12:08.249020+0000","last_undegraded":"2024-04-06T12:12:08.249020+0000","last_fullsized":"2024-04-06T12:12:08.249020+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:16.183719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232695+0000","last_change":"2024-04-06T12:12:03.798850+0000","last_active":"2024-04-06T12:12:08.232695+0000","last_peered":"2024-04-06T12:12:08.232695+0000","last_clean":"2024-04-06T12:12:08.232695+0000","last_became_active":"2024-04-06T12:12:03.798550+0000","last_became_peered":"2024-04-06T12:12:03.798550+0000","last_unstale":"2024-04-06T12:12:08.232695+0000","last_undegraded":"2024-04-06T12:12:08.232695+0000","last_fullsized":"2024-04-06T12:12:08.232695+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:05:14.778375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232223+0000","last_change":"2024-04-06T12:12:04.212276+0000","last_active":"2024-04-06T12:12:08.232223+0000","last_peered":"2024-04-06T12:12:08.232223+0000","last_clean":"2024-04-06T12:12:08.232223+0000","last_became_active":"2024-04-06T12:12:04.211681+0000","last_became_peered":"2024-04-06T12:12:04.211681+0000","last_unstale":"2024-04-06T12:12:08.232223+0000","last_undegraded":"2024-04-06T12:12:08.232223+0000","last_fullsized":"2024-04-06T12:12:08.232223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:21:43.141955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.248388+0000","last_change":"2024-04-06T12:12:08.244209+0000","last_active":"2024-04-06T12:12:08.248388+0000","last_peered":"2024-04-06T12:12:08.248388+0000","last_clean":"2024-04-06T12:12:08.248388+0000","last_became_active":"2024-04-06T12:12:08.243274+0000","last_became_peered":"2024-04-06T12:12:08.243274+0000","last_unstale":"2024-04-06T12:12:08.248388+0000","last_undegraded":"2024-04-06T12:12:08.248388+0000","last_fullsized":"2024-04-06T12:12:08.248388+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:03.696838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223468+0000","last_change":"2024-04-06T12:12:03.796799+0000","last_active":"2024-04-06T12:12:08.223468+0000","last_peered":"2024-04-06T12:12:08.223468+0000","last_clean":"2024-04-06T12:12:08.223468+0000","last_became_active":"2024-04-06T12:12:03.796536+0000","last_became_peered":"2024-04-06T12:12:03.796536+0000","last_unstale":"2024-04-06T12:12:08.223468+0000","last_undegraded":"2024-04-06T12:12:08.223468+0000","last_fullsized":"2024-04-06T12:12:08.223468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:15.699305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234439+0000","last_change":"2024-04-06T12:12:04.220607+0000","last_active":"2024-04-06T12:12:08.234439+0000","last_peered":"2024-04-06T12:12:08.234439+0000","last_clean":"2024-04-06T12:12:08.234439+0000","last_became_active":"2024-04-06T12:12:04.220242+0000","last_became_peered":"2024-04-06T12:12:04.220242+0000","last_unstale":"2024-04-06T12:12:08.234439+0000","last_undegraded":"2024-04-06T12:12:08.234439+0000","last_fullsized":"2024-04-06T12:12:08.234439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:00.258491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.243307+0000","last_change":"2024-04-06T12:12:08.237694+0000","last_active":"2024-04-06T12:12:08.243307+0000","last_peered":"2024-04-06T12:12:08.243307+0000","last_clean":"2024-04-06T12:12:08.243307+0000","last_became_active":"2024-04-06T12:12:08.237190+0000","last_became_peered":"2024-04-06T12:12:08.237190+0000","last_unstale":"2024-04-06T12:12:08.243307+0000","last_undegraded":"2024-04-06T12:12:08.243307+0000","last_fullsized":"2024-04-06T12:12:08.243307+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:50:32.228428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232240+0000","last_change":"2024-04-06T12:12:03.800903+0000","last_active":"2024-04-06T12:12:08.232240+0000","last_peered":"2024-04-06T12:12:08.232240+0000","last_clean":"2024-04-06T12:12:08.232240+0000","last_became_active":"2024-04-06T12:12:03.800590+0000","last_became_peered":"2024-04-06T12:12:03.800590+0000","last_unstale":"2024-04-06T12:12:08.232240+0000","last_undegraded":"2024-04-06T12:12:08.232240+0000","last_fullsized":"2024-04-06T12:12:08.232240+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:00:31.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232899+0000","last_change":"2024-04-06T12:12:04.216144+0000","last_active":"2024-04-06T12:12:08.232899+0000","last_peered":"2024-04-06T12:12:08.232899+0000","last_clean":"2024-04-06T12:12:08.232899+0000","last_became_active":"2024-04-06T12:12:04.215885+0000","last_became_peered":"2024-04-06T12:12:04.215885+0000","last_unstale":"2024-04-06T12:12:08.232899+0000","last_undegraded":"2024-04-06T12:12:08.232899+0000","last_fullsized":"2024-04-06T12:12:08.232899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:36:25.380472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256611+0000","last_change":"2024-04-06T12:12:08.252363+0000","last_active":"2024-04-06T12:12:08.256611+0000","last_peered":"2024-04-06T12:12:08.256611+0000","last_clean":"2024-04-06T12:12:08.256611+0000","last_became_active":"2024-04-06T12:12:08.252029+0000","last_became_peered":"2024-04-06T12:12:08.252029+0000","last_unstale":"2024-04-06T12:12:08.256611+0000","last_undegraded":"2024-04-06T12:12:08.256611+0000","last_fullsized":"2024-04-06T12:12:08.256611+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:35:54.255608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227386+0000","last_change":"2024-04-06T12:12:03.800462+0000","last_active":"2024-04-06T12:12:08.227386+0000","last_peered":"2024-04-06T12:12:08.227386+0000","last_clean":"2024-04-06T12:12:08.227386+0000","last_became_active":"2024-04-06T12:12:03.800210+0000","last_became_peered":"2024-04-06T12:12:03.800210+0000","last_unstale":"2024-04-06T12:12:08.227386+0000","last_undegraded":"2024-04-06T12:12:08.227386+0000","last_fullsized":"2024-04-06T12:12:08.227386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:00:31.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222742+0000","last_change":"2024-04-06T12:12:04.216700+0000","last_active":"2024-04-06T12:12:08.222742+0000","last_peered":"2024-04-06T12:12:08.222742+0000","last_clean":"2024-04-06T12:12:08.222742+0000","last_became_active":"2024-04-06T12:12:04.216435+0000","last_became_peered":"2024-04-06T12:12:04.216435+0000","last_unstale":"2024-04-06T12:12:08.222742+0000","last_undegraded":"2024-04-06T12:12:08.222742+0000","last_fullsized":"2024-04-06T12:12:08.222742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:36:25.380472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247221+0000","last_change":"2024-04-06T12:12:08.242079+0000","last_active":"2024-04-06T12:12:08.247221+0000","last_peered":"2024-04-06T12:12:08.247221+0000","last_clean":"2024-04-06T12:12:08.247221+0000","last_became_active":"2024-04-06T12:12:08.241648+0000","last_became_peered":"2024-04-06T12:12:08.241648+0000","last_unstale":"2024-04-06T12:12:08.247221+0000","last_undegraded":"2024-04-06T12:12:08.247221+0000","last_fullsized":"2024-04-06T12:12:08.247221+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:37:57.999646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224055+0000","last_change":"2024-04-06T12:12:03.799875+0000","last_active":"2024-04-06T12:12:08.224055+0000","last_peered":"2024-04-06T12:12:08.224055+0000","last_clean":"2024-04-06T12:12:08.224055+0000","last_became_active":"2024-04-06T12:12:03.799562+0000","last_became_peered":"2024-04-06T12:12:03.799562+0000","last_unstale":"2024-04-06T12:12:08.224055+0000","last_undegraded":"2024-04-06T12:12:08.224055+0000","last_fullsized":"2024-04-06T12:12:08.224055+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:08:55.770454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.229090+0000","last_change":"2024-04-06T12:12:04.225142+0000","last_active":"2024-04-06T12:12:08.229090+0000","last_peered":"2024-04-06T12:12:08.229090+0000","last_clean":"2024-04-06T12:12:08.229090+0000","last_became_active":"2024-04-06T12:12:04.224838+0000","last_became_peered":"2024-04-06T12:12:04.224838+0000","last_unstale":"2024-04-06T12:12:08.229090+0000","last_undegraded":"2024-04-06T12:12:08.229090+0000","last_fullsized":"2024-04-06T12:12:08.229090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:21:16.167775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247222+0000","last_change":"2024-04-06T12:12:08.244037+0000","last_active":"2024-04-06T12:12:08.247222+0000","last_peered":"2024-04-06T12:12:08.247222+0000","last_clean":"2024-04-06T12:12:08.247222+0000","last_became_active":"2024-04-06T12:12:08.243685+0000","last_became_peered":"2024-04-06T12:12:08.243685+0000","last_unstale":"2024-04-06T12:12:08.247222+0000","last_undegraded":"2024-04-06T12:12:08.247222+0000","last_fullsized":"2024-04-06T12:12:08.247222+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:02:51.216167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228999+0000","last_change":"2024-04-06T12:12:03.799591+0000","last_active":"2024-04-06T12:12:08.228999+0000","last_peered":"2024-04-06T12:12:08.228999+0000","last_clean":"2024-04-06T12:12:08.228999+0000","last_became_active":"2024-04-06T12:12:03.799333+0000","last_became_peered":"2024-04-06T12:12:03.799333+0000","last_unstale":"2024-04-06T12:12:08.228999+0000","last_undegraded":"2024-04-06T12:12:08.228999+0000","last_fullsized":"2024-04-06T12:12:08.228999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:11.635313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232670+0000","last_change":"2024-04-06T12:12:04.215620+0000","last_active":"2024-04-06T12:12:08.232670+0000","last_peered":"2024-04-06T12:12:08.232670+0000","last_clean":"2024-04-06T12:12:08.232670+0000","last_became_active":"2024-04-06T12:12:04.215189+0000","last_became_peered":"2024-04-06T12:12:04.215189+0000","last_unstale":"2024-04-06T12:12:08.232670+0000","last_undegraded":"2024-04-06T12:12:08.232670+0000","last_fullsized":"2024-04-06T12:12:08.232670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247852+0000","last_change":"2024-04-06T12:12:08.246657+0000","last_active":"2024-04-06T12:12:08.247852+0000","last_peered":"2024-04-06T12:12:08.247852+0000","last_clean":"2024-04-06T12:12:08.247852+0000","last_became_active":"2024-04-06T12:12:08.246310+0000","last_became_peered":"2024-04-06T12:12:08.246310+0000","last_unstale":"2024-04-06T12:12:08.247852+0000","last_undegraded":"2024-04-06T12:12:08.247852+0000","last_fullsized":"2024-04-06T12:12:08.247852+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:57:02.114694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223115+0000","last_change":"2024-04-06T12:12:03.802440+0000","last_active":"2024-04-06T12:12:08.223115+0000","last_peered":"2024-04-06T12:12:08.223115+0000","last_clean":"2024-04-06T12:12:08.223115+0000","last_became_active":"2024-04-06T12:12:03.802193+0000","last_became_peered":"2024-04-06T12:12:03.802193+0000","last_unstale":"2024-04-06T12:12:08.223115+0000","last_undegraded":"2024-04-06T12:12:08.223115+0000","last_fullsized":"2024-04-06T12:12:08.223115+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:04.938049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232783+0000","last_change":"2024-04-06T12:12:04.225740+0000","last_active":"2024-04-06T12:12:08.232783+0000","last_peered":"2024-04-06T12:12:08.232783+0000","last_clean":"2024-04-06T12:12:08.232783+0000","last_became_active":"2024-04-06T12:12:04.224605+0000","last_became_peered":"2024-04-06T12:12:04.224605+0000","last_unstale":"2024-04-06T12:12:08.232783+0000","last_undegraded":"2024-04-06T12:12:08.232783+0000","last_fullsized":"2024-04-06T12:12:08.232783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:26:22.511847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.245748+0000","last_change":"2024-04-06T12:12:08.242260+0000","last_active":"2024-04-06T12:12:08.245748+0000","last_peered":"2024-04-06T12:12:08.245748+0000","last_clean":"2024-04-06T12:12:08.245748+0000","last_became_active":"2024-04-06T12:12:08.241863+0000","last_became_peered":"2024-04-06T12:12:08.241863+0000","last_unstale":"2024-04-06T12:12:08.245748+0000","last_undegraded":"2024-04-06T12:12:08.245748+0000","last_fullsized":"2024-04-06T12:12:08.245748+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:06.085793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-06T12:12:13.609454+0000","last_change":"2024-04-06T12:12:03.799037+0000","last_active":"2024-04-06T12:12:13.609454+0000","last_peered":"2024-04-06T12:12:13.609454+0000","last_clean":"2024-04-06T12:12:13.609454+0000","last_became_active":"2024-04-06T12:12:03.798723+0000","last_became_peered":"2024-04-06T12:12:03.798723+0000","last_unstale":"2024-04-06T12:12:13.609454+0000","last_undegraded":"2024-04-06T12:12:13.609454+0000","last_fullsized":"2024-04-06T12:12:13.609454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:00:31.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3796,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224345+0000","last_change":"2024-04-06T12:12:04.226914+0000","last_active":"2024-04-06T12:12:08.224345+0000","last_peered":"2024-04-06T12:12:08.224345+0000","last_clean":"2024-04-06T12:12:08.224345+0000","last_became_active":"2024-04-06T12:12:04.226421+0000","last_became_peered":"2024-04-06T12:12:04.226421+0000","last_unstale":"2024-04-06T12:12:08.224345+0000","last_undegraded":"2024-04-06T12:12:08.224345+0000","last_fullsized":"2024-04-06T12:12:08.224345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:45:10.755538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247061+0000","last_change":"2024-04-06T12:12:08.242923+0000","last_active":"2024-04-06T12:12:08.247061+0000","last_peered":"2024-04-06T12:12:08.247061+0000","last_clean":"2024-04-06T12:12:08.247061+0000","last_became_active":"2024-04-06T12:12:08.242491+0000","last_became_peered":"2024-04-06T12:12:08.242491+0000","last_unstale":"2024-04-06T12:12:08.247061+0000","last_undegraded":"2024-04-06T12:12:08.247061+0000","last_fullsized":"2024-04-06T12:12:08.247061+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:57:02.114694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234681+0000","last_change":"2024-04-06T12:12:03.796757+0000","last_active":"2024-04-06T12:12:08.234681+0000","last_peered":"2024-04-06T12:12:08.234681+0000","last_clean":"2024-04-06T12:12:08.234681+0000","last_became_active":"2024-04-06T12:12:03.796519+0000","last_became_peered":"2024-04-06T12:12:03.796519+0000","last_unstale":"2024-04-06T12:12:08.234681+0000","last_undegraded":"2024-04-06T12:12:08.234681+0000","last_fullsized":"2024-04-06T12:12:08.234681+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:40:20.784453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.235485+0000","last_change":"2024-04-06T12:12:04.224445+0000","last_active":"2024-04-06T12:12:08.235485+0000","last_peered":"2024-04-06T12:12:08.235485+0000","last_clean":"2024-04-06T12:12:08.235485+0000","last_became_active":"2024-04-06T12:12:04.224153+0000","last_became_peered":"2024-04-06T12:12:04.224153+0000","last_unstale":"2024-04-06T12:12:08.235485+0000","last_undegraded":"2024-04-06T12:12:08.235485+0000","last_fullsized":"2024-04-06T12:12:08.235485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:15.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256724+0000","last_change":"2024-04-06T12:12:08.251536+0000","last_active":"2024-04-06T12:12:08.256724+0000","last_peered":"2024-04-06T12:12:08.256724+0000","last_clean":"2024-04-06T12:12:08.256724+0000","last_became_active":"2024-04-06T12:12:08.251142+0000","last_became_peered":"2024-04-06T12:12:08.251142+0000","last_unstale":"2024-04-06T12:12:08.256724+0000","last_undegraded":"2024-04-06T12:12:08.256724+0000","last_fullsized":"2024-04-06T12:12:08.256724+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:48:55.155611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228030+0000","last_change":"2024-04-06T12:12:03.797816+0000","last_active":"2024-04-06T12:12:08.228030+0000","last_peered":"2024-04-06T12:12:08.228030+0000","last_clean":"2024-04-06T12:12:08.228030+0000","last_became_active":"2024-04-06T12:12:03.797540+0000","last_became_peered":"2024-04-06T12:12:03.797540+0000","last_unstale":"2024-04-06T12:12:08.228030+0000","last_undegraded":"2024-04-06T12:12:08.228030+0000","last_fullsized":"2024-04-06T12:12:08.228030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:15.699305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224078+0000","last_change":"2024-04-06T12:12:04.223703+0000","last_active":"2024-04-06T12:12:08.224078+0000","last_peered":"2024-04-06T12:12:08.224078+0000","last_clean":"2024-04-06T12:12:08.224078+0000","last_became_active":"2024-04-06T12:12:04.223394+0000","last_became_peered":"2024-04-06T12:12:04.223394+0000","last_unstale":"2024-04-06T12:12:08.224078+0000","last_undegraded":"2024-04-06T12:12:08.224078+0000","last_fullsized":"2024-04-06T12:12:08.224078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:19:38.810657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256643+0000","last_change":"2024-04-06T12:12:08.250454+0000","last_active":"2024-04-06T12:12:08.256643+0000","last_peered":"2024-04-06T12:12:08.256643+0000","last_clean":"2024-04-06T12:12:08.256643+0000","last_became_active":"2024-04-06T12:12:08.250052+0000","last_became_peered":"2024-04-06T12:12:08.250052+0000","last_unstale":"2024-04-06T12:12:08.256643+0000","last_undegraded":"2024-04-06T12:12:08.256643+0000","last_fullsized":"2024-04-06T12:12:08.256643+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:11:04.976956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223382+0000","last_change":"2024-04-06T12:12:03.799361+0000","last_active":"2024-04-06T12:12:08.223382+0000","last_peered":"2024-04-06T12:12:08.223382+0000","last_clean":"2024-04-06T12:12:08.223382+0000","last_became_active":"2024-04-06T12:12:03.798981+0000","last_became_peered":"2024-04-06T12:12:03.798981+0000","last_unstale":"2024-04-06T12:12:08.223382+0000","last_undegraded":"2024-04-06T12:12:08.223382+0000","last_fullsized":"2024-04-06T12:12:08.223382+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:11.635313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224104+0000","last_change":"2024-04-06T12:12:04.223535+0000","last_active":"2024-04-06T12:12:08.224104+0000","last_peered":"2024-04-06T12:12:08.224104+0000","last_clean":"2024-04-06T12:12:08.224104+0000","last_became_active":"2024-04-06T12:12:04.217966+0000","last_became_peered":"2024-04-06T12:12:04.217966+0000","last_unstale":"2024-04-06T12:12:08.224104+0000","last_undegraded":"2024-04-06T12:12:08.224104+0000","last_fullsized":"2024-04-06T12:12:08.224104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:15.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256634+0000","last_change":"2024-04-06T12:12:08.252578+0000","last_active":"2024-04-06T12:12:08.256634+0000","last_peered":"2024-04-06T12:12:08.256634+0000","last_clean":"2024-04-06T12:12:08.256634+0000","last_became_active":"2024-04-06T12:12:08.252149+0000","last_became_peered":"2024-04-06T12:12:08.252149+0000","last_unstale":"2024-04-06T12:12:08.256634+0000","last_undegraded":"2024-04-06T12:12:08.256634+0000","last_fullsized":"2024-04-06T12:12:08.256634+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:06.085793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.226746+0000","last_change":"2024-04-06T12:12:03.799994+0000","last_active":"2024-04-06T12:12:08.226746+0000","last_peered":"2024-04-06T12:12:08.226746+0000","last_clean":"2024-04-06T12:12:08.226746+0000","last_became_active":"2024-04-06T12:12:03.799663+0000","last_became_peered":"2024-04-06T12:12:03.799663+0000","last_unstale":"2024-04-06T12:12:08.226746+0000","last_undegraded":"2024-04-06T12:12:08.226746+0000","last_fullsized":"2024-04-06T12:12:08.226746+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:08:55.770454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223501+0000","last_change":"2024-04-06T12:12:04.223425+0000","last_active":"2024-04-06T12:12:08.223501+0000","last_peered":"2024-04-06T12:12:08.223501+0000","last_clean":"2024-04-06T12:12:08.223501+0000","last_became_active":"2024-04-06T12:12:04.223101+0000","last_became_peered":"2024-04-06T12:12:04.223101+0000","last_unstale":"2024-04-06T12:12:08.223501+0000","last_undegraded":"2024-04-06T12:12:08.223501+0000","last_fullsized":"2024-04-06T12:12:08.223501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.250437+0000","last_change":"2024-04-06T12:12:08.249078+0000","last_active":"2024-04-06T12:12:08.250437+0000","last_peered":"2024-04-06T12:12:08.250437+0000","last_clean":"2024-04-06T12:12:08.250437+0000","last_became_active":"2024-04-06T12:12:08.248512+0000","last_became_peered":"2024-04-06T12:12:08.248512+0000","last_unstale":"2024-04-06T12:12:08.250437+0000","last_undegraded":"2024-04-06T12:12:08.250437+0000","last_fullsized":"2024-04-06T12:12:08.250437+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:24:33.553367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.230408+0000","last_change":"2024-04-06T12:12:03.793832+0000","last_active":"2024-04-06T12:12:08.230408+0000","last_peered":"2024-04-06T12:12:08.230408+0000","last_clean":"2024-04-06T12:12:08.230408+0000","last_became_active":"2024-04-06T12:12:03.793518+0000","last_became_peered":"2024-04-06T12:12:03.793518+0000","last_unstale":"2024-04-06T12:12:08.230408+0000","last_undegraded":"2024-04-06T12:12:08.230408+0000","last_fullsized":"2024-04-06T12:12:08.230408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:37:27.257815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224591+0000","last_change":"2024-04-06T12:12:04.224260+0000","last_active":"2024-04-06T12:12:08.224591+0000","last_peered":"2024-04-06T12:12:08.224591+0000","last_clean":"2024-04-06T12:12:08.224591+0000","last_became_active":"2024-04-06T12:12:04.223968+0000","last_became_peered":"2024-04-06T12:12:04.223968+0000","last_unstale":"2024-04-06T12:12:08.224591+0000","last_undegraded":"2024-04-06T12:12:08.224591+0000","last_fullsized":"2024-04-06T12:12:08.224591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.245727+0000","last_change":"2024-04-06T12:12:08.242094+0000","last_active":"2024-04-06T12:12:08.245727+0000","last_peered":"2024-04-06T12:12:08.245727+0000","last_clean":"2024-04-06T12:12:08.245727+0000","last_became_active":"2024-04-06T12:12:08.241707+0000","last_became_peered":"2024-04-06T12:12:08.241707+0000","last_unstale":"2024-04-06T12:12:08.245727+0000","last_undegraded":"2024-04-06T12:12:08.245727+0000","last_fullsized":"2024-04-06T12:12:08.245727+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:47:51.958478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222772+0000","last_change":"2024-04-06T12:12:03.800106+0000","last_active":"2024-04-06T12:12:08.222772+0000","last_peered":"2024-04-06T12:12:08.222772+0000","last_clean":"2024-04-06T12:12:08.222772+0000","last_became_active":"2024-04-06T12:12:03.799843+0000","last_became_peered":"2024-04-06T12:12:03.799843+0000","last_unstale":"2024-04-06T12:12:08.222772+0000","last_undegraded":"2024-04-06T12:12:08.222772+0000","last_fullsized":"2024-04-06T12:12:08.222772+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:26:21.506491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.229502+0000","last_change":"2024-04-06T12:12:04.214371+0000","last_active":"2024-04-06T12:12:08.229502+0000","last_peered":"2024-04-06T12:12:08.229502+0000","last_clean":"2024-04-06T12:12:08.229502+0000","last_became_active":"2024-04-06T12:12:04.214095+0000","last_became_peered":"2024-04-06T12:12:04.214095+0000","last_unstale":"2024-04-06T12:12:08.229502+0000","last_undegraded":"2024-04-06T12:12:08.229502+0000","last_fullsized":"2024-04-06T12:12:08.229502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:45:10.755538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.258734+0000","last_change":"2024-04-06T12:12:08.255356+0000","last_active":"2024-04-06T12:12:08.258734+0000","last_peered":"2024-04-06T12:12:08.258734+0000","last_clean":"2024-04-06T12:12:08.258734+0000","last_became_active":"2024-04-06T12:12:08.254148+0000","last_became_peered":"2024-04-06T12:12:08.254148+0000","last_unstale":"2024-04-06T12:12:08.258734+0000","last_undegraded":"2024-04-06T12:12:08.258734+0000","last_fullsized":"2024-04-06T12:12:08.258734+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:35:54.255608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227821+0000","last_change":"2024-04-06T12:12:03.801770+0000","last_active":"2024-04-06T12:12:08.227821+0000","last_peered":"2024-04-06T12:12:08.227821+0000","last_clean":"2024-04-06T12:12:08.227821+0000","last_became_active":"2024-04-06T12:12:03.801494+0000","last_became_peered":"2024-04-06T12:12:03.801494+0000","last_unstale":"2024-04-06T12:12:08.227821+0000","last_undegraded":"2024-04-06T12:12:08.227821+0000","last_fullsized":"2024-04-06T12:12:08.227821+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:08:55.770454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232799+0000","last_change":"2024-04-06T12:12:04.217734+0000","last_active":"2024-04-06T12:12:08.232799+0000","last_peered":"2024-04-06T12:12:08.232799+0000","last_clean":"2024-04-06T12:12:08.232799+0000","last_became_active":"2024-04-06T12:12:04.216838+0000","last_became_peered":"2024-04-06T12:12:04.216838+0000","last_unstale":"2024-04-06T12:12:08.232799+0000","last_undegraded":"2024-04-06T12:12:08.232799+0000","last_fullsized":"2024-04-06T12:12:08.232799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:05.943405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.258759+0000","last_change":"2024-04-06T12:12:08.255965+0000","last_active":"2024-04-06T12:12:08.258759+0000","last_peered":"2024-04-06T12:12:08.258759+0000","last_clean":"2024-04-06T12:12:08.258759+0000","last_became_active":"2024-04-06T12:12:08.251143+0000","last_became_peered":"2024-04-06T12:12:08.251143+0000","last_unstale":"2024-04-06T12:12:08.258759+0000","last_undegraded":"2024-04-06T12:12:08.258759+0000","last_fullsized":"2024-04-06T12:12:08.258759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:21:01.506046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222665+0000","last_change":"2024-04-06T12:12:04.208976+0000","last_active":"2024-04-06T12:12:08.222665+0000","last_peered":"2024-04-06T12:12:08.222665+0000","last_clean":"2024-04-06T12:12:08.222665+0000","last_became_active":"2024-04-06T12:12:04.208524+0000","last_became_peered":"2024-04-06T12:12:04.208524+0000","last_unstale":"2024-04-06T12:12:08.222665+0000","last_undegraded":"2024-04-06T12:12:08.222665+0000","last_fullsized":"2024-04-06T12:12:08.222665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.226929+0000","last_change":"2024-04-06T12:12:03.796992+0000","last_active":"2024-04-06T12:12:08.226929+0000","last_peered":"2024-04-06T12:12:08.226929+0000","last_clean":"2024-04-06T12:12:08.226929+0000","last_became_active":"2024-04-06T12:12:03.795824+0000","last_became_peered":"2024-04-06T12:12:03.795824+0000","last_unstale":"2024-04-06T12:12:08.226929+0000","last_undegraded":"2024-04-06T12:12:08.226929+0000","last_fullsized":"2024-04-06T12:12:08.226929+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:11.635313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.246589+0000","last_change":"2024-04-06T12:12:08.244008+0000","last_active":"2024-04-06T12:12:08.246589+0000","last_peered":"2024-04-06T12:12:08.246589+0000","last_clean":"2024-04-06T12:12:08.246589+0000","last_became_active":"2024-04-06T12:12:08.242895+0000","last_became_peered":"2024-04-06T12:12:08.242895+0000","last_unstale":"2024-04-06T12:12:08.246589+0000","last_undegraded":"2024-04-06T12:12:08.246589+0000","last_fullsized":"2024-04-06T12:12:08.246589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:54:54.771467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232422+0000","last_change":"2024-04-06T12:12:04.215137+0000","last_active":"2024-04-06T12:12:08.232422+0000","last_peered":"2024-04-06T12:12:08.232422+0000","last_clean":"2024-04-06T12:12:08.232422+0000","last_became_active":"2024-04-06T12:12:04.214872+0000","last_became_peered":"2024-04-06T12:12:04.214872+0000","last_unstale":"2024-04-06T12:12:08.232422+0000","last_undegraded":"2024-04-06T12:12:08.232422+0000","last_fullsized":"2024-04-06T12:12:08.232422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227366+0000","last_change":"2024-04-06T12:12:03.797439+0000","last_active":"2024-04-06T12:12:08.227366+0000","last_peered":"2024-04-06T12:12:08.227366+0000","last_clean":"2024-04-06T12:12:08.227366+0000","last_became_active":"2024-04-06T12:12:03.797176+0000","last_became_peered":"2024-04-06T12:12:03.797176+0000","last_unstale":"2024-04-06T12:12:08.227366+0000","last_undegraded":"2024-04-06T12:12:08.227366+0000","last_fullsized":"2024-04-06T12:12:08.227366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:26:21.506491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223206+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:12:08.223206+0000","last_peered":"2024-04-06T12:12:08.223206+0000","last_clean":"2024-04-06T12:12:08.223206+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:12:08.223206+0000","last_undegraded":"2024-04-06T12:12:08.223206+0000","last_fullsized":"2024-04-06T12:12:08.223206+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.252451+0000","last_change":"2024-04-06T12:12:08.248645+0000","last_active":"2024-04-06T12:12:08.252451+0000","last_peered":"2024-04-06T12:12:08.252451+0000","last_clean":"2024-04-06T12:12:08.252451+0000","last_became_active":"2024-04-06T12:12:08.247420+0000","last_became_peered":"2024-04-06T12:12:08.247420+0000","last_unstale":"2024-04-06T12:12:08.252451+0000","last_undegraded":"2024-04-06T12:12:08.252451+0000","last_fullsized":"2024-04-06T12:12:08.252451+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:31:21.467656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232818+0000","last_change":"2024-04-06T12:12:04.220328+0000","last_active":"2024-04-06T12:12:08.232818+0000","last_peered":"2024-04-06T12:12:08.232818+0000","last_clean":"2024-04-06T12:12:08.232818+0000","last_became_active":"2024-04-06T12:12:04.219948+0000","last_became_peered":"2024-04-06T12:12:04.219948+0000","last_unstale":"2024-04-06T12:12:08.232818+0000","last_undegraded":"2024-04-06T12:12:08.232818+0000","last_fullsized":"2024-04-06T12:12:08.232818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:36:25.380472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223326+0000","last_change":"2024-04-06T12:12:03.800906+0000","last_active":"2024-04-06T12:12:08.223326+0000","last_peered":"2024-04-06T12:12:08.223326+0000","last_clean":"2024-04-06T12:12:08.223326+0000","last_became_active":"2024-04-06T12:12:03.800460+0000","last_became_peered":"2024-04-06T12:12:03.800460+0000","last_unstale":"2024-04-06T12:12:08.223326+0000","last_undegraded":"2024-04-06T12:12:08.223326+0000","last_fullsized":"2024-04-06T12:12:08.223326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:11.635313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.251977+0000","last_change":"2024-04-06T12:12:08.246821+0000","last_active":"2024-04-06T12:12:08.251977+0000","last_peered":"2024-04-06T12:12:08.251977+0000","last_clean":"2024-04-06T12:12:08.251977+0000","last_became_active":"2024-04-06T12:12:08.246154+0000","last_became_peered":"2024-04-06T12:12:08.246154+0000","last_unstale":"2024-04-06T12:12:08.251977+0000","last_undegraded":"2024-04-06T12:12:08.251977+0000","last_fullsized":"2024-04-06T12:12:08.251977+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:47:51.958478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223565+0000","last_change":"2024-04-06T12:12:03.801675+0000","last_active":"2024-04-06T12:12:08.223565+0000","last_peered":"2024-04-06T12:12:08.223565+0000","last_clean":"2024-04-06T12:12:08.223565+0000","last_became_active":"2024-04-06T12:12:03.800427+0000","last_became_peered":"2024-04-06T12:12:03.800427+0000","last_unstale":"2024-04-06T12:12:08.223565+0000","last_undegraded":"2024-04-06T12:12:08.223565+0000","last_fullsized":"2024-04-06T12:12:08.223565+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:21:42.136599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222894+0000","last_change":"2024-04-06T12:12:04.216377+0000","last_active":"2024-04-06T12:12:08.222894+0000","last_peered":"2024-04-06T12:12:08.222894+0000","last_clean":"2024-04-06T12:12:08.222894+0000","last_became_active":"2024-04-06T12:12:04.216084+0000","last_became_peered":"2024-04-06T12:12:04.216084+0000","last_unstale":"2024-04-06T12:12:08.222894+0000","last_undegraded":"2024-04-06T12:12:08.222894+0000","last_fullsized":"2024-04-06T12:12:08.222894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:49:29.774968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.256642+0000","last_change":"2024-04-06T12:12:08.250308+0000","last_active":"2024-04-06T12:12:08.256642+0000","last_peered":"2024-04-06T12:12:08.256642+0000","last_clean":"2024-04-06T12:12:08.256642+0000","last_became_active":"2024-04-06T12:12:08.249904+0000","last_became_peered":"2024-04-06T12:12:08.249904+0000","last_unstale":"2024-04-06T12:12:08.256642+0000","last_undegraded":"2024-04-06T12:12:08.256642+0000","last_fullsized":"2024-04-06T12:12:08.256642+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:35:23.088849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222894+0000","last_change":"2024-04-06T12:12:03.803297+0000","last_active":"2024-04-06T12:12:08.222894+0000","last_peered":"2024-04-06T12:12:08.222894+0000","last_clean":"2024-04-06T12:12:08.222894+0000","last_became_active":"2024-04-06T12:12:03.803053+0000","last_became_peered":"2024-04-06T12:12:03.803053+0000","last_unstale":"2024-04-06T12:12:08.222894+0000","last_undegraded":"2024-04-06T12:12:08.222894+0000","last_fullsized":"2024-04-06T12:12:08.222894+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:21:42.136599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234446+0000","last_change":"2024-04-06T12:12:04.225056+0000","last_active":"2024-04-06T12:12:08.234446+0000","last_peered":"2024-04-06T12:12:08.234446+0000","last_clean":"2024-04-06T12:12:08.234446+0000","last_became_active":"2024-04-06T12:12:04.224635+0000","last_became_peered":"2024-04-06T12:12:04.224635+0000","last_unstale":"2024-04-06T12:12:08.234446+0000","last_undegraded":"2024-04-06T12:12:08.234446+0000","last_fullsized":"2024-04-06T12:12:08.234446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247224+0000","last_change":"2024-04-06T12:12:08.244844+0000","last_active":"2024-04-06T12:12:08.247224+0000","last_peered":"2024-04-06T12:12:08.247224+0000","last_clean":"2024-04-06T12:12:08.247224+0000","last_became_active":"2024-04-06T12:12:08.244422+0000","last_became_peered":"2024-04-06T12:12:08.244422+0000","last_unstale":"2024-04-06T12:12:08.247224+0000","last_undegraded":"2024-04-06T12:12:08.247224+0000","last_fullsized":"2024-04-06T12:12:08.247224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:54:54.771467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223933+0000","last_change":"2024-04-06T12:12:03.800460+0000","last_active":"2024-04-06T12:12:08.223933+0000","last_peered":"2024-04-06T12:12:08.223933+0000","last_clean":"2024-04-06T12:12:08.223933+0000","last_became_active":"2024-04-06T12:12:03.800183+0000","last_became_peered":"2024-04-06T12:12:03.800183+0000","last_unstale":"2024-04-06T12:12:08.223933+0000","last_undegraded":"2024-04-06T12:12:08.223933+0000","last_fullsized":"2024-04-06T12:12:08.223933+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:40:20.784453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.235166+0000","last_change":"2024-04-06T12:12:04.209360+0000","last_active":"2024-04-06T12:12:08.235166+0000","last_peered":"2024-04-06T12:12:08.235166+0000","last_clean":"2024-04-06T12:12:08.235166+0000","last_became_active":"2024-04-06T12:12:04.209105+0000","last_became_peered":"2024-04-06T12:12:04.209105+0000","last_unstale":"2024-04-06T12:12:08.235166+0000","last_undegraded":"2024-04-06T12:12:08.235166+0000","last_fullsized":"2024-04-06T12:12:08.235166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.252713+0000","last_change":"2024-04-06T12:12:08.249325+0000","last_active":"2024-04-06T12:12:08.252713+0000","last_peered":"2024-04-06T12:12:08.252713+0000","last_clean":"2024-04-06T12:12:08.252713+0000","last_became_active":"2024-04-06T12:12:08.248776+0000","last_became_peered":"2024-04-06T12:12:08.248776+0000","last_unstale":"2024-04-06T12:12:08.252713+0000","last_undegraded":"2024-04-06T12:12:08.252713+0000","last_fullsized":"2024-04-06T12:12:08.252713+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:31:21.467656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223337+0000","last_change":"2024-04-06T12:12:03.797556+0000","last_active":"2024-04-06T12:12:08.223337+0000","last_peered":"2024-04-06T12:12:08.223337+0000","last_clean":"2024-04-06T12:12:08.223337+0000","last_became_active":"2024-04-06T12:12:03.797288+0000","last_became_peered":"2024-04-06T12:12:03.797288+0000","last_unstale":"2024-04-06T12:12:08.223337+0000","last_undegraded":"2024-04-06T12:12:08.223337+0000","last_fullsized":"2024-04-06T12:12:08.223337+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:15.699305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.231368+0000","last_change":"2024-04-06T12:12:04.219859+0000","last_active":"2024-04-06T12:12:08.231368+0000","last_peered":"2024-04-06T12:12:08.231368+0000","last_clean":"2024-04-06T12:12:08.231368+0000","last_became_active":"2024-04-06T12:12:04.219470+0000","last_became_peered":"2024-04-06T12:12:04.219470+0000","last_unstale":"2024-04-06T12:12:08.231368+0000","last_undegraded":"2024-04-06T12:12:08.231368+0000","last_fullsized":"2024-04-06T12:12:08.231368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:16.704661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.258673+0000","last_change":"2024-04-06T12:12:08.254506+0000","last_active":"2024-04-06T12:12:08.258673+0000","last_peered":"2024-04-06T12:12:08.258673+0000","last_clean":"2024-04-06T12:12:08.258673+0000","last_became_active":"2024-04-06T12:12:08.250366+0000","last_became_peered":"2024-04-06T12:12:08.250366+0000","last_unstale":"2024-04-06T12:12:08.258673+0000","last_undegraded":"2024-04-06T12:12:08.258673+0000","last_fullsized":"2024-04-06T12:12:08.258673+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:29.953853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222644+0000","last_change":"2024-04-06T12:12:03.801858+0000","last_active":"2024-04-06T12:12:08.222644+0000","last_peered":"2024-04-06T12:12:08.222644+0000","last_clean":"2024-04-06T12:12:08.222644+0000","last_became_active":"2024-04-06T12:12:03.801643+0000","last_became_peered":"2024-04-06T12:12:03.801643+0000","last_unstale":"2024-04-06T12:12:08.222644+0000","last_undegraded":"2024-04-06T12:12:08.222644+0000","last_fullsized":"2024-04-06T12:12:08.222644+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:08:55.770454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232963+0000","last_change":"2024-04-06T12:12:04.213199+0000","last_active":"2024-04-06T12:12:08.232963+0000","last_peered":"2024-04-06T12:12:08.232963+0000","last_clean":"2024-04-06T12:12:08.232963+0000","last_became_active":"2024-04-06T12:12:04.212685+0000","last_became_peered":"2024-04-06T12:12:04.212685+0000","last_unstale":"2024-04-06T12:12:08.232963+0000","last_undegraded":"2024-04-06T12:12:08.232963+0000","last_fullsized":"2024-04-06T12:12:08.232963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:45:10.755538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.250618+0000","last_change":"2024-04-06T12:12:08.249234+0000","last_active":"2024-04-06T12:12:08.250618+0000","last_peered":"2024-04-06T12:12:08.250618+0000","last_clean":"2024-04-06T12:12:08.250618+0000","last_became_active":"2024-04-06T12:12:08.248705+0000","last_became_peered":"2024-04-06T12:12:08.248705+0000","last_unstale":"2024-04-06T12:12:08.250618+0000","last_undegraded":"2024-04-06T12:12:08.250618+0000","last_fullsized":"2024-04-06T12:12:08.250618+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:58:47.243168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222776+0000","last_change":"2024-04-06T12:12:03.798517+0000","last_active":"2024-04-06T12:12:08.222776+0000","last_peered":"2024-04-06T12:12:08.222776+0000","last_clean":"2024-04-06T12:12:08.222776+0000","last_became_active":"2024-04-06T12:12:03.798118+0000","last_became_peered":"2024-04-06T12:12:03.798118+0000","last_unstale":"2024-04-06T12:12:08.222776+0000","last_undegraded":"2024-04-06T12:12:08.222776+0000","last_fullsized":"2024-04-06T12:12:08.222776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:46:15.699305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232781+0000","last_change":"2024-04-06T12:12:04.216405+0000","last_active":"2024-04-06T12:12:08.232781+0000","last_peered":"2024-04-06T12:12:08.232781+0000","last_clean":"2024-04-06T12:12:08.232781+0000","last_became_active":"2024-04-06T12:12:04.216059+0000","last_became_peered":"2024-04-06T12:12:04.216059+0000","last_unstale":"2024-04-06T12:12:08.232781+0000","last_undegraded":"2024-04-06T12:12:08.232781+0000","last_fullsized":"2024-04-06T12:12:08.232781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.258081+0000","last_change":"2024-04-06T12:12:08.253652+0000","last_active":"2024-04-06T12:12:08.258081+0000","last_peered":"2024-04-06T12:12:08.258081+0000","last_clean":"2024-04-06T12:12:08.258081+0000","last_became_active":"2024-04-06T12:12:08.253321+0000","last_became_peered":"2024-04-06T12:12:08.253321+0000","last_unstale":"2024-04-06T12:12:08.258081+0000","last_undegraded":"2024-04-06T12:12:08.258081+0000","last_fullsized":"2024-04-06T12:12:08.258081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:31:50.315335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224175+0000","last_change":"2024-04-06T12:12:03.798088+0000","last_active":"2024-04-06T12:12:08.224175+0000","last_peered":"2024-04-06T12:12:08.224175+0000","last_clean":"2024-04-06T12:12:08.224175+0000","last_became_active":"2024-04-06T12:12:03.797844+0000","last_became_peered":"2024-04-06T12:12:03.797844+0000","last_unstale":"2024-04-06T12:12:08.224175+0000","last_undegraded":"2024-04-06T12:12:08.224175+0000","last_fullsized":"2024-04-06T12:12:08.224175+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:09.507432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223640+0000","last_change":"2024-04-06T12:12:04.224049+0000","last_active":"2024-04-06T12:12:08.223640+0000","last_peered":"2024-04-06T12:12:08.223640+0000","last_clean":"2024-04-06T12:12:08.223640+0000","last_became_active":"2024-04-06T12:12:04.223437+0000","last_became_peered":"2024-04-06T12:12:04.223437+0000","last_unstale":"2024-04-06T12:12:08.223640+0000","last_undegraded":"2024-04-06T12:12:08.223640+0000","last_fullsized":"2024-04-06T12:12:08.223640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:36:25.380472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.247262+0000","last_change":"2024-04-06T12:12:08.244925+0000","last_active":"2024-04-06T12:12:08.247262+0000","last_peered":"2024-04-06T12:12:08.247262+0000","last_clean":"2024-04-06T12:12:08.247262+0000","last_became_active":"2024-04-06T12:12:08.244571+0000","last_became_peered":"2024-04-06T12:12:08.244571+0000","last_unstale":"2024-04-06T12:12:08.247262+0000","last_undegraded":"2024-04-06T12:12:08.247262+0000","last_fullsized":"2024-04-06T12:12:08.247262+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:31:50.315335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227747+0000","last_change":"2024-04-06T12:12:03.800145+0000","last_active":"2024-04-06T12:12:08.227747+0000","last_peered":"2024-04-06T12:12:08.227747+0000","last_clean":"2024-04-06T12:12:08.227747+0000","last_became_active":"2024-04-06T12:12:03.799406+0000","last_became_peered":"2024-04-06T12:12:03.799406+0000","last_unstale":"2024-04-06T12:12:08.227747+0000","last_undegraded":"2024-04-06T12:12:08.227747+0000","last_fullsized":"2024-04-06T12:12:08.227747+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:40:20.784453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224246+0000","last_change":"2024-04-06T12:12:04.226479+0000","last_active":"2024-04-06T12:12:08.224246+0000","last_peered":"2024-04-06T12:12:08.224246+0000","last_clean":"2024-04-06T12:12:08.224246+0000","last_became_active":"2024-04-06T12:12:04.222457+0000","last_became_peered":"2024-04-06T12:12:04.222457+0000","last_unstale":"2024-04-06T12:12:08.224246+0000","last_undegraded":"2024-04-06T12:12:08.224246+0000","last_fullsized":"2024-04-06T12:12:08.224246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:41:20.002927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.252006+0000","last_change":"2024-04-06T12:12:08.247112+0000","last_active":"2024-04-06T12:12:08.252006+0000","last_peered":"2024-04-06T12:12:08.252006+0000","last_clean":"2024-04-06T12:12:08.252006+0000","last_became_active":"2024-04-06T12:12:08.246705+0000","last_became_peered":"2024-04-06T12:12:08.246705+0000","last_unstale":"2024-04-06T12:12:08.252006+0000","last_undegraded":"2024-04-06T12:12:08.252006+0000","last_fullsized":"2024-04-06T12:12:08.252006+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:29.953853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.222999+0000","last_change":"2024-04-06T12:12:03.799395+0000","last_active":"2024-04-06T12:12:08.222999+0000","last_peered":"2024-04-06T12:12:08.222999+0000","last_clean":"2024-04-06T12:12:08.222999+0000","last_became_active":"2024-04-06T12:12:03.799116+0000","last_became_peered":"2024-04-06T12:12:03.799116+0000","last_unstale":"2024-04-06T12:12:08.222999+0000","last_undegraded":"2024-04-06T12:12:08.222999+0000","last_fullsized":"2024-04-06T12:12:08.222999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:09.507432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227629+0000","last_change":"2024-04-06T12:12:04.221238+0000","last_active":"2024-04-06T12:12:08.227629+0000","last_peered":"2024-04-06T12:12:08.227629+0000","last_clean":"2024-04-06T12:12:08.227629+0000","last_became_active":"2024-04-06T12:12:04.220606+0000","last_became_peered":"2024-04-06T12:12:04.220606+0000","last_unstale":"2024-04-06T12:12:08.227629+0000","last_undegraded":"2024-04-06T12:12:08.227629+0000","last_fullsized":"2024-04-06T12:12:08.227629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.246867+0000","last_change":"2024-04-06T12:12:08.241508+0000","last_active":"2024-04-06T12:12:08.246867+0000","last_peered":"2024-04-06T12:12:08.246867+0000","last_clean":"2024-04-06T12:12:08.246867+0000","last_became_active":"2024-04-06T12:12:08.241184+0000","last_became_peered":"2024-04-06T12:12:08.241184+0000","last_unstale":"2024-04-06T12:12:08.246867+0000","last_undegraded":"2024-04-06T12:12:08.246867+0000","last_fullsized":"2024-04-06T12:12:08.246867+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:07.216556+0000","last_clean_scrub_stamp":"2024-04-06T12:12:07.216556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:35:54.255608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228578+0000","last_change":"2024-04-06T12:12:03.795282+0000","last_active":"2024-04-06T12:12:08.228578+0000","last_peered":"2024-04-06T12:12:08.228578+0000","last_clean":"2024-04-06T12:12:08.228578+0000","last_became_active":"2024-04-06T12:12:03.794928+0000","last_became_peered":"2024-04-06T12:12:03.794928+0000","last_unstale":"2024-04-06T12:12:08.228578+0000","last_undegraded":"2024-04-06T12:12:08.228578+0000","last_fullsized":"2024-04-06T12:12:08.228578+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:02.772853+0000","last_clean_scrub_stamp":"2024-04-06T12:12:02.772853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:00:31.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.225091+0000","last_change":"2024-04-06T12:12:04.224265+0000","last_active":"2024-04-06T12:12:08.225091+0000","last_peered":"2024-04-06T12:12:08.225091+0000","last_clean":"2024-04-06T12:12:08.225091+0000","last_became_active":"2024-04-06T12:12:04.223892+0000","last_became_peered":"2024-04-06T12:12:04.223892+0000","last_unstale":"2024-04-06T12:12:08.225091+0000","last_undegraded":"2024-04-06T12:12:08.225091+0000","last_fullsized":"2024-04-06T12:12:08.225091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:21:16.167775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223213+0000","last_change":"2024-04-06T12:12:04.223037+0000","last_active":"2024-04-06T12:12:08.223213+0000","last_peered":"2024-04-06T12:12:08.223213+0000","last_clean":"2024-04-06T12:12:08.223213+0000","last_became_active":"2024-04-06T12:12:04.218051+0000","last_became_peered":"2024-04-06T12:12:04.218051+0000","last_unstale":"2024-04-06T12:12:08.223213+0000","last_undegraded":"2024-04-06T12:12:08.223213+0000","last_fullsized":"2024-04-06T12:12:08.223213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:22:59.062918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227020+0000","last_change":"2024-04-06T12:12:04.214042+0000","last_active":"2024-04-06T12:12:08.227020+0000","last_peered":"2024-04-06T12:12:08.227020+0000","last_clean":"2024-04-06T12:12:08.227020+0000","last_became_active":"2024-04-06T12:12:04.213777+0000","last_became_peered":"2024-04-06T12:12:04.213777+0000","last_unstale":"2024-04-06T12:12:08.227020+0000","last_undegraded":"2024-04-06T12:12:08.227020+0000","last_fullsized":"2024-04-06T12:12:08.227020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:05.943405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228110+0000","last_change":"2024-04-06T12:12:04.216964+0000","last_active":"2024-04-06T12:12:08.228110+0000","last_peered":"2024-04-06T12:12:08.228110+0000","last_clean":"2024-04-06T12:12:08.228110+0000","last_became_active":"2024-04-06T12:12:04.216692+0000","last_became_peered":"2024-04-06T12:12:04.216692+0000","last_unstale":"2024-04-06T12:12:08.228110+0000","last_undegraded":"2024-04-06T12:12:08.228110+0000","last_fullsized":"2024-04-06T12:12:08.228110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:01:06.622244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234533+0000","last_change":"2024-04-06T12:12:04.219048+0000","last_active":"2024-04-06T12:12:08.234533+0000","last_peered":"2024-04-06T12:12:08.234533+0000","last_clean":"2024-04-06T12:12:08.234533+0000","last_became_active":"2024-04-06T12:12:04.218624+0000","last_became_peered":"2024-04-06T12:12:04.218624+0000","last_unstale":"2024-04-06T12:12:08.234533+0000","last_undegraded":"2024-04-06T12:12:08.234533+0000","last_fullsized":"2024-04-06T12:12:08.234533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:28:15.608394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.233536+0000","last_change":"2024-04-06T12:12:04.218893+0000","last_active":"2024-04-06T12:12:08.233536+0000","last_peered":"2024-04-06T12:12:08.233536+0000","last_clean":"2024-04-06T12:12:08.233536+0000","last_became_active":"2024-04-06T12:12:04.218526+0000","last_became_peered":"2024-04-06T12:12:04.218526+0000","last_unstale":"2024-04-06T12:12:08.233536+0000","last_undegraded":"2024-04-06T12:12:08.233536+0000","last_fullsized":"2024-04-06T12:12:08.233536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:15.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223037+0000","last_change":"2024-04-06T12:12:04.216794+0000","last_active":"2024-04-06T12:12:08.223037+0000","last_peered":"2024-04-06T12:12:08.223037+0000","last_clean":"2024-04-06T12:12:08.223037+0000","last_became_active":"2024-04-06T12:12:04.216532+0000","last_became_peered":"2024-04-06T12:12:04.216532+0000","last_unstale":"2024-04-06T12:12:08.223037+0000","last_undegraded":"2024-04-06T12:12:08.223037+0000","last_fullsized":"2024-04-06T12:12:08.223037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:15.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.224560+0000","last_change":"2024-04-06T12:12:04.229182+0000","last_active":"2024-04-06T12:12:08.224560+0000","last_peered":"2024-04-06T12:12:08.224560+0000","last_clean":"2024-04-06T12:12:08.224560+0000","last_became_active":"2024-04-06T12:12:04.228927+0000","last_became_peered":"2024-04-06T12:12:04.228927+0000","last_unstale":"2024-04-06T12:12:08.224560+0000","last_undegraded":"2024-04-06T12:12:08.224560+0000","last_fullsized":"2024-04-06T12:12:08.224560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:20.329194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234558+0000","last_change":"2024-04-06T12:12:04.217003+0000","last_active":"2024-04-06T12:12:08.234558+0000","last_peered":"2024-04-06T12:12:08.234558+0000","last_clean":"2024-04-06T12:12:08.234558+0000","last_became_active":"2024-04-06T12:12:04.216757+0000","last_became_peered":"2024-04-06T12:12:04.216757+0000","last_unstale":"2024-04-06T12:12:08.234558+0000","last_undegraded":"2024-04-06T12:12:08.234558+0000","last_fullsized":"2024-04-06T12:12:08.234558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:00.258491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.227358+0000","last_change":"2024-04-06T12:12:04.216442+0000","last_active":"2024-04-06T12:12:08.227358+0000","last_peered":"2024-04-06T12:12:08.227358+0000","last_clean":"2024-04-06T12:12:08.227358+0000","last_became_active":"2024-04-06T12:12:04.215051+0000","last_became_peered":"2024-04-06T12:12:04.215051+0000","last_unstale":"2024-04-06T12:12:08.227358+0000","last_undegraded":"2024-04-06T12:12:08.227358+0000","last_fullsized":"2024-04-06T12:12:08.227358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:19:38.810657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.232330+0000","last_change":"2024-04-06T12:12:04.219526+0000","last_active":"2024-04-06T12:12:08.232330+0000","last_peered":"2024-04-06T12:12:08.232330+0000","last_clean":"2024-04-06T12:12:08.232330+0000","last_became_active":"2024-04-06T12:12:04.219229+0000","last_became_peered":"2024-04-06T12:12:04.219229+0000","last_unstale":"2024-04-06T12:12:08.232330+0000","last_undegraded":"2024-04-06T12:12:08.232330+0000","last_fullsized":"2024-04-06T12:12:08.232330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:22:59.062918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.228186+0000","last_change":"2024-04-06T12:12:04.226116+0000","last_active":"2024-04-06T12:12:08.228186+0000","last_peered":"2024-04-06T12:12:08.228186+0000","last_clean":"2024-04-06T12:12:08.228186+0000","last_became_active":"2024-04-06T12:12:04.225820+0000","last_became_peered":"2024-04-06T12:12:04.225820+0000","last_unstale":"2024-04-06T12:12:08.228186+0000","last_undegraded":"2024-04-06T12:12:08.228186+0000","last_fullsized":"2024-04-06T12:12:08.228186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:41:20.002927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.234612+0000","last_change":"2024-04-06T12:12:04.228504+0000","last_active":"2024-04-06T12:12:08.234612+0000","last_peered":"2024-04-06T12:12:08.234612+0000","last_clean":"2024-04-06T12:12:08.234612+0000","last_became_active":"2024-04-06T12:12:04.228232+0000","last_became_peered":"2024-04-06T12:12:04.228232+0000","last_unstale":"2024-04-06T12:12:08.234612+0000","last_undegraded":"2024-04-06T12:12:08.234612+0000","last_fullsized":"2024-04-06T12:12:08.234612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:22:59.062918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-06T12:12:08.223649+0000","last_change":"2024-04-06T12:12:04.230394+0000","last_active":"2024-04-06T12:12:08.223649+0000","last_peered":"2024-04-06T12:12:08.223649+0000","last_clean":"2024-04-06T12:12:08.223649+0000","last_became_active":"2024-04-06T12:12:04.230065+0000","last_became_peered":"2024-04-06T12:12:04.230065+0000","last_unstale":"2024-04-06T12:12:08.223649+0000","last_undegraded":"2024-04-06T12:12:08.223649+0000","last_fullsized":"2024-04-06T12:12:08.223649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:12:03.778209+0000","last_clean_scrub_stamp":"2024-04-06T12:12:03.778209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:06.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4594,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":851968,"data_stored":690730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":262144,"data_stored":102557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":98921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":856064,"data_stored":689767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":98761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":103576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.223}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":99906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":98851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:12:32.690 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-06T12:12:32.690 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-06T12:12:32.690 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-06T12:12:32.690 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-06T12:12:32.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.849+0000 7f7d83f2c700 1 Processor -- start 2024-04-06T12:12:32.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.849+0000 7f7d83f2c700 1 -- start start 2024-04-06T12:12:32.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.850+0000 7f7d83f2c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d7c0cf500 0x7f7d7c0cf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:32.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.850+0000 7f7d83f2c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7d7c0cd3f0 0x7f7d7c06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:32.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.850+0000 7f7d83f2c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d7c0cd870 0x7f7d7c06ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:32.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.850+0000 7f7d83f2c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d7c06b340 con 0x7f7d7c0cd870 2024-04-06T12:12:32.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.850+0000 7f7d83f2c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d7c0ccfd0 con 0x7f7d7c0cd3f0 2024-04-06T12:12:32.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.850+0000 7f7d83f2c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7c0b27c0 con 0x7f7d7c0cf500 2024-04-06T12:12:32.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.850+0000 7f7d824c9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d7c0cd870 0x7f7d7c06ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:32.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.850+0000 7f7d824c9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d7c0cd870 0x7f7d7c06ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38368/0 (socket says 172.21.15.189:38368) 2024-04-06T12:12:32.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.850+0000 7f7d824c9700 1 -- 172.21.15.189:0/563254781 learned_addr learned my addr 172.21.15.189:0/563254781 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:32.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.850+0000 7f7d814c7700 1 --2- 172.21.15.189:0/563254781 >> v2:172.21.15.192:3300/0 conn(0x7f7d7c0cd3f0 0x7f7d7c06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:32.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.850+0000 7f7d81cc8700 1 --2- 172.21.15.189:0/563254781 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d7c0cf500 0x7f7d7c0cf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:32.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.850+0000 7f7d824c9700 1 -- 172.21.15.189:0/563254781 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d7c0cf500 msgr2=0x7f7d7c0cf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:32.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.850+0000 7f7d824c9700 1 --2- 172.21.15.189:0/563254781 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d7c0cf500 0x7f7d7c0cf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.850+0000 7f7d824c9700 1 -- 172.21.15.189:0/563254781 >> v2:172.21.15.192:3300/0 conn(0x7f7d7c0cd3f0 msgr2=0x7f7d7c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:32.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.850+0000 7f7d824c9700 1 --2- 172.21.15.189:0/563254781 >> v2:172.21.15.192:3300/0 conn(0x7f7d7c0cd3f0 0x7f7d7c06a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.850+0000 7f7d824c9700 1 -- 172.21.15.189:0/563254781 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7c0cfe20 con 0x7f7d7c0cd870 2024-04-06T12:12:32.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.850+0000 7f7d824c9700 1 --2- 172.21.15.189:0/563254781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d7c0cd870 0x7f7d7c06ac70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c0cdc50 tx=0x7f7d7400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a73fd14600afefd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:32.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.851+0000 7f7d80cc6700 1 -- 172.21.15.189:0/563254781 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7400e070 con 0x7f7d7c0cd870 2024-04-06T12:12:32.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.851+0000 7f7d80cc6700 1 -- 172.21.15.189:0/563254781 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7400b9c0 con 0x7f7d7c0cd870 2024-04-06T12:12:32.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.851+0000 7f7d80cc6700 1 -- 172.21.15.189:0/563254781 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7400bd70 con 0x7f7d7c0cd870 2024-04-06T12:12:32.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.851+0000 7f7d83f2c700 1 -- 172.21.15.189:0/563254781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d7c0cd870 msgr2=0x7f7d7c06ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:32.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.851+0000 7f7d83f2c700 1 --2- 172.21.15.189:0/563254781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d7c0cd870 0x7f7d7c06ac70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c0cdc50 tx=0x7f7d7400b330 comp rx=0 tx=0).stop 2024-04-06T12:12:32.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.851+0000 7f7d83f2c700 1 -- 172.21.15.189:0/563254781 shutdown_connections 2024-04-06T12:12:32.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.851+0000 7f7d83f2c700 1 --2- 172.21.15.189:0/563254781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d7c0cd870 0x7f7d7c06ac70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.851+0000 7f7d83f2c700 1 --2- 172.21.15.189:0/563254781 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d7c0cf500 0x7f7d7c0cf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.851+0000 7f7d83f2c700 1 --2- 172.21.15.189:0/563254781 >> v2:172.21.15.192:3300/0 conn(0x7f7d7c0cd3f0 0x7f7d7c06a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.851+0000 7f7d83f2c700 1 -- 172.21.15.189:0/563254781 >> 172.21.15.189:0/563254781 conn(0x7f7d7c099a50 msgr2=0x7f7d7c0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:32.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.851+0000 7f7d83f2c700 1 -- 172.21.15.189:0/563254781 shutdown_connections 2024-04-06T12:12:32.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.851+0000 7f7d83f2c700 1 -- 172.21.15.189:0/563254781 wait complete. 2024-04-06T12:12:32.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.852+0000 7f7d83f2c700 1 Processor -- start 2024-04-06T12:12:32.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.852+0000 7f7d83f2c700 1 -- start start 2024-04-06T12:12:32.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.853+0000 7f7d83f2c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d7c0cd3f0 0x7f7d7c1e7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:32.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.853+0000 7f7d83f2c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d7c0cd870 0x7f7d7c1d7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:32.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.853+0000 7f7d83f2c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d7c0cf500 0x7f7d7c1e5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:32.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.853+0000 7f7d83f2c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7c1e67d0 con 0x7f7d7c0cd3f0 2024-04-06T12:12:32.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.853+0000 7f7d83f2c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7c1e6ab0 con 0x7f7d7c0cf500 2024-04-06T12:12:32.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.853+0000 7f7d83f2c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7c1e6d90 con 0x7f7d7c0cd870 2024-04-06T12:12:32.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.853+0000 7f7d81cc8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d7c0cd3f0 0x7f7d7c1e7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:32.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.853+0000 7f7d81cc8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d7c0cd3f0 0x7f7d7c1e7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38378/0 (socket says 172.21.15.189:38378) 2024-04-06T12:12:32.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.853+0000 7f7d81cc8700 1 -- 172.21.15.189:0/4168948767 learned_addr learned my addr 172.21.15.189:0/4168948767 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:32.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.853+0000 7f7d824c9700 1 --2- 172.21.15.189:0/4168948767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d7c0cf500 0x7f7d7c1e5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:32.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.853+0000 7f7d814c7700 1 --2- 172.21.15.189:0/4168948767 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d7c0cd870 0x7f7d7c1d7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:32.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.854+0000 7f7d81cc8700 1 -- 172.21.15.189:0/4168948767 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d7c0cd870 msgr2=0x7f7d7c1d7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:32.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.854+0000 7f7d81cc8700 1 --2- 172.21.15.189:0/4168948767 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d7c0cd870 0x7f7d7c1d7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.854+0000 7f7d81cc8700 1 -- 172.21.15.189:0/4168948767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d7c0cf500 msgr2=0x7f7d7c1e5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:32.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.854+0000 7f7d81cc8700 1 --2- 172.21.15.189:0/4168948767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d7c0cf500 0x7f7d7c1e5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:32.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.854+0000 7f7d81cc8700 1 -- 172.21.15.189:0/4168948767 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d74009b30 con 0x7f7d7c0cd3f0 2024-04-06T12:12:32.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.854+0000 7f7d81cc8700 1 --2- 172.21.15.189:0/4168948767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d7c0cd3f0 0x7f7d7c1e7bb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800e810 tx=0x7f7d7800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:32.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.854+0000 7f7d727fc700 1 -- 172.21.15.189:0/4168948767 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d78013610 con 0x7f7d7c0cd3f0 2024-04-06T12:12:32.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.854+0000 7f7d727fc700 1 -- 172.21.15.189:0/4168948767 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d78013770 con 0x7f7d7c0cd3f0 2024-04-06T12:12:32.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.854+0000 7f7d83f2c700 1 -- 172.21.15.189:0/4168948767 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d7c0c3d70 con 0x7f7d7c0cd3f0 2024-04-06T12:12:32.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.854+0000 7f7d727fc700 1 -- 172.21.15.189:0/4168948767 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d78013b20 con 0x7f7d7c0cd3f0 2024-04-06T12:12:32.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.854+0000 7f7d83f2c700 1 -- 172.21.15.189:0/4168948767 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d7c1e0880 con 0x7f7d7c0cd3f0 2024-04-06T12:12:32.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.855+0000 7f7d83f2c700 1 -- 172.21.15.189:0/4168948767 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d7c0c40e0 con 0x7f7d7c0cd3f0 2024-04-06T12:12:32.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.856+0000 7f7d727fc700 1 -- 172.21.15.189:0/4168948767 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7d78013c80 con 0x7f7d7c0cd3f0 2024-04-06T12:12:32.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.856+0000 7f7d727fc700 1 --2- 172.21.15.189:0/4168948767 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d6006ac90 0x7f7d6006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:32.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.856+0000 7f7d727fc700 1 -- 172.21.15.189:0/4168948767 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7d780897b0 con 0x7f7d7c0cd3f0 2024-04-06T12:12:32.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.856+0000 7f7d814c7700 1 --2- 172.21.15.189:0/4168948767 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d6006ac90 0x7f7d6006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:32.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.857+0000 7f7d814c7700 1 --2- 172.21.15.189:0/4168948767 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d6006ac90 0x7f7d6006d150 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c0c5120 tx=0x7f7d6c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:32.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:32.860+0000 7f7d727fc700 1 -- 172.21.15.189:0/4168948767 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7d78054690 con 0x7f7d7c0cd3f0 2024-04-06T12:12:33.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.096+0000 7f7d83f2c700 1 -- 172.21.15.189:0/4168948767 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7d7c0c3bd0 con 0x7f7d7c0cd3f0 2024-04-06T12:12:33.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.096+0000 7f7d727fc700 1 -- 172.21.15.189:0/4168948767 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f7d7c0c3bd0 con 0x7f7d7c0cd3f0 2024-04-06T12:12:33.095 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:33.095 INFO:teuthology.orchestra.run.smithi189.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-06T12:12:33.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.098+0000 7f7d83f2c700 1 -- 172.21.15.189:0/4168948767 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d6006ac90 msgr2=0x7f7d6006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:33.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.098+0000 7f7d83f2c700 1 --2- 172.21.15.189:0/4168948767 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d6006ac90 0x7f7d6006d150 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c0c5120 tx=0x7f7d6c008040 comp rx=0 tx=0).stop 2024-04-06T12:12:33.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.098+0000 7f7d83f2c700 1 -- 172.21.15.189:0/4168948767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d7c0cd3f0 msgr2=0x7f7d7c1e7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:33.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.098+0000 7f7d83f2c700 1 --2- 172.21.15.189:0/4168948767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d7c0cd3f0 0x7f7d7c1e7bb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800e810 tx=0x7f7d7800ebd0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.099+0000 7f7d83f2c700 1 -- 172.21.15.189:0/4168948767 shutdown_connections 2024-04-06T12:12:33.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.099+0000 7f7d83f2c700 1 --2- 172.21.15.189:0/4168948767 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d6006ac90 0x7f7d6006d150 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.099+0000 7f7d83f2c700 1 --2- 172.21.15.189:0/4168948767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d7c0cf500 0x7f7d7c1e5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.099+0000 7f7d83f2c700 1 --2- 172.21.15.189:0/4168948767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d7c0cd3f0 0x7f7d7c1e7bb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.099+0000 7f7d83f2c700 1 --2- 172.21.15.189:0/4168948767 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d7c0cd870 0x7f7d7c1d7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.099+0000 7f7d83f2c700 1 -- 172.21.15.189:0/4168948767 >> 172.21.15.189:0/4168948767 conn(0x7f7d7c099a50 msgr2=0x7f7d7c1e8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:33.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.099+0000 7f7d83f2c700 1 -- 172.21.15.189:0/4168948767 shutdown_connections 2024-04-06T12:12:33.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.099+0000 7f7d83f2c700 1 -- 172.21.15.189:0/4168948767 wait complete. 2024-04-06T12:12:33.108 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-06T12:12:33.109 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:33.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.232+0000 7fdbcd3c9700 1 Processor -- start 2024-04-06T12:12:33.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.232+0000 7fdbcd3c9700 1 -- start start 2024-04-06T12:12:33.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.233+0000 7fdbcd3c9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbc80b5380 0x7fdbc80d2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:33.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.233+0000 7fdbcd3c9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc80b5cc0 con 0x7fdbc815d590 2024-04-06T12:12:33.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.233+0000 7fdbcd3c9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc80b57f0 con 0x7fdbc80b4ed0 2024-04-06T12:12:33.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.233+0000 7fdbcd3c9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc80d1500 con 0x7fdbc80b5380 2024-04-06T12:12:33.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.233+0000 7fdbc6ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdbc815d590 0x7fdbc80b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34346/0 (socket says 172.21.15.189:34346) 2024-04-06T12:12:33.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.233+0000 7fdbc6ffd700 1 -- 172.21.15.189:0/4186127749 learned_addr learned my addr 172.21.15.189:0/4186127749 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:33.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.233+0000 7fdbc77fe700 1 --2- 172.21.15.189:0/4186127749 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbc80b5380 0x7fdbc80d2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.233+0000 7fdbc5ffb700 1 -- 172.21.15.189:0/4186127749 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342740291 0 0) 0x7fdbc80b5cc0 con 0x7fdbc815d590 2024-04-06T12:12:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.233+0000 7fdbc5ffb700 1 -- 172.21.15.189:0/4186127749 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbb00035b0 con 0x7fdbc815d590 2024-04-06T12:12:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.233+0000 7fdbc5ffb700 1 -- 172.21.15.189:0/4186127749 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787204170 0 0) 0x7fdbc80b57f0 con 0x7fdbc80b4ed0 2024-04-06T12:12:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.233+0000 7fdbc5ffb700 1 -- 172.21.15.189:0/4186127749 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc80b5cc0 con 0x7fdbc80b4ed0 2024-04-06T12:12:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc5ffb700 1 -- 172.21.15.189:0/4186127749 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2556070163 0 0) 0x7fdbb00035b0 con 0x7fdbc815d590 2024-04-06T12:12:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc5ffb700 1 -- 172.21.15.189:0/4186127749 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbc80b5380 msgr2=0x7fdbc80d2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc5ffb700 1 --2- 172.21.15.189:0/4186127749 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbc80b5380 0x7fdbc80d2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc5ffb700 1 -- 172.21.15.189:0/4186127749 >> v1:172.21.15.192:6789/0 conn(0x7fdbc80b4ed0 legacy=0x7fdbc80d2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc5ffb700 1 -- 172.21.15.189:0/4186127749 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc80d17e0 con 0x7fdbc815d590 2024-04-06T12:12:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc5ffb700 1 -- 172.21.15.189:0/4186127749 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fdbbc001670 con 0x7fdbc815d590 2024-04-06T12:12:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc77fe700 1 --2- 172.21.15.189:0/4186127749 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbc80b5380 0x7fdbc80d2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc5ffb700 1 -- 172.21.15.189:0/4186127749 >> v1:172.21.15.189:6789/0 conn(0x7fdbc815d590 legacy=0x7fdbc80b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc5ffb700 1 --2- 172.21.15.189:0/4186127749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbb0003bc0 0x7fdbb0003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc5ffb700 1 --2- 172.21.15.189:0/4186127749 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbc80b5380 0x7fdbb0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc5ffb700 1 --2- 172.21.15.189:0/4186127749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbb0004d90 0x7fdbb0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc5ffb700 1 -- 172.21.15.189:0/4186127749 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc80d1500 con 0x7fdbb0003bc0 2024-04-06T12:12:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc5ffb700 1 -- 172.21.15.189:0/4186127749 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb000bac0 con 0x7fdbb0004d90 2024-04-06T12:12:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc5ffb700 1 -- 172.21.15.189:0/4186127749 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb000bc00 con 0x7fdbc80b5380 2024-04-06T12:12:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc6ffd700 1 --2- 172.21.15.189:0/4186127749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbb0003bc0 0x7fdbb0003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc77fe700 1 --2- 172.21.15.189:0/4186127749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbb0004d90 0x7fdbb0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc67fc700 1 --2- 172.21.15.189:0/4186127749 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbc80b5380 0x7fdbb0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc6ffd700 1 -- 172.21.15.189:0/4186127749 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbc80b5380 msgr2=0x7fdbb0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc6ffd700 1 --2- 172.21.15.189:0/4186127749 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbc80b5380 0x7fdbb0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc6ffd700 1 -- 172.21.15.189:0/4186127749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbb0004d90 msgr2=0x7fdbb0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc6ffd700 1 --2- 172.21.15.189:0/4186127749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbb0004d90 0x7fdbb0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc6ffd700 1 -- 172.21.15.189:0/4186127749 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc0009a70 con 0x7fdbb0003bc0 2024-04-06T12:12:33.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc67fc700 1 --2- 172.21.15.189:0/4186127749 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbc80b5380 0x7fdbb0004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:33.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc6ffd700 1 --2- 172.21.15.189:0/4186127749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbb0003bc0 0x7fdbb0003fc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc00c240 tx=0x7fdbb0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d418208ba7a434de server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:33.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc5ffb700 1 -- 172.21.15.189:0/4186127749 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc00cda0 con 0x7fdbb0003bc0 2024-04-06T12:12:33.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc5ffb700 1 -- 172.21.15.189:0/4186127749 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbbc001af0 con 0x7fdbb0003bc0 2024-04-06T12:12:33.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.234+0000 7fdbc5ffb700 1 -- 172.21.15.189:0/4186127749 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc005ba0 con 0x7fdbb0003bc0 2024-04-06T12:12:33.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.235+0000 7fdbcd3c9700 1 -- 172.21.15.189:0/4186127749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbb0003bc0 msgr2=0x7fdbb0003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:33.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.235+0000 7fdbcd3c9700 1 --2- 172.21.15.189:0/4186127749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbb0003bc0 0x7fdbb0003fc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc00c240 tx=0x7fdbb0005880 comp rx=0 tx=0).stop 2024-04-06T12:12:33.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.235+0000 7fdbc77fe700 1 -- 172.21.15.189:0/4186127749 reap_dead start 2024-04-06T12:12:33.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.235+0000 7fdbcd3c9700 1 -- 172.21.15.189:0/4186127749 shutdown_connections 2024-04-06T12:12:33.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.235+0000 7fdbcd3c9700 1 -- 172.21.15.189:0/4186127749 >> 172.21.15.189:0/4186127749 conn(0x7fdbc806bbc0 msgr2=0x7fdbc806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:33.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.235+0000 7fdbcd3c9700 1 -- 172.21.15.189:0/4186127749 shutdown_connections 2024-04-06T12:12:33.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.235+0000 7fdbcd3c9700 1 -- 172.21.15.189:0/4186127749 wait complete. 2024-04-06T12:12:33.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.236+0000 7fdbcd3c9700 1 Processor -- start 2024-04-06T12:12:33.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.236+0000 7fdbcd3c9700 1 -- start start 2024-04-06T12:12:33.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.237+0000 7fdbcd3c9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbb0003bc0 0x7fdbc80ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:33.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.237+0000 7fdbcd3c9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbc80ca520 0x7fdbc815f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:33.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.237+0000 7fdbcd3c9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbc81f6c50 0x7fdbc81f7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:33.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.237+0000 7fdbcd3c9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc815f4a0 con 0x7fdbc81f6c50 2024-04-06T12:12:33.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.237+0000 7fdbcd3c9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc80cb330 con 0x7fdbb0003bc0 2024-04-06T12:12:33.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.237+0000 7fdbcd3c9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc80caa90 con 0x7fdbc80ca520 2024-04-06T12:12:33.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.237+0000 7fdbc77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbc81f6c50 0x7fdbc81f7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:33.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.237+0000 7fdbc77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbc81f6c50 0x7fdbc81f7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38392/0 (socket says 172.21.15.189:38392) 2024-04-06T12:12:33.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.237+0000 7fdbc77fe700 1 -- 172.21.15.189:0/3147030600 learned_addr learned my addr 172.21.15.189:0/3147030600 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:33.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.237+0000 7fdbc6ffd700 1 --2- 172.21.15.189:0/3147030600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbb0003bc0 0x7fdbc80ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:33.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.237+0000 7fdbc67fc700 1 --2- 172.21.15.189:0/3147030600 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbc80ca520 0x7fdbc815f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:33.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.237+0000 7fdbc77fe700 1 -- 172.21.15.189:0/3147030600 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbc80ca520 msgr2=0x7fdbc815f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:33.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.237+0000 7fdbc77fe700 1 --2- 172.21.15.189:0/3147030600 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbc80ca520 0x7fdbc815f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.237+0000 7fdbc77fe700 1 -- 172.21.15.189:0/3147030600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbb0003bc0 msgr2=0x7fdbc80ccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:33.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.237+0000 7fdbc77fe700 1 --2- 172.21.15.189:0/3147030600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbb0003bc0 0x7fdbc80ccc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.237+0000 7fdbc77fe700 1 -- 172.21.15.189:0/3147030600 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc80cad70 con 0x7fdbc81f6c50 2024-04-06T12:12:33.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.237+0000 7fdbc67fc700 1 --2- 172.21.15.189:0/3147030600 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbc80ca520 0x7fdbc815f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:33.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.237+0000 7fdbc77fe700 1 --2- 172.21.15.189:0/3147030600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbc81f6c50 0x7fdbc81f7100 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdbc80b5fb0 tx=0x7fdbc0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:33.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.238+0000 7fdbaf7fe700 1 -- 172.21.15.189:0/3147030600 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc000e070 con 0x7fdbc81f6c50 2024-04-06T12:12:33.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.238+0000 7fdbaf7fe700 1 -- 172.21.15.189:0/3147030600 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc0003c30 con 0x7fdbc81f6c50 2024-04-06T12:12:33.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.238+0000 7fdbaf7fe700 1 -- 172.21.15.189:0/3147030600 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0003dc0 con 0x7fdbc81f6c50 2024-04-06T12:12:33.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.238+0000 7fdbcd3c9700 1 -- 172.21.15.189:0/3147030600 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc80cb050 con 0x7fdbc81f6c50 2024-04-06T12:12:33.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.238+0000 7fdbcd3c9700 1 -- 172.21.15.189:0/3147030600 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc815fcc0 con 0x7fdbc81f6c50 2024-04-06T12:12:33.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.239+0000 7fdbcd3c9700 1 -- 172.21.15.189:0/3147030600 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc80cf090 con 0x7fdbc81f6c50 2024-04-06T12:12:33.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.239+0000 7fdbaf7fe700 1 -- 172.21.15.189:0/3147030600 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdbc0011a60 con 0x7fdbc81f6c50 2024-04-06T12:12:33.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.239+0000 7fdbaf7fe700 1 --2- 172.21.15.189:0/3147030600 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdbb406ac90 0x7fdbb406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:33.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.239+0000 7fdbaf7fe700 1 -- 172.21.15.189:0/3147030600 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fdbc0089a70 con 0x7fdbc81f6c50 2024-04-06T12:12:33.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.239+0000 7fdbc6ffd700 1 --2- 172.21.15.189:0/3147030600 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdbb406ac90 0x7fdbb406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:33.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.240+0000 7fdbc6ffd700 1 --2- 172.21.15.189:0/3147030600 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdbb406ac90 0x7fdbb406d150 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc001eb0 tx=0x7fdbbc00a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:33.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.244+0000 7fdbaf7fe700 1 -- 172.21.15.189:0/3147030600 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdbc00549f0 con 0x7fdbc81f6c50 2024-04-06T12:12:33.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.439+0000 7fdbcd3c9700 1 -- 172.21.15.189:0/3147030600 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbc8074990 con 0x7fdbc81f6c50 2024-04-06T12:12:33.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.440+0000 7fdbaf7fe700 1 -- 172.21.15.189:0/3147030600 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17926 (secure 0 0 0) 0x7fdbc0058010 con 0x7fdbc81f6c50 2024-04-06T12:12:33.438 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:33.438 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:08.649170+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:33.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.442+0000 7fdbcd3c9700 1 -- 172.21.15.189:0/3147030600 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdbb406ac90 msgr2=0x7fdbb406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:33.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.442+0000 7fdbcd3c9700 1 --2- 172.21.15.189:0/3147030600 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdbb406ac90 0x7fdbb406d150 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc001eb0 tx=0x7fdbbc00a000 comp rx=0 tx=0).stop 2024-04-06T12:12:33.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.442+0000 7fdbcd3c9700 1 -- 172.21.15.189:0/3147030600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbc81f6c50 msgr2=0x7fdbc81f7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:33.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.442+0000 7fdbcd3c9700 1 --2- 172.21.15.189:0/3147030600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbc81f6c50 0x7fdbc81f7100 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdbc80b5fb0 tx=0x7fdbc0009f90 comp rx=0 tx=0).stop 2024-04-06T12:12:33.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.442+0000 7fdbcd3c9700 1 -- 172.21.15.189:0/3147030600 shutdown_connections 2024-04-06T12:12:33.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.442+0000 7fdbcd3c9700 1 --2- 172.21.15.189:0/3147030600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbc81f6c50 0x7fdbc81f7100 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.442+0000 7fdbcd3c9700 1 --2- 172.21.15.189:0/3147030600 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdbb406ac90 0x7fdbb406d150 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.442+0000 7fdbcd3c9700 1 --2- 172.21.15.189:0/3147030600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbb0003bc0 0x7fdbc80ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.442+0000 7fdbcd3c9700 1 --2- 172.21.15.189:0/3147030600 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbc80ca520 0x7fdbc815f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.442+0000 7fdbcd3c9700 1 -- 172.21.15.189:0/3147030600 >> 172.21.15.189:0/3147030600 conn(0x7fdbc806bbc0 msgr2=0x7fdbc80b4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:33.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.443+0000 7fdbcd3c9700 1 -- 172.21.15.189:0/3147030600 shutdown_connections 2024-04-06T12:12:33.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.443+0000 7fdbcd3c9700 1 -- 172.21.15.189:0/3147030600 wait complete. 2024-04-06T12:12:33.453 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:12:02.280866+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:12:03.156593+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-06T12:12:06.237184+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-06T12:12:33.453 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:12:33.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.580+0000 7f949d1ee700 1 Processor -- start 2024-04-06T12:12:33.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.580+0000 7f949d1ee700 1 -- start start 2024-04-06T12:12:33.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.581+0000 7f949d1ee700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f949806ae40 0x7f94980b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:33.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.581+0000 7f949d1ee700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f949815ddf0 0x7f949815e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:33.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.581+0000 7f949d1ee700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94980b3990 con 0x7f949815d7b0 2024-04-06T12:12:33.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.581+0000 7f949d1ee700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94980b3060 con 0x7f949806ae40 2024-04-06T12:12:33.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.581+0000 7f949d1ee700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94980b3340 con 0x7f949815ddf0 2024-04-06T12:12:33.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.581+0000 7f949659c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f949806ae40 0x7f94980b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:33.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.581+0000 7f9496d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f949815d7b0 0x7f949806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34352/0 (socket says 172.21.15.189:34352) 2024-04-06T12:12:33.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.581+0000 7f9496d9d700 1 -- 172.21.15.189:0/133840889 learned_addr learned my addr 172.21.15.189:0/133840889 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:33.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.582+0000 7f949759e700 1 --2- 172.21.15.189:0/133840889 >> v2:172.21.15.192:3300/0 conn(0x7f949815ddf0 0x7f949815e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:33.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.582+0000 7f949659c700 1 -- 172.21.15.189:0/133840889 >> v2:172.21.15.192:3300/0 conn(0x7f949815ddf0 msgr2=0x7f949815e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:33.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.582+0000 7f949659c700 1 --2- 172.21.15.189:0/133840889 >> v2:172.21.15.192:3300/0 conn(0x7f949815ddf0 0x7f949815e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.582+0000 7f949659c700 1 -- 172.21.15.189:0/133840889 >> v1:172.21.15.189:6789/0 conn(0x7f949815d7b0 legacy=0x7f949806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:33.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.582+0000 7f9495d9b700 1 -- 172.21.15.189:0/133840889 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679214086 0 0) 0x7f94980b3990 con 0x7f949815d7b0 2024-04-06T12:12:33.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.582+0000 7f949659c700 1 -- 172.21.15.189:0/133840889 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94980b3620 con 0x7f949806ae40 2024-04-06T12:12:33.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.582+0000 7f949759e700 1 --2- 172.21.15.189:0/133840889 >> v2:172.21.15.192:3300/0 conn(0x7f949815ddf0 0x7f949815e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:33.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.582+0000 7f949659c700 1 --2- 172.21.15.189:0/133840889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f949806ae40 0x7f94980b3e20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f949806d2c0 tx=0x7f9488009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b34cd5fe297182 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:33.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.582+0000 7f9495d9b700 1 -- 172.21.15.189:0/133840889 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9488011830 con 0x7f949806ae40 2024-04-06T12:12:33.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.582+0000 7f9495d9b700 1 -- 172.21.15.189:0/133840889 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9488011990 con 0x7f949806ae40 2024-04-06T12:12:33.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.582+0000 7f9495d9b700 1 -- 172.21.15.189:0/133840889 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9488011d20 con 0x7f949806ae40 2024-04-06T12:12:33.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.582+0000 7f949d1ee700 1 -- 172.21.15.189:0/133840889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f949806ae40 msgr2=0x7f94980b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:33.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.582+0000 7f949d1ee700 1 --2- 172.21.15.189:0/133840889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f949806ae40 0x7f94980b3e20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f949806d2c0 tx=0x7f9488009b20 comp rx=0 tx=0).stop 2024-04-06T12:12:33.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.582+0000 7f949d1ee700 1 -- 172.21.15.189:0/133840889 shutdown_connections 2024-04-06T12:12:33.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.582+0000 7f949d1ee700 1 --2- 172.21.15.189:0/133840889 >> v2:172.21.15.192:3300/0 conn(0x7f949815ddf0 0x7f949815e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.582+0000 7f949d1ee700 1 --2- 172.21.15.189:0/133840889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f949806ae40 0x7f94980b3e20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.582+0000 7f949d1ee700 1 -- 172.21.15.189:0/133840889 >> 172.21.15.189:0/133840889 conn(0x7f94980d00b0 msgr2=0x7f94980d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:33.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.582+0000 7f949d1ee700 1 -- 172.21.15.189:0/133840889 shutdown_connections 2024-04-06T12:12:33.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.583+0000 7f949d1ee700 1 -- 172.21.15.189:0/133840889 wait complete. 2024-04-06T12:12:33.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.584+0000 7f949d1ee700 1 Processor -- start 2024-04-06T12:12:33.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.584+0000 7f949d1ee700 1 -- start start 2024-04-06T12:12:33.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.585+0000 7f949d1ee700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f949806ae40 0x7f9498153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:33.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.585+0000 7f949d1ee700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f949815d7b0 0x7f9498152e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:33.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.585+0000 7f949d1ee700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f949815ddf0 0x7f949815c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:33.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.585+0000 7f949d1ee700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94980af800 con 0x7f949815ddf0 2024-04-06T12:12:33.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.585+0000 7f949d1ee700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9498153360 con 0x7f949806ae40 2024-04-06T12:12:33.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.585+0000 7f949d1ee700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94980aeb80 con 0x7f949815d7b0 2024-04-06T12:12:33.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.585+0000 7f949759e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f949815ddf0 0x7f949815c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:33.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.585+0000 7f949759e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f949815ddf0 0x7f949815c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38414/0 (socket says 172.21.15.189:38414) 2024-04-06T12:12:33.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.585+0000 7f949759e700 1 -- 172.21.15.189:0/2647515543 learned_addr learned my addr 172.21.15.189:0/2647515543 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:33.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.585+0000 7f9496d9d700 1 --2- 172.21.15.189:0/2647515543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f949806ae40 0x7f9498153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:33.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.585+0000 7f949659c700 1 --2- 172.21.15.189:0/2647515543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f949815d7b0 0x7f9498152e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:33.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.585+0000 7f949759e700 1 -- 172.21.15.189:0/2647515543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f949815d7b0 msgr2=0x7f9498152e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:33.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.585+0000 7f949759e700 1 --2- 172.21.15.189:0/2647515543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f949815d7b0 0x7f9498152e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.585+0000 7f949759e700 1 -- 172.21.15.189:0/2647515543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f949806ae40 msgr2=0x7f9498153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:33.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.585+0000 7f949759e700 1 --2- 172.21.15.189:0/2647515543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f949806ae40 0x7f9498153500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.585+0000 7f949759e700 1 -- 172.21.15.189:0/2647515543 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94880094a0 con 0x7f949815ddf0 2024-04-06T12:12:33.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.585+0000 7f949759e700 1 --2- 172.21.15.189:0/2647515543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f949815ddf0 0x7f949815c7c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f949000cd80 tx=0x7f949000a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:33.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.586+0000 7f947f7fe700 1 -- 172.21.15.189:0/2647515543 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949000f480 con 0x7f949815ddf0 2024-04-06T12:12:33.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.586+0000 7f947f7fe700 1 -- 172.21.15.189:0/2647515543 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f949000f600 con 0x7f949815ddf0 2024-04-06T12:12:33.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.586+0000 7f947f7fe700 1 -- 172.21.15.189:0/2647515543 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949000f9b0 con 0x7f949815ddf0 2024-04-06T12:12:33.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.586+0000 7f949d1ee700 1 -- 172.21.15.189:0/2647515543 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94980aee60 con 0x7f949815ddf0 2024-04-06T12:12:33.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.586+0000 7f949d1ee700 1 -- 172.21.15.189:0/2647515543 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94980af420 con 0x7f949815ddf0 2024-04-06T12:12:33.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.587+0000 7f949d1ee700 1 -- 172.21.15.189:0/2647515543 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9498067bb0 con 0x7f949815ddf0 2024-04-06T12:12:33.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.587+0000 7f947f7fe700 1 -- 172.21.15.189:0/2647515543 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f949000b040 con 0x7f949815ddf0 2024-04-06T12:12:33.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.587+0000 7f947f7fe700 1 --2- 172.21.15.189:0/2647515543 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f948406ac90 0x7f948406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:33.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.588+0000 7f947f7fe700 1 -- 172.21.15.189:0/2647515543 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9490089cf0 con 0x7f949815ddf0 2024-04-06T12:12:33.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.588+0000 7f9496d9d700 1 --2- 172.21.15.189:0/2647515543 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f948406ac90 0x7f948406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:33.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.588+0000 7f9496d9d700 1 --2- 172.21.15.189:0/2647515543 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f948406ac90 0x7f948406d150 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f948c007fd0 tx=0x7f948c007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:33.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.592+0000 7f947f7fe700 1 -- 172.21.15.189:0/2647515543 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f949008e070 con 0x7f949815ddf0 2024-04-06T12:12:33.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.786+0000 7f949d1ee700 1 -- 172.21.15.189:0/2647515543 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9498074990 con 0x7f949815ddf0 2024-04-06T12:12:33.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.786+0000 7f947f7fe700 1 -- 172.21.15.189:0/2647515543 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9490054bf0 con 0x7f949815ddf0 2024-04-06T12:12:33.785 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:12:33.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.788+0000 7f949d1ee700 1 -- 172.21.15.189:0/2647515543 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f948406ac90 msgr2=0x7f948406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:33.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.788+0000 7f949d1ee700 1 --2- 172.21.15.189:0/2647515543 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f948406ac90 0x7f948406d150 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f948c007fd0 tx=0x7f948c007de0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.788+0000 7f949d1ee700 1 -- 172.21.15.189:0/2647515543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f949815ddf0 msgr2=0x7f949815c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:33.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.788+0000 7f949d1ee700 1 --2- 172.21.15.189:0/2647515543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f949815ddf0 0x7f949815c7c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f949000cd80 tx=0x7f949000a560 comp rx=0 tx=0).stop 2024-04-06T12:12:33.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.788+0000 7f949d1ee700 1 -- 172.21.15.189:0/2647515543 shutdown_connections 2024-04-06T12:12:33.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.788+0000 7f949d1ee700 1 --2- 172.21.15.189:0/2647515543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f949815ddf0 0x7f949815c7c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.788+0000 7f949d1ee700 1 --2- 172.21.15.189:0/2647515543 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f948406ac90 0x7f948406d150 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.788+0000 7f949d1ee700 1 --2- 172.21.15.189:0/2647515543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f949806ae40 0x7f9498153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.788+0000 7f949d1ee700 1 --2- 172.21.15.189:0/2647515543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f949815d7b0 0x7f9498152e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.788+0000 7f949d1ee700 1 -- 172.21.15.189:0/2647515543 >> 172.21.15.189:0/2647515543 conn(0x7f94980d00b0 msgr2=0x7f949806eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:33.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.789+0000 7f949d1ee700 1 -- 172.21.15.189:0/2647515543 shutdown_connections 2024-04-06T12:12:33.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.789+0000 7f949d1ee700 1 -- 172.21.15.189:0/2647515543 wait complete. 2024-04-06T12:12:33.799 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:12:33.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.925+0000 7f68a1766700 1 Processor -- start 2024-04-06T12:12:33.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.925+0000 7f68a1766700 1 -- start start 2024-04-06T12:12:33.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.926+0000 7f68a1766700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f689c1592b0 0x7f689c0b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:33.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.926+0000 7f68a1766700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689c15c430 con 0x7f689c15c8c0 2024-04-06T12:12:33.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.926+0000 7f68a1766700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689c15bb80 con 0x7f689c15f8d0 2024-04-06T12:12:33.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.926+0000 7f68a1766700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f689c1601b0 con 0x7f689c1592b0 2024-04-06T12:12:33.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.926+0000 7f689affd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f689c1592b0 0x7f689c0b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:33.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.926+0000 7f689a7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f689c15c8c0 0x7f689c15f1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34368/0 (socket says 172.21.15.189:34368) 2024-04-06T12:12:33.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.926+0000 7f689a7fc700 1 -- 172.21.15.189:0/766025926 learned_addr learned my addr 172.21.15.189:0/766025926 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:33.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.926+0000 7f689affd700 1 -- 172.21.15.189:0/766025926 >> v1:172.21.15.192:6789/0 conn(0x7f689c15f8d0 legacy=0x7f689c0d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:33.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.926+0000 7f689affd700 1 -- 172.21.15.189:0/766025926 >> v1:172.21.15.189:6789/0 conn(0x7f689c15c8c0 legacy=0x7f689c15f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:33.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.926+0000 7f689affd700 1 -- 172.21.15.189:0/766025926 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689c15fed0 con 0x7f689c1592b0 2024-04-06T12:12:33.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.926+0000 7f6899ffb700 1 -- 172.21.15.189:0/766025926 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981530854 0 0) 0x7f689c15c430 con 0x7f689c15c8c0 2024-04-06T12:12:33.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.927+0000 7f689affd700 1 --2- 172.21.15.189:0/766025926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f689c1592b0 0x7f689c0b2ca0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6890009840 tx=0x7f6890009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d18decac356e01 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:33.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.927+0000 7f6899ffb700 1 -- 172.21.15.189:0/766025926 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6890004230 con 0x7f689c1592b0 2024-04-06T12:12:33.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.927+0000 7f6899ffb700 1 -- 172.21.15.189:0/766025926 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6890004390 con 0x7f689c1592b0 2024-04-06T12:12:33.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.927+0000 7f6899ffb700 1 -- 172.21.15.189:0/766025926 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68900036a0 con 0x7f689c1592b0 2024-04-06T12:12:33.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.927+0000 7f68a1766700 1 -- 172.21.15.189:0/766025926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f689c1592b0 msgr2=0x7f689c0b2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:33.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.927+0000 7f68a1766700 1 --2- 172.21.15.189:0/766025926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f689c1592b0 0x7f689c0b2ca0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6890009840 tx=0x7f6890009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:33.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.927+0000 7f68a1766700 1 -- 172.21.15.189:0/766025926 shutdown_connections 2024-04-06T12:12:33.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.927+0000 7f68a1766700 1 --2- 172.21.15.189:0/766025926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f689c1592b0 0x7f689c0b2ca0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.927+0000 7f68a1766700 1 -- 172.21.15.189:0/766025926 >> 172.21.15.189:0/766025926 conn(0x7f689c06eaa0 msgr2=0x7f689c06ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:33.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.927+0000 7f68a1766700 1 -- 172.21.15.189:0/766025926 shutdown_connections 2024-04-06T12:12:33.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.927+0000 7f68a1766700 1 -- 172.21.15.189:0/766025926 wait complete. 2024-04-06T12:12:33.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.929+0000 7f68a1766700 1 Processor -- start 2024-04-06T12:12:33.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.929+0000 7f68a1766700 1 -- start start 2024-04-06T12:12:33.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.929+0000 7f68a1766700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f689c1592b0 0x7f689c1f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:33.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.929+0000 7f68a1766700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f689c15c8c0 0x7f689c06c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:33.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.929+0000 7f68a1766700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f689c15f8d0 0x7f689c0c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:33.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.929+0000 7f68a1766700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689c0b4e50 con 0x7f689c1592b0 2024-04-06T12:12:33.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.929+0000 7f68a1766700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689c06c9f0 con 0x7f689c15c8c0 2024-04-06T12:12:33.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.929+0000 7f68a1766700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f689c06ccc0 con 0x7f689c15f8d0 2024-04-06T12:12:33.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.930+0000 7f689affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f689c1592b0 0x7f689c1f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:33.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.930+0000 7f689affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f689c1592b0 0x7f689c1f3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38440/0 (socket says 172.21.15.189:38440) 2024-04-06T12:12:33.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.930+0000 7f689affd700 1 -- 172.21.15.189:0/2247350593 learned_addr learned my addr 172.21.15.189:0/2247350593 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:33.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.930+0000 7f689a7fc700 1 --2- 172.21.15.189:0/2247350593 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f689c15c8c0 0x7f689c06c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:33.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.930+0000 7f689b7fe700 1 --2- 172.21.15.189:0/2247350593 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f689c15f8d0 0x7f689c0c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:33.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.930+0000 7f689affd700 1 -- 172.21.15.189:0/2247350593 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f689c15f8d0 msgr2=0x7f689c0c9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:33.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.930+0000 7f689affd700 1 --2- 172.21.15.189:0/2247350593 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f689c15f8d0 0x7f689c0c9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.930+0000 7f689affd700 1 -- 172.21.15.189:0/2247350593 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f689c15c8c0 msgr2=0x7f689c06c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:33.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.930+0000 7f689affd700 1 --2- 172.21.15.189:0/2247350593 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f689c15c8c0 0x7f689c06c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:33.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.930+0000 7f689affd700 1 -- 172.21.15.189:0/2247350593 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68900094a0 con 0x7f689c1592b0 2024-04-06T12:12:33.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.930+0000 7f689b7fe700 1 --2- 172.21.15.189:0/2247350593 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f689c15f8d0 0x7f689c0c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:33.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.930+0000 7f689affd700 1 --2- 172.21.15.189:0/2247350593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f689c1592b0 0x7f689c1f3e10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f68900090b0 tx=0x7f6890004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:33.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.930+0000 7f68837fe700 1 -- 172.21.15.189:0/2247350593 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6890011830 con 0x7f689c1592b0 2024-04-06T12:12:33.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.930+0000 7f68837fe700 1 -- 172.21.15.189:0/2247350593 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689001e070 con 0x7f689c1592b0 2024-04-06T12:12:33.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.930+0000 7f68837fe700 1 -- 172.21.15.189:0/2247350593 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6890011bb0 con 0x7f689c1592b0 2024-04-06T12:12:33.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.930+0000 7f68a1766700 1 -- 172.21.15.189:0/2247350593 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f689c06cfa0 con 0x7f689c1592b0 2024-04-06T12:12:33.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.930+0000 7f68a1766700 1 -- 172.21.15.189:0/2247350593 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f689c0b5380 con 0x7f689c1592b0 2024-04-06T12:12:33.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.931+0000 7f68a1766700 1 -- 172.21.15.189:0/2247350593 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f689c06d160 con 0x7f689c1592b0 2024-04-06T12:12:33.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.932+0000 7f68837fe700 1 -- 172.21.15.189:0/2247350593 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6890011d10 con 0x7f689c1592b0 2024-04-06T12:12:33.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.932+0000 7f68837fe700 1 --2- 172.21.15.189:0/2247350593 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f688806ac90 0x7f688806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:33.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.932+0000 7f68837fe700 1 -- 172.21.15.189:0/2247350593 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f689008a970 con 0x7f689c1592b0 2024-04-06T12:12:33.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.932+0000 7f689a7fc700 1 --2- 172.21.15.189:0/2247350593 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f688806ac90 0x7f688806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:33.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.932+0000 7f689a7fc700 1 --2- 172.21.15.189:0/2247350593 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f688806ac90 0x7f688806d150 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f689c0ce340 tx=0x7f688c007be0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:33.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:33.936+0000 7f68837fe700 1 -- 172.21.15.189:0/2247350593 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f68900558f0 con 0x7f689c1592b0 2024-04-06T12:12:34.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.130+0000 7f68a1766700 1 -- 172.21.15.189:0/2247350593 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f689c074990 con 0x7f689c1592b0 2024-04-06T12:12:34.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.130+0000 7f68837fe700 1 -- 172.21.15.189:0/2247350593 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6890058f10 con 0x7f689c1592b0 2024-04-06T12:12:34.128 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:12:34.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.132+0000 7f68a1766700 1 -- 172.21.15.189:0/2247350593 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f688806ac90 msgr2=0x7f688806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:34.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.132+0000 7f68a1766700 1 --2- 172.21.15.189:0/2247350593 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f688806ac90 0x7f688806d150 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f689c0ce340 tx=0x7f688c007be0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.132+0000 7f68a1766700 1 -- 172.21.15.189:0/2247350593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f689c1592b0 msgr2=0x7f689c1f3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:34.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.132+0000 7f68a1766700 1 --2- 172.21.15.189:0/2247350593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f689c1592b0 0x7f689c1f3e10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f68900090b0 tx=0x7f6890004750 comp rx=0 tx=0).stop 2024-04-06T12:12:34.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.132+0000 7f68a1766700 1 -- 172.21.15.189:0/2247350593 shutdown_connections 2024-04-06T12:12:34.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.132+0000 7f68a1766700 1 --2- 172.21.15.189:0/2247350593 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f689c15f8d0 0x7f689c0c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.132+0000 7f68a1766700 1 --2- 172.21.15.189:0/2247350593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f689c1592b0 0x7f689c1f3e10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.132+0000 7f68a1766700 1 --2- 172.21.15.189:0/2247350593 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f688806ac90 0x7f688806d150 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.132+0000 7f68a1766700 1 --2- 172.21.15.189:0/2247350593 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f689c15c8c0 0x7f689c06c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.132+0000 7f68a1766700 1 -- 172.21.15.189:0/2247350593 >> 172.21.15.189:0/2247350593 conn(0x7f689c06eaa0 msgr2=0x7f689c15d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:34.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.132+0000 7f68a1766700 1 -- 172.21.15.189:0/2247350593 shutdown_connections 2024-04-06T12:12:34.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.132+0000 7f68a1766700 1 -- 172.21.15.189:0/2247350593 wait complete. 2024-04-06T12:12:34.143 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:12:34.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.269+0000 7fcacdd1e700 1 Processor -- start 2024-04-06T12:12:34.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.269+0000 7fcacdd1e700 1 -- start start 2024-04-06T12:12:34.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.270+0000 7fcacdd1e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcac815d7b0 0x7fcac80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.270+0000 7fcacdd1e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcac806a730 0x7fcac806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.270+0000 7fcacdd1e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac80b3990 con 0x7fcac815ddf0 2024-04-06T12:12:34.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.270+0000 7fcacdd1e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcac806b0f0 con 0x7fcac815d7b0 2024-04-06T12:12:34.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.270+0000 7fcacdd1e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac80b3060 con 0x7fcac806a730 2024-04-06T12:12:34.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.270+0000 7fcac77fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcac815d7b0 0x7fcac80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.270+0000 7fcac7fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcac815ddf0 0x7fcac815e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34378/0 (socket says 172.21.15.189:34378) 2024-04-06T12:12:34.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.270+0000 7fcac7fff700 1 -- 172.21.15.189:0/3167809392 learned_addr learned my addr 172.21.15.189:0/3167809392 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:34.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.270+0000 7fcac6ffd700 1 --2- 172.21.15.189:0/3167809392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcac806a730 0x7fcac806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.270+0000 7fcac77fe700 1 -- 172.21.15.189:0/3167809392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcac806a730 msgr2=0x7fcac806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:34.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.270+0000 7fcac77fe700 1 --2- 172.21.15.189:0/3167809392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcac806a730 0x7fcac806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.270+0000 7fcac77fe700 1 -- 172.21.15.189:0/3167809392 >> v1:172.21.15.189:6789/0 conn(0x7fcac815ddf0 legacy=0x7fcac815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:34.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.270+0000 7fcac67fc700 1 -- 172.21.15.189:0/3167809392 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3100663175 0 0) 0x7fcac80b3990 con 0x7fcac815ddf0 2024-04-06T12:12:34.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.270+0000 7fcac77fe700 1 -- 172.21.15.189:0/3167809392 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac80b3340 con 0x7fcac815d7b0 2024-04-06T12:12:34.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.271+0000 7fcac77fe700 1 --2- 172.21.15.189:0/3167809392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac815d7b0 0x7fcac80b3e20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009840 tx=0x7fcab0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6681388dc183d6e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:34.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.271+0000 7fcac67fc700 1 -- 172.21.15.189:0/3167809392 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0011830 con 0x7fcac815d7b0 2024-04-06T12:12:34.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.271+0000 7fcac67fc700 1 -- 172.21.15.189:0/3167809392 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab0011990 con 0x7fcac815d7b0 2024-04-06T12:12:34.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.271+0000 7fcac67fc700 1 -- 172.21.15.189:0/3167809392 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0011d20 con 0x7fcac815d7b0 2024-04-06T12:12:34.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.271+0000 7fcacdd1e700 1 -- 172.21.15.189:0/3167809392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac815d7b0 msgr2=0x7fcac80b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:34.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.271+0000 7fcacdd1e700 1 --2- 172.21.15.189:0/3167809392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac815d7b0 0x7fcac80b3e20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009840 tx=0x7fcab0009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:34.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.271+0000 7fcacdd1e700 1 -- 172.21.15.189:0/3167809392 shutdown_connections 2024-04-06T12:12:34.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.271+0000 7fcacdd1e700 1 --2- 172.21.15.189:0/3167809392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac815d7b0 0x7fcac80b3e20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.271+0000 7fcacdd1e700 1 --2- 172.21.15.189:0/3167809392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcac806a730 0x7fcac806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.271+0000 7fcacdd1e700 1 -- 172.21.15.189:0/3167809392 >> 172.21.15.189:0/3167809392 conn(0x7fcac80d00b0 msgr2=0x7fcac80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:34.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.271+0000 7fcacdd1e700 1 -- 172.21.15.189:0/3167809392 shutdown_connections 2024-04-06T12:12:34.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.271+0000 7fcacdd1e700 1 -- 172.21.15.189:0/3167809392 wait complete. 2024-04-06T12:12:34.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.273+0000 7fcacdd1e700 1 Processor -- start 2024-04-06T12:12:34.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.273+0000 7fcacdd1e700 1 -- start start 2024-04-06T12:12:34.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.273+0000 7fcacdd1e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcac806a730 0x7fcac80cef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.273+0000 7fcacdd1e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcac815d7b0 0x7fcac815b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.273+0000 7fcacdd1e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac815ddf0 0x7fcac80cc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.273+0000 7fcacdd1e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac81433e0 con 0x7fcac815ddf0 2024-04-06T12:12:34.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.273+0000 7fcacdd1e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac815bac0 con 0x7fcac806a730 2024-04-06T12:12:34.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.273+0000 7fcacdd1e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac815bda0 con 0x7fcac815d7b0 2024-04-06T12:12:34.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.274+0000 7fcac7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac815ddf0 0x7fcac80cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.274+0000 7fcac7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac815ddf0 0x7fcac80cc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38452/0 (socket says 172.21.15.189:38452) 2024-04-06T12:12:34.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.274+0000 7fcac7fff700 1 -- 172.21.15.189:0/2073228197 learned_addr learned my addr 172.21.15.189:0/2073228197 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:34.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.274+0000 7fcac6ffd700 1 --2- 172.21.15.189:0/2073228197 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcac815d7b0 0x7fcac815b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.274+0000 7fcac77fe700 1 --2- 172.21.15.189:0/2073228197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcac806a730 0x7fcac80cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.274+0000 7fcac7fff700 1 -- 172.21.15.189:0/2073228197 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcac815d7b0 msgr2=0x7fcac815b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:34.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.274+0000 7fcac7fff700 1 --2- 172.21.15.189:0/2073228197 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcac815d7b0 0x7fcac815b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.274+0000 7fcac7fff700 1 -- 172.21.15.189:0/2073228197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcac806a730 msgr2=0x7fcac80cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:34.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.274+0000 7fcac7fff700 1 --2- 172.21.15.189:0/2073228197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcac806a730 0x7fcac80cef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.274+0000 7fcac7fff700 1 -- 172.21.15.189:0/2073228197 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab00094a0 con 0x7fcac815ddf0 2024-04-06T12:12:34.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.274+0000 7fcac7fff700 1 --2- 172.21.15.189:0/2073228197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac815ddf0 0x7fcac80cc680 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcac000cba0 tx=0x7fcac000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:34.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.274+0000 7fcabffff700 1 -- 172.21.15.189:0/2073228197 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac000ab80 con 0x7fcac815ddf0 2024-04-06T12:12:34.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.274+0000 7fcabffff700 1 -- 172.21.15.189:0/2073228197 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac000ad00 con 0x7fcac815ddf0 2024-04-06T12:12:34.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.274+0000 7fcabffff700 1 -- 172.21.15.189:0/2073228197 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac0007310 con 0x7fcac815ddf0 2024-04-06T12:12:34.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.274+0000 7fcacdd1e700 1 -- 172.21.15.189:0/2073228197 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac815c080 con 0x7fcac815ddf0 2024-04-06T12:12:34.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.274+0000 7fcacdd1e700 1 -- 172.21.15.189:0/2073228197 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac80ccbc0 con 0x7fcac815ddf0 2024-04-06T12:12:34.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.275+0000 7fcacdd1e700 1 -- 172.21.15.189:0/2073228197 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac8143590 con 0x7fcac815ddf0 2024-04-06T12:12:34.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.276+0000 7fcabffff700 1 -- 172.21.15.189:0/2073228197 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcac0007470 con 0x7fcac815ddf0 2024-04-06T12:12:34.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.276+0000 7fcabffff700 1 --2- 172.21.15.189:0/2073228197 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcaa806ac90 0x7fcaa806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.276+0000 7fcabffff700 1 -- 172.21.15.189:0/2073228197 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fcac0089190 con 0x7fcac815ddf0 2024-04-06T12:12:34.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.276+0000 7fcac77fe700 1 --2- 172.21.15.189:0/2073228197 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcaa806ac90 0x7fcaa806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.277+0000 7fcac77fe700 1 --2- 172.21.15.189:0/2073228197 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcaa806ac90 0x7fcaa806d150 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcab00090b0 tx=0x7fcab0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:34.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.280+0000 7fcabffff700 1 -- 172.21.15.189:0/2073228197 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fcac8143590 con 0x7fcac815ddf0 2024-04-06T12:12:34.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.477+0000 7fcacdd1e700 1 -- 172.21.15.189:0/2073228197 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcac8067bb0 con 0x7fcac815ddf0 2024-04-06T12:12:34.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.477+0000 7fcabffff700 1 -- 172.21.15.189:0/2073228197 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcac00180a0 con 0x7fcac815ddf0 2024-04-06T12:12:34.476 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:12:34.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.480+0000 7fcacdd1e700 1 -- 172.21.15.189:0/2073228197 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcaa806ac90 msgr2=0x7fcaa806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:34.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.480+0000 7fcacdd1e700 1 --2- 172.21.15.189:0/2073228197 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcaa806ac90 0x7fcaa806d150 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcab00090b0 tx=0x7fcab0008030 comp rx=0 tx=0).stop 2024-04-06T12:12:34.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.480+0000 7fcacdd1e700 1 -- 172.21.15.189:0/2073228197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac815ddf0 msgr2=0x7fcac80cc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:34.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.480+0000 7fcacdd1e700 1 --2- 172.21.15.189:0/2073228197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac815ddf0 0x7fcac80cc680 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcac000cba0 tx=0x7fcac000cf60 comp rx=0 tx=0).stop 2024-04-06T12:12:34.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.480+0000 7fcacdd1e700 1 -- 172.21.15.189:0/2073228197 shutdown_connections 2024-04-06T12:12:34.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.480+0000 7fcacdd1e700 1 --2- 172.21.15.189:0/2073228197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac815ddf0 0x7fcac80cc680 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.480+0000 7fcacdd1e700 1 --2- 172.21.15.189:0/2073228197 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcaa806ac90 0x7fcaa806d150 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.480+0000 7fcacdd1e700 1 --2- 172.21.15.189:0/2073228197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcac806a730 0x7fcac80cef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.480+0000 7fcacdd1e700 1 --2- 172.21.15.189:0/2073228197 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcac815d7b0 0x7fcac815b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.480+0000 7fcacdd1e700 1 -- 172.21.15.189:0/2073228197 >> 172.21.15.189:0/2073228197 conn(0x7fcac80d00b0 msgr2=0x7fcac8153c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:34.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.480+0000 7fcacdd1e700 1 -- 172.21.15.189:0/2073228197 shutdown_connections 2024-04-06T12:12:34.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.480+0000 7fcacdd1e700 1 -- 172.21.15.189:0/2073228197 wait complete. 2024-04-06T12:12:34.490 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-06T12:12:34.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.617+0000 7f8117919700 1 Processor -- start 2024-04-06T12:12:34.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.617+0000 7f8117919700 1 -- start start 2024-04-06T12:12:34.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.618+0000 7f8117919700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8110146dd0 0x7f811006d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.618+0000 7f8117919700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f81100cedf0 0x7f81100cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.618+0000 7f8117919700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81100cf760 con 0x7f81100d17a0 2024-04-06T12:12:34.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.618+0000 7f8117919700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f811006beb0 con 0x7f81100cedf0 2024-04-06T12:12:34.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.618+0000 7f8117919700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f811006c0a0 con 0x7f8110146dd0 2024-04-06T12:12:34.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.618+0000 7f8115eb6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f81100cedf0 0x7f81100cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.618+0000 7f8115eb6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f81100cedf0 0x7f81100cf220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38458/0 (socket says 172.21.15.189:38458) 2024-04-06T12:12:34.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.618+0000 7f8115eb6700 1 -- 172.21.15.189:0/4024498132 learned_addr learned my addr 172.21.15.189:0/4024498132 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:34.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.618+0000 7f8114eb4700 1 --2- 172.21.15.189:0/4024498132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8110146dd0 0x7f811006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.618+0000 7f8115eb6700 1 -- 172.21.15.189:0/4024498132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8110146dd0 msgr2=0x7f811006d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:34.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.618+0000 7f8115eb6700 1 --2- 172.21.15.189:0/4024498132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8110146dd0 0x7f811006d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.618+0000 7f8115eb6700 1 -- 172.21.15.189:0/4024498132 >> v1:172.21.15.192:6789/0 conn(0x7f81100d17a0 legacy=0x7f811006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:34.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.618+0000 7f8115eb6700 1 -- 172.21.15.189:0/4024498132 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811006c380 con 0x7f81100cedf0 2024-04-06T12:12:34.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.618+0000 7f8115eb6700 1 --2- 172.21.15.189:0/4024498132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81100cedf0 0x7f81100cf220 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8108009e50 tx=0x7f810800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e386bf79bc493a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:34.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.618+0000 7f8107fff700 1 -- 172.21.15.189:0/4024498132 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810800e070 con 0x7f81100cedf0 2024-04-06T12:12:34.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.619+0000 7f8107fff700 1 -- 172.21.15.189:0/4024498132 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810800b910 con 0x7f81100cedf0 2024-04-06T12:12:34.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.619+0000 7f8107fff700 1 -- 172.21.15.189:0/4024498132 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810800bcc0 con 0x7f81100cedf0 2024-04-06T12:12:34.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.619+0000 7f8117919700 1 -- 172.21.15.189:0/4024498132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81100cedf0 msgr2=0x7f81100cf220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:34.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.619+0000 7f8117919700 1 --2- 172.21.15.189:0/4024498132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81100cedf0 0x7f81100cf220 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8108009e50 tx=0x7f810800b330 comp rx=0 tx=0).stop 2024-04-06T12:12:34.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.619+0000 7f8117919700 1 -- 172.21.15.189:0/4024498132 shutdown_connections 2024-04-06T12:12:34.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.619+0000 7f8117919700 1 --2- 172.21.15.189:0/4024498132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81100cedf0 0x7f81100cf220 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.619+0000 7f8117919700 1 --2- 172.21.15.189:0/4024498132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8110146dd0 0x7f811006d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.619+0000 7f8117919700 1 -- 172.21.15.189:0/4024498132 >> 172.21.15.189:0/4024498132 conn(0x7f81100b2db0 msgr2=0x7f81100b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:34.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.619+0000 7f8117919700 1 -- 172.21.15.189:0/4024498132 shutdown_connections 2024-04-06T12:12:34.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.619+0000 7f8117919700 1 -- 172.21.15.189:0/4024498132 wait complete. 2024-04-06T12:12:34.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.620+0000 7f8117919700 1 Processor -- start 2024-04-06T12:12:34.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.620+0000 7f8117919700 1 -- start start 2024-04-06T12:12:34.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.621+0000 7f8117919700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81100cedf0 0x7f81100ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.621+0000 7f8117919700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81100d17a0 0x7f81100c89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.621+0000 7f8117919700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8110146dd0 0x7f81100c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.621+0000 7f8117919700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81101f40a0 con 0x7f8110146dd0 2024-04-06T12:12:34.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.621+0000 7f8117919700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81100ace10 con 0x7f81100d17a0 2024-04-06T12:12:34.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.621+0000 7f8117919700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81100adc50 con 0x7f81100cedf0 2024-04-06T12:12:34.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.621+0000 7f8115eb6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8110146dd0 0x7f81100c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.621+0000 7f8115eb6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8110146dd0 0x7f81100c8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38472/0 (socket says 172.21.15.189:38472) 2024-04-06T12:12:34.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.621+0000 7f8115eb6700 1 -- 172.21.15.189:0/2988246738 learned_addr learned my addr 172.21.15.189:0/2988246738 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:34.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.621+0000 7f8114eb4700 1 --2- 172.21.15.189:0/2988246738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81100d17a0 0x7f81100c89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.621+0000 7f81156b5700 1 --2- 172.21.15.189:0/2988246738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81100cedf0 0x7f81100ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.622+0000 7f8115eb6700 1 -- 172.21.15.189:0/2988246738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81100cedf0 msgr2=0x7f81100ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:34.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.622+0000 7f8115eb6700 1 --2- 172.21.15.189:0/2988246738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81100cedf0 0x7f81100ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.622+0000 7f8115eb6700 1 -- 172.21.15.189:0/2988246738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81100d17a0 msgr2=0x7f81100c89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:34.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.622+0000 7f8115eb6700 1 --2- 172.21.15.189:0/2988246738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81100d17a0 0x7f81100c89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.622+0000 7f8115eb6700 1 -- 172.21.15.189:0/2988246738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8108009b30 con 0x7f8110146dd0 2024-04-06T12:12:34.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.622+0000 7f81156b5700 1 --2- 172.21.15.189:0/2988246738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81100cedf0 0x7f81100ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:34.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.622+0000 7f8115eb6700 1 --2- 172.21.15.189:0/2988246738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8110146dd0 0x7f81100c8f20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f81100677e0 tx=0x7f8108004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:34.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.622+0000 7f8105ffb700 1 -- 172.21.15.189:0/2988246738 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81080048b0 con 0x7f8110146dd0 2024-04-06T12:12:34.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.622+0000 7f8105ffb700 1 -- 172.21.15.189:0/2988246738 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81080099b0 con 0x7f8110146dd0 2024-04-06T12:12:34.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.622+0000 7f8117919700 1 -- 172.21.15.189:0/2988246738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81100ad040 con 0x7f8110146dd0 2024-04-06T12:12:34.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.622+0000 7f8105ffb700 1 -- 172.21.15.189:0/2988246738 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810801b670 con 0x7f8110146dd0 2024-04-06T12:12:34.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.622+0000 7f8117919700 1 -- 172.21.15.189:0/2988246738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81100ad600 con 0x7f8110146dd0 2024-04-06T12:12:34.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.623+0000 7f8117919700 1 -- 172.21.15.189:0/2988246738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81100adde0 con 0x7f8110146dd0 2024-04-06T12:12:34.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.623+0000 7f8105ffb700 1 -- 172.21.15.189:0/2988246738 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f810801b7d0 con 0x7f8110146dd0 2024-04-06T12:12:34.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.624+0000 7f8105ffb700 1 --2- 172.21.15.189:0/2988246738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f80f406ac90 0x7f80f406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.624+0000 7f8105ffb700 1 -- 172.21.15.189:0/2988246738 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f810808ae40 con 0x7f8110146dd0 2024-04-06T12:12:34.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.624+0000 7f81156b5700 1 --2- 172.21.15.189:0/2988246738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f80f406ac90 0x7f80f406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.624+0000 7f81156b5700 1 --2- 172.21.15.189:0/2988246738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f80f406ac90 0x7f80f406d150 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8100000f60 tx=0x7f8100009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:34.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.628+0000 7f8105ffb700 1 -- 172.21.15.189:0/2988246738 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8108055dc0 con 0x7f8110146dd0 2024-04-06T12:12:34.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.822+0000 7f8117919700 1 -- 172.21.15.189:0/2988246738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f8110074990 con 0x7f8110146dd0 2024-04-06T12:12:34.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.822+0000 7f8105ffb700 1 -- 172.21.15.189:0/2988246738 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f810801f020 con 0x7f8110146dd0 2024-04-06T12:12:34.820 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:12:34.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.824+0000 7f8117919700 1 -- 172.21.15.189:0/2988246738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f80f406ac90 msgr2=0x7f80f406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:34.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.824+0000 7f8117919700 1 --2- 172.21.15.189:0/2988246738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f80f406ac90 0x7f80f406d150 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8100000f60 tx=0x7f8100009040 comp rx=0 tx=0).stop 2024-04-06T12:12:34.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.824+0000 7f8117919700 1 -- 172.21.15.189:0/2988246738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8110146dd0 msgr2=0x7f81100c8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:34.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.824+0000 7f8117919700 1 --2- 172.21.15.189:0/2988246738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8110146dd0 0x7f81100c8f20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f81100677e0 tx=0x7f8108004ad0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.824+0000 7f8117919700 1 -- 172.21.15.189:0/2988246738 shutdown_connections 2024-04-06T12:12:34.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.824+0000 7f8117919700 1 --2- 172.21.15.189:0/2988246738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8110146dd0 0x7f81100c8f20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.824+0000 7f8117919700 1 --2- 172.21.15.189:0/2988246738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81100cedf0 0x7f81100ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.824+0000 7f8117919700 1 --2- 172.21.15.189:0/2988246738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f80f406ac90 0x7f80f406d150 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.824+0000 7f8117919700 1 --2- 172.21.15.189:0/2988246738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81100d17a0 0x7f81100c89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.824+0000 7f8117919700 1 -- 172.21.15.189:0/2988246738 >> 172.21.15.189:0/2988246738 conn(0x7f81100b2db0 msgr2=0x7f81100b3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:34.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.824+0000 7f8117919700 1 -- 172.21.15.189:0/2988246738 shutdown_connections 2024-04-06T12:12:34.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.824+0000 7f8117919700 1 -- 172.21.15.189:0/2988246738 wait complete. 2024-04-06T12:12:34.835 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:12:34.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.958+0000 7f89d842e700 1 Processor -- start 2024-04-06T12:12:34.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.958+0000 7f89d842e700 1 -- start start 2024-04-06T12:12:34.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.959+0000 7f89d842e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f89d00d11b0 0x7f89d00d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.959+0000 7f89d842e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d00cb870 0x7f89d015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.959+0000 7f89d842e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89d00d1ad0 con 0x7f89d00cfa00 2024-04-06T12:12:34.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.959+0000 7f89d842e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89d00ccdc0 con 0x7f89d00d11b0 2024-04-06T12:12:34.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.959+0000 7f89d842e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89d00d0b50 con 0x7f89d00cb870 2024-04-06T12:12:34.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.959+0000 7f89d69cb700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f89d00cfa00 0x7f89d00cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34392/0 (socket says 172.21.15.189:34392) 2024-04-06T12:12:34.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.959+0000 7f89d69cb700 1 -- 172.21.15.189:0/1453141299 learned_addr learned my addr 172.21.15.189:0/1453141299 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:34.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.959+0000 7f89d61ca700 1 --2- 172.21.15.189:0/1453141299 >> v2:172.21.15.192:3300/0 conn(0x7f89d00d11b0 0x7f89d00d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.959+0000 7f89d59c9700 1 --2- 172.21.15.189:0/1453141299 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d00cb870 0x7f89d015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.959+0000 7f89d51c8700 1 -- 172.21.15.189:0/1453141299 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360271941 0 0) 0x7f89d00d1ad0 con 0x7f89d00cfa00 2024-04-06T12:12:34.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.959+0000 7f89d51c8700 1 -- 172.21.15.189:0/1453141299 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89bc0035b0 con 0x7f89d00cfa00 2024-04-06T12:12:34.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.959+0000 7f89d51c8700 1 -- 172.21.15.189:0/1453141299 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 212442406 0 0) 0x7f89bc0035b0 con 0x7f89d00cfa00 2024-04-06T12:12:34.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.959+0000 7f89d51c8700 1 -- 172.21.15.189:0/1453141299 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d00cb870 msgr2=0x7f89d015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:34.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.959+0000 7f89d51c8700 1 --2- 172.21.15.189:0/1453141299 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d00cb870 0x7f89d015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.959+0000 7f89d51c8700 1 -- 172.21.15.189:0/1453141299 >> v2:172.21.15.192:3300/0 conn(0x7f89d00d11b0 msgr2=0x7f89d00d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:34.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.959+0000 7f89d51c8700 1 --2- 172.21.15.189:0/1453141299 >> v2:172.21.15.192:3300/0 conn(0x7f89d00d11b0 0x7f89d00d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.959+0000 7f89d51c8700 1 -- 172.21.15.189:0/1453141299 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d00cbd80 con 0x7f89d00cfa00 2024-04-06T12:12:34.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.959+0000 7f89d51c8700 1 -- 172.21.15.189:0/1453141299 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f89c80019f0 con 0x7f89d00cfa00 2024-04-06T12:12:34.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d51c8700 1 -- 172.21.15.189:0/1453141299 >> v1:172.21.15.189:6789/0 conn(0x7f89d00cfa00 legacy=0x7f89d00cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:34.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d51c8700 1 --2- 172.21.15.189:0/1453141299 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89bc003a00 0x7f89bc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d51c8700 1 --2- 172.21.15.189:0/1453141299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc004850 0x7f89bc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d51c8700 1 --2- 172.21.15.189:0/1453141299 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d00cb870 0x7f89bc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d51c8700 1 -- 172.21.15.189:0/1453141299 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d00d0b50 con 0x7f89bc004850 2024-04-06T12:12:34.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d51c8700 1 -- 172.21.15.189:0/1453141299 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc00ba10 con 0x7f89bc003a00 2024-04-06T12:12:34.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d51c8700 1 -- 172.21.15.189:0/1453141299 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc00bb50 con 0x7f89d00cb870 2024-04-06T12:12:34.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d59c9700 1 --2- 172.21.15.189:0/1453141299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc004850 0x7f89bc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d61ca700 1 --2- 172.21.15.189:0/1453141299 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89bc003a00 0x7f89bc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d69cb700 1 --2- 172.21.15.189:0/1453141299 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d00cb870 0x7f89bc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d59c9700 1 -- 172.21.15.189:0/1453141299 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d00cb870 msgr2=0x7f89bc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:34.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d59c9700 1 --2- 172.21.15.189:0/1453141299 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d00cb870 0x7f89bc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d59c9700 1 -- 172.21.15.189:0/1453141299 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89bc003a00 msgr2=0x7f89bc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:34.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d59c9700 1 --2- 172.21.15.189:0/1453141299 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89bc003a00 0x7f89bc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d59c9700 1 -- 172.21.15.189:0/1453141299 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89bc0035b0 con 0x7f89bc004850 2024-04-06T12:12:34.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d69cb700 1 --2- 172.21.15.189:0/1453141299 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d00cb870 0x7f89bc005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:34.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d59c9700 1 --2- 172.21.15.189:0/1453141299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc004850 0x7f89bc004cd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f89c0009b60 tx=0x7f89bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9757a2df17af983e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:34.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d51c8700 1 -- 172.21.15.189:0/1453141299 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c000e6c0 con 0x7f89bc004850 2024-04-06T12:12:34.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d51c8700 1 -- 172.21.15.189:0/1453141299 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c000e820 con 0x7f89bc004850 2024-04-06T12:12:34.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.960+0000 7f89d51c8700 1 -- 172.21.15.189:0/1453141299 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c000eb90 con 0x7f89bc004850 2024-04-06T12:12:34.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.961+0000 7f89d842e700 1 -- 172.21.15.189:0/1453141299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc004850 msgr2=0x7f89bc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:34.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.961+0000 7f89d842e700 1 --2- 172.21.15.189:0/1453141299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc004850 0x7f89bc004cd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f89c0009b60 tx=0x7f89bc005880 comp rx=0 tx=0).stop 2024-04-06T12:12:34.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.961+0000 7f89d69cb700 1 -- 172.21.15.189:0/1453141299 reap_dead start 2024-04-06T12:12:34.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.961+0000 7f89d842e700 1 -- 172.21.15.189:0/1453141299 shutdown_connections 2024-04-06T12:12:34.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.961+0000 7f89d842e700 1 -- 172.21.15.189:0/1453141299 >> 172.21.15.189:0/1453141299 conn(0x7f89d00b2db0 msgr2=0x7f89d00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:34.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.961+0000 7f89d842e700 1 -- 172.21.15.189:0/1453141299 shutdown_connections 2024-04-06T12:12:34.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.961+0000 7f89d842e700 1 -- 172.21.15.189:0/1453141299 wait complete. 2024-04-06T12:12:34.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.962+0000 7f89d842e700 1 Processor -- start 2024-04-06T12:12:34.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.962+0000 7f89d842e700 1 -- start start 2024-04-06T12:12:34.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.963+0000 7f89d842e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89bc004850 0x7f89d00b5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.963+0000 7f89d842e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89d00c8d00 0x7f89d01476a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.963+0000 7f89d842e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d0147be0 0x7f89d01f4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.963+0000 7f89d842e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d0146ec0 con 0x7f89d00c8d00 2024-04-06T12:12:34.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.963+0000 7f89d842e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d01f1480 con 0x7f89bc004850 2024-04-06T12:12:34.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.963+0000 7f89d842e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89d01f1760 con 0x7f89d0147be0 2024-04-06T12:12:34.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.963+0000 7f89d59c9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89d00c8d00 0x7f89d01476a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.963+0000 7f89d59c9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89d00c8d00 0x7f89d01476a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38498/0 (socket says 172.21.15.189:38498) 2024-04-06T12:12:34.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.963+0000 7f89d59c9700 1 -- 172.21.15.189:0/4015259479 learned_addr learned my addr 172.21.15.189:0/4015259479 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:34.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.963+0000 7f89d61ca700 1 --2- 172.21.15.189:0/4015259479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89bc004850 0x7f89d00b5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.963+0000 7f89d69cb700 1 --2- 172.21.15.189:0/4015259479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d0147be0 0x7f89d01f4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.963+0000 7f89d59c9700 1 -- 172.21.15.189:0/4015259479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d0147be0 msgr2=0x7f89d01f4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:34.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.963+0000 7f89d59c9700 1 --2- 172.21.15.189:0/4015259479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d0147be0 0x7f89d01f4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.963+0000 7f89d59c9700 1 -- 172.21.15.189:0/4015259479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89bc004850 msgr2=0x7f89d00b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:34.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.963+0000 7f89d59c9700 1 --2- 172.21.15.189:0/4015259479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89bc004850 0x7f89d00b5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:34.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.963+0000 7f89d59c9700 1 -- 172.21.15.189:0/4015259479 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d01f1a40 con 0x7f89d00c8d00 2024-04-06T12:12:34.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.963+0000 7f89d61ca700 1 --2- 172.21.15.189:0/4015259479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89bc004850 0x7f89d00b5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:34.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.963+0000 7f89d59c9700 1 --2- 172.21.15.189:0/4015259479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89d00c8d00 0x7f89d01476a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f89bc004740 tx=0x7f89c00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:34.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.964+0000 7f89c6ffd700 1 -- 172.21.15.189:0/4015259479 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c0021070 con 0x7f89d00c8d00 2024-04-06T12:12:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.964+0000 7f89c6ffd700 1 -- 172.21.15.189:0/4015259479 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c00045d0 con 0x7f89d00c8d00 2024-04-06T12:12:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.964+0000 7f89d842e700 1 -- 172.21.15.189:0/4015259479 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89d0146ec0 con 0x7f89d00c8d00 2024-04-06T12:12:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.964+0000 7f89c6ffd700 1 -- 172.21.15.189:0/4015259479 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c000e6c0 con 0x7f89d00c8d00 2024-04-06T12:12:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.964+0000 7f89d842e700 1 -- 172.21.15.189:0/4015259479 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89d00b56a0 con 0x7f89d00c8d00 2024-04-06T12:12:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.965+0000 7f89d842e700 1 -- 172.21.15.189:0/4015259479 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89d01473c0 con 0x7f89d00c8d00 2024-04-06T12:12:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.965+0000 7f89c6ffd700 1 -- 172.21.15.189:0/4015259479 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f89c0025070 con 0x7f89d00c8d00 2024-04-06T12:12:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.965+0000 7f89c6ffd700 1 --2- 172.21.15.189:0/4015259479 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f89b406ac90 0x7f89b406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.965+0000 7f89c6ffd700 1 -- 172.21.15.189:0/4015259479 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f89c008a680 con 0x7f89d00c8d00 2024-04-06T12:12:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.965+0000 7f89d61ca700 1 --2- 172.21.15.189:0/4015259479 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f89b406ac90 0x7f89b406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.966+0000 7f89d61ca700 1 --2- 172.21.15.189:0/4015259479 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f89b406ac90 0x7f89b406d150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f89cc009ce0 tx=0x7f89cc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:34.970+0000 7f89c6ffd700 1 -- 172.21.15.189:0/4015259479 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f89d01473c0 con 0x7f89d00c8d00 2024-04-06T12:12:35.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.209+0000 7f89d842e700 1 -- 172.21.15.189:0/4015259479 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89d01473c0 con 0x7f89d00c8d00 2024-04-06T12:12:35.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.209+0000 7f89c6ffd700 1 -- 172.21.15.189:0/4015259479 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4739 (secure 0 0 0) 0x7f89d01473c0 con 0x7f89d00c8d00 2024-04-06T12:12:35.207 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:35.207 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2769778155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2769778155},{"type":"v1","addr":"172.21.15.189:6835","nonce":2769778155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4233,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3550531296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3550531296},{"type":"v1","addr":"172.21.15.189:6837","nonce":3550531296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3198235148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3198235148},{"type":"v1","addr":"172.21.15.192:6833","nonce":3198235148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:12:04.195581+0000","modified":"2024-04-06T12:12:26.535185+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/516212261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":516212261},{"type":"v1","addr":"172.21.15.192:6835","nonce":516212261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-06T12:12:35.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.211+0000 7f89d842e700 1 -- 172.21.15.189:0/4015259479 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f89b406ac90 msgr2=0x7f89b406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:35.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.211+0000 7f89d842e700 1 --2- 172.21.15.189:0/4015259479 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f89b406ac90 0x7f89b406d150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f89cc009ce0 tx=0x7f89cc008040 comp rx=0 tx=0).stop 2024-04-06T12:12:35.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.211+0000 7f89d842e700 1 -- 172.21.15.189:0/4015259479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89d00c8d00 msgr2=0x7f89d01476a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:35.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.211+0000 7f89d842e700 1 --2- 172.21.15.189:0/4015259479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89d00c8d00 0x7f89d01476a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f89bc004740 tx=0x7f89c00047c0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.211+0000 7f89d842e700 1 -- 172.21.15.189:0/4015259479 shutdown_connections 2024-04-06T12:12:35.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.211+0000 7f89d842e700 1 --2- 172.21.15.189:0/4015259479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d0147be0 0x7f89d01f4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.211+0000 7f89d842e700 1 --2- 172.21.15.189:0/4015259479 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f89b406ac90 0x7f89b406d150 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.211+0000 7f89d842e700 1 --2- 172.21.15.189:0/4015259479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89bc004850 0x7f89d00b5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.211+0000 7f89d842e700 1 --2- 172.21.15.189:0/4015259479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89d00c8d00 0x7f89d01476a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.211+0000 7f89d842e700 1 -- 172.21.15.189:0/4015259479 >> 172.21.15.189:0/4015259479 conn(0x7f89d00b2db0 msgr2=0x7f89d015a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:35.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.211+0000 7f89d842e700 1 -- 172.21.15.189:0/4015259479 shutdown_connections 2024-04-06T12:12:35.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.211+0000 7f89d842e700 1 -- 172.21.15.189:0/4015259479 wait complete. 2024-04-06T12:12:35.211 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 9 2024-04-06T12:12:35.222 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:35.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.344+0000 7f3abe60d700 1 Processor -- start 2024-04-06T12:12:35.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.344+0000 7f3abe60d700 1 -- start start 2024-04-06T12:12:35.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.345+0000 7f3abe60d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ab815d7b0 0x7f3ab80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:35.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.345+0000 7f3abe60d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab80b3990 con 0x7f3ab815e500 2024-04-06T12:12:35.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.345+0000 7f3abe60d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab80b3060 con 0x7f3ab806a730 2024-04-06T12:12:35.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.345+0000 7f3abe60d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab80b3340 con 0x7f3ab815d7b0 2024-04-06T12:12:35.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.345+0000 7f3abcbaa700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3ab815e500 0x7f3ab80717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34406/0 (socket says 172.21.15.189:34406) 2024-04-06T12:12:35.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.345+0000 7f3abcbaa700 1 -- 172.21.15.189:0/36290568 learned_addr learned my addr 172.21.15.189:0/36290568 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:35.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab7fff700 1 --2- 172.21.15.189:0/36290568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ab815d7b0 0x7f3ab80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:35.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/36290568 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896633417 0 0) 0x7f3ab80b3990 con 0x7f3ab815e500 2024-04-06T12:12:35.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/36290568 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aa00035b0 con 0x7f3ab815e500 2024-04-06T12:12:35.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/36290568 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1613523970 0 0) 0x7f3aa00035b0 con 0x7f3ab815e500 2024-04-06T12:12:35.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/36290568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ab815d7b0 msgr2=0x7f3ab80b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:35.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab6ffd700 1 --2- 172.21.15.189:0/36290568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ab815d7b0 0x7f3ab80b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/36290568 >> v1:172.21.15.192:6789/0 conn(0x7f3ab806a730 legacy=0x7f3ab815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:35.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/36290568 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab80b3620 con 0x7f3ab815e500 2024-04-06T12:12:35.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/36290568 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f3ab00019f0 con 0x7f3ab815e500 2024-04-06T12:12:35.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/36290568 >> v1:172.21.15.189:6789/0 conn(0x7f3ab815e500 legacy=0x7f3ab80717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:35.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab6ffd700 1 --2- 172.21.15.189:0/36290568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0003a30 0x7f3aa0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:35.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab6ffd700 1 --2- 172.21.15.189:0/36290568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3aa00043e0 0x7f3aa00048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:35.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab6ffd700 1 --2- 172.21.15.189:0/36290568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ab815d7b0 0x7f3aa00051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:35.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/36290568 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab80b3340 con 0x7f3aa0003a30 2024-04-06T12:12:35.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/36290568 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa000bac0 con 0x7f3aa00043e0 2024-04-06T12:12:35.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/36290568 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa000bc80 con 0x7f3ab815d7b0 2024-04-06T12:12:35.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab7fff700 1 --2- 172.21.15.189:0/36290568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0003a30 0x7f3aa0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:35.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3abcbaa700 1 --2- 172.21.15.189:0/36290568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ab815d7b0 0x7f3aa00051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:35.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab77fe700 1 --2- 172.21.15.189:0/36290568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3aa00043e0 0x7f3aa00048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:35.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab7fff700 1 -- 172.21.15.189:0/36290568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ab815d7b0 msgr2=0x7f3aa00051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:35.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab7fff700 1 --2- 172.21.15.189:0/36290568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ab815d7b0 0x7f3aa00051b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab7fff700 1 -- 172.21.15.189:0/36290568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3aa00043e0 msgr2=0x7f3aa00048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:35.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab7fff700 1 --2- 172.21.15.189:0/36290568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3aa00043e0 0x7f3aa00048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.346+0000 7f3ab7fff700 1 -- 172.21.15.189:0/36290568 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa00035b0 con 0x7f3aa0003a30 2024-04-06T12:12:35.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.347+0000 7f3ab7fff700 1 --2- 172.21.15.189:0/36290568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0003a30 0x7f3aa0003e10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8009b60 tx=0x7f3aa0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e56c0ac49adb0ca server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:35.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.347+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/36290568 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa800e6c0 con 0x7f3aa0003a30 2024-04-06T12:12:35.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.347+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/36290568 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa800e820 con 0x7f3aa0003a30 2024-04-06T12:12:35.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.347+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/36290568 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa800eb90 con 0x7f3aa0003a30 2024-04-06T12:12:35.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.347+0000 7f3abe60d700 1 -- 172.21.15.189:0/36290568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0003a30 msgr2=0x7f3aa0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:35.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.347+0000 7f3abe60d700 1 --2- 172.21.15.189:0/36290568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0003a30 0x7f3aa0003e10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8009b60 tx=0x7f3aa0005880 comp rx=0 tx=0).stop 2024-04-06T12:12:35.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.347+0000 7f3abcbaa700 1 -- 172.21.15.189:0/36290568 reap_dead start 2024-04-06T12:12:35.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.347+0000 7f3abe60d700 1 -- 172.21.15.189:0/36290568 shutdown_connections 2024-04-06T12:12:35.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.347+0000 7f3abe60d700 1 -- 172.21.15.189:0/36290568 >> 172.21.15.189:0/36290568 conn(0x7f3ab80d00b0 msgr2=0x7f3ab80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:35.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.347+0000 7f3abe60d700 1 -- 172.21.15.189:0/36290568 shutdown_connections 2024-04-06T12:12:35.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.347+0000 7f3abe60d700 1 -- 172.21.15.189:0/36290568 wait complete. 2024-04-06T12:12:35.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.348+0000 7f3abe60d700 1 Processor -- start 2024-04-06T12:12:35.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.349+0000 7f3abe60d700 1 -- start start 2024-04-06T12:12:35.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.349+0000 7f3abe60d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3aa0003a30 0x7f3ab814a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:35.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.349+0000 7f3abe60d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab8155a00 0x7f3ab8158f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:35.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.349+0000 7f3abe60d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ab8159460 0x7f3ab815eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:35.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.349+0000 7f3abe60d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab81598e0 con 0x7f3ab8155a00 2024-04-06T12:12:35.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.349+0000 7f3abe60d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab8154c40 con 0x7f3ab8159460 2024-04-06T12:12:35.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.349+0000 7f3abe60d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab8154f20 con 0x7f3aa0003a30 2024-04-06T12:12:35.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.350+0000 7f3ab77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab8155a00 0x7f3ab8158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:35.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.350+0000 7f3ab77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab8155a00 0x7f3ab8158f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38518/0 (socket says 172.21.15.189:38518) 2024-04-06T12:12:35.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.350+0000 7f3ab77fe700 1 -- 172.21.15.189:0/1129217485 learned_addr learned my addr 172.21.15.189:0/1129217485 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:35.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.350+0000 7f3abcbaa700 1 --2- 172.21.15.189:0/1129217485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ab8159460 0x7f3ab815eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:35.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.350+0000 7f3ab7fff700 1 --2- 172.21.15.189:0/1129217485 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3aa0003a30 0x7f3ab814a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:35.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.350+0000 7f3ab77fe700 1 -- 172.21.15.189:0/1129217485 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3aa0003a30 msgr2=0x7f3ab814a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:35.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.350+0000 7f3ab77fe700 1 --2- 172.21.15.189:0/1129217485 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3aa0003a30 0x7f3ab814a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.350+0000 7f3ab77fe700 1 -- 172.21.15.189:0/1129217485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ab8159460 msgr2=0x7f3ab815eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:35.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.350+0000 7f3ab77fe700 1 --2- 172.21.15.189:0/1129217485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ab8159460 0x7f3ab815eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.350+0000 7f3ab77fe700 1 -- 172.21.15.189:0/1129217485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab8155200 con 0x7f3ab8155a00 2024-04-06T12:12:35.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.350+0000 7f3ab7fff700 1 --2- 172.21.15.189:0/1129217485 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3aa0003a30 0x7f3ab814a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:35.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.350+0000 7f3ab77fe700 1 --2- 172.21.15.189:0/1129217485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab8155a00 0x7f3ab8158f20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3aac001eb0 tx=0x7f3aac00faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:35.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.350+0000 7f3ab4ff9700 1 -- 172.21.15.189:0/1129217485 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac0052e0 con 0x7f3ab8155a00 2024-04-06T12:12:35.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.350+0000 7f3ab4ff9700 1 -- 172.21.15.189:0/1129217485 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aac005460 con 0x7f3ab8155a00 2024-04-06T12:12:35.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.350+0000 7f3abe60d700 1 -- 172.21.15.189:0/1129217485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab81554e0 con 0x7f3ab8155a00 2024-04-06T12:12:35.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.350+0000 7f3ab4ff9700 1 -- 172.21.15.189:0/1129217485 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac0038c0 con 0x7f3ab8155a00 2024-04-06T12:12:35.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.350+0000 7f3abe60d700 1 -- 172.21.15.189:0/1129217485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab815f530 con 0x7f3ab8155a00 2024-04-06T12:12:35.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.351+0000 7f3abe60d700 1 -- 172.21.15.189:0/1129217485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab8154d80 con 0x7f3ab8155a00 2024-04-06T12:12:35.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.351+0000 7f3ab4ff9700 1 -- 172.21.15.189:0/1129217485 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3aac003a20 con 0x7f3ab8155a00 2024-04-06T12:12:35.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.352+0000 7f3ab4ff9700 1 --2- 172.21.15.189:0/1129217485 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3aa406ac90 0x7f3aa406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:35.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.352+0000 7f3ab4ff9700 1 -- 172.21.15.189:0/1129217485 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3aac0894d0 con 0x7f3ab8155a00 2024-04-06T12:12:35.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.352+0000 7f3ab7fff700 1 --2- 172.21.15.189:0/1129217485 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3aa406ac90 0x7f3aa406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:35.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.352+0000 7f3ab7fff700 1 --2- 172.21.15.189:0/1129217485 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3aa406ac90 0x7f3aa406d150 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80090b0 tx=0x7f3aa8008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:35.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.356+0000 7f3ab4ff9700 1 -- 172.21.15.189:0/1129217485 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3ab8154d80 con 0x7f3ab8155a00 2024-04-06T12:12:35.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.551+0000 7f3abe60d700 1 -- 172.21.15.189:0/1129217485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ab8154d80 con 0x7f3ab8155a00 2024-04-06T12:12:35.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.552+0000 7f3ab4ff9700 1 -- 172.21.15.189:0/1129217485 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f3ab8154d80 con 0x7f3ab8155a00 2024-04-06T12:12:35.551 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:35.551 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:08.649170+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:35.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.554+0000 7f3abe60d700 1 -- 172.21.15.189:0/1129217485 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3aa406ac90 msgr2=0x7f3aa406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:35.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.554+0000 7f3abe60d700 1 --2- 172.21.15.189:0/1129217485 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3aa406ac90 0x7f3aa406d150 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80090b0 tx=0x7f3aa8008030 comp rx=0 tx=0).stop 2024-04-06T12:12:35.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.554+0000 7f3abe60d700 1 -- 172.21.15.189:0/1129217485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab8155a00 msgr2=0x7f3ab8158f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:35.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.554+0000 7f3abe60d700 1 --2- 172.21.15.189:0/1129217485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab8155a00 0x7f3ab8158f20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3aac001eb0 tx=0x7f3aac00faa0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.554+0000 7f3abe60d700 1 -- 172.21.15.189:0/1129217485 shutdown_connections 2024-04-06T12:12:35.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.554+0000 7f3abe60d700 1 --2- 172.21.15.189:0/1129217485 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3aa406ac90 0x7f3aa406d150 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.554+0000 7f3abe60d700 1 --2- 172.21.15.189:0/1129217485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ab8159460 0x7f3ab815eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.554+0000 7f3abe60d700 1 --2- 172.21.15.189:0/1129217485 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3aa0003a30 0x7f3ab814a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.554+0000 7f3abe60d700 1 --2- 172.21.15.189:0/1129217485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab8155a00 0x7f3ab8158f20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.554+0000 7f3abe60d700 1 -- 172.21.15.189:0/1129217485 >> 172.21.15.189:0/1129217485 conn(0x7f3ab80d00b0 msgr2=0x7f3ab806f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:35.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.555+0000 7f3abe60d700 1 -- 172.21.15.189:0/1129217485 shutdown_connections 2024-04-06T12:12:35.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.555+0000 7f3abe60d700 1 -- 172.21.15.189:0/1129217485 wait complete. 2024-04-06T12:12:35.565 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:12:02.280866+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:12:03.156593+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-06T12:12:06.237184+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-06T12:12:35.566 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:12:35.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.694+0000 7f0ab7839700 1 Processor -- start 2024-04-06T12:12:35.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.695+0000 7f0ab7839700 1 -- start start 2024-04-06T12:12:35.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.695+0000 7f0ab7839700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ab00d17a0 0x7f0ab006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:35.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.695+0000 7f0ab7839700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0ab0146dd0 0x7f0ab006d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:35.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.695+0000 7f0ab7839700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab006c0a0 con 0x7f0ab00cedf0 2024-04-06T12:12:35.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.695+0000 7f0ab7839700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab006c380 con 0x7f0ab0146dd0 2024-04-06T12:12:35.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.695+0000 7f0ab7839700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab006c660 con 0x7f0ab00d17a0 2024-04-06T12:12:35.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.696+0000 7f0ab4dd4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0ab0146dd0 0x7f0ab006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:35.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.696+0000 7f0ab5dd6700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0ab00cedf0 0x7f0ab01441f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34420/0 (socket says 172.21.15.189:34420) 2024-04-06T12:12:35.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.696+0000 7f0ab5dd6700 1 -- 172.21.15.189:0/3607179761 learned_addr learned my addr 172.21.15.189:0/3607179761 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:35.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.696+0000 7f0ab55d5700 1 --2- 172.21.15.189:0/3607179761 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ab00d17a0 0x7f0ab006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:35.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.696+0000 7f0ab4dd4700 1 -- 172.21.15.189:0/3607179761 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ab00d17a0 msgr2=0x7f0ab006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:35.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.696+0000 7f0ab4dd4700 1 --2- 172.21.15.189:0/3607179761 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ab00d17a0 0x7f0ab006b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.696+0000 7f0ab4dd4700 1 -- 172.21.15.189:0/3607179761 >> v1:172.21.15.189:6789/0 conn(0x7f0ab00cedf0 legacy=0x7f0ab01441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:35.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.696+0000 7f0aa7fff700 1 -- 172.21.15.189:0/3607179761 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2432622618 0 0) 0x7f0ab006c0a0 con 0x7f0ab00cedf0 2024-04-06T12:12:35.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.696+0000 7f0ab4dd4700 1 -- 172.21.15.189:0/3607179761 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab006c940 con 0x7f0ab0146dd0 2024-04-06T12:12:35.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.696+0000 7f0ab4dd4700 1 --2- 172.21.15.189:0/3607179761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab0146dd0 0x7f0ab006d140 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0aac009840 tx=0x7f0aac009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f734793d4f3953fc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:35.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.696+0000 7f0aa7fff700 1 -- 172.21.15.189:0/3607179761 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac011830 con 0x7f0ab0146dd0 2024-04-06T12:12:35.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.696+0000 7f0aa7fff700 1 -- 172.21.15.189:0/3607179761 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aac011990 con 0x7f0ab0146dd0 2024-04-06T12:12:35.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.696+0000 7f0aa7fff700 1 -- 172.21.15.189:0/3607179761 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac011d20 con 0x7f0ab0146dd0 2024-04-06T12:12:35.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.697+0000 7f0ab7839700 1 -- 172.21.15.189:0/3607179761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab0146dd0 msgr2=0x7f0ab006d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:35.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.697+0000 7f0ab7839700 1 --2- 172.21.15.189:0/3607179761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab0146dd0 0x7f0ab006d140 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0aac009840 tx=0x7f0aac009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:35.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.697+0000 7f0ab7839700 1 -- 172.21.15.189:0/3607179761 shutdown_connections 2024-04-06T12:12:35.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.697+0000 7f0ab7839700 1 --2- 172.21.15.189:0/3607179761 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ab00d17a0 0x7f0ab006b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.697+0000 7f0ab7839700 1 --2- 172.21.15.189:0/3607179761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab0146dd0 0x7f0ab006d140 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.697+0000 7f0ab7839700 1 -- 172.21.15.189:0/3607179761 >> 172.21.15.189:0/3607179761 conn(0x7f0ab00b2db0 msgr2=0x7f0ab00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:35.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.697+0000 7f0ab7839700 1 -- 172.21.15.189:0/3607179761 shutdown_connections 2024-04-06T12:12:35.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.697+0000 7f0ab7839700 1 -- 172.21.15.189:0/3607179761 wait complete. 2024-04-06T12:12:35.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.698+0000 7f0ab7839700 1 Processor -- start 2024-04-06T12:12:35.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.698+0000 7f0ab7839700 1 -- start start 2024-04-06T12:12:35.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.699+0000 7f0ab7839700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ab00cedf0 0x7f0ab00ad4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:35.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.699+0000 7f0ab7839700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ab00d17a0 0x7f0ab00b4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:35.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.699+0000 7f0ab7839700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab0146dd0 0x7f0ab015b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:35.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.699+0000 7f0ab7839700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab00b4eb0 con 0x7f0ab0146dd0 2024-04-06T12:12:35.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.699+0000 7f0ab7839700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab00b5300 con 0x7f0ab00d17a0 2024-04-06T12:12:35.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.699+0000 7f0ab7839700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab00b55e0 con 0x7f0ab00cedf0 2024-04-06T12:12:35.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.699+0000 7f0ab5dd6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab0146dd0 0x7f0ab015b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:35.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.699+0000 7f0ab5dd6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab0146dd0 0x7f0ab015b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38532/0 (socket says 172.21.15.189:38532) 2024-04-06T12:12:35.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.699+0000 7f0ab5dd6700 1 -- 172.21.15.189:0/3180348926 learned_addr learned my addr 172.21.15.189:0/3180348926 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:35.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.699+0000 7f0ab4dd4700 1 --2- 172.21.15.189:0/3180348926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ab00d17a0 0x7f0ab00b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:35.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.699+0000 7f0ab55d5700 1 --2- 172.21.15.189:0/3180348926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ab00cedf0 0x7f0ab00ad4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:35.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.699+0000 7f0ab5dd6700 1 -- 172.21.15.189:0/3180348926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ab00cedf0 msgr2=0x7f0ab00ad4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:35.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.699+0000 7f0ab5dd6700 1 --2- 172.21.15.189:0/3180348926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ab00cedf0 0x7f0ab00ad4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.699+0000 7f0ab5dd6700 1 -- 172.21.15.189:0/3180348926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ab00d17a0 msgr2=0x7f0ab00b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:35.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.699+0000 7f0ab5dd6700 1 --2- 172.21.15.189:0/3180348926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ab00d17a0 0x7f0ab00b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.699+0000 7f0ab5dd6700 1 -- 172.21.15.189:0/3180348926 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac0094a0 con 0x7f0ab0146dd0 2024-04-06T12:12:35.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.699+0000 7f0ab55d5700 1 --2- 172.21.15.189:0/3180348926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ab00cedf0 0x7f0ab00ad4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:35.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.699+0000 7f0ab5dd6700 1 --2- 172.21.15.189:0/3180348926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab0146dd0 0x7f0ab015b8c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800cb60 tx=0x7f0aa800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:35.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.700+0000 7f0aa5ffb700 1 -- 172.21.15.189:0/3180348926 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa800ab50 con 0x7f0ab0146dd0 2024-04-06T12:12:35.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.700+0000 7f0aa5ffb700 1 -- 172.21.15.189:0/3180348926 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa800acb0 con 0x7f0ab0146dd0 2024-04-06T12:12:35.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.700+0000 7f0ab7839700 1 -- 172.21.15.189:0/3180348926 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab00b58c0 con 0x7f0ab0146dd0 2024-04-06T12:12:35.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.700+0000 7f0aa5ffb700 1 -- 172.21.15.189:0/3180348926 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8007310 con 0x7f0ab0146dd0 2024-04-06T12:12:35.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.700+0000 7f0ab7839700 1 -- 172.21.15.189:0/3180348926 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab00b5e80 con 0x7f0ab0146dd0 2024-04-06T12:12:35.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.701+0000 7f0ab7839700 1 -- 172.21.15.189:0/3180348926 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab00b6070 con 0x7f0ab0146dd0 2024-04-06T12:12:35.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.701+0000 7f0aa5ffb700 1 -- 172.21.15.189:0/3180348926 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0aa8007470 con 0x7f0ab0146dd0 2024-04-06T12:12:35.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.701+0000 7f0aa5ffb700 1 --2- 172.21.15.189:0/3180348926 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0a9406ac90 0x7f0a9406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:35.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.701+0000 7f0aa5ffb700 1 -- 172.21.15.189:0/3180348926 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0aa8089480 con 0x7f0ab0146dd0 2024-04-06T12:12:35.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.702+0000 7f0ab55d5700 1 --2- 172.21.15.189:0/3180348926 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0a9406ac90 0x7f0a9406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:35.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.702+0000 7f0ab55d5700 1 --2- 172.21.15.189:0/3180348926 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0a9406ac90 0x7f0a9406d150 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0006fd0 tx=0x7f0aa0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:35.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.706+0000 7f0aa5ffb700 1 -- 172.21.15.189:0/3180348926 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0aa80542e0 con 0x7f0ab0146dd0 2024-04-06T12:12:35.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.900+0000 7f0ab7839700 1 -- 172.21.15.189:0/3180348926 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0ab0074990 con 0x7f0ab0146dd0 2024-04-06T12:12:35.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.900+0000 7f0aa5ffb700 1 -- 172.21.15.189:0/3180348926 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0aa80180a0 con 0x7f0ab0146dd0 2024-04-06T12:12:35.899 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:12:35.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.902+0000 7f0ab7839700 1 -- 172.21.15.189:0/3180348926 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0a9406ac90 msgr2=0x7f0a9406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:35.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.902+0000 7f0ab7839700 1 --2- 172.21.15.189:0/3180348926 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0a9406ac90 0x7f0a9406d150 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0006fd0 tx=0x7f0aa0008040 comp rx=0 tx=0).stop 2024-04-06T12:12:35.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.902+0000 7f0ab7839700 1 -- 172.21.15.189:0/3180348926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab0146dd0 msgr2=0x7f0ab015b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:35.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.902+0000 7f0ab7839700 1 --2- 172.21.15.189:0/3180348926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab0146dd0 0x7f0ab015b8c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800cb60 tx=0x7f0aa800cf20 comp rx=0 tx=0).stop 2024-04-06T12:12:35.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.903+0000 7f0ab7839700 1 -- 172.21.15.189:0/3180348926 shutdown_connections 2024-04-06T12:12:35.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.903+0000 7f0ab7839700 1 --2- 172.21.15.189:0/3180348926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab0146dd0 0x7f0ab015b8c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.903+0000 7f0ab7839700 1 --2- 172.21.15.189:0/3180348926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ab00cedf0 0x7f0ab00ad4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.903+0000 7f0ab7839700 1 --2- 172.21.15.189:0/3180348926 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0a9406ac90 0x7f0a9406d150 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.903+0000 7f0ab7839700 1 --2- 172.21.15.189:0/3180348926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ab00d17a0 0x7f0ab00b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:35.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.903+0000 7f0ab7839700 1 -- 172.21.15.189:0/3180348926 >> 172.21.15.189:0/3180348926 conn(0x7f0ab00b2db0 msgr2=0x7f0ab00cf220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:35.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.903+0000 7f0ab7839700 1 -- 172.21.15.189:0/3180348926 shutdown_connections 2024-04-06T12:12:35.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:35.903+0000 7f0ab7839700 1 -- 172.21.15.189:0/3180348926 wait complete. 2024-04-06T12:12:35.913 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:12:36.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.040+0000 7eff7eda6700 1 Processor -- start 2024-04-06T12:12:36.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.040+0000 7eff7eda6700 1 -- start start 2024-04-06T12:12:36.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.041+0000 7eff7eda6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff78144f40 0x7eff7806a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.041+0000 7eff7eda6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff7815c740 con 0x7eff7815c150 2024-04-06T12:12:36.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.041+0000 7eff7eda6700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff7815f1f0 con 0x7eff7815fcd0 2024-04-06T12:12:36.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.041+0000 7eff7eda6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff7815c560 con 0x7eff78144f40 2024-04-06T12:12:36.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.041+0000 7eff7d343700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7eff7815c150 0x7eff780ceca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34424/0 (socket says 172.21.15.189:34424) 2024-04-06T12:12:36.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.041+0000 7eff7d343700 1 -- 172.21.15.189:0/3829087158 learned_addr learned my addr 172.21.15.189:0/3829087158 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:36.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.041+0000 7eff7cb42700 1 --2- 172.21.15.189:0/3829087158 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff78144f40 0x7eff7806a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.041+0000 7eff777fe700 1 -- 172.21.15.189:0/3829087158 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538002745 0 0) 0x7eff7815c740 con 0x7eff7815c150 2024-04-06T12:12:36.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.041+0000 7eff777fe700 1 -- 172.21.15.189:0/3829087158 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff680035b0 con 0x7eff7815c150 2024-04-06T12:12:36.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.041+0000 7eff777fe700 1 -- 172.21.15.189:0/3829087158 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639703936 0 0) 0x7eff7815f1f0 con 0x7eff7815fcd0 2024-04-06T12:12:36.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.041+0000 7eff777fe700 1 -- 172.21.15.189:0/3829087158 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff7815c740 con 0x7eff7815fcd0 2024-04-06T12:12:36.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.041+0000 7eff777fe700 1 -- 172.21.15.189:0/3829087158 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1688306028 0 0) 0x7eff680035b0 con 0x7eff7815c150 2024-04-06T12:12:36.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.041+0000 7eff777fe700 1 -- 172.21.15.189:0/3829087158 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff78144f40 msgr2=0x7eff7806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:36.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.041+0000 7eff777fe700 1 --2- 172.21.15.189:0/3829087158 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff78144f40 0x7eff7806a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.041+0000 7eff777fe700 1 -- 172.21.15.189:0/3829087158 >> v1:172.21.15.192:6789/0 conn(0x7eff7815fcd0 legacy=0x7eff7815ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:36.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.041+0000 7eff777fe700 1 -- 172.21.15.189:0/3829087158 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7815f4d0 con 0x7eff7815c150 2024-04-06T12:12:36.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.041+0000 7eff777fe700 1 -- 172.21.15.189:0/3829087158 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7eff700019f0 con 0x7eff7815c150 2024-04-06T12:12:36.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.041+0000 7eff777fe700 1 -- 172.21.15.189:0/3829087158 >> v1:172.21.15.189:6789/0 conn(0x7eff7815c150 legacy=0x7eff780ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:36.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.041+0000 7eff777fe700 1 --2- 172.21.15.189:0/3829087158 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff78144f40 0x7eff68003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.042+0000 7eff777fe700 1 --2- 172.21.15.189:0/3829087158 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff68004120 0x7eff68004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.042+0000 7eff777fe700 1 --2- 172.21.15.189:0/3829087158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff68004d90 0x7eff68005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.042+0000 7eff777fe700 1 -- 172.21.15.189:0/3829087158 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7815c560 con 0x7eff68004d90 2024-04-06T12:12:36.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.042+0000 7eff777fe700 1 -- 172.21.15.189:0/3829087158 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6800bba0 con 0x7eff68004120 2024-04-06T12:12:36.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.042+0000 7eff777fe700 1 -- 172.21.15.189:0/3829087158 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff6800bd30 con 0x7eff78144f40 2024-04-06T12:12:36.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.042+0000 7eff7d343700 1 --2- 172.21.15.189:0/3829087158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff68004d90 0x7eff68005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.042+0000 7eff7cb42700 1 --2- 172.21.15.189:0/3829087158 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff78144f40 0x7eff68003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.042+0000 7eff77fff700 1 --2- 172.21.15.189:0/3829087158 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff68004120 0x7eff68004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.042+0000 7eff7d343700 1 -- 172.21.15.189:0/3829087158 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff78144f40 msgr2=0x7eff68003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:36.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.042+0000 7eff7d343700 1 --2- 172.21.15.189:0/3829087158 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff78144f40 0x7eff68003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.042+0000 7eff7d343700 1 -- 172.21.15.189:0/3829087158 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff68004120 msgr2=0x7eff68004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:36.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.042+0000 7eff7d343700 1 --2- 172.21.15.189:0/3829087158 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff68004120 0x7eff68004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.042+0000 7eff7d343700 1 -- 172.21.15.189:0/3829087158 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff680035b0 con 0x7eff68004d90 2024-04-06T12:12:36.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.042+0000 7eff7d343700 1 --2- 172.21.15.189:0/3829087158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff68004d90 0x7eff68005210 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7eff7000a7a0 tx=0x7eff68005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38dfab06652c8104 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:36.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.042+0000 7eff777fe700 1 -- 172.21.15.189:0/3829087158 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff70012600 con 0x7eff68004d90 2024-04-06T12:12:36.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.042+0000 7eff777fe700 1 -- 172.21.15.189:0/3829087158 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff70012780 con 0x7eff68004d90 2024-04-06T12:12:36.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.042+0000 7eff777fe700 1 -- 172.21.15.189:0/3829087158 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff70012b30 con 0x7eff68004d90 2024-04-06T12:12:36.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.043+0000 7eff7eda6700 1 -- 172.21.15.189:0/3829087158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff68004d90 msgr2=0x7eff68005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:36.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.043+0000 7eff7eda6700 1 --2- 172.21.15.189:0/3829087158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff68004d90 0x7eff68005210 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7eff7000a7a0 tx=0x7eff68005880 comp rx=0 tx=0).stop 2024-04-06T12:12:36.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.043+0000 7eff7d343700 1 -- 172.21.15.189:0/3829087158 reap_dead start 2024-04-06T12:12:36.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.043+0000 7eff7eda6700 1 -- 172.21.15.189:0/3829087158 shutdown_connections 2024-04-06T12:12:36.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.043+0000 7eff7eda6700 1 -- 172.21.15.189:0/3829087158 >> 172.21.15.189:0/3829087158 conn(0x7eff780d00b0 msgr2=0x7eff780d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:36.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.043+0000 7eff7eda6700 1 -- 172.21.15.189:0/3829087158 shutdown_connections 2024-04-06T12:12:36.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.043+0000 7eff7eda6700 1 -- 172.21.15.189:0/3829087158 wait complete. 2024-04-06T12:12:36.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.044+0000 7eff7eda6700 1 Processor -- start 2024-04-06T12:12:36.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.044+0000 7eff7eda6700 1 -- start start 2024-04-06T12:12:36.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.045+0000 7eff7eda6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff7815acb0 0x7eff7813c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.045+0000 7eff7eda6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff7813c9b0 0x7eff78139b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.045+0000 7eff7eda6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff7813a060 0x7eff78138060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.045+0000 7eff7eda6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7813ce30 con 0x7eff7813c9b0 2024-04-06T12:12:36.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.045+0000 7eff7eda6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7815b090 con 0x7eff7815acb0 2024-04-06T12:12:36.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.045+0000 7eff7eda6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff7813a810 con 0x7eff7813a060 2024-04-06T12:12:36.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.045+0000 7eff77fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff7813c9b0 0x7eff78139b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.045+0000 7eff77fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff7813c9b0 0x7eff78139b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38540/0 (socket says 172.21.15.189:38540) 2024-04-06T12:12:36.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.045+0000 7eff77fff700 1 -- 172.21.15.189:0/3005133369 learned_addr learned my addr 172.21.15.189:0/3005133369 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:36.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.046+0000 7eff7cb42700 1 --2- 172.21.15.189:0/3005133369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff7815acb0 0x7eff7813c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.046+0000 7eff7d343700 1 --2- 172.21.15.189:0/3005133369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff7813a060 0x7eff78138060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.046+0000 7eff77fff700 1 -- 172.21.15.189:0/3005133369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff7813a060 msgr2=0x7eff78138060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:36.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.046+0000 7eff77fff700 1 --2- 172.21.15.189:0/3005133369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff7813a060 0x7eff78138060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.046+0000 7eff77fff700 1 -- 172.21.15.189:0/3005133369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff7815acb0 msgr2=0x7eff7813c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:36.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.046+0000 7eff77fff700 1 --2- 172.21.15.189:0/3005133369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff7815acb0 0x7eff7813c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.046+0000 7eff77fff700 1 -- 172.21.15.189:0/3005133369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7813aaf0 con 0x7eff7813c9b0 2024-04-06T12:12:36.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.046+0000 7eff77fff700 1 --2- 172.21.15.189:0/3005133369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff7813c9b0 0x7eff78139b20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7eff6400f7e0 tx=0x7eff6400fba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:36.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.046+0000 7eff757fa700 1 -- 172.21.15.189:0/3005133369 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff64001520 con 0x7eff7813c9b0 2024-04-06T12:12:36.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.046+0000 7eff757fa700 1 -- 172.21.15.189:0/3005133369 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff640052e0 con 0x7eff7813c9b0 2024-04-06T12:12:36.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.046+0000 7eff7eda6700 1 -- 172.21.15.189:0/3005133369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff7813add0 con 0x7eff7813c9b0 2024-04-06T12:12:36.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.046+0000 7eff757fa700 1 -- 172.21.15.189:0/3005133369 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff64003680 con 0x7eff7813c9b0 2024-04-06T12:12:36.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.046+0000 7eff7eda6700 1 -- 172.21.15.189:0/3005133369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff7813b390 con 0x7eff7813c9b0 2024-04-06T12:12:36.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.047+0000 7eff7eda6700 1 -- 172.21.15.189:0/3005133369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff78074990 con 0x7eff7813c9b0 2024-04-06T12:12:36.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.048+0000 7eff757fa700 1 -- 172.21.15.189:0/3005133369 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7eff64003800 con 0x7eff7813c9b0 2024-04-06T12:12:36.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.048+0000 7eff757fa700 1 --2- 172.21.15.189:0/3005133369 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7eff5c06ac90 0x7eff5c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.048+0000 7eff757fa700 1 -- 172.21.15.189:0/3005133369 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7eff64089580 con 0x7eff7813c9b0 2024-04-06T12:12:36.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.048+0000 7eff7cb42700 1 --2- 172.21.15.189:0/3005133369 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7eff5c06ac90 0x7eff5c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.048+0000 7eff7cb42700 1 --2- 172.21.15.189:0/3005133369 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7eff5c06ac90 0x7eff5c06d150 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009ba0 tx=0x7eff6c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:36.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.052+0000 7eff757fa700 1 -- 172.21.15.189:0/3005133369 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7eff64054470 con 0x7eff7813c9b0 2024-04-06T12:12:36.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.250+0000 7eff7eda6700 1 -- 172.21.15.189:0/3005133369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7eff7815f1e0 con 0x7eff7813c9b0 2024-04-06T12:12:36.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.250+0000 7eff757fa700 1 -- 172.21.15.189:0/3005133369 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7eff64057a90 con 0x7eff7813c9b0 2024-04-06T12:12:36.248 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:12:36.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.253+0000 7eff7eda6700 1 -- 172.21.15.189:0/3005133369 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7eff5c06ac90 msgr2=0x7eff5c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:36.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.253+0000 7eff7eda6700 1 --2- 172.21.15.189:0/3005133369 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7eff5c06ac90 0x7eff5c06d150 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009ba0 tx=0x7eff6c008040 comp rx=0 tx=0).stop 2024-04-06T12:12:36.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.253+0000 7eff7eda6700 1 -- 172.21.15.189:0/3005133369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff7813c9b0 msgr2=0x7eff78139b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:36.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.253+0000 7eff7eda6700 1 --2- 172.21.15.189:0/3005133369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff7813c9b0 0x7eff78139b20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7eff6400f7e0 tx=0x7eff6400fba0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.253+0000 7eff7eda6700 1 -- 172.21.15.189:0/3005133369 shutdown_connections 2024-04-06T12:12:36.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.253+0000 7eff7eda6700 1 --2- 172.21.15.189:0/3005133369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff7813a060 0x7eff78138060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.253+0000 7eff7eda6700 1 --2- 172.21.15.189:0/3005133369 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7eff5c06ac90 0x7eff5c06d150 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.253+0000 7eff7eda6700 1 --2- 172.21.15.189:0/3005133369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff7815acb0 0x7eff7813c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.253+0000 7eff7eda6700 1 --2- 172.21.15.189:0/3005133369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff7813c9b0 0x7eff78139b20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.253+0000 7eff7eda6700 1 -- 172.21.15.189:0/3005133369 >> 172.21.15.189:0/3005133369 conn(0x7eff780d00b0 msgr2=0x7eff7806f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:36.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.253+0000 7eff7eda6700 1 -- 172.21.15.189:0/3005133369 shutdown_connections 2024-04-06T12:12:36.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.253+0000 7eff7eda6700 1 -- 172.21.15.189:0/3005133369 wait complete. 2024-04-06T12:12:36.263 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:12:36.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.390+0000 7f5b2a46f700 1 Processor -- start 2024-04-06T12:12:36.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.390+0000 7f5b2a46f700 1 -- start start 2024-04-06T12:12:36.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.391+0000 7f5b2a46f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5b240cb870 0x7f5b2415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.391+0000 7f5b2a46f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b240cfa00 0x7f5b240cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.391+0000 7f5b2a46f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240cbd80 con 0x7f5b240d11b0 2024-04-06T12:12:36.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.391+0000 7f5b2a46f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b240ccdc0 con 0x7f5b240cfa00 2024-04-06T12:12:36.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.391+0000 7f5b2a46f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b240d0b50 con 0x7f5b240cb870 2024-04-06T12:12:36.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.391+0000 7f5b28a0c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b240cfa00 0x7f5b240cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.391+0000 7f5b28a0c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b240cfa00 0x7f5b240cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38554/0 (socket says 172.21.15.189:38554) 2024-04-06T12:12:36.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.391+0000 7f5b28a0c700 1 -- 172.21.15.189:0/3390959508 learned_addr learned my addr 172.21.15.189:0/3390959508 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:36.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.391+0000 7f5b237fe700 1 --2- 172.21.15.189:0/3390959508 >> v2:172.21.15.192:3300/0 conn(0x7f5b240cb870 0x7f5b2415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.391+0000 7f5b28a0c700 1 -- 172.21.15.189:0/3390959508 >> v2:172.21.15.192:3300/0 conn(0x7f5b240cb870 msgr2=0x7f5b2415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:36.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.391+0000 7f5b28a0c700 1 --2- 172.21.15.189:0/3390959508 >> v2:172.21.15.192:3300/0 conn(0x7f5b240cb870 0x7f5b2415c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.391+0000 7f5b28a0c700 1 -- 172.21.15.189:0/3390959508 >> v1:172.21.15.192:6789/0 conn(0x7f5b240d11b0 legacy=0x7f5b240d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:36.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.391+0000 7f5b28a0c700 1 -- 172.21.15.189:0/3390959508 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240cef20 con 0x7f5b240cfa00 2024-04-06T12:12:36.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.391+0000 7f5b28a0c700 1 --2- 172.21.15.189:0/3390959508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b240cfa00 0x7f5b240cd560 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5b2404fb40 tx=0x7f5b1c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=520b1458e29ec77b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:36.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.391+0000 7f5b22ffd700 1 -- 172.21.15.189:0/3390959508 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c00e070 con 0x7f5b240cfa00 2024-04-06T12:12:36.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.391+0000 7f5b22ffd700 1 -- 172.21.15.189:0/3390959508 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1c003a20 con 0x7f5b240cfa00 2024-04-06T12:12:36.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.391+0000 7f5b22ffd700 1 -- 172.21.15.189:0/3390959508 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c003dd0 con 0x7f5b240cfa00 2024-04-06T12:12:36.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.392+0000 7f5b2a46f700 1 -- 172.21.15.189:0/3390959508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b240cfa00 msgr2=0x7f5b240cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:36.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.392+0000 7f5b2a46f700 1 --2- 172.21.15.189:0/3390959508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b240cfa00 0x7f5b240cd560 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5b2404fb40 tx=0x7f5b1c00b330 comp rx=0 tx=0).stop 2024-04-06T12:12:36.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.392+0000 7f5b2a46f700 1 -- 172.21.15.189:0/3390959508 shutdown_connections 2024-04-06T12:12:36.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.392+0000 7f5b2a46f700 1 --2- 172.21.15.189:0/3390959508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b240cfa00 0x7f5b240cd560 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.392+0000 7f5b2a46f700 1 --2- 172.21.15.189:0/3390959508 >> v2:172.21.15.192:3300/0 conn(0x7f5b240cb870 0x7f5b2415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.392+0000 7f5b2a46f700 1 -- 172.21.15.189:0/3390959508 >> 172.21.15.189:0/3390959508 conn(0x7f5b240b2db0 msgr2=0x7f5b240b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:36.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.392+0000 7f5b2a46f700 1 -- 172.21.15.189:0/3390959508 shutdown_connections 2024-04-06T12:12:36.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.392+0000 7f5b2a46f700 1 -- 172.21.15.189:0/3390959508 wait complete. 2024-04-06T12:12:36.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.393+0000 7f5b2a46f700 1 Processor -- start 2024-04-06T12:12:36.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.393+0000 7f5b2a46f700 1 -- start start 2024-04-06T12:12:36.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.394+0000 7f5b2a46f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b240cb870 0x7f5b241f4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.394+0000 7f5b2a46f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b240cfa00 0x7f5b240affe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.394+0000 7f5b2a46f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b240d11b0 0x7f5b24155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.394+0000 7f5b2a46f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b24155cb0 con 0x7f5b240cfa00 2024-04-06T12:12:36.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.394+0000 7f5b2a46f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2406d6e0 con 0x7f5b240cb870 2024-04-06T12:12:36.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.394+0000 7f5b2a46f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b241f3680 con 0x7f5b240d11b0 2024-04-06T12:12:36.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.394+0000 7f5b237fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b240cfa00 0x7f5b240affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.394+0000 7f5b237fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b240cfa00 0x7f5b240affe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38562/0 (socket says 172.21.15.189:38562) 2024-04-06T12:12:36.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.394+0000 7f5b237fe700 1 -- 172.21.15.189:0/1176410400 learned_addr learned my addr 172.21.15.189:0/1176410400 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:36.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.394+0000 7f5b23fff700 1 --2- 172.21.15.189:0/1176410400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b240cb870 0x7f5b241f4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.394+0000 7f5b28a0c700 1 --2- 172.21.15.189:0/1176410400 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b240d11b0 0x7f5b24155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.394+0000 7f5b237fe700 1 -- 172.21.15.189:0/1176410400 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b240d11b0 msgr2=0x7f5b24155770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:36.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.394+0000 7f5b237fe700 1 --2- 172.21.15.189:0/1176410400 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b240d11b0 0x7f5b24155770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.394+0000 7f5b237fe700 1 -- 172.21.15.189:0/1176410400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b240cb870 msgr2=0x7f5b241f4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:36.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.394+0000 7f5b237fe700 1 --2- 172.21.15.189:0/1176410400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b240cb870 0x7f5b241f4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.394+0000 7f5b237fe700 1 -- 172.21.15.189:0/1176410400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c009b30 con 0x7f5b240cfa00 2024-04-06T12:12:36.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.395+0000 7f5b237fe700 1 --2- 172.21.15.189:0/1176410400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b240cfa00 0x7f5b240affe0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800c7f0 tx=0x7f5b1800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:36.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.395+0000 7f5b20ff9700 1 -- 172.21.15.189:0/1176410400 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18011800 con 0x7f5b240cfa00 2024-04-06T12:12:36.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.395+0000 7f5b20ff9700 1 -- 172.21.15.189:0/1176410400 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b18011960 con 0x7f5b240cfa00 2024-04-06T12:12:36.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.395+0000 7f5b2a46f700 1 -- 172.21.15.189:0/1176410400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b241f3960 con 0x7f5b240cfa00 2024-04-06T12:12:36.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.395+0000 7f5b20ff9700 1 -- 172.21.15.189:0/1176410400 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18011d10 con 0x7f5b240cfa00 2024-04-06T12:12:36.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.395+0000 7f5b2a46f700 1 -- 172.21.15.189:0/1176410400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b240afc90 con 0x7f5b240cfa00 2024-04-06T12:12:36.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.396+0000 7f5b2a46f700 1 -- 172.21.15.189:0/1176410400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b241f3e80 con 0x7f5b240cfa00 2024-04-06T12:12:36.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.396+0000 7f5b20ff9700 1 -- 172.21.15.189:0/1176410400 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5b18018360 con 0x7f5b240cfa00 2024-04-06T12:12:36.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.396+0000 7f5b20ff9700 1 --2- 172.21.15.189:0/1176410400 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b1006acc0 0x7f5b1006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.396+0000 7f5b20ff9700 1 -- 172.21.15.189:0/1176410400 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5b18089580 con 0x7f5b240cfa00 2024-04-06T12:12:36.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.397+0000 7f5b23fff700 1 --2- 172.21.15.189:0/1176410400 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b1006acc0 0x7f5b1006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.397+0000 7f5b23fff700 1 --2- 172.21.15.189:0/1176410400 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b1006acc0 0x7f5b1006d180 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5b14001730 tx=0x7f5b14009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:36.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.401+0000 7f5b20ff9700 1 -- 172.21.15.189:0/1176410400 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5b241f3e80 con 0x7f5b240cfa00 2024-04-06T12:12:36.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.595+0000 7f5b2a46f700 1 -- 172.21.15.189:0/1176410400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5b241f3e80 con 0x7f5b240cfa00 2024-04-06T12:12:36.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.595+0000 7f5b20ff9700 1 -- 172.21.15.189:0/1176410400 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5b241f3e80 con 0x7f5b240cfa00 2024-04-06T12:12:36.594 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:12:36.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.598+0000 7f5b2a46f700 1 -- 172.21.15.189:0/1176410400 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b1006acc0 msgr2=0x7f5b1006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:36.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.598+0000 7f5b2a46f700 1 --2- 172.21.15.189:0/1176410400 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b1006acc0 0x7f5b1006d180 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5b14001730 tx=0x7f5b14009040 comp rx=0 tx=0).stop 2024-04-06T12:12:36.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.598+0000 7f5b2a46f700 1 -- 172.21.15.189:0/1176410400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b240cfa00 msgr2=0x7f5b240affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:36.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.598+0000 7f5b2a46f700 1 --2- 172.21.15.189:0/1176410400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b240cfa00 0x7f5b240affe0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800c7f0 tx=0x7f5b1800cbb0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.598+0000 7f5b2a46f700 1 -- 172.21.15.189:0/1176410400 shutdown_connections 2024-04-06T12:12:36.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.598+0000 7f5b2a46f700 1 --2- 172.21.15.189:0/1176410400 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b240d11b0 0x7f5b24155770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.598+0000 7f5b2a46f700 1 --2- 172.21.15.189:0/1176410400 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b1006acc0 0x7f5b1006d180 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.598+0000 7f5b2a46f700 1 --2- 172.21.15.189:0/1176410400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b240cb870 0x7f5b241f4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.598+0000 7f5b2a46f700 1 --2- 172.21.15.189:0/1176410400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b240cfa00 0x7f5b240affe0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.598+0000 7f5b2a46f700 1 -- 172.21.15.189:0/1176410400 >> 172.21.15.189:0/1176410400 conn(0x7f5b240b2db0 msgr2=0x7f5b240b3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:36.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.598+0000 7f5b2a46f700 1 -- 172.21.15.189:0/1176410400 shutdown_connections 2024-04-06T12:12:36.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.598+0000 7f5b2a46f700 1 -- 172.21.15.189:0/1176410400 wait complete. 2024-04-06T12:12:36.608 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-06T12:12:36.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.735+0000 7ff0bd64e700 1 Processor -- start 2024-04-06T12:12:36.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.735+0000 7ff0bd64e700 1 -- start start 2024-04-06T12:12:36.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.736+0000 7ff0bd64e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff0b80d11b0 0x7ff0b80d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.736+0000 7ff0bd64e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff0b80cfa00 0x7ff0b80cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.736+0000 7ff0bd64e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b80d1ad0 con 0x7ff0b80cb870 2024-04-06T12:12:36.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.736+0000 7ff0bd64e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b80ccdc0 con 0x7ff0b80d11b0 2024-04-06T12:12:36.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.736+0000 7ff0bd64e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b80d0b50 con 0x7ff0b80cfa00 2024-04-06T12:12:36.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.736+0000 7ff0b6ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff0b80d11b0 0x7ff0b80d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.736+0000 7ff0b67fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff0b80cb870 0x7ff0b815c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34438/0 (socket says 172.21.15.189:34438) 2024-04-06T12:12:36.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.736+0000 7ff0b67fc700 1 -- 172.21.15.189:0/1298887575 learned_addr learned my addr 172.21.15.189:0/1298887575 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:36.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.737+0000 7ff0b77fe700 1 --2- 172.21.15.189:0/1298887575 >> v2:172.21.15.192:3300/0 conn(0x7ff0b80cfa00 0x7ff0b80cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.737+0000 7ff0b6ffd700 1 -- 172.21.15.189:0/1298887575 >> v2:172.21.15.192:3300/0 conn(0x7ff0b80cfa00 msgr2=0x7ff0b80cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:36.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.737+0000 7ff0b6ffd700 1 --2- 172.21.15.189:0/1298887575 >> v2:172.21.15.192:3300/0 conn(0x7ff0b80cfa00 0x7ff0b80cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.737+0000 7ff0b6ffd700 1 -- 172.21.15.189:0/1298887575 >> v1:172.21.15.189:6789/0 conn(0x7ff0b80cb870 legacy=0x7ff0b815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:36.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.737+0000 7ff0b6ffd700 1 -- 172.21.15.189:0/1298887575 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b80cbd80 con 0x7ff0b80d11b0 2024-04-06T12:12:36.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.737+0000 7ff0b5ffb700 1 -- 172.21.15.189:0/1298887575 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013741334 0 0) 0x7ff0b80d1ad0 con 0x7ff0b80cb870 2024-04-06T12:12:36.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.737+0000 7ff0b77fe700 1 --2- 172.21.15.189:0/1298887575 >> v2:172.21.15.192:3300/0 conn(0x7ff0b80cfa00 0x7ff0b80cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:36.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.737+0000 7ff0b6ffd700 1 --2- 172.21.15.189:0/1298887575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0b80d11b0 0x7ff0b80d1590 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff0b80b0cd0 tx=0x7ff0ac009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6195caa55c00af09 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:36.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.737+0000 7ff0b5ffb700 1 -- 172.21.15.189:0/1298887575 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac011830 con 0x7ff0b80d11b0 2024-04-06T12:12:36.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.737+0000 7ff0b5ffb700 1 -- 172.21.15.189:0/1298887575 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac011990 con 0x7ff0b80d11b0 2024-04-06T12:12:36.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.737+0000 7ff0b5ffb700 1 -- 172.21.15.189:0/1298887575 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac011d20 con 0x7ff0b80d11b0 2024-04-06T12:12:36.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.737+0000 7ff0bd64e700 1 -- 172.21.15.189:0/1298887575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0b80d11b0 msgr2=0x7ff0b80d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:36.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.737+0000 7ff0bd64e700 1 --2- 172.21.15.189:0/1298887575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0b80d11b0 0x7ff0b80d1590 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff0b80b0cd0 tx=0x7ff0ac009b20 comp rx=0 tx=0).stop 2024-04-06T12:12:36.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.738+0000 7ff0bd64e700 1 -- 172.21.15.189:0/1298887575 shutdown_connections 2024-04-06T12:12:36.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.738+0000 7ff0bd64e700 1 --2- 172.21.15.189:0/1298887575 >> v2:172.21.15.192:3300/0 conn(0x7ff0b80cfa00 0x7ff0b80cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.738+0000 7ff0bd64e700 1 --2- 172.21.15.189:0/1298887575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0b80d11b0 0x7ff0b80d1590 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.738+0000 7ff0bd64e700 1 -- 172.21.15.189:0/1298887575 >> 172.21.15.189:0/1298887575 conn(0x7ff0b80b2db0 msgr2=0x7ff0b80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:36.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.738+0000 7ff0bd64e700 1 -- 172.21.15.189:0/1298887575 shutdown_connections 2024-04-06T12:12:36.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.738+0000 7ff0bd64e700 1 -- 172.21.15.189:0/1298887575 wait complete. 2024-04-06T12:12:36.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.739+0000 7ff0bd64e700 1 Processor -- start 2024-04-06T12:12:36.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.739+0000 7ff0bd64e700 1 -- start start 2024-04-06T12:12:36.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0bd64e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0b80cb870 0x7ff0b815f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0bd64e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff0b80cfa00 0x7ff0b8154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0bd64e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff0b80d11b0 0x7ff0b80c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0bd64e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b80c8490 con 0x7ff0b80cb870 2024-04-06T12:12:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0bd64e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b81e7ad0 con 0x7ff0b80cfa00 2024-04-06T12:12:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0bd64e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b81e7230 con 0x7ff0b80d11b0 2024-04-06T12:12:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0b6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0b80cb870 0x7ff0b815f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0b6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0b80cb870 0x7ff0b815f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38580/0 (socket says 172.21.15.189:38580) 2024-04-06T12:12:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0b6ffd700 1 -- 172.21.15.189:0/52668928 learned_addr learned my addr 172.21.15.189:0/52668928 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0b67fc700 1 --2- 172.21.15.189:0/52668928 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff0b80cfa00 0x7ff0b8154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0b77fe700 1 --2- 172.21.15.189:0/52668928 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff0b80d11b0 0x7ff0b80c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0b6ffd700 1 -- 172.21.15.189:0/52668928 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff0b80d11b0 msgr2=0x7ff0b80c6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0b6ffd700 1 --2- 172.21.15.189:0/52668928 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff0b80d11b0 0x7ff0b80c6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0b6ffd700 1 -- 172.21.15.189:0/52668928 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff0b80cfa00 msgr2=0x7ff0b8154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0b6ffd700 1 --2- 172.21.15.189:0/52668928 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff0b80cfa00 0x7ff0b8154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0b6ffd700 1 -- 172.21.15.189:0/52668928 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ac0094a0 con 0x7ff0b80cb870 2024-04-06T12:12:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0b77fe700 1 --2- 172.21.15.189:0/52668928 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff0b80d11b0 0x7ff0b80c6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0b6ffd700 1 --2- 172.21.15.189:0/52668928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0b80cb870 0x7ff0b815f980 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac0090b0 tx=0x7ff0ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff09f7fe700 1 -- 172.21.15.189:0/52668928 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac0038a0 con 0x7ff0b80cb870 2024-04-06T12:12:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff09f7fe700 1 -- 172.21.15.189:0/52668928 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac01f070 con 0x7ff0b80cb870 2024-04-06T12:12:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0bd64e700 1 -- 172.21.15.189:0/52668928 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b81e7510 con 0x7ff0b80cb870 2024-04-06T12:12:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff09f7fe700 1 -- 172.21.15.189:0/52668928 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac003c60 con 0x7ff0b80cb870 2024-04-06T12:12:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.740+0000 7ff0bd64e700 1 -- 172.21.15.189:0/52668928 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b81e6e50 con 0x7ff0b80cb870 2024-04-06T12:12:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.742+0000 7ff0bd64e700 1 -- 172.21.15.189:0/52668928 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b81e6fe0 con 0x7ff0b80cb870 2024-04-06T12:12:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.742+0000 7ff09f7fe700 1 -- 172.21.15.189:0/52668928 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff0ac01a430 con 0x7ff0b80cb870 2024-04-06T12:12:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.742+0000 7ff09f7fe700 1 --2- 172.21.15.189:0/52668928 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff0a406ac90 0x7ff0a406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.742+0000 7ff09f7fe700 1 -- 172.21.15.189:0/52668928 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff0ac021070 con 0x7ff0b80cb870 2024-04-06T12:12:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.743+0000 7ff0b67fc700 1 --2- 172.21.15.189:0/52668928 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff0a406ac90 0x7ff0a406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.743+0000 7ff0b67fc700 1 --2- 172.21.15.189:0/52668928 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff0a406ac90 0x7ff0a406d150 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff0b815a480 tx=0x7ff0a8007e70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:36.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.747+0000 7ff09f7fe700 1 -- 172.21.15.189:0/52668928 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff0ac055f70 con 0x7ff0b80cb870 2024-04-06T12:12:36.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.941+0000 7ff0bd64e700 1 -- 172.21.15.189:0/52668928 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff0b8074990 con 0x7ff0b80cb870 2024-04-06T12:12:36.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.941+0000 7ff09f7fe700 1 -- 172.21.15.189:0/52668928 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff0ac059590 con 0x7ff0b80cb870 2024-04-06T12:12:36.940 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:12:36.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.943+0000 7ff0bd64e700 1 -- 172.21.15.189:0/52668928 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff0a406ac90 msgr2=0x7ff0a406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:36.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.943+0000 7ff0bd64e700 1 --2- 172.21.15.189:0/52668928 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff0a406ac90 0x7ff0a406d150 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff0b815a480 tx=0x7ff0a8007e70 comp rx=0 tx=0).stop 2024-04-06T12:12:36.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.943+0000 7ff0bd64e700 1 -- 172.21.15.189:0/52668928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0b80cb870 msgr2=0x7ff0b815f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:36.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.943+0000 7ff0bd64e700 1 --2- 172.21.15.189:0/52668928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0b80cb870 0x7ff0b815f980 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac0090b0 tx=0x7ff0ac004750 comp rx=0 tx=0).stop 2024-04-06T12:12:36.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.943+0000 7ff0bd64e700 1 -- 172.21.15.189:0/52668928 shutdown_connections 2024-04-06T12:12:36.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.943+0000 7ff0bd64e700 1 --2- 172.21.15.189:0/52668928 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff0b80d11b0 0x7ff0b80c6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.944+0000 7ff0bd64e700 1 --2- 172.21.15.189:0/52668928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0b80cb870 0x7ff0b815f980 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.944+0000 7ff0bd64e700 1 --2- 172.21.15.189:0/52668928 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff0a406ac90 0x7ff0a406d150 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.944+0000 7ff0bd64e700 1 --2- 172.21.15.189:0/52668928 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff0b80cfa00 0x7ff0b8154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:36.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.944+0000 7ff0bd64e700 1 -- 172.21.15.189:0/52668928 >> 172.21.15.189:0/52668928 conn(0x7ff0b80b2db0 msgr2=0x7ff0b80b69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:36.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.944+0000 7ff0bd64e700 1 -- 172.21.15.189:0/52668928 shutdown_connections 2024-04-06T12:12:36.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:36.944+0000 7ff0bd64e700 1 -- 172.21.15.189:0/52668928 wait complete. 2024-04-06T12:12:36.954 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:12:37.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.078+0000 7f6355823700 1 Processor -- start 2024-04-06T12:12:37.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.078+0000 7f6355823700 1 -- start start 2024-04-06T12:12:37.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.078+0000 7f6355823700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6350144f40 0x7f635006a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:37.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.078+0000 7f6355823700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f635015c150 0x7f635006ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:37.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.078+0000 7f6355823700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f635015c740 con 0x7f635015fcd0 2024-04-06T12:12:37.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.078+0000 7f6355823700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63500cfcb0 con 0x7f635015c150 2024-04-06T12:12:37.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.078+0000 7f6355823700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f635015ca20 con 0x7f6350144f40 2024-04-06T12:12:37.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.079+0000 7f634f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f635015c150 0x7f635006ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:37.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.079+0000 7f634e7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f635015fcd0 0x7f635015ba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34448/0 (socket says 172.21.15.189:34448) 2024-04-06T12:12:37.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.079+0000 7f634e7fc700 1 -- 172.21.15.189:0/3865973090 learned_addr learned my addr 172.21.15.189:0/3865973090 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:37.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.079+0000 7f634effd700 1 --2- 172.21.15.189:0/3865973090 >> v2:172.21.15.192:3300/0 conn(0x7f6350144f40 0x7f635006a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:37.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.079+0000 7f634f7fe700 1 -- 172.21.15.189:0/3865973090 >> v2:172.21.15.192:3300/0 conn(0x7f6350144f40 msgr2=0x7f635006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:37.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.079+0000 7f634f7fe700 1 --2- 172.21.15.189:0/3865973090 >> v2:172.21.15.192:3300/0 conn(0x7f6350144f40 0x7f635006a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.079+0000 7f634f7fe700 1 -- 172.21.15.189:0/3865973090 >> v1:172.21.15.189:6789/0 conn(0x7f635015fcd0 legacy=0x7f635015ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:37.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.079+0000 7f634f7fe700 1 -- 172.21.15.189:0/3865973090 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635015f1f0 con 0x7f635015c150 2024-04-06T12:12:37.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.079+0000 7f634dffb700 1 -- 172.21.15.189:0/3865973090 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333960753 0 0) 0x7f635015c740 con 0x7f635015fcd0 2024-04-06T12:12:37.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.079+0000 7f634f7fe700 1 --2- 172.21.15.189:0/3865973090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f635015c150 0x7f635006ac70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6350073470 tx=0x7f634800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3142ab095af6d66 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:37.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.079+0000 7f634dffb700 1 -- 172.21.15.189:0/3865973090 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634800e070 con 0x7f635015c150 2024-04-06T12:12:37.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.079+0000 7f634dffb700 1 -- 172.21.15.189:0/3865973090 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f634800bbd0 con 0x7f635015c150 2024-04-06T12:12:37.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.079+0000 7f634dffb700 1 -- 172.21.15.189:0/3865973090 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348012430 con 0x7f635015c150 2024-04-06T12:12:37.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.080+0000 7f6355823700 1 -- 172.21.15.189:0/3865973090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f635015c150 msgr2=0x7f635006ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:37.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.080+0000 7f6355823700 1 --2- 172.21.15.189:0/3865973090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f635015c150 0x7f635006ac70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6350073470 tx=0x7f634800b330 comp rx=0 tx=0).stop 2024-04-06T12:12:37.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.080+0000 7f6355823700 1 -- 172.21.15.189:0/3865973090 shutdown_connections 2024-04-06T12:12:37.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.080+0000 7f6355823700 1 --2- 172.21.15.189:0/3865973090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f635015c150 0x7f635006ac70 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.080+0000 7f6355823700 1 --2- 172.21.15.189:0/3865973090 >> v2:172.21.15.192:3300/0 conn(0x7f6350144f40 0x7f635006a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.080+0000 7f6355823700 1 -- 172.21.15.189:0/3865973090 >> 172.21.15.189:0/3865973090 conn(0x7f63500d00b0 msgr2=0x7f63500d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:37.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.080+0000 7f6355823700 1 -- 172.21.15.189:0/3865973090 shutdown_connections 2024-04-06T12:12:37.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.080+0000 7f6355823700 1 -- 172.21.15.189:0/3865973090 wait complete. 2024-04-06T12:12:37.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.081+0000 7f6355823700 1 Processor -- start 2024-04-06T12:12:37.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.081+0000 7f6355823700 1 -- start start 2024-04-06T12:12:37.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.082+0000 7f6355823700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6350144f40 0x7f63500ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:37.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.082+0000 7f6355823700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f635015c150 0x7f635012f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:37.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.082+0000 7f6355823700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f635015fcd0 0x7f6350130cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:37.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.082+0000 7f6355823700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635012fc30 con 0x7f635015c150 2024-04-06T12:12:37.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.082+0000 7f6355823700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635012ff10 con 0x7f635015fcd0 2024-04-06T12:12:37.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.082+0000 7f6355823700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63501301f0 con 0x7f6350144f40 2024-04-06T12:12:37.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.082+0000 7f634e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f635015c150 0x7f635012f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:37.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.082+0000 7f634e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f635015c150 0x7f635012f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38604/0 (socket says 172.21.15.189:38604) 2024-04-06T12:12:37.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.082+0000 7f634e7fc700 1 -- 172.21.15.189:0/166599386 learned_addr learned my addr 172.21.15.189:0/166599386 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:37.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.082+0000 7f634f7fe700 1 --2- 172.21.15.189:0/166599386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f635015fcd0 0x7f6350130cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:37.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.082+0000 7f634effd700 1 --2- 172.21.15.189:0/166599386 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6350144f40 0x7f63500ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:37.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.082+0000 7f634e7fc700 1 -- 172.21.15.189:0/166599386 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6350144f40 msgr2=0x7f63500ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:37.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.082+0000 7f634e7fc700 1 --2- 172.21.15.189:0/166599386 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6350144f40 0x7f63500ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.082+0000 7f634e7fc700 1 -- 172.21.15.189:0/166599386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f635015fcd0 msgr2=0x7f6350130cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:37.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.082+0000 7f634e7fc700 1 --2- 172.21.15.189:0/166599386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f635015fcd0 0x7f6350130cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.082+0000 7f634e7fc700 1 -- 172.21.15.189:0/166599386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6348009a70 con 0x7f635015c150 2024-04-06T12:12:37.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.082+0000 7f634f7fe700 1 --2- 172.21.15.189:0/166599386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f635015fcd0 0x7f6350130cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:37.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.082+0000 7f634e7fc700 1 --2- 172.21.15.189:0/166599386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f635015c150 0x7f635012f1e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f634400c430 tx=0x7f634400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:37.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.082+0000 7f6337fff700 1 -- 172.21.15.189:0/166599386 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6344004c80 con 0x7f635015c150 2024-04-06T12:12:37.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.083+0000 7f6337fff700 1 -- 172.21.15.189:0/166599386 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6344004e00 con 0x7f635015c150 2024-04-06T12:12:37.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.083+0000 7f6355823700 1 -- 172.21.15.189:0/166599386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63501304d0 con 0x7f635015c150 2024-04-06T12:12:37.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.083+0000 7f6337fff700 1 -- 172.21.15.189:0/166599386 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6344007310 con 0x7f635015c150 2024-04-06T12:12:37.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.083+0000 7f6355823700 1 -- 172.21.15.189:0/166599386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63501353d0 con 0x7f635015c150 2024-04-06T12:12:37.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.084+0000 7f6355823700 1 -- 172.21.15.189:0/166599386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6350067bb0 con 0x7f635015c150 2024-04-06T12:12:37.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.084+0000 7f6337fff700 1 -- 172.21.15.189:0/166599386 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6344007470 con 0x7f635015c150 2024-04-06T12:12:37.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.084+0000 7f6337fff700 1 --2- 172.21.15.189:0/166599386 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f633c06ac90 0x7f633c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:37.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.085+0000 7f6337fff700 1 -- 172.21.15.189:0/166599386 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f63440892f0 con 0x7f635015c150 2024-04-06T12:12:37.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.085+0000 7f634effd700 1 --2- 172.21.15.189:0/166599386 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f633c06ac90 0x7f633c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:37.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.085+0000 7f634effd700 1 --2- 172.21.15.189:0/166599386 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f633c06ac90 0x7f633c06d150 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6340009930 tx=0x7f6340008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:37.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.089+0000 7f6337fff700 1 -- 172.21.15.189:0/166599386 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6344054150 con 0x7f635015c150 2024-04-06T12:12:37.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.324+0000 7f6355823700 1 -- 172.21.15.189:0/166599386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6350074990 con 0x7f635015c150 2024-04-06T12:12:37.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.325+0000 7f6337fff700 1 -- 172.21.15.189:0/166599386 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4739 (secure 0 0 0) 0x7f6344057770 con 0x7f635015c150 2024-04-06T12:12:37.323 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:37.323 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2769778155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2769778155},{"type":"v1","addr":"172.21.15.189:6835","nonce":2769778155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4233,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3550531296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3550531296},{"type":"v1","addr":"172.21.15.189:6837","nonce":3550531296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3198235148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3198235148},{"type":"v1","addr":"172.21.15.192:6833","nonce":3198235148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:12:04.195581+0000","modified":"2024-04-06T12:12:26.535185+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/516212261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":516212261},{"type":"v1","addr":"172.21.15.192:6835","nonce":516212261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-06T12:12:37.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.327+0000 7f6355823700 1 -- 172.21.15.189:0/166599386 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f633c06ac90 msgr2=0x7f633c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:37.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.327+0000 7f6355823700 1 --2- 172.21.15.189:0/166599386 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f633c06ac90 0x7f633c06d150 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6340009930 tx=0x7f6340008040 comp rx=0 tx=0).stop 2024-04-06T12:12:37.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.327+0000 7f6355823700 1 -- 172.21.15.189:0/166599386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f635015c150 msgr2=0x7f635012f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:37.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.327+0000 7f6355823700 1 --2- 172.21.15.189:0/166599386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f635015c150 0x7f635012f1e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f634400c430 tx=0x7f634400c7f0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.327+0000 7f6355823700 1 -- 172.21.15.189:0/166599386 shutdown_connections 2024-04-06T12:12:37.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.327+0000 7f6355823700 1 --2- 172.21.15.189:0/166599386 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f633c06ac90 0x7f633c06d150 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.327+0000 7f6355823700 1 --2- 172.21.15.189:0/166599386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f635015fcd0 0x7f6350130cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.327+0000 7f6355823700 1 --2- 172.21.15.189:0/166599386 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6350144f40 0x7f63500ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.327+0000 7f6355823700 1 --2- 172.21.15.189:0/166599386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f635015c150 0x7f635012f1e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.327+0000 7f6355823700 1 -- 172.21.15.189:0/166599386 >> 172.21.15.189:0/166599386 conn(0x7f63500d00b0 msgr2=0x7f63500cf3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:37.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.327+0000 7f6355823700 1 -- 172.21.15.189:0/166599386 shutdown_connections 2024-04-06T12:12:37.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.327+0000 7f6355823700 1 -- 172.21.15.189:0/166599386 wait complete. 2024-04-06T12:12:37.326 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 9 2024-04-06T12:12:37.338 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:37.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.462+0000 7f12dedb2700 1 Processor -- start 2024-04-06T12:12:37.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.462+0000 7f12dedb2700 1 -- start start 2024-04-06T12:12:37.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.462+0000 7f12dedb2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f12d80cb870 0x7f12d815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:37.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.462+0000 7f12dedb2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12d80cbd80 con 0x7f12d80cfa00 2024-04-06T12:12:37.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.462+0000 7f12dedb2700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12d80cef20 con 0x7f12d80d11b0 2024-04-06T12:12:37.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.462+0000 7f12dedb2700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12d80ccdc0 con 0x7f12d80cb870 2024-04-06T12:12:37.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.463+0000 7f12d7fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f12d80cb870 0x7f12d815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:37.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.463+0000 7f12dd34f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f12d80cfa00 0x7f12d80cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34450/0 (socket says 172.21.15.189:34450) 2024-04-06T12:12:37.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.463+0000 7f12dd34f700 1 -- 172.21.15.189:0/509399370 learned_addr learned my addr 172.21.15.189:0/509399370 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:37.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.463+0000 7f12d7fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f12d80cb870 0x7f12d815c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38608/0 (socket says 172.21.15.189:38608) 2024-04-06T12:12:37.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.463+0000 7f12d7fff700 1 -- 172.21.15.189:0/509399370 >> v1:172.21.15.192:6789/0 conn(0x7f12d80d11b0 legacy=0x7f12d80d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:37.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.463+0000 7f12d7fff700 1 -- 172.21.15.189:0/509399370 >> v1:172.21.15.189:6789/0 conn(0x7f12d80cfa00 legacy=0x7f12d80cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:37.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.463+0000 7f12d7fff700 1 -- 172.21.15.189:0/509399370 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d80cf200 con 0x7f12d80cb870 2024-04-06T12:12:37.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.463+0000 7f12d77fe700 1 -- 172.21.15.189:0/509399370 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133407509 0 0) 0x7f12d80cbd80 con 0x7f12d80cfa00 2024-04-06T12:12:37.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.463+0000 7f12d7fff700 1 --2- 172.21.15.189:0/509399370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d80cb870 0x7f12d815c280 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009840 tx=0x7f12c4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2597dffd97516a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:37.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.463+0000 7f12d77fe700 1 -- 172.21.15.189:0/509399370 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4011830 con 0x7f12d80cb870 2024-04-06T12:12:37.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.463+0000 7f12d77fe700 1 -- 172.21.15.189:0/509399370 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c4011990 con 0x7f12d80cb870 2024-04-06T12:12:37.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.463+0000 7f12d77fe700 1 -- 172.21.15.189:0/509399370 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4011d20 con 0x7f12d80cb870 2024-04-06T12:12:37.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.464+0000 7f12dedb2700 1 -- 172.21.15.189:0/509399370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d80cb870 msgr2=0x7f12d815c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:37.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.464+0000 7f12dedb2700 1 --2- 172.21.15.189:0/509399370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d80cb870 0x7f12d815c280 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009840 tx=0x7f12c4009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:37.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.464+0000 7f12dedb2700 1 -- 172.21.15.189:0/509399370 shutdown_connections 2024-04-06T12:12:37.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.464+0000 7f12dedb2700 1 --2- 172.21.15.189:0/509399370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d80cb870 0x7f12d815c280 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.464+0000 7f12dedb2700 1 -- 172.21.15.189:0/509399370 >> 172.21.15.189:0/509399370 conn(0x7f12d80b2db0 msgr2=0x7f12d80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:37.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.464+0000 7f12dedb2700 1 -- 172.21.15.189:0/509399370 shutdown_connections 2024-04-06T12:12:37.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.464+0000 7f12dedb2700 1 -- 172.21.15.189:0/509399370 wait complete. 2024-04-06T12:12:37.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.465+0000 7f12dedb2700 1 Processor -- start 2024-04-06T12:12:37.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.465+0000 7f12dedb2700 1 -- start start 2024-04-06T12:12:37.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.466+0000 7f12dedb2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12d80cb870 0x7f12d80cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:37.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.466+0000 7f12dedb2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12d80cfa00 0x7f12d81f03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:37.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.466+0000 7f12dedb2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d80d11b0 0x7f12d81f6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:37.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.466+0000 7f12dedb2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d81f0930 con 0x7f12d80d11b0 2024-04-06T12:12:37.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.466+0000 7f12dedb2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d81f31b0 con 0x7f12d80cb870 2024-04-06T12:12:37.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.466+0000 7f12dedb2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12d81f2ea0 con 0x7f12d80cfa00 2024-04-06T12:12:37.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.466+0000 7f12dd34f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d80d11b0 0x7f12d81f6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:37.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.466+0000 7f12dd34f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d80d11b0 0x7f12d81f6490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38614/0 (socket says 172.21.15.189:38614) 2024-04-06T12:12:37.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.466+0000 7f12dd34f700 1 -- 172.21.15.189:0/728408466 learned_addr learned my addr 172.21.15.189:0/728408466 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:37.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.466+0000 7f12dcb4e700 1 --2- 172.21.15.189:0/728408466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12d80cb870 0x7f12d80cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:37.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.466+0000 7f12d7fff700 1 --2- 172.21.15.189:0/728408466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12d80cfa00 0x7f12d81f03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:37.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.466+0000 7f12dd34f700 1 -- 172.21.15.189:0/728408466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12d80cfa00 msgr2=0x7f12d81f03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:37.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.466+0000 7f12dd34f700 1 --2- 172.21.15.189:0/728408466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12d80cfa00 0x7f12d81f03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.466+0000 7f12dd34f700 1 -- 172.21.15.189:0/728408466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12d80cb870 msgr2=0x7f12d80cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:37.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.466+0000 7f12dd34f700 1 --2- 172.21.15.189:0/728408466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12d80cb870 0x7f12d80cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.466+0000 7f12dd34f700 1 -- 172.21.15.189:0/728408466 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c40094a0 con 0x7f12d80d11b0 2024-04-06T12:12:37.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.467+0000 7f12dd34f700 1 --2- 172.21.15.189:0/728408466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d80d11b0 0x7f12d81f6490 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f12d000cc70 tx=0x7f12d000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:37.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.467+0000 7f12d57fa700 1 -- 172.21.15.189:0/728408466 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d000aac0 con 0x7f12d80d11b0 2024-04-06T12:12:37.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.467+0000 7f12d57fa700 1 -- 172.21.15.189:0/728408466 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d000ac40 con 0x7f12d80d11b0 2024-04-06T12:12:37.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.467+0000 7f12dedb2700 1 -- 172.21.15.189:0/728408466 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d81f69d0 con 0x7f12d80d11b0 2024-04-06T12:12:37.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.467+0000 7f12d57fa700 1 -- 172.21.15.189:0/728408466 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d00070d0 con 0x7f12d80d11b0 2024-04-06T12:12:37.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.467+0000 7f12dedb2700 1 -- 172.21.15.189:0/728408466 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d81f6f50 con 0x7f12d80d11b0 2024-04-06T12:12:37.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.468+0000 7f12dedb2700 1 -- 172.21.15.189:0/728408466 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d81f3360 con 0x7f12d80d11b0 2024-04-06T12:12:37.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.468+0000 7f12d57fa700 1 -- 172.21.15.189:0/728408466 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f12d0007250 con 0x7f12d80d11b0 2024-04-06T12:12:37.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.468+0000 7f12d57fa700 1 --2- 172.21.15.189:0/728408466 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f12bc06ac90 0x7f12bc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:37.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.469+0000 7f12d57fa700 1 -- 172.21.15.189:0/728408466 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f12d0089320 con 0x7f12d80d11b0 2024-04-06T12:12:37.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.469+0000 7f12dcb4e700 1 --2- 172.21.15.189:0/728408466 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f12bc06ac90 0x7f12bc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:37.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.469+0000 7f12dcb4e700 1 --2- 172.21.15.189:0/728408466 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f12bc06ac90 0x7f12bc06d150 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f12d80b6a10 tx=0x7f12cc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:37.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.473+0000 7f12d57fa700 1 -- 172.21.15.189:0/728408466 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f12d81f3360 con 0x7f12d80d11b0 2024-04-06T12:12:37.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.671+0000 7f12dedb2700 1 -- 172.21.15.189:0/728408466 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12d81f3360 con 0x7f12d80d11b0 2024-04-06T12:12:37.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.672+0000 7f12d57fa700 1 -- 172.21.15.189:0/728408466 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f12d81f3360 con 0x7f12d80d11b0 2024-04-06T12:12:37.670 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:37.670 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:08.649170+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:37.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.674+0000 7f12dedb2700 1 -- 172.21.15.189:0/728408466 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f12bc06ac90 msgr2=0x7f12bc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:37.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.674+0000 7f12dedb2700 1 --2- 172.21.15.189:0/728408466 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f12bc06ac90 0x7f12bc06d150 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f12d80b6a10 tx=0x7f12cc009040 comp rx=0 tx=0).stop 2024-04-06T12:12:37.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.674+0000 7f12dedb2700 1 -- 172.21.15.189:0/728408466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d80d11b0 msgr2=0x7f12d81f6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:37.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.674+0000 7f12dedb2700 1 --2- 172.21.15.189:0/728408466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d80d11b0 0x7f12d81f6490 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f12d000cc70 tx=0x7f12d000cf80 comp rx=0 tx=0).stop 2024-04-06T12:12:37.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.674+0000 7f12dedb2700 1 -- 172.21.15.189:0/728408466 shutdown_connections 2024-04-06T12:12:37.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.674+0000 7f12dedb2700 1 --2- 172.21.15.189:0/728408466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d80d11b0 0x7f12d81f6490 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.674+0000 7f12dedb2700 1 --2- 172.21.15.189:0/728408466 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f12bc06ac90 0x7f12bc06d150 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.674+0000 7f12dedb2700 1 --2- 172.21.15.189:0/728408466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12d80cb870 0x7f12d80cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.674+0000 7f12dedb2700 1 --2- 172.21.15.189:0/728408466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12d80cfa00 0x7f12d81f03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.674+0000 7f12dedb2700 1 -- 172.21.15.189:0/728408466 >> 172.21.15.189:0/728408466 conn(0x7f12d80b2db0 msgr2=0x7f12d8159920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:37.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.674+0000 7f12dedb2700 1 -- 172.21.15.189:0/728408466 shutdown_connections 2024-04-06T12:12:37.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.674+0000 7f12dedb2700 1 -- 172.21.15.189:0/728408466 wait complete. 2024-04-06T12:12:37.685 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:12:37.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.809+0000 7fafed637700 1 Processor -- start 2024-04-06T12:12:37.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.809+0000 7fafed637700 1 -- start start 2024-04-06T12:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.810+0000 7fafed637700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fafe80b4660 0x7fafe80b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.810+0000 7fafed637700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fafe80b2fc0 0x7fafe80b3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.810+0000 7fafed637700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe80b5030 con 0x7fafe815a550 2024-04-06T12:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.810+0000 7fafed637700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafe80cdbe0 con 0x7fafe80b4660 2024-04-06T12:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.810+0000 7fafed637700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe815a940 con 0x7fafe80b2fc0 2024-04-06T12:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.811+0000 7fafe77fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fafe80b2fc0 0x7fafe80b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.811+0000 7fafe67fc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fafe80b4660 0x7fafe80b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.811+0000 7fafe67fc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fafe80b4660 0x7fafe80b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:59294/0 (socket says 172.21.15.189:59294) 2024-04-06T12:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.811+0000 7fafe77fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fafe80b2fc0 0x7fafe80b3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59942/0 (socket says 172.21.15.189:59942) 2024-04-06T12:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.811+0000 7fafe6ffd700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fafe815a550 0x7fafe80cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:52626/0 (socket says 172.21.15.189:52626) 2024-04-06T12:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.811+0000 7fafe67fc700 1 -- 172.21.15.189:0/1381511797 learned_addr learned my addr 172.21.15.189:0/1381511797 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.811+0000 7fafe77fe700 1 -- 172.21.15.189:0/1381511797 >> v2:172.21.15.192:3300/0 conn(0x7fafe80b4660 msgr2=0x7fafe80b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.811+0000 7fafe77fe700 1 --2- 172.21.15.189:0/1381511797 >> v2:172.21.15.192:3300/0 conn(0x7fafe80b4660 0x7fafe80b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.811+0000 7fafe77fe700 1 -- 172.21.15.189:0/1381511797 >> v1:172.21.15.192:6789/0 conn(0x7fafe815a550 legacy=0x7fafe80cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.811+0000 7fafe77fe700 1 -- 172.21.15.189:0/1381511797 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe815ac00 con 0x7fafe80b2fc0 2024-04-06T12:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.811+0000 7fafe67fc700 1 --2- 172.21.15.189:0/1381511797 >> v2:172.21.15.192:3300/0 conn(0x7fafe80b4660 0x7fafe80b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.811+0000 7fafe77fe700 1 --2- 172.21.15.189:0/1381511797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fafe80b2fc0 0x7fafe80b3410 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fafe0009d90 tx=0x7fafe000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=445819205653eff3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.811+0000 7fafe5ffb700 1 -- 172.21.15.189:0/1381511797 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe000e070 con 0x7fafe80b2fc0 2024-04-06T12:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.811+0000 7fafe5ffb700 1 -- 172.21.15.189:0/1381511797 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe000bb20 con 0x7fafe80b2fc0 2024-04-06T12:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.811+0000 7fafe5ffb700 1 -- 172.21.15.189:0/1381511797 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe0012430 con 0x7fafe80b2fc0 2024-04-06T12:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.812+0000 7fafed637700 1 -- 172.21.15.189:0/1381511797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fafe80b2fc0 msgr2=0x7fafe80b3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.812+0000 7fafed637700 1 --2- 172.21.15.189:0/1381511797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fafe80b2fc0 0x7fafe80b3410 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fafe0009d90 tx=0x7fafe000b330 comp rx=0 tx=0).stop 2024-04-06T12:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.812+0000 7fafed637700 1 -- 172.21.15.189:0/1381511797 shutdown_connections 2024-04-06T12:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.812+0000 7fafed637700 1 --2- 172.21.15.189:0/1381511797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fafe80b2fc0 0x7fafe80b3410 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.812+0000 7fafed637700 1 --2- 172.21.15.189:0/1381511797 >> v2:172.21.15.192:3300/0 conn(0x7fafe80b4660 0x7fafe80b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.812+0000 7fafed637700 1 -- 172.21.15.189:0/1381511797 >> 172.21.15.189:0/1381511797 conn(0x7fafe80d00b0 msgr2=0x7fafe80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.812+0000 7fafed637700 1 -- 172.21.15.189:0/1381511797 shutdown_connections 2024-04-06T12:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.812+0000 7fafed637700 1 -- 172.21.15.189:0/1381511797 wait complete. 2024-04-06T12:12:37.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.813+0000 7fafed637700 1 Processor -- start 2024-04-06T12:12:37.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.813+0000 7fafed637700 1 -- start start 2024-04-06T12:12:37.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.814+0000 7fafed637700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafe80b2fc0 0x7fafe814bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:37.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.814+0000 7fafed637700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fafe80b4660 0x7fafe81f1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:37.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.814+0000 7fafed637700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe81588c0 0x7fafe8158d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:37.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.814+0000 7fafed637700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe80a9380 con 0x7fafe81588c0 2024-04-06T12:12:37.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.814+0000 7fafed637700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe81592b0 con 0x7fafe80b2fc0 2024-04-06T12:12:37.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.814+0000 7fafed637700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe80a84b0 con 0x7fafe80b4660 2024-04-06T12:12:37.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.814+0000 7fafe77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe81588c0 0x7fafe8158d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:37.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.814+0000 7fafe6ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafe80b2fc0 0x7fafe814bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:37.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.814+0000 7fafe77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe81588c0 0x7fafe8158d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38628/0 (socket says 172.21.15.189:38628) 2024-04-06T12:12:37.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.814+0000 7fafe77fe700 1 -- 172.21.15.189:0/379601654 learned_addr learned my addr 172.21.15.189:0/379601654 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:37.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.814+0000 7fafe67fc700 1 --2- 172.21.15.189:0/379601654 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fafe80b4660 0x7fafe81f1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:37.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.814+0000 7fafe77fe700 1 -- 172.21.15.189:0/379601654 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fafe80b4660 msgr2=0x7fafe81f1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:37.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.814+0000 7fafe77fe700 1 --2- 172.21.15.189:0/379601654 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fafe80b4660 0x7fafe81f1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.814+0000 7fafe77fe700 1 -- 172.21.15.189:0/379601654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafe80b2fc0 msgr2=0x7fafe814bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:37.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.814+0000 7fafe77fe700 1 --2- 172.21.15.189:0/379601654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafe80b2fc0 0x7fafe814bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:37.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.814+0000 7fafe77fe700 1 -- 172.21.15.189:0/379601654 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd8009550 con 0x7fafe81588c0 2024-04-06T12:12:37.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.814+0000 7fafe6ffd700 1 --2- 172.21.15.189:0/379601654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafe80b2fc0 0x7fafe814bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:37.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.814+0000 7fafe77fe700 1 --2- 172.21.15.189:0/379601654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe81588c0 0x7fafe8158d70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fafe0014040 tx=0x7fafe0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:37.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.815+0000 7fafcf7fe700 1 -- 172.21.15.189:0/379601654 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe0012430 con 0x7fafe81588c0 2024-04-06T12:12:37.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.815+0000 7fafcf7fe700 1 -- 172.21.15.189:0/379601654 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe0020070 con 0x7fafe81588c0 2024-04-06T12:12:37.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.815+0000 7fafed637700 1 -- 172.21.15.189:0/379601654 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe0009a70 con 0x7fafe81588c0 2024-04-06T12:12:37.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.815+0000 7fafcf7fe700 1 -- 172.21.15.189:0/379601654 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe0003ea0 con 0x7fafe81588c0 2024-04-06T12:12:37.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.815+0000 7fafed637700 1 -- 172.21.15.189:0/379601654 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe80a89e0 con 0x7fafe81588c0 2024-04-06T12:12:37.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.816+0000 7fafed637700 1 -- 172.21.15.189:0/379601654 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe8067bb0 con 0x7fafe81588c0 2024-04-06T12:12:37.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.816+0000 7fafcf7fe700 1 -- 172.21.15.189:0/379601654 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fafe0004040 con 0x7fafe81588c0 2024-04-06T12:12:37.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.816+0000 7fafcf7fe700 1 --2- 172.21.15.189:0/379601654 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fafd406ac90 0x7fafd406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:37.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.817+0000 7fafe6ffd700 1 --2- 172.21.15.189:0/379601654 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fafd406ac90 0x7fafd406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:37.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.817+0000 7fafcf7fe700 1 -- 172.21.15.189:0/379601654 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fafe0022070 con 0x7fafe81588c0 2024-04-06T12:12:37.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.817+0000 7fafe6ffd700 1 --2- 172.21.15.189:0/379601654 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fafd406ac90 0x7fafd406d150 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fafdc000f60 tx=0x7fafdc007ed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:37.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:37.821+0000 7fafcf7fe700 1 -- 172.21.15.189:0/379601654 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fafe00562f0 con 0x7fafe81588c0 2024-04-06T12:12:38.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.059+0000 7fafed637700 1 -- 172.21.15.189:0/379601654 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafe8074990 con 0x7fafe81588c0 2024-04-06T12:12:38.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.059+0000 7fafcf7fe700 1 -- 172.21.15.189:0/379601654 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4739 (secure 0 0 0) 0x7fafe0059910 con 0x7fafe81588c0 2024-04-06T12:12:38.057 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:38.057 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2769778155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2769778155},{"type":"v1","addr":"172.21.15.189:6835","nonce":2769778155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4233,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3550531296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3550531296},{"type":"v1","addr":"172.21.15.189:6837","nonce":3550531296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3198235148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3198235148},{"type":"v1","addr":"172.21.15.192:6833","nonce":3198235148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:12:04.195581+0000","modified":"2024-04-06T12:12:26.535185+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/516212261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":516212261},{"type":"v1","addr":"172.21.15.192:6835","nonce":516212261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-06T12:12:38.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.061+0000 7fafed637700 1 -- 172.21.15.189:0/379601654 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fafd406ac90 msgr2=0x7fafd406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:38.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.061+0000 7fafed637700 1 --2- 172.21.15.189:0/379601654 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fafd406ac90 0x7fafd406d150 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fafdc000f60 tx=0x7fafdc007ed0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.061+0000 7fafed637700 1 -- 172.21.15.189:0/379601654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe81588c0 msgr2=0x7fafe8158d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:38.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.061+0000 7fafed637700 1 --2- 172.21.15.189:0/379601654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe81588c0 0x7fafe8158d70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fafe0014040 tx=0x7fafe0004ad0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.062+0000 7fafed637700 1 -- 172.21.15.189:0/379601654 shutdown_connections 2024-04-06T12:12:38.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.062+0000 7fafed637700 1 --2- 172.21.15.189:0/379601654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe81588c0 0x7fafe8158d70 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.062+0000 7fafed637700 1 --2- 172.21.15.189:0/379601654 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fafd406ac90 0x7fafd406d150 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.062+0000 7fafed637700 1 --2- 172.21.15.189:0/379601654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafe80b2fc0 0x7fafe814bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.062+0000 7fafed637700 1 --2- 172.21.15.189:0/379601654 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fafe80b4660 0x7fafe81f1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.062+0000 7fafed637700 1 -- 172.21.15.189:0/379601654 >> 172.21.15.189:0/379601654 conn(0x7fafe80d00b0 msgr2=0x7fafe806eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:38.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.062+0000 7fafed637700 1 -- 172.21.15.189:0/379601654 shutdown_connections 2024-04-06T12:12:38.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.062+0000 7fafed637700 1 -- 172.21.15.189:0/379601654 wait complete. 2024-04-06T12:12:38.061 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 9 2024-04-06T12:12:38.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-06T12:12:04.195581+0000', 'modified': '2024-04-06T12:12:26.535185+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4270': {'gid': 4270, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.192:6835/516212261', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6834', 'nonce': 516212261}, {'type': 'v1', 'addr': '172.21.15.192:6835', 'nonce': 516212261}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-06T12:12:38.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-06T12:12:38.475 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-06T12:12:38.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.603+0000 7ff3677f6700 1 Processor -- start 2024-04-06T12:12:38.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.603+0000 7ff3677f6700 1 -- start start 2024-04-06T12:12:38.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.604+0000 7ff3677f6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff36006ae40 0x7ff3600b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:38.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.604+0000 7ff3677f6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3600b3990 con 0x7ff36015ddf0 2024-04-06T12:12:38.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.604+0000 7ff3677f6700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3600b3060 con 0x7ff36015d7b0 2024-04-06T12:12:38.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.604+0000 7ff3677f6700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff36015e950 con 0x7ff36006ae40 2024-04-06T12:12:38.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.604+0000 7ff364d91700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff36006ae40 0x7ff3600b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:38.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.604+0000 7ff365d93700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff36015ddf0 0x7ff36015e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34466/0 (socket says 172.21.15.189:34466) 2024-04-06T12:12:38.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.604+0000 7ff365d93700 1 -- 172.21.15.189:0/171581221 learned_addr learned my addr 172.21.15.189:0/171581221 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:38.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.604+0000 7ff364d91700 1 -- 172.21.15.189:0/171581221 >> v1:172.21.15.192:6789/0 conn(0x7ff36015d7b0 legacy=0x7ff36006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:38.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.604+0000 7ff364d91700 1 -- 172.21.15.189:0/171581221 >> v1:172.21.15.189:6789/0 conn(0x7ff36015ddf0 legacy=0x7ff36015e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:38.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.604+0000 7ff357fff700 1 -- 172.21.15.189:0/171581221 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528602139 0 0) 0x7ff3600b3990 con 0x7ff36015ddf0 2024-04-06T12:12:38.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.605+0000 7ff364d91700 1 -- 172.21.15.189:0/171581221 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3600b3340 con 0x7ff36006ae40 2024-04-06T12:12:38.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.605+0000 7ff364d91700 1 --2- 172.21.15.189:0/171581221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff36006ae40 0x7ff3600b3e20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff35c009840 tx=0x7ff35c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae50eda3cd0da5f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:38.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.605+0000 7ff357fff700 1 -- 172.21.15.189:0/171581221 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c004230 con 0x7ff36006ae40 2024-04-06T12:12:38.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.605+0000 7ff357fff700 1 -- 172.21.15.189:0/171581221 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35c004390 con 0x7ff36006ae40 2024-04-06T12:12:38.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.605+0000 7ff357fff700 1 -- 172.21.15.189:0/171581221 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c0036a0 con 0x7ff36006ae40 2024-04-06T12:12:38.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.605+0000 7ff3677f6700 1 -- 172.21.15.189:0/171581221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff36006ae40 msgr2=0x7ff3600b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:38.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.605+0000 7ff3677f6700 1 --2- 172.21.15.189:0/171581221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff36006ae40 0x7ff3600b3e20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff35c009840 tx=0x7ff35c009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:38.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.605+0000 7ff3677f6700 1 -- 172.21.15.189:0/171581221 shutdown_connections 2024-04-06T12:12:38.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.605+0000 7ff3677f6700 1 --2- 172.21.15.189:0/171581221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff36006ae40 0x7ff3600b3e20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.605+0000 7ff3677f6700 1 -- 172.21.15.189:0/171581221 >> 172.21.15.189:0/171581221 conn(0x7ff3600d00b0 msgr2=0x7ff3600d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:38.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.605+0000 7ff3677f6700 1 -- 172.21.15.189:0/171581221 shutdown_connections 2024-04-06T12:12:38.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.605+0000 7ff3677f6700 1 -- 172.21.15.189:0/171581221 wait complete. 2024-04-06T12:12:38.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.607+0000 7ff3677f6700 1 Processor -- start 2024-04-06T12:12:38.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.607+0000 7ff3677f6700 1 -- start start 2024-04-06T12:12:38.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.607+0000 7ff3677f6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff36006ae40 0x7ff3600cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:38.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.607+0000 7ff3677f6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff36015d7b0 0x7ff3600b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:38.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.607+0000 7ff3677f6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff36015ddf0 0x7ff360153230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:38.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.607+0000 7ff3677f6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3600ced20 con 0x7ff36015ddf0 2024-04-06T12:12:38.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.607+0000 7ff3677f6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3600b1a90 con 0x7ff36006ae40 2024-04-06T12:12:38.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.607+0000 7ff3677f6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3600cf1b0 con 0x7ff36015d7b0 2024-04-06T12:12:38.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.607+0000 7ff365d93700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff36015ddf0 0x7ff360153230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:38.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.608+0000 7ff365d93700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff36015ddf0 0x7ff360153230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38644/0 (socket says 172.21.15.189:38644) 2024-04-06T12:12:38.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.608+0000 7ff365d93700 1 -- 172.21.15.189:0/82554428 learned_addr learned my addr 172.21.15.189:0/82554428 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:38.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.608+0000 7ff365592700 1 --2- 172.21.15.189:0/82554428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff36006ae40 0x7ff3600cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:38.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.608+0000 7ff365d93700 1 -- 172.21.15.189:0/82554428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff36015d7b0 msgr2=0x7ff3600b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:38.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.608+0000 7ff365d93700 1 --2- 172.21.15.189:0/82554428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff36015d7b0 0x7ff3600b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.608+0000 7ff365d93700 1 -- 172.21.15.189:0/82554428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff36006ae40 msgr2=0x7ff3600cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:38.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.608+0000 7ff365d93700 1 --2- 172.21.15.189:0/82554428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff36006ae40 0x7ff3600cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.608+0000 7ff365d93700 1 -- 172.21.15.189:0/82554428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c0094a0 con 0x7ff36015ddf0 2024-04-06T12:12:38.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.608+0000 7ff365592700 1 --2- 172.21.15.189:0/82554428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff36006ae40 0x7ff3600cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:38.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.608+0000 7ff365d93700 1 --2- 172.21.15.189:0/82554428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff36015ddf0 0x7ff360153230 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff35800cb90 tx=0x7ff35800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:38.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.608+0000 7ff355ffb700 1 -- 172.21.15.189:0/82554428 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35800ab30 con 0x7ff36015ddf0 2024-04-06T12:12:38.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.608+0000 7ff355ffb700 1 -- 172.21.15.189:0/82554428 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35800ac90 con 0x7ff36015ddf0 2024-04-06T12:12:38.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.608+0000 7ff355ffb700 1 -- 172.21.15.189:0/82554428 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3580070d0 con 0x7ff36015ddf0 2024-04-06T12:12:38.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.608+0000 7ff3677f6700 1 -- 172.21.15.189:0/82554428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3600ce110 con 0x7ff36015ddf0 2024-04-06T12:12:38.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.608+0000 7ff3677f6700 1 -- 172.21.15.189:0/82554428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3600ce6d0 con 0x7ff36015ddf0 2024-04-06T12:12:38.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.610+0000 7ff355ffb700 1 -- 172.21.15.189:0/82554428 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff358007230 con 0x7ff36015ddf0 2024-04-06T12:12:38.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.610+0000 7ff355ffb700 1 --2- 172.21.15.189:0/82554428 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff34406ac90 0x7ff34406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:38.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.610+0000 7ff355ffb700 1 -- 172.21.15.189:0/82554428 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff358089190 con 0x7ff36015ddf0 2024-04-06T12:12:38.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.610+0000 7ff365592700 1 --2- 172.21.15.189:0/82554428 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff34406ac90 0x7ff34406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:38.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.611+0000 7ff365592700 1 --2- 172.21.15.189:0/82554428 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff34406ac90 0x7ff34406d150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff350003f40 tx=0x7ff350009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:38.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.612+0000 7ff3739aab80 1 Processor -- start 2024-04-06T12:12:38.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.612+0000 7ff3739aab80 1 -- start start 2024-04-06T12:12:38.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.613+0000 7ff3739aab80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c30340c20 0x561c30344a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:38.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.613+0000 7ff3739aab80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c30345fc0 0x561c3031c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:38.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.613+0000 7ff3739aab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c3031b980 0x561c3031cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:38.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.613+0000 7ff3739aab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c30275940 con 0x561c3031b980 2024-04-06T12:12:38.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.613+0000 7ff3739aab80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c30273300 con 0x561c30340c20 2024-04-06T12:12:38.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.613+0000 7ff3739aab80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x561c3026ebf0 con 0x561c30345fc0 2024-04-06T12:12:38.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.613+0000 7ff365d93700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c3031b980 0x561c3031cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:38.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.613+0000 7ff365d93700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c3031b980 0x561c3031cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38656/0 (socket says 172.21.15.189:38656) 2024-04-06T12:12:38.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.613+0000 7ff365d93700 1 -- 172.21.15.189:0/1332939475 learned_addr learned my addr 172.21.15.189:0/1332939475 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:38.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.613+0000 7ff365592700 1 --2- 172.21.15.189:0/1332939475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c30340c20 0x561c30344a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:38.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.613+0000 7ff364d91700 1 --2- 172.21.15.189:0/1332939475 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c30345fc0 0x561c3031c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:38.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.614+0000 7ff365d93700 1 -- 172.21.15.189:0/1332939475 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c30345fc0 msgr2=0x561c3031c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:38.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.614+0000 7ff365d93700 1 --2- 172.21.15.189:0/1332939475 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c30345fc0 0x561c3031c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.614+0000 7ff365d93700 1 -- 172.21.15.189:0/1332939475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c30340c20 msgr2=0x561c30344a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:38.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.614+0000 7ff365d93700 1 --2- 172.21.15.189:0/1332939475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c30340c20 0x561c30344a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.614+0000 7ff365d93700 1 -- 172.21.15.189:0/1332939475 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c3009cb10 con 0x561c3031b980 2024-04-06T12:12:38.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.614+0000 7ff365592700 1 --2- 172.21.15.189:0/1332939475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c30340c20 0x561c30344a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:38.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.614+0000 7ff364d91700 1 --2- 172.21.15.189:0/1332939475 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c30345fc0 0x561c3031c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:38.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.614+0000 7ff365d93700 1 --2- 172.21.15.189:0/1332939475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c3031b980 0x561c3031cb30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff35805fbf0 tx=0x7ff358057760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d92aa660997ae8a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:38.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.614+0000 7ff342ffd700 1 -- 172.21.15.189:0/1332939475 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35805bb60 con 0x561c3031b980 2024-04-06T12:12:38.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.614+0000 7ff342ffd700 1 -- 172.21.15.189:0/1332939475 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff358060120 con 0x561c3031b980 2024-04-06T12:12:38.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.614+0000 7ff342ffd700 1 -- 172.21.15.189:0/1332939475 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff358057220 con 0x561c3031b980 2024-04-06T12:12:38.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.614+0000 7ff3739aab80 1 -- 172.21.15.189:0/1332939475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c3031b980 msgr2=0x561c3031cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:38.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.614+0000 7ff3739aab80 1 --2- 172.21.15.189:0/1332939475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c3031b980 0x561c3031cb30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff35805fbf0 tx=0x7ff358057760 comp rx=0 tx=0).stop 2024-04-06T12:12:38.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.614+0000 7ff3739aab80 1 -- 172.21.15.189:0/1332939475 shutdown_connections 2024-04-06T12:12:38.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.614+0000 7ff3739aab80 1 --2- 172.21.15.189:0/1332939475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c3031b980 0x561c3031cb30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.614+0000 7ff3739aab80 1 --2- 172.21.15.189:0/1332939475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c30340c20 0x561c30344a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.614+0000 7ff3739aab80 1 --2- 172.21.15.189:0/1332939475 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c30345fc0 0x561c3031c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.615+0000 7ff3739aab80 1 -- 172.21.15.189:0/1332939475 >> 172.21.15.189:0/1332939475 conn(0x561c3027c650 msgr2=0x561c303633e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:38.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.615+0000 7ff3739aab80 1 -- 172.21.15.189:0/1332939475 shutdown_connections 2024-04-06T12:12:38.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.615+0000 7ff3739aab80 1 -- 172.21.15.189:0/1332939475 wait complete. 2024-04-06T12:12:38.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.615+0000 7ff3739aab80 1 -- start start 2024-04-06T12:12:38.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.615+0000 7ff3739aab80 1 Processor -- start 2024-04-06T12:12:38.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff3739aab80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c3031b980 0x561c30343dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:38.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff3739aab80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c30340c20 0x561c30341000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:38.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff3739aab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c30345fc0 0x561c30341540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:38.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff3739aab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c3020ad30 con 0x561c30345fc0 2024-04-06T12:12:38.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff3739aab80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c3020a9d0 con 0x561c30340c20 2024-04-06T12:12:38.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff3739aab80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x561c300f2650 con 0x561c3031b980 2024-04-06T12:12:38.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff3739aab80 10 client.0 _ll_register_callbacks cb 0x561c30214420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:12:38.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff365d93700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c30345fc0 0x561c30341540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:38.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff365d93700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c30345fc0 0x561c30341540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38672/0 (socket says 172.21.15.189:38672) 2024-04-06T12:12:38.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff365d93700 1 -- 172.21.15.189:0/1874619206 learned_addr learned my addr 172.21.15.189:0/1874619206 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:38.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff364d91700 1 --2- 172.21.15.189:0/1874619206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c30340c20 0x561c30341000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:38.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff365592700 1 --2- 172.21.15.189:0/1874619206 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c3031b980 0x561c30343dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:38.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff365d93700 1 -- 172.21.15.189:0/1874619206 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c3031b980 msgr2=0x561c30343dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:38.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff365d93700 1 --2- 172.21.15.189:0/1874619206 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c3031b980 0x561c30343dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff365d93700 1 -- 172.21.15.189:0/1874619206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c30340c20 msgr2=0x561c30341000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:38.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff365d93700 1 --2- 172.21.15.189:0/1874619206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c30340c20 0x561c30341000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff365d93700 1 -- 172.21.15.189:0/1874619206 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3600ce6d0 con 0x561c30345fc0 2024-04-06T12:12:38.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff365592700 1 --2- 172.21.15.189:0/1874619206 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c3031b980 0x561c30343dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:38.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff365d93700 1 --2- 172.21.15.189:0/1874619206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c30345fc0 0x561c30341540 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff35805fbf0 tx=0x7ff35801e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2af263e0f32db164 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:38.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff33bfff700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:12:38.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff33bfff700 1 -- 172.21.15.189:0/1874619206 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35805fe90 con 0x561c30345fc0 2024-04-06T12:12:38.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff33bfff700 1 -- 172.21.15.189:0/1874619206 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff35805bb60 con 0x561c30345fc0 2024-04-06T12:12:38.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff3739aab80 1 -- 172.21.15.189:0/1874619206 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561c3022f330 con 0x561c30345fc0 2024-04-06T12:12:38.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff33bfff700 1 -- 172.21.15.189:0/1874619206 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff358087120 con 0x561c30345fc0 2024-04-06T12:12:38.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff3739aab80 10 client.4791 fetch_fsmap learned FSMap version 9 2024-04-06T12:12:38.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.617+0000 7ff3739aab80 1 -- 172.21.15.189:0/1874619206 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x561c30234e50 con 0x561c30345fc0 2024-04-06T12:12:38.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.618+0000 7ff33bfff700 1 -- 172.21.15.189:0/1874619206 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff358074020 con 0x561c30345fc0 2024-04-06T12:12:38.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.618+0000 7ff33bfff700 1 -- 172.21.15.189:0/1874619206 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 9) v1 ==== 3122+0+0 (secure 0 0 0) 0x7ff35806bb70 con 0x561c30345fc0 2024-04-06T12:12:38.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.618+0000 7ff3739aab80 10 client.4791 fetch_fsmap finished waiting for FSMap version 9 2024-04-06T12:12:38.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.618+0000 7ff3739aab80 10 client.4791 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-06T12:12:38.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.618+0000 7ff3739aab80 20 client.4791 populate_metadata read hostname 'smithi189' 2024-04-06T12:12:38.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.618+0000 7ff3739aab80 1 --2- 172.21.15.189:0/1874619206 >> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] conn(0x561c30346d20 0x561c30347100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:38.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.618+0000 7ff3739aab80 4 client.4791 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-06T12:12:38.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.618+0000 7ff3739aab80 1 -- 172.21.15.189:0/1874619206 --> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561c302676b0 con 0x561c30346d20 2024-04-06T12:12:38.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.618+0000 7ff365592700 1 --2- 172.21.15.189:0/1874619206 >> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] conn(0x561c30346d20 0x561c30347100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:38.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.618+0000 7ff33bfff700 10 client.4791 ms_handle_connect on v2:172.21.15.192:6834/516212261 2024-04-06T12:12:38.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.618+0000 7ff365592700 1 --2- 172.21.15.189:0/1874619206 >> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] conn(0x561c30346d20 0x561c30347100 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d59d5c1f45a6fd6 server_cookie=3b34eb18319a3e9d in_seq=0 out_seq=0 2024-04-06T12:12:38.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.619+0000 7ff33bfff700 1 -- 172.21.15.189:0/1874619206 <== mds.0 v2:172.21.15.192:6834/516212261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561c302676b0 con 0x561c30346d20 2024-04-06T12:12:38.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.619+0000 7ff33bfff700 10 client.4791 handle_command_reply: tid=0 2024-04-06T12:12:38.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.619+0000 7ff3739aab80 1 client.4791 shutdown 2024-04-06T12:12:38.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.619+0000 7ff3739aab80 1 -- 172.21.15.189:0/1874619206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c30345fc0 msgr2=0x561c30341540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:38.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.619+0000 7ff3739aab80 1 --2- 172.21.15.189:0/1874619206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c30345fc0 0x561c30341540 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff35805fbf0 tx=0x7ff35801e040 comp rx=0 tx=0).stop 2024-04-06T12:12:38.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.619+0000 7ff3739aab80 1 -- 172.21.15.189:0/1874619206 shutdown_connections 2024-04-06T12:12:38.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.619+0000 7ff3739aab80 1 --2- 172.21.15.189:0/1874619206 >> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] conn(0x561c30346d20 0x561c30347100 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.619+0000 7ff3739aab80 1 --2- 172.21.15.189:0/1874619206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c30345fc0 0x561c30341540 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.619+0000 7ff3739aab80 1 --2- 172.21.15.189:0/1874619206 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c3031b980 0x561c30343dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.619+0000 7ff3739aab80 1 --2- 172.21.15.189:0/1874619206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c30340c20 0x561c30341000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.619+0000 7ff3739aab80 1 -- 172.21.15.189:0/1874619206 >> 172.21.15.189:0/1874619206 conn(0x561c3027c650 msgr2=0x561c30363370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:38.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.619+0000 7ff33bfff700 0 client.4791 ms_handle_reset on v2:172.21.15.192:6834/516212261 2024-04-06T12:12:38.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.619+0000 7ff3739aab80 1 -- 172.21.15.189:0/1874619206 shutdown_connections 2024-04-06T12:12:38.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.619+0000 7ff3739aab80 1 -- 172.21.15.189:0/1874619206 wait complete. 2024-04-06T12:12:38.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.619+0000 7ff3739aab80 20 client.4791 trim_cache size 0 max 16384 2024-04-06T12:12:38.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.637+0000 7ff3739aab80 1 Processor -- start 2024-04-06T12:12:38.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.637+0000 7ff3739aab80 1 -- start start 2024-04-06T12:12:38.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.638+0000 7ff3739aab80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c3031b980 0x561c3031cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:38.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.638+0000 7ff3739aab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c30340c20 0x561c3031a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:38.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.638+0000 7ff3739aab80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c30345fc0 0x561c3031ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:38.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.638+0000 7ff3739aab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c3034b910 con 0x561c30340c20 2024-04-06T12:12:38.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.638+0000 7ff3739aab80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c30344390 con 0x561c3031b980 2024-04-06T12:12:38.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.638+0000 7ff3739aab80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x561c3035ced0 con 0x561c30345fc0 2024-04-06T12:12:38.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.639+0000 7ff364d91700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c30340c20 0x561c3031a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:38.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.639+0000 7ff364d91700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c30340c20 0x561c3031a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38684/0 (socket says 172.21.15.189:38684) 2024-04-06T12:12:38.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.639+0000 7ff364d91700 1 -- 172.21.15.189:0/3931975618 learned_addr learned my addr 172.21.15.189:0/3931975618 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:38.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.639+0000 7ff365d93700 1 --2- 172.21.15.189:0/3931975618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c30345fc0 0x561c3031ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:38.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.639+0000 7ff365592700 1 --2- 172.21.15.189:0/3931975618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c3031b980 0x561c3031cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:38.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.639+0000 7ff364d91700 1 -- 172.21.15.189:0/3931975618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c30345fc0 msgr2=0x561c3031ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:38.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.639+0000 7ff364d91700 1 --2- 172.21.15.189:0/3931975618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c30345fc0 0x561c3031ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.639+0000 7ff364d91700 1 -- 172.21.15.189:0/3931975618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c3031b980 msgr2=0x561c3031cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:38.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.639+0000 7ff364d91700 1 --2- 172.21.15.189:0/3931975618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c3031b980 0x561c3031cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.639+0000 7ff364d91700 1 -- 172.21.15.189:0/3931975618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c3034c750 con 0x561c30340c20 2024-04-06T12:12:38.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.639+0000 7ff365592700 1 --2- 172.21.15.189:0/3931975618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c3031b980 0x561c3031cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:38.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.639+0000 7ff365d93700 1 --2- 172.21.15.189:0/3931975618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c30345fc0 0x561c3031ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:38.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.639+0000 7ff364d91700 1 --2- 172.21.15.189:0/3931975618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c30340c20 0x561c3031a620 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff35c004000 tx=0x7ff35c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a4c91b7c8f06a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:38.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.639+0000 7ff3437fe700 1 -- 172.21.15.189:0/3931975618 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c004230 con 0x561c30340c20 2024-04-06T12:12:38.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.639+0000 7ff3437fe700 1 -- 172.21.15.189:0/3931975618 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35c003850 con 0x561c30340c20 2024-04-06T12:12:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.639+0000 7ff3437fe700 1 -- 172.21.15.189:0/3931975618 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c003c00 con 0x561c30340c20 2024-04-06T12:12:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.640+0000 7ff3739aab80 1 -- 172.21.15.189:0/3931975618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c30340c20 msgr2=0x561c3031a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.640+0000 7ff3739aab80 1 --2- 172.21.15.189:0/3931975618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c30340c20 0x561c3031a620 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff35c004000 tx=0x7ff35c010000 comp rx=0 tx=0).stop 2024-04-06T12:12:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.640+0000 7ff3739aab80 1 -- 172.21.15.189:0/3931975618 shutdown_connections 2024-04-06T12:12:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.640+0000 7ff3739aab80 1 --2- 172.21.15.189:0/3931975618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c30345fc0 0x561c3031ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.640+0000 7ff3739aab80 1 --2- 172.21.15.189:0/3931975618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c3031b980 0x561c3031cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.640+0000 7ff3739aab80 1 --2- 172.21.15.189:0/3931975618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c30340c20 0x561c3031a620 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.640+0000 7ff3739aab80 1 -- 172.21.15.189:0/3931975618 >> 172.21.15.189:0/3931975618 conn(0x561c3027c650 msgr2=0x561c30386450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.640+0000 7ff3739aab80 1 -- 172.21.15.189:0/3931975618 shutdown_connections 2024-04-06T12:12:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.640+0000 7ff3739aab80 1 -- 172.21.15.189:0/3931975618 wait complete. 2024-04-06T12:12:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.640+0000 7ff3739aab80 1 -- start start 2024-04-06T12:12:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.640+0000 7ff3739aab80 1 Processor -- start 2024-04-06T12:12:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.641+0000 7ff3739aab80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c3031b980 0x561c303173a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.641+0000 7ff3739aab80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c30340c20 0x561c3031a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.641+0000 7ff3739aab80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c30345fc0 0x561c3031abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.641+0000 7ff3739aab80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c30319c50 con 0x561c3031b980 2024-04-06T12:12:38.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.641+0000 7ff3739aab80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c303ca070 con 0x561c30340c20 2024-04-06T12:12:38.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.641+0000 7ff3739aab80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x561c30319a00 con 0x561c30345fc0 2024-04-06T12:12:38.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.641+0000 7ff3739aab80 10 client.0 _ll_register_callbacks cb 0x561c30214420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:12:38.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.641+0000 7ff365592700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c3031b980 0x561c303173a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:38.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.641+0000 7ff365592700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c3031b980 0x561c303173a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38692/0 (socket says 172.21.15.189:38692) 2024-04-06T12:12:38.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.641+0000 7ff365592700 1 -- 172.21.15.189:0/1314731879 learned_addr learned my addr 172.21.15.189:0/1314731879 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:38.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.641+0000 7ff365d93700 1 --2- 172.21.15.189:0/1314731879 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c30345fc0 0x561c3031abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:38.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.641+0000 7ff364d91700 1 --2- 172.21.15.189:0/1314731879 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c30340c20 0x561c3031a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:38.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff365592700 1 -- 172.21.15.189:0/1314731879 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c30345fc0 msgr2=0x561c3031abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:38.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff365592700 1 --2- 172.21.15.189:0/1314731879 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c30345fc0 0x561c3031abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff365592700 1 -- 172.21.15.189:0/1314731879 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c30340c20 msgr2=0x561c3031a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:38.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff365592700 1 --2- 172.21.15.189:0/1314731879 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c30340c20 0x561c3031a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff365592700 1 -- 172.21.15.189:0/1314731879 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3600b3340 con 0x561c3031b980 2024-04-06T12:12:38.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff365592700 1 --2- 172.21.15.189:0/1314731879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c3031b980 0x561c303173a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff350024540 tx=0x7ff350003780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d32da15b64d0e6f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:38.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff33bfff700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:12:38.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff33bfff700 1 -- 172.21.15.189:0/1314731879 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff350014c10 con 0x561c3031b980 2024-04-06T12:12:38.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff33bfff700 1 -- 172.21.15.189:0/1314731879 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff35002dd90 con 0x561c3031b980 2024-04-06T12:12:38.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff3739aab80 1 -- 172.21.15.189:0/1314731879 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff358060120 con 0x561c3031b980 2024-04-06T12:12:38.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff33bfff700 1 -- 172.21.15.189:0/1314731879 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff350014d70 con 0x561c3031b980 2024-04-06T12:12:38.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff3739aab80 10 client.4797 fetch_fsmap learned FSMap version 9 2024-04-06T12:12:38.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff3739aab80 1 -- 172.21.15.189:0/1314731879 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x561c300f4c80 con 0x561c3031b980 2024-04-06T12:12:38.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff33bfff700 1 -- 172.21.15.189:0/1314731879 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff358060120 con 0x561c3031b980 2024-04-06T12:12:38.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff33bfff700 1 -- 172.21.15.189:0/1314731879 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 9) v1 ==== 3122+0+0 (secure 0 0 0) 0x7ff350005020 con 0x561c3031b980 2024-04-06T12:12:38.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff3739aab80 10 client.4797 fetch_fsmap finished waiting for FSMap version 9 2024-04-06T12:12:38.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff3739aab80 10 client.4797 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-06T12:12:38.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff3739aab80 20 client.4797 populate_metadata read hostname 'smithi189' 2024-04-06T12:12:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff3739aab80 1 --2- 172.21.15.189:0/1314731879 >> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] conn(0x561c303c9320 0x561c303c9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff3739aab80 4 client.4797 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-06T12:12:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.642+0000 7ff3739aab80 1 -- 172.21.15.189:0/1314731879 --> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561c300bdca0 con 0x561c303c9320 2024-04-06T12:12:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.643+0000 7ff364d91700 1 --2- 172.21.15.189:0/1314731879 >> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] conn(0x561c303c9320 0x561c303c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.643+0000 7ff33bfff700 10 client.4797 ms_handle_connect on v2:172.21.15.192:6834/516212261 2024-04-06T12:12:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.643+0000 7ff364d91700 1 --2- 172.21.15.189:0/1314731879 >> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] conn(0x561c303c9320 0x561c303c9700 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a572a62e7e308bd7 server_cookie=89c79087f2a7b82f in_seq=0 out_seq=0 2024-04-06T12:12:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.643+0000 7ff33bfff700 1 -- 172.21.15.189:0/1314731879 <== mds.0 v2:172.21.15.192:6834/516212261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x561c300bdca0 con 0x561c303c9320 2024-04-06T12:12:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.643+0000 7ff33bfff700 10 client.4797 handle_command_reply: tid=0 2024-04-06T12:12:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.643+0000 7ff3739aab80 1 client.4797 shutdown 2024-04-06T12:12:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.643+0000 7ff3739aab80 1 -- 172.21.15.189:0/1314731879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c3031b980 msgr2=0x561c303173a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.643+0000 7ff3739aab80 1 --2- 172.21.15.189:0/1314731879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c3031b980 0x561c303173a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff350024540 tx=0x7ff350003780 comp rx=0 tx=0).stop 2024-04-06T12:12:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.643+0000 7ff3739aab80 1 -- 172.21.15.189:0/1314731879 shutdown_connections 2024-04-06T12:12:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.643+0000 7ff3739aab80 1 --2- 172.21.15.189:0/1314731879 >> [v2:172.21.15.192:6834/516212261,v1:172.21.15.192:6835/516212261] conn(0x561c303c9320 0x561c303c9700 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.643+0000 7ff3739aab80 1 --2- 172.21.15.189:0/1314731879 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x561c30345fc0 0x561c3031abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.643+0000 7ff3739aab80 1 --2- 172.21.15.189:0/1314731879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c3031b980 0x561c303173a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.643+0000 7ff3739aab80 1 --2- 172.21.15.189:0/1314731879 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c30340c20 0x561c3031a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.643+0000 7ff33bfff700 0 client.4797 ms_handle_reset on v2:172.21.15.192:6834/516212261 2024-04-06T12:12:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.643+0000 7ff3739aab80 1 -- 172.21.15.189:0/1314731879 >> 172.21.15.189:0/1314731879 conn(0x561c3027c650 msgr2=0x561c303145c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.643+0000 7ff3739aab80 1 -- 172.21.15.189:0/1314731879 shutdown_connections 2024-04-06T12:12:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.643+0000 7ff3739aab80 1 -- 172.21.15.189:0/1314731879 wait complete. 2024-04-06T12:12:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.644+0000 7ff3739aab80 20 client.4797 trim_cache size 0 max 16384 2024-04-06T12:12:38.655 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T12:12:38.655 INFO:teuthology.orchestra.run.smithi189.stdout: "cluster_fsid": "e30041de-e9a3-4351-8ca7-29d7f3e3728c", 2024-04-06T12:12:38.655 INFO:teuthology.orchestra.run.smithi189.stdout: "whoami": 0, 2024-04-06T12:12:38.655 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 4270, 2024-04-06T12:12:38.655 INFO:teuthology.orchestra.run.smithi189.stdout: "want_state": "up:active", 2024-04-06T12:12:38.655 INFO:teuthology.orchestra.run.smithi189.stdout: "state": "up:active", 2024-04-06T12:12:38.655 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs", 2024-04-06T12:12:38.655 INFO:teuthology.orchestra.run.smithi189.stdout: "rank_uptime": 34.445972597999997, 2024-04-06T12:12:38.655 INFO:teuthology.orchestra.run.smithi189.stdout: "mdsmap_epoch": 9, 2024-04-06T12:12:38.655 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch": 24, 2024-04-06T12:12:38.656 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch_barrier": 24, 2024-04-06T12:12:38.656 INFO:teuthology.orchestra.run.smithi189.stdout: "uptime": 40.890306076999998 2024-04-06T12:12:38.656 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T12:12:38.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.644+0000 7ff3677f6700 1 -- 172.21.15.189:0/82554428 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff34406ac90 msgr2=0x7ff34406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:38.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.644+0000 7ff3677f6700 1 --2- 172.21.15.189:0/82554428 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff34406ac90 0x7ff34406d150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff350003f40 tx=0x7ff350009040 comp rx=0 tx=0).stop 2024-04-06T12:12:38.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.644+0000 7ff3677f6700 1 -- 172.21.15.189:0/82554428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff36015ddf0 msgr2=0x7ff360153230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:38.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.644+0000 7ff3677f6700 1 --2- 172.21.15.189:0/82554428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff36015ddf0 0x7ff360153230 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff35800cb90 tx=0x7ff35800cf50 comp rx=0 tx=0).stop 2024-04-06T12:12:38.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.644+0000 7ff3677f6700 1 -- 172.21.15.189:0/82554428 shutdown_connections 2024-04-06T12:12:38.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.644+0000 7ff3677f6700 1 --2- 172.21.15.189:0/82554428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff36015ddf0 0x7ff360153230 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.644+0000 7ff3677f6700 1 --2- 172.21.15.189:0/82554428 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff34406ac90 0x7ff34406d150 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.644+0000 7ff3677f6700 1 --2- 172.21.15.189:0/82554428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff36006ae40 0x7ff3600cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.644+0000 7ff3677f6700 1 --2- 172.21.15.189:0/82554428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff36015d7b0 0x7ff3600b1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:38.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.644+0000 7ff3677f6700 1 -- 172.21.15.189:0/82554428 >> 172.21.15.189:0/82554428 conn(0x7ff3600d00b0 msgr2=0x7ff3600ae140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:38.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.644+0000 7ff3677f6700 1 -- 172.21.15.189:0/82554428 shutdown_connections 2024-04-06T12:12:38.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:38.644+0000 7ff3677f6700 1 -- 172.21.15.189:0/82554428 wait complete. 2024-04-06T12:12:38.659 INFO:teuthology.run_tasks:Running task sequential... 2024-04-06T12:12:38.668 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-06T12:12:38.669 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-06T12:12:38.669 INFO:teuthology.task.print:Enabling mgr modules 2024-04-06T12:12:38.669 INFO:teuthology.run_tasks:Running task kclient... 2024-04-06T12:12:38.718 INFO:tasks.kclient:Mounting kernel clients... 2024-04-06T12:12:38.719 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-06T12:12:38.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:38.719 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T12:12:38.736 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-06T12:12:38.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:38.737 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-06T12:12:38.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:38.815 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link delete ceph-brx 2024-04-06T12:12:38.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:38.911 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T12:12:38.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:38.928 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link delete ceph-brx 2024-04-06T12:12:38.998 INFO:teuthology.orchestra.run.smithi189.stderr:Cannot find device "ceph-brx" 2024-04-06T12:12:39.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T12:12:39.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:39.000 DEBUG:teuthology.orchestra.run.smithi192:> ip netns list 2024-04-06T12:12:39.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:39.022 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link delete ceph-brx 2024-04-06T12:12:39.098 INFO:teuthology.orchestra.run.smithi192.stderr:Cannot find device "ceph-brx" 2024-04-06T12:12:39.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T12:12:39.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:39.100 DEBUG:teuthology.orchestra.run.smithi192:> ip netns list 2024-04-06T12:12:39.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:39.159 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link delete ceph-brx 2024-04-06T12:12:39.231 INFO:teuthology.orchestra.run.smithi192.stderr:Cannot find device "ceph-brx" 2024-04-06T12:12:39.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T12:12:39.233 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-04-06T12:12:39.233 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-06T12:12:39.233 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T12:12:39.234 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T12:12:39.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.357+0000 7f56ae573700 1 Processor -- start 2024-04-06T12:12:39.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.357+0000 7f56ae573700 1 -- start start 2024-04-06T12:12:39.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.358+0000 7f56ae573700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f56a80d11b0 0x7f56a80d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:39.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.358+0000 7f56ae573700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56a80cfa00 0x7f56a80cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:39.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.358+0000 7f56ae573700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a80d1ad0 con 0x7f56a80cb870 2024-04-06T12:12:39.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.358+0000 7f56ae573700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a80ccdc0 con 0x7f56a80d11b0 2024-04-06T12:12:39.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.358+0000 7f56ae573700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a80d0b50 con 0x7f56a80cfa00 2024-04-06T12:12:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.358+0000 7f56acb10700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56a80cfa00 0x7f56a80cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.358+0000 7f56a7fff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f56a80d11b0 0x7f56a80d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.358+0000 7f56acb10700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56a80cfa00 0x7f56a80cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:51834/0 (socket says 172.21.15.189:51834) 2024-04-06T12:12:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.358+0000 7f56a7fff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f56a80d11b0 0x7f56a80d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:46724/0 (socket says 172.21.15.189:46724) 2024-04-06T12:12:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.358+0000 7f56a77fe700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f56a80cb870 0x7f56a815c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:49270/0 (socket says 172.21.15.189:49270) 2024-04-06T12:12:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.358+0000 7f56acb10700 1 -- 172.21.15.189:0/2605542144 learned_addr learned my addr 172.21.15.189:0/2605542144 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.359+0000 7f56a7fff700 1 -- 172.21.15.189:0/2605542144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56a80cfa00 msgr2=0x7f56a80cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.359+0000 7f56a7fff700 1 --2- 172.21.15.189:0/2605542144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56a80cfa00 0x7f56a80cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.359+0000 7f56a7fff700 1 -- 172.21.15.189:0/2605542144 >> v1:172.21.15.192:6789/0 conn(0x7f56a80cb870 legacy=0x7f56a815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.359+0000 7f56a7fff700 1 -- 172.21.15.189:0/2605542144 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a80cbd80 con 0x7f56a80d11b0 2024-04-06T12:12:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.359+0000 7f56a6ffd700 1 -- 172.21.15.189:0/2605542144 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393813128 0 0) 0x7f56a80d1ad0 con 0x7f56a80cb870 2024-04-06T12:12:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.359+0000 7f56acb10700 1 --2- 172.21.15.189:0/2605542144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56a80cfa00 0x7f56a80cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.359+0000 7f56a7fff700 1 --2- 172.21.15.189:0/2605542144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56a80d11b0 0x7f56a80d1590 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5698009840 tx=0x7f5698009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=673238d5079b1eab server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.359+0000 7f56a6ffd700 1 -- 172.21.15.189:0/2605542144 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5698011830 con 0x7f56a80d11b0 2024-04-06T12:12:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.359+0000 7f56a6ffd700 1 -- 172.21.15.189:0/2605542144 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5698011990 con 0x7f56a80d11b0 2024-04-06T12:12:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.359+0000 7f56a6ffd700 1 -- 172.21.15.189:0/2605542144 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5698011d20 con 0x7f56a80d11b0 2024-04-06T12:12:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.360+0000 7f56ae573700 1 -- 172.21.15.189:0/2605542144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56a80d11b0 msgr2=0x7f56a80d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.360+0000 7f56ae573700 1 --2- 172.21.15.189:0/2605542144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56a80d11b0 0x7f56a80d1590 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5698009840 tx=0x7f5698009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.360+0000 7f56ae573700 1 -- 172.21.15.189:0/2605542144 shutdown_connections 2024-04-06T12:12:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.360+0000 7f56ae573700 1 --2- 172.21.15.189:0/2605542144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56a80cfa00 0x7f56a80cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.360+0000 7f56ae573700 1 --2- 172.21.15.189:0/2605542144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56a80d11b0 0x7f56a80d1590 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.360+0000 7f56ae573700 1 -- 172.21.15.189:0/2605542144 >> 172.21.15.189:0/2605542144 conn(0x7f56a80b2db0 msgr2=0x7f56a80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.360+0000 7f56ae573700 1 -- 172.21.15.189:0/2605542144 shutdown_connections 2024-04-06T12:12:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.360+0000 7f56ae573700 1 -- 172.21.15.189:0/2605542144 wait complete. 2024-04-06T12:12:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.361+0000 7f56ae573700 1 Processor -- start 2024-04-06T12:12:39.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.361+0000 7f56ae573700 1 -- start start 2024-04-06T12:12:39.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.362+0000 7f56ae573700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56a80cb870 0x7f56a81f2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:39.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.362+0000 7f56ae573700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a80cfa00 0x7f56a806d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:39.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.362+0000 7f56ae573700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56a80d11b0 0x7f56a80c79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:39.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.362+0000 7f56ae573700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a806d880 con 0x7f56a80cfa00 2024-04-06T12:12:39.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.362+0000 7f56ae573700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a81f33e0 con 0x7f56a80cb870 2024-04-06T12:12:39.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.362+0000 7f56ae573700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a81f36c0 con 0x7f56a80d11b0 2024-04-06T12:12:39.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.362+0000 7f56a77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a80cfa00 0x7f56a806d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:39.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.362+0000 7f56a77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a80cfa00 0x7f56a806d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46606/0 (socket says 172.21.15.189:46606) 2024-04-06T12:12:39.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.362+0000 7f56a77fe700 1 -- 172.21.15.189:0/1880718570 learned_addr learned my addr 172.21.15.189:0/1880718570 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:39.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.362+0000 7f56acb10700 1 --2- 172.21.15.189:0/1880718570 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56a80d11b0 0x7f56a80c79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:39.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.362+0000 7f56a7fff700 1 --2- 172.21.15.189:0/1880718570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56a80cb870 0x7f56a81f2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:39.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.362+0000 7f56a77fe700 1 -- 172.21.15.189:0/1880718570 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56a80d11b0 msgr2=0x7f56a80c79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:39.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.362+0000 7f56a77fe700 1 --2- 172.21.15.189:0/1880718570 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56a80d11b0 0x7f56a80c79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:39.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.362+0000 7f56a77fe700 1 -- 172.21.15.189:0/1880718570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56a80cb870 msgr2=0x7f56a81f2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:39.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.362+0000 7f56a77fe700 1 --2- 172.21.15.189:0/1880718570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56a80cb870 0x7f56a81f2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:39.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.362+0000 7f56a77fe700 1 -- 172.21.15.189:0/1880718570 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a0009b30 con 0x7f56a80cfa00 2024-04-06T12:12:39.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.363+0000 7f56a77fe700 1 --2- 172.21.15.189:0/1880718570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a80cfa00 0x7f56a806d340 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f569c00c520 tx=0x7f569c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:39.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.363+0000 7f56a4ff9700 1 -- 172.21.15.189:0/1880718570 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c004c80 con 0x7f56a80cfa00 2024-04-06T12:12:39.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.363+0000 7f56a4ff9700 1 -- 172.21.15.189:0/1880718570 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569c004de0 con 0x7f56a80cfa00 2024-04-06T12:12:39.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.363+0000 7f56ae573700 1 -- 172.21.15.189:0/1880718570 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56980094a0 con 0x7f56a80cfa00 2024-04-06T12:12:39.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.363+0000 7f56a4ff9700 1 -- 172.21.15.189:0/1880718570 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c0072d0 con 0x7f56a80cfa00 2024-04-06T12:12:39.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.363+0000 7f56ae573700 1 -- 172.21.15.189:0/1880718570 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a81f3c80 con 0x7f56a80cfa00 2024-04-06T12:12:39.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.364+0000 7f56ae573700 1 -- 172.21.15.189:0/1880718570 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a8074990 con 0x7f56a80cfa00 2024-04-06T12:12:39.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.364+0000 7f56a4ff9700 1 -- 172.21.15.189:0/1880718570 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f569c007430 con 0x7f56a80cfa00 2024-04-06T12:12:39.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.365+0000 7f56a4ff9700 1 --2- 172.21.15.189:0/1880718570 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f569406ac90 0x7f569406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:39.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.365+0000 7f56a4ff9700 1 -- 172.21.15.189:0/1880718570 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f569c089320 con 0x7f56a80cfa00 2024-04-06T12:12:39.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.365+0000 7f56a7fff700 1 --2- 172.21.15.189:0/1880718570 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f569406ac90 0x7f569406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:39.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.365+0000 7f56a7fff700 1 --2- 172.21.15.189:0/1880718570 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f569406ac90 0x7f569406d150 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f56980090b0 tx=0x7f5698008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:39.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.369+0000 7f56a4ff9700 1 -- 172.21.15.189:0/1880718570 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f569c054210 con 0x7f56a80cfa00 2024-04-06T12:12:39.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.610+0000 7f56ae573700 1 -- 172.21.15.189:0/1880718570 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f56a80b0a00 con 0x7f56a80cfa00 2024-04-06T12:12:39.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.610+0000 7f56a4ff9700 1 -- 172.21.15.189:0/1880718570 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f569c057830 con 0x7f56a80cfa00 2024-04-06T12:12:39.609 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-06T12:12:39.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.613+0000 7f56ae573700 1 -- 172.21.15.189:0/1880718570 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f569406ac90 msgr2=0x7f569406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:39.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.613+0000 7f56ae573700 1 --2- 172.21.15.189:0/1880718570 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f569406ac90 0x7f569406d150 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f56980090b0 tx=0x7f5698008030 comp rx=0 tx=0).stop 2024-04-06T12:12:39.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.613+0000 7f56ae573700 1 -- 172.21.15.189:0/1880718570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a80cfa00 msgr2=0x7f56a806d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:39.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.613+0000 7f56ae573700 1 --2- 172.21.15.189:0/1880718570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a80cfa00 0x7f56a806d340 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f569c00c520 tx=0x7f569c00c8e0 comp rx=0 tx=0).stop 2024-04-06T12:12:39.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.613+0000 7f56ae573700 1 -- 172.21.15.189:0/1880718570 shutdown_connections 2024-04-06T12:12:39.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.613+0000 7f56ae573700 1 --2- 172.21.15.189:0/1880718570 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56a80d11b0 0x7f56a80c79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:39.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.613+0000 7f56ae573700 1 --2- 172.21.15.189:0/1880718570 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f569406ac90 0x7f569406d150 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:39.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.613+0000 7f56ae573700 1 --2- 172.21.15.189:0/1880718570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56a80cb870 0x7f56a81f2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:39.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.613+0000 7f56ae573700 1 --2- 172.21.15.189:0/1880718570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a80cfa00 0x7f56a806d340 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:39.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.613+0000 7f56ae573700 1 -- 172.21.15.189:0/1880718570 >> 172.21.15.189:0/1880718570 conn(0x7f56a80b2db0 msgr2=0x7f56a80b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:39.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.613+0000 7f56ae573700 1 -- 172.21.15.189:0/1880718570 shutdown_connections 2024-04-06T12:12:39.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:39.613+0000 7f56ae573700 1 -- 172.21.15.189:0/1880718570 wait complete. 2024-04-06T12:12:39.623 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-06T12:12:39.623 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T12:12:39.623 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-06T12:12:39.623 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-06T12:12:39.623 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T12:12:39.623 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-06T12:12:39.623 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-06T12:12:39.624 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T12:12:39.624 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:12:39.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:39.624 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-06T12:12:39.641 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T12:12:39.641 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T12:12:39.641 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T12:12:39.641 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:39.641 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T12:12:39.641 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:39.641 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:12:39.641 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:39.642 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-06T12:12:39.642 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:12:39.642 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:39.642 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-06T12:12:39.642 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:12:39.642 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:39.642 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T12:12:39.642 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:39.642 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T12:12:39.642 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 42330sec preferred_lft 42330sec 2024-04-06T12:12:39.642 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::8592:df2c:92e2:dace/64 scope link noprefixroute 2024-04-06T12:12:39.642 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:39.643 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-06T12:12:39.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:39.643 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:12:39.643 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:12:39.643 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link add name ceph-brx type bridge 2024-04-06T12:12:39.643 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip addr flush dev ceph-brx 2024-04-06T12:12:39.643 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set ceph-brx up 2024-04-06T12:12:39.643 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-06T12:12:39.643 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:12:39.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:39.760 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T12:12:39.786 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-06T12:12:39.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:39.788 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-06T12:12:39.847 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-06T12:12:39.848 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T12:12:39.848 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T12:12:39.848 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T12:12:39.848 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T12:12:39.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:39.848 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:12:39.848 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:12:39.848 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T12:12:39.848 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T12:12:39.848 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T12:12:39.848 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:12:39.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:39.955 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T12:12:39.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:39.973 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list-id 2024-04-06T12:12:40.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:40.030 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:12:40.030 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:12:40.030 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T12:12:40.030 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-06T12:12:40.030 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:12:40.124 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-06T12:12:40.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:40.124 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:12:40.124 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:12:40.125 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-06T12:12:40.125 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-06T12:12:40.125 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-06T12:12:40.125 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-06T12:12:40.125 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-06T12:12:40.125 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:12:40.156 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:12:40.170 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:12:40.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:40.334 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:12:40.334 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:12:40.334 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-04-06T12:12:40.334 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set dev brx.0 master ceph-brx 2024-04-06T12:12:40.334 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:12:40.432 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:12:40.433 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-06T12:12:40.450 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:12:40.450 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:12:40.450 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-06T12:12:40.506 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T12:12:40.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:40.507 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:12:40.593 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:12:40.593 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:12:40.593 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:12:40.719 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:12:40.719 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:12:40.719 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:12:40.719 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:12:40.719 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:12:40.737 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:12:40.738 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:12:40.738 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:12:40.738 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:12:40.738 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:12:40.738 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790" 2024-04-06T12:12:40.738 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2024-04-06T12:12:40.738 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e30041de-e9a3-4351-8ca7-29d7f3e3728c" will pass to kernel 2024-04-06T12:12:40.740 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T12:12:40.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:40.740 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-06T12:12:40.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:40.767 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:12:40.841 INFO:teuthology.orchestra.run.smithi189.stdout:client4732 2024-04-06T12:12:40.841 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:12:40.841 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:12:40.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:40.900 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:12:40.973 INFO:teuthology.orchestra.run.smithi189.stdout:client4732 2024-04-06T12:12:40.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:40.974 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:12:41.045 INFO:teuthology.orchestra.run.smithi189.stdout:client4732 2024-04-06T12:12:41.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:41.046 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e30041de-e9a3-4351-8ca7-29d7f3e3728c.client4732/status') 2024-04-06T12:12:41.127 INFO:teuthology.orchestra.run.smithi189.stdout:instance: client.4732 (3)172.21.15.189:0/1201138 2024-04-06T12:12:41.127 INFO:teuthology.orchestra.run.smithi189.stdout:blocklisted: false 2024-04-06T12:12:41.127 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2024-04-06T12:12:41.127 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2024-04-06T12:12:41.127 INFO:teuthology.orchestra.run.smithi189.stderr:68 bytes copied, 2.1701e-05 s, 3.1 MB/s 2024-04-06T12:12:41.128 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2024-04-06T12:12:41.128 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-06T12:12:41.128 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-06T12:12:41.129 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T12:12:41.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.292+0000 7f27c86ba700 1 Processor -- start 2024-04-06T12:12:41.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.292+0000 7f27c86ba700 1 -- start start 2024-04-06T12:12:41.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.293+0000 7f27c86ba700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f27c0158fe0 0x7f27c006a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:41.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.293+0000 7f27c86ba700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27c006ac70 0x7f27c015bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:41.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.293+0000 7f27c86ba700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c015f9e0 con 0x7f27c015c100 2024-04-06T12:12:41.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.293+0000 7f27c86ba700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27c006b0f0 con 0x7f27c0158fe0 2024-04-06T12:12:41.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.293+0000 7f27c86ba700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c015f0d0 con 0x7f27c006ac70 2024-04-06T12:12:41.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.293+0000 7f27c6456700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f27c0158fe0 0x7f27c006a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:41.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.293+0000 7f27c5c55700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27c006ac70 0x7f27c015bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:41.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.293+0000 7f27c6456700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f27c0158fe0 0x7f27c006a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:46750/0 (socket says 172.21.15.189:46750) 2024-04-06T12:12:41.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.293+0000 7f27c6456700 1 -- 172.21.15.189:0/3766186030 learned_addr learned my addr 172.21.15.189:0/3766186030 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:41.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.293+0000 7f27c5c55700 1 -- 172.21.15.189:0/3766186030 >> v2:172.21.15.192:3300/0 conn(0x7f27c0158fe0 msgr2=0x7f27c006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:41.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.293+0000 7f27c5c55700 1 --2- 172.21.15.189:0/3766186030 >> v2:172.21.15.192:3300/0 conn(0x7f27c0158fe0 0x7f27c006a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:41.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.293+0000 7f27c5c55700 1 -- 172.21.15.189:0/3766186030 >> v1:172.21.15.192:6789/0 conn(0x7f27c015c100 legacy=0x7f27c00717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:41.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.293+0000 7f27c5c55700 1 -- 172.21.15.189:0/3766186030 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c015c8f0 con 0x7f27c006ac70 2024-04-06T12:12:41.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.293+0000 7f27c6456700 1 --2- 172.21.15.189:0/3766186030 >> v2:172.21.15.192:3300/0 conn(0x7f27c0158fe0 0x7f27c006a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:41.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.294+0000 7f27c5c55700 1 --2- 172.21.15.189:0/3766186030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27c006ac70 0x7f27c015bbc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f27b0009840 tx=0x7f27b0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=763c80e123722590 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:41.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.294+0000 7f27c5454700 1 -- 172.21.15.189:0/3766186030 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b0011830 con 0x7f27c006ac70 2024-04-06T12:12:41.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.294+0000 7f27c5454700 1 -- 172.21.15.189:0/3766186030 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b0011990 con 0x7f27c006ac70 2024-04-06T12:12:41.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.294+0000 7f27c5454700 1 -- 172.21.15.189:0/3766186030 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b0011d20 con 0x7f27c006ac70 2024-04-06T12:12:41.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.294+0000 7f27c86ba700 1 -- 172.21.15.189:0/3766186030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27c006ac70 msgr2=0x7f27c015bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:41.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.294+0000 7f27c86ba700 1 --2- 172.21.15.189:0/3766186030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27c006ac70 0x7f27c015bbc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f27b0009840 tx=0x7f27b0009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:41.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.294+0000 7f27c86ba700 1 -- 172.21.15.189:0/3766186030 shutdown_connections 2024-04-06T12:12:41.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.294+0000 7f27c86ba700 1 --2- 172.21.15.189:0/3766186030 >> v2:172.21.15.192:3300/0 conn(0x7f27c0158fe0 0x7f27c006a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:41.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.294+0000 7f27c86ba700 1 --2- 172.21.15.189:0/3766186030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27c006ac70 0x7f27c015bbc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:41.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.294+0000 7f27c86ba700 1 -- 172.21.15.189:0/3766186030 >> 172.21.15.189:0/3766186030 conn(0x7f27c00d00b0 msgr2=0x7f27c00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:41.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.294+0000 7f27c86ba700 1 -- 172.21.15.189:0/3766186030 shutdown_connections 2024-04-06T12:12:41.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.294+0000 7f27c86ba700 1 -- 172.21.15.189:0/3766186030 wait complete. 2024-04-06T12:12:41.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.295+0000 7f27c86ba700 1 Processor -- start 2024-04-06T12:12:41.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.296+0000 7f27c86ba700 1 -- start start 2024-04-06T12:12:41.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.296+0000 7f27c86ba700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27c006ac70 0x7f27c013c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:41.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.296+0000 7f27c86ba700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27c0158fe0 0x7f27c0158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:41.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.296+0000 7f27c86ba700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27c015c100 0x7f27c01437d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:41.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.296+0000 7f27c86ba700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c013c6e0 con 0x7f27c015c100 2024-04-06T12:12:41.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.296+0000 7f27c86ba700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c013b100 con 0x7f27c0158fe0 2024-04-06T12:12:41.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.296+0000 7f27c86ba700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c013b3e0 con 0x7f27c006ac70 2024-04-06T12:12:41.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.296+0000 7f27c6c57700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27c015c100 0x7f27c01437d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:41.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.296+0000 7f27c6c57700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27c015c100 0x7f27c01437d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46618/0 (socket says 172.21.15.189:46618) 2024-04-06T12:12:41.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.296+0000 7f27c6c57700 1 -- 172.21.15.189:0/3339117872 learned_addr learned my addr 172.21.15.189:0/3339117872 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:41.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.296+0000 7f27c5c55700 1 --2- 172.21.15.189:0/3339117872 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27c0158fe0 0x7f27c0158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:41.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.296+0000 7f27c6456700 1 --2- 172.21.15.189:0/3339117872 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27c006ac70 0x7f27c013c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:41.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.297+0000 7f27c6c57700 1 -- 172.21.15.189:0/3339117872 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27c006ac70 msgr2=0x7f27c013c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:41.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.297+0000 7f27c6c57700 1 --2- 172.21.15.189:0/3339117872 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27c006ac70 0x7f27c013c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:41.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.297+0000 7f27c6c57700 1 -- 172.21.15.189:0/3339117872 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27c0158fe0 msgr2=0x7f27c0158030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:41.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.297+0000 7f27c6c57700 1 --2- 172.21.15.189:0/3339117872 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27c0158fe0 0x7f27c0158030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:41.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.297+0000 7f27c6c57700 1 -- 172.21.15.189:0/3339117872 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc009550 con 0x7f27c015c100 2024-04-06T12:12:41.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.297+0000 7f27c5c55700 1 --2- 172.21.15.189:0/3339117872 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27c0158fe0 0x7f27c0158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:41.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.297+0000 7f27c6c57700 1 --2- 172.21.15.189:0/3339117872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27c015c100 0x7f27c01437d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f27b800cc50 tx=0x7f27b800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:41.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.297+0000 7f27b6ffd700 1 -- 172.21.15.189:0/3339117872 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b8003c00 con 0x7f27c015c100 2024-04-06T12:12:41.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.297+0000 7f27b6ffd700 1 -- 172.21.15.189:0/3339117872 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b8003d60 con 0x7f27c015c100 2024-04-06T12:12:41.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.297+0000 7f27b6ffd700 1 -- 172.21.15.189:0/3339117872 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b800aab0 con 0x7f27c015c100 2024-04-06T12:12:41.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.297+0000 7f27c86ba700 1 -- 172.21.15.189:0/3339117872 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b00094a0 con 0x7f27c015c100 2024-04-06T12:12:41.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.297+0000 7f27c86ba700 1 -- 172.21.15.189:0/3339117872 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c013b9a0 con 0x7f27c015c100 2024-04-06T12:12:41.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.298+0000 7f27c86ba700 1 -- 172.21.15.189:0/3339117872 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27c0067bb0 con 0x7f27c015c100 2024-04-06T12:12:41.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.298+0000 7f27b6ffd700 1 -- 172.21.15.189:0/3339117872 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f27b800ac10 con 0x7f27c015c100 2024-04-06T12:12:41.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.299+0000 7f27b6ffd700 1 --2- 172.21.15.189:0/3339117872 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f27a406ac90 0x7f27a406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:41.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.299+0000 7f27b6ffd700 1 -- 172.21.15.189:0/3339117872 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f27b8089560 con 0x7f27c015c100 2024-04-06T12:12:41.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.299+0000 7f27c6456700 1 --2- 172.21.15.189:0/3339117872 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f27a406ac90 0x7f27a406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:41.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.299+0000 7f27c6456700 1 --2- 172.21.15.189:0/3339117872 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f27a406ac90 0x7f27a406d150 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f27bc006d30 tx=0x7f27bc0092c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:41.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.303+0000 7f27b6ffd700 1 -- 172.21.15.189:0/3339117872 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f27b80544e0 con 0x7f27c015c100 2024-04-06T12:12:41.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.545+0000 7f27c86ba700 1 -- 172.21.15.189:0/3339117872 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f27c0074990 con 0x7f27c015c100 2024-04-06T12:12:41.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.545+0000 7f27b6ffd700 1 -- 172.21.15.189:0/3339117872 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f27b8057b00 con 0x7f27c015c100 2024-04-06T12:12:41.544 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-06T12:12:41.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.547+0000 7f27c86ba700 1 -- 172.21.15.189:0/3339117872 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f27a406ac90 msgr2=0x7f27a406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:41.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.547+0000 7f27c86ba700 1 --2- 172.21.15.189:0/3339117872 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f27a406ac90 0x7f27a406d150 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f27bc006d30 tx=0x7f27bc0092c0 comp rx=0 tx=0).stop 2024-04-06T12:12:41.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.547+0000 7f27c86ba700 1 -- 172.21.15.189:0/3339117872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27c015c100 msgr2=0x7f27c01437d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:41.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.547+0000 7f27c86ba700 1 --2- 172.21.15.189:0/3339117872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27c015c100 0x7f27c01437d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f27b800cc50 tx=0x7f27b800cf60 comp rx=0 tx=0).stop 2024-04-06T12:12:41.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.547+0000 7f27c86ba700 1 -- 172.21.15.189:0/3339117872 shutdown_connections 2024-04-06T12:12:41.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.547+0000 7f27c86ba700 1 --2- 172.21.15.189:0/3339117872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27c015c100 0x7f27c01437d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:41.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.547+0000 7f27c86ba700 1 --2- 172.21.15.189:0/3339117872 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27c006ac70 0x7f27c013c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:41.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.548+0000 7f27c86ba700 1 --2- 172.21.15.189:0/3339117872 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f27a406ac90 0x7f27a406d150 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:41.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.548+0000 7f27c86ba700 1 --2- 172.21.15.189:0/3339117872 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27c0158fe0 0x7f27c0158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:41.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.548+0000 7f27c86ba700 1 -- 172.21.15.189:0/3339117872 >> 172.21.15.189:0/3339117872 conn(0x7f27c00d00b0 msgr2=0x7f27c006f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:41.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.548+0000 7f27c86ba700 1 -- 172.21.15.189:0/3339117872 shutdown_connections 2024-04-06T12:12:41.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:41.548+0000 7f27c86ba700 1 -- 172.21.15.189:0/3339117872 wait complete. 2024-04-06T12:12:41.558 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-06T12:12:41.558 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T12:12:41.558 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-06T12:12:41.558 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-06T12:12:41.558 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-06T12:12:41.558 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-06T12:12:41.558 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-06T12:12:41.558 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T12:12:41.558 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-06T12:12:41.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:41.558 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-06T12:12:41.576 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T12:12:41.576 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T12:12:41.576 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T12:12:41.576 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:41.576 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T12:12:41.576 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:41.576 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:12:41.576 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:41.576 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-06T12:12:41.576 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:12:41.577 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:41.577 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-06T12:12:41.577 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:12:41.577 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:41.577 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T12:12:41.577 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:41.577 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T12:12:41.577 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 42328sec preferred_lft 42328sec 2024-04-06T12:12:41.577 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::8592:df2c:92e2:dace/64 scope link noprefixroute 2024-04-06T12:12:41.577 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:41.577 INFO:teuthology.orchestra.run.smithi189.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T12:12:41.577 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 62:a5:8d:c5:1a:9c brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:41.577 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-06T12:12:41.577 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:41.577 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::88e:b6ff:fe6f:1cb0/64 scope link 2024-04-06T12:12:41.578 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:41.578 INFO:teuthology.orchestra.run.smithi189.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T12:12:41.578 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 62:a5:8d:c5:1a:9c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T12:12:41.578 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::60a5:8dff:fec5:1a9c/64 scope link tentative 2024-04-06T12:12:41.578 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:41.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:41.578 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T12:12:41.641 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-06T12:12:41.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:41.643 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-06T12:12:41.702 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-06T12:12:41.702 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T12:12:41.703 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T12:12:41.703 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T12:12:41.703 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T12:12:41.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:41.703 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:12:41.703 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:12:41.703 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T12:12:41.703 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T12:12:41.703 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T12:12:41.704 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:12:41.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:41.809 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T12:12:41.826 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-06T12:12:41.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:41.827 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list-id 2024-04-06T12:12:41.884 INFO:teuthology.orchestra.run.smithi189.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-06T12:12:41.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:41.884 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:12:41.884 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:12:41.885 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-06T12:12:41.885 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-06T12:12:41.885 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:12:41.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:41.979 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-06T12:12:42.017 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T12:12:42.017 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T12:12:42.017 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T12:12:42.017 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:42.017 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T12:12:42.017 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:42.017 INFO:teuthology.orchestra.run.smithi189.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T12:12:42.018 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 76:ba:e0:ce:99:58 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-06T12:12:42.018 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-06T12:12:42.018 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:42.018 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::74ba:e0ff:fece:9958/64 scope link tentative 2024-04-06T12:12:42.018 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:42.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:42.035 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-06T12:12:42.115 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T12:12:42.115 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T12:12:42.115 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T12:12:42.115 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:42.115 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T12:12:42.115 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:42.115 INFO:teuthology.orchestra.run.smithi189.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T12:12:42.115 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 76:ba:e0:ce:99:58 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-06T12:12:42.115 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-06T12:12:42.115 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:42.116 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::74ba:e0ff:fece:9958/64 scope link tentative 2024-04-06T12:12:42.116 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:42.134 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-06T12:12:42.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:42.135 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:12:42.135 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:12:42.135 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-06T12:12:42.135 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-06T12:12:42.135 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-06T12:12:42.135 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-06T12:12:42.135 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-06T12:12:42.135 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:12:42.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:42.379 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:12:42.380 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:12:42.380 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.1 up 2024-04-06T12:12:42.380 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set dev brx.1 master ceph-brx 2024-04-06T12:12:42.380 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:12:42.477 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:12:42.477 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-06T12:12:42.495 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-06T12:12:42.495 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:12:42.495 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-06T12:12:42.552 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T12:12:42.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:42.552 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-04-06T12:12:42.638 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-04-06T12:12:42.638 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-04-06T12:12:42.638 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:12:42.647 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-04-06T12:12:42.647 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-04-06T12:12:42.647 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:12:42.647 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:12:42.647 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:12:42.666 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-04-06T12:12:42.666 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-04-06T12:12:42.666 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:12:42.666 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:12:42.666 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:12:42.666 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790" 2024-04-06T12:12:42.667 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2024-04-06T12:12:42.667 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=e30041de-e9a3-4351-8ca7-29d7f3e3728c" will pass to kernel 2024-04-06T12:12:42.669 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T12:12:42.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:42.669 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-06T12:12:42.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:42.697 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:12:42.770 INFO:teuthology.orchestra.run.smithi189.stdout:client4601 2024-04-06T12:12:42.770 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:12:42.770 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:12:42.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:42.828 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:12:42.901 INFO:teuthology.orchestra.run.smithi189.stdout:client4601 2024-04-06T12:12:42.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:42.901 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:12:42.974 INFO:teuthology.orchestra.run.smithi189.stdout:client4601 2024-04-06T12:12:42.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:42.975 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e30041de-e9a3-4351-8ca7-29d7f3e3728c.client4601/status') 2024-04-06T12:12:43.056 INFO:teuthology.orchestra.run.smithi189.stdout:instance: client.4601 (3)172.21.15.189:0/1453088846 2024-04-06T12:12:43.056 INFO:teuthology.orchestra.run.smithi189.stdout:blocklisted: false 2024-04-06T12:12:43.057 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2024-04-06T12:12:43.057 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2024-04-06T12:12:43.057 INFO:teuthology.orchestra.run.smithi189.stderr:71 bytes copied, 2.1815e-05 s, 3.3 MB/s 2024-04-06T12:12:43.057 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-04-06T12:12:43.057 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-06T12:12:43.058 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-06T12:12:43.058 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T12:12:43.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.221+0000 7f72c1347700 1 Processor -- start 2024-04-06T12:12:43.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.222+0000 7f72c1347700 1 -- start start 2024-04-06T12:12:43.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.222+0000 7f72c1347700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f72bc15d040 0x7f72bc15dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:43.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.222+0000 7f72c1347700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f72bc15e2a0 0x7f72bc06c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:43.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.222+0000 7f72c1347700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72bc15d8d0 con 0x7f72bc0b2cd0 2024-04-06T12:12:43.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.222+0000 7f72c1347700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72bc0cba40 con 0x7f72bc15d040 2024-04-06T12:12:43.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.222+0000 7f72c1347700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72bc0cf660 con 0x7f72bc15e2a0 2024-04-06T12:12:43.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.222+0000 7f72ba7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f72bc15d040 0x7f72bc15dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:43.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.223+0000 7f72ba7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f72bc15d040 0x7f72bc15dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46628/0 (socket says 172.21.15.189:46628) 2024-04-06T12:12:43.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.223+0000 7f72ba7fc700 1 -- 172.21.15.189:0/1525602334 learned_addr learned my addr 172.21.15.189:0/1525602334 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:43.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.223+0000 7f72bb7fe700 1 --2- 172.21.15.189:0/1525602334 >> v2:172.21.15.192:3300/0 conn(0x7f72bc15e2a0 0x7f72bc06c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:43.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.223+0000 7f72ba7fc700 1 -- 172.21.15.189:0/1525602334 >> v2:172.21.15.192:3300/0 conn(0x7f72bc15e2a0 msgr2=0x7f72bc06c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:43.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.223+0000 7f72ba7fc700 1 --2- 172.21.15.189:0/1525602334 >> v2:172.21.15.192:3300/0 conn(0x7f72bc15e2a0 0x7f72bc06c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:43.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.223+0000 7f72ba7fc700 1 -- 172.21.15.189:0/1525602334 >> v1:172.21.15.192:6789/0 conn(0x7f72bc0b2cd0 legacy=0x7f72bc0cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:43.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.223+0000 7f72ba7fc700 1 -- 172.21.15.189:0/1525602334 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72bc15d580 con 0x7f72bc15d040 2024-04-06T12:12:43.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.223+0000 7f72ba7fc700 1 --2- 172.21.15.189:0/1525602334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72bc15d040 0x7f72bc15dd60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f72bc0b2c60 tx=0x7f72a400be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed683bd2f6c6e90 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:43.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.223+0000 7f72b9ffb700 1 -- 172.21.15.189:0/1525602334 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a400dcf0 con 0x7f72bc15d040 2024-04-06T12:12:43.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.223+0000 7f72b9ffb700 1 -- 172.21.15.189:0/1525602334 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a400de50 con 0x7f72bc15d040 2024-04-06T12:12:43.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.223+0000 7f72b9ffb700 1 -- 172.21.15.189:0/1525602334 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a4004250 con 0x7f72bc15d040 2024-04-06T12:12:43.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.224+0000 7f72c1347700 1 -- 172.21.15.189:0/1525602334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72bc15d040 msgr2=0x7f72bc15dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:43.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.224+0000 7f72c1347700 1 --2- 172.21.15.189:0/1525602334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72bc15d040 0x7f72bc15dd60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f72bc0b2c60 tx=0x7f72a400be70 comp rx=0 tx=0).stop 2024-04-06T12:12:43.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.224+0000 7f72c1347700 1 -- 172.21.15.189:0/1525602334 shutdown_connections 2024-04-06T12:12:43.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.224+0000 7f72c1347700 1 --2- 172.21.15.189:0/1525602334 >> v2:172.21.15.192:3300/0 conn(0x7f72bc15e2a0 0x7f72bc06c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:43.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.224+0000 7f72c1347700 1 --2- 172.21.15.189:0/1525602334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72bc15d040 0x7f72bc15dd60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:43.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.224+0000 7f72c1347700 1 -- 172.21.15.189:0/1525602334 >> 172.21.15.189:0/1525602334 conn(0x7f72bc06a730 msgr2=0x7f72bc06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:43.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.224+0000 7f72c1347700 1 -- 172.21.15.189:0/1525602334 shutdown_connections 2024-04-06T12:12:43.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.224+0000 7f72c1347700 1 -- 172.21.15.189:0/1525602334 wait complete. 2024-04-06T12:12:43.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.225+0000 7f72c1347700 1 Processor -- start 2024-04-06T12:12:43.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.225+0000 7f72c1347700 1 -- start start 2024-04-06T12:12:43.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72c1347700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72bc0b2cd0 0x7f72bc142a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:43.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72c1347700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72bc15d040 0x7f72bc152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:43.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72c1347700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72bc1553a0 0x7f72bc155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:43.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72c1347700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72bc155d90 con 0x7f72bc0b2cd0 2024-04-06T12:12:43.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72c1347700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72bc15b410 con 0x7f72bc15d040 2024-04-06T12:12:43.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72c1347700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72bc152f80 con 0x7f72bc1553a0 2024-04-06T12:12:43.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72baffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72bc0b2cd0 0x7f72bc142a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:43.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72baffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72bc0b2cd0 0x7f72bc142a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46634/0 (socket says 172.21.15.189:46634) 2024-04-06T12:12:43.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72baffd700 1 -- 172.21.15.189:0/3366205264 learned_addr learned my addr 172.21.15.189:0/3366205264 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:43.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72ba7fc700 1 --2- 172.21.15.189:0/3366205264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72bc15d040 0x7f72bc152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:43.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72bb7fe700 1 --2- 172.21.15.189:0/3366205264 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72bc1553a0 0x7f72bc155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:43.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72baffd700 1 -- 172.21.15.189:0/3366205264 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72bc1553a0 msgr2=0x7f72bc155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:43.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72baffd700 1 --2- 172.21.15.189:0/3366205264 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72bc1553a0 0x7f72bc155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:43.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72baffd700 1 -- 172.21.15.189:0/3366205264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72bc15d040 msgr2=0x7f72bc152a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:43.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72baffd700 1 --2- 172.21.15.189:0/3366205264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72bc15d040 0x7f72bc152a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:43.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72baffd700 1 -- 172.21.15.189:0/3366205264 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a400b7f0 con 0x7f72bc0b2cd0 2024-04-06T12:12:43.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72baffd700 1 --2- 172.21.15.189:0/3366205264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72bc0b2cd0 0x7f72bc142a70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f72ac00c530 tx=0x7f72ac00c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:43.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72b37fe700 1 -- 172.21.15.189:0/3366205264 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac011790 con 0x7f72bc0b2cd0 2024-04-06T12:12:43.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72b37fe700 1 -- 172.21.15.189:0/3366205264 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ac0118f0 con 0x7f72bc0b2cd0 2024-04-06T12:12:43.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72c1347700 1 -- 172.21.15.189:0/3366205264 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72bc1f2ea0 con 0x7f72bc0b2cd0 2024-04-06T12:12:43.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.226+0000 7f72b37fe700 1 -- 172.21.15.189:0/3366205264 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac011ca0 con 0x7f72bc0b2cd0 2024-04-06T12:12:43.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.227+0000 7f72c1347700 1 -- 172.21.15.189:0/3366205264 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72bc15b0a0 con 0x7f72bc0b2cd0 2024-04-06T12:12:43.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.227+0000 7f72c1347700 1 -- 172.21.15.189:0/3366205264 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72bc073270 con 0x7f72bc0b2cd0 2024-04-06T12:12:43.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.228+0000 7f72b37fe700 1 -- 172.21.15.189:0/3366205264 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f72ac01c460 con 0x7f72bc0b2cd0 2024-04-06T12:12:43.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.228+0000 7f72b37fe700 1 --2- 172.21.15.189:0/3366205264 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f729c06ac90 0x7f729c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:43.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.228+0000 7f72b37fe700 1 -- 172.21.15.189:0/3366205264 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f72ac089240 con 0x7f72bc0b2cd0 2024-04-06T12:12:43.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.228+0000 7f72ba7fc700 1 --2- 172.21.15.189:0/3366205264 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f729c06ac90 0x7f729c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:43.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.229+0000 7f72ba7fc700 1 --2- 172.21.15.189:0/3366205264 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f729c06ac90 0x7f729c06d150 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f72a400b340 tx=0x7f72a4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:43.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.233+0000 7f72b37fe700 1 -- 172.21.15.189:0/3366205264 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f72ac0541c0 con 0x7f72bc0b2cd0 2024-04-06T12:12:43.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.471+0000 7f72c1347700 1 -- 172.21.15.189:0/3366205264 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f72bc074990 con 0x7f72bc0b2cd0 2024-04-06T12:12:43.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.472+0000 7f72b37fe700 1 -- 172.21.15.189:0/3366205264 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f72ac0577e0 con 0x7f72bc0b2cd0 2024-04-06T12:12:43.470 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-06T12:12:43.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.474+0000 7f72c1347700 1 -- 172.21.15.189:0/3366205264 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f729c06ac90 msgr2=0x7f729c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:43.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.474+0000 7f72c1347700 1 --2- 172.21.15.189:0/3366205264 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f729c06ac90 0x7f729c06d150 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f72a400b340 tx=0x7f72a4009040 comp rx=0 tx=0).stop 2024-04-06T12:12:43.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.474+0000 7f72c1347700 1 -- 172.21.15.189:0/3366205264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72bc0b2cd0 msgr2=0x7f72bc142a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:43.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.474+0000 7f72c1347700 1 --2- 172.21.15.189:0/3366205264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72bc0b2cd0 0x7f72bc142a70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f72ac00c530 tx=0x7f72ac00c840 comp rx=0 tx=0).stop 2024-04-06T12:12:43.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.474+0000 7f72c1347700 1 -- 172.21.15.189:0/3366205264 shutdown_connections 2024-04-06T12:12:43.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.474+0000 7f72c1347700 1 --2- 172.21.15.189:0/3366205264 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72bc1553a0 0x7f72bc155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:43.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.474+0000 7f72c1347700 1 --2- 172.21.15.189:0/3366205264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72bc0b2cd0 0x7f72bc142a70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:43.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.474+0000 7f72c1347700 1 --2- 172.21.15.189:0/3366205264 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f729c06ac90 0x7f729c06d150 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:43.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.474+0000 7f72c1347700 1 --2- 172.21.15.189:0/3366205264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72bc15d040 0x7f72bc152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:43.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.474+0000 7f72c1347700 1 -- 172.21.15.189:0/3366205264 >> 172.21.15.189:0/3366205264 conn(0x7f72bc06a730 msgr2=0x7f72bc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:43.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.474+0000 7f72c1347700 1 -- 172.21.15.189:0/3366205264 shutdown_connections 2024-04-06T12:12:43.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:43.474+0000 7f72c1347700 1 -- 172.21.15.189:0/3366205264 wait complete. 2024-04-06T12:12:43.485 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-06T12:12:43.485 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T12:12:43.485 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi192.front.sepia.ceph.com 2024-04-06T12:12:43.485 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-06T12:12:43.485 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-06T12:12:43.485 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-06T12:12:43.485 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-06T12:12:43.485 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T12:12:43.485 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-06T12:12:43.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:43.486 DEBUG:teuthology.orchestra.run.smithi192:> ip addr 2024-04-06T12:12:43.507 INFO:teuthology.orchestra.run.smithi192.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T12:12:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T12:12:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T12:12:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 ::1/128 scope host 2024-04-06T12:12:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:43.508 INFO:teuthology.orchestra.run.smithi192.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:12:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:d9:38:4e brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: altname enp7s0f0 2024-04-06T12:12:43.509 INFO:teuthology.orchestra.run.smithi192.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:12:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:d9:38:4f brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: altname enp7s0f1 2024-04-06T12:12:43.509 INFO:teuthology.orchestra.run.smithi192.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:12:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:88:80:48 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:43.509 INFO:teuthology.orchestra.run.smithi192.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T12:12:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:88:80:49 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: inet 172.21.15.192/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T12:12:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft 42308sec preferred_lft 42308sec 2024-04-06T12:12:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 fe80::41b7:5014:a8f3:694f/64 scope link noprefixroute 2024-04-06T12:12:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:43.510 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-06T12:12:43.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:43.511 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2024-04-06T12:12:43.511 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2024-04-06T12:12:43.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link add name ceph-brx type bridge 2024-04-06T12:12:43.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip addr flush dev ceph-brx 2024-04-06T12:12:43.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link set ceph-brx up 2024-04-06T12:12:43.512 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-06T12:12:43.512 DEBUG:teuthology.orchestra.run.smithi192:> ') 2024-04-06T12:12:43.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:43.723 DEBUG:teuthology.orchestra.run.smithi192:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T12:12:43.752 INFO:teuthology.orchestra.run.smithi192.stdout:1 2024-04-06T12:12:43.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:43.753 DEBUG:teuthology.orchestra.run.smithi192:> route 2024-04-06T12:12:43.833 INFO:teuthology.orchestra.run.smithi192.stdout:Kernel IP routing table 2024-04-06T12:12:43.834 INFO:teuthology.orchestra.run.smithi192.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T12:12:43.834 INFO:teuthology.orchestra.run.smithi192.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T12:12:43.834 INFO:teuthology.orchestra.run.smithi192.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T12:12:43.834 INFO:teuthology.orchestra.run.smithi192.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T12:12:43.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:43.835 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2024-04-06T12:12:43.835 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2024-04-06T12:12:43.835 DEBUG:teuthology.orchestra.run.smithi192:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T12:12:43.835 DEBUG:teuthology.orchestra.run.smithi192:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T12:12:43.835 DEBUG:teuthology.orchestra.run.smithi192:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T12:12:43.835 DEBUG:teuthology.orchestra.run.smithi192:> ') 2024-04-06T12:12:44.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:44.065 DEBUG:teuthology.orchestra.run.smithi192:> ip netns list 2024-04-06T12:12:44.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:44.082 DEBUG:teuthology.orchestra.run.smithi192:> ip netns list-id 2024-04-06T12:12:44.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:44.139 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2024-04-06T12:12:44.139 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2024-04-06T12:12:44.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-06T12:12:44.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-06T12:12:44.140 DEBUG:teuthology.orchestra.run.smithi192:> ') 2024-04-06T12:12:44.237 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-06T12:12:44.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:44.237 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2024-04-06T12:12:44.237 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2024-04-06T12:12:44.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-06T12:12:44.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-06T12:12:44.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-06T12:12:44.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-06T12:12:44.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-06T12:12:44.238 DEBUG:teuthology.orchestra.run.smithi192:> ') 2024-04-06T12:12:44.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:44.466 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2024-04-06T12:12:44.466 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2024-04-06T12:12:44.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link set brx.0 up 2024-04-06T12:12:44.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link set dev brx.0 master ceph-brx 2024-04-06T12:12:44.466 DEBUG:teuthology.orchestra.run.smithi192:> ') 2024-04-06T12:12:44.564 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:12:44.564 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-06T12:12:44.581 INFO:teuthology.orchestra.run.smithi192.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-06T12:12:44.582 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:12:44.582 DEBUG:teuthology.orchestra.run.smithi192:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-06T12:12:44.638 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T12:12:44.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:44.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-06T12:12:44.725 INFO:teuthology.orchestra.run.smithi192.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-06T12:12:44.737 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-06T12:12:44.737 INFO:teuthology.orchestra.run.smithi192.stdout:invalid new device string format 2024-04-06T12:12:44.802 INFO:teuthology.orchestra.run.smithi192.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-06T12:12:44.802 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-06T12:12:44.802 INFO:teuthology.orchestra.run.smithi192.stdout:invalid new device string format 2024-04-06T12:12:44.802 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:12:44.803 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:12:44.820 INFO:teuthology.orchestra.run.smithi192.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-06T12:12:44.821 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-06T12:12:44.821 INFO:teuthology.orchestra.run.smithi192.stdout:invalid new device string format 2024-04-06T12:12:44.821 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:12:44.821 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:12:44.821 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: resolved to: "172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790" 2024-04-06T12:12:44.821 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2024-04-06T12:12:44.821 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=e30041de-e9a3-4351-8ca7-29d7f3e3728c" will pass to kernel 2024-04-06T12:12:44.823 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T12:12:44.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:44.823 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-06T12:12:44.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:44.849 DEBUG:teuthology.orchestra.run.smithi192:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:12:44.922 INFO:teuthology.orchestra.run.smithi192.stdout:client4607 2024-04-06T12:12:44.923 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:12:44.923 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:12:44.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:44.981 DEBUG:teuthology.orchestra.run.smithi192:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:12:45.053 INFO:teuthology.orchestra.run.smithi192.stdout:client4607 2024-04-06T12:12:45.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:45.053 DEBUG:teuthology.orchestra.run.smithi192:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:12:45.126 INFO:teuthology.orchestra.run.smithi192.stdout:client4607 2024-04-06T12:12:45.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:45.127 DEBUG:teuthology.orchestra.run.smithi192:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e30041de-e9a3-4351-8ca7-29d7f3e3728c.client4607/status') 2024-04-06T12:12:45.208 INFO:teuthology.orchestra.run.smithi192.stdout:instance: client.4607 (3)192.168.0.1:0/376716714 2024-04-06T12:12:45.209 INFO:teuthology.orchestra.run.smithi192.stdout:blocklisted: false 2024-04-06T12:12:45.210 INFO:teuthology.orchestra.run.smithi192.stderr:0+1 records in 2024-04-06T12:12:45.210 INFO:teuthology.orchestra.run.smithi192.stderr:0+1 records out 2024-04-06T12:12:45.210 INFO:teuthology.orchestra.run.smithi192.stderr:68 bytes copied, 2.0193e-05 s, 3.4 MB/s 2024-04-06T12:12:45.210 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2024-04-06T12:12:45.210 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-06T12:12:45.211 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-06T12:12:45.211 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T12:12:45.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.348+0000 7fd405b05700 1 Processor -- start 2024-04-06T12:12:45.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.348+0000 7fd405b05700 1 -- start start 2024-04-06T12:12:45.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.349+0000 7fd405b05700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd40015d7b0 0x7fd4000b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:45.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.349+0000 7fd405b05700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd40015e500 0x7fd40006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:45.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.349+0000 7fd405b05700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4000b3990 con 0x7fd40006a730 2024-04-06T12:12:45.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.349+0000 7fd405b05700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd40006b0f0 con 0x7fd40015d7b0 2024-04-06T12:12:45.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.349+0000 7fd405b05700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4000b3060 con 0x7fd40015e500 2024-04-06T12:12:45.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.349+0000 7fd3ff7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd40015d7b0 0x7fd4000b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:45.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.349+0000 7fd3feffd700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fd40006a730 0x7fd40015ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:49316/0 (socket says 172.21.15.189:49316) 2024-04-06T12:12:45.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.349+0000 7fd3feffd700 1 -- 172.21.15.189:0/4140278710 learned_addr learned my addr 172.21.15.189:0/4140278710 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:45.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.349+0000 7fd3fffff700 1 --2- 172.21.15.189:0/4140278710 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd40015e500 0x7fd40006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:45.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.350+0000 7fd3ff7fe700 1 -- 172.21.15.189:0/4140278710 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd40015e500 msgr2=0x7fd40006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:45.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.350+0000 7fd3ff7fe700 1 --2- 172.21.15.189:0/4140278710 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd40015e500 0x7fd40006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:45.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.350+0000 7fd3ff7fe700 1 -- 172.21.15.189:0/4140278710 >> v1:172.21.15.192:6789/0 conn(0x7fd40006a730 legacy=0x7fd40015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:45.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.350+0000 7fd3ff7fe700 1 -- 172.21.15.189:0/4140278710 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4000b3340 con 0x7fd40015d7b0 2024-04-06T12:12:45.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.350+0000 7fd3ff7fe700 1 --2- 172.21.15.189:0/4140278710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd40015d7b0 0x7fd4000b3e20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0009840 tx=0x7fd3f0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4145c2e3911daf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:45.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.353+0000 7fd3fe7fc700 1 -- 172.21.15.189:0/4140278710 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f0011830 con 0x7fd40015d7b0 2024-04-06T12:12:45.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.353+0000 7fd3fe7fc700 1 -- 172.21.15.189:0/4140278710 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f0011990 con 0x7fd40015d7b0 2024-04-06T12:12:45.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.353+0000 7fd3fe7fc700 1 -- 172.21.15.189:0/4140278710 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f0011d20 con 0x7fd40015d7b0 2024-04-06T12:12:45.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.354+0000 7fd405b05700 1 -- 172.21.15.189:0/4140278710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd40015d7b0 msgr2=0x7fd4000b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:45.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.354+0000 7fd405b05700 1 --2- 172.21.15.189:0/4140278710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd40015d7b0 0x7fd4000b3e20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0009840 tx=0x7fd3f0009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.354+0000 7fd405b05700 1 -- 172.21.15.189:0/4140278710 shutdown_connections 2024-04-06T12:12:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.354+0000 7fd405b05700 1 --2- 172.21.15.189:0/4140278710 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd40015e500 0x7fd40006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.354+0000 7fd405b05700 1 --2- 172.21.15.189:0/4140278710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd40015d7b0 0x7fd4000b3e20 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.354+0000 7fd405b05700 1 -- 172.21.15.189:0/4140278710 >> 172.21.15.189:0/4140278710 conn(0x7fd4000d00b0 msgr2=0x7fd4000d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.354+0000 7fd405b05700 1 -- 172.21.15.189:0/4140278710 shutdown_connections 2024-04-06T12:12:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.354+0000 7fd405b05700 1 -- 172.21.15.189:0/4140278710 wait complete. 2024-04-06T12:12:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.355+0000 7fd405b05700 1 Processor -- start 2024-04-06T12:12:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.356+0000 7fd405b05700 1 -- start start 2024-04-06T12:12:45.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.356+0000 7fd405b05700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd40006a730 0x7fd4001f4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:45.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.356+0000 7fd405b05700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd40015d7b0 0x7fd400146a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:45.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.356+0000 7fd405b05700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd40015e500 0x7fd4000ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:45.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.356+0000 7fd405b05700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40015ad90 con 0x7fd40015e500 2024-04-06T12:12:45.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.356+0000 7fd405b05700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd400160490 con 0x7fd40015d7b0 2024-04-06T12:12:45.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.356+0000 7fd405b05700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40015afe0 con 0x7fd40006a730 2024-04-06T12:12:45.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.356+0000 7fd3fffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd40015e500 0x7fd4000ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:45.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.356+0000 7fd3fffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd40015e500 0x7fd4000ccd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46640/0 (socket says 172.21.15.189:46640) 2024-04-06T12:12:45.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.357+0000 7fd3fffff700 1 -- 172.21.15.189:0/3680248091 learned_addr learned my addr 172.21.15.189:0/3680248091 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:45.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.357+0000 7fd3feffd700 1 --2- 172.21.15.189:0/3680248091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd40015d7b0 0x7fd400146a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:45.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.357+0000 7fd3ff7fe700 1 --2- 172.21.15.189:0/3680248091 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd40006a730 0x7fd4001f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:45.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.357+0000 7fd3fffff700 1 -- 172.21.15.189:0/3680248091 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd40006a730 msgr2=0x7fd4001f4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:45.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.357+0000 7fd3fffff700 1 --2- 172.21.15.189:0/3680248091 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd40006a730 0x7fd4001f4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:45.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.357+0000 7fd3fffff700 1 -- 172.21.15.189:0/3680248091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd40015d7b0 msgr2=0x7fd400146a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:45.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.357+0000 7fd3fffff700 1 --2- 172.21.15.189:0/3680248091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd40015d7b0 0x7fd400146a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:45.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.357+0000 7fd3fffff700 1 -- 172.21.15.189:0/3680248091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f00094a0 con 0x7fd40015e500 2024-04-06T12:12:45.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.357+0000 7fd3fffff700 1 --2- 172.21.15.189:0/3680248091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd40015e500 0x7fd4000ccd90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800efc0 tx=0x7fd3f800c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:45.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.357+0000 7fd3f7fff700 1 -- 172.21.15.189:0/3680248091 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f800cad0 con 0x7fd40015e500 2024-04-06T12:12:45.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.357+0000 7fd3f7fff700 1 -- 172.21.15.189:0/3680248091 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f800f040 con 0x7fd40015e500 2024-04-06T12:12:45.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.357+0000 7fd405b05700 1 -- 172.21.15.189:0/3680248091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd40015b2c0 con 0x7fd40015e500 2024-04-06T12:12:45.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.357+0000 7fd3f7fff700 1 -- 172.21.15.189:0/3680248091 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f800ce70 con 0x7fd40015e500 2024-04-06T12:12:45.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.357+0000 7fd405b05700 1 -- 172.21.15.189:0/3680248091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd400146f80 con 0x7fd40015e500 2024-04-06T12:12:45.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.358+0000 7fd405b05700 1 -- 172.21.15.189:0/3680248091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd40015b7e0 con 0x7fd40015e500 2024-04-06T12:12:45.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.359+0000 7fd3f7fff700 1 -- 172.21.15.189:0/3680248091 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd3f8014480 con 0x7fd40015e500 2024-04-06T12:12:45.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.359+0000 7fd3f7fff700 1 --2- 172.21.15.189:0/3680248091 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3e006ac90 0x7fd3e006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:45.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.359+0000 7fd3f7fff700 1 -- 172.21.15.189:0/3680248091 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd3f8089d20 con 0x7fd40015e500 2024-04-06T12:12:45.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.359+0000 7fd3ff7fe700 1 --2- 172.21.15.189:0/3680248091 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3e006ac90 0x7fd3e006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:45.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.360+0000 7fd3ff7fe700 1 --2- 172.21.15.189:0/3680248091 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3e006ac90 0x7fd3e006d150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0000c00 tx=0x7fd3f0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:45.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.363+0000 7fd3f7fff700 1 -- 172.21.15.189:0/3680248091 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd40015b7e0 con 0x7fd40015e500 2024-04-06T12:12:45.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.606+0000 7fd405b05700 1 -- 172.21.15.189:0/3680248091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd40015b7e0 con 0x7fd40015e500 2024-04-06T12:12:45.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.607+0000 7fd3f7fff700 1 -- 172.21.15.189:0/3680248091 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7fd40015b7e0 con 0x7fd40015e500 2024-04-06T12:12:45.605 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-06T12:12:45.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.609+0000 7fd405b05700 1 -- 172.21.15.189:0/3680248091 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3e006ac90 msgr2=0x7fd3e006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:45.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.609+0000 7fd405b05700 1 --2- 172.21.15.189:0/3680248091 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3e006ac90 0x7fd3e006d150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0000c00 tx=0x7fd3f0008030 comp rx=0 tx=0).stop 2024-04-06T12:12:45.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.609+0000 7fd405b05700 1 -- 172.21.15.189:0/3680248091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd40015e500 msgr2=0x7fd4000ccd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:45.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.609+0000 7fd405b05700 1 --2- 172.21.15.189:0/3680248091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd40015e500 0x7fd4000ccd90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800efc0 tx=0x7fd3f800c0c0 comp rx=0 tx=0).stop 2024-04-06T12:12:45.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.609+0000 7fd405b05700 1 -- 172.21.15.189:0/3680248091 shutdown_connections 2024-04-06T12:12:45.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.609+0000 7fd405b05700 1 --2- 172.21.15.189:0/3680248091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd40015e500 0x7fd4000ccd90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:45.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.609+0000 7fd405b05700 1 --2- 172.21.15.189:0/3680248091 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd40006a730 0x7fd4001f4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:45.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.609+0000 7fd405b05700 1 --2- 172.21.15.189:0/3680248091 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3e006ac90 0x7fd3e006d150 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:45.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.609+0000 7fd405b05700 1 --2- 172.21.15.189:0/3680248091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd40015d7b0 0x7fd400146a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:45.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.609+0000 7fd405b05700 1 -- 172.21.15.189:0/3680248091 >> 172.21.15.189:0/3680248091 conn(0x7fd4000d00b0 msgr2=0x7fd40006eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:45.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.609+0000 7fd405b05700 1 -- 172.21.15.189:0/3680248091 shutdown_connections 2024-04-06T12:12:45.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:45.609+0000 7fd405b05700 1 -- 172.21.15.189:0/3680248091 wait complete. 2024-04-06T12:12:45.619 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-06T12:12:45.619 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T12:12:45.619 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi192.front.sepia.ceph.com 2024-04-06T12:12:45.619 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-06T12:12:45.619 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-06T12:12:45.619 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-06T12:12:45.620 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-06T12:12:45.620 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T12:12:45.620 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-06T12:12:45.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:45.620 DEBUG:teuthology.orchestra.run.smithi192:> ip addr 2024-04-06T12:12:45.639 INFO:teuthology.orchestra.run.smithi192.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T12:12:45.639 INFO:teuthology.orchestra.run.smithi192.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T12:12:45.639 INFO:teuthology.orchestra.run.smithi192.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T12:12:45.639 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:45.639 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 ::1/128 scope host 2024-04-06T12:12:45.639 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:45.639 INFO:teuthology.orchestra.run.smithi192.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:12:45.639 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:d9:38:4e brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:45.639 INFO:teuthology.orchestra.run.smithi192.stdout: altname enp7s0f0 2024-04-06T12:12:45.639 INFO:teuthology.orchestra.run.smithi192.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:12:45.639 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:d9:38:4f brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:45.640 INFO:teuthology.orchestra.run.smithi192.stdout: altname enp7s0f1 2024-04-06T12:12:45.640 INFO:teuthology.orchestra.run.smithi192.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:12:45.640 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:88:80:48 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:45.640 INFO:teuthology.orchestra.run.smithi192.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T12:12:45.640 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:88:80:49 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:45.640 INFO:teuthology.orchestra.run.smithi192.stdout: inet 172.21.15.192/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T12:12:45.640 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft 42306sec preferred_lft 42306sec 2024-04-06T12:12:45.640 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 fe80::41b7:5014:a8f3:694f/64 scope link noprefixroute 2024-04-06T12:12:45.640 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:45.640 INFO:teuthology.orchestra.run.smithi192.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T12:12:45.640 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether d2:1d:4b:86:e6:c2 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:12:45.640 INFO:teuthology.orchestra.run.smithi192.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-06T12:12:45.640 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:45.640 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 fe80::c4cc:57ff:fe67:9e8/64 scope link tentative 2024-04-06T12:12:45.641 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:45.641 INFO:teuthology.orchestra.run.smithi192.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T12:12:45.641 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether d2:1d:4b:86:e6:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-06T12:12:45.641 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 fe80::d01d:4bff:fe86:e6c2/64 scope link tentative 2024-04-06T12:12:45.641 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:45.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:45.641 DEBUG:teuthology.orchestra.run.smithi192:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T12:12:45.705 INFO:teuthology.orchestra.run.smithi192.stdout:1 2024-04-06T12:12:45.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:45.707 DEBUG:teuthology.orchestra.run.smithi192:> route 2024-04-06T12:12:45.766 INFO:teuthology.orchestra.run.smithi192.stdout:Kernel IP routing table 2024-04-06T12:12:45.766 INFO:teuthology.orchestra.run.smithi192.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T12:12:45.767 INFO:teuthology.orchestra.run.smithi192.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T12:12:45.767 INFO:teuthology.orchestra.run.smithi192.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T12:12:45.767 INFO:teuthology.orchestra.run.smithi192.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T12:12:45.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:45.767 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2024-04-06T12:12:45.767 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2024-04-06T12:12:45.767 DEBUG:teuthology.orchestra.run.smithi192:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T12:12:45.767 DEBUG:teuthology.orchestra.run.smithi192:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T12:12:45.767 DEBUG:teuthology.orchestra.run.smithi192:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T12:12:45.767 DEBUG:teuthology.orchestra.run.smithi192:> ') 2024-04-06T12:12:45.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:45.873 DEBUG:teuthology.orchestra.run.smithi192:> ip netns list 2024-04-06T12:12:45.890 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-06T12:12:45.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:45.890 DEBUG:teuthology.orchestra.run.smithi192:> ip netns list-id 2024-04-06T12:12:45.946 INFO:teuthology.orchestra.run.smithi192.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-06T12:12:45.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:45.947 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2024-04-06T12:12:45.947 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2024-04-06T12:12:45.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-06T12:12:45.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-06T12:12:45.947 DEBUG:teuthology.orchestra.run.smithi192:> ') 2024-04-06T12:12:46.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:46.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-06T12:12:46.077 INFO:teuthology.orchestra.run.smithi192.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T12:12:46.077 INFO:teuthology.orchestra.run.smithi192.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T12:12:46.077 INFO:teuthology.orchestra.run.smithi192.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T12:12:46.077 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:46.077 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 ::1/128 scope host 2024-04-06T12:12:46.078 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:46.078 INFO:teuthology.orchestra.run.smithi192.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T12:12:46.078 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 7a:ca:e9:14:4f:24 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-06T12:12:46.078 INFO:teuthology.orchestra.run.smithi192.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-06T12:12:46.078 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:46.078 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 fe80::78ca:e9ff:fe14:4f24/64 scope link 2024-04-06T12:12:46.078 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:46.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:46.094 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-06T12:12:46.174 INFO:teuthology.orchestra.run.smithi192.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T12:12:46.174 INFO:teuthology.orchestra.run.smithi192.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T12:12:46.174 INFO:teuthology.orchestra.run.smithi192.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T12:12:46.174 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:46.174 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 ::1/128 scope host 2024-04-06T12:12:46.174 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:46.174 INFO:teuthology.orchestra.run.smithi192.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T12:12:46.174 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 7a:ca:e9:14:4f:24 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-06T12:12:46.175 INFO:teuthology.orchestra.run.smithi192.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-06T12:12:46.175 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:46.175 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 fe80::78ca:e9ff:fe14:4f24/64 scope link 2024-04-06T12:12:46.175 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:12:46.192 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-06T12:12:46.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:46.193 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2024-04-06T12:12:46.193 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2024-04-06T12:12:46.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-06T12:12:46.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-06T12:12:46.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-06T12:12:46.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-06T12:12:46.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-06T12:12:46.194 DEBUG:teuthology.orchestra.run.smithi192:> ') 2024-04-06T12:12:46.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:46.471 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2024-04-06T12:12:46.471 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2024-04-06T12:12:46.471 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link set brx.1 up 2024-04-06T12:12:46.471 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link set dev brx.1 master ceph-brx 2024-04-06T12:12:46.471 DEBUG:teuthology.orchestra.run.smithi192:> ') 2024-04-06T12:12:46.569 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:12:46.569 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-06T12:12:46.586 INFO:teuthology.orchestra.run.smithi192.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-06T12:12:46.586 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:12:46.586 DEBUG:teuthology.orchestra.run.smithi192:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-06T12:12:46.642 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T12:12:46.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:46.643 DEBUG:teuthology.orchestra.run.smithi192:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-04-06T12:12:46.728 INFO:teuthology.orchestra.run.smithi192.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-04-06T12:12:46.728 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-04-06T12:12:46.728 INFO:teuthology.orchestra.run.smithi192.stdout:invalid new device string format 2024-04-06T12:12:46.737 INFO:teuthology.orchestra.run.smithi192.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-04-06T12:12:46.737 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-04-06T12:12:46.737 INFO:teuthology.orchestra.run.smithi192.stdout:invalid new device string format 2024-04-06T12:12:46.737 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:12:46.737 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:12:46.756 INFO:teuthology.orchestra.run.smithi192.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-04-06T12:12:46.756 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-04-06T12:12:46.756 INFO:teuthology.orchestra.run.smithi192.stdout:invalid new device string format 2024-04-06T12:12:46.756 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:12:46.756 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:12:46.756 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: resolved to: "172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790" 2024-04-06T12:12:46.756 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2024-04-06T12:12:46.756 INFO:teuthology.orchestra.run.smithi192.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=e30041de-e9a3-4351-8ca7-29d7f3e3728c" will pass to kernel 2024-04-06T12:12:46.758 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T12:12:46.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:46.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-06T12:12:46.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:46.785 DEBUG:teuthology.orchestra.run.smithi192:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:12:46.857 INFO:teuthology.orchestra.run.smithi192.stdout:client4753 2024-04-06T12:12:46.857 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:12:46.857 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:12:46.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:46.915 DEBUG:teuthology.orchestra.run.smithi192:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:12:46.987 INFO:teuthology.orchestra.run.smithi192.stdout:client4753 2024-04-06T12:12:46.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:46.987 DEBUG:teuthology.orchestra.run.smithi192:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:12:47.059 INFO:teuthology.orchestra.run.smithi192.stdout:client4753 2024-04-06T12:12:47.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:47.060 DEBUG:teuthology.orchestra.run.smithi192:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e30041de-e9a3-4351-8ca7-29d7f3e3728c.client4753/status') 2024-04-06T12:12:47.140 INFO:teuthology.orchestra.run.smithi192.stdout:instance: client.4753 (3)192.168.0.2:0/1298705304 2024-04-06T12:12:47.140 INFO:teuthology.orchestra.run.smithi192.stdout:blocklisted: false 2024-04-06T12:12:47.141 INFO:teuthology.orchestra.run.smithi192.stderr:0+1 records in 2024-04-06T12:12:47.141 INFO:teuthology.orchestra.run.smithi192.stderr:0+1 records out 2024-04-06T12:12:47.141 INFO:teuthology.orchestra.run.smithi192.stderr:69 bytes copied, 2.0596e-05 s, 3.4 MB/s 2024-04-06T12:12:47.142 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-06T12:12:47.185 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:47.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.319+0000 7f27c3f78700 1 Processor -- start 2024-04-06T12:12:47.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.320+0000 7f27c3f78700 1 -- start start 2024-04-06T12:12:47.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.320+0000 7f27c3f78700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f27bc15d040 0x7f27bc15dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:47.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.320+0000 7f27c3f78700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27bc15e2a0 0x7f27bc06c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:47.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.320+0000 7f27c3f78700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc15d8d0 con 0x7f27bc0b2cd0 2024-04-06T12:12:47.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.320+0000 7f27c3f78700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27bc0cba40 con 0x7f27bc15d040 2024-04-06T12:12:47.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.320+0000 7f27c3f78700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc15d580 con 0x7f27bc15e2a0 2024-04-06T12:12:47.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.321+0000 7f27c1d14700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f27bc15d040 0x7f27bc15dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:47.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.321+0000 7f27c2515700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f27bc0b2cd0 0x7f27bc0cc1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44724/0 (socket says 172.21.15.189:44724) 2024-04-06T12:12:47.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.321+0000 7f27c2515700 1 -- 172.21.15.189:0/3156900677 learned_addr learned my addr 172.21.15.189:0/3156900677 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:47.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.321+0000 7f27c1513700 1 --2- 172.21.15.189:0/3156900677 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27bc15e2a0 0x7f27bc06c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:47.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.321+0000 7f27c1d14700 1 -- 172.21.15.189:0/3156900677 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27bc15e2a0 msgr2=0x7f27bc06c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:47.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.321+0000 7f27c1d14700 1 --2- 172.21.15.189:0/3156900677 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27bc15e2a0 0x7f27bc06c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:47.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.321+0000 7f27c1d14700 1 -- 172.21.15.189:0/3156900677 >> v1:172.21.15.189:6789/0 conn(0x7f27bc0b2cd0 legacy=0x7f27bc0cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:47.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.321+0000 7f27c1d14700 1 -- 172.21.15.189:0/3156900677 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc0b3240 con 0x7f27bc15d040 2024-04-06T12:12:47.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.321+0000 7f27c0d12700 1 -- 172.21.15.189:0/3156900677 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581039719 0 0) 0x7f27bc15d8d0 con 0x7f27bc0b2cd0 2024-04-06T12:12:47.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.321+0000 7f27c1d14700 1 --2- 172.21.15.189:0/3156900677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27bc15d040 0x7f27bc15dd60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f27b0009840 tx=0x7f27b0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84948dbe57b0187 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:47.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.321+0000 7f27c0d12700 1 -- 172.21.15.189:0/3156900677 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b0011830 con 0x7f27bc15d040 2024-04-06T12:12:47.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.321+0000 7f27c0d12700 1 -- 172.21.15.189:0/3156900677 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b0011990 con 0x7f27bc15d040 2024-04-06T12:12:47.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.321+0000 7f27c0d12700 1 -- 172.21.15.189:0/3156900677 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b0011d20 con 0x7f27bc15d040 2024-04-06T12:12:47.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.322+0000 7f27c3f78700 1 -- 172.21.15.189:0/3156900677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27bc15d040 msgr2=0x7f27bc15dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:47.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.322+0000 7f27c3f78700 1 --2- 172.21.15.189:0/3156900677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27bc15d040 0x7f27bc15dd60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f27b0009840 tx=0x7f27b0009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:47.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.322+0000 7f27c3f78700 1 -- 172.21.15.189:0/3156900677 shutdown_connections 2024-04-06T12:12:47.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.322+0000 7f27c3f78700 1 --2- 172.21.15.189:0/3156900677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27bc15d040 0x7f27bc15dd60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:47.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.322+0000 7f27c3f78700 1 --2- 172.21.15.189:0/3156900677 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27bc15e2a0 0x7f27bc06c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:47.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.322+0000 7f27c3f78700 1 -- 172.21.15.189:0/3156900677 >> 172.21.15.189:0/3156900677 conn(0x7f27bc06a730 msgr2=0x7f27bc06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:47.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.322+0000 7f27c3f78700 1 -- 172.21.15.189:0/3156900677 shutdown_connections 2024-04-06T12:12:47.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.322+0000 7f27c3f78700 1 -- 172.21.15.189:0/3156900677 wait complete. 2024-04-06T12:12:47.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.324+0000 7f27c3f78700 1 Processor -- start 2024-04-06T12:12:47.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.324+0000 7f27c3f78700 1 -- start start 2024-04-06T12:12:47.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.324+0000 7f27c3f78700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27bc0b2cd0 0x7f27bc147a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:47.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.324+0000 7f27c3f78700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27bc15d040 0x7f27bc1e33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:47.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.324+0000 7f27c3f78700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27bc0b0dd0 0x7f27bc0b1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:47.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.324+0000 7f27c3f78700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0b17c0 con 0x7f27bc0b0dd0 2024-04-06T12:12:47.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.324+0000 7f27c3f78700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0cc1e0 con 0x7f27bc15d040 2024-04-06T12:12:47.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.324+0000 7f27c3f78700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0cc4c0 con 0x7f27bc0b2cd0 2024-04-06T12:12:47.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.325+0000 7f27c2515700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27bc0b0dd0 0x7f27bc0b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:47.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.325+0000 7f27c2515700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27bc0b0dd0 0x7f27bc0b1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46662/0 (socket says 172.21.15.189:46662) 2024-04-06T12:12:47.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.325+0000 7f27c2515700 1 -- 172.21.15.189:0/1917845214 learned_addr learned my addr 172.21.15.189:0/1917845214 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:47.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.325+0000 7f27c1d14700 1 --2- 172.21.15.189:0/1917845214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27bc0b2cd0 0x7f27bc147a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:47.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.325+0000 7f27c1513700 1 --2- 172.21.15.189:0/1917845214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27bc15d040 0x7f27bc1e33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:47.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.325+0000 7f27c2515700 1 -- 172.21.15.189:0/1917845214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27bc0b2cd0 msgr2=0x7f27bc147a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:47.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.325+0000 7f27c2515700 1 --2- 172.21.15.189:0/1917845214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27bc0b2cd0 0x7f27bc147a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:47.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.325+0000 7f27c2515700 1 -- 172.21.15.189:0/1917845214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27bc15d040 msgr2=0x7f27bc1e33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:47.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.325+0000 7f27c2515700 1 --2- 172.21.15.189:0/1917845214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27bc15d040 0x7f27bc1e33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:47.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.325+0000 7f27c2515700 1 -- 172.21.15.189:0/1917845214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b00094a0 con 0x7f27bc0b0dd0 2024-04-06T12:12:47.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.325+0000 7f27c2515700 1 --2- 172.21.15.189:0/1917845214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27bc0b0dd0 0x7f27bc0b1280 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f27b4002230 tx=0x7f27b400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:47.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.325+0000 7f27ae7fc700 1 -- 172.21.15.189:0/1917845214 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b400aac0 con 0x7f27bc0b0dd0 2024-04-06T12:12:47.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.325+0000 7f27ae7fc700 1 -- 172.21.15.189:0/1917845214 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b400ac40 con 0x7f27bc0b0dd0 2024-04-06T12:12:47.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.325+0000 7f27c3f78700 1 -- 172.21.15.189:0/1917845214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27bc0cc7a0 con 0x7f27bc0b0dd0 2024-04-06T12:12:47.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.325+0000 7f27ae7fc700 1 -- 172.21.15.189:0/1917845214 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b40070d0 con 0x7f27bc0b0dd0 2024-04-06T12:12:47.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.325+0000 7f27c3f78700 1 -- 172.21.15.189:0/1917845214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27bc0ccd60 con 0x7f27bc0b0dd0 2024-04-06T12:12:47.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.326+0000 7f27c3f78700 1 -- 172.21.15.189:0/1917845214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27bc073270 con 0x7f27bc0b0dd0 2024-04-06T12:12:47.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.327+0000 7f27ae7fc700 1 -- 172.21.15.189:0/1917845214 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f27b4007250 con 0x7f27bc0b0dd0 2024-04-06T12:12:47.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.327+0000 7f27ae7fc700 1 --2- 172.21.15.189:0/1917845214 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f27a006ac90 0x7f27a006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:47.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.327+0000 7f27ae7fc700 1 -- 172.21.15.189:0/1917845214 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f27b4089320 con 0x7f27bc0b0dd0 2024-04-06T12:12:47.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.327+0000 7f27c1d14700 1 --2- 172.21.15.189:0/1917845214 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f27a006ac90 0x7f27a006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:47.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.328+0000 7f27c1d14700 1 --2- 172.21.15.189:0/1917845214 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f27a006ac90 0x7f27a006d150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f27b00090b0 tx=0x7f27b0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:47.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.332+0000 7f27ae7fc700 1 -- 172.21.15.189:0/1917845214 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f27b40542a0 con 0x7f27bc0b0dd0 2024-04-06T12:12:47.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.530+0000 7f27c3f78700 1 -- 172.21.15.189:0/1917845214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27bc074990 con 0x7f27bc0b0dd0 2024-04-06T12:12:47.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.532+0000 7f27ae7fc700 1 -- 172.21.15.189:0/1917845214 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f27b40578c0 con 0x7f27bc0b0dd0 2024-04-06T12:12:47.530 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:47.530 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:08.649170+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:47.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.534+0000 7f27c3f78700 1 -- 172.21.15.189:0/1917845214 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f27a006ac90 msgr2=0x7f27a006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:47.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.534+0000 7f27c3f78700 1 --2- 172.21.15.189:0/1917845214 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f27a006ac90 0x7f27a006d150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f27b00090b0 tx=0x7f27b0008030 comp rx=0 tx=0).stop 2024-04-06T12:12:47.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.534+0000 7f27c3f78700 1 -- 172.21.15.189:0/1917845214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27bc0b0dd0 msgr2=0x7f27bc0b1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:47.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.534+0000 7f27c3f78700 1 --2- 172.21.15.189:0/1917845214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27bc0b0dd0 0x7f27bc0b1280 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f27b4002230 tx=0x7f27b400cf50 comp rx=0 tx=0).stop 2024-04-06T12:12:47.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.534+0000 7f27c3f78700 1 -- 172.21.15.189:0/1917845214 shutdown_connections 2024-04-06T12:12:47.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.534+0000 7f27c3f78700 1 --2- 172.21.15.189:0/1917845214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27bc0b0dd0 0x7f27bc0b1280 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:47.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.534+0000 7f27c3f78700 1 --2- 172.21.15.189:0/1917845214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27bc0b2cd0 0x7f27bc147a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:47.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.534+0000 7f27c3f78700 1 --2- 172.21.15.189:0/1917845214 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f27a006ac90 0x7f27a006d150 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:47.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.534+0000 7f27c3f78700 1 --2- 172.21.15.189:0/1917845214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27bc15d040 0x7f27bc1e33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:47.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.534+0000 7f27c3f78700 1 -- 172.21.15.189:0/1917845214 >> 172.21.15.189:0/1917845214 conn(0x7f27bc06a730 msgr2=0x7f27bc156a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:47.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.534+0000 7f27c3f78700 1 -- 172.21.15.189:0/1917845214 shutdown_connections 2024-04-06T12:12:47.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.534+0000 7f27c3f78700 1 -- 172.21.15.189:0/1917845214 wait complete. 2024-04-06T12:12:47.545 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:12:02.280866+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:12:03.156593+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-06T12:12:06.237184+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-06T12:12:47.545 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:12:47.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.674+0000 7f1be6399700 1 Processor -- start 2024-04-06T12:12:47.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.674+0000 7f1be6399700 1 -- start start 2024-04-06T12:12:47.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.675+0000 7f1be6399700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1be015a550 0x7f1be00cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:47.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.675+0000 7f1be6399700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1be00cf900 0x7f1be00b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:47.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.675+0000 7f1be6399700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1be00cdbe0 con 0x7f1be00b4ba0 2024-04-06T12:12:47.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.675+0000 7f1be6399700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1be00b5010 con 0x7f1be015a550 2024-04-06T12:12:47.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.675+0000 7f1be6399700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be015a940 con 0x7f1be00cf900 2024-04-06T12:12:47.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.675+0000 7f1be4936700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1be00b4ba0 0x7f1be006eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44738/0 (socket says 172.21.15.189:44738) 2024-04-06T12:12:47.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.675+0000 7f1be4936700 1 -- 172.21.15.189:0/239986624 learned_addr learned my addr 172.21.15.189:0/239986624 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:47.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.675+0000 7f1bdf7fe700 1 --2- 172.21.15.189:0/239986624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1be00cf900 0x7f1be00b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:47.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.675+0000 7f1bdffff700 1 --2- 172.21.15.189:0/239986624 >> v2:172.21.15.192:3300/0 conn(0x7f1be015a550 0x7f1be00cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:47.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.675+0000 7f1bdeffd700 1 -- 172.21.15.189:0/239986624 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 896603997 0 0) 0x7f1be00cdbe0 con 0x7f1be00b4ba0 2024-04-06T12:12:47.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.675+0000 7f1bdeffd700 1 -- 172.21.15.189:0/239986624 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc80035b0 con 0x7f1be00b4ba0 2024-04-06T12:12:47.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.675+0000 7f1bdf7fe700 1 -- 172.21.15.189:0/239986624 >> v2:172.21.15.192:3300/0 conn(0x7f1be015a550 msgr2=0x7f1be00cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:47.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.675+0000 7f1bdf7fe700 1 --2- 172.21.15.189:0/239986624 >> v2:172.21.15.192:3300/0 conn(0x7f1be015a550 0x7f1be00cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:47.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.675+0000 7f1bdf7fe700 1 -- 172.21.15.189:0/239986624 >> v1:172.21.15.189:6789/0 conn(0x7f1be00b4ba0 legacy=0x7f1be006eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:47.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.675+0000 7f1bdf7fe700 1 -- 172.21.15.189:0/239986624 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be015ac00 con 0x7f1be00cf900 2024-04-06T12:12:47.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.676+0000 7f1bdf7fe700 1 --2- 172.21.15.189:0/239986624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1be00cf900 0x7f1be00b4660 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0009840 tx=0x7f1bd0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38f7688f920665b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:47.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.676+0000 7f1bdeffd700 1 -- 172.21.15.189:0/239986624 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd0011830 con 0x7f1be00cf900 2024-04-06T12:12:47.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.676+0000 7f1bdeffd700 1 -- 172.21.15.189:0/239986624 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd0011990 con 0x7f1be00cf900 2024-04-06T12:12:47.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.676+0000 7f1bdeffd700 1 -- 172.21.15.189:0/239986624 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd0011d00 con 0x7f1be00cf900 2024-04-06T12:12:47.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.676+0000 7f1be6399700 1 -- 172.21.15.189:0/239986624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1be00cf900 msgr2=0x7f1be00b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:47.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.676+0000 7f1be6399700 1 --2- 172.21.15.189:0/239986624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1be00cf900 0x7f1be00b4660 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0009840 tx=0x7f1bd0009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:47.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.676+0000 7f1be6399700 1 -- 172.21.15.189:0/239986624 shutdown_connections 2024-04-06T12:12:47.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.676+0000 7f1be6399700 1 --2- 172.21.15.189:0/239986624 >> v2:172.21.15.192:3300/0 conn(0x7f1be015a550 0x7f1be00cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:47.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.676+0000 7f1be6399700 1 --2- 172.21.15.189:0/239986624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1be00cf900 0x7f1be00b4660 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:47.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.676+0000 7f1be6399700 1 -- 172.21.15.189:0/239986624 >> 172.21.15.189:0/239986624 conn(0x7f1be00d00b0 msgr2=0x7f1be00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:47.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.676+0000 7f1be6399700 1 -- 172.21.15.189:0/239986624 shutdown_connections 2024-04-06T12:12:47.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.676+0000 7f1be6399700 1 -- 172.21.15.189:0/239986624 wait complete. 2024-04-06T12:12:47.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.677+0000 7f1be6399700 1 Processor -- start 2024-04-06T12:12:47.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.678+0000 7f1be6399700 1 -- start start 2024-04-06T12:12:47.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.678+0000 7f1be6399700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be00b4ba0 0x7f1be014e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:47.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.678+0000 7f1be6399700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1be00cf900 0x7f1be0154890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.678+0000 7f1be6399700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1be0147ac0 0x7f1be014c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.678+0000 7f1be6399700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be0154120 con 0x7f1be00b4ba0 2024-04-06T12:12:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.678+0000 7f1be6399700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be0154dd0 con 0x7f1be00cf900 2024-04-06T12:12:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.678+0000 7f1be6399700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be014f290 con 0x7f1be0147ac0 2024-04-06T12:12:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.679+0000 7f1bdffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be00b4ba0 0x7f1be014e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.679+0000 7f1bdffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be00b4ba0 0x7f1be014e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46678/0 (socket says 172.21.15.189:46678) 2024-04-06T12:12:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.679+0000 7f1bdffff700 1 -- 172.21.15.189:0/1705027010 learned_addr learned my addr 172.21.15.189:0/1705027010 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.679+0000 7f1be4936700 1 --2- 172.21.15.189:0/1705027010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1be0147ac0 0x7f1be014c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.679+0000 7f1bdf7fe700 1 --2- 172.21.15.189:0/1705027010 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1be00cf900 0x7f1be0154890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.679+0000 7f1bdffff700 1 -- 172.21.15.189:0/1705027010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1be0147ac0 msgr2=0x7f1be014c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.679+0000 7f1bdffff700 1 --2- 172.21.15.189:0/1705027010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1be0147ac0 0x7f1be014c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.679+0000 7f1bdffff700 1 -- 172.21.15.189:0/1705027010 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1be00cf900 msgr2=0x7f1be0154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.679+0000 7f1bdffff700 1 --2- 172.21.15.189:0/1705027010 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1be00cf900 0x7f1be0154890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.679+0000 7f1bdffff700 1 -- 172.21.15.189:0/1705027010 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd00094a0 con 0x7f1be00b4ba0 2024-04-06T12:12:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.679+0000 7f1bdffff700 1 --2- 172.21.15.189:0/1705027010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be00b4ba0 0x7f1be014e480 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1bd400e9b0 tx=0x7f1bd400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:47.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.679+0000 7f1bdcff9700 1 -- 172.21.15.189:0/1705027010 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4009710 con 0x7f1be00b4ba0 2024-04-06T12:12:47.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.679+0000 7f1bdcff9700 1 -- 172.21.15.189:0/1705027010 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd400ee50 con 0x7f1be00b4ba0 2024-04-06T12:12:47.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.679+0000 7f1be6399700 1 -- 172.21.15.189:0/1705027010 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be014f570 con 0x7f1be00b4ba0 2024-04-06T12:12:47.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.679+0000 7f1bdcff9700 1 -- 172.21.15.189:0/1705027010 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd400ca50 con 0x7f1be00b4ba0 2024-04-06T12:12:47.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.679+0000 7f1be6399700 1 -- 172.21.15.189:0/1705027010 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be0154370 con 0x7f1be00b4ba0 2024-04-06T12:12:47.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.680+0000 7f1be6399700 1 -- 172.21.15.189:0/1705027010 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1be0067bb0 con 0x7f1be00b4ba0 2024-04-06T12:12:47.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.681+0000 7f1bdcff9700 1 -- 172.21.15.189:0/1705027010 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1bd400cbb0 con 0x7f1be00b4ba0 2024-04-06T12:12:47.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.681+0000 7f1bdcff9700 1 --2- 172.21.15.189:0/1705027010 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1bcc06acc0 0x7f1bcc06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:47.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.681+0000 7f1bdf7fe700 1 --2- 172.21.15.189:0/1705027010 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1bcc06acc0 0x7f1bcc06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:47.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.681+0000 7f1bdcff9700 1 -- 172.21.15.189:0/1705027010 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1bd4089500 con 0x7f1be00b4ba0 2024-04-06T12:12:47.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.682+0000 7f1bdf7fe700 1 --2- 172.21.15.189:0/1705027010 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1bcc06acc0 0x7f1bcc06d180 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0000c00 tx=0x7f1bd001c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:47.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.686+0000 7f1bdcff9700 1 -- 172.21.15.189:0/1705027010 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1bd4054480 con 0x7f1be00b4ba0 2024-04-06T12:12:47.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.877+0000 7f1be6399700 1 -- 172.21.15.189:0/1705027010 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1be0154560 con 0x7f1be00b4ba0 2024-04-06T12:12:47.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.877+0000 7f1bdcff9700 1 -- 172.21.15.189:0/1705027010 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1bd400ce60 con 0x7f1be00b4ba0 2024-04-06T12:12:47.875 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:12:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.879+0000 7f1be6399700 1 -- 172.21.15.189:0/1705027010 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1bcc06acc0 msgr2=0x7f1bcc06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.879+0000 7f1be6399700 1 --2- 172.21.15.189:0/1705027010 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1bcc06acc0 0x7f1bcc06d180 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0000c00 tx=0x7f1bd001c040 comp rx=0 tx=0).stop 2024-04-06T12:12:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.879+0000 7f1be6399700 1 -- 172.21.15.189:0/1705027010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be00b4ba0 msgr2=0x7f1be014e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.879+0000 7f1be6399700 1 --2- 172.21.15.189:0/1705027010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be00b4ba0 0x7f1be014e480 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1bd400e9b0 tx=0x7f1bd400ecc0 comp rx=0 tx=0).stop 2024-04-06T12:12:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.879+0000 7f1be6399700 1 -- 172.21.15.189:0/1705027010 shutdown_connections 2024-04-06T12:12:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.879+0000 7f1be6399700 1 --2- 172.21.15.189:0/1705027010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1be0147ac0 0x7f1be014c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.879+0000 7f1be6399700 1 --2- 172.21.15.189:0/1705027010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be00b4ba0 0x7f1be014e480 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.879+0000 7f1be6399700 1 --2- 172.21.15.189:0/1705027010 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1bcc06acc0 0x7f1bcc06d180 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.879+0000 7f1be6399700 1 --2- 172.21.15.189:0/1705027010 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1be00cf900 0x7f1be0154890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.879+0000 7f1be6399700 1 -- 172.21.15.189:0/1705027010 >> 172.21.15.189:0/1705027010 conn(0x7f1be00d00b0 msgr2=0x7f1be006d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.879+0000 7f1be6399700 1 -- 172.21.15.189:0/1705027010 shutdown_connections 2024-04-06T12:12:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:47.879+0000 7f1be6399700 1 -- 172.21.15.189:0/1705027010 wait complete. 2024-04-06T12:12:47.889 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:12:48.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.016+0000 7ff600c6b700 1 Processor -- start 2024-04-06T12:12:48.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.016+0000 7ff600c6b700 1 -- start start 2024-04-06T12:12:48.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.017+0000 7ff600c6b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff5fc06ae40 0x7ff5fc0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.017+0000 7ff600c6b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff5fc15ddf0 0x7ff5fc15e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.017+0000 7ff600c6b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5fc0b3990 con 0x7ff5fc15d7b0 2024-04-06T12:12:48.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.017+0000 7ff600c6b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5fc0b3060 con 0x7ff5fc15ddf0 2024-04-06T12:12:48.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.017+0000 7ff600c6b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5fc0b3340 con 0x7ff5fc06ae40 2024-04-06T12:12:48.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.017+0000 7ff5fad9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff5fc15ddf0 0x7ff5fc15e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.017+0000 7ff5fa59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff5fc15d7b0 0x7ff5fc06a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44744/0 (socket says 172.21.15.189:44744) 2024-04-06T12:12:48.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.017+0000 7ff5fa59c700 1 -- 172.21.15.189:0/3913354468 learned_addr learned my addr 172.21.15.189:0/3913354468 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:48.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.017+0000 7ff5f9d9b700 1 --2- 172.21.15.189:0/3913354468 >> v2:172.21.15.192:3300/0 conn(0x7ff5fc06ae40 0x7ff5fc0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.017+0000 7ff5fad9d700 1 -- 172.21.15.189:0/3913354468 >> v2:172.21.15.192:3300/0 conn(0x7ff5fc06ae40 msgr2=0x7ff5fc0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:48.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.017+0000 7ff5fad9d700 1 --2- 172.21.15.189:0/3913354468 >> v2:172.21.15.192:3300/0 conn(0x7ff5fc06ae40 0x7ff5fc0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.017+0000 7ff5fad9d700 1 -- 172.21.15.189:0/3913354468 >> v1:172.21.15.189:6789/0 conn(0x7ff5fc15d7b0 legacy=0x7ff5fc06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:48.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.017+0000 7ff5fad9d700 1 -- 172.21.15.189:0/3913354468 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5fc0b3620 con 0x7ff5fc15ddf0 2024-04-06T12:12:48.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.017+0000 7ff5f959a700 1 -- 172.21.15.189:0/3913354468 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933105291 0 0) 0x7ff5fc0b3990 con 0x7ff5fc15d7b0 2024-04-06T12:12:48.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.018+0000 7ff5f9d9b700 1 --2- 172.21.15.189:0/3913354468 >> v2:172.21.15.192:3300/0 conn(0x7ff5fc06ae40 0x7ff5fc0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:48.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.018+0000 7ff5fad9d700 1 --2- 172.21.15.189:0/3913354468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5fc15ddf0 0x7ff5fc15e240 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc0afc00 tx=0x7ff5f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6086271c0ccc4d59 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:48.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.018+0000 7ff5f959a700 1 -- 172.21.15.189:0/3913354468 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f400e070 con 0x7ff5fc15ddf0 2024-04-06T12:12:48.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.018+0000 7ff5f959a700 1 -- 172.21.15.189:0/3913354468 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f400bbd0 con 0x7ff5fc15ddf0 2024-04-06T12:12:48.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.018+0000 7ff5f959a700 1 -- 172.21.15.189:0/3913354468 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4012430 con 0x7ff5fc15ddf0 2024-04-06T12:12:48.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.018+0000 7ff600c6b700 1 -- 172.21.15.189:0/3913354468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5fc15ddf0 msgr2=0x7ff5fc15e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:48.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.018+0000 7ff600c6b700 1 --2- 172.21.15.189:0/3913354468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5fc15ddf0 0x7ff5fc15e240 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc0afc00 tx=0x7ff5f400b330 comp rx=0 tx=0).stop 2024-04-06T12:12:48.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.018+0000 7ff600c6b700 1 -- 172.21.15.189:0/3913354468 shutdown_connections 2024-04-06T12:12:48.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.018+0000 7ff600c6b700 1 --2- 172.21.15.189:0/3913354468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5fc15ddf0 0x7ff5fc15e240 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.018+0000 7ff600c6b700 1 --2- 172.21.15.189:0/3913354468 >> v2:172.21.15.192:3300/0 conn(0x7ff5fc06ae40 0x7ff5fc0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.018+0000 7ff600c6b700 1 -- 172.21.15.189:0/3913354468 >> 172.21.15.189:0/3913354468 conn(0x7ff5fc0d00b0 msgr2=0x7ff5fc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:48.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.018+0000 7ff600c6b700 1 -- 172.21.15.189:0/3913354468 shutdown_connections 2024-04-06T12:12:48.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.018+0000 7ff600c6b700 1 -- 172.21.15.189:0/3913354468 wait complete. 2024-04-06T12:12:48.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.020+0000 7ff600c6b700 1 Processor -- start 2024-04-06T12:12:48.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.020+0000 7ff600c6b700 1 -- start start 2024-04-06T12:12:48.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.020+0000 7ff600c6b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc06ae40 0x7ff5fc15bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.020+0000 7ff600c6b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5fc15d7b0 0x7ff5fc14f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.020+0000 7ff600c6b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc15ddf0 0x7ff5fc0c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.020+0000 7ff600c6b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc14c440 con 0x7ff5fc15d7b0 2024-04-06T12:12:48.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.020+0000 7ff600c6b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc0c62e0 con 0x7ff5fc06ae40 2024-04-06T12:12:48.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.020+0000 7ff600c6b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc0c5470 con 0x7ff5fc15ddf0 2024-04-06T12:12:48.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.021+0000 7ff5f9d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5fc15d7b0 0x7ff5fc14f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.021+0000 7ff5f9d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5fc15d7b0 0x7ff5fc14f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46690/0 (socket says 172.21.15.189:46690) 2024-04-06T12:12:48.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.021+0000 7ff5f9d9b700 1 -- 172.21.15.189:0/1313551910 learned_addr learned my addr 172.21.15.189:0/1313551910 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:48.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.021+0000 7ff5fad9d700 1 --2- 172.21.15.189:0/1313551910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc15ddf0 0x7ff5fc0c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.021+0000 7ff5fa59c700 1 --2- 172.21.15.189:0/1313551910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc06ae40 0x7ff5fc15bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.021+0000 7ff5f9d9b700 1 -- 172.21.15.189:0/1313551910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc15ddf0 msgr2=0x7ff5fc0c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:48.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.021+0000 7ff5f9d9b700 1 --2- 172.21.15.189:0/1313551910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc15ddf0 0x7ff5fc0c4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.021+0000 7ff5f9d9b700 1 -- 172.21.15.189:0/1313551910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc06ae40 msgr2=0x7ff5fc15bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:48.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.021+0000 7ff5f9d9b700 1 --2- 172.21.15.189:0/1313551910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc06ae40 0x7ff5fc15bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.021+0000 7ff5f9d9b700 1 -- 172.21.15.189:0/1313551910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f4009a70 con 0x7ff5fc15d7b0 2024-04-06T12:12:48.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.021+0000 7ff5f9d9b700 1 --2- 172.21.15.189:0/1313551910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5fc15d7b0 0x7ff5fc14f430 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec009fd0 tx=0x7ff5ec00eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:48.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.021+0000 7ff5eaffd700 1 -- 172.21.15.189:0/1313551910 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec013610 con 0x7ff5fc15d7b0 2024-04-06T12:12:48.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.021+0000 7ff5eaffd700 1 -- 172.21.15.189:0/1313551910 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ec013770 con 0x7ff5fc15d7b0 2024-04-06T12:12:48.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.021+0000 7ff600c6b700 1 -- 172.21.15.189:0/1313551910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5fc0c56d0 con 0x7ff5fc15d7b0 2024-04-06T12:12:48.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.021+0000 7ff5eaffd700 1 -- 172.21.15.189:0/1313551910 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec013b20 con 0x7ff5fc15d7b0 2024-04-06T12:12:48.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.021+0000 7ff600c6b700 1 -- 172.21.15.189:0/1313551910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5fc0c5c90 con 0x7ff5fc15d7b0 2024-04-06T12:12:48.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.022+0000 7ff600c6b700 1 -- 172.21.15.189:0/1313551910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5fc067bb0 con 0x7ff5fc15d7b0 2024-04-06T12:12:48.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.023+0000 7ff5eaffd700 1 -- 172.21.15.189:0/1313551910 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff5ec013c80 con 0x7ff5fc15d7b0 2024-04-06T12:12:48.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.023+0000 7ff5eaffd700 1 --2- 172.21.15.189:0/1313551910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff5dc06ac90 0x7ff5dc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.023+0000 7ff5eaffd700 1 -- 172.21.15.189:0/1313551910 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff5ec089140 con 0x7ff5fc15d7b0 2024-04-06T12:12:48.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.023+0000 7ff5fa59c700 1 --2- 172.21.15.189:0/1313551910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff5dc06ac90 0x7ff5dc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.023+0000 7ff5fa59c700 1 --2- 172.21.15.189:0/1313551910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff5dc06ac90 0x7ff5dc06d150 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0007fd0 tx=0x7ff5f0007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:48.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.027+0000 7ff5eaffd700 1 -- 172.21.15.189:0/1313551910 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff5ec054030 con 0x7ff5fc15d7b0 2024-04-06T12:12:48.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.221+0000 7ff600c6b700 1 -- 172.21.15.189:0/1313551910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff5fc0c6450 con 0x7ff5fc15d7b0 2024-04-06T12:12:48.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.221+0000 7ff5eaffd700 1 -- 172.21.15.189:0/1313551910 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff5ec057650 con 0x7ff5fc15d7b0 2024-04-06T12:12:48.219 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:12:48.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.223+0000 7ff600c6b700 1 -- 172.21.15.189:0/1313551910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff5dc06ac90 msgr2=0x7ff5dc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:48.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.223+0000 7ff600c6b700 1 --2- 172.21.15.189:0/1313551910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff5dc06ac90 0x7ff5dc06d150 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0007fd0 tx=0x7ff5f0007de0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.223+0000 7ff600c6b700 1 -- 172.21.15.189:0/1313551910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5fc15d7b0 msgr2=0x7ff5fc14f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:48.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.223+0000 7ff600c6b700 1 --2- 172.21.15.189:0/1313551910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5fc15d7b0 0x7ff5fc14f430 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec009fd0 tx=0x7ff5ec00eac0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.223+0000 7ff600c6b700 1 -- 172.21.15.189:0/1313551910 shutdown_connections 2024-04-06T12:12:48.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.223+0000 7ff600c6b700 1 --2- 172.21.15.189:0/1313551910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc15ddf0 0x7ff5fc0c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.223+0000 7ff600c6b700 1 --2- 172.21.15.189:0/1313551910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff5dc06ac90 0x7ff5dc06d150 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.223+0000 7ff600c6b700 1 --2- 172.21.15.189:0/1313551910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc06ae40 0x7ff5fc15bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.223+0000 7ff600c6b700 1 --2- 172.21.15.189:0/1313551910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5fc15d7b0 0x7ff5fc14f430 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.223+0000 7ff600c6b700 1 -- 172.21.15.189:0/1313551910 >> 172.21.15.189:0/1313551910 conn(0x7ff5fc0d00b0 msgr2=0x7ff5fc071b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:48.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.223+0000 7ff600c6b700 1 -- 172.21.15.189:0/1313551910 shutdown_connections 2024-04-06T12:12:48.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.223+0000 7ff600c6b700 1 -- 172.21.15.189:0/1313551910 wait complete. 2024-04-06T12:12:48.234 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:12:48.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.363+0000 7f106cc2f700 1 Processor -- start 2024-04-06T12:12:48.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.363+0000 7f106cc2f700 1 -- start start 2024-04-06T12:12:48.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.364+0000 7f106cc2f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f106815d7c0 0x7f106806b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.364+0000 7f106cc2f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f106806b6f0 0x7f106806dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.364+0000 7f106cc2f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106806ceb0 con 0x7f10680cd3e0 2024-04-06T12:12:48.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.364+0000 7f106cc2f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10680cdfa0 con 0x7f106806b6f0 2024-04-06T12:12:48.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.364+0000 7f106cc2f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106806d750 con 0x7f106815d7c0 2024-04-06T12:12:48.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.364+0000 7f106659c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f10680cd3e0 0x7f10680cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44748/0 (socket says 172.21.15.189:44748) 2024-04-06T12:12:48.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.364+0000 7f106659c700 1 -- 172.21.15.189:0/416083615 learned_addr learned my addr 172.21.15.189:0/416083615 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:48.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.364+0000 7f1065d9b700 1 --2- 172.21.15.189:0/416083615 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f106815d7c0 0x7f106806b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.364+0000 7f106559a700 1 -- 172.21.15.189:0/416083615 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562391851 0 0) 0x7f106806ceb0 con 0x7f10680cd3e0 2024-04-06T12:12:48.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.364+0000 7f1066d9d700 1 --2- 172.21.15.189:0/416083615 >> v2:172.21.15.192:3300/0 conn(0x7f106806b6f0 0x7f106806dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.364+0000 7f106559a700 1 -- 172.21.15.189:0/416083615 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10540035b0 con 0x7f10680cd3e0 2024-04-06T12:12:48.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.364+0000 7f106559a700 1 -- 172.21.15.189:0/416083615 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3980137292 0 0) 0x7f10540035b0 con 0x7f10680cd3e0 2024-04-06T12:12:48.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.364+0000 7f106559a700 1 -- 172.21.15.189:0/416083615 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f106815d7c0 msgr2=0x7f106806b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:48.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.364+0000 7f106559a700 1 --2- 172.21.15.189:0/416083615 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f106815d7c0 0x7f106806b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f106559a700 1 -- 172.21.15.189:0/416083615 >> v2:172.21.15.192:3300/0 conn(0x7f106806b6f0 msgr2=0x7f106806dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:48.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f106559a700 1 --2- 172.21.15.189:0/416083615 >> v2:172.21.15.192:3300/0 conn(0x7f106806b6f0 0x7f106806dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f106559a700 1 -- 172.21.15.189:0/416083615 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106806bf00 con 0x7f10680cd3e0 2024-04-06T12:12:48.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f106559a700 1 -- 172.21.15.189:0/416083615 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f1050001670 con 0x7f10680cd3e0 2024-04-06T12:12:48.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f106559a700 1 -- 172.21.15.189:0/416083615 >> v1:172.21.15.189:6789/0 conn(0x7f10680cd3e0 legacy=0x7f10680cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:48.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f106559a700 1 --2- 172.21.15.189:0/416083615 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f106815d7c0 0x7f1054003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f106559a700 1 --2- 172.21.15.189:0/416083615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1054003f80 0x7f1054004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f106559a700 1 --2- 172.21.15.189:0/416083615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1054004d90 0x7f1054005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f106559a700 1 -- 172.21.15.189:0/416083615 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10680cdfa0 con 0x7f1054003f80 2024-04-06T12:12:48.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f106559a700 1 -- 172.21.15.189:0/416083615 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106806d750 con 0x7f1054004d90 2024-04-06T12:12:48.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f106559a700 1 -- 172.21.15.189:0/416083615 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f105400ba60 con 0x7f106815d7c0 2024-04-06T12:12:48.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f1065d9b700 1 --2- 172.21.15.189:0/416083615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1054003f80 0x7f1054004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f106659c700 1 --2- 172.21.15.189:0/416083615 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f106815d7c0 0x7f1054003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f1066d9d700 1 --2- 172.21.15.189:0/416083615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1054004d90 0x7f1054005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f1065d9b700 1 -- 172.21.15.189:0/416083615 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f106815d7c0 msgr2=0x7f1054003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:48.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f1065d9b700 1 --2- 172.21.15.189:0/416083615 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f106815d7c0 0x7f1054003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f1065d9b700 1 -- 172.21.15.189:0/416083615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1054004d90 msgr2=0x7f1054005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:48.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f1065d9b700 1 --2- 172.21.15.189:0/416083615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1054004d90 0x7f1054005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f1065d9b700 1 -- 172.21.15.189:0/416083615 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10540035b0 con 0x7f1054003f80 2024-04-06T12:12:48.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f1066d9d700 1 --2- 172.21.15.189:0/416083615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1054004d90 0x7f1054005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:48.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f1065d9b700 1 --2- 172.21.15.189:0/416083615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1054003f80 0x7f1054004850 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1058009b60 tx=0x7f1054005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9f93f028d4dee4c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:48.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f106559a700 1 -- 172.21.15.189:0/416083615 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105800e6c0 con 0x7f1054003f80 2024-04-06T12:12:48.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f106559a700 1 -- 172.21.15.189:0/416083615 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105800e820 con 0x7f1054003f80 2024-04-06T12:12:48.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.365+0000 7f106559a700 1 -- 172.21.15.189:0/416083615 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105800eb90 con 0x7f1054003f80 2024-04-06T12:12:48.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.366+0000 7f106cc2f700 1 -- 172.21.15.189:0/416083615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1054003f80 msgr2=0x7f1054004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:48.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.366+0000 7f106cc2f700 1 --2- 172.21.15.189:0/416083615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1054003f80 0x7f1054004850 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1058009b60 tx=0x7f1054005880 comp rx=0 tx=0).stop 2024-04-06T12:12:48.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.366+0000 7f1066d9d700 1 -- 172.21.15.189:0/416083615 reap_dead start 2024-04-06T12:12:48.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.366+0000 7f106cc2f700 1 -- 172.21.15.189:0/416083615 shutdown_connections 2024-04-06T12:12:48.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.366+0000 7f106cc2f700 1 -- 172.21.15.189:0/416083615 >> 172.21.15.189:0/416083615 conn(0x7f10680cc290 msgr2=0x7f10680cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:48.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.366+0000 7f106cc2f700 1 -- 172.21.15.189:0/416083615 shutdown_connections 2024-04-06T12:12:48.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.366+0000 7f106cc2f700 1 -- 172.21.15.189:0/416083615 wait complete. 2024-04-06T12:12:48.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.367+0000 7f106cc2f700 1 Processor -- start 2024-04-06T12:12:48.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.367+0000 7f106cc2f700 1 -- start start 2024-04-06T12:12:48.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.368+0000 7f106cc2f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1054003f80 0x7f10680b4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.368+0000 7f106cc2f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10680b4070 0x7f10681f6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.368+0000 7f106cc2f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10681f68a0 0x7f10681f6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.368+0000 7f106cc2f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10680b52e0 con 0x7f10681f68a0 2024-04-06T12:12:48.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.368+0000 7f106cc2f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106815a1f0 con 0x7f1054003f80 2024-04-06T12:12:48.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.368+0000 7f106cc2f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10681f7270 con 0x7f10680b4070 2024-04-06T12:12:48.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.368+0000 7f1066d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10681f68a0 0x7f10681f6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.368+0000 7f1066d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10681f68a0 0x7f10681f6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46708/0 (socket says 172.21.15.189:46708) 2024-04-06T12:12:48.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.368+0000 7f1066d9d700 1 -- 172.21.15.189:0/2468059959 learned_addr learned my addr 172.21.15.189:0/2468059959 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:48.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.368+0000 7f106659c700 1 --2- 172.21.15.189:0/2468059959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1054003f80 0x7f10680b4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.368+0000 7f1065d9b700 1 --2- 172.21.15.189:0/2468059959 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10680b4070 0x7f10681f6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.368+0000 7f1066d9d700 1 -- 172.21.15.189:0/2468059959 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10680b4070 msgr2=0x7f10681f6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:48.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.368+0000 7f1066d9d700 1 --2- 172.21.15.189:0/2468059959 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10680b4070 0x7f10681f6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.368+0000 7f1066d9d700 1 -- 172.21.15.189:0/2468059959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1054003f80 msgr2=0x7f10680b4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:48.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.368+0000 7f1066d9d700 1 --2- 172.21.15.189:0/2468059959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1054003f80 0x7f10680b4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.368+0000 7f1066d9d700 1 -- 172.21.15.189:0/2468059959 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1068159c30 con 0x7f10681f68a0 2024-04-06T12:12:48.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.369+0000 7f1066d9d700 1 --2- 172.21.15.189:0/2468059959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10681f68a0 0x7f10681f6d30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f106806d890 tx=0x7f10600104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:48.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.369+0000 7f105effd700 1 -- 172.21.15.189:0/2468059959 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106000e050 con 0x7f10681f68a0 2024-04-06T12:12:48.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.369+0000 7f105effd700 1 -- 172.21.15.189:0/2468059959 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106000f040 con 0x7f10681f68a0 2024-04-06T12:12:48.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.369+0000 7f106cc2f700 1 -- 172.21.15.189:0/2468059959 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1068159f10 con 0x7f10681f68a0 2024-04-06T12:12:48.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.369+0000 7f105effd700 1 -- 172.21.15.189:0/2468059959 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1060010dd0 con 0x7f10681f68a0 2024-04-06T12:12:48.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.369+0000 7f106cc2f700 1 -- 172.21.15.189:0/2468059959 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106815cc10 con 0x7f10681f68a0 2024-04-06T12:12:48.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.370+0000 7f106cc2f700 1 -- 172.21.15.189:0/2468059959 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f106815d030 con 0x7f10681f68a0 2024-04-06T12:12:48.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.370+0000 7f105effd700 1 -- 172.21.15.189:0/2468059959 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1060015440 con 0x7f10681f68a0 2024-04-06T12:12:48.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.371+0000 7f105effd700 1 --2- 172.21.15.189:0/2468059959 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f104806ac90 0x7f104806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.371+0000 7f105effd700 1 -- 172.21.15.189:0/2468059959 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f10600897c0 con 0x7f10681f68a0 2024-04-06T12:12:48.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.371+0000 7f106659c700 1 --2- 172.21.15.189:0/2468059959 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f104806ac90 0x7f104806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.371+0000 7f106659c700 1 --2- 172.21.15.189:0/2468059959 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f104806ac90 0x7f104806d150 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1050000c00 tx=0x7f1050009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:48.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.375+0000 7f105effd700 1 -- 172.21.15.189:0/2468059959 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f106815d030 con 0x7f10681f68a0 2024-04-06T12:12:48.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.570+0000 7f106cc2f700 1 -- 172.21.15.189:0/2468059959 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f106815d030 con 0x7f10681f68a0 2024-04-06T12:12:48.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.570+0000 7f105effd700 1 -- 172.21.15.189:0/2468059959 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f106815d030 con 0x7f10681f68a0 2024-04-06T12:12:48.569 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:12:48.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.573+0000 7f106cc2f700 1 -- 172.21.15.189:0/2468059959 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f104806ac90 msgr2=0x7f104806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:48.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.573+0000 7f106cc2f700 1 --2- 172.21.15.189:0/2468059959 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f104806ac90 0x7f104806d150 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1050000c00 tx=0x7f1050009040 comp rx=0 tx=0).stop 2024-04-06T12:12:48.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.573+0000 7f106cc2f700 1 -- 172.21.15.189:0/2468059959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10681f68a0 msgr2=0x7f10681f6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:48.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.573+0000 7f106cc2f700 1 --2- 172.21.15.189:0/2468059959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10681f68a0 0x7f10681f6d30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f106806d890 tx=0x7f10600104a0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.573+0000 7f106cc2f700 1 -- 172.21.15.189:0/2468059959 shutdown_connections 2024-04-06T12:12:48.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.573+0000 7f106cc2f700 1 --2- 172.21.15.189:0/2468059959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10681f68a0 0x7f10681f6d30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.573+0000 7f106cc2f700 1 --2- 172.21.15.189:0/2468059959 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f104806ac90 0x7f104806d150 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.573+0000 7f106cc2f700 1 --2- 172.21.15.189:0/2468059959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1054003f80 0x7f10680b4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.573+0000 7f106cc2f700 1 --2- 172.21.15.189:0/2468059959 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10680b4070 0x7f10681f6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.573+0000 7f106cc2f700 1 -- 172.21.15.189:0/2468059959 >> 172.21.15.189:0/2468059959 conn(0x7f10680cc290 msgr2=0x7f1068160db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:48.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.573+0000 7f106cc2f700 1 -- 172.21.15.189:0/2468059959 shutdown_connections 2024-04-06T12:12:48.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.573+0000 7f106cc2f700 1 -- 172.21.15.189:0/2468059959 wait complete. 2024-04-06T12:12:48.583 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-06T12:12:48.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.710+0000 7f1de3b26700 1 Processor -- start 2024-04-06T12:12:48.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.710+0000 7f1de3b26700 1 -- start start 2024-04-06T12:12:48.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.711+0000 7f1de3b26700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ddc15e8a0 0x7f1ddc0cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.711+0000 7f1de3b26700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1ddc15eee0 0x7f1ddc15f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.711+0000 7f1de3b26700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1ddc15f8a0 0x7f1ddc06ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.711+0000 7f1de3b26700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ddc15fd30 con 0x7f1ddc15eee0 2024-04-06T12:12:48.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.711+0000 7f1de3b26700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ddc0669d0 con 0x7f1ddc15f8a0 2024-04-06T12:12:48.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.711+0000 7f1de3b26700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc06ede0 con 0x7f1ddc15e8a0 2024-04-06T12:12:48.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.711+0000 7f1de10c1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1ddc15eee0 0x7f1ddc15f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.711+0000 7f1de10c1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1ddc15eee0 0x7f1ddc15f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46722/0 (socket says 172.21.15.189:46722) 2024-04-06T12:12:48.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.711+0000 7f1de10c1700 1 -- 172.21.15.189:0/1543409709 learned_addr learned my addr 172.21.15.189:0/1543409709 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:48.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.711+0000 7f1de20c3700 1 --2- 172.21.15.189:0/1543409709 >> v2:172.21.15.192:3300/0 conn(0x7f1ddc15f8a0 0x7f1ddc06ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.711+0000 7f1de18c2700 1 --2- 172.21.15.189:0/1543409709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ddc15e8a0 0x7f1ddc0cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.711+0000 7f1de10c1700 1 -- 172.21.15.189:0/1543409709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ddc15e8a0 msgr2=0x7f1ddc0cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:48.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.711+0000 7f1de10c1700 1 --2- 172.21.15.189:0/1543409709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ddc15e8a0 0x7f1ddc0cbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.711+0000 7f1de10c1700 1 -- 172.21.15.189:0/1543409709 >> v2:172.21.15.192:3300/0 conn(0x7f1ddc15f8a0 msgr2=0x7f1ddc06ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:48.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.711+0000 7f1de10c1700 1 --2- 172.21.15.189:0/1543409709 >> v2:172.21.15.192:3300/0 conn(0x7f1ddc15f8a0 0x7f1ddc06ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.711+0000 7f1de10c1700 1 -- 172.21.15.189:0/1543409709 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ddc15ec80 con 0x7f1ddc15eee0 2024-04-06T12:12:48.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.711+0000 7f1de20c3700 1 --2- 172.21.15.189:0/1543409709 >> v2:172.21.15.192:3300/0 conn(0x7f1ddc15f8a0 0x7f1ddc06ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:48.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.711+0000 7f1de10c1700 1 --2- 172.21.15.189:0/1543409709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ddc15eee0 0x7f1ddc15f360 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc0c9df0 tx=0x7f1dcc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a70105e5cc52dd55 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:48.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.712+0000 7f1de08c0700 1 -- 172.21.15.189:0/1543409709 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc00b870 con 0x7f1ddc15eee0 2024-04-06T12:12:48.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.712+0000 7f1de08c0700 1 -- 172.21.15.189:0/1543409709 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dcc00b9d0 con 0x7f1ddc15eee0 2024-04-06T12:12:48.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.712+0000 7f1de08c0700 1 -- 172.21.15.189:0/1543409709 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc00bd60 con 0x7f1ddc15eee0 2024-04-06T12:12:48.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.712+0000 7f1de3b26700 1 -- 172.21.15.189:0/1543409709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ddc15eee0 msgr2=0x7f1ddc15f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:48.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.712+0000 7f1de3b26700 1 --2- 172.21.15.189:0/1543409709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ddc15eee0 0x7f1ddc15f360 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc0c9df0 tx=0x7f1dcc009bd0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.712+0000 7f1de3b26700 1 -- 172.21.15.189:0/1543409709 shutdown_connections 2024-04-06T12:12:48.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.712+0000 7f1de3b26700 1 --2- 172.21.15.189:0/1543409709 >> v2:172.21.15.192:3300/0 conn(0x7f1ddc15f8a0 0x7f1ddc06ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.712+0000 7f1de3b26700 1 --2- 172.21.15.189:0/1543409709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ddc15e8a0 0x7f1ddc0cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.712+0000 7f1de3b26700 1 --2- 172.21.15.189:0/1543409709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ddc15eee0 0x7f1ddc15f360 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.712+0000 7f1de3b26700 1 -- 172.21.15.189:0/1543409709 >> 172.21.15.189:0/1543409709 conn(0x7f1ddc0cc290 msgr2=0x7f1ddc0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:48.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.712+0000 7f1de3b26700 1 -- 172.21.15.189:0/1543409709 shutdown_connections 2024-04-06T12:12:48.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.712+0000 7f1de3b26700 1 -- 172.21.15.189:0/1543409709 wait complete. 2024-04-06T12:12:48.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.713+0000 7f1de3b26700 1 Processor -- start 2024-04-06T12:12:48.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.713+0000 7f1de3b26700 1 -- start start 2024-04-06T12:12:48.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.714+0000 7f1de3b26700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ddc15e8a0 0x7f1ddc0b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.714+0000 7f1de3b26700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ddc15eee0 0x7f1ddc0c6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.714+0000 7f1de3b26700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ddc15f8a0 0x7f1ddc1542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.714+0000 7f1de3b26700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc0b28e0 con 0x7f1ddc15eee0 2024-04-06T12:12:48.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.714+0000 7f1de3b26700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc155100 con 0x7f1ddc15e8a0 2024-04-06T12:12:48.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.714+0000 7f1de3b26700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc0c66b0 con 0x7f1ddc15f8a0 2024-04-06T12:12:48.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.714+0000 7f1de10c1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ddc15eee0 0x7f1ddc0c6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.714+0000 7f1de10c1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ddc15eee0 0x7f1ddc0c6170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46726/0 (socket says 172.21.15.189:46726) 2024-04-06T12:12:48.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.714+0000 7f1de10c1700 1 -- 172.21.15.189:0/244400063 learned_addr learned my addr 172.21.15.189:0/244400063 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:48.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.714+0000 7f1de20c3700 1 --2- 172.21.15.189:0/244400063 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ddc15f8a0 0x7f1ddc1542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.714+0000 7f1de18c2700 1 --2- 172.21.15.189:0/244400063 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ddc15e8a0 0x7f1ddc0b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.715+0000 7f1de10c1700 1 -- 172.21.15.189:0/244400063 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ddc15f8a0 msgr2=0x7f1ddc1542f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:48.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.715+0000 7f1de10c1700 1 --2- 172.21.15.189:0/244400063 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ddc15f8a0 0x7f1ddc1542f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.715+0000 7f1de10c1700 1 -- 172.21.15.189:0/244400063 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ddc15e8a0 msgr2=0x7f1ddc0b50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:48.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.715+0000 7f1de10c1700 1 --2- 172.21.15.189:0/244400063 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ddc15e8a0 0x7f1ddc0b50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.715+0000 7f1de10c1700 1 -- 172.21.15.189:0/244400063 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc009550 con 0x7f1ddc15eee0 2024-04-06T12:12:48.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.715+0000 7f1de20c3700 1 --2- 172.21.15.189:0/244400063 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ddc15f8a0 0x7f1ddc1542f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:48.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.715+0000 7f1de18c2700 1 --2- 172.21.15.189:0/244400063 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ddc15e8a0 0x7f1ddc0b50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:48.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.715+0000 7f1de10c1700 1 --2- 172.21.15.189:0/244400063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ddc15eee0 0x7f1ddc0c6170 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc073430 tx=0x7f1dcc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:48.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.715+0000 7f1dd27fc700 1 -- 172.21.15.189:0/244400063 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc004540 con 0x7f1ddc15eee0 2024-04-06T12:12:48.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.715+0000 7f1dd27fc700 1 -- 172.21.15.189:0/244400063 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dcc0036a0 con 0x7f1ddc15eee0 2024-04-06T12:12:48.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.715+0000 7f1dd27fc700 1 -- 172.21.15.189:0/244400063 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc003a50 con 0x7f1ddc15eee0 2024-04-06T12:12:48.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.715+0000 7f1de3b26700 1 -- 172.21.15.189:0/244400063 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ddc154b40 con 0x7f1ddc15eee0 2024-04-06T12:12:48.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.715+0000 7f1de3b26700 1 -- 172.21.15.189:0/244400063 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ddc0b2500 con 0x7f1ddc15eee0 2024-04-06T12:12:48.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.716+0000 7f1de3b26700 1 -- 172.21.15.189:0/244400063 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ddc0b26c0 con 0x7f1ddc15eee0 2024-04-06T12:12:48.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.716+0000 7f1dd27fc700 1 -- 172.21.15.189:0/244400063 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1dcc021070 con 0x7f1ddc15eee0 2024-04-06T12:12:48.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.717+0000 7f1dd27fc700 1 --2- 172.21.15.189:0/244400063 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dc006ac90 0x7f1dc006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:48.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.717+0000 7f1de18c2700 1 --2- 172.21.15.189:0/244400063 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dc006ac90 0x7f1dc006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:48.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.717+0000 7f1dd27fc700 1 -- 172.21.15.189:0/244400063 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1dcc08ac50 con 0x7f1ddc15eee0 2024-04-06T12:12:48.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.717+0000 7f1de18c2700 1 --2- 172.21.15.189:0/244400063 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dc006ac90 0x7f1dc006d150 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8009970 tx=0x7f1dd8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:48.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.721+0000 7f1dd27fc700 1 -- 172.21.15.189:0/244400063 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1dcc090020 con 0x7f1ddc15eee0 2024-04-06T12:12:48.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.913+0000 7f1de3b26700 1 -- 172.21.15.189:0/244400063 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1ddc072140 con 0x7f1ddc15eee0 2024-04-06T12:12:48.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.913+0000 7f1dd27fc700 1 -- 172.21.15.189:0/244400063 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1dcc015020 con 0x7f1ddc15eee0 2024-04-06T12:12:48.911 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:12:48.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.915+0000 7f1de3b26700 1 -- 172.21.15.189:0/244400063 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dc006ac90 msgr2=0x7f1dc006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:48.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.915+0000 7f1de3b26700 1 --2- 172.21.15.189:0/244400063 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dc006ac90 0x7f1dc006d150 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8009970 tx=0x7f1dd8008040 comp rx=0 tx=0).stop 2024-04-06T12:12:48.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.915+0000 7f1de3b26700 1 -- 172.21.15.189:0/244400063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ddc15eee0 msgr2=0x7f1ddc0c6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:48.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.915+0000 7f1de3b26700 1 --2- 172.21.15.189:0/244400063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ddc15eee0 0x7f1ddc0c6170 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc073430 tx=0x7f1dcc004750 comp rx=0 tx=0).stop 2024-04-06T12:12:48.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.915+0000 7f1de3b26700 1 -- 172.21.15.189:0/244400063 shutdown_connections 2024-04-06T12:12:48.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.915+0000 7f1de3b26700 1 --2- 172.21.15.189:0/244400063 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ddc15f8a0 0x7f1ddc1542f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.915+0000 7f1de3b26700 1 --2- 172.21.15.189:0/244400063 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dc006ac90 0x7f1dc006d150 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.915+0000 7f1de3b26700 1 --2- 172.21.15.189:0/244400063 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ddc15e8a0 0x7f1ddc0b50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.915+0000 7f1de3b26700 1 --2- 172.21.15.189:0/244400063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ddc15eee0 0x7f1ddc0c6170 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:48.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.915+0000 7f1de3b26700 1 -- 172.21.15.189:0/244400063 >> 172.21.15.189:0/244400063 conn(0x7f1ddc0cc290 msgr2=0x7f1ddc0c9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:48.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.916+0000 7f1de3b26700 1 -- 172.21.15.189:0/244400063 shutdown_connections 2024-04-06T12:12:48.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:48.916+0000 7f1de3b26700 1 -- 172.21.15.189:0/244400063 wait complete. 2024-04-06T12:12:48.926 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:49.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.049+0000 7fa357d98700 1 Processor -- start 2024-04-06T12:12:49.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.049+0000 7fa357d98700 1 -- start start 2024-04-06T12:12:49.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.050+0000 7fa357d98700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3500cd3e0 0x7fa3500cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.050+0000 7fa357d98700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa35015d7c0 0x7fa35006b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.050+0000 7fa357d98700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35006ceb0 con 0x7fa35006b6f0 2024-04-06T12:12:49.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.050+0000 7fa357d98700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3500cdfa0 con 0x7fa35015d7c0 2024-04-06T12:12:49.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.050+0000 7fa357d98700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa35006d750 con 0x7fa3500cd3e0 2024-04-06T12:12:49.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.050+0000 7fa355333700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa35015d7c0 0x7fa35006b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.050+0000 7fa355b34700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3500cd3e0 0x7fa3500cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.050+0000 7fa356335700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fa35006b6f0 0x7fa3501610f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:49330/0 (socket says 172.21.15.189:49330) 2024-04-06T12:12:49.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.050+0000 7fa356335700 1 -- 172.21.15.189:0/2194550411 learned_addr learned my addr 172.21.15.189:0/2194550411 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:49.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.050+0000 7fa354b32700 1 -- 172.21.15.189:0/2194550411 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293951693 0 0) 0x7fa35006ceb0 con 0x7fa35006b6f0 2024-04-06T12:12:49.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.050+0000 7fa354b32700 1 -- 172.21.15.189:0/2194550411 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa33c0035b0 con 0x7fa35006b6f0 2024-04-06T12:12:49.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.050+0000 7fa355333700 1 -- 172.21.15.189:0/2194550411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3500cd3e0 msgr2=0x7fa3500cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:49.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.050+0000 7fa355333700 1 --2- 172.21.15.189:0/2194550411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3500cd3e0 0x7fa3500cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.050+0000 7fa355333700 1 -- 172.21.15.189:0/2194550411 >> v1:172.21.15.192:6789/0 conn(0x7fa35006b6f0 legacy=0x7fa3501610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:49.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.050+0000 7fa355333700 1 -- 172.21.15.189:0/2194550411 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3500cdd00 con 0x7fa35015d7c0 2024-04-06T12:12:49.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.050+0000 7fa355b34700 1 --2- 172.21.15.189:0/2194550411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3500cd3e0 0x7fa3500cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:49.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.051+0000 7fa355333700 1 --2- 172.21.15.189:0/2194550411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa35015d7c0 0x7fa35006b1b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa340009840 tx=0x7fa340009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afcaa6a783dea46d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:49.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.051+0000 7fa354b32700 1 -- 172.21.15.189:0/2194550411 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa340011830 con 0x7fa35015d7c0 2024-04-06T12:12:49.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.051+0000 7fa354b32700 1 -- 172.21.15.189:0/2194550411 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa340011990 con 0x7fa35015d7c0 2024-04-06T12:12:49.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.051+0000 7fa354b32700 1 -- 172.21.15.189:0/2194550411 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa340011d20 con 0x7fa35015d7c0 2024-04-06T12:12:49.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.051+0000 7fa357d98700 1 -- 172.21.15.189:0/2194550411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa35015d7c0 msgr2=0x7fa35006b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:49.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.051+0000 7fa357d98700 1 --2- 172.21.15.189:0/2194550411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa35015d7c0 0x7fa35006b1b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa340009840 tx=0x7fa340009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:49.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.051+0000 7fa357d98700 1 -- 172.21.15.189:0/2194550411 shutdown_connections 2024-04-06T12:12:49.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.051+0000 7fa357d98700 1 --2- 172.21.15.189:0/2194550411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3500cd3e0 0x7fa3500cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.051+0000 7fa357d98700 1 --2- 172.21.15.189:0/2194550411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa35015d7c0 0x7fa35006b1b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.051+0000 7fa357d98700 1 -- 172.21.15.189:0/2194550411 >> 172.21.15.189:0/2194550411 conn(0x7fa3500cc290 msgr2=0x7fa3500cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:49.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.051+0000 7fa357d98700 1 -- 172.21.15.189:0/2194550411 shutdown_connections 2024-04-06T12:12:49.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.051+0000 7fa357d98700 1 -- 172.21.15.189:0/2194550411 wait complete. 2024-04-06T12:12:49.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.053+0000 7fa357d98700 1 Processor -- start 2024-04-06T12:12:49.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.053+0000 7fa357d98700 1 -- start start 2024-04-06T12:12:49.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.053+0000 7fa357d98700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa35006b6f0 0x7fa350157be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.053+0000 7fa357d98700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3500cd3e0 0x7fa35015c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.053+0000 7fa357d98700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa35015d7c0 0x7fa3500b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.053+0000 7fa357d98700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35015b270 con 0x7fa35015d7c0 2024-04-06T12:12:49.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.053+0000 7fa357d98700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa350158120 con 0x7fa3500cd3e0 2024-04-06T12:12:49.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.053+0000 7fa357d98700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa35015a660 con 0x7fa35006b6f0 2024-04-06T12:12:49.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.053+0000 7fa356335700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa35015d7c0 0x7fa3500b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.053+0000 7fa356335700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa35015d7c0 0x7fa3500b44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46742/0 (socket says 172.21.15.189:46742) 2024-04-06T12:12:49.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.054+0000 7fa356335700 1 -- 172.21.15.189:0/2287583439 learned_addr learned my addr 172.21.15.189:0/2287583439 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:49.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.054+0000 7fa355333700 1 --2- 172.21.15.189:0/2287583439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3500cd3e0 0x7fa35015c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.054+0000 7fa355b34700 1 --2- 172.21.15.189:0/2287583439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa35006b6f0 0x7fa350157be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.054+0000 7fa356335700 1 -- 172.21.15.189:0/2287583439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa35006b6f0 msgr2=0x7fa350157be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:49.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.054+0000 7fa356335700 1 --2- 172.21.15.189:0/2287583439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa35006b6f0 0x7fa350157be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.054+0000 7fa356335700 1 -- 172.21.15.189:0/2287583439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3500cd3e0 msgr2=0x7fa35015c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:49.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.054+0000 7fa356335700 1 --2- 172.21.15.189:0/2287583439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3500cd3e0 0x7fa35015c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.054+0000 7fa356335700 1 -- 172.21.15.189:0/2287583439 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34c0094a0 con 0x7fa35015d7c0 2024-04-06T12:12:49.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.054+0000 7fa356335700 1 --2- 172.21.15.189:0/2287583439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa35015d7c0 0x7fa3500b44b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa34800caa0 tx=0x7fa34800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:49.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.054+0000 7fa3467fc700 1 -- 172.21.15.189:0/2287583439 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34800a9e0 con 0x7fa35015d7c0 2024-04-06T12:12:49.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.054+0000 7fa3467fc700 1 -- 172.21.15.189:0/2287583439 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34800ab40 con 0x7fa35015d7c0 2024-04-06T12:12:49.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.054+0000 7fa357d98700 1 -- 172.21.15.189:0/2287583439 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3400094a0 con 0x7fa35015d7c0 2024-04-06T12:12:49.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.054+0000 7fa3467fc700 1 -- 172.21.15.189:0/2287583439 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa348012400 con 0x7fa35015d7c0 2024-04-06T12:12:49.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.054+0000 7fa357d98700 1 -- 172.21.15.189:0/2287583439 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa35015ac20 con 0x7fa35015d7c0 2024-04-06T12:12:49.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.055+0000 7fa357d98700 1 -- 172.21.15.189:0/2287583439 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa35015b3e0 con 0x7fa35015d7c0 2024-04-06T12:12:49.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.055+0000 7fa3467fc700 1 -- 172.21.15.189:0/2287583439 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa34801b3b0 con 0x7fa35015d7c0 2024-04-06T12:12:49.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.056+0000 7fa3467fc700 1 --2- 172.21.15.189:0/2287583439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa33406ac90 0x7fa33406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.056+0000 7fa3467fc700 1 -- 172.21.15.189:0/2287583439 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa348089350 con 0x7fa35015d7c0 2024-04-06T12:12:49.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.056+0000 7fa355b34700 1 --2- 172.21.15.189:0/2287583439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa33406ac90 0x7fa33406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.056+0000 7fa355b34700 1 --2- 172.21.15.189:0/2287583439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa33406ac90 0x7fa33406d150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa34c009cd0 tx=0x7fa34c009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:49.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.060+0000 7fa3467fc700 1 -- 172.21.15.189:0/2287583439 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa3480541f0 con 0x7fa35015d7c0 2024-04-06T12:12:49.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.254+0000 7fa357d98700 1 -- 172.21.15.189:0/2287583439 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa350074990 con 0x7fa35015d7c0 2024-04-06T12:12:49.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.256+0000 7fa3467fc700 1 -- 172.21.15.189:0/2287583439 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17926 (secure 0 0 0) 0x7fa348057810 con 0x7fa35015d7c0 2024-04-06T12:12:49.256 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:49.256 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:08.649170+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:49.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.258+0000 7fa357d98700 1 -- 172.21.15.189:0/2287583439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa33406ac90 msgr2=0x7fa33406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.258+0000 7fa357d98700 1 --2- 172.21.15.189:0/2287583439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa33406ac90 0x7fa33406d150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa34c009cd0 tx=0x7fa34c009210 comp rx=0 tx=0).stop 2024-04-06T12:12:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.258+0000 7fa357d98700 1 -- 172.21.15.189:0/2287583439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa35015d7c0 msgr2=0x7fa3500b44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.258+0000 7fa357d98700 1 --2- 172.21.15.189:0/2287583439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa35015d7c0 0x7fa3500b44b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa34800caa0 tx=0x7fa34800cdb0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.258+0000 7fa357d98700 1 -- 172.21.15.189:0/2287583439 shutdown_connections 2024-04-06T12:12:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.258+0000 7fa357d98700 1 --2- 172.21.15.189:0/2287583439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa35015d7c0 0x7fa3500b44b0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.258+0000 7fa357d98700 1 --2- 172.21.15.189:0/2287583439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa35006b6f0 0x7fa350157be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.258+0000 7fa357d98700 1 --2- 172.21.15.189:0/2287583439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa33406ac90 0x7fa33406d150 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.258+0000 7fa357d98700 1 --2- 172.21.15.189:0/2287583439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3500cd3e0 0x7fa35015c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.258+0000 7fa357d98700 1 -- 172.21.15.189:0/2287583439 >> 172.21.15.189:0/2287583439 conn(0x7fa3500cc290 msgr2=0x7fa3500acf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:49.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.258+0000 7fa357d98700 1 -- 172.21.15.189:0/2287583439 shutdown_connections 2024-04-06T12:12:49.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.258+0000 7fa357d98700 1 -- 172.21.15.189:0/2287583439 wait complete. 2024-04-06T12:12:49.269 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:12:02.280866+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:12:03.156593+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-06T12:12:06.237184+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-06T12:12:49.269 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:12:49.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.398+0000 7f5db0c71700 1 Processor -- start 2024-04-06T12:12:49.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.398+0000 7f5db0c71700 1 -- start start 2024-04-06T12:12:49.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5db0c71700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5dac06cbf0 0x7f5dac0cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5db0c71700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dac15d8d0 con 0x7f5dac15d040 2024-04-06T12:12:49.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5db0c71700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dac15d580 con 0x7f5dac0b2cd0 2024-04-06T12:12:49.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5db0c71700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dac0cba40 con 0x7f5dac06cbf0 2024-04-06T12:12:49.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5daa59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5dac15d040 0x7f5dac15dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39024/0 (socket says 172.21.15.189:39024) 2024-04-06T12:12:49.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5daa59c700 1 -- 172.21.15.189:0/202115505 learned_addr learned my addr 172.21.15.189:0/202115505 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:49.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5da959a700 1 -- 172.21.15.189:0/202115505 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502446025 0 0) 0x7f5dac15d8d0 con 0x7f5dac15d040 2024-04-06T12:12:49.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5da959a700 1 -- 172.21.15.189:0/202115505 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d940034d0 con 0x7f5dac15d040 2024-04-06T12:12:49.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5daad9d700 1 --2- 172.21.15.189:0/202115505 >> v2:172.21.15.192:3300/0 conn(0x7f5dac06cbf0 0x7f5dac0cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5da959a700 1 -- 172.21.15.189:0/202115505 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1965495885 0 0) 0x7f5d940034d0 con 0x7f5dac15d040 2024-04-06T12:12:49.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5da959a700 1 -- 172.21.15.189:0/202115505 >> v2:172.21.15.192:3300/0 conn(0x7f5dac06cbf0 msgr2=0x7f5dac0cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:49.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5da959a700 1 --2- 172.21.15.189:0/202115505 >> v2:172.21.15.192:3300/0 conn(0x7f5dac06cbf0 0x7f5dac0cc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5da959a700 1 -- 172.21.15.189:0/202115505 >> v1:172.21.15.192:6789/0 conn(0x7f5dac0b2cd0 legacy=0x7f5dac06c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:49.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5da959a700 1 -- 172.21.15.189:0/202115505 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dac15e470 con 0x7f5dac15d040 2024-04-06T12:12:49.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5da959a700 1 -- 172.21.15.189:0/202115505 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f5d9c001670 con 0x7f5dac15d040 2024-04-06T12:12:49.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5daad9d700 1 --2- 172.21.15.189:0/202115505 >> v2:172.21.15.192:3300/0 conn(0x7f5dac06cbf0 0x7f5dac0cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:49.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5da959a700 1 -- 172.21.15.189:0/202115505 >> v1:172.21.15.189:6789/0 conn(0x7f5dac15d040 legacy=0x7f5dac15dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:49.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5da959a700 1 --2- 172.21.15.189:0/202115505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d94003af0 0x7f5d94003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5da959a700 1 --2- 172.21.15.189:0/202115505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d94008930 0x7f5d9400adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5da959a700 1 --2- 172.21.15.189:0/202115505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d9400b5c0 0x7f5d9400da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5da959a700 1 -- 172.21.15.189:0/202115505 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9400e330 con 0x7f5d9400b5c0 2024-04-06T12:12:49.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5da959a700 1 -- 172.21.15.189:0/202115505 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9400e4a0 con 0x7f5d94003af0 2024-04-06T12:12:49.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.399+0000 7f5da959a700 1 -- 172.21.15.189:0/202115505 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9400e660 con 0x7f5d94008930 2024-04-06T12:12:49.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.400+0000 7f5daad9d700 1 --2- 172.21.15.189:0/202115505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d9400b5c0 0x7f5d9400da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.400+0000 7f5daa59c700 1 --2- 172.21.15.189:0/202115505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d94003af0 0x7f5d94003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.400+0000 7f5da9d9b700 1 --2- 172.21.15.189:0/202115505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d94008930 0x7f5d9400adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.400+0000 7f5daad9d700 1 -- 172.21.15.189:0/202115505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d94008930 msgr2=0x7f5d9400adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:49.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.400+0000 7f5daad9d700 1 --2- 172.21.15.189:0/202115505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d94008930 0x7f5d9400adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.400+0000 7f5daad9d700 1 -- 172.21.15.189:0/202115505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d94003af0 msgr2=0x7f5d94003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:49.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.400+0000 7f5daad9d700 1 --2- 172.21.15.189:0/202115505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d94003af0 0x7f5d94003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.400+0000 7f5daad9d700 1 -- 172.21.15.189:0/202115505 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d940034d0 con 0x7f5d9400b5c0 2024-04-06T12:12:49.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.400+0000 7f5daad9d700 1 -- 172.21.15.189:0/202115505 reap_dead start 2024-04-06T12:12:49.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.400+0000 7f5daad9d700 1 --2- 172.21.15.189:0/202115505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d9400b5c0 0x7f5d9400da80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5da40073b0 tx=0x7f5d94003780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8437ea50d5133fef server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:49.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.400+0000 7f5da959a700 1 -- 172.21.15.189:0/202115505 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da4009490 con 0x7f5d9400b5c0 2024-04-06T12:12:49.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.400+0000 7f5da959a700 1 -- 172.21.15.189:0/202115505 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da4014400 con 0x7f5d9400b5c0 2024-04-06T12:12:49.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.400+0000 7f5da959a700 1 -- 172.21.15.189:0/202115505 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da40147b0 con 0x7f5d9400b5c0 2024-04-06T12:12:49.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.401+0000 7f5db0c71700 1 -- 172.21.15.189:0/202115505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d9400b5c0 msgr2=0x7f5d9400da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:49.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.401+0000 7f5db0c71700 1 --2- 172.21.15.189:0/202115505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d9400b5c0 0x7f5d9400da80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5da40073b0 tx=0x7f5d94003780 comp rx=0 tx=0).stop 2024-04-06T12:12:49.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.401+0000 7f5db0c71700 1 -- 172.21.15.189:0/202115505 shutdown_connections 2024-04-06T12:12:49.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.401+0000 7f5db0c71700 1 --2- 172.21.15.189:0/202115505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d9400b5c0 0x7f5d9400da80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.401+0000 7f5db0c71700 1 -- 172.21.15.189:0/202115505 >> 172.21.15.189:0/202115505 conn(0x7f5dac06a730 msgr2=0x7f5dac06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:49.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.401+0000 7f5db0c71700 1 -- 172.21.15.189:0/202115505 shutdown_connections 2024-04-06T12:12:49.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.401+0000 7f5db0c71700 1 -- 172.21.15.189:0/202115505 wait complete. 2024-04-06T12:12:49.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.402+0000 7f5db0c71700 1 Processor -- start 2024-04-06T12:12:49.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.402+0000 7f5db0c71700 1 -- start start 2024-04-06T12:12:49.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.403+0000 7f5db0c71700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d9400b5c0 0x7f5dac14fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.403+0000 7f5db0c71700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dac15afe0 0x7f5dac152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.403+0000 7f5db0c71700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dac1530d0 0x7f5dac1e43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.403+0000 7f5db0c71700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac153930 con 0x7f5dac1530d0 2024-04-06T12:12:49.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.403+0000 7f5db0c71700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac150430 con 0x7f5dac15afe0 2024-04-06T12:12:49.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.403+0000 7f5db0c71700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac153650 con 0x7f5d9400b5c0 2024-04-06T12:12:49.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.403+0000 7f5daad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dac1530d0 0x7f5dac1e43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.404+0000 7f5daad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dac1530d0 0x7f5dac1e43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57374/0 (socket says 172.21.15.189:57374) 2024-04-06T12:12:49.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.404+0000 7f5daad9d700 1 -- 172.21.15.189:0/1117832951 learned_addr learned my addr 172.21.15.189:0/1117832951 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:49.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.404+0000 7f5da9d9b700 1 --2- 172.21.15.189:0/1117832951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dac15afe0 0x7f5dac152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.404+0000 7f5daa59c700 1 --2- 172.21.15.189:0/1117832951 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d9400b5c0 0x7f5dac14fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.404+0000 7f5daad9d700 1 -- 172.21.15.189:0/1117832951 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d9400b5c0 msgr2=0x7f5dac14fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:49.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.404+0000 7f5daad9d700 1 --2- 172.21.15.189:0/1117832951 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d9400b5c0 0x7f5dac14fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.404+0000 7f5daad9d700 1 -- 172.21.15.189:0/1117832951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dac15afe0 msgr2=0x7f5dac152b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:49.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.404+0000 7f5daad9d700 1 --2- 172.21.15.189:0/1117832951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dac15afe0 0x7f5dac152b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.404+0000 7f5daad9d700 1 -- 172.21.15.189:0/1117832951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dac15a500 con 0x7f5dac1530d0 2024-04-06T12:12:49.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.404+0000 7f5daad9d700 1 --2- 172.21.15.189:0/1117832951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dac1530d0 0x7f5dac1e43c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0cf9c0 tx=0x7f5da4004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:49.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.404+0000 7f5d9affd700 1 -- 172.21.15.189:0/1117832951 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da4007000 con 0x7f5dac1530d0 2024-04-06T12:12:49.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.404+0000 7f5d9affd700 1 -- 172.21.15.189:0/1117832951 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da4004470 con 0x7f5dac1530d0 2024-04-06T12:12:49.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.404+0000 7f5db0c71700 1 -- 172.21.15.189:0/1117832951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dac15a7e0 con 0x7f5dac1530d0 2024-04-06T12:12:49.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.404+0000 7f5d9affd700 1 -- 172.21.15.189:0/1117832951 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da4004820 con 0x7f5dac1530d0 2024-04-06T12:12:49.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.404+0000 7f5db0c71700 1 -- 172.21.15.189:0/1117832951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dac150e60 con 0x7f5dac1530d0 2024-04-06T12:12:49.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.405+0000 7f5db0c71700 1 -- 172.21.15.189:0/1117832951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dac15a360 con 0x7f5dac1530d0 2024-04-06T12:12:49.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.406+0000 7f5d9affd700 1 -- 172.21.15.189:0/1117832951 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5da401b040 con 0x7f5dac1530d0 2024-04-06T12:12:49.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.406+0000 7f5d9affd700 1 --2- 172.21.15.189:0/1117832951 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d8c06ac90 0x7f5d8c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.406+0000 7f5d9affd700 1 -- 172.21.15.189:0/1117832951 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5da408a440 con 0x7f5dac1530d0 2024-04-06T12:12:49.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.406+0000 7f5daa59c700 1 --2- 172.21.15.189:0/1117832951 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d8c06ac90 0x7f5d8c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.406+0000 7f5daa59c700 1 --2- 172.21.15.189:0/1117832951 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d8c06ac90 0x7f5d8c06d150 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c000c00 tx=0x7f5d9c001b20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:49.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.410+0000 7f5d9affd700 1 -- 172.21.15.189:0/1117832951 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5dac15a360 con 0x7f5dac1530d0 2024-04-06T12:12:49.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.604+0000 7f5db0c71700 1 -- 172.21.15.189:0/1117832951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5dac15ad00 con 0x7f5dac1530d0 2024-04-06T12:12:49.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.605+0000 7f5d9affd700 1 -- 172.21.15.189:0/1117832951 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5dac15ad00 con 0x7f5dac1530d0 2024-04-06T12:12:49.603 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:12:49.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.607+0000 7f5db0c71700 1 -- 172.21.15.189:0/1117832951 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d8c06ac90 msgr2=0x7f5d8c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:49.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.607+0000 7f5db0c71700 1 --2- 172.21.15.189:0/1117832951 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d8c06ac90 0x7f5d8c06d150 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c000c00 tx=0x7f5d9c001b20 comp rx=0 tx=0).stop 2024-04-06T12:12:49.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.607+0000 7f5db0c71700 1 -- 172.21.15.189:0/1117832951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dac1530d0 msgr2=0x7f5dac1e43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:49.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.607+0000 7f5db0c71700 1 --2- 172.21.15.189:0/1117832951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dac1530d0 0x7f5dac1e43c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0cf9c0 tx=0x7f5da4004b40 comp rx=0 tx=0).stop 2024-04-06T12:12:49.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.607+0000 7f5db0c71700 1 -- 172.21.15.189:0/1117832951 shutdown_connections 2024-04-06T12:12:49.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.607+0000 7f5db0c71700 1 --2- 172.21.15.189:0/1117832951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dac1530d0 0x7f5dac1e43c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.607+0000 7f5db0c71700 1 --2- 172.21.15.189:0/1117832951 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d9400b5c0 0x7f5dac14fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.607+0000 7f5db0c71700 1 --2- 172.21.15.189:0/1117832951 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d8c06ac90 0x7f5d8c06d150 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.607+0000 7f5db0c71700 1 --2- 172.21.15.189:0/1117832951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dac15afe0 0x7f5dac152b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.607+0000 7f5db0c71700 1 -- 172.21.15.189:0/1117832951 >> 172.21.15.189:0/1117832951 conn(0x7f5dac06a730 msgr2=0x7f5dac1575e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:49.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.607+0000 7f5db0c71700 1 -- 172.21.15.189:0/1117832951 shutdown_connections 2024-04-06T12:12:49.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.607+0000 7f5db0c71700 1 -- 172.21.15.189:0/1117832951 wait complete. 2024-04-06T12:12:49.617 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:12:49.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.743+0000 7fcdac43b700 1 Processor -- start 2024-04-06T12:12:49.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.743+0000 7fcdac43b700 1 -- start start 2024-04-06T12:12:49.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.744+0000 7fcdac43b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcda4158fe0 0x7fcda406a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.744+0000 7fcdac43b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcda406ac70 0x7fcda415bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.744+0000 7fcdac43b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda415f9e0 con 0x7fcda415c100 2024-04-06T12:12:49.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.744+0000 7fcdac43b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcda406b0f0 con 0x7fcda406ac70 2024-04-06T12:12:49.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.744+0000 7fcdac43b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda415f0d0 con 0x7fcda4158fe0 2024-04-06T12:12:49.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.744+0000 7fcdaa9d8700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcda415c100 0x7fcda40717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39036/0 (socket says 172.21.15.189:39036) 2024-04-06T12:12:49.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.744+0000 7fcdaa9d8700 1 -- 172.21.15.189:0/2115139250 learned_addr learned my addr 172.21.15.189:0/2115139250 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:49.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.744+0000 7fcdaa1d7700 1 --2- 172.21.15.189:0/2115139250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcda4158fe0 0x7fcda406a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.744+0000 7fcda99d6700 1 --2- 172.21.15.189:0/2115139250 >> v2:172.21.15.192:3300/0 conn(0x7fcda406ac70 0x7fcda415bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.744+0000 7fcda91d5700 1 -- 172.21.15.189:0/2115139250 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195523951 0 0) 0x7fcda415f9e0 con 0x7fcda415c100 2024-04-06T12:12:49.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.744+0000 7fcda91d5700 1 -- 172.21.15.189:0/2115139250 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd900035b0 con 0x7fcda415c100 2024-04-06T12:12:49.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.744+0000 7fcdaa1d7700 1 -- 172.21.15.189:0/2115139250 >> v2:172.21.15.192:3300/0 conn(0x7fcda406ac70 msgr2=0x7fcda415bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:49.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.744+0000 7fcdaa1d7700 1 --2- 172.21.15.189:0/2115139250 >> v2:172.21.15.192:3300/0 conn(0x7fcda406ac70 0x7fcda415bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.744+0000 7fcdaa1d7700 1 -- 172.21.15.189:0/2115139250 >> v1:172.21.15.189:6789/0 conn(0x7fcda415c100 legacy=0x7fcda40717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:49.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.744+0000 7fcda91d5700 1 -- 172.21.15.189:0/2115139250 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3343706261 0 0) 0x7fcd900035b0 con 0x7fcda415c100 2024-04-06T12:12:49.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.744+0000 7fcdaa1d7700 1 -- 172.21.15.189:0/2115139250 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda415c8f0 con 0x7fcda4158fe0 2024-04-06T12:12:49.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.745+0000 7fcdaa1d7700 1 --2- 172.21.15.189:0/2115139250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcda4158fe0 0x7fcda406a730 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcda0009840 tx=0x7fcda0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a8a58e0d77a2719 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:49.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.745+0000 7fcda91d5700 1 -- 172.21.15.189:0/2115139250 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c0019f0 con 0x7fcda4158fe0 2024-04-06T12:12:49.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.745+0000 7fcda91d5700 1 -- 172.21.15.189:0/2115139250 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcda0004030 con 0x7fcda4158fe0 2024-04-06T12:12:49.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.745+0000 7fcda91d5700 1 -- 172.21.15.189:0/2115139250 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda00043c0 con 0x7fcda4158fe0 2024-04-06T12:12:49.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.745+0000 7fcdac43b700 1 -- 172.21.15.189:0/2115139250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcda4158fe0 msgr2=0x7fcda406a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:49.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.745+0000 7fcdac43b700 1 --2- 172.21.15.189:0/2115139250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcda4158fe0 0x7fcda406a730 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcda0009840 tx=0x7fcda0009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:49.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.745+0000 7fcdac43b700 1 -- 172.21.15.189:0/2115139250 shutdown_connections 2024-04-06T12:12:49.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.745+0000 7fcdac43b700 1 --2- 172.21.15.189:0/2115139250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcda4158fe0 0x7fcda406a730 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.745+0000 7fcdac43b700 1 --2- 172.21.15.189:0/2115139250 >> v2:172.21.15.192:3300/0 conn(0x7fcda406ac70 0x7fcda415bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.745+0000 7fcdac43b700 1 -- 172.21.15.189:0/2115139250 >> 172.21.15.189:0/2115139250 conn(0x7fcda40d00b0 msgr2=0x7fcda40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:49.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.745+0000 7fcdac43b700 1 -- 172.21.15.189:0/2115139250 shutdown_connections 2024-04-06T12:12:49.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.745+0000 7fcdac43b700 1 -- 172.21.15.189:0/2115139250 wait complete. 2024-04-06T12:12:49.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.747+0000 7fcdac43b700 1 Processor -- start 2024-04-06T12:12:49.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.747+0000 7fcdac43b700 1 -- start start 2024-04-06T12:12:49.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.747+0000 7fcdac43b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcda406ac70 0x7fcda40c9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.747+0000 7fcdac43b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcda4158fe0 0x7fcda4157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.747+0000 7fcdac43b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcda415c100 0x7fcda4141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.747+0000 7fcdac43b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda40b1ac0 con 0x7fcda415c100 2024-04-06T12:12:49.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.747+0000 7fcdac43b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda41f35a0 con 0x7fcda4158fe0 2024-04-06T12:12:49.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.747+0000 7fcdac43b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda40b31f0 con 0x7fcda406ac70 2024-04-06T12:12:49.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.748+0000 7fcdaa9d8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcda415c100 0x7fcda4141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.748+0000 7fcdaa9d8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcda415c100 0x7fcda4141fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57382/0 (socket says 172.21.15.189:57382) 2024-04-06T12:12:49.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.748+0000 7fcdaa9d8700 1 -- 172.21.15.189:0/1991915025 learned_addr learned my addr 172.21.15.189:0/1991915025 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:49.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.748+0000 7fcdaa1d7700 1 --2- 172.21.15.189:0/1991915025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcda406ac70 0x7fcda40c9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.748+0000 7fcda99d6700 1 --2- 172.21.15.189:0/1991915025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcda4158fe0 0x7fcda4157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.748+0000 7fcdaa9d8700 1 -- 172.21.15.189:0/1991915025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcda406ac70 msgr2=0x7fcda40c9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:49.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.748+0000 7fcdaa9d8700 1 --2- 172.21.15.189:0/1991915025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcda406ac70 0x7fcda40c9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.748+0000 7fcdaa9d8700 1 -- 172.21.15.189:0/1991915025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcda4158fe0 msgr2=0x7fcda4157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:49.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.748+0000 7fcdaa9d8700 1 --2- 172.21.15.189:0/1991915025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcda4158fe0 0x7fcda4157940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.748+0000 7fcdaa9d8700 1 -- 172.21.15.189:0/1991915025 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda00094a0 con 0x7fcda415c100 2024-04-06T12:12:49.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.748+0000 7fcdaa9d8700 1 --2- 172.21.15.189:0/1991915025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcda415c100 0x7fcda4141fa0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c00cca0 tx=0x7fcd9c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:49.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.748+0000 7fcd9affd700 1 -- 172.21.15.189:0/1991915025 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c00ab70 con 0x7fcda415c100 2024-04-06T12:12:49.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.748+0000 7fcd9affd700 1 -- 172.21.15.189:0/1991915025 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9c00acf0 con 0x7fcda415c100 2024-04-06T12:12:49.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.748+0000 7fcd9affd700 1 -- 172.21.15.189:0/1991915025 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c007310 con 0x7fcda415c100 2024-04-06T12:12:49.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.748+0000 7fcdac43b700 1 -- 172.21.15.189:0/1991915025 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda40b3490 con 0x7fcda415c100 2024-04-06T12:12:49.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.748+0000 7fcdac43b700 1 -- 172.21.15.189:0/1991915025 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda40b3a50 con 0x7fcda415c100 2024-04-06T12:12:49.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.749+0000 7fcdac43b700 1 -- 172.21.15.189:0/1991915025 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda4067bb0 con 0x7fcda415c100 2024-04-06T12:12:49.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.750+0000 7fcd9affd700 1 -- 172.21.15.189:0/1991915025 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcd9c007470 con 0x7fcda415c100 2024-04-06T12:12:49.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.750+0000 7fcd9affd700 1 --2- 172.21.15.189:0/1991915025 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcd8806ac90 0x7fcd8806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:49.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.750+0000 7fcd9affd700 1 -- 172.21.15.189:0/1991915025 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fcd9c0890c0 con 0x7fcda415c100 2024-04-06T12:12:49.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.750+0000 7fcdaa1d7700 1 --2- 172.21.15.189:0/1991915025 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcd8806ac90 0x7fcd8806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:49.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.750+0000 7fcdaa1d7700 1 --2- 172.21.15.189:0/1991915025 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcd8806ac90 0x7fcd8806d150 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcda00090b0 tx=0x7fcda0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:49.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.754+0000 7fcd9affd700 1 -- 172.21.15.189:0/1991915025 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fcd9c054040 con 0x7fcda415c100 2024-04-06T12:12:49.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.951+0000 7fcdac43b700 1 -- 172.21.15.189:0/1991915025 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcda40b2e90 con 0x7fcda415c100 2024-04-06T12:12:49.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.951+0000 7fcd9affd700 1 -- 172.21.15.189:0/1991915025 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcd9c00ae60 con 0x7fcda415c100 2024-04-06T12:12:49.950 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:12:49.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.953+0000 7fcdac43b700 1 -- 172.21.15.189:0/1991915025 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcd8806ac90 msgr2=0x7fcd8806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:49.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.953+0000 7fcdac43b700 1 --2- 172.21.15.189:0/1991915025 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcd8806ac90 0x7fcd8806d150 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcda00090b0 tx=0x7fcda0008030 comp rx=0 tx=0).stop 2024-04-06T12:12:49.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.953+0000 7fcdac43b700 1 -- 172.21.15.189:0/1991915025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcda415c100 msgr2=0x7fcda4141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:49.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.953+0000 7fcdac43b700 1 --2- 172.21.15.189:0/1991915025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcda415c100 0x7fcda4141fa0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c00cca0 tx=0x7fcd9c00a060 comp rx=0 tx=0).stop 2024-04-06T12:12:49.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.953+0000 7fcdac43b700 1 -- 172.21.15.189:0/1991915025 shutdown_connections 2024-04-06T12:12:49.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.953+0000 7fcdac43b700 1 --2- 172.21.15.189:0/1991915025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcda415c100 0x7fcda4141fa0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.953+0000 7fcdac43b700 1 --2- 172.21.15.189:0/1991915025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcda406ac70 0x7fcda40c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.953+0000 7fcdac43b700 1 --2- 172.21.15.189:0/1991915025 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcd8806ac90 0x7fcd8806d150 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.953+0000 7fcdac43b700 1 --2- 172.21.15.189:0/1991915025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcda4158fe0 0x7fcda4157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:49.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.953+0000 7fcdac43b700 1 -- 172.21.15.189:0/1991915025 >> 172.21.15.189:0/1991915025 conn(0x7fcda40d00b0 msgr2=0x7fcda4143930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:49.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.954+0000 7fcdac43b700 1 -- 172.21.15.189:0/1991915025 shutdown_connections 2024-04-06T12:12:49.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:49.954+0000 7fcdac43b700 1 -- 172.21.15.189:0/1991915025 wait complete. 2024-04-06T12:12:49.964 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:12:50.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.090+0000 7f65251f3700 1 Processor -- start 2024-04-06T12:12:50.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.090+0000 7f65251f3700 1 -- start start 2024-04-06T12:12:50.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.091+0000 7f65251f3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f652015a550 0x7f65200cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:50.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.091+0000 7f65251f3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65200cdbe0 con 0x7f65200cf900 2024-04-06T12:12:50.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.091+0000 7f65251f3700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f652015a940 con 0x7f65200b4d70 2024-04-06T12:12:50.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.091+0000 7f65251f3700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f652015ac00 con 0x7f652015a550 2024-04-06T12:12:50.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.091+0000 7f651ed9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f652015a550 0x7f65200cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:50.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.091+0000 7f651e59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f65200cf900 0x7f65200b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39050/0 (socket says 172.21.15.189:39050) 2024-04-06T12:12:50.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.091+0000 7f651e59c700 1 -- 172.21.15.189:0/2992880393 learned_addr learned my addr 172.21.15.189:0/2992880393 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:50.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.091+0000 7f651ed9d700 1 -- 172.21.15.189:0/2992880393 >> v1:172.21.15.192:6789/0 conn(0x7f65200b4d70 legacy=0x7f65200b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:50.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.091+0000 7f651ed9d700 1 -- 172.21.15.189:0/2992880393 >> v1:172.21.15.189:6789/0 conn(0x7f65200cf900 legacy=0x7f65200b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:50.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.091+0000 7f651dd9b700 1 -- 172.21.15.189:0/2992880393 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695688952 0 0) 0x7f65200cdbe0 con 0x7f65200cf900 2024-04-06T12:12:50.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.091+0000 7f651ed9d700 1 -- 172.21.15.189:0/2992880393 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f652015aee0 con 0x7f652015a550 2024-04-06T12:12:50.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.091+0000 7f651ed9d700 1 --2- 172.21.15.189:0/2992880393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f652015a550 0x7f65200cf3c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6510009840 tx=0x7f6510009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e08036d88130aba server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:50.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.091+0000 7f651dd9b700 1 -- 172.21.15.189:0/2992880393 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6510004230 con 0x7f652015a550 2024-04-06T12:12:50.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.091+0000 7f651dd9b700 1 -- 172.21.15.189:0/2992880393 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6510004390 con 0x7f652015a550 2024-04-06T12:12:50.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.091+0000 7f651dd9b700 1 -- 172.21.15.189:0/2992880393 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65100036a0 con 0x7f652015a550 2024-04-06T12:12:50.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.092+0000 7f65251f3700 1 -- 172.21.15.189:0/2992880393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f652015a550 msgr2=0x7f65200cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:50.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.092+0000 7f65251f3700 1 --2- 172.21.15.189:0/2992880393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f652015a550 0x7f65200cf3c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6510009840 tx=0x7f6510009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:50.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.092+0000 7f65251f3700 1 -- 172.21.15.189:0/2992880393 shutdown_connections 2024-04-06T12:12:50.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.092+0000 7f65251f3700 1 --2- 172.21.15.189:0/2992880393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f652015a550 0x7f65200cf3c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.092+0000 7f65251f3700 1 -- 172.21.15.189:0/2992880393 >> 172.21.15.189:0/2992880393 conn(0x7f65200d00b0 msgr2=0x7f65200d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:50.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.092+0000 7f65251f3700 1 -- 172.21.15.189:0/2992880393 shutdown_connections 2024-04-06T12:12:50.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.092+0000 7f65251f3700 1 -- 172.21.15.189:0/2992880393 wait complete. 2024-04-06T12:12:50.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.094+0000 7f65251f3700 1 Processor -- start 2024-04-06T12:12:50.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.094+0000 7f65251f3700 1 -- start start 2024-04-06T12:12:50.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f65251f3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65200b4d70 0x7f6520145e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:50.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f65251f3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65200cf900 0x7f6520148760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:50.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f65251f3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65201f3db0 0x7f65200b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:50.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f65251f3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65201490a0 con 0x7f65200b4d70 2024-04-06T12:12:50.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f65251f3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6520149380 con 0x7f65201f3db0 2024-04-06T12:12:50.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f65251f3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6520149660 con 0x7f65200cf900 2024-04-06T12:12:50.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f651ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65200b4d70 0x7f6520145e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:50.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f651ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65200b4d70 0x7f6520145e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57408/0 (socket says 172.21.15.189:57408) 2024-04-06T12:12:50.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f651ed9d700 1 -- 172.21.15.189:0/809916378 learned_addr learned my addr 172.21.15.189:0/809916378 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:50.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f651e59c700 1 --2- 172.21.15.189:0/809916378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65200cf900 0x7f6520148760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:50.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f651f59e700 1 --2- 172.21.15.189:0/809916378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65201f3db0 0x7f65200b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:50.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f651ed9d700 1 -- 172.21.15.189:0/809916378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65200cf900 msgr2=0x7f6520148760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:50.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f651ed9d700 1 --2- 172.21.15.189:0/809916378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65200cf900 0x7f6520148760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f651ed9d700 1 -- 172.21.15.189:0/809916378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65201f3db0 msgr2=0x7f65200b0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:50.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f651ed9d700 1 --2- 172.21.15.189:0/809916378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65201f3db0 0x7f65200b0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f651ed9d700 1 -- 172.21.15.189:0/809916378 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65100094a0 con 0x7f65200b4d70 2024-04-06T12:12:50.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f651ed9d700 1 --2- 172.21.15.189:0/809916378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65200b4d70 0x7f6520145e50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f65100090b0 tx=0x7f65100047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:50.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f65077fe700 1 -- 172.21.15.189:0/809916378 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6510011830 con 0x7f65200b4d70 2024-04-06T12:12:50.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f65077fe700 1 -- 172.21.15.189:0/809916378 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6510011990 con 0x7f65200b4d70 2024-04-06T12:12:50.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f65251f3700 1 -- 172.21.15.189:0/809916378 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65201467d0 con 0x7f65200b4d70 2024-04-06T12:12:50.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f65077fe700 1 -- 172.21.15.189:0/809916378 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6510011d40 con 0x7f65200b4d70 2024-04-06T12:12:50.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.095+0000 7f65251f3700 1 -- 172.21.15.189:0/809916378 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6520146390 con 0x7f65200b4d70 2024-04-06T12:12:50.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.096+0000 7f65251f3700 1 -- 172.21.15.189:0/809916378 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6520146cf0 con 0x7f65200b4d70 2024-04-06T12:12:50.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.097+0000 7f65077fe700 1 -- 172.21.15.189:0/809916378 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f651001b440 con 0x7f65200b4d70 2024-04-06T12:12:50.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.097+0000 7f65077fe700 1 --2- 172.21.15.189:0/809916378 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f650c06ac90 0x7f650c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:50.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.097+0000 7f651e59c700 1 --2- 172.21.15.189:0/809916378 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f650c06ac90 0x7f650c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:50.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.097+0000 7f65077fe700 1 -- 172.21.15.189:0/809916378 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f651008af70 con 0x7f65200b4d70 2024-04-06T12:12:50.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.098+0000 7f651e59c700 1 --2- 172.21.15.189:0/809916378 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f650c06ac90 0x7f650c06d150 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f65200aeae0 tx=0x7f6514007be0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:50.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.102+0000 7f65077fe700 1 -- 172.21.15.189:0/809916378 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6510055ef0 con 0x7f65200b4d70 2024-04-06T12:12:50.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.293+0000 7f65251f3700 1 -- 172.21.15.189:0/809916378 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6520067bb0 con 0x7f65200b4d70 2024-04-06T12:12:50.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.293+0000 7f65077fe700 1 -- 172.21.15.189:0/809916378 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6510059510 con 0x7f65200b4d70 2024-04-06T12:12:50.291 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:12:50.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.295+0000 7f65251f3700 1 -- 172.21.15.189:0/809916378 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f650c06ac90 msgr2=0x7f650c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:50.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.295+0000 7f65251f3700 1 --2- 172.21.15.189:0/809916378 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f650c06ac90 0x7f650c06d150 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f65200aeae0 tx=0x7f6514007be0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.295+0000 7f65251f3700 1 -- 172.21.15.189:0/809916378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65200b4d70 msgr2=0x7f6520145e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:50.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.295+0000 7f65251f3700 1 --2- 172.21.15.189:0/809916378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65200b4d70 0x7f6520145e50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f65100090b0 tx=0x7f65100047c0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.295+0000 7f65251f3700 1 -- 172.21.15.189:0/809916378 shutdown_connections 2024-04-06T12:12:50.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.295+0000 7f65251f3700 1 --2- 172.21.15.189:0/809916378 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f650c06ac90 0x7f650c06d150 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.295+0000 7f65251f3700 1 --2- 172.21.15.189:0/809916378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65201f3db0 0x7f65200b0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.295+0000 7f65251f3700 1 --2- 172.21.15.189:0/809916378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65200b4d70 0x7f6520145e50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.295+0000 7f65251f3700 1 --2- 172.21.15.189:0/809916378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65200cf900 0x7f6520148760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.295+0000 7f65251f3700 1 -- 172.21.15.189:0/809916378 >> 172.21.15.189:0/809916378 conn(0x7f65200d00b0 msgr2=0x7f652006edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:50.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.295+0000 7f65251f3700 1 -- 172.21.15.189:0/809916378 shutdown_connections 2024-04-06T12:12:50.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.295+0000 7f65251f3700 1 -- 172.21.15.189:0/809916378 wait complete. 2024-04-06T12:12:50.305 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-06T12:12:50.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.435+0000 7f21a4ad2700 1 Processor -- start 2024-04-06T12:12:50.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.435+0000 7f21a4ad2700 1 -- start start 2024-04-06T12:12:50.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.436+0000 7f21a4ad2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f21a00d11b0 0x7f21a00d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:50.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.436+0000 7f21a4ad2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f21a00cb870 0x7f21a015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:50.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.436+0000 7f21a4ad2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21a00cfa00 0x7f21a00cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:50.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.436+0000 7f21a4ad2700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a00ccdc0 con 0x7f21a00cb870 2024-04-06T12:12:50.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.436+0000 7f21a4ad2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a00d0b50 con 0x7f21a00d11b0 2024-04-06T12:12:50.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.436+0000 7f21a4ad2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a00d1ad0 con 0x7f21a00cfa00 2024-04-06T12:12:50.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.436+0000 7f219dd9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f21a00cb870 0x7f21a015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:50.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.436+0000 7f219dd9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f21a00cb870 0x7f21a015c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57414/0 (socket says 172.21.15.189:57414) 2024-04-06T12:12:50.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.436+0000 7f219dd9b700 1 -- 172.21.15.189:0/2220143736 learned_addr learned my addr 172.21.15.189:0/2220143736 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:50.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.437+0000 7f219ed9d700 1 --2- 172.21.15.189:0/2220143736 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21a00cfa00 0x7f21a00cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:50.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.437+0000 7f219e59c700 1 --2- 172.21.15.189:0/2220143736 >> v2:172.21.15.192:3300/0 conn(0x7f21a00d11b0 0x7f21a00d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:50.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.437+0000 7f219dd9b700 1 -- 172.21.15.189:0/2220143736 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21a00cfa00 msgr2=0x7f21a00cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:50.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.437+0000 7f219dd9b700 1 --2- 172.21.15.189:0/2220143736 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21a00cfa00 0x7f21a00cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.437+0000 7f219dd9b700 1 -- 172.21.15.189:0/2220143736 >> v2:172.21.15.192:3300/0 conn(0x7f21a00d11b0 msgr2=0x7f21a00d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:50.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.437+0000 7f219dd9b700 1 --2- 172.21.15.189:0/2220143736 >> v2:172.21.15.192:3300/0 conn(0x7f21a00d11b0 0x7f21a00d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.437+0000 7f219dd9b700 1 -- 172.21.15.189:0/2220143736 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a00cbd80 con 0x7f21a00cb870 2024-04-06T12:12:50.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.437+0000 7f219e59c700 1 --2- 172.21.15.189:0/2220143736 >> v2:172.21.15.192:3300/0 conn(0x7f21a00d11b0 0x7f21a00d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:50.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.437+0000 7f219dd9b700 1 --2- 172.21.15.189:0/2220143736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21a00cb870 0x7f21a015c280 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f21a00ced00 tx=0x7f2190009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705a57ac34fc1735 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:50.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.437+0000 7f219d59a700 1 -- 172.21.15.189:0/2220143736 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2190011830 con 0x7f21a00cb870 2024-04-06T12:12:50.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.437+0000 7f219d59a700 1 -- 172.21.15.189:0/2220143736 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2190011990 con 0x7f21a00cb870 2024-04-06T12:12:50.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.437+0000 7f219d59a700 1 -- 172.21.15.189:0/2220143736 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2190011d20 con 0x7f21a00cb870 2024-04-06T12:12:50.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.438+0000 7f21a4ad2700 1 -- 172.21.15.189:0/2220143736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21a00cb870 msgr2=0x7f21a015c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:50.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.438+0000 7f21a4ad2700 1 --2- 172.21.15.189:0/2220143736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21a00cb870 0x7f21a015c280 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f21a00ced00 tx=0x7f2190009bd0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.438+0000 7f21a4ad2700 1 -- 172.21.15.189:0/2220143736 shutdown_connections 2024-04-06T12:12:50.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.438+0000 7f21a4ad2700 1 --2- 172.21.15.189:0/2220143736 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21a00cfa00 0x7f21a00cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.438+0000 7f21a4ad2700 1 --2- 172.21.15.189:0/2220143736 >> v2:172.21.15.192:3300/0 conn(0x7f21a00d11b0 0x7f21a00d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.438+0000 7f21a4ad2700 1 --2- 172.21.15.189:0/2220143736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21a00cb870 0x7f21a015c280 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.438+0000 7f21a4ad2700 1 -- 172.21.15.189:0/2220143736 >> 172.21.15.189:0/2220143736 conn(0x7f21a00b2db0 msgr2=0x7f21a00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:50.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.438+0000 7f21a4ad2700 1 -- 172.21.15.189:0/2220143736 shutdown_connections 2024-04-06T12:12:50.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.438+0000 7f21a4ad2700 1 -- 172.21.15.189:0/2220143736 wait complete. 2024-04-06T12:12:50.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.439+0000 7f21a4ad2700 1 Processor -- start 2024-04-06T12:12:50.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.439+0000 7f21a4ad2700 1 -- start start 2024-04-06T12:12:50.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.440+0000 7f21a4ad2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21a00cb870 0x7f21a00ca350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:50.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.440+0000 7f21a4ad2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21a00cfa00 0x7f21a01f0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:50.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.440+0000 7f21a4ad2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21a00d11b0 0x7f21a00c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:50.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.440+0000 7f21a4ad2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a0156140 con 0x7f21a00d11b0 2024-04-06T12:12:50.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.440+0000 7f21a4ad2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a00c93e0 con 0x7f21a00cfa00 2024-04-06T12:12:50.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.440+0000 7f21a4ad2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a00c8b40 con 0x7f21a00cb870 2024-04-06T12:12:50.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.440+0000 7f219ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21a00d11b0 0x7f21a00c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:50.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.440+0000 7f219ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21a00d11b0 0x7f21a00c82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57420/0 (socket says 172.21.15.189:57420) 2024-04-06T12:12:50.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.440+0000 7f219ed9d700 1 -- 172.21.15.189:0/3538851722 learned_addr learned my addr 172.21.15.189:0/3538851722 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:50.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.440+0000 7f219e59c700 1 --2- 172.21.15.189:0/3538851722 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21a00cb870 0x7f21a00ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:50.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.440+0000 7f219dd9b700 1 --2- 172.21.15.189:0/3538851722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21a00cfa00 0x7f21a01f0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:50.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.440+0000 7f219ed9d700 1 -- 172.21.15.189:0/3538851722 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21a00cb870 msgr2=0x7f21a00ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:50.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.440+0000 7f219ed9d700 1 --2- 172.21.15.189:0/3538851722 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21a00cb870 0x7f21a00ca350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.440+0000 7f219ed9d700 1 -- 172.21.15.189:0/3538851722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21a00cfa00 msgr2=0x7f21a01f0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:50.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.440+0000 7f219ed9d700 1 --2- 172.21.15.189:0/3538851722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21a00cfa00 0x7f21a01f0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.440+0000 7f219ed9d700 1 -- 172.21.15.189:0/3538851722 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2190009550 con 0x7f21a00d11b0 2024-04-06T12:12:50.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.440+0000 7f219ed9d700 1 --2- 172.21.15.189:0/3538851722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21a00d11b0 0x7f21a00c82f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f219800eed0 tx=0x7f219800ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:50.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.440+0000 7f218effd700 1 -- 172.21.15.189:0/3538851722 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2198010040 con 0x7f21a00d11b0 2024-04-06T12:12:50.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.441+0000 7f218effd700 1 -- 172.21.15.189:0/3538851722 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219800b9f0 con 0x7f21a00d11b0 2024-04-06T12:12:50.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.441+0000 7f21a4ad2700 1 -- 172.21.15.189:0/3538851722 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a00c8e20 con 0x7f21a00d11b0 2024-04-06T12:12:50.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.441+0000 7f218effd700 1 -- 172.21.15.189:0/3538851722 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219800bda0 con 0x7f21a00d11b0 2024-04-06T12:12:50.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.441+0000 7f21a4ad2700 1 -- 172.21.15.189:0/3538851722 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a00c8830 con 0x7f21a00d11b0 2024-04-06T12:12:50.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.442+0000 7f21a4ad2700 1 -- 172.21.15.189:0/3538851722 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a00b38a0 con 0x7f21a00d11b0 2024-04-06T12:12:50.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.442+0000 7f218effd700 1 -- 172.21.15.189:0/3538851722 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2198014400 con 0x7f21a00d11b0 2024-04-06T12:12:50.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.442+0000 7f218effd700 1 --2- 172.21.15.189:0/3538851722 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f218006ac90 0x7f218006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:50.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.442+0000 7f218effd700 1 -- 172.21.15.189:0/3538851722 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2198089880 con 0x7f21a00d11b0 2024-04-06T12:12:50.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.442+0000 7f219e59c700 1 --2- 172.21.15.189:0/3538851722 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f218006ac90 0x7f218006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:50.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.443+0000 7f219e59c700 1 --2- 172.21.15.189:0/3538851722 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f218006ac90 0x7f218006d150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f21940099d0 tx=0x7f2194008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:50.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.447+0000 7f218effd700 1 -- 172.21.15.189:0/3538851722 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2198054800 con 0x7f21a00d11b0 2024-04-06T12:12:50.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.644+0000 7f21a4ad2700 1 -- 172.21.15.189:0/3538851722 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f21a0074990 con 0x7f21a00d11b0 2024-04-06T12:12:50.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.645+0000 7f218effd700 1 -- 172.21.15.189:0/3538851722 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2198018020 con 0x7f21a00d11b0 2024-04-06T12:12:50.643 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:12:50.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.647+0000 7f21a4ad2700 1 -- 172.21.15.189:0/3538851722 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f218006ac90 msgr2=0x7f218006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:50.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.647+0000 7f21a4ad2700 1 --2- 172.21.15.189:0/3538851722 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f218006ac90 0x7f218006d150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f21940099d0 tx=0x7f2194008040 comp rx=0 tx=0).stop 2024-04-06T12:12:50.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.647+0000 7f21a4ad2700 1 -- 172.21.15.189:0/3538851722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21a00d11b0 msgr2=0x7f21a00c82f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:50.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.647+0000 7f21a4ad2700 1 --2- 172.21.15.189:0/3538851722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21a00d11b0 0x7f21a00c82f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f219800eed0 tx=0x7f219800ef00 comp rx=0 tx=0).stop 2024-04-06T12:12:50.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.647+0000 7f21a4ad2700 1 -- 172.21.15.189:0/3538851722 shutdown_connections 2024-04-06T12:12:50.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.647+0000 7f21a4ad2700 1 --2- 172.21.15.189:0/3538851722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21a00d11b0 0x7f21a00c82f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.647+0000 7f21a4ad2700 1 --2- 172.21.15.189:0/3538851722 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21a00cb870 0x7f21a00ca350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.647+0000 7f21a4ad2700 1 --2- 172.21.15.189:0/3538851722 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f218006ac90 0x7f218006d150 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.647+0000 7f21a4ad2700 1 --2- 172.21.15.189:0/3538851722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21a00cfa00 0x7f21a01f0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.647+0000 7f21a4ad2700 1 -- 172.21.15.189:0/3538851722 >> 172.21.15.189:0/3538851722 conn(0x7f21a00b2db0 msgr2=0x7f21a00b3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:50.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.647+0000 7f21a4ad2700 1 -- 172.21.15.189:0/3538851722 shutdown_connections 2024-04-06T12:12:50.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.647+0000 7f21a4ad2700 1 -- 172.21.15.189:0/3538851722 wait complete. 2024-04-06T12:12:50.657 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:50.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.780+0000 7f5a5f5a7700 1 Processor -- start 2024-04-06T12:12:50.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.780+0000 7f5a5f5a7700 1 -- start start 2024-04-06T12:12:50.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.781+0000 7f5a5f5a7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5a58159b20 0x7f5a5806b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:50.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.781+0000 7f5a5f5a7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a580cdc60 0x7f5a580ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:50.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.781+0000 7f5a5f5a7700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5806a8d0 con 0x7f5a5806e470 2024-04-06T12:12:50.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.781+0000 7f5a5f5a7700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a580ce620 con 0x7f5a58159b20 2024-04-06T12:12:50.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.781+0000 7f5a5f5a7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a58159f00 con 0x7f5a580cdc60 2024-04-06T12:12:50.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.781+0000 7f5a5d343700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5a58159b20 0x7f5a5806b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:50.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.781+0000 7f5a5d343700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5a58159b20 0x7f5a5806b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57424/0 (socket says 172.21.15.189:57424) 2024-04-06T12:12:50.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.781+0000 7f5a5d343700 1 -- 172.21.15.189:0/2821545487 learned_addr learned my addr 172.21.15.189:0/2821545487 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:50.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.782+0000 7f5a5cb42700 1 --2- 172.21.15.189:0/2821545487 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a580cdc60 0x7f5a580ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:50.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.782+0000 7f5a5d343700 1 -- 172.21.15.189:0/2821545487 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a580cdc60 msgr2=0x7f5a580ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:50.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.782+0000 7f5a5d343700 1 --2- 172.21.15.189:0/2821545487 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a580cdc60 0x7f5a580ce0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.782+0000 7f5a5d343700 1 -- 172.21.15.189:0/2821545487 >> v1:172.21.15.192:6789/0 conn(0x7f5a5806e470 legacy=0x7f5a5806e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:50.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.782+0000 7f5a5d343700 1 -- 172.21.15.189:0/2821545487 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5806abb0 con 0x7f5a58159b20 2024-04-06T12:12:50.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.782+0000 7f5a5d343700 1 --2- 172.21.15.189:0/2821545487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a58159b20 0x7f5a5806b970 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5a540098f0 tx=0x7f5a54009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d62bd4ca5698f77 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:50.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.782+0000 7f5a4ffff700 1 -- 172.21.15.189:0/2821545487 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a54004230 con 0x7f5a58159b20 2024-04-06T12:12:50.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.782+0000 7f5a4ffff700 1 -- 172.21.15.189:0/2821545487 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a54004390 con 0x7f5a58159b20 2024-04-06T12:12:50.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.782+0000 7f5a4ffff700 1 -- 172.21.15.189:0/2821545487 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a540036a0 con 0x7f5a58159b20 2024-04-06T12:12:50.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.782+0000 7f5a5f5a7700 1 -- 172.21.15.189:0/2821545487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a58159b20 msgr2=0x7f5a5806b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:50.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.782+0000 7f5a5f5a7700 1 --2- 172.21.15.189:0/2821545487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a58159b20 0x7f5a5806b970 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5a540098f0 tx=0x7f5a54009c00 comp rx=0 tx=0).stop 2024-04-06T12:12:50.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.783+0000 7f5a5f5a7700 1 -- 172.21.15.189:0/2821545487 shutdown_connections 2024-04-06T12:12:50.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.783+0000 7f5a5f5a7700 1 --2- 172.21.15.189:0/2821545487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a58159b20 0x7f5a5806b970 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.783+0000 7f5a5f5a7700 1 --2- 172.21.15.189:0/2821545487 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a580cdc60 0x7f5a580ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.783+0000 7f5a5f5a7700 1 -- 172.21.15.189:0/2821545487 >> 172.21.15.189:0/2821545487 conn(0x7f5a580cc290 msgr2=0x7f5a580cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:50.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.783+0000 7f5a5f5a7700 1 -- 172.21.15.189:0/2821545487 shutdown_connections 2024-04-06T12:12:50.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.783+0000 7f5a5f5a7700 1 -- 172.21.15.189:0/2821545487 wait complete. 2024-04-06T12:12:50.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.784+0000 7f5a5f5a7700 1 Processor -- start 2024-04-06T12:12:50.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.784+0000 7f5a5f5a7700 1 -- start start 2024-04-06T12:12:50.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.784+0000 7f5a5f5a7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a5806e470 0x7f5a581511f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:50.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.784+0000 7f5a5f5a7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a580cdc60 0x7f5a5815b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:50.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.784+0000 7f5a5f5a7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a58159b20 0x7f5a5815ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:50.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.784+0000 7f5a5f5a7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a58151730 con 0x7f5a5806e470 2024-04-06T12:12:50.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.785+0000 7f5a5f5a7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a580c4bf0 con 0x7f5a58159b20 2024-04-06T12:12:50.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.785+0000 7f5a5f5a7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a580c3fe0 con 0x7f5a580cdc60 2024-04-06T12:12:50.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.785+0000 7f5a5d343700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a5806e470 0x7f5a581511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:50.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.785+0000 7f5a5d343700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a5806e470 0x7f5a581511f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57434/0 (socket says 172.21.15.189:57434) 2024-04-06T12:12:50.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.785+0000 7f5a5d343700 1 -- 172.21.15.189:0/2751713167 learned_addr learned my addr 172.21.15.189:0/2751713167 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:50.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.785+0000 7f5a5cb42700 1 --2- 172.21.15.189:0/2751713167 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a580cdc60 0x7f5a5815b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:50.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.785+0000 7f5a5db44700 1 --2- 172.21.15.189:0/2751713167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a58159b20 0x7f5a5815ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:50.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.785+0000 7f5a5d343700 1 -- 172.21.15.189:0/2751713167 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a580cdc60 msgr2=0x7f5a5815b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:50.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.785+0000 7f5a5d343700 1 --2- 172.21.15.189:0/2751713167 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a580cdc60 0x7f5a5815b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.785+0000 7f5a5d343700 1 -- 172.21.15.189:0/2751713167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a58159b20 msgr2=0x7f5a5815ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:50.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.785+0000 7f5a5d343700 1 --2- 172.21.15.189:0/2751713167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a58159b20 0x7f5a5815ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.785+0000 7f5a5d343700 1 -- 172.21.15.189:0/2751713167 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a54009550 con 0x7f5a5806e470 2024-04-06T12:12:50.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.785+0000 7f5a5d343700 1 --2- 172.21.15.189:0/2751713167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a5806e470 0x7f5a581511f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5a54009130 tx=0x7f5a54004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:50.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.785+0000 7f5a4dffb700 1 -- 172.21.15.189:0/2751713167 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5401d070 con 0x7f5a5806e470 2024-04-06T12:12:50.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.785+0000 7f5a4dffb700 1 -- 172.21.15.189:0/2751713167 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a540045e0 con 0x7f5a5806e470 2024-04-06T12:12:50.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.785+0000 7f5a4dffb700 1 -- 172.21.15.189:0/2751713167 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5400bab0 con 0x7f5a5806e470 2024-04-06T12:12:50.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.785+0000 7f5a5f5a7700 1 -- 172.21.15.189:0/2751713167 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a580c42c0 con 0x7f5a5806e470 2024-04-06T12:12:50.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.786+0000 7f5a5f5a7700 1 -- 172.21.15.189:0/2751713167 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a580c4880 con 0x7f5a5806e470 2024-04-06T12:12:50.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.786+0000 7f5a5f5a7700 1 -- 172.21.15.189:0/2751713167 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a58074990 con 0x7f5a5806e470 2024-04-06T12:12:50.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.787+0000 7f5a4dffb700 1 -- 172.21.15.189:0/2751713167 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5a5400bc10 con 0x7f5a5806e470 2024-04-06T12:12:50.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.787+0000 7f5a4dffb700 1 --2- 172.21.15.189:0/2751713167 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5a3c06ac90 0x7f5a3c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:50.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.787+0000 7f5a4dffb700 1 -- 172.21.15.189:0/2751713167 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5a5408aa90 con 0x7f5a5806e470 2024-04-06T12:12:50.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.787+0000 7f5a5cb42700 1 --2- 172.21.15.189:0/2751713167 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5a3c06ac90 0x7f5a3c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:50.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.788+0000 7f5a5cb42700 1 --2- 172.21.15.189:0/2751713167 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5a3c06ac90 0x7f5a3c06d150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5a58066720 tx=0x7f5a48008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:50.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.792+0000 7f5a4dffb700 1 -- 172.21.15.189:0/2751713167 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5a54055b40 con 0x7f5a5806e470 2024-04-06T12:12:50.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.987+0000 7f5a5f5a7700 1 -- 172.21.15.189:0/2751713167 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a580c4a10 con 0x7f5a5806e470 2024-04-06T12:12:50.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.988+0000 7f5a4dffb700 1 -- 172.21.15.189:0/2751713167 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f5a54059160 con 0x7f5a5806e470 2024-04-06T12:12:50.986 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:50.986 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:08.649170+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:50.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.990+0000 7f5a5f5a7700 1 -- 172.21.15.189:0/2751713167 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5a3c06ac90 msgr2=0x7f5a3c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:50.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.990+0000 7f5a5f5a7700 1 --2- 172.21.15.189:0/2751713167 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5a3c06ac90 0x7f5a3c06d150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5a58066720 tx=0x7f5a48008040 comp rx=0 tx=0).stop 2024-04-06T12:12:50.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.990+0000 7f5a5f5a7700 1 -- 172.21.15.189:0/2751713167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a5806e470 msgr2=0x7f5a581511f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:50.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.990+0000 7f5a5f5a7700 1 --2- 172.21.15.189:0/2751713167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a5806e470 0x7f5a581511f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5a54009130 tx=0x7f5a54004750 comp rx=0 tx=0).stop 2024-04-06T12:12:50.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.990+0000 7f5a5f5a7700 1 -- 172.21.15.189:0/2751713167 shutdown_connections 2024-04-06T12:12:50.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.990+0000 7f5a5f5a7700 1 --2- 172.21.15.189:0/2751713167 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5a3c06ac90 0x7f5a3c06d150 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.990+0000 7f5a5f5a7700 1 --2- 172.21.15.189:0/2751713167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a58159b20 0x7f5a5815ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.990+0000 7f5a5f5a7700 1 --2- 172.21.15.189:0/2751713167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a5806e470 0x7f5a581511f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.990+0000 7f5a5f5a7700 1 --2- 172.21.15.189:0/2751713167 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a580cdc60 0x7f5a5815b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:50.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.990+0000 7f5a5f5a7700 1 -- 172.21.15.189:0/2751713167 >> 172.21.15.189:0/2751713167 conn(0x7f5a580cc290 msgr2=0x7f5a580cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:50.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.990+0000 7f5a5f5a7700 1 -- 172.21.15.189:0/2751713167 shutdown_connections 2024-04-06T12:12:50.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:50.990+0000 7f5a5f5a7700 1 -- 172.21.15.189:0/2751713167 wait complete. 2024-04-06T12:12:51.000 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:12:02.280866+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:12:03.156593+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-06T12:12:06.237184+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-06T12:12:51.001 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:12:51.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.128+0000 7f6cad6d3700 1 Processor -- start 2024-04-06T12:12:51.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.128+0000 7f6cad6d3700 1 -- start start 2024-04-06T12:12:51.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.128+0000 7f6cad6d3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6ca815d590 0x7f6ca80b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.128+0000 7f6cad6d3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca80b4ed0 0x7f6ca80b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.128+0000 7f6cad6d3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca80b5cc0 con 0x7f6ca80b6580 2024-04-06T12:12:51.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.128+0000 7f6cad6d3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca80b5890 con 0x7f6ca815d590 2024-04-06T12:12:51.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.128+0000 7f6cad6d3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca80d1500 con 0x7f6ca80b4ed0 2024-04-06T12:12:51.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca77fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6ca80b6580 0x7f6ca80b6a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39062/0 (socket says 172.21.15.189:39062) 2024-04-06T12:12:51.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca77fe700 1 -- 172.21.15.189:0/3267809979 learned_addr learned my addr 172.21.15.189:0/3267809979 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:51.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca67fc700 1 --2- 172.21.15.189:0/3267809979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca80b4ed0 0x7f6ca80b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca6ffd700 1 --2- 172.21.15.189:0/3267809979 >> v2:172.21.15.192:3300/0 conn(0x7f6ca815d590 0x7f6ca80b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca5ffb700 1 -- 172.21.15.189:0/3267809979 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107848043 0 0) 0x7f6ca80b5cc0 con 0x7f6ca80b6580 2024-04-06T12:12:51.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca5ffb700 1 -- 172.21.15.189:0/3267809979 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c940035b0 con 0x7f6ca80b6580 2024-04-06T12:12:51.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca5ffb700 1 -- 172.21.15.189:0/3267809979 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 155815154 0 0) 0x7f6c940035b0 con 0x7f6ca80b6580 2024-04-06T12:12:51.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca5ffb700 1 -- 172.21.15.189:0/3267809979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca80b4ed0 msgr2=0x7f6ca80b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:51.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca5ffb700 1 --2- 172.21.15.189:0/3267809979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca80b4ed0 0x7f6ca80b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca5ffb700 1 -- 172.21.15.189:0/3267809979 >> v2:172.21.15.192:3300/0 conn(0x7f6ca815d590 msgr2=0x7f6ca80b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:51.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca5ffb700 1 --2- 172.21.15.189:0/3267809979 >> v2:172.21.15.192:3300/0 conn(0x7f6ca815d590 0x7f6ca80b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca5ffb700 1 -- 172.21.15.189:0/3267809979 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca80d17e0 con 0x7f6ca80b6580 2024-04-06T12:12:51.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca5ffb700 1 -- 172.21.15.189:0/3267809979 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f6ca00019f0 con 0x7f6ca80b6580 2024-04-06T12:12:51.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca5ffb700 1 -- 172.21.15.189:0/3267809979 >> v1:172.21.15.189:6789/0 conn(0x7f6ca80b6580 legacy=0x7f6ca80b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:51.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca5ffb700 1 --2- 172.21.15.189:0/3267809979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c94003a00 0x7f6c94003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca5ffb700 1 --2- 172.21.15.189:0/3267809979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca80b4ed0 0x7f6c94004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca5ffb700 1 --2- 172.21.15.189:0/3267809979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c94004d90 0x7f6c94005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca5ffb700 1 -- 172.21.15.189:0/3267809979 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca80b5890 con 0x7f6c94003a00 2024-04-06T12:12:51.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca5ffb700 1 -- 172.21.15.189:0/3267809979 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca80d1500 con 0x7f6c94004d90 2024-04-06T12:12:51.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca5ffb700 1 -- 172.21.15.189:0/3267809979 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9400ba10 con 0x7f6ca80b4ed0 2024-04-06T12:12:51.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca6ffd700 1 --2- 172.21.15.189:0/3267809979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c94003a00 0x7f6c94003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca67fc700 1 --2- 172.21.15.189:0/3267809979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca80b4ed0 0x7f6c94004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca77fe700 1 --2- 172.21.15.189:0/3267809979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c94004d90 0x7f6c94005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca6ffd700 1 -- 172.21.15.189:0/3267809979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca80b4ed0 msgr2=0x7f6c94004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:51.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca6ffd700 1 --2- 172.21.15.189:0/3267809979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca80b4ed0 0x7f6c94004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca6ffd700 1 -- 172.21.15.189:0/3267809979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c94004d90 msgr2=0x7f6c94005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:51.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca6ffd700 1 --2- 172.21.15.189:0/3267809979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c94004d90 0x7f6c94005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.129+0000 7f6ca6ffd700 1 -- 172.21.15.189:0/3267809979 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c940035b0 con 0x7f6c94003a00 2024-04-06T12:12:51.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.130+0000 7f6ca6ffd700 1 --2- 172.21.15.189:0/3267809979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c94003a00 0x7f6c94003e20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009b60 tx=0x7f6c94005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ada1205aa42468f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:51.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.130+0000 7f6ca5ffb700 1 -- 172.21.15.189:0/3267809979 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9800e660 con 0x7f6c94003a00 2024-04-06T12:12:51.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.130+0000 7f6ca5ffb700 1 -- 172.21.15.189:0/3267809979 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9800e7c0 con 0x7f6c94003a00 2024-04-06T12:12:51.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.130+0000 7f6ca5ffb700 1 -- 172.21.15.189:0/3267809979 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9800eb30 con 0x7f6c94003a00 2024-04-06T12:12:51.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.130+0000 7f6cad6d3700 1 -- 172.21.15.189:0/3267809979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c94003a00 msgr2=0x7f6c94003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:51.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.130+0000 7f6cad6d3700 1 --2- 172.21.15.189:0/3267809979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c94003a00 0x7f6c94003e20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009b60 tx=0x7f6c94005880 comp rx=0 tx=0).stop 2024-04-06T12:12:51.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.130+0000 7f6ca77fe700 1 -- 172.21.15.189:0/3267809979 reap_dead start 2024-04-06T12:12:51.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.130+0000 7f6cad6d3700 1 -- 172.21.15.189:0/3267809979 shutdown_connections 2024-04-06T12:12:51.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.130+0000 7f6cad6d3700 1 -- 172.21.15.189:0/3267809979 >> 172.21.15.189:0/3267809979 conn(0x7f6ca806bbc0 msgr2=0x7f6ca806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:51.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.130+0000 7f6cad6d3700 1 -- 172.21.15.189:0/3267809979 shutdown_connections 2024-04-06T12:12:51.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.130+0000 7f6cad6d3700 1 -- 172.21.15.189:0/3267809979 wait complete. 2024-04-06T12:12:51.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.132+0000 7f6cad6d3700 1 Processor -- start 2024-04-06T12:12:51.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.132+0000 7f6cad6d3700 1 -- start start 2024-04-06T12:12:51.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.132+0000 7f6cad6d3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c94003a00 0x7f6ca81538f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.132+0000 7f6cad6d3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca815dfa0 0x7f6ca80b7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.132+0000 7f6cad6d3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca80b7e50 0x7f6ca80cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.132+0000 7f6cad6d3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca80b8300 con 0x7f6c94003a00 2024-04-06T12:12:51.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.132+0000 7f6cad6d3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca81542f0 con 0x7f6ca815dfa0 2024-04-06T12:12:51.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.132+0000 7f6cad6d3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca81545d0 con 0x7f6ca80b7e50 2024-04-06T12:12:51.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.133+0000 7f6ca6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c94003a00 0x7f6ca81538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.133+0000 7f6ca6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c94003a00 0x7f6ca81538f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57448/0 (socket says 172.21.15.189:57448) 2024-04-06T12:12:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.133+0000 7f6ca6ffd700 1 -- 172.21.15.189:0/1691409227 learned_addr learned my addr 172.21.15.189:0/1691409227 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.133+0000 7f6ca67fc700 1 --2- 172.21.15.189:0/1691409227 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca815dfa0 0x7f6ca80b7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.133+0000 7f6ca77fe700 1 --2- 172.21.15.189:0/1691409227 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca80b7e50 0x7f6ca80cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.133+0000 7f6ca6ffd700 1 -- 172.21.15.189:0/1691409227 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca80b7e50 msgr2=0x7f6ca80cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.133+0000 7f6ca6ffd700 1 --2- 172.21.15.189:0/1691409227 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca80b7e50 0x7f6ca80cc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.133+0000 7f6ca6ffd700 1 -- 172.21.15.189:0/1691409227 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca815dfa0 msgr2=0x7f6ca80b7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.133+0000 7f6ca6ffd700 1 --2- 172.21.15.189:0/1691409227 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca815dfa0 0x7f6ca80b7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.133+0000 7f6ca6ffd700 1 -- 172.21.15.189:0/1691409227 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca81548b0 con 0x7f6c94003a00 2024-04-06T12:12:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.133+0000 7f6ca6ffd700 1 --2- 172.21.15.189:0/1691409227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c94003a00 0x7f6ca81538f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6c980090b0 tx=0x7f6c980047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.133+0000 7f6c9f7fe700 1 -- 172.21.15.189:0/1691409227 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9800e660 con 0x7f6c94003a00 2024-04-06T12:12:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.133+0000 7f6c9f7fe700 1 -- 172.21.15.189:0/1691409227 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c98003ad0 con 0x7f6c94003a00 2024-04-06T12:12:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.133+0000 7f6cad6d3700 1 -- 172.21.15.189:0/1691409227 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca8154b90 con 0x7f6c94003a00 2024-04-06T12:12:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.133+0000 7f6c9f7fe700 1 -- 172.21.15.189:0/1691409227 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98003e80 con 0x7f6c94003a00 2024-04-06T12:12:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.133+0000 7f6cad6d3700 1 -- 172.21.15.189:0/1691409227 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca8153e30 con 0x7f6c94003a00 2024-04-06T12:12:51.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.134+0000 7f6cad6d3700 1 -- 172.21.15.189:0/1691409227 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca81550b0 con 0x7f6c94003a00 2024-04-06T12:12:51.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.135+0000 7f6c9f7fe700 1 -- 172.21.15.189:0/1691409227 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6c98013470 con 0x7f6c94003a00 2024-04-06T12:12:51.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.135+0000 7f6c9f7fe700 1 --2- 172.21.15.189:0/1691409227 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c8806ac90 0x7f6c8806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.135+0000 7f6c9f7fe700 1 -- 172.21.15.189:0/1691409227 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6c9808ad80 con 0x7f6c94003a00 2024-04-06T12:12:51.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.135+0000 7f6ca67fc700 1 --2- 172.21.15.189:0/1691409227 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c8806ac90 0x7f6c8806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.135+0000 7f6ca67fc700 1 --2- 172.21.15.189:0/1691409227 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c8806ac90 0x7f6c8806d150 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6ca80b7030 tx=0x7f6c90008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:51.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.139+0000 7f6c9f7fe700 1 -- 172.21.15.189:0/1691409227 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6c98055e20 con 0x7f6c94003a00 2024-04-06T12:12:51.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.330+0000 7f6cad6d3700 1 -- 172.21.15.189:0/1691409227 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6ca8074990 con 0x7f6c94003a00 2024-04-06T12:12:51.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.331+0000 7f6c9f7fe700 1 -- 172.21.15.189:0/1691409227 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6c980170a0 con 0x7f6c94003a00 2024-04-06T12:12:51.329 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:12:51.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.333+0000 7f6cad6d3700 1 -- 172.21.15.189:0/1691409227 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c8806ac90 msgr2=0x7f6c8806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:51.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.333+0000 7f6cad6d3700 1 --2- 172.21.15.189:0/1691409227 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c8806ac90 0x7f6c8806d150 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6ca80b7030 tx=0x7f6c90008040 comp rx=0 tx=0).stop 2024-04-06T12:12:51.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.333+0000 7f6cad6d3700 1 -- 172.21.15.189:0/1691409227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c94003a00 msgr2=0x7f6ca81538f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:51.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.333+0000 7f6cad6d3700 1 --2- 172.21.15.189:0/1691409227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c94003a00 0x7f6ca81538f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6c980090b0 tx=0x7f6c980047c0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.333+0000 7f6cad6d3700 1 -- 172.21.15.189:0/1691409227 shutdown_connections 2024-04-06T12:12:51.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.333+0000 7f6cad6d3700 1 --2- 172.21.15.189:0/1691409227 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca80b7e50 0x7f6ca80cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.333+0000 7f6cad6d3700 1 --2- 172.21.15.189:0/1691409227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c94003a00 0x7f6ca81538f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.333+0000 7f6cad6d3700 1 --2- 172.21.15.189:0/1691409227 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c8806ac90 0x7f6c8806d150 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.333+0000 7f6cad6d3700 1 --2- 172.21.15.189:0/1691409227 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca815dfa0 0x7f6ca80b7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.333+0000 7f6cad6d3700 1 -- 172.21.15.189:0/1691409227 >> 172.21.15.189:0/1691409227 conn(0x7f6ca806bbc0 msgr2=0x7f6ca80d26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:51.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.333+0000 7f6cad6d3700 1 -- 172.21.15.189:0/1691409227 shutdown_connections 2024-04-06T12:12:51.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.333+0000 7f6cad6d3700 1 -- 172.21.15.189:0/1691409227 wait complete. 2024-04-06T12:12:51.344 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:12:51.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.470+0000 7f83b44b9700 1 Processor -- start 2024-04-06T12:12:51.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.470+0000 7f83b44b9700 1 -- start start 2024-04-06T12:12:51.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.471+0000 7f83b44b9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f83ac0b39d0 0x7f83ac0af040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.471+0000 7f83b44b9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f83ac0723f0 0x7f83ac072880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.471+0000 7f83b44b9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83ac071950 con 0x7f83ac15d7b0 2024-04-06T12:12:51.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.471+0000 7f83b44b9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83ac072dc0 con 0x7f83ac0b39d0 2024-04-06T12:12:51.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.471+0000 7f83b44b9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83ac0b4170 con 0x7f83ac0723f0 2024-04-06T12:12:51.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.471+0000 7f83b1a54700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f83ac0b39d0 0x7f83ac0af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.471+0000 7f83b2255700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f83ac15d7b0 0x7f83ac0ac8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39066/0 (socket says 172.21.15.189:39066) 2024-04-06T12:12:51.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.471+0000 7f83b2255700 1 -- 172.21.15.189:0/4117513645 learned_addr learned my addr 172.21.15.189:0/4117513645 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:51.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.472+0000 7f83b1253700 1 -- 172.21.15.189:0/4117513645 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583939638 0 0) 0x7f83ac071950 con 0x7f83ac15d7b0 2024-04-06T12:12:51.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.472+0000 7f83b1253700 1 -- 172.21.15.189:0/4117513645 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83980035b0 con 0x7f83ac15d7b0 2024-04-06T12:12:51.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.472+0000 7f83b2a56700 1 --2- 172.21.15.189:0/4117513645 >> v2:172.21.15.192:3300/0 conn(0x7f83ac0723f0 0x7f83ac072880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.472+0000 7f83b1a54700 1 -- 172.21.15.189:0/4117513645 >> v2:172.21.15.192:3300/0 conn(0x7f83ac0723f0 msgr2=0x7f83ac072880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:51.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.472+0000 7f83b1a54700 1 --2- 172.21.15.189:0/4117513645 >> v2:172.21.15.192:3300/0 conn(0x7f83ac0723f0 0x7f83ac072880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.472+0000 7f83b1a54700 1 -- 172.21.15.189:0/4117513645 >> v1:172.21.15.189:6789/0 conn(0x7f83ac15d7b0 legacy=0x7f83ac0ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:51.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.472+0000 7f83b1a54700 1 -- 172.21.15.189:0/4117513645 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac0b3e90 con 0x7f83ac0b39d0 2024-04-06T12:12:51.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.472+0000 7f83b1a54700 1 --2- 172.21.15.189:0/4117513645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83ac0b39d0 0x7f83ac0af040 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f83ac06c7d0 tx=0x7f83a8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af642b46751c56fa server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:51.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.472+0000 7f83b1253700 1 -- 172.21.15.189:0/4117513645 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8011830 con 0x7f83ac0b39d0 2024-04-06T12:12:51.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.472+0000 7f83b1253700 1 -- 172.21.15.189:0/4117513645 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a8011990 con 0x7f83ac0b39d0 2024-04-06T12:12:51.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.472+0000 7f83b1253700 1 -- 172.21.15.189:0/4117513645 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8011d20 con 0x7f83ac0b39d0 2024-04-06T12:12:51.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.473+0000 7f83b44b9700 1 -- 172.21.15.189:0/4117513645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83ac0b39d0 msgr2=0x7f83ac0af040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:51.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.473+0000 7f83b44b9700 1 --2- 172.21.15.189:0/4117513645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83ac0b39d0 0x7f83ac0af040 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f83ac06c7d0 tx=0x7f83a8009b20 comp rx=0 tx=0).stop 2024-04-06T12:12:51.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.473+0000 7f83b44b9700 1 -- 172.21.15.189:0/4117513645 shutdown_connections 2024-04-06T12:12:51.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.473+0000 7f83b44b9700 1 --2- 172.21.15.189:0/4117513645 >> v2:172.21.15.192:3300/0 conn(0x7f83ac0723f0 0x7f83ac072880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.473+0000 7f83b44b9700 1 --2- 172.21.15.189:0/4117513645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83ac0b39d0 0x7f83ac0af040 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.473+0000 7f83b44b9700 1 -- 172.21.15.189:0/4117513645 >> 172.21.15.189:0/4117513645 conn(0x7f83ac0d00b0 msgr2=0x7f83ac0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:51.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.473+0000 7f83b44b9700 1 -- 172.21.15.189:0/4117513645 shutdown_connections 2024-04-06T12:12:51.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.473+0000 7f83b44b9700 1 -- 172.21.15.189:0/4117513645 wait complete. 2024-04-06T12:12:51.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.474+0000 7f83b44b9700 1 Processor -- start 2024-04-06T12:12:51.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.474+0000 7f83b44b9700 1 -- start start 2024-04-06T12:12:51.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83b44b9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83ac0723f0 0x7f83ac0cef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83b44b9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83ac0b39d0 0x7f83ac153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83b44b9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83ac1542c0 0x7f83ac15dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83b44b9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac13f4d0 con 0x7f83ac0723f0 2024-04-06T12:12:51.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83b44b9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac154770 con 0x7f83ac1542c0 2024-04-06T12:12:51.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83b44b9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac13e600 con 0x7f83ac0b39d0 2024-04-06T12:12:51.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83b2255700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83ac0723f0 0x7f83ac0cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83b2255700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83ac0723f0 0x7f83ac0cef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57464/0 (socket says 172.21.15.189:57464) 2024-04-06T12:12:51.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83b2255700 1 -- 172.21.15.189:0/559106843 learned_addr learned my addr 172.21.15.189:0/559106843 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:51.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83b1a54700 1 --2- 172.21.15.189:0/559106843 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83ac0b39d0 0x7f83ac153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83b2a56700 1 --2- 172.21.15.189:0/559106843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83ac1542c0 0x7f83ac15dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83b2255700 1 -- 172.21.15.189:0/559106843 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83ac0b39d0 msgr2=0x7f83ac153d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:51.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83b2255700 1 --2- 172.21.15.189:0/559106843 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83ac0b39d0 0x7f83ac153d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83b2255700 1 -- 172.21.15.189:0/559106843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83ac1542c0 msgr2=0x7f83ac15dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:51.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83b2255700 1 --2- 172.21.15.189:0/559106843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83ac1542c0 0x7f83ac15dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83b2255700 1 -- 172.21.15.189:0/559106843 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a80094a0 con 0x7f83ac0723f0 2024-04-06T12:12:51.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83b2255700 1 --2- 172.21.15.189:0/559106843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83ac0723f0 0x7f83ac0cef10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f839c00c730 tx=0x7f839c00caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:51.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83a2ffd700 1 -- 172.21.15.189:0/559106843 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c005d70 con 0x7f83ac0723f0 2024-04-06T12:12:51.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83a2ffd700 1 -- 172.21.15.189:0/559106843 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839c0070d0 con 0x7f83ac0723f0 2024-04-06T12:12:51.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83b44b9700 1 -- 172.21.15.189:0/559106843 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83ac13e850 con 0x7f83ac0723f0 2024-04-06T12:12:51.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83a2ffd700 1 -- 172.21.15.189:0/559106843 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c007440 con 0x7f83ac0723f0 2024-04-06T12:12:51.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.475+0000 7f83b44b9700 1 -- 172.21.15.189:0/559106843 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83ac13ee10 con 0x7f83ac0723f0 2024-04-06T12:12:51.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.476+0000 7f83b44b9700 1 -- 172.21.15.189:0/559106843 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83ac067bb0 con 0x7f83ac0723f0 2024-04-06T12:12:51.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.477+0000 7f83a2ffd700 1 -- 172.21.15.189:0/559106843 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f839c0075a0 con 0x7f83ac0723f0 2024-04-06T12:12:51.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.477+0000 7f83a2ffd700 1 --2- 172.21.15.189:0/559106843 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f839006ac90 0x7f839006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.477+0000 7f83a2ffd700 1 -- 172.21.15.189:0/559106843 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f839c088a90 con 0x7f83ac0723f0 2024-04-06T12:12:51.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.477+0000 7f83b1a54700 1 --2- 172.21.15.189:0/559106843 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f839006ac90 0x7f839006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.478+0000 7f83b1a54700 1 --2- 172.21.15.189:0/559106843 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f839006ac90 0x7f839006d150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f83a80090b0 tx=0x7f83a8008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:51.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.482+0000 7f83a2ffd700 1 -- 172.21.15.189:0/559106843 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f839c053a30 con 0x7f83ac0723f0 2024-04-06T12:12:51.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.673+0000 7f83b44b9700 1 -- 172.21.15.189:0/559106843 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f83ac0af9f0 con 0x7f83ac0723f0 2024-04-06T12:12:51.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.673+0000 7f83a2ffd700 1 -- 172.21.15.189:0/559106843 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f839c017020 con 0x7f83ac0723f0 2024-04-06T12:12:51.672 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:12:51.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.676+0000 7f83b44b9700 1 -- 172.21.15.189:0/559106843 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f839006ac90 msgr2=0x7f839006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:51.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.676+0000 7f83b44b9700 1 --2- 172.21.15.189:0/559106843 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f839006ac90 0x7f839006d150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f83a80090b0 tx=0x7f83a8008030 comp rx=0 tx=0).stop 2024-04-06T12:12:51.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.676+0000 7f83b44b9700 1 -- 172.21.15.189:0/559106843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83ac0723f0 msgr2=0x7f83ac0cef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:51.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.676+0000 7f83b44b9700 1 --2- 172.21.15.189:0/559106843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83ac0723f0 0x7f83ac0cef10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f839c00c730 tx=0x7f839c00caf0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.676+0000 7f83b44b9700 1 -- 172.21.15.189:0/559106843 shutdown_connections 2024-04-06T12:12:51.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.676+0000 7f83b44b9700 1 --2- 172.21.15.189:0/559106843 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f839006ac90 0x7f839006d150 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.676+0000 7f83b44b9700 1 --2- 172.21.15.189:0/559106843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83ac1542c0 0x7f83ac15dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.676+0000 7f83b44b9700 1 --2- 172.21.15.189:0/559106843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83ac0723f0 0x7f83ac0cef10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.676+0000 7f83b44b9700 1 --2- 172.21.15.189:0/559106843 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83ac0b39d0 0x7f83ac153d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.676+0000 7f83b44b9700 1 -- 172.21.15.189:0/559106843 >> 172.21.15.189:0/559106843 conn(0x7f83ac0d00b0 msgr2=0x7f83ac0ad2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:51.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.676+0000 7f83b44b9700 1 -- 172.21.15.189:0/559106843 shutdown_connections 2024-04-06T12:12:51.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.676+0000 7f83b44b9700 1 -- 172.21.15.189:0/559106843 wait complete. 2024-04-06T12:12:51.686 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:12:51.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.814+0000 7f1269cf0700 1 Processor -- start 2024-04-06T12:12:51.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.814+0000 7f1269cf0700 1 -- start start 2024-04-06T12:12:51.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.815+0000 7f1269cf0700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f12640b4af0 0x7f12640b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.815+0000 7f1269cf0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12640cdbe0 con 0x7f12640b4660 2024-04-06T12:12:51.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.815+0000 7f1269cf0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126415a940 con 0x7f126415a550 2024-04-06T12:12:51.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.815+0000 7f1269cf0700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f126415ac00 con 0x7f12640b4af0 2024-04-06T12:12:51.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.815+0000 7f125bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f12640b4660 0x7f12640b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39068/0 (socket says 172.21.15.189:39068) 2024-04-06T12:12:51.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.815+0000 7f125bfff700 1 -- 172.21.15.189:0/2035281469 learned_addr learned my addr 172.21.15.189:0/2035281469 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:51.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.815+0000 7f1263fff700 1 --2- 172.21.15.189:0/2035281469 >> v2:172.21.15.192:3300/0 conn(0x7f12640b4af0 0x7f12640b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.815+0000 7f1262ffd700 1 -- 172.21.15.189:0/2035281469 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131882504 0 0) 0x7f12640cdbe0 con 0x7f12640b4660 2024-04-06T12:12:51.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.815+0000 7f1262ffd700 1 -- 172.21.15.189:0/2035281469 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12500035b0 con 0x7f12640b4660 2024-04-06T12:12:51.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.815+0000 7f1262ffd700 1 -- 172.21.15.189:0/2035281469 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071501782 0 0) 0x7f126415a940 con 0x7f126415a550 2024-04-06T12:12:51.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.815+0000 7f1262ffd700 1 -- 172.21.15.189:0/2035281469 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12640cdbe0 con 0x7f126415a550 2024-04-06T12:12:51.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.815+0000 7f1262ffd700 1 -- 172.21.15.189:0/2035281469 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3999535322 0 0) 0x7f12500035b0 con 0x7f12640b4660 2024-04-06T12:12:51.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.815+0000 7f1263fff700 1 -- 172.21.15.189:0/2035281469 >> v1:172.21.15.192:6789/0 conn(0x7f126415a550 legacy=0x7f12640cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:51.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.815+0000 7f1263fff700 1 -- 172.21.15.189:0/2035281469 >> v1:172.21.15.189:6789/0 conn(0x7f12640b4660 legacy=0x7f12640b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:51.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.816+0000 7f1263fff700 1 -- 172.21.15.189:0/2035281469 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126415aee0 con 0x7f12640b4af0 2024-04-06T12:12:51.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.816+0000 7f1263fff700 1 --2- 172.21.15.189:0/2035281469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12640b4af0 0x7f12640b36d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f125c009d90 tx=0x7f125c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3124a26573b1978c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:51.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.816+0000 7f1262ffd700 1 -- 172.21.15.189:0/2035281469 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c00e070 con 0x7f12640b4af0 2024-04-06T12:12:51.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.816+0000 7f1262ffd700 1 -- 172.21.15.189:0/2035281469 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125c003a20 con 0x7f12640b4af0 2024-04-06T12:12:51.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.816+0000 7f1262ffd700 1 -- 172.21.15.189:0/2035281469 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124c001670 con 0x7f12640b4af0 2024-04-06T12:12:51.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.816+0000 7f1269cf0700 1 -- 172.21.15.189:0/2035281469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12640b4af0 msgr2=0x7f12640b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:51.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.816+0000 7f1269cf0700 1 --2- 172.21.15.189:0/2035281469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12640b4af0 0x7f12640b36d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f125c009d90 tx=0x7f125c00b330 comp rx=0 tx=0).stop 2024-04-06T12:12:51.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.816+0000 7f1269cf0700 1 -- 172.21.15.189:0/2035281469 shutdown_connections 2024-04-06T12:12:51.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.817+0000 7f1269cf0700 1 --2- 172.21.15.189:0/2035281469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12640b4af0 0x7f12640b36d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.817+0000 7f1269cf0700 1 -- 172.21.15.189:0/2035281469 >> 172.21.15.189:0/2035281469 conn(0x7f12640d00b0 msgr2=0x7f12640d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:51.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.817+0000 7f1269cf0700 1 -- 172.21.15.189:0/2035281469 shutdown_connections 2024-04-06T12:12:51.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.817+0000 7f1269cf0700 1 -- 172.21.15.189:0/2035281469 wait complete. 2024-04-06T12:12:51.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.818+0000 7f1269cf0700 1 Processor -- start 2024-04-06T12:12:51.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.818+0000 7f1269cf0700 1 -- start start 2024-04-06T12:12:51.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.818+0000 7f1269cf0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12640b4660 0x7f12640a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.818+0000 7f1269cf0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12640b4af0 0x7f12640cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.818+0000 7f1269cf0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1264143870 0x7f1264143d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.818+0000 7f1269cf0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126415d3c0 con 0x7f12640b4660 2024-04-06T12:12:51.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.818+0000 7f1269cf0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1264144260 con 0x7f1264143870 2024-04-06T12:12:51.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.818+0000 7f1269cf0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12640a89f0 con 0x7f12640b4af0 2024-04-06T12:12:51.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.819+0000 7f12637fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12640b4660 0x7f12640a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.819+0000 7f12637fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12640b4660 0x7f12640a84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57476/0 (socket says 172.21.15.189:57476) 2024-04-06T12:12:51.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.819+0000 7f12637fe700 1 -- 172.21.15.189:0/1353161425 learned_addr learned my addr 172.21.15.189:0/1353161425 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:51.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.819+0000 7f125bfff700 1 --2- 172.21.15.189:0/1353161425 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12640b4af0 0x7f12640cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.819+0000 7f1263fff700 1 --2- 172.21.15.189:0/1353161425 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1264143870 0x7f1264143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.819+0000 7f12637fe700 1 -- 172.21.15.189:0/1353161425 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12640b4af0 msgr2=0x7f12640cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:51.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.819+0000 7f12637fe700 1 --2- 172.21.15.189:0/1353161425 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12640b4af0 0x7f12640cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.819+0000 7f12637fe700 1 -- 172.21.15.189:0/1353161425 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1264143870 msgr2=0x7f1264143d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:51.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.819+0000 7f12637fe700 1 --2- 172.21.15.189:0/1353161425 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1264143870 0x7f1264143d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:51.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.819+0000 7f12637fe700 1 -- 172.21.15.189:0/1353161425 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f125c009a70 con 0x7f12640b4660 2024-04-06T12:12:51.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.819+0000 7f125bfff700 1 --2- 172.21.15.189:0/1353161425 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12640b4af0 0x7f12640cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:51.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.819+0000 7f12637fe700 1 --2- 172.21.15.189:0/1353161425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12640b4660 0x7f12640a84b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f125400c530 tx=0x7f125400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:51.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.819+0000 7f1260ff9700 1 -- 172.21.15.189:0/1353161425 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12540052d0 con 0x7f12640b4660 2024-04-06T12:12:51.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.819+0000 7f1260ff9700 1 -- 172.21.15.189:0/1353161425 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1254005450 con 0x7f12640b4660 2024-04-06T12:12:51.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.819+0000 7f1269cf0700 1 -- 172.21.15.189:0/1353161425 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126415c740 con 0x7f12640b4660 2024-04-06T12:12:51.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.819+0000 7f1260ff9700 1 -- 172.21.15.189:0/1353161425 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12540038c0 con 0x7f12640b4660 2024-04-06T12:12:51.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.819+0000 7f1269cf0700 1 -- 172.21.15.189:0/1353161425 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f126415cd00 con 0x7f12640b4660 2024-04-06T12:12:51.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.820+0000 7f1269cf0700 1 -- 172.21.15.189:0/1353161425 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1264067bb0 con 0x7f12640b4660 2024-04-06T12:12:51.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.821+0000 7f1260ff9700 1 -- 172.21.15.189:0/1353161425 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1254003a20 con 0x7f12640b4660 2024-04-06T12:12:51.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.821+0000 7f1260ff9700 1 --2- 172.21.15.189:0/1353161425 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f124406ac90 0x7f124406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:51.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.821+0000 7f1260ff9700 1 -- 172.21.15.189:0/1353161425 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1254088a60 con 0x7f12640b4660 2024-04-06T12:12:51.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.821+0000 7f125bfff700 1 --2- 172.21.15.189:0/1353161425 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f124406ac90 0x7f124406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:51.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.821+0000 7f125bfff700 1 --2- 172.21.15.189:0/1353161425 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f124406ac90 0x7f124406d150 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f124c001eb0 tx=0x7f124c001ca0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:51.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:51.825+0000 7f1260ff9700 1 -- 172.21.15.189:0/1353161425 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1254053950 con 0x7f12640b4660 2024-04-06T12:12:52.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.018+0000 7f1269cf0700 1 -- 172.21.15.189:0/1353161425 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1264074990 con 0x7f12640b4660 2024-04-06T12:12:52.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.018+0000 7f1260ff9700 1 -- 172.21.15.189:0/1353161425 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f12540170a0 con 0x7f12640b4660 2024-04-06T12:12:52.016 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:12:52.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.020+0000 7f1269cf0700 1 -- 172.21.15.189:0/1353161425 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f124406ac90 msgr2=0x7f124406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:52.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.020+0000 7f1269cf0700 1 --2- 172.21.15.189:0/1353161425 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f124406ac90 0x7f124406d150 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f124c001eb0 tx=0x7f124c001ca0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.020+0000 7f1269cf0700 1 -- 172.21.15.189:0/1353161425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12640b4660 msgr2=0x7f12640a84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:52.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.020+0000 7f1269cf0700 1 --2- 172.21.15.189:0/1353161425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12640b4660 0x7f12640a84b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f125400c530 tx=0x7f125400c8f0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.020+0000 7f1269cf0700 1 -- 172.21.15.189:0/1353161425 shutdown_connections 2024-04-06T12:12:52.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.020+0000 7f1269cf0700 1 --2- 172.21.15.189:0/1353161425 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f124406ac90 0x7f124406d150 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.020+0000 7f1269cf0700 1 --2- 172.21.15.189:0/1353161425 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1264143870 0x7f1264143d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.020+0000 7f1269cf0700 1 --2- 172.21.15.189:0/1353161425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12640b4660 0x7f12640a84b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.020+0000 7f1269cf0700 1 --2- 172.21.15.189:0/1353161425 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12640b4af0 0x7f12640cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.020+0000 7f1269cf0700 1 -- 172.21.15.189:0/1353161425 >> 172.21.15.189:0/1353161425 conn(0x7f12640d00b0 msgr2=0x7f1264154d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:52.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.020+0000 7f1269cf0700 1 -- 172.21.15.189:0/1353161425 shutdown_connections 2024-04-06T12:12:52.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.020+0000 7f1269cf0700 1 -- 172.21.15.189:0/1353161425 wait complete. 2024-04-06T12:12:52.030 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-06T12:12:52.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.156+0000 7f9841189700 1 Processor -- start 2024-04-06T12:12:52.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.156+0000 7f9841189700 1 -- start start 2024-04-06T12:12:52.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.157+0000 7f9841189700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f983c06ae40 0x7f983c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.157+0000 7f9841189700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f983c15ddf0 0x7f983c15e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.157+0000 7f9841189700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f983c0b3990 con 0x7f983c15d7b0 2024-04-06T12:12:52.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.157+0000 7f9841189700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f983c0b3060 con 0x7f983c15ddf0 2024-04-06T12:12:52.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.157+0000 7f9841189700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f983c0b3340 con 0x7f983c06ae40 2024-04-06T12:12:52.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.157+0000 7f983b59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f983c15ddf0 0x7f983c15e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:52.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.157+0000 7f983b59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f983c15ddf0 0x7f983c15e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57484/0 (socket says 172.21.15.189:57484) 2024-04-06T12:12:52.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.157+0000 7f983b59e700 1 -- 172.21.15.189:0/2268504030 learned_addr learned my addr 172.21.15.189:0/2268504030 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:52.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.157+0000 7f983a59c700 1 --2- 172.21.15.189:0/2268504030 >> v2:172.21.15.192:3300/0 conn(0x7f983c06ae40 0x7f983c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:52.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.158+0000 7f983b59e700 1 -- 172.21.15.189:0/2268504030 >> v2:172.21.15.192:3300/0 conn(0x7f983c06ae40 msgr2=0x7f983c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:52.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.158+0000 7f983b59e700 1 --2- 172.21.15.189:0/2268504030 >> v2:172.21.15.192:3300/0 conn(0x7f983c06ae40 0x7f983c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.158+0000 7f983b59e700 1 -- 172.21.15.189:0/2268504030 >> v1:172.21.15.192:6789/0 conn(0x7f983c15d7b0 legacy=0x7f983c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:52.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.158+0000 7f983b59e700 1 -- 172.21.15.189:0/2268504030 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f983c0b3620 con 0x7f983c15ddf0 2024-04-06T12:12:52.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.158+0000 7f983a59c700 1 --2- 172.21.15.189:0/2268504030 >> v2:172.21.15.192:3300/0 conn(0x7f983c06ae40 0x7f983c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:52.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.158+0000 7f983b59e700 1 --2- 172.21.15.189:0/2268504030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f983c15ddf0 0x7f983c15e240 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f983c0afc00 tx=0x7f983400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2536d231cf68c047 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:52.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.158+0000 7f9839d9b700 1 -- 172.21.15.189:0/2268504030 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983400e070 con 0x7f983c15ddf0 2024-04-06T12:12:52.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.158+0000 7f9839d9b700 1 -- 172.21.15.189:0/2268504030 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983400b9c0 con 0x7f983c15ddf0 2024-04-06T12:12:52.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.158+0000 7f9839d9b700 1 -- 172.21.15.189:0/2268504030 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983400bd70 con 0x7f983c15ddf0 2024-04-06T12:12:52.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.158+0000 7f9841189700 1 -- 172.21.15.189:0/2268504030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f983c15ddf0 msgr2=0x7f983c15e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:52.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.158+0000 7f9841189700 1 --2- 172.21.15.189:0/2268504030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f983c15ddf0 0x7f983c15e240 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f983c0afc00 tx=0x7f983400b330 comp rx=0 tx=0).stop 2024-04-06T12:12:52.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.158+0000 7f9841189700 1 -- 172.21.15.189:0/2268504030 shutdown_connections 2024-04-06T12:12:52.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.158+0000 7f9841189700 1 --2- 172.21.15.189:0/2268504030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f983c15ddf0 0x7f983c15e240 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.158+0000 7f9841189700 1 --2- 172.21.15.189:0/2268504030 >> v2:172.21.15.192:3300/0 conn(0x7f983c06ae40 0x7f983c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.158+0000 7f9841189700 1 -- 172.21.15.189:0/2268504030 >> 172.21.15.189:0/2268504030 conn(0x7f983c0d00b0 msgr2=0x7f983c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:52.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.158+0000 7f9841189700 1 -- 172.21.15.189:0/2268504030 shutdown_connections 2024-04-06T12:12:52.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.159+0000 7f9841189700 1 -- 172.21.15.189:0/2268504030 wait complete. 2024-04-06T12:12:52.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.160+0000 7f9841189700 1 Processor -- start 2024-04-06T12:12:52.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.160+0000 7f9841189700 1 -- start start 2024-04-06T12:12:52.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f9841189700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f983c06ae40 0x7f983c0cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f9841189700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f983c15d7b0 0x7f983c0cd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f9841189700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f983c15ddf0 0x7f983c0cd890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f9841189700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983c1f6ed0 con 0x7f983c15ddf0 2024-04-06T12:12:52.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f9841189700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983c15d050 con 0x7f983c15d7b0 2024-04-06T12:12:52.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f9841189700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983c1f6250 con 0x7f983c06ae40 2024-04-06T12:12:52.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f983b59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f983c15ddf0 0x7f983c0cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:52.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f983b59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f983c15ddf0 0x7f983c0cd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57496/0 (socket says 172.21.15.189:57496) 2024-04-06T12:12:52.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f983b59e700 1 -- 172.21.15.189:0/380583026 learned_addr learned my addr 172.21.15.189:0/380583026 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:52.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f983ad9d700 1 --2- 172.21.15.189:0/380583026 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f983c06ae40 0x7f983c0cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:52.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f983a59c700 1 --2- 172.21.15.189:0/380583026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f983c15d7b0 0x7f983c0cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:52.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f983b59e700 1 -- 172.21.15.189:0/380583026 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f983c06ae40 msgr2=0x7f983c0cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:52.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f983b59e700 1 --2- 172.21.15.189:0/380583026 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f983c06ae40 0x7f983c0cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f983b59e700 1 -- 172.21.15.189:0/380583026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f983c15d7b0 msgr2=0x7f983c0cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:52.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f983b59e700 1 --2- 172.21.15.189:0/380583026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f983c15d7b0 0x7f983c0cd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f983b59e700 1 -- 172.21.15.189:0/380583026 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9834009b30 con 0x7f983c15ddf0 2024-04-06T12:12:52.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f983b59e700 1 --2- 172.21.15.189:0/380583026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f983c15ddf0 0x7f983c0cd890 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f983400b730 tx=0x7f9834004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:52.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f98237fe700 1 -- 172.21.15.189:0/380583026 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9834004870 con 0x7f983c15ddf0 2024-04-06T12:12:52.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f98237fe700 1 -- 172.21.15.189:0/380583026 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98340099b0 con 0x7f983c15ddf0 2024-04-06T12:12:52.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f9841189700 1 -- 172.21.15.189:0/380583026 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f983c1f6530 con 0x7f983c15ddf0 2024-04-06T12:12:52.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.161+0000 7f98237fe700 1 -- 172.21.15.189:0/380583026 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983401b670 con 0x7f983c15ddf0 2024-04-06T12:12:52.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.162+0000 7f9841189700 1 -- 172.21.15.189:0/380583026 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f983c1f6af0 con 0x7f983c15ddf0 2024-04-06T12:12:52.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.162+0000 7f9841189700 1 -- 172.21.15.189:0/380583026 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f983c1f7010 con 0x7f983c15ddf0 2024-04-06T12:12:52.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.163+0000 7f98237fe700 1 -- 172.21.15.189:0/380583026 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f983401b7d0 con 0x7f983c15ddf0 2024-04-06T12:12:52.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.163+0000 7f98237fe700 1 --2- 172.21.15.189:0/380583026 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f982806ac90 0x7f982806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.163+0000 7f98237fe700 1 -- 172.21.15.189:0/380583026 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f983408aa30 con 0x7f983c15ddf0 2024-04-06T12:12:52.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.163+0000 7f983ad9d700 1 --2- 172.21.15.189:0/380583026 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f982806ac90 0x7f982806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:52.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.164+0000 7f983ad9d700 1 --2- 172.21.15.189:0/380583026 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f982806ac90 0x7f982806d150 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9830007fd0 tx=0x7f9830009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:52.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.168+0000 7f98237fe700 1 -- 172.21.15.189:0/380583026 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f983c1f7010 con 0x7f983c15ddf0 2024-04-06T12:12:52.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.369+0000 7f9841189700 1 -- 172.21.15.189:0/380583026 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f983c1f7010 con 0x7f983c15ddf0 2024-04-06T12:12:52.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.369+0000 7f98237fe700 1 -- 172.21.15.189:0/380583026 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f983401f020 con 0x7f983c15ddf0 2024-04-06T12:12:52.367 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:12:52.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.371+0000 7f9841189700 1 -- 172.21.15.189:0/380583026 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f982806ac90 msgr2=0x7f982806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:52.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.371+0000 7f9841189700 1 --2- 172.21.15.189:0/380583026 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f982806ac90 0x7f982806d150 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9830007fd0 tx=0x7f9830009040 comp rx=0 tx=0).stop 2024-04-06T12:12:52.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.371+0000 7f9841189700 1 -- 172.21.15.189:0/380583026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f983c15ddf0 msgr2=0x7f983c0cd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:52.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.372+0000 7f9841189700 1 --2- 172.21.15.189:0/380583026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f983c15ddf0 0x7f983c0cd890 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f983400b730 tx=0x7f9834004ad0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.372+0000 7f9841189700 1 -- 172.21.15.189:0/380583026 shutdown_connections 2024-04-06T12:12:52.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.372+0000 7f9841189700 1 --2- 172.21.15.189:0/380583026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f983c15ddf0 0x7f983c0cd890 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.372+0000 7f9841189700 1 --2- 172.21.15.189:0/380583026 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f983c06ae40 0x7f983c0cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.372+0000 7f9841189700 1 --2- 172.21.15.189:0/380583026 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f982806ac90 0x7f982806d150 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.372+0000 7f9841189700 1 --2- 172.21.15.189:0/380583026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f983c15d7b0 0x7f983c0cd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.372+0000 7f9841189700 1 -- 172.21.15.189:0/380583026 >> 172.21.15.189:0/380583026 conn(0x7f983c0d00b0 msgr2=0x7f983c072400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:52.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.372+0000 7f9841189700 1 -- 172.21.15.189:0/380583026 shutdown_connections 2024-04-06T12:12:52.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.372+0000 7f9841189700 1 -- 172.21.15.189:0/380583026 wait complete. 2024-04-06T12:12:52.382 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:52.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.505+0000 7f0c3e209700 1 Processor -- start 2024-04-06T12:12:52.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.505+0000 7f0c3e209700 1 -- start start 2024-04-06T12:12:52.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.506+0000 7f0c3e209700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0c38144f40 0x7f0c3806a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.506+0000 7f0c3e209700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0c3815c150 0x7f0c3806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.506+0000 7f0c3e209700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c3815c740 con 0x7f0c3815fcd0 2024-04-06T12:12:52.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.506+0000 7f0c3e209700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c380cfcb0 con 0x7f0c3815c150 2024-04-06T12:12:52.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.506+0000 7f0c3e209700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c3815ca20 con 0x7f0c38144f40 2024-04-06T12:12:52.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.506+0000 7f0c37fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0c3815c150 0x7f0c3806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:52.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.506+0000 7f0c37fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0c3815c150 0x7f0c3806ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57508/0 (socket says 172.21.15.189:57508) 2024-04-06T12:12:52.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.506+0000 7f0c37fff700 1 -- 172.21.15.189:0/1177801119 learned_addr learned my addr 172.21.15.189:0/1177801119 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:52.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.507+0000 7f0c37fff700 1 -- 172.21.15.189:0/1177801119 >> v2:172.21.15.192:3300/0 conn(0x7f0c38144f40 msgr2=0x7f0c3806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:52.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.507+0000 7f0c37fff700 1 --2- 172.21.15.189:0/1177801119 >> v2:172.21.15.192:3300/0 conn(0x7f0c38144f40 0x7f0c3806a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.507+0000 7f0c37fff700 1 -- 172.21.15.189:0/1177801119 >> v1:172.21.15.192:6789/0 conn(0x7f0c3815fcd0 legacy=0x7f0c3815ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:52.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.507+0000 7f0c37fff700 1 -- 172.21.15.189:0/1177801119 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3815f1f0 con 0x7f0c3815c150 2024-04-06T12:12:52.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.507+0000 7f0c37fff700 1 --2- 172.21.15.189:0/1177801119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3815c150 0x7f0c3806ac70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0c38073470 tx=0x7f0c3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dbf437678d4a4f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:52.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.507+0000 7f0c367fc700 1 -- 172.21.15.189:0/1177801119 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3000e070 con 0x7f0c3815c150 2024-04-06T12:12:52.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.507+0000 7f0c367fc700 1 -- 172.21.15.189:0/1177801119 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c30003a20 con 0x7f0c3815c150 2024-04-06T12:12:52.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.507+0000 7f0c367fc700 1 -- 172.21.15.189:0/1177801119 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c30003dd0 con 0x7f0c3815c150 2024-04-06T12:12:52.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.507+0000 7f0c3e209700 1 -- 172.21.15.189:0/1177801119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3815c150 msgr2=0x7f0c3806ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:52.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.507+0000 7f0c3e209700 1 --2- 172.21.15.189:0/1177801119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3815c150 0x7f0c3806ac70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0c38073470 tx=0x7f0c3000b330 comp rx=0 tx=0).stop 2024-04-06T12:12:52.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.507+0000 7f0c3e209700 1 -- 172.21.15.189:0/1177801119 shutdown_connections 2024-04-06T12:12:52.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.507+0000 7f0c3e209700 1 --2- 172.21.15.189:0/1177801119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3815c150 0x7f0c3806ac70 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.507+0000 7f0c3e209700 1 --2- 172.21.15.189:0/1177801119 >> v2:172.21.15.192:3300/0 conn(0x7f0c38144f40 0x7f0c3806a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.507+0000 7f0c3e209700 1 -- 172.21.15.189:0/1177801119 >> 172.21.15.189:0/1177801119 conn(0x7f0c380d00b0 msgr2=0x7f0c380d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:52.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.507+0000 7f0c3e209700 1 -- 172.21.15.189:0/1177801119 shutdown_connections 2024-04-06T12:12:52.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.507+0000 7f0c3e209700 1 -- 172.21.15.189:0/1177801119 wait complete. 2024-04-06T12:12:52.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.509+0000 7f0c3e209700 1 Processor -- start 2024-04-06T12:12:52.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.509+0000 7f0c3e209700 1 -- start start 2024-04-06T12:12:52.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.510+0000 7f0c3e209700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c38144f40 0x7f0c380cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.510+0000 7f0c3e209700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c3815c150 0x7f0c38141470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.510+0000 7f0c3e209700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c3815fcd0 0x7f0c381419b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.510+0000 7f0c3e209700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c380b3a90 con 0x7f0c38144f40 2024-04-06T12:12:52.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.510+0000 7f0c3e209700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c381408f0 con 0x7f0c3815fcd0 2024-04-06T12:12:52.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.510+0000 7f0c3e209700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c380b3160 con 0x7f0c3815c150 2024-04-06T12:12:52.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.510+0000 7f0c377fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c38144f40 0x7f0c380cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:52.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.510+0000 7f0c377fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c38144f40 0x7f0c380cb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57510/0 (socket says 172.21.15.189:57510) 2024-04-06T12:12:52.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.510+0000 7f0c377fe700 1 -- 172.21.15.189:0/1057539128 learned_addr learned my addr 172.21.15.189:0/1057539128 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:52.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.510+0000 7f0c37fff700 1 --2- 172.21.15.189:0/1057539128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c3815fcd0 0x7f0c381419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:52.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.510+0000 7f0c36ffd700 1 --2- 172.21.15.189:0/1057539128 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c3815c150 0x7f0c38141470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:52.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.510+0000 7f0c377fe700 1 -- 172.21.15.189:0/1057539128 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c3815c150 msgr2=0x7f0c38141470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:52.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.510+0000 7f0c377fe700 1 --2- 172.21.15.189:0/1057539128 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c3815c150 0x7f0c38141470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.510+0000 7f0c377fe700 1 -- 172.21.15.189:0/1057539128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c3815fcd0 msgr2=0x7f0c381419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:52.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.510+0000 7f0c377fe700 1 --2- 172.21.15.189:0/1057539128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c3815fcd0 0x7f0c381419b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.510+0000 7f0c377fe700 1 -- 172.21.15.189:0/1057539128 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c30009b30 con 0x7f0c38144f40 2024-04-06T12:12:52.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.510+0000 7f0c37fff700 1 --2- 172.21.15.189:0/1057539128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c3815fcd0 0x7f0c381419b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:52.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.510+0000 7f0c36ffd700 1 --2- 172.21.15.189:0/1057539128 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c3815c150 0x7f0c38141470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:52.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.511+0000 7f0c377fe700 1 --2- 172.21.15.189:0/1057539128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c38144f40 0x7f0c380cb400 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0c2800b530 tx=0x7f0c2800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:52.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.511+0000 7f0c2ffff700 1 -- 172.21.15.189:0/1057539128 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c28011830 con 0x7f0c38144f40 2024-04-06T12:12:52.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.511+0000 7f0c2ffff700 1 -- 172.21.15.189:0/1057539128 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c280119b0 con 0x7f0c38144f40 2024-04-06T12:12:52.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.511+0000 7f0c2ffff700 1 -- 172.21.15.189:0/1057539128 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c28011d60 con 0x7f0c38144f40 2024-04-06T12:12:52.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.511+0000 7f0c3e209700 1 -- 172.21.15.189:0/1057539128 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c380b3440 con 0x7f0c38144f40 2024-04-06T12:12:52.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.511+0000 7f0c3e209700 1 -- 172.21.15.189:0/1057539128 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c380cbae0 con 0x7f0c38144f40 2024-04-06T12:12:52.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.512+0000 7f0c3e209700 1 -- 172.21.15.189:0/1057539128 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c38067bb0 con 0x7f0c38144f40 2024-04-06T12:12:52.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.512+0000 7f0c2ffff700 1 -- 172.21.15.189:0/1057539128 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0c2801a400 con 0x7f0c38144f40 2024-04-06T12:12:52.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.513+0000 7f0c2ffff700 1 --2- 172.21.15.189:0/1057539128 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c1806ac90 0x7f0c1806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.513+0000 7f0c2ffff700 1 -- 172.21.15.189:0/1057539128 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0c28089730 con 0x7f0c38144f40 2024-04-06T12:12:52.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.513+0000 7f0c36ffd700 1 --2- 172.21.15.189:0/1057539128 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c1806ac90 0x7f0c1806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:52.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.513+0000 7f0c36ffd700 1 --2- 172.21.15.189:0/1057539128 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c1806ac90 0x7f0c1806d150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0c381f3c90 tx=0x7f0c20008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:52.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.517+0000 7f0c2ffff700 1 -- 172.21.15.189:0/1057539128 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0c28054620 con 0x7f0c38144f40 2024-04-06T12:12:52.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.723+0000 7f0c3e209700 1 -- 172.21.15.189:0/1057539128 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c38074990 con 0x7f0c38144f40 2024-04-06T12:12:52.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.724+0000 7f0c2ffff700 1 -- 172.21.15.189:0/1057539128 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f0c28057c40 con 0x7f0c38144f40 2024-04-06T12:12:52.722 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:52.722 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:08.649170+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:52.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.726+0000 7f0c3e209700 1 -- 172.21.15.189:0/1057539128 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c1806ac90 msgr2=0x7f0c1806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:52.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.726+0000 7f0c3e209700 1 --2- 172.21.15.189:0/1057539128 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c1806ac90 0x7f0c1806d150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0c381f3c90 tx=0x7f0c20008040 comp rx=0 tx=0).stop 2024-04-06T12:12:52.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.726+0000 7f0c3e209700 1 -- 172.21.15.189:0/1057539128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c38144f40 msgr2=0x7f0c380cb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:52.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.726+0000 7f0c3e209700 1 --2- 172.21.15.189:0/1057539128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c38144f40 0x7f0c380cb400 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0c2800b530 tx=0x7f0c2800b8f0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.726+0000 7f0c3e209700 1 -- 172.21.15.189:0/1057539128 shutdown_connections 2024-04-06T12:12:52.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.726+0000 7f0c3e209700 1 --2- 172.21.15.189:0/1057539128 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c1806ac90 0x7f0c1806d150 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.726+0000 7f0c3e209700 1 --2- 172.21.15.189:0/1057539128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c3815fcd0 0x7f0c381419b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.726+0000 7f0c3e209700 1 --2- 172.21.15.189:0/1057539128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c38144f40 0x7f0c380cb400 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.726+0000 7f0c3e209700 1 --2- 172.21.15.189:0/1057539128 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c3815c150 0x7f0c38141470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.726+0000 7f0c3e209700 1 -- 172.21.15.189:0/1057539128 >> 172.21.15.189:0/1057539128 conn(0x7f0c380d00b0 msgr2=0x7f0c380d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:52.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.726+0000 7f0c3e209700 1 -- 172.21.15.189:0/1057539128 shutdown_connections 2024-04-06T12:12:52.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.726+0000 7f0c3e209700 1 -- 172.21.15.189:0/1057539128 wait complete. 2024-04-06T12:12:52.738 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:12:02.280866+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:12:03.156593+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-06T12:12:06.237184+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-06T12:12:52.739 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:12:52.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.866+0000 7f3e4fea5700 1 Processor -- start 2024-04-06T12:12:52.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.866+0000 7f3e4fea5700 1 -- start start 2024-04-06T12:12:52.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.867+0000 7f3e4fea5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e48146e70 0x7f3e4815db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.867+0000 7f3e4fea5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e481613c0 con 0x7f3e48072610 2024-04-06T12:12:52.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.867+0000 7f3e4fea5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4815e380 con 0x7f3e4806ec70 2024-04-06T12:12:52.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.867+0000 7f3e4fea5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e48147250 con 0x7f3e48146e70 2024-04-06T12:12:52.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.867+0000 7f3e4d440700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3e48072610 0x7f3e48161850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39074/0 (socket says 172.21.15.189:39074) 2024-04-06T12:12:52.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.867+0000 7f3e4d440700 1 -- 172.21.15.189:0/1166027552 learned_addr learned my addr 172.21.15.189:0/1166027552 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:52.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.867+0000 7f3e4dc41700 1 --2- 172.21.15.189:0/1166027552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e48146e70 0x7f3e4815db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:52.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.867+0000 7f3e4cc3f700 1 -- 172.21.15.189:0/1166027552 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079322394 0 0) 0x7f3e481613c0 con 0x7f3e48072610 2024-04-06T12:12:52.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.867+0000 7f3e4cc3f700 1 -- 172.21.15.189:0/1166027552 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e340035b0 con 0x7f3e48072610 2024-04-06T12:12:52.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.867+0000 7f3e4cc3f700 1 -- 172.21.15.189:0/1166027552 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1861363094 0 0) 0x7f3e340035b0 con 0x7f3e48072610 2024-04-06T12:12:52.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.867+0000 7f3e4cc3f700 1 -- 172.21.15.189:0/1166027552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e48146e70 msgr2=0x7f3e4815db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:52.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.867+0000 7f3e4cc3f700 1 --2- 172.21.15.189:0/1166027552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e48146e70 0x7f3e4815db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.867+0000 7f3e4cc3f700 1 -- 172.21.15.189:0/1166027552 >> v1:172.21.15.192:6789/0 conn(0x7f3e4806ec70 legacy=0x7f3e4806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:52.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.867+0000 7f3e4cc3f700 1 -- 172.21.15.189:0/1166027552 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4815e660 con 0x7f3e48072610 2024-04-06T12:12:52.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.867+0000 7f3e4cc3f700 1 -- 172.21.15.189:0/1166027552 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f3e38001670 con 0x7f3e48072610 2024-04-06T12:12:52.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.867+0000 7f3e4cc3f700 1 -- 172.21.15.189:0/1166027552 >> v1:172.21.15.189:6789/0 conn(0x7f3e48072610 legacy=0x7f3e48161850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:52.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.868+0000 7f3e4cc3f700 1 --2- 172.21.15.189:0/1166027552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e34003a30 0x7f3e34003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.868+0000 7f3e4cc3f700 1 --2- 172.21.15.189:0/1166027552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e48146e70 0x7f3e34004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.868+0000 7f3e4cc3f700 1 --2- 172.21.15.189:0/1166027552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e34004d90 0x7f3e34005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.868+0000 7f3e4cc3f700 1 -- 172.21.15.189:0/1166027552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e48147250 con 0x7f3e34003a30 2024-04-06T12:12:52.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.868+0000 7f3e4cc3f700 1 -- 172.21.15.189:0/1166027552 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3400bac0 con 0x7f3e34004d90 2024-04-06T12:12:52.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.868+0000 7f3e4cc3f700 1 -- 172.21.15.189:0/1166027552 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3400bc30 con 0x7f3e48146e70 2024-04-06T12:12:52.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.868+0000 7f3e4dc41700 1 --2- 172.21.15.189:0/1166027552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e34003a30 0x7f3e34003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:52.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.868+0000 7f3e4d440700 1 --2- 172.21.15.189:0/1166027552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e48146e70 0x7f3e34004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:52.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.868+0000 7f3e4e442700 1 --2- 172.21.15.189:0/1166027552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e34004d90 0x7f3e34005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:52.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.868+0000 7f3e4dc41700 1 -- 172.21.15.189:0/1166027552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e48146e70 msgr2=0x7f3e34004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:52.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.868+0000 7f3e4dc41700 1 --2- 172.21.15.189:0/1166027552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e48146e70 0x7f3e34004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.868+0000 7f3e4dc41700 1 -- 172.21.15.189:0/1166027552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e34004d90 msgr2=0x7f3e34005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:52.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.868+0000 7f3e4dc41700 1 --2- 172.21.15.189:0/1166027552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e34004d90 0x7f3e34005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.868+0000 7f3e4dc41700 1 -- 172.21.15.189:0/1166027552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e340035b0 con 0x7f3e34003a30 2024-04-06T12:12:52.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.868+0000 7f3e4dc41700 1 --2- 172.21.15.189:0/1166027552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e34003a30 0x7f3e34003e10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3e44009b60 tx=0x7f3e34005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4af62f5ba59e7c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:52.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.868+0000 7f3e4cc3f700 1 -- 172.21.15.189:0/1166027552 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4400e6c0 con 0x7f3e34003a30 2024-04-06T12:12:52.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.868+0000 7f3e4cc3f700 1 -- 172.21.15.189:0/1166027552 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4400e820 con 0x7f3e34003a30 2024-04-06T12:12:52.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.868+0000 7f3e4cc3f700 1 -- 172.21.15.189:0/1166027552 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4400eb90 con 0x7f3e34003a30 2024-04-06T12:12:52.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.869+0000 7f3e4fea5700 1 -- 172.21.15.189:0/1166027552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e34003a30 msgr2=0x7f3e34003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:52.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.869+0000 7f3e4fea5700 1 --2- 172.21.15.189:0/1166027552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e34003a30 0x7f3e34003e10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3e44009b60 tx=0x7f3e34005880 comp rx=0 tx=0).stop 2024-04-06T12:12:52.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.869+0000 7f3e4e442700 1 -- 172.21.15.189:0/1166027552 reap_dead start 2024-04-06T12:12:52.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.869+0000 7f3e4fea5700 1 -- 172.21.15.189:0/1166027552 shutdown_connections 2024-04-06T12:12:52.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.869+0000 7f3e4fea5700 1 -- 172.21.15.189:0/1166027552 >> 172.21.15.189:0/1166027552 conn(0x7f3e480b1e30 msgr2=0x7f3e480b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:52.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.869+0000 7f3e4fea5700 1 -- 172.21.15.189:0/1166027552 shutdown_connections 2024-04-06T12:12:52.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.869+0000 7f3e4fea5700 1 -- 172.21.15.189:0/1166027552 wait complete. 2024-04-06T12:12:52.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.870+0000 7f3e4fea5700 1 Processor -- start 2024-04-06T12:12:52.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.870+0000 7f3e4fea5700 1 -- start start 2024-04-06T12:12:52.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.871+0000 7f3e4fea5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e34003a30 0x7f3e48145d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.871+0000 7f3e4fea5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e480cc6b0 0x7f3e48140d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.871+0000 7f3e4fea5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e480ae8d0 0x7f3e480aed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.871+0000 7f3e4fea5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e480cdcc0 con 0x7f3e480ae8d0 2024-04-06T12:12:52.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.871+0000 7f3e4fea5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e480af290 con 0x7f3e480cc6b0 2024-04-06T12:12:52.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.871+0000 7f3e4fea5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e48146240 con 0x7f3e34003a30 2024-04-06T12:12:52.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.871+0000 7f3e4e442700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e480ae8d0 0x7f3e480aed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:52.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.871+0000 7f3e4e442700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e480ae8d0 0x7f3e480aed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57540/0 (socket says 172.21.15.189:57540) 2024-04-06T12:12:52.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.871+0000 7f3e4e442700 1 -- 172.21.15.189:0/3829319892 learned_addr learned my addr 172.21.15.189:0/3829319892 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:52.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.871+0000 7f3e4d440700 1 --2- 172.21.15.189:0/3829319892 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e480cc6b0 0x7f3e48140d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:52.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.871+0000 7f3e4dc41700 1 --2- 172.21.15.189:0/3829319892 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e34003a30 0x7f3e48145d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:52.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.871+0000 7f3e4e442700 1 -- 172.21.15.189:0/3829319892 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e34003a30 msgr2=0x7f3e48145d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:52.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.871+0000 7f3e4e442700 1 --2- 172.21.15.189:0/3829319892 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e34003a30 0x7f3e48145d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.871+0000 7f3e4e442700 1 -- 172.21.15.189:0/3829319892 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e480cc6b0 msgr2=0x7f3e48140d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:52.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.871+0000 7f3e4e442700 1 --2- 172.21.15.189:0/3829319892 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e480cc6b0 0x7f3e48140d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:52.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.871+0000 7f3e4e442700 1 -- 172.21.15.189:0/3829319892 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e480ccc50 con 0x7f3e480ae8d0 2024-04-06T12:12:52.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.871+0000 7f3e4e442700 1 --2- 172.21.15.189:0/3829319892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e480ae8d0 0x7f3e480aed50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3e480b2e40 tx=0x7f3e4000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:52.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.871+0000 7f3e3e7fc700 1 -- 172.21.15.189:0/3829319892 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e40003c10 con 0x7f3e480ae8d0 2024-04-06T12:12:52.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.872+0000 7f3e3e7fc700 1 -- 172.21.15.189:0/3829319892 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e40003d70 con 0x7f3e480ae8d0 2024-04-06T12:12:52.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.872+0000 7f3e3e7fc700 1 -- 172.21.15.189:0/3829319892 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4000daa0 con 0x7f3e480ae8d0 2024-04-06T12:12:52.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.872+0000 7f3e4fea5700 1 -- 172.21.15.189:0/3829319892 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e480ccf30 con 0x7f3e480ae8d0 2024-04-06T12:12:52.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.872+0000 7f3e4fea5700 1 -- 172.21.15.189:0/3829319892 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e480cd4f0 con 0x7f3e480ae8d0 2024-04-06T12:12:52.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.872+0000 7f3e4fea5700 1 -- 172.21.15.189:0/3829319892 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e480cd6b0 con 0x7f3e480ae8d0 2024-04-06T12:12:52.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.873+0000 7f3e3e7fc700 1 -- 172.21.15.189:0/3829319892 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3e4000dc00 con 0x7f3e480ae8d0 2024-04-06T12:12:52.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.873+0000 7f3e3e7fc700 1 --2- 172.21.15.189:0/3829319892 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3e2c06ac90 0x7f3e2c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:52.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.873+0000 7f3e3e7fc700 1 -- 172.21.15.189:0/3829319892 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3e400898f0 con 0x7f3e480ae8d0 2024-04-06T12:12:52.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.873+0000 7f3e4dc41700 1 --2- 172.21.15.189:0/3829319892 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3e2c06ac90 0x7f3e2c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:52.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.874+0000 7f3e4dc41700 1 --2- 172.21.15.189:0/3829319892 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3e2c06ac90 0x7f3e2c06d150 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3e440090b0 tx=0x7f3e4401f040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:52.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:52.878+0000 7f3e3e7fc700 1 -- 172.21.15.189:0/3829319892 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3e40054740 con 0x7f3e480ae8d0 2024-04-06T12:12:53.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.070+0000 7f3e4fea5700 1 -- 172.21.15.189:0/3829319892 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3e480748f0 con 0x7f3e480ae8d0 2024-04-06T12:12:53.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.070+0000 7f3e3e7fc700 1 -- 172.21.15.189:0/3829319892 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3e40001f40 con 0x7f3e480ae8d0 2024-04-06T12:12:53.068 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:12:53.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.072+0000 7f3e4fea5700 1 -- 172.21.15.189:0/3829319892 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3e2c06ac90 msgr2=0x7f3e2c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:53.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.072+0000 7f3e4fea5700 1 --2- 172.21.15.189:0/3829319892 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3e2c06ac90 0x7f3e2c06d150 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3e440090b0 tx=0x7f3e4401f040 comp rx=0 tx=0).stop 2024-04-06T12:12:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.072+0000 7f3e4fea5700 1 -- 172.21.15.189:0/3829319892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e480ae8d0 msgr2=0x7f3e480aed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.072+0000 7f3e4fea5700 1 --2- 172.21.15.189:0/3829319892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e480ae8d0 0x7f3e480aed50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3e480b2e40 tx=0x7f3e4000d430 comp rx=0 tx=0).stop 2024-04-06T12:12:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.072+0000 7f3e4fea5700 1 -- 172.21.15.189:0/3829319892 shutdown_connections 2024-04-06T12:12:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.072+0000 7f3e4fea5700 1 --2- 172.21.15.189:0/3829319892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e480ae8d0 0x7f3e480aed50 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.072+0000 7f3e4fea5700 1 --2- 172.21.15.189:0/3829319892 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e34003a30 0x7f3e48145d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.072+0000 7f3e4fea5700 1 --2- 172.21.15.189:0/3829319892 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3e2c06ac90 0x7f3e2c06d150 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.072+0000 7f3e4fea5700 1 --2- 172.21.15.189:0/3829319892 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e480cc6b0 0x7f3e48140d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.072+0000 7f3e4fea5700 1 -- 172.21.15.189:0/3829319892 >> 172.21.15.189:0/3829319892 conn(0x7f3e480b1e30 msgr2=0x7f3e48160260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.072+0000 7f3e4fea5700 1 -- 172.21.15.189:0/3829319892 shutdown_connections 2024-04-06T12:12:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.072+0000 7f3e4fea5700 1 -- 172.21.15.189:0/3829319892 wait complete. 2024-04-06T12:12:53.083 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:12:53.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.209+0000 7ff5f4d7f700 1 Processor -- start 2024-04-06T12:12:53.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.209+0000 7ff5f4d7f700 1 -- start start 2024-04-06T12:12:53.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.210+0000 7ff5f4d7f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5f00d11b0 0x7ff5f00d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.210+0000 7ff5f4d7f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff5f00cb870 0x7ff5f015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.211+0000 7ff5f4d7f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f00d1ad0 con 0x7ff5f00cfa00 2024-04-06T12:12:53.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.211+0000 7ff5f4d7f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f00ccdc0 con 0x7ff5f00cb870 2024-04-06T12:12:53.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.211+0000 7ff5f4d7f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f00d0b50 con 0x7ff5f00d11b0 2024-04-06T12:12:53.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.211+0000 7ff5edd9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff5f00cb870 0x7ff5f015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.211+0000 7ff5eed9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff5f00cfa00 0x7ff5f00cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39090/0 (socket says 172.21.15.189:39090) 2024-04-06T12:12:53.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.211+0000 7ff5eed9d700 1 -- 172.21.15.189:0/1797752876 learned_addr learned my addr 172.21.15.189:0/1797752876 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:53.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.211+0000 7ff5ee59c700 1 --2- 172.21.15.189:0/1797752876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5f00d11b0 0x7ff5f00d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.211+0000 7ff5edd9b700 1 -- 172.21.15.189:0/1797752876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5f00d11b0 msgr2=0x7ff5f00d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:53.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.211+0000 7ff5edd9b700 1 --2- 172.21.15.189:0/1797752876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5f00d11b0 0x7ff5f00d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.211+0000 7ff5edd9b700 1 -- 172.21.15.189:0/1797752876 >> v1:172.21.15.189:6789/0 conn(0x7ff5f00cfa00 legacy=0x7ff5f00cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:53.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.211+0000 7ff5edd9b700 1 -- 172.21.15.189:0/1797752876 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f00cbd80 con 0x7ff5f00cb870 2024-04-06T12:12:53.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.211+0000 7ff5ed59a700 1 -- 172.21.15.189:0/1797752876 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158746240 0 0) 0x7ff5f00d1ad0 con 0x7ff5f00cfa00 2024-04-06T12:12:53.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.211+0000 7ff5ee59c700 1 --2- 172.21.15.189:0/1797752876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5f00d11b0 0x7ff5f00d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:53.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.211+0000 7ff5edd9b700 1 --2- 172.21.15.189:0/1797752876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5f00cb870 0x7ff5f015c280 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0009840 tx=0x7ff5e0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e148613b2a3fc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:53.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.211+0000 7ff5ed59a700 1 -- 172.21.15.189:0/1797752876 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0011830 con 0x7ff5f00cb870 2024-04-06T12:12:53.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.211+0000 7ff5ed59a700 1 -- 172.21.15.189:0/1797752876 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e0011990 con 0x7ff5f00cb870 2024-04-06T12:12:53.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.211+0000 7ff5ed59a700 1 -- 172.21.15.189:0/1797752876 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0011d20 con 0x7ff5f00cb870 2024-04-06T12:12:53.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.212+0000 7ff5f4d7f700 1 -- 172.21.15.189:0/1797752876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5f00cb870 msgr2=0x7ff5f015c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:53.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.212+0000 7ff5f4d7f700 1 --2- 172.21.15.189:0/1797752876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5f00cb870 0x7ff5f015c280 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0009840 tx=0x7ff5e0009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:53.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.212+0000 7ff5f4d7f700 1 -- 172.21.15.189:0/1797752876 shutdown_connections 2024-04-06T12:12:53.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.212+0000 7ff5f4d7f700 1 --2- 172.21.15.189:0/1797752876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5f00d11b0 0x7ff5f00d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.212+0000 7ff5f4d7f700 1 --2- 172.21.15.189:0/1797752876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5f00cb870 0x7ff5f015c280 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.212+0000 7ff5f4d7f700 1 -- 172.21.15.189:0/1797752876 >> 172.21.15.189:0/1797752876 conn(0x7ff5f00b2db0 msgr2=0x7ff5f00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:53.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.212+0000 7ff5f4d7f700 1 -- 172.21.15.189:0/1797752876 shutdown_connections 2024-04-06T12:12:53.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.212+0000 7ff5f4d7f700 1 -- 172.21.15.189:0/1797752876 wait complete. 2024-04-06T12:12:53.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.213+0000 7ff5f4d7f700 1 Processor -- start 2024-04-06T12:12:53.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.213+0000 7ff5f4d7f700 1 -- start start 2024-04-06T12:12:53.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.214+0000 7ff5f4d7f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5f00cb870 0x7ff5f00c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.214+0000 7ff5f4d7f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5f00cfa00 0x7ff5f01555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.214+0000 7ff5f4d7f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5f00d11b0 0x7ff5f006d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.214+0000 7ff5f4d7f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f01f6240 con 0x7ff5f00d11b0 2024-04-06T12:12:53.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.214+0000 7ff5f4d7f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f01f6520 con 0x7ff5f00cfa00 2024-04-06T12:12:53.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.214+0000 7ff5f4d7f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f01f6800 con 0x7ff5f00cb870 2024-04-06T12:12:53.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.214+0000 7ff5eed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5f00d11b0 0x7ff5f006d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.214+0000 7ff5eed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5f00d11b0 0x7ff5f006d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57552/0 (socket says 172.21.15.189:57552) 2024-04-06T12:12:53.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.214+0000 7ff5eed9d700 1 -- 172.21.15.189:0/3785901552 learned_addr learned my addr 172.21.15.189:0/3785901552 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:53.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.214+0000 7ff5ee59c700 1 --2- 172.21.15.189:0/3785901552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5f00cb870 0x7ff5f00c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.214+0000 7ff5edd9b700 1 --2- 172.21.15.189:0/3785901552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5f00cfa00 0x7ff5f01555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.214+0000 7ff5eed9d700 1 -- 172.21.15.189:0/3785901552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5f00cb870 msgr2=0x7ff5f00c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:53.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.214+0000 7ff5eed9d700 1 --2- 172.21.15.189:0/3785901552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5f00cb870 0x7ff5f00c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.214+0000 7ff5eed9d700 1 -- 172.21.15.189:0/3785901552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5f00cfa00 msgr2=0x7ff5f01555d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:53.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.214+0000 7ff5eed9d700 1 --2- 172.21.15.189:0/3785901552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5f00cfa00 0x7ff5f01555d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.214+0000 7ff5eed9d700 1 -- 172.21.15.189:0/3785901552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e00094a0 con 0x7ff5f00d11b0 2024-04-06T12:12:53.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.214+0000 7ff5edd9b700 1 --2- 172.21.15.189:0/3785901552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5f00cfa00 0x7ff5f01555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:53.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.215+0000 7ff5eed9d700 1 --2- 172.21.15.189:0/3785901552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5f00d11b0 0x7ff5f006d250 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8002230 tx=0x7ff5e800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:53.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.215+0000 7ff5deffd700 1 -- 172.21.15.189:0/3785901552 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e800ac20 con 0x7ff5f00d11b0 2024-04-06T12:12:53.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.215+0000 7ff5deffd700 1 -- 172.21.15.189:0/3785901552 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e800ada0 con 0x7ff5f00d11b0 2024-04-06T12:12:53.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.215+0000 7ff5f4d7f700 1 -- 172.21.15.189:0/3785901552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f01f6ae0 con 0x7ff5f00d11b0 2024-04-06T12:12:53.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.215+0000 7ff5deffd700 1 -- 172.21.15.189:0/3785901552 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8007310 con 0x7ff5f00d11b0 2024-04-06T12:12:53.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.215+0000 7ff5f4d7f700 1 -- 172.21.15.189:0/3785901552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f01f70a0 con 0x7ff5f00d11b0 2024-04-06T12:12:53.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.216+0000 7ff5f4d7f700 1 -- 172.21.15.189:0/3785901552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f0074990 con 0x7ff5f00d11b0 2024-04-06T12:12:53.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.216+0000 7ff5deffd700 1 -- 172.21.15.189:0/3785901552 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff5e8007470 con 0x7ff5f00d11b0 2024-04-06T12:12:53.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.216+0000 7ff5deffd700 1 --2- 172.21.15.189:0/3785901552 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff5d006ac90 0x7ff5d006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.217+0000 7ff5deffd700 1 -- 172.21.15.189:0/3785901552 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff5e8089450 con 0x7ff5f00d11b0 2024-04-06T12:12:53.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.217+0000 7ff5ee59c700 1 --2- 172.21.15.189:0/3785901552 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff5d006ac90 0x7ff5d006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.217+0000 7ff5ee59c700 1 --2- 172.21.15.189:0/3785901552 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff5d006ac90 0x7ff5d006d150 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4006fd0 tx=0x7ff5e4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:53.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.221+0000 7ff5deffd700 1 -- 172.21.15.189:0/3785901552 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff5e80543d0 con 0x7ff5f00d11b0 2024-04-06T12:12:53.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.421+0000 7ff5f4d7f700 1 -- 172.21.15.189:0/3785901552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff5f00d1e20 con 0x7ff5f00d11b0 2024-04-06T12:12:53.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.421+0000 7ff5deffd700 1 -- 172.21.15.189:0/3785901552 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff5e80579f0 con 0x7ff5f00d11b0 2024-04-06T12:12:53.421 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:12:53.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.423+0000 7ff5f4d7f700 1 -- 172.21.15.189:0/3785901552 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff5d006ac90 msgr2=0x7ff5d006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:53.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.423+0000 7ff5f4d7f700 1 --2- 172.21.15.189:0/3785901552 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff5d006ac90 0x7ff5d006d150 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4006fd0 tx=0x7ff5e4008040 comp rx=0 tx=0).stop 2024-04-06T12:12:53.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.423+0000 7ff5f4d7f700 1 -- 172.21.15.189:0/3785901552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5f00d11b0 msgr2=0x7ff5f006d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:53.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.423+0000 7ff5f4d7f700 1 --2- 172.21.15.189:0/3785901552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5f00d11b0 0x7ff5f006d250 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8002230 tx=0x7ff5e800a060 comp rx=0 tx=0).stop 2024-04-06T12:12:53.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.423+0000 7ff5f4d7f700 1 -- 172.21.15.189:0/3785901552 shutdown_connections 2024-04-06T12:12:53.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.423+0000 7ff5f4d7f700 1 --2- 172.21.15.189:0/3785901552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5f00d11b0 0x7ff5f006d250 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.423+0000 7ff5f4d7f700 1 --2- 172.21.15.189:0/3785901552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5f00cb870 0x7ff5f00c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.423+0000 7ff5f4d7f700 1 --2- 172.21.15.189:0/3785901552 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff5d006ac90 0x7ff5d006d150 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.423+0000 7ff5f4d7f700 1 --2- 172.21.15.189:0/3785901552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5f00cfa00 0x7ff5f01555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.423+0000 7ff5f4d7f700 1 -- 172.21.15.189:0/3785901552 >> 172.21.15.189:0/3785901552 conn(0x7ff5f00b2db0 msgr2=0x7ff5f015a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:53.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.424+0000 7ff5f4d7f700 1 -- 172.21.15.189:0/3785901552 shutdown_connections 2024-04-06T12:12:53.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.424+0000 7ff5f4d7f700 1 -- 172.21.15.189:0/3785901552 wait complete. 2024-04-06T12:12:53.435 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:12:53.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.561+0000 7f0cc2b91700 1 Processor -- start 2024-04-06T12:12:53.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.561+0000 7f0cc2b91700 1 -- start start 2024-04-06T12:12:53.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.562+0000 7f0cc2b91700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0cbc0cea80 0x7f0cbc0cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.562+0000 7f0cc2b91700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cbc0cdae0 0x7f0cbc06ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.562+0000 7f0cc2b91700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cbc15a830 con 0x7f0cbc0aec70 2024-04-06T12:12:53.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.562+0000 7f0cc2b91700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cbc0cd920 con 0x7f0cbc0cea80 2024-04-06T12:12:53.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.562+0000 7f0cc2b91700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc0cc050 con 0x7f0cbc0cdae0 2024-04-06T12:12:53.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.562+0000 7f0cbbfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0cbc0cea80 0x7f0cbc0cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.562+0000 7f0cc092d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0cbc0aec70 0x7f0cbc0ce370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39096/0 (socket says 172.21.15.189:39096) 2024-04-06T12:12:53.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.562+0000 7f0cc092d700 1 -- 172.21.15.189:0/3535743784 learned_addr learned my addr 172.21.15.189:0/3535743784 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:53.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.562+0000 7f0cbbfff700 1 -- 172.21.15.189:0/3535743784 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cbc0cdae0 msgr2=0x7f0cbc06ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:53.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.562+0000 7f0cbbfff700 1 --2- 172.21.15.189:0/3535743784 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cbc0cdae0 0x7f0cbc06ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.562+0000 7f0cbbfff700 1 -- 172.21.15.189:0/3535743784 >> v1:172.21.15.189:6789/0 conn(0x7f0cbc0aec70 legacy=0x7f0cbc0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:53.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.562+0000 7f0cbbfff700 1 -- 172.21.15.189:0/3535743784 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cbc0666f0 con 0x7f0cbc0cea80 2024-04-06T12:12:53.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.562+0000 7f0cbb7fe700 1 -- 172.21.15.189:0/3535743784 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4179583578 0 0) 0x7f0cbc15a830 con 0x7f0cbc0aec70 2024-04-06T12:12:53.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.562+0000 7f0cbbfff700 1 --2- 172.21.15.189:0/3535743784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cbc0cea80 0x7f0cbc0cd3e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8009840 tx=0x7f0ca8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd4526345891b83 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:53.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.562+0000 7f0cbb7fe700 1 -- 172.21.15.189:0/3535743784 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8011830 con 0x7f0cbc0cea80 2024-04-06T12:12:53.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.563+0000 7f0cbb7fe700 1 -- 172.21.15.189:0/3535743784 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca8011990 con 0x7f0cbc0cea80 2024-04-06T12:12:53.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.563+0000 7f0cbb7fe700 1 -- 172.21.15.189:0/3535743784 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8011d20 con 0x7f0cbc0cea80 2024-04-06T12:12:53.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.563+0000 7f0cc2b91700 1 -- 172.21.15.189:0/3535743784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cbc0cea80 msgr2=0x7f0cbc0cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:53.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.563+0000 7f0cc2b91700 1 --2- 172.21.15.189:0/3535743784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cbc0cea80 0x7f0cbc0cd3e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8009840 tx=0x7f0ca8009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:53.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.563+0000 7f0cc2b91700 1 -- 172.21.15.189:0/3535743784 shutdown_connections 2024-04-06T12:12:53.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.563+0000 7f0cc2b91700 1 --2- 172.21.15.189:0/3535743784 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cbc0cdae0 0x7f0cbc06ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.563+0000 7f0cc2b91700 1 --2- 172.21.15.189:0/3535743784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cbc0cea80 0x7f0cbc0cd3e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.563+0000 7f0cc2b91700 1 -- 172.21.15.189:0/3535743784 >> 172.21.15.189:0/3535743784 conn(0x7f0cbc0cc290 msgr2=0x7f0cbc0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:53.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.563+0000 7f0cc2b91700 1 -- 172.21.15.189:0/3535743784 shutdown_connections 2024-04-06T12:12:53.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.563+0000 7f0cc2b91700 1 -- 172.21.15.189:0/3535743784 wait complete. 2024-04-06T12:12:53.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.565+0000 7f0cc2b91700 1 Processor -- start 2024-04-06T12:12:53.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.565+0000 7f0cc2b91700 1 -- start start 2024-04-06T12:12:53.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cc2b91700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cbc0aec70 0x7f0cbc0c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cc2b91700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cbc0cdae0 0x7f0cbc0b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cc2b91700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cbc0cea80 0x7f0cbc0c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cc2b91700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc1f22e0 con 0x7f0cbc0cdae0 2024-04-06T12:12:53.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cc2b91700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc0c6640 con 0x7f0cbc0aec70 2024-04-06T12:12:53.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cc2b91700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc156950 con 0x7f0cbc0cea80 2024-04-06T12:12:53.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cbbfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cbc0cdae0 0x7f0cbc0b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cbbfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cbc0cdae0 0x7f0cbc0b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57556/0 (socket says 172.21.15.189:57556) 2024-04-06T12:12:53.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cbbfff700 1 -- 172.21.15.189:0/1711002067 learned_addr learned my addr 172.21.15.189:0/1711002067 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:53.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cc112e700 1 --2- 172.21.15.189:0/1711002067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cbc0cea80 0x7f0cbc0c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cc092d700 1 --2- 172.21.15.189:0/1711002067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cbc0aec70 0x7f0cbc0c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cbbfff700 1 -- 172.21.15.189:0/1711002067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cbc0cea80 msgr2=0x7f0cbc0c8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:53.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cbbfff700 1 --2- 172.21.15.189:0/1711002067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cbc0cea80 0x7f0cbc0c8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cbbfff700 1 -- 172.21.15.189:0/1711002067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cbc0aec70 msgr2=0x7f0cbc0c6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:53.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cbbfff700 1 --2- 172.21.15.189:0/1711002067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cbc0aec70 0x7f0cbc0c6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cbbfff700 1 -- 172.21.15.189:0/1711002067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca80094a0 con 0x7f0cbc0cdae0 2024-04-06T12:12:53.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cc092d700 1 --2- 172.21.15.189:0/1711002067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cbc0aec70 0x7f0cbc0c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:53.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cbbfff700 1 --2- 172.21.15.189:0/1711002067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cbc0cdae0 0x7f0cbc0b4f20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0ca80090b0 tx=0x7f0ca8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:53.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cb97fa700 1 -- 172.21.15.189:0/1711002067 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca80036a0 con 0x7f0cbc0cdae0 2024-04-06T12:12:53.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.566+0000 7f0cb97fa700 1 -- 172.21.15.189:0/1711002067 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca801e070 con 0x7f0cbc0cdae0 2024-04-06T12:12:53.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.567+0000 7f0cc2b91700 1 -- 172.21.15.189:0/1711002067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cbc156670 con 0x7f0cbc0cdae0 2024-04-06T12:12:53.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.567+0000 7f0cb97fa700 1 -- 172.21.15.189:0/1711002067 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8003a40 con 0x7f0cbc0cdae0 2024-04-06T12:12:53.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.567+0000 7f0cc2b91700 1 -- 172.21.15.189:0/1711002067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cbc1f1f70 con 0x7f0cbc0cdae0 2024-04-06T12:12:53.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.568+0000 7f0cc2b91700 1 -- 172.21.15.189:0/1711002067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cbc074990 con 0x7f0cbc0cdae0 2024-04-06T12:12:53.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.568+0000 7f0cb97fa700 1 -- 172.21.15.189:0/1711002067 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0ca8003ba0 con 0x7f0cbc0cdae0 2024-04-06T12:12:53.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.568+0000 7f0cb97fa700 1 --2- 172.21.15.189:0/1711002067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0ca006ac90 0x7f0ca006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.568+0000 7f0cb97fa700 1 -- 172.21.15.189:0/1711002067 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0ca8020070 con 0x7f0cbc0cdae0 2024-04-06T12:12:53.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.568+0000 7f0cc092d700 1 --2- 172.21.15.189:0/1711002067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0ca006ac90 0x7f0ca006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.569+0000 7f0cc092d700 1 --2- 172.21.15.189:0/1711002067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0ca006ac90 0x7f0ca006d150 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0000f60 tx=0x7f0cb0007ca0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:53.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.573+0000 7f0cb97fa700 1 -- 172.21.15.189:0/1711002067 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0ca8055a00 con 0x7f0cbc0cdae0 2024-04-06T12:12:53.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.767+0000 7f0cc2b91700 1 -- 172.21.15.189:0/1711002067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0cbc067250 con 0x7f0cbc0cdae0 2024-04-06T12:12:53.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.767+0000 7f0cb97fa700 1 -- 172.21.15.189:0/1711002067 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0ca802a020 con 0x7f0cbc0cdae0 2024-04-06T12:12:53.766 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:12:53.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.769+0000 7f0cc2b91700 1 -- 172.21.15.189:0/1711002067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0ca006ac90 msgr2=0x7f0ca006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:53.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.769+0000 7f0cc2b91700 1 --2- 172.21.15.189:0/1711002067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0ca006ac90 0x7f0ca006d150 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0000f60 tx=0x7f0cb0007ca0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.769+0000 7f0cc2b91700 1 -- 172.21.15.189:0/1711002067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cbc0cdae0 msgr2=0x7f0cbc0b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:53.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.769+0000 7f0cc2b91700 1 --2- 172.21.15.189:0/1711002067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cbc0cdae0 0x7f0cbc0b4f20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0ca80090b0 tx=0x7f0ca8004750 comp rx=0 tx=0).stop 2024-04-06T12:12:53.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.769+0000 7f0cc2b91700 1 -- 172.21.15.189:0/1711002067 shutdown_connections 2024-04-06T12:12:53.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.769+0000 7f0cc2b91700 1 --2- 172.21.15.189:0/1711002067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cbc0cea80 0x7f0cbc0c8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.769+0000 7f0cc2b91700 1 --2- 172.21.15.189:0/1711002067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0ca006ac90 0x7f0ca006d150 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.769+0000 7f0cc2b91700 1 --2- 172.21.15.189:0/1711002067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cbc0aec70 0x7f0cbc0c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.769+0000 7f0cc2b91700 1 --2- 172.21.15.189:0/1711002067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cbc0cdae0 0x7f0cbc0b4f20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.769+0000 7f0cc2b91700 1 -- 172.21.15.189:0/1711002067 >> 172.21.15.189:0/1711002067 conn(0x7f0cbc0cc290 msgr2=0x7f0cbc0cd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:53.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.769+0000 7f0cc2b91700 1 -- 172.21.15.189:0/1711002067 shutdown_connections 2024-04-06T12:12:53.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.769+0000 7f0cc2b91700 1 -- 172.21.15.189:0/1711002067 wait complete. 2024-04-06T12:12:53.780 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-06T12:12:53.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.905+0000 7fd3cbbaf700 1 Processor -- start 2024-04-06T12:12:53.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.906+0000 7fd3cbbaf700 1 -- start start 2024-04-06T12:12:53.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.906+0000 7fd3cbbaf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3c415d7b0 0x7fd3c40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.906+0000 7fd3cbbaf700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd3c406a730 0x7fd3c406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3cbbaf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3c40b3990 con 0x7fd3c415ddf0 2024-04-06T12:12:53.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3cbbaf700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3c406b0f0 con 0x7fd3c406a730 2024-04-06T12:12:53.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3cbbaf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c40b3060 con 0x7fd3c415d7b0 2024-04-06T12:12:53.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3ca14c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd3c415ddf0 0x7fd3c415e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39110/0 (socket says 172.21.15.189:39110) 2024-04-06T12:12:53.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3ca14c700 1 -- 172.21.15.189:0/1225967693 learned_addr learned my addr 172.21.15.189:0/1225967693 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:53.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c8949700 1 -- 172.21.15.189:0/1225967693 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1304048282 0 0) 0x7fd3c40b3990 con 0x7fd3c415ddf0 2024-04-06T12:12:53.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c8949700 1 -- 172.21.15.189:0/1225967693 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3b00035b0 con 0x7fd3c415ddf0 2024-04-06T12:12:53.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c994b700 1 --2- 172.21.15.189:0/1225967693 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3c415d7b0 0x7fd3c40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c914a700 1 --2- 172.21.15.189:0/1225967693 >> v2:172.21.15.192:3300/0 conn(0x7fd3c406a730 0x7fd3c406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c8949700 1 -- 172.21.15.189:0/1225967693 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2717490021 0 0) 0x7fd3b00035b0 con 0x7fd3c415ddf0 2024-04-06T12:12:53.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c8949700 1 -- 172.21.15.189:0/1225967693 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3c415d7b0 msgr2=0x7fd3c40b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:53.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c8949700 1 --2- 172.21.15.189:0/1225967693 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3c415d7b0 0x7fd3c40b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c8949700 1 -- 172.21.15.189:0/1225967693 >> v2:172.21.15.192:3300/0 conn(0x7fd3c406a730 msgr2=0x7fd3c406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:53.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c8949700 1 --2- 172.21.15.189:0/1225967693 >> v2:172.21.15.192:3300/0 conn(0x7fd3c406a730 0x7fd3c406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c8949700 1 -- 172.21.15.189:0/1225967693 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c40b3340 con 0x7fd3c415ddf0 2024-04-06T12:12:53.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c8949700 1 -- 172.21.15.189:0/1225967693 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fd3bc0019f0 con 0x7fd3c415ddf0 2024-04-06T12:12:53.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c8949700 1 -- 172.21.15.189:0/1225967693 >> v1:172.21.15.189:6789/0 conn(0x7fd3c415ddf0 legacy=0x7fd3c415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:53.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c8949700 1 --2- 172.21.15.189:0/1225967693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3b0003a00 0x7fd3b0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c8949700 1 --2- 172.21.15.189:0/1225967693 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3c415d7b0 0x7fd3b0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c8949700 1 --2- 172.21.15.189:0/1225967693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3b0004d90 0x7fd3b0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c8949700 1 -- 172.21.15.189:0/1225967693 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c406b0f0 con 0x7fd3b0004d90 2024-04-06T12:12:53.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c8949700 1 -- 172.21.15.189:0/1225967693 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c40b3060 con 0x7fd3b0003a00 2024-04-06T12:12:53.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c8949700 1 -- 172.21.15.189:0/1225967693 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b000bac0 con 0x7fd3c415d7b0 2024-04-06T12:12:53.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3ca14c700 1 --2- 172.21.15.189:0/1225967693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3b0004d90 0x7fd3b0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c914a700 1 --2- 172.21.15.189:0/1225967693 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3c415d7b0 0x7fd3b0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.907+0000 7fd3c994b700 1 --2- 172.21.15.189:0/1225967693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3b0003a00 0x7fd3b0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.908+0000 7fd3ca14c700 1 -- 172.21.15.189:0/1225967693 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3c415d7b0 msgr2=0x7fd3b0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:53.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.908+0000 7fd3ca14c700 1 --2- 172.21.15.189:0/1225967693 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3c415d7b0 0x7fd3b0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.908+0000 7fd3ca14c700 1 -- 172.21.15.189:0/1225967693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3b0003a00 msgr2=0x7fd3b0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:53.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.908+0000 7fd3ca14c700 1 --2- 172.21.15.189:0/1225967693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3b0003a00 0x7fd3b0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.908+0000 7fd3ca14c700 1 -- 172.21.15.189:0/1225967693 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b00035b0 con 0x7fd3b0004d90 2024-04-06T12:12:53.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.908+0000 7fd3ca14c700 1 --2- 172.21.15.189:0/1225967693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3b0004d90 0x7fd3b0005210 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc00a8d0 tx=0x7fd3b0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f16a155a919b456d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:53.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.908+0000 7fd3c8949700 1 -- 172.21.15.189:0/1225967693 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc00a340 con 0x7fd3b0004d90 2024-04-06T12:12:53.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.908+0000 7fd3c8949700 1 -- 172.21.15.189:0/1225967693 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3bc012600 con 0x7fd3b0004d90 2024-04-06T12:12:53.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.908+0000 7fd3c8949700 1 -- 172.21.15.189:0/1225967693 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc0129b0 con 0x7fd3b0004d90 2024-04-06T12:12:53.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.908+0000 7fd3cbbaf700 1 -- 172.21.15.189:0/1225967693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3b0004d90 msgr2=0x7fd3b0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:53.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.908+0000 7fd3cbbaf700 1 --2- 172.21.15.189:0/1225967693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3b0004d90 0x7fd3b0005210 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc00a8d0 tx=0x7fd3b0005880 comp rx=0 tx=0).stop 2024-04-06T12:12:53.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.908+0000 7fd3ca14c700 1 -- 172.21.15.189:0/1225967693 reap_dead start 2024-04-06T12:12:53.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.908+0000 7fd3cbbaf700 1 -- 172.21.15.189:0/1225967693 shutdown_connections 2024-04-06T12:12:53.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.909+0000 7fd3cbbaf700 1 -- 172.21.15.189:0/1225967693 >> 172.21.15.189:0/1225967693 conn(0x7fd3c40d00b0 msgr2=0x7fd3c40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:53.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.909+0000 7fd3cbbaf700 1 -- 172.21.15.189:0/1225967693 shutdown_connections 2024-04-06T12:12:53.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.909+0000 7fd3cbbaf700 1 -- 172.21.15.189:0/1225967693 wait complete. 2024-04-06T12:12:53.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.910+0000 7fd3cbbaf700 1 Processor -- start 2024-04-06T12:12:53.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.910+0000 7fd3cbbaf700 1 -- start start 2024-04-06T12:12:53.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.910+0000 7fd3cbbaf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3b0004d90 0x7fd3c4146c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.910+0000 7fd3cbbaf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3c40cc570 0x7fd3c415ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.910+0000 7fd3cbbaf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3c40cd770 0x7fd3c40cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.910+0000 7fd3cbbaf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c40ce140 con 0x7fd3c40cc570 2024-04-06T12:12:53.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.910+0000 7fd3cbbaf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c415b750 con 0x7fd3c40cd770 2024-04-06T12:12:53.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.910+0000 7fd3cbbaf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c415ba30 con 0x7fd3b0004d90 2024-04-06T12:12:53.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.911+0000 7fd3c914a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3c40cc570 0x7fd3c415ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.911+0000 7fd3c914a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3c40cc570 0x7fd3c415ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57568/0 (socket says 172.21.15.189:57568) 2024-04-06T12:12:53.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.911+0000 7fd3c914a700 1 -- 172.21.15.189:0/2488409436 learned_addr learned my addr 172.21.15.189:0/2488409436 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:53.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.911+0000 7fd3c994b700 1 --2- 172.21.15.189:0/2488409436 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3b0004d90 0x7fd3c4146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.911+0000 7fd3ca14c700 1 --2- 172.21.15.189:0/2488409436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3c40cd770 0x7fd3c40cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.911+0000 7fd3c914a700 1 -- 172.21.15.189:0/2488409436 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3b0004d90 msgr2=0x7fd3c4146c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:53.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.911+0000 7fd3c914a700 1 --2- 172.21.15.189:0/2488409436 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3b0004d90 0x7fd3c4146c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.911+0000 7fd3c914a700 1 -- 172.21.15.189:0/2488409436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3c40cd770 msgr2=0x7fd3c40cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:53.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.911+0000 7fd3c914a700 1 --2- 172.21.15.189:0/2488409436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3c40cd770 0x7fd3c40cdc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:53.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.911+0000 7fd3c914a700 1 -- 172.21.15.189:0/2488409436 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c415bd10 con 0x7fd3c40cc570 2024-04-06T12:12:53.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.911+0000 7fd3ca14c700 1 --2- 172.21.15.189:0/2488409436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3c40cd770 0x7fd3c40cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:53.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.911+0000 7fd3c914a700 1 --2- 172.21.15.189:0/2488409436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3c40cc570 0x7fd3c415ae70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd3c0011900 tx=0x7fd3c0011cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:53.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.911+0000 7fd3b67fc700 1 -- 172.21.15.189:0/2488409436 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c000c070 con 0x7fd3c40cc570 2024-04-06T12:12:53.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.911+0000 7fd3b67fc700 1 -- 172.21.15.189:0/2488409436 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c000f8d0 con 0x7fd3c40cc570 2024-04-06T12:12:53.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.911+0000 7fd3cbbaf700 1 -- 172.21.15.189:0/2488409436 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3c415bff0 con 0x7fd3c40cc570 2024-04-06T12:12:53.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.911+0000 7fd3b67fc700 1 -- 172.21.15.189:0/2488409436 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c000fc80 con 0x7fd3c40cc570 2024-04-06T12:12:53.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.911+0000 7fd3cbbaf700 1 -- 172.21.15.189:0/2488409436 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3c41493f0 con 0x7fd3c40cc570 2024-04-06T12:12:53.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.912+0000 7fd3cbbaf700 1 -- 172.21.15.189:0/2488409436 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3c4067bb0 con 0x7fd3c40cc570 2024-04-06T12:12:53.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.913+0000 7fd3b67fc700 1 -- 172.21.15.189:0/2488409436 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd3c0015460 con 0x7fd3c40cc570 2024-04-06T12:12:53.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.913+0000 7fd3b67fc700 1 --2- 172.21.15.189:0/2488409436 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3a806ac90 0x7fd3a806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:53.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.913+0000 7fd3b67fc700 1 -- 172.21.15.189:0/2488409436 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd3c0089910 con 0x7fd3c40cc570 2024-04-06T12:12:53.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.913+0000 7fd3c994b700 1 --2- 172.21.15.189:0/2488409436 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3a806ac90 0x7fd3a806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:53.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.913+0000 7fd3c994b700 1 --2- 172.21.15.189:0/2488409436 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3a806ac90 0x7fd3a806d150 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8009be0 tx=0x7fd3b8009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:53.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:53.917+0000 7fd3b67fc700 1 -- 172.21.15.189:0/2488409436 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd3c0054730 con 0x7fd3c40cc570 2024-04-06T12:12:54.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.110+0000 7fd3cbbaf700 1 -- 172.21.15.189:0/2488409436 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd3c4074990 con 0x7fd3c40cc570 2024-04-06T12:12:54.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.110+0000 7fd3b67fc700 1 -- 172.21.15.189:0/2488409436 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd3c00230a0 con 0x7fd3c40cc570 2024-04-06T12:12:54.109 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:12:54.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.112+0000 7fd3cbbaf700 1 -- 172.21.15.189:0/2488409436 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3a806ac90 msgr2=0x7fd3a806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:54.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.112+0000 7fd3cbbaf700 1 --2- 172.21.15.189:0/2488409436 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3a806ac90 0x7fd3a806d150 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8009be0 tx=0x7fd3b8009210 comp rx=0 tx=0).stop 2024-04-06T12:12:54.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.113+0000 7fd3cbbaf700 1 -- 172.21.15.189:0/2488409436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3c40cc570 msgr2=0x7fd3c415ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:54.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.113+0000 7fd3cbbaf700 1 --2- 172.21.15.189:0/2488409436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3c40cc570 0x7fd3c415ae70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd3c0011900 tx=0x7fd3c0011cc0 comp rx=0 tx=0).stop 2024-04-06T12:12:54.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.113+0000 7fd3cbbaf700 1 -- 172.21.15.189:0/2488409436 shutdown_connections 2024-04-06T12:12:54.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.113+0000 7fd3cbbaf700 1 --2- 172.21.15.189:0/2488409436 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3a806ac90 0x7fd3a806d150 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:54.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.113+0000 7fd3cbbaf700 1 --2- 172.21.15.189:0/2488409436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3c40cd770 0x7fd3c40cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:54.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.113+0000 7fd3cbbaf700 1 --2- 172.21.15.189:0/2488409436 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3b0004d90 0x7fd3c4146c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:54.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.113+0000 7fd3cbbaf700 1 --2- 172.21.15.189:0/2488409436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3c40cc570 0x7fd3c415ae70 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:54.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.113+0000 7fd3cbbaf700 1 -- 172.21.15.189:0/2488409436 >> 172.21.15.189:0/2488409436 conn(0x7fd3c40d00b0 msgr2=0x7fd3c415ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:54.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.113+0000 7fd3cbbaf700 1 -- 172.21.15.189:0/2488409436 shutdown_connections 2024-04-06T12:12:54.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.113+0000 7fd3cbbaf700 1 -- 172.21.15.189:0/2488409436 wait complete. 2024-04-06T12:12:54.155 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2024-04-06T12:12:54.156 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-06T12:12:54.156 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2024-04-06T12:12:54.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.279+0000 7f5846373700 1 Processor -- start 2024-04-06T12:12:54.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.279+0000 7f5846373700 1 -- start start 2024-04-06T12:12:54.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f5846373700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f584015d7e0 0x7f58400d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:54.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f5846373700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58400d0890 0x7f58400d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:54.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f5846373700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f58400d1af0 0x7f58400b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:54.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f5846373700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58400d1f80 con 0x7f58400d1af0 2024-04-06T12:12:54.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f5846373700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58400cdf80 con 0x7f584015d7e0 2024-04-06T12:12:54.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f5846373700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58400d0cf0 con 0x7f58400d0890 2024-04-06T12:12:54.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f5844910700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f58400d1af0 0x7f58400b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:54.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f5844910700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f58400d1af0 0x7f58400b4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57572/0 (socket says 172.21.15.189:57572) 2024-04-06T12:12:54.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f5844910700 1 -- 172.21.15.189:0/3332554790 learned_addr learned my addr 172.21.15.189:0/3332554790 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:54.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f583f7fe700 1 --2- 172.21.15.189:0/3332554790 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58400d0890 0x7f58400d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:54.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f583ffff700 1 --2- 172.21.15.189:0/3332554790 >> v2:172.21.15.192:3300/0 conn(0x7f584015d7e0 0x7f58400d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:54.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f5844910700 1 -- 172.21.15.189:0/3332554790 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58400d0890 msgr2=0x7f58400d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:54.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f5844910700 1 --2- 172.21.15.189:0/3332554790 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58400d0890 0x7f58400d15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:54.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f5844910700 1 -- 172.21.15.189:0/3332554790 >> v2:172.21.15.192:3300/0 conn(0x7f584015d7e0 msgr2=0x7f58400d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:54.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f5844910700 1 --2- 172.21.15.189:0/3332554790 >> v2:172.21.15.192:3300/0 conn(0x7f584015d7e0 0x7f58400d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:54.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f5844910700 1 -- 172.21.15.189:0/3332554790 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58400d3230 con 0x7f58400d1af0 2024-04-06T12:12:54.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f583f7fe700 1 --2- 172.21.15.189:0/3332554790 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58400d0890 0x7f58400d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:54.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f5844910700 1 --2- 172.21.15.189:0/3332554790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58400d1af0 0x7f58400b4e50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f584008c190 tx=0x7f583800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f919ca70342157d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:54.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f583effd700 1 -- 172.21.15.189:0/3332554790 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583800e070 con 0x7f58400d1af0 2024-04-06T12:12:54.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f583effd700 1 -- 172.21.15.189:0/3332554790 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583800b9c0 con 0x7f58400d1af0 2024-04-06T12:12:54.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.280+0000 7f583effd700 1 -- 172.21.15.189:0/3332554790 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583800bd70 con 0x7f58400d1af0 2024-04-06T12:12:54.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.281+0000 7f5846373700 1 -- 172.21.15.189:0/3332554790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58400d1af0 msgr2=0x7f58400b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:54.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.281+0000 7f5846373700 1 --2- 172.21.15.189:0/3332554790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58400d1af0 0x7f58400b4e50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f584008c190 tx=0x7f583800b330 comp rx=0 tx=0).stop 2024-04-06T12:12:54.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.281+0000 7f5846373700 1 -- 172.21.15.189:0/3332554790 shutdown_connections 2024-04-06T12:12:54.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.281+0000 7f5846373700 1 --2- 172.21.15.189:0/3332554790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58400d1af0 0x7f58400b4e50 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:54.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.281+0000 7f5846373700 1 --2- 172.21.15.189:0/3332554790 >> v2:172.21.15.192:3300/0 conn(0x7f584015d7e0 0x7f58400d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:54.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.281+0000 7f5846373700 1 --2- 172.21.15.189:0/3332554790 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58400d0890 0x7f58400d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:54.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.281+0000 7f5846373700 1 -- 172.21.15.189:0/3332554790 >> 172.21.15.189:0/3332554790 conn(0x7f5840072250 msgr2=0x7f5840072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:54.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.281+0000 7f5846373700 1 -- 172.21.15.189:0/3332554790 shutdown_connections 2024-04-06T12:12:54.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.281+0000 7f5846373700 1 -- 172.21.15.189:0/3332554790 wait complete. 2024-04-06T12:12:54.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.282+0000 7f5846373700 1 Processor -- start 2024-04-06T12:12:54.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.282+0000 7f5846373700 1 -- start start 2024-04-06T12:12:54.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.283+0000 7f5846373700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58400d0890 0x7f58401f5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:54.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.283+0000 7f5846373700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58400d1af0 0x7f584015e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:54.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.283+0000 7f5846373700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f584015d7e0 0x7f584014e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:54.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.283+0000 7f5846373700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58400c9bf0 con 0x7f58400d1af0 2024-04-06T12:12:54.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.283+0000 7f5846373700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58401f6510 con 0x7f584015d7e0 2024-04-06T12:12:54.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.283+0000 7f5846373700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58401f6770 con 0x7f58400d0890 2024-04-06T12:12:54.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.283+0000 7f583f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58400d1af0 0x7f584015e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:54.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.283+0000 7f583f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58400d1af0 0x7f584015e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57586/0 (socket says 172.21.15.189:57586) 2024-04-06T12:12:54.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.283+0000 7f583f7fe700 1 -- 172.21.15.189:0/685778836 learned_addr learned my addr 172.21.15.189:0/685778836 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:54.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.283+0000 7f583ffff700 1 --2- 172.21.15.189:0/685778836 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58400d0890 0x7f58401f5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:54.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.283+0000 7f5844910700 1 --2- 172.21.15.189:0/685778836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f584015d7e0 0x7f584014e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:54.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.283+0000 7f583f7fe700 1 -- 172.21.15.189:0/685778836 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58400d0890 msgr2=0x7f58401f5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:54.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.283+0000 7f583f7fe700 1 --2- 172.21.15.189:0/685778836 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58400d0890 0x7f58401f5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:54.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.283+0000 7f583f7fe700 1 -- 172.21.15.189:0/685778836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f584015d7e0 msgr2=0x7f584014e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:54.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.283+0000 7f583f7fe700 1 --2- 172.21.15.189:0/685778836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f584015d7e0 0x7f584014e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:54.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.283+0000 7f583f7fe700 1 -- 172.21.15.189:0/685778836 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5838009b30 con 0x7f58400d1af0 2024-04-06T12:12:54.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.283+0000 7f583ffff700 1 --2- 172.21.15.189:0/685778836 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58400d0890 0x7f58401f5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:54.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.284+0000 7f583f7fe700 1 --2- 172.21.15.189:0/685778836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58400d1af0 0x7f584015e710 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f583400e760 tx=0x7f583400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:54.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.284+0000 7f583cff9700 1 -- 172.21.15.189:0/685778836 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5834013610 con 0x7f58400d1af0 2024-04-06T12:12:54.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.284+0000 7f583cff9700 1 -- 172.21.15.189:0/685778836 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5834013770 con 0x7f58400d1af0 2024-04-06T12:12:54.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.284+0000 7f5846373700 1 -- 172.21.15.189:0/685778836 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58401f6a50 con 0x7f58400d1af0 2024-04-06T12:12:54.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.284+0000 7f583cff9700 1 -- 172.21.15.189:0/685778836 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5834013b20 con 0x7f58400d1af0 2024-04-06T12:12:54.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.284+0000 7f5846373700 1 -- 172.21.15.189:0/685778836 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f584014dff0 con 0x7f58400d1af0 2024-04-06T12:12:54.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.285+0000 7f5846373700 1 -- 172.21.15.189:0/685778836 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f584014e1e0 con 0x7f58400d1af0 2024-04-06T12:12:54.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.285+0000 7f583cff9700 1 -- 172.21.15.189:0/685778836 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5834013c80 con 0x7f58400d1af0 2024-04-06T12:12:54.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.286+0000 7f583cff9700 1 --2- 172.21.15.189:0/685778836 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f582c06ac90 0x7f582c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:54.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.286+0000 7f583cff9700 1 -- 172.21.15.189:0/685778836 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5834089340 con 0x7f58400d1af0 2024-04-06T12:12:54.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.286+0000 7f583ffff700 1 --2- 172.21.15.189:0/685778836 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f582c06ac90 0x7f582c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:54.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.286+0000 7f583ffff700 1 --2- 172.21.15.189:0/685778836 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f582c06ac90 0x7f582c06d150 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5830009970 tx=0x7f5830008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:54.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.290+0000 7f583cff9700 1 -- 172.21.15.189:0/685778836 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5834054160 con 0x7f58400d1af0 2024-04-06T12:12:54.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:54.577+0000 7f5846373700 1 -- 172.21.15.189:0/685778836 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f5840074990 con 0x7f58400d1af0 2024-04-06T12:12:55.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.395+0000 7f583cff9700 1 -- 172.21.15.189:0/685778836 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v65) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5834057780 con 0x7f58400d1af0 2024-04-06T12:12:55.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.397+0000 7f5846373700 1 -- 172.21.15.189:0/685778836 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f582c06ac90 msgr2=0x7f582c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:55.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.397+0000 7f5846373700 1 --2- 172.21.15.189:0/685778836 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f582c06ac90 0x7f582c06d150 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5830009970 tx=0x7f5830008040 comp rx=0 tx=0).stop 2024-04-06T12:12:55.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.397+0000 7f5846373700 1 -- 172.21.15.189:0/685778836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58400d1af0 msgr2=0x7f584015e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:55.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.397+0000 7f5846373700 1 --2- 172.21.15.189:0/685778836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58400d1af0 0x7f584015e710 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f583400e760 tx=0x7f583400eb20 comp rx=0 tx=0).stop 2024-04-06T12:12:55.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.397+0000 7f5846373700 1 -- 172.21.15.189:0/685778836 shutdown_connections 2024-04-06T12:12:55.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.397+0000 7f5846373700 1 --2- 172.21.15.189:0/685778836 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f582c06ac90 0x7f582c06d150 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:55.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.397+0000 7f5846373700 1 --2- 172.21.15.189:0/685778836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f584015d7e0 0x7f584014e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:55.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.397+0000 7f5846373700 1 --2- 172.21.15.189:0/685778836 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58400d0890 0x7f58401f5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:55.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.397+0000 7f5846373700 1 --2- 172.21.15.189:0/685778836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58400d1af0 0x7f584015e710 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:55.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.397+0000 7f5846373700 1 -- 172.21.15.189:0/685778836 >> 172.21.15.189:0/685778836 conn(0x7f5840072250 msgr2=0x7f58400b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:55.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.397+0000 7f5846373700 1 -- 172.21.15.189:0/685778836 shutdown_connections 2024-04-06T12:12:55.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.397+0000 7f5846373700 1 -- 172.21.15.189:0/685778836 wait complete. 2024-04-06T12:12:55.411 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-06T12:12:55.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.540+0000 7f1205afd700 1 Processor -- start 2024-04-06T12:12:55.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.540+0000 7f1205afd700 1 -- start start 2024-04-06T12:12:55.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.541+0000 7f1205afd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f120015a550 0x7f12000cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:55.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.541+0000 7f1205afd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12000cdbe0 con 0x7f12000cf900 2024-04-06T12:12:55.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.541+0000 7f1205afd700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120015a940 con 0x7f12000b4d70 2024-04-06T12:12:55.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.541+0000 7f1205afd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f120015ac00 con 0x7f120015a550 2024-04-06T12:12:55.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.541+0000 7f11feffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f12000cf900 0x7f12000b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39120/0 (socket says 172.21.15.189:39120) 2024-04-06T12:12:55.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.541+0000 7f11feffd700 1 -- 172.21.15.189:0/1347356314 learned_addr learned my addr 172.21.15.189:0/1347356314 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:55.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.541+0000 7f11ff7fe700 1 --2- 172.21.15.189:0/1347356314 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f120015a550 0x7f12000cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:55.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.541+0000 7f11fe7fc700 1 -- 172.21.15.189:0/1347356314 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774106529 0 0) 0x7f12000cdbe0 con 0x7f12000cf900 2024-04-06T12:12:55.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.541+0000 7f11fe7fc700 1 -- 172.21.15.189:0/1347356314 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11e80035b0 con 0x7f12000cf900 2024-04-06T12:12:55.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fe7fc700 1 -- 172.21.15.189:0/1347356314 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514701199 0 0) 0x7f120015a940 con 0x7f12000b4d70 2024-04-06T12:12:55.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fe7fc700 1 -- 172.21.15.189:0/1347356314 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12000cdbe0 con 0x7f12000b4d70 2024-04-06T12:12:55.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fe7fc700 1 -- 172.21.15.189:0/1347356314 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2736174622 0 0) 0x7f11e80035b0 con 0x7f12000cf900 2024-04-06T12:12:55.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fe7fc700 1 -- 172.21.15.189:0/1347356314 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f120015a550 msgr2=0x7f12000cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:55.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fe7fc700 1 --2- 172.21.15.189:0/1347356314 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f120015a550 0x7f12000cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:55.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fe7fc700 1 -- 172.21.15.189:0/1347356314 >> v1:172.21.15.192:6789/0 conn(0x7f12000b4d70 legacy=0x7f12000b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:55.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fe7fc700 1 -- 172.21.15.189:0/1347356314 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120015aee0 con 0x7f12000cf900 2024-04-06T12:12:55.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fe7fc700 1 -- 172.21.15.189:0/1347356314 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f11f0001670 con 0x7f12000cf900 2024-04-06T12:12:55.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11ff7fe700 1 --2- 172.21.15.189:0/1347356314 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f120015a550 0x7f12000cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:55.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fe7fc700 1 -- 172.21.15.189:0/1347356314 >> v1:172.21.15.189:6789/0 conn(0x7f12000cf900 legacy=0x7f12000b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:55.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fe7fc700 1 --2- 172.21.15.189:0/1347356314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11e8003bc0 0x7f11e8003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:55.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fe7fc700 1 --2- 172.21.15.189:0/1347356314 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11e8004850 0x7f11e8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:55.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fe7fc700 1 --2- 172.21.15.189:0/1347356314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11e8005190 0x7f11e800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:55.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fe7fc700 1 -- 172.21.15.189:0/1347356314 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120015ac00 con 0x7f11e8005190 2024-04-06T12:12:55.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fe7fc700 1 -- 172.21.15.189:0/1347356314 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e800bf50 con 0x7f11e8003bc0 2024-04-06T12:12:55.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fe7fc700 1 -- 172.21.15.189:0/1347356314 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e800c090 con 0x7f11e8004850 2024-04-06T12:12:55.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fffff700 1 --2- 172.21.15.189:0/1347356314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11e8005190 0x7f11e800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:55.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11ff7fe700 1 --2- 172.21.15.189:0/1347356314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11e8003bc0 0x7f11e8003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:55.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11feffd700 1 --2- 172.21.15.189:0/1347356314 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11e8004850 0x7f11e8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:55.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fffff700 1 -- 172.21.15.189:0/1347356314 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11e8004850 msgr2=0x7f11e8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:55.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fffff700 1 --2- 172.21.15.189:0/1347356314 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11e8004850 0x7f11e8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:55.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fffff700 1 -- 172.21.15.189:0/1347356314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11e8003bc0 msgr2=0x7f11e8003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:55.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fffff700 1 --2- 172.21.15.189:0/1347356314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11e8003bc0 0x7f11e8003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:55.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fffff700 1 -- 172.21.15.189:0/1347356314 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f40094a0 con 0x7f11e8005190 2024-04-06T12:12:55.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fffff700 1 --2- 172.21.15.189:0/1347356314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11e8005190 0x7f11e800ba10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f11f8002230 tx=0x7f11e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab358a73e371d06 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:55.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fe7fc700 1 -- 172.21.15.189:0/1347356314 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f800ce20 con 0x7f11e8005190 2024-04-06T12:12:55.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fe7fc700 1 -- 172.21.15.189:0/1347356314 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f8003a00 con 0x7f11e8005190 2024-04-06T12:12:55.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.542+0000 7f11fe7fc700 1 -- 172.21.15.189:0/1347356314 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f8003db0 con 0x7f11e8005190 2024-04-06T12:12:55.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.543+0000 7f1205afd700 1 -- 172.21.15.189:0/1347356314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11e8005190 msgr2=0x7f11e800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:55.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.543+0000 7f1205afd700 1 --2- 172.21.15.189:0/1347356314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11e8005190 0x7f11e800ba10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f11f8002230 tx=0x7f11e8005880 comp rx=0 tx=0).stop 2024-04-06T12:12:55.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.543+0000 7f11fffff700 1 -- 172.21.15.189:0/1347356314 reap_dead start 2024-04-06T12:12:55.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.543+0000 7f1205afd700 1 -- 172.21.15.189:0/1347356314 shutdown_connections 2024-04-06T12:12:55.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.543+0000 7f1205afd700 1 -- 172.21.15.189:0/1347356314 >> 172.21.15.189:0/1347356314 conn(0x7f12000d00b0 msgr2=0x7f12000d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:55.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.543+0000 7f1205afd700 1 -- 172.21.15.189:0/1347356314 shutdown_connections 2024-04-06T12:12:55.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.543+0000 7f1205afd700 1 -- 172.21.15.189:0/1347356314 wait complete. 2024-04-06T12:12:55.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.544+0000 7f1205afd700 1 Processor -- start 2024-04-06T12:12:55.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.544+0000 7f1205afd700 1 -- start start 2024-04-06T12:12:55.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.545+0000 7f1205afd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1200153a70 0x7f12001ef210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:55.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.545+0000 7f1205afd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1200159d80 0x7f12000a93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:55.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.545+0000 7f1205afd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1200158720 0x7f12000a98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:55.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.545+0000 7f1205afd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12001ef750 con 0x7f1200158720 2024-04-06T12:12:55.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.545+0000 7f1205afd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1200158ce0 con 0x7f1200153a70 2024-04-06T12:12:55.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.545+0000 7f1205afd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1200158fc0 con 0x7f1200159d80 2024-04-06T12:12:55.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.545+0000 7f11fffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1200158720 0x7f12000a98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:55.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.545+0000 7f11fffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1200158720 0x7f12000a98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57608/0 (socket says 172.21.15.189:57608) 2024-04-06T12:12:55.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.545+0000 7f11fffff700 1 -- 172.21.15.189:0/2615313883 learned_addr learned my addr 172.21.15.189:0/2615313883 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:55.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.545+0000 7f11feffd700 1 --2- 172.21.15.189:0/2615313883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1200159d80 0x7f12000a93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:55.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.545+0000 7f11ff7fe700 1 --2- 172.21.15.189:0/2615313883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1200153a70 0x7f12001ef210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:55.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.545+0000 7f11fffff700 1 -- 172.21.15.189:0/2615313883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1200159d80 msgr2=0x7f12000a93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:55.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.545+0000 7f11fffff700 1 --2- 172.21.15.189:0/2615313883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1200159d80 0x7f12000a93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:55.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.545+0000 7f11fffff700 1 -- 172.21.15.189:0/2615313883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1200153a70 msgr2=0x7f12001ef210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:55.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.545+0000 7f11fffff700 1 --2- 172.21.15.189:0/2615313883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1200153a70 0x7f12001ef210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:55.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.545+0000 7f11fffff700 1 -- 172.21.15.189:0/2615313883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12001592a0 con 0x7f1200158720 2024-04-06T12:12:55.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.545+0000 7f11feffd700 1 --2- 172.21.15.189:0/2615313883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1200159d80 0x7f12000a93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:55.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.546+0000 7f11fffff700 1 --2- 172.21.15.189:0/2615313883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1200158720 0x7f12000a98e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f11f8005040 tx=0x7f11f8001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:55.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.546+0000 7f11e7fff700 1 -- 172.21.15.189:0/2615313883 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f80052d0 con 0x7f1200158720 2024-04-06T12:12:55.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.546+0000 7f11e7fff700 1 -- 172.21.15.189:0/2615313883 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f8021070 con 0x7f1200158720 2024-04-06T12:12:55.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.546+0000 7f1205afd700 1 -- 172.21.15.189:0/2615313883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1200159580 con 0x7f1200158720 2024-04-06T12:12:55.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.546+0000 7f11e7fff700 1 -- 172.21.15.189:0/2615313883 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f8005690 con 0x7f1200158720 2024-04-06T12:12:55.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.546+0000 7f1205afd700 1 -- 172.21.15.189:0/2615313883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1200153e50 con 0x7f1200158720 2024-04-06T12:12:55.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.547+0000 7f1205afd700 1 -- 172.21.15.189:0/2615313883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1200154010 con 0x7f1200158720 2024-04-06T12:12:55.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.547+0000 7f11e7fff700 1 -- 172.21.15.189:0/2615313883 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f11f8023030 con 0x7f1200158720 2024-04-06T12:12:55.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.547+0000 7f11e7fff700 1 --2- 172.21.15.189:0/2615313883 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f11ec06ac90 0x7f11ec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:55.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.548+0000 7f11e7fff700 1 -- 172.21.15.189:0/2615313883 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f11f808c5f0 con 0x7f1200158720 2024-04-06T12:12:55.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.548+0000 7f11ff7fe700 1 --2- 172.21.15.189:0/2615313883 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f11ec06ac90 0x7f11ec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:55.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.548+0000 7f11ff7fe700 1 --2- 172.21.15.189:0/2615313883 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f11ec06ac90 0x7f11ec06d150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009de0 tx=0x7f11f4009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:55.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.552+0000 7f11e7fff700 1 -- 172.21.15.189:0/2615313883 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f11f80574a0 con 0x7f1200158720 2024-04-06T12:12:55.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.749+0000 7f1205afd700 1 -- 172.21.15.189:0/2615313883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1200074990 con 0x7f1200158720 2024-04-06T12:12:55.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.755+0000 7f11e7fff700 1 -- 172.21.15.189:0/2615313883 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f11f80011c0 con 0x7f1200158720 2024-04-06T12:12:55.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.757+0000 7f1205afd700 1 -- 172.21.15.189:0/2615313883 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f11ec06ac90 msgr2=0x7f11ec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:55.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.757+0000 7f1205afd700 1 --2- 172.21.15.189:0/2615313883 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f11ec06ac90 0x7f11ec06d150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009de0 tx=0x7f11f4009210 comp rx=0 tx=0).stop 2024-04-06T12:12:55.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.757+0000 7f1205afd700 1 -- 172.21.15.189:0/2615313883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1200158720 msgr2=0x7f12000a98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:55.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.757+0000 7f1205afd700 1 --2- 172.21.15.189:0/2615313883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1200158720 0x7f12000a98e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f11f8005040 tx=0x7f11f8001650 comp rx=0 tx=0).stop 2024-04-06T12:12:55.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.757+0000 7f1205afd700 1 -- 172.21.15.189:0/2615313883 shutdown_connections 2024-04-06T12:12:55.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.757+0000 7f1205afd700 1 --2- 172.21.15.189:0/2615313883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1200158720 0x7f12000a98e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:55.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.757+0000 7f1205afd700 1 --2- 172.21.15.189:0/2615313883 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f11ec06ac90 0x7f11ec06d150 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:55.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.757+0000 7f1205afd700 1 --2- 172.21.15.189:0/2615313883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1200153a70 0x7f12001ef210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:55.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.757+0000 7f1205afd700 1 --2- 172.21.15.189:0/2615313883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1200159d80 0x7f12000a93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:55.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.757+0000 7f1205afd700 1 -- 172.21.15.189:0/2615313883 >> 172.21.15.189:0/2615313883 conn(0x7f12000d00b0 msgr2=0x7f1200154b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:55.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.758+0000 7f1205afd700 1 -- 172.21.15.189:0/2615313883 shutdown_connections 2024-04-06T12:12:55.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:55.758+0000 7f1205afd700 1 -- 172.21.15.189:0/2615313883 wait complete. 2024-04-06T12:12:55.769 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T12:12:55.769 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:12:55.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:12:55.859 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T12:12:55.859 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:12:55.859 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:12:55.877 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:12:55.877 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:12:55.934 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:12:55.934 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:12:55.992 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:56.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.154+0000 7f0ea7325700 1 Processor -- start 2024-04-06T12:12:56.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.154+0000 7f0ea7325700 1 -- start start 2024-04-06T12:12:56.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.155+0000 7f0ea7325700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0ea015a550 0x7f0ea00cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:56.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.155+0000 7f0ea7325700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea00cf900 0x7f0ea00b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:56.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.155+0000 7f0ea7325700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea00cdbe0 con 0x7f0ea00b4ba0 2024-04-06T12:12:56.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.155+0000 7f0ea7325700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea00b5010 con 0x7f0ea015a550 2024-04-06T12:12:56.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.155+0000 7f0ea7325700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea015a940 con 0x7f0ea00cf900 2024-04-06T12:12:56.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.155+0000 7f0ea50c1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0ea015a550 0x7f0ea00cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:56.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.155+0000 7f0ea58c2700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0ea00b4ba0 0x7f0ea006eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39124/0 (socket says 172.21.15.189:39124) 2024-04-06T12:12:56.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.155+0000 7f0ea58c2700 1 -- 172.21.15.189:0/2722403086 learned_addr learned my addr 172.21.15.189:0/2722403086 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:56.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.155+0000 7f0ea48c0700 1 --2- 172.21.15.189:0/2722403086 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea00cf900 0x7f0ea00b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:56.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.155+0000 7f0ea50c1700 1 -- 172.21.15.189:0/2722403086 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea00cf900 msgr2=0x7f0ea00b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:56.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.155+0000 7f0ea50c1700 1 --2- 172.21.15.189:0/2722403086 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea00cf900 0x7f0ea00b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.155+0000 7f0ea50c1700 1 -- 172.21.15.189:0/2722403086 >> v1:172.21.15.189:6789/0 conn(0x7f0ea00b4ba0 legacy=0x7f0ea006eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:56.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.155+0000 7f0ea50c1700 1 -- 172.21.15.189:0/2722403086 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea015ac00 con 0x7f0ea015a550 2024-04-06T12:12:56.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.155+0000 7f0e97fff700 1 -- 172.21.15.189:0/2722403086 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923453310 0 0) 0x7f0ea00cdbe0 con 0x7f0ea00b4ba0 2024-04-06T12:12:56.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.155+0000 7f0ea50c1700 1 --2- 172.21.15.189:0/2722403086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ea015a550 0x7f0ea00cf3c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c009840 tx=0x7f0e9c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2fcb3584ced635 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:56.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.156+0000 7f0e97fff700 1 -- 172.21.15.189:0/2722403086 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c011830 con 0x7f0ea015a550 2024-04-06T12:12:56.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.156+0000 7f0e97fff700 1 -- 172.21.15.189:0/2722403086 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9c011990 con 0x7f0ea015a550 2024-04-06T12:12:56.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.156+0000 7f0e97fff700 1 -- 172.21.15.189:0/2722403086 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c011d20 con 0x7f0ea015a550 2024-04-06T12:12:56.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.156+0000 7f0ea7325700 1 -- 172.21.15.189:0/2722403086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ea015a550 msgr2=0x7f0ea00cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:56.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.156+0000 7f0ea7325700 1 --2- 172.21.15.189:0/2722403086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ea015a550 0x7f0ea00cf3c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c009840 tx=0x7f0e9c009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:56.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.156+0000 7f0ea7325700 1 -- 172.21.15.189:0/2722403086 shutdown_connections 2024-04-06T12:12:56.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.156+0000 7f0ea7325700 1 --2- 172.21.15.189:0/2722403086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ea015a550 0x7f0ea00cf3c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.156+0000 7f0ea7325700 1 --2- 172.21.15.189:0/2722403086 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea00cf900 0x7f0ea00b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.156+0000 7f0ea7325700 1 -- 172.21.15.189:0/2722403086 >> 172.21.15.189:0/2722403086 conn(0x7f0ea00d00b0 msgr2=0x7f0ea00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:56.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.156+0000 7f0ea7325700 1 -- 172.21.15.189:0/2722403086 shutdown_connections 2024-04-06T12:12:56.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.156+0000 7f0ea7325700 1 -- 172.21.15.189:0/2722403086 wait complete. 2024-04-06T12:12:56.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.158+0000 7f0ea7325700 1 Processor -- start 2024-04-06T12:12:56.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.158+0000 7f0ea7325700 1 -- start start 2024-04-06T12:12:56.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0ea7325700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ea00b4ba0 0x7f0ea0146980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:56.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0ea7325700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea00cf900 0x7f0ea0147160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:56.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0ea7325700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ea01476a0 0x7f0ea00b0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:56.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0ea7325700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea0146ec0 con 0x7f0ea01476a0 2024-04-06T12:12:56.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0ea7325700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea01487c0 con 0x7f0ea00b4ba0 2024-04-06T12:12:56.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0ea7325700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea0147bb0 con 0x7f0ea00cf900 2024-04-06T12:12:56.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0ea58c2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ea01476a0 0x7f0ea00b0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:56.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0ea58c2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ea01476a0 0x7f0ea00b0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57622/0 (socket says 172.21.15.189:57622) 2024-04-06T12:12:56.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0ea58c2700 1 -- 172.21.15.189:0/1401330787 learned_addr learned my addr 172.21.15.189:0/1401330787 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:56.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0ea48c0700 1 --2- 172.21.15.189:0/1401330787 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea00cf900 0x7f0ea0147160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:56.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0ea50c1700 1 --2- 172.21.15.189:0/1401330787 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ea00b4ba0 0x7f0ea0146980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:56.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0ea58c2700 1 -- 172.21.15.189:0/1401330787 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea00cf900 msgr2=0x7f0ea0147160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:56.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0ea58c2700 1 --2- 172.21.15.189:0/1401330787 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea00cf900 0x7f0ea0147160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0ea58c2700 1 -- 172.21.15.189:0/1401330787 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ea00b4ba0 msgr2=0x7f0ea0146980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:56.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0ea58c2700 1 --2- 172.21.15.189:0/1401330787 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ea00b4ba0 0x7f0ea0146980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0ea58c2700 1 -- 172.21.15.189:0/1401330787 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9c0094a0 con 0x7f0ea01476a0 2024-04-06T12:12:56.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0ea50c1700 1 --2- 172.21.15.189:0/1401330787 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ea00b4ba0 0x7f0ea0146980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:56.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0ea58c2700 1 --2- 172.21.15.189:0/1401330787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ea01476a0 0x7f0ea00b0e00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0e9800cb60 tx=0x7f0e9800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:56.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0e95ffb700 1 -- 172.21.15.189:0/1401330787 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9800ab50 con 0x7f0ea01476a0 2024-04-06T12:12:56.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0e95ffb700 1 -- 172.21.15.189:0/1401330787 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9800acb0 con 0x7f0ea01476a0 2024-04-06T12:12:56.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0e95ffb700 1 -- 172.21.15.189:0/1401330787 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e98007310 con 0x7f0ea01476a0 2024-04-06T12:12:56.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.159+0000 7f0ea7325700 1 -- 172.21.15.189:0/1401330787 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea0147e90 con 0x7f0ea01476a0 2024-04-06T12:12:56.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.160+0000 7f0ea7325700 1 -- 172.21.15.189:0/1401330787 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea0148450 con 0x7f0ea01476a0 2024-04-06T12:12:56.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.161+0000 7f0ea7325700 1 -- 172.21.15.189:0/1401330787 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea0067bb0 con 0x7f0ea01476a0 2024-04-06T12:12:56.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.161+0000 7f0e95ffb700 1 -- 172.21.15.189:0/1401330787 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0e98007470 con 0x7f0ea01476a0 2024-04-06T12:12:56.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.161+0000 7f0e95ffb700 1 --2- 172.21.15.189:0/1401330787 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0e8406ac90 0x7f0e8406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:56.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.161+0000 7f0e95ffb700 1 -- 172.21.15.189:0/1401330787 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0e980891d0 con 0x7f0ea01476a0 2024-04-06T12:12:56.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.161+0000 7f0ea50c1700 1 --2- 172.21.15.189:0/1401330787 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0e8406ac90 0x7f0e8406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:56.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.162+0000 7f0ea50c1700 1 --2- 172.21.15.189:0/1401330787 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0e8406ac90 0x7f0e8406d150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0090b0 tx=0x7f0e9c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:56.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.166+0000 7f0e95ffb700 1 -- 172.21.15.189:0/1401330787 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0e98053ff0 con 0x7f0ea01476a0 2024-04-06T12:12:56.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.367+0000 7f0ea7325700 1 -- 172.21.15.189:0/1401330787 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ea0074990 con 0x7f0ea01476a0 2024-04-06T12:12:56.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.368+0000 7f0e95ffb700 1 -- 172.21.15.189:0/1401330787 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18314 (secure 0 0 0) 0x7f0e98057610 con 0x7f0ea01476a0 2024-04-06T12:12:56.367 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:56.367 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":26,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:54.376742+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:56.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.370+0000 7f0ea7325700 1 -- 172.21.15.189:0/1401330787 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0e8406ac90 msgr2=0x7f0e8406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:56.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.370+0000 7f0ea7325700 1 --2- 172.21.15.189:0/1401330787 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0e8406ac90 0x7f0e8406d150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0090b0 tx=0x7f0e9c008030 comp rx=0 tx=0).stop 2024-04-06T12:12:56.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.370+0000 7f0ea7325700 1 -- 172.21.15.189:0/1401330787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ea01476a0 msgr2=0x7f0ea00b0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:56.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.370+0000 7f0ea7325700 1 --2- 172.21.15.189:0/1401330787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ea01476a0 0x7f0ea00b0e00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0e9800cb60 tx=0x7f0e9800cf20 comp rx=0 tx=0).stop 2024-04-06T12:12:56.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.370+0000 7f0ea7325700 1 -- 172.21.15.189:0/1401330787 shutdown_connections 2024-04-06T12:12:56.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.370+0000 7f0ea7325700 1 --2- 172.21.15.189:0/1401330787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ea01476a0 0x7f0ea00b0e00 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.370+0000 7f0ea7325700 1 --2- 172.21.15.189:0/1401330787 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0e8406ac90 0x7f0e8406d150 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.370+0000 7f0ea7325700 1 --2- 172.21.15.189:0/1401330787 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ea00b4ba0 0x7f0ea0146980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.370+0000 7f0ea7325700 1 --2- 172.21.15.189:0/1401330787 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea00cf900 0x7f0ea0147160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.370+0000 7f0ea7325700 1 -- 172.21.15.189:0/1401330787 >> 172.21.15.189:0/1401330787 conn(0x7f0ea00d00b0 msgr2=0x7f0ea00b3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:56.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.370+0000 7f0ea7325700 1 -- 172.21.15.189:0/1401330787 shutdown_connections 2024-04-06T12:12:56.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.370+0000 7f0ea7325700 1 -- 172.21.15.189:0/1401330787 wait complete. 2024-04-06T12:12:56.382 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:12:02.280866+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:12:03.156593+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-06T12:12:06.237184+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-06T12:12:56.382 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:12:56.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.508+0000 7f0c117cb700 1 Processor -- start 2024-04-06T12:12:56.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.509+0000 7f0c117cb700 1 -- start start 2024-04-06T12:12:56.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.509+0000 7f0c117cb700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0c0c15d7b0 0x7f0c0c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:56.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.509+0000 7f0c117cb700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c0c06a730 0x7f0c0c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:56.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.509+0000 7f0c117cb700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0c0b3990 con 0x7f0c0c15ddf0 2024-04-06T12:12:56.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.509+0000 7f0c117cb700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c0c06b0f0 con 0x7f0c0c15d7b0 2024-04-06T12:12:56.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.509+0000 7f0c117cb700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c0b3060 con 0x7f0c0c06a730 2024-04-06T12:12:56.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.510+0000 7f0c0affd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0c0c15d7b0 0x7f0c0c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:56.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.510+0000 7f0c0a7fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c0c06a730 0x7f0c0c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:56.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.510+0000 7f0c0a7fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c0c06a730 0x7f0c0c06abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:60532/0 (socket says 172.21.15.189:60532) 2024-04-06T12:12:56.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.510+0000 7f0c0affd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0c0c15d7b0 0x7f0c0c0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:56440/0 (socket says 172.21.15.189:56440) 2024-04-06T12:12:56.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.510+0000 7f0c0a7fc700 1 -- 172.21.15.189:0/3201046205 learned_addr learned my addr 172.21.15.189:0/3201046205 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:56.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.510+0000 7f0c0b7fe700 1 --1- 172.21.15.189:0/3201046205 >> v1:172.21.15.192:6789/0 conn(0x7f0c0c15ddf0 0x7f0c0c15e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:51356/0 (socket says 172.21.15.189:51356) 2024-04-06T12:12:56.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.510+0000 7f0c0a7fc700 1 -- 172.21.15.189:0/3201046205 >> v2:172.21.15.192:3300/0 conn(0x7f0c0c15d7b0 msgr2=0x7f0c0c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:56.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.510+0000 7f0c0a7fc700 1 --2- 172.21.15.189:0/3201046205 >> v2:172.21.15.192:3300/0 conn(0x7f0c0c15d7b0 0x7f0c0c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.510+0000 7f0c0a7fc700 1 -- 172.21.15.189:0/3201046205 >> v1:172.21.15.192:6789/0 conn(0x7f0c0c15ddf0 legacy=0x7f0c0c15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:56.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.510+0000 7f0c0a7fc700 1 -- 172.21.15.189:0/3201046205 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c0b3340 con 0x7f0c0c06a730 2024-04-06T12:12:56.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.510+0000 7f0c0affd700 1 --2- 172.21.15.189:0/3201046205 >> v2:172.21.15.192:3300/0 conn(0x7f0c0c15d7b0 0x7f0c0c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:12:56.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.510+0000 7f0c0a7fc700 1 --2- 172.21.15.189:0/3201046205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c0c06a730 0x7f0c0c06abb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc0098f0 tx=0x7f0bfc009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95bb22300a98d701 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:56.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.510+0000 7f0c09ffb700 1 -- 172.21.15.189:0/3201046205 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc011830 con 0x7f0c0c06a730 2024-04-06T12:12:56.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.510+0000 7f0c09ffb700 1 -- 172.21.15.189:0/3201046205 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc011990 con 0x7f0c0c06a730 2024-04-06T12:12:56.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.510+0000 7f0c09ffb700 1 -- 172.21.15.189:0/3201046205 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc011d20 con 0x7f0c0c06a730 2024-04-06T12:12:56.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.511+0000 7f0c117cb700 1 -- 172.21.15.189:0/3201046205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c0c06a730 msgr2=0x7f0c0c06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:56.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.511+0000 7f0c117cb700 1 --2- 172.21.15.189:0/3201046205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c0c06a730 0x7f0c0c06abb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc0098f0 tx=0x7f0bfc009c00 comp rx=0 tx=0).stop 2024-04-06T12:12:56.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.511+0000 7f0c117cb700 1 -- 172.21.15.189:0/3201046205 shutdown_connections 2024-04-06T12:12:56.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.511+0000 7f0c117cb700 1 --2- 172.21.15.189:0/3201046205 >> v2:172.21.15.192:3300/0 conn(0x7f0c0c15d7b0 0x7f0c0c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.511+0000 7f0c117cb700 1 --2- 172.21.15.189:0/3201046205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c0c06a730 0x7f0c0c06abb0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.511+0000 7f0c117cb700 1 -- 172.21.15.189:0/3201046205 >> 172.21.15.189:0/3201046205 conn(0x7f0c0c0d00b0 msgr2=0x7f0c0c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:56.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.511+0000 7f0c117cb700 1 -- 172.21.15.189:0/3201046205 shutdown_connections 2024-04-06T12:12:56.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.511+0000 7f0c117cb700 1 -- 172.21.15.189:0/3201046205 wait complete. 2024-04-06T12:12:56.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.512+0000 7f0c117cb700 1 Processor -- start 2024-04-06T12:12:56.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.512+0000 7f0c117cb700 1 -- start start 2024-04-06T12:12:56.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.513+0000 7f0c117cb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c0c06a730 0x7f0c0c15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:56.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.513+0000 7f0c117cb700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c0c15d7b0 0x7f0c0c15ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:56.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.513+0000 7f0c117cb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c0c15ddf0 0x7f0c0c1550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:56.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.513+0000 7f0c117cb700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c15b290 con 0x7f0c0c06a730 2024-04-06T12:12:56.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.513+0000 7f0c117cb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c15cfc0 con 0x7f0c0c15ddf0 2024-04-06T12:12:56.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.513+0000 7f0c117cb700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c15b480 con 0x7f0c0c15d7b0 2024-04-06T12:12:56.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.513+0000 7f0c0affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c0c06a730 0x7f0c0c15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:56.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.513+0000 7f0c0affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c0c06a730 0x7f0c0c15ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57630/0 (socket says 172.21.15.189:57630) 2024-04-06T12:12:56.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.513+0000 7f0c0affd700 1 -- 172.21.15.189:0/1076247486 learned_addr learned my addr 172.21.15.189:0/1076247486 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:56.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.513+0000 7f0c0b7fe700 1 --2- 172.21.15.189:0/1076247486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c0c15ddf0 0x7f0c0c1550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:56.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.513+0000 7f0c0a7fc700 1 --2- 172.21.15.189:0/1076247486 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c0c15d7b0 0x7f0c0c15ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:56.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.513+0000 7f0c0affd700 1 -- 172.21.15.189:0/1076247486 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c0c15d7b0 msgr2=0x7f0c0c15ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:56.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.514+0000 7f0c0affd700 1 --2- 172.21.15.189:0/1076247486 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c0c15d7b0 0x7f0c0c15ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.514+0000 7f0c0affd700 1 -- 172.21.15.189:0/1076247486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c0c15ddf0 msgr2=0x7f0c0c1550e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:56.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.514+0000 7f0c0affd700 1 --2- 172.21.15.189:0/1076247486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c0c15ddf0 0x7f0c0c1550e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.514+0000 7f0c0affd700 1 -- 172.21.15.189:0/1076247486 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c000094a0 con 0x7f0c0c06a730 2024-04-06T12:12:56.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.514+0000 7f0c0b7fe700 1 --2- 172.21.15.189:0/1076247486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c0c15ddf0 0x7f0c0c1550e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:56.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.514+0000 7f0c0affd700 1 --2- 172.21.15.189:0/1076247486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c0c06a730 0x7f0c0c15ad50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0c0000ebe0 tx=0x7f0c00009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:56.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.514+0000 7f0bf37fe700 1 -- 172.21.15.189:0/1076247486 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0000c910 con 0x7f0c0c06a730 2024-04-06T12:12:56.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.514+0000 7f0bf37fe700 1 -- 172.21.15.189:0/1076247486 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0000ca70 con 0x7f0c0c06a730 2024-04-06T12:12:56.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.514+0000 7f0c117cb700 1 -- 172.21.15.189:0/1076247486 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bfc009550 con 0x7f0c0c06a730 2024-04-06T12:12:56.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.514+0000 7f0bf37fe700 1 -- 172.21.15.189:0/1076247486 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0000cde0 con 0x7f0c0c06a730 2024-04-06T12:12:56.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.514+0000 7f0c117cb700 1 -- 172.21.15.189:0/1076247486 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c0c15b810 con 0x7f0c0c06a730 2024-04-06T12:12:56.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.515+0000 7f0c117cb700 1 -- 172.21.15.189:0/1076247486 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0c067bb0 con 0x7f0c0c06a730 2024-04-06T12:12:56.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.515+0000 7f0bf37fe700 1 -- 172.21.15.189:0/1076247486 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0c00013470 con 0x7f0c0c06a730 2024-04-06T12:12:56.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.516+0000 7f0bf37fe700 1 --2- 172.21.15.189:0/1076247486 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0bf806ac90 0x7f0bf806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:56.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.516+0000 7f0c0a7fc700 1 --2- 172.21.15.189:0/1076247486 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0bf806ac90 0x7f0bf806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:56.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.516+0000 7f0bf37fe700 1 -- 172.21.15.189:0/1076247486 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0c000898c0 con 0x7f0c0c06a730 2024-04-06T12:12:56.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.516+0000 7f0c0a7fc700 1 --2- 172.21.15.189:0/1076247486 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0bf806ac90 0x7f0bf806d150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c15f6b0 tx=0x7f0bfc008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:56.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.520+0000 7f0bf37fe700 1 -- 172.21.15.189:0/1076247486 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0c000546e0 con 0x7f0c0c06a730 2024-04-06T12:12:56.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.714+0000 7f0c117cb700 1 -- 172.21.15.189:0/1076247486 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0c0c074990 con 0x7f0c0c06a730 2024-04-06T12:12:56.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.714+0000 7f0bf37fe700 1 -- 172.21.15.189:0/1076247486 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0c00057d00 con 0x7f0c0c06a730 2024-04-06T12:12:56.713 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:12:56.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.716+0000 7f0c117cb700 1 -- 172.21.15.189:0/1076247486 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0bf806ac90 msgr2=0x7f0bf806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:56.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.716+0000 7f0c117cb700 1 --2- 172.21.15.189:0/1076247486 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0bf806ac90 0x7f0bf806d150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c15f6b0 tx=0x7f0bfc008030 comp rx=0 tx=0).stop 2024-04-06T12:12:56.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.716+0000 7f0c117cb700 1 -- 172.21.15.189:0/1076247486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c0c06a730 msgr2=0x7f0c0c15ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:56.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.716+0000 7f0c117cb700 1 --2- 172.21.15.189:0/1076247486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c0c06a730 0x7f0c0c15ad50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0c0000ebe0 tx=0x7f0c00009ec0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.716+0000 7f0c117cb700 1 -- 172.21.15.189:0/1076247486 shutdown_connections 2024-04-06T12:12:56.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.716+0000 7f0c117cb700 1 --2- 172.21.15.189:0/1076247486 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0bf806ac90 0x7f0bf806d150 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.716+0000 7f0c117cb700 1 --2- 172.21.15.189:0/1076247486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c0c15ddf0 0x7f0c0c1550e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.716+0000 7f0c117cb700 1 --2- 172.21.15.189:0/1076247486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c0c06a730 0x7f0c0c15ad50 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.716+0000 7f0c117cb700 1 --2- 172.21.15.189:0/1076247486 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c0c15d7b0 0x7f0c0c15ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.716+0000 7f0c117cb700 1 -- 172.21.15.189:0/1076247486 >> 172.21.15.189:0/1076247486 conn(0x7f0c0c0d00b0 msgr2=0x7f0c0c071c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:56.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.716+0000 7f0c117cb700 1 -- 172.21.15.189:0/1076247486 shutdown_connections 2024-04-06T12:12:56.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.716+0000 7f0c117cb700 1 -- 172.21.15.189:0/1076247486 wait complete. 2024-04-06T12:12:56.728 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:12:56.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.854+0000 7f06c7183700 1 Processor -- start 2024-04-06T12:12:56.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.854+0000 7f06c7183700 1 -- start start 2024-04-06T12:12:56.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.854+0000 7f06c7183700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f06c00b4af0 0x7f06c00b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:56.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.855+0000 7f06c7183700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c00cdbe0 con 0x7f06c015a550 2024-04-06T12:12:56.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.855+0000 7f06c7183700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c015a940 con 0x7f06c00b4660 2024-04-06T12:12:56.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.855+0000 7f06c7183700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c015ac00 con 0x7f06c00b4af0 2024-04-06T12:12:56.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.855+0000 7f06c4f1f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f06c015a550 0x7f06c00cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39140/0 (socket says 172.21.15.189:39140) 2024-04-06T12:12:56.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.855+0000 7f06c4f1f700 1 -- 172.21.15.189:0/3184833217 learned_addr learned my addr 172.21.15.189:0/3184833217 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:56.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.855+0000 7f06c5720700 1 --2- 172.21.15.189:0/3184833217 >> v2:172.21.15.192:3300/0 conn(0x7f06c00b4af0 0x7f06c00b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:56.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.855+0000 7f06bf7fe700 1 -- 172.21.15.189:0/3184833217 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828162073 0 0) 0x7f06c00cdbe0 con 0x7f06c015a550 2024-04-06T12:12:56.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.855+0000 7f06bf7fe700 1 -- 172.21.15.189:0/3184833217 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06b00035b0 con 0x7f06c015a550 2024-04-06T12:12:56.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.855+0000 7f06c5720700 1 -- 172.21.15.189:0/3184833217 >> v1:172.21.15.192:6789/0 conn(0x7f06c00b4660 legacy=0x7f06c00b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:56.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.855+0000 7f06c5720700 1 -- 172.21.15.189:0/3184833217 >> v1:172.21.15.189:6789/0 conn(0x7f06c015a550 legacy=0x7f06c00cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:56.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.855+0000 7f06c5720700 1 -- 172.21.15.189:0/3184833217 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c015aee0 con 0x7f06c00b4af0 2024-04-06T12:12:56.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.855+0000 7f06c5720700 1 --2- 172.21.15.189:0/3184833217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06c00b4af0 0x7f06c00b36d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f06b8009d90 tx=0x7f06b800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11e7d728501e3f3a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:56.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.855+0000 7f06bf7fe700 1 -- 172.21.15.189:0/3184833217 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b800e070 con 0x7f06c00b4af0 2024-04-06T12:12:56.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.856+0000 7f06bf7fe700 1 -- 172.21.15.189:0/3184833217 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b800bb20 con 0x7f06c00b4af0 2024-04-06T12:12:56.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.856+0000 7f06bf7fe700 1 -- 172.21.15.189:0/3184833217 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b8012430 con 0x7f06c00b4af0 2024-04-06T12:12:56.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.856+0000 7f06c7183700 1 -- 172.21.15.189:0/3184833217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06c00b4af0 msgr2=0x7f06c00b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:56.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.856+0000 7f06c7183700 1 --2- 172.21.15.189:0/3184833217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06c00b4af0 0x7f06c00b36d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f06b8009d90 tx=0x7f06b800b330 comp rx=0 tx=0).stop 2024-04-06T12:12:56.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.856+0000 7f06c7183700 1 -- 172.21.15.189:0/3184833217 shutdown_connections 2024-04-06T12:12:56.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.856+0000 7f06c7183700 1 --2- 172.21.15.189:0/3184833217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06c00b4af0 0x7f06c00b36d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.856+0000 7f06c7183700 1 -- 172.21.15.189:0/3184833217 >> 172.21.15.189:0/3184833217 conn(0x7f06c00d00b0 msgr2=0x7f06c00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:56.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.856+0000 7f06c7183700 1 -- 172.21.15.189:0/3184833217 shutdown_connections 2024-04-06T12:12:56.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.856+0000 7f06c7183700 1 -- 172.21.15.189:0/3184833217 wait complete. 2024-04-06T12:12:56.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.857+0000 7f06c7183700 1 Processor -- start 2024-04-06T12:12:56.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.857+0000 7f06c7183700 1 -- start start 2024-04-06T12:12:56.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.858+0000 7f06c7183700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c00b4660 0x7f06c014bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:56.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.858+0000 7f06c7183700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06c00b4af0 0x7f06c00ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:56.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.858+0000 7f06c7183700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06c01535f0 0x7f06c00ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:56.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.858+0000 7f06c7183700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c01561f0 con 0x7f06c00b4660 2024-04-06T12:12:56.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.858+0000 7f06c7183700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c014c120 con 0x7f06c00b4af0 2024-04-06T12:12:56.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.858+0000 7f06c7183700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c00ad230 con 0x7f06c01535f0 2024-04-06T12:12:56.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.858+0000 7f06c4f1f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c00b4660 0x7f06c014bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:56.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.858+0000 7f06c4f1f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c00b4660 0x7f06c014bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57632/0 (socket says 172.21.15.189:57632) 2024-04-06T12:12:56.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.858+0000 7f06c4f1f700 1 -- 172.21.15.189:0/3869579161 learned_addr learned my addr 172.21.15.189:0/3869579161 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:56.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.858+0000 7f06c5720700 1 --2- 172.21.15.189:0/3869579161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06c01535f0 0x7f06c00ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:56.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.858+0000 7f06bffff700 1 --2- 172.21.15.189:0/3869579161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06c00b4af0 0x7f06c00ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:56.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.859+0000 7f06c4f1f700 1 -- 172.21.15.189:0/3869579161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06c01535f0 msgr2=0x7f06c00ae930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:56.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.859+0000 7f06c4f1f700 1 --2- 172.21.15.189:0/3869579161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06c01535f0 0x7f06c00ae930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.859+0000 7f06c4f1f700 1 -- 172.21.15.189:0/3869579161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06c00b4af0 msgr2=0x7f06c00ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:56.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.859+0000 7f06c4f1f700 1 --2- 172.21.15.189:0/3869579161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06c00b4af0 0x7f06c00ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:56.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.859+0000 7f06c4f1f700 1 -- 172.21.15.189:0/3869579161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b8009a70 con 0x7f06c00b4660 2024-04-06T12:12:56.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.859+0000 7f06c5720700 1 --2- 172.21.15.189:0/3869579161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06c01535f0 0x7f06c00ae930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:56.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.859+0000 7f06c4f1f700 1 --2- 172.21.15.189:0/3869579161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c00b4660 0x7f06c014bbe0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f06b400c4d0 tx=0x7f06b400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:56.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.859+0000 7f06bd7fa700 1 -- 172.21.15.189:0/3869579161 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b4005d70 con 0x7f06c00b4660 2024-04-06T12:12:56.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.859+0000 7f06bd7fa700 1 -- 172.21.15.189:0/3869579161 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b40070d0 con 0x7f06c00b4660 2024-04-06T12:12:56.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.859+0000 7f06c7183700 1 -- 172.21.15.189:0/3869579161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c00ad510 con 0x7f06c00b4660 2024-04-06T12:12:56.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.859+0000 7f06bd7fa700 1 -- 172.21.15.189:0/3869579161 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b4007460 con 0x7f06c00b4660 2024-04-06T12:12:56.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.859+0000 7f06c7183700 1 -- 172.21.15.189:0/3869579161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c00adad0 con 0x7f06c00b4660 2024-04-06T12:12:56.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.860+0000 7f06c7183700 1 -- 172.21.15.189:0/3869579161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c0067bb0 con 0x7f06c00b4660 2024-04-06T12:12:56.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.860+0000 7f06bd7fa700 1 -- 172.21.15.189:0/3869579161 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f06b40075c0 con 0x7f06c00b4660 2024-04-06T12:12:56.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.861+0000 7f06bd7fa700 1 --2- 172.21.15.189:0/3869579161 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06a406ac90 0x7f06a406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:56.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.861+0000 7f06bd7fa700 1 -- 172.21.15.189:0/3869579161 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f06b4088b70 con 0x7f06c00b4660 2024-04-06T12:12:56.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.861+0000 7f06bffff700 1 --2- 172.21.15.189:0/3869579161 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06a406ac90 0x7f06a406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:56.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.861+0000 7f06bffff700 1 --2- 172.21.15.189:0/3869579161 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06a406ac90 0x7f06a406d150 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f06c01f6210 tx=0x7f06ac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:56.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:56.865+0000 7f06bd7fa700 1 -- 172.21.15.189:0/3869579161 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f06b4053a20 con 0x7f06c00b4660 2024-04-06T12:12:57.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.060+0000 7f06c7183700 1 -- 172.21.15.189:0/3869579161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f06c0074990 con 0x7f06c00b4660 2024-04-06T12:12:57.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.060+0000 7f06bd7fa700 1 -- 172.21.15.189:0/3869579161 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f06b4057040 con 0x7f06c00b4660 2024-04-06T12:12:57.059 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:12:57.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.062+0000 7f06c7183700 1 -- 172.21.15.189:0/3869579161 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06a406ac90 msgr2=0x7f06a406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:57.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.062+0000 7f06c7183700 1 --2- 172.21.15.189:0/3869579161 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06a406ac90 0x7f06a406d150 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f06c01f6210 tx=0x7f06ac009040 comp rx=0 tx=0).stop 2024-04-06T12:12:57.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.062+0000 7f06c7183700 1 -- 172.21.15.189:0/3869579161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c00b4660 msgr2=0x7f06c014bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:57.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.062+0000 7f06c7183700 1 --2- 172.21.15.189:0/3869579161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c00b4660 0x7f06c014bbe0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f06b400c4d0 tx=0x7f06b400c890 comp rx=0 tx=0).stop 2024-04-06T12:12:57.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.062+0000 7f06c7183700 1 -- 172.21.15.189:0/3869579161 shutdown_connections 2024-04-06T12:12:57.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.062+0000 7f06c7183700 1 --2- 172.21.15.189:0/3869579161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06c01535f0 0x7f06c00ae930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.062+0000 7f06c7183700 1 --2- 172.21.15.189:0/3869579161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c00b4660 0x7f06c014bbe0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.062+0000 7f06c7183700 1 --2- 172.21.15.189:0/3869579161 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06a406ac90 0x7f06a406d150 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.062+0000 7f06c7183700 1 --2- 172.21.15.189:0/3869579161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06c00b4af0 0x7f06c00ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.062+0000 7f06c7183700 1 -- 172.21.15.189:0/3869579161 >> 172.21.15.189:0/3869579161 conn(0x7f06c00d00b0 msgr2=0x7f06c0158b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:57.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.062+0000 7f06c7183700 1 -- 172.21.15.189:0/3869579161 shutdown_connections 2024-04-06T12:12:57.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.062+0000 7f06c7183700 1 -- 172.21.15.189:0/3869579161 wait complete. 2024-04-06T12:12:57.073 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:12:57.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.199+0000 7fd2a4e41700 1 Processor -- start 2024-04-06T12:12:57.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.199+0000 7fd2a4e41700 1 -- start start 2024-04-06T12:12:57.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.200+0000 7fd2a4e41700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd2a015a550 0x7fd2a00cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.200+0000 7fd2a4e41700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2a00b4d70 0x7fd2a00b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.200+0000 7fd2a4e41700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a00cdbe0 con 0x7fd2a00cf900 2024-04-06T12:12:57.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.200+0000 7fd2a4e41700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2a015a940 con 0x7fd2a015a550 2024-04-06T12:12:57.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.200+0000 7fd2a4e41700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a015ac00 con 0x7fd2a00b4d70 2024-04-06T12:12:57.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.200+0000 7fd29dd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd2a00cf900 0x7fd2a00b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39142/0 (socket says 172.21.15.189:39142) 2024-04-06T12:12:57.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.200+0000 7fd29dd9b700 1 -- 172.21.15.189:0/4294134495 learned_addr learned my addr 172.21.15.189:0/4294134495 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:57.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29ed9d700 1 --2- 172.21.15.189:0/4294134495 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2a00b4d70 0x7fd2a00b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29e59c700 1 --2- 172.21.15.189:0/4294134495 >> v2:172.21.15.192:3300/0 conn(0x7fd2a015a550 0x7fd2a00cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29d59a700 1 -- 172.21.15.189:0/4294134495 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015263502 0 0) 0x7fd2a00cdbe0 con 0x7fd2a00cf900 2024-04-06T12:12:57.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29d59a700 1 -- 172.21.15.189:0/4294134495 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2880035b0 con 0x7fd2a00cf900 2024-04-06T12:12:57.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29d59a700 1 -- 172.21.15.189:0/4294134495 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2970206394 0 0) 0x7fd2880035b0 con 0x7fd2a00cf900 2024-04-06T12:12:57.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29d59a700 1 -- 172.21.15.189:0/4294134495 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2a00b4d70 msgr2=0x7fd2a00b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:57.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29d59a700 1 --2- 172.21.15.189:0/4294134495 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2a00b4d70 0x7fd2a00b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29d59a700 1 -- 172.21.15.189:0/4294134495 >> v2:172.21.15.192:3300/0 conn(0x7fd2a015a550 msgr2=0x7fd2a00cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:57.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29d59a700 1 --2- 172.21.15.189:0/4294134495 >> v2:172.21.15.192:3300/0 conn(0x7fd2a015a550 0x7fd2a00cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29d59a700 1 -- 172.21.15.189:0/4294134495 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a015aee0 con 0x7fd2a00cf900 2024-04-06T12:12:57.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29d59a700 1 -- 172.21.15.189:0/4294134495 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fd290001670 con 0x7fd2a00cf900 2024-04-06T12:12:57.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29d59a700 1 -- 172.21.15.189:0/4294134495 >> v1:172.21.15.189:6789/0 conn(0x7fd2a00cf900 legacy=0x7fd2a00b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:57.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29d59a700 1 --2- 172.21.15.189:0/4294134495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd288003a00 0x7fd288003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29d59a700 1 --2- 172.21.15.189:0/4294134495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd288004850 0x7fd288004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29d59a700 1 --2- 172.21.15.189:0/4294134495 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2a00b4d70 0x7fd288005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29d59a700 1 -- 172.21.15.189:0/4294134495 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a015a940 con 0x7fd288004850 2024-04-06T12:12:57.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29d59a700 1 -- 172.21.15.189:0/4294134495 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a015ac00 con 0x7fd288003a00 2024-04-06T12:12:57.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29d59a700 1 -- 172.21.15.189:0/4294134495 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28800bac0 con 0x7fd2a00b4d70 2024-04-06T12:12:57.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29dd9b700 1 --2- 172.21.15.189:0/4294134495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd288004850 0x7fd288004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29e59c700 1 --2- 172.21.15.189:0/4294134495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd288003a00 0x7fd288003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29ed9d700 1 --2- 172.21.15.189:0/4294134495 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2a00b4d70 0x7fd288005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29dd9b700 1 -- 172.21.15.189:0/4294134495 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2a00b4d70 msgr2=0x7fd288005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:57.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29dd9b700 1 --2- 172.21.15.189:0/4294134495 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2a00b4d70 0x7fd288005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29dd9b700 1 -- 172.21.15.189:0/4294134495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd288003a00 msgr2=0x7fd288003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:57.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29dd9b700 1 --2- 172.21.15.189:0/4294134495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd288003a00 0x7fd288003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29dd9b700 1 -- 172.21.15.189:0/4294134495 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2880035b0 con 0x7fd288004850 2024-04-06T12:12:57.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29ed9d700 1 --2- 172.21.15.189:0/4294134495 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2a00b4d70 0x7fd288005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:57.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.201+0000 7fd29dd9b700 1 --2- 172.21.15.189:0/4294134495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd288004850 0x7fd288004cd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd290001050 tx=0x7fd288005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e66bd34bc84df19 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:57.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.202+0000 7fd29d59a700 1 -- 172.21.15.189:0/4294134495 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd290007c60 con 0x7fd288004850 2024-04-06T12:12:57.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.202+0000 7fd29d59a700 1 -- 172.21.15.189:0/4294134495 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29000ce80 con 0x7fd288004850 2024-04-06T12:12:57.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.202+0000 7fd29d59a700 1 -- 172.21.15.189:0/4294134495 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd290005ba0 con 0x7fd288004850 2024-04-06T12:12:57.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.202+0000 7fd2a4e41700 1 -- 172.21.15.189:0/4294134495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd288004850 msgr2=0x7fd288004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:57.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.202+0000 7fd2a4e41700 1 --2- 172.21.15.189:0/4294134495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd288004850 0x7fd288004cd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd290001050 tx=0x7fd288005880 comp rx=0 tx=0).stop 2024-04-06T12:12:57.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.202+0000 7fd29ed9d700 1 -- 172.21.15.189:0/4294134495 reap_dead start 2024-04-06T12:12:57.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.202+0000 7fd2a4e41700 1 -- 172.21.15.189:0/4294134495 shutdown_connections 2024-04-06T12:12:57.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.202+0000 7fd2a4e41700 1 -- 172.21.15.189:0/4294134495 >> 172.21.15.189:0/4294134495 conn(0x7fd2a00d00b0 msgr2=0x7fd2a00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:57.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.202+0000 7fd2a4e41700 1 -- 172.21.15.189:0/4294134495 shutdown_connections 2024-04-06T12:12:57.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.202+0000 7fd2a4e41700 1 -- 172.21.15.189:0/4294134495 wait complete. 2024-04-06T12:12:57.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.204+0000 7fd2a4e41700 1 Processor -- start 2024-04-06T12:12:57.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.204+0000 7fd2a4e41700 1 -- start start 2024-04-06T12:12:57.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.204+0000 7fd2a4e41700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd288004850 0x7fd2a014d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.204+0000 7fd2a4e41700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2a01598f0 0x7fd2a00ad2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.204+0000 7fd2a4e41700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2a0158370 0x7fd2a01587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.204+0000 7fd2a4e41700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a0158d30 con 0x7fd288004850 2024-04-06T12:12:57.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.204+0000 7fd2a4e41700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a0152960 con 0x7fd2a0158370 2024-04-06T12:12:57.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.204+0000 7fd2a4e41700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a014df30 con 0x7fd2a01598f0 2024-04-06T12:12:57.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.205+0000 7fd29e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd288004850 0x7fd2a014d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.205+0000 7fd29e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd288004850 0x7fd2a014d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57644/0 (socket says 172.21.15.189:57644) 2024-04-06T12:12:57.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.205+0000 7fd29e59c700 1 -- 172.21.15.189:0/664005104 learned_addr learned my addr 172.21.15.189:0/664005104 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:57.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.205+0000 7fd29dd9b700 1 --2- 172.21.15.189:0/664005104 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2a01598f0 0x7fd2a00ad2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.205+0000 7fd29ed9d700 1 --2- 172.21.15.189:0/664005104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2a0158370 0x7fd2a01587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.205+0000 7fd29e59c700 1 -- 172.21.15.189:0/664005104 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2a01598f0 msgr2=0x7fd2a00ad2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:57.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.205+0000 7fd29e59c700 1 --2- 172.21.15.189:0/664005104 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2a01598f0 0x7fd2a00ad2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.205+0000 7fd29e59c700 1 -- 172.21.15.189:0/664005104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2a0158370 msgr2=0x7fd2a01587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:57.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.205+0000 7fd29e59c700 1 --2- 172.21.15.189:0/664005104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2a0158370 0x7fd2a01587f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.205+0000 7fd29e59c700 1 -- 172.21.15.189:0/664005104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a00adb00 con 0x7fd288004850 2024-04-06T12:12:57.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.205+0000 7fd29ed9d700 1 --2- 172.21.15.189:0/664005104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2a0158370 0x7fd2a01587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:57.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.205+0000 7fd29e59c700 1 --2- 172.21.15.189:0/664005104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd288004850 0x7fd2a014d9f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd294011930 tx=0x7fd294011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:57.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.205+0000 7fd28effd700 1 -- 172.21.15.189:0/664005104 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29400c070 con 0x7fd288004850 2024-04-06T12:12:57.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.205+0000 7fd28effd700 1 -- 172.21.15.189:0/664005104 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29400f8d0 con 0x7fd288004850 2024-04-06T12:12:57.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.205+0000 7fd2a4e41700 1 -- 172.21.15.189:0/664005104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a00adde0 con 0x7fd288004850 2024-04-06T12:12:57.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.205+0000 7fd28effd700 1 -- 172.21.15.189:0/664005104 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29400fc60 con 0x7fd288004850 2024-04-06T12:12:57.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.205+0000 7fd2a4e41700 1 -- 172.21.15.189:0/664005104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a00ae3a0 con 0x7fd288004850 2024-04-06T12:12:57.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.206+0000 7fd2a4e41700 1 -- 172.21.15.189:0/664005104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a0152af0 con 0x7fd288004850 2024-04-06T12:12:57.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.206+0000 7fd28effd700 1 -- 172.21.15.189:0/664005104 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd294015460 con 0x7fd288004850 2024-04-06T12:12:57.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.207+0000 7fd28effd700 1 --2- 172.21.15.189:0/664005104 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd28006acc0 0x7fd28006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.207+0000 7fd28effd700 1 -- 172.21.15.189:0/664005104 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd294089a20 con 0x7fd288004850 2024-04-06T12:12:57.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.207+0000 7fd29dd9b700 1 --2- 172.21.15.189:0/664005104 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd28006acc0 0x7fd28006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.207+0000 7fd29dd9b700 1 --2- 172.21.15.189:0/664005104 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd28006acc0 0x7fd28006d180 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd2a01590b0 tx=0x7fd29000a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:57.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.211+0000 7fd28effd700 1 -- 172.21.15.189:0/664005104 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd2a0152af0 con 0x7fd288004850 2024-04-06T12:12:57.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.404+0000 7fd2a4e41700 1 -- 172.21.15.189:0/664005104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd2a0067bb0 con 0x7fd288004850 2024-04-06T12:12:57.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.404+0000 7fd28effd700 1 -- 172.21.15.189:0/664005104 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd2940230a0 con 0x7fd288004850 2024-04-06T12:12:57.404 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:12:57.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.407+0000 7fd2a4e41700 1 -- 172.21.15.189:0/664005104 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd28006acc0 msgr2=0x7fd28006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:57.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.407+0000 7fd2a4e41700 1 --2- 172.21.15.189:0/664005104 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd28006acc0 0x7fd28006d180 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd2a01590b0 tx=0x7fd29000a000 comp rx=0 tx=0).stop 2024-04-06T12:12:57.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.407+0000 7fd2a4e41700 1 -- 172.21.15.189:0/664005104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd288004850 msgr2=0x7fd2a014d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:57.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.407+0000 7fd2a4e41700 1 --2- 172.21.15.189:0/664005104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd288004850 0x7fd2a014d9f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd294011930 tx=0x7fd294011cf0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.407+0000 7fd2a4e41700 1 -- 172.21.15.189:0/664005104 shutdown_connections 2024-04-06T12:12:57.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.407+0000 7fd2a4e41700 1 --2- 172.21.15.189:0/664005104 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd28006acc0 0x7fd28006d180 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.407+0000 7fd2a4e41700 1 --2- 172.21.15.189:0/664005104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2a0158370 0x7fd2a01587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.407+0000 7fd2a4e41700 1 --2- 172.21.15.189:0/664005104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd288004850 0x7fd2a014d9f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.407+0000 7fd2a4e41700 1 --2- 172.21.15.189:0/664005104 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2a01598f0 0x7fd2a00ad2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.407+0000 7fd2a4e41700 1 -- 172.21.15.189:0/664005104 >> 172.21.15.189:0/664005104 conn(0x7fd2a00d00b0 msgr2=0x7fd2a00cdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:57.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.407+0000 7fd2a4e41700 1 -- 172.21.15.189:0/664005104 shutdown_connections 2024-04-06T12:12:57.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.407+0000 7fd2a4e41700 1 -- 172.21.15.189:0/664005104 wait complete. 2024-04-06T12:12:57.418 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-06T12:12:57.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.543+0000 7fad08f2c700 1 Processor -- start 2024-04-06T12:12:57.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.543+0000 7fad08f2c700 1 -- start start 2024-04-06T12:12:57.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.544+0000 7fad08f2c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad0415a550 0x7fad040cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.544+0000 7fad08f2c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fad040cf900 0x7fad040b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.544+0000 7fad08f2c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad040cdbe0 con 0x7fad040b4ba0 2024-04-06T12:12:57.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.544+0000 7fad08f2c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad040b5010 con 0x7fad040cf900 2024-04-06T12:12:57.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.544+0000 7fad08f2c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0415a940 con 0x7fad0415a550 2024-04-06T12:12:57.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.544+0000 7fad01d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fad040cf900 0x7fad040b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.545+0000 7fad02d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fad040b4ba0 0x7fad0406eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39156/0 (socket says 172.21.15.189:39156) 2024-04-06T12:12:57.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.545+0000 7fad02d9d700 1 -- 172.21.15.189:0/4158478545 learned_addr learned my addr 172.21.15.189:0/4158478545 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:57.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.545+0000 7fad0259c700 1 --2- 172.21.15.189:0/4158478545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad0415a550 0x7fad040cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.545+0000 7fad01d9b700 1 -- 172.21.15.189:0/4158478545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad0415a550 msgr2=0x7fad040cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:57.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.545+0000 7fad01d9b700 1 --2- 172.21.15.189:0/4158478545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad0415a550 0x7fad040cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.545+0000 7fad01d9b700 1 -- 172.21.15.189:0/4158478545 >> v1:172.21.15.189:6789/0 conn(0x7fad040b4ba0 legacy=0x7fad0406eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:57.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.545+0000 7fad01d9b700 1 -- 172.21.15.189:0/4158478545 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0415ac00 con 0x7fad040cf900 2024-04-06T12:12:57.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.545+0000 7fad0159a700 1 -- 172.21.15.189:0/4158478545 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015035111 0 0) 0x7fad040cdbe0 con 0x7fad040b4ba0 2024-04-06T12:12:57.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.545+0000 7fad01d9b700 1 --2- 172.21.15.189:0/4158478545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad040cf900 0x7fad040b4660 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7facf4009840 tx=0x7facf4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ec68ecc407a3a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:57.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.545+0000 7fad0159a700 1 -- 172.21.15.189:0/4158478545 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf4011830 con 0x7fad040cf900 2024-04-06T12:12:57.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.545+0000 7fad0159a700 1 -- 172.21.15.189:0/4158478545 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf4011990 con 0x7fad040cf900 2024-04-06T12:12:57.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.545+0000 7fad0159a700 1 -- 172.21.15.189:0/4158478545 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf4011d20 con 0x7fad040cf900 2024-04-06T12:12:57.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.546+0000 7fad08f2c700 1 -- 172.21.15.189:0/4158478545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad040cf900 msgr2=0x7fad040b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:57.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.546+0000 7fad08f2c700 1 --2- 172.21.15.189:0/4158478545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad040cf900 0x7fad040b4660 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7facf4009840 tx=0x7facf4009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:57.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.546+0000 7fad08f2c700 1 -- 172.21.15.189:0/4158478545 shutdown_connections 2024-04-06T12:12:57.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.546+0000 7fad08f2c700 1 --2- 172.21.15.189:0/4158478545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad0415a550 0x7fad040cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.546+0000 7fad08f2c700 1 --2- 172.21.15.189:0/4158478545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad040cf900 0x7fad040b4660 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.546+0000 7fad08f2c700 1 -- 172.21.15.189:0/4158478545 >> 172.21.15.189:0/4158478545 conn(0x7fad040d00b0 msgr2=0x7fad040d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:57.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.546+0000 7fad08f2c700 1 -- 172.21.15.189:0/4158478545 shutdown_connections 2024-04-06T12:12:57.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.546+0000 7fad08f2c700 1 -- 172.21.15.189:0/4158478545 wait complete. 2024-04-06T12:12:57.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.547+0000 7fad08f2c700 1 Processor -- start 2024-04-06T12:12:57.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.547+0000 7fad08f2c700 1 -- start start 2024-04-06T12:12:57.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.548+0000 7fad08f2c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad040b4ba0 0x7fad041f0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.548+0000 7fad08f2c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad040cf900 0x7fad0414c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.548+0000 7fad08f2c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad04152e90 0x7fad041567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.548+0000 7fad08f2c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad04156310 con 0x7fad040b4ba0 2024-04-06T12:12:57.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.548+0000 7fad08f2c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad041f13b0 con 0x7fad040cf900 2024-04-06T12:12:57.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.548+0000 7fad08f2c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad04155700 con 0x7fad04152e90 2024-04-06T12:12:57.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.548+0000 7fad0259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad040b4ba0 0x7fad041f0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.548+0000 7fad0259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad040b4ba0 0x7fad041f0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57664/0 (socket says 172.21.15.189:57664) 2024-04-06T12:12:57.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.548+0000 7fad0259c700 1 -- 172.21.15.189:0/1077560373 learned_addr learned my addr 172.21.15.189:0/1077560373 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:57.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.548+0000 7fad02d9d700 1 --2- 172.21.15.189:0/1077560373 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad04152e90 0x7fad041567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.548+0000 7fad01d9b700 1 --2- 172.21.15.189:0/1077560373 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad040cf900 0x7fad0414c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.548+0000 7fad0259c700 1 -- 172.21.15.189:0/1077560373 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad04152e90 msgr2=0x7fad041567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:57.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.548+0000 7fad0259c700 1 --2- 172.21.15.189:0/1077560373 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad04152e90 0x7fad041567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.548+0000 7fad0259c700 1 -- 172.21.15.189:0/1077560373 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad040cf900 msgr2=0x7fad0414c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:57.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.548+0000 7fad0259c700 1 --2- 172.21.15.189:0/1077560373 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad040cf900 0x7fad0414c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.548+0000 7fad0259c700 1 -- 172.21.15.189:0/1077560373 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf40094a0 con 0x7fad040b4ba0 2024-04-06T12:12:57.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.549+0000 7fad0259c700 1 --2- 172.21.15.189:0/1077560373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad040b4ba0 0x7fad041f0e70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7facf800e870 tx=0x7facf800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:57.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.549+0000 7facf2ffd700 1 -- 172.21.15.189:0/1077560373 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf8009590 con 0x7fad040b4ba0 2024-04-06T12:12:57.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.549+0000 7facf2ffd700 1 -- 172.21.15.189:0/1077560373 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf8013680 con 0x7fad040b4ba0 2024-04-06T12:12:57.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.549+0000 7facf2ffd700 1 -- 172.21.15.189:0/1077560373 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf80139f0 con 0x7fad040b4ba0 2024-04-06T12:12:57.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.549+0000 7fad08f2c700 1 -- 172.21.15.189:0/1077560373 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad041559e0 con 0x7fad040b4ba0 2024-04-06T12:12:57.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.549+0000 7fad08f2c700 1 -- 172.21.15.189:0/1077560373 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad04155fa0 con 0x7fad040b4ba0 2024-04-06T12:12:57.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.550+0000 7fad08f2c700 1 -- 172.21.15.189:0/1077560373 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad04067bb0 con 0x7fad040b4ba0 2024-04-06T12:12:57.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.550+0000 7facf2ffd700 1 -- 172.21.15.189:0/1077560373 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7facf8013b50 con 0x7fad040b4ba0 2024-04-06T12:12:57.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.551+0000 7facf2ffd700 1 --2- 172.21.15.189:0/1077560373 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7face406ac90 0x7face406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.551+0000 7fad01d9b700 1 --2- 172.21.15.189:0/1077560373 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7face406ac90 0x7face406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.551+0000 7facf2ffd700 1 -- 172.21.15.189:0/1077560373 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7facf80895e0 con 0x7fad040b4ba0 2024-04-06T12:12:57.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.551+0000 7fad01d9b700 1 --2- 172.21.15.189:0/1077560373 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7face406ac90 0x7face406d150 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7facf40090b0 tx=0x7facf4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:57.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.555+0000 7facf2ffd700 1 -- 172.21.15.189:0/1077560373 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7facf80543f0 con 0x7fad040b4ba0 2024-04-06T12:12:57.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.750+0000 7fad08f2c700 1 -- 172.21.15.189:0/1077560373 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fad04074990 con 0x7fad040b4ba0 2024-04-06T12:12:57.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.750+0000 7facf2ffd700 1 -- 172.21.15.189:0/1077560373 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7facf8057a10 con 0x7fad040b4ba0 2024-04-06T12:12:57.750 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:12:57.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.753+0000 7fad08f2c700 1 -- 172.21.15.189:0/1077560373 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7face406ac90 msgr2=0x7face406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:57.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.753+0000 7fad08f2c700 1 --2- 172.21.15.189:0/1077560373 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7face406ac90 0x7face406d150 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7facf40090b0 tx=0x7facf4008030 comp rx=0 tx=0).stop 2024-04-06T12:12:57.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.753+0000 7fad08f2c700 1 -- 172.21.15.189:0/1077560373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad040b4ba0 msgr2=0x7fad041f0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:57.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.753+0000 7fad08f2c700 1 --2- 172.21.15.189:0/1077560373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad040b4ba0 0x7fad041f0e70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7facf800e870 tx=0x7facf800eb80 comp rx=0 tx=0).stop 2024-04-06T12:12:57.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.753+0000 7fad08f2c700 1 -- 172.21.15.189:0/1077560373 shutdown_connections 2024-04-06T12:12:57.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.753+0000 7fad08f2c700 1 --2- 172.21.15.189:0/1077560373 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad04152e90 0x7fad041567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.753+0000 7fad08f2c700 1 --2- 172.21.15.189:0/1077560373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad040b4ba0 0x7fad041f0e70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.753+0000 7fad08f2c700 1 --2- 172.21.15.189:0/1077560373 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7face406ac90 0x7face406d150 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.753+0000 7fad08f2c700 1 --2- 172.21.15.189:0/1077560373 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad040cf900 0x7fad0414c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.753+0000 7fad08f2c700 1 -- 172.21.15.189:0/1077560373 >> 172.21.15.189:0/1077560373 conn(0x7fad040d00b0 msgr2=0x7fad040cc3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:57.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.753+0000 7fad08f2c700 1 -- 172.21.15.189:0/1077560373 shutdown_connections 2024-04-06T12:12:57.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.753+0000 7fad08f2c700 1 -- 172.21.15.189:0/1077560373 wait complete. 2024-04-06T12:12:57.764 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T12:12:57.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.888+0000 7f2fb5351700 1 Processor -- start 2024-04-06T12:12:57.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.888+0000 7f2fb5351700 1 -- start start 2024-04-06T12:12:57.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fb5351700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2fb00b6560 0x7f2fb00ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fb5351700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fb00d1500 0x7f2fb00ce640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fb5351700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb00b90a0 con 0x7f2fb015d470 2024-04-06T12:12:57.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fb5351700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fb00b6a40 con 0x7f2fb00b6560 2024-04-06T12:12:57.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fb5351700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb00b6d20 con 0x7f2fb00d1500 2024-04-06T12:12:57.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2faeffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2fb015d470 0x7f2fb00d0670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39162/0 (socket says 172.21.15.189:39162) 2024-04-06T12:12:57.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2faeffd700 1 -- 172.21.15.189:0/569010704 learned_addr learned my addr 172.21.15.189:0/569010704 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:57.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2faf7fe700 1 --2- 172.21.15.189:0/569010704 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fb00d1500 0x7f2fb00ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fae7fc700 1 --2- 172.21.15.189:0/569010704 >> v2:172.21.15.192:3300/0 conn(0x7f2fb00b6560 0x7f2fb00ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fadffb700 1 -- 172.21.15.189:0/569010704 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640815035 0 0) 0x7f2fb00b90a0 con 0x7f2fb015d470 2024-04-06T12:12:57.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fadffb700 1 -- 172.21.15.189:0/569010704 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f980035b0 con 0x7f2fb015d470 2024-04-06T12:12:57.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fadffb700 1 -- 172.21.15.189:0/569010704 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4127923263 0 0) 0x7f2f980035b0 con 0x7f2fb015d470 2024-04-06T12:12:57.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fadffb700 1 -- 172.21.15.189:0/569010704 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fb00d1500 msgr2=0x7f2fb00ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:57.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fadffb700 1 --2- 172.21.15.189:0/569010704 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fb00d1500 0x7f2fb00ce640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fadffb700 1 -- 172.21.15.189:0/569010704 >> v2:172.21.15.192:3300/0 conn(0x7f2fb00b6560 msgr2=0x7f2fb00ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:57.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fadffb700 1 --2- 172.21.15.189:0/569010704 >> v2:172.21.15.192:3300/0 conn(0x7f2fb00b6560 0x7f2fb00ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fadffb700 1 -- 172.21.15.189:0/569010704 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb00b7000 con 0x7f2fb015d470 2024-04-06T12:12:57.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fadffb700 1 -- 172.21.15.189:0/569010704 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f2fa0001670 con 0x7f2fb015d470 2024-04-06T12:12:57.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fadffb700 1 -- 172.21.15.189:0/569010704 >> v1:172.21.15.189:6789/0 conn(0x7f2fb015d470 legacy=0x7f2fb00d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:57.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fadffb700 1 --2- 172.21.15.189:0/569010704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f98003a00 0x7f2f98003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fadffb700 1 --2- 172.21.15.189:0/569010704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f98004850 0x7f2f98004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fadffb700 1 --2- 172.21.15.189:0/569010704 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fb00d1500 0x7f2f98005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fadffb700 1 -- 172.21.15.189:0/569010704 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb00b6a40 con 0x7f2f98003a00 2024-04-06T12:12:57.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fadffb700 1 -- 172.21.15.189:0/569010704 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb00b6d20 con 0x7f2f98004850 2024-04-06T12:12:57.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2fadffb700 1 -- 172.21.15.189:0/569010704 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9800ba10 con 0x7f2fb00d1500 2024-04-06T12:12:57.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.889+0000 7f2faeffd700 1 --2- 172.21.15.189:0/569010704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f98003a00 0x7f2f98003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.890+0000 7f2fae7fc700 1 --2- 172.21.15.189:0/569010704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f98004850 0x7f2f98004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.890+0000 7f2faf7fe700 1 --2- 172.21.15.189:0/569010704 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fb00d1500 0x7f2f98005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.890+0000 7f2faeffd700 1 -- 172.21.15.189:0/569010704 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fb00d1500 msgr2=0x7f2f98005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:57.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.890+0000 7f2faeffd700 1 --2- 172.21.15.189:0/569010704 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fb00d1500 0x7f2f98005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.890+0000 7f2faeffd700 1 -- 172.21.15.189:0/569010704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f98004850 msgr2=0x7f2f98004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:57.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.890+0000 7f2faeffd700 1 --2- 172.21.15.189:0/569010704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f98004850 0x7f2f98004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.890+0000 7f2faeffd700 1 -- 172.21.15.189:0/569010704 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f980035b0 con 0x7f2f98003a00 2024-04-06T12:12:57.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.890+0000 7f2faeffd700 1 --2- 172.21.15.189:0/569010704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f98003a00 0x7f2f98003e20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000c370 tx=0x7f2f98005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e421cb873ea778 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:57.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.890+0000 7f2fadffb700 1 -- 172.21.15.189:0/569010704 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa0007c60 con 0x7f2f98003a00 2024-04-06T12:12:57.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.890+0000 7f2fadffb700 1 -- 172.21.15.189:0/569010704 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa0005960 con 0x7f2f98003a00 2024-04-06T12:12:57.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.890+0000 7f2fadffb700 1 -- 172.21.15.189:0/569010704 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa0005d10 con 0x7f2f98003a00 2024-04-06T12:12:57.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.890+0000 7f2fb5351700 1 -- 172.21.15.189:0/569010704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f98003a00 msgr2=0x7f2f98003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:57.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.890+0000 7f2fb5351700 1 --2- 172.21.15.189:0/569010704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f98003a00 0x7f2f98003e20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000c370 tx=0x7f2f98005880 comp rx=0 tx=0).stop 2024-04-06T12:12:57.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.890+0000 7f2faf7fe700 1 -- 172.21.15.189:0/569010704 reap_dead start 2024-04-06T12:12:57.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.891+0000 7f2fb5351700 1 -- 172.21.15.189:0/569010704 shutdown_connections 2024-04-06T12:12:57.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.891+0000 7f2fb5351700 1 -- 172.21.15.189:0/569010704 >> 172.21.15.189:0/569010704 conn(0x7f2fb006bbc0 msgr2=0x7f2fb006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:57.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.891+0000 7f2fb5351700 1 -- 172.21.15.189:0/569010704 shutdown_connections 2024-04-06T12:12:57.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.891+0000 7f2fb5351700 1 -- 172.21.15.189:0/569010704 wait complete. 2024-04-06T12:12:57.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.892+0000 7f2fb5351700 1 Processor -- start 2024-04-06T12:12:57.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.892+0000 7f2fb5351700 1 -- start start 2024-04-06T12:12:57.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2fb5351700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f98003a00 0x7f2fb01f60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2fb5351700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fb0159320 0x7f2fb01543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2fb5351700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fb0156c80 0x7f2fb0157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2fb5351700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb015a480 con 0x7f2fb0156c80 2024-04-06T12:12:57.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2fb5351700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb0157640 con 0x7f2fb0159320 2024-04-06T12:12:57.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2fb5351700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb00ce730 con 0x7f2f98003a00 2024-04-06T12:12:57.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2faf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fb0156c80 0x7f2fb0157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2faf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fb0156c80 0x7f2fb0157100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57694/0 (socket says 172.21.15.189:57694) 2024-04-06T12:12:57.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2faf7fe700 1 -- 172.21.15.189:0/1229944116 learned_addr learned my addr 172.21.15.189:0/1229944116 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:57.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2faeffd700 1 --2- 172.21.15.189:0/1229944116 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f98003a00 0x7f2fb01f60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2fae7fc700 1 --2- 172.21.15.189:0/1229944116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fb0159320 0x7f2fb01543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2faf7fe700 1 -- 172.21.15.189:0/1229944116 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f98003a00 msgr2=0x7f2fb01f60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:57.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2faf7fe700 1 --2- 172.21.15.189:0/1229944116 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f98003a00 0x7f2fb01f60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2faf7fe700 1 -- 172.21.15.189:0/1229944116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fb0159320 msgr2=0x7f2fb01543b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:57.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2faf7fe700 1 --2- 172.21.15.189:0/1229944116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fb0159320 0x7f2fb01543b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:57.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2faf7fe700 1 -- 172.21.15.189:0/1229944116 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb0159870 con 0x7f2fb0156c80 2024-04-06T12:12:57.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2faf7fe700 1 --2- 172.21.15.189:0/1229944116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fb0156c80 0x7f2fb0157100 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2fb00b6350 tx=0x7f2fa8011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:57.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2f977fe700 1 -- 172.21.15.189:0/1229944116 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa800e040 con 0x7f2fb0156c80 2024-04-06T12:12:57.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2f977fe700 1 -- 172.21.15.189:0/1229944116 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa800f040 con 0x7f2fb0156c80 2024-04-06T12:12:57.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2f977fe700 1 -- 172.21.15.189:0/1229944116 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa800fd00 con 0x7f2fb0156c80 2024-04-06T12:12:57.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2fb5351700 1 -- 172.21.15.189:0/1229944116 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb0159b50 con 0x7f2fb0156c80 2024-04-06T12:12:57.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.893+0000 7f2fb5351700 1 -- 172.21.15.189:0/1229944116 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb015a110 con 0x7f2fb0156c80 2024-04-06T12:12:57.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.894+0000 7f2fb5351700 1 -- 172.21.15.189:0/1229944116 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb00b58b0 con 0x7f2fb0156c80 2024-04-06T12:12:57.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.895+0000 7f2f977fe700 1 -- 172.21.15.189:0/1229944116 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2fa8015430 con 0x7f2fb0156c80 2024-04-06T12:12:57.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.895+0000 7f2f977fe700 1 --2- 172.21.15.189:0/1229944116 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2f9c06ac90 0x7f2f9c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:57.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.895+0000 7f2f977fe700 1 -- 172.21.15.189:0/1229944116 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2fa8089e40 con 0x7f2fb0156c80 2024-04-06T12:12:57.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.895+0000 7f2faeffd700 1 --2- 172.21.15.189:0/1229944116 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2f9c06ac90 0x7f2f9c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:57.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.896+0000 7f2faeffd700 1 --2- 172.21.15.189:0/1229944116 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2f9c06ac90 0x7f2f9c06d150 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2fb00cb710 tx=0x7f2fa000a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:57.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:57.900+0000 7f2f977fe700 1 -- 172.21.15.189:0/1229944116 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2fa8054cf0 con 0x7f2fb0156c80 2024-04-06T12:12:58.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.109+0000 7f2fb5351700 1 -- 172.21.15.189:0/1229944116 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2fb0074990 con 0x7f2fb0156c80 2024-04-06T12:12:58.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.110+0000 7f2f977fe700 1 -- 172.21.15.189:0/1229944116 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2fa8058310 con 0x7f2fb0156c80 2024-04-06T12:12:58.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.112+0000 7f2fb5351700 1 -- 172.21.15.189:0/1229944116 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2f9c06ac90 msgr2=0x7f2f9c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:58.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.112+0000 7f2fb5351700 1 --2- 172.21.15.189:0/1229944116 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2f9c06ac90 0x7f2f9c06d150 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2fb00cb710 tx=0x7f2fa000a000 comp rx=0 tx=0).stop 2024-04-06T12:12:58.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.112+0000 7f2fb5351700 1 -- 172.21.15.189:0/1229944116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fb0156c80 msgr2=0x7f2fb0157100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:58.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.112+0000 7f2fb5351700 1 --2- 172.21.15.189:0/1229944116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fb0156c80 0x7f2fb0157100 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2fb00b6350 tx=0x7f2fa8011e60 comp rx=0 tx=0).stop 2024-04-06T12:12:58.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.112+0000 7f2fb5351700 1 -- 172.21.15.189:0/1229944116 shutdown_connections 2024-04-06T12:12:58.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.112+0000 7f2fb5351700 1 --2- 172.21.15.189:0/1229944116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fb0156c80 0x7f2fb0157100 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:58.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.112+0000 7f2fb5351700 1 --2- 172.21.15.189:0/1229944116 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f98003a00 0x7f2fb01f60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:58.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.112+0000 7f2fb5351700 1 --2- 172.21.15.189:0/1229944116 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2f9c06ac90 0x7f2f9c06d150 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:58.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.112+0000 7f2fb5351700 1 --2- 172.21.15.189:0/1229944116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fb0159320 0x7f2fb01543b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:58.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.112+0000 7f2fb5351700 1 -- 172.21.15.189:0/1229944116 >> 172.21.15.189:0/1229944116 conn(0x7f2fb006bbc0 msgr2=0x7f2fb00c9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:58.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.112+0000 7f2fb5351700 1 -- 172.21.15.189:0/1229944116 shutdown_connections 2024-04-06T12:12:58.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.112+0000 7f2fb5351700 1 -- 172.21.15.189:0/1229944116 wait complete. 2024-04-06T12:12:58.112 INFO:teuthology.orchestra.run.smithi189.stderr:listed 0 entries 2024-04-06T12:12:58.123 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-06T12:12:58.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:58.124 DEBUG:teuthology.orchestra.run.smithi189:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-04-06T12:12:58.166 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T12:12:58.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:12:58.167 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-06T12:12:58.224 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:12:58.225 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:12:58.246 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:12:58.247 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:12:58.308 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:12:58.308 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:12:58.369 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:12:58.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.493+0000 7f7cd54b0700 1 Processor -- start 2024-04-06T12:12:58.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.493+0000 7f7cd54b0700 1 -- start start 2024-04-06T12:12:58.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.493+0000 7f7cd54b0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7cd00ce730 0x7f7cd015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:58.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.493+0000 7f7cd54b0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd015eac0 con 0x7f7cd015d6f0 2024-04-06T12:12:58.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.493+0000 7f7cd54b0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd015e220 con 0x7f7cd00b8d70 2024-04-06T12:12:58.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.493+0000 7f7cd54b0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cd015e010 con 0x7f7cd00ce730 2024-04-06T12:12:58.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.494+0000 7f7cce7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7cd00ce730 0x7f7cd015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:58.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.494+0000 7f7cceffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7cd015d6f0 0x7f7cd00ce020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39164/0 (socket says 172.21.15.189:39164) 2024-04-06T12:12:58.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.494+0000 7f7cceffd700 1 -- 172.21.15.189:0/3034657421 learned_addr learned my addr 172.21.15.189:0/3034657421 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.494+0000 7f7cce7fc700 1 --2- 172.21.15.189:0/3034657421 >> v2:172.21.15.189:3300/0 conn(0x7f7cd00ce730 0x7f7cd015dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57710/0 (socket says 172.21.15.189:57710) 2024-04-06T12:12:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.494+0000 7f7cce7fc700 1 -- 172.21.15.189:0/3034657421 >> v1:172.21.15.192:6789/0 conn(0x7f7cd00b8d70 legacy=0x7f7cd00d1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.494+0000 7f7cce7fc700 1 -- 172.21.15.189:0/3034657421 >> v1:172.21.15.189:6789/0 conn(0x7f7cd015d6f0 legacy=0x7f7cd00ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.494+0000 7f7cce7fc700 1 -- 172.21.15.189:0/3034657421 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd015e500 con 0x7f7cd00ce730 2024-04-06T12:12:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.494+0000 7f7ccdffb700 1 -- 172.21.15.189:0/3034657421 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2523591492 0 0) 0x7f7cd015eac0 con 0x7f7cd015d6f0 2024-04-06T12:12:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.494+0000 7f7cce7fc700 1 --2- 172.21.15.189:0/3034657421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cd00ce730 0x7f7cd015dad0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009840 tx=0x7f7cc4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a192a0f6bcc35dad server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.494+0000 7f7ccdffb700 1 -- 172.21.15.189:0/3034657421 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4004230 con 0x7f7cd00ce730 2024-04-06T12:12:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.494+0000 7f7ccdffb700 1 -- 172.21.15.189:0/3034657421 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc4004390 con 0x7f7cd00ce730 2024-04-06T12:12:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.494+0000 7f7ccdffb700 1 -- 172.21.15.189:0/3034657421 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc40036a0 con 0x7f7cd00ce730 2024-04-06T12:12:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.495+0000 7f7cd54b0700 1 -- 172.21.15.189:0/3034657421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cd00ce730 msgr2=0x7f7cd015dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.495+0000 7f7cd54b0700 1 --2- 172.21.15.189:0/3034657421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cd00ce730 0x7f7cd015dad0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009840 tx=0x7f7cc4009b50 comp rx=0 tx=0).stop 2024-04-06T12:12:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.495+0000 7f7cd54b0700 1 -- 172.21.15.189:0/3034657421 shutdown_connections 2024-04-06T12:12:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.495+0000 7f7cd54b0700 1 --2- 172.21.15.189:0/3034657421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cd00ce730 0x7f7cd015dad0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.495+0000 7f7cd54b0700 1 -- 172.21.15.189:0/3034657421 >> 172.21.15.189:0/3034657421 conn(0x7f7cd0072490 msgr2=0x7f7cd006b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.495+0000 7f7cd54b0700 1 -- 172.21.15.189:0/3034657421 shutdown_connections 2024-04-06T12:12:58.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.495+0000 7f7cd54b0700 1 -- 172.21.15.189:0/3034657421 wait complete. 2024-04-06T12:12:58.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.496+0000 7f7cd54b0700 1 Processor -- start 2024-04-06T12:12:58.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.496+0000 7f7cd54b0700 1 -- start start 2024-04-06T12:12:58.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.497+0000 7f7cd54b0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7cd00b8d70 0x7f7cd01f2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:58.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.497+0000 7f7cd54b0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7cd00ce730 0x7f7cd0159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:58.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.497+0000 7f7cd54b0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cd015d6f0 0x7f7cd00b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:58.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.497+0000 7f7cd54b0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd01f3130 con 0x7f7cd015d6f0 2024-04-06T12:12:58.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.497+0000 7f7cd54b0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd01f3410 con 0x7f7cd00ce730 2024-04-06T12:12:58.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.497+0000 7f7cd54b0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd01f36f0 con 0x7f7cd00b8d70 2024-04-06T12:12:58.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.497+0000 7f7ccf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cd015d6f0 0x7f7cd00b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:58.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.497+0000 7f7ccf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cd015d6f0 0x7f7cd00b3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57712/0 (socket says 172.21.15.189:57712) 2024-04-06T12:12:58.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.497+0000 7f7ccf7fe700 1 -- 172.21.15.189:0/2221775950 learned_addr learned my addr 172.21.15.189:0/2221775950 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:58.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.497+0000 7f7cceffd700 1 --2- 172.21.15.189:0/2221775950 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7cd00b8d70 0x7f7cd01f2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:58.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.497+0000 7f7cce7fc700 1 --2- 172.21.15.189:0/2221775950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7cd00ce730 0x7f7cd0159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:58.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.497+0000 7f7ccf7fe700 1 -- 172.21.15.189:0/2221775950 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7cd00b8d70 msgr2=0x7f7cd01f2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:58.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.497+0000 7f7ccf7fe700 1 --2- 172.21.15.189:0/2221775950 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7cd00b8d70 0x7f7cd01f2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:58.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.497+0000 7f7ccf7fe700 1 -- 172.21.15.189:0/2221775950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7cd00ce730 msgr2=0x7f7cd0159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:58.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.497+0000 7f7ccf7fe700 1 --2- 172.21.15.189:0/2221775950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7cd00ce730 0x7f7cd0159f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:58.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.497+0000 7f7ccf7fe700 1 -- 172.21.15.189:0/2221775950 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc40094a0 con 0x7f7cd015d6f0 2024-04-06T12:12:58.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.498+0000 7f7ccf7fe700 1 --2- 172.21.15.189:0/2221775950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cd015d6f0 0x7f7cd00b3b60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7cc800cc10 tx=0x7f7cc800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:58.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.498+0000 7f7cb77fe700 1 -- 172.21.15.189:0/2221775950 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc800abb0 con 0x7f7cd015d6f0 2024-04-06T12:12:58.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.498+0000 7f7cb77fe700 1 -- 172.21.15.189:0/2221775950 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc800ad10 con 0x7f7cd015d6f0 2024-04-06T12:12:58.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.498+0000 7f7cd54b0700 1 -- 172.21.15.189:0/2221775950 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd01f39d0 con 0x7f7cd015d6f0 2024-04-06T12:12:58.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.498+0000 7f7cb77fe700 1 -- 172.21.15.189:0/2221775950 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8012660 con 0x7f7cd015d6f0 2024-04-06T12:12:58.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.498+0000 7f7cd54b0700 1 -- 172.21.15.189:0/2221775950 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd0160230 con 0x7f7cd015d6f0 2024-04-06T12:12:58.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.499+0000 7f7cd54b0700 1 -- 172.21.15.189:0/2221775950 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd0074990 con 0x7f7cd015d6f0 2024-04-06T12:12:58.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.499+0000 7f7cb77fe700 1 -- 172.21.15.189:0/2221775950 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7cc80127c0 con 0x7f7cd015d6f0 2024-04-06T12:12:58.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.499+0000 7f7cb77fe700 1 --2- 172.21.15.189:0/2221775950 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7cbc06ac90 0x7f7cbc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:58.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.499+0000 7f7cb77fe700 1 -- 172.21.15.189:0/2221775950 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7cc8089eb0 con 0x7f7cd015d6f0 2024-04-06T12:12:58.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.500+0000 7f7cceffd700 1 --2- 172.21.15.189:0/2221775950 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7cbc06ac90 0x7f7cbc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:58.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.500+0000 7f7cceffd700 1 --2- 172.21.15.189:0/2221775950 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7cbc06ac90 0x7f7cbc06d150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0000f60 tx=0x7f7cc0007c90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:58.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.504+0000 7f7cb77fe700 1 -- 172.21.15.189:0/2221775950 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7cc8054cd0 con 0x7f7cd015d6f0 2024-04-06T12:12:58.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.699+0000 7f7cd54b0700 1 -- 172.21.15.189:0/2221775950 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7cd00c6860 con 0x7f7cd015d6f0 2024-04-06T12:12:58.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.701+0000 7f7cb77fe700 1 -- 172.21.15.189:0/2221775950 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18314 (secure 0 0 0) 0x7f7cc8016070 con 0x7f7cd015d6f0 2024-04-06T12:12:58.700 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:12:58.700 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":26,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:54.376742+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:12:58.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.703+0000 7f7cd54b0700 1 -- 172.21.15.189:0/2221775950 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7cbc06ac90 msgr2=0x7f7cbc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:58.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.703+0000 7f7cd54b0700 1 --2- 172.21.15.189:0/2221775950 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7cbc06ac90 0x7f7cbc06d150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0000f60 tx=0x7f7cc0007c90 comp rx=0 tx=0).stop 2024-04-06T12:12:58.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.703+0000 7f7cd54b0700 1 -- 172.21.15.189:0/2221775950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cd015d6f0 msgr2=0x7f7cd00b3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:58.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.703+0000 7f7cd54b0700 1 --2- 172.21.15.189:0/2221775950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cd015d6f0 0x7f7cd00b3b60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7cc800cc10 tx=0x7f7cc800a060 comp rx=0 tx=0).stop 2024-04-06T12:12:58.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.703+0000 7f7cd54b0700 1 -- 172.21.15.189:0/2221775950 shutdown_connections 2024-04-06T12:12:58.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.703+0000 7f7cd54b0700 1 --2- 172.21.15.189:0/2221775950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cd015d6f0 0x7f7cd00b3b60 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:58.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.703+0000 7f7cd54b0700 1 --2- 172.21.15.189:0/2221775950 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7cd00b8d70 0x7f7cd01f2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:58.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.703+0000 7f7cd54b0700 1 --2- 172.21.15.189:0/2221775950 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7cbc06ac90 0x7f7cbc06d150 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:58.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.703+0000 7f7cd54b0700 1 --2- 172.21.15.189:0/2221775950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7cd00ce730 0x7f7cd0159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:58.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.703+0000 7f7cd54b0700 1 -- 172.21.15.189:0/2221775950 >> 172.21.15.189:0/2221775950 conn(0x7f7cd0072490 msgr2=0x7f7cd00d1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:58.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.703+0000 7f7cd54b0700 1 -- 172.21.15.189:0/2221775950 shutdown_connections 2024-04-06T12:12:58.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.703+0000 7f7cd54b0700 1 -- 172.21.15.189:0/2221775950 wait complete. 2024-04-06T12:12:58.715 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:12:02.280866+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:12:03.156593+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-06T12:12:06.237184+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-06T12:12:58.716 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:12:58.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.841+0000 7f8bbeed9700 1 Processor -- start 2024-04-06T12:12:58.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.841+0000 7f8bbeed9700 1 -- start start 2024-04-06T12:12:58.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.842+0000 7f8bbeed9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bb80cb870 0x7f8bb815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:58.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.842+0000 7f8bbeed9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb80cbd80 con 0x7f8bb80d11b0 2024-04-06T12:12:58.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.842+0000 7f8bbeed9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb80cef20 con 0x7f8bb80cfa00 2024-04-06T12:12:58.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.842+0000 7f8bbeed9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb80ccdc0 con 0x7f8bb80cb870 2024-04-06T12:12:58.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.842+0000 7f8bbcc75700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8bb80d11b0 0x7f8bb80d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39174/0 (socket says 172.21.15.189:39174) 2024-04-06T12:12:58.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.842+0000 7f8bbcc75700 1 -- 172.21.15.189:0/1863831030 learned_addr learned my addr 172.21.15.189:0/1863831030 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:58.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.842+0000 7f8bb7fff700 1 --2- 172.21.15.189:0/1863831030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bb80cb870 0x7f8bb815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:58.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.842+0000 7f8bb77fe700 1 -- 172.21.15.189:0/1863831030 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610991090 0 0) 0x7f8bb80cbd80 con 0x7f8bb80d11b0 2024-04-06T12:12:58.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.842+0000 7f8bb77fe700 1 -- 172.21.15.189:0/1863831030 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ba80035b0 con 0x7f8bb80d11b0 2024-04-06T12:12:58.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.842+0000 7f8bb77fe700 1 -- 172.21.15.189:0/1863831030 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842074166 0 0) 0x7f8bb80cef20 con 0x7f8bb80cfa00 2024-04-06T12:12:58.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.842+0000 7f8bb77fe700 1 -- 172.21.15.189:0/1863831030 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb80cbd80 con 0x7f8bb80cfa00 2024-04-06T12:12:58.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.842+0000 7f8bb77fe700 1 -- 172.21.15.189:0/1863831030 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2954623825 0 0) 0x7f8ba80035b0 con 0x7f8bb80d11b0 2024-04-06T12:12:58.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.842+0000 7f8bb7fff700 1 -- 172.21.15.189:0/1863831030 >> v1:172.21.15.192:6789/0 conn(0x7f8bb80cfa00 legacy=0x7f8bb80cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:58.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.842+0000 7f8bb7fff700 1 -- 172.21.15.189:0/1863831030 >> v1:172.21.15.189:6789/0 conn(0x7f8bb80d11b0 legacy=0x7f8bb80d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:58.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.842+0000 7f8bb7fff700 1 -- 172.21.15.189:0/1863831030 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb80cf200 con 0x7f8bb80cb870 2024-04-06T12:12:58.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.843+0000 7f8bb7fff700 1 --2- 172.21.15.189:0/1863831030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bb80cb870 0x7f8bb815c280 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4009870 tx=0x7f8ba4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa7010db3d0904f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:58.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.843+0000 7f8bb77fe700 1 -- 172.21.15.189:0/1863831030 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac001670 con 0x7f8bb80cb870 2024-04-06T12:12:58.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.843+0000 7f8bb77fe700 1 -- 172.21.15.189:0/1863831030 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba4004030 con 0x7f8bb80cb870 2024-04-06T12:12:58.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.843+0000 7f8bb77fe700 1 -- 172.21.15.189:0/1863831030 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba40043c0 con 0x7f8bb80cb870 2024-04-06T12:12:58.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.843+0000 7f8bbeed9700 1 -- 172.21.15.189:0/1863831030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bb80cb870 msgr2=0x7f8bb815c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:58.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.843+0000 7f8bbeed9700 1 --2- 172.21.15.189:0/1863831030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bb80cb870 0x7f8bb815c280 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4009870 tx=0x7f8ba4009b80 comp rx=0 tx=0).stop 2024-04-06T12:12:58.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.843+0000 7f8bbeed9700 1 -- 172.21.15.189:0/1863831030 shutdown_connections 2024-04-06T12:12:58.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.843+0000 7f8bbeed9700 1 --2- 172.21.15.189:0/1863831030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bb80cb870 0x7f8bb815c280 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:58.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.843+0000 7f8bbeed9700 1 -- 172.21.15.189:0/1863831030 >> 172.21.15.189:0/1863831030 conn(0x7f8bb80b2db0 msgr2=0x7f8bb80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:58.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.843+0000 7f8bbeed9700 1 -- 172.21.15.189:0/1863831030 shutdown_connections 2024-04-06T12:12:58.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.843+0000 7f8bbeed9700 1 -- 172.21.15.189:0/1863831030 wait complete. 2024-04-06T12:12:58.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.845+0000 7f8bbeed9700 1 Processor -- start 2024-04-06T12:12:58.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.845+0000 7f8bbeed9700 1 -- start start 2024-04-06T12:12:58.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.845+0000 7f8bbeed9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bb80cb870 0x7f8bb81516c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:58.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.845+0000 7f8bbeed9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8bb80cfa00 0x7f8bb80b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:58.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bbeed9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bb80d11b0 0x7f8bb8157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:58.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bbeed9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb81562b0 con 0x7f8bb80cb870 2024-04-06T12:12:58.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bbeed9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb8156590 con 0x7f8bb80cfa00 2024-04-06T12:12:58.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bbeed9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb8156870 con 0x7f8bb80d11b0 2024-04-06T12:12:58.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bbcc75700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bb80cb870 0x7f8bb81516c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:58.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bbcc75700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bb80cb870 0x7f8bb81516c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57714/0 (socket says 172.21.15.189:57714) 2024-04-06T12:12:58.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bbcc75700 1 -- 172.21.15.189:0/2972541323 learned_addr learned my addr 172.21.15.189:0/2972541323 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:58.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bbd476700 1 --2- 172.21.15.189:0/2972541323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bb80d11b0 0x7f8bb8157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:58.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bb7fff700 1 --2- 172.21.15.189:0/2972541323 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8bb80cfa00 0x7f8bb80b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:58.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bbcc75700 1 -- 172.21.15.189:0/2972541323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bb80d11b0 msgr2=0x7f8bb8157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:58.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bbcc75700 1 --2- 172.21.15.189:0/2972541323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bb80d11b0 0x7f8bb8157350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:58.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bbcc75700 1 -- 172.21.15.189:0/2972541323 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8bb80cfa00 msgr2=0x7f8bb80b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:58.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bbcc75700 1 --2- 172.21.15.189:0/2972541323 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8bb80cfa00 0x7f8bb80b42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:58.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bbcc75700 1 -- 172.21.15.189:0/2972541323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba40094a0 con 0x7f8bb80cb870 2024-04-06T12:12:58.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bbd476700 1 --2- 172.21.15.189:0/2972541323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bb80d11b0 0x7f8bb8157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:58.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bbcc75700 1 --2- 172.21.15.189:0/2972541323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bb80cb870 0x7f8bb81516c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8bac00c6d0 tx=0x7f8bac00ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:58.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bb57fa700 1 -- 172.21.15.189:0/2972541323 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac004c80 con 0x7f8bb80cb870 2024-04-06T12:12:58.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bb57fa700 1 -- 172.21.15.189:0/2972541323 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bac004de0 con 0x7f8bb80cb870 2024-04-06T12:12:58.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bbeed9700 1 -- 172.21.15.189:0/2972541323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb8156b50 con 0x7f8bb80cb870 2024-04-06T12:12:58.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bb57fa700 1 -- 172.21.15.189:0/2972541323 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac007310 con 0x7f8bb80cb870 2024-04-06T12:12:58.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.846+0000 7f8bbeed9700 1 -- 172.21.15.189:0/2972541323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb8151c00 con 0x7f8bb80cb870 2024-04-06T12:12:58.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.847+0000 7f8bbeed9700 1 -- 172.21.15.189:0/2972541323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bb8074990 con 0x7f8bb80cb870 2024-04-06T12:12:58.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.848+0000 7f8bb57fa700 1 -- 172.21.15.189:0/2972541323 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8bac007470 con 0x7f8bb80cb870 2024-04-06T12:12:58.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.848+0000 7f8bb57fa700 1 --2- 172.21.15.189:0/2972541323 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8b9c06ac90 0x7f8b9c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:58.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.848+0000 7f8bb7fff700 1 --2- 172.21.15.189:0/2972541323 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8b9c06ac90 0x7f8b9c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:58.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.848+0000 7f8bb57fa700 1 -- 172.21.15.189:0/2972541323 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8bac0894c0 con 0x7f8bb80cb870 2024-04-06T12:12:58.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.849+0000 7f8bb7fff700 1 --2- 172.21.15.189:0/2972541323 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8b9c06ac90 0x7f8b9c06d150 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8bb80cbcb0 tx=0x7f8ba4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:58.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:58.853+0000 7f8bb57fa700 1 -- 172.21.15.189:0/2972541323 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8bac054370 con 0x7f8bb80cb870 2024-04-06T12:12:59.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.044+0000 7f8bbeed9700 1 -- 172.21.15.189:0/2972541323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8bb80b0a00 con 0x7f8bb80cb870 2024-04-06T12:12:59.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.044+0000 7f8bb57fa700 1 -- 172.21.15.189:0/2972541323 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8bac057990 con 0x7f8bb80cb870 2024-04-06T12:12:59.044 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:12:59.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.047+0000 7f8bbeed9700 1 -- 172.21.15.189:0/2972541323 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8b9c06ac90 msgr2=0x7f8b9c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:59.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.047+0000 7f8bbeed9700 1 --2- 172.21.15.189:0/2972541323 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8b9c06ac90 0x7f8b9c06d150 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8bb80cbcb0 tx=0x7f8ba4008030 comp rx=0 tx=0).stop 2024-04-06T12:12:59.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.047+0000 7f8bbeed9700 1 -- 172.21.15.189:0/2972541323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bb80cb870 msgr2=0x7f8bb81516c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:59.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.047+0000 7f8bbeed9700 1 --2- 172.21.15.189:0/2972541323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bb80cb870 0x7f8bb81516c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8bac00c6d0 tx=0x7f8bac00ca90 comp rx=0 tx=0).stop 2024-04-06T12:12:59.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.047+0000 7f8bbeed9700 1 -- 172.21.15.189:0/2972541323 shutdown_connections 2024-04-06T12:12:59.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.047+0000 7f8bbeed9700 1 --2- 172.21.15.189:0/2972541323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bb80d11b0 0x7f8bb8157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.047+0000 7f8bbeed9700 1 --2- 172.21.15.189:0/2972541323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bb80cb870 0x7f8bb81516c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.047+0000 7f8bbeed9700 1 --2- 172.21.15.189:0/2972541323 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8b9c06ac90 0x7f8b9c06d150 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.047+0000 7f8bbeed9700 1 --2- 172.21.15.189:0/2972541323 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8bb80cfa00 0x7f8bb80b42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.047+0000 7f8bbeed9700 1 -- 172.21.15.189:0/2972541323 >> 172.21.15.189:0/2972541323 conn(0x7f8bb80b2db0 msgr2=0x7f8bb80d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:59.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.047+0000 7f8bbeed9700 1 -- 172.21.15.189:0/2972541323 shutdown_connections 2024-04-06T12:12:59.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.047+0000 7f8bbeed9700 1 -- 172.21.15.189:0/2972541323 wait complete. 2024-04-06T12:12:59.058 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:12:59.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.182+0000 7f5c6be21700 1 Processor -- start 2024-04-06T12:12:59.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.183+0000 7f5c6be21700 1 -- start start 2024-04-06T12:12:59.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.183+0000 7f5c6be21700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5c640b3e20 0x7f5c6415e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.183+0000 7f5c6be21700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c640b3990 con 0x7f5c6406ae40 2024-04-06T12:12:59.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.183+0000 7f5c6be21700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c640b3060 con 0x7f5c6415d7b0 2024-04-06T12:12:59.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.183+0000 7f5c6be21700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c640b4230 con 0x7f5c640b3e20 2024-04-06T12:12:59.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c693bc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5c6406ae40 0x7f5c6415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39180/0 (socket says 172.21.15.189:39180) 2024-04-06T12:12:59.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c693bc700 1 -- 172.21.15.189:0/4056787249 learned_addr learned my addr 172.21.15.189:0/4056787249 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:12:59.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c68bbb700 1 -- 172.21.15.189:0/4056787249 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469399221 0 0) 0x7f5c640b3990 con 0x7f5c6406ae40 2024-04-06T12:12:59.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c6a3be700 1 --2- 172.21.15.189:0/4056787249 >> v2:172.21.15.192:3300/0 conn(0x7f5c640b3e20 0x7f5c6415e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c68bbb700 1 -- 172.21.15.189:0/4056787249 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c500035b0 con 0x7f5c6406ae40 2024-04-06T12:12:59.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c68bbb700 1 -- 172.21.15.189:0/4056787249 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1905694790 0 0) 0x7f5c500035b0 con 0x7f5c6406ae40 2024-04-06T12:12:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c68bbb700 1 -- 172.21.15.189:0/4056787249 >> v2:172.21.15.192:3300/0 conn(0x7f5c640b3e20 msgr2=0x7f5c6415e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c68bbb700 1 --2- 172.21.15.189:0/4056787249 >> v2:172.21.15.192:3300/0 conn(0x7f5c640b3e20 0x7f5c6415e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c68bbb700 1 -- 172.21.15.189:0/4056787249 >> v1:172.21.15.192:6789/0 conn(0x7f5c6415d7b0 legacy=0x7f5c6406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c68bbb700 1 -- 172.21.15.189:0/4056787249 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c640b3340 con 0x7f5c6406ae40 2024-04-06T12:12:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c68bbb700 1 -- 172.21.15.189:0/4056787249 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f5c60001670 con 0x7f5c6406ae40 2024-04-06T12:12:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c68bbb700 1 -- 172.21.15.189:0/4056787249 >> v1:172.21.15.189:6789/0 conn(0x7f5c6406ae40 legacy=0x7f5c6415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c68bbb700 1 --2- 172.21.15.189:0/4056787249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c50003a00 0x7f5c50003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c68bbb700 1 --2- 172.21.15.189:0/4056787249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c50004850 0x7f5c50004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c68bbb700 1 --2- 172.21.15.189:0/4056787249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c50005210 0x7f5c5000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c68bbb700 1 -- 172.21.15.189:0/4056787249 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c640b4230 con 0x7f5c50005210 2024-04-06T12:12:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c68bbb700 1 -- 172.21.15.189:0/4056787249 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5000c1b0 con 0x7f5c50004850 2024-04-06T12:12:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c68bbb700 1 -- 172.21.15.189:0/4056787249 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5000c370 con 0x7f5c50003a00 2024-04-06T12:12:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c6a3be700 1 --2- 172.21.15.189:0/4056787249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c50005210 0x7f5c5000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c693bc700 1 --2- 172.21.15.189:0/4056787249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c50004850 0x7f5c50004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.184+0000 7f5c69bbd700 1 --2- 172.21.15.189:0/4056787249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c50003a00 0x7f5c50003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.185+0000 7f5c6a3be700 1 -- 172.21.15.189:0/4056787249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c50003a00 msgr2=0x7f5c50003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.185+0000 7f5c6a3be700 1 --2- 172.21.15.189:0/4056787249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c50003a00 0x7f5c50003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.185+0000 7f5c6a3be700 1 -- 172.21.15.189:0/4056787249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c50004850 msgr2=0x7f5c50004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.185+0000 7f5c6a3be700 1 --2- 172.21.15.189:0/4056787249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c50004850 0x7f5c50004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.185+0000 7f5c6a3be700 1 -- 172.21.15.189:0/4056787249 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c500035b0 con 0x7f5c50005210 2024-04-06T12:12:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.185+0000 7f5c6a3be700 1 -- 172.21.15.189:0/4056787249 reap_dead start 2024-04-06T12:12:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.185+0000 7f5c6a3be700 1 --2- 172.21.15.189:0/4056787249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c50005210 0x7f5c5000ba10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c009fc0 tx=0x7f5c50005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0014933c8d82ae2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.185+0000 7f5c68bbb700 1 -- 172.21.15.189:0/4056787249 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c003a20 con 0x7f5c50005210 2024-04-06T12:12:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.185+0000 7f5c68bbb700 1 -- 172.21.15.189:0/4056787249 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c5c00f040 con 0x7f5c50005210 2024-04-06T12:12:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.185+0000 7f5c68bbb700 1 -- 172.21.15.189:0/4056787249 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c003de0 con 0x7f5c50005210 2024-04-06T12:12:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.185+0000 7f5c6be21700 1 -- 172.21.15.189:0/4056787249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c50005210 msgr2=0x7f5c5000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:59.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.185+0000 7f5c6be21700 1 --2- 172.21.15.189:0/4056787249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c50005210 0x7f5c5000ba10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c009fc0 tx=0x7f5c50005880 comp rx=0 tx=0).stop 2024-04-06T12:12:59.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.185+0000 7f5c6be21700 1 -- 172.21.15.189:0/4056787249 shutdown_connections 2024-04-06T12:12:59.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.185+0000 7f5c6be21700 1 --2- 172.21.15.189:0/4056787249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c50005210 0x7f5c5000ba10 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.185+0000 7f5c6be21700 1 -- 172.21.15.189:0/4056787249 >> 172.21.15.189:0/4056787249 conn(0x7f5c640d00b0 msgr2=0x7f5c640d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:59.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.185+0000 7f5c6be21700 1 -- 172.21.15.189:0/4056787249 shutdown_connections 2024-04-06T12:12:59.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.186+0000 7f5c6be21700 1 -- 172.21.15.189:0/4056787249 wait complete. 2024-04-06T12:12:59.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.187+0000 7f5c6be21700 1 Processor -- start 2024-04-06T12:12:59.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.187+0000 7f5c6be21700 1 -- start start 2024-04-06T12:12:59.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.188+0000 7f5c6be21700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c50005210 0x7f5c6414ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.188+0000 7f5c6be21700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c640cc260 0x7f5c640ce640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.188+0000 7f5c6be21700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c640cc6e0 0x7f5c640ceb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.188+0000 7f5c6be21700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6415cad0 con 0x7f5c50005210 2024-04-06T12:12:59.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.188+0000 7f5c6be21700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c640cf0c0 con 0x7f5c640cc260 2024-04-06T12:12:59.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.188+0000 7f5c6be21700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6414cfb0 con 0x7f5c640cc6e0 2024-04-06T12:12:59.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.188+0000 7f5c69bbd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c50005210 0x7f5c6414ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.188+0000 7f5c69bbd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c50005210 0x7f5c6414ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57730/0 (socket says 172.21.15.189:57730) 2024-04-06T12:12:59.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.188+0000 7f5c69bbd700 1 -- 172.21.15.189:0/1349673233 learned_addr learned my addr 172.21.15.189:0/1349673233 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:59.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.188+0000 7f5c693bc700 1 --2- 172.21.15.189:0/1349673233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c640cc260 0x7f5c640ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.188+0000 7f5c6a3be700 1 --2- 172.21.15.189:0/1349673233 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c640cc6e0 0x7f5c640ceb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.188+0000 7f5c69bbd700 1 -- 172.21.15.189:0/1349673233 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c640cc6e0 msgr2=0x7f5c640ceb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:59.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.188+0000 7f5c69bbd700 1 --2- 172.21.15.189:0/1349673233 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c640cc6e0 0x7f5c640ceb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.188+0000 7f5c69bbd700 1 -- 172.21.15.189:0/1349673233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c640cc260 msgr2=0x7f5c640ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:59.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.188+0000 7f5c69bbd700 1 --2- 172.21.15.189:0/1349673233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c640cc260 0x7f5c640ce640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.188+0000 7f5c69bbd700 1 -- 172.21.15.189:0/1349673233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6415be50 con 0x7f5c50005210 2024-04-06T12:12:59.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.188+0000 7f5c693bc700 1 --2- 172.21.15.189:0/1349673233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c640cc260 0x7f5c640ce640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:59.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.188+0000 7f5c6a3be700 1 --2- 172.21.15.189:0/1349673233 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c640cc6e0 0x7f5c640ceb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:59.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.189+0000 7f5c69bbd700 1 --2- 172.21.15.189:0/1349673233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c50005210 0x7f5c6414ca70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400f820 tx=0x7f5c5400fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:59.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.189+0000 7f5c5a7fc700 1 -- 172.21.15.189:0/1349673233 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c540052e0 con 0x7f5c50005210 2024-04-06T12:12:59.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.189+0000 7f5c5a7fc700 1 -- 172.21.15.189:0/1349673233 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c54005460 con 0x7f5c50005210 2024-04-06T12:12:59.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.189+0000 7f5c6be21700 1 -- 172.21.15.189:0/1349673233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c6415c130 con 0x7f5c50005210 2024-04-06T12:12:59.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.189+0000 7f5c5a7fc700 1 -- 172.21.15.189:0/1349673233 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c540038c0 con 0x7f5c50005210 2024-04-06T12:12:59.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.189+0000 7f5c6be21700 1 -- 172.21.15.189:0/1349673233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c6415c6f0 con 0x7f5c50005210 2024-04-06T12:12:59.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.190+0000 7f5c6be21700 1 -- 172.21.15.189:0/1349673233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c64067bb0 con 0x7f5c50005210 2024-04-06T12:12:59.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.190+0000 7f5c5a7fc700 1 -- 172.21.15.189:0/1349673233 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5c54003a20 con 0x7f5c50005210 2024-04-06T12:12:59.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.191+0000 7f5c5a7fc700 1 --2- 172.21.15.189:0/1349673233 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c4806ac90 0x7f5c4806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.191+0000 7f5c5a7fc700 1 -- 172.21.15.189:0/1349673233 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5c540894f0 con 0x7f5c50005210 2024-04-06T12:12:59.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.191+0000 7f5c693bc700 1 --2- 172.21.15.189:0/1349673233 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c4806ac90 0x7f5c4806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.191+0000 7f5c693bc700 1 --2- 172.21.15.189:0/1349673233 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c4806ac90 0x7f5c4806d150 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5c60007cf0 tx=0x7f5c60009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:59.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.195+0000 7f5c5a7fc700 1 -- 172.21.15.189:0/1349673233 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5c54054300 con 0x7f5c50005210 2024-04-06T12:12:59.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.387+0000 7f5c6be21700 1 -- 172.21.15.189:0/1349673233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5c640cf7a0 con 0x7f5c50005210 2024-04-06T12:12:59.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.387+0000 7f5c5a7fc700 1 -- 172.21.15.189:0/1349673233 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5c54057920 con 0x7f5c50005210 2024-04-06T12:12:59.386 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:12:59.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.389+0000 7f5c6be21700 1 -- 172.21.15.189:0/1349673233 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c4806ac90 msgr2=0x7f5c4806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:59.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.389+0000 7f5c6be21700 1 --2- 172.21.15.189:0/1349673233 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c4806ac90 0x7f5c4806d150 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5c60007cf0 tx=0x7f5c60009040 comp rx=0 tx=0).stop 2024-04-06T12:12:59.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.389+0000 7f5c6be21700 1 -- 172.21.15.189:0/1349673233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c50005210 msgr2=0x7f5c6414ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:59.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.389+0000 7f5c6be21700 1 --2- 172.21.15.189:0/1349673233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c50005210 0x7f5c6414ca70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400f820 tx=0x7f5c5400fbe0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.389+0000 7f5c6be21700 1 -- 172.21.15.189:0/1349673233 shutdown_connections 2024-04-06T12:12:59.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.389+0000 7f5c6be21700 1 --2- 172.21.15.189:0/1349673233 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c640cc6e0 0x7f5c640ceb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.389+0000 7f5c6be21700 1 --2- 172.21.15.189:0/1349673233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c50005210 0x7f5c6414ca70 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.389+0000 7f5c6be21700 1 --2- 172.21.15.189:0/1349673233 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c4806ac90 0x7f5c4806d150 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.389+0000 7f5c6be21700 1 --2- 172.21.15.189:0/1349673233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c640cc260 0x7f5c640ce640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.389+0000 7f5c6be21700 1 -- 172.21.15.189:0/1349673233 >> 172.21.15.189:0/1349673233 conn(0x7f5c640d00b0 msgr2=0x7f5c6406eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:59.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.389+0000 7f5c6be21700 1 -- 172.21.15.189:0/1349673233 shutdown_connections 2024-04-06T12:12:59.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.389+0000 7f5c6be21700 1 -- 172.21.15.189:0/1349673233 wait complete. 2024-04-06T12:12:59.400 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:12:59.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.526+0000 7fddf6e0d700 1 Processor -- start 2024-04-06T12:12:59.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.526+0000 7fddf6e0d700 1 -- start start 2024-04-06T12:12:59.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.527+0000 7fddf6e0d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fddf00b4660 0x7fddf00b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.527+0000 7fddf6e0d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddf00b2fc0 0x7fddf00b3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.528+0000 7fddf6e0d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf00b5030 con 0x7fddf015a550 2024-04-06T12:12:59.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.528+0000 7fddf6e0d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf00cdbe0 con 0x7fddf00b4660 2024-04-06T12:12:59.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.528+0000 7fddf6e0d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf015a940 con 0x7fddf00b2fc0 2024-04-06T12:12:59.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.528+0000 7fddeffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fddf00b4660 0x7fddf00b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.528+0000 7fddeffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fddf00b4660 0x7fddf00b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44646/0 (socket says 172.21.15.189:44646) 2024-04-06T12:12:59.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.528+0000 7fddeffff700 1 -- 172.21.15.189:0/1840950096 learned_addr learned my addr 172.21.15.189:0/1840950096 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:59.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.528+0000 7fddf53aa700 1 --2- 172.21.15.189:0/1840950096 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddf00b2fc0 0x7fddf00b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.528+0000 7fddeffff700 1 -- 172.21.15.189:0/1840950096 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddf00b2fc0 msgr2=0x7fddf00b3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:59.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.528+0000 7fddeffff700 1 --2- 172.21.15.189:0/1840950096 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddf00b2fc0 0x7fddf00b3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.528+0000 7fddeffff700 1 -- 172.21.15.189:0/1840950096 >> v1:172.21.15.192:6789/0 conn(0x7fddf015a550 legacy=0x7fddf00cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:59.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.528+0000 7fddeffff700 1 -- 172.21.15.189:0/1840950096 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf015ac00 con 0x7fddf00b4660 2024-04-06T12:12:59.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.528+0000 7fddf53aa700 1 --2- 172.21.15.189:0/1840950096 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddf00b2fc0 0x7fddf00b3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:59.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.528+0000 7fddeffff700 1 --2- 172.21.15.189:0/1840950096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf00b4660 0x7fddf00b4af0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fddd80098f0 tx=0x7fddd8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=261520ac51d42014 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:59.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.528+0000 7fddef7fe700 1 -- 172.21.15.189:0/1840950096 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd800b8a0 con 0x7fddf00b4660 2024-04-06T12:12:59.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.528+0000 7fddef7fe700 1 -- 172.21.15.189:0/1840950096 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd800ba00 con 0x7fddf00b4660 2024-04-06T12:12:59.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.528+0000 7fddef7fe700 1 -- 172.21.15.189:0/1840950096 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd800bd90 con 0x7fddf00b4660 2024-04-06T12:12:59.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.529+0000 7fddf6e0d700 1 -- 172.21.15.189:0/1840950096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf00b4660 msgr2=0x7fddf00b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:59.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.529+0000 7fddf6e0d700 1 --2- 172.21.15.189:0/1840950096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf00b4660 0x7fddf00b4af0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fddd80098f0 tx=0x7fddd8009c00 comp rx=0 tx=0).stop 2024-04-06T12:12:59.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.529+0000 7fddf6e0d700 1 -- 172.21.15.189:0/1840950096 shutdown_connections 2024-04-06T12:12:59.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.529+0000 7fddf6e0d700 1 --2- 172.21.15.189:0/1840950096 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddf00b2fc0 0x7fddf00b3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.529+0000 7fddf6e0d700 1 --2- 172.21.15.189:0/1840950096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf00b4660 0x7fddf00b4af0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.529+0000 7fddf6e0d700 1 -- 172.21.15.189:0/1840950096 >> 172.21.15.189:0/1840950096 conn(0x7fddf00d00b0 msgr2=0x7fddf00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:59.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.529+0000 7fddf6e0d700 1 -- 172.21.15.189:0/1840950096 shutdown_connections 2024-04-06T12:12:59.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.529+0000 7fddf6e0d700 1 -- 172.21.15.189:0/1840950096 wait complete. 2024-04-06T12:12:59.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.530+0000 7fddf6e0d700 1 Processor -- start 2024-04-06T12:12:59.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.531+0000 7fddf6e0d700 1 -- start start 2024-04-06T12:12:59.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fddf6e0d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddf00b2fc0 0x7fddf0154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fddf6e0d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddf00b4660 0x7fddf0154c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fddf6e0d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf006d610 0x7fddf006dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fddf6e0d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf006e000 con 0x7fddf006d610 2024-04-06T12:12:59.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fddf6e0d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf0154ac0 con 0x7fddf00b4660 2024-04-06T12:12:59.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fddf6e0d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf015d140 con 0x7fddf00b2fc0 2024-04-06T12:12:59.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fddf53aa700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf006d610 0x7fddf006dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fddf53aa700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf006d610 0x7fddf006dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44652/0 (socket says 172.21.15.189:44652) 2024-04-06T12:12:59.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fddf53aa700 1 -- 172.21.15.189:0/1931501376 learned_addr learned my addr 172.21.15.189:0/1931501376 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:59.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fddeffff700 1 --2- 172.21.15.189:0/1931501376 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddf00b4660 0x7fddf0154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fddf4ba9700 1 --2- 172.21.15.189:0/1931501376 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddf00b2fc0 0x7fddf0154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fddf53aa700 1 -- 172.21.15.189:0/1931501376 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddf00b2fc0 msgr2=0x7fddf0154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:59.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fddf53aa700 1 --2- 172.21.15.189:0/1931501376 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddf00b2fc0 0x7fddf0154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fddf53aa700 1 -- 172.21.15.189:0/1931501376 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddf00b4660 msgr2=0x7fddf0154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:59.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fddf53aa700 1 --2- 172.21.15.189:0/1931501376 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddf00b4660 0x7fddf0154c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fddf53aa700 1 -- 172.21.15.189:0/1931501376 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd8009550 con 0x7fddf006d610 2024-04-06T12:12:59.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fddf4ba9700 1 --2- 172.21.15.189:0/1931501376 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddf00b2fc0 0x7fddf0154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:12:59.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fddf53aa700 1 --2- 172.21.15.189:0/1931501376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf006d610 0x7fddf006dac0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdde000c7f0 tx=0x7fdde000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:59.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fdded7fa700 1 -- 172.21.15.189:0/1931501376 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde0011800 con 0x7fddf006d610 2024-04-06T12:12:59.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fdded7fa700 1 -- 172.21.15.189:0/1931501376 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde0011980 con 0x7fddf006d610 2024-04-06T12:12:59.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fddf6e0d700 1 -- 172.21.15.189:0/1931501376 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf01f16f0 con 0x7fddf006d610 2024-04-06T12:12:59.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.532+0000 7fdded7fa700 1 -- 172.21.15.189:0/1931501376 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde0011d30 con 0x7fddf006d610 2024-04-06T12:12:59.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.533+0000 7fddf6e0d700 1 -- 172.21.15.189:0/1931501376 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf00aba90 con 0x7fddf006d610 2024-04-06T12:12:59.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.534+0000 7fddf6e0d700 1 -- 172.21.15.189:0/1931501376 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf0067bb0 con 0x7fddf006d610 2024-04-06T12:12:59.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.534+0000 7fdded7fa700 1 -- 172.21.15.189:0/1931501376 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdde0022350 con 0x7fddf006d610 2024-04-06T12:12:59.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.534+0000 7fdded7fa700 1 --2- 172.21.15.189:0/1931501376 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fddd006ac90 0x7fddd006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.534+0000 7fdded7fa700 1 -- 172.21.15.189:0/1931501376 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fdde00895f0 con 0x7fddf006d610 2024-04-06T12:12:59.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.534+0000 7fddf4ba9700 1 --2- 172.21.15.189:0/1931501376 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fddd006ac90 0x7fddd006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.535+0000 7fddf4ba9700 1 --2- 172.21.15.189:0/1931501376 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fddd006ac90 0x7fddd006d150 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fddf006b770 tx=0x7fdde8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:59.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.539+0000 7fdded7fa700 1 -- 172.21.15.189:0/1931501376 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdde0054420 con 0x7fddf006d610 2024-04-06T12:12:59.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.734+0000 7fddf6e0d700 1 -- 172.21.15.189:0/1931501376 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fddf0074990 con 0x7fddf006d610 2024-04-06T12:12:59.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.735+0000 7fdded7fa700 1 -- 172.21.15.189:0/1931501376 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdde0057a40 con 0x7fddf006d610 2024-04-06T12:12:59.734 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:12:59.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.737+0000 7fddf6e0d700 1 -- 172.21.15.189:0/1931501376 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fddd006ac90 msgr2=0x7fddd006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:59.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.737+0000 7fddf6e0d700 1 --2- 172.21.15.189:0/1931501376 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fddd006ac90 0x7fddd006d150 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fddf006b770 tx=0x7fdde8009040 comp rx=0 tx=0).stop 2024-04-06T12:12:59.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.737+0000 7fddf6e0d700 1 -- 172.21.15.189:0/1931501376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf006d610 msgr2=0x7fddf006dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:59.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.737+0000 7fddf6e0d700 1 --2- 172.21.15.189:0/1931501376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf006d610 0x7fddf006dac0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdde000c7f0 tx=0x7fdde000cb00 comp rx=0 tx=0).stop 2024-04-06T12:12:59.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.737+0000 7fddf6e0d700 1 -- 172.21.15.189:0/1931501376 shutdown_connections 2024-04-06T12:12:59.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.737+0000 7fddf6e0d700 1 --2- 172.21.15.189:0/1931501376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf006d610 0x7fddf006dac0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.737+0000 7fddf6e0d700 1 --2- 172.21.15.189:0/1931501376 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddf00b2fc0 0x7fddf0154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.737+0000 7fddf6e0d700 1 --2- 172.21.15.189:0/1931501376 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fddd006ac90 0x7fddd006d150 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.737+0000 7fddf6e0d700 1 --2- 172.21.15.189:0/1931501376 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddf00b4660 0x7fddf0154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.737+0000 7fddf6e0d700 1 -- 172.21.15.189:0/1931501376 >> 172.21.15.189:0/1931501376 conn(0x7fddf00d00b0 msgr2=0x7fddf0156ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:59.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.737+0000 7fddf6e0d700 1 -- 172.21.15.189:0/1931501376 shutdown_connections 2024-04-06T12:12:59.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.737+0000 7fddf6e0d700 1 -- 172.21.15.189:0/1931501376 wait complete. 2024-04-06T12:12:59.748 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-06T12:12:59.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.875+0000 7f43a40b6700 1 Processor -- start 2024-04-06T12:12:59.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.875+0000 7f43a40b6700 1 -- start start 2024-04-06T12:12:59.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.876+0000 7f43a40b6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f439c06ae40 0x7f439c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.876+0000 7f43a40b6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f439c15ddf0 0x7f439c15e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.876+0000 7f43a40b6700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f439c0b3990 con 0x7f439c15d7b0 2024-04-06T12:12:59.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.876+0000 7f43a40b6700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f439c0b3060 con 0x7f439c15ddf0 2024-04-06T12:12:59.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.876+0000 7f43a40b6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f439c0b3340 con 0x7f439c06ae40 2024-04-06T12:12:59.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.876+0000 7f43a2653700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f439c15ddf0 0x7f439c15e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.876+0000 7f43a2653700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f439c15ddf0 0x7f439c15e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44664/0 (socket says 172.21.15.189:44664) 2024-04-06T12:12:59.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.876+0000 7f43a2653700 1 -- 172.21.15.189:0/3919661372 learned_addr learned my addr 172.21.15.189:0/3919661372 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:59.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.876+0000 7f43a1651700 1 --2- 172.21.15.189:0/3919661372 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f439c06ae40 0x7f439c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.876+0000 7f43a2653700 1 -- 172.21.15.189:0/3919661372 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f439c06ae40 msgr2=0x7f439c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:12:59.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.876+0000 7f43a2653700 1 --2- 172.21.15.189:0/3919661372 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f439c06ae40 0x7f439c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.876+0000 7f43a2653700 1 -- 172.21.15.189:0/3919661372 >> v1:172.21.15.192:6789/0 conn(0x7f439c15d7b0 legacy=0x7f439c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:59.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.876+0000 7f43a2653700 1 -- 172.21.15.189:0/3919661372 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439c0b3620 con 0x7f439c15ddf0 2024-04-06T12:12:59.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.876+0000 7f43a2653700 1 --2- 172.21.15.189:0/3919661372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f439c15ddf0 0x7f439c15e2a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4394009e50 tx=0x7f439400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f1b5423b9f58270 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:59.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.876+0000 7f43a0e50700 1 -- 172.21.15.189:0/3919661372 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439400e070 con 0x7f439c15ddf0 2024-04-06T12:12:59.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.876+0000 7f43a0e50700 1 -- 172.21.15.189:0/3919661372 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439400b9c0 con 0x7f439c15ddf0 2024-04-06T12:12:59.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.876+0000 7f43a0e50700 1 -- 172.21.15.189:0/3919661372 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439400bd70 con 0x7f439c15ddf0 2024-04-06T12:12:59.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.877+0000 7f43a40b6700 1 -- 172.21.15.189:0/3919661372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f439c15ddf0 msgr2=0x7f439c15e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:59.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.877+0000 7f43a40b6700 1 --2- 172.21.15.189:0/3919661372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f439c15ddf0 0x7f439c15e2a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4394009e50 tx=0x7f439400b330 comp rx=0 tx=0).stop 2024-04-06T12:12:59.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.877+0000 7f43a40b6700 1 -- 172.21.15.189:0/3919661372 shutdown_connections 2024-04-06T12:12:59.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.877+0000 7f43a40b6700 1 --2- 172.21.15.189:0/3919661372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f439c15ddf0 0x7f439c15e2a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.877+0000 7f43a40b6700 1 --2- 172.21.15.189:0/3919661372 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f439c06ae40 0x7f439c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.877+0000 7f43a40b6700 1 -- 172.21.15.189:0/3919661372 >> 172.21.15.189:0/3919661372 conn(0x7f439c0d00b0 msgr2=0x7f439c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:12:59.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.877+0000 7f43a40b6700 1 -- 172.21.15.189:0/3919661372 shutdown_connections 2024-04-06T12:12:59.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.877+0000 7f43a40b6700 1 -- 172.21.15.189:0/3919661372 wait complete. 2024-04-06T12:12:59.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.878+0000 7f43a40b6700 1 Processor -- start 2024-04-06T12:12:59.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.878+0000 7f43a40b6700 1 -- start start 2024-04-06T12:12:59.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.879+0000 7f43a40b6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f439c06ae40 0x7f439c0cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.879+0000 7f43a40b6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f439c15d7b0 0x7f439c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.879+0000 7f43a40b6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f439c15ddf0 0x7f439c158b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.879+0000 7f43a40b6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c0b0120 con 0x7f439c15ddf0 2024-04-06T12:12:59.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.879+0000 7f43a40b6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c0cc550 con 0x7f439c06ae40 2024-04-06T12:12:59.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.879+0000 7f43a40b6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f439c0cc830 con 0x7f439c15d7b0 2024-04-06T12:12:59.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.879+0000 7f43a2653700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f439c15ddf0 0x7f439c158b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.879+0000 7f43a2653700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f439c15ddf0 0x7f439c158b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44674/0 (socket says 172.21.15.189:44674) 2024-04-06T12:12:59.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.879+0000 7f43a2653700 1 -- 172.21.15.189:0/1280434419 learned_addr learned my addr 172.21.15.189:0/1280434419 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:12:59.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.879+0000 7f43a1e52700 1 --2- 172.21.15.189:0/1280434419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f439c06ae40 0x7f439c0cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.879+0000 7f43a1651700 1 --2- 172.21.15.189:0/1280434419 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f439c15d7b0 0x7f439c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.879+0000 7f43a2653700 1 -- 172.21.15.189:0/1280434419 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f439c15d7b0 msgr2=0x7f439c0af8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:59.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.879+0000 7f43a2653700 1 --2- 172.21.15.189:0/1280434419 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f439c15d7b0 0x7f439c0af8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.879+0000 7f43a2653700 1 -- 172.21.15.189:0/1280434419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f439c06ae40 msgr2=0x7f439c0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:12:59.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.879+0000 7f43a2653700 1 --2- 172.21.15.189:0/1280434419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f439c06ae40 0x7f439c0cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:12:59.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.879+0000 7f43a2653700 1 -- 172.21.15.189:0/1280434419 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4394009b30 con 0x7f439c15ddf0 2024-04-06T12:12:59.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.879+0000 7f43a1651700 1 --2- 172.21.15.189:0/1280434419 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f439c15d7b0 0x7f439c0af8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:59.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.879+0000 7f43a1e52700 1 --2- 172.21.15.189:0/1280434419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f439c06ae40 0x7f439c0cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:12:59.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.880+0000 7f43a2653700 1 --2- 172.21.15.189:0/1280434419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f439c15ddf0 0x7f439c158b70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f439c06c820 tx=0x7f4394004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:59.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.880+0000 7f43927fc700 1 -- 172.21.15.189:0/1280434419 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43940099b0 con 0x7f439c15ddf0 2024-04-06T12:12:59.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.880+0000 7f43927fc700 1 -- 172.21.15.189:0/1280434419 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43940092e0 con 0x7f439c15ddf0 2024-04-06T12:12:59.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.880+0000 7f43a40b6700 1 -- 172.21.15.189:0/1280434419 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f439c0ccb10 con 0x7f439c15ddf0 2024-04-06T12:12:59.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.880+0000 7f43927fc700 1 -- 172.21.15.189:0/1280434419 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439401c640 con 0x7f439c15ddf0 2024-04-06T12:12:59.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.880+0000 7f43a40b6700 1 -- 172.21.15.189:0/1280434419 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f439c0b0400 con 0x7f439c15ddf0 2024-04-06T12:12:59.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.881+0000 7f43a40b6700 1 -- 172.21.15.189:0/1280434419 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f439c067bb0 con 0x7f439c15ddf0 2024-04-06T12:12:59.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.881+0000 7f43927fc700 1 -- 172.21.15.189:0/1280434419 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f439401c7a0 con 0x7f439c15ddf0 2024-04-06T12:12:59.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.881+0000 7f43927fc700 1 --2- 172.21.15.189:0/1280434419 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f438006ac90 0x7f438006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:12:59.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.882+0000 7f43927fc700 1 -- 172.21.15.189:0/1280434419 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f439408b360 con 0x7f439c15ddf0 2024-04-06T12:12:59.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.882+0000 7f43a1e52700 1 --2- 172.21.15.189:0/1280434419 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f438006ac90 0x7f438006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:12:59.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.882+0000 7f43a1e52700 1 --2- 172.21.15.189:0/1280434419 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f438006ac90 0x7f438006d150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4398000f60 tx=0x7f4398009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:12:59.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:12:59.886+0000 7f43927fc700 1 -- 172.21.15.189:0/1280434419 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4394056210 con 0x7f439c15ddf0 2024-04-06T12:13:00.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.080+0000 7f43a40b6700 1 -- 172.21.15.189:0/1280434419 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f439c0ccd00 con 0x7f439c15ddf0 2024-04-06T12:13:00.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.080+0000 7f43927fc700 1 -- 172.21.15.189:0/1280434419 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4394059830 con 0x7f439c15ddf0 2024-04-06T12:13:00.079 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:13:00.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.082+0000 7f43a40b6700 1 -- 172.21.15.189:0/1280434419 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f438006ac90 msgr2=0x7f438006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:00.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.082+0000 7f43a40b6700 1 --2- 172.21.15.189:0/1280434419 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f438006ac90 0x7f438006d150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4398000f60 tx=0x7f4398009040 comp rx=0 tx=0).stop 2024-04-06T12:13:00.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.082+0000 7f43a40b6700 1 -- 172.21.15.189:0/1280434419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f439c15ddf0 msgr2=0x7f439c158b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:00.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.082+0000 7f43a40b6700 1 --2- 172.21.15.189:0/1280434419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f439c15ddf0 0x7f439c158b70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f439c06c820 tx=0x7f4394004ad0 comp rx=0 tx=0).stop 2024-04-06T12:13:00.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.082+0000 7f43a40b6700 1 -- 172.21.15.189:0/1280434419 shutdown_connections 2024-04-06T12:13:00.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.082+0000 7f43a40b6700 1 --2- 172.21.15.189:0/1280434419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f439c15ddf0 0x7f439c158b70 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:00.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.082+0000 7f43a40b6700 1 --2- 172.21.15.189:0/1280434419 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f438006ac90 0x7f438006d150 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:00.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.082+0000 7f43a40b6700 1 --2- 172.21.15.189:0/1280434419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f439c06ae40 0x7f439c0cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:00.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.082+0000 7f43a40b6700 1 --2- 172.21.15.189:0/1280434419 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f439c15d7b0 0x7f439c0af8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:00.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.082+0000 7f43a40b6700 1 -- 172.21.15.189:0/1280434419 >> 172.21.15.189:0/1280434419 conn(0x7f439c0d00b0 msgr2=0x7f439c15f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:00.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.082+0000 7f43a40b6700 1 -- 172.21.15.189:0/1280434419 shutdown_connections 2024-04-06T12:13:00.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.082+0000 7f43a40b6700 1 -- 172.21.15.189:0/1280434419 wait complete. 2024-04-06T12:13:00.094 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T12:13:00.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.218+0000 7fcc6ef0a700 1 Processor -- start 2024-04-06T12:13:00.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.218+0000 7fcc6ef0a700 1 -- start start 2024-04-06T12:13:00.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.219+0000 7fcc6ef0a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcc680d11b0 0x7fcc680d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:00.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.219+0000 7fcc6ef0a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc680d1ad0 con 0x7fcc680cb870 2024-04-06T12:13:00.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.219+0000 7fcc6ef0a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc680cbd80 con 0x7fcc680cfa00 2024-04-06T12:13:00.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.219+0000 7fcc6ef0a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc680ccdc0 con 0x7fcc680d11b0 2024-04-06T12:13:00.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.219+0000 7fcc67fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcc680cb870 0x7fcc6815c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42624/0 (socket says 172.21.15.189:42624) 2024-04-06T12:13:00.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.219+0000 7fcc67fff700 1 -- 172.21.15.189:0/2977555293 learned_addr learned my addr 172.21.15.189:0/2977555293 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:00.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.219+0000 7fcc6cca6700 1 --2- 172.21.15.189:0/2977555293 >> v2:172.21.15.192:3300/0 conn(0x7fcc680d11b0 0x7fcc680d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:00.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.220+0000 7fcc677fe700 1 -- 172.21.15.189:0/2977555293 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670852726 0 0) 0x7fcc680d1ad0 con 0x7fcc680cb870 2024-04-06T12:13:00.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.220+0000 7fcc677fe700 1 -- 172.21.15.189:0/2977555293 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc580035b0 con 0x7fcc680cb870 2024-04-06T12:13:00.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.220+0000 7fcc6cca6700 1 -- 172.21.15.189:0/2977555293 >> v1:172.21.15.192:6789/0 conn(0x7fcc680cfa00 legacy=0x7fcc680cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:00.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.220+0000 7fcc6cca6700 1 -- 172.21.15.189:0/2977555293 >> v1:172.21.15.189:6789/0 conn(0x7fcc680cb870 legacy=0x7fcc6815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:00.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.220+0000 7fcc6cca6700 1 -- 172.21.15.189:0/2977555293 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc680cef20 con 0x7fcc680d11b0 2024-04-06T12:13:00.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.220+0000 7fcc6cca6700 1 --2- 172.21.15.189:0/2977555293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc680d11b0 0x7fcc680d1590 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c009840 tx=0x7fcc5c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12d3db9ada9e20c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:00.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.220+0000 7fcc677fe700 1 -- 172.21.15.189:0/2977555293 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c011830 con 0x7fcc680d11b0 2024-04-06T12:13:00.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.220+0000 7fcc677fe700 1 -- 172.21.15.189:0/2977555293 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5c011990 con 0x7fcc680d11b0 2024-04-06T12:13:00.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.220+0000 7fcc677fe700 1 -- 172.21.15.189:0/2977555293 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c011d20 con 0x7fcc680d11b0 2024-04-06T12:13:00.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.221+0000 7fcc6ef0a700 1 -- 172.21.15.189:0/2977555293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc680d11b0 msgr2=0x7fcc680d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:00.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.221+0000 7fcc6ef0a700 1 --2- 172.21.15.189:0/2977555293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc680d11b0 0x7fcc680d1590 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c009840 tx=0x7fcc5c009b50 comp rx=0 tx=0).stop 2024-04-06T12:13:00.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.221+0000 7fcc6ef0a700 1 -- 172.21.15.189:0/2977555293 shutdown_connections 2024-04-06T12:13:00.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.221+0000 7fcc6ef0a700 1 --2- 172.21.15.189:0/2977555293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc680d11b0 0x7fcc680d1590 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:00.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.221+0000 7fcc6ef0a700 1 -- 172.21.15.189:0/2977555293 >> 172.21.15.189:0/2977555293 conn(0x7fcc680b2db0 msgr2=0x7fcc680b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:00.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.221+0000 7fcc6ef0a700 1 -- 172.21.15.189:0/2977555293 shutdown_connections 2024-04-06T12:13:00.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.221+0000 7fcc6ef0a700 1 -- 172.21.15.189:0/2977555293 wait complete. 2024-04-06T12:13:00.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.222+0000 7fcc6ef0a700 1 Processor -- start 2024-04-06T12:13:00.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.222+0000 7fcc6ef0a700 1 -- start start 2024-04-06T12:13:00.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.223+0000 7fcc6ef0a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc680cb870 0x7fcc68155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:00.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.223+0000 7fcc6ef0a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc680cfa00 0x7fcc680c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:00.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.223+0000 7fcc6ef0a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcc680d11b0 0x7fcc680c5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:00.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.223+0000 7fcc6ef0a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc68158f30 con 0x7fcc680cfa00 2024-04-06T12:13:00.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.223+0000 7fcc6ef0a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc681583b0 con 0x7fcc680cb870 2024-04-06T12:13:00.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.223+0000 7fcc6ef0a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc68158690 con 0x7fcc680d11b0 2024-04-06T12:13:00.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.223+0000 7fcc67fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc680cfa00 0x7fcc680c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:00.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.223+0000 7fcc67fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc680cfa00 0x7fcc680c5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44686/0 (socket says 172.21.15.189:44686) 2024-04-06T12:13:00.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.223+0000 7fcc67fff700 1 -- 172.21.15.189:0/3073277625 learned_addr learned my addr 172.21.15.189:0/3073277625 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:00.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.223+0000 7fcc6d4a7700 1 --2- 172.21.15.189:0/3073277625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcc680d11b0 0x7fcc680c5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:00.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.223+0000 7fcc6cca6700 1 --2- 172.21.15.189:0/3073277625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc680cb870 0x7fcc68155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:00.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.223+0000 7fcc67fff700 1 -- 172.21.15.189:0/3073277625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcc680d11b0 msgr2=0x7fcc680c5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:00.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.223+0000 7fcc67fff700 1 --2- 172.21.15.189:0/3073277625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcc680d11b0 0x7fcc680c5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:00.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.223+0000 7fcc67fff700 1 -- 172.21.15.189:0/3073277625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc680cb870 msgr2=0x7fcc68155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:00.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.223+0000 7fcc67fff700 1 --2- 172.21.15.189:0/3073277625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc680cb870 0x7fcc68155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:00.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.223+0000 7fcc67fff700 1 -- 172.21.15.189:0/3073277625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5c0094a0 con 0x7fcc680cfa00 2024-04-06T12:13:00.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.224+0000 7fcc67fff700 1 --2- 172.21.15.189:0/3073277625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc680cfa00 0x7fcc680c5ab0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fcc54005910 tx=0x7fcc5400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:00.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.224+0000 7fcc657fa700 1 -- 172.21.15.189:0/3073277625 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc54005d70 con 0x7fcc680cfa00 2024-04-06T12:13:00.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.224+0000 7fcc657fa700 1 -- 172.21.15.189:0/3073277625 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc540070d0 con 0x7fcc680cfa00 2024-04-06T12:13:00.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.224+0000 7fcc6ef0a700 1 -- 172.21.15.189:0/3073277625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc68158970 con 0x7fcc680cfa00 2024-04-06T12:13:00.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.224+0000 7fcc657fa700 1 -- 172.21.15.189:0/3073277625 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc54007460 con 0x7fcc680cfa00 2024-04-06T12:13:00.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.224+0000 7fcc6ef0a700 1 -- 172.21.15.189:0/3073277625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc68155bd0 con 0x7fcc680cfa00 2024-04-06T12:13:00.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.225+0000 7fcc6ef0a700 1 -- 172.21.15.189:0/3073277625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc68074990 con 0x7fcc680cfa00 2024-04-06T12:13:00.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.225+0000 7fcc657fa700 1 -- 172.21.15.189:0/3073277625 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcc540075c0 con 0x7fcc680cfa00 2024-04-06T12:13:00.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.226+0000 7fcc657fa700 1 --2- 172.21.15.189:0/3073277625 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcc4c06ac90 0x7fcc4c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:00.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.226+0000 7fcc657fa700 1 -- 172.21.15.189:0/3073277625 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fcc54088b50 con 0x7fcc680cfa00 2024-04-06T12:13:00.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.226+0000 7fcc6cca6700 1 --2- 172.21.15.189:0/3073277625 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcc4c06ac90 0x7fcc4c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:00.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.226+0000 7fcc6cca6700 1 --2- 172.21.15.189:0/3073277625 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcc4c06ac90 0x7fcc4c06d150 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c0090b0 tx=0x7fcc5c01c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:00.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.230+0000 7fcc657fa700 1 -- 172.21.15.189:0/3073277625 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fcc54053970 con 0x7fcc680cfa00 2024-04-06T12:13:00.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.443+0000 7fcc6ef0a700 1 -- 172.21.15.189:0/3073277625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fcc680b0a00 con 0x7fcc680cfa00 2024-04-06T12:13:00.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.444+0000 7fcc657fa700 1 -- 172.21.15.189:0/3073277625 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7fcc54017080 con 0x7fcc680cfa00 2024-04-06T12:13:00.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.446+0000 7fcc6ef0a700 1 -- 172.21.15.189:0/3073277625 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcc4c06ac90 msgr2=0x7fcc4c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:00.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.446+0000 7fcc6ef0a700 1 --2- 172.21.15.189:0/3073277625 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcc4c06ac90 0x7fcc4c06d150 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c0090b0 tx=0x7fcc5c01c040 comp rx=0 tx=0).stop 2024-04-06T12:13:00.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.446+0000 7fcc6ef0a700 1 -- 172.21.15.189:0/3073277625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc680cfa00 msgr2=0x7fcc680c5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:00.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.446+0000 7fcc6ef0a700 1 --2- 172.21.15.189:0/3073277625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc680cfa00 0x7fcc680c5ab0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fcc54005910 tx=0x7fcc5400c9e0 comp rx=0 tx=0).stop 2024-04-06T12:13:00.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.446+0000 7fcc6ef0a700 1 -- 172.21.15.189:0/3073277625 shutdown_connections 2024-04-06T12:13:00.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.446+0000 7fcc6ef0a700 1 --2- 172.21.15.189:0/3073277625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcc680d11b0 0x7fcc680c5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:00.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.446+0000 7fcc6ef0a700 1 --2- 172.21.15.189:0/3073277625 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcc4c06ac90 0x7fcc4c06d150 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:00.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.446+0000 7fcc6ef0a700 1 --2- 172.21.15.189:0/3073277625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc680cb870 0x7fcc68155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:00.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.446+0000 7fcc6ef0a700 1 --2- 172.21.15.189:0/3073277625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc680cfa00 0x7fcc680c5ab0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:00.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.446+0000 7fcc6ef0a700 1 -- 172.21.15.189:0/3073277625 >> 172.21.15.189:0/3073277625 conn(0x7fcc680b2db0 msgr2=0x7fcc6806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:00.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.446+0000 7fcc6ef0a700 1 -- 172.21.15.189:0/3073277625 shutdown_connections 2024-04-06T12:13:00.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.446+0000 7fcc6ef0a700 1 -- 172.21.15.189:0/3073277625 wait complete. 2024-04-06T12:13:00.447 INFO:teuthology.orchestra.run.smithi189.stderr:listed 0 entries 2024-04-06T12:13:00.457 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-06T12:13:00.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:00.458 DEBUG:teuthology.orchestra.run.smithi192:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-04-06T12:13:00.499 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T12:13:00.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:00.499 DEBUG:teuthology.orchestra.run.smithi192:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-06T12:13:00.589 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:13:00.589 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:13:00.607 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:13:00.607 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:13:00.665 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:13:00.665 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:13:00.726 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:13:00.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.888+0000 7f4e0eb0b700 1 Processor -- start 2024-04-06T12:13:00.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.888+0000 7f4e0eb0b700 1 -- start start 2024-04-06T12:13:00.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.889+0000 7f4e0eb0b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4e0815a550 0x7f4e080cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.889+0000 7f4e0eb0b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4e080b4d70 0x7f4e080b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.889+0000 7f4e0eb0b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e080cdbe0 con 0x7f4e080cf900 2024-04-06T12:13:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.889+0000 7f4e0eb0b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e0815a940 con 0x7f4e0815a550 2024-04-06T12:13:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.889+0000 7f4e0eb0b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e0815ac00 con 0x7f4e080b4d70 2024-04-06T12:13:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.889+0000 7f4e0c8a7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4e0815a550 0x7f4e080cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.889+0000 7f4e0c8a7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4e0815a550 0x7f4e080cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44688/0 (socket says 172.21.15.189:44688) 2024-04-06T12:13:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.889+0000 7f4e0c8a7700 1 -- 172.21.15.189:0/3666344745 learned_addr learned my addr 172.21.15.189:0/3666344745 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.889+0000 7f4e0d0a8700 1 --2- 172.21.15.189:0/3666344745 >> v2:172.21.15.192:3300/0 conn(0x7f4e080b4d70 0x7f4e080b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.889+0000 7f4e0c8a7700 1 -- 172.21.15.189:0/3666344745 >> v2:172.21.15.192:3300/0 conn(0x7f4e080b4d70 msgr2=0x7f4e080b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.889+0000 7f4e0c8a7700 1 --2- 172.21.15.189:0/3666344745 >> v2:172.21.15.192:3300/0 conn(0x7f4e080b4d70 0x7f4e080b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.889+0000 7f4e0c8a7700 1 -- 172.21.15.189:0/3666344745 >> v1:172.21.15.192:6789/0 conn(0x7f4e080cf900 legacy=0x7f4e080b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.889+0000 7f4e0c8a7700 1 -- 172.21.15.189:0/3666344745 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0815aee0 con 0x7f4e0815a550 2024-04-06T12:13:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.889+0000 7f4e0d0a8700 1 --2- 172.21.15.189:0/3666344745 >> v2:172.21.15.192:3300/0 conn(0x7f4e080b4d70 0x7f4e080b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.890+0000 7f4e0c8a7700 1 --2- 172.21.15.189:0/3666344745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e0815a550 0x7f4e080cf3c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4e08072830 tx=0x7f4dfc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d8b50c5149b2c05 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.890+0000 7f4e077fe700 1 -- 172.21.15.189:0/3666344745 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc004230 con 0x7f4e0815a550 2024-04-06T12:13:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.890+0000 7f4e077fe700 1 -- 172.21.15.189:0/3666344745 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dfc004390 con 0x7f4e0815a550 2024-04-06T12:13:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.890+0000 7f4e077fe700 1 -- 172.21.15.189:0/3666344745 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc0036a0 con 0x7f4e0815a550 2024-04-06T12:13:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.890+0000 7f4e0eb0b700 1 -- 172.21.15.189:0/3666344745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e0815a550 msgr2=0x7f4e080cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.890+0000 7f4e0eb0b700 1 --2- 172.21.15.189:0/3666344745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e0815a550 0x7f4e080cf3c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4e08072830 tx=0x7f4dfc009bd0 comp rx=0 tx=0).stop 2024-04-06T12:13:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.890+0000 7f4e0eb0b700 1 -- 172.21.15.189:0/3666344745 shutdown_connections 2024-04-06T12:13:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.890+0000 7f4e0eb0b700 1 --2- 172.21.15.189:0/3666344745 >> v2:172.21.15.192:3300/0 conn(0x7f4e080b4d70 0x7f4e080b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.890+0000 7f4e0eb0b700 1 --2- 172.21.15.189:0/3666344745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e0815a550 0x7f4e080cf3c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.890+0000 7f4e0eb0b700 1 -- 172.21.15.189:0/3666344745 >> 172.21.15.189:0/3666344745 conn(0x7f4e080d00b0 msgr2=0x7f4e080d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.890+0000 7f4e0eb0b700 1 -- 172.21.15.189:0/3666344745 shutdown_connections 2024-04-06T12:13:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.890+0000 7f4e0eb0b700 1 -- 172.21.15.189:0/3666344745 wait complete. 2024-04-06T12:13:00.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.892+0000 7f4e0eb0b700 1 Processor -- start 2024-04-06T12:13:00.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.892+0000 7f4e0eb0b700 1 -- start start 2024-04-06T12:13:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.892+0000 7f4e0eb0b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e080b4d70 0x7f4e081ed2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.892+0000 7f4e0eb0b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e080cf900 0x7f4e08151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.892+0000 7f4e0eb0b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e08152260 0x7f4e0815bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.892+0000 7f4e0eb0b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0815c8d0 con 0x7f4e080cf900 2024-04-06T12:13:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.892+0000 7f4e0eb0b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0815cbb0 con 0x7f4e08152260 2024-04-06T12:13:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.892+0000 7f4e0eb0b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0815ce90 con 0x7f4e080b4d70 2024-04-06T12:13:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.893+0000 7f4e07fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e080cf900 0x7f4e08151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.893+0000 7f4e07fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e080cf900 0x7f4e08151d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44696/0 (socket says 172.21.15.189:44696) 2024-04-06T12:13:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.893+0000 7f4e07fff700 1 -- 172.21.15.189:0/3892488702 learned_addr learned my addr 172.21.15.189:0/3892488702 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.893+0000 7f4e07fff700 1 -- 172.21.15.189:0/3892488702 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e080b4d70 msgr2=0x7f4e081ed2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.893+0000 7f4e07fff700 1 --2- 172.21.15.189:0/3892488702 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e080b4d70 0x7f4e081ed2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.893+0000 7f4e0d0a8700 1 --2- 172.21.15.189:0/3892488702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e08152260 0x7f4e0815bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.893+0000 7f4e07fff700 1 -- 172.21.15.189:0/3892488702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e08152260 msgr2=0x7f4e0815bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.893+0000 7f4e07fff700 1 --2- 172.21.15.189:0/3892488702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e08152260 0x7f4e0815bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.893+0000 7f4e07fff700 1 -- 172.21.15.189:0/3892488702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dfc009550 con 0x7f4e080cf900 2024-04-06T12:13:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.893+0000 7f4e0d0a8700 1 --2- 172.21.15.189:0/3892488702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e08152260 0x7f4e0815bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.893+0000 7f4e07fff700 1 --2- 172.21.15.189:0/3892488702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e080cf900 0x7f4e08151d20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4df400c4d0 tx=0x7f4df400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.893+0000 7f4e057fa700 1 -- 172.21.15.189:0/3892488702 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df4011790 con 0x7f4e080cf900 2024-04-06T12:13:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.893+0000 7f4e057fa700 1 -- 172.21.15.189:0/3892488702 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df40118f0 con 0x7f4e080cf900 2024-04-06T12:13:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.893+0000 7f4e0eb0b700 1 -- 172.21.15.189:0/3892488702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e08157a80 con 0x7f4e080cf900 2024-04-06T12:13:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.893+0000 7f4e057fa700 1 -- 172.21.15.189:0/3892488702 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df4011ca0 con 0x7f4e080cf900 2024-04-06T12:13:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.893+0000 7f4e0eb0b700 1 -- 172.21.15.189:0/3892488702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e0815d170 con 0x7f4e080cf900 2024-04-06T12:13:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.894+0000 7f4e0eb0b700 1 -- 172.21.15.189:0/3892488702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e08157ca0 con 0x7f4e080cf900 2024-04-06T12:13:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.895+0000 7f4e057fa700 1 -- 172.21.15.189:0/3892488702 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4df401c460 con 0x7f4e080cf900 2024-04-06T12:13:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.895+0000 7f4e057fa700 1 --2- 172.21.15.189:0/3892488702 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4dec06ac90 0x7f4dec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.895+0000 7f4e0c8a7700 1 --2- 172.21.15.189:0/3892488702 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4dec06ac90 0x7f4dec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.895+0000 7f4e057fa700 1 -- 172.21.15.189:0/3892488702 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f4df4089550 con 0x7f4e080cf900 2024-04-06T12:13:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.896+0000 7f4e0c8a7700 1 --2- 172.21.15.189:0/3892488702 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4dec06ac90 0x7f4dec06d150 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4e08159ab0 tx=0x7f4dfc010000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:00.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:00.900+0000 7f4e057fa700 1 -- 172.21.15.189:0/3892488702 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4df4054370 con 0x7f4e080cf900 2024-04-06T12:13:01.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.098+0000 7f4e0eb0b700 1 -- 172.21.15.189:0/3892488702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e08067bb0 con 0x7f4e080cf900 2024-04-06T12:13:01.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.099+0000 7f4e057fa700 1 -- 172.21.15.189:0/3892488702 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18314 (secure 0 0 0) 0x7f4df4057990 con 0x7f4e080cf900 2024-04-06T12:13:01.098 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:13:01.098 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":26,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:54.376742+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:13:01.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.101+0000 7f4e0eb0b700 1 -- 172.21.15.189:0/3892488702 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4dec06ac90 msgr2=0x7f4dec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:01.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.101+0000 7f4e0eb0b700 1 --2- 172.21.15.189:0/3892488702 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4dec06ac90 0x7f4dec06d150 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4e08159ab0 tx=0x7f4dfc010000 comp rx=0 tx=0).stop 2024-04-06T12:13:01.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.101+0000 7f4e0eb0b700 1 -- 172.21.15.189:0/3892488702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e080cf900 msgr2=0x7f4e08151d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:01.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.101+0000 7f4e0eb0b700 1 --2- 172.21.15.189:0/3892488702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e080cf900 0x7f4e08151d20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4df400c4d0 tx=0x7f4df400c890 comp rx=0 tx=0).stop 2024-04-06T12:13:01.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.101+0000 7f4e0eb0b700 1 -- 172.21.15.189:0/3892488702 shutdown_connections 2024-04-06T12:13:01.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.101+0000 7f4e0eb0b700 1 --2- 172.21.15.189:0/3892488702 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4dec06ac90 0x7f4dec06d150 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.101+0000 7f4e0eb0b700 1 --2- 172.21.15.189:0/3892488702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e08152260 0x7f4e0815bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.101+0000 7f4e0eb0b700 1 --2- 172.21.15.189:0/3892488702 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e080b4d70 0x7f4e081ed2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.101+0000 7f4e0eb0b700 1 --2- 172.21.15.189:0/3892488702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e080cf900 0x7f4e08151d20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.101+0000 7f4e0eb0b700 1 -- 172.21.15.189:0/3892488702 >> 172.21.15.189:0/3892488702 conn(0x7f4e080d00b0 msgr2=0x7f4e080afc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:01.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.101+0000 7f4e0eb0b700 1 -- 172.21.15.189:0/3892488702 shutdown_connections 2024-04-06T12:13:01.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.101+0000 7f4e0eb0b700 1 -- 172.21.15.189:0/3892488702 wait complete. 2024-04-06T12:13:01.113 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:12:02.280866+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:12:03.156593+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-06T12:12:06.237184+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-06T12:13:01.113 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:13:01.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.239+0000 7ff8433a4700 1 Processor -- start 2024-04-06T12:13:01.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.239+0000 7ff8433a4700 1 -- start start 2024-04-06T12:13:01.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.240+0000 7ff8433a4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff83c06ae40 0x7ff83c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:01.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.240+0000 7ff8433a4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff83c15ddf0 0x7ff83c15e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:01.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.240+0000 7ff8433a4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83c0b3990 con 0x7ff83c15d7b0 2024-04-06T12:13:01.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.240+0000 7ff8433a4700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff83c0b3060 con 0x7ff83c15ddf0 2024-04-06T12:13:01.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.240+0000 7ff8433a4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0b3340 con 0x7ff83c06ae40 2024-04-06T12:13:01.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.240+0000 7ff841941700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff83c15ddf0 0x7ff83c15e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:01.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.240+0000 7ff841140700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff83c15d7b0 0x7ff83c06a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42634/0 (socket says 172.21.15.189:42634) 2024-04-06T12:13:01.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.240+0000 7ff841140700 1 -- 172.21.15.189:0/56165455 learned_addr learned my addr 172.21.15.189:0/56165455 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:01.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.240+0000 7ff84093f700 1 --2- 172.21.15.189:0/56165455 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff83c06ae40 0x7ff83c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:01.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.240+0000 7ff841941700 1 -- 172.21.15.189:0/56165455 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff83c06ae40 msgr2=0x7ff83c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:01.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.240+0000 7ff841941700 1 --2- 172.21.15.189:0/56165455 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff83c06ae40 0x7ff83c0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.240+0000 7ff841941700 1 -- 172.21.15.189:0/56165455 >> v1:172.21.15.189:6789/0 conn(0x7ff83c15d7b0 legacy=0x7ff83c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:01.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.240+0000 7ff833fff700 1 -- 172.21.15.189:0/56165455 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896663298 0 0) 0x7ff83c0b3990 con 0x7ff83c15d7b0 2024-04-06T12:13:01.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.240+0000 7ff841941700 1 -- 172.21.15.189:0/56165455 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83c0b3620 con 0x7ff83c15ddf0 2024-04-06T12:13:01.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.240+0000 7ff841941700 1 --2- 172.21.15.189:0/56165455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff83c15ddf0 0x7ff83c15e2a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff834009d90 tx=0x7ff83400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36c0d97a030906ba server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:01.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.241+0000 7ff833fff700 1 -- 172.21.15.189:0/56165455 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83400e070 con 0x7ff83c15ddf0 2024-04-06T12:13:01.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.241+0000 7ff833fff700 1 -- 172.21.15.189:0/56165455 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83400bbd0 con 0x7ff83c15ddf0 2024-04-06T12:13:01.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.241+0000 7ff833fff700 1 -- 172.21.15.189:0/56165455 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff834012430 con 0x7ff83c15ddf0 2024-04-06T12:13:01.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.241+0000 7ff8433a4700 1 -- 172.21.15.189:0/56165455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff83c15ddf0 msgr2=0x7ff83c15e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:01.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.241+0000 7ff8433a4700 1 --2- 172.21.15.189:0/56165455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff83c15ddf0 0x7ff83c15e2a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff834009d90 tx=0x7ff83400b330 comp rx=0 tx=0).stop 2024-04-06T12:13:01.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.241+0000 7ff8433a4700 1 -- 172.21.15.189:0/56165455 shutdown_connections 2024-04-06T12:13:01.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.241+0000 7ff8433a4700 1 --2- 172.21.15.189:0/56165455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff83c15ddf0 0x7ff83c15e2a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.241+0000 7ff8433a4700 1 --2- 172.21.15.189:0/56165455 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff83c06ae40 0x7ff83c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.241+0000 7ff8433a4700 1 -- 172.21.15.189:0/56165455 >> 172.21.15.189:0/56165455 conn(0x7ff83c0d00b0 msgr2=0x7ff83c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:01.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.241+0000 7ff8433a4700 1 -- 172.21.15.189:0/56165455 shutdown_connections 2024-04-06T12:13:01.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.241+0000 7ff8433a4700 1 -- 172.21.15.189:0/56165455 wait complete. 2024-04-06T12:13:01.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.242+0000 7ff8433a4700 1 Processor -- start 2024-04-06T12:13:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.242+0000 7ff8433a4700 1 -- start start 2024-04-06T12:13:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.243+0000 7ff8433a4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff83c06ae40 0x7ff83c1f65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.243+0000 7ff8433a4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff83c15d7b0 0x7ff83c0af590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.243+0000 7ff8433a4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff83c15ddf0 0x7ff83c14dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.243+0000 7ff8433a4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0cc4c0 con 0x7ff83c15d7b0 2024-04-06T12:13:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.243+0000 7ff8433a4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0aff00 con 0x7ff83c06ae40 2024-04-06T12:13:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.243+0000 7ff8433a4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0b01e0 con 0x7ff83c15ddf0 2024-04-06T12:13:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.243+0000 7ff84093f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff83c15d7b0 0x7ff83c0af590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.243+0000 7ff84093f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff83c15d7b0 0x7ff83c0af590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44722/0 (socket says 172.21.15.189:44722) 2024-04-06T12:13:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.243+0000 7ff84093f700 1 -- 172.21.15.189:0/2026105793 learned_addr learned my addr 172.21.15.189:0/2026105793 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.243+0000 7ff841140700 1 --2- 172.21.15.189:0/2026105793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff83c06ae40 0x7ff83c1f65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.243+0000 7ff841941700 1 --2- 172.21.15.189:0/2026105793 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff83c15ddf0 0x7ff83c14dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.243+0000 7ff84093f700 1 -- 172.21.15.189:0/2026105793 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff83c15ddf0 msgr2=0x7ff83c14dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:01.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.243+0000 7ff84093f700 1 --2- 172.21.15.189:0/2026105793 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff83c15ddf0 0x7ff83c14dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.243+0000 7ff84093f700 1 -- 172.21.15.189:0/2026105793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff83c06ae40 msgr2=0x7ff83c1f65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:01.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.243+0000 7ff84093f700 1 --2- 172.21.15.189:0/2026105793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff83c06ae40 0x7ff83c1f65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.243+0000 7ff84093f700 1 -- 172.21.15.189:0/2026105793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff834009a70 con 0x7ff83c15d7b0 2024-04-06T12:13:01.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.244+0000 7ff84093f700 1 --2- 172.21.15.189:0/2026105793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff83c15d7b0 0x7ff83c0af590 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff82c00c7f0 tx=0x7ff82c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:01.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.244+0000 7ff831ffb700 1 -- 172.21.15.189:0/2026105793 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82c004220 con 0x7ff83c15d7b0 2024-04-06T12:13:01.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.244+0000 7ff831ffb700 1 -- 172.21.15.189:0/2026105793 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82c004380 con 0x7ff83c15d7b0 2024-04-06T12:13:01.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.244+0000 7ff8433a4700 1 -- 172.21.15.189:0/2026105793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff83c0b04c0 con 0x7ff83c15d7b0 2024-04-06T12:13:01.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.244+0000 7ff831ffb700 1 -- 172.21.15.189:0/2026105793 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82c0036a0 con 0x7ff83c15d7b0 2024-04-06T12:13:01.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.244+0000 7ff8433a4700 1 -- 172.21.15.189:0/2026105793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff83c1f6b00 con 0x7ff83c15d7b0 2024-04-06T12:13:01.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.245+0000 7ff8433a4700 1 -- 172.21.15.189:0/2026105793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff83c0b0650 con 0x7ff83c15d7b0 2024-04-06T12:13:01.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.245+0000 7ff831ffb700 1 -- 172.21.15.189:0/2026105793 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff82c003800 con 0x7ff83c15d7b0 2024-04-06T12:13:01.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.246+0000 7ff831ffb700 1 --2- 172.21.15.189:0/2026105793 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff82006ac90 0x7ff82006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:01.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.246+0000 7ff831ffb700 1 -- 172.21.15.189:0/2026105793 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff82c089350 con 0x7ff83c15d7b0 2024-04-06T12:13:01.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.246+0000 7ff841140700 1 --2- 172.21.15.189:0/2026105793 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff82006ac90 0x7ff82006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:01.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.246+0000 7ff841140700 1 --2- 172.21.15.189:0/2026105793 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff82006ac90 0x7ff82006d150 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff838000f60 tx=0x7ff838007be0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:01.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.250+0000 7ff831ffb700 1 -- 172.21.15.189:0/2026105793 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff82c054180 con 0x7ff83c15d7b0 2024-04-06T12:13:01.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.445+0000 7ff8433a4700 1 -- 172.21.15.189:0/2026105793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff83c067bb0 con 0x7ff83c15d7b0 2024-04-06T12:13:01.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.446+0000 7ff831ffb700 1 -- 172.21.15.189:0/2026105793 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff82c0577a0 con 0x7ff83c15d7b0 2024-04-06T12:13:01.445 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:13:01.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.448+0000 7ff8433a4700 1 -- 172.21.15.189:0/2026105793 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff82006ac90 msgr2=0x7ff82006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:01.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.448+0000 7ff8433a4700 1 --2- 172.21.15.189:0/2026105793 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff82006ac90 0x7ff82006d150 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff838000f60 tx=0x7ff838007be0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.448+0000 7ff8433a4700 1 -- 172.21.15.189:0/2026105793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff83c15d7b0 msgr2=0x7ff83c0af590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:01.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.448+0000 7ff8433a4700 1 --2- 172.21.15.189:0/2026105793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff83c15d7b0 0x7ff83c0af590 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff82c00c7f0 tx=0x7ff82c00cbb0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.448+0000 7ff8433a4700 1 -- 172.21.15.189:0/2026105793 shutdown_connections 2024-04-06T12:13:01.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.448+0000 7ff8433a4700 1 --2- 172.21.15.189:0/2026105793 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff83c15ddf0 0x7ff83c14dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.448+0000 7ff8433a4700 1 --2- 172.21.15.189:0/2026105793 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff82006ac90 0x7ff82006d150 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.448+0000 7ff8433a4700 1 --2- 172.21.15.189:0/2026105793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff83c06ae40 0x7ff83c1f65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.448+0000 7ff8433a4700 1 --2- 172.21.15.189:0/2026105793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff83c15d7b0 0x7ff83c0af590 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.448+0000 7ff8433a4700 1 -- 172.21.15.189:0/2026105793 >> 172.21.15.189:0/2026105793 conn(0x7ff83c0d00b0 msgr2=0x7ff83c15c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:01.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.448+0000 7ff8433a4700 1 -- 172.21.15.189:0/2026105793 shutdown_connections 2024-04-06T12:13:01.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.448+0000 7ff8433a4700 1 -- 172.21.15.189:0/2026105793 wait complete. 2024-04-06T12:13:01.459 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:13:01.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.585+0000 7f7d34e68700 1 Processor -- start 2024-04-06T12:13:01.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.585+0000 7f7d34e68700 1 -- start start 2024-04-06T12:13:01.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.586+0000 7f7d34e68700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d3006ae40 0x7f7d300b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:01.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.586+0000 7f7d34e68700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7d3015ddf0 0x7f7d3015e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:01.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.586+0000 7f7d34e68700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d300b3990 con 0x7f7d3015d7b0 2024-04-06T12:13:01.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.586+0000 7f7d34e68700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d300b3060 con 0x7f7d3006ae40 2024-04-06T12:13:01.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.586+0000 7f7d34e68700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d300b3340 con 0x7f7d3015ddf0 2024-04-06T12:13:01.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.586+0000 7f7d2dd9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d3006ae40 0x7f7d300b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:01.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.586+0000 7f7d2dd9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d3006ae40 0x7f7d300b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44734/0 (socket says 172.21.15.189:44734) 2024-04-06T12:13:01.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.586+0000 7f7d2dd9b700 1 -- 172.21.15.189:0/1572889818 learned_addr learned my addr 172.21.15.189:0/1572889818 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:01.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.586+0000 7f7d2ed9d700 1 --2- 172.21.15.189:0/1572889818 >> v2:172.21.15.192:3300/0 conn(0x7f7d3015ddf0 0x7f7d3015e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:01.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.586+0000 7f7d2dd9b700 1 -- 172.21.15.189:0/1572889818 >> v2:172.21.15.192:3300/0 conn(0x7f7d3015ddf0 msgr2=0x7f7d3015e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:01.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.586+0000 7f7d2dd9b700 1 --2- 172.21.15.189:0/1572889818 >> v2:172.21.15.192:3300/0 conn(0x7f7d3015ddf0 0x7f7d3015e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.586+0000 7f7d2dd9b700 1 -- 172.21.15.189:0/1572889818 >> v1:172.21.15.192:6789/0 conn(0x7f7d3015d7b0 legacy=0x7f7d3006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:01.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.586+0000 7f7d2dd9b700 1 -- 172.21.15.189:0/1572889818 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d300b3620 con 0x7f7d3006ae40 2024-04-06T12:13:01.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.586+0000 7f7d2dd9b700 1 --2- 172.21.15.189:0/1572889818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3006ae40 0x7f7d300b3e20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7d3006d2c0 tx=0x7f7d24009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e33930c1f3fdf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:01.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.586+0000 7f7d2d59a700 1 -- 172.21.15.189:0/1572889818 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2400b870 con 0x7f7d3006ae40 2024-04-06T12:13:01.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.587+0000 7f7d2d59a700 1 -- 172.21.15.189:0/1572889818 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2400b9d0 con 0x7f7d3006ae40 2024-04-06T12:13:01.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.587+0000 7f7d2d59a700 1 -- 172.21.15.189:0/1572889818 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2400bd60 con 0x7f7d3006ae40 2024-04-06T12:13:01.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.587+0000 7f7d34e68700 1 -- 172.21.15.189:0/1572889818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3006ae40 msgr2=0x7f7d300b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:01.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.587+0000 7f7d34e68700 1 --2- 172.21.15.189:0/1572889818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3006ae40 0x7f7d300b3e20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7d3006d2c0 tx=0x7f7d24009bd0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.587+0000 7f7d34e68700 1 -- 172.21.15.189:0/1572889818 shutdown_connections 2024-04-06T12:13:01.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.587+0000 7f7d34e68700 1 --2- 172.21.15.189:0/1572889818 >> v2:172.21.15.192:3300/0 conn(0x7f7d3015ddf0 0x7f7d3015e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.587+0000 7f7d34e68700 1 --2- 172.21.15.189:0/1572889818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3006ae40 0x7f7d300b3e20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.587+0000 7f7d34e68700 1 -- 172.21.15.189:0/1572889818 >> 172.21.15.189:0/1572889818 conn(0x7f7d300d00b0 msgr2=0x7f7d300d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:01.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.587+0000 7f7d34e68700 1 -- 172.21.15.189:0/1572889818 shutdown_connections 2024-04-06T12:13:01.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.587+0000 7f7d34e68700 1 -- 172.21.15.189:0/1572889818 wait complete. 2024-04-06T12:13:01.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.588+0000 7f7d34e68700 1 Processor -- start 2024-04-06T12:13:01.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.589+0000 7f7d34e68700 1 -- start start 2024-04-06T12:13:01.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.589+0000 7f7d34e68700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d3006ae40 0x7f7d30143e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:01.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.589+0000 7f7d34e68700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d3015d7b0 0x7f7d301f3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:01.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.589+0000 7f7d34e68700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3015ddf0 0x7f7d300ce960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:01.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.589+0000 7f7d34e68700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30154da0 con 0x7f7d3015ddf0 2024-04-06T12:13:01.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.589+0000 7f7d34e68700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d300ce540 con 0x7f7d3006ae40 2024-04-06T12:13:01.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.589+0000 7f7d34e68700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30144370 con 0x7f7d3015d7b0 2024-04-06T12:13:01.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.589+0000 7f7d2ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3015ddf0 0x7f7d300ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:01.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.589+0000 7f7d2ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3015ddf0 0x7f7d300ce960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44736/0 (socket says 172.21.15.189:44736) 2024-04-06T12:13:01.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.589+0000 7f7d2ed9d700 1 -- 172.21.15.189:0/2472019863 learned_addr learned my addr 172.21.15.189:0/2472019863 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:01.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.589+0000 7f7d2e59c700 1 --2- 172.21.15.189:0/2472019863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d3006ae40 0x7f7d30143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:01.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.589+0000 7f7d2dd9b700 1 --2- 172.21.15.189:0/2472019863 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d3015d7b0 0x7f7d301f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:01.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.590+0000 7f7d2ed9d700 1 -- 172.21.15.189:0/2472019863 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d3015d7b0 msgr2=0x7f7d301f3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:01.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.590+0000 7f7d2ed9d700 1 --2- 172.21.15.189:0/2472019863 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d3015d7b0 0x7f7d301f3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.590+0000 7f7d2ed9d700 1 -- 172.21.15.189:0/2472019863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d3006ae40 msgr2=0x7f7d30143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:01.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.590+0000 7f7d2ed9d700 1 --2- 172.21.15.189:0/2472019863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d3006ae40 0x7f7d30143e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.590+0000 7f7d2ed9d700 1 -- 172.21.15.189:0/2472019863 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d24009550 con 0x7f7d3015ddf0 2024-04-06T12:13:01.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.590+0000 7f7d2ed9d700 1 --2- 172.21.15.189:0/2472019863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3015ddf0 0x7f7d300ce960 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7d28007e90 tx=0x7f7d2800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:01.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.590+0000 7f7d1effd700 1 -- 172.21.15.189:0/2472019863 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2800ed30 con 0x7f7d3015ddf0 2024-04-06T12:13:01.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.590+0000 7f7d1effd700 1 -- 172.21.15.189:0/2472019863 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2800ee90 con 0x7f7d3015ddf0 2024-04-06T12:13:01.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.590+0000 7f7d34e68700 1 -- 172.21.15.189:0/2472019863 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d301f3980 con 0x7f7d3015ddf0 2024-04-06T12:13:01.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.590+0000 7f7d1effd700 1 -- 172.21.15.189:0/2472019863 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d28014660 con 0x7f7d3015ddf0 2024-04-06T12:13:01.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.590+0000 7f7d34e68700 1 -- 172.21.15.189:0/2472019863 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d300cdba0 con 0x7f7d3015ddf0 2024-04-06T12:13:01.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.591+0000 7f7d34e68700 1 -- 172.21.15.189:0/2472019863 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d300ce680 con 0x7f7d3015ddf0 2024-04-06T12:13:01.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.591+0000 7f7d1effd700 1 -- 172.21.15.189:0/2472019863 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7d280147c0 con 0x7f7d3015ddf0 2024-04-06T12:13:01.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.592+0000 7f7d1effd700 1 --2- 172.21.15.189:0/2472019863 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d1006ac90 0x7f7d1006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:01.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.592+0000 7f7d1effd700 1 -- 172.21.15.189:0/2472019863 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7d28089ca0 con 0x7f7d3015ddf0 2024-04-06T12:13:01.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.592+0000 7f7d2e59c700 1 --2- 172.21.15.189:0/2472019863 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d1006ac90 0x7f7d1006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:01.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.592+0000 7f7d2e59c700 1 --2- 172.21.15.189:0/2472019863 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d1006ac90 0x7f7d1006d150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7d20007fd0 tx=0x7f7d20009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:01.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.596+0000 7f7d1effd700 1 -- 172.21.15.189:0/2472019863 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7d300ce680 con 0x7f7d3015ddf0 2024-04-06T12:13:01.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.788+0000 7f7d34e68700 1 -- 172.21.15.189:0/2472019863 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7d300ce680 con 0x7f7d3015ddf0 2024-04-06T12:13:01.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.788+0000 7f7d1effd700 1 -- 172.21.15.189:0/2472019863 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7d300ce680 con 0x7f7d3015ddf0 2024-04-06T12:13:01.787 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:13:01.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.790+0000 7f7d34e68700 1 -- 172.21.15.189:0/2472019863 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d1006ac90 msgr2=0x7f7d1006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:01.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.790+0000 7f7d34e68700 1 --2- 172.21.15.189:0/2472019863 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d1006ac90 0x7f7d1006d150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7d20007fd0 tx=0x7f7d20009040 comp rx=0 tx=0).stop 2024-04-06T12:13:01.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.790+0000 7f7d34e68700 1 -- 172.21.15.189:0/2472019863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3015ddf0 msgr2=0x7f7d300ce960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:01.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.790+0000 7f7d34e68700 1 --2- 172.21.15.189:0/2472019863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3015ddf0 0x7f7d300ce960 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7d28007e90 tx=0x7f7d2800e3c0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.790+0000 7f7d34e68700 1 -- 172.21.15.189:0/2472019863 shutdown_connections 2024-04-06T12:13:01.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.790+0000 7f7d34e68700 1 --2- 172.21.15.189:0/2472019863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3015ddf0 0x7f7d300ce960 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.790+0000 7f7d34e68700 1 --2- 172.21.15.189:0/2472019863 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d1006ac90 0x7f7d1006d150 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.790+0000 7f7d34e68700 1 --2- 172.21.15.189:0/2472019863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d3006ae40 0x7f7d30143e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.790+0000 7f7d34e68700 1 --2- 172.21.15.189:0/2472019863 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d3015d7b0 0x7f7d301f3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.790+0000 7f7d34e68700 1 -- 172.21.15.189:0/2472019863 >> 172.21.15.189:0/2472019863 conn(0x7f7d300d00b0 msgr2=0x7f7d3015c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:01.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.791+0000 7f7d34e68700 1 -- 172.21.15.189:0/2472019863 shutdown_connections 2024-04-06T12:13:01.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.791+0000 7f7d34e68700 1 -- 172.21.15.189:0/2472019863 wait complete. 2024-04-06T12:13:01.802 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:13:01.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.928+0000 7fd68539b700 1 Processor -- start 2024-04-06T12:13:01.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.928+0000 7fd68539b700 1 -- start start 2024-04-06T12:13:01.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.929+0000 7fd68539b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd68015db10 0x7fd6800b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:01.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.929+0000 7fd68539b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd68015eb40 con 0x7fd680143470 2024-04-06T12:13:01.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.929+0000 7fd68539b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6800d0bf0 con 0x7fd680160b60 2024-04-06T12:13:01.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.929+0000 7fd68539b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd680160f40 con 0x7fd68015db10 2024-04-06T12:13:01.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.929+0000 7fd67f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd680143470 0x7fd6800cd890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42648/0 (socket says 172.21.15.189:42648) 2024-04-06T12:13:01.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.929+0000 7fd67f7fe700 1 -- 172.21.15.189:0/451748965 learned_addr learned my addr 172.21.15.189:0/451748965 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:01.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.929+0000 7fd67e7fc700 1 --2- 172.21.15.189:0/451748965 >> v2:172.21.15.192:3300/0 conn(0x7fd68015db10 0x7fd6800b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:01.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.929+0000 7fd67dffb700 1 -- 172.21.15.189:0/451748965 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203257344 0 0) 0x7fd678001490 con 0x7fd680143470 2024-04-06T12:13:01.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.929+0000 7fd67dffb700 1 -- 172.21.15.189:0/451748965 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6680035b0 con 0x7fd680143470 2024-04-06T12:13:01.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.929+0000 7fd67dffb700 1 -- 172.21.15.189:0/451748965 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263524126 0 0) 0x7fd6800d0bf0 con 0x7fd680160b60 2024-04-06T12:13:01.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.929+0000 7fd67dffb700 1 -- 172.21.15.189:0/451748965 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd678001490 con 0x7fd680160b60 2024-04-06T12:13:01.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.929+0000 7fd67e7fc700 1 -- 172.21.15.189:0/451748965 >> v1:172.21.15.192:6789/0 conn(0x7fd680160b60 legacy=0x7fd680142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:01.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.929+0000 7fd67e7fc700 1 -- 172.21.15.189:0/451748965 >> v1:172.21.15.189:6789/0 conn(0x7fd680143470 legacy=0x7fd6800cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:01.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.929+0000 7fd67e7fc700 1 -- 172.21.15.189:0/451748965 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68015dfc0 con 0x7fd68015db10 2024-04-06T12:13:01.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.930+0000 7fd67e7fc700 1 --2- 172.21.15.189:0/451748965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd68015db10 0x7fd6800b37f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd670009820 tx=0x7fd670009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad2137558aed7dfe server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:01.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.930+0000 7fd67dffb700 1 -- 172.21.15.189:0/451748965 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd670005e30 con 0x7fd68015db10 2024-04-06T12:13:01.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.930+0000 7fd67dffb700 1 -- 172.21.15.189:0/451748965 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd670006030 con 0x7fd68015db10 2024-04-06T12:13:01.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.930+0000 7fd67dffb700 1 -- 172.21.15.189:0/451748965 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6700063c0 con 0x7fd68015db10 2024-04-06T12:13:01.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.930+0000 7fd68539b700 1 -- 172.21.15.189:0/451748965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd68015db10 msgr2=0x7fd6800b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:01.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.930+0000 7fd68539b700 1 --2- 172.21.15.189:0/451748965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd68015db10 0x7fd6800b37f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd670009820 tx=0x7fd670009b30 comp rx=0 tx=0).stop 2024-04-06T12:13:01.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.930+0000 7fd68539b700 1 -- 172.21.15.189:0/451748965 shutdown_connections 2024-04-06T12:13:01.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.930+0000 7fd68539b700 1 --2- 172.21.15.189:0/451748965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd68015db10 0x7fd6800b37f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.930+0000 7fd68539b700 1 -- 172.21.15.189:0/451748965 >> 172.21.15.189:0/451748965 conn(0x7fd68006a730 msgr2=0x7fd68006ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:01.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.930+0000 7fd68539b700 1 -- 172.21.15.189:0/451748965 shutdown_connections 2024-04-06T12:13:01.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.931+0000 7fd68539b700 1 -- 172.21.15.189:0/451748965 wait complete. 2024-04-06T12:13:01.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.932+0000 7fd68539b700 1 Processor -- start 2024-04-06T12:13:01.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.932+0000 7fd68539b700 1 -- start start 2024-04-06T12:13:01.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd68539b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd680143470 0x7fd68015cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:01.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd68539b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd68015db10 0x7fd6800c7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:01.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd68539b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd680160b60 0x7fd6800c7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:01.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd68539b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6800b4dd0 con 0x7fd680143470 2024-04-06T12:13:01.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd68539b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6800b5c30 con 0x7fd68015db10 2024-04-06T12:13:01.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd68539b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6800b5020 con 0x7fd680160b60 2024-04-06T12:13:01.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd67effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd680143470 0x7fd68015cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:01.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd67effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd680143470 0x7fd68015cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44750/0 (socket says 172.21.15.189:44750) 2024-04-06T12:13:01.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd67effd700 1 -- 172.21.15.189:0/1061914488 learned_addr learned my addr 172.21.15.189:0/1061914488 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:01.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd67e7fc700 1 --2- 172.21.15.189:0/1061914488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd68015db10 0x7fd6800c7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:01.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd67f7fe700 1 --2- 172.21.15.189:0/1061914488 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd680160b60 0x7fd6800c7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:01.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd67effd700 1 -- 172.21.15.189:0/1061914488 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd680160b60 msgr2=0x7fd6800c7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:01.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd67effd700 1 --2- 172.21.15.189:0/1061914488 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd680160b60 0x7fd6800c7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd67effd700 1 -- 172.21.15.189:0/1061914488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd68015db10 msgr2=0x7fd6800c7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:01.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd67effd700 1 --2- 172.21.15.189:0/1061914488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd68015db10 0x7fd6800c7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:01.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd67effd700 1 -- 172.21.15.189:0/1061914488 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd670009430 con 0x7fd680143470 2024-04-06T12:13:01.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd67f7fe700 1 --2- 172.21.15.189:0/1061914488 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd680160b60 0x7fd6800c7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:01.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd67effd700 1 --2- 172.21.15.189:0/1061914488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd680143470 0x7fd68015cd60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd67400c500 tx=0x7fd67400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:01.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd6677fe700 1 -- 172.21.15.189:0/1061914488 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6740052d0 con 0x7fd680143470 2024-04-06T12:13:01.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd6677fe700 1 -- 172.21.15.189:0/1061914488 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd674005450 con 0x7fd680143470 2024-04-06T12:13:01.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd6677fe700 1 -- 172.21.15.189:0/1061914488 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6740038c0 con 0x7fd680143470 2024-04-06T12:13:01.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd68539b700 1 -- 172.21.15.189:0/1061914488 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6800b5300 con 0x7fd680143470 2024-04-06T12:13:01.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.933+0000 7fd68539b700 1 -- 172.21.15.189:0/1061914488 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6800b58c0 con 0x7fd680143470 2024-04-06T12:13:01.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.934+0000 7fd68539b700 1 -- 172.21.15.189:0/1061914488 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd680074990 con 0x7fd680143470 2024-04-06T12:13:01.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.935+0000 7fd6677fe700 1 -- 172.21.15.189:0/1061914488 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd674003a20 con 0x7fd680143470 2024-04-06T12:13:01.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.935+0000 7fd6677fe700 1 --2- 172.21.15.189:0/1061914488 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd66c06ac90 0x7fd66c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:01.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.935+0000 7fd6677fe700 1 -- 172.21.15.189:0/1061914488 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd674088af0 con 0x7fd680143470 2024-04-06T12:13:01.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.935+0000 7fd67e7fc700 1 --2- 172.21.15.189:0/1061914488 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd66c06ac90 0x7fd66c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:01.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.936+0000 7fd67e7fc700 1 --2- 172.21.15.189:0/1061914488 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd66c06ac90 0x7fd66c06d150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd670013040 tx=0x7fd670008070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:01.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:01.940+0000 7fd6677fe700 1 -- 172.21.15.189:0/1061914488 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd674053910 con 0x7fd680143470 2024-04-06T12:13:02.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.131+0000 7fd68539b700 1 -- 172.21.15.189:0/1061914488 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd6800b71a0 con 0x7fd680143470 2024-04-06T12:13:02.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.131+0000 7fd6677fe700 1 -- 172.21.15.189:0/1061914488 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd6740170a0 con 0x7fd680143470 2024-04-06T12:13:02.130 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:13:02.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.133+0000 7fd68539b700 1 -- 172.21.15.189:0/1061914488 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd66c06ac90 msgr2=0x7fd66c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:02.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.133+0000 7fd68539b700 1 --2- 172.21.15.189:0/1061914488 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd66c06ac90 0x7fd66c06d150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd670013040 tx=0x7fd670008070 comp rx=0 tx=0).stop 2024-04-06T12:13:02.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.133+0000 7fd68539b700 1 -- 172.21.15.189:0/1061914488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd680143470 msgr2=0x7fd68015cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:02.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.133+0000 7fd68539b700 1 --2- 172.21.15.189:0/1061914488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd680143470 0x7fd68015cd60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd67400c500 tx=0x7fd67400c810 comp rx=0 tx=0).stop 2024-04-06T12:13:02.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.133+0000 7fd68539b700 1 -- 172.21.15.189:0/1061914488 shutdown_connections 2024-04-06T12:13:02.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.133+0000 7fd68539b700 1 --2- 172.21.15.189:0/1061914488 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd680160b60 0x7fd6800c7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.133+0000 7fd68539b700 1 --2- 172.21.15.189:0/1061914488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd680143470 0x7fd68015cd60 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.133+0000 7fd68539b700 1 --2- 172.21.15.189:0/1061914488 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd66c06ac90 0x7fd66c06d150 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.133+0000 7fd68539b700 1 --2- 172.21.15.189:0/1061914488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd68015db10 0x7fd6800c7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.133+0000 7fd68539b700 1 -- 172.21.15.189:0/1061914488 >> 172.21.15.189:0/1061914488 conn(0x7fd68006a730 msgr2=0x7fd68006ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:02.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.133+0000 7fd68539b700 1 -- 172.21.15.189:0/1061914488 shutdown_connections 2024-04-06T12:13:02.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.133+0000 7fd68539b700 1 -- 172.21.15.189:0/1061914488 wait complete. 2024-04-06T12:13:02.145 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-06T12:13:02.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.271+0000 7fdb03868700 1 Processor -- start 2024-04-06T12:13:02.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.271+0000 7fdb03868700 1 -- start start 2024-04-06T12:13:02.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.272+0000 7fdb03868700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdafc0cbc20 0x7fdafc15f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:02.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.272+0000 7fdb03868700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdafc0669d0 con 0x7fdafc15e8a0 2024-04-06T12:13:02.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.272+0000 7fdb03868700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdafc06ede0 con 0x7fdafc06aa10 2024-04-06T12:13:02.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.272+0000 7fdb03868700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdafc15ec80 con 0x7fdafc0cbc20 2024-04-06T12:13:02.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.272+0000 7fdb01604700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdafc15e8a0 0x7fdafc15eee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42658/0 (socket says 172.21.15.189:42658) 2024-04-06T12:13:02.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.272+0000 7fdb01604700 1 -- 172.21.15.189:0/2048987938 learned_addr learned my addr 172.21.15.189:0/2048987938 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:02.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.272+0000 7fdb00e03700 1 --2- 172.21.15.189:0/2048987938 >> v2:172.21.15.192:3300/0 conn(0x7fdafc0cbc20 0x7fdafc15f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:02.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.272+0000 7fdaf3fff700 1 -- 172.21.15.189:0/2048987938 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291420655 0 0) 0x7fdafc0669d0 con 0x7fdafc15e8a0 2024-04-06T12:13:02.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.272+0000 7fdaf3fff700 1 -- 172.21.15.189:0/2048987938 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdae80035b0 con 0x7fdafc15e8a0 2024-04-06T12:13:02.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdaf3fff700 1 -- 172.21.15.189:0/2048987938 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444729425 0 0) 0x7fdafc06ede0 con 0x7fdafc06aa10 2024-04-06T12:13:02.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdaf3fff700 1 -- 172.21.15.189:0/2048987938 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdafc0669d0 con 0x7fdafc06aa10 2024-04-06T12:13:02.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdaf3fff700 1 -- 172.21.15.189:0/2048987938 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1032957542 0 0) 0x7fdae80035b0 con 0x7fdafc15e8a0 2024-04-06T12:13:02.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdaf3fff700 1 -- 172.21.15.189:0/2048987938 >> v2:172.21.15.192:3300/0 conn(0x7fdafc0cbc20 msgr2=0x7fdafc15f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:02.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdaf3fff700 1 --2- 172.21.15.189:0/2048987938 >> v2:172.21.15.192:3300/0 conn(0x7fdafc0cbc20 0x7fdafc15f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdaf3fff700 1 -- 172.21.15.189:0/2048987938 >> v1:172.21.15.192:6789/0 conn(0x7fdafc06aa10 legacy=0x7fdafc06ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:02.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdaf3fff700 1 -- 172.21.15.189:0/2048987938 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdafc0666f0 con 0x7fdafc15e8a0 2024-04-06T12:13:02.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdb00e03700 1 --2- 172.21.15.189:0/2048987938 >> v2:172.21.15.192:3300/0 conn(0x7fdafc0cbc20 0x7fdafc15f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:13:02.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdaf3fff700 1 -- 172.21.15.189:0/2048987938 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fdaec001670 con 0x7fdafc15e8a0 2024-04-06T12:13:02.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdaf3fff700 1 -- 172.21.15.189:0/2048987938 >> v1:172.21.15.189:6789/0 conn(0x7fdafc15e8a0 legacy=0x7fdafc15eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:02.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdaf3fff700 1 --2- 172.21.15.189:0/2048987938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdae8003b90 0x7fdae8003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:02.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdaf3fff700 1 --2- 172.21.15.189:0/2048987938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae8004850 0x7fdae8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:02.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdaf3fff700 1 --2- 172.21.15.189:0/2048987938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdae8005210 0x7fdae800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:02.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdaf3fff700 1 -- 172.21.15.189:0/2048987938 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc15ec80 con 0x7fdae8004850 2024-04-06T12:13:02.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdaf3fff700 1 -- 172.21.15.189:0/2048987938 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae800c140 con 0x7fdae8003b90 2024-04-06T12:13:02.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdaf3fff700 1 -- 172.21.15.189:0/2048987938 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae800c300 con 0x7fdae8005210 2024-04-06T12:13:02.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdb00e03700 1 --2- 172.21.15.189:0/2048987938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae8004850 0x7fdae8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:02.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdb01604700 1 --2- 172.21.15.189:0/2048987938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdae8003b90 0x7fdae8003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:02.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdb01e05700 1 --2- 172.21.15.189:0/2048987938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdae8005210 0x7fdae800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:02.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdb00e03700 1 -- 172.21.15.189:0/2048987938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdae8005210 msgr2=0x7fdae800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:02.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdb00e03700 1 --2- 172.21.15.189:0/2048987938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdae8005210 0x7fdae800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdb00e03700 1 -- 172.21.15.189:0/2048987938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdae8003b90 msgr2=0x7fdae8003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:02.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdb00e03700 1 --2- 172.21.15.189:0/2048987938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdae8003b90 0x7fdae8003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdb00e03700 1 -- 172.21.15.189:0/2048987938 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae80035b0 con 0x7fdae8004850 2024-04-06T12:13:02.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdb01e05700 1 -- 172.21.15.189:0/2048987938 reap_dead start 2024-04-06T12:13:02.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdb00e03700 1 --2- 172.21.15.189:0/2048987938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae8004850 0x7fdae8004cd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8009d30 tx=0x7fdae8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ba9ae0af356533 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:02.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdaf3fff700 1 -- 172.21.15.189:0/2048987938 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8009e80 con 0x7fdae8004850 2024-04-06T12:13:02.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdaf3fff700 1 -- 172.21.15.189:0/2048987938 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf800e9a0 con 0x7fdae8004850 2024-04-06T12:13:02.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.273+0000 7fdaf3fff700 1 -- 172.21.15.189:0/2048987938 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf800ed10 con 0x7fdae8004850 2024-04-06T12:13:02.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.274+0000 7fdb03868700 1 -- 172.21.15.189:0/2048987938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae8004850 msgr2=0x7fdae8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:02.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.274+0000 7fdb03868700 1 --2- 172.21.15.189:0/2048987938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae8004850 0x7fdae8004cd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8009d30 tx=0x7fdae8005880 comp rx=0 tx=0).stop 2024-04-06T12:13:02.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.274+0000 7fdb03868700 1 -- 172.21.15.189:0/2048987938 shutdown_connections 2024-04-06T12:13:02.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.274+0000 7fdb03868700 1 --2- 172.21.15.189:0/2048987938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae8004850 0x7fdae8004cd0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.274+0000 7fdb03868700 1 -- 172.21.15.189:0/2048987938 >> 172.21.15.189:0/2048987938 conn(0x7fdafc0cc290 msgr2=0x7fdafc0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:02.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.274+0000 7fdb03868700 1 -- 172.21.15.189:0/2048987938 shutdown_connections 2024-04-06T12:13:02.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.274+0000 7fdb03868700 1 -- 172.21.15.189:0/2048987938 wait complete. 2024-04-06T12:13:02.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.275+0000 7fdb03868700 1 Processor -- start 2024-04-06T12:13:02.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.275+0000 7fdb03868700 1 -- start start 2024-04-06T12:13:02.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.276+0000 7fdb03868700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae8004850 0x7fdafc0cf040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:02.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.276+0000 7fdb03868700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdafc152c60 0x7fdafc0c57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:02.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.276+0000 7fdb03868700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdafc0c5d20 0x7fdafc0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:02.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.276+0000 7fdb03868700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc0c4410 con 0x7fdae8004850 2024-04-06T12:13:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.276+0000 7fdb03868700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc0b02b0 con 0x7fdafc0c5d20 2024-04-06T12:13:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.276+0000 7fdb03868700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc0cd3e0 con 0x7fdafc152c60 2024-04-06T12:13:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.276+0000 7fdb01604700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae8004850 0x7fdafc0cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.276+0000 7fdb01604700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae8004850 0x7fdafc0cf040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44772/0 (socket says 172.21.15.189:44772) 2024-04-06T12:13:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.276+0000 7fdb01604700 1 -- 172.21.15.189:0/1191888056 learned_addr learned my addr 172.21.15.189:0/1191888056 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.276+0000 7fdb01e05700 1 --2- 172.21.15.189:0/1191888056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdafc0c5d20 0x7fdafc0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.276+0000 7fdb00e03700 1 --2- 172.21.15.189:0/1191888056 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdafc152c60 0x7fdafc0c57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.277+0000 7fdb01604700 1 -- 172.21.15.189:0/1191888056 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdafc152c60 msgr2=0x7fdafc0c57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.277+0000 7fdb01604700 1 --2- 172.21.15.189:0/1191888056 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdafc152c60 0x7fdafc0c57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.277+0000 7fdb01604700 1 -- 172.21.15.189:0/1191888056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdafc0c5d20 msgr2=0x7fdafc0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.277+0000 7fdb01604700 1 --2- 172.21.15.189:0/1191888056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdafc0c5d20 0x7fdafc0ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.277+0000 7fdb01604700 1 -- 172.21.15.189:0/1191888056 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdafc0c3e90 con 0x7fdae8004850 2024-04-06T12:13:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.277+0000 7fdb00e03700 1 --2- 172.21.15.189:0/1191888056 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdafc152c60 0x7fdafc0c57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.277+0000 7fdb01e05700 1 --2- 172.21.15.189:0/1191888056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdafc0c5d20 0x7fdafc0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.277+0000 7fdb01604700 1 --2- 172.21.15.189:0/1191888056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae8004850 0x7fdafc0cf040 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdaec0072c0 tx=0x7fdaec007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:02.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.277+0000 7fdaf1ffb700 1 -- 172.21.15.189:0/1191888056 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec001620 con 0x7fdae8004850 2024-04-06T12:13:02.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.277+0000 7fdaf1ffb700 1 -- 172.21.15.189:0/1191888056 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaec0077a0 con 0x7fdae8004850 2024-04-06T12:13:02.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.277+0000 7fdaf1ffb700 1 -- 172.21.15.189:0/1191888056 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec00fad0 con 0x7fdae8004850 2024-04-06T12:13:02.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.277+0000 7fdb03868700 1 -- 172.21.15.189:0/1191888056 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdafc0c3670 con 0x7fdae8004850 2024-04-06T12:13:02.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.277+0000 7fdb03868700 1 -- 172.21.15.189:0/1191888056 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdafc0c61a0 con 0x7fdae8004850 2024-04-06T12:13:02.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.278+0000 7fdb03868700 1 -- 172.21.15.189:0/1191888056 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdafc072140 con 0x7fdae8004850 2024-04-06T12:13:02.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.278+0000 7fdaf1ffb700 1 -- 172.21.15.189:0/1191888056 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdaec00fc30 con 0x7fdae8004850 2024-04-06T12:13:02.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.279+0000 7fdaf1ffb700 1 --2- 172.21.15.189:0/1191888056 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdae006ac90 0x7fdae006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:02.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.279+0000 7fdaf1ffb700 1 -- 172.21.15.189:0/1191888056 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fdaec0895b0 con 0x7fdae8004850 2024-04-06T12:13:02.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.279+0000 7fdb00e03700 1 --2- 172.21.15.189:0/1191888056 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdae006ac90 0x7fdae006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:02.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.279+0000 7fdb00e03700 1 --2- 172.21.15.189:0/1191888056 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdae006ac90 0x7fdae006d150 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdaf80090b0 tx=0x7fdaf8008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:02.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.283+0000 7fdaf1ffb700 1 -- 172.21.15.189:0/1191888056 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdaec054460 con 0x7fdae8004850 2024-04-06T12:13:02.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.479+0000 7fdb03868700 1 -- 172.21.15.189:0/1191888056 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fdafc074990 con 0x7fdae8004850 2024-04-06T12:13:02.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.479+0000 7fdaf1ffb700 1 -- 172.21.15.189:0/1191888056 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7fdaec057a80 con 0x7fdae8004850 2024-04-06T12:13:02.479 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:13:02.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.482+0000 7fdb03868700 1 -- 172.21.15.189:0/1191888056 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdae006ac90 msgr2=0x7fdae006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:02.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.482+0000 7fdb03868700 1 --2- 172.21.15.189:0/1191888056 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdae006ac90 0x7fdae006d150 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdaf80090b0 tx=0x7fdaf8008030 comp rx=0 tx=0).stop 2024-04-06T12:13:02.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.482+0000 7fdb03868700 1 -- 172.21.15.189:0/1191888056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae8004850 msgr2=0x7fdafc0cf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:02.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.482+0000 7fdb03868700 1 --2- 172.21.15.189:0/1191888056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae8004850 0x7fdafc0cf040 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdaec0072c0 tx=0x7fdaec007680 comp rx=0 tx=0).stop 2024-04-06T12:13:02.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.482+0000 7fdb03868700 1 -- 172.21.15.189:0/1191888056 shutdown_connections 2024-04-06T12:13:02.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.482+0000 7fdb03868700 1 --2- 172.21.15.189:0/1191888056 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdae006ac90 0x7fdae006d150 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.482+0000 7fdb03868700 1 --2- 172.21.15.189:0/1191888056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdafc0c5d20 0x7fdafc0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.482+0000 7fdb03868700 1 --2- 172.21.15.189:0/1191888056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae8004850 0x7fdafc0cf040 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.482+0000 7fdb03868700 1 --2- 172.21.15.189:0/1191888056 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdafc152c60 0x7fdafc0c57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.482+0000 7fdb03868700 1 -- 172.21.15.189:0/1191888056 >> 172.21.15.189:0/1191888056 conn(0x7fdafc0cc290 msgr2=0x7fdafc15cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:02.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.482+0000 7fdb03868700 1 -- 172.21.15.189:0/1191888056 shutdown_connections 2024-04-06T12:13:02.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.482+0000 7fdb03868700 1 -- 172.21.15.189:0/1191888056 wait complete. 2024-04-06T12:13:02.493 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T12:13:02.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.618+0000 7f60a3188700 1 Processor -- start 2024-04-06T12:13:02.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.619+0000 7f60a3188700 1 -- start start 2024-04-06T12:13:02.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.619+0000 7f60a3188700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f609c15b270 0x7f609c0ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:02.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.619+0000 7f60a3188700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f609c06be50 0x7f609c164db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:02.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.619+0000 7f60a3188700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f609c15b650 con 0x7f609c0aecc0 2024-04-06T12:13:02.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.619+0000 7f60a3188700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f609c0cf570 con 0x7f609c06be50 2024-04-06T12:13:02.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.619+0000 7f60a3188700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f609c06cdd0 con 0x7f609c15b270 2024-04-06T12:13:02.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.619+0000 7f609bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f609c0aecc0 0x7f609c06b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42660/0 (socket says 172.21.15.189:42660) 2024-04-06T12:13:02.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.619+0000 7f609bfff700 1 -- 172.21.15.189:0/2330139429 learned_addr learned my addr 172.21.15.189:0/2330139429 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:02.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.620+0000 7f60a1725700 1 --2- 172.21.15.189:0/2330139429 >> v2:172.21.15.192:3300/0 conn(0x7f609c06be50 0x7f609c164db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:02.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.620+0000 7f60a0f24700 1 --2- 172.21.15.189:0/2330139429 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f609c15b270 0x7f609c0ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:02.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.620+0000 7f609b7fe700 1 -- 172.21.15.189:0/2330139429 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53877645 0 0) 0x7f609c15b650 con 0x7f609c0aecc0 2024-04-06T12:13:02.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.620+0000 7f609b7fe700 1 -- 172.21.15.189:0/2330139429 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f608c0035b0 con 0x7f609c0aecc0 2024-04-06T12:13:02.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.620+0000 7f60a1725700 1 -- 172.21.15.189:0/2330139429 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f609c15b270 msgr2=0x7f609c0ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:02.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.620+0000 7f60a1725700 1 --2- 172.21.15.189:0/2330139429 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f609c15b270 0x7f609c0ce240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.620+0000 7f60a1725700 1 -- 172.21.15.189:0/2330139429 >> v1:172.21.15.189:6789/0 conn(0x7f609c0aecc0 legacy=0x7f609c06b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:02.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.620+0000 7f60a1725700 1 -- 172.21.15.189:0/2330139429 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609c06d0b0 con 0x7f609c06be50 2024-04-06T12:13:02.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.620+0000 7f60a1725700 1 --2- 172.21.15.189:0/2330139429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f609c06be50 0x7f609c164db0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6094009d90 tx=0x7f609400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=543575203264d9df server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:02.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.620+0000 7f609b7fe700 1 -- 172.21.15.189:0/2330139429 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609400e070 con 0x7f609c06be50 2024-04-06T12:13:02.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.620+0000 7f609b7fe700 1 -- 172.21.15.189:0/2330139429 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609400bb20 con 0x7f609c06be50 2024-04-06T12:13:02.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.620+0000 7f609b7fe700 1 -- 172.21.15.189:0/2330139429 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6094012430 con 0x7f609c06be50 2024-04-06T12:13:02.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.621+0000 7f60a3188700 1 -- 172.21.15.189:0/2330139429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f609c06be50 msgr2=0x7f609c164db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:02.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.621+0000 7f60a3188700 1 --2- 172.21.15.189:0/2330139429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f609c06be50 0x7f609c164db0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6094009d90 tx=0x7f609400b330 comp rx=0 tx=0).stop 2024-04-06T12:13:02.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.621+0000 7f60a3188700 1 -- 172.21.15.189:0/2330139429 shutdown_connections 2024-04-06T12:13:02.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.621+0000 7f60a3188700 1 --2- 172.21.15.189:0/2330139429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f609c06be50 0x7f609c164db0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.621+0000 7f60a3188700 1 --2- 172.21.15.189:0/2330139429 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f609c15b270 0x7f609c0ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.621+0000 7f60a3188700 1 -- 172.21.15.189:0/2330139429 >> 172.21.15.189:0/2330139429 conn(0x7f609c070e90 msgr2=0x7f609c0712a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:02.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.621+0000 7f60a3188700 1 -- 172.21.15.189:0/2330139429 shutdown_connections 2024-04-06T12:13:02.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.621+0000 7f60a3188700 1 -- 172.21.15.189:0/2330139429 wait complete. 2024-04-06T12:13:02.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.622+0000 7f60a3188700 1 Processor -- start 2024-04-06T12:13:02.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.622+0000 7f60a3188700 1 -- start start 2024-04-06T12:13:02.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.623+0000 7f60a3188700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f609c06be50 0x7f609c14f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:02.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.623+0000 7f60a3188700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f609c0aecc0 0x7f609c155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:02.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.623+0000 7f60a3188700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f609c15b270 0x7f609c15da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:02.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.623+0000 7f60a3188700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c0ae530 con 0x7f609c06be50 2024-04-06T12:13:02.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.623+0000 7f60a3188700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c156380 con 0x7f609c0aecc0 2024-04-06T12:13:02.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.623+0000 7f60a3188700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f609c156660 con 0x7f609c15b270 2024-04-06T12:13:02.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.623+0000 7f60a0f24700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f609c06be50 0x7f609c14f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:02.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.623+0000 7f60a0f24700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f609c06be50 0x7f609c14f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44780/0 (socket says 172.21.15.189:44780) 2024-04-06T12:13:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.623+0000 7f60a0f24700 1 -- 172.21.15.189:0/1449156159 learned_addr learned my addr 172.21.15.189:0/1449156159 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.623+0000 7f609bfff700 1 --2- 172.21.15.189:0/1449156159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f609c0aecc0 0x7f609c155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.623+0000 7f60a1725700 1 --2- 172.21.15.189:0/1449156159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f609c15b270 0x7f609c15da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.623+0000 7f60a0f24700 1 -- 172.21.15.189:0/1449156159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f609c15b270 msgr2=0x7f609c15da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.623+0000 7f60a0f24700 1 --2- 172.21.15.189:0/1449156159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f609c15b270 0x7f609c15da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.623+0000 7f60a0f24700 1 -- 172.21.15.189:0/1449156159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f609c0aecc0 msgr2=0x7f609c155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.623+0000 7f60a0f24700 1 --2- 172.21.15.189:0/1449156159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f609c0aecc0 0x7f609c155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.623+0000 7f60a0f24700 1 -- 172.21.15.189:0/1449156159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6094009a70 con 0x7f609c06be50 2024-04-06T12:13:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.624+0000 7f60a0f24700 1 --2- 172.21.15.189:0/1449156159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f609c06be50 0x7f609c14f0d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f609000e870 tx=0x7f609000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.624+0000 7f60997fa700 1 -- 172.21.15.189:0/1449156159 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6090009590 con 0x7f609c06be50 2024-04-06T12:13:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.624+0000 7f60997fa700 1 -- 172.21.15.189:0/1449156159 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609000ee70 con 0x7f609c06be50 2024-04-06T12:13:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.624+0000 7f60a3188700 1 -- 172.21.15.189:0/1449156159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f609c156940 con 0x7f609c06be50 2024-04-06T12:13:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.624+0000 7f60997fa700 1 -- 172.21.15.189:0/1449156159 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609000ca50 con 0x7f609c06be50 2024-04-06T12:13:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.624+0000 7f60a3188700 1 -- 172.21.15.189:0/1449156159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609c156f00 con 0x7f609c06be50 2024-04-06T12:13:02.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.625+0000 7f60a3188700 1 -- 172.21.15.189:0/1449156159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f609c0748f0 con 0x7f609c06be50 2024-04-06T12:13:02.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.625+0000 7f60997fa700 1 -- 172.21.15.189:0/1449156159 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f609000cbb0 con 0x7f609c06be50 2024-04-06T12:13:02.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.626+0000 7f60997fa700 1 --2- 172.21.15.189:0/1449156159 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f608006ac90 0x7f608006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:02.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.626+0000 7f60997fa700 1 -- 172.21.15.189:0/1449156159 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f60900896f0 con 0x7f609c06be50 2024-04-06T12:13:02.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.626+0000 7f609bfff700 1 --2- 172.21.15.189:0/1449156159 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f608006ac90 0x7f608006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:02.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.626+0000 7f609bfff700 1 --2- 172.21.15.189:0/1449156159 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f608006ac90 0x7f608006d150 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f60880011e0 tx=0x7f6088009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:02.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.630+0000 7f60997fa700 1 -- 172.21.15.189:0/1449156159 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6090054510 con 0x7f609c06be50 2024-04-06T12:13:02.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.837+0000 7f60a3188700 1 -- 172.21.15.189:0/1449156159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f609c06b580 con 0x7f609c06be50 2024-04-06T12:13:02.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.837+0000 7f60997fa700 1 -- 172.21.15.189:0/1449156159 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7f609000ce60 con 0x7f609c06be50 2024-04-06T12:13:02.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.839+0000 7f60a3188700 1 -- 172.21.15.189:0/1449156159 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f608006ac90 msgr2=0x7f608006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:02.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.839+0000 7f60a3188700 1 --2- 172.21.15.189:0/1449156159 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f608006ac90 0x7f608006d150 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f60880011e0 tx=0x7f6088009040 comp rx=0 tx=0).stop 2024-04-06T12:13:02.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.839+0000 7f60a3188700 1 -- 172.21.15.189:0/1449156159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f609c06be50 msgr2=0x7f609c14f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:02.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.839+0000 7f60a3188700 1 --2- 172.21.15.189:0/1449156159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f609c06be50 0x7f609c14f0d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f609000e870 tx=0x7f609000ec30 comp rx=0 tx=0).stop 2024-04-06T12:13:02.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.840+0000 7f60a3188700 1 -- 172.21.15.189:0/1449156159 shutdown_connections 2024-04-06T12:13:02.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.840+0000 7f60a3188700 1 --2- 172.21.15.189:0/1449156159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f609c15b270 0x7f609c15da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.840+0000 7f60a3188700 1 --2- 172.21.15.189:0/1449156159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f609c06be50 0x7f609c14f0d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.840+0000 7f60a3188700 1 --2- 172.21.15.189:0/1449156159 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f608006ac90 0x7f608006d150 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.840+0000 7f60a3188700 1 --2- 172.21.15.189:0/1449156159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f609c0aecc0 0x7f609c155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:02.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.840+0000 7f60a3188700 1 -- 172.21.15.189:0/1449156159 >> 172.21.15.189:0/1449156159 conn(0x7f609c070e90 msgr2=0x7f609c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:02.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.840+0000 7f60a3188700 1 -- 172.21.15.189:0/1449156159 shutdown_connections 2024-04-06T12:13:02.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:02.840+0000 7f60a3188700 1 -- 172.21.15.189:0/1449156159 wait complete. 2024-04-06T12:13:02.840 INFO:teuthology.orchestra.run.smithi189.stderr:listed 0 entries 2024-04-06T12:13:02.851 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-04-06T12:13:02.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:02.851 DEBUG:teuthology.orchestra.run.smithi189:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-04-06T12:13:02.888 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T12:13:02.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:02.888 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-06T12:13:02.946 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:13:02.946 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:13:02.968 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:13:02.968 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:13:03.028 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:13:03.028 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:13:03.089 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:13:03.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.212+0000 7faee970d700 1 Processor -- start 2024-04-06T12:13:03.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.212+0000 7faee970d700 1 -- start start 2024-04-06T12:13:03.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.213+0000 7faee970d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faee415d590 0x7faee40b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:03.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.213+0000 7faee970d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faee40b4ed0 0x7faee40b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:03.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.213+0000 7faee970d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee40b5cc0 con 0x7faee40b6580 2024-04-06T12:13:03.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.213+0000 7faee970d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee40b5890 con 0x7faee40b4ed0 2024-04-06T12:13:03.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.213+0000 7faee970d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee40d1500 con 0x7faee415d590 2024-04-06T12:13:03.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.213+0000 7faee27fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faee40b4ed0 0x7faee40b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:03.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.213+0000 7faee37fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7faee40b6580 0x7faee40b6a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42662/0 (socket says 172.21.15.189:42662) 2024-04-06T12:13:03.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.213+0000 7faee27fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faee40b4ed0 0x7faee40b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44794/0 (socket says 172.21.15.189:44794) 2024-04-06T12:13:03.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.213+0000 7faee37fe700 1 -- 172.21.15.189:0/3715807617 learned_addr learned my addr 172.21.15.189:0/3715807617 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:03.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.213+0000 7faee2ffd700 1 --2- 172.21.15.189:0/3715807617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faee415d590 0x7faee40b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:03.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.213+0000 7faee27fc700 1 -- 172.21.15.189:0/3715807617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faee415d590 msgr2=0x7faee40b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:03.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.213+0000 7faee27fc700 1 --2- 172.21.15.189:0/3715807617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faee415d590 0x7faee40b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.213+0000 7faee27fc700 1 -- 172.21.15.189:0/3715807617 >> v1:172.21.15.189:6789/0 conn(0x7faee40b6580 legacy=0x7faee40b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:03.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.213+0000 7faee1ffb700 1 -- 172.21.15.189:0/3715807617 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647511676 0 0) 0x7faee40b5cc0 con 0x7faee40b6580 2024-04-06T12:13:03.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.213+0000 7faee27fc700 1 -- 172.21.15.189:0/3715807617 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee40d17e0 con 0x7faee40b4ed0 2024-04-06T12:13:03.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.214+0000 7faee27fc700 1 --2- 172.21.15.189:0/3715807617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faee40b4ed0 0x7faee40b5350 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7faed4009870 tx=0x7faed4009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b3935d0053e604 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:03.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.214+0000 7faee1ffb700 1 -- 172.21.15.189:0/3715807617 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4011830 con 0x7faee40b4ed0 2024-04-06T12:13:03.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.214+0000 7faee1ffb700 1 -- 172.21.15.189:0/3715807617 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed4011990 con 0x7faee40b4ed0 2024-04-06T12:13:03.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.214+0000 7faee1ffb700 1 -- 172.21.15.189:0/3715807617 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4011d20 con 0x7faee40b4ed0 2024-04-06T12:13:03.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.214+0000 7faee970d700 1 -- 172.21.15.189:0/3715807617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faee40b4ed0 msgr2=0x7faee40b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:03.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.214+0000 7faee970d700 1 --2- 172.21.15.189:0/3715807617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faee40b4ed0 0x7faee40b5350 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7faed4009870 tx=0x7faed4009b80 comp rx=0 tx=0).stop 2024-04-06T12:13:03.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.214+0000 7faee970d700 1 -- 172.21.15.189:0/3715807617 shutdown_connections 2024-04-06T12:13:03.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.214+0000 7faee970d700 1 --2- 172.21.15.189:0/3715807617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faee415d590 0x7faee40b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.214+0000 7faee970d700 1 --2- 172.21.15.189:0/3715807617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faee40b4ed0 0x7faee40b5350 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.214+0000 7faee970d700 1 -- 172.21.15.189:0/3715807617 >> 172.21.15.189:0/3715807617 conn(0x7faee406bbc0 msgr2=0x7faee406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:03.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.214+0000 7faee970d700 1 -- 172.21.15.189:0/3715807617 shutdown_connections 2024-04-06T12:13:03.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.214+0000 7faee970d700 1 -- 172.21.15.189:0/3715807617 wait complete. 2024-04-06T12:13:03.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.216+0000 7faee970d700 1 Processor -- start 2024-04-06T12:13:03.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.216+0000 7faee970d700 1 -- start start 2024-04-06T12:13:03.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.216+0000 7faee970d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faee40b4ed0 0x7faee4149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:03.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.216+0000 7faee970d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faee40b6580 0x7faee41f2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:03.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.216+0000 7faee970d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faee415d590 0x7faee40b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:03.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.216+0000 7faee970d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee414ae20 con 0x7faee40b6580 2024-04-06T12:13:03.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.216+0000 7faee970d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee414a580 con 0x7faee40b4ed0 2024-04-06T12:13:03.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.216+0000 7faee970d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee414a860 con 0x7faee415d590 2024-04-06T12:13:03.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.216+0000 7faee27fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faee40b6580 0x7faee41f2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:03.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.216+0000 7faee27fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faee40b6580 0x7faee41f2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44800/0 (socket says 172.21.15.189:44800) 2024-04-06T12:13:03.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.217+0000 7faee27fc700 1 -- 172.21.15.189:0/3299637369 learned_addr learned my addr 172.21.15.189:0/3299637369 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:03.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.217+0000 7faee2ffd700 1 --2- 172.21.15.189:0/3299637369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faee40b4ed0 0x7faee4149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:03.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.217+0000 7faee37fe700 1 --2- 172.21.15.189:0/3299637369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faee415d590 0x7faee40b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:03.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.217+0000 7faee27fc700 1 -- 172.21.15.189:0/3299637369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faee415d590 msgr2=0x7faee40b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:03.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.217+0000 7faee27fc700 1 --2- 172.21.15.189:0/3299637369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faee415d590 0x7faee40b6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.217+0000 7faee27fc700 1 -- 172.21.15.189:0/3299637369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faee40b4ed0 msgr2=0x7faee4149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:03.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.217+0000 7faee27fc700 1 --2- 172.21.15.189:0/3299637369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faee40b4ed0 0x7faee4149f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.217+0000 7faee27fc700 1 -- 172.21.15.189:0/3299637369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed40094a0 con 0x7faee40b6580 2024-04-06T12:13:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.217+0000 7faee2ffd700 1 --2- 172.21.15.189:0/3299637369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faee40b4ed0 0x7faee4149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.217+0000 7faee27fc700 1 --2- 172.21.15.189:0/3299637369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faee40b6580 0x7faee41f2960 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7faed4009bc0 tx=0x7faed4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.217+0000 7faecb7fe700 1 -- 172.21.15.189:0/3299637369 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed40038a0 con 0x7faee40b6580 2024-04-06T12:13:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.217+0000 7faecb7fe700 1 -- 172.21.15.189:0/3299637369 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed401f070 con 0x7faee40b6580 2024-04-06T12:13:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.217+0000 7faecb7fe700 1 -- 172.21.15.189:0/3299637369 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4003c60 con 0x7faee40b6580 2024-04-06T12:13:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.217+0000 7faee970d700 1 -- 172.21.15.189:0/3299637369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee414ab40 con 0x7faee40b6580 2024-04-06T12:13:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.217+0000 7faee970d700 1 -- 172.21.15.189:0/3299637369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee40b7f50 con 0x7faee40b6580 2024-04-06T12:13:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.218+0000 7faee970d700 1 -- 172.21.15.189:0/3299637369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee41e6100 con 0x7faee40b6580 2024-04-06T12:13:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.219+0000 7faecb7fe700 1 -- 172.21.15.189:0/3299637369 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7faed401a430 con 0x7faee40b6580 2024-04-06T12:13:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.219+0000 7faecb7fe700 1 --2- 172.21.15.189:0/3299637369 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faed006ac90 0x7faed006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.219+0000 7faecb7fe700 1 -- 172.21.15.189:0/3299637369 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7faed4021070 con 0x7faee40b6580 2024-04-06T12:13:03.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.219+0000 7faee2ffd700 1 --2- 172.21.15.189:0/3299637369 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faed006ac90 0x7faed006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:03.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.219+0000 7faee2ffd700 1 --2- 172.21.15.189:0/3299637369 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faed006ac90 0x7faed006d150 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7faed8007a50 tx=0x7faed8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:03.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.223+0000 7faecb7fe700 1 -- 172.21.15.189:0/3299637369 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7faee41e6100 con 0x7faee40b6580 2024-04-06T12:13:03.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.416+0000 7faee970d700 1 -- 172.21.15.189:0/3299637369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faee41e6100 con 0x7faee40b6580 2024-04-06T12:13:03.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.417+0000 7faecb7fe700 1 -- 172.21.15.189:0/3299637369 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18314 (secure 0 0 0) 0x7faee41e6100 con 0x7faee40b6580 2024-04-06T12:13:03.417 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:13:03.417 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":26,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:54.376742+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:13:03.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.420+0000 7faee970d700 1 -- 172.21.15.189:0/3299637369 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faed006ac90 msgr2=0x7faed006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:03.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.420+0000 7faee970d700 1 --2- 172.21.15.189:0/3299637369 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faed006ac90 0x7faed006d150 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7faed8007a50 tx=0x7faed8008040 comp rx=0 tx=0).stop 2024-04-06T12:13:03.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.420+0000 7faee970d700 1 -- 172.21.15.189:0/3299637369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faee40b6580 msgr2=0x7faee41f2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:03.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.420+0000 7faee970d700 1 --2- 172.21.15.189:0/3299637369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faee40b6580 0x7faee41f2960 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7faed4009bc0 tx=0x7faed4004750 comp rx=0 tx=0).stop 2024-04-06T12:13:03.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.420+0000 7faee970d700 1 -- 172.21.15.189:0/3299637369 shutdown_connections 2024-04-06T12:13:03.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.420+0000 7faee970d700 1 --2- 172.21.15.189:0/3299637369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faee415d590 0x7faee40b6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.420+0000 7faee970d700 1 --2- 172.21.15.189:0/3299637369 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faed006ac90 0x7faed006d150 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.420+0000 7faee970d700 1 --2- 172.21.15.189:0/3299637369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faee40b4ed0 0x7faee4149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.420+0000 7faee970d700 1 --2- 172.21.15.189:0/3299637369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faee40b6580 0x7faee41f2960 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.420+0000 7faee970d700 1 -- 172.21.15.189:0/3299637369 >> 172.21.15.189:0/3299637369 conn(0x7faee406bbc0 msgr2=0x7faee40d05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:03.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.420+0000 7faee970d700 1 -- 172.21.15.189:0/3299637369 shutdown_connections 2024-04-06T12:13:03.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.420+0000 7faee970d700 1 -- 172.21.15.189:0/3299637369 wait complete. 2024-04-06T12:13:03.432 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:12:02.280866+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:12:03.156593+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-06T12:12:06.237184+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-06T12:13:03.432 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:13:03.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.558+0000 7f6accc3b700 1 Processor -- start 2024-04-06T12:13:03.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.558+0000 7f6accc3b700 1 -- start start 2024-04-06T12:13:03.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.559+0000 7f6accc3b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6ac806ae40 0x7f6ac80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:03.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.559+0000 7f6accc3b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac80b3990 con 0x7f6ac815ddf0 2024-04-06T12:13:03.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.559+0000 7f6accc3b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac80b3060 con 0x7f6ac815d7b0 2024-04-06T12:13:03.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.559+0000 7f6accc3b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ac815e950 con 0x7f6ac806ae40 2024-04-06T12:13:03.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.559+0000 7f6ac6d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6ac815ddf0 0x7f6ac815e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42664/0 (socket says 172.21.15.189:42664) 2024-04-06T12:13:03.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.559+0000 7f6ac6d9d700 1 -- 172.21.15.189:0/1858913809 learned_addr learned my addr 172.21.15.189:0/1858913809 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:03.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.559+0000 7f6ac559a700 1 -- 172.21.15.189:0/1858913809 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190381279 0 0) 0x7f6ac80b3990 con 0x7f6ac815ddf0 2024-04-06T12:13:03.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.559+0000 7f6ac559a700 1 -- 172.21.15.189:0/1858913809 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ab40035b0 con 0x7f6ac815ddf0 2024-04-06T12:13:03.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.559+0000 7f6ac5d9b700 1 --2- 172.21.15.189:0/1858913809 >> v2:172.21.15.192:3300/0 conn(0x7f6ac806ae40 0x7f6ac80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:03.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.559+0000 7f6ac559a700 1 -- 172.21.15.189:0/1858913809 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1982740727 0 0) 0x7f6ab40035b0 con 0x7f6ac815ddf0 2024-04-06T12:13:03.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.559+0000 7f6ac559a700 1 -- 172.21.15.189:0/1858913809 >> v2:172.21.15.192:3300/0 conn(0x7f6ac806ae40 msgr2=0x7f6ac80b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:03.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.559+0000 7f6ac559a700 1 --2- 172.21.15.189:0/1858913809 >> v2:172.21.15.192:3300/0 conn(0x7f6ac806ae40 0x7f6ac80b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac559a700 1 -- 172.21.15.189:0/1858913809 >> v1:172.21.15.192:6789/0 conn(0x7f6ac815d7b0 legacy=0x7f6ac806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:03.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac559a700 1 -- 172.21.15.189:0/1858913809 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac80b3340 con 0x7f6ac815ddf0 2024-04-06T12:13:03.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac559a700 1 -- 172.21.15.189:0/1858913809 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f6ac00019f0 con 0x7f6ac815ddf0 2024-04-06T12:13:03.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac559a700 1 -- 172.21.15.189:0/1858913809 >> v1:172.21.15.189:6789/0 conn(0x7f6ac815ddf0 legacy=0x7f6ac815e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:03.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac559a700 1 --2- 172.21.15.189:0/1858913809 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ab4003a00 0x7f6ab4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:03.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac559a700 1 --2- 172.21.15.189:0/1858913809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ab4004850 0x7f6ab4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:03.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac559a700 1 --2- 172.21.15.189:0/1858913809 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ab4005330 0x7f6ab400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:03.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac559a700 1 -- 172.21.15.189:0/1858913809 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab400c1d0 con 0x7f6ab4004850 2024-04-06T12:13:03.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac559a700 1 -- 172.21.15.189:0/1858913809 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab400c340 con 0x7f6ab4003a00 2024-04-06T12:13:03.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac559a700 1 -- 172.21.15.189:0/1858913809 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab400c500 con 0x7f6ab4005330 2024-04-06T12:13:03.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac5d9b700 1 --2- 172.21.15.189:0/1858913809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ab4004850 0x7f6ab4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:03.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac659c700 1 --2- 172.21.15.189:0/1858913809 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ab4003a00 0x7f6ab4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:03.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac6d9d700 1 --2- 172.21.15.189:0/1858913809 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ab4005330 0x7f6ab400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:03.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac5d9b700 1 -- 172.21.15.189:0/1858913809 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ab4005330 msgr2=0x7f6ab400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:03.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac5d9b700 1 --2- 172.21.15.189:0/1858913809 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ab4005330 0x7f6ab400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac5d9b700 1 -- 172.21.15.189:0/1858913809 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ab4003a00 msgr2=0x7f6ab4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:03.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac5d9b700 1 --2- 172.21.15.189:0/1858913809 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ab4003a00 0x7f6ab4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac5d9b700 1 -- 172.21.15.189:0/1858913809 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab40035b0 con 0x7f6ab4004850 2024-04-06T12:13:03.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac6d9d700 1 -- 172.21.15.189:0/1858913809 reap_dead start 2024-04-06T12:13:03.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac5d9b700 1 --2- 172.21.15.189:0/1858913809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ab4004850 0x7f6ab4004cd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6ac80b4360 tx=0x7f6ab4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be61f35b27db3d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:03.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac559a700 1 -- 172.21.15.189:0/1858913809 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab800c070 con 0x7f6ab4004850 2024-04-06T12:13:03.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac559a700 1 -- 172.21.15.189:0/1858913809 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab8004020 con 0x7f6ab4004850 2024-04-06T12:13:03.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.560+0000 7f6ac559a700 1 -- 172.21.15.189:0/1858913809 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab80043b0 con 0x7f6ab4004850 2024-04-06T12:13:03.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.561+0000 7f6accc3b700 1 -- 172.21.15.189:0/1858913809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ab4004850 msgr2=0x7f6ab4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:03.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.561+0000 7f6accc3b700 1 --2- 172.21.15.189:0/1858913809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ab4004850 0x7f6ab4004cd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6ac80b4360 tx=0x7f6ab4005880 comp rx=0 tx=0).stop 2024-04-06T12:13:03.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.561+0000 7f6accc3b700 1 -- 172.21.15.189:0/1858913809 shutdown_connections 2024-04-06T12:13:03.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.561+0000 7f6accc3b700 1 --2- 172.21.15.189:0/1858913809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ab4004850 0x7f6ab4004cd0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.561+0000 7f6accc3b700 1 -- 172.21.15.189:0/1858913809 >> 172.21.15.189:0/1858913809 conn(0x7f6ac80d00b0 msgr2=0x7f6ac80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:03.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.561+0000 7f6accc3b700 1 -- 172.21.15.189:0/1858913809 shutdown_connections 2024-04-06T12:13:03.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.561+0000 7f6accc3b700 1 -- 172.21.15.189:0/1858913809 wait complete. 2024-04-06T12:13:03.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.562+0000 7f6accc3b700 1 Processor -- start 2024-04-06T12:13:03.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.562+0000 7f6accc3b700 1 -- start start 2024-04-06T12:13:03.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.563+0000 7f6accc3b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ab4004850 0x7f6ac8151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:03.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.563+0000 7f6accc3b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac815bc90 0x7f6ac815c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.563+0000 7f6accc3b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ac8158990 0x7f6ac8158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.563+0000 7f6accc3b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac8159350 con 0x7f6ac815bc90 2024-04-06T12:13:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.563+0000 7f6accc3b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac81522a0 con 0x7f6ab4004850 2024-04-06T12:13:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.563+0000 7f6accc3b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac80cdbd0 con 0x7f6ac8158990 2024-04-06T12:13:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.563+0000 7f6ac5d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac815bc90 0x7f6ac815c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.563+0000 7f6ac5d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac815bc90 0x7f6ac815c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44810/0 (socket says 172.21.15.189:44810) 2024-04-06T12:13:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.563+0000 7f6ac5d9b700 1 -- 172.21.15.189:0/2203722468 learned_addr learned my addr 172.21.15.189:0/2203722468 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.563+0000 7f6ac659c700 1 --2- 172.21.15.189:0/2203722468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ab4004850 0x7f6ac8151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.563+0000 7f6ac6d9d700 1 --2- 172.21.15.189:0/2203722468 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ac8158990 0x7f6ac8158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.563+0000 7f6ac5d9b700 1 -- 172.21.15.189:0/2203722468 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ac8158990 msgr2=0x7f6ac8158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.563+0000 7f6ac5d9b700 1 --2- 172.21.15.189:0/2203722468 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ac8158990 0x7f6ac8158e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.563+0000 7f6ac5d9b700 1 -- 172.21.15.189:0/2203722468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ab4004850 msgr2=0x7f6ac8151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.563+0000 7f6ac5d9b700 1 --2- 172.21.15.189:0/2203722468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ab4004850 0x7f6ac8151d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.563+0000 7f6ac5d9b700 1 -- 172.21.15.189:0/2203722468 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac815f080 con 0x7f6ac815bc90 2024-04-06T12:13:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.564+0000 7f6ac5d9b700 1 --2- 172.21.15.189:0/2203722468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac815bc90 0x7f6ac815c110 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8015000 tx=0x7f6ab80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.564+0000 7f6abeffd700 1 -- 172.21.15.189:0/2203722468 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab80045c0 con 0x7f6ac815bc90 2024-04-06T12:13:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.564+0000 7f6abeffd700 1 -- 172.21.15.189:0/2203722468 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab8021070 con 0x7f6ac815bc90 2024-04-06T12:13:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.564+0000 7f6abeffd700 1 -- 172.21.15.189:0/2203722468 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab8003d10 con 0x7f6ac815bc90 2024-04-06T12:13:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.564+0000 7f6accc3b700 1 -- 172.21.15.189:0/2203722468 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac815f360 con 0x7f6ac815bc90 2024-04-06T12:13:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.564+0000 7f6accc3b700 1 -- 172.21.15.189:0/2203722468 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac815f920 con 0x7f6ac815bc90 2024-04-06T12:13:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.565+0000 7f6accc3b700 1 -- 172.21.15.189:0/2203722468 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac815fab0 con 0x7f6ac815bc90 2024-04-06T12:13:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.565+0000 7f6abeffd700 1 -- 172.21.15.189:0/2203722468 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6ab8023030 con 0x7f6ac815bc90 2024-04-06T12:13:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.566+0000 7f6abeffd700 1 --2- 172.21.15.189:0/2203722468 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6aa806ac90 0x7f6aa806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.566+0000 7f6abeffd700 1 -- 172.21.15.189:0/2203722468 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f6ab808a230 con 0x7f6ac815bc90 2024-04-06T12:13:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.566+0000 7f6ac659c700 1 --2- 172.21.15.189:0/2203722468 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6aa806ac90 0x7f6aa806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.566+0000 7f6ac659c700 1 --2- 172.21.15.189:0/2203722468 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6aa806ac90 0x7f6aa806d150 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0000c00 tx=0x7f6ab0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:03.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.570+0000 7f6abeffd700 1 -- 172.21.15.189:0/2203722468 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6ab80550e0 con 0x7f6ac815bc90 2024-04-06T12:13:03.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.763+0000 7f6accc3b700 1 -- 172.21.15.189:0/2203722468 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6ac8067bb0 con 0x7f6ac815bc90 2024-04-06T12:13:03.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.764+0000 7f6abeffd700 1 -- 172.21.15.189:0/2203722468 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6ab8058700 con 0x7f6ac815bc90 2024-04-06T12:13:03.763 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:13:03.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.766+0000 7f6accc3b700 1 -- 172.21.15.189:0/2203722468 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6aa806ac90 msgr2=0x7f6aa806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:03.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.766+0000 7f6accc3b700 1 --2- 172.21.15.189:0/2203722468 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6aa806ac90 0x7f6aa806d150 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0000c00 tx=0x7f6ab0009040 comp rx=0 tx=0).stop 2024-04-06T12:13:03.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.766+0000 7f6accc3b700 1 -- 172.21.15.189:0/2203722468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac815bc90 msgr2=0x7f6ac815c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:03.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.766+0000 7f6accc3b700 1 --2- 172.21.15.189:0/2203722468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac815bc90 0x7f6ac815c110 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8015000 tx=0x7f6ab80047c0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.766+0000 7f6accc3b700 1 -- 172.21.15.189:0/2203722468 shutdown_connections 2024-04-06T12:13:03.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.766+0000 7f6accc3b700 1 --2- 172.21.15.189:0/2203722468 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ac8158990 0x7f6ac8158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.766+0000 7f6accc3b700 1 --2- 172.21.15.189:0/2203722468 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6aa806ac90 0x7f6aa806d150 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.766+0000 7f6accc3b700 1 --2- 172.21.15.189:0/2203722468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ab4004850 0x7f6ac8151d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.766+0000 7f6accc3b700 1 --2- 172.21.15.189:0/2203722468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac815bc90 0x7f6ac815c110 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.766+0000 7f6accc3b700 1 -- 172.21.15.189:0/2203722468 >> 172.21.15.189:0/2203722468 conn(0x7f6ac80d00b0 msgr2=0x7f6ac806eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:03.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.766+0000 7f6accc3b700 1 -- 172.21.15.189:0/2203722468 shutdown_connections 2024-04-06T12:13:03.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.766+0000 7f6accc3b700 1 -- 172.21.15.189:0/2203722468 wait complete. 2024-04-06T12:13:03.779 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:13:03.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.903+0000 7fd9786e7700 1 Processor -- start 2024-04-06T12:13:03.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.904+0000 7fd9786e7700 1 -- start start 2024-04-06T12:13:03.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.904+0000 7fd9786e7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9700b4660 0x7fd9700b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:03.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.904+0000 7fd9786e7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9700cdbe0 con 0x7fd9700b2fc0 2024-04-06T12:13:03.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.904+0000 7fd9786e7700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9700b5000 con 0x7fd97015a550 2024-04-06T12:13:03.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.904+0000 7fd9786e7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd97015a940 con 0x7fd9700b4660 2024-04-06T12:13:03.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.905+0000 7fd976c84700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd9700b2fc0 0x7fd9700b3430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42674/0 (socket says 172.21.15.189:42674) 2024-04-06T12:13:03.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.905+0000 7fd976c84700 1 -- 172.21.15.189:0/1071426500 learned_addr learned my addr 172.21.15.189:0/1071426500 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.905+0000 7fd975c82700 1 --2- 172.21.15.189:0/1071426500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9700b4660 0x7fd9700b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.905+0000 7fd975481700 1 -- 172.21.15.189:0/1071426500 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596905200 0 0) 0x7fd9700cdbe0 con 0x7fd9700b2fc0 2024-04-06T12:13:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.905+0000 7fd975481700 1 -- 172.21.15.189:0/1071426500 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd95c0035b0 con 0x7fd9700b2fc0 2024-04-06T12:13:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.905+0000 7fd975481700 1 -- 172.21.15.189:0/1071426500 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298135174 0 0) 0x7fd9700b5000 con 0x7fd97015a550 2024-04-06T12:13:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.905+0000 7fd975481700 1 -- 172.21.15.189:0/1071426500 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9700cdbe0 con 0x7fd97015a550 2024-04-06T12:13:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.905+0000 7fd975c82700 1 -- 172.21.15.189:0/1071426500 >> v1:172.21.15.192:6789/0 conn(0x7fd97015a550 legacy=0x7fd9700cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.905+0000 7fd975c82700 1 -- 172.21.15.189:0/1071426500 >> v1:172.21.15.189:6789/0 conn(0x7fd9700b2fc0 legacy=0x7fd9700b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.905+0000 7fd975481700 1 -- 172.21.15.189:0/1071426500 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1728158648 0 0) 0x7fd95c0035b0 con 0x7fd9700b2fc0 2024-04-06T12:13:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.905+0000 7fd975c82700 1 -- 172.21.15.189:0/1071426500 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97015ac00 con 0x7fd9700b4660 2024-04-06T12:13:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.905+0000 7fd975c82700 1 --2- 172.21.15.189:0/1071426500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9700b4660 0x7fd9700b4ac0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd964009870 tx=0x7fd964009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3151a03b0699453 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.905+0000 7fd975481700 1 -- 172.21.15.189:0/1071426500 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9680019f0 con 0x7fd9700b4660 2024-04-06T12:13:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.906+0000 7fd975481700 1 -- 172.21.15.189:0/1071426500 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd964004030 con 0x7fd9700b4660 2024-04-06T12:13:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.906+0000 7fd975481700 1 -- 172.21.15.189:0/1071426500 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9640043c0 con 0x7fd9700b4660 2024-04-06T12:13:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.906+0000 7fd9786e7700 1 -- 172.21.15.189:0/1071426500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9700b4660 msgr2=0x7fd9700b4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:03.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.906+0000 7fd9786e7700 1 --2- 172.21.15.189:0/1071426500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9700b4660 0x7fd9700b4ac0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd964009870 tx=0x7fd964009b80 comp rx=0 tx=0).stop 2024-04-06T12:13:03.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.906+0000 7fd9786e7700 1 -- 172.21.15.189:0/1071426500 shutdown_connections 2024-04-06T12:13:03.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.906+0000 7fd9786e7700 1 --2- 172.21.15.189:0/1071426500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9700b4660 0x7fd9700b4ac0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.906+0000 7fd9786e7700 1 -- 172.21.15.189:0/1071426500 >> 172.21.15.189:0/1071426500 conn(0x7fd9700d00b0 msgr2=0x7fd9700d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:03.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.906+0000 7fd9786e7700 1 -- 172.21.15.189:0/1071426500 shutdown_connections 2024-04-06T12:13:03.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.906+0000 7fd9786e7700 1 -- 172.21.15.189:0/1071426500 wait complete. 2024-04-06T12:13:03.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.907+0000 7fd9786e7700 1 Processor -- start 2024-04-06T12:13:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.907+0000 7fd9786e7700 1 -- start start 2024-04-06T12:13:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.908+0000 7fd9786e7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9700b2fc0 0x7fd97014fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.908+0000 7fd9786e7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9700b4660 0x7fd970152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.908+0000 7fd9786e7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd97006eb50 0x7fd97006f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.908+0000 7fd9786e7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97006f540 con 0x7fd9700b2fc0 2024-04-06T12:13:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.908+0000 7fd9786e7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9701587a0 con 0x7fd97006eb50 2024-04-06T12:13:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.908+0000 7fd9786e7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd97014ff70 con 0x7fd9700b4660 2024-04-06T12:13:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.908+0000 7fd976483700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9700b2fc0 0x7fd97014fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.908+0000 7fd976483700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9700b2fc0 0x7fd97014fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44820/0 (socket says 172.21.15.189:44820) 2024-04-06T12:13:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.908+0000 7fd976483700 1 -- 172.21.15.189:0/2067533183 learned_addr learned my addr 172.21.15.189:0/2067533183 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.908+0000 7fd976c84700 1 --2- 172.21.15.189:0/2067533183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd97006eb50 0x7fd97006f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.908+0000 7fd975c82700 1 --2- 172.21.15.189:0/2067533183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9700b4660 0x7fd970152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.908+0000 7fd976483700 1 -- 172.21.15.189:0/2067533183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9700b4660 msgr2=0x7fd970152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.908+0000 7fd976483700 1 --2- 172.21.15.189:0/2067533183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9700b4660 0x7fd970152490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.908+0000 7fd976483700 1 -- 172.21.15.189:0/2067533183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd97006eb50 msgr2=0x7fd97006f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.908+0000 7fd976483700 1 --2- 172.21.15.189:0/2067533183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd97006eb50 0x7fd97006f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.908+0000 7fd976483700 1 -- 172.21.15.189:0/2067533183 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9640094a0 con 0x7fd9700b2fc0 2024-04-06T12:13:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.909+0000 7fd975c82700 1 --2- 172.21.15.189:0/2067533183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9700b4660 0x7fd970152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.909+0000 7fd976483700 1 --2- 172.21.15.189:0/2067533183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9700b2fc0 0x7fd97014fa30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd96c00c500 tx=0x7fd96c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.909+0000 7fd962ffd700 1 -- 172.21.15.189:0/2067533183 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c0052d0 con 0x7fd9700b2fc0 2024-04-06T12:13:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.909+0000 7fd962ffd700 1 -- 172.21.15.189:0/2067533183 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96c005450 con 0x7fd9700b2fc0 2024-04-06T12:13:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.909+0000 7fd9786e7700 1 -- 172.21.15.189:0/2067533183 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd970157b90 con 0x7fd9700b2fc0 2024-04-06T12:13:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.909+0000 7fd962ffd700 1 -- 172.21.15.189:0/2067533183 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c0038c0 con 0x7fd9700b2fc0 2024-04-06T12:13:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.909+0000 7fd9786e7700 1 -- 172.21.15.189:0/2067533183 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd970158150 con 0x7fd9700b2fc0 2024-04-06T12:13:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.910+0000 7fd9786e7700 1 -- 172.21.15.189:0/2067533183 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd970067bb0 con 0x7fd9700b2fc0 2024-04-06T12:13:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.910+0000 7fd962ffd700 1 -- 172.21.15.189:0/2067533183 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd96c003a20 con 0x7fd9700b2fc0 2024-04-06T12:13:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.911+0000 7fd962ffd700 1 --2- 172.21.15.189:0/2067533183 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd95406ac90 0x7fd95406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.911+0000 7fd962ffd700 1 -- 172.21.15.189:0/2067533183 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd96c088b90 con 0x7fd9700b2fc0 2024-04-06T12:13:03.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.911+0000 7fd975c82700 1 --2- 172.21.15.189:0/2067533183 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd95406ac90 0x7fd95406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:03.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.911+0000 7fd975c82700 1 --2- 172.21.15.189:0/2067533183 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd95406ac90 0x7fd95406d150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd9640090b0 tx=0x7fd964008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:03.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:03.915+0000 7fd962ffd700 1 -- 172.21.15.189:0/2067533183 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd96c053910 con 0x7fd9700b2fc0 2024-04-06T12:13:04.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.108+0000 7fd9786e7700 1 -- 172.21.15.189:0/2067533183 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd970074990 con 0x7fd9700b2fc0 2024-04-06T12:13:04.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.108+0000 7fd962ffd700 1 -- 172.21.15.189:0/2067533183 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd96c017020 con 0x7fd9700b2fc0 2024-04-06T12:13:04.107 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:13:04.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.110+0000 7fd9786e7700 1 -- 172.21.15.189:0/2067533183 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd95406ac90 msgr2=0x7fd95406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.110+0000 7fd9786e7700 1 --2- 172.21.15.189:0/2067533183 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd95406ac90 0x7fd95406d150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd9640090b0 tx=0x7fd964008030 comp rx=0 tx=0).stop 2024-04-06T12:13:04.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.110+0000 7fd9786e7700 1 -- 172.21.15.189:0/2067533183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9700b2fc0 msgr2=0x7fd97014fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.110+0000 7fd9786e7700 1 --2- 172.21.15.189:0/2067533183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9700b2fc0 0x7fd97014fa30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd96c00c500 tx=0x7fd96c00c810 comp rx=0 tx=0).stop 2024-04-06T12:13:04.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.110+0000 7fd9786e7700 1 -- 172.21.15.189:0/2067533183 shutdown_connections 2024-04-06T12:13:04.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.110+0000 7fd9786e7700 1 --2- 172.21.15.189:0/2067533183 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd95406ac90 0x7fd95406d150 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.110+0000 7fd9786e7700 1 --2- 172.21.15.189:0/2067533183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd97006eb50 0x7fd97006f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.111+0000 7fd9786e7700 1 --2- 172.21.15.189:0/2067533183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9700b2fc0 0x7fd97014fa30 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.111+0000 7fd9786e7700 1 --2- 172.21.15.189:0/2067533183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9700b4660 0x7fd970152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.111+0000 7fd9786e7700 1 -- 172.21.15.189:0/2067533183 >> 172.21.15.189:0/2067533183 conn(0x7fd9700d00b0 msgr2=0x7fd97006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:04.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.111+0000 7fd9786e7700 1 -- 172.21.15.189:0/2067533183 shutdown_connections 2024-04-06T12:13:04.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.111+0000 7fd9786e7700 1 -- 172.21.15.189:0/2067533183 wait complete. 2024-04-06T12:13:04.121 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:13:04.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.248+0000 7f5db88ac700 1 Processor -- start 2024-04-06T12:13:04.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.248+0000 7f5db88ac700 1 -- start start 2024-04-06T12:13:04.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.250+0000 7f5db88ac700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5db40d17a0 0x7f5db406b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.250+0000 7f5db88ac700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db406d850 con 0x7f5db4146dd0 2024-04-06T12:13:04.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.250+0000 7f5db88ac700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db406c0a0 con 0x7f5db40cedf0 2024-04-06T12:13:04.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.250+0000 7f5db88ac700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5db406c380 con 0x7f5db40d17a0 2024-04-06T12:13:04.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.250+0000 7f5db259c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5db40d17a0 0x7f5db406b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.250+0000 7f5db2d9d700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f5db40cedf0 0x7f5db40cf1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:37638/0 (socket says 172.21.15.189:37638) 2024-04-06T12:13:04.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.250+0000 7f5db2d9d700 1 -- 172.21.15.189:0/3723567021 learned_addr learned my addr 172.21.15.189:0/3723567021 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:04.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.250+0000 7f5db259c700 1 -- 172.21.15.189:0/3723567021 >> v1:172.21.15.192:6789/0 conn(0x7f5db40cedf0 legacy=0x7f5db40cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.250+0000 7f5db259c700 1 -- 172.21.15.189:0/3723567021 >> v1:172.21.15.189:6789/0 conn(0x7f5db4146dd0 legacy=0x7f5db406d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.250+0000 7f5db259c700 1 -- 172.21.15.189:0/3723567021 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db406c660 con 0x7f5db40d17a0 2024-04-06T12:13:04.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.251+0000 7f5db259c700 1 --2- 172.21.15.189:0/3723567021 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5db40d17a0 0x7f5db406b7a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5da4009840 tx=0x7f5da4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f92258ad1751fab1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:04.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.251+0000 7f5db159a700 1 -- 172.21.15.189:0/3723567021 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da4011830 con 0x7f5db40d17a0 2024-04-06T12:13:04.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.251+0000 7f5db159a700 1 -- 172.21.15.189:0/3723567021 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da4011990 con 0x7f5db40d17a0 2024-04-06T12:13:04.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.251+0000 7f5db159a700 1 -- 172.21.15.189:0/3723567021 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da4011d20 con 0x7f5db40d17a0 2024-04-06T12:13:04.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.252+0000 7f5db88ac700 1 -- 172.21.15.189:0/3723567021 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5db40d17a0 msgr2=0x7f5db406b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.252+0000 7f5db88ac700 1 --2- 172.21.15.189:0/3723567021 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5db40d17a0 0x7f5db406b7a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5da4009840 tx=0x7f5da4009b50 comp rx=0 tx=0).stop 2024-04-06T12:13:04.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.252+0000 7f5db88ac700 1 -- 172.21.15.189:0/3723567021 shutdown_connections 2024-04-06T12:13:04.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.252+0000 7f5db88ac700 1 --2- 172.21.15.189:0/3723567021 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5db40d17a0 0x7f5db406b7a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.252+0000 7f5db88ac700 1 -- 172.21.15.189:0/3723567021 >> 172.21.15.189:0/3723567021 conn(0x7f5db40b2db0 msgr2=0x7f5db40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:04.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.252+0000 7f5db88ac700 1 -- 172.21.15.189:0/3723567021 shutdown_connections 2024-04-06T12:13:04.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.252+0000 7f5db88ac700 1 -- 172.21.15.189:0/3723567021 wait complete. 2024-04-06T12:13:04.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.253+0000 7f5db88ac700 1 Processor -- start 2024-04-06T12:13:04.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.253+0000 7f5db88ac700 1 -- start start 2024-04-06T12:13:04.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.254+0000 7f5db88ac700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5db40cedf0 0x7f5db4151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.254+0000 7f5db88ac700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5db40d17a0 0x7f5db41458b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.254+0000 7f5db88ac700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db4146dd0 0x7f5db415f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.254+0000 7f5db88ac700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db41f3340 con 0x7f5db4146dd0 2024-04-06T12:13:04.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.254+0000 7f5db88ac700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db41f3620 con 0x7f5db40d17a0 2024-04-06T12:13:04.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.254+0000 7f5db88ac700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db41f3900 con 0x7f5db40cedf0 2024-04-06T12:13:04.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.254+0000 7f5db2d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db4146dd0 0x7f5db415f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.254+0000 7f5db2d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db4146dd0 0x7f5db415f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44830/0 (socket says 172.21.15.189:44830) 2024-04-06T12:13:04.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.254+0000 7f5db2d9d700 1 -- 172.21.15.189:0/608651038 learned_addr learned my addr 172.21.15.189:0/608651038 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:04.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.254+0000 7f5db259c700 1 --2- 172.21.15.189:0/608651038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5db40cedf0 0x7f5db4151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.254+0000 7f5db1d9b700 1 --2- 172.21.15.189:0/608651038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5db40d17a0 0x7f5db41458b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.254+0000 7f5db2d9d700 1 -- 172.21.15.189:0/608651038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5db40cedf0 msgr2=0x7f5db4151b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.254+0000 7f5db2d9d700 1 --2- 172.21.15.189:0/608651038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5db40cedf0 0x7f5db4151b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.254+0000 7f5db2d9d700 1 -- 172.21.15.189:0/608651038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5db40d17a0 msgr2=0x7f5db41458b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.254+0000 7f5db2d9d700 1 --2- 172.21.15.189:0/608651038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5db40d17a0 0x7f5db41458b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.254+0000 7f5db2d9d700 1 -- 172.21.15.189:0/608651038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da40094a0 con 0x7f5db4146dd0 2024-04-06T12:13:04.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.254+0000 7f5db2d9d700 1 --2- 172.21.15.189:0/608651038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db4146dd0 0x7f5db415f660 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5dac00ca80 tx=0x7f5dac00cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:04.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.255+0000 7f5da2ffd700 1 -- 172.21.15.189:0/608651038 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac00a9e0 con 0x7f5db4146dd0 2024-04-06T12:13:04.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.255+0000 7f5da2ffd700 1 -- 172.21.15.189:0/608651038 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dac00ab40 con 0x7f5db4146dd0 2024-04-06T12:13:04.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.255+0000 7f5db88ac700 1 -- 172.21.15.189:0/608651038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db41f3be0 con 0x7f5db4146dd0 2024-04-06T12:13:04.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.255+0000 7f5da2ffd700 1 -- 172.21.15.189:0/608651038 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac012400 con 0x7f5db4146dd0 2024-04-06T12:13:04.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.255+0000 7f5db88ac700 1 -- 172.21.15.189:0/608651038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db4145df0 con 0x7f5db4146dd0 2024-04-06T12:13:04.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.256+0000 7f5db88ac700 1 -- 172.21.15.189:0/608651038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db4074990 con 0x7f5db4146dd0 2024-04-06T12:13:04.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.256+0000 7f5da2ffd700 1 -- 172.21.15.189:0/608651038 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5dac01b3b0 con 0x7f5db4146dd0 2024-04-06T12:13:04.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.256+0000 7f5da2ffd700 1 --2- 172.21.15.189:0/608651038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d9406ac90 0x7f5d9406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.256+0000 7f5da2ffd700 1 -- 172.21.15.189:0/608651038 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5dac089100 con 0x7f5db4146dd0 2024-04-06T12:13:04.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.256+0000 7f5db259c700 1 --2- 172.21.15.189:0/608651038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d9406ac90 0x7f5d9406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.257+0000 7f5db259c700 1 --2- 172.21.15.189:0/608651038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d9406ac90 0x7f5d9406d150 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5da40090b0 tx=0x7f5da4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:04.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.261+0000 7f5da2ffd700 1 -- 172.21.15.189:0/608651038 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5dac053fd0 con 0x7f5db4146dd0 2024-04-06T12:13:04.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.453+0000 7f5db88ac700 1 -- 172.21.15.189:0/608651038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5db40d1e20 con 0x7f5db4146dd0 2024-04-06T12:13:04.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.453+0000 7f5da2ffd700 1 -- 172.21.15.189:0/608651038 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5dac0575f0 con 0x7f5db4146dd0 2024-04-06T12:13:04.453 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:13:04.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.455+0000 7f5db88ac700 1 -- 172.21.15.189:0/608651038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d9406ac90 msgr2=0x7f5d9406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.455+0000 7f5db88ac700 1 --2- 172.21.15.189:0/608651038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d9406ac90 0x7f5d9406d150 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5da40090b0 tx=0x7f5da4008030 comp rx=0 tx=0).stop 2024-04-06T12:13:04.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.455+0000 7f5db88ac700 1 -- 172.21.15.189:0/608651038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db4146dd0 msgr2=0x7f5db415f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.455+0000 7f5db88ac700 1 --2- 172.21.15.189:0/608651038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db4146dd0 0x7f5db415f660 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5dac00ca80 tx=0x7f5dac00cd90 comp rx=0 tx=0).stop 2024-04-06T12:13:04.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.455+0000 7f5db88ac700 1 -- 172.21.15.189:0/608651038 shutdown_connections 2024-04-06T12:13:04.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.455+0000 7f5db88ac700 1 --2- 172.21.15.189:0/608651038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db4146dd0 0x7f5db415f660 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.455+0000 7f5db88ac700 1 --2- 172.21.15.189:0/608651038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5db40cedf0 0x7f5db4151b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.455+0000 7f5db88ac700 1 --2- 172.21.15.189:0/608651038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d9406ac90 0x7f5d9406d150 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.455+0000 7f5db88ac700 1 --2- 172.21.15.189:0/608651038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5db40d17a0 0x7f5db41458b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.455+0000 7f5db88ac700 1 -- 172.21.15.189:0/608651038 >> 172.21.15.189:0/608651038 conn(0x7f5db40b2db0 msgr2=0x7f5db40b58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:04.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.456+0000 7f5db88ac700 1 -- 172.21.15.189:0/608651038 shutdown_connections 2024-04-06T12:13:04.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.456+0000 7f5db88ac700 1 -- 172.21.15.189:0/608651038 wait complete. 2024-04-06T12:13:04.467 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-06T12:13:04.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.592+0000 7f60cbde2700 1 Processor -- start 2024-04-06T12:13:04.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.592+0000 7f60cbde2700 1 -- start start 2024-04-06T12:13:04.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.593+0000 7f60cbde2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f60c406ae40 0x7f60c40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.593+0000 7f60cbde2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c40b3990 con 0x7f60c415d7b0 2024-04-06T12:13:04.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.593+0000 7f60cbde2700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c40b3060 con 0x7f60c415ddf0 2024-04-06T12:13:04.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.593+0000 7f60cbde2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c415e950 con 0x7f60c406ae40 2024-04-06T12:13:04.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.593+0000 7f60c9b7e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f60c415d7b0 0x7f60c406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42690/0 (socket says 172.21.15.189:42690) 2024-04-06T12:13:04.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.593+0000 7f60c9b7e700 1 -- 172.21.15.189:0/2978191978 learned_addr learned my addr 172.21.15.189:0/2978191978 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:04.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.593+0000 7f60c937d700 1 --2- 172.21.15.189:0/2978191978 >> v2:172.21.15.192:3300/0 conn(0x7f60c406ae40 0x7f60c40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.593+0000 7f60c8b7c700 1 -- 172.21.15.189:0/2978191978 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673666511 0 0) 0x7f60c40b3990 con 0x7f60c415d7b0 2024-04-06T12:13:04.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.593+0000 7f60c8b7c700 1 -- 172.21.15.189:0/2978191978 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60b00035b0 con 0x7f60c415d7b0 2024-04-06T12:13:04.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.593+0000 7f60c8b7c700 1 -- 172.21.15.189:0/2978191978 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1309378415 0 0) 0x7f60b00035b0 con 0x7f60c415d7b0 2024-04-06T12:13:04.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.593+0000 7f60c8b7c700 1 -- 172.21.15.189:0/2978191978 >> v2:172.21.15.192:3300/0 conn(0x7f60c406ae40 msgr2=0x7f60c40b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:04.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.593+0000 7f60c8b7c700 1 --2- 172.21.15.189:0/2978191978 >> v2:172.21.15.192:3300/0 conn(0x7f60c406ae40 0x7f60c40b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.593+0000 7f60c8b7c700 1 -- 172.21.15.189:0/2978191978 >> v1:172.21.15.192:6789/0 conn(0x7f60c415ddf0 legacy=0x7f60c415e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.593+0000 7f60c8b7c700 1 -- 172.21.15.189:0/2978191978 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c40b3340 con 0x7f60c415d7b0 2024-04-06T12:13:04.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c8b7c700 1 -- 172.21.15.189:0/2978191978 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f60c0001670 con 0x7f60c415d7b0 2024-04-06T12:13:04.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c8b7c700 1 -- 172.21.15.189:0/2978191978 >> v1:172.21.15.189:6789/0 conn(0x7f60c415d7b0 legacy=0x7f60c406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c8b7c700 1 --2- 172.21.15.189:0/2978191978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60b0003a50 0x7f60b0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c8b7c700 1 --2- 172.21.15.189:0/2978191978 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60b0004850 0x7f60b0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c8b7c700 1 --2- 172.21.15.189:0/2978191978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60b0005260 0x7f60b000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c8b7c700 1 -- 172.21.15.189:0/2978191978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b000c180 con 0x7f60b0003a50 2024-04-06T12:13:04.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c8b7c700 1 -- 172.21.15.189:0/2978191978 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b000c2f0 con 0x7f60b0005260 2024-04-06T12:13:04.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c8b7c700 1 -- 172.21.15.189:0/2978191978 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b000c4b0 con 0x7f60b0004850 2024-04-06T12:13:04.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c9b7e700 1 --2- 172.21.15.189:0/2978191978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60b0003a50 0x7f60b0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c937d700 1 --2- 172.21.15.189:0/2978191978 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60b0004850 0x7f60b0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60ca37f700 1 --2- 172.21.15.189:0/2978191978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60b0005260 0x7f60b000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c9b7e700 1 -- 172.21.15.189:0/2978191978 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60b0004850 msgr2=0x7f60b0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:04.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c9b7e700 1 --2- 172.21.15.189:0/2978191978 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60b0004850 0x7f60b0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c9b7e700 1 -- 172.21.15.189:0/2978191978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60b0005260 msgr2=0x7f60b000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:04.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c9b7e700 1 --2- 172.21.15.189:0/2978191978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60b0005260 0x7f60b000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c9b7e700 1 -- 172.21.15.189:0/2978191978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b00035b0 con 0x7f60b0003a50 2024-04-06T12:13:04.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c937d700 1 --2- 172.21.15.189:0/2978191978 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60b0004850 0x7f60b0004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:04.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60ca37f700 1 --2- 172.21.15.189:0/2978191978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60b0005260 0x7f60b000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:04.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60ca37f700 1 -- 172.21.15.189:0/2978191978 reap_dead start 2024-04-06T12:13:04.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c9b7e700 1 --2- 172.21.15.189:0/2978191978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60b0003a50 0x7f60b0003e70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f60c0001eb0 tx=0x7f60b0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196865bc443d39b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:04.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c8b7c700 1 -- 172.21.15.189:0/2978191978 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c0007b70 con 0x7f60b0003a50 2024-04-06T12:13:04.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c8b7c700 1 -- 172.21.15.189:0/2978191978 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c000cca0 con 0x7f60b0003a50 2024-04-06T12:13:04.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.594+0000 7f60c8b7c700 1 -- 172.21.15.189:0/2978191978 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c0005ba0 con 0x7f60b0003a50 2024-04-06T12:13:04.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.595+0000 7f60cbde2700 1 -- 172.21.15.189:0/2978191978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60b0003a50 msgr2=0x7f60b0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.595+0000 7f60cbde2700 1 --2- 172.21.15.189:0/2978191978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60b0003a50 0x7f60b0003e70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f60c0001eb0 tx=0x7f60b0005880 comp rx=0 tx=0).stop 2024-04-06T12:13:04.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.595+0000 7f60cbde2700 1 -- 172.21.15.189:0/2978191978 shutdown_connections 2024-04-06T12:13:04.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.595+0000 7f60cbde2700 1 --2- 172.21.15.189:0/2978191978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60b0003a50 0x7f60b0003e70 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.595+0000 7f60cbde2700 1 -- 172.21.15.189:0/2978191978 >> 172.21.15.189:0/2978191978 conn(0x7f60c40d00b0 msgr2=0x7f60c40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:04.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.595+0000 7f60cbde2700 1 -- 172.21.15.189:0/2978191978 shutdown_connections 2024-04-06T12:13:04.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.595+0000 7f60cbde2700 1 -- 172.21.15.189:0/2978191978 wait complete. 2024-04-06T12:13:04.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.596+0000 7f60cbde2700 1 Processor -- start 2024-04-06T12:13:04.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.596+0000 7f60cbde2700 1 -- start start 2024-04-06T12:13:04.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.597+0000 7f60cbde2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60b0003a50 0x7f60c41489a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.597+0000 7f60cbde2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c40cc580 0x7f60c40cecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.597+0000 7f60cbde2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60c40cca00 0x7f60c406f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.597+0000 7f60cbde2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c414a1e0 con 0x7f60c40cc580 2024-04-06T12:13:04.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.597+0000 7f60cbde2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c4148ee0 con 0x7f60c40cca00 2024-04-06T12:13:04.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.597+0000 7f60cbde2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c4149560 con 0x7f60b0003a50 2024-04-06T12:13:04.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.597+0000 7f60c937d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c40cc580 0x7f60c40cecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.597+0000 7f60c937d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c40cc580 0x7f60c40cecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44846/0 (socket says 172.21.15.189:44846) 2024-04-06T12:13:04.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.597+0000 7f60c937d700 1 -- 172.21.15.189:0/156620321 learned_addr learned my addr 172.21.15.189:0/156620321 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:04.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.597+0000 7f60c9b7e700 1 --2- 172.21.15.189:0/156620321 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60b0003a50 0x7f60c41489a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.597+0000 7f60ca37f700 1 --2- 172.21.15.189:0/156620321 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60c40cca00 0x7f60c406f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.597+0000 7f60c937d700 1 -- 172.21.15.189:0/156620321 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60b0003a50 msgr2=0x7f60c41489a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.597+0000 7f60c937d700 1 --2- 172.21.15.189:0/156620321 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60b0003a50 0x7f60c41489a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.597+0000 7f60c937d700 1 -- 172.21.15.189:0/156620321 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60c40cca00 msgr2=0x7f60c406f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.597+0000 7f60c937d700 1 --2- 172.21.15.189:0/156620321 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60c40cca00 0x7f60c406f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.597+0000 7f60c937d700 1 -- 172.21.15.189:0/156620321 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c4149840 con 0x7f60c40cc580 2024-04-06T12:13:04.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.597+0000 7f60ca37f700 1 --2- 172.21.15.189:0/156620321 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60c40cca00 0x7f60c406f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:04.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.597+0000 7f60c937d700 1 --2- 172.21.15.189:0/156620321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c40cc580 0x7f60c40cecd0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f60b4011550 tx=0x7f60b4011910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:04.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.597+0000 7f60ba7fc700 1 -- 172.21.15.189:0/156620321 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b4015870 con 0x7f60c40cc580 2024-04-06T12:13:04.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.598+0000 7f60ba7fc700 1 -- 172.21.15.189:0/156620321 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b40159d0 con 0x7f60c40cc580 2024-04-06T12:13:04.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.598+0000 7f60cbde2700 1 -- 172.21.15.189:0/156620321 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c4149b20 con 0x7f60c40cc580 2024-04-06T12:13:04.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.598+0000 7f60ba7fc700 1 -- 172.21.15.189:0/156620321 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b4015d80 con 0x7f60c40cc580 2024-04-06T12:13:04.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.598+0000 7f60cbde2700 1 -- 172.21.15.189:0/156620321 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c41491f0 con 0x7f60c40cc580 2024-04-06T12:13:04.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.598+0000 7f60cbde2700 1 -- 172.21.15.189:0/156620321 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c4067bb0 con 0x7f60c40cc580 2024-04-06T12:13:04.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.599+0000 7f60ba7fc700 1 -- 172.21.15.189:0/156620321 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f60b4013400 con 0x7f60c40cc580 2024-04-06T12:13:04.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.599+0000 7f60ba7fc700 1 --2- 172.21.15.189:0/156620321 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f60a806ac90 0x7f60a806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.600+0000 7f60c9b7e700 1 --2- 172.21.15.189:0/156620321 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f60a806ac90 0x7f60a806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.600+0000 7f60ba7fc700 1 -- 172.21.15.189:0/156620321 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f60b4089ac0 con 0x7f60c40cc580 2024-04-06T12:13:04.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.600+0000 7f60c9b7e700 1 --2- 172.21.15.189:0/156620321 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f60a806ac90 0x7f60a806d150 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f60c0000c00 tx=0x7f60c000a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:04.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.604+0000 7f60ba7fc700 1 -- 172.21.15.189:0/156620321 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f60b4054840 con 0x7f60c40cc580 2024-04-06T12:13:04.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.796+0000 7f60cbde2700 1 -- 172.21.15.189:0/156620321 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f60c4074990 con 0x7f60c40cc580 2024-04-06T12:13:04.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.796+0000 7f60ba7fc700 1 -- 172.21.15.189:0/156620321 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f60b4057e60 con 0x7f60c40cc580 2024-04-06T12:13:04.795 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:13:04.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.798+0000 7f60cbde2700 1 -- 172.21.15.189:0/156620321 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f60a806ac90 msgr2=0x7f60a806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.798+0000 7f60cbde2700 1 --2- 172.21.15.189:0/156620321 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f60a806ac90 0x7f60a806d150 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f60c0000c00 tx=0x7f60c000a000 comp rx=0 tx=0).stop 2024-04-06T12:13:04.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.798+0000 7f60cbde2700 1 -- 172.21.15.189:0/156620321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c40cc580 msgr2=0x7f60c40cecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.798+0000 7f60cbde2700 1 --2- 172.21.15.189:0/156620321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c40cc580 0x7f60c40cecd0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f60b4011550 tx=0x7f60b4011910 comp rx=0 tx=0).stop 2024-04-06T12:13:04.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.798+0000 7f60cbde2700 1 -- 172.21.15.189:0/156620321 shutdown_connections 2024-04-06T12:13:04.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.798+0000 7f60cbde2700 1 --2- 172.21.15.189:0/156620321 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f60a806ac90 0x7f60a806d150 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.798+0000 7f60cbde2700 1 --2- 172.21.15.189:0/156620321 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60c40cca00 0x7f60c406f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.798+0000 7f60cbde2700 1 --2- 172.21.15.189:0/156620321 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60b0003a50 0x7f60c41489a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.798+0000 7f60cbde2700 1 --2- 172.21.15.189:0/156620321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c40cc580 0x7f60c40cecd0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.798+0000 7f60cbde2700 1 -- 172.21.15.189:0/156620321 >> 172.21.15.189:0/156620321 conn(0x7f60c40d00b0 msgr2=0x7f60c4159530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:04.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.798+0000 7f60cbde2700 1 -- 172.21.15.189:0/156620321 shutdown_connections 2024-04-06T12:13:04.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.798+0000 7f60cbde2700 1 -- 172.21.15.189:0/156620321 wait complete. 2024-04-06T12:13:04.810 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T12:13:04.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.933+0000 7fee1c6d3700 1 Processor -- start 2024-04-06T12:13:04.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.933+0000 7fee1c6d3700 1 -- start start 2024-04-06T12:13:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.934+0000 7fee1c6d3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fee1406a730 0x7fee14071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.934+0000 7fee1c6d3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee1415e250 con 0x7fee1415da20 2024-04-06T12:13:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.934+0000 7fee1c6d3700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee1415e530 con 0x7fee1415ec30 2024-04-06T12:13:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.934+0000 7fee1c6d3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee1415e810 con 0x7fee1406a730 2024-04-06T12:13:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1ac70700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fee1415da20 0x7fee1406f310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42700/0 (socket says 172.21.15.189:42700) 2024-04-06T12:13:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1ac70700 1 -- 172.21.15.189:0/2469435997 learned_addr learned my addr 172.21.15.189:0/2469435997 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1946d700 1 -- 172.21.15.189:0/2469435997 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351577861 0 0) 0x7fee1415e250 con 0x7fee1415da20 2024-04-06T12:13:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1a46f700 1 --2- 172.21.15.189:0/2469435997 >> v2:172.21.15.192:3300/0 conn(0x7fee1406a730 0x7fee14071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1946d700 1 -- 172.21.15.189:0/2469435997 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee000035b0 con 0x7fee1415da20 2024-04-06T12:13:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1946d700 1 -- 172.21.15.189:0/2469435997 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2276422273 0 0) 0x7fee000035b0 con 0x7fee1415da20 2024-04-06T12:13:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1946d700 1 -- 172.21.15.189:0/2469435997 >> v2:172.21.15.192:3300/0 conn(0x7fee1406a730 msgr2=0x7fee14071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1946d700 1 --2- 172.21.15.189:0/2469435997 >> v2:172.21.15.192:3300/0 conn(0x7fee1406a730 0x7fee14071f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1946d700 1 -- 172.21.15.189:0/2469435997 >> v1:172.21.15.192:6789/0 conn(0x7fee1415ec30 legacy=0x7fee1406ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1946d700 1 -- 172.21.15.189:0/2469435997 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1406edf0 con 0x7fee1415da20 2024-04-06T12:13:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1946d700 1 -- 172.21.15.189:0/2469435997 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fee0c0019f0 con 0x7fee1415da20 2024-04-06T12:13:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1946d700 1 -- 172.21.15.189:0/2469435997 >> v1:172.21.15.189:6789/0 conn(0x7fee1415da20 legacy=0x7fee1406f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1946d700 1 --2- 172.21.15.189:0/2469435997 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee00003a00 0x7fee00003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1946d700 1 --2- 172.21.15.189:0/2469435997 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee00004400 0x7fee000048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1946d700 1 --2- 172.21.15.189:0/2469435997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee000050d0 0x7fee0000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1946d700 1 -- 172.21.15.189:0/2469435997 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1415e810 con 0x7fee000050d0 2024-04-06T12:13:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1946d700 1 -- 172.21.15.189:0/2469435997 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0000c030 con 0x7fee00003a00 2024-04-06T12:13:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1946d700 1 -- 172.21.15.189:0/2469435997 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee0000c1f0 con 0x7fee00004400 2024-04-06T12:13:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1ac70700 1 --2- 172.21.15.189:0/2469435997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee000050d0 0x7fee0000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.935+0000 7fee1a46f700 1 --2- 172.21.15.189:0/2469435997 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee00003a00 0x7fee00003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.936+0000 7fee19c6e700 1 --2- 172.21.15.189:0/2469435997 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee00004400 0x7fee000048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.936+0000 7fee1ac70700 1 -- 172.21.15.189:0/2469435997 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee00004400 msgr2=0x7fee000048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.936+0000 7fee1ac70700 1 --2- 172.21.15.189:0/2469435997 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee00004400 0x7fee000048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.936+0000 7fee1ac70700 1 -- 172.21.15.189:0/2469435997 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee00003a00 msgr2=0x7fee00003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:04.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.936+0000 7fee1ac70700 1 --2- 172.21.15.189:0/2469435997 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee00003a00 0x7fee00003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.936+0000 7fee1ac70700 1 -- 172.21.15.189:0/2469435997 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee000035b0 con 0x7fee000050d0 2024-04-06T12:13:04.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.936+0000 7fee19c6e700 1 --2- 172.21.15.189:0/2469435997 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee00004400 0x7fee000048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:04.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.936+0000 7fee1ac70700 1 -- 172.21.15.189:0/2469435997 reap_dead start 2024-04-06T12:13:04.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.936+0000 7fee1ac70700 1 --2- 172.21.15.189:0/2469435997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee000050d0 0x7fee0000ba10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00a7d0 tx=0x7fee00005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=823f2f6ffb8c5785 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:04.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.936+0000 7fee1946d700 1 -- 172.21.15.189:0/2469435997 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c012600 con 0x7fee000050d0 2024-04-06T12:13:04.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.936+0000 7fee1946d700 1 -- 172.21.15.189:0/2469435997 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0c012760 con 0x7fee000050d0 2024-04-06T12:13:04.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.936+0000 7fee1946d700 1 -- 172.21.15.189:0/2469435997 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c012af0 con 0x7fee000050d0 2024-04-06T12:13:04.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.936+0000 7fee1c6d3700 1 -- 172.21.15.189:0/2469435997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee000050d0 msgr2=0x7fee0000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.936+0000 7fee1c6d3700 1 --2- 172.21.15.189:0/2469435997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee000050d0 0x7fee0000ba10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00a7d0 tx=0x7fee00005880 comp rx=0 tx=0).stop 2024-04-06T12:13:04.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.937+0000 7fee1c6d3700 1 -- 172.21.15.189:0/2469435997 shutdown_connections 2024-04-06T12:13:04.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.937+0000 7fee1c6d3700 1 --2- 172.21.15.189:0/2469435997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee000050d0 0x7fee0000ba10 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.937+0000 7fee1c6d3700 1 -- 172.21.15.189:0/2469435997 >> 172.21.15.189:0/2469435997 conn(0x7fee140b1e30 msgr2=0x7fee140b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:04.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.937+0000 7fee1c6d3700 1 -- 172.21.15.189:0/2469435997 shutdown_connections 2024-04-06T12:13:04.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.937+0000 7fee1c6d3700 1 -- 172.21.15.189:0/2469435997 wait complete. 2024-04-06T12:13:04.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.938+0000 7fee1c6d3700 1 Processor -- start 2024-04-06T12:13:04.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.938+0000 7fee1c6d3700 1 -- start start 2024-04-06T12:13:04.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.938+0000 7fee1c6d3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee000050d0 0x7fee1414bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.938+0000 7fee1c6d3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee1414ef20 0x7fee1414f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee1c6d3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee140aecb0 0x7fee140af130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee1c6d3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee140af670 con 0x7fee140aecb0 2024-04-06T12:13:04.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee1c6d3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee140ac9f0 con 0x7fee1414ef20 2024-04-06T12:13:04.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee1c6d3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee140acc90 con 0x7fee000050d0 2024-04-06T12:13:04.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee1ac70700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee140aecb0 0x7fee140af130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee1ac70700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee140aecb0 0x7fee140af130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44874/0 (socket says 172.21.15.189:44874) 2024-04-06T12:13:04.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee1ac70700 1 -- 172.21.15.189:0/3429842527 learned_addr learned my addr 172.21.15.189:0/3429842527 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:04.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee19c6e700 1 --2- 172.21.15.189:0/3429842527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee1414ef20 0x7fee1414f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee1a46f700 1 --2- 172.21.15.189:0/3429842527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee000050d0 0x7fee1414bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee1ac70700 1 -- 172.21.15.189:0/3429842527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee000050d0 msgr2=0x7fee1414bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee1ac70700 1 --2- 172.21.15.189:0/3429842527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee000050d0 0x7fee1414bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee1ac70700 1 -- 172.21.15.189:0/3429842527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee1414ef20 msgr2=0x7fee1414f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:04.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee1ac70700 1 --2- 172.21.15.189:0/3429842527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee1414ef20 0x7fee1414f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:04.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee1ac70700 1 -- 172.21.15.189:0/3429842527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee140acf70 con 0x7fee140aecb0 2024-04-06T12:13:04.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee1ac70700 1 --2- 172.21.15.189:0/3429842527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee140aecb0 0x7fee140af130 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fee141580e0 tx=0x7fee0c001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:04.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee0affd700 1 -- 172.21.15.189:0/3429842527 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c0019c0 con 0x7fee140aecb0 2024-04-06T12:13:04.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee0affd700 1 -- 172.21.15.189:0/3429842527 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0c01d420 con 0x7fee140aecb0 2024-04-06T12:13:04.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee1c6d3700 1 -- 172.21.15.189:0/3429842527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee140ad250 con 0x7fee140aecb0 2024-04-06T12:13:04.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee0affd700 1 -- 172.21.15.189:0/3429842527 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c01d7d0 con 0x7fee140aecb0 2024-04-06T12:13:04.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.939+0000 7fee1c6d3700 1 -- 172.21.15.189:0/3429842527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1414c4c0 con 0x7fee140aecb0 2024-04-06T12:13:04.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.940+0000 7fee1c6d3700 1 -- 172.21.15.189:0/3429842527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee140748f0 con 0x7fee140aecb0 2024-04-06T12:13:04.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.941+0000 7fee0affd700 1 -- 172.21.15.189:0/3429842527 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fee0c01d930 con 0x7fee140aecb0 2024-04-06T12:13:04.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.941+0000 7fee0affd700 1 --2- 172.21.15.189:0/3429842527 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fedf806ac90 0x7fedf806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:04.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.941+0000 7fee0affd700 1 -- 172.21.15.189:0/3429842527 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fee0c08b610 con 0x7fee140aecb0 2024-04-06T12:13:04.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.941+0000 7fee1a46f700 1 --2- 172.21.15.189:0/3429842527 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fedf806ac90 0x7fedf806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:04.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.942+0000 7fee1a46f700 1 --2- 172.21.15.189:0/3429842527 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fedf806ac90 0x7fedf806d150 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fee140aea10 tx=0x7fee04008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:04.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:04.946+0000 7fee0affd700 1 -- 172.21.15.189:0/3429842527 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fee0c056540 con 0x7fee140aecb0 2024-04-06T12:13:05.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.158+0000 7fee1c6d3700 1 -- 172.21.15.189:0/3429842527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fee140afb80 con 0x7fee140aecb0 2024-04-06T12:13:05.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.158+0000 7fee0affd700 1 -- 172.21.15.189:0/3429842527 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7fee0c059b60 con 0x7fee140aecb0 2024-04-06T12:13:05.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.160+0000 7fee1c6d3700 1 -- 172.21.15.189:0/3429842527 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fedf806ac90 msgr2=0x7fedf806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:05.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.160+0000 7fee1c6d3700 1 --2- 172.21.15.189:0/3429842527 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fedf806ac90 0x7fedf806d150 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fee140aea10 tx=0x7fee04008040 comp rx=0 tx=0).stop 2024-04-06T12:13:05.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.160+0000 7fee1c6d3700 1 -- 172.21.15.189:0/3429842527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee140aecb0 msgr2=0x7fee140af130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:05.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.160+0000 7fee1c6d3700 1 --2- 172.21.15.189:0/3429842527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee140aecb0 0x7fee140af130 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fee141580e0 tx=0x7fee0c001d00 comp rx=0 tx=0).stop 2024-04-06T12:13:05.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.160+0000 7fee1c6d3700 1 -- 172.21.15.189:0/3429842527 shutdown_connections 2024-04-06T12:13:05.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.160+0000 7fee1c6d3700 1 --2- 172.21.15.189:0/3429842527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee140aecb0 0x7fee140af130 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.160+0000 7fee1c6d3700 1 --2- 172.21.15.189:0/3429842527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee000050d0 0x7fee1414bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.160+0000 7fee1c6d3700 1 --2- 172.21.15.189:0/3429842527 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fedf806ac90 0x7fedf806d150 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.160+0000 7fee1c6d3700 1 --2- 172.21.15.189:0/3429842527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee1414ef20 0x7fee1414f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.160+0000 7fee1c6d3700 1 -- 172.21.15.189:0/3429842527 >> 172.21.15.189:0/3429842527 conn(0x7fee140b1e30 msgr2=0x7fee1406ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:05.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.161+0000 7fee1c6d3700 1 -- 172.21.15.189:0/3429842527 shutdown_connections 2024-04-06T12:13:05.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.161+0000 7fee1c6d3700 1 -- 172.21.15.189:0/3429842527 wait complete. 2024-04-06T12:13:05.161 INFO:teuthology.orchestra.run.smithi189.stderr:listed 0 entries 2024-04-06T12:13:05.171 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-04-06T12:13:05.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:05.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-04-06T12:13:05.215 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T12:13:05.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:05.215 DEBUG:teuthology.orchestra.run.smithi192:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-06T12:13:05.275 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-06T12:13:05.275 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:13:05.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.397+0000 7fa68fa70700 1 Processor -- start 2024-04-06T12:13:05.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.397+0000 7fa68fa70700 1 -- start start 2024-04-06T12:13:05.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.398+0000 7fa68fa70700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa68815d7b0 0x7fa6880b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:05.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.398+0000 7fa68fa70700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6880b3990 con 0x7fa68815e500 2024-04-06T12:13:05.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.398+0000 7fa68fa70700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6880b3060 con 0x7fa68806a730 2024-04-06T12:13:05.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.398+0000 7fa68fa70700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa68815e950 con 0x7fa68815d7b0 2024-04-06T12:13:05.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.398+0000 7fa68e00d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa68815e500 0x7fa6880717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42706/0 (socket says 172.21.15.189:42706) 2024-04-06T12:13:05.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.398+0000 7fa68e00d700 1 -- 172.21.15.189:0/8246005 learned_addr learned my addr 172.21.15.189:0/8246005 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:05.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.399+0000 7fa68d80c700 1 --2- 172.21.15.189:0/8246005 >> v2:172.21.15.192:3300/0 conn(0x7fa68815d7b0 0x7fa6880b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:05.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.399+0000 7fa68c80a700 1 -- 172.21.15.189:0/8246005 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732796628 0 0) 0x7fa6880b3990 con 0x7fa68815e500 2024-04-06T12:13:05.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.399+0000 7fa68c80a700 1 -- 172.21.15.189:0/8246005 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6740035b0 con 0x7fa68815e500 2024-04-06T12:13:05.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.399+0000 7fa68c80a700 1 -- 172.21.15.189:0/8246005 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3359699613 0 0) 0x7fa6740035b0 con 0x7fa68815e500 2024-04-06T12:13:05.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.399+0000 7fa68d80c700 1 -- 172.21.15.189:0/8246005 >> v1:172.21.15.192:6789/0 conn(0x7fa68806a730 legacy=0x7fa68815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:05.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.399+0000 7fa68d80c700 1 -- 172.21.15.189:0/8246005 >> v1:172.21.15.189:6789/0 conn(0x7fa68815e500 legacy=0x7fa6880717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:05.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.399+0000 7fa68d80c700 1 -- 172.21.15.189:0/8246005 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6880b3340 con 0x7fa68815d7b0 2024-04-06T12:13:05.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.399+0000 7fa68d80c700 1 --2- 172.21.15.189:0/8246005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa68815d7b0 0x7fa6880b3e20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa678009840 tx=0x7fa678009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad2c9ba1628b6002 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:05.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.399+0000 7fa68c80a700 1 -- 172.21.15.189:0/8246005 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6800019f0 con 0x7fa68815d7b0 2024-04-06T12:13:05.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.399+0000 7fa68c80a700 1 -- 172.21.15.189:0/8246005 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa678004030 con 0x7fa68815d7b0 2024-04-06T12:13:05.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.399+0000 7fa68c80a700 1 -- 172.21.15.189:0/8246005 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6780043c0 con 0x7fa68815d7b0 2024-04-06T12:13:05.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.400+0000 7fa68fa70700 1 -- 172.21.15.189:0/8246005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa68815d7b0 msgr2=0x7fa6880b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:05.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.400+0000 7fa68fa70700 1 --2- 172.21.15.189:0/8246005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa68815d7b0 0x7fa6880b3e20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa678009840 tx=0x7fa678009b50 comp rx=0 tx=0).stop 2024-04-06T12:13:05.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.400+0000 7fa68fa70700 1 -- 172.21.15.189:0/8246005 shutdown_connections 2024-04-06T12:13:05.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.400+0000 7fa68fa70700 1 --2- 172.21.15.189:0/8246005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa68815d7b0 0x7fa6880b3e20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.400+0000 7fa68fa70700 1 -- 172.21.15.189:0/8246005 >> 172.21.15.189:0/8246005 conn(0x7fa6880d00b0 msgr2=0x7fa6880d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:05.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.400+0000 7fa68fa70700 1 -- 172.21.15.189:0/8246005 shutdown_connections 2024-04-06T12:13:05.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.400+0000 7fa68fa70700 1 -- 172.21.15.189:0/8246005 wait complete. 2024-04-06T12:13:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.401+0000 7fa68fa70700 1 Processor -- start 2024-04-06T12:13:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.401+0000 7fa68fa70700 1 -- start start 2024-04-06T12:13:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.402+0000 7fa68fa70700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa68806a730 0x7fa688151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.402+0000 7fa68fa70700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa68815d7b0 0x7fa6880b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.402+0000 7fa68fa70700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa68815e500 0x7fa6880aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.402+0000 7fa68fa70700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6880b1810 con 0x7fa68815e500 2024-04-06T12:13:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.402+0000 7fa68fa70700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6880af0c0 con 0x7fa68806a730 2024-04-06T12:13:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.402+0000 7fa68fa70700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6880af2f0 con 0x7fa68815d7b0 2024-04-06T12:13:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.402+0000 7fa68e00d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa68815e500 0x7fa6880aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.402+0000 7fa68e00d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa68815e500 0x7fa6880aeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44878/0 (socket says 172.21.15.189:44878) 2024-04-06T12:13:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.402+0000 7fa68e00d700 1 -- 172.21.15.189:0/1093587865 learned_addr learned my addr 172.21.15.189:0/1093587865 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.402+0000 7fa68d80c700 1 --2- 172.21.15.189:0/1093587865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa68806a730 0x7fa688151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.402+0000 7fa68d00b700 1 --2- 172.21.15.189:0/1093587865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa68815d7b0 0x7fa6880b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.402+0000 7fa68e00d700 1 -- 172.21.15.189:0/1093587865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa68815d7b0 msgr2=0x7fa6880b12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.402+0000 7fa68e00d700 1 --2- 172.21.15.189:0/1093587865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa68815d7b0 0x7fa6880b12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.402+0000 7fa68e00d700 1 -- 172.21.15.189:0/1093587865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa68806a730 msgr2=0x7fa688151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.402+0000 7fa68e00d700 1 --2- 172.21.15.189:0/1093587865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa68806a730 0x7fa688151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.402+0000 7fa68e00d700 1 -- 172.21.15.189:0/1093587865 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6780094a0 con 0x7fa68815e500 2024-04-06T12:13:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.403+0000 7fa68e00d700 1 --2- 172.21.15.189:0/1093587865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa68815e500 0x7fa6880aeb80 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa68000cc10 tx=0x7fa68000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.403+0000 7fa67e7fc700 1 -- 172.21.15.189:0/1093587865 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68000ab70 con 0x7fa68815e500 2024-04-06T12:13:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.403+0000 7fa67e7fc700 1 -- 172.21.15.189:0/1093587865 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa68000acf0 con 0x7fa68815e500 2024-04-06T12:13:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.403+0000 7fa68fa70700 1 -- 172.21.15.189:0/1093587865 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6880af5d0 con 0x7fa68815e500 2024-04-06T12:13:05.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.403+0000 7fa67e7fc700 1 -- 172.21.15.189:0/1093587865 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa680007310 con 0x7fa68815e500 2024-04-06T12:13:05.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.403+0000 7fa68fa70700 1 -- 172.21.15.189:0/1093587865 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6880afb90 con 0x7fa68815e500 2024-04-06T12:13:05.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.404+0000 7fa68fa70700 1 -- 172.21.15.189:0/1093587865 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6880aff00 con 0x7fa68815e500 2024-04-06T12:13:05.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.404+0000 7fa67e7fc700 1 -- 172.21.15.189:0/1093587865 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa680007470 con 0x7fa68815e500 2024-04-06T12:13:05.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.405+0000 7fa67e7fc700 1 --2- 172.21.15.189:0/1093587865 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa66c06ac90 0x7fa66c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:05.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.405+0000 7fa67e7fc700 1 -- 172.21.15.189:0/1093587865 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa6800895f0 con 0x7fa68815e500 2024-04-06T12:13:05.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.405+0000 7fa68d80c700 1 --2- 172.21.15.189:0/1093587865 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa66c06ac90 0x7fa66c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:05.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.405+0000 7fa68d80c700 1 --2- 172.21.15.189:0/1093587865 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa66c06ac90 0x7fa66c06d150 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa6780090b0 tx=0x7fa678008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:05.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.409+0000 7fa67e7fc700 1 -- 172.21.15.189:0/1093587865 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa6880aff00 con 0x7fa68815e500 2024-04-06T12:13:05.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.646+0000 7fa68fa70700 1 -- 172.21.15.189:0/1093587865 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6880aff00 con 0x7fa68815e500 2024-04-06T12:13:05.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.647+0000 7fa67e7fc700 1 -- 172.21.15.189:0/1093587865 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4739 (secure 0 0 0) 0x7fa6880aff00 con 0x7fa68815e500 2024-04-06T12:13:05.646 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:13:05.646 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2769778155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2769778155},{"type":"v1","addr":"172.21.15.189:6835","nonce":2769778155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4233,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3550531296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3550531296},{"type":"v1","addr":"172.21.15.189:6837","nonce":3550531296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3198235148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3198235148},{"type":"v1","addr":"172.21.15.192:6833","nonce":3198235148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:12:04.195581+0000","modified":"2024-04-06T12:12:26.535185+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/516212261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":516212261},{"type":"v1","addr":"172.21.15.192:6835","nonce":516212261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-06T12:13:05.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.649+0000 7fa68fa70700 1 -- 172.21.15.189:0/1093587865 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa66c06ac90 msgr2=0x7fa66c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:05.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.649+0000 7fa68fa70700 1 --2- 172.21.15.189:0/1093587865 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa66c06ac90 0x7fa66c06d150 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa6780090b0 tx=0x7fa678008030 comp rx=0 tx=0).stop 2024-04-06T12:13:05.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.649+0000 7fa68fa70700 1 -- 172.21.15.189:0/1093587865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa68815e500 msgr2=0x7fa6880aeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:05.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.649+0000 7fa68fa70700 1 --2- 172.21.15.189:0/1093587865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa68815e500 0x7fa6880aeb80 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa68000cc10 tx=0x7fa68000a060 comp rx=0 tx=0).stop 2024-04-06T12:13:05.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.649+0000 7fa68fa70700 1 -- 172.21.15.189:0/1093587865 shutdown_connections 2024-04-06T12:13:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.649+0000 7fa68fa70700 1 --2- 172.21.15.189:0/1093587865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa68815e500 0x7fa6880aeb80 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.649+0000 7fa68fa70700 1 --2- 172.21.15.189:0/1093587865 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa66c06ac90 0x7fa66c06d150 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.649+0000 7fa68fa70700 1 --2- 172.21.15.189:0/1093587865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa68806a730 0x7fa688151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.649+0000 7fa68fa70700 1 --2- 172.21.15.189:0/1093587865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa68815d7b0 0x7fa6880b12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.649+0000 7fa68fa70700 1 -- 172.21.15.189:0/1093587865 >> 172.21.15.189:0/1093587865 conn(0x7fa6880d00b0 msgr2=0x7fa6880ae210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.649+0000 7fa68fa70700 1 -- 172.21.15.189:0/1093587865 shutdown_connections 2024-04-06T12:13:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.649+0000 7fa68fa70700 1 -- 172.21.15.189:0/1093587865 wait complete. 2024-04-06T12:13:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 9 2024-04-06T12:13:05.660 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:13:05.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.784+0000 7efd00d8b700 1 Processor -- start 2024-04-06T12:13:05.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.784+0000 7efd00d8b700 1 -- start start 2024-04-06T12:13:05.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.785+0000 7efd00d8b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcfc0b4660 0x7efcfc0b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:05.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.785+0000 7efd00d8b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efcfc0b2fc0 0x7efcfc0b3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:05.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.785+0000 7efd00d8b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcfc0cdbe0 con 0x7efcfc15a550 2024-04-06T12:13:05.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.785+0000 7efd00d8b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcfc0b5000 con 0x7efcfc0b2fc0 2024-04-06T12:13:05.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.785+0000 7efd00d8b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc15a940 con 0x7efcfc0b4660 2024-04-06T12:13:05.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.785+0000 7efcfad9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efcfc0b2fc0 0x7efcfc0b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:05.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.785+0000 7efcfad9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efcfc0b2fc0 0x7efcfc0b3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44886/0 (socket says 172.21.15.189:44886) 2024-04-06T12:13:05.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.785+0000 7efcfad9d700 1 -- 172.21.15.189:0/914424403 learned_addr learned my addr 172.21.15.189:0/914424403 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:05.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.785+0000 7efcf9d9b700 1 --2- 172.21.15.189:0/914424403 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcfc0b4660 0x7efcfc0b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:05.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.785+0000 7efcfad9d700 1 -- 172.21.15.189:0/914424403 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcfc0b4660 msgr2=0x7efcfc0b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:05.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.785+0000 7efcfad9d700 1 --2- 172.21.15.189:0/914424403 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcfc0b4660 0x7efcfc0b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.785+0000 7efcfad9d700 1 -- 172.21.15.189:0/914424403 >> v1:172.21.15.192:6789/0 conn(0x7efcfc15a550 legacy=0x7efcfc0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:05.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.785+0000 7efcfad9d700 1 -- 172.21.15.189:0/914424403 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc15ac00 con 0x7efcfc0b2fc0 2024-04-06T12:13:05.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.785+0000 7efcf9d9b700 1 --2- 172.21.15.189:0/914424403 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcfc0b4660 0x7efcfc0b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:05.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.785+0000 7efcfad9d700 1 --2- 172.21.15.189:0/914424403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcfc0b2fc0 0x7efcfc0b3430 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7efcf400d150 tx=0x7efcf400d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec49b6c405d8e52a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:05.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.785+0000 7efcf959a700 1 -- 172.21.15.189:0/914424403 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf4010070 con 0x7efcfc0b2fc0 2024-04-06T12:13:05.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.786+0000 7efcf959a700 1 -- 172.21.15.189:0/914424403 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf400ddc0 con 0x7efcfc0b2fc0 2024-04-06T12:13:05.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.786+0000 7efcf959a700 1 -- 172.21.15.189:0/914424403 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf400b310 con 0x7efcfc0b2fc0 2024-04-06T12:13:05.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.786+0000 7efd00d8b700 1 -- 172.21.15.189:0/914424403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcfc0b2fc0 msgr2=0x7efcfc0b3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:05.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.786+0000 7efd00d8b700 1 --2- 172.21.15.189:0/914424403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcfc0b2fc0 0x7efcfc0b3430 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7efcf400d150 tx=0x7efcf400d460 comp rx=0 tx=0).stop 2024-04-06T12:13:05.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.786+0000 7efd00d8b700 1 -- 172.21.15.189:0/914424403 shutdown_connections 2024-04-06T12:13:05.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.786+0000 7efd00d8b700 1 --2- 172.21.15.189:0/914424403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcfc0b2fc0 0x7efcfc0b3430 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.786+0000 7efd00d8b700 1 --2- 172.21.15.189:0/914424403 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcfc0b4660 0x7efcfc0b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.786+0000 7efd00d8b700 1 -- 172.21.15.189:0/914424403 >> 172.21.15.189:0/914424403 conn(0x7efcfc0d00b0 msgr2=0x7efcfc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:05.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.786+0000 7efd00d8b700 1 -- 172.21.15.189:0/914424403 shutdown_connections 2024-04-06T12:13:05.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.786+0000 7efd00d8b700 1 -- 172.21.15.189:0/914424403 wait complete. 2024-04-06T12:13:05.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.787+0000 7efd00d8b700 1 Processor -- start 2024-04-06T12:13:05.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.788+0000 7efd00d8b700 1 -- start start 2024-04-06T12:13:05.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.788+0000 7efd00d8b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcfc0b2fc0 0x7efcfc15b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:05.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.788+0000 7efd00d8b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcfc0b4660 0x7efcfc15be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:05.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.788+0000 7efd00d8b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcfc1595d0 0x7efcfc15c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:05.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.788+0000 7efd00d8b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc14e9d0 con 0x7efcfc0b2fc0 2024-04-06T12:13:05.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.788+0000 7efd00d8b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc14e0e0 con 0x7efcfc1595d0 2024-04-06T12:13:05.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.788+0000 7efd00d8b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc159db0 con 0x7efcfc0b4660 2024-04-06T12:13:05.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.788+0000 7efcfa59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcfc0b2fc0 0x7efcfc15b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:05.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.788+0000 7efcfa59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcfc0b2fc0 0x7efcfc15b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44896/0 (socket says 172.21.15.189:44896) 2024-04-06T12:13:05.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.788+0000 7efcfa59c700 1 -- 172.21.15.189:0/3895595262 learned_addr learned my addr 172.21.15.189:0/3895595262 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:05.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.788+0000 7efcfad9d700 1 --2- 172.21.15.189:0/3895595262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcfc1595d0 0x7efcfc15c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:05.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.789+0000 7efcf9d9b700 1 --2- 172.21.15.189:0/3895595262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcfc0b4660 0x7efcfc15be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:05.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.789+0000 7efcfa59c700 1 -- 172.21.15.189:0/3895595262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcfc0b4660 msgr2=0x7efcfc15be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:05.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.789+0000 7efcfa59c700 1 --2- 172.21.15.189:0/3895595262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcfc0b4660 0x7efcfc15be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.789+0000 7efcfa59c700 1 -- 172.21.15.189:0/3895595262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcfc1595d0 msgr2=0x7efcfc15c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:05.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.789+0000 7efcfa59c700 1 --2- 172.21.15.189:0/3895595262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcfc1595d0 0x7efcfc15c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.789+0000 7efcfa59c700 1 -- 172.21.15.189:0/3895595262 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf400bd40 con 0x7efcfc0b2fc0 2024-04-06T12:13:05.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.789+0000 7efcfa59c700 1 --2- 172.21.15.189:0/3895595262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcfc0b2fc0 0x7efcfc15b8c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7efcec00c4e0 tx=0x7efcec00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:05.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.789+0000 7efceaffd700 1 -- 172.21.15.189:0/3895595262 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec0052d0 con 0x7efcfc0b2fc0 2024-04-06T12:13:05.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.789+0000 7efceaffd700 1 -- 172.21.15.189:0/3895595262 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcec005430 con 0x7efcfc0b2fc0 2024-04-06T12:13:05.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.789+0000 7efceaffd700 1 -- 172.21.15.189:0/3895595262 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec0038c0 con 0x7efcfc0b2fc0 2024-04-06T12:13:05.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.789+0000 7efd00d8b700 1 -- 172.21.15.189:0/3895595262 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcfc159ad0 con 0x7efcfc0b2fc0 2024-04-06T12:13:05.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.789+0000 7efd00d8b700 1 -- 172.21.15.189:0/3895595262 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcfc14e5f0 con 0x7efcfc0b2fc0 2024-04-06T12:13:05.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.790+0000 7efd00d8b700 1 -- 172.21.15.189:0/3895595262 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcfc067bb0 con 0x7efcfc0b2fc0 2024-04-06T12:13:05.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.790+0000 7efceaffd700 1 -- 172.21.15.189:0/3895595262 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efcec003a20 con 0x7efcfc0b2fc0 2024-04-06T12:13:05.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.791+0000 7efceaffd700 1 --2- 172.21.15.189:0/3895595262 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efcdc06ac90 0x7efcdc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:05.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.791+0000 7efceaffd700 1 -- 172.21.15.189:0/3895595262 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7efcec088a80 con 0x7efcfc0b2fc0 2024-04-06T12:13:05.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.791+0000 7efcf9d9b700 1 --2- 172.21.15.189:0/3895595262 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efcdc06ac90 0x7efcdc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:05.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.791+0000 7efcf9d9b700 1 --2- 172.21.15.189:0/3895595262 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efcdc06ac90 0x7efcdc06d150 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0cc270 tx=0x7efcf0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:05.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.795+0000 7efceaffd700 1 -- 172.21.15.189:0/3895595262 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efcec0538a0 con 0x7efcfc0b2fc0 2024-04-06T12:13:05.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.989+0000 7efd00d8b700 1 -- 172.21.15.189:0/3895595262 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efcfc06b250 con 0x7efcfc0b2fc0 2024-04-06T12:13:05.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.990+0000 7efceaffd700 1 -- 172.21.15.189:0/3895595262 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18314 (secure 0 0 0) 0x7efcec017090 con 0x7efcfc0b2fc0 2024-04-06T12:13:05.989 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:13:05.989 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":26,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:54.376742+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:13:05.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.992+0000 7efd00d8b700 1 -- 172.21.15.189:0/3895595262 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efcdc06ac90 msgr2=0x7efcdc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:05.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.992+0000 7efd00d8b700 1 --2- 172.21.15.189:0/3895595262 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efcdc06ac90 0x7efcdc06d150 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0cc270 tx=0x7efcf0008040 comp rx=0 tx=0).stop 2024-04-06T12:13:05.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.992+0000 7efd00d8b700 1 -- 172.21.15.189:0/3895595262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcfc0b2fc0 msgr2=0x7efcfc15b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:05.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.992+0000 7efd00d8b700 1 --2- 172.21.15.189:0/3895595262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcfc0b2fc0 0x7efcfc15b8c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7efcec00c4e0 tx=0x7efcec00c7f0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.992+0000 7efd00d8b700 1 -- 172.21.15.189:0/3895595262 shutdown_connections 2024-04-06T12:13:05.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.992+0000 7efd00d8b700 1 --2- 172.21.15.189:0/3895595262 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efcdc06ac90 0x7efcdc06d150 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.992+0000 7efd00d8b700 1 --2- 172.21.15.189:0/3895595262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcfc1595d0 0x7efcfc15c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.992+0000 7efd00d8b700 1 --2- 172.21.15.189:0/3895595262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcfc0b2fc0 0x7efcfc15b8c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.992+0000 7efd00d8b700 1 --2- 172.21.15.189:0/3895595262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcfc0b4660 0x7efcfc15be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:05.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.992+0000 7efd00d8b700 1 -- 172.21.15.189:0/3895595262 >> 172.21.15.189:0/3895595262 conn(0x7efcfc0d00b0 msgr2=0x7efcfc06eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:05.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.992+0000 7efd00d8b700 1 -- 172.21.15.189:0/3895595262 shutdown_connections 2024-04-06T12:13:05.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:05.992+0000 7efd00d8b700 1 -- 172.21.15.189:0/3895595262 wait complete. 2024-04-06T12:13:06.004 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:12:02.280866+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:12:03.156593+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-06T12:12:06.237184+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-06T12:13:06.004 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:13:06.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.130+0000 7f5d3c247700 1 Processor -- start 2024-04-06T12:13:06.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.130+0000 7f5d3c247700 1 -- start start 2024-04-06T12:13:06.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.131+0000 7f5d3c247700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d340cd3e0 0x7f5d340cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.131+0000 7f5d3c247700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5d3415d7c0 0x7f5d3406b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.131+0000 7f5d3c247700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3406ceb0 con 0x7f5d3406b6f0 2024-04-06T12:13:06.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.131+0000 7f5d3c247700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d340cdfa0 con 0x7f5d3415d7c0 2024-04-06T12:13:06.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.131+0000 7f5d3c247700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3406d750 con 0x7f5d340cd3e0 2024-04-06T12:13:06.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.131+0000 7f5d397e2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5d3415d7c0 0x7f5d3406b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.131+0000 7f5d397e2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5d3415d7c0 0x7f5d3406b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44902/0 (socket says 172.21.15.189:44902) 2024-04-06T12:13:06.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.131+0000 7f5d397e2700 1 -- 172.21.15.189:0/178907118 learned_addr learned my addr 172.21.15.189:0/178907118 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:06.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.131+0000 7f5d39fe3700 1 --2- 172.21.15.189:0/178907118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d340cd3e0 0x7f5d340cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.132+0000 7f5d397e2700 1 -- 172.21.15.189:0/178907118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d340cd3e0 msgr2=0x7f5d340cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:06.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.132+0000 7f5d397e2700 1 --2- 172.21.15.189:0/178907118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d340cd3e0 0x7f5d340cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.132+0000 7f5d397e2700 1 -- 172.21.15.189:0/178907118 >> v1:172.21.15.192:6789/0 conn(0x7f5d3406b6f0 legacy=0x7f5d341610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:06.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.132+0000 7f5d397e2700 1 -- 172.21.15.189:0/178907118 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d340cdd00 con 0x7f5d3415d7c0 2024-04-06T12:13:06.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.132+0000 7f5d397e2700 1 --2- 172.21.15.189:0/178907118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d3415d7c0 0x7f5d3406b1b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5d300098f0 tx=0x7f5d30009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d46fbf0c9739389 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:06.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.132+0000 7f5d38fe1700 1 -- 172.21.15.189:0/178907118 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d30011830 con 0x7f5d3415d7c0 2024-04-06T12:13:06.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.132+0000 7f5d38fe1700 1 -- 172.21.15.189:0/178907118 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d30011990 con 0x7f5d3415d7c0 2024-04-06T12:13:06.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.132+0000 7f5d38fe1700 1 -- 172.21.15.189:0/178907118 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d30011d20 con 0x7f5d3415d7c0 2024-04-06T12:13:06.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.132+0000 7f5d3c247700 1 -- 172.21.15.189:0/178907118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d3415d7c0 msgr2=0x7f5d3406b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:06.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.132+0000 7f5d3c247700 1 --2- 172.21.15.189:0/178907118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d3415d7c0 0x7f5d3406b1b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5d300098f0 tx=0x7f5d30009c00 comp rx=0 tx=0).stop 2024-04-06T12:13:06.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.133+0000 7f5d3c247700 1 -- 172.21.15.189:0/178907118 shutdown_connections 2024-04-06T12:13:06.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.133+0000 7f5d3c247700 1 --2- 172.21.15.189:0/178907118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d340cd3e0 0x7f5d340cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.133+0000 7f5d3c247700 1 --2- 172.21.15.189:0/178907118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d3415d7c0 0x7f5d3406b1b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.133+0000 7f5d3c247700 1 -- 172.21.15.189:0/178907118 >> 172.21.15.189:0/178907118 conn(0x7f5d340cc290 msgr2=0x7f5d340cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:06.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.133+0000 7f5d3c247700 1 -- 172.21.15.189:0/178907118 shutdown_connections 2024-04-06T12:13:06.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.133+0000 7f5d3c247700 1 -- 172.21.15.189:0/178907118 wait complete. 2024-04-06T12:13:06.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.134+0000 7f5d3c247700 1 Processor -- start 2024-04-06T12:13:06.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.134+0000 7f5d3c247700 1 -- start start 2024-04-06T12:13:06.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.134+0000 7f5d3c247700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d3406b6f0 0x7f5d34146ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.134+0000 7f5d3c247700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d340cd3e0 0x7f5d34154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.134+0000 7f5d3c247700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d3415d7c0 0x7f5d340b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.134+0000 7f5d3c247700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d34153be0 con 0x7f5d3415d7c0 2024-04-06T12:13:06.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.134+0000 7f5d3c247700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d34153ec0 con 0x7f5d340cd3e0 2024-04-06T12:13:06.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.134+0000 7f5d3c247700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d341541a0 con 0x7f5d3406b6f0 2024-04-06T12:13:06.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.135+0000 7f5d3a7e4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d3415d7c0 0x7f5d340b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.135+0000 7f5d3a7e4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d3415d7c0 0x7f5d340b44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44904/0 (socket says 172.21.15.189:44904) 2024-04-06T12:13:06.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.135+0000 7f5d3a7e4700 1 -- 172.21.15.189:0/2111667077 learned_addr learned my addr 172.21.15.189:0/2111667077 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:06.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.135+0000 7f5d39fe3700 1 --2- 172.21.15.189:0/2111667077 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d3406b6f0 0x7f5d34146ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.135+0000 7f5d397e2700 1 --2- 172.21.15.189:0/2111667077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d340cd3e0 0x7f5d34154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.135+0000 7f5d3a7e4700 1 -- 172.21.15.189:0/2111667077 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d3406b6f0 msgr2=0x7f5d34146ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:06.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.135+0000 7f5d3a7e4700 1 --2- 172.21.15.189:0/2111667077 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d3406b6f0 0x7f5d34146ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.135+0000 7f5d3a7e4700 1 -- 172.21.15.189:0/2111667077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d340cd3e0 msgr2=0x7f5d34154c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:06.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.135+0000 7f5d3a7e4700 1 --2- 172.21.15.189:0/2111667077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d340cd3e0 0x7f5d34154c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.135+0000 7f5d3a7e4700 1 -- 172.21.15.189:0/2111667077 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d30009550 con 0x7f5d3415d7c0 2024-04-06T12:13:06.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.135+0000 7f5d3a7e4700 1 --2- 172.21.15.189:0/2111667077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d3415d7c0 0x7f5d340b44b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00cb50 tx=0x7f5d2c00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:06.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.135+0000 7f5d2a7fc700 1 -- 172.21.15.189:0/2111667077 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c003c00 con 0x7f5d3415d7c0 2024-04-06T12:13:06.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.135+0000 7f5d2a7fc700 1 -- 172.21.15.189:0/2111667077 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d2c003d60 con 0x7f5d3415d7c0 2024-04-06T12:13:06.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.135+0000 7f5d3c247700 1 -- 172.21.15.189:0/2111667077 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d34154480 con 0x7f5d3415d7c0 2024-04-06T12:13:06.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.135+0000 7f5d2a7fc700 1 -- 172.21.15.189:0/2111667077 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c00aa20 con 0x7f5d3415d7c0 2024-04-06T12:13:06.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.135+0000 7f5d3c247700 1 -- 172.21.15.189:0/2111667077 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d340c5910 con 0x7f5d3415d7c0 2024-04-06T12:13:06.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.136+0000 7f5d3c247700 1 -- 172.21.15.189:0/2111667077 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d341549a0 con 0x7f5d3415d7c0 2024-04-06T12:13:06.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.137+0000 7f5d2a7fc700 1 -- 172.21.15.189:0/2111667077 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5d2c00ab80 con 0x7f5d3415d7c0 2024-04-06T12:13:06.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.137+0000 7f5d2a7fc700 1 --2- 172.21.15.189:0/2111667077 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d1806ac90 0x7f5d1806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.137+0000 7f5d2a7fc700 1 -- 172.21.15.189:0/2111667077 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5d2c089700 con 0x7f5d3415d7c0 2024-04-06T12:13:06.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.137+0000 7f5d39fe3700 1 --2- 172.21.15.189:0/2111667077 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d1806ac90 0x7f5d1806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.137+0000 7f5d39fe3700 1 --2- 172.21.15.189:0/2111667077 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d1806ac90 0x7f5d1806d150 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5d24009b40 tx=0x7f5d24009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:06.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.141+0000 7f5d2a7fc700 1 -- 172.21.15.189:0/2111667077 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5d341549a0 con 0x7f5d3415d7c0 2024-04-06T12:13:06.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.333+0000 7f5d3c247700 1 -- 172.21.15.189:0/2111667077 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5d341549a0 con 0x7f5d3415d7c0 2024-04-06T12:13:06.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.333+0000 7f5d2a7fc700 1 -- 172.21.15.189:0/2111667077 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5d341549a0 con 0x7f5d3415d7c0 2024-04-06T12:13:06.333 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:13:06.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.335+0000 7f5d3c247700 1 -- 172.21.15.189:0/2111667077 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d1806ac90 msgr2=0x7f5d1806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:06.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.335+0000 7f5d3c247700 1 --2- 172.21.15.189:0/2111667077 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d1806ac90 0x7f5d1806d150 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5d24009b40 tx=0x7f5d24009210 comp rx=0 tx=0).stop 2024-04-06T12:13:06.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.335+0000 7f5d3c247700 1 -- 172.21.15.189:0/2111667077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d3415d7c0 msgr2=0x7f5d340b44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:06.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.335+0000 7f5d3c247700 1 --2- 172.21.15.189:0/2111667077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d3415d7c0 0x7f5d340b44b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00cb50 tx=0x7f5d2c00ce60 comp rx=0 tx=0).stop 2024-04-06T12:13:06.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.335+0000 7f5d3c247700 1 -- 172.21.15.189:0/2111667077 shutdown_connections 2024-04-06T12:13:06.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.335+0000 7f5d3c247700 1 --2- 172.21.15.189:0/2111667077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d3415d7c0 0x7f5d340b44b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.335+0000 7f5d3c247700 1 --2- 172.21.15.189:0/2111667077 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d3406b6f0 0x7f5d34146ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.335+0000 7f5d3c247700 1 --2- 172.21.15.189:0/2111667077 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d1806ac90 0x7f5d1806d150 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.335+0000 7f5d3c247700 1 --2- 172.21.15.189:0/2111667077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d340cd3e0 0x7f5d34154c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.335+0000 7f5d3c247700 1 -- 172.21.15.189:0/2111667077 >> 172.21.15.189:0/2111667077 conn(0x7f5d340cc290 msgr2=0x7f5d340acb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:06.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.336+0000 7f5d3c247700 1 -- 172.21.15.189:0/2111667077 shutdown_connections 2024-04-06T12:13:06.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.336+0000 7f5d3c247700 1 -- 172.21.15.189:0/2111667077 wait complete. 2024-04-06T12:13:06.347 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:13:06.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.473+0000 7f2c5202f700 1 Processor -- start 2024-04-06T12:13:06.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.473+0000 7f2c5202f700 1 -- start start 2024-04-06T12:13:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.473+0000 7f2c5202f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c4c0b3e20 0x7f2c4c15e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.473+0000 7f2c5202f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c4c0b3990 con 0x7f2c4c15d7b0 2024-04-06T12:13:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.473+0000 7f2c5202f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c4c0b3060 con 0x7f2c4c06ae40 2024-04-06T12:13:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.473+0000 7f2c5202f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c0b3340 con 0x7f2c4c0b3e20 2024-04-06T12:13:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2c4c15d7b0 0x7f2c4c06a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42718/0 (socket says 172.21.15.189:42718) 2024-04-06T12:13:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4b7fe700 1 -- 172.21.15.189:0/3711301242 learned_addr learned my addr 172.21.15.189:0/3711301242 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4a7fc700 1 -- 172.21.15.189:0/3711301242 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619145350 0 0) 0x7f2c4c0b3990 con 0x7f2c4c15d7b0 2024-04-06T12:13:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4a7fc700 1 -- 172.21.15.189:0/3711301242 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c340035b0 con 0x7f2c4c15d7b0 2024-04-06T12:13:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4bfff700 1 --2- 172.21.15.189:0/3711301242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c4c0b3e20 0x7f2c4c15e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4a7fc700 1 -- 172.21.15.189:0/3711301242 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1070014787 0 0) 0x7f2c340035b0 con 0x7f2c4c15d7b0 2024-04-06T12:13:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4a7fc700 1 -- 172.21.15.189:0/3711301242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c4c0b3e20 msgr2=0x7f2c4c15e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4a7fc700 1 --2- 172.21.15.189:0/3711301242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c4c0b3e20 0x7f2c4c15e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4a7fc700 1 -- 172.21.15.189:0/3711301242 >> v1:172.21.15.192:6789/0 conn(0x7f2c4c06ae40 legacy=0x7f2c4c15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4a7fc700 1 -- 172.21.15.189:0/3711301242 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4c0b3620 con 0x7f2c4c15d7b0 2024-04-06T12:13:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4a7fc700 1 -- 172.21.15.189:0/3711301242 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f2c40001670 con 0x7f2c4c15d7b0 2024-04-06T12:13:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4a7fc700 1 -- 172.21.15.189:0/3711301242 >> v1:172.21.15.189:6789/0 conn(0x7f2c4c15d7b0 legacy=0x7f2c4c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:06.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4a7fc700 1 --2- 172.21.15.189:0/3711301242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c34003a30 0x7f2c34003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4a7fc700 1 --2- 172.21.15.189:0/3711301242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c34004850 0x7f2c34004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4a7fc700 1 --2- 172.21.15.189:0/3711301242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c4c0b3e20 0x7f2c34005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4a7fc700 1 -- 172.21.15.189:0/3711301242 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c0b3340 con 0x7f2c34004850 2024-04-06T12:13:06.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4a7fc700 1 -- 172.21.15.189:0/3711301242 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3400bb40 con 0x7f2c34003a30 2024-04-06T12:13:06.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4a7fc700 1 -- 172.21.15.189:0/3711301242 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3400bd00 con 0x7f2c4c0b3e20 2024-04-06T12:13:06.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4affd700 1 --2- 172.21.15.189:0/3711301242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c34004850 0x7f2c34004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4b7fe700 1 --2- 172.21.15.189:0/3711301242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c34003a30 0x7f2c34003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.474+0000 7f2c4bfff700 1 --2- 172.21.15.189:0/3711301242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c4c0b3e20 0x7f2c34005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.475+0000 7f2c4affd700 1 -- 172.21.15.189:0/3711301242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c4c0b3e20 msgr2=0x7f2c34005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:06.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.475+0000 7f2c4affd700 1 --2- 172.21.15.189:0/3711301242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c4c0b3e20 0x7f2c34005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.475+0000 7f2c4affd700 1 -- 172.21.15.189:0/3711301242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c34003a30 msgr2=0x7f2c34003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:06.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.475+0000 7f2c4affd700 1 --2- 172.21.15.189:0/3711301242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c34003a30 0x7f2c34003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.475+0000 7f2c4affd700 1 -- 172.21.15.189:0/3711301242 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c340035b0 con 0x7f2c34004850 2024-04-06T12:13:06.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.475+0000 7f2c4affd700 1 --2- 172.21.15.189:0/3711301242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c34004850 0x7f2c34004cd0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c001800 tx=0x7f2c34005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe1e8f6e983f19e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:06.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.475+0000 7f2c4a7fc700 1 -- 172.21.15.189:0/3711301242 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c0012f0 con 0x7f2c34004850 2024-04-06T12:13:06.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.475+0000 7f2c4a7fc700 1 -- 172.21.15.189:0/3711301242 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3c0052f0 con 0x7f2c34004850 2024-04-06T12:13:06.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.475+0000 7f2c4a7fc700 1 -- 172.21.15.189:0/3711301242 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c006860 con 0x7f2c34004850 2024-04-06T12:13:06.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.475+0000 7f2c5202f700 1 -- 172.21.15.189:0/3711301242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c34004850 msgr2=0x7f2c34004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:06.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.475+0000 7f2c5202f700 1 --2- 172.21.15.189:0/3711301242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c34004850 0x7f2c34004cd0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c001800 tx=0x7f2c34005880 comp rx=0 tx=0).stop 2024-04-06T12:13:06.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.475+0000 7f2c4bfff700 1 -- 172.21.15.189:0/3711301242 reap_dead start 2024-04-06T12:13:06.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.475+0000 7f2c5202f700 1 -- 172.21.15.189:0/3711301242 shutdown_connections 2024-04-06T12:13:06.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.475+0000 7f2c5202f700 1 -- 172.21.15.189:0/3711301242 >> 172.21.15.189:0/3711301242 conn(0x7f2c4c0d00b0 msgr2=0x7f2c4c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:06.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.475+0000 7f2c5202f700 1 -- 172.21.15.189:0/3711301242 shutdown_connections 2024-04-06T12:13:06.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.476+0000 7f2c5202f700 1 -- 172.21.15.189:0/3711301242 wait complete. 2024-04-06T12:13:06.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.477+0000 7f2c5202f700 1 Processor -- start 2024-04-06T12:13:06.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.477+0000 7f2c5202f700 1 -- start start 2024-04-06T12:13:06.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c5202f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c34004850 0x7f2c4c156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c5202f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c4c15b9b0 0x7f2c4c15be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c5202f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c4c1f33b0 0x7f2c4c1f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c5202f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c15c370 con 0x7f2c4c1f33b0 2024-04-06T12:13:06.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c5202f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c157150 con 0x7f2c34004850 2024-04-06T12:13:06.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c5202f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c1573b0 con 0x7f2c4c15b9b0 2024-04-06T12:13:06.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c4bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c4c1f33b0 0x7f2c4c1f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c4bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c4c1f33b0 0x7f2c4c1f3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44924/0 (socket says 172.21.15.189:44924) 2024-04-06T12:13:06.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c4bfff700 1 -- 172.21.15.189:0/2556804876 learned_addr learned my addr 172.21.15.189:0/2556804876 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:06.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c4b7fe700 1 --2- 172.21.15.189:0/2556804876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c34004850 0x7f2c4c156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c4affd700 1 --2- 172.21.15.189:0/2556804876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c4c15b9b0 0x7f2c4c15be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c4bfff700 1 -- 172.21.15.189:0/2556804876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c4c15b9b0 msgr2=0x7f2c4c15be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:06.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c4bfff700 1 --2- 172.21.15.189:0/2556804876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c4c15b9b0 0x7f2c4c15be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c4bfff700 1 -- 172.21.15.189:0/2556804876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c34004850 msgr2=0x7f2c4c156c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:06.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c4bfff700 1 --2- 172.21.15.189:0/2556804876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c34004850 0x7f2c4c156c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c4bfff700 1 -- 172.21.15.189:0/2556804876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4c1f7360 con 0x7f2c4c1f33b0 2024-04-06T12:13:06.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c4affd700 1 --2- 172.21.15.189:0/2556804876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c4c15b9b0 0x7f2c4c15be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:06.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c4bfff700 1 --2- 172.21.15.189:0/2556804876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c4c1f33b0 0x7f2c4c1f3860 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c0673e0 tx=0x7f2c44011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:06.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c33fff700 1 -- 172.21.15.189:0/2556804876 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4400e040 con 0x7f2c4c1f33b0 2024-04-06T12:13:06.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c33fff700 1 -- 172.21.15.189:0/2556804876 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4400f040 con 0x7f2c4c1f33b0 2024-04-06T12:13:06.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c5202f700 1 -- 172.21.15.189:0/2556804876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c4c1f3da0 con 0x7f2c4c1f33b0 2024-04-06T12:13:06.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c33fff700 1 -- 172.21.15.189:0/2556804876 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c44015430 con 0x7f2c4c1f33b0 2024-04-06T12:13:06.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.478+0000 7f2c5202f700 1 -- 172.21.15.189:0/2556804876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4c1f6880 con 0x7f2c4c1f33b0 2024-04-06T12:13:06.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.479+0000 7f2c5202f700 1 -- 172.21.15.189:0/2556804876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c4c067bb0 con 0x7f2c4c1f33b0 2024-04-06T12:13:06.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.480+0000 7f2c33fff700 1 -- 172.21.15.189:0/2556804876 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2c44015590 con 0x7f2c4c1f33b0 2024-04-06T12:13:06.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.480+0000 7f2c33fff700 1 --2- 172.21.15.189:0/2556804876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2c3806ac90 0x7f2c3806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.480+0000 7f2c33fff700 1 -- 172.21.15.189:0/2556804876 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2c44089ef0 con 0x7f2c4c1f33b0 2024-04-06T12:13:06.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.480+0000 7f2c4b7fe700 1 --2- 172.21.15.189:0/2556804876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2c3806ac90 0x7f2c3806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.481+0000 7f2c4b7fe700 1 --2- 172.21.15.189:0/2556804876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2c3806ac90 0x7f2c3806d150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c15a160 tx=0x7f2c40000ed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:06.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.485+0000 7f2c33fff700 1 -- 172.21.15.189:0/2556804876 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2c44054da0 con 0x7f2c4c1f33b0 2024-04-06T12:13:06.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.678+0000 7f2c5202f700 1 -- 172.21.15.189:0/2556804876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2c4c074990 con 0x7f2c4c1f33b0 2024-04-06T12:13:06.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.678+0000 7f2c33fff700 1 -- 172.21.15.189:0/2556804876 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2c440583c0 con 0x7f2c4c1f33b0 2024-04-06T12:13:06.678 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:13:06.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.680+0000 7f2c5202f700 1 -- 172.21.15.189:0/2556804876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2c3806ac90 msgr2=0x7f2c3806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:06.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.680+0000 7f2c5202f700 1 --2- 172.21.15.189:0/2556804876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2c3806ac90 0x7f2c3806d150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c15a160 tx=0x7f2c40000ed0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.680+0000 7f2c5202f700 1 -- 172.21.15.189:0/2556804876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c4c1f33b0 msgr2=0x7f2c4c1f3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:06.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.680+0000 7f2c5202f700 1 --2- 172.21.15.189:0/2556804876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c4c1f33b0 0x7f2c4c1f3860 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c0673e0 tx=0x7f2c44011f60 comp rx=0 tx=0).stop 2024-04-06T12:13:06.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.680+0000 7f2c5202f700 1 -- 172.21.15.189:0/2556804876 shutdown_connections 2024-04-06T12:13:06.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.680+0000 7f2c5202f700 1 --2- 172.21.15.189:0/2556804876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c4c1f33b0 0x7f2c4c1f3860 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.680+0000 7f2c5202f700 1 --2- 172.21.15.189:0/2556804876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2c3806ac90 0x7f2c3806d150 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.680+0000 7f2c5202f700 1 --2- 172.21.15.189:0/2556804876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c34004850 0x7f2c4c156c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.680+0000 7f2c5202f700 1 --2- 172.21.15.189:0/2556804876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c4c15b9b0 0x7f2c4c15be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.680+0000 7f2c5202f700 1 -- 172.21.15.189:0/2556804876 >> 172.21.15.189:0/2556804876 conn(0x7f2c4c0d00b0 msgr2=0x7f2c4c0717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:06.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.681+0000 7f2c5202f700 1 -- 172.21.15.189:0/2556804876 shutdown_connections 2024-04-06T12:13:06.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.681+0000 7f2c5202f700 1 -- 172.21.15.189:0/2556804876 wait complete. 2024-04-06T12:13:06.692 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:13:06.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.817+0000 7f5b83915700 1 Processor -- start 2024-04-06T12:13:06.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.817+0000 7f5b83915700 1 -- start start 2024-04-06T12:13:06.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.818+0000 7f5b83915700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b7c15d590 0x7f5b7c0b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.818+0000 7f5b83915700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b7c0b4ed0 0x7f5b7c0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.818+0000 7f5b83915700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5b7c0b6580 0x7f5b7c0b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.818+0000 7f5b83915700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b7c0b5890 con 0x7f5b7c15d590 2024-04-06T12:13:06.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.818+0000 7f5b83915700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b7c0b5cc0 con 0x7f5b7c0b6580 2024-04-06T12:13:06.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.818+0000 7f5b83915700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c0d1500 con 0x7f5b7c0b4ed0 2024-04-06T12:13:06.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.818+0000 7f5b816b1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b7c15d590 0x7f5b7c0b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.818+0000 7f5b816b1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b7c15d590 0x7f5b7c0b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44928/0 (socket says 172.21.15.189:44928) 2024-04-06T12:13:06.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.818+0000 7f5b816b1700 1 -- 172.21.15.189:0/3837167197 learned_addr learned my addr 172.21.15.189:0/3837167197 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:06.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.818+0000 7f5b81eb2700 1 --2- 172.21.15.189:0/3837167197 >> v2:172.21.15.192:3300/0 conn(0x7f5b7c0b6580 0x7f5b7c0b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.818+0000 7f5b80eb0700 1 --2- 172.21.15.189:0/3837167197 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b7c0b4ed0 0x7f5b7c0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.818+0000 7f5b816b1700 1 -- 172.21.15.189:0/3837167197 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b7c0b4ed0 msgr2=0x7f5b7c0b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.818+0000 7f5b816b1700 1 --2- 172.21.15.189:0/3837167197 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b7c0b4ed0 0x7f5b7c0b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.818+0000 7f5b816b1700 1 -- 172.21.15.189:0/3837167197 >> v2:172.21.15.192:3300/0 conn(0x7f5b7c0b6580 msgr2=0x7f5b7c0b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.818+0000 7f5b816b1700 1 --2- 172.21.15.189:0/3837167197 >> v2:172.21.15.192:3300/0 conn(0x7f5b7c0b6580 0x7f5b7c0b6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.818+0000 7f5b816b1700 1 -- 172.21.15.189:0/3837167197 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c0d17e0 con 0x7f5b7c15d590 2024-04-06T12:13:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.818+0000 7f5b80eb0700 1 --2- 172.21.15.189:0/3837167197 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b7c0b4ed0 0x7f5b7c0b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.819+0000 7f5b816b1700 1 --2- 172.21.15.189:0/3837167197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c15d590 0x7f5b7c0b8d70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0098f0 tx=0x7f5b6c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5e849e053293d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.819+0000 7f5b73fff700 1 -- 172.21.15.189:0/3837167197 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c00b860 con 0x7f5b7c15d590 2024-04-06T12:13:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.819+0000 7f5b73fff700 1 -- 172.21.15.189:0/3837167197 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6c00b9c0 con 0x7f5b7c15d590 2024-04-06T12:13:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.819+0000 7f5b73fff700 1 -- 172.21.15.189:0/3837167197 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c00bd50 con 0x7f5b7c15d590 2024-04-06T12:13:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.819+0000 7f5b83915700 1 -- 172.21.15.189:0/3837167197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c15d590 msgr2=0x7f5b7c0b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.819+0000 7f5b83915700 1 --2- 172.21.15.189:0/3837167197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c15d590 0x7f5b7c0b8d70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0098f0 tx=0x7f5b6c009c00 comp rx=0 tx=0).stop 2024-04-06T12:13:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.819+0000 7f5b83915700 1 -- 172.21.15.189:0/3837167197 shutdown_connections 2024-04-06T12:13:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.819+0000 7f5b83915700 1 --2- 172.21.15.189:0/3837167197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c15d590 0x7f5b7c0b8d70 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.819+0000 7f5b83915700 1 --2- 172.21.15.189:0/3837167197 >> v2:172.21.15.192:3300/0 conn(0x7f5b7c0b6580 0x7f5b7c0b6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.819+0000 7f5b83915700 1 --2- 172.21.15.189:0/3837167197 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b7c0b4ed0 0x7f5b7c0b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.819+0000 7f5b83915700 1 -- 172.21.15.189:0/3837167197 >> 172.21.15.189:0/3837167197 conn(0x7f5b7c06bbc0 msgr2=0x7f5b7c06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:06.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.819+0000 7f5b83915700 1 -- 172.21.15.189:0/3837167197 shutdown_connections 2024-04-06T12:13:06.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.819+0000 7f5b83915700 1 -- 172.21.15.189:0/3837167197 wait complete. 2024-04-06T12:13:06.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.821+0000 7f5b83915700 1 Processor -- start 2024-04-06T12:13:06.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.821+0000 7f5b83915700 1 -- start start 2024-04-06T12:13:06.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.821+0000 7f5b83915700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b7c0b4ed0 0x7f5b7c14d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.821+0000 7f5b83915700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b7c0b6580 0x7f5b7c15e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.821+0000 7f5b83915700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c15d590 0x7f5b7c15e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.821+0000 7f5b83915700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c0cb360 con 0x7f5b7c15d590 2024-04-06T12:13:06.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.821+0000 7f5b83915700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c14da00 con 0x7f5b7c0b6580 2024-04-06T12:13:06.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.821+0000 7f5b83915700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c15ca50 con 0x7f5b7c0b4ed0 2024-04-06T12:13:06.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.821+0000 7f5b81eb2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c15d590 0x7f5b7c15e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.821+0000 7f5b81eb2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c15d590 0x7f5b7c15e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44930/0 (socket says 172.21.15.189:44930) 2024-04-06T12:13:06.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.821+0000 7f5b81eb2700 1 -- 172.21.15.189:0/392938507 learned_addr learned my addr 172.21.15.189:0/392938507 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:06.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.822+0000 7f5b816b1700 1 --2- 172.21.15.189:0/392938507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b7c0b4ed0 0x7f5b7c14d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.822+0000 7f5b80eb0700 1 --2- 172.21.15.189:0/392938507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b7c0b6580 0x7f5b7c15e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.822+0000 7f5b81eb2700 1 -- 172.21.15.189:0/392938507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b7c0b4ed0 msgr2=0x7f5b7c14d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:06.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.822+0000 7f5b81eb2700 1 --2- 172.21.15.189:0/392938507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b7c0b4ed0 0x7f5b7c14d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.822+0000 7f5b81eb2700 1 -- 172.21.15.189:0/392938507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b7c0b6580 msgr2=0x7f5b7c15e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:06.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.822+0000 7f5b81eb2700 1 --2- 172.21.15.189:0/392938507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b7c0b6580 0x7f5b7c15e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:06.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.822+0000 7f5b81eb2700 1 -- 172.21.15.189:0/392938507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c009550 con 0x7f5b7c15d590 2024-04-06T12:13:06.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.822+0000 7f5b80eb0700 1 --2- 172.21.15.189:0/392938507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b7c0b6580 0x7f5b7c15e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:06.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.822+0000 7f5b81eb2700 1 --2- 172.21.15.189:0/392938507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c15d590 0x7f5b7c15e730 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5b74007e40 tx=0x7f5b7400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:06.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.822+0000 7f5b71ffb700 1 -- 172.21.15.189:0/392938507 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7400ede0 con 0x7f5b7c15d590 2024-04-06T12:13:06.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.822+0000 7f5b71ffb700 1 -- 172.21.15.189:0/392938507 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b74014420 con 0x7f5b7c15d590 2024-04-06T12:13:06.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.822+0000 7f5b71ffb700 1 -- 172.21.15.189:0/392938507 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b740147d0 con 0x7f5b7c15d590 2024-04-06T12:13:06.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.822+0000 7f5b83915700 1 -- 172.21.15.189:0/392938507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b7c0cb600 con 0x7f5b7c15d590 2024-04-06T12:13:06.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.822+0000 7f5b83915700 1 -- 172.21.15.189:0/392938507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b7c0cbbc0 con 0x7f5b7c15d590 2024-04-06T12:13:06.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.823+0000 7f5b83915700 1 -- 172.21.15.189:0/392938507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b7c0cbde0 con 0x7f5b7c15d590 2024-04-06T12:13:06.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.824+0000 7f5b71ffb700 1 -- 172.21.15.189:0/392938507 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5b74014930 con 0x7f5b7c15d590 2024-04-06T12:13:06.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.824+0000 7f5b71ffb700 1 --2- 172.21.15.189:0/392938507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b6006ac90 0x7f5b6006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:06.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.824+0000 7f5b71ffb700 1 -- 172.21.15.189:0/392938507 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5b74089dc0 con 0x7f5b7c15d590 2024-04-06T12:13:06.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.824+0000 7f5b816b1700 1 --2- 172.21.15.189:0/392938507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b6006ac90 0x7f5b6006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:06.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.824+0000 7f5b816b1700 1 --2- 172.21.15.189:0/392938507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b6006ac90 0x7f5b6006d150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c014040 tx=0x7f5b6c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:06.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:06.828+0000 7f5b71ffb700 1 -- 172.21.15.189:0/392938507 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5b74054c70 con 0x7f5b7c15d590 2024-04-06T12:13:07.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.023+0000 7f5b83915700 1 -- 172.21.15.189:0/392938507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5b7c074990 con 0x7f5b7c15d590 2024-04-06T12:13:07.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.023+0000 7f5b71ffb700 1 -- 172.21.15.189:0/392938507 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5b74058290 con 0x7f5b7c15d590 2024-04-06T12:13:07.022 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:13:07.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.025+0000 7f5b83915700 1 -- 172.21.15.189:0/392938507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b6006ac90 msgr2=0x7f5b6006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:07.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.025+0000 7f5b83915700 1 --2- 172.21.15.189:0/392938507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b6006ac90 0x7f5b6006d150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c014040 tx=0x7f5b6c008030 comp rx=0 tx=0).stop 2024-04-06T12:13:07.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.025+0000 7f5b83915700 1 -- 172.21.15.189:0/392938507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c15d590 msgr2=0x7f5b7c15e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:07.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.025+0000 7f5b83915700 1 --2- 172.21.15.189:0/392938507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c15d590 0x7f5b7c15e730 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5b74007e40 tx=0x7f5b7400e3c0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.025+0000 7f5b83915700 1 -- 172.21.15.189:0/392938507 shutdown_connections 2024-04-06T12:13:07.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.025+0000 7f5b83915700 1 --2- 172.21.15.189:0/392938507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b7c15d590 0x7f5b7c15e730 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.025+0000 7f5b83915700 1 --2- 172.21.15.189:0/392938507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b7c0b4ed0 0x7f5b7c14d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.025+0000 7f5b83915700 1 --2- 172.21.15.189:0/392938507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b6006ac90 0x7f5b6006d150 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.025+0000 7f5b83915700 1 --2- 172.21.15.189:0/392938507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b7c0b6580 0x7f5b7c15e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.025+0000 7f5b83915700 1 -- 172.21.15.189:0/392938507 >> 172.21.15.189:0/392938507 conn(0x7f5b7c06bbc0 msgr2=0x7f5b7c0b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:07.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.025+0000 7f5b83915700 1 -- 172.21.15.189:0/392938507 shutdown_connections 2024-04-06T12:13:07.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.025+0000 7f5b83915700 1 -- 172.21.15.189:0/392938507 wait complete. 2024-04-06T12:13:07.036 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-06T12:13:07.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.162+0000 7f18bd5b3700 1 Processor -- start 2024-04-06T12:13:07.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.162+0000 7f18bd5b3700 1 -- start start 2024-04-06T12:13:07.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.162+0000 7f18bd5b3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f18b815d040 0x7f18b815dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.162+0000 7f18bd5b3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18b815e2a0 0x7f18b806c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.162+0000 7f18bd5b3700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b815d8d0 con 0x7f18b80b2cd0 2024-04-06T12:13:07.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.162+0000 7f18bd5b3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b80cba40 con 0x7f18b815d040 2024-04-06T12:13:07.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.162+0000 7f18bd5b3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b80cf660 con 0x7f18b815e2a0 2024-04-06T12:13:07.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.163+0000 7f18b67fc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f18b815d040 0x7f18b815dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.163+0000 7f18b6ffd700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f18b80b2cd0 0x7f18b80cef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:37692/0 (socket says 172.21.15.189:37692) 2024-04-06T12:13:07.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.163+0000 7f18b77fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18b815e2a0 0x7f18b806c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.163+0000 7f18b6ffd700 1 -- 172.21.15.189:0/3191243005 learned_addr learned my addr 172.21.15.189:0/3191243005 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:07.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.163+0000 7f18b5ffb700 1 -- 172.21.15.189:0/3191243005 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481987396 0 0) 0x7f18b815d8d0 con 0x7f18b80b2cd0 2024-04-06T12:13:07.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.163+0000 7f18b5ffb700 1 -- 172.21.15.189:0/3191243005 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18a00035b0 con 0x7f18b80b2cd0 2024-04-06T12:13:07.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.163+0000 7f18b67fc700 1 -- 172.21.15.189:0/3191243005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18b815e2a0 msgr2=0x7f18b806c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:07.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.163+0000 7f18b67fc700 1 --2- 172.21.15.189:0/3191243005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18b815e2a0 0x7f18b806c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.163+0000 7f18b67fc700 1 -- 172.21.15.189:0/3191243005 >> v1:172.21.15.192:6789/0 conn(0x7f18b80b2cd0 legacy=0x7f18b80cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:07.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.163+0000 7f18b67fc700 1 -- 172.21.15.189:0/3191243005 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b815d580 con 0x7f18b815d040 2024-04-06T12:13:07.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.163+0000 7f18b77fe700 1 --2- 172.21.15.189:0/3191243005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18b815e2a0 0x7f18b806c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:13:07.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.163+0000 7f18b67fc700 1 --2- 172.21.15.189:0/3191243005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18b815d040 0x7f18b815dd60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f18ac009840 tx=0x7f18ac009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eea925878b1cf69c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:07.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.163+0000 7f18b5ffb700 1 -- 172.21.15.189:0/3191243005 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac011830 con 0x7f18b815d040 2024-04-06T12:13:07.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.164+0000 7f18b5ffb700 1 -- 172.21.15.189:0/3191243005 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ac011990 con 0x7f18b815d040 2024-04-06T12:13:07.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.164+0000 7f18b5ffb700 1 -- 172.21.15.189:0/3191243005 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac011d20 con 0x7f18b815d040 2024-04-06T12:13:07.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.164+0000 7f18bd5b3700 1 -- 172.21.15.189:0/3191243005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18b815d040 msgr2=0x7f18b815dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:07.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.164+0000 7f18bd5b3700 1 --2- 172.21.15.189:0/3191243005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18b815d040 0x7f18b815dd60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f18ac009840 tx=0x7f18ac009b50 comp rx=0 tx=0).stop 2024-04-06T12:13:07.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.164+0000 7f18bd5b3700 1 -- 172.21.15.189:0/3191243005 shutdown_connections 2024-04-06T12:13:07.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.164+0000 7f18bd5b3700 1 --2- 172.21.15.189:0/3191243005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18b815e2a0 0x7f18b806c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.164+0000 7f18bd5b3700 1 --2- 172.21.15.189:0/3191243005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18b815d040 0x7f18b815dd60 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.164+0000 7f18bd5b3700 1 -- 172.21.15.189:0/3191243005 >> 172.21.15.189:0/3191243005 conn(0x7f18b806a730 msgr2=0x7f18b806ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:07.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.164+0000 7f18bd5b3700 1 -- 172.21.15.189:0/3191243005 shutdown_connections 2024-04-06T12:13:07.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.164+0000 7f18bd5b3700 1 -- 172.21.15.189:0/3191243005 wait complete. 2024-04-06T12:13:07.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.165+0000 7f18bd5b3700 1 Processor -- start 2024-04-06T12:13:07.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.165+0000 7f18bd5b3700 1 -- start start 2024-04-06T12:13:07.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.166+0000 7f18bd5b3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18b80b2cd0 0x7f18b8150e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.166+0000 7f18bd5b3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18b8154570 0x7f18b814d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.166+0000 7f18bd5b3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18b80a84b0 0x7f18b814c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.166+0000 7f18bd5b3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b8152320 con 0x7f18b8154570 2024-04-06T12:13:07.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.166+0000 7f18bd5b3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b8151350 con 0x7f18b80a84b0 2024-04-06T12:13:07.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.166+0000 7f18bd5b3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b814f110 con 0x7f18b80b2cd0 2024-04-06T12:13:07.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.166+0000 7f18b67fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18b8154570 0x7f18b814d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.166+0000 7f18b67fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18b8154570 0x7f18b814d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44936/0 (socket says 172.21.15.189:44936) 2024-04-06T12:13:07.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.166+0000 7f18b67fc700 1 -- 172.21.15.189:0/3055056255 learned_addr learned my addr 172.21.15.189:0/3055056255 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:07.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.166+0000 7f18b77fe700 1 --2- 172.21.15.189:0/3055056255 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18b80a84b0 0x7f18b814c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.166+0000 7f18b6ffd700 1 --2- 172.21.15.189:0/3055056255 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18b80b2cd0 0x7f18b8150e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.167+0000 7f18b67fc700 1 -- 172.21.15.189:0/3055056255 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18b80b2cd0 msgr2=0x7f18b8150e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:07.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.167+0000 7f18b67fc700 1 --2- 172.21.15.189:0/3055056255 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18b80b2cd0 0x7f18b8150e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.167+0000 7f18b67fc700 1 -- 172.21.15.189:0/3055056255 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18b80a84b0 msgr2=0x7f18b814c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:07.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.167+0000 7f18b67fc700 1 --2- 172.21.15.189:0/3055056255 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18b80a84b0 0x7f18b814c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.167+0000 7f18b67fc700 1 -- 172.21.15.189:0/3055056255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b0009a70 con 0x7f18b8154570 2024-04-06T12:13:07.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.167+0000 7f18b67fc700 1 --2- 172.21.15.189:0/3055056255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18b8154570 0x7f18b814d4c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f18ac0090b0 tx=0x7f18ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:07.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.167+0000 7f189f7fe700 1 -- 172.21.15.189:0/3055056255 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac0045d0 con 0x7f18b8154570 2024-04-06T12:13:07.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.167+0000 7f189f7fe700 1 -- 172.21.15.189:0/3055056255 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ac01f070 con 0x7f18b8154570 2024-04-06T12:13:07.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.167+0000 7f18bd5b3700 1 -- 172.21.15.189:0/3055056255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18ac0094a0 con 0x7f18b8154570 2024-04-06T12:13:07.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.167+0000 7f189f7fe700 1 -- 172.21.15.189:0/3055056255 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac0038a0 con 0x7f18b8154570 2024-04-06T12:13:07.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.167+0000 7f18bd5b3700 1 -- 172.21.15.189:0/3055056255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b8152850 con 0x7f18b8154570 2024-04-06T12:13:07.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.168+0000 7f18bd5b3700 1 -- 172.21.15.189:0/3055056255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b8073270 con 0x7f18b8154570 2024-04-06T12:13:07.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.168+0000 7f189f7fe700 1 -- 172.21.15.189:0/3055056255 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f18ac003a00 con 0x7f18b8154570 2024-04-06T12:13:07.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.169+0000 7f189f7fe700 1 --2- 172.21.15.189:0/3055056255 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f18a406ac90 0x7f18a406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.169+0000 7f189f7fe700 1 -- 172.21.15.189:0/3055056255 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f18ac021070 con 0x7f18b8154570 2024-04-06T12:13:07.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.169+0000 7f18b6ffd700 1 --2- 172.21.15.189:0/3055056255 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f18a406ac90 0x7f18a406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.169+0000 7f18b6ffd700 1 --2- 172.21.15.189:0/3055056255 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f18a406ac90 0x7f18a406d150 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f18a80011e0 tx=0x7f18a8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:07.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.173+0000 7f189f7fe700 1 -- 172.21.15.189:0/3055056255 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f18ac055ca0 con 0x7f18b8154570 2024-04-06T12:13:07.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.364+0000 7f18bd5b3700 1 -- 172.21.15.189:0/3055056255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f18b814f2d0 con 0x7f18b8154570 2024-04-06T12:13:07.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.364+0000 7f189f7fe700 1 -- 172.21.15.189:0/3055056255 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f18ac015020 con 0x7f18b8154570 2024-04-06T12:13:07.363 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:13:07.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.366+0000 7f18bd5b3700 1 -- 172.21.15.189:0/3055056255 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f18a406ac90 msgr2=0x7f18a406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:07.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.366+0000 7f18bd5b3700 1 --2- 172.21.15.189:0/3055056255 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f18a406ac90 0x7f18a406d150 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f18a80011e0 tx=0x7f18a8009040 comp rx=0 tx=0).stop 2024-04-06T12:13:07.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.366+0000 7f18bd5b3700 1 -- 172.21.15.189:0/3055056255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18b8154570 msgr2=0x7f18b814d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:07.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.366+0000 7f18bd5b3700 1 --2- 172.21.15.189:0/3055056255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18b8154570 0x7f18b814d4c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f18ac0090b0 tx=0x7f18ac004750 comp rx=0 tx=0).stop 2024-04-06T12:13:07.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.366+0000 7f18bd5b3700 1 -- 172.21.15.189:0/3055056255 shutdown_connections 2024-04-06T12:13:07.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.366+0000 7f18bd5b3700 1 --2- 172.21.15.189:0/3055056255 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f18a406ac90 0x7f18a406d150 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.366+0000 7f18bd5b3700 1 --2- 172.21.15.189:0/3055056255 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18b80a84b0 0x7f18b814c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.366+0000 7f18bd5b3700 1 --2- 172.21.15.189:0/3055056255 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18b80b2cd0 0x7f18b8150e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.366+0000 7f18bd5b3700 1 --2- 172.21.15.189:0/3055056255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18b8154570 0x7f18b814d4c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.366+0000 7f18bd5b3700 1 -- 172.21.15.189:0/3055056255 >> 172.21.15.189:0/3055056255 conn(0x7f18b806a730 msgr2=0x7f18b80cc1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:07.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.366+0000 7f18bd5b3700 1 -- 172.21.15.189:0/3055056255 shutdown_connections 2024-04-06T12:13:07.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.367+0000 7f18bd5b3700 1 -- 172.21.15.189:0/3055056255 wait complete. 2024-04-06T12:13:07.378 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:13:07.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.502+0000 7f1d1d126700 1 Processor -- start 2024-04-06T12:13:07.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.502+0000 7f1d1d126700 1 -- start start 2024-04-06T12:13:07.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.503+0000 7f1d1d126700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d1815d7c0 0x7f1d1806b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.503+0000 7f1d1d126700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1d1806b6f0 0x7f1d1806dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.503+0000 7f1d1d126700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d1806ceb0 con 0x7f1d180cd3e0 2024-04-06T12:13:07.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.503+0000 7f1d1d126700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d180cdfa0 con 0x7f1d1806b6f0 2024-04-06T12:13:07.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.503+0000 7f1d1d126700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1806d750 con 0x7f1d1815d7c0 2024-04-06T12:13:07.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.503+0000 7f1d16d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1d180cd3e0 0x7f1d180cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42720/0 (socket says 172.21.15.189:42720) 2024-04-06T12:13:07.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.503+0000 7f1d16d9d700 1 -- 172.21.15.189:0/446086912 learned_addr learned my addr 172.21.15.189:0/446086912 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:07.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.503+0000 7f1d1759e700 1 --2- 172.21.15.189:0/446086912 >> v2:172.21.15.192:3300/0 conn(0x7f1d1806b6f0 0x7f1d1806dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.503+0000 7f1d1659c700 1 --2- 172.21.15.189:0/446086912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d1815d7c0 0x7f1d1806b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.503+0000 7f1d15d9b700 1 -- 172.21.15.189:0/446086912 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184901730 0 0) 0x7f1d1806ceb0 con 0x7f1d180cd3e0 2024-04-06T12:13:07.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.503+0000 7f1d15d9b700 1 -- 172.21.15.189:0/446086912 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d000035b0 con 0x7f1d180cd3e0 2024-04-06T12:13:07.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.503+0000 7f1d15d9b700 1 -- 172.21.15.189:0/446086912 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 163197428 0 0) 0x7f1d000035b0 con 0x7f1d180cd3e0 2024-04-06T12:13:07.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.503+0000 7f1d15d9b700 1 -- 172.21.15.189:0/446086912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d1815d7c0 msgr2=0x7f1d1806b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:07.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.503+0000 7f1d15d9b700 1 --2- 172.21.15.189:0/446086912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d1815d7c0 0x7f1d1806b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.503+0000 7f1d15d9b700 1 -- 172.21.15.189:0/446086912 >> v2:172.21.15.192:3300/0 conn(0x7f1d1806b6f0 msgr2=0x7f1d1806dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:07.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.503+0000 7f1d15d9b700 1 --2- 172.21.15.189:0/446086912 >> v2:172.21.15.192:3300/0 conn(0x7f1d1806b6f0 0x7f1d1806dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.503+0000 7f1d15d9b700 1 -- 172.21.15.189:0/446086912 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1806bf00 con 0x7f1d180cd3e0 2024-04-06T12:13:07.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.503+0000 7f1d15d9b700 1 -- 172.21.15.189:0/446086912 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f1d0c001670 con 0x7f1d180cd3e0 2024-04-06T12:13:07.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.504+0000 7f1d15d9b700 1 -- 172.21.15.189:0/446086912 >> v1:172.21.15.189:6789/0 conn(0x7f1d180cd3e0 legacy=0x7f1d180cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:07.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.504+0000 7f1d15d9b700 1 --2- 172.21.15.189:0/446086912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d00003a00 0x7f1d00003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.504+0000 7f1d15d9b700 1 --2- 172.21.15.189:0/446086912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d1815d7c0 0x7f1d00004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.504+0000 7f1d15d9b700 1 --2- 172.21.15.189:0/446086912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d00004d90 0x7f1d00005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.504+0000 7f1d15d9b700 1 -- 172.21.15.189:0/446086912 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d180cdfa0 con 0x7f1d00003a00 2024-04-06T12:13:07.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.504+0000 7f1d15d9b700 1 -- 172.21.15.189:0/446086912 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1806d750 con 0x7f1d00004d90 2024-04-06T12:13:07.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.504+0000 7f1d15d9b700 1 -- 172.21.15.189:0/446086912 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0000ba40 con 0x7f1d1815d7c0 2024-04-06T12:13:07.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.504+0000 7f1d16d9d700 1 --2- 172.21.15.189:0/446086912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d00003a00 0x7f1d00003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.504+0000 7f1d1659c700 1 --2- 172.21.15.189:0/446086912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d1815d7c0 0x7f1d00004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.504+0000 7f1d1759e700 1 --2- 172.21.15.189:0/446086912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d00004d90 0x7f1d00005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.504+0000 7f1d16d9d700 1 -- 172.21.15.189:0/446086912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d1815d7c0 msgr2=0x7f1d00004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:07.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.504+0000 7f1d16d9d700 1 --2- 172.21.15.189:0/446086912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d1815d7c0 0x7f1d00004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.504+0000 7f1d16d9d700 1 -- 172.21.15.189:0/446086912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d00004d90 msgr2=0x7f1d00005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:07.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.504+0000 7f1d16d9d700 1 --2- 172.21.15.189:0/446086912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d00004d90 0x7f1d00005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.504+0000 7f1d16d9d700 1 -- 172.21.15.189:0/446086912 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d000035b0 con 0x7f1d00003a00 2024-04-06T12:13:07.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.504+0000 7f1d16d9d700 1 --2- 172.21.15.189:0/446086912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d00003a00 0x7f1d00003e20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00c370 tx=0x7f1d00005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d951373760d7289 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:07.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.504+0000 7f1d15d9b700 1 -- 172.21.15.189:0/446086912 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c007c60 con 0x7f1d00003a00 2024-04-06T12:13:07.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.504+0000 7f1d15d9b700 1 -- 172.21.15.189:0/446086912 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0c005960 con 0x7f1d00003a00 2024-04-06T12:13:07.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.504+0000 7f1d15d9b700 1 -- 172.21.15.189:0/446086912 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c005d10 con 0x7f1d00003a00 2024-04-06T12:13:07.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.505+0000 7f1d1d126700 1 -- 172.21.15.189:0/446086912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d00003a00 msgr2=0x7f1d00003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:07.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.505+0000 7f1d1d126700 1 --2- 172.21.15.189:0/446086912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d00003a00 0x7f1d00003e20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00c370 tx=0x7f1d00005880 comp rx=0 tx=0).stop 2024-04-06T12:13:07.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.505+0000 7f1d1759e700 1 -- 172.21.15.189:0/446086912 reap_dead start 2024-04-06T12:13:07.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.505+0000 7f1d1d126700 1 -- 172.21.15.189:0/446086912 shutdown_connections 2024-04-06T12:13:07.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.505+0000 7f1d1d126700 1 -- 172.21.15.189:0/446086912 >> 172.21.15.189:0/446086912 conn(0x7f1d180cc290 msgr2=0x7f1d180cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:07.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.505+0000 7f1d1d126700 1 -- 172.21.15.189:0/446086912 shutdown_connections 2024-04-06T12:13:07.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.505+0000 7f1d1d126700 1 -- 172.21.15.189:0/446086912 wait complete. 2024-04-06T12:13:07.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.506+0000 7f1d1d126700 1 Processor -- start 2024-04-06T12:13:07.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.506+0000 7f1d1d126700 1 -- start start 2024-04-06T12:13:07.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.507+0000 7f1d1d126700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d00003a00 0x7f1d1815bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.507+0000 7f1d1d126700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d18159c10 0x7f1d1815a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.507+0000 7f1d1d126700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d180c5200 0x7f1d180c5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.507+0000 7f1d1d126700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d181607d0 con 0x7f1d00003a00 2024-04-06T12:13:07.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.507+0000 7f1d1d126700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1815b430 con 0x7f1d180c5200 2024-04-06T12:13:07.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.507+0000 7f1d1d126700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1815a5d0 con 0x7f1d18159c10 2024-04-06T12:13:07.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.507+0000 7f1d16d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d00003a00 0x7f1d1815bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.507+0000 7f1d16d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d00003a00 0x7f1d1815bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44948/0 (socket says 172.21.15.189:44948) 2024-04-06T12:13:07.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.507+0000 7f1d16d9d700 1 -- 172.21.15.189:0/2915288059 learned_addr learned my addr 172.21.15.189:0/2915288059 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:07.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.507+0000 7f1d1759e700 1 --2- 172.21.15.189:0/2915288059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d180c5200 0x7f1d180c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.507+0000 7f1d1659c700 1 --2- 172.21.15.189:0/2915288059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d18159c10 0x7f1d1815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.507+0000 7f1d16d9d700 1 -- 172.21.15.189:0/2915288059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d18159c10 msgr2=0x7f1d1815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:07.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.507+0000 7f1d16d9d700 1 --2- 172.21.15.189:0/2915288059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d18159c10 0x7f1d1815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.507+0000 7f1d16d9d700 1 -- 172.21.15.189:0/2915288059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d180c5200 msgr2=0x7f1d180c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:07.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.507+0000 7f1d16d9d700 1 --2- 172.21.15.189:0/2915288059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d180c5200 0x7f1d180c5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.507+0000 7f1d16d9d700 1 -- 172.21.15.189:0/2915288059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d180c5bd0 con 0x7f1d00003a00 2024-04-06T12:13:07.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.507+0000 7f1d1659c700 1 --2- 172.21.15.189:0/2915288059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d18159c10 0x7f1d1815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:07.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.508+0000 7f1d16d9d700 1 --2- 172.21.15.189:0/2915288059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d00003a00 0x7f1d1815bf60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00c960 tx=0x7f1d0c000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:07.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.508+0000 7f1cff7fe700 1 -- 172.21.15.189:0/2915288059 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c004870 con 0x7f1d00003a00 2024-04-06T12:13:07.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.508+0000 7f1cff7fe700 1 -- 172.21.15.189:0/2915288059 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0c0049f0 con 0x7f1d00003a00 2024-04-06T12:13:07.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.508+0000 7f1cff7fe700 1 -- 172.21.15.189:0/2915288059 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c004da0 con 0x7f1d00003a00 2024-04-06T12:13:07.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.508+0000 7f1d1d126700 1 -- 172.21.15.189:0/2915288059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d1815a820 con 0x7f1d00003a00 2024-04-06T12:13:07.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.508+0000 7f1d1d126700 1 -- 172.21.15.189:0/2915288059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1815ade0 con 0x7f1d00003a00 2024-04-06T12:13:07.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.509+0000 7f1d1d126700 1 -- 172.21.15.189:0/2915288059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d18074990 con 0x7f1d00003a00 2024-04-06T12:13:07.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.509+0000 7f1cff7fe700 1 -- 172.21.15.189:0/2915288059 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1d0c0070d0 con 0x7f1d00003a00 2024-04-06T12:13:07.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.510+0000 7f1cff7fe700 1 --2- 172.21.15.189:0/2915288059 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1d0406ac90 0x7f1d0406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.510+0000 7f1cff7fe700 1 -- 172.21.15.189:0/2915288059 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f1d0c08ab00 con 0x7f1d00003a00 2024-04-06T12:13:07.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.510+0000 7f1d1659c700 1 --2- 172.21.15.189:0/2915288059 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1d0406ac90 0x7f1d0406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.510+0000 7f1d1659c700 1 --2- 172.21.15.189:0/2915288059 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1d0406ac90 0x7f1d0406d150 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1d1815dbc0 tx=0x7f1d08008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:07.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.514+0000 7f1cff7fe700 1 -- 172.21.15.189:0/2915288059 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1d0c0559b0 con 0x7f1d00003a00 2024-04-06T12:13:07.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.753+0000 7f1d1d126700 1 -- 172.21.15.189:0/2915288059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d180b5790 con 0x7f1d00003a00 2024-04-06T12:13:07.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.753+0000 7f1cff7fe700 1 -- 172.21.15.189:0/2915288059 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4739 (secure 0 0 0) 0x7f1d0c007dc0 con 0x7f1d00003a00 2024-04-06T12:13:07.753 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:13:07.753 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2769778155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2769778155},{"type":"v1","addr":"172.21.15.189:6835","nonce":2769778155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4233,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3550531296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3550531296},{"type":"v1","addr":"172.21.15.189:6837","nonce":3550531296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3198235148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3198235148},{"type":"v1","addr":"172.21.15.192:6833","nonce":3198235148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:12:04.195581+0000","modified":"2024-04-06T12:12:26.535185+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/516212261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":516212261},{"type":"v1","addr":"172.21.15.192:6835","nonce":516212261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-06T12:13:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.755+0000 7f1d1d126700 1 -- 172.21.15.189:0/2915288059 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1d0406ac90 msgr2=0x7f1d0406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.755+0000 7f1d1d126700 1 --2- 172.21.15.189:0/2915288059 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1d0406ac90 0x7f1d0406d150 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1d1815dbc0 tx=0x7f1d08008040 comp rx=0 tx=0).stop 2024-04-06T12:13:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.755+0000 7f1d1d126700 1 -- 172.21.15.189:0/2915288059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d00003a00 msgr2=0x7f1d1815bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.755+0000 7f1d1d126700 1 --2- 172.21.15.189:0/2915288059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d00003a00 0x7f1d1815bf60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00c960 tx=0x7f1d0c000fe0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.755+0000 7f1d1d126700 1 -- 172.21.15.189:0/2915288059 shutdown_connections 2024-04-06T12:13:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.755+0000 7f1d1d126700 1 --2- 172.21.15.189:0/2915288059 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1d0406ac90 0x7f1d0406d150 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.755+0000 7f1d1d126700 1 --2- 172.21.15.189:0/2915288059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d180c5200 0x7f1d180c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.755+0000 7f1d1d126700 1 --2- 172.21.15.189:0/2915288059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d00003a00 0x7f1d1815bf60 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.755+0000 7f1d1d126700 1 --2- 172.21.15.189:0/2915288059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d18159c10 0x7f1d1815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.755+0000 7f1d1d126700 1 -- 172.21.15.189:0/2915288059 >> 172.21.15.189:0/2915288059 conn(0x7f1d180cc290 msgr2=0x7f1d180ae0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:07.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.755+0000 7f1d1d126700 1 -- 172.21.15.189:0/2915288059 shutdown_connections 2024-04-06T12:13:07.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.755+0000 7f1d1d126700 1 -- 172.21.15.189:0/2915288059 wait complete. 2024-04-06T12:13:07.756 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 9 2024-04-06T12:13:07.767 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:13:07.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.890+0000 7f788efaf700 1 Processor -- start 2024-04-06T12:13:07.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.890+0000 7f788efaf700 1 -- start start 2024-04-06T12:13:07.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.891+0000 7f788efaf700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f78880b4660 0x7f78880b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.891+0000 7f788efaf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78880b2fc0 0x7f78880b3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.891+0000 7f788efaf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78880b5030 con 0x7f788815a550 2024-04-06T12:13:07.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.891+0000 7f788efaf700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78880cdbe0 con 0x7f78880b4660 2024-04-06T12:13:07.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.891+0000 7f788efaf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788815a940 con 0x7f78880b2fc0 2024-04-06T12:13:07.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.891+0000 7f7887fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f78880b4660 0x7f78880b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.891+0000 7f788cd4b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f788815a550 0x7f78880cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42722/0 (socket says 172.21.15.189:42722) 2024-04-06T12:13:07.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.891+0000 7f788cd4b700 1 -- 172.21.15.189:0/1686444192 learned_addr learned my addr 172.21.15.189:0/1686444192 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:07.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.892+0000 7f788d54c700 1 --2- 172.21.15.189:0/1686444192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78880b2fc0 0x7f78880b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.892+0000 7f7887fff700 1 -- 172.21.15.189:0/1686444192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78880b2fc0 msgr2=0x7f78880b3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:07.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.892+0000 7f7887fff700 1 --2- 172.21.15.189:0/1686444192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78880b2fc0 0x7f78880b3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.892+0000 7f7887fff700 1 -- 172.21.15.189:0/1686444192 >> v1:172.21.15.189:6789/0 conn(0x7f788815a550 legacy=0x7f78880cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:07.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.892+0000 7f7887fff700 1 -- 172.21.15.189:0/1686444192 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788815ac00 con 0x7f78880b4660 2024-04-06T12:13:07.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.892+0000 7f78877fe700 1 -- 172.21.15.189:0/1686444192 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675958287 0 0) 0x7f78880b5030 con 0x7f788815a550 2024-04-06T12:13:07.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.892+0000 7f788d54c700 1 --2- 172.21.15.189:0/1686444192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78880b2fc0 0x7f78880b3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:07.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.892+0000 7f7887fff700 1 --2- 172.21.15.189:0/1686444192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78880b4660 0x7f78880b4af0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7874009840 tx=0x7f7874009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2344d50a6b244ef server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:07.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.892+0000 7f78877fe700 1 -- 172.21.15.189:0/1686444192 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7874011830 con 0x7f78880b4660 2024-04-06T12:13:07.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.892+0000 7f78877fe700 1 -- 172.21.15.189:0/1686444192 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7874011990 con 0x7f78880b4660 2024-04-06T12:13:07.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.892+0000 7f78877fe700 1 -- 172.21.15.189:0/1686444192 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7874011d20 con 0x7f78880b4660 2024-04-06T12:13:07.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.892+0000 7f788efaf700 1 -- 172.21.15.189:0/1686444192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78880b4660 msgr2=0x7f78880b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:07.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.892+0000 7f788efaf700 1 --2- 172.21.15.189:0/1686444192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78880b4660 0x7f78880b4af0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7874009840 tx=0x7f7874009b50 comp rx=0 tx=0).stop 2024-04-06T12:13:07.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.892+0000 7f788efaf700 1 -- 172.21.15.189:0/1686444192 shutdown_connections 2024-04-06T12:13:07.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.892+0000 7f788efaf700 1 --2- 172.21.15.189:0/1686444192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78880b2fc0 0x7f78880b3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.892+0000 7f788efaf700 1 --2- 172.21.15.189:0/1686444192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78880b4660 0x7f78880b4af0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.892+0000 7f788efaf700 1 -- 172.21.15.189:0/1686444192 >> 172.21.15.189:0/1686444192 conn(0x7f78880d00b0 msgr2=0x7f78880d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:07.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.893+0000 7f788efaf700 1 -- 172.21.15.189:0/1686444192 shutdown_connections 2024-04-06T12:13:07.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.893+0000 7f788efaf700 1 -- 172.21.15.189:0/1686444192 wait complete. 2024-04-06T12:13:07.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.894+0000 7f788efaf700 1 Processor -- start 2024-04-06T12:13:07.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.894+0000 7f788efaf700 1 -- start start 2024-04-06T12:13:07.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f788efaf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78880b2fc0 0x7f78880a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f788efaf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78880b4660 0x7f7888158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f788efaf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7888154200 0x7f7888159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f788efaf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7888153de0 con 0x7f78880b4660 2024-04-06T12:13:07.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f788efaf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78881534d0 con 0x7f78880b2fc0 2024-04-06T12:13:07.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f788efaf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78881554e0 con 0x7f7888154200 2024-04-06T12:13:07.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f7887fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78880b4660 0x7f7888158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f7887fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78880b4660 0x7f7888158b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44976/0 (socket says 172.21.15.189:44976) 2024-04-06T12:13:07.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f7887fff700 1 -- 172.21.15.189:0/1178736250 learned_addr learned my addr 172.21.15.189:0/1178736250 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:07.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f788cd4b700 1 --2- 172.21.15.189:0/1178736250 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78880b2fc0 0x7f78880a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f788d54c700 1 --2- 172.21.15.189:0/1178736250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7888154200 0x7f7888159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f7887fff700 1 -- 172.21.15.189:0/1178736250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7888154200 msgr2=0x7f7888159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:07.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f7887fff700 1 --2- 172.21.15.189:0/1178736250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7888154200 0x7f7888159090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f7887fff700 1 -- 172.21.15.189:0/1178736250 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78880b2fc0 msgr2=0x7f78880a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:07.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f7887fff700 1 --2- 172.21.15.189:0/1178736250 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78880b2fc0 0x7f78880a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:07.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f7887fff700 1 -- 172.21.15.189:0/1178736250 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78740094a0 con 0x7f78880b4660 2024-04-06T12:13:07.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f788cd4b700 1 --2- 172.21.15.189:0/1178736250 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78880b2fc0 0x7f78880a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:07.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f788d54c700 1 --2- 172.21.15.189:0/1178736250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7888154200 0x7f7888159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:07.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f7887fff700 1 --2- 172.21.15.189:0/1178736250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78880b4660 0x7f7888158b50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f7874000c00 tx=0x7f7874004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:07.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f78857fa700 1 -- 172.21.15.189:0/1178736250 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78740045f0 con 0x7f78880b4660 2024-04-06T12:13:07.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f78857fa700 1 -- 172.21.15.189:0/1178736250 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787401f070 con 0x7f78880b4660 2024-04-06T12:13:07.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f788efaf700 1 -- 172.21.15.189:0/1178736250 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7888153720 con 0x7f78880b4660 2024-04-06T12:13:07.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f78857fa700 1 -- 172.21.15.189:0/1178736250 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7874003ae0 con 0x7f78880b4660 2024-04-06T12:13:07.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.895+0000 7f788efaf700 1 -- 172.21.15.189:0/1178736250 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7888155100 con 0x7f78880b4660 2024-04-06T12:13:07.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.896+0000 7f788efaf700 1 -- 172.21.15.189:0/1178736250 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7888153f20 con 0x7f78880b4660 2024-04-06T12:13:07.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.897+0000 7f78857fa700 1 -- 172.21.15.189:0/1178736250 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7874003c40 con 0x7f78880b4660 2024-04-06T12:13:07.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.897+0000 7f78857fa700 1 --2- 172.21.15.189:0/1178736250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f786c06ac90 0x7f786c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:07.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.897+0000 7f788cd4b700 1 --2- 172.21.15.189:0/1178736250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f786c06ac90 0x7f786c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:07.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.897+0000 7f78857fa700 1 -- 172.21.15.189:0/1178736250 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7874021070 con 0x7f78880b4660 2024-04-06T12:13:07.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.898+0000 7f788cd4b700 1 --2- 172.21.15.189:0/1178736250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f786c06ac90 0x7f786c06d150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f787c000f60 tx=0x7f787c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:07.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:07.902+0000 7f78857fa700 1 -- 172.21.15.189:0/1178736250 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7888153f20 con 0x7f78880b4660 2024-04-06T12:13:08.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.097+0000 7f788efaf700 1 -- 172.21.15.189:0/1178736250 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7888153f20 con 0x7f78880b4660 2024-04-06T12:13:08.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.098+0000 7f78857fa700 1 -- 172.21.15.189:0/1178736250 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18314 (secure 0 0 0) 0x7f7888153f20 con 0x7f78880b4660 2024-04-06T12:13:08.097 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:13:08.097 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":26,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:54.376742+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:13:08.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.100+0000 7f788efaf700 1 -- 172.21.15.189:0/1178736250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f786c06ac90 msgr2=0x7f786c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:08.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.100+0000 7f788efaf700 1 --2- 172.21.15.189:0/1178736250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f786c06ac90 0x7f786c06d150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f787c000f60 tx=0x7f787c009040 comp rx=0 tx=0).stop 2024-04-06T12:13:08.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.100+0000 7f788efaf700 1 -- 172.21.15.189:0/1178736250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78880b4660 msgr2=0x7f7888158b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:08.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.100+0000 7f788efaf700 1 --2- 172.21.15.189:0/1178736250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78880b4660 0x7f7888158b50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f7874000c00 tx=0x7f7874004750 comp rx=0 tx=0).stop 2024-04-06T12:13:08.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.100+0000 7f788efaf700 1 -- 172.21.15.189:0/1178736250 shutdown_connections 2024-04-06T12:13:08.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.100+0000 7f788efaf700 1 --2- 172.21.15.189:0/1178736250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7888154200 0x7f7888159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.100+0000 7f788efaf700 1 --2- 172.21.15.189:0/1178736250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f786c06ac90 0x7f786c06d150 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.100+0000 7f788efaf700 1 --2- 172.21.15.189:0/1178736250 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78880b2fc0 0x7f78880a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.100+0000 7f788efaf700 1 --2- 172.21.15.189:0/1178736250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78880b4660 0x7f7888158b50 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.100+0000 7f788efaf700 1 -- 172.21.15.189:0/1178736250 >> 172.21.15.189:0/1178736250 conn(0x7f78880d00b0 msgr2=0x7f7888158000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:08.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.100+0000 7f788efaf700 1 -- 172.21.15.189:0/1178736250 shutdown_connections 2024-04-06T12:13:08.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.100+0000 7f788efaf700 1 -- 172.21.15.189:0/1178736250 wait complete. 2024-04-06T12:13:08.112 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-06T12:13:08.112 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:13:08.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.235+0000 7f3b076b5700 1 Processor -- start 2024-04-06T12:13:08.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.236+0000 7f3b076b5700 1 -- start start 2024-04-06T12:13:08.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.236+0000 7f3b076b5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b0015d7e0 0x7f3b000d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:08.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.236+0000 7f3b076b5700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3b000d1cc0 0x7f3b000b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:08.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.236+0000 7f3b076b5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b000cdf80 con 0x7f3b000d0890 2024-04-06T12:13:08.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.236+0000 7f3b076b5700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b000d0d10 con 0x7f3b000d1cc0 2024-04-06T12:13:08.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.236+0000 7f3b076b5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b000d3230 con 0x7f3b0015d7e0 2024-04-06T12:13:08.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.237+0000 7f3b04c50700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3b000d0890 0x7f3b000d15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42736/0 (socket says 172.21.15.189:42736) 2024-04-06T12:13:08.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.237+0000 7f3b05c52700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3b000d1cc0 0x7f3b000b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:08.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.237+0000 7f3b04c50700 1 -- 172.21.15.189:0/2217576035 learned_addr learned my addr 172.21.15.189:0/2217576035 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:08.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.237+0000 7f3b05451700 1 --2- 172.21.15.189:0/2217576035 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b0015d7e0 0x7f3b000d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:08.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.237+0000 7f3af7fff700 1 -- 172.21.15.189:0/2217576035 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949504071 0 0) 0x7f3b000cdf80 con 0x7f3b000d0890 2024-04-06T12:13:08.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.237+0000 7f3af7fff700 1 -- 172.21.15.189:0/2217576035 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aec0035b0 con 0x7f3b000d0890 2024-04-06T12:13:08.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.237+0000 7f3b05c52700 1 -- 172.21.15.189:0/2217576035 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b0015d7e0 msgr2=0x7f3b000d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:08.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.237+0000 7f3b05c52700 1 --2- 172.21.15.189:0/2217576035 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b0015d7e0 0x7f3b000d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.237+0000 7f3b05c52700 1 -- 172.21.15.189:0/2217576035 >> v1:172.21.15.189:6789/0 conn(0x7f3b000d0890 legacy=0x7f3b000d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:08.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.237+0000 7f3b05c52700 1 -- 172.21.15.189:0/2217576035 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b000d2da0 con 0x7f3b000d1cc0 2024-04-06T12:13:08.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.237+0000 7f3b05451700 1 --2- 172.21.15.189:0/2217576035 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b0015d7e0 0x7f3b000d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:13:08.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.237+0000 7f3b05c52700 1 --2- 172.21.15.189:0/2217576035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b000d1cc0 0x7f3b000b66b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3af8009d90 tx=0x7f3af800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=619f14991f87eb9b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:08.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.237+0000 7f3af7fff700 1 -- 172.21.15.189:0/2217576035 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af800e070 con 0x7f3b000d1cc0 2024-04-06T12:13:08.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.237+0000 7f3af7fff700 1 -- 172.21.15.189:0/2217576035 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af800bb20 con 0x7f3b000d1cc0 2024-04-06T12:13:08.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.237+0000 7f3af7fff700 1 -- 172.21.15.189:0/2217576035 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af8012430 con 0x7f3b000d1cc0 2024-04-06T12:13:08.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.238+0000 7f3b076b5700 1 -- 172.21.15.189:0/2217576035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b000d1cc0 msgr2=0x7f3b000b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:08.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.238+0000 7f3b076b5700 1 --2- 172.21.15.189:0/2217576035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b000d1cc0 0x7f3b000b66b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3af8009d90 tx=0x7f3af800b330 comp rx=0 tx=0).stop 2024-04-06T12:13:08.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.238+0000 7f3b076b5700 1 -- 172.21.15.189:0/2217576035 shutdown_connections 2024-04-06T12:13:08.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.238+0000 7f3b076b5700 1 --2- 172.21.15.189:0/2217576035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b000d1cc0 0x7f3b000b66b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.238+0000 7f3b076b5700 1 --2- 172.21.15.189:0/2217576035 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b0015d7e0 0x7f3b000d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.238+0000 7f3b076b5700 1 -- 172.21.15.189:0/2217576035 >> 172.21.15.189:0/2217576035 conn(0x7f3b00072250 msgr2=0x7f3b00072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:08.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.238+0000 7f3b076b5700 1 -- 172.21.15.189:0/2217576035 shutdown_connections 2024-04-06T12:13:08.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.238+0000 7f3b076b5700 1 -- 172.21.15.189:0/2217576035 wait complete. 2024-04-06T12:13:08.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.239+0000 7f3b076b5700 1 Processor -- start 2024-04-06T12:13:08.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.239+0000 7f3b076b5700 1 -- start start 2024-04-06T12:13:08.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.240+0000 7f3b076b5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b000d0890 0x7f3b0014ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:08.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.240+0000 7f3b076b5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b000d1cc0 0x7f3b0014b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:08.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.240+0000 7f3b076b5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b0015d7e0 0x7f3b001f63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:08.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.240+0000 7f3b076b5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0014b140 con 0x7f3b000d0890 2024-04-06T12:13:08.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.240+0000 7f3b076b5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b001f72e0 con 0x7f3b0015d7e0 2024-04-06T12:13:08.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.240+0000 7f3b076b5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b001f6d20 con 0x7f3b000d1cc0 2024-04-06T12:13:08.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.240+0000 7f3b05451700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b000d0890 0x7f3b0014ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:08.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.240+0000 7f3b05451700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b000d0890 0x7f3b0014ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44982/0 (socket says 172.21.15.189:44982) 2024-04-06T12:13:08.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.240+0000 7f3b05451700 1 -- 172.21.15.189:0/180559241 learned_addr learned my addr 172.21.15.189:0/180559241 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:08.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.240+0000 7f3b04c50700 1 --2- 172.21.15.189:0/180559241 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b000d1cc0 0x7f3b0014b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:08.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.240+0000 7f3b05c52700 1 --2- 172.21.15.189:0/180559241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b0015d7e0 0x7f3b001f63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:08.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.240+0000 7f3b05451700 1 -- 172.21.15.189:0/180559241 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b000d1cc0 msgr2=0x7f3b0014b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:08.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.240+0000 7f3b05451700 1 --2- 172.21.15.189:0/180559241 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b000d1cc0 0x7f3b0014b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.240+0000 7f3b05451700 1 -- 172.21.15.189:0/180559241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b0015d7e0 msgr2=0x7f3b001f63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:08.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.240+0000 7f3b05451700 1 --2- 172.21.15.189:0/180559241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b0015d7e0 0x7f3b001f63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.240+0000 7f3b05451700 1 -- 172.21.15.189:0/180559241 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3afc0094a0 con 0x7f3b000d0890 2024-04-06T12:13:08.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.240+0000 7f3b04c50700 1 --2- 172.21.15.189:0/180559241 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b000d1cc0 0x7f3b0014b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:08.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.240+0000 7f3b05c52700 1 --2- 172.21.15.189:0/180559241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b0015d7e0 0x7f3b001f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:08.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.240+0000 7f3b05451700 1 --2- 172.21.15.189:0/180559241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b000d0890 0x7f3b0014ac00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3afc00ebe0 tx=0x7f3afc009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:08.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.241+0000 7f3af5ffb700 1 -- 172.21.15.189:0/180559241 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc00c910 con 0x7f3b000d0890 2024-04-06T12:13:08.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.241+0000 7f3af5ffb700 1 -- 172.21.15.189:0/180559241 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3afc00ca70 con 0x7f3b000d0890 2024-04-06T12:13:08.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.241+0000 7f3b076b5700 1 -- 172.21.15.189:0/180559241 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af8009a70 con 0x7f3b000d0890 2024-04-06T12:13:08.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.241+0000 7f3af5ffb700 1 -- 172.21.15.189:0/180559241 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc00cde0 con 0x7f3b000d0890 2024-04-06T12:13:08.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.241+0000 7f3b076b5700 1 -- 172.21.15.189:0/180559241 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b000cc2d0 con 0x7f3b000d0890 2024-04-06T12:13:08.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.242+0000 7f3b076b5700 1 -- 172.21.15.189:0/180559241 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b00067080 con 0x7f3b000d0890 2024-04-06T12:13:08.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.242+0000 7f3af5ffb700 1 -- 172.21.15.189:0/180559241 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3afc013470 con 0x7f3b000d0890 2024-04-06T12:13:08.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.242+0000 7f3af5ffb700 1 --2- 172.21.15.189:0/180559241 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3ae406ac90 0x7f3ae406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:08.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.243+0000 7f3af5ffb700 1 -- 172.21.15.189:0/180559241 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3afc089d10 con 0x7f3b000d0890 2024-04-06T12:13:08.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.243+0000 7f3b04c50700 1 --2- 172.21.15.189:0/180559241 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3ae406ac90 0x7f3ae406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:08.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.243+0000 7f3b04c50700 1 --2- 172.21.15.189:0/180559241 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3ae406ac90 0x7f3ae406d150 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3af00011e0 tx=0x7f3af0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:08.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.247+0000 7f3af5ffb700 1 -- 172.21.15.189:0/180559241 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3afc054a90 con 0x7f3b000d0890 2024-04-06T12:13:08.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.488+0000 7f3b076b5700 1 -- 172.21.15.189:0/180559241 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b00074990 con 0x7f3b000d0890 2024-04-06T12:13:08.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.489+0000 7f3af5ffb700 1 -- 172.21.15.189:0/180559241 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4739 (secure 0 0 0) 0x7f3afc0580b0 con 0x7f3b000d0890 2024-04-06T12:13:08.488 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:13:08.488 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2769778155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2769778155},{"type":"v1","addr":"172.21.15.189:6835","nonce":2769778155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4233,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3550531296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3550531296},{"type":"v1","addr":"172.21.15.189:6837","nonce":3550531296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3198235148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3198235148},{"type":"v1","addr":"172.21.15.192:6833","nonce":3198235148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:12:04.195581+0000","modified":"2024-04-06T12:12:26.535185+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/516212261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":516212261},{"type":"v1","addr":"172.21.15.192:6835","nonce":516212261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-06T12:13:08.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.491+0000 7f3b076b5700 1 -- 172.21.15.189:0/180559241 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3ae406ac90 msgr2=0x7f3ae406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:08.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.491+0000 7f3b076b5700 1 --2- 172.21.15.189:0/180559241 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3ae406ac90 0x7f3ae406d150 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3af00011e0 tx=0x7f3af0009040 comp rx=0 tx=0).stop 2024-04-06T12:13:08.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.491+0000 7f3b076b5700 1 -- 172.21.15.189:0/180559241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b000d0890 msgr2=0x7f3b0014ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:08.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.491+0000 7f3b076b5700 1 --2- 172.21.15.189:0/180559241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b000d0890 0x7f3b0014ac00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3afc00ebe0 tx=0x7f3afc009ec0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.491+0000 7f3b076b5700 1 -- 172.21.15.189:0/180559241 shutdown_connections 2024-04-06T12:13:08.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.491+0000 7f3b076b5700 1 --2- 172.21.15.189:0/180559241 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3ae406ac90 0x7f3ae406d150 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.491+0000 7f3b076b5700 1 --2- 172.21.15.189:0/180559241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b0015d7e0 0x7f3b001f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.491+0000 7f3b076b5700 1 --2- 172.21.15.189:0/180559241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b000d0890 0x7f3b0014ac00 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.491+0000 7f3b076b5700 1 --2- 172.21.15.189:0/180559241 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b000d1cc0 0x7f3b0014b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.491+0000 7f3b076b5700 1 -- 172.21.15.189:0/180559241 >> 172.21.15.189:0/180559241 conn(0x7f3b00072250 msgr2=0x7f3b0015c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:08.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.491+0000 7f3b076b5700 1 -- 172.21.15.189:0/180559241 shutdown_connections 2024-04-06T12:13:08.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.491+0000 7f3b076b5700 1 -- 172.21.15.189:0/180559241 wait complete. 2024-04-06T12:13:08.491 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 9 2024-04-06T12:13:08.503 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:13:08.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.627+0000 7fa8526ff700 1 Processor -- start 2024-04-06T12:13:08.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.627+0000 7fa8526ff700 1 -- start start 2024-04-06T12:13:08.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.628+0000 7fa8526ff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa84c0b4660 0x7fa84c0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:08.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.628+0000 7fa8526ff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa84c0b5030 con 0x7fa84c15a550 2024-04-06T12:13:08.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.628+0000 7fa8526ff700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa84c0cdbe0 con 0x7fa84c0b2fc0 2024-04-06T12:13:08.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.628+0000 7fa8526ff700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa84c15a940 con 0x7fa84c0b4660 2024-04-06T12:13:08.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.628+0000 7fa84b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa84c0b4660 0x7fa84c0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:08.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.628+0000 7fa84bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa84c15a550 0x7fa84c0cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42750/0 (socket says 172.21.15.189:42750) 2024-04-06T12:13:08.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.628+0000 7fa84bfff700 1 -- 172.21.15.189:0/1800799627 learned_addr learned my addr 172.21.15.189:0/1800799627 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:08.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.629+0000 7fa84b7fe700 1 -- 172.21.15.189:0/1800799627 >> v1:172.21.15.192:6789/0 conn(0x7fa84c0b2fc0 legacy=0x7fa84c0b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:08.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.629+0000 7fa84b7fe700 1 -- 172.21.15.189:0/1800799627 >> v1:172.21.15.189:6789/0 conn(0x7fa84c15a550 legacy=0x7fa84c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:08.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.629+0000 7fa84b7fe700 1 -- 172.21.15.189:0/1800799627 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84c15ac00 con 0x7fa84c0b4660 2024-04-06T12:13:08.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.629+0000 7fa84affd700 1 -- 172.21.15.189:0/1800799627 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882004663 0 0) 0x7fa84c0b5030 con 0x7fa84c15a550 2024-04-06T12:13:08.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.629+0000 7fa84b7fe700 1 --2- 172.21.15.189:0/1800799627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84c0b4660 0x7fa84c0b4af0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa840009840 tx=0x7fa840009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed3943aee033e30e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:08.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.629+0000 7fa84affd700 1 -- 172.21.15.189:0/1800799627 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa840004230 con 0x7fa84c0b4660 2024-04-06T12:13:08.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.629+0000 7fa84affd700 1 -- 172.21.15.189:0/1800799627 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa840004390 con 0x7fa84c0b4660 2024-04-06T12:13:08.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.629+0000 7fa84affd700 1 -- 172.21.15.189:0/1800799627 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8400036a0 con 0x7fa84c0b4660 2024-04-06T12:13:08.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.629+0000 7fa8526ff700 1 -- 172.21.15.189:0/1800799627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84c0b4660 msgr2=0x7fa84c0b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:08.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.629+0000 7fa8526ff700 1 --2- 172.21.15.189:0/1800799627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84c0b4660 0x7fa84c0b4af0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa840009840 tx=0x7fa840009b50 comp rx=0 tx=0).stop 2024-04-06T12:13:08.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.629+0000 7fa8526ff700 1 -- 172.21.15.189:0/1800799627 shutdown_connections 2024-04-06T12:13:08.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.629+0000 7fa8526ff700 1 --2- 172.21.15.189:0/1800799627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84c0b4660 0x7fa84c0b4af0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.629+0000 7fa8526ff700 1 -- 172.21.15.189:0/1800799627 >> 172.21.15.189:0/1800799627 conn(0x7fa84c0d00b0 msgr2=0x7fa84c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:08.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.629+0000 7fa8526ff700 1 -- 172.21.15.189:0/1800799627 shutdown_connections 2024-04-06T12:13:08.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.629+0000 7fa8526ff700 1 -- 172.21.15.189:0/1800799627 wait complete. 2024-04-06T12:13:08.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.631+0000 7fa8526ff700 1 Processor -- start 2024-04-06T12:13:08.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.631+0000 7fa8526ff700 1 -- start start 2024-04-06T12:13:08.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.632+0000 7fa8526ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa84c0b2fc0 0x7fa84c15c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:08.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.632+0000 7fa8526ff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa84c0b4660 0x7fa84c06ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:08.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.632+0000 7fa8526ff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84c1f4050 0x7fa84c1517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:08.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.632+0000 7fa8526ff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c15c080 con 0x7fa84c1f4050 2024-04-06T12:13:08.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.632+0000 7fa8526ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c14f6a0 con 0x7fa84c0b2fc0 2024-04-06T12:13:08.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.632+0000 7fa8526ff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c15c9e0 con 0x7fa84c0b4660 2024-04-06T12:13:08.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.632+0000 7fa850c9c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84c1f4050 0x7fa84c1517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:08.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.632+0000 7fa850c9c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84c1f4050 0x7fa84c1517a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45012/0 (socket says 172.21.15.189:45012) 2024-04-06T12:13:08.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.632+0000 7fa850c9c700 1 -- 172.21.15.189:0/1791857620 learned_addr learned my addr 172.21.15.189:0/1791857620 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:08.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.632+0000 7fa84bfff700 1 --2- 172.21.15.189:0/1791857620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa84c0b2fc0 0x7fa84c15c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:08.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.632+0000 7fa84b7fe700 1 --2- 172.21.15.189:0/1791857620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa84c0b4660 0x7fa84c06ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:08.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.632+0000 7fa850c9c700 1 -- 172.21.15.189:0/1791857620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa84c0b4660 msgr2=0x7fa84c06ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:08.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.632+0000 7fa850c9c700 1 --2- 172.21.15.189:0/1791857620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa84c0b4660 0x7fa84c06ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.632+0000 7fa850c9c700 1 -- 172.21.15.189:0/1791857620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa84c0b2fc0 msgr2=0x7fa84c15c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:08.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.632+0000 7fa850c9c700 1 --2- 172.21.15.189:0/1791857620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa84c0b2fc0 0x7fa84c15c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.633+0000 7fa850c9c700 1 -- 172.21.15.189:0/1791857620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8400094a0 con 0x7fa84c1f4050 2024-04-06T12:13:08.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.633+0000 7fa850c9c700 1 --2- 172.21.15.189:0/1791857620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84c1f4050 0x7fa84c1517a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa84400cc10 tx=0x7fa84400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:08.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.633+0000 7fa848ff9700 1 -- 172.21.15.189:0/1791857620 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84400abb0 con 0x7fa84c1f4050 2024-04-06T12:13:08.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.633+0000 7fa848ff9700 1 -- 172.21.15.189:0/1791857620 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa84400ad10 con 0x7fa84c1f4050 2024-04-06T12:13:08.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.633+0000 7fa848ff9700 1 -- 172.21.15.189:0/1791857620 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa844012660 con 0x7fa84c1f4050 2024-04-06T12:13:08.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.633+0000 7fa8526ff700 1 -- 172.21.15.189:0/1791857620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84c1f4ad0 con 0x7fa84c1f4050 2024-04-06T12:13:08.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.633+0000 7fa8526ff700 1 -- 172.21.15.189:0/1791857620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84c15b9c0 con 0x7fa84c1f4050 2024-04-06T12:13:08.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.634+0000 7fa8526ff700 1 -- 172.21.15.189:0/1791857620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84c15c1c0 con 0x7fa84c1f4050 2024-04-06T12:13:08.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.634+0000 7fa848ff9700 1 -- 172.21.15.189:0/1791857620 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa8440127c0 con 0x7fa84c1f4050 2024-04-06T12:13:08.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.635+0000 7fa848ff9700 1 --2- 172.21.15.189:0/1791857620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa83806ac90 0x7fa83806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:08.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.635+0000 7fa848ff9700 1 -- 172.21.15.189:0/1791857620 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa844089ec0 con 0x7fa84c1f4050 2024-04-06T12:13:08.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.635+0000 7fa84bfff700 1 --2- 172.21.15.189:0/1791857620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa83806ac90 0x7fa83806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:08.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.635+0000 7fa84bfff700 1 --2- 172.21.15.189:0/1791857620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa83806ac90 0x7fa83806d150 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa83c000f60 tx=0x7fa83c007c90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:08.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.639+0000 7fa848ff9700 1 -- 172.21.15.189:0/1791857620 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa844054cd0 con 0x7fa84c1f4050 2024-04-06T12:13:08.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.875+0000 7fa8526ff700 1 -- 172.21.15.189:0/1791857620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa84c067bb0 con 0x7fa84c1f4050 2024-04-06T12:13:08.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.876+0000 7fa848ff9700 1 -- 172.21.15.189:0/1791857620 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4739 (secure 0 0 0) 0x7fa844016070 con 0x7fa84c1f4050 2024-04-06T12:13:08.875 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:13:08.875 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2769778155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2769778155},{"type":"v1","addr":"172.21.15.189:6835","nonce":2769778155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4233,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3550531296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3550531296},{"type":"v1","addr":"172.21.15.189:6837","nonce":3550531296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3198235148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3198235148},{"type":"v1","addr":"172.21.15.192:6833","nonce":3198235148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:12:04.195581+0000","modified":"2024-04-06T12:12:26.535185+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/516212261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":516212261},{"type":"v1","addr":"172.21.15.192:6835","nonce":516212261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-06T12:13:08.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.878+0000 7fa8526ff700 1 -- 172.21.15.189:0/1791857620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa83806ac90 msgr2=0x7fa83806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:08.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.878+0000 7fa8526ff700 1 --2- 172.21.15.189:0/1791857620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa83806ac90 0x7fa83806d150 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa83c000f60 tx=0x7fa83c007c90 comp rx=0 tx=0).stop 2024-04-06T12:13:08.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.878+0000 7fa8526ff700 1 -- 172.21.15.189:0/1791857620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84c1f4050 msgr2=0x7fa84c1517a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:08.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.878+0000 7fa8526ff700 1 --2- 172.21.15.189:0/1791857620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84c1f4050 0x7fa84c1517a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa84400cc10 tx=0x7fa84400a060 comp rx=0 tx=0).stop 2024-04-06T12:13:08.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.878+0000 7fa8526ff700 1 -- 172.21.15.189:0/1791857620 shutdown_connections 2024-04-06T12:13:08.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.878+0000 7fa8526ff700 1 --2- 172.21.15.189:0/1791857620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84c1f4050 0x7fa84c1517a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.878+0000 7fa8526ff700 1 --2- 172.21.15.189:0/1791857620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa83806ac90 0x7fa83806d150 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.878+0000 7fa8526ff700 1 --2- 172.21.15.189:0/1791857620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa84c0b2fc0 0x7fa84c15c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.878+0000 7fa8526ff700 1 --2- 172.21.15.189:0/1791857620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa84c0b4660 0x7fa84c06ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:08.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.878+0000 7fa8526ff700 1 -- 172.21.15.189:0/1791857620 >> 172.21.15.189:0/1791857620 conn(0x7fa84c0d00b0 msgr2=0x7fa84c0cddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:08.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.878+0000 7fa8526ff700 1 -- 172.21.15.189:0/1791857620 shutdown_connections 2024-04-06T12:13:08.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:08.878+0000 7fa8526ff700 1 -- 172.21.15.189:0/1791857620 wait complete. 2024-04-06T12:13:08.878 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 9 2024-04-06T12:13:08.890 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:13:09.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.011+0000 7fb74863e700 1 Processor -- start 2024-04-06T12:13:09.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.011+0000 7fb74863e700 1 -- start start 2024-04-06T12:13:09.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.012+0000 7fb74863e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb74015d7b0 0x7fb7400b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:09.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.012+0000 7fb74863e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb74015e500 0x7fb74006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:09.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.012+0000 7fb74863e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7400b3990 con 0x7fb74006a730 2024-04-06T12:13:09.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.012+0000 7fb74863e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74015e950 con 0x7fb74015e500 2024-04-06T12:13:09.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.012+0000 7fb74863e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74006b0f0 con 0x7fb74015d7b0 2024-04-06T12:13:09.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.012+0000 7fb746bdb700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb74015e500 0x7fb74006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:09.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.012+0000 7fb745bd9700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb74006a730 0x7fb74015ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42760/0 (socket says 172.21.15.189:42760) 2024-04-06T12:13:09.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.013+0000 7fb746bdb700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb74015e500 0x7fb74006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45020/0 (socket says 172.21.15.189:45020) 2024-04-06T12:13:09.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.013+0000 7fb745bd9700 1 -- 172.21.15.189:0/2770903832 learned_addr learned my addr 172.21.15.189:0/2770903832 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:09.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.013+0000 7fb7463da700 1 --2- 172.21.15.189:0/2770903832 >> v2:172.21.15.192:3300/0 conn(0x7fb74015d7b0 0x7fb7400b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:09.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.013+0000 7fb746bdb700 1 -- 172.21.15.189:0/2770903832 >> v2:172.21.15.192:3300/0 conn(0x7fb74015d7b0 msgr2=0x7fb7400b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:09.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.013+0000 7fb746bdb700 1 --2- 172.21.15.189:0/2770903832 >> v2:172.21.15.192:3300/0 conn(0x7fb74015d7b0 0x7fb7400b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.013+0000 7fb746bdb700 1 -- 172.21.15.189:0/2770903832 >> v1:172.21.15.189:6789/0 conn(0x7fb74006a730 legacy=0x7fb74015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.013+0000 7fb746bdb700 1 -- 172.21.15.189:0/2770903832 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7400b3060 con 0x7fb74015e500 2024-04-06T12:13:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.013+0000 7fb7453d8700 1 -- 172.21.15.189:0/2770903832 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64154234 0 0) 0x7fb7400b3990 con 0x7fb74006a730 2024-04-06T12:13:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.013+0000 7fb746bdb700 1 --2- 172.21.15.189:0/2770903832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb74015e500 0x7fb74006abb0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb7400b6200 tx=0x7fb73800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dea1d3c2aa277439 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.013+0000 7fb7453d8700 1 -- 172.21.15.189:0/2770903832 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73800e070 con 0x7fb74015e500 2024-04-06T12:13:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.013+0000 7fb7453d8700 1 -- 172.21.15.189:0/2770903832 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73800bb20 con 0x7fb74015e500 2024-04-06T12:13:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.013+0000 7fb7453d8700 1 -- 172.21.15.189:0/2770903832 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb738012430 con 0x7fb74015e500 2024-04-06T12:13:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.014+0000 7fb74863e700 1 -- 172.21.15.189:0/2770903832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb74015e500 msgr2=0x7fb74006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.014+0000 7fb74863e700 1 --2- 172.21.15.189:0/2770903832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb74015e500 0x7fb74006abb0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb7400b6200 tx=0x7fb73800b330 comp rx=0 tx=0).stop 2024-04-06T12:13:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.014+0000 7fb74863e700 1 -- 172.21.15.189:0/2770903832 shutdown_connections 2024-04-06T12:13:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.014+0000 7fb74863e700 1 --2- 172.21.15.189:0/2770903832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb74015e500 0x7fb74006abb0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.014+0000 7fb74863e700 1 --2- 172.21.15.189:0/2770903832 >> v2:172.21.15.192:3300/0 conn(0x7fb74015d7b0 0x7fb7400b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.014+0000 7fb74863e700 1 -- 172.21.15.189:0/2770903832 >> 172.21.15.189:0/2770903832 conn(0x7fb7400d00b0 msgr2=0x7fb7400d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.014+0000 7fb74863e700 1 -- 172.21.15.189:0/2770903832 shutdown_connections 2024-04-06T12:13:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.014+0000 7fb74863e700 1 -- 172.21.15.189:0/2770903832 wait complete. 2024-04-06T12:13:09.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.015+0000 7fb74863e700 1 Processor -- start 2024-04-06T12:13:09.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.015+0000 7fb74863e700 1 -- start start 2024-04-06T12:13:09.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.016+0000 7fb74863e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb74006a730 0x7fb7400cf100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:09.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.016+0000 7fb74863e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb74015d7b0 0x7fb7401f2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:09.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.016+0000 7fb74863e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb74015e500 0x7fb7400b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:09.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.016+0000 7fb74863e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7401f6e80 con 0x7fb74015d7b0 2024-04-06T12:13:09.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.016+0000 7fb74863e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7400cca70 con 0x7fb74006a730 2024-04-06T12:13:09.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.016+0000 7fb74863e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7400ccd50 con 0x7fb74015e500 2024-04-06T12:13:09.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.016+0000 7fb745bd9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb74015d7b0 0x7fb7401f2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:09.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.016+0000 7fb745bd9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb74015d7b0 0x7fb7401f2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45032/0 (socket says 172.21.15.189:45032) 2024-04-06T12:13:09.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.016+0000 7fb745bd9700 1 -- 172.21.15.189:0/3040262166 learned_addr learned my addr 172.21.15.189:0/3040262166 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:09.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.016+0000 7fb7463da700 1 --2- 172.21.15.189:0/3040262166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb74006a730 0x7fb7400cf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:09.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.016+0000 7fb746bdb700 1 --2- 172.21.15.189:0/3040262166 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb74015e500 0x7fb7400b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:09.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.016+0000 7fb745bd9700 1 -- 172.21.15.189:0/3040262166 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb74015e500 msgr2=0x7fb7400b06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:09.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.016+0000 7fb745bd9700 1 --2- 172.21.15.189:0/3040262166 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb74015e500 0x7fb7400b06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.016+0000 7fb745bd9700 1 -- 172.21.15.189:0/3040262166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb74006a730 msgr2=0x7fb7400cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:09.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.016+0000 7fb745bd9700 1 --2- 172.21.15.189:0/3040262166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb74006a730 0x7fb7400cf100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.016+0000 7fb745bd9700 1 -- 172.21.15.189:0/3040262166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb738009a70 con 0x7fb74015d7b0 2024-04-06T12:13:09.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.016+0000 7fb7463da700 1 --2- 172.21.15.189:0/3040262166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb74006a730 0x7fb7400cf100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:09.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.016+0000 7fb745bd9700 1 --2- 172.21.15.189:0/3040262166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb74015d7b0 0x7fb7401f2b00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0011e0 tx=0x7fb73c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:09.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.017+0000 7fb736ffd700 1 -- 172.21.15.189:0/3040262166 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c004c80 con 0x7fb74015d7b0 2024-04-06T12:13:09.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.017+0000 7fb736ffd700 1 -- 172.21.15.189:0/3040262166 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c004e00 con 0x7fb74015d7b0 2024-04-06T12:13:09.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.017+0000 7fb736ffd700 1 -- 172.21.15.189:0/3040262166 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c007310 con 0x7fb74015d7b0 2024-04-06T12:13:09.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.017+0000 7fb74863e700 1 -- 172.21.15.189:0/3040262166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7400cd030 con 0x7fb74015d7b0 2024-04-06T12:13:09.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.017+0000 7fb74863e700 1 -- 172.21.15.189:0/3040262166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7401f3220 con 0x7fb74015d7b0 2024-04-06T12:13:09.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.018+0000 7fb74863e700 1 -- 172.21.15.189:0/3040262166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7400cd550 con 0x7fb74015d7b0 2024-04-06T12:13:09.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.018+0000 7fb736ffd700 1 -- 172.21.15.189:0/3040262166 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb73c007470 con 0x7fb74015d7b0 2024-04-06T12:13:09.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.018+0000 7fb736ffd700 1 --2- 172.21.15.189:0/3040262166 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb72406ac90 0x7fb72406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:09.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.019+0000 7fb736ffd700 1 -- 172.21.15.189:0/3040262166 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fb73c0893e0 con 0x7fb74015d7b0 2024-04-06T12:13:09.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.019+0000 7fb7463da700 1 --2- 172.21.15.189:0/3040262166 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb72406ac90 0x7fb72406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:09.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.019+0000 7fb7463da700 1 --2- 172.21.15.189:0/3040262166 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb72406ac90 0x7fb72406d150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb730009c30 tx=0x7fb730008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:09.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.023+0000 7fb736ffd700 1 -- 172.21.15.189:0/3040262166 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb7400cd550 con 0x7fb74015d7b0 2024-04-06T12:13:09.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.221+0000 7fb74863e700 1 -- 172.21.15.189:0/3040262166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7400cd550 con 0x7fb74015d7b0 2024-04-06T12:13:09.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.222+0000 7fb736ffd700 1 -- 172.21.15.189:0/3040262166 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18314 (secure 0 0 0) 0x7fb7400cd550 con 0x7fb74015d7b0 2024-04-06T12:13:09.222 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:13:09.222 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":26,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:12:54.376742+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:12:02.280866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-06T12:12:03.156593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-06T12:12:06.237184+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:13:09.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.225+0000 7fb74863e700 1 -- 172.21.15.189:0/3040262166 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb72406ac90 msgr2=0x7fb72406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:09.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.225+0000 7fb74863e700 1 --2- 172.21.15.189:0/3040262166 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb72406ac90 0x7fb72406d150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb730009c30 tx=0x7fb730008040 comp rx=0 tx=0).stop 2024-04-06T12:13:09.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.225+0000 7fb74863e700 1 -- 172.21.15.189:0/3040262166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb74015d7b0 msgr2=0x7fb7401f2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:09.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.225+0000 7fb74863e700 1 --2- 172.21.15.189:0/3040262166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb74015d7b0 0x7fb7401f2b00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0011e0 tx=0x7fb73c00cae0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.225+0000 7fb74863e700 1 -- 172.21.15.189:0/3040262166 shutdown_connections 2024-04-06T12:13:09.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.225+0000 7fb74863e700 1 --2- 172.21.15.189:0/3040262166 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb74015e500 0x7fb7400b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.225+0000 7fb74863e700 1 --2- 172.21.15.189:0/3040262166 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb72406ac90 0x7fb72406d150 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.225+0000 7fb74863e700 1 --2- 172.21.15.189:0/3040262166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb74006a730 0x7fb7400cf100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.225+0000 7fb74863e700 1 --2- 172.21.15.189:0/3040262166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb74015d7b0 0x7fb7401f2b00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.225+0000 7fb74863e700 1 -- 172.21.15.189:0/3040262166 >> 172.21.15.189:0/3040262166 conn(0x7fb7400d00b0 msgr2=0x7fb7400ad100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:09.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.225+0000 7fb74863e700 1 -- 172.21.15.189:0/3040262166 shutdown_connections 2024-04-06T12:13:09.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.225+0000 7fb74863e700 1 -- 172.21.15.189:0/3040262166 wait complete. 2024-04-06T12:13:09.236 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-06T12:13:09.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.360+0000 7fdfef16b700 1 Processor -- start 2024-04-06T12:13:09.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.360+0000 7fdfef16b700 1 -- start start 2024-04-06T12:13:09.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.360+0000 7fdfef16b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfe8144f40 0x7fdfe806a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:09.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.361+0000 7fdfef16b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe815c740 con 0x7fdfe815c150 2024-04-06T12:13:09.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.361+0000 7fdfef16b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe815f1f0 con 0x7fdfe815fcd0 2024-04-06T12:13:09.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.361+0000 7fdfef16b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe815c560 con 0x7fdfe8144f40 2024-04-06T12:13:09.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.361+0000 7fdfed708700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdfe815c150 0x7fdfe80ceca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39994/0 (socket says 172.21.15.189:39994) 2024-04-06T12:13:09.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.361+0000 7fdfed708700 1 -- 172.21.15.189:0/1251291045 learned_addr learned my addr 172.21.15.189:0/1251291045 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:09.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.361+0000 7fdfecf07700 1 --2- 172.21.15.189:0/1251291045 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfe8144f40 0x7fdfe806a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:09.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.361+0000 7fdfe77fe700 1 -- 172.21.15.189:0/1251291045 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634895575 0 0) 0x7fdfe815c740 con 0x7fdfe815c150 2024-04-06T12:13:09.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.361+0000 7fdfe77fe700 1 -- 172.21.15.189:0/1251291045 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfd80035b0 con 0x7fdfe815c150 2024-04-06T12:13:09.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.361+0000 7fdfe77fe700 1 -- 172.21.15.189:0/1251291045 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790083895 0 0) 0x7fdfe815f1f0 con 0x7fdfe815fcd0 2024-04-06T12:13:09.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.361+0000 7fdfe77fe700 1 -- 172.21.15.189:0/1251291045 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfe815c740 con 0x7fdfe815fcd0 2024-04-06T12:13:09.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.361+0000 7fdfecf07700 1 -- 172.21.15.189:0/1251291045 >> v1:172.21.15.192:6789/0 conn(0x7fdfe815fcd0 legacy=0x7fdfe815ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:09.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.361+0000 7fdfecf07700 1 -- 172.21.15.189:0/1251291045 >> v1:172.21.15.189:6789/0 conn(0x7fdfe815c150 legacy=0x7fdfe80ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:09.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.361+0000 7fdfecf07700 1 -- 172.21.15.189:0/1251291045 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe815f4d0 con 0x7fdfe8144f40 2024-04-06T12:13:09.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.361+0000 7fdfecf07700 1 --2- 172.21.15.189:0/1251291045 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfe8144f40 0x7fdfe806a730 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009870 tx=0x7fdfdc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c267bae0c6a03e4b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:09.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.362+0000 7fdfe77fe700 1 -- 172.21.15.189:0/1251291045 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc011830 con 0x7fdfe8144f40 2024-04-06T12:13:09.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.362+0000 7fdfe77fe700 1 -- 172.21.15.189:0/1251291045 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfdc011990 con 0x7fdfe8144f40 2024-04-06T12:13:09.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.362+0000 7fdfe77fe700 1 -- 172.21.15.189:0/1251291045 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc011d00 con 0x7fdfe8144f40 2024-04-06T12:13:09.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.362+0000 7fdfef16b700 1 -- 172.21.15.189:0/1251291045 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfe8144f40 msgr2=0x7fdfe806a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:09.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.362+0000 7fdfef16b700 1 --2- 172.21.15.189:0/1251291045 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfe8144f40 0x7fdfe806a730 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009870 tx=0x7fdfdc009b80 comp rx=0 tx=0).stop 2024-04-06T12:13:09.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.362+0000 7fdfef16b700 1 -- 172.21.15.189:0/1251291045 shutdown_connections 2024-04-06T12:13:09.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.362+0000 7fdfef16b700 1 --2- 172.21.15.189:0/1251291045 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfe8144f40 0x7fdfe806a730 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.362+0000 7fdfef16b700 1 -- 172.21.15.189:0/1251291045 >> 172.21.15.189:0/1251291045 conn(0x7fdfe80d00b0 msgr2=0x7fdfe80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:09.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.362+0000 7fdfef16b700 1 -- 172.21.15.189:0/1251291045 shutdown_connections 2024-04-06T12:13:09.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.362+0000 7fdfef16b700 1 -- 172.21.15.189:0/1251291045 wait complete. 2024-04-06T12:13:09.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.364+0000 7fdfef16b700 1 Processor -- start 2024-04-06T12:13:09.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.364+0000 7fdfef16b700 1 -- start start 2024-04-06T12:13:09.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfef16b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfe8144f40 0x7fdfe80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:09.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfef16b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfe815c150 0x7fdfe8142f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:09.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfef16b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfe815fcd0 0x7fdfe81434a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:09.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfef16b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe814a840 con 0x7fdfe815c150 2024-04-06T12:13:09.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfef16b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe81439e0 con 0x7fdfe815fcd0 2024-04-06T12:13:09.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfef16b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe8143c90 con 0x7fdfe8144f40 2024-04-06T12:13:09.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfe7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfe815c150 0x7fdfe8142f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:09.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfe7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfe815c150 0x7fdfe8142f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57180/0 (socket says 172.21.15.189:57180) 2024-04-06T12:13:09.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfe7fff700 1 -- 172.21.15.189:0/1896024038 learned_addr learned my addr 172.21.15.189:0/1896024038 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:09.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfed708700 1 --2- 172.21.15.189:0/1896024038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfe815fcd0 0x7fdfe81434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:09.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfecf07700 1 --2- 172.21.15.189:0/1896024038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfe8144f40 0x7fdfe80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:09.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfe7fff700 1 -- 172.21.15.189:0/1896024038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfe8144f40 msgr2=0x7fdfe80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:09.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfe7fff700 1 --2- 172.21.15.189:0/1896024038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfe8144f40 0x7fdfe80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfe7fff700 1 -- 172.21.15.189:0/1896024038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfe815fcd0 msgr2=0x7fdfe81434a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:09.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfe7fff700 1 --2- 172.21.15.189:0/1896024038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfe815fcd0 0x7fdfe81434a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfe7fff700 1 -- 172.21.15.189:0/1896024038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc0094a0 con 0x7fdfe815c150 2024-04-06T12:13:09.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfecf07700 1 --2- 172.21.15.189:0/1896024038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfe8144f40 0x7fdfe80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:09.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfe7fff700 1 --2- 172.21.15.189:0/1896024038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfe815c150 0x7fdfe8142f60 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdfd400c450 tx=0x7fdfd400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:09.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfe57fa700 1 -- 172.21.15.189:0/1896024038 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd40052d0 con 0x7fdfe815c150 2024-04-06T12:13:09.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.365+0000 7fdfe57fa700 1 -- 172.21.15.189:0/1896024038 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd4005430 con 0x7fdfe815c150 2024-04-06T12:13:09.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.366+0000 7fdfef16b700 1 -- 172.21.15.189:0/1896024038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe8143f70 con 0x7fdfe815c150 2024-04-06T12:13:09.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.366+0000 7fdfe57fa700 1 -- 172.21.15.189:0/1896024038 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd40038c0 con 0x7fdfe815c150 2024-04-06T12:13:09.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.366+0000 7fdfef16b700 1 -- 172.21.15.189:0/1896024038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe8144530 con 0x7fdfe815c150 2024-04-06T12:13:09.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.366+0000 7fdfef16b700 1 -- 172.21.15.189:0/1896024038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe8067bb0 con 0x7fdfe815c150 2024-04-06T12:13:09.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.367+0000 7fdfe57fa700 1 -- 172.21.15.189:0/1896024038 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdfd4003a20 con 0x7fdfe815c150 2024-04-06T12:13:09.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.367+0000 7fdfe57fa700 1 --2- 172.21.15.189:0/1896024038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdfcc06ac90 0x7fdfcc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:09.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.367+0000 7fdfe57fa700 1 -- 172.21.15.189:0/1896024038 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fdfd4088a10 con 0x7fdfe815c150 2024-04-06T12:13:09.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.367+0000 7fdfecf07700 1 --2- 172.21.15.189:0/1896024038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdfcc06ac90 0x7fdfcc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:09.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.368+0000 7fdfecf07700 1 --2- 172.21.15.189:0/1896024038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdfcc06ac90 0x7fdfcc06d150 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc0090b0 tx=0x7fdfdc008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:09.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.372+0000 7fdfe57fa700 1 -- 172.21.15.189:0/1896024038 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdfd40538c0 con 0x7fdfe815c150 2024-04-06T12:13:09.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.614+0000 7fdfef16b700 1 -- 172.21.15.189:0/1896024038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fdfe8074990 con 0x7fdfe815c150 2024-04-06T12:13:09.614 INFO:tasks.ceph.mon.a.smithi189.stderr:2024-04-06T12:13:09.615+0000 7fa9fe1ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T12:13:09.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.628+0000 7fdfe57fa700 1 -- 172.21.15.189:0/1896024038 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) v1 ==== 161+0+0 (secure 0 0 0) 0x7fdfd4056ee0 con 0x7fdfe815c150 2024-04-06T12:13:09.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.630+0000 7fdfef16b700 1 -- 172.21.15.189:0/1896024038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdfcc06ac90 msgr2=0x7fdfcc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:09.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.630+0000 7fdfef16b700 1 --2- 172.21.15.189:0/1896024038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdfcc06ac90 0x7fdfcc06d150 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc0090b0 tx=0x7fdfdc008030 comp rx=0 tx=0).stop 2024-04-06T12:13:09.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.630+0000 7fdfef16b700 1 -- 172.21.15.189:0/1896024038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfe815c150 msgr2=0x7fdfe8142f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:09.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.630+0000 7fdfef16b700 1 --2- 172.21.15.189:0/1896024038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfe815c150 0x7fdfe8142f60 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdfd400c450 tx=0x7fdfd400c760 comp rx=0 tx=0).stop 2024-04-06T12:13:09.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.630+0000 7fdfef16b700 1 -- 172.21.15.189:0/1896024038 shutdown_connections 2024-04-06T12:13:09.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.630+0000 7fdfef16b700 1 --2- 172.21.15.189:0/1896024038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdfcc06ac90 0x7fdfcc06d150 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.630+0000 7fdfef16b700 1 --2- 172.21.15.189:0/1896024038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfe815fcd0 0x7fdfe81434a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.630+0000 7fdfef16b700 1 --2- 172.21.15.189:0/1896024038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfe8144f40 0x7fdfe80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.630+0000 7fdfef16b700 1 --2- 172.21.15.189:0/1896024038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfe815c150 0x7fdfe8142f60 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.630+0000 7fdfef16b700 1 -- 172.21.15.189:0/1896024038 >> 172.21.15.189:0/1896024038 conn(0x7fdfe80d00b0 msgr2=0x7fdfe815d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:09.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.630+0000 7fdfef16b700 1 -- 172.21.15.189:0/1896024038 shutdown_connections 2024-04-06T12:13:09.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.630+0000 7fdfef16b700 1 -- 172.21.15.189:0/1896024038 wait complete. 2024-04-06T12:13:09.630 INFO:teuthology.orchestra.run.smithi189.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-06T12:13:09.631 INFO:tasks.ceph.mds.b.smithi192.stderr: -5762> 2024-04-06T12:12:08.761+0000 7fdcdd88c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T12:13:09.641 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-06T12:13:09.664 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T12:13:09.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.766+0000 7f99fd229700 1 Processor -- start 2024-04-06T12:13:09.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.766+0000 7f99fd229700 1 -- start start 2024-04-06T12:13:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.767+0000 7f99fd229700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f99f80cb870 0x7f99f815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.767+0000 7f99fd229700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f80cbd80 con 0x7f99f80cfa00 2024-04-06T12:13:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.767+0000 7f99fd229700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f80cef20 con 0x7f99f80d11b0 2024-04-06T12:13:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.767+0000 7f99fd229700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f80ccdc0 con 0x7f99f80cb870 2024-04-06T12:13:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.767+0000 7f99f759e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f99f80cfa00 0x7f99f80cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40008/0 (socket says 172.21.15.189:40008) 2024-04-06T12:13:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.767+0000 7f99f759e700 1 -- 172.21.15.189:0/4047803481 learned_addr learned my addr 172.21.15.189:0/4047803481 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.767+0000 7f99f659c700 1 --2- 172.21.15.189:0/4047803481 >> v2:172.21.15.192:3300/0 conn(0x7f99f80cb870 0x7f99f815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.767+0000 7f99f5d9b700 1 -- 172.21.15.189:0/4047803481 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167346443 0 0) 0x7f99f80cbd80 con 0x7f99f80cfa00 2024-04-06T12:13:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.768+0000 7f99f5d9b700 1 -- 172.21.15.189:0/4047803481 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99e00035b0 con 0x7f99f80cfa00 2024-04-06T12:13:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.768+0000 7f99f659c700 1 -- 172.21.15.189:0/4047803481 >> v1:172.21.15.192:6789/0 conn(0x7f99f80d11b0 legacy=0x7f99f80d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.768+0000 7f99f659c700 1 -- 172.21.15.189:0/4047803481 >> v1:172.21.15.189:6789/0 conn(0x7f99f80cfa00 legacy=0x7f99f80cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.768+0000 7f99f5d9b700 1 -- 172.21.15.189:0/4047803481 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028085888 0 0) 0x7f99f80cef20 con 0x7f99f80d11b0 2024-04-06T12:13:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.768+0000 7f99f659c700 1 -- 172.21.15.189:0/4047803481 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f80cf200 con 0x7f99f80cb870 2024-04-06T12:13:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.768+0000 7f99f659c700 1 --2- 172.21.15.189:0/4047803481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99f80cb870 0x7f99f815c280 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f99e8009840 tx=0x7f99e8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b23cc10a1c5aaee8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.768+0000 7f99f5d9b700 1 -- 172.21.15.189:0/4047803481 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8011830 con 0x7f99f80cb870 2024-04-06T12:13:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.768+0000 7f99f5d9b700 1 -- 172.21.15.189:0/4047803481 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e8011990 con 0x7f99f80cb870 2024-04-06T12:13:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.768+0000 7f99f5d9b700 1 -- 172.21.15.189:0/4047803481 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8011d20 con 0x7f99f80cb870 2024-04-06T12:13:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.768+0000 7f99fd229700 1 -- 172.21.15.189:0/4047803481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99f80cb870 msgr2=0x7f99f815c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.768+0000 7f99fd229700 1 --2- 172.21.15.189:0/4047803481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99f80cb870 0x7f99f815c280 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f99e8009840 tx=0x7f99e8009b50 comp rx=0 tx=0).stop 2024-04-06T12:13:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.769+0000 7f99fd229700 1 -- 172.21.15.189:0/4047803481 shutdown_connections 2024-04-06T12:13:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.769+0000 7f99fd229700 1 --2- 172.21.15.189:0/4047803481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99f80cb870 0x7f99f815c280 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.769+0000 7f99fd229700 1 -- 172.21.15.189:0/4047803481 >> 172.21.15.189:0/4047803481 conn(0x7f99f80b2db0 msgr2=0x7f99f80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.769+0000 7f99fd229700 1 -- 172.21.15.189:0/4047803481 shutdown_connections 2024-04-06T12:13:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.769+0000 7f99fd229700 1 -- 172.21.15.189:0/4047803481 wait complete. 2024-04-06T12:13:09.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.770+0000 7f99fd229700 1 Processor -- start 2024-04-06T12:13:09.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.770+0000 7f99fd229700 1 -- start start 2024-04-06T12:13:09.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.771+0000 7f99fd229700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99f80cb870 0x7f99f8153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:09.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.771+0000 7f99fd229700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f80cfa00 0x7f99f81584f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:09.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.771+0000 7f99fd229700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99f80d11b0 0x7f99f8153750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:09.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.771+0000 7f99fd229700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f81e5470 con 0x7f99f80cfa00 2024-04-06T12:13:09.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.771+0000 7f99fd229700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f815cc80 con 0x7f99f80d11b0 2024-04-06T12:13:09.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.771+0000 7f99fd229700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f8158a30 con 0x7f99f80cb870 2024-04-06T12:13:09.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.771+0000 7f99f659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f80cfa00 0x7f99f81584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:09.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.771+0000 7f99f659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f80cfa00 0x7f99f81584f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57182/0 (socket says 172.21.15.189:57182) 2024-04-06T12:13:09.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.771+0000 7f99f659c700 1 -- 172.21.15.189:0/903576885 learned_addr learned my addr 172.21.15.189:0/903576885 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.771+0000 7f99f759e700 1 --2- 172.21.15.189:0/903576885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99f80d11b0 0x7f99f8153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.771+0000 7f99f6d9d700 1 --2- 172.21.15.189:0/903576885 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99f80cb870 0x7f99f8153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.771+0000 7f99f659c700 1 -- 172.21.15.189:0/903576885 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99f80cb870 msgr2=0x7f99f8153ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.771+0000 7f99f659c700 1 --2- 172.21.15.189:0/903576885 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99f80cb870 0x7f99f8153ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.771+0000 7f99f659c700 1 -- 172.21.15.189:0/903576885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99f80d11b0 msgr2=0x7f99f8153750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.771+0000 7f99f659c700 1 --2- 172.21.15.189:0/903576885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99f80d11b0 0x7f99f8153750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.771+0000 7f99f659c700 1 -- 172.21.15.189:0/903576885 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e80094a0 con 0x7f99f80cfa00 2024-04-06T12:13:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.771+0000 7f99f659c700 1 --2- 172.21.15.189:0/903576885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f80cfa00 0x7f99f81584f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f99e80090b0 tx=0x7f99e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.771+0000 7f99df7fe700 1 -- 172.21.15.189:0/903576885 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e801a7e0 con 0x7f99f80cfa00 2024-04-06T12:13:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.772+0000 7f99df7fe700 1 -- 172.21.15.189:0/903576885 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e801a940 con 0x7f99f80cfa00 2024-04-06T12:13:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.772+0000 7f99fd229700 1 -- 172.21.15.189:0/903576885 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f8153c90 con 0x7f99f80cfa00 2024-04-06T12:13:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.772+0000 7f99df7fe700 1 -- 172.21.15.189:0/903576885 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e801acd0 con 0x7f99f80cfa00 2024-04-06T12:13:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.772+0000 7f99fd229700 1 -- 172.21.15.189:0/903576885 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f81e4db0 con 0x7f99f80cfa00 2024-04-06T12:13:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.773+0000 7f99fd229700 1 -- 172.21.15.189:0/903576885 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f8074990 con 0x7f99f80cfa00 2024-04-06T12:13:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.773+0000 7f99df7fe700 1 -- 172.21.15.189:0/903576885 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f99e8022070 con 0x7f99f80cfa00 2024-04-06T12:13:09.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.773+0000 7f99df7fe700 1 --2- 172.21.15.189:0/903576885 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f99e406ac90 0x7f99e406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:09.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.773+0000 7f99df7fe700 1 -- 172.21.15.189:0/903576885 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f99e808a6e0 con 0x7f99f80cfa00 2024-04-06T12:13:09.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.774+0000 7f99f6d9d700 1 --2- 172.21.15.189:0/903576885 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f99e406ac90 0x7f99e406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:09.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.774+0000 7f99f6d9d700 1 --2- 172.21.15.189:0/903576885 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f99e406ac90 0x7f99e406d150 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f99ec000f60 tx=0x7f99ec009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:09.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:09.778+0000 7f99df7fe700 1 -- 172.21.15.189:0/903576885 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f99e8055540 con 0x7f99f80cfa00 2024-04-06T12:13:10.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.015+0000 7f99fd229700 1 -- 172.21.15.189:0/903576885 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f99f80cef10 con 0x7f99f80cfa00 2024-04-06T12:13:10.251 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:13:10.253 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:13:10.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.631+0000 7f99df7fe700 1 -- 172.21.15.189:0/903576885 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f99e8058b60 con 0x7f99f80cfa00 2024-04-06T12:13:10.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.633+0000 7f99fd229700 1 -- 172.21.15.189:0/903576885 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f99e406ac90 msgr2=0x7f99e406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:10.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.633+0000 7f99fd229700 1 --2- 172.21.15.189:0/903576885 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f99e406ac90 0x7f99e406d150 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f99ec000f60 tx=0x7f99ec009040 comp rx=0 tx=0).stop 2024-04-06T12:13:10.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.633+0000 7f99fd229700 1 -- 172.21.15.189:0/903576885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f80cfa00 msgr2=0x7f99f81584f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:10.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.633+0000 7f99fd229700 1 --2- 172.21.15.189:0/903576885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f80cfa00 0x7f99f81584f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f99e80090b0 tx=0x7f99e8004750 comp rx=0 tx=0).stop 2024-04-06T12:13:10.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.633+0000 7f99fd229700 1 -- 172.21.15.189:0/903576885 shutdown_connections 2024-04-06T12:13:10.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.633+0000 7f99fd229700 1 --2- 172.21.15.189:0/903576885 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f99e406ac90 0x7f99e406d150 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:10.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.633+0000 7f99fd229700 1 --2- 172.21.15.189:0/903576885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99f80d11b0 0x7f99f8153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:10.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.633+0000 7f99fd229700 1 --2- 172.21.15.189:0/903576885 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99f80cb870 0x7f99f8153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:10.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.633+0000 7f99fd229700 1 --2- 172.21.15.189:0/903576885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f80cfa00 0x7f99f81584f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:10.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.633+0000 7f99fd229700 1 -- 172.21.15.189:0/903576885 >> 172.21.15.189:0/903576885 conn(0x7f99f80b2db0 msgr2=0x7f99f80c6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:10.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.633+0000 7f99fd229700 1 -- 172.21.15.189:0/903576885 shutdown_connections 2024-04-06T12:13:10.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.633+0000 7f99fd229700 1 -- 172.21.15.189:0/903576885 wait complete. 2024-04-06T12:13:10.645 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-06T12:13:10.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.773+0000 7ff565d3a700 1 Processor -- start 2024-04-06T12:13:10.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.773+0000 7ff565d3a700 1 -- start start 2024-04-06T12:13:10.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.774+0000 7ff565d3a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff56015a550 0x7ff5600cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:10.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.774+0000 7ff565d3a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5600cf900 0x7ff5600b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:10.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.774+0000 7ff565d3a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5600cdbe0 con 0x7ff5600b4ba0 2024-04-06T12:13:10.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.774+0000 7ff565d3a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5600b5010 con 0x7ff56015a550 2024-04-06T12:13:10.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.774+0000 7ff565d3a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56015a940 con 0x7ff5600cf900 2024-04-06T12:13:10.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.774+0000 7ff55f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff56015a550 0x7ff5600cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:10.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.774+0000 7ff55f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff56015a550 0x7ff5600cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57190/0 (socket says 172.21.15.189:57190) 2024-04-06T12:13:10.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.774+0000 7ff55f7fe700 1 -- 172.21.15.189:0/2563981875 learned_addr learned my addr 172.21.15.189:0/2563981875 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:10.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.774+0000 7ff55effd700 1 --2- 172.21.15.189:0/2563981875 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5600cf900 0x7ff5600b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:10.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.774+0000 7ff55f7fe700 1 -- 172.21.15.189:0/2563981875 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5600cf900 msgr2=0x7ff5600b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:10.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.774+0000 7ff55f7fe700 1 --2- 172.21.15.189:0/2563981875 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5600cf900 0x7ff5600b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:10.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.774+0000 7ff55f7fe700 1 -- 172.21.15.189:0/2563981875 >> v1:172.21.15.192:6789/0 conn(0x7ff5600b4ba0 legacy=0x7ff56006eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:10.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.774+0000 7ff55f7fe700 1 -- 172.21.15.189:0/2563981875 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56015ac00 con 0x7ff56015a550 2024-04-06T12:13:10.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.774+0000 7ff55effd700 1 --2- 172.21.15.189:0/2563981875 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5600cf900 0x7ff5600b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:10.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.775+0000 7ff55f7fe700 1 --2- 172.21.15.189:0/2563981875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff56015a550 0x7ff5600cf3c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff55400bb90 tx=0x7ff55400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e3dfaebb56402d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:10.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.775+0000 7ff55e7fc700 1 -- 172.21.15.189:0/2563981875 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55400dd50 con 0x7ff56015a550 2024-04-06T12:13:10.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.775+0000 7ff55e7fc700 1 -- 172.21.15.189:0/2563981875 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff554010470 con 0x7ff56015a550 2024-04-06T12:13:10.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.775+0000 7ff55e7fc700 1 -- 172.21.15.189:0/2563981875 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff554010800 con 0x7ff56015a550 2024-04-06T12:13:10.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.775+0000 7ff565d3a700 1 -- 172.21.15.189:0/2563981875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff56015a550 msgr2=0x7ff5600cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:10.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.775+0000 7ff565d3a700 1 --2- 172.21.15.189:0/2563981875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff56015a550 0x7ff5600cf3c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff55400bb90 tx=0x7ff55400bea0 comp rx=0 tx=0).stop 2024-04-06T12:13:10.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.775+0000 7ff565d3a700 1 -- 172.21.15.189:0/2563981875 shutdown_connections 2024-04-06T12:13:10.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.775+0000 7ff565d3a700 1 --2- 172.21.15.189:0/2563981875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff56015a550 0x7ff5600cf3c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:10.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.775+0000 7ff565d3a700 1 --2- 172.21.15.189:0/2563981875 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5600cf900 0x7ff5600b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:10.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.775+0000 7ff565d3a700 1 -- 172.21.15.189:0/2563981875 >> 172.21.15.189:0/2563981875 conn(0x7ff5600d00b0 msgr2=0x7ff5600d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:10.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.775+0000 7ff565d3a700 1 -- 172.21.15.189:0/2563981875 shutdown_connections 2024-04-06T12:13:10.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.775+0000 7ff565d3a700 1 -- 172.21.15.189:0/2563981875 wait complete. 2024-04-06T12:13:10.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.776+0000 7ff565d3a700 1 Processor -- start 2024-04-06T12:13:10.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.777+0000 7ff565d3a700 1 -- start start 2024-04-06T12:13:10.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.777+0000 7ff565d3a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5600b4ba0 0x7ff5601504d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.777+0000 7ff565d3a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5600cf900 0x7ff5600c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.777+0000 7ff565d3a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5600c6410 0x7ff56015c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.777+0000 7ff565d3a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56014d450 con 0x7ff5600cf900 2024-04-06T12:13:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.777+0000 7ff565d3a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5600ae0f0 con 0x7ff5600c6410 2024-04-06T12:13:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.777+0000 7ff565d3a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5600ae3d0 con 0x7ff5600b4ba0 2024-04-06T12:13:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.778+0000 7ff55effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5600cf900 0x7ff5600c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.778+0000 7ff55effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5600cf900 0x7ff5600c5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57204/0 (socket says 172.21.15.189:57204) 2024-04-06T12:13:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.778+0000 7ff55effd700 1 -- 172.21.15.189:0/1062541506 learned_addr learned my addr 172.21.15.189:0/1062541506 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.778+0000 7ff55f7fe700 1 --2- 172.21.15.189:0/1062541506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5600b4ba0 0x7ff5601504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.778+0000 7ff55ffff700 1 --2- 172.21.15.189:0/1062541506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5600c6410 0x7ff56015c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.778+0000 7ff55effd700 1 -- 172.21.15.189:0/1062541506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5600b4ba0 msgr2=0x7ff5601504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.778+0000 7ff55effd700 1 --2- 172.21.15.189:0/1062541506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5600b4ba0 0x7ff5601504d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.778+0000 7ff55effd700 1 -- 172.21.15.189:0/1062541506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5600c6410 msgr2=0x7ff56015c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:10.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.778+0000 7ff55effd700 1 --2- 172.21.15.189:0/1062541506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5600c6410 0x7ff56015c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:10.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.778+0000 7ff55effd700 1 -- 172.21.15.189:0/1062541506 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55400b7f0 con 0x7ff5600cf900 2024-04-06T12:13:10.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.778+0000 7ff55effd700 1 --2- 172.21.15.189:0/1062541506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5600cf900 0x7ff5600c5ed0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff55000e810 tx=0x7ff55000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:10.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.778+0000 7ff547fff700 1 -- 172.21.15.189:0/1062541506 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff550004230 con 0x7ff5600cf900 2024-04-06T12:13:10.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.778+0000 7ff547fff700 1 -- 172.21.15.189:0/1062541506 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff550004390 con 0x7ff5600cf900 2024-04-06T12:13:10.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.778+0000 7ff547fff700 1 -- 172.21.15.189:0/1062541506 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5500036a0 con 0x7ff5600cf900 2024-04-06T12:13:10.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.778+0000 7ff565d3a700 1 -- 172.21.15.189:0/1062541506 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5600ae6b0 con 0x7ff5600cf900 2024-04-06T12:13:10.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.778+0000 7ff565d3a700 1 -- 172.21.15.189:0/1062541506 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff56014d940 con 0x7ff5600cf900 2024-04-06T12:13:10.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.779+0000 7ff565d3a700 1 -- 172.21.15.189:0/1062541506 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5600ac9f0 con 0x7ff5600cf900 2024-04-06T12:13:10.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.779+0000 7ff547fff700 1 -- 172.21.15.189:0/1062541506 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff550003800 con 0x7ff5600cf900 2024-04-06T12:13:10.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.780+0000 7ff547fff700 1 --2- 172.21.15.189:0/1062541506 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff54c06ac90 0x7ff54c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:10.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.780+0000 7ff547fff700 1 -- 172.21.15.189:0/1062541506 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6749+0+0 (secure 0 0 0) 0x7ff550089200 con 0x7ff5600cf900 2024-04-06T12:13:10.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.780+0000 7ff55f7fe700 1 --2- 172.21.15.189:0/1062541506 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff54c06ac90 0x7ff54c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:10.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.780+0000 7ff55f7fe700 1 --2- 172.21.15.189:0/1062541506 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff54c06ac90 0x7ff54c06d150 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff554000c00 tx=0x7ff55401b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:10.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.784+0000 7ff547fff700 1 -- 172.21.15.189:0/1062541506 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff550053fd0 con 0x7ff5600cf900 2024-04-06T12:13:10.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:10.979+0000 7ff565d3a700 1 -- 172.21.15.189:0/1062541506 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff5600c6820 con 0x7ff5600cf900 2024-04-06T12:13:11.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.634+0000 7ff547fff700 1 -- 172.21.15.189:0/1062541506 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v28) v1 ==== 181+0+0 (secure 0 0 0) 0x7ff5600c6820 con 0x7ff5600cf900 2024-04-06T12:13:11.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.639+0000 7ff565d3a700 1 -- 172.21.15.189:0/1062541506 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff54c06ac90 msgr2=0x7ff54c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:11.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.639+0000 7ff565d3a700 1 --2- 172.21.15.189:0/1062541506 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff54c06ac90 0x7ff54c06d150 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff554000c00 tx=0x7ff55401b040 comp rx=0 tx=0).stop 2024-04-06T12:13:11.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.639+0000 7ff565d3a700 1 -- 172.21.15.189:0/1062541506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5600cf900 msgr2=0x7ff5600c5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:11.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.639+0000 7ff565d3a700 1 --2- 172.21.15.189:0/1062541506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5600cf900 0x7ff5600c5ed0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff55000e810 tx=0x7ff55000ebd0 comp rx=0 tx=0).stop 2024-04-06T12:13:11.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.639+0000 7ff565d3a700 1 -- 172.21.15.189:0/1062541506 shutdown_connections 2024-04-06T12:13:11.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.639+0000 7ff565d3a700 1 --2- 172.21.15.189:0/1062541506 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff54c06ac90 0x7ff54c06d150 secure :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff554000c00 tx=0x7ff55401b040 comp rx=0 tx=0).stop 2024-04-06T12:13:11.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.639+0000 7ff565d3a700 1 --2- 172.21.15.189:0/1062541506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5600c6410 0x7ff56015c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:11.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.639+0000 7ff565d3a700 1 --2- 172.21.15.189:0/1062541506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5600b4ba0 0x7ff5601504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:11.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.639+0000 7ff565d3a700 1 --2- 172.21.15.189:0/1062541506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5600cf900 0x7ff5600c5ed0 secure :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff55000e810 tx=0x7ff55000ebd0 comp rx=0 tx=0).stop 2024-04-06T12:13:11.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.639+0000 7ff565d3a700 1 -- 172.21.15.189:0/1062541506 >> 172.21.15.189:0/1062541506 conn(0x7ff5600d00b0 msgr2=0x7ff5600717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:11.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.641+0000 7ff565d3a700 1 -- 172.21.15.189:0/1062541506 shutdown_connections 2024-04-06T12:13:11.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.641+0000 7ff565d3a700 1 -- 172.21.15.189:0/1062541506 wait complete. 2024-04-06T12:13:11.642 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' removed 2024-04-06T12:13:11.657 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-06T12:13:11.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.788+0000 7fb0a897d700 1 Processor -- start 2024-04-06T12:13:11.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.788+0000 7fb0a897d700 1 -- start start 2024-04-06T12:13:11.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.789+0000 7fb0a897d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb0a415a550 0x7fb0a40cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:11.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.789+0000 7fb0a897d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0a40cf900 0x7fb0a40b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:11.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.789+0000 7fb0a897d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a40cdbe0 con 0x7fb0a40b4ba0 2024-04-06T12:13:11.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.789+0000 7fb0a897d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a40b5010 con 0x7fb0a415a550 2024-04-06T12:13:11.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.789+0000 7fb0a897d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a415a940 con 0x7fb0a40cf900 2024-04-06T12:13:11.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.789+0000 7fb0a1d9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0a40cf900 0x7fb0a40b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:11.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.789+0000 7fb0a259c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb0a415a550 0x7fb0a40cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:11.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.789+0000 7fb0a1d9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0a40cf900 0x7fb0a40b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:40562/0 (socket says 172.21.15.189:40562) 2024-04-06T12:13:11.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.789+0000 7fb0a2d9d700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fb0a40b4ba0 0x7fb0a406eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:53048/0 (socket says 172.21.15.189:53048) 2024-04-06T12:13:11.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.789+0000 7fb0a259c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb0a415a550 0x7fb0a40cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:56770/0 (socket says 172.21.15.189:56770) 2024-04-06T12:13:11.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.789+0000 7fb0a1d9b700 1 -- 172.21.15.189:0/2996679742 learned_addr learned my addr 172.21.15.189:0/2996679742 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:11.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.790+0000 7fb0a1d9b700 1 -- 172.21.15.189:0/2996679742 >> v2:172.21.15.192:3300/0 conn(0x7fb0a415a550 msgr2=0x7fb0a40cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:11.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.790+0000 7fb0a1d9b700 1 --2- 172.21.15.189:0/2996679742 >> v2:172.21.15.192:3300/0 conn(0x7fb0a415a550 0x7fb0a40cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:11.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.790+0000 7fb0a1d9b700 1 -- 172.21.15.189:0/2996679742 >> v1:172.21.15.192:6789/0 conn(0x7fb0a40b4ba0 legacy=0x7fb0a406eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:11.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.790+0000 7fb0a1d9b700 1 -- 172.21.15.189:0/2996679742 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a415ac00 con 0x7fb0a40cf900 2024-04-06T12:13:11.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.790+0000 7fb0a259c700 1 --2- 172.21.15.189:0/2996679742 >> v2:172.21.15.192:3300/0 conn(0x7fb0a415a550 0x7fb0a40cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:13:11.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.790+0000 7fb0a1d9b700 1 --2- 172.21.15.189:0/2996679742 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0a40cf900 0x7fb0a40b4660 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb0940098f0 tx=0x7fb094009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecf1fc9cd23b2191 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:11.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.790+0000 7fb0a159a700 1 -- 172.21.15.189:0/2996679742 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094011830 con 0x7fb0a40cf900 2024-04-06T12:13:11.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.790+0000 7fb0a159a700 1 -- 172.21.15.189:0/2996679742 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb094011990 con 0x7fb0a40cf900 2024-04-06T12:13:11.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.790+0000 7fb0a159a700 1 -- 172.21.15.189:0/2996679742 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094011d20 con 0x7fb0a40cf900 2024-04-06T12:13:11.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.791+0000 7fb0a897d700 1 -- 172.21.15.189:0/2996679742 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0a40cf900 msgr2=0x7fb0a40b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:11.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.791+0000 7fb0a897d700 1 --2- 172.21.15.189:0/2996679742 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0a40cf900 0x7fb0a40b4660 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb0940098f0 tx=0x7fb094009c00 comp rx=0 tx=0).stop 2024-04-06T12:13:11.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.791+0000 7fb0a897d700 1 -- 172.21.15.189:0/2996679742 shutdown_connections 2024-04-06T12:13:11.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.791+0000 7fb0a897d700 1 --2- 172.21.15.189:0/2996679742 >> v2:172.21.15.192:3300/0 conn(0x7fb0a415a550 0x7fb0a40cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:11.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.791+0000 7fb0a897d700 1 --2- 172.21.15.189:0/2996679742 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0a40cf900 0x7fb0a40b4660 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:11.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.791+0000 7fb0a897d700 1 -- 172.21.15.189:0/2996679742 >> 172.21.15.189:0/2996679742 conn(0x7fb0a40d00b0 msgr2=0x7fb0a40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:11.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.791+0000 7fb0a897d700 1 -- 172.21.15.189:0/2996679742 shutdown_connections 2024-04-06T12:13:11.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.791+0000 7fb0a897d700 1 -- 172.21.15.189:0/2996679742 wait complete. 2024-04-06T12:13:11.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.792+0000 7fb0a897d700 1 Processor -- start 2024-04-06T12:13:11.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.792+0000 7fb0a897d700 1 -- start start 2024-04-06T12:13:11.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.793+0000 7fb0a897d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0a40b4ba0 0x7fb0a4156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:11.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.793+0000 7fb0a897d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0a40cf900 0x7fb0a40ad2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:11.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.793+0000 7fb0a897d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0a406ae60 0x7fb0a4151d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:11.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.793+0000 7fb0a897d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a4156fc0 con 0x7fb0a406ae60 2024-04-06T12:13:11.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.793+0000 7fb0a897d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a40adb40 con 0x7fb0a40b4ba0 2024-04-06T12:13:11.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.793+0000 7fb0a897d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a415b930 con 0x7fb0a40cf900 2024-04-06T12:13:11.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.793+0000 7fb0a2d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0a406ae60 0x7fb0a4151d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:11.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.793+0000 7fb0a2d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0a406ae60 0x7fb0a4151d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57210/0 (socket says 172.21.15.189:57210) 2024-04-06T12:13:11.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.793+0000 7fb0a2d9d700 1 -- 172.21.15.189:0/1508442077 learned_addr learned my addr 172.21.15.189:0/1508442077 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:11.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.793+0000 7fb0a1d9b700 1 --2- 172.21.15.189:0/1508442077 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0a40cf900 0x7fb0a40ad2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:11.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.793+0000 7fb0a259c700 1 --2- 172.21.15.189:0/1508442077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0a40b4ba0 0x7fb0a4156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:11.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.793+0000 7fb0a2d9d700 1 -- 172.21.15.189:0/1508442077 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0a40cf900 msgr2=0x7fb0a40ad2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:11.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.793+0000 7fb0a2d9d700 1 --2- 172.21.15.189:0/1508442077 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0a40cf900 0x7fb0a40ad2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:11.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.793+0000 7fb0a2d9d700 1 -- 172.21.15.189:0/1508442077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0a40b4ba0 msgr2=0x7fb0a4156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:11.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.793+0000 7fb0a2d9d700 1 --2- 172.21.15.189:0/1508442077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0a40b4ba0 0x7fb0a4156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:11.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.793+0000 7fb0a2d9d700 1 -- 172.21.15.189:0/1508442077 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0980094a0 con 0x7fb0a406ae60 2024-04-06T12:13:11.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.794+0000 7fb0a2d9d700 1 --2- 172.21.15.189:0/1508442077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0a406ae60 0x7fb0a4151d80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00cbd0 tx=0x7fb09c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:11.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.794+0000 7fb092ffd700 1 -- 172.21.15.189:0/1508442077 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c00abb0 con 0x7fb0a406ae60 2024-04-06T12:13:11.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.794+0000 7fb092ffd700 1 -- 172.21.15.189:0/1508442077 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09c00ad10 con 0x7fb0a406ae60 2024-04-06T12:13:11.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.794+0000 7fb0a897d700 1 -- 172.21.15.189:0/1508442077 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb094009550 con 0x7fb0a406ae60 2024-04-06T12:13:11.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.794+0000 7fb092ffd700 1 -- 172.21.15.189:0/1508442077 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c007270 con 0x7fb0a406ae60 2024-04-06T12:13:11.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.794+0000 7fb0a897d700 1 -- 172.21.15.189:0/1508442077 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a415bef0 con 0x7fb0a406ae60 2024-04-06T12:13:11.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.795+0000 7fb0a897d700 1 -- 172.21.15.189:0/1508442077 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a4067bb0 con 0x7fb0a406ae60 2024-04-06T12:13:11.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.795+0000 7fb092ffd700 1 -- 172.21.15.189:0/1508442077 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb09c0073d0 con 0x7fb0a406ae60 2024-04-06T12:13:11.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.795+0000 7fb092ffd700 1 --2- 172.21.15.189:0/1508442077 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb08406ac90 0x7fb08406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:11.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.796+0000 7fb092ffd700 1 -- 172.21.15.189:0/1508442077 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6311+0+0 (secure 0 0 0) 0x7fb09c089350 con 0x7fb0a406ae60 2024-04-06T12:13:11.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.796+0000 7fb0a259c700 1 --2- 172.21.15.189:0/1508442077 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb08406ac90 0x7fb08406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:11.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.796+0000 7fb0a259c700 1 --2- 172.21.15.189:0/1508442077 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb08406ac90 0x7fb08406d150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb098009c60 tx=0x7fb098008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:11.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.800+0000 7fb092ffd700 1 -- 172.21.15.189:0/1508442077 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb09c054300 con 0x7fb0a406ae60 2024-04-06T12:13:11.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:11.992+0000 7fb0a897d700 1 -- 172.21.15.189:0/1508442077 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb0a4074990 con 0x7fb0a406ae60 2024-04-06T12:13:12.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.647+0000 7fb092ffd700 1 -- 172.21.15.189:0/1508442077 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v29) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb09c057920 con 0x7fb0a406ae60 2024-04-06T12:13:12.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.652+0000 7fb0a897d700 1 -- 172.21.15.189:0/1508442077 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb08406ac90 msgr2=0x7fb08406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:12.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.652+0000 7fb0a897d700 1 --2- 172.21.15.189:0/1508442077 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb08406ac90 0x7fb08406d150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb098009c60 tx=0x7fb098008040 comp rx=0 tx=0).stop 2024-04-06T12:13:12.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.652+0000 7fb0a897d700 1 -- 172.21.15.189:0/1508442077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0a406ae60 msgr2=0x7fb0a4151d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:12.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.652+0000 7fb0a897d700 1 --2- 172.21.15.189:0/1508442077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0a406ae60 0x7fb0a4151d80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00cbd0 tx=0x7fb09c00cf90 comp rx=0 tx=0).stop 2024-04-06T12:13:12.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.652+0000 7fb0a897d700 1 -- 172.21.15.189:0/1508442077 shutdown_connections 2024-04-06T12:13:12.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.652+0000 7fb0a897d700 1 --2- 172.21.15.189:0/1508442077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0a406ae60 0x7fb0a4151d80 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:12.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.652+0000 7fb0a897d700 1 --2- 172.21.15.189:0/1508442077 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb08406ac90 0x7fb08406d150 secure :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb098009c60 tx=0x7fb098008040 comp rx=0 tx=0).stop 2024-04-06T12:13:12.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.652+0000 7fb0a897d700 1 --2- 172.21.15.189:0/1508442077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0a40b4ba0 0x7fb0a4156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:12.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.652+0000 7fb0a897d700 1 --2- 172.21.15.189:0/1508442077 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0a40cf900 0x7fb0a40ad2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:12.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.652+0000 7fb0a897d700 1 -- 172.21.15.189:0/1508442077 >> 172.21.15.189:0/1508442077 conn(0x7fb0a40d00b0 msgr2=0x7fb0a406eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:12.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.653+0000 7fb0a897d700 1 -- 172.21.15.189:0/1508442077 shutdown_connections 2024-04-06T12:13:12.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.653+0000 7fb0a897d700 1 -- 172.21.15.189:0/1508442077 wait complete. 2024-04-06T12:13:12.656 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' removed 2024-04-06T12:13:12.666 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-06T12:13:12.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.794+0000 7fc4878b8700 1 Processor -- start 2024-04-06T12:13:12.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.794+0000 7fc4878b8700 1 -- start start 2024-04-06T12:13:12.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.795+0000 7fc4878b8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc48015a550 0x7fc4800cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:12.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.795+0000 7fc4878b8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc4800cf900 0x7fc4800b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:12.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.795+0000 7fc4878b8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4800cdbe0 con 0x7fc4800b4ba0 2024-04-06T12:13:12.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.795+0000 7fc4878b8700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc48015a940 con 0x7fc4800cf900 2024-04-06T12:13:12.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.795+0000 7fc4878b8700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc48015ac00 con 0x7fc48015a550 2024-04-06T12:13:12.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.795+0000 7fc484e53700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc4800cf900 0x7fc4800b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:12.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.795+0000 7fc485e55700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc4800b4ba0 0x7fc48006eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40010/0 (socket says 172.21.15.189:40010) 2024-04-06T12:13:12.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.795+0000 7fc485e55700 1 -- 172.21.15.189:0/3806679199 learned_addr learned my addr 172.21.15.189:0/3806679199 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:12.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.796+0000 7fc485654700 1 --2- 172.21.15.189:0/3806679199 >> v2:172.21.15.192:3300/0 conn(0x7fc48015a550 0x7fc4800cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:12.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.796+0000 7fc484e53700 1 -- 172.21.15.189:0/3806679199 >> v2:172.21.15.192:3300/0 conn(0x7fc48015a550 msgr2=0x7fc4800cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:12.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.796+0000 7fc484e53700 1 --2- 172.21.15.189:0/3806679199 >> v2:172.21.15.192:3300/0 conn(0x7fc48015a550 0x7fc4800cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:12.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.796+0000 7fc484e53700 1 -- 172.21.15.189:0/3806679199 >> v1:172.21.15.189:6789/0 conn(0x7fc4800b4ba0 legacy=0x7fc48006eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:12.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.796+0000 7fc477fff700 1 -- 172.21.15.189:0/3806679199 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140923418 0 0) 0x7fc4800cdbe0 con 0x7fc4800b4ba0 2024-04-06T12:13:12.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.796+0000 7fc484e53700 1 -- 172.21.15.189:0/3806679199 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48015aee0 con 0x7fc4800cf900 2024-04-06T12:13:12.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.796+0000 7fc484e53700 1 --2- 172.21.15.189:0/3806679199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4800cf900 0x7fc4800b4660 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc480073470 tx=0x7fc47c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c7bd9af9c6d989 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:12.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.796+0000 7fc477fff700 1 -- 172.21.15.189:0/3806679199 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47c011830 con 0x7fc4800cf900 2024-04-06T12:13:12.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.796+0000 7fc477fff700 1 -- 172.21.15.189:0/3806679199 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc47c011990 con 0x7fc4800cf900 2024-04-06T12:13:12.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.796+0000 7fc477fff700 1 -- 172.21.15.189:0/3806679199 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47c011d20 con 0x7fc4800cf900 2024-04-06T12:13:12.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.797+0000 7fc4878b8700 1 -- 172.21.15.189:0/3806679199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4800cf900 msgr2=0x7fc4800b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:12.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.797+0000 7fc4878b8700 1 --2- 172.21.15.189:0/3806679199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4800cf900 0x7fc4800b4660 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc480073470 tx=0x7fc47c009b20 comp rx=0 tx=0).stop 2024-04-06T12:13:12.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.797+0000 7fc4878b8700 1 -- 172.21.15.189:0/3806679199 shutdown_connections 2024-04-06T12:13:12.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.797+0000 7fc4878b8700 1 --2- 172.21.15.189:0/3806679199 >> v2:172.21.15.192:3300/0 conn(0x7fc48015a550 0x7fc4800cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:12.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.797+0000 7fc4878b8700 1 --2- 172.21.15.189:0/3806679199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4800cf900 0x7fc4800b4660 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:12.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.797+0000 7fc4878b8700 1 -- 172.21.15.189:0/3806679199 >> 172.21.15.189:0/3806679199 conn(0x7fc4800d00b0 msgr2=0x7fc4800d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:12.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.797+0000 7fc4878b8700 1 -- 172.21.15.189:0/3806679199 shutdown_connections 2024-04-06T12:13:12.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.797+0000 7fc4878b8700 1 -- 172.21.15.189:0/3806679199 wait complete. 2024-04-06T12:13:12.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.798+0000 7fc4878b8700 1 Processor -- start 2024-04-06T12:13:12.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.798+0000 7fc4878b8700 1 -- start start 2024-04-06T12:13:12.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc4878b8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4800b4ba0 0x7fc480156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:12.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc4878b8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4800cf900 0x7fc4801ef250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:12.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc4878b8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4800a9050 0x7fc48006d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:12.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc4878b8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc480156fc0 con 0x7fc4800b4ba0 2024-04-06T12:13:12.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc4878b8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4801572a0 con 0x7fc4800cf900 2024-04-06T12:13:12.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc4878b8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc480157580 con 0x7fc4800a9050 2024-04-06T12:13:12.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc485654700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4800b4ba0 0x7fc480156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:12.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc485654700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4800b4ba0 0x7fc480156650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57236/0 (socket says 172.21.15.189:57236) 2024-04-06T12:13:12.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc485654700 1 -- 172.21.15.189:0/1210839023 learned_addr learned my addr 172.21.15.189:0/1210839023 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:12.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc484e53700 1 --2- 172.21.15.189:0/1210839023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4800cf900 0x7fc4801ef250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:12.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc485e55700 1 --2- 172.21.15.189:0/1210839023 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4800a9050 0x7fc48006d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:12.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc485654700 1 -- 172.21.15.189:0/1210839023 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4800a9050 msgr2=0x7fc48006d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:12.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc485654700 1 --2- 172.21.15.189:0/1210839023 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4800a9050 0x7fc48006d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:12.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc485654700 1 -- 172.21.15.189:0/1210839023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4800cf900 msgr2=0x7fc4801ef250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:12.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc485654700 1 --2- 172.21.15.189:0/1210839023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4800cf900 0x7fc4801ef250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:12.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc485654700 1 -- 172.21.15.189:0/1210839023 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47c0094a0 con 0x7fc4800b4ba0 2024-04-06T12:13:12.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc484e53700 1 --2- 172.21.15.189:0/1210839023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4800cf900 0x7fc4801ef250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:12.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc485e55700 1 --2- 172.21.15.189:0/1210839023 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4800a9050 0x7fc48006d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:12.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc485654700 1 --2- 172.21.15.189:0/1210839023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4800b4ba0 0x7fc480156650 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc47000d800 tx=0x7fc47000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:12.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc475ffb700 1 -- 172.21.15.189:0/1210839023 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4700125f0 con 0x7fc4800b4ba0 2024-04-06T12:13:12.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc475ffb700 1 -- 172.21.15.189:0/1210839023 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc470012750 con 0x7fc4800b4ba0 2024-04-06T12:13:12.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc4878b8700 1 -- 172.21.15.189:0/1210839023 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc480157860 con 0x7fc4800b4ba0 2024-04-06T12:13:12.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.799+0000 7fc475ffb700 1 -- 172.21.15.189:0/1210839023 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc470012b00 con 0x7fc4800b4ba0 2024-04-06T12:13:12.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.800+0000 7fc4878b8700 1 -- 172.21.15.189:0/1210839023 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc480156d70 con 0x7fc4800b4ba0 2024-04-06T12:13:12.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.801+0000 7fc4878b8700 1 -- 172.21.15.189:0/1210839023 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc480067bb0 con 0x7fc4800b4ba0 2024-04-06T12:13:12.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.801+0000 7fc475ffb700 1 -- 172.21.15.189:0/1210839023 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc470012c60 con 0x7fc4800b4ba0 2024-04-06T12:13:12.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.801+0000 7fc475ffb700 1 --2- 172.21.15.189:0/1210839023 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc46406ac90 0x7fc46406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:12.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.801+0000 7fc475ffb700 1 -- 172.21.15.189:0/1210839023 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fc470089700 con 0x7fc4800b4ba0 2024-04-06T12:13:12.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.801+0000 7fc484e53700 1 --2- 172.21.15.189:0/1210839023 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc46406ac90 0x7fc46406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:12.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.802+0000 7fc484e53700 1 --2- 172.21.15.189:0/1210839023 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc46406ac90 0x7fc46406d150 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc47c0090b0 tx=0x7fc47c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:12.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:12.806+0000 7fc475ffb700 1 -- 172.21.15.189:0/1210839023 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc470054890 con 0x7fc4800b4ba0 2024-04-06T12:13:13.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.001+0000 7fc4878b8700 1 -- 172.21.15.189:0/1210839023 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc480074990 con 0x7fc4800b4ba0 2024-04-06T12:13:13.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.651+0000 7fc475ffb700 1 -- 172.21.15.189:0/1210839023 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v30) v1 ==== 178+0+0 (secure 0 0 0) 0x7fc470057eb0 con 0x7fc4800b4ba0 2024-04-06T12:13:13.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.654+0000 7fc4878b8700 1 -- 172.21.15.189:0/1210839023 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc46406ac90 msgr2=0x7fc46406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:13.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.654+0000 7fc4878b8700 1 --2- 172.21.15.189:0/1210839023 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc46406ac90 0x7fc46406d150 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc47c0090b0 tx=0x7fc47c008030 comp rx=0 tx=0).stop 2024-04-06T12:13:13.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.654+0000 7fc4878b8700 1 -- 172.21.15.189:0/1210839023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4800b4ba0 msgr2=0x7fc480156650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:13.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.654+0000 7fc4878b8700 1 --2- 172.21.15.189:0/1210839023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4800b4ba0 0x7fc480156650 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc47000d800 tx=0x7fc47000dbc0 comp rx=0 tx=0).stop 2024-04-06T12:13:13.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.654+0000 7fc4878b8700 1 -- 172.21.15.189:0/1210839023 shutdown_connections 2024-04-06T12:13:13.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.654+0000 7fc4878b8700 1 --2- 172.21.15.189:0/1210839023 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4800a9050 0x7fc48006d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:13.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.654+0000 7fc4878b8700 1 --2- 172.21.15.189:0/1210839023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4800b4ba0 0x7fc480156650 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:13.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.654+0000 7fc4878b8700 1 --2- 172.21.15.189:0/1210839023 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc46406ac90 0x7fc46406d150 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:13.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.654+0000 7fc4878b8700 1 --2- 172.21.15.189:0/1210839023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4800cf900 0x7fc4801ef250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:13.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.654+0000 7fc4878b8700 1 -- 172.21.15.189:0/1210839023 >> 172.21.15.189:0/1210839023 conn(0x7fc4800d00b0 msgr2=0x7fc48006f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:13.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.654+0000 7fc4878b8700 1 -- 172.21.15.189:0/1210839023 shutdown_connections 2024-04-06T12:13:13.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.654+0000 7fc4878b8700 1 -- 172.21.15.189:0/1210839023 wait complete. 2024-04-06T12:13:13.654 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data_ec' removed 2024-04-06T12:13:13.668 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-06T12:13:13.668 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-06T12:13:13.668 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-06T12:13:13.669 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:13:13.669 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-06T12:13:13.669 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-06T12:13:13.669 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-06T12:13:13.669 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:13:13.669 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-06T12:13:13.669 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-06T12:13:13.670 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-06T12:13:13.670 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:13:13.670 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-06T12:13:13.670 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-06T12:13:13.670 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-06T12:13:13.671 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:13:13.679 INFO:tasks.ceph.mds.a:Stopped 2024-04-06T12:13:13.679 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-06T12:13:13.681 INFO:tasks.ceph.mds.c:Stopped 2024-04-06T12:13:13.681 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-06T12:13:13.683 INFO:tasks.ceph.mds.b:Stopped 2024-04-06T12:13:13.683 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-06T12:13:13.684 INFO:tasks.ceph.mds.d:Stopped 2024-04-06T12:13:13.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-06T12:13:13.723 INFO:tasks.ceph.mds.a:Started 2024-04-06T12:13:13.723 DEBUG:teuthology.parallel:result is None 2024-04-06T12:13:13.725 INFO:tasks.ceph.mds.c:Started 2024-04-06T12:13:13.725 DEBUG:teuthology.parallel:result is None 2024-04-06T12:13:13.726 INFO:tasks.ceph.mds.b:Started 2024-04-06T12:13:13.727 INFO:tasks.ceph.mds.d:Started 2024-04-06T12:13:13.727 DEBUG:teuthology.parallel:result is None 2024-04-06T12:13:13.728 DEBUG:teuthology.parallel:result is None 2024-04-06T12:13:13.728 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-06T12:13:13.806 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2024-04-06T12:13:13.807 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T12:13:13.811 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T12:13:13.811 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2024-04-06T12:13:13.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.866+0000 7f2aca36c700 1 Processor -- start 2024-04-06T12:13:13.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.866+0000 7f2aca36c700 1 -- start start 2024-04-06T12:13:13.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.867+0000 7f2aca36c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2ac415b510 0x7f2ac40724f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:13.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.867+0000 7f2aca36c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac40b1fb0 con 0x7f2ac40b3b80 2024-04-06T12:13:13.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.867+0000 7f2aca36c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac4071700 con 0x7f2ac415b130 2024-04-06T12:13:13.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.867+0000 7f2aca36c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac40b2200 con 0x7f2ac415b510 2024-04-06T12:13:13.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.867+0000 7f2ac37fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2ac40b3b80 0x7f2ac4071de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40026/0 (socket says 172.21.15.189:40026) 2024-04-06T12:13:13.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.867+0000 7f2ac37fe700 1 -- 172.21.15.189:0/3504560295 learned_addr learned my addr 172.21.15.189:0/3504560295 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:13.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.868+0000 7f2ac8909700 1 --2- 172.21.15.189:0/3504560295 >> v2:172.21.15.192:3300/0 conn(0x7f2ac415b510 0x7f2ac40724f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:13.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.868+0000 7f2ac2ffd700 1 -- 172.21.15.189:0/3504560295 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074160625 0 0) 0x7f2ac40b1fb0 con 0x7f2ac40b3b80 2024-04-06T12:13:13.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.868+0000 7f2ac2ffd700 1 -- 172.21.15.189:0/3504560295 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aac0035b0 con 0x7f2ac40b3b80 2024-04-06T12:13:13.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.868+0000 7f2ac2ffd700 1 -- 172.21.15.189:0/3504560295 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675255051 0 0) 0x7f2ac4071700 con 0x7f2ac415b130 2024-04-06T12:13:13.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.868+0000 7f2ac2ffd700 1 -- 172.21.15.189:0/3504560295 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac40b1fb0 con 0x7f2ac415b130 2024-04-06T12:13:13.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.868+0000 7f2ac8909700 1 -- 172.21.15.189:0/3504560295 >> v1:172.21.15.192:6789/0 conn(0x7f2ac415b130 legacy=0x7f2ac40b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:13.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.868+0000 7f2ac8909700 1 -- 172.21.15.189:0/3504560295 >> v1:172.21.15.189:6789/0 conn(0x7f2ac40b3b80 legacy=0x7f2ac4071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:13.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.868+0000 7f2ac8909700 1 -- 172.21.15.189:0/3504560295 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac40b24e0 con 0x7f2ac415b510 2024-04-06T12:13:13.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.868+0000 7f2ac8909700 1 --2- 172.21.15.189:0/3504560295 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ac415b510 0x7f2ac40724f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2abc009dc0 tx=0x7f2abc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5829d148e2303b22 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:13.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.868+0000 7f2ac2ffd700 1 -- 172.21.15.189:0/3504560295 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc00e070 con 0x7f2ac415b510 2024-04-06T12:13:13.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.868+0000 7f2ac2ffd700 1 -- 172.21.15.189:0/3504560295 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2abc00bae0 con 0x7f2ac415b510 2024-04-06T12:13:13.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.868+0000 7f2ac2ffd700 1 -- 172.21.15.189:0/3504560295 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc00be90 con 0x7f2ac415b510 2024-04-06T12:13:13.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.869+0000 7f2aca36c700 1 -- 172.21.15.189:0/3504560295 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ac415b510 msgr2=0x7f2ac40724f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:13.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.869+0000 7f2aca36c700 1 --2- 172.21.15.189:0/3504560295 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ac415b510 0x7f2ac40724f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2abc009dc0 tx=0x7f2abc00b330 comp rx=0 tx=0).stop 2024-04-06T12:13:13.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.869+0000 7f2aca36c700 1 -- 172.21.15.189:0/3504560295 shutdown_connections 2024-04-06T12:13:13.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.869+0000 7f2aca36c700 1 --2- 172.21.15.189:0/3504560295 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ac415b510 0x7f2ac40724f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:13.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.869+0000 7f2aca36c700 1 -- 172.21.15.189:0/3504560295 >> 172.21.15.189:0/3504560295 conn(0x7f2ac406a730 msgr2=0x7f2ac406ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:13.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.869+0000 7f2aca36c700 1 -- 172.21.15.189:0/3504560295 shutdown_connections 2024-04-06T12:13:13.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.869+0000 7f2aca36c700 1 -- 172.21.15.189:0/3504560295 wait complete. 2024-04-06T12:13:13.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.870+0000 7f2aca36c700 1 Processor -- start 2024-04-06T12:13:13.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.870+0000 7f2aca36c700 1 -- start start 2024-04-06T12:13:13.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.871+0000 7f2aca36c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ac40b3b80 0x7f2ac4150250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:13.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.871+0000 7f2aca36c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ac415b130 0x7f2ac4154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:13.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.871+0000 7f2aca36c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ac415b510 0x7f2ac4159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:13.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.871+0000 7f2aca36c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac4150790 con 0x7f2ac415b130 2024-04-06T12:13:13.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.871+0000 7f2aca36c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac41f1140 con 0x7f2ac415b510 2024-04-06T12:13:13.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.871+0000 7f2aca36c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac41f1420 con 0x7f2ac40b3b80 2024-04-06T12:13:13.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.871+0000 7f2ac37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ac415b130 0x7f2ac4154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:13.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.871+0000 7f2ac37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ac415b130 0x7f2ac4154530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57284/0 (socket says 172.21.15.189:57284) 2024-04-06T12:13:13.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.871+0000 7f2ac37fe700 1 -- 172.21.15.189:0/739131771 learned_addr learned my addr 172.21.15.189:0/739131771 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:13.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.871+0000 7f2ac8909700 1 --2- 172.21.15.189:0/739131771 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ac415b510 0x7f2ac4159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:13.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.871+0000 7f2ac3fff700 1 --2- 172.21.15.189:0/739131771 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ac40b3b80 0x7f2ac4150250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:13.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.871+0000 7f2ac37fe700 1 -- 172.21.15.189:0/739131771 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ac40b3b80 msgr2=0x7f2ac4150250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:13.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.871+0000 7f2ac37fe700 1 --2- 172.21.15.189:0/739131771 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ac40b3b80 0x7f2ac4150250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:13.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.871+0000 7f2ac37fe700 1 -- 172.21.15.189:0/739131771 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ac415b510 msgr2=0x7f2ac4159590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:13.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.871+0000 7f2ac37fe700 1 --2- 172.21.15.189:0/739131771 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ac415b510 0x7f2ac4159590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:13.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.871+0000 7f2ac37fe700 1 -- 172.21.15.189:0/739131771 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2abc009a70 con 0x7f2ac415b130 2024-04-06T12:13:13.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.871+0000 7f2ac37fe700 1 --2- 172.21.15.189:0/739131771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ac415b130 0x7f2ac4154530 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2ab400c4f0 tx=0x7f2ab400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:13.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.871+0000 7f2ac0ff9700 1 -- 172.21.15.189:0/739131771 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4005d70 con 0x7f2ac415b130 2024-04-06T12:13:13.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.872+0000 7f2ac0ff9700 1 -- 172.21.15.189:0/739131771 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab40070d0 con 0x7f2ac415b130 2024-04-06T12:13:13.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.872+0000 7f2aca36c700 1 -- 172.21.15.189:0/739131771 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac41f1700 con 0x7f2ac415b130 2024-04-06T12:13:13.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.872+0000 7f2ac0ff9700 1 -- 172.21.15.189:0/739131771 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4007440 con 0x7f2ac415b130 2024-04-06T12:13:13.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.872+0000 7f2aca36c700 1 -- 172.21.15.189:0/739131771 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac4154a70 con 0x7f2ac415b130 2024-04-06T12:13:13.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.872+0000 7f2aca36c700 1 -- 172.21.15.189:0/739131771 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac41f1c20 con 0x7f2ac415b130 2024-04-06T12:13:13.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.873+0000 7f2ac0ff9700 1 -- 172.21.15.189:0/739131771 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2ab40075a0 con 0x7f2ac415b130 2024-04-06T12:13:13.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.873+0000 7f2ac0ff9700 1 --2- 172.21.15.189:0/739131771 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2ab006ac90 0x7f2ab006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:13.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.873+0000 7f2ac0ff9700 1 -- 172.21.15.189:0/739131771 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f2ab40883f0 con 0x7f2ac415b130 2024-04-06T12:13:13.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.873+0000 7f2ac3fff700 1 --2- 172.21.15.189:0/739131771 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2ab006ac90 0x7f2ab006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:13.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.874+0000 7f2ac3fff700 1 --2- 172.21.15.189:0/739131771 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2ab006ac90 0x7f2ab006d150 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2ab80011e0 tx=0x7f2ab8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:13.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:13.878+0000 7f2ac0ff9700 1 -- 172.21.15.189:0/739131771 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2ab40538d0 con 0x7f2ac415b130 2024-04-06T12:13:14.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:14.096+0000 7f2aca36c700 1 -- 172.21.15.189:0/739131771 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2ac4073270 con 0x7f2ac415b130 2024-04-06T12:13:14.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:14.858+0000 7f2ac0ff9700 1 -- 172.21.15.189:0/739131771 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v35) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2ab4056ef0 con 0x7f2ac415b130 2024-04-06T12:13:14.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:14.861+0000 7f2aca36c700 1 -- 172.21.15.189:0/739131771 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2ab006ac90 msgr2=0x7f2ab006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:14.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:14.861+0000 7f2aca36c700 1 --2- 172.21.15.189:0/739131771 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2ab006ac90 0x7f2ab006d150 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2ab80011e0 tx=0x7f2ab8009040 comp rx=0 tx=0).stop 2024-04-06T12:13:14.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:14.861+0000 7f2aca36c700 1 -- 172.21.15.189:0/739131771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ac415b130 msgr2=0x7f2ac4154530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:14.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:14.861+0000 7f2aca36c700 1 --2- 172.21.15.189:0/739131771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ac415b130 0x7f2ac4154530 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2ab400c4f0 tx=0x7f2ab400c800 comp rx=0 tx=0).stop 2024-04-06T12:13:14.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:14.862+0000 7f2aca36c700 1 -- 172.21.15.189:0/739131771 shutdown_connections 2024-04-06T12:13:14.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:14.862+0000 7f2aca36c700 1 --2- 172.21.15.189:0/739131771 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2ab006ac90 0x7f2ab006d150 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:14.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:14.862+0000 7f2aca36c700 1 --2- 172.21.15.189:0/739131771 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ac415b510 0x7f2ac4159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:14.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:14.862+0000 7f2aca36c700 1 --2- 172.21.15.189:0/739131771 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ac40b3b80 0x7f2ac4150250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:14.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:14.862+0000 7f2aca36c700 1 --2- 172.21.15.189:0/739131771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ac415b130 0x7f2ac4154530 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:14.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:14.862+0000 7f2aca36c700 1 -- 172.21.15.189:0/739131771 >> 172.21.15.189:0/739131771 conn(0x7f2ac406a730 msgr2=0x7f2ac415e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:14.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:14.862+0000 7f2aca36c700 1 -- 172.21.15.189:0/739131771 shutdown_connections 2024-04-06T12:13:14.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:14.862+0000 7f2aca36c700 1 -- 172.21.15.189:0/739131771 wait complete. 2024-04-06T12:13:14.862 INFO:teuthology.orchestra.run.smithi189.stderr: removed all blocklist entries 2024-04-06T12:13:14.876 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-06T12:13:15.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.004+0000 7f8eb20a9700 1 Processor -- start 2024-04-06T12:13:15.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.004+0000 7f8eb20a9700 1 -- start start 2024-04-06T12:13:15.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.005+0000 7f8eb20a9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8eac15d7b0 0x7f8eac0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:15.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.005+0000 7f8eb20a9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8eac06a730 0x7f8eac06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:15.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.005+0000 7f8eb20a9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eac0b3990 con 0x7f8eac15ddf0 2024-04-06T12:13:15.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.005+0000 7f8eb20a9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eac15e950 con 0x7f8eac06a730 2024-04-06T12:13:15.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.005+0000 7f8eb20a9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eac06b0f0 con 0x7f8eac15d7b0 2024-04-06T12:13:15.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.005+0000 7f8eaaffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8eac06a730 0x7f8eac06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:15.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.005+0000 7f8eaaffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8eac06a730 0x7f8eac06abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57286/0 (socket says 172.21.15.189:57286) 2024-04-06T12:13:15.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.005+0000 7f8eaaffd700 1 -- 172.21.15.189:0/927703205 learned_addr learned my addr 172.21.15.189:0/927703205 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:15.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.005+0000 7f8eab7fe700 1 --2- 172.21.15.189:0/927703205 >> v2:172.21.15.192:3300/0 conn(0x7f8eac15d7b0 0x7f8eac0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:15.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.005+0000 7f8eaaffd700 1 -- 172.21.15.189:0/927703205 >> v2:172.21.15.192:3300/0 conn(0x7f8eac15d7b0 msgr2=0x7f8eac0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:15.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.005+0000 7f8eaaffd700 1 --2- 172.21.15.189:0/927703205 >> v2:172.21.15.192:3300/0 conn(0x7f8eac15d7b0 0x7f8eac0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.005+0000 7f8eaaffd700 1 -- 172.21.15.189:0/927703205 >> v1:172.21.15.192:6789/0 conn(0x7f8eac15ddf0 legacy=0x7f8eac15e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:15.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.005+0000 7f8eaaffd700 1 -- 172.21.15.189:0/927703205 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eac0b3060 con 0x7f8eac06a730 2024-04-06T12:13:15.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.005+0000 7f8eaaffd700 1 --2- 172.21.15.189:0/927703205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eac06a730 0x7f8eac06abb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8eac0b6200 tx=0x7f8e9c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49918929bd43348e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:15.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.006+0000 7f8eaa7fc700 1 -- 172.21.15.189:0/927703205 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c00b870 con 0x7f8eac06a730 2024-04-06T12:13:15.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.006+0000 7f8eaa7fc700 1 -- 172.21.15.189:0/927703205 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9c00b9d0 con 0x7f8eac06a730 2024-04-06T12:13:15.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.006+0000 7f8eaa7fc700 1 -- 172.21.15.189:0/927703205 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c00bd60 con 0x7f8eac06a730 2024-04-06T12:13:15.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.006+0000 7f8eb20a9700 1 -- 172.21.15.189:0/927703205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eac06a730 msgr2=0x7f8eac06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:15.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.006+0000 7f8eb20a9700 1 --2- 172.21.15.189:0/927703205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eac06a730 0x7f8eac06abb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8eac0b6200 tx=0x7f8e9c009bd0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.006+0000 7f8eb20a9700 1 -- 172.21.15.189:0/927703205 shutdown_connections 2024-04-06T12:13:15.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.006+0000 7f8eb20a9700 1 --2- 172.21.15.189:0/927703205 >> v2:172.21.15.192:3300/0 conn(0x7f8eac15d7b0 0x7f8eac0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.006+0000 7f8eb20a9700 1 --2- 172.21.15.189:0/927703205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eac06a730 0x7f8eac06abb0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.006+0000 7f8eb20a9700 1 -- 172.21.15.189:0/927703205 >> 172.21.15.189:0/927703205 conn(0x7f8eac0d00b0 msgr2=0x7f8eac0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:15.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.006+0000 7f8eb20a9700 1 -- 172.21.15.189:0/927703205 shutdown_connections 2024-04-06T12:13:15.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.006+0000 7f8eb20a9700 1 -- 172.21.15.189:0/927703205 wait complete. 2024-04-06T12:13:15.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.007+0000 7f8eb20a9700 1 Processor -- start 2024-04-06T12:13:15.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.008+0000 7f8eb20a9700 1 -- start start 2024-04-06T12:13:15.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.008+0000 7f8eb20a9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8eac06a730 0x7f8eac1f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:15.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.008+0000 7f8eb20a9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8eac15d7b0 0x7f8eac15b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:15.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.008+0000 7f8eb20a9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eac15ddf0 0x7f8eac15bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:15.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.008+0000 7f8eb20a9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac0aeb80 con 0x7f8eac15ddf0 2024-04-06T12:13:15.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.008+0000 7f8eb20a9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac0cde20 con 0x7f8eac06a730 2024-04-06T12:13:15.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.008+0000 7f8eb20a9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac0afb20 con 0x7f8eac15d7b0 2024-04-06T12:13:15.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.008+0000 7f8eabfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eac15ddf0 0x7f8eac15bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:15.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.008+0000 7f8eabfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eac15ddf0 0x7f8eac15bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57292/0 (socket says 172.21.15.189:57292) 2024-04-06T12:13:15.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.008+0000 7f8eabfff700 1 -- 172.21.15.189:0/3444948855 learned_addr learned my addr 172.21.15.189:0/3444948855 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:15.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.009+0000 7f8eab7fe700 1 --2- 172.21.15.189:0/3444948855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8eac06a730 0x7f8eac1f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:15.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.009+0000 7f8eaaffd700 1 --2- 172.21.15.189:0/3444948855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8eac15d7b0 0x7f8eac15b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:15.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.009+0000 7f8eabfff700 1 -- 172.21.15.189:0/3444948855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8eac15d7b0 msgr2=0x7f8eac15b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:15.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.009+0000 7f8eabfff700 1 --2- 172.21.15.189:0/3444948855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8eac15d7b0 0x7f8eac15b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.009+0000 7f8eabfff700 1 -- 172.21.15.189:0/3444948855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8eac06a730 msgr2=0x7f8eac1f2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:15.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.009+0000 7f8eabfff700 1 --2- 172.21.15.189:0/3444948855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8eac06a730 0x7f8eac1f2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.009+0000 7f8eabfff700 1 -- 172.21.15.189:0/3444948855 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c009550 con 0x7f8eac15ddf0 2024-04-06T12:13:15.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.009+0000 7f8eab7fe700 1 --2- 172.21.15.189:0/3444948855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8eac06a730 0x7f8eac1f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:15.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.009+0000 7f8eabfff700 1 --2- 172.21.15.189:0/3444948855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eac15ddf0 0x7f8eac15bf10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400cb50 tx=0x7f8ea400cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:15.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.009+0000 7f8ea3fff700 1 -- 172.21.15.189:0/3444948855 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4003c00 con 0x7f8eac15ddf0 2024-04-06T12:13:15.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.009+0000 7f8ea3fff700 1 -- 172.21.15.189:0/3444948855 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea4003d60 con 0x7f8eac15ddf0 2024-04-06T12:13:15.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.009+0000 7f8eb20a9700 1 -- 172.21.15.189:0/3444948855 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eac0aed60 con 0x7f8eac15ddf0 2024-04-06T12:13:15.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.009+0000 7f8ea3fff700 1 -- 172.21.15.189:0/3444948855 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea400ab10 con 0x7f8eac15ddf0 2024-04-06T12:13:15.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.009+0000 7f8eb20a9700 1 -- 172.21.15.189:0/3444948855 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eac0af320 con 0x7f8eac15ddf0 2024-04-06T12:13:15.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.010+0000 7f8eb20a9700 1 -- 172.21.15.189:0/3444948855 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eac067bb0 con 0x7f8eac15ddf0 2024-04-06T12:13:15.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.010+0000 7f8ea3fff700 1 -- 172.21.15.189:0/3444948855 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8ea400ac70 con 0x7f8eac15ddf0 2024-04-06T12:13:15.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.011+0000 7f8ea3fff700 1 --2- 172.21.15.189:0/3444948855 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8e8c06ac90 0x7f8e8c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:15.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.011+0000 7f8ea3fff700 1 -- 172.21.15.189:0/3444948855 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f8ea4089190 con 0x7f8eac15ddf0 2024-04-06T12:13:15.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.011+0000 7f8eab7fe700 1 --2- 172.21.15.189:0/3444948855 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8e8c06ac90 0x7f8e8c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:15.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.011+0000 7f8eab7fe700 1 --2- 172.21.15.189:0/3444948855 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8e8c06ac90 0x7f8e8c06d150 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8e940099f0 tx=0x7f8e94008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:15.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.015+0000 7f8ea3fff700 1 -- 172.21.15.189:0/3444948855 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8ea4054500 con 0x7f8eac15ddf0 2024-04-06T12:13:15.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.288+0000 7f8eb20a9700 1 -- 172.21.15.189:0/3444948855 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8eac074990 con 0x7f8eac15ddf0 2024-04-06T12:13:15.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.288+0000 7f8ea3fff700 1 -- 172.21.15.189:0/3444948855 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f8ea4057b20 con 0x7f8eac15ddf0 2024-04-06T12:13:15.287 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:13:15.287 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T12:13:15.288 INFO:teuthology.orchestra.run.smithi189.stdout: "auth_dump": [ 2024-04-06T12:13:15.288 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.288 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.a", 2024-04-06T12:13:15.288 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmTjYMNBAA5voWGV4Pa7pE27VogrdipQ==", 2024-04-06T12:13:15.288 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.288 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:13:15.288 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:13:15.288 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:13:15.288 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:13:15.288 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.288 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.288 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.289 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.b", 2024-04-06T12:13:15.289 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm9A1OBRAAnHzubCvRUX8UdW8hmDzKbQ==", 2024-04-06T12:13:15.289 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.289 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:13:15.289 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:13:15.289 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:13:15.289 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:13:15.289 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.289 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.289 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.289 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.c", 2024-04-06T12:13:15.289 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmz0hhOxAAD3/GINNmGoFk+oPyiHHJXg==", 2024-04-06T12:13:15.289 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.289 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:13:15.289 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:13:15.290 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:13:15.290 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:13:15.290 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.290 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.290 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.290 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.d", 2024-04-06T12:13:15.290 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmhoAeDRAAoQC0vicWumBRTwpi3PtYaw==", 2024-04-06T12:13:15.290 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.290 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:13:15.290 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:13:15.290 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:13:15.290 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:13:15.290 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.290 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.291 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.291 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.0", 2024-04-06T12:13:15.291 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDkOxFmkxLWJhAAFXewCNOFX7dhAZVgUmewpQ==", 2024-04-06T12:13:15.291 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.291 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:13:15.291 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:13:15.291 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:13:15.291 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.291 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.291 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.291 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.1", 2024-04-06T12:13:15.291 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDnOxFmHYqWJRAABlsVCiv3e7YJMbfodJTWag==", 2024-04-06T12:13:15.291 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.291 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:13:15.291 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:13:15.292 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:13:15.292 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.292 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.292 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.292 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.2", 2024-04-06T12:13:15.292 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDqOxFmkhcvJBAA8NMEpHVGbGALjV9RISKP8g==", 2024-04-06T12:13:15.292 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.292 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:13:15.292 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:13:15.292 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:13:15.292 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.292 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.292 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.292 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.3", 2024-04-06T12:13:15.292 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDtOxFmDGzhIxAAamBoAbE8+lAel4os67meyQ==", 2024-04-06T12:13:15.292 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.293 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:13:15.293 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:13:15.293 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:13:15.293 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.293 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.293 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.293 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.4", 2024-04-06T12:13:15.293 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDzOxFmobZDJxAA9l8wdS81Eyw6+17I5ac6bw==", 2024-04-06T12:13:15.293 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.293 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:13:15.293 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:13:15.293 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:13:15.293 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.293 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.293 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.294 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.5", 2024-04-06T12:13:15.294 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD2OxFmi+ruJRAASlQ0o9jX/wA7U5AosnL+Wg==", 2024-04-06T12:13:15.294 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.294 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:13:15.294 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:13:15.294 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:13:15.294 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.294 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.294 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.294 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.6", 2024-04-06T12:13:15.294 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD5OxFmbwuuJBAA0wqJRmnJyskQPZJ7/LnpvA==", 2024-04-06T12:13:15.294 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.294 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:13:15.294 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:13:15.294 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:13:15.295 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.295 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.295 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.295 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.7", 2024-04-06T12:13:15.295 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD8OxFmDmRgIxAAzyW9AGAbmb67ZvB5qunwLQ==", 2024-04-06T12:13:15.295 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.295 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:13:15.295 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:13:15.295 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:13:15.296 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.296 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.296 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.296 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.0", 2024-04-06T12:13:15.296 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm0ByJERAAoEV8Yfc09+Go6u95XafElQ==", 2024-04-06T12:13:15.296 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.296 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:13:15.296 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:13:15.296 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:13:15.296 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:13:15.296 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.296 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.296 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.296 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.1", 2024-04-06T12:13:15.296 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmSV5lGBAAavDEo2LGy0IHpWSAuitHXg==", 2024-04-06T12:13:15.297 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.297 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:13:15.297 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:13:15.297 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:13:15.297 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:13:15.297 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.297 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.297 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.297 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.2", 2024-04-06T12:13:15.297 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm/Gz5FBAA3xX/rsAyWZ1JoUVlQlBIvQ==", 2024-04-06T12:13:15.297 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.297 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:13:15.297 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:13:15.297 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:13:15.297 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:13:15.297 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.298 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.298 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.298 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.3", 2024-04-06T12:13:15.298 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmzra7GxAADbnzIvWnRTo65LvPwtxWrg==", 2024-04-06T12:13:15.298 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.298 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:13:15.298 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:13:15.298 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:13:15.298 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:13:15.298 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.298 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.298 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.298 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.admin", 2024-04-06T12:13:15.298 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmxG+wCBAANg5kt6TOUQdj3/X0Mqgx1g==", 2024-04-06T12:13:15.299 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.299 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:13:15.299 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:13:15.299 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:13:15.299 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:13:15.299 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.299 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.299 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.299 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.bootstrap-mds", 2024-04-06T12:13:15.299 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQACPBFmAHExIRAAVfz4d9Umg/eIvY7jcZ8Auw==", 2024-04-06T12:13:15.299 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.299 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile bootstrap-mds" 2024-04-06T12:13:15.299 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.299 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.299 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.299 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.bootstrap-mgr", 2024-04-06T12:13:15.300 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQACPBFmtXkxIRAA7wQ7n1XasqvpN0KwLuIbPA==", 2024-04-06T12:13:15.300 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.300 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-06T12:13:15.300 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.300 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.300 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.300 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.bootstrap-osd", 2024-04-06T12:13:15.300 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQACPBFmIoIxIRAA5eSoYtQ5VLFNPm6ptN7JZw==", 2024-04-06T12:13:15.300 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.300 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile bootstrap-osd" 2024-04-06T12:13:15.300 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.300 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.300 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.300 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.bootstrap-rbd", 2024-04-06T12:13:15.301 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQACPBFmuYkxIRAAjbIBuDWVo6Aih1l1e17g+g==", 2024-04-06T12:13:15.301 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.301 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-06T12:13:15.301 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.301 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.301 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.301 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-06T12:13:15.301 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQACPBFmV5ExIRAAgLdu9yj7TQc7jAhOLZo6kg==", 2024-04-06T12:13:15.301 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.301 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-06T12:13:15.301 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.301 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.301 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.301 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.bootstrap-rgw", 2024-04-06T12:13:15.301 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQACPBFmCZkxIRAABhXDJITys3oD2cYzVtHOsg==", 2024-04-06T12:13:15.302 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.302 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-06T12:13:15.302 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.302 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.302 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.302 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.x", 2024-04-06T12:13:15.302 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmPLyBMBAAYfhkI9Z1CPlf1Tijm+L7xA==", 2024-04-06T12:13:15.302 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.302 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:13:15.302 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T12:13:15.302 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:13:15.302 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.302 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:13:15.302 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:15.303 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.y", 2024-04-06T12:13:15.303 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFm1nHJHhAAQWRe8cU8duWuUJiYaDeTaQ==", 2024-04-06T12:13:15.303 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:13:15.303 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:13:15.303 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T12:13:15.303 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:13:15.303 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.303 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:15.303 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-04-06T12:13:15.303 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T12:13:15.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.290+0000 7f8eb20a9700 1 -- 172.21.15.189:0/3444948855 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8e8c06ac90 msgr2=0x7f8e8c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:15.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.290+0000 7f8eb20a9700 1 --2- 172.21.15.189:0/3444948855 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8e8c06ac90 0x7f8e8c06d150 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8e940099f0 tx=0x7f8e94008040 comp rx=0 tx=0).stop 2024-04-06T12:13:15.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.290+0000 7f8eb20a9700 1 -- 172.21.15.189:0/3444948855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eac15ddf0 msgr2=0x7f8eac15bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:15.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.290+0000 7f8eb20a9700 1 --2- 172.21.15.189:0/3444948855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eac15ddf0 0x7f8eac15bf10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400cb50 tx=0x7f8ea400cf10 comp rx=0 tx=0).stop 2024-04-06T12:13:15.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.291+0000 7f8eb20a9700 1 -- 172.21.15.189:0/3444948855 shutdown_connections 2024-04-06T12:13:15.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.291+0000 7f8eb20a9700 1 --2- 172.21.15.189:0/3444948855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eac15ddf0 0x7f8eac15bf10 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.291+0000 7f8eb20a9700 1 --2- 172.21.15.189:0/3444948855 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8e8c06ac90 0x7f8e8c06d150 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.291+0000 7f8eb20a9700 1 --2- 172.21.15.189:0/3444948855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8eac06a730 0x7f8eac1f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.291+0000 7f8eb20a9700 1 --2- 172.21.15.189:0/3444948855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8eac15d7b0 0x7f8eac15b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.291+0000 7f8eb20a9700 1 -- 172.21.15.189:0/3444948855 >> 172.21.15.189:0/3444948855 conn(0x7f8eac0d00b0 msgr2=0x7f8eac0726f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:15.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.291+0000 7f8eb20a9700 1 -- 172.21.15.189:0/3444948855 shutdown_connections 2024-04-06T12:13:15.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.291+0000 7f8eb20a9700 1 -- 172.21.15.189:0/3444948855 wait complete. 2024-04-06T12:13:15.306 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-06T12:13:15.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.429+0000 7fd27b01c700 1 Processor -- start 2024-04-06T12:13:15.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.430+0000 7fd27b01c700 1 -- start start 2024-04-06T12:13:15.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.430+0000 7fd27b01c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd2740d11b0 0x7fd2740d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:15.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.430+0000 7fd27b01c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd2740cb870 0x7fd27415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:15.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.430+0000 7fd27b01c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2740d1ad0 con 0x7fd2740cfa00 2024-04-06T12:13:15.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.430+0000 7fd27b01c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2740ccdc0 con 0x7fd2740cb870 2024-04-06T12:13:15.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.430+0000 7fd27b01c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2740d0b50 con 0x7fd2740d11b0 2024-04-06T12:13:15.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.430+0000 7fd273fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd2740cb870 0x7fd27415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:15.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.431+0000 7fd2795b9700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd2740cfa00 0x7fd2740cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40036/0 (socket says 172.21.15.189:40036) 2024-04-06T12:13:15.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.431+0000 7fd2795b9700 1 -- 172.21.15.189:0/3207184243 learned_addr learned my addr 172.21.15.189:0/3207184243 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:15.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.431+0000 7fd278db8700 1 --2- 172.21.15.189:0/3207184243 >> v2:172.21.15.192:3300/0 conn(0x7fd2740d11b0 0x7fd2740d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:15.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.431+0000 7fd273fff700 1 -- 172.21.15.189:0/3207184243 >> v2:172.21.15.192:3300/0 conn(0x7fd2740d11b0 msgr2=0x7fd2740d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:15.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.431+0000 7fd273fff700 1 --2- 172.21.15.189:0/3207184243 >> v2:172.21.15.192:3300/0 conn(0x7fd2740d11b0 0x7fd2740d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.431+0000 7fd273fff700 1 -- 172.21.15.189:0/3207184243 >> v1:172.21.15.189:6789/0 conn(0x7fd2740cfa00 legacy=0x7fd2740cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:15.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.431+0000 7fd273fff700 1 -- 172.21.15.189:0/3207184243 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2740cbd80 con 0x7fd2740cb870 2024-04-06T12:13:15.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.431+0000 7fd2737fe700 1 -- 172.21.15.189:0/3207184243 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930509387 0 0) 0x7fd2740d1ad0 con 0x7fd2740cfa00 2024-04-06T12:13:15.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.431+0000 7fd273fff700 1 --2- 172.21.15.189:0/3207184243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2740cb870 0x7fd27415c280 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd2740ced00 tx=0x7fd260009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246dd88ebd926f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:15.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.431+0000 7fd2737fe700 1 -- 172.21.15.189:0/3207184243 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd260011830 con 0x7fd2740cb870 2024-04-06T12:13:15.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.431+0000 7fd2737fe700 1 -- 172.21.15.189:0/3207184243 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd260011990 con 0x7fd2740cb870 2024-04-06T12:13:15.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.431+0000 7fd2737fe700 1 -- 172.21.15.189:0/3207184243 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd260011d20 con 0x7fd2740cb870 2024-04-06T12:13:15.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.432+0000 7fd27b01c700 1 -- 172.21.15.189:0/3207184243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2740cb870 msgr2=0x7fd27415c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:15.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.432+0000 7fd27b01c700 1 --2- 172.21.15.189:0/3207184243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2740cb870 0x7fd27415c280 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd2740ced00 tx=0x7fd260009b20 comp rx=0 tx=0).stop 2024-04-06T12:13:15.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.432+0000 7fd27b01c700 1 -- 172.21.15.189:0/3207184243 shutdown_connections 2024-04-06T12:13:15.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.432+0000 7fd27b01c700 1 --2- 172.21.15.189:0/3207184243 >> v2:172.21.15.192:3300/0 conn(0x7fd2740d11b0 0x7fd2740d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.432+0000 7fd27b01c700 1 --2- 172.21.15.189:0/3207184243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2740cb870 0x7fd27415c280 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.432+0000 7fd27b01c700 1 -- 172.21.15.189:0/3207184243 >> 172.21.15.189:0/3207184243 conn(0x7fd2740b2db0 msgr2=0x7fd2740b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:15.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.432+0000 7fd27b01c700 1 -- 172.21.15.189:0/3207184243 shutdown_connections 2024-04-06T12:13:15.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.432+0000 7fd27b01c700 1 -- 172.21.15.189:0/3207184243 wait complete. 2024-04-06T12:13:15.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.433+0000 7fd27b01c700 1 Processor -- start 2024-04-06T12:13:15.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.433+0000 7fd27b01c700 1 -- start start 2024-04-06T12:13:15.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.434+0000 7fd27b01c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2740cb870 0x7fd2740c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:15.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.434+0000 7fd27b01c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2740cfa00 0x7fd2741582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:15.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.434+0000 7fd27b01c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2740d11b0 0x7fd274158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:15.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.434+0000 7fd27b01c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2740c7da0 con 0x7fd2740d11b0 2024-04-06T12:13:15.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.434+0000 7fd27b01c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2740c7770 con 0x7fd2740cfa00 2024-04-06T12:13:15.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.434+0000 7fd27b01c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2740cafd0 con 0x7fd2740cb870 2024-04-06T12:13:15.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.434+0000 7fd2795b9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2740d11b0 0x7fd274158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:15.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.434+0000 7fd2795b9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2740d11b0 0x7fd274158810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57310/0 (socket says 172.21.15.189:57310) 2024-04-06T12:13:15.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.434+0000 7fd2795b9700 1 -- 172.21.15.189:0/2011695917 learned_addr learned my addr 172.21.15.189:0/2011695917 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:15.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.434+0000 7fd278db8700 1 --2- 172.21.15.189:0/2011695917 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2740cb870 0x7fd2740c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:15.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.434+0000 7fd273fff700 1 --2- 172.21.15.189:0/2011695917 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2740cfa00 0x7fd2741582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:15.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.434+0000 7fd2795b9700 1 -- 172.21.15.189:0/2011695917 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2740cb870 msgr2=0x7fd2740c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:15.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.434+0000 7fd2795b9700 1 --2- 172.21.15.189:0/2011695917 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2740cb870 0x7fd2740c8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.434+0000 7fd2795b9700 1 -- 172.21.15.189:0/2011695917 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2740cfa00 msgr2=0x7fd2741582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:15.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.434+0000 7fd2795b9700 1 --2- 172.21.15.189:0/2011695917 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2740cfa00 0x7fd2741582d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.434+0000 7fd2795b9700 1 -- 172.21.15.189:0/2011695917 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2600094a0 con 0x7fd2740d11b0 2024-04-06T12:13:15.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.434+0000 7fd2795b9700 1 --2- 172.21.15.189:0/2011695917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2740d11b0 0x7fd274158810 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00cb60 tx=0x7fd26c00ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:15.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.434+0000 7fd2717fa700 1 -- 172.21.15.189:0/2011695917 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c00a9e0 con 0x7fd2740d11b0 2024-04-06T12:13:15.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.435+0000 7fd2717fa700 1 -- 172.21.15.189:0/2011695917 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26c00ab40 con 0x7fd2740d11b0 2024-04-06T12:13:15.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.435+0000 7fd27b01c700 1 -- 172.21.15.189:0/2011695917 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2740cb270 con 0x7fd2740d11b0 2024-04-06T12:13:15.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.435+0000 7fd2717fa700 1 -- 172.21.15.189:0/2011695917 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c0070d0 con 0x7fd2740d11b0 2024-04-06T12:13:15.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.435+0000 7fd27b01c700 1 -- 172.21.15.189:0/2011695917 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2740c8cd0 con 0x7fd2740d11b0 2024-04-06T12:13:15.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.436+0000 7fd27b01c700 1 -- 172.21.15.189:0/2011695917 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd274074990 con 0x7fd2740d11b0 2024-04-06T12:13:15.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.436+0000 7fd2717fa700 1 -- 172.21.15.189:0/2011695917 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd26c007230 con 0x7fd2740d11b0 2024-04-06T12:13:15.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.437+0000 7fd2717fa700 1 --2- 172.21.15.189:0/2011695917 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd25806ac90 0x7fd25806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:15.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.437+0000 7fd2717fa700 1 -- 172.21.15.189:0/2011695917 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fd26c088d80 con 0x7fd2740d11b0 2024-04-06T12:13:15.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.437+0000 7fd278db8700 1 --2- 172.21.15.189:0/2011695917 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd25806ac90 0x7fd25806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:15.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.437+0000 7fd278db8700 1 --2- 172.21.15.189:0/2011695917 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd25806ac90 0x7fd25806d150 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd268009be0 tx=0x7fd268009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:15.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.441+0000 7fd2717fa700 1 -- 172.21.15.189:0/2011695917 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd26c054200 con 0x7fd2740d11b0 2024-04-06T12:13:15.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.718+0000 7fd27b01c700 1 -- 172.21.15.189:0/2011695917 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fd2740d1e20 con 0x7fd2740d11b0 2024-04-06T12:13:15.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.721+0000 7fd2717fa700 1 -- 172.21.15.189:0/2011695917 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7fd26c0180a0 con 0x7fd2740d11b0 2024-04-06T12:13:15.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.723+0000 7fd27b01c700 1 -- 172.21.15.189:0/2011695917 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd25806ac90 msgr2=0x7fd25806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:15.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.723+0000 7fd27b01c700 1 --2- 172.21.15.189:0/2011695917 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd25806ac90 0x7fd25806d150 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd268009be0 tx=0x7fd268009210 comp rx=0 tx=0).stop 2024-04-06T12:13:15.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.723+0000 7fd27b01c700 1 -- 172.21.15.189:0/2011695917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2740d11b0 msgr2=0x7fd274158810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:15.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.723+0000 7fd27b01c700 1 --2- 172.21.15.189:0/2011695917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2740d11b0 0x7fd274158810 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00cb60 tx=0x7fd26c00ce70 comp rx=0 tx=0).stop 2024-04-06T12:13:15.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.723+0000 7fd27b01c700 1 -- 172.21.15.189:0/2011695917 shutdown_connections 2024-04-06T12:13:15.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.723+0000 7fd27b01c700 1 --2- 172.21.15.189:0/2011695917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2740d11b0 0x7fd274158810 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.723+0000 7fd27b01c700 1 --2- 172.21.15.189:0/2011695917 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2740cb870 0x7fd2740c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.723+0000 7fd27b01c700 1 --2- 172.21.15.189:0/2011695917 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd25806ac90 0x7fd25806d150 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.723+0000 7fd27b01c700 1 --2- 172.21.15.189:0/2011695917 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2740cfa00 0x7fd2741582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.723+0000 7fd27b01c700 1 -- 172.21.15.189:0/2011695917 >> 172.21.15.189:0/2011695917 conn(0x7fd2740b2db0 msgr2=0x7fd2740cd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:15.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.724+0000 7fd27b01c700 1 -- 172.21.15.189:0/2011695917 shutdown_connections 2024-04-06T12:13:15.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.724+0000 7fd27b01c700 1 -- 172.21.15.189:0/2011695917 wait complete. 2024-04-06T12:13:15.735 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-06T12:13:15.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.859+0000 7fb119bb4700 1 Processor -- start 2024-04-06T12:13:15.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.859+0000 7fb119bb4700 1 -- start start 2024-04-06T12:13:15.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.860+0000 7fb119bb4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb11415a550 0x7fb1140cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:15.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.860+0000 7fb119bb4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb1140b4d70 0x7fb1140b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:15.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.860+0000 7fb119bb4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1140cdbe0 con 0x7fb1140cf900 2024-04-06T12:13:15.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.860+0000 7fb119bb4700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb11415a940 con 0x7fb1140b4d70 2024-04-06T12:13:15.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.860+0000 7fb119bb4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb11415ac00 con 0x7fb11415a550 2024-04-06T12:13:15.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.860+0000 7fb113fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb1140b4d70 0x7fb1140b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:15.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.860+0000 7fb112ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb1140cf900 0x7fb1140b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40040/0 (socket says 172.21.15.189:40040) 2024-04-06T12:13:15.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.860+0000 7fb113fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb1140b4d70 0x7fb1140b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57318/0 (socket says 172.21.15.189:57318) 2024-04-06T12:13:15.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.860+0000 7fb112ffd700 1 -- 172.21.15.189:0/3062774404 learned_addr learned my addr 172.21.15.189:0/3062774404 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:15.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.860+0000 7fb1137fe700 1 --2- 172.21.15.189:0/3062774404 >> v2:172.21.15.192:3300/0 conn(0x7fb11415a550 0x7fb1140cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:15.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.860+0000 7fb113fff700 1 -- 172.21.15.189:0/3062774404 >> v2:172.21.15.192:3300/0 conn(0x7fb11415a550 msgr2=0x7fb1140cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:15.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.860+0000 7fb113fff700 1 --2- 172.21.15.189:0/3062774404 >> v2:172.21.15.192:3300/0 conn(0x7fb11415a550 0x7fb1140cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.860+0000 7fb113fff700 1 -- 172.21.15.189:0/3062774404 >> v1:172.21.15.189:6789/0 conn(0x7fb1140cf900 legacy=0x7fb1140b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:15.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.860+0000 7fb113fff700 1 -- 172.21.15.189:0/3062774404 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11415aee0 con 0x7fb1140b4d70 2024-04-06T12:13:15.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.861+0000 7fb1137fe700 1 --2- 172.21.15.189:0/3062774404 >> v2:172.21.15.192:3300/0 conn(0x7fb11415a550 0x7fb1140cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:15.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.861+0000 7fb113fff700 1 --2- 172.21.15.189:0/3062774404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1140b4d70 0x7fb1140b2fc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fb114073470 tx=0x7fb10c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf09e04946c55f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:15.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.861+0000 7fb1127fc700 1 -- 172.21.15.189:0/3062774404 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c00e070 con 0x7fb1140b4d70 2024-04-06T12:13:15.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.861+0000 7fb1127fc700 1 -- 172.21.15.189:0/3062774404 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10c00bbd0 con 0x7fb1140b4d70 2024-04-06T12:13:15.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.861+0000 7fb1127fc700 1 -- 172.21.15.189:0/3062774404 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c012430 con 0x7fb1140b4d70 2024-04-06T12:13:15.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.861+0000 7fb119bb4700 1 -- 172.21.15.189:0/3062774404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1140b4d70 msgr2=0x7fb1140b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:15.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.861+0000 7fb119bb4700 1 --2- 172.21.15.189:0/3062774404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1140b4d70 0x7fb1140b2fc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fb114073470 tx=0x7fb10c00b330 comp rx=0 tx=0).stop 2024-04-06T12:13:15.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.861+0000 7fb119bb4700 1 -- 172.21.15.189:0/3062774404 shutdown_connections 2024-04-06T12:13:15.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.861+0000 7fb119bb4700 1 --2- 172.21.15.189:0/3062774404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1140b4d70 0x7fb1140b2fc0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.861+0000 7fb119bb4700 1 --2- 172.21.15.189:0/3062774404 >> v2:172.21.15.192:3300/0 conn(0x7fb11415a550 0x7fb1140cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.861+0000 7fb119bb4700 1 -- 172.21.15.189:0/3062774404 >> 172.21.15.189:0/3062774404 conn(0x7fb1140d00b0 msgr2=0x7fb1140d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:15.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.861+0000 7fb119bb4700 1 -- 172.21.15.189:0/3062774404 shutdown_connections 2024-04-06T12:13:15.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.861+0000 7fb119bb4700 1 -- 172.21.15.189:0/3062774404 wait complete. 2024-04-06T12:13:15.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.863+0000 7fb119bb4700 1 Processor -- start 2024-04-06T12:13:15.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.863+0000 7fb119bb4700 1 -- start start 2024-04-06T12:13:15.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.863+0000 7fb119bb4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb1140b4d70 0x7fb114153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:15.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.863+0000 7fb119bb4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1140cf900 0x7fb114156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:15.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.863+0000 7fb119bb4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb114157120 0x7fb11406d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:15.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.863+0000 7fb119bb4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb114153f70 con 0x7fb1140cf900 2024-04-06T12:13:15.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.863+0000 7fb119bb4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb114155b40 con 0x7fb1140b4d70 2024-04-06T12:13:15.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.863+0000 7fb119bb4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb114155e20 con 0x7fb114157120 2024-04-06T12:13:15.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.864+0000 7fb112ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1140cf900 0x7fb114156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:15.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.864+0000 7fb112ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1140cf900 0x7fb114156be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57334/0 (socket says 172.21.15.189:57334) 2024-04-06T12:13:15.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.864+0000 7fb112ffd700 1 -- 172.21.15.189:0/2840475723 learned_addr learned my addr 172.21.15.189:0/2840475723 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:15.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.864+0000 7fb1137fe700 1 --2- 172.21.15.189:0/2840475723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb1140b4d70 0x7fb114153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:15.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.864+0000 7fb113fff700 1 --2- 172.21.15.189:0/2840475723 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb114157120 0x7fb11406d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:15.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.864+0000 7fb112ffd700 1 -- 172.21.15.189:0/2840475723 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb114157120 msgr2=0x7fb11406d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:15.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.864+0000 7fb112ffd700 1 --2- 172.21.15.189:0/2840475723 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb114157120 0x7fb11406d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.864+0000 7fb112ffd700 1 -- 172.21.15.189:0/2840475723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb1140b4d70 msgr2=0x7fb114153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:15.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.864+0000 7fb112ffd700 1 --2- 172.21.15.189:0/2840475723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb1140b4d70 0x7fb114153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:15.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.864+0000 7fb112ffd700 1 -- 172.21.15.189:0/2840475723 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb10c009a70 con 0x7fb1140cf900 2024-04-06T12:13:15.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.864+0000 7fb1137fe700 1 --2- 172.21.15.189:0/2840475723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb1140b4d70 0x7fb114153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:15.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.864+0000 7fb112ffd700 1 --2- 172.21.15.189:0/2840475723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1140cf900 0x7fb114156be0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb10400c520 tx=0x7fb10400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:15.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.864+0000 7fb10bfff700 1 -- 172.21.15.189:0/2840475723 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb104005d70 con 0x7fb1140cf900 2024-04-06T12:13:15.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.864+0000 7fb10bfff700 1 -- 172.21.15.189:0/2840475723 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1040070d0 con 0x7fb1140cf900 2024-04-06T12:13:15.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.864+0000 7fb119bb4700 1 -- 172.21.15.189:0/2840475723 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb114156100 con 0x7fb1140cf900 2024-04-06T12:13:15.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.864+0000 7fb10bfff700 1 -- 172.21.15.189:0/2840475723 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb104007440 con 0x7fb1140cf900 2024-04-06T12:13:15.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.864+0000 7fb119bb4700 1 -- 172.21.15.189:0/2840475723 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1141566c0 con 0x7fb1140cf900 2024-04-06T12:13:15.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.865+0000 7fb119bb4700 1 -- 172.21.15.189:0/2840475723 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb114067bb0 con 0x7fb1140cf900 2024-04-06T12:13:15.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.866+0000 7fb10bfff700 1 -- 172.21.15.189:0/2840475723 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb1040075a0 con 0x7fb1140cf900 2024-04-06T12:13:15.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.866+0000 7fb10bfff700 1 --2- 172.21.15.189:0/2840475723 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb0f406ac90 0x7fb0f406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:15.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.866+0000 7fb10bfff700 1 -- 172.21.15.189:0/2840475723 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fb1040885a0 con 0x7fb1140cf900 2024-04-06T12:13:15.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.866+0000 7fb1137fe700 1 --2- 172.21.15.189:0/2840475723 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb0f406ac90 0x7fb0f406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:15.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.866+0000 7fb1137fe700 1 --2- 172.21.15.189:0/2840475723 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb0f406ac90 0x7fb0f406d150 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb114153000 tx=0x7fb0fc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:15.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:15.870+0000 7fb10bfff700 1 -- 172.21.15.189:0/2840475723 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb1040539c0 con 0x7fb1140cf900 2024-04-06T12:13:16.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.147+0000 7fb119bb4700 1 -- 172.21.15.189:0/2840475723 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fb114074990 con 0x7fb1140cf900 2024-04-06T12:13:16.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.151+0000 7fb10bfff700 1 -- 172.21.15.189:0/2840475723 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7fb104056fe0 con 0x7fb1140cf900 2024-04-06T12:13:16.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.153+0000 7fb119bb4700 1 -- 172.21.15.189:0/2840475723 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb0f406ac90 msgr2=0x7fb0f406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:16.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.153+0000 7fb119bb4700 1 --2- 172.21.15.189:0/2840475723 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb0f406ac90 0x7fb0f406d150 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb114153000 tx=0x7fb0fc008040 comp rx=0 tx=0).stop 2024-04-06T12:13:16.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.153+0000 7fb119bb4700 1 -- 172.21.15.189:0/2840475723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1140cf900 msgr2=0x7fb114156be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:16.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.153+0000 7fb119bb4700 1 --2- 172.21.15.189:0/2840475723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1140cf900 0x7fb114156be0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb10400c520 tx=0x7fb10400c8e0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.153+0000 7fb119bb4700 1 -- 172.21.15.189:0/2840475723 shutdown_connections 2024-04-06T12:13:16.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.153+0000 7fb119bb4700 1 --2- 172.21.15.189:0/2840475723 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb114157120 0x7fb11406d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.153+0000 7fb119bb4700 1 --2- 172.21.15.189:0/2840475723 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb0f406ac90 0x7fb0f406d150 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.153+0000 7fb119bb4700 1 --2- 172.21.15.189:0/2840475723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb1140b4d70 0x7fb114153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.153+0000 7fb119bb4700 1 --2- 172.21.15.189:0/2840475723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1140cf900 0x7fb114156be0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.153+0000 7fb119bb4700 1 -- 172.21.15.189:0/2840475723 >> 172.21.15.189:0/2840475723 conn(0x7fb1140d00b0 msgr2=0x7fb1140b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:16.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.154+0000 7fb119bb4700 1 -- 172.21.15.189:0/2840475723 shutdown_connections 2024-04-06T12:13:16.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.154+0000 7fb119bb4700 1 -- 172.21.15.189:0/2840475723 wait complete. 2024-04-06T12:13:16.165 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-06T12:13:16.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.289+0000 7fdd26d40700 1 Processor -- start 2024-04-06T12:13:16.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.289+0000 7fdd26d40700 1 -- start start 2024-04-06T12:13:16.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.290+0000 7fdd26d40700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd200d11b0 0x7fdd200d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:16.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.290+0000 7fdd26d40700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdd200cb870 0x7fdd2015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:16.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.290+0000 7fdd26d40700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdd200cfa00 0x7fdd200cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:16.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.290+0000 7fdd26d40700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd200ccdc0 con 0x7fdd200cb870 2024-04-06T12:13:16.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.290+0000 7fdd26d40700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd200d0b50 con 0x7fdd200cfa00 2024-04-06T12:13:16.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.290+0000 7fdd26d40700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd200d1ad0 con 0x7fdd200d11b0 2024-04-06T12:13:16.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.290+0000 7fdd1ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdd200cb870 0x7fdd2015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:16.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.290+0000 7fdd1ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdd200cb870 0x7fdd2015c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57350/0 (socket says 172.21.15.189:57350) 2024-04-06T12:13:16.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.290+0000 7fdd1ffff700 1 -- 172.21.15.189:0/3296929022 learned_addr learned my addr 172.21.15.189:0/3296929022 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:16.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.290+0000 7fdd24adc700 1 --2- 172.21.15.189:0/3296929022 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd200d11b0 0x7fdd200d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:16.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.290+0000 7fdd252dd700 1 --2- 172.21.15.189:0/3296929022 >> v2:172.21.15.192:3300/0 conn(0x7fdd200cfa00 0x7fdd200cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:16.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.290+0000 7fdd1ffff700 1 -- 172.21.15.189:0/3296929022 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd200d11b0 msgr2=0x7fdd200d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:16.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.290+0000 7fdd1ffff700 1 --2- 172.21.15.189:0/3296929022 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd200d11b0 0x7fdd200d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.290+0000 7fdd1ffff700 1 -- 172.21.15.189:0/3296929022 >> v2:172.21.15.192:3300/0 conn(0x7fdd200cfa00 msgr2=0x7fdd200cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:16.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.290+0000 7fdd1ffff700 1 --2- 172.21.15.189:0/3296929022 >> v2:172.21.15.192:3300/0 conn(0x7fdd200cfa00 0x7fdd200cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.290+0000 7fdd1ffff700 1 -- 172.21.15.189:0/3296929022 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd200cbd80 con 0x7fdd200cb870 2024-04-06T12:13:16.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.290+0000 7fdd1ffff700 1 --2- 172.21.15.189:0/3296929022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd200cb870 0x7fdd2015c280 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fdd200b0990 tx=0x7fdd0c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ec680c43979507 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:16.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.291+0000 7fdd1f7fe700 1 -- 172.21.15.189:0/3296929022 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c011830 con 0x7fdd200cb870 2024-04-06T12:13:16.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.291+0000 7fdd1f7fe700 1 -- 172.21.15.189:0/3296929022 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0c011990 con 0x7fdd200cb870 2024-04-06T12:13:16.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.291+0000 7fdd1f7fe700 1 -- 172.21.15.189:0/3296929022 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c011d20 con 0x7fdd200cb870 2024-04-06T12:13:16.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.291+0000 7fdd26d40700 1 -- 172.21.15.189:0/3296929022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd200cb870 msgr2=0x7fdd2015c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:16.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.291+0000 7fdd26d40700 1 --2- 172.21.15.189:0/3296929022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd200cb870 0x7fdd2015c280 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fdd200b0990 tx=0x7fdd0c009bd0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.291+0000 7fdd26d40700 1 -- 172.21.15.189:0/3296929022 shutdown_connections 2024-04-06T12:13:16.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.291+0000 7fdd26d40700 1 --2- 172.21.15.189:0/3296929022 >> v2:172.21.15.192:3300/0 conn(0x7fdd200cfa00 0x7fdd200cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.291+0000 7fdd26d40700 1 --2- 172.21.15.189:0/3296929022 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd200d11b0 0x7fdd200d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.291+0000 7fdd26d40700 1 --2- 172.21.15.189:0/3296929022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd200cb870 0x7fdd2015c280 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.291+0000 7fdd26d40700 1 -- 172.21.15.189:0/3296929022 >> 172.21.15.189:0/3296929022 conn(0x7fdd200b2db0 msgr2=0x7fdd200b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:16.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.291+0000 7fdd26d40700 1 -- 172.21.15.189:0/3296929022 shutdown_connections 2024-04-06T12:13:16.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.291+0000 7fdd26d40700 1 -- 172.21.15.189:0/3296929022 wait complete. 2024-04-06T12:13:16.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.292+0000 7fdd26d40700 1 Processor -- start 2024-04-06T12:13:16.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.292+0000 7fdd26d40700 1 -- start start 2024-04-06T12:13:16.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.293+0000 7fdd26d40700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd200cb870 0x7fdd200c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:16.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.293+0000 7fdd26d40700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd200cfa00 0x7fdd2015d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:16.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.293+0000 7fdd26d40700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd200d11b0 0x7fdd2014f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:16.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.293+0000 7fdd26d40700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2015df30 con 0x7fdd200d11b0 2024-04-06T12:13:16.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.293+0000 7fdd26d40700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2014fd70 con 0x7fdd200cfa00 2024-04-06T12:13:16.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.293+0000 7fdd26d40700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2015c280 con 0x7fdd200cb870 2024-04-06T12:13:16.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.293+0000 7fdd252dd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd200d11b0 0x7fdd2014f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:16.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.293+0000 7fdd252dd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd200d11b0 0x7fdd2014f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57360/0 (socket says 172.21.15.189:57360) 2024-04-06T12:13:16.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.293+0000 7fdd252dd700 1 -- 172.21.15.189:0/738715659 learned_addr learned my addr 172.21.15.189:0/738715659 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:16.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.293+0000 7fdd24adc700 1 --2- 172.21.15.189:0/738715659 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd200cb870 0x7fdd200c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:16.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.293+0000 7fdd1ffff700 1 --2- 172.21.15.189:0/738715659 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd200cfa00 0x7fdd2015d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:16.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.294+0000 7fdd252dd700 1 -- 172.21.15.189:0/738715659 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd200cb870 msgr2=0x7fdd200c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:16.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.294+0000 7fdd252dd700 1 --2- 172.21.15.189:0/738715659 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd200cb870 0x7fdd200c82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.294+0000 7fdd252dd700 1 -- 172.21.15.189:0/738715659 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd200cfa00 msgr2=0x7fdd2015d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:16.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.294+0000 7fdd252dd700 1 --2- 172.21.15.189:0/738715659 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd200cfa00 0x7fdd2015d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.294+0000 7fdd252dd700 1 -- 172.21.15.189:0/738715659 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd0c009550 con 0x7fdd200d11b0 2024-04-06T12:13:16.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.294+0000 7fdd1ffff700 1 --2- 172.21.15.189:0/738715659 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd200cfa00 0x7fdd2015d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:16.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.294+0000 7fdd252dd700 1 --2- 172.21.15.189:0/738715659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd200d11b0 0x7fdd2014f830 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800eed0 tx=0x7fdd1800b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:16.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.294+0000 7fdd1d7fa700 1 -- 172.21.15.189:0/738715659 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18010040 con 0x7fdd200d11b0 2024-04-06T12:13:16.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.294+0000 7fdd1d7fa700 1 -- 172.21.15.189:0/738715659 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1800bd30 con 0x7fdd200d11b0 2024-04-06T12:13:16.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.294+0000 7fdd1d7fa700 1 -- 172.21.15.189:0/738715659 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1800bea0 con 0x7fdd200d11b0 2024-04-06T12:13:16.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.294+0000 7fdd26d40700 1 -- 172.21.15.189:0/738715659 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd200c89d0 con 0x7fdd200d11b0 2024-04-06T12:13:16.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.294+0000 7fdd26d40700 1 -- 172.21.15.189:0/738715659 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2015c820 con 0x7fdd200d11b0 2024-04-06T12:13:16.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.295+0000 7fdd26d40700 1 -- 172.21.15.189:0/738715659 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd20074990 con 0x7fdd200d11b0 2024-04-06T12:13:16.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.295+0000 7fdd1d7fa700 1 -- 172.21.15.189:0/738715659 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdd18014620 con 0x7fdd200d11b0 2024-04-06T12:13:16.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.296+0000 7fdd1d7fa700 1 --2- 172.21.15.189:0/738715659 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdd0406ac90 0x7fdd0406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:16.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.296+0000 7fdd1d7fa700 1 -- 172.21.15.189:0/738715659 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fdd18089530 con 0x7fdd200d11b0 2024-04-06T12:13:16.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.296+0000 7fdd24adc700 1 --2- 172.21.15.189:0/738715659 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdd0406ac90 0x7fdd0406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:16.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.296+0000 7fdd24adc700 1 --2- 172.21.15.189:0/738715659 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdd0406ac90 0x7fdd0406d150 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdd14009be0 tx=0x7fdd14009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:16.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.300+0000 7fdd1d7fa700 1 -- 172.21.15.189:0/738715659 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdd180549b0 con 0x7fdd200d11b0 2024-04-06T12:13:16.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.573+0000 7fdd26d40700 1 -- 172.21.15.189:0/738715659 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fdd200d1e20 con 0x7fdd200d11b0 2024-04-06T12:13:16.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.578+0000 7fdd1d7fa700 1 -- 172.21.15.189:0/738715659 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fdd180180a0 con 0x7fdd200d11b0 2024-04-06T12:13:16.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.580+0000 7fdd26d40700 1 -- 172.21.15.189:0/738715659 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdd0406ac90 msgr2=0x7fdd0406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:16.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.580+0000 7fdd26d40700 1 --2- 172.21.15.189:0/738715659 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdd0406ac90 0x7fdd0406d150 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdd14009be0 tx=0x7fdd14009210 comp rx=0 tx=0).stop 2024-04-06T12:13:16.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.580+0000 7fdd26d40700 1 -- 172.21.15.189:0/738715659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd200d11b0 msgr2=0x7fdd2014f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:16.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.580+0000 7fdd26d40700 1 --2- 172.21.15.189:0/738715659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd200d11b0 0x7fdd2014f830 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800eed0 tx=0x7fdd1800b390 comp rx=0 tx=0).stop 2024-04-06T12:13:16.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.580+0000 7fdd26d40700 1 -- 172.21.15.189:0/738715659 shutdown_connections 2024-04-06T12:13:16.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.580+0000 7fdd26d40700 1 --2- 172.21.15.189:0/738715659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd200d11b0 0x7fdd2014f830 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.580+0000 7fdd26d40700 1 --2- 172.21.15.189:0/738715659 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd200cb870 0x7fdd200c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.580+0000 7fdd26d40700 1 --2- 172.21.15.189:0/738715659 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdd0406ac90 0x7fdd0406d150 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.580+0000 7fdd26d40700 1 --2- 172.21.15.189:0/738715659 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd200cfa00 0x7fdd2015d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.580+0000 7fdd26d40700 1 -- 172.21.15.189:0/738715659 >> 172.21.15.189:0/738715659 conn(0x7fdd200b2db0 msgr2=0x7fdd200b3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:16.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.580+0000 7fdd26d40700 1 -- 172.21.15.189:0/738715659 shutdown_connections 2024-04-06T12:13:16.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.581+0000 7fdd26d40700 1 -- 172.21.15.189:0/738715659 wait complete. 2024-04-06T12:13:16.592 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-06T12:13:16.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.717+0000 7ff0a35e5700 1 Processor -- start 2024-04-06T12:13:16.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.717+0000 7ff0a35e5700 1 -- start start 2024-04-06T12:13:16.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff0a35e5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff09c15d7b0 0x7ff09c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:16.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff0a35e5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff09c0b3990 con 0x7ff09c06a730 2024-04-06T12:13:16.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff0a35e5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff09c0b3060 con 0x7ff09c15e500 2024-04-06T12:13:16.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff0a35e5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c0b3340 con 0x7ff09c15d7b0 2024-04-06T12:13:16.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff0a0b80700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff09c06a730 0x7ff09c15ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40052/0 (socket says 172.21.15.189:40052) 2024-04-06T12:13:16.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff0a0b80700 1 -- 172.21.15.189:0/1728488505 learned_addr learned my addr 172.21.15.189:0/1728488505 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:16.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff0a1381700 1 --2- 172.21.15.189:0/1728488505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff09c15d7b0 0x7ff09c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:16.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff093fff700 1 -- 172.21.15.189:0/1728488505 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957527301 0 0) 0x7ff09c0b3990 con 0x7ff09c06a730 2024-04-06T12:13:16.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff093fff700 1 -- 172.21.15.189:0/1728488505 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0880035b0 con 0x7ff09c06a730 2024-04-06T12:13:16.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff093fff700 1 -- 172.21.15.189:0/1728488505 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2424541959 0 0) 0x7ff0880035b0 con 0x7ff09c06a730 2024-04-06T12:13:16.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff093fff700 1 -- 172.21.15.189:0/1728488505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff09c15d7b0 msgr2=0x7ff09c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:16.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff093fff700 1 --2- 172.21.15.189:0/1728488505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff09c15d7b0 0x7ff09c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff093fff700 1 -- 172.21.15.189:0/1728488505 >> v1:172.21.15.192:6789/0 conn(0x7ff09c15e500 legacy=0x7ff09c0717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:16.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff093fff700 1 -- 172.21.15.189:0/1728488505 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09c0b3620 con 0x7ff09c06a730 2024-04-06T12:13:16.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff093fff700 1 -- 172.21.15.189:0/1728488505 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7ff08c001670 con 0x7ff09c06a730 2024-04-06T12:13:16.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff093fff700 1 -- 172.21.15.189:0/1728488505 >> v1:172.21.15.189:6789/0 conn(0x7ff09c06a730 legacy=0x7ff09c15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:16.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff093fff700 1 --2- 172.21.15.189:0/1728488505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff088003a30 0x7ff088003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff093fff700 1 --2- 172.21.15.189:0/1728488505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff0880043e0 0x7ff0880048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff093fff700 1 --2- 172.21.15.189:0/1728488505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff09c15d7b0 0x7ff0880051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff093fff700 1 -- 172.21.15.189:0/1728488505 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c0b3340 con 0x7ff088003a30 2024-04-06T12:13:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff093fff700 1 -- 172.21.15.189:0/1728488505 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08800bac0 con 0x7ff0880043e0 2024-04-06T12:13:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff093fff700 1 -- 172.21.15.189:0/1728488505 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08800bc80 con 0x7ff09c15d7b0 2024-04-06T12:13:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.718+0000 7ff0a1381700 1 --2- 172.21.15.189:0/1728488505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff088003a30 0x7ff088003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.719+0000 7ff0a0b80700 1 --2- 172.21.15.189:0/1728488505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff0880043e0 0x7ff0880048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.719+0000 7ff0a1b82700 1 --2- 172.21.15.189:0/1728488505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff09c15d7b0 0x7ff0880051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.719+0000 7ff0a1381700 1 -- 172.21.15.189:0/1728488505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff09c15d7b0 msgr2=0x7ff0880051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.719+0000 7ff0a1381700 1 --2- 172.21.15.189:0/1728488505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff09c15d7b0 0x7ff0880051b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.719+0000 7ff0a1381700 1 -- 172.21.15.189:0/1728488505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff0880043e0 msgr2=0x7ff0880048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.719+0000 7ff0a1381700 1 --2- 172.21.15.189:0/1728488505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff0880043e0 0x7ff0880048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.719+0000 7ff0a1381700 1 -- 172.21.15.189:0/1728488505 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0880035b0 con 0x7ff088003a30 2024-04-06T12:13:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.719+0000 7ff0a1381700 1 --2- 172.21.15.189:0/1728488505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff088003a30 0x7ff088003e10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff098009b60 tx=0x7ff088005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb1f0e199b50f7b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.719+0000 7ff093fff700 1 -- 172.21.15.189:0/1728488505 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09800e6c0 con 0x7ff088003a30 2024-04-06T12:13:16.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.719+0000 7ff093fff700 1 -- 172.21.15.189:0/1728488505 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09800e820 con 0x7ff088003a30 2024-04-06T12:13:16.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.719+0000 7ff093fff700 1 -- 172.21.15.189:0/1728488505 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09800eb90 con 0x7ff088003a30 2024-04-06T12:13:16.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.720+0000 7ff0a35e5700 1 -- 172.21.15.189:0/1728488505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff088003a30 msgr2=0x7ff088003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:16.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.720+0000 7ff0a35e5700 1 --2- 172.21.15.189:0/1728488505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff088003a30 0x7ff088003e10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff098009b60 tx=0x7ff088005880 comp rx=0 tx=0).stop 2024-04-06T12:13:16.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.720+0000 7ff0a1b82700 1 -- 172.21.15.189:0/1728488505 reap_dead start 2024-04-06T12:13:16.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.720+0000 7ff0a35e5700 1 -- 172.21.15.189:0/1728488505 shutdown_connections 2024-04-06T12:13:16.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.720+0000 7ff0a35e5700 1 -- 172.21.15.189:0/1728488505 >> 172.21.15.189:0/1728488505 conn(0x7ff09c0d00b0 msgr2=0x7ff09c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:16.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.720+0000 7ff0a35e5700 1 -- 172.21.15.189:0/1728488505 shutdown_connections 2024-04-06T12:13:16.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.720+0000 7ff0a35e5700 1 -- 172.21.15.189:0/1728488505 wait complete. 2024-04-06T12:13:16.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.721+0000 7ff0a35e5700 1 Processor -- start 2024-04-06T12:13:16.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.721+0000 7ff0a35e5700 1 -- start start 2024-04-06T12:13:16.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff0a35e5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff088003a30 0x7ff09c147260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:16.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff0a35e5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff09c15bbc0 0x7ff09c143da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:16.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff0a35e5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff09c159d50 0x7ff09c15a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:16.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff0a35e5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c15a740 con 0x7ff09c159d50 2024-04-06T12:13:16.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff0a35e5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c15cf00 con 0x7ff088003a30 2024-04-06T12:13:16.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff0a35e5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c15c040 con 0x7ff09c15bbc0 2024-04-06T12:13:16.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff0a1b82700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff09c159d50 0x7ff09c15a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:16.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff0a1381700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff088003a30 0x7ff09c147260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:16.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff0a1b82700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff09c159d50 0x7ff09c15a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57376/0 (socket says 172.21.15.189:57376) 2024-04-06T12:13:16.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff0a1b82700 1 -- 172.21.15.189:0/996690834 learned_addr learned my addr 172.21.15.189:0/996690834 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:16.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff0a0b80700 1 --2- 172.21.15.189:0/996690834 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff09c15bbc0 0x7ff09c143da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:16.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff0a1b82700 1 -- 172.21.15.189:0/996690834 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff09c15bbc0 msgr2=0x7ff09c143da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:16.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff0a1b82700 1 --2- 172.21.15.189:0/996690834 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff09c15bbc0 0x7ff09c143da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff0a1b82700 1 -- 172.21.15.189:0/996690834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff088003a30 msgr2=0x7ff09c147260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:16.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff0a1b82700 1 --2- 172.21.15.189:0/996690834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff088003a30 0x7ff09c147260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:16.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff0a1b82700 1 -- 172.21.15.189:0/996690834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09c15c2f0 con 0x7ff09c159d50 2024-04-06T12:13:16.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff0a1b82700 1 --2- 172.21.15.189:0/996690834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff09c159d50 0x7ff09c15a200 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff09c06d2c0 tx=0x7ff09400d660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:16.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff091ffb700 1 -- 172.21.15.189:0/996690834 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09400db10 con 0x7ff09c159d50 2024-04-06T12:13:16.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff091ffb700 1 -- 172.21.15.189:0/996690834 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09400dc70 con 0x7ff09c159d50 2024-04-06T12:13:16.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff0a35e5700 1 -- 172.21.15.189:0/996690834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff09c15c5d0 con 0x7ff09c159d50 2024-04-06T12:13:16.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.722+0000 7ff091ffb700 1 -- 172.21.15.189:0/996690834 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff094003a00 con 0x7ff09c159d50 2024-04-06T12:13:16.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.723+0000 7ff0a35e5700 1 -- 172.21.15.189:0/996690834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff09c15cb90 con 0x7ff09c159d50 2024-04-06T12:13:16.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.724+0000 7ff0a35e5700 1 -- 172.21.15.189:0/996690834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff09c15cd50 con 0x7ff09c159d50 2024-04-06T12:13:16.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.724+0000 7ff091ffb700 1 -- 172.21.15.189:0/996690834 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff094003b60 con 0x7ff09c159d50 2024-04-06T12:13:16.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.724+0000 7ff091ffb700 1 --2- 172.21.15.189:0/996690834 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff08006ac90 0x7ff08006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:16.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.724+0000 7ff091ffb700 1 -- 172.21.15.189:0/996690834 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7ff0940893b0 con 0x7ff09c159d50 2024-04-06T12:13:16.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.724+0000 7ff0a1381700 1 --2- 172.21.15.189:0/996690834 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff08006ac90 0x7ff08006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:16.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.725+0000 7ff0a1381700 1 --2- 172.21.15.189:0/996690834 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff08006ac90 0x7ff08006d150 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff0980090b0 tx=0x7ff098008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:16.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:16.729+0000 7ff091ffb700 1 -- 172.21.15.189:0/996690834 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff0940547a0 con 0x7ff09c159d50 2024-04-06T12:13:17.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.003+0000 7ff0a35e5700 1 -- 172.21.15.189:0/996690834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7ff09c067bb0 con 0x7ff09c159d50 2024-04-06T12:13:17.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.006+0000 7ff091ffb700 1 -- 172.21.15.189:0/996690834 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff094057dc0 con 0x7ff09c159d50 2024-04-06T12:13:17.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.009+0000 7ff0a35e5700 1 -- 172.21.15.189:0/996690834 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff08006ac90 msgr2=0x7ff08006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:17.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.009+0000 7ff0a35e5700 1 --2- 172.21.15.189:0/996690834 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff08006ac90 0x7ff08006d150 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff0980090b0 tx=0x7ff098008030 comp rx=0 tx=0).stop 2024-04-06T12:13:17.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.009+0000 7ff0a35e5700 1 -- 172.21.15.189:0/996690834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff09c159d50 msgr2=0x7ff09c15a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:17.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.009+0000 7ff0a35e5700 1 --2- 172.21.15.189:0/996690834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff09c159d50 0x7ff09c15a200 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff09c06d2c0 tx=0x7ff09400d660 comp rx=0 tx=0).stop 2024-04-06T12:13:17.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.009+0000 7ff0a35e5700 1 -- 172.21.15.189:0/996690834 shutdown_connections 2024-04-06T12:13:17.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.009+0000 7ff0a35e5700 1 --2- 172.21.15.189:0/996690834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff09c159d50 0x7ff09c15a200 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.009+0000 7ff0a35e5700 1 --2- 172.21.15.189:0/996690834 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff08006ac90 0x7ff08006d150 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.009+0000 7ff0a35e5700 1 --2- 172.21.15.189:0/996690834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff088003a30 0x7ff09c147260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.009+0000 7ff0a35e5700 1 --2- 172.21.15.189:0/996690834 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff09c15bbc0 0x7ff09c143da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.009+0000 7ff0a35e5700 1 -- 172.21.15.189:0/996690834 >> 172.21.15.189:0/996690834 conn(0x7ff09c0d00b0 msgr2=0x7ff09c0ad220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:17.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.009+0000 7ff0a35e5700 1 -- 172.21.15.189:0/996690834 shutdown_connections 2024-04-06T12:13:17.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.009+0000 7ff0a35e5700 1 -- 172.21.15.189:0/996690834 wait complete. 2024-04-06T12:13:17.020 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-06T12:13:17.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.143+0000 7f06f9872700 1 Processor -- start 2024-04-06T12:13:17.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.143+0000 7f06f9872700 1 -- start start 2024-04-06T12:13:17.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.144+0000 7f06f9872700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f06f4143470 0x7f06f40b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:17.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.144+0000 7f06f9872700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f415eb40 con 0x7f06f4160b60 2024-04-06T12:13:17.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.144+0000 7f06f9872700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f40d0bf0 con 0x7f06f415db10 2024-04-06T12:13:17.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.144+0000 7f06f9872700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f4160f40 con 0x7f06f4143470 2024-04-06T12:13:17.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.144+0000 7f06f2ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f06f4160b60 0x7f06f4142d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40066/0 (socket says 172.21.15.189:40066) 2024-04-06T12:13:17.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.144+0000 7f06f2ffd700 1 -- 172.21.15.189:0/189876931 learned_addr learned my addr 172.21.15.189:0/189876931 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:17.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.144+0000 7f06f1ffb700 1 -- 172.21.15.189:0/189876931 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611361534 0 0) 0x7f06e8001110 con 0x7f06f4160b60 2024-04-06T12:13:17.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.144+0000 7f06f37fe700 1 --2- 172.21.15.189:0/189876931 >> v2:172.21.15.192:3300/0 conn(0x7f06f4143470 0x7f06f40b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:17.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.144+0000 7f06f1ffb700 1 -- 172.21.15.189:0/189876931 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06dc0035b0 con 0x7f06f4160b60 2024-04-06T12:13:17.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.144+0000 7f06f1ffb700 1 -- 172.21.15.189:0/189876931 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1162389002 0 0) 0x7f06dc0035b0 con 0x7f06f4160b60 2024-04-06T12:13:17.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.144+0000 7f06f1ffb700 1 -- 172.21.15.189:0/189876931 >> v2:172.21.15.192:3300/0 conn(0x7f06f4143470 msgr2=0x7f06f40b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:17.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.144+0000 7f06f1ffb700 1 --2- 172.21.15.189:0/189876931 >> v2:172.21.15.192:3300/0 conn(0x7f06f4143470 0x7f06f40b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.144+0000 7f06f1ffb700 1 -- 172.21.15.189:0/189876931 >> v1:172.21.15.192:6789/0 conn(0x7f06f415db10 legacy=0x7f06f40cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:17.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.144+0000 7f06f1ffb700 1 -- 172.21.15.189:0/189876931 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f415dfc0 con 0x7f06f4160b60 2024-04-06T12:13:17.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.144+0000 7f06f1ffb700 1 -- 172.21.15.189:0/189876931 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f06e8001800 con 0x7f06f4160b60 2024-04-06T12:13:17.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.144+0000 7f06f1ffb700 1 -- 172.21.15.189:0/189876931 >> v1:172.21.15.189:6789/0 conn(0x7f06f4160b60 legacy=0x7f06f4142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:17.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.144+0000 7f06f1ffb700 1 --2- 172.21.15.189:0/189876931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06dc003a50 0x7f06dc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:17.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.145+0000 7f06f1ffb700 1 --2- 172.21.15.189:0/189876931 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06dc004850 0x7f06dc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:17.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.145+0000 7f06f1ffb700 1 --2- 172.21.15.189:0/189876931 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06dc005210 0x7f06dc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:17.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.145+0000 7f06f1ffb700 1 -- 172.21.15.189:0/189876931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f4160f40 con 0x7f06dc003a50 2024-04-06T12:13:17.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.145+0000 7f06f1ffb700 1 -- 172.21.15.189:0/189876931 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06dc00c110 con 0x7f06dc005210 2024-04-06T12:13:17.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.145+0000 7f06f1ffb700 1 -- 172.21.15.189:0/189876931 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06dc00c2d0 con 0x7f06dc004850 2024-04-06T12:13:17.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.145+0000 7f06f2ffd700 1 --2- 172.21.15.189:0/189876931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06dc003a50 0x7f06dc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:17.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.145+0000 7f06f37fe700 1 --2- 172.21.15.189:0/189876931 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06dc005210 0x7f06dc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:17.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.145+0000 7f06f27fc700 1 --2- 172.21.15.189:0/189876931 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06dc004850 0x7f06dc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:17.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.145+0000 7f06f2ffd700 1 -- 172.21.15.189:0/189876931 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06dc004850 msgr2=0x7f06dc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:17.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.145+0000 7f06f2ffd700 1 --2- 172.21.15.189:0/189876931 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06dc004850 0x7f06dc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.145+0000 7f06f2ffd700 1 -- 172.21.15.189:0/189876931 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06dc005210 msgr2=0x7f06dc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:17.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.145+0000 7f06f2ffd700 1 --2- 172.21.15.189:0/189876931 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06dc005210 0x7f06dc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.145+0000 7f06f2ffd700 1 -- 172.21.15.189:0/189876931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06dc0035b0 con 0x7f06dc003a50 2024-04-06T12:13:17.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.145+0000 7f06f37fe700 1 --2- 172.21.15.189:0/189876931 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06dc005210 0x7f06dc00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:17.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.145+0000 7f06f37fe700 1 -- 172.21.15.189:0/189876931 reap_dead start 2024-04-06T12:13:17.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.145+0000 7f06f2ffd700 1 --2- 172.21.15.189:0/189876931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06dc003a50 0x7f06dc003e70 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f06e8007fd0 tx=0x7f06dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=414faf9a27e5da1b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:17.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.145+0000 7f06f1ffb700 1 -- 172.21.15.189:0/189876931 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e800ccd0 con 0x7f06dc003a50 2024-04-06T12:13:17.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.145+0000 7f06f1ffb700 1 -- 172.21.15.189:0/189876931 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e800ce50 con 0x7f06dc003a50 2024-04-06T12:13:17.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.145+0000 7f06f1ffb700 1 -- 172.21.15.189:0/189876931 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e8005ba0 con 0x7f06dc003a50 2024-04-06T12:13:17.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.146+0000 7f06f9872700 1 -- 172.21.15.189:0/189876931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06dc003a50 msgr2=0x7f06dc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:17.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.146+0000 7f06f9872700 1 --2- 172.21.15.189:0/189876931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06dc003a50 0x7f06dc003e70 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f06e8007fd0 tx=0x7f06dc005880 comp rx=0 tx=0).stop 2024-04-06T12:13:17.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.146+0000 7f06f9872700 1 -- 172.21.15.189:0/189876931 shutdown_connections 2024-04-06T12:13:17.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.146+0000 7f06f9872700 1 --2- 172.21.15.189:0/189876931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06dc003a50 0x7f06dc003e70 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.146+0000 7f06f9872700 1 -- 172.21.15.189:0/189876931 >> 172.21.15.189:0/189876931 conn(0x7f06f406a730 msgr2=0x7f06f406ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:17.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.146+0000 7f06f9872700 1 -- 172.21.15.189:0/189876931 shutdown_connections 2024-04-06T12:13:17.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.146+0000 7f06f9872700 1 -- 172.21.15.189:0/189876931 wait complete. 2024-04-06T12:13:17.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.147+0000 7f06f9872700 1 Processor -- start 2024-04-06T12:13:17.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.147+0000 7f06f9872700 1 -- start start 2024-04-06T12:13:17.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.148+0000 7f06f9872700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06dc003a50 0x7f06f40caf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:17.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.148+0000 7f06f9872700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06f40b1b60 0x7f06f415c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:17.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.148+0000 7f06f9872700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06f40b5b00 0x7f06f40b5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:17.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.148+0000 7f06f9872700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f40b64f0 con 0x7f06dc003a50 2024-04-06T12:13:17.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.148+0000 7f06f9872700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f40b2340 con 0x7f06f40b1b60 2024-04-06T12:13:17.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.148+0000 7f06f9872700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f40b2620 con 0x7f06f40b5b00 2024-04-06T12:13:17.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.148+0000 7f06f2ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06dc003a50 0x7f06f40caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:17.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.148+0000 7f06f2ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06dc003a50 0x7f06f40caf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57380/0 (socket says 172.21.15.189:57380) 2024-04-06T12:13:17.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.148+0000 7f06f2ffd700 1 -- 172.21.15.189:0/1723978194 learned_addr learned my addr 172.21.15.189:0/1723978194 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:17.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.148+0000 7f06f27fc700 1 --2- 172.21.15.189:0/1723978194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06f40b1b60 0x7f06f415c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:17.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.148+0000 7f06f37fe700 1 --2- 172.21.15.189:0/1723978194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06f40b5b00 0x7f06f40b5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:17.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.148+0000 7f06f2ffd700 1 -- 172.21.15.189:0/1723978194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06f40b5b00 msgr2=0x7f06f40b5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:17.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.148+0000 7f06f2ffd700 1 --2- 172.21.15.189:0/1723978194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06f40b5b00 0x7f06f40b5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.148+0000 7f06f2ffd700 1 -- 172.21.15.189:0/1723978194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06f40b1b60 msgr2=0x7f06f415c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:17.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.148+0000 7f06f2ffd700 1 --2- 172.21.15.189:0/1723978194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06f40b1b60 0x7f06f415c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.148+0000 7f06f2ffd700 1 -- 172.21.15.189:0/1723978194 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f40c8490 con 0x7f06dc003a50 2024-04-06T12:13:17.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.148+0000 7f06f37fe700 1 --2- 172.21.15.189:0/1723978194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06f40b5b00 0x7f06f40b5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:17.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.148+0000 7f06f2ffd700 1 --2- 172.21.15.189:0/1723978194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06dc003a50 0x7f06f40caf40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00c0e0 tx=0x7f06e8001ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:17.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.148+0000 7f06dbfff700 1 -- 172.21.15.189:0/1723978194 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e8004c80 con 0x7f06dc003a50 2024-04-06T12:13:17.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.149+0000 7f06dbfff700 1 -- 172.21.15.189:0/1723978194 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e8004e00 con 0x7f06dc003a50 2024-04-06T12:13:17.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.149+0000 7f06f9872700 1 -- 172.21.15.189:0/1723978194 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f40c8770 con 0x7f06dc003a50 2024-04-06T12:13:17.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.149+0000 7f06dbfff700 1 -- 172.21.15.189:0/1723978194 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e8007310 con 0x7f06dc003a50 2024-04-06T12:13:17.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.149+0000 7f06f9872700 1 -- 172.21.15.189:0/1723978194 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f415c8e0 con 0x7f06dc003a50 2024-04-06T12:13:17.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.149+0000 7f06f9872700 1 -- 172.21.15.189:0/1723978194 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f40c82d0 con 0x7f06dc003a50 2024-04-06T12:13:17.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.150+0000 7f06dbfff700 1 -- 172.21.15.189:0/1723978194 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f06e8007470 con 0x7f06dc003a50 2024-04-06T12:13:17.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.150+0000 7f06dbfff700 1 --2- 172.21.15.189:0/1723978194 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06e006ac90 0x7f06e006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:17.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.150+0000 7f06dbfff700 1 -- 172.21.15.189:0/1723978194 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f06e8089940 con 0x7f06dc003a50 2024-04-06T12:13:17.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.150+0000 7f06f27fc700 1 --2- 172.21.15.189:0/1723978194 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06e006ac90 0x7f06e006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:17.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.151+0000 7f06f27fc700 1 --2- 172.21.15.189:0/1723978194 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06e006ac90 0x7f06e006d150 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f06f4161d10 tx=0x7f06e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:17.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.155+0000 7f06dbfff700 1 -- 172.21.15.189:0/1723978194 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f06e8054d40 con 0x7f06dc003a50 2024-04-06T12:13:17.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.430+0000 7f06f9872700 1 -- 172.21.15.189:0/1723978194 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f06f415eb40 con 0x7f06dc003a50 2024-04-06T12:13:17.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.433+0000 7f06dbfff700 1 -- 172.21.15.189:0/1723978194 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f06f415eb40 con 0x7f06dc003a50 2024-04-06T12:13:17.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.436+0000 7f06f9872700 1 -- 172.21.15.189:0/1723978194 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06e006ac90 msgr2=0x7f06e006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:17.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.436+0000 7f06f9872700 1 --2- 172.21.15.189:0/1723978194 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06e006ac90 0x7f06e006d150 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f06f4161d10 tx=0x7f06e4009040 comp rx=0 tx=0).stop 2024-04-06T12:13:17.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.436+0000 7f06f9872700 1 -- 172.21.15.189:0/1723978194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06dc003a50 msgr2=0x7f06f40caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:17.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.436+0000 7f06f9872700 1 --2- 172.21.15.189:0/1723978194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06dc003a50 0x7f06f40caf40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00c0e0 tx=0x7f06e8001ac0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.436+0000 7f06f9872700 1 -- 172.21.15.189:0/1723978194 shutdown_connections 2024-04-06T12:13:17.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.436+0000 7f06f9872700 1 --2- 172.21.15.189:0/1723978194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06f40b5b00 0x7f06f40b5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.436+0000 7f06f9872700 1 --2- 172.21.15.189:0/1723978194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06dc003a50 0x7f06f40caf40 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.436+0000 7f06f9872700 1 --2- 172.21.15.189:0/1723978194 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06e006ac90 0x7f06e006d150 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.436+0000 7f06f9872700 1 --2- 172.21.15.189:0/1723978194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06f40b1b60 0x7f06f415c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.436+0000 7f06f9872700 1 -- 172.21.15.189:0/1723978194 >> 172.21.15.189:0/1723978194 conn(0x7f06f406a730 msgr2=0x7f06f415f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:17.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.436+0000 7f06f9872700 1 -- 172.21.15.189:0/1723978194 shutdown_connections 2024-04-06T12:13:17.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.436+0000 7f06f9872700 1 -- 172.21.15.189:0/1723978194 wait complete. 2024-04-06T12:13:17.447 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-06T12:13:17.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.571+0000 7fbf4a65d700 1 Processor -- start 2024-04-06T12:13:17.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.571+0000 7fbf4a65d700 1 -- start start 2024-04-06T12:13:17.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.572+0000 7fbf4a65d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbf4415d7e0 0x7fbf440d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:17.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.572+0000 7fbf4a65d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf440d1cc0 0x7fbf440b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:17.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.572+0000 7fbf4a65d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf440cdf80 con 0x7fbf440d0890 2024-04-06T12:13:17.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.572+0000 7fbf4a65d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf440d3230 con 0x7fbf4415d7e0 2024-04-06T12:13:17.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.572+0000 7fbf4a65d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf440d2da0 con 0x7fbf440d1cc0 2024-04-06T12:13:17.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.572+0000 7fbf43fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbf4415d7e0 0x7fbf440d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:17.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.572+0000 7fbf437fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbf440d0890 0x7fbf440d15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40068/0 (socket says 172.21.15.189:40068) 2024-04-06T12:13:17.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.572+0000 7fbf437fe700 1 -- 172.21.15.189:0/193821922 learned_addr learned my addr 172.21.15.189:0/193821922 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:17.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.572+0000 7fbf48bfa700 1 --2- 172.21.15.189:0/193821922 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf440d1cc0 0x7fbf440b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:17.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.573+0000 7fbf43fff700 1 -- 172.21.15.189:0/193821922 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf440d1cc0 msgr2=0x7fbf440b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:17.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.573+0000 7fbf43fff700 1 --2- 172.21.15.189:0/193821922 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf440d1cc0 0x7fbf440b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.573+0000 7fbf43fff700 1 -- 172.21.15.189:0/193821922 >> v1:172.21.15.189:6789/0 conn(0x7fbf440d0890 legacy=0x7fbf440d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:17.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.573+0000 7fbf42ffd700 1 -- 172.21.15.189:0/193821922 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076405946 0 0) 0x7fbf440cdf80 con 0x7fbf440d0890 2024-04-06T12:13:17.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.573+0000 7fbf43fff700 1 -- 172.21.15.189:0/193821922 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf440d0cf0 con 0x7fbf4415d7e0 2024-04-06T12:13:17.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.573+0000 7fbf43fff700 1 --2- 172.21.15.189:0/193821922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf4415d7e0 0x7fbf440d0350 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009840 tx=0x7fbf34009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899372593824f163 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:17.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.573+0000 7fbf42ffd700 1 -- 172.21.15.189:0/193821922 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf34011830 con 0x7fbf4415d7e0 2024-04-06T12:13:17.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.573+0000 7fbf42ffd700 1 -- 172.21.15.189:0/193821922 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf34011990 con 0x7fbf4415d7e0 2024-04-06T12:13:17.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.573+0000 7fbf42ffd700 1 -- 172.21.15.189:0/193821922 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf34011d20 con 0x7fbf4415d7e0 2024-04-06T12:13:17.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.573+0000 7fbf4a65d700 1 -- 172.21.15.189:0/193821922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf4415d7e0 msgr2=0x7fbf440d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:17.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.573+0000 7fbf4a65d700 1 --2- 172.21.15.189:0/193821922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf4415d7e0 0x7fbf440d0350 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009840 tx=0x7fbf34009b50 comp rx=0 tx=0).stop 2024-04-06T12:13:17.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.574+0000 7fbf4a65d700 1 -- 172.21.15.189:0/193821922 shutdown_connections 2024-04-06T12:13:17.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.574+0000 7fbf4a65d700 1 --2- 172.21.15.189:0/193821922 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf440d1cc0 0x7fbf440b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.574+0000 7fbf4a65d700 1 --2- 172.21.15.189:0/193821922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf4415d7e0 0x7fbf440d0350 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.574+0000 7fbf4a65d700 1 -- 172.21.15.189:0/193821922 >> 172.21.15.189:0/193821922 conn(0x7fbf44072250 msgr2=0x7fbf44072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:17.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.574+0000 7fbf4a65d700 1 -- 172.21.15.189:0/193821922 shutdown_connections 2024-04-06T12:13:17.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.574+0000 7fbf4a65d700 1 -- 172.21.15.189:0/193821922 wait complete. 2024-04-06T12:13:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.575+0000 7fbf4a65d700 1 Processor -- start 2024-04-06T12:13:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.575+0000 7fbf4a65d700 1 -- start start 2024-04-06T12:13:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf4a65d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf440d0890 0x7fbf4414d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf4a65d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf440d1cc0 0x7fbf440c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf4a65d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf4415d7e0 0x7fbf440cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf4a65d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4414dd30 con 0x7fbf4415d7e0 2024-04-06T12:13:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf4a65d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf440c7d10 con 0x7fbf440d0890 2024-04-06T12:13:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf4a65d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf440c7470 con 0x7fbf440d1cc0 2024-04-06T12:13:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf48bfa700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf4415d7e0 0x7fbf440cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf48bfa700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf4415d7e0 0x7fbf440cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57392/0 (socket says 172.21.15.189:57392) 2024-04-06T12:13:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf48bfa700 1 -- 172.21.15.189:0/2717033633 learned_addr learned my addr 172.21.15.189:0/2717033633 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:17.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf437fe700 1 --2- 172.21.15.189:0/2717033633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf440d1cc0 0x7fbf440c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:17.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf43fff700 1 --2- 172.21.15.189:0/2717033633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf440d0890 0x7fbf4414d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:17.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf48bfa700 1 -- 172.21.15.189:0/2717033633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf440d1cc0 msgr2=0x7fbf440c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:17.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf48bfa700 1 --2- 172.21.15.189:0/2717033633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf440d1cc0 0x7fbf440c6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf48bfa700 1 -- 172.21.15.189:0/2717033633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf440d0890 msgr2=0x7fbf4414d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:17.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf48bfa700 1 --2- 172.21.15.189:0/2717033633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf440d0890 0x7fbf4414d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf48bfa700 1 -- 172.21.15.189:0/2717033633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf340094a0 con 0x7fbf4415d7e0 2024-04-06T12:13:17.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf48bfa700 1 --2- 172.21.15.189:0/2717033633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf4415d7e0 0x7fbf440cc4b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c009fd0 tx=0x7fbf3c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:17.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf40ff9700 1 -- 172.21.15.189:0/2717033633 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c007ea0 con 0x7fbf4415d7e0 2024-04-06T12:13:17.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf40ff9700 1 -- 172.21.15.189:0/2717033633 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3c00ed30 con 0x7fbf4415d7e0 2024-04-06T12:13:17.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf40ff9700 1 -- 172.21.15.189:0/2717033633 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c00eea0 con 0x7fbf4415d7e0 2024-04-06T12:13:17.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf4a65d700 1 -- 172.21.15.189:0/2717033633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf440c7750 con 0x7fbf4415d7e0 2024-04-06T12:13:17.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.576+0000 7fbf4a65d700 1 -- 172.21.15.189:0/2717033633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf440b5950 con 0x7fbf4415d7e0 2024-04-06T12:13:17.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.577+0000 7fbf4a65d700 1 -- 172.21.15.189:0/2717033633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf440b5ae0 con 0x7fbf4415d7e0 2024-04-06T12:13:17.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.578+0000 7fbf40ff9700 1 -- 172.21.15.189:0/2717033633 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbf3c014660 con 0x7fbf4415d7e0 2024-04-06T12:13:17.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.578+0000 7fbf40ff9700 1 --2- 172.21.15.189:0/2717033633 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf3006ac90 0x7fbf3006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:17.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.578+0000 7fbf40ff9700 1 -- 172.21.15.189:0/2717033633 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fbf3c089750 con 0x7fbf4415d7e0 2024-04-06T12:13:17.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.578+0000 7fbf43fff700 1 --2- 172.21.15.189:0/2717033633 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf3006ac90 0x7fbf3006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:17.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.579+0000 7fbf43fff700 1 --2- 172.21.15.189:0/2717033633 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf3006ac90 0x7fbf3006d150 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbf340090b0 tx=0x7fbf34008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:17.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.583+0000 7fbf40ff9700 1 -- 172.21.15.189:0/2717033633 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbf3c054b30 con 0x7fbf4415d7e0 2024-04-06T12:13:17.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.858+0000 7fbf4a65d700 1 -- 172.21.15.189:0/2717033633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7fbf440c8210 con 0x7fbf4415d7e0 2024-04-06T12:13:17.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.862+0000 7fbf40ff9700 1 -- 172.21.15.189:0/2717033633 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7fbf440c8210 con 0x7fbf4415d7e0 2024-04-06T12:13:17.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.864+0000 7fbf4a65d700 1 -- 172.21.15.189:0/2717033633 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf3006ac90 msgr2=0x7fbf3006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:17.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.864+0000 7fbf4a65d700 1 --2- 172.21.15.189:0/2717033633 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf3006ac90 0x7fbf3006d150 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbf340090b0 tx=0x7fbf34008030 comp rx=0 tx=0).stop 2024-04-06T12:13:17.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.864+0000 7fbf4a65d700 1 -- 172.21.15.189:0/2717033633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf4415d7e0 msgr2=0x7fbf440cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:17.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.864+0000 7fbf4a65d700 1 --2- 172.21.15.189:0/2717033633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf4415d7e0 0x7fbf440cc4b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c009fd0 tx=0x7fbf3c00e3c0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.864+0000 7fbf4a65d700 1 -- 172.21.15.189:0/2717033633 shutdown_connections 2024-04-06T12:13:17.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.864+0000 7fbf4a65d700 1 --2- 172.21.15.189:0/2717033633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf4415d7e0 0x7fbf440cc4b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.864+0000 7fbf4a65d700 1 --2- 172.21.15.189:0/2717033633 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf3006ac90 0x7fbf3006d150 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.864+0000 7fbf4a65d700 1 --2- 172.21.15.189:0/2717033633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf440d0890 0x7fbf4414d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.864+0000 7fbf4a65d700 1 --2- 172.21.15.189:0/2717033633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf440d1cc0 0x7fbf440c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:17.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.864+0000 7fbf4a65d700 1 -- 172.21.15.189:0/2717033633 >> 172.21.15.189:0/2717033633 conn(0x7fbf44072250 msgr2=0x7fbf440b66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:17.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.864+0000 7fbf4a65d700 1 -- 172.21.15.189:0/2717033633 shutdown_connections 2024-04-06T12:13:17.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.864+0000 7fbf4a65d700 1 -- 172.21.15.189:0/2717033633 wait complete. 2024-04-06T12:13:17.876 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:13:17.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.998+0000 7f64f00f3700 1 Processor -- start 2024-04-06T12:13:17.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:17.999+0000 7f64f00f3700 1 -- start start 2024-04-06T12:13:18.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.000+0000 7f64f00f3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f64e80cd3e0 0x7f64e80cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:18.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.000+0000 7f64f00f3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f64e815d7c0 0x7f64e806b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:18.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.000+0000 7f64f00f3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e806ceb0 con 0x7f64e806b6f0 2024-04-06T12:13:18.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.000+0000 7f64f00f3700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e80cdfa0 con 0x7f64e815d7c0 2024-04-06T12:13:18.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.000+0000 7f64f00f3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e806d750 con 0x7f64e80cd3e0 2024-04-06T12:13:18.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.000+0000 7f64ed68e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f64e815d7c0 0x7f64e806b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:18.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.000+0000 7f64ee690700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f64e806b6f0 0x7f64e81610f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40080/0 (socket says 172.21.15.189:40080) 2024-04-06T12:13:18.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.000+0000 7f64ee690700 1 -- 172.21.15.189:0/1381267086 learned_addr learned my addr 172.21.15.189:0/1381267086 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:18.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.000+0000 7f64ede8f700 1 --2- 172.21.15.189:0/1381267086 >> v2:172.21.15.192:3300/0 conn(0x7f64e80cd3e0 0x7f64e80cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:18.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.000+0000 7f64ed68e700 1 -- 172.21.15.189:0/1381267086 >> v2:172.21.15.192:3300/0 conn(0x7f64e80cd3e0 msgr2=0x7f64e80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:18.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.000+0000 7f64ed68e700 1 --2- 172.21.15.189:0/1381267086 >> v2:172.21.15.192:3300/0 conn(0x7f64e80cd3e0 0x7f64e80cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.000+0000 7f64ed68e700 1 -- 172.21.15.189:0/1381267086 >> v1:172.21.15.189:6789/0 conn(0x7f64e806b6f0 legacy=0x7f64e81610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:18.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.000+0000 7f64ece8d700 1 -- 172.21.15.189:0/1381267086 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376546691 0 0) 0x7f64e806ceb0 con 0x7f64e806b6f0 2024-04-06T12:13:18.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.000+0000 7f64ed68e700 1 -- 172.21.15.189:0/1381267086 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e80cdd00 con 0x7f64e815d7c0 2024-04-06T12:13:18.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.000+0000 7f64ede8f700 1 --2- 172.21.15.189:0/1381267086 >> v2:172.21.15.192:3300/0 conn(0x7f64e80cd3e0 0x7f64e80cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:18.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.000+0000 7f64ed68e700 1 --2- 172.21.15.189:0/1381267086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64e815d7c0 0x7f64e806b1b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f64e80723a0 tx=0x7f64e4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1d9fa0853f8181d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:18.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.000+0000 7f64ece8d700 1 -- 172.21.15.189:0/1381267086 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e4011830 con 0x7f64e815d7c0 2024-04-06T12:13:18.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.000+0000 7f64ece8d700 1 -- 172.21.15.189:0/1381267086 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e4011990 con 0x7f64e815d7c0 2024-04-06T12:13:18.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.000+0000 7f64ece8d700 1 -- 172.21.15.189:0/1381267086 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e4011d20 con 0x7f64e815d7c0 2024-04-06T12:13:18.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.001+0000 7f64f00f3700 1 -- 172.21.15.189:0/1381267086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64e815d7c0 msgr2=0x7f64e806b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:18.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.001+0000 7f64f00f3700 1 --2- 172.21.15.189:0/1381267086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64e815d7c0 0x7f64e806b1b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f64e80723a0 tx=0x7f64e4009b20 comp rx=0 tx=0).stop 2024-04-06T12:13:18.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.001+0000 7f64f00f3700 1 -- 172.21.15.189:0/1381267086 shutdown_connections 2024-04-06T12:13:18.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.001+0000 7f64f00f3700 1 --2- 172.21.15.189:0/1381267086 >> v2:172.21.15.192:3300/0 conn(0x7f64e80cd3e0 0x7f64e80cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.001+0000 7f64f00f3700 1 --2- 172.21.15.189:0/1381267086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64e815d7c0 0x7f64e806b1b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.001+0000 7f64f00f3700 1 -- 172.21.15.189:0/1381267086 >> 172.21.15.189:0/1381267086 conn(0x7f64e80cc290 msgr2=0x7f64e80cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:18.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.001+0000 7f64f00f3700 1 -- 172.21.15.189:0/1381267086 shutdown_connections 2024-04-06T12:13:18.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.001+0000 7f64f00f3700 1 -- 172.21.15.189:0/1381267086 wait complete. 2024-04-06T12:13:18.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.002+0000 7f64f00f3700 1 Processor -- start 2024-04-06T12:13:18.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.002+0000 7f64f00f3700 1 -- start start 2024-04-06T12:13:18.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.003+0000 7f64f00f3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64e806b6f0 0x7f64e80b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:18.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.003+0000 7f64f00f3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64e80cd3e0 0x7f64e80c4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:18.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.003+0000 7f64f00f3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64e815d7c0 0x7f64e80c4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:18.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.003+0000 7f64f00f3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80c5550 con 0x7f64e806b6f0 2024-04-06T12:13:18.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.003+0000 7f64f00f3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80b2cb0 con 0x7f64e80cd3e0 2024-04-06T12:13:18.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.003+0000 7f64f00f3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80b3b10 con 0x7f64e815d7c0 2024-04-06T12:13:18.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.003+0000 7f64ede8f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64e806b6f0 0x7f64e80b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:18.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.003+0000 7f64ede8f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64e806b6f0 0x7f64e80b48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57410/0 (socket says 172.21.15.189:57410) 2024-04-06T12:13:18.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.003+0000 7f64ede8f700 1 -- 172.21.15.189:0/1451994338 learned_addr learned my addr 172.21.15.189:0/1451994338 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:18.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.003+0000 7f64ed68e700 1 --2- 172.21.15.189:0/1451994338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64e80cd3e0 0x7f64e80c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:18.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.003+0000 7f64ee690700 1 --2- 172.21.15.189:0/1451994338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64e815d7c0 0x7f64e80c4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:18.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.003+0000 7f64ede8f700 1 -- 172.21.15.189:0/1451994338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64e815d7c0 msgr2=0x7f64e80c4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:18.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.003+0000 7f64ede8f700 1 --2- 172.21.15.189:0/1451994338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64e815d7c0 0x7f64e80c4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.003+0000 7f64ede8f700 1 -- 172.21.15.189:0/1451994338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64e80cd3e0 msgr2=0x7f64e80c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:18.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.003+0000 7f64ede8f700 1 --2- 172.21.15.189:0/1451994338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64e80cd3e0 0x7f64e80c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.003+0000 7f64ede8f700 1 -- 172.21.15.189:0/1451994338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e40094a0 con 0x7f64e806b6f0 2024-04-06T12:13:18.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.003+0000 7f64ee690700 1 --2- 172.21.15.189:0/1451994338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64e815d7c0 0x7f64e80c4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:18.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.004+0000 7f64ede8f700 1 --2- 172.21.15.189:0/1451994338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64e806b6f0 0x7f64e80b48a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00c750 tx=0x7f64dc00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:18.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.004+0000 7f64da7fc700 1 -- 172.21.15.189:0/1451994338 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc011800 con 0x7f64e806b6f0 2024-04-06T12:13:18.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.004+0000 7f64da7fc700 1 -- 172.21.15.189:0/1451994338 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64dc011960 con 0x7f64e806b6f0 2024-04-06T12:13:18.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.004+0000 7f64f00f3700 1 -- 172.21.15.189:0/1451994338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e80b2f00 con 0x7f64e806b6f0 2024-04-06T12:13:18.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.004+0000 7f64da7fc700 1 -- 172.21.15.189:0/1451994338 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc011cd0 con 0x7f64e806b6f0 2024-04-06T12:13:18.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.004+0000 7f64f00f3700 1 -- 172.21.15.189:0/1451994338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e80b34c0 con 0x7f64e806b6f0 2024-04-06T12:13:18.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.005+0000 7f64f00f3700 1 -- 172.21.15.189:0/1451994338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e8074990 con 0x7f64e806b6f0 2024-04-06T12:13:18.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.005+0000 7f64da7fc700 1 -- 172.21.15.189:0/1451994338 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f64dc021350 con 0x7f64e806b6f0 2024-04-06T12:13:18.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.005+0000 7f64da7fc700 1 --2- 172.21.15.189:0/1451994338 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f64cc06ac90 0x7f64cc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:18.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.006+0000 7f64da7fc700 1 -- 172.21.15.189:0/1451994338 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f64dc089040 con 0x7f64e806b6f0 2024-04-06T12:13:18.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.006+0000 7f64ed68e700 1 --2- 172.21.15.189:0/1451994338 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f64cc06ac90 0x7f64cc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:18.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.006+0000 7f64ed68e700 1 --2- 172.21.15.189:0/1451994338 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f64cc06ac90 0x7f64cc06d150 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f64e40090b0 tx=0x7f64e401c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:18.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.010+0000 7f64da7fc700 1 -- 172.21.15.189:0/1451994338 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f64dc0544c0 con 0x7f64e806b6f0 2024-04-06T12:13:18.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.205+0000 7f64f00f3700 1 -- 172.21.15.189:0/1451994338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64e80b5790 con 0x7f64e806b6f0 2024-04-06T12:13:18.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.206+0000 7f64da7fc700 1 -- 172.21.15.189:0/1451994338 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+12642 (secure 0 0 0) 0x7f64dc057ae0 con 0x7f64e806b6f0 2024-04-06T12:13:18.205 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:13:18.206 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":35,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:13:14.854664+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:13:18.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.208+0000 7f64f00f3700 1 -- 172.21.15.189:0/1451994338 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f64cc06ac90 msgr2=0x7f64cc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:18.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.208+0000 7f64f00f3700 1 --2- 172.21.15.189:0/1451994338 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f64cc06ac90 0x7f64cc06d150 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f64e40090b0 tx=0x7f64e401c040 comp rx=0 tx=0).stop 2024-04-06T12:13:18.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.208+0000 7f64f00f3700 1 -- 172.21.15.189:0/1451994338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64e806b6f0 msgr2=0x7f64e80b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:18.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.208+0000 7f64f00f3700 1 --2- 172.21.15.189:0/1451994338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64e806b6f0 0x7f64e80b48a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00c750 tx=0x7f64dc00cb10 comp rx=0 tx=0).stop 2024-04-06T12:13:18.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.208+0000 7f64f00f3700 1 -- 172.21.15.189:0/1451994338 shutdown_connections 2024-04-06T12:13:18.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.208+0000 7f64f00f3700 1 --2- 172.21.15.189:0/1451994338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64e815d7c0 0x7f64e80c4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.208+0000 7f64f00f3700 1 --2- 172.21.15.189:0/1451994338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64e806b6f0 0x7f64e80b48a0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.208+0000 7f64f00f3700 1 --2- 172.21.15.189:0/1451994338 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f64cc06ac90 0x7f64cc06d150 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.208+0000 7f64f00f3700 1 --2- 172.21.15.189:0/1451994338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64e80cd3e0 0x7f64e80c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.208+0000 7f64f00f3700 1 -- 172.21.15.189:0/1451994338 >> 172.21.15.189:0/1451994338 conn(0x7f64e80cc290 msgr2=0x7f64e80ae050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:18.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.208+0000 7f64f00f3700 1 -- 172.21.15.189:0/1451994338 shutdown_connections 2024-04-06T12:13:18.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.208+0000 7f64f00f3700 1 -- 172.21.15.189:0/1451994338 wait complete. 2024-04-06T12:13:18.220 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:13:18.220 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:13:18.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.345+0000 7f4adfadd700 1 Processor -- start 2024-04-06T12:13:18.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.346+0000 7f4adfadd700 1 -- start start 2024-04-06T12:13:18.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.346+0000 7f4adfadd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4ad815d7b0 0x7f4ad80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:18.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.346+0000 7f4adfadd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ad806a730 0x7f4ad806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:18.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.346+0000 7f4adfadd700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad80b3990 con 0x7f4ad815ddf0 2024-04-06T12:13:18.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.346+0000 7f4adfadd700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad806b0f0 con 0x7f4ad815d7b0 2024-04-06T12:13:18.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.346+0000 7f4adfadd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad80b3060 con 0x7f4ad806a730 2024-04-06T12:13:18.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.347+0000 7f4add879700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4ad815d7b0 0x7f4ad80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:18.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.347+0000 7f4add879700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4ad815d7b0 0x7f4ad80b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57418/0 (socket says 172.21.15.189:57418) 2024-04-06T12:13:18.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.347+0000 7f4add879700 1 -- 172.21.15.189:0/2545802352 learned_addr learned my addr 172.21.15.189:0/2545802352 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:18.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.347+0000 7f4add078700 1 --2- 172.21.15.189:0/2545802352 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ad806a730 0x7f4ad806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:18.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.347+0000 7f4add879700 1 -- 172.21.15.189:0/2545802352 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ad806a730 msgr2=0x7f4ad806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:18.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.347+0000 7f4add879700 1 --2- 172.21.15.189:0/2545802352 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ad806a730 0x7f4ad806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.347+0000 7f4add879700 1 -- 172.21.15.189:0/2545802352 >> v1:172.21.15.192:6789/0 conn(0x7f4ad815ddf0 legacy=0x7f4ad815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:18.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.347+0000 7f4add879700 1 -- 172.21.15.189:0/2545802352 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad80b3340 con 0x7f4ad815d7b0 2024-04-06T12:13:18.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.347+0000 7f4add879700 1 --2- 172.21.15.189:0/2545802352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ad815d7b0 0x7f4ad80b3e20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4acc0098f0 tx=0x7f4acc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94351d39855142b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:18.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.347+0000 7f4adc877700 1 -- 172.21.15.189:0/2545802352 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc004230 con 0x7f4ad815d7b0 2024-04-06T12:13:18.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.347+0000 7f4adc877700 1 -- 172.21.15.189:0/2545802352 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4acc004390 con 0x7f4ad815d7b0 2024-04-06T12:13:18.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.347+0000 7f4adc877700 1 -- 172.21.15.189:0/2545802352 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc0036a0 con 0x7f4ad815d7b0 2024-04-06T12:13:18.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.348+0000 7f4adfadd700 1 -- 172.21.15.189:0/2545802352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ad815d7b0 msgr2=0x7f4ad80b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:18.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.348+0000 7f4adfadd700 1 --2- 172.21.15.189:0/2545802352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ad815d7b0 0x7f4ad80b3e20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4acc0098f0 tx=0x7f4acc009c00 comp rx=0 tx=0).stop 2024-04-06T12:13:18.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.348+0000 7f4adfadd700 1 -- 172.21.15.189:0/2545802352 shutdown_connections 2024-04-06T12:13:18.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.348+0000 7f4adfadd700 1 --2- 172.21.15.189:0/2545802352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ad815d7b0 0x7f4ad80b3e20 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.348+0000 7f4adfadd700 1 --2- 172.21.15.189:0/2545802352 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ad806a730 0x7f4ad806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.348+0000 7f4adfadd700 1 -- 172.21.15.189:0/2545802352 >> 172.21.15.189:0/2545802352 conn(0x7f4ad80d00b0 msgr2=0x7f4ad80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:18.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.348+0000 7f4adfadd700 1 -- 172.21.15.189:0/2545802352 shutdown_connections 2024-04-06T12:13:18.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.348+0000 7f4adfadd700 1 -- 172.21.15.189:0/2545802352 wait complete. 2024-04-06T12:13:18.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.349+0000 7f4adfadd700 1 Processor -- start 2024-04-06T12:13:18.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.349+0000 7f4adfadd700 1 -- start start 2024-04-06T12:13:18.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.350+0000 7f4adfadd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ad806a730 0x7f4ad8141f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:18.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.350+0000 7f4adfadd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ad815d7b0 0x7f4ad8142710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:18.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.350+0000 7f4adfadd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ad815ddf0 0x7f4ad80cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.350+0000 7f4adfadd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad80ce8d0 con 0x7f4ad806a730 2024-04-06T12:13:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.350+0000 7f4adfadd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad8142450 con 0x7f4ad815d7b0 2024-04-06T12:13:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.350+0000 7f4adfadd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad80cd990 con 0x7f4ad815ddf0 2024-04-06T12:13:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.350+0000 7f4add879700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ad806a730 0x7f4ad8141f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.350+0000 7f4add879700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ad806a730 0x7f4ad8141f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57422/0 (socket says 172.21.15.189:57422) 2024-04-06T12:13:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.350+0000 7f4add879700 1 -- 172.21.15.189:0/2493265943 learned_addr learned my addr 172.21.15.189:0/2493265943 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.350+0000 7f4add879700 1 -- 172.21.15.189:0/2493265943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ad815ddf0 msgr2=0x7f4ad80cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.350+0000 7f4add879700 1 --2- 172.21.15.189:0/2493265943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ad815ddf0 0x7f4ad80cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.350+0000 7f4add879700 1 -- 172.21.15.189:0/2493265943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ad815d7b0 msgr2=0x7f4ad8142710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.350+0000 7f4add879700 1 --2- 172.21.15.189:0/2493265943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ad815d7b0 0x7f4ad8142710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.350+0000 7f4add879700 1 -- 172.21.15.189:0/2493265943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4acc009550 con 0x7f4ad806a730 2024-04-06T12:13:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.350+0000 7f4add879700 1 --2- 172.21.15.189:0/2493265943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ad806a730 0x7f4ad8141f10 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009130 tx=0x7f4acc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.350+0000 7f4aca7fc700 1 -- 172.21.15.189:0/2493265943 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc003e70 con 0x7f4ad806a730 2024-04-06T12:13:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.351+0000 7f4aca7fc700 1 -- 172.21.15.189:0/2493265943 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4acc01f070 con 0x7f4ad806a730 2024-04-06T12:13:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.351+0000 7f4adfadd700 1 -- 172.21.15.189:0/2493265943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad80cdc50 con 0x7f4ad806a730 2024-04-06T12:13:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.351+0000 7f4aca7fc700 1 -- 172.21.15.189:0/2493265943 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc00b8a0 con 0x7f4ad806a730 2024-04-06T12:13:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.351+0000 7f4adfadd700 1 -- 172.21.15.189:0/2493265943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad80ce210 con 0x7f4ad806a730 2024-04-06T12:13:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.352+0000 7f4adfadd700 1 -- 172.21.15.189:0/2493265943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad8067bb0 con 0x7f4ad806a730 2024-04-06T12:13:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.352+0000 7f4aca7fc700 1 -- 172.21.15.189:0/2493265943 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4acc021030 con 0x7f4ad806a730 2024-04-06T12:13:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.352+0000 7f4aca7fc700 1 --2- 172.21.15.189:0/2493265943 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4abc06ac90 0x7f4abc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.352+0000 7f4aca7fc700 1 -- 172.21.15.189:0/2493265943 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f4acc08a8d0 con 0x7f4ad806a730 2024-04-06T12:13:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.353+0000 7f4add078700 1 --2- 172.21.15.189:0/2493265943 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4abc06ac90 0x7f4abc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.353+0000 7f4add078700 1 --2- 172.21.15.189:0/2493265943 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4abc06ac90 0x7f4abc06d150 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4ad806fe30 tx=0x7f4ad4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:18.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.357+0000 7f4aca7fc700 1 -- 172.21.15.189:0/2493265943 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4acc055cb0 con 0x7f4ad806a730 2024-04-06T12:13:18.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.551+0000 7f4adfadd700 1 -- 172.21.15.189:0/2493265943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4ad8067bb0 con 0x7f4ad806a730 2024-04-06T12:13:18.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.552+0000 7f4aca7fc700 1 -- 172.21.15.189:0/2493265943 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4acc0592d0 con 0x7f4ad806a730 2024-04-06T12:13:18.551 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:13:18.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.554+0000 7f4adfadd700 1 -- 172.21.15.189:0/2493265943 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4abc06ac90 msgr2=0x7f4abc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:18.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.554+0000 7f4adfadd700 1 --2- 172.21.15.189:0/2493265943 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4abc06ac90 0x7f4abc06d150 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4ad806fe30 tx=0x7f4ad4008040 comp rx=0 tx=0).stop 2024-04-06T12:13:18.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.554+0000 7f4adfadd700 1 -- 172.21.15.189:0/2493265943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ad806a730 msgr2=0x7f4ad8141f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:18.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.554+0000 7f4adfadd700 1 --2- 172.21.15.189:0/2493265943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ad806a730 0x7f4ad8141f10 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009130 tx=0x7f4acc004750 comp rx=0 tx=0).stop 2024-04-06T12:13:18.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.554+0000 7f4adfadd700 1 -- 172.21.15.189:0/2493265943 shutdown_connections 2024-04-06T12:13:18.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.554+0000 7f4adfadd700 1 --2- 172.21.15.189:0/2493265943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ad815ddf0 0x7f4ad80cecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.554+0000 7f4adfadd700 1 --2- 172.21.15.189:0/2493265943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ad806a730 0x7f4ad8141f10 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.554+0000 7f4adfadd700 1 --2- 172.21.15.189:0/2493265943 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4abc06ac90 0x7f4abc06d150 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.554+0000 7f4adfadd700 1 --2- 172.21.15.189:0/2493265943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ad815d7b0 0x7f4ad8142710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.554+0000 7f4adfadd700 1 -- 172.21.15.189:0/2493265943 >> 172.21.15.189:0/2493265943 conn(0x7f4ad80d00b0 msgr2=0x7f4ad815f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:18.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.554+0000 7f4adfadd700 1 -- 172.21.15.189:0/2493265943 shutdown_connections 2024-04-06T12:13:18.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.554+0000 7f4adfadd700 1 -- 172.21.15.189:0/2493265943 wait complete. 2024-04-06T12:13:18.566 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-06T12:13:18.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.689+0000 7f3cdfd3b700 1 Processor -- start 2024-04-06T12:13:18.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.690+0000 7f3cdfd3b700 1 -- start start 2024-04-06T12:13:18.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.690+0000 7f3cdfd3b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3cd806ae40 0x7f3cd80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:18.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.690+0000 7f3cdfd3b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd80b3990 con 0x7f3cd815d7b0 2024-04-06T12:13:18.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.690+0000 7f3cdfd3b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd80b3060 con 0x7f3cd815ddf0 2024-04-06T12:13:18.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.690+0000 7f3cdfd3b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd815e950 con 0x7f3cd806ae40 2024-04-06T12:13:18.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.691+0000 7f3cdd2d6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3cd806ae40 0x7f3cd80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:18.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.691+0000 7f3cddad7700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3cd815d7b0 0x7f3cd806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40086/0 (socket says 172.21.15.189:40086) 2024-04-06T12:13:18.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.691+0000 7f3cddad7700 1 -- 172.21.15.189:0/2921283107 learned_addr learned my addr 172.21.15.189:0/2921283107 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:18.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.691+0000 7f3cdd2d6700 1 -- 172.21.15.189:0/2921283107 >> v1:172.21.15.192:6789/0 conn(0x7f3cd815ddf0 legacy=0x7f3cd815e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:18.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.691+0000 7f3cdd2d6700 1 -- 172.21.15.189:0/2921283107 >> v1:172.21.15.189:6789/0 conn(0x7f3cd815d7b0 legacy=0x7f3cd806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:18.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.691+0000 7f3cdd2d6700 1 -- 172.21.15.189:0/2921283107 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd80b3340 con 0x7f3cd806ae40 2024-04-06T12:13:18.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.691+0000 7f3cdcad5700 1 -- 172.21.15.189:0/2921283107 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468318941 0 0) 0x7f3cd80b3990 con 0x7f3cd815d7b0 2024-04-06T12:13:18.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.691+0000 7f3cdd2d6700 1 --2- 172.21.15.189:0/2921283107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cd806ae40 0x7f3cd80b3e20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4009840 tx=0x7f3cd4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e578b16417ad306c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:18.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.691+0000 7f3cdcad5700 1 -- 172.21.15.189:0/2921283107 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4011830 con 0x7f3cd806ae40 2024-04-06T12:13:18.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.691+0000 7f3cdcad5700 1 -- 172.21.15.189:0/2921283107 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd4011990 con 0x7f3cd806ae40 2024-04-06T12:13:18.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.691+0000 7f3cdcad5700 1 -- 172.21.15.189:0/2921283107 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4011d20 con 0x7f3cd806ae40 2024-04-06T12:13:18.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.692+0000 7f3cdfd3b700 1 -- 172.21.15.189:0/2921283107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cd806ae40 msgr2=0x7f3cd80b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:18.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.692+0000 7f3cdfd3b700 1 --2- 172.21.15.189:0/2921283107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cd806ae40 0x7f3cd80b3e20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4009840 tx=0x7f3cd4009b50 comp rx=0 tx=0).stop 2024-04-06T12:13:18.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.692+0000 7f3cdfd3b700 1 -- 172.21.15.189:0/2921283107 shutdown_connections 2024-04-06T12:13:18.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.692+0000 7f3cdfd3b700 1 --2- 172.21.15.189:0/2921283107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cd806ae40 0x7f3cd80b3e20 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.692+0000 7f3cdfd3b700 1 -- 172.21.15.189:0/2921283107 >> 172.21.15.189:0/2921283107 conn(0x7f3cd80d00b0 msgr2=0x7f3cd80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:18.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.692+0000 7f3cdfd3b700 1 -- 172.21.15.189:0/2921283107 shutdown_connections 2024-04-06T12:13:18.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.692+0000 7f3cdfd3b700 1 -- 172.21.15.189:0/2921283107 wait complete. 2024-04-06T12:13:18.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.693+0000 7f3cdfd3b700 1 Processor -- start 2024-04-06T12:13:18.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.693+0000 7f3cdfd3b700 1 -- start start 2024-04-06T12:13:18.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cdfd3b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cd806ae40 0x7f3cd815c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:18.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cdfd3b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3cd815d7b0 0x7f3cd8156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:18.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cdfd3b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cd815ddf0 0x7f3cd80aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:18.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cdfd3b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd80b1470 con 0x7f3cd815ddf0 2024-04-06T12:13:18.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cdfd3b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd8157530 con 0x7f3cd806ae40 2024-04-06T12:13:18.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cdfd3b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd80b0ad0 con 0x7f3cd815d7b0 2024-04-06T12:13:18.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cde2d8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cd815ddf0 0x7f3cd80aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:18.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cde2d8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cd815ddf0 0x7f3cd80aeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57432/0 (socket says 172.21.15.189:57432) 2024-04-06T12:13:18.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cde2d8700 1 -- 172.21.15.189:0/2144059431 learned_addr learned my addr 172.21.15.189:0/2144059431 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:18.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cde2d8700 1 -- 172.21.15.189:0/2144059431 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3cd815d7b0 msgr2=0x7f3cd8156ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:18.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cde2d8700 1 --2- 172.21.15.189:0/2144059431 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3cd815d7b0 0x7f3cd8156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cde2d8700 1 -- 172.21.15.189:0/2144059431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cd806ae40 msgr2=0x7f3cd815c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:18.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cddad7700 1 --2- 172.21.15.189:0/2144059431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cd806ae40 0x7f3cd815c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:18.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cde2d8700 1 --2- 172.21.15.189:0/2144059431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cd806ae40 0x7f3cd815c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cde2d8700 1 -- 172.21.15.189:0/2144059431 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd40094a0 con 0x7f3cd815ddf0 2024-04-06T12:13:18.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cddad7700 1 --2- 172.21.15.189:0/2144059431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cd806ae40 0x7f3cd815c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:18.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cde2d8700 1 --2- 172.21.15.189:0/2144059431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cd815ddf0 0x7f3cd80aeb80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3cd000cb50 tx=0x7f3cd000cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:18.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cce7fc700 1 -- 172.21.15.189:0/2144059431 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0003c00 con 0x7f3cd815ddf0 2024-04-06T12:13:18.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cce7fc700 1 -- 172.21.15.189:0/2144059431 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd0003d60 con 0x7f3cd815ddf0 2024-04-06T12:13:18.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cce7fc700 1 -- 172.21.15.189:0/2144059431 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd000ab70 con 0x7f3cd815ddf0 2024-04-06T12:13:18.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cdfd3b700 1 -- 172.21.15.189:0/2144059431 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd80b0db0 con 0x7f3cd815ddf0 2024-04-06T12:13:18.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.694+0000 7f3cdfd3b700 1 -- 172.21.15.189:0/2144059431 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd80af0c0 con 0x7f3cd815ddf0 2024-04-06T12:13:18.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.695+0000 7f3cdfd3b700 1 -- 172.21.15.189:0/2144059431 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd80b15b0 con 0x7f3cd815ddf0 2024-04-06T12:13:18.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.696+0000 7f3cce7fc700 1 -- 172.21.15.189:0/2144059431 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3cd000acd0 con 0x7f3cd815ddf0 2024-04-06T12:13:18.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.696+0000 7f3cce7fc700 1 --2- 172.21.15.189:0/2144059431 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cbc06acc0 0x7f3cbc06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:18.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.696+0000 7f3cce7fc700 1 -- 172.21.15.189:0/2144059431 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f3cd00891a0 con 0x7f3cd815ddf0 2024-04-06T12:13:18.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.696+0000 7f3cddad7700 1 --2- 172.21.15.189:0/2144059431 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cbc06acc0 0x7f3cbc06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:18.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.697+0000 7f3cddad7700 1 --2- 172.21.15.189:0/2144059431 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cbc06acc0 0x7f3cbc06d180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8000f60 tx=0x7f3cc8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:18.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.701+0000 7f3cce7fc700 1 -- 172.21.15.189:0/2144059431 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3cd80b15b0 con 0x7f3cd815ddf0 2024-04-06T12:13:18.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.898+0000 7f3cdfd3b700 1 -- 172.21.15.189:0/2144059431 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3cd80b15b0 con 0x7f3cd815ddf0 2024-04-06T12:13:18.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.899+0000 7f3cce7fc700 1 -- 172.21.15.189:0/2144059431 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3cd80b15b0 con 0x7f3cd815ddf0 2024-04-06T12:13:18.898 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:13:18.898 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T12:13:18.898 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:18.898 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 1, 2024-04-06T12:13:18.898 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": ".mgr" 2024-04-06T12:13:18.898 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:18.899 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T12:13:18.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.901+0000 7f3cdfd3b700 1 -- 172.21.15.189:0/2144059431 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cbc06acc0 msgr2=0x7f3cbc06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:18.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.901+0000 7f3cdfd3b700 1 --2- 172.21.15.189:0/2144059431 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cbc06acc0 0x7f3cbc06d180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8000f60 tx=0x7f3cc8009040 comp rx=0 tx=0).stop 2024-04-06T12:13:18.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.901+0000 7f3cdfd3b700 1 -- 172.21.15.189:0/2144059431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cd815ddf0 msgr2=0x7f3cd80aeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:18.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.901+0000 7f3cdfd3b700 1 --2- 172.21.15.189:0/2144059431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cd815ddf0 0x7f3cd80aeb80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3cd000cb50 tx=0x7f3cd000cf10 comp rx=0 tx=0).stop 2024-04-06T12:13:18.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.901+0000 7f3cdfd3b700 1 -- 172.21.15.189:0/2144059431 shutdown_connections 2024-04-06T12:13:18.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.901+0000 7f3cdfd3b700 1 --2- 172.21.15.189:0/2144059431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cd815ddf0 0x7f3cd80aeb80 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.901+0000 7f3cdfd3b700 1 --2- 172.21.15.189:0/2144059431 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cbc06acc0 0x7f3cbc06d180 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.901+0000 7f3cdfd3b700 1 --2- 172.21.15.189:0/2144059431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cd806ae40 0x7f3cd815c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.901+0000 7f3cdfd3b700 1 --2- 172.21.15.189:0/2144059431 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3cd815d7b0 0x7f3cd8156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:18.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.901+0000 7f3cdfd3b700 1 -- 172.21.15.189:0/2144059431 >> 172.21.15.189:0/2144059431 conn(0x7f3cd80d00b0 msgr2=0x7f3cd815e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:18.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.901+0000 7f3cdfd3b700 1 -- 172.21.15.189:0/2144059431 shutdown_connections 2024-04-06T12:13:18.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:18.901+0000 7f3cdfd3b700 1 -- 172.21.15.189:0/2144059431 wait complete. 2024-04-06T12:13:18.912 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-06T12:13:18.912 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-06T12:13:19.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.039+0000 7f4a3318b700 1 Processor -- start 2024-04-06T12:13:19.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.039+0000 7f4a3318b700 1 -- start start 2024-04-06T12:13:19.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.040+0000 7f4a3318b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a2c15a550 0x7f4a2c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:19.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.040+0000 7f4a3318b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4a2c0b4d70 0x7f4a2c0b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:19.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.040+0000 7f4a3318b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c0cdbe0 con 0x7f4a2c0cf900 2024-04-06T12:13:19.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.040+0000 7f4a3318b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a2c15a940 con 0x7f4a2c0b4d70 2024-04-06T12:13:19.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.040+0000 7f4a3318b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c15ac00 con 0x7f4a2c15a550 2024-04-06T12:13:19.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.040+0000 7f4a31728700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4a2c0b4d70 0x7f4a2c0b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:19.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.040+0000 7f4a31728700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4a2c0b4d70 0x7f4a2c0b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57446/0 (socket says 172.21.15.189:57446) 2024-04-06T12:13:19.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.040+0000 7f4a31728700 1 -- 172.21.15.189:0/1993057078 learned_addr learned my addr 172.21.15.189:0/1993057078 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:19.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.040+0000 7f4a30f27700 1 --2- 172.21.15.189:0/1993057078 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a2c15a550 0x7f4a2c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:19.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.040+0000 7f4a31728700 1 -- 172.21.15.189:0/1993057078 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a2c15a550 msgr2=0x7f4a2c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:19.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.040+0000 7f4a31728700 1 --2- 172.21.15.189:0/1993057078 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a2c15a550 0x7f4a2c0cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:19.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.040+0000 7f4a31728700 1 -- 172.21.15.189:0/1993057078 >> v1:172.21.15.192:6789/0 conn(0x7f4a2c0cf900 legacy=0x7f4a2c0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:19.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.040+0000 7f4a31728700 1 -- 172.21.15.189:0/1993057078 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c15aee0 con 0x7f4a2c0b4d70 2024-04-06T12:13:19.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.040+0000 7f4a30f27700 1 --2- 172.21.15.189:0/1993057078 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a2c15a550 0x7f4a2c0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:19.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.040+0000 7f4a31728700 1 --2- 172.21.15.189:0/1993057078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a2c0b4d70 0x7f4a2c0b2fc0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4a2400d150 tx=0x7f4a2400d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bd2f7899fa8a117 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:19.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.041+0000 7f4a2b7fe700 1 -- 172.21.15.189:0/1993057078 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a24010070 con 0x7f4a2c0b4d70 2024-04-06T12:13:19.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.041+0000 7f4a2b7fe700 1 -- 172.21.15.189:0/1993057078 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2400ddc0 con 0x7f4a2c0b4d70 2024-04-06T12:13:19.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.041+0000 7f4a2b7fe700 1 -- 172.21.15.189:0/1993057078 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2400b310 con 0x7f4a2c0b4d70 2024-04-06T12:13:19.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.041+0000 7f4a3318b700 1 -- 172.21.15.189:0/1993057078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a2c0b4d70 msgr2=0x7f4a2c0b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:19.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.041+0000 7f4a3318b700 1 --2- 172.21.15.189:0/1993057078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a2c0b4d70 0x7f4a2c0b2fc0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4a2400d150 tx=0x7f4a2400d460 comp rx=0 tx=0).stop 2024-04-06T12:13:19.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.041+0000 7f4a3318b700 1 -- 172.21.15.189:0/1993057078 shutdown_connections 2024-04-06T12:13:19.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.041+0000 7f4a3318b700 1 --2- 172.21.15.189:0/1993057078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a2c0b4d70 0x7f4a2c0b2fc0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:19.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.041+0000 7f4a3318b700 1 --2- 172.21.15.189:0/1993057078 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a2c15a550 0x7f4a2c0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:19.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.041+0000 7f4a3318b700 1 -- 172.21.15.189:0/1993057078 >> 172.21.15.189:0/1993057078 conn(0x7f4a2c0d00b0 msgr2=0x7f4a2c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:19.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.041+0000 7f4a3318b700 1 -- 172.21.15.189:0/1993057078 shutdown_connections 2024-04-06T12:13:19.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.041+0000 7f4a3318b700 1 -- 172.21.15.189:0/1993057078 wait complete. 2024-04-06T12:13:19.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.042+0000 7f4a3318b700 1 Processor -- start 2024-04-06T12:13:19.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.042+0000 7f4a3318b700 1 -- start start 2024-04-06T12:13:19.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.043+0000 7f4a3318b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a2c0b4d70 0x7f4a2c13f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:19.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.043+0000 7f4a3318b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a2c0cf900 0x7f4a2c155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:19.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.043+0000 7f4a3318b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a2c1576e0 0x7f4a2c157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:19.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.043+0000 7f4a3318b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c142540 con 0x7f4a2c0b4d70 2024-04-06T12:13:19.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.043+0000 7f4a3318b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c15c0b0 con 0x7f4a2c0cf900 2024-04-06T12:13:19.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.043+0000 7f4a3318b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c156270 con 0x7f4a2c1576e0 2024-04-06T12:13:19.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.043+0000 7f4a30f27700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a2c0b4d70 0x7f4a2c13f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:19.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.043+0000 7f4a30f27700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a2c0b4d70 0x7f4a2c13f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57462/0 (socket says 172.21.15.189:57462) 2024-04-06T12:13:19.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.043+0000 7f4a30f27700 1 -- 172.21.15.189:0/3133869509 learned_addr learned my addr 172.21.15.189:0/3133869509 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:19.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.044+0000 7f4a31728700 1 --2- 172.21.15.189:0/3133869509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a2c1576e0 0x7f4a2c157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:19.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.044+0000 7f4a2bfff700 1 --2- 172.21.15.189:0/3133869509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a2c0cf900 0x7f4a2c155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:19.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.044+0000 7f4a30f27700 1 -- 172.21.15.189:0/3133869509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a2c1576e0 msgr2=0x7f4a2c157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:19.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.044+0000 7f4a30f27700 1 --2- 172.21.15.189:0/3133869509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a2c1576e0 0x7f4a2c157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:19.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.044+0000 7f4a30f27700 1 -- 172.21.15.189:0/3133869509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a2c0cf900 msgr2=0x7f4a2c155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:19.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.044+0000 7f4a30f27700 1 --2- 172.21.15.189:0/3133869509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a2c0cf900 0x7f4a2c155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:19.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.044+0000 7f4a30f27700 1 -- 172.21.15.189:0/3133869509 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2400bd40 con 0x7f4a2c0b4d70 2024-04-06T12:13:19.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.044+0000 7f4a2bfff700 1 --2- 172.21.15.189:0/3133869509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a2c0cf900 0x7f4a2c155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:19.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.044+0000 7f4a31728700 1 --2- 172.21.15.189:0/3133869509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a2c1576e0 0x7f4a2c157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:19.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.044+0000 7f4a30f27700 1 --2- 172.21.15.189:0/3133869509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a2c0b4d70 0x7f4a2c13f510 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000e810 tx=0x7f4a2000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:19.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.044+0000 7f4a297fa700 1 -- 172.21.15.189:0/3133869509 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2000c020 con 0x7f4a2c0b4d70 2024-04-06T12:13:19.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.044+0000 7f4a297fa700 1 -- 172.21.15.189:0/3133869509 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a20004250 con 0x7f4a2c0b4d70 2024-04-06T12:13:19.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.044+0000 7f4a3318b700 1 -- 172.21.15.189:0/3133869509 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2c142260 con 0x7f4a2c0b4d70 2024-04-06T12:13:19.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.044+0000 7f4a297fa700 1 -- 172.21.15.189:0/3133869509 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a200036a0 con 0x7f4a2c0b4d70 2024-04-06T12:13:19.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.044+0000 7f4a3318b700 1 -- 172.21.15.189:0/3133869509 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a2c15bd40 con 0x7f4a2c0b4d70 2024-04-06T12:13:19.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.045+0000 7f4a3318b700 1 -- 172.21.15.189:0/3133869509 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2c15b8c0 con 0x7f4a2c0b4d70 2024-04-06T12:13:19.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.045+0000 7f4a297fa700 1 -- 172.21.15.189:0/3133869509 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4a20003800 con 0x7f4a2c0b4d70 2024-04-06T12:13:19.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.046+0000 7f4a297fa700 1 --2- 172.21.15.189:0/3133869509 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4a1006ac90 0x7f4a1006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:19.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.046+0000 7f4a297fa700 1 -- 172.21.15.189:0/3133869509 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f4a20088bb0 con 0x7f4a2c0b4d70 2024-04-06T12:13:19.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.046+0000 7f4a2bfff700 1 --2- 172.21.15.189:0/3133869509 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4a1006ac90 0x7f4a1006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:19.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.046+0000 7f4a2bfff700 1 --2- 172.21.15.189:0/3133869509 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4a1006ac90 0x7f4a1006d150 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c154d60 tx=0x7f4a18009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:19.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.050+0000 7f4a297fa700 1 -- 172.21.15.189:0/3133869509 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4a2c15b8c0 con 0x7f4a2c0b4d70 2024-04-06T12:13:19.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.249+0000 7f4a3318b700 1 -- 172.21.15.189:0/3133869509 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4a2c15b8c0 con 0x7f4a2c0b4d70 2024-04-06T12:13:19.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.868+0000 7f4a297fa700 1 -- 172.21.15.189:0/3133869509 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4a2c15b8c0 con 0x7f4a2c0b4d70 2024-04-06T12:13:19.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.872+0000 7f4a3318b700 1 -- 172.21.15.189:0/3133869509 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4a1006ac90 msgr2=0x7f4a1006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:19.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.872+0000 7f4a3318b700 1 --2- 172.21.15.189:0/3133869509 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4a1006ac90 0x7f4a1006d150 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c154d60 tx=0x7f4a18009040 comp rx=0 tx=0).stop 2024-04-06T12:13:19.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.872+0000 7f4a3318b700 1 -- 172.21.15.189:0/3133869509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a2c0b4d70 msgr2=0x7f4a2c13f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:19.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.872+0000 7f4a3318b700 1 --2- 172.21.15.189:0/3133869509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a2c0b4d70 0x7f4a2c13f510 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000e810 tx=0x7f4a2000ebd0 comp rx=0 tx=0).stop 2024-04-06T12:13:19.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.873+0000 7f4a3318b700 1 -- 172.21.15.189:0/3133869509 shutdown_connections 2024-04-06T12:13:19.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.873+0000 7f4a3318b700 1 --2- 172.21.15.189:0/3133869509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a2c1576e0 0x7f4a2c157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:19.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.873+0000 7f4a3318b700 1 --2- 172.21.15.189:0/3133869509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a2c0b4d70 0x7f4a2c13f510 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:19.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.873+0000 7f4a3318b700 1 --2- 172.21.15.189:0/3133869509 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4a1006ac90 0x7f4a1006d150 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:19.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.873+0000 7f4a3318b700 1 --2- 172.21.15.189:0/3133869509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a2c0cf900 0x7f4a2c155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:19.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.873+0000 7f4a3318b700 1 -- 172.21.15.189:0/3133869509 >> 172.21.15.189:0/3133869509 conn(0x7f4a2c0d00b0 msgr2=0x7f4a2c0b3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:19.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.873+0000 7f4a3318b700 1 -- 172.21.15.189:0/3133869509 shutdown_connections 2024-04-06T12:13:19.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:19.873+0000 7f4a3318b700 1 -- 172.21.15.189:0/3133869509 wait complete. 2024-04-06T12:13:19.875 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' created 2024-04-06T12:13:19.888 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-06T12:13:20.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.020+0000 7f8f2049b700 1 Processor -- start 2024-04-06T12:13:20.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.020+0000 7f8f2049b700 1 -- start start 2024-04-06T12:13:20.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.021+0000 7f8f2049b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8f1815d7b0 0x7f8f180b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:20.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.021+0000 7f8f2049b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f1815e500 0x7f8f1806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:20.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.021+0000 7f8f2049b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f180b3990 con 0x7f8f1806a730 2024-04-06T12:13:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.021+0000 7f8f2049b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f1806b0f0 con 0x7f8f1815d7b0 2024-04-06T12:13:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.021+0000 7f8f2049b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f180b3060 con 0x7f8f1815e500 2024-04-06T12:13:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.021+0000 7f8f1e237700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8f1815d7b0 0x7f8f180b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.021+0000 7f8f1e237700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8f1815d7b0 0x7f8f180b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33024/0 (socket says 172.21.15.189:33024) 2024-04-06T12:13:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.021+0000 7f8f1e237700 1 -- 172.21.15.189:0/1476109378 learned_addr learned my addr 172.21.15.189:0/1476109378 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.021+0000 7f8f1e237700 1 -- 172.21.15.189:0/1476109378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f1815e500 msgr2=0x7f8f1806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.021+0000 7f8f1ea38700 1 --2- 172.21.15.189:0/1476109378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f1815e500 0x7f8f1806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.021+0000 7f8f1e237700 1 --2- 172.21.15.189:0/1476109378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f1815e500 0x7f8f1806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.021+0000 7f8f1e237700 1 -- 172.21.15.189:0/1476109378 >> v1:172.21.15.192:6789/0 conn(0x7f8f1806a730 legacy=0x7f8f1815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.021+0000 7f8f1e237700 1 -- 172.21.15.189:0/1476109378 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f180b3340 con 0x7f8f1815d7b0 2024-04-06T12:13:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.021+0000 7f8f1ea38700 1 --2- 172.21.15.189:0/1476109378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f1815e500 0x7f8f1806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.021+0000 7f8f1e237700 1 --2- 172.21.15.189:0/1476109378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f1815d7b0 0x7f8f180b3e20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8f080098f0 tx=0x7f8f08009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9dcbf60afc1c779 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.022+0000 7f8f1d235700 1 -- 172.21.15.189:0/1476109378 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f08004230 con 0x7f8f1815d7b0 2024-04-06T12:13:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.022+0000 7f8f1d235700 1 -- 172.21.15.189:0/1476109378 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f08004390 con 0x7f8f1815d7b0 2024-04-06T12:13:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.022+0000 7f8f1d235700 1 -- 172.21.15.189:0/1476109378 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f080036a0 con 0x7f8f1815d7b0 2024-04-06T12:13:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.022+0000 7f8f2049b700 1 -- 172.21.15.189:0/1476109378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f1815d7b0 msgr2=0x7f8f180b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.022+0000 7f8f2049b700 1 --2- 172.21.15.189:0/1476109378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f1815d7b0 0x7f8f180b3e20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8f080098f0 tx=0x7f8f08009c00 comp rx=0 tx=0).stop 2024-04-06T12:13:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.022+0000 7f8f2049b700 1 -- 172.21.15.189:0/1476109378 shutdown_connections 2024-04-06T12:13:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.022+0000 7f8f2049b700 1 --2- 172.21.15.189:0/1476109378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f1815e500 0x7f8f1806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.022+0000 7f8f2049b700 1 --2- 172.21.15.189:0/1476109378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f1815d7b0 0x7f8f180b3e20 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.022+0000 7f8f2049b700 1 -- 172.21.15.189:0/1476109378 >> 172.21.15.189:0/1476109378 conn(0x7f8f180d00b0 msgr2=0x7f8f180d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.022+0000 7f8f2049b700 1 -- 172.21.15.189:0/1476109378 shutdown_connections 2024-04-06T12:13:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.022+0000 7f8f2049b700 1 -- 172.21.15.189:0/1476109378 wait complete. 2024-04-06T12:13:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.023+0000 7f8f2049b700 1 Processor -- start 2024-04-06T12:13:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.023+0000 7f8f2049b700 1 -- start start 2024-04-06T12:13:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.024+0000 7f8f2049b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f1806a730 0x7f8f181f6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.024+0000 7f8f2049b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f1815d7b0 0x7f8f1815bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.024+0000 7f8f2049b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f1815e500 0x7f8f181502f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.024+0000 7f8f2049b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1815c2f0 con 0x7f8f1815d7b0 2024-04-06T12:13:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.024+0000 7f8f2049b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f18150830 con 0x7f8f1806a730 2024-04-06T12:13:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.024+0000 7f8f2049b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1815af20 con 0x7f8f1815e500 2024-04-06T12:13:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.024+0000 7f8f1da36700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f1815d7b0 0x7f8f1815bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.024+0000 7f8f1da36700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f1815d7b0 0x7f8f1815bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33032/0 (socket says 172.21.15.189:33032) 2024-04-06T12:13:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.024+0000 7f8f1da36700 1 -- 172.21.15.189:0/4063004985 learned_addr learned my addr 172.21.15.189:0/4063004985 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.024+0000 7f8f1ea38700 1 --2- 172.21.15.189:0/4063004985 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f1815e500 0x7f8f181502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.024+0000 7f8f1e237700 1 --2- 172.21.15.189:0/4063004985 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f1806a730 0x7f8f181f6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.025+0000 7f8f1da36700 1 -- 172.21.15.189:0/4063004985 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f1815e500 msgr2=0x7f8f181502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.025+0000 7f8f1da36700 1 --2- 172.21.15.189:0/4063004985 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f1815e500 0x7f8f181502f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.025+0000 7f8f1da36700 1 -- 172.21.15.189:0/4063004985 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f1806a730 msgr2=0x7f8f181f6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.025+0000 7f8f1da36700 1 --2- 172.21.15.189:0/4063004985 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f1806a730 0x7f8f181f6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.025+0000 7f8f1da36700 1 -- 172.21.15.189:0/4063004985 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f08009550 con 0x7f8f1815d7b0 2024-04-06T12:13:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.025+0000 7f8f1e237700 1 --2- 172.21.15.189:0/4063004985 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f1806a730 0x7f8f181f6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.025+0000 7f8f1da36700 1 --2- 172.21.15.189:0/4063004985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f1815d7b0 0x7f8f1815bdb0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8f1400c4d0 tx=0x7f8f1400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.025+0000 7f8f0effd700 1 -- 172.21.15.189:0/4063004985 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f14011790 con 0x7f8f1815d7b0 2024-04-06T12:13:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.025+0000 7f8f0effd700 1 -- 172.21.15.189:0/4063004985 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f140118f0 con 0x7f8f1815d7b0 2024-04-06T12:13:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.025+0000 7f8f0effd700 1 -- 172.21.15.189:0/4063004985 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f14011ca0 con 0x7f8f1815d7b0 2024-04-06T12:13:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.025+0000 7f8f2049b700 1 -- 172.21.15.189:0/4063004985 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1815b200 con 0x7f8f1815d7b0 2024-04-06T12:13:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.025+0000 7f8f2049b700 1 -- 172.21.15.189:0/4063004985 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f18151150 con 0x7f8f1815d7b0 2024-04-06T12:13:20.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.026+0000 7f8f2049b700 1 -- 172.21.15.189:0/4063004985 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f1815b720 con 0x7f8f1815d7b0 2024-04-06T12:13:20.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.026+0000 7f8f0effd700 1 -- 172.21.15.189:0/4063004985 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8f1401c460 con 0x7f8f1815d7b0 2024-04-06T12:13:20.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.027+0000 7f8f0effd700 1 --2- 172.21.15.189:0/4063004985 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8efc06ac90 0x7f8efc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:20.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.027+0000 7f8f0effd700 1 -- 172.21.15.189:0/4063004985 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f8f140890c0 con 0x7f8f1815d7b0 2024-04-06T12:13:20.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.027+0000 7f8f1e237700 1 --2- 172.21.15.189:0/4063004985 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8efc06ac90 0x7f8efc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:20.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.027+0000 7f8f1e237700 1 --2- 172.21.15.189:0/4063004985 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8efc06ac90 0x7f8efc06d150 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8f08009130 tx=0x7f8f08008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:20.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.032+0000 7f8f0effd700 1 -- 172.21.15.189:0/4063004985 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8f1815b720 con 0x7f8f1815d7b0 2024-04-06T12:13:20.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.225+0000 7f8f2049b700 1 -- 172.21.15.189:0/4063004985 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8f18150fb0 con 0x7f8f1815d7b0 2024-04-06T12:13:20.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.877+0000 7f8f0effd700 1 -- 172.21.15.189:0/4063004985 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8f18150fb0 con 0x7f8f1815d7b0 2024-04-06T12:13:20.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.882+0000 7f8f2049b700 1 -- 172.21.15.189:0/4063004985 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8efc06ac90 msgr2=0x7f8efc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:20.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.882+0000 7f8f2049b700 1 --2- 172.21.15.189:0/4063004985 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8efc06ac90 0x7f8efc06d150 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8f08009130 tx=0x7f8f08008030 comp rx=0 tx=0).stop 2024-04-06T12:13:20.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.882+0000 7f8f2049b700 1 -- 172.21.15.189:0/4063004985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f1815d7b0 msgr2=0x7f8f1815bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:20.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.882+0000 7f8f2049b700 1 --2- 172.21.15.189:0/4063004985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f1815d7b0 0x7f8f1815bdb0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8f1400c4d0 tx=0x7f8f1400c890 comp rx=0 tx=0).stop 2024-04-06T12:13:20.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.883+0000 7f8f2049b700 1 -- 172.21.15.189:0/4063004985 shutdown_connections 2024-04-06T12:13:20.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.883+0000 7f8f2049b700 1 --2- 172.21.15.189:0/4063004985 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f1815e500 0x7f8f181502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:20.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.883+0000 7f8f2049b700 1 --2- 172.21.15.189:0/4063004985 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8efc06ac90 0x7f8efc06d150 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:20.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.883+0000 7f8f2049b700 1 --2- 172.21.15.189:0/4063004985 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f1806a730 0x7f8f181f6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:20.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.883+0000 7f8f2049b700 1 --2- 172.21.15.189:0/4063004985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f1815d7b0 0x7f8f1815bdb0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:20.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.883+0000 7f8f2049b700 1 -- 172.21.15.189:0/4063004985 >> 172.21.15.189:0/4063004985 conn(0x7f8f180d00b0 msgr2=0x7f8f180afc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:20.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.885+0000 7f8f2049b700 1 -- 172.21.15.189:0/4063004985 shutdown_connections 2024-04-06T12:13:20.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:20.885+0000 7f8f2049b700 1 -- 172.21.15.189:0/4063004985 wait complete. 2024-04-06T12:13:20.887 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' created 2024-04-06T12:13:20.905 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-06T12:13:21.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.035+0000 7fc6f2306700 1 Processor -- start 2024-04-06T12:13:21.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.035+0000 7fc6f2306700 1 -- start start 2024-04-06T12:13:21.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.036+0000 7fc6f2306700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec15a550 0x7fc6ec0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.036+0000 7fc6f2306700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc6ec0cf900 0x7fc6ec0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.036+0000 7fc6f2306700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6ec0cdbe0 con 0x7fc6ec0b4ba0 2024-04-06T12:13:21.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.036+0000 7fc6f2306700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6ec0b5010 con 0x7fc6ec0cf900 2024-04-06T12:13:21.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.036+0000 7fc6f2306700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec15a940 con 0x7fc6ec15a550 2024-04-06T12:13:21.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.037+0000 7fc6eb7fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc6ec0cf900 0x7fc6ec0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.037+0000 7fc6ebfff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec15a550 0x7fc6ec0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.037+0000 7fc6ebfff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec15a550 0x7fc6ec0cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:33936/0 (socket says 172.21.15.189:33936) 2024-04-06T12:13:21.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.037+0000 7fc6eb7fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc6ec0cf900 0x7fc6ec0b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:35196/0 (socket says 172.21.15.189:35196) 2024-04-06T12:13:21.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.037+0000 7fc6f08a3700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fc6ec0b4ba0 0x7fc6ec06eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:47226/0 (socket says 172.21.15.189:47226) 2024-04-06T12:13:21.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.037+0000 7fc6ebfff700 1 -- 172.21.15.189:0/479796195 learned_addr learned my addr 172.21.15.189:0/479796195 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:21.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.037+0000 7fc6eb7fe700 1 -- 172.21.15.189:0/479796195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec15a550 msgr2=0x7fc6ec0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:21.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.037+0000 7fc6eb7fe700 1 --2- 172.21.15.189:0/479796195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec15a550 0x7fc6ec0cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.037+0000 7fc6eb7fe700 1 -- 172.21.15.189:0/479796195 >> v1:172.21.15.192:6789/0 conn(0x7fc6ec0b4ba0 legacy=0x7fc6ec06eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:21.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.037+0000 7fc6eaffd700 1 -- 172.21.15.189:0/479796195 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 771340131 0 0) 0x7fc6ec0cdbe0 con 0x7fc6ec0b4ba0 2024-04-06T12:13:21.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.037+0000 7fc6eb7fe700 1 -- 172.21.15.189:0/479796195 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ec15ac00 con 0x7fc6ec0cf900 2024-04-06T12:13:21.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.037+0000 7fc6ebfff700 1 --2- 172.21.15.189:0/479796195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec15a550 0x7fc6ec0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:13:21.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.037+0000 7fc6eb7fe700 1 --2- 172.21.15.189:0/479796195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6ec0cf900 0x7fc6ec0b4660 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc009840 tx=0x7fc6dc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e904077e6b1d4c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:21.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.037+0000 7fc6eaffd700 1 -- 172.21.15.189:0/479796195 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc011830 con 0x7fc6ec0cf900 2024-04-06T12:13:21.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.037+0000 7fc6eaffd700 1 -- 172.21.15.189:0/479796195 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6dc011990 con 0x7fc6ec0cf900 2024-04-06T12:13:21.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.037+0000 7fc6eaffd700 1 -- 172.21.15.189:0/479796195 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc011d20 con 0x7fc6ec0cf900 2024-04-06T12:13:21.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.038+0000 7fc6f2306700 1 -- 172.21.15.189:0/479796195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6ec0cf900 msgr2=0x7fc6ec0b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:21.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.038+0000 7fc6f2306700 1 --2- 172.21.15.189:0/479796195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6ec0cf900 0x7fc6ec0b4660 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc009840 tx=0x7fc6dc009b50 comp rx=0 tx=0).stop 2024-04-06T12:13:21.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.038+0000 7fc6f2306700 1 -- 172.21.15.189:0/479796195 shutdown_connections 2024-04-06T12:13:21.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.038+0000 7fc6f2306700 1 --2- 172.21.15.189:0/479796195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec15a550 0x7fc6ec0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.038+0000 7fc6f2306700 1 --2- 172.21.15.189:0/479796195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6ec0cf900 0x7fc6ec0b4660 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.038+0000 7fc6f2306700 1 -- 172.21.15.189:0/479796195 >> 172.21.15.189:0/479796195 conn(0x7fc6ec0d00b0 msgr2=0x7fc6ec0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:21.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.038+0000 7fc6f2306700 1 -- 172.21.15.189:0/479796195 shutdown_connections 2024-04-06T12:13:21.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.038+0000 7fc6f2306700 1 -- 172.21.15.189:0/479796195 wait complete. 2024-04-06T12:13:21.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.039+0000 7fc6f2306700 1 Processor -- start 2024-04-06T12:13:21.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.039+0000 7fc6f2306700 1 -- start start 2024-04-06T12:13:21.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.040+0000 7fc6f2306700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6ec0b4ba0 0x7fc6ec1ee8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.040+0000 7fc6f2306700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0cf900 0x7fc6ec0b0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.040+0000 7fc6f2306700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec1f1e10 0x7fc6ec1f22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.040+0000 7fc6f2306700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec1f2800 con 0x7fc6ec0cf900 2024-04-06T12:13:21.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.040+0000 7fc6f2306700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec151a10 con 0x7fc6ec0b4ba0 2024-04-06T12:13:21.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.040+0000 7fc6f2306700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec1f1be0 con 0x7fc6ec1f1e10 2024-04-06T12:13:21.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.040+0000 7fc6eb7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0cf900 0x7fc6ec0b0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.040+0000 7fc6eb7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0cf900 0x7fc6ec0b0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33048/0 (socket says 172.21.15.189:33048) 2024-04-06T12:13:21.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.040+0000 7fc6eb7fe700 1 -- 172.21.15.189:0/2575407240 learned_addr learned my addr 172.21.15.189:0/2575407240 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:21.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.040+0000 7fc6f08a3700 1 --2- 172.21.15.189:0/2575407240 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec1f1e10 0x7fc6ec1f22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.040+0000 7fc6ebfff700 1 --2- 172.21.15.189:0/2575407240 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6ec0b4ba0 0x7fc6ec1ee8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.041+0000 7fc6eb7fe700 1 -- 172.21.15.189:0/2575407240 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec1f1e10 msgr2=0x7fc6ec1f22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:21.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.041+0000 7fc6eb7fe700 1 --2- 172.21.15.189:0/2575407240 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec1f1e10 0x7fc6ec1f22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.041+0000 7fc6eb7fe700 1 -- 172.21.15.189:0/2575407240 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6ec0b4ba0 msgr2=0x7fc6ec1ee8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:21.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.041+0000 7fc6eb7fe700 1 --2- 172.21.15.189:0/2575407240 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6ec0b4ba0 0x7fc6ec1ee8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.041+0000 7fc6eb7fe700 1 -- 172.21.15.189:0/2575407240 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d400b7f0 con 0x7fc6ec0cf900 2024-04-06T12:13:21.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.041+0000 7fc6eb7fe700 1 --2- 172.21.15.189:0/2575407240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0cf900 0x7fc6ec0b0990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc009fd0 tx=0x7fc6dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:21.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.041+0000 7fc6e8ff9700 1 -- 172.21.15.189:0/2575407240 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc0036a0 con 0x7fc6ec0cf900 2024-04-06T12:13:21.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.041+0000 7fc6e8ff9700 1 -- 172.21.15.189:0/2575407240 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6dc01f070 con 0x7fc6ec0cf900 2024-04-06T12:13:21.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.041+0000 7fc6f2306700 1 -- 172.21.15.189:0/2575407240 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6dc0094a0 con 0x7fc6ec0cf900 2024-04-06T12:13:21.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.041+0000 7fc6e8ff9700 1 -- 172.21.15.189:0/2575407240 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc003a60 con 0x7fc6ec0cf900 2024-04-06T12:13:21.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.041+0000 7fc6f2306700 1 -- 172.21.15.189:0/2575407240 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ec1510e0 con 0x7fc6ec0cf900 2024-04-06T12:13:21.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.042+0000 7fc6f2306700 1 -- 172.21.15.189:0/2575407240 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6ec067bb0 con 0x7fc6ec0cf900 2024-04-06T12:13:21.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.042+0000 7fc6e8ff9700 1 -- 172.21.15.189:0/2575407240 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc6dc003bc0 con 0x7fc6ec0cf900 2024-04-06T12:13:21.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.043+0000 7fc6e8ff9700 1 --2- 172.21.15.189:0/2575407240 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc6cc06ac90 0x7fc6cc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.043+0000 7fc6e8ff9700 1 -- 172.21.15.189:0/2575407240 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5929+0+0 (secure 0 0 0) 0x7fc6dc021070 con 0x7fc6ec0cf900 2024-04-06T12:13:21.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.043+0000 7fc6ebfff700 1 --2- 172.21.15.189:0/2575407240 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc6cc06ac90 0x7fc6cc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.043+0000 7fc6ebfff700 1 --2- 172.21.15.189:0/2575407240 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc6cc06ac90 0x7fc6cc06d150 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4006d30 tx=0x7fc6d400b4d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:21.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.048+0000 7fc6e8ff9700 1 -- 172.21.15.189:0/2575407240 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc6dc055d10 con 0x7fc6ec0cf900 2024-04-06T12:13:21.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.286+0000 7fc6f2306700 1 -- 172.21.15.189:0/2575407240 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc6ec074990 con 0x7fc6ec0cf900 2024-04-06T12:13:21.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.294+0000 7fc6e8ff9700 1 -- 172.21.15.189:0/2575407240 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7fc6dc0150c0 con 0x7fc6ec0cf900 2024-04-06T12:13:21.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.298+0000 7fc6f2306700 1 -- 172.21.15.189:0/2575407240 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc6cc06ac90 msgr2=0x7fc6cc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:21.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.298+0000 7fc6f2306700 1 --2- 172.21.15.189:0/2575407240 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc6cc06ac90 0x7fc6cc06d150 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4006d30 tx=0x7fc6d400b4d0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.298+0000 7fc6f2306700 1 -- 172.21.15.189:0/2575407240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0cf900 msgr2=0x7fc6ec0b0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:21.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.298+0000 7fc6f2306700 1 --2- 172.21.15.189:0/2575407240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0cf900 0x7fc6ec0b0990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc009fd0 tx=0x7fc6dc004750 comp rx=0 tx=0).stop 2024-04-06T12:13:21.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.298+0000 7fc6f2306700 1 -- 172.21.15.189:0/2575407240 shutdown_connections 2024-04-06T12:13:21.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.298+0000 7fc6f2306700 1 --2- 172.21.15.189:0/2575407240 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec1f1e10 0x7fc6ec1f22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.298+0000 7fc6f2306700 1 --2- 172.21.15.189:0/2575407240 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc6cc06ac90 0x7fc6cc06d150 secure :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4006d30 tx=0x7fc6d400b4d0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.298+0000 7fc6f2306700 1 --2- 172.21.15.189:0/2575407240 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6ec0b4ba0 0x7fc6ec1ee8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.298+0000 7fc6f2306700 1 --2- 172.21.15.189:0/2575407240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0cf900 0x7fc6ec0b0990 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.298+0000 7fc6f2306700 1 -- 172.21.15.189:0/2575407240 >> 172.21.15.189:0/2575407240 conn(0x7fc6ec0d00b0 msgr2=0x7fc6ec06d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:21.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.300+0000 7fc6f2306700 1 -- 172.21.15.189:0/2575407240 shutdown_connections 2024-04-06T12:13:21.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.300+0000 7fc6f2306700 1 -- 172.21.15.189:0/2575407240 wait complete. 2024-04-06T12:13:21.301 INFO:teuthology.orchestra.run.smithi189.stderr:new fs with metadata pool 5 and data pool 6 2024-04-06T12:13:21.320 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:13:21.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.444+0000 7f4e2f990700 1 Processor -- start 2024-04-06T12:13:21.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.445+0000 7f4e2f990700 1 -- start start 2024-04-06T12:13:21.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.445+0000 7f4e2f990700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e280cbc20 0x7f4e2815f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.445+0000 7f4e2f990700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4e2806aa10 0x7f4e2806ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.445+0000 7f4e2f990700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e280669d0 con 0x7f4e2815e8a0 2024-04-06T12:13:21.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.445+0000 7f4e2f990700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e2806ede0 con 0x7f4e2806aa10 2024-04-06T12:13:21.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.445+0000 7f4e2f990700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2815ec80 con 0x7f4e280cbc20 2024-04-06T12:13:21.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.446+0000 7f4e2df2d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4e2806aa10 0x7f4e2806ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.446+0000 7f4e2d72c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4e2815e8a0 0x7f4e2815eee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33472/0 (socket says 172.21.15.189:33472) 2024-04-06T12:13:21.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.446+0000 7f4e2d72c700 1 -- 172.21.15.189:0/945090297 learned_addr learned my addr 172.21.15.189:0/945090297 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:21.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.446+0000 7f4e2cf2b700 1 --2- 172.21.15.189:0/945090297 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e280cbc20 0x7f4e2815f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.446+0000 7f4e2df2d700 1 -- 172.21.15.189:0/945090297 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e280cbc20 msgr2=0x7f4e2815f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:21.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.446+0000 7f4e2df2d700 1 --2- 172.21.15.189:0/945090297 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e280cbc20 0x7f4e2815f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.446+0000 7f4e2df2d700 1 -- 172.21.15.189:0/945090297 >> v1:172.21.15.189:6789/0 conn(0x7f4e2815e8a0 legacy=0x7f4e2815eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:21.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.446+0000 7f4e2df2d700 1 -- 172.21.15.189:0/945090297 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e280666f0 con 0x7f4e2806aa10 2024-04-06T12:13:21.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.446+0000 7f4e1bfff700 1 -- 172.21.15.189:0/945090297 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606051257 0 0) 0x7f4e280669d0 con 0x7f4e2815e8a0 2024-04-06T12:13:21.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.446+0000 7f4e2df2d700 1 --2- 172.21.15.189:0/945090297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e2806aa10 0x7f4e2806ae80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4e20009d90 tx=0x7f4e2000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a94818d4d3a4f53 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:21.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.446+0000 7f4e1bfff700 1 -- 172.21.15.189:0/945090297 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2000e070 con 0x7f4e2806aa10 2024-04-06T12:13:21.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.446+0000 7f4e1bfff700 1 -- 172.21.15.189:0/945090297 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2000bbd0 con 0x7f4e2806aa10 2024-04-06T12:13:21.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.446+0000 7f4e1bfff700 1 -- 172.21.15.189:0/945090297 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e20012430 con 0x7f4e2806aa10 2024-04-06T12:13:21.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.447+0000 7f4e2f990700 1 -- 172.21.15.189:0/945090297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e2806aa10 msgr2=0x7f4e2806ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:21.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.447+0000 7f4e2f990700 1 --2- 172.21.15.189:0/945090297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e2806aa10 0x7f4e2806ae80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4e20009d90 tx=0x7f4e2000b330 comp rx=0 tx=0).stop 2024-04-06T12:13:21.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.447+0000 7f4e2f990700 1 -- 172.21.15.189:0/945090297 shutdown_connections 2024-04-06T12:13:21.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.447+0000 7f4e2f990700 1 --2- 172.21.15.189:0/945090297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e2806aa10 0x7f4e2806ae80 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.447+0000 7f4e2f990700 1 --2- 172.21.15.189:0/945090297 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e280cbc20 0x7f4e2815f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.447+0000 7f4e2f990700 1 -- 172.21.15.189:0/945090297 >> 172.21.15.189:0/945090297 conn(0x7f4e280cc290 msgr2=0x7f4e280cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:21.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.447+0000 7f4e2f990700 1 -- 172.21.15.189:0/945090297 shutdown_connections 2024-04-06T12:13:21.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.447+0000 7f4e2f990700 1 -- 172.21.15.189:0/945090297 wait complete. 2024-04-06T12:13:21.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.448+0000 7f4e2f990700 1 Processor -- start 2024-04-06T12:13:21.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.448+0000 7f4e2f990700 1 -- start start 2024-04-06T12:13:21.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.449+0000 7f4e2f990700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e2806aa10 0x7f4e28159390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.449+0000 7f4e2f990700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e280cbc20 0x7f4e280b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.449+0000 7f4e2f990700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e2815e8a0 0x7f4e280b2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.449+0000 7f4e2f990700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e280c8be0 con 0x7f4e280cbc20 2024-04-06T12:13:21.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.449+0000 7f4e2f990700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e280c7fd0 con 0x7f4e2815e8a0 2024-04-06T12:13:21.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.449+0000 7f4e2f990700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e280c82b0 con 0x7f4e2806aa10 2024-04-06T12:13:21.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.449+0000 7f4e2cf2b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e280cbc20 0x7f4e280b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.449+0000 7f4e2cf2b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e280cbc20 0x7f4e280b25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33064/0 (socket says 172.21.15.189:33064) 2024-04-06T12:13:21.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.449+0000 7f4e2cf2b700 1 -- 172.21.15.189:0/1743824202 learned_addr learned my addr 172.21.15.189:0/1743824202 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:21.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.450+0000 7f4e2d72c700 1 --2- 172.21.15.189:0/1743824202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e2806aa10 0x7f4e28159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.450+0000 7f4e2df2d700 1 --2- 172.21.15.189:0/1743824202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e2815e8a0 0x7f4e280b2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.450+0000 7f4e2cf2b700 1 -- 172.21.15.189:0/1743824202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e2806aa10 msgr2=0x7f4e28159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:21.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.450+0000 7f4e2cf2b700 1 --2- 172.21.15.189:0/1743824202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e2806aa10 0x7f4e28159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.450+0000 7f4e2cf2b700 1 -- 172.21.15.189:0/1743824202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e2815e8a0 msgr2=0x7f4e280b2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:21.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.450+0000 7f4e2cf2b700 1 --2- 172.21.15.189:0/1743824202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e2815e8a0 0x7f4e280b2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.450+0000 7f4e2cf2b700 1 -- 172.21.15.189:0/1743824202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e20009a70 con 0x7f4e280cbc20 2024-04-06T12:13:21.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.450+0000 7f4e2df2d700 1 --2- 172.21.15.189:0/1743824202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e2815e8a0 0x7f4e280b2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:21.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.450+0000 7f4e2cf2b700 1 --2- 172.21.15.189:0/1743824202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e280cbc20 0x7f4e280b25b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400d800 tx=0x7f4e2400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:21.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.450+0000 7f4e19ffb700 1 -- 172.21.15.189:0/1743824202 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e240125f0 con 0x7f4e280cbc20 2024-04-06T12:13:21.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.450+0000 7f4e19ffb700 1 -- 172.21.15.189:0/1743824202 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e24012750 con 0x7f4e280cbc20 2024-04-06T12:13:21.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.450+0000 7f4e19ffb700 1 -- 172.21.15.189:0/1743824202 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e24012b00 con 0x7f4e280cbc20 2024-04-06T12:13:21.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.450+0000 7f4e2f990700 1 -- 172.21.15.189:0/1743824202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e280c8590 con 0x7f4e280cbc20 2024-04-06T12:13:21.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.450+0000 7f4e2f990700 1 -- 172.21.15.189:0/1743824202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e281598d0 con 0x7f4e280cbc20 2024-04-06T12:13:21.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.451+0000 7f4e2f990700 1 -- 172.21.15.189:0/1743824202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e28072140 con 0x7f4e280cbc20 2024-04-06T12:13:21.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.451+0000 7f4e19ffb700 1 -- 172.21.15.189:0/1743824202 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4e24012c60 con 0x7f4e280cbc20 2024-04-06T12:13:21.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.452+0000 7f4e19ffb700 1 --2- 172.21.15.189:0/1743824202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e0c06ac90 0x7f4e0c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.452+0000 7f4e19ffb700 1 -- 172.21.15.189:0/1743824202 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4e24089760 con 0x7f4e280cbc20 2024-04-06T12:13:21.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.452+0000 7f4e2d72c700 1 --2- 172.21.15.189:0/1743824202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e0c06ac90 0x7f4e0c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.452+0000 7f4e2d72c700 1 --2- 172.21.15.189:0/1743824202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e0c06ac90 0x7f4e0c06d150 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c007fd0 tx=0x7f4e1c007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:21.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.457+0000 7f4e19ffb700 1 -- 172.21.15.189:0/1743824202 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4e24054800 con 0x7f4e280cbc20 2024-04-06T12:13:21.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.653+0000 7f4e2f990700 1 -- 172.21.15.189:0/1743824202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e280c8d20 con 0x7f4e280cbc20 2024-04-06T12:13:21.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.654+0000 7f4e19ffb700 1 -- 172.21.15.189:0/1743824202 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16367 (secure 0 0 0) 0x7f4e280c8d20 con 0x7f4e280cbc20 2024-04-06T12:13:21.652 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:13:21.652 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":38,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:13:21.287596+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:13:19.268280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-06T12:13:20.243771+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:13:21.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.656+0000 7f4e2f990700 1 -- 172.21.15.189:0/1743824202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e0c06ac90 msgr2=0x7f4e0c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:21.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.656+0000 7f4e2f990700 1 --2- 172.21.15.189:0/1743824202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e0c06ac90 0x7f4e0c06d150 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c007fd0 tx=0x7f4e1c007de0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.656+0000 7f4e2f990700 1 -- 172.21.15.189:0/1743824202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e280cbc20 msgr2=0x7f4e280b25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:21.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.656+0000 7f4e2f990700 1 --2- 172.21.15.189:0/1743824202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e280cbc20 0x7f4e280b25b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400d800 tx=0x7f4e2400db10 comp rx=0 tx=0).stop 2024-04-06T12:13:21.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.656+0000 7f4e2f990700 1 -- 172.21.15.189:0/1743824202 shutdown_connections 2024-04-06T12:13:21.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.656+0000 7f4e2f990700 1 --2- 172.21.15.189:0/1743824202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e0c06ac90 0x7f4e0c06d150 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.656+0000 7f4e2f990700 1 --2- 172.21.15.189:0/1743824202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e2815e8a0 0x7f4e280b2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.656+0000 7f4e2f990700 1 --2- 172.21.15.189:0/1743824202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e2806aa10 0x7f4e28159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.656+0000 7f4e2f990700 1 --2- 172.21.15.189:0/1743824202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e280cbc20 0x7f4e280b25b0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.656+0000 7f4e2f990700 1 -- 172.21.15.189:0/1743824202 >> 172.21.15.189:0/1743824202 conn(0x7f4e280cc290 msgr2=0x7f4e2806cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:21.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.656+0000 7f4e2f990700 1 -- 172.21.15.189:0/1743824202 shutdown_connections 2024-04-06T12:13:21.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.656+0000 7f4e2f990700 1 -- 172.21.15.189:0/1743824202 wait complete. 2024-04-06T12:13:21.667 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:13:21.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.790+0000 7f45dadb3700 1 Processor -- start 2024-04-06T12:13:21.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.790+0000 7f45dadb3700 1 -- start start 2024-04-06T12:13:21.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45dadb3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f45d415d7b0 0x7f45d40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45dadb3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45d406a730 0x7f45d406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45dadb3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d40b3990 con 0x7f45d415ddf0 2024-04-06T12:13:21.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45dadb3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45d406b0f0 con 0x7f45d415d7b0 2024-04-06T12:13:21.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45dadb3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d40b3060 con 0x7f45d406a730 2024-04-06T12:13:21.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d9350700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f45d415ddf0 0x7f45d415e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33476/0 (socket says 172.21.15.189:33476) 2024-04-06T12:13:21.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d9350700 1 -- 172.21.15.189:0/844032552 learned_addr learned my addr 172.21.15.189:0/844032552 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:21.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d37fe700 1 -- 172.21.15.189:0/844032552 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848299925 0 0) 0x7f45d40b3990 con 0x7f45d415ddf0 2024-04-06T12:13:21.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d3fff700 1 --2- 172.21.15.189:0/844032552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45d406a730 0x7f45d406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d37fe700 1 -- 172.21.15.189:0/844032552 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45c40035b0 con 0x7f45d415ddf0 2024-04-06T12:13:21.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d8b4f700 1 --2- 172.21.15.189:0/844032552 >> v2:172.21.15.192:3300/0 conn(0x7f45d415d7b0 0x7f45d40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d37fe700 1 -- 172.21.15.189:0/844032552 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1278895001 0 0) 0x7f45c40035b0 con 0x7f45d415ddf0 2024-04-06T12:13:21.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d37fe700 1 -- 172.21.15.189:0/844032552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45d406a730 msgr2=0x7f45d406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:21.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d37fe700 1 --2- 172.21.15.189:0/844032552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45d406a730 0x7f45d406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d37fe700 1 -- 172.21.15.189:0/844032552 >> v2:172.21.15.192:3300/0 conn(0x7f45d415d7b0 msgr2=0x7f45d40b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:21.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d37fe700 1 --2- 172.21.15.189:0/844032552 >> v2:172.21.15.192:3300/0 conn(0x7f45d415d7b0 0x7f45d40b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d37fe700 1 -- 172.21.15.189:0/844032552 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d40b3340 con 0x7f45d415ddf0 2024-04-06T12:13:21.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d37fe700 1 -- 172.21.15.189:0/844032552 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f45cc0019f0 con 0x7f45d415ddf0 2024-04-06T12:13:21.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d37fe700 1 -- 172.21.15.189:0/844032552 >> v1:172.21.15.189:6789/0 conn(0x7f45d415ddf0 legacy=0x7f45d415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:21.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d37fe700 1 --2- 172.21.15.189:0/844032552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45d406a730 0x7f45c4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d37fe700 1 --2- 172.21.15.189:0/844032552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45c4003fd0 0x7f45c4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d37fe700 1 --2- 172.21.15.189:0/844032552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45c4004d90 0x7f45c4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d37fe700 1 -- 172.21.15.189:0/844032552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d406b0f0 con 0x7f45c4003fd0 2024-04-06T12:13:21.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d37fe700 1 -- 172.21.15.189:0/844032552 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d40b3060 con 0x7f45c4004d90 2024-04-06T12:13:21.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.791+0000 7f45d37fe700 1 -- 172.21.15.189:0/844032552 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45c400bac0 con 0x7f45d406a730 2024-04-06T12:13:21.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.792+0000 7f45d3fff700 1 --2- 172.21.15.189:0/844032552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45c4003fd0 0x7f45c4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.792+0000 7f45d8b4f700 1 --2- 172.21.15.189:0/844032552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45d406a730 0x7f45c4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.792+0000 7f45d9350700 1 --2- 172.21.15.189:0/844032552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45c4004d90 0x7f45c4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.792+0000 7f45d3fff700 1 -- 172.21.15.189:0/844032552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45d406a730 msgr2=0x7f45c4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:21.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.792+0000 7f45d3fff700 1 --2- 172.21.15.189:0/844032552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45d406a730 0x7f45c4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.792+0000 7f45d3fff700 1 -- 172.21.15.189:0/844032552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45c4004d90 msgr2=0x7f45c4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:21.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.792+0000 7f45d3fff700 1 --2- 172.21.15.189:0/844032552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45c4004d90 0x7f45c4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.792+0000 7f45d3fff700 1 -- 172.21.15.189:0/844032552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c40035b0 con 0x7f45c4003fd0 2024-04-06T12:13:21.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.792+0000 7f45d3fff700 1 --2- 172.21.15.189:0/844032552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45c4003fd0 0x7f45c4004850 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009b60 tx=0x7f45c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa5ac27ad0013ee server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:21.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.792+0000 7f45d37fe700 1 -- 172.21.15.189:0/844032552 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c000e6c0 con 0x7f45c4003fd0 2024-04-06T12:13:21.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.792+0000 7f45d37fe700 1 -- 172.21.15.189:0/844032552 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c000e820 con 0x7f45c4003fd0 2024-04-06T12:13:21.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.792+0000 7f45d37fe700 1 -- 172.21.15.189:0/844032552 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c000eb90 con 0x7f45c4003fd0 2024-04-06T12:13:21.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.793+0000 7f45dadb3700 1 -- 172.21.15.189:0/844032552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45c4003fd0 msgr2=0x7f45c4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:21.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.793+0000 7f45dadb3700 1 --2- 172.21.15.189:0/844032552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45c4003fd0 0x7f45c4004850 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009b60 tx=0x7f45c4005880 comp rx=0 tx=0).stop 2024-04-06T12:13:21.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.793+0000 7f45d9350700 1 -- 172.21.15.189:0/844032552 reap_dead start 2024-04-06T12:13:21.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.793+0000 7f45dadb3700 1 -- 172.21.15.189:0/844032552 shutdown_connections 2024-04-06T12:13:21.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.793+0000 7f45dadb3700 1 -- 172.21.15.189:0/844032552 >> 172.21.15.189:0/844032552 conn(0x7f45d40d00b0 msgr2=0x7f45d40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:21.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.793+0000 7f45dadb3700 1 -- 172.21.15.189:0/844032552 shutdown_connections 2024-04-06T12:13:21.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.793+0000 7f45dadb3700 1 -- 172.21.15.189:0/844032552 wait complete. 2024-04-06T12:13:21.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.794+0000 7f45dadb3700 1 Processor -- start 2024-04-06T12:13:21.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.794+0000 7f45dadb3700 1 -- start start 2024-04-06T12:13:21.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45dadb3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45c4003fd0 0x7f45d41f6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45dadb3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45d40b0c80 0x7f45d415c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45dadb3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45d415ce60 0x7f45d40ce310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45dadb3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d40cf260 con 0x7f45d40b0c80 2024-04-06T12:13:21.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45dadb3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d40cd240 con 0x7f45d415ce60 2024-04-06T12:13:21.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45dadb3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d41f7100 con 0x7f45c4003fd0 2024-04-06T12:13:21.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45d3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45d40b0c80 0x7f45d415c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45d3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45d40b0c80 0x7f45d415c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33084/0 (socket says 172.21.15.189:33084) 2024-04-06T12:13:21.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45d3fff700 1 -- 172.21.15.189:0/4186408971 learned_addr learned my addr 172.21.15.189:0/4186408971 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:21.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45d8b4f700 1 --2- 172.21.15.189:0/4186408971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45c4003fd0 0x7f45d41f6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45d9350700 1 --2- 172.21.15.189:0/4186408971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45d415ce60 0x7f45d40ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45d3fff700 1 -- 172.21.15.189:0/4186408971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45c4003fd0 msgr2=0x7f45d41f6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:21.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45d3fff700 1 --2- 172.21.15.189:0/4186408971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45c4003fd0 0x7f45d41f6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45d3fff700 1 -- 172.21.15.189:0/4186408971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45d415ce60 msgr2=0x7f45d40ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:21.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45d3fff700 1 --2- 172.21.15.189:0/4186408971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45d415ce60 0x7f45d40ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:21.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45d3fff700 1 -- 172.21.15.189:0/4186408971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d40cc630 con 0x7f45d40b0c80 2024-04-06T12:13:21.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45d9350700 1 --2- 172.21.15.189:0/4186408971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45d415ce60 0x7f45d40ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:21.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45d3fff700 1 --2- 172.21.15.189:0/4186408971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45d40b0c80 0x7f45d415c920 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f45c000e4b0 tx=0x7f45c00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:21.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45d17fa700 1 -- 172.21.15.189:0/4186408971 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0021070 con 0x7f45d40b0c80 2024-04-06T12:13:21.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45d17fa700 1 -- 172.21.15.189:0/4186408971 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c00045c0 con 0x7f45d40b0c80 2024-04-06T12:13:21.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45dadb3700 1 -- 172.21.15.189:0/4186408971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d40cc910 con 0x7f45d40b0c80 2024-04-06T12:13:21.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45d17fa700 1 -- 172.21.15.189:0/4186408971 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c000e6c0 con 0x7f45d40b0c80 2024-04-06T12:13:21.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.795+0000 7f45dadb3700 1 -- 172.21.15.189:0/4186408971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d40cced0 con 0x7f45d40b0c80 2024-04-06T12:13:21.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.796+0000 7f45dadb3700 1 -- 172.21.15.189:0/4186408971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d40cd3d0 con 0x7f45d40b0c80 2024-04-06T12:13:21.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.797+0000 7f45d17fa700 1 -- 172.21.15.189:0/4186408971 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f45c0025070 con 0x7f45d40b0c80 2024-04-06T12:13:21.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.797+0000 7f45d17fa700 1 --2- 172.21.15.189:0/4186408971 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f45b806ac90 0x7f45b806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:21.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.797+0000 7f45d17fa700 1 -- 172.21.15.189:0/4186408971 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f45c008a410 con 0x7f45d40b0c80 2024-04-06T12:13:21.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.797+0000 7f45d8b4f700 1 --2- 172.21.15.189:0/4186408971 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f45b806ac90 0x7f45b806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:21.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.798+0000 7f45d8b4f700 1 --2- 172.21.15.189:0/4186408971 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f45b806ac90 0x7f45b806d150 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f45c8009ce0 tx=0x7f45c8009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:21.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.802+0000 7f45d17fa700 1 -- 172.21.15.189:0/4186408971 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f45c0055660 con 0x7f45d40b0c80 2024-04-06T12:13:21.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.997+0000 7f45dadb3700 1 -- 172.21.15.189:0/4186408971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45d4067bb0 con 0x7f45d40b0c80 2024-04-06T12:13:21.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:21.998+0000 7f45d17fa700 1 -- 172.21.15.189:0/4186408971 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16367 (secure 0 0 0) 0x7f45c0058c80 con 0x7f45d40b0c80 2024-04-06T12:13:21.996 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:13:21.997 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":38,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:13:21.287596+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:13:19.268280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-06T12:13:20.243771+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:13:21.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.000+0000 7f45dadb3700 1 -- 172.21.15.189:0/4186408971 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f45b806ac90 msgr2=0x7f45b806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:21.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.000+0000 7f45dadb3700 1 --2- 172.21.15.189:0/4186408971 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f45b806ac90 0x7f45b806d150 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f45c8009ce0 tx=0x7f45c8009210 comp rx=0 tx=0).stop 2024-04-06T12:13:21.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.000+0000 7f45dadb3700 1 -- 172.21.15.189:0/4186408971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45d40b0c80 msgr2=0x7f45d415c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:22.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.000+0000 7f45dadb3700 1 --2- 172.21.15.189:0/4186408971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45d40b0c80 0x7f45d415c920 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f45c000e4b0 tx=0x7f45c00047c0 comp rx=0 tx=0).stop 2024-04-06T12:13:22.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.000+0000 7f45dadb3700 1 -- 172.21.15.189:0/4186408971 shutdown_connections 2024-04-06T12:13:22.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.000+0000 7f45dadb3700 1 --2- 172.21.15.189:0/4186408971 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f45b806ac90 0x7f45b806d150 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:22.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.000+0000 7f45dadb3700 1 --2- 172.21.15.189:0/4186408971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45d415ce60 0x7f45d40ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:22.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.000+0000 7f45dadb3700 1 --2- 172.21.15.189:0/4186408971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45c4003fd0 0x7f45d41f6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:22.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.000+0000 7f45dadb3700 1 --2- 172.21.15.189:0/4186408971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45d40b0c80 0x7f45d415c920 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:22.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.000+0000 7f45dadb3700 1 -- 172.21.15.189:0/4186408971 >> 172.21.15.189:0/4186408971 conn(0x7f45d40d00b0 msgr2=0x7f45d406f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:22.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.000+0000 7f45dadb3700 1 -- 172.21.15.189:0/4186408971 shutdown_connections 2024-04-06T12:13:22.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.000+0000 7f45dadb3700 1 -- 172.21.15.189:0/4186408971 wait complete. 2024-04-06T12:13:22.012 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-06T12:13:22.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.137+0000 7f6a28beb700 1 Processor -- start 2024-04-06T12:13:22.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.137+0000 7f6a28beb700 1 -- start start 2024-04-06T12:13:22.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.138+0000 7f6a28beb700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6a240b3e20 0x7f6a2415e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:22.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.138+0000 7f6a28beb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a240b3990 con 0x7f6a2415d7b0 2024-04-06T12:13:22.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.138+0000 7f6a28beb700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a240b3060 con 0x7f6a2406ae40 2024-04-06T12:13:22.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.138+0000 7f6a28beb700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a240b4230 con 0x7f6a240b3e20 2024-04-06T12:13:22.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a2259c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6a2415d7b0 0x7f6a2406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33490/0 (socket says 172.21.15.189:33490) 2024-04-06T12:13:22.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a2259c700 1 -- 172.21.15.189:0/1321795505 learned_addr learned my addr 172.21.15.189:0/1321795505 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:22.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a2159a700 1 -- 172.21.15.189:0/1321795505 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397741519 0 0) 0x7f6a240b3990 con 0x7f6a2415d7b0 2024-04-06T12:13:22.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a22d9d700 1 --2- 172.21.15.189:0/1321795505 >> v2:172.21.15.192:3300/0 conn(0x7f6a240b3e20 0x7f6a2415e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:22.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a2159a700 1 -- 172.21.15.189:0/1321795505 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a100035b0 con 0x7f6a2415d7b0 2024-04-06T12:13:22.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a2159a700 1 -- 172.21.15.189:0/1321795505 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3724502607 0 0) 0x7f6a100035b0 con 0x7f6a2415d7b0 2024-04-06T12:13:22.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a2159a700 1 -- 172.21.15.189:0/1321795505 >> v2:172.21.15.192:3300/0 conn(0x7f6a240b3e20 msgr2=0x7f6a2415e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a2159a700 1 --2- 172.21.15.189:0/1321795505 >> v2:172.21.15.192:3300/0 conn(0x7f6a240b3e20 0x7f6a2415e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a2159a700 1 -- 172.21.15.189:0/1321795505 >> v1:172.21.15.192:6789/0 conn(0x7f6a2406ae40 legacy=0x7f6a2415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a2159a700 1 -- 172.21.15.189:0/1321795505 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a240b3340 con 0x7f6a2415d7b0 2024-04-06T12:13:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a2159a700 1 -- 172.21.15.189:0/1321795505 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f6a14001670 con 0x7f6a2415d7b0 2024-04-06T12:13:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a2159a700 1 -- 172.21.15.189:0/1321795505 >> v1:172.21.15.189:6789/0 conn(0x7f6a2415d7b0 legacy=0x7f6a2406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a2159a700 1 --2- 172.21.15.189:0/1321795505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a10003a00 0x7f6a10003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a2159a700 1 --2- 172.21.15.189:0/1321795505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a10004850 0x7f6a10004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a2159a700 1 --2- 172.21.15.189:0/1321795505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a10005210 0x7f6a1000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a2159a700 1 -- 172.21.15.189:0/1321795505 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a240b4230 con 0x7f6a10004850 2024-04-06T12:13:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a2159a700 1 -- 172.21.15.189:0/1321795505 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1000c1b0 con 0x7f6a10003a00 2024-04-06T12:13:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a2159a700 1 -- 172.21.15.189:0/1321795505 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1000c370 con 0x7f6a10005210 2024-04-06T12:13:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a21d9b700 1 --2- 172.21.15.189:0/1321795505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a10004850 0x7f6a10004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a2259c700 1 --2- 172.21.15.189:0/1321795505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a10003a00 0x7f6a10003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.139+0000 7f6a22d9d700 1 --2- 172.21.15.189:0/1321795505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a10005210 0x7f6a1000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.140+0000 7f6a21d9b700 1 -- 172.21.15.189:0/1321795505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a10005210 msgr2=0x7f6a1000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.140+0000 7f6a21d9b700 1 --2- 172.21.15.189:0/1321795505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a10005210 0x7f6a1000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.140+0000 7f6a21d9b700 1 -- 172.21.15.189:0/1321795505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a10003a00 msgr2=0x7f6a10003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.140+0000 7f6a21d9b700 1 --2- 172.21.15.189:0/1321795505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a10003a00 0x7f6a10003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.140+0000 7f6a21d9b700 1 -- 172.21.15.189:0/1321795505 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a100035b0 con 0x7f6a10004850 2024-04-06T12:13:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.140+0000 7f6a22d9d700 1 -- 172.21.15.189:0/1321795505 reap_dead start 2024-04-06T12:13:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.140+0000 7f6a21d9b700 1 --2- 172.21.15.189:0/1321795505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a10004850 0x7f6a10004cd0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0017c0 tx=0x7f6a10005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d10d35a0304857b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.140+0000 7f6a2159a700 1 -- 172.21.15.189:0/1321795505 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c0052d0 con 0x7f6a10004850 2024-04-06T12:13:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.140+0000 7f6a2159a700 1 -- 172.21.15.189:0/1321795505 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0c005430 con 0x7f6a10004850 2024-04-06T12:13:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.140+0000 7f6a2159a700 1 -- 172.21.15.189:0/1321795505 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c0038c0 con 0x7f6a10004850 2024-04-06T12:13:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.140+0000 7f6a28beb700 1 -- 172.21.15.189:0/1321795505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a10004850 msgr2=0x7f6a10004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.140+0000 7f6a28beb700 1 --2- 172.21.15.189:0/1321795505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a10004850 0x7f6a10004cd0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0017c0 tx=0x7f6a10005880 comp rx=0 tx=0).stop 2024-04-06T12:13:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.140+0000 7f6a28beb700 1 -- 172.21.15.189:0/1321795505 shutdown_connections 2024-04-06T12:13:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.140+0000 7f6a28beb700 1 --2- 172.21.15.189:0/1321795505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a10004850 0x7f6a10004cd0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.141+0000 7f6a28beb700 1 -- 172.21.15.189:0/1321795505 >> 172.21.15.189:0/1321795505 conn(0x7f6a240d00b0 msgr2=0x7f6a240d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:22.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.141+0000 7f6a28beb700 1 -- 172.21.15.189:0/1321795505 shutdown_connections 2024-04-06T12:13:22.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.141+0000 7f6a28beb700 1 -- 172.21.15.189:0/1321795505 wait complete. 2024-04-06T12:13:22.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.142+0000 7f6a28beb700 1 Processor -- start 2024-04-06T12:13:22.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.142+0000 7f6a28beb700 1 -- start start 2024-04-06T12:13:22.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a28beb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a10004850 0x7f6a241f40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:22.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a28beb700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a2414bdb0 0x7f6a240c5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:22.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a28beb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a240aeb80 0x7f6a240acf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:22.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a28beb700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2414c230 con 0x7f6a240aeb80 2024-04-06T12:13:22.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a28beb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a241f4760 con 0x7f6a10004850 2024-04-06T12:13:22.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a28beb700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a241f4a40 con 0x7f6a2414bdb0 2024-04-06T12:13:22.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a22d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a240aeb80 0x7f6a240acf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:22.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a22d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a240aeb80 0x7f6a240acf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33094/0 (socket says 172.21.15.189:33094) 2024-04-06T12:13:22.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a22d9d700 1 -- 172.21.15.189:0/257296705 learned_addr learned my addr 172.21.15.189:0/257296705 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:22.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a2259c700 1 --2- 172.21.15.189:0/257296705 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a10004850 0x7f6a241f40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:22.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a21d9b700 1 --2- 172.21.15.189:0/257296705 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a2414bdb0 0x7f6a240c5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:22.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a22d9d700 1 -- 172.21.15.189:0/257296705 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a2414bdb0 msgr2=0x7f6a240c5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:22.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a22d9d700 1 --2- 172.21.15.189:0/257296705 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a2414bdb0 0x7f6a240c5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:22.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a22d9d700 1 -- 172.21.15.189:0/257296705 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a10004850 msgr2=0x7f6a241f40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:22.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a22d9d700 1 --2- 172.21.15.189:0/257296705 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a10004850 0x7f6a241f40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:22.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a22d9d700 1 -- 172.21.15.189:0/257296705 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a241f4d20 con 0x7f6a240aeb80 2024-04-06T12:13:22.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a22d9d700 1 --2- 172.21.15.189:0/257296705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a240aeb80 0x7f6a240acf30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6a24072eb0 tx=0x7f6a1c0111d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:22.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a1affd700 1 -- 172.21.15.189:0/257296705 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c00e070 con 0x7f6a240aeb80 2024-04-06T12:13:22.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a1affd700 1 -- 172.21.15.189:0/257296705 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1c003cf0 con 0x7f6a240aeb80 2024-04-06T12:13:22.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a1affd700 1 -- 172.21.15.189:0/257296705 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c011740 con 0x7f6a240aeb80 2024-04-06T12:13:22.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a28beb700 1 -- 172.21.15.189:0/257296705 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a241f5000 con 0x7f6a240aeb80 2024-04-06T12:13:22.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.143+0000 7f6a28beb700 1 -- 172.21.15.189:0/257296705 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a240af010 con 0x7f6a240aeb80 2024-04-06T12:13:22.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.144+0000 7f6a28beb700 1 -- 172.21.15.189:0/257296705 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a241f5520 con 0x7f6a240aeb80 2024-04-06T12:13:22.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.145+0000 7f6a1affd700 1 -- 172.21.15.189:0/257296705 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6a1c0118a0 con 0x7f6a240aeb80 2024-04-06T12:13:22.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.145+0000 7f6a1affd700 1 --2- 172.21.15.189:0/257296705 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6a0406ac90 0x7f6a0406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:22.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.146+0000 7f6a2259c700 1 --2- 172.21.15.189:0/257296705 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6a0406ac90 0x7f6a0406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:22.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.146+0000 7f6a1affd700 1 -- 172.21.15.189:0/257296705 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6a1c089960 con 0x7f6a240aeb80 2024-04-06T12:13:22.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.146+0000 7f6a2259c700 1 --2- 172.21.15.189:0/257296705 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6a0406ac90 0x7f6a0406d150 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6a24160ea0 tx=0x7f6a14000ed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:22.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.150+0000 7f6a1affd700 1 -- 172.21.15.189:0/257296705 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6a241f5520 con 0x7f6a240aeb80 2024-04-06T12:13:22.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:22.390+0000 7f6a28beb700 1 -- 172.21.15.189:0/257296705 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6a241f5520 con 0x7f6a240aeb80 2024-04-06T12:13:23.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.297+0000 7f6a1affd700 1 -- 172.21.15.189:0/257296705 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6a241f5520 con 0x7f6a240aeb80 2024-04-06T12:13:23.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.299+0000 7f6a28beb700 1 -- 172.21.15.189:0/257296705 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6a0406ac90 msgr2=0x7f6a0406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:23.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.300+0000 7f6a28beb700 1 --2- 172.21.15.189:0/257296705 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6a0406ac90 0x7f6a0406d150 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6a24160ea0 tx=0x7f6a14000ed0 comp rx=0 tx=0).stop 2024-04-06T12:13:23.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.300+0000 7f6a28beb700 1 -- 172.21.15.189:0/257296705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a240aeb80 msgr2=0x7f6a240acf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:23.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.300+0000 7f6a28beb700 1 --2- 172.21.15.189:0/257296705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a240aeb80 0x7f6a240acf30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6a24072eb0 tx=0x7f6a1c0111d0 comp rx=0 tx=0).stop 2024-04-06T12:13:23.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.300+0000 7f6a28beb700 1 -- 172.21.15.189:0/257296705 shutdown_connections 2024-04-06T12:13:23.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.300+0000 7f6a28beb700 1 --2- 172.21.15.189:0/257296705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a240aeb80 0x7f6a240acf30 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:23.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.300+0000 7f6a28beb700 1 --2- 172.21.15.189:0/257296705 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6a0406ac90 0x7f6a0406d150 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:23.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.300+0000 7f6a28beb700 1 --2- 172.21.15.189:0/257296705 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a10004850 0x7f6a241f40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:23.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.300+0000 7f6a28beb700 1 --2- 172.21.15.189:0/257296705 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a2414bdb0 0x7f6a240c5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:23.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.300+0000 7f6a28beb700 1 -- 172.21.15.189:0/257296705 >> 172.21.15.189:0/257296705 conn(0x7f6a240d00b0 msgr2=0x7f6a240d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:23.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.300+0000 7f6a28beb700 1 -- 172.21.15.189:0/257296705 shutdown_connections 2024-04-06T12:13:23.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.300+0000 7f6a28beb700 1 -- 172.21.15.189:0/257296705 wait complete. 2024-04-06T12:13:23.312 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-06T12:13:23.313 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-06T12:13:23.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.446+0000 7f067ebb2700 1 Processor -- start 2024-04-06T12:13:23.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.446+0000 7f067ebb2700 1 -- start start 2024-04-06T12:13:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.447+0000 7f067ebb2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06780d11b0 0x7f06780d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.447+0000 7f067ebb2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06780cb870 0x7f067815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.447+0000 7f067ebb2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f06780cfa00 0x7f06780cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.447+0000 7f067ebb2700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06780ccdc0 con 0x7f06780cb870 2024-04-06T12:13:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.447+0000 7f067ebb2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06780d0b50 con 0x7f06780cfa00 2024-04-06T12:13:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.447+0000 7f067ebb2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06780d1ad0 con 0x7f06780d11b0 2024-04-06T12:13:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.447+0000 7f0677fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06780cb870 0x7f067815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.447+0000 7f0677fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06780cb870 0x7f067815c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33106/0 (socket says 172.21.15.189:33106) 2024-04-06T12:13:23.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.447+0000 7f0677fff700 1 -- 172.21.15.189:0/39239474 learned_addr learned my addr 172.21.15.189:0/39239474 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:23.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.448+0000 7f067d14f700 1 --2- 172.21.15.189:0/39239474 >> v2:172.21.15.192:3300/0 conn(0x7f06780cfa00 0x7f06780cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:23.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.448+0000 7f067c94e700 1 --2- 172.21.15.189:0/39239474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06780d11b0 0x7f06780d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:23.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.448+0000 7f0677fff700 1 -- 172.21.15.189:0/39239474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06780d11b0 msgr2=0x7f06780d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:23.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.448+0000 7f0677fff700 1 --2- 172.21.15.189:0/39239474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06780d11b0 0x7f06780d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:23.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.448+0000 7f0677fff700 1 -- 172.21.15.189:0/39239474 >> v2:172.21.15.192:3300/0 conn(0x7f06780cfa00 msgr2=0x7f06780cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:23.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.448+0000 7f0677fff700 1 --2- 172.21.15.189:0/39239474 >> v2:172.21.15.192:3300/0 conn(0x7f06780cfa00 0x7f06780cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:23.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.448+0000 7f0677fff700 1 -- 172.21.15.189:0/39239474 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06780cbd80 con 0x7f06780cb870 2024-04-06T12:13:23.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.448+0000 7f0677fff700 1 --2- 172.21.15.189:0/39239474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06780cb870 0x7f067815c280 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f06780b0990 tx=0x7f0664009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4056fbf71c1582fe server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:23.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.448+0000 7f06777fe700 1 -- 172.21.15.189:0/39239474 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066400b870 con 0x7f06780cb870 2024-04-06T12:13:23.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.448+0000 7f06777fe700 1 -- 172.21.15.189:0/39239474 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066400b9d0 con 0x7f06780cb870 2024-04-06T12:13:23.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.448+0000 7f06777fe700 1 -- 172.21.15.189:0/39239474 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066400bd60 con 0x7f06780cb870 2024-04-06T12:13:23.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.449+0000 7f067ebb2700 1 -- 172.21.15.189:0/39239474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06780cb870 msgr2=0x7f067815c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:23.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.449+0000 7f067ebb2700 1 --2- 172.21.15.189:0/39239474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06780cb870 0x7f067815c280 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f06780b0990 tx=0x7f0664009bd0 comp rx=0 tx=0).stop 2024-04-06T12:13:23.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.449+0000 7f067ebb2700 1 -- 172.21.15.189:0/39239474 shutdown_connections 2024-04-06T12:13:23.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.449+0000 7f067ebb2700 1 --2- 172.21.15.189:0/39239474 >> v2:172.21.15.192:3300/0 conn(0x7f06780cfa00 0x7f06780cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:23.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.449+0000 7f067ebb2700 1 --2- 172.21.15.189:0/39239474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06780d11b0 0x7f06780d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:23.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.449+0000 7f067ebb2700 1 --2- 172.21.15.189:0/39239474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06780cb870 0x7f067815c280 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:23.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.449+0000 7f067ebb2700 1 -- 172.21.15.189:0/39239474 >> 172.21.15.189:0/39239474 conn(0x7f06780b2db0 msgr2=0x7f06780b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:23.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.449+0000 7f067ebb2700 1 -- 172.21.15.189:0/39239474 shutdown_connections 2024-04-06T12:13:23.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.449+0000 7f067ebb2700 1 -- 172.21.15.189:0/39239474 wait complete. 2024-04-06T12:13:23.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.450+0000 7f067ebb2700 1 Processor -- start 2024-04-06T12:13:23.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.450+0000 7f067ebb2700 1 -- start start 2024-04-06T12:13:23.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f067ebb2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06780cb870 0x7f06781586c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:23.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f067ebb2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06780cfa00 0x7f06780b5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:23.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f067ebb2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06780d11b0 0x7f06780b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:23.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f067ebb2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0678152f80 con 0x7f06780d11b0 2024-04-06T12:13:23.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f067ebb2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0678158230 con 0x7f06780cb870 2024-04-06T12:13:23.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f067ebb2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0678156f90 con 0x7f06780cfa00 2024-04-06T12:13:23.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f067d14f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06780d11b0 0x7f06780b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:23.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f067d14f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06780d11b0 0x7f06780b59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33116/0 (socket says 172.21.15.189:33116) 2024-04-06T12:13:23.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f067d14f700 1 -- 172.21.15.189:0/1066548732 learned_addr learned my addr 172.21.15.189:0/1066548732 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f067c94e700 1 --2- 172.21.15.189:0/1066548732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06780cb870 0x7f06781586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f0677fff700 1 --2- 172.21.15.189:0/1066548732 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06780cfa00 0x7f06780b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f067d14f700 1 -- 172.21.15.189:0/1066548732 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06780cfa00 msgr2=0x7f06780b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f067d14f700 1 --2- 172.21.15.189:0/1066548732 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06780cfa00 0x7f06780b5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f067d14f700 1 -- 172.21.15.189:0/1066548732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06780cb870 msgr2=0x7f06781586c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f067d14f700 1 --2- 172.21.15.189:0/1066548732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06780cb870 0x7f06781586c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f067d14f700 1 -- 172.21.15.189:0/1066548732 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0664009550 con 0x7f06780d11b0 2024-04-06T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f067d14f700 1 --2- 172.21.15.189:0/1066548732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06780d11b0 0x7f06780b59d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f067000ccf0 tx=0x7f067000a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f06757fa700 1 -- 172.21.15.189:0/1066548732 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067000ad60 con 0x7f06780d11b0 2024-04-06T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f06757fa700 1 -- 172.21.15.189:0/1066548732 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0670003a20 con 0x7f06780d11b0 2024-04-06T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.451+0000 7f06757fa700 1 -- 172.21.15.189:0/1066548732 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0670003db0 con 0x7f06780d11b0 2024-04-06T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.452+0000 7f067ebb2700 1 -- 172.21.15.189:0/1066548732 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0678156cb0 con 0x7f06780d11b0 2024-04-06T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.452+0000 7f067ebb2700 1 -- 172.21.15.189:0/1066548732 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0678157ec0 con 0x7f06780d11b0 2024-04-06T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.453+0000 7f067ebb2700 1 -- 172.21.15.189:0/1066548732 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06781583a0 con 0x7f06780d11b0 2024-04-06T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.453+0000 7f06757fa700 1 -- 172.21.15.189:0/1066548732 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0670003f50 con 0x7f06780d11b0 2024-04-06T12:13:23.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.453+0000 7f06757fa700 1 --2- 172.21.15.189:0/1066548732 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f065c06ac90 0x7f065c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:23.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.453+0000 7f06757fa700 1 -- 172.21.15.189:0/1066548732 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0670012070 con 0x7f06780d11b0 2024-04-06T12:13:23.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.453+0000 7f067c94e700 1 --2- 172.21.15.189:0/1066548732 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f065c06ac90 0x7f065c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:23.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.454+0000 7f067c94e700 1 --2- 172.21.15.189:0/1066548732 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f065c06ac90 0x7f065c06d150 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f066c006fd0 tx=0x7f066c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:23.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.458+0000 7f06757fa700 1 -- 172.21.15.189:0/1066548732 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0670054b50 con 0x7f06780d11b0 2024-04-06T12:13:23.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:23.698+0000 7f067ebb2700 1 -- 172.21.15.189:0/1066548732 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f0678074990 con 0x7f06780d11b0 2024-04-06T12:13:24.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.298+0000 7f06757fa700 1 -- 172.21.15.189:0/1066548732 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0670058170 con 0x7f06780d11b0 2024-04-06T12:13:24.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.300+0000 7f067ebb2700 1 -- 172.21.15.189:0/1066548732 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f065c06ac90 msgr2=0x7f065c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:24.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.300+0000 7f067ebb2700 1 --2- 172.21.15.189:0/1066548732 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f065c06ac90 0x7f065c06d150 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f066c006fd0 tx=0x7f066c008040 comp rx=0 tx=0).stop 2024-04-06T12:13:24.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.300+0000 7f067ebb2700 1 -- 172.21.15.189:0/1066548732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06780d11b0 msgr2=0x7f06780b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:24.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.300+0000 7f067ebb2700 1 --2- 172.21.15.189:0/1066548732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06780d11b0 0x7f06780b59d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f067000ccf0 tx=0x7f067000a290 comp rx=0 tx=0).stop 2024-04-06T12:13:24.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.301+0000 7f067ebb2700 1 -- 172.21.15.189:0/1066548732 shutdown_connections 2024-04-06T12:13:24.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.301+0000 7f067ebb2700 1 --2- 172.21.15.189:0/1066548732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06780d11b0 0x7f06780b59d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:24.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.301+0000 7f067ebb2700 1 --2- 172.21.15.189:0/1066548732 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f065c06ac90 0x7f065c06d150 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:24.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.301+0000 7f067ebb2700 1 --2- 172.21.15.189:0/1066548732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06780cb870 0x7f06781586c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:24.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.301+0000 7f067ebb2700 1 --2- 172.21.15.189:0/1066548732 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06780cfa00 0x7f06780b5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:24.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.301+0000 7f067ebb2700 1 -- 172.21.15.189:0/1066548732 >> 172.21.15.189:0/1066548732 conn(0x7f06780b2db0 msgr2=0x7f06780b1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:24.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.301+0000 7f067ebb2700 1 -- 172.21.15.189:0/1066548732 shutdown_connections 2024-04-06T12:13:24.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.301+0000 7f067ebb2700 1 -- 172.21.15.189:0/1066548732 wait complete. 2024-04-06T12:13:24.312 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-06T12:13:24.313 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:13:24.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.439+0000 7ffba3e55700 1 Processor -- start 2024-04-06T12:13:24.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.439+0000 7ffba3e55700 1 -- start start 2024-04-06T12:13:24.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.440+0000 7ffba3e55700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb9c144f40 0x7ffb9c06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:24.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.440+0000 7ffba3e55700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffb9c15fcd0 0x7ffb9c06ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:24.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.440+0000 7ffba3e55700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb9c15c740 con 0x7ffb9c15ba40 2024-04-06T12:13:24.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.440+0000 7ffba3e55700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb9c15c560 con 0x7ffb9c15fcd0 2024-04-06T12:13:24.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.440+0000 7ffba3e55700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c0cfcb0 con 0x7ffb9c144f40 2024-04-06T12:13:24.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.440+0000 7ffba23f2700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ffb9c15ba40 0x7ffb9c15be50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33494/0 (socket says 172.21.15.189:33494) 2024-04-06T12:13:24.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.440+0000 7ffba13f0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffb9c15fcd0 0x7ffb9c06ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:24.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.440+0000 7ffba23f2700 1 -- 172.21.15.189:0/3573884284 learned_addr learned my addr 172.21.15.189:0/3573884284 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:24.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.440+0000 7ffba1bf1700 1 --2- 172.21.15.189:0/3573884284 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb9c144f40 0x7ffb9c06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:24.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.440+0000 7ffba13f0700 1 -- 172.21.15.189:0/3573884284 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb9c144f40 msgr2=0x7ffb9c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:24.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.440+0000 7ffba13f0700 1 --2- 172.21.15.189:0/3573884284 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb9c144f40 0x7ffb9c06a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:24.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.440+0000 7ffba13f0700 1 -- 172.21.15.189:0/3573884284 >> v1:172.21.15.189:6789/0 conn(0x7ffb9c15ba40 legacy=0x7ffb9c15be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:24.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.440+0000 7ffba13f0700 1 -- 172.21.15.189:0/3573884284 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9c15f1f0 con 0x7ffb9c15fcd0 2024-04-06T12:13:24.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.440+0000 7ffba0bef700 1 -- 172.21.15.189:0/3573884284 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722914382 0 0) 0x7ffb9c15c740 con 0x7ffb9c15ba40 2024-04-06T12:13:24.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.441+0000 7ffba13f0700 1 --2- 172.21.15.189:0/3573884284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb9c15fcd0 0x7ffb9c06ac70 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009840 tx=0x7ffb8c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=232ea9630c2b4701 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:24.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.441+0000 7ffba0bef700 1 -- 172.21.15.189:0/3573884284 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c011830 con 0x7ffb9c15fcd0 2024-04-06T12:13:24.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.441+0000 7ffba0bef700 1 -- 172.21.15.189:0/3573884284 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb8c011990 con 0x7ffb9c15fcd0 2024-04-06T12:13:24.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.441+0000 7ffba0bef700 1 -- 172.21.15.189:0/3573884284 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c011d20 con 0x7ffb9c15fcd0 2024-04-06T12:13:24.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.441+0000 7ffba3e55700 1 -- 172.21.15.189:0/3573884284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb9c15fcd0 msgr2=0x7ffb9c06ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:24.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.441+0000 7ffba3e55700 1 --2- 172.21.15.189:0/3573884284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb9c15fcd0 0x7ffb9c06ac70 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009840 tx=0x7ffb8c009b50 comp rx=0 tx=0).stop 2024-04-06T12:13:24.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.441+0000 7ffba3e55700 1 -- 172.21.15.189:0/3573884284 shutdown_connections 2024-04-06T12:13:24.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.441+0000 7ffba3e55700 1 --2- 172.21.15.189:0/3573884284 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb9c144f40 0x7ffb9c06a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:24.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.441+0000 7ffba3e55700 1 --2- 172.21.15.189:0/3573884284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb9c15fcd0 0x7ffb9c06ac70 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:24.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.441+0000 7ffba3e55700 1 -- 172.21.15.189:0/3573884284 >> 172.21.15.189:0/3573884284 conn(0x7ffb9c0d00b0 msgr2=0x7ffb9c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:24.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.441+0000 7ffba3e55700 1 -- 172.21.15.189:0/3573884284 shutdown_connections 2024-04-06T12:13:24.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.441+0000 7ffba3e55700 1 -- 172.21.15.189:0/3573884284 wait complete. 2024-04-06T12:13:24.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.443+0000 7ffba3e55700 1 Processor -- start 2024-04-06T12:13:24.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.443+0000 7ffba3e55700 1 -- start start 2024-04-06T12:13:24.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.443+0000 7ffba3e55700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb9c144f40 0x7ffb9c13ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:24.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.443+0000 7ffba3e55700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb9c15ba40 0x7ffb9c06f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:24.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.443+0000 7ffba3e55700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb9c15fcd0 0x7ffb9c06fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:24.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.443+0000 7ffba3e55700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c0c5940 con 0x7ffb9c144f40 2024-04-06T12:13:24.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.443+0000 7ffba3e55700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c13f100 con 0x7ffb9c15ba40 2024-04-06T12:13:24.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.443+0000 7ffba3e55700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c159e30 con 0x7ffb9c15fcd0 2024-04-06T12:13:24.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.443+0000 7ffba1bf1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb9c144f40 0x7ffb9c13ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:24.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.443+0000 7ffba1bf1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb9c144f40 0x7ffb9c13ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33138/0 (socket says 172.21.15.189:33138) 2024-04-06T12:13:24.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.443+0000 7ffba1bf1700 1 -- 172.21.15.189:0/3621472323 learned_addr learned my addr 172.21.15.189:0/3621472323 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:24.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.444+0000 7ffba23f2700 1 --2- 172.21.15.189:0/3621472323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb9c15fcd0 0x7ffb9c06fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:24.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.444+0000 7ffba1bf1700 1 -- 172.21.15.189:0/3621472323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb9c15fcd0 msgr2=0x7ffb9c06fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:24.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.444+0000 7ffba1bf1700 1 --2- 172.21.15.189:0/3621472323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb9c15fcd0 0x7ffb9c06fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:24.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.444+0000 7ffba1bf1700 1 -- 172.21.15.189:0/3621472323 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb9c15ba40 msgr2=0x7ffb9c06f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:24.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.444+0000 7ffba13f0700 1 --2- 172.21.15.189:0/3621472323 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb9c15ba40 0x7ffb9c06f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:24.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.444+0000 7ffba1bf1700 1 --2- 172.21.15.189:0/3621472323 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb9c15ba40 0x7ffb9c06f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:24.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.444+0000 7ffba1bf1700 1 -- 172.21.15.189:0/3621472323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb8c0094a0 con 0x7ffb9c144f40 2024-04-06T12:13:24.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.444+0000 7ffba1bf1700 1 --2- 172.21.15.189:0/3621472323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb9c144f40 0x7ffb9c13ebc0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ffb9800e810 tx=0x7ffb9800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:24.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.444+0000 7ffb927fc700 1 -- 172.21.15.189:0/3621472323 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb98013610 con 0x7ffb9c144f40 2024-04-06T12:13:24.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.444+0000 7ffb927fc700 1 -- 172.21.15.189:0/3621472323 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb98013770 con 0x7ffb9c144f40 2024-04-06T12:13:24.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.444+0000 7ffba3e55700 1 -- 172.21.15.189:0/3621472323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb9c15a110 con 0x7ffb9c144f40 2024-04-06T12:13:24.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.444+0000 7ffb927fc700 1 -- 172.21.15.189:0/3621472323 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb98013b20 con 0x7ffb9c144f40 2024-04-06T12:13:24.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.444+0000 7ffba3e55700 1 -- 172.21.15.189:0/3621472323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb9c15a6d0 con 0x7ffb9c144f40 2024-04-06T12:13:24.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.445+0000 7ffba3e55700 1 -- 172.21.15.189:0/3621472323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb9c067bb0 con 0x7ffb9c144f40 2024-04-06T12:13:24.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.445+0000 7ffb927fc700 1 -- 172.21.15.189:0/3621472323 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ffb98013c80 con 0x7ffb9c144f40 2024-04-06T12:13:24.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.446+0000 7ffb927fc700 1 --2- 172.21.15.189:0/3621472323 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb8006ac90 0x7ffb8006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:24.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.446+0000 7ffb927fc700 1 -- 172.21.15.189:0/3621472323 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ffb980890e0 con 0x7ffb9c144f40 2024-04-06T12:13:24.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.446+0000 7ffba13f0700 1 --2- 172.21.15.189:0/3621472323 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb8006ac90 0x7ffb8006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:24.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.446+0000 7ffba13f0700 1 --2- 172.21.15.189:0/3621472323 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb8006ac90 0x7ffb8006d150 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c0d0c70 tx=0x7ffb8c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:24.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.450+0000 7ffb927fc700 1 -- 172.21.15.189:0/3621472323 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ffb98054190 con 0x7ffb9c144f40 2024-04-06T12:13:24.653 INFO:tasks.ceph.mds.c.smithi189.stderr:2024-04-06T12:13:24.655+0000 7f82a1b2b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T12:13:24.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.689+0000 7ffba3e55700 1 -- 172.21.15.189:0/3621472323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb9c074990 con 0x7ffb9c144f40 2024-04-06T12:13:24.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.690+0000 7ffb927fc700 1 -- 172.21.15.189:0/3621472323 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4740 (secure 0 0 0) 0x7ffb980577b0 con 0x7ffb9c144f40 2024-04-06T12:13:24.688 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:13:24.688 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4865,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/2279772906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2279772906},{"type":"v1","addr":"172.21.15.192:6834","nonce":2279772906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4868,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/454939556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":454939556},{"type":"v1","addr":"172.21.15.192:6835","nonce":454939556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5107,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/399541295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":399541295},{"type":"v1","addr":"172.21.15.189:6837","nonce":399541295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:13:21.287908+0000","modified":"2024-04-06T12:13:24.295625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5232":{"gid":5232,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/1024118918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1024118918},{"type":"v1","addr":"172.21.15.189:6836","nonce":1024118918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-04-06T12:13:24.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.692+0000 7ffba3e55700 1 -- 172.21.15.189:0/3621472323 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb8006ac90 msgr2=0x7ffb8006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:24.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.692+0000 7ffba3e55700 1 --2- 172.21.15.189:0/3621472323 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb8006ac90 0x7ffb8006d150 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c0d0c70 tx=0x7ffb8c008030 comp rx=0 tx=0).stop 2024-04-06T12:13:24.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.692+0000 7ffba3e55700 1 -- 172.21.15.189:0/3621472323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb9c144f40 msgr2=0x7ffb9c13ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:24.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.692+0000 7ffba3e55700 1 --2- 172.21.15.189:0/3621472323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb9c144f40 0x7ffb9c13ebc0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ffb9800e810 tx=0x7ffb9800ebd0 comp rx=0 tx=0).stop 2024-04-06T12:13:24.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.692+0000 7ffba3e55700 1 -- 172.21.15.189:0/3621472323 shutdown_connections 2024-04-06T12:13:24.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.692+0000 7ffba3e55700 1 --2- 172.21.15.189:0/3621472323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb9c15fcd0 0x7ffb9c06fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:24.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.692+0000 7ffba3e55700 1 --2- 172.21.15.189:0/3621472323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb9c144f40 0x7ffb9c13ebc0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:24.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.692+0000 7ffba3e55700 1 --2- 172.21.15.189:0/3621472323 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb8006ac90 0x7ffb8006d150 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:24.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.692+0000 7ffba3e55700 1 --2- 172.21.15.189:0/3621472323 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb9c15ba40 0x7ffb9c06f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:24.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.692+0000 7ffba3e55700 1 -- 172.21.15.189:0/3621472323 >> 172.21.15.189:0/3621472323 conn(0x7ffb9c0d00b0 msgr2=0x7ffb9c06ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:24.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.692+0000 7ffba3e55700 1 -- 172.21.15.189:0/3621472323 shutdown_connections 2024-04-06T12:13:24.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.692+0000 7ffba3e55700 1 -- 172.21.15.189:0/3621472323 wait complete. 2024-04-06T12:13:24.691 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 16 2024-04-06T12:13:24.702 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:13:24.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.825+0000 7fe55c620700 1 Processor -- start 2024-04-06T12:13:24.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.826+0000 7fe55c620700 1 -- start start 2024-04-06T12:13:24.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.826+0000 7fe55c620700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe55406ae40 0x7fe5540b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:24.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.826+0000 7fe55c620700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe55415ddf0 0x7fe55415e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:24.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.826+0000 7fe55c620700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5540b3990 con 0x7fe55415d7b0 2024-04-06T12:13:24.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.826+0000 7fe55c620700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5540b3060 con 0x7fe55415ddf0 2024-04-06T12:13:24.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.826+0000 7fe55c620700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5540b3340 con 0x7fe55406ae40 2024-04-06T12:13:24.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.827+0000 7fe55abbd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe55415ddf0 0x7fe55415e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:24.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.827+0000 7fe55a3bc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe55415d7b0 0x7fe55406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33500/0 (socket says 172.21.15.189:33500) 2024-04-06T12:13:24.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.827+0000 7fe55a3bc700 1 -- 172.21.15.189:0/54531019 learned_addr learned my addr 172.21.15.189:0/54531019 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:24.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.827+0000 7fe559bbb700 1 --2- 172.21.15.189:0/54531019 >> v2:172.21.15.192:3300/0 conn(0x7fe55406ae40 0x7fe5540b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:24.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.827+0000 7fe5593ba700 1 -- 172.21.15.189:0/54531019 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070286631 0 0) 0x7fe5540b3990 con 0x7fe55415d7b0 2024-04-06T12:13:24.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.827+0000 7fe5593ba700 1 -- 172.21.15.189:0/54531019 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5400035b0 con 0x7fe55415d7b0 2024-04-06T12:13:24.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.827+0000 7fe55abbd700 1 -- 172.21.15.189:0/54531019 >> v2:172.21.15.192:3300/0 conn(0x7fe55406ae40 msgr2=0x7fe5540b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:24.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.827+0000 7fe55abbd700 1 --2- 172.21.15.189:0/54531019 >> v2:172.21.15.192:3300/0 conn(0x7fe55406ae40 0x7fe5540b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:24.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.827+0000 7fe55abbd700 1 -- 172.21.15.189:0/54531019 >> v1:172.21.15.189:6789/0 conn(0x7fe55415d7b0 legacy=0x7fe55406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:24.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.827+0000 7fe55abbd700 1 -- 172.21.15.189:0/54531019 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5540b3620 con 0x7fe55415ddf0 2024-04-06T12:13:24.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.827+0000 7fe559bbb700 1 --2- 172.21.15.189:0/54531019 >> v2:172.21.15.192:3300/0 conn(0x7fe55406ae40 0x7fe5540b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:24.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.827+0000 7fe55abbd700 1 --2- 172.21.15.189:0/54531019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe55415ddf0 0x7fe55415e240 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe5540afc00 tx=0x7fe54c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dcaeb1a52f7db8e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:24.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.827+0000 7fe5593ba700 1 -- 172.21.15.189:0/54531019 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54c00e070 con 0x7fe55415ddf0 2024-04-06T12:13:24.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.827+0000 7fe5593ba700 1 -- 172.21.15.189:0/54531019 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54c00bb20 con 0x7fe55415ddf0 2024-04-06T12:13:24.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.827+0000 7fe5593ba700 1 -- 172.21.15.189:0/54531019 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54c012430 con 0x7fe55415ddf0 2024-04-06T12:13:24.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.828+0000 7fe55c620700 1 -- 172.21.15.189:0/54531019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe55415ddf0 msgr2=0x7fe55415e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:24.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.828+0000 7fe55c620700 1 --2- 172.21.15.189:0/54531019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe55415ddf0 0x7fe55415e240 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe5540afc00 tx=0x7fe54c00b330 comp rx=0 tx=0).stop 2024-04-06T12:13:24.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.828+0000 7fe55c620700 1 -- 172.21.15.189:0/54531019 shutdown_connections 2024-04-06T12:13:24.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.828+0000 7fe55c620700 1 --2- 172.21.15.189:0/54531019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe55415ddf0 0x7fe55415e240 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:24.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.828+0000 7fe55c620700 1 --2- 172.21.15.189:0/54531019 >> v2:172.21.15.192:3300/0 conn(0x7fe55406ae40 0x7fe5540b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:24.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.828+0000 7fe55c620700 1 -- 172.21.15.189:0/54531019 >> 172.21.15.189:0/54531019 conn(0x7fe5540d00b0 msgr2=0x7fe5540d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:24.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.828+0000 7fe55c620700 1 -- 172.21.15.189:0/54531019 shutdown_connections 2024-04-06T12:13:24.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.828+0000 7fe55c620700 1 -- 172.21.15.189:0/54531019 wait complete. 2024-04-06T12:13:24.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.829+0000 7fe55c620700 1 Processor -- start 2024-04-06T12:13:24.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.829+0000 7fe55c620700 1 -- start start 2024-04-06T12:13:24.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe55c620700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe55406ae40 0x7fe554159690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:24.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe55c620700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe55415d7b0 0x7fe5540cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:24.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe55c620700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe55415ddf0 0x7fe554149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:24.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe55c620700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55415a8f0 con 0x7fe55406ae40 2024-04-06T12:13:24.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe55c620700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5540cd680 con 0x7fe55415ddf0 2024-04-06T12:13:24.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe55c620700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5540cde30 con 0x7fe55415d7b0 2024-04-06T12:13:24.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe55a3bc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe55406ae40 0x7fe554159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:24.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe55a3bc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe55406ae40 0x7fe554159690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33164/0 (socket says 172.21.15.189:33164) 2024-04-06T12:13:24.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe55a3bc700 1 -- 172.21.15.189:0/3441651720 learned_addr learned my addr 172.21.15.189:0/3441651720 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:24.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe55abbd700 1 --2- 172.21.15.189:0/3441651720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe55415ddf0 0x7fe554149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:24.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe559bbb700 1 --2- 172.21.15.189:0/3441651720 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe55415d7b0 0x7fe5540cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:24.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe55a3bc700 1 -- 172.21.15.189:0/3441651720 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe55415d7b0 msgr2=0x7fe5540cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:24.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe55a3bc700 1 --2- 172.21.15.189:0/3441651720 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe55415d7b0 0x7fe5540cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:24.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe55a3bc700 1 -- 172.21.15.189:0/3441651720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe55415ddf0 msgr2=0x7fe554149ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:24.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe55a3bc700 1 --2- 172.21.15.189:0/3441651720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe55415ddf0 0x7fe554149ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:24.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe55a3bc700 1 -- 172.21.15.189:0/3441651720 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c009a70 con 0x7fe55406ae40 2024-04-06T12:13:24.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe559bbb700 1 --2- 172.21.15.189:0/3441651720 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe55415d7b0 0x7fe5540cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:24.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe55abbd700 1 --2- 172.21.15.189:0/3441651720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe55415ddf0 0x7fe554149ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:24.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe55a3bc700 1 --2- 172.21.15.189:0/3441651720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe55406ae40 0x7fe554159690 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe55000c700 tx=0x7fe55000cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:24.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe54affd700 1 -- 172.21.15.189:0/3441651720 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe550005d70 con 0x7fe55406ae40 2024-04-06T12:13:24.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe54affd700 1 -- 172.21.15.189:0/3441651720 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5500070d0 con 0x7fe55406ae40 2024-04-06T12:13:24.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe55c620700 1 -- 172.21.15.189:0/3441651720 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5540cd870 con 0x7fe55406ae40 2024-04-06T12:13:24.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.830+0000 7fe54affd700 1 -- 172.21.15.189:0/3441651720 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe550007460 con 0x7fe55406ae40 2024-04-06T12:13:24.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.831+0000 7fe55c620700 1 -- 172.21.15.189:0/3441651720 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe55415a510 con 0x7fe55406ae40 2024-04-06T12:13:24.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.831+0000 7fe55c620700 1 -- 172.21.15.189:0/3441651720 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe55415a370 con 0x7fe55406ae40 2024-04-06T12:13:24.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.832+0000 7fe54affd700 1 -- 172.21.15.189:0/3441651720 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe5500075c0 con 0x7fe55406ae40 2024-04-06T12:13:24.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.832+0000 7fe54affd700 1 --2- 172.21.15.189:0/3441651720 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe53806ac90 0x7fe53806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:24.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.832+0000 7fe54affd700 1 -- 172.21.15.189:0/3441651720 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe550088a90 con 0x7fe55406ae40 2024-04-06T12:13:24.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.832+0000 7fe559bbb700 1 --2- 172.21.15.189:0/3441651720 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe53806ac90 0x7fe53806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:24.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.833+0000 7fe559bbb700 1 --2- 172.21.15.189:0/3441651720 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe53806ac90 0x7fe53806d150 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe5541595b0 tx=0x7fe544008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:24.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:24.837+0000 7fe54affd700 1 -- 172.21.15.189:0/3441651720 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe55415a370 con 0x7fe55406ae40 2024-04-06T12:13:25.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.031+0000 7fe55c620700 1 -- 172.21.15.189:0/3441651720 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe55415a370 con 0x7fe55406ae40 2024-04-06T12:13:25.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.033+0000 7fe54affd700 1 -- 172.21.15.189:0/3441651720 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16354 (secure 0 0 0) 0x7fe55415a370 con 0x7fe55406ae40 2024-04-06T12:13:25.031 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:13:25.031 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":39,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:13:22.289399+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:13:19.268280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-06T12:13:20.243771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:13:25.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.035+0000 7fe55c620700 1 -- 172.21.15.189:0/3441651720 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe53806ac90 msgr2=0x7fe53806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:25.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.035+0000 7fe55c620700 1 --2- 172.21.15.189:0/3441651720 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe53806ac90 0x7fe53806d150 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe5541595b0 tx=0x7fe544008040 comp rx=0 tx=0).stop 2024-04-06T12:13:25.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.035+0000 7fe55c620700 1 -- 172.21.15.189:0/3441651720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe55406ae40 msgr2=0x7fe554159690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:25.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.035+0000 7fe55c620700 1 --2- 172.21.15.189:0/3441651720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe55406ae40 0x7fe554159690 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe55000c700 tx=0x7fe55000cac0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.035+0000 7fe55c620700 1 -- 172.21.15.189:0/3441651720 shutdown_connections 2024-04-06T12:13:25.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.035+0000 7fe55c620700 1 --2- 172.21.15.189:0/3441651720 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe53806ac90 0x7fe53806d150 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.035+0000 7fe55c620700 1 --2- 172.21.15.189:0/3441651720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe55415ddf0 0x7fe554149ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.035+0000 7fe55c620700 1 --2- 172.21.15.189:0/3441651720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe55406ae40 0x7fe554159690 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.035+0000 7fe55c620700 1 --2- 172.21.15.189:0/3441651720 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe55415d7b0 0x7fe5540cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.035+0000 7fe55c620700 1 -- 172.21.15.189:0/3441651720 >> 172.21.15.189:0/3441651720 conn(0x7fe5540d00b0 msgr2=0x7fe55415ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:25.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.035+0000 7fe55c620700 1 -- 172.21.15.189:0/3441651720 shutdown_connections 2024-04-06T12:13:25.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.035+0000 7fe55c620700 1 -- 172.21.15.189:0/3441651720 wait complete. 2024-04-06T12:13:25.045 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-06T12:13:25.046 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:13:25.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.169+0000 7ff54fdd2700 1 Processor -- start 2024-04-06T12:13:25.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.169+0000 7ff54fdd2700 1 -- start start 2024-04-06T12:13:25.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.170+0000 7ff54fdd2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff5480cbc20 0x7ff54815f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.170+0000 7ff54fdd2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff54806aa10 0x7ff54806ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.170+0000 7ff54fdd2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5480669d0 con 0x7ff54815e8a0 2024-04-06T12:13:25.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.170+0000 7ff54fdd2700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff54806ede0 con 0x7ff5480cbc20 2024-04-06T12:13:25.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.170+0000 7ff54fdd2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff54815ec80 con 0x7ff54806aa10 2024-04-06T12:13:25.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.170+0000 7ff54d36d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff5480cbc20 0x7ff54815f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:25.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.170+0000 7ff54db6e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff54815e8a0 0x7ff54815eee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33512/0 (socket says 172.21.15.189:33512) 2024-04-06T12:13:25.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.170+0000 7ff54db6e700 1 -- 172.21.15.189:0/439893441 learned_addr learned my addr 172.21.15.189:0/439893441 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:25.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.171+0000 7ff54e36f700 1 --2- 172.21.15.189:0/439893441 >> v2:172.21.15.192:3300/0 conn(0x7ff54806aa10 0x7ff54806ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:25.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.171+0000 7ff54d36d700 1 -- 172.21.15.189:0/439893441 >> v2:172.21.15.192:3300/0 conn(0x7ff54806aa10 msgr2=0x7ff54806ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:25.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.171+0000 7ff54d36d700 1 --2- 172.21.15.189:0/439893441 >> v2:172.21.15.192:3300/0 conn(0x7ff54806aa10 0x7ff54806ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.171+0000 7ff54d36d700 1 -- 172.21.15.189:0/439893441 >> v1:172.21.15.189:6789/0 conn(0x7ff54815e8a0 legacy=0x7ff54815eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:25.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.171+0000 7ff54d36d700 1 -- 172.21.15.189:0/439893441 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5480666f0 con 0x7ff5480cbc20 2024-04-06T12:13:25.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.171+0000 7ff54e36f700 1 --2- 172.21.15.189:0/439893441 >> v2:172.21.15.192:3300/0 conn(0x7ff54806aa10 0x7ff54806ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:25.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.171+0000 7ff54cb6c700 1 -- 172.21.15.189:0/439893441 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673105981 0 0) 0x7ff5480669d0 con 0x7ff54815e8a0 2024-04-06T12:13:25.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.171+0000 7ff54d36d700 1 --2- 172.21.15.189:0/439893441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5480cbc20 0x7ff54815f5f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff5480cce40 tx=0x7ff544009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5be5f1f73ca5f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:25.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.171+0000 7ff54cb6c700 1 -- 172.21.15.189:0/439893441 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544011830 con 0x7ff5480cbc20 2024-04-06T12:13:25.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.171+0000 7ff54cb6c700 1 -- 172.21.15.189:0/439893441 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff544011990 con 0x7ff5480cbc20 2024-04-06T12:13:25.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.171+0000 7ff54cb6c700 1 -- 172.21.15.189:0/439893441 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544011d20 con 0x7ff5480cbc20 2024-04-06T12:13:25.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.172+0000 7ff54fdd2700 1 -- 172.21.15.189:0/439893441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5480cbc20 msgr2=0x7ff54815f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:25.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.172+0000 7ff54fdd2700 1 --2- 172.21.15.189:0/439893441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5480cbc20 0x7ff54815f5f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff5480cce40 tx=0x7ff544009b20 comp rx=0 tx=0).stop 2024-04-06T12:13:25.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.172+0000 7ff54fdd2700 1 -- 172.21.15.189:0/439893441 shutdown_connections 2024-04-06T12:13:25.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.172+0000 7ff54fdd2700 1 --2- 172.21.15.189:0/439893441 >> v2:172.21.15.192:3300/0 conn(0x7ff54806aa10 0x7ff54806ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.172+0000 7ff54fdd2700 1 --2- 172.21.15.189:0/439893441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5480cbc20 0x7ff54815f5f0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.172+0000 7ff54fdd2700 1 -- 172.21.15.189:0/439893441 >> 172.21.15.189:0/439893441 conn(0x7ff5480cc290 msgr2=0x7ff5480cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:25.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.172+0000 7ff54fdd2700 1 -- 172.21.15.189:0/439893441 shutdown_connections 2024-04-06T12:13:25.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.172+0000 7ff54fdd2700 1 -- 172.21.15.189:0/439893441 wait complete. 2024-04-06T12:13:25.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.173+0000 7ff54fdd2700 1 Processor -- start 2024-04-06T12:13:25.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.173+0000 7ff54fdd2700 1 -- start start 2024-04-06T12:13:25.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff54fdd2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff54806aa10 0x7ff5481f43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff54fdd2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5480cbc20 0x7ff5480c4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff54fdd2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff54815e8a0 0x7ff5480c5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff54fdd2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5481574a0 con 0x7ff5480cbc20 2024-04-06T12:13:25.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff54fdd2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5480c5920 con 0x7ff54806aa10 2024-04-06T12:13:25.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff54fdd2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5480c5c00 con 0x7ff54815e8a0 2024-04-06T12:13:25.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff54d36d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5480cbc20 0x7ff5480c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:25.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff54d36d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5480cbc20 0x7ff5480c4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33188/0 (socket says 172.21.15.189:33188) 2024-04-06T12:13:25.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff54d36d700 1 -- 172.21.15.189:0/3141084654 learned_addr learned my addr 172.21.15.189:0/3141084654 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:25.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff54e36f700 1 --2- 172.21.15.189:0/3141084654 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff54815e8a0 0x7ff5480c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:25.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff54db6e700 1 --2- 172.21.15.189:0/3141084654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff54806aa10 0x7ff5481f43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:25.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff54d36d700 1 -- 172.21.15.189:0/3141084654 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff54815e8a0 msgr2=0x7ff5480c5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:25.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff54d36d700 1 --2- 172.21.15.189:0/3141084654 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff54815e8a0 0x7ff5480c5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff54d36d700 1 -- 172.21.15.189:0/3141084654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff54806aa10 msgr2=0x7ff5481f43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:25.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff54d36d700 1 --2- 172.21.15.189:0/3141084654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff54806aa10 0x7ff5481f43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff54d36d700 1 -- 172.21.15.189:0/3141084654 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5440094a0 con 0x7ff5480cbc20 2024-04-06T12:13:25.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff54e36f700 1 --2- 172.21.15.189:0/3141084654 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff54815e8a0 0x7ff5480c5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:25.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff54db6e700 1 --2- 172.21.15.189:0/3141084654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff54806aa10 0x7ff5481f43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:25.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff54d36d700 1 --2- 172.21.15.189:0/3141084654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5480cbc20 0x7ff5480c4d40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff544000c00 tx=0x7ff544004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:25.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.174+0000 7ff53e7fc700 1 -- 172.21.15.189:0/3141084654 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5440038a0 con 0x7ff5480cbc20 2024-04-06T12:13:25.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.175+0000 7ff53e7fc700 1 -- 172.21.15.189:0/3141084654 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54401f070 con 0x7ff5480cbc20 2024-04-06T12:13:25.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.175+0000 7ff54fdd2700 1 -- 172.21.15.189:0/3141084654 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5480c7e80 con 0x7ff5480cbc20 2024-04-06T12:13:25.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.175+0000 7ff53e7fc700 1 -- 172.21.15.189:0/3141084654 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544003c40 con 0x7ff5480cbc20 2024-04-06T12:13:25.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.175+0000 7ff54fdd2700 1 -- 172.21.15.189:0/3141084654 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5480c8440 con 0x7ff5480cbc20 2024-04-06T12:13:25.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.175+0000 7ff54fdd2700 1 -- 172.21.15.189:0/3141084654 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff548072140 con 0x7ff5480cbc20 2024-04-06T12:13:25.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.176+0000 7ff53e7fc700 1 -- 172.21.15.189:0/3141084654 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff544026480 con 0x7ff5480cbc20 2024-04-06T12:13:25.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.176+0000 7ff53e7fc700 1 --2- 172.21.15.189:0/3141084654 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff52c06ac90 0x7ff52c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.176+0000 7ff53e7fc700 1 -- 172.21.15.189:0/3141084654 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff544021070 con 0x7ff5480cbc20 2024-04-06T12:13:25.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.176+0000 7ff54db6e700 1 --2- 172.21.15.189:0/3141084654 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff52c06ac90 0x7ff52c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:25.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.177+0000 7ff54db6e700 1 --2- 172.21.15.189:0/3141084654 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff52c06ac90 0x7ff52c06d150 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff538000f60 tx=0x7ff538007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:25.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.181+0000 7ff53e7fc700 1 -- 172.21.15.189:0/3141084654 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff544056060 con 0x7ff5480cbc20 2024-04-06T12:13:25.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.373+0000 7ff54fdd2700 1 -- 172.21.15.189:0/3141084654 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff54806dfe0 con 0x7ff52c06ac90 2024-04-06T12:13:25.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.382+0000 7ff53e7fc700 1 -- 172.21.15.189:0/3141084654 <== mgr.4100 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289715 (secure 0 0 0) 0x7ff538004870 con 0x7ff52c06ac90 2024-04-06T12:13:25.384 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:13:25.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.385+0000 7ff54fdd2700 1 -- 172.21.15.189:0/3141084654 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff52c06ac90 msgr2=0x7ff52c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:25.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.385+0000 7ff54fdd2700 1 --2- 172.21.15.189:0/3141084654 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff52c06ac90 0x7ff52c06d150 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff538000f60 tx=0x7ff538007d30 comp rx=0 tx=0).stop 2024-04-06T12:13:25.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.385+0000 7ff54fdd2700 1 -- 172.21.15.189:0/3141084654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5480cbc20 msgr2=0x7ff5480c4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:25.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.385+0000 7ff54fdd2700 1 --2- 172.21.15.189:0/3141084654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5480cbc20 0x7ff5480c4d40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff544000c00 tx=0x7ff544004750 comp rx=0 tx=0).stop 2024-04-06T12:13:25.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.385+0000 7ff54fdd2700 1 -- 172.21.15.189:0/3141084654 shutdown_connections 2024-04-06T12:13:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.385+0000 7ff54fdd2700 1 --2- 172.21.15.189:0/3141084654 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff54815e8a0 0x7ff5480c5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.385+0000 7ff54fdd2700 1 --2- 172.21.15.189:0/3141084654 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff52c06ac90 0x7ff52c06d150 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.385+0000 7ff54fdd2700 1 --2- 172.21.15.189:0/3141084654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff54806aa10 0x7ff5481f43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.385+0000 7ff54fdd2700 1 --2- 172.21.15.189:0/3141084654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5480cbc20 0x7ff5480c4d40 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.385+0000 7ff54fdd2700 1 -- 172.21.15.189:0/3141084654 >> 172.21.15.189:0/3141084654 conn(0x7ff5480cc290 msgr2=0x7ff54815c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.385+0000 7ff54fdd2700 1 -- 172.21.15.189:0/3141084654 shutdown_connections 2024-04-06T12:13:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.385+0000 7ff54fdd2700 1 -- 172.21.15.189:0/3141084654 wait complete. 2024-04-06T12:13:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:13:25.395 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-04-06T12:13:24.583141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220928,"kb_used_data":4688,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753792,"statfs":{"total":773094113280,"available":772867883008,"internally_reserved":0,"allocated":4800512,"data_stored":2713425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.714287"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368838+0000","last_change":"2024-04-06T12:13:21.314648+0000","last_active":"2024-04-06T12:13:22.368838+0000","last_peered":"2024-04-06T12:13:22.368838+0000","last_clean":"2024-04-06T12:13:22.368838+0000","last_became_active":"2024-04-06T12:13:21.314388+0000","last_became_peered":"2024-04-06T12:13:21.314388+0000","last_unstale":"2024-04-06T12:13:22.368838+0000","last_undegraded":"2024-04-06T12:13:22.368838+0000","last_fullsized":"2024-04-06T12:13:22.368838+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:08:08.716105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368574+0000","last_change":"2024-04-06T12:13:21.324261+0000","last_active":"2024-04-06T12:13:22.368574+0000","last_peered":"2024-04-06T12:13:22.368574+0000","last_clean":"2024-04-06T12:13:22.368574+0000","last_became_active":"2024-04-06T12:13:21.324018+0000","last_became_peered":"2024-04-06T12:13:21.324018+0000","last_unstale":"2024-04-06T12:13:22.368574+0000","last_undegraded":"2024-04-06T12:13:22.368574+0000","last_fullsized":"2024-04-06T12:13:22.368574+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:27:48.973861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368382+0000","last_change":"2024-04-06T12:13:21.307590+0000","last_active":"2024-04-06T12:13:22.368382+0000","last_peered":"2024-04-06T12:13:22.368382+0000","last_clean":"2024-04-06T12:13:22.368382+0000","last_became_active":"2024-04-06T12:13:21.307318+0000","last_became_peered":"2024-04-06T12:13:21.307318+0000","last_unstale":"2024-04-06T12:13:22.368382+0000","last_undegraded":"2024-04-06T12:13:22.368382+0000","last_fullsized":"2024-04-06T12:13:22.368382+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:06:39.941523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368107+0000","last_change":"2024-04-06T12:13:21.308565+0000","last_active":"2024-04-06T12:13:22.368107+0000","last_peered":"2024-04-06T12:13:22.368107+0000","last_clean":"2024-04-06T12:13:22.368107+0000","last_became_active":"2024-04-06T12:13:21.308328+0000","last_became_peered":"2024-04-06T12:13:21.308328+0000","last_unstale":"2024-04-06T12:13:22.368107+0000","last_undegraded":"2024-04-06T12:13:22.368107+0000","last_fullsized":"2024-04-06T12:13:22.368107+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:14:53.401297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.367923+0000","last_change":"2024-04-06T12:13:21.298073+0000","last_active":"2024-04-06T12:13:22.367923+0000","last_peered":"2024-04-06T12:13:22.367923+0000","last_clean":"2024-04-06T12:13:22.367923+0000","last_became_active":"2024-04-06T12:13:21.297463+0000","last_became_peered":"2024-04-06T12:13:21.297463+0000","last_unstale":"2024-04-06T12:13:22.367923+0000","last_undegraded":"2024-04-06T12:13:22.367923+0000","last_fullsized":"2024-04-06T12:13:22.367923+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:44:09.673989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293010+0000","last_change":"2024-04-06T12:13:21.314937+0000","last_active":"2024-04-06T12:13:22.293010+0000","last_peered":"2024-04-06T12:13:22.293010+0000","last_clean":"2024-04-06T12:13:22.293010+0000","last_became_active":"2024-04-06T12:13:21.314677+0000","last_became_peered":"2024-04-06T12:13:21.314677+0000","last_unstale":"2024-04-06T12:13:22.293010+0000","last_undegraded":"2024-04-06T12:13:22.293010+0000","last_fullsized":"2024-04-06T12:13:22.293010+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:46:09.334185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369543+0000","last_change":"2024-04-06T12:13:21.315764+0000","last_active":"2024-04-06T12:13:22.369543+0000","last_peered":"2024-04-06T12:13:22.369543+0000","last_clean":"2024-04-06T12:13:22.369543+0000","last_became_active":"2024-04-06T12:13:21.315483+0000","last_became_peered":"2024-04-06T12:13:21.315483+0000","last_unstale":"2024-04-06T12:13:22.369543+0000","last_undegraded":"2024-04-06T12:13:22.369543+0000","last_fullsized":"2024-04-06T12:13:22.369543+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:53:38.483235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368052+0000","last_change":"2024-04-06T12:13:21.314740+0000","last_active":"2024-04-06T12:13:22.368052+0000","last_peered":"2024-04-06T12:13:22.368052+0000","last_clean":"2024-04-06T12:13:22.368052+0000","last_became_active":"2024-04-06T12:13:21.314477+0000","last_became_peered":"2024-04-06T12:13:21.314477+0000","last_unstale":"2024-04-06T12:13:22.368052+0000","last_undegraded":"2024-04-06T12:13:22.368052+0000","last_fullsized":"2024-04-06T12:13:22.368052+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:12:47.226472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368444+0000","last_change":"2024-04-06T12:13:21.313198+0000","last_active":"2024-04-06T12:13:22.368444+0000","last_peered":"2024-04-06T12:13:22.368444+0000","last_clean":"2024-04-06T12:13:22.368444+0000","last_became_active":"2024-04-06T12:13:21.312843+0000","last_became_peered":"2024-04-06T12:13:21.312843+0000","last_unstale":"2024-04-06T12:13:22.368444+0000","last_undegraded":"2024-04-06T12:13:22.368444+0000","last_fullsized":"2024-04-06T12:13:22.368444+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:41:19.205834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369663+0000","last_change":"2024-04-06T12:13:21.312756+0000","last_active":"2024-04-06T12:13:22.369663+0000","last_peered":"2024-04-06T12:13:22.369663+0000","last_clean":"2024-04-06T12:13:22.369663+0000","last_became_active":"2024-04-06T12:13:21.312386+0000","last_became_peered":"2024-04-06T12:13:21.312386+0000","last_unstale":"2024-04-06T12:13:22.369663+0000","last_undegraded":"2024-04-06T12:13:22.369663+0000","last_fullsized":"2024-04-06T12:13:22.369663+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:06:39.941523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368347+0000","last_change":"2024-04-06T12:13:21.321020+0000","last_active":"2024-04-06T12:13:22.368347+0000","last_peered":"2024-04-06T12:13:22.368347+0000","last_clean":"2024-04-06T12:13:22.368347+0000","last_became_active":"2024-04-06T12:13:21.320752+0000","last_became_peered":"2024-04-06T12:13:21.320752+0000","last_unstale":"2024-04-06T12:13:22.368347+0000","last_undegraded":"2024-04-06T12:13:22.368347+0000","last_fullsized":"2024-04-06T12:13:22.368347+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:01:46.620598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368263+0000","last_change":"2024-04-06T12:13:21.302283+0000","last_active":"2024-04-06T12:13:22.368263+0000","last_peered":"2024-04-06T12:13:22.368263+0000","last_clean":"2024-04-06T12:13:22.368263+0000","last_became_active":"2024-04-06T12:13:21.302109+0000","last_became_peered":"2024-04-06T12:13:21.302109+0000","last_unstale":"2024-04-06T12:13:22.368263+0000","last_undegraded":"2024-04-06T12:13:22.368263+0000","last_fullsized":"2024-04-06T12:13:22.368263+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:53:38.483235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368453+0000","last_change":"2024-04-06T12:13:21.312054+0000","last_active":"2024-04-06T12:13:22.368453+0000","last_peered":"2024-04-06T12:13:22.368453+0000","last_clean":"2024-04-06T12:13:22.368453+0000","last_became_active":"2024-04-06T12:13:21.311792+0000","last_became_peered":"2024-04-06T12:13:21.311792+0000","last_unstale":"2024-04-06T12:13:22.368453+0000","last_undegraded":"2024-04-06T12:13:22.368453+0000","last_fullsized":"2024-04-06T12:13:22.368453+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:29:43.926143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368447+0000","last_change":"2024-04-06T12:13:21.314135+0000","last_active":"2024-04-06T12:13:22.368447+0000","last_peered":"2024-04-06T12:13:22.368447+0000","last_clean":"2024-04-06T12:13:22.368447+0000","last_became_active":"2024-04-06T12:13:21.313888+0000","last_became_peered":"2024-04-06T12:13:21.313888+0000","last_unstale":"2024-04-06T12:13:22.368447+0000","last_undegraded":"2024-04-06T12:13:22.368447+0000","last_fullsized":"2024-04-06T12:13:22.368447+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:01.313739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.367937+0000","last_change":"2024-04-06T12:13:21.312811+0000","last_active":"2024-04-06T12:13:22.367937+0000","last_peered":"2024-04-06T12:13:22.367937+0000","last_clean":"2024-04-06T12:13:22.367937+0000","last_became_active":"2024-04-06T12:13:21.312482+0000","last_became_peered":"2024-04-06T12:13:21.312482+0000","last_unstale":"2024-04-06T12:13:22.367937+0000","last_undegraded":"2024-04-06T12:13:22.367937+0000","last_fullsized":"2024-04-06T12:13:22.367937+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:13:29.653277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293360+0000","last_change":"2024-04-06T12:13:21.313055+0000","last_active":"2024-04-06T12:13:22.293360+0000","last_peered":"2024-04-06T12:13:22.293360+0000","last_clean":"2024-04-06T12:13:22.293360+0000","last_became_active":"2024-04-06T12:13:21.312260+0000","last_became_peered":"2024-04-06T12:13:21.312260+0000","last_unstale":"2024-04-06T12:13:22.293360+0000","last_undegraded":"2024-04-06T12:13:22.293360+0000","last_fullsized":"2024-04-06T12:13:22.293360+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:53:51.147798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293848+0000","last_change":"2024-04-06T12:13:21.312705+0000","last_active":"2024-04-06T12:13:22.293848+0000","last_peered":"2024-04-06T12:13:22.293848+0000","last_clean":"2024-04-06T12:13:22.293848+0000","last_became_active":"2024-04-06T12:13:21.312019+0000","last_became_peered":"2024-04-06T12:13:21.312019+0000","last_unstale":"2024-04-06T12:13:22.293848+0000","last_undegraded":"2024-04-06T12:13:22.293848+0000","last_fullsized":"2024-04-06T12:13:22.293848+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:45:12.027545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.294427+0000","last_change":"2024-04-06T12:13:21.317041+0000","last_active":"2024-04-06T12:13:22.294427+0000","last_peered":"2024-04-06T12:13:22.294427+0000","last_clean":"2024-04-06T12:13:22.294427+0000","last_became_active":"2024-04-06T12:13:21.316786+0000","last_became_peered":"2024-04-06T12:13:21.316786+0000","last_unstale":"2024-04-06T12:13:22.294427+0000","last_undegraded":"2024-04-06T12:13:22.294427+0000","last_fullsized":"2024-04-06T12:13:22.294427+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:15:56.195554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368886+0000","last_change":"2024-04-06T12:13:21.300603+0000","last_active":"2024-04-06T12:13:22.368886+0000","last_peered":"2024-04-06T12:13:22.368886+0000","last_clean":"2024-04-06T12:13:22.368886+0000","last_became_active":"2024-04-06T12:13:21.300338+0000","last_became_peered":"2024-04-06T12:13:21.300338+0000","last_unstale":"2024-04-06T12:13:22.368886+0000","last_undegraded":"2024-04-06T12:13:22.368886+0000","last_fullsized":"2024-04-06T12:13:22.368886+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:45:15.762266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.294007+0000","last_change":"2024-04-06T12:13:21.316540+0000","last_active":"2024-04-06T12:13:22.294007+0000","last_peered":"2024-04-06T12:13:22.294007+0000","last_clean":"2024-04-06T12:13:22.294007+0000","last_became_active":"2024-04-06T12:13:21.316267+0000","last_became_peered":"2024-04-06T12:13:21.316267+0000","last_unstale":"2024-04-06T12:13:22.294007+0000","last_undegraded":"2024-04-06T12:13:22.294007+0000","last_fullsized":"2024-04-06T12:13:22.294007+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:50:45.615242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368164+0000","last_change":"2024-04-06T12:13:21.311462+0000","last_active":"2024-04-06T12:13:22.368164+0000","last_peered":"2024-04-06T12:13:22.368164+0000","last_clean":"2024-04-06T12:13:22.368164+0000","last_became_active":"2024-04-06T12:13:21.311187+0000","last_became_peered":"2024-04-06T12:13:21.311187+0000","last_unstale":"2024-04-06T12:13:22.368164+0000","last_undegraded":"2024-04-06T12:13:22.368164+0000","last_fullsized":"2024-04-06T12:13:22.368164+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:41:19.205834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368566+0000","last_change":"2024-04-06T12:13:21.308118+0000","last_active":"2024-04-06T12:13:22.368566+0000","last_peered":"2024-04-06T12:13:22.368566+0000","last_clean":"2024-04-06T12:13:22.368566+0000","last_became_active":"2024-04-06T12:13:21.307712+0000","last_became_peered":"2024-04-06T12:13:21.307712+0000","last_unstale":"2024-04-06T12:13:22.368566+0000","last_undegraded":"2024-04-06T12:13:22.368566+0000","last_fullsized":"2024-04-06T12:13:22.368566+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:19:19.798635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293486+0000","last_change":"2024-04-06T12:13:21.310424+0000","last_active":"2024-04-06T12:13:22.293486+0000","last_peered":"2024-04-06T12:13:22.293486+0000","last_clean":"2024-04-06T12:13:22.293486+0000","last_became_active":"2024-04-06T12:13:21.308250+0000","last_became_peered":"2024-04-06T12:13:21.308250+0000","last_unstale":"2024-04-06T12:13:22.293486+0000","last_undegraded":"2024-04-06T12:13:22.293486+0000","last_fullsized":"2024-04-06T12:13:22.293486+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:44:37.932979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368070+0000","last_change":"2024-04-06T12:13:21.312586+0000","last_active":"2024-04-06T12:13:22.368070+0000","last_peered":"2024-04-06T12:13:22.368070+0000","last_clean":"2024-04-06T12:13:22.368070+0000","last_became_active":"2024-04-06T12:13:21.312104+0000","last_became_peered":"2024-04-06T12:13:21.312104+0000","last_unstale":"2024-04-06T12:13:22.368070+0000","last_undegraded":"2024-04-06T12:13:22.368070+0000","last_fullsized":"2024-04-06T12:13:22.368070+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:54:06.007131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.294211+0000","last_change":"2024-04-06T12:13:21.317835+0000","last_active":"2024-04-06T12:13:22.294211+0000","last_peered":"2024-04-06T12:13:22.294211+0000","last_clean":"2024-04-06T12:13:22.294211+0000","last_became_active":"2024-04-06T12:13:21.317576+0000","last_became_peered":"2024-04-06T12:13:21.317576+0000","last_unstale":"2024-04-06T12:13:22.294211+0000","last_undegraded":"2024-04-06T12:13:22.294211+0000","last_fullsized":"2024-04-06T12:13:22.294211+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:54:06.007131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293033+0000","last_change":"2024-04-06T12:13:21.313596+0000","last_active":"2024-04-06T12:13:22.293033+0000","last_peered":"2024-04-06T12:13:22.293033+0000","last_clean":"2024-04-06T12:13:22.293033+0000","last_became_active":"2024-04-06T12:13:21.313334+0000","last_became_peered":"2024-04-06T12:13:21.313334+0000","last_unstale":"2024-04-06T12:13:22.293033+0000","last_undegraded":"2024-04-06T12:13:22.293033+0000","last_fullsized":"2024-04-06T12:13:22.293033+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:41:19.205834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293972+0000","last_change":"2024-04-06T12:13:21.313694+0000","last_active":"2024-04-06T12:13:22.293972+0000","last_peered":"2024-04-06T12:13:22.293972+0000","last_clean":"2024-04-06T12:13:22.293972+0000","last_became_active":"2024-04-06T12:13:21.313407+0000","last_became_peered":"2024-04-06T12:13:21.313407+0000","last_unstale":"2024-04-06T12:13:22.293972+0000","last_undegraded":"2024-04-06T12:13:22.293972+0000","last_fullsized":"2024-04-06T12:13:22.293972+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:57:05.900931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369453+0000","last_change":"2024-04-06T12:13:21.316069+0000","last_active":"2024-04-06T12:13:22.369453+0000","last_peered":"2024-04-06T12:13:22.369453+0000","last_clean":"2024-04-06T12:13:22.369453+0000","last_became_active":"2024-04-06T12:13:21.315815+0000","last_became_peered":"2024-04-06T12:13:21.315815+0000","last_unstale":"2024-04-06T12:13:22.369453+0000","last_undegraded":"2024-04-06T12:13:22.369453+0000","last_fullsized":"2024-04-06T12:13:22.369453+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:45:04.876402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.294218+0000","last_change":"2024-04-06T12:13:21.317125+0000","last_active":"2024-04-06T12:13:22.294218+0000","last_peered":"2024-04-06T12:13:22.294218+0000","last_clean":"2024-04-06T12:13:22.294218+0000","last_became_active":"2024-04-06T12:13:21.316868+0000","last_became_peered":"2024-04-06T12:13:21.316868+0000","last_unstale":"2024-04-06T12:13:22.294218+0000","last_undegraded":"2024-04-06T12:13:22.294218+0000","last_fullsized":"2024-04-06T12:13:22.294218+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:53:51.147798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293716+0000","last_change":"2024-04-06T12:13:21.318843+0000","last_active":"2024-04-06T12:13:22.293716+0000","last_peered":"2024-04-06T12:13:22.293716+0000","last_clean":"2024-04-06T12:13:22.293716+0000","last_became_active":"2024-04-06T12:13:21.318565+0000","last_became_peered":"2024-04-06T12:13:21.318565+0000","last_unstale":"2024-04-06T12:13:22.293716+0000","last_undegraded":"2024-04-06T12:13:22.293716+0000","last_fullsized":"2024-04-06T12:13:22.293716+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:48:58.061525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293362+0000","last_change":"2024-04-06T12:13:21.309134+0000","last_active":"2024-04-06T12:13:22.293362+0000","last_peered":"2024-04-06T12:13:22.293362+0000","last_clean":"2024-04-06T12:13:22.293362+0000","last_became_active":"2024-04-06T12:13:21.308873+0000","last_became_peered":"2024-04-06T12:13:21.308873+0000","last_unstale":"2024-04-06T12:13:22.293362+0000","last_undegraded":"2024-04-06T12:13:22.293362+0000","last_fullsized":"2024-04-06T12:13:22.293362+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:57:07.685321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293120+0000","last_change":"2024-04-06T12:13:21.303727+0000","last_active":"2024-04-06T12:13:22.293120+0000","last_peered":"2024-04-06T12:13:22.293120+0000","last_clean":"2024-04-06T12:13:22.293120+0000","last_became_active":"2024-04-06T12:13:21.303521+0000","last_became_peered":"2024-04-06T12:13:21.303521+0000","last_unstale":"2024-04-06T12:13:22.293120+0000","last_undegraded":"2024-04-06T12:13:22.293120+0000","last_fullsized":"2024-04-06T12:13:22.293120+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:13:29.653277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293542+0000","last_change":"2024-04-06T12:13:21.316236+0000","last_active":"2024-04-06T12:13:22.293542+0000","last_peered":"2024-04-06T12:13:22.293542+0000","last_clean":"2024-04-06T12:13:22.293542+0000","last_became_active":"2024-04-06T12:13:21.315956+0000","last_became_peered":"2024-04-06T12:13:21.315956+0000","last_unstale":"2024-04-06T12:13:22.293542+0000","last_undegraded":"2024-04-06T12:13:22.293542+0000","last_fullsized":"2024-04-06T12:13:22.293542+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:50:45.615242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.294292+0000","last_change":"2024-04-06T12:13:20.897573+0000","last_active":"2024-04-06T12:13:22.294292+0000","last_peered":"2024-04-06T12:13:22.294292+0000","last_clean":"2024-04-06T12:13:22.294292+0000","last_became_active":"2024-04-06T12:13:20.897299+0000","last_became_peered":"2024-04-06T12:13:20.897299+0000","last_unstale":"2024-04-06T12:13:22.294292+0000","last_undegraded":"2024-04-06T12:13:22.294292+0000","last_fullsized":"2024-04-06T12:13:22.294292+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:19:10.009040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293172+0000","last_change":"2024-04-06T12:13:21.316550+0000","last_active":"2024-04-06T12:13:22.293172+0000","last_peered":"2024-04-06T12:13:22.293172+0000","last_clean":"2024-04-06T12:13:22.293172+0000","last_became_active":"2024-04-06T12:13:21.316325+0000","last_became_peered":"2024-04-06T12:13:21.316325+0000","last_unstale":"2024-04-06T12:13:22.293172+0000","last_undegraded":"2024-04-06T12:13:22.293172+0000","last_fullsized":"2024-04-06T12:13:22.293172+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:17:40.977598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369175+0000","last_change":"2024-04-06T12:13:20.897660+0000","last_active":"2024-04-06T12:13:22.369175+0000","last_peered":"2024-04-06T12:13:22.369175+0000","last_clean":"2024-04-06T12:13:22.369175+0000","last_became_active":"2024-04-06T12:13:20.897405+0000","last_became_peered":"2024-04-06T12:13:20.897405+0000","last_unstale":"2024-04-06T12:13:22.369175+0000","last_undegraded":"2024-04-06T12:13:22.369175+0000","last_fullsized":"2024-04-06T12:13:22.369175+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:12:47.154213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369178+0000","last_change":"2024-04-06T12:13:21.309155+0000","last_active":"2024-04-06T12:13:22.369178+0000","last_peered":"2024-04-06T12:13:22.369178+0000","last_clean":"2024-04-06T12:13:22.369178+0000","last_became_active":"2024-04-06T12:13:21.308879+0000","last_became_peered":"2024-04-06T12:13:21.308879+0000","last_unstale":"2024-04-06T12:13:22.369178+0000","last_undegraded":"2024-04-06T12:13:22.369178+0000","last_fullsized":"2024-04-06T12:13:22.369178+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:17:40.977598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293195+0000","last_change":"2024-04-06T12:13:20.884932+0000","last_active":"2024-04-06T12:13:22.293195+0000","last_peered":"2024-04-06T12:13:22.293195+0000","last_clean":"2024-04-06T12:13:22.293195+0000","last_became_active":"2024-04-06T12:13:20.884681+0000","last_became_peered":"2024-04-06T12:13:20.884681+0000","last_unstale":"2024-04-06T12:13:22.293195+0000","last_undegraded":"2024-04-06T12:13:22.293195+0000","last_fullsized":"2024-04-06T12:13:22.293195+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:57:06.680055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368984+0000","last_change":"2024-04-06T12:13:21.317904+0000","last_active":"2024-04-06T12:13:22.368984+0000","last_peered":"2024-04-06T12:13:22.368984+0000","last_clean":"2024-04-06T12:13:22.368984+0000","last_became_active":"2024-04-06T12:13:21.317652+0000","last_became_peered":"2024-04-06T12:13:21.317652+0000","last_unstale":"2024-04-06T12:13:22.368984+0000","last_undegraded":"2024-04-06T12:13:22.368984+0000","last_fullsized":"2024-04-06T12:13:22.368984+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:17:40.174838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368705+0000","last_change":"2024-04-06T12:13:20.901272+0000","last_active":"2024-04-06T12:13:22.368705+0000","last_peered":"2024-04-06T12:13:22.368705+0000","last_clean":"2024-04-06T12:13:22.368705+0000","last_became_active":"2024-04-06T12:13:20.900984+0000","last_became_peered":"2024-04-06T12:13:20.900984+0000","last_unstale":"2024-04-06T12:13:22.368705+0000","last_undegraded":"2024-04-06T12:13:22.368705+0000","last_fullsized":"2024-04-06T12:13:22.368705+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:57:06.680055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368214+0000","last_change":"2024-04-06T12:13:21.303808+0000","last_active":"2024-04-06T12:13:22.368214+0000","last_peered":"2024-04-06T12:13:22.368214+0000","last_clean":"2024-04-06T12:13:22.368214+0000","last_became_active":"2024-04-06T12:13:21.303549+0000","last_became_peered":"2024-04-06T12:13:21.303549+0000","last_unstale":"2024-04-06T12:13:22.368214+0000","last_undegraded":"2024-04-06T12:13:22.368214+0000","last_fullsized":"2024-04-06T12:13:22.368214+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:37:27.117856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368289+0000","last_change":"2024-04-06T12:13:20.896675+0000","last_active":"2024-04-06T12:13:22.368289+0000","last_peered":"2024-04-06T12:13:22.368289+0000","last_clean":"2024-04-06T12:13:22.368289+0000","last_became_active":"2024-04-06T12:13:20.896395+0000","last_became_peered":"2024-04-06T12:13:20.896395+0000","last_unstale":"2024-04-06T12:13:22.368289+0000","last_undegraded":"2024-04-06T12:13:22.368289+0000","last_fullsized":"2024-04-06T12:13:22.368289+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:14:52.396032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.367910+0000","last_change":"2024-04-06T12:13:21.309885+0000","last_active":"2024-04-06T12:13:22.367910+0000","last_peered":"2024-04-06T12:13:22.367910+0000","last_clean":"2024-04-06T12:13:22.367910+0000","last_became_active":"2024-04-06T12:13:21.309596+0000","last_became_peered":"2024-04-06T12:13:21.309596+0000","last_unstale":"2024-04-06T12:13:22.367910+0000","last_undegraded":"2024-04-06T12:13:22.367910+0000","last_fullsized":"2024-04-06T12:13:22.367910+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:46:09.334185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293693+0000","last_change":"2024-04-06T12:13:20.897399+0000","last_active":"2024-04-06T12:13:22.293693+0000","last_peered":"2024-04-06T12:13:22.293693+0000","last_clean":"2024-04-06T12:13:22.293693+0000","last_became_active":"2024-04-06T12:13:20.897116+0000","last_became_peered":"2024-04-06T12:13:20.897116+0000","last_unstale":"2024-04-06T12:13:22.293693+0000","last_undegraded":"2024-04-06T12:13:22.293693+0000","last_fullsized":"2024-04-06T12:13:22.293693+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:19:10.009040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368683+0000","last_change":"2024-04-06T12:13:21.313322+0000","last_active":"2024-04-06T12:13:22.368683+0000","last_peered":"2024-04-06T12:13:22.368683+0000","last_clean":"2024-04-06T12:13:22.368683+0000","last_became_active":"2024-04-06T12:13:21.313042+0000","last_became_peered":"2024-04-06T12:13:21.313042+0000","last_unstale":"2024-04-06T12:13:22.368683+0000","last_undegraded":"2024-04-06T12:13:22.368683+0000","last_fullsized":"2024-04-06T12:13:22.368683+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:41:19.205834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.292966+0000","last_change":"2024-04-06T12:13:20.894229+0000","last_active":"2024-04-06T12:13:22.292966+0000","last_peered":"2024-04-06T12:13:22.292966+0000","last_clean":"2024-04-06T12:13:22.292966+0000","last_became_active":"2024-04-06T12:13:20.893620+0000","last_became_peered":"2024-04-06T12:13:20.893620+0000","last_unstale":"2024-04-06T12:13:22.292966+0000","last_undegraded":"2024-04-06T12:13:22.292966+0000","last_fullsized":"2024-04-06T12:13:22.292966+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:18:12.812027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368677+0000","last_change":"2024-04-06T12:13:21.318741+0000","last_active":"2024-04-06T12:13:22.368677+0000","last_peered":"2024-04-06T12:13:22.368677+0000","last_clean":"2024-04-06T12:13:22.368677+0000","last_became_active":"2024-04-06T12:13:21.318070+0000","last_became_peered":"2024-04-06T12:13:21.318070+0000","last_unstale":"2024-04-06T12:13:22.368677+0000","last_undegraded":"2024-04-06T12:13:22.368677+0000","last_fullsized":"2024-04-06T12:13:22.368677+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:53:09.306045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368366+0000","last_change":"2024-04-06T12:13:20.892999+0000","last_active":"2024-04-06T12:13:22.368366+0000","last_peered":"2024-04-06T12:13:22.368366+0000","last_clean":"2024-04-06T12:13:22.368366+0000","last_became_active":"2024-04-06T12:13:20.892205+0000","last_became_peered":"2024-04-06T12:13:20.892205+0000","last_unstale":"2024-04-06T12:13:22.368366+0000","last_undegraded":"2024-04-06T12:13:22.368366+0000","last_fullsized":"2024-04-06T12:13:22.368366+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:18:12.812027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368199+0000","last_change":"2024-04-06T12:13:21.297738+0000","last_active":"2024-04-06T12:13:22.368199+0000","last_peered":"2024-04-06T12:13:22.368199+0000","last_clean":"2024-04-06T12:13:22.368199+0000","last_became_active":"2024-04-06T12:13:21.297493+0000","last_became_peered":"2024-04-06T12:13:21.297493+0000","last_unstale":"2024-04-06T12:13:22.368199+0000","last_undegraded":"2024-04-06T12:13:22.368199+0000","last_fullsized":"2024-04-06T12:13:22.368199+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:38:07.521009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369057+0000","last_change":"2024-04-06T12:13:20.897312+0000","last_active":"2024-04-06T12:13:22.369057+0000","last_peered":"2024-04-06T12:13:22.369057+0000","last_clean":"2024-04-06T12:13:22.369057+0000","last_became_active":"2024-04-06T12:13:20.896922+0000","last_became_peered":"2024-04-06T12:13:20.896922+0000","last_unstale":"2024-04-06T12:13:22.369057+0000","last_undegraded":"2024-04-06T12:13:22.369057+0000","last_fullsized":"2024-04-06T12:13:22.369057+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:06:38.936257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293362+0000","last_change":"2024-04-06T12:13:21.316416+0000","last_active":"2024-04-06T12:13:22.293362+0000","last_peered":"2024-04-06T12:13:22.293362+0000","last_clean":"2024-04-06T12:13:22.293362+0000","last_became_active":"2024-04-06T12:13:21.311801+0000","last_became_peered":"2024-04-06T12:13:21.311801+0000","last_unstale":"2024-04-06T12:13:22.293362+0000","last_undegraded":"2024-04-06T12:13:22.293362+0000","last_fullsized":"2024-04-06T12:13:22.293362+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:53:40.611211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369721+0000","last_change":"2024-04-06T12:13:20.895917+0000","last_active":"2024-04-06T12:13:22.369721+0000","last_peered":"2024-04-06T12:13:22.369721+0000","last_clean":"2024-04-06T12:13:22.369721+0000","last_became_active":"2024-04-06T12:13:20.895648+0000","last_became_peered":"2024-04-06T12:13:20.895648+0000","last_unstale":"2024-04-06T12:13:22.369721+0000","last_undegraded":"2024-04-06T12:13:22.369721+0000","last_fullsized":"2024-04-06T12:13:22.369721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:44:08.668723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369094+0000","last_change":"2024-04-06T12:13:21.311428+0000","last_active":"2024-04-06T12:13:22.369094+0000","last_peered":"2024-04-06T12:13:22.369094+0000","last_clean":"2024-04-06T12:13:22.369094+0000","last_became_active":"2024-04-06T12:13:21.311163+0000","last_became_peered":"2024-04-06T12:13:21.311163+0000","last_unstale":"2024-04-06T12:13:22.369094+0000","last_undegraded":"2024-04-06T12:13:22.369094+0000","last_fullsized":"2024-04-06T12:13:22.369094+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:01.313739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368407+0000","last_change":"2024-04-06T12:13:20.897032+0000","last_active":"2024-04-06T12:13:22.368407+0000","last_peered":"2024-04-06T12:13:22.368407+0000","last_clean":"2024-04-06T12:13:22.368407+0000","last_became_active":"2024-04-06T12:13:20.896789+0000","last_became_peered":"2024-04-06T12:13:20.896789+0000","last_unstale":"2024-04-06T12:13:22.368407+0000","last_undegraded":"2024-04-06T12:13:22.368407+0000","last_fullsized":"2024-04-06T12:13:22.368407+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:53:50.142533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368915+0000","last_change":"2024-04-06T12:13:21.306919+0000","last_active":"2024-04-06T12:13:22.368915+0000","last_peered":"2024-04-06T12:13:22.368915+0000","last_clean":"2024-04-06T12:13:22.368915+0000","last_became_active":"2024-04-06T12:13:21.302205+0000","last_became_peered":"2024-04-06T12:13:21.302205+0000","last_unstale":"2024-04-06T12:13:22.368915+0000","last_undegraded":"2024-04-06T12:13:22.368915+0000","last_fullsized":"2024-04-06T12:13:22.368915+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:46:09.334185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368743+0000","last_change":"2024-04-06T12:13:20.895861+0000","last_active":"2024-04-06T12:13:22.368743+0000","last_peered":"2024-04-06T12:13:22.368743+0000","last_clean":"2024-04-06T12:13:22.368743+0000","last_became_active":"2024-04-06T12:13:20.895407+0000","last_became_peered":"2024-04-06T12:13:20.895407+0000","last_unstale":"2024-04-06T12:13:22.368743+0000","last_undegraded":"2024-04-06T12:13:22.368743+0000","last_fullsized":"2024-04-06T12:13:22.368743+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:19:18.793370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.367797+0000","last_change":"2024-04-06T12:13:21.313364+0000","last_active":"2024-04-06T12:13:22.367797+0000","last_peered":"2024-04-06T12:13:22.367797+0000","last_clean":"2024-04-06T12:13:22.367797+0000","last_became_active":"2024-04-06T12:13:21.312871+0000","last_became_peered":"2024-04-06T12:13:21.312871+0000","last_unstale":"2024-04-06T12:13:22.367797+0000","last_undegraded":"2024-04-06T12:13:22.367797+0000","last_fullsized":"2024-04-06T12:13:22.367797+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:46:09.334185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368773+0000","last_change":"2024-04-06T12:13:20.896174+0000","last_active":"2024-04-06T12:13:22.368773+0000","last_peered":"2024-04-06T12:13:22.368773+0000","last_clean":"2024-04-06T12:13:22.368773+0000","last_became_active":"2024-04-06T12:13:20.895869+0000","last_became_peered":"2024-04-06T12:13:20.895869+0000","last_unstale":"2024-04-06T12:13:22.368773+0000","last_undegraded":"2024-04-06T12:13:22.368773+0000","last_fullsized":"2024-04-06T12:13:22.368773+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:41:18.200569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369310+0000","last_change":"2024-04-06T12:13:21.314464+0000","last_active":"2024-04-06T12:13:22.369310+0000","last_peered":"2024-04-06T12:13:22.369310+0000","last_clean":"2024-04-06T12:13:22.369310+0000","last_became_active":"2024-04-06T12:13:21.314125+0000","last_became_peered":"2024-04-06T12:13:21.314125+0000","last_unstale":"2024-04-06T12:13:22.369310+0000","last_undegraded":"2024-04-06T12:13:22.369310+0000","last_fullsized":"2024-04-06T12:13:22.369310+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:52:36.596577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293092+0000","last_change":"2024-04-06T12:13:20.894824+0000","last_active":"2024-04-06T12:13:22.293092+0000","last_peered":"2024-04-06T12:13:22.293092+0000","last_clean":"2024-04-06T12:13:22.293092+0000","last_became_active":"2024-04-06T12:13:20.893644+0000","last_became_peered":"2024-04-06T12:13:20.893644+0000","last_unstale":"2024-04-06T12:13:22.293092+0000","last_undegraded":"2024-04-06T12:13:22.293092+0000","last_fullsized":"2024-04-06T12:13:22.293092+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:45:11.022279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293606+0000","last_change":"2024-04-06T12:13:21.310683+0000","last_active":"2024-04-06T12:13:22.293606+0000","last_peered":"2024-04-06T12:13:22.293606+0000","last_clean":"2024-04-06T12:13:22.293606+0000","last_became_active":"2024-04-06T12:13:21.308350+0000","last_became_peered":"2024-04-06T12:13:21.308350+0000","last_unstale":"2024-04-06T12:13:22.293606+0000","last_undegraded":"2024-04-06T12:13:22.293606+0000","last_fullsized":"2024-04-06T12:13:22.293606+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:12:48.159478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293454+0000","last_change":"2024-04-06T12:13:20.896087+0000","last_active":"2024-04-06T12:13:22.293454+0000","last_peered":"2024-04-06T12:13:22.293454+0000","last_clean":"2024-04-06T12:13:22.293454+0000","last_became_active":"2024-04-06T12:13:20.895819+0000","last_became_peered":"2024-04-06T12:13:20.895819+0000","last_unstale":"2024-04-06T12:13:22.293454+0000","last_undegraded":"2024-04-06T12:13:22.293454+0000","last_fullsized":"2024-04-06T12:13:22.293454+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:37:14.420774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368484+0000","last_change":"2024-04-06T12:13:21.318884+0000","last_active":"2024-04-06T12:13:22.368484+0000","last_peered":"2024-04-06T12:13:22.368484+0000","last_clean":"2024-04-06T12:13:22.368484+0000","last_became_active":"2024-04-06T12:13:21.318368+0000","last_became_peered":"2024-04-06T12:13:21.318368+0000","last_unstale":"2024-04-06T12:13:22.368484+0000","last_undegraded":"2024-04-06T12:13:22.368484+0000","last_fullsized":"2024-04-06T12:13:22.368484+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:49:24.182273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":19,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293790+0000","last_change":"2024-04-06T12:13:20.897418+0000","last_active":"2024-04-06T12:13:22.293790+0000","last_peered":"2024-04-06T12:13:22.293790+0000","last_clean":"2024-04-06T12:13:22.293790+0000","last_became_active":"2024-04-06T12:13:20.897159+0000","last_became_peered":"2024-04-06T12:13:20.897159+0000","last_unstale":"2024-04-06T12:13:22.293790+0000","last_undegraded":"2024-04-06T12:13:22.293790+0000","last_fullsized":"2024-04-06T12:13:22.293790+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:22:02.080030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368579+0000","last_change":"2024-04-06T12:13:21.319363+0000","last_active":"2024-04-06T12:13:22.368579+0000","last_peered":"2024-04-06T12:13:22.368579+0000","last_clean":"2024-04-06T12:13:22.368579+0000","last_became_active":"2024-04-06T12:13:21.318095+0000","last_became_peered":"2024-04-06T12:13:21.318095+0000","last_unstale":"2024-04-06T12:13:22.368579+0000","last_undegraded":"2024-04-06T12:13:22.368579+0000","last_fullsized":"2024-04-06T12:13:22.368579+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:24:27.755154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368768+0000","last_change":"2024-04-06T12:13:20.896444+0000","last_active":"2024-04-06T12:13:22.368768+0000","last_peered":"2024-04-06T12:13:22.368768+0000","last_clean":"2024-04-06T12:13:22.368768+0000","last_became_active":"2024-04-06T12:13:20.895602+0000","last_became_peered":"2024-04-06T12:13:20.895602+0000","last_unstale":"2024-04-06T12:13:22.368768+0000","last_undegraded":"2024-04-06T12:13:22.368768+0000","last_fullsized":"2024-04-06T12:13:22.368768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:38:06.515744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368245+0000","last_change":"2024-04-06T12:13:21.314552+0000","last_active":"2024-04-06T12:13:22.368245+0000","last_peered":"2024-04-06T12:13:22.368245+0000","last_clean":"2024-04-06T12:13:22.368245+0000","last_became_active":"2024-04-06T12:13:21.314233+0000","last_became_peered":"2024-04-06T12:13:21.314233+0000","last_unstale":"2024-04-06T12:13:22.368245+0000","last_undegraded":"2024-04-06T12:13:22.368245+0000","last_fullsized":"2024-04-06T12:13:22.368245+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:17:40.977598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369360+0000","last_change":"2024-04-06T12:13:20.893671+0000","last_active":"2024-04-06T12:13:22.369360+0000","last_peered":"2024-04-06T12:13:22.369360+0000","last_clean":"2024-04-06T12:13:22.369360+0000","last_became_active":"2024-04-06T12:13:20.893383+0000","last_became_peered":"2024-04-06T12:13:20.893383+0000","last_unstale":"2024-04-06T12:13:22.369360+0000","last_undegraded":"2024-04-06T12:13:22.369360+0000","last_fullsized":"2024-04-06T12:13:22.369360+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:20:58.659602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369274+0000","last_change":"2024-04-06T12:13:21.306280+0000","last_active":"2024-04-06T12:13:22.369274+0000","last_peered":"2024-04-06T12:13:22.369274+0000","last_clean":"2024-04-06T12:13:22.369274+0000","last_became_active":"2024-04-06T12:13:21.305980+0000","last_became_peered":"2024-04-06T12:13:21.305980+0000","last_unstale":"2024-04-06T12:13:22.369274+0000","last_undegraded":"2024-04-06T12:13:22.369274+0000","last_fullsized":"2024-04-06T12:13:22.369274+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:29:43.926143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368978+0000","last_change":"2024-04-06T12:13:20.897067+0000","last_active":"2024-04-06T12:13:22.368978+0000","last_peered":"2024-04-06T12:13:22.368978+0000","last_clean":"2024-04-06T12:13:22.368978+0000","last_became_active":"2024-04-06T12:13:20.896782+0000","last_became_peered":"2024-04-06T12:13:20.896782+0000","last_unstale":"2024-04-06T12:13:22.368978+0000","last_undegraded":"2024-04-06T12:13:22.368978+0000","last_fullsized":"2024-04-06T12:13:22.368978+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:25:39.354655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369197+0000","last_change":"2024-04-06T12:13:21.314760+0000","last_active":"2024-04-06T12:13:22.369197+0000","last_peered":"2024-04-06T12:13:22.369197+0000","last_clean":"2024-04-06T12:13:22.369197+0000","last_became_active":"2024-04-06T12:13:21.314498+0000","last_became_peered":"2024-04-06T12:13:21.314498+0000","last_unstale":"2024-04-06T12:13:22.369197+0000","last_undegraded":"2024-04-06T12:13:22.369197+0000","last_fullsized":"2024-04-06T12:13:22.369197+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:53:51.147798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.292905+0000","last_change":"2024-04-06T12:13:20.886151+0000","last_active":"2024-04-06T12:13:22.292905+0000","last_peered":"2024-04-06T12:13:22.292905+0000","last_clean":"2024-04-06T12:13:22.292905+0000","last_became_active":"2024-04-06T12:13:20.885883+0000","last_became_peered":"2024-04-06T12:13:20.885883+0000","last_unstale":"2024-04-06T12:13:22.292905+0000","last_undegraded":"2024-04-06T12:13:22.292905+0000","last_fullsized":"2024-04-06T12:13:22.292905+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:12:47.154213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293480+0000","last_change":"2024-04-06T12:13:21.311157+0000","last_active":"2024-04-06T12:13:22.293480+0000","last_peered":"2024-04-06T12:13:22.293480+0000","last_clean":"2024-04-06T12:13:22.293480+0000","last_became_active":"2024-04-06T12:13:21.310780+0000","last_became_peered":"2024-04-06T12:13:21.310780+0000","last_unstale":"2024-04-06T12:13:22.293480+0000","last_undegraded":"2024-04-06T12:13:22.293480+0000","last_fullsized":"2024-04-06T12:13:22.293480+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:20:59.664868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.367948+0000","last_change":"2024-04-06T12:13:20.891537+0000","last_active":"2024-04-06T12:13:22.367948+0000","last_peered":"2024-04-06T12:13:22.367948+0000","last_clean":"2024-04-06T12:13:22.367948+0000","last_became_active":"2024-04-06T12:13:20.891223+0000","last_became_peered":"2024-04-06T12:13:20.891223+0000","last_unstale":"2024-04-06T12:13:22.367948+0000","last_undegraded":"2024-04-06T12:13:22.367948+0000","last_fullsized":"2024-04-06T12:13:22.367948+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:20:49.414794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368376+0000","last_change":"2024-04-06T12:13:21.310442+0000","last_active":"2024-04-06T12:13:22.368376+0000","last_peered":"2024-04-06T12:13:22.368376+0000","last_clean":"2024-04-06T12:13:22.368376+0000","last_became_active":"2024-04-06T12:13:21.310207+0000","last_became_peered":"2024-04-06T12:13:21.310207+0000","last_unstale":"2024-04-06T12:13:22.368376+0000","last_undegraded":"2024-04-06T12:13:22.368376+0000","last_fullsized":"2024-04-06T12:13:22.368376+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:17:40.977598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368089+0000","last_change":"2024-04-06T12:13:20.894597+0000","last_active":"2024-04-06T12:13:22.368089+0000","last_peered":"2024-04-06T12:13:22.368089+0000","last_clean":"2024-04-06T12:13:22.368089+0000","last_became_active":"2024-04-06T12:13:20.894213+0000","last_became_peered":"2024-04-06T12:13:20.894213+0000","last_unstale":"2024-04-06T12:13:22.368089+0000","last_undegraded":"2024-04-06T12:13:22.368089+0000","last_fullsized":"2024-04-06T12:13:22.368089+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:30:01.607898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368206+0000","last_change":"2024-04-06T12:13:21.310004+0000","last_active":"2024-04-06T12:13:22.368206+0000","last_peered":"2024-04-06T12:13:22.368206+0000","last_clean":"2024-04-06T12:13:22.368206+0000","last_became_active":"2024-04-06T12:13:21.309673+0000","last_became_peered":"2024-04-06T12:13:21.309673+0000","last_unstale":"2024-04-06T12:13:22.368206+0000","last_undegraded":"2024-04-06T12:13:22.368206+0000","last_fullsized":"2024-04-06T12:13:22.368206+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:37:27.117856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293596+0000","last_change":"2024-04-06T12:13:20.898426+0000","last_active":"2024-04-06T12:13:22.293596+0000","last_peered":"2024-04-06T12:13:22.293596+0000","last_clean":"2024-04-06T12:13:22.293596+0000","last_became_active":"2024-04-06T12:13:20.898114+0000","last_became_peered":"2024-04-06T12:13:20.898114+0000","last_unstale":"2024-04-06T12:13:22.293596+0000","last_undegraded":"2024-04-06T12:13:22.293596+0000","last_fullsized":"2024-04-06T12:13:22.293596+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:37:14.420774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368575+0000","last_change":"2024-04-06T12:13:21.319258+0000","last_active":"2024-04-06T12:13:22.368575+0000","last_peered":"2024-04-06T12:13:22.368575+0000","last_clean":"2024-04-06T12:13:22.368575+0000","last_became_active":"2024-04-06T12:13:21.319017+0000","last_became_peered":"2024-04-06T12:13:21.319017+0000","last_unstale":"2024-04-06T12:13:22.368575+0000","last_undegraded":"2024-04-06T12:13:22.368575+0000","last_fullsized":"2024-04-06T12:13:22.368575+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:44:02.157487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368155+0000","last_change":"2024-04-06T12:13:20.891417+0000","last_active":"2024-04-06T12:13:22.368155+0000","last_peered":"2024-04-06T12:13:22.368155+0000","last_clean":"2024-04-06T12:13:22.368155+0000","last_became_active":"2024-04-06T12:13:20.891141+0000","last_became_peered":"2024-04-06T12:13:20.891141+0000","last_unstale":"2024-04-06T12:13:22.368155+0000","last_undegraded":"2024-04-06T12:13:22.368155+0000","last_fullsized":"2024-04-06T12:13:22.368155+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:17.138275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.292830+0000","last_change":"2024-04-06T12:13:21.313156+0000","last_active":"2024-04-06T12:13:22.292830+0000","last_peered":"2024-04-06T12:13:22.292830+0000","last_clean":"2024-04-06T12:13:22.292830+0000","last_became_active":"2024-04-06T12:13:21.312899+0000","last_became_peered":"2024-04-06T12:13:21.312899+0000","last_unstale":"2024-04-06T12:13:22.292830+0000","last_undegraded":"2024-04-06T12:13:22.292830+0000","last_fullsized":"2024-04-06T12:13:22.292830+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:06:39.941523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368818+0000","last_change":"2024-04-06T12:13:20.897723+0000","last_active":"2024-04-06T12:13:22.368818+0000","last_peered":"2024-04-06T12:13:22.368818+0000","last_clean":"2024-04-06T12:13:22.368818+0000","last_became_active":"2024-04-06T12:13:20.897450+0000","last_became_peered":"2024-04-06T12:13:20.897450+0000","last_unstale":"2024-04-06T12:13:22.368818+0000","last_undegraded":"2024-04-06T12:13:22.368818+0000","last_fullsized":"2024-04-06T12:13:22.368818+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:54:05.001865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293607+0000","last_change":"2024-04-06T12:13:21.311866+0000","last_active":"2024-04-06T12:13:22.293607+0000","last_peered":"2024-04-06T12:13:22.293607+0000","last_clean":"2024-04-06T12:13:22.293607+0000","last_became_active":"2024-04-06T12:13:21.311430+0000","last_became_peered":"2024-04-06T12:13:21.311430+0000","last_unstale":"2024-04-06T12:13:22.293607+0000","last_undegraded":"2024-04-06T12:13:22.293607+0000","last_fullsized":"2024-04-06T12:13:22.293607+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:18:13.817293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369013+0000","last_change":"2024-04-06T12:13:20.894358+0000","last_active":"2024-04-06T12:13:22.369013+0000","last_peered":"2024-04-06T12:13:22.369013+0000","last_clean":"2024-04-06T12:13:22.369013+0000","last_became_active":"2024-04-06T12:13:20.894085+0000","last_became_peered":"2024-04-06T12:13:20.894085+0000","last_unstale":"2024-04-06T12:13:22.369013+0000","last_undegraded":"2024-04-06T12:13:22.369013+0000","last_fullsized":"2024-04-06T12:13:22.369013+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:49:00.360938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368676+0000","last_change":"2024-04-06T12:13:21.318944+0000","last_active":"2024-04-06T12:13:22.368676+0000","last_peered":"2024-04-06T12:13:22.368676+0000","last_clean":"2024-04-06T12:13:22.368676+0000","last_became_active":"2024-04-06T12:13:21.318670+0000","last_became_peered":"2024-04-06T12:13:21.318670+0000","last_unstale":"2024-04-06T12:13:22.368676+0000","last_undegraded":"2024-04-06T12:13:22.368676+0000","last_fullsized":"2024-04-06T12:13:22.368676+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:48:53.537756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368461+0000","last_change":"2024-04-06T12:13:20.887045+0000","last_active":"2024-04-06T12:13:22.368461+0000","last_peered":"2024-04-06T12:13:22.368461+0000","last_clean":"2024-04-06T12:13:22.368461+0000","last_became_active":"2024-04-06T12:13:20.886784+0000","last_became_peered":"2024-04-06T12:13:20.886784+0000","last_unstale":"2024-04-06T12:13:22.368461+0000","last_undegraded":"2024-04-06T12:13:22.368461+0000","last_fullsized":"2024-04-06T12:13:22.368461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:44:36.927713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368121+0000","last_change":"2024-04-06T12:13:21.315549+0000","last_active":"2024-04-06T12:13:22.368121+0000","last_peered":"2024-04-06T12:13:22.368121+0000","last_clean":"2024-04-06T12:13:22.368121+0000","last_became_active":"2024-04-06T12:13:21.315250+0000","last_became_peered":"2024-04-06T12:13:21.315250+0000","last_unstale":"2024-04-06T12:13:22.368121+0000","last_undegraded":"2024-04-06T12:13:22.368121+0000","last_fullsized":"2024-04-06T12:13:22.368121+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:19:19.798635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368983+0000","last_change":"2024-04-06T12:13:20.896318+0000","last_active":"2024-04-06T12:13:22.368983+0000","last_peered":"2024-04-06T12:13:22.368983+0000","last_clean":"2024-04-06T12:13:22.368983+0000","last_became_active":"2024-04-06T12:13:20.896041+0000","last_became_peered":"2024-04-06T12:13:20.896041+0000","last_unstale":"2024-04-06T12:13:22.368983+0000","last_undegraded":"2024-04-06T12:13:22.368983+0000","last_fullsized":"2024-04-06T12:13:22.368983+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:20:58.659602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.292870+0000","last_change":"2024-04-06T12:13:21.310852+0000","last_active":"2024-04-06T12:13:22.292870+0000","last_peered":"2024-04-06T12:13:22.292870+0000","last_clean":"2024-04-06T12:13:22.292870+0000","last_became_active":"2024-04-06T12:13:21.310559+0000","last_became_peered":"2024-04-06T12:13:21.310559+0000","last_unstale":"2024-04-06T12:13:22.292870+0000","last_undegraded":"2024-04-06T12:13:22.292870+0000","last_fullsized":"2024-04-06T12:13:22.292870+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:19:19.798635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293810+0000","last_change":"2024-04-06T12:13:20.893519+0000","last_active":"2024-04-06T12:13:22.293810+0000","last_peered":"2024-04-06T12:13:22.293810+0000","last_clean":"2024-04-06T12:13:22.293810+0000","last_became_active":"2024-04-06T12:13:20.893267+0000","last_became_peered":"2024-04-06T12:13:20.893267+0000","last_unstale":"2024-04-06T12:13:22.293810+0000","last_undegraded":"2024-04-06T12:13:22.293810+0000","last_fullsized":"2024-04-06T12:13:22.293810+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:22:02.080030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368777+0000","last_change":"2024-04-06T12:13:21.318785+0000","last_active":"2024-04-06T12:13:22.368777+0000","last_peered":"2024-04-06T12:13:22.368777+0000","last_clean":"2024-04-06T12:13:22.368777+0000","last_became_active":"2024-04-06T12:13:21.318501+0000","last_became_peered":"2024-04-06T12:13:21.318501+0000","last_unstale":"2024-04-06T12:13:22.368777+0000","last_undegraded":"2024-04-06T12:13:22.368777+0000","last_fullsized":"2024-04-06T12:13:22.368777+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:06:39.941523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368521+0000","last_change":"2024-04-06T12:13:20.895139+0000","last_active":"2024-04-06T12:13:22.368521+0000","last_peered":"2024-04-06T12:13:22.368521+0000","last_clean":"2024-04-06T12:13:22.368521+0000","last_became_active":"2024-04-06T12:13:20.894846+0000","last_became_peered":"2024-04-06T12:13:20.894846+0000","last_unstale":"2024-04-06T12:13:22.368521+0000","last_undegraded":"2024-04-06T12:13:22.368521+0000","last_fullsized":"2024-04-06T12:13:22.368521+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:12:47.154213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293973+0000","last_change":"2024-04-06T12:13:21.311737+0000","last_active":"2024-04-06T12:13:22.293973+0000","last_peered":"2024-04-06T12:13:22.293973+0000","last_clean":"2024-04-06T12:13:22.293973+0000","last_became_active":"2024-04-06T12:13:21.311420+0000","last_became_peered":"2024-04-06T12:13:21.311420+0000","last_unstale":"2024-04-06T12:13:22.293973+0000","last_undegraded":"2024-04-06T12:13:22.293973+0000","last_fullsized":"2024-04-06T12:13:22.293973+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:54:06.007131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368129+0000","last_change":"2024-04-06T12:13:20.895166+0000","last_active":"2024-04-06T12:13:22.368129+0000","last_peered":"2024-04-06T12:13:22.368129+0000","last_clean":"2024-04-06T12:13:22.368129+0000","last_became_active":"2024-04-06T12:13:20.894882+0000","last_became_peered":"2024-04-06T12:13:20.894882+0000","last_unstale":"2024-04-06T12:13:22.368129+0000","last_undegraded":"2024-04-06T12:13:22.368129+0000","last_fullsized":"2024-04-06T12:13:22.368129+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:46:08.328919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368714+0000","last_change":"2024-04-06T12:13:21.315786+0000","last_active":"2024-04-06T12:13:22.368714+0000","last_peered":"2024-04-06T12:13:22.368714+0000","last_clean":"2024-04-06T12:13:22.368714+0000","last_became_active":"2024-04-06T12:13:21.315518+0000","last_became_peered":"2024-04-06T12:13:21.315518+0000","last_unstale":"2024-04-06T12:13:22.368714+0000","last_undegraded":"2024-04-06T12:13:22.368714+0000","last_fullsized":"2024-04-06T12:13:22.368714+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:19:19.798635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293694+0000","last_change":"2024-04-06T12:13:20.896439+0000","last_active":"2024-04-06T12:13:22.293694+0000","last_peered":"2024-04-06T12:13:22.293694+0000","last_clean":"2024-04-06T12:13:22.293694+0000","last_became_active":"2024-04-06T12:13:20.896156+0000","last_became_peered":"2024-04-06T12:13:20.896156+0000","last_unstale":"2024-04-06T12:13:22.293694+0000","last_undegraded":"2024-04-06T12:13:22.293694+0000","last_fullsized":"2024-04-06T12:13:22.293694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:49:00.360938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":183,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-06T12:13:21.302060+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:13:21.302060+0000","last_peered":"2024-04-06T12:13:21.302060+0000","last_clean":"2024-04-06T12:13:21.302060+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:13:21.302060+0000","last_undegraded":"2024-04-06T12:13:21.302060+0000","last_fullsized":"2024-04-06T12:13:21.302060+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28068,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343772,"statfs":{"total":96636764160,"available":96608022528,"internally_reserved":0,"allocated":1060864,"data_stored":786117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.47399999999999998}]},{"osd":1,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.60499999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.62}]},{"osd":3,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.52700000000000002}]},{"osd":4,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.69999999999999996}]},{"osd":5,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.71599999999999997}]},{"osd":6,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.63300000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":450560,"data_stored":193678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.436},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.55300000000000005}]},{"osd":1,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.58699999999999997}]},{"osd":2,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.45400000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.60399999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.66300000000000003}]},{"osd":5,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.67900000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.70799999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":438272,"data_stored":186566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.33900000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.315},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.36599999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.26500000000000001}]},{"osd":4,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":6.3739999999999997,"5min":6.3739999999999997,"15min":6.3739999999999997},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.495}]},{"osd":5,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.624},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.52000000000000002}]},{"osd":6,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.58899999999999997}]},{"osd":7,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.60399999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1044480,"data_stored":783498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.33400000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.8089999999999999,"5min":1.8089999999999999,"15min":1.8089999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":0.27700000000000002}]},{"osd":3,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.35899999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.626},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.77000000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.67400000000000004}]},{"osd":6,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.70199999999999996}]},{"osd":7,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.749}]}]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":450560,"data_stored":193166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.375}]},{"osd":1,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.6179999999999999,"5min":3.6179999999999999,"15min":3.6179999999999999},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.28100000000000003}]},{"osd":3,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":3.661,"5min":3.661,"15min":3.661},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.35799999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.68799999999999994}]},{"osd":5,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":3.6920000000000002,"5min":3.6920000000000002,"15min":3.6920000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.73499999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.7490000000000001,"5min":3.7490000000000001,"15min":3.7490000000000001},"last":0.67100000000000004}]},{"osd":7,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.7200000000000002,"5min":3.7200000000000002,"15min":3.7200000000000002},"last":0.747}]}]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27488,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344352,"statfs":{"total":96636764160,"available":96608616448,"internally_reserved":0,"allocated":466944,"data_stored":195713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"last":0.32600000000000001}]},{"osd":1,"last update":"Sat Apr 6 12:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.38300000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.311},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.39900000000000002}]},{"osd":4,"last update":"Sat Apr 6 12:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.68400000000000005}]},{"osd":5,"last update":"Sat Apr 6 12:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.72099999999999997}]},{"osd":6,"last update":"Sat Apr 6 12:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.73099999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.70899999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":446464,"data_stored":188099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.82199999999999995}]},{"osd":1,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":4.3479999999999999,"5min":4.3479999999999999,"15min":4.3479999999999999},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":4.3319999999999999,"5min":4.3319999999999999,"15min":4.3319999999999999},"last":0.80900000000000005}]},{"osd":2,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.71499999999999997}]},{"osd":3,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":4.1950000000000003,"5min":4.1950000000000003,"15min":4.1950000000000003},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.39400000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":1.196},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.66400000000000003}]},{"osd":6,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.76000000000000001}]},{"osd":7,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.77700000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":442368,"data_stored":186588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.41699999999999998}]},{"osd":1,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.57499999999999996}]},{"osd":2,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.59599999999999997}]},{"osd":3,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.48699999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.55100000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.56599999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:13:25.402 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:13:25.558 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.559+0000 7fece4240700 1 Processor -- start 2024-04-06T12:13:25.558 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.560+0000 7fece4240700 1 -- start start 2024-04-06T12:13:25.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.560+0000 7fece4240700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fecdc06ae40 0x7fecdc0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.560+0000 7fece4240700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecdc15ddf0 0x7fecdc15e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.560+0000 7fece4240700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecdc0b3990 con 0x7fecdc15d7b0 2024-04-06T12:13:25.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.560+0000 7fece4240700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecdc0b3060 con 0x7fecdc06ae40 2024-04-06T12:13:25.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.560+0000 7fece4240700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecdc0b3340 con 0x7fecdc15ddf0 2024-04-06T12:13:25.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.561+0000 7fece17db700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fecdc06ae40 0x7fecdc0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:25.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.561+0000 7fece1fdc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fecdc15d7b0 0x7fecdc06a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33518/0 (socket says 172.21.15.189:33518) 2024-04-06T12:13:25.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.561+0000 7fece1fdc700 1 -- 172.21.15.189:0/288852395 learned_addr learned my addr 172.21.15.189:0/288852395 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:25.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.561+0000 7fece27dd700 1 --2- 172.21.15.189:0/288852395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecdc15ddf0 0x7fecdc15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:25.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.561+0000 7fece17db700 1 -- 172.21.15.189:0/288852395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecdc15ddf0 msgr2=0x7fecdc15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:25.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.561+0000 7fece17db700 1 --2- 172.21.15.189:0/288852395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecdc15ddf0 0x7fecdc15e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.561+0000 7fece17db700 1 -- 172.21.15.189:0/288852395 >> v1:172.21.15.189:6789/0 conn(0x7fecdc15d7b0 legacy=0x7fecdc06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:25.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.561+0000 7fece17db700 1 -- 172.21.15.189:0/288852395 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecdc0b3620 con 0x7fecdc06ae40 2024-04-06T12:13:25.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.561+0000 7fece0fda700 1 -- 172.21.15.189:0/288852395 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471893906 0 0) 0x7fecdc0b3990 con 0x7fecdc15d7b0 2024-04-06T12:13:25.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.561+0000 7fece17db700 1 --2- 172.21.15.189:0/288852395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecdc06ae40 0x7fecdc0b3e20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fecd8009840 tx=0x7fecd8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=603fbd64705895e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:25.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.561+0000 7fece0fda700 1 -- 172.21.15.189:0/288852395 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd8011830 con 0x7fecdc06ae40 2024-04-06T12:13:25.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.561+0000 7fece0fda700 1 -- 172.21.15.189:0/288852395 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd8011990 con 0x7fecdc06ae40 2024-04-06T12:13:25.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.561+0000 7fece0fda700 1 -- 172.21.15.189:0/288852395 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd8011d20 con 0x7fecdc06ae40 2024-04-06T12:13:25.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.562+0000 7fece4240700 1 -- 172.21.15.189:0/288852395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecdc06ae40 msgr2=0x7fecdc0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:25.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.562+0000 7fece4240700 1 --2- 172.21.15.189:0/288852395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecdc06ae40 0x7fecdc0b3e20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fecd8009840 tx=0x7fecd8009b50 comp rx=0 tx=0).stop 2024-04-06T12:13:25.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.562+0000 7fece4240700 1 -- 172.21.15.189:0/288852395 shutdown_connections 2024-04-06T12:13:25.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.562+0000 7fece4240700 1 --2- 172.21.15.189:0/288852395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecdc15ddf0 0x7fecdc15e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.562+0000 7fece4240700 1 --2- 172.21.15.189:0/288852395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecdc06ae40 0x7fecdc0b3e20 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.562+0000 7fece4240700 1 -- 172.21.15.189:0/288852395 >> 172.21.15.189:0/288852395 conn(0x7fecdc0d00b0 msgr2=0x7fecdc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:25.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.562+0000 7fece4240700 1 -- 172.21.15.189:0/288852395 shutdown_connections 2024-04-06T12:13:25.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.562+0000 7fece4240700 1 -- 172.21.15.189:0/288852395 wait complete. 2024-04-06T12:13:25.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.563+0000 7fece4240700 1 Processor -- start 2024-04-06T12:13:25.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.563+0000 7fece4240700 1 -- start start 2024-04-06T12:13:25.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.564+0000 7fece4240700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecdc06ae40 0x7fecdc160920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.564+0000 7fece4240700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecdc15d7b0 0x7fecdc0b0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.564+0000 7fece4240700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecdc15ddf0 0x7fecdc14c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.564+0000 7fece4240700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecdc0c4f30 con 0x7fecdc15ddf0 2024-04-06T12:13:25.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.564+0000 7fece4240700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecdc0b0ad0 con 0x7fecdc06ae40 2024-04-06T12:13:25.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.564+0000 7fece4240700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecdc0b0da0 con 0x7fecdc15d7b0 2024-04-06T12:13:25.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.564+0000 7fece27dd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecdc15ddf0 0x7fecdc14c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:25.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.564+0000 7fece27dd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecdc15ddf0 0x7fecdc14c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33214/0 (socket says 172.21.15.189:33214) 2024-04-06T12:13:25.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.564+0000 7fece27dd700 1 -- 172.21.15.189:0/885423616 learned_addr learned my addr 172.21.15.189:0/885423616 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:25.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.564+0000 7fece1fdc700 1 --2- 172.21.15.189:0/885423616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecdc06ae40 0x7fecdc160920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:25.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.564+0000 7fece17db700 1 --2- 172.21.15.189:0/885423616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecdc15d7b0 0x7fecdc0b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:25.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.564+0000 7fece27dd700 1 -- 172.21.15.189:0/885423616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecdc15d7b0 msgr2=0x7fecdc0b0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:25.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.564+0000 7fece27dd700 1 --2- 172.21.15.189:0/885423616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecdc15d7b0 0x7fecdc0b0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.564+0000 7fece27dd700 1 -- 172.21.15.189:0/885423616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecdc06ae40 msgr2=0x7fecdc160920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:25.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.564+0000 7fece27dd700 1 --2- 172.21.15.189:0/885423616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecdc06ae40 0x7fecdc160920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.564+0000 7fece27dd700 1 -- 172.21.15.189:0/885423616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd80094a0 con 0x7fecdc15ddf0 2024-04-06T12:13:25.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.564+0000 7fece1fdc700 1 --2- 172.21.15.189:0/885423616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecdc06ae40 0x7fecdc160920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:25.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.564+0000 7fece27dd700 1 --2- 172.21.15.189:0/885423616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecdc15ddf0 0x7fecdc14c7c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fecd400efc0 tx=0x7fecd400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:25.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.565+0000 7fecd27fc700 1 -- 172.21.15.189:0/885423616 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd4009ba0 con 0x7fecdc15ddf0 2024-04-06T12:13:25.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.565+0000 7fecd27fc700 1 -- 172.21.15.189:0/885423616 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd400f040 con 0x7fecdc15ddf0 2024-04-06T12:13:25.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.565+0000 7fece4240700 1 -- 172.21.15.189:0/885423616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecdc0b1080 con 0x7fecdc15ddf0 2024-04-06T12:13:25.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.565+0000 7fecd27fc700 1 -- 172.21.15.189:0/885423616 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd4014480 con 0x7fecdc15ddf0 2024-04-06T12:13:25.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.565+0000 7fece4240700 1 -- 172.21.15.189:0/885423616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecdc0b1640 con 0x7fecdc15ddf0 2024-04-06T12:13:25.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.566+0000 7fece4240700 1 -- 172.21.15.189:0/885423616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecdc067bb0 con 0x7fecdc15ddf0 2024-04-06T12:13:25.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.566+0000 7fecd27fc700 1 -- 172.21.15.189:0/885423616 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fecd40145e0 con 0x7fecdc15ddf0 2024-04-06T12:13:25.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.567+0000 7fecd27fc700 1 --2- 172.21.15.189:0/885423616 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fecc006ac90 0x7fecc006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.567+0000 7fecd27fc700 1 -- 172.21.15.189:0/885423616 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fecd4089db0 con 0x7fecdc15ddf0 2024-04-06T12:13:25.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.567+0000 7fece1fdc700 1 --2- 172.21.15.189:0/885423616 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fecc006ac90 0x7fecc006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:25.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.567+0000 7fece1fdc700 1 --2- 172.21.15.189:0/885423616 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fecc006ac90 0x7fecc006d150 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7feccc000f60 tx=0x7feccc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:25.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.571+0000 7fecd27fc700 1 -- 172.21.15.189:0/885423616 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fecd408e070 con 0x7fecdc15ddf0 2024-04-06T12:13:25.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.761+0000 7fece4240700 1 -- 172.21.15.189:0/885423616 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fecdc15e7c0 con 0x7fecc006ac90 2024-04-06T12:13:25.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.769+0000 7fecd27fc700 1 -- 172.21.15.189:0/885423616 <== mgr.4100 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289715 (secure 0 0 0) 0x7feccc001cc0 con 0x7fecc006ac90 2024-04-06T12:13:25.770 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:13:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.772+0000 7fece4240700 1 -- 172.21.15.189:0/885423616 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fecc006ac90 msgr2=0x7fecc006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.772+0000 7fece4240700 1 --2- 172.21.15.189:0/885423616 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fecc006ac90 0x7fecc006d150 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7feccc000f60 tx=0x7feccc009040 comp rx=0 tx=0).stop 2024-04-06T12:13:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.772+0000 7fece4240700 1 -- 172.21.15.189:0/885423616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecdc15ddf0 msgr2=0x7fecdc14c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.772+0000 7fece4240700 1 --2- 172.21.15.189:0/885423616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecdc15ddf0 0x7fecdc14c7c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fecd400efc0 tx=0x7fecd400c3a0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.772+0000 7fece4240700 1 -- 172.21.15.189:0/885423616 shutdown_connections 2024-04-06T12:13:25.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.772+0000 7fece4240700 1 --2- 172.21.15.189:0/885423616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecdc15ddf0 0x7fecdc14c7c0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.772+0000 7fece4240700 1 --2- 172.21.15.189:0/885423616 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fecc006ac90 0x7fecc006d150 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.772+0000 7fece4240700 1 --2- 172.21.15.189:0/885423616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecdc06ae40 0x7fecdc160920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.772+0000 7fece4240700 1 --2- 172.21.15.189:0/885423616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecdc15d7b0 0x7fecdc0b0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.772+0000 7fece4240700 1 -- 172.21.15.189:0/885423616 >> 172.21.15.189:0/885423616 conn(0x7fecdc0d00b0 msgr2=0x7fecdc15c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:25.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.772+0000 7fece4240700 1 -- 172.21.15.189:0/885423616 shutdown_connections 2024-04-06T12:13:25.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.772+0000 7fece4240700 1 -- 172.21.15.189:0/885423616 wait complete. 2024-04-06T12:13:25.773 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:13:25.782 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-04-06T12:13:24.583141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220928,"kb_used_data":4688,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753792,"statfs":{"total":773094113280,"available":772867883008,"internally_reserved":0,"allocated":4800512,"data_stored":2713425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.714287"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368838+0000","last_change":"2024-04-06T12:13:21.314648+0000","last_active":"2024-04-06T12:13:22.368838+0000","last_peered":"2024-04-06T12:13:22.368838+0000","last_clean":"2024-04-06T12:13:22.368838+0000","last_became_active":"2024-04-06T12:13:21.314388+0000","last_became_peered":"2024-04-06T12:13:21.314388+0000","last_unstale":"2024-04-06T12:13:22.368838+0000","last_undegraded":"2024-04-06T12:13:22.368838+0000","last_fullsized":"2024-04-06T12:13:22.368838+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:08:08.716105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368574+0000","last_change":"2024-04-06T12:13:21.324261+0000","last_active":"2024-04-06T12:13:22.368574+0000","last_peered":"2024-04-06T12:13:22.368574+0000","last_clean":"2024-04-06T12:13:22.368574+0000","last_became_active":"2024-04-06T12:13:21.324018+0000","last_became_peered":"2024-04-06T12:13:21.324018+0000","last_unstale":"2024-04-06T12:13:22.368574+0000","last_undegraded":"2024-04-06T12:13:22.368574+0000","last_fullsized":"2024-04-06T12:13:22.368574+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:27:48.973861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368382+0000","last_change":"2024-04-06T12:13:21.307590+0000","last_active":"2024-04-06T12:13:22.368382+0000","last_peered":"2024-04-06T12:13:22.368382+0000","last_clean":"2024-04-06T12:13:22.368382+0000","last_became_active":"2024-04-06T12:13:21.307318+0000","last_became_peered":"2024-04-06T12:13:21.307318+0000","last_unstale":"2024-04-06T12:13:22.368382+0000","last_undegraded":"2024-04-06T12:13:22.368382+0000","last_fullsized":"2024-04-06T12:13:22.368382+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:06:39.941523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368107+0000","last_change":"2024-04-06T12:13:21.308565+0000","last_active":"2024-04-06T12:13:22.368107+0000","last_peered":"2024-04-06T12:13:22.368107+0000","last_clean":"2024-04-06T12:13:22.368107+0000","last_became_active":"2024-04-06T12:13:21.308328+0000","last_became_peered":"2024-04-06T12:13:21.308328+0000","last_unstale":"2024-04-06T12:13:22.368107+0000","last_undegraded":"2024-04-06T12:13:22.368107+0000","last_fullsized":"2024-04-06T12:13:22.368107+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:14:53.401297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.367923+0000","last_change":"2024-04-06T12:13:21.298073+0000","last_active":"2024-04-06T12:13:22.367923+0000","last_peered":"2024-04-06T12:13:22.367923+0000","last_clean":"2024-04-06T12:13:22.367923+0000","last_became_active":"2024-04-06T12:13:21.297463+0000","last_became_peered":"2024-04-06T12:13:21.297463+0000","last_unstale":"2024-04-06T12:13:22.367923+0000","last_undegraded":"2024-04-06T12:13:22.367923+0000","last_fullsized":"2024-04-06T12:13:22.367923+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:44:09.673989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293010+0000","last_change":"2024-04-06T12:13:21.314937+0000","last_active":"2024-04-06T12:13:22.293010+0000","last_peered":"2024-04-06T12:13:22.293010+0000","last_clean":"2024-04-06T12:13:22.293010+0000","last_became_active":"2024-04-06T12:13:21.314677+0000","last_became_peered":"2024-04-06T12:13:21.314677+0000","last_unstale":"2024-04-06T12:13:22.293010+0000","last_undegraded":"2024-04-06T12:13:22.293010+0000","last_fullsized":"2024-04-06T12:13:22.293010+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:46:09.334185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369543+0000","last_change":"2024-04-06T12:13:21.315764+0000","last_active":"2024-04-06T12:13:22.369543+0000","last_peered":"2024-04-06T12:13:22.369543+0000","last_clean":"2024-04-06T12:13:22.369543+0000","last_became_active":"2024-04-06T12:13:21.315483+0000","last_became_peered":"2024-04-06T12:13:21.315483+0000","last_unstale":"2024-04-06T12:13:22.369543+0000","last_undegraded":"2024-04-06T12:13:22.369543+0000","last_fullsized":"2024-04-06T12:13:22.369543+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:53:38.483235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368052+0000","last_change":"2024-04-06T12:13:21.314740+0000","last_active":"2024-04-06T12:13:22.368052+0000","last_peered":"2024-04-06T12:13:22.368052+0000","last_clean":"2024-04-06T12:13:22.368052+0000","last_became_active":"2024-04-06T12:13:21.314477+0000","last_became_peered":"2024-04-06T12:13:21.314477+0000","last_unstale":"2024-04-06T12:13:22.368052+0000","last_undegraded":"2024-04-06T12:13:22.368052+0000","last_fullsized":"2024-04-06T12:13:22.368052+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:12:47.226472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368444+0000","last_change":"2024-04-06T12:13:21.313198+0000","last_active":"2024-04-06T12:13:22.368444+0000","last_peered":"2024-04-06T12:13:22.368444+0000","last_clean":"2024-04-06T12:13:22.368444+0000","last_became_active":"2024-04-06T12:13:21.312843+0000","last_became_peered":"2024-04-06T12:13:21.312843+0000","last_unstale":"2024-04-06T12:13:22.368444+0000","last_undegraded":"2024-04-06T12:13:22.368444+0000","last_fullsized":"2024-04-06T12:13:22.368444+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:41:19.205834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369663+0000","last_change":"2024-04-06T12:13:21.312756+0000","last_active":"2024-04-06T12:13:22.369663+0000","last_peered":"2024-04-06T12:13:22.369663+0000","last_clean":"2024-04-06T12:13:22.369663+0000","last_became_active":"2024-04-06T12:13:21.312386+0000","last_became_peered":"2024-04-06T12:13:21.312386+0000","last_unstale":"2024-04-06T12:13:22.369663+0000","last_undegraded":"2024-04-06T12:13:22.369663+0000","last_fullsized":"2024-04-06T12:13:22.369663+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:06:39.941523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368347+0000","last_change":"2024-04-06T12:13:21.321020+0000","last_active":"2024-04-06T12:13:22.368347+0000","last_peered":"2024-04-06T12:13:22.368347+0000","last_clean":"2024-04-06T12:13:22.368347+0000","last_became_active":"2024-04-06T12:13:21.320752+0000","last_became_peered":"2024-04-06T12:13:21.320752+0000","last_unstale":"2024-04-06T12:13:22.368347+0000","last_undegraded":"2024-04-06T12:13:22.368347+0000","last_fullsized":"2024-04-06T12:13:22.368347+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:01:46.620598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368263+0000","last_change":"2024-04-06T12:13:21.302283+0000","last_active":"2024-04-06T12:13:22.368263+0000","last_peered":"2024-04-06T12:13:22.368263+0000","last_clean":"2024-04-06T12:13:22.368263+0000","last_became_active":"2024-04-06T12:13:21.302109+0000","last_became_peered":"2024-04-06T12:13:21.302109+0000","last_unstale":"2024-04-06T12:13:22.368263+0000","last_undegraded":"2024-04-06T12:13:22.368263+0000","last_fullsized":"2024-04-06T12:13:22.368263+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:53:38.483235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368453+0000","last_change":"2024-04-06T12:13:21.312054+0000","last_active":"2024-04-06T12:13:22.368453+0000","last_peered":"2024-04-06T12:13:22.368453+0000","last_clean":"2024-04-06T12:13:22.368453+0000","last_became_active":"2024-04-06T12:13:21.311792+0000","last_became_peered":"2024-04-06T12:13:21.311792+0000","last_unstale":"2024-04-06T12:13:22.368453+0000","last_undegraded":"2024-04-06T12:13:22.368453+0000","last_fullsized":"2024-04-06T12:13:22.368453+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:29:43.926143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368447+0000","last_change":"2024-04-06T12:13:21.314135+0000","last_active":"2024-04-06T12:13:22.368447+0000","last_peered":"2024-04-06T12:13:22.368447+0000","last_clean":"2024-04-06T12:13:22.368447+0000","last_became_active":"2024-04-06T12:13:21.313888+0000","last_became_peered":"2024-04-06T12:13:21.313888+0000","last_unstale":"2024-04-06T12:13:22.368447+0000","last_undegraded":"2024-04-06T12:13:22.368447+0000","last_fullsized":"2024-04-06T12:13:22.368447+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:01.313739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.367937+0000","last_change":"2024-04-06T12:13:21.312811+0000","last_active":"2024-04-06T12:13:22.367937+0000","last_peered":"2024-04-06T12:13:22.367937+0000","last_clean":"2024-04-06T12:13:22.367937+0000","last_became_active":"2024-04-06T12:13:21.312482+0000","last_became_peered":"2024-04-06T12:13:21.312482+0000","last_unstale":"2024-04-06T12:13:22.367937+0000","last_undegraded":"2024-04-06T12:13:22.367937+0000","last_fullsized":"2024-04-06T12:13:22.367937+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:13:29.653277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293360+0000","last_change":"2024-04-06T12:13:21.313055+0000","last_active":"2024-04-06T12:13:22.293360+0000","last_peered":"2024-04-06T12:13:22.293360+0000","last_clean":"2024-04-06T12:13:22.293360+0000","last_became_active":"2024-04-06T12:13:21.312260+0000","last_became_peered":"2024-04-06T12:13:21.312260+0000","last_unstale":"2024-04-06T12:13:22.293360+0000","last_undegraded":"2024-04-06T12:13:22.293360+0000","last_fullsized":"2024-04-06T12:13:22.293360+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:53:51.147798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293848+0000","last_change":"2024-04-06T12:13:21.312705+0000","last_active":"2024-04-06T12:13:22.293848+0000","last_peered":"2024-04-06T12:13:22.293848+0000","last_clean":"2024-04-06T12:13:22.293848+0000","last_became_active":"2024-04-06T12:13:21.312019+0000","last_became_peered":"2024-04-06T12:13:21.312019+0000","last_unstale":"2024-04-06T12:13:22.293848+0000","last_undegraded":"2024-04-06T12:13:22.293848+0000","last_fullsized":"2024-04-06T12:13:22.293848+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:45:12.027545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.294427+0000","last_change":"2024-04-06T12:13:21.317041+0000","last_active":"2024-04-06T12:13:22.294427+0000","last_peered":"2024-04-06T12:13:22.294427+0000","last_clean":"2024-04-06T12:13:22.294427+0000","last_became_active":"2024-04-06T12:13:21.316786+0000","last_became_peered":"2024-04-06T12:13:21.316786+0000","last_unstale":"2024-04-06T12:13:22.294427+0000","last_undegraded":"2024-04-06T12:13:22.294427+0000","last_fullsized":"2024-04-06T12:13:22.294427+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:15:56.195554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368886+0000","last_change":"2024-04-06T12:13:21.300603+0000","last_active":"2024-04-06T12:13:22.368886+0000","last_peered":"2024-04-06T12:13:22.368886+0000","last_clean":"2024-04-06T12:13:22.368886+0000","last_became_active":"2024-04-06T12:13:21.300338+0000","last_became_peered":"2024-04-06T12:13:21.300338+0000","last_unstale":"2024-04-06T12:13:22.368886+0000","last_undegraded":"2024-04-06T12:13:22.368886+0000","last_fullsized":"2024-04-06T12:13:22.368886+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:45:15.762266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.294007+0000","last_change":"2024-04-06T12:13:21.316540+0000","last_active":"2024-04-06T12:13:22.294007+0000","last_peered":"2024-04-06T12:13:22.294007+0000","last_clean":"2024-04-06T12:13:22.294007+0000","last_became_active":"2024-04-06T12:13:21.316267+0000","last_became_peered":"2024-04-06T12:13:21.316267+0000","last_unstale":"2024-04-06T12:13:22.294007+0000","last_undegraded":"2024-04-06T12:13:22.294007+0000","last_fullsized":"2024-04-06T12:13:22.294007+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:50:45.615242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368164+0000","last_change":"2024-04-06T12:13:21.311462+0000","last_active":"2024-04-06T12:13:22.368164+0000","last_peered":"2024-04-06T12:13:22.368164+0000","last_clean":"2024-04-06T12:13:22.368164+0000","last_became_active":"2024-04-06T12:13:21.311187+0000","last_became_peered":"2024-04-06T12:13:21.311187+0000","last_unstale":"2024-04-06T12:13:22.368164+0000","last_undegraded":"2024-04-06T12:13:22.368164+0000","last_fullsized":"2024-04-06T12:13:22.368164+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:41:19.205834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368566+0000","last_change":"2024-04-06T12:13:21.308118+0000","last_active":"2024-04-06T12:13:22.368566+0000","last_peered":"2024-04-06T12:13:22.368566+0000","last_clean":"2024-04-06T12:13:22.368566+0000","last_became_active":"2024-04-06T12:13:21.307712+0000","last_became_peered":"2024-04-06T12:13:21.307712+0000","last_unstale":"2024-04-06T12:13:22.368566+0000","last_undegraded":"2024-04-06T12:13:22.368566+0000","last_fullsized":"2024-04-06T12:13:22.368566+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:19:19.798635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293486+0000","last_change":"2024-04-06T12:13:21.310424+0000","last_active":"2024-04-06T12:13:22.293486+0000","last_peered":"2024-04-06T12:13:22.293486+0000","last_clean":"2024-04-06T12:13:22.293486+0000","last_became_active":"2024-04-06T12:13:21.308250+0000","last_became_peered":"2024-04-06T12:13:21.308250+0000","last_unstale":"2024-04-06T12:13:22.293486+0000","last_undegraded":"2024-04-06T12:13:22.293486+0000","last_fullsized":"2024-04-06T12:13:22.293486+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:44:37.932979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368070+0000","last_change":"2024-04-06T12:13:21.312586+0000","last_active":"2024-04-06T12:13:22.368070+0000","last_peered":"2024-04-06T12:13:22.368070+0000","last_clean":"2024-04-06T12:13:22.368070+0000","last_became_active":"2024-04-06T12:13:21.312104+0000","last_became_peered":"2024-04-06T12:13:21.312104+0000","last_unstale":"2024-04-06T12:13:22.368070+0000","last_undegraded":"2024-04-06T12:13:22.368070+0000","last_fullsized":"2024-04-06T12:13:22.368070+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:54:06.007131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.294211+0000","last_change":"2024-04-06T12:13:21.317835+0000","last_active":"2024-04-06T12:13:22.294211+0000","last_peered":"2024-04-06T12:13:22.294211+0000","last_clean":"2024-04-06T12:13:22.294211+0000","last_became_active":"2024-04-06T12:13:21.317576+0000","last_became_peered":"2024-04-06T12:13:21.317576+0000","last_unstale":"2024-04-06T12:13:22.294211+0000","last_undegraded":"2024-04-06T12:13:22.294211+0000","last_fullsized":"2024-04-06T12:13:22.294211+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:54:06.007131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293033+0000","last_change":"2024-04-06T12:13:21.313596+0000","last_active":"2024-04-06T12:13:22.293033+0000","last_peered":"2024-04-06T12:13:22.293033+0000","last_clean":"2024-04-06T12:13:22.293033+0000","last_became_active":"2024-04-06T12:13:21.313334+0000","last_became_peered":"2024-04-06T12:13:21.313334+0000","last_unstale":"2024-04-06T12:13:22.293033+0000","last_undegraded":"2024-04-06T12:13:22.293033+0000","last_fullsized":"2024-04-06T12:13:22.293033+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:41:19.205834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293972+0000","last_change":"2024-04-06T12:13:21.313694+0000","last_active":"2024-04-06T12:13:22.293972+0000","last_peered":"2024-04-06T12:13:22.293972+0000","last_clean":"2024-04-06T12:13:22.293972+0000","last_became_active":"2024-04-06T12:13:21.313407+0000","last_became_peered":"2024-04-06T12:13:21.313407+0000","last_unstale":"2024-04-06T12:13:22.293972+0000","last_undegraded":"2024-04-06T12:13:22.293972+0000","last_fullsized":"2024-04-06T12:13:22.293972+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:57:05.900931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369453+0000","last_change":"2024-04-06T12:13:21.316069+0000","last_active":"2024-04-06T12:13:22.369453+0000","last_peered":"2024-04-06T12:13:22.369453+0000","last_clean":"2024-04-06T12:13:22.369453+0000","last_became_active":"2024-04-06T12:13:21.315815+0000","last_became_peered":"2024-04-06T12:13:21.315815+0000","last_unstale":"2024-04-06T12:13:22.369453+0000","last_undegraded":"2024-04-06T12:13:22.369453+0000","last_fullsized":"2024-04-06T12:13:22.369453+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:45:04.876402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.294218+0000","last_change":"2024-04-06T12:13:21.317125+0000","last_active":"2024-04-06T12:13:22.294218+0000","last_peered":"2024-04-06T12:13:22.294218+0000","last_clean":"2024-04-06T12:13:22.294218+0000","last_became_active":"2024-04-06T12:13:21.316868+0000","last_became_peered":"2024-04-06T12:13:21.316868+0000","last_unstale":"2024-04-06T12:13:22.294218+0000","last_undegraded":"2024-04-06T12:13:22.294218+0000","last_fullsized":"2024-04-06T12:13:22.294218+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:53:51.147798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293716+0000","last_change":"2024-04-06T12:13:21.318843+0000","last_active":"2024-04-06T12:13:22.293716+0000","last_peered":"2024-04-06T12:13:22.293716+0000","last_clean":"2024-04-06T12:13:22.293716+0000","last_became_active":"2024-04-06T12:13:21.318565+0000","last_became_peered":"2024-04-06T12:13:21.318565+0000","last_unstale":"2024-04-06T12:13:22.293716+0000","last_undegraded":"2024-04-06T12:13:22.293716+0000","last_fullsized":"2024-04-06T12:13:22.293716+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:48:58.061525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293362+0000","last_change":"2024-04-06T12:13:21.309134+0000","last_active":"2024-04-06T12:13:22.293362+0000","last_peered":"2024-04-06T12:13:22.293362+0000","last_clean":"2024-04-06T12:13:22.293362+0000","last_became_active":"2024-04-06T12:13:21.308873+0000","last_became_peered":"2024-04-06T12:13:21.308873+0000","last_unstale":"2024-04-06T12:13:22.293362+0000","last_undegraded":"2024-04-06T12:13:22.293362+0000","last_fullsized":"2024-04-06T12:13:22.293362+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:57:07.685321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293120+0000","last_change":"2024-04-06T12:13:21.303727+0000","last_active":"2024-04-06T12:13:22.293120+0000","last_peered":"2024-04-06T12:13:22.293120+0000","last_clean":"2024-04-06T12:13:22.293120+0000","last_became_active":"2024-04-06T12:13:21.303521+0000","last_became_peered":"2024-04-06T12:13:21.303521+0000","last_unstale":"2024-04-06T12:13:22.293120+0000","last_undegraded":"2024-04-06T12:13:22.293120+0000","last_fullsized":"2024-04-06T12:13:22.293120+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:13:29.653277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293542+0000","last_change":"2024-04-06T12:13:21.316236+0000","last_active":"2024-04-06T12:13:22.293542+0000","last_peered":"2024-04-06T12:13:22.293542+0000","last_clean":"2024-04-06T12:13:22.293542+0000","last_became_active":"2024-04-06T12:13:21.315956+0000","last_became_peered":"2024-04-06T12:13:21.315956+0000","last_unstale":"2024-04-06T12:13:22.293542+0000","last_undegraded":"2024-04-06T12:13:22.293542+0000","last_fullsized":"2024-04-06T12:13:22.293542+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:50:45.615242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.294292+0000","last_change":"2024-04-06T12:13:20.897573+0000","last_active":"2024-04-06T12:13:22.294292+0000","last_peered":"2024-04-06T12:13:22.294292+0000","last_clean":"2024-04-06T12:13:22.294292+0000","last_became_active":"2024-04-06T12:13:20.897299+0000","last_became_peered":"2024-04-06T12:13:20.897299+0000","last_unstale":"2024-04-06T12:13:22.294292+0000","last_undegraded":"2024-04-06T12:13:22.294292+0000","last_fullsized":"2024-04-06T12:13:22.294292+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:19:10.009040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293172+0000","last_change":"2024-04-06T12:13:21.316550+0000","last_active":"2024-04-06T12:13:22.293172+0000","last_peered":"2024-04-06T12:13:22.293172+0000","last_clean":"2024-04-06T12:13:22.293172+0000","last_became_active":"2024-04-06T12:13:21.316325+0000","last_became_peered":"2024-04-06T12:13:21.316325+0000","last_unstale":"2024-04-06T12:13:22.293172+0000","last_undegraded":"2024-04-06T12:13:22.293172+0000","last_fullsized":"2024-04-06T12:13:22.293172+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:17:40.977598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369175+0000","last_change":"2024-04-06T12:13:20.897660+0000","last_active":"2024-04-06T12:13:22.369175+0000","last_peered":"2024-04-06T12:13:22.369175+0000","last_clean":"2024-04-06T12:13:22.369175+0000","last_became_active":"2024-04-06T12:13:20.897405+0000","last_became_peered":"2024-04-06T12:13:20.897405+0000","last_unstale":"2024-04-06T12:13:22.369175+0000","last_undegraded":"2024-04-06T12:13:22.369175+0000","last_fullsized":"2024-04-06T12:13:22.369175+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:12:47.154213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369178+0000","last_change":"2024-04-06T12:13:21.309155+0000","last_active":"2024-04-06T12:13:22.369178+0000","last_peered":"2024-04-06T12:13:22.369178+0000","last_clean":"2024-04-06T12:13:22.369178+0000","last_became_active":"2024-04-06T12:13:21.308879+0000","last_became_peered":"2024-04-06T12:13:21.308879+0000","last_unstale":"2024-04-06T12:13:22.369178+0000","last_undegraded":"2024-04-06T12:13:22.369178+0000","last_fullsized":"2024-04-06T12:13:22.369178+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:17:40.977598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293195+0000","last_change":"2024-04-06T12:13:20.884932+0000","last_active":"2024-04-06T12:13:22.293195+0000","last_peered":"2024-04-06T12:13:22.293195+0000","last_clean":"2024-04-06T12:13:22.293195+0000","last_became_active":"2024-04-06T12:13:20.884681+0000","last_became_peered":"2024-04-06T12:13:20.884681+0000","last_unstale":"2024-04-06T12:13:22.293195+0000","last_undegraded":"2024-04-06T12:13:22.293195+0000","last_fullsized":"2024-04-06T12:13:22.293195+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:57:06.680055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368984+0000","last_change":"2024-04-06T12:13:21.317904+0000","last_active":"2024-04-06T12:13:22.368984+0000","last_peered":"2024-04-06T12:13:22.368984+0000","last_clean":"2024-04-06T12:13:22.368984+0000","last_became_active":"2024-04-06T12:13:21.317652+0000","last_became_peered":"2024-04-06T12:13:21.317652+0000","last_unstale":"2024-04-06T12:13:22.368984+0000","last_undegraded":"2024-04-06T12:13:22.368984+0000","last_fullsized":"2024-04-06T12:13:22.368984+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:17:40.174838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368705+0000","last_change":"2024-04-06T12:13:20.901272+0000","last_active":"2024-04-06T12:13:22.368705+0000","last_peered":"2024-04-06T12:13:22.368705+0000","last_clean":"2024-04-06T12:13:22.368705+0000","last_became_active":"2024-04-06T12:13:20.900984+0000","last_became_peered":"2024-04-06T12:13:20.900984+0000","last_unstale":"2024-04-06T12:13:22.368705+0000","last_undegraded":"2024-04-06T12:13:22.368705+0000","last_fullsized":"2024-04-06T12:13:22.368705+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:57:06.680055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368214+0000","last_change":"2024-04-06T12:13:21.303808+0000","last_active":"2024-04-06T12:13:22.368214+0000","last_peered":"2024-04-06T12:13:22.368214+0000","last_clean":"2024-04-06T12:13:22.368214+0000","last_became_active":"2024-04-06T12:13:21.303549+0000","last_became_peered":"2024-04-06T12:13:21.303549+0000","last_unstale":"2024-04-06T12:13:22.368214+0000","last_undegraded":"2024-04-06T12:13:22.368214+0000","last_fullsized":"2024-04-06T12:13:22.368214+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:37:27.117856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368289+0000","last_change":"2024-04-06T12:13:20.896675+0000","last_active":"2024-04-06T12:13:22.368289+0000","last_peered":"2024-04-06T12:13:22.368289+0000","last_clean":"2024-04-06T12:13:22.368289+0000","last_became_active":"2024-04-06T12:13:20.896395+0000","last_became_peered":"2024-04-06T12:13:20.896395+0000","last_unstale":"2024-04-06T12:13:22.368289+0000","last_undegraded":"2024-04-06T12:13:22.368289+0000","last_fullsized":"2024-04-06T12:13:22.368289+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:14:52.396032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.367910+0000","last_change":"2024-04-06T12:13:21.309885+0000","last_active":"2024-04-06T12:13:22.367910+0000","last_peered":"2024-04-06T12:13:22.367910+0000","last_clean":"2024-04-06T12:13:22.367910+0000","last_became_active":"2024-04-06T12:13:21.309596+0000","last_became_peered":"2024-04-06T12:13:21.309596+0000","last_unstale":"2024-04-06T12:13:22.367910+0000","last_undegraded":"2024-04-06T12:13:22.367910+0000","last_fullsized":"2024-04-06T12:13:22.367910+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:46:09.334185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293693+0000","last_change":"2024-04-06T12:13:20.897399+0000","last_active":"2024-04-06T12:13:22.293693+0000","last_peered":"2024-04-06T12:13:22.293693+0000","last_clean":"2024-04-06T12:13:22.293693+0000","last_became_active":"2024-04-06T12:13:20.897116+0000","last_became_peered":"2024-04-06T12:13:20.897116+0000","last_unstale":"2024-04-06T12:13:22.293693+0000","last_undegraded":"2024-04-06T12:13:22.293693+0000","last_fullsized":"2024-04-06T12:13:22.293693+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:19:10.009040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368683+0000","last_change":"2024-04-06T12:13:21.313322+0000","last_active":"2024-04-06T12:13:22.368683+0000","last_peered":"2024-04-06T12:13:22.368683+0000","last_clean":"2024-04-06T12:13:22.368683+0000","last_became_active":"2024-04-06T12:13:21.313042+0000","last_became_peered":"2024-04-06T12:13:21.313042+0000","last_unstale":"2024-04-06T12:13:22.368683+0000","last_undegraded":"2024-04-06T12:13:22.368683+0000","last_fullsized":"2024-04-06T12:13:22.368683+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:41:19.205834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.292966+0000","last_change":"2024-04-06T12:13:20.894229+0000","last_active":"2024-04-06T12:13:22.292966+0000","last_peered":"2024-04-06T12:13:22.292966+0000","last_clean":"2024-04-06T12:13:22.292966+0000","last_became_active":"2024-04-06T12:13:20.893620+0000","last_became_peered":"2024-04-06T12:13:20.893620+0000","last_unstale":"2024-04-06T12:13:22.292966+0000","last_undegraded":"2024-04-06T12:13:22.292966+0000","last_fullsized":"2024-04-06T12:13:22.292966+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:18:12.812027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368677+0000","last_change":"2024-04-06T12:13:21.318741+0000","last_active":"2024-04-06T12:13:22.368677+0000","last_peered":"2024-04-06T12:13:22.368677+0000","last_clean":"2024-04-06T12:13:22.368677+0000","last_became_active":"2024-04-06T12:13:21.318070+0000","last_became_peered":"2024-04-06T12:13:21.318070+0000","last_unstale":"2024-04-06T12:13:22.368677+0000","last_undegraded":"2024-04-06T12:13:22.368677+0000","last_fullsized":"2024-04-06T12:13:22.368677+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:53:09.306045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368366+0000","last_change":"2024-04-06T12:13:20.892999+0000","last_active":"2024-04-06T12:13:22.368366+0000","last_peered":"2024-04-06T12:13:22.368366+0000","last_clean":"2024-04-06T12:13:22.368366+0000","last_became_active":"2024-04-06T12:13:20.892205+0000","last_became_peered":"2024-04-06T12:13:20.892205+0000","last_unstale":"2024-04-06T12:13:22.368366+0000","last_undegraded":"2024-04-06T12:13:22.368366+0000","last_fullsized":"2024-04-06T12:13:22.368366+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:18:12.812027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368199+0000","last_change":"2024-04-06T12:13:21.297738+0000","last_active":"2024-04-06T12:13:22.368199+0000","last_peered":"2024-04-06T12:13:22.368199+0000","last_clean":"2024-04-06T12:13:22.368199+0000","last_became_active":"2024-04-06T12:13:21.297493+0000","last_became_peered":"2024-04-06T12:13:21.297493+0000","last_unstale":"2024-04-06T12:13:22.368199+0000","last_undegraded":"2024-04-06T12:13:22.368199+0000","last_fullsized":"2024-04-06T12:13:22.368199+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:38:07.521009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369057+0000","last_change":"2024-04-06T12:13:20.897312+0000","last_active":"2024-04-06T12:13:22.369057+0000","last_peered":"2024-04-06T12:13:22.369057+0000","last_clean":"2024-04-06T12:13:22.369057+0000","last_became_active":"2024-04-06T12:13:20.896922+0000","last_became_peered":"2024-04-06T12:13:20.896922+0000","last_unstale":"2024-04-06T12:13:22.369057+0000","last_undegraded":"2024-04-06T12:13:22.369057+0000","last_fullsized":"2024-04-06T12:13:22.369057+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:06:38.936257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293362+0000","last_change":"2024-04-06T12:13:21.316416+0000","last_active":"2024-04-06T12:13:22.293362+0000","last_peered":"2024-04-06T12:13:22.293362+0000","last_clean":"2024-04-06T12:13:22.293362+0000","last_became_active":"2024-04-06T12:13:21.311801+0000","last_became_peered":"2024-04-06T12:13:21.311801+0000","last_unstale":"2024-04-06T12:13:22.293362+0000","last_undegraded":"2024-04-06T12:13:22.293362+0000","last_fullsized":"2024-04-06T12:13:22.293362+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:53:40.611211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369721+0000","last_change":"2024-04-06T12:13:20.895917+0000","last_active":"2024-04-06T12:13:22.369721+0000","last_peered":"2024-04-06T12:13:22.369721+0000","last_clean":"2024-04-06T12:13:22.369721+0000","last_became_active":"2024-04-06T12:13:20.895648+0000","last_became_peered":"2024-04-06T12:13:20.895648+0000","last_unstale":"2024-04-06T12:13:22.369721+0000","last_undegraded":"2024-04-06T12:13:22.369721+0000","last_fullsized":"2024-04-06T12:13:22.369721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:44:08.668723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369094+0000","last_change":"2024-04-06T12:13:21.311428+0000","last_active":"2024-04-06T12:13:22.369094+0000","last_peered":"2024-04-06T12:13:22.369094+0000","last_clean":"2024-04-06T12:13:22.369094+0000","last_became_active":"2024-04-06T12:13:21.311163+0000","last_became_peered":"2024-04-06T12:13:21.311163+0000","last_unstale":"2024-04-06T12:13:22.369094+0000","last_undegraded":"2024-04-06T12:13:22.369094+0000","last_fullsized":"2024-04-06T12:13:22.369094+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:01.313739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368407+0000","last_change":"2024-04-06T12:13:20.897032+0000","last_active":"2024-04-06T12:13:22.368407+0000","last_peered":"2024-04-06T12:13:22.368407+0000","last_clean":"2024-04-06T12:13:22.368407+0000","last_became_active":"2024-04-06T12:13:20.896789+0000","last_became_peered":"2024-04-06T12:13:20.896789+0000","last_unstale":"2024-04-06T12:13:22.368407+0000","last_undegraded":"2024-04-06T12:13:22.368407+0000","last_fullsized":"2024-04-06T12:13:22.368407+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:53:50.142533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368915+0000","last_change":"2024-04-06T12:13:21.306919+0000","last_active":"2024-04-06T12:13:22.368915+0000","last_peered":"2024-04-06T12:13:22.368915+0000","last_clean":"2024-04-06T12:13:22.368915+0000","last_became_active":"2024-04-06T12:13:21.302205+0000","last_became_peered":"2024-04-06T12:13:21.302205+0000","last_unstale":"2024-04-06T12:13:22.368915+0000","last_undegraded":"2024-04-06T12:13:22.368915+0000","last_fullsized":"2024-04-06T12:13:22.368915+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:46:09.334185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368743+0000","last_change":"2024-04-06T12:13:20.895861+0000","last_active":"2024-04-06T12:13:22.368743+0000","last_peered":"2024-04-06T12:13:22.368743+0000","last_clean":"2024-04-06T12:13:22.368743+0000","last_became_active":"2024-04-06T12:13:20.895407+0000","last_became_peered":"2024-04-06T12:13:20.895407+0000","last_unstale":"2024-04-06T12:13:22.368743+0000","last_undegraded":"2024-04-06T12:13:22.368743+0000","last_fullsized":"2024-04-06T12:13:22.368743+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:19:18.793370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.367797+0000","last_change":"2024-04-06T12:13:21.313364+0000","last_active":"2024-04-06T12:13:22.367797+0000","last_peered":"2024-04-06T12:13:22.367797+0000","last_clean":"2024-04-06T12:13:22.367797+0000","last_became_active":"2024-04-06T12:13:21.312871+0000","last_became_peered":"2024-04-06T12:13:21.312871+0000","last_unstale":"2024-04-06T12:13:22.367797+0000","last_undegraded":"2024-04-06T12:13:22.367797+0000","last_fullsized":"2024-04-06T12:13:22.367797+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:46:09.334185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368773+0000","last_change":"2024-04-06T12:13:20.896174+0000","last_active":"2024-04-06T12:13:22.368773+0000","last_peered":"2024-04-06T12:13:22.368773+0000","last_clean":"2024-04-06T12:13:22.368773+0000","last_became_active":"2024-04-06T12:13:20.895869+0000","last_became_peered":"2024-04-06T12:13:20.895869+0000","last_unstale":"2024-04-06T12:13:22.368773+0000","last_undegraded":"2024-04-06T12:13:22.368773+0000","last_fullsized":"2024-04-06T12:13:22.368773+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:41:18.200569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369310+0000","last_change":"2024-04-06T12:13:21.314464+0000","last_active":"2024-04-06T12:13:22.369310+0000","last_peered":"2024-04-06T12:13:22.369310+0000","last_clean":"2024-04-06T12:13:22.369310+0000","last_became_active":"2024-04-06T12:13:21.314125+0000","last_became_peered":"2024-04-06T12:13:21.314125+0000","last_unstale":"2024-04-06T12:13:22.369310+0000","last_undegraded":"2024-04-06T12:13:22.369310+0000","last_fullsized":"2024-04-06T12:13:22.369310+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:52:36.596577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293092+0000","last_change":"2024-04-06T12:13:20.894824+0000","last_active":"2024-04-06T12:13:22.293092+0000","last_peered":"2024-04-06T12:13:22.293092+0000","last_clean":"2024-04-06T12:13:22.293092+0000","last_became_active":"2024-04-06T12:13:20.893644+0000","last_became_peered":"2024-04-06T12:13:20.893644+0000","last_unstale":"2024-04-06T12:13:22.293092+0000","last_undegraded":"2024-04-06T12:13:22.293092+0000","last_fullsized":"2024-04-06T12:13:22.293092+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:45:11.022279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293606+0000","last_change":"2024-04-06T12:13:21.310683+0000","last_active":"2024-04-06T12:13:22.293606+0000","last_peered":"2024-04-06T12:13:22.293606+0000","last_clean":"2024-04-06T12:13:22.293606+0000","last_became_active":"2024-04-06T12:13:21.308350+0000","last_became_peered":"2024-04-06T12:13:21.308350+0000","last_unstale":"2024-04-06T12:13:22.293606+0000","last_undegraded":"2024-04-06T12:13:22.293606+0000","last_fullsized":"2024-04-06T12:13:22.293606+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:12:48.159478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293454+0000","last_change":"2024-04-06T12:13:20.896087+0000","last_active":"2024-04-06T12:13:22.293454+0000","last_peered":"2024-04-06T12:13:22.293454+0000","last_clean":"2024-04-06T12:13:22.293454+0000","last_became_active":"2024-04-06T12:13:20.895819+0000","last_became_peered":"2024-04-06T12:13:20.895819+0000","last_unstale":"2024-04-06T12:13:22.293454+0000","last_undegraded":"2024-04-06T12:13:22.293454+0000","last_fullsized":"2024-04-06T12:13:22.293454+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:37:14.420774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368484+0000","last_change":"2024-04-06T12:13:21.318884+0000","last_active":"2024-04-06T12:13:22.368484+0000","last_peered":"2024-04-06T12:13:22.368484+0000","last_clean":"2024-04-06T12:13:22.368484+0000","last_became_active":"2024-04-06T12:13:21.318368+0000","last_became_peered":"2024-04-06T12:13:21.318368+0000","last_unstale":"2024-04-06T12:13:22.368484+0000","last_undegraded":"2024-04-06T12:13:22.368484+0000","last_fullsized":"2024-04-06T12:13:22.368484+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:49:24.182273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":19,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293790+0000","last_change":"2024-04-06T12:13:20.897418+0000","last_active":"2024-04-06T12:13:22.293790+0000","last_peered":"2024-04-06T12:13:22.293790+0000","last_clean":"2024-04-06T12:13:22.293790+0000","last_became_active":"2024-04-06T12:13:20.897159+0000","last_became_peered":"2024-04-06T12:13:20.897159+0000","last_unstale":"2024-04-06T12:13:22.293790+0000","last_undegraded":"2024-04-06T12:13:22.293790+0000","last_fullsized":"2024-04-06T12:13:22.293790+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:22:02.080030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368579+0000","last_change":"2024-04-06T12:13:21.319363+0000","last_active":"2024-04-06T12:13:22.368579+0000","last_peered":"2024-04-06T12:13:22.368579+0000","last_clean":"2024-04-06T12:13:22.368579+0000","last_became_active":"2024-04-06T12:13:21.318095+0000","last_became_peered":"2024-04-06T12:13:21.318095+0000","last_unstale":"2024-04-06T12:13:22.368579+0000","last_undegraded":"2024-04-06T12:13:22.368579+0000","last_fullsized":"2024-04-06T12:13:22.368579+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:24:27.755154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368768+0000","last_change":"2024-04-06T12:13:20.896444+0000","last_active":"2024-04-06T12:13:22.368768+0000","last_peered":"2024-04-06T12:13:22.368768+0000","last_clean":"2024-04-06T12:13:22.368768+0000","last_became_active":"2024-04-06T12:13:20.895602+0000","last_became_peered":"2024-04-06T12:13:20.895602+0000","last_unstale":"2024-04-06T12:13:22.368768+0000","last_undegraded":"2024-04-06T12:13:22.368768+0000","last_fullsized":"2024-04-06T12:13:22.368768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:38:06.515744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368245+0000","last_change":"2024-04-06T12:13:21.314552+0000","last_active":"2024-04-06T12:13:22.368245+0000","last_peered":"2024-04-06T12:13:22.368245+0000","last_clean":"2024-04-06T12:13:22.368245+0000","last_became_active":"2024-04-06T12:13:21.314233+0000","last_became_peered":"2024-04-06T12:13:21.314233+0000","last_unstale":"2024-04-06T12:13:22.368245+0000","last_undegraded":"2024-04-06T12:13:22.368245+0000","last_fullsized":"2024-04-06T12:13:22.368245+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:17:40.977598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369360+0000","last_change":"2024-04-06T12:13:20.893671+0000","last_active":"2024-04-06T12:13:22.369360+0000","last_peered":"2024-04-06T12:13:22.369360+0000","last_clean":"2024-04-06T12:13:22.369360+0000","last_became_active":"2024-04-06T12:13:20.893383+0000","last_became_peered":"2024-04-06T12:13:20.893383+0000","last_unstale":"2024-04-06T12:13:22.369360+0000","last_undegraded":"2024-04-06T12:13:22.369360+0000","last_fullsized":"2024-04-06T12:13:22.369360+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:20:58.659602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369274+0000","last_change":"2024-04-06T12:13:21.306280+0000","last_active":"2024-04-06T12:13:22.369274+0000","last_peered":"2024-04-06T12:13:22.369274+0000","last_clean":"2024-04-06T12:13:22.369274+0000","last_became_active":"2024-04-06T12:13:21.305980+0000","last_became_peered":"2024-04-06T12:13:21.305980+0000","last_unstale":"2024-04-06T12:13:22.369274+0000","last_undegraded":"2024-04-06T12:13:22.369274+0000","last_fullsized":"2024-04-06T12:13:22.369274+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:29:43.926143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368978+0000","last_change":"2024-04-06T12:13:20.897067+0000","last_active":"2024-04-06T12:13:22.368978+0000","last_peered":"2024-04-06T12:13:22.368978+0000","last_clean":"2024-04-06T12:13:22.368978+0000","last_became_active":"2024-04-06T12:13:20.896782+0000","last_became_peered":"2024-04-06T12:13:20.896782+0000","last_unstale":"2024-04-06T12:13:22.368978+0000","last_undegraded":"2024-04-06T12:13:22.368978+0000","last_fullsized":"2024-04-06T12:13:22.368978+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:25:39.354655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369197+0000","last_change":"2024-04-06T12:13:21.314760+0000","last_active":"2024-04-06T12:13:22.369197+0000","last_peered":"2024-04-06T12:13:22.369197+0000","last_clean":"2024-04-06T12:13:22.369197+0000","last_became_active":"2024-04-06T12:13:21.314498+0000","last_became_peered":"2024-04-06T12:13:21.314498+0000","last_unstale":"2024-04-06T12:13:22.369197+0000","last_undegraded":"2024-04-06T12:13:22.369197+0000","last_fullsized":"2024-04-06T12:13:22.369197+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:53:51.147798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.292905+0000","last_change":"2024-04-06T12:13:20.886151+0000","last_active":"2024-04-06T12:13:22.292905+0000","last_peered":"2024-04-06T12:13:22.292905+0000","last_clean":"2024-04-06T12:13:22.292905+0000","last_became_active":"2024-04-06T12:13:20.885883+0000","last_became_peered":"2024-04-06T12:13:20.885883+0000","last_unstale":"2024-04-06T12:13:22.292905+0000","last_undegraded":"2024-04-06T12:13:22.292905+0000","last_fullsized":"2024-04-06T12:13:22.292905+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:12:47.154213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293480+0000","last_change":"2024-04-06T12:13:21.311157+0000","last_active":"2024-04-06T12:13:22.293480+0000","last_peered":"2024-04-06T12:13:22.293480+0000","last_clean":"2024-04-06T12:13:22.293480+0000","last_became_active":"2024-04-06T12:13:21.310780+0000","last_became_peered":"2024-04-06T12:13:21.310780+0000","last_unstale":"2024-04-06T12:13:22.293480+0000","last_undegraded":"2024-04-06T12:13:22.293480+0000","last_fullsized":"2024-04-06T12:13:22.293480+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:20:59.664868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.367948+0000","last_change":"2024-04-06T12:13:20.891537+0000","last_active":"2024-04-06T12:13:22.367948+0000","last_peered":"2024-04-06T12:13:22.367948+0000","last_clean":"2024-04-06T12:13:22.367948+0000","last_became_active":"2024-04-06T12:13:20.891223+0000","last_became_peered":"2024-04-06T12:13:20.891223+0000","last_unstale":"2024-04-06T12:13:22.367948+0000","last_undegraded":"2024-04-06T12:13:22.367948+0000","last_fullsized":"2024-04-06T12:13:22.367948+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:20:49.414794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368376+0000","last_change":"2024-04-06T12:13:21.310442+0000","last_active":"2024-04-06T12:13:22.368376+0000","last_peered":"2024-04-06T12:13:22.368376+0000","last_clean":"2024-04-06T12:13:22.368376+0000","last_became_active":"2024-04-06T12:13:21.310207+0000","last_became_peered":"2024-04-06T12:13:21.310207+0000","last_unstale":"2024-04-06T12:13:22.368376+0000","last_undegraded":"2024-04-06T12:13:22.368376+0000","last_fullsized":"2024-04-06T12:13:22.368376+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:17:40.977598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368089+0000","last_change":"2024-04-06T12:13:20.894597+0000","last_active":"2024-04-06T12:13:22.368089+0000","last_peered":"2024-04-06T12:13:22.368089+0000","last_clean":"2024-04-06T12:13:22.368089+0000","last_became_active":"2024-04-06T12:13:20.894213+0000","last_became_peered":"2024-04-06T12:13:20.894213+0000","last_unstale":"2024-04-06T12:13:22.368089+0000","last_undegraded":"2024-04-06T12:13:22.368089+0000","last_fullsized":"2024-04-06T12:13:22.368089+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:30:01.607898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368206+0000","last_change":"2024-04-06T12:13:21.310004+0000","last_active":"2024-04-06T12:13:22.368206+0000","last_peered":"2024-04-06T12:13:22.368206+0000","last_clean":"2024-04-06T12:13:22.368206+0000","last_became_active":"2024-04-06T12:13:21.309673+0000","last_became_peered":"2024-04-06T12:13:21.309673+0000","last_unstale":"2024-04-06T12:13:22.368206+0000","last_undegraded":"2024-04-06T12:13:22.368206+0000","last_fullsized":"2024-04-06T12:13:22.368206+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:37:27.117856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293596+0000","last_change":"2024-04-06T12:13:20.898426+0000","last_active":"2024-04-06T12:13:22.293596+0000","last_peered":"2024-04-06T12:13:22.293596+0000","last_clean":"2024-04-06T12:13:22.293596+0000","last_became_active":"2024-04-06T12:13:20.898114+0000","last_became_peered":"2024-04-06T12:13:20.898114+0000","last_unstale":"2024-04-06T12:13:22.293596+0000","last_undegraded":"2024-04-06T12:13:22.293596+0000","last_fullsized":"2024-04-06T12:13:22.293596+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:37:14.420774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368575+0000","last_change":"2024-04-06T12:13:21.319258+0000","last_active":"2024-04-06T12:13:22.368575+0000","last_peered":"2024-04-06T12:13:22.368575+0000","last_clean":"2024-04-06T12:13:22.368575+0000","last_became_active":"2024-04-06T12:13:21.319017+0000","last_became_peered":"2024-04-06T12:13:21.319017+0000","last_unstale":"2024-04-06T12:13:22.368575+0000","last_undegraded":"2024-04-06T12:13:22.368575+0000","last_fullsized":"2024-04-06T12:13:22.368575+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:44:02.157487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368155+0000","last_change":"2024-04-06T12:13:20.891417+0000","last_active":"2024-04-06T12:13:22.368155+0000","last_peered":"2024-04-06T12:13:22.368155+0000","last_clean":"2024-04-06T12:13:22.368155+0000","last_became_active":"2024-04-06T12:13:20.891141+0000","last_became_peered":"2024-04-06T12:13:20.891141+0000","last_unstale":"2024-04-06T12:13:22.368155+0000","last_undegraded":"2024-04-06T12:13:22.368155+0000","last_fullsized":"2024-04-06T12:13:22.368155+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:17.138275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.292830+0000","last_change":"2024-04-06T12:13:21.313156+0000","last_active":"2024-04-06T12:13:22.292830+0000","last_peered":"2024-04-06T12:13:22.292830+0000","last_clean":"2024-04-06T12:13:22.292830+0000","last_became_active":"2024-04-06T12:13:21.312899+0000","last_became_peered":"2024-04-06T12:13:21.312899+0000","last_unstale":"2024-04-06T12:13:22.292830+0000","last_undegraded":"2024-04-06T12:13:22.292830+0000","last_fullsized":"2024-04-06T12:13:22.292830+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:06:39.941523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368818+0000","last_change":"2024-04-06T12:13:20.897723+0000","last_active":"2024-04-06T12:13:22.368818+0000","last_peered":"2024-04-06T12:13:22.368818+0000","last_clean":"2024-04-06T12:13:22.368818+0000","last_became_active":"2024-04-06T12:13:20.897450+0000","last_became_peered":"2024-04-06T12:13:20.897450+0000","last_unstale":"2024-04-06T12:13:22.368818+0000","last_undegraded":"2024-04-06T12:13:22.368818+0000","last_fullsized":"2024-04-06T12:13:22.368818+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:54:05.001865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293607+0000","last_change":"2024-04-06T12:13:21.311866+0000","last_active":"2024-04-06T12:13:22.293607+0000","last_peered":"2024-04-06T12:13:22.293607+0000","last_clean":"2024-04-06T12:13:22.293607+0000","last_became_active":"2024-04-06T12:13:21.311430+0000","last_became_peered":"2024-04-06T12:13:21.311430+0000","last_unstale":"2024-04-06T12:13:22.293607+0000","last_undegraded":"2024-04-06T12:13:22.293607+0000","last_fullsized":"2024-04-06T12:13:22.293607+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:18:13.817293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.369013+0000","last_change":"2024-04-06T12:13:20.894358+0000","last_active":"2024-04-06T12:13:22.369013+0000","last_peered":"2024-04-06T12:13:22.369013+0000","last_clean":"2024-04-06T12:13:22.369013+0000","last_became_active":"2024-04-06T12:13:20.894085+0000","last_became_peered":"2024-04-06T12:13:20.894085+0000","last_unstale":"2024-04-06T12:13:22.369013+0000","last_undegraded":"2024-04-06T12:13:22.369013+0000","last_fullsized":"2024-04-06T12:13:22.369013+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:49:00.360938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368676+0000","last_change":"2024-04-06T12:13:21.318944+0000","last_active":"2024-04-06T12:13:22.368676+0000","last_peered":"2024-04-06T12:13:22.368676+0000","last_clean":"2024-04-06T12:13:22.368676+0000","last_became_active":"2024-04-06T12:13:21.318670+0000","last_became_peered":"2024-04-06T12:13:21.318670+0000","last_unstale":"2024-04-06T12:13:22.368676+0000","last_undegraded":"2024-04-06T12:13:22.368676+0000","last_fullsized":"2024-04-06T12:13:22.368676+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:48:53.537756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368461+0000","last_change":"2024-04-06T12:13:20.887045+0000","last_active":"2024-04-06T12:13:22.368461+0000","last_peered":"2024-04-06T12:13:22.368461+0000","last_clean":"2024-04-06T12:13:22.368461+0000","last_became_active":"2024-04-06T12:13:20.886784+0000","last_became_peered":"2024-04-06T12:13:20.886784+0000","last_unstale":"2024-04-06T12:13:22.368461+0000","last_undegraded":"2024-04-06T12:13:22.368461+0000","last_fullsized":"2024-04-06T12:13:22.368461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:44:36.927713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368121+0000","last_change":"2024-04-06T12:13:21.315549+0000","last_active":"2024-04-06T12:13:22.368121+0000","last_peered":"2024-04-06T12:13:22.368121+0000","last_clean":"2024-04-06T12:13:22.368121+0000","last_became_active":"2024-04-06T12:13:21.315250+0000","last_became_peered":"2024-04-06T12:13:21.315250+0000","last_unstale":"2024-04-06T12:13:22.368121+0000","last_undegraded":"2024-04-06T12:13:22.368121+0000","last_fullsized":"2024-04-06T12:13:22.368121+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:19:19.798635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368983+0000","last_change":"2024-04-06T12:13:20.896318+0000","last_active":"2024-04-06T12:13:22.368983+0000","last_peered":"2024-04-06T12:13:22.368983+0000","last_clean":"2024-04-06T12:13:22.368983+0000","last_became_active":"2024-04-06T12:13:20.896041+0000","last_became_peered":"2024-04-06T12:13:20.896041+0000","last_unstale":"2024-04-06T12:13:22.368983+0000","last_undegraded":"2024-04-06T12:13:22.368983+0000","last_fullsized":"2024-04-06T12:13:22.368983+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:20:58.659602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.292870+0000","last_change":"2024-04-06T12:13:21.310852+0000","last_active":"2024-04-06T12:13:22.292870+0000","last_peered":"2024-04-06T12:13:22.292870+0000","last_clean":"2024-04-06T12:13:22.292870+0000","last_became_active":"2024-04-06T12:13:21.310559+0000","last_became_peered":"2024-04-06T12:13:21.310559+0000","last_unstale":"2024-04-06T12:13:22.292870+0000","last_undegraded":"2024-04-06T12:13:22.292870+0000","last_fullsized":"2024-04-06T12:13:22.292870+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:19:19.798635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293810+0000","last_change":"2024-04-06T12:13:20.893519+0000","last_active":"2024-04-06T12:13:22.293810+0000","last_peered":"2024-04-06T12:13:22.293810+0000","last_clean":"2024-04-06T12:13:22.293810+0000","last_became_active":"2024-04-06T12:13:20.893267+0000","last_became_peered":"2024-04-06T12:13:20.893267+0000","last_unstale":"2024-04-06T12:13:22.293810+0000","last_undegraded":"2024-04-06T12:13:22.293810+0000","last_fullsized":"2024-04-06T12:13:22.293810+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:22:02.080030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368777+0000","last_change":"2024-04-06T12:13:21.318785+0000","last_active":"2024-04-06T12:13:22.368777+0000","last_peered":"2024-04-06T12:13:22.368777+0000","last_clean":"2024-04-06T12:13:22.368777+0000","last_became_active":"2024-04-06T12:13:21.318501+0000","last_became_peered":"2024-04-06T12:13:21.318501+0000","last_unstale":"2024-04-06T12:13:22.368777+0000","last_undegraded":"2024-04-06T12:13:22.368777+0000","last_fullsized":"2024-04-06T12:13:22.368777+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:06:39.941523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368521+0000","last_change":"2024-04-06T12:13:20.895139+0000","last_active":"2024-04-06T12:13:22.368521+0000","last_peered":"2024-04-06T12:13:22.368521+0000","last_clean":"2024-04-06T12:13:22.368521+0000","last_became_active":"2024-04-06T12:13:20.894846+0000","last_became_peered":"2024-04-06T12:13:20.894846+0000","last_unstale":"2024-04-06T12:13:22.368521+0000","last_undegraded":"2024-04-06T12:13:22.368521+0000","last_fullsized":"2024-04-06T12:13:22.368521+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:12:47.154213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293973+0000","last_change":"2024-04-06T12:13:21.311737+0000","last_active":"2024-04-06T12:13:22.293973+0000","last_peered":"2024-04-06T12:13:22.293973+0000","last_clean":"2024-04-06T12:13:22.293973+0000","last_became_active":"2024-04-06T12:13:21.311420+0000","last_became_peered":"2024-04-06T12:13:21.311420+0000","last_unstale":"2024-04-06T12:13:22.293973+0000","last_undegraded":"2024-04-06T12:13:22.293973+0000","last_fullsized":"2024-04-06T12:13:22.293973+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:54:06.007131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368129+0000","last_change":"2024-04-06T12:13:20.895166+0000","last_active":"2024-04-06T12:13:22.368129+0000","last_peered":"2024-04-06T12:13:22.368129+0000","last_clean":"2024-04-06T12:13:22.368129+0000","last_became_active":"2024-04-06T12:13:20.894882+0000","last_became_peered":"2024-04-06T12:13:20.894882+0000","last_unstale":"2024-04-06T12:13:22.368129+0000","last_undegraded":"2024-04-06T12:13:22.368129+0000","last_fullsized":"2024-04-06T12:13:22.368129+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:46:08.328919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.368714+0000","last_change":"2024-04-06T12:13:21.315786+0000","last_active":"2024-04-06T12:13:22.368714+0000","last_peered":"2024-04-06T12:13:22.368714+0000","last_clean":"2024-04-06T12:13:22.368714+0000","last_became_active":"2024-04-06T12:13:21.315518+0000","last_became_peered":"2024-04-06T12:13:21.315518+0000","last_unstale":"2024-04-06T12:13:22.368714+0000","last_undegraded":"2024-04-06T12:13:22.368714+0000","last_fullsized":"2024-04-06T12:13:22.368714+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:20.868947+0000","last_clean_scrub_stamp":"2024-04-06T12:13:20.868947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:19:19.798635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-06T12:13:22.293694+0000","last_change":"2024-04-06T12:13:20.896439+0000","last_active":"2024-04-06T12:13:22.293694+0000","last_peered":"2024-04-06T12:13:22.293694+0000","last_clean":"2024-04-06T12:13:22.293694+0000","last_became_active":"2024-04-06T12:13:20.896156+0000","last_became_peered":"2024-04-06T12:13:20.896156+0000","last_unstale":"2024-04-06T12:13:22.293694+0000","last_undegraded":"2024-04-06T12:13:22.293694+0000","last_fullsized":"2024-04-06T12:13:22.293694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:13:19.863681+0000","last_clean_scrub_stamp":"2024-04-06T12:13:19.863681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:49:00.360938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":183,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-06T12:13:21.302060+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:13:21.302060+0000","last_peered":"2024-04-06T12:13:21.302060+0000","last_clean":"2024-04-06T12:13:21.302060+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:13:21.302060+0000","last_undegraded":"2024-04-06T12:13:21.302060+0000","last_fullsized":"2024-04-06T12:13:21.302060+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28068,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343772,"statfs":{"total":96636764160,"available":96608022528,"internally_reserved":0,"allocated":1060864,"data_stored":786117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.47399999999999998}]},{"osd":1,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.60499999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.62}]},{"osd":3,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.52700000000000002}]},{"osd":4,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.69999999999999996}]},{"osd":5,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.71599999999999997}]},{"osd":6,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.63300000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":450560,"data_stored":193678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.436},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.55300000000000005}]},{"osd":1,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.58699999999999997}]},{"osd":2,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.45400000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.60399999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.66300000000000003}]},{"osd":5,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.67900000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.70799999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":438272,"data_stored":186566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.33900000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.315},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.36599999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.26500000000000001}]},{"osd":4,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":6.3739999999999997,"5min":6.3739999999999997,"15min":6.3739999999999997},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.495}]},{"osd":5,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.624},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.52000000000000002}]},{"osd":6,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.58899999999999997}]},{"osd":7,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.60399999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1044480,"data_stored":783498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.33400000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.8089999999999999,"5min":1.8089999999999999,"15min":1.8089999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":0.27700000000000002}]},{"osd":3,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.35899999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.626},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.77000000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.67400000000000004}]},{"osd":6,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.70199999999999996}]},{"osd":7,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.749}]}]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":450560,"data_stored":193166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.375}]},{"osd":1,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.6179999999999999,"5min":3.6179999999999999,"15min":3.6179999999999999},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.28100000000000003}]},{"osd":3,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":3.661,"5min":3.661,"15min":3.661},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.35799999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.68799999999999994}]},{"osd":5,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":3.6920000000000002,"5min":3.6920000000000002,"15min":3.6920000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.73499999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.7490000000000001,"5min":3.7490000000000001,"15min":3.7490000000000001},"last":0.67100000000000004}]},{"osd":7,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.7200000000000002,"5min":3.7200000000000002,"15min":3.7200000000000002},"last":0.747}]}]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27488,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344352,"statfs":{"total":96636764160,"available":96608616448,"internally_reserved":0,"allocated":466944,"data_stored":195713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"last":0.32600000000000001}]},{"osd":1,"last update":"Sat Apr 6 12:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.38300000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.311},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.39900000000000002}]},{"osd":4,"last update":"Sat Apr 6 12:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.68400000000000005}]},{"osd":5,"last update":"Sat Apr 6 12:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.72099999999999997}]},{"osd":6,"last update":"Sat Apr 6 12:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.73099999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.70899999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":446464,"data_stored":188099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.82199999999999995}]},{"osd":1,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":4.3479999999999999,"5min":4.3479999999999999,"15min":4.3479999999999999},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":4.3319999999999999,"5min":4.3319999999999999,"15min":4.3319999999999999},"last":0.80900000000000005}]},{"osd":2,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.71499999999999997}]},{"osd":3,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":4.1950000000000003,"5min":4.1950000000000003,"15min":4.1950000000000003},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.39400000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":1.196},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.66400000000000003}]},{"osd":6,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.76000000000000001}]},{"osd":7,"last update":"Sat Apr 6 12:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.77700000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":442368,"data_stored":186588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.41699999999999998}]},{"osd":1,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.57499999999999996}]},{"osd":2,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.59599999999999997}]},{"osd":3,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.48699999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.55100000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:12:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.56599999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:13:25.788 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-06T12:13:25.788 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-06T12:13:25.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.953+0000 7f06ebbbc700 1 Processor -- start 2024-04-06T12:13:25.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.953+0000 7f06ebbbc700 1 -- start start 2024-04-06T12:13:25.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06ebbbc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f06e40cb870 0x7f06e415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06ebbbc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06e40cbd80 con 0x7f06e40cfa00 2024-04-06T12:13:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06ebbbc700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06e40cef20 con 0x7f06e40d11b0 2024-04-06T12:13:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06ebbbc700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06e40ccdc0 con 0x7f06e40cb870 2024-04-06T12:13:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06ea159700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f06e40cfa00 0x7f06e40cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33532/0 (socket says 172.21.15.189:33532) 2024-04-06T12:13:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06ea159700 1 -- 172.21.15.189:0/2822831888 learned_addr learned my addr 172.21.15.189:0/2822831888 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06e9157700 1 --2- 172.21.15.189:0/2822831888 >> v2:172.21.15.192:3300/0 conn(0x7f06e40cb870 0x7f06e415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06e8956700 1 -- 172.21.15.189:0/2822831888 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220820741 0 0) 0x7f06e40cbd80 con 0x7f06e40cfa00 2024-04-06T12:13:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06e8956700 1 -- 172.21.15.189:0/2822831888 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06d00035b0 con 0x7f06e40cfa00 2024-04-06T12:13:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06e8956700 1 -- 172.21.15.189:0/2822831888 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1940404219 0 0) 0x7f06d00035b0 con 0x7f06e40cfa00 2024-04-06T12:13:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06e8956700 1 -- 172.21.15.189:0/2822831888 >> v2:172.21.15.192:3300/0 conn(0x7f06e40cb870 msgr2=0x7f06e415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06e8956700 1 --2- 172.21.15.189:0/2822831888 >> v2:172.21.15.192:3300/0 conn(0x7f06e40cb870 0x7f06e415c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06e8956700 1 -- 172.21.15.189:0/2822831888 >> v1:172.21.15.192:6789/0 conn(0x7f06e40d11b0 legacy=0x7f06e40d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06e8956700 1 -- 172.21.15.189:0/2822831888 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e40cf200 con 0x7f06e40cfa00 2024-04-06T12:13:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06e8956700 1 -- 172.21.15.189:0/2822831888 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f06dc0019f0 con 0x7f06e40cfa00 2024-04-06T12:13:25.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06e8956700 1 -- 172.21.15.189:0/2822831888 >> v1:172.21.15.189:6789/0 conn(0x7f06e40cfa00 legacy=0x7f06e40cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:25.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06e8956700 1 --2- 172.21.15.189:0/2822831888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06d0003a00 0x7f06d0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06e8956700 1 --2- 172.21.15.189:0/2822831888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06d0004850 0x7f06d0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06e8956700 1 --2- 172.21.15.189:0/2822831888 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06d0005210 0x7f06d000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06e8956700 1 -- 172.21.15.189:0/2822831888 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d000c180 con 0x7f06d0003a00 2024-04-06T12:13:25.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06e8956700 1 -- 172.21.15.189:0/2822831888 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d000c2f0 con 0x7f06d0004850 2024-04-06T12:13:25.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.954+0000 7f06e8956700 1 -- 172.21.15.189:0/2822831888 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d000c4b0 con 0x7f06d0005210 2024-04-06T12:13:25.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.955+0000 7f06e9958700 1 --2- 172.21.15.189:0/2822831888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06d0003a00 0x7f06d0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:25.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.955+0000 7f06e9157700 1 --2- 172.21.15.189:0/2822831888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06d0004850 0x7f06d0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:25.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.955+0000 7f06ea159700 1 --2- 172.21.15.189:0/2822831888 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06d0005210 0x7f06d000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:25.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.955+0000 7f06e9958700 1 -- 172.21.15.189:0/2822831888 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06d0005210 msgr2=0x7f06d000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:25.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.955+0000 7f06e9958700 1 --2- 172.21.15.189:0/2822831888 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06d0005210 0x7f06d000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.955+0000 7f06e9958700 1 -- 172.21.15.189:0/2822831888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06d0004850 msgr2=0x7f06d0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:25.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.955+0000 7f06e9958700 1 --2- 172.21.15.189:0/2822831888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06d0004850 0x7f06d0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.955+0000 7f06e9958700 1 -- 172.21.15.189:0/2822831888 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d00035b0 con 0x7f06d0003a00 2024-04-06T12:13:25.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.955+0000 7f06ea159700 1 -- 172.21.15.189:0/2822831888 reap_dead start 2024-04-06T12:13:25.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.955+0000 7f06e9958700 1 --2- 172.21.15.189:0/2822831888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06d0003a00 0x7f06d0003e20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f06d40017c0 tx=0x7f06d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82fef469d4c6f4e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:25.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.955+0000 7f06e8956700 1 -- 172.21.15.189:0/2822831888 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d40052d0 con 0x7f06d0003a00 2024-04-06T12:13:25.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.955+0000 7f06e8956700 1 -- 172.21.15.189:0/2822831888 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06d4005430 con 0x7f06d0003a00 2024-04-06T12:13:25.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.955+0000 7f06e8956700 1 -- 172.21.15.189:0/2822831888 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d40038c0 con 0x7f06d0003a00 2024-04-06T12:13:25.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.956+0000 7f06ebbbc700 1 -- 172.21.15.189:0/2822831888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06d0003a00 msgr2=0x7f06d0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:25.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.956+0000 7f06ebbbc700 1 --2- 172.21.15.189:0/2822831888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06d0003a00 0x7f06d0003e20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f06d40017c0 tx=0x7f06d0005880 comp rx=0 tx=0).stop 2024-04-06T12:13:25.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.956+0000 7f06ebbbc700 1 -- 172.21.15.189:0/2822831888 shutdown_connections 2024-04-06T12:13:25.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.956+0000 7f06ebbbc700 1 --2- 172.21.15.189:0/2822831888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06d0003a00 0x7f06d0003e20 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.956+0000 7f06ebbbc700 1 -- 172.21.15.189:0/2822831888 >> 172.21.15.189:0/2822831888 conn(0x7f06e40b2db0 msgr2=0x7f06e40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:25.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.956+0000 7f06ebbbc700 1 -- 172.21.15.189:0/2822831888 shutdown_connections 2024-04-06T12:13:25.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.956+0000 7f06ebbbc700 1 -- 172.21.15.189:0/2822831888 wait complete. 2024-04-06T12:13:25.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.957+0000 7f06ebbbc700 1 Processor -- start 2024-04-06T12:13:25.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.957+0000 7f06ebbbc700 1 -- start start 2024-04-06T12:13:25.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.958+0000 7f06ebbbc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06d0003a00 0x7f06e4154c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.958+0000 7f06ebbbc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06e41f10d0 0x7f06e41f1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.958+0000 7f06ebbbc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06e41f2270 0x7f06e41f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.958+0000 7f06ebbbc700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e4155160 con 0x7f06e41f2270 2024-04-06T12:13:25.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.958+0000 7f06ebbbc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e41f1b90 con 0x7f06e41f10d0 2024-04-06T12:13:25.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.958+0000 7f06ebbbc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06e41f1e70 con 0x7f06d0003a00 2024-04-06T12:13:25.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.958+0000 7f06ea159700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06e41f2270 0x7f06e41f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:25.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.958+0000 7f06ea159700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06e41f2270 0x7f06e41f26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33232/0 (socket says 172.21.15.189:33232) 2024-04-06T12:13:25.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.958+0000 7f06ea159700 1 -- 172.21.15.189:0/1411389215 learned_addr learned my addr 172.21.15.189:0/1411389215 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:25.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.959+0000 7f06ea159700 1 -- 172.21.15.189:0/1411389215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06d0003a00 msgr2=0x7f06e4154c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:25.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.959+0000 7f06ea159700 1 --2- 172.21.15.189:0/1411389215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06d0003a00 0x7f06e4154c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.959+0000 7f06ea159700 1 -- 172.21.15.189:0/1411389215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06e41f10d0 msgr2=0x7f06e41f1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:25.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.959+0000 7f06e9157700 1 --2- 172.21.15.189:0/1411389215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06e41f10d0 0x7f06e41f1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:25.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.959+0000 7f06ea159700 1 --2- 172.21.15.189:0/1411389215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06e41f10d0 0x7f06e41f1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:25.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.959+0000 7f06ea159700 1 -- 172.21.15.189:0/1411389215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e40c6000 con 0x7f06e41f2270 2024-04-06T12:13:25.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.959+0000 7f06ea159700 1 --2- 172.21.15.189:0/1411389215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06e41f2270 0x7f06e41f26f0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f06e406dce0 tx=0x7f06dc007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:25.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.959+0000 7f06da7fc700 1 -- 172.21.15.189:0/1411389215 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc00ec60 con 0x7f06e41f2270 2024-04-06T12:13:25.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.959+0000 7f06da7fc700 1 -- 172.21.15.189:0/1411389215 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06dc00edc0 con 0x7f06e41f2270 2024-04-06T12:13:25.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.959+0000 7f06ebbbc700 1 -- 172.21.15.189:0/1411389215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06e40c62e0 con 0x7f06e41f2270 2024-04-06T12:13:25.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.959+0000 7f06da7fc700 1 -- 172.21.15.189:0/1411389215 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc013660 con 0x7f06e41f2270 2024-04-06T12:13:25.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.959+0000 7f06ebbbc700 1 -- 172.21.15.189:0/1411389215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06e40c6800 con 0x7f06e41f2270 2024-04-06T12:13:25.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.960+0000 7f06ebbbc700 1 -- 172.21.15.189:0/1411389215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06e40c5e60 con 0x7f06e41f2270 2024-04-06T12:13:25.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.960+0000 7f06da7fc700 1 -- 172.21.15.189:0/1411389215 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f06dc0137c0 con 0x7f06e41f2270 2024-04-06T12:13:25.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.961+0000 7f06da7fc700 1 --2- 172.21.15.189:0/1411389215 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06c806ac90 0x7f06c806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:25.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.961+0000 7f06da7fc700 1 -- 172.21.15.189:0/1411389215 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f06dc089b70 con 0x7f06e41f2270 2024-04-06T12:13:25.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.961+0000 7f06e9958700 1 --2- 172.21.15.189:0/1411389215 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06c806ac90 0x7f06c806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:25.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.961+0000 7f06e9958700 1 --2- 172.21.15.189:0/1411389215 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06c806ac90 0x7f06c806d150 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f06d4000c00 tx=0x7f06d4011000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:25.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:25.965+0000 7f06da7fc700 1 -- 172.21.15.189:0/1411389215 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f06e40c5e60 con 0x7f06e41f2270 2024-04-06T12:13:26.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.239+0000 7f06ebbbc700 1 -- 172.21.15.189:0/1411389215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f06e40c5e60 con 0x7f06e41f2270 2024-04-06T12:13:26.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.243+0000 7f06da7fc700 1 -- 172.21.15.189:0/1411389215 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+0 (secure 0 0 0) 0x7f06e40c5e60 con 0x7f06e41f2270 2024-04-06T12:13:26.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.245+0000 7f06ebbbc700 1 -- 172.21.15.189:0/1411389215 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06c806ac90 msgr2=0x7f06c806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:26.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.245+0000 7f06ebbbc700 1 --2- 172.21.15.189:0/1411389215 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06c806ac90 0x7f06c806d150 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f06d4000c00 tx=0x7f06d4011000 comp rx=0 tx=0).stop 2024-04-06T12:13:26.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.245+0000 7f06ebbbc700 1 -- 172.21.15.189:0/1411389215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06e41f2270 msgr2=0x7f06e41f26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:26.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.245+0000 7f06ebbbc700 1 --2- 172.21.15.189:0/1411389215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06e41f2270 0x7f06e41f26f0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f06e406dce0 tx=0x7f06dc007c90 comp rx=0 tx=0).stop 2024-04-06T12:13:26.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.245+0000 7f06ebbbc700 1 -- 172.21.15.189:0/1411389215 shutdown_connections 2024-04-06T12:13:26.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.245+0000 7f06ebbbc700 1 --2- 172.21.15.189:0/1411389215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06e41f2270 0x7f06e41f26f0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:26.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.245+0000 7f06ebbbc700 1 --2- 172.21.15.189:0/1411389215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06d0003a00 0x7f06e4154c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:26.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.245+0000 7f06ebbbc700 1 --2- 172.21.15.189:0/1411389215 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06c806ac90 0x7f06c806d150 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:26.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.245+0000 7f06ebbbc700 1 --2- 172.21.15.189:0/1411389215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06e41f10d0 0x7f06e41f1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:26.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.245+0000 7f06ebbbc700 1 -- 172.21.15.189:0/1411389215 >> 172.21.15.189:0/1411389215 conn(0x7f06e40b2db0 msgr2=0x7f06e40b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:26.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.245+0000 7f06ebbbc700 1 -- 172.21.15.189:0/1411389215 shutdown_connections 2024-04-06T12:13:26.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.245+0000 7f06ebbbc700 1 -- 172.21.15.189:0/1411389215 wait complete. 2024-04-06T12:13:26.244 INFO:teuthology.orchestra.run.smithi189.stderr:updated caps for client.0 2024-04-06T12:13:26.255 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:13:26.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.380+0000 7f6f8177a700 1 Processor -- start 2024-04-06T12:13:26.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.380+0000 7f6f8177a700 1 -- start start 2024-04-06T12:13:26.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.381+0000 7f6f8177a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6f7c15b130 0x7f6f7c0b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:26.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.381+0000 7f6f8177a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f7c15b510 0x7f6f7c0b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:26.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.381+0000 7f6f8177a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7c0b1fb0 con 0x7f6f7c071de0 2024-04-06T12:13:26.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.381+0000 7f6f8177a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f7c0b2200 con 0x7f6f7c15b130 2024-04-06T12:13:26.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.381+0000 7f6f8177a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c0b24e0 con 0x7f6f7c15b510 2024-04-06T12:13:26.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.381+0000 7f6f7affd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6f7c15b130 0x7f6f7c0b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:26.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.381+0000 7f6f7b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6f7c071de0 0x7f6f7c072250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33544/0 (socket says 172.21.15.189:33544) 2024-04-06T12:13:26.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.381+0000 7f6f7b7fe700 1 -- 172.21.15.189:0/2667996205 learned_addr learned my addr 172.21.15.189:0/2667996205 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:26.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.381+0000 7f6f7affd700 1 -- 172.21.15.189:0/2667996205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f7c15b510 msgr2=0x7f6f7c0b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:26.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.381+0000 7f6f7affd700 1 --2- 172.21.15.189:0/2667996205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f7c15b510 0x7f6f7c0b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:26.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.381+0000 7f6f7affd700 1 -- 172.21.15.189:0/2667996205 >> v1:172.21.15.189:6789/0 conn(0x7f6f7c071de0 legacy=0x7f6f7c072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:26.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.381+0000 7f6f7affd700 1 -- 172.21.15.189:0/2667996205 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7c0b27c0 con 0x7f6f7c15b130 2024-04-06T12:13:26.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.381+0000 7f6f79ffb700 1 -- 172.21.15.189:0/2667996205 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181920763 0 0) 0x7f6f7c0b1fb0 con 0x7f6f7c071de0 2024-04-06T12:13:26.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.382+0000 7f6f7affd700 1 --2- 172.21.15.189:0/2667996205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f7c15b130 0x7f6f7c0b3470 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6f70009840 tx=0x7f6f70009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65820761e15eb70c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:26.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.382+0000 7f6f79ffb700 1 -- 172.21.15.189:0/2667996205 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f70011830 con 0x7f6f7c15b130 2024-04-06T12:13:26.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.382+0000 7f6f79ffb700 1 -- 172.21.15.189:0/2667996205 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f70011990 con 0x7f6f7c15b130 2024-04-06T12:13:26.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.382+0000 7f6f79ffb700 1 -- 172.21.15.189:0/2667996205 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f70011d20 con 0x7f6f7c15b130 2024-04-06T12:13:26.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.382+0000 7f6f8177a700 1 -- 172.21.15.189:0/2667996205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f7c15b130 msgr2=0x7f6f7c0b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:26.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.382+0000 7f6f8177a700 1 --2- 172.21.15.189:0/2667996205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f7c15b130 0x7f6f7c0b3470 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6f70009840 tx=0x7f6f70009b50 comp rx=0 tx=0).stop 2024-04-06T12:13:26.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.382+0000 7f6f8177a700 1 -- 172.21.15.189:0/2667996205 shutdown_connections 2024-04-06T12:13:26.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.382+0000 7f6f8177a700 1 --2- 172.21.15.189:0/2667996205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f7c15b130 0x7f6f7c0b3470 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:26.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.382+0000 7f6f8177a700 1 --2- 172.21.15.189:0/2667996205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f7c15b510 0x7f6f7c0b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:26.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.382+0000 7f6f8177a700 1 -- 172.21.15.189:0/2667996205 >> 172.21.15.189:0/2667996205 conn(0x7f6f7c06a730 msgr2=0x7f6f7c06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:26.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.382+0000 7f6f8177a700 1 -- 172.21.15.189:0/2667996205 shutdown_connections 2024-04-06T12:13:26.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.382+0000 7f6f8177a700 1 -- 172.21.15.189:0/2667996205 wait complete. 2024-04-06T12:13:26.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.384+0000 7f6f8177a700 1 Processor -- start 2024-04-06T12:13:26.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.384+0000 7f6f8177a700 1 -- start start 2024-04-06T12:13:26.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.384+0000 7f6f8177a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f7c071de0 0x7f6f7c14f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:26.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.384+0000 7f6f8177a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f7c15b130 0x7f6f7c13f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:26.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.384+0000 7f6f8177a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f7c15b510 0x7f6f7c1506f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:26.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.384+0000 7f6f8177a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c1502d0 con 0x7f6f7c15b510 2024-04-06T12:13:26.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.384+0000 7f6f8177a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c0b0cf0 con 0x7f6f7c071de0 2024-04-06T12:13:26.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.384+0000 7f6f8177a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c0b0fd0 con 0x7f6f7c15b130 2024-04-06T12:13:26.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.384+0000 7f6f7b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f7c15b510 0x7f6f7c1506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:26.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.384+0000 7f6f7b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f7c15b510 0x7f6f7c1506f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33240/0 (socket says 172.21.15.189:33240) 2024-04-06T12:13:26.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.384+0000 7f6f7b7fe700 1 -- 172.21.15.189:0/3827873236 learned_addr learned my addr 172.21.15.189:0/3827873236 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:26.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.385+0000 7f6f7affd700 1 --2- 172.21.15.189:0/3827873236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f7c071de0 0x7f6f7c14f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:26.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.385+0000 7f6f7a7fc700 1 --2- 172.21.15.189:0/3827873236 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f7c15b130 0x7f6f7c13f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:26.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.385+0000 7f6f7b7fe700 1 -- 172.21.15.189:0/3827873236 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f7c15b130 msgr2=0x7f6f7c13f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:26.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.385+0000 7f6f7b7fe700 1 --2- 172.21.15.189:0/3827873236 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f7c15b130 0x7f6f7c13f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:26.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.385+0000 7f6f7b7fe700 1 -- 172.21.15.189:0/3827873236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f7c071de0 msgr2=0x7f6f7c14f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:26.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.385+0000 7f6f7b7fe700 1 --2- 172.21.15.189:0/3827873236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f7c071de0 0x7f6f7c14f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:26.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.385+0000 7f6f7b7fe700 1 -- 172.21.15.189:0/3827873236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f700094a0 con 0x7f6f7c15b510 2024-04-06T12:13:26.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.385+0000 7f6f7b7fe700 1 --2- 172.21.15.189:0/3827873236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f7c15b510 0x7f6f7c1506f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6f74002230 tx=0x7f6f7400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:26.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.385+0000 7f6f637fe700 1 -- 172.21.15.189:0/3827873236 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7400ac20 con 0x7f6f7c15b510 2024-04-06T12:13:26.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.385+0000 7f6f637fe700 1 -- 172.21.15.189:0/3827873236 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7400ada0 con 0x7f6f7c15b510 2024-04-06T12:13:26.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.385+0000 7f6f8177a700 1 -- 172.21.15.189:0/3827873236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7c14fc10 con 0x7f6f7c15b510 2024-04-06T12:13:26.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.385+0000 7f6f637fe700 1 -- 172.21.15.189:0/3827873236 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f74007310 con 0x7f6f7c15b510 2024-04-06T12:13:26.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.385+0000 7f6f8177a700 1 -- 172.21.15.189:0/3827873236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7c13f6b0 con 0x7f6f7c15b510 2024-04-06T12:13:26.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.386+0000 7f6f8177a700 1 -- 172.21.15.189:0/3827873236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f7c150410 con 0x7f6f7c15b510 2024-04-06T12:13:26.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.387+0000 7f6f637fe700 1 -- 172.21.15.189:0/3827873236 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6f74007470 con 0x7f6f7c15b510 2024-04-06T12:13:26.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.387+0000 7f6f637fe700 1 --2- 172.21.15.189:0/3827873236 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f6806ac90 0x7f6f6806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:26.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.387+0000 7f6f637fe700 1 -- 172.21.15.189:0/3827873236 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6f74089390 con 0x7f6f7c15b510 2024-04-06T12:13:26.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.387+0000 7f6f7affd700 1 --2- 172.21.15.189:0/3827873236 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f6806ac90 0x7f6f6806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:26.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.387+0000 7f6f7affd700 1 --2- 172.21.15.189:0/3827873236 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f6806ac90 0x7f6f6806d150 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6f700090b0 tx=0x7f6f70008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:26.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.391+0000 7f6f637fe700 1 -- 172.21.15.189:0/3827873236 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6f7c150410 con 0x7f6f7c15b510 2024-04-06T12:13:26.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.628+0000 7f6f8177a700 1 -- 172.21.15.189:0/3827873236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f7c150410 con 0x7f6f7c15b510 2024-04-06T12:13:26.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.628+0000 7f6f637fe700 1 -- 172.21.15.189:0/3827873236 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f6f7c150410 con 0x7f6f7c15b510 2024-04-06T12:13:26.627 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:13:26.627 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4865,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/2279772906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2279772906},{"type":"v1","addr":"172.21.15.192:6834","nonce":2279772906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4868,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/454939556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":454939556},{"type":"v1","addr":"172.21.15.192:6835","nonce":454939556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5107,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/399541295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":399541295},{"type":"v1","addr":"172.21.15.189:6837","nonce":399541295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:13:21.287908+0000","modified":"2024-04-06T12:13:24.295625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5232":{"gid":5232,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/1024118918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1024118918},{"type":"v1","addr":"172.21.15.189:6836","nonce":1024118918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-04-06T12:13:26.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.631+0000 7f6f8177a700 1 -- 172.21.15.189:0/3827873236 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f6806ac90 msgr2=0x7f6f6806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:26.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.631+0000 7f6f8177a700 1 --2- 172.21.15.189:0/3827873236 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f6806ac90 0x7f6f6806d150 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6f700090b0 tx=0x7f6f70008030 comp rx=0 tx=0).stop 2024-04-06T12:13:26.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.631+0000 7f6f8177a700 1 -- 172.21.15.189:0/3827873236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f7c15b510 msgr2=0x7f6f7c1506f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:26.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.631+0000 7f6f8177a700 1 --2- 172.21.15.189:0/3827873236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f7c15b510 0x7f6f7c1506f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6f74002230 tx=0x7f6f7400a060 comp rx=0 tx=0).stop 2024-04-06T12:13:26.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.631+0000 7f6f8177a700 1 -- 172.21.15.189:0/3827873236 shutdown_connections 2024-04-06T12:13:26.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.631+0000 7f6f8177a700 1 --2- 172.21.15.189:0/3827873236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f7c15b510 0x7f6f7c1506f0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:26.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.631+0000 7f6f8177a700 1 --2- 172.21.15.189:0/3827873236 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f6806ac90 0x7f6f6806d150 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:26.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.631+0000 7f6f8177a700 1 --2- 172.21.15.189:0/3827873236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f7c071de0 0x7f6f7c14f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:26.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.631+0000 7f6f8177a700 1 --2- 172.21.15.189:0/3827873236 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f7c15b130 0x7f6f7c13f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:26.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.631+0000 7f6f8177a700 1 -- 172.21.15.189:0/3827873236 >> 172.21.15.189:0/3827873236 conn(0x7f6f7c06a730 msgr2=0x7f6f7c070b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:26.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.631+0000 7f6f8177a700 1 -- 172.21.15.189:0/3827873236 shutdown_connections 2024-04-06T12:13:26.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:26.631+0000 7f6f8177a700 1 -- 172.21.15.189:0/3827873236 wait complete. 2024-04-06T12:13:26.630 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 16 2024-04-06T12:13:27.044 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-06T12:13:21.287908+0000', 'modified': '2024-04-06T12:13:24.295625+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5232}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5232': {'gid': 5232, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.189:6836/1024118918', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6835', 'nonce': 1024118918}, {'type': 'v1', 'addr': '172.21.15.189:6836', 'nonce': 1024118918}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-06T12:13:27.044 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-06T12:13:27.044 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-06T12:13:27.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.170+0000 7f5c480b9700 1 Processor -- start 2024-04-06T12:13:27.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.170+0000 7f5c480b9700 1 -- start start 2024-04-06T12:13:27.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.171+0000 7f5c480b9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5c4015d7b0 0x7f5c400b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.171+0000 7f5c480b9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c4006a730 0x7f5c4006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.171+0000 7f5c480b9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5c4015ddf0 0x7f5c4015e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.171+0000 7f5c480b9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c4006b0f0 con 0x7f5c4015d7b0 2024-04-06T12:13:27.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.171+0000 7f5c480b9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c400b3990 con 0x7f5c4015ddf0 2024-04-06T12:13:27.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.171+0000 7f5c480b9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c400b3060 con 0x7f5c4006a730 2024-04-06T12:13:27.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.171+0000 7f5c45e55700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5c4015d7b0 0x7f5c400b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.171+0000 7f5c45e55700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5c4015d7b0 0x7f5c400b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33254/0 (socket says 172.21.15.189:33254) 2024-04-06T12:13:27.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.171+0000 7f5c45e55700 1 -- 172.21.15.189:0/2065687845 learned_addr learned my addr 172.21.15.189:0/2065687845 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:27.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.172+0000 7f5c45654700 1 --2- 172.21.15.189:0/2065687845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c4006a730 0x7f5c4006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.172+0000 7f5c46656700 1 --2- 172.21.15.189:0/2065687845 >> v2:172.21.15.192:3300/0 conn(0x7f5c4015ddf0 0x7f5c4015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.172+0000 7f5c45e55700 1 -- 172.21.15.189:0/2065687845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c4006a730 msgr2=0x7f5c4006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:27.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.172+0000 7f5c45e55700 1 --2- 172.21.15.189:0/2065687845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c4006a730 0x7f5c4006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.172+0000 7f5c45e55700 1 -- 172.21.15.189:0/2065687845 >> v2:172.21.15.192:3300/0 conn(0x7f5c4015ddf0 msgr2=0x7f5c4015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:27.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.172+0000 7f5c45e55700 1 --2- 172.21.15.189:0/2065687845 >> v2:172.21.15.192:3300/0 conn(0x7f5c4015ddf0 0x7f5c4015e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.172+0000 7f5c45e55700 1 -- 172.21.15.189:0/2065687845 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c400b3340 con 0x7f5c4015d7b0 2024-04-06T12:13:27.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.172+0000 7f5c45654700 1 --2- 172.21.15.189:0/2065687845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c4006a730 0x7f5c4006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:27.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.172+0000 7f5c45e55700 1 --2- 172.21.15.189:0/2065687845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c4015d7b0 0x7f5c400b3e20 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5c400b4360 tx=0x7f5c30009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e28ae70690754ca server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:27.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.172+0000 7f5c44e53700 1 -- 172.21.15.189:0/2065687845 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3000b870 con 0x7f5c4015d7b0 2024-04-06T12:13:27.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.172+0000 7f5c44e53700 1 -- 172.21.15.189:0/2065687845 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3000b9d0 con 0x7f5c4015d7b0 2024-04-06T12:13:27.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.172+0000 7f5c44e53700 1 -- 172.21.15.189:0/2065687845 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3000bd60 con 0x7f5c4015d7b0 2024-04-06T12:13:27.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.172+0000 7f5c480b9700 1 -- 172.21.15.189:0/2065687845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c4015d7b0 msgr2=0x7f5c400b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:27.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.172+0000 7f5c480b9700 1 --2- 172.21.15.189:0/2065687845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c4015d7b0 0x7f5c400b3e20 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5c400b4360 tx=0x7f5c30009bd0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.173+0000 7f5c480b9700 1 -- 172.21.15.189:0/2065687845 shutdown_connections 2024-04-06T12:13:27.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.173+0000 7f5c480b9700 1 --2- 172.21.15.189:0/2065687845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c4015d7b0 0x7f5c400b3e20 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.173+0000 7f5c480b9700 1 --2- 172.21.15.189:0/2065687845 >> v2:172.21.15.192:3300/0 conn(0x7f5c4015ddf0 0x7f5c4015e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.173+0000 7f5c480b9700 1 --2- 172.21.15.189:0/2065687845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c4006a730 0x7f5c4006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.173+0000 7f5c480b9700 1 -- 172.21.15.189:0/2065687845 >> 172.21.15.189:0/2065687845 conn(0x7f5c400d00b0 msgr2=0x7f5c400d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:27.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.173+0000 7f5c480b9700 1 -- 172.21.15.189:0/2065687845 shutdown_connections 2024-04-06T12:13:27.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.173+0000 7f5c480b9700 1 -- 172.21.15.189:0/2065687845 wait complete. 2024-04-06T12:13:27.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.174+0000 7f5c480b9700 1 Processor -- start 2024-04-06T12:13:27.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.174+0000 7f5c480b9700 1 -- start start 2024-04-06T12:13:27.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.175+0000 7f5c480b9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c4006a730 0x7f5c40149970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.175+0000 7f5c480b9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c4015d7b0 0x7f5c40154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.175+0000 7f5c480b9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c4015ddf0 0x7f5c40155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.175+0000 7f5c480b9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c40154d30 con 0x7f5c4006a730 2024-04-06T12:13:27.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.175+0000 7f5c480b9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c40155010 con 0x7f5c4015ddf0 2024-04-06T12:13:27.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.175+0000 7f5c480b9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c401552f0 con 0x7f5c4015d7b0 2024-04-06T12:13:27.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.175+0000 7f5c45e55700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c4006a730 0x7f5c40149970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.175+0000 7f5c45e55700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c4006a730 0x7f5c40149970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33256/0 (socket says 172.21.15.189:33256) 2024-04-06T12:13:27.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.175+0000 7f5c45e55700 1 -- 172.21.15.189:0/829683753 learned_addr learned my addr 172.21.15.189:0/829683753 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:27.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.175+0000 7f5c45e55700 1 -- 172.21.15.189:0/829683753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c4015d7b0 msgr2=0x7f5c40154450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:27.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.175+0000 7f5c45e55700 1 --2- 172.21.15.189:0/829683753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c4015d7b0 0x7f5c40154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.175+0000 7f5c45e55700 1 -- 172.21.15.189:0/829683753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c4015ddf0 msgr2=0x7f5c40155dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:27.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.175+0000 7f5c45e55700 1 --2- 172.21.15.189:0/829683753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c4015ddf0 0x7f5c40155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.175+0000 7f5c45e55700 1 -- 172.21.15.189:0/829683753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c30009550 con 0x7f5c4006a730 2024-04-06T12:13:27.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.176+0000 7f5c45e55700 1 --2- 172.21.15.189:0/829683753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c4006a730 0x7f5c40149970 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5c4006c820 tx=0x7f5c30004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:27.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.176+0000 7f5c367fc700 1 -- 172.21.15.189:0/829683753 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c30004310 con 0x7f5c4006a730 2024-04-06T12:13:27.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.176+0000 7f5c367fc700 1 -- 172.21.15.189:0/829683753 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3001f070 con 0x7f5c4006a730 2024-04-06T12:13:27.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.176+0000 7f5c367fc700 1 -- 172.21.15.189:0/829683753 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c300036a0 con 0x7f5c4006a730 2024-04-06T12:13:27.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.176+0000 7f5c480b9700 1 -- 172.21.15.189:0/829683753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c401555d0 con 0x7f5c4006a730 2024-04-06T12:13:27.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.176+0000 7f5c480b9700 1 -- 172.21.15.189:0/829683753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c40149eb0 con 0x7f5c4006a730 2024-04-06T12:13:27.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.177+0000 7f5c367fc700 1 -- 172.21.15.189:0/829683753 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5c30021030 con 0x7f5c4006a730 2024-04-06T12:13:27.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.178+0000 7f5c367fc700 1 --2- 172.21.15.189:0/829683753 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2406ac90 0x7f5c2406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.178+0000 7f5c367fc700 1 -- 172.21.15.189:0/829683753 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5c3008a920 con 0x7f5c4006a730 2024-04-06T12:13:27.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.178+0000 7f5c45654700 1 --2- 172.21.15.189:0/829683753 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2406ac90 0x7f5c2406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.178+0000 7f5c45654700 1 --2- 172.21.15.189:0/829683753 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2406ac90 0x7f5c2406d150 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5c40154120 tx=0x7f5c3c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:27.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.180+0000 7f5c5426db80 1 Processor -- start 2024-04-06T12:13:27.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.180+0000 7f5c5426db80 1 -- start start 2024-04-06T12:13:27.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.181+0000 7f5c5426db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5f6c20 0x56122f5faa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.181+0000 7f5c5426db80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5fbfc0 0x56122f5d25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.181+0000 7f5c5426db80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5d1980 0x56122f5d2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.181+0000 7f5c5426db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56122f52b940 con 0x56122f5f6c20 2024-04-06T12:13:27.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.181+0000 7f5c5426db80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x56122f529300 con 0x56122f5d1980 2024-04-06T12:13:27.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.181+0000 7f5c5426db80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x56122f524bf0 con 0x56122f5fbfc0 2024-04-06T12:13:27.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.181+0000 7f5c45e55700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5f6c20 0x56122f5faa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.181+0000 7f5c45e55700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5f6c20 0x56122f5faa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33268/0 (socket says 172.21.15.189:33268) 2024-04-06T12:13:27.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.181+0000 7f5c45e55700 1 -- 172.21.15.189:0/2879066330 learned_addr learned my addr 172.21.15.189:0/2879066330 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:27.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.181+0000 7f5c45654700 1 --2- 172.21.15.189:0/2879066330 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5fbfc0 0x56122f5d25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.181+0000 7f5c46656700 1 --2- 172.21.15.189:0/2879066330 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5d1980 0x56122f5d2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.181+0000 7f5c45e55700 1 -- 172.21.15.189:0/2879066330 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5fbfc0 msgr2=0x56122f5d25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:27.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.181+0000 7f5c45e55700 1 --2- 172.21.15.189:0/2879066330 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5fbfc0 0x56122f5d25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.181+0000 7f5c45e55700 1 -- 172.21.15.189:0/2879066330 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5d1980 msgr2=0x56122f5d2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:27.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.181+0000 7f5c45e55700 1 --2- 172.21.15.189:0/2879066330 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5d1980 0x56122f5d2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.181+0000 7f5c45e55700 1 -- 172.21.15.189:0/2879066330 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56122f352b10 con 0x56122f5f6c20 2024-04-06T12:13:27.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.181+0000 7f5c45e55700 1 --2- 172.21.15.189:0/2879066330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5f6c20 0x56122f5faa60 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5c300596a0 tx=0x7f5c300092c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d60beee87ab774c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:27.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.181+0000 7f5c237fe700 1 -- 172.21.15.189:0/2879066330 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3008a040 con 0x56122f5f6c20 2024-04-06T12:13:27.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.182+0000 7f5c237fe700 1 -- 172.21.15.189:0/2879066330 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3005d540 con 0x56122f5f6c20 2024-04-06T12:13:27.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.182+0000 7f5c237fe700 1 -- 172.21.15.189:0/2879066330 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c30061b00 con 0x56122f5f6c20 2024-04-06T12:13:27.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.182+0000 7f5c5426db80 1 -- 172.21.15.189:0/2879066330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5f6c20 msgr2=0x56122f5faa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:27.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.182+0000 7f5c5426db80 1 --2- 172.21.15.189:0/2879066330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5f6c20 0x56122f5faa60 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5c300596a0 tx=0x7f5c300092c0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.182+0000 7f5c5426db80 1 -- 172.21.15.189:0/2879066330 shutdown_connections 2024-04-06T12:13:27.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.182+0000 7f5c5426db80 1 --2- 172.21.15.189:0/2879066330 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5d1980 0x56122f5d2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.182+0000 7f5c5426db80 1 --2- 172.21.15.189:0/2879066330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5f6c20 0x56122f5faa60 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.182+0000 7f5c5426db80 1 --2- 172.21.15.189:0/2879066330 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5fbfc0 0x56122f5d25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.182+0000 7f5c5426db80 1 -- 172.21.15.189:0/2879066330 >> 172.21.15.189:0/2879066330 conn(0x56122f532650 msgr2=0x56122f6193e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:27.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.182+0000 7f5c5426db80 1 -- 172.21.15.189:0/2879066330 shutdown_connections 2024-04-06T12:13:27.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.182+0000 7f5c5426db80 1 -- 172.21.15.189:0/2879066330 wait complete. 2024-04-06T12:13:27.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.183+0000 7f5c5426db80 1 -- start start 2024-04-06T12:13:27.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.183+0000 7f5c5426db80 1 Processor -- start 2024-04-06T12:13:27.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c5426db80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5d1980 0x56122f5e4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c5426db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5f6c20 0x56122f5f7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c5426db80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5fbfc0 0x56122f5f9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c5426db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56122f4c0d30 con 0x56122f5f6c20 2024-04-06T12:13:27.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c5426db80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x56122f4c09d0 con 0x56122f5d1980 2024-04-06T12:13:27.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c5426db80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x56122f3a8650 con 0x56122f5fbfc0 2024-04-06T12:13:27.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c5426db80 10 client.0 _ll_register_callbacks cb 0x56122f4ca420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:13:27.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c45654700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5f6c20 0x56122f5f7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c45654700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5f6c20 0x56122f5f7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33270/0 (socket says 172.21.15.189:33270) 2024-04-06T12:13:27.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c45654700 1 -- 172.21.15.189:0/2983523115 learned_addr learned my addr 172.21.15.189:0/2983523115 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:27.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c46656700 1 --2- 172.21.15.189:0/2983523115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5fbfc0 0x56122f5f9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c45e55700 1 --2- 172.21.15.189:0/2983523115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5d1980 0x56122f5e4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c45654700 1 -- 172.21.15.189:0/2983523115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5fbfc0 msgr2=0x56122f5f9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:27.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c45654700 1 --2- 172.21.15.189:0/2983523115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5fbfc0 0x56122f5f9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c45654700 1 -- 172.21.15.189:0/2983523115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5d1980 msgr2=0x56122f5e4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:27.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c45654700 1 --2- 172.21.15.189:0/2983523115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5d1980 0x56122f5e4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c45654700 1 -- 172.21.15.189:0/2983523115 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c40149eb0 con 0x56122f5f6c20 2024-04-06T12:13:27.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c45654700 1 --2- 172.21.15.189:0/2983523115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5f6c20 0x56122f5f7400 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5c401f5e50 tx=0x7f5c3c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96aca564668ca2a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:27.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c20ff9700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:13:27.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.184+0000 7f5c20ff9700 1 -- 172.21.15.189:0/2983523115 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c013c10 con 0x56122f5f6c20 2024-04-06T12:13:27.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.185+0000 7f5c20ff9700 1 -- 172.21.15.189:0/2983523115 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5c3c013d70 con 0x56122f5f6c20 2024-04-06T12:13:27.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.185+0000 7f5c5426db80 1 -- 172.21.15.189:0/2983523115 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56122f4e5330 con 0x56122f5f6c20 2024-04-06T12:13:27.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.185+0000 7f5c20ff9700 1 -- 172.21.15.189:0/2983523115 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c3c01f3f0 con 0x56122f5f6c20 2024-04-06T12:13:27.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.185+0000 7f5c5426db80 10 client.5433 fetch_fsmap learned FSMap version 16 2024-04-06T12:13:27.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.185+0000 7f5c5426db80 1 -- 172.21.15.189:0/2983523115 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x56122f4eae50 con 0x56122f5f6c20 2024-04-06T12:13:27.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.185+0000 7f5c20ff9700 1 -- 172.21.15.189:0/2983523115 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56122f4e5330 con 0x56122f5f6c20 2024-04-06T12:13:27.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.185+0000 7f5c20ff9700 1 -- 172.21.15.189:0/2983523115 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 16) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5c3c01e070 con 0x56122f5f6c20 2024-04-06T12:13:27.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.185+0000 7f5c5426db80 10 client.5433 fetch_fsmap finished waiting for FSMap version 16 2024-04-06T12:13:27.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.185+0000 7f5c5426db80 10 client.5433 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-06T12:13:27.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.185+0000 7f5c5426db80 20 client.5433 populate_metadata read hostname 'smithi189' 2024-04-06T12:13:27.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.185+0000 7f5c5426db80 1 --2- 172.21.15.189:0/2983523115 >> [v2:172.21.15.189:6835/1024118918,v1:172.21.15.189:6836/1024118918] conn(0x56122f5f9fc0 0x56122f601aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.185+0000 7f5c5426db80 4 client.5433 mds_command: new command op to 5232 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-06T12:13:27.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.185+0000 7f5c5426db80 1 -- 172.21.15.189:0/2983523115 --> [v2:172.21.15.189:6835/1024118918,v1:172.21.15.189:6836/1024118918] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56122f51d6b0 con 0x56122f5f9fc0 2024-04-06T12:13:27.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.185+0000 7f5c45e55700 1 --2- 172.21.15.189:0/2983523115 >> [v2:172.21.15.189:6835/1024118918,v1:172.21.15.189:6836/1024118918] conn(0x56122f5f9fc0 0x56122f601aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.186+0000 7f5c20ff9700 10 client.5433 ms_handle_connect on v2:172.21.15.189:6835/1024118918 2024-04-06T12:13:27.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.186+0000 7f5c45e55700 1 --2- 172.21.15.189:0/2983523115 >> [v2:172.21.15.189:6835/1024118918,v1:172.21.15.189:6836/1024118918] conn(0x56122f5f9fc0 0x56122f601aa0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b227b299cfe13273 server_cookie=2a1545f68cf40b38 in_seq=0 out_seq=0 2024-04-06T12:13:27.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.186+0000 7f5c20ff9700 1 -- 172.21.15.189:0/2983523115 <== mds.0 v2:172.21.15.189:6835/1024118918 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56122f51d6b0 con 0x56122f5f9fc0 2024-04-06T12:13:27.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.186+0000 7f5c20ff9700 10 client.5433 handle_command_reply: tid=0 2024-04-06T12:13:27.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.186+0000 7f5c5426db80 1 client.5433 shutdown 2024-04-06T12:13:27.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.186+0000 7f5c5426db80 1 -- 172.21.15.189:0/2983523115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5f6c20 msgr2=0x56122f5f7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:27.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.186+0000 7f5c5426db80 1 --2- 172.21.15.189:0/2983523115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5f6c20 0x56122f5f7400 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5c401f5e50 tx=0x7f5c3c004750 comp rx=0 tx=0).stop 2024-04-06T12:13:27.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.187+0000 7f5c5426db80 1 -- 172.21.15.189:0/2983523115 shutdown_connections 2024-04-06T12:13:27.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.187+0000 7f5c5426db80 1 --2- 172.21.15.189:0/2983523115 >> [v2:172.21.15.189:6835/1024118918,v1:172.21.15.189:6836/1024118918] conn(0x56122f5f9fc0 0x56122f601aa0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.187+0000 7f5c5426db80 1 --2- 172.21.15.189:0/2983523115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5fbfc0 0x56122f5f9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.187+0000 7f5c20ff9700 0 client.5433 ms_handle_reset on v2:172.21.15.189:6835/1024118918 2024-04-06T12:13:27.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.187+0000 7f5c5426db80 1 --2- 172.21.15.189:0/2983523115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5d1980 0x56122f5e4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.187+0000 7f5c5426db80 1 --2- 172.21.15.189:0/2983523115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5f6c20 0x56122f5f7400 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.187+0000 7f5c5426db80 1 -- 172.21.15.189:0/2983523115 >> 172.21.15.189:0/2983523115 conn(0x56122f532650 msgr2=0x56122f5d39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:27.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.187+0000 7f5c5426db80 1 -- 172.21.15.189:0/2983523115 shutdown_connections 2024-04-06T12:13:27.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.187+0000 7f5c5426db80 1 -- 172.21.15.189:0/2983523115 wait complete. 2024-04-06T12:13:27.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.187+0000 7f5c5426db80 20 client.5433 trim_cache size 0 max 16384 2024-04-06T12:13:27.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.204+0000 7f5c5426db80 1 Processor -- start 2024-04-06T12:13:27.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.204+0000 7f5c5426db80 1 -- start start 2024-04-06T12:13:27.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.205+0000 7f5c5426db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5d1980 0x56122f5fb100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.205+0000 7f5c5426db80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5f6c20 0x56122f5fb640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.205+0000 7f5c5426db80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5f9fc0 0x56122f5f8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.205+0000 7f5c5426db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56122f5fe900 con 0x56122f5d1980 2024-04-06T12:13:27.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.205+0000 7f5c5426db80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x56122f5e42b0 con 0x56122f5f9fc0 2024-04-06T12:13:27.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.205+0000 7f5c5426db80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x56122f5cf9d0 con 0x56122f5f6c20 2024-04-06T12:13:27.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.205+0000 7f5c45e55700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5d1980 0x56122f5fb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.205+0000 7f5c45e55700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5d1980 0x56122f5fb100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33276/0 (socket says 172.21.15.189:33276) 2024-04-06T12:13:27.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.205+0000 7f5c45e55700 1 -- 172.21.15.189:0/3357553935 learned_addr learned my addr 172.21.15.189:0/3357553935 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:27.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.205+0000 7f5c45654700 1 --2- 172.21.15.189:0/3357553935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5f6c20 0x56122f5fb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.205+0000 7f5c46656700 1 --2- 172.21.15.189:0/3357553935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5f9fc0 0x56122f5f8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.205+0000 7f5c45e55700 1 -- 172.21.15.189:0/3357553935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5f6c20 msgr2=0x56122f5fb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:27.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.205+0000 7f5c45e55700 1 --2- 172.21.15.189:0/3357553935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5f6c20 0x56122f5fb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.205+0000 7f5c45e55700 1 -- 172.21.15.189:0/3357553935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5f9fc0 msgr2=0x56122f5f8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:27.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.205+0000 7f5c45e55700 1 --2- 172.21.15.189:0/3357553935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5f9fc0 0x56122f5f8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.205+0000 7f5c45e55700 1 -- 172.21.15.189:0/3357553935 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56122f3aac80 con 0x56122f5d1980 2024-04-06T12:13:27.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.205+0000 7f5c45e55700 1 --2- 172.21.15.189:0/3357553935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5d1980 0x56122f5fb100 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5c30088df0 tx=0x7f5c3005f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a78626e012b7d12 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:27.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.205+0000 7f5c23fff700 1 -- 172.21.15.189:0/3357553935 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3008a030 con 0x56122f5d1980 2024-04-06T12:13:27.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.205+0000 7f5c23fff700 1 -- 172.21.15.189:0/3357553935 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c30071150 con 0x56122f5d1980 2024-04-06T12:13:27.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.206+0000 7f5c23fff700 1 -- 172.21.15.189:0/3357553935 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c30061b00 con 0x56122f5d1980 2024-04-06T12:13:27.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.206+0000 7f5c5426db80 1 -- 172.21.15.189:0/3357553935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5d1980 msgr2=0x56122f5fb100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:27.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.206+0000 7f5c5426db80 1 --2- 172.21.15.189:0/3357553935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5d1980 0x56122f5fb100 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5c30088df0 tx=0x7f5c3005f000 comp rx=0 tx=0).stop 2024-04-06T12:13:27.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.206+0000 7f5c5426db80 1 -- 172.21.15.189:0/3357553935 shutdown_connections 2024-04-06T12:13:27.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.206+0000 7f5c5426db80 1 --2- 172.21.15.189:0/3357553935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5f9fc0 0x56122f5f8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.206+0000 7f5c5426db80 1 --2- 172.21.15.189:0/3357553935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5d1980 0x56122f5fb100 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.206+0000 7f5c5426db80 1 --2- 172.21.15.189:0/3357553935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5f6c20 0x56122f5fb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.206+0000 7f5c5426db80 1 -- 172.21.15.189:0/3357553935 >> 172.21.15.189:0/3357553935 conn(0x56122f532650 msgr2=0x56122f63c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:27.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.206+0000 7f5c5426db80 1 -- 172.21.15.189:0/3357553935 shutdown_connections 2024-04-06T12:13:27.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.206+0000 7f5c5426db80 1 -- 172.21.15.189:0/3357553935 wait complete. 2024-04-06T12:13:27.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.207+0000 7f5c5426db80 1 -- start start 2024-04-06T12:13:27.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.207+0000 7f5c5426db80 1 Processor -- start 2024-04-06T12:13:27.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c5426db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5d1980 0x56122f668bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c5426db80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5f6c20 0x56122f5e6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c5426db80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5f9fc0 0x56122f619cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c5426db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56122f5f90e0 con 0x56122f5d1980 2024-04-06T12:13:27.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c5426db80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x56122f5ff940 con 0x56122f5f9fc0 2024-04-06T12:13:27.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c5426db80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x56122f5f8dd0 con 0x56122f5f6c20 2024-04-06T12:13:27.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c5426db80 10 client.0 _ll_register_callbacks cb 0x7f5c3c00c730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:13:27.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c45e55700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5d1980 0x56122f668bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c45e55700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5d1980 0x56122f668bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33288/0 (socket says 172.21.15.189:33288) 2024-04-06T12:13:27.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c45e55700 1 -- 172.21.15.189:0/1665615771 learned_addr learned my addr 172.21.15.189:0/1665615771 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:27.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c45654700 1 --2- 172.21.15.189:0/1665615771 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5f6c20 0x56122f5e6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c46656700 1 --2- 172.21.15.189:0/1665615771 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5f9fc0 0x56122f619cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c45e55700 1 -- 172.21.15.189:0/1665615771 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5f6c20 msgr2=0x56122f5e6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:27.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c45e55700 1 --2- 172.21.15.189:0/1665615771 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5f6c20 0x56122f5e6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c45e55700 1 -- 172.21.15.189:0/1665615771 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5f9fc0 msgr2=0x56122f619cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:27.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c45e55700 1 --2- 172.21.15.189:0/1665615771 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5f9fc0 0x56122f619cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c45e55700 1 -- 172.21.15.189:0/1665615771 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56122f352b10 con 0x56122f5d1980 2024-04-06T12:13:27.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c46656700 1 --2- 172.21.15.189:0/1665615771 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5f9fc0 0x56122f619cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:27.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c45654700 1 --2- 172.21.15.189:0/1665615771 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5f6c20 0x56122f5e6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:27.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c45e55700 1 --2- 172.21.15.189:0/1665615771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5d1980 0x56122f668bd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5c30009fd0 tx=0x7f5c3000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f146e3ce05140d2c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:27.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c20ff9700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:13:27.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c20ff9700 1 -- 172.21.15.189:0/1665615771 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c30057910 con 0x56122f5d1980 2024-04-06T12:13:27.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c20ff9700 1 -- 172.21.15.189:0/1665615771 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5c3005d160 con 0x56122f5d1980 2024-04-06T12:13:27.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c5426db80 1 -- 172.21.15.189:0/1665615771 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5c3005d540 con 0x56122f5d1980 2024-04-06T12:13:27.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c20ff9700 1 -- 172.21.15.189:0/1665615771 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c3005d320 con 0x56122f5d1980 2024-04-06T12:13:27.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c5426db80 10 client.5439 fetch_fsmap learned FSMap version 16 2024-04-06T12:13:27.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.208+0000 7f5c5426db80 1 -- 172.21.15.189:0/1665615771 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x56122f350fe0 con 0x56122f5d1980 2024-04-06T12:13:27.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c20ff9700 1 -- 172.21.15.189:0/1665615771 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3005d540 con 0x56122f5d1980 2024-04-06T12:13:27.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c20ff9700 1 -- 172.21.15.189:0/1665615771 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 16) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5c30059970 con 0x56122f5d1980 2024-04-06T12:13:27.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c5426db80 10 client.5439 fetch_fsmap finished waiting for FSMap version 16 2024-04-06T12:13:27.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c5426db80 10 client.5439 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-06T12:13:27.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c5426db80 20 client.5439 populate_metadata read hostname 'smithi189' 2024-04-06T12:13:27.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c5426db80 1 --2- 172.21.15.189:0/1665615771 >> [v2:172.21.15.189:6835/1024118918,v1:172.21.15.189:6836/1024118918] conn(0x56122f5fbfc0 0x56122f5e94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c5426db80 4 client.5439 mds_command: new command op to 5232 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-06T12:13:27.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c5426db80 1 -- 172.21.15.189:0/1665615771 --> [v2:172.21.15.189:6835/1024118918,v1:172.21.15.189:6836/1024118918] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56122f373ca0 con 0x56122f5fbfc0 2024-04-06T12:13:27.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c45654700 1 --2- 172.21.15.189:0/1665615771 >> [v2:172.21.15.189:6835/1024118918,v1:172.21.15.189:6836/1024118918] conn(0x56122f5fbfc0 0x56122f5e94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c20ff9700 10 client.5439 ms_handle_connect on v2:172.21.15.189:6835/1024118918 2024-04-06T12:13:27.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c45654700 1 --2- 172.21.15.189:0/1665615771 >> [v2:172.21.15.189:6835/1024118918,v1:172.21.15.189:6836/1024118918] conn(0x56122f5fbfc0 0x56122f5e94e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66abc77f94bf9aea server_cookie=eac3707bdf11baca in_seq=0 out_seq=0 2024-04-06T12:13:27.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c20ff9700 1 -- 172.21.15.189:0/1665615771 <== mds.0 v2:172.21.15.189:6835/1024118918 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x56122f373ca0 con 0x56122f5fbfc0 2024-04-06T12:13:27.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c20ff9700 10 client.5439 handle_command_reply: tid=0 2024-04-06T12:13:27.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c5426db80 1 client.5439 shutdown 2024-04-06T12:13:27.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c5426db80 1 -- 172.21.15.189:0/1665615771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5d1980 msgr2=0x56122f668bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:27.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c5426db80 1 --2- 172.21.15.189:0/1665615771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5d1980 0x56122f668bd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5c30009fd0 tx=0x7f5c3000bfd0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c5426db80 1 -- 172.21.15.189:0/1665615771 shutdown_connections 2024-04-06T12:13:27.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c5426db80 1 --2- 172.21.15.189:0/1665615771 >> [v2:172.21.15.189:6835/1024118918,v1:172.21.15.189:6836/1024118918] conn(0x56122f5fbfc0 0x56122f5e94e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c5426db80 1 --2- 172.21.15.189:0/1665615771 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56122f5f9fc0 0x56122f619cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c5426db80 1 --2- 172.21.15.189:0/1665615771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56122f5d1980 0x56122f668bd0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c20ff9700 0 client.5439 ms_handle_reset on v2:172.21.15.189:6835/1024118918 2024-04-06T12:13:27.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c5426db80 1 --2- 172.21.15.189:0/1665615771 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56122f5f6c20 0x56122f5e6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.209+0000 7f5c5426db80 1 -- 172.21.15.189:0/1665615771 >> 172.21.15.189:0/1665615771 conn(0x56122f532650 msgr2=0x56122f5d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:27.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.210+0000 7f5c5426db80 1 -- 172.21.15.189:0/1665615771 shutdown_connections 2024-04-06T12:13:27.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.210+0000 7f5c5426db80 1 -- 172.21.15.189:0/1665615771 wait complete. 2024-04-06T12:13:27.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.210+0000 7f5c5426db80 20 client.5439 trim_cache size 0 max 16384 2024-04-06T12:13:27.216 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T12:13:27.216 INFO:teuthology.orchestra.run.smithi189.stdout: "cluster_fsid": "e30041de-e9a3-4351-8ca7-29d7f3e3728c", 2024-04-06T12:13:27.216 INFO:teuthology.orchestra.run.smithi189.stdout: "whoami": 0, 2024-04-06T12:13:27.216 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 5232, 2024-04-06T12:13:27.216 INFO:teuthology.orchestra.run.smithi189.stdout: "want_state": "up:active", 2024-04-06T12:13:27.217 INFO:teuthology.orchestra.run.smithi189.stdout: "state": "up:active", 2024-04-06T12:13:27.217 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs", 2024-04-06T12:13:27.217 INFO:teuthology.orchestra.run.smithi189.stdout: "rank_uptime": 5.9203832529999998, 2024-04-06T12:13:27.217 INFO:teuthology.orchestra.run.smithi189.stdout: "mdsmap_epoch": 16, 2024-04-06T12:13:27.217 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch": 39, 2024-04-06T12:13:27.217 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch_barrier": 39, 2024-04-06T12:13:27.217 INFO:teuthology.orchestra.run.smithi189.stdout: "uptime": 13.399684662 2024-04-06T12:13:27.217 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T12:13:27.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.210+0000 7f5c480b9700 1 -- 172.21.15.189:0/829683753 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2406ac90 msgr2=0x7f5c2406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:27.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.210+0000 7f5c480b9700 1 --2- 172.21.15.189:0/829683753 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2406ac90 0x7f5c2406d150 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5c40154120 tx=0x7f5c3c008040 comp rx=0 tx=0).stop 2024-04-06T12:13:27.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.210+0000 7f5c480b9700 1 -- 172.21.15.189:0/829683753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c4006a730 msgr2=0x7f5c40149970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:27.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.210+0000 7f5c480b9700 1 --2- 172.21.15.189:0/829683753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c4006a730 0x7f5c40149970 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5c4006c820 tx=0x7f5c30004750 comp rx=0 tx=0).stop 2024-04-06T12:13:27.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.210+0000 7f5c480b9700 1 -- 172.21.15.189:0/829683753 shutdown_connections 2024-04-06T12:13:27.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.210+0000 7f5c480b9700 1 --2- 172.21.15.189:0/829683753 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2406ac90 0x7f5c2406d150 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.210+0000 7f5c480b9700 1 --2- 172.21.15.189:0/829683753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c4015ddf0 0x7f5c40155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.210+0000 7f5c480b9700 1 --2- 172.21.15.189:0/829683753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c4006a730 0x7f5c40149970 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.210+0000 7f5c480b9700 1 --2- 172.21.15.189:0/829683753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c4015d7b0 0x7f5c40154450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.210+0000 7f5c480b9700 1 -- 172.21.15.189:0/829683753 >> 172.21.15.189:0/829683753 conn(0x7f5c400d00b0 msgr2=0x7f5c400ae0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:27.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.210+0000 7f5c480b9700 1 -- 172.21.15.189:0/829683753 shutdown_connections 2024-04-06T12:13:27.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.210+0000 7f5c480b9700 1 -- 172.21.15.189:0/829683753 wait complete. 2024-04-06T12:13:27.220 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T12:13:27.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.343+0000 7f07253c5700 1 Processor -- start 2024-04-06T12:13:27.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.343+0000 7f07253c5700 1 -- start start 2024-04-06T12:13:27.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.344+0000 7f07253c5700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f072006ae40 0x7f07200b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.344+0000 7f07253c5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f072015ddf0 0x7f072015e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.344+0000 7f07253c5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07200b3990 con 0x7f072015d7b0 2024-04-06T12:13:27.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.344+0000 7f07253c5700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07200b3060 con 0x7f072006ae40 2024-04-06T12:13:27.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.344+0000 7f07253c5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07200b3340 con 0x7f072015ddf0 2024-04-06T12:13:27.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.344+0000 7f071e7fc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f072006ae40 0x7f07200b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.344+0000 7f071f7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f072015ddf0 0x7f072015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.344+0000 7f071effd700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f072015d7b0 0x7f072006a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:47250/0 (socket says 172.21.15.189:47250) 2024-04-06T12:13:27.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.344+0000 7f071e7fc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f072006ae40 0x7f07200b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:35430/0 (socket says 172.21.15.189:35430) 2024-04-06T12:13:27.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.344+0000 7f071effd700 1 -- 172.21.15.189:0/607978392 learned_addr learned my addr 172.21.15.189:0/607978392 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:27.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.345+0000 7f071f7fe700 1 -- 172.21.15.189:0/607978392 >> v2:172.21.15.192:3300/0 conn(0x7f072006ae40 msgr2=0x7f07200b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:27.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.345+0000 7f071f7fe700 1 --2- 172.21.15.189:0/607978392 >> v2:172.21.15.192:3300/0 conn(0x7f072006ae40 0x7f07200b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.345+0000 7f071f7fe700 1 -- 172.21.15.189:0/607978392 >> v1:172.21.15.192:6789/0 conn(0x7f072015d7b0 legacy=0x7f072006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:27.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.345+0000 7f071f7fe700 1 -- 172.21.15.189:0/607978392 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07200b3620 con 0x7f072015ddf0 2024-04-06T12:13:27.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.345+0000 7f071f7fe700 1 --2- 172.21.15.189:0/607978392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f072015ddf0 0x7f072015e280 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0718009d90 tx=0x7f071800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec41123997267598 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:27.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.347+0000 7f071dffb700 1 -- 172.21.15.189:0/607978392 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071800e070 con 0x7f072015ddf0 2024-04-06T12:13:27.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.347+0000 7f071dffb700 1 -- 172.21.15.189:0/607978392 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071800bb20 con 0x7f072015ddf0 2024-04-06T12:13:27.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.347+0000 7f071dffb700 1 -- 172.21.15.189:0/607978392 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718012430 con 0x7f072015ddf0 2024-04-06T12:13:27.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.348+0000 7f07253c5700 1 -- 172.21.15.189:0/607978392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f072015ddf0 msgr2=0x7f072015e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:27.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.348+0000 7f07253c5700 1 --2- 172.21.15.189:0/607978392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f072015ddf0 0x7f072015e280 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0718009d90 tx=0x7f071800b330 comp rx=0 tx=0).stop 2024-04-06T12:13:27.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.348+0000 7f07253c5700 1 -- 172.21.15.189:0/607978392 shutdown_connections 2024-04-06T12:13:27.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.348+0000 7f07253c5700 1 --2- 172.21.15.189:0/607978392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f072015ddf0 0x7f072015e280 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.348+0000 7f07253c5700 1 --2- 172.21.15.189:0/607978392 >> v2:172.21.15.192:3300/0 conn(0x7f072006ae40 0x7f07200b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.348+0000 7f07253c5700 1 -- 172.21.15.189:0/607978392 >> 172.21.15.189:0/607978392 conn(0x7f07200d00b0 msgr2=0x7f07200d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:27.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.348+0000 7f07253c5700 1 -- 172.21.15.189:0/607978392 shutdown_connections 2024-04-06T12:13:27.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.348+0000 7f07253c5700 1 -- 172.21.15.189:0/607978392 wait complete. 2024-04-06T12:13:27.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.349+0000 7f07253c5700 1 Processor -- start 2024-04-06T12:13:27.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.349+0000 7f07253c5700 1 -- start start 2024-04-06T12:13:27.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.350+0000 7f07253c5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f072006ae40 0x7f07200cef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.350+0000 7f07253c5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f072015d7b0 0x7f0720151f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.350+0000 7f07253c5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f072015ddf0 0x7f07201524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.350+0000 7f07253c5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07200ccb00 con 0x7f072015d7b0 2024-04-06T12:13:27.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.350+0000 7f07253c5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07200ccf60 con 0x7f072006ae40 2024-04-06T12:13:27.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.350+0000 7f07253c5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07200cd240 con 0x7f072015ddf0 2024-04-06T12:13:27.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.350+0000 7f071e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f072015d7b0 0x7f0720151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.350+0000 7f071e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f072015d7b0 0x7f0720151f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33300/0 (socket says 172.21.15.189:33300) 2024-04-06T12:13:27.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.350+0000 7f071e7fc700 1 -- 172.21.15.189:0/566007046 learned_addr learned my addr 172.21.15.189:0/566007046 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:27.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.350+0000 7f071f7fe700 1 --2- 172.21.15.189:0/566007046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f072015ddf0 0x7f07201524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.350+0000 7f071effd700 1 --2- 172.21.15.189:0/566007046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f072006ae40 0x7f07200cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.351+0000 7f071e7fc700 1 -- 172.21.15.189:0/566007046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f072015ddf0 msgr2=0x7f07201524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:27.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.351+0000 7f071e7fc700 1 --2- 172.21.15.189:0/566007046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f072015ddf0 0x7f07201524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.351+0000 7f071e7fc700 1 -- 172.21.15.189:0/566007046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f072006ae40 msgr2=0x7f07200cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:27.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.351+0000 7f071e7fc700 1 --2- 172.21.15.189:0/566007046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f072006ae40 0x7f07200cef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.351+0000 7f071e7fc700 1 -- 172.21.15.189:0/566007046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0718009a70 con 0x7f072015d7b0 2024-04-06T12:13:27.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.351+0000 7f071e7fc700 1 --2- 172.21.15.189:0/566007046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f072015d7b0 0x7f0720151f70 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f071400e980 tx=0x7f071400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:27.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.351+0000 7f07077fe700 1 -- 172.21.15.189:0/566007046 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0714009710 con 0x7f072015d7b0 2024-04-06T12:13:27.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.351+0000 7f07077fe700 1 -- 172.21.15.189:0/566007046 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071400c690 con 0x7f072015d7b0 2024-04-06T12:13:27.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.351+0000 7f07253c5700 1 -- 172.21.15.189:0/566007046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07200cd520 con 0x7f072015d7b0 2024-04-06T12:13:27.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.351+0000 7f07077fe700 1 -- 172.21.15.189:0/566007046 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071400ca20 con 0x7f072015d7b0 2024-04-06T12:13:27.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.351+0000 7f07253c5700 1 -- 172.21.15.189:0/566007046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07200cf470 con 0x7f072015d7b0 2024-04-06T12:13:27.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.352+0000 7f07253c5700 1 -- 172.21.15.189:0/566007046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07200cf660 con 0x7f072015d7b0 2024-04-06T12:13:27.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.352+0000 7f07077fe700 1 -- 172.21.15.189:0/566007046 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f071400cb80 con 0x7f072015d7b0 2024-04-06T12:13:27.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.353+0000 7f07077fe700 1 --2- 172.21.15.189:0/566007046 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f070c06ac90 0x7f070c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:27.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.353+0000 7f071effd700 1 --2- 172.21.15.189:0/566007046 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f070c06ac90 0x7f070c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:27.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.353+0000 7f07077fe700 1 -- 172.21.15.189:0/566007046 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0714089340 con 0x7f072015d7b0 2024-04-06T12:13:27.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.353+0000 7f071effd700 1 --2- 172.21.15.189:0/566007046 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f070c06ac90 0x7f070c06d150 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0710005910 tx=0x7f0710009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:27.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.357+0000 7f07077fe700 1 -- 172.21.15.189:0/566007046 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0714054470 con 0x7f072015d7b0 2024-04-06T12:13:27.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.595+0000 7f07253c5700 1 -- 172.21.15.189:0/566007046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0720151c90 con 0x7f072015d7b0 2024-04-06T12:13:27.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.595+0000 7f07077fe700 1 -- 172.21.15.189:0/566007046 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0720151c90 con 0x7f072015d7b0 2024-04-06T12:13:27.593 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-06T12:13:27.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.597+0000 7f07253c5700 1 -- 172.21.15.189:0/566007046 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f070c06ac90 msgr2=0x7f070c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:27.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.597+0000 7f07253c5700 1 --2- 172.21.15.189:0/566007046 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f070c06ac90 0x7f070c06d150 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0710005910 tx=0x7f0710009040 comp rx=0 tx=0).stop 2024-04-06T12:13:27.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.597+0000 7f07253c5700 1 -- 172.21.15.189:0/566007046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f072015d7b0 msgr2=0x7f0720151f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:27.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.597+0000 7f07253c5700 1 --2- 172.21.15.189:0/566007046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f072015d7b0 0x7f0720151f70 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f071400e980 tx=0x7f071400ec90 comp rx=0 tx=0).stop 2024-04-06T12:13:27.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.597+0000 7f07253c5700 1 -- 172.21.15.189:0/566007046 shutdown_connections 2024-04-06T12:13:27.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.597+0000 7f07253c5700 1 --2- 172.21.15.189:0/566007046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f072015ddf0 0x7f07201524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.597+0000 7f07253c5700 1 --2- 172.21.15.189:0/566007046 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f070c06ac90 0x7f070c06d150 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.597+0000 7f07253c5700 1 --2- 172.21.15.189:0/566007046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f072006ae40 0x7f07200cef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.597+0000 7f07253c5700 1 --2- 172.21.15.189:0/566007046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f072015d7b0 0x7f0720151f70 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:27.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.597+0000 7f07253c5700 1 -- 172.21.15.189:0/566007046 >> 172.21.15.189:0/566007046 conn(0x7f07200d00b0 msgr2=0x7f07200ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:27.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.598+0000 7f07253c5700 1 -- 172.21.15.189:0/566007046 shutdown_connections 2024-04-06T12:13:27.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:27.598+0000 7f07253c5700 1 -- 172.21.15.189:0/566007046 wait complete. 2024-04-06T12:13:27.607 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-06T12:13:27.608 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T12:13:27.608 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-06T12:13:27.608 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-06T12:13:27.608 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T12:13:27.608 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-06T12:13:27.608 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-06T12:13:27.608 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T12:13:27.608 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:13:27.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:27.608 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-06T12:13:27.626 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T12:13:27.627 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T12:13:27.627 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T12:13:27.627 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:13:27.627 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T12:13:27.627 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:13:27.627 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:13:27.627 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:13:27.627 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-06T12:13:27.627 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:13:27.627 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:13:27.627 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-06T12:13:27.627 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:13:27.627 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:13:27.628 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T12:13:27.628 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:13:27.628 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T12:13:27.628 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 42282sec preferred_lft 42282sec 2024-04-06T12:13:27.628 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::8592:df2c:92e2:dace/64 scope link noprefixroute 2024-04-06T12:13:27.628 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:13:27.628 INFO:teuthology.orchestra.run.smithi189.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T12:13:27.628 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff 2024-04-06T12:13:27.628 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-06T12:13:27.628 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:13:27.628 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::88e:b6ff:fe6f:1cb0/64 scope link 2024-04-06T12:13:27.628 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:13:27.628 INFO:teuthology.orchestra.run.smithi189.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T12:13:27.628 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 62:a5:8d:c5:1a:9c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T12:13:27.628 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::60a5:8dff:fec5:1a9c/64 scope link 2024-04-06T12:13:27.629 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:13:27.629 INFO:teuthology.orchestra.run.smithi189.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T12:13:27.629 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-06T12:13:27.629 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::1c7e:c6ff:febb:1dab/64 scope link 2024-04-06T12:13:27.629 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:13:27.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:27.629 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T12:13:27.693 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-06T12:13:27.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:27.694 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-06T12:13:27.752 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-06T12:13:27.753 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T12:13:27.753 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T12:13:27.753 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T12:13:27.753 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T12:13:27.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:27.753 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:13:27.753 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:13:27.753 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T12:13:27.753 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T12:13:27.754 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T12:13:27.754 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:13:27.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:27.860 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T12:13:27.876 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-06T12:13:27.876 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-06T12:13:27.877 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:13:27.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:27.877 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-04-06T12:13:27.945 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:13:27.945 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-06T12:13:28.002 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:13:28.002 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:13:28.002 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-06T12:13:28.058 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T12:13:28.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:28.059 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:13:28.145 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:13:28.145 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:13:28.145 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:13:28.154 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:13:28.154 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:13:28.154 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:13:28.154 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:13:28.154 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:13:28.173 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:13:28.173 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:13:28.174 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:13:28.174 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:13:28.174 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:13:28.174 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790" 2024-04-06T12:13:28.174 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2024-04-06T12:13:28.174 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e30041de-e9a3-4351-8ca7-29d7f3e3728c" will pass to kernel 2024-04-06T12:13:28.176 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T12:13:28.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:28.176 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-06T12:13:28.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:28.204 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:13:28.290 INFO:teuthology.orchestra.run.smithi189.stdout:client4967 2024-04-06T12:13:28.290 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:13:28.290 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:13:28.308 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-06T12:13:28.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.472+0000 7f5be3a44700 1 Processor -- start 2024-04-06T12:13:28.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.472+0000 7f5be3a44700 1 -- start start 2024-04-06T12:13:28.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.473+0000 7f5be3a44700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5bdc15d7b0 0x7f5bdc0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:28.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.473+0000 7f5be3a44700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5bdc15e500 0x7f5bdc06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:28.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.473+0000 7f5be3a44700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc0b3990 con 0x7f5bdc06a730 2024-04-06T12:13:28.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.473+0000 7f5be3a44700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bdc15e950 con 0x7f5bdc15d7b0 2024-04-06T12:13:28.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.473+0000 7f5be3a44700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bdc06b0f0 con 0x7f5bdc15e500 2024-04-06T12:13:28.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.473+0000 7f5be17e0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5bdc15d7b0 0x7f5bdc0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:28.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.473+0000 7f5be0fdf700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5bdc06a730 0x7f5bdc15ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33546/0 (socket says 172.21.15.189:33546) 2024-04-06T12:13:28.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.473+0000 7f5be0fdf700 1 -- 172.21.15.189:0/275388222 learned_addr learned my addr 172.21.15.189:0/275388222 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:28.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.473+0000 7f5be1fe1700 1 --2- 172.21.15.189:0/275388222 >> v2:172.21.15.192:3300/0 conn(0x7f5bdc15e500 0x7f5bdc06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:28.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.473+0000 7f5bd3fff700 1 -- 172.21.15.189:0/275388222 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 341781943 0 0) 0x7f5bdc0b3990 con 0x7f5bdc06a730 2024-04-06T12:13:28.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.473+0000 7f5bd3fff700 1 -- 172.21.15.189:0/275388222 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bc80035b0 con 0x7f5bdc06a730 2024-04-06T12:13:28.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.473+0000 7f5be17e0700 1 -- 172.21.15.189:0/275388222 >> v2:172.21.15.192:3300/0 conn(0x7f5bdc15e500 msgr2=0x7f5bdc06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:28.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.473+0000 7f5be17e0700 1 --2- 172.21.15.189:0/275388222 >> v2:172.21.15.192:3300/0 conn(0x7f5bdc15e500 0x7f5bdc06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:28.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.473+0000 7f5be17e0700 1 -- 172.21.15.189:0/275388222 >> v1:172.21.15.189:6789/0 conn(0x7f5bdc06a730 legacy=0x7f5bdc15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:28.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.473+0000 7f5be17e0700 1 -- 172.21.15.189:0/275388222 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bdc0b3060 con 0x7f5bdc15d7b0 2024-04-06T12:13:28.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.473+0000 7f5be17e0700 1 --2- 172.21.15.189:0/275388222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bdc15d7b0 0x7f5bdc0b3e20 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc0673e0 tx=0x7f5bd8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de537e897de664c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:28.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.474+0000 7f5bd3fff700 1 -- 172.21.15.189:0/275388222 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8011830 con 0x7f5bdc15d7b0 2024-04-06T12:13:28.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.474+0000 7f5bd3fff700 1 -- 172.21.15.189:0/275388222 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd8011990 con 0x7f5bdc15d7b0 2024-04-06T12:13:28.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.474+0000 7f5bd3fff700 1 -- 172.21.15.189:0/275388222 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8011d20 con 0x7f5bdc15d7b0 2024-04-06T12:13:28.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.474+0000 7f5be3a44700 1 -- 172.21.15.189:0/275388222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bdc15d7b0 msgr2=0x7f5bdc0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:28.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.474+0000 7f5be3a44700 1 --2- 172.21.15.189:0/275388222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bdc15d7b0 0x7f5bdc0b3e20 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc0673e0 tx=0x7f5bd8009b20 comp rx=0 tx=0).stop 2024-04-06T12:13:28.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.474+0000 7f5be3a44700 1 -- 172.21.15.189:0/275388222 shutdown_connections 2024-04-06T12:13:28.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.474+0000 7f5be3a44700 1 --2- 172.21.15.189:0/275388222 >> v2:172.21.15.192:3300/0 conn(0x7f5bdc15e500 0x7f5bdc06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:28.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.474+0000 7f5be3a44700 1 --2- 172.21.15.189:0/275388222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bdc15d7b0 0x7f5bdc0b3e20 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:28.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.474+0000 7f5be3a44700 1 -- 172.21.15.189:0/275388222 >> 172.21.15.189:0/275388222 conn(0x7f5bdc0d00b0 msgr2=0x7f5bdc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:28.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.474+0000 7f5be3a44700 1 -- 172.21.15.189:0/275388222 shutdown_connections 2024-04-06T12:13:28.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.474+0000 7f5be3a44700 1 -- 172.21.15.189:0/275388222 wait complete. 2024-04-06T12:13:28.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.475+0000 7f5be3a44700 1 Processor -- start 2024-04-06T12:13:28.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.475+0000 7f5be3a44700 1 -- start start 2024-04-06T12:13:28.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.476+0000 7f5be3a44700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bdc06a730 0x7f5bdc0acf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:28.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.476+0000 7f5be3a44700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5bdc15d7b0 0x7f5bdc149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:28.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.476+0000 7f5be3a44700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bdc15e500 0x7f5bdc14a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:28.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.476+0000 7f5be3a44700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc15a370 con 0x7f5bdc06a730 2024-04-06T12:13:28.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.476+0000 7f5be3a44700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc15ca80 con 0x7f5bdc15e500 2024-04-06T12:13:28.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.476+0000 7f5be3a44700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc15a610 con 0x7f5bdc15d7b0 2024-04-06T12:13:28.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.476+0000 7f5be17e0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bdc06a730 0x7f5bdc0acf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:28.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.476+0000 7f5be17e0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bdc06a730 0x7f5bdc0acf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33310/0 (socket says 172.21.15.189:33310) 2024-04-06T12:13:28.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.476+0000 7f5be17e0700 1 -- 172.21.15.189:0/934156953 learned_addr learned my addr 172.21.15.189:0/934156953 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:28.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.476+0000 7f5be17e0700 1 -- 172.21.15.189:0/934156953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5bdc15d7b0 msgr2=0x7f5bdc149b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:28.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.476+0000 7f5be17e0700 1 --2- 172.21.15.189:0/934156953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5bdc15d7b0 0x7f5bdc149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:28.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.476+0000 7f5be17e0700 1 -- 172.21.15.189:0/934156953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bdc15e500 msgr2=0x7f5bdc14a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:28.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.476+0000 7f5be17e0700 1 --2- 172.21.15.189:0/934156953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bdc15e500 0x7f5bdc14a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:28.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.476+0000 7f5be17e0700 1 -- 172.21.15.189:0/934156953 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd80094a0 con 0x7f5bdc06a730 2024-04-06T12:13:28.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.477+0000 7f5be17e0700 1 --2- 172.21.15.189:0/934156953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bdc06a730 0x7f5bdc0acf70 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8009b60 tx=0x7f5bd8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:28.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.477+0000 7f5bd1ffb700 1 -- 172.21.15.189:0/934156953 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd80036a0 con 0x7f5bdc06a730 2024-04-06T12:13:28.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.477+0000 7f5bd1ffb700 1 -- 172.21.15.189:0/934156953 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd801f070 con 0x7f5bdc06a730 2024-04-06T12:13:28.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.477+0000 7f5be3a44700 1 -- 172.21.15.189:0/934156953 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bdc15a8f0 con 0x7f5bdc06a730 2024-04-06T12:13:28.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.477+0000 7f5bd1ffb700 1 -- 172.21.15.189:0/934156953 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8003a40 con 0x7f5bdc06a730 2024-04-06T12:13:28.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.477+0000 7f5be3a44700 1 -- 172.21.15.189:0/934156953 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bdc0ad4b0 con 0x7f5bdc06a730 2024-04-06T12:13:28.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.478+0000 7f5be3a44700 1 -- 172.21.15.189:0/934156953 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bdc0ad700 con 0x7f5bdc06a730 2024-04-06T12:13:28.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.479+0000 7f5bd1ffb700 1 -- 172.21.15.189:0/934156953 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5bd8003ba0 con 0x7f5bdc06a730 2024-04-06T12:13:28.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.480+0000 7f5bd1ffb700 1 --2- 172.21.15.189:0/934156953 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5bc006ac90 0x7f5bc006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:28.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.480+0000 7f5bd1ffb700 1 -- 172.21.15.189:0/934156953 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5bd8021070 con 0x7f5bdc06a730 2024-04-06T12:13:28.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.480+0000 7f5be0fdf700 1 --2- 172.21.15.189:0/934156953 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5bc006ac90 0x7f5bc006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:28.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.480+0000 7f5be0fdf700 1 --2- 172.21.15.189:0/934156953 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5bc006ac90 0x7f5bc006d150 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc072650 tx=0x7f5bcc0079c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:28.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.483+0000 7f5bd1ffb700 1 -- 172.21.15.189:0/934156953 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5bd8056520 con 0x7f5bdc06a730 2024-04-06T12:13:28.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.708+0000 7f5be3a44700 1 -- 172.21.15.189:0/934156953 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f5bdc15eb40 con 0x7f5bc006ac90 2024-04-06T12:13:28.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.710+0000 7f5bd1ffb700 1 -- 172.21.15.189:0/934156953 <== mgr.4100 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f5bcc005940 con 0x7f5bc006ac90 2024-04-06T12:13:28.708 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T12:13:28.708 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:13:28.708 INFO:teuthology.orchestra.run.smithi189.stdout: "name": "cephfs" 2024-04-06T12:13:28.708 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:13:28.708 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T12:13:28.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.712+0000 7f5be3a44700 1 -- 172.21.15.189:0/934156953 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5bc006ac90 msgr2=0x7f5bc006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:28.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.712+0000 7f5be3a44700 1 --2- 172.21.15.189:0/934156953 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5bc006ac90 0x7f5bc006d150 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc072650 tx=0x7f5bcc0079c0 comp rx=0 tx=0).stop 2024-04-06T12:13:28.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.712+0000 7f5be3a44700 1 -- 172.21.15.189:0/934156953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bdc06a730 msgr2=0x7f5bdc0acf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:28.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.712+0000 7f5be3a44700 1 --2- 172.21.15.189:0/934156953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bdc06a730 0x7f5bdc0acf70 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8009b60 tx=0x7f5bd8004750 comp rx=0 tx=0).stop 2024-04-06T12:13:28.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.712+0000 7f5be3a44700 1 -- 172.21.15.189:0/934156953 shutdown_connections 2024-04-06T12:13:28.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.713+0000 7f5be3a44700 1 --2- 172.21.15.189:0/934156953 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5bc006ac90 0x7f5bc006d150 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:28.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.713+0000 7f5be3a44700 1 --2- 172.21.15.189:0/934156953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bdc15e500 0x7f5bdc14a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:28.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.713+0000 7f5be3a44700 1 --2- 172.21.15.189:0/934156953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bdc06a730 0x7f5bdc0acf70 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:28.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.713+0000 7f5be3a44700 1 --2- 172.21.15.189:0/934156953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5bdc15d7b0 0x7f5bdc149b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:28.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.713+0000 7f5be3a44700 1 -- 172.21.15.189:0/934156953 >> 172.21.15.189:0/934156953 conn(0x7f5bdc0d00b0 msgr2=0x7f5bdc072730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:28.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.713+0000 7f5be3a44700 1 -- 172.21.15.189:0/934156953 shutdown_connections 2024-04-06T12:13:28.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.713+0000 7f5be3a44700 1 -- 172.21.15.189:0/934156953 wait complete. 2024-04-06T12:13:28.723 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-06T12:13:28.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.848+0000 7f7e76b39700 1 Processor -- start 2024-04-06T12:13:28.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.848+0000 7f7e76b39700 1 -- start start 2024-04-06T12:13:28.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.849+0000 7f7e76b39700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7e7015a550 0x7f7e700cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:28.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.849+0000 7f7e76b39700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e700cdbe0 con 0x7f7e700cf900 2024-04-06T12:13:28.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.849+0000 7f7e76b39700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e7015a940 con 0x7f7e700b4d70 2024-04-06T12:13:28.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.849+0000 7f7e76b39700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7015ac00 con 0x7f7e7015a550 2024-04-06T12:13:28.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.849+0000 7f7e6ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7e700cf900 0x7f7e700b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33560/0 (socket says 172.21.15.189:33560) 2024-04-06T12:13:28.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.849+0000 7f7e6ffff700 1 -- 172.21.15.189:0/4215687660 learned_addr learned my addr 172.21.15.189:0/4215687660 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:28.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.849+0000 7f7e748d5700 1 --2- 172.21.15.189:0/4215687660 >> v2:172.21.15.192:3300/0 conn(0x7f7e7015a550 0x7f7e700cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:28.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.849+0000 7f7e6f7fe700 1 -- 172.21.15.189:0/4215687660 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202562603 0 0) 0x7f7e700cdbe0 con 0x7f7e700cf900 2024-04-06T12:13:28.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.849+0000 7f7e6f7fe700 1 -- 172.21.15.189:0/4215687660 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e600035b0 con 0x7f7e700cf900 2024-04-06T12:13:28.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6f7fe700 1 -- 172.21.15.189:0/4215687660 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2612261722 0 0) 0x7f7e600035b0 con 0x7f7e700cf900 2024-04-06T12:13:28.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6f7fe700 1 -- 172.21.15.189:0/4215687660 >> v2:172.21.15.192:3300/0 conn(0x7f7e7015a550 msgr2=0x7f7e700cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:28.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6f7fe700 1 --2- 172.21.15.189:0/4215687660 >> v2:172.21.15.192:3300/0 conn(0x7f7e7015a550 0x7f7e700cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:28.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6f7fe700 1 -- 172.21.15.189:0/4215687660 >> v1:172.21.15.192:6789/0 conn(0x7f7e700b4d70 legacy=0x7f7e700b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:28.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6f7fe700 1 -- 172.21.15.189:0/4215687660 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7015aee0 con 0x7f7e700cf900 2024-04-06T12:13:28.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6f7fe700 1 -- 172.21.15.189:0/4215687660 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f7e5c001670 con 0x7f7e700cf900 2024-04-06T12:13:28.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6f7fe700 1 -- 172.21.15.189:0/4215687660 >> v1:172.21.15.189:6789/0 conn(0x7f7e700cf900 legacy=0x7f7e700b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:28.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6f7fe700 1 --2- 172.21.15.189:0/4215687660 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e60003a00 0x7f7e60003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:28.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6f7fe700 1 --2- 172.21.15.189:0/4215687660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e60004850 0x7f7e60004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:28.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6f7fe700 1 --2- 172.21.15.189:0/4215687660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e60005210 0x7f7e6000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:28.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6f7fe700 1 -- 172.21.15.189:0/4215687660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7015ac00 con 0x7f7e60004850 2024-04-06T12:13:28.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6f7fe700 1 -- 172.21.15.189:0/4215687660 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6000c0b0 con 0x7f7e60005210 2024-04-06T12:13:28.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6f7fe700 1 -- 172.21.15.189:0/4215687660 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6000c270 con 0x7f7e60003a00 2024-04-06T12:13:28.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6ffff700 1 --2- 172.21.15.189:0/4215687660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e60004850 0x7f7e60004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:28.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e748d5700 1 --2- 172.21.15.189:0/4215687660 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e60003a00 0x7f7e60003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:28.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e750d6700 1 --2- 172.21.15.189:0/4215687660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e60005210 0x7f7e6000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:28.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6ffff700 1 -- 172.21.15.189:0/4215687660 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e60003a00 msgr2=0x7f7e60003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:28.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6ffff700 1 --2- 172.21.15.189:0/4215687660 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e60003a00 0x7f7e60003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:28.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6ffff700 1 -- 172.21.15.189:0/4215687660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e60005210 msgr2=0x7f7e6000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:28.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6ffff700 1 --2- 172.21.15.189:0/4215687660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e60005210 0x7f7e6000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:28.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e750d6700 1 -- 172.21.15.189:0/4215687660 reap_dead start 2024-04-06T12:13:28.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6ffff700 1 -- 172.21.15.189:0/4215687660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e600035b0 con 0x7f7e60004850 2024-04-06T12:13:28.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.850+0000 7f7e6ffff700 1 --2- 172.21.15.189:0/4215687660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e60004850 0x7f7e60004cd0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00c240 tx=0x7f7e60005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31de68973a0bb59 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:28.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.851+0000 7f7e6f7fe700 1 -- 172.21.15.189:0/4215687660 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c007c10 con 0x7f7e60004850 2024-04-06T12:13:28.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.851+0000 7f7e6f7fe700 1 -- 172.21.15.189:0/4215687660 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5c00cde0 con 0x7f7e60004850 2024-04-06T12:13:28.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.851+0000 7f7e6f7fe700 1 -- 172.21.15.189:0/4215687660 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c005ba0 con 0x7f7e60004850 2024-04-06T12:13:28.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.851+0000 7f7e76b39700 1 -- 172.21.15.189:0/4215687660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e60004850 msgr2=0x7f7e60004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:28.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.851+0000 7f7e76b39700 1 --2- 172.21.15.189:0/4215687660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e60004850 0x7f7e60004cd0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00c240 tx=0x7f7e60005880 comp rx=0 tx=0).stop 2024-04-06T12:13:28.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.851+0000 7f7e76b39700 1 -- 172.21.15.189:0/4215687660 shutdown_connections 2024-04-06T12:13:28.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.851+0000 7f7e76b39700 1 --2- 172.21.15.189:0/4215687660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e60004850 0x7f7e60004cd0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:28.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.851+0000 7f7e76b39700 1 -- 172.21.15.189:0/4215687660 >> 172.21.15.189:0/4215687660 conn(0x7f7e700d00b0 msgr2=0x7f7e700d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:28.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.851+0000 7f7e76b39700 1 -- 172.21.15.189:0/4215687660 shutdown_connections 2024-04-06T12:13:28.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.851+0000 7f7e76b39700 1 -- 172.21.15.189:0/4215687660 wait complete. 2024-04-06T12:13:28.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.852+0000 7f7e76b39700 1 Processor -- start 2024-04-06T12:13:28.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.852+0000 7f7e76b39700 1 -- start start 2024-04-06T12:13:28.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.853+0000 7f7e76b39700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e60004850 0x7f7e70150750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:28.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.853+0000 7f7e76b39700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e701f3e40 0x7f7e700b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:28.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.853+0000 7f7e76b39700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e7014b160 0x7f7e700b1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:28.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.853+0000 7f7e76b39700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7014b660 con 0x7f7e60004850 2024-04-06T12:13:28.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.853+0000 7f7e76b39700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e701f4320 con 0x7f7e7014b160 2024-04-06T12:13:28.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.853+0000 7f7e76b39700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e701f4600 con 0x7f7e701f3e40 2024-04-06T12:13:28.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.853+0000 7f7e748d5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e60004850 0x7f7e70150750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:28.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.853+0000 7f7e748d5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e60004850 0x7f7e70150750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33320/0 (socket says 172.21.15.189:33320) 2024-04-06T12:13:28.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.853+0000 7f7e748d5700 1 -- 172.21.15.189:0/3356695431 learned_addr learned my addr 172.21.15.189:0/3356695431 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:28.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.853+0000 7f7e6ffff700 1 --2- 172.21.15.189:0/3356695431 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e701f3e40 0x7f7e700b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:28.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.853+0000 7f7e750d6700 1 --2- 172.21.15.189:0/3356695431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e7014b160 0x7f7e700b1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:28.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.854+0000 7f7e748d5700 1 -- 172.21.15.189:0/3356695431 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e701f3e40 msgr2=0x7f7e700b0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:28.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.854+0000 7f7e748d5700 1 --2- 172.21.15.189:0/3356695431 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e701f3e40 0x7f7e700b0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:28.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.854+0000 7f7e748d5700 1 -- 172.21.15.189:0/3356695431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e7014b160 msgr2=0x7f7e700b1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:28.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.854+0000 7f7e748d5700 1 --2- 172.21.15.189:0/3356695431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e7014b160 0x7f7e700b1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:28.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.854+0000 7f7e748d5700 1 -- 172.21.15.189:0/3356695431 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e701f48e0 con 0x7f7e60004850 2024-04-06T12:13:28.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.854+0000 7f7e750d6700 1 --2- 172.21.15.189:0/3356695431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e7014b160 0x7f7e700b1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:28.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.854+0000 7f7e748d5700 1 --2- 172.21.15.189:0/3356695431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e60004850 0x7f7e70150750 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7e64011aa0 tx=0x7f7e64011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:28.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.854+0000 7f7e6d7fa700 1 -- 172.21.15.189:0/3356695431 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400f940 con 0x7f7e60004850 2024-04-06T12:13:28.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.854+0000 7f7e6d7fa700 1 -- 172.21.15.189:0/3356695431 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6400faa0 con 0x7f7e60004850 2024-04-06T12:13:28.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.854+0000 7f7e76b39700 1 -- 172.21.15.189:0/3356695431 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e701f4bc0 con 0x7f7e60004850 2024-04-06T12:13:28.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.854+0000 7f7e6d7fa700 1 -- 172.21.15.189:0/3356695431 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400fe50 con 0x7f7e60004850 2024-04-06T12:13:28.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.854+0000 7f7e76b39700 1 -- 172.21.15.189:0/3356695431 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e70158830 con 0x7f7e60004850 2024-04-06T12:13:28.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.855+0000 7f7e76b39700 1 -- 172.21.15.189:0/3356695431 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7014b900 con 0x7f7e60004850 2024-04-06T12:13:28.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.855+0000 7f7e6d7fa700 1 -- 172.21.15.189:0/3356695431 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7e64014440 con 0x7f7e60004850 2024-04-06T12:13:28.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.856+0000 7f7e6d7fa700 1 --2- 172.21.15.189:0/3356695431 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e5406ac90 0x7f7e5406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:28.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.856+0000 7f7e6d7fa700 1 -- 172.21.15.189:0/3356695431 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7e64089c80 con 0x7f7e60004850 2024-04-06T12:13:28.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.856+0000 7f7e6ffff700 1 --2- 172.21.15.189:0/3356695431 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e5406ac90 0x7f7e5406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:28.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.856+0000 7f7e6ffff700 1 --2- 172.21.15.189:0/3356695431 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e5406ac90 0x7f7e5406d150 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c001930 tx=0x7f7e5c007000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:28.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:28.860+0000 7f7e6d7fa700 1 -- 172.21.15.189:0/3356695431 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7e64054d20 con 0x7f7e60004850 2024-04-06T12:13:29.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.112+0000 7f7e76b39700 1 -- 172.21.15.189:0/3356695431 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f7e701f4d50 con 0x7f7e60004850 2024-04-06T12:13:29.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.360+0000 7f7e6d7fa700 1 -- 172.21.15.189:0/3356695431 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 6) v1 ==== 89162+0+0 (secure 0 0 0) 0x7f7e640550f0 con 0x7f7e60004850 2024-04-06T12:13:29.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.361+0000 7f7e6d7fa700 1 -- 172.21.15.189:0/3356695431 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) v1 ==== 92+0+0 (secure 0 0 0) 0x7f7e64054d20 con 0x7f7e60004850 2024-04-06T12:13:29.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.364+0000 7f7e76b39700 1 -- 172.21.15.189:0/3356695431 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e5406ac90 msgr2=0x7f7e5406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:29.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.364+0000 7f7e76b39700 1 --2- 172.21.15.189:0/3356695431 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e5406ac90 0x7f7e5406d150 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c001930 tx=0x7f7e5c007000 comp rx=0 tx=0).stop 2024-04-06T12:13:29.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.364+0000 7f7e76b39700 1 -- 172.21.15.189:0/3356695431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e60004850 msgr2=0x7f7e70150750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:29.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.364+0000 7f7e76b39700 1 --2- 172.21.15.189:0/3356695431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e60004850 0x7f7e70150750 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7e64011aa0 tx=0x7f7e64011db0 comp rx=0 tx=0).stop 2024-04-06T12:13:29.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.364+0000 7f7e76b39700 1 -- 172.21.15.189:0/3356695431 shutdown_connections 2024-04-06T12:13:29.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.364+0000 7f7e76b39700 1 --2- 172.21.15.189:0/3356695431 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e5406ac90 0x7f7e5406d150 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:29.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.364+0000 7f7e76b39700 1 --2- 172.21.15.189:0/3356695431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e7014b160 0x7f7e700b1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:29.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.364+0000 7f7e76b39700 1 --2- 172.21.15.189:0/3356695431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e60004850 0x7f7e70150750 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:29.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.364+0000 7f7e76b39700 1 --2- 172.21.15.189:0/3356695431 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e701f3e40 0x7f7e700b0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:29.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.364+0000 7f7e76b39700 1 -- 172.21.15.189:0/3356695431 >> 172.21.15.189:0/3356695431 conn(0x7f7e700d00b0 msgr2=0x7f7e7014d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:29.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.364+0000 7f7e76b39700 1 -- 172.21.15.189:0/3356695431 shutdown_connections 2024-04-06T12:13:29.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.365+0000 7f7e76b39700 1 -- 172.21.15.189:0/3356695431 wait complete. 2024-04-06T12:13:29.379 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-06T12:13:29.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.511+0000 7f097c4b6700 1 Processor -- start 2024-04-06T12:13:29.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.511+0000 7f097c4b6700 1 -- start start 2024-04-06T12:13:29.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.512+0000 7f097c4b6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f09740d0a60 0x7f09740d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:29.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.512+0000 7f097c4b6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09740cdf80 con 0x7f09740d1af0 2024-04-06T12:13:29.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.512+0000 7f097c4b6700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09740d1f60 con 0x7f097415d7e0 2024-04-06T12:13:29.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.512+0000 7f097c4b6700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09740d3230 con 0x7f09740d0a60 2024-04-06T12:13:29.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.512+0000 7f0979a51700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f09740d0a60 0x7f09740d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:29.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.512+0000 7f097aa53700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f09740d1af0 0x7f09740b66b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39754/0 (socket says 172.21.15.189:39754) 2024-04-06T12:13:29.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.512+0000 7f097aa53700 1 -- 172.21.15.189:0/3829633213 learned_addr learned my addr 172.21.15.189:0/3829633213 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:29.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.512+0000 7f0979250700 1 -- 172.21.15.189:0/3829633213 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116050609 0 0) 0x7f09740cdf80 con 0x7f09740d1af0 2024-04-06T12:13:29.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.512+0000 7f0979250700 1 -- 172.21.15.189:0/3829633213 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09600035b0 con 0x7f09740d1af0 2024-04-06T12:13:29.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.512+0000 7f0979a51700 1 -- 172.21.15.189:0/3829633213 >> v1:172.21.15.192:6789/0 conn(0x7f097415d7e0 legacy=0x7f09740d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:29.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.512+0000 7f0979a51700 1 -- 172.21.15.189:0/3829633213 >> v1:172.21.15.189:6789/0 conn(0x7f09740d1af0 legacy=0x7f09740b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:29.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.512+0000 7f0979a51700 1 -- 172.21.15.189:0/3829633213 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09740d2da0 con 0x7f09740d0a60 2024-04-06T12:13:29.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.512+0000 7f0979a51700 1 --2- 172.21.15.189:0/3829633213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09740d0a60 0x7f09740d15b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0964009840 tx=0x7f0964009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e19478a1aea93cb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:29.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.513+0000 7f0979250700 1 -- 172.21.15.189:0/3829633213 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0964011830 con 0x7f09740d0a60 2024-04-06T12:13:29.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.513+0000 7f0979250700 1 -- 172.21.15.189:0/3829633213 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0964011990 con 0x7f09740d0a60 2024-04-06T12:13:29.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.513+0000 7f0979250700 1 -- 172.21.15.189:0/3829633213 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0964011d00 con 0x7f09740d0a60 2024-04-06T12:13:29.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.513+0000 7f097c4b6700 1 -- 172.21.15.189:0/3829633213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09740d0a60 msgr2=0x7f09740d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:29.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.513+0000 7f097c4b6700 1 --2- 172.21.15.189:0/3829633213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09740d0a60 0x7f09740d15b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0964009840 tx=0x7f0964009b50 comp rx=0 tx=0).stop 2024-04-06T12:13:29.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.513+0000 7f097c4b6700 1 -- 172.21.15.189:0/3829633213 shutdown_connections 2024-04-06T12:13:29.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.513+0000 7f097c4b6700 1 --2- 172.21.15.189:0/3829633213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09740d0a60 0x7f09740d15b0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:29.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.513+0000 7f097c4b6700 1 -- 172.21.15.189:0/3829633213 >> 172.21.15.189:0/3829633213 conn(0x7f0974072250 msgr2=0x7f0974072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:29.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.513+0000 7f097c4b6700 1 -- 172.21.15.189:0/3829633213 shutdown_connections 2024-04-06T12:13:29.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.513+0000 7f097c4b6700 1 -- 172.21.15.189:0/3829633213 wait complete. 2024-04-06T12:13:29.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.515+0000 7f097c4b6700 1 Processor -- start 2024-04-06T12:13:29.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.515+0000 7f097c4b6700 1 -- start start 2024-04-06T12:13:29.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.515+0000 7f097c4b6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09740d0a60 0x7f09740b4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:29.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.515+0000 7f097c4b6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09740d1af0 0x7f09741496a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:29.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.515+0000 7f097c4b6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f097415d7e0 0x7f09740b2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:29.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.515+0000 7f097c4b6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09740b3cf0 con 0x7f097415d7e0 2024-04-06T12:13:29.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.515+0000 7f097c4b6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09740b4650 con 0x7f09740d0a60 2024-04-06T12:13:29.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.515+0000 7f097c4b6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09740b3070 con 0x7f09740d1af0 2024-04-06T12:13:29.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.515+0000 7f097aa53700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f097415d7e0 0x7f09740b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:29.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.515+0000 7f097aa53700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f097415d7e0 0x7f09740b2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51170/0 (socket says 172.21.15.189:51170) 2024-04-06T12:13:29.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.515+0000 7f097aa53700 1 -- 172.21.15.189:0/3812307513 learned_addr learned my addr 172.21.15.189:0/3812307513 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:29.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.515+0000 7f097a252700 1 --2- 172.21.15.189:0/3812307513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09740d0a60 0x7f09740b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:29.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.516+0000 7f0979a51700 1 --2- 172.21.15.189:0/3812307513 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09740d1af0 0x7f09741496a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:29.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.516+0000 7f097aa53700 1 -- 172.21.15.189:0/3812307513 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09740d1af0 msgr2=0x7f09741496a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:29.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.516+0000 7f097aa53700 1 --2- 172.21.15.189:0/3812307513 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09740d1af0 0x7f09741496a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:29.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.516+0000 7f097aa53700 1 -- 172.21.15.189:0/3812307513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09740d0a60 msgr2=0x7f09740b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:29.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.516+0000 7f097aa53700 1 --2- 172.21.15.189:0/3812307513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09740d0a60 0x7f09740b4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:29.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.516+0000 7f097aa53700 1 -- 172.21.15.189:0/3812307513 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09640094a0 con 0x7f097415d7e0 2024-04-06T12:13:29.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.516+0000 7f097aa53700 1 --2- 172.21.15.189:0/3812307513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f097415d7e0 0x7f09740b2620 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f096c00cb40 tx=0x7f096c00ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:29.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.516+0000 7f096affd700 1 -- 172.21.15.189:0/3812307513 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096c00a9e0 con 0x7f097415d7e0 2024-04-06T12:13:29.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.516+0000 7f096affd700 1 -- 172.21.15.189:0/3812307513 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f096c00ab40 con 0x7f097415d7e0 2024-04-06T12:13:29.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.516+0000 7f097c4b6700 1 -- 172.21.15.189:0/3812307513 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09740b3350 con 0x7f097415d7e0 2024-04-06T12:13:29.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.516+0000 7f096affd700 1 -- 172.21.15.189:0/3812307513 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096c012400 con 0x7f097415d7e0 2024-04-06T12:13:29.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.516+0000 7f097c4b6700 1 -- 172.21.15.189:0/3812307513 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09740b3910 con 0x7f097415d7e0 2024-04-06T12:13:29.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.517+0000 7f097c4b6700 1 -- 172.21.15.189:0/3812307513 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0974074990 con 0x7f097415d7e0 2024-04-06T12:13:29.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.518+0000 7f096affd700 1 -- 172.21.15.189:0/3812307513 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 6) v1 ==== 89162+0+0 (secure 0 0 0) 0x7f096c01b3b0 con 0x7f097415d7e0 2024-04-06T12:13:29.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.518+0000 7f096affd700 1 --2- 172.21.15.189:0/3812307513 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f095806ace0 0x7f095806d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:29.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.518+0000 7f097a252700 1 -- 172.21.15.189:0/3812307513 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f095806ace0 msgr2=0x7f095806d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:13:29.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.518+0000 7f096affd700 1 -- 172.21.15.189:0/3812307513 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f096c088fe0 con 0x7f097415d7e0 2024-04-06T12:13:29.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.518+0000 7f097a252700 1 --2- 172.21.15.189:0/3812307513 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f095806ace0 0x7f095806d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:13:29.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.523+0000 7f096affd700 1 -- 172.21.15.189:0/3812307513 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f096c054160 con 0x7f097415d7e0 2024-04-06T12:13:29.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.719+0000 7f097a252700 1 -- 172.21.15.189:0/3812307513 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f095806ace0 msgr2=0x7f095806d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:13:29.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.719+0000 7f097a252700 1 --2- 172.21.15.189:0/3812307513 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f095806ace0 0x7f095806d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:13:29.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.728+0000 7f097c4b6700 1 -- 172.21.15.189:0/3812307513 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f09740d1640 con 0x7f097415d7e0 2024-04-06T12:13:29.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.732+0000 7f096affd700 1 -- 172.21.15.189:0/3812307513 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v11)=0 v11) v1 ==== 138+0+0 (secure 0 0 0) 0x7f096c0170d0 con 0x7f097415d7e0 2024-04-06T12:13:29.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.734+0000 7f097c4b6700 1 -- 172.21.15.189:0/3812307513 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f095806ace0 msgr2=0x7f095806d1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:13:29.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.734+0000 7f097c4b6700 1 --2- 172.21.15.189:0/3812307513 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f095806ace0 0x7f095806d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:29.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.734+0000 7f097c4b6700 1 -- 172.21.15.189:0/3812307513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f097415d7e0 msgr2=0x7f09740b2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:29.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.734+0000 7f097c4b6700 1 --2- 172.21.15.189:0/3812307513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f097415d7e0 0x7f09740b2620 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f096c00cb40 tx=0x7f096c00ce50 comp rx=0 tx=0).stop 2024-04-06T12:13:29.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.734+0000 7f097c4b6700 1 -- 172.21.15.189:0/3812307513 shutdown_connections 2024-04-06T12:13:29.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.734+0000 7f097c4b6700 1 --2- 172.21.15.189:0/3812307513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f097415d7e0 0x7f09740b2620 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:29.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.734+0000 7f097c4b6700 1 --2- 172.21.15.189:0/3812307513 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f095806ace0 0x7f095806d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:29.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.734+0000 7f097c4b6700 1 --2- 172.21.15.189:0/3812307513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09740d0a60 0x7f09740b4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:29.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.734+0000 7f097c4b6700 1 --2- 172.21.15.189:0/3812307513 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09740d1af0 0x7f09741496a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:29.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.734+0000 7f097c4b6700 1 -- 172.21.15.189:0/3812307513 >> 172.21.15.189:0/3812307513 conn(0x7f0974072250 msgr2=0x7f09740b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:29.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.734+0000 7f097c4b6700 1 -- 172.21.15.189:0/3812307513 shutdown_connections 2024-04-06T12:13:29.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.734+0000 7f097c4b6700 1 -- 172.21.15.189:0/3812307513 wait complete. 2024-04-06T12:13:29.741 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:29.743+0000 7f8973863200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:13:29.745 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-06T12:13:29.760 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:29.762+0000 7ff9d7f03200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:13:29.820 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:29.822+0000 7f8973863200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:13:29.844 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:29.846+0000 7ff9d7f03200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:13:29.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.875+0000 7f7d87952700 1 Processor -- start 2024-04-06T12:13:29.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.875+0000 7f7d87952700 1 -- start start 2024-04-06T12:13:29.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.876+0000 7f7d87952700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d8015d590 0x7f7d800b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:29.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.876+0000 7f7d87952700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7d800b4ed0 0x7f7d800b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:29.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.876+0000 7f7d87952700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d800b5cc0 con 0x7f7d800b6580 2024-04-06T12:13:29.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.876+0000 7f7d87952700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d800b58c0 con 0x7f7d8015d590 2024-04-06T12:13:29.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.876+0000 7f7d87952700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d800d1500 con 0x7f7d800b4ed0 2024-04-06T12:13:29.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.876+0000 7f7d856ee700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d8015d590 0x7f7d800b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:29.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.876+0000 7f7d856ee700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d8015d590 0x7f7d800b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51174/0 (socket says 172.21.15.189:51174) 2024-04-06T12:13:29.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.876+0000 7f7d856ee700 1 -- 172.21.15.189:0/2874130278 learned_addr learned my addr 172.21.15.189:0/2874130278 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:29.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.876+0000 7f7d84eed700 1 --2- 172.21.15.189:0/2874130278 >> v2:172.21.15.192:3300/0 conn(0x7f7d800b4ed0 0x7f7d800b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:29.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.876+0000 7f7d856ee700 1 -- 172.21.15.189:0/2874130278 >> v2:172.21.15.192:3300/0 conn(0x7f7d800b4ed0 msgr2=0x7f7d800b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:29.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.876+0000 7f7d856ee700 1 --2- 172.21.15.189:0/2874130278 >> v2:172.21.15.192:3300/0 conn(0x7f7d800b4ed0 0x7f7d800b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:29.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.876+0000 7f7d856ee700 1 -- 172.21.15.189:0/2874130278 >> v1:172.21.15.189:6789/0 conn(0x7f7d800b6580 legacy=0x7f7d800b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:29.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.876+0000 7f7d77fff700 1 -- 172.21.15.189:0/2874130278 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251151467 0 0) 0x7f7d800b5cc0 con 0x7f7d800b6580 2024-04-06T12:13:29.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.876+0000 7f7d856ee700 1 -- 172.21.15.189:0/2874130278 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d800d17e0 con 0x7f7d8015d590 2024-04-06T12:13:29.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.877+0000 7f7d856ee700 1 --2- 172.21.15.189:0/2874130278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d8015d590 0x7f7d800b8d70 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7d800b6330 tx=0x7f7d7c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f67612b7a9cc188 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:29.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.877+0000 7f7d77fff700 1 -- 172.21.15.189:0/2874130278 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c011830 con 0x7f7d8015d590 2024-04-06T12:13:29.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.877+0000 7f7d77fff700 1 -- 172.21.15.189:0/2874130278 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7c011990 con 0x7f7d8015d590 2024-04-06T12:13:29.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.877+0000 7f7d77fff700 1 -- 172.21.15.189:0/2874130278 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c011d20 con 0x7f7d8015d590 2024-04-06T12:13:29.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.877+0000 7f7d87952700 1 -- 172.21.15.189:0/2874130278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d8015d590 msgr2=0x7f7d800b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:29.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.877+0000 7f7d87952700 1 --2- 172.21.15.189:0/2874130278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d8015d590 0x7f7d800b8d70 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7d800b6330 tx=0x7f7d7c009bd0 comp rx=0 tx=0).stop 2024-04-06T12:13:29.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.877+0000 7f7d87952700 1 -- 172.21.15.189:0/2874130278 shutdown_connections 2024-04-06T12:13:29.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.877+0000 7f7d87952700 1 --2- 172.21.15.189:0/2874130278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d8015d590 0x7f7d800b8d70 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:29.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.877+0000 7f7d87952700 1 --2- 172.21.15.189:0/2874130278 >> v2:172.21.15.192:3300/0 conn(0x7f7d800b4ed0 0x7f7d800b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:29.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.877+0000 7f7d87952700 1 -- 172.21.15.189:0/2874130278 >> 172.21.15.189:0/2874130278 conn(0x7f7d8006bbc0 msgr2=0x7f7d8006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:29.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.877+0000 7f7d87952700 1 -- 172.21.15.189:0/2874130278 shutdown_connections 2024-04-06T12:13:29.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.878+0000 7f7d87952700 1 -- 172.21.15.189:0/2874130278 wait complete. 2024-04-06T12:13:29.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.879+0000 7f7d87952700 1 Processor -- start 2024-04-06T12:13:29.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.879+0000 7f7d87952700 1 -- start start 2024-04-06T12:13:29.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.880+0000 7f7d87952700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d800b4ed0 0x7f7d8014d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:29.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.880+0000 7f7d87952700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d800b6580 0x7f7d8015e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:29.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.880+0000 7f7d87952700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d8015d590 0x7f7d8015e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:29.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.880+0000 7f7d87952700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8014d7d0 con 0x7f7d8015d590 2024-04-06T12:13:29.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.880+0000 7f7d87952700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d80159590 con 0x7f7d800b4ed0 2024-04-06T12:13:29.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.880+0000 7f7d87952700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d801592b0 con 0x7f7d800b6580 2024-04-06T12:13:29.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.880+0000 7f7d85eef700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d8015d590 0x7f7d8015e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:29.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.880+0000 7f7d856ee700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d800b4ed0 0x7f7d8014d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:29.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.880+0000 7f7d84eed700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d800b6580 0x7f7d8015e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:29.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.880+0000 7f7d85eef700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d8015d590 0x7f7d8015e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51190/0 (socket says 172.21.15.189:51190) 2024-04-06T12:13:29.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.880+0000 7f7d85eef700 1 -- 172.21.15.189:0/3016378276 learned_addr learned my addr 172.21.15.189:0/3016378276 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:29.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.880+0000 7f7d85eef700 1 -- 172.21.15.189:0/3016378276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d800b6580 msgr2=0x7f7d8015e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:29.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.880+0000 7f7d85eef700 1 --2- 172.21.15.189:0/3016378276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d800b6580 0x7f7d8015e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:29.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.880+0000 7f7d85eef700 1 -- 172.21.15.189:0/3016378276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d800b4ed0 msgr2=0x7f7d8014d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:29.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.880+0000 7f7d85eef700 1 --2- 172.21.15.189:0/3016378276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d800b4ed0 0x7f7d8014d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:29.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.880+0000 7f7d85eef700 1 -- 172.21.15.189:0/3016378276 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7c009550 con 0x7f7d8015d590 2024-04-06T12:13:29.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.880+0000 7f7d856ee700 1 --2- 172.21.15.189:0/3016378276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d800b4ed0 0x7f7d8014d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:13:29.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.881+0000 7f7d85eef700 1 --2- 172.21.15.189:0/3016378276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d8015d590 0x7f7d8015e6a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7d78001560 tx=0x7f7d7800cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:29.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.881+0000 7f7d75ffb700 1 -- 172.21.15.189:0/3016378276 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d78003e30 con 0x7f7d8015d590 2024-04-06T12:13:29.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.881+0000 7f7d75ffb700 1 -- 172.21.15.189:0/3016378276 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d78005070 con 0x7f7d8015d590 2024-04-06T12:13:29.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.881+0000 7f7d87952700 1 -- 172.21.15.189:0/3016378276 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d800cb2e0 con 0x7f7d8015d590 2024-04-06T12:13:29.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.881+0000 7f7d75ffb700 1 -- 172.21.15.189:0/3016378276 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d78005420 con 0x7f7d8015d590 2024-04-06T12:13:29.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.881+0000 7f7d87952700 1 -- 172.21.15.189:0/3016378276 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d800cb8a0 con 0x7f7d8015d590 2024-04-06T12:13:29.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.882+0000 7f7d87952700 1 -- 172.21.15.189:0/3016378276 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d800cbac0 con 0x7f7d8015d590 2024-04-06T12:13:29.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.882+0000 7f7d75ffb700 1 -- 172.21.15.189:0/3016378276 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 6) v1 ==== 89162+0+0 (secure 0 0 0) 0x7f7d78005580 con 0x7f7d8015d590 2024-04-06T12:13:29.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.883+0000 7f7d75ffb700 1 --2- 172.21.15.189:0/3016378276 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d6406ace0 0x7f7d6406d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:29.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.883+0000 7f7d856ee700 1 -- 172.21.15.189:0/3016378276 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d6406ace0 msgr2=0x7f7d6406d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:13:29.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.883+0000 7f7d75ffb700 1 -- 172.21.15.189:0/3016378276 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7d78088af0 con 0x7f7d8015d590 2024-04-06T12:13:29.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.883+0000 7f7d856ee700 1 --2- 172.21.15.189:0/3016378276 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d6406ace0 0x7f7d6406d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:13:29.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:29.887+0000 7f7d75ffb700 1 -- 172.21.15.189:0/3016378276 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7d78053b90 con 0x7f7d8015d590 2024-04-06T12:13:29.893 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:29.895+0000 7f8973863200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:13:29.923 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:29.925+0000 7ff9d7f03200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:13:29.960 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:29.962+0000 7f8973863200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:13:29.993 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:29.996+0000 7ff9d7f03200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:13:30.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.084+0000 7f7d856ee700 1 -- 172.21.15.189:0/3016378276 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d6406ace0 msgr2=0x7f7d6406d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:13:30.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.084+0000 7f7d856ee700 1 --2- 172.21.15.189:0/3016378276 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d6406ace0 0x7f7d6406d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:13:30.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.093+0000 7f7d87952700 1 -- 172.21.15.189:0/3016378276 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f7d80074990 con 0x7f7d8015d590 2024-04-06T12:13:30.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.098+0000 7f7d75ffb700 1 -- 172.21.15.189:0/3016378276 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v12)=0 v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7f7d78017070 con 0x7f7d8015d590 2024-04-06T12:13:30.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.101+0000 7f7d87952700 1 -- 172.21.15.189:0/3016378276 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d6406ace0 msgr2=0x7f7d6406d1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:13:30.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.101+0000 7f7d87952700 1 --2- 172.21.15.189:0/3016378276 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d6406ace0 0x7f7d6406d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:30.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.101+0000 7f7d87952700 1 -- 172.21.15.189:0/3016378276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d8015d590 msgr2=0x7f7d8015e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:30.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.101+0000 7f7d87952700 1 --2- 172.21.15.189:0/3016378276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d8015d590 0x7f7d8015e6a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7d78001560 tx=0x7f7d7800cb50 comp rx=0 tx=0).stop 2024-04-06T12:13:30.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.101+0000 7f7d87952700 1 -- 172.21.15.189:0/3016378276 shutdown_connections 2024-04-06T12:13:30.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.101+0000 7f7d87952700 1 --2- 172.21.15.189:0/3016378276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d8015d590 0x7f7d8015e6a0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:30.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.101+0000 7f7d87952700 1 --2- 172.21.15.189:0/3016378276 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d6406ace0 0x7f7d6406d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:30.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.101+0000 7f7d87952700 1 --2- 172.21.15.189:0/3016378276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d800b4ed0 0x7f7d8014d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:30.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.101+0000 7f7d87952700 1 --2- 172.21.15.189:0/3016378276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d800b6580 0x7f7d8015e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:30.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.101+0000 7f7d87952700 1 -- 172.21.15.189:0/3016378276 >> 172.21.15.189:0/3016378276 conn(0x7f7d8006bbc0 msgr2=0x7f7d800b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:30.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.101+0000 7f7d87952700 1 -- 172.21.15.189:0/3016378276 shutdown_connections 2024-04-06T12:13:30.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.101+0000 7f7d87952700 1 -- 172.21.15.189:0/3016378276 wait complete. 2024-04-06T12:13:30.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:30.111 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-06T12:13:30.135 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:30.137+0000 7f8973863200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:13:30.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:30.172 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2024-04-06T12:13:30.176 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:30.178+0000 7ff9d7f03200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:13:30.193 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir0 1m 2024-04-06T12:13:30.290 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:30.292+0000 7f8973863200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:13:30.343 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:30.346+0000 7ff9d7f03200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:13:30.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.366+0000 7f5c4f28b700 1 Processor -- start 2024-04-06T12:13:30.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.366+0000 7f5c4f28b700 1 -- start start 2024-04-06T12:13:30.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.367+0000 7f5c4f28b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c480b2bf0 0x7f5c480b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:30.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.367+0000 7f5c4f28b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5c480cf280 0x7f5c48144830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:30.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.367+0000 7f5c4f28b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c4815eac0 con 0x7f5c480cee00 2024-04-06T12:13:30.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.367+0000 7f5c4f28b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c48161dc0 con 0x7f5c480cf280 2024-04-06T12:13:30.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.367+0000 7f5c4f28b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c480b3510 con 0x7f5c480b2bf0 2024-04-06T12:13:30.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.367+0000 7f5c4d828700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5c480cf280 0x7f5c48144830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:30.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.367+0000 7f5c4d027700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c480b2bf0 0x7f5c480b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:30.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.367+0000 7f5c4d828700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5c480cf280 0x7f5c48144830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:37428/0 (socket says 172.21.15.189:37428) 2024-04-06T12:13:30.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.367+0000 7f5c4c826700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f5c480cee00 0x7f5c48144120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:36252/0 (socket says 172.21.15.189:36252) 2024-04-06T12:13:30.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.367+0000 7f5c4d027700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c480b2bf0 0x7f5c480b2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:52978/0 (socket says 172.21.15.189:52978) 2024-04-06T12:13:30.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.367+0000 7f5c4d828700 1 -- 172.21.15.189:0/1126409769 learned_addr learned my addr 172.21.15.189:0/1126409769 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:30.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.367+0000 7f5c4d828700 1 -- 172.21.15.189:0/1126409769 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c480b2bf0 msgr2=0x7f5c480b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:30.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.367+0000 7f5c4d828700 1 --2- 172.21.15.189:0/1126409769 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c480b2bf0 0x7f5c480b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:30.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.367+0000 7f5c4d828700 1 -- 172.21.15.189:0/1126409769 >> v1:172.21.15.192:6789/0 conn(0x7f5c480cee00 legacy=0x7f5c48144120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:30.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.367+0000 7f5c4d828700 1 -- 172.21.15.189:0/1126409769 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c480cf6f0 con 0x7f5c480cf280 2024-04-06T12:13:30.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.367+0000 7f5c3ffff700 1 -- 172.21.15.189:0/1126409769 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355682092 0 0) 0x7f5c4815eac0 con 0x7f5c480cee00 2024-04-06T12:13:30.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.367+0000 7f5c4d027700 1 --2- 172.21.15.189:0/1126409769 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c480b2bf0 0x7f5c480b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:13:30.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.367+0000 7f5c4d828700 1 --2- 172.21.15.189:0/1126409769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c480cf280 0x7f5c48144830 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5c40009e50 tx=0x7f5c4000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1973e405292f15da server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:30.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.367+0000 7f5c3ffff700 1 -- 172.21.15.189:0/1126409769 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4000e070 con 0x7f5c480cf280 2024-04-06T12:13:30.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.368+0000 7f5c3ffff700 1 -- 172.21.15.189:0/1126409769 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4000b910 con 0x7f5c480cf280 2024-04-06T12:13:30.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.368+0000 7f5c3ffff700 1 -- 172.21.15.189:0/1126409769 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4000bcc0 con 0x7f5c480cf280 2024-04-06T12:13:30.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.368+0000 7f5c4f28b700 1 -- 172.21.15.189:0/1126409769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c480cf280 msgr2=0x7f5c48144830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:30.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.368+0000 7f5c4f28b700 1 --2- 172.21.15.189:0/1126409769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c480cf280 0x7f5c48144830 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5c40009e50 tx=0x7f5c4000b330 comp rx=0 tx=0).stop 2024-04-06T12:13:30.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.368+0000 7f5c4f28b700 1 -- 172.21.15.189:0/1126409769 shutdown_connections 2024-04-06T12:13:30.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.368+0000 7f5c4f28b700 1 --2- 172.21.15.189:0/1126409769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c480cf280 0x7f5c48144830 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:30.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.368+0000 7f5c4f28b700 1 --2- 172.21.15.189:0/1126409769 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c480b2bf0 0x7f5c480b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:30.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.368+0000 7f5c4f28b700 1 -- 172.21.15.189:0/1126409769 >> 172.21.15.189:0/1126409769 conn(0x7f5c48066b80 msgr2=0x7f5c4806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:30.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.368+0000 7f5c4f28b700 1 -- 172.21.15.189:0/1126409769 shutdown_connections 2024-04-06T12:13:30.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.368+0000 7f5c4f28b700 1 -- 172.21.15.189:0/1126409769 wait complete. 2024-04-06T12:13:30.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.369+0000 7f5c4f28b700 1 Processor -- start 2024-04-06T12:13:30.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.369+0000 7f5c4f28b700 1 -- start start 2024-04-06T12:13:30.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.370+0000 7f5c4f28b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c480b2bf0 0x7f5c48156920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:30.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.370+0000 7f5c4f28b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c480cee00 0x7f5c480c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:30.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.370+0000 7f5c4f28b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c480cf280 0x7f5c480c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:30.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.370+0000 7f5c4f28b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c48156e60 con 0x7f5c480cf280 2024-04-06T12:13:30.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.370+0000 7f5c4f28b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c480c6770 con 0x7f5c480b2bf0 2024-04-06T12:13:30.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.370+0000 7f5c4f28b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c480c56d0 con 0x7f5c480cee00 2024-04-06T12:13:30.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.370+0000 7f5c4d828700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c480cf280 0x7f5c480c9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:30.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.370+0000 7f5c4d828700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c480cf280 0x7f5c480c9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51196/0 (socket says 172.21.15.189:51196) 2024-04-06T12:13:30.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.370+0000 7f5c4c826700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c480cee00 0x7f5c480c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:30.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.370+0000 7f5c4d828700 1 -- 172.21.15.189:0/1826227495 learned_addr learned my addr 172.21.15.189:0/1826227495 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:30.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.370+0000 7f5c4d027700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c480b2bf0 0x7f5c48156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:30.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.371+0000 7f5c4c826700 1 -- 172.21.15.189:0/1826227495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c480b2bf0 msgr2=0x7f5c48156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:30.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.371+0000 7f5c4c826700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c480b2bf0 0x7f5c48156920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:30.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.371+0000 7f5c4c826700 1 -- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c480cf280 msgr2=0x7f5c480c9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:30.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.371+0000 7f5c4c826700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c480cf280 0x7f5c480c9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:30.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.371+0000 7f5c4c826700 1 -- 172.21.15.189:0/1826227495 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c440094a0 con 0x7f5c480cee00 2024-04-06T12:13:30.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.371+0000 7f5c4d828700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c480cf280 0x7f5c480c9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:13:30.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.371+0000 7f5c4c826700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c480cee00 0x7f5c480c4c80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5c3800c4b0 tx=0x7f5c3800c7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:30.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.371+0000 7f5c3dffb700 1 -- 172.21.15.189:0/1826227495 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c38004c80 con 0x7f5c480cee00 2024-04-06T12:13:30.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.371+0000 7f5c3dffb700 1 -- 172.21.15.189:0/1826227495 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c38004e00 con 0x7f5c480cee00 2024-04-06T12:13:30.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.371+0000 7f5c4f28b700 1 -- 172.21.15.189:0/1826227495 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c40009b30 con 0x7f5c480cee00 2024-04-06T12:13:30.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.371+0000 7f5c3dffb700 1 -- 172.21.15.189:0/1826227495 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c38007310 con 0x7f5c480cee00 2024-04-06T12:13:30.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.371+0000 7f5c4f28b700 1 -- 172.21.15.189:0/1826227495 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c480c5c90 con 0x7f5c480cee00 2024-04-06T12:13:30.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.372+0000 7f5c4f28b700 1 -- 172.21.15.189:0/1826227495 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c480748f0 con 0x7f5c480cee00 2024-04-06T12:13:30.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.372+0000 7f5c3dffb700 1 -- 172.21.15.189:0/1826227495 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 6) v1 ==== 89162+0+0 (secure 0 0 0) 0x7f5c38007470 con 0x7f5c480cee00 2024-04-06T12:13:30.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.373+0000 7f5c3dffb700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2c06ace0 0x7f5c2c06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:30.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.373+0000 7f5c4d027700 1 -- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2c06ace0 msgr2=0x7f5c2c06d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:13:30.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.373+0000 7f5c3dffb700 1 -- 172.21.15.189:0/1826227495 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5c380890e0 con 0x7f5c480cee00 2024-04-06T12:13:30.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.373+0000 7f5c4d027700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2c06ace0 0x7f5c2c06d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:13:30.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.377+0000 7f5c3dffb700 1 -- 172.21.15.189:0/1826227495 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5c38054240 con 0x7f5c480cee00 2024-04-06T12:13:30.401 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:30.403+0000 7f8973863200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:13:30.460 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:30.461+0000 7ff9d7f03200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:13:30.482 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:30.484+0000 7f8973863200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:13:30.543 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:30.546+0000 7ff9d7f03200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:13:30.556 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:30.558+0000 7f8973863200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:13:30.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.574+0000 7f5c4d027700 1 -- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2c06ace0 msgr2=0x7f5c2c06d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:13:30.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.574+0000 7f5c4d027700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2c06ace0 0x7f5c2c06d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:13:30.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.606+0000 7f5c4f28b700 1 -- 172.21.15.189:0/1826227495 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c48161700 con 0x7f5c2c06ace0 2024-04-06T12:13:30.621 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:30.623+0000 7ff9d7f03200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:13:30.758 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:30.760+0000 7f8973863200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:13:30.825 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:30.827+0000 7ff9d7f03200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:13:30.840 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:30.836+0000 7f8973863200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:13:30.900 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:30.902+0000 7ff9d7f03200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:13:30.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.975+0000 7f5c4d027700 1 -- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2c06ace0 msgr2=0x7f5c2c06d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:13:30.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:30.975+0000 7f5c4d027700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2c06ace0 0x7f5c2c06d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-06T12:13:31.152 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:31.154+0000 7f8973863200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:13:31.221 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:31.223+0000 7ff9d7f03200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:13:31.470 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:31.472+0000 7f8973863200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:13:31.543 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:31.545+0000 7ff9d7f03200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:13:31.628 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:31.630+0000 7f8973863200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:13:31.700 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:31.702+0000 7f8973863200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:13:31.701 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:31.704+0000 7ff9d7f03200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:13:31.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:31.776+0000 7f5c4d027700 1 -- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2c06ace0 msgr2=0x7f5c2c06d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:13:31.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:31.776+0000 7f5c4d027700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2c06ace0 0x7f5c2c06d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-06T12:13:31.774 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:31.776+0000 7ff9d7f03200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:13:32.313 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:32.315+0000 7f8973863200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:13:32.390 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:32.392+0000 7f8973863200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:13:32.395 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:32.397+0000 7ff9d7f03200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:13:32.473 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:32.475+0000 7ff9d7f03200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:13:32.514 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:32.516+0000 7f8973863200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:13:32.598 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:32.600+0000 7ff9d7f03200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:13:33.022 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:33.024+0000 7f8973863200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:13:33.109 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:33.111+0000 7ff9d7f03200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:13:33.172 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:33.174+0000 7f8973863200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:13:33.261 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:33.263+0000 7ff9d7f03200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:13:33.325 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:33.327+0000 7f8973863200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:13:33.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:33.377+0000 7f5c4d027700 1 -- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2c06ace0 msgr2=0x7f5c2c06d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:13:33.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:33.377+0000 7f5c4d027700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2c06ace0 0x7f5c2c06d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-06T12:13:33.414 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:33.416+0000 7ff9d7f03200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:13:34.042 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:34.044+0000 7f8973863200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:13:34.135 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:34.137+0000 7ff9d7f03200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:13:34.276 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:34.278+0000 7f8973863200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:13:34.350 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:34.352+0000 7f8973863200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:13:34.370 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:34.372+0000 7ff9d7f03200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:13:34.445 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:34.447+0000 7ff9d7f03200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:13:34.723 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:13:34.725+0000 7f8973863200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:13:34.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:34.796+0000 7f5c3dffb700 1 -- 172.21.15.189:0/1826227495 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mgrmap(e 7) v1 ==== 89162+0+0 (secure 0 0 0) 0x7f5c38054610 con 0x7f5c480cee00 2024-04-06T12:13:34.821 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:34.823+0000 7ff9d7f03200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:13:34.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:34.847+0000 7f5c3dffb700 1 -- 172.21.15.189:0/1826227495 <== mon.2 v2:172.21.15.192:3301/0 8 ==== mgrmap(e 8) v1 ==== 88873+0+0 (secure 0 0 0) 0x7f5c3805fe60 con 0x7f5c480cee00 2024-04-06T12:13:34.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:34.848+0000 7f5c3dffb700 1 -- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2c06ace0 msgr2=0x7f5c2c06d1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:13:34.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:34.848+0000 7f5c3dffb700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2c06ace0 0x7f5c2c06d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:35.023 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:35.025+0000 7ff94ae11700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:13:35.023 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:35.025+0000 7ff94ae11700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:13:35.024 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:35.025+0000 7ff94ae11700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:13:35.024 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:35.025+0000 7ff94ae11700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:13:35.024 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:35.025+0000 7ff94ae11700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:13:35.024 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:35.025+0000 7ff94de17700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:13:35.024 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:35.025+0000 7ff94de17700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:13:35.024 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:35.025+0000 7ff94de17700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:13:35.024 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:35.025+0000 7ff94de17700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:13:35.024 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:13:35.025+0000 7ff94de17700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:13:35.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:35.853+0000 7f5c3dffb700 1 -- 172.21.15.189:0/1826227495 <== mon.2 v2:172.21.15.192:3301/0 9 ==== mgrmap(e 9) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f5c38054ee0 con 0x7f5c480cee00 2024-04-06T12:13:35.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:35.853+0000 7f5c3dffb700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2c06ace0 0x7f5c2c06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:35.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:35.853+0000 7f5c3dffb700 1 -- 172.21.15.189:0/1826227495 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c2c027950 con 0x7f5c2c06ace0 2024-04-06T12:13:35.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:35.854+0000 7f5c4d027700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2c06ace0 0x7f5c2c06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:35.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:35.856+0000 7f5c4d027700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2c06ace0 0x7f5c2c06d1a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5c480b4220 tx=0x7f5c44009210 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:35.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:35.944+0000 7f5c3dffb700 1 -- 172.21.15.189:0/1826227495 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f5c2c027950 con 0x7f5c2c06ace0 2024-04-06T12:13:35.943 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path /snap_test_dir1/dir0 2024-04-06T12:13:35.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:35.947+0000 7f5c4f28b700 1 -- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2c06ace0 msgr2=0x7f5c2c06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:35.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:35.947+0000 7f5c4f28b700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2c06ace0 0x7f5c2c06d1a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5c480b4220 tx=0x7f5c44009210 comp rx=0 tx=0).stop 2024-04-06T12:13:35.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:35.947+0000 7f5c4f28b700 1 -- 172.21.15.189:0/1826227495 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c480cee00 msgr2=0x7f5c480c4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:35.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:35.947+0000 7f5c4f28b700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c480cee00 0x7f5c480c4c80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5c3800c4b0 tx=0x7f5c3800c7c0 comp rx=0 tx=0).stop 2024-04-06T12:13:35.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:35.947+0000 7f5c4f28b700 1 -- 172.21.15.189:0/1826227495 shutdown_connections 2024-04-06T12:13:35.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:35.947+0000 7f5c4f28b700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c480cf280 0x7f5c480c9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:35.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:35.947+0000 7f5c4f28b700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2c06ace0 0x7f5c2c06d1a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:35.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:35.947+0000 7f5c4f28b700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c480b2bf0 0x7f5c48156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:35.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:35.947+0000 7f5c4f28b700 1 --2- 172.21.15.189:0/1826227495 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c480cee00 0x7f5c480c4c80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:35.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:35.947+0000 7f5c4f28b700 1 -- 172.21.15.189:0/1826227495 >> 172.21.15.189:0/1826227495 conn(0x7f5c48066b80 msgr2=0x7f5c480b0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:35.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:35.947+0000 7f5c4f28b700 1 -- 172.21.15.189:0/1826227495 shutdown_connections 2024-04-06T12:13:35.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:35.947+0000 7f5c4f28b700 1 -- 172.21.15.189:0/1826227495 wait complete. 2024-04-06T12:13:35.958 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2024-04-06T12:13:35.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:35.958 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2024-04-06T12:13:35.977 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir1 1m 2024-04-06T12:13:36.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.145+0000 7f50b62bd700 1 Processor -- start 2024-04-06T12:13:36.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.145+0000 7f50b62bd700 1 -- start start 2024-04-06T12:13:36.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.146+0000 7f50b62bd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f50b006ae40 0x7f50b00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:36.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.146+0000 7f50b62bd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50b015ddf0 0x7f50b015e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:36.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.146+0000 7f50b62bd700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b00b3990 con 0x7f50b015d7b0 2024-04-06T12:13:36.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.146+0000 7f50b62bd700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b00b3060 con 0x7f50b006ae40 2024-04-06T12:13:36.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.146+0000 7f50b62bd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b00b3340 con 0x7f50b015ddf0 2024-04-06T12:13:36.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.146+0000 7f50af7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f50b006ae40 0x7f50b00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:36.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.146+0000 7f50af7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f50b006ae40 0x7f50b00b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51320/0 (socket says 172.21.15.189:51320) 2024-04-06T12:13:36.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.146+0000 7f50af7fe700 1 -- 172.21.15.189:0/2219372860 learned_addr learned my addr 172.21.15.189:0/2219372860 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:36.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.146+0000 7f50b485a700 1 --2- 172.21.15.189:0/2219372860 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50b015ddf0 0x7f50b015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:36.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.146+0000 7f50af7fe700 1 -- 172.21.15.189:0/2219372860 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50b015ddf0 msgr2=0x7f50b015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:36.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.146+0000 7f50af7fe700 1 --2- 172.21.15.189:0/2219372860 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50b015ddf0 0x7f50b015e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.146+0000 7f50af7fe700 1 -- 172.21.15.189:0/2219372860 >> v1:172.21.15.192:6789/0 conn(0x7f50b015d7b0 legacy=0x7f50b006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:36.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.146+0000 7f50af7fe700 1 -- 172.21.15.189:0/2219372860 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b00b3620 con 0x7f50b006ae40 2024-04-06T12:13:36.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.146+0000 7f50af7fe700 1 --2- 172.21.15.189:0/2219372860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50b006ae40 0x7f50b00b3e20 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f50a00098f0 tx=0x7f50a0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=486d388bd3830bb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:36.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.146+0000 7f50aeffd700 1 -- 172.21.15.189:0/2219372860 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a000b8a0 con 0x7f50b006ae40 2024-04-06T12:13:36.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.147+0000 7f50aeffd700 1 -- 172.21.15.189:0/2219372860 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a000ba00 con 0x7f50b006ae40 2024-04-06T12:13:36.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.147+0000 7f50aeffd700 1 -- 172.21.15.189:0/2219372860 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a000bd90 con 0x7f50b006ae40 2024-04-06T12:13:36.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.147+0000 7f50b62bd700 1 -- 172.21.15.189:0/2219372860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50b006ae40 msgr2=0x7f50b00b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:36.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.147+0000 7f50b62bd700 1 --2- 172.21.15.189:0/2219372860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50b006ae40 0x7f50b00b3e20 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f50a00098f0 tx=0x7f50a0009c00 comp rx=0 tx=0).stop 2024-04-06T12:13:36.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.147+0000 7f50b62bd700 1 -- 172.21.15.189:0/2219372860 shutdown_connections 2024-04-06T12:13:36.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.147+0000 7f50b62bd700 1 --2- 172.21.15.189:0/2219372860 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50b015ddf0 0x7f50b015e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.147+0000 7f50b62bd700 1 --2- 172.21.15.189:0/2219372860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50b006ae40 0x7f50b00b3e20 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.147+0000 7f50b62bd700 1 -- 172.21.15.189:0/2219372860 >> 172.21.15.189:0/2219372860 conn(0x7f50b00d00b0 msgr2=0x7f50b00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:36.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.147+0000 7f50b62bd700 1 -- 172.21.15.189:0/2219372860 shutdown_connections 2024-04-06T12:13:36.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.147+0000 7f50b62bd700 1 -- 172.21.15.189:0/2219372860 wait complete. 2024-04-06T12:13:36.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.148+0000 7f50b62bd700 1 Processor -- start 2024-04-06T12:13:36.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.149+0000 7f50b62bd700 1 -- start start 2024-04-06T12:13:36.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.149+0000 7f50b62bd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50b006ae40 0x7f50b01f28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:36.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.149+0000 7f50b62bd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50b015d7b0 0x7f50b015a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:36.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.149+0000 7f50b62bd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b015ddf0 0x7f50b00cebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:36.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.149+0000 7f50b62bd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b00ce7d0 con 0x7f50b006ae40 2024-04-06T12:13:36.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.149+0000 7f50b62bd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b0148780 con 0x7f50b015ddf0 2024-04-06T12:13:36.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.149+0000 7f50b62bd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b01425d0 con 0x7f50b015d7b0 2024-04-06T12:13:36.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.149+0000 7f50affff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50b006ae40 0x7f50b01f28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:36.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.149+0000 7f50affff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50b006ae40 0x7f50b01f28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51334/0 (socket says 172.21.15.189:51334) 2024-04-06T12:13:36.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.149+0000 7f50affff700 1 -- 172.21.15.189:0/3472765696 learned_addr learned my addr 172.21.15.189:0/3472765696 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:36.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.149+0000 7f50af7fe700 1 --2- 172.21.15.189:0/3472765696 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50b015d7b0 0x7f50b015a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:36.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.149+0000 7f50b485a700 1 --2- 172.21.15.189:0/3472765696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b015ddf0 0x7f50b00cebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:36.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.150+0000 7f50affff700 1 -- 172.21.15.189:0/3472765696 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50b015d7b0 msgr2=0x7f50b015a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:36.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.150+0000 7f50affff700 1 --2- 172.21.15.189:0/3472765696 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50b015d7b0 0x7f50b015a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.150+0000 7f50affff700 1 -- 172.21.15.189:0/3472765696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b015ddf0 msgr2=0x7f50b00cebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:36.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.150+0000 7f50affff700 1 --2- 172.21.15.189:0/3472765696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b015ddf0 0x7f50b00cebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.150+0000 7f50affff700 1 -- 172.21.15.189:0/3472765696 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a0009550 con 0x7f50b006ae40 2024-04-06T12:13:36.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.150+0000 7f50affff700 1 --2- 172.21.15.189:0/3472765696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50b006ae40 0x7f50b01f28f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f50a400c4e0 tx=0x7f50a400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:36.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.150+0000 7f50acff9700 1 -- 172.21.15.189:0/3472765696 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a40052d0 con 0x7f50b006ae40 2024-04-06T12:13:36.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.150+0000 7f50acff9700 1 -- 172.21.15.189:0/3472765696 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a4005430 con 0x7f50b006ae40 2024-04-06T12:13:36.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.150+0000 7f50acff9700 1 -- 172.21.15.189:0/3472765696 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a40038c0 con 0x7f50b006ae40 2024-04-06T12:13:36.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.150+0000 7f50b62bd700 1 -- 172.21.15.189:0/3472765696 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b01573b0 con 0x7f50b006ae40 2024-04-06T12:13:36.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.150+0000 7f50b62bd700 1 -- 172.21.15.189:0/3472765696 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b0156df0 con 0x7f50b006ae40 2024-04-06T12:13:36.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.151+0000 7f50b62bd700 1 -- 172.21.15.189:0/3472765696 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b0067bb0 con 0x7f50b006ae40 2024-04-06T12:13:36.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.151+0000 7f50acff9700 1 -- 172.21.15.189:0/3472765696 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 9) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f50a4003a20 con 0x7f50b006ae40 2024-04-06T12:13:36.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.152+0000 7f50acff9700 1 --2- 172.21.15.189:0/3472765696 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f509c06ab30 0x7f509c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:36.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.152+0000 7f50acff9700 1 -- 172.21.15.189:0/3472765696 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f50a4088580 con 0x7f50b006ae40 2024-04-06T12:13:36.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.152+0000 7f50af7fe700 1 --2- 172.21.15.189:0/3472765696 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f509c06ab30 0x7f509c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:36.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.152+0000 7f50af7fe700 1 --2- 172.21.15.189:0/3472765696 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f509c06ab30 0x7f509c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f50a00098f0 tx=0x7f50a0008030 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:36.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.156+0000 7f50acff9700 1 -- 172.21.15.189:0/3472765696 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f50a4053860 con 0x7f50b006ae40 2024-04-06T12:13:36.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.377+0000 7f50b62bd700 1 -- 172.21.15.189:0/3472765696 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f50b015e7c0 con 0x7f509c06ab30 2024-04-06T12:13:36.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.452+0000 7f50acff9700 1 -- 172.21.15.189:0/3472765696 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f50a000bb70 con 0x7f509c06ab30 2024-04-06T12:13:36.450 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path /snap_test_dir1/dir1 2024-04-06T12:13:36.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.454+0000 7f50b62bd700 1 -- 172.21.15.189:0/3472765696 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f509c06ab30 msgr2=0x7f509c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:36.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.454+0000 7f50b62bd700 1 --2- 172.21.15.189:0/3472765696 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f509c06ab30 0x7f509c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f50a00098f0 tx=0x7f50a0008030 comp rx=0 tx=0).stop 2024-04-06T12:13:36.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.454+0000 7f50b62bd700 1 -- 172.21.15.189:0/3472765696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50b006ae40 msgr2=0x7f50b01f28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:36.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.454+0000 7f50b62bd700 1 --2- 172.21.15.189:0/3472765696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50b006ae40 0x7f50b01f28f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f50a400c4e0 tx=0x7f50a400c7f0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.454+0000 7f50b62bd700 1 -- 172.21.15.189:0/3472765696 shutdown_connections 2024-04-06T12:13:36.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.454+0000 7f50b62bd700 1 --2- 172.21.15.189:0/3472765696 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f509c06ab30 0x7f509c06cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.454+0000 7f50b62bd700 1 --2- 172.21.15.189:0/3472765696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b015ddf0 0x7f50b00cebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.454+0000 7f50b62bd700 1 --2- 172.21.15.189:0/3472765696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50b006ae40 0x7f50b01f28f0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.454+0000 7f50b62bd700 1 --2- 172.21.15.189:0/3472765696 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50b015d7b0 0x7f50b015a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.454+0000 7f50b62bd700 1 -- 172.21.15.189:0/3472765696 >> 172.21.15.189:0/3472765696 conn(0x7f50b00d00b0 msgr2=0x7f50b00ae0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:36.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.455+0000 7f50b62bd700 1 -- 172.21.15.189:0/3472765696 shutdown_connections 2024-04-06T12:13:36.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.455+0000 7f50b62bd700 1 -- 172.21.15.189:0/3472765696 wait complete. 2024-04-06T12:13:36.465 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2024-04-06T12:13:36.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:36.465 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2024-04-06T12:13:36.484 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir2 1m 2024-04-06T12:13:36.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.650+0000 7f49d2d4f700 1 Processor -- start 2024-04-06T12:13:36.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.650+0000 7f49d2d4f700 1 -- start start 2024-04-06T12:13:36.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.651+0000 7f49d2d4f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f49cc15d7b0 0x7f49cc0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:36.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.651+0000 7f49d2d4f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49cc15e500 0x7f49cc06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:36.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.651+0000 7f49d2d4f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49cc0b3990 con 0x7f49cc06a730 2024-04-06T12:13:36.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.651+0000 7f49d2d4f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49cc06b0f0 con 0x7f49cc15d7b0 2024-04-06T12:13:36.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.651+0000 7f49d2d4f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc0b3060 con 0x7f49cc15e500 2024-04-06T12:13:36.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.651+0000 7f49cbfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f49cc06a730 0x7f49cc15ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39762/0 (socket says 172.21.15.189:39762) 2024-04-06T12:13:36.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.651+0000 7f49cbfff700 1 -- 172.21.15.189:0/1475814846 learned_addr learned my addr 172.21.15.189:0/1475814846 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:36.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49d12ec700 1 --2- 172.21.15.189:0/1475814846 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49cc15e500 0x7f49cc06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:36.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49cb7fe700 1 -- 172.21.15.189:0/1475814846 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257294077 0 0) 0x7f49cc0b3990 con 0x7f49cc06a730 2024-04-06T12:13:36.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49d0aeb700 1 --2- 172.21.15.189:0/1475814846 >> v2:172.21.15.192:3300/0 conn(0x7f49cc15d7b0 0x7f49cc0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:36.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49cb7fe700 1 -- 172.21.15.189:0/1475814846 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49bc0035b0 con 0x7f49cc06a730 2024-04-06T12:13:36.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49cb7fe700 1 -- 172.21.15.189:0/1475814846 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 701085073 0 0) 0x7f49bc0035b0 con 0x7f49cc06a730 2024-04-06T12:13:36.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49cb7fe700 1 -- 172.21.15.189:0/1475814846 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49cc15e500 msgr2=0x7f49cc06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:36.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49cb7fe700 1 --2- 172.21.15.189:0/1475814846 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49cc15e500 0x7f49cc06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49cb7fe700 1 -- 172.21.15.189:0/1475814846 >> v2:172.21.15.192:3300/0 conn(0x7f49cc15d7b0 msgr2=0x7f49cc0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:36.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49cb7fe700 1 --2- 172.21.15.189:0/1475814846 >> v2:172.21.15.192:3300/0 conn(0x7f49cc15d7b0 0x7f49cc0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49cb7fe700 1 -- 172.21.15.189:0/1475814846 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49cc0b3340 con 0x7f49cc06a730 2024-04-06T12:13:36.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49cb7fe700 1 -- 172.21.15.189:0/1475814846 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f49b8001670 con 0x7f49cc06a730 2024-04-06T12:13:36.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49cb7fe700 1 -- 172.21.15.189:0/1475814846 >> v1:172.21.15.189:6789/0 conn(0x7f49cc06a730 legacy=0x7f49cc15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:36.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49cb7fe700 1 --2- 172.21.15.189:0/1475814846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49bc003a00 0x7f49bc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:36.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49cb7fe700 1 --2- 172.21.15.189:0/1475814846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49bc004850 0x7f49bc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:36.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49cb7fe700 1 --2- 172.21.15.189:0/1475814846 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49cc15e500 0x7f49bc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:36.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49cb7fe700 1 -- 172.21.15.189:0/1475814846 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc06b0f0 con 0x7f49bc003a00 2024-04-06T12:13:36.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49cb7fe700 1 -- 172.21.15.189:0/1475814846 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc0b3060 con 0x7f49bc004850 2024-04-06T12:13:36.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49cb7fe700 1 -- 172.21.15.189:0/1475814846 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc00ba10 con 0x7f49cc15e500 2024-04-06T12:13:36.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49d0aeb700 1 --2- 172.21.15.189:0/1475814846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49bc003a00 0x7f49bc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:36.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49cbfff700 1 --2- 172.21.15.189:0/1475814846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49bc004850 0x7f49bc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:36.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49d12ec700 1 --2- 172.21.15.189:0/1475814846 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49cc15e500 0x7f49bc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:36.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49d0aeb700 1 -- 172.21.15.189:0/1475814846 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49cc15e500 msgr2=0x7f49bc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:36.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49d0aeb700 1 --2- 172.21.15.189:0/1475814846 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49cc15e500 0x7f49bc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49d0aeb700 1 -- 172.21.15.189:0/1475814846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49bc004850 msgr2=0x7f49bc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:36.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49d0aeb700 1 --2- 172.21.15.189:0/1475814846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49bc004850 0x7f49bc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.652+0000 7f49d0aeb700 1 -- 172.21.15.189:0/1475814846 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49bc0035b0 con 0x7f49bc003a00 2024-04-06T12:13:36.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.653+0000 7f49d0aeb700 1 --2- 172.21.15.189:0/1475814846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49bc003a00 0x7f49bc003e20 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009b90 tx=0x7f49bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=354a9b7a1521193a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:36.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.653+0000 7f49cb7fe700 1 -- 172.21.15.189:0/1475814846 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c000e6c0 con 0x7f49bc003a00 2024-04-06T12:13:36.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.653+0000 7f49cb7fe700 1 -- 172.21.15.189:0/1475814846 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c000e820 con 0x7f49bc003a00 2024-04-06T12:13:36.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.653+0000 7f49cb7fe700 1 -- 172.21.15.189:0/1475814846 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c000eb90 con 0x7f49bc003a00 2024-04-06T12:13:36.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.653+0000 7f49d2d4f700 1 -- 172.21.15.189:0/1475814846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49bc003a00 msgr2=0x7f49bc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:36.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.653+0000 7f49d2d4f700 1 --2- 172.21.15.189:0/1475814846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49bc003a00 0x7f49bc003e20 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009b90 tx=0x7f49bc005880 comp rx=0 tx=0).stop 2024-04-06T12:13:36.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.653+0000 7f49d12ec700 1 -- 172.21.15.189:0/1475814846 reap_dead start 2024-04-06T12:13:36.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.653+0000 7f49d2d4f700 1 -- 172.21.15.189:0/1475814846 shutdown_connections 2024-04-06T12:13:36.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.653+0000 7f49d2d4f700 1 -- 172.21.15.189:0/1475814846 >> 172.21.15.189:0/1475814846 conn(0x7f49cc0d00b0 msgr2=0x7f49cc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:36.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.653+0000 7f49d2d4f700 1 -- 172.21.15.189:0/1475814846 shutdown_connections 2024-04-06T12:13:36.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.653+0000 7f49d2d4f700 1 -- 172.21.15.189:0/1475814846 wait complete. 2024-04-06T12:13:36.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.655+0000 7f49d2d4f700 1 Processor -- start 2024-04-06T12:13:36.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.655+0000 7f49d2d4f700 1 -- start start 2024-04-06T12:13:36.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.655+0000 7f49d2d4f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49bc003a00 0x7f49cc0cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:36.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.655+0000 7f49d2d4f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49cc1f2760 0x7f49cc145150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:36.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.655+0000 7f49d2d4f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49cc06f690 0x7f49cc06fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:36.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.656+0000 7f49d2d4f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc070050 con 0x7f49cc06f690 2024-04-06T12:13:36.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.656+0000 7f49d2d4f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc145690 con 0x7f49bc003a00 2024-04-06T12:13:36.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.656+0000 7f49d2d4f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc0ce3f0 con 0x7f49cc1f2760 2024-04-06T12:13:36.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.656+0000 7f49d12ec700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49cc06f690 0x7f49cc06fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:36.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.656+0000 7f49d12ec700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49cc06f690 0x7f49cc06fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51358/0 (socket says 172.21.15.189:51358) 2024-04-06T12:13:36.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.656+0000 7f49d12ec700 1 -- 172.21.15.189:0/3652510417 learned_addr learned my addr 172.21.15.189:0/3652510417 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:36.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.656+0000 7f49d0aeb700 1 --2- 172.21.15.189:0/3652510417 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49bc003a00 0x7f49cc0cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:36.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.656+0000 7f49cbfff700 1 --2- 172.21.15.189:0/3652510417 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49cc1f2760 0x7f49cc145150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:36.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.656+0000 7f49d12ec700 1 -- 172.21.15.189:0/3652510417 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49cc1f2760 msgr2=0x7f49cc145150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:36.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.656+0000 7f49d12ec700 1 --2- 172.21.15.189:0/3652510417 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49cc1f2760 0x7f49cc145150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.656+0000 7f49d12ec700 1 -- 172.21.15.189:0/3652510417 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49bc003a00 msgr2=0x7f49cc0cdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:36.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.656+0000 7f49d12ec700 1 --2- 172.21.15.189:0/3652510417 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49bc003a00 0x7f49cc0cdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.656+0000 7f49d12ec700 1 -- 172.21.15.189:0/3652510417 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49cc0ce6d0 con 0x7f49cc06f690 2024-04-06T12:13:36.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.656+0000 7f49d12ec700 1 --2- 172.21.15.189:0/3652510417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49cc06f690 0x7f49cc06fb10 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f49cc06c7f0 tx=0x7f49c4011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:36.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.656+0000 7f49c97fa700 1 -- 172.21.15.189:0/3652510417 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c400e040 con 0x7f49cc06f690 2024-04-06T12:13:36.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.656+0000 7f49c97fa700 1 -- 172.21.15.189:0/3652510417 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c400f040 con 0x7f49cc06f690 2024-04-06T12:13:36.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.656+0000 7f49c97fa700 1 -- 172.21.15.189:0/3652510417 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c400fe40 con 0x7f49cc06f690 2024-04-06T12:13:36.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.656+0000 7f49d2d4f700 1 -- 172.21.15.189:0/3652510417 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49cc0ce9b0 con 0x7f49cc06f690 2024-04-06T12:13:36.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.656+0000 7f49d2d4f700 1 -- 172.21.15.189:0/3652510417 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49cc0cef70 con 0x7f49cc06f690 2024-04-06T12:13:36.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.657+0000 7f49d2d4f700 1 -- 172.21.15.189:0/3652510417 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49cc0cf190 con 0x7f49cc06f690 2024-04-06T12:13:36.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.658+0000 7f49c97fa700 1 -- 172.21.15.189:0/3652510417 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 9) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f49c4015430 con 0x7f49cc06f690 2024-04-06T12:13:36.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.658+0000 7f49c97fa700 1 --2- 172.21.15.189:0/3652510417 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f49b006ab30 0x7f49b006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:36.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.658+0000 7f49c97fa700 1 -- 172.21.15.189:0/3652510417 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f49c4089680 con 0x7f49cc06f690 2024-04-06T12:13:36.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.658+0000 7f49d0aeb700 1 --2- 172.21.15.189:0/3652510417 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f49b006ab30 0x7f49b006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:36.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.659+0000 7f49d0aeb700 1 --2- 172.21.15.189:0/3652510417 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f49b006ab30 0x7f49b006cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f49c00090b0 tx=0x7f49c0008030 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:36.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.663+0000 7f49c97fa700 1 -- 172.21.15.189:0/3652510417 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f49c4054960 con 0x7f49cc06f690 2024-04-06T12:13:36.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.887+0000 7f49d2d4f700 1 -- 172.21.15.189:0/3652510417 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir2", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f49cc15eb40 con 0x7f49b006ab30 2024-04-06T12:13:36.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.950+0000 7f49c97fa700 1 -- 172.21.15.189:0/3652510417 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f49c0022040 con 0x7f49b006ab30 2024-04-06T12:13:36.949 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path /snap_test_dir1/dir2 2024-04-06T12:13:36.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.953+0000 7f49d2d4f700 1 -- 172.21.15.189:0/3652510417 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f49b006ab30 msgr2=0x7f49b006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:36.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.953+0000 7f49d2d4f700 1 --2- 172.21.15.189:0/3652510417 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f49b006ab30 0x7f49b006cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f49c00090b0 tx=0x7f49c0008030 comp rx=0 tx=0).stop 2024-04-06T12:13:36.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.953+0000 7f49d2d4f700 1 -- 172.21.15.189:0/3652510417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49cc06f690 msgr2=0x7f49cc06fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:36.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.953+0000 7f49d2d4f700 1 --2- 172.21.15.189:0/3652510417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49cc06f690 0x7f49cc06fb10 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f49cc06c7f0 tx=0x7f49c4011f60 comp rx=0 tx=0).stop 2024-04-06T12:13:36.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.953+0000 7f49d2d4f700 1 -- 172.21.15.189:0/3652510417 shutdown_connections 2024-04-06T12:13:36.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.953+0000 7f49d2d4f700 1 --2- 172.21.15.189:0/3652510417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49cc06f690 0x7f49cc06fb10 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.953+0000 7f49d2d4f700 1 --2- 172.21.15.189:0/3652510417 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f49b006ab30 0x7f49b006cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.953+0000 7f49d2d4f700 1 --2- 172.21.15.189:0/3652510417 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49bc003a00 0x7f49cc0cdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.953+0000 7f49d2d4f700 1 --2- 172.21.15.189:0/3652510417 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49cc1f2760 0x7f49cc145150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:36.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.953+0000 7f49d2d4f700 1 -- 172.21.15.189:0/3652510417 >> 172.21.15.189:0/3652510417 conn(0x7f49cc0d00b0 msgr2=0x7f49cc072040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:36.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.953+0000 7f49d2d4f700 1 -- 172.21.15.189:0/3652510417 shutdown_connections 2024-04-06T12:13:36.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:36.953+0000 7f49d2d4f700 1 -- 172.21.15.189:0/3652510417 wait complete. 2024-04-06T12:13:36.964 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2024-04-06T12:13:36.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:36.964 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2024-04-06T12:13:36.983 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir3 1m 2024-04-06T12:13:37.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.150+0000 7fdc4b1f8700 1 Processor -- start 2024-04-06T12:13:37.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.150+0000 7fdc4b1f8700 1 -- start start 2024-04-06T12:13:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.151+0000 7fdc4b1f8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdc440b3e20 0x7fdc4415e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.151+0000 7fdc4b1f8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc440b3990 con 0x7fdc4406ae40 2024-04-06T12:13:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.151+0000 7fdc4b1f8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc440b3060 con 0x7fdc4415d7b0 2024-04-06T12:13:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.151+0000 7fdc4b1f8700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc440b4230 con 0x7fdc440b3e20 2024-04-06T12:13:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.151+0000 7fdc49795700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdc440b3e20 0x7fdc4415e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.151+0000 7fdc49795700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdc440b3e20 0x7fdc4415e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51382/0 (socket says 172.21.15.189:51382) 2024-04-06T12:13:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.151+0000 7fdc49795700 1 -- 172.21.15.189:0/1734604100 learned_addr learned my addr 172.21.15.189:0/1734604100 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.151+0000 7fdc43fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdc4406ae40 0x7fdc4415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39770/0 (socket says 172.21.15.189:39770) 2024-04-06T12:13:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.152+0000 7fdc49795700 1 -- 172.21.15.189:0/1734604100 >> v1:172.21.15.192:6789/0 conn(0x7fdc4415d7b0 legacy=0x7fdc4406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.152+0000 7fdc49795700 1 -- 172.21.15.189:0/1734604100 >> v1:172.21.15.189:6789/0 conn(0x7fdc4406ae40 legacy=0x7fdc4415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.152+0000 7fdc49795700 1 -- 172.21.15.189:0/1734604100 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc440b3340 con 0x7fdc440b3e20 2024-04-06T12:13:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.152+0000 7fdc49795700 1 --2- 172.21.15.189:0/1734604100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc440b3e20 0x7fdc4415e500 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c009e50 tx=0x7fdc3c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23eedebb1f06bb36 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.152+0000 7fdc437fe700 1 -- 172.21.15.189:0/1734604100 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c00e070 con 0x7fdc440b3e20 2024-04-06T12:13:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.152+0000 7fdc437fe700 1 -- 172.21.15.189:0/1734604100 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3c003a20 con 0x7fdc440b3e20 2024-04-06T12:13:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.152+0000 7fdc437fe700 1 -- 172.21.15.189:0/1734604100 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c003dd0 con 0x7fdc440b3e20 2024-04-06T12:13:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.153+0000 7fdc4b1f8700 1 -- 172.21.15.189:0/1734604100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc440b3e20 msgr2=0x7fdc4415e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.153+0000 7fdc4b1f8700 1 --2- 172.21.15.189:0/1734604100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc440b3e20 0x7fdc4415e500 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c009e50 tx=0x7fdc3c00b330 comp rx=0 tx=0).stop 2024-04-06T12:13:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.153+0000 7fdc4b1f8700 1 -- 172.21.15.189:0/1734604100 shutdown_connections 2024-04-06T12:13:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.153+0000 7fdc4b1f8700 1 --2- 172.21.15.189:0/1734604100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc440b3e20 0x7fdc4415e500 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.153+0000 7fdc4b1f8700 1 -- 172.21.15.189:0/1734604100 >> 172.21.15.189:0/1734604100 conn(0x7fdc440d00b0 msgr2=0x7fdc440d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.153+0000 7fdc4b1f8700 1 -- 172.21.15.189:0/1734604100 shutdown_connections 2024-04-06T12:13:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.153+0000 7fdc4b1f8700 1 -- 172.21.15.189:0/1734604100 wait complete. 2024-04-06T12:13:37.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.154+0000 7fdc4b1f8700 1 Processor -- start 2024-04-06T12:13:37.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.154+0000 7fdc4b1f8700 1 -- start start 2024-04-06T12:13:37.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.155+0000 7fdc4b1f8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc4406ae40 0x7fdc441f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:37.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.155+0000 7fdc4b1f8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc440b3e20 0x7fdc4415b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:37.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.155+0000 7fdc4b1f8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc4415d7b0 0x7fdc4415b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:37.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.155+0000 7fdc4b1f8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4415afb0 con 0x7fdc4415d7b0 2024-04-06T12:13:37.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.155+0000 7fdc4b1f8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc440cda60 con 0x7fdc440b3e20 2024-04-06T12:13:37.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.155+0000 7fdc4b1f8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4415a680 con 0x7fdc4406ae40 2024-04-06T12:13:37.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.155+0000 7fdc49795700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc4415d7b0 0x7fdc4415b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:37.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.155+0000 7fdc49795700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc4415d7b0 0x7fdc4415b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51398/0 (socket says 172.21.15.189:51398) 2024-04-06T12:13:37.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.155+0000 7fdc49795700 1 -- 172.21.15.189:0/1132251752 learned_addr learned my addr 172.21.15.189:0/1132251752 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:37.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.155+0000 7fdc48f94700 1 --2- 172.21.15.189:0/1132251752 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc4406ae40 0x7fdc441f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:37.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.155+0000 7fdc43fff700 1 --2- 172.21.15.189:0/1132251752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc440b3e20 0x7fdc4415b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:37.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.155+0000 7fdc49795700 1 -- 172.21.15.189:0/1132251752 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc4406ae40 msgr2=0x7fdc441f26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:37.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.155+0000 7fdc49795700 1 --2- 172.21.15.189:0/1132251752 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc4406ae40 0x7fdc441f26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.155+0000 7fdc49795700 1 -- 172.21.15.189:0/1132251752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc440b3e20 msgr2=0x7fdc4415b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:37.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.155+0000 7fdc49795700 1 --2- 172.21.15.189:0/1132251752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc440b3e20 0x7fdc4415b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.155+0000 7fdc49795700 1 -- 172.21.15.189:0/1132251752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3c009b30 con 0x7fdc4415d7b0 2024-04-06T12:13:37.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.155+0000 7fdc49795700 1 --2- 172.21.15.189:0/1132251752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc4415d7b0 0x7fdc4415b980 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fdc4406c820 tx=0x7fdc3c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:37.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.156+0000 7fdc417fa700 1 -- 172.21.15.189:0/1132251752 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c004440 con 0x7fdc4415d7b0 2024-04-06T12:13:37.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.156+0000 7fdc417fa700 1 -- 172.21.15.189:0/1132251752 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3c0045c0 con 0x7fdc4415d7b0 2024-04-06T12:13:37.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.156+0000 7fdc4b1f8700 1 -- 172.21.15.189:0/1132251752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4415a960 con 0x7fdc4415d7b0 2024-04-06T12:13:37.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.156+0000 7fdc417fa700 1 -- 172.21.15.189:0/1132251752 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c004970 con 0x7fdc4415d7b0 2024-04-06T12:13:37.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.156+0000 7fdc4b1f8700 1 -- 172.21.15.189:0/1132251752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc44150bf0 con 0x7fdc4415d7b0 2024-04-06T12:13:37.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.157+0000 7fdc4b1f8700 1 -- 172.21.15.189:0/1132251752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc44150d80 con 0x7fdc4415d7b0 2024-04-06T12:13:37.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.157+0000 7fdc417fa700 1 -- 172.21.15.189:0/1132251752 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 9) v1 ==== 89000+0+0 (secure 0 0 0) 0x7fdc3c025450 con 0x7fdc4415d7b0 2024-04-06T12:13:37.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.157+0000 7fdc417fa700 1 --2- 172.21.15.189:0/1132251752 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdc2806ab30 0x7fdc2806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:37.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.157+0000 7fdc417fa700 1 -- 172.21.15.189:0/1132251752 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fdc3c08a9b0 con 0x7fdc4415d7b0 2024-04-06T12:13:37.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.157+0000 7fdc48f94700 1 --2- 172.21.15.189:0/1132251752 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdc2806ab30 0x7fdc2806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:37.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.158+0000 7fdc48f94700 1 --2- 172.21.15.189:0/1132251752 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdc2806ab30 0x7fdc2806cff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009960 tx=0x7fdc380090e0 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:37.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.162+0000 7fdc417fa700 1 -- 172.21.15.189:0/1132251752 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdc3c0596a0 con 0x7fdc4415d7b0 2024-04-06T12:13:37.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.351+0000 7fdc417fa700 1 -- 172.21.15.189:0/1132251752 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 10) v1 ==== 89265+0+0 (secure 0 0 0) 0x7fdc3c0093a0 con 0x7fdc4415d7b0 2024-04-06T12:13:37.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.389+0000 7fdc4b1f8700 1 -- 172.21.15.189:0/1132251752 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir3", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc4415ae20 con 0x7fdc2806ab30 2024-04-06T12:13:37.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.458+0000 7fdc417fa700 1 -- 172.21.15.189:0/1132251752 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fdc4415ae20 con 0x7fdc2806ab30 2024-04-06T12:13:37.456 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path /snap_test_dir1/dir3 2024-04-06T12:13:37.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.460+0000 7fdc4b1f8700 1 -- 172.21.15.189:0/1132251752 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdc2806ab30 msgr2=0x7fdc2806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:37.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.460+0000 7fdc4b1f8700 1 --2- 172.21.15.189:0/1132251752 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdc2806ab30 0x7fdc2806cff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009960 tx=0x7fdc380090e0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.460+0000 7fdc4b1f8700 1 -- 172.21.15.189:0/1132251752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc4415d7b0 msgr2=0x7fdc4415b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:37.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.460+0000 7fdc4b1f8700 1 --2- 172.21.15.189:0/1132251752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc4415d7b0 0x7fdc4415b980 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fdc4406c820 tx=0x7fdc3c004ad0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.460+0000 7fdc4b1f8700 1 -- 172.21.15.189:0/1132251752 shutdown_connections 2024-04-06T12:13:37.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.460+0000 7fdc4b1f8700 1 --2- 172.21.15.189:0/1132251752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc4415d7b0 0x7fdc4415b980 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.460+0000 7fdc4b1f8700 1 --2- 172.21.15.189:0/1132251752 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc4406ae40 0x7fdc441f26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.460+0000 7fdc4b1f8700 1 --2- 172.21.15.189:0/1132251752 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdc2806ab30 0x7fdc2806cff0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.460+0000 7fdc4b1f8700 1 --2- 172.21.15.189:0/1132251752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc440b3e20 0x7fdc4415b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.460+0000 7fdc4b1f8700 1 -- 172.21.15.189:0/1132251752 >> 172.21.15.189:0/1132251752 conn(0x7fdc440d00b0 msgr2=0x7fdc4415c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:37.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.460+0000 7fdc4b1f8700 1 -- 172.21.15.189:0/1132251752 shutdown_connections 2024-04-06T12:13:37.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.460+0000 7fdc4b1f8700 1 -- 172.21.15.189:0/1132251752 wait complete. 2024-04-06T12:13:37.471 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2024-04-06T12:13:37.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:37.471 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2024-04-06T12:13:37.489 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir4 1m 2024-04-06T12:13:37.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.656+0000 7ffb13874700 1 Processor -- start 2024-04-06T12:13:37.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.656+0000 7ffb13874700 1 -- start start 2024-04-06T12:13:37.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.656+0000 7ffb13874700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffb0c0d17a0 0x7ffb0c06b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:37.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb13874700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ffb0c146dd0 0x7ffb0c06d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:37.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb13874700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb0c0cedf0 0x7ffb0c0cf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:37.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb13874700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb0c0cf780 con 0x7ffb0c0d17a0 2024-04-06T12:13:37.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb13874700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb0c06c0a0 con 0x7ffb0c146dd0 2024-04-06T12:13:37.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb13874700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c06c380 con 0x7ffb0c0cedf0 2024-04-06T12:13:37.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb11610700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffb0c0d17a0 0x7ffb0c06b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:37.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb11610700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffb0c0d17a0 0x7ffb0c06b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51402/0 (socket says 172.21.15.189:51402) 2024-04-06T12:13:37.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb11610700 1 -- 172.21.15.189:0/2279553034 learned_addr learned my addr 172.21.15.189:0/2279553034 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:37.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb10e0f700 1 --2- 172.21.15.189:0/2279553034 >> v2:172.21.15.192:3300/0 conn(0x7ffb0c146dd0 0x7ffb0c06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:37.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb11610700 1 -- 172.21.15.189:0/2279553034 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb0c0cedf0 msgr2=0x7ffb0c0cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:37.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb11e11700 1 --2- 172.21.15.189:0/2279553034 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb0c0cedf0 0x7ffb0c0cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:37.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb11610700 1 --2- 172.21.15.189:0/2279553034 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb0c0cedf0 0x7ffb0c0cf240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb11610700 1 -- 172.21.15.189:0/2279553034 >> v2:172.21.15.192:3300/0 conn(0x7ffb0c146dd0 msgr2=0x7ffb0c06d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:37.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb11610700 1 --2- 172.21.15.189:0/2279553034 >> v2:172.21.15.192:3300/0 conn(0x7ffb0c146dd0 0x7ffb0c06d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb11610700 1 -- 172.21.15.189:0/2279553034 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c06c660 con 0x7ffb0c0d17a0 2024-04-06T12:13:37.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb10e0f700 1 --2- 172.21.15.189:0/2279553034 >> v2:172.21.15.192:3300/0 conn(0x7ffb0c146dd0 0x7ffb0c06d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:37.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb11e11700 1 --2- 172.21.15.189:0/2279553034 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb0c0cedf0 0x7ffb0c0cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:37.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb11610700 1 --2- 172.21.15.189:0/2279553034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb0c0d17a0 0x7ffb0c06b7a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0d2300 tx=0x7ffafc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71719408e9972236 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:37.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb03fff700 1 -- 172.21.15.189:0/2279553034 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc00b870 con 0x7ffb0c0d17a0 2024-04-06T12:13:37.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb03fff700 1 -- 172.21.15.189:0/2279553034 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffafc00b9d0 con 0x7ffb0c0d17a0 2024-04-06T12:13:37.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.657+0000 7ffb03fff700 1 -- 172.21.15.189:0/2279553034 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc00bd60 con 0x7ffb0c0d17a0 2024-04-06T12:13:37.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.658+0000 7ffb13874700 1 -- 172.21.15.189:0/2279553034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb0c0d17a0 msgr2=0x7ffb0c06b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:37.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.658+0000 7ffb13874700 1 --2- 172.21.15.189:0/2279553034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb0c0d17a0 0x7ffb0c06b7a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0d2300 tx=0x7ffafc009bd0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.658+0000 7ffb13874700 1 -- 172.21.15.189:0/2279553034 shutdown_connections 2024-04-06T12:13:37.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.658+0000 7ffb13874700 1 --2- 172.21.15.189:0/2279553034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb0c0d17a0 0x7ffb0c06b7a0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.658+0000 7ffb13874700 1 --2- 172.21.15.189:0/2279553034 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb0c0cedf0 0x7ffb0c0cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.658+0000 7ffb13874700 1 --2- 172.21.15.189:0/2279553034 >> v2:172.21.15.192:3300/0 conn(0x7ffb0c146dd0 0x7ffb0c06d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.658+0000 7ffb13874700 1 -- 172.21.15.189:0/2279553034 >> 172.21.15.189:0/2279553034 conn(0x7ffb0c0b2db0 msgr2=0x7ffb0c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:37.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.658+0000 7ffb13874700 1 -- 172.21.15.189:0/2279553034 shutdown_connections 2024-04-06T12:13:37.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.658+0000 7ffb13874700 1 -- 172.21.15.189:0/2279553034 wait complete. 2024-04-06T12:13:37.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.659+0000 7ffb13874700 1 Processor -- start 2024-04-06T12:13:37.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.659+0000 7ffb13874700 1 -- start start 2024-04-06T12:13:37.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.660+0000 7ffb13874700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb0c0cedf0 0x7ffb0c158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:37.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.660+0000 7ffb13874700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb0c0d17a0 0x7ffb0c158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:37.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.660+0000 7ffb13874700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb0c146dd0 0x7ffb0c0acb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:37.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.660+0000 7ffb13874700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c0af260 con 0x7ffb0c0cedf0 2024-04-06T12:13:37.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.660+0000 7ffb13874700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c0aea50 con 0x7ffb0c0d17a0 2024-04-06T12:13:37.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.660+0000 7ffb13874700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c0c4f80 con 0x7ffb0c146dd0 2024-04-06T12:13:37.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.660+0000 7ffb11610700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb0c0cedf0 0x7ffb0c158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:37.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.660+0000 7ffb11610700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb0c0cedf0 0x7ffb0c158a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51404/0 (socket says 172.21.15.189:51404) 2024-04-06T12:13:37.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.660+0000 7ffb11610700 1 -- 172.21.15.189:0/2539621894 learned_addr learned my addr 172.21.15.189:0/2539621894 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:37.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.660+0000 7ffb10e0f700 1 --2- 172.21.15.189:0/2539621894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb0c0d17a0 0x7ffb0c158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:37.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.660+0000 7ffb11e11700 1 --2- 172.21.15.189:0/2539621894 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb0c146dd0 0x7ffb0c0acb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:37.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.660+0000 7ffb11610700 1 -- 172.21.15.189:0/2539621894 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb0c146dd0 msgr2=0x7ffb0c0acb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:37.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.660+0000 7ffb11610700 1 --2- 172.21.15.189:0/2539621894 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb0c146dd0 0x7ffb0c0acb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.660+0000 7ffb11610700 1 -- 172.21.15.189:0/2539621894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb0c0d17a0 msgr2=0x7ffb0c158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:37.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.660+0000 7ffb11610700 1 --2- 172.21.15.189:0/2539621894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb0c0d17a0 0x7ffb0c158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.660+0000 7ffb11610700 1 -- 172.21.15.189:0/2539621894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffafc009550 con 0x7ffb0c0cedf0 2024-04-06T12:13:37.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.660+0000 7ffb11610700 1 --2- 172.21.15.189:0/2539621894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb0c0cedf0 0x7ffb0c158a40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0677e0 tx=0x7ffafc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:37.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.661+0000 7ffb01ffb700 1 -- 172.21.15.189:0/2539621894 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc011830 con 0x7ffb0c0cedf0 2024-04-06T12:13:37.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.661+0000 7ffb01ffb700 1 -- 172.21.15.189:0/2539621894 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffafc01e070 con 0x7ffb0c0cedf0 2024-04-06T12:13:37.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.661+0000 7ffb13874700 1 -- 172.21.15.189:0/2539621894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb0c0aeca0 con 0x7ffb0c0cedf0 2024-04-06T12:13:37.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.661+0000 7ffb01ffb700 1 -- 172.21.15.189:0/2539621894 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc011bb0 con 0x7ffb0c0cedf0 2024-04-06T12:13:37.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.661+0000 7ffb13874700 1 -- 172.21.15.189:0/2539621894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb0c1612e0 con 0x7ffb0c0cedf0 2024-04-06T12:13:37.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.662+0000 7ffb13874700 1 -- 172.21.15.189:0/2539621894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb0c0c4cf0 con 0x7ffb0c0cedf0 2024-04-06T12:13:37.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.662+0000 7ffb01ffb700 1 -- 172.21.15.189:0/2539621894 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 10) v1 ==== 89265+0+0 (secure 0 0 0) 0x7ffafc011d10 con 0x7ffb0c0cedf0 2024-04-06T12:13:37.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.663+0000 7ffb01ffb700 1 --2- 172.21.15.189:0/2539621894 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffaf006ae00 0x7ffaf006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:37.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.663+0000 7ffb10e0f700 1 --2- 172.21.15.189:0/2539621894 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffaf006ae00 0x7ffaf006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:37.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.663+0000 7ffb01ffb700 1 -- 172.21.15.189:0/2539621894 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7ffafc020070 con 0x7ffb0c0cedf0 2024-04-06T12:13:37.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.663+0000 7ffb10e0f700 1 --2- 172.21.15.189:0/2539621894 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffaf006ae00 0x7ffaf006d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffb08006fd0 tx=0x7ffb08008040 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:37.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.667+0000 7ffb01ffb700 1 -- 172.21.15.189:0/2539621894 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ffafc055be0 con 0x7ffb0c0cedf0 2024-04-06T12:13:37.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.893+0000 7ffb13874700 1 -- 172.21.15.189:0/2539621894 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir4", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb0c0d1b80 con 0x7ffaf006ae00 2024-04-06T12:13:37.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.953+0000 7ffb01ffb700 1 -- 172.21.15.189:0/2539621894 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7ffb0800c040 con 0x7ffaf006ae00 2024-04-06T12:13:37.951 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path /snap_test_dir1/dir4 2024-04-06T12:13:37.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.956+0000 7ffb13874700 1 -- 172.21.15.189:0/2539621894 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffaf006ae00 msgr2=0x7ffaf006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:37.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.956+0000 7ffb13874700 1 --2- 172.21.15.189:0/2539621894 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffaf006ae00 0x7ffaf006d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffb08006fd0 tx=0x7ffb08008040 comp rx=0 tx=0).stop 2024-04-06T12:13:37.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.956+0000 7ffb13874700 1 -- 172.21.15.189:0/2539621894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb0c0cedf0 msgr2=0x7ffb0c158a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:37.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.956+0000 7ffb13874700 1 --2- 172.21.15.189:0/2539621894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb0c0cedf0 0x7ffb0c158a40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0677e0 tx=0x7ffafc004750 comp rx=0 tx=0).stop 2024-04-06T12:13:37.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.956+0000 7ffb13874700 1 -- 172.21.15.189:0/2539621894 shutdown_connections 2024-04-06T12:13:37.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.956+0000 7ffb13874700 1 --2- 172.21.15.189:0/2539621894 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb0c146dd0 0x7ffb0c0acb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.956+0000 7ffb13874700 1 --2- 172.21.15.189:0/2539621894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb0c0cedf0 0x7ffb0c158a40 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.956+0000 7ffb13874700 1 --2- 172.21.15.189:0/2539621894 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffaf006ae00 0x7ffaf006d2c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.956+0000 7ffb13874700 1 --2- 172.21.15.189:0/2539621894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb0c0d17a0 0x7ffb0c158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:37.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.956+0000 7ffb13874700 1 -- 172.21.15.189:0/2539621894 >> 172.21.15.189:0/2539621894 conn(0x7ffb0c0b2db0 msgr2=0x7ffb0c15d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:37.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.956+0000 7ffb13874700 1 -- 172.21.15.189:0/2539621894 shutdown_connections 2024-04-06T12:13:37.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:37.956+0000 7ffb13874700 1 -- 172.21.15.189:0/2539621894 wait complete. 2024-04-06T12:13:37.966 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2024-04-06T12:13:37.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:37.966 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2024-04-06T12:13:37.985 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir5 1m 2024-04-06T12:13:38.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.152+0000 7f2993ead700 1 Processor -- start 2024-04-06T12:13:38.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.153+0000 7f2993ead700 1 -- start start 2024-04-06T12:13:38.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.153+0000 7f2993ead700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f298c0d11b0 0x7f298c0d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:38.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.153+0000 7f2993ead700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f298c0cb870 0x7f298c15c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:38.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.153+0000 7f2993ead700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f298c0cfa00 0x7f298c0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:38.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.153+0000 7f2993ead700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f298c0ccdc0 con 0x7f298c0cb870 2024-04-06T12:13:38.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.153+0000 7f2993ead700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f298c0d0b50 con 0x7f298c0cfa00 2024-04-06T12:13:38.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.153+0000 7f2993ead700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f298c0d1ad0 con 0x7f298c0d11b0 2024-04-06T12:13:38.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.154+0000 7f2991448700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f298c0cb870 0x7f298c15c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:38.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.154+0000 7f2991448700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f298c0cb870 0x7f298c15c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51412/0 (socket says 172.21.15.189:51412) 2024-04-06T12:13:38.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.154+0000 7f2991448700 1 -- 172.21.15.189:0/1637002962 learned_addr learned my addr 172.21.15.189:0/1637002962 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:38.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.154+0000 7f299244a700 1 --2- 172.21.15.189:0/1637002962 >> v2:172.21.15.192:3300/0 conn(0x7f298c0cfa00 0x7f298c0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:38.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.154+0000 7f2991c49700 1 --2- 172.21.15.189:0/1637002962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f298c0d11b0 0x7f298c0d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:38.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.154+0000 7f2991448700 1 -- 172.21.15.189:0/1637002962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f298c0d11b0 msgr2=0x7f298c0d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:38.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.154+0000 7f2991448700 1 --2- 172.21.15.189:0/1637002962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f298c0d11b0 0x7f298c0d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.154+0000 7f2991448700 1 -- 172.21.15.189:0/1637002962 >> v2:172.21.15.192:3300/0 conn(0x7f298c0cfa00 msgr2=0x7f298c0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:38.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.154+0000 7f2991448700 1 --2- 172.21.15.189:0/1637002962 >> v2:172.21.15.192:3300/0 conn(0x7f298c0cfa00 0x7f298c0cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.154+0000 7f2991448700 1 -- 172.21.15.189:0/1637002962 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298c0cbd80 con 0x7f298c0cb870 2024-04-06T12:13:38.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.154+0000 7f2991448700 1 --2- 172.21.15.189:0/1637002962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f298c0cb870 0x7f298c15c280 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f298c0b0990 tx=0x7f2988009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83020be8eab1bca1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:38.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.154+0000 7f2990c47700 1 -- 172.21.15.189:0/1637002962 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298800b870 con 0x7f298c0cb870 2024-04-06T12:13:38.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.154+0000 7f2990c47700 1 -- 172.21.15.189:0/1637002962 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f298800b9d0 con 0x7f298c0cb870 2024-04-06T12:13:38.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.154+0000 7f2990c47700 1 -- 172.21.15.189:0/1637002962 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298800bd60 con 0x7f298c0cb870 2024-04-06T12:13:38.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.155+0000 7f2993ead700 1 -- 172.21.15.189:0/1637002962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f298c0cb870 msgr2=0x7f298c15c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:38.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.155+0000 7f2993ead700 1 --2- 172.21.15.189:0/1637002962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f298c0cb870 0x7f298c15c280 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f298c0b0990 tx=0x7f2988009bd0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.155+0000 7f2993ead700 1 -- 172.21.15.189:0/1637002962 shutdown_connections 2024-04-06T12:13:38.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.155+0000 7f2993ead700 1 --2- 172.21.15.189:0/1637002962 >> v2:172.21.15.192:3300/0 conn(0x7f298c0cfa00 0x7f298c0cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.155+0000 7f2993ead700 1 --2- 172.21.15.189:0/1637002962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f298c0d11b0 0x7f298c0d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.155+0000 7f2993ead700 1 --2- 172.21.15.189:0/1637002962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f298c0cb870 0x7f298c15c280 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.155+0000 7f2993ead700 1 -- 172.21.15.189:0/1637002962 >> 172.21.15.189:0/1637002962 conn(0x7f298c0b2db0 msgr2=0x7f298c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:38.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.155+0000 7f2993ead700 1 -- 172.21.15.189:0/1637002962 shutdown_connections 2024-04-06T12:13:38.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.155+0000 7f2993ead700 1 -- 172.21.15.189:0/1637002962 wait complete. 2024-04-06T12:13:38.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.156+0000 7f2993ead700 1 Processor -- start 2024-04-06T12:13:38.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.156+0000 7f2993ead700 1 -- start start 2024-04-06T12:13:38.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.157+0000 7f2993ead700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f298c0cb870 0x7f298c152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:38.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.157+0000 7f2993ead700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f298c0cfa00 0x7f298c0c6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:38.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.157+0000 7f2993ead700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f298c0d11b0 0x7f298c06d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:38.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.157+0000 7f2993ead700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298c0c60f0 con 0x7f298c0cb870 2024-04-06T12:13:38.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.157+0000 7f2993ead700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298c0c63d0 con 0x7f298c0cfa00 2024-04-06T12:13:38.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.157+0000 7f2993ead700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f298c0c66b0 con 0x7f298c0d11b0 2024-04-06T12:13:38.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.157+0000 7f2991c49700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f298c0cb870 0x7f298c152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:38.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.157+0000 7f2991c49700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f298c0cb870 0x7f298c152280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51416/0 (socket says 172.21.15.189:51416) 2024-04-06T12:13:38.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.157+0000 7f2991c49700 1 -- 172.21.15.189:0/5587332 learned_addr learned my addr 172.21.15.189:0/5587332 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:38.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.157+0000 7f299244a700 1 --2- 172.21.15.189:0/5587332 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f298c0d11b0 0x7f298c06d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:38.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.157+0000 7f2991448700 1 --2- 172.21.15.189:0/5587332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f298c0cfa00 0x7f298c0c6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:38.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.157+0000 7f2991c49700 1 -- 172.21.15.189:0/5587332 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f298c0d11b0 msgr2=0x7f298c06d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:38.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.157+0000 7f2991c49700 1 --2- 172.21.15.189:0/5587332 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f298c0d11b0 0x7f298c06d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.157+0000 7f2991c49700 1 -- 172.21.15.189:0/5587332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f298c0cfa00 msgr2=0x7f298c0c6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:38.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.157+0000 7f2991c49700 1 --2- 172.21.15.189:0/5587332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f298c0cfa00 0x7f298c0c6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.157+0000 7f2991c49700 1 -- 172.21.15.189:0/5587332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2988009550 con 0x7f298c0cb870 2024-04-06T12:13:38.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.157+0000 7f2991c49700 1 --2- 172.21.15.189:0/5587332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f298c0cb870 0x7f298c152280 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f297c00e810 tx=0x7f297c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:38.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.158+0000 7f29827fc700 1 -- 172.21.15.189:0/5587332 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297c00c070 con 0x7f298c0cb870 2024-04-06T12:13:38.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.158+0000 7f29827fc700 1 -- 172.21.15.189:0/5587332 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f297c013610 con 0x7f298c0cb870 2024-04-06T12:13:38.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.158+0000 7f2993ead700 1 -- 172.21.15.189:0/5587332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f298c0c6990 con 0x7f298c0cb870 2024-04-06T12:13:38.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.158+0000 7f29827fc700 1 -- 172.21.15.189:0/5587332 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297c0139c0 con 0x7f298c0cb870 2024-04-06T12:13:38.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.158+0000 7f2993ead700 1 -- 172.21.15.189:0/5587332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f298c06d640 con 0x7f298c0cb870 2024-04-06T12:13:38.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.158+0000 7f2993ead700 1 -- 172.21.15.189:0/5587332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f298c0c6230 con 0x7f298c0cb870 2024-04-06T12:13:38.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.159+0000 7f29827fc700 1 -- 172.21.15.189:0/5587332 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 10) v1 ==== 89265+0+0 (secure 0 0 0) 0x7f297c013b20 con 0x7f298c0cb870 2024-04-06T12:13:38.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.159+0000 7f29827fc700 1 --2- 172.21.15.189:0/5587332 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f297006ae00 0x7f297006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:38.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.159+0000 7f29827fc700 1 -- 172.21.15.189:0/5587332 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f297c089590 con 0x7f298c0cb870 2024-04-06T12:13:38.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.159+0000 7f2991448700 1 --2- 172.21.15.189:0/5587332 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f297006ae00 0x7f297006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:38.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.160+0000 7f2991448700 1 --2- 172.21.15.189:0/5587332 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f297006ae00 0x7f297006d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f298c0cbcb0 tx=0x7f2988008030 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:38.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.164+0000 7f29827fc700 1 -- 172.21.15.189:0/5587332 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f298c0c6230 con 0x7f298c0cb870 2024-04-06T12:13:38.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.385+0000 7f2993ead700 1 -- 172.21.15.189:0/5587332 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir5", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f298c0cf4e0 con 0x7f297006ae00 2024-04-06T12:13:38.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.448+0000 7f29827fc700 1 -- 172.21.15.189:0/5587332 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f298800bb40 con 0x7f297006ae00 2024-04-06T12:13:38.446 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path /snap_test_dir1/dir5 2024-04-06T12:13:38.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.450+0000 7f2993ead700 1 -- 172.21.15.189:0/5587332 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f297006ae00 msgr2=0x7f297006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:38.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.450+0000 7f2993ead700 1 --2- 172.21.15.189:0/5587332 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f297006ae00 0x7f297006d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f298c0cbcb0 tx=0x7f2988008030 comp rx=0 tx=0).stop 2024-04-06T12:13:38.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.450+0000 7f2993ead700 1 -- 172.21.15.189:0/5587332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f298c0cb870 msgr2=0x7f298c152280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:38.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.450+0000 7f2993ead700 1 --2- 172.21.15.189:0/5587332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f298c0cb870 0x7f298c152280 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f297c00e810 tx=0x7f297c00eb20 comp rx=0 tx=0).stop 2024-04-06T12:13:38.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.450+0000 7f2993ead700 1 -- 172.21.15.189:0/5587332 shutdown_connections 2024-04-06T12:13:38.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.450+0000 7f2993ead700 1 --2- 172.21.15.189:0/5587332 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f298c0d11b0 0x7f298c06d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.450+0000 7f2993ead700 1 --2- 172.21.15.189:0/5587332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f298c0cb870 0x7f298c152280 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.450+0000 7f2993ead700 1 --2- 172.21.15.189:0/5587332 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f297006ae00 0x7f297006d2c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.450+0000 7f2993ead700 1 --2- 172.21.15.189:0/5587332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f298c0cfa00 0x7f298c0c6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.450+0000 7f2993ead700 1 -- 172.21.15.189:0/5587332 >> 172.21.15.189:0/5587332 conn(0x7f298c0b2db0 msgr2=0x7f298c0cd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:38.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.450+0000 7f2993ead700 1 -- 172.21.15.189:0/5587332 shutdown_connections 2024-04-06T12:13:38.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.450+0000 7f2993ead700 1 -- 172.21.15.189:0/5587332 wait complete. 2024-04-06T12:13:38.460 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2024-04-06T12:13:38.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:38.460 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2024-04-06T12:13:38.479 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir6 1m 2024-04-06T12:13:38.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.647+0000 7f9333d7c700 1 Processor -- start 2024-04-06T12:13:38.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.647+0000 7f9333d7c700 1 -- start start 2024-04-06T12:13:38.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.648+0000 7f9333d7c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f932c158fe0 0x7f932c06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:38.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.648+0000 7f9333d7c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f932c06ac70 0x7f932c15bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:38.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.648+0000 7f9333d7c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f932c15c100 0x7f932c0717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:38.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.648+0000 7f9333d7c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f932c06b0f0 con 0x7f932c15c100 2024-04-06T12:13:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.648+0000 7f9333d7c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f932c15f9e0 con 0x7f932c158fe0 2024-04-06T12:13:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.648+0000 7f9333d7c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f932c15f0d0 con 0x7f932c06ac70 2024-04-06T12:13:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.648+0000 7f9332319700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f932c15c100 0x7f932c0717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.648+0000 7f9332319700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f932c15c100 0x7f932c0717b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51426/0 (socket says 172.21.15.189:51426) 2024-04-06T12:13:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.648+0000 7f9332319700 1 -- 172.21.15.189:0/725285507 learned_addr learned my addr 172.21.15.189:0/725285507 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.648+0000 7f9332319700 1 -- 172.21.15.189:0/725285507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f932c06ac70 msgr2=0x7f932c15bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.648+0000 7f9331b18700 1 --2- 172.21.15.189:0/725285507 >> v2:172.21.15.192:3300/0 conn(0x7f932c158fe0 0x7f932c06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.648+0000 7f9331317700 1 --2- 172.21.15.189:0/725285507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f932c06ac70 0x7f932c15bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.648+0000 7f9332319700 1 --2- 172.21.15.189:0/725285507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f932c06ac70 0x7f932c15bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.648+0000 7f9332319700 1 -- 172.21.15.189:0/725285507 >> v2:172.21.15.192:3300/0 conn(0x7f932c158fe0 msgr2=0x7f932c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.648+0000 7f9332319700 1 --2- 172.21.15.189:0/725285507 >> v2:172.21.15.192:3300/0 conn(0x7f932c158fe0 0x7f932c06a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.648+0000 7f9332319700 1 -- 172.21.15.189:0/725285507 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932c15c8f0 con 0x7f932c15c100 2024-04-06T12:13:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.648+0000 7f9331b18700 1 --2- 172.21.15.189:0/725285507 >> v2:172.21.15.192:3300/0 conn(0x7f932c158fe0 0x7f932c06a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:38.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.648+0000 7f9331317700 1 --2- 172.21.15.189:0/725285507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f932c06ac70 0x7f932c15bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.648+0000 7f9332319700 1 --2- 172.21.15.189:0/725285507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f932c15c100 0x7f932c0717b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f932c073470 tx=0x7f932400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51148d83f45ccdec server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.649+0000 7f9330b16700 1 -- 172.21.15.189:0/725285507 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932400e070 con 0x7f932c15c100 2024-04-06T12:13:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.649+0000 7f9330b16700 1 -- 172.21.15.189:0/725285507 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932400b9c0 con 0x7f932c15c100 2024-04-06T12:13:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.649+0000 7f9330b16700 1 -- 172.21.15.189:0/725285507 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932400bd70 con 0x7f932c15c100 2024-04-06T12:13:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.649+0000 7f9333d7c700 1 -- 172.21.15.189:0/725285507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f932c15c100 msgr2=0x7f932c0717b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.649+0000 7f9333d7c700 1 --2- 172.21.15.189:0/725285507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f932c15c100 0x7f932c0717b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f932c073470 tx=0x7f932400b330 comp rx=0 tx=0).stop 2024-04-06T12:13:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.649+0000 7f9333d7c700 1 -- 172.21.15.189:0/725285507 shutdown_connections 2024-04-06T12:13:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.649+0000 7f9333d7c700 1 --2- 172.21.15.189:0/725285507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f932c15c100 0x7f932c0717b0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.649+0000 7f9333d7c700 1 --2- 172.21.15.189:0/725285507 >> v2:172.21.15.192:3300/0 conn(0x7f932c158fe0 0x7f932c06a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.649+0000 7f9333d7c700 1 --2- 172.21.15.189:0/725285507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f932c06ac70 0x7f932c15bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.649+0000 7f9333d7c700 1 -- 172.21.15.189:0/725285507 >> 172.21.15.189:0/725285507 conn(0x7f932c0d00b0 msgr2=0x7f932c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.649+0000 7f9333d7c700 1 -- 172.21.15.189:0/725285507 shutdown_connections 2024-04-06T12:13:38.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.649+0000 7f9333d7c700 1 -- 172.21.15.189:0/725285507 wait complete. 2024-04-06T12:13:38.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.650+0000 7f9333d7c700 1 Processor -- start 2024-04-06T12:13:38.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.650+0000 7f9333d7c700 1 -- start start 2024-04-06T12:13:38.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.651+0000 7f9333d7c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f932c06ac70 0x7f932c15df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.651+0000 7f9333d7c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f932c158fe0 0x7f932c0c7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.651+0000 7f9333d7c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f932c15c100 0x7f932c0c7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.651+0000 7f9333d7c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c0cbb00 con 0x7f932c06ac70 2024-04-06T12:13:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.651+0000 7f9333d7c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c0c97b0 con 0x7f932c158fe0 2024-04-06T12:13:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.651+0000 7f9333d7c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f932c0c9a90 con 0x7f932c15c100 2024-04-06T12:13:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.651+0000 7f9331b18700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f932c06ac70 0x7f932c15df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.651+0000 7f9331b18700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f932c06ac70 0x7f932c15df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51438/0 (socket says 172.21.15.189:51438) 2024-04-06T12:13:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.651+0000 7f9331b18700 1 -- 172.21.15.189:0/3482463684 learned_addr learned my addr 172.21.15.189:0/3482463684 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.651+0000 7f9332319700 1 --2- 172.21.15.189:0/3482463684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f932c15c100 0x7f932c0c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.651+0000 7f9331317700 1 --2- 172.21.15.189:0/3482463684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f932c158fe0 0x7f932c0c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.652+0000 7f9331b18700 1 -- 172.21.15.189:0/3482463684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f932c15c100 msgr2=0x7f932c0c7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.652+0000 7f9331b18700 1 --2- 172.21.15.189:0/3482463684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f932c15c100 0x7f932c0c7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.652+0000 7f9331b18700 1 -- 172.21.15.189:0/3482463684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f932c158fe0 msgr2=0x7f932c0c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:38.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.652+0000 7f9331b18700 1 --2- 172.21.15.189:0/3482463684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f932c158fe0 0x7f932c0c7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.652+0000 7f9331b18700 1 -- 172.21.15.189:0/3482463684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9324009b30 con 0x7f932c06ac70 2024-04-06T12:13:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.652+0000 7f9331b18700 1 --2- 172.21.15.189:0/3482463684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f932c06ac70 0x7f932c15df30 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f932800c750 tx=0x7f932800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.652+0000 7f93227fc700 1 -- 172.21.15.189:0/3482463684 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9328011800 con 0x7f932c06ac70 2024-04-06T12:13:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.652+0000 7f93227fc700 1 -- 172.21.15.189:0/3482463684 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9328011960 con 0x7f932c06ac70 2024-04-06T12:13:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.652+0000 7f93227fc700 1 -- 172.21.15.189:0/3482463684 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9328011cd0 con 0x7f932c06ac70 2024-04-06T12:13:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.652+0000 7f9333d7c700 1 -- 172.21.15.189:0/3482463684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f932c0c9d70 con 0x7f932c06ac70 2024-04-06T12:13:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.652+0000 7f9333d7c700 1 -- 172.21.15.189:0/3482463684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f932c0ca330 con 0x7f932c06ac70 2024-04-06T12:13:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.653+0000 7f9333d7c700 1 -- 172.21.15.189:0/3482463684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f932c067bb0 con 0x7f932c06ac70 2024-04-06T12:13:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.653+0000 7f93227fc700 1 -- 172.21.15.189:0/3482463684 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 10) v1 ==== 89265+0+0 (secure 0 0 0) 0x7f9328021350 con 0x7f932c06ac70 2024-04-06T12:13:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.654+0000 7f93227fc700 1 --2- 172.21.15.189:0/3482463684 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f931006ae00 0x7f931006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.654+0000 7f93227fc700 1 -- 172.21.15.189:0/3482463684 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f9328089810 con 0x7f932c06ac70 2024-04-06T12:13:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.654+0000 7f9331317700 1 --2- 172.21.15.189:0/3482463684 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f931006ae00 0x7f931006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:38.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.654+0000 7f9331317700 1 --2- 172.21.15.189:0/3482463684 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f931006ae00 0x7f931006d2c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f932c0cbed0 tx=0x7f931c008040 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:38.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.658+0000 7f93227fc700 1 -- 172.21.15.189:0/3482463684 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9328054590 con 0x7f932c06ac70 2024-04-06T12:13:38.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.882+0000 7f9333d7c700 1 -- 172.21.15.189:0/3482463684 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir6", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f932c1593c0 con 0x7f931006ae00 2024-04-06T12:13:38.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.963+0000 7f93227fc700 1 -- 172.21.15.189:0/3482463684 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f931c00ae70 con 0x7f931006ae00 2024-04-06T12:13:38.962 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path /snap_test_dir1/dir6 2024-04-06T12:13:38.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.966+0000 7f9333d7c700 1 -- 172.21.15.189:0/3482463684 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f931006ae00 msgr2=0x7f931006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:38.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.966+0000 7f9333d7c700 1 --2- 172.21.15.189:0/3482463684 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f931006ae00 0x7f931006d2c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f932c0cbed0 tx=0x7f931c008040 comp rx=0 tx=0).stop 2024-04-06T12:13:38.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.966+0000 7f9333d7c700 1 -- 172.21.15.189:0/3482463684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f932c06ac70 msgr2=0x7f932c15df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:38.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.966+0000 7f9333d7c700 1 --2- 172.21.15.189:0/3482463684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f932c06ac70 0x7f932c15df30 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f932800c750 tx=0x7f932800cb10 comp rx=0 tx=0).stop 2024-04-06T12:13:38.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.966+0000 7f9333d7c700 1 -- 172.21.15.189:0/3482463684 shutdown_connections 2024-04-06T12:13:38.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.966+0000 7f9333d7c700 1 --2- 172.21.15.189:0/3482463684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f932c15c100 0x7f932c0c7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.966+0000 7f9333d7c700 1 --2- 172.21.15.189:0/3482463684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f932c06ac70 0x7f932c15df30 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.966+0000 7f9333d7c700 1 --2- 172.21.15.189:0/3482463684 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f931006ae00 0x7f931006d2c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.966+0000 7f9333d7c700 1 --2- 172.21.15.189:0/3482463684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f932c158fe0 0x7f932c0c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:38.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.966+0000 7f9333d7c700 1 -- 172.21.15.189:0/3482463684 >> 172.21.15.189:0/3482463684 conn(0x7f932c0d00b0 msgr2=0x7f932c1425d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:38.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.966+0000 7f9333d7c700 1 -- 172.21.15.189:0/3482463684 shutdown_connections 2024-04-06T12:13:38.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:38.966+0000 7f9333d7c700 1 -- 172.21.15.189:0/3482463684 wait complete. 2024-04-06T12:13:38.976 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2024-04-06T12:13:38.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:38.976 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2024-04-06T12:13:38.994 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir7 1m 2024-04-06T12:13:39.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.161+0000 7f1ac49cf700 1 Processor -- start 2024-04-06T12:13:39.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.161+0000 7f1ac49cf700 1 -- start start 2024-04-06T12:13:39.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.162+0000 7f1ac49cf700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1ac00d17a0 0x7f1ac006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:39.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.162+0000 7f1ac49cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac0146dd0 0x7f1ac006d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:39.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.162+0000 7f1ac49cf700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac006c0a0 con 0x7f1ac00cedf0 2024-04-06T12:13:39.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.162+0000 7f1ac49cf700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac006c380 con 0x7f1ac00d17a0 2024-04-06T12:13:39.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.162+0000 7f1ac49cf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac006c660 con 0x7f1ac0146dd0 2024-04-06T12:13:39.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.163+0000 7f1abdd9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac0146dd0 0x7f1ac006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:39.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.163+0000 7f1abe59c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1ac00d17a0 0x7f1ac006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:39.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.163+0000 7f1abed9d700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f1ac00cedf0 0x7f1ac01441f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:36276/0 (socket says 172.21.15.189:36276) 2024-04-06T12:13:39.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.163+0000 7f1abdd9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac0146dd0 0x7f1ac006d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:53236/0 (socket says 172.21.15.189:53236) 2024-04-06T12:13:39.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.163+0000 7f1abed9d700 1 -- 172.21.15.189:0/345449174 learned_addr learned my addr 172.21.15.189:0/345449174 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:39.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.163+0000 7f1abe59c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1ac00d17a0 0x7f1ac006b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:37686/0 (socket says 172.21.15.189:37686) 2024-04-06T12:13:39.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.163+0000 7f1abdd9b700 1 -- 172.21.15.189:0/345449174 >> v2:172.21.15.192:3300/0 conn(0x7f1ac00d17a0 msgr2=0x7f1ac006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:39.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.163+0000 7f1abdd9b700 1 --2- 172.21.15.189:0/345449174 >> v2:172.21.15.192:3300/0 conn(0x7f1ac00d17a0 0x7f1ac006b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:39.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.163+0000 7f1abdd9b700 1 -- 172.21.15.189:0/345449174 >> v1:172.21.15.192:6789/0 conn(0x7f1ac00cedf0 legacy=0x7f1ac01441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:39.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.163+0000 7f1abdd9b700 1 -- 172.21.15.189:0/345449174 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac006c940 con 0x7f1ac0146dd0 2024-04-06T12:13:39.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.163+0000 7f1abe59c700 1 --2- 172.21.15.189:0/345449174 >> v2:172.21.15.192:3300/0 conn(0x7f1ac00d17a0 0x7f1ac006b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:13:39.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.163+0000 7f1abdd9b700 1 --2- 172.21.15.189:0/345449174 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac0146dd0 0x7f1ac006d140 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0009840 tx=0x7f1ab0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6f6872aea70cbe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:39.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.163+0000 7f1abd59a700 1 -- 172.21.15.189:0/345449174 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0011830 con 0x7f1ac0146dd0 2024-04-06T12:13:39.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.163+0000 7f1abd59a700 1 -- 172.21.15.189:0/345449174 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab0011990 con 0x7f1ac0146dd0 2024-04-06T12:13:39.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.163+0000 7f1abd59a700 1 -- 172.21.15.189:0/345449174 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0011d20 con 0x7f1ac0146dd0 2024-04-06T12:13:39.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.164+0000 7f1ac49cf700 1 -- 172.21.15.189:0/345449174 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac0146dd0 msgr2=0x7f1ac006d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:39.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.164+0000 7f1ac49cf700 1 --2- 172.21.15.189:0/345449174 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac0146dd0 0x7f1ac006d140 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0009840 tx=0x7f1ab0009b50 comp rx=0 tx=0).stop 2024-04-06T12:13:39.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.164+0000 7f1ac49cf700 1 -- 172.21.15.189:0/345449174 shutdown_connections 2024-04-06T12:13:39.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.164+0000 7f1ac49cf700 1 --2- 172.21.15.189:0/345449174 >> v2:172.21.15.192:3300/0 conn(0x7f1ac00d17a0 0x7f1ac006b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:39.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.164+0000 7f1ac49cf700 1 --2- 172.21.15.189:0/345449174 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac0146dd0 0x7f1ac006d140 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:39.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.164+0000 7f1ac49cf700 1 -- 172.21.15.189:0/345449174 >> 172.21.15.189:0/345449174 conn(0x7f1ac00b2db0 msgr2=0x7f1ac00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:39.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.164+0000 7f1ac49cf700 1 -- 172.21.15.189:0/345449174 shutdown_connections 2024-04-06T12:13:39.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.164+0000 7f1ac49cf700 1 -- 172.21.15.189:0/345449174 wait complete. 2024-04-06T12:13:39.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.166+0000 7f1ac49cf700 1 Processor -- start 2024-04-06T12:13:39.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.166+0000 7f1ac49cf700 1 -- start start 2024-04-06T12:13:39.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1ac49cf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ac00cedf0 0x7f1ac01560d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:39.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1ac49cf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac00d17a0 0x7f1ac0157840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:39.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1ac49cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac0146dd0 0x7f1ac0157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:39.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1ac49cf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac00b2820 con 0x7f1ac00d17a0 2024-04-06T12:13:39.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1ac49cf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac0156610 con 0x7f1ac00cedf0 2024-04-06T12:13:39.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1ac49cf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac00c6b50 con 0x7f1ac0146dd0 2024-04-06T12:13:39.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1abdd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac00d17a0 0x7f1ac0157840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:39.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1abdd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac00d17a0 0x7f1ac0157840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51444/0 (socket says 172.21.15.189:51444) 2024-04-06T12:13:39.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1abdd9b700 1 -- 172.21.15.189:0/1860203764 learned_addr learned my addr 172.21.15.189:0/1860203764 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:39.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1abe59c700 1 --2- 172.21.15.189:0/1860203764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ac00cedf0 0x7f1ac01560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:39.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1abed9d700 1 --2- 172.21.15.189:0/1860203764 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac0146dd0 0x7f1ac0157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:39.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1abdd9b700 1 -- 172.21.15.189:0/1860203764 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac0146dd0 msgr2=0x7f1ac0157d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:39.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1abdd9b700 1 --2- 172.21.15.189:0/1860203764 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac0146dd0 0x7f1ac0157d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:39.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1abdd9b700 1 -- 172.21.15.189:0/1860203764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ac00cedf0 msgr2=0x7f1ac01560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:39.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1abdd9b700 1 --2- 172.21.15.189:0/1860203764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ac00cedf0 0x7f1ac01560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:39.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1abdd9b700 1 -- 172.21.15.189:0/1860203764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab40094a0 con 0x7f1ac00d17a0 2024-04-06T12:13:39.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1abdd9b700 1 --2- 172.21.15.189:0/1860203764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac00d17a0 0x7f1ac0157840 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0009fd0 tx=0x7f1ab0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:39.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1aaeffd700 1 -- 172.21.15.189:0/1860203764 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00036a0 con 0x7f1ac00d17a0 2024-04-06T12:13:39.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1aaeffd700 1 -- 172.21.15.189:0/1860203764 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab001f070 con 0x7f1ac00d17a0 2024-04-06T12:13:39.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1aaeffd700 1 -- 172.21.15.189:0/1860203764 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0003a60 con 0x7f1ac00d17a0 2024-04-06T12:13:39.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.167+0000 7f1ac49cf700 1 -- 172.21.15.189:0/1860203764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab00094a0 con 0x7f1ac00d17a0 2024-04-06T12:13:39.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.168+0000 7f1ac49cf700 1 -- 172.21.15.189:0/1860203764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac00c6df0 con 0x7f1ac00d17a0 2024-04-06T12:13:39.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.168+0000 7f1ac49cf700 1 -- 172.21.15.189:0/1860203764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac0074990 con 0x7f1ac00d17a0 2024-04-06T12:13:39.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.169+0000 7f1aaeffd700 1 -- 172.21.15.189:0/1860203764 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 10) v1 ==== 89265+0+0 (secure 0 0 0) 0x7f1ab0003bc0 con 0x7f1ac00d17a0 2024-04-06T12:13:39.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.169+0000 7f1aaeffd700 1 --2- 172.21.15.189:0/1860203764 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1aa006ae00 0x7f1aa006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:39.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.169+0000 7f1aaeffd700 1 -- 172.21.15.189:0/1860203764 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f1ab0021070 con 0x7f1ac00d17a0 2024-04-06T12:13:39.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.169+0000 7f1abe59c700 1 --2- 172.21.15.189:0/1860203764 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1aa006ae00 0x7f1aa006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:39.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.170+0000 7f1abe59c700 1 --2- 172.21.15.189:0/1860203764 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1aa006ae00 0x7f1aa006d2c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009d00 tx=0x7f1ab4008040 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:39.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.174+0000 7f1aaeffd700 1 -- 172.21.15.189:0/1860203764 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1ab0055ee0 con 0x7f1ac00d17a0 2024-04-06T12:13:39.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.360+0000 7f1aaeffd700 1 -- 172.21.15.189:0/1860203764 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 11) v1 ==== 89321+0+0 (secure 0 0 0) 0x7f1ab008f070 con 0x7f1ac00d17a0 2024-04-06T12:13:39.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.397+0000 7f1ac49cf700 1 -- 172.21.15.189:0/1860203764 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir7", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ac00d1b80 con 0x7f1aa006ae00 2024-04-06T12:13:39.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.463+0000 7f1aaeffd700 1 -- 172.21.15.189:0/1860203764 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f1ab4009e40 con 0x7f1aa006ae00 2024-04-06T12:13:39.461 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path /snap_test_dir1/dir7 2024-04-06T12:13:39.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.465+0000 7f1ac49cf700 1 -- 172.21.15.189:0/1860203764 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1aa006ae00 msgr2=0x7f1aa006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:39.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.465+0000 7f1ac49cf700 1 --2- 172.21.15.189:0/1860203764 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1aa006ae00 0x7f1aa006d2c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009d00 tx=0x7f1ab4008040 comp rx=0 tx=0).stop 2024-04-06T12:13:39.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.465+0000 7f1ac49cf700 1 -- 172.21.15.189:0/1860203764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac00d17a0 msgr2=0x7f1ac0157840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:39.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.465+0000 7f1ac49cf700 1 --2- 172.21.15.189:0/1860203764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac00d17a0 0x7f1ac0157840 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0009fd0 tx=0x7f1ab0004750 comp rx=0 tx=0).stop 2024-04-06T12:13:39.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.465+0000 7f1ac49cf700 1 -- 172.21.15.189:0/1860203764 shutdown_connections 2024-04-06T12:13:39.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.465+0000 7f1ac49cf700 1 --2- 172.21.15.189:0/1860203764 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac0146dd0 0x7f1ac0157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:39.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.465+0000 7f1ac49cf700 1 --2- 172.21.15.189:0/1860203764 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1aa006ae00 0x7f1aa006d2c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:39.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.465+0000 7f1ac49cf700 1 --2- 172.21.15.189:0/1860203764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ac00cedf0 0x7f1ac01560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:39.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.466+0000 7f1ac49cf700 1 --2- 172.21.15.189:0/1860203764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac00d17a0 0x7f1ac0157840 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:39.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.466+0000 7f1ac49cf700 1 -- 172.21.15.189:0/1860203764 >> 172.21.15.189:0/1860203764 conn(0x7f1ac00b2db0 msgr2=0x7f1ac015dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:39.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.466+0000 7f1ac49cf700 1 -- 172.21.15.189:0/1860203764 shutdown_connections 2024-04-06T12:13:39.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.466+0000 7f1ac49cf700 1 -- 172.21.15.189:0/1860203764 wait complete. 2024-04-06T12:13:39.476 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2024-04-06T12:13:39.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:39.476 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2024-04-06T12:13:39.494 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir8 1m 2024-04-06T12:13:39.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.662+0000 7f5a81331700 1 Processor -- start 2024-04-06T12:13:39.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.662+0000 7f5a81331700 1 -- start start 2024-04-06T12:13:39.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.663+0000 7f5a81331700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a7c06ae40 0x7f5a7c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:39.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.663+0000 7f5a81331700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5a7c15ddf0 0x7f5a7c15e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:39.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.663+0000 7f5a81331700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7c0b3990 con 0x7f5a7c15d7b0 2024-04-06T12:13:39.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.663+0000 7f5a81331700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a7c0b3060 con 0x7f5a7c15ddf0 2024-04-06T12:13:39.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.663+0000 7f5a81331700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c0b3340 con 0x7f5a7c06ae40 2024-04-06T12:13:39.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.663+0000 7f5a7a7fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a7c06ae40 0x7f5a7c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:39.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.663+0000 7f5a7b7fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5a7c15ddf0 0x7f5a7c15e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:39.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.663+0000 7f5a7a7fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a7c06ae40 0x7f5a7c0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:58118/0 (socket says 172.21.15.189:58118) 2024-04-06T12:13:39.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.663+0000 7f5a7affd700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f5a7c15d7b0 0x7f5a7c06a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:53786/0 (socket says 172.21.15.189:53786) 2024-04-06T12:13:39.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.663+0000 7f5a7a7fc700 1 -- 172.21.15.189:0/2061898671 learned_addr learned my addr 172.21.15.189:0/2061898671 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:39.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.663+0000 7f5a79ffb700 1 -- 172.21.15.189:0/2061898671 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399858646 0 0) 0x7f5a7c0b3990 con 0x7f5a7c15d7b0 2024-04-06T12:13:39.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.664+0000 7f5a79ffb700 1 -- 172.21.15.189:0/2061898671 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a640035b0 con 0x7f5a7c15d7b0 2024-04-06T12:13:39.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.664+0000 7f5a7a7fc700 1 -- 172.21.15.189:0/2061898671 >> v2:172.21.15.192:3300/0 conn(0x7f5a7c15ddf0 msgr2=0x7f5a7c15e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:39.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.664+0000 7f5a7a7fc700 1 --2- 172.21.15.189:0/2061898671 >> v2:172.21.15.192:3300/0 conn(0x7f5a7c15ddf0 0x7f5a7c15e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:39.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.664+0000 7f5a7a7fc700 1 -- 172.21.15.189:0/2061898671 >> v1:172.21.15.192:6789/0 conn(0x7f5a7c15d7b0 legacy=0x7f5a7c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:39.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.664+0000 7f5a7a7fc700 1 -- 172.21.15.189:0/2061898671 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7c0b3620 con 0x7f5a7c06ae40 2024-04-06T12:13:39.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.664+0000 7f5a7b7fe700 1 --2- 172.21.15.189:0/2061898671 >> v2:172.21.15.192:3300/0 conn(0x7f5a7c15ddf0 0x7f5a7c15e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:13:39.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.664+0000 7f5a7a7fc700 1 --2- 172.21.15.189:0/2061898671 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a7c06ae40 0x7f5a7c0b3e20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c0098f0 tx=0x7f5a6c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e731b9aec667b739 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:39.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.664+0000 7f5a79ffb700 1 -- 172.21.15.189:0/2061898671 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c011830 con 0x7f5a7c06ae40 2024-04-06T12:13:39.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.664+0000 7f5a79ffb700 1 -- 172.21.15.189:0/2061898671 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6c011990 con 0x7f5a7c06ae40 2024-04-06T12:13:39.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.664+0000 7f5a79ffb700 1 -- 172.21.15.189:0/2061898671 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c011d00 con 0x7f5a7c06ae40 2024-04-06T12:13:39.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.664+0000 7f5a81331700 1 -- 172.21.15.189:0/2061898671 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a7c06ae40 msgr2=0x7f5a7c0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:39.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.664+0000 7f5a81331700 1 --2- 172.21.15.189:0/2061898671 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a7c06ae40 0x7f5a7c0b3e20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c0098f0 tx=0x7f5a6c009c00 comp rx=0 tx=0).stop 2024-04-06T12:13:39.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.664+0000 7f5a81331700 1 -- 172.21.15.189:0/2061898671 shutdown_connections 2024-04-06T12:13:39.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.664+0000 7f5a81331700 1 --2- 172.21.15.189:0/2061898671 >> v2:172.21.15.192:3300/0 conn(0x7f5a7c15ddf0 0x7f5a7c15e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:39.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.664+0000 7f5a81331700 1 --2- 172.21.15.189:0/2061898671 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a7c06ae40 0x7f5a7c0b3e20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:39.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.664+0000 7f5a81331700 1 -- 172.21.15.189:0/2061898671 >> 172.21.15.189:0/2061898671 conn(0x7f5a7c0d00b0 msgr2=0x7f5a7c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:39.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.664+0000 7f5a81331700 1 -- 172.21.15.189:0/2061898671 shutdown_connections 2024-04-06T12:13:39.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.665+0000 7f5a81331700 1 -- 172.21.15.189:0/2061898671 wait complete. 2024-04-06T12:13:39.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.666+0000 7f5a81331700 1 Processor -- start 2024-04-06T12:13:39.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.666+0000 7f5a81331700 1 -- start start 2024-04-06T12:13:39.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a81331700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a7c06ae40 0x7f5a7c153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:39.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a81331700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a7c15d7b0 0x7f5a7c160260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:39.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a81331700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a7c15ddf0 0x7f5a7c1479e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:39.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a81331700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c154e20 con 0x7f5a7c15d7b0 2024-04-06T12:13:39.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a81331700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c1541a0 con 0x7f5a7c15ddf0 2024-04-06T12:13:39.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a81331700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c154480 con 0x7f5a7c06ae40 2024-04-06T12:13:39.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a7a7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a7c15d7b0 0x7f5a7c160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:39.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a7a7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a7c15d7b0 0x7f5a7c160260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57598/0 (socket says 172.21.15.189:57598) 2024-04-06T12:13:39.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a7a7fc700 1 -- 172.21.15.189:0/2412172111 learned_addr learned my addr 172.21.15.189:0/2412172111 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:39.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a7b7fe700 1 --2- 172.21.15.189:0/2412172111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a7c15ddf0 0x7f5a7c1479e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:39.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a7affd700 1 --2- 172.21.15.189:0/2412172111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a7c06ae40 0x7f5a7c153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:39.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a7a7fc700 1 -- 172.21.15.189:0/2412172111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a7c06ae40 msgr2=0x7f5a7c153870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:39.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a7a7fc700 1 --2- 172.21.15.189:0/2412172111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a7c06ae40 0x7f5a7c153870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:39.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a7a7fc700 1 -- 172.21.15.189:0/2412172111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a7c15ddf0 msgr2=0x7f5a7c1479e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:39.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a7a7fc700 1 --2- 172.21.15.189:0/2412172111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a7c15ddf0 0x7f5a7c1479e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:39.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a7a7fc700 1 -- 172.21.15.189:0/2412172111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a74009a70 con 0x7f5a7c15d7b0 2024-04-06T12:13:39.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a7a7fc700 1 --2- 172.21.15.189:0/2412172111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a7c15d7b0 0x7f5a7c160260 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009c40 tx=0x7f5a6c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:39.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a637fe700 1 -- 172.21.15.189:0/2412172111 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c0045d0 con 0x7f5a7c15d7b0 2024-04-06T12:13:39.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a637fe700 1 -- 172.21.15.189:0/2412172111 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6c0036a0 con 0x7f5a7c15d7b0 2024-04-06T12:13:39.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a81331700 1 -- 172.21.15.189:0/2412172111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6c009550 con 0x7f5a7c15d7b0 2024-04-06T12:13:39.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a637fe700 1 -- 172.21.15.189:0/2412172111 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c003a30 con 0x7f5a7c15d7b0 2024-04-06T12:13:39.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.667+0000 7f5a81331700 1 -- 172.21.15.189:0/2412172111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a7c154a40 con 0x7f5a7c15d7b0 2024-04-06T12:13:39.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.668+0000 7f5a81331700 1 -- 172.21.15.189:0/2412172111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7c067bb0 con 0x7f5a7c15d7b0 2024-04-06T12:13:39.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.669+0000 7f5a637fe700 1 -- 172.21.15.189:0/2412172111 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 11) v1 ==== 89321+0+0 (secure 0 0 0) 0x7f5a6c021070 con 0x7f5a7c15d7b0 2024-04-06T12:13:39.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.669+0000 7f5a637fe700 1 --2- 172.21.15.189:0/2412172111 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5a6806ae90 0x7f5a6806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:39.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.669+0000 7f5a637fe700 1 -- 172.21.15.189:0/2412172111 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f5a6c08ad20 con 0x7f5a7c15d7b0 2024-04-06T12:13:39.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.669+0000 7f5a7affd700 1 --2- 172.21.15.189:0/2412172111 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5a6806ae90 0x7f5a6806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:39.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.670+0000 7f5a7affd700 1 --2- 172.21.15.189:0/2412172111 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5a6806ae90 0x7f5a6806d350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5a70007fd0 tx=0x7f5a70007d80 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:39.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.674+0000 7f5a637fe700 1 -- 172.21.15.189:0/2412172111 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5a6c055ad0 con 0x7f5a7c15d7b0 2024-04-06T12:13:39.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.891+0000 7f5a81331700 1 -- 172.21.15.189:0/2412172111 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir8", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a7c15e7e0 con 0x7f5a6806ae90 2024-04-06T12:13:39.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.963+0000 7f5a637fe700 1 -- 172.21.15.189:0/2412172111 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f5a70004ad0 con 0x7f5a6806ae90 2024-04-06T12:13:39.961 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path /snap_test_dir1/dir8 2024-04-06T12:13:39.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.965+0000 7f5a81331700 1 -- 172.21.15.189:0/2412172111 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5a6806ae90 msgr2=0x7f5a6806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:39.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.965+0000 7f5a81331700 1 --2- 172.21.15.189:0/2412172111 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5a6806ae90 0x7f5a6806d350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5a70007fd0 tx=0x7f5a70007d80 comp rx=0 tx=0).stop 2024-04-06T12:13:39.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.965+0000 7f5a81331700 1 -- 172.21.15.189:0/2412172111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a7c15d7b0 msgr2=0x7f5a7c160260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:39.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.965+0000 7f5a81331700 1 --2- 172.21.15.189:0/2412172111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a7c15d7b0 0x7f5a7c160260 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009c40 tx=0x7f5a6c004750 comp rx=0 tx=0).stop 2024-04-06T12:13:39.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.965+0000 7f5a81331700 1 -- 172.21.15.189:0/2412172111 shutdown_connections 2024-04-06T12:13:39.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.965+0000 7f5a81331700 1 --2- 172.21.15.189:0/2412172111 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5a6806ae90 0x7f5a6806d350 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:39.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.965+0000 7f5a81331700 1 --2- 172.21.15.189:0/2412172111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a7c15ddf0 0x7f5a7c1479e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:39.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.965+0000 7f5a81331700 1 --2- 172.21.15.189:0/2412172111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a7c06ae40 0x7f5a7c153870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:39.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.965+0000 7f5a81331700 1 --2- 172.21.15.189:0/2412172111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a7c15d7b0 0x7f5a7c160260 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:39.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.965+0000 7f5a81331700 1 -- 172.21.15.189:0/2412172111 >> 172.21.15.189:0/2412172111 conn(0x7f5a7c0d00b0 msgr2=0x7f5a7c06eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:39.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.965+0000 7f5a81331700 1 -- 172.21.15.189:0/2412172111 shutdown_connections 2024-04-06T12:13:39.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:39.965+0000 7f5a81331700 1 -- 172.21.15.189:0/2412172111 wait complete. 2024-04-06T12:13:39.975 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2024-04-06T12:13:39.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:13:39.975 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2024-04-06T12:13:39.994 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir9 1m 2024-04-06T12:13:40.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.160+0000 7f2c8d440700 1 Processor -- start 2024-04-06T12:13:40.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.160+0000 7f2c8d440700 1 -- start start 2024-04-06T12:13:40.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.161+0000 7f2c8d440700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2c8815d7b0 0x7f2c880b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:40.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.161+0000 7f2c8d440700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c8806a730 0x7f2c8806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:40.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.161+0000 7f2c8d440700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c880b3990 con 0x7f2c8815ddf0 2024-04-06T12:13:40.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.161+0000 7f2c8d440700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c8806b0f0 con 0x7f2c8815d7b0 2024-04-06T12:13:40.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.161+0000 7f2c8d440700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c880b3060 con 0x7f2c8806a730 2024-04-06T12:13:40.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.161+0000 7f2c86ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2c8815d7b0 0x7f2c880b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:40.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.161+0000 7f2c86ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2c8815d7b0 0x7f2c880b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57600/0 (socket says 172.21.15.189:57600) 2024-04-06T12:13:40.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.161+0000 7f2c86ffd700 1 -- 172.21.15.189:0/652705178 learned_addr learned my addr 172.21.15.189:0/652705178 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:40.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.162+0000 7f2c867fc700 1 --2- 172.21.15.189:0/652705178 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c8806a730 0x7f2c8806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:40.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.162+0000 7f2c86ffd700 1 -- 172.21.15.189:0/652705178 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c8806a730 msgr2=0x7f2c8806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:40.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.162+0000 7f2c86ffd700 1 --2- 172.21.15.189:0/652705178 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c8806a730 0x7f2c8806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.162+0000 7f2c86ffd700 1 -- 172.21.15.189:0/652705178 >> v1:172.21.15.192:6789/0 conn(0x7f2c8815ddf0 legacy=0x7f2c8815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:40.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.162+0000 7f2c86ffd700 1 -- 172.21.15.189:0/652705178 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c880b3340 con 0x7f2c8815d7b0 2024-04-06T12:13:40.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.162+0000 7f2c86ffd700 1 --2- 172.21.15.189:0/652705178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c8815d7b0 0x7f2c880b3e20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c0098f0 tx=0x7f2c7c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b78ba37f866102 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:40.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.162+0000 7f2c85ffb700 1 -- 172.21.15.189:0/652705178 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c004230 con 0x7f2c8815d7b0 2024-04-06T12:13:40.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.162+0000 7f2c85ffb700 1 -- 172.21.15.189:0/652705178 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7c004390 con 0x7f2c8815d7b0 2024-04-06T12:13:40.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.162+0000 7f2c85ffb700 1 -- 172.21.15.189:0/652705178 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c0036a0 con 0x7f2c8815d7b0 2024-04-06T12:13:40.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.163+0000 7f2c8d440700 1 -- 172.21.15.189:0/652705178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c8815d7b0 msgr2=0x7f2c880b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:40.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.163+0000 7f2c8d440700 1 --2- 172.21.15.189:0/652705178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c8815d7b0 0x7f2c880b3e20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c0098f0 tx=0x7f2c7c009c00 comp rx=0 tx=0).stop 2024-04-06T12:13:40.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.163+0000 7f2c8d440700 1 -- 172.21.15.189:0/652705178 shutdown_connections 2024-04-06T12:13:40.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.163+0000 7f2c8d440700 1 --2- 172.21.15.189:0/652705178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c8815d7b0 0x7f2c880b3e20 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.163+0000 7f2c8d440700 1 --2- 172.21.15.189:0/652705178 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c8806a730 0x7f2c8806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.163+0000 7f2c8d440700 1 -- 172.21.15.189:0/652705178 >> 172.21.15.189:0/652705178 conn(0x7f2c880d00b0 msgr2=0x7f2c880d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:40.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.163+0000 7f2c8d440700 1 -- 172.21.15.189:0/652705178 shutdown_connections 2024-04-06T12:13:40.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.163+0000 7f2c8d440700 1 -- 172.21.15.189:0/652705178 wait complete. 2024-04-06T12:13:40.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.164+0000 7f2c8d440700 1 Processor -- start 2024-04-06T12:13:40.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.164+0000 7f2c8d440700 1 -- start start 2024-04-06T12:13:40.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c8d440700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c8806a730 0x7f2c881517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:40.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c8d440700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c8815d7b0 0x7f2c880b0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:40.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c8d440700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c8815ddf0 0x7f2c880b10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:40.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c8d440700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c88152210 con 0x7f2c8815d7b0 2024-04-06T12:13:40.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c8d440700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c88151ce0 con 0x7f2c8806a730 2024-04-06T12:13:40.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c8d440700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c88158470 con 0x7f2c8815ddf0 2024-04-06T12:13:40.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c867fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c8815d7b0 0x7f2c880b0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:40.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c867fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c8815d7b0 0x7f2c880b0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57602/0 (socket says 172.21.15.189:57602) 2024-04-06T12:13:40.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c867fc700 1 -- 172.21.15.189:0/1594682713 learned_addr learned my addr 172.21.15.189:0/1594682713 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:40.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c86ffd700 1 --2- 172.21.15.189:0/1594682713 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c8806a730 0x7f2c881517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:40.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c877fe700 1 --2- 172.21.15.189:0/1594682713 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c8815ddf0 0x7f2c880b10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:40.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c867fc700 1 -- 172.21.15.189:0/1594682713 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c8815ddf0 msgr2=0x7f2c880b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:40.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c867fc700 1 --2- 172.21.15.189:0/1594682713 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c8815ddf0 0x7f2c880b10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c867fc700 1 -- 172.21.15.189:0/1594682713 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c8806a730 msgr2=0x7f2c881517a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:40.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c867fc700 1 --2- 172.21.15.189:0/1594682713 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c8806a730 0x7f2c881517a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c867fc700 1 -- 172.21.15.189:0/1594682713 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c009550 con 0x7f2c8815d7b0 2024-04-06T12:13:40.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c86ffd700 1 --2- 172.21.15.189:0/1594682713 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c8806a730 0x7f2c881517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:40.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c867fc700 1 --2- 172.21.15.189:0/1594682713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c8815d7b0 0x7f2c880b0b60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2c7800c7f0 tx=0x7f2c7800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:40.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c6f7fe700 1 -- 172.21.15.189:0/1594682713 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c78011800 con 0x7f2c8815d7b0 2024-04-06T12:13:40.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c6f7fe700 1 -- 172.21.15.189:0/1594682713 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c78011960 con 0x7f2c8815d7b0 2024-04-06T12:13:40.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c8d440700 1 -- 172.21.15.189:0/1594682713 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c88158710 con 0x7f2c8815d7b0 2024-04-06T12:13:40.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c6f7fe700 1 -- 172.21.15.189:0/1594682713 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c78011cf0 con 0x7f2c8815d7b0 2024-04-06T12:13:40.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.165+0000 7f2c8d440700 1 -- 172.21.15.189:0/1594682713 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c88158cd0 con 0x7f2c8815d7b0 2024-04-06T12:13:40.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.166+0000 7f2c8d440700 1 -- 172.21.15.189:0/1594682713 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c88067bb0 con 0x7f2c8815d7b0 2024-04-06T12:13:40.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.167+0000 7f2c6f7fe700 1 -- 172.21.15.189:0/1594682713 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 11) v1 ==== 89321+0+0 (secure 0 0 0) 0x7f2c78018360 con 0x7f2c8815d7b0 2024-04-06T12:13:40.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.167+0000 7f2c6f7fe700 1 --2- 172.21.15.189:0/1594682713 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2c7406ae90 0x7f2c7406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:40.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.167+0000 7f2c6f7fe700 1 -- 172.21.15.189:0/1594682713 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f2c78089a50 con 0x7f2c8815d7b0 2024-04-06T12:13:40.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.167+0000 7f2c86ffd700 1 --2- 172.21.15.189:0/1594682713 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2c7406ae90 0x7f2c7406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:40.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.168+0000 7f2c86ffd700 1 --2- 172.21.15.189:0/1594682713 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2c7406ae90 0x7f2c7406d350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c009c40 tx=0x7f2c7c008030 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:40.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.172+0000 7f2c6f7fe700 1 -- 172.21.15.189:0/1594682713 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2c780545d0 con 0x7f2c8815d7b0 2024-04-06T12:13:40.321 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:13:40.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:13:40.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.394+0000 7f2c8d440700 1 -- 172.21.15.189:0/1594682713 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir9", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c8815eb40 con 0x7f2c7406ae90 2024-04-06T12:13:40.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.473+0000 7f2c6f7fe700 1 -- 172.21.15.189:0/1594682713 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f2c7c020070 con 0x7f2c7406ae90 2024-04-06T12:13:40.471 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path /snap_test_dir1/dir9 2024-04-06T12:13:40.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.475+0000 7f2c8d440700 1 -- 172.21.15.189:0/1594682713 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2c7406ae90 msgr2=0x7f2c7406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:40.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.475+0000 7f2c8d440700 1 --2- 172.21.15.189:0/1594682713 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2c7406ae90 0x7f2c7406d350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c009c40 tx=0x7f2c7c008030 comp rx=0 tx=0).stop 2024-04-06T12:13:40.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.475+0000 7f2c8d440700 1 -- 172.21.15.189:0/1594682713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c8815d7b0 msgr2=0x7f2c880b0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:40.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.475+0000 7f2c8d440700 1 --2- 172.21.15.189:0/1594682713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c8815d7b0 0x7f2c880b0b60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2c7800c7f0 tx=0x7f2c7800cbb0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.475+0000 7f2c8d440700 1 -- 172.21.15.189:0/1594682713 shutdown_connections 2024-04-06T12:13:40.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.475+0000 7f2c8d440700 1 --2- 172.21.15.189:0/1594682713 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c8815ddf0 0x7f2c880b10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.475+0000 7f2c8d440700 1 --2- 172.21.15.189:0/1594682713 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2c7406ae90 0x7f2c7406d350 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.475+0000 7f2c8d440700 1 --2- 172.21.15.189:0/1594682713 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c8806a730 0x7f2c881517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.475+0000 7f2c8d440700 1 --2- 172.21.15.189:0/1594682713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c8815d7b0 0x7f2c880b0b60 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.475+0000 7f2c8d440700 1 -- 172.21.15.189:0/1594682713 >> 172.21.15.189:0/1594682713 conn(0x7f2c880d00b0 msgr2=0x7f2c88071c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:40.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.476+0000 7f2c8d440700 1 -- 172.21.15.189:0/1594682713 shutdown_connections 2024-04-06T12:13:40.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.476+0000 7f2c8d440700 1 -- 172.21.15.189:0/1594682713 wait complete. 2024-04-06T12:13:40.485 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2024-04-06T12:13:40.486 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir0 1m 2024-04-06T12:13:40.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.653+0000 7f34435b8700 1 Processor -- start 2024-04-06T12:13:40.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.653+0000 7f34435b8700 1 -- start start 2024-04-06T12:13:40.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.654+0000 7f34435b8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f343c15d7b0 0x7f343c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:40.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.654+0000 7f34435b8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f343c15e500 0x7f343c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:40.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.654+0000 7f34435b8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343c0b3990 con 0x7f343c06a730 2024-04-06T12:13:40.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.654+0000 7f34435b8700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343c06b0f0 con 0x7f343c15d7b0 2024-04-06T12:13:40.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.654+0000 7f34435b8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c0b3060 con 0x7f343c15e500 2024-04-06T12:13:40.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.654+0000 7f3441354700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f343c15d7b0 0x7f343c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:40.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.654+0000 7f3441b55700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f343c15e500 0x7f343c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:40.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.654+0000 7f3441354700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f343c15d7b0 0x7f343c0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:57128/0 (socket says 172.21.15.189:57128) 2024-04-06T12:13:40.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.654+0000 7f3441b55700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f343c15e500 0x7f343c06abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:58166/0 (socket says 172.21.15.189:58166) 2024-04-06T12:13:40.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.655+0000 7f3441354700 1 -- 172.21.15.189:0/2883427961 learned_addr learned my addr 172.21.15.189:0/2883427961 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:40.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.655+0000 7f3440b53700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f343c06a730 0x7f343c15ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:53790/0 (socket says 172.21.15.189:53790) 2024-04-06T12:13:40.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.655+0000 7f3433fff700 1 -- 172.21.15.189:0/2883427961 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282866049 0 0) 0x7f343c0b3990 con 0x7f343c06a730 2024-04-06T12:13:40.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.655+0000 7f3433fff700 1 -- 172.21.15.189:0/2883427961 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34280035b0 con 0x7f343c06a730 2024-04-06T12:13:40.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.655+0000 7f3441b55700 1 -- 172.21.15.189:0/2883427961 >> v2:172.21.15.192:3300/0 conn(0x7f343c15d7b0 msgr2=0x7f343c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:40.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.655+0000 7f3441b55700 1 --2- 172.21.15.189:0/2883427961 >> v2:172.21.15.192:3300/0 conn(0x7f343c15d7b0 0x7f343c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.655+0000 7f3441b55700 1 -- 172.21.15.189:0/2883427961 >> v1:172.21.15.192:6789/0 conn(0x7f343c06a730 legacy=0x7f343c15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:40.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.655+0000 7f3441b55700 1 -- 172.21.15.189:0/2883427961 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343c0b3340 con 0x7f343c15e500 2024-04-06T12:13:40.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.655+0000 7f3441354700 1 --2- 172.21.15.189:0/2883427961 >> v2:172.21.15.192:3300/0 conn(0x7f343c15d7b0 0x7f343c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:13:40.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.655+0000 7f3441b55700 1 --2- 172.21.15.189:0/2883427961 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f343c15e500 0x7f343c06abb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3434009d90 tx=0x7f343400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13a95c91d953ce06 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:40.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.655+0000 7f3433fff700 1 -- 172.21.15.189:0/2883427961 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343400e070 con 0x7f343c15e500 2024-04-06T12:13:40.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.655+0000 7f3433fff700 1 -- 172.21.15.189:0/2883427961 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343400bb20 con 0x7f343c15e500 2024-04-06T12:13:40.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.655+0000 7f3433fff700 1 -- 172.21.15.189:0/2883427961 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343400be90 con 0x7f343c15e500 2024-04-06T12:13:40.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.656+0000 7f34435b8700 1 -- 172.21.15.189:0/2883427961 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f343c15e500 msgr2=0x7f343c06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:40.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.656+0000 7f34435b8700 1 --2- 172.21.15.189:0/2883427961 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f343c15e500 0x7f343c06abb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3434009d90 tx=0x7f343400b330 comp rx=0 tx=0).stop 2024-04-06T12:13:40.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.656+0000 7f34435b8700 1 -- 172.21.15.189:0/2883427961 shutdown_connections 2024-04-06T12:13:40.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.656+0000 7f34435b8700 1 --2- 172.21.15.189:0/2883427961 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f343c15e500 0x7f343c06abb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.656+0000 7f34435b8700 1 --2- 172.21.15.189:0/2883427961 >> v2:172.21.15.192:3300/0 conn(0x7f343c15d7b0 0x7f343c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.656+0000 7f34435b8700 1 -- 172.21.15.189:0/2883427961 >> 172.21.15.189:0/2883427961 conn(0x7f343c0d00b0 msgr2=0x7f343c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:40.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.656+0000 7f34435b8700 1 -- 172.21.15.189:0/2883427961 shutdown_connections 2024-04-06T12:13:40.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.656+0000 7f34435b8700 1 -- 172.21.15.189:0/2883427961 wait complete. 2024-04-06T12:13:40.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.657+0000 7f34435b8700 1 Processor -- start 2024-04-06T12:13:40.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.657+0000 7f34435b8700 1 -- start start 2024-04-06T12:13:40.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.658+0000 7f34435b8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f343c06a730 0x7f343c15cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:40.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.658+0000 7f34435b8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f343c15d7b0 0x7f343c14e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:40.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.658+0000 7f34435b8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f343c15e500 0x7f343c0cda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:40.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.658+0000 7f34435b8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c14ecb0 con 0x7f343c15d7b0 2024-04-06T12:13:40.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.658+0000 7f34435b8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c0cf390 con 0x7f343c15e500 2024-04-06T12:13:40.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.658+0000 7f34435b8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c0cc9b0 con 0x7f343c06a730 2024-04-06T12:13:40.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.658+0000 7f3440b53700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f343c15d7b0 0x7f343c14e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:40.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.658+0000 7f3440b53700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f343c15d7b0 0x7f343c14e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57606/0 (socket says 172.21.15.189:57606) 2024-04-06T12:13:40.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.658+0000 7f3440b53700 1 -- 172.21.15.189:0/2855301462 learned_addr learned my addr 172.21.15.189:0/2855301462 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:40.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.658+0000 7f3441b55700 1 --2- 172.21.15.189:0/2855301462 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f343c15e500 0x7f343c0cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:40.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.658+0000 7f3441354700 1 --2- 172.21.15.189:0/2855301462 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f343c06a730 0x7f343c15cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:40.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.658+0000 7f3440b53700 1 -- 172.21.15.189:0/2855301462 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f343c06a730 msgr2=0x7f343c15cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:40.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.658+0000 7f3440b53700 1 --2- 172.21.15.189:0/2855301462 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f343c06a730 0x7f343c15cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.658+0000 7f3440b53700 1 -- 172.21.15.189:0/2855301462 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f343c15e500 msgr2=0x7f343c0cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:40.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.658+0000 7f3440b53700 1 --2- 172.21.15.189:0/2855301462 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f343c15e500 0x7f343c0cda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.658+0000 7f3440b53700 1 -- 172.21.15.189:0/2855301462 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342c009550 con 0x7f343c15d7b0 2024-04-06T12:13:40.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.658+0000 7f3441354700 1 --2- 172.21.15.189:0/2855301462 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f343c06a730 0x7f343c15cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.658+0000 7f3440b53700 1 --2- 172.21.15.189:0/2855301462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f343c15d7b0 0x7f343c14e770 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f343800c500 tx=0x7f343800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.659+0000 7f3431ffb700 1 -- 172.21.15.189:0/2855301462 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438005cb0 con 0x7f343c15d7b0 2024-04-06T12:13:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.659+0000 7f3431ffb700 1 -- 172.21.15.189:0/2855301462 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3438005e10 con 0x7f343c15d7b0 2024-04-06T12:13:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.659+0000 7f34435b8700 1 -- 172.21.15.189:0/2855301462 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3434009a70 con 0x7f343c15d7b0 2024-04-06T12:13:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.659+0000 7f3431ffb700 1 -- 172.21.15.189:0/2855301462 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438007310 con 0x7f343c15d7b0 2024-04-06T12:13:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.659+0000 7f34435b8700 1 -- 172.21.15.189:0/2855301462 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f343c0ccd40 con 0x7f343c15d7b0 2024-04-06T12:13:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.660+0000 7f34435b8700 1 -- 172.21.15.189:0/2855301462 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343c067bb0 con 0x7f343c15d7b0 2024-04-06T12:13:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.660+0000 7f3431ffb700 1 -- 172.21.15.189:0/2855301462 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 11) v1 ==== 89321+0+0 (secure 0 0 0) 0x7f3438007470 con 0x7f343c15d7b0 2024-04-06T12:13:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.660+0000 7f3431ffb700 1 --2- 172.21.15.189:0/2855301462 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f342006ae90 0x7f342006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.661+0000 7f3431ffb700 1 -- 172.21.15.189:0/2855301462 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f3438088e90 con 0x7f343c15d7b0 2024-04-06T12:13:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.661+0000 7f3441354700 1 --2- 172.21.15.189:0/2855301462 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f342006ae90 0x7f342006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.661+0000 7f3441354700 1 --2- 172.21.15.189:0/2855301462 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f342006ae90 0x7f342006d350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f342c009ce0 tx=0x7f342c008040 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:40.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.665+0000 7f3431ffb700 1 -- 172.21.15.189:0/2855301462 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3438053c70 con 0x7f343c15d7b0 2024-04-06T12:13:40.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.889+0000 7f34435b8700 1 -- 172.21.15.189:0/2855301462 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f343c074990 con 0x7f342006ae90 2024-04-06T12:13:40.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.946+0000 7f3431ffb700 1 -- 172.21.15.189:0/2855301462 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f343c074990 con 0x7f342006ae90 2024-04-06T12:13:40.944 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule activated for path /snap_test_dir1/dir0 2024-04-06T12:13:40.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.948+0000 7f34435b8700 1 -- 172.21.15.189:0/2855301462 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f342006ae90 msgr2=0x7f342006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:40.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.948+0000 7f34435b8700 1 --2- 172.21.15.189:0/2855301462 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f342006ae90 0x7f342006d350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f342c009ce0 tx=0x7f342c008040 comp rx=0 tx=0).stop 2024-04-06T12:13:40.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.948+0000 7f34435b8700 1 -- 172.21.15.189:0/2855301462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f343c15d7b0 msgr2=0x7f343c14e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:40.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.948+0000 7f34435b8700 1 --2- 172.21.15.189:0/2855301462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f343c15d7b0 0x7f343c14e770 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f343800c500 tx=0x7f343800c8c0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.948+0000 7f34435b8700 1 -- 172.21.15.189:0/2855301462 shutdown_connections 2024-04-06T12:13:40.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.948+0000 7f34435b8700 1 --2- 172.21.15.189:0/2855301462 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f342006ae90 0x7f342006d350 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.948+0000 7f34435b8700 1 --2- 172.21.15.189:0/2855301462 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f343c15e500 0x7f343c0cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.948+0000 7f34435b8700 1 --2- 172.21.15.189:0/2855301462 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f343c06a730 0x7f343c15cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.948+0000 7f34435b8700 1 --2- 172.21.15.189:0/2855301462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f343c15d7b0 0x7f343c14e770 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:40.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.948+0000 7f34435b8700 1 -- 172.21.15.189:0/2855301462 >> 172.21.15.189:0/2855301462 conn(0x7f343c0d00b0 msgr2=0x7f343c159510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:40.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.949+0000 7f34435b8700 1 -- 172.21.15.189:0/2855301462 shutdown_connections 2024-04-06T12:13:40.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:40.949+0000 7f34435b8700 1 -- 172.21.15.189:0/2855301462 wait complete. 2024-04-06T12:13:40.958 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2024-04-06T12:13:40.959 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir1 1m 2024-04-06T12:13:41.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.085+0000 7fb61388d700 1 Processor -- start 2024-04-06T12:13:41.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.085+0000 7fb61388d700 1 -- start start 2024-04-06T12:13:41.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.086+0000 7fb61388d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb60c15a550 0x7fb60c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:41.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.086+0000 7fb61388d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb60c0cf900 0x7fb60c0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:41.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.086+0000 7fb61388d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb60c0cdbe0 con 0x7fb60c0b4ba0 2024-04-06T12:13:41.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.086+0000 7fb61388d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb60c0b5010 con 0x7fb60c15a550 2024-04-06T12:13:41.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.086+0000 7fb61388d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c15a940 con 0x7fb60c0cf900 2024-04-06T12:13:41.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.086+0000 7fb611629700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb60c15a550 0x7fb60c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:41.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.086+0000 7fb611629700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb60c15a550 0x7fb60c0cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57610/0 (socket says 172.21.15.189:57610) 2024-04-06T12:13:41.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.086+0000 7fb611629700 1 -- 172.21.15.189:0/2324521375 learned_addr learned my addr 172.21.15.189:0/2324521375 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:41.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.086+0000 7fb610e28700 1 --2- 172.21.15.189:0/2324521375 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb60c0cf900 0x7fb60c0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:41.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.086+0000 7fb611629700 1 -- 172.21.15.189:0/2324521375 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb60c0cf900 msgr2=0x7fb60c0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:41.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.086+0000 7fb611629700 1 --2- 172.21.15.189:0/2324521375 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb60c0cf900 0x7fb60c0b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.086+0000 7fb611629700 1 -- 172.21.15.189:0/2324521375 >> v1:172.21.15.192:6789/0 conn(0x7fb60c0b4ba0 legacy=0x7fb60c06eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:41.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.086+0000 7fb611629700 1 -- 172.21.15.189:0/2324521375 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60c15ac00 con 0x7fb60c15a550 2024-04-06T12:13:41.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.087+0000 7fb611629700 1 --2- 172.21.15.189:0/2324521375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c15a550 0x7fb60c0cf3c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb60800bb90 tx=0x7fb60800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d937e23c62efca4a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:41.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.087+0000 7fb603fff700 1 -- 172.21.15.189:0/2324521375 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60800dd30 con 0x7fb60c15a550 2024-04-06T12:13:41.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.087+0000 7fb603fff700 1 -- 172.21.15.189:0/2324521375 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb60800de90 con 0x7fb60c15a550 2024-04-06T12:13:41.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.087+0000 7fb603fff700 1 -- 172.21.15.189:0/2324521375 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb608010690 con 0x7fb60c15a550 2024-04-06T12:13:41.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.087+0000 7fb61388d700 1 -- 172.21.15.189:0/2324521375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c15a550 msgr2=0x7fb60c0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:41.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.087+0000 7fb61388d700 1 --2- 172.21.15.189:0/2324521375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c15a550 0x7fb60c0cf3c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb60800bb90 tx=0x7fb60800bea0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.087+0000 7fb61388d700 1 -- 172.21.15.189:0/2324521375 shutdown_connections 2024-04-06T12:13:41.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.087+0000 7fb61388d700 1 --2- 172.21.15.189:0/2324521375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c15a550 0x7fb60c0cf3c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.087+0000 7fb61388d700 1 --2- 172.21.15.189:0/2324521375 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb60c0cf900 0x7fb60c0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.087+0000 7fb61388d700 1 -- 172.21.15.189:0/2324521375 >> 172.21.15.189:0/2324521375 conn(0x7fb60c0d00b0 msgr2=0x7fb60c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:41.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.087+0000 7fb61388d700 1 -- 172.21.15.189:0/2324521375 shutdown_connections 2024-04-06T12:13:41.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.087+0000 7fb61388d700 1 -- 172.21.15.189:0/2324521375 wait complete. 2024-04-06T12:13:41.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.089+0000 7fb61388d700 1 Processor -- start 2024-04-06T12:13:41.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.089+0000 7fb61388d700 1 -- start start 2024-04-06T12:13:41.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.089+0000 7fb61388d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb60c0b4ba0 0x7fb60c14bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:41.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.089+0000 7fb61388d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c0cf900 0x7fb60c14ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:41.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.089+0000 7fb61388d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb60c14d470 0x7fb60c14f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:41.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.089+0000 7fb61388d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c1574b0 con 0x7fb60c0cf900 2024-04-06T12:13:41.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.089+0000 7fb61388d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c14c2f0 con 0x7fb60c14d470 2024-04-06T12:13:41.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.089+0000 7fb61388d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c14fa10 con 0x7fb60c0b4ba0 2024-04-06T12:13:41.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.089+0000 7fb610e28700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c0cf900 0x7fb60c14ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:41.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.089+0000 7fb610e28700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c0cf900 0x7fb60c14ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57624/0 (socket says 172.21.15.189:57624) 2024-04-06T12:13:41.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.089+0000 7fb610e28700 1 -- 172.21.15.189:0/2565079272 learned_addr learned my addr 172.21.15.189:0/2565079272 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:41.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.090+0000 7fb611629700 1 --2- 172.21.15.189:0/2565079272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb60c0b4ba0 0x7fb60c14bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:41.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.090+0000 7fb611e2a700 1 --2- 172.21.15.189:0/2565079272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb60c14d470 0x7fb60c14f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:41.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.090+0000 7fb610e28700 1 -- 172.21.15.189:0/2565079272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb60c0b4ba0 msgr2=0x7fb60c14bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:41.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.090+0000 7fb610e28700 1 --2- 172.21.15.189:0/2565079272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb60c0b4ba0 0x7fb60c14bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.090+0000 7fb610e28700 1 -- 172.21.15.189:0/2565079272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb60c14d470 msgr2=0x7fb60c14f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:41.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.090+0000 7fb610e28700 1 --2- 172.21.15.189:0/2565079272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb60c14d470 0x7fb60c14f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.090+0000 7fb610e28700 1 -- 172.21.15.189:0/2565079272 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60800b7f0 con 0x7fb60c0cf900 2024-04-06T12:13:41.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.090+0000 7fb610e28700 1 --2- 172.21.15.189:0/2565079272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c0cf900 0x7fb60c14ef90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00e9e0 tx=0x7fb5fc00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:41.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.090+0000 7fb601ffb700 1 -- 172.21.15.189:0/2565079272 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc009710 con 0x7fb60c0cf900 2024-04-06T12:13:41.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.090+0000 7fb601ffb700 1 -- 172.21.15.189:0/2565079272 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5fc00ee80 con 0x7fb60c0cf900 2024-04-06T12:13:41.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.090+0000 7fb61388d700 1 -- 172.21.15.189:0/2565079272 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60c14d9e0 con 0x7fb60c0cf900 2024-04-06T12:13:41.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.090+0000 7fb601ffb700 1 -- 172.21.15.189:0/2565079272 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc00ca50 con 0x7fb60c0cf900 2024-04-06T12:13:41.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.090+0000 7fb61388d700 1 -- 172.21.15.189:0/2565079272 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb60c14ce10 con 0x7fb60c0cf900 2024-04-06T12:13:41.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.091+0000 7fb61388d700 1 -- 172.21.15.189:0/2565079272 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb60c1575f0 con 0x7fb60c0cf900 2024-04-06T12:13:41.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.092+0000 7fb601ffb700 1 -- 172.21.15.189:0/2565079272 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 11) v1 ==== 89321+0+0 (secure 0 0 0) 0x7fb5fc00cbb0 con 0x7fb60c0cf900 2024-04-06T12:13:41.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.092+0000 7fb601ffb700 1 --2- 172.21.15.189:0/2565079272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb5f006aec0 0x7fb5f006d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:41.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.092+0000 7fb601ffb700 1 -- 172.21.15.189:0/2565079272 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fb5fc089bd0 con 0x7fb60c0cf900 2024-04-06T12:13:41.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.092+0000 7fb611629700 1 --2- 172.21.15.189:0/2565079272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb5f006aec0 0x7fb5f006d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:41.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.093+0000 7fb611629700 1 --2- 172.21.15.189:0/2565079272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb5f006aec0 0x7fb5f006d380 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb60800bee0 tx=0x7fb608013000 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:41.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.097+0000 7fb601ffb700 1 -- 172.21.15.189:0/2565079272 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb60c1575f0 con 0x7fb60c0cf900 2024-04-06T12:13:41.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.319+0000 7fb61388d700 1 -- 172.21.15.189:0/2565079272 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb60c15b4a0 con 0x7fb5f006aec0 2024-04-06T12:13:41.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.376+0000 7fb601ffb700 1 -- 172.21.15.189:0/2565079272 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fb608020040 con 0x7fb5f006aec0 2024-04-06T12:13:41.374 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule activated for path /snap_test_dir1/dir1 2024-04-06T12:13:41.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.378+0000 7fb61388d700 1 -- 172.21.15.189:0/2565079272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb5f006aec0 msgr2=0x7fb5f006d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:41.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.378+0000 7fb61388d700 1 --2- 172.21.15.189:0/2565079272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb5f006aec0 0x7fb5f006d380 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb60800bee0 tx=0x7fb608013000 comp rx=0 tx=0).stop 2024-04-06T12:13:41.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.378+0000 7fb61388d700 1 -- 172.21.15.189:0/2565079272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c0cf900 msgr2=0x7fb60c14ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:41.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.378+0000 7fb61388d700 1 --2- 172.21.15.189:0/2565079272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c0cf900 0x7fb60c14ef90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00e9e0 tx=0x7fb5fc00ecf0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.378+0000 7fb61388d700 1 -- 172.21.15.189:0/2565079272 shutdown_connections 2024-04-06T12:13:41.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.378+0000 7fb61388d700 1 --2- 172.21.15.189:0/2565079272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb5f006aec0 0x7fb5f006d380 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.378+0000 7fb61388d700 1 --2- 172.21.15.189:0/2565079272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb60c14d470 0x7fb60c14f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.378+0000 7fb61388d700 1 --2- 172.21.15.189:0/2565079272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb60c0b4ba0 0x7fb60c14bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.378+0000 7fb61388d700 1 --2- 172.21.15.189:0/2565079272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c0cf900 0x7fb60c14ef90 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.378+0000 7fb61388d700 1 -- 172.21.15.189:0/2565079272 >> 172.21.15.189:0/2565079272 conn(0x7fb60c0d00b0 msgr2=0x7fb60c0b2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:41.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.379+0000 7fb61388d700 1 -- 172.21.15.189:0/2565079272 shutdown_connections 2024-04-06T12:13:41.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.379+0000 7fb61388d700 1 -- 172.21.15.189:0/2565079272 wait complete. 2024-04-06T12:13:41.389 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2024-04-06T12:13:41.389 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir2 1m 2024-04-06T12:13:41.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.516+0000 7f797a3f0700 1 Processor -- start 2024-04-06T12:13:41.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.516+0000 7f797a3f0700 1 -- start start 2024-04-06T12:13:41.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.517+0000 7f797a3f0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f797415d7b0 0x7f79740b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:41.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.517+0000 7f797a3f0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f797406a730 0x7f797406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:41.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.517+0000 7f797a3f0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79740b3990 con 0x7f797415ddf0 2024-04-06T12:13:41.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.517+0000 7f797a3f0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f797406b0f0 con 0x7f797406a730 2024-04-06T12:13:41.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.517+0000 7f797a3f0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79740b3060 con 0x7f797415d7b0 2024-04-06T12:13:41.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.517+0000 7f79737fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f797406a730 0x7f797406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:41.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.517+0000 7f797898d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f797415ddf0 0x7f797415e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38980/0 (socket says 172.21.15.189:38980) 2024-04-06T12:13:41.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.517+0000 7f797898d700 1 -- 172.21.15.189:0/1910098617 learned_addr learned my addr 172.21.15.189:0/1910098617 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:41.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.517+0000 7f7973fff700 1 --2- 172.21.15.189:0/1910098617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f797415d7b0 0x7f79740b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:41.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.518+0000 7f79737fe700 1 -- 172.21.15.189:0/1910098617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f797415d7b0 msgr2=0x7f79740b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:41.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.518+0000 7f79737fe700 1 --2- 172.21.15.189:0/1910098617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f797415d7b0 0x7f79740b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.518+0000 7f79737fe700 1 -- 172.21.15.189:0/1910098617 >> v1:172.21.15.189:6789/0 conn(0x7f797415ddf0 legacy=0x7f797415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:41.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.518+0000 7f7972ffd700 1 -- 172.21.15.189:0/1910098617 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658487146 0 0) 0x7f79740b3990 con 0x7f797415ddf0 2024-04-06T12:13:41.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.518+0000 7f79737fe700 1 -- 172.21.15.189:0/1910098617 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79740b3340 con 0x7f797406a730 2024-04-06T12:13:41.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.518+0000 7f7973fff700 1 --2- 172.21.15.189:0/1910098617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f797415d7b0 0x7f79740b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:41.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.518+0000 7f79737fe700 1 --2- 172.21.15.189:0/1910098617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797406a730 0x7f797406abb0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f7964009840 tx=0x7f7964009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a17ea4dc7e7570 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:41.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.518+0000 7f7972ffd700 1 -- 172.21.15.189:0/1910098617 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7964011830 con 0x7f797406a730 2024-04-06T12:13:41.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.518+0000 7f7972ffd700 1 -- 172.21.15.189:0/1910098617 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7964011990 con 0x7f797406a730 2024-04-06T12:13:41.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.518+0000 7f7972ffd700 1 -- 172.21.15.189:0/1910098617 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7964011d20 con 0x7f797406a730 2024-04-06T12:13:41.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.518+0000 7f797a3f0700 1 -- 172.21.15.189:0/1910098617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797406a730 msgr2=0x7f797406abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.518+0000 7f797a3f0700 1 --2- 172.21.15.189:0/1910098617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797406a730 0x7f797406abb0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f7964009840 tx=0x7f7964009b50 comp rx=0 tx=0).stop 2024-04-06T12:13:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.518+0000 7f797a3f0700 1 -- 172.21.15.189:0/1910098617 shutdown_connections 2024-04-06T12:13:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.518+0000 7f797a3f0700 1 --2- 172.21.15.189:0/1910098617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f797415d7b0 0x7f79740b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.518+0000 7f797a3f0700 1 --2- 172.21.15.189:0/1910098617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797406a730 0x7f797406abb0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.518+0000 7f797a3f0700 1 -- 172.21.15.189:0/1910098617 >> 172.21.15.189:0/1910098617 conn(0x7f79740d00b0 msgr2=0x7f79740d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.519+0000 7f797a3f0700 1 -- 172.21.15.189:0/1910098617 shutdown_connections 2024-04-06T12:13:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.519+0000 7f797a3f0700 1 -- 172.21.15.189:0/1910098617 wait complete. 2024-04-06T12:13:41.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.520+0000 7f797a3f0700 1 Processor -- start 2024-04-06T12:13:41.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.520+0000 7f797a3f0700 1 -- start start 2024-04-06T12:13:41.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.521+0000 7f797a3f0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f797406a730 0x7f79740cf190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:41.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.521+0000 7f797a3f0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797415d7b0 0x7f797414eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:41.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.521+0000 7f797a3f0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f797415ddf0 0x7f79740ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:41.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.521+0000 7f797a3f0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79740cf6d0 con 0x7f797415d7b0 2024-04-06T12:13:41.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.521+0000 7f797a3f0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79740cd620 con 0x7f797415ddf0 2024-04-06T12:13:41.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.521+0000 7f797a3f0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79740cd900 con 0x7f797406a730 2024-04-06T12:13:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.521+0000 7f79737fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797415d7b0 0x7f797414eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.521+0000 7f79737fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797415d7b0 0x7f797414eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57636/0 (socket says 172.21.15.189:57636) 2024-04-06T12:13:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.521+0000 7f79737fe700 1 -- 172.21.15.189:0/738944544 learned_addr learned my addr 172.21.15.189:0/738944544 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.521+0000 7f7973fff700 1 --2- 172.21.15.189:0/738944544 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f797406a730 0x7f79740cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.521+0000 7f797898d700 1 --2- 172.21.15.189:0/738944544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f797415ddf0 0x7f79740ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.522+0000 7f79737fe700 1 -- 172.21.15.189:0/738944544 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f797406a730 msgr2=0x7f79740cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.522+0000 7f79737fe700 1 --2- 172.21.15.189:0/738944544 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f797406a730 0x7f79740cf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.522+0000 7f79737fe700 1 -- 172.21.15.189:0/738944544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f797415ddf0 msgr2=0x7f79740ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.522+0000 7f79737fe700 1 --2- 172.21.15.189:0/738944544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f797415ddf0 0x7f79740ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.522+0000 7f79737fe700 1 -- 172.21.15.189:0/738944544 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79640094a0 con 0x7f797415d7b0 2024-04-06T12:13:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.522+0000 7f79737fe700 1 --2- 172.21.15.189:0/738944544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797415d7b0 0x7f797414eaf0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f79640090b0 tx=0x7f7964004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.522+0000 7f7970ff9700 1 -- 172.21.15.189:0/738944544 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79640038a0 con 0x7f797415d7b0 2024-04-06T12:13:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.522+0000 7f7970ff9700 1 -- 172.21.15.189:0/738944544 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796401f070 con 0x7f797415d7b0 2024-04-06T12:13:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.522+0000 7f797a3f0700 1 -- 172.21.15.189:0/738944544 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79740cdbe0 con 0x7f797415d7b0 2024-04-06T12:13:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.522+0000 7f7970ff9700 1 -- 172.21.15.189:0/738944544 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7964003c40 con 0x7f797415d7b0 2024-04-06T12:13:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.522+0000 7f797a3f0700 1 -- 172.21.15.189:0/738944544 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79740ce1a0 con 0x7f797415d7b0 2024-04-06T12:13:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.523+0000 7f797a3f0700 1 -- 172.21.15.189:0/738944544 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7974067bb0 con 0x7f797415d7b0 2024-04-06T12:13:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.523+0000 7f7970ff9700 1 -- 172.21.15.189:0/738944544 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 11) v1 ==== 89321+0+0 (secure 0 0 0) 0x7f796401a430 con 0x7f797415d7b0 2024-04-06T12:13:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.524+0000 7f7970ff9700 1 --2- 172.21.15.189:0/738944544 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f796006ae90 0x7f796006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.524+0000 7f7970ff9700 1 -- 172.21.15.189:0/738944544 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f7964021070 con 0x7f797415d7b0 2024-04-06T12:13:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.524+0000 7f7973fff700 1 --2- 172.21.15.189:0/738944544 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f796006ae90 0x7f796006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.524+0000 7f7973fff700 1 --2- 172.21.15.189:0/738944544 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f796006ae90 0x7f796006d350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7968006fd0 tx=0x7f7968008040 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:41.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.528+0000 7f7970ff9700 1 -- 172.21.15.189:0/738944544 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7964056190 con 0x7f797415d7b0 2024-04-06T12:13:41.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.749+0000 7f797a3f0700 1 -- 172.21.15.189:0/738944544 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f797415eb40 con 0x7f796006ae90 2024-04-06T12:13:41.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.798+0000 7f7970ff9700 1 -- 172.21.15.189:0/738944544 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f796800c050 con 0x7f796006ae90 2024-04-06T12:13:41.796 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule activated for path /snap_test_dir1/dir2 2024-04-06T12:13:41.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.800+0000 7f797a3f0700 1 -- 172.21.15.189:0/738944544 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f796006ae90 msgr2=0x7f796006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:41.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.800+0000 7f797a3f0700 1 --2- 172.21.15.189:0/738944544 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f796006ae90 0x7f796006d350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7968006fd0 tx=0x7f7968008040 comp rx=0 tx=0).stop 2024-04-06T12:13:41.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.800+0000 7f797a3f0700 1 -- 172.21.15.189:0/738944544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797415d7b0 msgr2=0x7f797414eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:41.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.800+0000 7f797a3f0700 1 --2- 172.21.15.189:0/738944544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797415d7b0 0x7f797414eaf0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f79640090b0 tx=0x7f7964004750 comp rx=0 tx=0).stop 2024-04-06T12:13:41.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.800+0000 7f797a3f0700 1 -- 172.21.15.189:0/738944544 shutdown_connections 2024-04-06T12:13:41.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.800+0000 7f797a3f0700 1 --2- 172.21.15.189:0/738944544 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f796006ae90 0x7f796006d350 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.800+0000 7f797a3f0700 1 --2- 172.21.15.189:0/738944544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f797415ddf0 0x7f79740ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.800+0000 7f797a3f0700 1 --2- 172.21.15.189:0/738944544 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f797406a730 0x7f79740cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.800+0000 7f797a3f0700 1 --2- 172.21.15.189:0/738944544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797415d7b0 0x7f797414eaf0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.800+0000 7f797a3f0700 1 -- 172.21.15.189:0/738944544 >> 172.21.15.189:0/738944544 conn(0x7f79740d00b0 msgr2=0x7f797406eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:41.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.800+0000 7f797a3f0700 1 -- 172.21.15.189:0/738944544 shutdown_connections 2024-04-06T12:13:41.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.800+0000 7f797a3f0700 1 -- 172.21.15.189:0/738944544 wait complete. 2024-04-06T12:13:41.810 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2024-04-06T12:13:41.811 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir3 1m 2024-04-06T12:13:41.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.938+0000 7fe36a322700 1 Processor -- start 2024-04-06T12:13:41.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.938+0000 7fe36a322700 1 -- start start 2024-04-06T12:13:41.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.939+0000 7fe36a322700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe364144f40 0x7fe36406a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:41.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.939+0000 7fe36a322700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe36415fcd0 0x7fe36406ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:41.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.939+0000 7fe36a322700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36415c740 con 0x7fe36415ba40 2024-04-06T12:13:41.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.939+0000 7fe36a322700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3640cfcb0 con 0x7fe36415fcd0 2024-04-06T12:13:41.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.939+0000 7fe36a322700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe36415ca20 con 0x7fe364144f40 2024-04-06T12:13:41.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.939+0000 7fe3637fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe36415fcd0 0x7fe36406ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:41.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.939+0000 7fe3637fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe36415fcd0 0x7fe36406ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57642/0 (socket says 172.21.15.189:57642) 2024-04-06T12:13:41.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.939+0000 7fe3637fe700 1 -- 172.21.15.189:0/1432934645 learned_addr learned my addr 172.21.15.189:0/1432934645 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:41.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.939+0000 7fe363fff700 1 --2- 172.21.15.189:0/1432934645 >> v2:172.21.15.192:3300/0 conn(0x7fe364144f40 0x7fe36406a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:41.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.939+0000 7fe3637fe700 1 -- 172.21.15.189:0/1432934645 >> v2:172.21.15.192:3300/0 conn(0x7fe364144f40 msgr2=0x7fe36406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:41.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.939+0000 7fe3637fe700 1 --2- 172.21.15.189:0/1432934645 >> v2:172.21.15.192:3300/0 conn(0x7fe364144f40 0x7fe36406a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.939+0000 7fe3637fe700 1 -- 172.21.15.189:0/1432934645 >> v1:172.21.15.192:6789/0 conn(0x7fe36415ba40 legacy=0x7fe36415be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:41.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.939+0000 7fe3637fe700 1 -- 172.21.15.189:0/1432934645 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36415f1f0 con 0x7fe36415fcd0 2024-04-06T12:13:41.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.939+0000 7fe363fff700 1 --2- 172.21.15.189:0/1432934645 >> v2:172.21.15.192:3300/0 conn(0x7fe364144f40 0x7fe36406a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:41.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.939+0000 7fe3637fe700 1 --2- 172.21.15.189:0/1432934645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe36415fcd0 0x7fe36406ac70 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fe364073470 tx=0x7fe354009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=877e2224b5f0e2a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:41.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.940+0000 7fe362ffd700 1 -- 172.21.15.189:0/1432934645 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe354004230 con 0x7fe36415fcd0 2024-04-06T12:13:41.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.940+0000 7fe362ffd700 1 -- 172.21.15.189:0/1432934645 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe354004390 con 0x7fe36415fcd0 2024-04-06T12:13:41.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.940+0000 7fe362ffd700 1 -- 172.21.15.189:0/1432934645 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3540036a0 con 0x7fe36415fcd0 2024-04-06T12:13:41.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.940+0000 7fe36a322700 1 -- 172.21.15.189:0/1432934645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe36415fcd0 msgr2=0x7fe36406ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:41.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.940+0000 7fe36a322700 1 --2- 172.21.15.189:0/1432934645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe36415fcd0 0x7fe36406ac70 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fe364073470 tx=0x7fe354009bd0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.940+0000 7fe36a322700 1 -- 172.21.15.189:0/1432934645 shutdown_connections 2024-04-06T12:13:41.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.940+0000 7fe36a322700 1 --2- 172.21.15.189:0/1432934645 >> v2:172.21.15.192:3300/0 conn(0x7fe364144f40 0x7fe36406a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.940+0000 7fe36a322700 1 --2- 172.21.15.189:0/1432934645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe36415fcd0 0x7fe36406ac70 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.940+0000 7fe36a322700 1 -- 172.21.15.189:0/1432934645 >> 172.21.15.189:0/1432934645 conn(0x7fe3640d00b0 msgr2=0x7fe3640d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:41.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.940+0000 7fe36a322700 1 -- 172.21.15.189:0/1432934645 shutdown_connections 2024-04-06T12:13:41.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.940+0000 7fe36a322700 1 -- 172.21.15.189:0/1432934645 wait complete. 2024-04-06T12:13:41.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.941+0000 7fe36a322700 1 Processor -- start 2024-04-06T12:13:41.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.942+0000 7fe36a322700 1 -- start start 2024-04-06T12:13:41.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.942+0000 7fe36a322700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe364144f40 0x7fe3641588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:41.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.942+0000 7fe36a322700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe36415ba40 0x7fe36413e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:41.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.942+0000 7fe36a322700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe36415fcd0 0x7fe3641f1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:41.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.942+0000 7fe36a322700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3641f0c40 con 0x7fe364144f40 2024-04-06T12:13:41.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.942+0000 7fe36a322700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36413f030 con 0x7fe36415fcd0 2024-04-06T12:13:41.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.942+0000 7fe36a322700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3640c7050 con 0x7fe36415ba40 2024-04-06T12:13:41.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.942+0000 7fe363fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe364144f40 0x7fe3641588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:41.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.942+0000 7fe363fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe364144f40 0x7fe3641588f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57656/0 (socket says 172.21.15.189:57656) 2024-04-06T12:13:41.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.942+0000 7fe363fff700 1 -- 172.21.15.189:0/3309650369 learned_addr learned my addr 172.21.15.189:0/3309650369 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:41.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.943+0000 7fe3688bf700 1 --2- 172.21.15.189:0/3309650369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe36415fcd0 0x7fe3641f1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:41.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.943+0000 7fe3637fe700 1 --2- 172.21.15.189:0/3309650369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe36415ba40 0x7fe36413e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:41.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.943+0000 7fe363fff700 1 -- 172.21.15.189:0/3309650369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe36415ba40 msgr2=0x7fe36413e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:41.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.943+0000 7fe363fff700 1 --2- 172.21.15.189:0/3309650369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe36415ba40 0x7fe36413e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.943+0000 7fe363fff700 1 -- 172.21.15.189:0/3309650369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe36415fcd0 msgr2=0x7fe3641f1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:41.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.943+0000 7fe363fff700 1 --2- 172.21.15.189:0/3309650369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe36415fcd0 0x7fe3641f1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:41.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.943+0000 7fe363fff700 1 -- 172.21.15.189:0/3309650369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe354009550 con 0x7fe364144f40 2024-04-06T12:13:41.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.943+0000 7fe3637fe700 1 --2- 172.21.15.189:0/3309650369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe36415ba40 0x7fe36413e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:41.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.943+0000 7fe363fff700 1 --2- 172.21.15.189:0/3309650369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe364144f40 0x7fe3641588f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe35800c750 tx=0x7fe35800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:41.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.943+0000 7fe360ff9700 1 -- 172.21.15.189:0/3309650369 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358011800 con 0x7fe364144f40 2024-04-06T12:13:41.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.943+0000 7fe360ff9700 1 -- 172.21.15.189:0/3309650369 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe358011960 con 0x7fe364144f40 2024-04-06T12:13:41.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.943+0000 7fe36a322700 1 -- 172.21.15.189:0/3309650369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3640c7330 con 0x7fe364144f40 2024-04-06T12:13:41.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.943+0000 7fe360ff9700 1 -- 172.21.15.189:0/3309650369 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358011cd0 con 0x7fe364144f40 2024-04-06T12:13:41.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.943+0000 7fe36a322700 1 -- 172.21.15.189:0/3309650369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3640c7610 con 0x7fe364144f40 2024-04-06T12:13:41.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.944+0000 7fe36a322700 1 -- 172.21.15.189:0/3309650369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe364067bb0 con 0x7fe364144f40 2024-04-06T12:13:41.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.945+0000 7fe360ff9700 1 -- 172.21.15.189:0/3309650369 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 11) v1 ==== 89321+0+0 (secure 0 0 0) 0x7fe358021350 con 0x7fe364144f40 2024-04-06T12:13:41.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.945+0000 7fe360ff9700 1 --2- 172.21.15.189:0/3309650369 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe35006ae90 0x7fe35006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:41.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.945+0000 7fe360ff9700 1 -- 172.21.15.189:0/3309650369 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fe358089850 con 0x7fe364144f40 2024-04-06T12:13:41.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.945+0000 7fe3637fe700 1 --2- 172.21.15.189:0/3309650369 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe35006ae90 0x7fe35006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:41.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.945+0000 7fe3637fe700 1 --2- 172.21.15.189:0/3309650369 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe35006ae90 0x7fe35006d350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe3641600e0 tx=0x7fe354008030 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:41.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:41.949+0000 7fe360ff9700 1 -- 172.21.15.189:0/3309650369 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe3580544f0 con 0x7fe364144f40 2024-04-06T12:13:42.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.170+0000 7fe36a322700 1 -- 172.21.15.189:0/3309650369 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe36415f7b0 con 0x7fe35006ae90 2024-04-06T12:13:42.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.229+0000 7fe360ff9700 1 -- 172.21.15.189:0/3309650369 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fe354020070 con 0x7fe35006ae90 2024-04-06T12:13:42.228 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule activated for path /snap_test_dir1/dir3 2024-04-06T12:13:42.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.232+0000 7fe36a322700 1 -- 172.21.15.189:0/3309650369 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe35006ae90 msgr2=0x7fe35006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:42.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.232+0000 7fe36a322700 1 --2- 172.21.15.189:0/3309650369 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe35006ae90 0x7fe35006d350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe3641600e0 tx=0x7fe354008030 comp rx=0 tx=0).stop 2024-04-06T12:13:42.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.232+0000 7fe36a322700 1 -- 172.21.15.189:0/3309650369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe364144f40 msgr2=0x7fe3641588f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:42.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.232+0000 7fe36a322700 1 --2- 172.21.15.189:0/3309650369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe364144f40 0x7fe3641588f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe35800c750 tx=0x7fe35800ca60 comp rx=0 tx=0).stop 2024-04-06T12:13:42.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.232+0000 7fe36a322700 1 -- 172.21.15.189:0/3309650369 shutdown_connections 2024-04-06T12:13:42.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.232+0000 7fe36a322700 1 --2- 172.21.15.189:0/3309650369 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe35006ae90 0x7fe35006d350 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.232+0000 7fe36a322700 1 --2- 172.21.15.189:0/3309650369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe36415fcd0 0x7fe3641f1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.232+0000 7fe36a322700 1 --2- 172.21.15.189:0/3309650369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe364144f40 0x7fe3641588f0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.232+0000 7fe36a322700 1 --2- 172.21.15.189:0/3309650369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe36415ba40 0x7fe36413e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.232+0000 7fe36a322700 1 -- 172.21.15.189:0/3309650369 >> 172.21.15.189:0/3309650369 conn(0x7fe3640d00b0 msgr2=0x7fe3640cf240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:42.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.232+0000 7fe36a322700 1 -- 172.21.15.189:0/3309650369 shutdown_connections 2024-04-06T12:13:42.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.232+0000 7fe36a322700 1 -- 172.21.15.189:0/3309650369 wait complete. 2024-04-06T12:13:42.242 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2024-04-06T12:13:42.242 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir4 1m 2024-04-06T12:13:42.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.369+0000 7f5c4b987700 1 Processor -- start 2024-04-06T12:13:42.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.369+0000 7f5c4b987700 1 -- start start 2024-04-06T12:13:42.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c4b987700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5c440b4660 0x7f5c440b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:42.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c4b987700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c440b5030 con 0x7f5c4415a550 2024-04-06T12:13:42.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c4b987700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c440cdbe0 con 0x7f5c440b2fc0 2024-04-06T12:13:42.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c4b987700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c4415a940 con 0x7f5c440b4660 2024-04-06T12:13:42.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c49723700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5c4415a550 0x7f5c440cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38996/0 (socket says 172.21.15.189:38996) 2024-04-06T12:13:42.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c49723700 1 -- 172.21.15.189:0/3152569658 learned_addr learned my addr 172.21.15.189:0/3152569658 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:42.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c3bfff700 1 -- 172.21.15.189:0/3152569658 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850110852 0 0) 0x7f5c440b5030 con 0x7f5c4415a550 2024-04-06T12:13:42.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c3bfff700 1 -- 172.21.15.189:0/3152569658 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c300035b0 con 0x7f5c4415a550 2024-04-06T12:13:42.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c48f22700 1 --2- 172.21.15.189:0/3152569658 >> v2:172.21.15.192:3300/0 conn(0x7f5c440b4660 0x7f5c440b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:42.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c3bfff700 1 -- 172.21.15.189:0/3152569658 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3034834107 0 0) 0x7f5c300035b0 con 0x7f5c4415a550 2024-04-06T12:13:42.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c3bfff700 1 -- 172.21.15.189:0/3152569658 >> v2:172.21.15.192:3300/0 conn(0x7f5c440b4660 msgr2=0x7f5c440b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:42.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c3bfff700 1 --2- 172.21.15.189:0/3152569658 >> v2:172.21.15.192:3300/0 conn(0x7f5c440b4660 0x7f5c440b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c3bfff700 1 -- 172.21.15.189:0/3152569658 >> v1:172.21.15.192:6789/0 conn(0x7f5c440b2fc0 legacy=0x7f5c440b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:42.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c3bfff700 1 -- 172.21.15.189:0/3152569658 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4415ac00 con 0x7f5c4415a550 2024-04-06T12:13:42.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c3bfff700 1 -- 172.21.15.189:0/3152569658 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f5c40001670 con 0x7f5c4415a550 2024-04-06T12:13:42.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c3bfff700 1 -- 172.21.15.189:0/3152569658 >> v1:172.21.15.189:6789/0 conn(0x7f5c4415a550 legacy=0x7f5c440cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:42.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c3bfff700 1 --2- 172.21.15.189:0/3152569658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c30003a50 0x7f5c30003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:42.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c3bfff700 1 --2- 172.21.15.189:0/3152569658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c30004850 0x7f5c30004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:42.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c3bfff700 1 --2- 172.21.15.189:0/3152569658 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c30005260 0x7f5c3000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:42.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c3bfff700 1 -- 172.21.15.189:0/3152569658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4415a940 con 0x7f5c30004850 2024-04-06T12:13:42.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c3bfff700 1 -- 172.21.15.189:0/3152569658 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3000c190 con 0x7f5c30003a50 2024-04-06T12:13:42.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c3bfff700 1 -- 172.21.15.189:0/3152569658 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3000c350 con 0x7f5c30005260 2024-04-06T12:13:42.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.370+0000 7f5c48f22700 1 --2- 172.21.15.189:0/3152569658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c30004850 0x7f5c30004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:42.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.371+0000 7f5c49723700 1 --2- 172.21.15.189:0/3152569658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c30003a50 0x7f5c30003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:42.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.371+0000 7f5c49f24700 1 --2- 172.21.15.189:0/3152569658 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c30005260 0x7f5c3000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:42.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.371+0000 7f5c48f22700 1 -- 172.21.15.189:0/3152569658 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c30005260 msgr2=0x7f5c3000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:42.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.371+0000 7f5c48f22700 1 --2- 172.21.15.189:0/3152569658 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c30005260 0x7f5c3000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.371+0000 7f5c48f22700 1 -- 172.21.15.189:0/3152569658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c30003a50 msgr2=0x7f5c30003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:42.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.371+0000 7f5c48f22700 1 --2- 172.21.15.189:0/3152569658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c30003a50 0x7f5c30003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.371+0000 7f5c48f22700 1 -- 172.21.15.189:0/3152569658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c300035b0 con 0x7f5c30004850 2024-04-06T12:13:42.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.371+0000 7f5c49f24700 1 -- 172.21.15.189:0/3152569658 reap_dead start 2024-04-06T12:13:42.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.371+0000 7f5c49723700 1 --2- 172.21.15.189:0/3152569658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c30003a50 0x7f5c30003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:42.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.371+0000 7f5c48f22700 1 --2- 172.21.15.189:0/3152569658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c30004850 0x7f5c30004cd0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5c34009860 tx=0x7f5c30005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e99237f0d6e7febd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:42.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.371+0000 7f5c3bfff700 1 -- 172.21.15.189:0/3152569658 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3400c070 con 0x7f5c30004850 2024-04-06T12:13:42.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.371+0000 7f5c3bfff700 1 -- 172.21.15.189:0/3152569658 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c34013610 con 0x7f5c30004850 2024-04-06T12:13:42.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.371+0000 7f5c3bfff700 1 -- 172.21.15.189:0/3152569658 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c340139a0 con 0x7f5c30004850 2024-04-06T12:13:42.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.371+0000 7f5c4b987700 1 -- 172.21.15.189:0/3152569658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c30004850 msgr2=0x7f5c30004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:42.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.371+0000 7f5c4b987700 1 --2- 172.21.15.189:0/3152569658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c30004850 0x7f5c30004cd0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5c34009860 tx=0x7f5c30005880 comp rx=0 tx=0).stop 2024-04-06T12:13:42.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.372+0000 7f5c4b987700 1 -- 172.21.15.189:0/3152569658 shutdown_connections 2024-04-06T12:13:42.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.372+0000 7f5c4b987700 1 --2- 172.21.15.189:0/3152569658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c30004850 0x7f5c30004cd0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.372+0000 7f5c4b987700 1 -- 172.21.15.189:0/3152569658 >> 172.21.15.189:0/3152569658 conn(0x7f5c440d00b0 msgr2=0x7f5c440d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:42.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.372+0000 7f5c4b987700 1 -- 172.21.15.189:0/3152569658 shutdown_connections 2024-04-06T12:13:42.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.372+0000 7f5c4b987700 1 -- 172.21.15.189:0/3152569658 wait complete. 2024-04-06T12:13:42.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.373+0000 7f5c4b987700 1 Processor -- start 2024-04-06T12:13:42.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.373+0000 7f5c4b987700 1 -- start start 2024-04-06T12:13:42.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c4b987700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c30004850 0x7f5c441406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:42.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c4b987700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c44157110 0x7f5c44157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:42.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c4b987700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c441f0ca0 0x7f5c441f1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:42.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c4b987700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c44140bf0 con 0x7f5c441f0ca0 2024-04-06T12:13:42.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c4b987700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c44141680 con 0x7f5c30004850 2024-04-06T12:13:42.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c4b987700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c44157ad0 con 0x7f5c44157110 2024-04-06T12:13:42.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c49f24700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c441f0ca0 0x7f5c441f1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:42.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c49f24700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c441f0ca0 0x7f5c441f1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57666/0 (socket says 172.21.15.189:57666) 2024-04-06T12:13:42.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c49f24700 1 -- 172.21.15.189:0/184019079 learned_addr learned my addr 172.21.15.189:0/184019079 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:42.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c48f22700 1 --2- 172.21.15.189:0/184019079 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c44157110 0x7f5c44157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:42.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c49723700 1 --2- 172.21.15.189:0/184019079 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c30004850 0x7f5c441406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:42.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c49f24700 1 -- 172.21.15.189:0/184019079 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c44157110 msgr2=0x7f5c44157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:42.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c49f24700 1 --2- 172.21.15.189:0/184019079 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c44157110 0x7f5c44157590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c49f24700 1 -- 172.21.15.189:0/184019079 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c30004850 msgr2=0x7f5c441406b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:42.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c49f24700 1 --2- 172.21.15.189:0/184019079 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c30004850 0x7f5c441406b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c49f24700 1 -- 172.21.15.189:0/184019079 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c441f1660 con 0x7f5c441f0ca0 2024-04-06T12:13:42.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c49f24700 1 --2- 172.21.15.189:0/184019079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c441f0ca0 0x7f5c441f1120 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5c440d1110 tx=0x7f5c3c00edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:42.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c39ffb700 1 -- 172.21.15.189:0/184019079 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c00c7b0 con 0x7f5c441f0ca0 2024-04-06T12:13:42.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c39ffb700 1 -- 172.21.15.189:0/184019079 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3c00c910 con 0x7f5c441f0ca0 2024-04-06T12:13:42.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c4b987700 1 -- 172.21.15.189:0/184019079 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c4414f230 con 0x7f5c441f0ca0 2024-04-06T12:13:42.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.374+0000 7f5c39ffb700 1 -- 172.21.15.189:0/184019079 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c00ccc0 con 0x7f5c441f0ca0 2024-04-06T12:13:42.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.375+0000 7f5c4b987700 1 -- 172.21.15.189:0/184019079 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c4413e360 con 0x7f5c441f0ca0 2024-04-06T12:13:42.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.375+0000 7f5c4b987700 1 -- 172.21.15.189:0/184019079 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c4414f3c0 con 0x7f5c441f0ca0 2024-04-06T12:13:42.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.377+0000 7f5c39ffb700 1 -- 172.21.15.189:0/184019079 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 11) v1 ==== 89321+0+0 (secure 0 0 0) 0x7f5c3c01f460 con 0x7f5c441f0ca0 2024-04-06T12:13:42.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.377+0000 7f5c39ffb700 1 --2- 172.21.15.189:0/184019079 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2806ae90 0x7f5c2806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:42.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.378+0000 7f5c39ffb700 1 -- 172.21.15.189:0/184019079 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f5c3c08a9b0 con 0x7f5c441f0ca0 2024-04-06T12:13:42.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.378+0000 7f5c49723700 1 --2- 172.21.15.189:0/184019079 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2806ae90 0x7f5c2806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:42.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.378+0000 7f5c49723700 1 --2- 172.21.15.189:0/184019079 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2806ae90 0x7f5c2806d350 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5c40001be0 tx=0x7f5c40007040 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:42.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.381+0000 7f5c39ffb700 1 -- 172.21.15.189:0/184019079 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5c3c0556b0 con 0x7f5c441f0ca0 2024-04-06T12:13:42.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.604+0000 7f5c4b987700 1 -- 172.21.15.189:0/184019079 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c4415b4a0 con 0x7f5c2806ae90 2024-04-06T12:13:42.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.671+0000 7f5c39ffb700 1 -- 172.21.15.189:0/184019079 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f5c400116f0 con 0x7f5c2806ae90 2024-04-06T12:13:42.669 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule activated for path /snap_test_dir1/dir4 2024-04-06T12:13:42.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.673+0000 7f5c4b987700 1 -- 172.21.15.189:0/184019079 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2806ae90 msgr2=0x7f5c2806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:42.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.673+0000 7f5c4b987700 1 --2- 172.21.15.189:0/184019079 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2806ae90 0x7f5c2806d350 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5c40001be0 tx=0x7f5c40007040 comp rx=0 tx=0).stop 2024-04-06T12:13:42.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.673+0000 7f5c4b987700 1 -- 172.21.15.189:0/184019079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c441f0ca0 msgr2=0x7f5c441f1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:42.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.673+0000 7f5c4b987700 1 --2- 172.21.15.189:0/184019079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c441f0ca0 0x7f5c441f1120 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5c440d1110 tx=0x7f5c3c00edd0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.673+0000 7f5c4b987700 1 -- 172.21.15.189:0/184019079 shutdown_connections 2024-04-06T12:13:42.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.673+0000 7f5c4b987700 1 --2- 172.21.15.189:0/184019079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c441f0ca0 0x7f5c441f1120 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.673+0000 7f5c4b987700 1 --2- 172.21.15.189:0/184019079 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c2806ae90 0x7f5c2806d350 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.673+0000 7f5c4b987700 1 --2- 172.21.15.189:0/184019079 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c30004850 0x7f5c441406b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.673+0000 7f5c4b987700 1 --2- 172.21.15.189:0/184019079 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c44157110 0x7f5c44157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.673+0000 7f5c4b987700 1 -- 172.21.15.189:0/184019079 >> 172.21.15.189:0/184019079 conn(0x7f5c440d00b0 msgr2=0x7f5c4406eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:42.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.673+0000 7f5c4b987700 1 -- 172.21.15.189:0/184019079 shutdown_connections 2024-04-06T12:13:42.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.673+0000 7f5c4b987700 1 -- 172.21.15.189:0/184019079 wait complete. 2024-04-06T12:13:42.684 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2024-04-06T12:13:42.684 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir5 1m 2024-04-06T12:13:42.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.810+0000 7f94f7b68700 1 Processor -- start 2024-04-06T12:13:42.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.810+0000 7f94f7b68700 1 -- start start 2024-04-06T12:13:42.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.811+0000 7f94f7b68700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f94f015d7b0 0x7f94f00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:42.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.811+0000 7f94f7b68700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f94f006a730 0x7f94f006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:42.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.811+0000 7f94f7b68700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f00b3990 con 0x7f94f015ddf0 2024-04-06T12:13:42.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.811+0000 7f94f7b68700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f015e950 con 0x7f94f006a730 2024-04-06T12:13:42.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.811+0000 7f94f7b68700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f006b0f0 con 0x7f94f015d7b0 2024-04-06T12:13:42.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.811+0000 7f94f5103700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f94f006a730 0x7f94f006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:42.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.811+0000 7f94f6105700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f94f015ddf0 0x7f94f015e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39008/0 (socket says 172.21.15.189:39008) 2024-04-06T12:13:42.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.811+0000 7f94f6105700 1 -- 172.21.15.189:0/3693133649 learned_addr learned my addr 172.21.15.189:0/3693133649 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:42.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.811+0000 7f94f5904700 1 --2- 172.21.15.189:0/3693133649 >> v2:172.21.15.192:3300/0 conn(0x7f94f015d7b0 0x7f94f00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:42.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.811+0000 7f94f5103700 1 -- 172.21.15.189:0/3693133649 >> v2:172.21.15.192:3300/0 conn(0x7f94f015d7b0 msgr2=0x7f94f00b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:42.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.811+0000 7f94f5103700 1 --2- 172.21.15.189:0/3693133649 >> v2:172.21.15.192:3300/0 conn(0x7f94f015d7b0 0x7f94f00b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.811+0000 7f94f5103700 1 -- 172.21.15.189:0/3693133649 >> v1:172.21.15.189:6789/0 conn(0x7f94f015ddf0 legacy=0x7f94f015e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:42.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.811+0000 7f94f5103700 1 -- 172.21.15.189:0/3693133649 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f00b3060 con 0x7f94f006a730 2024-04-06T12:13:42.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.811+0000 7f94f4902700 1 -- 172.21.15.189:0/3693133649 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172192404 0 0) 0x7f94f00b3990 con 0x7f94f015ddf0 2024-04-06T12:13:42.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.811+0000 7f94f5103700 1 --2- 172.21.15.189:0/3693133649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94f006a730 0x7f94f006abb0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f94f00b6200 tx=0x7f94e0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0207875070972ac server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:42.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.811+0000 7f94f4902700 1 -- 172.21.15.189:0/3693133649 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e0011830 con 0x7f94f006a730 2024-04-06T12:13:42.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.811+0000 7f94f4902700 1 -- 172.21.15.189:0/3693133649 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e0011990 con 0x7f94f006a730 2024-04-06T12:13:42.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.812+0000 7f94f4902700 1 -- 172.21.15.189:0/3693133649 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e0011d20 con 0x7f94f006a730 2024-04-06T12:13:42.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.812+0000 7f94f7b68700 1 -- 172.21.15.189:0/3693133649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94f006a730 msgr2=0x7f94f006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:42.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.812+0000 7f94f7b68700 1 --2- 172.21.15.189:0/3693133649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94f006a730 0x7f94f006abb0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f94f00b6200 tx=0x7f94e0009b20 comp rx=0 tx=0).stop 2024-04-06T12:13:42.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.812+0000 7f94f7b68700 1 -- 172.21.15.189:0/3693133649 shutdown_connections 2024-04-06T12:13:42.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.812+0000 7f94f7b68700 1 --2- 172.21.15.189:0/3693133649 >> v2:172.21.15.192:3300/0 conn(0x7f94f015d7b0 0x7f94f00b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.812+0000 7f94f7b68700 1 --2- 172.21.15.189:0/3693133649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94f006a730 0x7f94f006abb0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.812+0000 7f94f7b68700 1 -- 172.21.15.189:0/3693133649 >> 172.21.15.189:0/3693133649 conn(0x7f94f00d00b0 msgr2=0x7f94f00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:42.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.812+0000 7f94f7b68700 1 -- 172.21.15.189:0/3693133649 shutdown_connections 2024-04-06T12:13:42.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.812+0000 7f94f7b68700 1 -- 172.21.15.189:0/3693133649 wait complete. 2024-04-06T12:13:42.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.813+0000 7f94f7b68700 1 Processor -- start 2024-04-06T12:13:42.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.813+0000 7f94f7b68700 1 -- start start 2024-04-06T12:13:42.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.814+0000 7f94f7b68700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94f006a730 0x7f94f0156750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:42.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.814+0000 7f94f7b68700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94f015d7b0 0x7f94f00cc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:42.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.814+0000 7f94f7b68700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94f015ddf0 0x7f94f00cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:42.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.814+0000 7f94f7b68700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f01556b0 con 0x7f94f015d7b0 2024-04-06T12:13:42.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.814+0000 7f94f7b68700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f0155990 con 0x7f94f006a730 2024-04-06T12:13:42.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.814+0000 7f94f7b68700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f0155c70 con 0x7f94f015ddf0 2024-04-06T12:13:42.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.814+0000 7f94f5103700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94f015d7b0 0x7f94f00cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:42.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.814+0000 7f94f5103700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94f015d7b0 0x7f94f00cc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57674/0 (socket says 172.21.15.189:57674) 2024-04-06T12:13:42.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.814+0000 7f94f5103700 1 -- 172.21.15.189:0/2541523799 learned_addr learned my addr 172.21.15.189:0/2541523799 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:42.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.814+0000 7f94f5103700 1 -- 172.21.15.189:0/2541523799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94f015ddf0 msgr2=0x7f94f00cc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:42.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.814+0000 7f94f5103700 1 --2- 172.21.15.189:0/2541523799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94f015ddf0 0x7f94f00cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.814+0000 7f94f5103700 1 -- 172.21.15.189:0/2541523799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94f006a730 msgr2=0x7f94f0156750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:42.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.814+0000 7f94f5904700 1 --2- 172.21.15.189:0/2541523799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94f006a730 0x7f94f0156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:42.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.814+0000 7f94f5103700 1 --2- 172.21.15.189:0/2541523799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94f006a730 0x7f94f0156750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:42.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.814+0000 7f94f5103700 1 -- 172.21.15.189:0/2541523799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e00094a0 con 0x7f94f015d7b0 2024-04-06T12:13:42.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.815+0000 7f94f5103700 1 --2- 172.21.15.189:0/2541523799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94f015d7b0 0x7f94f00cc460 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f94e00090b0 tx=0x7f94e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:42.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.815+0000 7f94e67fc700 1 -- 172.21.15.189:0/2541523799 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e00038a0 con 0x7f94f015d7b0 2024-04-06T12:13:42.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.815+0000 7f94e67fc700 1 -- 172.21.15.189:0/2541523799 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e001f070 con 0x7f94f015d7b0 2024-04-06T12:13:42.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.815+0000 7f94e67fc700 1 -- 172.21.15.189:0/2541523799 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e0003c20 con 0x7f94f015d7b0 2024-04-06T12:13:42.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.815+0000 7f94f7b68700 1 -- 172.21.15.189:0/2541523799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f0155f50 con 0x7f94f015d7b0 2024-04-06T12:13:42.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.815+0000 7f94f7b68700 1 -- 172.21.15.189:0/2541523799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f0156c90 con 0x7f94f015d7b0 2024-04-06T12:13:42.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.816+0000 7f94f7b68700 1 -- 172.21.15.189:0/2541523799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f0067bb0 con 0x7f94f015d7b0 2024-04-06T12:13:42.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.816+0000 7f94e67fc700 1 -- 172.21.15.189:0/2541523799 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 11) v1 ==== 89321+0+0 (secure 0 0 0) 0x7f94e001a430 con 0x7f94f015d7b0 2024-04-06T12:13:42.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.817+0000 7f94e67fc700 1 --2- 172.21.15.189:0/2541523799 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f94d406ae90 0x7f94d406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:42.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.817+0000 7f94e67fc700 1 -- 172.21.15.189:0/2541523799 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f94e0021070 con 0x7f94f015d7b0 2024-04-06T12:13:42.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.817+0000 7f94f5904700 1 --2- 172.21.15.189:0/2541523799 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f94d406ae90 0x7f94d406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:42.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.817+0000 7f94f5904700 1 --2- 172.21.15.189:0/2541523799 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f94d406ae90 0x7f94d406d350 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f94ec009950 tx=0x7f94ec008040 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:42.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:42.821+0000 7f94e67fc700 1 -- 172.21.15.189:0/2541523799 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f94e0056040 con 0x7f94f015d7b0 2024-04-06T12:13:43.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.043+0000 7f94f7b68700 1 -- 172.21.15.189:0/2541523799 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f94f015eb40 con 0x7f94d406ae90 2024-04-06T12:13:43.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.107+0000 7f94e67fc700 1 -- 172.21.15.189:0/2541523799 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f94ec0095e0 con 0x7f94d406ae90 2024-04-06T12:13:43.106 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule activated for path /snap_test_dir1/dir5 2024-04-06T12:13:43.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.110+0000 7f94f7b68700 1 -- 172.21.15.189:0/2541523799 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f94d406ae90 msgr2=0x7f94d406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:43.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.110+0000 7f94f7b68700 1 --2- 172.21.15.189:0/2541523799 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f94d406ae90 0x7f94d406d350 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f94ec009950 tx=0x7f94ec008040 comp rx=0 tx=0).stop 2024-04-06T12:13:43.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.110+0000 7f94f7b68700 1 -- 172.21.15.189:0/2541523799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94f015d7b0 msgr2=0x7f94f00cc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:43.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.110+0000 7f94f7b68700 1 --2- 172.21.15.189:0/2541523799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94f015d7b0 0x7f94f00cc460 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f94e00090b0 tx=0x7f94e0004750 comp rx=0 tx=0).stop 2024-04-06T12:13:43.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.110+0000 7f94f7b68700 1 -- 172.21.15.189:0/2541523799 shutdown_connections 2024-04-06T12:13:43.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.110+0000 7f94f7b68700 1 --2- 172.21.15.189:0/2541523799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94f015ddf0 0x7f94f00cc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.110+0000 7f94f7b68700 1 --2- 172.21.15.189:0/2541523799 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f94d406ae90 0x7f94d406d350 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.110+0000 7f94f7b68700 1 --2- 172.21.15.189:0/2541523799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94f006a730 0x7f94f0156750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.110+0000 7f94f7b68700 1 --2- 172.21.15.189:0/2541523799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94f015d7b0 0x7f94f00cc460 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.110+0000 7f94f7b68700 1 -- 172.21.15.189:0/2541523799 >> 172.21.15.189:0/2541523799 conn(0x7f94f00d00b0 msgr2=0x7f94f00adc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:43.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.110+0000 7f94f7b68700 1 -- 172.21.15.189:0/2541523799 shutdown_connections 2024-04-06T12:13:43.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.110+0000 7f94f7b68700 1 -- 172.21.15.189:0/2541523799 wait complete. 2024-04-06T12:13:43.120 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2024-04-06T12:13:43.120 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir6 1m 2024-04-06T12:13:43.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.247+0000 7f5682dbe700 1 Processor -- start 2024-04-06T12:13:43.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.247+0000 7f5682dbe700 1 -- start start 2024-04-06T12:13:43.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.247+0000 7f5682dbe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f567c0cbc20 0x7f567c15f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:43.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.247+0000 7f5682dbe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f567c06aa10 0x7f567c06ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:43.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.247+0000 7f5682dbe700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567c0669d0 con 0x7f567c15e8a0 2024-04-06T12:13:43.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.247+0000 7f5682dbe700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f567c06ede0 con 0x7f567c0cbc20 2024-04-06T12:13:43.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.247+0000 7f5682dbe700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f567c15ec80 con 0x7f567c06aa10 2024-04-06T12:13:43.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.248+0000 7f567bfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f567c0cbc20 0x7f567c15f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:43.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.248+0000 7f567bfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f567c0cbc20 0x7f567c15f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57682/0 (socket says 172.21.15.189:57682) 2024-04-06T12:13:43.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.248+0000 7f5680b5a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f567c15e8a0 0x7f567c15eee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39010/0 (socket says 172.21.15.189:39010) 2024-04-06T12:13:43.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.248+0000 7f567bfff700 1 -- 172.21.15.189:0/964614227 learned_addr learned my addr 172.21.15.189:0/964614227 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:43.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.248+0000 7f568135b700 1 --2- 172.21.15.189:0/964614227 >> v2:172.21.15.192:3300/0 conn(0x7f567c06aa10 0x7f567c06ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:43.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.248+0000 7f567bfff700 1 -- 172.21.15.189:0/964614227 >> v2:172.21.15.192:3300/0 conn(0x7f567c06aa10 msgr2=0x7f567c06ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:43.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.248+0000 7f567bfff700 1 --2- 172.21.15.189:0/964614227 >> v2:172.21.15.192:3300/0 conn(0x7f567c06aa10 0x7f567c06ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.248+0000 7f567bfff700 1 -- 172.21.15.189:0/964614227 >> v1:172.21.15.189:6789/0 conn(0x7f567c15e8a0 legacy=0x7f567c15eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:43.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.248+0000 7f567b7fe700 1 -- 172.21.15.189:0/964614227 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840664549 0 0) 0x7f567c0669d0 con 0x7f567c15e8a0 2024-04-06T12:13:43.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.248+0000 7f567bfff700 1 -- 172.21.15.189:0/964614227 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567c0666f0 con 0x7f567c0cbc20 2024-04-06T12:13:43.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.248+0000 7f567bfff700 1 --2- 172.21.15.189:0/964614227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f567c0cbc20 0x7f567c15f5f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f567c0cce40 tx=0x7f5668009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a5c2d9fca09f72 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:43.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.248+0000 7f567b7fe700 1 -- 172.21.15.189:0/964614227 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668011830 con 0x7f567c0cbc20 2024-04-06T12:13:43.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.248+0000 7f567b7fe700 1 -- 172.21.15.189:0/964614227 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5668011990 con 0x7f567c0cbc20 2024-04-06T12:13:43.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.248+0000 7f567b7fe700 1 -- 172.21.15.189:0/964614227 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668011d20 con 0x7f567c0cbc20 2024-04-06T12:13:43.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.249+0000 7f5682dbe700 1 -- 172.21.15.189:0/964614227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f567c0cbc20 msgr2=0x7f567c15f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:43.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.249+0000 7f5682dbe700 1 --2- 172.21.15.189:0/964614227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f567c0cbc20 0x7f567c15f5f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f567c0cce40 tx=0x7f5668009bd0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.249+0000 7f5682dbe700 1 -- 172.21.15.189:0/964614227 shutdown_connections 2024-04-06T12:13:43.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.249+0000 7f5682dbe700 1 --2- 172.21.15.189:0/964614227 >> v2:172.21.15.192:3300/0 conn(0x7f567c06aa10 0x7f567c06ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.249+0000 7f5682dbe700 1 --2- 172.21.15.189:0/964614227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f567c0cbc20 0x7f567c15f5f0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.249+0000 7f5682dbe700 1 -- 172.21.15.189:0/964614227 >> 172.21.15.189:0/964614227 conn(0x7f567c0cc290 msgr2=0x7f567c0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:43.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.249+0000 7f5682dbe700 1 -- 172.21.15.189:0/964614227 shutdown_connections 2024-04-06T12:13:43.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.249+0000 7f5682dbe700 1 -- 172.21.15.189:0/964614227 wait complete. 2024-04-06T12:13:43.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.250+0000 7f5682dbe700 1 Processor -- start 2024-04-06T12:13:43.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.250+0000 7f5682dbe700 1 -- start start 2024-04-06T12:13:43.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.251+0000 7f5682dbe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f567c06aa10 0x7f567c156a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:43.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.251+0000 7f5682dbe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f567c0cbc20 0x7f567c0adbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:43.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.251+0000 7f5682dbe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f567c15e8a0 0x7f567c0c7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:43.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.251+0000 7f5682dbe700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c0ac8d0 con 0x7f567c0cbc20 2024-04-06T12:13:43.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.251+0000 7f5682dbe700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c156f80 con 0x7f567c15e8a0 2024-04-06T12:13:43.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.251+0000 7f5682dbe700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567c0acb20 con 0x7f567c06aa10 2024-04-06T12:13:43.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.251+0000 7f567bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f567c0cbc20 0x7f567c0adbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:43.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.251+0000 7f567bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f567c0cbc20 0x7f567c0adbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57686/0 (socket says 172.21.15.189:57686) 2024-04-06T12:13:43.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.251+0000 7f567bfff700 1 -- 172.21.15.189:0/2116989433 learned_addr learned my addr 172.21.15.189:0/2116989433 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:43.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.251+0000 7f5680b5a700 1 --2- 172.21.15.189:0/2116989433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f567c06aa10 0x7f567c156a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:43.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.251+0000 7f568135b700 1 --2- 172.21.15.189:0/2116989433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f567c15e8a0 0x7f567c0c7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:43.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.251+0000 7f567bfff700 1 -- 172.21.15.189:0/2116989433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f567c06aa10 msgr2=0x7f567c156a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:43.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.251+0000 7f567bfff700 1 --2- 172.21.15.189:0/2116989433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f567c06aa10 0x7f567c156a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.251+0000 7f567bfff700 1 -- 172.21.15.189:0/2116989433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f567c15e8a0 msgr2=0x7f567c0c7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:43.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.251+0000 7f567bfff700 1 --2- 172.21.15.189:0/2116989433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f567c15e8a0 0x7f567c0c7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.251+0000 7f567bfff700 1 -- 172.21.15.189:0/2116989433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5668009550 con 0x7f567c0cbc20 2024-04-06T12:13:43.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.252+0000 7f567bfff700 1 --2- 172.21.15.189:0/2116989433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f567c0cbc20 0x7f567c0adbc0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f567c073430 tx=0x7f5668004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:43.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.252+0000 7f56797fa700 1 -- 172.21.15.189:0/2116989433 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668004510 con 0x7f567c0cbc20 2024-04-06T12:13:43.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.252+0000 7f56797fa700 1 -- 172.21.15.189:0/2116989433 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56680036a0 con 0x7f567c0cbc20 2024-04-06T12:13:43.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.252+0000 7f56797fa700 1 -- 172.21.15.189:0/2116989433 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668003a10 con 0x7f567c0cbc20 2024-04-06T12:13:43.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.252+0000 7f5682dbe700 1 -- 172.21.15.189:0/2116989433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f567c0ace00 con 0x7f567c0cbc20 2024-04-06T12:13:43.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.252+0000 7f5682dbe700 1 -- 172.21.15.189:0/2116989433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f567c0ad3c0 con 0x7f567c0cbc20 2024-04-06T12:13:43.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.253+0000 7f5682dbe700 1 -- 172.21.15.189:0/2116989433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567c072140 con 0x7f567c0cbc20 2024-04-06T12:13:43.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.253+0000 7f56797fa700 1 -- 172.21.15.189:0/2116989433 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 11) v1 ==== 89321+0+0 (secure 0 0 0) 0x7f5668003b70 con 0x7f567c0cbc20 2024-04-06T12:13:43.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.254+0000 7f56797fa700 1 --2- 172.21.15.189:0/2116989433 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f566006ae90 0x7f566006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:43.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.254+0000 7f5680b5a700 1 --2- 172.21.15.189:0/2116989433 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f566006ae90 0x7f566006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:43.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.254+0000 7f56797fa700 1 -- 172.21.15.189:0/2116989433 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f5668021030 con 0x7f567c0cbc20 2024-04-06T12:13:43.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.254+0000 7f5680b5a700 1 --2- 172.21.15.189:0/2116989433 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f566006ae90 0x7f566006d350 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5670000f60 tx=0x7f5670007e70 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:43.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.258+0000 7f56797fa700 1 -- 172.21.15.189:0/2116989433 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5668055c50 con 0x7f567c0cbc20 2024-04-06T12:13:43.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.479+0000 7f5682dbe700 1 -- 172.21.15.189:0/2116989433 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f567c06dfe0 con 0x7f566006ae90 2024-04-06T12:13:43.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.527+0000 7f56797fa700 1 -- 172.21.15.189:0/2116989433 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f5670004ab0 con 0x7f566006ae90 2024-04-06T12:13:43.525 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule activated for path /snap_test_dir1/dir6 2024-04-06T12:13:43.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.529+0000 7f5682dbe700 1 -- 172.21.15.189:0/2116989433 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f566006ae90 msgr2=0x7f566006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:43.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.529+0000 7f5682dbe700 1 --2- 172.21.15.189:0/2116989433 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f566006ae90 0x7f566006d350 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5670000f60 tx=0x7f5670007e70 comp rx=0 tx=0).stop 2024-04-06T12:13:43.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.529+0000 7f5682dbe700 1 -- 172.21.15.189:0/2116989433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f567c0cbc20 msgr2=0x7f567c0adbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:43.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.529+0000 7f5682dbe700 1 --2- 172.21.15.189:0/2116989433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f567c0cbc20 0x7f567c0adbc0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f567c073430 tx=0x7f5668004750 comp rx=0 tx=0).stop 2024-04-06T12:13:43.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.529+0000 7f5682dbe700 1 -- 172.21.15.189:0/2116989433 shutdown_connections 2024-04-06T12:13:43.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.529+0000 7f5682dbe700 1 --2- 172.21.15.189:0/2116989433 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f566006ae90 0x7f566006d350 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.529+0000 7f5682dbe700 1 --2- 172.21.15.189:0/2116989433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f567c15e8a0 0x7f567c0c7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.529+0000 7f5682dbe700 1 --2- 172.21.15.189:0/2116989433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f567c06aa10 0x7f567c156a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.529+0000 7f5682dbe700 1 --2- 172.21.15.189:0/2116989433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f567c0cbc20 0x7f567c0adbc0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.529+0000 7f5682dbe700 1 -- 172.21.15.189:0/2116989433 >> 172.21.15.189:0/2116989433 conn(0x7f567c0cc290 msgr2=0x7f567c15eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:43.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.529+0000 7f5682dbe700 1 -- 172.21.15.189:0/2116989433 shutdown_connections 2024-04-06T12:13:43.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.529+0000 7f5682dbe700 1 -- 172.21.15.189:0/2116989433 wait complete. 2024-04-06T12:13:43.539 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2024-04-06T12:13:43.539 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir7 1m 2024-04-06T12:13:43.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.666+0000 7f66db606700 1 Processor -- start 2024-04-06T12:13:43.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.666+0000 7f66db606700 1 -- start start 2024-04-06T12:13:43.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.667+0000 7f66db606700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f66d40cb870 0x7f66d415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:43.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.667+0000 7f66db606700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66d40cfa00 0x7f66d40cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:43.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.667+0000 7f66db606700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d40cbd80 con 0x7f66d40d11b0 2024-04-06T12:13:43.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.667+0000 7f66db606700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d40ccdc0 con 0x7f66d40cb870 2024-04-06T12:13:43.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.667+0000 7f66db606700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d40d0b50 con 0x7f66d40cfa00 2024-04-06T12:13:43.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.667+0000 7f66d93a2700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f66d40d11b0 0x7f66d40d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39022/0 (socket says 172.21.15.189:39022) 2024-04-06T12:13:43.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.667+0000 7f66d93a2700 1 -- 172.21.15.189:0/600454897 learned_addr learned my addr 172.21.15.189:0/600454897 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:13:43.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.667+0000 7f66d9ba3700 1 --2- 172.21.15.189:0/600454897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66d40cfa00 0x7f66d40cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:43.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.667+0000 7f66d8ba1700 1 --2- 172.21.15.189:0/600454897 >> v2:172.21.15.192:3300/0 conn(0x7f66d40cb870 0x7f66d415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:43.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.667+0000 7f66cbfff700 1 -- 172.21.15.189:0/600454897 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582257026 0 0) 0x7f66d40cbd80 con 0x7f66d40d11b0 2024-04-06T12:13:43.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.667+0000 7f66cbfff700 1 -- 172.21.15.189:0/600454897 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66c00035b0 con 0x7f66d40d11b0 2024-04-06T12:13:43.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.667+0000 7f66d8ba1700 1 -- 172.21.15.189:0/600454897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66d40cfa00 msgr2=0x7f66d40cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:43.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.667+0000 7f66cbfff700 1 -- 172.21.15.189:0/600454897 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4146311125 0 0) 0x7f66c00035b0 con 0x7f66d40d11b0 2024-04-06T12:13:43.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.667+0000 7f66d8ba1700 1 --2- 172.21.15.189:0/600454897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66d40cfa00 0x7f66d40cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.667+0000 7f66d8ba1700 1 -- 172.21.15.189:0/600454897 >> v1:172.21.15.189:6789/0 conn(0x7f66d40d11b0 legacy=0x7f66d40d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:43.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.667+0000 7f66d8ba1700 1 -- 172.21.15.189:0/600454897 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d40cef20 con 0x7f66d40cb870 2024-04-06T12:13:43.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.668+0000 7f66d8ba1700 1 --2- 172.21.15.189:0/600454897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66d40cb870 0x7f66d415c280 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f66c4009840 tx=0x7f66c4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a20c32fc95a0a5e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:43.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.668+0000 7f66cbfff700 1 -- 172.21.15.189:0/600454897 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d0001670 con 0x7f66d40cb870 2024-04-06T12:13:43.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.668+0000 7f66cbfff700 1 -- 172.21.15.189:0/600454897 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c4004030 con 0x7f66d40cb870 2024-04-06T12:13:43.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.668+0000 7f66cbfff700 1 -- 172.21.15.189:0/600454897 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c40043a0 con 0x7f66d40cb870 2024-04-06T12:13:43.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.668+0000 7f66db606700 1 -- 172.21.15.189:0/600454897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66d40cb870 msgr2=0x7f66d415c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:43.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.668+0000 7f66db606700 1 --2- 172.21.15.189:0/600454897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66d40cb870 0x7f66d415c280 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f66c4009840 tx=0x7f66c4009b50 comp rx=0 tx=0).stop 2024-04-06T12:13:43.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.668+0000 7f66db606700 1 -- 172.21.15.189:0/600454897 shutdown_connections 2024-04-06T12:13:43.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.668+0000 7f66db606700 1 --2- 172.21.15.189:0/600454897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66d40cfa00 0x7f66d40cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.668+0000 7f66db606700 1 --2- 172.21.15.189:0/600454897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66d40cb870 0x7f66d415c280 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.668+0000 7f66db606700 1 -- 172.21.15.189:0/600454897 >> 172.21.15.189:0/600454897 conn(0x7f66d40b2db0 msgr2=0x7f66d40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:43.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.668+0000 7f66db606700 1 -- 172.21.15.189:0/600454897 shutdown_connections 2024-04-06T12:13:43.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.668+0000 7f66db606700 1 -- 172.21.15.189:0/600454897 wait complete. 2024-04-06T12:13:43.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.670+0000 7f66db606700 1 Processor -- start 2024-04-06T12:13:43.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.670+0000 7f66db606700 1 -- start start 2024-04-06T12:13:43.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.670+0000 7f66db606700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66d40cb870 0x7f66d4154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:43.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.670+0000 7f66db606700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66d40cfa00 0x7f66d415cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:43.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.670+0000 7f66db606700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66d40d11b0 0x7f66d415d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:43.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.670+0000 7f66db606700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d4154dc0 con 0x7f66d40cfa00 2024-04-06T12:13:43.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.670+0000 7f66db606700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d40b4a50 con 0x7f66d40d11b0 2024-04-06T12:13:43.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.670+0000 7f66db606700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d40cab30 con 0x7f66d40cb870 2024-04-06T12:13:43.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.670+0000 7f66d8ba1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66d40cfa00 0x7f66d415cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:43.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.670+0000 7f66d8ba1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66d40cfa00 0x7f66d415cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57698/0 (socket says 172.21.15.189:57698) 2024-04-06T12:13:43.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.670+0000 7f66d8ba1700 1 -- 172.21.15.189:0/2385473354 learned_addr learned my addr 172.21.15.189:0/2385473354 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:43.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.671+0000 7f66d93a2700 1 --2- 172.21.15.189:0/2385473354 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66d40cb870 0x7f66d4154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:43.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.671+0000 7f66d9ba3700 1 --2- 172.21.15.189:0/2385473354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66d40d11b0 0x7f66d415d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:43.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.671+0000 7f66d8ba1700 1 -- 172.21.15.189:0/2385473354 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66d40cb870 msgr2=0x7f66d4154880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:43.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.671+0000 7f66d8ba1700 1 --2- 172.21.15.189:0/2385473354 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66d40cb870 0x7f66d4154880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.671+0000 7f66d8ba1700 1 -- 172.21.15.189:0/2385473354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66d40d11b0 msgr2=0x7f66d415d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:43.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.671+0000 7f66d8ba1700 1 --2- 172.21.15.189:0/2385473354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66d40d11b0 0x7f66d415d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.671+0000 7f66d8ba1700 1 -- 172.21.15.189:0/2385473354 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c40094a0 con 0x7f66d40cfa00 2024-04-06T12:13:43.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.671+0000 7f66d8ba1700 1 --2- 172.21.15.189:0/2385473354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66d40cfa00 0x7f66d415cf20 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f66c4000c00 tx=0x7f66c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:43.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.671+0000 7f66c9ffb700 1 -- 172.21.15.189:0/2385473354 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c4003df0 con 0x7f66d40cfa00 2024-04-06T12:13:43.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.671+0000 7f66c9ffb700 1 -- 172.21.15.189:0/2385473354 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c401f070 con 0x7f66d40cfa00 2024-04-06T12:13:43.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.671+0000 7f66db606700 1 -- 172.21.15.189:0/2385473354 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d40b4ca0 con 0x7f66d40cfa00 2024-04-06T12:13:43.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.671+0000 7f66c9ffb700 1 -- 172.21.15.189:0/2385473354 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c4011870 con 0x7f66d40cfa00 2024-04-06T12:13:43.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.671+0000 7f66db606700 1 -- 172.21.15.189:0/2385473354 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d40b5260 con 0x7f66d40cfa00 2024-04-06T12:13:43.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.672+0000 7f66db606700 1 -- 172.21.15.189:0/2385473354 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d4074990 con 0x7f66d40cfa00 2024-04-06T12:13:43.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.673+0000 7f66c9ffb700 1 -- 172.21.15.189:0/2385473354 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 11) v1 ==== 89321+0+0 (secure 0 0 0) 0x7f66c4021030 con 0x7f66d40cfa00 2024-04-06T12:13:43.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.673+0000 7f66c9ffb700 1 --2- 172.21.15.189:0/2385473354 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66b806ae90 0x7f66b806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:43.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.673+0000 7f66c9ffb700 1 -- 172.21.15.189:0/2385473354 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f66c408ac90 con 0x7f66d40cfa00 2024-04-06T12:13:43.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.673+0000 7f66d93a2700 1 --2- 172.21.15.189:0/2385473354 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66b806ae90 0x7f66b806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:43.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.673+0000 7f66d93a2700 1 --2- 172.21.15.189:0/2385473354 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66b806ae90 0x7f66b806d350 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f66d415a9a0 tx=0x7f66d0009040 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:43.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.677+0000 7f66c9ffb700 1 -- 172.21.15.189:0/2385473354 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f66c40559c0 con 0x7f66d40cfa00 2024-04-06T12:13:43.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.902+0000 7f66db606700 1 -- 172.21.15.189:0/2385473354 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f66d40cc060 con 0x7f66b806ae90 2024-04-06T12:13:43.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.949+0000 7f66c9ffb700 1 -- 172.21.15.189:0/2385473354 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f66d0007d30 con 0x7f66b806ae90 2024-04-06T12:13:43.947 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule activated for path /snap_test_dir1/dir7 2024-04-06T12:13:43.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.951+0000 7f66db606700 1 -- 172.21.15.189:0/2385473354 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66b806ae90 msgr2=0x7f66b806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:43.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.951+0000 7f66db606700 1 --2- 172.21.15.189:0/2385473354 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66b806ae90 0x7f66b806d350 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f66d415a9a0 tx=0x7f66d0009040 comp rx=0 tx=0).stop 2024-04-06T12:13:43.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.951+0000 7f66db606700 1 -- 172.21.15.189:0/2385473354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66d40cfa00 msgr2=0x7f66d415cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:43.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.951+0000 7f66db606700 1 --2- 172.21.15.189:0/2385473354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66d40cfa00 0x7f66d415cf20 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f66c4000c00 tx=0x7f66c4004750 comp rx=0 tx=0).stop 2024-04-06T12:13:43.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.951+0000 7f66db606700 1 -- 172.21.15.189:0/2385473354 shutdown_connections 2024-04-06T12:13:43.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.951+0000 7f66db606700 1 --2- 172.21.15.189:0/2385473354 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66b806ae90 0x7f66b806d350 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.951+0000 7f66db606700 1 --2- 172.21.15.189:0/2385473354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66d40d11b0 0x7f66d415d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.951+0000 7f66db606700 1 --2- 172.21.15.189:0/2385473354 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66d40cb870 0x7f66d4154880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.951+0000 7f66db606700 1 --2- 172.21.15.189:0/2385473354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66d40cfa00 0x7f66d415cf20 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:43.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.951+0000 7f66db606700 1 -- 172.21.15.189:0/2385473354 >> 172.21.15.189:0/2385473354 conn(0x7f66d40b2db0 msgr2=0x7f66d40d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:43.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.951+0000 7f66db606700 1 -- 172.21.15.189:0/2385473354 shutdown_connections 2024-04-06T12:13:43.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:43.951+0000 7f66db606700 1 -- 172.21.15.189:0/2385473354 wait complete. 2024-04-06T12:13:43.961 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2024-04-06T12:13:43.961 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir8 1m 2024-04-06T12:13:44.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.089+0000 7f0bd22d0700 1 Processor -- start 2024-04-06T12:13:44.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.089+0000 7f0bd22d0700 1 -- start start 2024-04-06T12:13:44.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.090+0000 7f0bd22d0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0bcc15d7b0 0x7f0bcc0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:44.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.090+0000 7f0bd22d0700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0bcc06a730 0x7f0bcc06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:44.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.090+0000 7f0bd22d0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0bcc15ddf0 0x7f0bcc15e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:44.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.090+0000 7f0bd22d0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bcc06b0f0 con 0x7f0bcc15ddf0 2024-04-06T12:13:44.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.090+0000 7f0bd22d0700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bcc0b3990 con 0x7f0bcc06a730 2024-04-06T12:13:44.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.090+0000 7f0bd22d0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc0b3060 con 0x7f0bcc15d7b0 2024-04-06T12:13:44.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.090+0000 7f0bd086d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0bcc15ddf0 0x7f0bcc15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:44.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.090+0000 7f0bd086d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0bcc15ddf0 0x7f0bcc15e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57702/0 (socket says 172.21.15.189:57702) 2024-04-06T12:13:44.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.090+0000 7f0bd086d700 1 -- 172.21.15.189:0/10906304 learned_addr learned my addr 172.21.15.189:0/10906304 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:44.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.090+0000 7f0bcbfff700 1 --2- 172.21.15.189:0/10906304 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0bcc15d7b0 0x7f0bcc0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:44.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.090+0000 7f0bcb7fe700 1 --2- 172.21.15.189:0/10906304 >> v2:172.21.15.192:3300/0 conn(0x7f0bcc06a730 0x7f0bcc06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:44.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.090+0000 7f0bd086d700 1 -- 172.21.15.189:0/10906304 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0bcc15d7b0 msgr2=0x7f0bcc0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:44.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.090+0000 7f0bd086d700 1 --2- 172.21.15.189:0/10906304 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0bcc15d7b0 0x7f0bcc0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.090+0000 7f0bd086d700 1 -- 172.21.15.189:0/10906304 >> v2:172.21.15.192:3300/0 conn(0x7f0bcc06a730 msgr2=0x7f0bcc06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:44.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.090+0000 7f0bd086d700 1 --2- 172.21.15.189:0/10906304 >> v2:172.21.15.192:3300/0 conn(0x7f0bcc06a730 0x7f0bcc06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.090+0000 7f0bd086d700 1 -- 172.21.15.189:0/10906304 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc0b3340 con 0x7f0bcc15ddf0 2024-04-06T12:13:44.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.090+0000 7f0bcb7fe700 1 --2- 172.21.15.189:0/10906304 >> v2:172.21.15.192:3300/0 conn(0x7f0bcc06a730 0x7f0bcc06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:13:44.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.090+0000 7f0bd086d700 1 --2- 172.21.15.189:0/10906304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bcc15ddf0 0x7f0bcc15e280 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc06cb90 tx=0x7f0bc400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a48ec67220d468f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:44.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.091+0000 7f0bcaffd700 1 -- 172.21.15.189:0/10906304 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc400e070 con 0x7f0bcc15ddf0 2024-04-06T12:13:44.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.091+0000 7f0bcaffd700 1 -- 172.21.15.189:0/10906304 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc400b9c0 con 0x7f0bcc15ddf0 2024-04-06T12:13:44.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.091+0000 7f0bcaffd700 1 -- 172.21.15.189:0/10906304 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc400bd70 con 0x7f0bcc15ddf0 2024-04-06T12:13:44.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.091+0000 7f0bd22d0700 1 -- 172.21.15.189:0/10906304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bcc15ddf0 msgr2=0x7f0bcc15e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:44.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.091+0000 7f0bd22d0700 1 --2- 172.21.15.189:0/10906304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bcc15ddf0 0x7f0bcc15e280 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc06cb90 tx=0x7f0bc400b330 comp rx=0 tx=0).stop 2024-04-06T12:13:44.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.091+0000 7f0bd22d0700 1 -- 172.21.15.189:0/10906304 shutdown_connections 2024-04-06T12:13:44.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.091+0000 7f0bd22d0700 1 --2- 172.21.15.189:0/10906304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bcc15ddf0 0x7f0bcc15e280 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.091+0000 7f0bd22d0700 1 --2- 172.21.15.189:0/10906304 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0bcc15d7b0 0x7f0bcc0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.091+0000 7f0bd22d0700 1 --2- 172.21.15.189:0/10906304 >> v2:172.21.15.192:3300/0 conn(0x7f0bcc06a730 0x7f0bcc06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.091+0000 7f0bd22d0700 1 -- 172.21.15.189:0/10906304 >> 172.21.15.189:0/10906304 conn(0x7f0bcc0d00b0 msgr2=0x7f0bcc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:44.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.091+0000 7f0bd22d0700 1 -- 172.21.15.189:0/10906304 shutdown_connections 2024-04-06T12:13:44.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.091+0000 7f0bd22d0700 1 -- 172.21.15.189:0/10906304 wait complete. 2024-04-06T12:13:44.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.092+0000 7f0bd22d0700 1 Processor -- start 2024-04-06T12:13:44.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.093+0000 7f0bd22d0700 1 -- start start 2024-04-06T12:13:44.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.093+0000 7f0bd22d0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bcc06a730 0x7f0bcc0cf480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:44.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.093+0000 7f0bd22d0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0bcc15d7b0 0x7f0bcc14c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:44.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.093+0000 7f0bd22d0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bcc15ddf0 0x7f0bcc1f5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:44.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.093+0000 7f0bd22d0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc0cfad0 con 0x7f0bcc15ddf0 2024-04-06T12:13:44.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.093+0000 7f0bd22d0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc1605c0 con 0x7f0bcc06a730 2024-04-06T12:13:44.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.093+0000 7f0bd22d0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc15f800 con 0x7f0bcc15d7b0 2024-04-06T12:13:44.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.093+0000 7f0bd086d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bcc15ddf0 0x7f0bcc1f5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:44.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.093+0000 7f0bd086d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bcc15ddf0 0x7f0bcc1f5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57716/0 (socket says 172.21.15.189:57716) 2024-04-06T12:13:44.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.093+0000 7f0bd086d700 1 -- 172.21.15.189:0/3419377361 learned_addr learned my addr 172.21.15.189:0/3419377361 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:44.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.093+0000 7f0bcbfff700 1 --2- 172.21.15.189:0/3419377361 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bcc06a730 0x7f0bcc0cf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:44.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.093+0000 7f0bcb7fe700 1 --2- 172.21.15.189:0/3419377361 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0bcc15d7b0 0x7f0bcc14c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:44.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.094+0000 7f0bd086d700 1 -- 172.21.15.189:0/3419377361 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0bcc15d7b0 msgr2=0x7f0bcc14c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:44.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.094+0000 7f0bd086d700 1 --2- 172.21.15.189:0/3419377361 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0bcc15d7b0 0x7f0bcc14c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.094+0000 7f0bd086d700 1 -- 172.21.15.189:0/3419377361 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bcc06a730 msgr2=0x7f0bcc0cf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:44.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.094+0000 7f0bd086d700 1 --2- 172.21.15.189:0/3419377361 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bcc06a730 0x7f0bcc0cf480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.094+0000 7f0bd086d700 1 -- 172.21.15.189:0/3419377361 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc4009b30 con 0x7f0bcc15ddf0 2024-04-06T12:13:44.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.094+0000 7f0bd086d700 1 --2- 172.21.15.189:0/3419377361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bcc15ddf0 0x7f0bcc1f5f70 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400b370 tx=0x7f0bc4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:44.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.094+0000 7f0bc8ff9700 1 -- 172.21.15.189:0/3419377361 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4004310 con 0x7f0bcc15ddf0 2024-04-06T12:13:44.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.094+0000 7f0bc8ff9700 1 -- 172.21.15.189:0/3419377361 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc4004470 con 0x7f0bcc15ddf0 2024-04-06T12:13:44.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.094+0000 7f0bd22d0700 1 -- 172.21.15.189:0/3419377361 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bcc15fae0 con 0x7f0bcc15ddf0 2024-04-06T12:13:44.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.094+0000 7f0bc8ff9700 1 -- 172.21.15.189:0/3419377361 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4004820 con 0x7f0bcc15ddf0 2024-04-06T12:13:44.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.094+0000 7f0bd22d0700 1 -- 172.21.15.189:0/3419377361 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bcc1600a0 con 0x7f0bcc15ddf0 2024-04-06T12:13:44.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.095+0000 7f0bd22d0700 1 -- 172.21.15.189:0/3419377361 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bcc067bb0 con 0x7f0bcc15ddf0 2024-04-06T12:13:44.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.096+0000 7f0bc8ff9700 1 -- 172.21.15.189:0/3419377361 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 11) v1 ==== 89321+0+0 (secure 0 0 0) 0x7f0bc401c440 con 0x7f0bcc15ddf0 2024-04-06T12:13:44.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.096+0000 7f0bc8ff9700 1 --2- 172.21.15.189:0/3419377361 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0bb806aec0 0x7f0bb806d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:44.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.096+0000 7f0bc8ff9700 1 -- 172.21.15.189:0/3419377361 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f0bc408ae40 con 0x7f0bcc15ddf0 2024-04-06T12:13:44.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.096+0000 7f0bcbfff700 1 --2- 172.21.15.189:0/3419377361 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0bb806aec0 0x7f0bb806d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:44.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.096+0000 7f0bcbfff700 1 --2- 172.21.15.189:0/3419377361 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0bb806aec0 0x7f0bb806d380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000aae0 tx=0x7f0bc0008040 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:44.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.100+0000 7f0bc8ff9700 1 -- 172.21.15.189:0/3419377361 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0bc4055ae0 con 0x7f0bcc15ddf0 2024-04-06T12:13:44.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.324+0000 7f0bd22d0700 1 -- 172.21.15.189:0/3419377361 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bcc15e7c0 con 0x7f0bb806aec0 2024-04-06T12:13:44.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.385+0000 7f0bc8ff9700 1 -- 172.21.15.189:0/3419377361 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f0bc000c040 con 0x7f0bb806aec0 2024-04-06T12:13:44.383 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule activated for path /snap_test_dir1/dir8 2024-04-06T12:13:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.387+0000 7f0bd22d0700 1 -- 172.21.15.189:0/3419377361 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0bb806aec0 msgr2=0x7f0bb806d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.388+0000 7f0bd22d0700 1 --2- 172.21.15.189:0/3419377361 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0bb806aec0 0x7f0bb806d380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000aae0 tx=0x7f0bc0008040 comp rx=0 tx=0).stop 2024-04-06T12:13:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.388+0000 7f0bd22d0700 1 -- 172.21.15.189:0/3419377361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bcc15ddf0 msgr2=0x7f0bcc1f5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.388+0000 7f0bd22d0700 1 --2- 172.21.15.189:0/3419377361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bcc15ddf0 0x7f0bcc1f5f70 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400b370 tx=0x7f0bc4004ad0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.388+0000 7f0bd22d0700 1 -- 172.21.15.189:0/3419377361 shutdown_connections 2024-04-06T12:13:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.388+0000 7f0bd22d0700 1 --2- 172.21.15.189:0/3419377361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bcc15ddf0 0x7f0bcc1f5f70 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.388+0000 7f0bd22d0700 1 --2- 172.21.15.189:0/3419377361 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0bb806aec0 0x7f0bb806d380 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.388+0000 7f0bd22d0700 1 --2- 172.21.15.189:0/3419377361 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bcc06a730 0x7f0bcc0cf480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.388+0000 7f0bd22d0700 1 --2- 172.21.15.189:0/3419377361 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0bcc15d7b0 0x7f0bcc14c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.388+0000 7f0bd22d0700 1 -- 172.21.15.189:0/3419377361 >> 172.21.15.189:0/3419377361 conn(0x7f0bcc0d00b0 msgr2=0x7f0bcc06f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:44.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.388+0000 7f0bd22d0700 1 -- 172.21.15.189:0/3419377361 shutdown_connections 2024-04-06T12:13:44.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.388+0000 7f0bd22d0700 1 -- 172.21.15.189:0/3419377361 wait complete. 2024-04-06T12:13:44.398 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2024-04-06T12:13:44.398 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir9 1m 2024-04-06T12:13:44.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.524+0000 7ff3e8b3f700 1 Processor -- start 2024-04-06T12:13:44.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.524+0000 7ff3e8b3f700 1 -- start start 2024-04-06T12:13:44.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.525+0000 7ff3e8b3f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff3e40cd3e0 0x7ff3e40cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:44.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.525+0000 7ff3e8b3f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff3e415d7c0 0x7ff3e406b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:44.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.525+0000 7ff3e8b3f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3e406b6f0 0x7ff3e41610f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:44.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.525+0000 7ff3e8b3f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e40cdfa0 con 0x7ff3e415d7c0 2024-04-06T12:13:44.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.525+0000 7ff3e8b3f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e406ceb0 con 0x7ff3e40cd3e0 2024-04-06T12:13:44.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.525+0000 7ff3e8b3f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e406d750 con 0x7ff3e406b6f0 2024-04-06T12:13:44.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.525+0000 7ff3e1d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff3e415d7c0 0x7ff3e406b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:44.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.525+0000 7ff3e1d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff3e415d7c0 0x7ff3e406b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57726/0 (socket says 172.21.15.189:57726) 2024-04-06T12:13:44.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.525+0000 7ff3e1d9b700 1 -- 172.21.15.189:0/232507959 learned_addr learned my addr 172.21.15.189:0/232507959 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:44.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.525+0000 7ff3e2d9d700 1 --2- 172.21.15.189:0/232507959 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3e406b6f0 0x7ff3e41610f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:44.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.525+0000 7ff3e259c700 1 --2- 172.21.15.189:0/232507959 >> v2:172.21.15.192:3300/0 conn(0x7ff3e40cd3e0 0x7ff3e40cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:44.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.525+0000 7ff3e1d9b700 1 -- 172.21.15.189:0/232507959 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3e406b6f0 msgr2=0x7ff3e41610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:44.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.525+0000 7ff3e1d9b700 1 --2- 172.21.15.189:0/232507959 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3e406b6f0 0x7ff3e41610f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.525+0000 7ff3e1d9b700 1 -- 172.21.15.189:0/232507959 >> v2:172.21.15.192:3300/0 conn(0x7ff3e40cd3e0 msgr2=0x7ff3e40cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:13:44.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.525+0000 7ff3e1d9b700 1 --2- 172.21.15.189:0/232507959 >> v2:172.21.15.192:3300/0 conn(0x7ff3e40cd3e0 0x7ff3e40cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.525+0000 7ff3e1d9b700 1 -- 172.21.15.189:0/232507959 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e40cdd00 con 0x7ff3e415d7c0 2024-04-06T12:13:44.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.525+0000 7ff3e259c700 1 --2- 172.21.15.189:0/232507959 >> v2:172.21.15.192:3300/0 conn(0x7ff3e40cd3e0 0x7ff3e40cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:13:44.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.525+0000 7ff3e1d9b700 1 --2- 172.21.15.189:0/232507959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3e415d7c0 0x7ff3e406b1b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff3e40723a0 tx=0x7ff3d8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c50e3bd9bec0f3fd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:44.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.525+0000 7ff3e159a700 1 -- 172.21.15.189:0/232507959 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d800b870 con 0x7ff3e415d7c0 2024-04-06T12:13:44.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.526+0000 7ff3e159a700 1 -- 172.21.15.189:0/232507959 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d800b9d0 con 0x7ff3e415d7c0 2024-04-06T12:13:44.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.526+0000 7ff3e159a700 1 -- 172.21.15.189:0/232507959 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d800bd60 con 0x7ff3e415d7c0 2024-04-06T12:13:44.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.526+0000 7ff3e8b3f700 1 -- 172.21.15.189:0/232507959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3e415d7c0 msgr2=0x7ff3e406b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:44.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.526+0000 7ff3e8b3f700 1 --2- 172.21.15.189:0/232507959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3e415d7c0 0x7ff3e406b1b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff3e40723a0 tx=0x7ff3d8009bd0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.526+0000 7ff3e8b3f700 1 -- 172.21.15.189:0/232507959 shutdown_connections 2024-04-06T12:13:44.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.526+0000 7ff3e8b3f700 1 --2- 172.21.15.189:0/232507959 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3e406b6f0 0x7ff3e41610f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.526+0000 7ff3e8b3f700 1 --2- 172.21.15.189:0/232507959 >> v2:172.21.15.192:3300/0 conn(0x7ff3e40cd3e0 0x7ff3e40cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.526+0000 7ff3e8b3f700 1 --2- 172.21.15.189:0/232507959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3e415d7c0 0x7ff3e406b1b0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.526+0000 7ff3e8b3f700 1 -- 172.21.15.189:0/232507959 >> 172.21.15.189:0/232507959 conn(0x7ff3e40cc290 msgr2=0x7ff3e40cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:44.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.526+0000 7ff3e8b3f700 1 -- 172.21.15.189:0/232507959 shutdown_connections 2024-04-06T12:13:44.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.526+0000 7ff3e8b3f700 1 -- 172.21.15.189:0/232507959 wait complete. 2024-04-06T12:13:44.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.527+0000 7ff3e8b3f700 1 Processor -- start 2024-04-06T12:13:44.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.528+0000 7ff3e8b3f700 1 -- start start 2024-04-06T12:13:44.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.528+0000 7ff3e8b3f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3e406b6f0 0x7ff3e40c56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:44.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.528+0000 7ff3e8b3f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3e40cd3e0 0x7ff3e41f56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:44.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.528+0000 7ff3e8b3f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3e415d7c0 0x7ff3e41f5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:44.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.528+0000 7ff3e8b3f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e4154180 con 0x7ff3e406b6f0 2024-04-06T12:13:44.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.528+0000 7ff3e8b3f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e41f6140 con 0x7ff3e415d7c0 2024-04-06T12:13:44.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.528+0000 7ff3e8b3f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e40c4f40 con 0x7ff3e40cd3e0 2024-04-06T12:13:44.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.528+0000 7ff3e259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3e406b6f0 0x7ff3e40c56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:44.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.528+0000 7ff3e259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3e406b6f0 0x7ff3e40c56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57742/0 (socket says 172.21.15.189:57742) 2024-04-06T12:13:44.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.528+0000 7ff3e259c700 1 -- 172.21.15.189:0/966873142 learned_addr learned my addr 172.21.15.189:0/966873142 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:13:44.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.529+0000 7ff3e2d9d700 1 --2- 172.21.15.189:0/966873142 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3e415d7c0 0x7ff3e41f5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:44.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.529+0000 7ff3e1d9b700 1 --2- 172.21.15.189:0/966873142 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3e40cd3e0 0x7ff3e41f56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:44.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.529+0000 7ff3e259c700 1 -- 172.21.15.189:0/966873142 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3e40cd3e0 msgr2=0x7ff3e41f56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:44.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.529+0000 7ff3e259c700 1 --2- 172.21.15.189:0/966873142 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3e40cd3e0 0x7ff3e41f56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.529+0000 7ff3e259c700 1 -- 172.21.15.189:0/966873142 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3e415d7c0 msgr2=0x7ff3e41f5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:44.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.529+0000 7ff3e259c700 1 --2- 172.21.15.189:0/966873142 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3e415d7c0 0x7ff3e41f5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.529+0000 7ff3e259c700 1 -- 172.21.15.189:0/966873142 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d8009550 con 0x7ff3e406b6f0 2024-04-06T12:13:44.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.529+0000 7ff3e259c700 1 --2- 172.21.15.189:0/966873142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3e406b6f0 0x7ff3e40c56f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400e760 tx=0x7ff3d400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:44.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.529+0000 7ff3d2ffd700 1 -- 172.21.15.189:0/966873142 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4013610 con 0x7ff3e406b6f0 2024-04-06T12:13:44.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.529+0000 7ff3d2ffd700 1 -- 172.21.15.189:0/966873142 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d4013770 con 0x7ff3e406b6f0 2024-04-06T12:13:44.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.529+0000 7ff3e8b3f700 1 -- 172.21.15.189:0/966873142 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e41537e0 con 0x7ff3e406b6f0 2024-04-06T12:13:44.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.529+0000 7ff3d2ffd700 1 -- 172.21.15.189:0/966873142 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4013b20 con 0x7ff3e406b6f0 2024-04-06T12:13:44.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.529+0000 7ff3e8b3f700 1 -- 172.21.15.189:0/966873142 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e4153da0 con 0x7ff3e406b6f0 2024-04-06T12:13:44.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.530+0000 7ff3e8b3f700 1 -- 172.21.15.189:0/966873142 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e41543f0 con 0x7ff3e406b6f0 2024-04-06T12:13:44.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.531+0000 7ff3d2ffd700 1 -- 172.21.15.189:0/966873142 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 11) v1 ==== 89321+0+0 (secure 0 0 0) 0x7ff3d4013c80 con 0x7ff3e406b6f0 2024-04-06T12:13:44.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.531+0000 7ff3d2ffd700 1 --2- 172.21.15.189:0/966873142 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff3c406ae90 0x7ff3c406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:13:44.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.531+0000 7ff3d2ffd700 1 -- 172.21.15.189:0/966873142 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7ff3d4089570 con 0x7ff3e406b6f0 2024-04-06T12:13:44.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.531+0000 7ff3e1d9b700 1 --2- 172.21.15.189:0/966873142 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff3c406ae90 0x7ff3c406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:13:44.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.531+0000 7ff3e1d9b700 1 --2- 172.21.15.189:0/966873142 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff3c406ae90 0x7ff3c406d350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff3e40ae690 tx=0x7ff3d8008030 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:13:44.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.535+0000 7ff3d2ffd700 1 -- 172.21.15.189:0/966873142 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff3d4054210 con 0x7ff3e406b6f0 2024-04-06T12:13:44.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.755+0000 7ff3e8b3f700 1 -- 172.21.15.189:0/966873142 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3e406d3e0 con 0x7ff3c406ae90 2024-04-06T12:13:44.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.809+0000 7ff3d2ffd700 1 -- 172.21.15.189:0/966873142 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ff3d801f070 con 0x7ff3c406ae90 2024-04-06T12:13:44.809 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule activated for path /snap_test_dir1/dir9 2024-04-06T12:13:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.812+0000 7ff3e8b3f700 1 -- 172.21.15.189:0/966873142 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff3c406ae90 msgr2=0x7ff3c406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.812+0000 7ff3e8b3f700 1 --2- 172.21.15.189:0/966873142 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff3c406ae90 0x7ff3c406d350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff3e40ae690 tx=0x7ff3d8008030 comp rx=0 tx=0).stop 2024-04-06T12:13:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.812+0000 7ff3e8b3f700 1 -- 172.21.15.189:0/966873142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3e406b6f0 msgr2=0x7ff3e40c56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:13:44.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.812+0000 7ff3e8b3f700 1 --2- 172.21.15.189:0/966873142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3e406b6f0 0x7ff3e40c56f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400e760 tx=0x7ff3d400eb20 comp rx=0 tx=0).stop 2024-04-06T12:13:44.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.812+0000 7ff3e8b3f700 1 -- 172.21.15.189:0/966873142 shutdown_connections 2024-04-06T12:13:44.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.812+0000 7ff3e8b3f700 1 --2- 172.21.15.189:0/966873142 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff3c406ae90 0x7ff3c406d350 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.812+0000 7ff3e8b3f700 1 --2- 172.21.15.189:0/966873142 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3e415d7c0 0x7ff3e41f5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.812+0000 7ff3e8b3f700 1 --2- 172.21.15.189:0/966873142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3e406b6f0 0x7ff3e40c56f0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.812+0000 7ff3e8b3f700 1 --2- 172.21.15.189:0/966873142 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3e40cd3e0 0x7ff3e41f56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:13:44.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.812+0000 7ff3e8b3f700 1 -- 172.21.15.189:0/966873142 >> 172.21.15.189:0/966873142 conn(0x7ff3e40cc290 msgr2=0x7ff3e40ae800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:13:44.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.812+0000 7ff3e8b3f700 1 -- 172.21.15.189:0/966873142 shutdown_connections 2024-04-06T12:13:44.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:13:44.812+0000 7ff3e8b3f700 1 -- 172.21.15.189:0/966873142 wait complete. 2024-04-06T12:13:44.823 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2024-04-06T12:14:00.116 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:14:00.117+0000 7ff93756a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:14:00.117 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:14:00.117+0000 7ff93756a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:14:00.117 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:14:00.117+0000 7ff93756a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:14:00.117 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:14:00.117+0000 7ff93756a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:14:00.117 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:14:00.117+0000 7ff93756a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:14:10.402 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:14:10.406 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:14:40.480 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:14:40.484 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:15:10.563 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:15:10.567 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:15:40.641 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:15:40.645 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:16:10.707 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:16:10.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:16:40.792 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:16:40.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:17:00.745 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:17:00.747+0000 7ff93856c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:17:00.746 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:17:00.747+0000 7ff93856c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:17:00.746 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:17:00.747+0000 7ff93856c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:17:00.746 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:17:00.747+0000 7ff93856c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:17:00.746 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:17:00.747+0000 7ff93856c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:17:10.864 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:17:10.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:17:40.928 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:17:40.932 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:18:11.009 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:18:11.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:18:41.089 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:18:41.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:19:11.163 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:19:11.171 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:19:41.246 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:19:41.253 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:20:11.332 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:20:11.336 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:20:41.409 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:20:41.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:21:11.482 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:21:11.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:21:41.571 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:21:41.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:22:11.642 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:22:11.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:22:41.715 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:22:41.719 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:23:11.831 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:23:11.834 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:23:41.932 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:23:41.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:24:12.038 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:24:12.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:24:19.824 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir0 1m 2024-04-06T12:24:19.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.964+0000 7f6a5b747700 1 Processor -- start 2024-04-06T12:24:19.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.964+0000 7f6a5b747700 1 -- start start 2024-04-06T12:24:19.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.965+0000 7f6a5b747700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a5406ae40 0x7f6a540b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:19.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.965+0000 7f6a5b747700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6a5415ddf0 0x7f6a5415e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:19.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.965+0000 7f6a5b747700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a540b3990 con 0x7f6a5415d7b0 2024-04-06T12:24:19.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.965+0000 7f6a5b747700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a540b3060 con 0x7f6a5415ddf0 2024-04-06T12:24:19.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.965+0000 7f6a5b747700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a540b3340 con 0x7f6a5406ae40 2024-04-06T12:24:19.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.965+0000 7f6a594e3700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6a5415d7b0 0x7f6a5406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48650/0 (socket says 172.21.15.189:48650) 2024-04-06T12:24:19.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.965+0000 7f6a594e3700 1 -- 172.21.15.189:0/689516855 learned_addr learned my addr 172.21.15.189:0/689516855 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:19.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a53fff700 1 -- 172.21.15.189:0/689516855 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599019513 0 0) 0x7f6a540b3990 con 0x7f6a5415d7b0 2024-04-06T12:24:19.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a58ce2700 1 --2- 172.21.15.189:0/689516855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a5406ae40 0x7f6a540b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:19.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a59ce4700 1 --2- 172.21.15.189:0/689516855 >> v2:172.21.15.192:3300/0 conn(0x7f6a5415ddf0 0x7f6a5415e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:19.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a53fff700 1 -- 172.21.15.189:0/689516855 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a3c0035b0 con 0x7f6a5415d7b0 2024-04-06T12:24:19.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a53fff700 1 -- 172.21.15.189:0/689516855 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4206891400 0 0) 0x7f6a3c0035b0 con 0x7f6a5415d7b0 2024-04-06T12:24:19.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a53fff700 1 -- 172.21.15.189:0/689516855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a5406ae40 msgr2=0x7f6a540b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:19.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a53fff700 1 --2- 172.21.15.189:0/689516855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a5406ae40 0x7f6a540b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:19.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a53fff700 1 -- 172.21.15.189:0/689516855 >> v2:172.21.15.192:3300/0 conn(0x7f6a5415ddf0 msgr2=0x7f6a5415e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:19.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a53fff700 1 --2- 172.21.15.189:0/689516855 >> v2:172.21.15.192:3300/0 conn(0x7f6a5415ddf0 0x7f6a5415e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:19.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a53fff700 1 -- 172.21.15.189:0/689516855 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a540b3620 con 0x7f6a5415d7b0 2024-04-06T12:24:19.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a53fff700 1 -- 172.21.15.189:0/689516855 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f6a440019f0 con 0x7f6a5415d7b0 2024-04-06T12:24:19.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a53fff700 1 -- 172.21.15.189:0/689516855 >> v1:172.21.15.189:6789/0 conn(0x7f6a5415d7b0 legacy=0x7f6a5406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:19.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a53fff700 1 --2- 172.21.15.189:0/689516855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a5406ae40 0x7f6a3c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:19.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a53fff700 1 --2- 172.21.15.189:0/689516855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a3c003f80 0x7f6a3c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:19.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a53fff700 1 --2- 172.21.15.189:0/689516855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a3c004d90 0x7f6a3c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:19.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a53fff700 1 -- 172.21.15.189:0/689516855 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a540b3060 con 0x7f6a3c004d90 2024-04-06T12:24:19.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a53fff700 1 -- 172.21.15.189:0/689516855 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a540b3340 con 0x7f6a3c003f80 2024-04-06T12:24:19.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a53fff700 1 -- 172.21.15.189:0/689516855 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3c00ba60 con 0x7f6a5406ae40 2024-04-06T12:24:19.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a59ce4700 1 --2- 172.21.15.189:0/689516855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a3c004d90 0x7f6a3c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:19.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a594e3700 1 --2- 172.21.15.189:0/689516855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a5406ae40 0x7f6a3c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:19.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.966+0000 7f6a58ce2700 1 --2- 172.21.15.189:0/689516855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a3c003f80 0x7f6a3c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:19.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.967+0000 7f6a59ce4700 1 -- 172.21.15.189:0/689516855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a5406ae40 msgr2=0x7f6a3c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:19.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.967+0000 7f6a59ce4700 1 --2- 172.21.15.189:0/689516855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a5406ae40 0x7f6a3c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:19.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.967+0000 7f6a59ce4700 1 -- 172.21.15.189:0/689516855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a3c003f80 msgr2=0x7f6a3c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:19.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.967+0000 7f6a59ce4700 1 --2- 172.21.15.189:0/689516855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a3c003f80 0x7f6a3c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:19.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.967+0000 7f6a59ce4700 1 -- 172.21.15.189:0/689516855 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3c0035b0 con 0x7f6a3c004d90 2024-04-06T12:24:19.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.967+0000 7f6a59ce4700 1 --2- 172.21.15.189:0/689516855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a3c004d90 0x7f6a3c005210 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c009b90 tx=0x7f6a3c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ec091a2ae324c61 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:19.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.967+0000 7f6a53fff700 1 -- 172.21.15.189:0/689516855 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c00e6c0 con 0x7f6a3c004d90 2024-04-06T12:24:19.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.967+0000 7f6a53fff700 1 -- 172.21.15.189:0/689516855 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4c00e820 con 0x7f6a3c004d90 2024-04-06T12:24:19.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.967+0000 7f6a53fff700 1 -- 172.21.15.189:0/689516855 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c00eb90 con 0x7f6a3c004d90 2024-04-06T12:24:19.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.967+0000 7f6a5b747700 1 -- 172.21.15.189:0/689516855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a3c004d90 msgr2=0x7f6a3c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:19.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.967+0000 7f6a5b747700 1 --2- 172.21.15.189:0/689516855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a3c004d90 0x7f6a3c005210 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c009b90 tx=0x7f6a3c005880 comp rx=0 tx=0).stop 2024-04-06T12:24:19.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.967+0000 7f6a59ce4700 1 -- 172.21.15.189:0/689516855 reap_dead start 2024-04-06T12:24:19.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.967+0000 7f6a5b747700 1 -- 172.21.15.189:0/689516855 shutdown_connections 2024-04-06T12:24:19.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.967+0000 7f6a5b747700 1 -- 172.21.15.189:0/689516855 >> 172.21.15.189:0/689516855 conn(0x7f6a540d00b0 msgr2=0x7f6a540d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:19.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.968+0000 7f6a5b747700 1 -- 172.21.15.189:0/689516855 shutdown_connections 2024-04-06T12:24:19.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.968+0000 7f6a5b747700 1 -- 172.21.15.189:0/689516855 wait complete. 2024-04-06T12:24:19.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.969+0000 7f6a5b747700 1 Processor -- start 2024-04-06T12:24:19.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.969+0000 7f6a5b747700 1 -- start start 2024-04-06T12:24:19.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a5b747700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a3c004d90 0x7f6a541f3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:19.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a5b747700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a541436c0 0x7f6a5415f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:19.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a5b747700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a5415fb00 0x7f6a540b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:19.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a5b747700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5415ff80 con 0x7f6a3c004d90 2024-04-06T12:24:19.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a5b747700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a541f3840 con 0x7f6a5415fb00 2024-04-06T12:24:19.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a5b747700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a541462c0 con 0x7f6a541436c0 2024-04-06T12:24:19.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a594e3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a3c004d90 0x7f6a541f3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:19.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a594e3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a3c004d90 0x7f6a541f3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33744/0 (socket says 172.21.15.189:33744) 2024-04-06T12:24:19.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a594e3700 1 -- 172.21.15.189:0/1723799085 learned_addr learned my addr 172.21.15.189:0/1723799085 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:19.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a59ce4700 1 --2- 172.21.15.189:0/1723799085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a5415fb00 0x7f6a540b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:19.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a58ce2700 1 --2- 172.21.15.189:0/1723799085 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a541436c0 0x7f6a5415f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:19.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a594e3700 1 -- 172.21.15.189:0/1723799085 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a541436c0 msgr2=0x7f6a5415f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:19.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a594e3700 1 --2- 172.21.15.189:0/1723799085 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a541436c0 0x7f6a5415f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:19.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a594e3700 1 -- 172.21.15.189:0/1723799085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a5415fb00 msgr2=0x7f6a540b0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:19.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a594e3700 1 --2- 172.21.15.189:0/1723799085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a5415fb00 0x7f6a540b0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:19.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a594e3700 1 -- 172.21.15.189:0/1723799085 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a541465a0 con 0x7f6a3c004d90 2024-04-06T12:24:19.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a594e3700 1 --2- 172.21.15.189:0/1723799085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a3c004d90 0x7f6a541f3300 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6a44001330 tx=0x7f6a4400a8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:19.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a51ffb700 1 -- 172.21.15.189:0/1723799085 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a440133f0 con 0x7f6a3c004d90 2024-04-06T12:24:19.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a51ffb700 1 -- 172.21.15.189:0/1723799085 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a44013570 con 0x7f6a3c004d90 2024-04-06T12:24:19.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a5b747700 1 -- 172.21.15.189:0/1723799085 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a54146880 con 0x7f6a3c004d90 2024-04-06T12:24:19.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.970+0000 7f6a51ffb700 1 -- 172.21.15.189:0/1723799085 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a44013920 con 0x7f6a3c004d90 2024-04-06T12:24:19.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.971+0000 7f6a5b747700 1 -- 172.21.15.189:0/1723799085 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a54146e40 con 0x7f6a3c004d90 2024-04-06T12:24:19.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.972+0000 7f6a5b747700 1 -- 172.21.15.189:0/1723799085 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a54067bb0 con 0x7f6a3c004d90 2024-04-06T12:24:19.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.972+0000 7f6a51ffb700 1 -- 172.21.15.189:0/1723799085 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f6a44013a80 con 0x7f6a3c004d90 2024-04-06T12:24:19.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.972+0000 7f6a51ffb700 1 --2- 172.21.15.189:0/1723799085 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6a4006ae90 0x7f6a4006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:19.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.973+0000 7f6a58ce2700 1 --2- 172.21.15.189:0/1723799085 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6a4006ae90 0x7f6a4006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:19.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.973+0000 7f6a51ffb700 1 -- 172.21.15.189:0/1723799085 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f6a4408a700 con 0x7f6a3c004d90 2024-04-06T12:24:19.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.973+0000 7f6a58ce2700 1 --2- 172.21.15.189:0/1723799085 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6a4006ae90 0x7f6a4006d350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6a5415acd0 tx=0x7f6a48009210 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:19.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:19.977+0000 7f6a51ffb700 1 -- 172.21.15.189:0/1723799085 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6a44055490 con 0x7f6a3c004d90 2024-04-06T12:24:20.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.201+0000 7f6a5b747700 1 -- 172.21.15.189:0/1723799085 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a5415e7e0 con 0x7f6a4006ae90 2024-04-06T12:24:20.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.273+0000 7f6a51ffb700 1 -- 172.21.15.189:0/1723799085 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f6a480098d0 con 0x7f6a4006ae90 2024-04-06T12:24:20.271 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2024-04-06T12:24:20.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.275+0000 7f6a5b747700 1 -- 172.21.15.189:0/1723799085 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6a4006ae90 msgr2=0x7f6a4006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:20.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.275+0000 7f6a5b747700 1 --2- 172.21.15.189:0/1723799085 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6a4006ae90 0x7f6a4006d350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6a5415acd0 tx=0x7f6a48009210 comp rx=0 tx=0).stop 2024-04-06T12:24:20.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.275+0000 7f6a5b747700 1 -- 172.21.15.189:0/1723799085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a3c004d90 msgr2=0x7f6a541f3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:20.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.275+0000 7f6a5b747700 1 --2- 172.21.15.189:0/1723799085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a3c004d90 0x7f6a541f3300 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6a44001330 tx=0x7f6a4400a8f0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.275+0000 7f6a5b747700 1 -- 172.21.15.189:0/1723799085 shutdown_connections 2024-04-06T12:24:20.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.275+0000 7f6a5b747700 1 --2- 172.21.15.189:0/1723799085 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6a4006ae90 0x7f6a4006d350 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.275+0000 7f6a5b747700 1 --2- 172.21.15.189:0/1723799085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a5415fb00 0x7f6a540b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.275+0000 7f6a5b747700 1 --2- 172.21.15.189:0/1723799085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a3c004d90 0x7f6a541f3300 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.275+0000 7f6a5b747700 1 --2- 172.21.15.189:0/1723799085 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a541436c0 0x7f6a5415f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.275+0000 7f6a5b747700 1 -- 172.21.15.189:0/1723799085 >> 172.21.15.189:0/1723799085 conn(0x7f6a540d00b0 msgr2=0x7f6a5415ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:20.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.275+0000 7f6a5b747700 1 -- 172.21.15.189:0/1723799085 shutdown_connections 2024-04-06T12:24:20.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.275+0000 7f6a5b747700 1 -- 172.21.15.189:0/1723799085 wait complete. 2024-04-06T12:24:20.284 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2024-04-06T12:24:20.284 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir1 1m 2024-04-06T12:24:20.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.412+0000 7f5e513c8700 1 Processor -- start 2024-04-06T12:24:20.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.412+0000 7f5e513c8700 1 -- start start 2024-04-06T12:24:20.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.413+0000 7f5e513c8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5e4c0b4ed0 0x7f5e4c0b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:20.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.413+0000 7f5e513c8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5e4c0d2880 0x7f5e4c0d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:20.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.413+0000 7f5e513c8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e4c0b5cc0 con 0x7f5e4c15d590 2024-04-06T12:24:20.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.413+0000 7f5e513c8700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e4c0b58c0 con 0x7f5e4c0b4ed0 2024-04-06T12:24:20.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.413+0000 7f5e513c8700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e4c0d1500 con 0x7f5e4c0d2880 2024-04-06T12:24:20.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.413+0000 7f5e4a7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5e4c0b4ed0 0x7f5e4c0b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:20.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.413+0000 7f5e4a7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5e4c0b4ed0 0x7f5e4c0b5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33754/0 (socket says 172.21.15.189:33754) 2024-04-06T12:24:20.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.413+0000 7f5e4a7fc700 1 -- 172.21.15.189:0/2384644385 learned_addr learned my addr 172.21.15.189:0/2384644385 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:20.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.413+0000 7f5e4b7fe700 1 --2- 172.21.15.189:0/2384644385 >> v2:172.21.15.192:3300/0 conn(0x7f5e4c0d2880 0x7f5e4c0d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:20.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.413+0000 7f5e4a7fc700 1 -- 172.21.15.189:0/2384644385 >> v2:172.21.15.192:3300/0 conn(0x7f5e4c0d2880 msgr2=0x7f5e4c0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:20.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.413+0000 7f5e4a7fc700 1 --2- 172.21.15.189:0/2384644385 >> v2:172.21.15.192:3300/0 conn(0x7f5e4c0d2880 0x7f5e4c0d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.413+0000 7f5e4a7fc700 1 -- 172.21.15.189:0/2384644385 >> v1:172.21.15.192:6789/0 conn(0x7f5e4c15d590 legacy=0x7f5e4c0b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.413+0000 7f5e4a7fc700 1 -- 172.21.15.189:0/2384644385 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4c0d17e0 con 0x7f5e4c0b4ed0 2024-04-06T12:24:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.413+0000 7f5e4b7fe700 1 --2- 172.21.15.189:0/2384644385 >> v2:172.21.15.192:3300/0 conn(0x7f5e4c0d2880 0x7f5e4c0d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:24:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.413+0000 7f5e4a7fc700 1 --2- 172.21.15.189:0/2384644385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e4c0b4ed0 0x7f5e4c0b5380 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c0b75c0 tx=0x7f5e3c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839ca93835e2e4bb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.413+0000 7f5e49ffb700 1 -- 172.21.15.189:0/2384644385 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c00b870 con 0x7f5e4c0b4ed0 2024-04-06T12:24:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.414+0000 7f5e49ffb700 1 -- 172.21.15.189:0/2384644385 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3c00b9d0 con 0x7f5e4c0b4ed0 2024-04-06T12:24:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.414+0000 7f5e49ffb700 1 -- 172.21.15.189:0/2384644385 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c00bd60 con 0x7f5e4c0b4ed0 2024-04-06T12:24:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.414+0000 7f5e513c8700 1 -- 172.21.15.189:0/2384644385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e4c0b4ed0 msgr2=0x7f5e4c0b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.414+0000 7f5e513c8700 1 --2- 172.21.15.189:0/2384644385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e4c0b4ed0 0x7f5e4c0b5380 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c0b75c0 tx=0x7f5e3c009bd0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.414+0000 7f5e513c8700 1 -- 172.21.15.189:0/2384644385 shutdown_connections 2024-04-06T12:24:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.414+0000 7f5e513c8700 1 --2- 172.21.15.189:0/2384644385 >> v2:172.21.15.192:3300/0 conn(0x7f5e4c0d2880 0x7f5e4c0d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.414+0000 7f5e513c8700 1 --2- 172.21.15.189:0/2384644385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e4c0b4ed0 0x7f5e4c0b5380 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.414+0000 7f5e513c8700 1 -- 172.21.15.189:0/2384644385 >> 172.21.15.189:0/2384644385 conn(0x7f5e4c06bbc0 msgr2=0x7f5e4c06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:20.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.414+0000 7f5e513c8700 1 -- 172.21.15.189:0/2384644385 shutdown_connections 2024-04-06T12:24:20.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.414+0000 7f5e513c8700 1 -- 172.21.15.189:0/2384644385 wait complete. 2024-04-06T12:24:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.415+0000 7f5e513c8700 1 Processor -- start 2024-04-06T12:24:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.415+0000 7f5e513c8700 1 -- start start 2024-04-06T12:24:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.416+0000 7f5e513c8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e4c0b4ed0 0x7f5e4c0cb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.416+0000 7f5e513c8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5e4c0d2880 0x7f5e4c15d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.416+0000 7f5e513c8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e4c15d590 0x7f5e4c0c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.416+0000 7f5e513c8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c15deb0 con 0x7f5e4c0b4ed0 2024-04-06T12:24:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.416+0000 7f5e513c8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c0caca0 con 0x7f5e4c15d590 2024-04-06T12:24:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.416+0000 7f5e513c8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c0cc050 con 0x7f5e4c0d2880 2024-04-06T12:24:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.416+0000 7f5e4affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e4c0b4ed0 0x7f5e4c0cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.416+0000 7f5e4affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e4c0b4ed0 0x7f5e4c0cb130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33764/0 (socket says 172.21.15.189:33764) 2024-04-06T12:24:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.416+0000 7f5e4affd700 1 -- 172.21.15.189:0/2775512331 learned_addr learned my addr 172.21.15.189:0/2775512331 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.417+0000 7f5e4b7fe700 1 --2- 172.21.15.189:0/2775512331 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e4c15d590 0x7f5e4c0c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.417+0000 7f5e4a7fc700 1 --2- 172.21.15.189:0/2775512331 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5e4c0d2880 0x7f5e4c15d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.417+0000 7f5e4affd700 1 -- 172.21.15.189:0/2775512331 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5e4c0d2880 msgr2=0x7f5e4c15d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.417+0000 7f5e4affd700 1 --2- 172.21.15.189:0/2775512331 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5e4c0d2880 0x7f5e4c15d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.417+0000 7f5e4affd700 1 -- 172.21.15.189:0/2775512331 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e4c15d590 msgr2=0x7f5e4c0c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.417+0000 7f5e4affd700 1 --2- 172.21.15.189:0/2775512331 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e4c15d590 0x7f5e4c0c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.417+0000 7f5e4affd700 1 -- 172.21.15.189:0/2775512331 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c009550 con 0x7f5e4c0b4ed0 2024-04-06T12:24:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.417+0000 7f5e4affd700 1 --2- 172.21.15.189:0/2775512331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e4c0b4ed0 0x7f5e4c0cb130 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f5e4000c450 tx=0x7f5e4000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.417+0000 7f5e337fe700 1 -- 172.21.15.189:0/2775512331 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e400052d0 con 0x7f5e4c0b4ed0 2024-04-06T12:24:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.417+0000 7f5e337fe700 1 -- 172.21.15.189:0/2775512331 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e40005430 con 0x7f5e4c0b4ed0 2024-04-06T12:24:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.417+0000 7f5e513c8700 1 -- 172.21.15.189:0/2775512331 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e4c0c9610 con 0x7f5e4c0b4ed0 2024-04-06T12:24:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.417+0000 7f5e337fe700 1 -- 172.21.15.189:0/2775512331 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e400038c0 con 0x7f5e4c0b4ed0 2024-04-06T12:24:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.417+0000 7f5e513c8700 1 -- 172.21.15.189:0/2775512331 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e4c0ca650 con 0x7f5e4c0b4ed0 2024-04-06T12:24:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.418+0000 7f5e513c8700 1 -- 172.21.15.189:0/2775512331 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e4c0caf80 con 0x7f5e4c0b4ed0 2024-04-06T12:24:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.418+0000 7f5e337fe700 1 -- 172.21.15.189:0/2775512331 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f5e40003a20 con 0x7f5e4c0b4ed0 2024-04-06T12:24:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.419+0000 7f5e337fe700 1 --2- 172.21.15.189:0/2775512331 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5e3806ae90 0x7f5e3806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.419+0000 7f5e337fe700 1 -- 172.21.15.189:0/2775512331 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f5e40088c30 con 0x7f5e4c0b4ed0 2024-04-06T12:24:20.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.419+0000 7f5e4a7fc700 1 --2- 172.21.15.189:0/2775512331 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5e3806ae90 0x7f5e3806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:20.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.419+0000 7f5e4a7fc700 1 --2- 172.21.15.189:0/2775512331 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5e3806ae90 0x7f5e3806d350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c000c00 tx=0x7f5e3c008030 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:20.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.423+0000 7f5e337fe700 1 -- 172.21.15.189:0/2775512331 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5e400539c0 con 0x7f5e4c0b4ed0 2024-04-06T12:24:20.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.646+0000 7f5e513c8700 1 -- 172.21.15.189:0/2775512331 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e4c0d2080 con 0x7f5e3806ae90 2024-04-06T12:24:20.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.710+0000 7f5e337fe700 1 -- 172.21.15.189:0/2775512331 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f5e3c00bb40 con 0x7f5e3806ae90 2024-04-06T12:24:20.707 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2024-04-06T12:24:20.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.712+0000 7f5e513c8700 1 -- 172.21.15.189:0/2775512331 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5e3806ae90 msgr2=0x7f5e3806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:20.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.712+0000 7f5e513c8700 1 --2- 172.21.15.189:0/2775512331 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5e3806ae90 0x7f5e3806d350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c000c00 tx=0x7f5e3c008030 comp rx=0 tx=0).stop 2024-04-06T12:24:20.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.712+0000 7f5e513c8700 1 -- 172.21.15.189:0/2775512331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e4c0b4ed0 msgr2=0x7f5e4c0cb130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:20.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.712+0000 7f5e513c8700 1 --2- 172.21.15.189:0/2775512331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e4c0b4ed0 0x7f5e4c0cb130 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f5e4000c450 tx=0x7f5e4000c810 comp rx=0 tx=0).stop 2024-04-06T12:24:20.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.712+0000 7f5e513c8700 1 -- 172.21.15.189:0/2775512331 shutdown_connections 2024-04-06T12:24:20.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.712+0000 7f5e513c8700 1 --2- 172.21.15.189:0/2775512331 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5e3806ae90 0x7f5e3806d350 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.712+0000 7f5e513c8700 1 --2- 172.21.15.189:0/2775512331 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e4c15d590 0x7f5e4c0c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.712+0000 7f5e513c8700 1 --2- 172.21.15.189:0/2775512331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e4c0b4ed0 0x7f5e4c0cb130 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.712+0000 7f5e513c8700 1 --2- 172.21.15.189:0/2775512331 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5e4c0d2880 0x7f5e4c15d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.712+0000 7f5e513c8700 1 -- 172.21.15.189:0/2775512331 >> 172.21.15.189:0/2775512331 conn(0x7f5e4c06bbc0 msgr2=0x7f5e4c0ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:20.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.712+0000 7f5e513c8700 1 -- 172.21.15.189:0/2775512331 shutdown_connections 2024-04-06T12:24:20.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.712+0000 7f5e513c8700 1 -- 172.21.15.189:0/2775512331 wait complete. 2024-04-06T12:24:20.721 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2024-04-06T12:24:20.722 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir2 1m 2024-04-06T12:24:20.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.849+0000 7f7dd069f700 1 Processor -- start 2024-04-06T12:24:20.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.849+0000 7f7dd069f700 1 -- start start 2024-04-06T12:24:20.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.850+0000 7f7dd069f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7dc815a550 0x7f7dc80cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:20.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.850+0000 7f7dd069f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dc80b4d70 0x7f7dc80b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:20.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.850+0000 7f7dd069f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc80cdbe0 con 0x7f7dc80cf900 2024-04-06T12:24:20.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.850+0000 7f7dd069f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc815a940 con 0x7f7dc815a550 2024-04-06T12:24:20.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.850+0000 7f7dd069f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc815ac00 con 0x7f7dc80b4d70 2024-04-06T12:24:20.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.850+0000 7f7dce43b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7dc815a550 0x7f7dc80cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:20.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.850+0000 7f7dcdc3a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7dc80cf900 0x7f7dc80b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48666/0 (socket says 172.21.15.189:48666) 2024-04-06T12:24:20.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.850+0000 7f7dcdc3a700 1 -- 172.21.15.189:0/1778719416 learned_addr learned my addr 172.21.15.189:0/1778719416 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:20.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.850+0000 7f7dcec3c700 1 --2- 172.21.15.189:0/1778719416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dc80b4d70 0x7f7dc80b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:20.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.850+0000 7f7dce43b700 1 -- 172.21.15.189:0/1778719416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dc80b4d70 msgr2=0x7f7dc80b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:20.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.850+0000 7f7dce43b700 1 --2- 172.21.15.189:0/1778719416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dc80b4d70 0x7f7dc80b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.850+0000 7f7dce43b700 1 -- 172.21.15.189:0/1778719416 >> v1:172.21.15.189:6789/0 conn(0x7f7dc80cf900 legacy=0x7f7dc80b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:20.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.850+0000 7f7dce43b700 1 -- 172.21.15.189:0/1778719416 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc815aee0 con 0x7f7dc815a550 2024-04-06T12:24:20.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.850+0000 7f7dcd439700 1 -- 172.21.15.189:0/1778719416 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060225563 0 0) 0x7f7dc80cdbe0 con 0x7f7dc80cf900 2024-04-06T12:24:20.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.851+0000 7f7dce43b700 1 --2- 172.21.15.189:0/1778719416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7dc815a550 0x7f7dc80cf3c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7db800bae0 tx=0x7f7db800bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d41793a55f75041 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:20.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.851+0000 7f7dcd439700 1 -- 172.21.15.189:0/1778719416 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db800dc70 con 0x7f7dc815a550 2024-04-06T12:24:20.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.851+0000 7f7dcd439700 1 -- 172.21.15.189:0/1778719416 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db800ddd0 con 0x7f7dc815a550 2024-04-06T12:24:20.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.851+0000 7f7dcd439700 1 -- 172.21.15.189:0/1778719416 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db8010690 con 0x7f7dc815a550 2024-04-06T12:24:20.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.851+0000 7f7dd069f700 1 -- 172.21.15.189:0/1778719416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7dc815a550 msgr2=0x7f7dc80cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:20.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.851+0000 7f7dd069f700 1 --2- 172.21.15.189:0/1778719416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7dc815a550 0x7f7dc80cf3c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7db800bae0 tx=0x7f7db800bdf0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.851+0000 7f7dd069f700 1 -- 172.21.15.189:0/1778719416 shutdown_connections 2024-04-06T12:24:20.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.851+0000 7f7dd069f700 1 --2- 172.21.15.189:0/1778719416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dc80b4d70 0x7f7dc80b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.851+0000 7f7dd069f700 1 --2- 172.21.15.189:0/1778719416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7dc815a550 0x7f7dc80cf3c0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.851+0000 7f7dd069f700 1 -- 172.21.15.189:0/1778719416 >> 172.21.15.189:0/1778719416 conn(0x7f7dc80d00b0 msgr2=0x7f7dc80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:20.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.851+0000 7f7dd069f700 1 -- 172.21.15.189:0/1778719416 shutdown_connections 2024-04-06T12:24:20.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.851+0000 7f7dd069f700 1 -- 172.21.15.189:0/1778719416 wait complete. 2024-04-06T12:24:20.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.853+0000 7f7dd069f700 1 Processor -- start 2024-04-06T12:24:20.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.853+0000 7f7dd069f700 1 -- start start 2024-04-06T12:24:20.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.853+0000 7f7dd069f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dc80b4d70 0x7f7dc813e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:20.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.853+0000 7f7dd069f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dc80cf900 0x7f7dc813ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:20.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.853+0000 7f7dd069f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7dc813f490 0x7f7dc8156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:20.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.853+0000 7f7dd069f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc806b470 con 0x7f7dc813f490 2024-04-06T12:24:20.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.853+0000 7f7dd069f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc8141bc0 con 0x7f7dc80cf900 2024-04-06T12:24:20.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.853+0000 7f7dd069f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc8141ea0 con 0x7f7dc80b4d70 2024-04-06T12:24:20.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.853+0000 7f7dcec3c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7dc813f490 0x7f7dc8156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:20.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.853+0000 7f7dcec3c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7dc813f490 0x7f7dc8156030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33788/0 (socket says 172.21.15.189:33788) 2024-04-06T12:24:20.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.853+0000 7f7dcec3c700 1 -- 172.21.15.189:0/187177916 learned_addr learned my addr 172.21.15.189:0/187177916 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:20.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.853+0000 7f7dce43b700 1 --2- 172.21.15.189:0/187177916 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dc80b4d70 0x7f7dc813e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:20.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.853+0000 7f7dcdc3a700 1 --2- 172.21.15.189:0/187177916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dc80cf900 0x7f7dc813ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:20.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.854+0000 7f7dcec3c700 1 -- 172.21.15.189:0/187177916 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dc80b4d70 msgr2=0x7f7dc813e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:20.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.854+0000 7f7dcec3c700 1 --2- 172.21.15.189:0/187177916 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dc80b4d70 0x7f7dc813e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.854+0000 7f7dcec3c700 1 -- 172.21.15.189:0/187177916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dc80cf900 msgr2=0x7f7dc813ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:20.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.854+0000 7f7dcec3c700 1 --2- 172.21.15.189:0/187177916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dc80cf900 0x7f7dc813ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:20.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.854+0000 7f7dcec3c700 1 -- 172.21.15.189:0/187177916 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db800b740 con 0x7f7dc813f490 2024-04-06T12:24:20.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.854+0000 7f7dcec3c700 1 --2- 172.21.15.189:0/187177916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7dc813f490 0x7f7dc8156030 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7dc000efc0 tx=0x7f7dc000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:20.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.854+0000 7f7dbeffd700 1 -- 172.21.15.189:0/187177916 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc000cc90 con 0x7f7dc813f490 2024-04-06T12:24:20.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.854+0000 7f7dbeffd700 1 -- 172.21.15.189:0/187177916 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc000f040 con 0x7f7dc813f490 2024-04-06T12:24:20.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.854+0000 7f7dd069f700 1 -- 172.21.15.189:0/187177916 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc8155550 con 0x7f7dc813f490 2024-04-06T12:24:20.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.854+0000 7f7dbeffd700 1 -- 172.21.15.189:0/187177916 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc00146c0 con 0x7f7dc813f490 2024-04-06T12:24:20.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.854+0000 7f7dd069f700 1 -- 172.21.15.189:0/187177916 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc8155b10 con 0x7f7dc813f490 2024-04-06T12:24:20.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.855+0000 7f7dd069f700 1 -- 172.21.15.189:0/187177916 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc81553b0 con 0x7f7dc813f490 2024-04-06T12:24:20.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.855+0000 7f7dbeffd700 1 -- 172.21.15.189:0/187177916 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f7dc0014820 con 0x7f7dc813f490 2024-04-06T12:24:20.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.856+0000 7f7dbeffd700 1 --2- 172.21.15.189:0/187177916 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7dac06ae90 0x7f7dac06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:20.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.856+0000 7f7dbeffd700 1 -- 172.21.15.189:0/187177916 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f7dc008a540 con 0x7f7dc813f490 2024-04-06T12:24:20.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.856+0000 7f7dce43b700 1 --2- 172.21.15.189:0/187177916 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7dac06ae90 0x7f7dac06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:20.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.856+0000 7f7dce43b700 1 --2- 172.21.15.189:0/187177916 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7dac06ae90 0x7f7dac06d350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7db800b2c0 tx=0x7f7db8013000 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:20.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:20.860+0000 7f7dbeffd700 1 -- 172.21.15.189:0/187177916 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7dc81553b0 con 0x7f7dc813f490 2024-04-06T12:24:21.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.087+0000 7f7dd069f700 1 -- 172.21.15.189:0/187177916 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7dc815b4a0 con 0x7f7dac06ae90 2024-04-06T12:24:21.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.158+0000 7f7dbeffd700 1 -- 172.21.15.189:0/187177916 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f7db8020040 con 0x7f7dac06ae90 2024-04-06T12:24:21.155 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2024-04-06T12:24:21.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.160+0000 7f7dd069f700 1 -- 172.21.15.189:0/187177916 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7dac06ae90 msgr2=0x7f7dac06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:21.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.160+0000 7f7dd069f700 1 --2- 172.21.15.189:0/187177916 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7dac06ae90 0x7f7dac06d350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7db800b2c0 tx=0x7f7db8013000 comp rx=0 tx=0).stop 2024-04-06T12:24:21.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.160+0000 7f7dd069f700 1 -- 172.21.15.189:0/187177916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7dc813f490 msgr2=0x7f7dc8156030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:21.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.160+0000 7f7dd069f700 1 --2- 172.21.15.189:0/187177916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7dc813f490 0x7f7dc8156030 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7dc000efc0 tx=0x7f7dc000c3a0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.160+0000 7f7dd069f700 1 -- 172.21.15.189:0/187177916 shutdown_connections 2024-04-06T12:24:21.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.160+0000 7f7dd069f700 1 --2- 172.21.15.189:0/187177916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7dc813f490 0x7f7dc8156030 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.160+0000 7f7dd069f700 1 --2- 172.21.15.189:0/187177916 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dc80b4d70 0x7f7dc813e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.160+0000 7f7dd069f700 1 --2- 172.21.15.189:0/187177916 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7dac06ae90 0x7f7dac06d350 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.160+0000 7f7dd069f700 1 --2- 172.21.15.189:0/187177916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dc80cf900 0x7f7dc813ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.160+0000 7f7dd069f700 1 -- 172.21.15.189:0/187177916 >> 172.21.15.189:0/187177916 conn(0x7f7dc80d00b0 msgr2=0x7f7dc80b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:21.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.160+0000 7f7dd069f700 1 -- 172.21.15.189:0/187177916 shutdown_connections 2024-04-06T12:24:21.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.160+0000 7f7dd069f700 1 -- 172.21.15.189:0/187177916 wait complete. 2024-04-06T12:24:21.169 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2024-04-06T12:24:21.169 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir3 1m 2024-04-06T12:24:21.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.296+0000 7fd794ffb700 1 Processor -- start 2024-04-06T12:24:21.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.296+0000 7fd794ffb700 1 -- start start 2024-04-06T12:24:21.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.297+0000 7fd794ffb700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd7900d11b0 0x7fd7900d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:21.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.297+0000 7fd794ffb700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd7900cb870 0x7fd79015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:21.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.297+0000 7fd794ffb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7900d1ad0 con 0x7fd7900cfa00 2024-04-06T12:24:21.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.297+0000 7fd794ffb700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7900ccdc0 con 0x7fd7900cb870 2024-04-06T12:24:21.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.297+0000 7fd794ffb700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7900d0b50 con 0x7fd7900d11b0 2024-04-06T12:24:21.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.297+0000 7fd78dd9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd7900cb870 0x7fd79015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:21.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.297+0000 7fd78dd9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd7900cb870 0x7fd79015c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33798/0 (socket says 172.21.15.189:33798) 2024-04-06T12:24:21.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.297+0000 7fd78ed9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd7900cfa00 0x7fd7900cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48676/0 (socket says 172.21.15.189:48676) 2024-04-06T12:24:21.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.297+0000 7fd78dd9b700 1 -- 172.21.15.189:0/3736261582 learned_addr learned my addr 172.21.15.189:0/3736261582 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:21.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.297+0000 7fd78e59c700 1 --2- 172.21.15.189:0/3736261582 >> v2:172.21.15.192:3300/0 conn(0x7fd7900d11b0 0x7fd7900d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:21.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.297+0000 7fd78dd9b700 1 -- 172.21.15.189:0/3736261582 >> v2:172.21.15.192:3300/0 conn(0x7fd7900d11b0 msgr2=0x7fd7900d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:21.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.297+0000 7fd78dd9b700 1 --2- 172.21.15.189:0/3736261582 >> v2:172.21.15.192:3300/0 conn(0x7fd7900d11b0 0x7fd7900d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.297+0000 7fd78d59a700 1 -- 172.21.15.189:0/3736261582 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623037205 0 0) 0x7fd7900d1ad0 con 0x7fd7900cfa00 2024-04-06T12:24:21.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.297+0000 7fd78dd9b700 1 -- 172.21.15.189:0/3736261582 >> v1:172.21.15.189:6789/0 conn(0x7fd7900cfa00 legacy=0x7fd7900cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:21.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.297+0000 7fd78dd9b700 1 -- 172.21.15.189:0/3736261582 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7900cbd80 con 0x7fd7900cb870 2024-04-06T12:24:21.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.298+0000 7fd78dd9b700 1 --2- 172.21.15.189:0/3736261582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7900cb870 0x7fd79015c280 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd7900ced00 tx=0x7fd780009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d1ae043aa9033f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:21.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.298+0000 7fd78d59a700 1 -- 172.21.15.189:0/3736261582 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78000b870 con 0x7fd7900cb870 2024-04-06T12:24:21.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.298+0000 7fd78d59a700 1 -- 172.21.15.189:0/3736261582 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78000b9d0 con 0x7fd7900cb870 2024-04-06T12:24:21.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.298+0000 7fd78d59a700 1 -- 172.21.15.189:0/3736261582 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78000bd60 con 0x7fd7900cb870 2024-04-06T12:24:21.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.298+0000 7fd794ffb700 1 -- 172.21.15.189:0/3736261582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7900cb870 msgr2=0x7fd79015c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:21.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.298+0000 7fd794ffb700 1 --2- 172.21.15.189:0/3736261582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7900cb870 0x7fd79015c280 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd7900ced00 tx=0x7fd780009bd0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.298+0000 7fd794ffb700 1 -- 172.21.15.189:0/3736261582 shutdown_connections 2024-04-06T12:24:21.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.298+0000 7fd794ffb700 1 --2- 172.21.15.189:0/3736261582 >> v2:172.21.15.192:3300/0 conn(0x7fd7900d11b0 0x7fd7900d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.298+0000 7fd794ffb700 1 --2- 172.21.15.189:0/3736261582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7900cb870 0x7fd79015c280 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.298+0000 7fd794ffb700 1 -- 172.21.15.189:0/3736261582 >> 172.21.15.189:0/3736261582 conn(0x7fd7900b2db0 msgr2=0x7fd7900b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:21.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.298+0000 7fd794ffb700 1 -- 172.21.15.189:0/3736261582 shutdown_connections 2024-04-06T12:24:21.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.298+0000 7fd794ffb700 1 -- 172.21.15.189:0/3736261582 wait complete. 2024-04-06T12:24:21.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.300+0000 7fd794ffb700 1 Processor -- start 2024-04-06T12:24:21.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.300+0000 7fd794ffb700 1 -- start start 2024-04-06T12:24:21.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.300+0000 7fd794ffb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7900cb870 0x7fd79014c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:21.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd794ffb700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7900cfa00 0x7fd79015cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:21.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd794ffb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7900d11b0 0x7fd7900b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:21.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd794ffb700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79015d4e0 con 0x7fd7900cb870 2024-04-06T12:24:21.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd794ffb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79015d7c0 con 0x7fd7900d11b0 2024-04-06T12:24:21.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd794ffb700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79015daa0 con 0x7fd7900cfa00 2024-04-06T12:24:21.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd78e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7900cb870 0x7fd79014c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:21.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd78e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7900cb870 0x7fd79014c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33806/0 (socket says 172.21.15.189:33806) 2024-04-06T12:24:21.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd78e59c700 1 -- 172.21.15.189:0/296255530 learned_addr learned my addr 172.21.15.189:0/296255530 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:21.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd78dd9b700 1 --2- 172.21.15.189:0/296255530 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7900cfa00 0x7fd79015cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:21.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd78ed9d700 1 --2- 172.21.15.189:0/296255530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7900d11b0 0x7fd7900b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:21.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd78e59c700 1 -- 172.21.15.189:0/296255530 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7900cfa00 msgr2=0x7fd79015cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:21.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd78e59c700 1 --2- 172.21.15.189:0/296255530 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7900cfa00 0x7fd79015cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd78e59c700 1 -- 172.21.15.189:0/296255530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7900d11b0 msgr2=0x7fd7900b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:21.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd78e59c700 1 --2- 172.21.15.189:0/296255530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7900d11b0 0x7fd7900b47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd78e59c700 1 -- 172.21.15.189:0/296255530 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd780009550 con 0x7fd7900cb870 2024-04-06T12:24:21.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd78e59c700 1 --2- 172.21.15.189:0/296255530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7900cb870 0x7fd79014c220 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd78400c7f0 tx=0x7fd78400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:21.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd77effd700 1 -- 172.21.15.189:0/296255530 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd784004220 con 0x7fd7900cb870 2024-04-06T12:24:21.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd77effd700 1 -- 172.21.15.189:0/296255530 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd784004380 con 0x7fd7900cb870 2024-04-06T12:24:21.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd794ffb700 1 -- 172.21.15.189:0/296255530 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd79015dd80 con 0x7fd7900cb870 2024-04-06T12:24:21.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd77effd700 1 -- 172.21.15.189:0/296255530 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7840036a0 con 0x7fd7900cb870 2024-04-06T12:24:21.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.301+0000 7fd794ffb700 1 -- 172.21.15.189:0/296255530 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd79015d1d0 con 0x7fd7900cb870 2024-04-06T12:24:21.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.302+0000 7fd794ffb700 1 -- 172.21.15.189:0/296255530 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd790074990 con 0x7fd7900cb870 2024-04-06T12:24:21.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.303+0000 7fd77effd700 1 -- 172.21.15.189:0/296255530 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fd784003800 con 0x7fd7900cb870 2024-04-06T12:24:21.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.303+0000 7fd77effd700 1 --2- 172.21.15.189:0/296255530 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd77006ae90 0x7fd77006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:21.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.303+0000 7fd77effd700 1 -- 172.21.15.189:0/296255530 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fd7840895f0 con 0x7fd7900cb870 2024-04-06T12:24:21.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.303+0000 7fd78dd9b700 1 --2- 172.21.15.189:0/296255530 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd77006ae90 0x7fd77006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:21.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.304+0000 7fd78dd9b700 1 --2- 172.21.15.189:0/296255530 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd77006ae90 0x7fd77006d350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd7900cbcb0 tx=0x7fd780008030 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:21.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.308+0000 7fd77effd700 1 -- 172.21.15.189:0/296255530 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd784054140 con 0x7fd7900cb870 2024-04-06T12:24:21.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.531+0000 7fd794ffb700 1 -- 172.21.15.189:0/296255530 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd79015d930 con 0x7fd77006ae90 2024-04-06T12:24:21.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.596+0000 7fd77effd700 1 -- 172.21.15.189:0/296255530 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fd79015d930 con 0x7fd77006ae90 2024-04-06T12:24:21.594 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2024-04-06T12:24:21.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.599+0000 7fd794ffb700 1 -- 172.21.15.189:0/296255530 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd77006ae90 msgr2=0x7fd77006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:21.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.599+0000 7fd794ffb700 1 --2- 172.21.15.189:0/296255530 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd77006ae90 0x7fd77006d350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd7900cbcb0 tx=0x7fd780008030 comp rx=0 tx=0).stop 2024-04-06T12:24:21.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.599+0000 7fd794ffb700 1 -- 172.21.15.189:0/296255530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7900cb870 msgr2=0x7fd79014c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:21.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.599+0000 7fd794ffb700 1 --2- 172.21.15.189:0/296255530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7900cb870 0x7fd79014c220 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd78400c7f0 tx=0x7fd78400cb00 comp rx=0 tx=0).stop 2024-04-06T12:24:21.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.599+0000 7fd794ffb700 1 -- 172.21.15.189:0/296255530 shutdown_connections 2024-04-06T12:24:21.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.599+0000 7fd794ffb700 1 --2- 172.21.15.189:0/296255530 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd77006ae90 0x7fd77006d350 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.599+0000 7fd794ffb700 1 --2- 172.21.15.189:0/296255530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7900d11b0 0x7fd7900b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.599+0000 7fd794ffb700 1 --2- 172.21.15.189:0/296255530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7900cb870 0x7fd79014c220 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.599+0000 7fd794ffb700 1 --2- 172.21.15.189:0/296255530 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7900cfa00 0x7fd79015cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.599+0000 7fd794ffb700 1 -- 172.21.15.189:0/296255530 >> 172.21.15.189:0/296255530 conn(0x7fd7900b2db0 msgr2=0x7fd79015a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:21.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.599+0000 7fd794ffb700 1 -- 172.21.15.189:0/296255530 shutdown_connections 2024-04-06T12:24:21.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.599+0000 7fd794ffb700 1 -- 172.21.15.189:0/296255530 wait complete. 2024-04-06T12:24:21.608 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2024-04-06T12:24:21.608 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir4 1m 2024-04-06T12:24:21.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.736+0000 7febb4761700 1 Processor -- start 2024-04-06T12:24:21.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.736+0000 7febb4761700 1 -- start start 2024-04-06T12:24:21.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.737+0000 7febb4761700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7febac157c50 0x7febac15e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:21.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.737+0000 7febb4761700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7febac0b3480 0x7febac0b3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:21.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.737+0000 7febb4761700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febac0716e0 con 0x7febac0b3ac0 2024-04-06T12:24:21.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.737+0000 7febb4761700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febac0b4480 con 0x7febac0b3480 2024-04-06T12:24:21.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.737+0000 7febb4761700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febac158030 con 0x7febac157c50 2024-04-06T12:24:21.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.737+0000 7febb2cfe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7febac0b3480 0x7febac0b3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:21.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.737+0000 7febb1cfc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7febac0b3ac0 0x7febac06c4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48688/0 (socket says 172.21.15.189:48688) 2024-04-06T12:24:21.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.737+0000 7febb1cfc700 1 -- 172.21.15.189:0/4218978277 learned_addr learned my addr 172.21.15.189:0/4218978277 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:21.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.737+0000 7febb24fd700 1 --2- 172.21.15.189:0/4218978277 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7febac157c50 0x7febac15e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:21.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.737+0000 7febb2cfe700 1 -- 172.21.15.189:0/4218978277 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7febac157c50 msgr2=0x7febac15e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:21.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.737+0000 7febb2cfe700 1 --2- 172.21.15.189:0/4218978277 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7febac157c50 0x7febac15e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.737+0000 7febb2cfe700 1 -- 172.21.15.189:0/4218978277 >> v1:172.21.15.189:6789/0 conn(0x7febac0b3ac0 legacy=0x7febac06c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:21.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.738+0000 7febb2cfe700 1 -- 172.21.15.189:0/4218978277 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febac0b21c0 con 0x7febac0b3480 2024-04-06T12:24:21.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.738+0000 7febb14fb700 1 -- 172.21.15.189:0/4218978277 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461655049 0 0) 0x7febac0716e0 con 0x7febac0b3ac0 2024-04-06T12:24:21.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.738+0000 7febb2cfe700 1 --2- 172.21.15.189:0/4218978277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac0b3480 0x7febac0b3f40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7feba4009d90 tx=0x7feba400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee5665e61b21ed18 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:21.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.738+0000 7febb14fb700 1 -- 172.21.15.189:0/4218978277 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba400e070 con 0x7febac0b3480 2024-04-06T12:24:21.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.738+0000 7febb14fb700 1 -- 172.21.15.189:0/4218978277 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba400bb20 con 0x7febac0b3480 2024-04-06T12:24:21.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.738+0000 7febb14fb700 1 -- 172.21.15.189:0/4218978277 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba4012430 con 0x7febac0b3480 2024-04-06T12:24:21.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.738+0000 7febb4761700 1 -- 172.21.15.189:0/4218978277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac0b3480 msgr2=0x7febac0b3f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:21.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.738+0000 7febb4761700 1 --2- 172.21.15.189:0/4218978277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac0b3480 0x7febac0b3f40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7feba4009d90 tx=0x7feba400b330 comp rx=0 tx=0).stop 2024-04-06T12:24:21.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.738+0000 7febb4761700 1 -- 172.21.15.189:0/4218978277 shutdown_connections 2024-04-06T12:24:21.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.738+0000 7febb4761700 1 --2- 172.21.15.189:0/4218978277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac0b3480 0x7febac0b3f40 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.738+0000 7febb4761700 1 --2- 172.21.15.189:0/4218978277 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7febac157c50 0x7febac15e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.738+0000 7febb4761700 1 -- 172.21.15.189:0/4218978277 >> 172.21.15.189:0/4218978277 conn(0x7febac06a730 msgr2=0x7febac06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:21.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.739+0000 7febb4761700 1 -- 172.21.15.189:0/4218978277 shutdown_connections 2024-04-06T12:24:21.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.739+0000 7febb4761700 1 -- 172.21.15.189:0/4218978277 wait complete. 2024-04-06T12:24:21.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.740+0000 7febb4761700 1 Processor -- start 2024-04-06T12:24:21.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.740+0000 7febb4761700 1 -- start start 2024-04-06T12:24:21.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.740+0000 7febb4761700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febac0b3480 0x7febac15de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:21.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.740+0000 7febb4761700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac0b3ac0 0x7febac159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:21.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.740+0000 7febb4761700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7febac157c50 0x7febac14ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:21.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.740+0000 7febb4761700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac15a170 con 0x7febac0b3ac0 2024-04-06T12:24:21.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.740+0000 7febb4761700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac0adcf0 con 0x7febac0b3480 2024-04-06T12:24:21.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.740+0000 7febb4761700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febac14cf70 con 0x7febac157c50 2024-04-06T12:24:21.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.741+0000 7febb1cfc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac0b3ac0 0x7febac159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:21.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.741+0000 7febb1cfc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac0b3ac0 0x7febac159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33830/0 (socket says 172.21.15.189:33830) 2024-04-06T12:24:21.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.741+0000 7febb1cfc700 1 -- 172.21.15.189:0/2594535390 learned_addr learned my addr 172.21.15.189:0/2594535390 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:21.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.741+0000 7febb24fd700 1 --2- 172.21.15.189:0/2594535390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febac0b3480 0x7febac15de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:21.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.741+0000 7febb2cfe700 1 --2- 172.21.15.189:0/2594535390 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7febac157c50 0x7febac14ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:21.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.741+0000 7febb1cfc700 1 -- 172.21.15.189:0/2594535390 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7febac157c50 msgr2=0x7febac14ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:21.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.741+0000 7febb1cfc700 1 --2- 172.21.15.189:0/2594535390 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7febac157c50 0x7febac14ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.741+0000 7febb1cfc700 1 -- 172.21.15.189:0/2594535390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febac0b3480 msgr2=0x7febac15de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:21.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.741+0000 7febb1cfc700 1 --2- 172.21.15.189:0/2594535390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febac0b3480 0x7febac15de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:21.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.741+0000 7febb1cfc700 1 -- 172.21.15.189:0/2594535390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba4009a70 con 0x7febac0b3ac0 2024-04-06T12:24:21.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.741+0000 7febb24fd700 1 --2- 172.21.15.189:0/2594535390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febac0b3480 0x7febac15de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:24:21.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.741+0000 7febb2cfe700 1 --2- 172.21.15.189:0/2594535390 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7febac157c50 0x7febac14ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:24:21.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.741+0000 7febb1cfc700 1 --2- 172.21.15.189:0/2594535390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac0b3ac0 0x7febac159c30 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7feba80011e0 tx=0x7feba800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:21.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.741+0000 7feba2ffd700 1 -- 172.21.15.189:0/2594535390 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba8004c80 con 0x7febac0b3ac0 2024-04-06T12:24:21.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.741+0000 7feba2ffd700 1 -- 172.21.15.189:0/2594535390 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba8004e00 con 0x7febac0b3ac0 2024-04-06T12:24:21.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.741+0000 7febb4761700 1 -- 172.21.15.189:0/2594535390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febac15a380 con 0x7febac0b3ac0 2024-04-06T12:24:21.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.741+0000 7feba2ffd700 1 -- 172.21.15.189:0/2594535390 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba8007310 con 0x7febac0b3ac0 2024-04-06T12:24:21.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.741+0000 7febb4761700 1 -- 172.21.15.189:0/2594535390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febac15a940 con 0x7febac0b3ac0 2024-04-06T12:24:21.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.742+0000 7febb4761700 1 -- 172.21.15.189:0/2594535390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febac0adac0 con 0x7febac0b3ac0 2024-04-06T12:24:21.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.743+0000 7feba2ffd700 1 -- 172.21.15.189:0/2594535390 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7feba8007470 con 0x7febac0b3ac0 2024-04-06T12:24:21.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.743+0000 7feba2ffd700 1 --2- 172.21.15.189:0/2594535390 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb9006ae90 0x7feb9006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:21.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.743+0000 7febb24fd700 1 --2- 172.21.15.189:0/2594535390 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb9006ae90 0x7feb9006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:21.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.743+0000 7feba2ffd700 1 -- 172.21.15.189:0/2594535390 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7feba80896a0 con 0x7febac0b3ac0 2024-04-06T12:24:21.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.744+0000 7febb24fd700 1 --2- 172.21.15.189:0/2594535390 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb9006ae90 0x7feb9006d350 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009b90 tx=0x7feb9c008040 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:21.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.748+0000 7feba2ffd700 1 -- 172.21.15.189:0/2594535390 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7febac0adac0 con 0x7febac0b3ac0 2024-04-06T12:24:21.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:21.968+0000 7febb4761700 1 -- 172.21.15.189:0/2594535390 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7febac070ad0 con 0x7feb9006ae90 2024-04-06T12:24:22.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.033+0000 7feba2ffd700 1 -- 172.21.15.189:0/2594535390 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7feb9c0097e0 con 0x7feb9006ae90 2024-04-06T12:24:22.030 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2024-04-06T12:24:22.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.035+0000 7febb4761700 1 -- 172.21.15.189:0/2594535390 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb9006ae90 msgr2=0x7feb9006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:22.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.035+0000 7febb4761700 1 --2- 172.21.15.189:0/2594535390 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb9006ae90 0x7feb9006d350 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009b90 tx=0x7feb9c008040 comp rx=0 tx=0).stop 2024-04-06T12:24:22.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.035+0000 7febb4761700 1 -- 172.21.15.189:0/2594535390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac0b3ac0 msgr2=0x7febac159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:22.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.035+0000 7febb4761700 1 --2- 172.21.15.189:0/2594535390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac0b3ac0 0x7febac159c30 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7feba80011e0 tx=0x7feba800cae0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.035+0000 7febb4761700 1 -- 172.21.15.189:0/2594535390 shutdown_connections 2024-04-06T12:24:22.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.035+0000 7febb4761700 1 --2- 172.21.15.189:0/2594535390 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7febac157c50 0x7febac14ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.035+0000 7febb4761700 1 --2- 172.21.15.189:0/2594535390 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb9006ae90 0x7feb9006d350 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.035+0000 7febb4761700 1 --2- 172.21.15.189:0/2594535390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febac0b3480 0x7febac15de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.035+0000 7febb4761700 1 --2- 172.21.15.189:0/2594535390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac0b3ac0 0x7febac159c30 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.035+0000 7febb4761700 1 -- 172.21.15.189:0/2594535390 >> 172.21.15.189:0/2594535390 conn(0x7febac06a730 msgr2=0x7febac06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:22.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.035+0000 7febb4761700 1 -- 172.21.15.189:0/2594535390 shutdown_connections 2024-04-06T12:24:22.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.035+0000 7febb4761700 1 -- 172.21.15.189:0/2594535390 wait complete. 2024-04-06T12:24:22.044 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2024-04-06T12:24:22.045 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir5 1m 2024-04-06T12:24:22.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.172+0000 7fbcbaf2a700 1 Processor -- start 2024-04-06T12:24:22.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.172+0000 7fbcbaf2a700 1 -- start start 2024-04-06T12:24:22.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.173+0000 7fbcbaf2a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbcb415d7b0 0x7fbcb40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:22.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.173+0000 7fbcbaf2a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbcb415e500 0x7fbcb406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:22.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.173+0000 7fbcbaf2a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb40b3990 con 0x7fbcb406a730 2024-04-06T12:24:22.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.173+0000 7fbcbaf2a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcb415e950 con 0x7fbcb415d7b0 2024-04-06T12:24:22.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.173+0000 7fbcbaf2a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcb406b0f0 con 0x7fbcb415e500 2024-04-06T12:24:22.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.173+0000 7fbcb8cc6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbcb415d7b0 0x7fbcb40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:22.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.173+0000 7fbcb3fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbcb406a730 0x7fbcb415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48700/0 (socket says 172.21.15.189:48700) 2024-04-06T12:24:22.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.173+0000 7fbcb3fff700 1 -- 172.21.15.189:0/59813690 learned_addr learned my addr 172.21.15.189:0/59813690 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:22.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.173+0000 7fbcb8cc6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbcb415d7b0 0x7fbcb40b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33832/0 (socket says 172.21.15.189:33832) 2024-04-06T12:24:22.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.173+0000 7fbcb94c7700 1 --2- 172.21.15.189:0/59813690 >> v2:172.21.15.192:3300/0 conn(0x7fbcb415e500 0x7fbcb406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:22.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.173+0000 7fbcb8cc6700 1 -- 172.21.15.189:0/59813690 >> v2:172.21.15.192:3300/0 conn(0x7fbcb415e500 msgr2=0x7fbcb406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:22.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.173+0000 7fbcb8cc6700 1 --2- 172.21.15.189:0/59813690 >> v2:172.21.15.192:3300/0 conn(0x7fbcb415e500 0x7fbcb406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.173+0000 7fbcb8cc6700 1 -- 172.21.15.189:0/59813690 >> v1:172.21.15.189:6789/0 conn(0x7fbcb406a730 legacy=0x7fbcb415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:22.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.173+0000 7fbcb8cc6700 1 -- 172.21.15.189:0/59813690 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb40b3060 con 0x7fbcb415d7b0 2024-04-06T12:24:22.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.173+0000 7fbcb37fe700 1 -- 172.21.15.189:0/59813690 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960597089 0 0) 0x7fbcb40b3990 con 0x7fbcb406a730 2024-04-06T12:24:22.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.173+0000 7fbcb8cc6700 1 --2- 172.21.15.189:0/59813690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcb415d7b0 0x7fbcb40b3e20 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fbcb40673e0 tx=0x7fbca8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f62d28154494655e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:22.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.173+0000 7fbcb37fe700 1 -- 172.21.15.189:0/59813690 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca8011830 con 0x7fbcb415d7b0 2024-04-06T12:24:22.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.174+0000 7fbcb37fe700 1 -- 172.21.15.189:0/59813690 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca8011990 con 0x7fbcb415d7b0 2024-04-06T12:24:22.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.174+0000 7fbcb37fe700 1 -- 172.21.15.189:0/59813690 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca8011d20 con 0x7fbcb415d7b0 2024-04-06T12:24:22.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.174+0000 7fbcbaf2a700 1 -- 172.21.15.189:0/59813690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcb415d7b0 msgr2=0x7fbcb40b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:22.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.174+0000 7fbcbaf2a700 1 --2- 172.21.15.189:0/59813690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcb415d7b0 0x7fbcb40b3e20 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fbcb40673e0 tx=0x7fbca8009b20 comp rx=0 tx=0).stop 2024-04-06T12:24:22.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.174+0000 7fbcbaf2a700 1 -- 172.21.15.189:0/59813690 shutdown_connections 2024-04-06T12:24:22.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.174+0000 7fbcbaf2a700 1 --2- 172.21.15.189:0/59813690 >> v2:172.21.15.192:3300/0 conn(0x7fbcb415e500 0x7fbcb406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.174+0000 7fbcbaf2a700 1 --2- 172.21.15.189:0/59813690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcb415d7b0 0x7fbcb40b3e20 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.174+0000 7fbcbaf2a700 1 -- 172.21.15.189:0/59813690 >> 172.21.15.189:0/59813690 conn(0x7fbcb40d00b0 msgr2=0x7fbcb40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:22.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.174+0000 7fbcbaf2a700 1 -- 172.21.15.189:0/59813690 shutdown_connections 2024-04-06T12:24:22.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.174+0000 7fbcbaf2a700 1 -- 172.21.15.189:0/59813690 wait complete. 2024-04-06T12:24:22.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.175+0000 7fbcbaf2a700 1 Processor -- start 2024-04-06T12:24:22.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.175+0000 7fbcbaf2a700 1 -- start start 2024-04-06T12:24:22.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.176+0000 7fbcbaf2a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcb406a730 0x7fbcb415adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:22.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.176+0000 7fbcbaf2a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb415d7b0 0x7fbcb40ad090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:22.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.176+0000 7fbcbaf2a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbcb415e500 0x7fbcb40cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:22.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.176+0000 7fbcbaf2a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb41f2a30 con 0x7fbcb406a730 2024-04-06T12:24:22.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.176+0000 7fbcbaf2a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb41f64b0 con 0x7fbcb415d7b0 2024-04-06T12:24:22.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.176+0000 7fbcbaf2a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb41f6790 con 0x7fbcb415e500 2024-04-06T12:24:22.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.176+0000 7fbcb8cc6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcb406a730 0x7fbcb415adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:22.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.176+0000 7fbcb8cc6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcb406a730 0x7fbcb415adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33846/0 (socket says 172.21.15.189:33846) 2024-04-06T12:24:22.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.176+0000 7fbcb8cc6700 1 -- 172.21.15.189:0/3175789533 learned_addr learned my addr 172.21.15.189:0/3175789533 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:22.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.176+0000 7fbcb3fff700 1 --2- 172.21.15.189:0/3175789533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb415d7b0 0x7fbcb40ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:22.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.176+0000 7fbcb94c7700 1 --2- 172.21.15.189:0/3175789533 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbcb415e500 0x7fbcb40cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:22.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.177+0000 7fbcb8cc6700 1 -- 172.21.15.189:0/3175789533 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbcb415e500 msgr2=0x7fbcb40cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:22.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.177+0000 7fbcb8cc6700 1 --2- 172.21.15.189:0/3175789533 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbcb415e500 0x7fbcb40cc790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.177+0000 7fbcb8cc6700 1 -- 172.21.15.189:0/3175789533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb415d7b0 msgr2=0x7fbcb40ad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:22.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.177+0000 7fbcb8cc6700 1 --2- 172.21.15.189:0/3175789533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb415d7b0 0x7fbcb40ad090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.177+0000 7fbcb8cc6700 1 -- 172.21.15.189:0/3175789533 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca80094a0 con 0x7fbcb406a730 2024-04-06T12:24:22.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.177+0000 7fbcb8cc6700 1 --2- 172.21.15.189:0/3175789533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcb406a730 0x7fbcb415adb0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fbca8009b60 tx=0x7fbca8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:22.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.177+0000 7fbcb17fa700 1 -- 172.21.15.189:0/3175789533 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca80036a0 con 0x7fbcb406a730 2024-04-06T12:24:22.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.177+0000 7fbcb17fa700 1 -- 172.21.15.189:0/3175789533 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca801f070 con 0x7fbcb406a730 2024-04-06T12:24:22.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.177+0000 7fbcbaf2a700 1 -- 172.21.15.189:0/3175789533 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb41f6a70 con 0x7fbcb406a730 2024-04-06T12:24:22.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.177+0000 7fbcb17fa700 1 -- 172.21.15.189:0/3175789533 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca8003a40 con 0x7fbcb406a730 2024-04-06T12:24:22.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.177+0000 7fbcbaf2a700 1 -- 172.21.15.189:0/3175789533 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb415b2f0 con 0x7fbcb406a730 2024-04-06T12:24:22.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.178+0000 7fbcbaf2a700 1 -- 172.21.15.189:0/3175789533 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb41f6f90 con 0x7fbcb406a730 2024-04-06T12:24:22.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.178+0000 7fbcb17fa700 1 -- 172.21.15.189:0/3175789533 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fbca8003ba0 con 0x7fbcb406a730 2024-04-06T12:24:22.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.179+0000 7fbcb17fa700 1 --2- 172.21.15.189:0/3175789533 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc9806ae90 0x7fbc9806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:22.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.179+0000 7fbcb17fa700 1 -- 172.21.15.189:0/3175789533 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fbca8021070 con 0x7fbcb406a730 2024-04-06T12:24:22.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.179+0000 7fbcb3fff700 1 --2- 172.21.15.189:0/3175789533 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc9806ae90 0x7fbc9806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:22.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.179+0000 7fbcb3fff700 1 --2- 172.21.15.189:0/3175789533 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc9806ae90 0x7fbc9806d350 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbcb40add70 tx=0x7fbca0007be0 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:22.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.183+0000 7fbcb17fa700 1 -- 172.21.15.189:0/3175789533 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbca8055f40 con 0x7fbcb406a730 2024-04-06T12:24:22.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.404+0000 7fbcbaf2a700 1 -- 172.21.15.189:0/3175789533 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fbcb415eb40 con 0x7fbc9806ae90 2024-04-06T12:24:22.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.470+0000 7fbcb17fa700 1 -- 172.21.15.189:0/3175789533 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fbca0005aa0 con 0x7fbc9806ae90 2024-04-06T12:24:22.467 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2024-04-06T12:24:22.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.472+0000 7fbcbaf2a700 1 -- 172.21.15.189:0/3175789533 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc9806ae90 msgr2=0x7fbc9806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:22.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.472+0000 7fbcbaf2a700 1 --2- 172.21.15.189:0/3175789533 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc9806ae90 0x7fbc9806d350 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbcb40add70 tx=0x7fbca0007be0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.472+0000 7fbcbaf2a700 1 -- 172.21.15.189:0/3175789533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcb406a730 msgr2=0x7fbcb415adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:22.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.472+0000 7fbcbaf2a700 1 --2- 172.21.15.189:0/3175789533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcb406a730 0x7fbcb415adb0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fbca8009b60 tx=0x7fbca8004750 comp rx=0 tx=0).stop 2024-04-06T12:24:22.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.472+0000 7fbcbaf2a700 1 -- 172.21.15.189:0/3175789533 shutdown_connections 2024-04-06T12:24:22.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.472+0000 7fbcbaf2a700 1 --2- 172.21.15.189:0/3175789533 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbcb415e500 0x7fbcb40cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.472+0000 7fbcbaf2a700 1 --2- 172.21.15.189:0/3175789533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcb406a730 0x7fbcb415adb0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.472+0000 7fbcbaf2a700 1 --2- 172.21.15.189:0/3175789533 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc9806ae90 0x7fbc9806d350 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.472+0000 7fbcbaf2a700 1 --2- 172.21.15.189:0/3175789533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb415d7b0 0x7fbcb40ad090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.472+0000 7fbcbaf2a700 1 -- 172.21.15.189:0/3175789533 >> 172.21.15.189:0/3175789533 conn(0x7fbcb40d00b0 msgr2=0x7fbcb406eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:22.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.472+0000 7fbcbaf2a700 1 -- 172.21.15.189:0/3175789533 shutdown_connections 2024-04-06T12:24:22.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.472+0000 7fbcbaf2a700 1 -- 172.21.15.189:0/3175789533 wait complete. 2024-04-06T12:24:22.481 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2024-04-06T12:24:22.481 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir6 1m 2024-04-06T12:24:22.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.609+0000 7f51a709f700 1 Processor -- start 2024-04-06T12:24:22.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.609+0000 7f51a709f700 1 -- start start 2024-04-06T12:24:22.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.610+0000 7f51a709f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f51a00b4af0 0x7f51a00b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:22.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.610+0000 7f51a709f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51a00cdbe0 con 0x7f51a015a550 2024-04-06T12:24:22.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.610+0000 7f51a709f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51a015a940 con 0x7f51a00b4660 2024-04-06T12:24:22.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.610+0000 7f51a709f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51a015ac00 con 0x7f51a00b4af0 2024-04-06T12:24:22.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.610+0000 7f51a4e3b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f51a015a550 0x7f51a00cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48702/0 (socket says 172.21.15.189:48702) 2024-04-06T12:24:22.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.610+0000 7f51a4e3b700 1 -- 172.21.15.189:0/1110727558 learned_addr learned my addr 172.21.15.189:0/1110727558 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:22.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.610+0000 7f51a563c700 1 --2- 172.21.15.189:0/1110727558 >> v2:172.21.15.192:3300/0 conn(0x7f51a00b4af0 0x7f51a00b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:22.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.610+0000 7f519f7fe700 1 -- 172.21.15.189:0/1110727558 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198742817 0 0) 0x7f51a00cdbe0 con 0x7f51a015a550 2024-04-06T12:24:22.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.610+0000 7f519f7fe700 1 -- 172.21.15.189:0/1110727558 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51900035b0 con 0x7f51a015a550 2024-04-06T12:24:22.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.610+0000 7f519f7fe700 1 -- 172.21.15.189:0/1110727558 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2234968559 0 0) 0x7f51a015a940 con 0x7f51a00b4660 2024-04-06T12:24:22.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.610+0000 7f519f7fe700 1 -- 172.21.15.189:0/1110727558 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51a00cdbe0 con 0x7f51a00b4660 2024-04-06T12:24:22.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.610+0000 7f51a563c700 1 -- 172.21.15.189:0/1110727558 >> v1:172.21.15.192:6789/0 conn(0x7f51a00b4660 legacy=0x7f51a00b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:22.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.610+0000 7f51a563c700 1 -- 172.21.15.189:0/1110727558 >> v1:172.21.15.189:6789/0 conn(0x7f51a015a550 legacy=0x7f51a00cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:22.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.610+0000 7f51a563c700 1 -- 172.21.15.189:0/1110727558 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a015aee0 con 0x7f51a00b4af0 2024-04-06T12:24:22.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.610+0000 7f51a563c700 1 --2- 172.21.15.189:0/1110727558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51a00b4af0 0x7f51a00b36d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5198009d90 tx=0x7f519800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c4af545492f3e54 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:22.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.611+0000 7f519f7fe700 1 -- 172.21.15.189:0/1110727558 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519800e070 con 0x7f51a00b4af0 2024-04-06T12:24:22.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.611+0000 7f519f7fe700 1 -- 172.21.15.189:0/1110727558 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f519800bb20 con 0x7f51a00b4af0 2024-04-06T12:24:22.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.611+0000 7f519f7fe700 1 -- 172.21.15.189:0/1110727558 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5198012430 con 0x7f51a00b4af0 2024-04-06T12:24:22.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.611+0000 7f51a709f700 1 -- 172.21.15.189:0/1110727558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51a00b4af0 msgr2=0x7f51a00b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:22.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.611+0000 7f51a709f700 1 --2- 172.21.15.189:0/1110727558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51a00b4af0 0x7f51a00b36d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5198009d90 tx=0x7f519800b330 comp rx=0 tx=0).stop 2024-04-06T12:24:22.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.611+0000 7f51a709f700 1 -- 172.21.15.189:0/1110727558 shutdown_connections 2024-04-06T12:24:22.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.611+0000 7f51a709f700 1 --2- 172.21.15.189:0/1110727558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51a00b4af0 0x7f51a00b36d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.611+0000 7f51a709f700 1 -- 172.21.15.189:0/1110727558 >> 172.21.15.189:0/1110727558 conn(0x7f51a00d00b0 msgr2=0x7f51a00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:22.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.611+0000 7f51a709f700 1 -- 172.21.15.189:0/1110727558 shutdown_connections 2024-04-06T12:24:22.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.611+0000 7f51a709f700 1 -- 172.21.15.189:0/1110727558 wait complete. 2024-04-06T12:24:22.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.612+0000 7f51a709f700 1 Processor -- start 2024-04-06T12:24:22.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.612+0000 7f51a709f700 1 -- start start 2024-04-06T12:24:22.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.613+0000 7f51a709f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51a00b4660 0x7f51a015d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:22.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.613+0000 7f51a709f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51a00b4af0 0x7f51a013b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:22.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.613+0000 7f51a709f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51a015c680 0x7f51a014cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:22.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.613+0000 7f51a709f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a013bf30 con 0x7f51a00b4660 2024-04-06T12:24:22.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.613+0000 7f51a709f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a015ce90 con 0x7f51a015c680 2024-04-06T12:24:22.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.613+0000 7f51a709f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51a013af10 con 0x7f51a00b4af0 2024-04-06T12:24:22.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.613+0000 7f51a4e3b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51a00b4660 0x7f51a015d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:22.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.613+0000 7f51a4e3b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51a00b4660 0x7f51a015d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33854/0 (socket says 172.21.15.189:33854) 2024-04-06T12:24:22.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.613+0000 7f51a4e3b700 1 -- 172.21.15.189:0/1621388706 learned_addr learned my addr 172.21.15.189:0/1621388706 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:22.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.613+0000 7f51a563c700 1 --2- 172.21.15.189:0/1621388706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51a015c680 0x7f51a014cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:22.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.613+0000 7f51a4e3b700 1 -- 172.21.15.189:0/1621388706 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51a00b4af0 msgr2=0x7f51a013b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:22.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.613+0000 7f51a4e3b700 1 --2- 172.21.15.189:0/1621388706 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51a00b4af0 0x7f51a013b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.614+0000 7f51a4e3b700 1 -- 172.21.15.189:0/1621388706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51a015c680 msgr2=0x7f51a014cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:22.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.614+0000 7f51a4e3b700 1 --2- 172.21.15.189:0/1621388706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51a015c680 0x7f51a014cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.614+0000 7f51a4e3b700 1 -- 172.21.15.189:0/1621388706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5198009a70 con 0x7f51a00b4660 2024-04-06T12:24:22.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.614+0000 7f51a4e3b700 1 --2- 172.21.15.189:0/1621388706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51a00b4660 0x7f51a015d580 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f519400c4d0 tx=0x7f519400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:22.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.614+0000 7f519d7fa700 1 -- 172.21.15.189:0/1621388706 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5194005d70 con 0x7f51a00b4660 2024-04-06T12:24:22.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.614+0000 7f519d7fa700 1 -- 172.21.15.189:0/1621388706 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51940070d0 con 0x7f51a00b4660 2024-04-06T12:24:22.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.614+0000 7f519d7fa700 1 -- 172.21.15.189:0/1621388706 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5194007480 con 0x7f51a00b4660 2024-04-06T12:24:22.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.614+0000 7f51a709f700 1 -- 172.21.15.189:0/1621388706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a013b1f0 con 0x7f51a00b4660 2024-04-06T12:24:22.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.614+0000 7f51a709f700 1 -- 172.21.15.189:0/1621388706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a015cb00 con 0x7f51a00b4660 2024-04-06T12:24:22.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.615+0000 7f51a709f700 1 -- 172.21.15.189:0/1621388706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a013b710 con 0x7f51a00b4660 2024-04-06T12:24:22.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.615+0000 7f519d7fa700 1 -- 172.21.15.189:0/1621388706 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f51940075e0 con 0x7f51a00b4660 2024-04-06T12:24:22.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.616+0000 7f519d7fa700 1 --2- 172.21.15.189:0/1621388706 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f518406ae90 0x7f518406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:22.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.616+0000 7f519d7fa700 1 -- 172.21.15.189:0/1621388706 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f5194088ed0 con 0x7f51a00b4660 2024-04-06T12:24:22.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.616+0000 7f519ffff700 1 --2- 172.21.15.189:0/1621388706 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f518406ae90 0x7f518406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:22.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.616+0000 7f519ffff700 1 --2- 172.21.15.189:0/1621388706 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f518406ae90 0x7f518406d350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f518c0010e0 tx=0x7f518c0079e0 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:22.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.620+0000 7f519d7fa700 1 -- 172.21.15.189:0/1621388706 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f51a013b710 con 0x7f51a00b4660 2024-04-06T12:24:22.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.845+0000 7f51a709f700 1 -- 172.21.15.189:0/1621388706 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f51a006fdd0 con 0x7f518406ae90 2024-04-06T12:24:22.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.911+0000 7f519d7fa700 1 -- 172.21.15.189:0/1621388706 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f518c0050d0 con 0x7f518406ae90 2024-04-06T12:24:22.908 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2024-04-06T12:24:22.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.913+0000 7f51a709f700 1 -- 172.21.15.189:0/1621388706 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f518406ae90 msgr2=0x7f518406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:22.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.913+0000 7f51a709f700 1 --2- 172.21.15.189:0/1621388706 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f518406ae90 0x7f518406d350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f518c0010e0 tx=0x7f518c0079e0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.913+0000 7f51a709f700 1 -- 172.21.15.189:0/1621388706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51a00b4660 msgr2=0x7f51a015d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:22.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.913+0000 7f51a709f700 1 --2- 172.21.15.189:0/1621388706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51a00b4660 0x7f51a015d580 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f519400c4d0 tx=0x7f519400c890 comp rx=0 tx=0).stop 2024-04-06T12:24:22.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.913+0000 7f51a709f700 1 -- 172.21.15.189:0/1621388706 shutdown_connections 2024-04-06T12:24:22.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.913+0000 7f51a709f700 1 --2- 172.21.15.189:0/1621388706 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f518406ae90 0x7f518406d350 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.913+0000 7f51a709f700 1 --2- 172.21.15.189:0/1621388706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51a015c680 0x7f51a014cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.913+0000 7f51a709f700 1 --2- 172.21.15.189:0/1621388706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51a00b4660 0x7f51a015d580 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.913+0000 7f51a709f700 1 --2- 172.21.15.189:0/1621388706 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51a00b4af0 0x7f51a013b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:22.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.913+0000 7f51a709f700 1 -- 172.21.15.189:0/1621388706 >> 172.21.15.189:0/1621388706 conn(0x7f51a00d00b0 msgr2=0x7f51a006afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:22.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.913+0000 7f51a709f700 1 -- 172.21.15.189:0/1621388706 shutdown_connections 2024-04-06T12:24:22.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:22.913+0000 7f51a709f700 1 -- 172.21.15.189:0/1621388706 wait complete. 2024-04-06T12:24:22.922 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2024-04-06T12:24:22.922 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir7 1m 2024-04-06T12:24:23.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.050+0000 7f6baf3c9700 1 Processor -- start 2024-04-06T12:24:23.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.050+0000 7f6baf3c9700 1 -- start start 2024-04-06T12:24:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.051+0000 7f6baf3c9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6ba815a550 0x7f6ba80cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.051+0000 7f6baf3c9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba80b4d70 0x7f6ba80b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.051+0000 7f6baf3c9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba80cdbe0 con 0x7f6ba80cf900 2024-04-06T12:24:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.051+0000 7f6baf3c9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ba815a940 con 0x7f6ba815a550 2024-04-06T12:24:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.051+0000 7f6baf3c9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba815ac00 con 0x7f6ba80b4d70 2024-04-06T12:24:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.051+0000 7f6bad165700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6ba815a550 0x7f6ba80cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.051+0000 7f6bad165700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6ba815a550 0x7f6ba80cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33858/0 (socket says 172.21.15.189:33858) 2024-04-06T12:24:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.051+0000 7f6bad165700 1 -- 172.21.15.189:0/717169509 learned_addr learned my addr 172.21.15.189:0/717169509 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.051+0000 7f6bad165700 1 -- 172.21.15.189:0/717169509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba80b4d70 msgr2=0x7f6ba80b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.051+0000 7f6bad165700 1 --2- 172.21.15.189:0/717169509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba80b4d70 0x7f6ba80b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.051+0000 7f6bad165700 1 -- 172.21.15.189:0/717169509 >> v1:172.21.15.192:6789/0 conn(0x7f6ba80cf900 legacy=0x7f6ba80b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.051+0000 7f6bad165700 1 -- 172.21.15.189:0/717169509 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba815aee0 con 0x7f6ba815a550 2024-04-06T12:24:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.051+0000 7f6bad165700 1 --2- 172.21.15.189:0/717169509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ba815a550 0x7f6ba80cf3c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f6ba400bb90 tx=0x7f6ba400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8201d83986303e05 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.051+0000 7f6b9ffff700 1 -- 172.21.15.189:0/717169509 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba400dd10 con 0x7f6ba815a550 2024-04-06T12:24:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.052+0000 7f6b9ffff700 1 -- 172.21.15.189:0/717169509 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba400de70 con 0x7f6ba815a550 2024-04-06T12:24:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.052+0000 7f6b9ffff700 1 -- 172.21.15.189:0/717169509 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4004230 con 0x7f6ba815a550 2024-04-06T12:24:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.052+0000 7f6baf3c9700 1 -- 172.21.15.189:0/717169509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ba815a550 msgr2=0x7f6ba80cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.052+0000 7f6baf3c9700 1 --2- 172.21.15.189:0/717169509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ba815a550 0x7f6ba80cf3c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f6ba400bb90 tx=0x7f6ba400bea0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.052+0000 7f6baf3c9700 1 -- 172.21.15.189:0/717169509 shutdown_connections 2024-04-06T12:24:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.052+0000 7f6baf3c9700 1 --2- 172.21.15.189:0/717169509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba80b4d70 0x7f6ba80b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.052+0000 7f6baf3c9700 1 --2- 172.21.15.189:0/717169509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ba815a550 0x7f6ba80cf3c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.052+0000 7f6baf3c9700 1 -- 172.21.15.189:0/717169509 >> 172.21.15.189:0/717169509 conn(0x7f6ba80d00b0 msgr2=0x7f6ba80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.052+0000 7f6baf3c9700 1 -- 172.21.15.189:0/717169509 shutdown_connections 2024-04-06T12:24:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.052+0000 7f6baf3c9700 1 -- 172.21.15.189:0/717169509 wait complete. 2024-04-06T12:24:23.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.053+0000 7f6baf3c9700 1 Processor -- start 2024-04-06T12:24:23.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.053+0000 7f6baf3c9700 1 -- start start 2024-04-06T12:24:23.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.054+0000 7f6baf3c9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ba80b4d70 0x7f6ba80aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:23.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.054+0000 7f6baf3c9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba80cf900 0x7f6ba8152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:23.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.054+0000 7f6baf3c9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba80ac8c0 0x7f6ba815cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:23.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.054+0000 7f6baf3c9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba80ad9b0 con 0x7f6ba80b4d70 2024-04-06T12:24:23.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.054+0000 7f6baf3c9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba8153190 con 0x7f6ba80ac8c0 2024-04-06T12:24:23.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.054+0000 7f6baf3c9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba80acd40 con 0x7f6ba80cf900 2024-04-06T12:24:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.054+0000 7f6bad165700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ba80b4d70 0x7f6ba80aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.054+0000 7f6bad165700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ba80b4d70 0x7f6ba80aefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33870/0 (socket says 172.21.15.189:33870) 2024-04-06T12:24:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.054+0000 7f6bad165700 1 -- 172.21.15.189:0/3542452124 learned_addr learned my addr 172.21.15.189:0/3542452124 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.054+0000 7f6bad966700 1 --2- 172.21.15.189:0/3542452124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba80ac8c0 0x7f6ba815cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.054+0000 7f6bac964700 1 --2- 172.21.15.189:0/3542452124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba80cf900 0x7f6ba8152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.054+0000 7f6bad165700 1 -- 172.21.15.189:0/3542452124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba80cf900 msgr2=0x7f6ba8152c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.054+0000 7f6bad165700 1 --2- 172.21.15.189:0/3542452124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba80cf900 0x7f6ba8152c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.054+0000 7f6bad165700 1 -- 172.21.15.189:0/3542452124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba80ac8c0 msgr2=0x7f6ba815cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.054+0000 7f6bad165700 1 --2- 172.21.15.189:0/3542452124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba80ac8c0 0x7f6ba815cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.054+0000 7f6bad165700 1 -- 172.21.15.189:0/3542452124 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba400b7f0 con 0x7f6ba80b4d70 2024-04-06T12:24:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.055+0000 7f6bad165700 1 --2- 172.21.15.189:0/3542452124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ba80b4d70 0x7f6ba80aefb0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f6ba806c820 tx=0x7f6ba4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.055+0000 7f6b9dffb700 1 -- 172.21.15.189:0/3542452124 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4003ea0 con 0x7f6ba80b4d70 2024-04-06T12:24:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.055+0000 7f6b9dffb700 1 -- 172.21.15.189:0/3542452124 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba400dcc0 con 0x7f6ba80b4d70 2024-04-06T12:24:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.055+0000 7f6baf3c9700 1 -- 172.21.15.189:0/3542452124 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba80ad3f0 con 0x7f6ba80b4d70 2024-04-06T12:24:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.055+0000 7f6b9dffb700 1 -- 172.21.15.189:0/3542452124 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4010670 con 0x7f6ba80b4d70 2024-04-06T12:24:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.055+0000 7f6baf3c9700 1 -- 172.21.15.189:0/3542452124 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba815d070 con 0x7f6ba80b4d70 2024-04-06T12:24:23.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.056+0000 7f6baf3c9700 1 -- 172.21.15.189:0/3542452124 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ba80ad250 con 0x7f6ba80b4d70 2024-04-06T12:24:23.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.056+0000 7f6b9dffb700 1 -- 172.21.15.189:0/3542452124 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f6ba40193f0 con 0x7f6ba80b4d70 2024-04-06T12:24:23.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.057+0000 7f6b9dffb700 1 --2- 172.21.15.189:0/3542452124 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6b8c06ae90 0x7f6b8c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:23.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.057+0000 7f6b9dffb700 1 -- 172.21.15.189:0/3542452124 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f6ba408b870 con 0x7f6ba80b4d70 2024-04-06T12:24:23.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.057+0000 7f6bac964700 1 --2- 172.21.15.189:0/3542452124 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6b8c06ae90 0x7f6b8c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:23.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.057+0000 7f6bac964700 1 --2- 172.21.15.189:0/3542452124 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6b8c06ae90 0x7f6b8c06d350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6b98007fd0 tx=0x7f6b98007ec0 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:23.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.061+0000 7f6b9dffb700 1 -- 172.21.15.189:0/3542452124 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6ba80ad250 con 0x7f6ba80b4d70 2024-04-06T12:24:23.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.284+0000 7f6baf3c9700 1 -- 172.21.15.189:0/3542452124 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ba806fdd0 con 0x7f6b8c06ae90 2024-04-06T12:24:23.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.352+0000 7f6b9dffb700 1 -- 172.21.15.189:0/3542452124 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f6b98004d00 con 0x7f6b8c06ae90 2024-04-06T12:24:23.349 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2024-04-06T12:24:23.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.354+0000 7f6baf3c9700 1 -- 172.21.15.189:0/3542452124 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6b8c06ae90 msgr2=0x7f6b8c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:23.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.354+0000 7f6baf3c9700 1 --2- 172.21.15.189:0/3542452124 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6b8c06ae90 0x7f6b8c06d350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6b98007fd0 tx=0x7f6b98007ec0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.354+0000 7f6baf3c9700 1 -- 172.21.15.189:0/3542452124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ba80b4d70 msgr2=0x7f6ba80aefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:23.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.354+0000 7f6baf3c9700 1 --2- 172.21.15.189:0/3542452124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ba80b4d70 0x7f6ba80aefb0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f6ba806c820 tx=0x7f6ba4004750 comp rx=0 tx=0).stop 2024-04-06T12:24:23.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.354+0000 7f6baf3c9700 1 -- 172.21.15.189:0/3542452124 shutdown_connections 2024-04-06T12:24:23.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.354+0000 7f6baf3c9700 1 --2- 172.21.15.189:0/3542452124 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6b8c06ae90 0x7f6b8c06d350 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.354+0000 7f6baf3c9700 1 --2- 172.21.15.189:0/3542452124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba80ac8c0 0x7f6ba815cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.354+0000 7f6baf3c9700 1 --2- 172.21.15.189:0/3542452124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ba80b4d70 0x7f6ba80aefb0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.354+0000 7f6baf3c9700 1 --2- 172.21.15.189:0/3542452124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba80cf900 0x7f6ba8152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.354+0000 7f6baf3c9700 1 -- 172.21.15.189:0/3542452124 >> 172.21.15.189:0/3542452124 conn(0x7f6ba80d00b0 msgr2=0x7f6ba806f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:23.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.354+0000 7f6baf3c9700 1 -- 172.21.15.189:0/3542452124 shutdown_connections 2024-04-06T12:24:23.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.354+0000 7f6baf3c9700 1 -- 172.21.15.189:0/3542452124 wait complete. 2024-04-06T12:24:23.363 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2024-04-06T12:24:23.363 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir8 1m 2024-04-06T12:24:23.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.490+0000 7fc4f5f19700 1 Processor -- start 2024-04-06T12:24:23.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.490+0000 7fc4f5f19700 1 -- start start 2024-04-06T12:24:23.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.491+0000 7fc4f5f19700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4f015a550 0x7fc4f00cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:23.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.491+0000 7fc4f5f19700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc4f00cf900 0x7fc4f00b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:23.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.491+0000 7fc4f5f19700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f00cdbe0 con 0x7fc4f00b4ba0 2024-04-06T12:24:23.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.491+0000 7fc4f5f19700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f00b5010 con 0x7fc4f00cf900 2024-04-06T12:24:23.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.491+0000 7fc4f5f19700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f015a940 con 0x7fc4f015a550 2024-04-06T12:24:23.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.491+0000 7fc4eeffd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc4f00cf900 0x7fc4f00b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:23.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.491+0000 7fc4ef7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4f015a550 0x7fc4f00cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:23.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.491+0000 7fc4eeffd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc4f00cf900 0x7fc4f00b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:34472/0 (socket says 172.21.15.189:34472) 2024-04-06T12:24:23.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.491+0000 7fc4eeffd700 1 -- 172.21.15.189:0/2147588644 learned_addr learned my addr 172.21.15.189:0/2147588644 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:23.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.492+0000 7fc4eeffd700 1 -- 172.21.15.189:0/2147588644 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4f015a550 msgr2=0x7fc4f00cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:23.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.492+0000 7fc4eeffd700 1 --2- 172.21.15.189:0/2147588644 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4f015a550 0x7fc4f00cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.492+0000 7fc4eeffd700 1 -- 172.21.15.189:0/2147588644 >> v1:172.21.15.192:6789/0 conn(0x7fc4f00b4ba0 legacy=0x7fc4f006eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:23.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.492+0000 7fc4eeffd700 1 -- 172.21.15.189:0/2147588644 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f015ac00 con 0x7fc4f00cf900 2024-04-06T12:24:23.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.492+0000 7fc4ee7fc700 1 -- 172.21.15.189:0/2147588644 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275069110 0 0) 0x7fc4f00cdbe0 con 0x7fc4f00b4ba0 2024-04-06T12:24:23.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.492+0000 7fc4eeffd700 1 --2- 172.21.15.189:0/2147588644 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4f00cf900 0x7fc4f00b4660 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc4e00098f0 tx=0x7fc4e0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e274d3533dec496a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:23.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.492+0000 7fc4ee7fc700 1 -- 172.21.15.189:0/2147588644 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0011830 con 0x7fc4f00cf900 2024-04-06T12:24:23.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.492+0000 7fc4ee7fc700 1 -- 172.21.15.189:0/2147588644 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e0011990 con 0x7fc4f00cf900 2024-04-06T12:24:23.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.492+0000 7fc4ee7fc700 1 -- 172.21.15.189:0/2147588644 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0011d20 con 0x7fc4f00cf900 2024-04-06T12:24:23.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.493+0000 7fc4f5f19700 1 -- 172.21.15.189:0/2147588644 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4f00cf900 msgr2=0x7fc4f00b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:23.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.493+0000 7fc4f5f19700 1 --2- 172.21.15.189:0/2147588644 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4f00cf900 0x7fc4f00b4660 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc4e00098f0 tx=0x7fc4e0009c00 comp rx=0 tx=0).stop 2024-04-06T12:24:23.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.493+0000 7fc4f5f19700 1 -- 172.21.15.189:0/2147588644 shutdown_connections 2024-04-06T12:24:23.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.493+0000 7fc4f5f19700 1 --2- 172.21.15.189:0/2147588644 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4f015a550 0x7fc4f00cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.493+0000 7fc4f5f19700 1 --2- 172.21.15.189:0/2147588644 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4f00cf900 0x7fc4f00b4660 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.493+0000 7fc4f5f19700 1 -- 172.21.15.189:0/2147588644 >> 172.21.15.189:0/2147588644 conn(0x7fc4f00d00b0 msgr2=0x7fc4f00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:23.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.493+0000 7fc4f5f19700 1 -- 172.21.15.189:0/2147588644 shutdown_connections 2024-04-06T12:24:23.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.493+0000 7fc4f5f19700 1 -- 172.21.15.189:0/2147588644 wait complete. 2024-04-06T12:24:23.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.494+0000 7fc4f5f19700 1 Processor -- start 2024-04-06T12:24:23.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.494+0000 7fc4f5f19700 1 -- start start 2024-04-06T12:24:23.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.495+0000 7fc4f5f19700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4f00b4ba0 0x7fc4f0143ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:23.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.495+0000 7fc4f5f19700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4f00cf900 0x7fc4f014be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:23.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.495+0000 7fc4f5f19700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4f01f4f80 0x7fc4f0155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:23.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.495+0000 7fc4f5f19700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f014b0c0 con 0x7fc4f01f4f80 2024-04-06T12:24:23.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.495+0000 7fc4f5f19700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f014b380 con 0x7fc4f00cf900 2024-04-06T12:24:23.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.495+0000 7fc4f5f19700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f014b660 con 0x7fc4f00b4ba0 2024-04-06T12:24:23.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.495+0000 7fc4effff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4f01f4f80 0x7fc4f0155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:23.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.495+0000 7fc4effff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4f01f4f80 0x7fc4f0155f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33882/0 (socket says 172.21.15.189:33882) 2024-04-06T12:24:23.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.495+0000 7fc4effff700 1 -- 172.21.15.189:0/1628614196 learned_addr learned my addr 172.21.15.189:0/1628614196 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:23.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.495+0000 7fc4eeffd700 1 --2- 172.21.15.189:0/1628614196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4f00cf900 0x7fc4f014be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:23.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.495+0000 7fc4ef7fe700 1 --2- 172.21.15.189:0/1628614196 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4f00b4ba0 0x7fc4f0143ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:23.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.495+0000 7fc4effff700 1 -- 172.21.15.189:0/1628614196 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4f00b4ba0 msgr2=0x7fc4f0143ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:23.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.495+0000 7fc4effff700 1 --2- 172.21.15.189:0/1628614196 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4f00b4ba0 0x7fc4f0143ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.495+0000 7fc4effff700 1 -- 172.21.15.189:0/1628614196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4f00cf900 msgr2=0x7fc4f014be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:23.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.495+0000 7fc4effff700 1 --2- 172.21.15.189:0/1628614196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4f00cf900 0x7fc4f014be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.495+0000 7fc4effff700 1 -- 172.21.15.189:0/1628614196 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e0009550 con 0x7fc4f01f4f80 2024-04-06T12:24:23.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.495+0000 7fc4effff700 1 --2- 172.21.15.189:0/1628614196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4f01f4f80 0x7fc4f0155f90 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8001560 tx=0x7fc4e800cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:23.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.496+0000 7fc4d7fff700 1 -- 172.21.15.189:0/1628614196 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8003e30 con 0x7fc4f01f4f80 2024-04-06T12:24:23.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.496+0000 7fc4d7fff700 1 -- 172.21.15.189:0/1628614196 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e8005070 con 0x7fc4f01f4f80 2024-04-06T12:24:23.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.496+0000 7fc4d7fff700 1 -- 172.21.15.189:0/1628614196 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8005420 con 0x7fc4f01f4f80 2024-04-06T12:24:23.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.496+0000 7fc4f5f19700 1 -- 172.21.15.189:0/1628614196 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f014b940 con 0x7fc4f01f4f80 2024-04-06T12:24:23.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.496+0000 7fc4f5f19700 1 -- 172.21.15.189:0/1628614196 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f0152390 con 0x7fc4f01f4f80 2024-04-06T12:24:23.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.497+0000 7fc4f5f19700 1 -- 172.21.15.189:0/1628614196 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f0067bb0 con 0x7fc4f01f4f80 2024-04-06T12:24:23.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.497+0000 7fc4d7fff700 1 -- 172.21.15.189:0/1628614196 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fc4e8005580 con 0x7fc4f01f4f80 2024-04-06T12:24:23.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.497+0000 7fc4d7fff700 1 --2- 172.21.15.189:0/1628614196 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc4dc06ae90 0x7fc4dc06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:23.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.498+0000 7fc4d7fff700 1 -- 172.21.15.189:0/1628614196 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fc4e8088ec0 con 0x7fc4f01f4f80 2024-04-06T12:24:23.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.498+0000 7fc4ef7fe700 1 --2- 172.21.15.189:0/1628614196 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc4dc06ae90 0x7fc4dc06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:23.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.498+0000 7fc4ef7fe700 1 --2- 172.21.15.189:0/1628614196 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc4dc06ae90 0x7fc4dc06d350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009b40 tx=0x7fc4e4009210 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:23.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.502+0000 7fc4d7fff700 1 -- 172.21.15.189:0/1628614196 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc4e8053c50 con 0x7fc4f01f4f80 2024-04-06T12:24:23.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.725+0000 7fc4f5f19700 1 -- 172.21.15.189:0/1628614196 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4f014b4f0 con 0x7fc4dc06ae90 2024-04-06T12:24:23.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.794+0000 7fc4d7fff700 1 -- 172.21.15.189:0/1628614196 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fc4f014b4f0 con 0x7fc4dc06ae90 2024-04-06T12:24:23.792 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2024-04-06T12:24:23.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.796+0000 7fc4f5f19700 1 -- 172.21.15.189:0/1628614196 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc4dc06ae90 msgr2=0x7fc4dc06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:23.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.796+0000 7fc4f5f19700 1 --2- 172.21.15.189:0/1628614196 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc4dc06ae90 0x7fc4dc06d350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009b40 tx=0x7fc4e4009210 comp rx=0 tx=0).stop 2024-04-06T12:24:23.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.796+0000 7fc4f5f19700 1 -- 172.21.15.189:0/1628614196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4f01f4f80 msgr2=0x7fc4f0155f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:23.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.796+0000 7fc4f5f19700 1 --2- 172.21.15.189:0/1628614196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4f01f4f80 0x7fc4f0155f90 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8001560 tx=0x7fc4e800cb50 comp rx=0 tx=0).stop 2024-04-06T12:24:23.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.797+0000 7fc4f5f19700 1 -- 172.21.15.189:0/1628614196 shutdown_connections 2024-04-06T12:24:23.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.797+0000 7fc4f5f19700 1 --2- 172.21.15.189:0/1628614196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4f01f4f80 0x7fc4f0155f90 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.797+0000 7fc4f5f19700 1 --2- 172.21.15.189:0/1628614196 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4f00b4ba0 0x7fc4f0143ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.797+0000 7fc4f5f19700 1 --2- 172.21.15.189:0/1628614196 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc4dc06ae90 0x7fc4dc06d350 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.797+0000 7fc4f5f19700 1 --2- 172.21.15.189:0/1628614196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4f00cf900 0x7fc4f014be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.797+0000 7fc4f5f19700 1 -- 172.21.15.189:0/1628614196 >> 172.21.15.189:0/1628614196 conn(0x7fc4f00d00b0 msgr2=0x7fc4f00cc3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:23.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.797+0000 7fc4f5f19700 1 -- 172.21.15.189:0/1628614196 shutdown_connections 2024-04-06T12:24:23.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.797+0000 7fc4f5f19700 1 -- 172.21.15.189:0/1628614196 wait complete. 2024-04-06T12:24:23.806 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2024-04-06T12:24:23.806 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir9 1m 2024-04-06T12:24:23.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.934+0000 7fe3e4bae700 1 Processor -- start 2024-04-06T12:24:23.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.934+0000 7fe3e4bae700 1 -- start start 2024-04-06T12:24:23.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.935+0000 7fe3e4bae700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe3e00cbc20 0x7fe3e015f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:23.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.935+0000 7fe3e4bae700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e006aa10 0x7fe3e006ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:23.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.935+0000 7fe3e4bae700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3e00669d0 con 0x7fe3e015e8a0 2024-04-06T12:24:23.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.935+0000 7fe3e4bae700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3e006ede0 con 0x7fe3e00cbc20 2024-04-06T12:24:23.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.935+0000 7fe3e4bae700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e015ec80 con 0x7fe3e006aa10 2024-04-06T12:24:23.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.935+0000 7fe3ddd9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe3e00cbc20 0x7fe3e015f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:23.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.935+0000 7fe3ddd9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe3e00cbc20 0x7fe3e015f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33890/0 (socket says 172.21.15.189:33890) 2024-04-06T12:24:23.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.935+0000 7fe3ddd9b700 1 -- 172.21.15.189:0/3592181668 learned_addr learned my addr 172.21.15.189:0/3592181668 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:23.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.935+0000 7fe3ded9d700 1 --2- 172.21.15.189:0/3592181668 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e006aa10 0x7fe3e006ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:23.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.935+0000 7fe3ddd9b700 1 -- 172.21.15.189:0/3592181668 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e006aa10 msgr2=0x7fe3e006ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:23.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.935+0000 7fe3ddd9b700 1 --2- 172.21.15.189:0/3592181668 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e006aa10 0x7fe3e006ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.935+0000 7fe3ddd9b700 1 -- 172.21.15.189:0/3592181668 >> v1:172.21.15.192:6789/0 conn(0x7fe3e015e8a0 legacy=0x7fe3e015eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:23.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.935+0000 7fe3ddd9b700 1 -- 172.21.15.189:0/3592181668 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e00666f0 con 0x7fe3e00cbc20 2024-04-06T12:24:23.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.935+0000 7fe3ddd9b700 1 --2- 172.21.15.189:0/3592181668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e00cbc20 0x7fe3e015f5f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe3d40098f0 tx=0x7fe3d4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f4a03870b34d46c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:23.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.935+0000 7fe3dd59a700 1 -- 172.21.15.189:0/3592181668 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d400b8a0 con 0x7fe3e00cbc20 2024-04-06T12:24:23.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.935+0000 7fe3dd59a700 1 -- 172.21.15.189:0/3592181668 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d400ba00 con 0x7fe3e00cbc20 2024-04-06T12:24:23.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.935+0000 7fe3dd59a700 1 -- 172.21.15.189:0/3592181668 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d400bd90 con 0x7fe3e00cbc20 2024-04-06T12:24:23.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.936+0000 7fe3e4bae700 1 -- 172.21.15.189:0/3592181668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e00cbc20 msgr2=0x7fe3e015f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:23.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.936+0000 7fe3e4bae700 1 --2- 172.21.15.189:0/3592181668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e00cbc20 0x7fe3e015f5f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe3d40098f0 tx=0x7fe3d4009c00 comp rx=0 tx=0).stop 2024-04-06T12:24:23.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.936+0000 7fe3e4bae700 1 -- 172.21.15.189:0/3592181668 shutdown_connections 2024-04-06T12:24:23.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.936+0000 7fe3e4bae700 1 --2- 172.21.15.189:0/3592181668 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e006aa10 0x7fe3e006ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.936+0000 7fe3e4bae700 1 --2- 172.21.15.189:0/3592181668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e00cbc20 0x7fe3e015f5f0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.936+0000 7fe3e4bae700 1 -- 172.21.15.189:0/3592181668 >> 172.21.15.189:0/3592181668 conn(0x7fe3e00cc290 msgr2=0x7fe3e00cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:23.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.936+0000 7fe3e4bae700 1 -- 172.21.15.189:0/3592181668 shutdown_connections 2024-04-06T12:24:23.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.936+0000 7fe3e4bae700 1 -- 172.21.15.189:0/3592181668 wait complete. 2024-04-06T12:24:23.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.937+0000 7fe3e4bae700 1 Processor -- start 2024-04-06T12:24:23.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.937+0000 7fe3e4bae700 1 -- start start 2024-04-06T12:24:23.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.938+0000 7fe3e4bae700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e006aa10 0x7fe3e01f2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:23.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.938+0000 7fe3e4bae700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e00cbc20 0x7fe3e01451e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:23.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.938+0000 7fe3e4bae700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3e015e8a0 0x7fe3e015b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:23.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.938+0000 7fe3e4bae700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e00b5330 con 0x7fe3e006aa10 2024-04-06T12:24:23.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.938+0000 7fe3e4bae700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e01f2d40 con 0x7fe3e015e8a0 2024-04-06T12:24:23.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.938+0000 7fe3e4bae700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e00c4f00 con 0x7fe3e00cbc20 2024-04-06T12:24:23.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.938+0000 7fe3de59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e006aa10 0x7fe3e01f2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:23.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.938+0000 7fe3de59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e006aa10 0x7fe3e01f2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33892/0 (socket says 172.21.15.189:33892) 2024-04-06T12:24:23.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.938+0000 7fe3de59c700 1 -- 172.21.15.189:0/3032094628 learned_addr learned my addr 172.21.15.189:0/3032094628 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:23.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.938+0000 7fe3ded9d700 1 --2- 172.21.15.189:0/3032094628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3e015e8a0 0x7fe3e015b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:23.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.938+0000 7fe3ddd9b700 1 --2- 172.21.15.189:0/3032094628 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e00cbc20 0x7fe3e01451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:23.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.938+0000 7fe3de59c700 1 -- 172.21.15.189:0/3032094628 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e00cbc20 msgr2=0x7fe3e01451e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:23.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.938+0000 7fe3de59c700 1 --2- 172.21.15.189:0/3032094628 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e00cbc20 0x7fe3e01451e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.939+0000 7fe3de59c700 1 -- 172.21.15.189:0/3032094628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3e015e8a0 msgr2=0x7fe3e015b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:23.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.939+0000 7fe3de59c700 1 --2- 172.21.15.189:0/3032094628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3e015e8a0 0x7fe3e015b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:23.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.939+0000 7fe3de59c700 1 -- 172.21.15.189:0/3032094628 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d4009550 con 0x7fe3e006aa10 2024-04-06T12:24:23.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.939+0000 7fe3ddd9b700 1 --2- 172.21.15.189:0/3032094628 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e00cbc20 0x7fe3e01451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:24:23.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.939+0000 7fe3de59c700 1 --2- 172.21.15.189:0/3032094628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e006aa10 0x7fe3e01f2800 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe3d000c4e0 tx=0x7fe3d000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:23.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.939+0000 7fe3ceffd700 1 -- 172.21.15.189:0/3032094628 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d00052d0 con 0x7fe3e006aa10 2024-04-06T12:24:23.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.939+0000 7fe3ceffd700 1 -- 172.21.15.189:0/3032094628 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d0005430 con 0x7fe3e006aa10 2024-04-06T12:24:23.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.939+0000 7fe3ceffd700 1 -- 172.21.15.189:0/3032094628 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d00038c0 con 0x7fe3e006aa10 2024-04-06T12:24:23.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.939+0000 7fe3e4bae700 1 -- 172.21.15.189:0/3032094628 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e00b4db0 con 0x7fe3e006aa10 2024-04-06T12:24:23.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.939+0000 7fe3e4bae700 1 -- 172.21.15.189:0/3032094628 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3e00c4bd0 con 0x7fe3e006aa10 2024-04-06T12:24:23.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.940+0000 7fe3e4bae700 1 -- 172.21.15.189:0/3032094628 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e0072140 con 0x7fe3e006aa10 2024-04-06T12:24:23.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.940+0000 7fe3ceffd700 1 -- 172.21.15.189:0/3032094628 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fe3d0003a20 con 0x7fe3e006aa10 2024-04-06T12:24:23.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.941+0000 7fe3ceffd700 1 --2- 172.21.15.189:0/3032094628 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe3c006ae90 0x7fe3c006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:23.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.941+0000 7fe3ddd9b700 1 --2- 172.21.15.189:0/3032094628 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe3c006ae90 0x7fe3c006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:23.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.941+0000 7fe3ceffd700 1 -- 172.21.15.189:0/3032094628 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fe3d0088dd0 con 0x7fe3e006aa10 2024-04-06T12:24:23.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.941+0000 7fe3ddd9b700 1 --2- 172.21.15.189:0/3032094628 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe3c006ae90 0x7fe3c006d350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe3e00c9b20 tx=0x7fe3d4008030 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:23.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:23.945+0000 7fe3ceffd700 1 -- 172.21.15.189:0/3032094628 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe3d0053ad0 con 0x7fe3e006aa10 2024-04-06T12:24:24.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.168+0000 7fe3e4bae700 1 -- 172.21.15.189:0/3032094628 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3e006dfe0 con 0x7fe3c006ae90 2024-04-06T12:24:24.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.263+0000 7fe3ceffd700 1 -- 172.21.15.189:0/3032094628 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fe3d401f070 con 0x7fe3c006ae90 2024-04-06T12:24:24.260 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2024-04-06T12:24:24.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.265+0000 7fe3e4bae700 1 -- 172.21.15.189:0/3032094628 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe3c006ae90 msgr2=0x7fe3c006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:24.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.265+0000 7fe3e4bae700 1 --2- 172.21.15.189:0/3032094628 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe3c006ae90 0x7fe3c006d350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe3e00c9b20 tx=0x7fe3d4008030 comp rx=0 tx=0).stop 2024-04-06T12:24:24.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.265+0000 7fe3e4bae700 1 -- 172.21.15.189:0/3032094628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e006aa10 msgr2=0x7fe3e01f2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:24.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.265+0000 7fe3e4bae700 1 --2- 172.21.15.189:0/3032094628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e006aa10 0x7fe3e01f2800 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe3d000c4e0 tx=0x7fe3d000c8a0 comp rx=0 tx=0).stop 2024-04-06T12:24:24.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.265+0000 7fe3e4bae700 1 -- 172.21.15.189:0/3032094628 shutdown_connections 2024-04-06T12:24:24.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.265+0000 7fe3e4bae700 1 --2- 172.21.15.189:0/3032094628 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe3c006ae90 0x7fe3c006d350 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:24.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.265+0000 7fe3e4bae700 1 --2- 172.21.15.189:0/3032094628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3e015e8a0 0x7fe3e015b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:24.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.265+0000 7fe3e4bae700 1 --2- 172.21.15.189:0/3032094628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e006aa10 0x7fe3e01f2800 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:24.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.265+0000 7fe3e4bae700 1 --2- 172.21.15.189:0/3032094628 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e00cbc20 0x7fe3e01451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:24.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.265+0000 7fe3e4bae700 1 -- 172.21.15.189:0/3032094628 >> 172.21.15.189:0/3032094628 conn(0x7fe3e00cc290 msgr2=0x7fe3e006cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:24.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.265+0000 7fe3e4bae700 1 -- 172.21.15.189:0/3032094628 shutdown_connections 2024-04-06T12:24:24.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.265+0000 7fe3e4bae700 1 -- 172.21.15.189:0/3032094628 wait complete. 2024-04-06T12:24:24.274 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2024-04-06T12:24:24.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:24.275 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2024-04-06T12:24:24.294 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:24.294 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:24.294 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:24.294 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:24.294 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:24.294 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:24.294 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:24.294 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:24.294 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:24.294 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:24.294 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:24.295 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-06-12_14_00_UTC', 'scheduled-2024-04-06-12_15_00_UTC', 'scheduled-2024-04-06-12_16_00_UTC', 'scheduled-2024-04-06-12_17_00_UTC', 'scheduled-2024-04-06-12_18_00_UTC', 'scheduled-2024-04-06-12_19_00_UTC', 'scheduled-2024-04-06-12_20_00_UTC', 'scheduled-2024-04-06-12_21_00_UTC', 'scheduled-2024-04-06-12_22_00_UTC', 'scheduled-2024-04-06-12_23_00_UTC', 'scheduled-2024-04-06-12_24_00_UTC'] 2024-04-06T12:24:24.295 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir0 2024-04-06T12:24:24.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.460+0000 7fc7b386b700 1 Processor -- start 2024-04-06T12:24:24.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.460+0000 7fc7b386b700 1 -- start start 2024-04-06T12:24:24.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.461+0000 7fc7b386b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7ac0b3290 0x7fc7ac06cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:24.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.461+0000 7fc7b386b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc7ac15da30 0x7fc7ac06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:24.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.461+0000 7fc7b386b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ac06df40 con 0x7fc7ac0aee10 2024-04-06T12:24:24.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.461+0000 7fc7b386b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7ac06d330 con 0x7fc7ac15da30 2024-04-06T12:24:24.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.461+0000 7fc7b386b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac06d610 con 0x7fc7ac0b3290 2024-04-06T12:24:24.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.461+0000 7fc7b1e08700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc7ac15da30 0x7fc7ac06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:24.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.461+0000 7fc7b1607700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc7ac0aee10 0x7fc7ac0b2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48716/0 (socket says 172.21.15.189:48716) 2024-04-06T12:24:24.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.461+0000 7fc7b1607700 1 -- 172.21.15.189:0/2518729877 learned_addr learned my addr 172.21.15.189:0/2518729877 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:24.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.461+0000 7fc7b0e06700 1 --2- 172.21.15.189:0/2518729877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7ac0b3290 0x7fc7ac06cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:24.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.461+0000 7fc7b1e08700 1 -- 172.21.15.189:0/2518729877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7ac0b3290 msgr2=0x7fc7ac06cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:24.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.461+0000 7fc7b1e08700 1 --2- 172.21.15.189:0/2518729877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7ac0b3290 0x7fc7ac06cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:24.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.461+0000 7fc7b1e08700 1 -- 172.21.15.189:0/2518729877 >> v1:172.21.15.189:6789/0 conn(0x7fc7ac0aee10 legacy=0x7fc7ac0b2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:24.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.461+0000 7fc7b1e08700 1 -- 172.21.15.189:0/2518729877 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac06d8f0 con 0x7fc7ac15da30 2024-04-06T12:24:24.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.461+0000 7fc7b0e06700 1 --2- 172.21.15.189:0/2518729877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7ac0b3290 0x7fc7ac06cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:24:24.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.461+0000 7fc7b1e08700 1 --2- 172.21.15.189:0/2518729877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7ac15da30 0x7fc7ac06a730 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4009d90 tx=0x7fc7a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=134814bf75745349 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:24.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.461+0000 7fc7a3fff700 1 -- 172.21.15.189:0/2518729877 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a400e070 con 0x7fc7ac15da30 2024-04-06T12:24:24.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.462+0000 7fc7a3fff700 1 -- 172.21.15.189:0/2518729877 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a400bbd0 con 0x7fc7ac15da30 2024-04-06T12:24:24.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.462+0000 7fc7a3fff700 1 -- 172.21.15.189:0/2518729877 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a4012430 con 0x7fc7ac15da30 2024-04-06T12:24:24.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.462+0000 7fc7b386b700 1 -- 172.21.15.189:0/2518729877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7ac15da30 msgr2=0x7fc7ac06a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:24.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.462+0000 7fc7b386b700 1 --2- 172.21.15.189:0/2518729877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7ac15da30 0x7fc7ac06a730 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4009d90 tx=0x7fc7a400b330 comp rx=0 tx=0).stop 2024-04-06T12:24:24.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.462+0000 7fc7b386b700 1 -- 172.21.15.189:0/2518729877 shutdown_connections 2024-04-06T12:24:24.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.462+0000 7fc7b386b700 1 --2- 172.21.15.189:0/2518729877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7ac15da30 0x7fc7ac06a730 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:24.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.462+0000 7fc7b386b700 1 --2- 172.21.15.189:0/2518729877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7ac0b3290 0x7fc7ac06cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:24.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.462+0000 7fc7b386b700 1 -- 172.21.15.189:0/2518729877 >> 172.21.15.189:0/2518729877 conn(0x7fc7ac0b1d90 msgr2=0x7fc7ac0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:24.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.462+0000 7fc7b386b700 1 -- 172.21.15.189:0/2518729877 shutdown_connections 2024-04-06T12:24:24.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.462+0000 7fc7b386b700 1 -- 172.21.15.189:0/2518729877 wait complete. 2024-04-06T12:24:24.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.463+0000 7fc7b386b700 1 Processor -- start 2024-04-06T12:24:24.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.463+0000 7fc7b386b700 1 -- start start 2024-04-06T12:24:24.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.464+0000 7fc7b386b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7ac0aee10 0x7fc7ac1f63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:24.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.464+0000 7fc7b386b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7ac0b3290 0x7fc7ac1f6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:24.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.464+0000 7fc7b386b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7ac15da30 0x7fc7ac150830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:24.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.464+0000 7fc7b386b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac1515a0 con 0x7fc7ac0aee10 2024-04-06T12:24:24.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.464+0000 7fc7b386b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0cc0f0 con 0x7fc7ac0b3290 2024-04-06T12:24:24.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.464+0000 7fc7b386b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0cc3d0 con 0x7fc7ac15da30 2024-04-06T12:24:24.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.464+0000 7fc7b1607700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7ac0aee10 0x7fc7ac1f63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:24.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.464+0000 7fc7b1607700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7ac0aee10 0x7fc7ac1f63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33908/0 (socket says 172.21.15.189:33908) 2024-04-06T12:24:24.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.464+0000 7fc7b1607700 1 -- 172.21.15.189:0/4216747207 learned_addr learned my addr 172.21.15.189:0/4216747207 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:24.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.464+0000 7fc7b1e08700 1 --2- 172.21.15.189:0/4216747207 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7ac15da30 0x7fc7ac150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:24.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.464+0000 7fc7b0e06700 1 --2- 172.21.15.189:0/4216747207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7ac0b3290 0x7fc7ac1f6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:24.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.464+0000 7fc7b1607700 1 -- 172.21.15.189:0/4216747207 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7ac15da30 msgr2=0x7fc7ac150830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:24.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.464+0000 7fc7b1607700 1 --2- 172.21.15.189:0/4216747207 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7ac15da30 0x7fc7ac150830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:24.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.464+0000 7fc7b1607700 1 -- 172.21.15.189:0/4216747207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7ac0b3290 msgr2=0x7fc7ac1f6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:24.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.464+0000 7fc7b1607700 1 --2- 172.21.15.189:0/4216747207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7ac0b3290 0x7fc7ac1f6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:24.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.464+0000 7fc7b1607700 1 -- 172.21.15.189:0/4216747207 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a4009a70 con 0x7fc7ac0aee10 2024-04-06T12:24:24.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.465+0000 7fc7b1607700 1 --2- 172.21.15.189:0/4216747207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7ac0aee10 0x7fc7ac1f63f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800c520 tx=0x7fc7a800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:24.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.465+0000 7fc7a1ffb700 1 -- 172.21.15.189:0/4216747207 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8005d70 con 0x7fc7ac0aee10 2024-04-06T12:24:24.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.465+0000 7fc7a1ffb700 1 -- 172.21.15.189:0/4216747207 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a80070d0 con 0x7fc7ac0aee10 2024-04-06T12:24:24.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.465+0000 7fc7b386b700 1 -- 172.21.15.189:0/4216747207 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7ac0cc6b0 con 0x7fc7ac0aee10 2024-04-06T12:24:24.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.465+0000 7fc7a1ffb700 1 -- 172.21.15.189:0/4216747207 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8007440 con 0x7fc7ac0aee10 2024-04-06T12:24:24.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.465+0000 7fc7b386b700 1 -- 172.21.15.189:0/4216747207 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ac1f0610 con 0x7fc7ac0aee10 2024-04-06T12:24:24.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.466+0000 7fc7b386b700 1 -- 172.21.15.189:0/4216747207 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7ac0ccbd0 con 0x7fc7ac0aee10 2024-04-06T12:24:24.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.466+0000 7fc7a1ffb700 1 -- 172.21.15.189:0/4216747207 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fc7a80075a0 con 0x7fc7ac0aee10 2024-04-06T12:24:24.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.466+0000 7fc7a1ffb700 1 --2- 172.21.15.189:0/4216747207 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc79006ae90 0x7fc79006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:24.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.467+0000 7fc7a1ffb700 1 -- 172.21.15.189:0/4216747207 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fc7a8088cb0 con 0x7fc7ac0aee10 2024-04-06T12:24:24.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.467+0000 7fc7b0e06700 1 --2- 172.21.15.189:0/4216747207 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc79006ae90 0x7fc79006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:24.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.467+0000 7fc7b0e06700 1 --2- 172.21.15.189:0/4216747207 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc79006ae90 0x7fc79006d350 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac0cfff0 tx=0x7fc79c008040 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:24.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.471+0000 7fc7a1ffb700 1 -- 172.21.15.189:0/4216747207 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc7ac0ccbd0 con 0x7fc7ac0aee10 2024-04-06T12:24:24.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.693+0000 7fc7b386b700 1 -- 172.21.15.189:0/4216747207 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc7ac0af1f0 con 0x7fc79006ae90 2024-04-06T12:24:24.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.701+0000 7fc7a1ffb700 1 -- 172.21.15.189:0/4216747207 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fc79c00c050 con 0x7fc79006ae90 2024-04-06T12:24:24.698 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:24.699 INFO:teuthology.orchestra.run.smithi189.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:35", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:24.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.703+0000 7fc7b386b700 1 -- 172.21.15.189:0/4216747207 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc79006ae90 msgr2=0x7fc79006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:24.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.703+0000 7fc7b386b700 1 --2- 172.21.15.189:0/4216747207 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc79006ae90 0x7fc79006d350 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac0cfff0 tx=0x7fc79c008040 comp rx=0 tx=0).stop 2024-04-06T12:24:24.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.703+0000 7fc7b386b700 1 -- 172.21.15.189:0/4216747207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7ac0aee10 msgr2=0x7fc7ac1f63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:24.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.703+0000 7fc7b386b700 1 --2- 172.21.15.189:0/4216747207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7ac0aee10 0x7fc7ac1f63f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800c520 tx=0x7fc7a800c8e0 comp rx=0 tx=0).stop 2024-04-06T12:24:24.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.703+0000 7fc7b386b700 1 -- 172.21.15.189:0/4216747207 shutdown_connections 2024-04-06T12:24:24.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.703+0000 7fc7b386b700 1 --2- 172.21.15.189:0/4216747207 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7ac15da30 0x7fc7ac150830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:24.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.703+0000 7fc7b386b700 1 --2- 172.21.15.189:0/4216747207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7ac0aee10 0x7fc7ac1f63f0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:24.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.703+0000 7fc7b386b700 1 --2- 172.21.15.189:0/4216747207 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc79006ae90 0x7fc79006d350 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:24.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.703+0000 7fc7b386b700 1 --2- 172.21.15.189:0/4216747207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7ac0b3290 0x7fc7ac1f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:24.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.703+0000 7fc7b386b700 1 -- 172.21.15.189:0/4216747207 >> 172.21.15.189:0/4216747207 conn(0x7fc7ac0b1d90 msgr2=0x7fc7ac0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:24.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.703+0000 7fc7b386b700 1 -- 172.21.15.189:0/4216747207 shutdown_connections 2024-04-06T12:24:24.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.703+0000 7fc7b386b700 1 -- 172.21.15.189:0/4216747207 wait complete. 2024-04-06T12:24:24.712 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:35", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:24.712 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1m', 'retention': {}, 'start': '2024-04-06T00:00:00', 'created': '2024-04-06T12:13:35', 'first': '2024-04-06T12:14:00', 'last': '2024-04-06T12:24:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-06T12:24:24.712 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-06T12:24:24.713 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-06T12:24:24.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:24.713 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2024-04-06T12:24:24.731 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:24.732 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:24.732 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:24.732 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:24.732 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:24.732 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:24.732 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:24.732 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:24.732 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:24.732 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:24.732 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:24.732 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-06-12_14_00_UTC', 'scheduled-2024-04-06-12_15_00_UTC', 'scheduled-2024-04-06-12_16_00_UTC', 'scheduled-2024-04-06-12_17_00_UTC', 'scheduled-2024-04-06-12_18_00_UTC', 'scheduled-2024-04-06-12_19_00_UTC', 'scheduled-2024-04-06-12_20_00_UTC', 'scheduled-2024-04-06-12_21_00_UTC', 'scheduled-2024-04-06-12_22_00_UTC', 'scheduled-2024-04-06-12_23_00_UTC', 'scheduled-2024-04-06-12_24_00_UTC'] 2024-04-06T12:24:24.733 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir1 2024-04-06T12:24:24.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.898+0000 7f4d8a2ad700 1 Processor -- start 2024-04-06T12:24:24.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.898+0000 7f4d8a2ad700 1 -- start start 2024-04-06T12:24:24.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.899+0000 7f4d8a2ad700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4d8415a550 0x7f4d840cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:24.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.899+0000 7f4d8a2ad700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d840cdbe0 con 0x7f4d840cf900 2024-04-06T12:24:24.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.899+0000 7f4d8a2ad700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8415a940 con 0x7f4d840b4d70 2024-04-06T12:24:24.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.899+0000 7f4d8a2ad700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d8415ac00 con 0x7f4d8415a550 2024-04-06T12:24:24.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.899+0000 7f4d83fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4d8415a550 0x7f4d840cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:24.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.899+0000 7f4d837fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4d840cf900 0x7f4d840b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48718/0 (socket says 172.21.15.189:48718) 2024-04-06T12:24:24.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.899+0000 7f4d837fe700 1 -- 172.21.15.189:0/1855087931 learned_addr learned my addr 172.21.15.189:0/1855087931 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:24.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.899+0000 7f4d83fff700 1 -- 172.21.15.189:0/1855087931 >> v1:172.21.15.192:6789/0 conn(0x7f4d840b4d70 legacy=0x7f4d840b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:24.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.899+0000 7f4d83fff700 1 -- 172.21.15.189:0/1855087931 >> v1:172.21.15.189:6789/0 conn(0x7f4d840cf900 legacy=0x7f4d840b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:24.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.899+0000 7f4d82ffd700 1 -- 172.21.15.189:0/1855087931 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871007612 0 0) 0x7f4d840cdbe0 con 0x7f4d840cf900 2024-04-06T12:24:24.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.899+0000 7f4d83fff700 1 -- 172.21.15.189:0/1855087931 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8415aee0 con 0x7f4d8415a550 2024-04-06T12:24:24.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.899+0000 7f4d83fff700 1 --2- 172.21.15.189:0/1855087931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d8415a550 0x7f4d840cf3c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f4d78009840 tx=0x7f4d78009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cadd86d4800b1781 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:24.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.900+0000 7f4d82ffd700 1 -- 172.21.15.189:0/1855087931 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d78004230 con 0x7f4d8415a550 2024-04-06T12:24:24.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.900+0000 7f4d82ffd700 1 -- 172.21.15.189:0/1855087931 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d78004390 con 0x7f4d8415a550 2024-04-06T12:24:24.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.900+0000 7f4d82ffd700 1 -- 172.21.15.189:0/1855087931 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d780036a0 con 0x7f4d8415a550 2024-04-06T12:24:24.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.900+0000 7f4d8a2ad700 1 -- 172.21.15.189:0/1855087931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d8415a550 msgr2=0x7f4d840cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:24.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.900+0000 7f4d8a2ad700 1 --2- 172.21.15.189:0/1855087931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d8415a550 0x7f4d840cf3c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f4d78009840 tx=0x7f4d78009b50 comp rx=0 tx=0).stop 2024-04-06T12:24:24.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.900+0000 7f4d8a2ad700 1 -- 172.21.15.189:0/1855087931 shutdown_connections 2024-04-06T12:24:24.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.900+0000 7f4d8a2ad700 1 --2- 172.21.15.189:0/1855087931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d8415a550 0x7f4d840cf3c0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:24.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.900+0000 7f4d8a2ad700 1 -- 172.21.15.189:0/1855087931 >> 172.21.15.189:0/1855087931 conn(0x7f4d840d00b0 msgr2=0x7f4d840d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:24.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.900+0000 7f4d8a2ad700 1 -- 172.21.15.189:0/1855087931 shutdown_connections 2024-04-06T12:24:24.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.900+0000 7f4d8a2ad700 1 -- 172.21.15.189:0/1855087931 wait complete. 2024-04-06T12:24:24.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.901+0000 7f4d8a2ad700 1 Processor -- start 2024-04-06T12:24:24.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.901+0000 7f4d8a2ad700 1 -- start start 2024-04-06T12:24:24.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.902+0000 7f4d8a2ad700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d840b4d70 0x7f4d840aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:24.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.902+0000 7f4d8a2ad700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d840cf900 0x7f4d841534d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.902+0000 7f4d8a2ad700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d84153a10 0x7f4d84155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.902+0000 7f4d8a2ad700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d840af2b0 con 0x7f4d840b4d70 2024-04-06T12:24:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.902+0000 7f4d8a2ad700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d84153f20 con 0x7f4d840cf900 2024-04-06T12:24:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.902+0000 7f4d8a2ad700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d84154200 con 0x7f4d84153a10 2024-04-06T12:24:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.902+0000 7f4d83fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d840b4d70 0x7f4d840aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.902+0000 7f4d83fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d840b4d70 0x7f4d840aed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33926/0 (socket says 172.21.15.189:33926) 2024-04-06T12:24:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.902+0000 7f4d83fff700 1 -- 172.21.15.189:0/3251580831 learned_addr learned my addr 172.21.15.189:0/3251580831 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.903+0000 7f4d837fe700 1 --2- 172.21.15.189:0/3251580831 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d840cf900 0x7f4d841534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.903+0000 7f4d8884a700 1 --2- 172.21.15.189:0/3251580831 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d84153a10 0x7f4d84155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.903+0000 7f4d83fff700 1 -- 172.21.15.189:0/3251580831 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d84153a10 msgr2=0x7f4d84155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.903+0000 7f4d83fff700 1 --2- 172.21.15.189:0/3251580831 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d84153a10 0x7f4d84155920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.903+0000 7f4d83fff700 1 -- 172.21.15.189:0/3251580831 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d840cf900 msgr2=0x7f4d841534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.903+0000 7f4d83fff700 1 --2- 172.21.15.189:0/3251580831 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d840cf900 0x7f4d841534d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.903+0000 7f4d83fff700 1 -- 172.21.15.189:0/3251580831 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d780094a0 con 0x7f4d840b4d70 2024-04-06T12:24:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.903+0000 7f4d8884a700 1 --2- 172.21.15.189:0/3251580831 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d84153a10 0x7f4d84155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:24:24.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.903+0000 7f4d83fff700 1 --2- 172.21.15.189:0/3251580831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d840b4d70 0x7f4d840aed70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f4d780090b0 tx=0x7f4d780047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:24.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.903+0000 7f4d80ff9700 1 -- 172.21.15.189:0/3251580831 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d78011630 con 0x7f4d840b4d70 2024-04-06T12:24:24.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.903+0000 7f4d80ff9700 1 -- 172.21.15.189:0/3251580831 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d78011790 con 0x7f4d840b4d70 2024-04-06T12:24:24.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.903+0000 7f4d8a2ad700 1 -- 172.21.15.189:0/3251580831 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d841544e0 con 0x7f4d840b4d70 2024-04-06T12:24:24.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.903+0000 7f4d80ff9700 1 -- 172.21.15.189:0/3251580831 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d78011b40 con 0x7f4d840b4d70 2024-04-06T12:24:24.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.903+0000 7f4d8a2ad700 1 -- 172.21.15.189:0/3251580831 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d84154aa0 con 0x7f4d840b4d70 2024-04-06T12:24:24.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.904+0000 7f4d8a2ad700 1 -- 172.21.15.189:0/3251580831 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d84067bb0 con 0x7f4d840b4d70 2024-04-06T12:24:24.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.904+0000 7f4d80ff9700 1 -- 172.21.15.189:0/3251580831 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f4d78011ca0 con 0x7f4d840b4d70 2024-04-06T12:24:24.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.905+0000 7f4d80ff9700 1 --2- 172.21.15.189:0/3251580831 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4d7006ae90 0x7f4d7006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:24.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.905+0000 7f4d80ff9700 1 -- 172.21.15.189:0/3251580831 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f4d7808b1c0 con 0x7f4d840b4d70 2024-04-06T12:24:24.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.905+0000 7f4d837fe700 1 --2- 172.21.15.189:0/3251580831 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4d7006ae90 0x7f4d7006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:24.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.905+0000 7f4d837fe700 1 --2- 172.21.15.189:0/3251580831 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4d7006ae90 0x7f4d7006d350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4d84070000 tx=0x7f4d74007be0 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:24.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:24.909+0000 7f4d80ff9700 1 -- 172.21.15.189:0/3251580831 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4d78056070 con 0x7f4d840b4d70 2024-04-06T12:24:25.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.129+0000 7f4d8a2ad700 1 -- 172.21.15.189:0/3251580831 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4d8406fdd0 con 0x7f4d7006ae90 2024-04-06T12:24:25.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.137+0000 7f4d80ff9700 1 -- 172.21.15.189:0/3251580831 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f4d74005aa0 con 0x7f4d7006ae90 2024-04-06T12:24:25.134 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:25.134 INFO:teuthology.orchestra.run.smithi189.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:36", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:25.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.139+0000 7f4d8a2ad700 1 -- 172.21.15.189:0/3251580831 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4d7006ae90 msgr2=0x7f4d7006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:25.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.139+0000 7f4d8a2ad700 1 --2- 172.21.15.189:0/3251580831 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4d7006ae90 0x7f4d7006d350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4d84070000 tx=0x7f4d74007be0 comp rx=0 tx=0).stop 2024-04-06T12:24:25.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.139+0000 7f4d8a2ad700 1 -- 172.21.15.189:0/3251580831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d840b4d70 msgr2=0x7f4d840aed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:25.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.139+0000 7f4d8a2ad700 1 --2- 172.21.15.189:0/3251580831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d840b4d70 0x7f4d840aed70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f4d780090b0 tx=0x7f4d780047c0 comp rx=0 tx=0).stop 2024-04-06T12:24:25.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.139+0000 7f4d8a2ad700 1 -- 172.21.15.189:0/3251580831 shutdown_connections 2024-04-06T12:24:25.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.139+0000 7f4d8a2ad700 1 --2- 172.21.15.189:0/3251580831 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d84153a10 0x7f4d84155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:25.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.139+0000 7f4d8a2ad700 1 --2- 172.21.15.189:0/3251580831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d840b4d70 0x7f4d840aed70 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:25.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.139+0000 7f4d8a2ad700 1 --2- 172.21.15.189:0/3251580831 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4d7006ae90 0x7f4d7006d350 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:25.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.139+0000 7f4d8a2ad700 1 --2- 172.21.15.189:0/3251580831 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d840cf900 0x7f4d841534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:25.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.139+0000 7f4d8a2ad700 1 -- 172.21.15.189:0/3251580831 >> 172.21.15.189:0/3251580831 conn(0x7f4d840d00b0 msgr2=0x7f4d840b44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:25.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.139+0000 7f4d8a2ad700 1 -- 172.21.15.189:0/3251580831 shutdown_connections 2024-04-06T12:24:25.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.139+0000 7f4d8a2ad700 1 -- 172.21.15.189:0/3251580831 wait complete. 2024-04-06T12:24:25.148 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:36", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:25.148 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1m', 'retention': {}, 'start': '2024-04-06T00:00:00', 'created': '2024-04-06T12:13:36', 'first': '2024-04-06T12:14:00', 'last': '2024-04-06T12:24:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-06T12:24:25.148 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-06T12:24:25.148 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-06T12:24:25.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:25.149 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2024-04-06T12:24:25.167 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:25.167 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:25.167 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:25.167 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:25.167 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:25.167 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:25.167 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:25.167 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:25.167 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:25.167 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:25.167 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:25.168 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-06-12_14_00_UTC', 'scheduled-2024-04-06-12_15_00_UTC', 'scheduled-2024-04-06-12_16_00_UTC', 'scheduled-2024-04-06-12_17_00_UTC', 'scheduled-2024-04-06-12_18_00_UTC', 'scheduled-2024-04-06-12_19_00_UTC', 'scheduled-2024-04-06-12_20_00_UTC', 'scheduled-2024-04-06-12_21_00_UTC', 'scheduled-2024-04-06-12_22_00_UTC', 'scheduled-2024-04-06-12_23_00_UTC', 'scheduled-2024-04-06-12_24_00_UTC'] 2024-04-06T12:24:25.168 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir2 2024-04-06T12:24:25.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.335+0000 7f10ed616700 1 Processor -- start 2024-04-06T12:24:25.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.335+0000 7f10ed616700 1 -- start start 2024-04-06T12:24:25.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.336+0000 7f10ed616700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f10e80ce730 0x7f10e815dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:25.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.336+0000 7f10ed616700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f10e80b8d70 0x7f10e80d1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:25.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.336+0000 7f10ed616700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e815eac0 con 0x7f10e815d6f0 2024-04-06T12:24:25.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.336+0000 7f10ed616700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e815e010 con 0x7f10e80b8d70 2024-04-06T12:24:25.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.336+0000 7f10ed616700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e815e220 con 0x7f10e80ce730 2024-04-06T12:24:25.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.336+0000 7f10e77fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f10e80b8d70 0x7f10e80d1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:25.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.336+0000 7f10e6ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f10e815d6f0 0x7f10e80ce020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48720/0 (socket says 172.21.15.189:48720) 2024-04-06T12:24:25.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.336+0000 7f10e6ffd700 1 -- 172.21.15.189:0/4185469293 learned_addr learned my addr 172.21.15.189:0/4185469293 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:25.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.336+0000 7f10e67fc700 1 --2- 172.21.15.189:0/4185469293 >> v2:172.21.15.192:3300/0 conn(0x7f10e80ce730 0x7f10e815dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:25.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.336+0000 7f10e77fe700 1 -- 172.21.15.189:0/4185469293 >> v2:172.21.15.192:3300/0 conn(0x7f10e80ce730 msgr2=0x7f10e815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:25.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.336+0000 7f10e77fe700 1 --2- 172.21.15.189:0/4185469293 >> v2:172.21.15.192:3300/0 conn(0x7f10e80ce730 0x7f10e815dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:25.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.336+0000 7f10e77fe700 1 -- 172.21.15.189:0/4185469293 >> v1:172.21.15.189:6789/0 conn(0x7f10e815d6f0 legacy=0x7f10e80ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:25.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.336+0000 7f10e5ffb700 1 -- 172.21.15.189:0/4185469293 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427872043 0 0) 0x7f10e815eac0 con 0x7f10e815d6f0 2024-04-06T12:24:25.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.336+0000 7f10e77fe700 1 -- 172.21.15.189:0/4185469293 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e815e500 con 0x7f10e80b8d70 2024-04-06T12:24:25.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.336+0000 7f10e77fe700 1 --2- 172.21.15.189:0/4185469293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e80b8d70 0x7f10e80d1e00 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f10e80b6350 tx=0x7f10e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e4ba52f7766758d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:25.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.336+0000 7f10e5ffb700 1 -- 172.21.15.189:0/4185469293 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e000e070 con 0x7f10e80b8d70 2024-04-06T12:24:25.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.336+0000 7f10e5ffb700 1 -- 172.21.15.189:0/4185469293 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10e000bbd0 con 0x7f10e80b8d70 2024-04-06T12:24:25.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.336+0000 7f10e5ffb700 1 -- 172.21.15.189:0/4185469293 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e0012430 con 0x7f10e80b8d70 2024-04-06T12:24:25.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.337+0000 7f10ed616700 1 -- 172.21.15.189:0/4185469293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e80b8d70 msgr2=0x7f10e80d1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:25.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.337+0000 7f10ed616700 1 --2- 172.21.15.189:0/4185469293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e80b8d70 0x7f10e80d1e00 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f10e80b6350 tx=0x7f10e000b330 comp rx=0 tx=0).stop 2024-04-06T12:24:25.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.337+0000 7f10ed616700 1 -- 172.21.15.189:0/4185469293 shutdown_connections 2024-04-06T12:24:25.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.337+0000 7f10ed616700 1 --2- 172.21.15.189:0/4185469293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e80b8d70 0x7f10e80d1e00 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:25.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.337+0000 7f10ed616700 1 --2- 172.21.15.189:0/4185469293 >> v2:172.21.15.192:3300/0 conn(0x7f10e80ce730 0x7f10e815dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:25.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.337+0000 7f10ed616700 1 -- 172.21.15.189:0/4185469293 >> 172.21.15.189:0/4185469293 conn(0x7f10e8072490 msgr2=0x7f10e806b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:25.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.337+0000 7f10ed616700 1 -- 172.21.15.189:0/4185469293 shutdown_connections 2024-04-06T12:24:25.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.337+0000 7f10ed616700 1 -- 172.21.15.189:0/4185469293 wait complete. 2024-04-06T12:24:25.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.338+0000 7f10ed616700 1 Processor -- start 2024-04-06T12:24:25.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.338+0000 7f10ed616700 1 -- start start 2024-04-06T12:24:25.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.339+0000 7f10ed616700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e80b8d70 0x7f10e80b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:25.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.339+0000 7f10ed616700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e80ce730 0x7f10e8155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:25.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.339+0000 7f10ed616700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e815d6f0 0x7f10e80b6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:25.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.339+0000 7f10ed616700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e8156430 con 0x7f10e80ce730 2024-04-06T12:24:25.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.339+0000 7f10ed616700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e81570f0 con 0x7f10e815d6f0 2024-04-06T12:24:25.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.339+0000 7f10ed616700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e81573d0 con 0x7f10e80b8d70 2024-04-06T12:24:25.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.339+0000 7f10e67fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e80ce730 0x7f10e8155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:25.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.339+0000 7f10e67fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e80ce730 0x7f10e8155ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33942/0 (socket says 172.21.15.189:33942) 2024-04-06T12:24:25.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.339+0000 7f10e67fc700 1 -- 172.21.15.189:0/2878923234 learned_addr learned my addr 172.21.15.189:0/2878923234 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:25.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.339+0000 7f10e6ffd700 1 --2- 172.21.15.189:0/2878923234 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e80b8d70 0x7f10e80b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:25.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.339+0000 7f10e77fe700 1 --2- 172.21.15.189:0/2878923234 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e815d6f0 0x7f10e80b6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:25.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.340+0000 7f10e67fc700 1 -- 172.21.15.189:0/2878923234 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e80b8d70 msgr2=0x7f10e80b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:25.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.340+0000 7f10e67fc700 1 --2- 172.21.15.189:0/2878923234 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e80b8d70 0x7f10e80b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:25.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.340+0000 7f10e67fc700 1 -- 172.21.15.189:0/2878923234 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e815d6f0 msgr2=0x7f10e80b6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:25.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.340+0000 7f10e67fc700 1 --2- 172.21.15.189:0/2878923234 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e815d6f0 0x7f10e80b6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:25.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.340+0000 7f10e67fc700 1 -- 172.21.15.189:0/2878923234 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e0009a70 con 0x7f10e80ce730 2024-04-06T12:24:25.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.340+0000 7f10e67fc700 1 --2- 172.21.15.189:0/2878923234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e80ce730 0x7f10e8155ef0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00e7e0 tx=0x7f10dc00eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:25.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.340+0000 7f10cf7fe700 1 -- 172.21.15.189:0/2878923234 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc013610 con 0x7f10e80ce730 2024-04-06T12:24:25.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.340+0000 7f10cf7fe700 1 -- 172.21.15.189:0/2878923234 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10dc013770 con 0x7f10e80ce730 2024-04-06T12:24:25.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.340+0000 7f10ed616700 1 -- 172.21.15.189:0/2878923234 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e81576b0 con 0x7f10e80ce730 2024-04-06T12:24:25.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.340+0000 7f10cf7fe700 1 -- 172.21.15.189:0/2878923234 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc013ae0 con 0x7f10e80ce730 2024-04-06T12:24:25.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.340+0000 7f10ed616700 1 -- 172.21.15.189:0/2878923234 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e8157c70 con 0x7f10e80ce730 2024-04-06T12:24:25.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.341+0000 7f10ed616700 1 -- 172.21.15.189:0/2878923234 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e80b76b0 con 0x7f10e80ce730 2024-04-06T12:24:25.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.341+0000 7f10cf7fe700 1 -- 172.21.15.189:0/2878923234 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f10dc013c40 con 0x7f10e80ce730 2024-04-06T12:24:25.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.342+0000 7f10cf7fe700 1 --2- 172.21.15.189:0/2878923234 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10d406ae90 0x7f10d406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:25.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.342+0000 7f10cf7fe700 1 -- 172.21.15.189:0/2878923234 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f10dc089750 con 0x7f10e80ce730 2024-04-06T12:24:25.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.342+0000 7f10e6ffd700 1 --2- 172.21.15.189:0/2878923234 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10d406ae90 0x7f10d406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:25.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.342+0000 7f10e6ffd700 1 --2- 172.21.15.189:0/2878923234 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10d406ae90 0x7f10d406d350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f10d8000f60 tx=0x7f10d8007c90 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:25.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.346+0000 7f10cf7fe700 1 -- 172.21.15.189:0/2878923234 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f10dc0542c0 con 0x7f10e80ce730 2024-04-06T12:24:25.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.565+0000 7f10ed616700 1 -- 172.21.15.189:0/2878923234 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f10e80d0750 con 0x7f10d406ae90 2024-04-06T12:24:25.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.573+0000 7f10cf7fe700 1 -- 172.21.15.189:0/2878923234 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f10d8013420 con 0x7f10d406ae90 2024-04-06T12:24:25.570 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:25.570 INFO:teuthology.orchestra.run.smithi189.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:36", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:25.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.575+0000 7f10ed616700 1 -- 172.21.15.189:0/2878923234 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10d406ae90 msgr2=0x7f10d406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:25.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.575+0000 7f10ed616700 1 --2- 172.21.15.189:0/2878923234 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10d406ae90 0x7f10d406d350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f10d8000f60 tx=0x7f10d8007c90 comp rx=0 tx=0).stop 2024-04-06T12:24:25.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.575+0000 7f10ed616700 1 -- 172.21.15.189:0/2878923234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e80ce730 msgr2=0x7f10e8155ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:25.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.575+0000 7f10ed616700 1 --2- 172.21.15.189:0/2878923234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e80ce730 0x7f10e8155ef0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00e7e0 tx=0x7f10dc00eba0 comp rx=0 tx=0).stop 2024-04-06T12:24:25.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.575+0000 7f10ed616700 1 -- 172.21.15.189:0/2878923234 shutdown_connections 2024-04-06T12:24:25.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.575+0000 7f10ed616700 1 --2- 172.21.15.189:0/2878923234 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10d406ae90 0x7f10d406d350 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:25.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.575+0000 7f10ed616700 1 --2- 172.21.15.189:0/2878923234 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e815d6f0 0x7f10e80b6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:25.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.575+0000 7f10ed616700 1 --2- 172.21.15.189:0/2878923234 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e80b8d70 0x7f10e80b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:25.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.575+0000 7f10ed616700 1 --2- 172.21.15.189:0/2878923234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e80ce730 0x7f10e8155ef0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:25.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.575+0000 7f10ed616700 1 -- 172.21.15.189:0/2878923234 >> 172.21.15.189:0/2878923234 conn(0x7f10e8072490 msgr2=0x7f10e806b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:25.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.575+0000 7f10ed616700 1 -- 172.21.15.189:0/2878923234 shutdown_connections 2024-04-06T12:24:25.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.575+0000 7f10ed616700 1 -- 172.21.15.189:0/2878923234 wait complete. 2024-04-06T12:24:25.584 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:36", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:25.584 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1m', 'retention': {}, 'start': '2024-04-06T00:00:00', 'created': '2024-04-06T12:13:36', 'first': '2024-04-06T12:14:00', 'last': '2024-04-06T12:24:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-06T12:24:25.584 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-06T12:24:25.584 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-06T12:24:25.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:25.585 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2024-04-06T12:24:25.603 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:25.603 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:25.603 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:25.603 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:25.603 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:25.603 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:25.603 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:25.603 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:25.603 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:25.603 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:25.603 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:25.604 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-06-12_14_00_UTC', 'scheduled-2024-04-06-12_15_00_UTC', 'scheduled-2024-04-06-12_16_00_UTC', 'scheduled-2024-04-06-12_17_00_UTC', 'scheduled-2024-04-06-12_18_00_UTC', 'scheduled-2024-04-06-12_19_00_UTC', 'scheduled-2024-04-06-12_20_00_UTC', 'scheduled-2024-04-06-12_21_00_UTC', 'scheduled-2024-04-06-12_22_00_UTC', 'scheduled-2024-04-06-12_23_00_UTC', 'scheduled-2024-04-06-12_24_00_UTC'] 2024-04-06T12:24:25.604 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir3 2024-04-06T12:24:25.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.771+0000 7f05f1356700 1 Processor -- start 2024-04-06T12:24:25.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.771+0000 7f05f1356700 1 -- start start 2024-04-06T12:24:25.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.772+0000 7f05f1356700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ec143470 0x7f05ec0b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:25.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.772+0000 7f05f1356700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05ec15eb40 con 0x7f05ec15db10 2024-04-06T12:24:25.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.772+0000 7f05f1356700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05ec0d0bf0 con 0x7f05ec160b60 2024-04-06T12:24:25.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.772+0000 7f05f1356700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ec160f40 con 0x7f05ec143470 2024-04-06T12:24:25.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.772+0000 7f05ea7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f05ec15db10 0x7f05ec0cd890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48736/0 (socket says 172.21.15.189:48736) 2024-04-06T12:24:25.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.772+0000 7f05ea7fc700 1 -- 172.21.15.189:0/2393307350 learned_addr learned my addr 172.21.15.189:0/2393307350 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:25.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.772+0000 7f05eb7fe700 1 --2- 172.21.15.189:0/2393307350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ec143470 0x7f05ec0b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:25.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.772+0000 7f05e9ffb700 1 -- 172.21.15.189:0/2393307350 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93519574 0 0) 0x7f05d4001290 con 0x7f05ec15db10 2024-04-06T12:24:25.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.772+0000 7f05e9ffb700 1 -- 172.21.15.189:0/2393307350 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05d80035b0 con 0x7f05ec15db10 2024-04-06T12:24:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.772+0000 7f05e9ffb700 1 -- 172.21.15.189:0/2393307350 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328809118 0 0) 0x7f05ec0d0bf0 con 0x7f05ec160b60 2024-04-06T12:24:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.772+0000 7f05e9ffb700 1 -- 172.21.15.189:0/2393307350 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05d4001290 con 0x7f05ec160b60 2024-04-06T12:24:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.773+0000 7f05eb7fe700 1 -- 172.21.15.189:0/2393307350 >> v1:172.21.15.192:6789/0 conn(0x7f05ec160b60 legacy=0x7f05ec142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.773+0000 7f05eb7fe700 1 -- 172.21.15.189:0/2393307350 >> v1:172.21.15.189:6789/0 conn(0x7f05ec15db10 legacy=0x7f05ec0cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.773+0000 7f05e9ffb700 1 -- 172.21.15.189:0/2393307350 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1889132315 0 0) 0x7f05d80035b0 con 0x7f05ec15db10 2024-04-06T12:24:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.773+0000 7f05eb7fe700 1 -- 172.21.15.189:0/2393307350 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ec15dfc0 con 0x7f05ec143470 2024-04-06T12:24:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.773+0000 7f05eb7fe700 1 --2- 172.21.15.189:0/2393307350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ec143470 0x7f05ec0b37f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f05e4009dc0 tx=0x7f05e400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2c560e0ada3a6f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.773+0000 7f05e9ffb700 1 -- 172.21.15.189:0/2393307350 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e400e070 con 0x7f05ec143470 2024-04-06T12:24:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.773+0000 7f05e9ffb700 1 -- 172.21.15.189:0/2393307350 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e4003a20 con 0x7f05ec143470 2024-04-06T12:24:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.773+0000 7f05e9ffb700 1 -- 172.21.15.189:0/2393307350 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d4001980 con 0x7f05ec143470 2024-04-06T12:24:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.773+0000 7f05f1356700 1 -- 172.21.15.189:0/2393307350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ec143470 msgr2=0x7f05ec0b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.773+0000 7f05f1356700 1 --2- 172.21.15.189:0/2393307350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ec143470 0x7f05ec0b37f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f05e4009dc0 tx=0x7f05e400b330 comp rx=0 tx=0).stop 2024-04-06T12:24:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.774+0000 7f05f1356700 1 -- 172.21.15.189:0/2393307350 shutdown_connections 2024-04-06T12:24:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.774+0000 7f05f1356700 1 --2- 172.21.15.189:0/2393307350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ec143470 0x7f05ec0b37f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.774+0000 7f05f1356700 1 -- 172.21.15.189:0/2393307350 >> 172.21.15.189:0/2393307350 conn(0x7f05ec06a730 msgr2=0x7f05ec06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:25.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.774+0000 7f05f1356700 1 -- 172.21.15.189:0/2393307350 shutdown_connections 2024-04-06T12:24:25.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.774+0000 7f05f1356700 1 -- 172.21.15.189:0/2393307350 wait complete. 2024-04-06T12:24:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.775+0000 7f05f1356700 1 Processor -- start 2024-04-06T12:24:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.775+0000 7f05f1356700 1 -- start start 2024-04-06T12:24:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05f1356700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05ec143470 0x7f05ec0b4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05f1356700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ec15db10 0x7f05ec0cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05f1356700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05ec160b60 0x7f05ec1f3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05f1356700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ec0cd650 con 0x7f05ec143470 2024-04-06T12:24:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05f1356700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ec0b4740 con 0x7f05ec160b60 2024-04-06T12:24:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05f1356700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ec0cca40 con 0x7f05ec15db10 2024-04-06T12:24:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05eaffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05ec143470 0x7f05ec0b4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05eaffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05ec143470 0x7f05ec0b4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33950/0 (socket says 172.21.15.189:33950) 2024-04-06T12:24:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05eaffd700 1 -- 172.21.15.189:0/2236980437 learned_addr learned my addr 172.21.15.189:0/2236980437 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05eb7fe700 1 --2- 172.21.15.189:0/2236980437 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05ec160b60 0x7f05ec1f3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05ea7fc700 1 --2- 172.21.15.189:0/2236980437 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ec15db10 0x7f05ec0cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05eaffd700 1 -- 172.21.15.189:0/2236980437 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ec15db10 msgr2=0x7f05ec0cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05eaffd700 1 --2- 172.21.15.189:0/2236980437 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ec15db10 0x7f05ec0cdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05eaffd700 1 -- 172.21.15.189:0/2236980437 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05ec160b60 msgr2=0x7f05ec1f3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05eaffd700 1 --2- 172.21.15.189:0/2236980437 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05ec160b60 0x7f05ec1f3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05eaffd700 1 -- 172.21.15.189:0/2236980437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e4009a70 con 0x7f05ec143470 2024-04-06T12:24:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05eb7fe700 1 --2- 172.21.15.189:0/2236980437 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05ec160b60 0x7f05ec1f3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:24:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05eaffd700 1 --2- 172.21.15.189:0/2236980437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05ec143470 0x7f05ec0b4200 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00c500 tx=0x7f05dc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05e37fe700 1 -- 172.21.15.189:0/2236980437 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc0052d0 con 0x7f05ec143470 2024-04-06T12:24:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05e37fe700 1 -- 172.21.15.189:0/2236980437 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05dc005450 con 0x7f05ec143470 2024-04-06T12:24:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05e37fe700 1 -- 172.21.15.189:0/2236980437 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc0038c0 con 0x7f05ec143470 2024-04-06T12:24:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05f1356700 1 -- 172.21.15.189:0/2236980437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ec0ccd20 con 0x7f05ec143470 2024-04-06T12:24:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.776+0000 7f05f1356700 1 -- 172.21.15.189:0/2236980437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05ec0cd2e0 con 0x7f05ec143470 2024-04-06T12:24:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.777+0000 7f05f1356700 1 -- 172.21.15.189:0/2236980437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05ec0cd4a0 con 0x7f05ec143470 2024-04-06T12:24:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.778+0000 7f05e37fe700 1 -- 172.21.15.189:0/2236980437 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f05dc003a20 con 0x7f05ec143470 2024-04-06T12:24:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.778+0000 7f05e37fe700 1 --2- 172.21.15.189:0/2236980437 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f05cc06ae90 0x7f05cc06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:25.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.778+0000 7f05e37fe700 1 -- 172.21.15.189:0/2236980437 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f05dc088df0 con 0x7f05ec143470 2024-04-06T12:24:25.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.778+0000 7f05ea7fc700 1 --2- 172.21.15.189:0/2236980437 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f05cc06ae90 0x7f05cc06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:25.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.779+0000 7f05ea7fc700 1 --2- 172.21.15.189:0/2236980437 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f05cc06ae90 0x7f05cc06d350 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f05d4001eb0 tx=0x7f05d4009040 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:25.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:25.782+0000 7f05e37fe700 1 -- 172.21.15.189:0/2236980437 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f05dc053ae0 con 0x7f05ec143470 2024-04-06T12:24:25.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.002+0000 7f05f1356700 1 -- 172.21.15.189:0/2236980437 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f05ec15e860 con 0x7f05cc06ae90 2024-04-06T12:24:26.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.009+0000 7f05e37fe700 1 -- 172.21.15.189:0/2236980437 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f05d4001940 con 0x7f05cc06ae90 2024-04-06T12:24:26.007 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:26.007 INFO:teuthology.orchestra.run.smithi189.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:37", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:26.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.012+0000 7f05f1356700 1 -- 172.21.15.189:0/2236980437 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f05cc06ae90 msgr2=0x7f05cc06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:26.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.012+0000 7f05f1356700 1 --2- 172.21.15.189:0/2236980437 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f05cc06ae90 0x7f05cc06d350 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f05d4001eb0 tx=0x7f05d4009040 comp rx=0 tx=0).stop 2024-04-06T12:24:26.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.012+0000 7f05f1356700 1 -- 172.21.15.189:0/2236980437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05ec143470 msgr2=0x7f05ec0b4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:26.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.012+0000 7f05f1356700 1 --2- 172.21.15.189:0/2236980437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05ec143470 0x7f05ec0b4200 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00c500 tx=0x7f05dc00c8c0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.012+0000 7f05f1356700 1 -- 172.21.15.189:0/2236980437 shutdown_connections 2024-04-06T12:24:26.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.012+0000 7f05f1356700 1 --2- 172.21.15.189:0/2236980437 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f05cc06ae90 0x7f05cc06d350 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.012+0000 7f05f1356700 1 --2- 172.21.15.189:0/2236980437 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05ec160b60 0x7f05ec1f3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.012+0000 7f05f1356700 1 --2- 172.21.15.189:0/2236980437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05ec143470 0x7f05ec0b4200 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.012+0000 7f05f1356700 1 --2- 172.21.15.189:0/2236980437 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ec15db10 0x7f05ec0cdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.012+0000 7f05f1356700 1 -- 172.21.15.189:0/2236980437 >> 172.21.15.189:0/2236980437 conn(0x7f05ec06a730 msgr2=0x7f05ec06d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:26.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.012+0000 7f05f1356700 1 -- 172.21.15.189:0/2236980437 shutdown_connections 2024-04-06T12:24:26.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.012+0000 7f05f1356700 1 -- 172.21.15.189:0/2236980437 wait complete. 2024-04-06T12:24:26.021 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:37", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:26.021 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1m', 'retention': {}, 'start': '2024-04-06T00:00:00', 'created': '2024-04-06T12:13:37', 'first': '2024-04-06T12:14:00', 'last': '2024-04-06T12:24:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-06T12:24:26.021 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-06T12:24:26.021 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-06T12:24:26.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:26.022 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2024-04-06T12:24:26.039 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:26.039 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:26.040 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:26.040 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:26.040 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:26.040 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_01_UTC 2024-04-06T12:24:26.040 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:26.040 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:26.040 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:26.040 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:26.040 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:26.040 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-06-12_14_00_UTC', 'scheduled-2024-04-06-12_15_00_UTC', 'scheduled-2024-04-06-12_16_00_UTC', 'scheduled-2024-04-06-12_17_00_UTC', 'scheduled-2024-04-06-12_18_00_UTC', 'scheduled-2024-04-06-12_19_01_UTC', 'scheduled-2024-04-06-12_20_00_UTC', 'scheduled-2024-04-06-12_21_00_UTC', 'scheduled-2024-04-06-12_22_00_UTC', 'scheduled-2024-04-06-12_23_00_UTC', 'scheduled-2024-04-06-12_24_00_UTC'] 2024-04-06T12:24:26.041 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir4 2024-04-06T12:24:26.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.207+0000 7f40ba9f6700 1 Processor -- start 2024-04-06T12:24:26.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.207+0000 7f40ba9f6700 1 -- start start 2024-04-06T12:24:26.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.207+0000 7f40ba9f6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f40b406ae40 0x7f40b415ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:26.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40ba9f6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b415c740 con 0x7f40b4144f40 2024-04-06T12:24:26.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40ba9f6700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b415f1f0 con 0x7f40b415fcd0 2024-04-06T12:24:26.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40ba9f6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b40cfcb0 con 0x7f40b406ae40 2024-04-06T12:24:26.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b3fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f40b4144f40 0x7f40b406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48752/0 (socket says 172.21.15.189:48752) 2024-04-06T12:24:26.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b3fff700 1 -- 172.21.15.189:0/512860502 learned_addr learned my addr 172.21.15.189:0/512860502 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:26.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b37fe700 1 --2- 172.21.15.189:0/512860502 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f40b406ae40 0x7f40b415ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:26.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b2ffd700 1 -- 172.21.15.189:0/512860502 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093701833 0 0) 0x7f40b415c740 con 0x7f40b4144f40 2024-04-06T12:24:26.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b2ffd700 1 -- 172.21.15.189:0/512860502 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f409c0035b0 con 0x7f40b4144f40 2024-04-06T12:24:26.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b2ffd700 1 -- 172.21.15.189:0/512860502 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4189172275 0 0) 0x7f409c0035b0 con 0x7f40b4144f40 2024-04-06T12:24:26.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b2ffd700 1 -- 172.21.15.189:0/512860502 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f40b406ae40 msgr2=0x7f40b415ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:26.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b2ffd700 1 --2- 172.21.15.189:0/512860502 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f40b406ae40 0x7f40b415ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b2ffd700 1 -- 172.21.15.189:0/512860502 >> v1:172.21.15.192:6789/0 conn(0x7f40b415fcd0 legacy=0x7f40b415bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:26.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b2ffd700 1 -- 172.21.15.189:0/512860502 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b415f4d0 con 0x7f40b4144f40 2024-04-06T12:24:26.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b2ffd700 1 -- 172.21.15.189:0/512860502 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f40a8001670 con 0x7f40b4144f40 2024-04-06T12:24:26.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b2ffd700 1 -- 172.21.15.189:0/512860502 >> v1:172.21.15.189:6789/0 conn(0x7f40b4144f40 legacy=0x7f40b406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:26.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b2ffd700 1 --2- 172.21.15.189:0/512860502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f409c003a30 0x7f409c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:26.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b2ffd700 1 --2- 172.21.15.189:0/512860502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f409c004850 0x7f409c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:26.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b2ffd700 1 --2- 172.21.15.189:0/512860502 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f40b406ae40 0x7f409c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:26.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b2ffd700 1 -- 172.21.15.189:0/512860502 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b40cfcb0 con 0x7f409c004850 2024-04-06T12:24:26.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b2ffd700 1 -- 172.21.15.189:0/512860502 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c00bb40 con 0x7f409c003a30 2024-04-06T12:24:26.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b2ffd700 1 -- 172.21.15.189:0/512860502 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f409c00bd00 con 0x7f40b406ae40 2024-04-06T12:24:26.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b37fe700 1 --2- 172.21.15.189:0/512860502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f409c004850 0x7f409c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:26.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.208+0000 7f40b3fff700 1 --2- 172.21.15.189:0/512860502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f409c003a30 0x7f409c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:26.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.209+0000 7f40b8f93700 1 --2- 172.21.15.189:0/512860502 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f40b406ae40 0x7f409c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:26.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.209+0000 7f40b37fe700 1 -- 172.21.15.189:0/512860502 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f40b406ae40 msgr2=0x7f409c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:26.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.209+0000 7f40b37fe700 1 --2- 172.21.15.189:0/512860502 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f40b406ae40 0x7f409c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.209+0000 7f40b37fe700 1 -- 172.21.15.189:0/512860502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f409c003a30 msgr2=0x7f409c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:26.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.209+0000 7f40b37fe700 1 --2- 172.21.15.189:0/512860502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f409c003a30 0x7f409c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.209+0000 7f40b37fe700 1 -- 172.21.15.189:0/512860502 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409c0035b0 con 0x7f409c004850 2024-04-06T12:24:26.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.209+0000 7f40b37fe700 1 --2- 172.21.15.189:0/512860502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f409c004850 0x7f409c004cd0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f40a4009b30 tx=0x7f409c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc9eae5bb91a495 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:26.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.209+0000 7f40b2ffd700 1 -- 172.21.15.189:0/512860502 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a400e6c0 con 0x7f409c004850 2024-04-06T12:24:26.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.209+0000 7f40b2ffd700 1 -- 172.21.15.189:0/512860502 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a400e820 con 0x7f409c004850 2024-04-06T12:24:26.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.209+0000 7f40b2ffd700 1 -- 172.21.15.189:0/512860502 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a400eb90 con 0x7f409c004850 2024-04-06T12:24:26.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.209+0000 7f40ba9f6700 1 -- 172.21.15.189:0/512860502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f409c004850 msgr2=0x7f409c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:26.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.209+0000 7f40ba9f6700 1 --2- 172.21.15.189:0/512860502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f409c004850 0x7f409c004cd0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f40a4009b30 tx=0x7f409c005880 comp rx=0 tx=0).stop 2024-04-06T12:24:26.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.209+0000 7f40b8f93700 1 -- 172.21.15.189:0/512860502 reap_dead start 2024-04-06T12:24:26.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.209+0000 7f40ba9f6700 1 -- 172.21.15.189:0/512860502 shutdown_connections 2024-04-06T12:24:26.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.209+0000 7f40ba9f6700 1 -- 172.21.15.189:0/512860502 >> 172.21.15.189:0/512860502 conn(0x7f40b40d00b0 msgr2=0x7f40b40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:26.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.210+0000 7f40ba9f6700 1 -- 172.21.15.189:0/512860502 shutdown_connections 2024-04-06T12:24:26.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.210+0000 7f40ba9f6700 1 -- 172.21.15.189:0/512860502 wait complete. 2024-04-06T12:24:26.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.211+0000 7f40ba9f6700 1 Processor -- start 2024-04-06T12:24:26.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.211+0000 7f40ba9f6700 1 -- start start 2024-04-06T12:24:26.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.212+0000 7f40ba9f6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f409c004850 0x7f40b41300f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:26.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.212+0000 7f40ba9f6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40b4130630 0x7f40b40717b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:26.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.212+0000 7f40ba9f6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40b4071cf0 0x7f40b40cb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:26.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.212+0000 7f40ba9f6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b40721a0 con 0x7f40b4130630 2024-04-06T12:24:26.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.212+0000 7f40ba9f6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b40cc060 con 0x7f40b4071cf0 2024-04-06T12:24:26.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.212+0000 7f40ba9f6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b4130b50 con 0x7f409c004850 2024-04-06T12:24:26.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.212+0000 7f40b37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40b4130630 0x7f40b40717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:26.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.212+0000 7f40b37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40b4130630 0x7f40b40717b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33956/0 (socket says 172.21.15.189:33956) 2024-04-06T12:24:26.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.212+0000 7f40b37fe700 1 -- 172.21.15.189:0/641589219 learned_addr learned my addr 172.21.15.189:0/641589219 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:26.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.212+0000 7f40b8f93700 1 --2- 172.21.15.189:0/641589219 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40b4071cf0 0x7f40b40cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:26.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.212+0000 7f40b3fff700 1 --2- 172.21.15.189:0/641589219 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f409c004850 0x7f40b41300f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:26.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.213+0000 7f40b37fe700 1 -- 172.21.15.189:0/641589219 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f409c004850 msgr2=0x7f40b41300f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:26.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.213+0000 7f40b37fe700 1 --2- 172.21.15.189:0/641589219 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f409c004850 0x7f40b41300f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.213+0000 7f40b37fe700 1 -- 172.21.15.189:0/641589219 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40b4071cf0 msgr2=0x7f40b40cb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:26.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.213+0000 7f40b37fe700 1 --2- 172.21.15.189:0/641589219 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40b4071cf0 0x7f40b40cb250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.213+0000 7f40b37fe700 1 -- 172.21.15.189:0/641589219 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b4130e30 con 0x7f40b4130630 2024-04-06T12:24:26.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.213+0000 7f40b37fe700 1 --2- 172.21.15.189:0/641589219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40b4130630 0x7f40b40717b0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f40a4000c00 tx=0x7f40a40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:26.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.213+0000 7f40b0ff9700 1 -- 172.21.15.189:0/641589219 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a400e6c0 con 0x7f40b4130630 2024-04-06T12:24:26.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.213+0000 7f40b0ff9700 1 -- 172.21.15.189:0/641589219 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a4013e00 con 0x7f40b4130630 2024-04-06T12:24:26.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.213+0000 7f40ba9f6700 1 -- 172.21.15.189:0/641589219 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b4131110 con 0x7f40b4130630 2024-04-06T12:24:26.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.213+0000 7f40b0ff9700 1 -- 172.21.15.189:0/641589219 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a4028690 con 0x7f40b4130630 2024-04-06T12:24:26.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.213+0000 7f40ba9f6700 1 -- 172.21.15.189:0/641589219 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b4131810 con 0x7f40b4130630 2024-04-06T12:24:26.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.214+0000 7f40ba9f6700 1 -- 172.21.15.189:0/641589219 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b4067bb0 con 0x7f40b4130630 2024-04-06T12:24:26.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.214+0000 7f40b0ff9700 1 -- 172.21.15.189:0/641589219 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f40a401c450 con 0x7f40b4130630 2024-04-06T12:24:26.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.215+0000 7f40b0ff9700 1 --2- 172.21.15.189:0/641589219 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f40a006ae90 0x7f40a006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:26.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.215+0000 7f40b0ff9700 1 -- 172.21.15.189:0/641589219 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f40a408b390 con 0x7f40b4130630 2024-04-06T12:24:26.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.215+0000 7f40b3fff700 1 --2- 172.21.15.189:0/641589219 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f40a006ae90 0x7f40a006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:26.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.215+0000 7f40b3fff700 1 --2- 172.21.15.189:0/641589219 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f40a006ae90 0x7f40a006d350 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f40a8007c30 tx=0x7f40a8009040 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:26.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.219+0000 7f40b0ff9700 1 -- 172.21.15.189:0/641589219 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f40a40562f0 con 0x7f40b4130630 2024-04-06T12:24:26.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.438+0000 7f40ba9f6700 1 -- 172.21.15.189:0/641589219 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f40b40671d0 con 0x7f40a006ae90 2024-04-06T12:24:26.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.445+0000 7f40b0ff9700 1 -- 172.21.15.189:0/641589219 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f40b40671d0 con 0x7f40a006ae90 2024-04-06T12:24:26.442 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:26.442 INFO:teuthology.orchestra.run.smithi189.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:37", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:26.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.447+0000 7f40ba9f6700 1 -- 172.21.15.189:0/641589219 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f40a006ae90 msgr2=0x7f40a006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:26.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.447+0000 7f40ba9f6700 1 --2- 172.21.15.189:0/641589219 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f40a006ae90 0x7f40a006d350 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f40a8007c30 tx=0x7f40a8009040 comp rx=0 tx=0).stop 2024-04-06T12:24:26.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.447+0000 7f40ba9f6700 1 -- 172.21.15.189:0/641589219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40b4130630 msgr2=0x7f40b40717b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:26.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.447+0000 7f40ba9f6700 1 --2- 172.21.15.189:0/641589219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40b4130630 0x7f40b40717b0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f40a4000c00 tx=0x7f40a40047c0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.447+0000 7f40ba9f6700 1 -- 172.21.15.189:0/641589219 shutdown_connections 2024-04-06T12:24:26.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.447+0000 7f40ba9f6700 1 --2- 172.21.15.189:0/641589219 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f40a006ae90 0x7f40a006d350 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.447+0000 7f40ba9f6700 1 --2- 172.21.15.189:0/641589219 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40b4071cf0 0x7f40b40cb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.447+0000 7f40ba9f6700 1 --2- 172.21.15.189:0/641589219 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f409c004850 0x7f40b41300f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.447+0000 7f40ba9f6700 1 --2- 172.21.15.189:0/641589219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40b4130630 0x7f40b40717b0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.447+0000 7f40ba9f6700 1 -- 172.21.15.189:0/641589219 >> 172.21.15.189:0/641589219 conn(0x7f40b40d00b0 msgr2=0x7f40b41401d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:26.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.447+0000 7f40ba9f6700 1 -- 172.21.15.189:0/641589219 shutdown_connections 2024-04-06T12:24:26.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.447+0000 7f40ba9f6700 1 -- 172.21.15.189:0/641589219 wait complete. 2024-04-06T12:24:26.457 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:37", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:26.457 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1m', 'retention': {}, 'start': '2024-04-06T00:00:00', 'created': '2024-04-06T12:13:37', 'first': '2024-04-06T12:14:00', 'last': '2024-04-06T12:24:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-06T12:24:26.457 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-06T12:24:26.457 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-06T12:24:26.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:26.457 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2024-04-06T12:24:26.475 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:26.475 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:26.476 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:26.476 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:26.476 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:26.476 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:26.476 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:26.476 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:26.476 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:26.476 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:26.476 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:26.476 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-06-12_14_00_UTC', 'scheduled-2024-04-06-12_15_00_UTC', 'scheduled-2024-04-06-12_16_00_UTC', 'scheduled-2024-04-06-12_17_00_UTC', 'scheduled-2024-04-06-12_18_00_UTC', 'scheduled-2024-04-06-12_19_00_UTC', 'scheduled-2024-04-06-12_20_00_UTC', 'scheduled-2024-04-06-12_21_00_UTC', 'scheduled-2024-04-06-12_22_00_UTC', 'scheduled-2024-04-06-12_23_00_UTC', 'scheduled-2024-04-06-12_24_00_UTC'] 2024-04-06T12:24:26.476 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir5 2024-04-06T12:24:26.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.643+0000 7f50c0beb700 1 Processor -- start 2024-04-06T12:24:26.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.643+0000 7f50c0beb700 1 -- start start 2024-04-06T12:24:26.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.644+0000 7f50c0beb700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f50bc15a550 0x7f50bc0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:26.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.644+0000 7f50c0beb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50bc0cdbe0 con 0x7f50bc0b4d70 2024-04-06T12:24:26.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.644+0000 7f50c0beb700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50bc15a940 con 0x7f50bc0cf900 2024-04-06T12:24:26.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.644+0000 7f50c0beb700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50bc15ac00 con 0x7f50bc15a550 2024-04-06T12:24:26.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.644+0000 7f50bad9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f50bc0b4d70 0x7f50bc0b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48760/0 (socket says 172.21.15.189:48760) 2024-04-06T12:24:26.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.644+0000 7f50bad9d700 1 -- 172.21.15.189:0/2345789456 learned_addr learned my addr 172.21.15.189:0/2345789456 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:26.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50ba59c700 1 --2- 172.21.15.189:0/2345789456 >> v2:172.21.15.192:3300/0 conn(0x7f50bc15a550 0x7f50bc0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:26.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50b959a700 1 -- 172.21.15.189:0/2345789456 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042915220 0 0) 0x7f50bc0cdbe0 con 0x7f50bc0b4d70 2024-04-06T12:24:26.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50b959a700 1 -- 172.21.15.189:0/2345789456 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50a40057c0 con 0x7f50bc0b4d70 2024-04-06T12:24:26.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50b959a700 1 -- 172.21.15.189:0/2345789456 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525796668 0 0) 0x7f50bc15a940 con 0x7f50bc0cf900 2024-04-06T12:24:26.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50b959a700 1 -- 172.21.15.189:0/2345789456 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50bc0cdbe0 con 0x7f50bc0cf900 2024-04-06T12:24:26.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50b959a700 1 -- 172.21.15.189:0/2345789456 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3671837908 0 0) 0x7f50a40057c0 con 0x7f50bc0b4d70 2024-04-06T12:24:26.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50b959a700 1 -- 172.21.15.189:0/2345789456 >> v2:172.21.15.192:3300/0 conn(0x7f50bc15a550 msgr2=0x7f50bc0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:26.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50b959a700 1 --2- 172.21.15.189:0/2345789456 >> v2:172.21.15.192:3300/0 conn(0x7f50bc15a550 0x7f50bc0cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50b959a700 1 -- 172.21.15.189:0/2345789456 >> v1:172.21.15.192:6789/0 conn(0x7f50bc0cf900 legacy=0x7f50bc0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:26.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50b959a700 1 -- 172.21.15.189:0/2345789456 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50bc15aee0 con 0x7f50bc0b4d70 2024-04-06T12:24:26.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50b959a700 1 -- 172.21.15.189:0/2345789456 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f50b40019f0 con 0x7f50bc0b4d70 2024-04-06T12:24:26.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50b959a700 1 -- 172.21.15.189:0/2345789456 >> v1:172.21.15.189:6789/0 conn(0x7f50bc0b4d70 legacy=0x7f50bc0b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:26.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50b959a700 1 --2- 172.21.15.189:0/2345789456 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50a4005da0 0x7f50a40061a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:26.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50b959a700 1 --2- 172.21.15.189:0/2345789456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50a4006a60 0x7f50a4006ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:26.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50b959a700 1 --2- 172.21.15.189:0/2345789456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50a4007420 0x7f50a400dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:26.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50b959a700 1 -- 172.21.15.189:0/2345789456 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc15ac00 con 0x7f50a4007420 2024-04-06T12:24:26.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50b959a700 1 -- 172.21.15.189:0/2345789456 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a400e2a0 con 0x7f50a4006a60 2024-04-06T12:24:26.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50b959a700 1 -- 172.21.15.189:0/2345789456 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a400e460 con 0x7f50a4005da0 2024-04-06T12:24:26.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50bad9d700 1 --2- 172.21.15.189:0/2345789456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50a4007420 0x7f50a400dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:26.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50ba59c700 1 --2- 172.21.15.189:0/2345789456 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50a4005da0 0x7f50a40061a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:26.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50b9d9b700 1 --2- 172.21.15.189:0/2345789456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50a4006a60 0x7f50a4006ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:26.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50bad9d700 1 -- 172.21.15.189:0/2345789456 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50a4005da0 msgr2=0x7f50a40061a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:26.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50bad9d700 1 --2- 172.21.15.189:0/2345789456 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50a4005da0 0x7f50a40061a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50bad9d700 1 -- 172.21.15.189:0/2345789456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50a4006a60 msgr2=0x7f50a4006ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:26.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50bad9d700 1 --2- 172.21.15.189:0/2345789456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50a4006a60 0x7f50a4006ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.645+0000 7f50bad9d700 1 -- 172.21.15.189:0/2345789456 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a40057c0 con 0x7f50a4007420 2024-04-06T12:24:26.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.646+0000 7f50bad9d700 1 -- 172.21.15.189:0/2345789456 reap_dead start 2024-04-06T12:24:26.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.646+0000 7f50bad9d700 1 --2- 172.21.15.189:0/2345789456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50a4007420 0x7f50a400dc20 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f50b400a7a0 tx=0x7f50a4007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=990b492d03b754f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:26.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.646+0000 7f50b959a700 1 -- 172.21.15.189:0/2345789456 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b4012600 con 0x7f50a4007420 2024-04-06T12:24:26.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.646+0000 7f50b959a700 1 -- 172.21.15.189:0/2345789456 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b4012780 con 0x7f50a4007420 2024-04-06T12:24:26.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.646+0000 7f50b959a700 1 -- 172.21.15.189:0/2345789456 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b4012b30 con 0x7f50a4007420 2024-04-06T12:24:26.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.646+0000 7f50c0beb700 1 -- 172.21.15.189:0/2345789456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50a4007420 msgr2=0x7f50a400dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:26.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.646+0000 7f50c0beb700 1 --2- 172.21.15.189:0/2345789456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50a4007420 0x7f50a400dc20 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f50b400a7a0 tx=0x7f50a4007a90 comp rx=0 tx=0).stop 2024-04-06T12:24:26.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.646+0000 7f50c0beb700 1 -- 172.21.15.189:0/2345789456 shutdown_connections 2024-04-06T12:24:26.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.646+0000 7f50c0beb700 1 --2- 172.21.15.189:0/2345789456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50a4007420 0x7f50a400dc20 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.646+0000 7f50c0beb700 1 -- 172.21.15.189:0/2345789456 >> 172.21.15.189:0/2345789456 conn(0x7f50bc0d00b0 msgr2=0x7f50bc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:26.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.646+0000 7f50c0beb700 1 -- 172.21.15.189:0/2345789456 shutdown_connections 2024-04-06T12:24:26.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.646+0000 7f50c0beb700 1 -- 172.21.15.189:0/2345789456 wait complete. 2024-04-06T12:24:26.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.648+0000 7f50c0beb700 1 Processor -- start 2024-04-06T12:24:26.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.648+0000 7f50c0beb700 1 -- start start 2024-04-06T12:24:26.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.648+0000 7f50c0beb700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50a4007420 0x7f50bc152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:26.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.648+0000 7f50c0beb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50bc158eb0 0x7f50bc0a8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:26.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.648+0000 7f50c0beb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50bc1eebc0 0x7f50bc1ef050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:26.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.648+0000 7f50c0beb700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc159330 con 0x7f50bc158eb0 2024-04-06T12:24:26.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.648+0000 7f50c0beb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc157360 con 0x7f50bc1eebc0 2024-04-06T12:24:26.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.648+0000 7f50c0beb700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc1583d0 con 0x7f50a4007420 2024-04-06T12:24:26.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.649+0000 7f50b9d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50bc158eb0 0x7f50bc0a8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:26.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.649+0000 7f50b9d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50bc158eb0 0x7f50bc0a8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33966/0 (socket says 172.21.15.189:33966) 2024-04-06T12:24:26.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.649+0000 7f50b9d9b700 1 -- 172.21.15.189:0/913568690 learned_addr learned my addr 172.21.15.189:0/913568690 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:26.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.649+0000 7f50bad9d700 1 --2- 172.21.15.189:0/913568690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50bc1eebc0 0x7f50bc1ef050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:26.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.649+0000 7f50ba59c700 1 --2- 172.21.15.189:0/913568690 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50a4007420 0x7f50bc152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:26.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.649+0000 7f50b9d9b700 1 -- 172.21.15.189:0/913568690 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50a4007420 msgr2=0x7f50bc152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:26.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.649+0000 7f50b9d9b700 1 --2- 172.21.15.189:0/913568690 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50a4007420 0x7f50bc152a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.649+0000 7f50b9d9b700 1 -- 172.21.15.189:0/913568690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50bc1eebc0 msgr2=0x7f50bc1ef050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:26.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.649+0000 7f50b9d9b700 1 --2- 172.21.15.189:0/913568690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50bc1eebc0 0x7f50bc1ef050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.649+0000 7f50b9d9b700 1 -- 172.21.15.189:0/913568690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50bc1586b0 con 0x7f50bc158eb0 2024-04-06T12:24:26.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.649+0000 7f50b9d9b700 1 --2- 172.21.15.189:0/913568690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50bc158eb0 0x7f50bc0a8e70 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f50ac00f770 tx=0x7f50ac00fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:26.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.649+0000 7f50aaffd700 1 -- 172.21.15.189:0/913568690 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac005320 con 0x7f50bc158eb0 2024-04-06T12:24:26.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.649+0000 7f50aaffd700 1 -- 172.21.15.189:0/913568690 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50ac0054a0 con 0x7f50bc158eb0 2024-04-06T12:24:26.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.649+0000 7f50aaffd700 1 -- 172.21.15.189:0/913568690 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac006aa0 con 0x7f50bc158eb0 2024-04-06T12:24:26.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.649+0000 7f50c0beb700 1 -- 172.21.15.189:0/913568690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50bc158990 con 0x7f50bc158eb0 2024-04-06T12:24:26.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.649+0000 7f50c0beb700 1 -- 172.21.15.189:0/913568690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50bc157010 con 0x7f50bc158eb0 2024-04-06T12:24:26.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.650+0000 7f50c0beb700 1 -- 172.21.15.189:0/913568690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50bc158bd0 con 0x7f50bc158eb0 2024-04-06T12:24:26.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.651+0000 7f50aaffd700 1 -- 172.21.15.189:0/913568690 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f50ac006c00 con 0x7f50bc158eb0 2024-04-06T12:24:26.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.651+0000 7f50aaffd700 1 --2- 172.21.15.189:0/913568690 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f509c06ae90 0x7f509c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:26.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.651+0000 7f50aaffd700 1 -- 172.21.15.189:0/913568690 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f50ac088e60 con 0x7f50bc158eb0 2024-04-06T12:24:26.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.651+0000 7f50ba59c700 1 --2- 172.21.15.189:0/913568690 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f509c06ae90 0x7f509c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:26.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.651+0000 7f50ba59c700 1 --2- 172.21.15.189:0/913568690 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f509c06ae90 0x7f509c06d350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009a20 tx=0x7f50b0008040 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:26.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.655+0000 7f50aaffd700 1 -- 172.21.15.189:0/913568690 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f50bc158bd0 con 0x7f50bc158eb0 2024-04-06T12:24:26.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.877+0000 7f50c0beb700 1 -- 172.21.15.189:0/913568690 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f50bc06fdd0 con 0x7f509c06ae90 2024-04-06T12:24:26.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.884+0000 7f50aaffd700 1 -- 172.21.15.189:0/913568690 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f50b0009e90 con 0x7f509c06ae90 2024-04-06T12:24:26.882 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:26.882 INFO:teuthology.orchestra.run.smithi189.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:38", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:26.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.887+0000 7f50c0beb700 1 -- 172.21.15.189:0/913568690 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f509c06ae90 msgr2=0x7f509c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:26.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.887+0000 7f50c0beb700 1 --2- 172.21.15.189:0/913568690 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f509c06ae90 0x7f509c06d350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009a20 tx=0x7f50b0008040 comp rx=0 tx=0).stop 2024-04-06T12:24:26.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.887+0000 7f50c0beb700 1 -- 172.21.15.189:0/913568690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50bc158eb0 msgr2=0x7f50bc0a8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:26.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.887+0000 7f50c0beb700 1 --2- 172.21.15.189:0/913568690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50bc158eb0 0x7f50bc0a8e70 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f50ac00f770 tx=0x7f50ac00fb30 comp rx=0 tx=0).stop 2024-04-06T12:24:26.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.887+0000 7f50c0beb700 1 -- 172.21.15.189:0/913568690 shutdown_connections 2024-04-06T12:24:26.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.887+0000 7f50c0beb700 1 --2- 172.21.15.189:0/913568690 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f509c06ae90 0x7f509c06d350 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.887+0000 7f50c0beb700 1 --2- 172.21.15.189:0/913568690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50bc1eebc0 0x7f50bc1ef050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.887+0000 7f50c0beb700 1 --2- 172.21.15.189:0/913568690 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50a4007420 0x7f50bc152a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.887+0000 7f50c0beb700 1 --2- 172.21.15.189:0/913568690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50bc158eb0 0x7f50bc0a8e70 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:26.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.887+0000 7f50c0beb700 1 -- 172.21.15.189:0/913568690 >> 172.21.15.189:0/913568690 conn(0x7f50bc0d00b0 msgr2=0x7f50bc1596a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:26.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.887+0000 7f50c0beb700 1 -- 172.21.15.189:0/913568690 shutdown_connections 2024-04-06T12:24:26.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:26.887+0000 7f50c0beb700 1 -- 172.21.15.189:0/913568690 wait complete. 2024-04-06T12:24:26.896 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:38", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:26.896 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1m', 'retention': {}, 'start': '2024-04-06T00:00:00', 'created': '2024-04-06T12:13:38', 'first': '2024-04-06T12:14:00', 'last': '2024-04-06T12:24:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-06T12:24:26.896 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-06T12:24:26.896 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-06T12:24:26.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:26.896 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2024-04-06T12:24:26.914 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:26.914 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:26.914 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:26.914 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:26.914 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:26.914 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:26.914 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:26.914 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_01_UTC 2024-04-06T12:24:26.915 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:26.915 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:26.915 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:26.915 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-06-12_14_00_UTC', 'scheduled-2024-04-06-12_15_00_UTC', 'scheduled-2024-04-06-12_16_00_UTC', 'scheduled-2024-04-06-12_17_00_UTC', 'scheduled-2024-04-06-12_18_00_UTC', 'scheduled-2024-04-06-12_19_00_UTC', 'scheduled-2024-04-06-12_20_00_UTC', 'scheduled-2024-04-06-12_21_01_UTC', 'scheduled-2024-04-06-12_22_00_UTC', 'scheduled-2024-04-06-12_23_00_UTC', 'scheduled-2024-04-06-12_24_00_UTC'] 2024-04-06T12:24:26.915 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir6 2024-04-06T12:24:27.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.080+0000 7fe2e2f79700 1 Processor -- start 2024-04-06T12:24:27.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.080+0000 7fe2e2f79700 1 -- start start 2024-04-06T12:24:27.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.081+0000 7fe2e2f79700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe2dc15a550 0x7fe2dc0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.081+0000 7fe2e2f79700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe2dc0cf900 0x7fe2dc0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.081+0000 7fe2e2f79700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2dc0cdbe0 con 0x7fe2dc0b4ba0 2024-04-06T12:24:27.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.081+0000 7fe2e2f79700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2dc15a940 con 0x7fe2dc15a550 2024-04-06T12:24:27.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.081+0000 7fe2e2f79700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2dc15ac00 con 0x7fe2dc0cf900 2024-04-06T12:24:27.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.081+0000 7fe2e0d15700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe2dc15a550 0x7fe2dc0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.081+0000 7fe2e0d15700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe2dc15a550 0x7fe2dc0cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33970/0 (socket says 172.21.15.189:33970) 2024-04-06T12:24:27.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.081+0000 7fe2e0d15700 1 -- 172.21.15.189:0/216923072 learned_addr learned my addr 172.21.15.189:0/216923072 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:27.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.081+0000 7fe2dbfff700 1 --2- 172.21.15.189:0/216923072 >> v2:172.21.15.192:3300/0 conn(0x7fe2dc0cf900 0x7fe2dc0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.081+0000 7fe2e0d15700 1 -- 172.21.15.189:0/216923072 >> v2:172.21.15.192:3300/0 conn(0x7fe2dc0cf900 msgr2=0x7fe2dc0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:27.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.081+0000 7fe2e0d15700 1 --2- 172.21.15.189:0/216923072 >> v2:172.21.15.192:3300/0 conn(0x7fe2dc0cf900 0x7fe2dc0b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.081+0000 7fe2e0d15700 1 -- 172.21.15.189:0/216923072 >> v1:172.21.15.192:6789/0 conn(0x7fe2dc0b4ba0 legacy=0x7fe2dc06eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:27.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.081+0000 7fe2e0d15700 1 -- 172.21.15.189:0/216923072 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc15aee0 con 0x7fe2dc15a550 2024-04-06T12:24:27.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.081+0000 7fe2e0d15700 1 --2- 172.21.15.189:0/216923072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc15a550 0x7fe2dc0cf3c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0b2d40 tx=0x7fe2d0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9364730801e8bb0b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:27.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.081+0000 7fe2db7fe700 1 -- 172.21.15.189:0/216923072 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0004230 con 0x7fe2dc15a550 2024-04-06T12:24:27.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.081+0000 7fe2db7fe700 1 -- 172.21.15.189:0/216923072 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d0004390 con 0x7fe2dc15a550 2024-04-06T12:24:27.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.081+0000 7fe2db7fe700 1 -- 172.21.15.189:0/216923072 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d00036a0 con 0x7fe2dc15a550 2024-04-06T12:24:27.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.082+0000 7fe2e2f79700 1 -- 172.21.15.189:0/216923072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc15a550 msgr2=0x7fe2dc0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:27.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.082+0000 7fe2e2f79700 1 --2- 172.21.15.189:0/216923072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc15a550 0x7fe2dc0cf3c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0b2d40 tx=0x7fe2d0009bd0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.082+0000 7fe2e2f79700 1 -- 172.21.15.189:0/216923072 shutdown_connections 2024-04-06T12:24:27.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.082+0000 7fe2e2f79700 1 --2- 172.21.15.189:0/216923072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc15a550 0x7fe2dc0cf3c0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.082+0000 7fe2e2f79700 1 --2- 172.21.15.189:0/216923072 >> v2:172.21.15.192:3300/0 conn(0x7fe2dc0cf900 0x7fe2dc0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.082+0000 7fe2e2f79700 1 -- 172.21.15.189:0/216923072 >> 172.21.15.189:0/216923072 conn(0x7fe2dc0d00b0 msgr2=0x7fe2dc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:27.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.082+0000 7fe2e2f79700 1 -- 172.21.15.189:0/216923072 shutdown_connections 2024-04-06T12:24:27.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.082+0000 7fe2e2f79700 1 -- 172.21.15.189:0/216923072 wait complete. 2024-04-06T12:24:27.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.083+0000 7fe2e2f79700 1 Processor -- start 2024-04-06T12:24:27.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.083+0000 7fe2e2f79700 1 -- start start 2024-04-06T12:24:27.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.084+0000 7fe2e2f79700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc0b4ba0 0x7fe2dc0cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.084+0000 7fe2e2f79700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2dc0cf900 0x7fe2dc140fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.084+0000 7fe2e2f79700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2dc143de0 0x7fe2dc14d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.084+0000 7fe2e2f79700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc0b1aa0 con 0x7fe2dc0b4ba0 2024-04-06T12:24:27.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.084+0000 7fe2e2f79700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc0b17c0 con 0x7fe2dc143de0 2024-04-06T12:24:27.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.084+0000 7fe2e2f79700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc0ae9b0 con 0x7fe2dc0cf900 2024-04-06T12:24:27.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.084+0000 7fe2e0d15700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc0b4ba0 0x7fe2dc0cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.084+0000 7fe2e0d15700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc0b4ba0 0x7fe2dc0cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33974/0 (socket says 172.21.15.189:33974) 2024-04-06T12:24:27.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.084+0000 7fe2e0d15700 1 -- 172.21.15.189:0/152536658 learned_addr learned my addr 172.21.15.189:0/152536658 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:27.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.084+0000 7fe2dbfff700 1 --2- 172.21.15.189:0/152536658 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2dc0cf900 0x7fe2dc140fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.084+0000 7fe2e0d15700 1 -- 172.21.15.189:0/152536658 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2dc0cf900 msgr2=0x7fe2dc140fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:27.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.084+0000 7fe2e0d15700 1 --2- 172.21.15.189:0/152536658 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2dc0cf900 0x7fe2dc140fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.084+0000 7fe2e1516700 1 --2- 172.21.15.189:0/152536658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2dc143de0 0x7fe2dc14d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.084+0000 7fe2e0d15700 1 -- 172.21.15.189:0/152536658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2dc143de0 msgr2=0x7fe2dc14d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:27.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.084+0000 7fe2e0d15700 1 --2- 172.21.15.189:0/152536658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2dc143de0 0x7fe2dc14d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.084+0000 7fe2e0d15700 1 -- 172.21.15.189:0/152536658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d0009550 con 0x7fe2dc0b4ba0 2024-04-06T12:24:27.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.084+0000 7fe2e0d15700 1 --2- 172.21.15.189:0/152536658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc0b4ba0 0x7fe2dc0cc4b0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc06c820 tx=0x7fe2d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:27.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.085+0000 7fe2d97fa700 1 -- 172.21.15.189:0/152536658 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0011630 con 0x7fe2dc0b4ba0 2024-04-06T12:24:27.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.085+0000 7fe2d97fa700 1 -- 172.21.15.189:0/152536658 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d0011790 con 0x7fe2dc0b4ba0 2024-04-06T12:24:27.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.085+0000 7fe2e2f79700 1 -- 172.21.15.189:0/152536658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2dc0aec90 con 0x7fe2dc0b4ba0 2024-04-06T12:24:27.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.085+0000 7fe2d97fa700 1 -- 172.21.15.189:0/152536658 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0011b40 con 0x7fe2dc0b4ba0 2024-04-06T12:24:27.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.085+0000 7fe2e2f79700 1 -- 172.21.15.189:0/152536658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2dc1414e0 con 0x7fe2dc0b4ba0 2024-04-06T12:24:27.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.086+0000 7fe2e2f79700 1 -- 172.21.15.189:0/152536658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2dc0af1b0 con 0x7fe2dc0b4ba0 2024-04-06T12:24:27.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.086+0000 7fe2d97fa700 1 -- 172.21.15.189:0/152536658 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fe2d0011ca0 con 0x7fe2dc0b4ba0 2024-04-06T12:24:27.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.086+0000 7fe2d97fa700 1 --2- 172.21.15.189:0/152536658 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2c006ae90 0x7fe2c006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.087+0000 7fe2d97fa700 1 -- 172.21.15.189:0/152536658 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fe2d008afd0 con 0x7fe2dc0b4ba0 2024-04-06T12:24:27.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.087+0000 7fe2dbfff700 1 --2- 172.21.15.189:0/152536658 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2c006ae90 0x7fe2c006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.087+0000 7fe2dbfff700 1 --2- 172.21.15.189:0/152536658 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2c006ae90 0x7fe2c006d350 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc1f6110 tx=0x7fe2c8008040 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:27.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.091+0000 7fe2d97fa700 1 -- 172.21.15.189:0/152536658 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe2d0055d60 con 0x7fe2dc0b4ba0 2024-04-06T12:24:27.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.310+0000 7fe2e2f79700 1 -- 172.21.15.189:0/152536658 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe2dc0b4fb0 con 0x7fe2c006ae90 2024-04-06T12:24:27.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.317+0000 7fe2d97fa700 1 -- 172.21.15.189:0/152536658 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fe2c800c040 con 0x7fe2c006ae90 2024-04-06T12:24:27.314 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:27.315 INFO:teuthology.orchestra.run.smithi189.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:38", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:27.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.319+0000 7fe2e2f79700 1 -- 172.21.15.189:0/152536658 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2c006ae90 msgr2=0x7fe2c006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:27.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.319+0000 7fe2e2f79700 1 --2- 172.21.15.189:0/152536658 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2c006ae90 0x7fe2c006d350 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc1f6110 tx=0x7fe2c8008040 comp rx=0 tx=0).stop 2024-04-06T12:24:27.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.319+0000 7fe2e2f79700 1 -- 172.21.15.189:0/152536658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc0b4ba0 msgr2=0x7fe2dc0cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:27.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.319+0000 7fe2e2f79700 1 --2- 172.21.15.189:0/152536658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc0b4ba0 0x7fe2dc0cc4b0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc06c820 tx=0x7fe2d0004750 comp rx=0 tx=0).stop 2024-04-06T12:24:27.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.320+0000 7fe2e2f79700 1 -- 172.21.15.189:0/152536658 shutdown_connections 2024-04-06T12:24:27.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.320+0000 7fe2e2f79700 1 --2- 172.21.15.189:0/152536658 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2c006ae90 0x7fe2c006d350 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.320+0000 7fe2e2f79700 1 --2- 172.21.15.189:0/152536658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2dc143de0 0x7fe2dc14d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.320+0000 7fe2e2f79700 1 --2- 172.21.15.189:0/152536658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc0b4ba0 0x7fe2dc0cc4b0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.320+0000 7fe2e2f79700 1 --2- 172.21.15.189:0/152536658 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2dc0cf900 0x7fe2dc140fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.320+0000 7fe2e2f79700 1 -- 172.21.15.189:0/152536658 >> 172.21.15.189:0/152536658 conn(0x7fe2dc0d00b0 msgr2=0x7fe2dc06f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:27.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.320+0000 7fe2e2f79700 1 -- 172.21.15.189:0/152536658 shutdown_connections 2024-04-06T12:24:27.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.320+0000 7fe2e2f79700 1 -- 172.21.15.189:0/152536658 wait complete. 2024-04-06T12:24:27.329 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:38", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:27.329 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1m', 'retention': {}, 'start': '2024-04-06T00:00:00', 'created': '2024-04-06T12:13:38', 'first': '2024-04-06T12:14:00', 'last': '2024-04-06T12:24:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-06T12:24:27.329 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-06T12:24:27.330 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-06T12:24:27.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:27.330 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2024-04-06T12:24:27.348 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:27.348 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:27.349 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:27.349 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:27.349 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:27.349 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:27.349 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:27.349 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:27.349 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:27.349 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:27.349 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:27.350 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-06-12_14_00_UTC', 'scheduled-2024-04-06-12_15_00_UTC', 'scheduled-2024-04-06-12_16_00_UTC', 'scheduled-2024-04-06-12_17_00_UTC', 'scheduled-2024-04-06-12_18_00_UTC', 'scheduled-2024-04-06-12_19_00_UTC', 'scheduled-2024-04-06-12_20_00_UTC', 'scheduled-2024-04-06-12_21_00_UTC', 'scheduled-2024-04-06-12_22_00_UTC', 'scheduled-2024-04-06-12_23_00_UTC', 'scheduled-2024-04-06-12_24_00_UTC'] 2024-04-06T12:24:27.350 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir7 2024-04-06T12:24:27.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.515+0000 7efd3e345700 1 Processor -- start 2024-04-06T12:24:27.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.515+0000 7efd3e345700 1 -- start start 2024-04-06T12:24:27.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.516+0000 7efd3e345700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efd3815d7b0 0x7efd380b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.516+0000 7efd3e345700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd3806a730 0x7efd3806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.516+0000 7efd3e345700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd380b3990 con 0x7efd3815ddf0 2024-04-06T12:24:27.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.516+0000 7efd3e345700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd3806b0f0 con 0x7efd3815d7b0 2024-04-06T12:24:27.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.516+0000 7efd3e345700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd380b3060 con 0x7efd3806a730 2024-04-06T12:24:27.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.516+0000 7efd3c8e2700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efd3815ddf0 0x7efd3815e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48766/0 (socket says 172.21.15.189:48766) 2024-04-06T12:24:27.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.516+0000 7efd3c8e2700 1 -- 172.21.15.189:0/1589464547 learned_addr learned my addr 172.21.15.189:0/1589464547 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:27.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.516+0000 7efd37fff700 1 --2- 172.21.15.189:0/1589464547 >> v2:172.21.15.192:3300/0 conn(0x7efd3815d7b0 0x7efd380b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.516+0000 7efd377fe700 1 --2- 172.21.15.189:0/1589464547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd3806a730 0x7efd3806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.516+0000 7efd36ffd700 1 -- 172.21.15.189:0/1589464547 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854560500 0 0) 0x7efd380b3990 con 0x7efd3815ddf0 2024-04-06T12:24:27.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.516+0000 7efd36ffd700 1 -- 172.21.15.189:0/1589464547 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd200035b0 con 0x7efd3815ddf0 2024-04-06T12:24:27.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd36ffd700 1 -- 172.21.15.189:0/1589464547 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 903641687 0 0) 0x7efd200035b0 con 0x7efd3815ddf0 2024-04-06T12:24:27.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd36ffd700 1 -- 172.21.15.189:0/1589464547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd3806a730 msgr2=0x7efd3806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:27.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd36ffd700 1 --2- 172.21.15.189:0/1589464547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd3806a730 0x7efd3806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd36ffd700 1 -- 172.21.15.189:0/1589464547 >> v2:172.21.15.192:3300/0 conn(0x7efd3815d7b0 msgr2=0x7efd380b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:27.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd36ffd700 1 --2- 172.21.15.189:0/1589464547 >> v2:172.21.15.192:3300/0 conn(0x7efd3815d7b0 0x7efd380b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd36ffd700 1 -- 172.21.15.189:0/1589464547 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd380b3340 con 0x7efd3815ddf0 2024-04-06T12:24:27.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd36ffd700 1 -- 172.21.15.189:0/1589464547 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7efd300019f0 con 0x7efd3815ddf0 2024-04-06T12:24:27.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd36ffd700 1 -- 172.21.15.189:0/1589464547 >> v1:172.21.15.189:6789/0 conn(0x7efd3815ddf0 legacy=0x7efd3815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:27.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd36ffd700 1 --2- 172.21.15.189:0/1589464547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd20003a00 0x7efd20003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd36ffd700 1 --2- 172.21.15.189:0/1589464547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd3806a730 0x7efd20004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd36ffd700 1 --2- 172.21.15.189:0/1589464547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd20004d90 0x7efd20005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd36ffd700 1 -- 172.21.15.189:0/1589464547 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3806b0f0 con 0x7efd20003a00 2024-04-06T12:24:27.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd36ffd700 1 -- 172.21.15.189:0/1589464547 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd380b3060 con 0x7efd20004d90 2024-04-06T12:24:27.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd36ffd700 1 -- 172.21.15.189:0/1589464547 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd2000ba10 con 0x7efd3806a730 2024-04-06T12:24:27.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd37fff700 1 --2- 172.21.15.189:0/1589464547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd20003a00 0x7efd20003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd3c8e2700 1 --2- 172.21.15.189:0/1589464547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd20004d90 0x7efd20005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd377fe700 1 --2- 172.21.15.189:0/1589464547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd3806a730 0x7efd20004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd37fff700 1 -- 172.21.15.189:0/1589464547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd3806a730 msgr2=0x7efd20004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:27.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd37fff700 1 --2- 172.21.15.189:0/1589464547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd3806a730 0x7efd20004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd37fff700 1 -- 172.21.15.189:0/1589464547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd20004d90 msgr2=0x7efd20005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:27.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd37fff700 1 --2- 172.21.15.189:0/1589464547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd20004d90 0x7efd20005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd37fff700 1 -- 172.21.15.189:0/1589464547 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd200035b0 con 0x7efd20003a00 2024-04-06T12:24:27.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd377fe700 1 --2- 172.21.15.189:0/1589464547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd3806a730 0x7efd20004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:24:27.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.517+0000 7efd37fff700 1 --2- 172.21.15.189:0/1589464547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd20003a00 0x7efd20003e20 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7efd28009b60 tx=0x7efd20005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44372fa4f5568144 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:27.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.519+0000 7efd36ffd700 1 -- 172.21.15.189:0/1589464547 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2800e6c0 con 0x7efd20003a00 2024-04-06T12:24:27.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.519+0000 7efd36ffd700 1 -- 172.21.15.189:0/1589464547 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2800e820 con 0x7efd20003a00 2024-04-06T12:24:27.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.519+0000 7efd36ffd700 1 -- 172.21.15.189:0/1589464547 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2800eb90 con 0x7efd20003a00 2024-04-06T12:24:27.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.519+0000 7efd3e345700 1 -- 172.21.15.189:0/1589464547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd20003a00 msgr2=0x7efd20003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:27.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.519+0000 7efd3e345700 1 --2- 172.21.15.189:0/1589464547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd20003a00 0x7efd20003e20 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7efd28009b60 tx=0x7efd20005880 comp rx=0 tx=0).stop 2024-04-06T12:24:27.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.519+0000 7efd3c8e2700 1 -- 172.21.15.189:0/1589464547 reap_dead start 2024-04-06T12:24:27.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.520+0000 7efd3e345700 1 -- 172.21.15.189:0/1589464547 shutdown_connections 2024-04-06T12:24:27.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.520+0000 7efd3e345700 1 -- 172.21.15.189:0/1589464547 >> 172.21.15.189:0/1589464547 conn(0x7efd380d00b0 msgr2=0x7efd380d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:27.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.520+0000 7efd3e345700 1 -- 172.21.15.189:0/1589464547 shutdown_connections 2024-04-06T12:24:27.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.520+0000 7efd3e345700 1 -- 172.21.15.189:0/1589464547 wait complete. 2024-04-06T12:24:27.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.521+0000 7efd3e345700 1 Processor -- start 2024-04-06T12:24:27.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.521+0000 7efd3e345700 1 -- start start 2024-04-06T12:24:27.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.522+0000 7efd3e345700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd20003a00 0x7efd380cca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.522+0000 7efd3e345700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd3815f800 0x7efd3815fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.522+0000 7efd3e345700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd380cd3f0 0x7efd380cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.522+0000 7efd3e345700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd380ce040 con 0x7efd3815f800 2024-04-06T12:24:27.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.522+0000 7efd3e345700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd381601c0 con 0x7efd380cd3f0 2024-04-06T12:24:27.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.522+0000 7efd3e345700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd380cddb0 con 0x7efd20003a00 2024-04-06T12:24:27.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.522+0000 7efd377fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd3815f800 0x7efd3815fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.522+0000 7efd377fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd3815f800 0x7efd3815fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33992/0 (socket says 172.21.15.189:33992) 2024-04-06T12:24:27.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.522+0000 7efd377fe700 1 -- 172.21.15.189:0/3160385649 learned_addr learned my addr 172.21.15.189:0/3160385649 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:27.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.522+0000 7efd3c8e2700 1 --2- 172.21.15.189:0/3160385649 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd380cd3f0 0x7efd380cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.522+0000 7efd37fff700 1 --2- 172.21.15.189:0/3160385649 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd20003a00 0x7efd380cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.522+0000 7efd377fe700 1 -- 172.21.15.189:0/3160385649 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd20003a00 msgr2=0x7efd380cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:27.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.522+0000 7efd377fe700 1 --2- 172.21.15.189:0/3160385649 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd20003a00 0x7efd380cca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.522+0000 7efd377fe700 1 -- 172.21.15.189:0/3160385649 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd380cd3f0 msgr2=0x7efd380cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:27.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.522+0000 7efd377fe700 1 --2- 172.21.15.189:0/3160385649 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd380cd3f0 0x7efd380cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.522+0000 7efd377fe700 1 -- 172.21.15.189:0/3160385649 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd381f6120 con 0x7efd3815f800 2024-04-06T12:24:27.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.523+0000 7efd377fe700 1 --2- 172.21.15.189:0/3160385649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd3815f800 0x7efd3815fc80 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7efd2c0118d0 tx=0x7efd2c011be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:27.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.523+0000 7efd34ff9700 1 -- 172.21.15.189:0/3160385649 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c004220 con 0x7efd3815f800 2024-04-06T12:24:27.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.523+0000 7efd34ff9700 1 -- 172.21.15.189:0/3160385649 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2c004380 con 0x7efd3815f800 2024-04-06T12:24:27.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.523+0000 7efd3e345700 1 -- 172.21.15.189:0/3160385649 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd381f6400 con 0x7efd3815f800 2024-04-06T12:24:27.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.523+0000 7efd34ff9700 1 -- 172.21.15.189:0/3160385649 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c0036a0 con 0x7efd3815f800 2024-04-06T12:24:27.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.523+0000 7efd3e345700 1 -- 172.21.15.189:0/3160385649 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd381f69c0 con 0x7efd3815f800 2024-04-06T12:24:27.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.524+0000 7efd3e345700 1 -- 172.21.15.189:0/3160385649 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd381f6d30 con 0x7efd3815f800 2024-04-06T12:24:27.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.524+0000 7efd34ff9700 1 -- 172.21.15.189:0/3160385649 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7efd2c003800 con 0x7efd3815f800 2024-04-06T12:24:27.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.525+0000 7efd34ff9700 1 --2- 172.21.15.189:0/3160385649 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efd2406ae90 0x7efd2406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.525+0000 7efd37fff700 1 --2- 172.21.15.189:0/3160385649 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efd2406ae90 0x7efd2406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.525+0000 7efd34ff9700 1 -- 172.21.15.189:0/3160385649 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7efd2c089bf0 con 0x7efd3815f800 2024-04-06T12:24:27.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.525+0000 7efd37fff700 1 --2- 172.21.15.189:0/3160385649 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efd2406ae90 0x7efd2406d350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efd280090b0 tx=0x7efd28008030 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:27.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.529+0000 7efd34ff9700 1 -- 172.21.15.189:0/3160385649 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efd2c0548f0 con 0x7efd3815f800 2024-04-06T12:24:27.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.748+0000 7efd3e345700 1 -- 172.21.15.189:0/3160385649 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efd3815eb40 con 0x7efd2406ae90 2024-04-06T12:24:27.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.756+0000 7efd34ff9700 1 -- 172.21.15.189:0/3160385649 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7efd28022040 con 0x7efd2406ae90 2024-04-06T12:24:27.753 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:27.753 INFO:teuthology.orchestra.run.smithi189.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:39", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:27.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.758+0000 7efd3e345700 1 -- 172.21.15.189:0/3160385649 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efd2406ae90 msgr2=0x7efd2406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:27.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.758+0000 7efd3e345700 1 --2- 172.21.15.189:0/3160385649 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efd2406ae90 0x7efd2406d350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efd280090b0 tx=0x7efd28008030 comp rx=0 tx=0).stop 2024-04-06T12:24:27.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.758+0000 7efd3e345700 1 -- 172.21.15.189:0/3160385649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd3815f800 msgr2=0x7efd3815fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:27.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.758+0000 7efd3e345700 1 --2- 172.21.15.189:0/3160385649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd3815f800 0x7efd3815fc80 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7efd2c0118d0 tx=0x7efd2c011be0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.758+0000 7efd3e345700 1 -- 172.21.15.189:0/3160385649 shutdown_connections 2024-04-06T12:24:27.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.758+0000 7efd3e345700 1 --2- 172.21.15.189:0/3160385649 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efd2406ae90 0x7efd2406d350 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.758+0000 7efd3e345700 1 --2- 172.21.15.189:0/3160385649 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd380cd3f0 0x7efd380cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.758+0000 7efd3e345700 1 --2- 172.21.15.189:0/3160385649 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd20003a00 0x7efd380cca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.758+0000 7efd3e345700 1 --2- 172.21.15.189:0/3160385649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd3815f800 0x7efd3815fc80 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.758+0000 7efd3e345700 1 -- 172.21.15.189:0/3160385649 >> 172.21.15.189:0/3160385649 conn(0x7efd380d00b0 msgr2=0x7efd38159510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:27.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.758+0000 7efd3e345700 1 -- 172.21.15.189:0/3160385649 shutdown_connections 2024-04-06T12:24:27.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.758+0000 7efd3e345700 1 -- 172.21.15.189:0/3160385649 wait complete. 2024-04-06T12:24:27.767 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:39", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:27.767 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1m', 'retention': {}, 'start': '2024-04-06T00:00:00', 'created': '2024-04-06T12:13:39', 'first': '2024-04-06T12:14:00', 'last': '2024-04-06T12:24:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-06T12:24:27.768 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-06T12:24:27.768 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-06T12:24:27.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:27.768 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2024-04-06T12:24:27.786 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:27.786 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:27.786 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:27.786 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:27.786 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:27.786 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:27.786 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:27.786 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:27.787 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:27.787 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:27.787 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:27.787 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-06-12_14_00_UTC', 'scheduled-2024-04-06-12_15_00_UTC', 'scheduled-2024-04-06-12_16_00_UTC', 'scheduled-2024-04-06-12_17_00_UTC', 'scheduled-2024-04-06-12_18_00_UTC', 'scheduled-2024-04-06-12_19_00_UTC', 'scheduled-2024-04-06-12_20_00_UTC', 'scheduled-2024-04-06-12_21_00_UTC', 'scheduled-2024-04-06-12_22_00_UTC', 'scheduled-2024-04-06-12_23_00_UTC', 'scheduled-2024-04-06-12_24_00_UTC'] 2024-04-06T12:24:27.787 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir8 2024-04-06T12:24:27.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.952+0000 7f4baecb9700 1 Processor -- start 2024-04-06T12:24:27.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.952+0000 7f4baecb9700 1 -- start start 2024-04-06T12:24:27.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.953+0000 7f4baecb9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ba8071f00 0x7f4ba806f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.953+0000 7f4baecb9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba815e250 con 0x7f4ba806a730 2024-04-06T12:24:27.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.953+0000 7f4baecb9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba815e530 con 0x7f4ba815ec30 2024-04-06T12:24:27.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.953+0000 7f4baecb9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba8072350 con 0x7f4ba8071f00 2024-04-06T12:24:27.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.953+0000 7f4baca55700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4ba806a730 0x7f4ba806ab10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48780/0 (socket says 172.21.15.189:48780) 2024-04-06T12:24:27.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.953+0000 7f4baca55700 1 -- 172.21.15.189:0/2222094430 learned_addr learned my addr 172.21.15.189:0/2222094430 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:27.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4ba77fe700 1 -- 172.21.15.189:0/2222094430 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120415512 0 0) 0x7f4ba815e250 con 0x7f4ba806a730 2024-04-06T12:24:27.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4bad256700 1 --2- 172.21.15.189:0/2222094430 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ba8071f00 0x7f4ba806f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4ba77fe700 1 -- 172.21.15.189:0/2222094430 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b980035b0 con 0x7f4ba806a730 2024-04-06T12:24:27.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4ba77fe700 1 -- 172.21.15.189:0/2222094430 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 228820505 0 0) 0x7f4b980035b0 con 0x7f4ba806a730 2024-04-06T12:24:27.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4ba77fe700 1 -- 172.21.15.189:0/2222094430 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ba8071f00 msgr2=0x7f4ba806f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:27.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4ba77fe700 1 --2- 172.21.15.189:0/2222094430 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ba8071f00 0x7f4ba806f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4ba77fe700 1 -- 172.21.15.189:0/2222094430 >> v1:172.21.15.192:6789/0 conn(0x7f4ba815ec30 legacy=0x7f4ba815da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:27.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4ba77fe700 1 -- 172.21.15.189:0/2222094430 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba815e810 con 0x7f4ba806a730 2024-04-06T12:24:27.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4ba77fe700 1 -- 172.21.15.189:0/2222094430 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f4b9c001670 con 0x7f4ba806a730 2024-04-06T12:24:27.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4ba77fe700 1 -- 172.21.15.189:0/2222094430 >> v1:172.21.15.189:6789/0 conn(0x7f4ba806a730 legacy=0x7f4ba806ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:27.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4ba77fe700 1 --2- 172.21.15.189:0/2222094430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b98003a30 0x7f4b98003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4ba77fe700 1 --2- 172.21.15.189:0/2222094430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b980043e0 0x7f4b980048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4ba77fe700 1 --2- 172.21.15.189:0/2222094430 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ba8071f00 0x7f4b98005110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4ba77fe700 1 -- 172.21.15.189:0/2222094430 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba8072350 con 0x7f4b98003a30 2024-04-06T12:24:27.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4ba77fe700 1 -- 172.21.15.189:0/2222094430 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9800ba10 con 0x7f4b980043e0 2024-04-06T12:24:27.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4ba77fe700 1 -- 172.21.15.189:0/2222094430 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9800bbd0 con 0x7f4ba8071f00 2024-04-06T12:24:27.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4baca55700 1 --2- 172.21.15.189:0/2222094430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b98003a30 0x7f4b98003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4bad256700 1 --2- 172.21.15.189:0/2222094430 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ba8071f00 0x7f4b98005110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4ba7fff700 1 --2- 172.21.15.189:0/2222094430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b980043e0 0x7f4b980048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4baca55700 1 -- 172.21.15.189:0/2222094430 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ba8071f00 msgr2=0x7f4b98005110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:27.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4baca55700 1 --2- 172.21.15.189:0/2222094430 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ba8071f00 0x7f4b98005110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4baca55700 1 -- 172.21.15.189:0/2222094430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b980043e0 msgr2=0x7f4b980048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:27.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4baca55700 1 --2- 172.21.15.189:0/2222094430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b980043e0 0x7f4b980048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4baca55700 1 -- 172.21.15.189:0/2222094430 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b980035b0 con 0x7f4b98003a30 2024-04-06T12:24:27.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.954+0000 7f4baca55700 1 --2- 172.21.15.189:0/2222094430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b98003a30 0x7f4b98003e10 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c00c2f0 tx=0x7f4b98005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c232625b6ae856fd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:27.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.955+0000 7f4ba77fe700 1 -- 172.21.15.189:0/2222094430 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c00ce30 con 0x7f4b98003a30 2024-04-06T12:24:27.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.955+0000 7f4ba77fe700 1 -- 172.21.15.189:0/2222094430 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9c005960 con 0x7f4b98003a30 2024-04-06T12:24:27.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.955+0000 7f4ba77fe700 1 -- 172.21.15.189:0/2222094430 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c005cf0 con 0x7f4b98003a30 2024-04-06T12:24:27.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.955+0000 7f4baecb9700 1 -- 172.21.15.189:0/2222094430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b98003a30 msgr2=0x7f4b98003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:27.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.955+0000 7f4baecb9700 1 --2- 172.21.15.189:0/2222094430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b98003a30 0x7f4b98003e10 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c00c2f0 tx=0x7f4b98005880 comp rx=0 tx=0).stop 2024-04-06T12:24:27.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.955+0000 7f4bad256700 1 -- 172.21.15.189:0/2222094430 reap_dead start 2024-04-06T12:24:27.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.955+0000 7f4baecb9700 1 -- 172.21.15.189:0/2222094430 shutdown_connections 2024-04-06T12:24:27.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.955+0000 7f4baecb9700 1 -- 172.21.15.189:0/2222094430 >> 172.21.15.189:0/2222094430 conn(0x7f4ba80b1e30 msgr2=0x7f4ba80b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:27.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.955+0000 7f4baecb9700 1 -- 172.21.15.189:0/2222094430 shutdown_connections 2024-04-06T12:24:27.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.955+0000 7f4baecb9700 1 -- 172.21.15.189:0/2222094430 wait complete. 2024-04-06T12:24:27.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.957+0000 7f4baecb9700 1 Processor -- start 2024-04-06T12:24:27.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.957+0000 7f4baecb9700 1 -- start start 2024-04-06T12:24:27.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.958+0000 7f4baecb9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b98003a30 0x7f4ba8152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.958+0000 7f4baecb9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ba8159fe0 0x7f4ba81543c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.958+0000 7f4baecb9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ba8154d50 0x7f4ba815b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.958+0000 7f4baecb9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba8152f50 con 0x7f4ba8159fe0 2024-04-06T12:24:27.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.958+0000 7f4baecb9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba80cd900 con 0x7f4ba8154d50 2024-04-06T12:24:27.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.958+0000 7f4baecb9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba815a680 con 0x7f4b98003a30 2024-04-06T12:24:27.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.958+0000 7f4ba7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ba8159fe0 0x7f4ba81543c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.958+0000 7f4ba7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ba8159fe0 0x7f4ba81543c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34006/0 (socket says 172.21.15.189:34006) 2024-04-06T12:24:27.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.958+0000 7f4ba7fff700 1 -- 172.21.15.189:0/465419978 learned_addr learned my addr 172.21.15.189:0/465419978 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:27.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.958+0000 7f4baca55700 1 --2- 172.21.15.189:0/465419978 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b98003a30 0x7f4ba8152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.958+0000 7f4ba7fff700 1 -- 172.21.15.189:0/465419978 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b98003a30 msgr2=0x7f4ba8152a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:27.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.958+0000 7f4ba7fff700 1 --2- 172.21.15.189:0/465419978 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b98003a30 0x7f4ba8152a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.958+0000 7f4ba7fff700 1 -- 172.21.15.189:0/465419978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ba8154d50 msgr2=0x7f4ba815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:27.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.958+0000 7f4bad256700 1 --2- 172.21.15.189:0/465419978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ba8154d50 0x7f4ba815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.958+0000 7f4ba7fff700 1 --2- 172.21.15.189:0/465419978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ba8154d50 0x7f4ba815b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:27.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.958+0000 7f4ba7fff700 1 -- 172.21.15.189:0/465419978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba8154900 con 0x7f4ba8159fe0 2024-04-06T12:24:27.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.958+0000 7f4ba7fff700 1 --2- 172.21.15.189:0/465419978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ba8159fe0 0x7f4ba81543c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4b94001eb0 tx=0x7f4b9400fb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:27.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.959+0000 7f4ba57fa700 1 -- 172.21.15.189:0/465419978 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b940052e0 con 0x7f4ba8159fe0 2024-04-06T12:24:27.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.959+0000 7f4ba57fa700 1 -- 172.21.15.189:0/465419978 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b94005460 con 0x7f4ba8159fe0 2024-04-06T12:24:27.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.959+0000 7f4baecb9700 1 -- 172.21.15.189:0/465419978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba815a8d0 con 0x7f4ba8159fe0 2024-04-06T12:24:27.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.959+0000 7f4ba57fa700 1 -- 172.21.15.189:0/465419978 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b940038c0 con 0x7f4ba8159fe0 2024-04-06T12:24:27.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.959+0000 7f4baecb9700 1 -- 172.21.15.189:0/465419978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba815ae90 con 0x7f4ba8159fe0 2024-04-06T12:24:27.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.960+0000 7f4baecb9700 1 -- 172.21.15.189:0/465419978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba80748f0 con 0x7f4ba8159fe0 2024-04-06T12:24:27.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.960+0000 7f4ba57fa700 1 -- 172.21.15.189:0/465419978 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f4b94003a20 con 0x7f4ba8159fe0 2024-04-06T12:24:27.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.960+0000 7f4ba57fa700 1 --2- 172.21.15.189:0/465419978 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b8c06ae90 0x7f4b8c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:27.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.960+0000 7f4ba57fa700 1 -- 172.21.15.189:0/465419978 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f4b940897c0 con 0x7f4ba8159fe0 2024-04-06T12:24:27.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.960+0000 7f4baca55700 1 --2- 172.21.15.189:0/465419978 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b8c06ae90 0x7f4b8c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:27.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.961+0000 7f4baca55700 1 --2- 172.21.15.189:0/465419978 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b8c06ae90 0x7f4b8c06d350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c015040 tx=0x7f4b9c007000 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:27.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:27.965+0000 7f4ba57fa700 1 -- 172.21.15.189:0/465419978 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4b940544c0 con 0x7f4ba8159fe0 2024-04-06T12:24:28.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.183+0000 7f4baecb9700 1 -- 172.21.15.189:0/465419978 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4ba806f310 con 0x7f4b8c06ae90 2024-04-06T12:24:28.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.191+0000 7f4ba57fa700 1 -- 172.21.15.189:0/465419978 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f4b9c001430 con 0x7f4b8c06ae90 2024-04-06T12:24:28.190 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:28.190 INFO:teuthology.orchestra.run.smithi189.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:39", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:28.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.193+0000 7f4baecb9700 1 -- 172.21.15.189:0/465419978 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b8c06ae90 msgr2=0x7f4b8c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:28.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.193+0000 7f4baecb9700 1 --2- 172.21.15.189:0/465419978 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b8c06ae90 0x7f4b8c06d350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c015040 tx=0x7f4b9c007000 comp rx=0 tx=0).stop 2024-04-06T12:24:28.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.193+0000 7f4baecb9700 1 -- 172.21.15.189:0/465419978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ba8159fe0 msgr2=0x7f4ba81543c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:28.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.193+0000 7f4baecb9700 1 --2- 172.21.15.189:0/465419978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ba8159fe0 0x7f4ba81543c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4b94001eb0 tx=0x7f4b9400fb50 comp rx=0 tx=0).stop 2024-04-06T12:24:28.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.193+0000 7f4baecb9700 1 -- 172.21.15.189:0/465419978 shutdown_connections 2024-04-06T12:24:28.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.193+0000 7f4baecb9700 1 --2- 172.21.15.189:0/465419978 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b8c06ae90 0x7f4b8c06d350 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.193+0000 7f4baecb9700 1 --2- 172.21.15.189:0/465419978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ba8154d50 0x7f4ba815b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.193+0000 7f4baecb9700 1 --2- 172.21.15.189:0/465419978 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b98003a30 0x7f4ba8152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.193+0000 7f4baecb9700 1 --2- 172.21.15.189:0/465419978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ba8159fe0 0x7f4ba81543c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.193+0000 7f4baecb9700 1 -- 172.21.15.189:0/465419978 >> 172.21.15.189:0/465419978 conn(0x7f4ba80b1e30 msgr2=0x7f4ba80b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:28.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.193+0000 7f4baecb9700 1 -- 172.21.15.189:0/465419978 shutdown_connections 2024-04-06T12:24:28.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.193+0000 7f4baecb9700 1 -- 172.21.15.189:0/465419978 wait complete. 2024-04-06T12:24:28.203 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:39", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:28.203 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1m', 'retention': {}, 'start': '2024-04-06T00:00:00', 'created': '2024-04-06T12:13:39', 'first': '2024-04-06T12:14:00', 'last': '2024-04-06T12:24:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-06T12:24:28.203 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-06T12:24:28.203 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-06T12:24:28.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:28.203 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2024-04-06T12:24:28.221 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:28.221 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:28.221 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:28.221 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:28.221 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:28.221 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:28.221 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:28.221 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:28.221 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:28.221 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:28.222 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:28.222 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-06-12_14_00_UTC', 'scheduled-2024-04-06-12_15_00_UTC', 'scheduled-2024-04-06-12_16_00_UTC', 'scheduled-2024-04-06-12_17_00_UTC', 'scheduled-2024-04-06-12_18_00_UTC', 'scheduled-2024-04-06-12_19_00_UTC', 'scheduled-2024-04-06-12_20_00_UTC', 'scheduled-2024-04-06-12_21_00_UTC', 'scheduled-2024-04-06-12_22_00_UTC', 'scheduled-2024-04-06-12_23_00_UTC', 'scheduled-2024-04-06-12_24_00_UTC'] 2024-04-06T12:24:28.222 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir9 2024-04-06T12:24:28.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.387+0000 7f45b1008700 1 Processor -- start 2024-04-06T12:24:28.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.387+0000 7f45b1008700 1 -- start start 2024-04-06T12:24:28.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.388+0000 7f45b1008700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45ac15d7b0 0x7f45ac0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:28.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.388+0000 7f45b1008700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ac0b3990 con 0x7f45ac06a730 2024-04-06T12:24:28.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.388+0000 7f45b1008700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ac0b3060 con 0x7f45ac15e500 2024-04-06T12:24:28.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.388+0000 7f45b1008700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac0b3340 con 0x7f45ac15d7b0 2024-04-06T12:24:28.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.388+0000 7f45aa59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f45ac06a730 0x7f45ac15ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48788/0 (socket says 172.21.15.189:48788) 2024-04-06T12:24:28.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.388+0000 7f45aa59c700 1 -- 172.21.15.189:0/1151779118 learned_addr learned my addr 172.21.15.189:0/1151779118 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:28.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.388+0000 7f45aad9d700 1 --2- 172.21.15.189:0/1151779118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45ac15d7b0 0x7f45ac0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:28.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.388+0000 7f45a9d9b700 1 -- 172.21.15.189:0/1151779118 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899052008 0 0) 0x7f45ac0b3990 con 0x7f45ac06a730 2024-04-06T12:24:28.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45a9d9b700 1 -- 172.21.15.189:0/1151779118 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45940035b0 con 0x7f45ac06a730 2024-04-06T12:24:28.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45a9d9b700 1 -- 172.21.15.189:0/1151779118 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1709226308 0 0) 0x7f45940035b0 con 0x7f45ac06a730 2024-04-06T12:24:28.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45a9d9b700 1 -- 172.21.15.189:0/1151779118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45ac15d7b0 msgr2=0x7f45ac0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:28.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45a9d9b700 1 --2- 172.21.15.189:0/1151779118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45ac15d7b0 0x7f45ac0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45a9d9b700 1 -- 172.21.15.189:0/1151779118 >> v1:172.21.15.192:6789/0 conn(0x7f45ac15e500 legacy=0x7f45ac0717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:28.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45a9d9b700 1 -- 172.21.15.189:0/1151779118 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac0b3620 con 0x7f45ac06a730 2024-04-06T12:24:28.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45a9d9b700 1 -- 172.21.15.189:0/1151779118 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f459c001670 con 0x7f45ac06a730 2024-04-06T12:24:28.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45a9d9b700 1 -- 172.21.15.189:0/1151779118 >> v1:172.21.15.189:6789/0 conn(0x7f45ac06a730 legacy=0x7f45ac15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:28.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45a9d9b700 1 --2- 172.21.15.189:0/1151779118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45ac15d7b0 0x7f4594003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:28.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45a9d9b700 1 --2- 172.21.15.189:0/1151779118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4594003f70 0x7f4594004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:28.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45a9d9b700 1 --2- 172.21.15.189:0/1151779118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4594004d90 0x7f4594005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:28.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45a9d9b700 1 -- 172.21.15.189:0/1151779118 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac0b3340 con 0x7f4594004d90 2024-04-06T12:24:28.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45a9d9b700 1 -- 172.21.15.189:0/1151779118 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459400bb20 con 0x7f4594003f70 2024-04-06T12:24:28.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45a9d9b700 1 -- 172.21.15.189:0/1151779118 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459400bce0 con 0x7f45ac15d7b0 2024-04-06T12:24:28.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45ab59e700 1 --2- 172.21.15.189:0/1151779118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4594004d90 0x7f4594005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:28.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45aa59c700 1 --2- 172.21.15.189:0/1151779118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4594003f70 0x7f4594004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:28.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45aad9d700 1 --2- 172.21.15.189:0/1151779118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45ac15d7b0 0x7f4594003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:28.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45ab59e700 1 -- 172.21.15.189:0/1151779118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45ac15d7b0 msgr2=0x7f4594003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:28.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45ab59e700 1 --2- 172.21.15.189:0/1151779118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45ac15d7b0 0x7f4594003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45ab59e700 1 -- 172.21.15.189:0/1151779118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4594003f70 msgr2=0x7f4594004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:28.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45ab59e700 1 --2- 172.21.15.189:0/1151779118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4594003f70 0x7f4594004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45ab59e700 1 -- 172.21.15.189:0/1151779118 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45940035b0 con 0x7f4594004d90 2024-04-06T12:24:28.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.389+0000 7f45ab59e700 1 --2- 172.21.15.189:0/1151779118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4594004d90 0x7f4594005210 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f45a400c370 tx=0x7f4594005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e697a751c278d0b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:28.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.390+0000 7f45a9d9b700 1 -- 172.21.15.189:0/1151779118 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4001f20 con 0x7f4594004d90 2024-04-06T12:24:28.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.390+0000 7f45a9d9b700 1 -- 172.21.15.189:0/1151779118 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a4012420 con 0x7f4594004d90 2024-04-06T12:24:28.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.390+0000 7f45a9d9b700 1 -- 172.21.15.189:0/1151779118 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a40127d0 con 0x7f4594004d90 2024-04-06T12:24:28.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.390+0000 7f45b1008700 1 -- 172.21.15.189:0/1151779118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4594004d90 msgr2=0x7f4594005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:28.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.390+0000 7f45b1008700 1 --2- 172.21.15.189:0/1151779118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4594004d90 0x7f4594005210 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f45a400c370 tx=0x7f4594005880 comp rx=0 tx=0).stop 2024-04-06T12:24:28.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.390+0000 7f45ab59e700 1 -- 172.21.15.189:0/1151779118 reap_dead start 2024-04-06T12:24:28.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.390+0000 7f45b1008700 1 -- 172.21.15.189:0/1151779118 shutdown_connections 2024-04-06T12:24:28.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.390+0000 7f45b1008700 1 -- 172.21.15.189:0/1151779118 >> 172.21.15.189:0/1151779118 conn(0x7f45ac0d00b0 msgr2=0x7f45ac0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:28.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.390+0000 7f45b1008700 1 -- 172.21.15.189:0/1151779118 shutdown_connections 2024-04-06T12:24:28.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.390+0000 7f45b1008700 1 -- 172.21.15.189:0/1151779118 wait complete. 2024-04-06T12:24:28.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.392+0000 7f45b1008700 1 Processor -- start 2024-04-06T12:24:28.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.392+0000 7f45b1008700 1 -- start start 2024-04-06T12:24:28.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.392+0000 7f45b1008700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45ac0cb8c0 0x7f45ac0b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:28.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.392+0000 7f45b1008700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45ac0cbca0 0x7f45ac0b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:28.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.392+0000 7f45b1008700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45ac0cc130 0x7f45ac0cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:28.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.392+0000 7f45b1008700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac15cb50 con 0x7f45ac0cbca0 2024-04-06T12:24:28.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.392+0000 7f45b1008700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac0ccaf0 con 0x7f45ac0cc130 2024-04-06T12:24:28.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.392+0000 7f45b1008700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac0ccd80 con 0x7f45ac0cb8c0 2024-04-06T12:24:28.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.392+0000 7f45aa59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45ac0cbca0 0x7f45ac0b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:28.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.392+0000 7f45aa59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45ac0cbca0 0x7f45ac0b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34016/0 (socket says 172.21.15.189:34016) 2024-04-06T12:24:28.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.393+0000 7f45aa59c700 1 -- 172.21.15.189:0/1589890371 learned_addr learned my addr 172.21.15.189:0/1589890371 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:28.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.393+0000 7f45aa59c700 1 -- 172.21.15.189:0/1589890371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45ac0cb8c0 msgr2=0x7f45ac0b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:28.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.393+0000 7f45aa59c700 1 --2- 172.21.15.189:0/1589890371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45ac0cb8c0 0x7f45ac0b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.393+0000 7f45aa59c700 1 -- 172.21.15.189:0/1589890371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45ac0cc130 msgr2=0x7f45ac0cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:28.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.393+0000 7f45ab59e700 1 --2- 172.21.15.189:0/1589890371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45ac0cc130 0x7f45ac0cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:28.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.393+0000 7f45aa59c700 1 --2- 172.21.15.189:0/1589890371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45ac0cc130 0x7f45ac0cc5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.393+0000 7f45aa59c700 1 -- 172.21.15.189:0/1589890371 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac0cd060 con 0x7f45ac0cbca0 2024-04-06T12:24:28.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.393+0000 7f45ab59e700 1 --2- 172.21.15.189:0/1589890371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45ac0cc130 0x7f45ac0cc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:24:28.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.393+0000 7f45aa59c700 1 --2- 172.21.15.189:0/1589890371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45ac0cbca0 0x7f45ac0b12f0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f459c0072c0 tx=0x7f459c007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:28.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.393+0000 7f45937fe700 1 -- 172.21.15.189:0/1589890371 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c0077a0 con 0x7f45ac0cbca0 2024-04-06T12:24:28.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.393+0000 7f45937fe700 1 -- 172.21.15.189:0/1589890371 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459c00f870 con 0x7f45ac0cbca0 2024-04-06T12:24:28.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.393+0000 7f45b1008700 1 -- 172.21.15.189:0/1589890371 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45ac0cd340 con 0x7f45ac0cbca0 2024-04-06T12:24:28.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.393+0000 7f45937fe700 1 -- 172.21.15.189:0/1589890371 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c00fc20 con 0x7f45ac0cbca0 2024-04-06T12:24:28.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.393+0000 7f45b1008700 1 -- 172.21.15.189:0/1589890371 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ac0cd900 con 0x7f45ac0cbca0 2024-04-06T12:24:28.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.394+0000 7f45b1008700 1 -- 172.21.15.189:0/1589890371 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45ac074990 con 0x7f45ac0cbca0 2024-04-06T12:24:28.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.395+0000 7f45937fe700 1 -- 172.21.15.189:0/1589890371 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f459c00fd80 con 0x7f45ac0cbca0 2024-04-06T12:24:28.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.395+0000 7f45937fe700 1 --2- 172.21.15.189:0/1589890371 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f459806ae90 0x7f459806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:28.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.395+0000 7f45937fe700 1 -- 172.21.15.189:0/1589890371 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f459c089950 con 0x7f45ac0cbca0 2024-04-06T12:24:28.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.395+0000 7f45aad9d700 1 --2- 172.21.15.189:0/1589890371 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f459806ae90 0x7f459806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:28.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.396+0000 7f45aad9d700 1 --2- 172.21.15.189:0/1589890371 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f459806ae90 0x7f459806d350 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f45a0009b50 tx=0x7f45a0009670 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:28.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.400+0000 7f45937fe700 1 -- 172.21.15.189:0/1589890371 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f459c054700 con 0x7f45ac0cbca0 2024-04-06T12:24:28.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.619+0000 7f45b1008700 1 -- 172.21.15.189:0/1589890371 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f45ac06af30 con 0x7f459806ae90 2024-04-06T12:24:28.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.627+0000 7f45937fe700 1 -- 172.21.15.189:0/1589890371 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f45ac06af30 con 0x7f459806ae90 2024-04-06T12:24:28.624 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:28.624 INFO:teuthology.orchestra.run.smithi189.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:40", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:28.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.629+0000 7f45b1008700 1 -- 172.21.15.189:0/1589890371 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f459806ae90 msgr2=0x7f459806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:28.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.629+0000 7f45b1008700 1 --2- 172.21.15.189:0/1589890371 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f459806ae90 0x7f459806d350 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f45a0009b50 tx=0x7f45a0009670 comp rx=0 tx=0).stop 2024-04-06T12:24:28.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.629+0000 7f45b1008700 1 -- 172.21.15.189:0/1589890371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45ac0cbca0 msgr2=0x7f45ac0b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:28.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.629+0000 7f45b1008700 1 --2- 172.21.15.189:0/1589890371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45ac0cbca0 0x7f45ac0b12f0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f459c0072c0 tx=0x7f459c007680 comp rx=0 tx=0).stop 2024-04-06T12:24:28.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.629+0000 7f45b1008700 1 -- 172.21.15.189:0/1589890371 shutdown_connections 2024-04-06T12:24:28.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.629+0000 7f45b1008700 1 --2- 172.21.15.189:0/1589890371 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f459806ae90 0x7f459806d350 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.629+0000 7f45b1008700 1 --2- 172.21.15.189:0/1589890371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45ac0cc130 0x7f45ac0cc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.629+0000 7f45b1008700 1 --2- 172.21.15.189:0/1589890371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45ac0cb8c0 0x7f45ac0b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.629+0000 7f45b1008700 1 --2- 172.21.15.189:0/1589890371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45ac0cbca0 0x7f45ac0b12f0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.629+0000 7f45b1008700 1 -- 172.21.15.189:0/1589890371 >> 172.21.15.189:0/1589890371 conn(0x7f45ac0d00b0 msgr2=0x7f45ac06fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:28.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.629+0000 7f45b1008700 1 -- 172.21.15.189:0/1589890371 shutdown_connections 2024-04-06T12:24:28.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.629+0000 7f45b1008700 1 -- 172.21.15.189:0/1589890371 wait complete. 2024-04-06T12:24:28.639 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:13:40", "first": "2024-04-06T12:14:00", "last": "2024-04-06T12:24:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-06T12:24:28.639 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1m', 'retention': {}, 'start': '2024-04-06T00:00:00', 'created': '2024-04-06T12:13:40', 'first': '2024-04-06T12:14:00', 'last': '2024-04-06T12:24:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-06T12:24:28.639 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-06T12:24:28.639 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-06T12:24:28.639 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir0 1m 2024-04-06T12:24:28.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.766+0000 7f282d79d700 1 Processor -- start 2024-04-06T12:24:28.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.766+0000 7f282d79d700 1 -- start start 2024-04-06T12:24:28.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.767+0000 7f282d79d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2828161a70 0x7f28280b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:28.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.767+0000 7f282d79d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f28280b23d0 0x7f28280b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:28.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.767+0000 7f282d79d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28280b3490 0x7f28281571f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:28.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.767+0000 7f282d79d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28280b2860 con 0x7f2828161a70 2024-04-06T12:24:28.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.767+0000 7f282d79d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2828072520 con 0x7f28280b23d0 2024-04-06T12:24:28.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.767+0000 7f282d79d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2828072090 con 0x7f28280b3490 2024-04-06T12:24:28.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.767+0000 7f2827fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2828161a70 0x7f28280b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:28.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.767+0000 7f2827fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2828161a70 0x7f28280b1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34032/0 (socket says 172.21.15.189:34032) 2024-04-06T12:24:28.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.767+0000 7f2827fff700 1 -- 172.21.15.189:0/1224832768 learned_addr learned my addr 172.21.15.189:0/1224832768 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:28.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.767+0000 7f28277fe700 1 --2- 172.21.15.189:0/1224832768 >> v2:172.21.15.192:3300/0 conn(0x7f28280b23d0 0x7f28280b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:28.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.767+0000 7f282cf9c700 1 --2- 172.21.15.189:0/1224832768 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28280b3490 0x7f28281571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:28.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.767+0000 7f2827fff700 1 -- 172.21.15.189:0/1224832768 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28280b3490 msgr2=0x7f28281571f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:28.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.767+0000 7f2827fff700 1 --2- 172.21.15.189:0/1224832768 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28280b3490 0x7f28281571f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.767+0000 7f2827fff700 1 -- 172.21.15.189:0/1224832768 >> v2:172.21.15.192:3300/0 conn(0x7f28280b23d0 msgr2=0x7f28280b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:28.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.767+0000 7f2827fff700 1 --2- 172.21.15.189:0/1224832768 >> v2:172.21.15.192:3300/0 conn(0x7f28280b23d0 0x7f28280b2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.767+0000 7f2827fff700 1 -- 172.21.15.189:0/1224832768 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2828071760 con 0x7f2828161a70 2024-04-06T12:24:28.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.768+0000 7f2827fff700 1 --2- 172.21.15.189:0/1224832768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2828161a70 0x7f28280b1e90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f281c009920 tx=0x7f281c009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=629e731796b3f8b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:28.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.768+0000 7f2826ffd700 1 -- 172.21.15.189:0/1224832768 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281c00b9e0 con 0x7f2828161a70 2024-04-06T12:24:28.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.768+0000 7f2826ffd700 1 -- 172.21.15.189:0/1224832768 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281c00bb40 con 0x7f2828161a70 2024-04-06T12:24:28.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.768+0000 7f2826ffd700 1 -- 172.21.15.189:0/1224832768 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281c009e70 con 0x7f2828161a70 2024-04-06T12:24:28.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.768+0000 7f282d79d700 1 -- 172.21.15.189:0/1224832768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2828161a70 msgr2=0x7f28280b1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:28.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.768+0000 7f282d79d700 1 --2- 172.21.15.189:0/1224832768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2828161a70 0x7f28280b1e90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f281c009920 tx=0x7f281c009c30 comp rx=0 tx=0).stop 2024-04-06T12:24:28.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.768+0000 7f282d79d700 1 -- 172.21.15.189:0/1224832768 shutdown_connections 2024-04-06T12:24:28.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.768+0000 7f282d79d700 1 --2- 172.21.15.189:0/1224832768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2828161a70 0x7f28280b1e90 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.768+0000 7f282d79d700 1 --2- 172.21.15.189:0/1224832768 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28280b3490 0x7f28281571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.768+0000 7f282d79d700 1 --2- 172.21.15.189:0/1224832768 >> v2:172.21.15.192:3300/0 conn(0x7f28280b23d0 0x7f28280b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.768+0000 7f282d79d700 1 -- 172.21.15.189:0/1224832768 >> 172.21.15.189:0/1224832768 conn(0x7f28280b5470 msgr2=0x7f28280b5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:28.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.768+0000 7f282d79d700 1 -- 172.21.15.189:0/1224832768 shutdown_connections 2024-04-06T12:24:28.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.768+0000 7f282d79d700 1 -- 172.21.15.189:0/1224832768 wait complete. 2024-04-06T12:24:28.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.769+0000 7f282d79d700 1 Processor -- start 2024-04-06T12:24:28.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.770+0000 7f282d79d700 1 -- start start 2024-04-06T12:24:28.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.770+0000 7f282d79d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28280b23d0 0x7f282814fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:28.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.770+0000 7f282d79d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28280b3490 0x7f28281512e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:28.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.770+0000 7f282d79d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2828161a70 0x7f28280b04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:28.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.770+0000 7f282d79d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2828150010 con 0x7f28280b3490 2024-04-06T12:24:28.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.770+0000 7f282d79d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28280cc950 con 0x7f28280b23d0 2024-04-06T12:24:28.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.770+0000 7f282d79d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28280d0620 con 0x7f2828161a70 2024-04-06T12:24:28.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.771+0000 7f28277fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28280b3490 0x7f28281512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:28.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.771+0000 7f28277fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28280b3490 0x7f28281512e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34048/0 (socket says 172.21.15.189:34048) 2024-04-06T12:24:28.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.771+0000 7f28277fe700 1 -- 172.21.15.189:0/3840696640 learned_addr learned my addr 172.21.15.189:0/3840696640 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:28.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.771+0000 7f282cf9c700 1 --2- 172.21.15.189:0/3840696640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2828161a70 0x7f28280b04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:28.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.771+0000 7f2827fff700 1 --2- 172.21.15.189:0/3840696640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28280b23d0 0x7f282814fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:28.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.771+0000 7f28277fe700 1 -- 172.21.15.189:0/3840696640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2828161a70 msgr2=0x7f28280b04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:28.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.771+0000 7f28277fe700 1 --2- 172.21.15.189:0/3840696640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2828161a70 0x7f28280b04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.771+0000 7f28277fe700 1 -- 172.21.15.189:0/3840696640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28280b23d0 msgr2=0x7f282814fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:28.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.771+0000 7f28277fe700 1 --2- 172.21.15.189:0/3840696640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28280b23d0 0x7f282814fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:28.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.771+0000 7f28277fe700 1 -- 172.21.15.189:0/3840696640 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281c009550 con 0x7f28280b3490 2024-04-06T12:24:28.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.771+0000 7f282cf9c700 1 --2- 172.21.15.189:0/3840696640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2828161a70 0x7f28280b04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:24:28.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.771+0000 7f2827fff700 1 --2- 172.21.15.189:0/3840696640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28280b23d0 0x7f282814fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:24:28.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.771+0000 7f28277fe700 1 --2- 172.21.15.189:0/3840696640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28280b3490 0x7f28281512e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f281800d830 tx=0x7f281800dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:28.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.771+0000 7f2824ff9700 1 -- 172.21.15.189:0/3840696640 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28180125f0 con 0x7f28280b3490 2024-04-06T12:24:28.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.771+0000 7f2824ff9700 1 -- 172.21.15.189:0/3840696640 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2818012750 con 0x7f28280b3490 2024-04-06T12:24:28.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.771+0000 7f2824ff9700 1 -- 172.21.15.189:0/3840696640 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2818012ae0 con 0x7f28280b3490 2024-04-06T12:24:28.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.771+0000 7f282d79d700 1 -- 172.21.15.189:0/3840696640 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2828151820 con 0x7f28280b3490 2024-04-06T12:24:28.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.771+0000 7f282d79d700 1 -- 172.21.15.189:0/3840696640 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28281f3990 con 0x7f28280b3490 2024-04-06T12:24:28.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.772+0000 7f282d79d700 1 -- 172.21.15.189:0/3840696640 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2828151130 con 0x7f28280b3490 2024-04-06T12:24:28.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.773+0000 7f2824ff9700 1 -- 172.21.15.189:0/3840696640 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f2818012c40 con 0x7f28280b3490 2024-04-06T12:24:28.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.773+0000 7f2824ff9700 1 --2- 172.21.15.189:0/3840696640 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f281406ae90 0x7f281406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:28.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.773+0000 7f2824ff9700 1 -- 172.21.15.189:0/3840696640 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f2818089ca0 con 0x7f28280b3490 2024-04-06T12:24:28.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.773+0000 7f2827fff700 1 --2- 172.21.15.189:0/3840696640 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f281406ae90 0x7f281406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:28.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.773+0000 7f2827fff700 1 --2- 172.21.15.189:0/3840696640 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f281406ae90 0x7f281406d350 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f281c000c00 tx=0x7f281c008030 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:28.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.777+0000 7f2824ff9700 1 -- 172.21.15.189:0/3840696640 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f28180549a0 con 0x7f28280b3490 2024-04-06T12:24:28.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:28.997+0000 7f282d79d700 1 -- 172.21.15.189:0/3840696640 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2828071d20 con 0x7f281406ae90 2024-04-06T12:24:29.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:29.073+0000 7f2824ff9700 1 -- 172.21.15.189:0/3840696640 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f281c00bcb0 con 0x7f281406ae90 2024-04-06T12:24:29.071 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule removed for path /snap_test_dir1/dir0 2024-04-06T12:24:29.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:29.075+0000 7f282d79d700 1 -- 172.21.15.189:0/3840696640 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f281406ae90 msgr2=0x7f281406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:29.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:29.075+0000 7f282d79d700 1 --2- 172.21.15.189:0/3840696640 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f281406ae90 0x7f281406d350 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f281c000c00 tx=0x7f281c008030 comp rx=0 tx=0).stop 2024-04-06T12:24:29.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:29.075+0000 7f282d79d700 1 -- 172.21.15.189:0/3840696640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28280b3490 msgr2=0x7f28281512e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:29.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:29.075+0000 7f282d79d700 1 --2- 172.21.15.189:0/3840696640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28280b3490 0x7f28281512e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f281800d830 tx=0x7f281800dbf0 comp rx=0 tx=0).stop 2024-04-06T12:24:29.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:29.075+0000 7f282d79d700 1 -- 172.21.15.189:0/3840696640 shutdown_connections 2024-04-06T12:24:29.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:29.075+0000 7f282d79d700 1 --2- 172.21.15.189:0/3840696640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2828161a70 0x7f28280b04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:29.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:29.075+0000 7f282d79d700 1 --2- 172.21.15.189:0/3840696640 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f281406ae90 0x7f281406d350 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:29.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:29.075+0000 7f282d79d700 1 --2- 172.21.15.189:0/3840696640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28280b23d0 0x7f282814fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:29.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:29.075+0000 7f282d79d700 1 --2- 172.21.15.189:0/3840696640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28280b3490 0x7f28281512e0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:29.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:29.075+0000 7f282d79d700 1 -- 172.21.15.189:0/3840696640 >> 172.21.15.189:0/3840696640 conn(0x7f28280b5470 msgr2=0x7f282815ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:29.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:29.076+0000 7f282d79d700 1 -- 172.21.15.189:0/3840696640 shutdown_connections 2024-04-06T12:24:29.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:29.076+0000 7f282d79d700 1 -- 172.21.15.189:0/3840696640 wait complete. 2024-04-06T12:24:29.085 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2024-04-06T12:24:29.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:29.085 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2024-04-06T12:24:29.102 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:29.103 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:29.103 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:29.103 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:29.103 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:29.103 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:29.103 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:29.103 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:29.103 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:29.103 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:29.103 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:29.103 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:29.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:29.104 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_14_00_UTC) 2024-04-06T12:24:29.166 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:29.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:29.166 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_15_00_UTC) 2024-04-06T12:24:29.229 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:29.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:29.229 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_16_00_UTC) 2024-04-06T12:24:29.291 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:29.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:29.292 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_17_00_UTC) 2024-04-06T12:24:29.356 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:29.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:29.357 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_18_00_UTC) 2024-04-06T12:24:29.422 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:29.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:29.422 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_19_00_UTC) 2024-04-06T12:24:29.489 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:29.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:29.489 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_20_00_UTC) 2024-04-06T12:24:29.555 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:29.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:29.555 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_21_00_UTC) 2024-04-06T12:24:29.618 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:29.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:29.618 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_22_00_UTC) 2024-04-06T12:24:29.681 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:29.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:29.682 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_23_00_UTC) 2024-04-06T12:24:29.745 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:29.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:29.745 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-06-12_24_00_UTC) 2024-04-06T12:24:29.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:29.808 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2024-04-06T12:24:29.869 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir1 1m 2024-04-06T12:24:30.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.036+0000 7f95efd18700 1 Processor -- start 2024-04-06T12:24:30.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.037+0000 7f95efd18700 1 -- start start 2024-04-06T12:24:30.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.037+0000 7f95efd18700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f95e80b3e20 0x7f95e815e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:30.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.037+0000 7f95efd18700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95e80b3990 con 0x7f95e815d7b0 2024-04-06T12:24:30.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.037+0000 7f95efd18700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95e80b3060 con 0x7f95e806ae40 2024-04-06T12:24:30.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.037+0000 7f95efd18700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95e80b4230 con 0x7f95e80b3e20 2024-04-06T12:24:30.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.037+0000 7f95edab4700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f95e815d7b0 0x7f95e806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36336/0 (socket says 172.21.15.189:36336) 2024-04-06T12:24:30.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.037+0000 7f95edab4700 1 -- 172.21.15.189:0/1769522882 learned_addr learned my addr 172.21.15.189:0/1769522882 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:30.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.037+0000 7f95ee2b5700 1 --2- 172.21.15.189:0/1769522882 >> v2:172.21.15.189:3300/0 conn(0x7f95e80b3e20 0x7f95e815e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:30.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.038+0000 7f95ee2b5700 1 -- 172.21.15.189:0/1769522882 >> v1:172.21.15.192:6789/0 conn(0x7f95e806ae40 legacy=0x7f95e815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:30.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.038+0000 7f95ee2b5700 1 -- 172.21.15.189:0/1769522882 >> v1:172.21.15.189:6789/0 conn(0x7f95e815d7b0 legacy=0x7f95e806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:30.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.038+0000 7f95ee2b5700 1 -- 172.21.15.189:0/1769522882 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e80b3340 con 0x7f95e80b3e20 2024-04-06T12:24:30.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.038+0000 7f95ecab2700 1 -- 172.21.15.189:0/1769522882 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62800698 0 0) 0x7f95e80b3990 con 0x7f95e815d7b0 2024-04-06T12:24:30.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.038+0000 7f95ee2b5700 1 --2- 172.21.15.189:0/1769522882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95e80b3e20 0x7f95e815e500 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f95e0009d90 tx=0x7f95e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26dc140dd9e2249 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:30.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.038+0000 7f95ecab2700 1 -- 172.21.15.189:0/1769522882 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e000e070 con 0x7f95e80b3e20 2024-04-06T12:24:30.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.038+0000 7f95ecab2700 1 -- 172.21.15.189:0/1769522882 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e000bb90 con 0x7f95e80b3e20 2024-04-06T12:24:30.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.038+0000 7f95ecab2700 1 -- 172.21.15.189:0/1769522882 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e0003a20 con 0x7f95e80b3e20 2024-04-06T12:24:30.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.039+0000 7f95efd18700 1 -- 172.21.15.189:0/1769522882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95e80b3e20 msgr2=0x7f95e815e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:30.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.039+0000 7f95efd18700 1 --2- 172.21.15.189:0/1769522882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95e80b3e20 0x7f95e815e500 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f95e0009d90 tx=0x7f95e000b330 comp rx=0 tx=0).stop 2024-04-06T12:24:30.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.039+0000 7f95efd18700 1 -- 172.21.15.189:0/1769522882 shutdown_connections 2024-04-06T12:24:30.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.039+0000 7f95efd18700 1 --2- 172.21.15.189:0/1769522882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95e80b3e20 0x7f95e815e500 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:30.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.039+0000 7f95efd18700 1 -- 172.21.15.189:0/1769522882 >> 172.21.15.189:0/1769522882 conn(0x7f95e80d00b0 msgr2=0x7f95e80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:30.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.039+0000 7f95efd18700 1 -- 172.21.15.189:0/1769522882 shutdown_connections 2024-04-06T12:24:30.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.039+0000 7f95efd18700 1 -- 172.21.15.189:0/1769522882 wait complete. 2024-04-06T12:24:30.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.040+0000 7f95efd18700 1 Processor -- start 2024-04-06T12:24:30.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.040+0000 7f95efd18700 1 -- start start 2024-04-06T12:24:30.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95efd18700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95e806ae40 0x7f95e8160550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:30.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95efd18700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95e80b3e20 0x7f95e80cf170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:30.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95efd18700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95e815d7b0 0x7f95e80aff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:30.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95efd18700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e80cdff0 con 0x7f95e806ae40 2024-04-06T12:24:30.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95efd18700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e80ce890 con 0x7f95e80b3e20 2024-04-06T12:24:30.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95efd18700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e80cf6b0 con 0x7f95e815d7b0 2024-04-06T12:24:30.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95edab4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95e806ae40 0x7f95e8160550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:30.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95edab4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95e806ae40 0x7f95e8160550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60110/0 (socket says 172.21.15.189:60110) 2024-04-06T12:24:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95edab4700 1 -- 172.21.15.189:0/1539581966 learned_addr learned my addr 172.21.15.189:0/1539581966 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95ed2b3700 1 --2- 172.21.15.189:0/1539581966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95e80b3e20 0x7f95e80cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95ee2b5700 1 --2- 172.21.15.189:0/1539581966 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95e815d7b0 0x7f95e80aff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95edab4700 1 -- 172.21.15.189:0/1539581966 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95e815d7b0 msgr2=0x7f95e80aff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95edab4700 1 --2- 172.21.15.189:0/1539581966 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95e815d7b0 0x7f95e80aff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95edab4700 1 -- 172.21.15.189:0/1539581966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95e80b3e20 msgr2=0x7f95e80cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95edab4700 1 --2- 172.21.15.189:0/1539581966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95e80b3e20 0x7f95e80cf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95edab4700 1 -- 172.21.15.189:0/1539581966 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e0009a70 con 0x7f95e806ae40 2024-04-06T12:24:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95ee2b5700 1 --2- 172.21.15.189:0/1539581966 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95e815d7b0 0x7f95e80aff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:24:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95ed2b3700 1 --2- 172.21.15.189:0/1539581966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95e80b3e20 0x7f95e80cf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:24:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95edab4700 1 --2- 172.21.15.189:0/1539581966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95e806ae40 0x7f95e8160550 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f95d800c580 tx=0x7f95d800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95de7fc700 1 -- 172.21.15.189:0/1539581966 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d8004c80 con 0x7f95e806ae40 2024-04-06T12:24:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95de7fc700 1 -- 172.21.15.189:0/1539581966 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d8004e00 con 0x7f95e806ae40 2024-04-06T12:24:30.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95efd18700 1 -- 172.21.15.189:0/1539581966 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e8160ba0 con 0x7f95e806ae40 2024-04-06T12:24:30.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95de7fc700 1 -- 172.21.15.189:0/1539581966 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d8007310 con 0x7f95e806ae40 2024-04-06T12:24:30.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.041+0000 7f95efd18700 1 -- 172.21.15.189:0/1539581966 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e80ce520 con 0x7f95e806ae40 2024-04-06T12:24:30.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.042+0000 7f95efd18700 1 -- 172.21.15.189:0/1539581966 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95e8067bb0 con 0x7f95e806ae40 2024-04-06T12:24:30.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.043+0000 7f95de7fc700 1 -- 172.21.15.189:0/1539581966 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f95d8007470 con 0x7f95e806ae40 2024-04-06T12:24:30.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.043+0000 7f95de7fc700 1 --2- 172.21.15.189:0/1539581966 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f95cc06aec0 0x7f95cc06d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:30.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.043+0000 7f95de7fc700 1 -- 172.21.15.189:0/1539581966 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f95d8089590 con 0x7f95e806ae40 2024-04-06T12:24:30.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.043+0000 7f95ed2b3700 1 --2- 172.21.15.189:0/1539581966 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f95cc06aec0 0x7f95cc06d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:30.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.044+0000 7f95ed2b3700 1 --2- 172.21.15.189:0/1539581966 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f95cc06aec0 0x7f95cc06d380 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f95e81f6fa0 tx=0x7f95e4009040 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:30.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.048+0000 7f95de7fc700 1 -- 172.21.15.189:0/1539581966 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f95d8054320 con 0x7f95e806ae40 2024-04-06T12:24:30.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.266+0000 7f95efd18700 1 -- 172.21.15.189:0/1539581966 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f95e8152330 con 0x7f95cc06aec0 2024-04-06T12:24:30.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.349+0000 7f95de7fc700 1 -- 172.21.15.189:0/1539581966 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f95e8152330 con 0x7f95cc06aec0 2024-04-06T12:24:30.346 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule removed for path /snap_test_dir1/dir1 2024-04-06T12:24:30.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.351+0000 7f95efd18700 1 -- 172.21.15.189:0/1539581966 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f95cc06aec0 msgr2=0x7f95cc06d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:30.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.351+0000 7f95efd18700 1 --2- 172.21.15.189:0/1539581966 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f95cc06aec0 0x7f95cc06d380 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f95e81f6fa0 tx=0x7f95e4009040 comp rx=0 tx=0).stop 2024-04-06T12:24:30.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.351+0000 7f95efd18700 1 -- 172.21.15.189:0/1539581966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95e806ae40 msgr2=0x7f95e8160550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:30.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.351+0000 7f95efd18700 1 --2- 172.21.15.189:0/1539581966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95e806ae40 0x7f95e8160550 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f95d800c580 tx=0x7f95d800c940 comp rx=0 tx=0).stop 2024-04-06T12:24:30.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.351+0000 7f95efd18700 1 -- 172.21.15.189:0/1539581966 shutdown_connections 2024-04-06T12:24:30.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.351+0000 7f95efd18700 1 --2- 172.21.15.189:0/1539581966 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95e815d7b0 0x7f95e80aff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:30.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.351+0000 7f95efd18700 1 --2- 172.21.15.189:0/1539581966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95e806ae40 0x7f95e8160550 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:30.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.351+0000 7f95efd18700 1 --2- 172.21.15.189:0/1539581966 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f95cc06aec0 0x7f95cc06d380 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:30.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.351+0000 7f95efd18700 1 --2- 172.21.15.189:0/1539581966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95e80b3e20 0x7f95e80cf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:30.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.351+0000 7f95efd18700 1 -- 172.21.15.189:0/1539581966 >> 172.21.15.189:0/1539581966 conn(0x7f95e80d00b0 msgr2=0x7f95e80adef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:30.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.351+0000 7f95efd18700 1 -- 172.21.15.189:0/1539581966 shutdown_connections 2024-04-06T12:24:30.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:30.351+0000 7f95efd18700 1 -- 172.21.15.189:0/1539581966 wait complete. 2024-04-06T12:24:30.361 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2024-04-06T12:24:30.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:30.361 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2024-04-06T12:24:30.379 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:30.379 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:30.379 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:30.379 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:30.379 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:30.380 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:30.380 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:30.380 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:30.380 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:30.380 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:30.380 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:30.380 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:30.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:30.380 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_14_00_UTC) 2024-04-06T12:24:30.443 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:30.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:30.443 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_15_00_UTC) 2024-04-06T12:24:30.506 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:30.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:30.506 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_16_00_UTC) 2024-04-06T12:24:30.569 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:30.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:30.569 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_17_00_UTC) 2024-04-06T12:24:30.632 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:30.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:30.632 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_18_00_UTC) 2024-04-06T12:24:30.695 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:30.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:30.695 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_19_00_UTC) 2024-04-06T12:24:30.758 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:30.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:30.758 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_20_00_UTC) 2024-04-06T12:24:30.827 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:30.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:30.827 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_21_00_UTC) 2024-04-06T12:24:30.889 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:30.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:30.890 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_22_00_UTC) 2024-04-06T12:24:30.953 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:30.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:30.953 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_23_00_UTC) 2024-04-06T12:24:31.016 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:31.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:31.016 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-06-12_24_00_UTC) 2024-04-06T12:24:31.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:31.080 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2024-04-06T12:24:31.139 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir2 1m 2024-04-06T12:24:31.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.308+0000 7f1bf0774700 1 Processor -- start 2024-04-06T12:24:31.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.308+0000 7f1bf0774700 1 -- start start 2024-04-06T12:24:31.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.309+0000 7f1bf0774700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1be806b8c0 0x7f1be8168420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:31.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.309+0000 7f1bf0774700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1be806bcd0 con 0x7f1be815d7c0 2024-04-06T12:24:31.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.309+0000 7f1bf0774700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1be806ceb0 con 0x7f1be80cd3e0 2024-04-06T12:24:31.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.309+0000 7f1bf0774700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1be80cdfa0 con 0x7f1be806b8c0 2024-04-06T12:24:31.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.309+0000 7f1beed11700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1be806b8c0 0x7f1be8168420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:31.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.309+0000 7f1bedd0f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1be815d7c0 0x7f1be806b1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36338/0 (socket says 172.21.15.189:36338) 2024-04-06T12:24:31.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.309+0000 7f1bedd0f700 1 -- 172.21.15.189:0/1283036119 learned_addr learned my addr 172.21.15.189:0/1283036119 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:31.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.309+0000 7f1beed11700 1 --2- 172.21.15.189:0/1283036119 >> v2:172.21.15.189:3300/0 conn(0x7f1be806b8c0 0x7f1be8168420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60114/0 (socket says 172.21.15.189:60114) 2024-04-06T12:24:31.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.309+0000 7f1beed11700 1 -- 172.21.15.189:0/1283036119 >> v1:172.21.15.192:6789/0 conn(0x7f1be80cd3e0 legacy=0x7f1be80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:31.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.309+0000 7f1beed11700 1 -- 172.21.15.189:0/1283036119 >> v1:172.21.15.189:6789/0 conn(0x7f1be815d7c0 legacy=0x7f1be806b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:31.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.309+0000 7f1beed11700 1 -- 172.21.15.189:0/1283036119 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be806d750 con 0x7f1be806b8c0 2024-04-06T12:24:31.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.309+0000 7f1bed50e700 1 -- 172.21.15.189:0/1283036119 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173958238 0 0) 0x7f1be806bcd0 con 0x7f1be815d7c0 2024-04-06T12:24:31.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.309+0000 7f1beed11700 1 --2- 172.21.15.189:0/1283036119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be806b8c0 0x7f1be8168420 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1be0009d90 tx=0x7f1be000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abfb4dbacf6da502 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:31.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.310+0000 7f1bed50e700 1 -- 172.21.15.189:0/1283036119 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be000e070 con 0x7f1be806b8c0 2024-04-06T12:24:31.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.310+0000 7f1bed50e700 1 -- 172.21.15.189:0/1283036119 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be000bb90 con 0x7f1be806b8c0 2024-04-06T12:24:31.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.310+0000 7f1bed50e700 1 -- 172.21.15.189:0/1283036119 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be0003a20 con 0x7f1be806b8c0 2024-04-06T12:24:31.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.310+0000 7f1bf0774700 1 -- 172.21.15.189:0/1283036119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be806b8c0 msgr2=0x7f1be8168420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:31.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.310+0000 7f1bf0774700 1 --2- 172.21.15.189:0/1283036119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be806b8c0 0x7f1be8168420 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1be0009d90 tx=0x7f1be000b330 comp rx=0 tx=0).stop 2024-04-06T12:24:31.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.310+0000 7f1bf0774700 1 -- 172.21.15.189:0/1283036119 shutdown_connections 2024-04-06T12:24:31.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.310+0000 7f1bf0774700 1 --2- 172.21.15.189:0/1283036119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be806b8c0 0x7f1be8168420 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:31.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.310+0000 7f1bf0774700 1 -- 172.21.15.189:0/1283036119 >> 172.21.15.189:0/1283036119 conn(0x7f1be80cc290 msgr2=0x7f1be80cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:31.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.310+0000 7f1bf0774700 1 -- 172.21.15.189:0/1283036119 shutdown_connections 2024-04-06T12:24:31.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.310+0000 7f1bf0774700 1 -- 172.21.15.189:0/1283036119 wait complete. 2024-04-06T12:24:31.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.311+0000 7f1bf0774700 1 Processor -- start 2024-04-06T12:24:31.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.311+0000 7f1bf0774700 1 -- start start 2024-04-06T12:24:31.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.312+0000 7f1bf0774700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be806b8c0 0x7f1be81f36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:31.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.312+0000 7f1bf0774700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1be80cd3e0 0x7f1be80b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:31.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.312+0000 7f1bf0774700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1be815d7c0 0x7f1be8157d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.312+0000 7f1bf0774700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be80c5310 con 0x7f1be806b8c0 2024-04-06T12:24:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.312+0000 7f1bf0774700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be81f2fe0 con 0x7f1be815d7c0 2024-04-06T12:24:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.312+0000 7f1bf0774700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be81f3c00 con 0x7f1be80cd3e0 2024-04-06T12:24:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.312+0000 7f1bee510700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be806b8c0 0x7f1be81f36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.312+0000 7f1bee510700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be806b8c0 0x7f1be81f36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60130/0 (socket says 172.21.15.189:60130) 2024-04-06T12:24:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.312+0000 7f1bee510700 1 -- 172.21.15.189:0/908509660 learned_addr learned my addr 172.21.15.189:0/908509660 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.312+0000 7f1beed11700 1 --2- 172.21.15.189:0/908509660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1be815d7c0 0x7f1be8157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.312+0000 7f1bedd0f700 1 --2- 172.21.15.189:0/908509660 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1be80cd3e0 0x7f1be80b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.313+0000 7f1bee510700 1 -- 172.21.15.189:0/908509660 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1be80cd3e0 msgr2=0x7f1be80b22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.313+0000 7f1bee510700 1 --2- 172.21.15.189:0/908509660 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1be80cd3e0 0x7f1be80b22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.313+0000 7f1bee510700 1 -- 172.21.15.189:0/908509660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1be815d7c0 msgr2=0x7f1be8157d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.313+0000 7f1bee510700 1 --2- 172.21.15.189:0/908509660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1be815d7c0 0x7f1be8157d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.313+0000 7f1bee510700 1 -- 172.21.15.189:0/908509660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be0009a70 con 0x7f1be806b8c0 2024-04-06T12:24:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.313+0000 7f1beed11700 1 --2- 172.21.15.189:0/908509660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1be815d7c0 0x7f1be8157d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:24:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.313+0000 7f1bee510700 1 --2- 172.21.15.189:0/908509660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be806b8c0 0x7f1be81f36c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f1be400c4d0 tx=0x7f1be400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.313+0000 7f1bdeffd700 1 -- 172.21.15.189:0/908509660 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be40054e0 con 0x7f1be806b8c0 2024-04-06T12:24:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.313+0000 7f1bdeffd700 1 -- 172.21.15.189:0/908509660 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be4003680 con 0x7f1be806b8c0 2024-04-06T12:24:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.313+0000 7f1bf0774700 1 -- 172.21.15.189:0/908509660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be80c4270 con 0x7f1be806b8c0 2024-04-06T12:24:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.313+0000 7f1bdeffd700 1 -- 172.21.15.189:0/908509660 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be4003a30 con 0x7f1be806b8c0 2024-04-06T12:24:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.313+0000 7f1bf0774700 1 -- 172.21.15.189:0/908509660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be80c4c50 con 0x7f1be806b8c0 2024-04-06T12:24:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.314+0000 7f1bf0774700 1 -- 172.21.15.189:0/908509660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1be8074990 con 0x7f1be806b8c0 2024-04-06T12:24:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.314+0000 7f1bdeffd700 1 -- 172.21.15.189:0/908509660 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f1be4003b90 con 0x7f1be806b8c0 2024-04-06T12:24:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.315+0000 7f1bdeffd700 1 --2- 172.21.15.189:0/908509660 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1bcc06ae90 0x7f1bcc06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.315+0000 7f1bdeffd700 1 -- 172.21.15.189:0/908509660 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f1be4088e20 con 0x7f1be806b8c0 2024-04-06T12:24:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.315+0000 7f1bedd0f700 1 --2- 172.21.15.189:0/908509660 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1bcc06ae90 0x7f1bcc06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.315+0000 7f1bedd0f700 1 --2- 172.21.15.189:0/908509660 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1bcc06ae90 0x7f1bcc06d350 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1be81561e0 tx=0x7f1bd8007e70 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:31.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.319+0000 7f1bdeffd700 1 -- 172.21.15.189:0/908509660 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1be4053a90 con 0x7f1be806b8c0 2024-04-06T12:24:31.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.541+0000 7f1bf0774700 1 -- 172.21.15.189:0/908509660 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1be806d3e0 con 0x7f1bcc06ae90 2024-04-06T12:24:31.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.620+0000 7f1bdeffd700 1 -- 172.21.15.189:0/908509660 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f1bd8004ab0 con 0x7f1bcc06ae90 2024-04-06T12:24:31.618 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule removed for path /snap_test_dir1/dir2 2024-04-06T12:24:31.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.622+0000 7f1bf0774700 1 -- 172.21.15.189:0/908509660 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1bcc06ae90 msgr2=0x7f1bcc06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.622+0000 7f1bf0774700 1 --2- 172.21.15.189:0/908509660 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1bcc06ae90 0x7f1bcc06d350 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1be81561e0 tx=0x7f1bd8007e70 comp rx=0 tx=0).stop 2024-04-06T12:24:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.623+0000 7f1bf0774700 1 -- 172.21.15.189:0/908509660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be806b8c0 msgr2=0x7f1be81f36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.623+0000 7f1bf0774700 1 --2- 172.21.15.189:0/908509660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be806b8c0 0x7f1be81f36c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f1be400c4d0 tx=0x7f1be400c7e0 comp rx=0 tx=0).stop 2024-04-06T12:24:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.623+0000 7f1bf0774700 1 -- 172.21.15.189:0/908509660 shutdown_connections 2024-04-06T12:24:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.623+0000 7f1bf0774700 1 --2- 172.21.15.189:0/908509660 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1bcc06ae90 0x7f1bcc06d350 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.623+0000 7f1bf0774700 1 --2- 172.21.15.189:0/908509660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1be815d7c0 0x7f1be8157d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.623+0000 7f1bf0774700 1 --2- 172.21.15.189:0/908509660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be806b8c0 0x7f1be81f36c0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.623+0000 7f1bf0774700 1 --2- 172.21.15.189:0/908509660 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1be80cd3e0 0x7f1be80b22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.623+0000 7f1bf0774700 1 -- 172.21.15.189:0/908509660 >> 172.21.15.189:0/908509660 conn(0x7f1be80cc290 msgr2=0x7f1be80b08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.623+0000 7f1bf0774700 1 -- 172.21.15.189:0/908509660 shutdown_connections 2024-04-06T12:24:31.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:31.623+0000 7f1bf0774700 1 -- 172.21.15.189:0/908509660 wait complete. 2024-04-06T12:24:31.632 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2024-04-06T12:24:31.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:31.632 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2024-04-06T12:24:31.651 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:31.651 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:31.651 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:31.651 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:31.651 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:31.651 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:31.652 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:31.652 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:31.652 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:31.652 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:31.652 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:31.652 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:31.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:31.652 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_14_00_UTC) 2024-04-06T12:24:31.716 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:31.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:31.717 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_15_00_UTC) 2024-04-06T12:24:31.780 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:31.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:31.780 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_16_00_UTC) 2024-04-06T12:24:31.843 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:31.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:31.844 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_17_00_UTC) 2024-04-06T12:24:31.908 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:31.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:31.908 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_18_00_UTC) 2024-04-06T12:24:31.971 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:31.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:31.971 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_19_00_UTC) 2024-04-06T12:24:32.034 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:32.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:32.034 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_20_00_UTC) 2024-04-06T12:24:32.101 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:32.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:32.101 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_21_00_UTC) 2024-04-06T12:24:32.164 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:32.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:32.165 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_22_00_UTC) 2024-04-06T12:24:32.228 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:32.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:32.228 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_23_00_UTC) 2024-04-06T12:24:32.290 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:32.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:32.291 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-06-12_24_00_UTC) 2024-04-06T12:24:32.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:32.354 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2024-04-06T12:24:32.413 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir3 1m 2024-04-06T12:24:32.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.581+0000 7f9ba6f7a700 1 Processor -- start 2024-04-06T12:24:32.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.581+0000 7f9ba6f7a700 1 -- start start 2024-04-06T12:24:32.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.582+0000 7f9ba6f7a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ba00aec70 0x7f9ba00cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.582+0000 7f9ba6f7a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba015a830 con 0x7f9ba00cea80 2024-04-06T12:24:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.582+0000 7f9ba6f7a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba00cc050 con 0x7f9ba00cdae0 2024-04-06T12:24:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.582+0000 7f9ba6f7a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba00cd920 con 0x7f9ba00aec70 2024-04-06T12:24:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.582+0000 7f9ba5517700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9ba00cea80 0x7f9ba006ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36348/0 (socket says 172.21.15.189:36348) 2024-04-06T12:24:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.582+0000 7f9ba5517700 1 -- 172.21.15.189:0/4268270971 learned_addr learned my addr 172.21.15.189:0/4268270971 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9ba4d16700 1 --2- 172.21.15.189:0/4268270971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ba00aec70 0x7f9ba00cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9f7fe700 1 -- 172.21.15.189:0/4268270971 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882764039 0 0) 0x7f9ba015a830 con 0x7f9ba00cea80 2024-04-06T12:24:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9f7fe700 1 -- 172.21.15.189:0/4268270971 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b900035b0 con 0x7f9ba00cea80 2024-04-06T12:24:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9f7fe700 1 -- 172.21.15.189:0/4268270971 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 95823322 0 0) 0x7f9b900035b0 con 0x7f9ba00cea80 2024-04-06T12:24:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9f7fe700 1 -- 172.21.15.189:0/4268270971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ba00aec70 msgr2=0x7f9ba00cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9f7fe700 1 --2- 172.21.15.189:0/4268270971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ba00aec70 0x7f9ba00cd3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9f7fe700 1 -- 172.21.15.189:0/4268270971 >> v1:172.21.15.192:6789/0 conn(0x7f9ba00cdae0 legacy=0x7f9ba00ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9f7fe700 1 -- 172.21.15.189:0/4268270971 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba00666f0 con 0x7f9ba00cea80 2024-04-06T12:24:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9f7fe700 1 -- 172.21.15.189:0/4268270971 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f9b980019f0 con 0x7f9ba00cea80 2024-04-06T12:24:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9f7fe700 1 -- 172.21.15.189:0/4268270971 >> v1:172.21.15.189:6789/0 conn(0x7f9ba00cea80 legacy=0x7f9ba006ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9f7fe700 1 --2- 172.21.15.189:0/4268270971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ba00aec70 0x7f9b90003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9f7fe700 1 --2- 172.21.15.189:0/4268270971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b90003f70 0x7f9b90004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9f7fe700 1 --2- 172.21.15.189:0/4268270971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b90004d90 0x7f9b90005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9f7fe700 1 -- 172.21.15.189:0/4268270971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba00cd920 con 0x7f9b90003f70 2024-04-06T12:24:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9f7fe700 1 -- 172.21.15.189:0/4268270971 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9000bb20 con 0x7f9b90004d90 2024-04-06T12:24:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9f7fe700 1 -- 172.21.15.189:0/4268270971 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9000bce0 con 0x7f9ba00aec70 2024-04-06T12:24:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9ffff700 1 --2- 172.21.15.189:0/4268270971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b90003f70 0x7f9b90004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9ba4d16700 1 --2- 172.21.15.189:0/4268270971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ba00aec70 0x7f9b90003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9ba5517700 1 --2- 172.21.15.189:0/4268270971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b90004d90 0x7f9b90005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9ffff700 1 -- 172.21.15.189:0/4268270971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ba00aec70 msgr2=0x7f9b90003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9ffff700 1 --2- 172.21.15.189:0/4268270971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ba00aec70 0x7f9b90003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9ffff700 1 -- 172.21.15.189:0/4268270971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b90004d90 msgr2=0x7f9b90005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9ffff700 1 --2- 172.21.15.189:0/4268270971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b90004d90 0x7f9b90005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9ffff700 1 -- 172.21.15.189:0/4268270971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b900035b0 con 0x7f9b90003f70 2024-04-06T12:24:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9ba4d16700 1 --2- 172.21.15.189:0/4268270971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ba00aec70 0x7f9b90003a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:24:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.583+0000 7f9b9ffff700 1 --2- 172.21.15.189:0/4268270971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b90003f70 0x7f9b90004850 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c0018d0 tx=0x7f9b90005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec0d676118016bc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.584+0000 7f9b9f7fe700 1 -- 172.21.15.189:0/4268270971 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c00c690 con 0x7f9b90003f70 2024-04-06T12:24:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.584+0000 7f9b9f7fe700 1 -- 172.21.15.189:0/4268270971 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8c00c7f0 con 0x7f9b90003f70 2024-04-06T12:24:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.584+0000 7f9b9f7fe700 1 -- 172.21.15.189:0/4268270971 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c00cba0 con 0x7f9b90003f70 2024-04-06T12:24:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.584+0000 7f9ba6f7a700 1 -- 172.21.15.189:0/4268270971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b90003f70 msgr2=0x7f9b90004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.584+0000 7f9ba6f7a700 1 --2- 172.21.15.189:0/4268270971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b90003f70 0x7f9b90004850 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c0018d0 tx=0x7f9b90005880 comp rx=0 tx=0).stop 2024-04-06T12:24:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.584+0000 7f9ba5517700 1 -- 172.21.15.189:0/4268270971 reap_dead start 2024-04-06T12:24:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.584+0000 7f9ba6f7a700 1 -- 172.21.15.189:0/4268270971 shutdown_connections 2024-04-06T12:24:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.584+0000 7f9ba6f7a700 1 -- 172.21.15.189:0/4268270971 >> 172.21.15.189:0/4268270971 conn(0x7f9ba00cc290 msgr2=0x7f9ba00cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.584+0000 7f9ba6f7a700 1 -- 172.21.15.189:0/4268270971 shutdown_connections 2024-04-06T12:24:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.584+0000 7f9ba6f7a700 1 -- 172.21.15.189:0/4268270971 wait complete. 2024-04-06T12:24:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.585+0000 7f9ba6f7a700 1 Processor -- start 2024-04-06T12:24:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.585+0000 7f9ba6f7a700 1 -- start start 2024-04-06T12:24:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.586+0000 7f9ba6f7a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b90003f70 0x7f9ba006b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.586+0000 7f9ba6f7a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ba01f13d0 0x7f9ba00aa680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.586+0000 7f9ba6f7a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ba00aabc0 0x7f9ba00a9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.586+0000 7f9ba6f7a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba00ac590 con 0x7f9ba00aabc0 2024-04-06T12:24:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.586+0000 7f9ba6f7a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba00ac870 con 0x7f9ba01f13d0 2024-04-06T12:24:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.586+0000 7f9ba6f7a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba00acb50 con 0x7f9b90003f70 2024-04-06T12:24:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.586+0000 7f9ba5517700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ba00aabc0 0x7f9ba00a9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.586+0000 7f9ba5517700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ba00aabc0 0x7f9ba00a9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60146/0 (socket says 172.21.15.189:60146) 2024-04-06T12:24:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.586+0000 7f9ba5517700 1 -- 172.21.15.189:0/361801978 learned_addr learned my addr 172.21.15.189:0/361801978 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:32.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.587+0000 7f9b9ffff700 1 --2- 172.21.15.189:0/361801978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ba01f13d0 0x7f9ba00aa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:32.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.587+0000 7f9ba4d16700 1 --2- 172.21.15.189:0/361801978 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b90003f70 0x7f9ba006b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:32.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.587+0000 7f9ba5517700 1 -- 172.21.15.189:0/361801978 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b90003f70 msgr2=0x7f9ba006b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:32.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.587+0000 7f9ba5517700 1 --2- 172.21.15.189:0/361801978 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b90003f70 0x7f9ba006b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:32.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.587+0000 7f9ba5517700 1 -- 172.21.15.189:0/361801978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ba01f13d0 msgr2=0x7f9ba00aa680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:32.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.587+0000 7f9ba5517700 1 --2- 172.21.15.189:0/361801978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ba01f13d0 0x7f9ba00aa680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:32.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.587+0000 7f9ba5517700 1 -- 172.21.15.189:0/361801978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba00ace30 con 0x7f9ba00aabc0 2024-04-06T12:24:32.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.587+0000 7f9b9ffff700 1 --2- 172.21.15.189:0/361801978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ba01f13d0 0x7f9ba00aa680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:24:32.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.587+0000 7f9ba5517700 1 --2- 172.21.15.189:0/361801978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ba00aabc0 0x7f9ba00a9660 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9ba006ef10 tx=0x7f9b9800e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:32.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.587+0000 7f9b9d7fa700 1 -- 172.21.15.189:0/361801978 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9800ec50 con 0x7f9ba00aabc0 2024-04-06T12:24:32.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.587+0000 7f9b9d7fa700 1 -- 172.21.15.189:0/361801978 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9800edb0 con 0x7f9ba00aabc0 2024-04-06T12:24:32.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.587+0000 7f9ba6f7a700 1 -- 172.21.15.189:0/361801978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba00ad110 con 0x7f9ba00aabc0 2024-04-06T12:24:32.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.587+0000 7f9b9d7fa700 1 -- 172.21.15.189:0/361801978 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b98013660 con 0x7f9ba00aabc0 2024-04-06T12:24:32.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.587+0000 7f9ba6f7a700 1 -- 172.21.15.189:0/361801978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba00ad630 con 0x7f9ba00aabc0 2024-04-06T12:24:32.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.588+0000 7f9ba6f7a700 1 -- 172.21.15.189:0/361801978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba00ad300 con 0x7f9ba00aabc0 2024-04-06T12:24:32.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.588+0000 7f9b9d7fa700 1 -- 172.21.15.189:0/361801978 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f9b980137c0 con 0x7f9ba00aabc0 2024-04-06T12:24:32.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.589+0000 7f9b9d7fa700 1 --2- 172.21.15.189:0/361801978 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b8406ae90 0x7f9b8406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:32.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.589+0000 7f9b9d7fa700 1 -- 172.21.15.189:0/361801978 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f9b98089e90 con 0x7f9ba00aabc0 2024-04-06T12:24:32.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.589+0000 7f9ba4d16700 1 --2- 172.21.15.189:0/361801978 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b8406ae90 0x7f9b8406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:32.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.589+0000 7f9ba4d16700 1 --2- 172.21.15.189:0/361801978 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b8406ae90 0x7f9b8406d350 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009a50 tx=0x7f9b94008040 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:32.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.593+0000 7f9b9d7fa700 1 -- 172.21.15.189:0/361801978 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9b98054b90 con 0x7f9ba00aabc0 2024-04-06T12:24:32.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.815+0000 7f9ba6f7a700 1 -- 172.21.15.189:0/361801978 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ba00cbb30 con 0x7f9b8406ae90 2024-04-06T12:24:32.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.888+0000 7f9b9d7fa700 1 -- 172.21.15.189:0/361801978 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f9b940096e0 con 0x7f9b8406ae90 2024-04-06T12:24:32.885 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule removed for path /snap_test_dir1/dir3 2024-04-06T12:24:32.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.890+0000 7f9ba6f7a700 1 -- 172.21.15.189:0/361801978 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b8406ae90 msgr2=0x7f9b8406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:32.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.890+0000 7f9ba6f7a700 1 --2- 172.21.15.189:0/361801978 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b8406ae90 0x7f9b8406d350 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009a50 tx=0x7f9b94008040 comp rx=0 tx=0).stop 2024-04-06T12:24:32.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.890+0000 7f9ba6f7a700 1 -- 172.21.15.189:0/361801978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ba00aabc0 msgr2=0x7f9ba00a9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:32.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.890+0000 7f9ba6f7a700 1 --2- 172.21.15.189:0/361801978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ba00aabc0 0x7f9ba00a9660 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9ba006ef10 tx=0x7f9b9800e440 comp rx=0 tx=0).stop 2024-04-06T12:24:32.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.890+0000 7f9ba6f7a700 1 -- 172.21.15.189:0/361801978 shutdown_connections 2024-04-06T12:24:32.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.890+0000 7f9ba6f7a700 1 --2- 172.21.15.189:0/361801978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ba00aabc0 0x7f9ba00a9660 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:32.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.890+0000 7f9ba6f7a700 1 --2- 172.21.15.189:0/361801978 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b90003f70 0x7f9ba006b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:32.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.890+0000 7f9ba6f7a700 1 --2- 172.21.15.189:0/361801978 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b8406ae90 0x7f9b8406d350 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:32.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.890+0000 7f9ba6f7a700 1 --2- 172.21.15.189:0/361801978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ba01f13d0 0x7f9ba00aa680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:32.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.890+0000 7f9ba6f7a700 1 -- 172.21.15.189:0/361801978 >> 172.21.15.189:0/361801978 conn(0x7f9ba00cc290 msgr2=0x7f9ba00af520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:32.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.890+0000 7f9ba6f7a700 1 -- 172.21.15.189:0/361801978 shutdown_connections 2024-04-06T12:24:32.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:32.890+0000 7f9ba6f7a700 1 -- 172.21.15.189:0/361801978 wait complete. 2024-04-06T12:24:32.899 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2024-04-06T12:24:32.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:32.900 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2024-04-06T12:24:32.918 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:32.918 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:32.918 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:32.919 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:32.919 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:32.919 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:32.919 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:32.919 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:32.919 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:32.919 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:32.919 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:32.919 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:32.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:32.920 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_14_00_UTC) 2024-04-06T12:24:32.983 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:32.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:32.984 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_15_00_UTC) 2024-04-06T12:24:33.047 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:33.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:33.047 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_16_00_UTC) 2024-04-06T12:24:33.111 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:33.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:33.111 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_17_00_UTC) 2024-04-06T12:24:33.178 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:33.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:33.178 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_18_00_UTC) 2024-04-06T12:24:33.241 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:33.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:33.241 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_19_00_UTC) 2024-04-06T12:24:33.304 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:33.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:33.305 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_20_00_UTC) 2024-04-06T12:24:33.369 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:33.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:33.370 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_21_00_UTC) 2024-04-06T12:24:33.434 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:33.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:33.434 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_22_00_UTC) 2024-04-06T12:24:33.497 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:33.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:33.497 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_23_00_UTC) 2024-04-06T12:24:33.561 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:33.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:33.561 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-06-12_24_00_UTC) 2024-04-06T12:24:33.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:33.625 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2024-04-06T12:24:33.684 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir4 1m 2024-04-06T12:24:33.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.853+0000 7fc934e7a700 1 Processor -- start 2024-04-06T12:24:33.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.853+0000 7fc934e7a700 1 -- start start 2024-04-06T12:24:33.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.854+0000 7fc934e7a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc93006ae40 0x7fc9300b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:33.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.854+0000 7fc934e7a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc93015ddf0 0x7fc93015e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:33.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.854+0000 7fc934e7a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9300b3990 con 0x7fc93015d7b0 2024-04-06T12:24:33.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.854+0000 7fc934e7a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9300b3060 con 0x7fc93015ddf0 2024-04-06T12:24:33.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.854+0000 7fc934e7a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9300b3340 con 0x7fc93006ae40 2024-04-06T12:24:33.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.854+0000 7fc92ed9d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc93015ddf0 0x7fc93015e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:33.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.854+0000 7fc92dd9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc93006ae40 0x7fc9300b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:33.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.854+0000 7fc92e59c700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fc93015d7b0 0x7fc93006a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:41306/0 (socket says 172.21.15.189:41306) 2024-04-06T12:24:33.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.854+0000 7fc92e59c700 1 -- 172.21.15.189:0/916224273 learned_addr learned my addr 172.21.15.189:0/916224273 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:33.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.855+0000 7fc92d59a700 1 -- 172.21.15.189:0/916224273 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311135747 0 0) 0x7fc9300b3990 con 0x7fc93015d7b0 2024-04-06T12:24:33.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.855+0000 7fc92d59a700 1 -- 172.21.15.189:0/916224273 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9180035b0 con 0x7fc93015d7b0 2024-04-06T12:24:33.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.855+0000 7fc92dd9b700 1 -- 172.21.15.189:0/916224273 >> v2:172.21.15.192:3300/0 conn(0x7fc93015ddf0 msgr2=0x7fc93015e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:33.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.855+0000 7fc92dd9b700 1 --2- 172.21.15.189:0/916224273 >> v2:172.21.15.192:3300/0 conn(0x7fc93015ddf0 0x7fc93015e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:33.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.855+0000 7fc92dd9b700 1 -- 172.21.15.189:0/916224273 >> v1:172.21.15.192:6789/0 conn(0x7fc93015d7b0 legacy=0x7fc93006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:33.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.855+0000 7fc92dd9b700 1 -- 172.21.15.189:0/916224273 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9300b3620 con 0x7fc93006ae40 2024-04-06T12:24:33.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.855+0000 7fc92ed9d700 1 --2- 172.21.15.189:0/916224273 >> v2:172.21.15.192:3300/0 conn(0x7fc93015ddf0 0x7fc93015e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:33.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.855+0000 7fc92dd9b700 1 --2- 172.21.15.189:0/916224273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc93006ae40 0x7fc9300b3e20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc920009840 tx=0x7fc920009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d2851fb8f9e1398 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:33.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.855+0000 7fc92d59a700 1 -- 172.21.15.189:0/916224273 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc920011830 con 0x7fc93006ae40 2024-04-06T12:24:33.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.855+0000 7fc92d59a700 1 -- 172.21.15.189:0/916224273 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc920011990 con 0x7fc93006ae40 2024-04-06T12:24:33.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.855+0000 7fc92d59a700 1 -- 172.21.15.189:0/916224273 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc920011d00 con 0x7fc93006ae40 2024-04-06T12:24:33.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.856+0000 7fc934e7a700 1 -- 172.21.15.189:0/916224273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc93006ae40 msgr2=0x7fc9300b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:33.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.856+0000 7fc934e7a700 1 --2- 172.21.15.189:0/916224273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc93006ae40 0x7fc9300b3e20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc920009840 tx=0x7fc920009b50 comp rx=0 tx=0).stop 2024-04-06T12:24:33.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.856+0000 7fc934e7a700 1 -- 172.21.15.189:0/916224273 shutdown_connections 2024-04-06T12:24:33.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.856+0000 7fc934e7a700 1 --2- 172.21.15.189:0/916224273 >> v2:172.21.15.192:3300/0 conn(0x7fc93015ddf0 0x7fc93015e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:33.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.856+0000 7fc934e7a700 1 --2- 172.21.15.189:0/916224273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc93006ae40 0x7fc9300b3e20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:33.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.856+0000 7fc934e7a700 1 -- 172.21.15.189:0/916224273 >> 172.21.15.189:0/916224273 conn(0x7fc9300d00b0 msgr2=0x7fc9300d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:33.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.856+0000 7fc934e7a700 1 -- 172.21.15.189:0/916224273 shutdown_connections 2024-04-06T12:24:33.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.856+0000 7fc934e7a700 1 -- 172.21.15.189:0/916224273 wait complete. 2024-04-06T12:24:33.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.857+0000 7fc934e7a700 1 Processor -- start 2024-04-06T12:24:33.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.858+0000 7fc934e7a700 1 -- start start 2024-04-06T12:24:33.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.858+0000 7fc934e7a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc93006ae40 0x7fc9301517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:33.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.858+0000 7fc934e7a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc93015d7b0 0x7fc9301f69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:33.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.858+0000 7fc934e7a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc93015ddf0 0x7fc930160110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:33.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.858+0000 7fc934e7a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9301f6ef0 con 0x7fc93006ae40 2024-04-06T12:24:33.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.858+0000 7fc934e7a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc930157110 con 0x7fc93015ddf0 2024-04-06T12:24:33.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.858+0000 7fc934e7a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc930156500 con 0x7fc93015d7b0 2024-04-06T12:24:33.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.859+0000 7fc92e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc93006ae40 0x7fc9301517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:33.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.859+0000 7fc92e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc93006ae40 0x7fc9301517a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60158/0 (socket says 172.21.15.189:60158) 2024-04-06T12:24:33.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.859+0000 7fc92e59c700 1 -- 172.21.15.189:0/1966544278 learned_addr learned my addr 172.21.15.189:0/1966544278 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:33.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.859+0000 7fc92dd9b700 1 --2- 172.21.15.189:0/1966544278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc93015d7b0 0x7fc9301f69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:33.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.859+0000 7fc92ed9d700 1 --2- 172.21.15.189:0/1966544278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc93015ddf0 0x7fc930160110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:33.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.859+0000 7fc92e59c700 1 -- 172.21.15.189:0/1966544278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc93015d7b0 msgr2=0x7fc9301f69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:33.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.859+0000 7fc92e59c700 1 --2- 172.21.15.189:0/1966544278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc93015d7b0 0x7fc9301f69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:33.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.859+0000 7fc92e59c700 1 -- 172.21.15.189:0/1966544278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc93015ddf0 msgr2=0x7fc930160110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:33.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.859+0000 7fc92e59c700 1 --2- 172.21.15.189:0/1966544278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc93015ddf0 0x7fc930160110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:33.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.859+0000 7fc92e59c700 1 -- 172.21.15.189:0/1966544278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc928009a70 con 0x7fc93006ae40 2024-04-06T12:24:33.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.859+0000 7fc92ed9d700 1 --2- 172.21.15.189:0/1966544278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc93015ddf0 0x7fc930160110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:24:33.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.859+0000 7fc92e59c700 1 --2- 172.21.15.189:0/1966544278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc93006ae40 0x7fc9301517a0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc92400c500 tx=0x7fc92400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:33.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.859+0000 7fc91effd700 1 -- 172.21.15.189:0/1966544278 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc924005d70 con 0x7fc93006ae40 2024-04-06T12:24:33.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.859+0000 7fc91effd700 1 -- 172.21.15.189:0/1966544278 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9240070d0 con 0x7fc93006ae40 2024-04-06T12:24:33.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.859+0000 7fc91effd700 1 -- 172.21.15.189:0/1966544278 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc924007440 con 0x7fc93006ae40 2024-04-06T12:24:33.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.859+0000 7fc934e7a700 1 -- 172.21.15.189:0/1966544278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9200094a0 con 0x7fc93006ae40 2024-04-06T12:24:33.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.859+0000 7fc934e7a700 1 -- 172.21.15.189:0/1966544278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc930156ac0 con 0x7fc93006ae40 2024-04-06T12:24:33.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.860+0000 7fc934e7a700 1 -- 172.21.15.189:0/1966544278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc930067bb0 con 0x7fc93006ae40 2024-04-06T12:24:33.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.861+0000 7fc91effd700 1 -- 172.21.15.189:0/1966544278 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fc9240075a0 con 0x7fc93006ae40 2024-04-06T12:24:33.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.861+0000 7fc91effd700 1 --2- 172.21.15.189:0/1966544278 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc91006ae90 0x7fc91006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:33.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.861+0000 7fc91effd700 1 -- 172.21.15.189:0/1966544278 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fc924088ca0 con 0x7fc93006ae40 2024-04-06T12:24:33.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.861+0000 7fc92dd9b700 1 --2- 172.21.15.189:0/1966544278 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc91006ae90 0x7fc91006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:33.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.861+0000 7fc92dd9b700 1 --2- 172.21.15.189:0/1966544278 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc91006ae90 0x7fc91006d350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc920009840 tx=0x7fc920008030 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:33.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:33.865+0000 7fc91effd700 1 -- 172.21.15.189:0/1966544278 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc924053a50 con 0x7fc93006ae40 2024-04-06T12:24:34.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:34.088+0000 7fc934e7a700 1 -- 172.21.15.189:0/1966544278 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc93015e7e0 con 0x7fc91006ae90 2024-04-06T12:24:34.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:34.167+0000 7fc91effd700 1 -- 172.21.15.189:0/1966544278 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fc920011630 con 0x7fc91006ae90 2024-04-06T12:24:34.165 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule removed for path /snap_test_dir1/dir4 2024-04-06T12:24:34.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:34.170+0000 7fc934e7a700 1 -- 172.21.15.189:0/1966544278 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc91006ae90 msgr2=0x7fc91006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:34.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:34.170+0000 7fc934e7a700 1 --2- 172.21.15.189:0/1966544278 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc91006ae90 0x7fc91006d350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc920009840 tx=0x7fc920008030 comp rx=0 tx=0).stop 2024-04-06T12:24:34.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:34.170+0000 7fc934e7a700 1 -- 172.21.15.189:0/1966544278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc93006ae40 msgr2=0x7fc9301517a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:34.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:34.170+0000 7fc934e7a700 1 --2- 172.21.15.189:0/1966544278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc93006ae40 0x7fc9301517a0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc92400c500 tx=0x7fc92400c810 comp rx=0 tx=0).stop 2024-04-06T12:24:34.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:34.170+0000 7fc934e7a700 1 -- 172.21.15.189:0/1966544278 shutdown_connections 2024-04-06T12:24:34.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:34.170+0000 7fc934e7a700 1 --2- 172.21.15.189:0/1966544278 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc91006ae90 0x7fc91006d350 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:34.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:34.170+0000 7fc934e7a700 1 --2- 172.21.15.189:0/1966544278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc93015ddf0 0x7fc930160110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:34.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:34.170+0000 7fc934e7a700 1 --2- 172.21.15.189:0/1966544278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc93006ae40 0x7fc9301517a0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:34.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:34.170+0000 7fc934e7a700 1 --2- 172.21.15.189:0/1966544278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc93015d7b0 0x7fc9301f69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:34.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:34.170+0000 7fc934e7a700 1 -- 172.21.15.189:0/1966544278 >> 172.21.15.189:0/1966544278 conn(0x7fc9300d00b0 msgr2=0x7fc9300ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:34.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:34.170+0000 7fc934e7a700 1 -- 172.21.15.189:0/1966544278 shutdown_connections 2024-04-06T12:24:34.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:34.170+0000 7fc934e7a700 1 -- 172.21.15.189:0/1966544278 wait complete. 2024-04-06T12:24:34.179 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2024-04-06T12:24:34.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:34.179 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2024-04-06T12:24:34.197 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:34.197 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:34.197 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:34.197 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:34.197 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:34.197 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_01_UTC 2024-04-06T12:24:34.197 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:34.197 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:34.197 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:34.197 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:34.198 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:34.198 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:34.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:34.198 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_14_00_UTC) 2024-04-06T12:24:34.260 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:34.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:34.261 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_15_00_UTC) 2024-04-06T12:24:34.325 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:34.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:34.326 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_16_00_UTC) 2024-04-06T12:24:34.389 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:34.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:34.389 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_17_00_UTC) 2024-04-06T12:24:34.452 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:34.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:34.452 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_18_00_UTC) 2024-04-06T12:24:34.515 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_19_01_UTC 2024-04-06T12:24:34.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:34.515 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_19_01_UTC) 2024-04-06T12:24:34.618 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:34.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:34.618 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_20_00_UTC) 2024-04-06T12:24:34.674 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:34.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:34.674 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_21_00_UTC) 2024-04-06T12:24:34.771 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:34.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:34.771 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_22_00_UTC) 2024-04-06T12:24:34.834 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:34.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:34.834 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_23_00_UTC) 2024-04-06T12:24:34.891 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:34.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:34.892 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-06-12_24_00_UTC) 2024-04-06T12:24:34.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:34.927 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2024-04-06T12:24:34.986 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir5 1m 2024-04-06T12:24:35.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.155+0000 7f8d53953700 1 Processor -- start 2024-04-06T12:24:35.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.155+0000 7f8d53953700 1 -- start start 2024-04-06T12:24:35.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.156+0000 7f8d53953700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d4c15a550 0x7f8d4c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:35.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.156+0000 7f8d53953700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8d4c0cf900 0x7f8d4c0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:35.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.156+0000 7f8d53953700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d4c0cdbe0 con 0x7f8d4c0b4ba0 2024-04-06T12:24:35.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.156+0000 7f8d53953700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d4c0b5010 con 0x7f8d4c0cf900 2024-04-06T12:24:35.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.156+0000 7f8d53953700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c15a940 con 0x7f8d4c15a550 2024-04-06T12:24:35.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.156+0000 7f8d50eee700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8d4c0cf900 0x7f8d4c0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:35.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.156+0000 7f8d51ef0700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8d4c0b4ba0 0x7f8d4c06eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36356/0 (socket says 172.21.15.189:36356) 2024-04-06T12:24:35.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.156+0000 7f8d51ef0700 1 -- 172.21.15.189:0/290415089 learned_addr learned my addr 172.21.15.189:0/290415089 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:35.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.156+0000 7f8d50eee700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8d4c0cf900 0x7f8d4c0b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60166/0 (socket says 172.21.15.189:60166) 2024-04-06T12:24:35.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.156+0000 7f8d516ef700 1 --2- 172.21.15.189:0/290415089 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d4c15a550 0x7f8d4c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:35.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.156+0000 7f8d50eee700 1 -- 172.21.15.189:0/290415089 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d4c15a550 msgr2=0x7f8d4c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:35.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.156+0000 7f8d50eee700 1 --2- 172.21.15.189:0/290415089 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d4c15a550 0x7f8d4c0cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:35.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.156+0000 7f8d50eee700 1 -- 172.21.15.189:0/290415089 >> v1:172.21.15.189:6789/0 conn(0x7f8d4c0b4ba0 legacy=0x7f8d4c06eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:35.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.156+0000 7f8d50eee700 1 -- 172.21.15.189:0/290415089 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4c15ac00 con 0x7f8d4c0cf900 2024-04-06T12:24:35.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.156+0000 7f8d43fff700 1 -- 172.21.15.189:0/290415089 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221532338 0 0) 0x7f8d4c0cdbe0 con 0x7f8d4c0b4ba0 2024-04-06T12:24:35.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.156+0000 7f8d50eee700 1 --2- 172.21.15.189:0/290415089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d4c0cf900 0x7f8d4c0b4660 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f8d4800bae0 tx=0x7f8d4800bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341b51a30aacbadd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:35.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.157+0000 7f8d43fff700 1 -- 172.21.15.189:0/290415089 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4800dd30 con 0x7f8d4c0cf900 2024-04-06T12:24:35.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.157+0000 7f8d43fff700 1 -- 172.21.15.189:0/290415089 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4800de90 con 0x7f8d4c0cf900 2024-04-06T12:24:35.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.157+0000 7f8d43fff700 1 -- 172.21.15.189:0/290415089 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d48010690 con 0x7f8d4c0cf900 2024-04-06T12:24:35.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.157+0000 7f8d53953700 1 -- 172.21.15.189:0/290415089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d4c0cf900 msgr2=0x7f8d4c0b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:35.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.157+0000 7f8d53953700 1 --2- 172.21.15.189:0/290415089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d4c0cf900 0x7f8d4c0b4660 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f8d4800bae0 tx=0x7f8d4800bdf0 comp rx=0 tx=0).stop 2024-04-06T12:24:35.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.157+0000 7f8d53953700 1 -- 172.21.15.189:0/290415089 shutdown_connections 2024-04-06T12:24:35.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.157+0000 7f8d53953700 1 --2- 172.21.15.189:0/290415089 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d4c15a550 0x7f8d4c0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:35.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.157+0000 7f8d53953700 1 --2- 172.21.15.189:0/290415089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d4c0cf900 0x7f8d4c0b4660 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:35.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.157+0000 7f8d53953700 1 -- 172.21.15.189:0/290415089 >> 172.21.15.189:0/290415089 conn(0x7f8d4c0d00b0 msgr2=0x7f8d4c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:35.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.157+0000 7f8d53953700 1 -- 172.21.15.189:0/290415089 shutdown_connections 2024-04-06T12:24:35.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.157+0000 7f8d53953700 1 -- 172.21.15.189:0/290415089 wait complete. 2024-04-06T12:24:35.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.158+0000 7f8d53953700 1 Processor -- start 2024-04-06T12:24:35.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.158+0000 7f8d53953700 1 -- start start 2024-04-06T12:24:35.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.159+0000 7f8d53953700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d4c0b4ba0 0x7f8d4c151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:35.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.159+0000 7f8d53953700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d4c0cf900 0x7f8d4c155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:35.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.159+0000 7f8d53953700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d4c156520 0x7f8d4c157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:35.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.159+0000 7f8d53953700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c152450 con 0x7f8d4c156520 2024-04-06T12:24:35.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.159+0000 7f8d53953700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c156aa0 con 0x7f8d4c0b4ba0 2024-04-06T12:24:35.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.159+0000 7f8d53953700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c156d80 con 0x7f8d4c0cf900 2024-04-06T12:24:35.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.159+0000 7f8d51ef0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d4c156520 0x7f8d4c157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:35.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.159+0000 7f8d51ef0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d4c156520 0x7f8d4c157630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60182/0 (socket says 172.21.15.189:60182) 2024-04-06T12:24:35.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.159+0000 7f8d51ef0700 1 -- 172.21.15.189:0/152551973 learned_addr learned my addr 172.21.15.189:0/152551973 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:35.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.159+0000 7f8d50eee700 1 --2- 172.21.15.189:0/152551973 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d4c0cf900 0x7f8d4c155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:35.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.159+0000 7f8d51ef0700 1 -- 172.21.15.189:0/152551973 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d4c0cf900 msgr2=0x7f8d4c155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:35.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.159+0000 7f8d51ef0700 1 --2- 172.21.15.189:0/152551973 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d4c0cf900 0x7f8d4c155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:35.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.159+0000 7f8d51ef0700 1 -- 172.21.15.189:0/152551973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d4c0b4ba0 msgr2=0x7f8d4c151f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:35.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.159+0000 7f8d516ef700 1 --2- 172.21.15.189:0/152551973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d4c0b4ba0 0x7f8d4c151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:35.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.159+0000 7f8d51ef0700 1 --2- 172.21.15.189:0/152551973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d4c0b4ba0 0x7f8d4c151f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:35.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.159+0000 7f8d51ef0700 1 -- 172.21.15.189:0/152551973 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4800b740 con 0x7f8d4c156520 2024-04-06T12:24:35.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.159+0000 7f8d50eee700 1 --2- 172.21.15.189:0/152551973 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d4c0cf900 0x7f8d4c155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:24:35.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.159+0000 7f8d516ef700 1 --2- 172.21.15.189:0/152551973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d4c0b4ba0 0x7f8d4c151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:24:35.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.160+0000 7f8d51ef0700 1 --2- 172.21.15.189:0/152551973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d4c156520 0x7f8d4c157630 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400cb60 tx=0x7f8d4400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:35.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.160+0000 7f8d41ffb700 1 -- 172.21.15.189:0/152551973 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4400ab50 con 0x7f8d4c156520 2024-04-06T12:24:35.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.160+0000 7f8d41ffb700 1 -- 172.21.15.189:0/152551973 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4400acb0 con 0x7f8d4c156520 2024-04-06T12:24:35.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.160+0000 7f8d41ffb700 1 -- 172.21.15.189:0/152551973 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d44007310 con 0x7f8d4c156520 2024-04-06T12:24:35.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.160+0000 7f8d53953700 1 -- 172.21.15.189:0/152551973 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d4c0af8b0 con 0x7f8d4c156520 2024-04-06T12:24:35.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.160+0000 7f8d53953700 1 -- 172.21.15.189:0/152551973 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d4c0af200 con 0x7f8d4c156520 2024-04-06T12:24:35.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.161+0000 7f8d53953700 1 -- 172.21.15.189:0/152551973 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4c0afdd0 con 0x7f8d4c156520 2024-04-06T12:24:35.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.161+0000 7f8d41ffb700 1 -- 172.21.15.189:0/152551973 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f8d44007470 con 0x7f8d4c156520 2024-04-06T12:24:35.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.162+0000 7f8d41ffb700 1 --2- 172.21.15.189:0/152551973 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8d3006ae90 0x7f8d3006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:35.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.162+0000 7f8d41ffb700 1 -- 172.21.15.189:0/152551973 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 7364+0+0 (secure 0 0 0) 0x7f8d440898e0 con 0x7f8d4c156520 2024-04-06T12:24:35.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.162+0000 7f8d516ef700 1 --2- 172.21.15.189:0/152551973 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8d3006ae90 0x7f8d3006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:35.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.162+0000 7f8d516ef700 1 --2- 172.21.15.189:0/152551973 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8d3006ae90 0x7f8d3006d350 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c009a00 tx=0x7f8d3c008040 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:35.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.166+0000 7f8d41ffb700 1 -- 172.21.15.189:0/152551973 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8d4c0afdd0 con 0x7f8d4c156520 2024-04-06T12:24:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.389+0000 7f8d53953700 1 -- 172.21.15.189:0/152551973 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d4c0af730 con 0x7f8d3006ae90 2024-04-06T12:24:35.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.461+0000 7f8d41ffb700 1 -- 172.21.15.189:0/152551973 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f8d3c009610 con 0x7f8d3006ae90 2024-04-06T12:24:35.458 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule removed for path /snap_test_dir1/dir5 2024-04-06T12:24:35.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.463+0000 7f8d53953700 1 -- 172.21.15.189:0/152551973 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8d3006ae90 msgr2=0x7f8d3006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:35.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.463+0000 7f8d53953700 1 --2- 172.21.15.189:0/152551973 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8d3006ae90 0x7f8d3006d350 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c009a00 tx=0x7f8d3c008040 comp rx=0 tx=0).stop 2024-04-06T12:24:35.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.463+0000 7f8d53953700 1 -- 172.21.15.189:0/152551973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d4c156520 msgr2=0x7f8d4c157630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:35.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.463+0000 7f8d53953700 1 --2- 172.21.15.189:0/152551973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d4c156520 0x7f8d4c157630 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400cb60 tx=0x7f8d4400cf20 comp rx=0 tx=0).stop 2024-04-06T12:24:35.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.463+0000 7f8d53953700 1 -- 172.21.15.189:0/152551973 shutdown_connections 2024-04-06T12:24:35.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.463+0000 7f8d53953700 1 --2- 172.21.15.189:0/152551973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d4c156520 0x7f8d4c157630 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:35.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.463+0000 7f8d53953700 1 --2- 172.21.15.189:0/152551973 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8d3006ae90 0x7f8d3006d350 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:35.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.463+0000 7f8d53953700 1 --2- 172.21.15.189:0/152551973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d4c0b4ba0 0x7f8d4c151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:35.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.463+0000 7f8d53953700 1 --2- 172.21.15.189:0/152551973 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d4c0cf900 0x7f8d4c155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:35.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.463+0000 7f8d53953700 1 -- 172.21.15.189:0/152551973 >> 172.21.15.189:0/152551973 conn(0x7f8d4c0d00b0 msgr2=0x7f8d4c152c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:35.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.463+0000 7f8d53953700 1 -- 172.21.15.189:0/152551973 shutdown_connections 2024-04-06T12:24:35.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:35.463+0000 7f8d53953700 1 -- 172.21.15.189:0/152551973 wait complete. 2024-04-06T12:24:35.473 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2024-04-06T12:24:35.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:35.474 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2024-04-06T12:24:35.492 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:35.492 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:35.492 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:35.492 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:35.492 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:35.492 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:35.493 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:35.493 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:35.493 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:35.493 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:35.493 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:35.493 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:35.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:35.493 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_14_00_UTC) 2024-04-06T12:24:35.572 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:35.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:35.573 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_15_00_UTC) 2024-04-06T12:24:35.635 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:35.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:35.635 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_16_00_UTC) 2024-04-06T12:24:35.697 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:35.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:35.698 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_17_00_UTC) 2024-04-06T12:24:35.761 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:35.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:35.761 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_18_00_UTC) 2024-04-06T12:24:35.824 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:35.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:35.825 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_19_00_UTC) 2024-04-06T12:24:35.888 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:35.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:35.888 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_20_00_UTC) 2024-04-06T12:24:35.951 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:35.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:35.951 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_21_00_UTC) 2024-04-06T12:24:36.016 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:36.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:36.016 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_22_00_UTC) 2024-04-06T12:24:36.078 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:36.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:36.079 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_23_00_UTC) 2024-04-06T12:24:36.141 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:36.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:36.142 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-06-12_24_00_UTC) 2024-04-06T12:24:36.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:36.205 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2024-04-06T12:24:36.264 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir6 1m 2024-04-06T12:24:36.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.432+0000 7f36564d4700 1 Processor -- start 2024-04-06T12:24:36.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.432+0000 7f36564d4700 1 -- start start 2024-04-06T12:24:36.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.433+0000 7f36564d4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f365015d7b0 0x7f36500b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:36.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.433+0000 7f36564d4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f365006a730 0x7f365006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:36.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.433+0000 7f36564d4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36500b3990 con 0x7f365015ddf0 2024-04-06T12:24:36.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.433+0000 7f36564d4700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f365015e950 con 0x7f365006a730 2024-04-06T12:24:36.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.433+0000 7f36564d4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f365006b0f0 con 0x7f365015d7b0 2024-04-06T12:24:36.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.433+0000 7f364f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f365006a730 0x7f365006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:36.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.433+0000 7f3654a71700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f365015ddf0 0x7f365015e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36368/0 (socket says 172.21.15.189:36368) 2024-04-06T12:24:36.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.433+0000 7f3654a71700 1 -- 172.21.15.189:0/2862722982 learned_addr learned my addr 172.21.15.189:0/2862722982 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:36.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.433+0000 7f364ffff700 1 --2- 172.21.15.189:0/2862722982 >> v2:172.21.15.192:3300/0 conn(0x7f365015d7b0 0x7f36500b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:36.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.433+0000 7f364f7fe700 1 -- 172.21.15.189:0/2862722982 >> v2:172.21.15.192:3300/0 conn(0x7f365015d7b0 msgr2=0x7f36500b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:36.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.433+0000 7f364f7fe700 1 --2- 172.21.15.189:0/2862722982 >> v2:172.21.15.192:3300/0 conn(0x7f365015d7b0 0x7f36500b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:36.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.433+0000 7f364f7fe700 1 -- 172.21.15.189:0/2862722982 >> v1:172.21.15.189:6789/0 conn(0x7f365015ddf0 legacy=0x7f365015e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:36.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.433+0000 7f364effd700 1 -- 172.21.15.189:0/2862722982 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618397369 0 0) 0x7f36500b3990 con 0x7f365015ddf0 2024-04-06T12:24:36.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.433+0000 7f364f7fe700 1 -- 172.21.15.189:0/2862722982 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36500b3060 con 0x7f365006a730 2024-04-06T12:24:36.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.433+0000 7f364ffff700 1 --2- 172.21.15.189:0/2862722982 >> v2:172.21.15.192:3300/0 conn(0x7f365015d7b0 0x7f36500b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:24:36.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.434+0000 7f364f7fe700 1 --2- 172.21.15.189:0/2862722982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f365006a730 0x7f365006abb0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f36500b6200 tx=0x7f3640009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db579f423dd68234 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:36.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.434+0000 7f364effd700 1 -- 172.21.15.189:0/2862722982 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3640011830 con 0x7f365006a730 2024-04-06T12:24:36.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.434+0000 7f364effd700 1 -- 172.21.15.189:0/2862722982 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3640011990 con 0x7f365006a730 2024-04-06T12:24:36.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.434+0000 7f364effd700 1 -- 172.21.15.189:0/2862722982 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3640011d20 con 0x7f365006a730 2024-04-06T12:24:36.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.434+0000 7f36564d4700 1 -- 172.21.15.189:0/2862722982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f365006a730 msgr2=0x7f365006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:36.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.434+0000 7f36564d4700 1 --2- 172.21.15.189:0/2862722982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f365006a730 0x7f365006abb0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f36500b6200 tx=0x7f3640009b20 comp rx=0 tx=0).stop 2024-04-06T12:24:36.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.434+0000 7f36564d4700 1 -- 172.21.15.189:0/2862722982 shutdown_connections 2024-04-06T12:24:36.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.434+0000 7f36564d4700 1 --2- 172.21.15.189:0/2862722982 >> v2:172.21.15.192:3300/0 conn(0x7f365015d7b0 0x7f36500b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:36.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.434+0000 7f36564d4700 1 --2- 172.21.15.189:0/2862722982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f365006a730 0x7f365006abb0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:36.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.434+0000 7f36564d4700 1 -- 172.21.15.189:0/2862722982 >> 172.21.15.189:0/2862722982 conn(0x7f36500d00b0 msgr2=0x7f36500d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:36.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.434+0000 7f36564d4700 1 -- 172.21.15.189:0/2862722982 shutdown_connections 2024-04-06T12:24:36.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.434+0000 7f36564d4700 1 -- 172.21.15.189:0/2862722982 wait complete. 2024-04-06T12:24:36.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.435+0000 7f36564d4700 1 Processor -- start 2024-04-06T12:24:36.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.436+0000 7f36564d4700 1 -- start start 2024-04-06T12:24:36.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.436+0000 7f36564d4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f365006a730 0x7f3650149b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:36.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.436+0000 7f36564d4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f365015d7b0 0x7f36500cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:36.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.436+0000 7f36564d4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f365015ddf0 0x7f36500adc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:36.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.436+0000 7f36564d4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36500cd910 con 0x7f365006a730 2024-04-06T12:24:36.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.436+0000 7f36564d4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365014a390 con 0x7f365015ddf0 2024-04-06T12:24:36.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.436+0000 7f36564d4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f365014a090 con 0x7f365015d7b0 2024-04-06T12:24:36.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.437+0000 7f364ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f365006a730 0x7f3650149b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:36.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.437+0000 7f364ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f365006a730 0x7f3650149b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60198/0 (socket says 172.21.15.189:60198) 2024-04-06T12:24:36.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.437+0000 7f364ffff700 1 -- 172.21.15.189:0/4085416580 learned_addr learned my addr 172.21.15.189:0/4085416580 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:36.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.437+0000 7f3654a71700 1 --2- 172.21.15.189:0/4085416580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f365015ddf0 0x7f36500adc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:36.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.437+0000 7f364f7fe700 1 --2- 172.21.15.189:0/4085416580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f365015d7b0 0x7f36500cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:36.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.437+0000 7f364ffff700 1 -- 172.21.15.189:0/4085416580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f365015d7b0 msgr2=0x7f36500cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:36.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.437+0000 7f364ffff700 1 --2- 172.21.15.189:0/4085416580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f365015d7b0 0x7f36500cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:36.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.437+0000 7f364ffff700 1 -- 172.21.15.189:0/4085416580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f365015ddf0 msgr2=0x7f36500adc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:36.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.437+0000 7f364ffff700 1 --2- 172.21.15.189:0/4085416580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f365015ddf0 0x7f36500adc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:36.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.437+0000 7f364ffff700 1 -- 172.21.15.189:0/4085416580 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36400094a0 con 0x7f365006a730 2024-04-06T12:24:36.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.437+0000 7f3654a71700 1 --2- 172.21.15.189:0/4085416580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f365015ddf0 0x7f36500adc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:24:36.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.437+0000 7f364ffff700 1 --2- 172.21.15.189:0/4085416580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f365006a730 0x7f3650149b50 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f364400c750 tx=0x7f364400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:36.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.437+0000 7f364cff9700 1 -- 172.21.15.189:0/4085416580 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3644011800 con 0x7f365006a730 2024-04-06T12:24:36.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.437+0000 7f364cff9700 1 -- 172.21.15.189:0/4085416580 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3644011960 con 0x7f365006a730 2024-04-06T12:24:36.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.437+0000 7f364cff9700 1 -- 172.21.15.189:0/4085416580 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3644011cd0 con 0x7f365006a730 2024-04-06T12:24:36.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.437+0000 7f36564d4700 1 -- 172.21.15.189:0/4085416580 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36500cd630 con 0x7f365006a730 2024-04-06T12:24:36.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.437+0000 7f36564d4700 1 -- 172.21.15.189:0/4085416580 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f365014a880 con 0x7f365006a730 2024-04-06T12:24:36.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.438+0000 7f36564d4700 1 -- 172.21.15.189:0/4085416580 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3650067bb0 con 0x7f365006a730 2024-04-06T12:24:36.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.439+0000 7f364cff9700 1 -- 172.21.15.189:0/4085416580 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3644021350 con 0x7f365006a730 2024-04-06T12:24:36.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.439+0000 7f364cff9700 1 --2- 172.21.15.189:0/4085416580 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f363c06ae90 0x7f363c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:36.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.439+0000 7f364f7fe700 1 --2- 172.21.15.189:0/4085416580 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f363c06ae90 0x7f363c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:36.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.439+0000 7f364cff9700 1 -- 172.21.15.189:0/4085416580 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6948+0+0 (secure 0 0 0) 0x7f3644089c30 con 0x7f365006a730 2024-04-06T12:24:36.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.439+0000 7f364f7fe700 1 --2- 172.21.15.189:0/4085416580 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f363c06ae90 0x7f363c06d350 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f36500cc930 tx=0x7f3640008030 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:36.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.444+0000 7f364cff9700 1 -- 172.21.15.189:0/4085416580 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3644054630 con 0x7f365006a730 2024-04-06T12:24:36.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.677+0000 7f36564d4700 1 -- 172.21.15.189:0/4085416580 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f365015eb40 con 0x7f363c06ae90 2024-04-06T12:24:36.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.777+0000 7f364cff9700 1 -- 172.21.15.189:0/4085416580 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f3640011b80 con 0x7f363c06ae90 2024-04-06T12:24:36.774 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule removed for path /snap_test_dir1/dir6 2024-04-06T12:24:36.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.779+0000 7f36564d4700 1 -- 172.21.15.189:0/4085416580 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f363c06ae90 msgr2=0x7f363c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.779+0000 7f36564d4700 1 --2- 172.21.15.189:0/4085416580 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f363c06ae90 0x7f363c06d350 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f36500cc930 tx=0x7f3640008030 comp rx=0 tx=0).stop 2024-04-06T12:24:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.779+0000 7f36564d4700 1 -- 172.21.15.189:0/4085416580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f365006a730 msgr2=0x7f3650149b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.779+0000 7f36564d4700 1 --2- 172.21.15.189:0/4085416580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f365006a730 0x7f3650149b50 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f364400c750 tx=0x7f364400cb10 comp rx=0 tx=0).stop 2024-04-06T12:24:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.779+0000 7f36564d4700 1 -- 172.21.15.189:0/4085416580 shutdown_connections 2024-04-06T12:24:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.779+0000 7f36564d4700 1 --2- 172.21.15.189:0/4085416580 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f363c06ae90 0x7f363c06d350 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.779+0000 7f36564d4700 1 --2- 172.21.15.189:0/4085416580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f365015ddf0 0x7f36500adc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.779+0000 7f36564d4700 1 --2- 172.21.15.189:0/4085416580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f365006a730 0x7f3650149b50 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.779+0000 7f36564d4700 1 --2- 172.21.15.189:0/4085416580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f365015d7b0 0x7f36500cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.779+0000 7f36564d4700 1 -- 172.21.15.189:0/4085416580 >> 172.21.15.189:0/4085416580 conn(0x7f36500d00b0 msgr2=0x7f365015f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.779+0000 7f36564d4700 1 -- 172.21.15.189:0/4085416580 shutdown_connections 2024-04-06T12:24:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:36.779+0000 7f36564d4700 1 -- 172.21.15.189:0/4085416580 wait complete. 2024-04-06T12:24:36.788 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2024-04-06T12:24:36.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:36.789 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2024-04-06T12:24:36.807 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:36.807 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:36.807 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:36.807 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:36.807 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:36.807 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:36.807 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:36.807 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_01_UTC 2024-04-06T12:24:36.807 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:36.807 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:36.808 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:36.808 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:36.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:36.808 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_14_00_UTC) 2024-04-06T12:24:36.873 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:36.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:36.873 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_15_00_UTC) 2024-04-06T12:24:36.938 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:36.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:36.938 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_16_00_UTC) 2024-04-06T12:24:37.003 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:37.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:37.003 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_17_00_UTC) 2024-04-06T12:24:37.066 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:37.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:37.066 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_18_00_UTC) 2024-04-06T12:24:37.129 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:37.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:37.129 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_19_00_UTC) 2024-04-06T12:24:37.195 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:37.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:37.195 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_20_00_UTC) 2024-04-06T12:24:37.258 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_21_01_UTC 2024-04-06T12:24:37.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:37.258 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_21_01_UTC) 2024-04-06T12:24:37.320 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:37.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:37.320 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_22_00_UTC) 2024-04-06T12:24:37.382 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:37.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:37.383 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_23_00_UTC) 2024-04-06T12:24:37.445 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:37.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:37.446 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-06-12_24_00_UTC) 2024-04-06T12:24:37.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:37.512 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2024-04-06T12:24:37.574 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir7 1m 2024-04-06T12:24:37.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.751+0000 7f71c256e700 1 Processor -- start 2024-04-06T12:24:37.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.751+0000 7f71c256e700 1 -- start start 2024-04-06T12:24:37.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.752+0000 7f71c256e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f71bc15a550 0x7f71bc0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:37.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.752+0000 7f71c256e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71bc0b4d70 0x7f71bc0b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:37.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.752+0000 7f71c256e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71bc0cdbe0 con 0x7f71bc0cf900 2024-04-06T12:24:37.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.752+0000 7f71c256e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71bc15a940 con 0x7f71bc15a550 2024-04-06T12:24:37.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.752+0000 7f71c256e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc15ac00 con 0x7f71bc0b4d70 2024-04-06T12:24:37.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.752+0000 7f71bbfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f71bc15a550 0x7f71bc0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:37.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.752+0000 7f71bbfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f71bc15a550 0x7f71bc0cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60204/0 (socket says 172.21.15.189:60204) 2024-04-06T12:24:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.752+0000 7f71bbfff700 1 -- 172.21.15.189:0/715972614 learned_addr learned my addr 172.21.15.189:0/715972614 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.752+0000 7f71c0b0b700 1 --2- 172.21.15.189:0/715972614 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71bc0b4d70 0x7f71bc0b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.752+0000 7f71bbfff700 1 -- 172.21.15.189:0/715972614 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71bc0b4d70 msgr2=0x7f71bc0b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.752+0000 7f71bbfff700 1 --2- 172.21.15.189:0/715972614 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71bc0b4d70 0x7f71bc0b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.752+0000 7f71bbfff700 1 -- 172.21.15.189:0/715972614 >> v1:172.21.15.192:6789/0 conn(0x7f71bc0cf900 legacy=0x7f71bc0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.752+0000 7f71bbfff700 1 -- 172.21.15.189:0/715972614 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71bc15aee0 con 0x7f71bc15a550 2024-04-06T12:24:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.753+0000 7f71bbfff700 1 --2- 172.21.15.189:0/715972614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71bc15a550 0x7f71bc0cf3c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00bb90 tx=0x7f71ac00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=817c7f84a33ffa2a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.753+0000 7f71baffd700 1 -- 172.21.15.189:0/715972614 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ac00dd50 con 0x7f71bc15a550 2024-04-06T12:24:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.753+0000 7f71baffd700 1 -- 172.21.15.189:0/715972614 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71ac010470 con 0x7f71bc15a550 2024-04-06T12:24:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.753+0000 7f71baffd700 1 -- 172.21.15.189:0/715972614 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ac010800 con 0x7f71bc15a550 2024-04-06T12:24:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.753+0000 7f71c256e700 1 -- 172.21.15.189:0/715972614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71bc15a550 msgr2=0x7f71bc0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.753+0000 7f71c256e700 1 --2- 172.21.15.189:0/715972614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71bc15a550 0x7f71bc0cf3c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00bb90 tx=0x7f71ac00bea0 comp rx=0 tx=0).stop 2024-04-06T12:24:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.753+0000 7f71c256e700 1 -- 172.21.15.189:0/715972614 shutdown_connections 2024-04-06T12:24:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.753+0000 7f71c256e700 1 --2- 172.21.15.189:0/715972614 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71bc0b4d70 0x7f71bc0b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.753+0000 7f71c256e700 1 --2- 172.21.15.189:0/715972614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71bc15a550 0x7f71bc0cf3c0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.753+0000 7f71c256e700 1 -- 172.21.15.189:0/715972614 >> 172.21.15.189:0/715972614 conn(0x7f71bc0d00b0 msgr2=0x7f71bc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.753+0000 7f71c256e700 1 -- 172.21.15.189:0/715972614 shutdown_connections 2024-04-06T12:24:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.753+0000 7f71c256e700 1 -- 172.21.15.189:0/715972614 wait complete. 2024-04-06T12:24:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.755+0000 7f71c256e700 1 Processor -- start 2024-04-06T12:24:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.755+0000 7f71c256e700 1 -- start start 2024-04-06T12:24:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.756+0000 7f71c256e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71bc0b4d70 0x7f71bc152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.756+0000 7f71c256e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71bc0cf900 0x7f71bc0a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.756+0000 7f71c256e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71bc154860 0x7f71bc154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.756+0000 7f71c256e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc155250 con 0x7f71bc0cf900 2024-04-06T12:24:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.756+0000 7f71c256e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc1f2d30 con 0x7f71bc0b4d70 2024-04-06T12:24:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.756+0000 7f71c256e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc1f2120 con 0x7f71bc154860 2024-04-06T12:24:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.756+0000 7f71bb7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71bc0cf900 0x7f71bc0a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.756+0000 7f71bb7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71bc0cf900 0x7f71bc0a84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60218/0 (socket says 172.21.15.189:60218) 2024-04-06T12:24:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.756+0000 7f71bb7fe700 1 -- 172.21.15.189:0/2374721020 learned_addr learned my addr 172.21.15.189:0/2374721020 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.756+0000 7f71bbfff700 1 --2- 172.21.15.189:0/2374721020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71bc0b4d70 0x7f71bc152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.756+0000 7f71c0b0b700 1 --2- 172.21.15.189:0/2374721020 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71bc154860 0x7f71bc154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.756+0000 7f71bb7fe700 1 -- 172.21.15.189:0/2374721020 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71bc154860 msgr2=0x7f71bc154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.756+0000 7f71bb7fe700 1 --2- 172.21.15.189:0/2374721020 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71bc154860 0x7f71bc154d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.756+0000 7f71bb7fe700 1 -- 172.21.15.189:0/2374721020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71bc0b4d70 msgr2=0x7f71bc152d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.756+0000 7f71bb7fe700 1 --2- 172.21.15.189:0/2374721020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71bc0b4d70 0x7f71bc152d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.756+0000 7f71bb7fe700 1 -- 172.21.15.189:0/2374721020 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ac00b7f0 con 0x7f71bc0cf900 2024-04-06T12:24:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.756+0000 7f71bb7fe700 1 --2- 172.21.15.189:0/2374721020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71bc0cf900 0x7f71bc0a84b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f71b000c4e0 tx=0x7f71b000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.757+0000 7f71b8ff9700 1 -- 172.21.15.189:0/2374721020 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b00052d0 con 0x7f71bc0cf900 2024-04-06T12:24:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.757+0000 7f71b8ff9700 1 -- 172.21.15.189:0/2374721020 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b0005430 con 0x7f71bc0cf900 2024-04-06T12:24:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.757+0000 7f71c256e700 1 -- 172.21.15.189:0/2374721020 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71bc1f2400 con 0x7f71bc0cf900 2024-04-06T12:24:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.757+0000 7f71b8ff9700 1 -- 172.21.15.189:0/2374721020 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b00038c0 con 0x7f71bc0cf900 2024-04-06T12:24:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.757+0000 7f71c256e700 1 -- 172.21.15.189:0/2374721020 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71bc1f29c0 con 0x7f71bc0cf900 2024-04-06T12:24:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.758+0000 7f71c256e700 1 -- 172.21.15.189:0/2374721020 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71bc067bb0 con 0x7f71bc0cf900 2024-04-06T12:24:37.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.758+0000 7f71b8ff9700 1 -- 172.21.15.189:0/2374721020 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f71b0003a20 con 0x7f71bc0cf900 2024-04-06T12:24:37.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.758+0000 7f71b8ff9700 1 --2- 172.21.15.189:0/2374721020 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f71a806ae90 0x7f71a806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:37.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.759+0000 7f71b8ff9700 1 -- 172.21.15.189:0/2374721020 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7044+0+0 (secure 0 0 0) 0x7f71b0088ef0 con 0x7f71bc0cf900 2024-04-06T12:24:37.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.759+0000 7f71bbfff700 1 --2- 172.21.15.189:0/2374721020 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f71a806ae90 0x7f71a806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:37.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.759+0000 7f71bbfff700 1 --2- 172.21.15.189:0/2374721020 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f71a806ae90 0x7f71a806d350 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00bee0 tx=0x7f71ac013000 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:37.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:37.763+0000 7f71b8ff9700 1 -- 172.21.15.189:0/2374721020 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f71b00539a0 con 0x7f71bc0cf900 2024-04-06T12:24:37.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:38.000+0000 7f71c256e700 1 -- 172.21.15.189:0/2374721020 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f71bc1f2ea0 con 0x7f71a806ae90 2024-04-06T12:24:38.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:38.080+0000 7f71b8ff9700 1 -- 172.21.15.189:0/2374721020 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f71bc1f2ea0 con 0x7f71a806ae90 2024-04-06T12:24:38.077 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule removed for path /snap_test_dir1/dir7 2024-04-06T12:24:38.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:38.082+0000 7f71c256e700 1 -- 172.21.15.189:0/2374721020 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f71a806ae90 msgr2=0x7f71a806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:38.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:38.082+0000 7f71c256e700 1 --2- 172.21.15.189:0/2374721020 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f71a806ae90 0x7f71a806d350 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00bee0 tx=0x7f71ac013000 comp rx=0 tx=0).stop 2024-04-06T12:24:38.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:38.082+0000 7f71c256e700 1 -- 172.21.15.189:0/2374721020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71bc0cf900 msgr2=0x7f71bc0a84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:38.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:38.082+0000 7f71c256e700 1 --2- 172.21.15.189:0/2374721020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71bc0cf900 0x7f71bc0a84b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f71b000c4e0 tx=0x7f71b000c7f0 comp rx=0 tx=0).stop 2024-04-06T12:24:38.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:38.082+0000 7f71c256e700 1 -- 172.21.15.189:0/2374721020 shutdown_connections 2024-04-06T12:24:38.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:38.082+0000 7f71c256e700 1 --2- 172.21.15.189:0/2374721020 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71bc154860 0x7f71bc154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:38.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:38.082+0000 7f71c256e700 1 --2- 172.21.15.189:0/2374721020 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f71a806ae90 0x7f71a806d350 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:38.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:38.082+0000 7f71c256e700 1 --2- 172.21.15.189:0/2374721020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71bc0b4d70 0x7f71bc152d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:38.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:38.082+0000 7f71c256e700 1 --2- 172.21.15.189:0/2374721020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71bc0cf900 0x7f71bc0a84b0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:38.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:38.082+0000 7f71c256e700 1 -- 172.21.15.189:0/2374721020 >> 172.21.15.189:0/2374721020 conn(0x7f71bc0d00b0 msgr2=0x7f71bc06f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:38.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:38.082+0000 7f71c256e700 1 -- 172.21.15.189:0/2374721020 shutdown_connections 2024-04-06T12:24:38.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:38.082+0000 7f71c256e700 1 -- 172.21.15.189:0/2374721020 wait complete. 2024-04-06T12:24:38.092 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2024-04-06T12:24:38.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:38.092 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2024-04-06T12:24:38.111 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:38.111 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:38.111 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:38.111 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:38.111 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:38.111 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:38.112 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:38.112 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:38.112 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:38.112 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:38.112 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:38.112 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:38.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:38.113 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_14_00_UTC) 2024-04-06T12:24:38.175 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:38.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:38.175 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_15_00_UTC) 2024-04-06T12:24:38.238 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:38.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:38.238 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_16_00_UTC) 2024-04-06T12:24:38.301 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:38.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:38.301 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_17_00_UTC) 2024-04-06T12:24:38.366 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:38.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:38.366 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_18_00_UTC) 2024-04-06T12:24:38.433 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:38.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:38.433 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_19_00_UTC) 2024-04-06T12:24:38.496 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:38.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:38.497 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_20_00_UTC) 2024-04-06T12:24:38.562 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:38.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:38.562 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_21_00_UTC) 2024-04-06T12:24:38.640 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:38.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:38.640 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_22_00_UTC) 2024-04-06T12:24:38.704 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:38.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:38.704 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_23_00_UTC) 2024-04-06T12:24:38.769 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:38.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:38.769 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-06-12_24_00_UTC) 2024-04-06T12:24:38.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:38.843 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2024-04-06T12:24:38.902 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir8 1m 2024-04-06T12:24:39.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.070+0000 7f0dfe6bd700 1 Processor -- start 2024-04-06T12:24:39.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.070+0000 7f0dfe6bd700 1 -- start start 2024-04-06T12:24:39.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.071+0000 7f0dfe6bd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0df815d7b0 0x7f0df80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:39.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.071+0000 7f0dfe6bd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0df815e500 0x7f0df806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:39.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.071+0000 7f0dfe6bd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df80b3990 con 0x7f0df806a730 2024-04-06T12:24:39.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.071+0000 7f0dfe6bd700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0df815e950 con 0x7f0df815d7b0 2024-04-06T12:24:39.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.071+0000 7f0dfe6bd700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0df806b0f0 con 0x7f0df815e500 2024-04-06T12:24:39.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.071+0000 7f0df7fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0df815d7b0 0x7f0df80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:39.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.071+0000 7f0df77fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0df806a730 0x7f0df815ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36370/0 (socket says 172.21.15.189:36370) 2024-04-06T12:24:39.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.071+0000 7f0df77fe700 1 -- 172.21.15.189:0/3178013857 learned_addr learned my addr 172.21.15.189:0/3178013857 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:39.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.071+0000 7f0df7fff700 1 -- 172.21.15.189:0/3178013857 >> v2:172.21.15.192:3300/0 conn(0x7f0df815e500 msgr2=0x7f0df806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:39.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.071+0000 7f0dfcc5a700 1 --2- 172.21.15.189:0/3178013857 >> v2:172.21.15.192:3300/0 conn(0x7f0df815e500 0x7f0df806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:39.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.071+0000 7f0df7fff700 1 --2- 172.21.15.189:0/3178013857 >> v2:172.21.15.192:3300/0 conn(0x7f0df815e500 0x7f0df806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:39.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.071+0000 7f0df7fff700 1 -- 172.21.15.189:0/3178013857 >> v1:172.21.15.189:6789/0 conn(0x7f0df806a730 legacy=0x7f0df815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:39.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.071+0000 7f0df7fff700 1 -- 172.21.15.189:0/3178013857 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df80b3060 con 0x7f0df815d7b0 2024-04-06T12:24:39.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.071+0000 7f0df6ffd700 1 -- 172.21.15.189:0/3178013857 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172855557 0 0) 0x7f0df80b3990 con 0x7f0df806a730 2024-04-06T12:24:39.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.071+0000 7f0dfcc5a700 1 --2- 172.21.15.189:0/3178013857 >> v2:172.21.15.192:3300/0 conn(0x7f0df815e500 0x7f0df806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:24:39.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.071+0000 7f0df7fff700 1 --2- 172.21.15.189:0/3178013857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0df815d7b0 0x7f0df80b3e20 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f0df80673e0 tx=0x7f0dec009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc14757a560b799e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:39.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.072+0000 7f0df6ffd700 1 -- 172.21.15.189:0/3178013857 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec011830 con 0x7f0df815d7b0 2024-04-06T12:24:39.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.072+0000 7f0df6ffd700 1 -- 172.21.15.189:0/3178013857 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dec011990 con 0x7f0df815d7b0 2024-04-06T12:24:39.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.072+0000 7f0df6ffd700 1 -- 172.21.15.189:0/3178013857 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec011d20 con 0x7f0df815d7b0 2024-04-06T12:24:39.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.072+0000 7f0dfe6bd700 1 -- 172.21.15.189:0/3178013857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0df815d7b0 msgr2=0x7f0df80b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:39.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.072+0000 7f0dfe6bd700 1 --2- 172.21.15.189:0/3178013857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0df815d7b0 0x7f0df80b3e20 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f0df80673e0 tx=0x7f0dec009b20 comp rx=0 tx=0).stop 2024-04-06T12:24:39.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.072+0000 7f0dfe6bd700 1 -- 172.21.15.189:0/3178013857 shutdown_connections 2024-04-06T12:24:39.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.072+0000 7f0dfe6bd700 1 --2- 172.21.15.189:0/3178013857 >> v2:172.21.15.192:3300/0 conn(0x7f0df815e500 0x7f0df806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:39.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.072+0000 7f0dfe6bd700 1 --2- 172.21.15.189:0/3178013857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0df815d7b0 0x7f0df80b3e20 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:39.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.072+0000 7f0dfe6bd700 1 -- 172.21.15.189:0/3178013857 >> 172.21.15.189:0/3178013857 conn(0x7f0df80d00b0 msgr2=0x7f0df80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:39.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.072+0000 7f0dfe6bd700 1 -- 172.21.15.189:0/3178013857 shutdown_connections 2024-04-06T12:24:39.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.072+0000 7f0dfe6bd700 1 -- 172.21.15.189:0/3178013857 wait complete. 2024-04-06T12:24:39.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.073+0000 7f0dfe6bd700 1 Processor -- start 2024-04-06T12:24:39.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.074+0000 7f0dfe6bd700 1 -- start start 2024-04-06T12:24:39.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.074+0000 7f0dfe6bd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0df806a730 0x7f0df814d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:39.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.074+0000 7f0dfe6bd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0df815d7b0 0x7f0df80ced90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:39.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.074+0000 7f0dfe6bd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0df815e500 0x7f0df814dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:39.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.074+0000 7f0dfe6bd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df815b570 con 0x7f0df806a730 2024-04-06T12:24:39.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.074+0000 7f0dfe6bd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df80cd4d0 con 0x7f0df815e500 2024-04-06T12:24:39.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.074+0000 7f0dfe6bd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df815b7c0 con 0x7f0df815d7b0 2024-04-06T12:24:39.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.074+0000 7f0df7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0df806a730 0x7f0df814d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:39.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.074+0000 7f0df7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0df806a730 0x7f0df814d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60236/0 (socket says 172.21.15.189:60236) 2024-04-06T12:24:39.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.074+0000 7f0df7fff700 1 -- 172.21.15.189:0/2773478114 learned_addr learned my addr 172.21.15.189:0/2773478114 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:39.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.075+0000 7f0df7fff700 1 -- 172.21.15.189:0/2773478114 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0df815d7b0 msgr2=0x7f0df80ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:39.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.075+0000 7f0df7fff700 1 --2- 172.21.15.189:0/2773478114 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0df815d7b0 0x7f0df80ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:39.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.075+0000 7f0df7fff700 1 -- 172.21.15.189:0/2773478114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0df815e500 msgr2=0x7f0df814dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:39.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.075+0000 7f0df7fff700 1 --2- 172.21.15.189:0/2773478114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0df815e500 0x7f0df814dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:39.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.075+0000 7f0df7fff700 1 -- 172.21.15.189:0/2773478114 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dec0094a0 con 0x7f0df806a730 2024-04-06T12:24:39.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.075+0000 7f0df7fff700 1 --2- 172.21.15.189:0/2773478114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0df806a730 0x7f0df814d530 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f0dec000c00 tx=0x7f0dec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:39.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.075+0000 7f0df4ff9700 1 -- 172.21.15.189:0/2773478114 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec0038a0 con 0x7f0df806a730 2024-04-06T12:24:39.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.075+0000 7f0df4ff9700 1 -- 172.21.15.189:0/2773478114 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dec01f070 con 0x7f0df806a730 2024-04-06T12:24:39.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.075+0000 7f0dfe6bd700 1 -- 172.21.15.189:0/2773478114 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df815baa0 con 0x7f0df806a730 2024-04-06T12:24:39.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.075+0000 7f0df4ff9700 1 -- 172.21.15.189:0/2773478114 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec003c40 con 0x7f0df806a730 2024-04-06T12:24:39.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.075+0000 7f0dfe6bd700 1 -- 172.21.15.189:0/2773478114 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df80cf2d0 con 0x7f0df806a730 2024-04-06T12:24:39.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.076+0000 7f0dfe6bd700 1 -- 172.21.15.189:0/2773478114 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df8067bb0 con 0x7f0df806a730 2024-04-06T12:24:39.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.076+0000 7f0df4ff9700 1 -- 172.21.15.189:0/2773478114 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f0dec026480 con 0x7f0df806a730 2024-04-06T12:24:39.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.077+0000 7f0df4ff9700 1 --2- 172.21.15.189:0/2773478114 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0de406ae90 0x7f0de406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:39.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.077+0000 7f0df4ff9700 1 -- 172.21.15.189:0/2773478114 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7060+0+0 (secure 0 0 0) 0x7f0dec021070 con 0x7f0df806a730 2024-04-06T12:24:39.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.077+0000 7f0df77fe700 1 --2- 172.21.15.189:0/2773478114 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0de406ae90 0x7f0de406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:39.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.077+0000 7f0df77fe700 1 --2- 172.21.15.189:0/2773478114 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0de406ae90 0x7f0de406d350 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0de80011e0 tx=0x7f0de80079b0 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:39.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.081+0000 7f0df4ff9700 1 -- 172.21.15.189:0/2773478114 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0dec056170 con 0x7f0df806a730 2024-04-06T12:24:39.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.313+0000 7f0dfe6bd700 1 -- 172.21.15.189:0/2773478114 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0df80b3620 con 0x7f0de406ae90 2024-04-06T12:24:39.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.384+0000 7f0df4ff9700 1 -- 172.21.15.189:0/2773478114 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f0de8001370 con 0x7f0de406ae90 2024-04-06T12:24:39.381 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule removed for path /snap_test_dir1/dir8 2024-04-06T12:24:39.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.386+0000 7f0dfe6bd700 1 -- 172.21.15.189:0/2773478114 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0de406ae90 msgr2=0x7f0de406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:39.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.386+0000 7f0dfe6bd700 1 --2- 172.21.15.189:0/2773478114 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0de406ae90 0x7f0de406d350 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0de80011e0 tx=0x7f0de80079b0 comp rx=0 tx=0).stop 2024-04-06T12:24:39.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.386+0000 7f0dfe6bd700 1 -- 172.21.15.189:0/2773478114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0df806a730 msgr2=0x7f0df814d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:39.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.386+0000 7f0dfe6bd700 1 --2- 172.21.15.189:0/2773478114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0df806a730 0x7f0df814d530 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f0dec000c00 tx=0x7f0dec004750 comp rx=0 tx=0).stop 2024-04-06T12:24:39.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.386+0000 7f0dfe6bd700 1 -- 172.21.15.189:0/2773478114 shutdown_connections 2024-04-06T12:24:39.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.386+0000 7f0dfe6bd700 1 --2- 172.21.15.189:0/2773478114 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0de406ae90 0x7f0de406d350 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:39.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.386+0000 7f0dfe6bd700 1 --2- 172.21.15.189:0/2773478114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0df815e500 0x7f0df814dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:39.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.386+0000 7f0dfe6bd700 1 --2- 172.21.15.189:0/2773478114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0df806a730 0x7f0df814d530 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:39.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.386+0000 7f0dfe6bd700 1 --2- 172.21.15.189:0/2773478114 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0df815d7b0 0x7f0df80ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:39.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.386+0000 7f0dfe6bd700 1 -- 172.21.15.189:0/2773478114 >> 172.21.15.189:0/2773478114 conn(0x7f0df80d00b0 msgr2=0x7f0df806fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:39.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.386+0000 7f0dfe6bd700 1 -- 172.21.15.189:0/2773478114 shutdown_connections 2024-04-06T12:24:39.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:39.386+0000 7f0dfe6bd700 1 -- 172.21.15.189:0/2773478114 wait complete. 2024-04-06T12:24:39.395 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2024-04-06T12:24:39.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:39.396 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2024-04-06T12:24:39.413 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:39.413 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:39.413 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:39.414 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:39.414 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:39.414 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:39.414 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:39.414 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:39.414 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:39.414 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:39.414 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:39.415 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:39.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:39.415 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_14_00_UTC) 2024-04-06T12:24:39.479 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:39.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:39.479 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_15_00_UTC) 2024-04-06T12:24:39.544 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:39.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:39.544 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_16_00_UTC) 2024-04-06T12:24:39.641 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:39.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:39.641 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_17_00_UTC) 2024-04-06T12:24:39.670 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:39.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:39.670 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_18_00_UTC) 2024-04-06T12:24:39.735 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:39.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:39.735 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_19_00_UTC) 2024-04-06T12:24:39.799 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:39.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:39.799 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_20_00_UTC) 2024-04-06T12:24:39.863 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:39.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:39.863 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_21_00_UTC) 2024-04-06T12:24:39.927 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:39.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:39.927 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_22_00_UTC) 2024-04-06T12:24:39.991 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:39.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:39.991 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_23_00_UTC) 2024-04-06T12:24:40.055 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:40.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:40.055 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-06-12_24_00_UTC) 2024-04-06T12:24:40.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:40.118 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2024-04-06T12:24:40.180 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir9 1m 2024-04-06T12:24:40.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.348+0000 7f88aa778700 1 Processor -- start 2024-04-06T12:24:40.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.348+0000 7f88aa778700 1 -- start start 2024-04-06T12:24:40.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88aa778700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88a40d17a0 0x7f88a406b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:40.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88aa778700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f88a40cedf0 0x7f88a40cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:40.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88aa778700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a40cf760 con 0x7f88a4146dd0 2024-04-06T12:24:40.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88aa778700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88a406d850 con 0x7f88a40cedf0 2024-04-06T12:24:40.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88aa778700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a406c0a0 con 0x7f88a40d17a0 2024-04-06T12:24:40.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88a37fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f88a4146dd0 0x7f88a406d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47084/0 (socket says 172.21.15.189:47084) 2024-04-06T12:24:40.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88a37fe700 1 -- 172.21.15.189:0/398190986 learned_addr learned my addr 172.21.15.189:0/398190986 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:40.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88a8d15700 1 --2- 172.21.15.189:0/398190986 >> v2:172.21.15.192:3300/0 conn(0x7f88a40cedf0 0x7f88a40cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:40.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88a2ffd700 1 -- 172.21.15.189:0/398190986 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240525051 0 0) 0x7f88a40cf760 con 0x7f88a4146dd0 2024-04-06T12:24:40.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88a3fff700 1 --2- 172.21.15.189:0/398190986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88a40d17a0 0x7f88a406b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:40.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88a2ffd700 1 -- 172.21.15.189:0/398190986 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f888c0035b0 con 0x7f88a4146dd0 2024-04-06T12:24:40.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88a2ffd700 1 -- 172.21.15.189:0/398190986 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1352954568 0 0) 0x7f888c0035b0 con 0x7f88a4146dd0 2024-04-06T12:24:40.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88a2ffd700 1 -- 172.21.15.189:0/398190986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88a40d17a0 msgr2=0x7f88a406b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:40.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88a2ffd700 1 --2- 172.21.15.189:0/398190986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88a40d17a0 0x7f88a406b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:40.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88a2ffd700 1 -- 172.21.15.189:0/398190986 >> v2:172.21.15.192:3300/0 conn(0x7f88a40cedf0 msgr2=0x7f88a40cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:40.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88a2ffd700 1 --2- 172.21.15.189:0/398190986 >> v2:172.21.15.192:3300/0 conn(0x7f88a40cedf0 0x7f88a40cf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:40.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88a2ffd700 1 -- 172.21.15.189:0/398190986 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a406c380 con 0x7f88a4146dd0 2024-04-06T12:24:40.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88a2ffd700 1 -- 172.21.15.189:0/398190986 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f8894001670 con 0x7f88a4146dd0 2024-04-06T12:24:40.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88a2ffd700 1 -- 172.21.15.189:0/398190986 >> v1:172.21.15.189:6789/0 conn(0x7f88a4146dd0 legacy=0x7f88a406d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:40.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88a2ffd700 1 --2- 172.21.15.189:0/398190986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88a40d17a0 0x7f888c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:40.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.349+0000 7f88a2ffd700 1 --2- 172.21.15.189:0/398190986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f888c003f80 0x7f888c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:40.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.350+0000 7f88a2ffd700 1 --2- 172.21.15.189:0/398190986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f888c004d90 0x7f888c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:40.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.350+0000 7f88a2ffd700 1 -- 172.21.15.189:0/398190986 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a406d850 con 0x7f888c004d90 2024-04-06T12:24:40.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.350+0000 7f88a2ffd700 1 -- 172.21.15.189:0/398190986 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a406c0a0 con 0x7f888c003f80 2024-04-06T12:24:40.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.350+0000 7f88a2ffd700 1 -- 172.21.15.189:0/398190986 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888c00ba10 con 0x7f88a40d17a0 2024-04-06T12:24:40.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.350+0000 7f88a8d15700 1 --2- 172.21.15.189:0/398190986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f888c004d90 0x7f888c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:40.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.350+0000 7f88a3fff700 1 --2- 172.21.15.189:0/398190986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88a40d17a0 0x7f888c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:40.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.350+0000 7f88a37fe700 1 --2- 172.21.15.189:0/398190986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f888c003f80 0x7f888c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:40.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.350+0000 7f88a8d15700 1 -- 172.21.15.189:0/398190986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88a40d17a0 msgr2=0x7f888c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:40.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.350+0000 7f88a8d15700 1 --2- 172.21.15.189:0/398190986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88a40d17a0 0x7f888c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:40.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.350+0000 7f88a8d15700 1 -- 172.21.15.189:0/398190986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f888c003f80 msgr2=0x7f888c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:40.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.350+0000 7f88a8d15700 1 --2- 172.21.15.189:0/398190986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f888c003f80 0x7f888c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:40.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.350+0000 7f88a8d15700 1 -- 172.21.15.189:0/398190986 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888c0035b0 con 0x7f888c004d90 2024-04-06T12:24:40.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.350+0000 7f88a8d15700 1 --2- 172.21.15.189:0/398190986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f888c004d90 0x7f888c005210 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f889c009fc0 tx=0x7f888c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faaa033274cb975a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:40.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.350+0000 7f88a2ffd700 1 -- 172.21.15.189:0/398190986 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c00ebd0 con 0x7f888c004d90 2024-04-06T12:24:40.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.350+0000 7f88a2ffd700 1 -- 172.21.15.189:0/398190986 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889c00f040 con 0x7f888c004d90 2024-04-06T12:24:40.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.350+0000 7f88a2ffd700 1 -- 172.21.15.189:0/398190986 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c014480 con 0x7f888c004d90 2024-04-06T12:24:40.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.351+0000 7f88aa778700 1 -- 172.21.15.189:0/398190986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f888c004d90 msgr2=0x7f888c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:40.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.351+0000 7f88aa778700 1 --2- 172.21.15.189:0/398190986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f888c004d90 0x7f888c005210 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f889c009fc0 tx=0x7f888c005880 comp rx=0 tx=0).stop 2024-04-06T12:24:40.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.351+0000 7f88a8d15700 1 -- 172.21.15.189:0/398190986 reap_dead start 2024-04-06T12:24:40.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.351+0000 7f88aa778700 1 -- 172.21.15.189:0/398190986 shutdown_connections 2024-04-06T12:24:40.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.351+0000 7f88aa778700 1 -- 172.21.15.189:0/398190986 >> 172.21.15.189:0/398190986 conn(0x7f88a40b2db0 msgr2=0x7f88a40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:40.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.351+0000 7f88aa778700 1 -- 172.21.15.189:0/398190986 shutdown_connections 2024-04-06T12:24:40.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.351+0000 7f88aa778700 1 -- 172.21.15.189:0/398190986 wait complete. 2024-04-06T12:24:40.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.352+0000 7f88aa778700 1 Processor -- start 2024-04-06T12:24:40.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.352+0000 7f88aa778700 1 -- start start 2024-04-06T12:24:40.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.353+0000 7f88aa778700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88a41f5fb0 0x7f88a41f6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:40.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.353+0000 7f88aa778700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88a40ad2e0 0x7f88a40b0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:40.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.353+0000 7f88aa778700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88a40c6d60 0x7f88a40c71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:40.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.353+0000 7f88aa778700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a40ae910 con 0x7f88a41f5fb0 2024-04-06T12:24:40.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.353+0000 7f88aa778700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a40c7720 con 0x7f88a40ad2e0 2024-04-06T12:24:40.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.353+0000 7f88aa778700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a40b0790 con 0x7f88a40c6d60 2024-04-06T12:24:40.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.353+0000 7f88a3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88a41f5fb0 0x7f88a41f6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:40.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.353+0000 7f88a3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88a41f5fb0 0x7f88a41f6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60436/0 (socket says 172.21.15.189:60436) 2024-04-06T12:24:40.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.353+0000 7f88a3fff700 1 -- 172.21.15.189:0/1633112620 learned_addr learned my addr 172.21.15.189:0/1633112620 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:40.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.353+0000 7f88a37fe700 1 --2- 172.21.15.189:0/1633112620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88a40ad2e0 0x7f88a40b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:40.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.353+0000 7f88a8d15700 1 --2- 172.21.15.189:0/1633112620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88a40c6d60 0x7f88a40c71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:40.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.353+0000 7f88a3fff700 1 -- 172.21.15.189:0/1633112620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88a40c6d60 msgr2=0x7f88a40c71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:40.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.353+0000 7f88a3fff700 1 --2- 172.21.15.189:0/1633112620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88a40c6d60 0x7f88a40c71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:40.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.353+0000 7f88a3fff700 1 -- 172.21.15.189:0/1633112620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88a40ad2e0 msgr2=0x7f88a40b0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:40.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.353+0000 7f88a3fff700 1 --2- 172.21.15.189:0/1633112620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88a40ad2e0 0x7f88a40b0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:40.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.353+0000 7f88a3fff700 1 -- 172.21.15.189:0/1633112620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a40ad760 con 0x7f88a41f5fb0 2024-04-06T12:24:40.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.353+0000 7f88a3fff700 1 --2- 172.21.15.189:0/1633112620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88a41f5fb0 0x7f88a41f6390 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f8898011ad0 tx=0x7f8898011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:40.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.354+0000 7f88a0ff9700 1 -- 172.21.15.189:0/1633112620 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889800f940 con 0x7f88a41f5fb0 2024-04-06T12:24:40.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.354+0000 7f88a0ff9700 1 -- 172.21.15.189:0/1633112620 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889800faa0 con 0x7f88a41f5fb0 2024-04-06T12:24:40.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.354+0000 7f88a0ff9700 1 -- 172.21.15.189:0/1633112620 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889800fe50 con 0x7f88a41f5fb0 2024-04-06T12:24:40.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.354+0000 7f88aa778700 1 -- 172.21.15.189:0/1633112620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a40adf80 con 0x7f88a41f5fb0 2024-04-06T12:24:40.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.354+0000 7f88aa778700 1 -- 172.21.15.189:0/1633112620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a40adca0 con 0x7f88a41f5fb0 2024-04-06T12:24:40.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.355+0000 7f88aa778700 1 -- 172.21.15.189:0/1633112620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a4074990 con 0x7f88a41f5fb0 2024-04-06T12:24:40.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.355+0000 7f88a0ff9700 1 -- 172.21.15.189:0/1633112620 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f8898014440 con 0x7f88a41f5fb0 2024-04-06T12:24:40.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.355+0000 7f88a0ff9700 1 --2- 172.21.15.189:0/1633112620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f889006ae90 0x7f889006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:40.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.356+0000 7f88a37fe700 1 --2- 172.21.15.189:0/1633112620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f889006ae90 0x7f889006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:40.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.356+0000 7f88a0ff9700 1 -- 172.21.15.189:0/1633112620 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6900+0+0 (secure 0 0 0) 0x7f889808a170 con 0x7f88a41f5fb0 2024-04-06T12:24:40.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.356+0000 7f88a37fe700 1 --2- 172.21.15.189:0/1633112620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f889006ae90 0x7f889006d350 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f88a4146600 tx=0x7f8894007980 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:40.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.360+0000 7f88a0ff9700 1 -- 172.21.15.189:0/1633112620 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8898054ce0 con 0x7f88a41f5fb0 2024-04-06T12:24:40.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.581+0000 7f88aa778700 1 -- 172.21.15.189:0/1633112620 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f88a406cc20 con 0x7f889006ae90 2024-04-06T12:24:40.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.673+0000 7f88a0ff9700 1 -- 172.21.15.189:0/1633112620 <== mgr.5469 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f8894007200 con 0x7f889006ae90 2024-04-06T12:24:40.670 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule removed for path /snap_test_dir1/dir9 2024-04-06T12:24:40.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.675+0000 7f88aa778700 1 -- 172.21.15.189:0/1633112620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f889006ae90 msgr2=0x7f889006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:40.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.675+0000 7f88aa778700 1 --2- 172.21.15.189:0/1633112620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f889006ae90 0x7f889006d350 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f88a4146600 tx=0x7f8894007980 comp rx=0 tx=0).stop 2024-04-06T12:24:40.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.675+0000 7f88aa778700 1 -- 172.21.15.189:0/1633112620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88a41f5fb0 msgr2=0x7f88a41f6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:40.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.675+0000 7f88aa778700 1 --2- 172.21.15.189:0/1633112620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88a41f5fb0 0x7f88a41f6390 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f8898011ad0 tx=0x7f8898011de0 comp rx=0 tx=0).stop 2024-04-06T12:24:40.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.675+0000 7f88aa778700 1 -- 172.21.15.189:0/1633112620 shutdown_connections 2024-04-06T12:24:40.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.675+0000 7f88aa778700 1 --2- 172.21.15.189:0/1633112620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88a40c6d60 0x7f88a40c71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:40.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.675+0000 7f88aa778700 1 --2- 172.21.15.189:0/1633112620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88a41f5fb0 0x7f88a41f6390 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:40.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.675+0000 7f88aa778700 1 --2- 172.21.15.189:0/1633112620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f889006ae90 0x7f889006d350 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:40.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.675+0000 7f88aa778700 1 --2- 172.21.15.189:0/1633112620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88a40ad2e0 0x7f88a40b0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:40.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.675+0000 7f88aa778700 1 -- 172.21.15.189:0/1633112620 >> 172.21.15.189:0/1633112620 conn(0x7f88a40b2db0 msgr2=0x7f88a4160410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:40.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.675+0000 7f88aa778700 1 -- 172.21.15.189:0/1633112620 shutdown_connections 2024-04-06T12:24:40.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:40.675+0000 7f88aa778700 1 -- 172.21.15.189:0/1633112620 wait complete. 2024-04-06T12:24:40.684 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2024-04-06T12:24:40.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:40.684 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2024-04-06T12:24:40.703 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:40.703 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:40.703 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:40.703 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:40.703 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:40.703 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:40.703 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:40.703 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:40.703 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:40.704 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:40.704 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:40.704 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_14_00_UTC 2024-04-06T12:24:40.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:40.704 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_14_00_UTC) 2024-04-06T12:24:40.768 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_15_00_UTC 2024-04-06T12:24:40.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:40.768 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_15_00_UTC) 2024-04-06T12:24:40.833 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_16_00_UTC 2024-04-06T12:24:40.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:40.834 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_16_00_UTC) 2024-04-06T12:24:40.897 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_17_00_UTC 2024-04-06T12:24:40.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:40.897 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_17_00_UTC) 2024-04-06T12:24:40.963 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_18_00_UTC 2024-04-06T12:24:40.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:40.963 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_18_00_UTC) 2024-04-06T12:24:41.030 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_19_00_UTC 2024-04-06T12:24:41.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:41.030 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_19_00_UTC) 2024-04-06T12:24:41.099 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_20_00_UTC 2024-04-06T12:24:41.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:41.100 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_20_00_UTC) 2024-04-06T12:24:41.165 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_21_00_UTC 2024-04-06T12:24:41.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:41.166 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_21_00_UTC) 2024-04-06T12:24:41.229 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_22_00_UTC 2024-04-06T12:24:41.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:41.230 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_22_00_UTC) 2024-04-06T12:24:41.294 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_23_00_UTC 2024-04-06T12:24:41.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:41.294 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_23_00_UTC) 2024-04-06T12:24:41.357 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_24_00_UTC 2024-04-06T12:24:41.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:41.357 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-06-12_24_00_UTC) 2024-04-06T12:24:41.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:41.421 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2024-04-06T12:24:41.486 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-06T12:24:41.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.648+0000 7f5111251700 1 Processor -- start 2024-04-06T12:24:41.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.649+0000 7f5111251700 1 -- start start 2024-04-06T12:24:41.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.649+0000 7f5111251700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f510c0b4660 0x7f510c0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:41.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.649+0000 7f5111251700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f510c0b2fc0 0x7f510c0b3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:41.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.649+0000 7f5111251700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510c0b5030 con 0x7f510c15a550 2024-04-06T12:24:41.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.649+0000 7f5111251700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f510c0cdbe0 con 0x7f510c0b4660 2024-04-06T12:24:41.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.649+0000 7f5111251700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f510c15a940 con 0x7f510c0b2fc0 2024-04-06T12:24:41.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.649+0000 7f510a59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f510c0b4660 0x7f510c0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:41.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.649+0000 7f510ad9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f510c15a550 0x7f510c0cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47092/0 (socket says 172.21.15.189:47092) 2024-04-06T12:24:41.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.649+0000 7f510ad9d700 1 -- 172.21.15.189:0/2628550522 learned_addr learned my addr 172.21.15.189:0/2628550522 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:41.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.650+0000 7f510a59c700 1 -- 172.21.15.189:0/2628550522 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f510c0b2fc0 msgr2=0x7f510c0b3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:41.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.650+0000 7f510a59c700 1 --2- 172.21.15.189:0/2628550522 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f510c0b2fc0 0x7f510c0b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:41.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.650+0000 7f510a59c700 1 -- 172.21.15.189:0/2628550522 >> v1:172.21.15.189:6789/0 conn(0x7f510c15a550 legacy=0x7f510c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:41.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.650+0000 7f510a59c700 1 -- 172.21.15.189:0/2628550522 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510c15ac00 con 0x7f510c0b4660 2024-04-06T12:24:41.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.650+0000 7f5109d9b700 1 -- 172.21.15.189:0/2628550522 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478952863 0 0) 0x7f510c0b5030 con 0x7f510c15a550 2024-04-06T12:24:41.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.650+0000 7f510a59c700 1 --2- 172.21.15.189:0/2628550522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f510c0b4660 0x7f510c0b4af0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f50f4009840 tx=0x7f50f4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3071034fd7d0db server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:41.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.650+0000 7f5109d9b700 1 -- 172.21.15.189:0/2628550522 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f4011830 con 0x7f510c0b4660 2024-04-06T12:24:41.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.650+0000 7f5109d9b700 1 -- 172.21.15.189:0/2628550522 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f4011990 con 0x7f510c0b4660 2024-04-06T12:24:41.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.650+0000 7f5109d9b700 1 -- 172.21.15.189:0/2628550522 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f4011d20 con 0x7f510c0b4660 2024-04-06T12:24:41.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.651+0000 7f5111251700 1 -- 172.21.15.189:0/2628550522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f510c0b4660 msgr2=0x7f510c0b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:41.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.651+0000 7f5111251700 1 --2- 172.21.15.189:0/2628550522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f510c0b4660 0x7f510c0b4af0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f50f4009840 tx=0x7f50f4009b50 comp rx=0 tx=0).stop 2024-04-06T12:24:41.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.651+0000 7f5111251700 1 -- 172.21.15.189:0/2628550522 shutdown_connections 2024-04-06T12:24:41.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.651+0000 7f5111251700 1 --2- 172.21.15.189:0/2628550522 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f510c0b2fc0 0x7f510c0b3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:41.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.651+0000 7f5111251700 1 --2- 172.21.15.189:0/2628550522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f510c0b4660 0x7f510c0b4af0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:41.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.651+0000 7f5111251700 1 -- 172.21.15.189:0/2628550522 >> 172.21.15.189:0/2628550522 conn(0x7f510c0d00b0 msgr2=0x7f510c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:41.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.651+0000 7f5111251700 1 -- 172.21.15.189:0/2628550522 shutdown_connections 2024-04-06T12:24:41.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.651+0000 7f5111251700 1 -- 172.21.15.189:0/2628550522 wait complete. 2024-04-06T12:24:41.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.652+0000 7f5111251700 1 Processor -- start 2024-04-06T12:24:41.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.652+0000 7f5111251700 1 -- start start 2024-04-06T12:24:41.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.653+0000 7f5111251700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f510c0b2fc0 0x7f510c153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:41.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.653+0000 7f5111251700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f510c0b4660 0x7f510c15bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:41.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.653+0000 7f5111251700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f510c0aad60 0x7f510c06a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:41.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.653+0000 7f5111251700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510c154150 con 0x7f510c0aad60 2024-04-06T12:24:41.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.653+0000 7f5111251700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510c15c880 con 0x7f510c0b2fc0 2024-04-06T12:24:41.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.653+0000 7f5111251700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f510c15cb60 con 0x7f510c0b4660 2024-04-06T12:24:41.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.653+0000 7f510b59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f510c0aad60 0x7f510c06a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:41.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.653+0000 7f510b59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f510c0aad60 0x7f510c06a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60446/0 (socket says 172.21.15.189:60446) 2024-04-06T12:24:41.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.653+0000 7f510b59e700 1 -- 172.21.15.189:0/3753056121 learned_addr learned my addr 172.21.15.189:0/3753056121 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:41.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.653+0000 7f510a59c700 1 --2- 172.21.15.189:0/3753056121 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f510c0b4660 0x7f510c15bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:41.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.653+0000 7f510ad9d700 1 --2- 172.21.15.189:0/3753056121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f510c0b2fc0 0x7f510c153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:41.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.653+0000 7f510b59e700 1 -- 172.21.15.189:0/3753056121 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f510c0b4660 msgr2=0x7f510c15bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:41.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.653+0000 7f510b59e700 1 --2- 172.21.15.189:0/3753056121 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f510c0b4660 0x7f510c15bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:41.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.653+0000 7f510b59e700 1 -- 172.21.15.189:0/3753056121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f510c0b2fc0 msgr2=0x7f510c153c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:41.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.653+0000 7f510b59e700 1 --2- 172.21.15.189:0/3753056121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f510c0b2fc0 0x7f510c153c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:41.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.653+0000 7f510b59e700 1 -- 172.21.15.189:0/3753056121 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f40094a0 con 0x7f510c0aad60 2024-04-06T12:24:41.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.654+0000 7f510b59e700 1 --2- 172.21.15.189:0/3753056121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f510c0aad60 0x7f510c06a760 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f510400b970 tx=0x7f510400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:41.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.654+0000 7f51037fe700 1 -- 172.21.15.189:0/3753056121 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510400d860 con 0x7f510c0aad60 2024-04-06T12:24:41.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.654+0000 7f51037fe700 1 -- 172.21.15.189:0/3753056121 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510400d9e0 con 0x7f510c0aad60 2024-04-06T12:24:41.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.654+0000 7f51037fe700 1 -- 172.21.15.189:0/3753056121 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510400dd90 con 0x7f510c0aad60 2024-04-06T12:24:41.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.654+0000 7f5111251700 1 -- 172.21.15.189:0/3753056121 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f510c15ce40 con 0x7f510c0aad60 2024-04-06T12:24:41.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.654+0000 7f5111251700 1 -- 172.21.15.189:0/3753056121 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f510c15d400 con 0x7f510c0aad60 2024-04-06T12:24:41.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.655+0000 7f5111251700 1 -- 172.21.15.189:0/3753056121 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f510c067bb0 con 0x7f510c0aad60 2024-04-06T12:24:41.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.655+0000 7f51037fe700 1 -- 172.21.15.189:0/3753056121 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 14) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f510400e430 con 0x7f510c0aad60 2024-04-06T12:24:41.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.656+0000 7f51037fe700 1 --2- 172.21.15.189:0/3753056121 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f50ec06ae90 0x7f50ec06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:41.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.656+0000 7f510ad9d700 1 --2- 172.21.15.189:0/3753056121 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f50ec06ae90 0x7f50ec06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:41.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.656+0000 7f51037fe700 1 -- 172.21.15.189:0/3753056121 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7392+0+0 (secure 0 0 0) 0x7f5104089f90 con 0x7f510c0aad60 2024-04-06T12:24:41.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.656+0000 7f510ad9d700 1 --2- 172.21.15.189:0/3753056121 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f50ec06ae90 0x7f50ec06d350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f50fc007fd0 tx=0x7f50fc007de0 comp rx=0 tx=0).ready entity=mgr.5469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:41.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.660+0000 7f51037fe700 1 -- 172.21.15.189:0/3753056121 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f51040bd650 con 0x7f510c0aad60 2024-04-06T12:24:41.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:41.934+0000 7f5111251700 1 -- 172.21.15.189:0/3753056121 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f510c074990 con 0x7f510c0aad60 2024-04-06T12:24:42.102 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:24:42.105 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:24:42.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.600+0000 7f51037fe700 1 -- 172.21.15.189:0/3753056121 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 15) v1 ==== 89316+0+0 (secure 0 0 0) 0x7f5104054b20 con 0x7f510c0aad60 2024-04-06T12:24:42.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.601+0000 7f51037fe700 1 -- 172.21.15.189:0/3753056121 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v15) v1 ==== 93+0+0 (secure 0 0 0) 0x7f5104072320 con 0x7f510c0aad60 2024-04-06T12:24:42.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.603+0000 7f5111251700 1 -- 172.21.15.189:0/3753056121 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f50ec06ae90 msgr2=0x7f50ec06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:42.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.603+0000 7f5111251700 1 --2- 172.21.15.189:0/3753056121 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f50ec06ae90 0x7f50ec06d350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f50fc007fd0 tx=0x7f50fc007de0 comp rx=0 tx=0).stop 2024-04-06T12:24:42.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.603+0000 7f5111251700 1 -- 172.21.15.189:0/3753056121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f510c0aad60 msgr2=0x7f510c06a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:42.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.603+0000 7f5111251700 1 --2- 172.21.15.189:0/3753056121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f510c0aad60 0x7f510c06a760 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f510400b970 tx=0x7f510400bc80 comp rx=0 tx=0).stop 2024-04-06T12:24:42.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.603+0000 7f5111251700 1 -- 172.21.15.189:0/3753056121 shutdown_connections 2024-04-06T12:24:42.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.603+0000 7f5111251700 1 --2- 172.21.15.189:0/3753056121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f510c0aad60 0x7f510c06a760 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:42.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.603+0000 7f5111251700 1 --2- 172.21.15.189:0/3753056121 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f50ec06ae90 0x7f50ec06d350 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:42.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.603+0000 7f5111251700 1 --2- 172.21.15.189:0/3753056121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f510c0b2fc0 0x7f510c153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:42.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.603+0000 7f5111251700 1 --2- 172.21.15.189:0/3753056121 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f510c0b4660 0x7f510c15bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:42.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.603+0000 7f5111251700 1 -- 172.21.15.189:0/3753056121 >> 172.21.15.189:0/3753056121 conn(0x7f510c0d00b0 msgr2=0x7f510c0cddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:42.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.603+0000 7f5111251700 1 -- 172.21.15.189:0/3753056121 shutdown_connections 2024-04-06T12:24:42.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.604+0000 7f5111251700 1 -- 172.21.15.189:0/3753056121 wait complete. 2024-04-06T12:24:42.615 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T12:24:42.615 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:24:42.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:24:42.697 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T12:24:42.697 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:24:42.697 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:24:42.717 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:24:42.717 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:24:42.795 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:24:42.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.961+0000 7f42e52a5700 1 Processor -- start 2024-04-06T12:24:42.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.961+0000 7f42e52a5700 1 -- start start 2024-04-06T12:24:42.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.962+0000 7f42e52a5700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f42e00b4ed0 0x7f42e00b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:42.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.962+0000 7f42e52a5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f42e00d2880 0x7f42e00d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:42.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.962+0000 7f42e52a5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e00b5cc0 con 0x7f42e015d590 2024-04-06T12:24:42.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.962+0000 7f42e52a5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e00b58c0 con 0x7f42e00d2880 2024-04-06T12:24:42.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.962+0000 7f42e52a5700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e00d1500 con 0x7f42e00b4ed0 2024-04-06T12:24:42.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.962+0000 7f42df7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f42e00d2880 0x7f42e00d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:42.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.962+0000 7f42df7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f42e00d2880 0x7f42e00d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60452/0 (socket says 172.21.15.189:60452) 2024-04-06T12:24:42.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.962+0000 7f42df7fe700 1 -- 172.21.15.189:0/68937258 learned_addr learned my addr 172.21.15.189:0/68937258 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:42.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.962+0000 7f42de7fc700 1 --2- 172.21.15.189:0/68937258 >> v2:172.21.15.192:3300/0 conn(0x7f42e00b4ed0 0x7f42e00b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:42.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.963+0000 7f42df7fe700 1 -- 172.21.15.189:0/68937258 >> v2:172.21.15.192:3300/0 conn(0x7f42e00b4ed0 msgr2=0x7f42e00b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:42.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.963+0000 7f42df7fe700 1 --2- 172.21.15.189:0/68937258 >> v2:172.21.15.192:3300/0 conn(0x7f42e00b4ed0 0x7f42e00b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:42.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.963+0000 7f42df7fe700 1 -- 172.21.15.189:0/68937258 >> v1:172.21.15.192:6789/0 conn(0x7f42e015d590 legacy=0x7f42e00b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:42.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.963+0000 7f42df7fe700 1 -- 172.21.15.189:0/68937258 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e00d17e0 con 0x7f42e00d2880 2024-04-06T12:24:42.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.963+0000 7f42df7fe700 1 --2- 172.21.15.189:0/68937258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42e00d2880 0x7f42e00d2cd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f42e00b6330 tx=0x7f42d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f367f9e090efe834 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:42.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.963+0000 7f42ddffb700 1 -- 172.21.15.189:0/68937258 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d800e070 con 0x7f42e00d2880 2024-04-06T12:24:42.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.963+0000 7f42ddffb700 1 -- 172.21.15.189:0/68937258 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d800b910 con 0x7f42e00d2880 2024-04-06T12:24:42.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.963+0000 7f42ddffb700 1 -- 172.21.15.189:0/68937258 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d800bcc0 con 0x7f42e00d2880 2024-04-06T12:24:42.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.963+0000 7f42e52a5700 1 -- 172.21.15.189:0/68937258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42e00d2880 msgr2=0x7f42e00d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:42.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.963+0000 7f42e52a5700 1 --2- 172.21.15.189:0/68937258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42e00d2880 0x7f42e00d2cd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f42e00b6330 tx=0x7f42d800b330 comp rx=0 tx=0).stop 2024-04-06T12:24:42.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.963+0000 7f42e52a5700 1 -- 172.21.15.189:0/68937258 shutdown_connections 2024-04-06T12:24:42.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.963+0000 7f42e52a5700 1 --2- 172.21.15.189:0/68937258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42e00d2880 0x7f42e00d2cd0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:42.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.963+0000 7f42e52a5700 1 --2- 172.21.15.189:0/68937258 >> v2:172.21.15.192:3300/0 conn(0x7f42e00b4ed0 0x7f42e00b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:42.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.964+0000 7f42e52a5700 1 -- 172.21.15.189:0/68937258 >> 172.21.15.189:0/68937258 conn(0x7f42e006bbc0 msgr2=0x7f42e006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:42.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.964+0000 7f42e52a5700 1 -- 172.21.15.189:0/68937258 shutdown_connections 2024-04-06T12:24:42.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.964+0000 7f42e52a5700 1 -- 172.21.15.189:0/68937258 wait complete. 2024-04-06T12:24:42.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.965+0000 7f42e52a5700 1 Processor -- start 2024-04-06T12:24:42.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.965+0000 7f42e52a5700 1 -- start start 2024-04-06T12:24:42.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42e52a5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42e00b4ed0 0x7f42e0158610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:42.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42e52a5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42e00d2880 0x7f42e01f24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:42.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42e52a5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42e015d590 0x7f42e00cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:42.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42e52a5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e00ca830 con 0x7f42e015d590 2024-04-06T12:24:42.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42e52a5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e00ca520 con 0x7f42e00d2880 2024-04-06T12:24:42.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42e52a5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e0158e40 con 0x7f42e00b4ed0 2024-04-06T12:24:42.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42df7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42e015d590 0x7f42e00cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:42.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42df7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42e015d590 0x7f42e00cb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60458/0 (socket says 172.21.15.189:60458) 2024-04-06T12:24:42.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42df7fe700 1 -- 172.21.15.189:0/567591786 learned_addr learned my addr 172.21.15.189:0/567591786 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:42.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42de7fc700 1 --2- 172.21.15.189:0/567591786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42e00d2880 0x7f42e01f24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:42.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42deffd700 1 --2- 172.21.15.189:0/567591786 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42e00b4ed0 0x7f42e0158610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:42.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42df7fe700 1 -- 172.21.15.189:0/567591786 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42e00b4ed0 msgr2=0x7f42e0158610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:42.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42df7fe700 1 --2- 172.21.15.189:0/567591786 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42e00b4ed0 0x7f42e0158610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:42.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42df7fe700 1 -- 172.21.15.189:0/567591786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42e00d2880 msgr2=0x7f42e01f24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:42.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42df7fe700 1 --2- 172.21.15.189:0/567591786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42e00d2880 0x7f42e01f24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:42.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42df7fe700 1 -- 172.21.15.189:0/567591786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d8009b30 con 0x7f42e015d590 2024-04-06T12:24:42.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42df7fe700 1 --2- 172.21.15.189:0/567591786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42e015d590 0x7f42e00cb960 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f42e00d41b0 tx=0x7f42d8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:42.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42c77fe700 1 -- 172.21.15.189:0/567591786 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d8004880 con 0x7f42e015d590 2024-04-06T12:24:42.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42c77fe700 1 -- 172.21.15.189:0/567591786 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d80099b0 con 0x7f42e015d590 2024-04-06T12:24:42.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42e52a5700 1 -- 172.21.15.189:0/567591786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e0159120 con 0x7f42e015d590 2024-04-06T12:24:42.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.966+0000 7f42c77fe700 1 -- 172.21.15.189:0/567591786 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d801c680 con 0x7f42e015d590 2024-04-06T12:24:42.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.967+0000 7f42e52a5700 1 -- 172.21.15.189:0/567591786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e0158b50 con 0x7f42e015d590 2024-04-06T12:24:42.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.967+0000 7f42e52a5700 1 -- 172.21.15.189:0/567591786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e00ca970 con 0x7f42e015d590 2024-04-06T12:24:42.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.968+0000 7f42c77fe700 1 -- 172.21.15.189:0/567591786 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 15) v1 ==== 89316+0+0 (secure 0 0 0) 0x7f42d801c7e0 con 0x7f42e015d590 2024-04-06T12:24:42.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.968+0000 7f42c77fe700 1 --2- 172.21.15.189:0/567591786 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42cc06ae40 0x7f42cc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:42.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.968+0000 7f42deffd700 1 -- 172.21.15.189:0/567591786 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42cc06ae40 msgr2=0x7f42cc06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:42.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.968+0000 7f42c77fe700 1 -- 172.21.15.189:0/567591786 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f42d808b7f0 con 0x7f42e015d590 2024-04-06T12:24:42.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.968+0000 7f42deffd700 1 --2- 172.21.15.189:0/567591786 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42cc06ae40 0x7f42cc06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:24:42.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:42.973+0000 7f42c77fe700 1 -- 172.21.15.189:0/567591786 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f42e00ca970 con 0x7f42e015d590 2024-04-06T12:24:43.018 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:43.021+0000 7f44d481c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:24:43.032 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:43.034+0000 7f51465c0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:24:43.095 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:43.098+0000 7f44d481c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:24:43.113 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:43.115+0000 7f51465c0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:24:43.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.169+0000 7f42deffd700 1 -- 172.21.15.189:0/567591786 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42cc06ae40 msgr2=0x7f42cc06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:43.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.169+0000 7f42deffd700 1 --2- 172.21.15.189:0/567591786 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42cc06ae40 0x7f42cc06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:24:43.167 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:43.170+0000 7f44d481c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:24:43.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.178+0000 7f42e52a5700 1 -- 172.21.15.189:0/567591786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42e00ca970 con 0x7f42e015d590 2024-04-06T12:24:43.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.180+0000 7f42c77fe700 1 -- 172.21.15.189:0/567591786 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+17818 (secure 0 0 0) 0x7f42e00ca970 con 0x7f42e015d590 2024-04-06T12:24:43.178 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:43.178 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":51,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:24:42.483517+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:13:19.268280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-06T12:13:20.243771+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":220,"snap_epoch":51,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.35","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/155214328":"2024-04-07T12:13:34.836450+0000","172.21.15.189:0/3092563682":"2024-04-07T12:13:34.836450+0000","172.21.15.189:0/1075995033":"2024-04-07T12:13:34.836450+0000","172.21.15.189:6833/47128":"2024-04-07T12:13:34.836450+0000","172.21.15.189:0/3015633422":"2024-04-07T12:13:34.836450+0000","172.21.15.189:6832/47128":"2024-04-07T12:13:34.836450+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":2,"length":1},{"begin":5,"length":2},{"begin":9,"length":1},{"begin":11,"length":1},{"begin":13,"length":2},{"begin":17,"length":2},{"begin":21,"length":2},{"begin":24,"length":2},{"begin":28,"length":2},{"begin":33,"length":1},{"begin":35,"length":2},{"begin":39,"length":2},{"begin":43,"length":1},{"begin":45,"length":2},{"begin":49,"length":3},{"begin":53,"length":1},{"begin":55,"length":2},{"begin":59,"length":4},{"begin":64,"length":1},{"begin":66,"length":2},{"begin":70,"length":3},{"begin":74,"length":1},{"begin":76,"length":2},{"begin":80,"length":5},{"begin":86,"length":1},{"begin":88,"length":2},{"begin":93,"length":3},{"begin":97,"length":1},{"begin":99,"length":2},{"begin":103,"length":3},{"begin":109,"length":2},{"begin":160,"length":61}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":9,"length":1},{"begin":21,"length":2},{"begin":33,"length":1},{"begin":43,"length":1},{"begin":53,"length":1},{"begin":64,"length":1},{"begin":74,"length":1},{"begin":86,"length":1},{"begin":97,"length":1},{"begin":103,"length":1},{"begin":210,"length":11}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:24:43.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.182+0000 7f42e52a5700 1 -- 172.21.15.189:0/567591786 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42cc06ae40 msgr2=0x7f42cc06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:24:43.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.182+0000 7f42e52a5700 1 --2- 172.21.15.189:0/567591786 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42cc06ae40 0x7f42cc06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.182+0000 7f42e52a5700 1 -- 172.21.15.189:0/567591786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42e015d590 msgr2=0x7f42e00cb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:43.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.182+0000 7f42e52a5700 1 --2- 172.21.15.189:0/567591786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42e015d590 0x7f42e00cb960 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f42e00d41b0 tx=0x7f42d8004ad0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.182+0000 7f42e52a5700 1 -- 172.21.15.189:0/567591786 shutdown_connections 2024-04-06T12:24:43.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.182+0000 7f42e52a5700 1 --2- 172.21.15.189:0/567591786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42e015d590 0x7f42e00cb960 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.182+0000 7f42e52a5700 1 --2- 172.21.15.189:0/567591786 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42e00b4ed0 0x7f42e0158610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.182+0000 7f42e52a5700 1 --2- 172.21.15.189:0/567591786 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42cc06ae40 0x7f42cc06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.182+0000 7f42e52a5700 1 --2- 172.21.15.189:0/567591786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42e00d2880 0x7f42e01f24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.182+0000 7f42e52a5700 1 -- 172.21.15.189:0/567591786 >> 172.21.15.189:0/567591786 conn(0x7f42e006bbc0 msgr2=0x7f42e0071e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:43.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.182+0000 7f42e52a5700 1 -- 172.21.15.189:0/567591786 shutdown_connections 2024-04-06T12:24:43.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.182+0000 7f42e52a5700 1 -- 172.21.15.189:0/567591786 wait complete. 2024-04-06T12:24:43.189 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:43.191+0000 7f51465c0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:24:43.193 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:13:19.268280+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:13:20.243771+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 220, 'snap_epoch': 51, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:24:43.194 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:24:43.234 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:43.237+0000 7f44d481c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:24:43.264 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:43.265+0000 7f51465c0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:24:43.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.325+0000 7f462689b700 1 Processor -- start 2024-04-06T12:24:43.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.325+0000 7f462689b700 1 -- start start 2024-04-06T12:24:43.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.326+0000 7f462689b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f46200d17a0 0x7f462006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:43.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.326+0000 7f462689b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f46200cedf0 0x7f46200cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:43.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.326+0000 7f462689b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46200cf730 con 0x7f4620146dd0 2024-04-06T12:24:43.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.326+0000 7f462689b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f462006d850 con 0x7f46200cedf0 2024-04-06T12:24:43.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.326+0000 7f462689b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f462006c0a0 con 0x7f46200d17a0 2024-04-06T12:24:43.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.326+0000 7f4624e38700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f46200cedf0 0x7f46200cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:43.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.326+0000 7f461f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4620146dd0 0x7f462006d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47102/0 (socket says 172.21.15.189:47102) 2024-04-06T12:24:43.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.326+0000 7f461f7fe700 1 -- 172.21.15.189:0/4203687703 learned_addr learned my addr 172.21.15.189:0/4203687703 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:43.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.326+0000 7f461ffff700 1 --2- 172.21.15.189:0/4203687703 >> v2:172.21.15.192:3300/0 conn(0x7f46200d17a0 0x7f462006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:43.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.326+0000 7f461effd700 1 -- 172.21.15.189:0/4203687703 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323240974 0 0) 0x7f46200cf730 con 0x7f4620146dd0 2024-04-06T12:24:43.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.326+0000 7f461effd700 1 -- 172.21.15.189:0/4203687703 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46080035b0 con 0x7f4620146dd0 2024-04-06T12:24:43.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.326+0000 7f4624e38700 1 -- 172.21.15.189:0/4203687703 >> v2:172.21.15.192:3300/0 conn(0x7f46200d17a0 msgr2=0x7f462006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:43.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.326+0000 7f4624e38700 1 --2- 172.21.15.189:0/4203687703 >> v2:172.21.15.192:3300/0 conn(0x7f46200d17a0 0x7f462006b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.326+0000 7f4624e38700 1 -- 172.21.15.189:0/4203687703 >> v1:172.21.15.189:6789/0 conn(0x7f4620146dd0 legacy=0x7f462006d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:43.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.326+0000 7f4624e38700 1 -- 172.21.15.189:0/4203687703 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f462006c380 con 0x7f46200cedf0 2024-04-06T12:24:43.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.327+0000 7f4624e38700 1 --2- 172.21.15.189:0/4203687703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46200cedf0 0x7f46200cf1f0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f462004fb40 tx=0x7f461800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf9cb6a534c6a024 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:43.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.327+0000 7f461effd700 1 -- 172.21.15.189:0/4203687703 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461800e070 con 0x7f46200cedf0 2024-04-06T12:24:43.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.327+0000 7f461effd700 1 -- 172.21.15.189:0/4203687703 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461800bb20 con 0x7f46200cedf0 2024-04-06T12:24:43.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.327+0000 7f461effd700 1 -- 172.21.15.189:0/4203687703 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461800be90 con 0x7f46200cedf0 2024-04-06T12:24:43.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.327+0000 7f462689b700 1 -- 172.21.15.189:0/4203687703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46200cedf0 msgr2=0x7f46200cf1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:43.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.327+0000 7f462689b700 1 --2- 172.21.15.189:0/4203687703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46200cedf0 0x7f46200cf1f0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f462004fb40 tx=0x7f461800b330 comp rx=0 tx=0).stop 2024-04-06T12:24:43.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.327+0000 7f462689b700 1 -- 172.21.15.189:0/4203687703 shutdown_connections 2024-04-06T12:24:43.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.327+0000 7f462689b700 1 --2- 172.21.15.189:0/4203687703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46200cedf0 0x7f46200cf1f0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.327+0000 7f462689b700 1 --2- 172.21.15.189:0/4203687703 >> v2:172.21.15.192:3300/0 conn(0x7f46200d17a0 0x7f462006b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.327+0000 7f462689b700 1 -- 172.21.15.189:0/4203687703 >> 172.21.15.189:0/4203687703 conn(0x7f46200b2db0 msgr2=0x7f46200b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:43.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.328+0000 7f462689b700 1 -- 172.21.15.189:0/4203687703 shutdown_connections 2024-04-06T12:24:43.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.328+0000 7f462689b700 1 -- 172.21.15.189:0/4203687703 wait complete. 2024-04-06T12:24:43.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.329+0000 7f462689b700 1 Processor -- start 2024-04-06T12:24:43.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.329+0000 7f462689b700 1 -- start start 2024-04-06T12:24:43.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.330+0000 7f462689b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46200cedf0 0x7f46201580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:43.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.330+0000 7f462689b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46200d17a0 0x7f462015e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:43.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.330+0000 7f462689b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4620146dd0 0x7f46200ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:43.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.330+0000 7f462689b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462015f460 con 0x7f4620146dd0 2024-04-06T12:24:43.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.330+0000 7f462689b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462015f740 con 0x7f46200cedf0 2024-04-06T12:24:43.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.330+0000 7f462689b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f462015fa20 con 0x7f46200d17a0 2024-04-06T12:24:43.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.330+0000 7f4624e38700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4620146dd0 0x7f46200ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:43.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.330+0000 7f461ffff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46200cedf0 0x7f46201580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:43.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.330+0000 7f4624e38700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4620146dd0 0x7f46200ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60470/0 (socket says 172.21.15.189:60470) 2024-04-06T12:24:43.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.330+0000 7f4624e38700 1 -- 172.21.15.189:0/4077071395 learned_addr learned my addr 172.21.15.189:0/4077071395 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:43.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.330+0000 7f461f7fe700 1 --2- 172.21.15.189:0/4077071395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46200d17a0 0x7f462015e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:43.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.331+0000 7f461ffff700 1 -- 172.21.15.189:0/4077071395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46200d17a0 msgr2=0x7f462015e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:43.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.331+0000 7f461ffff700 1 --2- 172.21.15.189:0/4077071395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46200d17a0 0x7f462015e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.331+0000 7f461ffff700 1 -- 172.21.15.189:0/4077071395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4620146dd0 msgr2=0x7f46200ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:43.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.331+0000 7f461ffff700 1 --2- 172.21.15.189:0/4077071395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4620146dd0 0x7f46200ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.331+0000 7f461ffff700 1 -- 172.21.15.189:0/4077071395 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4618009a70 con 0x7f46200cedf0 2024-04-06T12:24:43.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.331+0000 7f4624e38700 1 --2- 172.21.15.189:0/4077071395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4620146dd0 0x7f46200ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:43.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.331+0000 7f461ffff700 1 --2- 172.21.15.189:0/4077071395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46200cedf0 0x7f46201580b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f461000e870 tx=0x7f461000eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:43.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.331+0000 7f461cff9700 1 -- 172.21.15.189:0/4077071395 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4610013680 con 0x7f46200cedf0 2024-04-06T12:24:43.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.331+0000 7f461cff9700 1 -- 172.21.15.189:0/4077071395 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46100137e0 con 0x7f46200cedf0 2024-04-06T12:24:43.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.331+0000 7f462689b700 1 -- 172.21.15.189:0/4077071395 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f462015fd00 con 0x7f46200cedf0 2024-04-06T12:24:43.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.331+0000 7f461cff9700 1 -- 172.21.15.189:0/4077071395 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4610013b90 con 0x7f46200cedf0 2024-04-06T12:24:43.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.331+0000 7f462689b700 1 -- 172.21.15.189:0/4077071395 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f462015b600 con 0x7f46200cedf0 2024-04-06T12:24:43.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.332+0000 7f462689b700 1 -- 172.21.15.189:0/4077071395 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4620160350 con 0x7f46200cedf0 2024-04-06T12:24:43.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.332+0000 7f461cff9700 1 -- 172.21.15.189:0/4077071395 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 15) v1 ==== 89316+0+0 (secure 0 0 0) 0x7f4610013cf0 con 0x7f46200cedf0 2024-04-06T12:24:43.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.333+0000 7f461cff9700 1 --2- 172.21.15.189:0/4077071395 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f460c06ae40 0x7f460c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:43.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.333+0000 7f461f7fe700 1 -- 172.21.15.189:0/4077071395 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f460c06ae40 msgr2=0x7f460c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:43.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.333+0000 7f461cff9700 1 -- 172.21.15.189:0/4077071395 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f4610089a00 con 0x7f46200cedf0 2024-04-06T12:24:43.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.333+0000 7f461f7fe700 1 --2- 172.21.15.189:0/4077071395 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f460c06ae40 0x7f460c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:24:43.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.337+0000 7f461cff9700 1 -- 172.21.15.189:0/4077071395 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4610054390 con 0x7f46200cedf0 2024-04-06T12:24:43.409 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:43.412+0000 7f44d481c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:24:43.447 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:43.449+0000 7f51465c0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:24:43.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.534+0000 7f461f7fe700 1 -- 172.21.15.189:0/4077071395 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f460c06ae40 msgr2=0x7f460c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:43.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.534+0000 7f461f7fe700 1 --2- 172.21.15.189:0/4077071395 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f460c06ae40 0x7f460c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:24:43.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.539+0000 7f462689b700 1 -- 172.21.15.189:0/4077071395 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4620074990 con 0x7f46200cedf0 2024-04-06T12:24:43.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.540+0000 7f461cff9700 1 -- 172.21.15.189:0/4077071395 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) v1 ==== 93+0+10 (secure 0 0 0) 0x7f46100579b0 con 0x7f46200cedf0 2024-04-06T12:24:43.538 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:24:43.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.542+0000 7f462689b700 1 -- 172.21.15.189:0/4077071395 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f460c06ae40 msgr2=0x7f460c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:24:43.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.542+0000 7f462689b700 1 --2- 172.21.15.189:0/4077071395 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f460c06ae40 0x7f460c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.542+0000 7f462689b700 1 -- 172.21.15.189:0/4077071395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46200cedf0 msgr2=0x7f46201580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:43.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.542+0000 7f462689b700 1 --2- 172.21.15.189:0/4077071395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46200cedf0 0x7f46201580b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f461000e870 tx=0x7f461000eb80 comp rx=0 tx=0).stop 2024-04-06T12:24:43.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.542+0000 7f462689b700 1 -- 172.21.15.189:0/4077071395 shutdown_connections 2024-04-06T12:24:43.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.542+0000 7f462689b700 1 --2- 172.21.15.189:0/4077071395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4620146dd0 0x7f46200ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.542+0000 7f462689b700 1 --2- 172.21.15.189:0/4077071395 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f460c06ae40 0x7f460c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.542+0000 7f462689b700 1 --2- 172.21.15.189:0/4077071395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46200cedf0 0x7f46201580b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.542+0000 7f462689b700 1 --2- 172.21.15.189:0/4077071395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46200d17a0 0x7f462015e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.542+0000 7f462689b700 1 -- 172.21.15.189:0/4077071395 >> 172.21.15.189:0/4077071395 conn(0x7f46200b2db0 msgr2=0x7f46200b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:43.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.542+0000 7f462689b700 1 -- 172.21.15.189:0/4077071395 shutdown_connections 2024-04-06T12:24:43.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.542+0000 7f462689b700 1 -- 172.21.15.189:0/4077071395 wait complete. 2024-04-06T12:24:43.553 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:24:43.564 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:43.567+0000 7f44d481c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:24:43.693 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:43.612+0000 7f51465c0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:24:43.693 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:43.676+0000 7f44d481c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:24:43.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.685+0000 7faacda2d700 1 Processor -- start 2024-04-06T12:24:43.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.685+0000 7faacda2d700 1 -- start start 2024-04-06T12:24:43.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.686+0000 7faacda2d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7faac80d11b0 0x7faac80d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:43.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.686+0000 7faacda2d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faac80cb870 0x7faac815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:43.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.686+0000 7faacda2d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faac80d1ad0 con 0x7faac80cfa00 2024-04-06T12:24:43.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.686+0000 7faacda2d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faac80ccdc0 con 0x7faac80d11b0 2024-04-06T12:24:43.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.686+0000 7faacda2d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faac80d0b50 con 0x7faac80cb870 2024-04-06T12:24:43.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.686+0000 7faac77fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7faac80cfa00 0x7faac80cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47106/0 (socket says 172.21.15.189:47106) 2024-04-06T12:24:43.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.686+0000 7faac77fe700 1 -- 172.21.15.189:0/3580180528 learned_addr learned my addr 172.21.15.189:0/3580180528 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:43.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.686+0000 7faac6ffd700 1 --2- 172.21.15.189:0/3580180528 >> v2:172.21.15.192:3300/0 conn(0x7faac80d11b0 0x7faac80d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:43.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.686+0000 7faac67fc700 1 --2- 172.21.15.189:0/3580180528 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faac80cb870 0x7faac815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:43.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.686+0000 7faac5ffb700 1 -- 172.21.15.189:0/3580180528 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748604889 0 0) 0x7faac80d1ad0 con 0x7faac80cfa00 2024-04-06T12:24:43.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.686+0000 7faac5ffb700 1 -- 172.21.15.189:0/3580180528 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab00035b0 con 0x7faac80cfa00 2024-04-06T12:24:43.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.686+0000 7faac6ffd700 1 -- 172.21.15.189:0/3580180528 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faac80cb870 msgr2=0x7faac815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:43.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.686+0000 7faac6ffd700 1 --2- 172.21.15.189:0/3580180528 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faac80cb870 0x7faac815c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.686+0000 7faac6ffd700 1 -- 172.21.15.189:0/3580180528 >> v1:172.21.15.189:6789/0 conn(0x7faac80cfa00 legacy=0x7faac80cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:43.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.686+0000 7faac6ffd700 1 -- 172.21.15.189:0/3580180528 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac80cbd80 con 0x7faac80d11b0 2024-04-06T12:24:43.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.686+0000 7faac67fc700 1 --2- 172.21.15.189:0/3580180528 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faac80cb870 0x7faac815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:43.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.687+0000 7faac6ffd700 1 --2- 172.21.15.189:0/3580180528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faac80d11b0 0x7faac80d1590 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7faab8009840 tx=0x7faab8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=deea618bdfae4b9d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:43.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.687+0000 7faac5ffb700 1 -- 172.21.15.189:0/3580180528 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab8011830 con 0x7faac80d11b0 2024-04-06T12:24:43.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.687+0000 7faac5ffb700 1 -- 172.21.15.189:0/3580180528 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faab8011990 con 0x7faac80d11b0 2024-04-06T12:24:43.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.687+0000 7faac5ffb700 1 -- 172.21.15.189:0/3580180528 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab8011d20 con 0x7faac80d11b0 2024-04-06T12:24:43.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.687+0000 7faacda2d700 1 -- 172.21.15.189:0/3580180528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faac80d11b0 msgr2=0x7faac80d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:43.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.687+0000 7faacda2d700 1 --2- 172.21.15.189:0/3580180528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faac80d11b0 0x7faac80d1590 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7faab8009840 tx=0x7faab8009b50 comp rx=0 tx=0).stop 2024-04-06T12:24:43.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.687+0000 7faacda2d700 1 -- 172.21.15.189:0/3580180528 shutdown_connections 2024-04-06T12:24:43.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.687+0000 7faacda2d700 1 --2- 172.21.15.189:0/3580180528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faac80d11b0 0x7faac80d1590 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.687+0000 7faacda2d700 1 --2- 172.21.15.189:0/3580180528 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faac80cb870 0x7faac815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.687+0000 7faacda2d700 1 -- 172.21.15.189:0/3580180528 >> 172.21.15.189:0/3580180528 conn(0x7faac80b2db0 msgr2=0x7faac80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:43.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.687+0000 7faacda2d700 1 -- 172.21.15.189:0/3580180528 shutdown_connections 2024-04-06T12:24:43.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.687+0000 7faacda2d700 1 -- 172.21.15.189:0/3580180528 wait complete. 2024-04-06T12:24:43.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.689+0000 7faacda2d700 1 Processor -- start 2024-04-06T12:24:43.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.689+0000 7faacda2d700 1 -- start start 2024-04-06T12:24:43.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faacda2d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faac80cb870 0x7faac80b6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:43.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faacda2d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faac80cfa00 0x7faac81557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:43.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faacda2d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac80d11b0 0x7faac81f4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:43.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faacda2d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac80b3ea0 con 0x7faac80d11b0 2024-04-06T12:24:43.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faacda2d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac8155d20 con 0x7faac80cb870 2024-04-06T12:24:43.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faacda2d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faac81f4750 con 0x7faac80cfa00 2024-04-06T12:24:43.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faac77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac80d11b0 0x7faac81f4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:43.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faac77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac80d11b0 0x7faac81f4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60482/0 (socket says 172.21.15.189:60482) 2024-04-06T12:24:43.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faac77fe700 1 -- 172.21.15.189:0/3183114541 learned_addr learned my addr 172.21.15.189:0/3183114541 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:43.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faac67fc700 1 --2- 172.21.15.189:0/3183114541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faac80cfa00 0x7faac81557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:43.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faac6ffd700 1 --2- 172.21.15.189:0/3183114541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faac80cb870 0x7faac80b6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:43.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faac77fe700 1 -- 172.21.15.189:0/3183114541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faac80cfa00 msgr2=0x7faac81557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:43.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faac77fe700 1 --2- 172.21.15.189:0/3183114541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faac80cfa00 0x7faac81557e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faac77fe700 1 -- 172.21.15.189:0/3183114541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faac80cb870 msgr2=0x7faac80b6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:43.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faac77fe700 1 --2- 172.21.15.189:0/3183114541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faac80cb870 0x7faac80b6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faac77fe700 1 -- 172.21.15.189:0/3183114541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faabc0094a0 con 0x7faac80d11b0 2024-04-06T12:24:43.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faac77fe700 1 --2- 172.21.15.189:0/3183114541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac80d11b0 0x7faac81f4210 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7faac000cb70 tx=0x7faac000ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:43.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faaaffff700 1 -- 172.21.15.189:0/3183114541 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac000aa60 con 0x7faac80d11b0 2024-04-06T12:24:43.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faaaffff700 1 -- 172.21.15.189:0/3183114541 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac000abc0 con 0x7faac80d11b0 2024-04-06T12:24:43.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faacda2d700 1 -- 172.21.15.189:0/3183114541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab80094a0 con 0x7faac80d11b0 2024-04-06T12:24:43.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faaaffff700 1 -- 172.21.15.189:0/3183114541 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac0012400 con 0x7faac80d11b0 2024-04-06T12:24:43.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.690+0000 7faacda2d700 1 -- 172.21.15.189:0/3183114541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faac815ddc0 con 0x7faac80d11b0 2024-04-06T12:24:43.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.691+0000 7faacda2d700 1 -- 172.21.15.189:0/3183114541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faac8074990 con 0x7faac80d11b0 2024-04-06T12:24:43.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.692+0000 7faaaffff700 1 -- 172.21.15.189:0/3183114541 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 15) v1 ==== 89316+0+0 (secure 0 0 0) 0x7faac001b3b0 con 0x7faac80d11b0 2024-04-06T12:24:43.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.692+0000 7faaaffff700 1 --2- 172.21.15.189:0/3183114541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faab406ae40 0x7faab406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:43.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.692+0000 7faac6ffd700 1 -- 172.21.15.189:0/3183114541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faab406ae40 msgr2=0x7faab406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:43.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.692+0000 7faaaffff700 1 -- 172.21.15.189:0/3183114541 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7028+0+0 (secure 0 0 0) 0x7faac00897e0 con 0x7faac80d11b0 2024-04-06T12:24:43.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.692+0000 7faac6ffd700 1 --2- 172.21.15.189:0/3183114541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faab406ae40 0x7faab406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:24:43.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.696+0000 7faaaffff700 1 -- 172.21.15.189:0/3183114541 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7faac0054200 con 0x7faac80d11b0 2024-04-06T12:24:43.727 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:43.729+0000 7f51465c0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:24:43.756 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:43.758+0000 7f44d481c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:24:43.811 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:43.812+0000 7f51465c0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:24:43.830 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:43.833+0000 7f44d481c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:24:43.888 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:43.890+0000 7f51465c0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:24:43.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.893+0000 7faac6ffd700 1 -- 172.21.15.189:0/3183114541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faab406ae40 msgr2=0x7faab406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:43.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.893+0000 7faac6ffd700 1 --2- 172.21.15.189:0/3183114541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faab406ae40 0x7faab406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:24:43.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.899+0000 7faacda2d700 1 -- 172.21.15.189:0/3183114541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faac80b0a00 con 0x7faac80d11b0 2024-04-06T12:24:43.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.899+0000 7faaaffff700 1 -- 172.21.15.189:0/3183114541 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v51) v1 ==== 104+0+11 (secure 0 0 0) 0x7faac0057820 con 0x7faac80d11b0 2024-04-06T12:24:43.900 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:24:43.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.901+0000 7faacda2d700 1 -- 172.21.15.189:0/3183114541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faab406ae40 msgr2=0x7faab406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:24:43.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.902+0000 7faacda2d700 1 --2- 172.21.15.189:0/3183114541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faab406ae40 0x7faab406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.902+0000 7faacda2d700 1 -- 172.21.15.189:0/3183114541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac80d11b0 msgr2=0x7faac81f4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:43.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.902+0000 7faacda2d700 1 --2- 172.21.15.189:0/3183114541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac80d11b0 0x7faac81f4210 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7faac000cb70 tx=0x7faac000ce80 comp rx=0 tx=0).stop 2024-04-06T12:24:43.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.902+0000 7faacda2d700 1 -- 172.21.15.189:0/3183114541 shutdown_connections 2024-04-06T12:24:43.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.902+0000 7faacda2d700 1 --2- 172.21.15.189:0/3183114541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac80d11b0 0x7faac81f4210 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.902+0000 7faacda2d700 1 --2- 172.21.15.189:0/3183114541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faab406ae40 0x7faab406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.902+0000 7faacda2d700 1 --2- 172.21.15.189:0/3183114541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faac80cb870 0x7faac80b6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.902+0000 7faacda2d700 1 --2- 172.21.15.189:0/3183114541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faac80cfa00 0x7faac81557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:43.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.902+0000 7faacda2d700 1 -- 172.21.15.189:0/3183114541 >> 172.21.15.189:0/3183114541 conn(0x7faac80b2db0 msgr2=0x7faac806d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:43.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.902+0000 7faacda2d700 1 -- 172.21.15.189:0/3183114541 shutdown_connections 2024-04-06T12:24:43.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:43.902+0000 7faacda2d700 1 -- 172.21.15.189:0/3183114541 wait complete. 2024-04-06T12:24:43.913 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:24:44.032 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:44.034+0000 7f44d481c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:24:44.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.084+0000 7fb2fde79700 1 Processor -- start 2024-04-06T12:24:44.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.084+0000 7fb2fde79700 1 -- start start 2024-04-06T12:24:44.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.086+0000 7fb2fde79700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb2f8161ac0 0x7fb2f8161ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:44.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.086+0000 7fb2fde79700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2f80b2ac0 0x7fb2f806da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:44.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.086+0000 7fb2fde79700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f806e910 con 0x7fb2f80b2f20 2024-04-06T12:24:44.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.086+0000 7fb2fde79700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2f806ef40 con 0x7fb2f8161ac0 2024-04-06T12:24:44.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.086+0000 7fb2fde79700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f80b1fb0 con 0x7fb2f80b2ac0 2024-04-06T12:24:44.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.086+0000 7fb2f77fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb2f8161ac0 0x7fb2f8161ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:44.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.086+0000 7fb2f77fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb2f8161ac0 0x7fb2f8161ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60496/0 (socket says 172.21.15.189:60496) 2024-04-06T12:24:44.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.086+0000 7fb2f77fe700 1 -- 172.21.15.189:0/282485573 learned_addr learned my addr 172.21.15.189:0/282485573 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:44.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.086+0000 7fb2f6ffd700 1 --2- 172.21.15.189:0/282485573 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2f80b2ac0 0x7fb2f806da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:44.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.086+0000 7fb2f77fe700 1 -- 172.21.15.189:0/282485573 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2f80b2ac0 msgr2=0x7fb2f806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:44.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.086+0000 7fb2f77fe700 1 --2- 172.21.15.189:0/282485573 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2f80b2ac0 0x7fb2f806da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.086+0000 7fb2f77fe700 1 -- 172.21.15.189:0/282485573 >> v1:172.21.15.192:6789/0 conn(0x7fb2f80b2f20 legacy=0x7fb2f80711e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-06T12:24:44.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.086+0000 7fb2f77fe700 1 -- 172.21.15.189:0/282485573 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f80b2270 con 0x7fb2f8161ac0 2024-04-06T12:24:44.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.086+0000 7fb2f77fe700 1 --2- 172.21.15.189:0/282485573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2f8161ac0 0x7fb2f8161ea0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0009e50 tx=0x7fb2f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e484b3b9f678f30 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:44.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.087+0000 7fb2f67fc700 1 -- 172.21.15.189:0/282485573 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f000e070 con 0x7fb2f8161ac0 2024-04-06T12:24:44.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.087+0000 7fb2f67fc700 1 -- 172.21.15.189:0/282485573 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f0003a20 con 0x7fb2f8161ac0 2024-04-06T12:24:44.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.087+0000 7fb2f67fc700 1 -- 172.21.15.189:0/282485573 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0003dd0 con 0x7fb2f8161ac0 2024-04-06T12:24:44.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.088+0000 7fb2fde79700 1 -- 172.21.15.189:0/282485573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2f8161ac0 msgr2=0x7fb2f8161ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:44.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.088+0000 7fb2fde79700 1 --2- 172.21.15.189:0/282485573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2f8161ac0 0x7fb2f8161ea0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0009e50 tx=0x7fb2f000b330 comp rx=0 tx=0).stop 2024-04-06T12:24:44.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.088+0000 7fb2fde79700 1 -- 172.21.15.189:0/282485573 shutdown_connections 2024-04-06T12:24:44.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.088+0000 7fb2fde79700 1 --2- 172.21.15.189:0/282485573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2f8161ac0 0x7fb2f8161ea0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.088+0000 7fb2fde79700 1 --2- 172.21.15.189:0/282485573 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2f80b2ac0 0x7fb2f806da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.088+0000 7fb2fde79700 1 -- 172.21.15.189:0/282485573 >> 172.21.15.189:0/282485573 conn(0x7fb2f80cefa0 msgr2=0x7fb2f80cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:44.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.088+0000 7fb2fde79700 1 -- 172.21.15.189:0/282485573 shutdown_connections 2024-04-06T12:24:44.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.088+0000 7fb2fde79700 1 -- 172.21.15.189:0/282485573 wait complete. 2024-04-06T12:24:44.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.090+0000 7fb2fde79700 1 Processor -- start 2024-04-06T12:24:44.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.090+0000 7fb2fde79700 1 -- start start 2024-04-06T12:24:44.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.091+0000 7fb2fde79700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2f80b2ac0 0x7fb2f80adb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:44.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.091+0000 7fb2fde79700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2f80b2f20 0x7fb2f8144c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:44.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.091+0000 7fb2fde79700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2f8161ac0 0x7fb2f8151ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:44.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.091+0000 7fb2fde79700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f8152540 con 0x7fb2f80b2ac0 2024-04-06T12:24:44.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.091+0000 7fb2fde79700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f814a970 con 0x7fb2f80b2f20 2024-04-06T12:24:44.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.091+0000 7fb2fde79700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f814ac50 con 0x7fb2f8161ac0 2024-04-06T12:24:44.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.092+0000 7fb2f6ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2f80b2f20 0x7fb2f8144c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:44.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.092+0000 7fb2f6ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2f80b2f20 0x7fb2f8144c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:57660/0 (socket says 172.21.15.189:57660) 2024-04-06T12:24:44.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.092+0000 7fb2f6ffd700 1 -- 172.21.15.189:0/1243917866 learned_addr learned my addr 172.21.15.189:0/1243917866 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:44.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.092+0000 7fb2f6ffd700 1 -- 172.21.15.189:0/1243917866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2f8161ac0 msgr2=0x7fb2f8151ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:24:44.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.092+0000 7fb2f6ffd700 1 --2- 172.21.15.189:0/1243917866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2f8161ac0 0x7fb2f8151ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.092+0000 7fb2f6ffd700 1 -- 172.21.15.189:0/1243917866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2f80b2ac0 msgr2=0x7fb2f80adb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:24:44.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.092+0000 7fb2f6ffd700 1 --2- 172.21.15.189:0/1243917866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2f80b2ac0 0x7fb2f80adb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.092+0000 7fb2f6ffd700 1 -- 172.21.15.189:0/1243917866 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f0009b30 con 0x7fb2f80b2f20 2024-04-06T12:24:44.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.092+0000 7fb2f6ffd700 1 --2- 172.21.15.189:0/1243917866 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2f80b2f20 0x7fb2f8144c40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb2e000e9b0 tx=0x7fb2e000ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:44.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.092+0000 7fb2effff700 1 -- 172.21.15.189:0/1243917866 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0009710 con 0x7fb2f80b2f20 2024-04-06T12:24:44.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.092+0000 7fb2effff700 1 -- 172.21.15.189:0/1243917866 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e000c850 con 0x7fb2f80b2f20 2024-04-06T12:24:44.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.092+0000 7fb2effff700 1 -- 172.21.15.189:0/1243917866 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e000cbc0 con 0x7fb2f80b2f20 2024-04-06T12:24:44.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.093+0000 7fb2fde79700 1 -- 172.21.15.189:0/1243917866 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f814af30 con 0x7fb2f80b2f20 2024-04-06T12:24:44.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.093+0000 7fb2fde79700 1 -- 172.21.15.189:0/1243917866 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f80ae0c0 con 0x7fb2f80b2f20 2024-04-06T12:24:44.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.094+0000 7fb2fde79700 1 -- 172.21.15.189:0/1243917866 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2f8074990 con 0x7fb2f80b2f20 2024-04-06T12:24:44.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.094+0000 7fb2effff700 1 -- 172.21.15.189:0/1243917866 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 15) v1 ==== 89316+0+0 (secure 0 0 0) 0x7fb2e000cd20 con 0x7fb2f80b2f20 2024-04-06T12:24:44.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.095+0000 7fb2effff700 1 --2- 172.21.15.189:0/1243917866 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2d806ae40 0x7fb2d806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:44.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.095+0000 7fb2effff700 1 -- 172.21.15.189:0/1243917866 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7028+0+0 (secure 0 0 0) 0x7fb2e0089ca0 con 0x7fb2f80b2f20 2024-04-06T12:24:44.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.095+0000 7fb2f77fe700 1 -- 172.21.15.189:0/1243917866 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2d806ae40 msgr2=0x7fb2d806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:44.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.095+0000 7fb2f77fe700 1 --2- 172.21.15.189:0/1243917866 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2d806ae40 0x7fb2d806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:24:44.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.099+0000 7fb2effff700 1 -- 172.21.15.189:0/1243917866 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb2e0054760 con 0x7fb2f80b2f20 2024-04-06T12:24:44.106 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:44.109+0000 7f44d481c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:24:44.130 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:44.131+0000 7f51465c0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:24:44.205 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:44.207+0000 7f51465c0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:24:44.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.296+0000 7fb2f77fe700 1 -- 172.21.15.189:0/1243917866 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2d806ae40 msgr2=0x7fb2d806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:44.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.296+0000 7fb2f77fe700 1 --2- 172.21.15.189:0/1243917866 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2d806ae40 0x7fb2d806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:24:44.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.324+0000 7fb2fde79700 1 -- 172.21.15.189:0/1243917866 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb2f806e330 con 0x7fb2f80b2f20 2024-04-06T12:24:44.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.325+0000 7fb2effff700 1 -- 172.21.15.189:0/1243917866 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v51) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb2e005e070 con 0x7fb2f80b2f20 2024-04-06T12:24:44.323 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:24:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.327+0000 7fb2fde79700 1 -- 172.21.15.189:0/1243917866 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2d806ae40 msgr2=0x7fb2d806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:24:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.327+0000 7fb2fde79700 1 --2- 172.21.15.189:0/1243917866 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2d806ae40 0x7fb2d806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.327+0000 7fb2fde79700 1 -- 172.21.15.189:0/1243917866 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2f80b2f20 msgr2=0x7fb2f8144c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.327+0000 7fb2fde79700 1 --2- 172.21.15.189:0/1243917866 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2f80b2f20 0x7fb2f8144c40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb2e000e9b0 tx=0x7fb2e000ed70 comp rx=0 tx=0).stop 2024-04-06T12:24:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.327+0000 7fb2fde79700 1 -- 172.21.15.189:0/1243917866 shutdown_connections 2024-04-06T12:24:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.327+0000 7fb2fde79700 1 --2- 172.21.15.189:0/1243917866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2f8161ac0 0x7fb2f8151ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.327+0000 7fb2fde79700 1 --2- 172.21.15.189:0/1243917866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2f80b2ac0 0x7fb2f80adb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.327+0000 7fb2fde79700 1 --2- 172.21.15.189:0/1243917866 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2d806ae40 0x7fb2d806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.327+0000 7fb2fde79700 1 --2- 172.21.15.189:0/1243917866 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2f80b2f20 0x7fb2f8144c40 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.327+0000 7fb2fde79700 1 -- 172.21.15.189:0/1243917866 >> 172.21.15.189:0/1243917866 conn(0x7fb2f80cefa0 msgr2=0x7fb2f80cce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.327+0000 7fb2fde79700 1 -- 172.21.15.189:0/1243917866 shutdown_connections 2024-04-06T12:24:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.327+0000 7fb2fde79700 1 -- 172.21.15.189:0/1243917866 wait complete. 2024-04-06T12:24:44.339 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T12:24:44.422 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:44.424+0000 7f44d481c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:24:44.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.469+0000 7f328bdf9700 1 Processor -- start 2024-04-06T12:24:44.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.469+0000 7f328bdf9700 1 -- start start 2024-04-06T12:24:44.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f328bdf9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328415d710 0x7f32840cec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:44.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f328bdf9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f32840cf980 0x7f32840d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:44.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f328bdf9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f32840d1e90 0x7f32841612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:44.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f328bdf9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32840d0b00 con 0x7f32840d1e90 2024-04-06T12:24:44.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f328bdf9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32840d2300 con 0x7f32840cf980 2024-04-06T12:24:44.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f328bdf9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32840cfe80 con 0x7f328415d710 2024-04-06T12:24:44.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f328a396700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f32840d1e90 0x7f32841612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:44.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f328a396700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f32840d1e90 0x7f32841612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60508/0 (socket says 172.21.15.189:60508) 2024-04-06T12:24:44.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f328a396700 1 -- 172.21.15.189:0/3911239118 learned_addr learned my addr 172.21.15.189:0/3911239118 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:44.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f3289394700 1 --2- 172.21.15.189:0/3911239118 >> v2:172.21.15.192:3300/0 conn(0x7f32840cf980 0x7f32840d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:44.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f3289b95700 1 --2- 172.21.15.189:0/3911239118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328415d710 0x7f32840cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:44.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f328a396700 1 -- 172.21.15.189:0/3911239118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328415d710 msgr2=0x7f32840cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:44.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f328a396700 1 --2- 172.21.15.189:0/3911239118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328415d710 0x7f32840cec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f328a396700 1 -- 172.21.15.189:0/3911239118 >> v2:172.21.15.192:3300/0 conn(0x7f32840cf980 msgr2=0x7f32840d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:44.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f328a396700 1 --2- 172.21.15.189:0/3911239118 >> v2:172.21.15.192:3300/0 conn(0x7f32840cf980 0x7f32840d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f328a396700 1 -- 172.21.15.189:0/3911239118 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32840d0160 con 0x7f32840d1e90 2024-04-06T12:24:44.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f3289394700 1 --2- 172.21.15.189:0/3911239118 >> v2:172.21.15.192:3300/0 conn(0x7f32840cf980 0x7f32840d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:44.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f328a396700 1 --2- 172.21.15.189:0/3911239118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32840d1e90 0x7f32841612c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f328408b660 tx=0x7f327c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d75ca01ca969dc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:44.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f3288b93700 1 -- 172.21.15.189:0/3911239118 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c00e070 con 0x7f32840d1e90 2024-04-06T12:24:44.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.470+0000 7f3288b93700 1 -- 172.21.15.189:0/3911239118 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327c00b8d0 con 0x7f32840d1e90 2024-04-06T12:24:44.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.471+0000 7f3288b93700 1 -- 172.21.15.189:0/3911239118 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c00bc80 con 0x7f32840d1e90 2024-04-06T12:24:44.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.471+0000 7f328bdf9700 1 -- 172.21.15.189:0/3911239118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32840d1e90 msgr2=0x7f32841612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:44.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.471+0000 7f328bdf9700 1 --2- 172.21.15.189:0/3911239118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32840d1e90 0x7f32841612c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f328408b660 tx=0x7f327c00b330 comp rx=0 tx=0).stop 2024-04-06T12:24:44.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.471+0000 7f328bdf9700 1 -- 172.21.15.189:0/3911239118 shutdown_connections 2024-04-06T12:24:44.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.471+0000 7f328bdf9700 1 --2- 172.21.15.189:0/3911239118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32840d1e90 0x7f32841612c0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.471+0000 7f328bdf9700 1 --2- 172.21.15.189:0/3911239118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328415d710 0x7f32840cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.471+0000 7f328bdf9700 1 --2- 172.21.15.189:0/3911239118 >> v2:172.21.15.192:3300/0 conn(0x7f32840cf980 0x7f32840d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.471+0000 7f328bdf9700 1 -- 172.21.15.189:0/3911239118 >> 172.21.15.189:0/3911239118 conn(0x7f328409c6b0 msgr2=0x7f32840d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:44.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.471+0000 7f328bdf9700 1 -- 172.21.15.189:0/3911239118 shutdown_connections 2024-04-06T12:24:44.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.471+0000 7f328bdf9700 1 -- 172.21.15.189:0/3911239118 wait complete. 2024-04-06T12:24:44.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.472+0000 7f328bdf9700 1 Processor -- start 2024-04-06T12:24:44.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.472+0000 7f328bdf9700 1 -- start start 2024-04-06T12:24:44.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.473+0000 7f328bdf9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32840cf980 0x7f328415bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:44.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.473+0000 7f328bdf9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32840d1e90 0x7f32840b27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:44.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.473+0000 7f328bdf9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328415d710 0x7f32840c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:44.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.473+0000 7f328bdf9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328415c400 con 0x7f32840cf980 2024-04-06T12:24:44.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.473+0000 7f328bdf9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32840b3210 con 0x7f32840d1e90 2024-04-06T12:24:44.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.473+0000 7f328bdf9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32840b34f0 con 0x7f328415d710 2024-04-06T12:24:44.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.473+0000 7f3289b95700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32840cf980 0x7f328415bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:44.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.473+0000 7f3289b95700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32840cf980 0x7f328415bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60514/0 (socket says 172.21.15.189:60514) 2024-04-06T12:24:44.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.473+0000 7f3289b95700 1 -- 172.21.15.189:0/1297354475 learned_addr learned my addr 172.21.15.189:0/1297354475 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:44.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.473+0000 7f3289394700 1 --2- 172.21.15.189:0/1297354475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32840d1e90 0x7f32840b27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:44.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.473+0000 7f328a396700 1 --2- 172.21.15.189:0/1297354475 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328415d710 0x7f32840c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:44.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.474+0000 7f3289b95700 1 -- 172.21.15.189:0/1297354475 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328415d710 msgr2=0x7f32840c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:44.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.474+0000 7f3289b95700 1 --2- 172.21.15.189:0/1297354475 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328415d710 0x7f32840c85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.474+0000 7f3289b95700 1 -- 172.21.15.189:0/1297354475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32840d1e90 msgr2=0x7f32840b27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:44.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.474+0000 7f3289b95700 1 --2- 172.21.15.189:0/1297354475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32840d1e90 0x7f32840b27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.474+0000 7f3289b95700 1 -- 172.21.15.189:0/1297354475 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32740094a0 con 0x7f32840cf980 2024-04-06T12:24:44.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.474+0000 7f3289b95700 1 --2- 172.21.15.189:0/1297354475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32840cf980 0x7f328415bec0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f328000e870 tx=0x7f328000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:44.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.474+0000 7f327a7fc700 1 -- 172.21.15.189:0/1297354475 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3280009590 con 0x7f32840cf980 2024-04-06T12:24:44.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.474+0000 7f327a7fc700 1 -- 172.21.15.189:0/1297354475 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3280013680 con 0x7f32840cf980 2024-04-06T12:24:44.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.474+0000 7f328bdf9700 1 -- 172.21.15.189:0/1297354475 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f327c009b30 con 0x7f32840cf980 2024-04-06T12:24:44.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.474+0000 7f327a7fc700 1 -- 172.21.15.189:0/1297354475 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32800139f0 con 0x7f32840cf980 2024-04-06T12:24:44.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.474+0000 7f328bdf9700 1 -- 172.21.15.189:0/1297354475 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32840b3ab0 con 0x7f32840cf980 2024-04-06T12:24:44.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.475+0000 7f328bdf9700 1 -- 172.21.15.189:0/1297354475 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32840b4100 con 0x7f32840cf980 2024-04-06T12:24:44.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.476+0000 7f327a7fc700 1 -- 172.21.15.189:0/1297354475 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 15) v1 ==== 89316+0+0 (secure 0 0 0) 0x7f3280013b50 con 0x7f32840cf980 2024-04-06T12:24:44.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.476+0000 7f327a7fc700 1 --2- 172.21.15.189:0/1297354475 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f326806ae40 0x7f326806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:44.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.476+0000 7f3289394700 1 -- 172.21.15.189:0/1297354475 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f326806ae40 msgr2=0x7f326806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:44.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.476+0000 7f327a7fc700 1 -- 172.21.15.189:0/1297354475 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f3280089da0 con 0x7f32840cf980 2024-04-06T12:24:44.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.476+0000 7f3289394700 1 --2- 172.21.15.189:0/1297354475 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f326806ae40 0x7f326806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:24:44.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.480+0000 7f327a7fc700 1 -- 172.21.15.189:0/1297354475 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f32800547d0 con 0x7f32840cf980 2024-04-06T12:24:44.542 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:44.544+0000 7f51465c0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:24:44.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.677+0000 7f3289394700 1 -- 172.21.15.189:0/1297354475 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f326806ae40 msgr2=0x7f326806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:44.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.677+0000 7f3289394700 1 --2- 172.21.15.189:0/1297354475 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f326806ae40 0x7f326806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:24:44.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.700+0000 7f328bdf9700 1 -- 172.21.15.189:0/1297354475 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f32840748f0 con 0x7f32840cf980 2024-04-06T12:24:44.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.700+0000 7f327a7fc700 1 -- 172.21.15.189:0/1297354475 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v51) v1 ==== 80+0+349 (secure 0 0 0) 0x7f3280021170 con 0x7f32840cf980 2024-04-06T12:24:44.699 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/155214328 2024-04-07T12:13:34.836450+0000 2024-04-06T12:24:44.699 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/3092563682 2024-04-07T12:13:34.836450+0000 2024-04-06T12:24:44.699 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1075995033 2024-04-07T12:13:34.836450+0000 2024-04-06T12:24:44.699 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6833/47128 2024-04-07T12:13:34.836450+0000 2024-04-06T12:24:44.700 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/3015633422 2024-04-07T12:13:34.836450+0000 2024-04-06T12:24:44.700 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6832/47128 2024-04-07T12:13:34.836450+0000 2024-04-06T12:24:44.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.702+0000 7f328bdf9700 1 -- 172.21.15.189:0/1297354475 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f326806ae40 msgr2=0x7f326806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:24:44.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.702+0000 7f328bdf9700 1 --2- 172.21.15.189:0/1297354475 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f326806ae40 0x7f326806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.702+0000 7f328bdf9700 1 -- 172.21.15.189:0/1297354475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32840cf980 msgr2=0x7f328415bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:44.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.702+0000 7f328bdf9700 1 --2- 172.21.15.189:0/1297354475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32840cf980 0x7f328415bec0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f328000e870 tx=0x7f328000eb80 comp rx=0 tx=0).stop 2024-04-06T12:24:44.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.702+0000 7f328bdf9700 1 -- 172.21.15.189:0/1297354475 shutdown_connections 2024-04-06T12:24:44.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.702+0000 7f328bdf9700 1 --2- 172.21.15.189:0/1297354475 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328415d710 0x7f32840c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.702+0000 7f328bdf9700 1 --2- 172.21.15.189:0/1297354475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32840cf980 0x7f328415bec0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.702+0000 7f328bdf9700 1 --2- 172.21.15.189:0/1297354475 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f326806ae40 0x7f326806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.702+0000 7f328bdf9700 1 --2- 172.21.15.189:0/1297354475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32840d1e90 0x7f32840b27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:44.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.702+0000 7f328bdf9700 1 -- 172.21.15.189:0/1297354475 >> 172.21.15.189:0/1297354475 conn(0x7f328409c6b0 msgr2=0x7f32840ce100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:44.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.702+0000 7f328bdf9700 1 -- 172.21.15.189:0/1297354475 shutdown_connections 2024-04-06T12:24:44.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:44.702+0000 7f328bdf9700 1 -- 172.21.15.189:0/1297354475 wait complete. 2024-04-06T12:24:44.703 INFO:teuthology.orchestra.run.smithi189.stderr:listed 6 entries 2024-04-06T12:24:44.713 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-06T12:24:44.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:44.713 DEBUG:teuthology.orchestra.run.smithi189:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-06T12:24:44.742 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:44.745+0000 7f44d481c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:24:44.764 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T12:24:44.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:24:44.765 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-06T12:24:44.784 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:24:44.784 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:24:44.801 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:24:44.802 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:24:44.802 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:24:44.841 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:24:44.841 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:24:44.841 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:24:44.859 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:24:44.859 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:24:44.877 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:44.879+0000 7f51465c0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:24:44.905 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:44.907+0000 7f44d481c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:24:44.977 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:44.979+0000 7f44d481c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:24:45.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.005+0000 7f4ebc83a700 1 Processor -- start 2024-04-06T12:24:45.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.005+0000 7f4ebc83a700 1 -- start start 2024-04-06T12:24:45.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.006+0000 7f4ebc83a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4eb815d7b0 0x7f4eb80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:45.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.006+0000 7f4ebc83a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4eb806a730 0x7f4eb806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:45.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.006+0000 7f4ebc83a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb80b3990 con 0x7f4eb815ddf0 2024-04-06T12:24:45.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.006+0000 7f4ebc83a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eb806b0f0 con 0x7f4eb806a730 2024-04-06T12:24:45.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.006+0000 7f4ebc83a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb80b3060 con 0x7f4eb815d7b0 2024-04-06T12:24:45.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.006+0000 7f4eb5d9b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4eb806a730 0x7f4eb806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:45.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.006+0000 7f4eb659c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4eb815d7b0 0x7f4eb80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:45.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.006+0000 7f4eb5d9b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4eb806a730 0x7f4eb806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:57676/0 (socket says 172.21.15.189:57676) 2024-04-06T12:24:45.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.006+0000 7f4eb659c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4eb815d7b0 0x7f4eb80b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:55284/0 (socket says 172.21.15.189:55284) 2024-04-06T12:24:45.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.006+0000 7f4eb5d9b700 1 -- 172.21.15.189:0/1234727084 learned_addr learned my addr 172.21.15.189:0/1234727084 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:45.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.006+0000 7f4eb659c700 1 -- 172.21.15.189:0/1234727084 >> v2:172.21.15.192:3300/0 conn(0x7f4eb806a730 msgr2=0x7f4eb806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:45.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.006+0000 7f4eb659c700 1 --2- 172.21.15.189:0/1234727084 >> v2:172.21.15.192:3300/0 conn(0x7f4eb806a730 0x7f4eb806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.006+0000 7f4eb659c700 1 -- 172.21.15.189:0/1234727084 >> v1:172.21.15.192:6789/0 conn(0x7f4eb815ddf0 legacy=0x7f4eb815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:45.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.006+0000 7f4eb659c700 1 -- 172.21.15.189:0/1234727084 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb80b3340 con 0x7f4eb815d7b0 2024-04-06T12:24:45.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.006+0000 7f4eb5d9b700 1 --2- 172.21.15.189:0/1234727084 >> v2:172.21.15.192:3300/0 conn(0x7f4eb806a730 0x7f4eb806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:45.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.006+0000 7f4eb659c700 1 --2- 172.21.15.189:0/1234727084 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4eb815d7b0 0x7f4eb80b3e20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8009840 tx=0x7f4ea8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c57d79e24bace0b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:45.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.007+0000 7f4eb559a700 1 -- 172.21.15.189:0/1234727084 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea8011830 con 0x7f4eb815d7b0 2024-04-06T12:24:45.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.007+0000 7f4eb559a700 1 -- 172.21.15.189:0/1234727084 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea8011990 con 0x7f4eb815d7b0 2024-04-06T12:24:45.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.007+0000 7f4eb559a700 1 -- 172.21.15.189:0/1234727084 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea8011d20 con 0x7f4eb815d7b0 2024-04-06T12:24:45.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.007+0000 7f4ebc83a700 1 -- 172.21.15.189:0/1234727084 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4eb815d7b0 msgr2=0x7f4eb80b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:45.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.007+0000 7f4ebc83a700 1 --2- 172.21.15.189:0/1234727084 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4eb815d7b0 0x7f4eb80b3e20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8009840 tx=0x7f4ea8009b50 comp rx=0 tx=0).stop 2024-04-06T12:24:45.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.007+0000 7f4ebc83a700 1 -- 172.21.15.189:0/1234727084 shutdown_connections 2024-04-06T12:24:45.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.007+0000 7f4ebc83a700 1 --2- 172.21.15.189:0/1234727084 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4eb815d7b0 0x7f4eb80b3e20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.007+0000 7f4ebc83a700 1 --2- 172.21.15.189:0/1234727084 >> v2:172.21.15.192:3300/0 conn(0x7f4eb806a730 0x7f4eb806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.007+0000 7f4ebc83a700 1 -- 172.21.15.189:0/1234727084 >> 172.21.15.189:0/1234727084 conn(0x7f4eb80d00b0 msgr2=0x7f4eb80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:45.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.007+0000 7f4ebc83a700 1 -- 172.21.15.189:0/1234727084 shutdown_connections 2024-04-06T12:24:45.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.007+0000 7f4ebc83a700 1 -- 172.21.15.189:0/1234727084 wait complete. 2024-04-06T12:24:45.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.009+0000 7f4ebc83a700 1 Processor -- start 2024-04-06T12:24:45.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.009+0000 7f4ebc83a700 1 -- start start 2024-04-06T12:24:45.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.009+0000 7f4ebc83a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4eb806a730 0x7f4eb814fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:45.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.009+0000 7f4ebc83a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4eb815d7b0 0x7f4eb80ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:45.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.009+0000 7f4ebc83a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4eb815ddf0 0x7f4eb80cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:45.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.009+0000 7f4ebc83a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb8158370 con 0x7f4eb815ddf0 2024-04-06T12:24:45.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.009+0000 7f4ebc83a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb8159600 con 0x7f4eb815d7b0 2024-04-06T12:24:45.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.009+0000 7f4ebc83a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb8158560 con 0x7f4eb806a730 2024-04-06T12:24:45.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.010+0000 7f4eb6d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4eb815ddf0 0x7f4eb80cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:45.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.010+0000 7f4eb5d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4eb815d7b0 0x7f4eb80ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:45.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.010+0000 7f4eb5d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4eb815d7b0 0x7f4eb80ccec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:57680/0 (socket says 172.21.15.189:57680) 2024-04-06T12:24:45.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.010+0000 7f4eb659c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4eb806a730 0x7f4eb814fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:45.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.010+0000 7f4eb6d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4eb815ddf0 0x7f4eb80cd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60524/0 (socket says 172.21.15.189:60524) 2024-04-06T12:24:45.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.010+0000 7f4eb5d9b700 1 -- 172.21.15.189:0/2804365975 learned_addr learned my addr 172.21.15.189:0/2804365975 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:45.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.010+0000 7f4eb6d9d700 1 -- 172.21.15.189:0/2804365975 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4eb806a730 msgr2=0x7f4eb814fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:45.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.010+0000 7f4eb6d9d700 1 --2- 172.21.15.189:0/2804365975 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4eb806a730 0x7f4eb814fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.010+0000 7f4eb6d9d700 1 -- 172.21.15.189:0/2804365975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4eb815d7b0 msgr2=0x7f4eb80ccec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:45.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.010+0000 7f4eb6d9d700 1 --2- 172.21.15.189:0/2804365975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4eb815d7b0 0x7f4eb80ccec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.010+0000 7f4eb6d9d700 1 -- 172.21.15.189:0/2804365975 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac009550 con 0x7f4eb815ddf0 2024-04-06T12:24:45.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.010+0000 7f4eb5d9b700 1 --2- 172.21.15.189:0/2804365975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4eb815d7b0 0x7f4eb80ccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:45.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.010+0000 7f4eb6d9d700 1 --2- 172.21.15.189:0/2804365975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4eb815ddf0 0x7f4eb80cd400 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000cb10 tx=0x7f4eb000ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:45.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.010+0000 7f4ea6ffd700 1 -- 172.21.15.189:0/2804365975 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0003c00 con 0x7f4eb815ddf0 2024-04-06T12:24:45.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.010+0000 7f4ea6ffd700 1 -- 172.21.15.189:0/2804365975 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb0003d60 con 0x7f4eb815ddf0 2024-04-06T12:24:45.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.010+0000 7f4ebc83a700 1 -- 172.21.15.189:0/2804365975 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea80094a0 con 0x7f4eb815ddf0 2024-04-06T12:24:45.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.010+0000 7f4ea6ffd700 1 -- 172.21.15.189:0/2804365975 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb000a9e0 con 0x7f4eb815ddf0 2024-04-06T12:24:45.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.010+0000 7f4ebc83a700 1 -- 172.21.15.189:0/2804365975 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb8158b20 con 0x7f4eb815ddf0 2024-04-06T12:24:45.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.011+0000 7f4ebc83a700 1 -- 172.21.15.189:0/2804365975 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb8067bb0 con 0x7f4eb815ddf0 2024-04-06T12:24:45.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.012+0000 7f4ea6ffd700 1 -- 172.21.15.189:0/2804365975 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 15) v1 ==== 89316+0+0 (secure 0 0 0) 0x7f4eb000ab40 con 0x7f4eb815ddf0 2024-04-06T12:24:45.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.012+0000 7f4ea6ffd700 1 --2- 172.21.15.189:0/2804365975 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e9806ae40 0x7f4e9806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:45.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.012+0000 7f4eb659c700 1 -- 172.21.15.189:0/2804365975 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e9806ae40 msgr2=0x7f4e9806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:45.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.013+0000 7f4ea6ffd700 1 -- 172.21.15.189:0/2804365975 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f4eb00899f0 con 0x7f4eb815ddf0 2024-04-06T12:24:45.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.013+0000 7f4eb659c700 1 --2- 172.21.15.189:0/2804365975 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e9806ae40 0x7f4e9806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:24:45.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.017+0000 7f4ea6ffd700 1 -- 172.21.15.189:0/2804365975 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4eb0054420 con 0x7f4eb815ddf0 2024-04-06T12:24:45.056 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:45.058+0000 7f51465c0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:24:45.132 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:45.134+0000 7f51465c0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:24:45.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.214+0000 7f4eb659c700 1 -- 172.21.15.189:0/2804365975 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e9806ae40 msgr2=0x7f4e9806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:45.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.214+0000 7f4eb659c700 1 --2- 172.21.15.189:0/2804365975 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e9806ae40 0x7f4e9806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:24:45.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.279+0000 7f4ebc83a700 1 -- 172.21.15.189:0/2804365975 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4eb8074990 con 0x7f4eb815ddf0 2024-04-06T12:24:45.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.280+0000 7f4ea6ffd700 1 -- 172.21.15.189:0/2804365975 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f4eb0057a40 con 0x7f4eb815ddf0 2024-04-06T12:24:45.278 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:45.278 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4865,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/2279772906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2279772906},{"type":"v1","addr":"172.21.15.192:6834","nonce":2279772906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4868,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/454939556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":454939556},{"type":"v1","addr":"172.21.15.192:6835","nonce":454939556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5107,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/399541295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":399541295},{"type":"v1","addr":"172.21.15.189:6837","nonce":399541295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:13:21.287908+0000","modified":"2024-04-06T12:13:24.295625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5232":{"gid":5232,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/1024118918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1024118918},{"type":"v1","addr":"172.21.15.189:6836","nonce":1024118918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-04-06T12:24:45.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.282+0000 7f4ebc83a700 1 -- 172.21.15.189:0/2804365975 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e9806ae40 msgr2=0x7f4e9806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:24:45.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.282+0000 7f4ebc83a700 1 --2- 172.21.15.189:0/2804365975 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e9806ae40 0x7f4e9806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.282+0000 7f4ebc83a700 1 -- 172.21.15.189:0/2804365975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4eb815ddf0 msgr2=0x7f4eb80cd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:45.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.282+0000 7f4ebc83a700 1 --2- 172.21.15.189:0/2804365975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4eb815ddf0 0x7f4eb80cd400 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000cb10 tx=0x7f4eb000ce20 comp rx=0 tx=0).stop 2024-04-06T12:24:45.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.282+0000 7f4ebc83a700 1 -- 172.21.15.189:0/2804365975 shutdown_connections 2024-04-06T12:24:45.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.282+0000 7f4ebc83a700 1 --2- 172.21.15.189:0/2804365975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4eb815ddf0 0x7f4eb80cd400 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.282+0000 7f4ebc83a700 1 --2- 172.21.15.189:0/2804365975 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4eb806a730 0x7f4eb814fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.282+0000 7f4ebc83a700 1 --2- 172.21.15.189:0/2804365975 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e9806ae40 0x7f4e9806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.282+0000 7f4ebc83a700 1 --2- 172.21.15.189:0/2804365975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4eb815d7b0 0x7f4eb80ccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.282+0000 7f4ebc83a700 1 -- 172.21.15.189:0/2804365975 >> 172.21.15.189:0/2804365975 conn(0x7f4eb80d00b0 msgr2=0x7f4eb806fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:45.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.282+0000 7f4ebc83a700 1 -- 172.21.15.189:0/2804365975 shutdown_connections 2024-04-06T12:24:45.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.282+0000 7f4ebc83a700 1 -- 172.21.15.189:0/2804365975 wait complete. 2024-04-06T12:24:45.282 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 16 2024-04-06T12:24:45.293 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:24:45.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.424+0000 7f772565f700 1 Processor -- start 2024-04-06T12:24:45.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.424+0000 7f772565f700 1 -- start start 2024-04-06T12:24:45.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.424+0000 7f772565f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f772015d7e0 0x7f77200d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:45.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.424+0000 7f772565f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f77200d1cc0 0x7f77200b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:45.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.424+0000 7f772565f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77200cdf80 con 0x7f77200d0890 2024-04-06T12:24:45.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.424+0000 7f772565f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77200d0d10 con 0x7f77200d1cc0 2024-04-06T12:24:45.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.424+0000 7f772565f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77200d3230 con 0x7f772015d7e0 2024-04-06T12:24:45.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.425+0000 7f771e7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f77200d0890 0x7f77200d15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47110/0 (socket says 172.21.15.189:47110) 2024-04-06T12:24:45.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.425+0000 7f771e7fc700 1 -- 172.21.15.189:0/1685683552 learned_addr learned my addr 172.21.15.189:0/1685683552 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:45.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.425+0000 7f771f7fe700 1 --2- 172.21.15.189:0/1685683552 >> v2:172.21.15.192:3300/0 conn(0x7f77200d1cc0 0x7f77200b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:45.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.425+0000 7f771effd700 1 --2- 172.21.15.189:0/1685683552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f772015d7e0 0x7f77200d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:45.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.425+0000 7f771dffb700 1 -- 172.21.15.189:0/1685683552 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616065245 0 0) 0x7f77200cdf80 con 0x7f77200d0890 2024-04-06T12:24:45.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.425+0000 7f771dffb700 1 -- 172.21.15.189:0/1685683552 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77080035b0 con 0x7f77200d0890 2024-04-06T12:24:45.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.425+0000 7f771effd700 1 -- 172.21.15.189:0/1685683552 >> v2:172.21.15.192:3300/0 conn(0x7f77200d1cc0 msgr2=0x7f77200b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:45.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.425+0000 7f771effd700 1 --2- 172.21.15.189:0/1685683552 >> v2:172.21.15.192:3300/0 conn(0x7f77200d1cc0 0x7f77200b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.425+0000 7f771effd700 1 -- 172.21.15.189:0/1685683552 >> v1:172.21.15.189:6789/0 conn(0x7f77200d0890 legacy=0x7f77200d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:45.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.425+0000 7f771effd700 1 -- 172.21.15.189:0/1685683552 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77200d2da0 con 0x7f772015d7e0 2024-04-06T12:24:45.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.425+0000 7f771f7fe700 1 --2- 172.21.15.189:0/1685683552 >> v2:172.21.15.192:3300/0 conn(0x7f77200d1cc0 0x7f77200b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:45.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.425+0000 7f771effd700 1 --2- 172.21.15.189:0/1685683552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f772015d7e0 0x7f77200d0350 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7710009840 tx=0x7f7710009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f3dcde8483d390f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:45.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.425+0000 7f771dffb700 1 -- 172.21.15.189:0/1685683552 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710011830 con 0x7f772015d7e0 2024-04-06T12:24:45.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.425+0000 7f771dffb700 1 -- 172.21.15.189:0/1685683552 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7710011990 con 0x7f772015d7e0 2024-04-06T12:24:45.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.425+0000 7f771dffb700 1 -- 172.21.15.189:0/1685683552 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710011d20 con 0x7f772015d7e0 2024-04-06T12:24:45.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.426+0000 7f772565f700 1 -- 172.21.15.189:0/1685683552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f772015d7e0 msgr2=0x7f77200d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:45.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.426+0000 7f772565f700 1 --2- 172.21.15.189:0/1685683552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f772015d7e0 0x7f77200d0350 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7710009840 tx=0x7f7710009b50 comp rx=0 tx=0).stop 2024-04-06T12:24:45.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.426+0000 7f772565f700 1 -- 172.21.15.189:0/1685683552 shutdown_connections 2024-04-06T12:24:45.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.426+0000 7f772565f700 1 --2- 172.21.15.189:0/1685683552 >> v2:172.21.15.192:3300/0 conn(0x7f77200d1cc0 0x7f77200b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.426+0000 7f772565f700 1 --2- 172.21.15.189:0/1685683552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f772015d7e0 0x7f77200d0350 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.426+0000 7f772565f700 1 -- 172.21.15.189:0/1685683552 >> 172.21.15.189:0/1685683552 conn(0x7f7720072250 msgr2=0x7f7720072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:45.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.426+0000 7f772565f700 1 -- 172.21.15.189:0/1685683552 shutdown_connections 2024-04-06T12:24:45.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.426+0000 7f772565f700 1 -- 172.21.15.189:0/1685683552 wait complete. 2024-04-06T12:24:45.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.428+0000 7f772565f700 1 Processor -- start 2024-04-06T12:24:45.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.428+0000 7f772565f700 1 -- start start 2024-04-06T12:24:45.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.429+0000 7f772565f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77200d0890 0x7f7720149630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:45.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.429+0000 7f772565f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77200d1cc0 0x7f77200b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:45.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.429+0000 7f772565f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f772015d7e0 0x7f77200ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:45.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.429+0000 7f772565f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7720149b70 con 0x7f772015d7e0 2024-04-06T12:24:45.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.429+0000 7f772565f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77200c76a0 con 0x7f77200d0890 2024-04-06T12:24:45.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.429+0000 7f772565f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77200b7460 con 0x7f77200d1cc0 2024-04-06T12:24:45.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.429+0000 7f771f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f772015d7e0 0x7f77200ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:45.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.429+0000 7f771f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f772015d7e0 0x7f77200ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60526/0 (socket says 172.21.15.189:60526) 2024-04-06T12:24:45.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.429+0000 7f771e7fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77200d1cc0 0x7f77200b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:45.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.429+0000 7f771f7fe700 1 -- 172.21.15.189:0/1347288613 learned_addr learned my addr 172.21.15.189:0/1347288613 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:45.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.429+0000 7f771effd700 1 --2- 172.21.15.189:0/1347288613 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77200d0890 0x7f7720149630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:45.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.430+0000 7f771f7fe700 1 -- 172.21.15.189:0/1347288613 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77200d1cc0 msgr2=0x7f77200b78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.430+0000 7f771f7fe700 1 --2- 172.21.15.189:0/1347288613 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77200d1cc0 0x7f77200b78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.430+0000 7f771f7fe700 1 -- 172.21.15.189:0/1347288613 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77200d0890 msgr2=0x7f7720149630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.430+0000 7f771f7fe700 1 --2- 172.21.15.189:0/1347288613 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77200d0890 0x7f7720149630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.430+0000 7f771f7fe700 1 -- 172.21.15.189:0/1347288613 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7718009a70 con 0x7f772015d7e0 2024-04-06T12:24:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.430+0000 7f771e7fc700 1 --2- 172.21.15.189:0/1347288613 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77200d1cc0 0x7f77200b78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.430+0000 7f771effd700 1 --2- 172.21.15.189:0/1347288613 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77200d0890 0x7f7720149630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.430+0000 7f771f7fe700 1 --2- 172.21.15.189:0/1347288613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f772015d7e0 0x7f77200ccf60 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f7718009e60 tx=0x7f771800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.430+0000 7f77077fe700 1 -- 172.21.15.189:0/1347288613 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7718009cd0 con 0x7f772015d7e0 2024-04-06T12:24:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.430+0000 7f77077fe700 1 -- 172.21.15.189:0/1347288613 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771800f040 con 0x7f772015d7e0 2024-04-06T12:24:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.430+0000 7f772565f700 1 -- 172.21.15.189:0/1347288613 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77100094a0 con 0x7f772015d7e0 2024-04-06T12:24:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.430+0000 7f77077fe700 1 -- 172.21.15.189:0/1347288613 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771800ccb0 con 0x7f772015d7e0 2024-04-06T12:24:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.430+0000 7f772565f700 1 -- 172.21.15.189:0/1347288613 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77200b6b30 con 0x7f772015d7e0 2024-04-06T12:24:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.432+0000 7f772565f700 1 -- 172.21.15.189:0/1347288613 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77200b75d0 con 0x7f772015d7e0 2024-04-06T12:24:45.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.432+0000 7f77077fe700 1 -- 172.21.15.189:0/1347288613 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 15) v1 ==== 89316+0+0 (secure 0 0 0) 0x7f7718014480 con 0x7f772015d7e0 2024-04-06T12:24:45.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.432+0000 7f77077fe700 1 --2- 172.21.15.189:0/1347288613 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f770c06ae40 0x7f770c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:45.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.432+0000 7f771effd700 1 -- 172.21.15.189:0/1347288613 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f770c06ae40 msgr2=0x7f770c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:45.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.432+0000 7f77077fe700 1 -- 172.21.15.189:0/1347288613 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f771808a140 con 0x7f772015d7e0 2024-04-06T12:24:45.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.432+0000 7f771effd700 1 --2- 172.21.15.189:0/1347288613 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f770c06ae40 0x7f770c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:24:45.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.437+0000 7f77077fe700 1 -- 172.21.15.189:0/1347288613 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7718054af0 con 0x7f772015d7e0 2024-04-06T12:24:45.596 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:45.599+0000 7f44d481c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:24:45.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.633+0000 7f771effd700 1 -- 172.21.15.189:0/1347288613 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f770c06ae40 msgr2=0x7f770c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:45.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.633+0000 7f771effd700 1 --2- 172.21.15.189:0/1347288613 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f770c06ae40 0x7f770c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:24:45.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.642+0000 7f772565f700 1 -- 172.21.15.189:0/1347288613 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7720074990 con 0x7f772015d7e0 2024-04-06T12:24:45.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.644+0000 7f77077fe700 1 -- 172.21.15.189:0/1347288613 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+17818 (secure 0 0 0) 0x7f7718058110 con 0x7f772015d7e0 2024-04-06T12:24:45.642 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:45.643 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":51,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:24:42.483517+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:13:19.268280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-06T12:13:20.243771+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":220,"snap_epoch":51,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.35","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/155214328":"2024-04-07T12:13:34.836450+0000","172.21.15.189:0/3092563682":"2024-04-07T12:13:34.836450+0000","172.21.15.189:0/1075995033":"2024-04-07T12:13:34.836450+0000","172.21.15.189:6833/47128":"2024-04-07T12:13:34.836450+0000","172.21.15.189:0/3015633422":"2024-04-07T12:13:34.836450+0000","172.21.15.189:6832/47128":"2024-04-07T12:13:34.836450+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":2,"length":1},{"begin":5,"length":2},{"begin":9,"length":1},{"begin":11,"length":1},{"begin":13,"length":2},{"begin":17,"length":2},{"begin":21,"length":2},{"begin":24,"length":2},{"begin":28,"length":2},{"begin":33,"length":1},{"begin":35,"length":2},{"begin":39,"length":2},{"begin":43,"length":1},{"begin":45,"length":2},{"begin":49,"length":3},{"begin":53,"length":1},{"begin":55,"length":2},{"begin":59,"length":4},{"begin":64,"length":1},{"begin":66,"length":2},{"begin":70,"length":3},{"begin":74,"length":1},{"begin":76,"length":2},{"begin":80,"length":5},{"begin":86,"length":1},{"begin":88,"length":2},{"begin":93,"length":3},{"begin":97,"length":1},{"begin":99,"length":2},{"begin":103,"length":3},{"begin":109,"length":2},{"begin":160,"length":61}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":9,"length":1},{"begin":21,"length":2},{"begin":33,"length":1},{"begin":43,"length":1},{"begin":53,"length":1},{"begin":64,"length":1},{"begin":74,"length":1},{"begin":86,"length":1},{"begin":97,"length":1},{"begin":103,"length":1},{"begin":210,"length":11}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:24:45.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.646+0000 7f772565f700 1 -- 172.21.15.189:0/1347288613 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f770c06ae40 msgr2=0x7f770c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:24:45.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.646+0000 7f772565f700 1 --2- 172.21.15.189:0/1347288613 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f770c06ae40 0x7f770c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.646+0000 7f772565f700 1 -- 172.21.15.189:0/1347288613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f772015d7e0 msgr2=0x7f77200ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:45.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.646+0000 7f772565f700 1 --2- 172.21.15.189:0/1347288613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f772015d7e0 0x7f77200ccf60 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f7718009e60 tx=0x7f771800c3a0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.646+0000 7f772565f700 1 -- 172.21.15.189:0/1347288613 shutdown_connections 2024-04-06T12:24:45.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.646+0000 7f772565f700 1 --2- 172.21.15.189:0/1347288613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f772015d7e0 0x7f77200ccf60 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.646+0000 7f772565f700 1 --2- 172.21.15.189:0/1347288613 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f770c06ae40 0x7f770c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.646+0000 7f772565f700 1 --2- 172.21.15.189:0/1347288613 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77200d0890 0x7f7720149630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.646+0000 7f772565f700 1 --2- 172.21.15.189:0/1347288613 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77200d1cc0 0x7f77200b78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.646+0000 7f772565f700 1 -- 172.21.15.189:0/1347288613 >> 172.21.15.189:0/1347288613 conn(0x7f7720072250 msgr2=0x7f772015bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:45.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.646+0000 7f772565f700 1 -- 172.21.15.189:0/1347288613 shutdown_connections 2024-04-06T12:24:45.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.646+0000 7f772565f700 1 -- 172.21.15.189:0/1347288613 wait complete. 2024-04-06T12:24:45.661 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:13:19.268280+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:13:20.243771+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 220, 'snap_epoch': 51, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:24:45.662 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:24:45.675 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:45.678+0000 7f44d481c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:24:45.783 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:45.785+0000 7f51465c0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:24:45.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.791+0000 7ff37561b700 1 Processor -- start 2024-04-06T12:24:45.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.791+0000 7ff37561b700 1 -- start start 2024-04-06T12:24:45.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.792+0000 7ff37561b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3700d11b0 0x7ff3700d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:45.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.792+0000 7ff37561b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff3700cb870 0x7ff37015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:45.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.792+0000 7ff37561b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3700d1ad0 con 0x7ff3700cfa00 2024-04-06T12:24:45.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.792+0000 7ff37561b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3700ccdc0 con 0x7ff3700cb870 2024-04-06T12:24:45.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.792+0000 7ff37561b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3700d0b50 con 0x7ff3700d11b0 2024-04-06T12:24:45.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.792+0000 7ff36e7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff3700cb870 0x7ff37015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:45.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.792+0000 7ff36e7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff3700cb870 0x7ff37015c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60532/0 (socket says 172.21.15.189:60532) 2024-04-06T12:24:45.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.792+0000 7ff36e7fc700 1 -- 172.21.15.189:0/2264194612 learned_addr learned my addr 172.21.15.189:0/2264194612 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:45.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.792+0000 7ff36effd700 1 --2- 172.21.15.189:0/2264194612 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3700d11b0 0x7ff3700d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:45.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.793+0000 7ff36e7fc700 1 -- 172.21.15.189:0/2264194612 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3700d11b0 msgr2=0x7ff3700d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:45.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.793+0000 7ff36e7fc700 1 --2- 172.21.15.189:0/2264194612 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3700d11b0 0x7ff3700d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.793+0000 7ff36e7fc700 1 -- 172.21.15.189:0/2264194612 >> v1:172.21.15.192:6789/0 conn(0x7ff3700cfa00 legacy=0x7ff3700cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:45.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.793+0000 7ff36e7fc700 1 -- 172.21.15.189:0/2264194612 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3700cbd80 con 0x7ff3700cb870 2024-04-06T12:24:45.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.793+0000 7ff36e7fc700 1 --2- 172.21.15.189:0/2264194612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3700cb870 0x7ff37015c280 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ff3640098f0 tx=0x7ff364009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cff3050977ee0d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:45.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.793+0000 7ff36dffb700 1 -- 172.21.15.189:0/2264194612 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff364011830 con 0x7ff3700cb870 2024-04-06T12:24:45.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.793+0000 7ff36dffb700 1 -- 172.21.15.189:0/2264194612 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff364011990 con 0x7ff3700cb870 2024-04-06T12:24:45.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.793+0000 7ff36dffb700 1 -- 172.21.15.189:0/2264194612 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff364011d20 con 0x7ff3700cb870 2024-04-06T12:24:45.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.793+0000 7ff37561b700 1 -- 172.21.15.189:0/2264194612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3700cb870 msgr2=0x7ff37015c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:45.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.793+0000 7ff37561b700 1 --2- 172.21.15.189:0/2264194612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3700cb870 0x7ff37015c280 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ff3640098f0 tx=0x7ff364009c00 comp rx=0 tx=0).stop 2024-04-06T12:24:45.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.794+0000 7ff37561b700 1 -- 172.21.15.189:0/2264194612 shutdown_connections 2024-04-06T12:24:45.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.794+0000 7ff37561b700 1 --2- 172.21.15.189:0/2264194612 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3700d11b0 0x7ff3700d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.794+0000 7ff37561b700 1 --2- 172.21.15.189:0/2264194612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3700cb870 0x7ff37015c280 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.794+0000 7ff37561b700 1 -- 172.21.15.189:0/2264194612 >> 172.21.15.189:0/2264194612 conn(0x7ff3700b2db0 msgr2=0x7ff3700b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:45.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.794+0000 7ff37561b700 1 -- 172.21.15.189:0/2264194612 shutdown_connections 2024-04-06T12:24:45.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.794+0000 7ff37561b700 1 -- 172.21.15.189:0/2264194612 wait complete. 2024-04-06T12:24:45.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.796+0000 7ff37561b700 1 Processor -- start 2024-04-06T12:24:45.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.796+0000 7ff37561b700 1 -- start start 2024-04-06T12:24:45.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.797+0000 7ff37561b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3700cb870 0x7ff37014f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:45.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.797+0000 7ff37561b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3700cfa00 0x7ff37015c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:45.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.797+0000 7ff37561b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3700d11b0 0x7ff37015c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:45.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.797+0000 7ff37561b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37015da00 con 0x7ff3700cb870 2024-04-06T12:24:45.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.797+0000 7ff37561b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37015cdf0 con 0x7ff3700d11b0 2024-04-06T12:24:45.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.797+0000 7ff37561b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff37015d0d0 con 0x7ff3700cfa00 2024-04-06T12:24:45.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.797+0000 7ff36effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3700cb870 0x7ff37014f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:45.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.797+0000 7ff36effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3700cb870 0x7ff37014f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60542/0 (socket says 172.21.15.189:60542) 2024-04-06T12:24:45.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.797+0000 7ff36effd700 1 -- 172.21.15.189:0/1624496282 learned_addr learned my addr 172.21.15.189:0/1624496282 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:45.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.797+0000 7ff36e7fc700 1 --2- 172.21.15.189:0/1624496282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3700cfa00 0x7ff37015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:45.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.797+0000 7ff36f7fe700 1 --2- 172.21.15.189:0/1624496282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3700d11b0 0x7ff37015c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:45.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.797+0000 7ff36effd700 1 -- 172.21.15.189:0/1624496282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3700cfa00 msgr2=0x7ff37015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:45.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.797+0000 7ff36effd700 1 --2- 172.21.15.189:0/1624496282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3700cfa00 0x7ff37015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.797+0000 7ff36effd700 1 -- 172.21.15.189:0/1624496282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3700d11b0 msgr2=0x7ff37015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:45.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.797+0000 7ff36effd700 1 --2- 172.21.15.189:0/1624496282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3700d11b0 0x7ff37015c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:45.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.797+0000 7ff36effd700 1 -- 172.21.15.189:0/1624496282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff364009550 con 0x7ff3700cb870 2024-04-06T12:24:45.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.797+0000 7ff36effd700 1 --2- 172.21.15.189:0/1624496282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3700cb870 0x7ff37014f610 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ff36000e870 tx=0x7ff36000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:45.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.798+0000 7ff3577fe700 1 -- 172.21.15.189:0/1624496282 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff360009590 con 0x7ff3700cb870 2024-04-06T12:24:45.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.798+0000 7ff3577fe700 1 -- 172.21.15.189:0/1624496282 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff360013680 con 0x7ff3700cb870 2024-04-06T12:24:45.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.798+0000 7ff3577fe700 1 -- 172.21.15.189:0/1624496282 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3600139f0 con 0x7ff3700cb870 2024-04-06T12:24:45.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.798+0000 7ff37561b700 1 -- 172.21.15.189:0/1624496282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37015d3b0 con 0x7ff3700cb870 2024-04-06T12:24:45.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.798+0000 7ff37561b700 1 -- 172.21.15.189:0/1624496282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff370157650 con 0x7ff3700cb870 2024-04-06T12:24:45.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.799+0000 7ff37561b700 1 -- 172.21.15.189:0/1624496282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff370074990 con 0x7ff3700cb870 2024-04-06T12:24:45.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.799+0000 7ff3577fe700 1 -- 172.21.15.189:0/1624496282 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 15) v1 ==== 89316+0+0 (secure 0 0 0) 0x7ff360013b50 con 0x7ff3700cb870 2024-04-06T12:24:45.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.800+0000 7ff3577fe700 1 --2- 172.21.15.189:0/1624496282 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff35c06ae40 0x7ff35c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:45.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.800+0000 7ff36e7fc700 1 -- 172.21.15.189:0/1624496282 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff35c06ae40 msgr2=0x7ff35c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:45.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.800+0000 7ff3577fe700 1 -- 172.21.15.189:0/1624496282 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7028+0+0 (secure 0 0 0) 0x7ff3600899e0 con 0x7ff3700cb870 2024-04-06T12:24:45.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.800+0000 7ff36e7fc700 1 --2- 172.21.15.189:0/1624496282 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff35c06ae40 0x7ff35c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:24:45.799 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:45.802+0000 7f44d481c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:24:45.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:45.804+0000 7ff3577fe700 1 -- 172.21.15.189:0/1624496282 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff360054410 con 0x7ff3700cb870 2024-04-06T12:24:45.867 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:45.869+0000 7f51465c0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:24:45.996 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:45.998+0000 7f51465c0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:24:45.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.001+0000 7ff36e7fc700 1 -- 172.21.15.189:0/1624496282 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff35c06ae40 msgr2=0x7ff35c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:45.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.001+0000 7ff36e7fc700 1 --2- 172.21.15.189:0/1624496282 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff35c06ae40 0x7ff35c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:24:46.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.006+0000 7ff37561b700 1 -- 172.21.15.189:0/1624496282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff3700cb630 con 0x7ff3700cb870 2024-04-06T12:24:46.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.006+0000 7ff3577fe700 1 -- 172.21.15.189:0/1624496282 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff360057a30 con 0x7ff3700cb870 2024-04-06T12:24:46.004 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:24:46.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.008+0000 7ff37561b700 1 -- 172.21.15.189:0/1624496282 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff35c06ae40 msgr2=0x7ff35c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:24:46.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.008+0000 7ff37561b700 1 --2- 172.21.15.189:0/1624496282 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff35c06ae40 0x7ff35c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.008+0000 7ff37561b700 1 -- 172.21.15.189:0/1624496282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3700cb870 msgr2=0x7ff37014f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:46.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.008+0000 7ff37561b700 1 --2- 172.21.15.189:0/1624496282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3700cb870 0x7ff37014f610 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ff36000e870 tx=0x7ff36000eb80 comp rx=0 tx=0).stop 2024-04-06T12:24:46.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.008+0000 7ff37561b700 1 -- 172.21.15.189:0/1624496282 shutdown_connections 2024-04-06T12:24:46.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.008+0000 7ff37561b700 1 --2- 172.21.15.189:0/1624496282 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff35c06ae40 0x7ff35c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.008+0000 7ff37561b700 1 --2- 172.21.15.189:0/1624496282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3700d11b0 0x7ff37015c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.008+0000 7ff37561b700 1 --2- 172.21.15.189:0/1624496282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3700cb870 0x7ff37014f610 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.008+0000 7ff37561b700 1 --2- 172.21.15.189:0/1624496282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3700cfa00 0x7ff37015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.008+0000 7ff37561b700 1 -- 172.21.15.189:0/1624496282 >> 172.21.15.189:0/1624496282 conn(0x7ff3700b2db0 msgr2=0x7ff3700cd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:46.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.009+0000 7ff37561b700 1 -- 172.21.15.189:0/1624496282 shutdown_connections 2024-04-06T12:24:46.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.009+0000 7ff37561b700 1 -- 172.21.15.189:0/1624496282 wait complete. 2024-04-06T12:24:46.019 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:24:46.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.152+0000 7f2582478700 1 Processor -- start 2024-04-06T12:24:46.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.152+0000 7f2582478700 1 -- start start 2024-04-06T12:24:46.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.154+0000 7f2582478700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f257c1547c0 0x7f257c0b4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:46.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.154+0000 7f2582478700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f257c15b3e0 0x7f257c0b4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:46.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.154+0000 7f2582478700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257c154ba0 con 0x7f257c0b3130 2024-04-06T12:24:46.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.154+0000 7f2582478700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f257c15a710 con 0x7f257c15b3e0 2024-04-06T12:24:46.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.154+0000 7f2582478700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f257c0b3e60 con 0x7f257c1547c0 2024-04-06T12:24:46.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.154+0000 7f257b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f257c15b3e0 0x7f257c0b4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:46.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.154+0000 7f2580a15700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f257c0b3130 0x7f257c0b35a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47124/0 (socket says 172.21.15.189:47124) 2024-04-06T12:24:46.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.154+0000 7f257b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f257c15b3e0 0x7f257c0b4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60546/0 (socket says 172.21.15.189:60546) 2024-04-06T12:24:46.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.154+0000 7f2580a15700 1 -- 172.21.15.189:0/1939701157 learned_addr learned my addr 172.21.15.189:0/1939701157 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:46.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.154+0000 7f257bfff700 1 --2- 172.21.15.189:0/1939701157 >> v2:172.21.15.192:3300/0 conn(0x7f257c1547c0 0x7f257c0b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:46.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.154+0000 7f257b7fe700 1 -- 172.21.15.189:0/1939701157 >> v2:172.21.15.192:3300/0 conn(0x7f257c1547c0 msgr2=0x7f257c0b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:46.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.154+0000 7f257b7fe700 1 --2- 172.21.15.189:0/1939701157 >> v2:172.21.15.192:3300/0 conn(0x7f257c1547c0 0x7f257c0b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.154+0000 7f257b7fe700 1 -- 172.21.15.189:0/1939701157 >> v1:172.21.15.189:6789/0 conn(0x7f257c0b3130 legacy=0x7f257c0b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:46.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.154+0000 7f257affd700 1 -- 172.21.15.189:0/1939701157 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162710376 0 0) 0x7f257c154ba0 con 0x7f257c0b3130 2024-04-06T12:24:46.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.154+0000 7f257b7fe700 1 -- 172.21.15.189:0/1939701157 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c15a900 con 0x7f257c15b3e0 2024-04-06T12:24:46.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.154+0000 7f257b7fe700 1 --2- 172.21.15.189:0/1939701157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c15b3e0 0x7f257c0b4990 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f257c0d1130 tx=0x7f2570009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c6de485b3e721e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:46.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.154+0000 7f257affd700 1 -- 172.21.15.189:0/1939701157 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2570011830 con 0x7f257c15b3e0 2024-04-06T12:24:46.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.154+0000 7f257affd700 1 -- 172.21.15.189:0/1939701157 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2570011990 con 0x7f257c15b3e0 2024-04-06T12:24:46.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.155+0000 7f257affd700 1 -- 172.21.15.189:0/1939701157 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2570011d20 con 0x7f257c15b3e0 2024-04-06T12:24:46.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.155+0000 7f2582478700 1 -- 172.21.15.189:0/1939701157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c15b3e0 msgr2=0x7f257c0b4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:46.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.155+0000 7f2582478700 1 --2- 172.21.15.189:0/1939701157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c15b3e0 0x7f257c0b4990 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f257c0d1130 tx=0x7f2570009b20 comp rx=0 tx=0).stop 2024-04-06T12:24:46.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.155+0000 7f2582478700 1 -- 172.21.15.189:0/1939701157 shutdown_connections 2024-04-06T12:24:46.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.155+0000 7f2582478700 1 --2- 172.21.15.189:0/1939701157 >> v2:172.21.15.192:3300/0 conn(0x7f257c1547c0 0x7f257c0b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.155+0000 7f2582478700 1 --2- 172.21.15.189:0/1939701157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c15b3e0 0x7f257c0b4990 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.155+0000 7f2582478700 1 -- 172.21.15.189:0/1939701157 >> 172.21.15.189:0/1939701157 conn(0x7f257c0b1e30 msgr2=0x7f257c0b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:46.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.155+0000 7f2582478700 1 -- 172.21.15.189:0/1939701157 shutdown_connections 2024-04-06T12:24:46.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.155+0000 7f2582478700 1 -- 172.21.15.189:0/1939701157 wait complete. 2024-04-06T12:24:46.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.156+0000 7f2582478700 1 Processor -- start 2024-04-06T12:24:46.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.157+0000 7f2582478700 1 -- start start 2024-04-06T12:24:46.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.157+0000 7f2582478700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c0b3130 0x7f257c147f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:46.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.157+0000 7f2582478700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f257c1547c0 0x7f257c14bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:46.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.157+0000 7f2582478700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f257c15b3e0 0x7f257c149550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:46.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.157+0000 7f2582478700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c142c00 con 0x7f257c0b3130 2024-04-06T12:24:46.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.157+0000 7f2582478700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c157c80 con 0x7f257c15b3e0 2024-04-06T12:24:46.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.157+0000 7f2582478700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257c1490c0 con 0x7f257c1547c0 2024-04-06T12:24:46.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.158+0000 7f257bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c0b3130 0x7f257c147f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:46.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.158+0000 7f257bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c0b3130 0x7f257c147f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60552/0 (socket says 172.21.15.189:60552) 2024-04-06T12:24:46.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.158+0000 7f257bfff700 1 -- 172.21.15.189:0/399116468 learned_addr learned my addr 172.21.15.189:0/399116468 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:46.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.158+0000 7f2580a15700 1 --2- 172.21.15.189:0/399116468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f257c15b3e0 0x7f257c149550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:46.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.158+0000 7f257b7fe700 1 --2- 172.21.15.189:0/399116468 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f257c1547c0 0x7f257c14bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:46.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.158+0000 7f257bfff700 1 -- 172.21.15.189:0/399116468 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f257c1547c0 msgr2=0x7f257c14bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:46.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.158+0000 7f257bfff700 1 --2- 172.21.15.189:0/399116468 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f257c1547c0 0x7f257c14bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.158+0000 7f257bfff700 1 -- 172.21.15.189:0/399116468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f257c15b3e0 msgr2=0x7f257c149550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:46.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.158+0000 7f257bfff700 1 --2- 172.21.15.189:0/399116468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f257c15b3e0 0x7f257c149550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.158+0000 7f257bfff700 1 -- 172.21.15.189:0/399116468 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25700094a0 con 0x7f257c0b3130 2024-04-06T12:24:46.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.158+0000 7f257bfff700 1 --2- 172.21.15.189:0/399116468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c0b3130 0x7f257c147f60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f256c00e9e0 tx=0x7f256c00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:46.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.158+0000 7f2578ff9700 1 -- 172.21.15.189:0/399116468 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c009710 con 0x7f257c0b3130 2024-04-06T12:24:46.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.158+0000 7f2578ff9700 1 -- 172.21.15.189:0/399116468 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f256c00ee80 con 0x7f257c0b3130 2024-04-06T12:24:46.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.158+0000 7f2582478700 1 -- 172.21.15.189:0/399116468 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f257c154f30 con 0x7f257c0b3130 2024-04-06T12:24:46.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.158+0000 7f2578ff9700 1 -- 172.21.15.189:0/399116468 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c00ca50 con 0x7f257c0b3130 2024-04-06T12:24:46.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.158+0000 7f2582478700 1 -- 172.21.15.189:0/399116468 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f257c1554b0 con 0x7f257c0b3130 2024-04-06T12:24:46.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.159+0000 7f2582478700 1 -- 172.21.15.189:0/399116468 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f257c0748f0 con 0x7f257c0b3130 2024-04-06T12:24:46.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.160+0000 7f2578ff9700 1 -- 172.21.15.189:0/399116468 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 15) v1 ==== 89316+0+0 (secure 0 0 0) 0x7f256c00cbb0 con 0x7f257c0b3130 2024-04-06T12:24:46.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.160+0000 7f2578ff9700 1 --2- 172.21.15.189:0/399116468 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f256806ae40 0x7f256806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:46.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.160+0000 7f257b7fe700 1 -- 172.21.15.189:0/399116468 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f256806ae40 msgr2=0x7f256806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:46.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.160+0000 7f2578ff9700 1 -- 172.21.15.189:0/399116468 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f256c089ba0 con 0x7f257c0b3130 2024-04-06T12:24:46.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.160+0000 7f257b7fe700 1 --2- 172.21.15.189:0/399116468 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f256806ae40 0x7f256806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:24:46.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.165+0000 7f2578ff9700 1 -- 172.21.15.189:0/399116468 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f256c0545d0 con 0x7f257c0b3130 2024-04-06T12:24:46.313 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:46.316+0000 7f44d481c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:24:46.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.361+0000 7f257b7fe700 1 -- 172.21.15.189:0/399116468 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f256806ae40 msgr2=0x7f256806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:46.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.361+0000 7f257b7fe700 1 --2- 172.21.15.189:0/399116468 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f256806ae40 0x7f256806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:24:46.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.364+0000 7f2582478700 1 -- 172.21.15.189:0/399116468 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f257c0b4c30 con 0x7f257c0b3130 2024-04-06T12:24:46.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.365+0000 7f2578ff9700 1 -- 172.21.15.189:0/399116468 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v51) v1 ==== 104+0+11 (secure 0 0 0) 0x7f256c057bf0 con 0x7f257c0b3130 2024-04-06T12:24:46.363 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:24:46.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.367+0000 7f2582478700 1 -- 172.21.15.189:0/399116468 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f256806ae40 msgr2=0x7f256806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:24:46.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.367+0000 7f2582478700 1 --2- 172.21.15.189:0/399116468 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f256806ae40 0x7f256806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.367+0000 7f2582478700 1 -- 172.21.15.189:0/399116468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c0b3130 msgr2=0x7f257c147f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:46.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.367+0000 7f2582478700 1 --2- 172.21.15.189:0/399116468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c0b3130 0x7f257c147f60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f256c00e9e0 tx=0x7f256c00ecf0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.367+0000 7f2582478700 1 -- 172.21.15.189:0/399116468 shutdown_connections 2024-04-06T12:24:46.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.367+0000 7f2582478700 1 --2- 172.21.15.189:0/399116468 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f256806ae40 0x7f256806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.367+0000 7f2582478700 1 --2- 172.21.15.189:0/399116468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f257c15b3e0 0x7f257c149550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.367+0000 7f2582478700 1 --2- 172.21.15.189:0/399116468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c0b3130 0x7f257c147f60 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.367+0000 7f2582478700 1 --2- 172.21.15.189:0/399116468 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f257c1547c0 0x7f257c14bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.367+0000 7f2582478700 1 -- 172.21.15.189:0/399116468 >> 172.21.15.189:0/399116468 conn(0x7f257c0b1e30 msgr2=0x7f257c15c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:46.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.367+0000 7f2582478700 1 -- 172.21.15.189:0/399116468 shutdown_connections 2024-04-06T12:24:46.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.367+0000 7f2582478700 1 -- 172.21.15.189:0/399116468 wait complete. 2024-04-06T12:24:46.378 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:24:46.465 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:46.468+0000 7f44d481c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:24:46.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.511+0000 7f88cbe7f700 1 Processor -- start 2024-04-06T12:24:46.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.511+0000 7f88cbe7f700 1 -- start start 2024-04-06T12:24:46.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.512+0000 7f88cbe7f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88c40cd3e0 0x7f88c40cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:46.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.512+0000 7f88cbe7f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f88c406b8c0 0x7f88c406dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:46.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.512+0000 7f88cbe7f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c406ceb0 con 0x7f88c415d7c0 2024-04-06T12:24:46.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.512+0000 7f88cbe7f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c40cdfa0 con 0x7f88c406b8c0 2024-04-06T12:24:46.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.512+0000 7f88cbe7f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c406d750 con 0x7f88c40cd3e0 2024-04-06T12:24:46.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.512+0000 7f88ca41c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f88c406b8c0 0x7f88c406dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:46.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.512+0000 7f88c9c1b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88c40cd3e0 0x7f88c40cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:46.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.512+0000 7f88ca41c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f88c406b8c0 0x7f88c406dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:57730/0 (socket says 172.21.15.189:57730) 2024-04-06T12:24:46.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.512+0000 7f88ca41c700 1 -- 172.21.15.189:0/4085935384 learned_addr learned my addr 172.21.15.189:0/4085935384 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:46.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.512+0000 7f88c9c1b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88c40cd3e0 0x7f88c40cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:55348/0 (socket says 172.21.15.189:55348) 2024-04-06T12:24:46.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.512+0000 7f88c9c1b700 1 -- 172.21.15.189:0/4085935384 >> v2:172.21.15.192:3300/0 conn(0x7f88c406b8c0 msgr2=0x7f88c406dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:46.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.512+0000 7f88c9c1b700 1 --2- 172.21.15.189:0/4085935384 >> v2:172.21.15.192:3300/0 conn(0x7f88c406b8c0 0x7f88c406dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.512+0000 7f88c9c1b700 1 -- 172.21.15.189:0/4085935384 >> v1:172.21.15.192:6789/0 conn(0x7f88c415d7c0 legacy=0x7f88c406b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:46.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.512+0000 7f88c9c1b700 1 -- 172.21.15.189:0/4085935384 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c40cdd00 con 0x7f88c40cd3e0 2024-04-06T12:24:46.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.512+0000 7f88ca41c700 1 --2- 172.21.15.189:0/4085935384 >> v2:172.21.15.192:3300/0 conn(0x7f88c406b8c0 0x7f88c406dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:46.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.513+0000 7f88c9c1b700 1 --2- 172.21.15.189:0/4085935384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88c40cd3e0 0x7f88c40cd7c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009840 tx=0x7f88c0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ed20e94e62fddee server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:46.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.513+0000 7f88c8c19700 1 -- 172.21.15.189:0/4085935384 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c0011830 con 0x7f88c40cd3e0 2024-04-06T12:24:46.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.513+0000 7f88c8c19700 1 -- 172.21.15.189:0/4085935384 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c0011990 con 0x7f88c40cd3e0 2024-04-06T12:24:46.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.513+0000 7f88c8c19700 1 -- 172.21.15.189:0/4085935384 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c0011d20 con 0x7f88c40cd3e0 2024-04-06T12:24:46.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.513+0000 7f88cbe7f700 1 -- 172.21.15.189:0/4085935384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88c40cd3e0 msgr2=0x7f88c40cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:46.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.513+0000 7f88cbe7f700 1 --2- 172.21.15.189:0/4085935384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88c40cd3e0 0x7f88c40cd7c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009840 tx=0x7f88c0009b50 comp rx=0 tx=0).stop 2024-04-06T12:24:46.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.513+0000 7f88cbe7f700 1 -- 172.21.15.189:0/4085935384 shutdown_connections 2024-04-06T12:24:46.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.513+0000 7f88cbe7f700 1 --2- 172.21.15.189:0/4085935384 >> v2:172.21.15.192:3300/0 conn(0x7f88c406b8c0 0x7f88c406dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.513+0000 7f88cbe7f700 1 --2- 172.21.15.189:0/4085935384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88c40cd3e0 0x7f88c40cd7c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.513+0000 7f88cbe7f700 1 -- 172.21.15.189:0/4085935384 >> 172.21.15.189:0/4085935384 conn(0x7f88c40cc290 msgr2=0x7f88c40cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:46.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.513+0000 7f88cbe7f700 1 -- 172.21.15.189:0/4085935384 shutdown_connections 2024-04-06T12:24:46.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.513+0000 7f88cbe7f700 1 -- 172.21.15.189:0/4085935384 wait complete. 2024-04-06T12:24:46.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.514+0000 7f88cbe7f700 1 Processor -- start 2024-04-06T12:24:46.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.515+0000 7f88cbe7f700 1 -- start start 2024-04-06T12:24:46.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.515+0000 7f88cbe7f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88c406b8c0 0x7f88c40c5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:46.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.515+0000 7f88cbe7f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88c40cd3e0 0x7f88c41f6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:46.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.515+0000 7f88cbe7f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88c415d7c0 0x7f88c40b3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:46.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.515+0000 7f88cbe7f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c413e4c0 con 0x7f88c40cd3e0 2024-04-06T12:24:46.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.515+0000 7f88cbe7f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c41f6f50 con 0x7f88c415d7c0 2024-04-06T12:24:46.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.515+0000 7f88cbe7f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c41f7230 con 0x7f88c406b8c0 2024-04-06T12:24:46.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.515+0000 7f88c941a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88c40cd3e0 0x7f88c41f6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:46.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.515+0000 7f88c941a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88c40cd3e0 0x7f88c41f6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60558/0 (socket says 172.21.15.189:60558) 2024-04-06T12:24:46.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.515+0000 7f88c941a700 1 -- 172.21.15.189:0/4211917629 learned_addr learned my addr 172.21.15.189:0/4211917629 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:46.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.515+0000 7f88ca41c700 1 --2- 172.21.15.189:0/4211917629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88c415d7c0 0x7f88c40b3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:46.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.515+0000 7f88c9c1b700 1 --2- 172.21.15.189:0/4211917629 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88c406b8c0 0x7f88c40c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:46.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.516+0000 7f88c941a700 1 -- 172.21.15.189:0/4211917629 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88c406b8c0 msgr2=0x7f88c40c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:46.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.516+0000 7f88c941a700 1 --2- 172.21.15.189:0/4211917629 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88c406b8c0 0x7f88c40c5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.516+0000 7f88c941a700 1 -- 172.21.15.189:0/4211917629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88c415d7c0 msgr2=0x7f88c40b3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:46.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.516+0000 7f88c941a700 1 --2- 172.21.15.189:0/4211917629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88c415d7c0 0x7f88c40b3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.516+0000 7f88c941a700 1 -- 172.21.15.189:0/4211917629 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88bc009b30 con 0x7f88c40cd3e0 2024-04-06T12:24:46.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.516+0000 7f88c9c1b700 1 --2- 172.21.15.189:0/4211917629 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88c406b8c0 0x7f88c40c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:46.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.516+0000 7f88c941a700 1 --2- 172.21.15.189:0/4211917629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88c40cd3e0 0x7f88c41f6700 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f88b400c420 tx=0x7f88b400c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:46.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.516+0000 7f88ba7fc700 1 -- 172.21.15.189:0/4211917629 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b40052d0 con 0x7f88c40cd3e0 2024-04-06T12:24:46.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.516+0000 7f88ba7fc700 1 -- 172.21.15.189:0/4211917629 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b4005430 con 0x7f88c40cd3e0 2024-04-06T12:24:46.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.516+0000 7f88cbe7f700 1 -- 172.21.15.189:0/4211917629 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c00094a0 con 0x7f88c40cd3e0 2024-04-06T12:24:46.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.516+0000 7f88ba7fc700 1 -- 172.21.15.189:0/4211917629 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b40038c0 con 0x7f88c40cd3e0 2024-04-06T12:24:46.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.516+0000 7f88cbe7f700 1 -- 172.21.15.189:0/4211917629 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c413e760 con 0x7f88c40cd3e0 2024-04-06T12:24:46.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.517+0000 7f88cbe7f700 1 -- 172.21.15.189:0/4211917629 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c40cbcf0 con 0x7f88c40cd3e0 2024-04-06T12:24:46.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.518+0000 7f88ba7fc700 1 -- 172.21.15.189:0/4211917629 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 15) v1 ==== 89316+0+0 (secure 0 0 0) 0x7f88b4003a20 con 0x7f88c40cd3e0 2024-04-06T12:24:46.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.518+0000 7f88ba7fc700 1 --2- 172.21.15.189:0/4211917629 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f88a806ae40 0x7f88a806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:46.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.518+0000 7f88c9c1b700 1 -- 172.21.15.189:0/4211917629 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f88a806ae40 msgr2=0x7f88a806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:46.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.518+0000 7f88ba7fc700 1 -- 172.21.15.189:0/4211917629 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6868+0+0 (secure 0 0 0) 0x7f88b4089250 con 0x7f88c40cd3e0 2024-04-06T12:24:46.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.518+0000 7f88c9c1b700 1 --2- 172.21.15.189:0/4211917629 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f88a806ae40 0x7f88a806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:24:46.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.522+0000 7f88ba7fc700 1 -- 172.21.15.189:0/4211917629 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f88b4053d00 con 0x7f88c40cd3e0 2024-04-06T12:24:46.536 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:46.538+0000 7f51465c0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:24:46.624 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:46.627+0000 7f44d481c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:24:46.695 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:46.697+0000 7f51465c0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:24:46.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.719+0000 7f88c9c1b700 1 -- 172.21.15.189:0/4211917629 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f88a806ae40 msgr2=0x7f88a806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:46.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.719+0000 7f88c9c1b700 1 --2- 172.21.15.189:0/4211917629 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f88a806ae40 0x7f88a806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:24:46.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.725+0000 7f88cbe7f700 1 -- 172.21.15.189:0/4211917629 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f88c413e950 con 0x7f88c40cd3e0 2024-04-06T12:24:46.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.725+0000 7f88ba7fc700 1 -- 172.21.15.189:0/4211917629 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) v1 ==== 100+0+11 (secure 0 0 0) 0x7f88b4017020 con 0x7f88c40cd3e0 2024-04-06T12:24:46.724 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:24:46.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.728+0000 7f88cbe7f700 1 -- 172.21.15.189:0/4211917629 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f88a806ae40 msgr2=0x7f88a806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:24:46.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.728+0000 7f88cbe7f700 1 --2- 172.21.15.189:0/4211917629 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f88a806ae40 0x7f88a806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.728+0000 7f88cbe7f700 1 -- 172.21.15.189:0/4211917629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88c40cd3e0 msgr2=0x7f88c41f6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:46.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.728+0000 7f88cbe7f700 1 --2- 172.21.15.189:0/4211917629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88c40cd3e0 0x7f88c41f6700 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f88b400c420 tx=0x7f88b400c730 comp rx=0 tx=0).stop 2024-04-06T12:24:46.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.728+0000 7f88cbe7f700 1 -- 172.21.15.189:0/4211917629 shutdown_connections 2024-04-06T12:24:46.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.728+0000 7f88cbe7f700 1 --2- 172.21.15.189:0/4211917629 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f88a806ae40 0x7f88a806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.728+0000 7f88cbe7f700 1 --2- 172.21.15.189:0/4211917629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88c415d7c0 0x7f88c40b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.728+0000 7f88cbe7f700 1 --2- 172.21.15.189:0/4211917629 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88c406b8c0 0x7f88c40c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.728+0000 7f88cbe7f700 1 --2- 172.21.15.189:0/4211917629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88c40cd3e0 0x7f88c41f6700 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.728+0000 7f88cbe7f700 1 -- 172.21.15.189:0/4211917629 >> 172.21.15.189:0/4211917629 conn(0x7f88c40cc290 msgr2=0x7f88c40af9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:46.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.728+0000 7f88cbe7f700 1 -- 172.21.15.189:0/4211917629 shutdown_connections 2024-04-06T12:24:46.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.728+0000 7f88cbe7f700 1 -- 172.21.15.189:0/4211917629 wait complete. 2024-04-06T12:24:46.739 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:24:46.860 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:46.862+0000 7f51465c0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:24:46.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.867+0000 7fe6310cf700 1 Processor -- start 2024-04-06T12:24:46.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.867+0000 7fe6310cf700 1 -- start start 2024-04-06T12:24:46.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.868+0000 7fe6310cf700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe62c15a550 0x7fe62c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:46.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.868+0000 7fe6310cf700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe62c0cf900 0x7fe62c0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:46.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.868+0000 7fe6310cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe62c0b4ba0 0x7fe62c06eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:46.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.868+0000 7fe6310cf700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe62c0cdbe0 con 0x7fe62c15a550 2024-04-06T12:24:46.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.868+0000 7fe6310cf700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe62c0b4ff0 con 0x7fe62c0cf900 2024-04-06T12:24:46.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.868+0000 7fe6310cf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c15a940 con 0x7fe62c0b4ba0 2024-04-06T12:24:46.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.868+0000 7fe62ad9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe62c15a550 0x7fe62c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:46.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.868+0000 7fe62ad9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe62c15a550 0x7fe62c0cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60562/0 (socket says 172.21.15.189:60562) 2024-04-06T12:24:46.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.868+0000 7fe62ad9d700 1 -- 172.21.15.189:0/906703947 learned_addr learned my addr 172.21.15.189:0/906703947 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:46.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.868+0000 7fe62b59e700 1 --2- 172.21.15.189:0/906703947 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe62c0b4ba0 0x7fe62c06eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:46.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.868+0000 7fe62a59c700 1 --2- 172.21.15.189:0/906703947 >> v2:172.21.15.192:3300/0 conn(0x7fe62c0cf900 0x7fe62c0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:46.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.869+0000 7fe62ad9d700 1 -- 172.21.15.189:0/906703947 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe62c0b4ba0 msgr2=0x7fe62c06eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:46.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.869+0000 7fe62ad9d700 1 --2- 172.21.15.189:0/906703947 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe62c0b4ba0 0x7fe62c06eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.869+0000 7fe62ad9d700 1 -- 172.21.15.189:0/906703947 >> v2:172.21.15.192:3300/0 conn(0x7fe62c0cf900 msgr2=0x7fe62c0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:46.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.869+0000 7fe62ad9d700 1 --2- 172.21.15.189:0/906703947 >> v2:172.21.15.192:3300/0 conn(0x7fe62c0cf900 0x7fe62c0b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.869+0000 7fe62ad9d700 1 -- 172.21.15.189:0/906703947 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe62c15ac00 con 0x7fe62c15a550 2024-04-06T12:24:46.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.869+0000 7fe62b59e700 1 --2- 172.21.15.189:0/906703947 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe62c0b4ba0 0x7fe62c06eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:24:46.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.869+0000 7fe62a59c700 1 --2- 172.21.15.189:0/906703947 >> v2:172.21.15.192:3300/0 conn(0x7fe62c0cf900 0x7fe62c0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:24:46.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.869+0000 7fe62ad9d700 1 --2- 172.21.15.189:0/906703947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe62c15a550 0x7fe62c0cf3c0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe62c0b2d40 tx=0x7fe61c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=350dba0bdc2ca530 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:46.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.869+0000 7fe629d9b700 1 -- 172.21.15.189:0/906703947 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c00b870 con 0x7fe62c15a550 2024-04-06T12:24:46.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.869+0000 7fe629d9b700 1 -- 172.21.15.189:0/906703947 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61c00b9d0 con 0x7fe62c15a550 2024-04-06T12:24:46.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.869+0000 7fe629d9b700 1 -- 172.21.15.189:0/906703947 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c00bd60 con 0x7fe62c15a550 2024-04-06T12:24:46.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.869+0000 7fe6310cf700 1 -- 172.21.15.189:0/906703947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe62c15a550 msgr2=0x7fe62c0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:46.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.869+0000 7fe6310cf700 1 --2- 172.21.15.189:0/906703947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe62c15a550 0x7fe62c0cf3c0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe62c0b2d40 tx=0x7fe61c009bd0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.870+0000 7fe6310cf700 1 -- 172.21.15.189:0/906703947 shutdown_connections 2024-04-06T12:24:46.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.870+0000 7fe6310cf700 1 --2- 172.21.15.189:0/906703947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe62c15a550 0x7fe62c0cf3c0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.870+0000 7fe6310cf700 1 --2- 172.21.15.189:0/906703947 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe62c0b4ba0 0x7fe62c06eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.870+0000 7fe6310cf700 1 --2- 172.21.15.189:0/906703947 >> v2:172.21.15.192:3300/0 conn(0x7fe62c0cf900 0x7fe62c0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.870+0000 7fe6310cf700 1 -- 172.21.15.189:0/906703947 >> 172.21.15.189:0/906703947 conn(0x7fe62c0d00b0 msgr2=0x7fe62c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:46.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.870+0000 7fe6310cf700 1 -- 172.21.15.189:0/906703947 shutdown_connections 2024-04-06T12:24:46.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.870+0000 7fe6310cf700 1 -- 172.21.15.189:0/906703947 wait complete. 2024-04-06T12:24:46.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.871+0000 7fe6310cf700 1 Processor -- start 2024-04-06T12:24:46.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.871+0000 7fe6310cf700 1 -- start start 2024-04-06T12:24:46.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.872+0000 7fe6310cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe62c0b4ba0 0x7fe62c13d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:46.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.872+0000 7fe6310cf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe62c0cf900 0x7fe62c06a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:46.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.872+0000 7fe6310cf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe62c06ac70 0x7fe62c152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:46.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.872+0000 7fe6310cf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c06b100 con 0x7fe62c0cf900 2024-04-06T12:24:46.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.872+0000 7fe6310cf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c151510 con 0x7fe62c06ac70 2024-04-06T12:24:46.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.872+0000 7fe6310cf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c153710 con 0x7fe62c0b4ba0 2024-04-06T12:24:46.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe62a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe62c0cf900 0x7fe62c06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:46.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe62ad9d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe62c0b4ba0 0x7fe62c13d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:46.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe62b59e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe62c06ac70 0x7fe62c152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:46.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe62ad9d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe62c0b4ba0 0x7fe62c13d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:55374/0 (socket says 172.21.15.189:55374) 2024-04-06T12:24:46.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe62a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe62c0cf900 0x7fe62c06a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60574/0 (socket says 172.21.15.189:60574) 2024-04-06T12:24:46.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe62ad9d700 1 -- 172.21.15.189:0/1993649567 learned_addr learned my addr 172.21.15.189:0/1993649567 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:46.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe62b59e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe62c06ac70 0x7fe62c152a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:57762/0 (socket says 172.21.15.189:57762) 2024-04-06T12:24:46.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe62ad9d700 1 -- 172.21.15.189:0/1993649567 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe62c06ac70 msgr2=0x7fe62c152a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:46.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe62ad9d700 1 --2- 172.21.15.189:0/1993649567 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe62c06ac70 0x7fe62c152a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe62ad9d700 1 -- 172.21.15.189:0/1993649567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe62c0cf900 msgr2=0x7fe62c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:46.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe62ad9d700 1 --2- 172.21.15.189:0/1993649567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe62c0cf900 0x7fe62c06a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:46.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe62ad9d700 1 -- 172.21.15.189:0/1993649567 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61c009550 con 0x7fe62c0b4ba0 2024-04-06T12:24:46.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe62a59c700 1 --2- 172.21.15.189:0/1993649567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe62c0cf900 0x7fe62c06a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:46.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe62b59e700 1 --2- 172.21.15.189:0/1993649567 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe62c06ac70 0x7fe62c152a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:46.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe62ad9d700 1 --2- 172.21.15.189:0/1993649567 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe62c0b4ba0 0x7fe62c13d980 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe61c000c00 tx=0x7fe61c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:46.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe6137fe700 1 -- 172.21.15.189:0/1993649567 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c004480 con 0x7fe62c0b4ba0 2024-04-06T12:24:46.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe6137fe700 1 -- 172.21.15.189:0/1993649567 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61c0045e0 con 0x7fe62c0b4ba0 2024-04-06T12:24:46.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe6310cf700 1 -- 172.21.15.189:0/1993649567 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe62c1539f0 con 0x7fe62c0b4ba0 2024-04-06T12:24:46.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe6137fe700 1 -- 172.21.15.189:0/1993649567 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c0038e0 con 0x7fe62c0b4ba0 2024-04-06T12:24:46.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.873+0000 7fe6310cf700 1 -- 172.21.15.189:0/1993649567 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe62c153fb0 con 0x7fe62c0b4ba0 2024-04-06T12:24:46.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.874+0000 7fe6310cf700 1 -- 172.21.15.189:0/1993649567 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe62c067bb0 con 0x7fe62c0b4ba0 2024-04-06T12:24:46.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.875+0000 7fe6137fe700 1 -- 172.21.15.189:0/1993649567 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 15) v1 ==== 89316+0+0 (secure 0 0 0) 0x7fe61c021070 con 0x7fe62c0b4ba0 2024-04-06T12:24:46.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.875+0000 7fe6137fe700 1 --2- 172.21.15.189:0/1993649567 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe61806ae40 0x7fe61806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:46.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.875+0000 7fe62a59c700 1 -- 172.21.15.189:0/1993649567 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe61806ae40 msgr2=0x7fe61806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:46.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.875+0000 7fe6137fe700 1 -- 172.21.15.189:0/1993649567 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6868+0+0 (secure 0 0 0) 0x7fe61c08b250 con 0x7fe62c0b4ba0 2024-04-06T12:24:46.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.875+0000 7fe62a59c700 1 --2- 172.21.15.189:0/1993649567 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe61806ae40 0x7fe61806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:24:46.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:46.880+0000 7fe6137fe700 1 -- 172.21.15.189:0/1993649567 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe61c055db0 con 0x7fe62c0b4ba0 2024-04-06T12:24:47.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.076+0000 7fe62a59c700 1 -- 172.21.15.189:0/1993649567 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe61806ae40 msgr2=0x7fe61806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:47.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.076+0000 7fe62a59c700 1 --2- 172.21.15.189:0/1993649567 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe61806ae40 0x7fe61806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:24:47.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.133+0000 7fe6310cf700 1 -- 172.21.15.189:0/1993649567 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe62c074990 con 0x7fe62c0b4ba0 2024-04-06T12:24:47.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.133+0000 7fe6137fe700 1 -- 172.21.15.189:0/1993649567 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4740 (secure 0 0 0) 0x7fe61c0593d0 con 0x7fe62c0b4ba0 2024-04-06T12:24:47.131 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:47.132 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4865,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/2279772906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2279772906},{"type":"v1","addr":"172.21.15.192:6834","nonce":2279772906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4868,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/454939556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":454939556},{"type":"v1","addr":"172.21.15.192:6835","nonce":454939556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5107,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/399541295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":399541295},{"type":"v1","addr":"172.21.15.189:6837","nonce":399541295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:13:21.287908+0000","modified":"2024-04-06T12:13:24.295625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5232":{"gid":5232,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/1024118918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1024118918},{"type":"v1","addr":"172.21.15.189:6836","nonce":1024118918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-04-06T12:24:47.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.136+0000 7fe6310cf700 1 -- 172.21.15.189:0/1993649567 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe61806ae40 msgr2=0x7fe61806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:24:47.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.136+0000 7fe6310cf700 1 --2- 172.21.15.189:0/1993649567 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe61806ae40 0x7fe61806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.136+0000 7fe6310cf700 1 -- 172.21.15.189:0/1993649567 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe62c0b4ba0 msgr2=0x7fe62c13d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:47.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.136+0000 7fe6310cf700 1 --2- 172.21.15.189:0/1993649567 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe62c0b4ba0 0x7fe62c13d980 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe61c000c00 tx=0x7fe61c004750 comp rx=0 tx=0).stop 2024-04-06T12:24:47.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.136+0000 7fe6310cf700 1 -- 172.21.15.189:0/1993649567 shutdown_connections 2024-04-06T12:24:47.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.136+0000 7fe6310cf700 1 --2- 172.21.15.189:0/1993649567 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe61806ae40 0x7fe61806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.136+0000 7fe6310cf700 1 --2- 172.21.15.189:0/1993649567 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe62c06ac70 0x7fe62c152a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.136+0000 7fe6310cf700 1 --2- 172.21.15.189:0/1993649567 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe62c0b4ba0 0x7fe62c13d980 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.136+0000 7fe6310cf700 1 --2- 172.21.15.189:0/1993649567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe62c0cf900 0x7fe62c06a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.136+0000 7fe6310cf700 1 -- 172.21.15.189:0/1993649567 >> 172.21.15.189:0/1993649567 conn(0x7fe62c0d00b0 msgr2=0x7fe62c0b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:47.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.136+0000 7fe6310cf700 1 -- 172.21.15.189:0/1993649567 shutdown_connections 2024-04-06T12:24:47.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.136+0000 7fe6310cf700 1 -- 172.21.15.189:0/1993649567 wait complete. 2024-04-06T12:24:47.136 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 16 2024-04-06T12:24:47.146 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:24:47.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.273+0000 7fec7a74d700 1 Processor -- start 2024-04-06T12:24:47.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.273+0000 7fec7a74d700 1 -- start start 2024-04-06T12:24:47.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.274+0000 7fec7a74d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fec7415d7b0 0x7fec740b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:47.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.274+0000 7fec7a74d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec740b3990 con 0x7fec7406a730 2024-04-06T12:24:47.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.274+0000 7fec7a74d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec740b3060 con 0x7fec7415e500 2024-04-06T12:24:47.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.274+0000 7fec7a74d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec7415e950 con 0x7fec7415d7b0 2024-04-06T12:24:47.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.274+0000 7fec737fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fec7406a730 0x7fec7415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47132/0 (socket says 172.21.15.189:47132) 2024-04-06T12:24:47.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.274+0000 7fec737fe700 1 -- 172.21.15.189:0/151365810 learned_addr learned my addr 172.21.15.189:0/151365810 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:47.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.274+0000 7fec73fff700 1 --2- 172.21.15.189:0/151365810 >> v2:172.21.15.192:3300/0 conn(0x7fec7415d7b0 0x7fec740b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:47.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.274+0000 7fec72ffd700 1 -- 172.21.15.189:0/151365810 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994146810 0 0) 0x7fec740b3990 con 0x7fec7406a730 2024-04-06T12:24:47.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.274+0000 7fec72ffd700 1 -- 172.21.15.189:0/151365810 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec5c0035b0 con 0x7fec7406a730 2024-04-06T12:24:47.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.274+0000 7fec73fff700 1 -- 172.21.15.189:0/151365810 >> v1:172.21.15.192:6789/0 conn(0x7fec7415e500 legacy=0x7fec740717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:47.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.274+0000 7fec72ffd700 1 -- 172.21.15.189:0/151365810 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397919308 0 0) 0x7fec740b3060 con 0x7fec7415e500 2024-04-06T12:24:47.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.274+0000 7fec73fff700 1 -- 172.21.15.189:0/151365810 >> v1:172.21.15.189:6789/0 conn(0x7fec7406a730 legacy=0x7fec7415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:47.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.274+0000 7fec73fff700 1 -- 172.21.15.189:0/151365810 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec740b3340 con 0x7fec7415d7b0 2024-04-06T12:24:47.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.275+0000 7fec73fff700 1 --2- 172.21.15.189:0/151365810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec7415d7b0 0x7fec740b3e20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fec64009840 tx=0x7fec64009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26dea58f2b14b381 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:47.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.275+0000 7fec72ffd700 1 -- 172.21.15.189:0/151365810 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec64011830 con 0x7fec7415d7b0 2024-04-06T12:24:47.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.275+0000 7fec72ffd700 1 -- 172.21.15.189:0/151365810 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec64011990 con 0x7fec7415d7b0 2024-04-06T12:24:47.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.275+0000 7fec72ffd700 1 -- 172.21.15.189:0/151365810 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec64011d20 con 0x7fec7415d7b0 2024-04-06T12:24:47.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.275+0000 7fec7a74d700 1 -- 172.21.15.189:0/151365810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec7415d7b0 msgr2=0x7fec740b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:47.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.275+0000 7fec7a74d700 1 --2- 172.21.15.189:0/151365810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec7415d7b0 0x7fec740b3e20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fec64009840 tx=0x7fec64009b50 comp rx=0 tx=0).stop 2024-04-06T12:24:47.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.275+0000 7fec7a74d700 1 -- 172.21.15.189:0/151365810 shutdown_connections 2024-04-06T12:24:47.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.275+0000 7fec7a74d700 1 --2- 172.21.15.189:0/151365810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec7415d7b0 0x7fec740b3e20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.275+0000 7fec7a74d700 1 -- 172.21.15.189:0/151365810 >> 172.21.15.189:0/151365810 conn(0x7fec740d00b0 msgr2=0x7fec740d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:47.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.275+0000 7fec7a74d700 1 -- 172.21.15.189:0/151365810 shutdown_connections 2024-04-06T12:24:47.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.275+0000 7fec7a74d700 1 -- 172.21.15.189:0/151365810 wait complete. 2024-04-06T12:24:47.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.277+0000 7fec7a74d700 1 Processor -- start 2024-04-06T12:24:47.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.277+0000 7fec7a74d700 1 -- start start 2024-04-06T12:24:47.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.277+0000 7fec7a74d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec7406a730 0x7fec74159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:47.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.277+0000 7fec7a74d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec7415d7b0 0x7fec741f6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:47.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.277+0000 7fec7a74d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec7415e500 0x7fec74146960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:47.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.277+0000 7fec7a74d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec74146540 con 0x7fec7415d7b0 2024-04-06T12:24:47.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.277+0000 7fec7a74d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec741f6750 con 0x7fec7406a730 2024-04-06T12:24:47.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.277+0000 7fec7a74d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec741f6980 con 0x7fec7415e500 2024-04-06T12:24:47.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.278+0000 7fec737fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec7415d7b0 0x7fec741f6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:47.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.278+0000 7fec73fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec7406a730 0x7fec74159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:47.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.278+0000 7fec737fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec7415d7b0 0x7fec741f6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60586/0 (socket says 172.21.15.189:60586) 2024-04-06T12:24:47.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.278+0000 7fec737fe700 1 -- 172.21.15.189:0/264465924 learned_addr learned my addr 172.21.15.189:0/264465924 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:47.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.278+0000 7fec78cea700 1 --2- 172.21.15.189:0/264465924 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec7415e500 0x7fec74146960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:47.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.278+0000 7fec737fe700 1 -- 172.21.15.189:0/264465924 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec7415e500 msgr2=0x7fec74146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:47.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.278+0000 7fec737fe700 1 --2- 172.21.15.189:0/264465924 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec7415e500 0x7fec74146960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.278+0000 7fec737fe700 1 -- 172.21.15.189:0/264465924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec7406a730 msgr2=0x7fec74159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:47.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.278+0000 7fec737fe700 1 --2- 172.21.15.189:0/264465924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec7406a730 0x7fec74159450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.278+0000 7fec737fe700 1 -- 172.21.15.189:0/264465924 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec640094a0 con 0x7fec7415d7b0 2024-04-06T12:24:47.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.278+0000 7fec73fff700 1 --2- 172.21.15.189:0/264465924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec7406a730 0x7fec74159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:47.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.278+0000 7fec737fe700 1 --2- 172.21.15.189:0/264465924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec7415d7b0 0x7fec741f6210 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fec6800c4d0 tx=0x7fec6800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:47.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.278+0000 7fec70ff9700 1 -- 172.21.15.189:0/264465924 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68005d70 con 0x7fec7415d7b0 2024-04-06T12:24:47.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.278+0000 7fec70ff9700 1 -- 172.21.15.189:0/264465924 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec680070d0 con 0x7fec7415d7b0 2024-04-06T12:24:47.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.278+0000 7fec7a74d700 1 -- 172.21.15.189:0/264465924 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec741f6c60 con 0x7fec7415d7b0 2024-04-06T12:24:47.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.278+0000 7fec70ff9700 1 -- 172.21.15.189:0/264465924 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68007440 con 0x7fec7415d7b0 2024-04-06T12:24:47.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.278+0000 7fec7a74d700 1 -- 172.21.15.189:0/264465924 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec74146ea0 con 0x7fec7415d7b0 2024-04-06T12:24:47.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.279+0000 7fec7a74d700 1 -- 172.21.15.189:0/264465924 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec74146680 con 0x7fec7415d7b0 2024-04-06T12:24:47.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.280+0000 7fec70ff9700 1 -- 172.21.15.189:0/264465924 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 15) v1 ==== 89316+0+0 (secure 0 0 0) 0x7fec680075a0 con 0x7fec7415d7b0 2024-04-06T12:24:47.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.280+0000 7fec70ff9700 1 --2- 172.21.15.189:0/264465924 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec6006ae40 0x7fec6006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:47.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.280+0000 7fec73fff700 1 -- 172.21.15.189:0/264465924 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec6006ae40 msgr2=0x7fec6006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:47.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.280+0000 7fec70ff9700 1 -- 172.21.15.189:0/264465924 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6868+0+0 (secure 0 0 0) 0x7fec68088db0 con 0x7fec7415d7b0 2024-04-06T12:24:47.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.280+0000 7fec73fff700 1 --2- 172.21.15.189:0/264465924 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec6006ae40 0x7fec6006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:24:47.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.285+0000 7fec70ff9700 1 -- 172.21.15.189:0/264465924 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fec74146680 con 0x7fec7415d7b0 2024-04-06T12:24:47.349 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:47.352+0000 7f44d481c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:24:47.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.481+0000 7fec73fff700 1 -- 172.21.15.189:0/264465924 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec6006ae40 msgr2=0x7fec6006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:47.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.481+0000 7fec73fff700 1 --2- 172.21.15.189:0/264465924 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec6006ae40 0x7fec6006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:24:47.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.492+0000 7fec7a74d700 1 -- 172.21.15.189:0/264465924 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec74146680 con 0x7fec7415d7b0 2024-04-06T12:24:47.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.493+0000 7fec70ff9700 1 -- 172.21.15.189:0/264465924 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+17604 (secure 0 0 0) 0x7fec74146680 con 0x7fec7415d7b0 2024-04-06T12:24:47.491 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:47.491 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":53,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:24:47.484617+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:13:19.268280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-06T12:13:20.243771+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":53,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.35","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/155214328":"2024-04-07T12:13:34.836450+0000","172.21.15.189:0/3092563682":"2024-04-07T12:13:34.836450+0000","172.21.15.189:0/1075995033":"2024-04-07T12:13:34.836450+0000","172.21.15.189:6833/47128":"2024-04-07T12:13:34.836450+0000","172.21.15.189:0/3015633422":"2024-04-07T12:13:34.836450+0000","172.21.15.189:6832/47128":"2024-04-07T12:13:34.836450+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":1},{"begin":9,"length":1},{"begin":11,"length":1},{"begin":13,"length":2},{"begin":17,"length":2},{"begin":21,"length":2},{"begin":24,"length":2},{"begin":28,"length":2},{"begin":33,"length":1},{"begin":35,"length":2},{"begin":39,"length":2},{"begin":43,"length":1},{"begin":45,"length":2},{"begin":49,"length":3},{"begin":53,"length":1},{"begin":55,"length":2},{"begin":59,"length":4},{"begin":64,"length":1},{"begin":66,"length":2},{"begin":70,"length":3},{"begin":74,"length":1},{"begin":76,"length":2},{"begin":80,"length":5},{"begin":86,"length":1},{"begin":88,"length":2},{"begin":93,"length":3},{"begin":97,"length":1},{"begin":99,"length":2},{"begin":103,"length":3},{"begin":107,"length":1},{"begin":109,"length":2},{"begin":160,"length":62}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":107,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:24:47.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.495+0000 7fec7a74d700 1 -- 172.21.15.189:0/264465924 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec6006ae40 msgr2=0x7fec6006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:24:47.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.495+0000 7fec7a74d700 1 --2- 172.21.15.189:0/264465924 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec6006ae40 0x7fec6006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.495+0000 7fec7a74d700 1 -- 172.21.15.189:0/264465924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec7415d7b0 msgr2=0x7fec741f6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:47.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.495+0000 7fec7a74d700 1 --2- 172.21.15.189:0/264465924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec7415d7b0 0x7fec741f6210 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fec6800c4d0 tx=0x7fec6800c7e0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.495+0000 7fec7a74d700 1 -- 172.21.15.189:0/264465924 shutdown_connections 2024-04-06T12:24:47.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.495+0000 7fec7a74d700 1 --2- 172.21.15.189:0/264465924 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec7415e500 0x7fec74146960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.495+0000 7fec7a74d700 1 --2- 172.21.15.189:0/264465924 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec6006ae40 0x7fec6006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.495+0000 7fec7a74d700 1 --2- 172.21.15.189:0/264465924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec7406a730 0x7fec74159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.495+0000 7fec7a74d700 1 --2- 172.21.15.189:0/264465924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec7415d7b0 0x7fec741f6210 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.495+0000 7fec7a74d700 1 -- 172.21.15.189:0/264465924 >> 172.21.15.189:0/264465924 conn(0x7fec740d00b0 msgr2=0x7fec7406eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:47.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.495+0000 7fec7a74d700 1 -- 172.21.15.189:0/264465924 shutdown_connections 2024-04-06T12:24:47.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.495+0000 7fec7a74d700 1 -- 172.21.15.189:0/264465924 wait complete. 2024-04-06T12:24:47.507 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-06T12:24:47.507 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:24:47.588 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:47.591+0000 7f44d481c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:24:47.627 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:47.629+0000 7f51465c0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:24:47.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.635+0000 7f26b1a83700 1 Processor -- start 2024-04-06T12:24:47.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.635+0000 7f26b1a83700 1 -- start start 2024-04-06T12:24:47.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.636+0000 7f26b1a83700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f26ac0d11b0 0x7f26ac0d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:47.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.636+0000 7f26b1a83700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ac0cfa00 0x7f26ac0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:47.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.636+0000 7f26b1a83700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ac0d1ad0 con 0x7f26ac0cb870 2024-04-06T12:24:47.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.636+0000 7f26b1a83700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26ac0ccdc0 con 0x7f26ac0d11b0 2024-04-06T12:24:47.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.636+0000 7f26b1a83700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac0d0b50 con 0x7f26ac0cfa00 2024-04-06T12:24:47.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.636+0000 7f26aaffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f26ac0cb870 0x7f26ac15c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47146/0 (socket says 172.21.15.189:47146) 2024-04-06T12:24:47.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.636+0000 7f26aaffd700 1 -- 172.21.15.189:0/3026707534 learned_addr learned my addr 172.21.15.189:0/3026707534 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:47.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.636+0000 7f26ab7fe700 1 --2- 172.21.15.189:0/3026707534 >> v2:172.21.15.192:3300/0 conn(0x7f26ac0d11b0 0x7f26ac0d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:47.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.636+0000 7f26abfff700 1 --2- 172.21.15.189:0/3026707534 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ac0cfa00 0x7f26ac0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:47.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.637+0000 7f26aa7fc700 1 -- 172.21.15.189:0/3026707534 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843150264 0 0) 0x7f26ac0d1ad0 con 0x7f26ac0cb870 2024-04-06T12:24:47.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.637+0000 7f26aa7fc700 1 -- 172.21.15.189:0/3026707534 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26940035b0 con 0x7f26ac0cb870 2024-04-06T12:24:47.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.637+0000 7f26ab7fe700 1 -- 172.21.15.189:0/3026707534 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ac0cfa00 msgr2=0x7f26ac0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:47.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.637+0000 7f26ab7fe700 1 --2- 172.21.15.189:0/3026707534 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ac0cfa00 0x7f26ac0cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.637+0000 7f26ab7fe700 1 -- 172.21.15.189:0/3026707534 >> v1:172.21.15.189:6789/0 conn(0x7f26ac0cb870 legacy=0x7f26ac15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:47.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.637+0000 7f26ab7fe700 1 -- 172.21.15.189:0/3026707534 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ac0cbd80 con 0x7f26ac0d11b0 2024-04-06T12:24:47.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.637+0000 7f26abfff700 1 --2- 172.21.15.189:0/3026707534 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ac0cfa00 0x7f26ac0cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:47.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.637+0000 7f26ab7fe700 1 --2- 172.21.15.189:0/3026707534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ac0d11b0 0x7f26ac0d1590 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f269c009d90 tx=0x7f269c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1c9d211aad8711d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:47.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.637+0000 7f26aa7fc700 1 -- 172.21.15.189:0/3026707534 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c00e070 con 0x7f26ac0d11b0 2024-04-06T12:24:47.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.637+0000 7f26aa7fc700 1 -- 172.21.15.189:0/3026707534 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269c00bb20 con 0x7f26ac0d11b0 2024-04-06T12:24:47.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.637+0000 7f26aa7fc700 1 -- 172.21.15.189:0/3026707534 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c012430 con 0x7f26ac0d11b0 2024-04-06T12:24:47.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.638+0000 7f26b1a83700 1 -- 172.21.15.189:0/3026707534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ac0d11b0 msgr2=0x7f26ac0d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:47.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.638+0000 7f26b1a83700 1 --2- 172.21.15.189:0/3026707534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ac0d11b0 0x7f26ac0d1590 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f269c009d90 tx=0x7f269c00b330 comp rx=0 tx=0).stop 2024-04-06T12:24:47.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.638+0000 7f26b1a83700 1 -- 172.21.15.189:0/3026707534 shutdown_connections 2024-04-06T12:24:47.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.638+0000 7f26b1a83700 1 --2- 172.21.15.189:0/3026707534 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ac0cfa00 0x7f26ac0cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.638+0000 7f26b1a83700 1 --2- 172.21.15.189:0/3026707534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ac0d11b0 0x7f26ac0d1590 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.638+0000 7f26b1a83700 1 -- 172.21.15.189:0/3026707534 >> 172.21.15.189:0/3026707534 conn(0x7f26ac0b2db0 msgr2=0x7f26ac0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:47.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.638+0000 7f26b1a83700 1 -- 172.21.15.189:0/3026707534 shutdown_connections 2024-04-06T12:24:47.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.638+0000 7f26b1a83700 1 -- 172.21.15.189:0/3026707534 wait complete. 2024-04-06T12:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.639+0000 7f26b1a83700 1 Processor -- start 2024-04-06T12:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.639+0000 7f26b1a83700 1 -- start start 2024-04-06T12:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.640+0000 7f26b1a83700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ac0cb870 0x7f26ac15a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.640+0000 7f26b1a83700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ac0cfa00 0x7f26ac158820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.640+0000 7f26b1a83700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26ac0d11b0 0x7f26ac158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.640+0000 7f26b1a83700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac1592a0 con 0x7f26ac0d11b0 2024-04-06T12:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.640+0000 7f26b1a83700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac0ca400 con 0x7f26ac0cfa00 2024-04-06T12:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.640+0000 7f26b1a83700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac0ca6e0 con 0x7f26ac0cb870 2024-04-06T12:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.640+0000 7f26abfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26ac0d11b0 0x7f26ac158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.640+0000 7f26abfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26ac0d11b0 0x7f26ac158d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60590/0 (socket says 172.21.15.189:60590) 2024-04-06T12:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.640+0000 7f26abfff700 1 -- 172.21.15.189:0/4047088751 learned_addr learned my addr 172.21.15.189:0/4047088751 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.640+0000 7f26ab7fe700 1 --2- 172.21.15.189:0/4047088751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ac0cb870 0x7f26ac15a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.640+0000 7f26aaffd700 1 --2- 172.21.15.189:0/4047088751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ac0cfa00 0x7f26ac158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.640+0000 7f26abfff700 1 -- 172.21.15.189:0/4047088751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ac0cb870 msgr2=0x7f26ac15a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:47.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.640+0000 7f26abfff700 1 --2- 172.21.15.189:0/4047088751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ac0cb870 0x7f26ac15a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.640+0000 7f26abfff700 1 -- 172.21.15.189:0/4047088751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ac0cfa00 msgr2=0x7f26ac158820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:47.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.640+0000 7f26abfff700 1 --2- 172.21.15.189:0/4047088751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ac0cfa00 0x7f26ac158820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.640+0000 7f26abfff700 1 -- 172.21.15.189:0/4047088751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a40094a0 con 0x7f26ac0d11b0 2024-04-06T12:24:47.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.640+0000 7f26ab7fe700 1 --2- 172.21.15.189:0/4047088751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ac0cb870 0x7f26ac15a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:47.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.640+0000 7f26abfff700 1 --2- 172.21.15.189:0/4047088751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26ac0d11b0 0x7f26ac158d60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f26a400e930 tx=0x7f26a400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:47.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.641+0000 7f2693fff700 1 -- 172.21.15.189:0/4047088751 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a4009700 con 0x7f26ac0d11b0 2024-04-06T12:24:47.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.641+0000 7f2693fff700 1 -- 172.21.15.189:0/4047088751 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a4004020 con 0x7f26ac0d11b0 2024-04-06T12:24:47.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.641+0000 7f26b1a83700 1 -- 172.21.15.189:0/4047088751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f269c009a70 con 0x7f26ac0d11b0 2024-04-06T12:24:47.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.641+0000 7f2693fff700 1 -- 172.21.15.189:0/4047088751 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a40043b0 con 0x7f26ac0d11b0 2024-04-06T12:24:47.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.641+0000 7f26b1a83700 1 -- 172.21.15.189:0/4047088751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26ac0caca0 con 0x7f26ac0d11b0 2024-04-06T12:24:47.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.642+0000 7f26b1a83700 1 -- 172.21.15.189:0/4047088751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26ac074990 con 0x7f26ac0d11b0 2024-04-06T12:24:47.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.642+0000 7f2693fff700 1 -- 172.21.15.189:0/4047088751 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 15) v1 ==== 89316+0+0 (secure 0 0 0) 0x7f26a40036a0 con 0x7f26ac0d11b0 2024-04-06T12:24:47.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.642+0000 7f2693fff700 1 --2- 172.21.15.189:0/4047088751 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f269806ae40 0x7f269806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:47.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.642+0000 7f26ab7fe700 1 -- 172.21.15.189:0/4047088751 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f269806ae40 msgr2=0x7f269806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:47.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.642+0000 7f2693fff700 1 -- 172.21.15.189:0/4047088751 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6884+0+0 (secure 0 0 0) 0x7f26a40897b0 con 0x7f26ac0d11b0 2024-04-06T12:24:47.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.643+0000 7f26ab7fe700 1 --2- 172.21.15.189:0/4047088751 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f269806ae40 0x7f269806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:24:47.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.647+0000 7f2693fff700 1 -- 172.21.15.189:0/4047088751 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f26a40541e0 con 0x7f26ac0d11b0 2024-04-06T12:24:47.664 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:47.667+0000 7f44d481c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:24:47.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.844+0000 7f26ab7fe700 1 -- 172.21.15.189:0/4047088751 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f269806ae40 msgr2=0x7f269806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:47.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.844+0000 7f26ab7fe700 1 --2- 172.21.15.189:0/4047088751 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f269806ae40 0x7f269806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:24:47.885 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:47.887+0000 7f51465c0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:24:47.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.903+0000 7f26b1a83700 1 -- 172.21.15.189:0/4047088751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26ac0ce730 con 0x7f26ac0d11b0 2024-04-06T12:24:47.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.903+0000 7f2693fff700 1 -- 172.21.15.189:0/4047088751 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f26a4057800 con 0x7f26ac0d11b0 2024-04-06T12:24:47.901 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:47.902 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4865,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/2279772906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2279772906},{"type":"v1","addr":"172.21.15.192:6834","nonce":2279772906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4868,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/454939556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":454939556},{"type":"v1","addr":"172.21.15.192:6835","nonce":454939556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5107,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/399541295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":399541295},{"type":"v1","addr":"172.21.15.189:6837","nonce":399541295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:13:21.287908+0000","modified":"2024-04-06T12:13:24.295625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5232":{"gid":5232,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/1024118918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1024118918},{"type":"v1","addr":"172.21.15.189:6836","nonce":1024118918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-04-06T12:24:47.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.906+0000 7f26b1a83700 1 -- 172.21.15.189:0/4047088751 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f269806ae40 msgr2=0x7f269806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:24:47.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.906+0000 7f26b1a83700 1 --2- 172.21.15.189:0/4047088751 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f269806ae40 0x7f269806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.906+0000 7f26b1a83700 1 -- 172.21.15.189:0/4047088751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26ac0d11b0 msgr2=0x7f26ac158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:47.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.906+0000 7f26b1a83700 1 --2- 172.21.15.189:0/4047088751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26ac0d11b0 0x7f26ac158d60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f26a400e930 tx=0x7f26a400ec40 comp rx=0 tx=0).stop 2024-04-06T12:24:47.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.906+0000 7f26b1a83700 1 -- 172.21.15.189:0/4047088751 shutdown_connections 2024-04-06T12:24:47.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.906+0000 7f26b1a83700 1 --2- 172.21.15.189:0/4047088751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26ac0d11b0 0x7f26ac158d60 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.906+0000 7f26b1a83700 1 --2- 172.21.15.189:0/4047088751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ac0cb870 0x7f26ac15a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.906+0000 7f26b1a83700 1 --2- 172.21.15.189:0/4047088751 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f269806ae40 0x7f269806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.906+0000 7f26b1a83700 1 --2- 172.21.15.189:0/4047088751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ac0cfa00 0x7f26ac158820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:47.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.906+0000 7f26b1a83700 1 -- 172.21.15.189:0/4047088751 >> 172.21.15.189:0/4047088751 conn(0x7f26ac0b2db0 msgr2=0x7f26ac0cd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:47.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.906+0000 7f26b1a83700 1 -- 172.21.15.189:0/4047088751 shutdown_connections 2024-04-06T12:24:47.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:47.906+0000 7f26b1a83700 1 -- 172.21.15.189:0/4047088751 wait complete. 2024-04-06T12:24:47.905 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 16 2024-04-06T12:24:47.916 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:24:47.967 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:47.969+0000 7f51465c0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:24:48.036 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:24:48.039+0000 7f44d481c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:24:48.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.044+0000 7f04cd6cf700 1 Processor -- start 2024-04-06T12:24:48.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.044+0000 7f04cd6cf700 1 -- start start 2024-04-06T12:24:48.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.045+0000 7f04cd6cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04c8143470 0x7f04c80b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:48.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.045+0000 7f04cd6cf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c815eb40 con 0x7f04c815db10 2024-04-06T12:24:48.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.045+0000 7f04cd6cf700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c80d0bf0 con 0x7f04c8160b60 2024-04-06T12:24:48.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.045+0000 7f04cd6cf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c8160f40 con 0x7f04c8143470 2024-04-06T12:24:48.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.045+0000 7f04c67fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f04c815db10 0x7f04c80cd890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47160/0 (socket says 172.21.15.189:47160) 2024-04-06T12:24:48.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.045+0000 7f04c67fc700 1 -- 172.21.15.189:0/3433781148 learned_addr learned my addr 172.21.15.189:0/3433781148 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:48.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.045+0000 7f04c77fe700 1 --2- 172.21.15.189:0/3433781148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04c8143470 0x7f04c80b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:48.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.045+0000 7f04c5ffb700 1 -- 172.21.15.189:0/3433781148 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536465249 0 0) 0x7f04b8001290 con 0x7f04c815db10 2024-04-06T12:24:48.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.045+0000 7f04c5ffb700 1 -- 172.21.15.189:0/3433781148 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b00035b0 con 0x7f04c815db10 2024-04-06T12:24:48.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.045+0000 7f04c5ffb700 1 -- 172.21.15.189:0/3433781148 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066565197 0 0) 0x7f04c80d0bf0 con 0x7f04c8160b60 2024-04-06T12:24:48.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.045+0000 7f04c5ffb700 1 -- 172.21.15.189:0/3433781148 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b8001290 con 0x7f04c8160b60 2024-04-06T12:24:48.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.045+0000 7f04c77fe700 1 -- 172.21.15.189:0/3433781148 >> v1:172.21.15.192:6789/0 conn(0x7f04c8160b60 legacy=0x7f04c8142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:48.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.045+0000 7f04c5ffb700 1 -- 172.21.15.189:0/3433781148 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1878185478 0 0) 0x7f04b00035b0 con 0x7f04c815db10 2024-04-06T12:24:48.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.045+0000 7f04c77fe700 1 -- 172.21.15.189:0/3433781148 >> v1:172.21.15.189:6789/0 conn(0x7f04c815db10 legacy=0x7f04c80cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:48.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.045+0000 7f04c77fe700 1 -- 172.21.15.189:0/3433781148 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c815dfc0 con 0x7f04c8143470 2024-04-06T12:24:48.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.046+0000 7f04c77fe700 1 --2- 172.21.15.189:0/3433781148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04c8143470 0x7f04c80b37f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f04c0009dc0 tx=0x7f04c000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ac3f9f5058efced server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:48.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.046+0000 7f04c5ffb700 1 -- 172.21.15.189:0/3433781148 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c000e070 con 0x7f04c8143470 2024-04-06T12:24:48.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.046+0000 7f04c5ffb700 1 -- 172.21.15.189:0/3433781148 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c0003a20 con 0x7f04c8143470 2024-04-06T12:24:48.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.046+0000 7f04c5ffb700 1 -- 172.21.15.189:0/3433781148 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b8001980 con 0x7f04c8143470 2024-04-06T12:24:48.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.046+0000 7f04cd6cf700 1 -- 172.21.15.189:0/3433781148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04c8143470 msgr2=0x7f04c80b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:48.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.046+0000 7f04cd6cf700 1 --2- 172.21.15.189:0/3433781148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04c8143470 0x7f04c80b37f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f04c0009dc0 tx=0x7f04c000b330 comp rx=0 tx=0).stop 2024-04-06T12:24:48.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.046+0000 7f04cd6cf700 1 -- 172.21.15.189:0/3433781148 shutdown_connections 2024-04-06T12:24:48.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.046+0000 7f04cd6cf700 1 --2- 172.21.15.189:0/3433781148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04c8143470 0x7f04c80b37f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.046+0000 7f04cd6cf700 1 -- 172.21.15.189:0/3433781148 >> 172.21.15.189:0/3433781148 conn(0x7f04c806a730 msgr2=0x7f04c806ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:48.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.046+0000 7f04cd6cf700 1 -- 172.21.15.189:0/3433781148 shutdown_connections 2024-04-06T12:24:48.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.046+0000 7f04cd6cf700 1 -- 172.21.15.189:0/3433781148 wait complete. 2024-04-06T12:24:48.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.048+0000 7f04cd6cf700 1 Processor -- start 2024-04-06T12:24:48.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.048+0000 7f04cd6cf700 1 -- start start 2024-04-06T12:24:48.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.048+0000 7f04cd6cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04c8143470 0x7f04c81f5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:48.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.048+0000 7f04cd6cf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04c815db10 0x7f04c80b5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:48.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.048+0000 7f04cd6cf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04c8160b60 0x7f04c80b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:48.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.048+0000 7f04cd6cf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c80c51d0 con 0x7f04c815db10 2024-04-06T12:24:48.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.048+0000 7f04cd6cf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c80b67e0 con 0x7f04c8160b60 2024-04-06T12:24:48.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.048+0000 7f04cd6cf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c80b6220 con 0x7f04c8143470 2024-04-06T12:24:48.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.048+0000 7f04c67fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04c815db10 0x7f04c80b5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:48.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.049+0000 7f04c67fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04c815db10 0x7f04c80b5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60600/0 (socket says 172.21.15.189:60600) 2024-04-06T12:24:48.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.049+0000 7f04c67fc700 1 -- 172.21.15.189:0/1271494844 learned_addr learned my addr 172.21.15.189:0/1271494844 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:48.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.049+0000 7f04c6ffd700 1 --2- 172.21.15.189:0/1271494844 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04c8143470 0x7f04c81f5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:48.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.049+0000 7f04c77fe700 1 --2- 172.21.15.189:0/1271494844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04c8160b60 0x7f04c80b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:48.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.049+0000 7f04c67fc700 1 -- 172.21.15.189:0/1271494844 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04c8143470 msgr2=0x7f04c81f5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:48.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.049+0000 7f04c67fc700 1 --2- 172.21.15.189:0/1271494844 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04c8143470 0x7f04c81f5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.049+0000 7f04c67fc700 1 -- 172.21.15.189:0/1271494844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04c8160b60 msgr2=0x7f04c80b5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:48.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.049+0000 7f04c67fc700 1 --2- 172.21.15.189:0/1271494844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04c8160b60 0x7f04c80b5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.049+0000 7f04c67fc700 1 -- 172.21.15.189:0/1271494844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c0009a70 con 0x7f04c815db10 2024-04-06T12:24:48.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.049+0000 7f04c67fc700 1 --2- 172.21.15.189:0/1271494844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04c815db10 0x7f04c80b5660 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f04b800c730 tx=0x7f04b800caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:48.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.049+0000 7f04af7fe700 1 -- 172.21.15.189:0/1271494844 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b8007500 con 0x7f04c815db10 2024-04-06T12:24:48.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.049+0000 7f04af7fe700 1 -- 172.21.15.189:0/1271494844 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b8007660 con 0x7f04c815db10 2024-04-06T12:24:48.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.049+0000 7f04cd6cf700 1 -- 172.21.15.189:0/1271494844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c80b6500 con 0x7f04c815db10 2024-04-06T12:24:48.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.049+0000 7f04af7fe700 1 -- 172.21.15.189:0/1271494844 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b8005a50 con 0x7f04c815db10 2024-04-06T12:24:48.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.049+0000 7f04cd6cf700 1 -- 172.21.15.189:0/1271494844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c80b3a10 con 0x7f04c815db10 2024-04-06T12:24:48.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.050+0000 7f04cd6cf700 1 -- 172.21.15.189:0/1271494844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c8074990 con 0x7f04c815db10 2024-04-06T12:24:48.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.051+0000 7f04af7fe700 1 -- 172.21.15.189:0/1271494844 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 15) v1 ==== 89316+0+0 (secure 0 0 0) 0x7f04b8005bb0 con 0x7f04c815db10 2024-04-06T12:24:48.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.051+0000 7f04af7fe700 1 --2- 172.21.15.189:0/1271494844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f04b406ae40 0x7f04b406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:48.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.051+0000 7f04c6ffd700 1 -- 172.21.15.189:0/1271494844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f04b406ae40 msgr2=0x7f04b406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:48.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.051+0000 7f04af7fe700 1 -- 172.21.15.189:0/1271494844 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6884+0+0 (secure 0 0 0) 0x7f04b8088fe0 con 0x7f04c815db10 2024-04-06T12:24:48.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.051+0000 7f04c6ffd700 1 --2- 172.21.15.189:0/1271494844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f04b406ae40 0x7f04b406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:24:48.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.055+0000 7f04af7fe700 1 -- 172.21.15.189:0/1271494844 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f04b8053b30 con 0x7f04c815db10 2024-04-06T12:24:48.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.252+0000 7f04c6ffd700 1 -- 172.21.15.189:0/1271494844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f04b406ae40 msgr2=0x7f04b406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:24:48.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.252+0000 7f04c6ffd700 1 --2- 172.21.15.189:0/1271494844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f04b406ae40 0x7f04b406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:24:48.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.305+0000 7f04cd6cf700 1 -- 172.21.15.189:0/1271494844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04c815eb40 con 0x7f04c815db10 2024-04-06T12:24:48.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.305+0000 7f04af7fe700 1 -- 172.21.15.189:0/1271494844 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f04c815eb40 con 0x7f04c815db10 2024-04-06T12:24:48.304 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:48.304 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4865,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/2279772906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2279772906},{"type":"v1","addr":"172.21.15.192:6834","nonce":2279772906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4868,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/454939556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":454939556},{"type":"v1","addr":"172.21.15.192:6835","nonce":454939556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5107,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/399541295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":399541295},{"type":"v1","addr":"172.21.15.189:6837","nonce":399541295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:13:21.287908+0000","modified":"2024-04-06T12:13:24.295625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5232":{"gid":5232,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/1024118918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1024118918},{"type":"v1","addr":"172.21.15.189:6836","nonce":1024118918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-04-06T12:24:48.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.307+0000 7f04cd6cf700 1 -- 172.21.15.189:0/1271494844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f04b406ae40 msgr2=0x7f04b406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:24:48.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.308+0000 7f04cd6cf700 1 --2- 172.21.15.189:0/1271494844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f04b406ae40 0x7f04b406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.308+0000 7f04cd6cf700 1 -- 172.21.15.189:0/1271494844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04c815db10 msgr2=0x7f04c80b5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:48.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.308+0000 7f04cd6cf700 1 --2- 172.21.15.189:0/1271494844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04c815db10 0x7f04c80b5660 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f04b800c730 tx=0x7f04b800caf0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.308+0000 7f04cd6cf700 1 -- 172.21.15.189:0/1271494844 shutdown_connections 2024-04-06T12:24:48.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.308+0000 7f04cd6cf700 1 --2- 172.21.15.189:0/1271494844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f04b406ae40 0x7f04b406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.308+0000 7f04cd6cf700 1 --2- 172.21.15.189:0/1271494844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04c8160b60 0x7f04c80b5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.308+0000 7f04cd6cf700 1 --2- 172.21.15.189:0/1271494844 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04c8143470 0x7f04c81f5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.308+0000 7f04cd6cf700 1 --2- 172.21.15.189:0/1271494844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04c815db10 0x7f04c80b5660 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.308+0000 7f04cd6cf700 1 -- 172.21.15.189:0/1271494844 >> 172.21.15.189:0/1271494844 conn(0x7f04c806a730 msgr2=0x7f04c806f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:48.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.308+0000 7f04cd6cf700 1 -- 172.21.15.189:0/1271494844 shutdown_connections 2024-04-06T12:24:48.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.308+0000 7f04cd6cf700 1 -- 172.21.15.189:0/1271494844 wait complete. 2024-04-06T12:24:48.307 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 16 2024-04-06T12:24:48.319 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:24:48.362 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:48.363+0000 7f51465c0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:24:48.443 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:48.445+0000 7f50c1cdf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:24:48.443 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:48.445+0000 7f50c1cdf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:24:48.444 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:48.445+0000 7f50c1cdf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:24:48.444 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:48.445+0000 7f50c1cdf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:24:48.444 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:48.445+0000 7f50c1cdf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:24:48.444 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:48.446+0000 7f50bdcd7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:24:48.445 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:48.446+0000 7f50bdcd7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:24:48.445 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:48.446+0000 7f50bdcd7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:24:48.445 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:48.446+0000 7f50bdcd7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:24:48.445 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:24:48.446+0000 7f50bdcd7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:24:48.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.460+0000 7f538fd35700 1 Processor -- start 2024-04-06T12:24:48.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.460+0000 7f538fd35700 1 -- start start 2024-04-06T12:24:48.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.461+0000 7f538fd35700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53880d17a0 0x7f538806b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:48.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.461+0000 7f538fd35700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f53880cedf0 0x7f53880cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:48.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.461+0000 7f538fd35700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53880cf760 con 0x7f5388146dd0 2024-04-06T12:24:48.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.461+0000 7f538fd35700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f538806d850 con 0x7f53880cedf0 2024-04-06T12:24:48.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.461+0000 7f538fd35700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538806c0a0 con 0x7f53880d17a0 2024-04-06T12:24:48.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.461+0000 7f538d2d0700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5388146dd0 0x7f538806d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47164/0 (socket says 172.21.15.189:47164) 2024-04-06T12:24:48.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.461+0000 7f538d2d0700 1 -- 172.21.15.189:0/1922703753 learned_addr learned my addr 172.21.15.189:0/1922703753 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:48.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.461+0000 7f538dad1700 1 --2- 172.21.15.189:0/1922703753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53880d17a0 0x7f538806b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:48.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.461+0000 7f538e2d2700 1 --2- 172.21.15.189:0/1922703753 >> v2:172.21.15.192:3300/0 conn(0x7f53880cedf0 0x7f53880cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:48.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.461+0000 7f538cacf700 1 -- 172.21.15.189:0/1922703753 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744238011 0 0) 0x7f53880cf760 con 0x7f5388146dd0 2024-04-06T12:24:48.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.461+0000 7f538cacf700 1 -- 172.21.15.189:0/1922703753 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53740035b0 con 0x7f5388146dd0 2024-04-06T12:24:48.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.462+0000 7f538e2d2700 1 -- 172.21.15.189:0/1922703753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53880d17a0 msgr2=0x7f538806b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:48.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.462+0000 7f538e2d2700 1 --2- 172.21.15.189:0/1922703753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53880d17a0 0x7f538806b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.462+0000 7f538e2d2700 1 -- 172.21.15.189:0/1922703753 >> v1:172.21.15.189:6789/0 conn(0x7f5388146dd0 legacy=0x7f538806d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:48.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.462+0000 7f538cacf700 1 -- 172.21.15.189:0/1922703753 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3610491656 0 0) 0x7f53740035b0 con 0x7f5388146dd0 2024-04-06T12:24:48.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.462+0000 7f538e2d2700 1 -- 172.21.15.189:0/1922703753 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538806c380 con 0x7f53880cedf0 2024-04-06T12:24:48.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.462+0000 7f538dad1700 1 --2- 172.21.15.189:0/1922703753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53880d17a0 0x7f538806b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:48.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.462+0000 7f538e2d2700 1 --2- 172.21.15.189:0/1922703753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53880cedf0 0x7f53880cf220 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5380009d90 tx=0x7f538000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20dbd4b17d62f4d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:48.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.462+0000 7f538cacf700 1 -- 172.21.15.189:0/1922703753 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538000e070 con 0x7f53880cedf0 2024-04-06T12:24:48.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.462+0000 7f538cacf700 1 -- 172.21.15.189:0/1922703753 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5380003a20 con 0x7f53880cedf0 2024-04-06T12:24:48.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.462+0000 7f538cacf700 1 -- 172.21.15.189:0/1922703753 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5384001670 con 0x7f53880cedf0 2024-04-06T12:24:48.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.462+0000 7f538fd35700 1 -- 172.21.15.189:0/1922703753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53880cedf0 msgr2=0x7f53880cf220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:48.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.462+0000 7f538fd35700 1 --2- 172.21.15.189:0/1922703753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53880cedf0 0x7f53880cf220 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5380009d90 tx=0x7f538000b330 comp rx=0 tx=0).stop 2024-04-06T12:24:48.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.463+0000 7f538fd35700 1 -- 172.21.15.189:0/1922703753 shutdown_connections 2024-04-06T12:24:48.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.463+0000 7f538fd35700 1 --2- 172.21.15.189:0/1922703753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53880cedf0 0x7f53880cf220 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.463+0000 7f538fd35700 1 --2- 172.21.15.189:0/1922703753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53880d17a0 0x7f538806b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.463+0000 7f538fd35700 1 -- 172.21.15.189:0/1922703753 >> 172.21.15.189:0/1922703753 conn(0x7f53880b2db0 msgr2=0x7f53880b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:48.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.463+0000 7f538fd35700 1 -- 172.21.15.189:0/1922703753 shutdown_connections 2024-04-06T12:24:48.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.463+0000 7f538fd35700 1 -- 172.21.15.189:0/1922703753 wait complete. 2024-04-06T12:24:48.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.464+0000 7f538fd35700 1 Processor -- start 2024-04-06T12:24:48.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.464+0000 7f538fd35700 1 -- start start 2024-04-06T12:24:48.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538fd35700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53880cedf0 0x7f53880c5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:48.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538fd35700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53880d17a0 0x7f53880b5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:48.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538fd35700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5388146dd0 0x7f53880b6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:48.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538fd35700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53880c7040 con 0x7f5388146dd0 2024-04-06T12:24:48.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538fd35700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53880acb60 con 0x7f53880d17a0 2024-04-06T12:24:48.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538fd35700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53880c6430 con 0x7f53880cedf0 2024-04-06T12:24:48.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538e2d2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5388146dd0 0x7f53880b6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:48.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538e2d2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5388146dd0 0x7f53880b6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60702/0 (socket says 172.21.15.189:60702) 2024-04-06T12:24:48.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538e2d2700 1 -- 172.21.15.189:0/2657581137 learned_addr learned my addr 172.21.15.189:0/2657581137 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:48.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538dad1700 1 --2- 172.21.15.189:0/2657581137 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53880cedf0 0x7f53880c5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:48.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538d2d0700 1 --2- 172.21.15.189:0/2657581137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53880d17a0 0x7f53880b5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:48.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538e2d2700 1 -- 172.21.15.189:0/2657581137 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53880cedf0 msgr2=0x7f53880c5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:48.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538e2d2700 1 --2- 172.21.15.189:0/2657581137 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53880cedf0 0x7f53880c5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538e2d2700 1 -- 172.21.15.189:0/2657581137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53880d17a0 msgr2=0x7f53880b5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:48.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538e2d2700 1 --2- 172.21.15.189:0/2657581137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53880d17a0 0x7f53880b5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538e2d2700 1 -- 172.21.15.189:0/2657581137 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53780094a0 con 0x7f5388146dd0 2024-04-06T12:24:48.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538dad1700 1 --2- 172.21.15.189:0/2657581137 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53880cedf0 0x7f53880c5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:24:48.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538e2d2700 1 --2- 172.21.15.189:0/2657581137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5388146dd0 0x7f53880b6620 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f53880677e0 tx=0x7f5380004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:48.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f537e7fc700 1 -- 172.21.15.189:0/2657581137 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53800045f0 con 0x7f5388146dd0 2024-04-06T12:24:48.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f537e7fc700 1 -- 172.21.15.189:0/2657581137 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5380004750 con 0x7f5388146dd0 2024-04-06T12:24:48.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538fd35700 1 -- 172.21.15.189:0/2657581137 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5380009a70 con 0x7f5388146dd0 2024-04-06T12:24:48.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f537e7fc700 1 -- 172.21.15.189:0/2657581137 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5380009450 con 0x7f5388146dd0 2024-04-06T12:24:48.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.465+0000 7f538fd35700 1 -- 172.21.15.189:0/2657581137 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53880c69f0 con 0x7f5388146dd0 2024-04-06T12:24:48.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.466+0000 7f538fd35700 1 -- 172.21.15.189:0/2657581137 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5388074990 con 0x7f5388146dd0 2024-04-06T12:24:48.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.467+0000 7f537e7fc700 1 -- 172.21.15.189:0/2657581137 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f5380012450 con 0x7f5388146dd0 2024-04-06T12:24:48.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.467+0000 7f537e7fc700 1 -- 172.21.15.189:0/2657581137 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7141+0+0 (secure 0 0 0) 0x7f538008ad50 con 0x7f5388146dd0 2024-04-06T12:24:48.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.472+0000 7f537e7fc700 1 -- 172.21.15.189:0/2657581137 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5380055b90 con 0x7f5388146dd0 2024-04-06T12:24:48.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.667+0000 7f538fd35700 1 -- 172.21.15.189:0/2657581137 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53880d1e20 con 0x7f5388146dd0 2024-04-06T12:24:48.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.668+0000 7f537e7fc700 1 -- 172.21.15.189:0/2657581137 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+17974 (secure 0 0 0) 0x7f5380059380 con 0x7f5388146dd0 2024-04-06T12:24:48.667 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:48.667 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":54,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:24:48.379030+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:13:19.268280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-06T12:13:20.243771+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":53,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.35","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/3227363819":"2024-04-07T12:24:48.378992+0000","172.21.15.189:0/3742046684":"2024-04-07T12:24:48.378992+0000","172.21.15.189:0/2548985455":"2024-04-07T12:24:48.378992+0000","172.21.15.189:0/3219398994":"2024-04-07T12:24:48.378992+0000","172.21.15.189:6832/47128":"2024-04-07T12:13:34.836450+0000","172.21.15.189:0/1812383853":"2024-04-07T12:24:48.378992+0000","172.21.15.189:0/3015633422":"2024-04-07T12:13:34.836450+0000","172.21.15.189:0/934958820":"2024-04-07T12:24:48.378992+0000","172.21.15.189:6833/47128":"2024-04-07T12:13:34.836450+0000","172.21.15.189:0/1075995033":"2024-04-07T12:13:34.836450+0000","172.21.15.189:0/3092563682":"2024-04-07T12:13:34.836450+0000","172.21.15.189:0/1087803804":"2024-04-07T12:24:48.378992+0000","172.21.15.189:0/155214328":"2024-04-07T12:13:34.836450+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":1},{"begin":9,"length":1},{"begin":11,"length":1},{"begin":13,"length":2},{"begin":17,"length":2},{"begin":21,"length":2},{"begin":24,"length":2},{"begin":28,"length":2},{"begin":33,"length":1},{"begin":35,"length":2},{"begin":39,"length":2},{"begin":43,"length":1},{"begin":45,"length":2},{"begin":49,"length":3},{"begin":53,"length":1},{"begin":55,"length":2},{"begin":59,"length":4},{"begin":64,"length":1},{"begin":66,"length":2},{"begin":70,"length":3},{"begin":74,"length":1},{"begin":76,"length":2},{"begin":80,"length":5},{"begin":86,"length":1},{"begin":88,"length":2},{"begin":93,"length":3},{"begin":97,"length":1},{"begin":99,"length":2},{"begin":103,"length":3},{"begin":107,"length":1},{"begin":109,"length":2},{"begin":160,"length":62}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:24:48.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.670+0000 7f538fd35700 1 -- 172.21.15.189:0/2657581137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5388146dd0 msgr2=0x7f53880b6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:48.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.670+0000 7f538fd35700 1 --2- 172.21.15.189:0/2657581137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5388146dd0 0x7f53880b6620 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f53880677e0 tx=0x7f5380004ad0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.671+0000 7f538fd35700 1 -- 172.21.15.189:0/2657581137 shutdown_connections 2024-04-06T12:24:48.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.671+0000 7f538fd35700 1 --2- 172.21.15.189:0/2657581137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5388146dd0 0x7f53880b6620 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.671+0000 7f538fd35700 1 --2- 172.21.15.189:0/2657581137 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53880cedf0 0x7f53880c5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.671+0000 7f538fd35700 1 --2- 172.21.15.189:0/2657581137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53880d17a0 0x7f53880b5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.671+0000 7f538fd35700 1 -- 172.21.15.189:0/2657581137 >> 172.21.15.189:0/2657581137 conn(0x7f53880b2db0 msgr2=0x7f5388160fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:48.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.671+0000 7f538fd35700 1 -- 172.21.15.189:0/2657581137 shutdown_connections 2024-04-06T12:24:48.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.671+0000 7f538fd35700 1 -- 172.21.15.189:0/2657581137 wait complete. 2024-04-06T12:24:48.681 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-06T12:24:48.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.849+0000 7f4797acc700 1 Processor -- start 2024-04-06T12:24:48.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.849+0000 7f4797acc700 1 -- start start 2024-04-06T12:24:48.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.850+0000 7f4797acc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47900d11b0 0x7f47900d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:48.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.850+0000 7f4797acc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f47900cfa00 0x7f47900cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:48.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.850+0000 7f4797acc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47900d1ad0 con 0x7f47900cb870 2024-04-06T12:24:48.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.850+0000 7f4797acc700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47900ccdc0 con 0x7f47900cfa00 2024-04-06T12:24:48.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.850+0000 7f4797acc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47900d0b50 con 0x7f47900d11b0 2024-04-06T12:24:48.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.850+0000 7f4795067700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f47900cb870 0x7f479015c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47180/0 (socket says 172.21.15.189:47180) 2024-04-06T12:24:48.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.850+0000 7f4795067700 1 -- 172.21.15.189:0/2453187556 learned_addr learned my addr 172.21.15.189:0/2453187556 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:48.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.850+0000 7f4796069700 1 --2- 172.21.15.189:0/2453187556 >> v2:172.21.15.192:3300/0 conn(0x7f47900cfa00 0x7f47900cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:48.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.850+0000 7f4795868700 1 --2- 172.21.15.189:0/2453187556 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47900d11b0 0x7f47900d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:48.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.850+0000 7f4794866700 1 -- 172.21.15.189:0/2453187556 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422234818 0 0) 0x7f47900d1ad0 con 0x7f47900cb870 2024-04-06T12:24:48.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.850+0000 7f4794866700 1 -- 172.21.15.189:0/2453187556 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f477c0035b0 con 0x7f47900cb870 2024-04-06T12:24:48.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.850+0000 7f4796069700 1 -- 172.21.15.189:0/2453187556 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47900d11b0 msgr2=0x7f47900d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:48.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.850+0000 7f4796069700 1 --2- 172.21.15.189:0/2453187556 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47900d11b0 0x7f47900d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.850+0000 7f4794866700 1 -- 172.21.15.189:0/2453187556 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 296389534 0 0) 0x7f477c0035b0 con 0x7f47900cb870 2024-04-06T12:24:48.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.850+0000 7f4796069700 1 -- 172.21.15.189:0/2453187556 >> v1:172.21.15.189:6789/0 conn(0x7f47900cb870 legacy=0x7f479015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:48.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.850+0000 7f4796069700 1 -- 172.21.15.189:0/2453187556 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47900cbd80 con 0x7f47900cfa00 2024-04-06T12:24:48.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.850+0000 7f4795868700 1 --2- 172.21.15.189:0/2453187556 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47900d11b0 0x7f47900d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:48.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.851+0000 7f4796069700 1 --2- 172.21.15.189:0/2453187556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47900cfa00 0x7f47900cd560 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4788009d90 tx=0x7f478800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d06c54752e8baf19 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:48.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.851+0000 7f4794866700 1 -- 172.21.15.189:0/2453187556 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478800e070 con 0x7f47900cfa00 2024-04-06T12:24:48.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.851+0000 7f4794866700 1 -- 172.21.15.189:0/2453187556 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4788003a20 con 0x7f47900cfa00 2024-04-06T12:24:48.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.851+0000 7f4794866700 1 -- 172.21.15.189:0/2453187556 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4780001670 con 0x7f47900cfa00 2024-04-06T12:24:48.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.851+0000 7f4797acc700 1 -- 172.21.15.189:0/2453187556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47900cfa00 msgr2=0x7f47900cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:48.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.851+0000 7f4797acc700 1 --2- 172.21.15.189:0/2453187556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47900cfa00 0x7f47900cd560 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4788009d90 tx=0x7f478800b330 comp rx=0 tx=0).stop 2024-04-06T12:24:48.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.851+0000 7f4797acc700 1 -- 172.21.15.189:0/2453187556 shutdown_connections 2024-04-06T12:24:48.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.851+0000 7f4797acc700 1 --2- 172.21.15.189:0/2453187556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47900cfa00 0x7f47900cd560 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.851+0000 7f4797acc700 1 --2- 172.21.15.189:0/2453187556 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47900d11b0 0x7f47900d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.851+0000 7f4797acc700 1 -- 172.21.15.189:0/2453187556 >> 172.21.15.189:0/2453187556 conn(0x7f47900b2db0 msgr2=0x7f47900b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:48.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.851+0000 7f4797acc700 1 -- 172.21.15.189:0/2453187556 shutdown_connections 2024-04-06T12:24:48.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.851+0000 7f4797acc700 1 -- 172.21.15.189:0/2453187556 wait complete. 2024-04-06T12:24:48.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.852+0000 7f4797acc700 1 Processor -- start 2024-04-06T12:24:48.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.853+0000 7f4797acc700 1 -- start start 2024-04-06T12:24:48.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.853+0000 7f4797acc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47900cb870 0x7f4790152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:48.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.853+0000 7f4797acc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47900cfa00 0x7f47900b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:48.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.853+0000 7f4797acc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47900d11b0 0x7f47900b5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:48.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.853+0000 7f4797acc700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4790153030 con 0x7f47900cfa00 2024-04-06T12:24:48.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.853+0000 7f4797acc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479006c8e0 con 0x7f47900cb870 2024-04-06T12:24:48.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.853+0000 7f4797acc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f479006b840 con 0x7f47900d11b0 2024-04-06T12:24:48.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.853+0000 7f4795067700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47900cfa00 0x7f47900b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:48.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.853+0000 7f4795067700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47900cfa00 0x7f47900b4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60704/0 (socket says 172.21.15.189:60704) 2024-04-06T12:24:48.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.853+0000 7f4795067700 1 -- 172.21.15.189:0/2983735582 learned_addr learned my addr 172.21.15.189:0/2983735582 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:48.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.853+0000 7f4795868700 1 --2- 172.21.15.189:0/2983735582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47900cb870 0x7f4790152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:48.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.853+0000 7f4796069700 1 --2- 172.21.15.189:0/2983735582 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47900d11b0 0x7f47900b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:48.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.854+0000 7f4795067700 1 -- 172.21.15.189:0/2983735582 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47900d11b0 msgr2=0x7f47900b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:48.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.854+0000 7f4795067700 1 --2- 172.21.15.189:0/2983735582 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47900d11b0 0x7f47900b5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.854+0000 7f4795067700 1 -- 172.21.15.189:0/2983735582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47900cb870 msgr2=0x7f4790152af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:48.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.854+0000 7f4795067700 1 --2- 172.21.15.189:0/2983735582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47900cb870 0x7f4790152af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:48.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.854+0000 7f4795067700 1 -- 172.21.15.189:0/2983735582 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478c0094a0 con 0x7f47900cfa00 2024-04-06T12:24:48.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.854+0000 7f4795067700 1 --2- 172.21.15.189:0/2983735582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47900cfa00 0x7f47900b4ef0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f478000c750 tx=0x7f478000ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.854+0000 7f47867fc700 1 -- 172.21.15.189:0/2983735582 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4780004c80 con 0x7f47900cfa00 2024-04-06T12:24:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.854+0000 7f47867fc700 1 -- 172.21.15.189:0/2983735582 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4780004de0 con 0x7f47900cfa00 2024-04-06T12:24:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.854+0000 7f47867fc700 1 -- 172.21.15.189:0/2983735582 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4780007310 con 0x7f47900cfa00 2024-04-06T12:24:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.854+0000 7f4797acc700 1 -- 172.21.15.189:0/2983735582 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4788009a70 con 0x7f47900cfa00 2024-04-06T12:24:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.854+0000 7f4797acc700 1 -- 172.21.15.189:0/2983735582 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f479006be00 con 0x7f47900cfa00 2024-04-06T12:24:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.855+0000 7f4797acc700 1 -- 172.21.15.189:0/2983735582 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4790074990 con 0x7f47900cfa00 2024-04-06T12:24:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.855+0000 7f47867fc700 1 -- 172.21.15.189:0/2983735582 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f4780007470 con 0x7f47900cfa00 2024-04-06T12:24:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.856+0000 7f47867fc700 1 -- 172.21.15.189:0/2983735582 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7141+0+0 (secure 0 0 0) 0x7f4780089070 con 0x7f47900cfa00 2024-04-06T12:24:48.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:48.860+0000 7f47867fc700 1 -- 172.21.15.189:0/2983735582 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4780054100 con 0x7f47900cfa00 2024-04-06T12:24:49.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.099+0000 7f4797acc700 1 -- 172.21.15.189:0/2983735582 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f47900cb630 con 0x7f47900cfa00 2024-04-06T12:24:49.097 INFO:tasks.ceph.mon.a.smithi189.stderr:2024-04-06T12:24:49.100+0000 7fa9fe1ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T12:24:49.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.111+0000 7f47867fc700 1 -- 172.21.15.189:0/2983735582 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4780057720 con 0x7f47900cfa00 2024-04-06T12:24:49.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.114+0000 7f4797acc700 1 -- 172.21.15.189:0/2983735582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47900cfa00 msgr2=0x7f47900b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:49.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.114+0000 7f4797acc700 1 --2- 172.21.15.189:0/2983735582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47900cfa00 0x7f47900b4ef0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f478000c750 tx=0x7f478000ca60 comp rx=0 tx=0).stop 2024-04-06T12:24:49.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.114+0000 7f4797acc700 1 -- 172.21.15.189:0/2983735582 shutdown_connections 2024-04-06T12:24:49.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.114+0000 7f4797acc700 1 --2- 172.21.15.189:0/2983735582 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47900d11b0 0x7f47900b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:49.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.114+0000 7f4797acc700 1 --2- 172.21.15.189:0/2983735582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47900cb870 0x7f4790152af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:49.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.114+0000 7f4797acc700 1 --2- 172.21.15.189:0/2983735582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47900cfa00 0x7f47900b4ef0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:49.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.114+0000 7f4797acc700 1 -- 172.21.15.189:0/2983735582 >> 172.21.15.189:0/2983735582 conn(0x7f47900b2db0 msgr2=0x7f47900cd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:49.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.114+0000 7f4797acc700 1 -- 172.21.15.189:0/2983735582 shutdown_connections 2024-04-06T12:24:49.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.114+0000 7f4797acc700 1 -- 172.21.15.189:0/2983735582 wait complete. 2024-04-06T12:24:49.113 INFO:teuthology.orchestra.run.smithi189.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-06T12:24:49.125 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-06T12:24:49.147 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T12:24:49.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.252+0000 7f024bfff700 1 Processor -- start 2024-04-06T12:24:49.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.252+0000 7f024bfff700 1 -- start start 2024-04-06T12:24:49.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.252+0000 7f024bfff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024c06f7b0 0x7f024c0b3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:49.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.252+0000 7f024bfff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f024c071d20 0x7f024c15ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:49.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.252+0000 7f024bfff700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c15b220 con 0x7f024c15a6a0 2024-04-06T12:24:49.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.253+0000 7f024bfff700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f024c06fb90 con 0x7f024c071d20 2024-04-06T12:24:49.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.253+0000 7f024bfff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f024c0b2f50 con 0x7f024c06f7b0 2024-04-06T12:24:49.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.253+0000 7f024b7fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f024c071d20 0x7f024c15ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:49.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.253+0000 7f024affd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024c06f7b0 0x7f024c0b3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:49.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.253+0000 7f024b7fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f024c071d20 0x7f024c15ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:57972/0 (socket says 172.21.15.189:57972) 2024-04-06T12:24:49.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.253+0000 7f024affd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024c06f7b0 0x7f024c0b3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:55588/0 (socket says 172.21.15.189:55588) 2024-04-06T12:24:49.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.253+0000 7f024b7fe700 1 -- 172.21.15.189:0/2094592623 learned_addr learned my addr 172.21.15.189:0/2094592623 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:49.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.253+0000 7f024b7fe700 1 -- 172.21.15.189:0/2094592623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024c06f7b0 msgr2=0x7f024c0b3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:49.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.253+0000 7f024b7fe700 1 --2- 172.21.15.189:0/2094592623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024c06f7b0 0x7f024c0b3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:49.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.253+0000 7f024b7fe700 1 -- 172.21.15.189:0/2094592623 >> v1:172.21.15.192:6789/0 conn(0x7f024c15a6a0 legacy=0x7f024c071610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:49.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.253+0000 7f024b7fe700 1 -- 172.21.15.189:0/2094592623 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024c15b500 con 0x7f024c071d20 2024-04-06T12:24:49.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.253+0000 7f024b7fe700 1 --2- 172.21.15.189:0/2094592623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f024c071d20 0x7f024c15ab20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0244009e50 tx=0x7f024400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61c86168cdc64e40 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:49.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.253+0000 7f0249ffb700 1 -- 172.21.15.189:0/2094592623 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024400e070 con 0x7f024c071d20 2024-04-06T12:24:49.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.253+0000 7f0249ffb700 1 -- 172.21.15.189:0/2094592623 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f024400b910 con 0x7f024c071d20 2024-04-06T12:24:49.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.254+0000 7f0249ffb700 1 -- 172.21.15.189:0/2094592623 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024400bcc0 con 0x7f024c071d20 2024-04-06T12:24:49.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.254+0000 7f024bfff700 1 -- 172.21.15.189:0/2094592623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f024c071d20 msgr2=0x7f024c15ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:49.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.254+0000 7f024bfff700 1 --2- 172.21.15.189:0/2094592623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f024c071d20 0x7f024c15ab20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0244009e50 tx=0x7f024400b330 comp rx=0 tx=0).stop 2024-04-06T12:24:49.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.254+0000 7f024bfff700 1 -- 172.21.15.189:0/2094592623 shutdown_connections 2024-04-06T12:24:49.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.254+0000 7f024bfff700 1 --2- 172.21.15.189:0/2094592623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f024c071d20 0x7f024c15ab20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:49.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.254+0000 7f024bfff700 1 --2- 172.21.15.189:0/2094592623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024c06f7b0 0x7f024c0b3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:49.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.254+0000 7f024bfff700 1 -- 172.21.15.189:0/2094592623 >> 172.21.15.189:0/2094592623 conn(0x7f024c0cff80 msgr2=0x7f024c0d0360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:49.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.254+0000 7f024bfff700 1 -- 172.21.15.189:0/2094592623 shutdown_connections 2024-04-06T12:24:49.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.254+0000 7f024bfff700 1 -- 172.21.15.189:0/2094592623 wait complete. 2024-04-06T12:24:49.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.256+0000 7f024bfff700 1 Processor -- start 2024-04-06T12:24:49.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.256+0000 7f024bfff700 1 -- start start 2024-04-06T12:24:49.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.256+0000 7f024bfff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f024c06f7b0 0x7f024c153690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:49.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.256+0000 7f024bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024c071d20 0x7f024c158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:49.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.256+0000 7f024bfff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024c15a6a0 0x7f024c0cb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:49.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.256+0000 7f024bfff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024c156f80 con 0x7f024c071d20 2024-04-06T12:24:49.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.256+0000 7f024bfff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024c1effa0 con 0x7f024c06f7b0 2024-04-06T12:24:49.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.256+0000 7f024bfff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f024c1f0280 con 0x7f024c15a6a0 2024-04-06T12:24:49.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.256+0000 7f024a7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024c071d20 0x7f024c158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:49.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.256+0000 7f024a7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024c071d20 0x7f024c158a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60714/0 (socket says 172.21.15.189:60714) 2024-04-06T12:24:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.256+0000 7f024a7fc700 1 -- 172.21.15.189:0/3567652269 learned_addr learned my addr 172.21.15.189:0/3567652269 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.257+0000 7f024affd700 1 --2- 172.21.15.189:0/3567652269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f024c06f7b0 0x7f024c153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.257+0000 7f024b7fe700 1 --2- 172.21.15.189:0/3567652269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024c15a6a0 0x7f024c0cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.257+0000 7f024a7fc700 1 -- 172.21.15.189:0/3567652269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024c15a6a0 msgr2=0x7f024c0cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.257+0000 7f024a7fc700 1 --2- 172.21.15.189:0/3567652269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024c15a6a0 0x7f024c0cb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.257+0000 7f024a7fc700 1 -- 172.21.15.189:0/3567652269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f024c06f7b0 msgr2=0x7f024c153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.257+0000 7f024a7fc700 1 --2- 172.21.15.189:0/3567652269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f024c06f7b0 0x7f024c153690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.257+0000 7f024a7fc700 1 -- 172.21.15.189:0/3567652269 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0244009b30 con 0x7f024c071d20 2024-04-06T12:24:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.257+0000 7f024a7fc700 1 --2- 172.21.15.189:0/3567652269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024c071d20 0x7f024c158a10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f024000c450 tx=0x7f024000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.257+0000 7f02337fe700 1 -- 172.21.15.189:0/3567652269 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02400052d0 con 0x7f024c071d20 2024-04-06T12:24:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.257+0000 7f02337fe700 1 -- 172.21.15.189:0/3567652269 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0240005430 con 0x7f024c071d20 2024-04-06T12:24:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.257+0000 7f024bfff700 1 -- 172.21.15.189:0/3567652269 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f024c1f0560 con 0x7f024c071d20 2024-04-06T12:24:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.257+0000 7f02337fe700 1 -- 172.21.15.189:0/3567652269 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02400038c0 con 0x7f024c071d20 2024-04-06T12:24:49.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.257+0000 7f024bfff700 1 -- 172.21.15.189:0/3567652269 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f024c158f50 con 0x7f024c071d20 2024-04-06T12:24:49.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.258+0000 7f024bfff700 1 -- 172.21.15.189:0/3567652269 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f024c1590e0 con 0x7f024c071d20 2024-04-06T12:24:49.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.258+0000 7f02337fe700 1 -- 172.21.15.189:0/3567652269 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f0240003a20 con 0x7f024c071d20 2024-04-06T12:24:49.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.259+0000 7f02337fe700 1 -- 172.21.15.189:0/3567652269 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7227+0+0 (secure 0 0 0) 0x7f0240088740 con 0x7f024c071d20 2024-04-06T12:24:49.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.263+0000 7f02337fe700 1 -- 172.21.15.189:0/3567652269 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f024c1590e0 con 0x7f024c071d20 2024-04-06T12:24:49.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.387+0000 7f02337fe700 1 -- 172.21.15.189:0/3567652269 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 17) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f02400531b0 con 0x7f024c071d20 2024-04-06T12:24:49.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.388+0000 7f02337fe700 1 --2- 172.21.15.189:0/3567652269 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0238036150 0x7f0238072e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:49.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.390+0000 7f024affd700 1 --2- 172.21.15.189:0/3567652269 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0238036150 0x7f0238072e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:49.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.391+0000 7f024affd700 1 --2- 172.21.15.189:0/3567652269 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0238036150 0x7f0238072e20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f023c009be0 tx=0x7f023c009210 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:49.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:49.512+0000 7f024bfff700 1 -- 172.21.15.189:0/3567652269 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f024c1590e0 con 0x7f024c071d20 2024-04-06T12:24:50.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.113+0000 7f02337fe700 1 -- 172.21.15.189:0/3567652269 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) v1 ==== 104+0+0 (secure 0 0 0) 0x7f024c1590e0 con 0x7f024c071d20 2024-04-06T12:24:50.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.116+0000 7f024bfff700 1 -- 172.21.15.189:0/3567652269 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0238036150 msgr2=0x7f0238072e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:50.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.116+0000 7f024bfff700 1 --2- 172.21.15.189:0/3567652269 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0238036150 0x7f0238072e20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f023c009be0 tx=0x7f023c009210 comp rx=0 tx=0).stop 2024-04-06T12:24:50.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.116+0000 7f024bfff700 1 -- 172.21.15.189:0/3567652269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024c071d20 msgr2=0x7f024c158a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:50.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.116+0000 7f024bfff700 1 --2- 172.21.15.189:0/3567652269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024c071d20 0x7f024c158a10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f024000c450 tx=0x7f024000c760 comp rx=0 tx=0).stop 2024-04-06T12:24:50.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.116+0000 7f024bfff700 1 -- 172.21.15.189:0/3567652269 shutdown_connections 2024-04-06T12:24:50.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.116+0000 7f024bfff700 1 --2- 172.21.15.189:0/3567652269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f024c15a6a0 0x7f024c0cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:50.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.116+0000 7f024bfff700 1 --2- 172.21.15.189:0/3567652269 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0238036150 0x7f0238072e20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:50.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.116+0000 7f024bfff700 1 --2- 172.21.15.189:0/3567652269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f024c06f7b0 0x7f024c153690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:50.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.116+0000 7f024bfff700 1 --2- 172.21.15.189:0/3567652269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024c071d20 0x7f024c158a10 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:50.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.116+0000 7f024bfff700 1 -- 172.21.15.189:0/3567652269 >> 172.21.15.189:0/3567652269 conn(0x7f024c0cff80 msgr2=0x7f024c0cd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:50.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.116+0000 7f024bfff700 1 -- 172.21.15.189:0/3567652269 shutdown_connections 2024-04-06T12:24:50.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.116+0000 7f024bfff700 1 -- 172.21.15.189:0/3567652269 wait complete. 2024-04-06T12:24:50.130 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-06T12:24:50.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.303+0000 7f4fa5d04700 1 Processor -- start 2024-04-06T12:24:50.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.303+0000 7f4fa5d04700 1 -- start start 2024-04-06T12:24:50.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.304+0000 7f4fa5d04700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4fa015a550 0x7f4fa00cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:50.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.304+0000 7f4fa5d04700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4fa00cf900 0x7f4fa00b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:50.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.304+0000 7f4fa5d04700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa00cdbe0 con 0x7f4fa00b4ba0 2024-04-06T12:24:50.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.304+0000 7f4fa5d04700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fa015a940 con 0x7f4fa015a550 2024-04-06T12:24:50.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.304+0000 7f4fa5d04700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fa015ac00 con 0x7f4fa00cf900 2024-04-06T12:24:50.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.305+0000 7f4f9f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4fa015a550 0x7f4fa00cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:50.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.305+0000 7f4f9ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4fa00b4ba0 0x7f4fa006eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49376/0 (socket says 172.21.15.189:49376) 2024-04-06T12:24:50.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.305+0000 7f4f9ffff700 1 -- 172.21.15.189:0/971570152 learned_addr learned my addr 172.21.15.189:0/971570152 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:50.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.305+0000 7f4f9effd700 1 --2- 172.21.15.189:0/971570152 >> v2:172.21.15.192:3300/0 conn(0x7f4fa00cf900 0x7f4fa00b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:50.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.305+0000 7f4f9f7fe700 1 -- 172.21.15.189:0/971570152 >> v2:172.21.15.192:3300/0 conn(0x7f4fa00cf900 msgr2=0x7f4fa00b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:50.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.305+0000 7f4f9f7fe700 1 --2- 172.21.15.189:0/971570152 >> v2:172.21.15.192:3300/0 conn(0x7f4fa00cf900 0x7f4fa00b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:50.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.305+0000 7f4f9f7fe700 1 -- 172.21.15.189:0/971570152 >> v1:172.21.15.189:6789/0 conn(0x7f4fa00b4ba0 legacy=0x7f4fa006eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:50.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.305+0000 7f4f9e7fc700 1 -- 172.21.15.189:0/971570152 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232264109 0 0) 0x7f4fa00cdbe0 con 0x7f4fa00b4ba0 2024-04-06T12:24:50.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.305+0000 7f4f9f7fe700 1 -- 172.21.15.189:0/971570152 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa015aee0 con 0x7f4fa015a550 2024-04-06T12:24:50.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.305+0000 7f4f9effd700 1 --2- 172.21.15.189:0/971570152 >> v2:172.21.15.192:3300/0 conn(0x7f4fa00cf900 0x7f4fa00b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:24:50.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.305+0000 7f4f9f7fe700 1 --2- 172.21.15.189:0/971570152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa015a550 0x7f4fa00cf3c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f4fa00b2d40 tx=0x7f4f88009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1324c38f58efd05 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:50.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.305+0000 7f4f9e7fc700 1 -- 172.21.15.189:0/971570152 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f88011830 con 0x7f4fa015a550 2024-04-06T12:24:50.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.305+0000 7f4f9e7fc700 1 -- 172.21.15.189:0/971570152 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f88011990 con 0x7f4fa015a550 2024-04-06T12:24:50.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.305+0000 7f4f9e7fc700 1 -- 172.21.15.189:0/971570152 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f88011d20 con 0x7f4fa015a550 2024-04-06T12:24:50.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.306+0000 7f4fa5d04700 1 -- 172.21.15.189:0/971570152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa015a550 msgr2=0x7f4fa00cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:50.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.306+0000 7f4fa5d04700 1 --2- 172.21.15.189:0/971570152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa015a550 0x7f4fa00cf3c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f4fa00b2d40 tx=0x7f4f88009b20 comp rx=0 tx=0).stop 2024-04-06T12:24:50.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.306+0000 7f4fa5d04700 1 -- 172.21.15.189:0/971570152 shutdown_connections 2024-04-06T12:24:50.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.306+0000 7f4fa5d04700 1 --2- 172.21.15.189:0/971570152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa015a550 0x7f4fa00cf3c0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:50.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.306+0000 7f4fa5d04700 1 --2- 172.21.15.189:0/971570152 >> v2:172.21.15.192:3300/0 conn(0x7f4fa00cf900 0x7f4fa00b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:50.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.306+0000 7f4fa5d04700 1 -- 172.21.15.189:0/971570152 >> 172.21.15.189:0/971570152 conn(0x7f4fa00d00b0 msgr2=0x7f4fa00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:50.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.306+0000 7f4fa5d04700 1 -- 172.21.15.189:0/971570152 shutdown_connections 2024-04-06T12:24:50.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.306+0000 7f4fa5d04700 1 -- 172.21.15.189:0/971570152 wait complete. 2024-04-06T12:24:50.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.307+0000 7f4fa5d04700 1 Processor -- start 2024-04-06T12:24:50.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.307+0000 7f4fa5d04700 1 -- start start 2024-04-06T12:24:50.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.308+0000 7f4fa5d04700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fa00b4ba0 0x7f4fa014a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:50.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.308+0000 7f4fa5d04700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fa00cf900 0x7f4fa00cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:50.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.308+0000 7f4fa5d04700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa0149b00 0x7f4fa0149f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:50.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.308+0000 7f4fa5d04700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa0148260 con 0x7f4fa0149b00 2024-04-06T12:24:50.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.308+0000 7f4fa5d04700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa014a4d0 con 0x7f4fa00b4ba0 2024-04-06T12:24:50.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.308+0000 7f4fa5d04700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa00aaf90 con 0x7f4fa00cf900 2024-04-06T12:24:50.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.308+0000 7f4f9ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa0149b00 0x7f4fa0149f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:50.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.308+0000 7f4f9ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa0149b00 0x7f4fa0149f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42680/0 (socket says 172.21.15.189:42680) 2024-04-06T12:24:50.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.308+0000 7f4f9ffff700 1 -- 172.21.15.189:0/1272243566 learned_addr learned my addr 172.21.15.189:0/1272243566 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:50.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.308+0000 7f4f9f7fe700 1 --2- 172.21.15.189:0/1272243566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fa00b4ba0 0x7f4fa014a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:50.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.308+0000 7f4f9effd700 1 --2- 172.21.15.189:0/1272243566 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fa00cf900 0x7f4fa00cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:50.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.308+0000 7f4f9ffff700 1 -- 172.21.15.189:0/1272243566 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fa00cf900 msgr2=0x7f4fa00cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:50.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.308+0000 7f4f9ffff700 1 --2- 172.21.15.189:0/1272243566 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fa00cf900 0x7f4fa00cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:50.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.308+0000 7f4f9ffff700 1 -- 172.21.15.189:0/1272243566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fa00b4ba0 msgr2=0x7f4fa014a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:50.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.308+0000 7f4f9ffff700 1 --2- 172.21.15.189:0/1272243566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fa00b4ba0 0x7f4fa014a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:50.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.308+0000 7f4f9ffff700 1 -- 172.21.15.189:0/1272243566 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f880094a0 con 0x7f4fa0149b00 2024-04-06T12:24:50.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.308+0000 7f4f9effd700 1 --2- 172.21.15.189:0/1272243566 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fa00cf900 0x7f4fa00cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:24:50.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.308+0000 7f4f9ffff700 1 --2- 172.21.15.189:0/1272243566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa0149b00 0x7f4fa0149f90 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f4f98002230 tx=0x7f4f9800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:50.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.309+0000 7f4f97fff700 1 -- 172.21.15.189:0/1272243566 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9800ab70 con 0x7f4fa0149b00 2024-04-06T12:24:50.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.309+0000 7f4f97fff700 1 -- 172.21.15.189:0/1272243566 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9800acf0 con 0x7f4fa0149b00 2024-04-06T12:24:50.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.309+0000 7f4fa5d04700 1 -- 172.21.15.189:0/1272243566 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa00aacb0 con 0x7f4fa0149b00 2024-04-06T12:24:50.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.309+0000 7f4f97fff700 1 -- 172.21.15.189:0/1272243566 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f98007310 con 0x7f4fa0149b00 2024-04-06T12:24:50.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.309+0000 7f4fa5d04700 1 -- 172.21.15.189:0/1272243566 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa0147e80 con 0x7f4fa0149b00 2024-04-06T12:24:50.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.310+0000 7f4fa5d04700 1 -- 172.21.15.189:0/1272243566 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa01483a0 con 0x7f4fa0149b00 2024-04-06T12:24:50.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.310+0000 7f4f97fff700 1 -- 172.21.15.189:0/1272243566 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f4f98007470 con 0x7f4fa0149b00 2024-04-06T12:24:50.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.310+0000 7f4f97fff700 1 --2- 172.21.15.189:0/1272243566 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f8006aae0 0x7f4f8006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:50.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.311+0000 7f4f97fff700 1 -- 172.21.15.189:0/1272243566 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7227+0+0 (secure 0 0 0) 0x7f4f980890a0 con 0x7f4fa0149b00 2024-04-06T12:24:50.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.311+0000 7f4f9f7fe700 1 --2- 172.21.15.189:0/1272243566 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f8006aae0 0x7f4f8006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:50.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.311+0000 7f4f9f7fe700 1 --2- 172.21.15.189:0/1272243566 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f8006aae0 0x7f4f8006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4f880090b0 tx=0x7f4f8801c040 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:50.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.315+0000 7f4f97fff700 1 -- 172.21.15.189:0/1272243566 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4fa01483a0 con 0x7f4fa0149b00 2024-04-06T12:24:50.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:50.529+0000 7f4fa5d04700 1 -- 172.21.15.189:0/1272243566 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4fa01483a0 con 0x7f4fa0149b00 2024-04-06T12:24:51.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.399+0000 7f4f97fff700 1 -- 172.21.15.189:0/1272243566 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 18) v1 ==== 89191+0+0 (secure 0 0 0) 0x7f4f98054320 con 0x7f4fa0149b00 2024-04-06T12:24:51.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.401+0000 7f4f97fff700 1 -- 172.21.15.189:0/1272243566 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v56) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4f98053f50 con 0x7f4fa0149b00 2024-04-06T12:24:51.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.404+0000 7f4fa5d04700 1 -- 172.21.15.189:0/1272243566 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f8006aae0 msgr2=0x7f4f8006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:51.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.404+0000 7f4fa5d04700 1 --2- 172.21.15.189:0/1272243566 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f8006aae0 0x7f4f8006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4f880090b0 tx=0x7f4f8801c040 comp rx=0 tx=0).stop 2024-04-06T12:24:51.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.404+0000 7f4fa5d04700 1 -- 172.21.15.189:0/1272243566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa0149b00 msgr2=0x7f4fa0149f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:51.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.404+0000 7f4fa5d04700 1 --2- 172.21.15.189:0/1272243566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa0149b00 0x7f4fa0149f90 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f4f98002230 tx=0x7f4f9800cf50 comp rx=0 tx=0).stop 2024-04-06T12:24:51.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.404+0000 7f4fa5d04700 1 -- 172.21.15.189:0/1272243566 shutdown_connections 2024-04-06T12:24:51.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.404+0000 7f4fa5d04700 1 --2- 172.21.15.189:0/1272243566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fa0149b00 0x7f4fa0149f90 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:51.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.404+0000 7f4fa5d04700 1 --2- 172.21.15.189:0/1272243566 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f8006aae0 0x7f4f8006cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:51.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.404+0000 7f4fa5d04700 1 --2- 172.21.15.189:0/1272243566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fa00b4ba0 0x7f4fa014a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:51.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.404+0000 7f4fa5d04700 1 --2- 172.21.15.189:0/1272243566 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fa00cf900 0x7f4fa00cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:51.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.404+0000 7f4fa5d04700 1 -- 172.21.15.189:0/1272243566 >> 172.21.15.189:0/1272243566 conn(0x7f4fa00d00b0 msgr2=0x7f4fa00b2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:51.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.405+0000 7f4fa5d04700 1 -- 172.21.15.189:0/1272243566 shutdown_connections 2024-04-06T12:24:51.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.405+0000 7f4fa5d04700 1 -- 172.21.15.189:0/1272243566 wait complete. 2024-04-06T12:24:51.404 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' removed 2024-04-06T12:24:51.419 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-06T12:24:51.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.559+0000 7fe780dac700 1 Processor -- start 2024-04-06T12:24:51.558 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.559+0000 7fe780dac700 1 -- start start 2024-04-06T12:24:51.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.560+0000 7fe780dac700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe77c15d7b0 0x7fe77c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:51.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.560+0000 7fe780dac700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe77c06a730 0x7fe77c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:51.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.560+0000 7fe780dac700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe77c15ddf0 0x7fe77c15e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:51.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.560+0000 7fe780dac700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe77c06b0f0 con 0x7fe77c15ddf0 2024-04-06T12:24:51.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.560+0000 7fe780dac700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe77c0b3990 con 0x7fe77c15d7b0 2024-04-06T12:24:51.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.560+0000 7fe780dac700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe77c0b3060 con 0x7fe77c06a730 2024-04-06T12:24:51.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.560+0000 7fe77ad9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe77c15ddf0 0x7fe77c15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:51.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.560+0000 7fe77ad9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe77c15ddf0 0x7fe77c15e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42718/0 (socket says 172.21.15.189:42718) 2024-04-06T12:24:51.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.560+0000 7fe77ad9d700 1 -- 172.21.15.189:0/683385520 learned_addr learned my addr 172.21.15.189:0/683385520 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:51.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.560+0000 7fe77a59c700 1 --2- 172.21.15.189:0/683385520 >> v2:172.21.15.192:3300/0 conn(0x7fe77c15d7b0 0x7fe77c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:51.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.560+0000 7fe779d9b700 1 --2- 172.21.15.189:0/683385520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe77c06a730 0x7fe77c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:51.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.560+0000 7fe77ad9d700 1 -- 172.21.15.189:0/683385520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe77c06a730 msgr2=0x7fe77c06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:51.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.560+0000 7fe77ad9d700 1 --2- 172.21.15.189:0/683385520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe77c06a730 0x7fe77c06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:51.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.560+0000 7fe77ad9d700 1 -- 172.21.15.189:0/683385520 >> v2:172.21.15.192:3300/0 conn(0x7fe77c15d7b0 msgr2=0x7fe77c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:51.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.560+0000 7fe77ad9d700 1 --2- 172.21.15.189:0/683385520 >> v2:172.21.15.192:3300/0 conn(0x7fe77c15d7b0 0x7fe77c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:51.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.560+0000 7fe77ad9d700 1 -- 172.21.15.189:0/683385520 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe77c0b3340 con 0x7fe77c15ddf0 2024-04-06T12:24:51.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.561+0000 7fe77ad9d700 1 --2- 172.21.15.189:0/683385520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77c15ddf0 0x7fe77c15e280 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe77c0b6200 tx=0x7fe77400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afaab08f1fb2f7b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:51.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.561+0000 7fe77959a700 1 -- 172.21.15.189:0/683385520 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77400e070 con 0x7fe77c15ddf0 2024-04-06T12:24:51.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.561+0000 7fe77959a700 1 -- 172.21.15.189:0/683385520 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe77400b9c0 con 0x7fe77c15ddf0 2024-04-06T12:24:51.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.561+0000 7fe77959a700 1 -- 172.21.15.189:0/683385520 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77400bd70 con 0x7fe77c15ddf0 2024-04-06T12:24:51.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.561+0000 7fe780dac700 1 -- 172.21.15.189:0/683385520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77c15ddf0 msgr2=0x7fe77c15e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:51.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.561+0000 7fe780dac700 1 --2- 172.21.15.189:0/683385520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77c15ddf0 0x7fe77c15e280 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe77c0b6200 tx=0x7fe77400b330 comp rx=0 tx=0).stop 2024-04-06T12:24:51.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.561+0000 7fe780dac700 1 -- 172.21.15.189:0/683385520 shutdown_connections 2024-04-06T12:24:51.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.561+0000 7fe780dac700 1 --2- 172.21.15.189:0/683385520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77c15ddf0 0x7fe77c15e280 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:51.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.561+0000 7fe780dac700 1 --2- 172.21.15.189:0/683385520 >> v2:172.21.15.192:3300/0 conn(0x7fe77c15d7b0 0x7fe77c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:51.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.561+0000 7fe780dac700 1 --2- 172.21.15.189:0/683385520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe77c06a730 0x7fe77c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:51.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.561+0000 7fe780dac700 1 -- 172.21.15.189:0/683385520 >> 172.21.15.189:0/683385520 conn(0x7fe77c0d00b0 msgr2=0x7fe77c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:51.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.561+0000 7fe780dac700 1 -- 172.21.15.189:0/683385520 shutdown_connections 2024-04-06T12:24:51.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.561+0000 7fe780dac700 1 -- 172.21.15.189:0/683385520 wait complete. 2024-04-06T12:24:51.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.563+0000 7fe780dac700 1 Processor -- start 2024-04-06T12:24:51.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.563+0000 7fe780dac700 1 -- start start 2024-04-06T12:24:51.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe780dac700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77c06a730 0x7fe77c1f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:51.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe780dac700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe77c15d7b0 0x7fe77c0c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:51.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe780dac700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe77c15ddf0 0x7fe77c0ac950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:51.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe780dac700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77c14f740 con 0x7fe77c06a730 2024-04-06T12:24:51.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe780dac700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77c1f67d0 con 0x7fe77c15ddf0 2024-04-06T12:24:51.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe780dac700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe77c1f6ab0 con 0x7fe77c15d7b0 2024-04-06T12:24:51.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe77a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77c06a730 0x7fe77c1f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:51.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe77a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77c06a730 0x7fe77c1f26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42726/0 (socket says 172.21.15.189:42726) 2024-04-06T12:24:51.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe77a59c700 1 -- 172.21.15.189:0/1050953579 learned_addr learned my addr 172.21.15.189:0/1050953579 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:51.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe77ad9d700 1 --2- 172.21.15.189:0/1050953579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe77c15ddf0 0x7fe77c0ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:51.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe779d9b700 1 --2- 172.21.15.189:0/1050953579 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe77c15d7b0 0x7fe77c0c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:51.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe77a59c700 1 -- 172.21.15.189:0/1050953579 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe77c15d7b0 msgr2=0x7fe77c0c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:51.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe77a59c700 1 --2- 172.21.15.189:0/1050953579 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe77c15d7b0 0x7fe77c0c4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:51.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe77a59c700 1 -- 172.21.15.189:0/1050953579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe77c15ddf0 msgr2=0x7fe77c0ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:51.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe77a59c700 1 --2- 172.21.15.189:0/1050953579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe77c15ddf0 0x7fe77c0ac950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:51.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe77a59c700 1 -- 172.21.15.189:0/1050953579 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe774009b30 con 0x7fe77c06a730 2024-04-06T12:24:51.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe77a59c700 1 --2- 172.21.15.189:0/1050953579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77c06a730 0x7fe77c1f26b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe77000e810 tx=0x7fe77000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:51.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe76affd700 1 -- 172.21.15.189:0/1050953579 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77000c070 con 0x7fe77c06a730 2024-04-06T12:24:51.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe76affd700 1 -- 172.21.15.189:0/1050953579 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe770013610 con 0x7fe77c06a730 2024-04-06T12:24:51.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe780dac700 1 -- 172.21.15.189:0/1050953579 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe77c1f6d90 con 0x7fe77c06a730 2024-04-06T12:24:51.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe76affd700 1 -- 172.21.15.189:0/1050953579 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7700139c0 con 0x7fe77c06a730 2024-04-06T12:24:51.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.564+0000 7fe780dac700 1 -- 172.21.15.189:0/1050953579 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe77c1f2bf0 con 0x7fe77c06a730 2024-04-06T12:24:51.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.565+0000 7fe780dac700 1 -- 172.21.15.189:0/1050953579 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe77c149520 con 0x7fe77c06a730 2024-04-06T12:24:51.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.566+0000 7fe76affd700 1 -- 172.21.15.189:0/1050953579 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 89191+0+0 (secure 0 0 0) 0x7fe770013b20 con 0x7fe77c06a730 2024-04-06T12:24:51.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.566+0000 7fe76affd700 1 --2- 172.21.15.189:0/1050953579 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe75c06ad20 0x7fe75c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:51.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.566+0000 7fe779d9b700 1 --2- 172.21.15.189:0/1050953579 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe75c06ad20 0x7fe75c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:51.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.566+0000 7fe76affd700 1 -- 172.21.15.189:0/1050953579 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6789+0+0 (secure 0 0 0) 0x7fe7700895e0 con 0x7fe77c06a730 2024-04-06T12:24:51.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.567+0000 7fe779d9b700 1 --2- 172.21.15.189:0/1050953579 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe75c06ad20 0x7fe75c06d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe77c0cc340 tx=0x7fe76c008040 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:51.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.571+0000 7fe76affd700 1 -- 172.21.15.189:0/1050953579 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe770054290 con 0x7fe77c06a730 2024-04-06T12:24:51.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:51.772+0000 7fe780dac700 1 -- 172.21.15.189:0/1050953579 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe77c067bb0 con 0x7fe77c06a730 2024-04-06T12:24:52.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.623+0000 7fe76affd700 1 -- 172.21.15.189:0/1050953579 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fe770054660 con 0x7fe77c06a730 2024-04-06T12:24:52.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.625+0000 7fe76affd700 1 -- 172.21.15.189:0/1050953579 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v58) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe770072c20 con 0x7fe77c06a730 2024-04-06T12:24:52.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.630+0000 7fe780dac700 1 -- 172.21.15.189:0/1050953579 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe75c06ad20 msgr2=0x7fe75c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:52.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.630+0000 7fe780dac700 1 --2- 172.21.15.189:0/1050953579 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe75c06ad20 0x7fe75c06d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe77c0cc340 tx=0x7fe76c008040 comp rx=0 tx=0).stop 2024-04-06T12:24:52.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.630+0000 7fe780dac700 1 -- 172.21.15.189:0/1050953579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77c06a730 msgr2=0x7fe77c1f26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:52.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.630+0000 7fe780dac700 1 --2- 172.21.15.189:0/1050953579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77c06a730 0x7fe77c1f26b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe77000e810 tx=0x7fe77000eb20 comp rx=0 tx=0).stop 2024-04-06T12:24:52.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.630+0000 7fe780dac700 1 -- 172.21.15.189:0/1050953579 shutdown_connections 2024-04-06T12:24:52.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.630+0000 7fe780dac700 1 --2- 172.21.15.189:0/1050953579 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe75c06ad20 0x7fe75c06d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:52.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.630+0000 7fe780dac700 1 --2- 172.21.15.189:0/1050953579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe77c15ddf0 0x7fe77c0ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:52.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.630+0000 7fe780dac700 1 --2- 172.21.15.189:0/1050953579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77c06a730 0x7fe77c1f26b0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:52.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.630+0000 7fe780dac700 1 --2- 172.21.15.189:0/1050953579 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe77c15d7b0 0x7fe77c0c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:52.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.630+0000 7fe780dac700 1 -- 172.21.15.189:0/1050953579 >> 172.21.15.189:0/1050953579 conn(0x7fe77c0d00b0 msgr2=0x7fe77c0ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:52.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.630+0000 7fe780dac700 1 -- 172.21.15.189:0/1050953579 shutdown_connections 2024-04-06T12:24:52.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.630+0000 7fe780dac700 1 -- 172.21.15.189:0/1050953579 wait complete. 2024-04-06T12:24:52.629 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' removed 2024-04-06T12:24:52.643 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-06T12:24:52.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.777+0000 7f1a7e426700 1 Processor -- start 2024-04-06T12:24:52.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.777+0000 7f1a7e426700 1 -- start start 2024-04-06T12:24:52.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a7e426700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1a780cfa00 0x7f1a780cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:52.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a7e426700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a780cbd80 con 0x7f1a780cb870 2024-04-06T12:24:52.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a7e426700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a780cef20 con 0x7f1a780d11b0 2024-04-06T12:24:52.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a7e426700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a780ccdc0 con 0x7f1a780cfa00 2024-04-06T12:24:52.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a777fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1a780cb870 0x7f1a7815c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49388/0 (socket says 172.21.15.189:49388) 2024-04-06T12:24:52.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a777fe700 1 -- 172.21.15.189:0/785387342 learned_addr learned my addr 172.21.15.189:0/785387342 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:52.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a7c9c3700 1 --2- 172.21.15.189:0/785387342 >> v2:172.21.15.192:3300/0 conn(0x7f1a780cfa00 0x7f1a780cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:52.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a76ffd700 1 -- 172.21.15.189:0/785387342 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979578970 0 0) 0x7f1a780cbd80 con 0x7f1a780cb870 2024-04-06T12:24:52.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a76ffd700 1 -- 172.21.15.189:0/785387342 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a600035b0 con 0x7f1a780cb870 2024-04-06T12:24:52.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a76ffd700 1 -- 172.21.15.189:0/785387342 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228335319 0 0) 0x7f1a780cef20 con 0x7f1a780d11b0 2024-04-06T12:24:52.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a76ffd700 1 -- 172.21.15.189:0/785387342 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a780cbd80 con 0x7f1a780d11b0 2024-04-06T12:24:52.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a76ffd700 1 -- 172.21.15.189:0/785387342 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3877333645 0 0) 0x7f1a600035b0 con 0x7f1a780cb870 2024-04-06T12:24:52.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a76ffd700 1 -- 172.21.15.189:0/785387342 >> v2:172.21.15.192:3300/0 conn(0x7f1a780cfa00 msgr2=0x7f1a780cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:52.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a76ffd700 1 --2- 172.21.15.189:0/785387342 >> v2:172.21.15.192:3300/0 conn(0x7f1a780cfa00 0x7f1a780cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:52.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a76ffd700 1 -- 172.21.15.189:0/785387342 >> v1:172.21.15.192:6789/0 conn(0x7f1a780d11b0 legacy=0x7f1a780d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:52.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a76ffd700 1 -- 172.21.15.189:0/785387342 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a780cf200 con 0x7f1a780cb870 2024-04-06T12:24:52.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a76ffd700 1 -- 172.21.15.189:0/785387342 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f1a6c001670 con 0x7f1a780cb870 2024-04-06T12:24:52.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a76ffd700 1 -- 172.21.15.189:0/785387342 >> v1:172.21.15.189:6789/0 conn(0x7f1a780cb870 legacy=0x7f1a7815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:52.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a76ffd700 1 --2- 172.21.15.189:0/785387342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a60003b90 0x7f1a60003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:52.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a76ffd700 1 --2- 172.21.15.189:0/785387342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a60004850 0x7f1a60004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:52.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a76ffd700 1 --2- 172.21.15.189:0/785387342 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a60005210 0x7f1a6000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:52.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a76ffd700 1 -- 172.21.15.189:0/785387342 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6000c110 con 0x7f1a60004850 2024-04-06T12:24:52.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a76ffd700 1 -- 172.21.15.189:0/785387342 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6000c280 con 0x7f1a60003b90 2024-04-06T12:24:52.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.778+0000 7f1a76ffd700 1 -- 172.21.15.189:0/785387342 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6000c440 con 0x7f1a60005210 2024-04-06T12:24:52.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.779+0000 7f1a777fe700 1 --2- 172.21.15.189:0/785387342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a60004850 0x7f1a60004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:52.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.779+0000 7f1a7c9c3700 1 --2- 172.21.15.189:0/785387342 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a60005210 0x7f1a6000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:52.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.779+0000 7f1a77fff700 1 --2- 172.21.15.189:0/785387342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a60003b90 0x7f1a60003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:52.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.779+0000 7f1a777fe700 1 -- 172.21.15.189:0/785387342 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a60005210 msgr2=0x7f1a6000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:52.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.779+0000 7f1a777fe700 1 --2- 172.21.15.189:0/785387342 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a60005210 0x7f1a6000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:52.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.779+0000 7f1a777fe700 1 -- 172.21.15.189:0/785387342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a60003b90 msgr2=0x7f1a60003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:52.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.779+0000 7f1a777fe700 1 --2- 172.21.15.189:0/785387342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a60003b90 0x7f1a60003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:52.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.779+0000 7f1a777fe700 1 -- 172.21.15.189:0/785387342 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a600035b0 con 0x7f1a60004850 2024-04-06T12:24:52.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.779+0000 7f1a7c9c3700 1 -- 172.21.15.189:0/785387342 reap_dead start 2024-04-06T12:24:52.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.779+0000 7f1a777fe700 1 --2- 172.21.15.189:0/785387342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a60004850 0x7f1a60004cd0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1a780cbcb0 tx=0x7f1a60005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68ead2677d8d7c48 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:52.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.779+0000 7f1a76ffd700 1 -- 172.21.15.189:0/785387342 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c00cce0 con 0x7f1a60004850 2024-04-06T12:24:52.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.779+0000 7f1a76ffd700 1 -- 172.21.15.189:0/785387342 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a6c00ce60 con 0x7f1a60004850 2024-04-06T12:24:52.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.779+0000 7f1a76ffd700 1 -- 172.21.15.189:0/785387342 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c005ba0 con 0x7f1a60004850 2024-04-06T12:24:52.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.780+0000 7f1a7e426700 1 -- 172.21.15.189:0/785387342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a60004850 msgr2=0x7f1a60004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:52.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.780+0000 7f1a7e426700 1 --2- 172.21.15.189:0/785387342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a60004850 0x7f1a60004cd0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1a780cbcb0 tx=0x7f1a60005880 comp rx=0 tx=0).stop 2024-04-06T12:24:52.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.780+0000 7f1a7e426700 1 -- 172.21.15.189:0/785387342 shutdown_connections 2024-04-06T12:24:52.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.780+0000 7f1a7e426700 1 --2- 172.21.15.189:0/785387342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a60004850 0x7f1a60004cd0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:52.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.780+0000 7f1a7e426700 1 -- 172.21.15.189:0/785387342 >> 172.21.15.189:0/785387342 conn(0x7f1a780b2db0 msgr2=0x7f1a780b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:52.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.780+0000 7f1a7e426700 1 -- 172.21.15.189:0/785387342 shutdown_connections 2024-04-06T12:24:52.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.780+0000 7f1a7e426700 1 -- 172.21.15.189:0/785387342 wait complete. 2024-04-06T12:24:52.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.781+0000 7f1a7e426700 1 Processor -- start 2024-04-06T12:24:52.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.781+0000 7f1a7e426700 1 -- start start 2024-04-06T12:24:52.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.782+0000 7f1a7e426700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a60004850 0x7f1a7806d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:52.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.782+0000 7f1a7e426700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a780c5da0 0x7f1a781555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:52.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.782+0000 7f1a7e426700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a7815dc30 0x7f1a7815a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:52.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.782+0000 7f1a7e426700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a781f6610 con 0x7f1a7815dc30 2024-04-06T12:24:52.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.782+0000 7f1a7e426700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a780c62a0 con 0x7f1a780c5da0 2024-04-06T12:24:52.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.782+0000 7f1a7e426700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a780c6580 con 0x7f1a60004850 2024-04-06T12:24:52.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.782+0000 7f1a7c9c3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a7815dc30 0x7f1a7815a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:52.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.782+0000 7f1a7c9c3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a7815dc30 0x7f1a7815a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42746/0 (socket says 172.21.15.189:42746) 2024-04-06T12:24:52.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.782+0000 7f1a7c9c3700 1 -- 172.21.15.189:0/1317823104 learned_addr learned my addr 172.21.15.189:0/1317823104 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:52.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.782+0000 7f1a777fe700 1 --2- 172.21.15.189:0/1317823104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a780c5da0 0x7f1a781555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:52.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.782+0000 7f1a77fff700 1 --2- 172.21.15.189:0/1317823104 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a60004850 0x7f1a7806d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:52.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.782+0000 7f1a7c9c3700 1 -- 172.21.15.189:0/1317823104 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a60004850 msgr2=0x7f1a7806d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:52.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.782+0000 7f1a7c9c3700 1 --2- 172.21.15.189:0/1317823104 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a60004850 0x7f1a7806d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:52.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.782+0000 7f1a7c9c3700 1 -- 172.21.15.189:0/1317823104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a780c5da0 msgr2=0x7f1a781555d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:52.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.782+0000 7f1a7c9c3700 1 --2- 172.21.15.189:0/1317823104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a780c5da0 0x7f1a781555d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:52.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.782+0000 7f1a7c9c3700 1 -- 172.21.15.189:0/1317823104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a780c6860 con 0x7f1a7815dc30 2024-04-06T12:24:52.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.782+0000 7f1a7c9c3700 1 --2- 172.21.15.189:0/1317823104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a7815dc30 0x7f1a7815a390 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1a7806dce0 tx=0x7f1a70011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:52.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.782+0000 7f1a74ff9700 1 -- 172.21.15.189:0/1317823104 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7000e040 con 0x7f1a7815dc30 2024-04-06T12:24:52.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.783+0000 7f1a74ff9700 1 -- 172.21.15.189:0/1317823104 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a7000f040 con 0x7f1a7815dc30 2024-04-06T12:24:52.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.783+0000 7f1a7e426700 1 -- 172.21.15.189:0/1317823104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a780c6b40 con 0x7f1a7815dc30 2024-04-06T12:24:52.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.783+0000 7f1a74ff9700 1 -- 172.21.15.189:0/1317823104 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7000fc40 con 0x7f1a7815dc30 2024-04-06T12:24:52.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.783+0000 7f1a7e426700 1 -- 172.21.15.189:0/1317823104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a78155b10 con 0x7f1a7815dc30 2024-04-06T12:24:52.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.783+0000 7f1a7e426700 1 -- 172.21.15.189:0/1317823104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a78072dc0 con 0x7f1a7815dc30 2024-04-06T12:24:52.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.784+0000 7f1a74ff9700 1 -- 172.21.15.189:0/1317823104 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f1a70022460 con 0x7f1a7815dc30 2024-04-06T12:24:52.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.784+0000 7f1a74ff9700 1 --2- 172.21.15.189:0/1317823104 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a6406adb0 0x7f1a6406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:52.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.784+0000 7f1a74ff9700 1 -- 172.21.15.189:0/1317823104 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f1a70089a30 con 0x7f1a7815dc30 2024-04-06T12:24:52.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.784+0000 7f1a77fff700 1 --2- 172.21.15.189:0/1317823104 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a6406adb0 0x7f1a6406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:52.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.785+0000 7f1a77fff700 1 --2- 172.21.15.189:0/1317823104 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a6406adb0 0x7f1a6406d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1a68000c00 tx=0x7f1a68009040 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:52.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.789+0000 7f1a74ff9700 1 -- 172.21.15.189:0/1317823104 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1a70054980 con 0x7f1a7815dc30 2024-04-06T12:24:52.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.983+0000 7f1a7e426700 1 -- 172.21.15.189:0/1317823104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1a78074990 con 0x7f1a7815dc30 2024-04-06T12:24:52.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.987+0000 7f1a74ff9700 1 -- 172.21.15.189:0/1317823104 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1a70057fa0 con 0x7f1a7815dc30 2024-04-06T12:24:52.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.989+0000 7f1a7e426700 1 -- 172.21.15.189:0/1317823104 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a6406adb0 msgr2=0x7f1a6406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:52.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.989+0000 7f1a7e426700 1 --2- 172.21.15.189:0/1317823104 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a6406adb0 0x7f1a6406d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1a68000c00 tx=0x7f1a68009040 comp rx=0 tx=0).stop 2024-04-06T12:24:52.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.989+0000 7f1a7e426700 1 -- 172.21.15.189:0/1317823104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a7815dc30 msgr2=0x7f1a7815a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:52.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.989+0000 7f1a7e426700 1 --2- 172.21.15.189:0/1317823104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a7815dc30 0x7f1a7815a390 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1a7806dce0 tx=0x7f1a70011ed0 comp rx=0 tx=0).stop 2024-04-06T12:24:52.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.990+0000 7f1a7e426700 1 -- 172.21.15.189:0/1317823104 shutdown_connections 2024-04-06T12:24:52.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.990+0000 7f1a7e426700 1 --2- 172.21.15.189:0/1317823104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a7815dc30 0x7f1a7815a390 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:52.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.990+0000 7f1a7e426700 1 --2- 172.21.15.189:0/1317823104 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a60004850 0x7f1a7806d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:52.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.990+0000 7f1a7e426700 1 --2- 172.21.15.189:0/1317823104 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a6406adb0 0x7f1a6406d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:52.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.990+0000 7f1a7e426700 1 --2- 172.21.15.189:0/1317823104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a780c5da0 0x7f1a781555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:52.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.990+0000 7f1a7e426700 1 -- 172.21.15.189:0/1317823104 >> 172.21.15.189:0/1317823104 conn(0x7f1a780b2db0 msgr2=0x7f1a780b3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:52.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.990+0000 7f1a7e426700 1 -- 172.21.15.189:0/1317823104 shutdown_connections 2024-04-06T12:24:52.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:52.990+0000 7f1a7e426700 1 -- 172.21.15.189:0/1317823104 wait complete. 2024-04-06T12:24:53.000 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-06T12:24:53.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.131+0000 7f52a5602700 1 Processor -- start 2024-04-06T12:24:53.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.131+0000 7f52a5602700 1 -- start start 2024-04-06T12:24:53.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.131+0000 7f52a5602700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f52a00b2cd0 0x7f52a006c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f52a5602700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a015d8d0 con 0x7f52a00b2680 2024-04-06T12:24:53.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f52a5602700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a015d580 con 0x7f52a015d040 2024-04-06T12:24:53.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f52a5602700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a00cba40 con 0x7f52a00b2cd0 2024-04-06T12:24:53.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f52a00b2680 0x7f52a00cc1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49396/0 (socket says 172.21.15.189:49396) 2024-04-06T12:24:53.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529f7fe700 1 -- 172.21.15.189:0/177863757 learned_addr learned my addr 172.21.15.189:0/177863757 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:53.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529e7fc700 1 --2- 172.21.15.189:0/177863757 >> v2:172.21.15.192:3300/0 conn(0x7f52a00b2cd0 0x7f52a006c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529dffb700 1 -- 172.21.15.189:0/177863757 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161744780 0 0) 0x7f52a015d8d0 con 0x7f52a00b2680 2024-04-06T12:24:53.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529dffb700 1 -- 172.21.15.189:0/177863757 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52880035b0 con 0x7f52a00b2680 2024-04-06T12:24:53.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529dffb700 1 -- 172.21.15.189:0/177863757 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3000789501 0 0) 0x7f52880035b0 con 0x7f52a00b2680 2024-04-06T12:24:53.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529dffb700 1 -- 172.21.15.189:0/177863757 >> v2:172.21.15.192:3300/0 conn(0x7f52a00b2cd0 msgr2=0x7f52a006c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:53.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529dffb700 1 --2- 172.21.15.189:0/177863757 >> v2:172.21.15.192:3300/0 conn(0x7f52a00b2cd0 0x7f52a006c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529dffb700 1 -- 172.21.15.189:0/177863757 >> v1:172.21.15.192:6789/0 conn(0x7f52a015d040 legacy=0x7f52a015dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:53.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529dffb700 1 -- 172.21.15.189:0/177863757 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a015e470 con 0x7f52a00b2680 2024-04-06T12:24:53.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529dffb700 1 -- 172.21.15.189:0/177863757 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f52980019f0 con 0x7f52a00b2680 2024-04-06T12:24:53.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529dffb700 1 -- 172.21.15.189:0/177863757 >> v1:172.21.15.189:6789/0 conn(0x7f52a00b2680 legacy=0x7f52a00cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:53.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529dffb700 1 --2- 172.21.15.189:0/177863757 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5288003a00 0x7f5288003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529dffb700 1 --2- 172.21.15.189:0/177863757 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5288004850 0x7f5288004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529dffb700 1 --2- 172.21.15.189:0/177863757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5288005210 0x7f528800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529dffb700 1 -- 172.21.15.189:0/177863757 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528800c180 con 0x7f5288005210 2024-04-06T12:24:53.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529dffb700 1 -- 172.21.15.189:0/177863757 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528800c2f0 con 0x7f5288004850 2024-04-06T12:24:53.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529dffb700 1 -- 172.21.15.189:0/177863757 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f528800c4b0 con 0x7f5288003a00 2024-04-06T12:24:53.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529f7fe700 1 --2- 172.21.15.189:0/177863757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5288005210 0x7f528800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.132+0000 7f529e7fc700 1 --2- 172.21.15.189:0/177863757 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5288004850 0x7f5288004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.133+0000 7f529effd700 1 --2- 172.21.15.189:0/177863757 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5288003a00 0x7f5288003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.133+0000 7f529f7fe700 1 -- 172.21.15.189:0/177863757 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5288003a00 msgr2=0x7f5288003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:53.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.133+0000 7f529f7fe700 1 --2- 172.21.15.189:0/177863757 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5288003a00 0x7f5288003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.133+0000 7f529f7fe700 1 -- 172.21.15.189:0/177863757 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5288004850 msgr2=0x7f5288004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:53.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.133+0000 7f529f7fe700 1 --2- 172.21.15.189:0/177863757 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5288004850 0x7f5288004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.133+0000 7f529f7fe700 1 -- 172.21.15.189:0/177863757 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52880035b0 con 0x7f5288005210 2024-04-06T12:24:53.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.133+0000 7f529f7fe700 1 -- 172.21.15.189:0/177863757 reap_dead start 2024-04-06T12:24:53.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.133+0000 7f529f7fe700 1 --2- 172.21.15.189:0/177863757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5288005210 0x7f528800ba10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f529800a830 tx=0x7f5288005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9c2ee9252a2ec5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:53.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.133+0000 7f529dffb700 1 -- 172.21.15.189:0/177863757 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298010690 con 0x7f5288005210 2024-04-06T12:24:53.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.133+0000 7f529dffb700 1 -- 172.21.15.189:0/177863757 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5298010810 con 0x7f5288005210 2024-04-06T12:24:53.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.133+0000 7f529dffb700 1 -- 172.21.15.189:0/177863757 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298010bc0 con 0x7f5288005210 2024-04-06T12:24:53.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.133+0000 7f52a5602700 1 -- 172.21.15.189:0/177863757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5288005210 msgr2=0x7f528800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:53.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.133+0000 7f52a5602700 1 --2- 172.21.15.189:0/177863757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5288005210 0x7f528800ba10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f529800a830 tx=0x7f5288005880 comp rx=0 tx=0).stop 2024-04-06T12:24:53.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.133+0000 7f52a5602700 1 -- 172.21.15.189:0/177863757 shutdown_connections 2024-04-06T12:24:53.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.133+0000 7f52a5602700 1 --2- 172.21.15.189:0/177863757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5288005210 0x7f528800ba10 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.133+0000 7f52a5602700 1 -- 172.21.15.189:0/177863757 >> 172.21.15.189:0/177863757 conn(0x7f52a006a730 msgr2=0x7f52a006ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:53.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.133+0000 7f52a5602700 1 -- 172.21.15.189:0/177863757 shutdown_connections 2024-04-06T12:24:53.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.133+0000 7f52a5602700 1 -- 172.21.15.189:0/177863757 wait complete. 2024-04-06T12:24:53.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.135+0000 7f52a5602700 1 Processor -- start 2024-04-06T12:24:53.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.135+0000 7f52a5602700 1 -- start start 2024-04-06T12:24:53.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.136+0000 7f52a5602700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5288005210 0x7f52a0153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.136+0000 7f52a5602700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52a01523a0 0x7f52a00ca460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.136+0000 7f52a5602700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52a00ca9a0 0x7f52a00cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.136+0000 7f52a5602700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a00cae20 con 0x7f52a00ca9a0 2024-04-06T12:24:53.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.136+0000 7f52a5602700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a00cdb10 con 0x7f5288005210 2024-04-06T12:24:53.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.136+0000 7f52a5602700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a0154300 con 0x7f52a01523a0 2024-04-06T12:24:53.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.136+0000 7f529f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52a00ca9a0 0x7f52a00cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.136+0000 7f529f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52a00ca9a0 0x7f52a00cce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42768/0 (socket says 172.21.15.189:42768) 2024-04-06T12:24:53.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.136+0000 7f529f7fe700 1 -- 172.21.15.189:0/3466803545 learned_addr learned my addr 172.21.15.189:0/3466803545 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:53.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.136+0000 7f529effd700 1 --2- 172.21.15.189:0/3466803545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5288005210 0x7f52a0153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.136+0000 7f529e7fc700 1 --2- 172.21.15.189:0/3466803545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52a01523a0 0x7f52a00ca460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.136+0000 7f529f7fe700 1 -- 172.21.15.189:0/3466803545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52a01523a0 msgr2=0x7f52a00ca460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.137+0000 7f529f7fe700 1 --2- 172.21.15.189:0/3466803545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52a01523a0 0x7f52a00ca460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.137+0000 7f529f7fe700 1 -- 172.21.15.189:0/3466803545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5288005210 msgr2=0x7f52a0153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.137+0000 7f529f7fe700 1 --2- 172.21.15.189:0/3466803545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5288005210 0x7f52a0153dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.137+0000 7f529f7fe700 1 -- 172.21.15.189:0/3466803545 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a01515e0 con 0x7f52a00ca9a0 2024-04-06T12:24:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.137+0000 7f529f7fe700 1 --2- 172.21.15.189:0/3466803545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52a00ca9a0 0x7f52a00cce90 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f52a00cf9c0 tx=0x7f5298001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.137+0000 7f52877fe700 1 -- 172.21.15.189:0/3466803545 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529800a2f0 con 0x7f52a00ca9a0 2024-04-06T12:24:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.137+0000 7f52877fe700 1 -- 172.21.15.189:0/3466803545 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5298011670 con 0x7f52a00ca9a0 2024-04-06T12:24:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.137+0000 7f52a5602700 1 -- 172.21.15.189:0/3466803545 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a01518c0 con 0x7f52a00ca9a0 2024-04-06T12:24:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.137+0000 7f52877fe700 1 -- 172.21.15.189:0/3466803545 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298011a20 con 0x7f52a00ca9a0 2024-04-06T12:24:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.137+0000 7f52a5602700 1 -- 172.21.15.189:0/3466803545 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a0151e80 con 0x7f52a00ca9a0 2024-04-06T12:24:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.138+0000 7f52a5602700 1 -- 172.21.15.189:0/3466803545 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a0073270 con 0x7f52a00ca9a0 2024-04-06T12:24:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.138+0000 7f52877fe700 1 -- 172.21.15.189:0/3466803545 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f5298011b80 con 0x7f52a00ca9a0 2024-04-06T12:24:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.138+0000 7f52877fe700 1 --2- 172.21.15.189:0/3466803545 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f528c06adb0 0x7f528c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.139+0000 7f52877fe700 1 -- 172.21.15.189:0/3466803545 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f529808ce90 con 0x7f52a00ca9a0 2024-04-06T12:24:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.139+0000 7f529effd700 1 --2- 172.21.15.189:0/3466803545 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f528c06adb0 0x7f528c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.139+0000 7f529effd700 1 --2- 172.21.15.189:0/3466803545 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f528c06adb0 0x7f528c06d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5290000c00 tx=0x7f5290009040 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:53.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.143+0000 7f52877fe700 1 -- 172.21.15.189:0/3466803545 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5298057f00 con 0x7f52a00ca9a0 2024-04-06T12:24:53.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.336+0000 7f52a5602700 1 -- 172.21.15.189:0/3466803545 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f52a0072840 con 0x7f52a00ca9a0 2024-04-06T12:24:53.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.340+0000 7f52877fe700 1 -- 172.21.15.189:0/3466803545 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v14) v1 ==== 115+0+0 (secure 0 0 0) 0x7f529805b520 con 0x7f52a00ca9a0 2024-04-06T12:24:53.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.342+0000 7f52a5602700 1 -- 172.21.15.189:0/3466803545 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f528c06adb0 msgr2=0x7f528c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:53.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.342+0000 7f52a5602700 1 --2- 172.21.15.189:0/3466803545 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f528c06adb0 0x7f528c06d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5290000c00 tx=0x7f5290009040 comp rx=0 tx=0).stop 2024-04-06T12:24:53.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.343+0000 7f52a5602700 1 -- 172.21.15.189:0/3466803545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52a00ca9a0 msgr2=0x7f52a00cce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:53.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.343+0000 7f52a5602700 1 --2- 172.21.15.189:0/3466803545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52a00ca9a0 0x7f52a00cce90 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f52a00cf9c0 tx=0x7f5298001d80 comp rx=0 tx=0).stop 2024-04-06T12:24:53.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.343+0000 7f52a5602700 1 -- 172.21.15.189:0/3466803545 shutdown_connections 2024-04-06T12:24:53.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.343+0000 7f52a5602700 1 --2- 172.21.15.189:0/3466803545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52a00ca9a0 0x7f52a00cce90 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.343+0000 7f52a5602700 1 --2- 172.21.15.189:0/3466803545 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f528c06adb0 0x7f528c06d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.343+0000 7f52a5602700 1 --2- 172.21.15.189:0/3466803545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5288005210 0x7f52a0153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.343+0000 7f52a5602700 1 --2- 172.21.15.189:0/3466803545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52a01523a0 0x7f52a00ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.343+0000 7f52a5602700 1 -- 172.21.15.189:0/3466803545 >> 172.21.15.189:0/3466803545 conn(0x7f52a006a730 msgr2=0x7f52a0071bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:53.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.343+0000 7f52a5602700 1 -- 172.21.15.189:0/3466803545 shutdown_connections 2024-04-06T12:24:53.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.343+0000 7f52a5602700 1 -- 172.21.15.189:0/3466803545 wait complete. 2024-04-06T12:24:53.354 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-06T12:24:53.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.480+0000 7f69aebe0700 1 Processor -- start 2024-04-06T12:24:53.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.480+0000 7f69aebe0700 1 -- start start 2024-04-06T12:24:53.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.481+0000 7f69aebe0700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f69a815a550 0x7f69a80cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.481+0000 7f69aebe0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a80cdbe0 con 0x7f69a80cf900 2024-04-06T12:24:53.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.481+0000 7f69aebe0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a815a940 con 0x7f69a80b4d70 2024-04-06T12:24:53.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.481+0000 7f69aebe0700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a815ac00 con 0x7f69a815a550 2024-04-06T12:24:53.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.481+0000 7f69a7fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f69a80cf900 0x7f69a80b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49412/0 (socket says 172.21.15.189:49412) 2024-04-06T12:24:53.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.481+0000 7f69a7fff700 1 -- 172.21.15.189:0/751920543 learned_addr learned my addr 172.21.15.189:0/751920543 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:53.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a77fe700 1 -- 172.21.15.189:0/751920543 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869208431 0 0) 0x7f69a80cdbe0 con 0x7f69a80cf900 2024-04-06T12:24:53.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69ac97c700 1 --2- 172.21.15.189:0/751920543 >> v2:172.21.15.192:3300/0 conn(0x7f69a815a550 0x7f69a80cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a77fe700 1 -- 172.21.15.189:0/751920543 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69980057c0 con 0x7f69a80cf900 2024-04-06T12:24:53.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a77fe700 1 -- 172.21.15.189:0/751920543 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 877743621 0 0) 0x7f69980057c0 con 0x7f69a80cf900 2024-04-06T12:24:53.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a77fe700 1 -- 172.21.15.189:0/751920543 >> v2:172.21.15.192:3300/0 conn(0x7f69a815a550 msgr2=0x7f69a80cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:53.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a77fe700 1 --2- 172.21.15.189:0/751920543 >> v2:172.21.15.192:3300/0 conn(0x7f69a815a550 0x7f69a80cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a77fe700 1 -- 172.21.15.189:0/751920543 >> v1:172.21.15.192:6789/0 conn(0x7f69a80b4d70 legacy=0x7f69a80b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:53.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a77fe700 1 -- 172.21.15.189:0/751920543 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a815aee0 con 0x7f69a80cf900 2024-04-06T12:24:53.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a77fe700 1 -- 172.21.15.189:0/751920543 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f6994001670 con 0x7f69a80cf900 2024-04-06T12:24:53.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a77fe700 1 -- 172.21.15.189:0/751920543 >> v1:172.21.15.189:6789/0 conn(0x7f69a80cf900 legacy=0x7f69a80b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:53.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a77fe700 1 --2- 172.21.15.189:0/751920543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6998005c10 0x7f6998006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a77fe700 1 --2- 172.21.15.189:0/751920543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6998006a60 0x7f6998006ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a77fe700 1 --2- 172.21.15.189:0/751920543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6998007420 0x7f699800dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a77fe700 1 -- 172.21.15.189:0/751920543 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a815ac00 con 0x7f6998006a60 2024-04-06T12:24:53.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a77fe700 1 -- 172.21.15.189:0/751920543 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699800e2c0 con 0x7f6998007420 2024-04-06T12:24:53.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a77fe700 1 -- 172.21.15.189:0/751920543 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699800e480 con 0x7f6998005c10 2024-04-06T12:24:53.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a7fff700 1 --2- 172.21.15.189:0/751920543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6998006a60 0x7f6998006ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69ad17d700 1 --2- 172.21.15.189:0/751920543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6998007420 0x7f699800dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69ac97c700 1 --2- 172.21.15.189:0/751920543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6998005c10 0x7f6998006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a7fff700 1 -- 172.21.15.189:0/751920543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6998005c10 msgr2=0x7f6998006030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:53.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a7fff700 1 --2- 172.21.15.189:0/751920543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6998005c10 0x7f6998006030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a7fff700 1 -- 172.21.15.189:0/751920543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6998007420 msgr2=0x7f699800dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:53.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a7fff700 1 --2- 172.21.15.189:0/751920543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6998007420 0x7f699800dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69ad17d700 1 -- 172.21.15.189:0/751920543 reap_dead start 2024-04-06T12:24:53.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a7fff700 1 -- 172.21.15.189:0/751920543 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69980057c0 con 0x7f6998006a60 2024-04-06T12:24:53.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69ac97c700 1 --2- 172.21.15.189:0/751920543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6998005c10 0x7f6998006030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:24:53.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.482+0000 7f69a7fff700 1 --2- 172.21.15.189:0/751920543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6998006a60 0x7f6998006ee0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f699400c240 tx=0x7f6998007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbda7e0e1baca993 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:53.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.483+0000 7f69a77fe700 1 -- 172.21.15.189:0/751920543 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6994007c10 con 0x7f6998006a60 2024-04-06T12:24:53.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.483+0000 7f69a77fe700 1 -- 172.21.15.189:0/751920543 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699400ce00 con 0x7f6998006a60 2024-04-06T12:24:53.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.483+0000 7f69a77fe700 1 -- 172.21.15.189:0/751920543 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6994005ba0 con 0x7f6998006a60 2024-04-06T12:24:53.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.483+0000 7f69aebe0700 1 -- 172.21.15.189:0/751920543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6998006a60 msgr2=0x7f6998006ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:53.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.483+0000 7f69aebe0700 1 --2- 172.21.15.189:0/751920543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6998006a60 0x7f6998006ee0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f699400c240 tx=0x7f6998007a90 comp rx=0 tx=0).stop 2024-04-06T12:24:53.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.483+0000 7f69aebe0700 1 -- 172.21.15.189:0/751920543 shutdown_connections 2024-04-06T12:24:53.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.483+0000 7f69aebe0700 1 --2- 172.21.15.189:0/751920543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6998006a60 0x7f6998006ee0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.483+0000 7f69aebe0700 1 -- 172.21.15.189:0/751920543 >> 172.21.15.189:0/751920543 conn(0x7f69a80d00b0 msgr2=0x7f69a80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:53.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.483+0000 7f69aebe0700 1 -- 172.21.15.189:0/751920543 shutdown_connections 2024-04-06T12:24:53.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.483+0000 7f69aebe0700 1 -- 172.21.15.189:0/751920543 wait complete. 2024-04-06T12:24:53.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.484+0000 7f69aebe0700 1 Processor -- start 2024-04-06T12:24:53.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.484+0000 7f69aebe0700 1 -- start start 2024-04-06T12:24:53.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.485+0000 7f69aebe0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6998006a60 0x7f69a813e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.485+0000 7f69aebe0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69a813b880 0x7f69a80b1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.485+0000 7f69aebe0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69a814b950 0x7f69a813aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.485+0000 7f69aebe0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a815da70 con 0x7f69a814b950 2024-04-06T12:24:53.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.485+0000 7f69aebe0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a814cc60 con 0x7f69a813b880 2024-04-06T12:24:53.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.485+0000 7f69aebe0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a813e810 con 0x7f6998006a60 2024-04-06T12:24:53.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.485+0000 7f69ad17d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69a814b950 0x7f69a813aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.485+0000 7f69ad17d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69a814b950 0x7f69a813aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42794/0 (socket says 172.21.15.189:42794) 2024-04-06T12:24:53.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.485+0000 7f69ad17d700 1 -- 172.21.15.189:0/720932066 learned_addr learned my addr 172.21.15.189:0/720932066 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:53.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.485+0000 7f69a7fff700 1 --2- 172.21.15.189:0/720932066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69a813b880 0x7f69a80b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.486+0000 7f69ac97c700 1 --2- 172.21.15.189:0/720932066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6998006a60 0x7f69a813e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.486+0000 7f69ad17d700 1 -- 172.21.15.189:0/720932066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6998006a60 msgr2=0x7f69a813e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:53.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.486+0000 7f69ad17d700 1 --2- 172.21.15.189:0/720932066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6998006a60 0x7f69a813e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.486+0000 7f69ad17d700 1 -- 172.21.15.189:0/720932066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69a813b880 msgr2=0x7f69a80b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:53.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.486+0000 7f69ad17d700 1 --2- 172.21.15.189:0/720932066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69a813b880 0x7f69a80b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.486+0000 7f69ad17d700 1 -- 172.21.15.189:0/720932066 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a814ced0 con 0x7f69a814b950 2024-04-06T12:24:53.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.486+0000 7f69a7fff700 1 --2- 172.21.15.189:0/720932066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69a813b880 0x7f69a80b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:24:53.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.486+0000 7f69ad17d700 1 --2- 172.21.15.189:0/720932066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69a814b950 0x7f69a813aba0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f69a80cf160 tx=0x7f69a000d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:53.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.486+0000 7f69a57fa700 1 -- 172.21.15.189:0/720932066 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a0003cd0 con 0x7f69a814b950 2024-04-06T12:24:53.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.486+0000 7f69a57fa700 1 -- 172.21.15.189:0/720932066 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a0003e30 con 0x7f69a814b950 2024-04-06T12:24:53.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.486+0000 7f69aebe0700 1 -- 172.21.15.189:0/720932066 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a814d1b0 con 0x7f69a814b950 2024-04-06T12:24:53.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.486+0000 7f69a57fa700 1 -- 172.21.15.189:0/720932066 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a000d9e0 con 0x7f69a814b950 2024-04-06T12:24:53.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.486+0000 7f69aebe0700 1 -- 172.21.15.189:0/720932066 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a814e830 con 0x7f69a814b950 2024-04-06T12:24:53.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.487+0000 7f69aebe0700 1 -- 172.21.15.189:0/720932066 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a814edc0 con 0x7f69a814b950 2024-04-06T12:24:53.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.488+0000 7f69a57fa700 1 -- 172.21.15.189:0/720932066 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f69a000db40 con 0x7f69a814b950 2024-04-06T12:24:53.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.488+0000 7f69a57fa700 1 --2- 172.21.15.189:0/720932066 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f698c06adb0 0x7f698c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.488+0000 7f69a57fa700 1 -- 172.21.15.189:0/720932066 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f69a008a050 con 0x7f69a814b950 2024-04-06T12:24:53.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.488+0000 7f69ac97c700 1 --2- 172.21.15.189:0/720932066 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f698c06adb0 0x7f698c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.489+0000 7f69ac97c700 1 --2- 172.21.15.189:0/720932066 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f698c06adb0 0x7f698c06d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f69a8068450 tx=0x7f699c008040 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:53.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.492+0000 7f69a57fa700 1 -- 172.21.15.189:0/720932066 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f69a0055120 con 0x7f69a814b950 2024-04-06T12:24:53.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.688+0000 7f69aebe0700 1 -- 172.21.15.189:0/720932066 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f69a814edc0 con 0x7f69a814b950 2024-04-06T12:24:53.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.692+0000 7f69a57fa700 1 -- 172.21.15.189:0/720932066 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v15) v1 ==== 120+0+0 (secure 0 0 0) 0x7f69a814edc0 con 0x7f69a814b950 2024-04-06T12:24:53.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.694+0000 7f69aebe0700 1 -- 172.21.15.189:0/720932066 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f698c06adb0 msgr2=0x7f698c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:53.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.694+0000 7f69aebe0700 1 --2- 172.21.15.189:0/720932066 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f698c06adb0 0x7f698c06d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f69a8068450 tx=0x7f699c008040 comp rx=0 tx=0).stop 2024-04-06T12:24:53.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.694+0000 7f69aebe0700 1 -- 172.21.15.189:0/720932066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69a814b950 msgr2=0x7f69a813aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:53.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.694+0000 7f69aebe0700 1 --2- 172.21.15.189:0/720932066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69a814b950 0x7f69a813aba0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f69a80cf160 tx=0x7f69a000d150 comp rx=0 tx=0).stop 2024-04-06T12:24:53.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.694+0000 7f69aebe0700 1 -- 172.21.15.189:0/720932066 shutdown_connections 2024-04-06T12:24:53.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.694+0000 7f69aebe0700 1 --2- 172.21.15.189:0/720932066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69a814b950 0x7f69a813aba0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.694+0000 7f69aebe0700 1 --2- 172.21.15.189:0/720932066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6998006a60 0x7f69a813e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.694+0000 7f69aebe0700 1 --2- 172.21.15.189:0/720932066 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f698c06adb0 0x7f698c06d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.694+0000 7f69aebe0700 1 --2- 172.21.15.189:0/720932066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69a813b880 0x7f69a80b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.694+0000 7f69aebe0700 1 -- 172.21.15.189:0/720932066 >> 172.21.15.189:0/720932066 conn(0x7f69a80d00b0 msgr2=0x7f69a80b2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:53.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.695+0000 7f69aebe0700 1 -- 172.21.15.189:0/720932066 shutdown_connections 2024-04-06T12:24:53.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.695+0000 7f69aebe0700 1 -- 172.21.15.189:0/720932066 wait complete. 2024-04-06T12:24:53.705 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2024-04-06T12:24:53.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.830+0000 7f13977fc700 1 Processor -- start 2024-04-06T12:24:53.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.830+0000 7f13977fc700 1 -- start start 2024-04-06T12:24:53.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.831+0000 7f13977fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f139015d7b0 0x7f13900b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.831+0000 7f13977fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f139006a730 0x7f139006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.831+0000 7f13977fc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13900b3990 con 0x7f139015ddf0 2024-04-06T12:24:53.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.831+0000 7f13977fc700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f139006b0f0 con 0x7f139006a730 2024-04-06T12:24:53.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.831+0000 7f13977fc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13900b3060 con 0x7f139015d7b0 2024-04-06T12:24:53.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.831+0000 7f1394d97700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f139006a730 0x7f139006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.831+0000 7f1394d97700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f139006a730 0x7f139006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42810/0 (socket says 172.21.15.189:42810) 2024-04-06T12:24:53.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.832+0000 7f1394d97700 1 -- 172.21.15.189:0/864395907 learned_addr learned my addr 172.21.15.189:0/864395907 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:53.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.832+0000 7f1395598700 1 --2- 172.21.15.189:0/864395907 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f139015d7b0 0x7f13900b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.832+0000 7f1394d97700 1 -- 172.21.15.189:0/864395907 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f139015d7b0 msgr2=0x7f13900b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:53.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.832+0000 7f1394d97700 1 --2- 172.21.15.189:0/864395907 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f139015d7b0 0x7f13900b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.832+0000 7f1394d97700 1 -- 172.21.15.189:0/864395907 >> v1:172.21.15.189:6789/0 conn(0x7f139015ddf0 legacy=0x7f139015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:53.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.832+0000 7f1394d97700 1 -- 172.21.15.189:0/864395907 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13900b3340 con 0x7f139006a730 2024-04-06T12:24:53.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.832+0000 7f1394d97700 1 --2- 172.21.15.189:0/864395907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f139006a730 0x7f139006abb0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f13840098f0 tx=0x7f1384009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a91c1a9e43daeb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:53.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.832+0000 7f1383fff700 1 -- 172.21.15.189:0/864395907 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1384011830 con 0x7f139006a730 2024-04-06T12:24:53.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.832+0000 7f1383fff700 1 -- 172.21.15.189:0/864395907 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1384011990 con 0x7f139006a730 2024-04-06T12:24:53.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.832+0000 7f1383fff700 1 -- 172.21.15.189:0/864395907 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1384011d20 con 0x7f139006a730 2024-04-06T12:24:53.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.833+0000 7f13977fc700 1 -- 172.21.15.189:0/864395907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f139006a730 msgr2=0x7f139006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:53.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.833+0000 7f13977fc700 1 --2- 172.21.15.189:0/864395907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f139006a730 0x7f139006abb0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f13840098f0 tx=0x7f1384009c00 comp rx=0 tx=0).stop 2024-04-06T12:24:53.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.833+0000 7f13977fc700 1 -- 172.21.15.189:0/864395907 shutdown_connections 2024-04-06T12:24:53.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.833+0000 7f13977fc700 1 --2- 172.21.15.189:0/864395907 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f139015d7b0 0x7f13900b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.833+0000 7f13977fc700 1 --2- 172.21.15.189:0/864395907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f139006a730 0x7f139006abb0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.833+0000 7f13977fc700 1 -- 172.21.15.189:0/864395907 >> 172.21.15.189:0/864395907 conn(0x7f13900d00b0 msgr2=0x7f13900d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:53.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.833+0000 7f13977fc700 1 -- 172.21.15.189:0/864395907 shutdown_connections 2024-04-06T12:24:53.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.833+0000 7f13977fc700 1 -- 172.21.15.189:0/864395907 wait complete. 2024-04-06T12:24:53.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.834+0000 7f13977fc700 1 Processor -- start 2024-04-06T12:24:53.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.834+0000 7f13977fc700 1 -- start start 2024-04-06T12:24:53.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f13977fc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f139006a730 0x7f13900af090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f13977fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f139015d7b0 0x7f13901f6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f13977fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f139015ddf0 0x7f1390146630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f13977fc700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139015af70 con 0x7f139015ddf0 2024-04-06T12:24:53.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f13977fc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13901f71c0 con 0x7f139006a730 2024-04-06T12:24:53.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f13977fc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13901f74a0 con 0x7f139015d7b0 2024-04-06T12:24:53.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f1395d99700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f139015ddf0 0x7f1390146630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f1395d99700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f139015ddf0 0x7f1390146630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42814/0 (socket says 172.21.15.189:42814) 2024-04-06T12:24:53.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f1395d99700 1 -- 172.21.15.189:0/1023925067 learned_addr learned my addr 172.21.15.189:0/1023925067 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:53.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f1395598700 1 --2- 172.21.15.189:0/1023925067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f139006a730 0x7f13900af090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f1394d97700 1 --2- 172.21.15.189:0/1023925067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f139015d7b0 0x7f13901f6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f1395d99700 1 -- 172.21.15.189:0/1023925067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f139015d7b0 msgr2=0x7f13901f6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:53.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f1395d99700 1 --2- 172.21.15.189:0/1023925067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f139015d7b0 0x7f13901f6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f1395d99700 1 -- 172.21.15.189:0/1023925067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f139006a730 msgr2=0x7f13900af090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:53.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f1395d99700 1 --2- 172.21.15.189:0/1023925067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f139006a730 0x7f13900af090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:53.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f1395d99700 1 -- 172.21.15.189:0/1023925067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1384009550 con 0x7f139015ddf0 2024-04-06T12:24:53.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f1395d99700 1 --2- 172.21.15.189:0/1023925067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f139015ddf0 0x7f1390146630 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f138800cb50 tx=0x7f138800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:53.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f1381ffb700 1 -- 172.21.15.189:0/1023925067 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1388003c00 con 0x7f139015ddf0 2024-04-06T12:24:53.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f1381ffb700 1 -- 172.21.15.189:0/1023925067 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1388003d60 con 0x7f139015ddf0 2024-04-06T12:24:53.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f13977fc700 1 -- 172.21.15.189:0/1023925067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f139015b210 con 0x7f139015ddf0 2024-04-06T12:24:53.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.835+0000 7f1381ffb700 1 -- 172.21.15.189:0/1023925067 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138800aa20 con 0x7f139015ddf0 2024-04-06T12:24:53.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.836+0000 7f13977fc700 1 -- 172.21.15.189:0/1023925067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13900cc630 con 0x7f139015ddf0 2024-04-06T12:24:53.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.836+0000 7f13977fc700 1 -- 172.21.15.189:0/1023925067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f139015b730 con 0x7f139015ddf0 2024-04-06T12:24:53.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.837+0000 7f1381ffb700 1 -- 172.21.15.189:0/1023925067 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f138800ab80 con 0x7f139015ddf0 2024-04-06T12:24:53.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.837+0000 7f1381ffb700 1 --2- 172.21.15.189:0/1023925067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f137406adb0 0x7f137406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:53.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.837+0000 7f1381ffb700 1 -- 172.21.15.189:0/1023925067 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f1388089520 con 0x7f139015ddf0 2024-04-06T12:24:53.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.837+0000 7f1395598700 1 --2- 172.21.15.189:0/1023925067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f137406adb0 0x7f137406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:53.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.838+0000 7f1395598700 1 --2- 172.21.15.189:0/1023925067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f137406adb0 0x7f137406d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f138c009be0 tx=0x7f138c009210 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:53.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:53.842+0000 7f1381ffb700 1 -- 172.21.15.189:0/1023925067 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1388054580 con 0x7f139015ddf0 2024-04-06T12:24:54.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.118+0000 7f13977fc700 1 -- 172.21.15.189:0/1023925067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f1390067bb0 con 0x7f139015ddf0 2024-04-06T12:24:54.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.622+0000 7f1381ffb700 1 -- 172.21.15.189:0/1023925067 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v465) v1 ==== 158+0+0 (secure 0 0 0) 0x7f1388057ba0 con 0x7f139015ddf0 2024-04-06T12:24:54.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.625+0000 7f13977fc700 1 -- 172.21.15.189:0/1023925067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f137406adb0 msgr2=0x7f137406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:54.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.625+0000 7f13977fc700 1 --2- 172.21.15.189:0/1023925067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f137406adb0 0x7f137406d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f138c009be0 tx=0x7f138c009210 comp rx=0 tx=0).stop 2024-04-06T12:24:54.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.625+0000 7f13977fc700 1 -- 172.21.15.189:0/1023925067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f139015ddf0 msgr2=0x7f1390146630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:54.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.625+0000 7f13977fc700 1 --2- 172.21.15.189:0/1023925067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f139015ddf0 0x7f1390146630 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f138800cb50 tx=0x7f138800ce60 comp rx=0 tx=0).stop 2024-04-06T12:24:54.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.625+0000 7f13977fc700 1 -- 172.21.15.189:0/1023925067 shutdown_connections 2024-04-06T12:24:54.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.625+0000 7f13977fc700 1 --2- 172.21.15.189:0/1023925067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f139015ddf0 0x7f1390146630 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:54.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.625+0000 7f13977fc700 1 --2- 172.21.15.189:0/1023925067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f137406adb0 0x7f137406d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:54.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.625+0000 7f13977fc700 1 --2- 172.21.15.189:0/1023925067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f139006a730 0x7f13900af090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:54.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.625+0000 7f13977fc700 1 --2- 172.21.15.189:0/1023925067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f139015d7b0 0x7f13901f6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:54.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.625+0000 7f13977fc700 1 -- 172.21.15.189:0/1023925067 >> 172.21.15.189:0/1023925067 conn(0x7f13900d00b0 msgr2=0x7f13900ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:54.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.625+0000 7f13977fc700 1 -- 172.21.15.189:0/1023925067 shutdown_connections 2024-04-06T12:24:54.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.625+0000 7f13977fc700 1 -- 172.21.15.189:0/1023925067 wait complete. 2024-04-06T12:24:54.638 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2024-04-06T12:24:54.638 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2024-04-06T12:24:54.638 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-06T12:24:54.638 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2024-04-06T12:24:54.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.767+0000 7f60405ec700 1 Processor -- start 2024-04-06T12:24:54.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.767+0000 7f60405ec700 1 -- start start 2024-04-06T12:24:54.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.768+0000 7f60405ec700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f60380aec70 0x7f60380cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:54.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.768+0000 7f60405ec700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f60380cea80 0x7f603806ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:54.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.768+0000 7f60405ec700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603815a830 con 0x7f60380cdae0 2024-04-06T12:24:54.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.768+0000 7f60405ec700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60380cd920 con 0x7f60380cea80 2024-04-06T12:24:54.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.768+0000 7f60405ec700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60380cc050 con 0x7f60380aec70 2024-04-06T12:24:54.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.768+0000 7f603eb89700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f60380cea80 0x7f603806ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:54.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.768+0000 7f603eb89700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f60380cea80 0x7f603806ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42828/0 (socket says 172.21.15.189:42828) 2024-04-06T12:24:54.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.768+0000 7f603eb89700 1 -- 172.21.15.189:0/3524109041 learned_addr learned my addr 172.21.15.189:0/3524109041 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:54.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.768+0000 7f603e388700 1 --2- 172.21.15.189:0/3524109041 >> v2:172.21.15.192:3300/0 conn(0x7f60380aec70 0x7f60380cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:54.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.769+0000 7f603eb89700 1 -- 172.21.15.189:0/3524109041 >> v2:172.21.15.192:3300/0 conn(0x7f60380aec70 msgr2=0x7f60380cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:54.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.769+0000 7f603eb89700 1 --2- 172.21.15.189:0/3524109041 >> v2:172.21.15.192:3300/0 conn(0x7f60380aec70 0x7f60380cd3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:54.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.769+0000 7f603eb89700 1 -- 172.21.15.189:0/3524109041 >> v1:172.21.15.189:6789/0 conn(0x7f60380cdae0 legacy=0x7f60380ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:54.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.769+0000 7f603d386700 1 -- 172.21.15.189:0/3524109041 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393944216 0 0) 0x7f603815a830 con 0x7f60380cdae0 2024-04-06T12:24:54.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.769+0000 7f603eb89700 1 -- 172.21.15.189:0/3524109041 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60380666f0 con 0x7f60380cea80 2024-04-06T12:24:54.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.769+0000 7f603eb89700 1 --2- 172.21.15.189:0/3524109041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60380cea80 0x7f603806ceb0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f60380b2420 tx=0x7f603000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13e74b691c9d2d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:54.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.769+0000 7f603d386700 1 -- 172.21.15.189:0/3524109041 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603000e070 con 0x7f60380cea80 2024-04-06T12:24:54.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.769+0000 7f603d386700 1 -- 172.21.15.189:0/3524109041 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603000b9c0 con 0x7f60380cea80 2024-04-06T12:24:54.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.769+0000 7f603d386700 1 -- 172.21.15.189:0/3524109041 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603000bd70 con 0x7f60380cea80 2024-04-06T12:24:54.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.769+0000 7f60405ec700 1 -- 172.21.15.189:0/3524109041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60380cea80 msgr2=0x7f603806ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:54.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.769+0000 7f60405ec700 1 --2- 172.21.15.189:0/3524109041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60380cea80 0x7f603806ceb0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f60380b2420 tx=0x7f603000b330 comp rx=0 tx=0).stop 2024-04-06T12:24:54.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.770+0000 7f60405ec700 1 -- 172.21.15.189:0/3524109041 shutdown_connections 2024-04-06T12:24:54.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.770+0000 7f60405ec700 1 --2- 172.21.15.189:0/3524109041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60380cea80 0x7f603806ceb0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:54.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.770+0000 7f60405ec700 1 --2- 172.21.15.189:0/3524109041 >> v2:172.21.15.192:3300/0 conn(0x7f60380aec70 0x7f60380cd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:54.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.770+0000 7f60405ec700 1 -- 172.21.15.189:0/3524109041 >> 172.21.15.189:0/3524109041 conn(0x7f60380cc290 msgr2=0x7f60380cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:54.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.770+0000 7f60405ec700 1 -- 172.21.15.189:0/3524109041 shutdown_connections 2024-04-06T12:24:54.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.770+0000 7f60405ec700 1 -- 172.21.15.189:0/3524109041 wait complete. 2024-04-06T12:24:54.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.771+0000 7f60405ec700 1 Processor -- start 2024-04-06T12:24:54.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.771+0000 7f60405ec700 1 -- start start 2024-04-06T12:24:54.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.771+0000 7f60405ec700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60380aec70 0x7f60380c4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:54.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.771+0000 7f60405ec700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60380cdae0 0x7f60381f0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:54.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f60405ec700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60380cea80 0x7f60380c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:54.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f60405ec700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603814afc0 con 0x7f60380aec70 2024-04-06T12:24:54.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f60405ec700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603814a130 con 0x7f60380cdae0 2024-04-06T12:24:54.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f60405ec700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603814a340 con 0x7f60380cea80 2024-04-06T12:24:54.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f603e388700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60380aec70 0x7f60380c4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:54.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f603e388700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60380aec70 0x7f60380c4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42840/0 (socket says 172.21.15.189:42840) 2024-04-06T12:24:54.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f603e388700 1 -- 172.21.15.189:0/88379108 learned_addr learned my addr 172.21.15.189:0/88379108 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:54.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f603db87700 1 --2- 172.21.15.189:0/88379108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60380cdae0 0x7f60381f0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:54.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f603eb89700 1 --2- 172.21.15.189:0/88379108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60380cea80 0x7f60380c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:54.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f603e388700 1 -- 172.21.15.189:0/88379108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60380cea80 msgr2=0x7f60380c8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:54.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f603e388700 1 --2- 172.21.15.189:0/88379108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60380cea80 0x7f60380c8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:54.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f603e388700 1 -- 172.21.15.189:0/88379108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60380cdae0 msgr2=0x7f60381f0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:54.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f603e388700 1 --2- 172.21.15.189:0/88379108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60380cdae0 0x7f60381f0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:54.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f603e388700 1 -- 172.21.15.189:0/88379108 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6030009b30 con 0x7f60380aec70 2024-04-06T12:24:54.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f603e388700 1 --2- 172.21.15.189:0/88379108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60380aec70 0x7f60380c4e70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f603400e810 tx=0x7f603400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:54.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f602effd700 1 -- 172.21.15.189:0/88379108 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034013610 con 0x7f60380aec70 2024-04-06T12:24:54.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f602effd700 1 -- 172.21.15.189:0/88379108 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6034013770 con 0x7f60380aec70 2024-04-06T12:24:54.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f60405ec700 1 -- 172.21.15.189:0/88379108 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f603814a620 con 0x7f60380aec70 2024-04-06T12:24:54.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f602effd700 1 -- 172.21.15.189:0/88379108 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034013b20 con 0x7f60380aec70 2024-04-06T12:24:54.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.772+0000 7f60405ec700 1 -- 172.21.15.189:0/88379108 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f603814abe0 con 0x7f60380aec70 2024-04-06T12:24:54.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.773+0000 7f60405ec700 1 -- 172.21.15.189:0/88379108 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6038074990 con 0x7f60380aec70 2024-04-06T12:24:54.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.774+0000 7f602effd700 1 -- 172.21.15.189:0/88379108 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f6034013c80 con 0x7f60380aec70 2024-04-06T12:24:54.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.774+0000 7f602effd700 1 --2- 172.21.15.189:0/88379108 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f601c06adb0 0x7f601c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:54.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.774+0000 7f602effd700 1 -- 172.21.15.189:0/88379108 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f6034089300 con 0x7f60380aec70 2024-04-06T12:24:54.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.774+0000 7f603db87700 1 --2- 172.21.15.189:0/88379108 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f601c06adb0 0x7f601c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:54.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.774+0000 7f603db87700 1 --2- 172.21.15.189:0/88379108 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f601c06adb0 0x7f601c06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f60380c98e0 tx=0x7f6028009040 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:54.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:54.778+0000 7f602effd700 1 -- 172.21.15.189:0/88379108 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6034054260 con 0x7f60380aec70 2024-04-06T12:24:55.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.051+0000 7f60405ec700 1 -- 172.21.15.189:0/88379108 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7f60380b5790 con 0x7f60380aec70 2024-04-06T12:24:55.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.625+0000 7f602effd700 1 -- 172.21.15.189:0/88379108 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v466) v1 ==== 157+0+0 (secure 0 0 0) 0x7f6034057880 con 0x7f60380aec70 2024-04-06T12:24:55.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.628+0000 7f60405ec700 1 -- 172.21.15.189:0/88379108 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f601c06adb0 msgr2=0x7f601c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:55.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.628+0000 7f60405ec700 1 --2- 172.21.15.189:0/88379108 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f601c06adb0 0x7f601c06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f60380c98e0 tx=0x7f6028009040 comp rx=0 tx=0).stop 2024-04-06T12:24:55.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.628+0000 7f60405ec700 1 -- 172.21.15.189:0/88379108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60380aec70 msgr2=0x7f60380c4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:55.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.628+0000 7f60405ec700 1 --2- 172.21.15.189:0/88379108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60380aec70 0x7f60380c4e70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f603400e810 tx=0x7f603400ebd0 comp rx=0 tx=0).stop 2024-04-06T12:24:55.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.628+0000 7f60405ec700 1 -- 172.21.15.189:0/88379108 shutdown_connections 2024-04-06T12:24:55.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.628+0000 7f60405ec700 1 --2- 172.21.15.189:0/88379108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60380cea80 0x7f60380c8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:55.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.628+0000 7f60405ec700 1 --2- 172.21.15.189:0/88379108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60380aec70 0x7f60380c4e70 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:55.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.628+0000 7f60405ec700 1 --2- 172.21.15.189:0/88379108 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f601c06adb0 0x7f601c06d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:55.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.628+0000 7f60405ec700 1 --2- 172.21.15.189:0/88379108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60380cdae0 0x7f60381f0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:55.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.628+0000 7f60405ec700 1 -- 172.21.15.189:0/88379108 >> 172.21.15.189:0/88379108 conn(0x7f60380cc290 msgr2=0x7f60380b2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:55.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.628+0000 7f60405ec700 1 -- 172.21.15.189:0/88379108 shutdown_connections 2024-04-06T12:24:55.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.628+0000 7f60405ec700 1 -- 172.21.15.189:0/88379108 wait complete. 2024-04-06T12:24:55.641 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-06T12:24:55.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.773+0000 7fed2c44e700 1 Processor -- start 2024-04-06T12:24:55.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.773+0000 7fed2c44e700 1 -- start start 2024-04-06T12:24:55.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.774+0000 7fed2c44e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fed2415a550 0x7fed240cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:55.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.774+0000 7fed2c44e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fed240b4d70 0x7fed240b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:55.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.774+0000 7fed2c44e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed240cdbe0 con 0x7fed240cf900 2024-04-06T12:24:55.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.774+0000 7fed2c44e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed2415a940 con 0x7fed2415a550 2024-04-06T12:24:55.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.774+0000 7fed2c44e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed2415ac00 con 0x7fed240b4d70 2024-04-06T12:24:55.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.775+0000 7fed2a1ea700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fed2415a550 0x7fed240cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:55.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.775+0000 7fed2a1ea700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fed2415a550 0x7fed240cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42846/0 (socket says 172.21.15.189:42846) 2024-04-06T12:24:55.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.775+0000 7fed2a1ea700 1 -- 172.21.15.189:0/2312745585 learned_addr learned my addr 172.21.15.189:0/2312745585 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:55.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.775+0000 7fed2a9eb700 1 --2- 172.21.15.189:0/2312745585 >> v2:172.21.15.192:3300/0 conn(0x7fed240b4d70 0x7fed240b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:55.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.775+0000 7fed2a1ea700 1 -- 172.21.15.189:0/2312745585 >> v2:172.21.15.192:3300/0 conn(0x7fed240b4d70 msgr2=0x7fed240b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:55.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.775+0000 7fed2a1ea700 1 --2- 172.21.15.189:0/2312745585 >> v2:172.21.15.192:3300/0 conn(0x7fed240b4d70 0x7fed240b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:55.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.775+0000 7fed2a1ea700 1 -- 172.21.15.189:0/2312745585 >> v1:172.21.15.192:6789/0 conn(0x7fed240cf900 legacy=0x7fed240b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:55.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.775+0000 7fed2a1ea700 1 -- 172.21.15.189:0/2312745585 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2415aee0 con 0x7fed2415a550 2024-04-06T12:24:55.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.775+0000 7fed2a1ea700 1 --2- 172.21.15.189:0/2312745585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed2415a550 0x7fed240cf3c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fed24072830 tx=0x7fed18009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba46ee17a459e734 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:55.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.775+0000 7fed291e8700 1 -- 172.21.15.189:0/2312745585 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed18004230 con 0x7fed2415a550 2024-04-06T12:24:55.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.775+0000 7fed291e8700 1 -- 172.21.15.189:0/2312745585 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed18004390 con 0x7fed2415a550 2024-04-06T12:24:55.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.775+0000 7fed291e8700 1 -- 172.21.15.189:0/2312745585 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed180036a0 con 0x7fed2415a550 2024-04-06T12:24:55.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.776+0000 7fed2c44e700 1 -- 172.21.15.189:0/2312745585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed2415a550 msgr2=0x7fed240cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:55.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.776+0000 7fed2c44e700 1 --2- 172.21.15.189:0/2312745585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed2415a550 0x7fed240cf3c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fed24072830 tx=0x7fed18009bd0 comp rx=0 tx=0).stop 2024-04-06T12:24:55.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.776+0000 7fed2c44e700 1 -- 172.21.15.189:0/2312745585 shutdown_connections 2024-04-06T12:24:55.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.776+0000 7fed2c44e700 1 --2- 172.21.15.189:0/2312745585 >> v2:172.21.15.192:3300/0 conn(0x7fed240b4d70 0x7fed240b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:55.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.776+0000 7fed2c44e700 1 --2- 172.21.15.189:0/2312745585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed2415a550 0x7fed240cf3c0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:55.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.776+0000 7fed2c44e700 1 -- 172.21.15.189:0/2312745585 >> 172.21.15.189:0/2312745585 conn(0x7fed240d00b0 msgr2=0x7fed240d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:55.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.776+0000 7fed2c44e700 1 -- 172.21.15.189:0/2312745585 shutdown_connections 2024-04-06T12:24:55.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.776+0000 7fed2c44e700 1 -- 172.21.15.189:0/2312745585 wait complete. 2024-04-06T12:24:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.777+0000 7fed2c44e700 1 Processor -- start 2024-04-06T12:24:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.777+0000 7fed2c44e700 1 -- start start 2024-04-06T12:24:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.778+0000 7fed2c44e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed240b4d70 0x7fed241471d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.778+0000 7fed2c44e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed240cf900 0x7fed2415c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.778+0000 7fed2c44e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed24149260 0x7fed2415ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.778+0000 7fed2c44e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed240ad860 con 0x7fed24149260 2024-04-06T12:24:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.778+0000 7fed2c44e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed240ad330 con 0x7fed240b4d70 2024-04-06T12:24:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.778+0000 7fed2c44e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed241f0800 con 0x7fed240cf900 2024-04-06T12:24:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.778+0000 7fed2a9eb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed24149260 0x7fed2415ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.778+0000 7fed2a9eb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed24149260 0x7fed2415ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42856/0 (socket says 172.21.15.189:42856) 2024-04-06T12:24:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.778+0000 7fed2a9eb700 1 -- 172.21.15.189:0/806237682 learned_addr learned my addr 172.21.15.189:0/806237682 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:55.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.778+0000 7fed2a1ea700 1 --2- 172.21.15.189:0/806237682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed240b4d70 0x7fed241471d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:55.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.778+0000 7fed299e9700 1 --2- 172.21.15.189:0/806237682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed240cf900 0x7fed2415c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:55.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.778+0000 7fed2a9eb700 1 -- 172.21.15.189:0/806237682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed240cf900 msgr2=0x7fed2415c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:55.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.778+0000 7fed2a9eb700 1 --2- 172.21.15.189:0/806237682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed240cf900 0x7fed2415c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:55.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.778+0000 7fed2a9eb700 1 -- 172.21.15.189:0/806237682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed240b4d70 msgr2=0x7fed241471d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:55.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.778+0000 7fed2a9eb700 1 --2- 172.21.15.189:0/806237682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed240b4d70 0x7fed241471d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:55.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.778+0000 7fed2a9eb700 1 -- 172.21.15.189:0/806237682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed18009550 con 0x7fed24149260 2024-04-06T12:24:55.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.779+0000 7fed2a9eb700 1 --2- 172.21.15.189:0/806237682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed24149260 0x7fed2415ce50 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fed1c00cd70 tx=0x7fed1c00a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:55.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.779+0000 7fed16ffd700 1 -- 172.21.15.189:0/806237682 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1c003a20 con 0x7fed24149260 2024-04-06T12:24:55.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.779+0000 7fed16ffd700 1 -- 172.21.15.189:0/806237682 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1c003b80 con 0x7fed24149260 2024-04-06T12:24:55.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.779+0000 7fed2c44e700 1 -- 172.21.15.189:0/806237682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed24149790 con 0x7fed24149260 2024-04-06T12:24:55.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.779+0000 7fed16ffd700 1 -- 172.21.15.189:0/806237682 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1c004020 con 0x7fed24149260 2024-04-06T12:24:55.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.779+0000 7fed2c44e700 1 -- 172.21.15.189:0/806237682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed24149d50 con 0x7fed24149260 2024-04-06T12:24:55.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.780+0000 7fed2c44e700 1 -- 172.21.15.189:0/806237682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed24067bb0 con 0x7fed24149260 2024-04-06T12:24:55.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.780+0000 7fed16ffd700 1 -- 172.21.15.189:0/806237682 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fed1c004180 con 0x7fed24149260 2024-04-06T12:24:55.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.780+0000 7fed16ffd700 1 --2- 172.21.15.189:0/806237682 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fed0806adb0 0x7fed0806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:55.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.780+0000 7fed16ffd700 1 -- 172.21.15.189:0/806237682 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5822+0+0 (secure 0 0 0) 0x7fed1c012070 con 0x7fed24149260 2024-04-06T12:24:55.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.780+0000 7fed2a1ea700 1 --2- 172.21.15.189:0/806237682 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fed0806adb0 0x7fed0806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:55.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.781+0000 7fed2a1ea700 1 --2- 172.21.15.189:0/806237682 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fed0806adb0 0x7fed0806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fed18009130 tx=0x7fed18008030 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:55.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.785+0000 7fed16ffd700 1 -- 172.21.15.189:0/806237682 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fed1c054c40 con 0x7fed24149260 2024-04-06T12:24:55.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.980+0000 7fed2c44e700 1 -- 172.21.15.189:0/806237682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fed24074990 con 0x7fed24149260 2024-04-06T12:24:55.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.985+0000 7fed16ffd700 1 -- 172.21.15.189:0/806237682 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fed1c058260 con 0x7fed24149260 2024-04-06T12:24:55.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.987+0000 7fed2c44e700 1 -- 172.21.15.189:0/806237682 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fed0806adb0 msgr2=0x7fed0806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:55.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.987+0000 7fed2c44e700 1 --2- 172.21.15.189:0/806237682 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fed0806adb0 0x7fed0806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fed18009130 tx=0x7fed18008030 comp rx=0 tx=0).stop 2024-04-06T12:24:55.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.987+0000 7fed2c44e700 1 -- 172.21.15.189:0/806237682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed24149260 msgr2=0x7fed2415ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:55.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.987+0000 7fed2c44e700 1 --2- 172.21.15.189:0/806237682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed24149260 0x7fed2415ce50 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fed1c00cd70 tx=0x7fed1c00a570 comp rx=0 tx=0).stop 2024-04-06T12:24:55.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.987+0000 7fed2c44e700 1 -- 172.21.15.189:0/806237682 shutdown_connections 2024-04-06T12:24:55.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.987+0000 7fed2c44e700 1 --2- 172.21.15.189:0/806237682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed24149260 0x7fed2415ce50 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:55.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.987+0000 7fed2c44e700 1 --2- 172.21.15.189:0/806237682 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fed0806adb0 0x7fed0806d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:55.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.987+0000 7fed2c44e700 1 --2- 172.21.15.189:0/806237682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed240b4d70 0x7fed241471d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:55.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.987+0000 7fed2c44e700 1 --2- 172.21.15.189:0/806237682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed240cf900 0x7fed2415c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:55.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.987+0000 7fed2c44e700 1 -- 172.21.15.189:0/806237682 >> 172.21.15.189:0/806237682 conn(0x7fed240d00b0 msgr2=0x7fed240b3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:55.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.987+0000 7fed2c44e700 1 -- 172.21.15.189:0/806237682 shutdown_connections 2024-04-06T12:24:55.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:55.987+0000 7fed2c44e700 1 -- 172.21.15.189:0/806237682 wait complete. 2024-04-06T12:24:55.997 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T12:24:55.998 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:24:56.036 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:24:56.083 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T12:24:56.083 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:24:56.083 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:24:56.101 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:24:56.101 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:24:56.101 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:24:56.120 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:24:56.121 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:24:56.121 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:24:56.159 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:24:56.159 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:24:56.159 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:24:56.180 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:24:56.180 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-06T12:24:56.181 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:24:56.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.322+0000 7febb4a79700 1 Processor -- start 2024-04-06T12:24:56.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.322+0000 7febb4a79700 1 -- start start 2024-04-06T12:24:56.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.323+0000 7febb4a79700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7febb015d7b0 0x7febb00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:56.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.323+0000 7febb4a79700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7febb015e500 0x7febb006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:56.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.323+0000 7febb4a79700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb00b3990 con 0x7febb006a730 2024-04-06T12:24:56.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.323+0000 7febb4a79700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb015e950 con 0x7febb015e500 2024-04-06T12:24:56.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.323+0000 7febb4a79700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb006b0f0 con 0x7febb015d7b0 2024-04-06T12:24:56.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.323+0000 7febaed9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7febb015e500 0x7febb006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:56.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.323+0000 7febaed9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7febb015e500 0x7febb006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42872/0 (socket says 172.21.15.189:42872) 2024-04-06T12:24:56.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.323+0000 7febaed9d700 1 -- 172.21.15.189:0/738339171 learned_addr learned my addr 172.21.15.189:0/738339171 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:56.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.323+0000 7febae59c700 1 --2- 172.21.15.189:0/738339171 >> v2:172.21.15.192:3300/0 conn(0x7febb015d7b0 0x7febb00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:56.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.323+0000 7febaed9d700 1 -- 172.21.15.189:0/738339171 >> v2:172.21.15.192:3300/0 conn(0x7febb015d7b0 msgr2=0x7febb00b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:56.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.323+0000 7febaed9d700 1 --2- 172.21.15.189:0/738339171 >> v2:172.21.15.192:3300/0 conn(0x7febb015d7b0 0x7febb00b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:56.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.323+0000 7febaed9d700 1 -- 172.21.15.189:0/738339171 >> v1:172.21.15.189:6789/0 conn(0x7febb006a730 legacy=0x7febb015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:56.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.323+0000 7febaed9d700 1 -- 172.21.15.189:0/738339171 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb00b3060 con 0x7febb015e500 2024-04-06T12:24:56.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.323+0000 7febae59c700 1 --2- 172.21.15.189:0/738339171 >> v2:172.21.15.192:3300/0 conn(0x7febb015d7b0 0x7febb00b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:24:56.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.323+0000 7febaed9d700 1 --2- 172.21.15.189:0/738339171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb015e500 0x7febb006abb0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7febb00b6200 tx=0x7feba800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f84e362c4c583f33 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:56.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.323+0000 7febad59a700 1 -- 172.21.15.189:0/738339171 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba800e070 con 0x7febb015e500 2024-04-06T12:24:56.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.324+0000 7febad59a700 1 -- 172.21.15.189:0/738339171 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba800b9c0 con 0x7febb015e500 2024-04-06T12:24:56.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.324+0000 7febad59a700 1 -- 172.21.15.189:0/738339171 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba800bd70 con 0x7febb015e500 2024-04-06T12:24:56.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.324+0000 7febb4a79700 1 -- 172.21.15.189:0/738339171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb015e500 msgr2=0x7febb006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:56.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.324+0000 7febb4a79700 1 --2- 172.21.15.189:0/738339171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb015e500 0x7febb006abb0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7febb00b6200 tx=0x7feba800b330 comp rx=0 tx=0).stop 2024-04-06T12:24:56.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.324+0000 7febb4a79700 1 -- 172.21.15.189:0/738339171 shutdown_connections 2024-04-06T12:24:56.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.324+0000 7febb4a79700 1 --2- 172.21.15.189:0/738339171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb015e500 0x7febb006abb0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:56.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.324+0000 7febb4a79700 1 --2- 172.21.15.189:0/738339171 >> v2:172.21.15.192:3300/0 conn(0x7febb015d7b0 0x7febb00b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:56.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.324+0000 7febb4a79700 1 -- 172.21.15.189:0/738339171 >> 172.21.15.189:0/738339171 conn(0x7febb00d00b0 msgr2=0x7febb00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:56.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.324+0000 7febb4a79700 1 -- 172.21.15.189:0/738339171 shutdown_connections 2024-04-06T12:24:56.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.324+0000 7febb4a79700 1 -- 172.21.15.189:0/738339171 wait complete. 2024-04-06T12:24:56.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.325+0000 7febb4a79700 1 Processor -- start 2024-04-06T12:24:56.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.325+0000 7febb4a79700 1 -- start start 2024-04-06T12:24:56.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.326+0000 7febb4a79700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7febb006a730 0x7febb00ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:56.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.326+0000 7febb4a79700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb015d7b0 0x7febb0157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:56.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.326+0000 7febb4a79700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febb015e500 0x7febb0145a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:56.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.326+0000 7febb4a79700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb0158f10 con 0x7febb015d7b0 2024-04-06T12:24:56.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.326+0000 7febb4a79700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb0145f50 con 0x7febb015e500 2024-04-06T12:24:56.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.326+0000 7febb4a79700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb0146470 con 0x7febb006a730 2024-04-06T12:24:56.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.326+0000 7febadd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb015d7b0 0x7febb0157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:56.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.326+0000 7febadd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb015d7b0 0x7febb0157c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42886/0 (socket says 172.21.15.189:42886) 2024-04-06T12:24:56.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.326+0000 7febadd9b700 1 -- 172.21.15.189:0/1454899879 learned_addr learned my addr 172.21.15.189:0/1454899879 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:56.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.326+0000 7febaed9d700 1 --2- 172.21.15.189:0/1454899879 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febb015e500 0x7febb0145a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:56.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.326+0000 7febae59c700 1 --2- 172.21.15.189:0/1454899879 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7febb006a730 0x7febb00ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:56.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.327+0000 7febadd9b700 1 -- 172.21.15.189:0/1454899879 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7febb006a730 msgr2=0x7febb00ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:56.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.327+0000 7febadd9b700 1 --2- 172.21.15.189:0/1454899879 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7febb006a730 0x7febb00ccdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:56.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.327+0000 7febadd9b700 1 -- 172.21.15.189:0/1454899879 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febb015e500 msgr2=0x7febb0145a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:56.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.327+0000 7febadd9b700 1 --2- 172.21.15.189:0/1454899879 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febb015e500 0x7febb0145a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:56.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.327+0000 7febadd9b700 1 -- 172.21.15.189:0/1454899879 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba8009b30 con 0x7febb015d7b0 2024-04-06T12:24:56.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.327+0000 7febae59c700 1 --2- 172.21.15.189:0/1454899879 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7febb006a730 0x7febb00ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:24:56.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.327+0000 7febadd9b700 1 --2- 172.21.15.189:0/1454899879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb015d7b0 0x7febb0157c70 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7feba400c510 tx=0x7feba400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:56.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.327+0000 7feb9effd700 1 -- 172.21.15.189:0/1454899879 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba40052d0 con 0x7febb015d7b0 2024-04-06T12:24:56.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.327+0000 7feb9effd700 1 -- 172.21.15.189:0/1454899879 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba4005430 con 0x7febb015d7b0 2024-04-06T12:24:56.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.327+0000 7febb4a79700 1 -- 172.21.15.189:0/1454899879 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb0158300 con 0x7febb015d7b0 2024-04-06T12:24:56.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.327+0000 7feb9effd700 1 -- 172.21.15.189:0/1454899879 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba40038c0 con 0x7febb015d7b0 2024-04-06T12:24:56.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.327+0000 7febb4a79700 1 -- 172.21.15.189:0/1454899879 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb01588c0 con 0x7febb015d7b0 2024-04-06T12:24:56.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.328+0000 7febb4a79700 1 -- 172.21.15.189:0/1454899879 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb0067bb0 con 0x7febb015d7b0 2024-04-06T12:24:56.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.328+0000 7feb9effd700 1 -- 172.21.15.189:0/1454899879 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7feba4003a20 con 0x7febb015d7b0 2024-04-06T12:24:56.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.329+0000 7feb9effd700 1 --2- 172.21.15.189:0/1454899879 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb9006adb0 0x7feb9006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:56.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.329+0000 7febae59c700 1 --2- 172.21.15.189:0/1454899879 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb9006adb0 0x7feb9006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:56.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.329+0000 7feb9effd700 1 -- 172.21.15.189:0/1454899879 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5822+0+0 (secure 0 0 0) 0x7feba4088930 con 0x7febb015d7b0 2024-04-06T12:24:56.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.329+0000 7febae59c700 1 --2- 172.21.15.189:0/1454899879 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb9006adb0 0x7feb9006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7feba0006fd0 tx=0x7feba0008040 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:56.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.333+0000 7feb9effd700 1 -- 172.21.15.189:0/1454899879 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7feba4053990 con 0x7febb015d7b0 2024-04-06T12:24:56.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.575+0000 7febb4a79700 1 -- 172.21.15.189:0/1454899879 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febb0074990 con 0x7febb015d7b0 2024-04-06T12:24:56.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.575+0000 7feb9effd700 1 -- 172.21.15.189:0/1454899879 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+3535 (secure 0 0 0) 0x7feba4017090 con 0x7febb015d7b0 2024-04-06T12:24:56.574 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:56.574 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4865,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/2279772906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2279772906},{"type":"v1","addr":"172.21.15.192:6834","nonce":2279772906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4868,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/454939556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":454939556},{"type":"v1","addr":"172.21.15.192:6835","nonce":454939556}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5107,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/399541295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":399541295},{"type":"v1","addr":"172.21.15.189:6837","nonce":399541295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":6021,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2144304712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2144304712},{"type":"v1","addr":"172.21.15.189:6836","nonce":2144304712}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[]} 2024-04-06T12:24:56.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.577+0000 7febb4a79700 1 -- 172.21.15.189:0/1454899879 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb9006adb0 msgr2=0x7feb9006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:56.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.577+0000 7febb4a79700 1 --2- 172.21.15.189:0/1454899879 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb9006adb0 0x7feb9006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7feba0006fd0 tx=0x7feba0008040 comp rx=0 tx=0).stop 2024-04-06T12:24:56.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.577+0000 7febb4a79700 1 -- 172.21.15.189:0/1454899879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb015d7b0 msgr2=0x7febb0157c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:56.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.577+0000 7febb4a79700 1 --2- 172.21.15.189:0/1454899879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb015d7b0 0x7febb0157c70 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7feba400c510 tx=0x7feba400c820 comp rx=0 tx=0).stop 2024-04-06T12:24:56.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.577+0000 7febb4a79700 1 -- 172.21.15.189:0/1454899879 shutdown_connections 2024-04-06T12:24:56.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.577+0000 7febb4a79700 1 --2- 172.21.15.189:0/1454899879 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb9006adb0 0x7feb9006d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:56.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.577+0000 7febb4a79700 1 --2- 172.21.15.189:0/1454899879 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febb015e500 0x7febb0145a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:56.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.577+0000 7febb4a79700 1 --2- 172.21.15.189:0/1454899879 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7febb006a730 0x7febb00ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:56.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.577+0000 7febb4a79700 1 --2- 172.21.15.189:0/1454899879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb015d7b0 0x7febb0157c70 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:56.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.577+0000 7febb4a79700 1 -- 172.21.15.189:0/1454899879 >> 172.21.15.189:0/1454899879 conn(0x7febb00d00b0 msgr2=0x7febb00acdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:56.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.577+0000 7febb4a79700 1 -- 172.21.15.189:0/1454899879 shutdown_connections 2024-04-06T12:24:56.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.577+0000 7febb4a79700 1 -- 172.21.15.189:0/1454899879 wait complete. 2024-04-06T12:24:56.577 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 18 2024-04-06T12:24:56.588 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-06T12:24:56.588 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-06T12:24:56.588 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-06T12:24:56.588 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:24:56.589 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-06T12:24:56.589 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-06T12:24:56.589 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-06T12:24:56.589 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:24:56.589 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-06T12:24:56.589 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-06T12:24:56.590 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-06T12:24:56.590 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:24:56.590 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-06T12:24:56.590 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-06T12:24:56.590 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-06T12:24:56.590 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:24:56.597 INFO:tasks.ceph.mds.a:Stopped 2024-04-06T12:24:56.598 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-06T12:24:56.598 INFO:tasks.ceph.mds.c:Stopped 2024-04-06T12:24:56.599 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-06T12:24:56.604 INFO:tasks.ceph.mds.b:Stopped 2024-04-06T12:24:56.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-06T12:24:56.605 INFO:tasks.ceph.mds.d:Stopped 2024-04-06T12:24:56.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-06T12:24:56.640 INFO:tasks.ceph.mds.a:Started 2024-04-06T12:24:56.641 DEBUG:teuthology.parallel:result is None 2024-04-06T12:24:56.641 INFO:tasks.ceph.mds.c:Started 2024-04-06T12:24:56.641 DEBUG:teuthology.parallel:result is None 2024-04-06T12:24:56.647 INFO:tasks.ceph.mds.b:Started 2024-04-06T12:24:56.648 DEBUG:teuthology.parallel:result is None 2024-04-06T12:24:56.648 INFO:tasks.ceph.mds.d:Started 2024-04-06T12:24:56.648 DEBUG:teuthology.parallel:result is None 2024-04-06T12:24:56.648 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-06T12:24:56.721 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T12:24:56.721 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2024-04-06T12:24:56.735 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2024-04-06T12:24:56.735 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T12:24:56.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.782+0000 7f0c36044700 1 Processor -- start 2024-04-06T12:24:56.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.782+0000 7f0c36044700 1 -- start start 2024-04-06T12:24:56.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.783+0000 7f0c36044700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c300d11b0 0x7f0c300d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:56.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.783+0000 7f0c36044700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0c300cfa00 0x7f0c300cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:56.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.783+0000 7f0c36044700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c300d1ad0 con 0x7f0c300cb870 2024-04-06T12:24:56.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.783+0000 7f0c36044700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c300ccdc0 con 0x7f0c300cfa00 2024-04-06T12:24:56.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.783+0000 7f0c36044700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c300d0b50 con 0x7f0c300d11b0 2024-04-06T12:24:56.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.783+0000 7f0c2f7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c300d11b0 0x7f0c300d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:56.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.783+0000 7f0c2ffff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0c300cfa00 0x7f0c300cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:56.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.784+0000 7f0c2effd700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f0c300cb870 0x7f0c3015c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:39090/0 (socket says 172.21.15.189:39090) 2024-04-06T12:24:56.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.784+0000 7f0c2effd700 1 -- 172.21.15.189:0/3387187213 learned_addr learned my addr 172.21.15.189:0/3387187213 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:56.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.784+0000 7f0c2f7fe700 1 -- 172.21.15.189:0/3387187213 >> v2:172.21.15.192:3300/0 conn(0x7f0c300cfa00 msgr2=0x7f0c300cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:56.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.784+0000 7f0c2f7fe700 1 --2- 172.21.15.189:0/3387187213 >> v2:172.21.15.192:3300/0 conn(0x7f0c300cfa00 0x7f0c300cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:56.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.784+0000 7f0c2f7fe700 1 -- 172.21.15.189:0/3387187213 >> v1:172.21.15.192:6789/0 conn(0x7f0c300cb870 legacy=0x7f0c3015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:56.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.784+0000 7f0c2e7fc700 1 -- 172.21.15.189:0/3387187213 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9677810 0 0) 0x7f0c300d1ad0 con 0x7f0c300cb870 2024-04-06T12:24:56.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.784+0000 7f0c2f7fe700 1 -- 172.21.15.189:0/3387187213 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c300cbd80 con 0x7f0c300d11b0 2024-04-06T12:24:56.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.784+0000 7f0c2ffff700 1 --2- 172.21.15.189:0/3387187213 >> v2:172.21.15.192:3300/0 conn(0x7f0c300cfa00 0x7f0c300cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:56.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.784+0000 7f0c2f7fe700 1 --2- 172.21.15.189:0/3387187213 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c300d11b0 0x7f0c300d1590 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0c20009840 tx=0x7f0c20009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c15d7660b8a366a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:56.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.784+0000 7f0c2e7fc700 1 -- 172.21.15.189:0/3387187213 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c20011830 con 0x7f0c300d11b0 2024-04-06T12:24:56.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.784+0000 7f0c2e7fc700 1 -- 172.21.15.189:0/3387187213 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c20011990 con 0x7f0c300d11b0 2024-04-06T12:24:56.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.784+0000 7f0c2e7fc700 1 -- 172.21.15.189:0/3387187213 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c20011d20 con 0x7f0c300d11b0 2024-04-06T12:24:56.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.785+0000 7f0c36044700 1 -- 172.21.15.189:0/3387187213 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c300d11b0 msgr2=0x7f0c300d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:56.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.785+0000 7f0c36044700 1 --2- 172.21.15.189:0/3387187213 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c300d11b0 0x7f0c300d1590 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0c20009840 tx=0x7f0c20009b50 comp rx=0 tx=0).stop 2024-04-06T12:24:56.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.785+0000 7f0c36044700 1 -- 172.21.15.189:0/3387187213 shutdown_connections 2024-04-06T12:24:56.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.785+0000 7f0c36044700 1 --2- 172.21.15.189:0/3387187213 >> v2:172.21.15.192:3300/0 conn(0x7f0c300cfa00 0x7f0c300cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:56.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.785+0000 7f0c36044700 1 --2- 172.21.15.189:0/3387187213 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c300d11b0 0x7f0c300d1590 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:56.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.785+0000 7f0c36044700 1 -- 172.21.15.189:0/3387187213 >> 172.21.15.189:0/3387187213 conn(0x7f0c300b2db0 msgr2=0x7f0c300b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:56.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.785+0000 7f0c36044700 1 -- 172.21.15.189:0/3387187213 shutdown_connections 2024-04-06T12:24:56.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.785+0000 7f0c36044700 1 -- 172.21.15.189:0/3387187213 wait complete. 2024-04-06T12:24:56.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.786+0000 7f0c36044700 1 Processor -- start 2024-04-06T12:24:56.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.786+0000 7f0c36044700 1 -- start start 2024-04-06T12:24:56.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.787+0000 7f0c36044700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c300cb870 0x7f0c301579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:56.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.787+0000 7f0c36044700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c300cfa00 0x7f0c301f3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:56.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.787+0000 7f0c36044700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c300d11b0 0x7f0c3015e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:56.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.787+0000 7f0c36044700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c301f30b0 con 0x7f0c300cb870 2024-04-06T12:24:56.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.787+0000 7f0c36044700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c301581f0 con 0x7f0c300d11b0 2024-04-06T12:24:56.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.787+0000 7f0c36044700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c301584d0 con 0x7f0c300cfa00 2024-04-06T12:24:56.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.787+0000 7f0c2f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c300cb870 0x7f0c301579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:56.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.787+0000 7f0c2f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c300cb870 0x7f0c301579c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42914/0 (socket says 172.21.15.189:42914) 2024-04-06T12:24:56.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.787+0000 7f0c2f7fe700 1 -- 172.21.15.189:0/2318599038 learned_addr learned my addr 172.21.15.189:0/2318599038 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:56.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.787+0000 7f0c2effd700 1 --2- 172.21.15.189:0/2318599038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c300cfa00 0x7f0c301f3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:56.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.787+0000 7f0c2ffff700 1 --2- 172.21.15.189:0/2318599038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c300d11b0 0x7f0c3015e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:56.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.787+0000 7f0c2f7fe700 1 -- 172.21.15.189:0/2318599038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c300cfa00 msgr2=0x7f0c301f3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:56.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.787+0000 7f0c2f7fe700 1 --2- 172.21.15.189:0/2318599038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c300cfa00 0x7f0c301f3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:56.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.787+0000 7f0c2f7fe700 1 -- 172.21.15.189:0/2318599038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c300d11b0 msgr2=0x7f0c3015e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:56.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.787+0000 7f0c2f7fe700 1 --2- 172.21.15.189:0/2318599038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c300d11b0 0x7f0c3015e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:56.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.787+0000 7f0c2f7fe700 1 -- 172.21.15.189:0/2318599038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c28009a70 con 0x7f0c300cb870 2024-04-06T12:24:56.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.788+0000 7f0c2f7fe700 1 --2- 172.21.15.189:0/2318599038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c300cb870 0x7f0c301579c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f0c20009fd0 tx=0x7f0c20004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:56.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.788+0000 7f0c17fff700 1 -- 172.21.15.189:0/2318599038 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c200036a0 con 0x7f0c300cb870 2024-04-06T12:24:56.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.788+0000 7f0c17fff700 1 -- 172.21.15.189:0/2318599038 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2001f070 con 0x7f0c300cb870 2024-04-06T12:24:56.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.788+0000 7f0c17fff700 1 -- 172.21.15.189:0/2318599038 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c20003a60 con 0x7f0c300cb870 2024-04-06T12:24:56.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.788+0000 7f0c36044700 1 -- 172.21.15.189:0/2318599038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c200094a0 con 0x7f0c300cb870 2024-04-06T12:24:56.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.788+0000 7f0c36044700 1 -- 172.21.15.189:0/2318599038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c30157f00 con 0x7f0c300cb870 2024-04-06T12:24:56.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.789+0000 7f0c15ffb700 1 -- 172.21.15.189:0/2318599038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c080052f0 con 0x7f0c300cb870 2024-04-06T12:24:56.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.789+0000 7f0c17fff700 1 -- 172.21.15.189:0/2318599038 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0c20003bc0 con 0x7f0c300cb870 2024-04-06T12:24:56.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.789+0000 7f0c17fff700 1 --2- 172.21.15.189:0/2318599038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c1c06adb0 0x7f0c1c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:56.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.789+0000 7f0c17fff700 1 -- 172.21.15.189:0/2318599038 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f0c20021070 con 0x7f0c300cb870 2024-04-06T12:24:56.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.790+0000 7f0c2effd700 1 --2- 172.21.15.189:0/2318599038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c1c06adb0 0x7f0c1c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:56.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.790+0000 7f0c2effd700 1 --2- 172.21.15.189:0/2318599038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c1c06adb0 0x7f0c1c06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0c301573d0 tx=0x7f0c24007be0 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:56.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:56.794+0000 7f0c17fff700 1 -- 172.21.15.189:0/2318599038 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0c20055cb0 con 0x7f0c300cb870 2024-04-06T12:24:57.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:57.002+0000 7f0c15ffb700 1 -- 172.21.15.189:0/2318599038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0c080055a0 con 0x7f0c300cb870 2024-04-06T12:24:57.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:57.988+0000 7f0c17fff700 1 -- 172.21.15.189:0/2318599038 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v63) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0c2002b080 con 0x7f0c300cb870 2024-04-06T12:24:57.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:57.991+0000 7f0c15ffb700 1 -- 172.21.15.189:0/2318599038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c1c06adb0 msgr2=0x7f0c1c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:57.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:57.991+0000 7f0c15ffb700 1 --2- 172.21.15.189:0/2318599038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c1c06adb0 0x7f0c1c06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0c301573d0 tx=0x7f0c24007be0 comp rx=0 tx=0).stop 2024-04-06T12:24:57.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:57.991+0000 7f0c15ffb700 1 -- 172.21.15.189:0/2318599038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c300cb870 msgr2=0x7f0c301579c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:57.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:57.991+0000 7f0c15ffb700 1 --2- 172.21.15.189:0/2318599038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c300cb870 0x7f0c301579c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f0c20009fd0 tx=0x7f0c20004750 comp rx=0 tx=0).stop 2024-04-06T12:24:57.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:57.991+0000 7f0c15ffb700 1 -- 172.21.15.189:0/2318599038 shutdown_connections 2024-04-06T12:24:57.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:57.991+0000 7f0c15ffb700 1 --2- 172.21.15.189:0/2318599038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c1c06adb0 0x7f0c1c06d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:57.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:57.991+0000 7f0c15ffb700 1 --2- 172.21.15.189:0/2318599038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c300d11b0 0x7f0c3015e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:57.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:57.991+0000 7f0c15ffb700 1 --2- 172.21.15.189:0/2318599038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c300cb870 0x7f0c301579c0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:57.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:57.991+0000 7f0c15ffb700 1 --2- 172.21.15.189:0/2318599038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c300cfa00 0x7f0c301f3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:57.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:57.991+0000 7f0c15ffb700 1 -- 172.21.15.189:0/2318599038 >> 172.21.15.189:0/2318599038 conn(0x7f0c300b2db0 msgr2=0x7f0c300cddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:57.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:57.992+0000 7f0c15ffb700 1 -- 172.21.15.189:0/2318599038 shutdown_connections 2024-04-06T12:24:57.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:57.992+0000 7f0c15ffb700 1 -- 172.21.15.189:0/2318599038 wait complete. 2024-04-06T12:24:57.991 INFO:teuthology.orchestra.run.smithi189.stderr: removed all blocklist entries 2024-04-06T12:24:58.005 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-06T12:24:58.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.134+0000 7f150bc25700 1 Processor -- start 2024-04-06T12:24:58.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.134+0000 7f150bc25700 1 -- start start 2024-04-06T12:24:58.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.135+0000 7f150bc25700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f150415d610 0x7f15040b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:58.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.135+0000 7f150bc25700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150406abb0 0x7f1504161810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:58.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.135+0000 7f150bc25700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f150415d9f0 con 0x7f150406a730 2024-04-06T12:24:58.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.135+0000 7f150bc25700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15040d1040 con 0x7f150415d610 2024-04-06T12:24:58.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.135+0000 7f150bc25700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1504072760 con 0x7f150406abb0 2024-04-06T12:24:58.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.135+0000 7f15099c1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f150415d610 0x7f15040b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:58.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.135+0000 7f15099c1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f150415d610 0x7f15040b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42928/0 (socket says 172.21.15.189:42928) 2024-04-06T12:24:58.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.135+0000 7f15099c1700 1 -- 172.21.15.189:0/2952365580 learned_addr learned my addr 172.21.15.189:0/2952365580 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:58.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.135+0000 7f150a1c2700 1 --2- 172.21.15.189:0/2952365580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150406abb0 0x7f1504161810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:58.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.135+0000 7f15099c1700 1 -- 172.21.15.189:0/2952365580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150406abb0 msgr2=0x7f1504161810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:58.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.135+0000 7f15099c1700 1 --2- 172.21.15.189:0/2952365580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150406abb0 0x7f1504161810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.135+0000 7f15099c1700 1 -- 172.21.15.189:0/2952365580 >> v1:172.21.15.192:6789/0 conn(0x7f150406a730 legacy=0x7f1504161100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:58.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.135+0000 7f15099c1700 1 -- 172.21.15.189:0/2952365580 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15040719a0 con 0x7f150415d610 2024-04-06T12:24:58.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.136+0000 7f15099c1700 1 --2- 172.21.15.189:0/2952365580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150415d610 0x7f15040b3de0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f14f40098f0 tx=0x7f14f4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac1f64f251be0358 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:58.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.136+0000 7f15089bf700 1 -- 172.21.15.189:0/2952365580 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f4004230 con 0x7f150415d610 2024-04-06T12:24:58.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.136+0000 7f15089bf700 1 -- 172.21.15.189:0/2952365580 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f4004390 con 0x7f150415d610 2024-04-06T12:24:58.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.136+0000 7f15089bf700 1 -- 172.21.15.189:0/2952365580 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f40036a0 con 0x7f150415d610 2024-04-06T12:24:58.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.136+0000 7f150bc25700 1 -- 172.21.15.189:0/2952365580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150415d610 msgr2=0x7f15040b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:58.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.136+0000 7f150bc25700 1 --2- 172.21.15.189:0/2952365580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150415d610 0x7f15040b3de0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f14f40098f0 tx=0x7f14f4009c00 comp rx=0 tx=0).stop 2024-04-06T12:24:58.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.136+0000 7f150bc25700 1 -- 172.21.15.189:0/2952365580 shutdown_connections 2024-04-06T12:24:58.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.136+0000 7f150bc25700 1 --2- 172.21.15.189:0/2952365580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150406abb0 0x7f1504161810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.136+0000 7f150bc25700 1 --2- 172.21.15.189:0/2952365580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150415d610 0x7f15040b3de0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.136+0000 7f150bc25700 1 -- 172.21.15.189:0/2952365580 >> 172.21.15.189:0/2952365580 conn(0x7f15040d00b0 msgr2=0x7f15040d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:58.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.136+0000 7f150bc25700 1 -- 172.21.15.189:0/2952365580 shutdown_connections 2024-04-06T12:24:58.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.136+0000 7f150bc25700 1 -- 172.21.15.189:0/2952365580 wait complete. 2024-04-06T12:24:58.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.138+0000 7f150bc25700 1 Processor -- start 2024-04-06T12:24:58.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.138+0000 7f150bc25700 1 -- start start 2024-04-06T12:24:58.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.138+0000 7f150bc25700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f150406a730 0x7f15040c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:58.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.138+0000 7f150bc25700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150406abb0 0x7f15040c5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:58.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.138+0000 7f150bc25700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150415d610 0x7f15040ce590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:58.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.138+0000 7f150bc25700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15040c6420 con 0x7f150406abb0 2024-04-06T12:24:58.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.138+0000 7f150bc25700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15040c5570 con 0x7f150406a730 2024-04-06T12:24:58.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.138+0000 7f150bc25700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15040c5810 con 0x7f150415d610 2024-04-06T12:24:58.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.138+0000 7f15091c0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150406abb0 0x7f15040c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:58.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.138+0000 7f15091c0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150406abb0 0x7f15040c5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42940/0 (socket says 172.21.15.189:42940) 2024-04-06T12:24:58.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.138+0000 7f15091c0700 1 -- 172.21.15.189:0/1626445982 learned_addr learned my addr 172.21.15.189:0/1626445982 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:58.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.139+0000 7f150a1c2700 1 --2- 172.21.15.189:0/1626445982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150415d610 0x7f15040ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:58.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.139+0000 7f15099c1700 1 --2- 172.21.15.189:0/1626445982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f150406a730 0x7f15040c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:58.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.139+0000 7f15091c0700 1 -- 172.21.15.189:0/1626445982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150415d610 msgr2=0x7f15040ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:58.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.139+0000 7f15091c0700 1 --2- 172.21.15.189:0/1626445982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150415d610 0x7f15040ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.139+0000 7f15091c0700 1 -- 172.21.15.189:0/1626445982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f150406a730 msgr2=0x7f15040c68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:58.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.139+0000 7f15091c0700 1 --2- 172.21.15.189:0/1626445982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f150406a730 0x7f15040c68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.139+0000 7f15091c0700 1 -- 172.21.15.189:0/1626445982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f4009550 con 0x7f150406abb0 2024-04-06T12:24:58.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.139+0000 7f150a1c2700 1 --2- 172.21.15.189:0/1626445982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150415d610 0x7f15040ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:24:58.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.139+0000 7f15091c0700 1 --2- 172.21.15.189:0/1626445982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150406abb0 0x7f15040c5030 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f150000c4d0 tx=0x7f150000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:58.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.139+0000 7f14fa7fc700 1 -- 172.21.15.189:0/1626445982 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1500011790 con 0x7f150406abb0 2024-04-06T12:24:58.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.139+0000 7f14fa7fc700 1 -- 172.21.15.189:0/1626445982 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15000118f0 con 0x7f150406abb0 2024-04-06T12:24:58.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.139+0000 7f150bc25700 1 -- 172.21.15.189:0/1626445982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15040c5af0 con 0x7f150406abb0 2024-04-06T12:24:58.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.139+0000 7f14fa7fc700 1 -- 172.21.15.189:0/1626445982 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1500011ca0 con 0x7f150406abb0 2024-04-06T12:24:58.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.139+0000 7f150bc25700 1 -- 172.21.15.189:0/1626445982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15040c60b0 con 0x7f150406abb0 2024-04-06T12:24:58.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.140+0000 7f150bc25700 1 -- 172.21.15.189:0/1626445982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15040b38c0 con 0x7f150406abb0 2024-04-06T12:24:58.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.141+0000 7f14fa7fc700 1 -- 172.21.15.189:0/1626445982 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f150001c460 con 0x7f150406abb0 2024-04-06T12:24:58.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.141+0000 7f14fa7fc700 1 --2- 172.21.15.189:0/1626445982 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f14e806adb0 0x7f14e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:58.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.141+0000 7f15099c1700 1 --2- 172.21.15.189:0/1626445982 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f14e806adb0 0x7f14e806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:58.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.141+0000 7f14fa7fc700 1 -- 172.21.15.189:0/1626445982 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f15000890c0 con 0x7f150406abb0 2024-04-06T12:24:58.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.141+0000 7f15099c1700 1 --2- 172.21.15.189:0/1626445982 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f14e806adb0 0x7f14e806d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f14f4009c40 tx=0x7f14f4008030 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:58.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.146+0000 7f14fa7fc700 1 -- 172.21.15.189:0/1626445982 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f15000543a0 con 0x7f150406abb0 2024-04-06T12:24:58.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.415+0000 7f150bc25700 1 -- 172.21.15.189:0/1626445982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f15040c6240 con 0x7f150406abb0 2024-04-06T12:24:58.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.416+0000 7f14fa7fc700 1 -- 172.21.15.189:0/1626445982 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f15000579c0 con 0x7f150406abb0 2024-04-06T12:24:58.414 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:58.414 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T12:24:58.415 INFO:teuthology.orchestra.run.smithi189.stdout: "auth_dump": [ 2024-04-06T12:24:58.415 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:58.415 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.a", 2024-04-06T12:24:58.415 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmTjYMNBAA5voWGV4Pa7pE27VogrdipQ==", 2024-04-06T12:24:58.415 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:24:58.415 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:24:58.415 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:24:58.415 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:24:58.415 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:24:58.415 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.415 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:24:58.415 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:58.415 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.b", 2024-04-06T12:24:58.415 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm9A1OBRAAnHzubCvRUX8UdW8hmDzKbQ==", 2024-04-06T12:24:58.416 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:24:58.416 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:24:58.416 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:24:58.416 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:24:58.416 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:24:58.416 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.416 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:24:58.416 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:58.416 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.c", 2024-04-06T12:24:58.416 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmz0hhOxAAD3/GINNmGoFk+oPyiHHJXg==", 2024-04-06T12:24:58.416 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:24:58.416 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:24:58.416 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:24:58.416 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:24:58.416 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:24:58.416 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.417 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:24:58.417 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:58.417 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.d", 2024-04-06T12:24:58.417 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmhoAeDRAAoQC0vicWumBRTwpi3PtYaw==", 2024-04-06T12:24:58.417 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:24:58.417 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:24:58.417 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:24:58.417 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:24:58.417 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:24:58.417 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.417 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:24:58.417 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:58.417 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.0", 2024-04-06T12:24:58.417 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDkOxFmkxLWJhAAFXewCNOFX7dhAZVgUmewpQ==", 2024-04-06T12:24:58.417 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:24:58.418 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:24:58.418 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:24:58.418 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:24:58.418 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.418 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:24:58.418 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:58.418 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.1", 2024-04-06T12:24:58.418 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDnOxFmHYqWJRAABlsVCiv3e7YJMbfodJTWag==", 2024-04-06T12:24:58.418 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:24:58.418 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:24:58.418 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:24:58.418 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:24:58.418 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.418 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:24:58.419 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:58.419 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.2", 2024-04-06T12:24:58.419 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDqOxFmkhcvJBAA8NMEpHVGbGALjV9RISKP8g==", 2024-04-06T12:24:58.419 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:24:58.419 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:24:58.419 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:24:58.419 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:24:58.419 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.419 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:24:58.419 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:58.419 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.3", 2024-04-06T12:24:58.419 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDtOxFmDGzhIxAAamBoAbE8+lAel4os67meyQ==", 2024-04-06T12:24:58.419 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:24:58.419 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:24:58.419 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:24:58.419 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:24:58.420 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.420 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:24:58.420 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:58.420 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.4", 2024-04-06T12:24:58.420 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDzOxFmobZDJxAA9l8wdS81Eyw6+17I5ac6bw==", 2024-04-06T12:24:58.420 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:24:58.420 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:24:58.420 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:24:58.420 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:24:58.420 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.420 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:24:58.421 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:58.421 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.5", 2024-04-06T12:24:58.421 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD2OxFmi+ruJRAASlQ0o9jX/wA7U5AosnL+Wg==", 2024-04-06T12:24:58.421 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:24:58.421 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:24:58.421 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:24:58.421 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:24:58.421 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.421 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:24:58.421 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:58.421 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.6", 2024-04-06T12:24:58.421 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD5OxFmbwuuJBAA0wqJRmnJyskQPZJ7/LnpvA==", 2024-04-06T12:24:58.421 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:24:58.421 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:24:58.422 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:24:58.422 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:24:58.422 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.422 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:24:58.422 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:58.422 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.7", 2024-04-06T12:24:58.422 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD8OxFmDmRgIxAAzyW9AGAbmb67ZvB5qunwLQ==", 2024-04-06T12:24:58.422 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:24:58.422 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:24:58.422 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:24:58.422 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:24:58.422 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.422 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:24:58.422 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:58.422 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.0", 2024-04-06T12:24:58.423 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm0ByJERAAoEV8Yfc09+Go6u95XafElQ==", 2024-04-06T12:24:58.423 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:24:58.423 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:24:58.423 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow r", 2024-04-06T12:24:58.423 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-06T12:24:58.423 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.423 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:24:58.423 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:58.423 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.1", 2024-04-06T12:24:58.423 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmSV5lGBAAavDEo2LGy0IHpWSAuitHXg==", 2024-04-06T12:24:58.423 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:24:58.423 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:24:58.423 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:24:58.423 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:24:58.424 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:24:58.424 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.424 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:24:58.424 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:58.424 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.2", 2024-04-06T12:24:58.424 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm/Gz5FBAA3xX/rsAyWZ1JoUVlQlBIvQ==", 2024-04-06T12:24:58.424 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:24:58.424 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:24:58.424 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:24:58.424 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:24:58.424 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:24:58.424 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.424 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:24:58.424 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:58.424 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.3", 2024-04-06T12:24:58.425 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmzra7GxAADbnzIvWnRTo65LvPwtxWrg==", 2024-04-06T12:24:58.425 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:24:58.425 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:24:58.425 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:24:58.425 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:24:58.425 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:24:58.425 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.425 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:24:58.425 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:58.425 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.admin", 2024-04-06T12:24:58.425 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmxG+wCBAANg5kt6TOUQdj3/X0Mqgx1g==", 2024-04-06T12:24:58.425 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:24:58.425 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:24:58.425 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:24:58.425 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:24:58.426 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:24:58.426 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.426 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:24:58.426 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:58.426 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.x", 2024-04-06T12:24:58.426 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmPLyBMBAAYfhkI9Z1CPlf1Tijm+L7xA==", 2024-04-06T12:24:58.426 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:24:58.426 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:24:58.426 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T12:24:58.426 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:24:58.426 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.426 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:24:58.426 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:58.426 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.y", 2024-04-06T12:24:58.426 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFm1nHJHhAAQWRe8cU8duWuUJiYaDeTaQ==", 2024-04-06T12:24:58.427 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:24:58.427 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:24:58.427 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T12:24:58.427 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:24:58.427 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.427 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:58.427 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-04-06T12:24:58.427 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T12:24:58.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.418+0000 7f150bc25700 1 -- 172.21.15.189:0/1626445982 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f14e806adb0 msgr2=0x7f14e806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:58.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.418+0000 7f150bc25700 1 --2- 172.21.15.189:0/1626445982 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f14e806adb0 0x7f14e806d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f14f4009c40 tx=0x7f14f4008030 comp rx=0 tx=0).stop 2024-04-06T12:24:58.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.418+0000 7f150bc25700 1 -- 172.21.15.189:0/1626445982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150406abb0 msgr2=0x7f15040c5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:58.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.418+0000 7f150bc25700 1 --2- 172.21.15.189:0/1626445982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150406abb0 0x7f15040c5030 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f150000c4d0 tx=0x7f150000c890 comp rx=0 tx=0).stop 2024-04-06T12:24:58.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.418+0000 7f150bc25700 1 -- 172.21.15.189:0/1626445982 shutdown_connections 2024-04-06T12:24:58.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.418+0000 7f150bc25700 1 --2- 172.21.15.189:0/1626445982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150415d610 0x7f15040ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.418+0000 7f150bc25700 1 --2- 172.21.15.189:0/1626445982 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f14e806adb0 0x7f14e806d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.418+0000 7f150bc25700 1 --2- 172.21.15.189:0/1626445982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f150406a730 0x7f15040c68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.418+0000 7f150bc25700 1 --2- 172.21.15.189:0/1626445982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150406abb0 0x7f15040c5030 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.418+0000 7f150bc25700 1 -- 172.21.15.189:0/1626445982 >> 172.21.15.189:0/1626445982 conn(0x7f15040d00b0 msgr2=0x7f15040af800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:58.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.418+0000 7f150bc25700 1 -- 172.21.15.189:0/1626445982 shutdown_connections 2024-04-06T12:24:58.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.418+0000 7f150bc25700 1 -- 172.21.15.189:0/1626445982 wait complete. 2024-04-06T12:24:58.430 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:24:58.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.555+0000 7f359cd2e700 1 Processor -- start 2024-04-06T12:24:58.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.555+0000 7f359cd2e700 1 -- start start 2024-04-06T12:24:58.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.556+0000 7f359cd2e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f359815a550 0x7f35980cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:58.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.556+0000 7f359cd2e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f35980b4d70 0x7f35980b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:58.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.556+0000 7f359cd2e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35980cdbe0 con 0x7f35980cf900 2024-04-06T12:24:58.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.556+0000 7f359cd2e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f359815a940 con 0x7f35980b4d70 2024-04-06T12:24:58.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.556+0000 7f359cd2e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359815ac00 con 0x7f359815a550 2024-04-06T12:24:58.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.556+0000 7f359659c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f359815a550 0x7f35980cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:58.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.556+0000 7f3596d9d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f35980b4d70 0x7f35980b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:58.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.556+0000 7f3595d9b700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f35980cf900 0x7f35980b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:39104/0 (socket says 172.21.15.189:39104) 2024-04-06T12:24:58.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.556+0000 7f359659c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f359815a550 0x7f35980cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59990/0 (socket says 172.21.15.189:59990) 2024-04-06T12:24:58.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.556+0000 7f3596d9d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f35980b4d70 0x7f35980b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:52328/0 (socket says 172.21.15.189:52328) 2024-04-06T12:24:58.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.556+0000 7f3595d9b700 1 -- 172.21.15.189:0/922040151 learned_addr learned my addr 172.21.15.189:0/922040151 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:58.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.556+0000 7f359559a700 1 -- 172.21.15.189:0/922040151 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919972364 0 0) 0x7f35980cdbe0 con 0x7f35980cf900 2024-04-06T12:24:58.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.556+0000 7f359559a700 1 -- 172.21.15.189:0/922040151 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35800035b0 con 0x7f35980cf900 2024-04-06T12:24:58.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.556+0000 7f359659c700 1 -- 172.21.15.189:0/922040151 >> v2:172.21.15.192:3300/0 conn(0x7f35980b4d70 msgr2=0x7f35980b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:58.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.556+0000 7f359659c700 1 --2- 172.21.15.189:0/922040151 >> v2:172.21.15.192:3300/0 conn(0x7f35980b4d70 0x7f35980b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.556+0000 7f359659c700 1 -- 172.21.15.189:0/922040151 >> v1:172.21.15.192:6789/0 conn(0x7f35980cf900 legacy=0x7f35980b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:58.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.556+0000 7f359659c700 1 -- 172.21.15.189:0/922040151 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359815aee0 con 0x7f359815a550 2024-04-06T12:24:58.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.556+0000 7f3596d9d700 1 --2- 172.21.15.189:0/922040151 >> v2:172.21.15.192:3300/0 conn(0x7f35980b4d70 0x7f35980b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:58.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.557+0000 7f359659c700 1 --2- 172.21.15.189:0/922040151 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f359815a550 0x7f35980cf3c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3588009840 tx=0x7f3588009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e44329a359bc2e64 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:58.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.557+0000 7f359559a700 1 -- 172.21.15.189:0/922040151 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3588011830 con 0x7f359815a550 2024-04-06T12:24:58.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.557+0000 7f359559a700 1 -- 172.21.15.189:0/922040151 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3588011990 con 0x7f359815a550 2024-04-06T12:24:58.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.557+0000 7f359559a700 1 -- 172.21.15.189:0/922040151 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3588011d20 con 0x7f359815a550 2024-04-06T12:24:58.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.557+0000 7f359cd2e700 1 -- 172.21.15.189:0/922040151 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f359815a550 msgr2=0x7f35980cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:58.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.557+0000 7f359cd2e700 1 --2- 172.21.15.189:0/922040151 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f359815a550 0x7f35980cf3c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3588009840 tx=0x7f3588009b50 comp rx=0 tx=0).stop 2024-04-06T12:24:58.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.557+0000 7f359cd2e700 1 -- 172.21.15.189:0/922040151 shutdown_connections 2024-04-06T12:24:58.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.557+0000 7f359cd2e700 1 --2- 172.21.15.189:0/922040151 >> v2:172.21.15.192:3300/0 conn(0x7f35980b4d70 0x7f35980b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.557+0000 7f359cd2e700 1 --2- 172.21.15.189:0/922040151 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f359815a550 0x7f35980cf3c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.557+0000 7f359cd2e700 1 -- 172.21.15.189:0/922040151 >> 172.21.15.189:0/922040151 conn(0x7f35980d00b0 msgr2=0x7f35980d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:58.558 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.557+0000 7f359cd2e700 1 -- 172.21.15.189:0/922040151 shutdown_connections 2024-04-06T12:24:58.558 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.557+0000 7f359cd2e700 1 -- 172.21.15.189:0/922040151 wait complete. 2024-04-06T12:24:58.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.559+0000 7f359cd2e700 1 Processor -- start 2024-04-06T12:24:58.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.559+0000 7f359cd2e700 1 -- start start 2024-04-06T12:24:58.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.559+0000 7f359cd2e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35980b4d70 0x7f35980cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:58.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.559+0000 7f359cd2e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35980cf900 0x7f359814c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:58.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.559+0000 7f359cd2e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35980caeb0 0x7f359814f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:58.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.559+0000 7f359cd2e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35980ca940 con 0x7f35980caeb0 2024-04-06T12:24:58.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.560+0000 7f359cd2e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35980cb360 con 0x7f35980b4d70 2024-04-06T12:24:58.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.560+0000 7f359cd2e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35980cb570 con 0x7f35980cf900 2024-04-06T12:24:58.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.560+0000 7f3596d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35980caeb0 0x7f359814f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:58.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.560+0000 7f3596d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35980caeb0 0x7f359814f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42948/0 (socket says 172.21.15.189:42948) 2024-04-06T12:24:58.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.560+0000 7f3596d9d700 1 -- 172.21.15.189:0/1229786991 learned_addr learned my addr 172.21.15.189:0/1229786991 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:58.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.560+0000 7f359659c700 1 --2- 172.21.15.189:0/1229786991 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35980b4d70 0x7f35980cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:58.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.560+0000 7f3595d9b700 1 --2- 172.21.15.189:0/1229786991 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35980cf900 0x7f359814c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:58.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.560+0000 7f3596d9d700 1 -- 172.21.15.189:0/1229786991 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35980cf900 msgr2=0x7f359814c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:58.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.560+0000 7f3596d9d700 1 --2- 172.21.15.189:0/1229786991 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35980cf900 0x7f359814c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.560+0000 7f3596d9d700 1 -- 172.21.15.189:0/1229786991 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35980b4d70 msgr2=0x7f35980cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:58.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.560+0000 7f3596d9d700 1 --2- 172.21.15.189:0/1229786991 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35980b4d70 0x7f35980cc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.560+0000 7f3596d9d700 1 -- 172.21.15.189:0/1229786991 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3590009a70 con 0x7f35980caeb0 2024-04-06T12:24:58.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.560+0000 7f3596d9d700 1 --2- 172.21.15.189:0/1229786991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35980caeb0 0x7f359814f840 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f359000efb0 tx=0x7f359000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:58.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.560+0000 7f3586ffd700 1 -- 172.21.15.189:0/1229786991 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359000cc90 con 0x7f35980caeb0 2024-04-06T12:24:58.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.560+0000 7f3586ffd700 1 -- 172.21.15.189:0/1229786991 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359000f040 con 0x7f35980caeb0 2024-04-06T12:24:58.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.560+0000 7f359cd2e700 1 -- 172.21.15.189:0/1229786991 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35880094a0 con 0x7f35980caeb0 2024-04-06T12:24:58.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.560+0000 7f3586ffd700 1 -- 172.21.15.189:0/1229786991 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3590014480 con 0x7f35980caeb0 2024-04-06T12:24:58.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.560+0000 7f359cd2e700 1 -- 172.21.15.189:0/1229786991 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35980cbb30 con 0x7f35980caeb0 2024-04-06T12:24:58.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.561+0000 7f359cd2e700 1 -- 172.21.15.189:0/1229786991 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3598067bb0 con 0x7f35980caeb0 2024-04-06T12:24:58.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.562+0000 7f3586ffd700 1 -- 172.21.15.189:0/1229786991 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f35900145e0 con 0x7f35980caeb0 2024-04-06T12:24:58.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.562+0000 7f3586ffd700 1 --2- 172.21.15.189:0/1229786991 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f357806adb0 0x7f357806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:58.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.562+0000 7f3586ffd700 1 -- 172.21.15.189:0/1229786991 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f3590089fb0 con 0x7f35980caeb0 2024-04-06T12:24:58.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.562+0000 7f359659c700 1 --2- 172.21.15.189:0/1229786991 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f357806adb0 0x7f357806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:58.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.563+0000 7f359659c700 1 --2- 172.21.15.189:0/1229786991 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f357806adb0 0x7f357806d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3588000c00 tx=0x7f3588008030 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:58.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.567+0000 7f3586ffd700 1 -- 172.21.15.189:0/1229786991 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f35900bd650 con 0x7f35980caeb0 2024-04-06T12:24:58.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.762+0000 7f359cd2e700 1 -- 172.21.15.189:0/1229786991 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35980b34d0 con 0x7f35980caeb0 2024-04-06T12:24:58.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.763+0000 7f3586ffd700 1 -- 172.21.15.189:0/1229786991 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+12642 (secure 0 0 0) 0x7f3590018090 con 0x7f35980caeb0 2024-04-06T12:24:58.761 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:58.761 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":63,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:24:57.986128+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:24:58.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.765+0000 7f359cd2e700 1 -- 172.21.15.189:0/1229786991 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f357806adb0 msgr2=0x7f357806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:58.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.765+0000 7f359cd2e700 1 --2- 172.21.15.189:0/1229786991 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f357806adb0 0x7f357806d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3588000c00 tx=0x7f3588008030 comp rx=0 tx=0).stop 2024-04-06T12:24:58.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.765+0000 7f359cd2e700 1 -- 172.21.15.189:0/1229786991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35980caeb0 msgr2=0x7f359814f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:58.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.765+0000 7f359cd2e700 1 --2- 172.21.15.189:0/1229786991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35980caeb0 0x7f359814f840 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f359000efb0 tx=0x7f359000c3a0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.765+0000 7f359cd2e700 1 -- 172.21.15.189:0/1229786991 shutdown_connections 2024-04-06T12:24:58.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.765+0000 7f359cd2e700 1 --2- 172.21.15.189:0/1229786991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35980caeb0 0x7f359814f840 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.765+0000 7f359cd2e700 1 --2- 172.21.15.189:0/1229786991 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f357806adb0 0x7f357806d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.765+0000 7f359cd2e700 1 --2- 172.21.15.189:0/1229786991 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35980b4d70 0x7f35980cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.765+0000 7f359cd2e700 1 --2- 172.21.15.189:0/1229786991 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35980cf900 0x7f359814c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.765+0000 7f359cd2e700 1 -- 172.21.15.189:0/1229786991 >> 172.21.15.189:0/1229786991 conn(0x7f35980d00b0 msgr2=0x7f359806ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:58.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.765+0000 7f359cd2e700 1 -- 172.21.15.189:0/1229786991 shutdown_connections 2024-04-06T12:24:58.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.765+0000 7f359cd2e700 1 -- 172.21.15.189:0/1229786991 wait complete. 2024-04-06T12:24:58.776 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:24:58.776 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:24:58.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.904+0000 7fbf752b1700 1 Processor -- start 2024-04-06T12:24:58.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.904+0000 7fbf752b1700 1 -- start start 2024-04-06T12:24:58.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.905+0000 7fbf752b1700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbf7015d590 0x7fbf700b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:58.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.905+0000 7fbf752b1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf700b4ed0 0x7fbf700b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:58.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.905+0000 7fbf752b1700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf700b5cc0 con 0x7fbf700b6580 2024-04-06T12:24:58.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.905+0000 7fbf752b1700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf700b5890 con 0x7fbf7015d590 2024-04-06T12:24:58.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.905+0000 7fbf752b1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf700d1500 con 0x7fbf700b4ed0 2024-04-06T12:24:58.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.905+0000 7fbf6e7fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf700b4ed0 0x7fbf700b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:58.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.905+0000 7fbf6effd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbf7015d590 0x7fbf700b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:58.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.905+0000 7fbf6f7fe700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fbf700b6580 0x7fbf700b6a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:39112/0 (socket says 172.21.15.189:39112) 2024-04-06T12:24:58.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.905+0000 7fbf6e7fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf700b4ed0 0x7fbf700b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:60010/0 (socket says 172.21.15.189:60010) 2024-04-06T12:24:58.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.905+0000 7fbf6f7fe700 1 -- 172.21.15.189:0/1496732079 learned_addr learned my addr 172.21.15.189:0/1496732079 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:58.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.905+0000 7fbf6effd700 1 -- 172.21.15.189:0/1496732079 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf700b4ed0 msgr2=0x7fbf700b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:58.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.905+0000 7fbf6effd700 1 --2- 172.21.15.189:0/1496732079 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf700b4ed0 0x7fbf700b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.905+0000 7fbf6effd700 1 -- 172.21.15.189:0/1496732079 >> v1:172.21.15.192:6789/0 conn(0x7fbf700b6580 legacy=0x7fbf700b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:58.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.905+0000 7fbf6effd700 1 -- 172.21.15.189:0/1496732079 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf700d17e0 con 0x7fbf7015d590 2024-04-06T12:24:58.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.905+0000 7fbf6dffb700 1 -- 172.21.15.189:0/1496732079 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992655356 0 0) 0x7fbf700b5cc0 con 0x7fbf700b6580 2024-04-06T12:24:58.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.905+0000 7fbf6e7fc700 1 --2- 172.21.15.189:0/1496732079 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf700b4ed0 0x7fbf700b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:24:58.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.906+0000 7fbf6effd700 1 --2- 172.21.15.189:0/1496732079 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf7015d590 0x7fbf700b8d70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009870 tx=0x7fbf60009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=423f1218ec7f0f21 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:58.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.906+0000 7fbf6dffb700 1 -- 172.21.15.189:0/1496732079 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf60011830 con 0x7fbf7015d590 2024-04-06T12:24:58.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.906+0000 7fbf6dffb700 1 -- 172.21.15.189:0/1496732079 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf60011990 con 0x7fbf7015d590 2024-04-06T12:24:58.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.906+0000 7fbf6dffb700 1 -- 172.21.15.189:0/1496732079 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf60011d20 con 0x7fbf7015d590 2024-04-06T12:24:58.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.906+0000 7fbf752b1700 1 -- 172.21.15.189:0/1496732079 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf7015d590 msgr2=0x7fbf700b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:58.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.906+0000 7fbf752b1700 1 --2- 172.21.15.189:0/1496732079 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf7015d590 0x7fbf700b8d70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009870 tx=0x7fbf60009b80 comp rx=0 tx=0).stop 2024-04-06T12:24:58.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.906+0000 7fbf752b1700 1 -- 172.21.15.189:0/1496732079 shutdown_connections 2024-04-06T12:24:58.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.906+0000 7fbf752b1700 1 --2- 172.21.15.189:0/1496732079 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf7015d590 0x7fbf700b8d70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.906+0000 7fbf752b1700 1 --2- 172.21.15.189:0/1496732079 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf700b4ed0 0x7fbf700b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.906+0000 7fbf752b1700 1 -- 172.21.15.189:0/1496732079 >> 172.21.15.189:0/1496732079 conn(0x7fbf7006bbc0 msgr2=0x7fbf7006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:58.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.906+0000 7fbf752b1700 1 -- 172.21.15.189:0/1496732079 shutdown_connections 2024-04-06T12:24:58.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.906+0000 7fbf752b1700 1 -- 172.21.15.189:0/1496732079 wait complete. 2024-04-06T12:24:58.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.908+0000 7fbf752b1700 1 Processor -- start 2024-04-06T12:24:58.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.908+0000 7fbf752b1700 1 -- start start 2024-04-06T12:24:58.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.908+0000 7fbf752b1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf700b4ed0 0x7fbf700cc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:58.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.908+0000 7fbf752b1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf700b6580 0x7fbf701f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:58.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.908+0000 7fbf752b1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf7015d590 0x7fbf70159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:58.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.908+0000 7fbf752b1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7015e890 con 0x7fbf7015d590 2024-04-06T12:24:58.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.908+0000 7fbf752b1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7015d970 con 0x7fbf700b6580 2024-04-06T12:24:58.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.908+0000 7fbf752b1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7015dc10 con 0x7fbf700b4ed0 2024-04-06T12:24:58.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.908+0000 7fbf6f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf7015d590 0x7fbf70159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:58.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.909+0000 7fbf6f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf7015d590 0x7fbf70159930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42962/0 (socket says 172.21.15.189:42962) 2024-04-06T12:24:58.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.909+0000 7fbf6f7fe700 1 -- 172.21.15.189:0/2606964449 learned_addr learned my addr 172.21.15.189:0/2606964449 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:58.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.909+0000 7fbf6effd700 1 --2- 172.21.15.189:0/2606964449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf700b4ed0 0x7fbf700cc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:58.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.909+0000 7fbf6e7fc700 1 --2- 172.21.15.189:0/2606964449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf700b6580 0x7fbf701f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:58.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.909+0000 7fbf6f7fe700 1 -- 172.21.15.189:0/2606964449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf700b4ed0 msgr2=0x7fbf700cc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:58.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.909+0000 7fbf6f7fe700 1 --2- 172.21.15.189:0/2606964449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf700b4ed0 0x7fbf700cc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.909+0000 7fbf6f7fe700 1 -- 172.21.15.189:0/2606964449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf700b6580 msgr2=0x7fbf701f26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:58.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.909+0000 7fbf6f7fe700 1 --2- 172.21.15.189:0/2606964449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf700b6580 0x7fbf701f26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:58.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.909+0000 7fbf6f7fe700 1 -- 172.21.15.189:0/2606964449 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf640094a0 con 0x7fbf7015d590 2024-04-06T12:24:58.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.909+0000 7fbf6f7fe700 1 --2- 172.21.15.189:0/2606964449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf7015d590 0x7fbf70159930 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbf6800cca0 tx=0x7fbf6800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:58.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.909+0000 7fbf577fe700 1 -- 172.21.15.189:0/2606964449 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6800cea0 con 0x7fbf7015d590 2024-04-06T12:24:58.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.909+0000 7fbf577fe700 1 -- 172.21.15.189:0/2606964449 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6800ade0 con 0x7fbf7015d590 2024-04-06T12:24:58.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.909+0000 7fbf577fe700 1 -- 172.21.15.189:0/2606964449 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf68007310 con 0x7fbf7015d590 2024-04-06T12:24:58.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.909+0000 7fbf752b1700 1 -- 172.21.15.189:0/2606964449 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf600094a0 con 0x7fbf7015d590 2024-04-06T12:24:58.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.909+0000 7fbf752b1700 1 -- 172.21.15.189:0/2606964449 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf7015e1d0 con 0x7fbf7015d590 2024-04-06T12:24:58.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.910+0000 7fbf752b1700 1 -- 172.21.15.189:0/2606964449 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf70074990 con 0x7fbf7015d590 2024-04-06T12:24:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.911+0000 7fbf577fe700 1 -- 172.21.15.189:0/2606964449 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fbf68007470 con 0x7fbf7015d590 2024-04-06T12:24:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.911+0000 7fbf577fe700 1 --2- 172.21.15.189:0/2606964449 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf5c06adb0 0x7fbf5c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.911+0000 7fbf577fe700 1 -- 172.21.15.189:0/2606964449 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fbf68088e80 con 0x7fbf7015d590 2024-04-06T12:24:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.911+0000 7fbf6effd700 1 --2- 172.21.15.189:0/2606964449 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf5c06adb0 0x7fbf5c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.911+0000 7fbf6effd700 1 --2- 172.21.15.189:0/2606964449 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf5c06adb0 0x7fbf5c06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbf701f63d0 tx=0x7fbf6001c040 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:58.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:58.915+0000 7fbf577fe700 1 -- 172.21.15.189:0/2606964449 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbf680541f0 con 0x7fbf7015d590 2024-04-06T12:24:59.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.109+0000 7fbf752b1700 1 -- 172.21.15.189:0/2606964449 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbf700d30a0 con 0x7fbf7015d590 2024-04-06T12:24:59.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.109+0000 7fbf577fe700 1 -- 172.21.15.189:0/2606964449 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbf68018020 con 0x7fbf7015d590 2024-04-06T12:24:59.108 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:24:59.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.112+0000 7fbf752b1700 1 -- 172.21.15.189:0/2606964449 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf5c06adb0 msgr2=0x7fbf5c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:59.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.112+0000 7fbf752b1700 1 --2- 172.21.15.189:0/2606964449 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf5c06adb0 0x7fbf5c06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbf701f63d0 tx=0x7fbf6001c040 comp rx=0 tx=0).stop 2024-04-06T12:24:59.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.112+0000 7fbf752b1700 1 -- 172.21.15.189:0/2606964449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf7015d590 msgr2=0x7fbf70159930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:59.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.112+0000 7fbf752b1700 1 --2- 172.21.15.189:0/2606964449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf7015d590 0x7fbf70159930 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbf6800cca0 tx=0x7fbf6800a340 comp rx=0 tx=0).stop 2024-04-06T12:24:59.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.112+0000 7fbf752b1700 1 -- 172.21.15.189:0/2606964449 shutdown_connections 2024-04-06T12:24:59.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.112+0000 7fbf752b1700 1 --2- 172.21.15.189:0/2606964449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf7015d590 0x7fbf70159930 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:59.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.112+0000 7fbf752b1700 1 --2- 172.21.15.189:0/2606964449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf700b4ed0 0x7fbf700cc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:59.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.112+0000 7fbf752b1700 1 --2- 172.21.15.189:0/2606964449 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf5c06adb0 0x7fbf5c06d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:59.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.112+0000 7fbf752b1700 1 --2- 172.21.15.189:0/2606964449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf700b6580 0x7fbf701f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:59.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.112+0000 7fbf752b1700 1 -- 172.21.15.189:0/2606964449 >> 172.21.15.189:0/2606964449 conn(0x7fbf7006bbc0 msgr2=0x7fbf700cd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:59.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.112+0000 7fbf752b1700 1 -- 172.21.15.189:0/2606964449 shutdown_connections 2024-04-06T12:24:59.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.112+0000 7fbf752b1700 1 -- 172.21.15.189:0/2606964449 wait complete. 2024-04-06T12:24:59.122 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-06T12:24:59.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.245+0000 7fb322177700 1 Processor -- start 2024-04-06T12:24:59.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.245+0000 7fb322177700 1 -- start start 2024-04-06T12:24:59.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.246+0000 7fb322177700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb31c15e8a0 0x7fb31c0cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:59.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.246+0000 7fb322177700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb31c15fa70 0x7fb31c06aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:59.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.246+0000 7fb322177700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31c0669d0 con 0x7fb31c15eee0 2024-04-06T12:24:59.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.246+0000 7fb322177700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb31c06ede0 con 0x7fb31c15fa70 2024-04-06T12:24:59.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.246+0000 7fb322177700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c15ec80 con 0x7fb31c15e8a0 2024-04-06T12:24:59.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.246+0000 7fb31affd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb31c15eee0 0x7fb31c15f360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49476/0 (socket says 172.21.15.189:49476) 2024-04-06T12:24:59.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.246+0000 7fb31affd700 1 -- 172.21.15.189:0/2623366885 learned_addr learned my addr 172.21.15.189:0/2623366885 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:59.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.246+0000 7fb31b7fe700 1 --2- 172.21.15.189:0/2623366885 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb31c15e8a0 0x7fb31c0cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:59.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.246+0000 7fb31bfff700 1 --2- 172.21.15.189:0/2623366885 >> v2:172.21.15.192:3300/0 conn(0x7fb31c15fa70 0x7fb31c06aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:59.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.246+0000 7fb31a7fc700 1 -- 172.21.15.189:0/2623366885 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177230453 0 0) 0x7fb31c0669d0 con 0x7fb31c15eee0 2024-04-06T12:24:59.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.246+0000 7fb31a7fc700 1 -- 172.21.15.189:0/2623366885 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3080035b0 con 0x7fb31c15eee0 2024-04-06T12:24:59.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31a7fc700 1 -- 172.21.15.189:0/2623366885 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3620038499 0 0) 0x7fb3080035b0 con 0x7fb31c15eee0 2024-04-06T12:24:59.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31a7fc700 1 -- 172.21.15.189:0/2623366885 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb31c15e8a0 msgr2=0x7fb31c0cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:59.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31a7fc700 1 --2- 172.21.15.189:0/2623366885 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb31c15e8a0 0x7fb31c0cbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:59.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31a7fc700 1 -- 172.21.15.189:0/2623366885 >> v2:172.21.15.192:3300/0 conn(0x7fb31c15fa70 msgr2=0x7fb31c06aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:59.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31a7fc700 1 --2- 172.21.15.189:0/2623366885 >> v2:172.21.15.192:3300/0 conn(0x7fb31c15fa70 0x7fb31c06aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:59.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31a7fc700 1 -- 172.21.15.189:0/2623366885 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c0666f0 con 0x7fb31c15eee0 2024-04-06T12:24:59.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31a7fc700 1 -- 172.21.15.189:0/2623366885 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fb30c001670 con 0x7fb31c15eee0 2024-04-06T12:24:59.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31a7fc700 1 -- 172.21.15.189:0/2623366885 >> v1:172.21.15.189:6789/0 conn(0x7fb31c15eee0 legacy=0x7fb31c15f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:59.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31a7fc700 1 --2- 172.21.15.189:0/2623366885 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb31c15e8a0 0x7fb308003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:59.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31a7fc700 1 --2- 172.21.15.189:0/2623366885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb308003f80 0x7fb308004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:59.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31a7fc700 1 --2- 172.21.15.189:0/2623366885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb308004d90 0x7fb308005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:59.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31a7fc700 1 -- 172.21.15.189:0/2623366885 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c06ede0 con 0x7fb308003f80 2024-04-06T12:24:59.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31a7fc700 1 -- 172.21.15.189:0/2623366885 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c15ec80 con 0x7fb308004d90 2024-04-06T12:24:59.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31a7fc700 1 -- 172.21.15.189:0/2623366885 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb30800ba60 con 0x7fb31c15e8a0 2024-04-06T12:24:59.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31affd700 1 --2- 172.21.15.189:0/2623366885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb308003f80 0x7fb308004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:59.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31b7fe700 1 --2- 172.21.15.189:0/2623366885 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb31c15e8a0 0x7fb308003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:59.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31bfff700 1 --2- 172.21.15.189:0/2623366885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb308004d90 0x7fb308005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:59.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31affd700 1 -- 172.21.15.189:0/2623366885 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb31c15e8a0 msgr2=0x7fb308003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:59.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31affd700 1 --2- 172.21.15.189:0/2623366885 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb31c15e8a0 0x7fb308003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:59.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31affd700 1 -- 172.21.15.189:0/2623366885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb308004d90 msgr2=0x7fb308005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:24:59.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31affd700 1 --2- 172.21.15.189:0/2623366885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb308004d90 0x7fb308005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:59.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31affd700 1 -- 172.21.15.189:0/2623366885 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3080035b0 con 0x7fb308003f80 2024-04-06T12:24:59.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31affd700 1 --2- 172.21.15.189:0/2623366885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb308003f80 0x7fb308004850 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00c270 tx=0x7fb308005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2e88e28b7fc618 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:59.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.247+0000 7fb31a7fc700 1 -- 172.21.15.189:0/2623366885 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c007c10 con 0x7fb308003f80 2024-04-06T12:24:59.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.248+0000 7fb31a7fc700 1 -- 172.21.15.189:0/2623366885 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30c00cdd0 con 0x7fb308003f80 2024-04-06T12:24:59.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.248+0000 7fb31a7fc700 1 -- 172.21.15.189:0/2623366885 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c005ba0 con 0x7fb308003f80 2024-04-06T12:24:59.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.248+0000 7fb322177700 1 -- 172.21.15.189:0/2623366885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb308003f80 msgr2=0x7fb308004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:59.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.248+0000 7fb322177700 1 --2- 172.21.15.189:0/2623366885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb308003f80 0x7fb308004850 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00c270 tx=0x7fb308005880 comp rx=0 tx=0).stop 2024-04-06T12:24:59.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.248+0000 7fb31bfff700 1 -- 172.21.15.189:0/2623366885 reap_dead start 2024-04-06T12:24:59.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.248+0000 7fb322177700 1 -- 172.21.15.189:0/2623366885 shutdown_connections 2024-04-06T12:24:59.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.248+0000 7fb322177700 1 -- 172.21.15.189:0/2623366885 >> 172.21.15.189:0/2623366885 conn(0x7fb31c0cc290 msgr2=0x7fb31c0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:59.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.248+0000 7fb322177700 1 -- 172.21.15.189:0/2623366885 shutdown_connections 2024-04-06T12:24:59.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.248+0000 7fb322177700 1 -- 172.21.15.189:0/2623366885 wait complete. 2024-04-06T12:24:59.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.249+0000 7fb322177700 1 Processor -- start 2024-04-06T12:24:59.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.250+0000 7fb322177700 1 -- start start 2024-04-06T12:24:59.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.250+0000 7fb322177700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb308003f80 0x7fb31c0c32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:59.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.250+0000 7fb322177700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb31c0c71b0 0x7fb31c0af810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:59.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.250+0000 7fb322177700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb31c0afd50 0x7fb31c0c8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:59.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.250+0000 7fb322177700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c0b01d0 con 0x7fb31c0c71b0 2024-04-06T12:24:59.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.250+0000 7fb322177700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c0ae400 con 0x7fb308003f80 2024-04-06T12:24:59.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.250+0000 7fb322177700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c0c9ba0 con 0x7fb31c0afd50 2024-04-06T12:24:59.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.250+0000 7fb31affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb31c0c71b0 0x7fb31c0af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:59.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.250+0000 7fb31affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb31c0c71b0 0x7fb31c0af810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42980/0 (socket says 172.21.15.189:42980) 2024-04-06T12:24:59.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.250+0000 7fb31affd700 1 -- 172.21.15.189:0/2251656517 learned_addr learned my addr 172.21.15.189:0/2251656517 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:59.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.250+0000 7fb31bfff700 1 --2- 172.21.15.189:0/2251656517 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb31c0afd50 0x7fb31c0c8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:59.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.251+0000 7fb31b7fe700 1 --2- 172.21.15.189:0/2251656517 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb308003f80 0x7fb31c0c32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:59.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.251+0000 7fb31affd700 1 -- 172.21.15.189:0/2251656517 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb31c0afd50 msgr2=0x7fb31c0c8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:59.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.251+0000 7fb31affd700 1 --2- 172.21.15.189:0/2251656517 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb31c0afd50 0x7fb31c0c8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:59.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.251+0000 7fb31affd700 1 -- 172.21.15.189:0/2251656517 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb308003f80 msgr2=0x7fb31c0c32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:59.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.251+0000 7fb31affd700 1 --2- 172.21.15.189:0/2251656517 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb308003f80 0x7fb31c0c32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:59.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.251+0000 7fb31affd700 1 -- 172.21.15.189:0/2251656517 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c0c98c0 con 0x7fb31c0c71b0 2024-04-06T12:24:59.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.251+0000 7fb31affd700 1 --2- 172.21.15.189:0/2251656517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb31c0c71b0 0x7fb31c0af810 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00c860 tx=0x7fb30c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:59.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.251+0000 7fb313fff700 1 -- 172.21.15.189:0/2251656517 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c001640 con 0x7fb31c0c71b0 2024-04-06T12:24:59.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.251+0000 7fb313fff700 1 -- 172.21.15.189:0/2251656517 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30c0070d0 con 0x7fb31c0c71b0 2024-04-06T12:24:59.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.251+0000 7fb322177700 1 -- 172.21.15.189:0/2251656517 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31c0cd480 con 0x7fb31c0c71b0 2024-04-06T12:24:59.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.251+0000 7fb313fff700 1 -- 172.21.15.189:0/2251656517 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c0074a0 con 0x7fb31c0c71b0 2024-04-06T12:24:59.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.251+0000 7fb322177700 1 -- 172.21.15.189:0/2251656517 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb31c0cda40 con 0x7fb31c0c71b0 2024-04-06T12:24:59.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.252+0000 7fb322177700 1 -- 172.21.15.189:0/2251656517 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31c072140 con 0x7fb31c0c71b0 2024-04-06T12:24:59.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.252+0000 7fb313fff700 1 -- 172.21.15.189:0/2251656517 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fb30c019030 con 0x7fb31c0c71b0 2024-04-06T12:24:59.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.253+0000 7fb313fff700 1 --2- 172.21.15.189:0/2251656517 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2fc06ade0 0x7fb2fc06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:59.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.253+0000 7fb31b7fe700 1 --2- 172.21.15.189:0/2251656517 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2fc06ade0 0x7fb2fc06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:59.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.253+0000 7fb313fff700 1 -- 172.21.15.189:0/2251656517 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fb30c08a8d0 con 0x7fb31c0c71b0 2024-04-06T12:24:59.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.253+0000 7fb31b7fe700 1 --2- 172.21.15.189:0/2251656517 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2fc06ade0 0x7fb2fc06d2a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb304009a60 tx=0x7fb304008040 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:59.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.257+0000 7fb313fff700 1 -- 172.21.15.189:0/2251656517 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb30c055bc0 con 0x7fb31c0c71b0 2024-04-06T12:24:59.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.456+0000 7fb322177700 1 -- 172.21.15.189:0/2251656517 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb31c074990 con 0x7fb31c0c71b0 2024-04-06T12:24:59.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.456+0000 7fb313fff700 1 -- 172.21.15.189:0/2251656517 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v63) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb30c0591e0 con 0x7fb31c0c71b0 2024-04-06T12:24:59.454 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:24:59.454 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T12:24:59.455 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:24:59.455 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 1, 2024-04-06T12:24:59.455 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": ".mgr" 2024-04-06T12:24:59.455 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:24:59.455 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T12:24:59.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.458+0000 7fb322177700 1 -- 172.21.15.189:0/2251656517 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2fc06ade0 msgr2=0x7fb2fc06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:59.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.458+0000 7fb322177700 1 --2- 172.21.15.189:0/2251656517 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2fc06ade0 0x7fb2fc06d2a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb304009a60 tx=0x7fb304008040 comp rx=0 tx=0).stop 2024-04-06T12:24:59.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.458+0000 7fb322177700 1 -- 172.21.15.189:0/2251656517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb31c0c71b0 msgr2=0x7fb31c0af810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:59.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.458+0000 7fb322177700 1 --2- 172.21.15.189:0/2251656517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb31c0c71b0 0x7fb31c0af810 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00c860 tx=0x7fb30c001a00 comp rx=0 tx=0).stop 2024-04-06T12:24:59.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.458+0000 7fb322177700 1 -- 172.21.15.189:0/2251656517 shutdown_connections 2024-04-06T12:24:59.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.458+0000 7fb322177700 1 --2- 172.21.15.189:0/2251656517 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb31c0afd50 0x7fb31c0c8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:59.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.458+0000 7fb322177700 1 --2- 172.21.15.189:0/2251656517 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2fc06ade0 0x7fb2fc06d2a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:59.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.458+0000 7fb322177700 1 --2- 172.21.15.189:0/2251656517 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb308003f80 0x7fb31c0c32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:59.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.458+0000 7fb322177700 1 --2- 172.21.15.189:0/2251656517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb31c0c71b0 0x7fb31c0af810 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:59.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.458+0000 7fb322177700 1 -- 172.21.15.189:0/2251656517 >> 172.21.15.189:0/2251656517 conn(0x7fb31c0cc290 msgr2=0x7fb31c0ca020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:59.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.458+0000 7fb322177700 1 -- 172.21.15.189:0/2251656517 shutdown_connections 2024-04-06T12:24:59.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.458+0000 7fb322177700 1 -- 172.21.15.189:0/2251656517 wait complete. 2024-04-06T12:24:59.469 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-06T12:24:59.469 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-06T12:24:59.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.597+0000 7f85df8f3700 1 Processor -- start 2024-04-06T12:24:59.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.597+0000 7f85df8f3700 1 -- start start 2024-04-06T12:24:59.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.598+0000 7f85df8f3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85d80b4660 0x7f85d80b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:59.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.598+0000 7f85df8f3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d80cdbe0 con 0x7f85d815a550 2024-04-06T12:24:59.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.598+0000 7f85df8f3700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d80b5000 con 0x7f85d80b2fc0 2024-04-06T12:24:59.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.598+0000 7f85df8f3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d815a940 con 0x7f85d80b4660 2024-04-06T12:24:59.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.598+0000 7f85dd68f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f85d815a550 0x7f85d80cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37230/0 (socket says 172.21.15.189:37230) 2024-04-06T12:24:59.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.598+0000 7f85dd68f700 1 -- 172.21.15.189:0/2633864503 learned_addr learned my addr 172.21.15.189:0/2633864503 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:24:59.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.598+0000 7f85dce8e700 1 --2- 172.21.15.189:0/2633864503 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85d80b4660 0x7f85d80b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:59.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.598+0000 7f85cffff700 1 -- 172.21.15.189:0/2633864503 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151084400 0 0) 0x7f85d80cdbe0 con 0x7f85d815a550 2024-04-06T12:24:59.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.598+0000 7f85cffff700 1 -- 172.21.15.189:0/2633864503 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85c40035b0 con 0x7f85d815a550 2024-04-06T12:24:59.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.598+0000 7f85cffff700 1 -- 172.21.15.189:0/2633864503 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330645428 0 0) 0x7f85d80b5000 con 0x7f85d80b2fc0 2024-04-06T12:24:59.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.598+0000 7f85cffff700 1 -- 172.21.15.189:0/2633864503 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85d80cdbe0 con 0x7f85d80b2fc0 2024-04-06T12:24:59.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.598+0000 7f85dce8e700 1 -- 172.21.15.189:0/2633864503 >> v1:172.21.15.192:6789/0 conn(0x7f85d80b2fc0 legacy=0x7f85d80b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:59.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.598+0000 7f85dce8e700 1 -- 172.21.15.189:0/2633864503 >> v1:172.21.15.189:6789/0 conn(0x7f85d815a550 legacy=0x7f85d80cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:59.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.598+0000 7f85dce8e700 1 -- 172.21.15.189:0/2633864503 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d815ac00 con 0x7f85d80b4660 2024-04-06T12:24:59.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.599+0000 7f85dce8e700 1 --2- 172.21.15.189:0/2633864503 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85d80b4660 0x7f85d80b4ac0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f85c8009870 tx=0x7f85c8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0a362cd4acccc58 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:59.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.599+0000 7f85cffff700 1 -- 172.21.15.189:0/2633864503 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8011830 con 0x7f85d80b4660 2024-04-06T12:24:59.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.599+0000 7f85cffff700 1 -- 172.21.15.189:0/2633864503 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c8011990 con 0x7f85d80b4660 2024-04-06T12:24:59.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.599+0000 7f85cffff700 1 -- 172.21.15.189:0/2633864503 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8011d20 con 0x7f85d80b4660 2024-04-06T12:24:59.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.599+0000 7f85df8f3700 1 -- 172.21.15.189:0/2633864503 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85d80b4660 msgr2=0x7f85d80b4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:59.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.599+0000 7f85df8f3700 1 --2- 172.21.15.189:0/2633864503 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85d80b4660 0x7f85d80b4ac0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f85c8009870 tx=0x7f85c8009b80 comp rx=0 tx=0).stop 2024-04-06T12:24:59.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.599+0000 7f85df8f3700 1 -- 172.21.15.189:0/2633864503 shutdown_connections 2024-04-06T12:24:59.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.599+0000 7f85df8f3700 1 --2- 172.21.15.189:0/2633864503 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85d80b4660 0x7f85d80b4ac0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:59.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.599+0000 7f85df8f3700 1 -- 172.21.15.189:0/2633864503 >> 172.21.15.189:0/2633864503 conn(0x7f85d80d00b0 msgr2=0x7f85d80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:24:59.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.599+0000 7f85df8f3700 1 -- 172.21.15.189:0/2633864503 shutdown_connections 2024-04-06T12:24:59.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.599+0000 7f85df8f3700 1 -- 172.21.15.189:0/2633864503 wait complete. 2024-04-06T12:24:59.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.601+0000 7f85df8f3700 1 Processor -- start 2024-04-06T12:24:59.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.601+0000 7f85df8f3700 1 -- start start 2024-04-06T12:24:59.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.601+0000 7f85df8f3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85d80b2fc0 0x7f85d81f6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:59.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.601+0000 7f85df8f3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85d80b4660 0x7f85d8151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:59.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.601+0000 7f85df8f3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85d806a730 0x7f85d806abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:59.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.601+0000 7f85df8f3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d81f6820 con 0x7f85d80b4660 2024-04-06T12:24:59.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.601+0000 7f85df8f3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d806b2a0 con 0x7f85d806a730 2024-04-06T12:24:59.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.601+0000 7f85df8f3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d81f6160 con 0x7f85d80b2fc0 2024-04-06T12:24:59.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.601+0000 7f85dce8e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85d80b4660 0x7f85d8151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:59.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.602+0000 7f85dce8e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85d80b4660 0x7f85d8151e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57756/0 (socket says 172.21.15.189:57756) 2024-04-06T12:24:59.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.602+0000 7f85dce8e700 1 -- 172.21.15.189:0/157402450 learned_addr learned my addr 172.21.15.189:0/157402450 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:24:59.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.602+0000 7f85dd68f700 1 --2- 172.21.15.189:0/157402450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85d80b2fc0 0x7f85d81f6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:59.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.602+0000 7f85dde90700 1 --2- 172.21.15.189:0/157402450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85d806a730 0x7f85d806abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:59.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.602+0000 7f85dce8e700 1 -- 172.21.15.189:0/157402450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85d80b2fc0 msgr2=0x7f85d81f6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:59.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.602+0000 7f85dce8e700 1 --2- 172.21.15.189:0/157402450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85d80b2fc0 0x7f85d81f6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:59.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.602+0000 7f85dce8e700 1 -- 172.21.15.189:0/157402450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85d806a730 msgr2=0x7f85d806abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:24:59.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.602+0000 7f85dce8e700 1 --2- 172.21.15.189:0/157402450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85d806a730 0x7f85d806abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:24:59.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.602+0000 7f85dce8e700 1 -- 172.21.15.189:0/157402450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c80094a0 con 0x7f85d80b4660 2024-04-06T12:24:59.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.602+0000 7f85dde90700 1 --2- 172.21.15.189:0/157402450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85d806a730 0x7f85d806abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:24:59.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.602+0000 7f85dce8e700 1 --2- 172.21.15.189:0/157402450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85d80b4660 0x7f85d8151e10 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f85c80090b0 tx=0x7f85c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:59.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.602+0000 7f85cdffb700 1 -- 172.21.15.189:0/157402450 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c801a860 con 0x7f85d80b4660 2024-04-06T12:24:59.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.602+0000 7f85cdffb700 1 -- 172.21.15.189:0/157402450 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c801a9c0 con 0x7f85d80b4660 2024-04-06T12:24:59.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.602+0000 7f85df8f3700 1 -- 172.21.15.189:0/157402450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d81f6440 con 0x7f85d80b4660 2024-04-06T12:24:59.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.602+0000 7f85cdffb700 1 -- 172.21.15.189:0/157402450 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c801ad30 con 0x7f85d80b4660 2024-04-06T12:24:59.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.602+0000 7f85df8f3700 1 -- 172.21.15.189:0/157402450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d806b580 con 0x7f85d80b4660 2024-04-06T12:24:59.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.603+0000 7f85df8f3700 1 -- 172.21.15.189:0/157402450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d81f6960 con 0x7f85d80b4660 2024-04-06T12:24:59.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.604+0000 7f85cdffb700 1 -- 172.21.15.189:0/157402450 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f85c8022070 con 0x7f85d80b4660 2024-04-06T12:24:59.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.604+0000 7f85cdffb700 1 --2- 172.21.15.189:0/157402450 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f85bc06ade0 0x7f85bc06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:24:59.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.604+0000 7f85cdffb700 1 -- 172.21.15.189:0/157402450 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f85c808a440 con 0x7f85d80b4660 2024-04-06T12:24:59.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.604+0000 7f85dd68f700 1 --2- 172.21.15.189:0/157402450 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f85bc06ade0 0x7f85bc06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:24:59.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.604+0000 7f85dd68f700 1 --2- 172.21.15.189:0/157402450 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f85bc06ade0 0x7f85bc06d2a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f85d4000f60 tx=0x7f85d4009040 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:24:59.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.608+0000 7f85cdffb700 1 -- 172.21.15.189:0/157402450 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f85d81f6960 con 0x7f85d80b4660 2024-04-06T12:24:59.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:24:59.809+0000 7f85df8f3700 1 -- 172.21.15.189:0/157402450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f85d81f6960 con 0x7f85d80b4660 2024-04-06T12:25:00.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.640+0000 7f85cdffb700 1 -- 172.21.15.189:0/157402450 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v64) v1 ==== 138+0+0 (secure 0 0 0) 0x7f85d81f6960 con 0x7f85d80b4660 2024-04-06T12:25:00.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.644+0000 7f85df8f3700 1 -- 172.21.15.189:0/157402450 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f85bc06ade0 msgr2=0x7f85bc06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:00.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.644+0000 7f85df8f3700 1 --2- 172.21.15.189:0/157402450 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f85bc06ade0 0x7f85bc06d2a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f85d4000f60 tx=0x7f85d4009040 comp rx=0 tx=0).stop 2024-04-06T12:25:00.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.644+0000 7f85df8f3700 1 -- 172.21.15.189:0/157402450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85d80b4660 msgr2=0x7f85d8151e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:00.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.644+0000 7f85df8f3700 1 --2- 172.21.15.189:0/157402450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85d80b4660 0x7f85d8151e10 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f85c80090b0 tx=0x7f85c8004750 comp rx=0 tx=0).stop 2024-04-06T12:25:00.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.645+0000 7f85df8f3700 1 -- 172.21.15.189:0/157402450 shutdown_connections 2024-04-06T12:25:00.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.645+0000 7f85df8f3700 1 --2- 172.21.15.189:0/157402450 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f85bc06ade0 0x7f85bc06d2a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:00.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.645+0000 7f85df8f3700 1 --2- 172.21.15.189:0/157402450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85d806a730 0x7f85d806abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:00.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.645+0000 7f85df8f3700 1 --2- 172.21.15.189:0/157402450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85d80b2fc0 0x7f85d81f6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:00.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.645+0000 7f85df8f3700 1 --2- 172.21.15.189:0/157402450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85d80b4660 0x7f85d8151e10 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:00.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.645+0000 7f85df8f3700 1 -- 172.21.15.189:0/157402450 >> 172.21.15.189:0/157402450 conn(0x7f85d80d00b0 msgr2=0x7f85d8154e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:00.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.645+0000 7f85df8f3700 1 -- 172.21.15.189:0/157402450 shutdown_connections 2024-04-06T12:25:00.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.645+0000 7f85df8f3700 1 -- 172.21.15.189:0/157402450 wait complete. 2024-04-06T12:25:00.645 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' created 2024-04-06T12:25:00.659 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-06T12:25:00.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.796+0000 7f519d258700 1 Processor -- start 2024-04-06T12:25:00.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.796+0000 7f519d258700 1 -- start start 2024-04-06T12:25:00.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.797+0000 7f519d258700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f519806eb80 0x7f51980ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:00.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.797+0000 7f519d258700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519806c080 con 0x7f51980cdc60 2024-04-06T12:25:00.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.797+0000 7f519d258700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519806a8d0 con 0x7f5198159b20 2024-04-06T12:25:00.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.797+0000 7f519d258700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51980ce650 con 0x7f519806eb80 2024-04-06T12:25:00.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.797+0000 7f519659c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f51980cdc60 0x7f519806e470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37234/0 (socket says 172.21.15.189:37234) 2024-04-06T12:25:00.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.797+0000 7f519659c700 1 -- 172.21.15.189:0/1877792194 learned_addr learned my addr 172.21.15.189:0/1877792194 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:25:00.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f5195d9b700 1 -- 172.21.15.189:0/1877792194 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127067545 0 0) 0x7f519806c080 con 0x7f51980cdc60 2024-04-06T12:25:00.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f519759e700 1 --2- 172.21.15.189:0/1877792194 >> v2:172.21.15.192:3300/0 conn(0x7f519806eb80 0x7f51980ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:00.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f5195d9b700 1 -- 172.21.15.189:0/1877792194 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51800046c0 con 0x7f51980cdc60 2024-04-06T12:25:00.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f5195d9b700 1 -- 172.21.15.189:0/1877792194 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3220974177 0 0) 0x7f51800046c0 con 0x7f51980cdc60 2024-04-06T12:25:00.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f5195d9b700 1 -- 172.21.15.189:0/1877792194 >> v2:172.21.15.192:3300/0 conn(0x7f519806eb80 msgr2=0x7f51980ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:00.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f5195d9b700 1 --2- 172.21.15.189:0/1877792194 >> v2:172.21.15.192:3300/0 conn(0x7f519806eb80 0x7f51980ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:00.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f5195d9b700 1 -- 172.21.15.189:0/1877792194 >> v1:172.21.15.192:6789/0 conn(0x7f5198159b20 legacy=0x7f519806b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:00.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f5195d9b700 1 -- 172.21.15.189:0/1877792194 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519806abb0 con 0x7f51980cdc60 2024-04-06T12:25:00.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f5195d9b700 1 -- 172.21.15.189:0/1877792194 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f5188001670 con 0x7f51980cdc60 2024-04-06T12:25:00.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f5195d9b700 1 -- 172.21.15.189:0/1877792194 >> v1:172.21.15.189:6789/0 conn(0x7f51980cdc60 legacy=0x7f519806e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:00.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f5195d9b700 1 --2- 172.21.15.189:0/1877792194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5180004b10 0x7f5180004f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:00.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f5195d9b700 1 --2- 172.21.15.189:0/1877792194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5180005960 0x7f5180005de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:00.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f5195d9b700 1 --2- 172.21.15.189:0/1877792194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5180006320 0x7f518000cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:00.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f5195d9b700 1 -- 172.21.15.189:0/1877792194 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51980ce650 con 0x7f5180005960 2024-04-06T12:25:00.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f5195d9b700 1 -- 172.21.15.189:0/1877792194 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518000d2c0 con 0x7f5180004b10 2024-04-06T12:25:00.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f5195d9b700 1 -- 172.21.15.189:0/1877792194 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5180003550 con 0x7f5180006320 2024-04-06T12:25:00.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f519659c700 1 --2- 172.21.15.189:0/1877792194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5180005960 0x7f5180005de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:00.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f5196d9d700 1 --2- 172.21.15.189:0/1877792194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5180004b10 0x7f5180004f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:00.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f519759e700 1 --2- 172.21.15.189:0/1877792194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5180006320 0x7f518000cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:00.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f519659c700 1 -- 172.21.15.189:0/1877792194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5180006320 msgr2=0x7f518000cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:00.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f519659c700 1 --2- 172.21.15.189:0/1877792194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5180006320 0x7f518000cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:00.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f519659c700 1 -- 172.21.15.189:0/1877792194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5180004b10 msgr2=0x7f5180004f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:00.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f519659c700 1 --2- 172.21.15.189:0/1877792194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5180004b10 0x7f5180004f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:00.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f519659c700 1 -- 172.21.15.189:0/1877792194 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51800046c0 con 0x7f5180005960 2024-04-06T12:25:00.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f519759e700 1 -- 172.21.15.189:0/1877792194 reap_dead start 2024-04-06T12:25:00.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.798+0000 7f519659c700 1 --2- 172.21.15.189:0/1877792194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5180005960 0x7f5180005de0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f5188001050 tx=0x7f5180006990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a50c8f79a30f5c1c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:00.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.799+0000 7f5195d9b700 1 -- 172.21.15.189:0/1877792194 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518800cd80 con 0x7f5180005960 2024-04-06T12:25:00.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.799+0000 7f5195d9b700 1 -- 172.21.15.189:0/1877792194 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5188005960 con 0x7f5180005960 2024-04-06T12:25:00.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.799+0000 7f5195d9b700 1 -- 172.21.15.189:0/1877792194 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188005d10 con 0x7f5180005960 2024-04-06T12:25:00.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.799+0000 7f519d258700 1 -- 172.21.15.189:0/1877792194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5180005960 msgr2=0x7f5180005de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:00.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.799+0000 7f519d258700 1 --2- 172.21.15.189:0/1877792194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5180005960 0x7f5180005de0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f5188001050 tx=0x7f5180006990 comp rx=0 tx=0).stop 2024-04-06T12:25:00.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.799+0000 7f519d258700 1 -- 172.21.15.189:0/1877792194 shutdown_connections 2024-04-06T12:25:00.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.799+0000 7f519d258700 1 --2- 172.21.15.189:0/1877792194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5180005960 0x7f5180005de0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:00.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.799+0000 7f519d258700 1 -- 172.21.15.189:0/1877792194 >> 172.21.15.189:0/1877792194 conn(0x7f51980cc290 msgr2=0x7f51980cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:00.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.799+0000 7f519d258700 1 -- 172.21.15.189:0/1877792194 shutdown_connections 2024-04-06T12:25:00.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.799+0000 7f519d258700 1 -- 172.21.15.189:0/1877792194 wait complete. 2024-04-06T12:25:00.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.801+0000 7f519d258700 1 Processor -- start 2024-04-06T12:25:00.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.801+0000 7f519d258700 1 -- start start 2024-04-06T12:25:00.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.801+0000 7f519d258700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5180005960 0x7f51981e43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:00.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.801+0000 7f519d258700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51981e4930 0x7f51980ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:00.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.801+0000 7f519d258700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51980ace10 0x7f51980ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:00.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.801+0000 7f519d258700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51981e4db0 con 0x7f5180005960 2024-04-06T12:25:00.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.801+0000 7f519d258700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519814d2c0 con 0x7f51981e4930 2024-04-06T12:25:00.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.801+0000 7f519d258700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51980b23b0 con 0x7f51980ace10 2024-04-06T12:25:00.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.802+0000 7f5196d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5180005960 0x7f51981e43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:00.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.802+0000 7f5196d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5180005960 0x7f51981e43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57766/0 (socket says 172.21.15.189:57766) 2024-04-06T12:25:00.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.802+0000 7f5196d9d700 1 -- 172.21.15.189:0/1346132343 learned_addr learned my addr 172.21.15.189:0/1346132343 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:00.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.802+0000 7f519659c700 1 --2- 172.21.15.189:0/1346132343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51981e4930 0x7f51980ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:00.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.802+0000 7f5196d9d700 1 -- 172.21.15.189:0/1346132343 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51980ace10 msgr2=0x7f51980ad2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:00.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.802+0000 7f519759e700 1 --2- 172.21.15.189:0/1346132343 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51980ace10 0x7f51980ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:00.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.802+0000 7f5196d9d700 1 --2- 172.21.15.189:0/1346132343 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51980ace10 0x7f51980ad2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:00.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.802+0000 7f5196d9d700 1 -- 172.21.15.189:0/1346132343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51981e4930 msgr2=0x7f51980ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:00.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.802+0000 7f5196d9d700 1 --2- 172.21.15.189:0/1346132343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51981e4930 0x7f51980ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:00.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.802+0000 7f5196d9d700 1 -- 172.21.15.189:0/1346132343 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51980b2690 con 0x7f5180005960 2024-04-06T12:25:00.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.802+0000 7f519659c700 1 --2- 172.21.15.189:0/1346132343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51981e4930 0x7f51980ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:25:00.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.802+0000 7f5196d9d700 1 --2- 172.21.15.189:0/1346132343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5180005960 0x7f51981e43f0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f518c00f770 tx=0x7f518c00fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:00.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.802+0000 7f517f7fe700 1 -- 172.21.15.189:0/1346132343 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c0052e0 con 0x7f5180005960 2024-04-06T12:25:00.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.802+0000 7f517f7fe700 1 -- 172.21.15.189:0/1346132343 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f518c005440 con 0x7f5180005960 2024-04-06T12:25:00.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.802+0000 7f519d258700 1 -- 172.21.15.189:0/1346132343 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51980b2970 con 0x7f5180005960 2024-04-06T12:25:00.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.802+0000 7f517f7fe700 1 -- 172.21.15.189:0/1346132343 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c0038a0 con 0x7f5180005960 2024-04-06T12:25:00.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.802+0000 7f519d258700 1 -- 172.21.15.189:0/1346132343 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51980c7740 con 0x7f5180005960 2024-04-06T12:25:00.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.803+0000 7f519d258700 1 -- 172.21.15.189:0/1346132343 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51980c78d0 con 0x7f5180005960 2024-04-06T12:25:00.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.804+0000 7f517f7fe700 1 -- 172.21.15.189:0/1346132343 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f518c003a00 con 0x7f5180005960 2024-04-06T12:25:00.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.804+0000 7f517f7fe700 1 --2- 172.21.15.189:0/1346132343 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f518406adb0 0x7f518406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:00.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.804+0000 7f519659c700 1 --2- 172.21.15.189:0/1346132343 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f518406adb0 0x7f518406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:00.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.804+0000 7f517f7fe700 1 -- 172.21.15.189:0/1346132343 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f518c089140 con 0x7f5180005960 2024-04-06T12:25:00.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.805+0000 7f519659c700 1 --2- 172.21.15.189:0/1346132343 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f518406adb0 0x7f518406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f51981f6890 tx=0x7f518800a000 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:00.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:00.809+0000 7f517f7fe700 1 -- 172.21.15.189:0/1346132343 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f51980c78d0 con 0x7f5180005960 2024-04-06T12:25:00.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.000+0000 7f519d258700 1 -- 172.21.15.189:0/1346132343 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f51980b2e90 con 0x7f5180005960 2024-04-06T12:25:01.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.642+0000 7f517f7fe700 1 -- 172.21.15.189:0/1346132343 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v65) v1 ==== 170+0+0 (secure 0 0 0) 0x7f51980b2e90 con 0x7f5180005960 2024-04-06T12:25:01.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.650+0000 7f519d258700 1 -- 172.21.15.189:0/1346132343 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f518406adb0 msgr2=0x7f518406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:01.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.650+0000 7f519d258700 1 --2- 172.21.15.189:0/1346132343 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f518406adb0 0x7f518406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f51981f6890 tx=0x7f518800a000 comp rx=0 tx=0).stop 2024-04-06T12:25:01.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.650+0000 7f519d258700 1 -- 172.21.15.189:0/1346132343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5180005960 msgr2=0x7f51981e43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:01.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.650+0000 7f519d258700 1 --2- 172.21.15.189:0/1346132343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5180005960 0x7f51981e43f0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f518c00f770 tx=0x7f518c00fb30 comp rx=0 tx=0).stop 2024-04-06T12:25:01.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.650+0000 7f519d258700 1 -- 172.21.15.189:0/1346132343 shutdown_connections 2024-04-06T12:25:01.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.650+0000 7f519d258700 1 --2- 172.21.15.189:0/1346132343 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51980ace10 0x7f51980ad2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:01.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.650+0000 7f519d258700 1 --2- 172.21.15.189:0/1346132343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5180005960 0x7f51981e43f0 secure :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f518c00f770 tx=0x7f518c00fb30 comp rx=0 tx=0).stop 2024-04-06T12:25:01.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.650+0000 7f519d258700 1 --2- 172.21.15.189:0/1346132343 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f518406adb0 0x7f518406d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:01.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.650+0000 7f519d258700 1 --2- 172.21.15.189:0/1346132343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51981e4930 0x7f51980ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:01.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.650+0000 7f519d258700 1 -- 172.21.15.189:0/1346132343 >> 172.21.15.189:0/1346132343 conn(0x7f51980cc290 msgr2=0x7f519806baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:01.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.654+0000 7f519d258700 1 -- 172.21.15.189:0/1346132343 shutdown_connections 2024-04-06T12:25:01.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.654+0000 7f519d258700 1 -- 172.21.15.189:0/1346132343 wait complete. 2024-04-06T12:25:01.655 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' created 2024-04-06T12:25:01.674 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-06T12:25:01.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.805+0000 7f5d90096700 1 Processor -- start 2024-04-06T12:25:01.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.805+0000 7f5d90096700 1 -- start start 2024-04-06T12:25:01.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.806+0000 7f5d90096700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d880b4660 0x7f5d880b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:01.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.806+0000 7f5d90096700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5d880b2fc0 0x7f5d880b3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:01.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.806+0000 7f5d90096700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d880cdbe0 con 0x7f5d8815a550 2024-04-06T12:25:01.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.806+0000 7f5d90096700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d880b5000 con 0x7f5d880b2fc0 2024-04-06T12:25:01.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.806+0000 7f5d90096700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8815a940 con 0x7f5d880b4660 2024-04-06T12:25:01.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.806+0000 7f5d8de32700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5d8815a550 0x7f5d880cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37248/0 (socket says 172.21.15.189:37248) 2024-04-06T12:25:01.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.806+0000 7f5d8de32700 1 -- 172.21.15.189:0/3048834266 learned_addr learned my addr 172.21.15.189:0/3048834266 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:25:01.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.806+0000 7f5d8ce30700 1 -- 172.21.15.189:0/3048834266 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 666743841 0 0) 0x7f5d880cdbe0 con 0x7f5d8815a550 2024-04-06T12:25:01.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.806+0000 7f5d8d631700 1 --2- 172.21.15.189:0/3048834266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d880b4660 0x7f5d880b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:01.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.806+0000 7f5d8e633700 1 --2- 172.21.15.189:0/3048834266 >> v2:172.21.15.192:3300/0 conn(0x7f5d880b2fc0 0x7f5d880b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:01.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.806+0000 7f5d8ce30700 1 -- 172.21.15.189:0/3048834266 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d740035b0 con 0x7f5d8815a550 2024-04-06T12:25:01.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8ce30700 1 -- 172.21.15.189:0/3048834266 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 354887791 0 0) 0x7f5d740035b0 con 0x7f5d8815a550 2024-04-06T12:25:01.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8ce30700 1 -- 172.21.15.189:0/3048834266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d880b4660 msgr2=0x7f5d880b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:01.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8ce30700 1 --2- 172.21.15.189:0/3048834266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d880b4660 0x7f5d880b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:01.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8ce30700 1 -- 172.21.15.189:0/3048834266 >> v2:172.21.15.192:3300/0 conn(0x7f5d880b2fc0 msgr2=0x7f5d880b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:01.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8ce30700 1 --2- 172.21.15.189:0/3048834266 >> v2:172.21.15.192:3300/0 conn(0x7f5d880b2fc0 0x7f5d880b3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:01.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8ce30700 1 -- 172.21.15.189:0/3048834266 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8815ac00 con 0x7f5d8815a550 2024-04-06T12:25:01.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8ce30700 1 -- 172.21.15.189:0/3048834266 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f5d78001670 con 0x7f5d8815a550 2024-04-06T12:25:01.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8ce30700 1 -- 172.21.15.189:0/3048834266 >> v1:172.21.15.189:6789/0 conn(0x7f5d8815a550 legacy=0x7f5d880cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:01.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8ce30700 1 --2- 172.21.15.189:0/3048834266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d74003a00 0x7f5d74003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:01.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8ce30700 1 --2- 172.21.15.189:0/3048834266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d880b4660 0x7f5d74004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:01.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8ce30700 1 --2- 172.21.15.189:0/3048834266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d74004d90 0x7f5d74005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:01.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8ce30700 1 -- 172.21.15.189:0/3048834266 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880b5000 con 0x7f5d74003a00 2024-04-06T12:25:01.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8ce30700 1 -- 172.21.15.189:0/3048834266 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8815a940 con 0x7f5d74004d90 2024-04-06T12:25:01.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8ce30700 1 -- 172.21.15.189:0/3048834266 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7400ba10 con 0x7f5d880b4660 2024-04-06T12:25:01.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8de32700 1 --2- 172.21.15.189:0/3048834266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d74003a00 0x7f5d74003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:01.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8d631700 1 --2- 172.21.15.189:0/3048834266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d880b4660 0x7f5d74004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:01.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8e633700 1 --2- 172.21.15.189:0/3048834266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d74004d90 0x7f5d74005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:01.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8de32700 1 -- 172.21.15.189:0/3048834266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d880b4660 msgr2=0x7f5d74004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:01.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8de32700 1 --2- 172.21.15.189:0/3048834266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d880b4660 0x7f5d74004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:01.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8de32700 1 -- 172.21.15.189:0/3048834266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d74004d90 msgr2=0x7f5d74005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:01.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8de32700 1 --2- 172.21.15.189:0/3048834266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d74004d90 0x7f5d74005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:01.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8de32700 1 -- 172.21.15.189:0/3048834266 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d740035b0 con 0x7f5d74003a00 2024-04-06T12:25:01.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8d631700 1 --2- 172.21.15.189:0/3048834266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d880b4660 0x7f5d74004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:25:01.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8de32700 1 --2- 172.21.15.189:0/3048834266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d74003a00 0x7f5d74003e20 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800c3a0 tx=0x7f5d74005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e09fc972b2b01f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:01.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.807+0000 7f5d8ce30700 1 -- 172.21.15.189:0/3048834266 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78007c60 con 0x7f5d74003a00 2024-04-06T12:25:01.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.808+0000 7f5d8ce30700 1 -- 172.21.15.189:0/3048834266 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d78005960 con 0x7f5d74003a00 2024-04-06T12:25:01.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.808+0000 7f5d8ce30700 1 -- 172.21.15.189:0/3048834266 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7800cea0 con 0x7f5d74003a00 2024-04-06T12:25:01.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.808+0000 7f5d90096700 1 -- 172.21.15.189:0/3048834266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d74003a00 msgr2=0x7f5d74003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:01.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.808+0000 7f5d90096700 1 --2- 172.21.15.189:0/3048834266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d74003a00 0x7f5d74003e20 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800c3a0 tx=0x7f5d74005880 comp rx=0 tx=0).stop 2024-04-06T12:25:01.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.808+0000 7f5d8e633700 1 -- 172.21.15.189:0/3048834266 reap_dead start 2024-04-06T12:25:01.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.808+0000 7f5d90096700 1 -- 172.21.15.189:0/3048834266 shutdown_connections 2024-04-06T12:25:01.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.808+0000 7f5d90096700 1 -- 172.21.15.189:0/3048834266 >> 172.21.15.189:0/3048834266 conn(0x7f5d880d00b0 msgr2=0x7f5d880d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:01.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.808+0000 7f5d90096700 1 -- 172.21.15.189:0/3048834266 shutdown_connections 2024-04-06T12:25:01.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.808+0000 7f5d90096700 1 -- 172.21.15.189:0/3048834266 wait complete. 2024-04-06T12:25:01.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.809+0000 7f5d90096700 1 Processor -- start 2024-04-06T12:25:01.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.809+0000 7f5d90096700 1 -- start start 2024-04-06T12:25:01.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.810+0000 7f5d90096700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d74003a00 0x7f5d88154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.810+0000 7f5d90096700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d881f0660 0x7f5d881ef3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.810+0000 7f5d90096700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d8815c860 0x7f5d8815cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.810+0000 7f5d90096700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8815d220 con 0x7f5d8815c860 2024-04-06T12:25:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.810+0000 7f5d90096700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8814dc40 con 0x7f5d74003a00 2024-04-06T12:25:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.810+0000 7f5d90096700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d88154d20 con 0x7f5d881f0660 2024-04-06T12:25:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.810+0000 7f5d8e633700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d8815c860 0x7f5d8815cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.810+0000 7f5d8e633700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d8815c860 0x7f5d8815cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57792/0 (socket says 172.21.15.189:57792) 2024-04-06T12:25:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.810+0000 7f5d8e633700 1 -- 172.21.15.189:0/2207130568 learned_addr learned my addr 172.21.15.189:0/2207130568 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.810+0000 7f5d8de32700 1 --2- 172.21.15.189:0/2207130568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d74003a00 0x7f5d88154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.810+0000 7f5d8d631700 1 --2- 172.21.15.189:0/2207130568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d881f0660 0x7f5d881ef3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.810+0000 7f5d8e633700 1 -- 172.21.15.189:0/2207130568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d881f0660 msgr2=0x7f5d881ef3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.810+0000 7f5d8e633700 1 --2- 172.21.15.189:0/2207130568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d881f0660 0x7f5d881ef3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.810+0000 7f5d8e633700 1 -- 172.21.15.189:0/2207130568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d74003a00 msgr2=0x7f5d88154740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.810+0000 7f5d8e633700 1 --2- 172.21.15.189:0/2207130568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d74003a00 0x7f5d88154740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.810+0000 7f5d8e633700 1 -- 172.21.15.189:0/2207130568 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d88155000 con 0x7f5d8815c860 2024-04-06T12:25:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.810+0000 7f5d8de32700 1 --2- 172.21.15.189:0/2207130568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d74003a00 0x7f5d88154740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:25:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.811+0000 7f5d8e633700 1 --2- 172.21.15.189:0/2207130568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d8815c860 0x7f5d8815cce0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5d88072830 tx=0x7f5d80011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.811+0000 7f5d7e7fc700 1 -- 172.21.15.189:0/2207130568 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8000e040 con 0x7f5d8815c860 2024-04-06T12:25:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.811+0000 7f5d7e7fc700 1 -- 172.21.15.189:0/2207130568 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8000f040 con 0x7f5d8815c860 2024-04-06T12:25:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.811+0000 7f5d7e7fc700 1 -- 172.21.15.189:0/2207130568 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8000fde0 con 0x7f5d8815c860 2024-04-06T12:25:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.811+0000 7f5d90096700 1 -- 172.21.15.189:0/2207130568 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d881552e0 con 0x7f5d8815c860 2024-04-06T12:25:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.811+0000 7f5d90096700 1 -- 172.21.15.189:0/2207130568 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d881f0ae0 con 0x7f5d8815c860 2024-04-06T12:25:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.812+0000 7f5d90096700 1 -- 172.21.15.189:0/2207130568 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d8814ddd0 con 0x7f5d8815c860 2024-04-06T12:25:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.812+0000 7f5d7e7fc700 1 -- 172.21.15.189:0/2207130568 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f5d80022460 con 0x7f5d8815c860 2024-04-06T12:25:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.813+0000 7f5d7e7fc700 1 --2- 172.21.15.189:0/2207130568 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d6c06adb0 0x7f5d6c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.813+0000 7f5d7e7fc700 1 -- 172.21.15.189:0/2207130568 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f5d8008a6d0 con 0x7f5d8815c860 2024-04-06T12:25:01.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.813+0000 7f5d8de32700 1 --2- 172.21.15.189:0/2207130568 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d6c06adb0 0x7f5d6c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:01.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.813+0000 7f5d8de32700 1 --2- 172.21.15.189:0/2207130568 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d6c06adb0 0x7f5d6c06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5d78001050 tx=0x7f5d7800a000 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:01.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:01.817+0000 7f5d7e7fc700 1 -- 172.21.15.189:0/2207130568 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5d80055690 con 0x7f5d8815c860 2024-04-06T12:25:02.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.060+0000 7f5d90096700 1 -- 172.21.15.189:0/2207130568 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f5d88067b80 con 0x7f5d8815c860 2024-04-06T12:25:02.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.065+0000 7f5d7e7fc700 1 -- 172.21.15.189:0/2207130568 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v20) v1 ==== 172+0+0 (secure 0 0 0) 0x7f5d80058cb0 con 0x7f5d8815c860 2024-04-06T12:25:02.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.074+0000 7f5d90096700 1 -- 172.21.15.189:0/2207130568 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d6c06adb0 msgr2=0x7f5d6c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:02.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.074+0000 7f5d90096700 1 --2- 172.21.15.189:0/2207130568 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d6c06adb0 0x7f5d6c06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5d78001050 tx=0x7f5d7800a000 comp rx=0 tx=0).stop 2024-04-06T12:25:02.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.074+0000 7f5d90096700 1 -- 172.21.15.189:0/2207130568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d8815c860 msgr2=0x7f5d8815cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:02.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.074+0000 7f5d90096700 1 --2- 172.21.15.189:0/2207130568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d8815c860 0x7f5d8815cce0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5d88072830 tx=0x7f5d80011f80 comp rx=0 tx=0).stop 2024-04-06T12:25:02.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.079+0000 7f5d90096700 1 -- 172.21.15.189:0/2207130568 shutdown_connections 2024-04-06T12:25:02.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.079+0000 7f5d90096700 1 --2- 172.21.15.189:0/2207130568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d8815c860 0x7f5d8815cce0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.079+0000 7f5d90096700 1 --2- 172.21.15.189:0/2207130568 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5d6c06adb0 0x7f5d6c06d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.079+0000 7f5d90096700 1 --2- 172.21.15.189:0/2207130568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d74003a00 0x7f5d88154740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.079+0000 7f5d90096700 1 --2- 172.21.15.189:0/2207130568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d881f0660 0x7f5d881ef3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.079+0000 7f5d90096700 1 -- 172.21.15.189:0/2207130568 >> 172.21.15.189:0/2207130568 conn(0x7f5d880d00b0 msgr2=0x7f5d8806eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:02.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.080+0000 7f5d90096700 1 -- 172.21.15.189:0/2207130568 shutdown_connections 2024-04-06T12:25:02.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.080+0000 7f5d90096700 1 -- 172.21.15.189:0/2207130568 wait complete. 2024-04-06T12:25:02.087 INFO:teuthology.orchestra.run.smithi189.stderr:new fs with metadata pool 7 and data pool 8 2024-04-06T12:25:02.099 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:25:02.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.222+0000 7fe68a45f700 1 Processor -- start 2024-04-06T12:25:02.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.223+0000 7fe68a45f700 1 -- start start 2024-04-06T12:25:02.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.223+0000 7fe68a45f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe68415d7e0 0x7fe6840d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:02.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.223+0000 7fe68a45f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6840d1cc0 0x7fe6840b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:02.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.223+0000 7fe68a45f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6840cdf80 con 0x7fe6840d0890 2024-04-06T12:25:02.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.223+0000 7fe68a45f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6840d3230 con 0x7fe68415d7e0 2024-04-06T12:25:02.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.223+0000 7fe68a45f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6840d2da0 con 0x7fe6840d1cc0 2024-04-06T12:25:02.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.224+0000 7fe6889fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6840d1cc0 0x7fe6840b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:02.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.224+0000 7fe683fff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe68415d7e0 0x7fe6840d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:02.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.224+0000 7fe6889fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6840d1cc0 0x7fe6840b66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:49698/0 (socket says 172.21.15.189:49698) 2024-04-06T12:25:02.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.224+0000 7fe6889fc700 1 -- 172.21.15.189:0/325736783 learned_addr learned my addr 172.21.15.189:0/325736783 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:02.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.224+0000 7fe6889fc700 1 -- 172.21.15.189:0/325736783 >> v2:172.21.15.192:3300/0 conn(0x7fe68415d7e0 msgr2=0x7fe6840d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:02.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.224+0000 7fe6889fc700 1 --2- 172.21.15.189:0/325736783 >> v2:172.21.15.192:3300/0 conn(0x7fe68415d7e0 0x7fe6840d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.224+0000 7fe6889fc700 1 -- 172.21.15.189:0/325736783 >> v1:172.21.15.192:6789/0 conn(0x7fe6840d0890 legacy=0x7fe6840d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:02.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.224+0000 7fe6889fc700 1 -- 172.21.15.189:0/325736783 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6840d0cf0 con 0x7fe6840d1cc0 2024-04-06T12:25:02.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.224+0000 7fe682ffd700 1 -- 172.21.15.189:0/325736783 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489457078 0 0) 0x7fe6840cdf80 con 0x7fe6840d0890 2024-04-06T12:25:02.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.224+0000 7fe683fff700 1 --2- 172.21.15.189:0/325736783 >> v2:172.21.15.192:3300/0 conn(0x7fe68415d7e0 0x7fe6840d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:25:02.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.224+0000 7fe6889fc700 1 --2- 172.21.15.189:0/325736783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6840d1cc0 0x7fe6840b66b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009e50 tx=0x7fe67c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32d63b8d526756d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:02.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.224+0000 7fe682ffd700 1 -- 172.21.15.189:0/325736783 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c00e070 con 0x7fe6840d1cc0 2024-04-06T12:25:02.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.224+0000 7fe682ffd700 1 -- 172.21.15.189:0/325736783 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67c00b910 con 0x7fe6840d1cc0 2024-04-06T12:25:02.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.225+0000 7fe682ffd700 1 -- 172.21.15.189:0/325736783 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c00bcc0 con 0x7fe6840d1cc0 2024-04-06T12:25:02.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.225+0000 7fe68a45f700 1 -- 172.21.15.189:0/325736783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6840d1cc0 msgr2=0x7fe6840b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:02.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.225+0000 7fe68a45f700 1 --2- 172.21.15.189:0/325736783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6840d1cc0 0x7fe6840b66b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009e50 tx=0x7fe67c00b330 comp rx=0 tx=0).stop 2024-04-06T12:25:02.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.225+0000 7fe68a45f700 1 -- 172.21.15.189:0/325736783 shutdown_connections 2024-04-06T12:25:02.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.225+0000 7fe68a45f700 1 --2- 172.21.15.189:0/325736783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6840d1cc0 0x7fe6840b66b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.225+0000 7fe68a45f700 1 --2- 172.21.15.189:0/325736783 >> v2:172.21.15.192:3300/0 conn(0x7fe68415d7e0 0x7fe6840d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.225+0000 7fe68a45f700 1 -- 172.21.15.189:0/325736783 >> 172.21.15.189:0/325736783 conn(0x7fe684072250 msgr2=0x7fe684072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:02.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.225+0000 7fe68a45f700 1 -- 172.21.15.189:0/325736783 shutdown_connections 2024-04-06T12:25:02.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.225+0000 7fe68a45f700 1 -- 172.21.15.189:0/325736783 wait complete. 2024-04-06T12:25:02.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.226+0000 7fe68a45f700 1 Processor -- start 2024-04-06T12:25:02.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.227+0000 7fe68a45f700 1 -- start start 2024-04-06T12:25:02.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.227+0000 7fe68a45f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6840d0890 0x7fe6840c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:02.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.227+0000 7fe68a45f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6840d1cc0 0x7fe68414e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:02.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.227+0000 7fe68a45f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe68415d7e0 0x7fe6840b5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:02.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.227+0000 7fe68a45f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6840c7880 con 0x7fe6840d0890 2024-04-06T12:25:02.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.227+0000 7fe68a45f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6840c81e0 con 0x7fe6840d1cc0 2024-04-06T12:25:02.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.227+0000 7fe68a45f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe684157de0 con 0x7fe68415d7e0 2024-04-06T12:25:02.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.227+0000 7fe683fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6840d0890 0x7fe6840c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:02.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.227+0000 7fe683fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6840d0890 0x7fe6840c7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57800/0 (socket says 172.21.15.189:57800) 2024-04-06T12:25:02.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.227+0000 7fe683fff700 1 -- 172.21.15.189:0/624948059 learned_addr learned my addr 172.21.15.189:0/624948059 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:02.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.228+0000 7fe6889fc700 1 --2- 172.21.15.189:0/624948059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe68415d7e0 0x7fe6840b5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:02.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.228+0000 7fe6837fe700 1 --2- 172.21.15.189:0/624948059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6840d1cc0 0x7fe68414e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:02.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.228+0000 7fe683fff700 1 -- 172.21.15.189:0/624948059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe68415d7e0 msgr2=0x7fe6840b5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:02.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.228+0000 7fe683fff700 1 --2- 172.21.15.189:0/624948059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe68415d7e0 0x7fe6840b5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.228+0000 7fe683fff700 1 -- 172.21.15.189:0/624948059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6840d1cc0 msgr2=0x7fe68414e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:02.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.228+0000 7fe683fff700 1 --2- 172.21.15.189:0/624948059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6840d1cc0 0x7fe68414e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.228+0000 7fe683fff700 1 -- 172.21.15.189:0/624948059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6740094a0 con 0x7fe6840d0890 2024-04-06T12:25:02.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.228+0000 7fe6837fe700 1 --2- 172.21.15.189:0/624948059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6840d1cc0 0x7fe68414e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:25:02.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.228+0000 7fe683fff700 1 --2- 172.21.15.189:0/624948059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6840d0890 0x7fe6840c7ca0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe67400ebe0 tx=0x7fe67400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:02.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.228+0000 7fe680ff9700 1 -- 172.21.15.189:0/624948059 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe674009800 con 0x7fe6840d0890 2024-04-06T12:25:02.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.228+0000 7fe680ff9700 1 -- 172.21.15.189:0/624948059 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67400c910 con 0x7fe6840d0890 2024-04-06T12:25:02.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.228+0000 7fe68a45f700 1 -- 172.21.15.189:0/624948059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67c009b30 con 0x7fe6840d0890 2024-04-06T12:25:02.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.228+0000 7fe680ff9700 1 -- 172.21.15.189:0/624948059 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67400cc80 con 0x7fe6840d0890 2024-04-06T12:25:02.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.228+0000 7fe68a45f700 1 -- 172.21.15.189:0/624948059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6840c6ee0 con 0x7fe6840d0890 2024-04-06T12:25:02.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.229+0000 7fe68a45f700 1 -- 172.21.15.189:0/624948059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe684074990 con 0x7fe6840d0890 2024-04-06T12:25:02.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.230+0000 7fe680ff9700 1 -- 172.21.15.189:0/624948059 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fe674013470 con 0x7fe6840d0890 2024-04-06T12:25:02.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.230+0000 7fe680ff9700 1 --2- 172.21.15.189:0/624948059 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe67006adb0 0x7fe67006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:02.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.230+0000 7fe680ff9700 1 -- 172.21.15.189:0/624948059 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe674089710 con 0x7fe6840d0890 2024-04-06T12:25:02.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.230+0000 7fe6837fe700 1 --2- 172.21.15.189:0/624948059 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe67006adb0 0x7fe67006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:02.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.230+0000 7fe6837fe700 1 --2- 172.21.15.189:0/624948059 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe67006adb0 0x7fe67006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe68415e9b0 tx=0x7fe678009040 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:02.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.234+0000 7fe680ff9700 1 -- 172.21.15.189:0/624948059 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe6740546a0 con 0x7fe6840d0890 2024-04-06T12:25:02.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.432+0000 7fe68a45f700 1 -- 172.21.15.189:0/624948059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6840b1f20 con 0x7fe6840d0890 2024-04-06T12:25:02.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.433+0000 7fe680ff9700 1 -- 172.21.15.189:0/624948059 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16367 (secure 0 0 0) 0x7fe674057cc0 con 0x7fe6840d0890 2024-04-06T12:25:02.432 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:25:02.432 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":66,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:25:02.062144+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:24:59.828905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-06T12:25:01.020253+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:25:02.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.435+0000 7fe68a45f700 1 -- 172.21.15.189:0/624948059 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe67006adb0 msgr2=0x7fe67006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:02.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.435+0000 7fe68a45f700 1 --2- 172.21.15.189:0/624948059 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe67006adb0 0x7fe67006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe68415e9b0 tx=0x7fe678009040 comp rx=0 tx=0).stop 2024-04-06T12:25:02.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.435+0000 7fe68a45f700 1 -- 172.21.15.189:0/624948059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6840d0890 msgr2=0x7fe6840c7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:02.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.435+0000 7fe68a45f700 1 --2- 172.21.15.189:0/624948059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6840d0890 0x7fe6840c7ca0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe67400ebe0 tx=0x7fe67400eef0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.435+0000 7fe68a45f700 1 -- 172.21.15.189:0/624948059 shutdown_connections 2024-04-06T12:25:02.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.435+0000 7fe68a45f700 1 --2- 172.21.15.189:0/624948059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe68415d7e0 0x7fe6840b5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.435+0000 7fe68a45f700 1 --2- 172.21.15.189:0/624948059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6840d0890 0x7fe6840c7ca0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.435+0000 7fe68a45f700 1 --2- 172.21.15.189:0/624948059 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe67006adb0 0x7fe67006d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.435+0000 7fe68a45f700 1 --2- 172.21.15.189:0/624948059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6840d1cc0 0x7fe68414e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.435+0000 7fe68a45f700 1 -- 172.21.15.189:0/624948059 >> 172.21.15.189:0/624948059 conn(0x7fe684072250 msgr2=0x7fe6840cdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:02.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.436+0000 7fe68a45f700 1 -- 172.21.15.189:0/624948059 shutdown_connections 2024-04-06T12:25:02.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.436+0000 7fe68a45f700 1 -- 172.21.15.189:0/624948059 wait complete. 2024-04-06T12:25:02.447 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:25:02.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.570+0000 7fee18c20700 1 Processor -- start 2024-04-06T12:25:02.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.570+0000 7fee18c20700 1 -- start start 2024-04-06T12:25:02.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.572+0000 7fee18c20700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fee1415a550 0x7fee140cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:02.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.572+0000 7fee18c20700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fee140cf900 0x7fee140b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:02.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.572+0000 7fee18c20700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee140cdbe0 con 0x7fee140b4ba0 2024-04-06T12:25:02.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.572+0000 7fee18c20700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee1415a940 con 0x7fee140cf900 2024-04-06T12:25:02.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.572+0000 7fee18c20700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee1415ac00 con 0x7fee1415a550 2024-04-06T12:25:02.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.572+0000 7fee11d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fee140cf900 0x7fee140b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:02.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.572+0000 7fee11d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fee140cf900 0x7fee140b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57814/0 (socket says 172.21.15.189:57814) 2024-04-06T12:25:02.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.572+0000 7fee11d9b700 1 -- 172.21.15.189:0/3000873688 learned_addr learned my addr 172.21.15.189:0/3000873688 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:02.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.572+0000 7fee1259c700 1 --2- 172.21.15.189:0/3000873688 >> v2:172.21.15.192:3300/0 conn(0x7fee1415a550 0x7fee140cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:02.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.572+0000 7fee11d9b700 1 -- 172.21.15.189:0/3000873688 >> v2:172.21.15.192:3300/0 conn(0x7fee1415a550 msgr2=0x7fee140cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:02.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.572+0000 7fee11d9b700 1 --2- 172.21.15.189:0/3000873688 >> v2:172.21.15.192:3300/0 conn(0x7fee1415a550 0x7fee140cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.572+0000 7fee11d9b700 1 -- 172.21.15.189:0/3000873688 >> v1:172.21.15.192:6789/0 conn(0x7fee140b4ba0 legacy=0x7fee1406eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:02.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.572+0000 7fee11d9b700 1 -- 172.21.15.189:0/3000873688 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1415aee0 con 0x7fee140cf900 2024-04-06T12:25:02.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.572+0000 7fee1259c700 1 --2- 172.21.15.189:0/3000873688 >> v2:172.21.15.192:3300/0 conn(0x7fee1415a550 0x7fee140cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:25:02.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.572+0000 7fee11d9b700 1 --2- 172.21.15.189:0/3000873688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee140cf900 0x7fee140b4660 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fee14073470 tx=0x7fee04009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e3b4c6e55426d2b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:02.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.572+0000 7fee1159a700 1 -- 172.21.15.189:0/3000873688 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0400b870 con 0x7fee140cf900 2024-04-06T12:25:02.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.572+0000 7fee1159a700 1 -- 172.21.15.189:0/3000873688 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0400b9d0 con 0x7fee140cf900 2024-04-06T12:25:02.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.572+0000 7fee1159a700 1 -- 172.21.15.189:0/3000873688 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0400bd60 con 0x7fee140cf900 2024-04-06T12:25:02.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.573+0000 7fee18c20700 1 -- 172.21.15.189:0/3000873688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee140cf900 msgr2=0x7fee140b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:02.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.573+0000 7fee18c20700 1 --2- 172.21.15.189:0/3000873688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee140cf900 0x7fee140b4660 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fee14073470 tx=0x7fee04009bd0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.573+0000 7fee18c20700 1 -- 172.21.15.189:0/3000873688 shutdown_connections 2024-04-06T12:25:02.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.573+0000 7fee18c20700 1 --2- 172.21.15.189:0/3000873688 >> v2:172.21.15.192:3300/0 conn(0x7fee1415a550 0x7fee140cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.573+0000 7fee18c20700 1 --2- 172.21.15.189:0/3000873688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee140cf900 0x7fee140b4660 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.573+0000 7fee18c20700 1 -- 172.21.15.189:0/3000873688 >> 172.21.15.189:0/3000873688 conn(0x7fee140d00b0 msgr2=0x7fee140d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:02.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.573+0000 7fee18c20700 1 -- 172.21.15.189:0/3000873688 shutdown_connections 2024-04-06T12:25:02.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.573+0000 7fee18c20700 1 -- 172.21.15.189:0/3000873688 wait complete. 2024-04-06T12:25:02.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.574+0000 7fee18c20700 1 Processor -- start 2024-04-06T12:25:02.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.574+0000 7fee18c20700 1 -- start start 2024-04-06T12:25:02.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.575+0000 7fee18c20700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee140b4ba0 0x7fee140cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:02.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.575+0000 7fee18c20700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee140cf900 0x7fee140aaa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:02.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.575+0000 7fee18c20700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee1414a770 0x7fee140aafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:02.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.575+0000 7fee18c20700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1415d3e0 con 0x7fee140cf900 2024-04-06T12:25:02.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.575+0000 7fee18c20700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1415c7d0 con 0x7fee140b4ba0 2024-04-06T12:25:02.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.575+0000 7fee18c20700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1415cab0 con 0x7fee1414a770 2024-04-06T12:25:02.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.575+0000 7fee11d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee140cf900 0x7fee140aaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:02.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.575+0000 7fee11d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee140cf900 0x7fee140aaa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57822/0 (socket says 172.21.15.189:57822) 2024-04-06T12:25:02.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.575+0000 7fee11d9b700 1 -- 172.21.15.189:0/3098000299 learned_addr learned my addr 172.21.15.189:0/3098000299 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:02.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.576+0000 7fee12d9d700 1 --2- 172.21.15.189:0/3098000299 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee1414a770 0x7fee140aafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:02.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.576+0000 7fee11d9b700 1 -- 172.21.15.189:0/3098000299 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee1414a770 msgr2=0x7fee140aafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:02.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.576+0000 7fee1259c700 1 --2- 172.21.15.189:0/3098000299 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee140b4ba0 0x7fee140cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:02.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.576+0000 7fee11d9b700 1 --2- 172.21.15.189:0/3098000299 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee1414a770 0x7fee140aafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.576+0000 7fee11d9b700 1 -- 172.21.15.189:0/3098000299 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee140b4ba0 msgr2=0x7fee140cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:02.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.576+0000 7fee11d9b700 1 --2- 172.21.15.189:0/3098000299 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee140b4ba0 0x7fee140cc1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.576+0000 7fee11d9b700 1 -- 172.21.15.189:0/3098000299 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee04009550 con 0x7fee140cf900 2024-04-06T12:25:02.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.576+0000 7fee11d9b700 1 --2- 172.21.15.189:0/3098000299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee140cf900 0x7fee140aaa60 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fee04000c00 tx=0x7fee04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:02.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.576+0000 7fee02ffd700 1 -- 172.21.15.189:0/3098000299 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04004540 con 0x7fee140cf900 2024-04-06T12:25:02.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.576+0000 7fee02ffd700 1 -- 172.21.15.189:0/3098000299 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee040036a0 con 0x7fee140cf900 2024-04-06T12:25:02.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.576+0000 7fee18c20700 1 -- 172.21.15.189:0/3098000299 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1415cd90 con 0x7fee140cf900 2024-04-06T12:25:02.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.576+0000 7fee02ffd700 1 -- 172.21.15.189:0/3098000299 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04003a30 con 0x7fee140cf900 2024-04-06T12:25:02.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.576+0000 7fee18c20700 1 -- 172.21.15.189:0/3098000299 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee140cc710 con 0x7fee140cf900 2024-04-06T12:25:02.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.577+0000 7fee18c20700 1 -- 172.21.15.189:0/3098000299 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee140cc8d0 con 0x7fee140cf900 2024-04-06T12:25:02.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.577+0000 7fee02ffd700 1 -- 172.21.15.189:0/3098000299 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fee04021070 con 0x7fee140cf900 2024-04-06T12:25:02.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.578+0000 7fee02ffd700 1 --2- 172.21.15.189:0/3098000299 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fedf406adb0 0x7fedf406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:02.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.578+0000 7fee02ffd700 1 -- 172.21.15.189:0/3098000299 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fee0408ad10 con 0x7fee140cf900 2024-04-06T12:25:02.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.578+0000 7fee1259c700 1 --2- 172.21.15.189:0/3098000299 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fedf406adb0 0x7fedf406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:02.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.578+0000 7fee1259c700 1 --2- 172.21.15.189:0/3098000299 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fedf406adb0 0x7fedf406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fee0800d8b0 tx=0x7fee08008040 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:02.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.582+0000 7fee02ffd700 1 -- 172.21.15.189:0/3098000299 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fee04055d30 con 0x7fee140cf900 2024-04-06T12:25:02.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.779+0000 7fee18c20700 1 -- 172.21.15.189:0/3098000299 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee14067bb0 con 0x7fee140cf900 2024-04-06T12:25:02.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.780+0000 7fee02ffd700 1 -- 172.21.15.189:0/3098000299 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16367 (secure 0 0 0) 0x7fee04015020 con 0x7fee140cf900 2024-04-06T12:25:02.779 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:25:02.779 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":66,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:25:02.062144+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:24:59.828905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-06T12:25:01.020253+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:25:02.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.782+0000 7fee18c20700 1 -- 172.21.15.189:0/3098000299 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fedf406adb0 msgr2=0x7fedf406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:02.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.782+0000 7fee18c20700 1 --2- 172.21.15.189:0/3098000299 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fedf406adb0 0x7fedf406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fee0800d8b0 tx=0x7fee08008040 comp rx=0 tx=0).stop 2024-04-06T12:25:02.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.782+0000 7fee18c20700 1 -- 172.21.15.189:0/3098000299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee140cf900 msgr2=0x7fee140aaa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:02.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.782+0000 7fee18c20700 1 --2- 172.21.15.189:0/3098000299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee140cf900 0x7fee140aaa60 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fee04000c00 tx=0x7fee04004750 comp rx=0 tx=0).stop 2024-04-06T12:25:02.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.782+0000 7fee18c20700 1 -- 172.21.15.189:0/3098000299 shutdown_connections 2024-04-06T12:25:02.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.782+0000 7fee18c20700 1 --2- 172.21.15.189:0/3098000299 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee1414a770 0x7fee140aafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.782+0000 7fee18c20700 1 --2- 172.21.15.189:0/3098000299 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fedf406adb0 0x7fedf406d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.782+0000 7fee18c20700 1 --2- 172.21.15.189:0/3098000299 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee140b4ba0 0x7fee140cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.782+0000 7fee18c20700 1 --2- 172.21.15.189:0/3098000299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee140cf900 0x7fee140aaa60 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.782+0000 7fee18c20700 1 -- 172.21.15.189:0/3098000299 >> 172.21.15.189:0/3098000299 conn(0x7fee140d00b0 msgr2=0x7fee14073b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:02.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.783+0000 7fee18c20700 1 -- 172.21.15.189:0/3098000299 shutdown_connections 2024-04-06T12:25:02.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.783+0000 7fee18c20700 1 -- 172.21.15.189:0/3098000299 wait complete. 2024-04-06T12:25:02.793 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-06T12:25:02.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.920+0000 7f9d28e3a700 1 Processor -- start 2024-04-06T12:25:02.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.920+0000 7f9d28e3a700 1 -- start start 2024-04-06T12:25:02.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.921+0000 7f9d28e3a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9d240cb870 0x7f9d2415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:02.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.921+0000 7f9d28e3a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9d240cfa00 0x7f9d240cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:02.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.921+0000 7f9d28e3a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d240cbd80 con 0x7f9d240d11b0 2024-04-06T12:25:02.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.921+0000 7f9d28e3a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d240ccdc0 con 0x7f9d240cb870 2024-04-06T12:25:02.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.921+0000 7f9d28e3a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d240d0b50 con 0x7f9d240cfa00 2024-04-06T12:25:02.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.921+0000 7f9d21d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9d240cb870 0x7f9d2415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:02.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.921+0000 7f9d21d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9d240cb870 0x7f9d2415c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57826/0 (socket says 172.21.15.189:57826) 2024-04-06T12:25:02.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.921+0000 7f9d21d9b700 1 -- 172.21.15.189:0/566965590 learned_addr learned my addr 172.21.15.189:0/566965590 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:02.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.922+0000 7f9d22d9d700 1 --2- 172.21.15.189:0/566965590 >> v2:172.21.15.192:3300/0 conn(0x7f9d240cfa00 0x7f9d240cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:02.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.922+0000 7f9d21d9b700 1 -- 172.21.15.189:0/566965590 >> v2:172.21.15.192:3300/0 conn(0x7f9d240cfa00 msgr2=0x7f9d240cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:02.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.922+0000 7f9d21d9b700 1 --2- 172.21.15.189:0/566965590 >> v2:172.21.15.192:3300/0 conn(0x7f9d240cfa00 0x7f9d240cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.922+0000 7f9d21d9b700 1 -- 172.21.15.189:0/566965590 >> v1:172.21.15.192:6789/0 conn(0x7f9d240d11b0 legacy=0x7f9d240d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:02.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.922+0000 7f9d21d9b700 1 -- 172.21.15.189:0/566965590 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d240cef20 con 0x7f9d240cb870 2024-04-06T12:25:02.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.922+0000 7f9d21d9b700 1 --2- 172.21.15.189:0/566965590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d240cb870 0x7f9d2415c280 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f9d2415ff70 tx=0x7f9d14009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e3eb0121253068 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:02.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.922+0000 7f9d2159a700 1 -- 172.21.15.189:0/566965590 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d14011830 con 0x7f9d240cb870 2024-04-06T12:25:02.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.922+0000 7f9d2159a700 1 -- 172.21.15.189:0/566965590 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d14011990 con 0x7f9d240cb870 2024-04-06T12:25:02.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.922+0000 7f9d2159a700 1 -- 172.21.15.189:0/566965590 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d14011d20 con 0x7f9d240cb870 2024-04-06T12:25:02.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.923+0000 7f9d28e3a700 1 -- 172.21.15.189:0/566965590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d240cb870 msgr2=0x7f9d2415c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:02.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.923+0000 7f9d28e3a700 1 --2- 172.21.15.189:0/566965590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d240cb870 0x7f9d2415c280 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f9d2415ff70 tx=0x7f9d14009bd0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.923+0000 7f9d28e3a700 1 -- 172.21.15.189:0/566965590 shutdown_connections 2024-04-06T12:25:02.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.923+0000 7f9d28e3a700 1 --2- 172.21.15.189:0/566965590 >> v2:172.21.15.192:3300/0 conn(0x7f9d240cfa00 0x7f9d240cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.923+0000 7f9d28e3a700 1 --2- 172.21.15.189:0/566965590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d240cb870 0x7f9d2415c280 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.923+0000 7f9d28e3a700 1 -- 172.21.15.189:0/566965590 >> 172.21.15.189:0/566965590 conn(0x7f9d240b2db0 msgr2=0x7f9d240b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:02.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.923+0000 7f9d28e3a700 1 -- 172.21.15.189:0/566965590 shutdown_connections 2024-04-06T12:25:02.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.923+0000 7f9d28e3a700 1 -- 172.21.15.189:0/566965590 wait complete. 2024-04-06T12:25:02.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.924+0000 7f9d28e3a700 1 Processor -- start 2024-04-06T12:25:02.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.924+0000 7f9d28e3a700 1 -- start start 2024-04-06T12:25:02.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d28e3a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d240cb870 0x7f9d241f4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:02.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d28e3a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d240cfa00 0x7f9d2414d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:02.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d28e3a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d240d11b0 0x7f9d24159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:02.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d28e3a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d241f5400 con 0x7f9d240cb870 2024-04-06T12:25:02.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d28e3a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d241573d0 con 0x7f9d240cfa00 2024-04-06T12:25:02.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d28e3a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d241567c0 con 0x7f9d240d11b0 2024-04-06T12:25:02.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d2259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d240cb870 0x7f9d241f4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:02.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d2259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d240cb870 0x7f9d241f4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57840/0 (socket says 172.21.15.189:57840) 2024-04-06T12:25:02.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d2259c700 1 -- 172.21.15.189:0/4245123532 learned_addr learned my addr 172.21.15.189:0/4245123532 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:02.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d21d9b700 1 --2- 172.21.15.189:0/4245123532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d240cfa00 0x7f9d2414d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:02.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d22d9d700 1 --2- 172.21.15.189:0/4245123532 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d240d11b0 0x7f9d24159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:02.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d2259c700 1 -- 172.21.15.189:0/4245123532 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d240d11b0 msgr2=0x7f9d24159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:02.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d2259c700 1 --2- 172.21.15.189:0/4245123532 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d240d11b0 0x7f9d24159cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d2259c700 1 -- 172.21.15.189:0/4245123532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d240cfa00 msgr2=0x7f9d2414d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:02.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d2259c700 1 --2- 172.21.15.189:0/4245123532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d240cfa00 0x7f9d2414d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:02.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d2259c700 1 -- 172.21.15.189:0/4245123532 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d14009550 con 0x7f9d240cb870 2024-04-06T12:25:02.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d2259c700 1 --2- 172.21.15.189:0/4245123532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d240cb870 0x7f9d241f4ec0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800c4e0 tx=0x7f9d1800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:02.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d12ffd700 1 -- 172.21.15.189:0/4245123532 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d180052d0 con 0x7f9d240cb870 2024-04-06T12:25:02.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d12ffd700 1 -- 172.21.15.189:0/4245123532 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d18005430 con 0x7f9d240cb870 2024-04-06T12:25:02.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d28e3a700 1 -- 172.21.15.189:0/4245123532 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d24156aa0 con 0x7f9d240cb870 2024-04-06T12:25:02.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.925+0000 7f9d12ffd700 1 -- 172.21.15.189:0/4245123532 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d180038c0 con 0x7f9d240cb870 2024-04-06T12:25:02.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.926+0000 7f9d28e3a700 1 -- 172.21.15.189:0/4245123532 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d24157060 con 0x7f9d240cb870 2024-04-06T12:25:02.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.927+0000 7f9d28e3a700 1 -- 172.21.15.189:0/4245123532 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d24074990 con 0x7f9d240cb870 2024-04-06T12:25:02.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.927+0000 7f9d12ffd700 1 -- 172.21.15.189:0/4245123532 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f9d18003a20 con 0x7f9d240cb870 2024-04-06T12:25:02.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.927+0000 7f9d12ffd700 1 --2- 172.21.15.189:0/4245123532 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9d0406adb0 0x7f9d0406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:02.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.927+0000 7f9d21d9b700 1 --2- 172.21.15.189:0/4245123532 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9d0406adb0 0x7f9d0406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:02.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.927+0000 7f9d12ffd700 1 -- 172.21.15.189:0/4245123532 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9d180888e0 con 0x7f9d240cb870 2024-04-06T12:25:02.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.928+0000 7f9d21d9b700 1 --2- 172.21.15.189:0/4245123532 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9d0406adb0 0x7f9d0406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9d240cbcb0 tx=0x7f9d14008030 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:02.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:02.932+0000 7f9d12ffd700 1 -- 172.21.15.189:0/4245123532 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9d18053900 con 0x7f9d240cb870 2024-04-06T12:25:03.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:03.174+0000 7f9d28e3a700 1 -- 172.21.15.189:0/4245123532 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9d240d1e20 con 0x7f9d240cb870 2024-04-06T12:25:04.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.078+0000 7f9d12ffd700 1 -- 172.21.15.189:0/4245123532 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v23) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9d180170b0 con 0x7f9d240cb870 2024-04-06T12:25:04.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.081+0000 7f9d28e3a700 1 -- 172.21.15.189:0/4245123532 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9d0406adb0 msgr2=0x7f9d0406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:04.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.081+0000 7f9d28e3a700 1 --2- 172.21.15.189:0/4245123532 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9d0406adb0 0x7f9d0406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9d240cbcb0 tx=0x7f9d14008030 comp rx=0 tx=0).stop 2024-04-06T12:25:04.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.081+0000 7f9d28e3a700 1 -- 172.21.15.189:0/4245123532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d240cb870 msgr2=0x7f9d241f4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:04.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.081+0000 7f9d28e3a700 1 --2- 172.21.15.189:0/4245123532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d240cb870 0x7f9d241f4ec0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800c4e0 tx=0x7f9d1800c7f0 comp rx=0 tx=0).stop 2024-04-06T12:25:04.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.081+0000 7f9d28e3a700 1 -- 172.21.15.189:0/4245123532 shutdown_connections 2024-04-06T12:25:04.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.081+0000 7f9d28e3a700 1 --2- 172.21.15.189:0/4245123532 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d240d11b0 0x7f9d24159cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:04.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.081+0000 7f9d28e3a700 1 --2- 172.21.15.189:0/4245123532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d240cb870 0x7f9d241f4ec0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:04.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.081+0000 7f9d28e3a700 1 --2- 172.21.15.189:0/4245123532 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9d0406adb0 0x7f9d0406d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:04.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.081+0000 7f9d28e3a700 1 --2- 172.21.15.189:0/4245123532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d240cfa00 0x7f9d2414d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:04.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.081+0000 7f9d28e3a700 1 -- 172.21.15.189:0/4245123532 >> 172.21.15.189:0/4245123532 conn(0x7f9d240b2db0 msgr2=0x7f9d2406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:04.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.081+0000 7f9d28e3a700 1 -- 172.21.15.189:0/4245123532 shutdown_connections 2024-04-06T12:25:04.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.081+0000 7f9d28e3a700 1 -- 172.21.15.189:0/4245123532 wait complete. 2024-04-06T12:25:04.095 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-06T12:25:04.096 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-06T12:25:04.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.231+0000 7f609da02700 1 Processor -- start 2024-04-06T12:25:04.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.231+0000 7f609da02700 1 -- start start 2024-04-06T12:25:04.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.232+0000 7f609da02700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60980ce730 0x7f609815dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:04.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.232+0000 7f609da02700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f60980b8d70 0x7f60980d1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:04.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.232+0000 7f609da02700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f609815eac0 con 0x7f609815d6f0 2024-04-06T12:25:04.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.232+0000 7f609da02700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f609815e010 con 0x7f60980b8d70 2024-04-06T12:25:04.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.232+0000 7f609da02700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f609815e220 con 0x7f60980ce730 2024-04-06T12:25:04.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.232+0000 7f60977fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f60980b8d70 0x7f60980d1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:04.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.232+0000 7f60977fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f60980b8d70 0x7f60980d1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57854/0 (socket says 172.21.15.189:57854) 2024-04-06T12:25:04.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.232+0000 7f60977fe700 1 -- 172.21.15.189:0/1250051992 learned_addr learned my addr 172.21.15.189:0/1250051992 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:04.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.232+0000 7f60967fc700 1 --2- 172.21.15.189:0/1250051992 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60980ce730 0x7f609815dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:04.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.232+0000 7f60977fe700 1 -- 172.21.15.189:0/1250051992 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60980ce730 msgr2=0x7f609815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:04.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.232+0000 7f60977fe700 1 --2- 172.21.15.189:0/1250051992 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60980ce730 0x7f609815dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:04.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.232+0000 7f60977fe700 1 -- 172.21.15.189:0/1250051992 >> v1:172.21.15.192:6789/0 conn(0x7f609815d6f0 legacy=0x7f60980ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:04.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.232+0000 7f60977fe700 1 -- 172.21.15.189:0/1250051992 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609815e500 con 0x7f60980b8d70 2024-04-06T12:25:04.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.232+0000 7f60967fc700 1 --2- 172.21.15.189:0/1250051992 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60980ce730 0x7f609815dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:25:04.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.232+0000 7f60977fe700 1 --2- 172.21.15.189:0/1250051992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60980b8d70 0x7f60980d1e00 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f6090009e50 tx=0x7f609000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b9d3f7893141c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:04.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.232+0000 7f6095ffb700 1 -- 172.21.15.189:0/1250051992 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609000e070 con 0x7f60980b8d70 2024-04-06T12:25:04.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.233+0000 7f6095ffb700 1 -- 172.21.15.189:0/1250051992 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609000b9c0 con 0x7f60980b8d70 2024-04-06T12:25:04.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.233+0000 7f6095ffb700 1 -- 172.21.15.189:0/1250051992 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609000bd70 con 0x7f60980b8d70 2024-04-06T12:25:04.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.233+0000 7f609da02700 1 -- 172.21.15.189:0/1250051992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60980b8d70 msgr2=0x7f60980d1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:04.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.233+0000 7f609da02700 1 --2- 172.21.15.189:0/1250051992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60980b8d70 0x7f60980d1e00 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f6090009e50 tx=0x7f609000b330 comp rx=0 tx=0).stop 2024-04-06T12:25:04.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.233+0000 7f609da02700 1 -- 172.21.15.189:0/1250051992 shutdown_connections 2024-04-06T12:25:04.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.233+0000 7f609da02700 1 --2- 172.21.15.189:0/1250051992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60980b8d70 0x7f60980d1e00 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:04.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.233+0000 7f609da02700 1 --2- 172.21.15.189:0/1250051992 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60980ce730 0x7f609815dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:04.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.233+0000 7f609da02700 1 -- 172.21.15.189:0/1250051992 >> 172.21.15.189:0/1250051992 conn(0x7f6098072490 msgr2=0x7f609806b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:04.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.233+0000 7f609da02700 1 -- 172.21.15.189:0/1250051992 shutdown_connections 2024-04-06T12:25:04.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.233+0000 7f609da02700 1 -- 172.21.15.189:0/1250051992 wait complete. 2024-04-06T12:25:04.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.234+0000 7f609da02700 1 Processor -- start 2024-04-06T12:25:04.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.235+0000 7f609da02700 1 -- start start 2024-04-06T12:25:04.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.235+0000 7f609da02700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60980b8d70 0x7f60980ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:04.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.235+0000 7f609da02700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60980ce730 0x7f60980c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:04.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.235+0000 7f609da02700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f609815d6f0 0x7f609815aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:04.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.235+0000 7f609da02700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60980c9310 con 0x7f60980b8d70 2024-04-06T12:25:04.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.235+0000 7f609da02700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60980c8700 con 0x7f60980ce730 2024-04-06T12:25:04.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.235+0000 7f609da02700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60980c89e0 con 0x7f609815d6f0 2024-04-06T12:25:04.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.235+0000 7f6096ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60980b8d70 0x7f60980ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:04.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.235+0000 7f6096ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60980b8d70 0x7f60980ccfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57862/0 (socket says 172.21.15.189:57862) 2024-04-06T12:25:04.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.235+0000 7f6096ffd700 1 -- 172.21.15.189:0/3478171637 learned_addr learned my addr 172.21.15.189:0/3478171637 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:04.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.236+0000 7f60967fc700 1 --2- 172.21.15.189:0/3478171637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60980ce730 0x7f60980c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:04.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.236+0000 7f6096ffd700 1 -- 172.21.15.189:0/3478171637 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f609815d6f0 msgr2=0x7f609815aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:04.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.236+0000 7f6096ffd700 1 --2- 172.21.15.189:0/3478171637 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f609815d6f0 0x7f609815aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:04.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.236+0000 7f6096ffd700 1 -- 172.21.15.189:0/3478171637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60980ce730 msgr2=0x7f60980c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:04.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.236+0000 7f6096ffd700 1 --2- 172.21.15.189:0/3478171637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60980ce730 0x7f60980c97a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:04.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.236+0000 7f6096ffd700 1 -- 172.21.15.189:0/3478171637 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6090009b30 con 0x7f60980b8d70 2024-04-06T12:25:04.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.236+0000 7f6096ffd700 1 --2- 172.21.15.189:0/3478171637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60980b8d70 0x7f60980ccfe0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f608c00c450 tx=0x7f608c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:04.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.236+0000 7f607ffff700 1 -- 172.21.15.189:0/3478171637 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608c0052d0 con 0x7f60980b8d70 2024-04-06T12:25:04.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.236+0000 7f607ffff700 1 -- 172.21.15.189:0/3478171637 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f608c005430 con 0x7f60980b8d70 2024-04-06T12:25:04.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.236+0000 7f609da02700 1 -- 172.21.15.189:0/3478171637 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60980c8cc0 con 0x7f60980b8d70 2024-04-06T12:25:04.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.236+0000 7f607ffff700 1 -- 172.21.15.189:0/3478171637 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608c0038c0 con 0x7f60980b8d70 2024-04-06T12:25:04.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.236+0000 7f609da02700 1 -- 172.21.15.189:0/3478171637 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60980c9ce0 con 0x7f60980b8d70 2024-04-06T12:25:04.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.237+0000 7f609da02700 1 -- 172.21.15.189:0/3478171637 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6098074990 con 0x7f60980b8d70 2024-04-06T12:25:04.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.238+0000 7f607ffff700 1 -- 172.21.15.189:0/3478171637 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f608c003a20 con 0x7f60980b8d70 2024-04-06T12:25:04.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.238+0000 7f607ffff700 1 --2- 172.21.15.189:0/3478171637 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f608406adb0 0x7f608406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:04.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.238+0000 7f607ffff700 1 -- 172.21.15.189:0/3478171637 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f608c088b00 con 0x7f60980b8d70 2024-04-06T12:25:04.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.239+0000 7f60967fc700 1 --2- 172.21.15.189:0/3478171637 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f608406adb0 0x7f608406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:04.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.239+0000 7f60967fc700 1 --2- 172.21.15.189:0/3478171637 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f608406adb0 0x7f608406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f60980ca430 tx=0x7f6088008040 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:04.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.242+0000 7f607ffff700 1 -- 172.21.15.189:0/3478171637 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f608c053a70 con 0x7f60980b8d70 2024-04-06T12:25:04.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:04.482+0000 7f609da02700 1 -- 172.21.15.189:0/3478171637 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f60980ce240 con 0x7f60980b8d70 2024-04-06T12:25:05.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.084+0000 7f607ffff700 1 -- 172.21.15.189:0/3478171637 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7f608c057090 con 0x7f60980b8d70 2024-04-06T12:25:05.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.087+0000 7f609da02700 1 -- 172.21.15.189:0/3478171637 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f608406adb0 msgr2=0x7f608406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.087+0000 7f609da02700 1 --2- 172.21.15.189:0/3478171637 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f608406adb0 0x7f608406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f60980ca430 tx=0x7f6088008040 comp rx=0 tx=0).stop 2024-04-06T12:25:05.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.087+0000 7f609da02700 1 -- 172.21.15.189:0/3478171637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60980b8d70 msgr2=0x7f60980ccfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.087+0000 7f609da02700 1 --2- 172.21.15.189:0/3478171637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60980b8d70 0x7f60980ccfe0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f608c00c450 tx=0x7f608c00c810 comp rx=0 tx=0).stop 2024-04-06T12:25:05.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.087+0000 7f609da02700 1 -- 172.21.15.189:0/3478171637 shutdown_connections 2024-04-06T12:25:05.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.087+0000 7f609da02700 1 --2- 172.21.15.189:0/3478171637 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f609815d6f0 0x7f609815aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.087+0000 7f609da02700 1 --2- 172.21.15.189:0/3478171637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60980b8d70 0x7f60980ccfe0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.087+0000 7f609da02700 1 --2- 172.21.15.189:0/3478171637 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f608406adb0 0x7f608406d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.087+0000 7f609da02700 1 --2- 172.21.15.189:0/3478171637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60980ce730 0x7f60980c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.087+0000 7f609da02700 1 -- 172.21.15.189:0/3478171637 >> 172.21.15.189:0/3478171637 conn(0x7f6098072490 msgr2=0x7f60980b67c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:05.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.087+0000 7f609da02700 1 -- 172.21.15.189:0/3478171637 shutdown_connections 2024-04-06T12:25:05.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.088+0000 7f609da02700 1 -- 172.21.15.189:0/3478171637 wait complete. 2024-04-06T12:25:05.101 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-06T12:25:05.101 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:25:05.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.231+0000 7f47f7c48700 1 Processor -- start 2024-04-06T12:25:05.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.231+0000 7f47f7c48700 1 -- start start 2024-04-06T12:25:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.232+0000 7f47f7c48700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f47f00cea80 0x7f47f00cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.232+0000 7f47f7c48700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47f00cdae0 0x7f47f006ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.232+0000 7f47f7c48700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f015a830 con 0x7f47f00aec70 2024-04-06T12:25:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.232+0000 7f47f7c48700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f00cd920 con 0x7f47f00cea80 2024-04-06T12:25:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.232+0000 7f47f7c48700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f00cc050 con 0x7f47f00cdae0 2024-04-06T12:25:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.232+0000 7f47f61e5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47f00cdae0 0x7f47f006ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.232+0000 7f47f51e3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f47f00cea80 0x7f47f00cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.232+0000 7f47f51e3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f47f00cea80 0x7f47f00cd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:47346/0 (socket says 172.21.15.189:47346) 2024-04-06T12:25:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.232+0000 7f47f61e5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47f00cdae0 0x7f47f006ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:49748/0 (socket says 172.21.15.189:49748) 2024-04-06T12:25:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.232+0000 7f47f51e3700 1 -- 172.21.15.189:0/1720767246 learned_addr learned my addr 172.21.15.189:0/1720767246 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:05.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.232+0000 7f47f59e4700 1 --1- 172.21.15.189:0/1720767246 >> v1:172.21.15.192:6789/0 conn(0x7f47f00aec70 0x7f47f00ce370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:55868/0 (socket says 172.21.15.189:55868) 2024-04-06T12:25:05.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.232+0000 7f47f51e3700 1 -- 172.21.15.189:0/1720767246 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47f00cdae0 msgr2=0x7f47f006ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:05.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.232+0000 7f47f51e3700 1 --2- 172.21.15.189:0/1720767246 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47f00cdae0 0x7f47f006ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.232+0000 7f47f51e3700 1 -- 172.21.15.189:0/1720767246 >> v1:172.21.15.192:6789/0 conn(0x7f47f00aec70 legacy=0x7f47f00ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.232+0000 7f47f51e3700 1 -- 172.21.15.189:0/1720767246 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f00666f0 con 0x7f47f00cea80 2024-04-06T12:25:05.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.232+0000 7f47f61e5700 1 --2- 172.21.15.189:0/1720767246 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47f00cdae0 0x7f47f006ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:25:05.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.232+0000 7f47f51e3700 1 --2- 172.21.15.189:0/1720767246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47f00cea80 0x7f47f00cd3e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f47e00098f0 tx=0x7f47e0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2eac6083b360a9d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:05.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.233+0000 7f47f49e2700 1 -- 172.21.15.189:0/1720767246 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e0011830 con 0x7f47f00cea80 2024-04-06T12:25:05.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.233+0000 7f47f49e2700 1 -- 172.21.15.189:0/1720767246 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e0011990 con 0x7f47f00cea80 2024-04-06T12:25:05.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.233+0000 7f47f49e2700 1 -- 172.21.15.189:0/1720767246 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e0011d20 con 0x7f47f00cea80 2024-04-06T12:25:05.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.233+0000 7f47f7c48700 1 -- 172.21.15.189:0/1720767246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47f00cea80 msgr2=0x7f47f00cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.233+0000 7f47f7c48700 1 --2- 172.21.15.189:0/1720767246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47f00cea80 0x7f47f00cd3e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f47e00098f0 tx=0x7f47e0009c00 comp rx=0 tx=0).stop 2024-04-06T12:25:05.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.233+0000 7f47f7c48700 1 -- 172.21.15.189:0/1720767246 shutdown_connections 2024-04-06T12:25:05.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.233+0000 7f47f7c48700 1 --2- 172.21.15.189:0/1720767246 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47f00cdae0 0x7f47f006ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.233+0000 7f47f7c48700 1 --2- 172.21.15.189:0/1720767246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47f00cea80 0x7f47f00cd3e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.233+0000 7f47f7c48700 1 -- 172.21.15.189:0/1720767246 >> 172.21.15.189:0/1720767246 conn(0x7f47f00cc290 msgr2=0x7f47f00cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:05.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.233+0000 7f47f7c48700 1 -- 172.21.15.189:0/1720767246 shutdown_connections 2024-04-06T12:25:05.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.233+0000 7f47f7c48700 1 -- 172.21.15.189:0/1720767246 wait complete. 2024-04-06T12:25:05.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.235+0000 7f47f7c48700 1 Processor -- start 2024-04-06T12:25:05.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.235+0000 7f47f7c48700 1 -- start start 2024-04-06T12:25:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.236+0000 7f47f7c48700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47f00aec70 0x7f47f00c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.236+0000 7f47f7c48700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47f00cdae0 0x7f47f014ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.236+0000 7f47f7c48700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47f00cea80 0x7f47f00c9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.236+0000 7f47f7c48700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f014b1a0 con 0x7f47f00aec70 2024-04-06T12:25:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.236+0000 7f47f7c48700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f00c7ff0 con 0x7f47f00cea80 2024-04-06T12:25:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.236+0000 7f47f7c48700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f00c82d0 con 0x7f47f00cdae0 2024-04-06T12:25:05.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.236+0000 7f47f59e4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47f00aec70 0x7f47f00c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.236+0000 7f47f59e4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47f00aec70 0x7f47f00c7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57864/0 (socket says 172.21.15.189:57864) 2024-04-06T12:25:05.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.236+0000 7f47f59e4700 1 -- 172.21.15.189:0/1750697660 learned_addr learned my addr 172.21.15.189:0/1750697660 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:05.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.236+0000 7f47f51e3700 1 --2- 172.21.15.189:0/1750697660 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47f00cdae0 0x7f47f014ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.236+0000 7f47f61e5700 1 --2- 172.21.15.189:0/1750697660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47f00cea80 0x7f47f00c9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.236+0000 7f47f59e4700 1 -- 172.21.15.189:0/1750697660 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47f00cdae0 msgr2=0x7f47f014ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.236+0000 7f47f59e4700 1 --2- 172.21.15.189:0/1750697660 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47f00cdae0 0x7f47f014ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.236+0000 7f47f59e4700 1 -- 172.21.15.189:0/1750697660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47f00cea80 msgr2=0x7f47f00c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.236+0000 7f47f59e4700 1 --2- 172.21.15.189:0/1750697660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47f00cea80 0x7f47f00c9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.236+0000 7f47f59e4700 1 -- 172.21.15.189:0/1750697660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e8009a70 con 0x7f47f00aec70 2024-04-06T12:25:05.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.236+0000 7f47f59e4700 1 --2- 172.21.15.189:0/1750697660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47f00aec70 0x7f47f00c7a10 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f47ec00c490 tx=0x7f47ec00c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:05.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.237+0000 7f47e67fc700 1 -- 172.21.15.189:0/1750697660 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec005cb0 con 0x7f47f00aec70 2024-04-06T12:25:05.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.237+0000 7f47e67fc700 1 -- 172.21.15.189:0/1750697660 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ec00ce90 con 0x7f47f00aec70 2024-04-06T12:25:05.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.237+0000 7f47f7c48700 1 -- 172.21.15.189:0/1750697660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e0009550 con 0x7f47f00aec70 2024-04-06T12:25:05.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.237+0000 7f47e67fc700 1 -- 172.21.15.189:0/1750697660 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec0072d0 con 0x7f47f00aec70 2024-04-06T12:25:05.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.237+0000 7f47f7c48700 1 -- 172.21.15.189:0/1750697660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f00c8890 con 0x7f47f00aec70 2024-04-06T12:25:05.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.238+0000 7f47f7c48700 1 -- 172.21.15.189:0/1750697660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f0074990 con 0x7f47f00aec70 2024-04-06T12:25:05.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.238+0000 7f47e67fc700 1 -- 172.21.15.189:0/1750697660 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f47ec007430 con 0x7f47f00aec70 2024-04-06T12:25:05.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.238+0000 7f47e67fc700 1 --2- 172.21.15.189:0/1750697660 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f47d406adb0 0x7f47d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.239+0000 7f47e67fc700 1 -- 172.21.15.189:0/1750697660 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f47ec088760 con 0x7f47f00aec70 2024-04-06T12:25:05.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.239+0000 7f47f51e3700 1 --2- 172.21.15.189:0/1750697660 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f47d406adb0 0x7f47d406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.239+0000 7f47f51e3700 1 --2- 172.21.15.189:0/1750697660 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f47d406adb0 0x7f47d406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f47e0014040 tx=0x7f47e0008030 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:05.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.243+0000 7f47e67fc700 1 -- 172.21.15.189:0/1750697660 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f47ec053830 con 0x7f47f00aec70 2024-04-06T12:25:05.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.480+0000 7f47f7c48700 1 -- 172.21.15.189:0/1750697660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47f006bc70 con 0x7f47f00aec70 2024-04-06T12:25:05.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.481+0000 7f47e67fc700 1 -- 172.21.15.189:0/1750697660 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f47ec056e50 con 0x7f47f00aec70 2024-04-06T12:25:05.479 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:25:05.479 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/527093473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":527093473},{"type":"v1","addr":"172.21.15.192:6835","nonce":527093473}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6105,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2142144473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2142144473},{"type":"v1","addr":"172.21.15.189:6836","nonce":2142144473}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6108,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2285762709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2285762709},{"type":"v1","addr":"172.21.15.189:6837","nonce":2285762709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:25:02.062461+0000","modified":"2024-04-06T12:25:05.077648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6834/525827513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":525827513},{"type":"v1","addr":"172.21.15.192:6834","nonce":525827513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-06T12:25:05.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.483+0000 7f47f7c48700 1 -- 172.21.15.189:0/1750697660 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f47d406adb0 msgr2=0x7f47d406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.483+0000 7f47f7c48700 1 --2- 172.21.15.189:0/1750697660 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f47d406adb0 0x7f47d406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f47e0014040 tx=0x7f47e0008030 comp rx=0 tx=0).stop 2024-04-06T12:25:05.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.483+0000 7f47f7c48700 1 -- 172.21.15.189:0/1750697660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47f00aec70 msgr2=0x7f47f00c7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.483+0000 7f47f7c48700 1 --2- 172.21.15.189:0/1750697660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47f00aec70 0x7f47f00c7a10 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f47ec00c490 tx=0x7f47ec00c7a0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.483+0000 7f47f7c48700 1 -- 172.21.15.189:0/1750697660 shutdown_connections 2024-04-06T12:25:05.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.483+0000 7f47f7c48700 1 --2- 172.21.15.189:0/1750697660 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f47d406adb0 0x7f47d406d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.483+0000 7f47f7c48700 1 --2- 172.21.15.189:0/1750697660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47f00cea80 0x7f47f00c9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.483+0000 7f47f7c48700 1 --2- 172.21.15.189:0/1750697660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47f00aec70 0x7f47f00c7a10 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.483+0000 7f47f7c48700 1 --2- 172.21.15.189:0/1750697660 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47f00cdae0 0x7f47f014ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.483+0000 7f47f7c48700 1 -- 172.21.15.189:0/1750697660 >> 172.21.15.189:0/1750697660 conn(0x7f47f00cc290 msgr2=0x7f47f00af1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:05.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.483+0000 7f47f7c48700 1 -- 172.21.15.189:0/1750697660 shutdown_connections 2024-04-06T12:25:05.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.483+0000 7f47f7c48700 1 -- 172.21.15.189:0/1750697660 wait complete. 2024-04-06T12:25:05.483 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 24 2024-04-06T12:25:05.494 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:25:05.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.617+0000 7efefc16e700 1 Processor -- start 2024-04-06T12:25:05.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.617+0000 7efefc16e700 1 -- start start 2024-04-06T12:25:05.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.618+0000 7efefc16e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efef40b4660 0x7efef40b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.618+0000 7efefc16e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef40cdbe0 con 0x7efef40b2fc0 2024-04-06T12:25:05.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.618+0000 7efefc16e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef40b5000 con 0x7efef415a550 2024-04-06T12:25:05.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.618+0000 7efefc16e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef415a940 con 0x7efef40b4660 2024-04-06T12:25:05.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.618+0000 7efefa70b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efef40b2fc0 0x7efef40b3430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37264/0 (socket says 172.21.15.189:37264) 2024-04-06T12:25:05.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.618+0000 7efefa70b700 1 -- 172.21.15.189:0/1971910302 learned_addr learned my addr 172.21.15.189:0/1971910302 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:25:05.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.618+0000 7efef9709700 1 --2- 172.21.15.189:0/1971910302 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efef40b4660 0x7efef40b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef8f08700 1 -- 172.21.15.189:0/1971910302 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167832313 0 0) 0x7efef40cdbe0 con 0x7efef40b2fc0 2024-04-06T12:25:05.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef8f08700 1 -- 172.21.15.189:0/1971910302 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efee00035b0 con 0x7efef40b2fc0 2024-04-06T12:25:05.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef8f08700 1 -- 172.21.15.189:0/1971910302 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036646388 0 0) 0x7efef40b5000 con 0x7efef415a550 2024-04-06T12:25:05.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef8f08700 1 -- 172.21.15.189:0/1971910302 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efef40cdbe0 con 0x7efef415a550 2024-04-06T12:25:05.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef8f08700 1 -- 172.21.15.189:0/1971910302 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2565732595 0 0) 0x7efee00035b0 con 0x7efef40b2fc0 2024-04-06T12:25:05.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef8f08700 1 -- 172.21.15.189:0/1971910302 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efef40b4660 msgr2=0x7efef40b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:05.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef8f08700 1 --2- 172.21.15.189:0/1971910302 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efef40b4660 0x7efef40b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef8f08700 1 -- 172.21.15.189:0/1971910302 >> v1:172.21.15.192:6789/0 conn(0x7efef415a550 legacy=0x7efef40cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef8f08700 1 -- 172.21.15.189:0/1971910302 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef415ac00 con 0x7efef40b2fc0 2024-04-06T12:25:05.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef8f08700 1 -- 172.21.15.189:0/1971910302 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7efeec0019f0 con 0x7efef40b2fc0 2024-04-06T12:25:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef8f08700 1 -- 172.21.15.189:0/1971910302 >> v1:172.21.15.189:6789/0 conn(0x7efef40b2fc0 legacy=0x7efef40b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef8f08700 1 --2- 172.21.15.189:0/1971910302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efee0003bc0 0x7efee0003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef8f08700 1 --2- 172.21.15.189:0/1971910302 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efef40b4660 0x7efee0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef8f08700 1 --2- 172.21.15.189:0/1971910302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efee0004d90 0x7efee0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef8f08700 1 -- 172.21.15.189:0/1971910302 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef415a940 con 0x7efee0003bc0 2024-04-06T12:25:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef8f08700 1 -- 172.21.15.189:0/1971910302 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee000dcd0 con 0x7efee0004d90 2024-04-06T12:25:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef8f08700 1 -- 172.21.15.189:0/1971910302 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee000de90 con 0x7efef40b4660 2024-04-06T12:25:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef9f0a700 1 --2- 172.21.15.189:0/1971910302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efee0003bc0 0x7efee0003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efefa70b700 1 --2- 172.21.15.189:0/1971910302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efee0004d90 0x7efee0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef9709700 1 --2- 172.21.15.189:0/1971910302 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efef40b4660 0x7efee0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef9f0a700 1 -- 172.21.15.189:0/1971910302 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efef40b4660 msgr2=0x7efee0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef9f0a700 1 --2- 172.21.15.189:0/1971910302 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efef40b4660 0x7efee0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef9f0a700 1 -- 172.21.15.189:0/1971910302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efee0004d90 msgr2=0x7efee0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef9f0a700 1 --2- 172.21.15.189:0/1971910302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efee0004d90 0x7efee0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef9f0a700 1 -- 172.21.15.189:0/1971910302 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee00035b0 con 0x7efee0003bc0 2024-04-06T12:25:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efefa70b700 1 --2- 172.21.15.189:0/1971910302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efee0004d90 0x7efee0005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:25:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.619+0000 7efef9f0a700 1 --2- 172.21.15.189:0/1971910302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efee0003bc0 0x7efee0003fc0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7efee4001940 tx=0x7efee0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aeb5cfd989fed39 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.620+0000 7efef8f08700 1 -- 172.21.15.189:0/1971910302 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee4001430 con 0x7efee0003bc0 2024-04-06T12:25:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.620+0000 7efef8f08700 1 -- 172.21.15.189:0/1971910302 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee4001d70 con 0x7efee0003bc0 2024-04-06T12:25:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.620+0000 7efef8f08700 1 -- 172.21.15.189:0/1971910302 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee4005510 con 0x7efee0003bc0 2024-04-06T12:25:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.620+0000 7efefc16e700 1 -- 172.21.15.189:0/1971910302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efee0003bc0 msgr2=0x7efee0003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.620+0000 7efefc16e700 1 --2- 172.21.15.189:0/1971910302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efee0003bc0 0x7efee0003fc0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7efee4001940 tx=0x7efee0005880 comp rx=0 tx=0).stop 2024-04-06T12:25:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.620+0000 7efefa70b700 1 -- 172.21.15.189:0/1971910302 reap_dead start 2024-04-06T12:25:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.620+0000 7efefc16e700 1 -- 172.21.15.189:0/1971910302 shutdown_connections 2024-04-06T12:25:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.620+0000 7efefc16e700 1 -- 172.21.15.189:0/1971910302 >> 172.21.15.189:0/1971910302 conn(0x7efef40d00b0 msgr2=0x7efef40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.620+0000 7efefc16e700 1 -- 172.21.15.189:0/1971910302 shutdown_connections 2024-04-06T12:25:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.620+0000 7efefc16e700 1 -- 172.21.15.189:0/1971910302 wait complete. 2024-04-06T12:25:05.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.622+0000 7efefc16e700 1 Processor -- start 2024-04-06T12:25:05.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.622+0000 7efefc16e700 1 -- start start 2024-04-06T12:25:05.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efefc16e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efee0003bc0 0x7efef4134900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efefc16e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efef40b14a0 0x7efef40ce590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efefc16e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef40cead0 0x7efef415c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efefc16e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef414fae0 con 0x7efef40cead0 2024-04-06T12:25:05.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efefc16e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef40cef80 con 0x7efef40b14a0 2024-04-06T12:25:05.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efefc16e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef414c650 con 0x7efee0003bc0 2024-04-06T12:25:05.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efefa70b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef40cead0 0x7efef415c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efefa70b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef40cead0 0x7efef415c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57882/0 (socket says 172.21.15.189:57882) 2024-04-06T12:25:05.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efefa70b700 1 -- 172.21.15.189:0/3229688207 learned_addr learned my addr 172.21.15.189:0/3229688207 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:05.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efef9f0a700 1 --2- 172.21.15.189:0/3229688207 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efee0003bc0 0x7efef4134900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efef9709700 1 --2- 172.21.15.189:0/3229688207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efef40b14a0 0x7efef40ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efefa70b700 1 -- 172.21.15.189:0/3229688207 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efee0003bc0 msgr2=0x7efef4134900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efefa70b700 1 --2- 172.21.15.189:0/3229688207 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efee0003bc0 0x7efef4134900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efefa70b700 1 -- 172.21.15.189:0/3229688207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efef40b14a0 msgr2=0x7efef40ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efefa70b700 1 --2- 172.21.15.189:0/3229688207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efef40b14a0 0x7efef40ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efefa70b700 1 -- 172.21.15.189:0/3229688207 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef4134e40 con 0x7efef40cead0 2024-04-06T12:25:05.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efefa70b700 1 --2- 172.21.15.189:0/3229688207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef40cead0 0x7efef415c910 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7efef40b6230 tx=0x7efeec007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:05.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efeea7fc700 1 -- 172.21.15.189:0/3229688207 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec017070 con 0x7efef40cead0 2024-04-06T12:25:05.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efeea7fc700 1 -- 172.21.15.189:0/3229688207 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeec00ebd0 con 0x7efef40cead0 2024-04-06T12:25:05.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efefc16e700 1 -- 172.21.15.189:0/3229688207 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef413eb50 con 0x7efef40cead0 2024-04-06T12:25:05.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efeea7fc700 1 -- 172.21.15.189:0/3229688207 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec00ed60 con 0x7efef40cead0 2024-04-06T12:25:05.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.623+0000 7efefc16e700 1 -- 172.21.15.189:0/3229688207 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef414f140 con 0x7efef40cead0 2024-04-06T12:25:05.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.624+0000 7efefc16e700 1 -- 172.21.15.189:0/3229688207 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef414fc20 con 0x7efef40cead0 2024-04-06T12:25:05.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.625+0000 7efeea7fc700 1 -- 172.21.15.189:0/3229688207 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7efeec013420 con 0x7efef40cead0 2024-04-06T12:25:05.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.625+0000 7efeea7fc700 1 --2- 172.21.15.189:0/3229688207 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed806ade0 0x7efed806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.626+0000 7efeea7fc700 1 -- 172.21.15.189:0/3229688207 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7efeec089a90 con 0x7efef40cead0 2024-04-06T12:25:05.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.626+0000 7efef9f0a700 1 --2- 172.21.15.189:0/3229688207 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed806ade0 0x7efed806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.626+0000 7efef9f0a700 1 --2- 172.21.15.189:0/3229688207 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed806ade0 0x7efed806d2a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efee4015040 tx=0x7efee400a000 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:05.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.629+0000 7efeea7fc700 1 -- 172.21.15.189:0/3229688207 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efeec054a00 con 0x7efef40cead0 2024-04-06T12:25:05.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.825+0000 7efefc16e700 1 -- 172.21.15.189:0/3229688207 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efef414fc20 con 0x7efef40cead0 2024-04-06T12:25:05.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.826+0000 7efeea7fc700 1 -- 172.21.15.189:0/3229688207 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+16354 (secure 0 0 0) 0x7efef414fc20 con 0x7efef40cead0 2024-04-06T12:25:05.824 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:25:05.825 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":67,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:25:03.064289+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:24:59.828905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-06T12:25:01.020253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:25:05.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.828+0000 7efefc16e700 1 -- 172.21.15.189:0/3229688207 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed806ade0 msgr2=0x7efed806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.828+0000 7efefc16e700 1 --2- 172.21.15.189:0/3229688207 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed806ade0 0x7efed806d2a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efee4015040 tx=0x7efee400a000 comp rx=0 tx=0).stop 2024-04-06T12:25:05.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.828+0000 7efefc16e700 1 -- 172.21.15.189:0/3229688207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef40cead0 msgr2=0x7efef415c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.828+0000 7efefc16e700 1 --2- 172.21.15.189:0/3229688207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef40cead0 0x7efef415c910 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7efef40b6230 tx=0x7efeec007c90 comp rx=0 tx=0).stop 2024-04-06T12:25:05.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.828+0000 7efefc16e700 1 -- 172.21.15.189:0/3229688207 shutdown_connections 2024-04-06T12:25:05.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.828+0000 7efefc16e700 1 --2- 172.21.15.189:0/3229688207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef40cead0 0x7efef415c910 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.828+0000 7efefc16e700 1 --2- 172.21.15.189:0/3229688207 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efee0003bc0 0x7efef4134900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.828+0000 7efefc16e700 1 --2- 172.21.15.189:0/3229688207 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed806ade0 0x7efed806d2a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.828+0000 7efefc16e700 1 --2- 172.21.15.189:0/3229688207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efef40b14a0 0x7efef40ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.828+0000 7efefc16e700 1 -- 172.21.15.189:0/3229688207 >> 172.21.15.189:0/3229688207 conn(0x7efef40d00b0 msgr2=0x7efef40cf190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:05.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.829+0000 7efefc16e700 1 -- 172.21.15.189:0/3229688207 shutdown_connections 2024-04-06T12:25:05.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.829+0000 7efefc16e700 1 -- 172.21.15.189:0/3229688207 wait complete. 2024-04-06T12:25:05.840 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-06T12:25:05.840 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:25:05.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.963+0000 7fa4e81ec700 1 Processor -- start 2024-04-06T12:25:05.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.963+0000 7fa4e81ec700 1 -- start start 2024-04-06T12:25:05.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.964+0000 7fa4e81ec700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa4e015fcd0 0x7fa4e015c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.964+0000 7fa4e81ec700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e015c740 con 0x7fa4e006ae40 2024-04-06T12:25:05.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.964+0000 7fa4e81ec700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e015f1f0 con 0x7fa4e0144f40 2024-04-06T12:25:05.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.964+0000 7fa4e81ec700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4e00cfcb0 con 0x7fa4e015fcd0 2024-04-06T12:25:05.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.964+0000 7fa4e5787700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa4e006ae40 0x7fa4e015ba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37280/0 (socket says 172.21.15.189:37280) 2024-04-06T12:25:05.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e5787700 1 -- 172.21.15.189:0/3270779977 learned_addr learned my addr 172.21.15.189:0/3270779977 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:25:05.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e6789700 1 --2- 172.21.15.189:0/3270779977 >> v2:172.21.15.192:3300/0 conn(0x7fa4e015fcd0 0x7fa4e015c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e4f86700 1 -- 172.21.15.189:0/3270779977 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240156190 0 0) 0x7fa4e015c740 con 0x7fa4e006ae40 2024-04-06T12:25:05.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e4f86700 1 -- 172.21.15.189:0/3270779977 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4cc0035b0 con 0x7fa4e006ae40 2024-04-06T12:25:05.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e4f86700 1 -- 172.21.15.189:0/3270779977 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2187779719 0 0) 0x7fa4cc0035b0 con 0x7fa4e006ae40 2024-04-06T12:25:05.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e4f86700 1 -- 172.21.15.189:0/3270779977 >> v2:172.21.15.192:3300/0 conn(0x7fa4e015fcd0 msgr2=0x7fa4e015c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:05.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e4f86700 1 --2- 172.21.15.189:0/3270779977 >> v2:172.21.15.192:3300/0 conn(0x7fa4e015fcd0 0x7fa4e015c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e4f86700 1 -- 172.21.15.189:0/3270779977 >> v1:172.21.15.192:6789/0 conn(0x7fa4e0144f40 legacy=0x7fa4e006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e4f86700 1 -- 172.21.15.189:0/3270779977 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e015f4d0 con 0x7fa4e006ae40 2024-04-06T12:25:05.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e4f86700 1 -- 172.21.15.189:0/3270779977 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fa4dc001670 con 0x7fa4e006ae40 2024-04-06T12:25:05.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e4f86700 1 -- 172.21.15.189:0/3270779977 >> v1:172.21.15.189:6789/0 conn(0x7fa4e006ae40 legacy=0x7fa4e015ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e4f86700 1 --2- 172.21.15.189:0/3270779977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4cc003a00 0x7fa4cc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e4f86700 1 --2- 172.21.15.189:0/3270779977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4cc004850 0x7fa4cc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e4f86700 1 --2- 172.21.15.189:0/3270779977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4cc005210 0x7fa4cc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e4f86700 1 -- 172.21.15.189:0/3270779977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e00cfcb0 con 0x7fa4cc003a00 2024-04-06T12:25:05.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e4f86700 1 -- 172.21.15.189:0/3270779977 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4cc00c1b0 con 0x7fa4cc004850 2024-04-06T12:25:05.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e4f86700 1 -- 172.21.15.189:0/3270779977 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4cc00c370 con 0x7fa4cc005210 2024-04-06T12:25:05.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e5f88700 1 --2- 172.21.15.189:0/3270779977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4cc003a00 0x7fa4cc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e5787700 1 --2- 172.21.15.189:0/3270779977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4cc004850 0x7fa4cc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e6789700 1 --2- 172.21.15.189:0/3270779977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4cc005210 0x7fa4cc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.965+0000 7fa4e5f88700 1 -- 172.21.15.189:0/3270779977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4cc005210 msgr2=0x7fa4cc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:05.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.966+0000 7fa4e5f88700 1 --2- 172.21.15.189:0/3270779977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4cc005210 0x7fa4cc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.966+0000 7fa4e5f88700 1 -- 172.21.15.189:0/3270779977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4cc004850 msgr2=0x7fa4cc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:05.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.966+0000 7fa4e5f88700 1 --2- 172.21.15.189:0/3270779977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4cc004850 0x7fa4cc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.966+0000 7fa4e5f88700 1 -- 172.21.15.189:0/3270779977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4cc0035b0 con 0x7fa4cc003a00 2024-04-06T12:25:05.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.966+0000 7fa4e6789700 1 -- 172.21.15.189:0/3270779977 reap_dead start 2024-04-06T12:25:05.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.966+0000 7fa4e5f88700 1 --2- 172.21.15.189:0/3270779977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4cc003a00 0x7fa4cc003e20 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fa4d40017c0 tx=0x7fa4cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae216335eb6e0f6a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:05.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.966+0000 7fa4e4f86700 1 -- 172.21.15.189:0/3270779977 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d40052d0 con 0x7fa4cc003a00 2024-04-06T12:25:05.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.966+0000 7fa4e4f86700 1 -- 172.21.15.189:0/3270779977 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d4005450 con 0x7fa4cc003a00 2024-04-06T12:25:05.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.966+0000 7fa4e4f86700 1 -- 172.21.15.189:0/3270779977 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d40038c0 con 0x7fa4cc003a00 2024-04-06T12:25:05.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.966+0000 7fa4e81ec700 1 -- 172.21.15.189:0/3270779977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4cc003a00 msgr2=0x7fa4cc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.966+0000 7fa4e81ec700 1 --2- 172.21.15.189:0/3270779977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4cc003a00 0x7fa4cc003e20 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fa4d40017c0 tx=0x7fa4cc005880 comp rx=0 tx=0).stop 2024-04-06T12:25:05.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.966+0000 7fa4e81ec700 1 -- 172.21.15.189:0/3270779977 shutdown_connections 2024-04-06T12:25:05.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.966+0000 7fa4e81ec700 1 --2- 172.21.15.189:0/3270779977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4cc003a00 0x7fa4cc003e20 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.966+0000 7fa4e81ec700 1 -- 172.21.15.189:0/3270779977 >> 172.21.15.189:0/3270779977 conn(0x7fa4e00d00b0 msgr2=0x7fa4e00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:05.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.966+0000 7fa4e81ec700 1 -- 172.21.15.189:0/3270779977 shutdown_connections 2024-04-06T12:25:05.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.966+0000 7fa4e81ec700 1 -- 172.21.15.189:0/3270779977 wait complete. 2024-04-06T12:25:05.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.968+0000 7fa4e81ec700 1 Processor -- start 2024-04-06T12:25:05.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.968+0000 7fa4e81ec700 1 -- start start 2024-04-06T12:25:05.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.968+0000 7fa4e81ec700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4cc003a00 0x7fa4e013c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.968+0000 7fa4e81ec700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4e01365a0 0x7fa4e015acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.968+0000 7fa4e81ec700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4e0139b20 0x7fa4e0139fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.968+0000 7fa4e81ec700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e013ce70 con 0x7fa4cc003a00 2024-04-06T12:25:05.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.968+0000 7fa4e81ec700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e0136b60 con 0x7fa4e01365a0 2024-04-06T12:25:05.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.968+0000 7fa4e81ec700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e0136e40 con 0x7fa4e0139b20 2024-04-06T12:25:05.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.968+0000 7fa4e5f88700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4cc003a00 0x7fa4e013c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.968+0000 7fa4e5f88700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4cc003a00 0x7fa4e013c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57902/0 (socket says 172.21.15.189:57902) 2024-04-06T12:25:05.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.968+0000 7fa4e5f88700 1 -- 172.21.15.189:0/3949088697 learned_addr learned my addr 172.21.15.189:0/3949088697 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:05.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.969+0000 7fa4e6789700 1 --2- 172.21.15.189:0/3949088697 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4e0139b20 0x7fa4e0139fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.969+0000 7fa4e5787700 1 --2- 172.21.15.189:0/3949088697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4e01365a0 0x7fa4e015acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.969+0000 7fa4e5f88700 1 -- 172.21.15.189:0/3949088697 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4e0139b20 msgr2=0x7fa4e0139fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.969+0000 7fa4e5f88700 1 --2- 172.21.15.189:0/3949088697 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4e0139b20 0x7fa4e0139fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.969+0000 7fa4e5f88700 1 -- 172.21.15.189:0/3949088697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4e01365a0 msgr2=0x7fa4e015acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:05.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.969+0000 7fa4e5f88700 1 --2- 172.21.15.189:0/3949088697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4e01365a0 0x7fa4e015acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:05.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.969+0000 7fa4e5f88700 1 -- 172.21.15.189:0/3949088697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e0137120 con 0x7fa4cc003a00 2024-04-06T12:25:05.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.969+0000 7fa4e5787700 1 --2- 172.21.15.189:0/3949088697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4e01365a0 0x7fa4e015acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:25:05.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.969+0000 7fa4e5f88700 1 --2- 172.21.15.189:0/3949088697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4cc003a00 0x7fa4e013c930 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4000c00 tx=0x7fa4d4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:05.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.969+0000 7fa4d27fc700 1 -- 172.21.15.189:0/3949088697 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d4001290 con 0x7fa4cc003a00 2024-04-06T12:25:05.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.969+0000 7fa4d27fc700 1 -- 172.21.15.189:0/3949088697 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d4007040 con 0x7fa4cc003a00 2024-04-06T12:25:05.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.969+0000 7fa4e81ec700 1 -- 172.21.15.189:0/3949088697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e0137400 con 0x7fa4cc003a00 2024-04-06T12:25:05.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.969+0000 7fa4d27fc700 1 -- 172.21.15.189:0/3949088697 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d40073f0 con 0x7fa4cc003a00 2024-04-06T12:25:05.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.969+0000 7fa4e81ec700 1 -- 172.21.15.189:0/3949088697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e015b1f0 con 0x7fa4cc003a00 2024-04-06T12:25:05.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.970+0000 7fa4e81ec700 1 -- 172.21.15.189:0/3949088697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e0067bb0 con 0x7fa4cc003a00 2024-04-06T12:25:05.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.971+0000 7fa4d27fc700 1 -- 172.21.15.189:0/3949088697 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fa4d4007550 con 0x7fa4cc003a00 2024-04-06T12:25:05.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.971+0000 7fa4d27fc700 1 --2- 172.21.15.189:0/3949088697 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa4c406adb0 0x7fa4c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:05.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.971+0000 7fa4d27fc700 1 -- 172.21.15.189:0/3949088697 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa4d4089fc0 con 0x7fa4cc003a00 2024-04-06T12:25:05.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.971+0000 7fa4e5787700 1 --2- 172.21.15.189:0/3949088697 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa4c406adb0 0x7fa4c406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:05.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.971+0000 7fa4e5787700 1 --2- 172.21.15.189:0/3949088697 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa4c406adb0 0x7fa4c406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00cc900 tx=0x7fa4dc009040 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:05.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:05.975+0000 7fa4d27fc700 1 -- 172.21.15.189:0/3949088697 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa4d40bd650 con 0x7fa4cc003a00 2024-04-06T12:25:06.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.166+0000 7fa4e81ec700 1 -- 172.21.15.189:0/3949088697 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa4e0074990 con 0x7fa4c406adb0 2024-04-06T12:25:06.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.174+0000 7fa4d27fc700 1 -- 172.21.15.189:0/3949088697 <== mgr.5225 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289571 (secure 0 0 0) 0x7fa4e0074990 con 0x7fa4c406adb0 2024-04-06T12:25:06.177 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:25:06.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.178+0000 7fa4e81ec700 1 -- 172.21.15.189:0/3949088697 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa4c406adb0 msgr2=0x7fa4c406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.178+0000 7fa4e81ec700 1 --2- 172.21.15.189:0/3949088697 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa4c406adb0 0x7fa4c406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00cc900 tx=0x7fa4dc009040 comp rx=0 tx=0).stop 2024-04-06T12:25:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.178+0000 7fa4e81ec700 1 -- 172.21.15.189:0/3949088697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4cc003a00 msgr2=0x7fa4e013c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.178+0000 7fa4e81ec700 1 --2- 172.21.15.189:0/3949088697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4cc003a00 0x7fa4e013c930 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4000c00 tx=0x7fa4d4001120 comp rx=0 tx=0).stop 2024-04-06T12:25:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.178+0000 7fa4e81ec700 1 -- 172.21.15.189:0/3949088697 shutdown_connections 2024-04-06T12:25:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.178+0000 7fa4e81ec700 1 --2- 172.21.15.189:0/3949088697 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4e0139b20 0x7fa4e0139fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.178+0000 7fa4e81ec700 1 --2- 172.21.15.189:0/3949088697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4cc003a00 0x7fa4e013c930 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.178+0000 7fa4e81ec700 1 --2- 172.21.15.189:0/3949088697 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa4c406adb0 0x7fa4c406d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.178+0000 7fa4e81ec700 1 --2- 172.21.15.189:0/3949088697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4e01365a0 0x7fa4e015acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.178+0000 7fa4e81ec700 1 -- 172.21.15.189:0/3949088697 >> 172.21.15.189:0/3949088697 conn(0x7fa4e00d00b0 msgr2=0x7fa4e0143ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.178+0000 7fa4e81ec700 1 -- 172.21.15.189:0/3949088697 shutdown_connections 2024-04-06T12:25:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.178+0000 7fa4e81ec700 1 -- 172.21.15.189:0/3949088697 wait complete. 2024-04-06T12:25:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:25:06.188 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-06T12:25:04.396106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":284300,"kb_used_data":7428,"kb_used_omap":12,"kb_used_meta":276723,"kb_avail":754690420,"statfs":{"total":773094113280,"available":772802990080,"internally_reserved":0,"allocated":7606272,"data_stored":4295373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":283364920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000802"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.048034+0000","last_change":"2024-04-06T12:25:02.091668+0000","last_active":"2024-04-06T12:25:04.048034+0000","last_peered":"2024-04-06T12:25:04.048034+0000","last_clean":"2024-04-06T12:25:04.048034+0000","last_became_active":"2024-04-06T12:25:02.091427+0000","last_became_peered":"2024-04-06T12:25:02.091427+0000","last_unstale":"2024-04-06T12:25:04.048034+0000","last_undegraded":"2024-04-06T12:25:04.048034+0000","last_fullsized":"2024-04-06T12:25:04.048034+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:18:36.819518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073692+0000","last_change":"2024-04-06T12:25:02.093907+0000","last_active":"2024-04-06T12:25:03.073692+0000","last_peered":"2024-04-06T12:25:03.073692+0000","last_clean":"2024-04-06T12:25:03.073692+0000","last_became_active":"2024-04-06T12:25:02.093615+0000","last_became_peered":"2024-04-06T12:25:02.093615+0000","last_unstale":"2024-04-06T12:25:03.073692+0000","last_undegraded":"2024-04-06T12:25:03.073692+0000","last_fullsized":"2024-04-06T12:25:03.073692+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:39:34.096584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.084504+0000","last_change":"2024-04-06T12:25:02.080403+0000","last_active":"2024-04-06T12:25:02.084504+0000","last_peered":"2024-04-06T12:25:02.084504+0000","last_clean":"2024-04-06T12:25:02.084504+0000","last_became_active":"2024-04-06T12:25:02.080139+0000","last_became_peered":"2024-04-06T12:25:02.080139+0000","last_unstale":"2024-04-06T12:25:02.084504+0000","last_undegraded":"2024-04-06T12:25:02.084504+0000","last_fullsized":"2024-04-06T12:25:02.084504+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:08:57.284175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073630+0000","last_change":"2024-04-06T12:25:02.085112+0000","last_active":"2024-04-06T12:25:03.073630+0000","last_peered":"2024-04-06T12:25:03.073630+0000","last_clean":"2024-04-06T12:25:03.073630+0000","last_became_active":"2024-04-06T12:25:02.084865+0000","last_became_peered":"2024-04-06T12:25:02.084865+0000","last_unstale":"2024-04-06T12:25:03.073630+0000","last_undegraded":"2024-04-06T12:25:03.073630+0000","last_fullsized":"2024-04-06T12:25:03.073630+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:17:26.548465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071988+0000","last_change":"2024-04-06T12:25:02.086962+0000","last_active":"2024-04-06T12:25:03.071988+0000","last_peered":"2024-04-06T12:25:03.071988+0000","last_clean":"2024-04-06T12:25:03.071988+0000","last_became_active":"2024-04-06T12:25:02.086701+0000","last_became_peered":"2024-04-06T12:25:02.086701+0000","last_unstale":"2024-04-06T12:25:03.071988+0000","last_undegraded":"2024-04-06T12:25:03.071988+0000","last_fullsized":"2024-04-06T12:25:03.071988+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:40.362773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.047869+0000","last_change":"2024-04-06T12:25:02.086636+0000","last_active":"2024-04-06T12:25:04.047869+0000","last_peered":"2024-04-06T12:25:04.047869+0000","last_clean":"2024-04-06T12:25:04.047869+0000","last_became_active":"2024-04-06T12:25:02.086253+0000","last_became_peered":"2024-04-06T12:25:02.086253+0000","last_unstale":"2024-04-06T12:25:04.047869+0000","last_undegraded":"2024-04-06T12:25:04.047869+0000","last_fullsized":"2024-04-06T12:25:04.047869+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:40.362773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.084492+0000","last_change":"2024-04-06T12:25:02.078319+0000","last_active":"2024-04-06T12:25:02.084492+0000","last_peered":"2024-04-06T12:25:02.084492+0000","last_clean":"2024-04-06T12:25:02.084492+0000","last_became_active":"2024-04-06T12:25:02.075399+0000","last_became_peered":"2024-04-06T12:25:02.075399+0000","last_unstale":"2024-04-06T12:25:02.084492+0000","last_undegraded":"2024-04-06T12:25:02.084492+0000","last_fullsized":"2024-04-06T12:25:02.084492+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:55:23.868104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071737+0000","last_change":"2024-04-06T12:25:02.083809+0000","last_active":"2024-04-06T12:25:03.071737+0000","last_peered":"2024-04-06T12:25:03.071737+0000","last_clean":"2024-04-06T12:25:03.071737+0000","last_became_active":"2024-04-06T12:25:02.083403+0000","last_became_peered":"2024-04-06T12:25:02.083403+0000","last_unstale":"2024-04-06T12:25:03.071737+0000","last_undegraded":"2024-04-06T12:25:03.071737+0000","last_fullsized":"2024-04-06T12:25:03.071737+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:43:10.545910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.072369+0000","last_change":"2024-04-06T12:25:02.091817+0000","last_active":"2024-04-06T12:25:03.072369+0000","last_peered":"2024-04-06T12:25:03.072369+0000","last_clean":"2024-04-06T12:25:03.072369+0000","last_became_active":"2024-04-06T12:25:02.091558+0000","last_became_peered":"2024-04-06T12:25:02.091558+0000","last_unstale":"2024-04-06T12:25:03.072369+0000","last_undegraded":"2024-04-06T12:25:03.072369+0000","last_fullsized":"2024-04-06T12:25:03.072369+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:02:05.954262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.048042+0000","last_change":"2024-04-06T12:25:02.082783+0000","last_active":"2024-04-06T12:25:04.048042+0000","last_peered":"2024-04-06T12:25:04.048042+0000","last_clean":"2024-04-06T12:25:04.048042+0000","last_became_active":"2024-04-06T12:25:02.082539+0000","last_became_peered":"2024-04-06T12:25:02.082539+0000","last_unstale":"2024-04-06T12:25:04.048042+0000","last_undegraded":"2024-04-06T12:25:04.048042+0000","last_fullsized":"2024-04-06T12:25:04.048042+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:35:08.942032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.081831+0000","last_change":"2024-04-06T12:25:02.079287+0000","last_active":"2024-04-06T12:25:02.081831+0000","last_peered":"2024-04-06T12:25:02.081831+0000","last_clean":"2024-04-06T12:25:02.081831+0000","last_became_active":"2024-04-06T12:25:02.079033+0000","last_became_peered":"2024-04-06T12:25:02.079033+0000","last_unstale":"2024-04-06T12:25:02.081831+0000","last_undegraded":"2024-04-06T12:25:02.081831+0000","last_fullsized":"2024-04-06T12:25:02.081831+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:38:32.224967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.072006+0000","last_change":"2024-04-06T12:25:02.095882+0000","last_active":"2024-04-06T12:25:03.072006+0000","last_peered":"2024-04-06T12:25:03.072006+0000","last_clean":"2024-04-06T12:25:03.072006+0000","last_became_active":"2024-04-06T12:25:02.095625+0000","last_became_peered":"2024-04-06T12:25:02.095625+0000","last_unstale":"2024-04-06T12:25:03.072006+0000","last_undegraded":"2024-04-06T12:25:03.072006+0000","last_fullsized":"2024-04-06T12:25:03.072006+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:08:57.284175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.047837+0000","last_change":"2024-04-06T12:25:02.082956+0000","last_active":"2024-04-06T12:25:04.047837+0000","last_peered":"2024-04-06T12:25:04.047837+0000","last_clean":"2024-04-06T12:25:04.047837+0000","last_became_active":"2024-04-06T12:25:02.082703+0000","last_became_peered":"2024-04-06T12:25:02.082703+0000","last_unstale":"2024-04-06T12:25:04.047837+0000","last_undegraded":"2024-04-06T12:25:04.047837+0000","last_fullsized":"2024-04-06T12:25:04.047837+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:23:58.314601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.081064+0000","last_change":"2024-04-06T12:25:02.076125+0000","last_active":"2024-04-06T12:25:02.081064+0000","last_peered":"2024-04-06T12:25:02.081064+0000","last_clean":"2024-04-06T12:25:02.081064+0000","last_became_active":"2024-04-06T12:25:02.075868+0000","last_became_peered":"2024-04-06T12:25:02.075868+0000","last_unstale":"2024-04-06T12:25:02.081064+0000","last_undegraded":"2024-04-06T12:25:02.081064+0000","last_fullsized":"2024-04-06T12:25:02.081064+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:15:22.102032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073232+0000","last_change":"2024-04-06T12:25:02.088728+0000","last_active":"2024-04-06T12:25:03.073232+0000","last_peered":"2024-04-06T12:25:03.073232+0000","last_clean":"2024-04-06T12:25:03.073232+0000","last_became_active":"2024-04-06T12:25:02.088455+0000","last_became_peered":"2024-04-06T12:25:02.088455+0000","last_unstale":"2024-04-06T12:25:03.073232+0000","last_undegraded":"2024-04-06T12:25:03.073232+0000","last_fullsized":"2024-04-06T12:25:03.073232+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:44:39.649767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.047793+0000","last_change":"2024-04-06T12:25:02.083055+0000","last_active":"2024-04-06T12:25:04.047793+0000","last_peered":"2024-04-06T12:25:04.047793+0000","last_clean":"2024-04-06T12:25:04.047793+0000","last_became_active":"2024-04-06T12:25:02.082275+0000","last_became_peered":"2024-04-06T12:25:02.082275+0000","last_unstale":"2024-04-06T12:25:04.047793+0000","last_undegraded":"2024-04-06T12:25:04.047793+0000","last_fullsized":"2024-04-06T12:25:04.047793+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:58:02.162124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.090737+0000","last_change":"2024-04-06T12:25:02.090005+0000","last_active":"2024-04-06T12:25:02.090737+0000","last_peered":"2024-04-06T12:25:02.090737+0000","last_clean":"2024-04-06T12:25:02.090737+0000","last_became_active":"2024-04-06T12:25:02.089551+0000","last_became_peered":"2024-04-06T12:25:02.089551+0000","last_unstale":"2024-04-06T12:25:02.090737+0000","last_undegraded":"2024-04-06T12:25:02.090737+0000","last_fullsized":"2024-04-06T12:25:02.090737+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:50:29.941582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073656+0000","last_change":"2024-04-06T12:25:02.093300+0000","last_active":"2024-04-06T12:25:03.073656+0000","last_peered":"2024-04-06T12:25:03.073656+0000","last_clean":"2024-04-06T12:25:03.073656+0000","last_became_active":"2024-04-06T12:25:02.093076+0000","last_became_peered":"2024-04-06T12:25:02.093076+0000","last_unstale":"2024-04-06T12:25:03.073656+0000","last_undegraded":"2024-04-06T12:25:03.073656+0000","last_fullsized":"2024-04-06T12:25:03.073656+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:01:57.407601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.084958+0000","last_change":"2024-04-06T12:25:02.081681+0000","last_active":"2024-04-06T12:25:02.084958+0000","last_peered":"2024-04-06T12:25:02.084958+0000","last_clean":"2024-04-06T12:25:02.084958+0000","last_became_active":"2024-04-06T12:25:02.081424+0000","last_became_peered":"2024-04-06T12:25:02.081424+0000","last_unstale":"2024-04-06T12:25:02.084958+0000","last_undegraded":"2024-04-06T12:25:02.084958+0000","last_fullsized":"2024-04-06T12:25:02.084958+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:23:01.737309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.098641+0000","last_change":"2024-04-06T12:25:02.093057+0000","last_active":"2024-04-06T12:25:02.098641+0000","last_peered":"2024-04-06T12:25:02.098641+0000","last_clean":"2024-04-06T12:25:02.098641+0000","last_became_active":"2024-04-06T12:25:02.092777+0000","last_became_peered":"2024-04-06T12:25:02.092777+0000","last_unstale":"2024-04-06T12:25:02.098641+0000","last_undegraded":"2024-04-06T12:25:02.098641+0000","last_fullsized":"2024-04-06T12:25:02.098641+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:06:53.161677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.098420+0000","last_change":"2024-04-06T12:25:02.087620+0000","last_active":"2024-04-06T12:25:02.098420+0000","last_peered":"2024-04-06T12:25:02.098420+0000","last_clean":"2024-04-06T12:25:02.098420+0000","last_became_active":"2024-04-06T12:25:02.087350+0000","last_became_peered":"2024-04-06T12:25:02.087350+0000","last_unstale":"2024-04-06T12:25:02.098420+0000","last_undegraded":"2024-04-06T12:25:02.098420+0000","last_fullsized":"2024-04-06T12:25:02.098420+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:27:28.547443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071728+0000","last_change":"2024-04-06T12:25:02.091096+0000","last_active":"2024-04-06T12:25:03.071728+0000","last_peered":"2024-04-06T12:25:03.071728+0000","last_clean":"2024-04-06T12:25:03.071728+0000","last_became_active":"2024-04-06T12:25:02.090869+0000","last_became_peered":"2024-04-06T12:25:02.090869+0000","last_unstale":"2024-04-06T12:25:03.071728+0000","last_undegraded":"2024-04-06T12:25:03.071728+0000","last_fullsized":"2024-04-06T12:25:03.071728+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:07.223156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.085109+0000","last_change":"2024-04-06T12:25:02.078998+0000","last_active":"2024-04-06T12:25:02.085109+0000","last_peered":"2024-04-06T12:25:02.085109+0000","last_clean":"2024-04-06T12:25:02.085109+0000","last_became_active":"2024-04-06T12:25:02.078653+0000","last_became_peered":"2024-04-06T12:25:02.078653+0000","last_unstale":"2024-04-06T12:25:02.085109+0000","last_undegraded":"2024-04-06T12:25:02.085109+0000","last_fullsized":"2024-04-06T12:25:02.085109+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:23:29.611147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.081007+0000","last_change":"2024-04-06T12:25:02.075714+0000","last_active":"2024-04-06T12:25:02.081007+0000","last_peered":"2024-04-06T12:25:02.081007+0000","last_clean":"2024-04-06T12:25:02.081007+0000","last_became_active":"2024-04-06T12:25:02.075370+0000","last_became_peered":"2024-04-06T12:25:02.075370+0000","last_unstale":"2024-04-06T12:25:02.081007+0000","last_undegraded":"2024-04-06T12:25:02.081007+0000","last_fullsized":"2024-04-06T12:25:02.081007+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:53:56.158924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.089978+0000","last_change":"2024-04-06T12:25:02.084771+0000","last_active":"2024-04-06T12:25:02.089978+0000","last_peered":"2024-04-06T12:25:02.089978+0000","last_clean":"2024-04-06T12:25:02.089978+0000","last_became_active":"2024-04-06T12:25:02.084522+0000","last_became_peered":"2024-04-06T12:25:02.084522+0000","last_unstale":"2024-04-06T12:25:02.089978+0000","last_undegraded":"2024-04-06T12:25:02.089978+0000","last_fullsized":"2024-04-06T12:25:02.089978+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:20:14.893701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.072108+0000","last_change":"2024-04-06T12:25:02.095349+0000","last_active":"2024-04-06T12:25:03.072108+0000","last_peered":"2024-04-06T12:25:03.072108+0000","last_clean":"2024-04-06T12:25:03.072108+0000","last_became_active":"2024-04-06T12:25:02.092121+0000","last_became_peered":"2024-04-06T12:25:02.092121+0000","last_unstale":"2024-04-06T12:25:03.072108+0000","last_undegraded":"2024-04-06T12:25:03.072108+0000","last_fullsized":"2024-04-06T12:25:03.072108+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:50:29.941582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.048135+0000","last_change":"2024-04-06T12:25:02.085811+0000","last_active":"2024-04-06T12:25:04.048135+0000","last_peered":"2024-04-06T12:25:04.048135+0000","last_clean":"2024-04-06T12:25:04.048135+0000","last_became_active":"2024-04-06T12:25:02.082852+0000","last_became_peered":"2024-04-06T12:25:02.082852+0000","last_unstale":"2024-04-06T12:25:04.048135+0000","last_undegraded":"2024-04-06T12:25:04.048135+0000","last_fullsized":"2024-04-06T12:25:04.048135+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:19:09.708529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.098403+0000","last_change":"2024-04-06T12:25:02.087472+0000","last_active":"2024-04-06T12:25:02.098403+0000","last_peered":"2024-04-06T12:25:02.098403+0000","last_clean":"2024-04-06T12:25:02.098403+0000","last_became_active":"2024-04-06T12:25:02.087163+0000","last_became_peered":"2024-04-06T12:25:02.087163+0000","last_unstale":"2024-04-06T12:25:02.098403+0000","last_undegraded":"2024-04-06T12:25:02.098403+0000","last_fullsized":"2024-04-06T12:25:02.098403+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:00:56.274953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.090634+0000","last_change":"2024-04-06T12:25:02.088495+0000","last_active":"2024-04-06T12:25:02.090634+0000","last_peered":"2024-04-06T12:25:02.090634+0000","last_clean":"2024-04-06T12:25:02.090634+0000","last_became_active":"2024-04-06T12:25:02.088224+0000","last_became_peered":"2024-04-06T12:25:02.088224+0000","last_unstale":"2024-04-06T12:25:02.090634+0000","last_undegraded":"2024-04-06T12:25:02.090634+0000","last_fullsized":"2024-04-06T12:25:02.090634+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:42:56.852103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073403+0000","last_change":"2024-04-06T12:25:02.084707+0000","last_active":"2024-04-06T12:25:03.073403+0000","last_peered":"2024-04-06T12:25:03.073403+0000","last_clean":"2024-04-06T12:25:03.073403+0000","last_became_active":"2024-04-06T12:25:02.083891+0000","last_became_peered":"2024-04-06T12:25:02.083891+0000","last_unstale":"2024-04-06T12:25:03.073403+0000","last_undegraded":"2024-04-06T12:25:03.073403+0000","last_fullsized":"2024-04-06T12:25:03.073403+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:35:04.762031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.081751+0000","last_change":"2024-04-06T12:25:02.077742+0000","last_active":"2024-04-06T12:25:02.081751+0000","last_peered":"2024-04-06T12:25:02.081751+0000","last_clean":"2024-04-06T12:25:02.081751+0000","last_became_active":"2024-04-06T12:25:02.075972+0000","last_became_peered":"2024-04-06T12:25:02.075972+0000","last_unstale":"2024-04-06T12:25:02.081751+0000","last_undegraded":"2024-04-06T12:25:02.081751+0000","last_fullsized":"2024-04-06T12:25:02.081751+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:43:10.545910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.091981+0000","last_change":"2024-04-06T12:25:02.089285+0000","last_active":"2024-04-06T12:25:02.091981+0000","last_peered":"2024-04-06T12:25:02.091981+0000","last_clean":"2024-04-06T12:25:02.091981+0000","last_became_active":"2024-04-06T12:25:02.088982+0000","last_became_peered":"2024-04-06T12:25:02.088982+0000","last_unstale":"2024-04-06T12:25:02.091981+0000","last_undegraded":"2024-04-06T12:25:02.091981+0000","last_fullsized":"2024-04-06T12:25:02.091981+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:54:46.589494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.072901+0000","last_change":"2024-04-06T12:25:02.093182+0000","last_active":"2024-04-06T12:25:03.072901+0000","last_peered":"2024-04-06T12:25:03.072901+0000","last_clean":"2024-04-06T12:25:03.072901+0000","last_became_active":"2024-04-06T12:25:02.092907+0000","last_became_peered":"2024-04-06T12:25:02.092907+0000","last_unstale":"2024-04-06T12:25:03.072901+0000","last_undegraded":"2024-04-06T12:25:03.072901+0000","last_fullsized":"2024-04-06T12:25:03.072901+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:47:17.110195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"66'4","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.098690+0000","last_change":"2024-04-06T12:25:01.657867+0000","last_active":"2024-04-06T12:25:02.098690+0000","last_peered":"2024-04-06T12:25:02.098690+0000","last_clean":"2024-04-06T12:25:02.098690+0000","last_became_active":"2024-04-06T12:25:01.657651+0000","last_became_peered":"2024-04-06T12:25:01.657651+0000","last_unstale":"2024-04-06T12:25:02.098690+0000","last_undegraded":"2024-04-06T12:25:02.098690+0000","last_fullsized":"2024-04-06T12:25:02.098690+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:47:08.361971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.092065+0000","last_change":"2024-04-06T12:25:02.089234+0000","last_active":"2024-04-06T12:25:02.092065+0000","last_peered":"2024-04-06T12:25:02.092065+0000","last_clean":"2024-04-06T12:25:02.092065+0000","last_became_active":"2024-04-06T12:25:02.088889+0000","last_became_peered":"2024-04-06T12:25:02.088889+0000","last_unstale":"2024-04-06T12:25:02.092065+0000","last_undegraded":"2024-04-06T12:25:02.092065+0000","last_fullsized":"2024-04-06T12:25:02.092065+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:48:19.997695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"66'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.048105+0000","last_change":"2024-04-06T12:25:01.664838+0000","last_active":"2024-04-06T12:25:04.048105+0000","last_peered":"2024-04-06T12:25:04.048105+0000","last_clean":"2024-04-06T12:25:04.048105+0000","last_became_active":"2024-04-06T12:25:01.664565+0000","last_became_peered":"2024-04-06T12:25:01.664565+0000","last_unstale":"2024-04-06T12:25:04.048105+0000","last_undegraded":"2024-04-06T12:25:04.048105+0000","last_fullsized":"2024-04-06T12:25:04.048105+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:06:52.156959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.098453+0000","last_change":"2024-04-06T12:25:02.088084+0000","last_active":"2024-04-06T12:25:02.098453+0000","last_peered":"2024-04-06T12:25:02.098453+0000","last_clean":"2024-04-06T12:25:02.098453+0000","last_became_active":"2024-04-06T12:25:02.087796+0000","last_became_peered":"2024-04-06T12:25:02.087796+0000","last_unstale":"2024-04-06T12:25:02.098453+0000","last_undegraded":"2024-04-06T12:25:02.098453+0000","last_fullsized":"2024-04-06T12:25:02.098453+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:47:52.425649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.048118+0000","last_change":"2024-04-06T12:25:01.668912+0000","last_active":"2024-04-06T12:25:04.048118+0000","last_peered":"2024-04-06T12:25:04.048118+0000","last_clean":"2024-04-06T12:25:04.048118+0000","last_became_active":"2024-04-06T12:25:01.668707+0000","last_became_peered":"2024-04-06T12:25:01.668707+0000","last_unstale":"2024-04-06T12:25:04.048118+0000","last_undegraded":"2024-04-06T12:25:04.048118+0000","last_fullsized":"2024-04-06T12:25:04.048118+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:47:51.420931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071843+0000","last_change":"2024-04-06T12:25:02.093680+0000","last_active":"2024-04-06T12:25:03.071843+0000","last_peered":"2024-04-06T12:25:03.071843+0000","last_clean":"2024-04-06T12:25:03.071843+0000","last_became_active":"2024-04-06T12:25:02.093385+0000","last_became_peered":"2024-04-06T12:25:02.093385+0000","last_unstale":"2024-04-06T12:25:03.071843+0000","last_undegraded":"2024-04-06T12:25:03.071843+0000","last_fullsized":"2024-04-06T12:25:03.071843+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:57:44.456276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"66'2","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.091004+0000","last_change":"2024-04-06T12:25:01.658400+0000","last_active":"2024-04-06T12:25:02.091004+0000","last_peered":"2024-04-06T12:25:02.091004+0000","last_clean":"2024-04-06T12:25:02.091004+0000","last_became_active":"2024-04-06T12:25:01.658129+0000","last_became_peered":"2024-04-06T12:25:01.658129+0000","last_unstale":"2024-04-06T12:25:02.091004+0000","last_undegraded":"2024-04-06T12:25:02.091004+0000","last_fullsized":"2024-04-06T12:25:02.091004+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:10:32.281402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.072749+0000","last_change":"2024-04-06T12:25:02.084471+0000","last_active":"2024-04-06T12:25:03.072749+0000","last_peered":"2024-04-06T12:25:03.072749+0000","last_clean":"2024-04-06T12:25:03.072749+0000","last_became_active":"2024-04-06T12:25:02.084198+0000","last_became_peered":"2024-04-06T12:25:02.084198+0000","last_unstale":"2024-04-06T12:25:03.072749+0000","last_undegraded":"2024-04-06T12:25:03.072749+0000","last_fullsized":"2024-04-06T12:25:03.072749+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:10:27.727009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"66'1","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.079889+0000","last_change":"2024-04-06T12:25:01.655266+0000","last_active":"2024-04-06T12:25:02.079889+0000","last_peered":"2024-04-06T12:25:02.079889+0000","last_clean":"2024-04-06T12:25:02.079889+0000","last_became_active":"2024-04-06T12:25:01.655002+0000","last_became_peered":"2024-04-06T12:25:01.655002+0000","last_unstale":"2024-04-06T12:25:02.079889+0000","last_undegraded":"2024-04-06T12:25:02.079889+0000","last_fullsized":"2024-04-06T12:25:02.079889+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:18:35.814800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073528+0000","last_change":"2024-04-06T12:25:02.085799+0000","last_active":"2024-04-06T12:25:03.073528+0000","last_peered":"2024-04-06T12:25:03.073528+0000","last_clean":"2024-04-06T12:25:03.073528+0000","last_became_active":"2024-04-06T12:25:02.085541+0000","last_became_peered":"2024-04-06T12:25:02.085541+0000","last_unstale":"2024-04-06T12:25:03.073528+0000","last_undegraded":"2024-04-06T12:25:03.073528+0000","last_fullsized":"2024-04-06T12:25:03.073528+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:26:00.272122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"66'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071842+0000","last_change":"2024-04-06T12:25:01.657325+0000","last_active":"2024-04-06T12:25:03.071842+0000","last_peered":"2024-04-06T12:25:03.071842+0000","last_clean":"2024-04-06T12:25:03.071842+0000","last_became_active":"2024-04-06T12:25:01.657029+0000","last_became_peered":"2024-04-06T12:25:01.657029+0000","last_unstale":"2024-04-06T12:25:03.071842+0000","last_undegraded":"2024-04-06T12:25:03.071842+0000","last_fullsized":"2024-04-06T12:25:03.071842+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:10:32.281402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071948+0000","last_change":"2024-04-06T12:25:02.093758+0000","last_active":"2024-04-06T12:25:03.071948+0000","last_peered":"2024-04-06T12:25:03.071948+0000","last_clean":"2024-04-06T12:25:03.071948+0000","last_became_active":"2024-04-06T12:25:02.093517+0000","last_became_peered":"2024-04-06T12:25:02.093517+0000","last_unstale":"2024-04-06T12:25:03.071948+0000","last_undegraded":"2024-04-06T12:25:03.071948+0000","last_fullsized":"2024-04-06T12:25:03.071948+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:20:14.893701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.083893+0000","last_change":"2024-04-06T12:25:01.659950+0000","last_active":"2024-04-06T12:25:02.083893+0000","last_peered":"2024-04-06T12:25:02.083893+0000","last_clean":"2024-04-06T12:25:02.083893+0000","last_became_active":"2024-04-06T12:25:01.659663+0000","last_became_peered":"2024-04-06T12:25:01.659663+0000","last_unstale":"2024-04-06T12:25:02.083893+0000","last_undegraded":"2024-04-06T12:25:02.083893+0000","last_fullsized":"2024-04-06T12:25:02.083893+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:47:20.358209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073616+0000","last_change":"2024-04-06T12:25:02.076306+0000","last_active":"2024-04-06T12:25:03.073616+0000","last_peered":"2024-04-06T12:25:03.073616+0000","last_clean":"2024-04-06T12:25:03.073616+0000","last_became_active":"2024-04-06T12:25:02.076017+0000","last_became_peered":"2024-04-06T12:25:02.076017+0000","last_unstale":"2024-04-06T12:25:03.073616+0000","last_undegraded":"2024-04-06T12:25:03.073616+0000","last_fullsized":"2024-04-06T12:25:03.073616+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:58:02.162124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.068870+0000","last_change":"2024-04-06T12:25:01.667208+0000","last_active":"2024-04-06T12:25:02.068870+0000","last_peered":"2024-04-06T12:25:02.068870+0000","last_clean":"2024-04-06T12:25:02.068870+0000","last_became_active":"2024-04-06T12:25:01.666880+0000","last_became_peered":"2024-04-06T12:25:01.666880+0000","last_unstale":"2024-04-06T12:25:02.068870+0000","last_undegraded":"2024-04-06T12:25:02.068870+0000","last_fullsized":"2024-04-06T12:25:02.068870+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:05:05.517843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071988+0000","last_change":"2024-04-06T12:25:02.084907+0000","last_active":"2024-04-06T12:25:03.071988+0000","last_peered":"2024-04-06T12:25:03.071988+0000","last_clean":"2024-04-06T12:25:03.071988+0000","last_became_active":"2024-04-06T12:25:02.084631+0000","last_became_peered":"2024-04-06T12:25:02.084631+0000","last_unstale":"2024-04-06T12:25:03.071988+0000","last_undegraded":"2024-04-06T12:25:03.071988+0000","last_fullsized":"2024-04-06T12:25:03.071988+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:48:19.997695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073636+0000","last_change":"2024-04-06T12:25:01.656618+0000","last_active":"2024-04-06T12:25:03.073636+0000","last_peered":"2024-04-06T12:25:03.073636+0000","last_clean":"2024-04-06T12:25:03.073636+0000","last_became_active":"2024-04-06T12:25:01.656325+0000","last_became_peered":"2024-04-06T12:25:01.656325+0000","last_unstale":"2024-04-06T12:25:03.073636+0000","last_undegraded":"2024-04-06T12:25:03.073636+0000","last_fullsized":"2024-04-06T12:25:03.073636+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:25:33.521667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.092044+0000","last_change":"2024-04-06T12:25:02.088517+0000","last_active":"2024-04-06T12:25:02.092044+0000","last_peered":"2024-04-06T12:25:02.092044+0000","last_clean":"2024-04-06T12:25:02.092044+0000","last_became_active":"2024-04-06T12:25:02.087772+0000","last_became_peered":"2024-04-06T12:25:02.087772+0000","last_unstale":"2024-04-06T12:25:02.092044+0000","last_undegraded":"2024-04-06T12:25:02.092044+0000","last_fullsized":"2024-04-06T12:25:02.092044+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:28:27.128910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.072830+0000","last_change":"2024-04-06T12:25:01.665289+0000","last_active":"2024-04-06T12:25:03.072830+0000","last_peered":"2024-04-06T12:25:03.072830+0000","last_clean":"2024-04-06T12:25:03.072830+0000","last_became_active":"2024-04-06T12:25:01.664968+0000","last_became_peered":"2024-04-06T12:25:01.664968+0000","last_unstale":"2024-04-06T12:25:03.072830+0000","last_undegraded":"2024-04-06T12:25:03.072830+0000","last_fullsized":"2024-04-06T12:25:03.072830+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:35:03.757313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.094055+0000","last_change":"2024-04-06T12:25:02.090419+0000","last_active":"2024-04-06T12:25:02.094055+0000","last_peered":"2024-04-06T12:25:02.094055+0000","last_clean":"2024-04-06T12:25:02.094055+0000","last_became_active":"2024-04-06T12:25:02.090087+0000","last_became_peered":"2024-04-06T12:25:02.090087+0000","last_unstale":"2024-04-06T12:25:02.094055+0000","last_undegraded":"2024-04-06T12:25:02.094055+0000","last_fullsized":"2024-04-06T12:25:02.094055+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:02:05.954262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"66'3","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073580+0000","last_change":"2024-04-06T12:25:01.666204+0000","last_active":"2024-04-06T12:25:03.073580+0000","last_peered":"2024-04-06T12:25:03.073580+0000","last_clean":"2024-04-06T12:25:03.073580+0000","last_became_active":"2024-04-06T12:25:01.665978+0000","last_became_peered":"2024-04-06T12:25:01.665978+0000","last_unstale":"2024-04-06T12:25:03.073580+0000","last_undegraded":"2024-04-06T12:25:03.073580+0000","last_fullsized":"2024-04-06T12:25:03.073580+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:31:42.445760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.072056+0000","last_change":"2024-04-06T12:25:02.080992+0000","last_active":"2024-04-06T12:25:03.072056+0000","last_peered":"2024-04-06T12:25:03.072056+0000","last_clean":"2024-04-06T12:25:03.072056+0000","last_became_active":"2024-04-06T12:25:02.080635+0000","last_became_peered":"2024-04-06T12:25:02.080635+0000","last_unstale":"2024-04-06T12:25:03.072056+0000","last_undegraded":"2024-04-06T12:25:03.072056+0000","last_fullsized":"2024-04-06T12:25:03.072056+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:54:46.589494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"66'6","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.072062+0000","last_change":"2024-04-06T12:25:01.655646+0000","last_active":"2024-04-06T12:25:03.072062+0000","last_peered":"2024-04-06T12:25:03.072062+0000","last_clean":"2024-04-06T12:25:03.072062+0000","last_became_active":"2024-04-06T12:25:01.655306+0000","last_became_peered":"2024-04-06T12:25:01.655306+0000","last_unstale":"2024-04-06T12:25:03.072062+0000","last_undegraded":"2024-04-06T12:25:03.072062+0000","last_fullsized":"2024-04-06T12:25:03.072062+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:47:00.426367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.092226+0000","last_change":"2024-04-06T12:25:02.090390+0000","last_active":"2024-04-06T12:25:02.092226+0000","last_peered":"2024-04-06T12:25:02.092226+0000","last_clean":"2024-04-06T12:25:02.092226+0000","last_became_active":"2024-04-06T12:25:02.090115+0000","last_became_peered":"2024-04-06T12:25:02.090115+0000","last_unstale":"2024-04-06T12:25:02.092226+0000","last_undegraded":"2024-04-06T12:25:02.092226+0000","last_fullsized":"2024-04-06T12:25:02.092226+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:40.362773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"66'2","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.081034+0000","last_change":"2024-04-06T12:25:01.666676+0000","last_active":"2024-04-06T12:25:02.081034+0000","last_peered":"2024-04-06T12:25:02.081034+0000","last_clean":"2024-04-06T12:25:02.081034+0000","last_became_active":"2024-04-06T12:25:01.664796+0000","last_became_peered":"2024-04-06T12:25:01.664796+0000","last_unstale":"2024-04-06T12:25:02.081034+0000","last_undegraded":"2024-04-06T12:25:02.081034+0000","last_fullsized":"2024-04-06T12:25:02.081034+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:10:26.722291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.085063+0000","last_change":"2024-04-06T12:25:02.083800+0000","last_active":"2024-04-06T12:25:02.085063+0000","last_peered":"2024-04-06T12:25:02.085063+0000","last_clean":"2024-04-06T12:25:02.085063+0000","last_became_active":"2024-04-06T12:25:02.083562+0000","last_became_peered":"2024-04-06T12:25:02.083562+0000","last_unstale":"2024-04-06T12:25:02.085063+0000","last_undegraded":"2024-04-06T12:25:02.085063+0000","last_fullsized":"2024-04-06T12:25:02.085063+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:34:26.095936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.082173+0000","last_change":"2024-04-06T12:25:01.668367+0000","last_active":"2024-04-06T12:25:02.082173+0000","last_peered":"2024-04-06T12:25:02.082173+0000","last_clean":"2024-04-06T12:25:02.082173+0000","last_became_active":"2024-04-06T12:25:01.668089+0000","last_became_peered":"2024-04-06T12:25:01.668089+0000","last_unstale":"2024-04-06T12:25:02.082173+0000","last_undegraded":"2024-04-06T12:25:02.082173+0000","last_fullsized":"2024-04-06T12:25:02.082173+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:52:47.398773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071844+0000","last_change":"2024-04-06T12:25:02.091381+0000","last_active":"2024-04-06T12:25:03.071844+0000","last_peered":"2024-04-06T12:25:03.071844+0000","last_clean":"2024-04-06T12:25:03.071844+0000","last_became_active":"2024-04-06T12:25:02.091145+0000","last_became_peered":"2024-04-06T12:25:02.091145+0000","last_unstale":"2024-04-06T12:25:03.071844+0000","last_undegraded":"2024-04-06T12:25:03.071844+0000","last_fullsized":"2024-04-06T12:25:03.071844+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:23:01.737309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073281+0000","last_change":"2024-04-06T12:25:01.657373+0000","last_active":"2024-04-06T12:25:03.073281+0000","last_peered":"2024-04-06T12:25:03.073281+0000","last_clean":"2024-04-06T12:25:03.073281+0000","last_became_active":"2024-04-06T12:25:01.657078+0000","last_became_peered":"2024-04-06T12:25:01.657078+0000","last_unstale":"2024-04-06T12:25:03.073281+0000","last_undegraded":"2024-04-06T12:25:03.073281+0000","last_fullsized":"2024-04-06T12:25:03.073281+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:47:04.026567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.090717+0000","last_change":"2024-04-06T12:25:02.089758+0000","last_active":"2024-04-06T12:25:02.090717+0000","last_peered":"2024-04-06T12:25:02.090717+0000","last_clean":"2024-04-06T12:25:02.090717+0000","last_became_active":"2024-04-06T12:25:02.089241+0000","last_became_peered":"2024-04-06T12:25:02.089241+0000","last_unstale":"2024-04-06T12:25:02.090717+0000","last_undegraded":"2024-04-06T12:25:02.090717+0000","last_fullsized":"2024-04-06T12:25:02.090717+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:33:27.461189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"66'8","reported_seq":19,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071846+0000","last_change":"2024-04-06T12:25:01.655439+0000","last_active":"2024-04-06T12:25:03.071846+0000","last_peered":"2024-04-06T12:25:03.071846+0000","last_clean":"2024-04-06T12:25:03.071846+0000","last_became_active":"2024-04-06T12:25:01.655176+0000","last_became_peered":"2024-04-06T12:25:01.655176+0000","last_unstale":"2024-04-06T12:25:03.071846+0000","last_undegraded":"2024-04-06T12:25:03.071846+0000","last_fullsized":"2024-04-06T12:25:03.071846+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:17:11.808726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071740+0000","last_change":"2024-04-06T12:25:02.095985+0000","last_active":"2024-04-06T12:25:03.071740+0000","last_peered":"2024-04-06T12:25:03.071740+0000","last_clean":"2024-04-06T12:25:03.071740+0000","last_became_active":"2024-04-06T12:25:02.095599+0000","last_became_peered":"2024-04-06T12:25:02.095599+0000","last_unstale":"2024-04-06T12:25:03.071740+0000","last_undegraded":"2024-04-06T12:25:03.071740+0000","last_fullsized":"2024-04-06T12:25:03.071740+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:15:22.102032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"66'1","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.077013+0000","last_change":"2024-04-06T12:25:01.671174+0000","last_active":"2024-04-06T12:25:02.077013+0000","last_peered":"2024-04-06T12:25:02.077013+0000","last_clean":"2024-04-06T12:25:02.077013+0000","last_became_active":"2024-04-06T12:25:01.670893+0000","last_became_peered":"2024-04-06T12:25:01.670893+0000","last_unstale":"2024-04-06T12:25:02.077013+0000","last_undegraded":"2024-04-06T12:25:02.077013+0000","last_fullsized":"2024-04-06T12:25:02.077013+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:48:18.992977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.047828+0000","last_change":"2024-04-06T12:25:02.086649+0000","last_active":"2024-04-06T12:25:04.047828+0000","last_peered":"2024-04-06T12:25:04.047828+0000","last_clean":"2024-04-06T12:25:04.047828+0000","last_became_active":"2024-04-06T12:25:02.086370+0000","last_became_peered":"2024-04-06T12:25:02.086370+0000","last_unstale":"2024-04-06T12:25:04.047828+0000","last_undegraded":"2024-04-06T12:25:04.047828+0000","last_fullsized":"2024-04-06T12:25:04.047828+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:54:46.589494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"66'2","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.088269+0000","last_change":"2024-04-06T12:25:01.657487+0000","last_active":"2024-04-06T12:25:02.088269+0000","last_peered":"2024-04-06T12:25:02.088269+0000","last_clean":"2024-04-06T12:25:02.088269+0000","last_became_active":"2024-04-06T12:25:01.657195+0000","last_became_peered":"2024-04-06T12:25:01.657195+0000","last_unstale":"2024-04-06T12:25:02.088269+0000","last_undegraded":"2024-04-06T12:25:02.088269+0000","last_fullsized":"2024-04-06T12:25:02.088269+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:35:07.572364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071812+0000","last_change":"2024-04-06T12:25:02.082577+0000","last_active":"2024-04-06T12:25:03.071812+0000","last_peered":"2024-04-06T12:25:03.071812+0000","last_clean":"2024-04-06T12:25:03.071812+0000","last_became_active":"2024-04-06T12:25:02.082262+0000","last_became_peered":"2024-04-06T12:25:02.082262+0000","last_unstale":"2024-04-06T12:25:03.071812+0000","last_undegraded":"2024-04-06T12:25:03.071812+0000","last_fullsized":"2024-04-06T12:25:03.071812+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:20:08.845705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"66'1","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.094811+0000","last_change":"2024-04-06T12:25:01.669469+0000","last_active":"2024-04-06T12:25:02.094811+0000","last_peered":"2024-04-06T12:25:02.094811+0000","last_clean":"2024-04-06T12:25:02.094811+0000","last_became_active":"2024-04-06T12:25:01.669226+0000","last_became_peered":"2024-04-06T12:25:01.669226+0000","last_unstale":"2024-04-06T12:25:02.094811+0000","last_undegraded":"2024-04-06T12:25:02.094811+0000","last_fullsized":"2024-04-06T12:25:02.094811+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:20:34.472848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.047978+0000","last_change":"2024-04-06T12:25:02.089505+0000","last_active":"2024-04-06T12:25:04.047978+0000","last_peered":"2024-04-06T12:25:04.047978+0000","last_clean":"2024-04-06T12:25:04.047978+0000","last_became_active":"2024-04-06T12:25:02.089216+0000","last_became_peered":"2024-04-06T12:25:02.089216+0000","last_unstale":"2024-04-06T12:25:04.047978+0000","last_undegraded":"2024-04-06T12:25:04.047978+0000","last_fullsized":"2024-04-06T12:25:04.047978+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:28:27.128910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.082321+0000","last_change":"2024-04-06T12:25:01.658691+0000","last_active":"2024-04-06T12:25:02.082321+0000","last_peered":"2024-04-06T12:25:02.082321+0000","last_clean":"2024-04-06T12:25:02.082321+0000","last_became_active":"2024-04-06T12:25:01.658426+0000","last_became_peered":"2024-04-06T12:25:01.658426+0000","last_unstale":"2024-04-06T12:25:02.082321+0000","last_undegraded":"2024-04-06T12:25:02.082321+0000","last_fullsized":"2024-04-06T12:25:02.082321+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:50:22.888848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.091058+0000","last_change":"2024-04-06T12:25:02.085021+0000","last_active":"2024-04-06T12:25:02.091058+0000","last_peered":"2024-04-06T12:25:02.091058+0000","last_clean":"2024-04-06T12:25:02.091058+0000","last_became_active":"2024-04-06T12:25:02.084751+0000","last_became_peered":"2024-04-06T12:25:02.084751+0000","last_unstale":"2024-04-06T12:25:02.091058+0000","last_undegraded":"2024-04-06T12:25:02.091058+0000","last_fullsized":"2024-04-06T12:25:02.091058+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:10:27.727009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.048001+0000","last_change":"2024-04-06T12:25:01.664195+0000","last_active":"2024-04-06T12:25:04.048001+0000","last_peered":"2024-04-06T12:25:04.048001+0000","last_clean":"2024-04-06T12:25:04.048001+0000","last_became_active":"2024-04-06T12:25:01.663920+0000","last_became_peered":"2024-04-06T12:25:01.663920+0000","last_unstale":"2024-04-06T12:25:04.048001+0000","last_undegraded":"2024-04-06T12:25:04.048001+0000","last_fullsized":"2024-04-06T12:25:04.048001+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:39:59.484595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.098574+0000","last_change":"2024-04-06T12:25:02.091450+0000","last_active":"2024-04-06T12:25:02.098574+0000","last_peered":"2024-04-06T12:25:02.098574+0000","last_clean":"2024-04-06T12:25:02.098574+0000","last_became_active":"2024-04-06T12:25:02.091137+0000","last_became_peered":"2024-04-06T12:25:02.091137+0000","last_unstale":"2024-04-06T12:25:02.098574+0000","last_undegraded":"2024-04-06T12:25:02.098574+0000","last_fullsized":"2024-04-06T12:25:02.098574+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:35:02.409261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071728+0000","last_change":"2024-04-06T12:25:01.654111+0000","last_active":"2024-04-06T12:25:03.071728+0000","last_peered":"2024-04-06T12:25:03.071728+0000","last_clean":"2024-04-06T12:25:03.071728+0000","last_became_active":"2024-04-06T12:25:01.653816+0000","last_became_peered":"2024-04-06T12:25:01.653816+0000","last_unstale":"2024-04-06T12:25:03.071728+0000","last_undegraded":"2024-04-06T12:25:03.071728+0000","last_fullsized":"2024-04-06T12:25:03.071728+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:05.968823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.090611+0000","last_change":"2024-04-06T12:25:02.088434+0000","last_active":"2024-04-06T12:25:02.090611+0000","last_peered":"2024-04-06T12:25:02.090611+0000","last_clean":"2024-04-06T12:25:02.090611+0000","last_became_active":"2024-04-06T12:25:02.088270+0000","last_became_peered":"2024-04-06T12:25:02.088270+0000","last_unstale":"2024-04-06T12:25:02.090611+0000","last_undegraded":"2024-04-06T12:25:02.090611+0000","last_fullsized":"2024-04-06T12:25:02.090611+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:12:13.258920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.069357+0000","last_change":"2024-04-06T12:25:01.667322+0000","last_active":"2024-04-06T12:25:02.069357+0000","last_peered":"2024-04-06T12:25:02.069357+0000","last_clean":"2024-04-06T12:25:02.069357+0000","last_became_active":"2024-04-06T12:25:01.664317+0000","last_became_peered":"2024-04-06T12:25:01.664317+0000","last_unstale":"2024-04-06T12:25:02.069357+0000","last_undegraded":"2024-04-06T12:25:02.069357+0000","last_fullsized":"2024-04-06T12:25:02.069357+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:20:07.840987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.048133+0000","last_change":"2024-04-06T12:25:02.093006+0000","last_active":"2024-04-06T12:25:04.048133+0000","last_peered":"2024-04-06T12:25:04.048133+0000","last_clean":"2024-04-06T12:25:04.048133+0000","last_became_active":"2024-04-06T12:25:02.092754+0000","last_became_peered":"2024-04-06T12:25:02.092754+0000","last_unstale":"2024-04-06T12:25:04.048133+0000","last_undegraded":"2024-04-06T12:25:04.048133+0000","last_fullsized":"2024-04-06T12:25:04.048133+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:10:27.727009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.066153+0000","last_change":"2024-04-06T12:25:01.648228+0000","last_active":"2024-04-06T12:25:02.066153+0000","last_peered":"2024-04-06T12:25:02.066153+0000","last_clean":"2024-04-06T12:25:02.066153+0000","last_became_active":"2024-04-06T12:25:01.647374+0000","last_became_peered":"2024-04-06T12:25:01.647374+0000","last_unstale":"2024-04-06T12:25:02.066153+0000","last_undegraded":"2024-04-06T12:25:02.066153+0000","last_fullsized":"2024-04-06T12:25:02.066153+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:47:20.358209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.091999+0000","last_change":"2024-04-06T12:25:02.089303+0000","last_active":"2024-04-06T12:25:02.091999+0000","last_peered":"2024-04-06T12:25:02.091999+0000","last_clean":"2024-04-06T12:25:02.091999+0000","last_became_active":"2024-04-06T12:25:02.088994+0000","last_became_peered":"2024-04-06T12:25:02.088994+0000","last_unstale":"2024-04-06T12:25:02.091999+0000","last_undegraded":"2024-04-06T12:25:02.091999+0000","last_fullsized":"2024-04-06T12:25:02.091999+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:05:06.522562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"66'2","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.090423+0000","last_change":"2024-04-06T12:25:01.668356+0000","last_active":"2024-04-06T12:25:02.090423+0000","last_peered":"2024-04-06T12:25:02.090423+0000","last_clean":"2024-04-06T12:25:02.090423+0000","last_became_active":"2024-04-06T12:25:01.668085+0000","last_became_peered":"2024-04-06T12:25:01.668085+0000","last_unstale":"2024-04-06T12:25:02.090423+0000","last_undegraded":"2024-04-06T12:25:02.090423+0000","last_fullsized":"2024-04-06T12:25:02.090423+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:23:28.606429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.047969+0000","last_change":"2024-04-06T12:25:02.082171+0000","last_active":"2024-04-06T12:25:04.047969+0000","last_peered":"2024-04-06T12:25:04.047969+0000","last_clean":"2024-04-06T12:25:04.047969+0000","last_became_active":"2024-04-06T12:25:02.081923+0000","last_became_peered":"2024-04-06T12:25:02.081923+0000","last_unstale":"2024-04-06T12:25:04.047969+0000","last_undegraded":"2024-04-06T12:25:04.047969+0000","last_fullsized":"2024-04-06T12:25:04.047969+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:20:08.845705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073635+0000","last_change":"2024-04-06T12:25:01.654961+0000","last_active":"2024-04-06T12:25:03.073635+0000","last_peered":"2024-04-06T12:25:03.073635+0000","last_clean":"2024-04-06T12:25:03.073635+0000","last_became_active":"2024-04-06T12:25:01.654643+0000","last_became_peered":"2024-04-06T12:25:01.654643+0000","last_unstale":"2024-04-06T12:25:03.073635+0000","last_undegraded":"2024-04-06T12:25:03.073635+0000","last_fullsized":"2024-04-06T12:25:03.073635+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:25:27.977561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071886+0000","last_change":"2024-04-06T12:25:02.093277+0000","last_active":"2024-04-06T12:25:03.071886+0000","last_peered":"2024-04-06T12:25:03.071886+0000","last_clean":"2024-04-06T12:25:03.071886+0000","last_became_active":"2024-04-06T12:25:02.092152+0000","last_became_peered":"2024-04-06T12:25:02.092152+0000","last_unstale":"2024-04-06T12:25:03.071886+0000","last_undegraded":"2024-04-06T12:25:03.071886+0000","last_fullsized":"2024-04-06T12:25:03.071886+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:58:40.462728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.069819+0000","last_change":"2024-04-06T12:25:01.666679+0000","last_active":"2024-04-06T12:25:02.069819+0000","last_peered":"2024-04-06T12:25:02.069819+0000","last_clean":"2024-04-06T12:25:02.069819+0000","last_became_active":"2024-04-06T12:25:01.664967+0000","last_became_peered":"2024-04-06T12:25:01.664967+0000","last_unstale":"2024-04-06T12:25:02.069819+0000","last_undegraded":"2024-04-06T12:25:02.069819+0000","last_fullsized":"2024-04-06T12:25:02.069819+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:39.358055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.089911+0000","last_change":"2024-04-06T12:25:02.085535+0000","last_active":"2024-04-06T12:25:02.089911+0000","last_peered":"2024-04-06T12:25:02.089911+0000","last_clean":"2024-04-06T12:25:02.089911+0000","last_became_active":"2024-04-06T12:25:02.085258+0000","last_became_peered":"2024-04-06T12:25:02.085258+0000","last_unstale":"2024-04-06T12:25:02.089911+0000","last_undegraded":"2024-04-06T12:25:02.089911+0000","last_fullsized":"2024-04-06T12:25:02.089911+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:28:24.392874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071904+0000","last_change":"2024-04-06T12:25:01.647993+0000","last_active":"2024-04-06T12:25:03.071904+0000","last_peered":"2024-04-06T12:25:03.071904+0000","last_clean":"2024-04-06T12:25:03.071904+0000","last_became_active":"2024-04-06T12:25:01.647625+0000","last_became_peered":"2024-04-06T12:25:01.647625+0000","last_unstale":"2024-04-06T12:25:03.071904+0000","last_undegraded":"2024-04-06T12:25:03.071904+0000","last_fullsized":"2024-04-06T12:25:03.071904+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:36:59.654441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.091936+0000","last_change":"2024-04-06T12:25:02.088411+0000","last_active":"2024-04-06T12:25:02.091936+0000","last_peered":"2024-04-06T12:25:02.091936+0000","last_clean":"2024-04-06T12:25:02.091936+0000","last_became_active":"2024-04-06T12:25:02.087731+0000","last_became_peered":"2024-04-06T12:25:02.087731+0000","last_unstale":"2024-04-06T12:25:02.091936+0000","last_undegraded":"2024-04-06T12:25:02.091936+0000","last_fullsized":"2024-04-06T12:25:02.091936+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:20:08.845705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071820+0000","last_change":"2024-04-06T12:25:01.664380+0000","last_active":"2024-04-06T12:25:03.071820+0000","last_peered":"2024-04-06T12:25:03.071820+0000","last_clean":"2024-04-06T12:25:03.071820+0000","last_became_active":"2024-04-06T12:25:01.664071+0000","last_became_peered":"2024-04-06T12:25:01.664071+0000","last_unstale":"2024-04-06T12:25:03.071820+0000","last_undegraded":"2024-04-06T12:25:03.071820+0000","last_fullsized":"2024-04-06T12:25:03.071820+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:57:11.331245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.078188+0000","last_change":"2024-04-06T12:25:02.074385+0000","last_active":"2024-04-06T12:25:02.078188+0000","last_peered":"2024-04-06T12:25:02.078188+0000","last_clean":"2024-04-06T12:25:02.078188+0000","last_became_active":"2024-04-06T12:25:02.074137+0000","last_became_peered":"2024-04-06T12:25:02.074137+0000","last_unstale":"2024-04-06T12:25:02.078188+0000","last_undegraded":"2024-04-06T12:25:02.078188+0000","last_fullsized":"2024-04-06T12:25:02.078188+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:07.223156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071894+0000","last_change":"2024-04-06T12:25:01.655721+0000","last_active":"2024-04-06T12:25:03.071894+0000","last_peered":"2024-04-06T12:25:03.071894+0000","last_clean":"2024-04-06T12:25:03.071894+0000","last_became_active":"2024-04-06T12:25:01.655377+0000","last_became_peered":"2024-04-06T12:25:01.655377+0000","last_unstale":"2024-04-06T12:25:03.071894+0000","last_undegraded":"2024-04-06T12:25:03.071894+0000","last_fullsized":"2024-04-06T12:25:03.071894+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:17:25.543747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.072458+0000","last_change":"2024-04-06T12:25:02.093377+0000","last_active":"2024-04-06T12:25:03.072458+0000","last_peered":"2024-04-06T12:25:03.072458+0000","last_clean":"2024-04-06T12:25:03.072458+0000","last_became_active":"2024-04-06T12:25:02.093105+0000","last_became_peered":"2024-04-06T12:25:02.093105+0000","last_unstale":"2024-04-06T12:25:03.072458+0000","last_undegraded":"2024-04-06T12:25:03.072458+0000","last_fullsized":"2024-04-06T12:25:03.072458+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:34:26.095936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.076903+0000","last_change":"2024-04-06T12:25:01.664323+0000","last_active":"2024-04-06T12:25:02.076903+0000","last_peered":"2024-04-06T12:25:02.076903+0000","last_clean":"2024-04-06T12:25:02.076903+0000","last_became_active":"2024-04-06T12:25:01.664098+0000","last_became_peered":"2024-04-06T12:25:01.664098+0000","last_unstale":"2024-04-06T12:25:02.076903+0000","last_undegraded":"2024-04-06T12:25:02.076903+0000","last_fullsized":"2024-04-06T12:25:02.076903+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:38:03.670811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.098565+0000","last_change":"2024-04-06T12:25:02.091379+0000","last_active":"2024-04-06T12:25:02.098565+0000","last_peered":"2024-04-06T12:25:02.098565+0000","last_clean":"2024-04-06T12:25:02.098565+0000","last_became_active":"2024-04-06T12:25:02.091141+0000","last_became_peered":"2024-04-06T12:25:02.091141+0000","last_unstale":"2024-04-06T12:25:02.098565+0000","last_undegraded":"2024-04-06T12:25:02.098565+0000","last_fullsized":"2024-04-06T12:25:02.098565+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:40:00.489313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.069450+0000","last_change":"2024-04-06T12:25:01.667874+0000","last_active":"2024-04-06T12:25:02.069450+0000","last_peered":"2024-04-06T12:25:02.069450+0000","last_clean":"2024-04-06T12:25:02.069450+0000","last_became_active":"2024-04-06T12:25:01.667525+0000","last_became_peered":"2024-04-06T12:25:01.667525+0000","last_unstale":"2024-04-06T12:25:02.069450+0000","last_undegraded":"2024-04-06T12:25:02.069450+0000","last_fullsized":"2024-04-06T12:25:02.069450+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:28:26.124191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"55'139","reported_seq":295,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.071533+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:25:02.071533+0000","last_peered":"2024-04-06T12:25:02.071533+0000","last_clean":"2024-04-06T12:25:02.071533+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:25:02.071533+0000","last_undegraded":"2024-04-06T12:25:02.071533+0000","last_fullsized":"2024-04-06T12:25:02.071533+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575008,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54632,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317208,"statfs":{"total":96636764160,"available":96580820992,"internally_reserved":0,"allocated":1404928,"data_stored":983178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56899999999999995,"15min":0.53900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.77500000000000002,"5min":0.78300000000000003,"15min":0.80200000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56499999999999995,"15min":0.55100000000000005},"min":{"1min":0.35399999999999998,"5min":0.32000000000000001,"15min":0.25900000000000001},"max":{"1min":0.71999999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.44600000000000001}]},{"osd":1,"last update":"Sat Apr 6 12:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56100000000000005,"15min":0.54500000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.31900000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.81399999999999995},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58399999999999996,"15min":0.56499999999999995},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79600000000000004},"last":0.51500000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57499999999999996,"15min":0.54300000000000004},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.78500000000000003,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59099999999999997,"15min":0.58199999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.24199999999999999},"max":{"1min":0.751,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.624}]},{"osd":3,"last update":"Sat Apr 6 12:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58699999999999997,"15min":0.56599999999999995},"min":{"1min":0.34799999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.73099999999999998,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59799999999999998,"15min":0.60099999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.34699999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.92000000000000004},"last":0.57799999999999996}]},{"osd":4,"last update":"Sat Apr 6 12:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.44,"15min":0.46400000000000002},"min":{"1min":0.29699999999999999,"5min":0.25,"15min":0.25},"max":{"1min":0.70399999999999996,"5min":0.72999999999999998,"15min":0.83599999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.496,"5min":0.52300000000000002,"15min":0.54200000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.64200000000000002,"5min":0.74199999999999999,"15min":1.0780000000000001},"last":0.65000000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.48599999999999999,"15min":0.52600000000000002},"min":{"1min":0.29299999999999998,"5min":0.26600000000000001,"15min":0.23799999999999999},"max":{"1min":0.76900000000000002,"5min":0.86099999999999999,"15min":0.98199999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.5,"5min":0.55100000000000005,"15min":0.57499999999999996},"min":{"1min":0.316,"5min":0.314,"15min":0.29299999999999998},"max":{"1min":0.76000000000000001,"5min":0.81499999999999995,"15min":0.93600000000000005},"last":0.61099999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.52000000000000002,"15min":0.55400000000000005},"min":{"1min":0.34799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66000000000000003,"5min":0.77400000000000002,"15min":0.80200000000000005},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.56000000000000005,"15min":0.57599999999999996},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.26900000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.88500000000000001},"last":0.68100000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575007,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":794624,"data_stored":385699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55100000000000005,"15min":0.52300000000000002},"min":{"1min":0.379,"5min":0.26200000000000001,"15min":0.23000000000000001},"max":{"1min":0.70499999999999996,"5min":0.80700000000000005,"15min":0.82099999999999995},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.55200000000000005,"15min":0.52300000000000002},"min":{"1min":0.378,"5min":0.30199999999999999,"15min":0.245},"max":{"1min":0.66900000000000004,"5min":0.76200000000000001,"15min":0.76400000000000001},"last":0.37}]},{"osd":1,"last update":"Sat Apr 6 12:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.60699999999999998,"15min":0.56499999999999995},"min":{"1min":0.39100000000000001,"5min":0.32600000000000001,"15min":0.24399999999999999},"max":{"1min":0.753,"5min":0.84599999999999997,"15min":0.84999999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61299999999999999,"15min":0.58199999999999996},"min":{"1min":0.46800000000000003,"5min":0.221,"15min":0.221},"max":{"1min":0.76700000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57399999999999995,"15min":0.54200000000000004},"min":{"1min":0.443,"5min":0.35599999999999998,"15min":0.29899999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58499999999999996,"15min":0.56200000000000006},"min":{"1min":0.41399999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.70099999999999996,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.32500000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59699999999999998,"15min":0.57699999999999996},"min":{"1min":0.42799999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.69199999999999995,"5min":0.79000000000000004,"15min":0.81200000000000006},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63500000000000001,"15min":0.61499999999999999},"min":{"1min":0.45600000000000002,"5min":0.42899999999999999,"15min":0.374},"max":{"1min":0.753,"5min":0.872,"15min":0.872},"last":0.42899999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.495,"15min":0.50600000000000001},"min":{"1min":0.309,"5min":0.27500000000000002,"15min":0.27300000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.54200000000000004},"min":{"1min":0.33700000000000002,"5min":0.309,"15min":0.30599999999999999},"max":{"1min":0.73999999999999999,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.56499999999999995}]},{"osd":5,"last update":"Sat Apr 6 12:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.495,"15min":0.50900000000000001},"min":{"1min":0.33200000000000002,"5min":0.32100000000000001,"15min":0.313},"max":{"1min":0.755,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51100000000000001,"15min":0.53400000000000003},"min":{"1min":0.35999999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.80200000000000005,"5min":0.83699999999999997,"15min":0.92800000000000005},"last":0.55100000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.53900000000000003,"15min":0.55000000000000004},"min":{"1min":0.40200000000000002,"5min":0.35299999999999998,"15min":0.33300000000000002},"max":{"1min":0.78300000000000003,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56899999999999995,"15min":0.57699999999999996},"min":{"1min":0.39100000000000001,"5min":0.379,"15min":0.32700000000000001},"max":{"1min":0.82899999999999996,"5min":0.88900000000000001,"15min":0.96099999999999997},"last":0.502}]}]},{"osd":1,"up_from":13,"seq":55834575006,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":806912,"data_stored":391827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.35999999999999999,"15min":0.35399999999999998},"min":{"1min":0.26500000000000001,"5min":0.22600000000000001,"15min":0.224},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.71699999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.41299999999999998,"15min":0.39400000000000002},"min":{"1min":0.30299999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.56499999999999995,"5min":0.59799999999999998,"15min":0.61699999999999999},"last":0.56499999999999995}]},{"osd":2,"last update":"Sat Apr 6 12:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.40699999999999997,"15min":0.39200000000000002},"min":{"1min":0.245,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.624,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.437,"15min":0.42999999999999999},"min":{"1min":0.30299999999999999,"5min":0.29199999999999998,"15min":0.249},"max":{"1min":0.61699999999999999,"5min":0.64900000000000002,"15min":0.79000000000000004},"last":0.63600000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.45300000000000001,"15min":0.44400000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.29699999999999999},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.84299999999999997},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42999999999999999,"15min":0.41399999999999998},"min":{"1min":0.34399999999999997,"5min":0.26600000000000001,"15min":0.253},"max":{"1min":0.55500000000000005,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.69599999999999995}]},{"osd":4,"last update":"Sat Apr 6 12:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63700000000000001,"15min":0.68100000000000005},"min":{"1min":0.51400000000000001,"5min":0.41399999999999998,"15min":0.34799999999999998},"max":{"1min":0.82899999999999996,"5min":0.84099999999999997,"15min":6.3739999999999997},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.64200000000000002,"15min":0.61899999999999999},"min":{"1min":0.441,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.84299999999999997,"5min":0.94899999999999995,"15min":1.0269999999999999},"last":0.85199999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63500000000000001,"15min":0.60999999999999999},"min":{"1min":0.48699999999999999,"5min":0.442,"15min":0.35999999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.97699999999999998},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64600000000000002,"15min":0.625},"min":{"1min":0.46300000000000002,"5min":0.42499999999999999,"15min":0.35099999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":1.0089999999999999},"last":0.78100000000000003}]},{"osd":6,"last update":"Sat Apr 6 12:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.60899999999999999},"min":{"1min":0.48899999999999999,"5min":0.439,"15min":0.38200000000000001},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.96299999999999997},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67000000000000004,"15min":0.65100000000000002},"min":{"1min":0.54500000000000004,"5min":0.439,"15min":0.41799999999999998},"max":{"1min":0.878,"5min":0.92600000000000005,"15min":1.151},"last":0.77200000000000002}]},{"osd":7,"last update":"Sat Apr 6 12:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.64400000000000002,"15min":0.70599999999999996},"min":{"1min":0.51400000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":6.3639999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.65700000000000003,"15min":0.63800000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.39300000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":1.073},"last":0.95599999999999996}]}]},{"osd":0,"up_from":13,"seq":55834575006,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32480,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339360,"statfs":{"total":96636764160,"available":96603504640,"internally_reserved":0,"allocated":1388544,"data_stored":975471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 12:24:23 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.38600000000000001,"15min":0.373},"min":{"1min":0.29299999999999998,"5min":0.23200000000000001,"15min":0.222},"max":{"1min":0.52400000000000002,"5min":0.627,"15min":0.89100000000000001},"last":0.311},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.41099999999999998,"15min":0.39500000000000002},"min":{"1min":0.26300000000000001,"5min":0.25800000000000001,"15min":0.20899999999999999},"max":{"1min":0.60899999999999999,"5min":0.61099999999999999,"15min":0.67100000000000004},"last":0.29799999999999999}]},{"osd":2,"last update":"Sat Apr 6 12:24:23 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.41799999999999998,"15min":0.41599999999999998},"min":{"1min":0.316,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.59199999999999997,"5min":0.63600000000000001,"15min":1.8089999999999999},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.436,"15min":0.42499999999999999},"min":{"1min":0.34200000000000003,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":0.54800000000000004,"5min":0.64100000000000001,"15min":1.764},"last":0.374}]},{"osd":3,"last update":"Sat Apr 6 12:24:23 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41999999999999998,"15min":0.40200000000000002},"min":{"1min":0.311,"5min":0.30299999999999999,"15min":0.21299999999999999},"max":{"1min":0.55500000000000005,"5min":0.64700000000000002,"15min":0.64800000000000002},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.47399999999999998,"15min":0.45600000000000002},"min":{"1min":0.42099999999999999,"5min":0.313,"15min":0.249},"max":{"1min":0.55600000000000005,"5min":0.64500000000000002,"15min":0.92000000000000004},"last":0.28499999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:24:23 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64200000000000002,"15min":0.61499999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.35699999999999998},"max":{"1min":0.82599999999999996,"5min":0.90000000000000002,"15min":0.93799999999999994},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64400000000000002,"15min":0.622},"min":{"1min":0.47399999999999998,"5min":0.37,"15min":0.311},"max":{"1min":0.81200000000000006,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.69199999999999995}]},{"osd":5,"last update":"Sat Apr 6 12:24:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63800000000000001,"15min":0.60899999999999999},"min":{"1min":0.437,"5min":0.40400000000000003,"15min":0.33700000000000002},"max":{"1min":0.78700000000000003,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.65600000000000003,"15min":0.64800000000000002},"min":{"1min":0.5,"5min":0.315,"15min":0.315},"max":{"1min":0.84099999999999997,"5min":0.94499999999999995,"15min":0.97899999999999998},"last":0.57899999999999996}]},{"osd":6,"last update":"Sat Apr 6 12:24:23 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66700000000000004,"15min":0.64600000000000002},"min":{"1min":0.502,"5min":0.39300000000000002,"15min":0.378},"max":{"1min":0.87,"5min":0.91000000000000003,"15min":0.96699999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.67600000000000005,"15min":0.65900000000000003},"min":{"1min":0.53100000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82899999999999996,"5min":0.92500000000000004,"15min":0.99199999999999999},"last":0.66700000000000004}]},{"osd":7,"last update":"Sat Apr 6 12:24:23 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.67500000000000004,"15min":0.65600000000000003},"min":{"1min":0.56100000000000005,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86399999999999999,"5min":0.89400000000000002,"15min":0.96899999999999997},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.64600000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.36699999999999999},"max":{"1min":0.82699999999999996,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.63200000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575008,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49952,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321888,"statfs":{"total":96636764160,"available":96585613312,"internally_reserved":0,"allocated":798720,"data_stored":386611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39800000000000002,"15min":0.38},"min":{"1min":0.28499999999999998,"5min":0.26800000000000002,"15min":0.21099999999999999},"max":{"1min":0.53300000000000003,"5min":0.56399999999999995,"15min":1.159},"last":0.313},{"interface":"front","average":{"1min":0.436,"5min":0.41899999999999998,"15min":0.40999999999999998},"min":{"1min":0.33000000000000002,"5min":0.25700000000000001,"15min":0.24199999999999999},"max":{"1min":0.58799999999999997,"5min":0.60899999999999999,"15min":1.075},"last":0.33200000000000002}]},{"osd":1,"last update":"Sat Apr 6 12:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.41599999999999998,"15min":0.436},"min":{"1min":0.34599999999999997,"5min":0.27500000000000002,"15min":0.24299999999999999},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":3.6179999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.443,"15min":0.43099999999999999},"min":{"1min":0.33900000000000002,"5min":0.32300000000000001,"15min":0.23599999999999999},"max":{"1min":0.59099999999999997,"5min":0.61399999999999999,"15min":1.125},"last":0.28999999999999998}]},{"osd":3,"last update":"Sat Apr 6 12:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.45900000000000002,"15min":0.47899999999999998},"min":{"1min":0.35299999999999998,"5min":0.29799999999999999,"15min":0.29699999999999999},"max":{"1min":0.56799999999999995,"5min":0.65500000000000003,"15min":3.661},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.439,"15min":0.437},"min":{"1min":0.36399999999999999,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":1.1910000000000001},"last":0.34200000000000003}]},{"osd":4,"last update":"Sat Apr 6 12:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63800000000000001,"15min":0.60199999999999998},"min":{"1min":0.51700000000000002,"5min":0.41899999999999998,"15min":0.36199999999999999},"max":{"1min":0.73199999999999998,"5min":0.88200000000000001,"15min":1.2170000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.65500000000000003,"15min":0.628},"min":{"1min":0.48099999999999998,"5min":0.45900000000000002,"15min":0.30499999999999999},"max":{"1min":0.90200000000000002,"5min":0.92600000000000005,"15min":1.101},"last":0.47499999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65100000000000002,"15min":0.66400000000000003},"min":{"1min":0.498,"5min":0.38400000000000001,"15min":0.32900000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":3.6920000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65100000000000002,"15min":0.626},"min":{"1min":0.46100000000000002,"5min":0.438,"15min":0.39300000000000002},"max":{"1min":0.83799999999999997,"5min":0.91800000000000004,"15min":1.244},"last":0.42399999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.63200000000000001},"min":{"1min":0.50700000000000001,"5min":0.42899999999999999,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":0.93400000000000005,"15min":1.012},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.67400000000000004,"15min":0.68000000000000005},"min":{"1min":0.57599999999999996,"5min":0.44600000000000001,"15min":0.39300000000000002},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":3.7490000000000001},"last":0.58499999999999996}]},{"osd":7,"last update":"Sat Apr 6 12:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.67600000000000005,"15min":0.64900000000000002},"min":{"1min":0.503,"5min":0.47499999999999998,"15min":0.35599999999999998},"max":{"1min":0.86899999999999999,"5min":0.95799999999999996,"15min":1.0469999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.53400000000000003,"5min":0.438,"15min":0.42499999999999999},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":3.7200000000000002},"last":0.60699999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575007,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31924,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339916,"statfs":{"total":96636764160,"available":96604073984,"internally_reserved":0,"allocated":819200,"data_stored":394272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.36399999999999999,"15min":0.35699999999999998},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.191},"max":{"1min":0.48699999999999999,"5min":0.59699999999999998,"15min":0.63900000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.39000000000000001,"15min":0.37},"min":{"1min":0.30199999999999999,"5min":0.25,"15min":0.19600000000000001},"max":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.64400000000000002},"last":0.433}]},{"osd":1,"last update":"Sat Apr 6 12:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.38500000000000001,"15min":0.36799999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.215},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.62},"last":0.373},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.41399999999999998,"15min":0.40300000000000002},"min":{"1min":0.32900000000000001,"5min":0.29199999999999998,"15min":0.247},"max":{"1min":0.53900000000000003,"5min":0.57799999999999996,"15min":0.59099999999999997},"last":0.44900000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.25700000000000001},"max":{"1min":0.64900000000000002,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.47799999999999998,"15min":0.46600000000000003},"min":{"1min":0.372,"5min":0.35699999999999998,"15min":0.29899999999999999},"max":{"1min":0.628,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.51100000000000001}]},{"osd":4,"last update":"Sat Apr 6 12:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.61299999999999999,"15min":0.58799999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.85999999999999999,"5min":0.874,"15min":0.95999999999999996},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.63900000000000001,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.90300000000000002,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.60199999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.66200000000000003,"15min":0.65300000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.33700000000000002},"max":{"1min":0.88900000000000001,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66600000000000004,"15min":0.64400000000000002},"min":{"1min":0.48099999999999998,"5min":0.437,"15min":0.36899999999999999},"max":{"1min":0.86099999999999999,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.74199999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.623,"15min":0.61499999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.77500000000000002,"5min":0.95799999999999996,"15min":0.96999999999999997},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.65900000000000003,"15min":0.64600000000000002},"min":{"1min":0.45500000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84999999999999998,"5min":0.92600000000000005,"15min":0.94699999999999995},"last":0.61899999999999999}]},{"osd":7,"last update":"Sat Apr 6 12:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65000000000000002,"15min":0.63100000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.33300000000000002},"max":{"1min":0.89000000000000001,"5min":0.90700000000000003,"15min":0.93300000000000005},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.66400000000000003,"15min":0.65300000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.373},"max":{"1min":0.81899999999999995,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.629}]}]},{"osd":4,"up_from":13,"seq":55834575007,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":806912,"data_stored":393268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57299999999999995,"15min":0.55800000000000005},"min":{"1min":0.45600000000000002,"5min":0.35299999999999998,"15min":0.23899999999999999},"max":{"1min":0.72099999999999997,"5min":0.80800000000000005,"15min":0.82999999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56499999999999995,"15min":0.54400000000000004},"min":{"1min":0.41399999999999998,"5min":0.38800000000000001,"15min":0.29199999999999998},"max":{"1min":0.68300000000000005,"5min":0.71699999999999997,"15min":0.82199999999999995},"last":0.46400000000000002}]},{"osd":1,"last update":"Sat Apr 6 12:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61499999999999999,"15min":0.66100000000000003},"min":{"1min":0.47699999999999998,"5min":0.36199999999999999,"15min":0.33900000000000002},"max":{"1min":0.70799999999999996,"5min":0.76900000000000002,"15min":4.3479999999999999},"last":0.502},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59899999999999998,"15min":0.65100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70699999999999996,"5min":0.78400000000000003,"15min":4.3319999999999999},"last":0.49399999999999999}]},{"osd":2,"last update":"Sat Apr 6 12:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57599999999999996,"15min":0.56200000000000006},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.311},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.78400000000000003},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59499999999999997,"15min":0.57799999999999996},"min":{"1min":0.45500000000000002,"5min":0.39100000000000001,"15min":0.35199999999999998},"max":{"1min":0.73899999999999999,"5min":0.754,"15min":0.82199999999999995},"last":0.54000000000000004}]},{"osd":3,"last update":"Sat Apr 6 12:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56000000000000005,"15min":0.625},"min":{"1min":0.43099999999999999,"5min":0.39900000000000002,"15min":0.34899999999999998},"max":{"1min":0.71399999999999997,"5min":0.72699999999999998,"15min":4.1950000000000003},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.61099999999999999,"15min":0.60599999999999998},"min":{"1min":0.47199999999999998,"5min":0.441,"15min":0.36099999999999999},"max":{"1min":0.71299999999999997,"5min":0.75900000000000001,"15min":1.2330000000000001},"last":0.52600000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.47599999999999998,"15min":0.5},"min":{"1min":0.316,"5min":0.313,"15min":0.29099999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":1.196},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.49199999999999999,"15min":0.504},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.312},"max":{"1min":0.65800000000000003,"5min":0.745,"15min":0.77000000000000002},"last":0.443}]},{"osd":6,"last update":"Sat Apr 6 12:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.497,"15min":0.52200000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.32000000000000001},"max":{"1min":0.67800000000000005,"5min":0.755,"15min":1.022},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.53700000000000003,"15min":0.55700000000000005},"min":{"1min":0.39200000000000002,"5min":0.378,"15min":0.35699999999999998},"max":{"1min":0.71799999999999997,"5min":0.73699999999999999,"15min":0.872},"last":0.39600000000000002}]},{"osd":7,"last update":"Sat Apr 6 12:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52500000000000002,"15min":0.55200000000000005},"min":{"1min":0.372,"5min":0.36899999999999999,"15min":0.34899999999999998},"max":{"1min":0.65700000000000003,"5min":0.77500000000000002,"15min":0.91500000000000004},"last":0.37},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56299999999999994,"15min":0.59199999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.36799999999999999},"max":{"1min":0.73099999999999998,"5min":0.84299999999999997,"15min":1.0589999999999999},"last":0.58199999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575007,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":786432,"data_stored":385047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:24:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59099999999999997,"15min":0.56899999999999995},"min":{"1min":0.32700000000000001,"5min":0.29099999999999998,"15min":0.24199999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56499999999999995,"15min":0.54700000000000004},"min":{"1min":0.38900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.746,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.73499999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:24:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57799999999999996,"15min":0.55700000000000005},"min":{"1min":0.47399999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.497},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56599999999999995,"15min":0.54900000000000004},"min":{"1min":0.35799999999999998,"5min":0.34499999999999997,"15min":0.30599999999999999},"max":{"1min":0.72099999999999997,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.70599999999999996}]},{"osd":2,"last update":"Sat Apr 6 12:24:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58599999999999997,"15min":0.55700000000000005},"min":{"1min":0.38400000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.57899999999999996,"15min":0.56699999999999995},"min":{"1min":0.47499999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.76100000000000001,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.747}]},{"osd":3,"last update":"Sat Apr 6 12:24:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58599999999999997,"15min":0.57199999999999995},"min":{"1min":0.39500000000000002,"5min":0.32500000000000001,"15min":0.31},"max":{"1min":0.76900000000000002,"5min":0.77300000000000002,"15min":0.79400000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61099999999999999,"15min":0.59799999999999998},"min":{"1min":0.45200000000000001,"5min":0.377,"15min":0.32300000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.58699999999999997}]},{"osd":4,"last update":"Sat Apr 6 12:24:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.47499999999999998,"15min":0.48699999999999999},"min":{"1min":0.32600000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.70299999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.437},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.54900000000000004},"min":{"1min":0.40500000000000003,"5min":0.33300000000000002,"15min":0.29499999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.91800000000000004},"last":0.45100000000000001}]},{"osd":6,"last update":"Sat Apr 6 12:24:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.48999999999999999,"15min":0.505},"min":{"1min":0.36499999999999999,"5min":0.34100000000000003,"15min":0.28000000000000003},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.88700000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.503,"15min":0.51200000000000001},"min":{"1min":0.41599999999999998,"5min":0.33800000000000002,"15min":0.32100000000000001},"max":{"1min":0.72799999999999998,"5min":0.77000000000000002,"15min":0.83299999999999996},"last":0.34699999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:24:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.51700000000000002,"15min":0.52600000000000002},"min":{"1min":0.40000000000000002,"5min":0.314,"15min":0.252},"max":{"1min":0.72399999999999998,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.35199999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.85599999999999998},"last":0.48499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:25:06.194 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:25:06.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.353+0000 7ff26a9ae700 1 Processor -- start 2024-04-06T12:25:06.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.353+0000 7ff26a9ae700 1 -- start start 2024-04-06T12:25:06.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.354+0000 7ff26a9ae700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff26415d7b0 0x7ff2640b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:06.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.354+0000 7ff26a9ae700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff26406a730 0x7ff26406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:06.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.354+0000 7ff26a9ae700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2640b3990 con 0x7ff26415ddf0 2024-04-06T12:25:06.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.354+0000 7ff26a9ae700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff26406b0f0 con 0x7ff26415d7b0 2024-04-06T12:25:06.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.354+0000 7ff26a9ae700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2640b3060 con 0x7ff26406a730 2024-04-06T12:25:06.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.354+0000 7ff263fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff26415d7b0 0x7ff2640b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:06.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.354+0000 7ff263fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff26415d7b0 0x7ff2640b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57904/0 (socket says 172.21.15.189:57904) 2024-04-06T12:25:06.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.354+0000 7ff263fff700 1 -- 172.21.15.189:0/2221779060 learned_addr learned my addr 172.21.15.189:0/2221779060 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:06.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.354+0000 7ff2637fe700 1 --2- 172.21.15.189:0/2221779060 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff26406a730 0x7ff26406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:06.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.354+0000 7ff263fff700 1 -- 172.21.15.189:0/2221779060 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff26406a730 msgr2=0x7ff26406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:06.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.354+0000 7ff263fff700 1 --2- 172.21.15.189:0/2221779060 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff26406a730 0x7ff26406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:06.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.354+0000 7ff263fff700 1 -- 172.21.15.189:0/2221779060 >> v1:172.21.15.192:6789/0 conn(0x7ff26415ddf0 legacy=0x7ff26415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:06.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.354+0000 7ff263fff700 1 -- 172.21.15.189:0/2221779060 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2640b3340 con 0x7ff26415d7b0 2024-04-06T12:25:06.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.354+0000 7ff263fff700 1 --2- 172.21.15.189:0/2221779060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff26415d7b0 0x7ff2640b3e20 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff2540098f0 tx=0x7ff254009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e93cf2cd9bf4135 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:06.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.354+0000 7ff262ffd700 1 -- 172.21.15.189:0/2221779060 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25400b8a0 con 0x7ff26415d7b0 2024-04-06T12:25:06.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.354+0000 7ff262ffd700 1 -- 172.21.15.189:0/2221779060 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25400ba00 con 0x7ff26415d7b0 2024-04-06T12:25:06.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.354+0000 7ff262ffd700 1 -- 172.21.15.189:0/2221779060 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25400bd90 con 0x7ff26415d7b0 2024-04-06T12:25:06.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.355+0000 7ff26a9ae700 1 -- 172.21.15.189:0/2221779060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff26415d7b0 msgr2=0x7ff2640b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:06.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.355+0000 7ff26a9ae700 1 --2- 172.21.15.189:0/2221779060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff26415d7b0 0x7ff2640b3e20 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff2540098f0 tx=0x7ff254009c00 comp rx=0 tx=0).stop 2024-04-06T12:25:06.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.355+0000 7ff26a9ae700 1 -- 172.21.15.189:0/2221779060 shutdown_connections 2024-04-06T12:25:06.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.355+0000 7ff26a9ae700 1 --2- 172.21.15.189:0/2221779060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff26415d7b0 0x7ff2640b3e20 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:06.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.355+0000 7ff26a9ae700 1 --2- 172.21.15.189:0/2221779060 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff26406a730 0x7ff26406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:06.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.355+0000 7ff26a9ae700 1 -- 172.21.15.189:0/2221779060 >> 172.21.15.189:0/2221779060 conn(0x7ff2640d00b0 msgr2=0x7ff2640d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:06.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.355+0000 7ff26a9ae700 1 -- 172.21.15.189:0/2221779060 shutdown_connections 2024-04-06T12:25:06.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.355+0000 7ff26a9ae700 1 -- 172.21.15.189:0/2221779060 wait complete. 2024-04-06T12:25:06.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.356+0000 7ff26a9ae700 1 Processor -- start 2024-04-06T12:25:06.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.356+0000 7ff26a9ae700 1 -- start start 2024-04-06T12:25:06.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.357+0000 7ff26a9ae700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff26406a730 0x7ff2640af590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:06.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.357+0000 7ff26a9ae700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff26415d7b0 0x7ff2640cea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:06.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.357+0000 7ff26a9ae700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff26415ddf0 0x7ff2640b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:06.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.357+0000 7ff26a9ae700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2640cef40 con 0x7ff26415ddf0 2024-04-06T12:25:06.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.357+0000 7ff26a9ae700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2640afc20 con 0x7ff26406a730 2024-04-06T12:25:06.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.357+0000 7ff26a9ae700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2640aff00 con 0x7ff26415d7b0 2024-04-06T12:25:06.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.357+0000 7ff268f4b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff26415ddf0 0x7ff2640b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:06.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.357+0000 7ff268f4b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff26415ddf0 0x7ff2640b13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57906/0 (socket says 172.21.15.189:57906) 2024-04-06T12:25:06.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.357+0000 7ff268f4b700 1 -- 172.21.15.189:0/2503578360 learned_addr learned my addr 172.21.15.189:0/2503578360 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:06.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.357+0000 7ff2637fe700 1 --2- 172.21.15.189:0/2503578360 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff26415d7b0 0x7ff2640cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:06.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.357+0000 7ff263fff700 1 --2- 172.21.15.189:0/2503578360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff26406a730 0x7ff2640af590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:06.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.358+0000 7ff268f4b700 1 -- 172.21.15.189:0/2503578360 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff26415d7b0 msgr2=0x7ff2640cea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:06.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.358+0000 7ff268f4b700 1 --2- 172.21.15.189:0/2503578360 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff26415d7b0 0x7ff2640cea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:06.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.358+0000 7ff268f4b700 1 -- 172.21.15.189:0/2503578360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff26406a730 msgr2=0x7ff2640af590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:06.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.358+0000 7ff268f4b700 1 --2- 172.21.15.189:0/2503578360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff26406a730 0x7ff2640af590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:06.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.358+0000 7ff268f4b700 1 -- 172.21.15.189:0/2503578360 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff254009550 con 0x7ff26415ddf0 2024-04-06T12:25:06.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.358+0000 7ff2637fe700 1 --2- 172.21.15.189:0/2503578360 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff26415d7b0 0x7ff2640cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:25:06.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.358+0000 7ff268f4b700 1 --2- 172.21.15.189:0/2503578360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff26415ddf0 0x7ff2640b13e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff25c00cb50 tx=0x7ff25c00cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:06.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.358+0000 7ff260ff9700 1 -- 172.21.15.189:0/2503578360 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c003c00 con 0x7ff26415ddf0 2024-04-06T12:25:06.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.358+0000 7ff260ff9700 1 -- 172.21.15.189:0/2503578360 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25c003d60 con 0x7ff26415ddf0 2024-04-06T12:25:06.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.358+0000 7ff26a9ae700 1 -- 172.21.15.189:0/2503578360 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2640b01e0 con 0x7ff26415ddf0 2024-04-06T12:25:06.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.358+0000 7ff260ff9700 1 -- 172.21.15.189:0/2503578360 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c00ab70 con 0x7ff26415ddf0 2024-04-06T12:25:06.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.358+0000 7ff26a9ae700 1 -- 172.21.15.189:0/2503578360 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff264155990 con 0x7ff26415ddf0 2024-04-06T12:25:06.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.359+0000 7ff26a9ae700 1 -- 172.21.15.189:0/2503578360 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff264155c40 con 0x7ff26415ddf0 2024-04-06T12:25:06.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.359+0000 7ff260ff9700 1 -- 172.21.15.189:0/2503578360 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7ff25c00acd0 con 0x7ff26415ddf0 2024-04-06T12:25:06.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.360+0000 7ff260ff9700 1 --2- 172.21.15.189:0/2503578360 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff25006adb0 0x7ff25006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:06.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.360+0000 7ff263fff700 1 --2- 172.21.15.189:0/2503578360 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff25006adb0 0x7ff25006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:06.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.360+0000 7ff260ff9700 1 -- 172.21.15.189:0/2503578360 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff25c0896c0 con 0x7ff26415ddf0 2024-04-06T12:25:06.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.360+0000 7ff263fff700 1 --2- 172.21.15.189:0/2503578360 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff25006adb0 0x7ff25006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff254009c40 tx=0x7ff254008030 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:06.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.364+0000 7ff260ff9700 1 -- 172.21.15.189:0/2503578360 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff264155c40 con 0x7ff26415ddf0 2024-04-06T12:25:06.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.555+0000 7ff26a9ae700 1 -- 172.21.15.189:0/2503578360 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff26415eb40 con 0x7ff25006adb0 2024-04-06T12:25:06.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.563+0000 7ff260ff9700 1 -- 172.21.15.189:0/2503578360 <== mgr.5225 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289571 (secure 0 0 0) 0x7ff25400bb70 con 0x7ff25006adb0 2024-04-06T12:25:06.565 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:25:06.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.567+0000 7ff26a9ae700 1 -- 172.21.15.189:0/2503578360 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff25006adb0 msgr2=0x7ff25006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:06.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.567+0000 7ff26a9ae700 1 --2- 172.21.15.189:0/2503578360 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff25006adb0 0x7ff25006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff254009c40 tx=0x7ff254008030 comp rx=0 tx=0).stop 2024-04-06T12:25:06.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.567+0000 7ff26a9ae700 1 -- 172.21.15.189:0/2503578360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff26415ddf0 msgr2=0x7ff2640b13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:06.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.567+0000 7ff26a9ae700 1 --2- 172.21.15.189:0/2503578360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff26415ddf0 0x7ff2640b13e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff25c00cb50 tx=0x7ff25c00cf10 comp rx=0 tx=0).stop 2024-04-06T12:25:06.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.567+0000 7ff26a9ae700 1 -- 172.21.15.189:0/2503578360 shutdown_connections 2024-04-06T12:25:06.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.567+0000 7ff26a9ae700 1 --2- 172.21.15.189:0/2503578360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff26415ddf0 0x7ff2640b13e0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:06.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.567+0000 7ff26a9ae700 1 --2- 172.21.15.189:0/2503578360 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff25006adb0 0x7ff25006d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:06.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.567+0000 7ff26a9ae700 1 --2- 172.21.15.189:0/2503578360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff26406a730 0x7ff2640af590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:06.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.567+0000 7ff26a9ae700 1 --2- 172.21.15.189:0/2503578360 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff26415d7b0 0x7ff2640cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:06.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.567+0000 7ff26a9ae700 1 -- 172.21.15.189:0/2503578360 >> 172.21.15.189:0/2503578360 conn(0x7ff2640d00b0 msgr2=0x7ff26415c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:06.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.567+0000 7ff26a9ae700 1 -- 172.21.15.189:0/2503578360 shutdown_connections 2024-04-06T12:25:06.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.567+0000 7ff26a9ae700 1 -- 172.21.15.189:0/2503578360 wait complete. 2024-04-06T12:25:06.568 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:25:06.577 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-04-06T12:25:06.396501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":284300,"kb_used_data":7428,"kb_used_omap":12,"kb_used_meta":276723,"kb_avail":754690420,"statfs":{"total":773094113280,"available":772802990080,"internally_reserved":0,"allocated":7606272,"data_stored":4295373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":283364920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.758119"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.048034+0000","last_change":"2024-04-06T12:25:02.091668+0000","last_active":"2024-04-06T12:25:04.048034+0000","last_peered":"2024-04-06T12:25:04.048034+0000","last_clean":"2024-04-06T12:25:04.048034+0000","last_became_active":"2024-04-06T12:25:02.091427+0000","last_became_peered":"2024-04-06T12:25:02.091427+0000","last_unstale":"2024-04-06T12:25:04.048034+0000","last_undegraded":"2024-04-06T12:25:04.048034+0000","last_fullsized":"2024-04-06T12:25:04.048034+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:18:36.819518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073692+0000","last_change":"2024-04-06T12:25:02.093907+0000","last_active":"2024-04-06T12:25:03.073692+0000","last_peered":"2024-04-06T12:25:03.073692+0000","last_clean":"2024-04-06T12:25:03.073692+0000","last_became_active":"2024-04-06T12:25:02.093615+0000","last_became_peered":"2024-04-06T12:25:02.093615+0000","last_unstale":"2024-04-06T12:25:03.073692+0000","last_undegraded":"2024-04-06T12:25:03.073692+0000","last_fullsized":"2024-04-06T12:25:03.073692+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:39:34.096584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.084504+0000","last_change":"2024-04-06T12:25:02.080403+0000","last_active":"2024-04-06T12:25:02.084504+0000","last_peered":"2024-04-06T12:25:02.084504+0000","last_clean":"2024-04-06T12:25:02.084504+0000","last_became_active":"2024-04-06T12:25:02.080139+0000","last_became_peered":"2024-04-06T12:25:02.080139+0000","last_unstale":"2024-04-06T12:25:02.084504+0000","last_undegraded":"2024-04-06T12:25:02.084504+0000","last_fullsized":"2024-04-06T12:25:02.084504+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:08:57.284175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073630+0000","last_change":"2024-04-06T12:25:02.085112+0000","last_active":"2024-04-06T12:25:03.073630+0000","last_peered":"2024-04-06T12:25:03.073630+0000","last_clean":"2024-04-06T12:25:03.073630+0000","last_became_active":"2024-04-06T12:25:02.084865+0000","last_became_peered":"2024-04-06T12:25:02.084865+0000","last_unstale":"2024-04-06T12:25:03.073630+0000","last_undegraded":"2024-04-06T12:25:03.073630+0000","last_fullsized":"2024-04-06T12:25:03.073630+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:17:26.548465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071988+0000","last_change":"2024-04-06T12:25:02.086962+0000","last_active":"2024-04-06T12:25:03.071988+0000","last_peered":"2024-04-06T12:25:03.071988+0000","last_clean":"2024-04-06T12:25:03.071988+0000","last_became_active":"2024-04-06T12:25:02.086701+0000","last_became_peered":"2024-04-06T12:25:02.086701+0000","last_unstale":"2024-04-06T12:25:03.071988+0000","last_undegraded":"2024-04-06T12:25:03.071988+0000","last_fullsized":"2024-04-06T12:25:03.071988+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:40.362773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.047869+0000","last_change":"2024-04-06T12:25:02.086636+0000","last_active":"2024-04-06T12:25:04.047869+0000","last_peered":"2024-04-06T12:25:04.047869+0000","last_clean":"2024-04-06T12:25:04.047869+0000","last_became_active":"2024-04-06T12:25:02.086253+0000","last_became_peered":"2024-04-06T12:25:02.086253+0000","last_unstale":"2024-04-06T12:25:04.047869+0000","last_undegraded":"2024-04-06T12:25:04.047869+0000","last_fullsized":"2024-04-06T12:25:04.047869+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:40.362773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.084492+0000","last_change":"2024-04-06T12:25:02.078319+0000","last_active":"2024-04-06T12:25:02.084492+0000","last_peered":"2024-04-06T12:25:02.084492+0000","last_clean":"2024-04-06T12:25:02.084492+0000","last_became_active":"2024-04-06T12:25:02.075399+0000","last_became_peered":"2024-04-06T12:25:02.075399+0000","last_unstale":"2024-04-06T12:25:02.084492+0000","last_undegraded":"2024-04-06T12:25:02.084492+0000","last_fullsized":"2024-04-06T12:25:02.084492+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:55:23.868104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071737+0000","last_change":"2024-04-06T12:25:02.083809+0000","last_active":"2024-04-06T12:25:03.071737+0000","last_peered":"2024-04-06T12:25:03.071737+0000","last_clean":"2024-04-06T12:25:03.071737+0000","last_became_active":"2024-04-06T12:25:02.083403+0000","last_became_peered":"2024-04-06T12:25:02.083403+0000","last_unstale":"2024-04-06T12:25:03.071737+0000","last_undegraded":"2024-04-06T12:25:03.071737+0000","last_fullsized":"2024-04-06T12:25:03.071737+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:43:10.545910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.072369+0000","last_change":"2024-04-06T12:25:02.091817+0000","last_active":"2024-04-06T12:25:03.072369+0000","last_peered":"2024-04-06T12:25:03.072369+0000","last_clean":"2024-04-06T12:25:03.072369+0000","last_became_active":"2024-04-06T12:25:02.091558+0000","last_became_peered":"2024-04-06T12:25:02.091558+0000","last_unstale":"2024-04-06T12:25:03.072369+0000","last_undegraded":"2024-04-06T12:25:03.072369+0000","last_fullsized":"2024-04-06T12:25:03.072369+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:02:05.954262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.048042+0000","last_change":"2024-04-06T12:25:02.082783+0000","last_active":"2024-04-06T12:25:04.048042+0000","last_peered":"2024-04-06T12:25:04.048042+0000","last_clean":"2024-04-06T12:25:04.048042+0000","last_became_active":"2024-04-06T12:25:02.082539+0000","last_became_peered":"2024-04-06T12:25:02.082539+0000","last_unstale":"2024-04-06T12:25:04.048042+0000","last_undegraded":"2024-04-06T12:25:04.048042+0000","last_fullsized":"2024-04-06T12:25:04.048042+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:35:08.942032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.081831+0000","last_change":"2024-04-06T12:25:02.079287+0000","last_active":"2024-04-06T12:25:02.081831+0000","last_peered":"2024-04-06T12:25:02.081831+0000","last_clean":"2024-04-06T12:25:02.081831+0000","last_became_active":"2024-04-06T12:25:02.079033+0000","last_became_peered":"2024-04-06T12:25:02.079033+0000","last_unstale":"2024-04-06T12:25:02.081831+0000","last_undegraded":"2024-04-06T12:25:02.081831+0000","last_fullsized":"2024-04-06T12:25:02.081831+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:38:32.224967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.072006+0000","last_change":"2024-04-06T12:25:02.095882+0000","last_active":"2024-04-06T12:25:03.072006+0000","last_peered":"2024-04-06T12:25:03.072006+0000","last_clean":"2024-04-06T12:25:03.072006+0000","last_became_active":"2024-04-06T12:25:02.095625+0000","last_became_peered":"2024-04-06T12:25:02.095625+0000","last_unstale":"2024-04-06T12:25:03.072006+0000","last_undegraded":"2024-04-06T12:25:03.072006+0000","last_fullsized":"2024-04-06T12:25:03.072006+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:08:57.284175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.047837+0000","last_change":"2024-04-06T12:25:02.082956+0000","last_active":"2024-04-06T12:25:04.047837+0000","last_peered":"2024-04-06T12:25:04.047837+0000","last_clean":"2024-04-06T12:25:04.047837+0000","last_became_active":"2024-04-06T12:25:02.082703+0000","last_became_peered":"2024-04-06T12:25:02.082703+0000","last_unstale":"2024-04-06T12:25:04.047837+0000","last_undegraded":"2024-04-06T12:25:04.047837+0000","last_fullsized":"2024-04-06T12:25:04.047837+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:23:58.314601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.081064+0000","last_change":"2024-04-06T12:25:02.076125+0000","last_active":"2024-04-06T12:25:02.081064+0000","last_peered":"2024-04-06T12:25:02.081064+0000","last_clean":"2024-04-06T12:25:02.081064+0000","last_became_active":"2024-04-06T12:25:02.075868+0000","last_became_peered":"2024-04-06T12:25:02.075868+0000","last_unstale":"2024-04-06T12:25:02.081064+0000","last_undegraded":"2024-04-06T12:25:02.081064+0000","last_fullsized":"2024-04-06T12:25:02.081064+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:15:22.102032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073232+0000","last_change":"2024-04-06T12:25:02.088728+0000","last_active":"2024-04-06T12:25:03.073232+0000","last_peered":"2024-04-06T12:25:03.073232+0000","last_clean":"2024-04-06T12:25:03.073232+0000","last_became_active":"2024-04-06T12:25:02.088455+0000","last_became_peered":"2024-04-06T12:25:02.088455+0000","last_unstale":"2024-04-06T12:25:03.073232+0000","last_undegraded":"2024-04-06T12:25:03.073232+0000","last_fullsized":"2024-04-06T12:25:03.073232+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:44:39.649767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.047793+0000","last_change":"2024-04-06T12:25:02.083055+0000","last_active":"2024-04-06T12:25:04.047793+0000","last_peered":"2024-04-06T12:25:04.047793+0000","last_clean":"2024-04-06T12:25:04.047793+0000","last_became_active":"2024-04-06T12:25:02.082275+0000","last_became_peered":"2024-04-06T12:25:02.082275+0000","last_unstale":"2024-04-06T12:25:04.047793+0000","last_undegraded":"2024-04-06T12:25:04.047793+0000","last_fullsized":"2024-04-06T12:25:04.047793+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:58:02.162124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.090737+0000","last_change":"2024-04-06T12:25:02.090005+0000","last_active":"2024-04-06T12:25:02.090737+0000","last_peered":"2024-04-06T12:25:02.090737+0000","last_clean":"2024-04-06T12:25:02.090737+0000","last_became_active":"2024-04-06T12:25:02.089551+0000","last_became_peered":"2024-04-06T12:25:02.089551+0000","last_unstale":"2024-04-06T12:25:02.090737+0000","last_undegraded":"2024-04-06T12:25:02.090737+0000","last_fullsized":"2024-04-06T12:25:02.090737+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:50:29.941582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073656+0000","last_change":"2024-04-06T12:25:02.093300+0000","last_active":"2024-04-06T12:25:03.073656+0000","last_peered":"2024-04-06T12:25:03.073656+0000","last_clean":"2024-04-06T12:25:03.073656+0000","last_became_active":"2024-04-06T12:25:02.093076+0000","last_became_peered":"2024-04-06T12:25:02.093076+0000","last_unstale":"2024-04-06T12:25:03.073656+0000","last_undegraded":"2024-04-06T12:25:03.073656+0000","last_fullsized":"2024-04-06T12:25:03.073656+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:01:57.407601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.084958+0000","last_change":"2024-04-06T12:25:02.081681+0000","last_active":"2024-04-06T12:25:02.084958+0000","last_peered":"2024-04-06T12:25:02.084958+0000","last_clean":"2024-04-06T12:25:02.084958+0000","last_became_active":"2024-04-06T12:25:02.081424+0000","last_became_peered":"2024-04-06T12:25:02.081424+0000","last_unstale":"2024-04-06T12:25:02.084958+0000","last_undegraded":"2024-04-06T12:25:02.084958+0000","last_fullsized":"2024-04-06T12:25:02.084958+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:23:01.737309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.098641+0000","last_change":"2024-04-06T12:25:02.093057+0000","last_active":"2024-04-06T12:25:02.098641+0000","last_peered":"2024-04-06T12:25:02.098641+0000","last_clean":"2024-04-06T12:25:02.098641+0000","last_became_active":"2024-04-06T12:25:02.092777+0000","last_became_peered":"2024-04-06T12:25:02.092777+0000","last_unstale":"2024-04-06T12:25:02.098641+0000","last_undegraded":"2024-04-06T12:25:02.098641+0000","last_fullsized":"2024-04-06T12:25:02.098641+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:06:53.161677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.098420+0000","last_change":"2024-04-06T12:25:02.087620+0000","last_active":"2024-04-06T12:25:02.098420+0000","last_peered":"2024-04-06T12:25:02.098420+0000","last_clean":"2024-04-06T12:25:02.098420+0000","last_became_active":"2024-04-06T12:25:02.087350+0000","last_became_peered":"2024-04-06T12:25:02.087350+0000","last_unstale":"2024-04-06T12:25:02.098420+0000","last_undegraded":"2024-04-06T12:25:02.098420+0000","last_fullsized":"2024-04-06T12:25:02.098420+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:27:28.547443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071728+0000","last_change":"2024-04-06T12:25:02.091096+0000","last_active":"2024-04-06T12:25:03.071728+0000","last_peered":"2024-04-06T12:25:03.071728+0000","last_clean":"2024-04-06T12:25:03.071728+0000","last_became_active":"2024-04-06T12:25:02.090869+0000","last_became_peered":"2024-04-06T12:25:02.090869+0000","last_unstale":"2024-04-06T12:25:03.071728+0000","last_undegraded":"2024-04-06T12:25:03.071728+0000","last_fullsized":"2024-04-06T12:25:03.071728+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:07.223156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.085109+0000","last_change":"2024-04-06T12:25:02.078998+0000","last_active":"2024-04-06T12:25:02.085109+0000","last_peered":"2024-04-06T12:25:02.085109+0000","last_clean":"2024-04-06T12:25:02.085109+0000","last_became_active":"2024-04-06T12:25:02.078653+0000","last_became_peered":"2024-04-06T12:25:02.078653+0000","last_unstale":"2024-04-06T12:25:02.085109+0000","last_undegraded":"2024-04-06T12:25:02.085109+0000","last_fullsized":"2024-04-06T12:25:02.085109+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:23:29.611147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.081007+0000","last_change":"2024-04-06T12:25:02.075714+0000","last_active":"2024-04-06T12:25:02.081007+0000","last_peered":"2024-04-06T12:25:02.081007+0000","last_clean":"2024-04-06T12:25:02.081007+0000","last_became_active":"2024-04-06T12:25:02.075370+0000","last_became_peered":"2024-04-06T12:25:02.075370+0000","last_unstale":"2024-04-06T12:25:02.081007+0000","last_undegraded":"2024-04-06T12:25:02.081007+0000","last_fullsized":"2024-04-06T12:25:02.081007+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:53:56.158924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.089978+0000","last_change":"2024-04-06T12:25:02.084771+0000","last_active":"2024-04-06T12:25:02.089978+0000","last_peered":"2024-04-06T12:25:02.089978+0000","last_clean":"2024-04-06T12:25:02.089978+0000","last_became_active":"2024-04-06T12:25:02.084522+0000","last_became_peered":"2024-04-06T12:25:02.084522+0000","last_unstale":"2024-04-06T12:25:02.089978+0000","last_undegraded":"2024-04-06T12:25:02.089978+0000","last_fullsized":"2024-04-06T12:25:02.089978+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:20:14.893701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.072108+0000","last_change":"2024-04-06T12:25:02.095349+0000","last_active":"2024-04-06T12:25:03.072108+0000","last_peered":"2024-04-06T12:25:03.072108+0000","last_clean":"2024-04-06T12:25:03.072108+0000","last_became_active":"2024-04-06T12:25:02.092121+0000","last_became_peered":"2024-04-06T12:25:02.092121+0000","last_unstale":"2024-04-06T12:25:03.072108+0000","last_undegraded":"2024-04-06T12:25:03.072108+0000","last_fullsized":"2024-04-06T12:25:03.072108+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:50:29.941582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.048135+0000","last_change":"2024-04-06T12:25:02.085811+0000","last_active":"2024-04-06T12:25:04.048135+0000","last_peered":"2024-04-06T12:25:04.048135+0000","last_clean":"2024-04-06T12:25:04.048135+0000","last_became_active":"2024-04-06T12:25:02.082852+0000","last_became_peered":"2024-04-06T12:25:02.082852+0000","last_unstale":"2024-04-06T12:25:04.048135+0000","last_undegraded":"2024-04-06T12:25:04.048135+0000","last_fullsized":"2024-04-06T12:25:04.048135+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:19:09.708529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.098403+0000","last_change":"2024-04-06T12:25:02.087472+0000","last_active":"2024-04-06T12:25:02.098403+0000","last_peered":"2024-04-06T12:25:02.098403+0000","last_clean":"2024-04-06T12:25:02.098403+0000","last_became_active":"2024-04-06T12:25:02.087163+0000","last_became_peered":"2024-04-06T12:25:02.087163+0000","last_unstale":"2024-04-06T12:25:02.098403+0000","last_undegraded":"2024-04-06T12:25:02.098403+0000","last_fullsized":"2024-04-06T12:25:02.098403+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:00:56.274953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.090634+0000","last_change":"2024-04-06T12:25:02.088495+0000","last_active":"2024-04-06T12:25:02.090634+0000","last_peered":"2024-04-06T12:25:02.090634+0000","last_clean":"2024-04-06T12:25:02.090634+0000","last_became_active":"2024-04-06T12:25:02.088224+0000","last_became_peered":"2024-04-06T12:25:02.088224+0000","last_unstale":"2024-04-06T12:25:02.090634+0000","last_undegraded":"2024-04-06T12:25:02.090634+0000","last_fullsized":"2024-04-06T12:25:02.090634+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:42:56.852103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073403+0000","last_change":"2024-04-06T12:25:02.084707+0000","last_active":"2024-04-06T12:25:03.073403+0000","last_peered":"2024-04-06T12:25:03.073403+0000","last_clean":"2024-04-06T12:25:03.073403+0000","last_became_active":"2024-04-06T12:25:02.083891+0000","last_became_peered":"2024-04-06T12:25:02.083891+0000","last_unstale":"2024-04-06T12:25:03.073403+0000","last_undegraded":"2024-04-06T12:25:03.073403+0000","last_fullsized":"2024-04-06T12:25:03.073403+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:35:04.762031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.081751+0000","last_change":"2024-04-06T12:25:02.077742+0000","last_active":"2024-04-06T12:25:02.081751+0000","last_peered":"2024-04-06T12:25:02.081751+0000","last_clean":"2024-04-06T12:25:02.081751+0000","last_became_active":"2024-04-06T12:25:02.075972+0000","last_became_peered":"2024-04-06T12:25:02.075972+0000","last_unstale":"2024-04-06T12:25:02.081751+0000","last_undegraded":"2024-04-06T12:25:02.081751+0000","last_fullsized":"2024-04-06T12:25:02.081751+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:43:10.545910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.091981+0000","last_change":"2024-04-06T12:25:02.089285+0000","last_active":"2024-04-06T12:25:02.091981+0000","last_peered":"2024-04-06T12:25:02.091981+0000","last_clean":"2024-04-06T12:25:02.091981+0000","last_became_active":"2024-04-06T12:25:02.088982+0000","last_became_peered":"2024-04-06T12:25:02.088982+0000","last_unstale":"2024-04-06T12:25:02.091981+0000","last_undegraded":"2024-04-06T12:25:02.091981+0000","last_fullsized":"2024-04-06T12:25:02.091981+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:54:46.589494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.072901+0000","last_change":"2024-04-06T12:25:02.093182+0000","last_active":"2024-04-06T12:25:03.072901+0000","last_peered":"2024-04-06T12:25:03.072901+0000","last_clean":"2024-04-06T12:25:03.072901+0000","last_became_active":"2024-04-06T12:25:02.092907+0000","last_became_peered":"2024-04-06T12:25:02.092907+0000","last_unstale":"2024-04-06T12:25:03.072901+0000","last_undegraded":"2024-04-06T12:25:03.072901+0000","last_fullsized":"2024-04-06T12:25:03.072901+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:47:17.110195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"66'4","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.098690+0000","last_change":"2024-04-06T12:25:01.657867+0000","last_active":"2024-04-06T12:25:02.098690+0000","last_peered":"2024-04-06T12:25:02.098690+0000","last_clean":"2024-04-06T12:25:02.098690+0000","last_became_active":"2024-04-06T12:25:01.657651+0000","last_became_peered":"2024-04-06T12:25:01.657651+0000","last_unstale":"2024-04-06T12:25:02.098690+0000","last_undegraded":"2024-04-06T12:25:02.098690+0000","last_fullsized":"2024-04-06T12:25:02.098690+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:47:08.361971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.092065+0000","last_change":"2024-04-06T12:25:02.089234+0000","last_active":"2024-04-06T12:25:02.092065+0000","last_peered":"2024-04-06T12:25:02.092065+0000","last_clean":"2024-04-06T12:25:02.092065+0000","last_became_active":"2024-04-06T12:25:02.088889+0000","last_became_peered":"2024-04-06T12:25:02.088889+0000","last_unstale":"2024-04-06T12:25:02.092065+0000","last_undegraded":"2024-04-06T12:25:02.092065+0000","last_fullsized":"2024-04-06T12:25:02.092065+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:48:19.997695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"66'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.048105+0000","last_change":"2024-04-06T12:25:01.664838+0000","last_active":"2024-04-06T12:25:04.048105+0000","last_peered":"2024-04-06T12:25:04.048105+0000","last_clean":"2024-04-06T12:25:04.048105+0000","last_became_active":"2024-04-06T12:25:01.664565+0000","last_became_peered":"2024-04-06T12:25:01.664565+0000","last_unstale":"2024-04-06T12:25:04.048105+0000","last_undegraded":"2024-04-06T12:25:04.048105+0000","last_fullsized":"2024-04-06T12:25:04.048105+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:06:52.156959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.098453+0000","last_change":"2024-04-06T12:25:02.088084+0000","last_active":"2024-04-06T12:25:02.098453+0000","last_peered":"2024-04-06T12:25:02.098453+0000","last_clean":"2024-04-06T12:25:02.098453+0000","last_became_active":"2024-04-06T12:25:02.087796+0000","last_became_peered":"2024-04-06T12:25:02.087796+0000","last_unstale":"2024-04-06T12:25:02.098453+0000","last_undegraded":"2024-04-06T12:25:02.098453+0000","last_fullsized":"2024-04-06T12:25:02.098453+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:47:52.425649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.048118+0000","last_change":"2024-04-06T12:25:01.668912+0000","last_active":"2024-04-06T12:25:04.048118+0000","last_peered":"2024-04-06T12:25:04.048118+0000","last_clean":"2024-04-06T12:25:04.048118+0000","last_became_active":"2024-04-06T12:25:01.668707+0000","last_became_peered":"2024-04-06T12:25:01.668707+0000","last_unstale":"2024-04-06T12:25:04.048118+0000","last_undegraded":"2024-04-06T12:25:04.048118+0000","last_fullsized":"2024-04-06T12:25:04.048118+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:47:51.420931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071843+0000","last_change":"2024-04-06T12:25:02.093680+0000","last_active":"2024-04-06T12:25:03.071843+0000","last_peered":"2024-04-06T12:25:03.071843+0000","last_clean":"2024-04-06T12:25:03.071843+0000","last_became_active":"2024-04-06T12:25:02.093385+0000","last_became_peered":"2024-04-06T12:25:02.093385+0000","last_unstale":"2024-04-06T12:25:03.071843+0000","last_undegraded":"2024-04-06T12:25:03.071843+0000","last_fullsized":"2024-04-06T12:25:03.071843+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:57:44.456276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"66'2","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.091004+0000","last_change":"2024-04-06T12:25:01.658400+0000","last_active":"2024-04-06T12:25:02.091004+0000","last_peered":"2024-04-06T12:25:02.091004+0000","last_clean":"2024-04-06T12:25:02.091004+0000","last_became_active":"2024-04-06T12:25:01.658129+0000","last_became_peered":"2024-04-06T12:25:01.658129+0000","last_unstale":"2024-04-06T12:25:02.091004+0000","last_undegraded":"2024-04-06T12:25:02.091004+0000","last_fullsized":"2024-04-06T12:25:02.091004+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:10:32.281402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.072749+0000","last_change":"2024-04-06T12:25:02.084471+0000","last_active":"2024-04-06T12:25:03.072749+0000","last_peered":"2024-04-06T12:25:03.072749+0000","last_clean":"2024-04-06T12:25:03.072749+0000","last_became_active":"2024-04-06T12:25:02.084198+0000","last_became_peered":"2024-04-06T12:25:02.084198+0000","last_unstale":"2024-04-06T12:25:03.072749+0000","last_undegraded":"2024-04-06T12:25:03.072749+0000","last_fullsized":"2024-04-06T12:25:03.072749+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:10:27.727009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"66'1","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.079889+0000","last_change":"2024-04-06T12:25:01.655266+0000","last_active":"2024-04-06T12:25:02.079889+0000","last_peered":"2024-04-06T12:25:02.079889+0000","last_clean":"2024-04-06T12:25:02.079889+0000","last_became_active":"2024-04-06T12:25:01.655002+0000","last_became_peered":"2024-04-06T12:25:01.655002+0000","last_unstale":"2024-04-06T12:25:02.079889+0000","last_undegraded":"2024-04-06T12:25:02.079889+0000","last_fullsized":"2024-04-06T12:25:02.079889+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:18:35.814800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073528+0000","last_change":"2024-04-06T12:25:02.085799+0000","last_active":"2024-04-06T12:25:03.073528+0000","last_peered":"2024-04-06T12:25:03.073528+0000","last_clean":"2024-04-06T12:25:03.073528+0000","last_became_active":"2024-04-06T12:25:02.085541+0000","last_became_peered":"2024-04-06T12:25:02.085541+0000","last_unstale":"2024-04-06T12:25:03.073528+0000","last_undegraded":"2024-04-06T12:25:03.073528+0000","last_fullsized":"2024-04-06T12:25:03.073528+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:26:00.272122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"66'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071842+0000","last_change":"2024-04-06T12:25:01.657325+0000","last_active":"2024-04-06T12:25:03.071842+0000","last_peered":"2024-04-06T12:25:03.071842+0000","last_clean":"2024-04-06T12:25:03.071842+0000","last_became_active":"2024-04-06T12:25:01.657029+0000","last_became_peered":"2024-04-06T12:25:01.657029+0000","last_unstale":"2024-04-06T12:25:03.071842+0000","last_undegraded":"2024-04-06T12:25:03.071842+0000","last_fullsized":"2024-04-06T12:25:03.071842+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:10:32.281402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071948+0000","last_change":"2024-04-06T12:25:02.093758+0000","last_active":"2024-04-06T12:25:03.071948+0000","last_peered":"2024-04-06T12:25:03.071948+0000","last_clean":"2024-04-06T12:25:03.071948+0000","last_became_active":"2024-04-06T12:25:02.093517+0000","last_became_peered":"2024-04-06T12:25:02.093517+0000","last_unstale":"2024-04-06T12:25:03.071948+0000","last_undegraded":"2024-04-06T12:25:03.071948+0000","last_fullsized":"2024-04-06T12:25:03.071948+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:20:14.893701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.083893+0000","last_change":"2024-04-06T12:25:01.659950+0000","last_active":"2024-04-06T12:25:02.083893+0000","last_peered":"2024-04-06T12:25:02.083893+0000","last_clean":"2024-04-06T12:25:02.083893+0000","last_became_active":"2024-04-06T12:25:01.659663+0000","last_became_peered":"2024-04-06T12:25:01.659663+0000","last_unstale":"2024-04-06T12:25:02.083893+0000","last_undegraded":"2024-04-06T12:25:02.083893+0000","last_fullsized":"2024-04-06T12:25:02.083893+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:47:20.358209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073616+0000","last_change":"2024-04-06T12:25:02.076306+0000","last_active":"2024-04-06T12:25:03.073616+0000","last_peered":"2024-04-06T12:25:03.073616+0000","last_clean":"2024-04-06T12:25:03.073616+0000","last_became_active":"2024-04-06T12:25:02.076017+0000","last_became_peered":"2024-04-06T12:25:02.076017+0000","last_unstale":"2024-04-06T12:25:03.073616+0000","last_undegraded":"2024-04-06T12:25:03.073616+0000","last_fullsized":"2024-04-06T12:25:03.073616+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:58:02.162124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.068870+0000","last_change":"2024-04-06T12:25:01.667208+0000","last_active":"2024-04-06T12:25:02.068870+0000","last_peered":"2024-04-06T12:25:02.068870+0000","last_clean":"2024-04-06T12:25:02.068870+0000","last_became_active":"2024-04-06T12:25:01.666880+0000","last_became_peered":"2024-04-06T12:25:01.666880+0000","last_unstale":"2024-04-06T12:25:02.068870+0000","last_undegraded":"2024-04-06T12:25:02.068870+0000","last_fullsized":"2024-04-06T12:25:02.068870+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:05:05.517843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071988+0000","last_change":"2024-04-06T12:25:02.084907+0000","last_active":"2024-04-06T12:25:03.071988+0000","last_peered":"2024-04-06T12:25:03.071988+0000","last_clean":"2024-04-06T12:25:03.071988+0000","last_became_active":"2024-04-06T12:25:02.084631+0000","last_became_peered":"2024-04-06T12:25:02.084631+0000","last_unstale":"2024-04-06T12:25:03.071988+0000","last_undegraded":"2024-04-06T12:25:03.071988+0000","last_fullsized":"2024-04-06T12:25:03.071988+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:48:19.997695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073636+0000","last_change":"2024-04-06T12:25:01.656618+0000","last_active":"2024-04-06T12:25:03.073636+0000","last_peered":"2024-04-06T12:25:03.073636+0000","last_clean":"2024-04-06T12:25:03.073636+0000","last_became_active":"2024-04-06T12:25:01.656325+0000","last_became_peered":"2024-04-06T12:25:01.656325+0000","last_unstale":"2024-04-06T12:25:03.073636+0000","last_undegraded":"2024-04-06T12:25:03.073636+0000","last_fullsized":"2024-04-06T12:25:03.073636+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:25:33.521667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.092044+0000","last_change":"2024-04-06T12:25:02.088517+0000","last_active":"2024-04-06T12:25:02.092044+0000","last_peered":"2024-04-06T12:25:02.092044+0000","last_clean":"2024-04-06T12:25:02.092044+0000","last_became_active":"2024-04-06T12:25:02.087772+0000","last_became_peered":"2024-04-06T12:25:02.087772+0000","last_unstale":"2024-04-06T12:25:02.092044+0000","last_undegraded":"2024-04-06T12:25:02.092044+0000","last_fullsized":"2024-04-06T12:25:02.092044+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:28:27.128910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.072830+0000","last_change":"2024-04-06T12:25:01.665289+0000","last_active":"2024-04-06T12:25:03.072830+0000","last_peered":"2024-04-06T12:25:03.072830+0000","last_clean":"2024-04-06T12:25:03.072830+0000","last_became_active":"2024-04-06T12:25:01.664968+0000","last_became_peered":"2024-04-06T12:25:01.664968+0000","last_unstale":"2024-04-06T12:25:03.072830+0000","last_undegraded":"2024-04-06T12:25:03.072830+0000","last_fullsized":"2024-04-06T12:25:03.072830+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:35:03.757313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.094055+0000","last_change":"2024-04-06T12:25:02.090419+0000","last_active":"2024-04-06T12:25:02.094055+0000","last_peered":"2024-04-06T12:25:02.094055+0000","last_clean":"2024-04-06T12:25:02.094055+0000","last_became_active":"2024-04-06T12:25:02.090087+0000","last_became_peered":"2024-04-06T12:25:02.090087+0000","last_unstale":"2024-04-06T12:25:02.094055+0000","last_undegraded":"2024-04-06T12:25:02.094055+0000","last_fullsized":"2024-04-06T12:25:02.094055+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:02:05.954262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"66'3","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073580+0000","last_change":"2024-04-06T12:25:01.666204+0000","last_active":"2024-04-06T12:25:03.073580+0000","last_peered":"2024-04-06T12:25:03.073580+0000","last_clean":"2024-04-06T12:25:03.073580+0000","last_became_active":"2024-04-06T12:25:01.665978+0000","last_became_peered":"2024-04-06T12:25:01.665978+0000","last_unstale":"2024-04-06T12:25:03.073580+0000","last_undegraded":"2024-04-06T12:25:03.073580+0000","last_fullsized":"2024-04-06T12:25:03.073580+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:31:42.445760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.072056+0000","last_change":"2024-04-06T12:25:02.080992+0000","last_active":"2024-04-06T12:25:03.072056+0000","last_peered":"2024-04-06T12:25:03.072056+0000","last_clean":"2024-04-06T12:25:03.072056+0000","last_became_active":"2024-04-06T12:25:02.080635+0000","last_became_peered":"2024-04-06T12:25:02.080635+0000","last_unstale":"2024-04-06T12:25:03.072056+0000","last_undegraded":"2024-04-06T12:25:03.072056+0000","last_fullsized":"2024-04-06T12:25:03.072056+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:54:46.589494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"66'6","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.072062+0000","last_change":"2024-04-06T12:25:01.655646+0000","last_active":"2024-04-06T12:25:03.072062+0000","last_peered":"2024-04-06T12:25:03.072062+0000","last_clean":"2024-04-06T12:25:03.072062+0000","last_became_active":"2024-04-06T12:25:01.655306+0000","last_became_peered":"2024-04-06T12:25:01.655306+0000","last_unstale":"2024-04-06T12:25:03.072062+0000","last_undegraded":"2024-04-06T12:25:03.072062+0000","last_fullsized":"2024-04-06T12:25:03.072062+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:47:00.426367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.092226+0000","last_change":"2024-04-06T12:25:02.090390+0000","last_active":"2024-04-06T12:25:02.092226+0000","last_peered":"2024-04-06T12:25:02.092226+0000","last_clean":"2024-04-06T12:25:02.092226+0000","last_became_active":"2024-04-06T12:25:02.090115+0000","last_became_peered":"2024-04-06T12:25:02.090115+0000","last_unstale":"2024-04-06T12:25:02.092226+0000","last_undegraded":"2024-04-06T12:25:02.092226+0000","last_fullsized":"2024-04-06T12:25:02.092226+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:40.362773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"66'2","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.081034+0000","last_change":"2024-04-06T12:25:01.666676+0000","last_active":"2024-04-06T12:25:02.081034+0000","last_peered":"2024-04-06T12:25:02.081034+0000","last_clean":"2024-04-06T12:25:02.081034+0000","last_became_active":"2024-04-06T12:25:01.664796+0000","last_became_peered":"2024-04-06T12:25:01.664796+0000","last_unstale":"2024-04-06T12:25:02.081034+0000","last_undegraded":"2024-04-06T12:25:02.081034+0000","last_fullsized":"2024-04-06T12:25:02.081034+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:10:26.722291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.085063+0000","last_change":"2024-04-06T12:25:02.083800+0000","last_active":"2024-04-06T12:25:02.085063+0000","last_peered":"2024-04-06T12:25:02.085063+0000","last_clean":"2024-04-06T12:25:02.085063+0000","last_became_active":"2024-04-06T12:25:02.083562+0000","last_became_peered":"2024-04-06T12:25:02.083562+0000","last_unstale":"2024-04-06T12:25:02.085063+0000","last_undegraded":"2024-04-06T12:25:02.085063+0000","last_fullsized":"2024-04-06T12:25:02.085063+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:34:26.095936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.082173+0000","last_change":"2024-04-06T12:25:01.668367+0000","last_active":"2024-04-06T12:25:02.082173+0000","last_peered":"2024-04-06T12:25:02.082173+0000","last_clean":"2024-04-06T12:25:02.082173+0000","last_became_active":"2024-04-06T12:25:01.668089+0000","last_became_peered":"2024-04-06T12:25:01.668089+0000","last_unstale":"2024-04-06T12:25:02.082173+0000","last_undegraded":"2024-04-06T12:25:02.082173+0000","last_fullsized":"2024-04-06T12:25:02.082173+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:52:47.398773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071844+0000","last_change":"2024-04-06T12:25:02.091381+0000","last_active":"2024-04-06T12:25:03.071844+0000","last_peered":"2024-04-06T12:25:03.071844+0000","last_clean":"2024-04-06T12:25:03.071844+0000","last_became_active":"2024-04-06T12:25:02.091145+0000","last_became_peered":"2024-04-06T12:25:02.091145+0000","last_unstale":"2024-04-06T12:25:03.071844+0000","last_undegraded":"2024-04-06T12:25:03.071844+0000","last_fullsized":"2024-04-06T12:25:03.071844+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:23:01.737309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073281+0000","last_change":"2024-04-06T12:25:01.657373+0000","last_active":"2024-04-06T12:25:03.073281+0000","last_peered":"2024-04-06T12:25:03.073281+0000","last_clean":"2024-04-06T12:25:03.073281+0000","last_became_active":"2024-04-06T12:25:01.657078+0000","last_became_peered":"2024-04-06T12:25:01.657078+0000","last_unstale":"2024-04-06T12:25:03.073281+0000","last_undegraded":"2024-04-06T12:25:03.073281+0000","last_fullsized":"2024-04-06T12:25:03.073281+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:47:04.026567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.090717+0000","last_change":"2024-04-06T12:25:02.089758+0000","last_active":"2024-04-06T12:25:02.090717+0000","last_peered":"2024-04-06T12:25:02.090717+0000","last_clean":"2024-04-06T12:25:02.090717+0000","last_became_active":"2024-04-06T12:25:02.089241+0000","last_became_peered":"2024-04-06T12:25:02.089241+0000","last_unstale":"2024-04-06T12:25:02.090717+0000","last_undegraded":"2024-04-06T12:25:02.090717+0000","last_fullsized":"2024-04-06T12:25:02.090717+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:33:27.461189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"66'8","reported_seq":19,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071846+0000","last_change":"2024-04-06T12:25:01.655439+0000","last_active":"2024-04-06T12:25:03.071846+0000","last_peered":"2024-04-06T12:25:03.071846+0000","last_clean":"2024-04-06T12:25:03.071846+0000","last_became_active":"2024-04-06T12:25:01.655176+0000","last_became_peered":"2024-04-06T12:25:01.655176+0000","last_unstale":"2024-04-06T12:25:03.071846+0000","last_undegraded":"2024-04-06T12:25:03.071846+0000","last_fullsized":"2024-04-06T12:25:03.071846+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:17:11.808726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071740+0000","last_change":"2024-04-06T12:25:02.095985+0000","last_active":"2024-04-06T12:25:03.071740+0000","last_peered":"2024-04-06T12:25:03.071740+0000","last_clean":"2024-04-06T12:25:03.071740+0000","last_became_active":"2024-04-06T12:25:02.095599+0000","last_became_peered":"2024-04-06T12:25:02.095599+0000","last_unstale":"2024-04-06T12:25:03.071740+0000","last_undegraded":"2024-04-06T12:25:03.071740+0000","last_fullsized":"2024-04-06T12:25:03.071740+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:15:22.102032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"66'1","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.077013+0000","last_change":"2024-04-06T12:25:01.671174+0000","last_active":"2024-04-06T12:25:02.077013+0000","last_peered":"2024-04-06T12:25:02.077013+0000","last_clean":"2024-04-06T12:25:02.077013+0000","last_became_active":"2024-04-06T12:25:01.670893+0000","last_became_peered":"2024-04-06T12:25:01.670893+0000","last_unstale":"2024-04-06T12:25:02.077013+0000","last_undegraded":"2024-04-06T12:25:02.077013+0000","last_fullsized":"2024-04-06T12:25:02.077013+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:48:18.992977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.047828+0000","last_change":"2024-04-06T12:25:02.086649+0000","last_active":"2024-04-06T12:25:04.047828+0000","last_peered":"2024-04-06T12:25:04.047828+0000","last_clean":"2024-04-06T12:25:04.047828+0000","last_became_active":"2024-04-06T12:25:02.086370+0000","last_became_peered":"2024-04-06T12:25:02.086370+0000","last_unstale":"2024-04-06T12:25:04.047828+0000","last_undegraded":"2024-04-06T12:25:04.047828+0000","last_fullsized":"2024-04-06T12:25:04.047828+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:54:46.589494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"66'2","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.088269+0000","last_change":"2024-04-06T12:25:01.657487+0000","last_active":"2024-04-06T12:25:02.088269+0000","last_peered":"2024-04-06T12:25:02.088269+0000","last_clean":"2024-04-06T12:25:02.088269+0000","last_became_active":"2024-04-06T12:25:01.657195+0000","last_became_peered":"2024-04-06T12:25:01.657195+0000","last_unstale":"2024-04-06T12:25:02.088269+0000","last_undegraded":"2024-04-06T12:25:02.088269+0000","last_fullsized":"2024-04-06T12:25:02.088269+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:35:07.572364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071812+0000","last_change":"2024-04-06T12:25:02.082577+0000","last_active":"2024-04-06T12:25:03.071812+0000","last_peered":"2024-04-06T12:25:03.071812+0000","last_clean":"2024-04-06T12:25:03.071812+0000","last_became_active":"2024-04-06T12:25:02.082262+0000","last_became_peered":"2024-04-06T12:25:02.082262+0000","last_unstale":"2024-04-06T12:25:03.071812+0000","last_undegraded":"2024-04-06T12:25:03.071812+0000","last_fullsized":"2024-04-06T12:25:03.071812+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:20:08.845705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"66'1","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.094811+0000","last_change":"2024-04-06T12:25:01.669469+0000","last_active":"2024-04-06T12:25:02.094811+0000","last_peered":"2024-04-06T12:25:02.094811+0000","last_clean":"2024-04-06T12:25:02.094811+0000","last_became_active":"2024-04-06T12:25:01.669226+0000","last_became_peered":"2024-04-06T12:25:01.669226+0000","last_unstale":"2024-04-06T12:25:02.094811+0000","last_undegraded":"2024-04-06T12:25:02.094811+0000","last_fullsized":"2024-04-06T12:25:02.094811+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:20:34.472848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.047978+0000","last_change":"2024-04-06T12:25:02.089505+0000","last_active":"2024-04-06T12:25:04.047978+0000","last_peered":"2024-04-06T12:25:04.047978+0000","last_clean":"2024-04-06T12:25:04.047978+0000","last_became_active":"2024-04-06T12:25:02.089216+0000","last_became_peered":"2024-04-06T12:25:02.089216+0000","last_unstale":"2024-04-06T12:25:04.047978+0000","last_undegraded":"2024-04-06T12:25:04.047978+0000","last_fullsized":"2024-04-06T12:25:04.047978+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:28:27.128910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.082321+0000","last_change":"2024-04-06T12:25:01.658691+0000","last_active":"2024-04-06T12:25:02.082321+0000","last_peered":"2024-04-06T12:25:02.082321+0000","last_clean":"2024-04-06T12:25:02.082321+0000","last_became_active":"2024-04-06T12:25:01.658426+0000","last_became_peered":"2024-04-06T12:25:01.658426+0000","last_unstale":"2024-04-06T12:25:02.082321+0000","last_undegraded":"2024-04-06T12:25:02.082321+0000","last_fullsized":"2024-04-06T12:25:02.082321+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:50:22.888848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.091058+0000","last_change":"2024-04-06T12:25:02.085021+0000","last_active":"2024-04-06T12:25:02.091058+0000","last_peered":"2024-04-06T12:25:02.091058+0000","last_clean":"2024-04-06T12:25:02.091058+0000","last_became_active":"2024-04-06T12:25:02.084751+0000","last_became_peered":"2024-04-06T12:25:02.084751+0000","last_unstale":"2024-04-06T12:25:02.091058+0000","last_undegraded":"2024-04-06T12:25:02.091058+0000","last_fullsized":"2024-04-06T12:25:02.091058+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:10:27.727009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.048001+0000","last_change":"2024-04-06T12:25:01.664195+0000","last_active":"2024-04-06T12:25:04.048001+0000","last_peered":"2024-04-06T12:25:04.048001+0000","last_clean":"2024-04-06T12:25:04.048001+0000","last_became_active":"2024-04-06T12:25:01.663920+0000","last_became_peered":"2024-04-06T12:25:01.663920+0000","last_unstale":"2024-04-06T12:25:04.048001+0000","last_undegraded":"2024-04-06T12:25:04.048001+0000","last_fullsized":"2024-04-06T12:25:04.048001+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:39:59.484595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.098574+0000","last_change":"2024-04-06T12:25:02.091450+0000","last_active":"2024-04-06T12:25:02.098574+0000","last_peered":"2024-04-06T12:25:02.098574+0000","last_clean":"2024-04-06T12:25:02.098574+0000","last_became_active":"2024-04-06T12:25:02.091137+0000","last_became_peered":"2024-04-06T12:25:02.091137+0000","last_unstale":"2024-04-06T12:25:02.098574+0000","last_undegraded":"2024-04-06T12:25:02.098574+0000","last_fullsized":"2024-04-06T12:25:02.098574+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:35:02.409261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071728+0000","last_change":"2024-04-06T12:25:01.654111+0000","last_active":"2024-04-06T12:25:03.071728+0000","last_peered":"2024-04-06T12:25:03.071728+0000","last_clean":"2024-04-06T12:25:03.071728+0000","last_became_active":"2024-04-06T12:25:01.653816+0000","last_became_peered":"2024-04-06T12:25:01.653816+0000","last_unstale":"2024-04-06T12:25:03.071728+0000","last_undegraded":"2024-04-06T12:25:03.071728+0000","last_fullsized":"2024-04-06T12:25:03.071728+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:05.968823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.090611+0000","last_change":"2024-04-06T12:25:02.088434+0000","last_active":"2024-04-06T12:25:02.090611+0000","last_peered":"2024-04-06T12:25:02.090611+0000","last_clean":"2024-04-06T12:25:02.090611+0000","last_became_active":"2024-04-06T12:25:02.088270+0000","last_became_peered":"2024-04-06T12:25:02.088270+0000","last_unstale":"2024-04-06T12:25:02.090611+0000","last_undegraded":"2024-04-06T12:25:02.090611+0000","last_fullsized":"2024-04-06T12:25:02.090611+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:12:13.258920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.069357+0000","last_change":"2024-04-06T12:25:01.667322+0000","last_active":"2024-04-06T12:25:02.069357+0000","last_peered":"2024-04-06T12:25:02.069357+0000","last_clean":"2024-04-06T12:25:02.069357+0000","last_became_active":"2024-04-06T12:25:01.664317+0000","last_became_peered":"2024-04-06T12:25:01.664317+0000","last_unstale":"2024-04-06T12:25:02.069357+0000","last_undegraded":"2024-04-06T12:25:02.069357+0000","last_fullsized":"2024-04-06T12:25:02.069357+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:20:07.840987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.048133+0000","last_change":"2024-04-06T12:25:02.093006+0000","last_active":"2024-04-06T12:25:04.048133+0000","last_peered":"2024-04-06T12:25:04.048133+0000","last_clean":"2024-04-06T12:25:04.048133+0000","last_became_active":"2024-04-06T12:25:02.092754+0000","last_became_peered":"2024-04-06T12:25:02.092754+0000","last_unstale":"2024-04-06T12:25:04.048133+0000","last_undegraded":"2024-04-06T12:25:04.048133+0000","last_fullsized":"2024-04-06T12:25:04.048133+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:10:27.727009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.066153+0000","last_change":"2024-04-06T12:25:01.648228+0000","last_active":"2024-04-06T12:25:02.066153+0000","last_peered":"2024-04-06T12:25:02.066153+0000","last_clean":"2024-04-06T12:25:02.066153+0000","last_became_active":"2024-04-06T12:25:01.647374+0000","last_became_peered":"2024-04-06T12:25:01.647374+0000","last_unstale":"2024-04-06T12:25:02.066153+0000","last_undegraded":"2024-04-06T12:25:02.066153+0000","last_fullsized":"2024-04-06T12:25:02.066153+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:47:20.358209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.091999+0000","last_change":"2024-04-06T12:25:02.089303+0000","last_active":"2024-04-06T12:25:02.091999+0000","last_peered":"2024-04-06T12:25:02.091999+0000","last_clean":"2024-04-06T12:25:02.091999+0000","last_became_active":"2024-04-06T12:25:02.088994+0000","last_became_peered":"2024-04-06T12:25:02.088994+0000","last_unstale":"2024-04-06T12:25:02.091999+0000","last_undegraded":"2024-04-06T12:25:02.091999+0000","last_fullsized":"2024-04-06T12:25:02.091999+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:05:06.522562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"66'2","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.090423+0000","last_change":"2024-04-06T12:25:01.668356+0000","last_active":"2024-04-06T12:25:02.090423+0000","last_peered":"2024-04-06T12:25:02.090423+0000","last_clean":"2024-04-06T12:25:02.090423+0000","last_became_active":"2024-04-06T12:25:01.668085+0000","last_became_peered":"2024-04-06T12:25:01.668085+0000","last_unstale":"2024-04-06T12:25:02.090423+0000","last_undegraded":"2024-04-06T12:25:02.090423+0000","last_fullsized":"2024-04-06T12:25:02.090423+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:23:28.606429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:04.047969+0000","last_change":"2024-04-06T12:25:02.082171+0000","last_active":"2024-04-06T12:25:04.047969+0000","last_peered":"2024-04-06T12:25:04.047969+0000","last_clean":"2024-04-06T12:25:04.047969+0000","last_became_active":"2024-04-06T12:25:02.081923+0000","last_became_peered":"2024-04-06T12:25:02.081923+0000","last_unstale":"2024-04-06T12:25:04.047969+0000","last_undegraded":"2024-04-06T12:25:04.047969+0000","last_fullsized":"2024-04-06T12:25:04.047969+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:20:08.845705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.073635+0000","last_change":"2024-04-06T12:25:01.654961+0000","last_active":"2024-04-06T12:25:03.073635+0000","last_peered":"2024-04-06T12:25:03.073635+0000","last_clean":"2024-04-06T12:25:03.073635+0000","last_became_active":"2024-04-06T12:25:01.654643+0000","last_became_peered":"2024-04-06T12:25:01.654643+0000","last_unstale":"2024-04-06T12:25:03.073635+0000","last_undegraded":"2024-04-06T12:25:03.073635+0000","last_fullsized":"2024-04-06T12:25:03.073635+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:25:27.977561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071886+0000","last_change":"2024-04-06T12:25:02.093277+0000","last_active":"2024-04-06T12:25:03.071886+0000","last_peered":"2024-04-06T12:25:03.071886+0000","last_clean":"2024-04-06T12:25:03.071886+0000","last_became_active":"2024-04-06T12:25:02.092152+0000","last_became_peered":"2024-04-06T12:25:02.092152+0000","last_unstale":"2024-04-06T12:25:03.071886+0000","last_undegraded":"2024-04-06T12:25:03.071886+0000","last_fullsized":"2024-04-06T12:25:03.071886+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:58:40.462728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.069819+0000","last_change":"2024-04-06T12:25:01.666679+0000","last_active":"2024-04-06T12:25:02.069819+0000","last_peered":"2024-04-06T12:25:02.069819+0000","last_clean":"2024-04-06T12:25:02.069819+0000","last_became_active":"2024-04-06T12:25:01.664967+0000","last_became_peered":"2024-04-06T12:25:01.664967+0000","last_unstale":"2024-04-06T12:25:02.069819+0000","last_undegraded":"2024-04-06T12:25:02.069819+0000","last_fullsized":"2024-04-06T12:25:02.069819+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:39.358055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.089911+0000","last_change":"2024-04-06T12:25:02.085535+0000","last_active":"2024-04-06T12:25:02.089911+0000","last_peered":"2024-04-06T12:25:02.089911+0000","last_clean":"2024-04-06T12:25:02.089911+0000","last_became_active":"2024-04-06T12:25:02.085258+0000","last_became_peered":"2024-04-06T12:25:02.085258+0000","last_unstale":"2024-04-06T12:25:02.089911+0000","last_undegraded":"2024-04-06T12:25:02.089911+0000","last_fullsized":"2024-04-06T12:25:02.089911+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:28:24.392874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071904+0000","last_change":"2024-04-06T12:25:01.647993+0000","last_active":"2024-04-06T12:25:03.071904+0000","last_peered":"2024-04-06T12:25:03.071904+0000","last_clean":"2024-04-06T12:25:03.071904+0000","last_became_active":"2024-04-06T12:25:01.647625+0000","last_became_peered":"2024-04-06T12:25:01.647625+0000","last_unstale":"2024-04-06T12:25:03.071904+0000","last_undegraded":"2024-04-06T12:25:03.071904+0000","last_fullsized":"2024-04-06T12:25:03.071904+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:36:59.654441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.091936+0000","last_change":"2024-04-06T12:25:02.088411+0000","last_active":"2024-04-06T12:25:02.091936+0000","last_peered":"2024-04-06T12:25:02.091936+0000","last_clean":"2024-04-06T12:25:02.091936+0000","last_became_active":"2024-04-06T12:25:02.087731+0000","last_became_peered":"2024-04-06T12:25:02.087731+0000","last_unstale":"2024-04-06T12:25:02.091936+0000","last_undegraded":"2024-04-06T12:25:02.091936+0000","last_fullsized":"2024-04-06T12:25:02.091936+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:20:08.845705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071820+0000","last_change":"2024-04-06T12:25:01.664380+0000","last_active":"2024-04-06T12:25:03.071820+0000","last_peered":"2024-04-06T12:25:03.071820+0000","last_clean":"2024-04-06T12:25:03.071820+0000","last_became_active":"2024-04-06T12:25:01.664071+0000","last_became_peered":"2024-04-06T12:25:01.664071+0000","last_unstale":"2024-04-06T12:25:03.071820+0000","last_undegraded":"2024-04-06T12:25:03.071820+0000","last_fullsized":"2024-04-06T12:25:03.071820+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:57:11.331245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.078188+0000","last_change":"2024-04-06T12:25:02.074385+0000","last_active":"2024-04-06T12:25:02.078188+0000","last_peered":"2024-04-06T12:25:02.078188+0000","last_clean":"2024-04-06T12:25:02.078188+0000","last_became_active":"2024-04-06T12:25:02.074137+0000","last_became_peered":"2024-04-06T12:25:02.074137+0000","last_unstale":"2024-04-06T12:25:02.078188+0000","last_undegraded":"2024-04-06T12:25:02.078188+0000","last_fullsized":"2024-04-06T12:25:02.078188+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:07.223156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.071894+0000","last_change":"2024-04-06T12:25:01.655721+0000","last_active":"2024-04-06T12:25:03.071894+0000","last_peered":"2024-04-06T12:25:03.071894+0000","last_clean":"2024-04-06T12:25:03.071894+0000","last_became_active":"2024-04-06T12:25:01.655377+0000","last_became_peered":"2024-04-06T12:25:01.655377+0000","last_unstale":"2024-04-06T12:25:03.071894+0000","last_undegraded":"2024-04-06T12:25:03.071894+0000","last_fullsized":"2024-04-06T12:25:03.071894+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:17:25.543747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-06T12:25:03.072458+0000","last_change":"2024-04-06T12:25:02.093377+0000","last_active":"2024-04-06T12:25:03.072458+0000","last_peered":"2024-04-06T12:25:03.072458+0000","last_clean":"2024-04-06T12:25:03.072458+0000","last_became_active":"2024-04-06T12:25:02.093105+0000","last_became_peered":"2024-04-06T12:25:02.093105+0000","last_unstale":"2024-04-06T12:25:03.072458+0000","last_undegraded":"2024-04-06T12:25:03.072458+0000","last_fullsized":"2024-04-06T12:25:03.072458+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:34:26.095936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.076903+0000","last_change":"2024-04-06T12:25:01.664323+0000","last_active":"2024-04-06T12:25:02.076903+0000","last_peered":"2024-04-06T12:25:02.076903+0000","last_clean":"2024-04-06T12:25:02.076903+0000","last_became_active":"2024-04-06T12:25:01.664098+0000","last_became_peered":"2024-04-06T12:25:01.664098+0000","last_unstale":"2024-04-06T12:25:02.076903+0000","last_undegraded":"2024-04-06T12:25:02.076903+0000","last_fullsized":"2024-04-06T12:25:02.076903+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:38:03.670811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.098565+0000","last_change":"2024-04-06T12:25:02.091379+0000","last_active":"2024-04-06T12:25:02.098565+0000","last_peered":"2024-04-06T12:25:02.098565+0000","last_clean":"2024-04-06T12:25:02.098565+0000","last_became_active":"2024-04-06T12:25:02.091141+0000","last_became_peered":"2024-04-06T12:25:02.091141+0000","last_unstale":"2024-04-06T12:25:02.098565+0000","last_undegraded":"2024-04-06T12:25:02.098565+0000","last_fullsized":"2024-04-06T12:25:02.098565+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:01.637355+0000","last_clean_scrub_stamp":"2024-04-06T12:25:01.637355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:40:00.489313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.069450+0000","last_change":"2024-04-06T12:25:01.667874+0000","last_active":"2024-04-06T12:25:02.069450+0000","last_peered":"2024-04-06T12:25:02.069450+0000","last_clean":"2024-04-06T12:25:02.069450+0000","last_became_active":"2024-04-06T12:25:01.667525+0000","last_became_peered":"2024-04-06T12:25:01.667525+0000","last_unstale":"2024-04-06T12:25:02.069450+0000","last_undegraded":"2024-04-06T12:25:02.069450+0000","last_fullsized":"2024-04-06T12:25:02.069450+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:25:00.632637+0000","last_clean_scrub_stamp":"2024-04-06T12:25:00.632637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:28:26.124191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"55'139","reported_seq":295,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-06T12:25:02.071533+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:25:02.071533+0000","last_peered":"2024-04-06T12:25:02.071533+0000","last_clean":"2024-04-06T12:25:02.071533+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:25:02.071533+0000","last_undegraded":"2024-04-06T12:25:02.071533+0000","last_fullsized":"2024-04-06T12:25:02.071533+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575008,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54632,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317208,"statfs":{"total":96636764160,"available":96580820992,"internally_reserved":0,"allocated":1404928,"data_stored":983178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56899999999999995,"15min":0.53900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.77500000000000002,"5min":0.78300000000000003,"15min":0.80200000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56499999999999995,"15min":0.55100000000000005},"min":{"1min":0.35399999999999998,"5min":0.32000000000000001,"15min":0.25900000000000001},"max":{"1min":0.71999999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.44600000000000001}]},{"osd":1,"last update":"Sat Apr 6 12:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56100000000000005,"15min":0.54500000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.31900000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.81399999999999995},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58399999999999996,"15min":0.56499999999999995},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79600000000000004},"last":0.51500000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57499999999999996,"15min":0.54300000000000004},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.78500000000000003,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59099999999999997,"15min":0.58199999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.24199999999999999},"max":{"1min":0.751,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.624}]},{"osd":3,"last update":"Sat Apr 6 12:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58699999999999997,"15min":0.56599999999999995},"min":{"1min":0.34799999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.73099999999999998,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59799999999999998,"15min":0.60099999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.34699999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.92000000000000004},"last":0.57799999999999996}]},{"osd":4,"last update":"Sat Apr 6 12:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.44,"15min":0.46400000000000002},"min":{"1min":0.29699999999999999,"5min":0.25,"15min":0.25},"max":{"1min":0.70399999999999996,"5min":0.72999999999999998,"15min":0.83599999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.496,"5min":0.52300000000000002,"15min":0.54200000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.64200000000000002,"5min":0.74199999999999999,"15min":1.0780000000000001},"last":0.65000000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.48599999999999999,"15min":0.52600000000000002},"min":{"1min":0.29299999999999998,"5min":0.26600000000000001,"15min":0.23799999999999999},"max":{"1min":0.76900000000000002,"5min":0.86099999999999999,"15min":0.98199999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.5,"5min":0.55100000000000005,"15min":0.57499999999999996},"min":{"1min":0.316,"5min":0.314,"15min":0.29299999999999998},"max":{"1min":0.76000000000000001,"5min":0.81499999999999995,"15min":0.93600000000000005},"last":0.61099999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.52000000000000002,"15min":0.55400000000000005},"min":{"1min":0.34799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66000000000000003,"5min":0.77400000000000002,"15min":0.80200000000000005},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.56000000000000005,"15min":0.57599999999999996},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.26900000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.88500000000000001},"last":0.68100000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575007,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":794624,"data_stored":385699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55100000000000005,"15min":0.52300000000000002},"min":{"1min":0.379,"5min":0.26200000000000001,"15min":0.23000000000000001},"max":{"1min":0.70499999999999996,"5min":0.80700000000000005,"15min":0.82099999999999995},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.55200000000000005,"15min":0.52300000000000002},"min":{"1min":0.378,"5min":0.30199999999999999,"15min":0.245},"max":{"1min":0.66900000000000004,"5min":0.76200000000000001,"15min":0.76400000000000001},"last":0.37}]},{"osd":1,"last update":"Sat Apr 6 12:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.60699999999999998,"15min":0.56499999999999995},"min":{"1min":0.39100000000000001,"5min":0.32600000000000001,"15min":0.24399999999999999},"max":{"1min":0.753,"5min":0.84599999999999997,"15min":0.84999999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61299999999999999,"15min":0.58199999999999996},"min":{"1min":0.46800000000000003,"5min":0.221,"15min":0.221},"max":{"1min":0.76700000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57399999999999995,"15min":0.54200000000000004},"min":{"1min":0.443,"5min":0.35599999999999998,"15min":0.29899999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58499999999999996,"15min":0.56200000000000006},"min":{"1min":0.41399999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.70099999999999996,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.32500000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59699999999999998,"15min":0.57699999999999996},"min":{"1min":0.42799999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.69199999999999995,"5min":0.79000000000000004,"15min":0.81200000000000006},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63500000000000001,"15min":0.61499999999999999},"min":{"1min":0.45600000000000002,"5min":0.42899999999999999,"15min":0.374},"max":{"1min":0.753,"5min":0.872,"15min":0.872},"last":0.42899999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.495,"15min":0.50600000000000001},"min":{"1min":0.309,"5min":0.27500000000000002,"15min":0.27300000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.54200000000000004},"min":{"1min":0.33700000000000002,"5min":0.309,"15min":0.30599999999999999},"max":{"1min":0.73999999999999999,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.56499999999999995}]},{"osd":5,"last update":"Sat Apr 6 12:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.495,"15min":0.50900000000000001},"min":{"1min":0.33200000000000002,"5min":0.32100000000000001,"15min":0.313},"max":{"1min":0.755,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51100000000000001,"15min":0.53400000000000003},"min":{"1min":0.35999999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.80200000000000005,"5min":0.83699999999999997,"15min":0.92800000000000005},"last":0.55100000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.53900000000000003,"15min":0.55000000000000004},"min":{"1min":0.40200000000000002,"5min":0.35299999999999998,"15min":0.33300000000000002},"max":{"1min":0.78300000000000003,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56899999999999995,"15min":0.57699999999999996},"min":{"1min":0.39100000000000001,"5min":0.379,"15min":0.32700000000000001},"max":{"1min":0.82899999999999996,"5min":0.88900000000000001,"15min":0.96099999999999997},"last":0.502}]}]},{"osd":1,"up_from":13,"seq":55834575006,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":806912,"data_stored":391827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.35999999999999999,"15min":0.35399999999999998},"min":{"1min":0.26500000000000001,"5min":0.22600000000000001,"15min":0.224},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.71699999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.41299999999999998,"15min":0.39400000000000002},"min":{"1min":0.30299999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.56499999999999995,"5min":0.59799999999999998,"15min":0.61699999999999999},"last":0.56499999999999995}]},{"osd":2,"last update":"Sat Apr 6 12:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.40699999999999997,"15min":0.39200000000000002},"min":{"1min":0.245,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.624,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.437,"15min":0.42999999999999999},"min":{"1min":0.30299999999999999,"5min":0.29199999999999998,"15min":0.249},"max":{"1min":0.61699999999999999,"5min":0.64900000000000002,"15min":0.79000000000000004},"last":0.63600000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.45300000000000001,"15min":0.44400000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.29699999999999999},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.84299999999999997},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42999999999999999,"15min":0.41399999999999998},"min":{"1min":0.34399999999999997,"5min":0.26600000000000001,"15min":0.253},"max":{"1min":0.55500000000000005,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.69599999999999995}]},{"osd":4,"last update":"Sat Apr 6 12:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63700000000000001,"15min":0.68100000000000005},"min":{"1min":0.51400000000000001,"5min":0.41399999999999998,"15min":0.34799999999999998},"max":{"1min":0.82899999999999996,"5min":0.84099999999999997,"15min":6.3739999999999997},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.64200000000000002,"15min":0.61899999999999999},"min":{"1min":0.441,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.84299999999999997,"5min":0.94899999999999995,"15min":1.0269999999999999},"last":0.85199999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63500000000000001,"15min":0.60999999999999999},"min":{"1min":0.48699999999999999,"5min":0.442,"15min":0.35999999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.97699999999999998},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64600000000000002,"15min":0.625},"min":{"1min":0.46300000000000002,"5min":0.42499999999999999,"15min":0.35099999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":1.0089999999999999},"last":0.78100000000000003}]},{"osd":6,"last update":"Sat Apr 6 12:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.60899999999999999},"min":{"1min":0.48899999999999999,"5min":0.439,"15min":0.38200000000000001},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.96299999999999997},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67000000000000004,"15min":0.65100000000000002},"min":{"1min":0.54500000000000004,"5min":0.439,"15min":0.41799999999999998},"max":{"1min":0.878,"5min":0.92600000000000005,"15min":1.151},"last":0.77200000000000002}]},{"osd":7,"last update":"Sat Apr 6 12:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.64400000000000002,"15min":0.70599999999999996},"min":{"1min":0.51400000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":6.3639999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.65700000000000003,"15min":0.63800000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.39300000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":1.073},"last":0.95599999999999996}]}]},{"osd":0,"up_from":13,"seq":55834575006,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32480,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339360,"statfs":{"total":96636764160,"available":96603504640,"internally_reserved":0,"allocated":1388544,"data_stored":975471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 12:24:23 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.38600000000000001,"15min":0.373},"min":{"1min":0.29299999999999998,"5min":0.23200000000000001,"15min":0.222},"max":{"1min":0.52400000000000002,"5min":0.627,"15min":0.89100000000000001},"last":0.311},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.41099999999999998,"15min":0.39500000000000002},"min":{"1min":0.26300000000000001,"5min":0.25800000000000001,"15min":0.20899999999999999},"max":{"1min":0.60899999999999999,"5min":0.61099999999999999,"15min":0.67100000000000004},"last":0.29799999999999999}]},{"osd":2,"last update":"Sat Apr 6 12:24:23 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.41799999999999998,"15min":0.41599999999999998},"min":{"1min":0.316,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.59199999999999997,"5min":0.63600000000000001,"15min":1.8089999999999999},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.436,"15min":0.42499999999999999},"min":{"1min":0.34200000000000003,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":0.54800000000000004,"5min":0.64100000000000001,"15min":1.764},"last":0.374}]},{"osd":3,"last update":"Sat Apr 6 12:24:23 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41999999999999998,"15min":0.40200000000000002},"min":{"1min":0.311,"5min":0.30299999999999999,"15min":0.21299999999999999},"max":{"1min":0.55500000000000005,"5min":0.64700000000000002,"15min":0.64800000000000002},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.47399999999999998,"15min":0.45600000000000002},"min":{"1min":0.42099999999999999,"5min":0.313,"15min":0.249},"max":{"1min":0.55600000000000005,"5min":0.64500000000000002,"15min":0.92000000000000004},"last":0.28499999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:24:23 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64200000000000002,"15min":0.61499999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.35699999999999998},"max":{"1min":0.82599999999999996,"5min":0.90000000000000002,"15min":0.93799999999999994},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64400000000000002,"15min":0.622},"min":{"1min":0.47399999999999998,"5min":0.37,"15min":0.311},"max":{"1min":0.81200000000000006,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.69199999999999995}]},{"osd":5,"last update":"Sat Apr 6 12:24:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63800000000000001,"15min":0.60899999999999999},"min":{"1min":0.437,"5min":0.40400000000000003,"15min":0.33700000000000002},"max":{"1min":0.78700000000000003,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.65600000000000003,"15min":0.64800000000000002},"min":{"1min":0.5,"5min":0.315,"15min":0.315},"max":{"1min":0.84099999999999997,"5min":0.94499999999999995,"15min":0.97899999999999998},"last":0.57899999999999996}]},{"osd":6,"last update":"Sat Apr 6 12:24:23 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66700000000000004,"15min":0.64600000000000002},"min":{"1min":0.502,"5min":0.39300000000000002,"15min":0.378},"max":{"1min":0.87,"5min":0.91000000000000003,"15min":0.96699999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.67600000000000005,"15min":0.65900000000000003},"min":{"1min":0.53100000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82899999999999996,"5min":0.92500000000000004,"15min":0.99199999999999999},"last":0.66700000000000004}]},{"osd":7,"last update":"Sat Apr 6 12:24:23 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.67500000000000004,"15min":0.65600000000000003},"min":{"1min":0.56100000000000005,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86399999999999999,"5min":0.89400000000000002,"15min":0.96899999999999997},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.64600000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.36699999999999999},"max":{"1min":0.82699999999999996,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.63200000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575008,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49952,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321888,"statfs":{"total":96636764160,"available":96585613312,"internally_reserved":0,"allocated":798720,"data_stored":386611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39800000000000002,"15min":0.38},"min":{"1min":0.28499999999999998,"5min":0.26800000000000002,"15min":0.21099999999999999},"max":{"1min":0.53300000000000003,"5min":0.56399999999999995,"15min":1.159},"last":0.313},{"interface":"front","average":{"1min":0.436,"5min":0.41899999999999998,"15min":0.40999999999999998},"min":{"1min":0.33000000000000002,"5min":0.25700000000000001,"15min":0.24199999999999999},"max":{"1min":0.58799999999999997,"5min":0.60899999999999999,"15min":1.075},"last":0.33200000000000002}]},{"osd":1,"last update":"Sat Apr 6 12:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.41599999999999998,"15min":0.436},"min":{"1min":0.34599999999999997,"5min":0.27500000000000002,"15min":0.24299999999999999},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":3.6179999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.443,"15min":0.43099999999999999},"min":{"1min":0.33900000000000002,"5min":0.32300000000000001,"15min":0.23599999999999999},"max":{"1min":0.59099999999999997,"5min":0.61399999999999999,"15min":1.125},"last":0.28999999999999998}]},{"osd":3,"last update":"Sat Apr 6 12:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.45900000000000002,"15min":0.47899999999999998},"min":{"1min":0.35299999999999998,"5min":0.29799999999999999,"15min":0.29699999999999999},"max":{"1min":0.56799999999999995,"5min":0.65500000000000003,"15min":3.661},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.439,"15min":0.437},"min":{"1min":0.36399999999999999,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":1.1910000000000001},"last":0.34200000000000003}]},{"osd":4,"last update":"Sat Apr 6 12:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63800000000000001,"15min":0.60199999999999998},"min":{"1min":0.51700000000000002,"5min":0.41899999999999998,"15min":0.36199999999999999},"max":{"1min":0.73199999999999998,"5min":0.88200000000000001,"15min":1.2170000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.65500000000000003,"15min":0.628},"min":{"1min":0.48099999999999998,"5min":0.45900000000000002,"15min":0.30499999999999999},"max":{"1min":0.90200000000000002,"5min":0.92600000000000005,"15min":1.101},"last":0.47499999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65100000000000002,"15min":0.66400000000000003},"min":{"1min":0.498,"5min":0.38400000000000001,"15min":0.32900000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":3.6920000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65100000000000002,"15min":0.626},"min":{"1min":0.46100000000000002,"5min":0.438,"15min":0.39300000000000002},"max":{"1min":0.83799999999999997,"5min":0.91800000000000004,"15min":1.244},"last":0.42399999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.63200000000000001},"min":{"1min":0.50700000000000001,"5min":0.42899999999999999,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":0.93400000000000005,"15min":1.012},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.67400000000000004,"15min":0.68000000000000005},"min":{"1min":0.57599999999999996,"5min":0.44600000000000001,"15min":0.39300000000000002},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":3.7490000000000001},"last":0.58499999999999996}]},{"osd":7,"last update":"Sat Apr 6 12:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.67600000000000005,"15min":0.64900000000000002},"min":{"1min":0.503,"5min":0.47499999999999998,"15min":0.35599999999999998},"max":{"1min":0.86899999999999999,"5min":0.95799999999999996,"15min":1.0469999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.53400000000000003,"5min":0.438,"15min":0.42499999999999999},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":3.7200000000000002},"last":0.60699999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575007,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31924,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339916,"statfs":{"total":96636764160,"available":96604073984,"internally_reserved":0,"allocated":819200,"data_stored":394272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.36399999999999999,"15min":0.35699999999999998},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.191},"max":{"1min":0.48699999999999999,"5min":0.59699999999999998,"15min":0.63900000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.39000000000000001,"15min":0.37},"min":{"1min":0.30199999999999999,"5min":0.25,"15min":0.19600000000000001},"max":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.64400000000000002},"last":0.433}]},{"osd":1,"last update":"Sat Apr 6 12:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.38500000000000001,"15min":0.36799999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.215},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.62},"last":0.373},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.41399999999999998,"15min":0.40300000000000002},"min":{"1min":0.32900000000000001,"5min":0.29199999999999998,"15min":0.247},"max":{"1min":0.53900000000000003,"5min":0.57799999999999996,"15min":0.59099999999999997},"last":0.44900000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.25700000000000001},"max":{"1min":0.64900000000000002,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.47799999999999998,"15min":0.46600000000000003},"min":{"1min":0.372,"5min":0.35699999999999998,"15min":0.29899999999999999},"max":{"1min":0.628,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.51100000000000001}]},{"osd":4,"last update":"Sat Apr 6 12:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.61299999999999999,"15min":0.58799999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.85999999999999999,"5min":0.874,"15min":0.95999999999999996},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.63900000000000001,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.90300000000000002,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.60199999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.66200000000000003,"15min":0.65300000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.33700000000000002},"max":{"1min":0.88900000000000001,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66600000000000004,"15min":0.64400000000000002},"min":{"1min":0.48099999999999998,"5min":0.437,"15min":0.36899999999999999},"max":{"1min":0.86099999999999999,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.74199999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.623,"15min":0.61499999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.77500000000000002,"5min":0.95799999999999996,"15min":0.96999999999999997},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.65900000000000003,"15min":0.64600000000000002},"min":{"1min":0.45500000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84999999999999998,"5min":0.92600000000000005,"15min":0.94699999999999995},"last":0.61899999999999999}]},{"osd":7,"last update":"Sat Apr 6 12:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65000000000000002,"15min":0.63100000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.33300000000000002},"max":{"1min":0.89000000000000001,"5min":0.90700000000000003,"15min":0.93300000000000005},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.66400000000000003,"15min":0.65300000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.373},"max":{"1min":0.81899999999999995,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.629}]}]},{"osd":4,"up_from":13,"seq":55834575007,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":806912,"data_stored":393268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57299999999999995,"15min":0.55800000000000005},"min":{"1min":0.45600000000000002,"5min":0.35299999999999998,"15min":0.23899999999999999},"max":{"1min":0.72099999999999997,"5min":0.80800000000000005,"15min":0.82999999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56499999999999995,"15min":0.54400000000000004},"min":{"1min":0.41399999999999998,"5min":0.38800000000000001,"15min":0.29199999999999998},"max":{"1min":0.68300000000000005,"5min":0.71699999999999997,"15min":0.82199999999999995},"last":0.46400000000000002}]},{"osd":1,"last update":"Sat Apr 6 12:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61499999999999999,"15min":0.66100000000000003},"min":{"1min":0.47699999999999998,"5min":0.36199999999999999,"15min":0.33900000000000002},"max":{"1min":0.70799999999999996,"5min":0.76900000000000002,"15min":4.3479999999999999},"last":0.502},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59899999999999998,"15min":0.65100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70699999999999996,"5min":0.78400000000000003,"15min":4.3319999999999999},"last":0.49399999999999999}]},{"osd":2,"last update":"Sat Apr 6 12:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57599999999999996,"15min":0.56200000000000006},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.311},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.78400000000000003},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59499999999999997,"15min":0.57799999999999996},"min":{"1min":0.45500000000000002,"5min":0.39100000000000001,"15min":0.35199999999999998},"max":{"1min":0.73899999999999999,"5min":0.754,"15min":0.82199999999999995},"last":0.54000000000000004}]},{"osd":3,"last update":"Sat Apr 6 12:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56000000000000005,"15min":0.625},"min":{"1min":0.43099999999999999,"5min":0.39900000000000002,"15min":0.34899999999999998},"max":{"1min":0.71399999999999997,"5min":0.72699999999999998,"15min":4.1950000000000003},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.61099999999999999,"15min":0.60599999999999998},"min":{"1min":0.47199999999999998,"5min":0.441,"15min":0.36099999999999999},"max":{"1min":0.71299999999999997,"5min":0.75900000000000001,"15min":1.2330000000000001},"last":0.52600000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.47599999999999998,"15min":0.5},"min":{"1min":0.316,"5min":0.313,"15min":0.29099999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":1.196},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.49199999999999999,"15min":0.504},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.312},"max":{"1min":0.65800000000000003,"5min":0.745,"15min":0.77000000000000002},"last":0.443}]},{"osd":6,"last update":"Sat Apr 6 12:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.497,"15min":0.52200000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.32000000000000001},"max":{"1min":0.67800000000000005,"5min":0.755,"15min":1.022},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.53700000000000003,"15min":0.55700000000000005},"min":{"1min":0.39200000000000002,"5min":0.378,"15min":0.35699999999999998},"max":{"1min":0.71799999999999997,"5min":0.73699999999999999,"15min":0.872},"last":0.39600000000000002}]},{"osd":7,"last update":"Sat Apr 6 12:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52500000000000002,"15min":0.55200000000000005},"min":{"1min":0.372,"5min":0.36899999999999999,"15min":0.34899999999999998},"max":{"1min":0.65700000000000003,"5min":0.77500000000000002,"15min":0.91500000000000004},"last":0.37},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56299999999999994,"15min":0.59199999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.36799999999999999},"max":{"1min":0.73099999999999998,"5min":0.84299999999999997,"15min":1.0589999999999999},"last":0.58199999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575007,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":786432,"data_stored":385047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:24:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59099999999999997,"15min":0.56899999999999995},"min":{"1min":0.32700000000000001,"5min":0.29099999999999998,"15min":0.24199999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56499999999999995,"15min":0.54700000000000004},"min":{"1min":0.38900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.746,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.73499999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:24:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57799999999999996,"15min":0.55700000000000005},"min":{"1min":0.47399999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.497},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56599999999999995,"15min":0.54900000000000004},"min":{"1min":0.35799999999999998,"5min":0.34499999999999997,"15min":0.30599999999999999},"max":{"1min":0.72099999999999997,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.70599999999999996}]},{"osd":2,"last update":"Sat Apr 6 12:24:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58599999999999997,"15min":0.55700000000000005},"min":{"1min":0.38400000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.57899999999999996,"15min":0.56699999999999995},"min":{"1min":0.47499999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.76100000000000001,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.747}]},{"osd":3,"last update":"Sat Apr 6 12:24:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58599999999999997,"15min":0.57199999999999995},"min":{"1min":0.39500000000000002,"5min":0.32500000000000001,"15min":0.31},"max":{"1min":0.76900000000000002,"5min":0.77300000000000002,"15min":0.79400000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61099999999999999,"15min":0.59799999999999998},"min":{"1min":0.45200000000000001,"5min":0.377,"15min":0.32300000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.58699999999999997}]},{"osd":4,"last update":"Sat Apr 6 12:24:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.47499999999999998,"15min":0.48699999999999999},"min":{"1min":0.32600000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.70299999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.437},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.54900000000000004},"min":{"1min":0.40500000000000003,"5min":0.33300000000000002,"15min":0.29499999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.91800000000000004},"last":0.45100000000000001}]},{"osd":6,"last update":"Sat Apr 6 12:24:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.48999999999999999,"15min":0.505},"min":{"1min":0.36499999999999999,"5min":0.34100000000000003,"15min":0.28000000000000003},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.88700000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.503,"15min":0.51200000000000001},"min":{"1min":0.41599999999999998,"5min":0.33800000000000002,"15min":0.32100000000000001},"max":{"1min":0.72799999999999998,"5min":0.77000000000000002,"15min":0.83299999999999996},"last":0.34699999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:24:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.51700000000000002,"15min":0.52600000000000002},"min":{"1min":0.40000000000000002,"5min":0.314,"15min":0.252},"max":{"1min":0.72399999999999998,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.35199999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.85599999999999998},"last":0.48499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:25:06.582 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-06T12:25:06.582 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-06T12:25:06.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.749+0000 7f0b2c9ec700 1 Processor -- start 2024-04-06T12:25:06.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.749+0000 7f0b2c9ec700 1 -- start start 2024-04-06T12:25:06.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.750+0000 7f0b2c9ec700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0b2815d7b0 0x7f0b280b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:06.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.750+0000 7f0b2c9ec700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b280b3990 con 0x7f0b2806a730 2024-04-06T12:25:06.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.750+0000 7f0b2c9ec700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b280b3060 con 0x7f0b2815e500 2024-04-06T12:25:06.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.750+0000 7f0b2c9ec700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b2815e950 con 0x7f0b2815d7b0 2024-04-06T12:25:06.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b25d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0b2806a730 0x7f0b2815ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37286/0 (socket says 172.21.15.189:37286) 2024-04-06T12:25:06.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b25d9b700 1 -- 172.21.15.189:0/4173015737 learned_addr learned my addr 172.21.15.189:0/4173015737 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:25:06.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b2559a700 1 -- 172.21.15.189:0/4173015737 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993385037 0 0) 0x7f0b280b3990 con 0x7f0b2806a730 2024-04-06T12:25:06.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b2659c700 1 --2- 172.21.15.189:0/4173015737 >> v2:172.21.15.192:3300/0 conn(0x7f0b2815d7b0 0x7f0b280b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:06.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b2559a700 1 -- 172.21.15.189:0/4173015737 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b100035b0 con 0x7f0b2806a730 2024-04-06T12:25:06.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b2559a700 1 -- 172.21.15.189:0/4173015737 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1884515729 0 0) 0x7f0b100035b0 con 0x7f0b2806a730 2024-04-06T12:25:06.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b2559a700 1 -- 172.21.15.189:0/4173015737 >> v2:172.21.15.192:3300/0 conn(0x7f0b2815d7b0 msgr2=0x7f0b280b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:06.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b2559a700 1 --2- 172.21.15.189:0/4173015737 >> v2:172.21.15.192:3300/0 conn(0x7f0b2815d7b0 0x7f0b280b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:06.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b2559a700 1 -- 172.21.15.189:0/4173015737 >> v1:172.21.15.192:6789/0 conn(0x7f0b2815e500 legacy=0x7f0b280717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:06.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b2559a700 1 -- 172.21.15.189:0/4173015737 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b280b3340 con 0x7f0b2806a730 2024-04-06T12:25:06.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b2559a700 1 -- 172.21.15.189:0/4173015737 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f0b18001670 con 0x7f0b2806a730 2024-04-06T12:25:06.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b2559a700 1 -- 172.21.15.189:0/4173015737 >> v1:172.21.15.189:6789/0 conn(0x7f0b2806a730 legacy=0x7f0b2815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b2559a700 1 --2- 172.21.15.189:0/4173015737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b10003a00 0x7f0b10003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b2559a700 1 --2- 172.21.15.189:0/4173015737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b10004850 0x7f0b10004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b2559a700 1 --2- 172.21.15.189:0/4173015737 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b10005210 0x7f0b1000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b2559a700 1 -- 172.21.15.189:0/4173015737 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1000c180 con 0x7f0b10004850 2024-04-06T12:25:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b2559a700 1 -- 172.21.15.189:0/4173015737 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1000c2f0 con 0x7f0b10005210 2024-04-06T12:25:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b2559a700 1 -- 172.21.15.189:0/4173015737 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1000c4b0 con 0x7f0b10003a00 2024-04-06T12:25:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b25d9b700 1 --2- 172.21.15.189:0/4173015737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b10004850 0x7f0b10004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b2659c700 1 --2- 172.21.15.189:0/4173015737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b10003a00 0x7f0b10003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.751+0000 7f0b26d9d700 1 --2- 172.21.15.189:0/4173015737 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b10005210 0x7f0b1000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.752+0000 7f0b25d9b700 1 -- 172.21.15.189:0/4173015737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b10003a00 msgr2=0x7f0b10003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.752+0000 7f0b25d9b700 1 --2- 172.21.15.189:0/4173015737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b10003a00 0x7f0b10003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.752+0000 7f0b25d9b700 1 -- 172.21.15.189:0/4173015737 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b10005210 msgr2=0x7f0b1000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.752+0000 7f0b25d9b700 1 --2- 172.21.15.189:0/4173015737 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b10005210 0x7f0b1000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.752+0000 7f0b25d9b700 1 -- 172.21.15.189:0/4173015737 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b100035b0 con 0x7f0b10004850 2024-04-06T12:25:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.752+0000 7f0b26d9d700 1 -- 172.21.15.189:0/4173015737 reap_dead start 2024-04-06T12:25:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.752+0000 7f0b25d9b700 1 --2- 172.21.15.189:0/4173015737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b10004850 0x7f0b10004cd0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f0b1800c240 tx=0x7f0b10005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9213a440c6bd1f2c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.752+0000 7f0b2559a700 1 -- 172.21.15.189:0/4173015737 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18007c10 con 0x7f0b10004850 2024-04-06T12:25:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.752+0000 7f0b2559a700 1 -- 172.21.15.189:0/4173015737 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1800cde0 con 0x7f0b10004850 2024-04-06T12:25:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.752+0000 7f0b2559a700 1 -- 172.21.15.189:0/4173015737 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18005ba0 con 0x7f0b10004850 2024-04-06T12:25:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.752+0000 7f0b2c9ec700 1 -- 172.21.15.189:0/4173015737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b10004850 msgr2=0x7f0b10004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.752+0000 7f0b2c9ec700 1 --2- 172.21.15.189:0/4173015737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b10004850 0x7f0b10004cd0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f0b1800c240 tx=0x7f0b10005880 comp rx=0 tx=0).stop 2024-04-06T12:25:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.753+0000 7f0b2c9ec700 1 -- 172.21.15.189:0/4173015737 shutdown_connections 2024-04-06T12:25:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.753+0000 7f0b2c9ec700 1 --2- 172.21.15.189:0/4173015737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b10004850 0x7f0b10004cd0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.753+0000 7f0b2c9ec700 1 -- 172.21.15.189:0/4173015737 >> 172.21.15.189:0/4173015737 conn(0x7f0b280d00b0 msgr2=0x7f0b280d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.753+0000 7f0b2c9ec700 1 -- 172.21.15.189:0/4173015737 shutdown_connections 2024-04-06T12:25:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.753+0000 7f0b2c9ec700 1 -- 172.21.15.189:0/4173015737 wait complete. 2024-04-06T12:25:06.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.754+0000 7f0b2c9ec700 1 Processor -- start 2024-04-06T12:25:06.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.754+0000 7f0b2c9ec700 1 -- start start 2024-04-06T12:25:06.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.755+0000 7f0b2c9ec700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b10004850 0x7f0b28155040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.755+0000 7f0b2c9ec700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b280cc580 0x7f0b280b1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.755+0000 7f0b2c9ec700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b2815c030 0x7f0b2815c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.755+0000 7f0b2c9ec700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2815c9f0 con 0x7f0b10004850 2024-04-06T12:25:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.755+0000 7f0b2c9ec700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b28155580 con 0x7f0b2815c030 2024-04-06T12:25:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.755+0000 7f0b2c9ec700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b280ccaf0 con 0x7f0b280cc580 2024-04-06T12:25:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.755+0000 7f0b2659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b10004850 0x7f0b28155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.755+0000 7f0b2659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b10004850 0x7f0b28155040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57922/0 (socket says 172.21.15.189:57922) 2024-04-06T12:25:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.755+0000 7f0b2659c700 1 -- 172.21.15.189:0/2607073233 learned_addr learned my addr 172.21.15.189:0/2607073233 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.755+0000 7f0b26d9d700 1 --2- 172.21.15.189:0/2607073233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b2815c030 0x7f0b2815c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.755+0000 7f0b25d9b700 1 --2- 172.21.15.189:0/2607073233 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b280cc580 0x7f0b280b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.755+0000 7f0b2659c700 1 -- 172.21.15.189:0/2607073233 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b280cc580 msgr2=0x7f0b280b1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.755+0000 7f0b2659c700 1 --2- 172.21.15.189:0/2607073233 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b280cc580 0x7f0b280b1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.755+0000 7f0b2659c700 1 -- 172.21.15.189:0/2607073233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b2815c030 msgr2=0x7f0b2815c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.755+0000 7f0b2659c700 1 --2- 172.21.15.189:0/2607073233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b2815c030 0x7f0b2815c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.755+0000 7f0b2659c700 1 -- 172.21.15.189:0/2607073233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b280ccdd0 con 0x7f0b10004850 2024-04-06T12:25:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.755+0000 7f0b2659c700 1 --2- 172.21.15.189:0/2607073233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b10004850 0x7f0b28155040 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c011aa0 tx=0x7f0b1c011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.755+0000 7f0b16ffd700 1 -- 172.21.15.189:0/2607073233 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c00f940 con 0x7f0b10004850 2024-04-06T12:25:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.756+0000 7f0b16ffd700 1 -- 172.21.15.189:0/2607073233 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1c00faa0 con 0x7f0b10004850 2024-04-06T12:25:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.756+0000 7f0b16ffd700 1 -- 172.21.15.189:0/2607073233 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c00fe50 con 0x7f0b10004850 2024-04-06T12:25:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.756+0000 7f0b2c9ec700 1 -- 172.21.15.189:0/2607073233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b280cd0b0 con 0x7f0b10004850 2024-04-06T12:25:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.756+0000 7f0b2c9ec700 1 -- 172.21.15.189:0/2607073233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b280cd670 con 0x7f0b10004850 2024-04-06T12:25:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.757+0000 7f0b2c9ec700 1 -- 172.21.15.189:0/2607073233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b280afcc0 con 0x7f0b10004850 2024-04-06T12:25:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.757+0000 7f0b16ffd700 1 -- 172.21.15.189:0/2607073233 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0b1c014440 con 0x7f0b10004850 2024-04-06T12:25:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.757+0000 7f0b16ffd700 1 --2- 172.21.15.189:0/2607073233 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b0806adb0 0x7f0b0806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.758+0000 7f0b16ffd700 1 -- 172.21.15.189:0/2607073233 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0b1c089cb0 con 0x7f0b10004850 2024-04-06T12:25:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.758+0000 7f0b25d9b700 1 --2- 172.21.15.189:0/2607073233 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b0806adb0 0x7f0b0806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.758+0000 7f0b25d9b700 1 --2- 172.21.15.189:0/2607073233 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b0806adb0 0x7f0b0806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0b18001930 tx=0x7f0b18007000 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:06.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:06.763+0000 7f0b16ffd700 1 -- 172.21.15.189:0/2607073233 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0b1c054c40 con 0x7f0b10004850 2024-04-06T12:25:07.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.032+0000 7f0b2c9ec700 1 -- 172.21.15.189:0/2607073233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f0b28070170 con 0x7f0b10004850 2024-04-06T12:25:07.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.038+0000 7f0b16ffd700 1 -- 172.21.15.189:0/2607073233 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+0 (secure 0 0 0) 0x7f0b1c058260 con 0x7f0b10004850 2024-04-06T12:25:07.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.041+0000 7f0b2c9ec700 1 -- 172.21.15.189:0/2607073233 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b0806adb0 msgr2=0x7f0b0806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:07.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.041+0000 7f0b2c9ec700 1 --2- 172.21.15.189:0/2607073233 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b0806adb0 0x7f0b0806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0b18001930 tx=0x7f0b18007000 comp rx=0 tx=0).stop 2024-04-06T12:25:07.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.041+0000 7f0b2c9ec700 1 -- 172.21.15.189:0/2607073233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b10004850 msgr2=0x7f0b28155040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:07.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.041+0000 7f0b2c9ec700 1 --2- 172.21.15.189:0/2607073233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b10004850 0x7f0b28155040 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c011aa0 tx=0x7f0b1c011db0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.041+0000 7f0b2c9ec700 1 -- 172.21.15.189:0/2607073233 shutdown_connections 2024-04-06T12:25:07.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.041+0000 7f0b2c9ec700 1 --2- 172.21.15.189:0/2607073233 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b0806adb0 0x7f0b0806d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.041+0000 7f0b2c9ec700 1 --2- 172.21.15.189:0/2607073233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b2815c030 0x7f0b2815c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.041+0000 7f0b2c9ec700 1 --2- 172.21.15.189:0/2607073233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b10004850 0x7f0b28155040 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.041+0000 7f0b2c9ec700 1 --2- 172.21.15.189:0/2607073233 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b280cc580 0x7f0b280b1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.041+0000 7f0b2c9ec700 1 -- 172.21.15.189:0/2607073233 >> 172.21.15.189:0/2607073233 conn(0x7f0b280d00b0 msgr2=0x7f0b280ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:07.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.041+0000 7f0b2c9ec700 1 -- 172.21.15.189:0/2607073233 shutdown_connections 2024-04-06T12:25:07.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.041+0000 7f0b2c9ec700 1 -- 172.21.15.189:0/2607073233 wait complete. 2024-04-06T12:25:07.040 INFO:teuthology.orchestra.run.smithi189.stderr:updated caps for client.0 2024-04-06T12:25:07.051 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:25:07.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.175+0000 7f2bd5663700 1 Processor -- start 2024-04-06T12:25:07.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.175+0000 7f2bd5663700 1 -- start start 2024-04-06T12:25:07.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.175+0000 7f2bd5663700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2bd00ce730 0x7f2bd015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.175+0000 7f2bd5663700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bd00b8d70 0x7f2bd00d1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.175+0000 7f2bd5663700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bd015eac0 con 0x7f2bd015d6f0 2024-04-06T12:25:07.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.175+0000 7f2bd5663700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd015e010 con 0x7f2bd00ce730 2024-04-06T12:25:07.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.175+0000 7f2bd5663700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd015e220 con 0x7f2bd00b8d70 2024-04-06T12:25:07.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bceffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2bd015d6f0 0x7f2bd00ce020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37296/0 (socket says 172.21.15.189:37296) 2024-04-06T12:25:07.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bceffd700 1 -- 172.21.15.189:0/275519856 learned_addr learned my addr 172.21.15.189:0/275519856 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:25:07.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bcf7fe700 1 --2- 172.21.15.189:0/275519856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bd00b8d70 0x7f2bd00d1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bcdffb700 1 -- 172.21.15.189:0/275519856 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007832433 0 0) 0x7f2bd015eac0 con 0x7f2bd015d6f0 2024-04-06T12:25:07.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bce7fc700 1 --2- 172.21.15.189:0/275519856 >> v2:172.21.15.192:3300/0 conn(0x7f2bd00ce730 0x7f2bd015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bcdffb700 1 -- 172.21.15.189:0/275519856 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bb80035b0 con 0x7f2bd015d6f0 2024-04-06T12:25:07.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bcdffb700 1 -- 172.21.15.189:0/275519856 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3097220823 0 0) 0x7f2bb80035b0 con 0x7f2bd015d6f0 2024-04-06T12:25:07.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bcdffb700 1 -- 172.21.15.189:0/275519856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bd00b8d70 msgr2=0x7f2bd00d1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:07.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bcdffb700 1 --2- 172.21.15.189:0/275519856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bd00b8d70 0x7f2bd00d1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bcdffb700 1 -- 172.21.15.189:0/275519856 >> v2:172.21.15.192:3300/0 conn(0x7f2bd00ce730 msgr2=0x7f2bd015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:07.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bcdffb700 1 --2- 172.21.15.189:0/275519856 >> v2:172.21.15.192:3300/0 conn(0x7f2bd00ce730 0x7f2bd015dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bcdffb700 1 -- 172.21.15.189:0/275519856 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd015e500 con 0x7f2bd015d6f0 2024-04-06T12:25:07.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bcdffb700 1 -- 172.21.15.189:0/275519856 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f2bc4001670 con 0x7f2bd015d6f0 2024-04-06T12:25:07.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bcdffb700 1 -- 172.21.15.189:0/275519856 >> v1:172.21.15.189:6789/0 conn(0x7f2bd015d6f0 legacy=0x7f2bd00ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:07.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bcdffb700 1 --2- 172.21.15.189:0/275519856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bb8003a00 0x7f2bb8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bcdffb700 1 --2- 172.21.15.189:0/275519856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bd00b8d70 0x7f2bb8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bcdffb700 1 --2- 172.21.15.189:0/275519856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bb8004d90 0x7f2bb8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bcdffb700 1 -- 172.21.15.189:0/275519856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd015e010 con 0x7f2bb8003a00 2024-04-06T12:25:07.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bcdffb700 1 -- 172.21.15.189:0/275519856 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd015e220 con 0x7f2bb8004d90 2024-04-06T12:25:07.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bcdffb700 1 -- 172.21.15.189:0/275519856 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb800ba10 con 0x7f2bd00b8d70 2024-04-06T12:25:07.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bceffd700 1 --2- 172.21.15.189:0/275519856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bb8003a00 0x7f2bb8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bcf7fe700 1 --2- 172.21.15.189:0/275519856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bb8004d90 0x7f2bb8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.176+0000 7f2bce7fc700 1 --2- 172.21.15.189:0/275519856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bd00b8d70 0x7f2bb8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.177+0000 7f2bceffd700 1 -- 172.21.15.189:0/275519856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bd00b8d70 msgr2=0x7f2bb8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:07.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.177+0000 7f2bceffd700 1 --2- 172.21.15.189:0/275519856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bd00b8d70 0x7f2bb8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.177+0000 7f2bceffd700 1 -- 172.21.15.189:0/275519856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bb8004d90 msgr2=0x7f2bb8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:07.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.177+0000 7f2bceffd700 1 --2- 172.21.15.189:0/275519856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bb8004d90 0x7f2bb8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.177+0000 7f2bceffd700 1 -- 172.21.15.189:0/275519856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb80035b0 con 0x7f2bb8003a00 2024-04-06T12:25:07.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.177+0000 7f2bceffd700 1 --2- 172.21.15.189:0/275519856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bb8003a00 0x7f2bb8003e20 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400c320 tx=0x7f2bb8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bdd973ef2745a0a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:07.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.177+0000 7f2bcdffb700 1 -- 172.21.15.189:0/275519856 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4007c40 con 0x7f2bb8003a00 2024-04-06T12:25:07.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.177+0000 7f2bcdffb700 1 -- 172.21.15.189:0/275519856 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc400ce20 con 0x7f2bb8003a00 2024-04-06T12:25:07.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.177+0000 7f2bcdffb700 1 -- 172.21.15.189:0/275519856 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4005ba0 con 0x7f2bb8003a00 2024-04-06T12:25:07.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.177+0000 7f2bd5663700 1 -- 172.21.15.189:0/275519856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bb8003a00 msgr2=0x7f2bb8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:07.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.177+0000 7f2bd5663700 1 --2- 172.21.15.189:0/275519856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bb8003a00 0x7f2bb8003e20 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400c320 tx=0x7f2bb8005880 comp rx=0 tx=0).stop 2024-04-06T12:25:07.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.177+0000 7f2bcf7fe700 1 -- 172.21.15.189:0/275519856 reap_dead start 2024-04-06T12:25:07.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.177+0000 7f2bd5663700 1 -- 172.21.15.189:0/275519856 shutdown_connections 2024-04-06T12:25:07.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.177+0000 7f2bd5663700 1 -- 172.21.15.189:0/275519856 >> 172.21.15.189:0/275519856 conn(0x7f2bd0072490 msgr2=0x7f2bd006b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:07.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.177+0000 7f2bd5663700 1 -- 172.21.15.189:0/275519856 shutdown_connections 2024-04-06T12:25:07.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.177+0000 7f2bd5663700 1 -- 172.21.15.189:0/275519856 wait complete. 2024-04-06T12:25:07.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.179+0000 7f2bd5663700 1 Processor -- start 2024-04-06T12:25:07.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.179+0000 7f2bd5663700 1 -- start start 2024-04-06T12:25:07.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.179+0000 7f2bd5663700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bb8003a00 0x7f2bd0157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bd5663700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bd00b3ea0 0x7f2bd01f5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bd5663700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bd00b54b0 0x7f2bd01537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bd5663700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd0159bf0 con 0x7f2bd00b3ea0 2024-04-06T12:25:07.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bd5663700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd00b4410 con 0x7f2bd00b54b0 2024-04-06T12:25:07.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bd5663700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd00b46f0 con 0x7f2bb8003a00 2024-04-06T12:25:07.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bce7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bd00b3ea0 0x7f2bd01f5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bce7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bd00b3ea0 0x7f2bd01f5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57944/0 (socket says 172.21.15.189:57944) 2024-04-06T12:25:07.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bce7fc700 1 -- 172.21.15.189:0/723090507 learned_addr learned my addr 172.21.15.189:0/723090507 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:07.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bceffd700 1 --2- 172.21.15.189:0/723090507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bb8003a00 0x7f2bd0157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bcf7fe700 1 --2- 172.21.15.189:0/723090507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bd00b54b0 0x7f2bd01537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bce7fc700 1 -- 172.21.15.189:0/723090507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bb8003a00 msgr2=0x7f2bd0157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:07.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bce7fc700 1 --2- 172.21.15.189:0/723090507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bb8003a00 0x7f2bd0157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bce7fc700 1 -- 172.21.15.189:0/723090507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bd00b54b0 msgr2=0x7f2bd01537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:07.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bce7fc700 1 --2- 172.21.15.189:0/723090507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bd00b54b0 0x7f2bd01537e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bce7fc700 1 -- 172.21.15.189:0/723090507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd00b49d0 con 0x7f2bd00b3ea0 2024-04-06T12:25:07.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bce7fc700 1 --2- 172.21.15.189:0/723090507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bd00b3ea0 0x7f2bd01f5810 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0006dd0 tx=0x7f2bc0011d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:07.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bb77fe700 1 -- 172.21.15.189:0/723090507 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0011e80 con 0x7f2bd00b3ea0 2024-04-06T12:25:07.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bb77fe700 1 -- 172.21.15.189:0/723090507 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc0004030 con 0x7f2bd00b3ea0 2024-04-06T12:25:07.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bd5663700 1 -- 172.21.15.189:0/723090507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd00b4cb0 con 0x7f2bd00b3ea0 2024-04-06T12:25:07.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bb77fe700 1 -- 172.21.15.189:0/723090507 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc00043e0 con 0x7f2bd00b3ea0 2024-04-06T12:25:07.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.180+0000 7f2bd5663700 1 -- 172.21.15.189:0/723090507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd01f5d50 con 0x7f2bd00b3ea0 2024-04-06T12:25:07.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.181+0000 7f2bd5663700 1 -- 172.21.15.189:0/723090507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd0074990 con 0x7f2bd00b3ea0 2024-04-06T12:25:07.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.182+0000 7f2bb77fe700 1 -- 172.21.15.189:0/723090507 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f2bc000f940 con 0x7f2bd00b3ea0 2024-04-06T12:25:07.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.182+0000 7f2bb77fe700 1 --2- 172.21.15.189:0/723090507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2bbc06adb0 0x7f2bbc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.182+0000 7f2bb77fe700 1 -- 172.21.15.189:0/723090507 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2bc0089800 con 0x7f2bd00b3ea0 2024-04-06T12:25:07.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.182+0000 7f2bceffd700 1 --2- 172.21.15.189:0/723090507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2bbc06adb0 0x7f2bbc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.183+0000 7f2bceffd700 1 --2- 172.21.15.189:0/723090507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2bbc06adb0 0x7f2bbc06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4008fd0 tx=0x7f2bc400a000 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:07.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.188+0000 7f2bb77fe700 1 -- 172.21.15.189:0/723090507 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2bc0054790 con 0x7f2bd00b3ea0 2024-04-06T12:25:07.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.429+0000 7f2bd5663700 1 -- 172.21.15.189:0/723090507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bd015f1f0 con 0x7f2bd00b3ea0 2024-04-06T12:25:07.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.429+0000 7f2bb77fe700 1 -- 172.21.15.189:0/723090507 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f2bc0022020 con 0x7f2bd00b3ea0 2024-04-06T12:25:07.428 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:25:07.428 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/527093473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":527093473},{"type":"v1","addr":"172.21.15.192:6835","nonce":527093473}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6105,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2142144473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2142144473},{"type":"v1","addr":"172.21.15.189:6836","nonce":2142144473}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6108,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2285762709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2285762709},{"type":"v1","addr":"172.21.15.189:6837","nonce":2285762709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:25:02.062461+0000","modified":"2024-04-06T12:25:05.077648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6834/525827513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":525827513},{"type":"v1","addr":"172.21.15.192:6834","nonce":525827513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-06T12:25:07.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.431+0000 7f2bd5663700 1 -- 172.21.15.189:0/723090507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2bbc06adb0 msgr2=0x7f2bbc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:07.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.431+0000 7f2bd5663700 1 --2- 172.21.15.189:0/723090507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2bbc06adb0 0x7f2bbc06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4008fd0 tx=0x7f2bc400a000 comp rx=0 tx=0).stop 2024-04-06T12:25:07.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.431+0000 7f2bd5663700 1 -- 172.21.15.189:0/723090507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bd00b3ea0 msgr2=0x7f2bd01f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:07.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.431+0000 7f2bd5663700 1 --2- 172.21.15.189:0/723090507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bd00b3ea0 0x7f2bd01f5810 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0006dd0 tx=0x7f2bc0011d50 comp rx=0 tx=0).stop 2024-04-06T12:25:07.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.431+0000 7f2bd5663700 1 -- 172.21.15.189:0/723090507 shutdown_connections 2024-04-06T12:25:07.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.431+0000 7f2bd5663700 1 --2- 172.21.15.189:0/723090507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2bbc06adb0 0x7f2bbc06d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.431+0000 7f2bd5663700 1 --2- 172.21.15.189:0/723090507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bd00b54b0 0x7f2bd01537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.431+0000 7f2bd5663700 1 --2- 172.21.15.189:0/723090507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bb8003a00 0x7f2bd0157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.431+0000 7f2bd5663700 1 --2- 172.21.15.189:0/723090507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bd00b3ea0 0x7f2bd01f5810 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.431+0000 7f2bd5663700 1 -- 172.21.15.189:0/723090507 >> 172.21.15.189:0/723090507 conn(0x7f2bd0072490 msgr2=0x7f2bd00d21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:07.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.431+0000 7f2bd5663700 1 -- 172.21.15.189:0/723090507 shutdown_connections 2024-04-06T12:25:07.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.431+0000 7f2bd5663700 1 -- 172.21.15.189:0/723090507 wait complete. 2024-04-06T12:25:07.431 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 24 2024-04-06T12:25:07.633 INFO:tasks.ceph.mds.b.smithi192.stderr:2024-04-06T12:25:07.635+0000 7fefd514a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T12:25:07.845 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-06T12:25:02.062461+0000', 'modified': '2024-04-06T12:25:05.077648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6114}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6114': {'gid': 6114, 'name': 'b', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.192:6834/525827513', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6832', 'nonce': 525827513}, {'type': 'v1', 'addr': '172.21.15.192:6834', 'nonce': 525827513}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-06T12:25:07.845 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-06T12:25:07.845 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-06T12:25:07.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.973+0000 7fa843c8a700 1 Processor -- start 2024-04-06T12:25:07.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.973+0000 7fa843c8a700 1 -- start start 2024-04-06T12:25:07.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.974+0000 7fa843c8a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa83c06ae40 0x7fa83c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.974+0000 7fa843c8a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa83c0b3990 con 0x7fa83c15ddf0 2024-04-06T12:25:07.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.974+0000 7fa843c8a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa83c0b3060 con 0x7fa83c15d7b0 2024-04-06T12:25:07.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.974+0000 7fa843c8a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa83c15e950 con 0x7fa83c06ae40 2024-04-06T12:25:07.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.974+0000 7fa842227700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa83c15ddf0 0x7fa83c15e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37308/0 (socket says 172.21.15.189:37308) 2024-04-06T12:25:07.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.974+0000 7fa842227700 1 -- 172.21.15.189:0/3815308865 learned_addr learned my addr 172.21.15.189:0/3815308865 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:25:07.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.974+0000 7fa841225700 1 --2- 172.21.15.189:0/3815308865 >> v2:172.21.15.192:3300/0 conn(0x7fa83c06ae40 0x7fa83c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.974+0000 7fa840a24700 1 -- 172.21.15.189:0/3815308865 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771304462 0 0) 0x7fa83c0b3990 con 0x7fa83c15ddf0 2024-04-06T12:25:07.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.974+0000 7fa840a24700 1 -- 172.21.15.189:0/3815308865 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8280035b0 con 0x7fa83c15ddf0 2024-04-06T12:25:07.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa840a24700 1 -- 172.21.15.189:0/3815308865 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553749041 0 0) 0x7fa83c0b3060 con 0x7fa83c15d7b0 2024-04-06T12:25:07.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa840a24700 1 -- 172.21.15.189:0/3815308865 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa83c0b3990 con 0x7fa83c15d7b0 2024-04-06T12:25:07.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa840a24700 1 -- 172.21.15.189:0/3815308865 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1216548811 0 0) 0x7fa8280035b0 con 0x7fa83c15ddf0 2024-04-06T12:25:07.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa840a24700 1 -- 172.21.15.189:0/3815308865 >> v2:172.21.15.192:3300/0 conn(0x7fa83c06ae40 msgr2=0x7fa83c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:07.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa840a24700 1 --2- 172.21.15.189:0/3815308865 >> v2:172.21.15.192:3300/0 conn(0x7fa83c06ae40 0x7fa83c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa840a24700 1 -- 172.21.15.189:0/3815308865 >> v1:172.21.15.192:6789/0 conn(0x7fa83c15d7b0 legacy=0x7fa83c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:07.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa840a24700 1 -- 172.21.15.189:0/3815308865 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83c0b3340 con 0x7fa83c15ddf0 2024-04-06T12:25:07.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa840a24700 1 -- 172.21.15.189:0/3815308865 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fa8340019f0 con 0x7fa83c15ddf0 2024-04-06T12:25:07.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa840a24700 1 -- 172.21.15.189:0/3815308865 >> v1:172.21.15.189:6789/0 conn(0x7fa83c15ddf0 legacy=0x7fa83c15e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:07.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa840a24700 1 --2- 172.21.15.189:0/3815308865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa828003be0 0x7fa828003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa840a24700 1 --2- 172.21.15.189:0/3815308865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa828004850 0x7fa828004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa840a24700 1 --2- 172.21.15.189:0/3815308865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa828005260 0x7fa82800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa840a24700 1 -- 172.21.15.189:0/3815308865 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82800c160 con 0x7fa828003be0 2024-04-06T12:25:07.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa840a24700 1 -- 172.21.15.189:0/3815308865 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82800c2d0 con 0x7fa828004850 2024-04-06T12:25:07.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa840a24700 1 -- 172.21.15.189:0/3815308865 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82800c490 con 0x7fa828005260 2024-04-06T12:25:07.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa841a26700 1 --2- 172.21.15.189:0/3815308865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa828003be0 0x7fa828003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa841225700 1 --2- 172.21.15.189:0/3815308865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa828004850 0x7fa828004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa842227700 1 --2- 172.21.15.189:0/3815308865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa828005260 0x7fa82800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa841a26700 1 -- 172.21.15.189:0/3815308865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa828005260 msgr2=0x7fa82800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:07.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa841a26700 1 --2- 172.21.15.189:0/3815308865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa828005260 0x7fa82800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa841a26700 1 -- 172.21.15.189:0/3815308865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa828004850 msgr2=0x7fa828004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:07.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa841a26700 1 --2- 172.21.15.189:0/3815308865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa828004850 0x7fa828004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa841a26700 1 -- 172.21.15.189:0/3815308865 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8280035b0 con 0x7fa828003be0 2024-04-06T12:25:07.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa842227700 1 -- 172.21.15.189:0/3815308865 reap_dead start 2024-04-06T12:25:07.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa841a26700 1 --2- 172.21.15.189:0/3815308865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa828003be0 0x7fa828003fe0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa838001940 tx=0x7fa828005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c14670daea262ea server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:07.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa840a24700 1 -- 172.21.15.189:0/3815308865 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838001430 con 0x7fa828003be0 2024-04-06T12:25:07.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa840a24700 1 -- 172.21.15.189:0/3815308865 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa838001d70 con 0x7fa828003be0 2024-04-06T12:25:07.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.975+0000 7fa840a24700 1 -- 172.21.15.189:0/3815308865 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838005530 con 0x7fa828003be0 2024-04-06T12:25:07.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.976+0000 7fa843c8a700 1 -- 172.21.15.189:0/3815308865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa828003be0 msgr2=0x7fa828003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:07.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.976+0000 7fa843c8a700 1 --2- 172.21.15.189:0/3815308865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa828003be0 0x7fa828003fe0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa838001940 tx=0x7fa828005880 comp rx=0 tx=0).stop 2024-04-06T12:25:07.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.976+0000 7fa843c8a700 1 -- 172.21.15.189:0/3815308865 shutdown_connections 2024-04-06T12:25:07.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.976+0000 7fa843c8a700 1 --2- 172.21.15.189:0/3815308865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa828003be0 0x7fa828003fe0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.976+0000 7fa843c8a700 1 -- 172.21.15.189:0/3815308865 >> 172.21.15.189:0/3815308865 conn(0x7fa83c0d00b0 msgr2=0x7fa83c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:07.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.976+0000 7fa843c8a700 1 -- 172.21.15.189:0/3815308865 shutdown_connections 2024-04-06T12:25:07.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.976+0000 7fa843c8a700 1 -- 172.21.15.189:0/3815308865 wait complete. 2024-04-06T12:25:07.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.977+0000 7fa843c8a700 1 Processor -- start 2024-04-06T12:25:07.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.977+0000 7fa843c8a700 1 -- start start 2024-04-06T12:25:07.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.978+0000 7fa843c8a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa828003be0 0x7fa83c0cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.978+0000 7fa843c8a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa83c154ad0 0x7fa83c0b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.978+0000 7fa843c8a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa83c159070 0x7fa83c0aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.978+0000 7fa843c8a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c158c50 con 0x7fa83c154ad0 2024-04-06T12:25:07.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.978+0000 7fa843c8a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c155420 con 0x7fa828003be0 2024-04-06T12:25:07.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.978+0000 7fa843c8a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c154f50 con 0x7fa83c159070 2024-04-06T12:25:07.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.978+0000 7fa841225700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa83c154ad0 0x7fa83c0b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.978+0000 7fa841225700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa83c154ad0 0x7fa83c0b14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57968/0 (socket says 172.21.15.189:57968) 2024-04-06T12:25:07.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.978+0000 7fa841225700 1 -- 172.21.15.189:0/310585546 learned_addr learned my addr 172.21.15.189:0/310585546 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:07.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.978+0000 7fa842227700 1 --2- 172.21.15.189:0/310585546 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa83c159070 0x7fa83c0aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.978+0000 7fa841a26700 1 --2- 172.21.15.189:0/310585546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa828003be0 0x7fa83c0cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.978+0000 7fa841225700 1 -- 172.21.15.189:0/310585546 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa83c159070 msgr2=0x7fa83c0aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:07.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.978+0000 7fa841225700 1 --2- 172.21.15.189:0/310585546 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa83c159070 0x7fa83c0aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.978+0000 7fa841225700 1 -- 172.21.15.189:0/310585546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa828003be0 msgr2=0x7fa83c0cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:07.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.978+0000 7fa841225700 1 --2- 172.21.15.189:0/310585546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa828003be0 0x7fa83c0cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.978+0000 7fa841225700 1 -- 172.21.15.189:0/310585546 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83c1594f0 con 0x7fa83c154ad0 2024-04-06T12:25:07.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.979+0000 7fa841225700 1 --2- 172.21.15.189:0/310585546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa83c154ad0 0x7fa83c0b14c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fa82c011a80 tx=0x7fa82c011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:07.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.979+0000 7fa8327fc700 1 -- 172.21.15.189:0/310585546 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c00fa00 con 0x7fa83c154ad0 2024-04-06T12:25:07.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.979+0000 7fa8327fc700 1 -- 172.21.15.189:0/310585546 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82c00fb60 con 0x7fa83c154ad0 2024-04-06T12:25:07.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.979+0000 7fa843c8a700 1 -- 172.21.15.189:0/310585546 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa83c1582f0 con 0x7fa83c154ad0 2024-04-06T12:25:07.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.979+0000 7fa8327fc700 1 -- 172.21.15.189:0/310585546 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c014440 con 0x7fa83c154ad0 2024-04-06T12:25:07.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.979+0000 7fa843c8a700 1 -- 172.21.15.189:0/310585546 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa83c158870 con 0x7fa83c154ad0 2024-04-06T12:25:07.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.980+0000 7fa8327fc700 1 -- 172.21.15.189:0/310585546 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fa82c0145a0 con 0x7fa83c154ad0 2024-04-06T12:25:07.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.981+0000 7fa8327fc700 1 --2- 172.21.15.189:0/310585546 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa82006adb0 0x7fa82006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.981+0000 7fa8327fc700 1 -- 172.21.15.189:0/310585546 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa82c089cc0 con 0x7fa83c154ad0 2024-04-06T12:25:07.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.981+0000 7fa841a26700 1 --2- 172.21.15.189:0/310585546 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa82006adb0 0x7fa82006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.981+0000 7fa841a26700 1 --2- 172.21.15.189:0/310585546 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa82006adb0 0x7fa82006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa838015040 tx=0x7fa83800a000 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:07.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.983+0000 7fa84fe3eb80 1 Processor -- start 2024-04-06T12:25:07.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.983+0000 7fa84fe3eb80 1 -- start start 2024-04-06T12:25:07.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.984+0000 7fa84fe3eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377357c20 0x56437735ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.984+0000 7fa84fe3eb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 0x5643773335f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.984+0000 7fa84fe3eb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377332980 0x564377333b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.984+0000 7fa84fe3eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56437728c940 con 0x564377357c20 2024-04-06T12:25:07.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.984+0000 7fa84fe3eb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x56437728a300 con 0x564377332980 2024-04-06T12:25:07.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.984+0000 7fa84fe3eb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x564377285bf0 con 0x56437735cfc0 2024-04-06T12:25:07.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.984+0000 7fa841a26700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377357c20 0x56437735ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.984+0000 7fa841a26700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377357c20 0x56437735ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57980/0 (socket says 172.21.15.189:57980) 2024-04-06T12:25:07.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.984+0000 7fa841a26700 1 -- 172.21.15.189:0/275653477 learned_addr learned my addr 172.21.15.189:0/275653477 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:07.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.984+0000 7fa841225700 1 --2- 172.21.15.189:0/275653477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 0x5643773335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.984+0000 7fa842227700 1 --2- 172.21.15.189:0/275653477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377332980 0x564377333b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.984+0000 7fa841a26700 1 -- 172.21.15.189:0/275653477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 msgr2=0x5643773335f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:07.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.984+0000 7fa841a26700 1 --2- 172.21.15.189:0/275653477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 0x5643773335f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.984+0000 7fa841a26700 1 -- 172.21.15.189:0/275653477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377332980 msgr2=0x564377333b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:07.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.984+0000 7fa841a26700 1 --2- 172.21.15.189:0/275653477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377332980 0x564377333b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.985+0000 7fa841a26700 1 -- 172.21.15.189:0/275653477 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5643770b3b10 con 0x564377357c20 2024-04-06T12:25:07.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.985+0000 7fa842227700 1 --2- 172.21.15.189:0/275653477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377332980 0x564377333b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:25:07.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.985+0000 7fa841a26700 1 --2- 172.21.15.189:0/275653477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377357c20 0x56437735ba60 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fa83800c3c0 tx=0x7fa838009030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99b0af26d6b5ff77 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:07.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.985+0000 7fa81f7fe700 1 -- 172.21.15.189:0/275653477 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838005530 con 0x564377357c20 2024-04-06T12:25:07.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.985+0000 7fa81f7fe700 1 -- 172.21.15.189:0/275653477 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa838007040 con 0x564377357c20 2024-04-06T12:25:07.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.985+0000 7fa81f7fe700 1 -- 172.21.15.189:0/275653477 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8380073b0 con 0x564377357c20 2024-04-06T12:25:07.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.985+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/275653477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377357c20 msgr2=0x56437735ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:07.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.985+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/275653477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377357c20 0x56437735ba60 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fa83800c3c0 tx=0x7fa838009030 comp rx=0 tx=0).stop 2024-04-06T12:25:07.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.985+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/275653477 shutdown_connections 2024-04-06T12:25:07.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.986+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/275653477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377332980 0x564377333b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.986+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/275653477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377357c20 0x56437735ba60 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.986+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/275653477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 0x5643773335f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.986+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/275653477 >> 172.21.15.189:0/275653477 conn(0x564377293650 msgr2=0x56437737a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:07.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.986+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/275653477 shutdown_connections 2024-04-06T12:25:07.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.986+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/275653477 wait complete. 2024-04-06T12:25:07.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.986+0000 7fa84fe3eb80 1 -- start start 2024-04-06T12:25:07.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.986+0000 7fa84fe3eb80 1 Processor -- start 2024-04-06T12:25:07.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa84fe3eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 0x56437735a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa84fe3eb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377357c20 0x5643773480b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa84fe3eb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 0x564377379050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa84fe3eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x564377221d30 con 0x564377332980 2024-04-06T12:25:07.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa84fe3eb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643772219d0 con 0x564377357c20 2024-04-06T12:25:07.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa84fe3eb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x564377109650 con 0x56437735cfc0 2024-04-06T12:25:07.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa84fe3eb80 10 client.0 _ll_register_callbacks cb 0x56437722b420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:25:07.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa841a26700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 0x56437735a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa841a26700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 0x56437735a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57994/0 (socket says 172.21.15.189:57994) 2024-04-06T12:25:07.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa841a26700 1 -- 172.21.15.189:0/1555404520 learned_addr learned my addr 172.21.15.189:0/1555404520 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:07.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa842227700 1 --2- 172.21.15.189:0/1555404520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 0x564377379050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa841225700 1 --2- 172.21.15.189:0/1555404520 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377357c20 0x5643773480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa841a26700 1 -- 172.21.15.189:0/1555404520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 msgr2=0x564377379050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:07.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa841a26700 1 --2- 172.21.15.189:0/1555404520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 0x564377379050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa841a26700 1 -- 172.21.15.189:0/1555404520 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377357c20 msgr2=0x5643773480b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:07.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa841a26700 1 --2- 172.21.15.189:0/1555404520 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377357c20 0x5643773480b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa841a26700 1 -- 172.21.15.189:0/1555404520 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8280035b0 con 0x564377332980 2024-04-06T12:25:07.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa841a26700 1 --2- 172.21.15.189:0/1555404520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 0x56437735a010 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fa838005040 tx=0x7fa83800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d082706d07a92f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:07.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa81cff9700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:25:07.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa81cff9700 1 -- 172.21.15.189:0/1555404520 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838038070 con 0x564377332980 2024-04-06T12:25:07.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa81cff9700 1 -- 172.21.15.189:0/1555404520 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa8380073c0 con 0x564377332980 2024-04-06T12:25:07.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.988+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/1555404520 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564377246330 con 0x564377332980 2024-04-06T12:25:07.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.989+0000 7fa81cff9700 1 -- 172.21.15.189:0/1555404520 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa838007580 con 0x564377332980 2024-04-06T12:25:07.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.989+0000 7fa84fe3eb80 10 client.6222 fetch_fsmap learned FSMap version 24 2024-04-06T12:25:07.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.989+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/1555404520 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x56437724be50 con 0x564377332980 2024-04-06T12:25:07.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.989+0000 7fa81cff9700 1 -- 172.21.15.189:0/1555404520 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564377246330 con 0x564377332980 2024-04-06T12:25:07.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.989+0000 7fa81cff9700 1 -- 172.21.15.189:0/1555404520 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 24) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa838003680 con 0x564377332980 2024-04-06T12:25:07.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.989+0000 7fa84fe3eb80 10 client.6222 fetch_fsmap finished waiting for FSMap version 24 2024-04-06T12:25:07.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.989+0000 7fa84fe3eb80 10 client.6222 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-06T12:25:07.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.989+0000 7fa84fe3eb80 20 client.6222 populate_metadata read hostname 'smithi189' 2024-04-06T12:25:07.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.989+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/1555404520 >> [v2:172.21.15.192:6832/525827513,v1:172.21.15.192:6834/525827513] conn(0x564377388b50 0x564377388f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:07.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.989+0000 7fa84fe3eb80 4 client.6222 mds_command: new command op to 6114 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-06T12:25:07.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.989+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/1555404520 --> [v2:172.21.15.192:6832/525827513,v1:172.21.15.192:6834/525827513] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56437727e6b0 con 0x564377388b50 2024-04-06T12:25:07.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.989+0000 7fa841225700 1 --2- 172.21.15.189:0/1555404520 >> [v2:172.21.15.192:6832/525827513,v1:172.21.15.192:6834/525827513] conn(0x564377388b50 0x564377388f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:07.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.989+0000 7fa81cff9700 10 client.6222 ms_handle_connect on v2:172.21.15.192:6832/525827513 2024-04-06T12:25:07.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.989+0000 7fa841225700 1 --2- 172.21.15.189:0/1555404520 >> [v2:172.21.15.192:6832/525827513,v1:172.21.15.192:6834/525827513] conn(0x564377388b50 0x564377388f30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecc4cf4a26fa8e64 server_cookie=698bc700093e3a82 in_seq=0 out_seq=0 2024-04-06T12:25:07.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.990+0000 7fa81cff9700 1 -- 172.21.15.189:0/1555404520 <== mds.0 v2:172.21.15.192:6832/525827513 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56437727e6b0 con 0x564377388b50 2024-04-06T12:25:07.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.990+0000 7fa81cff9700 10 client.6222 handle_command_reply: tid=0 2024-04-06T12:25:07.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.990+0000 7fa84fe3eb80 1 client.6222 shutdown 2024-04-06T12:25:07.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.990+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/1555404520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 msgr2=0x56437735a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:07.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.990+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/1555404520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 0x56437735a010 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fa838005040 tx=0x7fa83800f000 comp rx=0 tx=0).stop 2024-04-06T12:25:07.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.990+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/1555404520 shutdown_connections 2024-04-06T12:25:07.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.990+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/1555404520 >> [v2:172.21.15.192:6832/525827513,v1:172.21.15.192:6834/525827513] conn(0x564377388b50 0x564377388f30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.990+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/1555404520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 0x564377379050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.990+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/1555404520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 0x56437735a010 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.990+0000 7fa81cff9700 0 client.6222 ms_handle_reset on v2:172.21.15.192:6832/525827513 2024-04-06T12:25:07.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.990+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/1555404520 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377357c20 0x5643773480b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:07.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.990+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/1555404520 >> 172.21.15.189:0/1555404520 conn(0x564377293650 msgr2=0x5643773335f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:07.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.990+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/1555404520 shutdown_connections 2024-04-06T12:25:07.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.990+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/1555404520 wait complete. 2024-04-06T12:25:07.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:07.991+0000 7fa84fe3eb80 20 client.6222 trim_cache size 0 max 16384 2024-04-06T12:25:08.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.008+0000 7fa84fe3eb80 1 Processor -- start 2024-04-06T12:25:08.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.008+0000 7fa84fe3eb80 1 -- start start 2024-04-06T12:25:08.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.009+0000 7fa84fe3eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 0x56437737a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:08.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.009+0000 7fa84fe3eb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377357c20 0x5643773d8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:08.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.009+0000 7fa84fe3eb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 0x564377359210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:08.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.009+0000 7fa84fe3eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x564377332840 con 0x564377332980 2024-04-06T12:25:08.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.009+0000 7fa84fe3eb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564377330ad0 con 0x564377357c20 2024-04-06T12:25:08.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.009+0000 7fa84fe3eb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x56437739c330 con 0x56437735cfc0 2024-04-06T12:25:08.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.009+0000 7fa841a26700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 0x56437737a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:08.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.009+0000 7fa841a26700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 0x56437737a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58004/0 (socket says 172.21.15.189:58004) 2024-04-06T12:25:08.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.009+0000 7fa841a26700 1 -- 172.21.15.189:0/4224101112 learned_addr learned my addr 172.21.15.189:0/4224101112 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:08.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.009+0000 7fa842227700 1 --2- 172.21.15.189:0/4224101112 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 0x564377359210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:08.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.009+0000 7fa841225700 1 --2- 172.21.15.189:0/4224101112 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377357c20 0x5643773d8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:08.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.009+0000 7fa841a26700 1 -- 172.21.15.189:0/4224101112 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 msgr2=0x564377359210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:08.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.009+0000 7fa841a26700 1 --2- 172.21.15.189:0/4224101112 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 0x564377359210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.009+0000 7fa841a26700 1 -- 172.21.15.189:0/4224101112 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377357c20 msgr2=0x5643773d8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:08.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.009+0000 7fa841a26700 1 --2- 172.21.15.189:0/4224101112 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377357c20 0x5643773d8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.009+0000 7fa841a26700 1 -- 172.21.15.189:0/4224101112 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56437710bc80 con 0x564377332980 2024-04-06T12:25:08.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.009+0000 7fa841a26700 1 --2- 172.21.15.189:0/4224101112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 0x56437737a650 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa838006b40 tx=0x7fa838014050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb94194ba1c453a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:08.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.010+0000 7fa81ffff700 1 -- 172.21.15.189:0/4224101112 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838003b40 con 0x564377332980 2024-04-06T12:25:08.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.010+0000 7fa81ffff700 1 -- 172.21.15.189:0/4224101112 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa838003680 con 0x564377332980 2024-04-06T12:25:08.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.010+0000 7fa81ffff700 1 -- 172.21.15.189:0/4224101112 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83800cdc0 con 0x564377332980 2024-04-06T12:25:08.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.010+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/4224101112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 msgr2=0x56437737a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:08.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.010+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/4224101112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 0x56437737a650 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa838006b40 tx=0x7fa838014050 comp rx=0 tx=0).stop 2024-04-06T12:25:08.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.010+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/4224101112 shutdown_connections 2024-04-06T12:25:08.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.010+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/4224101112 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 0x564377359210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.010+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/4224101112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 0x56437737a650 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.010+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/4224101112 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377357c20 0x5643773d8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.010+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/4224101112 >> 172.21.15.189:0/4224101112 conn(0x564377293650 msgr2=0x56437738c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:08.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.010+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/4224101112 shutdown_connections 2024-04-06T12:25:08.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.010+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/4224101112 wait complete. 2024-04-06T12:25:08.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.011+0000 7fa84fe3eb80 1 -- start start 2024-04-06T12:25:08.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.011+0000 7fa84fe3eb80 1 Processor -- start 2024-04-06T12:25:08.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa84fe3eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 0x564377356df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:08.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa84fe3eb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377357c20 0x5643773591d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:08.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa84fe3eb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 0x564377359710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:08.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa84fe3eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643773573f0 con 0x564377332980 2024-04-06T12:25:08.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa84fe3eb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x56437738bfe0 con 0x564377357c20 2024-04-06T12:25:08.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa84fe3eb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x564377357640 con 0x56437735cfc0 2024-04-06T12:25:08.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa84fe3eb80 10 client.0 _ll_register_callbacks cb 0x56437722b420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:25:08.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa841a26700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 0x564377356df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:08.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa841a26700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 0x564377356df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58006/0 (socket says 172.21.15.189:58006) 2024-04-06T12:25:08.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa841a26700 1 -- 172.21.15.189:0/3591270316 learned_addr learned my addr 172.21.15.189:0/3591270316 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:08.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa841225700 1 --2- 172.21.15.189:0/3591270316 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377357c20 0x5643773591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:08.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa842227700 1 --2- 172.21.15.189:0/3591270316 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 0x564377359710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:08.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa841a26700 1 -- 172.21.15.189:0/3591270316 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 msgr2=0x564377359710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:08.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa841a26700 1 --2- 172.21.15.189:0/3591270316 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 0x564377359710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa841a26700 1 -- 172.21.15.189:0/3591270316 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377357c20 msgr2=0x5643773591d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:08.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa841a26700 1 --2- 172.21.15.189:0/3591270316 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377357c20 0x5643773591d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa841a26700 1 -- 172.21.15.189:0/3591270316 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56437724be50 con 0x564377332980 2024-04-06T12:25:08.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa841a26700 1 --2- 172.21.15.189:0/3591270316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 0x564377356df0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fa838006bc0 tx=0x7fa838012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbb8a04fbd6a14df server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:08.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa81cff9700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:25:08.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa81cff9700 1 -- 172.21.15.189:0/3591270316 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83800cd70 con 0x564377332980 2024-04-06T12:25:08.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa81cff9700 1 -- 172.21.15.189:0/3591270316 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa838003b40 con 0x564377332980 2024-04-06T12:25:08.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.012+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/3591270316 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa838007040 con 0x564377332980 2024-04-06T12:25:08.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.013+0000 7fa81cff9700 1 -- 172.21.15.189:0/3591270316 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8380037f0 con 0x564377332980 2024-04-06T12:25:08.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.013+0000 7fa84fe3eb80 10 client.6228 fetch_fsmap learned FSMap version 24 2024-04-06T12:25:08.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.013+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/3591270316 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x5643770b1fe0 con 0x564377332980 2024-04-06T12:25:08.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.013+0000 7fa81cff9700 1 -- 172.21.15.189:0/3591270316 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa838007040 con 0x564377332980 2024-04-06T12:25:08.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.013+0000 7fa81cff9700 1 -- 172.21.15.189:0/3591270316 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 24) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa8380073c0 con 0x564377332980 2024-04-06T12:25:08.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.013+0000 7fa84fe3eb80 10 client.6228 fetch_fsmap finished waiting for FSMap version 24 2024-04-06T12:25:08.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.013+0000 7fa84fe3eb80 10 client.6228 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-06T12:25:08.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.013+0000 7fa84fe3eb80 20 client.6228 populate_metadata read hostname 'smithi189' 2024-04-06T12:25:08.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.013+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/3591270316 >> [v2:172.21.15.192:6832/525827513,v1:172.21.15.192:6834/525827513] conn(0x5643773c7d50 0x56437738af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:08.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.013+0000 7fa84fe3eb80 4 client.6228 mds_command: new command op to 6114 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-06T12:25:08.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.013+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/3591270316 --> [v2:172.21.15.192:6832/525827513,v1:172.21.15.192:6834/525827513] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5643770d4ca0 con 0x5643773c7d50 2024-04-06T12:25:08.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.013+0000 7fa841225700 1 --2- 172.21.15.189:0/3591270316 >> [v2:172.21.15.192:6832/525827513,v1:172.21.15.192:6834/525827513] conn(0x5643773c7d50 0x56437738af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:08.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.013+0000 7fa81cff9700 10 client.6228 ms_handle_connect on v2:172.21.15.192:6832/525827513 2024-04-06T12:25:08.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.013+0000 7fa841225700 1 --2- 172.21.15.189:0/3591270316 >> [v2:172.21.15.192:6832/525827513,v1:172.21.15.192:6834/525827513] conn(0x5643773c7d50 0x56437738af40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e68e23d76194c4f server_cookie=a3bb28b919f3259d in_seq=0 out_seq=0 2024-04-06T12:25:08.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.014+0000 7fa81cff9700 1 -- 172.21.15.189:0/3591270316 <== mds.0 v2:172.21.15.192:6832/525827513 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5643770d4ca0 con 0x5643773c7d50 2024-04-06T12:25:08.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.014+0000 7fa81cff9700 10 client.6228 handle_command_reply: tid=0 2024-04-06T12:25:08.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.014+0000 7fa84fe3eb80 1 client.6228 shutdown 2024-04-06T12:25:08.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.014+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/3591270316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 msgr2=0x564377356df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:08.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.014+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/3591270316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 0x564377356df0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fa838006bc0 tx=0x7fa838012000 comp rx=0 tx=0).stop 2024-04-06T12:25:08.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.014+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/3591270316 shutdown_connections 2024-04-06T12:25:08.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.014+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/3591270316 >> [v2:172.21.15.192:6832/525827513,v1:172.21.15.192:6834/525827513] conn(0x5643773c7d50 0x56437738af40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.014+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/3591270316 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x56437735cfc0 0x564377359710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.014+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/3591270316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564377332980 0x564377356df0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.014+0000 7fa81cff9700 0 client.6228 ms_handle_reset on v2:172.21.15.192:6832/525827513 2024-04-06T12:25:08.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.014+0000 7fa84fe3eb80 1 --2- 172.21.15.189:0/3591270316 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564377357c20 0x5643773591d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.014+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/3591270316 >> 172.21.15.189:0/3591270316 conn(0x564377293650 msgr2=0x5643773d8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:08.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.014+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/3591270316 shutdown_connections 2024-04-06T12:25:08.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.014+0000 7fa84fe3eb80 1 -- 172.21.15.189:0/3591270316 wait complete. 2024-04-06T12:25:08.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.014+0000 7fa84fe3eb80 20 client.6228 trim_cache size 0 max 16384 2024-04-06T12:25:08.017 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T12:25:08.017 INFO:teuthology.orchestra.run.smithi189.stdout: "cluster_fsid": "e30041de-e9a3-4351-8ca7-29d7f3e3728c", 2024-04-06T12:25:08.017 INFO:teuthology.orchestra.run.smithi189.stdout: "whoami": 0, 2024-04-06T12:25:08.017 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 6114, 2024-04-06T12:25:08.018 INFO:teuthology.orchestra.run.smithi189.stdout: "want_state": "up:active", 2024-04-06T12:25:08.018 INFO:teuthology.orchestra.run.smithi189.stdout: "state": "up:active", 2024-04-06T12:25:08.018 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs", 2024-04-06T12:25:08.018 INFO:teuthology.orchestra.run.smithi189.stdout: "rank_uptime": 5.9500711940000004, 2024-04-06T12:25:08.018 INFO:teuthology.orchestra.run.smithi189.stdout: "mdsmap_epoch": 24, 2024-04-06T12:25:08.018 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch": 67, 2024-04-06T12:25:08.018 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch_barrier": 67, 2024-04-06T12:25:08.018 INFO:teuthology.orchestra.run.smithi189.stdout: "uptime": 11.274447027000001 2024-04-06T12:25:08.018 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T12:25:08.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.014+0000 7fa843c8a700 1 -- 172.21.15.189:0/310585546 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa82006adb0 msgr2=0x7fa82006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:08.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.014+0000 7fa843c8a700 1 --2- 172.21.15.189:0/310585546 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa82006adb0 0x7fa82006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa838015040 tx=0x7fa83800a000 comp rx=0 tx=0).stop 2024-04-06T12:25:08.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.014+0000 7fa843c8a700 1 -- 172.21.15.189:0/310585546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa83c154ad0 msgr2=0x7fa83c0b14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:08.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.014+0000 7fa843c8a700 1 --2- 172.21.15.189:0/310585546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa83c154ad0 0x7fa83c0b14c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fa82c011a80 tx=0x7fa82c011e40 comp rx=0 tx=0).stop 2024-04-06T12:25:08.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.015+0000 7fa843c8a700 1 -- 172.21.15.189:0/310585546 shutdown_connections 2024-04-06T12:25:08.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.015+0000 7fa843c8a700 1 --2- 172.21.15.189:0/310585546 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa83c159070 0x7fa83c0aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.015+0000 7fa843c8a700 1 --2- 172.21.15.189:0/310585546 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa82006adb0 0x7fa82006d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.015+0000 7fa843c8a700 1 --2- 172.21.15.189:0/310585546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa828003be0 0x7fa83c0cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.015+0000 7fa843c8a700 1 --2- 172.21.15.189:0/310585546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa83c154ad0 0x7fa83c0b14c0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.015+0000 7fa843c8a700 1 -- 172.21.15.189:0/310585546 >> 172.21.15.189:0/310585546 conn(0x7fa83c0d00b0 msgr2=0x7fa83c06fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:08.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.015+0000 7fa843c8a700 1 -- 172.21.15.189:0/310585546 shutdown_connections 2024-04-06T12:25:08.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.015+0000 7fa843c8a700 1 -- 172.21.15.189:0/310585546 wait complete. 2024-04-06T12:25:08.024 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T12:25:08.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.148+0000 7f1abe4d4700 1 Processor -- start 2024-04-06T12:25:08.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.148+0000 7f1abe4d4700 1 -- start start 2024-04-06T12:25:08.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.149+0000 7f1abe4d4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ab815d7b0 0x7f1ab80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:08.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.149+0000 7f1abe4d4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1ab806a730 0x7f1ab806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:08.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.149+0000 7f1abe4d4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab80b3990 con 0x7f1ab815ddf0 2024-04-06T12:25:08.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.149+0000 7f1abe4d4700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab806b0f0 con 0x7f1ab806a730 2024-04-06T12:25:08.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.149+0000 7f1abe4d4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab80b3060 con 0x7f1ab815d7b0 2024-04-06T12:25:08.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.149+0000 7f1ab77fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1ab806a730 0x7f1ab806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:08.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.149+0000 7f1ab77fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1ab806a730 0x7f1ab806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58008/0 (socket says 172.21.15.189:58008) 2024-04-06T12:25:08.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.149+0000 7f1ab77fe700 1 -- 172.21.15.189:0/2902054611 learned_addr learned my addr 172.21.15.189:0/2902054611 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:08.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.149+0000 7f1ab7fff700 1 --2- 172.21.15.189:0/2902054611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ab815d7b0 0x7f1ab80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:08.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.149+0000 7f1ab77fe700 1 -- 172.21.15.189:0/2902054611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ab815d7b0 msgr2=0x7f1ab80b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:08.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.149+0000 7f1ab77fe700 1 --2- 172.21.15.189:0/2902054611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ab815d7b0 0x7f1ab80b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.149+0000 7f1ab77fe700 1 -- 172.21.15.189:0/2902054611 >> v1:172.21.15.192:6789/0 conn(0x7f1ab815ddf0 legacy=0x7f1ab815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:08.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.149+0000 7f1ab77fe700 1 -- 172.21.15.189:0/2902054611 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab80b3340 con 0x7f1ab806a730 2024-04-06T12:25:08.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.149+0000 7f1ab77fe700 1 --2- 172.21.15.189:0/2902054611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ab806a730 0x7f1ab806abb0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80098f0 tx=0x7f1aa8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f46d7dae2efa6ccd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:08.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.150+0000 7f1ab6ffd700 1 -- 172.21.15.189:0/2902054611 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa800b8a0 con 0x7f1ab806a730 2024-04-06T12:25:08.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.150+0000 7f1ab6ffd700 1 -- 172.21.15.189:0/2902054611 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa800ba00 con 0x7f1ab806a730 2024-04-06T12:25:08.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.150+0000 7f1ab6ffd700 1 -- 172.21.15.189:0/2902054611 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa800bd90 con 0x7f1ab806a730 2024-04-06T12:25:08.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.150+0000 7f1abe4d4700 1 -- 172.21.15.189:0/2902054611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ab806a730 msgr2=0x7f1ab806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:08.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.150+0000 7f1abe4d4700 1 --2- 172.21.15.189:0/2902054611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ab806a730 0x7f1ab806abb0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80098f0 tx=0x7f1aa8009c00 comp rx=0 tx=0).stop 2024-04-06T12:25:08.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.150+0000 7f1abe4d4700 1 -- 172.21.15.189:0/2902054611 shutdown_connections 2024-04-06T12:25:08.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.150+0000 7f1abe4d4700 1 --2- 172.21.15.189:0/2902054611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ab815d7b0 0x7f1ab80b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.150+0000 7f1abe4d4700 1 --2- 172.21.15.189:0/2902054611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ab806a730 0x7f1ab806abb0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.150+0000 7f1abe4d4700 1 -- 172.21.15.189:0/2902054611 >> 172.21.15.189:0/2902054611 conn(0x7f1ab80d00b0 msgr2=0x7f1ab80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:08.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.150+0000 7f1abe4d4700 1 -- 172.21.15.189:0/2902054611 shutdown_connections 2024-04-06T12:25:08.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.150+0000 7f1abe4d4700 1 -- 172.21.15.189:0/2902054611 wait complete. 2024-04-06T12:25:08.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.152+0000 7f1abe4d4700 1 Processor -- start 2024-04-06T12:25:08.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.152+0000 7f1abe4d4700 1 -- start start 2024-04-06T12:25:08.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.153+0000 7f1abe4d4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ab806a730 0x7f1ab80af090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:08.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.153+0000 7f1abe4d4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ab815d7b0 0x7f1ab81f6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:08.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.153+0000 7f1abe4d4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ab815ddf0 0x7f1ab8146630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:08.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.153+0000 7f1abe4d4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab815af70 con 0x7f1ab815ddf0 2024-04-06T12:25:08.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.153+0000 7f1abe4d4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab81f71c0 con 0x7f1ab815d7b0 2024-04-06T12:25:08.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.153+0000 7f1abe4d4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab81f74a0 con 0x7f1ab806a730 2024-04-06T12:25:08.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.153+0000 7f1abca71700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ab815ddf0 0x7f1ab8146630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:08.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.153+0000 7f1abca71700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ab815ddf0 0x7f1ab8146630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58010/0 (socket says 172.21.15.189:58010) 2024-04-06T12:25:08.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.153+0000 7f1abca71700 1 -- 172.21.15.189:0/1849039631 learned_addr learned my addr 172.21.15.189:0/1849039631 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:08.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.153+0000 7f1ab7fff700 1 --2- 172.21.15.189:0/1849039631 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ab806a730 0x7f1ab80af090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:08.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.153+0000 7f1ab77fe700 1 --2- 172.21.15.189:0/1849039631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ab815d7b0 0x7f1ab81f6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:08.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.153+0000 7f1abca71700 1 -- 172.21.15.189:0/1849039631 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ab806a730 msgr2=0x7f1ab80af090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:08.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.153+0000 7f1abca71700 1 --2- 172.21.15.189:0/1849039631 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ab806a730 0x7f1ab80af090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.153+0000 7f1abca71700 1 -- 172.21.15.189:0/1849039631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ab815d7b0 msgr2=0x7f1ab81f6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:08.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.153+0000 7f1abca71700 1 --2- 172.21.15.189:0/1849039631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ab815d7b0 0x7f1ab81f6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.153+0000 7f1abca71700 1 -- 172.21.15.189:0/1849039631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa8009550 con 0x7f1ab815ddf0 2024-04-06T12:25:08.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.153+0000 7f1abca71700 1 --2- 172.21.15.189:0/1849039631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ab815ddf0 0x7f1ab8146630 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000cb50 tx=0x7f1ab000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:08.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.153+0000 7f1ab4ff9700 1 -- 172.21.15.189:0/1849039631 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0003c00 con 0x7f1ab815ddf0 2024-04-06T12:25:08.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.154+0000 7f1ab4ff9700 1 -- 172.21.15.189:0/1849039631 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab0003d60 con 0x7f1ab815ddf0 2024-04-06T12:25:08.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.154+0000 7f1abe4d4700 1 -- 172.21.15.189:0/1849039631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab815b210 con 0x7f1ab815ddf0 2024-04-06T12:25:08.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.154+0000 7f1ab4ff9700 1 -- 172.21.15.189:0/1849039631 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab000aa20 con 0x7f1ab815ddf0 2024-04-06T12:25:08.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.154+0000 7f1abe4d4700 1 -- 172.21.15.189:0/1849039631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab80cc630 con 0x7f1ab815ddf0 2024-04-06T12:25:08.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.155+0000 7f1abe4d4700 1 -- 172.21.15.189:0/1849039631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab815b730 con 0x7f1ab815ddf0 2024-04-06T12:25:08.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.155+0000 7f1ab4ff9700 1 -- 172.21.15.189:0/1849039631 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f1ab000ab80 con 0x7f1ab815ddf0 2024-04-06T12:25:08.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.155+0000 7f1ab4ff9700 1 --2- 172.21.15.189:0/1849039631 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1aa406adb0 0x7f1aa406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:08.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.155+0000 7f1ab4ff9700 1 -- 172.21.15.189:0/1849039631 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1ab0089690 con 0x7f1ab815ddf0 2024-04-06T12:25:08.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.155+0000 7f1ab7fff700 1 --2- 172.21.15.189:0/1849039631 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1aa406adb0 0x7f1aa406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:08.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.156+0000 7f1ab7fff700 1 --2- 172.21.15.189:0/1849039631 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1aa406adb0 0x7f1aa406d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1aac006b20 tx=0x7f1aac008040 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:08.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.160+0000 7f1ab4ff9700 1 -- 172.21.15.189:0/1849039631 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1ab0054580 con 0x7f1ab815ddf0 2024-04-06T12:25:08.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.399+0000 7f1abe4d4700 1 -- 172.21.15.189:0/1849039631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1ab8067bb0 con 0x7f1ab815ddf0 2024-04-06T12:25:08.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.399+0000 7f1ab4ff9700 1 -- 172.21.15.189:0/1849039631 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1ab0057ba0 con 0x7f1ab815ddf0 2024-04-06T12:25:08.396 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-06T12:25:08.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.401+0000 7f1abe4d4700 1 -- 172.21.15.189:0/1849039631 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1aa406adb0 msgr2=0x7f1aa406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.401+0000 7f1abe4d4700 1 --2- 172.21.15.189:0/1849039631 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1aa406adb0 0x7f1aa406d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1aac006b20 tx=0x7f1aac008040 comp rx=0 tx=0).stop 2024-04-06T12:25:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.401+0000 7f1abe4d4700 1 -- 172.21.15.189:0/1849039631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ab815ddf0 msgr2=0x7f1ab8146630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.401+0000 7f1abe4d4700 1 --2- 172.21.15.189:0/1849039631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ab815ddf0 0x7f1ab8146630 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000cb50 tx=0x7f1ab000ce60 comp rx=0 tx=0).stop 2024-04-06T12:25:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.401+0000 7f1abe4d4700 1 -- 172.21.15.189:0/1849039631 shutdown_connections 2024-04-06T12:25:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.401+0000 7f1abe4d4700 1 --2- 172.21.15.189:0/1849039631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ab815ddf0 0x7f1ab8146630 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.401+0000 7f1abe4d4700 1 --2- 172.21.15.189:0/1849039631 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ab806a730 0x7f1ab80af090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.401+0000 7f1abe4d4700 1 --2- 172.21.15.189:0/1849039631 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1aa406adb0 0x7f1aa406d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.401+0000 7f1abe4d4700 1 --2- 172.21.15.189:0/1849039631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ab815d7b0 0x7f1ab81f6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.401+0000 7f1abe4d4700 1 -- 172.21.15.189:0/1849039631 >> 172.21.15.189:0/1849039631 conn(0x7f1ab80d00b0 msgr2=0x7f1ab80ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.401+0000 7f1abe4d4700 1 -- 172.21.15.189:0/1849039631 shutdown_connections 2024-04-06T12:25:08.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:08.401+0000 7f1abe4d4700 1 -- 172.21.15.189:0/1849039631 wait complete. 2024-04-06T12:25:08.410 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-06T12:25:08.411 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T12:25:08.411 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-06T12:25:08.411 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-06T12:25:08.411 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T12:25:08.411 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-06T12:25:08.411 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-06T12:25:08.411 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T12:25:08.411 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:25:08.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:08.411 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-06T12:25:08.430 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T12:25:08.430 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T12:25:08.430 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T12:25:08.431 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:25:08.431 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T12:25:08.431 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:25:08.431 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:25:08.431 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:25:08.431 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-06T12:25:08.431 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:25:08.431 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:25:08.431 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-06T12:25:08.431 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:25:08.431 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:25:08.431 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T12:25:08.431 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:25:08.431 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T12:25:08.431 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 41581sec preferred_lft 41581sec 2024-04-06T12:25:08.431 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::8592:df2c:92e2:dace/64 scope link noprefixroute 2024-04-06T12:25:08.432 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:25:08.432 INFO:teuthology.orchestra.run.smithi189.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T12:25:08.432 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff 2024-04-06T12:25:08.432 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-06T12:25:08.432 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:25:08.432 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::88e:b6ff:fe6f:1cb0/64 scope link 2024-04-06T12:25:08.432 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:25:08.432 INFO:teuthology.orchestra.run.smithi189.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T12:25:08.432 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 62:a5:8d:c5:1a:9c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T12:25:08.432 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::60a5:8dff:fec5:1a9c/64 scope link 2024-04-06T12:25:08.432 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:25:08.433 INFO:teuthology.orchestra.run.smithi189.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T12:25:08.433 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-06T12:25:08.433 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::1c7e:c6ff:febb:1dab/64 scope link 2024-04-06T12:25:08.433 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:25:08.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:08.434 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T12:25:08.497 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-06T12:25:08.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:08.499 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-06T12:25:08.558 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-06T12:25:08.559 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T12:25:08.559 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T12:25:08.559 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T12:25:08.559 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T12:25:08.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:08.559 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:25:08.559 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:25:08.559 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T12:25:08.560 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T12:25:08.560 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T12:25:08.560 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:25:08.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:08.666 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T12:25:08.682 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-06T12:25:08.683 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-06T12:25:08.683 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:25:08.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:08.683 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-04-06T12:25:08.750 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:25:08.750 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-06T12:25:08.807 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:25:08.807 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:25:08.807 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-06T12:25:08.864 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T12:25:08.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:08.864 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:25:08.950 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:25:08.950 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:25:08.950 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:25:08.959 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:25:08.959 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:25:08.959 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:25:08.959 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:25:08.960 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:25:08.978 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:25:08.978 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:25:08.978 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:25:08.979 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:25:08.979 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:25:08.979 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790" 2024-04-06T12:25:08.979 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2024-04-06T12:25:08.979 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e30041de-e9a3-4351-8ca7-29d7f3e3728c" will pass to kernel 2024-04-06T12:25:08.981 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T12:25:08.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:08.981 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-06T12:25:09.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:09.010 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:25:09.105 INFO:teuthology.orchestra.run.smithi189.stdout:client5932 2024-04-06T12:25:09.105 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:25:09.105 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:25:09.123 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-06T12:25:09.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.288+0000 7f933f7f4700 1 Processor -- start 2024-04-06T12:25:09.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.288+0000 7f933f7f4700 1 -- start start 2024-04-06T12:25:09.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.289+0000 7f933f7f4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f93380d11b0 0x7f93380d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:09.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.289+0000 7f933f7f4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93380d1ad0 con 0x7f93380cb870 2024-04-06T12:25:09.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.289+0000 7f933f7f4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93380cbd80 con 0x7f93380cfa00 2024-04-06T12:25:09.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.289+0000 7f933f7f4700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93380ccdc0 con 0x7f93380d11b0 2024-04-06T12:25:09.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.289+0000 7f933d590700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f93380d11b0 0x7f93380d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:09.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.289+0000 7f933cd8f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f93380cb870 0x7f933815c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46714/0 (socket says 172.21.15.189:46714) 2024-04-06T12:25:09.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.289+0000 7f933cd8f700 1 -- 172.21.15.189:0/3271666598 learned_addr learned my addr 172.21.15.189:0/3271666598 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:25:09.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.289+0000 7f933d590700 1 -- 172.21.15.189:0/3271666598 >> v1:172.21.15.192:6789/0 conn(0x7f93380cfa00 legacy=0x7f93380cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:09.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.289+0000 7f933d590700 1 -- 172.21.15.189:0/3271666598 >> v1:172.21.15.189:6789/0 conn(0x7f93380cb870 legacy=0x7f933815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:09.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.289+0000 7f933d590700 1 -- 172.21.15.189:0/3271666598 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93380cef20 con 0x7f93380d11b0 2024-04-06T12:25:09.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.290+0000 7f933d590700 1 --2- 172.21.15.189:0/3271666598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93380d11b0 0x7f93380d1590 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f9334009840 tx=0x7f9334009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edbe019a73c24c29 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:09.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.290+0000 7f932ffff700 1 -- 172.21.15.189:0/3271666598 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9334004230 con 0x7f93380d11b0 2024-04-06T12:25:09.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.290+0000 7f932ffff700 1 -- 172.21.15.189:0/3271666598 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9334004390 con 0x7f93380d11b0 2024-04-06T12:25:09.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.290+0000 7f932ffff700 1 -- 172.21.15.189:0/3271666598 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93340036a0 con 0x7f93380d11b0 2024-04-06T12:25:09.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.290+0000 7f933f7f4700 1 -- 172.21.15.189:0/3271666598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93380d11b0 msgr2=0x7f93380d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:09.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.290+0000 7f933f7f4700 1 --2- 172.21.15.189:0/3271666598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93380d11b0 0x7f93380d1590 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f9334009840 tx=0x7f9334009b50 comp rx=0 tx=0).stop 2024-04-06T12:25:09.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.290+0000 7f933f7f4700 1 -- 172.21.15.189:0/3271666598 shutdown_connections 2024-04-06T12:25:09.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.290+0000 7f933f7f4700 1 --2- 172.21.15.189:0/3271666598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93380d11b0 0x7f93380d1590 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:09.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.290+0000 7f933f7f4700 1 -- 172.21.15.189:0/3271666598 >> 172.21.15.189:0/3271666598 conn(0x7f93380b2db0 msgr2=0x7f93380b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:09.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.290+0000 7f933f7f4700 1 -- 172.21.15.189:0/3271666598 shutdown_connections 2024-04-06T12:25:09.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.290+0000 7f933f7f4700 1 -- 172.21.15.189:0/3271666598 wait complete. 2024-04-06T12:25:09.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.292+0000 7f933f7f4700 1 Processor -- start 2024-04-06T12:25:09.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.292+0000 7f933f7f4700 1 -- start start 2024-04-06T12:25:09.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.292+0000 7f933f7f4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93380cb870 0x7f933814b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:09.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.292+0000 7f933f7f4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93380cfa00 0x7f93380c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:09.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.292+0000 7f933f7f4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93380d11b0 0x7f933815c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:09.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.292+0000 7f933f7f4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93380c5f70 con 0x7f93380cfa00 2024-04-06T12:25:09.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.292+0000 7f933f7f4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93380c6250 con 0x7f93380d11b0 2024-04-06T12:25:09.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.292+0000 7f933f7f4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93380c6530 con 0x7f93380cb870 2024-04-06T12:25:09.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.292+0000 7f933cd8f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93380cfa00 0x7f93380c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:09.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.292+0000 7f933cd8f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93380cfa00 0x7f93380c8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38674/0 (socket says 172.21.15.189:38674) 2024-04-06T12:25:09.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.292+0000 7f933cd8f700 1 -- 172.21.15.189:0/1780810245 learned_addr learned my addr 172.21.15.189:0/1780810245 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:09.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.293+0000 7f933d590700 1 --2- 172.21.15.189:0/1780810245 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93380cb870 0x7f933814b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:09.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.293+0000 7f933dd91700 1 --2- 172.21.15.189:0/1780810245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93380d11b0 0x7f933815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:09.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.293+0000 7f933cd8f700 1 -- 172.21.15.189:0/1780810245 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93380cb870 msgr2=0x7f933814b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:09.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.293+0000 7f933cd8f700 1 --2- 172.21.15.189:0/1780810245 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93380cb870 0x7f933814b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:09.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.293+0000 7f933cd8f700 1 -- 172.21.15.189:0/1780810245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93380d11b0 msgr2=0x7f933815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:09.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.293+0000 7f933cd8f700 1 --2- 172.21.15.189:0/1780810245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93380d11b0 0x7f933815c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:09.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.293+0000 7f933cd8f700 1 -- 172.21.15.189:0/1780810245 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93340094a0 con 0x7f93380cfa00 2024-04-06T12:25:09.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.293+0000 7f933dd91700 1 --2- 172.21.15.189:0/1780810245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93380d11b0 0x7f933815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:25:09.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.293+0000 7f933cd8f700 1 --2- 172.21.15.189:0/1780810245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93380cfa00 0x7f93380c8d00 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f932800c500 tx=0x7f932800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:09.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.293+0000 7f932dffb700 1 -- 172.21.15.189:0/1780810245 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9328005d70 con 0x7f93380cfa00 2024-04-06T12:25:09.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.293+0000 7f932dffb700 1 -- 172.21.15.189:0/1780810245 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93280070d0 con 0x7f93380cfa00 2024-04-06T12:25:09.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.293+0000 7f932dffb700 1 -- 172.21.15.189:0/1780810245 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9328007460 con 0x7f93380cfa00 2024-04-06T12:25:09.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.293+0000 7f933f7f4700 1 -- 172.21.15.189:0/1780810245 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93380c6810 con 0x7f93380cfa00 2024-04-06T12:25:09.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.293+0000 7f933f7f4700 1 -- 172.21.15.189:0/1780810245 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f933814ba00 con 0x7f93380cfa00 2024-04-06T12:25:09.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.294+0000 7f933f7f4700 1 -- 172.21.15.189:0/1780810245 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93380c60b0 con 0x7f93380cfa00 2024-04-06T12:25:09.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.295+0000 7f932dffb700 1 -- 172.21.15.189:0/1780810245 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f93280075c0 con 0x7f93380cfa00 2024-04-06T12:25:09.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.295+0000 7f932dffb700 1 --2- 172.21.15.189:0/1780810245 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f931c06adb0 0x7f931c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:09.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.295+0000 7f932dffb700 1 -- 172.21.15.189:0/1780810245 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9328088a70 con 0x7f93380cfa00 2024-04-06T12:25:09.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.295+0000 7f933d590700 1 --2- 172.21.15.189:0/1780810245 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f931c06adb0 0x7f931c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:09.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.295+0000 7f933d590700 1 --2- 172.21.15.189:0/1780810245 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f931c06adb0 0x7f931c06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f93340090b0 tx=0x7f9334008030 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:09.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.299+0000 7f932dffb700 1 -- 172.21.15.189:0/1780810245 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f93380c60b0 con 0x7f93380cfa00 2024-04-06T12:25:09.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.525+0000 7f933f7f4700 1 -- 172.21.15.189:0/1780810245 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f93380cc060 con 0x7f931c06adb0 2024-04-06T12:25:09.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.526+0000 7f932dffb700 1 -- 172.21.15.189:0/1780810245 <== mgr.5225 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f93340041b0 con 0x7f931c06adb0 2024-04-06T12:25:09.524 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T12:25:09.524 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:25:09.524 INFO:teuthology.orchestra.run.smithi189.stdout: "name": "cephfs" 2024-04-06T12:25:09.524 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:25:09.524 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T12:25:09.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.529+0000 7f933f7f4700 1 -- 172.21.15.189:0/1780810245 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f931c06adb0 msgr2=0x7f931c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:09.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.529+0000 7f933f7f4700 1 --2- 172.21.15.189:0/1780810245 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f931c06adb0 0x7f931c06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f93340090b0 tx=0x7f9334008030 comp rx=0 tx=0).stop 2024-04-06T12:25:09.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.529+0000 7f933f7f4700 1 -- 172.21.15.189:0/1780810245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93380cfa00 msgr2=0x7f93380c8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:09.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.529+0000 7f933f7f4700 1 --2- 172.21.15.189:0/1780810245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93380cfa00 0x7f93380c8d00 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f932800c500 tx=0x7f932800c8c0 comp rx=0 tx=0).stop 2024-04-06T12:25:09.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.529+0000 7f933f7f4700 1 -- 172.21.15.189:0/1780810245 shutdown_connections 2024-04-06T12:25:09.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.529+0000 7f933f7f4700 1 --2- 172.21.15.189:0/1780810245 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f931c06adb0 0x7f931c06d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:09.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.529+0000 7f933f7f4700 1 --2- 172.21.15.189:0/1780810245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93380d11b0 0x7f933815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:09.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.529+0000 7f933f7f4700 1 --2- 172.21.15.189:0/1780810245 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93380cb870 0x7f933814b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:09.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.529+0000 7f933f7f4700 1 --2- 172.21.15.189:0/1780810245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93380cfa00 0x7f93380c8d00 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:09.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.529+0000 7f933f7f4700 1 -- 172.21.15.189:0/1780810245 >> 172.21.15.189:0/1780810245 conn(0x7f93380b2db0 msgr2=0x7f93380b2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:09.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.529+0000 7f933f7f4700 1 -- 172.21.15.189:0/1780810245 shutdown_connections 2024-04-06T12:25:09.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.529+0000 7f933f7f4700 1 -- 172.21.15.189:0/1780810245 wait complete. 2024-04-06T12:25:09.538 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-06T12:25:09.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.666+0000 7f6684a9a700 1 Processor -- start 2024-04-06T12:25:09.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.666+0000 7f6684a9a700 1 -- start start 2024-04-06T12:25:09.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.667+0000 7f6684a9a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6680158fe0 0x7f668006a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:09.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.667+0000 7f6684a9a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f668015c2d0 0x7f66800ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:09.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.667+0000 7f6684a9a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f668015f9e0 con 0x7f668006ac70 2024-04-06T12:25:09.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.667+0000 7f6684a9a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f668006b0f0 con 0x7f6680158fe0 2024-04-06T12:25:09.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.667+0000 7f6684a9a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f668015f0d0 con 0x7f668015c2d0 2024-04-06T12:25:09.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.667+0000 7f667e59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6680158fe0 0x7f668006a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:09.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.667+0000 7f667e59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6680158fe0 0x7f668006a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38680/0 (socket says 172.21.15.189:38680) 2024-04-06T12:25:09.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.667+0000 7f667e59c700 1 -- 172.21.15.189:0/3348406165 learned_addr learned my addr 172.21.15.189:0/3348406165 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:09.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.667+0000 7f667dd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f668006ac70 0x7f668015bbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46724/0 (socket says 172.21.15.189:46724) 2024-04-06T12:25:09.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.667+0000 7f667ed9d700 1 --2- 172.21.15.189:0/3348406165 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f668015c2d0 0x7f66800ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:09.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.667+0000 7f667e59c700 1 -- 172.21.15.189:0/3348406165 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f668015c2d0 msgr2=0x7f66800ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:09.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.667+0000 7f667e59c700 1 --2- 172.21.15.189:0/3348406165 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f668015c2d0 0x7f66800ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:09.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.667+0000 7f667e59c700 1 -- 172.21.15.189:0/3348406165 >> v1:172.21.15.189:6789/0 conn(0x7f668006ac70 legacy=0x7f668015bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:09.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.667+0000 7f667d59a700 1 -- 172.21.15.189:0/3348406165 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115077463 0 0) 0x7f668015f9e0 con 0x7f668006ac70 2024-04-06T12:25:09.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.667+0000 7f667e59c700 1 -- 172.21.15.189:0/3348406165 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668015c8f0 con 0x7f6680158fe0 2024-04-06T12:25:09.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.667+0000 7f667e59c700 1 --2- 172.21.15.189:0/3348406165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6680158fe0 0x7f668006a730 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f66700098f0 tx=0x7f6670009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e4400874bef476e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:09.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.668+0000 7f667d59a700 1 -- 172.21.15.189:0/3348406165 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670011830 con 0x7f6680158fe0 2024-04-06T12:25:09.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.668+0000 7f667d59a700 1 -- 172.21.15.189:0/3348406165 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6670011990 con 0x7f6680158fe0 2024-04-06T12:25:09.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.668+0000 7f667d59a700 1 -- 172.21.15.189:0/3348406165 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670011d20 con 0x7f6680158fe0 2024-04-06T12:25:09.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.668+0000 7f6684a9a700 1 -- 172.21.15.189:0/3348406165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6680158fe0 msgr2=0x7f668006a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:09.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.668+0000 7f6684a9a700 1 --2- 172.21.15.189:0/3348406165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6680158fe0 0x7f668006a730 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f66700098f0 tx=0x7f6670009c00 comp rx=0 tx=0).stop 2024-04-06T12:25:09.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.668+0000 7f6684a9a700 1 -- 172.21.15.189:0/3348406165 shutdown_connections 2024-04-06T12:25:09.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.668+0000 7f6684a9a700 1 --2- 172.21.15.189:0/3348406165 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f668015c2d0 0x7f66800ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:09.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.668+0000 7f6684a9a700 1 --2- 172.21.15.189:0/3348406165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6680158fe0 0x7f668006a730 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:09.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.668+0000 7f6684a9a700 1 -- 172.21.15.189:0/3348406165 >> 172.21.15.189:0/3348406165 conn(0x7f66800d00b0 msgr2=0x7f66800d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:09.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.668+0000 7f6684a9a700 1 -- 172.21.15.189:0/3348406165 shutdown_connections 2024-04-06T12:25:09.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.668+0000 7f6684a9a700 1 -- 172.21.15.189:0/3348406165 wait complete. 2024-04-06T12:25:09.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.670+0000 7f6684a9a700 1 Processor -- start 2024-04-06T12:25:09.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.670+0000 7f6684a9a700 1 -- start start 2024-04-06T12:25:09.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.671+0000 7f6684a9a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f668006ac70 0x7f66801f6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:09.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.671+0000 7f6684a9a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6680158fe0 0x7f66801415f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:09.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.671+0000 7f6684a9a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f668015c2d0 0x7f6680141b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:09.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.671+0000 7f6684a9a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66801f2be0 con 0x7f668006ac70 2024-04-06T12:25:09.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.671+0000 7f6684a9a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66801f3bb0 con 0x7f6680158fe0 2024-04-06T12:25:09.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.671+0000 7f6684a9a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66801f2df0 con 0x7f668015c2d0 2024-04-06T12:25:09.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.671+0000 7f667e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f668006ac70 0x7f66801f6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:09.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.671+0000 7f667e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f668006ac70 0x7f66801f6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38694/0 (socket says 172.21.15.189:38694) 2024-04-06T12:25:09.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.671+0000 7f667e59c700 1 -- 172.21.15.189:0/450822215 learned_addr learned my addr 172.21.15.189:0/450822215 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:09.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.671+0000 7f667dd9b700 1 --2- 172.21.15.189:0/450822215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6680158fe0 0x7f66801415f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:09.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.671+0000 7f667ed9d700 1 --2- 172.21.15.189:0/450822215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f668015c2d0 0x7f6680141b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:09.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.671+0000 7f667e59c700 1 -- 172.21.15.189:0/450822215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f668015c2d0 msgr2=0x7f6680141b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:09.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.671+0000 7f667e59c700 1 --2- 172.21.15.189:0/450822215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f668015c2d0 0x7f6680141b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:09.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.671+0000 7f667e59c700 1 -- 172.21.15.189:0/450822215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6680158fe0 msgr2=0x7f66801415f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:09.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.671+0000 7f667e59c700 1 --2- 172.21.15.189:0/450822215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6680158fe0 0x7f66801415f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:09.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.671+0000 7f667e59c700 1 -- 172.21.15.189:0/450822215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6670009550 con 0x7f668006ac70 2024-04-06T12:25:09.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.671+0000 7f667e59c700 1 --2- 172.21.15.189:0/450822215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f668006ac70 0x7f66801f6920 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f668006c820 tx=0x7f6670004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:09.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.672+0000 7f666effd700 1 -- 172.21.15.189:0/450822215 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670004480 con 0x7f668006ac70 2024-04-06T12:25:09.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.672+0000 7f666effd700 1 -- 172.21.15.189:0/450822215 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667001f070 con 0x7f668006ac70 2024-04-06T12:25:09.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.672+0000 7f6684a9a700 1 -- 172.21.15.189:0/450822215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66801f30d0 con 0x7f668006ac70 2024-04-06T12:25:09.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.672+0000 7f666effd700 1 -- 172.21.15.189:0/450822215 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66700038c0 con 0x7f668006ac70 2024-04-06T12:25:09.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.672+0000 7f6684a9a700 1 -- 172.21.15.189:0/450822215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66801f3690 con 0x7f668006ac70 2024-04-06T12:25:09.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.673+0000 7f6684a9a700 1 -- 172.21.15.189:0/450822215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f668015fe50 con 0x7f668006ac70 2024-04-06T12:25:09.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.673+0000 7f666effd700 1 -- 172.21.15.189:0/450822215 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f6670021030 con 0x7f668006ac70 2024-04-06T12:25:09.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.673+0000 7f666effd700 1 --2- 172.21.15.189:0/450822215 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006adb0 0x7f666006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:09.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.674+0000 7f667dd9b700 1 --2- 172.21.15.189:0/450822215 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006adb0 0x7f666006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:09.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.674+0000 7f666effd700 1 -- 172.21.15.189:0/450822215 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f667008a9c0 con 0x7f668006ac70 2024-04-06T12:25:09.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.674+0000 7f667dd9b700 1 --2- 172.21.15.189:0/450822215 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006adb0 0x7f666006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f66800af5f0 tx=0x7f6674009040 comp rx=0 tx=0).ready entity=mgr.5225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:09.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.678+0000 7f666effd700 1 -- 172.21.15.189:0/450822215 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f668015fe50 con 0x7f668006ac70 2024-04-06T12:25:09.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:09.930+0000 7f6684a9a700 1 -- 172.21.15.189:0/450822215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f668015fe50 con 0x7f668006ac70 2024-04-06T12:25:10.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.514+0000 7f666effd700 1 -- 172.21.15.189:0/450822215 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 20) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f6670055db0 con 0x7f668006ac70 2024-04-06T12:25:10.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.515+0000 7f666effd700 1 -- 172.21.15.189:0/450822215 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v20) v1 ==== 92+0+0 (secure 0 0 0) 0x7f66700705f0 con 0x7f668006ac70 2024-04-06T12:25:10.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.518+0000 7f6684a9a700 1 -- 172.21.15.189:0/450822215 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006adb0 msgr2=0x7f666006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:10.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.518+0000 7f6684a9a700 1 --2- 172.21.15.189:0/450822215 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006adb0 0x7f666006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f66800af5f0 tx=0x7f6674009040 comp rx=0 tx=0).stop 2024-04-06T12:25:10.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.518+0000 7f6684a9a700 1 -- 172.21.15.189:0/450822215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f668006ac70 msgr2=0x7f66801f6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:10.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.518+0000 7f6684a9a700 1 --2- 172.21.15.189:0/450822215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f668006ac70 0x7f66801f6920 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f668006c820 tx=0x7f6670004750 comp rx=0 tx=0).stop 2024-04-06T12:25:10.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.518+0000 7f6684a9a700 1 -- 172.21.15.189:0/450822215 shutdown_connections 2024-04-06T12:25:10.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.518+0000 7f6684a9a700 1 --2- 172.21.15.189:0/450822215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f668015c2d0 0x7f6680141b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:10.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.518+0000 7f6684a9a700 1 --2- 172.21.15.189:0/450822215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f668006ac70 0x7f66801f6920 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:10.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.518+0000 7f6684a9a700 1 --2- 172.21.15.189:0/450822215 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006adb0 0x7f666006d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:10.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.518+0000 7f6684a9a700 1 --2- 172.21.15.189:0/450822215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6680158fe0 0x7f66801415f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:10.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.518+0000 7f6684a9a700 1 -- 172.21.15.189:0/450822215 >> 172.21.15.189:0/450822215 conn(0x7f66800d00b0 msgr2=0x7f66800d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:10.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.518+0000 7f6684a9a700 1 -- 172.21.15.189:0/450822215 shutdown_connections 2024-04-06T12:25:10.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.518+0000 7f6684a9a700 1 -- 172.21.15.189:0/450822215 wait complete. 2024-04-06T12:25:10.529 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-06T12:25:10.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.661+0000 7f6bbddc9700 1 Processor -- start 2024-04-06T12:25:10.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.661+0000 7f6bbddc9700 1 -- start start 2024-04-06T12:25:10.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.662+0000 7f6bbddc9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6bb80d11b0 0x7f6bb80d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:10.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.662+0000 7f6bbddc9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6bb80cb870 0x7f6bb815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:10.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.662+0000 7f6bbddc9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bb80cfa00 0x7f6bb80cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:10.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.662+0000 7f6bbddc9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb80ccdc0 con 0x7f6bb80d11b0 2024-04-06T12:25:10.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.662+0000 7f6bbddc9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb80d0b50 con 0x7f6bb80cb870 2024-04-06T12:25:10.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.662+0000 7f6bbddc9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb80d1ad0 con 0x7f6bb80cfa00 2024-04-06T12:25:10.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.662+0000 7f6bb77fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6bb80d11b0 0x7f6bb80d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:10.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.662+0000 7f6bb77fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6bb80d11b0 0x7f6bb80d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38728/0 (socket says 172.21.15.189:38728) 2024-04-06T12:25:10.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.662+0000 7f6bb77fe700 1 -- 172.21.15.189:0/2932019825 learned_addr learned my addr 172.21.15.189:0/2932019825 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:10.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.662+0000 7f6bb7fff700 1 --2- 172.21.15.189:0/2932019825 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bb80cfa00 0x7f6bb80cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:10.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.662+0000 7f6bb6ffd700 1 --2- 172.21.15.189:0/2932019825 >> v2:172.21.15.192:3300/0 conn(0x7f6bb80cb870 0x7f6bb815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:10.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.663+0000 7f6bb77fe700 1 -- 172.21.15.189:0/2932019825 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bb80cfa00 msgr2=0x7f6bb80cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:10.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.663+0000 7f6bb77fe700 1 --2- 172.21.15.189:0/2932019825 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bb80cfa00 0x7f6bb80cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:10.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.663+0000 7f6bb77fe700 1 -- 172.21.15.189:0/2932019825 >> v2:172.21.15.192:3300/0 conn(0x7f6bb80cb870 msgr2=0x7f6bb815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:10.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.663+0000 7f6bb77fe700 1 --2- 172.21.15.189:0/2932019825 >> v2:172.21.15.192:3300/0 conn(0x7f6bb80cb870 0x7f6bb815c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:10.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.663+0000 7f6bb77fe700 1 -- 172.21.15.189:0/2932019825 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb80cbd80 con 0x7f6bb80d11b0 2024-04-06T12:25:10.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.663+0000 7f6bb6ffd700 1 --2- 172.21.15.189:0/2932019825 >> v2:172.21.15.192:3300/0 conn(0x7f6bb80cb870 0x7f6bb815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:25:10.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.663+0000 7f6bb7fff700 1 --2- 172.21.15.189:0/2932019825 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bb80cfa00 0x7f6bb80cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:25:10.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.663+0000 7f6bb77fe700 1 --2- 172.21.15.189:0/2932019825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bb80d11b0 0x7f6bb80d1590 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f6bb804fb40 tx=0x7f6ba800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67d4691ab530a3e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:10.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.663+0000 7f6bb67fc700 1 -- 172.21.15.189:0/2932019825 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba800e070 con 0x7f6bb80d11b0 2024-04-06T12:25:10.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.663+0000 7f6bb67fc700 1 -- 172.21.15.189:0/2932019825 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba800b9c0 con 0x7f6bb80d11b0 2024-04-06T12:25:10.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.663+0000 7f6bb67fc700 1 -- 172.21.15.189:0/2932019825 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba800bd70 con 0x7f6bb80d11b0 2024-04-06T12:25:10.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.663+0000 7f6bbddc9700 1 -- 172.21.15.189:0/2932019825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bb80d11b0 msgr2=0x7f6bb80d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:10.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.663+0000 7f6bbddc9700 1 --2- 172.21.15.189:0/2932019825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bb80d11b0 0x7f6bb80d1590 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f6bb804fb40 tx=0x7f6ba800b330 comp rx=0 tx=0).stop 2024-04-06T12:25:10.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.663+0000 7f6bbddc9700 1 -- 172.21.15.189:0/2932019825 shutdown_connections 2024-04-06T12:25:10.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.663+0000 7f6bbddc9700 1 --2- 172.21.15.189:0/2932019825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bb80d11b0 0x7f6bb80d1590 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:10.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.663+0000 7f6bbddc9700 1 --2- 172.21.15.189:0/2932019825 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bb80cfa00 0x7f6bb80cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:10.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.663+0000 7f6bbddc9700 1 --2- 172.21.15.189:0/2932019825 >> v2:172.21.15.192:3300/0 conn(0x7f6bb80cb870 0x7f6bb815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:10.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.663+0000 7f6bbddc9700 1 -- 172.21.15.189:0/2932019825 >> 172.21.15.189:0/2932019825 conn(0x7f6bb80b2db0 msgr2=0x7f6bb80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:10.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.664+0000 7f6bbddc9700 1 -- 172.21.15.189:0/2932019825 shutdown_connections 2024-04-06T12:25:10.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.664+0000 7f6bbddc9700 1 -- 172.21.15.189:0/2932019825 wait complete. 2024-04-06T12:25:10.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.665+0000 7f6bbddc9700 1 Processor -- start 2024-04-06T12:25:10.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.665+0000 7f6bbddc9700 1 -- start start 2024-04-06T12:25:10.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bbddc9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bb80cb870 0x7f6bb8156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:10.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bbddc9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bb80cfa00 0x7f6bb8158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:10.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bbddc9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bb80d11b0 0x7f6bb81552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:10.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bbddc9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb81f50d0 con 0x7f6bb80cfa00 2024-04-06T12:25:10.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bbddc9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb81f4450 con 0x7f6bb80cb870 2024-04-06T12:25:10.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bbddc9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb81f4730 con 0x7f6bb80d11b0 2024-04-06T12:25:10.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bb6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bb80cfa00 0x7f6bb8158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:10.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bb77fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bb80cb870 0x7f6bb8156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:10.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bb6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bb80cfa00 0x7f6bb8158940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38734/0 (socket says 172.21.15.189:38734) 2024-04-06T12:25:10.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bb77fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bb80cb870 0x7f6bb8156be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:43646/0 (socket says 172.21.15.189:43646) 2024-04-06T12:25:10.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bb6ffd700 1 -- 172.21.15.189:0/1997038709 learned_addr learned my addr 172.21.15.189:0/1997038709 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:10.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bb7fff700 1 --2- 172.21.15.189:0/1997038709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bb80d11b0 0x7f6bb81552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:10.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bb6ffd700 1 -- 172.21.15.189:0/1997038709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bb80d11b0 msgr2=0x7f6bb81552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:10.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bb6ffd700 1 --2- 172.21.15.189:0/1997038709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bb80d11b0 0x7f6bb81552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:10.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bb6ffd700 1 -- 172.21.15.189:0/1997038709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bb80cb870 msgr2=0x7f6bb8156be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:10.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bb6ffd700 1 --2- 172.21.15.189:0/1997038709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bb80cb870 0x7f6bb8156be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:10.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bb6ffd700 1 -- 172.21.15.189:0/1997038709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba8009b30 con 0x7f6bb80cfa00 2024-04-06T12:25:10.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bb77fe700 1 --2- 172.21.15.189:0/1997038709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bb80cb870 0x7f6bb8156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:25:10.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bb6ffd700 1 --2- 172.21.15.189:0/1997038709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bb80cfa00 0x7f6bb8158940 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f6bac00d7d0 tx=0x7f6bac00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:10.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6b9ffff700 1 -- 172.21.15.189:0/1997038709 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac004030 con 0x7f6bb80cfa00 2024-04-06T12:25:10.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6bbddc9700 1 -- 172.21.15.189:0/1997038709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb81f4a10 con 0x7f6bb80cfa00 2024-04-06T12:25:10.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6b9ffff700 1 -- 172.21.15.189:0/1997038709 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bac004190 con 0x7f6bb80cfa00 2024-04-06T12:25:10.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.666+0000 7f6b9ffff700 1 -- 172.21.15.189:0/1997038709 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac004500 con 0x7f6bb80cfa00 2024-04-06T12:25:10.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.667+0000 7f6bbddc9700 1 -- 172.21.15.189:0/1997038709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb8156220 con 0x7f6bb80cfa00 2024-04-06T12:25:10.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.667+0000 7f6bbddc9700 1 -- 172.21.15.189:0/1997038709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb8074990 con 0x7f6bb80cfa00 2024-04-06T12:25:10.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.668+0000 7f6b9ffff700 1 -- 172.21.15.189:0/1997038709 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f6bac0036a0 con 0x7f6bb80cfa00 2024-04-06T12:25:10.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.668+0000 7f6b9ffff700 1 --2- 172.21.15.189:0/1997038709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ba406ae00 0x7f6ba406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:10.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.668+0000 7f6bb77fe700 1 -- 172.21.15.189:0/1997038709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ba406ae00 msgr2=0x7f6ba406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:25:10.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.668+0000 7f6b9ffff700 1 -- 172.21.15.189:0/1997038709 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6bac0892b0 con 0x7f6bb80cfa00 2024-04-06T12:25:10.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.668+0000 7f6bb77fe700 1 --2- 172.21.15.189:0/1997038709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ba406ae00 0x7f6ba406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:25:10.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.673+0000 7f6b9ffff700 1 -- 172.21.15.189:0/1997038709 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6bac054250 con 0x7f6bb80cfa00 2024-04-06T12:25:10.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.869+0000 7f6bb77fe700 1 -- 172.21.15.189:0/1997038709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ba406ae00 msgr2=0x7f6ba406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:25:10.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.869+0000 7f6bb77fe700 1 --2- 172.21.15.189:0/1997038709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ba406ae00 0x7f6ba406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:25:10.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.877+0000 7f6bbddc9700 1 -- 172.21.15.189:0/1997038709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f6bb80b0a00 con 0x7f6bb80cfa00 2024-04-06T12:25:10.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.881+0000 7f6b9ffff700 1 -- 172.21.15.189:0/1997038709 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v17)=0 v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6bac057870 con 0x7f6bb80cfa00 2024-04-06T12:25:10.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.884+0000 7f6bbddc9700 1 -- 172.21.15.189:0/1997038709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ba406ae00 msgr2=0x7f6ba406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:25:10.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.884+0000 7f6bbddc9700 1 --2- 172.21.15.189:0/1997038709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ba406ae00 0x7f6ba406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:10.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.884+0000 7f6bbddc9700 1 -- 172.21.15.189:0/1997038709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bb80cfa00 msgr2=0x7f6bb8158940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:10.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.884+0000 7f6bbddc9700 1 --2- 172.21.15.189:0/1997038709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bb80cfa00 0x7f6bb8158940 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f6bac00d7d0 tx=0x7f6bac00dae0 comp rx=0 tx=0).stop 2024-04-06T12:25:10.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.884+0000 7f6bbddc9700 1 -- 172.21.15.189:0/1997038709 shutdown_connections 2024-04-06T12:25:10.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.884+0000 7f6bbddc9700 1 --2- 172.21.15.189:0/1997038709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bb80d11b0 0x7f6bb81552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:10.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.884+0000 7f6bbddc9700 1 --2- 172.21.15.189:0/1997038709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ba406ae00 0x7f6ba406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:10.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.884+0000 7f6bbddc9700 1 --2- 172.21.15.189:0/1997038709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bb80cb870 0x7f6bb8156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:10.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.884+0000 7f6bbddc9700 1 --2- 172.21.15.189:0/1997038709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bb80cfa00 0x7f6bb8158940 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:10.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.884+0000 7f6bbddc9700 1 -- 172.21.15.189:0/1997038709 >> 172.21.15.189:0/1997038709 conn(0x7f6bb80b2db0 msgr2=0x7f6bb815a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:10.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.884+0000 7f6bbddc9700 1 -- 172.21.15.189:0/1997038709 shutdown_connections 2024-04-06T12:25:10.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:10.884+0000 7f6bbddc9700 1 -- 172.21.15.189:0/1997038709 wait complete. 2024-04-06T12:25:10.889 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:10.891+0000 7f238f529200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:25:10.894 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-06T12:25:10.908 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:10.911+0000 7f6db0024200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:25:10.967 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:10.968+0000 7f238f529200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:25:10.991 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:10.994+0000 7f6db0024200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:25:11.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.025+0000 7fe38ecac700 1 Processor -- start 2024-04-06T12:25:11.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.025+0000 7fe38ecac700 1 -- start start 2024-04-06T12:25:11.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.026+0000 7fe38ecac700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3880d11b0 0x7fe3880d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:11.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.026+0000 7fe38ecac700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe3880cfa00 0x7fe3880cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:11.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.026+0000 7fe38ecac700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3880d1ad0 con 0x7fe3880cb870 2024-04-06T12:25:11.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.026+0000 7fe38ecac700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3880ccdc0 con 0x7fe3880cfa00 2024-04-06T12:25:11.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.026+0000 7fe38ecac700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3880d0b50 con 0x7fe3880d11b0 2024-04-06T12:25:11.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.026+0000 7fe38d249700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe3880cfa00 0x7fe3880cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:11.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.026+0000 7fe38d249700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe3880cfa00 0x7fe3880cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38746/0 (socket says 172.21.15.189:38746) 2024-04-06T12:25:11.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.026+0000 7fe38d249700 1 -- 172.21.15.189:0/2099216988 learned_addr learned my addr 172.21.15.189:0/2099216988 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:11.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.026+0000 7fe38ca48700 1 --2- 172.21.15.189:0/2099216988 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3880d11b0 0x7fe3880d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:11.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.027+0000 7fe38d249700 1 -- 172.21.15.189:0/2099216988 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3880d11b0 msgr2=0x7fe3880d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:11.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.027+0000 7fe38d249700 1 --2- 172.21.15.189:0/2099216988 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3880d11b0 0x7fe3880d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:11.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.027+0000 7fe38d249700 1 -- 172.21.15.189:0/2099216988 >> v1:172.21.15.192:6789/0 conn(0x7fe3880cb870 legacy=0x7fe38815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:11.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.027+0000 7fe38d249700 1 -- 172.21.15.189:0/2099216988 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3880cbd80 con 0x7fe3880cfa00 2024-04-06T12:25:11.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.027+0000 7fe3877fe700 1 -- 172.21.15.189:0/2099216988 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536116431 0 0) 0x7fe3880d1ad0 con 0x7fe3880cb870 2024-04-06T12:25:11.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.027+0000 7fe38d249700 1 --2- 172.21.15.189:0/2099216988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3880cfa00 0x7fe3880cd560 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fe380009e50 tx=0x7fe38000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f8b135269188802 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:11.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.027+0000 7fe3877fe700 1 -- 172.21.15.189:0/2099216988 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38000e070 con 0x7fe3880cfa00 2024-04-06T12:25:11.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.027+0000 7fe3877fe700 1 -- 172.21.15.189:0/2099216988 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38000b910 con 0x7fe3880cfa00 2024-04-06T12:25:11.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.027+0000 7fe3877fe700 1 -- 172.21.15.189:0/2099216988 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38000bcc0 con 0x7fe3880cfa00 2024-04-06T12:25:11.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.027+0000 7fe38ecac700 1 -- 172.21.15.189:0/2099216988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3880cfa00 msgr2=0x7fe3880cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:11.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.028+0000 7fe38ecac700 1 --2- 172.21.15.189:0/2099216988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3880cfa00 0x7fe3880cd560 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fe380009e50 tx=0x7fe38000b330 comp rx=0 tx=0).stop 2024-04-06T12:25:11.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.028+0000 7fe38ecac700 1 -- 172.21.15.189:0/2099216988 shutdown_connections 2024-04-06T12:25:11.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.028+0000 7fe38ecac700 1 --2- 172.21.15.189:0/2099216988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3880cfa00 0x7fe3880cd560 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:11.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.028+0000 7fe38ecac700 1 --2- 172.21.15.189:0/2099216988 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3880d11b0 0x7fe3880d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:11.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.028+0000 7fe38ecac700 1 -- 172.21.15.189:0/2099216988 >> 172.21.15.189:0/2099216988 conn(0x7fe3880b2db0 msgr2=0x7fe3880b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:11.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.028+0000 7fe38ecac700 1 -- 172.21.15.189:0/2099216988 shutdown_connections 2024-04-06T12:25:11.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.028+0000 7fe38ecac700 1 -- 172.21.15.189:0/2099216988 wait complete. 2024-04-06T12:25:11.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.029+0000 7fe38ecac700 1 Processor -- start 2024-04-06T12:25:11.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.029+0000 7fe38ecac700 1 -- start start 2024-04-06T12:25:11.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe38ecac700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3880cb870 0x7fe3881552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:11.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe38ecac700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3880cfa00 0x7fe3880c6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:11.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe38ecac700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3880d11b0 0x7fe3880c7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:11.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe38ecac700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3880c8d70 con 0x7fe3880d11b0 2024-04-06T12:25:11.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe38ecac700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3881557f0 con 0x7fe3880cfa00 2024-04-06T12:25:11.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe38ecac700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe38806d790 con 0x7fe3880cb870 2024-04-06T12:25:11.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe38d249700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3880d11b0 0x7fe3880c7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:11.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe38ca48700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3880cb870 0x7fe3881552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:11.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe38ca48700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3880cb870 0x7fe3881552b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:53834/0 (socket says 172.21.15.189:53834) 2024-04-06T12:25:11.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe38d249700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3880d11b0 0x7fe3880c7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38760/0 (socket says 172.21.15.189:38760) 2024-04-06T12:25:11.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe387fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3880cfa00 0x7fe3880c6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:11.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe38ca48700 1 -- 172.21.15.189:0/2713063535 learned_addr learned my addr 172.21.15.189:0/2713063535 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:11.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe38ca48700 1 -- 172.21.15.189:0/2713063535 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3880cfa00 msgr2=0x7fe3880c6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:11.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe38ca48700 1 --2- 172.21.15.189:0/2713063535 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3880cfa00 0x7fe3880c6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:11.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe38ca48700 1 -- 172.21.15.189:0/2713063535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3880d11b0 msgr2=0x7fe3880c7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:11.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe38ca48700 1 --2- 172.21.15.189:0/2713063535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3880d11b0 0x7fe3880c7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:11.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe38ca48700 1 -- 172.21.15.189:0/2713063535 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe380009b30 con 0x7fe3880cb870 2024-04-06T12:25:11.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe38d249700 1 --2- 172.21.15.189:0/2713063535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3880d11b0 0x7fe3880c7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:25:11.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe38ca48700 1 --2- 172.21.15.189:0/2713063535 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3880cb870 0x7fe3881552b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe37c00e870 tx=0x7fe37c00eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:11.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe3857fa700 1 -- 172.21.15.189:0/2713063535 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c009590 con 0x7fe3880cb870 2024-04-06T12:25:11.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.030+0000 7fe3857fa700 1 -- 172.21.15.189:0/2713063535 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe37c013680 con 0x7fe3880cb870 2024-04-06T12:25:11.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.031+0000 7fe38ecac700 1 -- 172.21.15.189:0/2713063535 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3881f6260 con 0x7fe3880cb870 2024-04-06T12:25:11.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.031+0000 7fe3857fa700 1 -- 172.21.15.189:0/2713063535 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c0139f0 con 0x7fe3880cb870 2024-04-06T12:25:11.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.031+0000 7fe38ecac700 1 -- 172.21.15.189:0/2713063535 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3881f6820 con 0x7fe3880cb870 2024-04-06T12:25:11.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.031+0000 7fe38ecac700 1 -- 172.21.15.189:0/2713063535 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3880cf4c0 con 0x7fe3880cb870 2024-04-06T12:25:11.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.032+0000 7fe3857fa700 1 -- 172.21.15.189:0/2713063535 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 20) v1 ==== 89264+0+0 (secure 0 0 0) 0x7fe37c013b50 con 0x7fe3880cb870 2024-04-06T12:25:11.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.032+0000 7fe3857fa700 1 --2- 172.21.15.189:0/2713063535 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe36c06ae00 0x7fe36c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:11.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.032+0000 7fe387fff700 1 -- 172.21.15.189:0/2713063535 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe36c06ae00 msgr2=0x7fe36c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:25:11.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.032+0000 7fe3857fa700 1 -- 172.21.15.189:0/2713063535 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe37c0894f0 con 0x7fe3880cb870 2024-04-06T12:25:11.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.032+0000 7fe387fff700 1 --2- 172.21.15.189:0/2713063535 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe36c06ae00 0x7fe36c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:25:11.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.037+0000 7fe3857fa700 1 -- 172.21.15.189:0/2713063535 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe3880cf4c0 con 0x7fe3880cb870 2024-04-06T12:25:11.040 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:11.042+0000 7f238f529200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:25:11.069 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:11.072+0000 7f6db0024200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:25:11.108 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:11.110+0000 7f238f529200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:25:11.140 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:11.143+0000 7f6db0024200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:25:11.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.233+0000 7fe387fff700 1 -- 172.21.15.189:0/2713063535 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe36c06ae00 msgr2=0x7fe36c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:25:11.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.233+0000 7fe387fff700 1 --2- 172.21.15.189:0/2713063535 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe36c06ae00 0x7fe36c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:25:11.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.242+0000 7fe38ecac700 1 -- 172.21.15.189:0/2713063535 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fe3880cf4c0 con 0x7fe3880cb870 2024-04-06T12:25:11.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.247+0000 7fe3857fa700 1 -- 172.21.15.189:0/2713063535 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v18)=0 v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe3880cf4c0 con 0x7fe3880cb870 2024-04-06T12:25:11.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.250+0000 7fe38ecac700 1 -- 172.21.15.189:0/2713063535 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe36c06ae00 msgr2=0x7fe36c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:25:11.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.250+0000 7fe38ecac700 1 --2- 172.21.15.189:0/2713063535 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe36c06ae00 0x7fe36c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:11.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.250+0000 7fe38ecac700 1 -- 172.21.15.189:0/2713063535 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3880cb870 msgr2=0x7fe3881552b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:11.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.250+0000 7fe38ecac700 1 --2- 172.21.15.189:0/2713063535 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3880cb870 0x7fe3881552b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe37c00e870 tx=0x7fe37c00eb80 comp rx=0 tx=0).stop 2024-04-06T12:25:11.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.250+0000 7fe38ecac700 1 -- 172.21.15.189:0/2713063535 shutdown_connections 2024-04-06T12:25:11.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.250+0000 7fe38ecac700 1 --2- 172.21.15.189:0/2713063535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3880d11b0 0x7fe3880c7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:11.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.250+0000 7fe38ecac700 1 --2- 172.21.15.189:0/2713063535 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3880cb870 0x7fe3881552b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:11.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.250+0000 7fe38ecac700 1 --2- 172.21.15.189:0/2713063535 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe36c06ae00 0x7fe36c06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:11.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.250+0000 7fe38ecac700 1 --2- 172.21.15.189:0/2713063535 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3880cfa00 0x7fe3880c6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:11.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.250+0000 7fe38ecac700 1 -- 172.21.15.189:0/2713063535 >> 172.21.15.189:0/2713063535 conn(0x7fe3880b2db0 msgr2=0x7fe3880cd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:11.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.250+0000 7fe38ecac700 1 -- 172.21.15.189:0/2713063535 shutdown_connections 2024-04-06T12:25:11.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.250+0000 7fe38ecac700 1 -- 172.21.15.189:0/2713063535 wait complete. 2024-04-06T12:25:11.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:11.260 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-06T12:25:11.284 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:11.285+0000 7f238f529200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:25:11.321 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-04-06T12:25:11.322 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:11.325+0000 7f6db0024200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:25:11.439 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:11.441+0000 7f238f529200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:25:11.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.451+0000 7f3cf1730700 1 Processor -- start 2024-04-06T12:25:11.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.451+0000 7f3cf1730700 1 -- start start 2024-04-06T12:25:11.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.452+0000 7f3cf1730700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3cec06ae40 0x7f3cec0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:11.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.452+0000 7f3cf1730700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3cec15ddf0 0x7f3cec15e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:11.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.452+0000 7f3cf1730700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cec0b3990 con 0x7f3cec15d7b0 2024-04-06T12:25:11.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.452+0000 7f3cf1730700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cec0b3060 con 0x7f3cec15ddf0 2024-04-06T12:25:11.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.452+0000 7f3cf1730700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec0b3340 con 0x7f3cec06ae40 2024-04-06T12:25:11.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.452+0000 7f3ceb7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3cec15ddf0 0x7f3cec15e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:11.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.452+0000 7f3ceaffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3cec15d7b0 0x7f3cec06a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46732/0 (socket says 172.21.15.189:46732) 2024-04-06T12:25:11.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.452+0000 7f3ceb7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3cec15ddf0 0x7f3cec15e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38764/0 (socket says 172.21.15.189:38764) 2024-04-06T12:25:11.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.452+0000 7f3ceaffd700 1 -- 172.21.15.189:0/2368188636 learned_addr learned my addr 172.21.15.189:0/2368188636 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:25:11.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.452+0000 7f3cea7fc700 1 --2- 172.21.15.189:0/2368188636 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3cec06ae40 0x7f3cec0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:11.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.452+0000 7f3ceb7fe700 1 -- 172.21.15.189:0/2368188636 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3cec06ae40 msgr2=0x7f3cec0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:11.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.452+0000 7f3ceb7fe700 1 --2- 172.21.15.189:0/2368188636 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3cec06ae40 0x7f3cec0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:11.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.452+0000 7f3ceb7fe700 1 -- 172.21.15.189:0/2368188636 >> v1:172.21.15.189:6789/0 conn(0x7f3cec15d7b0 legacy=0x7f3cec06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:11.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.452+0000 7f3ceb7fe700 1 -- 172.21.15.189:0/2368188636 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cec0b3620 con 0x7f3cec15ddf0 2024-04-06T12:25:11.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.452+0000 7f3ce9ffb700 1 -- 172.21.15.189:0/2368188636 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787789676 0 0) 0x7f3cec0b3990 con 0x7f3cec15d7b0 2024-04-06T12:25:11.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.453+0000 7f3ceb7fe700 1 --2- 172.21.15.189:0/2368188636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cec15ddf0 0x7f3cec15e2a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4009d90 tx=0x7f3ce400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=959c1aeb20e86def server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:11.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.453+0000 7f3ce9ffb700 1 -- 172.21.15.189:0/2368188636 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce400e070 con 0x7f3cec15ddf0 2024-04-06T12:25:11.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.453+0000 7f3ce9ffb700 1 -- 172.21.15.189:0/2368188636 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce400bb20 con 0x7f3cec15ddf0 2024-04-06T12:25:11.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.453+0000 7f3ce9ffb700 1 -- 172.21.15.189:0/2368188636 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4012430 con 0x7f3cec15ddf0 2024-04-06T12:25:11.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.453+0000 7f3cf1730700 1 -- 172.21.15.189:0/2368188636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cec15ddf0 msgr2=0x7f3cec15e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:11.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.453+0000 7f3cf1730700 1 --2- 172.21.15.189:0/2368188636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cec15ddf0 0x7f3cec15e2a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4009d90 tx=0x7f3ce400b330 comp rx=0 tx=0).stop 2024-04-06T12:25:11.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.453+0000 7f3cf1730700 1 -- 172.21.15.189:0/2368188636 shutdown_connections 2024-04-06T12:25:11.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.453+0000 7f3cf1730700 1 --2- 172.21.15.189:0/2368188636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cec15ddf0 0x7f3cec15e2a0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:11.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.453+0000 7f3cf1730700 1 --2- 172.21.15.189:0/2368188636 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3cec06ae40 0x7f3cec0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:11.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.453+0000 7f3cf1730700 1 -- 172.21.15.189:0/2368188636 >> 172.21.15.189:0/2368188636 conn(0x7f3cec0d00b0 msgr2=0x7f3cec0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:11.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.454+0000 7f3cf1730700 1 -- 172.21.15.189:0/2368188636 shutdown_connections 2024-04-06T12:25:11.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.454+0000 7f3cf1730700 1 -- 172.21.15.189:0/2368188636 wait complete. 2024-04-06T12:25:11.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.455+0000 7f3cf1730700 1 Processor -- start 2024-04-06T12:25:11.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.455+0000 7f3cf1730700 1 -- start start 2024-04-06T12:25:11.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.456+0000 7f3cf1730700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cec06ae40 0x7f3cec1f65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:11.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.456+0000 7f3cf1730700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3cec15d7b0 0x7f3cec0af590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:11.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.456+0000 7f3cf1730700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cec15ddf0 0x7f3cec14dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:11.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.456+0000 7f3cf1730700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec0cc4c0 con 0x7f3cec15ddf0 2024-04-06T12:25:11.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.456+0000 7f3cf1730700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec0aff00 con 0x7f3cec06ae40 2024-04-06T12:25:11.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.456+0000 7f3cf1730700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec0b01e0 con 0x7f3cec15d7b0 2024-04-06T12:25:11.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.456+0000 7f3ceb7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cec15ddf0 0x7f3cec14dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:11.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.456+0000 7f3ceb7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cec15ddf0 0x7f3cec14dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38776/0 (socket says 172.21.15.189:38776) 2024-04-06T12:25:11.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.456+0000 7f3ceb7fe700 1 -- 172.21.15.189:0/121505274 learned_addr learned my addr 172.21.15.189:0/121505274 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:11.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.456+0000 7f3cea7fc700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3cec15d7b0 0x7f3cec0af590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:11.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.456+0000 7f3ceaffd700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cec06ae40 0x7f3cec1f65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:11.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.456+0000 7f3ceb7fe700 1 -- 172.21.15.189:0/121505274 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3cec15d7b0 msgr2=0x7f3cec0af590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:11.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.456+0000 7f3ceb7fe700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3cec15d7b0 0x7f3cec0af590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:11.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.456+0000 7f3ceb7fe700 1 -- 172.21.15.189:0/121505274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cec06ae40 msgr2=0x7f3cec1f65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:11.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.456+0000 7f3ceb7fe700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cec06ae40 0x7f3cec1f65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:11.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.456+0000 7f3ceb7fe700 1 -- 172.21.15.189:0/121505274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce4009a70 con 0x7f3cec15ddf0 2024-04-06T12:25:11.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.456+0000 7f3ceb7fe700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cec15ddf0 0x7f3cec14dad0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4014040 tx=0x7f3ce4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:11.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.456+0000 7f3cd37fe700 1 -- 172.21.15.189:0/121505274 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4012430 con 0x7f3cec15ddf0 2024-04-06T12:25:11.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.456+0000 7f3cd37fe700 1 -- 172.21.15.189:0/121505274 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce4020070 con 0x7f3cec15ddf0 2024-04-06T12:25:11.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.457+0000 7f3cf1730700 1 -- 172.21.15.189:0/121505274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cec0b04c0 con 0x7f3cec15ddf0 2024-04-06T12:25:11.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.457+0000 7f3cd37fe700 1 -- 172.21.15.189:0/121505274 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4004280 con 0x7f3cec15ddf0 2024-04-06T12:25:11.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.457+0000 7f3cf1730700 1 -- 172.21.15.189:0/121505274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cec1f6b00 con 0x7f3cec15ddf0 2024-04-06T12:25:11.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.458+0000 7f3cf1730700 1 -- 172.21.15.189:0/121505274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cec0cc950 con 0x7f3cec15ddf0 2024-04-06T12:25:11.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.458+0000 7f3cd37fe700 1 -- 172.21.15.189:0/121505274 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f3ce40043e0 con 0x7f3cec15ddf0 2024-04-06T12:25:11.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.458+0000 7f3cd37fe700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cd806ae00 0x7f3cd806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:11.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.458+0000 7f3ceaffd700 1 -- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cd806ae00 msgr2=0x7f3cd806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:25:11.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.459+0000 7f3cd37fe700 1 -- 172.21.15.189:0/121505274 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3ce4022070 con 0x7f3cec15ddf0 2024-04-06T12:25:11.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.459+0000 7f3ceaffd700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cd806ae00 0x7f3cd806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:25:11.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.463+0000 7f3cd37fe700 1 -- 172.21.15.189:0/121505274 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3ce4056410 con 0x7f3cec15ddf0 2024-04-06T12:25:11.490 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:11.493+0000 7f6db0024200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:25:11.552 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:11.554+0000 7f238f529200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:25:11.603 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:11.606+0000 7f6db0024200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:25:11.633 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:11.635+0000 7f238f529200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:25:11.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.660+0000 7f3ceaffd700 1 -- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cd806ae00 msgr2=0x7f3cd806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:25:11.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.660+0000 7f3ceaffd700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cd806ae00 0x7f3cd806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:25:11.687 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:11.690+0000 7f6db0024200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:25:11.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:11.693+0000 7f3cf1730700 1 -- 172.21.15.189:0/121505274 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3cec15e7e0 con 0x7f3cd806ae00 2024-04-06T12:25:11.708 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:11.709+0000 7f238f529200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:25:11.762 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:11.765+0000 7f6db0024200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:25:11.910 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:11.911+0000 7f238f529200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:25:11.964 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:11.967+0000 7f6db0024200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:25:11.985 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:11.987+0000 7f238f529200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:25:12.039 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:12.042+0000 7f6db0024200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:25:12.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:12.061+0000 7f3ceaffd700 1 -- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cd806ae00 msgr2=0x7f3cd806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:25:12.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:12.061+0000 7f3ceaffd700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cd806ae00 0x7f3cd806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-06T12:25:12.174 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:25:12.177 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:25:12.307 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:12.308+0000 7f238f529200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:25:12.363 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:12.366+0000 7f6db0024200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:25:12.623 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:12.625+0000 7f238f529200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:25:12.687 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:12.690+0000 7f6db0024200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:25:12.782 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:12.784+0000 7f238f529200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:25:12.847 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:12.850+0000 7f6db0024200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:25:12.854 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:12.856+0000 7f238f529200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:25:12.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:12.862+0000 7f3ceaffd700 1 -- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cd806ae00 msgr2=0x7f3cd806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:25:12.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:12.862+0000 7f3ceaffd700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cd806ae00 0x7f3cd806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-06T12:25:12.921 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:12.924+0000 7f6db0024200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:25:13.462 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:13.464+0000 7f238f529200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:25:13.539 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:13.541+0000 7f238f529200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:25:13.544 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:13.547+0000 7f6db0024200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:25:13.622 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:13.625+0000 7f6db0024200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:25:13.663 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:13.664+0000 7f238f529200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:25:13.746 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:13.749+0000 7f6db0024200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:25:14.168 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:14.169+0000 7f238f529200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:25:14.254 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:14.256+0000 7f6db0024200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:25:14.319 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:14.320+0000 7f238f529200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:25:14.406 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:14.408+0000 7f6db0024200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:25:14.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:14.463+0000 7f3ceaffd700 1 -- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cd806ae00 msgr2=0x7f3cd806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:25:14.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:14.463+0000 7f3ceaffd700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cd806ae00 0x7f3cd806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-06T12:25:14.470 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:14.472+0000 7f238f529200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:25:14.558 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:14.561+0000 7f6db0024200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:25:15.185 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:15.188+0000 7f238f529200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:25:15.277 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:15.280+0000 7f6db0024200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:25:15.419 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:15.422+0000 7f238f529200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:25:15.494 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:15.497+0000 7f238f529200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:25:15.517 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:15.520+0000 7f6db0024200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:25:15.592 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:15.595+0000 7f6db0024200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:25:15.865 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:25:15.868+0000 7f238f529200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:25:15.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:15.939+0000 7f3cd37fe700 1 -- 172.21.15.189:0/121505274 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 21) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f3ce40567e0 con 0x7f3cec15ddf0 2024-04-06T12:25:15.966 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:15.969+0000 7f6db0024200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:25:15.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:15.991+0000 7f3cd37fe700 1 -- 172.21.15.189:0/121505274 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mgrmap(e 22) v1 ==== 88873+0+0 (secure 0 0 0) 0x7f3ce406df10 con 0x7f3cec15ddf0 2024-04-06T12:25:15.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:15.991+0000 7f3cd37fe700 1 -- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cd806ae00 msgr2=0x7f3cd806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:25:15.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:15.991+0000 7f3cd37fe700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cd806ae00 0x7f3cd806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:16.164 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:16.167+0000 7f6d25f38700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:25:16.164 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:16.167+0000 7f6d25f38700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:25:16.164 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:16.167+0000 7f6d25f38700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:25:16.164 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:16.167+0000 7f6d25f38700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:25:16.165 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:16.167+0000 7f6d25f38700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:25:16.165 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:16.167+0000 7f6d22f32700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:25:16.165 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:16.167+0000 7f6d22f32700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:25:16.165 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:16.167+0000 7f6d22f32700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:25:16.165 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:16.167+0000 7f6d22f32700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:25:16.165 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:25:16.167+0000 7f6d22f32700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:25:16.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:16.997+0000 7f3cd37fe700 1 -- 172.21.15.189:0/121505274 <== mon.0 v2:172.21.15.189:3300/0 9 ==== mgrmap(e 23) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f3ce405d650 con 0x7f3cec15ddf0 2024-04-06T12:25:16.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:16.998+0000 7f3cd37fe700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cd806ae00 0x7f3cd806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:16.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:16.998+0000 7f3cd37fe700 1 -- 172.21.15.189:0/121505274 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3cd801cd40 con 0x7f3cd806ae00 2024-04-06T12:25:16.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:16.999+0000 7f3ceaffd700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cd806ae00 0x7f3cd806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:16.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.001+0000 7f3ceaffd700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cd806ae00 0x7f3cd806d2c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0007fd0 tx=0x7f3ce0007de0 comp rx=0 tx=0).ready entity=mgr.6264 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:17.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.064+0000 7f3cd37fe700 1 -- 172.21.15.189:0/121505274 <== mgr.6264 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3cd801cd40 con 0x7f3cd806ae00 2024-04-06T12:25:17.061 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path snap_test_dir1 2024-04-06T12:25:17.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.066+0000 7f3cf1730700 1 -- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cd806ae00 msgr2=0x7f3cd806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:17.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.066+0000 7f3cf1730700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cd806ae00 0x7f3cd806d2c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0007fd0 tx=0x7f3ce0007de0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.066+0000 7f3cf1730700 1 -- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cec15ddf0 msgr2=0x7f3cec14dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:17.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.066+0000 7f3cf1730700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cec15ddf0 0x7f3cec14dad0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4014040 tx=0x7f3ce4004ad0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.066+0000 7f3cf1730700 1 -- 172.21.15.189:0/121505274 shutdown_connections 2024-04-06T12:25:17.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.066+0000 7f3cf1730700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cec15ddf0 0x7f3cec14dad0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.066+0000 7f3cf1730700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3cd806ae00 0x7f3cd806d2c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.066+0000 7f3cf1730700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cec06ae40 0x7f3cec1f65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.066+0000 7f3cf1730700 1 --2- 172.21.15.189:0/121505274 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3cec15d7b0 0x7f3cec0af590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.066+0000 7f3cf1730700 1 -- 172.21.15.189:0/121505274 >> 172.21.15.189:0/121505274 conn(0x7f3cec0d00b0 msgr2=0x7f3cec15c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:17.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.066+0000 7f3cf1730700 1 -- 172.21.15.189:0/121505274 shutdown_connections 2024-04-06T12:25:17.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.066+0000 7f3cf1730700 1 -- 172.21.15.189:0/121505274 wait complete. 2024-04-06T12:25:17.076 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-04-06T12:25:17.076 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 2m 2024-04-06T12:25:17.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.203+0000 7f081c60e700 1 Processor -- start 2024-04-06T12:25:17.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.203+0000 7f081c60e700 1 -- start start 2024-04-06T12:25:17.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.204+0000 7f081c60e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08140b4660 0x7f08140b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:17.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.204+0000 7f081c60e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08140cdbe0 con 0x7f081415a550 2024-04-06T12:25:17.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.204+0000 7f081c60e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08140b5000 con 0x7f08140b2fc0 2024-04-06T12:25:17.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.204+0000 7f081c60e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f081415a940 con 0x7f08140b4660 2024-04-06T12:25:17.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.204+0000 7f081a3aa700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f081415a550 0x7f08140cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46734/0 (socket says 172.21.15.189:46734) 2024-04-06T12:25:17.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.204+0000 7f081a3aa700 1 -- 172.21.15.189:0/925109862 learned_addr learned my addr 172.21.15.189:0/925109862 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:25:17.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.205+0000 7f0819ba9700 1 --2- 172.21.15.189:0/925109862 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08140b4660 0x7f08140b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:17.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.205+0000 7f08193a8700 1 -- 172.21.15.189:0/925109862 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185096202 0 0) 0x7f08140cdbe0 con 0x7f081415a550 2024-04-06T12:25:17.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.205+0000 7f08193a8700 1 -- 172.21.15.189:0/925109862 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08000035b0 con 0x7f081415a550 2024-04-06T12:25:17.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.205+0000 7f08193a8700 1 -- 172.21.15.189:0/925109862 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611695887 0 0) 0x7f08140b5000 con 0x7f08140b2fc0 2024-04-06T12:25:17.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.205+0000 7f08193a8700 1 -- 172.21.15.189:0/925109862 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08140cdbe0 con 0x7f08140b2fc0 2024-04-06T12:25:17.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.205+0000 7f0819ba9700 1 -- 172.21.15.189:0/925109862 >> v1:172.21.15.192:6789/0 conn(0x7f08140b2fc0 legacy=0x7f08140b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:17.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.205+0000 7f0819ba9700 1 -- 172.21.15.189:0/925109862 >> v1:172.21.15.189:6789/0 conn(0x7f081415a550 legacy=0x7f08140cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:17.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.205+0000 7f0819ba9700 1 -- 172.21.15.189:0/925109862 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081415ac00 con 0x7f08140b4660 2024-04-06T12:25:17.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.205+0000 7f0819ba9700 1 --2- 172.21.15.189:0/925109862 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08140b4660 0x7f08140b4ac0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0810009870 tx=0x7f0810009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a888a810a5a7658 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:17.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.205+0000 7f08193a8700 1 -- 172.21.15.189:0/925109862 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0810011830 con 0x7f08140b4660 2024-04-06T12:25:17.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.205+0000 7f08193a8700 1 -- 172.21.15.189:0/925109862 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0810011990 con 0x7f08140b4660 2024-04-06T12:25:17.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.205+0000 7f08193a8700 1 -- 172.21.15.189:0/925109862 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0810011d20 con 0x7f08140b4660 2024-04-06T12:25:17.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.206+0000 7f081c60e700 1 -- 172.21.15.189:0/925109862 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08140b4660 msgr2=0x7f08140b4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:17.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.206+0000 7f081c60e700 1 --2- 172.21.15.189:0/925109862 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08140b4660 0x7f08140b4ac0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0810009870 tx=0x7f0810009b80 comp rx=0 tx=0).stop 2024-04-06T12:25:17.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.206+0000 7f081c60e700 1 -- 172.21.15.189:0/925109862 shutdown_connections 2024-04-06T12:25:17.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.206+0000 7f081c60e700 1 --2- 172.21.15.189:0/925109862 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08140b4660 0x7f08140b4ac0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.206+0000 7f081c60e700 1 -- 172.21.15.189:0/925109862 >> 172.21.15.189:0/925109862 conn(0x7f08140d00b0 msgr2=0x7f08140d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:17.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.206+0000 7f081c60e700 1 -- 172.21.15.189:0/925109862 shutdown_connections 2024-04-06T12:25:17.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.206+0000 7f081c60e700 1 -- 172.21.15.189:0/925109862 wait complete. 2024-04-06T12:25:17.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.207+0000 7f081c60e700 1 Processor -- start 2024-04-06T12:25:17.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.208+0000 7f081c60e700 1 -- start start 2024-04-06T12:25:17.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.208+0000 7f081c60e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08140b2fc0 0x7f0814152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:17.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.208+0000 7f081c60e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08140b4660 0x7f08140aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:17.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.208+0000 7f081c60e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08141f2ae0 0x7f08141ef190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:17.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.208+0000 7f081c60e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08141eed70 con 0x7f08140b4660 2024-04-06T12:25:17.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.208+0000 7f081c60e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08141ee9d0 con 0x7f08140b2fc0 2024-04-06T12:25:17.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.208+0000 7f081c60e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0814155670 con 0x7f08141f2ae0 2024-04-06T12:25:17.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.208+0000 7f0819ba9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08140b4660 0x7f08140aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:17.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.209+0000 7f0819ba9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08140b4660 0x7f08140aec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38892/0 (socket says 172.21.15.189:38892) 2024-04-06T12:25:17.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.209+0000 7f0819ba9700 1 -- 172.21.15.189:0/1784778143 learned_addr learned my addr 172.21.15.189:0/1784778143 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:17.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.209+0000 7f081a3aa700 1 --2- 172.21.15.189:0/1784778143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08140b2fc0 0x7f0814152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:17.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.209+0000 7f081abab700 1 --2- 172.21.15.189:0/1784778143 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08141f2ae0 0x7f08141ef190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:17.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.209+0000 7f0819ba9700 1 -- 172.21.15.189:0/1784778143 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08141f2ae0 msgr2=0x7f08141ef190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:17.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.209+0000 7f0819ba9700 1 --2- 172.21.15.189:0/1784778143 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08141f2ae0 0x7f08141ef190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.209+0000 7f0819ba9700 1 -- 172.21.15.189:0/1784778143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08140b2fc0 msgr2=0x7f0814152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:17.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.209+0000 7f0819ba9700 1 --2- 172.21.15.189:0/1784778143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08140b2fc0 0x7f0814152350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.209+0000 7f0819ba9700 1 -- 172.21.15.189:0/1784778143 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08100094a0 con 0x7f08140b4660 2024-04-06T12:25:17.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.209+0000 7f081a3aa700 1 --2- 172.21.15.189:0/1784778143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08140b2fc0 0x7f0814152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:25:17.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.209+0000 7f0819ba9700 1 --2- 172.21.15.189:0/1784778143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08140b4660 0x7f08140aec60 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f08100090b0 tx=0x7f0810004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:17.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.209+0000 7f080affd700 1 -- 172.21.15.189:0/1784778143 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081001a860 con 0x7f08140b4660 2024-04-06T12:25:17.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.209+0000 7f080affd700 1 -- 172.21.15.189:0/1784778143 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f081001a9c0 con 0x7f08140b4660 2024-04-06T12:25:17.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.209+0000 7f081c60e700 1 -- 172.21.15.189:0/1784778143 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0814155950 con 0x7f08140b4660 2024-04-06T12:25:17.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.209+0000 7f080affd700 1 -- 172.21.15.189:0/1784778143 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081001ad30 con 0x7f08140b4660 2024-04-06T12:25:17.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.209+0000 7f081c60e700 1 -- 172.21.15.189:0/1784778143 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0814155c30 con 0x7f08140b4660 2024-04-06T12:25:17.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.210+0000 7f081c60e700 1 -- 172.21.15.189:0/1784778143 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08141eeeb0 con 0x7f08140b4660 2024-04-06T12:25:17.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.211+0000 7f080affd700 1 -- 172.21.15.189:0/1784778143 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 23) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f0810022070 con 0x7f08140b4660 2024-04-06T12:25:17.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.211+0000 7f080affd700 1 --2- 172.21.15.189:0/1784778143 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f07f806ab30 0x7f07f806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:17.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.211+0000 7f080affd700 1 -- 172.21.15.189:0/1784778143 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f081008a3c0 con 0x7f08140b4660 2024-04-06T12:25:17.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.211+0000 7f081a3aa700 1 --2- 172.21.15.189:0/1784778143 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f07f806ab30 0x7f07f806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:17.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.211+0000 7f081a3aa700 1 --2- 172.21.15.189:0/1784778143 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f07f806ab30 0x7f07f806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0804000f60 tx=0x7f0804009040 comp rx=0 tx=0).ready entity=mgr.6264 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:17.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.216+0000 7f080affd700 1 -- 172.21.15.189:0/1784778143 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f08141eeeb0 con 0x7f08140b4660 2024-04-06T12:25:17.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.435+0000 7f081c60e700 1 -- 172.21.15.189:0/1784778143 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "2m", "target": ["mon-mgr", ""]}) v1 -- 0x7f081415b4a0 con 0x7f07f806ab30 2024-04-06T12:25:17.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.490+0000 7f080affd700 1 -- 172.21.15.189:0/1784778143 <== mgr.6264 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0804001450 con 0x7f07f806ab30 2024-04-06T12:25:17.487 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path snap_test_dir1 2024-04-06T12:25:17.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.492+0000 7f081c60e700 1 -- 172.21.15.189:0/1784778143 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f07f806ab30 msgr2=0x7f07f806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:17.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.492+0000 7f081c60e700 1 --2- 172.21.15.189:0/1784778143 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f07f806ab30 0x7f07f806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0804000f60 tx=0x7f0804009040 comp rx=0 tx=0).stop 2024-04-06T12:25:17.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.492+0000 7f081c60e700 1 -- 172.21.15.189:0/1784778143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08140b4660 msgr2=0x7f08140aec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:17.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.492+0000 7f081c60e700 1 --2- 172.21.15.189:0/1784778143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08140b4660 0x7f08140aec60 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f08100090b0 tx=0x7f0810004750 comp rx=0 tx=0).stop 2024-04-06T12:25:17.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.492+0000 7f081c60e700 1 -- 172.21.15.189:0/1784778143 shutdown_connections 2024-04-06T12:25:17.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.492+0000 7f081c60e700 1 --2- 172.21.15.189:0/1784778143 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08141f2ae0 0x7f08141ef190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.492+0000 7f081c60e700 1 --2- 172.21.15.189:0/1784778143 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f07f806ab30 0x7f07f806cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.492+0000 7f081c60e700 1 --2- 172.21.15.189:0/1784778143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08140b2fc0 0x7f0814152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.492+0000 7f081c60e700 1 --2- 172.21.15.189:0/1784778143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08140b4660 0x7f08140aec60 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.492+0000 7f081c60e700 1 -- 172.21.15.189:0/1784778143 >> 172.21.15.189:0/1784778143 conn(0x7f08140d00b0 msgr2=0x7f0814153fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:17.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.492+0000 7f081c60e700 1 -- 172.21.15.189:0/1784778143 shutdown_connections 2024-04-06T12:25:17.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.492+0000 7f081c60e700 1 -- 172.21.15.189:0/1784778143 wait complete. 2024-04-06T12:25:17.502 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-04-06T12:25:17.502 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-04-06-12_26_00 in ~43s... 2024-04-06T12:25:17.502 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-04-06-12_27_00 in ~103s... 2024-04-06T12:25:17.502 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m', '2m'], expected_retention: [] 2024-04-06T12:25:17.503 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-04-06T12:25:17.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.629+0000 7f6404121700 1 Processor -- start 2024-04-06T12:25:17.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.629+0000 7f6404121700 1 -- start start 2024-04-06T12:25:17.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.630+0000 7f6404121700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f63fc15a550 0x7f63fc0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:17.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.630+0000 7f6404121700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63fc0cf900 0x7f63fc0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:17.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.630+0000 7f6404121700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63fc0cdbe0 con 0x7f63fc0b4ba0 2024-04-06T12:25:17.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.630+0000 7f6404121700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63fc0b5010 con 0x7f63fc15a550 2024-04-06T12:25:17.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.630+0000 7f6404121700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc15a940 con 0x7f63fc0cf900 2024-04-06T12:25:17.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.630+0000 7f64026be700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f63fc0b4ba0 0x7f63fc06eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46748/0 (socket says 172.21.15.189:46748) 2024-04-06T12:25:17.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.630+0000 7f64026be700 1 -- 172.21.15.189:0/2319027721 learned_addr learned my addr 172.21.15.189:0/2319027721 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:25:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.631+0000 7f6401ebd700 1 --2- 172.21.15.189:0/2319027721 >> v2:172.21.15.192:3300/0 conn(0x7f63fc15a550 0x7f63fc0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.631+0000 7f64016bc700 1 --2- 172.21.15.189:0/2319027721 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63fc0cf900 0x7f63fc0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.631+0000 7f6400ebb700 1 -- 172.21.15.189:0/2319027721 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001108020 0 0) 0x7f63fc0cdbe0 con 0x7f63fc0b4ba0 2024-04-06T12:25:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.631+0000 7f6400ebb700 1 -- 172.21.15.189:0/2319027721 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63e80035b0 con 0x7f63fc0b4ba0 2024-04-06T12:25:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.631+0000 7f6401ebd700 1 -- 172.21.15.189:0/2319027721 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63fc0cf900 msgr2=0x7f63fc0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:25:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.631+0000 7f6401ebd700 1 --2- 172.21.15.189:0/2319027721 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63fc0cf900 0x7f63fc0b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.631+0000 7f6401ebd700 1 -- 172.21.15.189:0/2319027721 >> v1:172.21.15.189:6789/0 conn(0x7f63fc0b4ba0 legacy=0x7f63fc06eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.631+0000 7f6401ebd700 1 -- 172.21.15.189:0/2319027721 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63fc15ac00 con 0x7f63fc15a550 2024-04-06T12:25:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.631+0000 7f64016bc700 1 --2- 172.21.15.189:0/2319027721 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63fc0cf900 0x7f63fc0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:25:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.631+0000 7f6401ebd700 1 --2- 172.21.15.189:0/2319027721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63fc15a550 0x7f63fc0cf3c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f63ec009840 tx=0x7f63ec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaae6b271743b122 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.631+0000 7f6400ebb700 1 -- 172.21.15.189:0/2319027721 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec011830 con 0x7f63fc15a550 2024-04-06T12:25:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.631+0000 7f6400ebb700 1 -- 172.21.15.189:0/2319027721 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63ec011990 con 0x7f63fc15a550 2024-04-06T12:25:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.631+0000 7f6400ebb700 1 -- 172.21.15.189:0/2319027721 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec011d20 con 0x7f63fc15a550 2024-04-06T12:25:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.632+0000 7f6404121700 1 -- 172.21.15.189:0/2319027721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63fc15a550 msgr2=0x7f63fc0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.632+0000 7f6404121700 1 --2- 172.21.15.189:0/2319027721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63fc15a550 0x7f63fc0cf3c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f63ec009840 tx=0x7f63ec009b50 comp rx=0 tx=0).stop 2024-04-06T12:25:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.632+0000 7f6404121700 1 -- 172.21.15.189:0/2319027721 shutdown_connections 2024-04-06T12:25:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.632+0000 7f6404121700 1 --2- 172.21.15.189:0/2319027721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63fc15a550 0x7f63fc0cf3c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.632+0000 7f6404121700 1 --2- 172.21.15.189:0/2319027721 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63fc0cf900 0x7f63fc0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.632+0000 7f6404121700 1 -- 172.21.15.189:0/2319027721 >> 172.21.15.189:0/2319027721 conn(0x7f63fc0d00b0 msgr2=0x7f63fc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.632+0000 7f6404121700 1 -- 172.21.15.189:0/2319027721 shutdown_connections 2024-04-06T12:25:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.632+0000 7f6404121700 1 -- 172.21.15.189:0/2319027721 wait complete. 2024-04-06T12:25:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.633+0000 7f6404121700 1 Processor -- start 2024-04-06T12:25:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.633+0000 7f6404121700 1 -- start start 2024-04-06T12:25:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.634+0000 7f6404121700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63fc0b4ba0 0x7f63fc13caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.634+0000 7f6404121700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63fc0cf900 0x7f63fc0717b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.634+0000 7f6404121700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63fc071cf0 0x7f63fc158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.634+0000 7f6404121700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc14e190 con 0x7f63fc071cf0 2024-04-06T12:25:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.634+0000 7f6404121700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc072170 con 0x7f63fc0cf900 2024-04-06T12:25:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.634+0000 7f6404121700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc13d030 con 0x7f63fc0b4ba0 2024-04-06T12:25:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.634+0000 7f64026be700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63fc071cf0 0x7f63fc158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.634+0000 7f64026be700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63fc071cf0 0x7f63fc158cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38900/0 (socket says 172.21.15.189:38900) 2024-04-06T12:25:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.634+0000 7f64026be700 1 -- 172.21.15.189:0/2658917106 learned_addr learned my addr 172.21.15.189:0/2658917106 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:25:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.634+0000 7f6401ebd700 1 --2- 172.21.15.189:0/2658917106 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63fc0b4ba0 0x7f63fc13caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.634+0000 7f64016bc700 1 --2- 172.21.15.189:0/2658917106 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63fc0cf900 0x7f63fc0717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.634+0000 7f64026be700 1 -- 172.21.15.189:0/2658917106 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63fc0b4ba0 msgr2=0x7f63fc13caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.634+0000 7f64026be700 1 --2- 172.21.15.189:0/2658917106 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63fc0b4ba0 0x7f63fc13caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.634+0000 7f64026be700 1 -- 172.21.15.189:0/2658917106 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63fc0cf900 msgr2=0x7f63fc0717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.634+0000 7f64026be700 1 --2- 172.21.15.189:0/2658917106 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63fc0cf900 0x7f63fc0717b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.634+0000 7f64026be700 1 -- 172.21.15.189:0/2658917106 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f80094a0 con 0x7f63fc071cf0 2024-04-06T12:25:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.634+0000 7f64026be700 1 --2- 172.21.15.189:0/2658917106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63fc071cf0 0x7f63fc158cb0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f63f400caa0 tx=0x7f63f400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.635+0000 7f63f27fc700 1 -- 172.21.15.189:0/2658917106 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400a9e0 con 0x7f63fc071cf0 2024-04-06T12:25:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.635+0000 7f63f27fc700 1 -- 172.21.15.189:0/2658917106 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f400ab40 con 0x7f63fc071cf0 2024-04-06T12:25:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.635+0000 7f6404121700 1 -- 172.21.15.189:0/2658917106 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63ec0094a0 con 0x7f63fc071cf0 2024-04-06T12:25:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.635+0000 7f63f27fc700 1 -- 172.21.15.189:0/2658917106 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4012400 con 0x7f63fc071cf0 2024-04-06T12:25:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.635+0000 7f6404121700 1 -- 172.21.15.189:0/2658917106 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63fc159a20 con 0x7f63fc071cf0 2024-04-06T12:25:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.636+0000 7f6404121700 1 -- 172.21.15.189:0/2658917106 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63fc067bb0 con 0x7f63fc071cf0 2024-04-06T12:25:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.636+0000 7f63f27fc700 1 -- 172.21.15.189:0/2658917106 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 23) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f63f401b3b0 con 0x7f63fc071cf0 2024-04-06T12:25:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.636+0000 7f63f27fc700 1 --2- 172.21.15.189:0/2658917106 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f63e006ab30 0x7f63e006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:25:17.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.637+0000 7f6401ebd700 1 --2- 172.21.15.189:0/2658917106 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f63e006ab30 0x7f63e006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:25:17.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.637+0000 7f63f27fc700 1 -- 172.21.15.189:0/2658917106 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f63f4088f20 con 0x7f63fc071cf0 2024-04-06T12:25:17.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.637+0000 7f6401ebd700 1 --2- 172.21.15.189:0/2658917106 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f63e006ab30 0x7f63e006cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f63ec000c00 tx=0x7f63ec008030 comp rx=0 tx=0).ready entity=mgr.6264 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:25:17.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.641+0000 7f63f27fc700 1 -- 172.21.15.189:0/2658917106 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f63f4054190 con 0x7f63fc071cf0 2024-04-06T12:25:17.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.864+0000 7f6404121700 1 -- 172.21.15.189:0/2658917106 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f63fc15b1c0 con 0x7f63e006ab30 2024-04-06T12:25:17.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.873+0000 7f63f27fc700 1 -- 172.21.15.189:0/2658917106 <== mgr.6264 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+76 (secure 0 0 0) 0x7f63ec011630 con 0x7f63e006ab30 2024-04-06T12:25:17.871 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:25:17.871 INFO:teuthology.orchestra.run.smithi189.stdout:{"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2024-04-06T12:25:17.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.875+0000 7f6404121700 1 -- 172.21.15.189:0/2658917106 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f63e006ab30 msgr2=0x7f63e006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:17.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.875+0000 7f6404121700 1 --2- 172.21.15.189:0/2658917106 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f63e006ab30 0x7f63e006cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f63ec000c00 tx=0x7f63ec008030 comp rx=0 tx=0).stop 2024-04-06T12:25:17.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.875+0000 7f6404121700 1 -- 172.21.15.189:0/2658917106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63fc071cf0 msgr2=0x7f63fc158cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:25:17.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.875+0000 7f6404121700 1 --2- 172.21.15.189:0/2658917106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63fc071cf0 0x7f63fc158cb0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f63f400caa0 tx=0x7f63f400cdb0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.876+0000 7f6404121700 1 -- 172.21.15.189:0/2658917106 shutdown_connections 2024-04-06T12:25:17.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.876+0000 7f6404121700 1 --2- 172.21.15.189:0/2658917106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63fc071cf0 0x7f63fc158cb0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.876+0000 7f6404121700 1 --2- 172.21.15.189:0/2658917106 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63fc0b4ba0 0x7f63fc13caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.876+0000 7f6404121700 1 --2- 172.21.15.189:0/2658917106 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f63e006ab30 0x7f63e006cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.876+0000 7f6404121700 1 --2- 172.21.15.189:0/2658917106 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63fc0cf900 0x7f63fc0717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:25:17.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.876+0000 7f6404121700 1 -- 172.21.15.189:0/2658917106 >> 172.21.15.189:0/2658917106 conn(0x7f63fc0d00b0 msgr2=0x7f63fc0b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:25:17.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.876+0000 7f6404121700 1 -- 172.21.15.189:0/2658917106 shutdown_connections 2024-04-06T12:25:17.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:25:17.876+0000 7f6404121700 1 -- 172.21.15.189:0/2658917106 wait complete. 2024-04-06T12:25:17.885 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2024-04-06T12:25:17.885 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m', '2m'], 'retention': [{}, {}]} 2024-04-06T12:25:17.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:17.885 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:17.903 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:17.903 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:17.903 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:18.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:18.904 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:18.927 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:18.927 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:18.927 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:19.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:19.929 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:19.951 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:19.951 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:19.951 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:20.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:20.952 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:20.975 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:20.975 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:20.975 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:21.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:21.976 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:21.999 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:21.999 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:21.999 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:23.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:23.000 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:23.023 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:23.023 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:23.023 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:24.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:24.024 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:24.046 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:24.046 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:24.046 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:25.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:25.047 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:25.070 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:25.070 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:25.070 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:26.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:26.071 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:26.093 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:26.093 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:26.094 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:27.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:27.094 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:27.117 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:27.117 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:27.117 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:28.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:28.118 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:28.141 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:28.141 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:28.142 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:29.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:29.143 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:29.165 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:29.166 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:29.166 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:30.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:30.167 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:30.189 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:30.189 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:30.189 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:31.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:31.190 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:31.212 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:31.212 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:31.212 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:32.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:32.213 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:32.236 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:32.236 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:32.236 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:33.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:33.236 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:33.259 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:33.259 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:33.259 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:34.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:34.261 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:34.283 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:34.283 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:34.283 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:35.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:35.285 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:35.307 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:35.307 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:35.307 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:36.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:36.308 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:36.331 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:36.331 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:36.331 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:37.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:37.332 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:37.355 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:37.355 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:37.355 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:38.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:38.356 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:38.378 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:38.378 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:38.378 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:39.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:39.380 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:39.402 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:39.402 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:39.402 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:40.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:40.403 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:40.426 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:40.426 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:40.426 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:41.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:41.428 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:41.451 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:41.451 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:41.451 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:42.239 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:25:42.242 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:25:42.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:42.451 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:42.475 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:42.475 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:42.475 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:43.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:43.477 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:43.499 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:43.499 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:43.499 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:44.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:44.501 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:44.523 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:44.524 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:44.524 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:45.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:45.525 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:45.547 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:45.547 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:45.548 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:46.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:46.549 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:46.571 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:46.571 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:46.571 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:47.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:47.572 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:47.594 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:47.594 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:47.594 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:48.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:48.596 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:48.618 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:48.618 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:48.618 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:49.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:49.619 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:49.642 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:49.642 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:49.642 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:50.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:50.644 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:50.667 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:50.667 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:50.667 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:51.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:51.669 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:51.691 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:51.691 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:51.691 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:52.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:52.692 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:52.714 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:52.714 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:52.714 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:53.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:53.716 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:53.738 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:53.738 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:53.738 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:54.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:54.739 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:54.762 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:54.762 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:54.762 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:55.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:55.763 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:55.785 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:55.785 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:55.785 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:56.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:56.787 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:56.809 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:56.809 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:56.809 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:57.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:57.811 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:57.833 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:57.833 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:57.833 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:58.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:58.834 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:58.856 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:58.856 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:58.856 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:25:59.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:25:59.857 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:25:59.879 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:25:59.879 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:25:59.879 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:00.497 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:26:00.500+0000 7f6d14f16700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:26:00.497 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:26:00.500+0000 7f6d14f16700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:26:00.498 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:26:00.500+0000 7f6d14f16700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:26:00.498 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:26:00.500+0000 7f6d14f16700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:26:00.498 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:26:00.500+0000 7f6d14f16700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:26:00.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:00.880 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:00.902 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:00.903 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:00.903 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:00.903 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:00.903 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-04-06-12_26_00_UTC'] 2024-04-06T12:26:00.903 DEBUG:tasks.cephfs.test_snap_schedules:exec=1712406317.5025842, now = 1712406360.903878, timo = 43 2024-04-06T12:26:01.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:01.904 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:01.926 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:01.927 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:01.927 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:01.927 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:02.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:02.928 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:02.948 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:02.949 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:02.949 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:02.949 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:03.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:03.950 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:03.972 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:03.973 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:03.973 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:03.973 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:04.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:04.975 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:04.996 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:04.997 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:04.997 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:04.997 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:05.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:05.999 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:06.020 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:06.021 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:06.021 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:06.021 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:07.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:07.023 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:07.044 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:07.045 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:07.045 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:07.045 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:08.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:08.046 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:08.068 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:08.068 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:08.068 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:08.068 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:09.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:09.069 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:09.091 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:09.091 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:09.092 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:09.092 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:10.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:10.092 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:10.114 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:10.115 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:10.115 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:10.115 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:11.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:11.116 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:11.139 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:11.139 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:11.139 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:11.139 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:12.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:12.141 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:12.164 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:12.164 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:12.164 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:12.164 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:12.318 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:26:12.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:26:13.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:13.165 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:13.186 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:13.187 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:13.188 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:13.188 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:14.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:14.188 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:14.210 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:14.211 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:14.211 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:14.211 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:15.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:15.212 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:15.234 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:15.234 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:15.234 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:15.235 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:16.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:16.236 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:16.257 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:16.258 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:16.258 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:16.258 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:17.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:17.259 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:17.281 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:17.282 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:17.282 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:17.282 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:18.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:18.284 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:18.305 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:18.306 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:18.306 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:18.306 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:19.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:19.307 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:19.328 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:19.329 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:19.329 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:19.329 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:20.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:20.330 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:20.352 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:20.352 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:20.353 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:20.353 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:21.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:21.354 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:21.376 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:21.377 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:21.377 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:21.377 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:22.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:22.378 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:22.399 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:22.400 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:22.400 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:22.400 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:23.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:23.401 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:23.423 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:23.424 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:23.424 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:23.424 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:24.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:24.426 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:24.447 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:24.448 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:24.448 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:24.448 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:25.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:25.449 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:25.471 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:25.472 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:25.472 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:25.472 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:26.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:26.473 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:26.494 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:26.495 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:26.495 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:26.495 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:27.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:27.496 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:27.518 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:27.519 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:27.519 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:27.519 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:28.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:28.520 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:28.541 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:28.542 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:28.542 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:28.542 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:29.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:29.543 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:29.565 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:29.566 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:29.566 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:29.566 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:30.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:30.567 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:30.589 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:30.590 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:30.590 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:30.590 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:31.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:31.591 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:31.613 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:31.614 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:31.614 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:31.615 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:32.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:32.616 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:32.637 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:32.638 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:32.638 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:32.638 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:33.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:33.640 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:33.662 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:33.662 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:33.663 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:33.663 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:34.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:34.664 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:34.686 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:34.687 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:34.687 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:34.688 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:35.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:35.688 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:35.710 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:35.711 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:35.711 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:35.711 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:36.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:36.711 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:36.733 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:36.734 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:36.734 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:36.735 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:37.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:37.736 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:37.757 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:37.758 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:37.758 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:37.758 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:38.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:38.759 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:38.781 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:38.782 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:38.782 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:38.783 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:39.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:39.784 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:39.805 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:39.806 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:39.806 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:39.806 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:40.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:40.808 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:40.830 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:40.831 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:40.831 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:40.831 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:41.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:41.832 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:41.854 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:41.854 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:41.855 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:41.855 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:42.407 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:26:42.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:26:42.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:42.856 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:42.877 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:42.878 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:42.878 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:42.878 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:43.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:43.880 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:43.901 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:43.902 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:43.902 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:43.902 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:44.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:44.903 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:44.925 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:44.926 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:44.926 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:44.926 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:45.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:45.927 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:45.949 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:45.950 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:45.950 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:45.950 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:46.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:46.952 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:46.974 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:46.975 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:46.975 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:46.975 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:47.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:47.976 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:47.998 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:47.999 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:47.999 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:47.999 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:49.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:49.000 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:49.023 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:49.023 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:49.023 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:49.023 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:50.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:50.024 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:50.046 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:50.047 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:50.047 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:50.047 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:51.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:51.048 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:51.070 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:51.071 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:51.071 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:51.071 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:52.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:52.073 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:52.094 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:52.095 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:52.095 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:52.095 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:53.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:53.097 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:53.119 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:53.120 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:53.120 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:53.120 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:54.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:54.121 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:54.142 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:54.143 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:54.143 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:54.143 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:55.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:55.145 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:55.166 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:55.167 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:55.167 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:55.167 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:56.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:56.168 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:56.190 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:56.191 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:56.191 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:56.191 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:57.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:57.192 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:57.213 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:57.214 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:57.214 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:57.214 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:58.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:58.215 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:58.238 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:58.238 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:58.238 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:58.238 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:26:59.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:26:59.239 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:26:59.261 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:26:59.262 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:26:59.262 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:26:59.262 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:27:00.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:00.263 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:27:00.285 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:27:00.286 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:27:00.286 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:27:00.286 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:27:01.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:01.287 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:27:01.309 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:27:01.309 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_27_00_UTC 2024-04-06T12:27:01.309 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_27_00_UTC', 'scheduled-2024-04-06-12_26_00_UTC'} 2024-04-06T12:27:01.309 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-04-06-12_27_00_UTC'} 2024-04-06T12:27:01.310 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:27:01.310 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-04-06-12_27_00_UTC'] 2024-04-06T12:27:01.310 DEBUG:tasks.cephfs.test_snap_schedules:exec=1712406317.5025842, now = 1712406421.3102741, timo = 103 2024-04-06T12:27:02.311 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-04-06T12:27:02.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.450+0000 7f4ee0d55700 1 Processor -- start 2024-04-06T12:27:02.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.450+0000 7f4ee0d55700 1 -- start start 2024-04-06T12:27:02.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.451+0000 7f4ee0d55700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4edc0b4660 0x7f4edc0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:02.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.451+0000 7f4ee0d55700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4edc0b2fc0 0x7f4edc0b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:02.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.451+0000 7f4ee0d55700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4edc0b5030 con 0x7f4edc15a550 2024-04-06T12:27:02.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.451+0000 7f4ee0d55700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4edc0cdbe0 con 0x7f4edc0b2fc0 2024-04-06T12:27:02.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.451+0000 7f4ee0d55700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4edc15a940 con 0x7f4edc0b4660 2024-04-06T12:27:02.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.452+0000 7f4edad9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4edc0b2fc0 0x7f4edc0b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:02.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.452+0000 7f4edad9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4edc0b2fc0 0x7f4edc0b33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36346/0 (socket says 172.21.15.189:36346) 2024-04-06T12:27:02.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.452+0000 7f4edad9d700 1 -- 172.21.15.189:0/2328708724 learned_addr learned my addr 172.21.15.189:0/2328708724 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:02.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.452+0000 7f4ed9d9b700 1 --2- 172.21.15.189:0/2328708724 >> v2:172.21.15.192:3300/0 conn(0x7f4edc0b4660 0x7f4edc0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:02.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.452+0000 7f4edad9d700 1 -- 172.21.15.189:0/2328708724 >> v2:172.21.15.192:3300/0 conn(0x7f4edc0b4660 msgr2=0x7f4edc0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:02.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.452+0000 7f4edad9d700 1 --2- 172.21.15.189:0/2328708724 >> v2:172.21.15.192:3300/0 conn(0x7f4edc0b4660 0x7f4edc0b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:02.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.452+0000 7f4edad9d700 1 -- 172.21.15.189:0/2328708724 >> v1:172.21.15.192:6789/0 conn(0x7f4edc15a550 legacy=0x7f4edc0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:02.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.452+0000 7f4edad9d700 1 -- 172.21.15.189:0/2328708724 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4edc15ac00 con 0x7f4edc0b2fc0 2024-04-06T12:27:02.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.452+0000 7f4edad9d700 1 --2- 172.21.15.189:0/2328708724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4edc0b2fc0 0x7f4edc0b33d0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4edc0b2d40 tx=0x7f4ed400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b1569b9895fbff server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:02.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.452+0000 7f4ed959a700 1 -- 172.21.15.189:0/2328708724 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed400e070 con 0x7f4edc0b2fc0 2024-04-06T12:27:02.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.452+0000 7f4ed959a700 1 -- 172.21.15.189:0/2328708724 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed4003a20 con 0x7f4edc0b2fc0 2024-04-06T12:27:02.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.452+0000 7f4ed959a700 1 -- 172.21.15.189:0/2328708724 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed4003dd0 con 0x7f4edc0b2fc0 2024-04-06T12:27:02.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.453+0000 7f4ee0d55700 1 -- 172.21.15.189:0/2328708724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4edc0b2fc0 msgr2=0x7f4edc0b33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:02.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.453+0000 7f4ee0d55700 1 --2- 172.21.15.189:0/2328708724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4edc0b2fc0 0x7f4edc0b33d0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4edc0b2d40 tx=0x7f4ed400b330 comp rx=0 tx=0).stop 2024-04-06T12:27:02.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.453+0000 7f4ee0d55700 1 -- 172.21.15.189:0/2328708724 shutdown_connections 2024-04-06T12:27:02.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.453+0000 7f4ee0d55700 1 --2- 172.21.15.189:0/2328708724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4edc0b2fc0 0x7f4edc0b33d0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:02.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.453+0000 7f4ee0d55700 1 --2- 172.21.15.189:0/2328708724 >> v2:172.21.15.192:3300/0 conn(0x7f4edc0b4660 0x7f4edc0b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:02.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.453+0000 7f4ee0d55700 1 -- 172.21.15.189:0/2328708724 >> 172.21.15.189:0/2328708724 conn(0x7f4edc0d00b0 msgr2=0x7f4edc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:02.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.453+0000 7f4ee0d55700 1 -- 172.21.15.189:0/2328708724 shutdown_connections 2024-04-06T12:27:02.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.453+0000 7f4ee0d55700 1 -- 172.21.15.189:0/2328708724 wait complete. 2024-04-06T12:27:02.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.454+0000 7f4ee0d55700 1 Processor -- start 2024-04-06T12:27:02.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.454+0000 7f4ee0d55700 1 -- start start 2024-04-06T12:27:02.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.455+0000 7f4ee0d55700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4edc0b2fc0 0x7f4edc14e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:02.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.455+0000 7f4ee0d55700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4edc0b4660 0x7f4edc158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:02.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.455+0000 7f4ee0d55700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4edc15d4d0 0x7f4edc159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:02.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.455+0000 7f4ee0d55700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4edc0b14d0 con 0x7f4edc0b4660 2024-04-06T12:27:02.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.455+0000 7f4ee0d55700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4edc141000 con 0x7f4edc15d4d0 2024-04-06T12:27:02.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.455+0000 7f4ee0d55700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4edc0b1720 con 0x7f4edc0b2fc0 2024-04-06T12:27:02.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.455+0000 7f4ed9d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4edc0b4660 0x7f4edc158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:02.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.455+0000 7f4ed9d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4edc0b4660 0x7f4edc158b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36354/0 (socket says 172.21.15.189:36354) 2024-04-06T12:27:02.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.455+0000 7f4ed9d9b700 1 -- 172.21.15.189:0/3021869095 learned_addr learned my addr 172.21.15.189:0/3021869095 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:02.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.455+0000 7f4edad9d700 1 --2- 172.21.15.189:0/3021869095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4edc15d4d0 0x7f4edc159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:02.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.455+0000 7f4eda59c700 1 --2- 172.21.15.189:0/3021869095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4edc0b2fc0 0x7f4edc14e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:02.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.455+0000 7f4ed9d9b700 1 -- 172.21.15.189:0/3021869095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4edc0b2fc0 msgr2=0x7f4edc14e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:02.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.455+0000 7f4ed9d9b700 1 --2- 172.21.15.189:0/3021869095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4edc0b2fc0 0x7f4edc14e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:02.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.455+0000 7f4ed9d9b700 1 -- 172.21.15.189:0/3021869095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4edc15d4d0 msgr2=0x7f4edc159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:02.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.455+0000 7f4ed9d9b700 1 --2- 172.21.15.189:0/3021869095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4edc15d4d0 0x7f4edc159090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:02.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.455+0000 7f4ed9d9b700 1 -- 172.21.15.189:0/3021869095 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed4009b30 con 0x7f4edc0b4660 2024-04-06T12:27:02.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.456+0000 7f4ed9d9b700 1 --2- 172.21.15.189:0/3021869095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4edc0b4660 0x7f4edc158b50 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc00e810 tx=0x7f4ecc00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:02.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.456+0000 7f4ecaffd700 1 -- 172.21.15.189:0/3021869095 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc00c070 con 0x7f4edc0b4660 2024-04-06T12:27:02.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.456+0000 7f4ecaffd700 1 -- 172.21.15.189:0/3021869095 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ecc013610 con 0x7f4edc0b4660 2024-04-06T12:27:02.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.456+0000 7f4ee0d55700 1 -- 172.21.15.189:0/3021869095 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4edc0b1a00 con 0x7f4edc0b4660 2024-04-06T12:27:02.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.456+0000 7f4ecaffd700 1 -- 172.21.15.189:0/3021869095 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc0139c0 con 0x7f4edc0b4660 2024-04-06T12:27:02.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.456+0000 7f4ee0d55700 1 -- 172.21.15.189:0/3021869095 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4edc15bc40 con 0x7f4edc0b4660 2024-04-06T12:27:02.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.457+0000 7f4ee0d55700 1 -- 172.21.15.189:0/3021869095 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4edc15be00 con 0x7f4edc0b4660 2024-04-06T12:27:02.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.457+0000 7f4ecaffd700 1 -- 172.21.15.189:0/3021869095 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) v1 ==== 89276+0+0 (secure 0 0 0) 0x7f4ecc013b20 con 0x7f4edc0b4660 2024-04-06T12:27:02.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.458+0000 7f4ecaffd700 1 --2- 172.21.15.189:0/3021869095 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4ebc06ae00 0x7f4ebc06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:02.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.458+0000 7f4ecaffd700 1 -- 172.21.15.189:0/3021869095 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f4ecc089650 con 0x7f4edc0b4660 2024-04-06T12:27:02.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.458+0000 7f4eda59c700 1 --2- 172.21.15.189:0/3021869095 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4ebc06ae00 0x7f4ebc06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:02.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.458+0000 7f4eda59c700 1 --2- 172.21.15.189:0/3021869095 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4ebc06ae00 0x7f4ebc06d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0007fd0 tx=0x7f4ed0007ec0 comp rx=0 tx=0).ready entity=mgr.6264 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:02.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.462+0000 7f4ecaffd700 1 -- 172.21.15.189:0/3021869095 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4ecc054350 con 0x7f4edc0b4660 2024-04-06T12:27:02.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.684+0000 7f4ee0d55700 1 -- 172.21.15.189:0/3021869095 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4edc15b4a0 con 0x7f4ebc06ae00 2024-04-06T12:27:02.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.754+0000 7f4ecaffd700 1 -- 172.21.15.189:0/3021869095 <== mgr.6264 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f4ed0003a90 con 0x7f4ebc06ae00 2024-04-06T12:27:02.751 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule removed for path snap_test_dir1 2024-04-06T12:27:02.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.756+0000 7f4ee0d55700 1 -- 172.21.15.189:0/3021869095 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4ebc06ae00 msgr2=0x7f4ebc06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:02.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.756+0000 7f4ee0d55700 1 --2- 172.21.15.189:0/3021869095 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4ebc06ae00 0x7f4ebc06d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0007fd0 tx=0x7f4ed0007ec0 comp rx=0 tx=0).stop 2024-04-06T12:27:02.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.756+0000 7f4ee0d55700 1 -- 172.21.15.189:0/3021869095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4edc0b4660 msgr2=0x7f4edc158b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:02.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.756+0000 7f4ee0d55700 1 --2- 172.21.15.189:0/3021869095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4edc0b4660 0x7f4edc158b50 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc00e810 tx=0x7f4ecc00eb20 comp rx=0 tx=0).stop 2024-04-06T12:27:02.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.756+0000 7f4ee0d55700 1 -- 172.21.15.189:0/3021869095 shutdown_connections 2024-04-06T12:27:02.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.756+0000 7f4ee0d55700 1 --2- 172.21.15.189:0/3021869095 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4ebc06ae00 0x7f4ebc06d2c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:02.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.756+0000 7f4ee0d55700 1 --2- 172.21.15.189:0/3021869095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4edc15d4d0 0x7f4edc159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:02.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.756+0000 7f4ee0d55700 1 --2- 172.21.15.189:0/3021869095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4edc0b2fc0 0x7f4edc14e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:02.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.756+0000 7f4ee0d55700 1 --2- 172.21.15.189:0/3021869095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4edc0b4660 0x7f4edc158b50 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:02.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.756+0000 7f4ee0d55700 1 -- 172.21.15.189:0/3021869095 >> 172.21.15.189:0/3021869095 conn(0x7f4edc0d00b0 msgr2=0x7f4edc06d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:02.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.756+0000 7f4ee0d55700 1 -- 172.21.15.189:0/3021869095 shutdown_connections 2024-04-06T12:27:02.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:02.756+0000 7f4ee0d55700 1 -- 172.21.15.189:0/3021869095 wait complete. 2024-04-06T12:27:02.765 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-04-06T12:27:02.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:02.766 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:27:02.783 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:27:02.783 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_27_00_UTC 2024-04-06T12:27:02.784 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_26_00_UTC 2024-04-06T12:27:02.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:02.784 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_26_00_UTC) 2024-04-06T12:27:02.846 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_27_00_UTC 2024-04-06T12:27:02.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:02.847 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_27_00_UTC) 2024-04-06T12:27:02.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:02.911 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-06T12:27:02.974 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-06T12:27:03.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.140+0000 7fdc280f0700 1 Processor -- start 2024-04-06T12:27:03.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.140+0000 7fdc280f0700 1 -- start start 2024-04-06T12:27:03.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.141+0000 7fdc280f0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdc200d11b0 0x7fdc200d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:03.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.141+0000 7fdc280f0700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdc200cb870 0x7fdc2015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:03.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.141+0000 7fdc280f0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc200cfa00 0x7fdc200cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.141+0000 7fdc280f0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc200ccdc0 con 0x7fdc200d11b0 2024-04-06T12:27:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.141+0000 7fdc280f0700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc200d0b50 con 0x7fdc200cb870 2024-04-06T12:27:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.141+0000 7fdc280f0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc200d1ad0 con 0x7fdc200cfa00 2024-04-06T12:27:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.141+0000 7fdc25e8c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdc200d11b0 0x7fdc200d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.141+0000 7fdc25e8c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdc200d11b0 0x7fdc200d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36362/0 (socket says 172.21.15.189:36362) 2024-04-06T12:27:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.141+0000 7fdc25e8c700 1 -- 172.21.15.189:0/3556180235 learned_addr learned my addr 172.21.15.189:0/3556180235 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.141+0000 7fdc2668d700 1 --2- 172.21.15.189:0/3556180235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc200cfa00 0x7fdc200cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.141+0000 7fdc2568b700 1 --2- 172.21.15.189:0/3556180235 >> v2:172.21.15.192:3300/0 conn(0x7fdc200cb870 0x7fdc2015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.141+0000 7fdc25e8c700 1 -- 172.21.15.189:0/3556180235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc200cfa00 msgr2=0x7fdc200cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.141+0000 7fdc25e8c700 1 --2- 172.21.15.189:0/3556180235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc200cfa00 0x7fdc200cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.141+0000 7fdc25e8c700 1 -- 172.21.15.189:0/3556180235 >> v2:172.21.15.192:3300/0 conn(0x7fdc200cb870 msgr2=0x7fdc2015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.141+0000 7fdc25e8c700 1 --2- 172.21.15.189:0/3556180235 >> v2:172.21.15.192:3300/0 conn(0x7fdc200cb870 0x7fdc2015c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.141+0000 7fdc25e8c700 1 -- 172.21.15.189:0/3556180235 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc200cbd80 con 0x7fdc200d11b0 2024-04-06T12:27:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.141+0000 7fdc25e8c700 1 --2- 172.21.15.189:0/3556180235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc200d11b0 0x7fdc200d1590 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fdc2004fb40 tx=0x7fdc1c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27c428f2dfa899af server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.142+0000 7fdc24e8a700 1 -- 172.21.15.189:0/3556180235 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c00b870 con 0x7fdc200d11b0 2024-04-06T12:27:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.142+0000 7fdc24e8a700 1 -- 172.21.15.189:0/3556180235 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1c00b9d0 con 0x7fdc200d11b0 2024-04-06T12:27:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.142+0000 7fdc24e8a700 1 -- 172.21.15.189:0/3556180235 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c00bd60 con 0x7fdc200d11b0 2024-04-06T12:27:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.142+0000 7fdc280f0700 1 -- 172.21.15.189:0/3556180235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc200d11b0 msgr2=0x7fdc200d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.142+0000 7fdc280f0700 1 --2- 172.21.15.189:0/3556180235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc200d11b0 0x7fdc200d1590 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fdc2004fb40 tx=0x7fdc1c009bd0 comp rx=0 tx=0).stop 2024-04-06T12:27:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.142+0000 7fdc280f0700 1 -- 172.21.15.189:0/3556180235 shutdown_connections 2024-04-06T12:27:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.142+0000 7fdc280f0700 1 --2- 172.21.15.189:0/3556180235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc200d11b0 0x7fdc200d1590 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.142+0000 7fdc280f0700 1 --2- 172.21.15.189:0/3556180235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc200cfa00 0x7fdc200cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.142+0000 7fdc280f0700 1 --2- 172.21.15.189:0/3556180235 >> v2:172.21.15.192:3300/0 conn(0x7fdc200cb870 0x7fdc2015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.142+0000 7fdc280f0700 1 -- 172.21.15.189:0/3556180235 >> 172.21.15.189:0/3556180235 conn(0x7fdc200b2db0 msgr2=0x7fdc200b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.142+0000 7fdc280f0700 1 -- 172.21.15.189:0/3556180235 shutdown_connections 2024-04-06T12:27:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.142+0000 7fdc280f0700 1 -- 172.21.15.189:0/3556180235 wait complete. 2024-04-06T12:27:03.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.143+0000 7fdc280f0700 1 Processor -- start 2024-04-06T12:27:03.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.143+0000 7fdc280f0700 1 -- start start 2024-04-06T12:27:03.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.144+0000 7fdc280f0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc200cb870 0x7fdc20156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:03.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.144+0000 7fdc280f0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc200cfa00 0x7fdc20158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:03.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.144+0000 7fdc280f0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc200d11b0 0x7fdc201552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:03.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.144+0000 7fdc280f0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc201f50d0 con 0x7fdc200d11b0 2024-04-06T12:27:03.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.144+0000 7fdc280f0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc201f4450 con 0x7fdc200cb870 2024-04-06T12:27:03.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.144+0000 7fdc280f0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc201f4730 con 0x7fdc200cfa00 2024-04-06T12:27:03.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.144+0000 7fdc2668d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc200d11b0 0x7fdc201552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:03.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.144+0000 7fdc2668d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc200d11b0 0x7fdc201552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36366/0 (socket says 172.21.15.189:36366) 2024-04-06T12:27:03.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.144+0000 7fdc2668d700 1 -- 172.21.15.189:0/2637888801 learned_addr learned my addr 172.21.15.189:0/2637888801 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:03.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.144+0000 7fdc25e8c700 1 --2- 172.21.15.189:0/2637888801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc200cb870 0x7fdc20156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:03.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.144+0000 7fdc2568b700 1 --2- 172.21.15.189:0/2637888801 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc200cfa00 0x7fdc20158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:03.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.144+0000 7fdc2668d700 1 -- 172.21.15.189:0/2637888801 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc200cfa00 msgr2=0x7fdc20158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:03.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.145+0000 7fdc2668d700 1 --2- 172.21.15.189:0/2637888801 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc200cfa00 0x7fdc20158940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:03.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.145+0000 7fdc2668d700 1 -- 172.21.15.189:0/2637888801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc200cb870 msgr2=0x7fdc20156be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:03.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.145+0000 7fdc2668d700 1 --2- 172.21.15.189:0/2637888801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc200cb870 0x7fdc20156be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:03.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.145+0000 7fdc2668d700 1 -- 172.21.15.189:0/2637888801 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1c009550 con 0x7fdc200d11b0 2024-04-06T12:27:03.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.145+0000 7fdc25e8c700 1 --2- 172.21.15.189:0/2637888801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc200cb870 0x7fdc20156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:27:03.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.145+0000 7fdc2668d700 1 --2- 172.21.15.189:0/2637888801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc200d11b0 0x7fdc201552d0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800cc30 tx=0x7fdc1800a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:03.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.145+0000 7fdc167fc700 1 -- 172.21.15.189:0/2637888801 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1800ab70 con 0x7fdc200d11b0 2024-04-06T12:27:03.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.145+0000 7fdc167fc700 1 -- 172.21.15.189:0/2637888801 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1800acd0 con 0x7fdc200d11b0 2024-04-06T12:27:03.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.145+0000 7fdc280f0700 1 -- 172.21.15.189:0/2637888801 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc201f4a10 con 0x7fdc200d11b0 2024-04-06T12:27:03.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.145+0000 7fdc167fc700 1 -- 172.21.15.189:0/2637888801 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc18003c40 con 0x7fdc200d11b0 2024-04-06T12:27:03.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.145+0000 7fdc280f0700 1 -- 172.21.15.189:0/2637888801 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc20156220 con 0x7fdc200d11b0 2024-04-06T12:27:03.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.146+0000 7fdc280f0700 1 -- 172.21.15.189:0/2637888801 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc20074990 con 0x7fdc200d11b0 2024-04-06T12:27:03.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.146+0000 7fdc167fc700 1 -- 172.21.15.189:0/2637888801 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) v1 ==== 89276+0+0 (secure 0 0 0) 0x7fdc1800f3f0 con 0x7fdc200d11b0 2024-04-06T12:27:03.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.147+0000 7fdc167fc700 1 --2- 172.21.15.189:0/2637888801 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdc0406ae00 0x7fdc0406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:03.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.147+0000 7fdc25e8c700 1 --2- 172.21.15.189:0/2637888801 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdc0406ae00 0x7fdc0406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:03.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.147+0000 7fdc167fc700 1 -- 172.21.15.189:0/2637888801 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fdc180899d0 con 0x7fdc200d11b0 2024-04-06T12:27:03.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.147+0000 7fdc25e8c700 1 --2- 172.21.15.189:0/2637888801 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdc0406ae00 0x7fdc0406d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c009130 tx=0x7fdc1c008030 comp rx=0 tx=0).ready entity=mgr.6264 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:03.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.151+0000 7fdc167fc700 1 -- 172.21.15.189:0/2637888801 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdc18054750 con 0x7fdc200d11b0 2024-04-06T12:27:03.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.401+0000 7fdc280f0700 1 -- 172.21.15.189:0/2637888801 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fdc200b0a00 con 0x7fdc200d11b0 2024-04-06T12:27:03.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.566+0000 7fdc167fc700 1 -- 172.21.15.189:0/2637888801 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 28) v1 ==== 89259+0+0 (secure 0 0 0) 0x7fdc18087c20 con 0x7fdc200d11b0 2024-04-06T12:27:03.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.567+0000 7fdc167fc700 1 -- 172.21.15.189:0/2637888801 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v28) v1 ==== 93+0+0 (secure 0 0 0) 0x7fdc18073d90 con 0x7fdc200d11b0 2024-04-06T12:27:03.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.570+0000 7fdc280f0700 1 -- 172.21.15.189:0/2637888801 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdc0406ae00 msgr2=0x7fdc0406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:03.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.570+0000 7fdc280f0700 1 --2- 172.21.15.189:0/2637888801 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdc0406ae00 0x7fdc0406d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c009130 tx=0x7fdc1c008030 comp rx=0 tx=0).stop 2024-04-06T12:27:03.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.570+0000 7fdc280f0700 1 -- 172.21.15.189:0/2637888801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc200d11b0 msgr2=0x7fdc201552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:03.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.570+0000 7fdc280f0700 1 --2- 172.21.15.189:0/2637888801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc200d11b0 0x7fdc201552d0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800cc30 tx=0x7fdc1800a1f0 comp rx=0 tx=0).stop 2024-04-06T12:27:03.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.570+0000 7fdc280f0700 1 -- 172.21.15.189:0/2637888801 shutdown_connections 2024-04-06T12:27:03.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.570+0000 7fdc280f0700 1 --2- 172.21.15.189:0/2637888801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc200d11b0 0x7fdc201552d0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:03.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.570+0000 7fdc280f0700 1 --2- 172.21.15.189:0/2637888801 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdc0406ae00 0x7fdc0406d2c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:03.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.570+0000 7fdc280f0700 1 --2- 172.21.15.189:0/2637888801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc200cb870 0x7fdc20156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.570+0000 7fdc280f0700 1 --2- 172.21.15.189:0/2637888801 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdc200cfa00 0x7fdc20158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.570+0000 7fdc280f0700 1 -- 172.21.15.189:0/2637888801 >> 172.21.15.189:0/2637888801 conn(0x7fdc200b2db0 msgr2=0x7fdc2015a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.570+0000 7fdc280f0700 1 -- 172.21.15.189:0/2637888801 shutdown_connections 2024-04-06T12:27:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.570+0000 7fdc280f0700 1 -- 172.21.15.189:0/2637888801 wait complete. 2024-04-06T12:27:03.580 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T12:27:03.581 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:27:03.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:27:03.659 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T12:27:03.660 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:27:03.660 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:27:03.677 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:27:03.678 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:27:03.736 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:27:03.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.904+0000 7f66f1f04700 1 Processor -- start 2024-04-06T12:27:03.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.904+0000 7f66f1f04700 1 -- start start 2024-04-06T12:27:03.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.905+0000 7f66f1f04700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f66ec0d0a60 0x7f66ec0d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:03.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.905+0000 7f66f1f04700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66ec0d1af0 0x7f66ec0b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:03.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.905+0000 7f66f1f04700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ec0cdf80 con 0x7f66ec15d7e0 2024-04-06T12:27:03.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.905+0000 7f66f1f04700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ec0d1f80 con 0x7f66ec0d0a60 2024-04-06T12:27:03.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.905+0000 7f66f1f04700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec0d3230 con 0x7f66ec0d1af0 2024-04-06T12:27:03.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.905+0000 7f66eaffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f66ec0d0a60 0x7f66ec0d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:03.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.905+0000 7f66eaffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f66ec0d0a60 0x7f66ec0d15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36382/0 (socket says 172.21.15.189:36382) 2024-04-06T12:27:03.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.905+0000 7f66eaffd700 1 -- 172.21.15.189:0/2151013685 learned_addr learned my addr 172.21.15.189:0/2151013685 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:03.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.905+0000 7f66ebfff700 1 --2- 172.21.15.189:0/2151013685 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66ec0d1af0 0x7f66ec0b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:03.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.905+0000 7f66eaffd700 1 -- 172.21.15.189:0/2151013685 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66ec0d1af0 msgr2=0x7f66ec0b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:03.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.905+0000 7f66eaffd700 1 --2- 172.21.15.189:0/2151013685 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66ec0d1af0 0x7f66ec0b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:03.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.905+0000 7f66eaffd700 1 -- 172.21.15.189:0/2151013685 >> v1:172.21.15.192:6789/0 conn(0x7f66ec15d7e0 legacy=0x7f66ec0d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:03.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.905+0000 7f66eaffd700 1 -- 172.21.15.189:0/2151013685 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ec0d2da0 con 0x7f66ec0d0a60 2024-04-06T12:27:03.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.906+0000 7f66eaffd700 1 --2- 172.21.15.189:0/2151013685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66ec0d0a60 0x7f66ec0d15b0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f66e00098f0 tx=0x7f66e0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=718ea0a5cf711eb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:03.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.906+0000 7f66ea7fc700 1 -- 172.21.15.189:0/2151013685 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e0011830 con 0x7f66ec0d0a60 2024-04-06T12:27:03.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.906+0000 7f66ea7fc700 1 -- 172.21.15.189:0/2151013685 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e0011990 con 0x7f66ec0d0a60 2024-04-06T12:27:03.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.906+0000 7f66ea7fc700 1 -- 172.21.15.189:0/2151013685 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e0011d20 con 0x7f66ec0d0a60 2024-04-06T12:27:03.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.906+0000 7f66f1f04700 1 -- 172.21.15.189:0/2151013685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66ec0d0a60 msgr2=0x7f66ec0d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:03.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.906+0000 7f66f1f04700 1 --2- 172.21.15.189:0/2151013685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66ec0d0a60 0x7f66ec0d15b0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f66e00098f0 tx=0x7f66e0009c00 comp rx=0 tx=0).stop 2024-04-06T12:27:03.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.907+0000 7f66f1f04700 1 -- 172.21.15.189:0/2151013685 shutdown_connections 2024-04-06T12:27:03.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.907+0000 7f66f1f04700 1 --2- 172.21.15.189:0/2151013685 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66ec0d1af0 0x7f66ec0b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.907+0000 7f66f1f04700 1 --2- 172.21.15.189:0/2151013685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66ec0d0a60 0x7f66ec0d15b0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.907+0000 7f66f1f04700 1 -- 172.21.15.189:0/2151013685 >> 172.21.15.189:0/2151013685 conn(0x7f66ec072250 msgr2=0x7f66ec072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.907+0000 7f66f1f04700 1 -- 172.21.15.189:0/2151013685 shutdown_connections 2024-04-06T12:27:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.907+0000 7f66f1f04700 1 -- 172.21.15.189:0/2151013685 wait complete. 2024-04-06T12:27:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.908+0000 7f66f1f04700 1 Processor -- start 2024-04-06T12:27:03.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.908+0000 7f66f1f04700 1 -- start start 2024-04-06T12:27:03.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.909+0000 7f66f1f04700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ec0d0a60 0x7f66ec0c7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:03.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.909+0000 7f66f1f04700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66ec0d1af0 0x7f66ec159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:03.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.909+0000 7f66f1f04700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66ec15d7e0 0x7f66ec0c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:03.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.909+0000 7f66f1f04700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec1f3b80 con 0x7f66ec0d1af0 2024-04-06T12:27:03.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.909+0000 7f66f1f04700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec0c74c0 con 0x7f66ec0d0a60 2024-04-06T12:27:03.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.909+0000 7f66f1f04700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec0c6b90 con 0x7f66ec15d7e0 2024-04-06T12:27:03.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.910+0000 7f66eaffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66ec0d1af0 0x7f66ec159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:03.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.910+0000 7f66eaffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66ec0d1af0 0x7f66ec159840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36396/0 (socket says 172.21.15.189:36396) 2024-04-06T12:27:03.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.910+0000 7f66eaffd700 1 -- 172.21.15.189:0/3439649093 learned_addr learned my addr 172.21.15.189:0/3439649093 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:03.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.910+0000 7f66eb7fe700 1 --2- 172.21.15.189:0/3439649093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ec0d0a60 0x7f66ec0c7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:03.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.910+0000 7f66ebfff700 1 --2- 172.21.15.189:0/3439649093 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66ec15d7e0 0x7f66ec0c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:03.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.910+0000 7f66eaffd700 1 -- 172.21.15.189:0/3439649093 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66ec15d7e0 msgr2=0x7f66ec0c92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:03.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.910+0000 7f66eaffd700 1 --2- 172.21.15.189:0/3439649093 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66ec15d7e0 0x7f66ec0c92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:03.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.910+0000 7f66eaffd700 1 -- 172.21.15.189:0/3439649093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ec0d0a60 msgr2=0x7f66ec0c7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:03.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.910+0000 7f66eaffd700 1 --2- 172.21.15.189:0/3439649093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ec0d0a60 0x7f66ec0c7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:03.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.910+0000 7f66eaffd700 1 -- 172.21.15.189:0/3439649093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e0009550 con 0x7f66ec0d1af0 2024-04-06T12:27:03.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.910+0000 7f66eaffd700 1 --2- 172.21.15.189:0/3439649093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66ec0d1af0 0x7f66ec159840 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f66e0009c40 tx=0x7f66e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.910+0000 7f66d3fff700 1 -- 172.21.15.189:0/3439649093 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e0004510 con 0x7f66ec0d1af0 2024-04-06T12:27:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.910+0000 7f66d3fff700 1 -- 172.21.15.189:0/3439649093 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e001e070 con 0x7f66ec0d1af0 2024-04-06T12:27:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.910+0000 7f66f1f04700 1 -- 172.21.15.189:0/3439649093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ec0c6e70 con 0x7f66ec0d1af0 2024-04-06T12:27:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.910+0000 7f66d3fff700 1 -- 172.21.15.189:0/3439649093 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e00038e0 con 0x7f66ec0d1af0 2024-04-06T12:27:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.910+0000 7f66f1f04700 1 -- 172.21.15.189:0/3439649093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ec159d80 con 0x7f66ec0d1af0 2024-04-06T12:27:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.911+0000 7f66f1f04700 1 -- 172.21.15.189:0/3439649093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66ec073f90 con 0x7f66ec0d1af0 2024-04-06T12:27:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.912+0000 7f66d3fff700 1 -- 172.21.15.189:0/3439649093 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 28) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f66e0003a40 con 0x7f66ec0d1af0 2024-04-06T12:27:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.913+0000 7f66d3fff700 1 --2- 172.21.15.189:0/3439649093 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66d806adb0 0x7f66d806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.913+0000 7f66eb7fe700 1 -- 172.21.15.189:0/3439649093 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66d806adb0 msgr2=0x7f66d806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.913+0000 7f66d3fff700 1 -- 172.21.15.189:0/3439649093 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f66e008a600 con 0x7f66ec0d1af0 2024-04-06T12:27:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.913+0000 7f66eb7fe700 1 --2- 172.21.15.189:0/3439649093 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66d806adb0 0x7f66d806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:03.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:03.921+0000 7f66d3fff700 1 -- 172.21.15.189:0/3439649093 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f66e0055410 con 0x7f66ec0d1af0 2024-04-06T12:27:03.951 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:03.954+0000 7fe88a431200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:27:03.963 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:03.966+0000 7fbc5e4e5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:27:04.029 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:04.032+0000 7fe88a431200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:27:04.043 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:04.046+0000 7fbc5e4e5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:27:04.101 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:04.104+0000 7fe88a431200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:27:04.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.114+0000 7f66eb7fe700 1 -- 172.21.15.189:0/3439649093 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66d806adb0 msgr2=0x7f66d806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:04.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.114+0000 7f66eb7fe700 1 --2- 172.21.15.189:0/3439649093 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66d806adb0 0x7f66d806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:04.118 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:04.121+0000 7fbc5e4e5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:27:04.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.125+0000 7f66f1f04700 1 -- 172.21.15.189:0/3439649093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66ec0c7630 con 0x7f66ec0d1af0 2024-04-06T12:27:04.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.127+0000 7f66d3fff700 1 -- 172.21.15.189:0/3439649093 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16892 (secure 0 0 0) 0x7f66e0058a30 con 0x7f66ec0d1af0 2024-04-06T12:27:04.125 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:04.125 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":70,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:26:17.522565+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:24:59.828905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-06T12:25:01.020253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.23","mappings":[{"from":0,"to":1}]},{"pgid":"8.38","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/1187685283":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/1356350621":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/162292108":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/2706754156":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/1122382676":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/2132037261":"2024-04-07T12:25:15.982358+0000","172.21.15.189:6833/47128":"2024-04-07T12:25:15.982358+0000","172.21.15.189:6832/47128":"2024-04-07T12:25:15.982358+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:27:04.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.129+0000 7f66f1f04700 1 -- 172.21.15.189:0/3439649093 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66d806adb0 msgr2=0x7f66d806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:04.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.130+0000 7f66f1f04700 1 --2- 172.21.15.189:0/3439649093 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66d806adb0 0x7f66d806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.130+0000 7f66f1f04700 1 -- 172.21.15.189:0/3439649093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66ec0d1af0 msgr2=0x7f66ec159840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:04.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.130+0000 7f66f1f04700 1 --2- 172.21.15.189:0/3439649093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66ec0d1af0 0x7f66ec159840 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f66e0009c40 tx=0x7f66e0004750 comp rx=0 tx=0).stop 2024-04-06T12:27:04.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.130+0000 7f66f1f04700 1 -- 172.21.15.189:0/3439649093 shutdown_connections 2024-04-06T12:27:04.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.130+0000 7f66f1f04700 1 --2- 172.21.15.189:0/3439649093 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66ec15d7e0 0x7f66ec0c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.130+0000 7f66f1f04700 1 --2- 172.21.15.189:0/3439649093 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66d806adb0 0x7f66d806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.130+0000 7f66f1f04700 1 --2- 172.21.15.189:0/3439649093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ec0d0a60 0x7f66ec0c7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.130+0000 7f66f1f04700 1 --2- 172.21.15.189:0/3439649093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66ec0d1af0 0x7f66ec159840 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.130+0000 7f66f1f04700 1 -- 172.21.15.189:0/3439649093 >> 172.21.15.189:0/3439649093 conn(0x7f66ec072250 msgr2=0x7f66ec0cd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:04.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.130+0000 7f66f1f04700 1 -- 172.21.15.189:0/3439649093 shutdown_connections 2024-04-06T12:27:04.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.130+0000 7f66f1f04700 1 -- 172.21.15.189:0/3439649093 wait complete. 2024-04-06T12:27:04.140 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:24:59.828905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:25:01.020253+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:27:04.140 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:27:04.168 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:04.171+0000 7fe88a431200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:27:04.191 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:04.194+0000 7fbc5e4e5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:27:04.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.272+0000 7fb27650b700 1 Processor -- start 2024-04-06T12:27:04.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.272+0000 7fb27650b700 1 -- start start 2024-04-06T12:27:04.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.273+0000 7fb27650b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb270158fe0 0x7fb27006a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:04.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.273+0000 7fb27650b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb27006ac70 0x7fb27015bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:04.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.273+0000 7fb27650b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb27015f9e0 con 0x7fb27015c100 2024-04-06T12:27:04.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.273+0000 7fb27650b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb27006b0f0 con 0x7fb270158fe0 2024-04-06T12:27:04.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.273+0000 7fb27650b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27015f0d0 con 0x7fb27006ac70 2024-04-06T12:27:04.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.273+0000 7fb274aa8700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb27015c100 0x7fb2700717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34436/0 (socket says 172.21.15.189:34436) 2024-04-06T12:27:04.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.273+0000 7fb274aa8700 1 -- 172.21.15.189:0/4065140580 learned_addr learned my addr 172.21.15.189:0/4065140580 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:04.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.273+0000 7fb26f7fe700 1 --2- 172.21.15.189:0/4065140580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb27006ac70 0x7fb27015bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:04.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.273+0000 7fb26ffff700 1 --2- 172.21.15.189:0/4065140580 >> v2:172.21.15.192:3300/0 conn(0x7fb270158fe0 0x7fb27006a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:04.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.273+0000 7fb26effd700 1 -- 172.21.15.189:0/4065140580 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146010636 0 0) 0x7fb27015f9e0 con 0x7fb27015c100 2024-04-06T12:27:04.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.273+0000 7fb26effd700 1 -- 172.21.15.189:0/4065140580 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2580035b0 con 0x7fb27015c100 2024-04-06T12:27:04.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.274+0000 7fb26f7fe700 1 -- 172.21.15.189:0/4065140580 >> v2:172.21.15.192:3300/0 conn(0x7fb270158fe0 msgr2=0x7fb27006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:04.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.274+0000 7fb26f7fe700 1 --2- 172.21.15.189:0/4065140580 >> v2:172.21.15.192:3300/0 conn(0x7fb270158fe0 0x7fb27006a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.274+0000 7fb26f7fe700 1 -- 172.21.15.189:0/4065140580 >> v1:172.21.15.189:6789/0 conn(0x7fb27015c100 legacy=0x7fb2700717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:04.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.274+0000 7fb26f7fe700 1 -- 172.21.15.189:0/4065140580 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27015c8f0 con 0x7fb27006ac70 2024-04-06T12:27:04.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.274+0000 7fb26f7fe700 1 --2- 172.21.15.189:0/4065140580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb27006ac70 0x7fb27015bbc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb264009840 tx=0x7fb264009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5db98e990b9d2b9c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:04.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.274+0000 7fb26effd700 1 -- 172.21.15.189:0/4065140580 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264011830 con 0x7fb27006ac70 2024-04-06T12:27:04.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.274+0000 7fb26effd700 1 -- 172.21.15.189:0/4065140580 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb264011990 con 0x7fb27006ac70 2024-04-06T12:27:04.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.274+0000 7fb26effd700 1 -- 172.21.15.189:0/4065140580 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264011d20 con 0x7fb27006ac70 2024-04-06T12:27:04.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.275+0000 7fb27650b700 1 -- 172.21.15.189:0/4065140580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb27006ac70 msgr2=0x7fb27015bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:04.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.275+0000 7fb27650b700 1 --2- 172.21.15.189:0/4065140580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb27006ac70 0x7fb27015bbc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb264009840 tx=0x7fb264009b50 comp rx=0 tx=0).stop 2024-04-06T12:27:04.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.275+0000 7fb27650b700 1 -- 172.21.15.189:0/4065140580 shutdown_connections 2024-04-06T12:27:04.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.275+0000 7fb27650b700 1 --2- 172.21.15.189:0/4065140580 >> v2:172.21.15.192:3300/0 conn(0x7fb270158fe0 0x7fb27006a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.275+0000 7fb27650b700 1 --2- 172.21.15.189:0/4065140580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb27006ac70 0x7fb27015bbc0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.275+0000 7fb27650b700 1 -- 172.21.15.189:0/4065140580 >> 172.21.15.189:0/4065140580 conn(0x7fb2700d00b0 msgr2=0x7fb2700d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:04.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.275+0000 7fb27650b700 1 -- 172.21.15.189:0/4065140580 shutdown_connections 2024-04-06T12:27:04.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.275+0000 7fb27650b700 1 -- 172.21.15.189:0/4065140580 wait complete. 2024-04-06T12:27:04.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.276+0000 7fb27650b700 1 Processor -- start 2024-04-06T12:27:04.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.276+0000 7fb27650b700 1 -- start start 2024-04-06T12:27:04.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.277+0000 7fb27650b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb27006ac70 0x7fb2701f35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:04.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.277+0000 7fb27650b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb270158fe0 0x7fb2700ad280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:04.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.277+0000 7fb27650b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb27015c100 0x7fb2701f6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:04.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.277+0000 7fb27650b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2701f7000 con 0x7fb270158fe0 2024-04-06T12:27:04.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.277+0000 7fb27650b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2701f2b60 con 0x7fb27015c100 2024-04-06T12:27:04.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.277+0000 7fb27650b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2700ad960 con 0x7fb27006ac70 2024-04-06T12:27:04.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.277+0000 7fb26f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb270158fe0 0x7fb2700ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:04.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.277+0000 7fb26f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb270158fe0 0x7fb2700ad280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36406/0 (socket says 172.21.15.189:36406) 2024-04-06T12:27:04.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.277+0000 7fb26f7fe700 1 -- 172.21.15.189:0/3595234809 learned_addr learned my addr 172.21.15.189:0/3595234809 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:04.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.277+0000 7fb26ffff700 1 --2- 172.21.15.189:0/3595234809 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb27006ac70 0x7fb2701f35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:04.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.277+0000 7fb274aa8700 1 --2- 172.21.15.189:0/3595234809 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb27015c100 0x7fb2701f6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:04.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.277+0000 7fb26f7fe700 1 -- 172.21.15.189:0/3595234809 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb27006ac70 msgr2=0x7fb2701f35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:04.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.277+0000 7fb26f7fe700 1 --2- 172.21.15.189:0/3595234809 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb27006ac70 0x7fb2701f35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.277+0000 7fb26f7fe700 1 -- 172.21.15.189:0/3595234809 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb27015c100 msgr2=0x7fb2701f6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:04.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.277+0000 7fb26f7fe700 1 --2- 172.21.15.189:0/3595234809 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb27015c100 0x7fb2701f6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.277+0000 7fb26f7fe700 1 -- 172.21.15.189:0/3595234809 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2640094a0 con 0x7fb270158fe0 2024-04-06T12:27:04.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.277+0000 7fb26ffff700 1 --2- 172.21.15.189:0/3595234809 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb27006ac70 0x7fb2701f35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:04.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.278+0000 7fb26f7fe700 1 --2- 172.21.15.189:0/3595234809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb270158fe0 0x7fb2700ad280 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fb264009fd0 tx=0x7fb264004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:04.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.278+0000 7fb26cff9700 1 -- 172.21.15.189:0/3595234809 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2640036a0 con 0x7fb270158fe0 2024-04-06T12:27:04.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.278+0000 7fb26cff9700 1 -- 172.21.15.189:0/3595234809 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26401e070 con 0x7fb270158fe0 2024-04-06T12:27:04.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.278+0000 7fb27650b700 1 -- 172.21.15.189:0/3595234809 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2701f2dd0 con 0x7fb270158fe0 2024-04-06T12:27:04.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.278+0000 7fb26cff9700 1 -- 172.21.15.189:0/3595234809 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264003a60 con 0x7fb270158fe0 2024-04-06T12:27:04.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.278+0000 7fb27650b700 1 -- 172.21.15.189:0/3595234809 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2700adc40 con 0x7fb270158fe0 2024-04-06T12:27:04.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.279+0000 7fb27650b700 1 -- 172.21.15.189:0/3595234809 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2701f32f0 con 0x7fb270158fe0 2024-04-06T12:27:04.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.279+0000 7fb26cff9700 1 -- 172.21.15.189:0/3595234809 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 28) v1 ==== 89259+0+0 (secure 0 0 0) 0x7fb264003bc0 con 0x7fb270158fe0 2024-04-06T12:27:04.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.280+0000 7fb26cff9700 1 --2- 172.21.15.189:0/3595234809 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25c06adb0 0x7fb25c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:04.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.280+0000 7fb26ffff700 1 -- 172.21.15.189:0/3595234809 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25c06adb0 msgr2=0x7fb25c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:04.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.280+0000 7fb26cff9700 1 -- 172.21.15.189:0/3595234809 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb26408a550 con 0x7fb270158fe0 2024-04-06T12:27:04.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.280+0000 7fb26ffff700 1 --2- 172.21.15.189:0/3595234809 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25c06adb0 0x7fb25c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:04.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.284+0000 7fb26cff9700 1 -- 172.21.15.189:0/3595234809 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb2701f32f0 con 0x7fb270158fe0 2024-04-06T12:27:04.344 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:04.347+0000 7fe88a431200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:27:04.376 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:04.379+0000 7fbc5e4e5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:27:04.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.481+0000 7fb26ffff700 1 -- 172.21.15.189:0/3595234809 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25c06adb0 msgr2=0x7fb25c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:04.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.481+0000 7fb26ffff700 1 --2- 172.21.15.189:0/3595234809 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25c06adb0 0x7fb25c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:04.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.485+0000 7fb27650b700 1 -- 172.21.15.189:0/3595234809 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb2701f32f0 con 0x7fb270158fe0 2024-04-06T12:27:04.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.486+0000 7fb26cff9700 1 -- 172.21.15.189:0/3595234809 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb2701f32f0 con 0x7fb270158fe0 2024-04-06T12:27:04.483 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:27:04.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.488+0000 7fb27650b700 1 -- 172.21.15.189:0/3595234809 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25c06adb0 msgr2=0x7fb25c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:04.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.488+0000 7fb27650b700 1 --2- 172.21.15.189:0/3595234809 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25c06adb0 0x7fb25c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.488+0000 7fb27650b700 1 -- 172.21.15.189:0/3595234809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb270158fe0 msgr2=0x7fb2700ad280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:04.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.488+0000 7fb27650b700 1 --2- 172.21.15.189:0/3595234809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb270158fe0 0x7fb2700ad280 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fb264009fd0 tx=0x7fb264004750 comp rx=0 tx=0).stop 2024-04-06T12:27:04.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.488+0000 7fb27650b700 1 -- 172.21.15.189:0/3595234809 shutdown_connections 2024-04-06T12:27:04.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.488+0000 7fb27650b700 1 --2- 172.21.15.189:0/3595234809 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25c06adb0 0x7fb25c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.488+0000 7fb27650b700 1 --2- 172.21.15.189:0/3595234809 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb27015c100 0x7fb2701f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.488+0000 7fb27650b700 1 --2- 172.21.15.189:0/3595234809 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb27006ac70 0x7fb2701f35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.488+0000 7fb27650b700 1 --2- 172.21.15.189:0/3595234809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb270158fe0 0x7fb2700ad280 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.488+0000 7fb27650b700 1 -- 172.21.15.189:0/3595234809 >> 172.21.15.189:0/3595234809 conn(0x7fb2700d00b0 msgr2=0x7fb27006fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:04.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.488+0000 7fb27650b700 1 -- 172.21.15.189:0/3595234809 shutdown_connections 2024-04-06T12:27:04.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.488+0000 7fb27650b700 1 -- 172.21.15.189:0/3595234809 wait complete. 2024-04-06T12:27:04.499 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:27:04.499 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:04.502+0000 7fe88a431200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:27:04.540 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:04.543+0000 7fbc5e4e5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:27:04.608 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:04.611+0000 7fe88a431200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:27:04.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.631+0000 7fbfa8e27700 1 Processor -- start 2024-04-06T12:27:04.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.631+0000 7fbfa8e27700 1 -- start start 2024-04-06T12:27:04.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.632+0000 7fbfa8e27700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfa40cb870 0x7fbfa415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:04.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.632+0000 7fbfa8e27700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbfa40cfa00 0x7fbfa40cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:04.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.632+0000 7fbfa8e27700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa40cbd80 con 0x7fbfa40d11b0 2024-04-06T12:27:04.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.632+0000 7fbfa8e27700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa40ccdc0 con 0x7fbfa40cfa00 2024-04-06T12:27:04.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.632+0000 7fbfa8e27700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa40d0b50 con 0x7fbfa40cb870 2024-04-06T12:27:04.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.632+0000 7fbfa2d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbfa40cfa00 0x7fbfa40cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:04.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.632+0000 7fbfa2d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbfa40cfa00 0x7fbfa40cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36418/0 (socket says 172.21.15.189:36418) 2024-04-06T12:27:04.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.632+0000 7fbfa2d9d700 1 -- 172.21.15.189:0/609762589 learned_addr learned my addr 172.21.15.189:0/609762589 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:04.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.632+0000 7fbfa259c700 1 --1- 172.21.15.189:0/609762589 >> v1:172.21.15.189:6789/0 conn(0x7fbfa40d11b0 0x7fbfa40d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34438/0 (socket says 172.21.15.189:34438) 2024-04-06T12:27:04.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.632+0000 7fbfa1d9b700 1 --2- 172.21.15.189:0/609762589 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfa40cb870 0x7fbfa415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:04.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.633+0000 7fbfa2d9d700 1 -- 172.21.15.189:0/609762589 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfa40cb870 msgr2=0x7fbfa415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:04.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.633+0000 7fbfa2d9d700 1 --2- 172.21.15.189:0/609762589 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfa40cb870 0x7fbfa415c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.633+0000 7fbfa2d9d700 1 -- 172.21.15.189:0/609762589 >> v1:172.21.15.189:6789/0 conn(0x7fbfa40d11b0 legacy=0x7fbfa40d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:04.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.633+0000 7fbfa2d9d700 1 -- 172.21.15.189:0/609762589 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa40cef20 con 0x7fbfa40cfa00 2024-04-06T12:27:04.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.633+0000 7fbfa2d9d700 1 --2- 172.21.15.189:0/609762589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfa40cfa00 0x7fbfa40cd560 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c009e50 tx=0x7fbf9c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a58950371e9d19 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:04.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.633+0000 7fbfa159a700 1 -- 172.21.15.189:0/609762589 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c00e070 con 0x7fbfa40cfa00 2024-04-06T12:27:04.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.633+0000 7fbfa159a700 1 -- 172.21.15.189:0/609762589 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9c00b910 con 0x7fbfa40cfa00 2024-04-06T12:27:04.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.633+0000 7fbfa159a700 1 -- 172.21.15.189:0/609762589 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c00bcc0 con 0x7fbfa40cfa00 2024-04-06T12:27:04.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.634+0000 7fbfa8e27700 1 -- 172.21.15.189:0/609762589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfa40cfa00 msgr2=0x7fbfa40cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:04.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.634+0000 7fbfa8e27700 1 --2- 172.21.15.189:0/609762589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfa40cfa00 0x7fbfa40cd560 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c009e50 tx=0x7fbf9c00b330 comp rx=0 tx=0).stop 2024-04-06T12:27:04.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.634+0000 7fbfa8e27700 1 -- 172.21.15.189:0/609762589 shutdown_connections 2024-04-06T12:27:04.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.634+0000 7fbfa8e27700 1 --2- 172.21.15.189:0/609762589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfa40cfa00 0x7fbfa40cd560 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.634+0000 7fbfa8e27700 1 --2- 172.21.15.189:0/609762589 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfa40cb870 0x7fbfa415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.634+0000 7fbfa8e27700 1 -- 172.21.15.189:0/609762589 >> 172.21.15.189:0/609762589 conn(0x7fbfa40b2db0 msgr2=0x7fbfa40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:04.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.634+0000 7fbfa8e27700 1 -- 172.21.15.189:0/609762589 shutdown_connections 2024-04-06T12:27:04.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.634+0000 7fbfa8e27700 1 -- 172.21.15.189:0/609762589 wait complete. 2024-04-06T12:27:04.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.635+0000 7fbfa8e27700 1 Processor -- start 2024-04-06T12:27:04.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.635+0000 7fbfa8e27700 1 -- start start 2024-04-06T12:27:04.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.636+0000 7fbfa8e27700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfa40cb870 0x7fbfa4157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:04.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.636+0000 7fbfa8e27700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbfa40cfa00 0x7fbfa40c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:04.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.636+0000 7fbfa8e27700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfa40d11b0 0x7fbfa40c9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:04.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.636+0000 7fbfa8e27700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa415e6f0 con 0x7fbfa40cb870 2024-04-06T12:27:04.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.636+0000 7fbfa8e27700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa415db20 con 0x7fbfa40cfa00 2024-04-06T12:27:04.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.636+0000 7fbfa8e27700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa40b1b60 con 0x7fbfa40d11b0 2024-04-06T12:27:04.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.637+0000 7fbfa259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfa40cb870 0x7fbfa4157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:04.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.637+0000 7fbfa259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfa40cb870 0x7fbfa4157050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36424/0 (socket says 172.21.15.189:36424) 2024-04-06T12:27:04.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.637+0000 7fbfa259c700 1 -- 172.21.15.189:0/4225752619 learned_addr learned my addr 172.21.15.189:0/4225752619 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:04.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.637+0000 7fbfa2d9d700 1 --2- 172.21.15.189:0/4225752619 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfa40d11b0 0x7fbfa40c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:04.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.637+0000 7fbfa1d9b700 1 --2- 172.21.15.189:0/4225752619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbfa40cfa00 0x7fbfa40c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:04.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.637+0000 7fbfa259c700 1 -- 172.21.15.189:0/4225752619 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfa40d11b0 msgr2=0x7fbfa40c9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:04.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.637+0000 7fbfa259c700 1 --2- 172.21.15.189:0/4225752619 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfa40d11b0 0x7fbfa40c9240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.637+0000 7fbfa259c700 1 -- 172.21.15.189:0/4225752619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbfa40cfa00 msgr2=0x7fbfa40c8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:04.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.637+0000 7fbfa259c700 1 --2- 172.21.15.189:0/4225752619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbfa40cfa00 0x7fbfa40c8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.637+0000 7fbfa259c700 1 -- 172.21.15.189:0/4225752619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c009b30 con 0x7fbfa40cb870 2024-04-06T12:27:04.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.637+0000 7fbfa259c700 1 --2- 172.21.15.189:0/4225752619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfa40cb870 0x7fbfa4157050 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fbf9400c500 tx=0x7fbf9400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:04.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.637+0000 7fbf92ffd700 1 -- 172.21.15.189:0/4225752619 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94005cb0 con 0x7fbfa40cb870 2024-04-06T12:27:04.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.637+0000 7fbf92ffd700 1 -- 172.21.15.189:0/4225752619 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf94005e10 con 0x7fbfa40cb870 2024-04-06T12:27:04.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.637+0000 7fbfa8e27700 1 -- 172.21.15.189:0/4225752619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa415ddc0 con 0x7fbfa40cb870 2024-04-06T12:27:04.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.637+0000 7fbf92ffd700 1 -- 172.21.15.189:0/4225752619 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf940072d0 con 0x7fbfa40cb870 2024-04-06T12:27:04.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.637+0000 7fbfa8e27700 1 -- 172.21.15.189:0/4225752619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa415e380 con 0x7fbfa40cb870 2024-04-06T12:27:04.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.639+0000 7fbfa8e27700 1 -- 172.21.15.189:0/4225752619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa4074990 con 0x7fbfa40cb870 2024-04-06T12:27:04.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.639+0000 7fbf92ffd700 1 -- 172.21.15.189:0/4225752619 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 28) v1 ==== 89259+0+0 (secure 0 0 0) 0x7fbf94007430 con 0x7fbfa40cb870 2024-04-06T12:27:04.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.639+0000 7fbf92ffd700 1 --2- 172.21.15.189:0/4225752619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf8406adb0 0x7fbf8406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:04.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.639+0000 7fbfa1d9b700 1 -- 172.21.15.189:0/4225752619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf8406adb0 msgr2=0x7fbf8406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:04.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.639+0000 7fbf92ffd700 1 -- 172.21.15.189:0/4225752619 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fbf940888c0 con 0x7fbfa40cb870 2024-04-06T12:27:04.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.639+0000 7fbfa1d9b700 1 --2- 172.21.15.189:0/4225752619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf8406adb0 0x7fbf8406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:04.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.644+0000 7fbf92ffd700 1 -- 172.21.15.189:0/4225752619 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbf94053800 con 0x7fbfa40cb870 2024-04-06T12:27:04.657 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:04.660+0000 7fbc5e4e5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:27:04.690 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:04.693+0000 7fe88a431200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:27:04.742 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:04.745+0000 7fbc5e4e5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:27:04.765 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:04.767+0000 7fe88a431200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:27:04.819 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:04.822+0000 7fbc5e4e5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:27:04.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.840+0000 7fbfa1d9b700 1 -- 172.21.15.189:0/4225752619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf8406adb0 msgr2=0x7fbf8406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:04.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.840+0000 7fbfa1d9b700 1 --2- 172.21.15.189:0/4225752619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf8406adb0 0x7fbf8406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:04.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.848+0000 7fbfa8e27700 1 -- 172.21.15.189:0/4225752619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbfa40b0a00 con 0x7fbfa40cb870 2024-04-06T12:27:04.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.849+0000 7fbf92ffd700 1 -- 172.21.15.189:0/4225752619 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbf94017020 con 0x7fbfa40cb870 2024-04-06T12:27:04.846 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:27:04.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.851+0000 7fbfa8e27700 1 -- 172.21.15.189:0/4225752619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf8406adb0 msgr2=0x7fbf8406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:04.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.851+0000 7fbfa8e27700 1 --2- 172.21.15.189:0/4225752619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf8406adb0 0x7fbf8406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.851+0000 7fbfa8e27700 1 -- 172.21.15.189:0/4225752619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfa40cb870 msgr2=0x7fbfa4157050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:04.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.851+0000 7fbfa8e27700 1 --2- 172.21.15.189:0/4225752619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfa40cb870 0x7fbfa4157050 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fbf9400c500 tx=0x7fbf9400c810 comp rx=0 tx=0).stop 2024-04-06T12:27:04.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.851+0000 7fbfa8e27700 1 -- 172.21.15.189:0/4225752619 shutdown_connections 2024-04-06T12:27:04.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.851+0000 7fbfa8e27700 1 --2- 172.21.15.189:0/4225752619 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfa40d11b0 0x7fbfa40c9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.851+0000 7fbfa8e27700 1 --2- 172.21.15.189:0/4225752619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfa40cb870 0x7fbfa4157050 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.851+0000 7fbfa8e27700 1 --2- 172.21.15.189:0/4225752619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf8406adb0 0x7fbf8406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.851+0000 7fbfa8e27700 1 --2- 172.21.15.189:0/4225752619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbfa40cfa00 0x7fbfa40c8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:04.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.851+0000 7fbfa8e27700 1 -- 172.21.15.189:0/4225752619 >> 172.21.15.189:0/4225752619 conn(0x7fbfa40b2db0 msgr2=0x7fbfa406f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:04.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.851+0000 7fbfa8e27700 1 -- 172.21.15.189:0/4225752619 shutdown_connections 2024-04-06T12:27:04.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:04.851+0000 7fbfa8e27700 1 -- 172.21.15.189:0/4225752619 wait complete. 2024-04-06T12:27:04.861 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:27:04.969 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:04.968+0000 7fe88a431200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:27:05.038 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:05.037+0000 7fbc5e4e5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:27:05.041 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:05.044+0000 7fe88a431200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:27:05.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.058+0000 7f077d822700 1 Processor -- start 2024-04-06T12:27:05.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.058+0000 7f077d822700 1 -- start start 2024-04-06T12:27:05.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.058+0000 7f077d822700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f07780c2a70 0x7f077806d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:05.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.058+0000 7f077d822700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f07780cdaf0 0x7f077806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:05.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.058+0000 7f077d822700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0778161d60 con 0x7f0778161980 2024-04-06T12:27:05.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.058+0000 7f077d822700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f077806c130 con 0x7f07780cdaf0 2024-04-06T12:27:05.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.058+0000 7f077d822700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f077806b5b0 con 0x7f07780c2a70 2024-04-06T12:27:05.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.059+0000 7f07777fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f07780cdaf0 0x7f077806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:05.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.059+0000 7f07777fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f07780cdaf0 0x7f077806d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36426/0 (socket says 172.21.15.189:36426) 2024-04-06T12:27:05.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.059+0000 7f07777fe700 1 -- 172.21.15.189:0/642481105 learned_addr learned my addr 172.21.15.189:0/642481105 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:05.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.059+0000 7f07767fc700 1 --2- 172.21.15.189:0/642481105 >> v2:172.21.15.192:3300/0 conn(0x7f07780c2a70 0x7f077806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:05.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.059+0000 7f07777fe700 1 -- 172.21.15.189:0/642481105 >> v2:172.21.15.192:3300/0 conn(0x7f07780c2a70 msgr2=0x7f077806d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:05.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.059+0000 7f07777fe700 1 --2- 172.21.15.189:0/642481105 >> v2:172.21.15.192:3300/0 conn(0x7f07780c2a70 0x7f077806d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.059+0000 7f07777fe700 1 -- 172.21.15.189:0/642481105 >> v1:172.21.15.192:6789/0 conn(0x7f0778161980 legacy=0x7f07780cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:05.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.059+0000 7f07777fe700 1 -- 172.21.15.189:0/642481105 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f077806b890 con 0x7f07780cdaf0 2024-04-06T12:27:05.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.059+0000 7f07777fe700 1 --2- 172.21.15.189:0/642481105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07780cdaf0 0x7f077806d7b0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f0778071f60 tx=0x7f077000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=143628858e65642 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:05.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.059+0000 7f0775ffb700 1 -- 172.21.15.189:0/642481105 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077000e070 con 0x7f07780cdaf0 2024-04-06T12:27:05.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.059+0000 7f0775ffb700 1 -- 172.21.15.189:0/642481105 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077000b910 con 0x7f07780cdaf0 2024-04-06T12:27:05.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.059+0000 7f0775ffb700 1 -- 172.21.15.189:0/642481105 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077000bcc0 con 0x7f07780cdaf0 2024-04-06T12:27:05.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.060+0000 7f077d822700 1 -- 172.21.15.189:0/642481105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07780cdaf0 msgr2=0x7f077806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:05.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.060+0000 7f077d822700 1 --2- 172.21.15.189:0/642481105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07780cdaf0 0x7f077806d7b0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f0778071f60 tx=0x7f077000b330 comp rx=0 tx=0).stop 2024-04-06T12:27:05.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.060+0000 7f077d822700 1 -- 172.21.15.189:0/642481105 shutdown_connections 2024-04-06T12:27:05.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.060+0000 7f077d822700 1 --2- 172.21.15.189:0/642481105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07780cdaf0 0x7f077806d7b0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.060+0000 7f077d822700 1 --2- 172.21.15.189:0/642481105 >> v2:172.21.15.192:3300/0 conn(0x7f07780c2a70 0x7f077806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.060+0000 7f077d822700 1 -- 172.21.15.189:0/642481105 >> 172.21.15.189:0/642481105 conn(0x7f07780cc290 msgr2=0x7f07780cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:05.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.060+0000 7f077d822700 1 -- 172.21.15.189:0/642481105 shutdown_connections 2024-04-06T12:27:05.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.060+0000 7f077d822700 1 -- 172.21.15.189:0/642481105 wait complete. 2024-04-06T12:27:05.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.061+0000 7f077d822700 1 Processor -- start 2024-04-06T12:27:05.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.061+0000 7f077d822700 1 -- start start 2024-04-06T12:27:05.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.062+0000 7f077d822700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07780c2a70 0x7f07780c2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:05.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.062+0000 7f077d822700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07780cdaf0 0x7f07780be3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:05.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.062+0000 7f077d822700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0778161980 0x7f0778163e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:05.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.062+0000 7f077d822700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07780be910 con 0x7f07780c2a70 2024-04-06T12:27:05.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.062+0000 7f077d822700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07780bebc0 con 0x7f07780cdaf0 2024-04-06T12:27:05.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.062+0000 7f077d822700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07780beea0 con 0x7f0778161980 2024-04-06T12:27:05.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.062+0000 7f0776ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07780c2a70 0x7f07780c2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:05.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.062+0000 7f0776ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07780c2a70 0x7f07780c2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36428/0 (socket says 172.21.15.189:36428) 2024-04-06T12:27:05.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.062+0000 7f0776ffd700 1 -- 172.21.15.189:0/3895780250 learned_addr learned my addr 172.21.15.189:0/3895780250 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:05.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.062+0000 7f07767fc700 1 --2- 172.21.15.189:0/3895780250 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07780cdaf0 0x7f07780be3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:05.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.062+0000 7f07777fe700 1 --2- 172.21.15.189:0/3895780250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0778161980 0x7f0778163e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:05.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.063+0000 7f0776ffd700 1 -- 172.21.15.189:0/3895780250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0778161980 msgr2=0x7f0778163e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:05.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.063+0000 7f0776ffd700 1 --2- 172.21.15.189:0/3895780250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0778161980 0x7f0778163e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.063+0000 7f0776ffd700 1 -- 172.21.15.189:0/3895780250 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07780cdaf0 msgr2=0x7f07780be3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:05.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.063+0000 7f0776ffd700 1 --2- 172.21.15.189:0/3895780250 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07780cdaf0 0x7f07780be3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.063+0000 7f0776ffd700 1 -- 172.21.15.189:0/3895780250 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0770009b30 con 0x7f07780c2a70 2024-04-06T12:27:05.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.063+0000 7f07767fc700 1 --2- 172.21.15.189:0/3895780250 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07780cdaf0 0x7f07780be3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:05.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.063+0000 7f0776ffd700 1 --2- 172.21.15.189:0/3895780250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07780c2a70 0x7f07780c2510 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f076800c4e0 tx=0x7f076800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:05.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.063+0000 7f075ffff700 1 -- 172.21.15.189:0/3895780250 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07680052d0 con 0x7f07780c2a70 2024-04-06T12:27:05.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.063+0000 7f075ffff700 1 -- 172.21.15.189:0/3895780250 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0768005430 con 0x7f07780c2a70 2024-04-06T12:27:05.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.063+0000 7f077d822700 1 -- 172.21.15.189:0/3895780250 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07780bf180 con 0x7f07780c2a70 2024-04-06T12:27:05.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.063+0000 7f075ffff700 1 -- 172.21.15.189:0/3895780250 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07680038c0 con 0x7f07780c2a70 2024-04-06T12:27:05.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.063+0000 7f077d822700 1 -- 172.21.15.189:0/3895780250 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07780bf740 con 0x7f07780c2a70 2024-04-06T12:27:05.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.064+0000 7f077d822700 1 -- 172.21.15.189:0/3895780250 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07780bf960 con 0x7f07780c2a70 2024-04-06T12:27:05.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.064+0000 7f075ffff700 1 -- 172.21.15.189:0/3895780250 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 28) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f0768003a20 con 0x7f07780c2a70 2024-04-06T12:27:05.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.065+0000 7f075ffff700 1 --2- 172.21.15.189:0/3895780250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f076406adb0 0x7f076406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:05.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.065+0000 7f075ffff700 1 -- 172.21.15.189:0/3895780250 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f0768088900 con 0x7f07780c2a70 2024-04-06T12:27:05.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.065+0000 7f07767fc700 1 -- 172.21.15.189:0/3895780250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f076406adb0 msgr2=0x7f076406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:05.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.065+0000 7f07767fc700 1 --2- 172.21.15.189:0/3895780250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f076406adb0 0x7f076406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:05.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.069+0000 7f075ffff700 1 -- 172.21.15.189:0/3895780250 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0768053790 con 0x7f07780c2a70 2024-04-06T12:27:05.113 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:05.116+0000 7fbc5e4e5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:27:05.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.266+0000 7f07767fc700 1 -- 172.21.15.189:0/3895780250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f076406adb0 msgr2=0x7f076406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:05.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.266+0000 7f07767fc700 1 --2- 172.21.15.189:0/3895780250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f076406adb0 0x7f076406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:05.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.273+0000 7f077d822700 1 -- 172.21.15.189:0/3895780250 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0778161d60 con 0x7f07780c2a70 2024-04-06T12:27:05.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.274+0000 7f075ffff700 1 -- 172.21.15.189:0/3895780250 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0768017020 con 0x7f07780c2a70 2024-04-06T12:27:05.271 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:27:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.276+0000 7f077d822700 1 -- 172.21.15.189:0/3895780250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f076406adb0 msgr2=0x7f076406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.276+0000 7f077d822700 1 --2- 172.21.15.189:0/3895780250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f076406adb0 0x7f076406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.276+0000 7f077d822700 1 -- 172.21.15.189:0/3895780250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07780c2a70 msgr2=0x7f07780c2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.276+0000 7f077d822700 1 --2- 172.21.15.189:0/3895780250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07780c2a70 0x7f07780c2510 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f076800c4e0 tx=0x7f076800c7f0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.276+0000 7f077d822700 1 -- 172.21.15.189:0/3895780250 shutdown_connections 2024-04-06T12:27:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.276+0000 7f077d822700 1 --2- 172.21.15.189:0/3895780250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0778161980 0x7f0778163e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.276+0000 7f077d822700 1 --2- 172.21.15.189:0/3895780250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07780c2a70 0x7f07780c2510 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.276+0000 7f077d822700 1 --2- 172.21.15.189:0/3895780250 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f076406adb0 0x7f076406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.276+0000 7f077d822700 1 --2- 172.21.15.189:0/3895780250 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07780cdaf0 0x7f07780be3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.276+0000 7f077d822700 1 -- 172.21.15.189:0/3895780250 >> 172.21.15.189:0/3895780250 conn(0x7f07780cc290 msgr2=0x7f07780b0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.276+0000 7f077d822700 1 -- 172.21.15.189:0/3895780250 shutdown_connections 2024-04-06T12:27:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.276+0000 7f077d822700 1 -- 172.21.15.189:0/3895780250 wait complete. 2024-04-06T12:27:05.288 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T12:27:05.357 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:05.360+0000 7fe88a431200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:27:05.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.416+0000 7feeb77f4700 1 Processor -- start 2024-04-06T12:27:05.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.417+0000 7feeb77f4700 1 -- start start 2024-04-06T12:27:05.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.417+0000 7feeb77f4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7feeb006ae40 0x7feeb00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:05.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.417+0000 7feeb77f4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb00b3990 con 0x7feeb015ddf0 2024-04-06T12:27:05.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.417+0000 7feeb77f4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb00b3060 con 0x7feeb015d7b0 2024-04-06T12:27:05.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.417+0000 7feeb77f4700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeb015e950 con 0x7feeb006ae40 2024-04-06T12:27:05.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.418+0000 7feeb4d8f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7feeb006ae40 0x7feeb00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:05.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.418+0000 7feeb4d8f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7feeb006ae40 0x7feeb00b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36438/0 (socket says 172.21.15.189:36438) 2024-04-06T12:27:05.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.418+0000 7feeb4d8f700 1 -- 172.21.15.189:0/2156246594 learned_addr learned my addr 172.21.15.189:0/2156246594 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:05.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.418+0000 7feeb4d8f700 1 -- 172.21.15.189:0/2156246594 >> v1:172.21.15.192:6789/0 conn(0x7feeb015d7b0 legacy=0x7feeb006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:05.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.418+0000 7feeb4d8f700 1 -- 172.21.15.189:0/2156246594 >> v1:172.21.15.189:6789/0 conn(0x7feeb015ddf0 legacy=0x7feeb015e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:05.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.418+0000 7feeb4d8f700 1 -- 172.21.15.189:0/2156246594 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb00b3340 con 0x7feeb006ae40 2024-04-06T12:27:05.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.418+0000 7feeb4d8f700 1 --2- 172.21.15.189:0/2156246594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeb006ae40 0x7feeb00b3e20 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7feeac0098f0 tx=0x7feeac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e30d304d618b184a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:05.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.418+0000 7feea7fff700 1 -- 172.21.15.189:0/2156246594 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac00b8a0 con 0x7feeb006ae40 2024-04-06T12:27:05.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.418+0000 7feea7fff700 1 -- 172.21.15.189:0/2156246594 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeac00ba00 con 0x7feeb006ae40 2024-04-06T12:27:05.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.418+0000 7feea7fff700 1 -- 172.21.15.189:0/2156246594 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac00bd90 con 0x7feeb006ae40 2024-04-06T12:27:05.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.419+0000 7feeb77f4700 1 -- 172.21.15.189:0/2156246594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeb006ae40 msgr2=0x7feeb00b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:05.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.419+0000 7feeb77f4700 1 --2- 172.21.15.189:0/2156246594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeb006ae40 0x7feeb00b3e20 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7feeac0098f0 tx=0x7feeac009c00 comp rx=0 tx=0).stop 2024-04-06T12:27:05.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.419+0000 7feeb77f4700 1 -- 172.21.15.189:0/2156246594 shutdown_connections 2024-04-06T12:27:05.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.419+0000 7feeb77f4700 1 --2- 172.21.15.189:0/2156246594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeb006ae40 0x7feeb00b3e20 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.419+0000 7feeb77f4700 1 -- 172.21.15.189:0/2156246594 >> 172.21.15.189:0/2156246594 conn(0x7feeb00d00b0 msgr2=0x7feeb00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:05.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.419+0000 7feeb77f4700 1 -- 172.21.15.189:0/2156246594 shutdown_connections 2024-04-06T12:27:05.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.419+0000 7feeb77f4700 1 -- 172.21.15.189:0/2156246594 wait complete. 2024-04-06T12:27:05.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.420+0000 7feeb77f4700 1 Processor -- start 2024-04-06T12:27:05.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.420+0000 7feeb77f4700 1 -- start start 2024-04-06T12:27:05.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.421+0000 7feeb77f4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feeb006ae40 0x7feeb0152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:05.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.421+0000 7feeb77f4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feeb015d7b0 0x7feeb00b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:05.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.421+0000 7feeb77f4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeb015ddf0 0x7feeb00aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:05.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.421+0000 7feeb77f4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb00cc870 con 0x7feeb015ddf0 2024-04-06T12:27:05.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.421+0000 7feeb77f4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb00cdcb0 con 0x7feeb006ae40 2024-04-06T12:27:05.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.421+0000 7feeb77f4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb01525d0 con 0x7feeb015d7b0 2024-04-06T12:27:05.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.421+0000 7feeb5d91700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeb015ddf0 0x7feeb00aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:05.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.421+0000 7feeb5d91700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeb015ddf0 0x7feeb00aeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36454/0 (socket says 172.21.15.189:36454) 2024-04-06T12:27:05.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.421+0000 7feeb5d91700 1 -- 172.21.15.189:0/171056642 learned_addr learned my addr 172.21.15.189:0/171056642 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:05.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.421+0000 7feeb5590700 1 --2- 172.21.15.189:0/171056642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feeb006ae40 0x7feeb0152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:05.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.421+0000 7feeb4d8f700 1 --2- 172.21.15.189:0/171056642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feeb015d7b0 0x7feeb00b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:05.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.421+0000 7feeb5d91700 1 -- 172.21.15.189:0/171056642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feeb015d7b0 msgr2=0x7feeb00b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:05.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.421+0000 7feeb5d91700 1 --2- 172.21.15.189:0/171056642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feeb015d7b0 0x7feeb00b11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.421+0000 7feeb5d91700 1 -- 172.21.15.189:0/171056642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feeb006ae40 msgr2=0x7feeb0152090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:05.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.421+0000 7feeb5d91700 1 --2- 172.21.15.189:0/171056642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feeb006ae40 0x7feeb0152090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.421+0000 7feeb5d91700 1 -- 172.21.15.189:0/171056642 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac009550 con 0x7feeb015ddf0 2024-04-06T12:27:05.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.422+0000 7feeb5d91700 1 --2- 172.21.15.189:0/171056642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeb015ddf0 0x7feeb00aeb80 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7feea800cb50 tx=0x7feea800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:05.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.422+0000 7feea5ffb700 1 -- 172.21.15.189:0/171056642 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea8003c00 con 0x7feeb015ddf0 2024-04-06T12:27:05.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.422+0000 7feea5ffb700 1 -- 172.21.15.189:0/171056642 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea8003d60 con 0x7feeb015ddf0 2024-04-06T12:27:05.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.422+0000 7feeb77f4700 1 -- 172.21.15.189:0/171056642 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeb00cdee0 con 0x7feeb015ddf0 2024-04-06T12:27:05.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.422+0000 7feea5ffb700 1 -- 172.21.15.189:0/171056642 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea800aa20 con 0x7feeb015ddf0 2024-04-06T12:27:05.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.422+0000 7feeb77f4700 1 -- 172.21.15.189:0/171056642 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb00ce4a0 con 0x7feeb015ddf0 2024-04-06T12:27:05.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.423+0000 7feeb77f4700 1 -- 172.21.15.189:0/171056642 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeb0067bb0 con 0x7feeb015ddf0 2024-04-06T12:27:05.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.424+0000 7feea5ffb700 1 -- 172.21.15.189:0/171056642 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 28) v1 ==== 89259+0+0 (secure 0 0 0) 0x7feea800ab80 con 0x7feeb015ddf0 2024-04-06T12:27:05.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.424+0000 7feea5ffb700 1 --2- 172.21.15.189:0/171056642 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fee9406adb0 0x7fee9406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:05.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.424+0000 7feeb5590700 1 -- 172.21.15.189:0/171056642 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fee9406adb0 msgr2=0x7fee9406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:05.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.425+0000 7feea5ffb700 1 -- 172.21.15.189:0/171056642 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6431+0+0 (secure 0 0 0) 0x7feea80896b0 con 0x7feeb015ddf0 2024-04-06T12:27:05.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.425+0000 7feeb5590700 1 --2- 172.21.15.189:0/171056642 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fee9406adb0 0x7fee9406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:05.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.429+0000 7feea5ffb700 1 -- 172.21.15.189:0/171056642 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7feea8054540 con 0x7feeb015ddf0 2024-04-06T12:27:05.453 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:05.456+0000 7fbc5e4e5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:27:05.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.626+0000 7feeb5590700 1 -- 172.21.15.189:0/171056642 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fee9406adb0 msgr2=0x7fee9406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:05.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.626+0000 7feeb5590700 1 --2- 172.21.15.189:0/171056642 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fee9406adb0 0x7fee9406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:05.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.648+0000 7feeb77f4700 1 -- 172.21.15.189:0/171056642 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7feeb0074990 con 0x7feeb015ddf0 2024-04-06T12:27:05.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.649+0000 7feea5ffb700 1 -- 172.21.15.189:0/171056642 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v70) v1 ==== 80+0+467 (secure 0 0 0) 0x7feea8057b60 con 0x7feeb015ddf0 2024-04-06T12:27:05.646 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1187685283 2024-04-07T12:25:15.982358+0000 2024-04-06T12:27:05.646 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1356350621 2024-04-07T12:25:15.982358+0000 2024-04-06T12:27:05.646 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/162292108 2024-04-07T12:25:15.982358+0000 2024-04-06T12:27:05.646 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2706754156 2024-04-07T12:25:15.982358+0000 2024-04-06T12:27:05.646 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1122382676 2024-04-07T12:25:15.982358+0000 2024-04-06T12:27:05.647 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2132037261 2024-04-07T12:25:15.982358+0000 2024-04-06T12:27:05.647 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6833/47128 2024-04-07T12:25:15.982358+0000 2024-04-06T12:27:05.647 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6832/47128 2024-04-07T12:25:15.982358+0000 2024-04-06T12:27:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.651+0000 7feeb77f4700 1 -- 172.21.15.189:0/171056642 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fee9406adb0 msgr2=0x7fee9406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.651+0000 7feeb77f4700 1 --2- 172.21.15.189:0/171056642 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fee9406adb0 0x7fee9406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.651+0000 7feeb77f4700 1 -- 172.21.15.189:0/171056642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeb015ddf0 msgr2=0x7feeb00aeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.651+0000 7feeb77f4700 1 --2- 172.21.15.189:0/171056642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeb015ddf0 0x7feeb00aeb80 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7feea800cb50 tx=0x7feea800ce60 comp rx=0 tx=0).stop 2024-04-06T12:27:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.651+0000 7feeb77f4700 1 -- 172.21.15.189:0/171056642 shutdown_connections 2024-04-06T12:27:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.651+0000 7feeb77f4700 1 --2- 172.21.15.189:0/171056642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeb015ddf0 0x7feeb00aeb80 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.651+0000 7feeb77f4700 1 --2- 172.21.15.189:0/171056642 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fee9406adb0 0x7fee9406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.651+0000 7feeb77f4700 1 --2- 172.21.15.189:0/171056642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feeb006ae40 0x7feeb0152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.651+0000 7feeb77f4700 1 --2- 172.21.15.189:0/171056642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feeb015d7b0 0x7feeb00b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.651+0000 7feeb77f4700 1 -- 172.21.15.189:0/171056642 >> 172.21.15.189:0/171056642 conn(0x7feeb00d00b0 msgr2=0x7feeb015f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.651+0000 7feeb77f4700 1 -- 172.21.15.189:0/171056642 shutdown_connections 2024-04-06T12:27:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.651+0000 7feeb77f4700 1 -- 172.21.15.189:0/171056642 wait complete. 2024-04-06T12:27:05.649 INFO:teuthology.orchestra.run.smithi189.stderr:listed 8 entries 2024-04-06T12:27:05.661 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-06T12:27:05.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:05.661 DEBUG:teuthology.orchestra.run.smithi189:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-06T12:27:05.675 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:05.678+0000 7fe88a431200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:27:05.705 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T12:27:05.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:05.705 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-06T12:27:05.762 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:27:05.763 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:27:05.780 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:27:05.780 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:27:05.780 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:27:05.789 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:05.792+0000 7fbc5e4e5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:27:05.806 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:27:05.806 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:27:05.806 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:27:05.837 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:27:05.838 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:27:05.842 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:05.845+0000 7fe88a431200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:27:05.915 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:05.918+0000 7fe88a431200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:27:05.959 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:05.962+0000 7fbc5e4e5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:27:05.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.975+0000 7f6f4fb8f700 1 Processor -- start 2024-04-06T12:27:05.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.976+0000 7f6f4fb8f700 1 -- start start 2024-04-06T12:27:05.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.977+0000 7f6f4fb8f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f480cb870 0x7f6f4815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:05.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.977+0000 7f6f4fb8f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6f480cfa00 0x7f6f480cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:05.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.977+0000 7f6f4fb8f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f480cbd80 con 0x7f6f480d11b0 2024-04-06T12:27:05.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.977+0000 7f6f4fb8f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f480ccdc0 con 0x7f6f480cfa00 2024-04-06T12:27:05.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.977+0000 7f6f4fb8f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f480d0b50 con 0x7f6f480cb870 2024-04-06T12:27:05.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.977+0000 7f6f4e12c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6f480cfa00 0x7f6f480cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:05.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.977+0000 7f6f4e12c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6f480cfa00 0x7f6f480cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36468/0 (socket says 172.21.15.189:36468) 2024-04-06T12:27:05.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.977+0000 7f6f4e12c700 1 -- 172.21.15.189:0/2385661513 learned_addr learned my addr 172.21.15.189:0/2385661513 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:05.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.977+0000 7f6f4d12a700 1 --2- 172.21.15.189:0/2385661513 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f480cb870 0x7f6f4815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:05.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.978+0000 7f6f4e12c700 1 -- 172.21.15.189:0/2385661513 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f480cb870 msgr2=0x7f6f4815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:05.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.978+0000 7f6f4e12c700 1 --2- 172.21.15.189:0/2385661513 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f480cb870 0x7f6f4815c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.978+0000 7f6f4e12c700 1 -- 172.21.15.189:0/2385661513 >> v1:172.21.15.192:6789/0 conn(0x7f6f480d11b0 legacy=0x7f6f480d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:05.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.978+0000 7f6f4c929700 1 -- 172.21.15.189:0/2385661513 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364712643 0 0) 0x7f6f480cbd80 con 0x7f6f480d11b0 2024-04-06T12:27:05.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.978+0000 7f6f4e12c700 1 -- 172.21.15.189:0/2385661513 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f480cef20 con 0x7f6f480cfa00 2024-04-06T12:27:05.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.978+0000 7f6f4e12c700 1 --2- 172.21.15.189:0/2385661513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f480cfa00 0x7f6f480cd560 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f6f40009e50 tx=0x7f6f4000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac4977f365a104cd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:05.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.978+0000 7f6f4c929700 1 -- 172.21.15.189:0/2385661513 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4000e070 con 0x7f6f480cfa00 2024-04-06T12:27:05.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.978+0000 7f6f4c929700 1 -- 172.21.15.189:0/2385661513 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4000b910 con 0x7f6f480cfa00 2024-04-06T12:27:05.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.978+0000 7f6f4c929700 1 -- 172.21.15.189:0/2385661513 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4000bcc0 con 0x7f6f480cfa00 2024-04-06T12:27:05.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.978+0000 7f6f4fb8f700 1 -- 172.21.15.189:0/2385661513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f480cfa00 msgr2=0x7f6f480cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:05.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.979+0000 7f6f4fb8f700 1 --2- 172.21.15.189:0/2385661513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f480cfa00 0x7f6f480cd560 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f6f40009e50 tx=0x7f6f4000b330 comp rx=0 tx=0).stop 2024-04-06T12:27:05.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.979+0000 7f6f4fb8f700 1 -- 172.21.15.189:0/2385661513 shutdown_connections 2024-04-06T12:27:05.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.979+0000 7f6f4fb8f700 1 --2- 172.21.15.189:0/2385661513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f480cfa00 0x7f6f480cd560 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.979+0000 7f6f4fb8f700 1 --2- 172.21.15.189:0/2385661513 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f480cb870 0x7f6f4815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.979+0000 7f6f4fb8f700 1 -- 172.21.15.189:0/2385661513 >> 172.21.15.189:0/2385661513 conn(0x7f6f480b2db0 msgr2=0x7f6f480b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:05.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.979+0000 7f6f4fb8f700 1 -- 172.21.15.189:0/2385661513 shutdown_connections 2024-04-06T12:27:05.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.979+0000 7f6f4fb8f700 1 -- 172.21.15.189:0/2385661513 wait complete. 2024-04-06T12:27:05.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.980+0000 7f6f4fb8f700 1 Processor -- start 2024-04-06T12:27:05.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.980+0000 7f6f4fb8f700 1 -- start start 2024-04-06T12:27:05.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.980+0000 7f6f4fb8f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f480cb870 0x7f6f480b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:05.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f4fb8f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f480cfa00 0x7f6f480cab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:05.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f4fb8f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f480d11b0 0x7f6f4806bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:05.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f4fb8f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f481557d0 con 0x7f6f480d11b0 2024-04-06T12:27:05.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f4fb8f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f480b4480 con 0x7f6f480cb870 2024-04-06T12:27:05.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f4fb8f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f480b4760 con 0x7f6f480cfa00 2024-04-06T12:27:05.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f4e12c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f480d11b0 0x7f6f4806bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:05.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f4e12c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f480d11b0 0x7f6f4806bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36482/0 (socket says 172.21.15.189:36482) 2024-04-06T12:27:05.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f4e12c700 1 -- 172.21.15.189:0/3436362411 learned_addr learned my addr 172.21.15.189:0/3436362411 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:05.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f4d12a700 1 --2- 172.21.15.189:0/3436362411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f480cfa00 0x7f6f480cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:05.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f4d92b700 1 --2- 172.21.15.189:0/3436362411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f480cb870 0x7f6f480b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:05.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f4e12c700 1 -- 172.21.15.189:0/3436362411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f480cfa00 msgr2=0x7f6f480cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:05.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f4e12c700 1 --2- 172.21.15.189:0/3436362411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f480cfa00 0x7f6f480cab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f4e12c700 1 -- 172.21.15.189:0/3436362411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f480cb870 msgr2=0x7f6f480b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:05.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f4e12c700 1 --2- 172.21.15.189:0/3436362411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f480cb870 0x7f6f480b3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:05.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f4e12c700 1 -- 172.21.15.189:0/3436362411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f40009b30 con 0x7f6f480d11b0 2024-04-06T12:27:05.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f4e12c700 1 --2- 172.21.15.189:0/3436362411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f480d11b0 0x7f6f4806bf80 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f6f40009e20 tx=0x7f6f40004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:05.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f3e7fc700 1 -- 172.21.15.189:0/3436362411 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f40004880 con 0x7f6f480d11b0 2024-04-06T12:27:05.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f3e7fc700 1 -- 172.21.15.189:0/3436362411 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f400099b0 con 0x7f6f480d11b0 2024-04-06T12:27:05.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f4fb8f700 1 -- 172.21.15.189:0/3436362411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f480b4a40 con 0x7f6f480d11b0 2024-04-06T12:27:05.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f3e7fc700 1 -- 172.21.15.189:0/3436362411 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4001c680 con 0x7f6f480d11b0 2024-04-06T12:27:05.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.981+0000 7f6f4fb8f700 1 -- 172.21.15.189:0/3436362411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f480b5000 con 0x7f6f480d11b0 2024-04-06T12:27:05.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.982+0000 7f6f4fb8f700 1 -- 172.21.15.189:0/3436362411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f48074990 con 0x7f6f480d11b0 2024-04-06T12:27:05.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.983+0000 7f6f3e7fc700 1 -- 172.21.15.189:0/3436362411 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 28) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f6f4001c7e0 con 0x7f6f480d11b0 2024-04-06T12:27:05.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.983+0000 7f6f3e7fc700 1 --2- 172.21.15.189:0/3436362411 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f2c06adb0 0x7f6f2c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:05.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.983+0000 7f6f4d92b700 1 -- 172.21.15.189:0/3436362411 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f2c06adb0 msgr2=0x7f6f2c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:05.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.983+0000 7f6f3e7fc700 1 -- 172.21.15.189:0/3436362411 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f6f4008b3a0 con 0x7f6f480d11b0 2024-04-06T12:27:05.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.983+0000 7f6f4d92b700 1 --2- 172.21.15.189:0/3436362411 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f2c06adb0 0x7f6f2c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:05.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:05.992+0000 7f6f3e7fc700 1 -- 172.21.15.189:0/3436362411 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6f40056120 con 0x7f6f480d11b0 2024-04-06T12:27:06.040 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:06.043+0000 7fbc5e4e5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:27:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.184+0000 7f6f4d92b700 1 -- 172.21.15.189:0/3436362411 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f2c06adb0 msgr2=0x7f6f2c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.184+0000 7f6f4d92b700 1 --2- 172.21.15.189:0/3436362411 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f2c06adb0 0x7f6f2c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:06.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.242+0000 7f6f4fb8f700 1 -- 172.21.15.189:0/3436362411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f480cb100 con 0x7f6f480d11b0 2024-04-06T12:27:06.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.243+0000 7f6f3e7fc700 1 -- 172.21.15.189:0/3436362411 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f6f40059740 con 0x7f6f480d11b0 2024-04-06T12:27:06.240 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:06.240 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/527093473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":527093473},{"type":"v1","addr":"172.21.15.192:6835","nonce":527093473}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6105,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2142144473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2142144473},{"type":"v1","addr":"172.21.15.189:6836","nonce":2142144473}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6108,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2285762709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2285762709},{"type":"v1","addr":"172.21.15.189:6837","nonce":2285762709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:25:02.062461+0000","modified":"2024-04-06T12:25:05.077648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6834/525827513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":525827513},{"type":"v1","addr":"172.21.15.192:6834","nonce":525827513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-06T12:27:06.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.245+0000 7f6f4fb8f700 1 -- 172.21.15.189:0/3436362411 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f2c06adb0 msgr2=0x7f6f2c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:06.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.245+0000 7f6f4fb8f700 1 --2- 172.21.15.189:0/3436362411 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f2c06adb0 0x7f6f2c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.245+0000 7f6f4fb8f700 1 -- 172.21.15.189:0/3436362411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f480d11b0 msgr2=0x7f6f4806bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:06.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.245+0000 7f6f4fb8f700 1 --2- 172.21.15.189:0/3436362411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f480d11b0 0x7f6f4806bf80 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f6f40009e20 tx=0x7f6f40004ad0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.245+0000 7f6f4fb8f700 1 -- 172.21.15.189:0/3436362411 shutdown_connections 2024-04-06T12:27:06.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.245+0000 7f6f4fb8f700 1 --2- 172.21.15.189:0/3436362411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f480d11b0 0x7f6f4806bf80 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.245+0000 7f6f4fb8f700 1 --2- 172.21.15.189:0/3436362411 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f2c06adb0 0x7f6f2c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.245+0000 7f6f4fb8f700 1 --2- 172.21.15.189:0/3436362411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f480cb870 0x7f6f480b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.245+0000 7f6f4fb8f700 1 --2- 172.21.15.189:0/3436362411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f480cfa00 0x7f6f480cab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.245+0000 7f6f4fb8f700 1 -- 172.21.15.189:0/3436362411 >> 172.21.15.189:0/3436362411 conn(0x7f6f480b2db0 msgr2=0x7f6f4806d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:06.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.245+0000 7f6f4fb8f700 1 -- 172.21.15.189:0/3436362411 shutdown_connections 2024-04-06T12:27:06.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.245+0000 7f6f4fb8f700 1 -- 172.21.15.189:0/3436362411 wait complete. 2024-04-06T12:27:06.244 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 24 2024-04-06T12:27:06.255 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:27:06.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.389+0000 7fb8a87c5700 1 Processor -- start 2024-04-06T12:27:06.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.389+0000 7fb8a87c5700 1 -- start start 2024-04-06T12:27:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.390+0000 7fb8a87c5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8a00b4ed0 0x7fb8a00b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.390+0000 7fb8a87c5700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb8a00d2880 0x7fb8a00d2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.390+0000 7fb8a87c5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a00b5cc0 con 0x7fb8a015d590 2024-04-06T12:27:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.390+0000 7fb8a87c5700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a00b5890 con 0x7fb8a00d2880 2024-04-06T12:27:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.390+0000 7fb8a87c5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a00d1500 con 0x7fb8a00b4ed0 2024-04-06T12:27:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.390+0000 7fb8a6561700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb8a015d590 0x7fb8a00b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34460/0 (socket says 172.21.15.189:34460) 2024-04-06T12:27:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.390+0000 7fb8a6561700 1 -- 172.21.15.189:0/228696933 learned_addr learned my addr 172.21.15.189:0/228696933 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.390+0000 7fb8a6d62700 1 --2- 172.21.15.189:0/228696933 >> v2:172.21.15.192:3300/0 conn(0x7fb8a00d2880 0x7fb8a00d2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.390+0000 7fb8a5d60700 1 --2- 172.21.15.189:0/228696933 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8a00b4ed0 0x7fb8a00b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.390+0000 7fb8a555f700 1 -- 172.21.15.189:0/228696933 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162118644 0 0) 0x7fb8a00b5cc0 con 0x7fb8a015d590 2024-04-06T12:27:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.390+0000 7fb8a555f700 1 -- 172.21.15.189:0/228696933 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb88c0035b0 con 0x7fb8a015d590 2024-04-06T12:27:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.390+0000 7fb8a6d62700 1 -- 172.21.15.189:0/228696933 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8a00b4ed0 msgr2=0x7fb8a00b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.390+0000 7fb8a6d62700 1 --2- 172.21.15.189:0/228696933 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8a00b4ed0 0x7fb8a00b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.390+0000 7fb8a6d62700 1 -- 172.21.15.189:0/228696933 >> v1:172.21.15.189:6789/0 conn(0x7fb8a015d590 legacy=0x7fb8a00b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.390+0000 7fb8a6d62700 1 -- 172.21.15.189:0/228696933 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a00d17e0 con 0x7fb8a00d2880 2024-04-06T12:27:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.391+0000 7fb8a6d62700 1 --2- 172.21.15.189:0/228696933 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8a00d2880 0x7fb8a00d2d10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb898009d90 tx=0x7fb89800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ae52a4d8ae546bc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.391+0000 7fb8a555f700 1 -- 172.21.15.189:0/228696933 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89800e070 con 0x7fb8a00d2880 2024-04-06T12:27:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.391+0000 7fb8a555f700 1 -- 172.21.15.189:0/228696933 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89800bb20 con 0x7fb8a00d2880 2024-04-06T12:27:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.391+0000 7fb8a555f700 1 -- 172.21.15.189:0/228696933 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898012430 con 0x7fb8a00d2880 2024-04-06T12:27:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.391+0000 7fb8a87c5700 1 -- 172.21.15.189:0/228696933 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8a00d2880 msgr2=0x7fb8a00d2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.391+0000 7fb8a87c5700 1 --2- 172.21.15.189:0/228696933 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8a00d2880 0x7fb8a00d2d10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb898009d90 tx=0x7fb89800b330 comp rx=0 tx=0).stop 2024-04-06T12:27:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.391+0000 7fb8a87c5700 1 -- 172.21.15.189:0/228696933 shutdown_connections 2024-04-06T12:27:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.391+0000 7fb8a87c5700 1 --2- 172.21.15.189:0/228696933 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8a00d2880 0x7fb8a00d2d10 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.392+0000 7fb8a87c5700 1 --2- 172.21.15.189:0/228696933 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8a00b4ed0 0x7fb8a00b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.392+0000 7fb8a87c5700 1 -- 172.21.15.189:0/228696933 >> 172.21.15.189:0/228696933 conn(0x7fb8a006bbc0 msgr2=0x7fb8a006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:06.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.392+0000 7fb8a87c5700 1 -- 172.21.15.189:0/228696933 shutdown_connections 2024-04-06T12:27:06.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.392+0000 7fb8a87c5700 1 -- 172.21.15.189:0/228696933 wait complete. 2024-04-06T12:27:06.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.393+0000 7fb8a87c5700 1 Processor -- start 2024-04-06T12:27:06.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.393+0000 7fb8a87c5700 1 -- start start 2024-04-06T12:27:06.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.393+0000 7fb8a87c5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8a00b4ed0 0x7fb8a00cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:06.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.393+0000 7fb8a87c5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8a00d2880 0x7fb8a01562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:06.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.393+0000 7fb8a87c5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8a015d590 0x7fb8a00caf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:06.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.393+0000 7fb8a87c5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a014d050 con 0x7fb8a00b4ed0 2024-04-06T12:27:06.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.394+0000 7fb8a87c5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a01e8a40 con 0x7fb8a00d2880 2024-04-06T12:27:06.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.394+0000 7fb8a87c5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a00cbd80 con 0x7fb8a015d590 2024-04-06T12:27:06.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.394+0000 7fb8a6561700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8a00b4ed0 0x7fb8a00cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:06.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.394+0000 7fb8a6561700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8a00b4ed0 0x7fb8a00cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36484/0 (socket says 172.21.15.189:36484) 2024-04-06T12:27:06.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.394+0000 7fb8a6561700 1 -- 172.21.15.189:0/2152728953 learned_addr learned my addr 172.21.15.189:0/2152728953 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:06.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.394+0000 7fb8a5d60700 1 --2- 172.21.15.189:0/2152728953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8a00d2880 0x7fb8a01562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:06.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.394+0000 7fb8a6d62700 1 --2- 172.21.15.189:0/2152728953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8a015d590 0x7fb8a00caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:06.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.394+0000 7fb8a6561700 1 -- 172.21.15.189:0/2152728953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8a015d590 msgr2=0x7fb8a00caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:06.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.394+0000 7fb8a6561700 1 --2- 172.21.15.189:0/2152728953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8a015d590 0x7fb8a00caf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.394+0000 7fb8a6561700 1 -- 172.21.15.189:0/2152728953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8a00d2880 msgr2=0x7fb8a01562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:06.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.394+0000 7fb8a6561700 1 --2- 172.21.15.189:0/2152728953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8a00d2880 0x7fb8a01562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.394+0000 7fb8a6561700 1 -- 172.21.15.189:0/2152728953 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb898009a70 con 0x7fb8a00b4ed0 2024-04-06T12:27:06.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.394+0000 7fb8a6561700 1 --2- 172.21.15.189:0/2152728953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8a00b4ed0 0x7fb8a00cce00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fb89c001eb0 tx=0x7fb89c00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:06.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.394+0000 7fb896ffd700 1 -- 172.21.15.189:0/2152728953 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c005d70 con 0x7fb8a00b4ed0 2024-04-06T12:27:06.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.394+0000 7fb896ffd700 1 -- 172.21.15.189:0/2152728953 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89c0070d0 con 0x7fb8a00b4ed0 2024-04-06T12:27:06.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.394+0000 7fb8a87c5700 1 -- 172.21.15.189:0/2152728953 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a00cc0c0 con 0x7fb8a00b4ed0 2024-04-06T12:27:06.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.394+0000 7fb896ffd700 1 -- 172.21.15.189:0/2152728953 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c007460 con 0x7fb8a00b4ed0 2024-04-06T12:27:06.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.394+0000 7fb8a87c5700 1 -- 172.21.15.189:0/2152728953 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a00cc900 con 0x7fb8a00b4ed0 2024-04-06T12:27:06.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.395+0000 7fb8a87c5700 1 -- 172.21.15.189:0/2152728953 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a00ccc70 con 0x7fb8a00b4ed0 2024-04-06T12:27:06.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.396+0000 7fb896ffd700 1 -- 172.21.15.189:0/2152728953 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 28) v1 ==== 89259+0+0 (secure 0 0 0) 0x7fb89c0075c0 con 0x7fb8a00b4ed0 2024-04-06T12:27:06.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.396+0000 7fb896ffd700 1 --2- 172.21.15.189:0/2152728953 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb88406adb0 0x7fb88406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:06.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.396+0000 7fb8a5d60700 1 -- 172.21.15.189:0/2152728953 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb88406adb0 msgr2=0x7fb88406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:06.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.396+0000 7fb896ffd700 1 -- 172.21.15.189:0/2152728953 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb89c088ad0 con 0x7fb8a00b4ed0 2024-04-06T12:27:06.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.396+0000 7fb8a5d60700 1 --2- 172.21.15.189:0/2152728953 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb88406adb0 0x7fb88406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:06.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.400+0000 7fb896ffd700 1 -- 172.21.15.189:0/2152728953 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb89c053960 con 0x7fb8a00b4ed0 2024-04-06T12:27:06.528 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:06.531+0000 7fe88a431200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:27:06.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.597+0000 7fb8a5d60700 1 -- 172.21.15.189:0/2152728953 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb88406adb0 msgr2=0x7fb88406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:06.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.597+0000 7fb8a5d60700 1 --2- 172.21.15.189:0/2152728953 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb88406adb0 0x7fb88406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:06.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.606+0000 7fb8a87c5700 1 -- 172.21.15.189:0/2152728953 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8a00cca90 con 0x7fb8a00b4ed0 2024-04-06T12:27:06.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.607+0000 7fb896ffd700 1 -- 172.21.15.189:0/2152728953 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16892 (secure 0 0 0) 0x7fb89c017090 con 0x7fb8a00b4ed0 2024-04-06T12:27:06.604 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:06.605 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":70,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:26:17.522565+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:24:59.828905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-06T12:25:01.020253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.23","mappings":[{"from":0,"to":1}]},{"pgid":"8.38","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/1187685283":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/1356350621":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/162292108":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/2706754156":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/1122382676":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/2132037261":"2024-04-07T12:25:15.982358+0000","172.21.15.189:6833/47128":"2024-04-07T12:25:15.982358+0000","172.21.15.189:6832/47128":"2024-04-07T12:25:15.982358+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:27:06.605 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:06.608+0000 7fe88a431200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:27:06.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.609+0000 7fb8a87c5700 1 -- 172.21.15.189:0/2152728953 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb88406adb0 msgr2=0x7fb88406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:06.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.609+0000 7fb8a87c5700 1 --2- 172.21.15.189:0/2152728953 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb88406adb0 0x7fb88406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.609+0000 7fb8a87c5700 1 -- 172.21.15.189:0/2152728953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8a00b4ed0 msgr2=0x7fb8a00cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:06.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.609+0000 7fb8a87c5700 1 --2- 172.21.15.189:0/2152728953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8a00b4ed0 0x7fb8a00cce00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fb89c001eb0 tx=0x7fb89c00c7b0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.609+0000 7fb8a87c5700 1 -- 172.21.15.189:0/2152728953 shutdown_connections 2024-04-06T12:27:06.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.609+0000 7fb8a87c5700 1 --2- 172.21.15.189:0/2152728953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8a015d590 0x7fb8a00caf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.609+0000 7fb8a87c5700 1 --2- 172.21.15.189:0/2152728953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8a00b4ed0 0x7fb8a00cce00 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.609+0000 7fb8a87c5700 1 --2- 172.21.15.189:0/2152728953 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb88406adb0 0x7fb88406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.609+0000 7fb8a87c5700 1 --2- 172.21.15.189:0/2152728953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8a00d2880 0x7fb8a01562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.609+0000 7fb8a87c5700 1 -- 172.21.15.189:0/2152728953 >> 172.21.15.189:0/2152728953 conn(0x7fb8a006bbc0 msgr2=0x7fb8a015bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:06.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.609+0000 7fb8a87c5700 1 -- 172.21.15.189:0/2152728953 shutdown_connections 2024-04-06T12:27:06.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.609+0000 7fb8a87c5700 1 -- 172.21.15.189:0/2152728953 wait complete. 2024-04-06T12:27:06.619 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:24:59.828905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:25:01.020253+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:27:06.620 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:27:06.696 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:06.698+0000 7fbc5e4e5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:27:06.728 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:06.730+0000 7fe88a431200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:27:06.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.753+0000 7f3e0e5f6700 1 Processor -- start 2024-04-06T12:27:06.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.753+0000 7f3e0e5f6700 1 -- start start 2024-04-06T12:27:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.754+0000 7f3e0e5f6700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3e0806ae40 0x7f3e0815bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.754+0000 7f3e0e5f6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3e0815c100 0x7f3e080ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.754+0000 7f3e0e5f6700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e0815f9e0 con 0x7f3e08158fe0 2024-04-06T12:27:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.754+0000 7f3e0e5f6700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e0815f0d0 con 0x7f3e0815c100 2024-04-06T12:27:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.754+0000 7f3e0e5f6700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e0815c8f0 con 0x7f3e0806ae40 2024-04-06T12:27:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.754+0000 7f3e0cb93700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3e0815c100 0x7f3e080ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.754+0000 7f3e0cb93700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3e0815c100 0x7f3e080ceca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36492/0 (socket says 172.21.15.189:36492) 2024-04-06T12:27:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.754+0000 7f3e077fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3e0806ae40 0x7f3e0815bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.754+0000 7f3e0cb93700 1 -- 172.21.15.189:0/2515091012 learned_addr learned my addr 172.21.15.189:0/2515091012 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.754+0000 7f3e07fff700 1 --1- 172.21.15.189:0/2515091012 >> v1:172.21.15.192:6789/0 conn(0x7f3e08158fe0 0x7f3e0806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:59080/0 (socket says 172.21.15.189:59080) 2024-04-06T12:27:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.755+0000 7f3e077fe700 1 -- 172.21.15.189:0/2515091012 >> v2:172.21.15.189:3300/0 conn(0x7f3e0815c100 msgr2=0x7f3e080ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.755+0000 7f3e077fe700 1 --2- 172.21.15.189:0/2515091012 >> v2:172.21.15.189:3300/0 conn(0x7f3e0815c100 0x7f3e080ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.755+0000 7f3e077fe700 1 -- 172.21.15.189:0/2515091012 >> v1:172.21.15.192:6789/0 conn(0x7f3e08158fe0 legacy=0x7f3e0806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.755+0000 7f3e077fe700 1 -- 172.21.15.189:0/2515091012 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0815f390 con 0x7f3e0806ae40 2024-04-06T12:27:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.755+0000 7f3e06ffd700 1 -- 172.21.15.189:0/2515091012 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291903083 0 0) 0x7f3e0815f9e0 con 0x7f3e08158fe0 2024-04-06T12:27:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.755+0000 7f3e0cb93700 1 --2- 172.21.15.189:0/2515091012 >> v2:172.21.15.189:3300/0 conn(0x7f3e0815c100 0x7f3e080ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.755+0000 7f3e077fe700 1 --2- 172.21.15.189:0/2515091012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e0806ae40 0x7f3e0815bbc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3e08067b80 tx=0x7f3dfc009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fb600c48d01cc99 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.755+0000 7f3e06ffd700 1 -- 172.21.15.189:0/2515091012 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc011830 con 0x7f3e0806ae40 2024-04-06T12:27:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.755+0000 7f3e06ffd700 1 -- 172.21.15.189:0/2515091012 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dfc011990 con 0x7f3e0806ae40 2024-04-06T12:27:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.755+0000 7f3e06ffd700 1 -- 172.21.15.189:0/2515091012 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc011d20 con 0x7f3e0806ae40 2024-04-06T12:27:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.756+0000 7f3e0e5f6700 1 -- 172.21.15.189:0/2515091012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e0806ae40 msgr2=0x7f3e0815bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.756+0000 7f3e0e5f6700 1 --2- 172.21.15.189:0/2515091012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e0806ae40 0x7f3e0815bbc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3e08067b80 tx=0x7f3dfc009b20 comp rx=0 tx=0).stop 2024-04-06T12:27:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.756+0000 7f3e0e5f6700 1 -- 172.21.15.189:0/2515091012 shutdown_connections 2024-04-06T12:27:06.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.756+0000 7f3e0e5f6700 1 --2- 172.21.15.189:0/2515091012 >> v2:172.21.15.189:3300/0 conn(0x7f3e0815c100 0x7f3e080ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.756+0000 7f3e0e5f6700 1 --2- 172.21.15.189:0/2515091012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e0806ae40 0x7f3e0815bbc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.756+0000 7f3e0e5f6700 1 -- 172.21.15.189:0/2515091012 >> 172.21.15.189:0/2515091012 conn(0x7f3e080d00b0 msgr2=0x7f3e080d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:06.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.756+0000 7f3e0e5f6700 1 -- 172.21.15.189:0/2515091012 shutdown_connections 2024-04-06T12:27:06.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.756+0000 7f3e0e5f6700 1 -- 172.21.15.189:0/2515091012 wait complete. 2024-04-06T12:27:06.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.757+0000 7f3e0e5f6700 1 Processor -- start 2024-04-06T12:27:06.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.757+0000 7f3e0e5f6700 1 -- start start 2024-04-06T12:27:06.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.758+0000 7f3e0e5f6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e0806ae40 0x7f3e081580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:06.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.758+0000 7f3e0e5f6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e08158fe0 0x7f3e0813f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:06.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.758+0000 7f3e0e5f6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e0815c100 0x7f3e0813f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:06.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.758+0000 7f3e0e5f6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e080c9340 con 0x7f3e0815c100 2024-04-06T12:27:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.758+0000 7f3e0e5f6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e08158600 con 0x7f3e0806ae40 2024-04-06T12:27:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.758+0000 7f3e0e5f6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e080b4a10 con 0x7f3e08158fe0 2024-04-06T12:27:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.758+0000 7f3e0cb93700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e0815c100 0x7f3e0813f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.758+0000 7f3e07fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e0806ae40 0x7f3e081580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.758+0000 7f3e0cb93700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e0815c100 0x7f3e0813f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36506/0 (socket says 172.21.15.189:36506) 2024-04-06T12:27:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.758+0000 7f3e0cb93700 1 -- 172.21.15.189:0/2063477401 learned_addr learned my addr 172.21.15.189:0/2063477401 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.758+0000 7f3e077fe700 1 --2- 172.21.15.189:0/2063477401 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e08158fe0 0x7f3e0813f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.759+0000 7f3e07fff700 1 -- 172.21.15.189:0/2063477401 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e08158fe0 msgr2=0x7f3e0813f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.759+0000 7f3e07fff700 1 --2- 172.21.15.189:0/2063477401 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e08158fe0 0x7f3e0813f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.759+0000 7f3e07fff700 1 -- 172.21.15.189:0/2063477401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e0815c100 msgr2=0x7f3e0813f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.759+0000 7f3e07fff700 1 --2- 172.21.15.189:0/2063477401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e0815c100 0x7f3e0813f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.759+0000 7f3e07fff700 1 -- 172.21.15.189:0/2063477401 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e00009b30 con 0x7f3e0806ae40 2024-04-06T12:27:06.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.759+0000 7f3e07fff700 1 --2- 172.21.15.189:0/2063477401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e0806ae40 0x7f3e081580c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3df800c4b0 tx=0x7f3df800c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.759+0000 7f3e04ff9700 1 -- 172.21.15.189:0/2063477401 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df8004c80 con 0x7f3e0806ae40 2024-04-06T12:27:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.759+0000 7f3e04ff9700 1 -- 172.21.15.189:0/2063477401 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df8004e00 con 0x7f3e0806ae40 2024-04-06T12:27:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.759+0000 7f3e0e5f6700 1 -- 172.21.15.189:0/2063477401 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dfc0094a0 con 0x7f3e0806ae40 2024-04-06T12:27:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.759+0000 7f3e04ff9700 1 -- 172.21.15.189:0/2063477401 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df8007310 con 0x7f3e0806ae40 2024-04-06T12:27:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.759+0000 7f3e0e5f6700 1 -- 172.21.15.189:0/2063477401 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e080c9830 con 0x7f3e0806ae40 2024-04-06T12:27:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.760+0000 7f3e0e5f6700 1 -- 172.21.15.189:0/2063477401 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e08067bb0 con 0x7f3e0806ae40 2024-04-06T12:27:06.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.760+0000 7f3e04ff9700 1 -- 172.21.15.189:0/2063477401 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 28) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f3df8007470 con 0x7f3e0806ae40 2024-04-06T12:27:06.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.761+0000 7f3e04ff9700 1 --2- 172.21.15.189:0/2063477401 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3df406adb0 0x7f3df406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:06.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.761+0000 7f3e077fe700 1 -- 172.21.15.189:0/2063477401 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3df406adb0 msgr2=0x7f3df406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:06.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.761+0000 7f3e04ff9700 1 -- 172.21.15.189:0/2063477401 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f3df8089340 con 0x7f3e0806ae40 2024-04-06T12:27:06.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.761+0000 7f3e077fe700 1 --2- 172.21.15.189:0/2063477401 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3df406adb0 0x7f3df406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:06.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.766+0000 7f3e04ff9700 1 -- 172.21.15.189:0/2063477401 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3df8054200 con 0x7f3e0806ae40 2024-04-06T12:27:06.780 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:06.783+0000 7fbc5e4e5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:27:06.910 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:06.913+0000 7fbc5e4e5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:27:06.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.962+0000 7f3e077fe700 1 -- 172.21.15.189:0/2063477401 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3df406adb0 msgr2=0x7f3df406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:06.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.962+0000 7f3e077fe700 1 --2- 172.21.15.189:0/2063477401 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3df406adb0 0x7f3df406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:06.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.966+0000 7f3e0e5f6700 1 -- 172.21.15.189:0/2063477401 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3e08158b90 con 0x7f3e0806ae40 2024-04-06T12:27:06.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.967+0000 7f3e04ff9700 1 -- 172.21.15.189:0/2063477401 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3df8057820 con 0x7f3e0806ae40 2024-04-06T12:27:06.964 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:27:06.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.969+0000 7f3e0e5f6700 1 -- 172.21.15.189:0/2063477401 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3df406adb0 msgr2=0x7f3df406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:06.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.969+0000 7f3e0e5f6700 1 --2- 172.21.15.189:0/2063477401 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3df406adb0 0x7f3df406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.969+0000 7f3e0e5f6700 1 -- 172.21.15.189:0/2063477401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e0806ae40 msgr2=0x7f3e081580c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:06.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.969+0000 7f3e0e5f6700 1 --2- 172.21.15.189:0/2063477401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e0806ae40 0x7f3e081580c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3df800c4b0 tx=0x7f3df800c7c0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.969+0000 7f3e0e5f6700 1 -- 172.21.15.189:0/2063477401 shutdown_connections 2024-04-06T12:27:06.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.969+0000 7f3e0e5f6700 1 --2- 172.21.15.189:0/2063477401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e0815c100 0x7f3e0813f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.969+0000 7f3e0e5f6700 1 --2- 172.21.15.189:0/2063477401 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3df406adb0 0x7f3df406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.969+0000 7f3e0e5f6700 1 --2- 172.21.15.189:0/2063477401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e0806ae40 0x7f3e081580c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.969+0000 7f3e0e5f6700 1 --2- 172.21.15.189:0/2063477401 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e08158fe0 0x7f3e0813f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:06.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.969+0000 7f3e0e5f6700 1 -- 172.21.15.189:0/2063477401 >> 172.21.15.189:0/2063477401 conn(0x7f3e080d00b0 msgr2=0x7f3e080cf870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:06.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.969+0000 7f3e0e5f6700 1 -- 172.21.15.189:0/2063477401 shutdown_connections 2024-04-06T12:27:06.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:06.969+0000 7f3e0e5f6700 1 -- 172.21.15.189:0/2063477401 wait complete. 2024-04-06T12:27:06.979 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:27:07.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.114+0000 7f5365633700 1 Processor -- start 2024-04-06T12:27:07.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.114+0000 7f5365633700 1 -- start start 2024-04-06T12:27:07.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.115+0000 7f5365633700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f536015d7b0 0x7f53600b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:07.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.115+0000 7f5365633700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f536006a730 0x7f536006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:07.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.115+0000 7f5365633700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f536015ddf0 0x7f536015e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:07.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.115+0000 7f5365633700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f536006b0f0 con 0x7f536015ddf0 2024-04-06T12:27:07.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.115+0000 7f5365633700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53600b3990 con 0x7f536015d7b0 2024-04-06T12:27:07.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.115+0000 7f5365633700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53600b3060 con 0x7f536006a730 2024-04-06T12:27:07.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.115+0000 7f535f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f536015ddf0 0x7f536015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:07.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.115+0000 7f535f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f536015ddf0 0x7f536015e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36512/0 (socket says 172.21.15.189:36512) 2024-04-06T12:27:07.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.115+0000 7f535f7fe700 1 -- 172.21.15.189:0/655051462 learned_addr learned my addr 172.21.15.189:0/655051462 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:07.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.115+0000 7f535effd700 1 --2- 172.21.15.189:0/655051462 >> v2:172.21.15.192:3300/0 conn(0x7f536015d7b0 0x7f53600b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:07.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.115+0000 7f535e7fc700 1 --2- 172.21.15.189:0/655051462 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f536006a730 0x7f536006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:07.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.115+0000 7f535f7fe700 1 -- 172.21.15.189:0/655051462 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f536006a730 msgr2=0x7f536006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:07.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.115+0000 7f535f7fe700 1 --2- 172.21.15.189:0/655051462 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f536006a730 0x7f536006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.115+0000 7f535f7fe700 1 -- 172.21.15.189:0/655051462 >> v2:172.21.15.192:3300/0 conn(0x7f536015d7b0 msgr2=0x7f53600b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:07.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.115+0000 7f535f7fe700 1 --2- 172.21.15.189:0/655051462 >> v2:172.21.15.192:3300/0 conn(0x7f536015d7b0 0x7f53600b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.115+0000 7f535f7fe700 1 -- 172.21.15.189:0/655051462 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53600b3340 con 0x7f536015ddf0 2024-04-06T12:27:07.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.116+0000 7f535f7fe700 1 --2- 172.21.15.189:0/655051462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f536015ddf0 0x7f536015e280 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f53600b6200 tx=0x7f535800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e9cff0e4ff19e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:07.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.116+0000 7f535dffb700 1 -- 172.21.15.189:0/655051462 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535800e070 con 0x7f536015ddf0 2024-04-06T12:27:07.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.116+0000 7f535dffb700 1 -- 172.21.15.189:0/655051462 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535800b910 con 0x7f536015ddf0 2024-04-06T12:27:07.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.116+0000 7f535dffb700 1 -- 172.21.15.189:0/655051462 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535800bcc0 con 0x7f536015ddf0 2024-04-06T12:27:07.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.117+0000 7f5365633700 1 -- 172.21.15.189:0/655051462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f536015ddf0 msgr2=0x7f536015e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:07.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.117+0000 7f5365633700 1 --2- 172.21.15.189:0/655051462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f536015ddf0 0x7f536015e280 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f53600b6200 tx=0x7f535800b330 comp rx=0 tx=0).stop 2024-04-06T12:27:07.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.117+0000 7f5365633700 1 -- 172.21.15.189:0/655051462 shutdown_connections 2024-04-06T12:27:07.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.117+0000 7f5365633700 1 --2- 172.21.15.189:0/655051462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f536015ddf0 0x7f536015e280 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.117+0000 7f5365633700 1 --2- 172.21.15.189:0/655051462 >> v2:172.21.15.192:3300/0 conn(0x7f536015d7b0 0x7f53600b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.117+0000 7f5365633700 1 --2- 172.21.15.189:0/655051462 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f536006a730 0x7f536006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.117+0000 7f5365633700 1 -- 172.21.15.189:0/655051462 >> 172.21.15.189:0/655051462 conn(0x7f53600d00b0 msgr2=0x7f53600d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:07.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.117+0000 7f5365633700 1 -- 172.21.15.189:0/655051462 shutdown_connections 2024-04-06T12:27:07.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.117+0000 7f5365633700 1 -- 172.21.15.189:0/655051462 wait complete. 2024-04-06T12:27:07.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.119+0000 7f5365633700 1 Processor -- start 2024-04-06T12:27:07.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.119+0000 7f5365633700 1 -- start start 2024-04-06T12:27:07.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.119+0000 7f5365633700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f536006a730 0x7f53600aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:07.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.119+0000 7f5365633700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f536015d7b0 0x7f5360149be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:07.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.119+0000 7f5365633700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f536015ddf0 0x7f536015c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:07.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.119+0000 7f5365633700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536014a6e0 con 0x7f536015ddf0 2024-04-06T12:27:07.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.119+0000 7f5365633700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53600af0c0 con 0x7f536006a730 2024-04-06T12:27:07.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.119+0000 7f5365633700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f536014a910 con 0x7f536015d7b0 2024-04-06T12:27:07.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.120+0000 7f535f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f536015ddf0 0x7f536015c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:07.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.120+0000 7f535f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f536015ddf0 0x7f536015c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36522/0 (socket says 172.21.15.189:36522) 2024-04-06T12:27:07.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.120+0000 7f535f7fe700 1 -- 172.21.15.189:0/3299477223 learned_addr learned my addr 172.21.15.189:0/3299477223 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:07.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.120+0000 7f535effd700 1 --2- 172.21.15.189:0/3299477223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f536006a730 0x7f53600aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:07.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.120+0000 7f535e7fc700 1 --2- 172.21.15.189:0/3299477223 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f536015d7b0 0x7f5360149be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:07.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.120+0000 7f535effd700 1 -- 172.21.15.189:0/3299477223 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f536015d7b0 msgr2=0x7f5360149be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:07.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.120+0000 7f535effd700 1 --2- 172.21.15.189:0/3299477223 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f536015d7b0 0x7f5360149be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.120+0000 7f535effd700 1 -- 172.21.15.189:0/3299477223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f536015ddf0 msgr2=0x7f536015c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:07.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.120+0000 7f535effd700 1 --2- 172.21.15.189:0/3299477223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f536015ddf0 0x7f536015c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.120+0000 7f535effd700 1 -- 172.21.15.189:0/3299477223 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5358009b30 con 0x7f536006a730 2024-04-06T12:27:07.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.120+0000 7f535e7fc700 1 --2- 172.21.15.189:0/3299477223 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f536015d7b0 0x7f5360149be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:07.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.120+0000 7f535f7fe700 1 --2- 172.21.15.189:0/3299477223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f536015ddf0 0x7f536015c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:07.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.120+0000 7f535effd700 1 --2- 172.21.15.189:0/3299477223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f536006a730 0x7f53600aeb80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f535000e930 tx=0x7f535000ec40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:07.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.120+0000 7f53477fe700 1 -- 172.21.15.189:0/3299477223 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5350013680 con 0x7f536006a730 2024-04-06T12:27:07.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.120+0000 7f53477fe700 1 -- 172.21.15.189:0/3299477223 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53500137e0 con 0x7f536006a730 2024-04-06T12:27:07.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.120+0000 7f53477fe700 1 -- 172.21.15.189:0/3299477223 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5350013b50 con 0x7f536006a730 2024-04-06T12:27:07.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.120+0000 7f5365633700 1 -- 172.21.15.189:0/3299477223 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536014abf0 con 0x7f536006a730 2024-04-06T12:27:07.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.120+0000 7f5365633700 1 -- 172.21.15.189:0/3299477223 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536014b1b0 con 0x7f536006a730 2024-04-06T12:27:07.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.121+0000 7f5365633700 1 -- 172.21.15.189:0/3299477223 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5360067bb0 con 0x7f536006a730 2024-04-06T12:27:07.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.122+0000 7f53477fe700 1 -- 172.21.15.189:0/3299477223 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 28) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f5350013cb0 con 0x7f536006a730 2024-04-06T12:27:07.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.123+0000 7f53477fe700 1 --2- 172.21.15.189:0/3299477223 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f534c06adb0 0x7f534c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:07.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.123+0000 7f535e7fc700 1 -- 172.21.15.189:0/3299477223 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f534c06adb0 msgr2=0x7f534c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:07.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.123+0000 7f53477fe700 1 -- 172.21.15.189:0/3299477223 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5350089560 con 0x7f536006a730 2024-04-06T12:27:07.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.123+0000 7f535e7fc700 1 --2- 172.21.15.189:0/3299477223 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f534c06adb0 0x7f534c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:07.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.127+0000 7f53477fe700 1 -- 172.21.15.189:0/3299477223 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f53500543f0 con 0x7f536006a730 2024-04-06T12:27:07.235 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:07.237+0000 7fe88a431200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:27:07.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.324+0000 7f535e7fc700 1 -- 172.21.15.189:0/3299477223 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f534c06adb0 msgr2=0x7f534c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:07.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.324+0000 7f535e7fc700 1 --2- 172.21.15.189:0/3299477223 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f534c06adb0 0x7f534c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:07.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.327+0000 7f5365633700 1 -- 172.21.15.189:0/3299477223 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f53600b4fc0 con 0x7f536006a730 2024-04-06T12:27:07.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.327+0000 7f53477fe700 1 -- 172.21.15.189:0/3299477223 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5350057a10 con 0x7f536006a730 2024-04-06T12:27:07.324 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:27:07.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.330+0000 7f5365633700 1 -- 172.21.15.189:0/3299477223 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f534c06adb0 msgr2=0x7f534c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:07.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.330+0000 7f5365633700 1 --2- 172.21.15.189:0/3299477223 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f534c06adb0 0x7f534c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.330+0000 7f5365633700 1 -- 172.21.15.189:0/3299477223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f536006a730 msgr2=0x7f53600aeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:07.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.330+0000 7f5365633700 1 --2- 172.21.15.189:0/3299477223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f536006a730 0x7f53600aeb80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f535000e930 tx=0x7f535000ec40 comp rx=0 tx=0).stop 2024-04-06T12:27:07.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.330+0000 7f5365633700 1 -- 172.21.15.189:0/3299477223 shutdown_connections 2024-04-06T12:27:07.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.330+0000 7f5365633700 1 --2- 172.21.15.189:0/3299477223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f536015ddf0 0x7f536015c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.330+0000 7f5365633700 1 --2- 172.21.15.189:0/3299477223 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f534c06adb0 0x7f534c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.330+0000 7f5365633700 1 --2- 172.21.15.189:0/3299477223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f536006a730 0x7f53600aeb80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.330+0000 7f5365633700 1 --2- 172.21.15.189:0/3299477223 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f536015d7b0 0x7f5360149be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.330+0000 7f5365633700 1 -- 172.21.15.189:0/3299477223 >> 172.21.15.189:0/3299477223 conn(0x7f53600d00b0 msgr2=0x7f536006eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:07.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.330+0000 7f5365633700 1 -- 172.21.15.189:0/3299477223 shutdown_connections 2024-04-06T12:27:07.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.330+0000 7f5365633700 1 -- 172.21.15.189:0/3299477223 wait complete. 2024-04-06T12:27:07.339 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:27:07.384 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:07.387+0000 7fe88a431200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:27:07.454 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:07.454+0000 7fbc5e4e5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:27:07.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.469+0000 7f9b4c479700 1 Processor -- start 2024-04-06T12:27:07.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.469+0000 7f9b4c479700 1 -- start start 2024-04-06T12:27:07.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.470+0000 7f9b4c479700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b440d0a60 0x7f9b440d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:07.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.470+0000 7f9b4c479700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9b440d1af0 0x7f9b440b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:07.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.470+0000 7f9b4c479700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b440cdf80 con 0x7f9b4415d7e0 2024-04-06T12:27:07.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.470+0000 7f9b4c479700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b440d1f80 con 0x7f9b440d1af0 2024-04-06T12:27:07.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.470+0000 7f9b4c479700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b440d3230 con 0x7f9b440d0a60 2024-04-06T12:27:07.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.470+0000 7f9b4aa16700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9b440d1af0 0x7f9b440b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:07.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.470+0000 7f9b4aa16700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9b440d1af0 0x7f9b440b66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36534/0 (socket says 172.21.15.189:36534) 2024-04-06T12:27:07.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.470+0000 7f9b4aa16700 1 -- 172.21.15.189:0/3380153203 learned_addr learned my addr 172.21.15.189:0/3380153203 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:07.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.470+0000 7f9b49a14700 1 --2- 172.21.15.189:0/3380153203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b440d0a60 0x7f9b440d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:07.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.470+0000 7f9b4aa16700 1 -- 172.21.15.189:0/3380153203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b440d0a60 msgr2=0x7f9b440d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:07.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.470+0000 7f9b4aa16700 1 --2- 172.21.15.189:0/3380153203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b440d0a60 0x7f9b440d15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.470+0000 7f9b4aa16700 1 -- 172.21.15.189:0/3380153203 >> v1:172.21.15.192:6789/0 conn(0x7f9b4415d7e0 legacy=0x7f9b440d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:07.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.470+0000 7f9b4aa16700 1 -- 172.21.15.189:0/3380153203 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b440d2da0 con 0x7f9b440d1af0 2024-04-06T12:27:07.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.470+0000 7f9b49213700 1 -- 172.21.15.189:0/3380153203 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973585537 0 0) 0x7f9b440cdf80 con 0x7f9b4415d7e0 2024-04-06T12:27:07.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.470+0000 7f9b4aa16700 1 --2- 172.21.15.189:0/3380153203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b440d1af0 0x7f9b440b66b0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009e50 tx=0x7f9b3c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de499957eac1cff0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:07.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.471+0000 7f9b49213700 1 -- 172.21.15.189:0/3380153203 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c00e070 con 0x7f9b440d1af0 2024-04-06T12:27:07.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.471+0000 7f9b49213700 1 -- 172.21.15.189:0/3380153203 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3c00b910 con 0x7f9b440d1af0 2024-04-06T12:27:07.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.471+0000 7f9b49213700 1 -- 172.21.15.189:0/3380153203 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c00bcc0 con 0x7f9b440d1af0 2024-04-06T12:27:07.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.471+0000 7f9b4c479700 1 -- 172.21.15.189:0/3380153203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b440d1af0 msgr2=0x7f9b440b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:07.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.471+0000 7f9b4c479700 1 --2- 172.21.15.189:0/3380153203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b440d1af0 0x7f9b440b66b0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009e50 tx=0x7f9b3c00b330 comp rx=0 tx=0).stop 2024-04-06T12:27:07.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.471+0000 7f9b4c479700 1 -- 172.21.15.189:0/3380153203 shutdown_connections 2024-04-06T12:27:07.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.471+0000 7f9b4c479700 1 --2- 172.21.15.189:0/3380153203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b440d1af0 0x7f9b440b66b0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.471+0000 7f9b4c479700 1 --2- 172.21.15.189:0/3380153203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b440d0a60 0x7f9b440d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.471+0000 7f9b4c479700 1 -- 172.21.15.189:0/3380153203 >> 172.21.15.189:0/3380153203 conn(0x7f9b44072250 msgr2=0x7f9b44072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:07.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.471+0000 7f9b4c479700 1 -- 172.21.15.189:0/3380153203 shutdown_connections 2024-04-06T12:27:07.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.471+0000 7f9b4c479700 1 -- 172.21.15.189:0/3380153203 wait complete. 2024-04-06T12:27:07.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.473+0000 7f9b4c479700 1 Processor -- start 2024-04-06T12:27:07.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.473+0000 7f9b4c479700 1 -- start start 2024-04-06T12:27:07.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.473+0000 7f9b4c479700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b440d0a60 0x7f9b441599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:07.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.473+0000 7f9b4c479700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b440d1af0 0x7f9b440c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:07.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.473+0000 7f9b4c479700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b4415d7e0 0x7f9b440b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:07.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.473+0000 7f9b4c479700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b440b26a0 con 0x7f9b440d0a60 2024-04-06T12:27:07.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.473+0000 7f9b4c479700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b44159f30 con 0x7f9b4415d7e0 2024-04-06T12:27:07.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.473+0000 7f9b4c479700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b440c70e0 con 0x7f9b440d1af0 2024-04-06T12:27:07.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.473+0000 7f9b4a215700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b440d0a60 0x7f9b441599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:07.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.473+0000 7f9b4a215700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b440d0a60 0x7f9b441599f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36542/0 (socket says 172.21.15.189:36542) 2024-04-06T12:27:07.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.473+0000 7f9b4a215700 1 -- 172.21.15.189:0/1620432294 learned_addr learned my addr 172.21.15.189:0/1620432294 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:07.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.474+0000 7f9b49a14700 1 --2- 172.21.15.189:0/1620432294 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b440d1af0 0x7f9b440c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:07.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.474+0000 7f9b4aa16700 1 --2- 172.21.15.189:0/1620432294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b4415d7e0 0x7f9b440b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:07.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.474+0000 7f9b49a14700 1 -- 172.21.15.189:0/1620432294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b4415d7e0 msgr2=0x7f9b440b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:07.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.474+0000 7f9b49a14700 1 --2- 172.21.15.189:0/1620432294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b4415d7e0 0x7f9b440b3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.474+0000 7f9b49a14700 1 -- 172.21.15.189:0/1620432294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b440d0a60 msgr2=0x7f9b441599f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:07.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.474+0000 7f9b49a14700 1 --2- 172.21.15.189:0/1620432294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b440d0a60 0x7f9b441599f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.474+0000 7f9b49a14700 1 -- 172.21.15.189:0/1620432294 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3c009b30 con 0x7f9b440d1af0 2024-04-06T12:27:07.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.474+0000 7f9b4a215700 1 --2- 172.21.15.189:0/1620432294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b440d0a60 0x7f9b441599f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:07.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.474+0000 7f9b49a14700 1 --2- 172.21.15.189:0/1620432294 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b440d1af0 0x7f9b440c9150 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400e8a0 tx=0x7f9b3400ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:07.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.474+0000 7f9b3affd700 1 -- 172.21.15.189:0/1620432294 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b34013680 con 0x7f9b440d1af0 2024-04-06T12:27:07.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.474+0000 7f9b3affd700 1 -- 172.21.15.189:0/1620432294 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b340137e0 con 0x7f9b440d1af0 2024-04-06T12:27:07.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.474+0000 7f9b4c479700 1 -- 172.21.15.189:0/1620432294 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b440c6e00 con 0x7f9b440d1af0 2024-04-06T12:27:07.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.474+0000 7f9b3affd700 1 -- 172.21.15.189:0/1620432294 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b34013b70 con 0x7f9b440d1af0 2024-04-06T12:27:07.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.474+0000 7f9b4c479700 1 -- 172.21.15.189:0/1620432294 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b440c77f0 con 0x7f9b440d1af0 2024-04-06T12:27:07.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.475+0000 7f9b4c479700 1 -- 172.21.15.189:0/1620432294 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b440c7d10 con 0x7f9b440d1af0 2024-04-06T12:27:07.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.476+0000 7f9b3affd700 1 -- 172.21.15.189:0/1620432294 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 28) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f9b34013cd0 con 0x7f9b440d1af0 2024-04-06T12:27:07.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.476+0000 7f9b3affd700 1 --2- 172.21.15.189:0/1620432294 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b2806adb0 0x7f9b2806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:07.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.476+0000 7f9b4a215700 1 -- 172.21.15.189:0/1620432294 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b2806adb0 msgr2=0x7f9b2806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:07.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.476+0000 7f9b3affd700 1 -- 172.21.15.189:0/1620432294 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f9b340895d0 con 0x7f9b440d1af0 2024-04-06T12:27:07.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.476+0000 7f9b4a215700 1 --2- 172.21.15.189:0/1620432294 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b2806adb0 0x7f9b2806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:07.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.481+0000 7f9b3affd700 1 -- 172.21.15.189:0/1620432294 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9b34054330 con 0x7f9b440d1af0 2024-04-06T12:27:07.536 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:07.539+0000 7fe88a431200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:27:07.619 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:07.622+0000 7fbc5e4e5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:27:07.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.677+0000 7f9b4a215700 1 -- 172.21.15.189:0/1620432294 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b2806adb0 msgr2=0x7f9b2806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:07.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.677+0000 7f9b4a215700 1 --2- 172.21.15.189:0/1620432294 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b2806adb0 0x7f9b2806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:07.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.692+0000 7f9b4c479700 1 -- 172.21.15.189:0/1620432294 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9b44074990 con 0x7f9b440d1af0 2024-04-06T12:27:07.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.692+0000 7f9b3affd700 1 -- 172.21.15.189:0/1620432294 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9b34057950 con 0x7f9b440d1af0 2024-04-06T12:27:07.689 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:27:07.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.694+0000 7f9b4c479700 1 -- 172.21.15.189:0/1620432294 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b2806adb0 msgr2=0x7f9b2806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:07.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.694+0000 7f9b4c479700 1 --2- 172.21.15.189:0/1620432294 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b2806adb0 0x7f9b2806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.694+0000 7f9b4c479700 1 -- 172.21.15.189:0/1620432294 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b440d1af0 msgr2=0x7f9b440c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:07.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.694+0000 7f9b4c479700 1 --2- 172.21.15.189:0/1620432294 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b440d1af0 0x7f9b440c9150 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400e8a0 tx=0x7f9b3400ebb0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.694+0000 7f9b4c479700 1 -- 172.21.15.189:0/1620432294 shutdown_connections 2024-04-06T12:27:07.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.694+0000 7f9b4c479700 1 --2- 172.21.15.189:0/1620432294 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b2806adb0 0x7f9b2806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.694+0000 7f9b4c479700 1 --2- 172.21.15.189:0/1620432294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b4415d7e0 0x7f9b440b3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.694+0000 7f9b4c479700 1 --2- 172.21.15.189:0/1620432294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b440d0a60 0x7f9b441599f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.694+0000 7f9b4c479700 1 --2- 172.21.15.189:0/1620432294 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b440d1af0 0x7f9b440c9150 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.695+0000 7f9b4c479700 1 -- 172.21.15.189:0/1620432294 >> 172.21.15.189:0/1620432294 conn(0x7f9b44072250 msgr2=0x7f9b440cd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:07.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.695+0000 7f9b4c479700 1 -- 172.21.15.189:0/1620432294 shutdown_connections 2024-04-06T12:27:07.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.695+0000 7f9b4c479700 1 -- 172.21.15.189:0/1620432294 wait complete. 2024-04-06T12:27:07.705 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:27:07.782 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:07.785+0000 7fbc5e4e5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:27:07.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.834+0000 7f7e80cbd700 1 Processor -- start 2024-04-06T12:27:07.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.834+0000 7f7e80cbd700 1 -- start start 2024-04-06T12:27:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.834+0000 7f7e80cbd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e7c15d590 0x7f7e7c0b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.835+0000 7f7e80cbd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7e7c0d3890 0x7f7e7c0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.835+0000 7f7e80cbd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e7c0b5cc0 con 0x7f7e7c0b4ed0 2024-04-06T12:27:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.835+0000 7f7e80cbd700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7c0b5890 con 0x7f7e7c0d3890 2024-04-06T12:27:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.835+0000 7f7e80cbd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c0d1500 con 0x7f7e7c15d590 2024-04-06T12:27:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.835+0000 7f7e79d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7e7c0b4ed0 0x7f7e7c0b6580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34470/0 (socket says 172.21.15.189:34470) 2024-04-06T12:27:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.835+0000 7f7e79d9b700 1 -- 172.21.15.189:0/818837065 learned_addr learned my addr 172.21.15.189:0/818837065 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.835+0000 7f7e7a59c700 1 --2- 172.21.15.189:0/818837065 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e7c15d590 0x7f7e7c0b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.835+0000 7f7e7ad9d700 1 --2- 172.21.15.189:0/818837065 >> v2:172.21.15.192:3300/0 conn(0x7f7e7c0d3890 0x7f7e7c0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.835+0000 7f7e7959a700 1 -- 172.21.15.189:0/818837065 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690633372 0 0) 0x7f7e7c0b5cc0 con 0x7f7e7c0b4ed0 2024-04-06T12:27:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.835+0000 7f7e7959a700 1 -- 172.21.15.189:0/818837065 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e640035b0 con 0x7f7e7c0b4ed0 2024-04-06T12:27:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.835+0000 7f7e7a59c700 1 -- 172.21.15.189:0/818837065 >> v2:172.21.15.192:3300/0 conn(0x7f7e7c0d3890 msgr2=0x7f7e7c0b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.835+0000 7f7e7a59c700 1 --2- 172.21.15.189:0/818837065 >> v2:172.21.15.192:3300/0 conn(0x7f7e7c0d3890 0x7f7e7c0b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.835+0000 7f7e7a59c700 1 -- 172.21.15.189:0/818837065 >> v1:172.21.15.189:6789/0 conn(0x7f7e7c0b4ed0 legacy=0x7f7e7c0b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.835+0000 7f7e7a59c700 1 -- 172.21.15.189:0/818837065 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c0d17e0 con 0x7f7e7c15d590 2024-04-06T12:27:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.835+0000 7f7e7ad9d700 1 --2- 172.21.15.189:0/818837065 >> v2:172.21.15.192:3300/0 conn(0x7f7e7c0d3890 0x7f7e7c0b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.835+0000 7f7e7a59c700 1 --2- 172.21.15.189:0/818837065 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e7c15d590 0x7f7e7c0b8d70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009870 tx=0x7f7e6c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=310fdc05cc2d615c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.835+0000 7f7e7959a700 1 -- 172.21.15.189:0/818837065 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c011830 con 0x7f7e7c15d590 2024-04-06T12:27:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.836+0000 7f7e7959a700 1 -- 172.21.15.189:0/818837065 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6c011990 con 0x7f7e7c15d590 2024-04-06T12:27:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.836+0000 7f7e7959a700 1 -- 172.21.15.189:0/818837065 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c011d20 con 0x7f7e7c15d590 2024-04-06T12:27:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.836+0000 7f7e80cbd700 1 -- 172.21.15.189:0/818837065 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e7c15d590 msgr2=0x7f7e7c0b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.836+0000 7f7e80cbd700 1 --2- 172.21.15.189:0/818837065 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e7c15d590 0x7f7e7c0b8d70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009870 tx=0x7f7e6c009b80 comp rx=0 tx=0).stop 2024-04-06T12:27:07.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.836+0000 7f7e80cbd700 1 -- 172.21.15.189:0/818837065 shutdown_connections 2024-04-06T12:27:07.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.836+0000 7f7e80cbd700 1 --2- 172.21.15.189:0/818837065 >> v2:172.21.15.192:3300/0 conn(0x7f7e7c0d3890 0x7f7e7c0b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.836+0000 7f7e80cbd700 1 --2- 172.21.15.189:0/818837065 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e7c15d590 0x7f7e7c0b8d70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.836+0000 7f7e80cbd700 1 -- 172.21.15.189:0/818837065 >> 172.21.15.189:0/818837065 conn(0x7f7e7c06bbc0 msgr2=0x7f7e7c06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:07.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.836+0000 7f7e80cbd700 1 -- 172.21.15.189:0/818837065 shutdown_connections 2024-04-06T12:27:07.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.836+0000 7f7e80cbd700 1 -- 172.21.15.189:0/818837065 wait complete. 2024-04-06T12:27:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.838+0000 7f7e80cbd700 1 Processor -- start 2024-04-06T12:27:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.838+0000 7f7e80cbd700 1 -- start start 2024-04-06T12:27:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e80cbd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e7c0b4ed0 0x7f7e7c160140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e80cbd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e7c0d3890 0x7f7e7c0ca520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e80cbd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e7c15d590 0x7f7e7c14fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e80cbd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c1524e0 con 0x7f7e7c15d590 2024-04-06T12:27:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e80cbd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c1500f0 con 0x7f7e7c0b4ed0 2024-04-06T12:27:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e80cbd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c153dd0 con 0x7f7e7c0d3890 2024-04-06T12:27:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e7ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e7c15d590 0x7f7e7c14fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e7a59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e7c0b4ed0 0x7f7e7c160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e79d9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e7c0d3890 0x7f7e7c0ca520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e7a59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e7c0b4ed0 0x7f7e7c160140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:38822/0 (socket says 172.21.15.189:38822) 2024-04-06T12:27:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e7a59c700 1 -- 172.21.15.189:0/1650356529 learned_addr learned my addr 172.21.15.189:0/1650356529 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e7ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e7c15d590 0x7f7e7c14fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36548/0 (socket says 172.21.15.189:36548) 2024-04-06T12:27:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e7a59c700 1 -- 172.21.15.189:0/1650356529 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e7c0d3890 msgr2=0x7f7e7c0ca520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e7a59c700 1 --2- 172.21.15.189:0/1650356529 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e7c0d3890 0x7f7e7c0ca520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e7a59c700 1 -- 172.21.15.189:0/1650356529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e7c15d590 msgr2=0x7f7e7c14fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e7a59c700 1 --2- 172.21.15.189:0/1650356529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e7c15d590 0x7f7e7c14fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e7a59c700 1 -- 172.21.15.189:0/1650356529 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e74009a70 con 0x7f7e7c0b4ed0 2024-04-06T12:27:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e79d9b700 1 --2- 172.21.15.189:0/1650356529 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e7c0d3890 0x7f7e7c0ca520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e7a59c700 1 --2- 172.21.15.189:0/1650356529 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e7c0b4ed0 0x7f7e7c160140 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009840 tx=0x7f7e6c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e6affd700 1 -- 172.21.15.189:0/1650356529 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c004550 con 0x7f7e7c0b4ed0 2024-04-06T12:27:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e6affd700 1 -- 172.21.15.189:0/1650356529 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6c01f070 con 0x7f7e7c0b4ed0 2024-04-06T12:27:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e80cbd700 1 -- 172.21.15.189:0/1650356529 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e6c0094a0 con 0x7f7e7c0b4ed0 2024-04-06T12:27:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e6affd700 1 -- 172.21.15.189:0/1650356529 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c0038e0 con 0x7f7e7c0b4ed0 2024-04-06T12:27:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.839+0000 7f7e80cbd700 1 -- 172.21.15.189:0/1650356529 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e7c151e90 con 0x7f7e7c0b4ed0 2024-04-06T12:27:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.841+0000 7f7e80cbd700 1 -- 172.21.15.189:0/1650356529 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7c074990 con 0x7f7e7c0b4ed0 2024-04-06T12:27:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.841+0000 7f7e6affd700 1 -- 172.21.15.189:0/1650356529 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 28) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f7e6c003a40 con 0x7f7e7c0b4ed0 2024-04-06T12:27:07.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.841+0000 7f7e6affd700 1 --2- 172.21.15.189:0/1650356529 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e5c06adb0 0x7f7e5c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:07.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.841+0000 7f7e79d9b700 1 -- 172.21.15.189:0/1650356529 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e5c06adb0 msgr2=0x7f7e5c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:07.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.841+0000 7f7e6affd700 1 -- 172.21.15.189:0/1650356529 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f7e6c021070 con 0x7f7e7c0b4ed0 2024-04-06T12:27:07.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.841+0000 7f7e79d9b700 1 --2- 172.21.15.189:0/1650356529 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e5c06adb0 0x7f7e5c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:07.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:07.846+0000 7f7e6affd700 1 -- 172.21.15.189:0/1650356529 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7e6c055d20 con 0x7f7e7c0b4ed0 2024-04-06T12:27:08.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.042+0000 7f7e79d9b700 1 -- 172.21.15.189:0/1650356529 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e5c06adb0 msgr2=0x7f7e5c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:08.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.042+0000 7f7e79d9b700 1 --2- 172.21.15.189:0/1650356529 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e5c06adb0 0x7f7e5c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:08.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.099+0000 7f7e80cbd700 1 -- 172.21.15.189:0/1650356529 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e7c0b4090 con 0x7f7e7c0b4ed0 2024-04-06T12:27:08.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.100+0000 7f7e6affd700 1 -- 172.21.15.189:0/1650356529 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f7e6c02b090 con 0x7f7e7c0b4ed0 2024-04-06T12:27:08.097 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:08.097 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/527093473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":527093473},{"type":"v1","addr":"172.21.15.192:6835","nonce":527093473}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6105,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2142144473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2142144473},{"type":"v1","addr":"172.21.15.189:6836","nonce":2142144473}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6108,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2285762709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2285762709},{"type":"v1","addr":"172.21.15.189:6837","nonce":2285762709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:25:02.062461+0000","modified":"2024-04-06T12:25:05.077648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6834/525827513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":525827513},{"type":"v1","addr":"172.21.15.192:6834","nonce":525827513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-06T12:27:08.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.103+0000 7f7e80cbd700 1 -- 172.21.15.189:0/1650356529 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e5c06adb0 msgr2=0x7f7e5c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:08.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.103+0000 7f7e80cbd700 1 --2- 172.21.15.189:0/1650356529 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e5c06adb0 0x7f7e5c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.103+0000 7f7e80cbd700 1 -- 172.21.15.189:0/1650356529 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e7c0b4ed0 msgr2=0x7f7e7c160140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:08.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.103+0000 7f7e80cbd700 1 --2- 172.21.15.189:0/1650356529 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e7c0b4ed0 0x7f7e7c160140 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009840 tx=0x7f7e6c004750 comp rx=0 tx=0).stop 2024-04-06T12:27:08.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.103+0000 7f7e80cbd700 1 -- 172.21.15.189:0/1650356529 shutdown_connections 2024-04-06T12:27:08.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.103+0000 7f7e80cbd700 1 --2- 172.21.15.189:0/1650356529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e7c15d590 0x7f7e7c14fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.103+0000 7f7e80cbd700 1 --2- 172.21.15.189:0/1650356529 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7e5c06adb0 0x7f7e5c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.103+0000 7f7e80cbd700 1 --2- 172.21.15.189:0/1650356529 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e7c0b4ed0 0x7f7e7c160140 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.103+0000 7f7e80cbd700 1 --2- 172.21.15.189:0/1650356529 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e7c0d3890 0x7f7e7c0ca520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.103+0000 7f7e80cbd700 1 -- 172.21.15.189:0/1650356529 >> 172.21.15.189:0/1650356529 conn(0x7f7e7c06bbc0 msgr2=0x7f7e7c0b2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:08.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.103+0000 7f7e80cbd700 1 -- 172.21.15.189:0/1650356529 shutdown_connections 2024-04-06T12:27:08.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.103+0000 7f7e80cbd700 1 -- 172.21.15.189:0/1650356529 wait complete. 2024-04-06T12:27:08.101 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 24 2024-04-06T12:27:08.112 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:27:08.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.241+0000 7f4736905700 1 Processor -- start 2024-04-06T12:27:08.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.241+0000 7f4736905700 1 -- start start 2024-04-06T12:27:08.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.242+0000 7f4736905700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f47300b76c0 0x7f47301682f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:08.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.242+0000 7f4736905700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47300cdf80 con 0x7f473015d7e0 2024-04-06T12:27:08.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.242+0000 7f4736905700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47300d3230 con 0x7f47300d0a60 2024-04-06T12:27:08.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.242+0000 7f4736905700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47300d2da0 con 0x7f47300b76c0 2024-04-06T12:27:08.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.242+0000 7f4734ea2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f47300b76c0 0x7f47301682f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:08.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.242+0000 7f4734ea2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f47300b76c0 0x7f47301682f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36550/0 (socket says 172.21.15.189:36550) 2024-04-06T12:27:08.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.242+0000 7f4734ea2700 1 -- 172.21.15.189:0/83842881 learned_addr learned my addr 172.21.15.189:0/83842881 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:08.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.242+0000 7f4734ea2700 1 -- 172.21.15.189:0/83842881 >> v1:172.21.15.192:6789/0 conn(0x7f47300d0a60 legacy=0x7f47300d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:08.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.242+0000 7f4734ea2700 1 -- 172.21.15.189:0/83842881 >> v1:172.21.15.189:6789/0 conn(0x7f473015d7e0 legacy=0x7f47300d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:08.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.242+0000 7f4734ea2700 1 -- 172.21.15.189:0/83842881 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473015dbc0 con 0x7f47300b76c0 2024-04-06T12:27:08.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.242+0000 7f4734ea2700 1 --2- 172.21.15.189:0/83842881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47300b76c0 0x7f47301682f0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f4728009e50 tx=0x7f472800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0e1dcde9566e9f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:08.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.242+0000 7f472effd700 1 -- 172.21.15.189:0/83842881 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472800e070 con 0x7f47300b76c0 2024-04-06T12:27:08.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.242+0000 7f472effd700 1 -- 172.21.15.189:0/83842881 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472800b9c0 con 0x7f47300b76c0 2024-04-06T12:27:08.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.243+0000 7f472effd700 1 -- 172.21.15.189:0/83842881 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472800bd70 con 0x7f47300b76c0 2024-04-06T12:27:08.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.243+0000 7f4736905700 1 -- 172.21.15.189:0/83842881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47300b76c0 msgr2=0x7f47301682f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:08.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.243+0000 7f4736905700 1 --2- 172.21.15.189:0/83842881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47300b76c0 0x7f47301682f0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f4728009e50 tx=0x7f472800b330 comp rx=0 tx=0).stop 2024-04-06T12:27:08.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.243+0000 7f4736905700 1 -- 172.21.15.189:0/83842881 shutdown_connections 2024-04-06T12:27:08.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.243+0000 7f4736905700 1 --2- 172.21.15.189:0/83842881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47300b76c0 0x7f47301682f0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.243+0000 7f4736905700 1 -- 172.21.15.189:0/83842881 >> 172.21.15.189:0/83842881 conn(0x7f4730072250 msgr2=0x7f4730072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:08.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.243+0000 7f4736905700 1 -- 172.21.15.189:0/83842881 shutdown_connections 2024-04-06T12:27:08.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.243+0000 7f4736905700 1 -- 172.21.15.189:0/83842881 wait complete. 2024-04-06T12:27:08.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.244+0000 7f4736905700 1 Processor -- start 2024-04-06T12:27:08.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.245+0000 7f4736905700 1 -- start start 2024-04-06T12:27:08.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.245+0000 7f4736905700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47300b76c0 0x7f47300b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:08.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.245+0000 7f4736905700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47300d0a60 0x7f47300c7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:08.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.245+0000 7f4736905700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f473015d7e0 0x7f47301f2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:08.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.245+0000 7f4736905700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47300c76a0 con 0x7f47300b76c0 2024-04-06T12:27:08.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.245+0000 7f4736905700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47300b38a0 con 0x7f47300d0a60 2024-04-06T12:27:08.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.245+0000 7f4736905700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47300b3b80 con 0x7f473015d7e0 2024-04-06T12:27:08.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.245+0000 7f472ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47300b76c0 0x7f47300b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:08.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.245+0000 7f472ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47300b76c0 0x7f47300b31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36552/0 (socket says 172.21.15.189:36552) 2024-04-06T12:27:08.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.245+0000 7f472ffff700 1 -- 172.21.15.189:0/2912330372 learned_addr learned my addr 172.21.15.189:0/2912330372 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:08.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.245+0000 7f472f7fe700 1 --2- 172.21.15.189:0/2912330372 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47300d0a60 0x7f47300c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:08.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.245+0000 7f4734ea2700 1 --2- 172.21.15.189:0/2912330372 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f473015d7e0 0x7f47301f2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:08.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.246+0000 7f472ffff700 1 -- 172.21.15.189:0/2912330372 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f473015d7e0 msgr2=0x7f47301f2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:08.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.246+0000 7f472ffff700 1 --2- 172.21.15.189:0/2912330372 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f473015d7e0 0x7f47301f2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.246+0000 7f472ffff700 1 -- 172.21.15.189:0/2912330372 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47300d0a60 msgr2=0x7f47300c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:08.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.246+0000 7f472ffff700 1 --2- 172.21.15.189:0/2912330372 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47300d0a60 0x7f47300c7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.246+0000 7f472ffff700 1 -- 172.21.15.189:0/2912330372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4728009b30 con 0x7f47300b76c0 2024-04-06T12:27:08.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.246+0000 7f472ffff700 1 --2- 172.21.15.189:0/2912330372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47300b76c0 0x7f47300b31e0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f472000c450 tx=0x7f472000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:08.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.246+0000 7f472cff9700 1 -- 172.21.15.189:0/2912330372 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47200052d0 con 0x7f47300b76c0 2024-04-06T12:27:08.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.246+0000 7f472cff9700 1 -- 172.21.15.189:0/2912330372 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4720005430 con 0x7f47300b76c0 2024-04-06T12:27:08.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.246+0000 7f4736905700 1 -- 172.21.15.189:0/2912330372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47300b3e60 con 0x7f47300b76c0 2024-04-06T12:27:08.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.246+0000 7f472cff9700 1 -- 172.21.15.189:0/2912330372 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47200038c0 con 0x7f47300b76c0 2024-04-06T12:27:08.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.246+0000 7f4736905700 1 -- 172.21.15.189:0/2912330372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47300b4420 con 0x7f47300b76c0 2024-04-06T12:27:08.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.247+0000 7f4736905700 1 -- 172.21.15.189:0/2912330372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4730074990 con 0x7f47300b76c0 2024-04-06T12:27:08.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.248+0000 7f472cff9700 1 -- 172.21.15.189:0/2912330372 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 28) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f4720003a20 con 0x7f47300b76c0 2024-04-06T12:27:08.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.248+0000 7f472cff9700 1 --2- 172.21.15.189:0/2912330372 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f471c06adb0 0x7f471c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:08.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.248+0000 7f472f7fe700 1 -- 172.21.15.189:0/2912330372 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f471c06adb0 msgr2=0x7f471c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:08.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.248+0000 7f472cff9700 1 -- 172.21.15.189:0/2912330372 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f4720088b50 con 0x7f47300b76c0 2024-04-06T12:27:08.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.248+0000 7f472f7fe700 1 --2- 172.21.15.189:0/2912330372 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f471c06adb0 0x7f471c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:08.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.253+0000 7f472cff9700 1 -- 172.21.15.189:0/2912330372 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4720053890 con 0x7f47300b76c0 2024-04-06T12:27:08.259 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:08.262+0000 7fe88a431200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:27:08.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.449+0000 7f472f7fe700 1 -- 172.21.15.189:0/2912330372 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f471c06adb0 msgr2=0x7f471c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:08.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.449+0000 7f472f7fe700 1 --2- 172.21.15.189:0/2912330372 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f471c06adb0 0x7f471c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:08.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.458+0000 7f4736905700 1 -- 172.21.15.189:0/2912330372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47300b45e0 con 0x7f47300b76c0 2024-04-06T12:27:08.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.459+0000 7f472cff9700 1 -- 172.21.15.189:0/2912330372 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+17000 (secure 0 0 0) 0x7f472000ce60 con 0x7f47300b76c0 2024-04-06T12:27:08.457 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:08.457 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":71,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:27:07.574700+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:24:59.828905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-06T12:25:01.020253+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.23","mappings":[{"from":0,"to":1}]},{"pgid":"8.38","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/1187685283":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/1356350621":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/162292108":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/2706754156":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/1122382676":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/2132037261":"2024-04-07T12:25:15.982358+0000","172.21.15.189:6833/47128":"2024-04-07T12:25:15.982358+0000","172.21.15.189:6832/47128":"2024-04-07T12:25:15.982358+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:27:08.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.462+0000 7f4736905700 1 -- 172.21.15.189:0/2912330372 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f471c06adb0 msgr2=0x7f471c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:08.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.462+0000 7f4736905700 1 --2- 172.21.15.189:0/2912330372 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f471c06adb0 0x7f471c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.462+0000 7f4736905700 1 -- 172.21.15.189:0/2912330372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47300b76c0 msgr2=0x7f47300b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:08.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.462+0000 7f4736905700 1 --2- 172.21.15.189:0/2912330372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47300b76c0 0x7f47300b31e0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f472000c450 tx=0x7f472000c760 comp rx=0 tx=0).stop 2024-04-06T12:27:08.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.462+0000 7f4736905700 1 -- 172.21.15.189:0/2912330372 shutdown_connections 2024-04-06T12:27:08.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.462+0000 7f4736905700 1 --2- 172.21.15.189:0/2912330372 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f473015d7e0 0x7f47301f2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.462+0000 7f4736905700 1 --2- 172.21.15.189:0/2912330372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47300b76c0 0x7f47300b31e0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.462+0000 7f4736905700 1 --2- 172.21.15.189:0/2912330372 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f471c06adb0 0x7f471c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.462+0000 7f4736905700 1 --2- 172.21.15.189:0/2912330372 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47300d0a60 0x7f47300c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.462+0000 7f4736905700 1 -- 172.21.15.189:0/2912330372 >> 172.21.15.189:0/2912330372 conn(0x7f4730072250 msgr2=0x7f47300cd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:08.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.462+0000 7f4736905700 1 -- 172.21.15.189:0/2912330372 shutdown_connections 2024-04-06T12:27:08.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.462+0000 7f4736905700 1 -- 172.21.15.189:0/2912330372 wait complete. 2024-04-06T12:27:08.471 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-06T12:27:08.472 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:27:08.499 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:08.502+0000 7fe88a431200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:27:08.549 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:08.552+0000 7fbc5e4e5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:27:08.572 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:08.575+0000 7fe88a431200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:27:08.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.603+0000 7f054e038700 1 Processor -- start 2024-04-06T12:27:08.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.603+0000 7f054e038700 1 -- start start 2024-04-06T12:27:08.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.604+0000 7f054e038700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0548146dd0 0x7f054806d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:08.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.604+0000 7f054e038700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f05480cedf0 0x7f05480cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:08.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.604+0000 7f054e038700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05480cf760 con 0x7f05480d17a0 2024-04-06T12:27:08.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.604+0000 7f054e038700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f054806beb0 con 0x7f05480cedf0 2024-04-06T12:27:08.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.604+0000 7f054e038700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054806c0a0 con 0x7f0548146dd0 2024-04-06T12:27:08.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.604+0000 7f05477fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f05480d17a0 0x7f054806b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34502/0 (socket says 172.21.15.189:34502) 2024-04-06T12:27:08.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.604+0000 7f05477fe700 1 -- 172.21.15.189:0/3404838435 learned_addr learned my addr 172.21.15.189:0/3404838435 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:08.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.604+0000 7f0546ffd700 1 --2- 172.21.15.189:0/3404838435 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0548146dd0 0x7f054806d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:08.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.604+0000 7f0547fff700 1 --2- 172.21.15.189:0/3404838435 >> v2:172.21.15.192:3300/0 conn(0x7f05480cedf0 0x7f05480cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:08.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.605+0000 7f05467fc700 1 -- 172.21.15.189:0/3404838435 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663981973 0 0) 0x7f05480cf760 con 0x7f05480d17a0 2024-04-06T12:27:08.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.605+0000 7f05467fc700 1 -- 172.21.15.189:0/3404838435 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05300035b0 con 0x7f05480d17a0 2024-04-06T12:27:08.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.605+0000 7f0546ffd700 1 -- 172.21.15.189:0/3404838435 >> v2:172.21.15.192:3300/0 conn(0x7f05480cedf0 msgr2=0x7f05480cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:08.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.605+0000 7f0546ffd700 1 --2- 172.21.15.189:0/3404838435 >> v2:172.21.15.192:3300/0 conn(0x7f05480cedf0 0x7f05480cf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.605+0000 7f0546ffd700 1 -- 172.21.15.189:0/3404838435 >> v1:172.21.15.189:6789/0 conn(0x7f05480d17a0 legacy=0x7f054806b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:08.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.605+0000 7f0546ffd700 1 -- 172.21.15.189:0/3404838435 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054806c380 con 0x7f0548146dd0 2024-04-06T12:27:08.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.605+0000 7f0547fff700 1 --2- 172.21.15.189:0/3404838435 >> v2:172.21.15.192:3300/0 conn(0x7f05480cedf0 0x7f05480cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:08.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.605+0000 7f0546ffd700 1 --2- 172.21.15.189:0/3404838435 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0548146dd0 0x7f054806d140 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f053c009840 tx=0x7f053c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbc72a5f72449008 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:08.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.605+0000 7f05467fc700 1 -- 172.21.15.189:0/3404838435 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c011830 con 0x7f0548146dd0 2024-04-06T12:27:08.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.605+0000 7f05467fc700 1 -- 172.21.15.189:0/3404838435 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053c011990 con 0x7f0548146dd0 2024-04-06T12:27:08.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.605+0000 7f05467fc700 1 -- 172.21.15.189:0/3404838435 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c011d20 con 0x7f0548146dd0 2024-04-06T12:27:08.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.606+0000 7f054e038700 1 -- 172.21.15.189:0/3404838435 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0548146dd0 msgr2=0x7f054806d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:08.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.606+0000 7f054e038700 1 --2- 172.21.15.189:0/3404838435 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0548146dd0 0x7f054806d140 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f053c009840 tx=0x7f053c009b50 comp rx=0 tx=0).stop 2024-04-06T12:27:08.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.606+0000 7f054e038700 1 -- 172.21.15.189:0/3404838435 shutdown_connections 2024-04-06T12:27:08.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.606+0000 7f054e038700 1 --2- 172.21.15.189:0/3404838435 >> v2:172.21.15.192:3300/0 conn(0x7f05480cedf0 0x7f05480cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.606+0000 7f054e038700 1 --2- 172.21.15.189:0/3404838435 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0548146dd0 0x7f054806d140 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.606+0000 7f054e038700 1 -- 172.21.15.189:0/3404838435 >> 172.21.15.189:0/3404838435 conn(0x7f05480b2db0 msgr2=0x7f05480b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:08.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.606+0000 7f054e038700 1 -- 172.21.15.189:0/3404838435 shutdown_connections 2024-04-06T12:27:08.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.606+0000 7f054e038700 1 -- 172.21.15.189:0/3404838435 wait complete. 2024-04-06T12:27:08.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.607+0000 7f054e038700 1 Processor -- start 2024-04-06T12:27:08.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.607+0000 7f054e038700 1 -- start start 2024-04-06T12:27:08.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.608+0000 7f054e038700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05480cedf0 0x7f05480b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:08.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.608+0000 7f054e038700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05480d17a0 0x7f05480c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:08.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.608+0000 7f054e038700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0548146dd0 0x7f054815aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:08.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.608+0000 7f054e038700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05480afda0 con 0x7f05480d17a0 2024-04-06T12:27:08.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.608+0000 7f054e038700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05481f7490 con 0x7f05480cedf0 2024-04-06T12:27:08.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.608+0000 7f054e038700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05480b0700 con 0x7f0548146dd0 2024-04-06T12:27:08.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.608+0000 7f0546ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05480d17a0 0x7f05480c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:08.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.608+0000 7f0546ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05480d17a0 0x7f05480c6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36554/0 (socket says 172.21.15.189:36554) 2024-04-06T12:27:08.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.608+0000 7f0546ffd700 1 -- 172.21.15.189:0/1490767497 learned_addr learned my addr 172.21.15.189:0/1490767497 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:08.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.608+0000 7f0547fff700 1 --2- 172.21.15.189:0/1490767497 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0548146dd0 0x7f054815aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:08.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.608+0000 7f05477fe700 1 --2- 172.21.15.189:0/1490767497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05480cedf0 0x7f05480b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:08.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.609+0000 7f0546ffd700 1 -- 172.21.15.189:0/1490767497 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0548146dd0 msgr2=0x7f054815aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:08.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.609+0000 7f0546ffd700 1 --2- 172.21.15.189:0/1490767497 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0548146dd0 0x7f054815aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.609+0000 7f0546ffd700 1 -- 172.21.15.189:0/1490767497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05480cedf0 msgr2=0x7f05480b01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:08.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.609+0000 7f0546ffd700 1 --2- 172.21.15.189:0/1490767497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05480cedf0 0x7f05480b01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.609+0000 7f0546ffd700 1 -- 172.21.15.189:0/1490767497 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05400094a0 con 0x7f05480d17a0 2024-04-06T12:27:08.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.609+0000 7f0547fff700 1 --2- 172.21.15.189:0/1490767497 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0548146dd0 0x7f054815aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:08.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.609+0000 7f0546ffd700 1 --2- 172.21.15.189:0/1490767497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05480d17a0 0x7f05480c6f40 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f053c009b90 tx=0x7f053c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:08.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.609+0000 7f052ffff700 1 -- 172.21.15.189:0/1490767497 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c0044d0 con 0x7f05480d17a0 2024-04-06T12:27:08.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.609+0000 7f052ffff700 1 -- 172.21.15.189:0/1490767497 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053c0036a0 con 0x7f05480d17a0 2024-04-06T12:27:08.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.609+0000 7f054e038700 1 -- 172.21.15.189:0/1490767497 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053c0094a0 con 0x7f05480d17a0 2024-04-06T12:27:08.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.609+0000 7f052ffff700 1 -- 172.21.15.189:0/1490767497 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c003a30 con 0x7f05480d17a0 2024-04-06T12:27:08.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.609+0000 7f054e038700 1 -- 172.21.15.189:0/1490767497 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05480af6e0 con 0x7f05480d17a0 2024-04-06T12:27:08.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.610+0000 7f054e038700 1 -- 172.21.15.189:0/1490767497 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05480afee0 con 0x7f05480d17a0 2024-04-06T12:27:08.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.610+0000 7f052ffff700 1 -- 172.21.15.189:0/1490767497 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 28) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f053c003b90 con 0x7f05480d17a0 2024-04-06T12:27:08.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.611+0000 7f052ffff700 1 --2- 172.21.15.189:0/1490767497 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f053406adb0 0x7f053406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:08.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.611+0000 7f05477fe700 1 -- 172.21.15.189:0/1490767497 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f053406adb0 msgr2=0x7f053406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:08.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.611+0000 7f052ffff700 1 -- 172.21.15.189:0/1490767497 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f053c08aa00 con 0x7f05480d17a0 2024-04-06T12:27:08.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.611+0000 7f05477fe700 1 --2- 172.21.15.189:0/1490767497 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f053406adb0 0x7f053406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:08.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.615+0000 7f052ffff700 1 -- 172.21.15.189:0/1490767497 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f05480afee0 con 0x7f05480d17a0 2024-04-06T12:27:08.803 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:08.806+0000 7fbc5e4e5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:27:08.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.812+0000 7f05477fe700 1 -- 172.21.15.189:0/1490767497 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f053406adb0 msgr2=0x7f053406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:08.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.812+0000 7f05477fe700 1 --2- 172.21.15.189:0/1490767497 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f053406adb0 0x7f053406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:08.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.869+0000 7f054e038700 1 -- 172.21.15.189:0/1490767497 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05480afee0 con 0x7f05480d17a0 2024-04-06T12:27:08.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.869+0000 7f052ffff700 1 -- 172.21.15.189:0/1490767497 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f05480afee0 con 0x7f05480d17a0 2024-04-06T12:27:08.867 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:08.867 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/527093473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":527093473},{"type":"v1","addr":"172.21.15.192:6835","nonce":527093473}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6105,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2142144473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2142144473},{"type":"v1","addr":"172.21.15.189:6836","nonce":2142144473}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6108,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2285762709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2285762709},{"type":"v1","addr":"172.21.15.189:6837","nonce":2285762709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:25:02.062461+0000","modified":"2024-04-06T12:25:05.077648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6834/525827513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":525827513},{"type":"v1","addr":"172.21.15.192:6834","nonce":525827513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-06T12:27:08.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.872+0000 7f054e038700 1 -- 172.21.15.189:0/1490767497 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f053406adb0 msgr2=0x7f053406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:08.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.872+0000 7f054e038700 1 --2- 172.21.15.189:0/1490767497 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f053406adb0 0x7f053406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.872+0000 7f054e038700 1 -- 172.21.15.189:0/1490767497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05480d17a0 msgr2=0x7f05480c6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:08.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.872+0000 7f054e038700 1 --2- 172.21.15.189:0/1490767497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05480d17a0 0x7f05480c6f40 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f053c009b90 tx=0x7f053c004750 comp rx=0 tx=0).stop 2024-04-06T12:27:08.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.872+0000 7f054e038700 1 -- 172.21.15.189:0/1490767497 shutdown_connections 2024-04-06T12:27:08.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.872+0000 7f054e038700 1 --2- 172.21.15.189:0/1490767497 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0548146dd0 0x7f054815aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.872+0000 7f054e038700 1 --2- 172.21.15.189:0/1490767497 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f053406adb0 0x7f053406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.872+0000 7f054e038700 1 --2- 172.21.15.189:0/1490767497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05480cedf0 0x7f05480b01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.872+0000 7f054e038700 1 --2- 172.21.15.189:0/1490767497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05480d17a0 0x7f05480c6f40 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:08.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.872+0000 7f054e038700 1 -- 172.21.15.189:0/1490767497 >> 172.21.15.189:0/1490767497 conn(0x7f05480b2db0 msgr2=0x7f0548160250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:08.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.872+0000 7f054e038700 1 -- 172.21.15.189:0/1490767497 shutdown_connections 2024-04-06T12:27:08.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:08.872+0000 7f054e038700 1 -- 172.21.15.189:0/1490767497 wait complete. 2024-04-06T12:27:08.871 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 24 2024-04-06T12:27:08.882 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:27:08.882 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:08.884+0000 7fbc5e4e5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:27:08.945 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:08.948+0000 7fe88a431200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:27:09.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.010+0000 7f0b3c13a700 1 Processor -- start 2024-04-06T12:27:09.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.011+0000 7f0b3c13a700 1 -- start start 2024-04-06T12:27:09.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.012+0000 7f0b3c13a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0b340cb870 0x7f0b3415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:09.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.012+0000 7f0b3c13a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b340cbd80 con 0x7f0b340cfa00 2024-04-06T12:27:09.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.012+0000 7f0b3c13a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b340cef20 con 0x7f0b340d11b0 2024-04-06T12:27:09.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.012+0000 7f0b3c13a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b340ccdc0 con 0x7f0b340cb870 2024-04-06T12:27:09.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.012+0000 7f0b396d5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0b340cb870 0x7f0b3415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:09.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.012+0000 7f0b396d5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0b340cb870 0x7f0b3415c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36556/0 (socket says 172.21.15.189:36556) 2024-04-06T12:27:09.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.012+0000 7f0b396d5700 1 -- 172.21.15.189:0/2996867752 learned_addr learned my addr 172.21.15.189:0/2996867752 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:09.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.012+0000 7f0b396d5700 1 -- 172.21.15.189:0/2996867752 >> v1:172.21.15.192:6789/0 conn(0x7f0b340d11b0 legacy=0x7f0b340d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:09.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.012+0000 7f0b396d5700 1 -- 172.21.15.189:0/2996867752 >> v1:172.21.15.189:6789/0 conn(0x7f0b340cfa00 legacy=0x7f0b340cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:09.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.012+0000 7f0b396d5700 1 -- 172.21.15.189:0/2996867752 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b340cf200 con 0x7f0b340cb870 2024-04-06T12:27:09.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.012+0000 7f0b396d5700 1 --2- 172.21.15.189:0/2996867752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b340cb870 0x7f0b3415c280 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f0b240098f0 tx=0x7f0b24009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dab4c381b84ffdc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:09.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.013+0000 7f0b38ed4700 1 -- 172.21.15.189:0/2996867752 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2400b8a0 con 0x7f0b340cb870 2024-04-06T12:27:09.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.013+0000 7f0b38ed4700 1 -- 172.21.15.189:0/2996867752 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2400ba00 con 0x7f0b340cb870 2024-04-06T12:27:09.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.013+0000 7f0b38ed4700 1 -- 172.21.15.189:0/2996867752 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2400bd90 con 0x7f0b340cb870 2024-04-06T12:27:09.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.013+0000 7f0b3c13a700 1 -- 172.21.15.189:0/2996867752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b340cb870 msgr2=0x7f0b3415c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:09.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.013+0000 7f0b3c13a700 1 --2- 172.21.15.189:0/2996867752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b340cb870 0x7f0b3415c280 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f0b240098f0 tx=0x7f0b24009c00 comp rx=0 tx=0).stop 2024-04-06T12:27:09.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.013+0000 7f0b3c13a700 1 -- 172.21.15.189:0/2996867752 shutdown_connections 2024-04-06T12:27:09.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.013+0000 7f0b3c13a700 1 --2- 172.21.15.189:0/2996867752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b340cb870 0x7f0b3415c280 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.013+0000 7f0b3c13a700 1 -- 172.21.15.189:0/2996867752 >> 172.21.15.189:0/2996867752 conn(0x7f0b340b2db0 msgr2=0x7f0b340b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:09.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.013+0000 7f0b3c13a700 1 -- 172.21.15.189:0/2996867752 shutdown_connections 2024-04-06T12:27:09.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.013+0000 7f0b3c13a700 1 -- 172.21.15.189:0/2996867752 wait complete. 2024-04-06T12:27:09.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.015+0000 7f0b3c13a700 1 Processor -- start 2024-04-06T12:27:09.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.015+0000 7f0b3c13a700 1 -- start start 2024-04-06T12:27:09.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.016+0000 7f0b3c13a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b340cb870 0x7f0b34157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.016+0000 7f0b3c13a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b340cfa00 0x7f0b34155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.016+0000 7f0b3c13a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b340d11b0 0x7f0b341564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.016+0000 7f0b3c13a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b341582f0 con 0x7f0b340cfa00 2024-04-06T12:27:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.016+0000 7f0b3c13a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3406bf30 con 0x7f0b340d11b0 2024-04-06T12:27:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.016+0000 7f0b3c13a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b340c7d50 con 0x7f0b340cb870 2024-04-06T12:27:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.016+0000 7f0b396d5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b340cfa00 0x7f0b34155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.016+0000 7f0b396d5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b340cfa00 0x7f0b34155f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36572/0 (socket says 172.21.15.189:36572) 2024-04-06T12:27:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.016+0000 7f0b396d5700 1 -- 172.21.15.189:0/3752492363 learned_addr learned my addr 172.21.15.189:0/3752492363 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.016+0000 7f0b3a6d7700 1 --2- 172.21.15.189:0/3752492363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b340d11b0 0x7f0b341564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.016+0000 7f0b39ed6700 1 --2- 172.21.15.189:0/3752492363 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b340cb870 0x7f0b34157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.016+0000 7f0b396d5700 1 -- 172.21.15.189:0/3752492363 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b340cb870 msgr2=0x7f0b34157380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.016+0000 7f0b396d5700 1 --2- 172.21.15.189:0/3752492363 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b340cb870 0x7f0b34157380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.016+0000 7f0b396d5700 1 -- 172.21.15.189:0/3752492363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b340d11b0 msgr2=0x7f0b341564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.016+0000 7f0b396d5700 1 --2- 172.21.15.189:0/3752492363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b340d11b0 0x7f0b341564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.016+0000 7f0b396d5700 1 -- 172.21.15.189:0/3752492363 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b24009550 con 0x7f0b340cfa00 2024-04-06T12:27:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.016+0000 7f0b396d5700 1 --2- 172.21.15.189:0/3752492363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b340cfa00 0x7f0b34155f80 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0b24000c00 tx=0x7f0b24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.017+0000 7f0b2a7fc700 1 -- 172.21.15.189:0/3752492363 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b24004590 con 0x7f0b340cfa00 2024-04-06T12:27:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.017+0000 7f0b2a7fc700 1 -- 172.21.15.189:0/3752492363 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b240036a0 con 0x7f0b340cfa00 2024-04-06T12:27:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.017+0000 7f0b2a7fc700 1 -- 172.21.15.189:0/3752492363 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b24003a50 con 0x7f0b340cfa00 2024-04-06T12:27:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.017+0000 7f0b3c13a700 1 -- 172.21.15.189:0/3752492363 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b340c7a70 con 0x7f0b340cfa00 2024-04-06T12:27:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.017+0000 7f0b3c13a700 1 -- 172.21.15.189:0/3752492363 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3406bbc0 con 0x7f0b340cfa00 2024-04-06T12:27:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.018+0000 7f0b3c13a700 1 -- 172.21.15.189:0/3752492363 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3406c0e0 con 0x7f0b340cfa00 2024-04-06T12:27:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.018+0000 7f0b2a7fc700 1 -- 172.21.15.189:0/3752492363 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 28) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f0b24021070 con 0x7f0b340cfa00 2024-04-06T12:27:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.018+0000 7f0b2a7fc700 1 --2- 172.21.15.189:0/3752492363 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b1806adb0 0x7f0b1806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.019+0000 7f0b39ed6700 1 -- 172.21.15.189:0/3752492363 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b1806adb0 msgr2=0x7f0b1806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.019+0000 7f0b2a7fc700 1 -- 172.21.15.189:0/3752492363 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f0b2408ac90 con 0x7f0b340cfa00 2024-04-06T12:27:09.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.019+0000 7f0b39ed6700 1 --2- 172.21.15.189:0/3752492363 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b1806adb0 0x7f0b1806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:09.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.023+0000 7f0b2a7fc700 1 -- 172.21.15.189:0/3752492363 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0b3406c0e0 con 0x7f0b340cfa00 2024-04-06T12:27:09.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.220+0000 7f0b39ed6700 1 -- 172.21.15.189:0/3752492363 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b1806adb0 msgr2=0x7f0b1806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:09.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.220+0000 7f0b39ed6700 1 --2- 172.21.15.189:0/3752492363 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b1806adb0 0x7f0b1806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:09.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.270+0000 7f0b3c13a700 1 -- 172.21.15.189:0/3752492363 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b3406c0e0 con 0x7f0b340cfa00 2024-04-06T12:27:09.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.270+0000 7f0b2a7fc700 1 -- 172.21.15.189:0/3752492363 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f0b3406c0e0 con 0x7f0b340cfa00 2024-04-06T12:27:09.268 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:09.268 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/527093473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":527093473},{"type":"v1","addr":"172.21.15.192:6835","nonce":527093473}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6105,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2142144473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2142144473},{"type":"v1","addr":"172.21.15.189:6836","nonce":2142144473}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6108,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2285762709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2285762709},{"type":"v1","addr":"172.21.15.189:6837","nonce":2285762709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:25:02.062461+0000","modified":"2024-04-06T12:25:05.077648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6834/525827513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":525827513},{"type":"v1","addr":"172.21.15.192:6834","nonce":525827513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-06T12:27:09.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.273+0000 7f0b3c13a700 1 -- 172.21.15.189:0/3752492363 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b1806adb0 msgr2=0x7f0b1806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:09.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.273+0000 7f0b3c13a700 1 --2- 172.21.15.189:0/3752492363 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b1806adb0 0x7f0b1806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.273+0000 7f0b3c13a700 1 -- 172.21.15.189:0/3752492363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b340cfa00 msgr2=0x7f0b34155f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:09.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.273+0000 7f0b3c13a700 1 --2- 172.21.15.189:0/3752492363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b340cfa00 0x7f0b34155f80 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0b24000c00 tx=0x7f0b24004750 comp rx=0 tx=0).stop 2024-04-06T12:27:09.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.273+0000 7f0b3c13a700 1 -- 172.21.15.189:0/3752492363 shutdown_connections 2024-04-06T12:27:09.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.273+0000 7f0b3c13a700 1 --2- 172.21.15.189:0/3752492363 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b1806adb0 0x7f0b1806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.273+0000 7f0b3c13a700 1 --2- 172.21.15.189:0/3752492363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b340d11b0 0x7f0b341564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.273+0000 7f0b3c13a700 1 --2- 172.21.15.189:0/3752492363 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b340cb870 0x7f0b34157380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.273+0000 7f0b3c13a700 1 --2- 172.21.15.189:0/3752492363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b340cfa00 0x7f0b34155f80 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.273+0000 7f0b3c13a700 1 -- 172.21.15.189:0/3752492363 >> 172.21.15.189:0/3752492363 conn(0x7f0b340b2db0 msgr2=0x7f0b340b3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:09.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.273+0000 7f0b3c13a700 1 -- 172.21.15.189:0/3752492363 shutdown_connections 2024-04-06T12:27:09.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.273+0000 7f0b3c13a700 1 -- 172.21.15.189:0/3752492363 wait complete. 2024-04-06T12:27:09.273 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 24 2024-04-06T12:27:09.274 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:09.277+0000 7fbc5e4e5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:27:09.283 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:27:09.355 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:09.358+0000 7fbbd9bc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:09.355 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:09.358+0000 7fbbd9bc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:09.355 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:09.358+0000 7fbbd9bc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:09.355 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:09.358+0000 7fbbd9bc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:09.355 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:09.358+0000 7fbbd9bc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:09.355 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:09.358+0000 7fbbd5bbc700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:09.355 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:09.358+0000 7fbbd5bbc700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:09.355 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:09.358+0000 7fbbd5bbc700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:09.355 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:09.358+0000 7fbbd5bbc700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:09.356 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:09.358+0000 7fbbd5bbc700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:09.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.420+0000 7f22e0bc1700 1 Processor -- start 2024-04-06T12:27:09.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.420+0000 7f22e0bc1700 1 -- start start 2024-04-06T12:27:09.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.421+0000 7f22e0bc1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22dc06b8c0 0x7f22dc168420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:09.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.421+0000 7f22e0bc1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22dc06ceb0 con 0x7f22dc0cd3e0 2024-04-06T12:27:09.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.421+0000 7f22e0bc1700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22dc06d750 con 0x7f22dc15d7c0 2024-04-06T12:27:09.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.421+0000 7f22e0bc1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22dc0cdfa0 con 0x7f22dc06b8c0 2024-04-06T12:27:09.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.421+0000 7f22da59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f22dc0cd3e0 0x7f22dc0cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37452/0 (socket says 172.21.15.189:37452) 2024-04-06T12:27:09.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.421+0000 7f22da59c700 1 -- 172.21.15.189:0/4107062800 learned_addr learned my addr 172.21.15.189:0/4107062800 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:09.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.421+0000 7f22dad9d700 1 --2- 172.21.15.189:0/4107062800 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22dc06b8c0 0x7f22dc168420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:09.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.421+0000 7f22d959a700 1 -- 172.21.15.189:0/4107062800 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984488365 0 0) 0x7f22dc06ceb0 con 0x7f22dc0cd3e0 2024-04-06T12:27:09.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.421+0000 7f22d959a700 1 -- 172.21.15.189:0/4107062800 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22c40035b0 con 0x7f22dc0cd3e0 2024-04-06T12:27:09.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.421+0000 7f22d959a700 1 -- 172.21.15.189:0/4107062800 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3862350316 0 0) 0x7f22c40035b0 con 0x7f22dc0cd3e0 2024-04-06T12:27:09.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.421+0000 7f22d959a700 1 -- 172.21.15.189:0/4107062800 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22dc06b8c0 msgr2=0x7f22dc168420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:09.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.421+0000 7f22d959a700 1 --2- 172.21.15.189:0/4107062800 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22dc06b8c0 0x7f22dc168420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.421+0000 7f22d959a700 1 -- 172.21.15.189:0/4107062800 >> v1:172.21.15.192:6789/0 conn(0x7f22dc15d7c0 legacy=0x7f22dc06b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:09.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.421+0000 7f22d959a700 1 -- 172.21.15.189:0/4107062800 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22dc06bf00 con 0x7f22dc0cd3e0 2024-04-06T12:27:09.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.421+0000 7f22d959a700 1 -- 172.21.15.189:0/4107062800 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f22cc001670 con 0x7f22dc0cd3e0 2024-04-06T12:27:09.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.421+0000 7f22d959a700 1 -- 172.21.15.189:0/4107062800 >> v1:172.21.15.189:6789/0 conn(0x7f22dc0cd3e0 legacy=0x7f22dc0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:09.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.422+0000 7f22d959a700 1 --2- 172.21.15.189:0/4107062800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22c4003a30 0x7f22c4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:09.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.422+0000 7f22d959a700 1 --2- 172.21.15.189:0/4107062800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22c4004850 0x7f22c4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:09.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.422+0000 7f22d959a700 1 --2- 172.21.15.189:0/4107062800 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22dc06b8c0 0x7f22c4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:09.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.422+0000 7f22d959a700 1 -- 172.21.15.189:0/4107062800 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22dc0cdfa0 con 0x7f22c4004850 2024-04-06T12:27:09.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.422+0000 7f22d959a700 1 -- 172.21.15.189:0/4107062800 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c400bb10 con 0x7f22c4003a30 2024-04-06T12:27:09.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.422+0000 7f22d959a700 1 -- 172.21.15.189:0/4107062800 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c400bcd0 con 0x7f22dc06b8c0 2024-04-06T12:27:09.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.422+0000 7f22d9d9b700 1 --2- 172.21.15.189:0/4107062800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22c4004850 0x7f22c4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:09.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.422+0000 7f22da59c700 1 --2- 172.21.15.189:0/4107062800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22c4003a30 0x7f22c4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:09.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.422+0000 7f22dad9d700 1 --2- 172.21.15.189:0/4107062800 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22dc06b8c0 0x7f22c4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:09.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.422+0000 7f22d9d9b700 1 -- 172.21.15.189:0/4107062800 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22dc06b8c0 msgr2=0x7f22c4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:09.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.422+0000 7f22d9d9b700 1 --2- 172.21.15.189:0/4107062800 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22dc06b8c0 0x7f22c4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.422+0000 7f22d9d9b700 1 -- 172.21.15.189:0/4107062800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22c4003a30 msgr2=0x7f22c4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:09.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.422+0000 7f22d9d9b700 1 --2- 172.21.15.189:0/4107062800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22c4003a30 0x7f22c4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.422+0000 7f22d9d9b700 1 -- 172.21.15.189:0/4107062800 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c40035b0 con 0x7f22c4004850 2024-04-06T12:27:09.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.422+0000 7f22d9d9b700 1 --2- 172.21.15.189:0/4107062800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22c4004850 0x7f22c4004cd0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f22d00018d0 tx=0x7f22c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ca0d93295cd8a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:09.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.422+0000 7f22d959a700 1 -- 172.21.15.189:0/4107062800 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d000c690 con 0x7f22c4004850 2024-04-06T12:27:09.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.422+0000 7f22d959a700 1 -- 172.21.15.189:0/4107062800 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d000c7f0 con 0x7f22c4004850 2024-04-06T12:27:09.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.422+0000 7f22d959a700 1 -- 172.21.15.189:0/4107062800 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d000cba0 con 0x7f22c4004850 2024-04-06T12:27:09.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.423+0000 7f22e0bc1700 1 -- 172.21.15.189:0/4107062800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22c4004850 msgr2=0x7f22c4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:09.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.423+0000 7f22e0bc1700 1 --2- 172.21.15.189:0/4107062800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22c4004850 0x7f22c4004cd0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f22d00018d0 tx=0x7f22c4005880 comp rx=0 tx=0).stop 2024-04-06T12:27:09.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.423+0000 7f22dad9d700 1 -- 172.21.15.189:0/4107062800 reap_dead start 2024-04-06T12:27:09.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.423+0000 7f22e0bc1700 1 -- 172.21.15.189:0/4107062800 shutdown_connections 2024-04-06T12:27:09.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.423+0000 7f22e0bc1700 1 -- 172.21.15.189:0/4107062800 >> 172.21.15.189:0/4107062800 conn(0x7f22dc0cc290 msgr2=0x7f22dc0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:09.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.423+0000 7f22e0bc1700 1 -- 172.21.15.189:0/4107062800 shutdown_connections 2024-04-06T12:27:09.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.423+0000 7f22e0bc1700 1 -- 172.21.15.189:0/4107062800 wait complete. 2024-04-06T12:27:09.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.424+0000 7f22e0bc1700 1 Processor -- start 2024-04-06T12:27:09.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.424+0000 7f22e0bc1700 1 -- start start 2024-04-06T12:27:09.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.425+0000 7f22e0bc1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22c4004850 0x7f22dc0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:09.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.425+0000 7f22e0bc1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22dc159480 0x7f22dc157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:09.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.425+0000 7f22e0bc1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22dc15bfb0 0x7f22dc0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:09.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.425+0000 7f22e0bc1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22dc15b840 con 0x7f22dc15bfb0 2024-04-06T12:27:09.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.425+0000 7f22e0bc1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22dc15bb20 con 0x7f22dc159480 2024-04-06T12:27:09.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.425+0000 7f22e0bc1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22dc0c5a50 con 0x7f22c4004850 2024-04-06T12:27:09.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.425+0000 7f22dad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22dc15bfb0 0x7f22dc0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:09.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.425+0000 7f22dad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22dc15bfb0 0x7f22dc0b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37222/0 (socket says 172.21.15.189:37222) 2024-04-06T12:27:09.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.425+0000 7f22dad9d700 1 -- 172.21.15.189:0/2813230808 learned_addr learned my addr 172.21.15.189:0/2813230808 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:09.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.425+0000 7f22d9d9b700 1 --2- 172.21.15.189:0/2813230808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22dc159480 0x7f22dc157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:09.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.425+0000 7f22da59c700 1 --2- 172.21.15.189:0/2813230808 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22c4004850 0x7f22dc0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:09.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.425+0000 7f22dad9d700 1 -- 172.21.15.189:0/2813230808 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22c4004850 msgr2=0x7f22dc0b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:09.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.425+0000 7f22dad9d700 1 --2- 172.21.15.189:0/2813230808 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22c4004850 0x7f22dc0b4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.425+0000 7f22dad9d700 1 -- 172.21.15.189:0/2813230808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22dc159480 msgr2=0x7f22dc157e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:09.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.425+0000 7f22dad9d700 1 --2- 172.21.15.189:0/2813230808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22dc159480 0x7f22dc157e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.425+0000 7f22dad9d700 1 -- 172.21.15.189:0/2813230808 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22dc0c5cf0 con 0x7f22dc15bfb0 2024-04-06T12:27:09.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.425+0000 7f22da59c700 1 --2- 172.21.15.189:0/2813230808 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22c4004850 0x7f22dc0b4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:09.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.426+0000 7f22dad9d700 1 --2- 172.21.15.189:0/2813230808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22dc15bfb0 0x7f22dc0b3e30 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f22c40047d0 tx=0x7f22d4011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:09.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.426+0000 7f22caffd700 1 -- 172.21.15.189:0/2813230808 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d400e040 con 0x7f22dc15bfb0 2024-04-06T12:27:09.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.426+0000 7f22caffd700 1 -- 172.21.15.189:0/2813230808 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d400f040 con 0x7f22dc15bfb0 2024-04-06T12:27:09.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.426+0000 7f22e0bc1700 1 -- 172.21.15.189:0/2813230808 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22dc0c5fd0 con 0x7f22dc15bfb0 2024-04-06T12:27:09.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.426+0000 7f22caffd700 1 -- 172.21.15.189:0/2813230808 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d4015430 con 0x7f22dc15bfb0 2024-04-06T12:27:09.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.426+0000 7f22e0bc1700 1 -- 172.21.15.189:0/2813230808 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22dc0b0fa0 con 0x7f22dc15bfb0 2024-04-06T12:27:09.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.427+0000 7f22e0bc1700 1 -- 172.21.15.189:0/2813230808 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22dc0c61c0 con 0x7f22dc15bfb0 2024-04-06T12:27:09.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.427+0000 7f22caffd700 1 -- 172.21.15.189:0/2813230808 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 29) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f22d4015590 con 0x7f22dc15bfb0 2024-04-06T12:27:09.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.428+0000 7f22caffd700 1 -- 172.21.15.189:0/2813230808 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6717+0+0 (secure 0 0 0) 0x7f22d4089740 con 0x7f22dc15bfb0 2024-04-06T12:27:09.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.432+0000 7f22caffd700 1 -- 172.21.15.189:0/2813230808 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f22d4054980 con 0x7f22dc15bfb0 2024-04-06T12:27:09.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.627+0000 7f22e0bc1700 1 -- 172.21.15.189:0/2813230808 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22dc074990 con 0x7f22dc15bfb0 2024-04-06T12:27:09.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.629+0000 7f22caffd700 1 -- 172.21.15.189:0/2813230808 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+17332 (secure 0 0 0) 0x7f22d4057fa0 con 0x7f22dc15bfb0 2024-04-06T12:27:09.626 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:09.626 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":72,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:27:09.291244+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:24:59.828905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-06T12:25:01.020253+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.23","mappings":[{"from":0,"to":1}]},{"pgid":"8.38","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/2310280005":"2024-04-07T12:27:09.291204+0000","172.21.15.189:0/1187685283":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/1356350621":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/1784361417":"2024-04-07T12:27:09.291204+0000","172.21.15.189:0/162292108":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/829802120":"2024-04-07T12:27:09.291204+0000","172.21.15.189:0/2706754156":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/979582492":"2024-04-07T12:27:09.291204+0000","172.21.15.189:0/1122382676":"2024-04-07T12:25:15.982358+0000","172.21.15.189:0/448783303":"2024-04-07T12:27:09.291204+0000","172.21.15.189:0/2189204741":"2024-04-07T12:27:09.291204+0000","172.21.15.189:0/2132037261":"2024-04-07T12:25:15.982358+0000","172.21.15.189:6833/47128":"2024-04-07T12:25:15.982358+0000","172.21.15.189:6832/47128":"2024-04-07T12:25:15.982358+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:27:09.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.631+0000 7f22e0bc1700 1 -- 172.21.15.189:0/2813230808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22dc15bfb0 msgr2=0x7f22dc0b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:09.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.631+0000 7f22e0bc1700 1 --2- 172.21.15.189:0/2813230808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22dc15bfb0 0x7f22dc0b3e30 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f22c40047d0 tx=0x7f22d4011f60 comp rx=0 tx=0).stop 2024-04-06T12:27:09.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.631+0000 7f22e0bc1700 1 -- 172.21.15.189:0/2813230808 shutdown_connections 2024-04-06T12:27:09.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.631+0000 7f22e0bc1700 1 --2- 172.21.15.189:0/2813230808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22dc15bfb0 0x7f22dc0b3e30 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.631+0000 7f22e0bc1700 1 --2- 172.21.15.189:0/2813230808 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22c4004850 0x7f22dc0b4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.631+0000 7f22e0bc1700 1 --2- 172.21.15.189:0/2813230808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22dc159480 0x7f22dc157e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.631+0000 7f22e0bc1700 1 -- 172.21.15.189:0/2813230808 >> 172.21.15.189:0/2813230808 conn(0x7f22dc0cc290 msgr2=0x7f22dc0ac8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:09.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.631+0000 7f22e0bc1700 1 -- 172.21.15.189:0/2813230808 shutdown_connections 2024-04-06T12:27:09.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.631+0000 7f22e0bc1700 1 -- 172.21.15.189:0/2813230808 wait complete. 2024-04-06T12:27:09.640 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-06T12:27:09.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.766+0000 7f5df1f36700 1 Processor -- start 2024-04-06T12:27:09.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.766+0000 7f5df1f36700 1 -- start start 2024-04-06T12:27:09.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5df1f36700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5dec0b39d0 0x7f5dec0af040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:09.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5df1f36700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dec071950 con 0x7f5dec15d7b0 2024-04-06T12:27:09.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5df1f36700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dec0b4170 con 0x7f5dec0723f0 2024-04-06T12:27:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5df1f36700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec0b3e90 con 0x7f5dec0b39d0 2024-04-06T12:27:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5deb7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5dec15d7b0 0x7f5dec0ac8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37464/0 (socket says 172.21.15.189:37464) 2024-04-06T12:27:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5deb7fe700 1 -- 172.21.15.189:0/3688861571 learned_addr learned my addr 172.21.15.189:0/3688861571 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5deaffd700 1 --2- 172.21.15.189:0/3688861571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5dec0b39d0 0x7f5dec0af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5dea7fc700 1 -- 172.21.15.189:0/3688861571 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754024178 0 0) 0x7f5dec071950 con 0x7f5dec15d7b0 2024-04-06T12:27:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5dea7fc700 1 -- 172.21.15.189:0/3688861571 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dd40035b0 con 0x7f5dec15d7b0 2024-04-06T12:27:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5dea7fc700 1 -- 172.21.15.189:0/3688861571 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3798166915 0 0) 0x7f5dd40035b0 con 0x7f5dec15d7b0 2024-04-06T12:27:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5dea7fc700 1 -- 172.21.15.189:0/3688861571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5dec0b39d0 msgr2=0x7f5dec0af040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5dea7fc700 1 --2- 172.21.15.189:0/3688861571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5dec0b39d0 0x7f5dec0af040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5dea7fc700 1 -- 172.21.15.189:0/3688861571 >> v1:172.21.15.192:6789/0 conn(0x7f5dec0723f0 legacy=0x7f5dec06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5dea7fc700 1 -- 172.21.15.189:0/3688861571 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec071bf0 con 0x7f5dec15d7b0 2024-04-06T12:27:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5dea7fc700 1 -- 172.21.15.189:0/3688861571 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f5de0001670 con 0x7f5dec15d7b0 2024-04-06T12:27:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5dea7fc700 1 -- 172.21.15.189:0/3688861571 >> v1:172.21.15.189:6789/0 conn(0x7f5dec15d7b0 legacy=0x7f5dec0ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:09.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5dea7fc700 1 --2- 172.21.15.189:0/3688861571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5dec0b39d0 0x7f5dd4003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5dea7fc700 1 --2- 172.21.15.189:0/3688861571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dd4003fb0 0x7f5dd4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5dea7fc700 1 --2- 172.21.15.189:0/3688861571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dd4004d90 0x7f5dd4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5dea7fc700 1 -- 172.21.15.189:0/3688861571 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec0b3e90 con 0x7f5dd4003fb0 2024-04-06T12:27:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5dea7fc700 1 -- 172.21.15.189:0/3688861571 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd400baf0 con 0x7f5dd4004d90 2024-04-06T12:27:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5dea7fc700 1 -- 172.21.15.189:0/3688861571 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd400bcb0 con 0x7f5dec0b39d0 2024-04-06T12:27:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5deaffd700 1 --2- 172.21.15.189:0/3688861571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dd4003fb0 0x7f5dd4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5deb7fe700 1 --2- 172.21.15.189:0/3688861571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5dec0b39d0 0x7f5dd4003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5debfff700 1 --2- 172.21.15.189:0/3688861571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dd4004d90 0x7f5dd4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5deaffd700 1 -- 172.21.15.189:0/3688861571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5dec0b39d0 msgr2=0x7f5dd4003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5deaffd700 1 --2- 172.21.15.189:0/3688861571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5dec0b39d0 0x7f5dd4003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5deaffd700 1 -- 172.21.15.189:0/3688861571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dd4004d90 msgr2=0x7f5dd4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5deaffd700 1 --2- 172.21.15.189:0/3688861571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dd4004d90 0x7f5dd4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5deaffd700 1 -- 172.21.15.189:0/3688861571 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd40035b0 con 0x7f5dd4003fb0 2024-04-06T12:27:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5deaffd700 1 --2- 172.21.15.189:0/3688861571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dd4003fb0 0x7f5dd4004850 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc009b60 tx=0x7f5dd4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f223a7833d0a8106 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:09.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5dea7fc700 1 -- 172.21.15.189:0/3688861571 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc00e6c0 con 0x7f5dd4003fb0 2024-04-06T12:27:09.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5dea7fc700 1 -- 172.21.15.189:0/3688861571 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ddc00e820 con 0x7f5dd4003fb0 2024-04-06T12:27:09.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.767+0000 7f5dea7fc700 1 -- 172.21.15.189:0/3688861571 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc00eb90 con 0x7f5dd4003fb0 2024-04-06T12:27:09.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.768+0000 7f5df1f36700 1 -- 172.21.15.189:0/3688861571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dd4003fb0 msgr2=0x7f5dd4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:09.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.768+0000 7f5df1f36700 1 --2- 172.21.15.189:0/3688861571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dd4003fb0 0x7f5dd4004850 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc009b60 tx=0x7f5dd4005880 comp rx=0 tx=0).stop 2024-04-06T12:27:09.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.768+0000 7f5debfff700 1 -- 172.21.15.189:0/3688861571 reap_dead start 2024-04-06T12:27:09.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.768+0000 7f5df1f36700 1 -- 172.21.15.189:0/3688861571 shutdown_connections 2024-04-06T12:27:09.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.768+0000 7f5df1f36700 1 -- 172.21.15.189:0/3688861571 >> 172.21.15.189:0/3688861571 conn(0x7f5dec0d00b0 msgr2=0x7f5dec0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:09.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.768+0000 7f5df1f36700 1 -- 172.21.15.189:0/3688861571 shutdown_connections 2024-04-06T12:27:09.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.768+0000 7f5df1f36700 1 -- 172.21.15.189:0/3688861571 wait complete. 2024-04-06T12:27:09.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.770+0000 7f5df1f36700 1 Processor -- start 2024-04-06T12:27:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.770+0000 7f5df1f36700 1 -- start start 2024-04-06T12:27:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5df1f36700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5dd4003fb0 0x7f5dec150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5df1f36700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dec1f3f30 0x7f5dec0cec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5df1f36700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dec15f600 0x7f5dec15fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5df1f36700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec15ffc0 con 0x7f5dec15f600 2024-04-06T12:27:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5df1f36700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec0cd1d0 con 0x7f5dec1f3f30 2024-04-06T12:27:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5df1f36700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec0cc5c0 con 0x7f5dd4003fb0 2024-04-06T12:27:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5debfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dec15f600 0x7f5dec15fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5debfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dec15f600 0x7f5dec15fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37238/0 (socket says 172.21.15.189:37238) 2024-04-06T12:27:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5debfff700 1 -- 172.21.15.189:0/665412687 learned_addr learned my addr 172.21.15.189:0/665412687 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5deb7fe700 1 --2- 172.21.15.189:0/665412687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5dd4003fb0 0x7f5dec150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5deaffd700 1 --2- 172.21.15.189:0/665412687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dec1f3f30 0x7f5dec0cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:09.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5debfff700 1 -- 172.21.15.189:0/665412687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5dd4003fb0 msgr2=0x7f5dec150f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:09.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5debfff700 1 --2- 172.21.15.189:0/665412687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5dd4003fb0 0x7f5dec150f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5debfff700 1 -- 172.21.15.189:0/665412687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dec1f3f30 msgr2=0x7f5dec0cec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:09.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5debfff700 1 --2- 172.21.15.189:0/665412687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dec1f3f30 0x7f5dec0cec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:09.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5debfff700 1 -- 172.21.15.189:0/665412687 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec0cc8a0 con 0x7f5dec15f600 2024-04-06T12:27:09.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5debfff700 1 --2- 172.21.15.189:0/665412687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dec15f600 0x7f5dec15fa80 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f5dec0d0ba0 tx=0x7f5de400d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:09.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5dd3fff700 1 -- 172.21.15.189:0/665412687 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de40020c0 con 0x7f5dec15f600 2024-04-06T12:27:09.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5dd3fff700 1 -- 172.21.15.189:0/665412687 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de4003c10 con 0x7f5dec15f600 2024-04-06T12:27:09.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5df1f36700 1 -- 172.21.15.189:0/665412687 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dec0ccb80 con 0x7f5dec15f600 2024-04-06T12:27:09.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.771+0000 7f5dd3fff700 1 -- 172.21.15.189:0/665412687 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de400d8e0 con 0x7f5dec15f600 2024-04-06T12:27:09.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.772+0000 7f5df1f36700 1 -- 172.21.15.189:0/665412687 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dec1f43b0 con 0x7f5dec15f600 2024-04-06T12:27:09.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.772+0000 7f5df1f36700 1 -- 172.21.15.189:0/665412687 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dec073200 con 0x7f5dec15f600 2024-04-06T12:27:09.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.773+0000 7f5dd3fff700 1 -- 172.21.15.189:0/665412687 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 29) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f5de400da40 con 0x7f5dec15f600 2024-04-06T12:27:09.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.773+0000 7f5dd3fff700 1 -- 172.21.15.189:0/665412687 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6717+0+0 (secure 0 0 0) 0x7f5de4089350 con 0x7f5dec15f600 2024-04-06T12:27:09.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:09.778+0000 7f5dd3fff700 1 -- 172.21.15.189:0/665412687 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5dec073200 con 0x7f5dec15f600 2024-04-06T12:27:10.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.018+0000 7f5df1f36700 1 -- 172.21.15.189:0/665412687 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5dec073200 con 0x7f5dec15f600 2024-04-06T12:27:10.017 INFO:tasks.ceph.mon.a.smithi189.stderr:2024-04-06T12:27:10.019+0000 7fa9fe1ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T12:27:10.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.029+0000 7f5dd3fff700 1 -- 172.21.15.189:0/665412687 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v25) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5dec073200 con 0x7f5dec15f600 2024-04-06T12:27:10.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.031+0000 7f5df1f36700 1 -- 172.21.15.189:0/665412687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dec15f600 msgr2=0x7f5dec15fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:10.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.031+0000 7f5df1f36700 1 --2- 172.21.15.189:0/665412687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dec15f600 0x7f5dec15fa80 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f5dec0d0ba0 tx=0x7f5de400d430 comp rx=0 tx=0).stop 2024-04-06T12:27:10.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.031+0000 7f5df1f36700 1 -- 172.21.15.189:0/665412687 shutdown_connections 2024-04-06T12:27:10.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.031+0000 7f5df1f36700 1 --2- 172.21.15.189:0/665412687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dec15f600 0x7f5dec15fa80 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:10.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.031+0000 7f5df1f36700 1 --2- 172.21.15.189:0/665412687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5dd4003fb0 0x7f5dec150f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:10.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.031+0000 7f5df1f36700 1 --2- 172.21.15.189:0/665412687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dec1f3f30 0x7f5dec0cec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:10.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.031+0000 7f5df1f36700 1 -- 172.21.15.189:0/665412687 >> 172.21.15.189:0/665412687 conn(0x7f5dec0d00b0 msgr2=0x7f5dec06afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:10.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.032+0000 7f5df1f36700 1 -- 172.21.15.189:0/665412687 shutdown_connections 2024-04-06T12:27:10.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.032+0000 7f5df1f36700 1 -- 172.21.15.189:0/665412687 wait complete. 2024-04-06T12:27:10.031 INFO:teuthology.orchestra.run.smithi189.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-06T12:27:10.042 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-06T12:27:10.065 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T12:27:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.169+0000 7f5c30d16700 1 Processor -- start 2024-04-06T12:27:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.169+0000 7f5c30d16700 1 -- start start 2024-04-06T12:27:10.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.170+0000 7f5c30d16700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5c2c143470 0x7f5c2c0b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:10.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.170+0000 7f5c30d16700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c15eb40 con 0x7f5c2c160b60 2024-04-06T12:27:10.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.170+0000 7f5c30d16700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c0d0bf0 con 0x7f5c2c15db10 2024-04-06T12:27:10.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.170+0000 7f5c30d16700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c2c160f40 con 0x7f5c2c143470 2024-04-06T12:27:10.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.170+0000 7f5c2ad9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5c2c143470 0x7f5c2c0b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:10.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.170+0000 7f5c2a59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5c2c160b60 0x7f5c2c142d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37478/0 (socket says 172.21.15.189:37478) 2024-04-06T12:27:10.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.170+0000 7f5c2a59c700 1 -- 172.21.15.189:0/2619384219 learned_addr learned my addr 172.21.15.189:0/2619384219 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:10.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.170+0000 7f5c2ad9d700 1 -- 172.21.15.189:0/2619384219 >> v1:172.21.15.192:6789/0 conn(0x7f5c2c15db10 legacy=0x7f5c2c0cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:10.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.170+0000 7f5c2ad9d700 1 -- 172.21.15.189:0/2619384219 >> v1:172.21.15.189:6789/0 conn(0x7f5c2c160b60 legacy=0x7f5c2c142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:10.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.170+0000 7f5c2ad9d700 1 -- 172.21.15.189:0/2619384219 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2c15dfc0 con 0x7f5c2c143470 2024-04-06T12:27:10.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.170+0000 7f5c2959a700 1 -- 172.21.15.189:0/2619384219 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205370019 0 0) 0x7f5c20001110 con 0x7f5c2c160b60 2024-04-06T12:27:10.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.171+0000 7f5c2ad9d700 1 --2- 172.21.15.189:0/2619384219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2c143470 0x7f5c2c0b37f0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f5c24009d90 tx=0x7f5c2400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9142d3752f885f5d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:10.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.171+0000 7f5c2959a700 1 -- 172.21.15.189:0/2619384219 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2400e070 con 0x7f5c2c143470 2024-04-06T12:27:10.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.171+0000 7f5c2959a700 1 -- 172.21.15.189:0/2619384219 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2400bbd0 con 0x7f5c2c143470 2024-04-06T12:27:10.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.171+0000 7f5c2959a700 1 -- 172.21.15.189:0/2619384219 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c24012430 con 0x7f5c2c143470 2024-04-06T12:27:10.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.171+0000 7f5c30d16700 1 -- 172.21.15.189:0/2619384219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2c143470 msgr2=0x7f5c2c0b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:10.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.171+0000 7f5c30d16700 1 --2- 172.21.15.189:0/2619384219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2c143470 0x7f5c2c0b37f0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f5c24009d90 tx=0x7f5c2400b330 comp rx=0 tx=0).stop 2024-04-06T12:27:10.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.171+0000 7f5c30d16700 1 -- 172.21.15.189:0/2619384219 shutdown_connections 2024-04-06T12:27:10.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.171+0000 7f5c30d16700 1 --2- 172.21.15.189:0/2619384219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2c143470 0x7f5c2c0b37f0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:10.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.171+0000 7f5c30d16700 1 -- 172.21.15.189:0/2619384219 >> 172.21.15.189:0/2619384219 conn(0x7f5c2c06a730 msgr2=0x7f5c2c06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:10.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.171+0000 7f5c30d16700 1 -- 172.21.15.189:0/2619384219 shutdown_connections 2024-04-06T12:27:10.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.171+0000 7f5c30d16700 1 -- 172.21.15.189:0/2619384219 wait complete. 2024-04-06T12:27:10.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.173+0000 7f5c30d16700 1 Processor -- start 2024-04-06T12:27:10.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.173+0000 7f5c30d16700 1 -- start start 2024-04-06T12:27:10.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.173+0000 7f5c30d16700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c2c143470 0x7f5c2c0c4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.173+0000 7f5c30d16700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c2c15db10 0x7f5c2c145560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.173+0000 7f5c30d16700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2c160b60 0x7f5c2c1448a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.173+0000 7f5c30d16700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c0c7bd0 con 0x7f5c2c160b60 2024-04-06T12:27:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.173+0000 7f5c30d16700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c0c6df0 con 0x7f5c2c143470 2024-04-06T12:27:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.173+0000 7f5c30d16700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c0c7090 con 0x7f5c2c15db10 2024-04-06T12:27:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.173+0000 7f5c2ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2c160b60 0x7f5c2c1448a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.173+0000 7f5c2ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2c160b60 0x7f5c2c1448a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37254/0 (socket says 172.21.15.189:37254) 2024-04-06T12:27:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.173+0000 7f5c2ad9d700 1 -- 172.21.15.189:0/3758075084 learned_addr learned my addr 172.21.15.189:0/3758075084 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.174+0000 7f5c2a59c700 1 --2- 172.21.15.189:0/3758075084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c2c143470 0x7f5c2c0c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.174+0000 7f5c29d9b700 1 --2- 172.21.15.189:0/3758075084 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c2c15db10 0x7f5c2c145560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.174+0000 7f5c2ad9d700 1 -- 172.21.15.189:0/3758075084 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c2c15db10 msgr2=0x7f5c2c145560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.174+0000 7f5c2ad9d700 1 --2- 172.21.15.189:0/3758075084 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c2c15db10 0x7f5c2c145560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.174+0000 7f5c2ad9d700 1 -- 172.21.15.189:0/3758075084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c2c143470 msgr2=0x7f5c2c0c4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:10.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.174+0000 7f5c2ad9d700 1 --2- 172.21.15.189:0/3758075084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c2c143470 0x7f5c2c0c4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:10.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.174+0000 7f5c2ad9d700 1 -- 172.21.15.189:0/3758075084 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c24009a70 con 0x7f5c2c160b60 2024-04-06T12:27:10.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.174+0000 7f5c2a59c700 1 --2- 172.21.15.189:0/3758075084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c2c143470 0x7f5c2c0c4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:10.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.174+0000 7f5c2ad9d700 1 --2- 172.21.15.189:0/3758075084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2c160b60 0x7f5c2c1448a0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f5c24014040 tx=0x7f5c24004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:10.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.174+0000 7f5c1affd700 1 -- 172.21.15.189:0/3758075084 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2401b810 con 0x7f5c2c160b60 2024-04-06T12:27:10.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.174+0000 7f5c1affd700 1 -- 172.21.15.189:0/3758075084 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c24020070 con 0x7f5c2c160b60 2024-04-06T12:27:10.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.174+0000 7f5c30d16700 1 -- 172.21.15.189:0/3758075084 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c2c0c49d0 con 0x7f5c2c160b60 2024-04-06T12:27:10.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.174+0000 7f5c1affd700 1 -- 172.21.15.189:0/3758075084 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2401bbb0 con 0x7f5c2c160b60 2024-04-06T12:27:10.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.174+0000 7f5c30d16700 1 -- 172.21.15.189:0/3758075084 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c2c0c7610 con 0x7f5c2c160b60 2024-04-06T12:27:10.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.175+0000 7f5c30d16700 1 -- 172.21.15.189:0/3758075084 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c2c1600a0 con 0x7f5c2c160b60 2024-04-06T12:27:10.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.175+0000 7f5c1affd700 1 -- 172.21.15.189:0/3758075084 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 29) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f5c2401bd10 con 0x7f5c2c160b60 2024-04-06T12:27:10.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.176+0000 7f5c1affd700 1 -- 172.21.15.189:0/3758075084 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6803+0+0 (secure 0 0 0) 0x7f5c24022070 con 0x7f5c2c160b60 2024-04-06T12:27:10.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.180+0000 7f5c1affd700 1 -- 172.21.15.189:0/3758075084 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5c24056650 con 0x7f5c2c160b60 2024-04-06T12:27:10.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.302+0000 7f5c1affd700 1 -- 172.21.15.189:0/3758075084 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 30) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f5c240562d0 con 0x7f5c2c160b60 2024-04-06T12:27:10.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.303+0000 7f5c1affd700 1 --2- 172.21.15.189:0/3758075084 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c0c036150 0x7f5c0c072a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:10.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.303+0000 7f5c2a59c700 1 --2- 172.21.15.189:0/3758075084 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c0c036150 0x7f5c0c072a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:10.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.305+0000 7f5c2a59c700 1 --2- 172.21.15.189:0/3758075084 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c0c036150 0x7f5c0c072a50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5c20000f60 tx=0x7f5c20007e50 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:10.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:10.429+0000 7f5c30d16700 1 -- 172.21.15.189:0/3758075084 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5c2c074990 con 0x7f5c2c160b60 2024-04-06T12:27:11.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.031+0000 7f5c1affd700 1 -- 172.21.15.189:0/3758075084 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5c2406e320 con 0x7f5c2c160b60 2024-04-06T12:27:11.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.034+0000 7f5c30d16700 1 -- 172.21.15.189:0/3758075084 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c0c036150 msgr2=0x7f5c0c072a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:11.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.034+0000 7f5c30d16700 1 --2- 172.21.15.189:0/3758075084 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c0c036150 0x7f5c0c072a50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5c20000f60 tx=0x7f5c20007e50 comp rx=0 tx=0).stop 2024-04-06T12:27:11.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.034+0000 7f5c30d16700 1 -- 172.21.15.189:0/3758075084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2c160b60 msgr2=0x7f5c2c1448a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:11.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.034+0000 7f5c30d16700 1 --2- 172.21.15.189:0/3758075084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2c160b60 0x7f5c2c1448a0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f5c24014040 tx=0x7f5c24004ad0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.034+0000 7f5c30d16700 1 -- 172.21.15.189:0/3758075084 shutdown_connections 2024-04-06T12:27:11.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.034+0000 7f5c30d16700 1 --2- 172.21.15.189:0/3758075084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2c160b60 0x7f5c2c1448a0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.034+0000 7f5c30d16700 1 --2- 172.21.15.189:0/3758075084 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c0c036150 0x7f5c0c072a50 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.034+0000 7f5c30d16700 1 --2- 172.21.15.189:0/3758075084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c2c143470 0x7f5c2c0c4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.034+0000 7f5c30d16700 1 --2- 172.21.15.189:0/3758075084 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c2c15db10 0x7f5c2c145560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.034+0000 7f5c30d16700 1 -- 172.21.15.189:0/3758075084 >> 172.21.15.189:0/3758075084 conn(0x7f5c2c06a730 msgr2=0x7f5c2c15f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:11.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.034+0000 7f5c30d16700 1 -- 172.21.15.189:0/3758075084 shutdown_connections 2024-04-06T12:27:11.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.034+0000 7f5c30d16700 1 -- 172.21.15.189:0/3758075084 wait complete. 2024-04-06T12:27:11.047 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-06T12:27:11.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.178+0000 7f1e0ffc7700 1 Processor -- start 2024-04-06T12:27:11.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.179+0000 7f1e0ffc7700 1 -- start start 2024-04-06T12:27:11.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.179+0000 7f1e0ffc7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e080c2a70 0x7f1e0806d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:11.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.179+0000 7f1e0ffc7700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1e080cdaf0 0x7f1e0806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:11.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.179+0000 7f1e0ffc7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e08161d60 con 0x7f1e08161980 2024-04-06T12:27:11.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.179+0000 7f1e0ffc7700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e0806c130 con 0x7f1e080cdaf0 2024-04-06T12:27:11.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.179+0000 7f1e0ffc7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0806b5b0 con 0x7f1e080c2a70 2024-04-06T12:27:11.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0dd63700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1e08161980 0x7f1e080cd3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37484/0 (socket says 172.21.15.189:37484) 2024-04-06T12:27:11.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0dd63700 1 -- 172.21.15.189:0/2692779535 learned_addr learned my addr 172.21.15.189:0/2692779535 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:11.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0d562700 1 --2- 172.21.15.189:0/2692779535 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e080c2a70 0x7f1e0806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:11.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0e564700 1 --2- 172.21.15.189:0/2692779535 >> v2:172.21.15.192:3300/0 conn(0x7f1e080cdaf0 0x7f1e0806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:11.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0cd61700 1 -- 172.21.15.189:0/2692779535 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743669788 0 0) 0x7f1e00001110 con 0x7f1e08161980 2024-04-06T12:27:11.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0cd61700 1 -- 172.21.15.189:0/2692779535 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1df40035b0 con 0x7f1e08161980 2024-04-06T12:27:11.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0cd61700 1 -- 172.21.15.189:0/2692779535 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1289944808 0 0) 0x7f1df40035b0 con 0x7f1e08161980 2024-04-06T12:27:11.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0cd61700 1 -- 172.21.15.189:0/2692779535 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e080c2a70 msgr2=0x7f1e0806d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:11.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0cd61700 1 --2- 172.21.15.189:0/2692779535 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e080c2a70 0x7f1e0806d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0cd61700 1 -- 172.21.15.189:0/2692779535 >> v2:172.21.15.192:3300/0 conn(0x7f1e080cdaf0 msgr2=0x7f1e0806d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:11.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0cd61700 1 --2- 172.21.15.189:0/2692779535 >> v2:172.21.15.192:3300/0 conn(0x7f1e080cdaf0 0x7f1e0806d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0cd61700 1 -- 172.21.15.189:0/2692779535 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0806b890 con 0x7f1e08161980 2024-04-06T12:27:11.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0cd61700 1 -- 172.21.15.189:0/2692779535 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f1e00001800 con 0x7f1e08161980 2024-04-06T12:27:11.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0cd61700 1 -- 172.21.15.189:0/2692779535 >> v1:172.21.15.189:6789/0 conn(0x7f1e08161980 legacy=0x7f1e080cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:11.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0cd61700 1 --2- 172.21.15.189:0/2692779535 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1df4003a00 0x7f1df4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:11.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0cd61700 1 --2- 172.21.15.189:0/2692779535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1df40043f0 0x7f1df40048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:11.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0cd61700 1 --2- 172.21.15.189:0/2692779535 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1df4004f40 0x7f1df400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:11.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0cd61700 1 -- 172.21.15.189:0/2692779535 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0806c130 con 0x7f1df40043f0 2024-04-06T12:27:11.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0cd61700 1 -- 172.21.15.189:0/2692779535 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0806b5b0 con 0x7f1df4004f40 2024-04-06T12:27:11.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0cd61700 1 -- 172.21.15.189:0/2692779535 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df400bf50 con 0x7f1df4003a00 2024-04-06T12:27:11.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0e564700 1 --2- 172.21.15.189:0/2692779535 >> v2:172.21.15.192:3300/0 conn(0x7f1e080cdaf0 0x7f1e0806d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:11.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0d562700 1 --2- 172.21.15.189:0/2692779535 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e080c2a70 0x7f1e0806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:11.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0d562700 1 --2- 172.21.15.189:0/2692779535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1df40043f0 0x7f1df40048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:11.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0dd63700 1 --2- 172.21.15.189:0/2692779535 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1df4003a00 0x7f1df4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:11.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.180+0000 7f1e0e564700 1 --2- 172.21.15.189:0/2692779535 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1df4004f40 0x7f1df400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:11.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.181+0000 7f1e0d562700 1 -- 172.21.15.189:0/2692779535 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1df4003a00 msgr2=0x7f1df4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:11.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.181+0000 7f1e0d562700 1 --2- 172.21.15.189:0/2692779535 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1df4003a00 0x7f1df4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.181+0000 7f1e0d562700 1 -- 172.21.15.189:0/2692779535 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1df4004f40 msgr2=0x7f1df400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:11.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.181+0000 7f1e0d562700 1 --2- 172.21.15.189:0/2692779535 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1df4004f40 0x7f1df400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.181+0000 7f1e0d562700 1 -- 172.21.15.189:0/2692779535 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df40035b0 con 0x7f1df40043f0 2024-04-06T12:27:11.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.181+0000 7f1e0d562700 1 --2- 172.21.15.189:0/2692779535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1df40043f0 0x7f1df40048d0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f1e080cf770 tx=0x7f1df4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5166992b080aa186 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:11.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.181+0000 7f1e0cd61700 1 -- 172.21.15.189:0/2692779535 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df8004030 con 0x7f1df40043f0 2024-04-06T12:27:11.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.181+0000 7f1e0cd61700 1 -- 172.21.15.189:0/2692779535 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df8004190 con 0x7f1df40043f0 2024-04-06T12:27:11.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.181+0000 7f1e0cd61700 1 -- 172.21.15.189:0/2692779535 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df8004540 con 0x7f1df40043f0 2024-04-06T12:27:11.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.181+0000 7f1e0ffc7700 1 -- 172.21.15.189:0/2692779535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1df40043f0 msgr2=0x7f1df40048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:11.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.181+0000 7f1e0ffc7700 1 --2- 172.21.15.189:0/2692779535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1df40043f0 0x7f1df40048d0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f1e080cf770 tx=0x7f1df4005880 comp rx=0 tx=0).stop 2024-04-06T12:27:11.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.181+0000 7f1e0e564700 1 -- 172.21.15.189:0/2692779535 reap_dead start 2024-04-06T12:27:11.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.181+0000 7f1e0ffc7700 1 -- 172.21.15.189:0/2692779535 shutdown_connections 2024-04-06T12:27:11.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.182+0000 7f1e0ffc7700 1 -- 172.21.15.189:0/2692779535 >> 172.21.15.189:0/2692779535 conn(0x7f1e080cc290 msgr2=0x7f1e080cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:11.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.182+0000 7f1e0ffc7700 1 -- 172.21.15.189:0/2692779535 shutdown_connections 2024-04-06T12:27:11.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.182+0000 7f1e0ffc7700 1 -- 172.21.15.189:0/2692779535 wait complete. 2024-04-06T12:27:11.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.183+0000 7f1e0ffc7700 1 Processor -- start 2024-04-06T12:27:11.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.183+0000 7f1e0ffc7700 1 -- start start 2024-04-06T12:27:11.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.184+0000 7f1e0ffc7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1df40043f0 0x7f1e081608e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:11.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.184+0000 7f1e0ffc7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e080c0730 0x7f1e080c5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:11.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.184+0000 7f1e0ffc7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e080bf0c0 0x7f1e080c1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:11.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.184+0000 7f1e0ffc7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e080c0310 con 0x7f1e080c0730 2024-04-06T12:27:11.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.184+0000 7f1e0ffc7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e080c6090 con 0x7f1df40043f0 2024-04-06T12:27:11.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.184+0000 7f1e0ffc7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e08160e20 con 0x7f1e080bf0c0 2024-04-06T12:27:11.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.184+0000 7f1e0d562700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e080c0730 0x7f1e080c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:11.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.184+0000 7f1e0d562700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e080c0730 0x7f1e080c5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37274/0 (socket says 172.21.15.189:37274) 2024-04-06T12:27:11.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.184+0000 7f1e0d562700 1 -- 172.21.15.189:0/3361654248 learned_addr learned my addr 172.21.15.189:0/3361654248 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:11.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.184+0000 7f1e0dd63700 1 --2- 172.21.15.189:0/3361654248 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1df40043f0 0x7f1e081608e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:11.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.184+0000 7f1e0d562700 1 -- 172.21.15.189:0/3361654248 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e080bf0c0 msgr2=0x7f1e080c1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:11.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.184+0000 7f1e0e564700 1 --2- 172.21.15.189:0/3361654248 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e080bf0c0 0x7f1e080c1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:11.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.184+0000 7f1e0d562700 1 --2- 172.21.15.189:0/3361654248 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e080bf0c0 0x7f1e080c1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.184+0000 7f1e0d562700 1 -- 172.21.15.189:0/3361654248 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1df40043f0 msgr2=0x7f1e081608e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:11.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.184+0000 7f1e0d562700 1 --2- 172.21.15.189:0/3361654248 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1df40043f0 0x7f1e081608e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.184+0000 7f1e0d562700 1 -- 172.21.15.189:0/3361654248 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e080bf9b0 con 0x7f1e080c0730 2024-04-06T12:27:11.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.184+0000 7f1e0dd63700 1 --2- 172.21.15.189:0/3361654248 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1df40043f0 0x7f1e081608e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:27:11.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.184+0000 7f1e0e564700 1 --2- 172.21.15.189:0/3361654248 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e080bf0c0 0x7f1e080c1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:27:11.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.185+0000 7f1e0d562700 1 --2- 172.21.15.189:0/3361654248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e080c0730 0x7f1e080c5810 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f1df4005730 tx=0x7f1df80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:11.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.185+0000 7f1dfe7fc700 1 -- 172.21.15.189:0/3361654248 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df8004030 con 0x7f1e080c0730 2024-04-06T12:27:11.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.185+0000 7f1dfe7fc700 1 -- 172.21.15.189:0/3361654248 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df8003ab0 con 0x7f1e080c0730 2024-04-06T12:27:11.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.185+0000 7f1e0ffc7700 1 -- 172.21.15.189:0/3361654248 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e080bfc50 con 0x7f1e080c0730 2024-04-06T12:27:11.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.185+0000 7f1dfe7fc700 1 -- 172.21.15.189:0/3361654248 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df8003e40 con 0x7f1e080c0730 2024-04-06T12:27:11.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.185+0000 7f1e0ffc7700 1 -- 172.21.15.189:0/3361654248 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e080c5d50 con 0x7f1e080c0730 2024-04-06T12:27:11.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.186+0000 7f1e0ffc7700 1 -- 172.21.15.189:0/3361654248 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e080af040 con 0x7f1e080c0730 2024-04-06T12:27:11.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.186+0000 7f1dfe7fc700 1 -- 172.21.15.189:0/3361654248 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 30) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f1df8013470 con 0x7f1e080c0730 2024-04-06T12:27:11.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.186+0000 7f1dfe7fc700 1 --2- 172.21.15.189:0/3361654248 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dec06aae0 0x7f1dec06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:11.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.187+0000 7f1dfe7fc700 1 -- 172.21.15.189:0/3361654248 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6803+0+0 (secure 0 0 0) 0x7f1df808ad10 con 0x7f1e080c0730 2024-04-06T12:27:11.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.187+0000 7f1e0dd63700 1 --2- 172.21.15.189:0/3361654248 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dec06aae0 0x7f1dec06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:11.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.187+0000 7f1e0dd63700 1 --2- 172.21.15.189:0/3361654248 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dec06aae0 0x7f1dec06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1e00001eb0 tx=0x7f1e00009040 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:11.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.191+0000 7f1dfe7fc700 1 -- 172.21.15.189:0/3361654248 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1df8055de0 con 0x7f1e080c0730 2024-04-06T12:27:11.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.401+0000 7f1e0ffc7700 1 -- 172.21.15.189:0/3361654248 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1e08074990 con 0x7f1e080c0730 2024-04-06T12:27:11.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.519+0000 7f1dfe7fc700 1 -- 172.21.15.189:0/3361654248 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v74) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1df8059400 con 0x7f1e080c0730 2024-04-06T12:27:11.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.522+0000 7f1e0ffc7700 1 -- 172.21.15.189:0/3361654248 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dec06aae0 msgr2=0x7f1dec06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:11.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.522+0000 7f1e0ffc7700 1 --2- 172.21.15.189:0/3361654248 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dec06aae0 0x7f1dec06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1e00001eb0 tx=0x7f1e00009040 comp rx=0 tx=0).stop 2024-04-06T12:27:11.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.522+0000 7f1e0ffc7700 1 -- 172.21.15.189:0/3361654248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e080c0730 msgr2=0x7f1e080c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:11.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.522+0000 7f1e0ffc7700 1 --2- 172.21.15.189:0/3361654248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e080c0730 0x7f1e080c5810 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f1df4005730 tx=0x7f1df80047c0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.523+0000 7f1e0ffc7700 1 -- 172.21.15.189:0/3361654248 shutdown_connections 2024-04-06T12:27:11.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.523+0000 7f1e0ffc7700 1 --2- 172.21.15.189:0/3361654248 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e080bf0c0 0x7f1e080c1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.523+0000 7f1e0ffc7700 1 --2- 172.21.15.189:0/3361654248 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dec06aae0 0x7f1dec06cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.523+0000 7f1e0ffc7700 1 --2- 172.21.15.189:0/3361654248 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1df40043f0 0x7f1e081608e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.523+0000 7f1e0ffc7700 1 --2- 172.21.15.189:0/3361654248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e080c0730 0x7f1e080c5810 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.523+0000 7f1e0ffc7700 1 -- 172.21.15.189:0/3361654248 >> 172.21.15.189:0/3361654248 conn(0x7f1e080cc290 msgr2=0x7f1e080b2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:11.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.523+0000 7f1e0ffc7700 1 -- 172.21.15.189:0/3361654248 shutdown_connections 2024-04-06T12:27:11.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.523+0000 7f1e0ffc7700 1 -- 172.21.15.189:0/3361654248 wait complete. 2024-04-06T12:27:11.522 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' removed 2024-04-06T12:27:11.533 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-06T12:27:11.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.665+0000 7f141f323700 1 Processor -- start 2024-04-06T12:27:11.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.665+0000 7f141f323700 1 -- start start 2024-04-06T12:27:11.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.666+0000 7f141f323700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f141815a550 0x7f14180cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:11.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.666+0000 7f141f323700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f14180cf900 0x7f14180b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:11.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.666+0000 7f141f323700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14180cdbe0 con 0x7f14180b4ba0 2024-04-06T12:27:11.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.666+0000 7f141f323700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14180b5010 con 0x7f141815a550 2024-04-06T12:27:11.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.666+0000 7f141f323700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f141815a940 con 0x7f14180cf900 2024-04-06T12:27:11.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.666+0000 7f141d8c0700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f14180b4ba0 0x7f141806eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37486/0 (socket says 172.21.15.189:37486) 2024-04-06T12:27:11.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.666+0000 7f141d8c0700 1 -- 172.21.15.189:0/916159263 learned_addr learned my addr 172.21.15.189:0/916159263 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:11.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.666+0000 7f141c8be700 1 --2- 172.21.15.189:0/916159263 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f14180cf900 0x7f14180b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:11.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.666+0000 7f141d0bf700 1 --2- 172.21.15.189:0/916159263 >> v2:172.21.15.192:3300/0 conn(0x7f141815a550 0x7f14180cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:11.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.666+0000 7f140ffff700 1 -- 172.21.15.189:0/916159263 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625685736 0 0) 0x7f14180cdbe0 con 0x7f14180b4ba0 2024-04-06T12:27:11.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.666+0000 7f140ffff700 1 -- 172.21.15.189:0/916159263 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14040035b0 con 0x7f14180b4ba0 2024-04-06T12:27:11.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.666+0000 7f141d0bf700 1 -- 172.21.15.189:0/916159263 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f14180cf900 msgr2=0x7f14180b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:11.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.666+0000 7f141d0bf700 1 --2- 172.21.15.189:0/916159263 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f14180cf900 0x7f14180b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.666+0000 7f141d0bf700 1 -- 172.21.15.189:0/916159263 >> v1:172.21.15.189:6789/0 conn(0x7f14180b4ba0 legacy=0x7f141806eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:11.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.666+0000 7f140ffff700 1 -- 172.21.15.189:0/916159263 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2027264899 0 0) 0x7f14040035b0 con 0x7f14180b4ba0 2024-04-06T12:27:11.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.666+0000 7f141d0bf700 1 -- 172.21.15.189:0/916159263 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f141815ac00 con 0x7f141815a550 2024-04-06T12:27:11.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.666+0000 7f141c8be700 1 --2- 172.21.15.189:0/916159263 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f14180cf900 0x7f14180b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:11.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.666+0000 7f141d0bf700 1 --2- 172.21.15.189:0/916159263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f141815a550 0x7f14180cf3c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1408009840 tx=0x7f1408009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20c417fc50f587c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:11.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.667+0000 7f140ffff700 1 -- 172.21.15.189:0/916159263 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14100019f0 con 0x7f141815a550 2024-04-06T12:27:11.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.667+0000 7f140ffff700 1 -- 172.21.15.189:0/916159263 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1408004030 con 0x7f141815a550 2024-04-06T12:27:11.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.667+0000 7f140ffff700 1 -- 172.21.15.189:0/916159263 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14080043c0 con 0x7f141815a550 2024-04-06T12:27:11.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.667+0000 7f141f323700 1 -- 172.21.15.189:0/916159263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f141815a550 msgr2=0x7f14180cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:11.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.667+0000 7f141f323700 1 --2- 172.21.15.189:0/916159263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f141815a550 0x7f14180cf3c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1408009840 tx=0x7f1408009b50 comp rx=0 tx=0).stop 2024-04-06T12:27:11.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.667+0000 7f141f323700 1 -- 172.21.15.189:0/916159263 shutdown_connections 2024-04-06T12:27:11.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.667+0000 7f141f323700 1 --2- 172.21.15.189:0/916159263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f141815a550 0x7f14180cf3c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.667+0000 7f141f323700 1 --2- 172.21.15.189:0/916159263 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f14180cf900 0x7f14180b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.667+0000 7f141f323700 1 -- 172.21.15.189:0/916159263 >> 172.21.15.189:0/916159263 conn(0x7f14180d00b0 msgr2=0x7f14180d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:11.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.667+0000 7f141f323700 1 -- 172.21.15.189:0/916159263 shutdown_connections 2024-04-06T12:27:11.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.667+0000 7f141f323700 1 -- 172.21.15.189:0/916159263 wait complete. 2024-04-06T12:27:11.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.668+0000 7f141f323700 1 Processor -- start 2024-04-06T12:27:11.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.668+0000 7f141f323700 1 -- start start 2024-04-06T12:27:11.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.669+0000 7f141f323700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14180b4ba0 0x7f1418150e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:11.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.669+0000 7f141f323700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14180cf900 0x7f141814b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:11.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.669+0000 7f141f323700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f14181471b0 0x7f141814b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:11.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.669+0000 7f141f323700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1418149300 con 0x7f14180b4ba0 2024-04-06T12:27:11.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.669+0000 7f141f323700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1418149530 con 0x7f14180cf900 2024-04-06T12:27:11.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.669+0000 7f141f323700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1418149810 con 0x7f14181471b0 2024-04-06T12:27:11.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.669+0000 7f141d0bf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14180b4ba0 0x7f1418150e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:11.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.669+0000 7f141d0bf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14180b4ba0 0x7f1418150e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37300/0 (socket says 172.21.15.189:37300) 2024-04-06T12:27:11.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.669+0000 7f141d0bf700 1 -- 172.21.15.189:0/2889383129 learned_addr learned my addr 172.21.15.189:0/2889383129 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:11.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.669+0000 7f141c8be700 1 --2- 172.21.15.189:0/2889383129 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14180cf900 0x7f141814b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:11.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.669+0000 7f141d0bf700 1 -- 172.21.15.189:0/2889383129 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f14181471b0 msgr2=0x7f141814b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:11.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.669+0000 7f141d8c0700 1 --2- 172.21.15.189:0/2889383129 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f14181471b0 0x7f141814b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:11.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.670+0000 7f141d0bf700 1 --2- 172.21.15.189:0/2889383129 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f14181471b0 0x7f141814b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.670+0000 7f141d0bf700 1 -- 172.21.15.189:0/2889383129 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14180cf900 msgr2=0x7f141814b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:11.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.670+0000 7f141d0bf700 1 --2- 172.21.15.189:0/2889383129 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14180cf900 0x7f141814b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:11.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.670+0000 7f141d0bf700 1 -- 172.21.15.189:0/2889383129 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14140094a0 con 0x7f14180b4ba0 2024-04-06T12:27:11.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.670+0000 7f141d0bf700 1 --2- 172.21.15.189:0/2889383129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14180b4ba0 0x7f1418150e80 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f140800a010 tx=0x7f1408004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:11.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.670+0000 7f140dffb700 1 -- 172.21.15.189:0/2889383129 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1408003e10 con 0x7f14180b4ba0 2024-04-06T12:27:11.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.670+0000 7f140dffb700 1 -- 172.21.15.189:0/2889383129 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140801f070 con 0x7f14180b4ba0 2024-04-06T12:27:11.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.670+0000 7f141f323700 1 -- 172.21.15.189:0/2889383129 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14080094a0 con 0x7f14180b4ba0 2024-04-06T12:27:11.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.670+0000 7f140dffb700 1 -- 172.21.15.189:0/2889383129 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1408011870 con 0x7f14180b4ba0 2024-04-06T12:27:11.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.670+0000 7f141f323700 1 -- 172.21.15.189:0/2889383129 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1418149dd0 con 0x7f14180b4ba0 2024-04-06T12:27:11.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.671+0000 7f141f323700 1 -- 172.21.15.189:0/2889383129 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1418067bb0 con 0x7f14180b4ba0 2024-04-06T12:27:11.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.671+0000 7f140dffb700 1 -- 172.21.15.189:0/2889383129 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 30) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f14080119d0 con 0x7f14180b4ba0 2024-04-06T12:27:11.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.672+0000 7f140dffb700 1 --2- 172.21.15.189:0/2889383129 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13fc06aae0 0x7f13fc06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:11.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.672+0000 7f140dffb700 1 -- 172.21.15.189:0/2889383129 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6365+0+0 (secure 0 0 0) 0x7f1408021070 con 0x7f14180b4ba0 2024-04-06T12:27:11.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.672+0000 7f141c8be700 1 --2- 172.21.15.189:0/2889383129 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13fc06aae0 0x7f13fc06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:11.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.672+0000 7f141c8be700 1 --2- 172.21.15.189:0/2889383129 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13fc06aae0 0x7f13fc06cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1414009d00 tx=0x7f1414008040 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:11.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.676+0000 7f140dffb700 1 -- 172.21.15.189:0/2889383129 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1408055e20 con 0x7f14180b4ba0 2024-04-06T12:27:11.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:11.871+0000 7f141f323700 1 -- 172.21.15.189:0/2889383129 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1418074990 con 0x7f14180b4ba0 2024-04-06T12:27:12.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.308+0000 7f140dffb700 1 -- 172.21.15.189:0/2889383129 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 31) v1 ==== 89191+0+0 (secure 0 0 0) 0x7f14080561f0 con 0x7f14180b4ba0 2024-04-06T12:27:12.473 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:27:12.476 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:27:12.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.589+0000 7f140dffb700 1 -- 172.21.15.189:0/2889383129 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v75) v1 ==== 169+0+0 (secure 0 0 0) 0x7f1408055e20 con 0x7f14180b4ba0 2024-04-06T12:27:12.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.591+0000 7f141f323700 1 -- 172.21.15.189:0/2889383129 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13fc06aae0 msgr2=0x7f13fc06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:12.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.591+0000 7f141f323700 1 --2- 172.21.15.189:0/2889383129 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13fc06aae0 0x7f13fc06cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1414009d00 tx=0x7f1414008040 comp rx=0 tx=0).stop 2024-04-06T12:27:12.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.591+0000 7f141f323700 1 -- 172.21.15.189:0/2889383129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14180b4ba0 msgr2=0x7f1418150e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:12.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.591+0000 7f141f323700 1 --2- 172.21.15.189:0/2889383129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14180b4ba0 0x7f1418150e80 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f140800a010 tx=0x7f1408004750 comp rx=0 tx=0).stop 2024-04-06T12:27:12.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.591+0000 7f141f323700 1 -- 172.21.15.189:0/2889383129 shutdown_connections 2024-04-06T12:27:12.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.591+0000 7f141f323700 1 --2- 172.21.15.189:0/2889383129 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f14181471b0 0x7f141814b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:12.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.591+0000 7f141f323700 1 --2- 172.21.15.189:0/2889383129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14180b4ba0 0x7f1418150e80 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:12.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.591+0000 7f141f323700 1 --2- 172.21.15.189:0/2889383129 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13fc06aae0 0x7f13fc06cfa0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:12.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.591+0000 7f141f323700 1 --2- 172.21.15.189:0/2889383129 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14180cf900 0x7f141814b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:12.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.591+0000 7f141f323700 1 -- 172.21.15.189:0/2889383129 >> 172.21.15.189:0/2889383129 conn(0x7f14180d00b0 msgr2=0x7f14180b31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:12.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.591+0000 7f141f323700 1 -- 172.21.15.189:0/2889383129 shutdown_connections 2024-04-06T12:27:12.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.591+0000 7f141f323700 1 -- 172.21.15.189:0/2889383129 wait complete. 2024-04-06T12:27:12.591 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' removed 2024-04-06T12:27:12.602 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-06T12:27:12.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.728+0000 7f474191a700 1 Processor -- start 2024-04-06T12:27:12.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.728+0000 7f474191a700 1 -- start start 2024-04-06T12:27:12.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.729+0000 7f474191a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f473c15a550 0x7f473c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:12.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.729+0000 7f474191a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f473c0cf900 0x7f473c0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:12.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.729+0000 7f474191a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f473c0cdbe0 con 0x7f473c0b4ba0 2024-04-06T12:27:12.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.729+0000 7f474191a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f473c15a940 con 0x7f473c0cf900 2024-04-06T12:27:12.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.729+0000 7f474191a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f473c15ac00 con 0x7f473c15a550 2024-04-06T12:27:12.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.729+0000 7f473a7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f473c0cf900 0x7f473c0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:12.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.729+0000 7f473a7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f473c0cf900 0x7f473c0b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37306/0 (socket says 172.21.15.189:37306) 2024-04-06T12:27:12.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.729+0000 7f473a7fc700 1 -- 172.21.15.189:0/1973631813 learned_addr learned my addr 172.21.15.189:0/1973631813 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:12.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.729+0000 7f473affd700 1 --2- 172.21.15.189:0/1973631813 >> v2:172.21.15.192:3300/0 conn(0x7f473c15a550 0x7f473c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:12.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.729+0000 7f473a7fc700 1 -- 172.21.15.189:0/1973631813 >> v2:172.21.15.192:3300/0 conn(0x7f473c15a550 msgr2=0x7f473c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:12.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.729+0000 7f473a7fc700 1 --2- 172.21.15.189:0/1973631813 >> v2:172.21.15.192:3300/0 conn(0x7f473c15a550 0x7f473c0cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:12.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.729+0000 7f473a7fc700 1 -- 172.21.15.189:0/1973631813 >> v1:172.21.15.192:6789/0 conn(0x7f473c0b4ba0 legacy=0x7f473c06eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:12.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.729+0000 7f473a7fc700 1 -- 172.21.15.189:0/1973631813 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473c15aee0 con 0x7f473c0cf900 2024-04-06T12:27:12.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.729+0000 7f473a7fc700 1 --2- 172.21.15.189:0/1973631813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f473c0cf900 0x7f473c0b4660 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f473c073470 tx=0x7f473000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7339788fb7504810 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:12.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.729+0000 7f4739ffb700 1 -- 172.21.15.189:0/1973631813 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473000dc70 con 0x7f473c0cf900 2024-04-06T12:27:12.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.729+0000 7f4739ffb700 1 -- 172.21.15.189:0/1973631813 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473000ddd0 con 0x7f473c0cf900 2024-04-06T12:27:12.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.729+0000 7f4739ffb700 1 -- 172.21.15.189:0/1973631813 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4730010690 con 0x7f473c0cf900 2024-04-06T12:27:12.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.730+0000 7f474191a700 1 -- 172.21.15.189:0/1973631813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f473c0cf900 msgr2=0x7f473c0b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:12.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.730+0000 7f474191a700 1 --2- 172.21.15.189:0/1973631813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f473c0cf900 0x7f473c0b4660 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f473c073470 tx=0x7f473000be70 comp rx=0 tx=0).stop 2024-04-06T12:27:12.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.730+0000 7f474191a700 1 -- 172.21.15.189:0/1973631813 shutdown_connections 2024-04-06T12:27:12.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.730+0000 7f474191a700 1 --2- 172.21.15.189:0/1973631813 >> v2:172.21.15.192:3300/0 conn(0x7f473c15a550 0x7f473c0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:12.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.730+0000 7f474191a700 1 --2- 172.21.15.189:0/1973631813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f473c0cf900 0x7f473c0b4660 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:12.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.730+0000 7f474191a700 1 -- 172.21.15.189:0/1973631813 >> 172.21.15.189:0/1973631813 conn(0x7f473c0d00b0 msgr2=0x7f473c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:12.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.730+0000 7f474191a700 1 -- 172.21.15.189:0/1973631813 shutdown_connections 2024-04-06T12:27:12.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.730+0000 7f474191a700 1 -- 172.21.15.189:0/1973631813 wait complete. 2024-04-06T12:27:12.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.731+0000 7f474191a700 1 Processor -- start 2024-04-06T12:27:12.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.731+0000 7f474191a700 1 -- start start 2024-04-06T12:27:12.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.732+0000 7f474191a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f473c0b4ba0 0x7f473c1f4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:12.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.732+0000 7f474191a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f473c0cf900 0x7f473c150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:12.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.732+0000 7f474191a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f473c153100 0x7f473c1f2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:12.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.732+0000 7f474191a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473c0ab3c0 con 0x7f473c0cf900 2024-04-06T12:27:12.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.732+0000 7f474191a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473c0a84b0 con 0x7f473c153100 2024-04-06T12:27:12.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.732+0000 7f474191a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f473c1f3810 con 0x7f473c0b4ba0 2024-04-06T12:27:12.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.732+0000 7f473a7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f473c0cf900 0x7f473c150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:12.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.732+0000 7f473a7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f473c0cf900 0x7f473c150560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37310/0 (socket says 172.21.15.189:37310) 2024-04-06T12:27:12.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.732+0000 7f473a7fc700 1 -- 172.21.15.189:0/2958712911 learned_addr learned my addr 172.21.15.189:0/2958712911 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:12.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.732+0000 7f473affd700 1 --2- 172.21.15.189:0/2958712911 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f473c0b4ba0 0x7f473c1f4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:12.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.732+0000 7f473b7fe700 1 --2- 172.21.15.189:0/2958712911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f473c153100 0x7f473c1f2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:12.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.732+0000 7f473a7fc700 1 -- 172.21.15.189:0/2958712911 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f473c0b4ba0 msgr2=0x7f473c1f4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:12.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.733+0000 7f473a7fc700 1 --2- 172.21.15.189:0/2958712911 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f473c0b4ba0 0x7f473c1f4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:12.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.733+0000 7f473a7fc700 1 -- 172.21.15.189:0/2958712911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f473c153100 msgr2=0x7f473c1f2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:12.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.733+0000 7f473a7fc700 1 --2- 172.21.15.189:0/2958712911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f473c153100 0x7f473c1f2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:12.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.733+0000 7f473a7fc700 1 -- 172.21.15.189:0/2958712911 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473000b7f0 con 0x7f473c0cf900 2024-04-06T12:27:12.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.733+0000 7f473affd700 1 --2- 172.21.15.189:0/2958712911 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f473c0b4ba0 0x7f473c1f4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:12.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.733+0000 7f473a7fc700 1 --2- 172.21.15.189:0/2958712911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f473c0cf900 0x7f473c150560 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f473000beb0 tx=0x7f4730004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:12.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.733+0000 7f4723fff700 1 -- 172.21.15.189:0/2958712911 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4730003c70 con 0x7f473c0cf900 2024-04-06T12:27:12.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.733+0000 7f4723fff700 1 -- 172.21.15.189:0/2958712911 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4730003df0 con 0x7f473c0cf900 2024-04-06T12:27:12.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.733+0000 7f4723fff700 1 -- 172.21.15.189:0/2958712911 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47300106b0 con 0x7f473c0cf900 2024-04-06T12:27:12.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.733+0000 7f474191a700 1 -- 172.21.15.189:0/2958712911 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f473c1f3af0 con 0x7f473c0cf900 2024-04-06T12:27:12.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.733+0000 7f474191a700 1 -- 172.21.15.189:0/2958712911 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f473c0a8750 con 0x7f473c0cf900 2024-04-06T12:27:12.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.734+0000 7f474191a700 1 -- 172.21.15.189:0/2958712911 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f473c067bb0 con 0x7f473c0cf900 2024-04-06T12:27:12.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.734+0000 7f4723fff700 1 -- 172.21.15.189:0/2958712911 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 31) v1 ==== 89191+0+0 (secure 0 0 0) 0x7f4730019400 con 0x7f473c0cf900 2024-04-06T12:27:12.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.735+0000 7f4723fff700 1 --2- 172.21.15.189:0/2958712911 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f472806ad20 0x7f472806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:12.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.735+0000 7f4723fff700 1 -- 172.21.15.189:0/2958712911 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f473008aca0 con 0x7f473c0cf900 2024-04-06T12:27:12.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.735+0000 7f473affd700 1 --2- 172.21.15.189:0/2958712911 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f472806ad20 0x7f472806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:12.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.735+0000 7f473affd700 1 --2- 172.21.15.189:0/2958712911 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f472806ad20 0x7f472806d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f472c009b90 tx=0x7f472c008040 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:12.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.739+0000 7f4723fff700 1 -- 172.21.15.189:0/2958712911 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4730055c70 con 0x7f473c0cf900 2024-04-06T12:27:12.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.945+0000 7f474191a700 1 -- 172.21.15.189:0/2958712911 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f473c074990 con 0x7f473c0cf900 2024-04-06T12:27:12.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.949+0000 7f4723fff700 1 -- 172.21.15.189:0/2958712911 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4730059290 con 0x7f473c0cf900 2024-04-06T12:27:12.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.951+0000 7f474191a700 1 -- 172.21.15.189:0/2958712911 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f472806ad20 msgr2=0x7f472806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:12.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.951+0000 7f474191a700 1 --2- 172.21.15.189:0/2958712911 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f472806ad20 0x7f472806d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f472c009b90 tx=0x7f472c008040 comp rx=0 tx=0).stop 2024-04-06T12:27:12.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.951+0000 7f474191a700 1 -- 172.21.15.189:0/2958712911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f473c0cf900 msgr2=0x7f473c150560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:12.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.951+0000 7f474191a700 1 --2- 172.21.15.189:0/2958712911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f473c0cf900 0x7f473c150560 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f473000beb0 tx=0x7f4730004750 comp rx=0 tx=0).stop 2024-04-06T12:27:12.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.951+0000 7f474191a700 1 -- 172.21.15.189:0/2958712911 shutdown_connections 2024-04-06T12:27:12.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.951+0000 7f474191a700 1 --2- 172.21.15.189:0/2958712911 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f472806ad20 0x7f472806d1e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:12.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.951+0000 7f474191a700 1 --2- 172.21.15.189:0/2958712911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f473c153100 0x7f473c1f2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:12.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.951+0000 7f474191a700 1 --2- 172.21.15.189:0/2958712911 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f473c0b4ba0 0x7f473c1f4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:12.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.951+0000 7f474191a700 1 --2- 172.21.15.189:0/2958712911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f473c0cf900 0x7f473c150560 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:12.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.951+0000 7f474191a700 1 -- 172.21.15.189:0/2958712911 >> 172.21.15.189:0/2958712911 conn(0x7f473c0d00b0 msgr2=0x7f473c06d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:12.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.951+0000 7f474191a700 1 -- 172.21.15.189:0/2958712911 shutdown_connections 2024-04-06T12:27:12.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:12.951+0000 7f474191a700 1 -- 172.21.15.189:0/2958712911 wait complete. 2024-04-06T12:27:12.962 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-06T12:27:13.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.087+0000 7efc4305c700 1 Processor -- start 2024-04-06T12:27:13.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.087+0000 7efc4305c700 1 -- start start 2024-04-06T12:27:13.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.088+0000 7efc4305c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efc3c15d7e0 0x7efc3c0d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.088+0000 7efc4305c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc3c0d1cc0 0x7efc3c0b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.088+0000 7efc4305c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc3c0cdf80 con 0x7efc3c0d0890 2024-04-06T12:27:13.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.088+0000 7efc4305c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc3c0d3230 con 0x7efc3c15d7e0 2024-04-06T12:27:13.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.088+0000 7efc4305c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c0d2da0 con 0x7efc3c0d1cc0 2024-04-06T12:27:13.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.088+0000 7efc3bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efc3c0d0890 0x7efc3c0d15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37490/0 (socket says 172.21.15.189:37490) 2024-04-06T12:27:13.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.088+0000 7efc3bfff700 1 -- 172.21.15.189:0/4064595521 learned_addr learned my addr 172.21.15.189:0/4064595521 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:13.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc40df8700 1 --2- 172.21.15.189:0/4064595521 >> v2:172.21.15.192:3300/0 conn(0x7efc3c15d7e0 0x7efc3c0d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc415f9700 1 --2- 172.21.15.189:0/4064595521 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc3c0d1cc0 0x7efc3c0b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3b7fe700 1 -- 172.21.15.189:0/4064595521 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762562751 0 0) 0x7efc3c0cdf80 con 0x7efc3c0d0890 2024-04-06T12:27:13.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3b7fe700 1 -- 172.21.15.189:0/4064595521 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc2c0035b0 con 0x7efc3c0d0890 2024-04-06T12:27:13.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3b7fe700 1 -- 172.21.15.189:0/4064595521 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 590834428 0 0) 0x7efc2c0035b0 con 0x7efc3c0d0890 2024-04-06T12:27:13.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3b7fe700 1 -- 172.21.15.189:0/4064595521 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc3c0d1cc0 msgr2=0x7efc3c0b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:13.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3b7fe700 1 --2- 172.21.15.189:0/4064595521 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc3c0d1cc0 0x7efc3c0b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3b7fe700 1 -- 172.21.15.189:0/4064595521 >> v2:172.21.15.192:3300/0 conn(0x7efc3c15d7e0 msgr2=0x7efc3c0d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:13.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3b7fe700 1 --2- 172.21.15.189:0/4064595521 >> v2:172.21.15.192:3300/0 conn(0x7efc3c15d7e0 0x7efc3c0d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3b7fe700 1 -- 172.21.15.189:0/4064595521 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3c0d0cf0 con 0x7efc3c0d0890 2024-04-06T12:27:13.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3b7fe700 1 -- 172.21.15.189:0/4064595521 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7efc28001670 con 0x7efc3c0d0890 2024-04-06T12:27:13.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3b7fe700 1 -- 172.21.15.189:0/4064595521 >> v1:172.21.15.189:6789/0 conn(0x7efc3c0d0890 legacy=0x7efc3c0d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:13.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3b7fe700 1 --2- 172.21.15.189:0/4064595521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc2c003a00 0x7efc2c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3b7fe700 1 --2- 172.21.15.189:0/4064595521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc2c004850 0x7efc2c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3b7fe700 1 --2- 172.21.15.189:0/4064595521 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc3c0d1cc0 0x7efc2c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3b7fe700 1 -- 172.21.15.189:0/4064595521 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c0d3230 con 0x7efc2c004850 2024-04-06T12:27:13.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3b7fe700 1 -- 172.21.15.189:0/4064595521 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c0d2da0 con 0x7efc2c003a00 2024-04-06T12:27:13.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3b7fe700 1 -- 172.21.15.189:0/4064595521 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc2c00ba10 con 0x7efc3c0d1cc0 2024-04-06T12:27:13.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3bfff700 1 --2- 172.21.15.189:0/4064595521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc2c004850 0x7efc2c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc40df8700 1 --2- 172.21.15.189:0/4064595521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc2c003a00 0x7efc2c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc415f9700 1 --2- 172.21.15.189:0/4064595521 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc3c0d1cc0 0x7efc2c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3bfff700 1 -- 172.21.15.189:0/4064595521 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc3c0d1cc0 msgr2=0x7efc2c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:13.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3bfff700 1 --2- 172.21.15.189:0/4064595521 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc3c0d1cc0 0x7efc2c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3bfff700 1 -- 172.21.15.189:0/4064595521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc2c003a00 msgr2=0x7efc2c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:13.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3bfff700 1 --2- 172.21.15.189:0/4064595521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc2c003a00 0x7efc2c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc3bfff700 1 -- 172.21.15.189:0/4064595521 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc2c0035b0 con 0x7efc2c004850 2024-04-06T12:27:13.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.089+0000 7efc415f9700 1 --2- 172.21.15.189:0/4064595521 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc3c0d1cc0 0x7efc2c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:13.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.090+0000 7efc3bfff700 1 --2- 172.21.15.189:0/4064595521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc2c004850 0x7efc2c004cd0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7efc2800c370 tx=0x7efc2c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3098ab8c2bec799 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:13.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.090+0000 7efc3b7fe700 1 -- 172.21.15.189:0/4064595521 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc28007c60 con 0x7efc2c004850 2024-04-06T12:27:13.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.090+0000 7efc3b7fe700 1 -- 172.21.15.189:0/4064595521 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc28005960 con 0x7efc2c004850 2024-04-06T12:27:13.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.090+0000 7efc3b7fe700 1 -- 172.21.15.189:0/4064595521 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc28005d10 con 0x7efc2c004850 2024-04-06T12:27:13.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.090+0000 7efc4305c700 1 -- 172.21.15.189:0/4064595521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc2c004850 msgr2=0x7efc2c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:13.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.090+0000 7efc4305c700 1 --2- 172.21.15.189:0/4064595521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc2c004850 0x7efc2c004cd0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7efc2800c370 tx=0x7efc2c005880 comp rx=0 tx=0).stop 2024-04-06T12:27:13.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.090+0000 7efc415f9700 1 -- 172.21.15.189:0/4064595521 reap_dead start 2024-04-06T12:27:13.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.090+0000 7efc4305c700 1 -- 172.21.15.189:0/4064595521 shutdown_connections 2024-04-06T12:27:13.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.090+0000 7efc4305c700 1 -- 172.21.15.189:0/4064595521 >> 172.21.15.189:0/4064595521 conn(0x7efc3c072250 msgr2=0x7efc3c072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:13.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.090+0000 7efc4305c700 1 -- 172.21.15.189:0/4064595521 shutdown_connections 2024-04-06T12:27:13.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.090+0000 7efc4305c700 1 -- 172.21.15.189:0/4064595521 wait complete. 2024-04-06T12:27:13.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.091+0000 7efc4305c700 1 Processor -- start 2024-04-06T12:27:13.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.091+0000 7efc4305c700 1 -- start start 2024-04-06T12:27:13.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.092+0000 7efc4305c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc2c004850 0x7efc3c14dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.092+0000 7efc4305c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc3c14e3c0 0x7efc3c0b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.092+0000 7efc4305c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc3c0b3250 0x7efc3c0b3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.092+0000 7efc4305c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c0cbba0 con 0x7efc3c0b3250 2024-04-06T12:27:13.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.092+0000 7efc4305c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c0caf90 con 0x7efc2c004850 2024-04-06T12:27:13.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.092+0000 7efc4305c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c0cb270 con 0x7efc3c14e3c0 2024-04-06T12:27:13.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.092+0000 7efc415f9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc3c0b3250 0x7efc3c0b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.093+0000 7efc415f9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc3c0b3250 0x7efc3c0b3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37322/0 (socket says 172.21.15.189:37322) 2024-04-06T12:27:13.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.093+0000 7efc415f9700 1 -- 172.21.15.189:0/2084944581 learned_addr learned my addr 172.21.15.189:0/2084944581 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:13.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.093+0000 7efc3bfff700 1 --2- 172.21.15.189:0/2084944581 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc3c14e3c0 0x7efc3c0b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.093+0000 7efc40df8700 1 --2- 172.21.15.189:0/2084944581 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc2c004850 0x7efc3c14dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.093+0000 7efc415f9700 1 -- 172.21.15.189:0/2084944581 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc3c14e3c0 msgr2=0x7efc3c0b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:13.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.093+0000 7efc415f9700 1 --2- 172.21.15.189:0/2084944581 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc3c14e3c0 0x7efc3c0b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.093+0000 7efc415f9700 1 -- 172.21.15.189:0/2084944581 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc2c004850 msgr2=0x7efc3c14dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:13.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.093+0000 7efc415f9700 1 --2- 172.21.15.189:0/2084944581 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc2c004850 0x7efc3c14dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.093+0000 7efc415f9700 1 -- 172.21.15.189:0/2084944581 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3c0cb550 con 0x7efc3c0b3250 2024-04-06T12:27:13.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.093+0000 7efc3bfff700 1 --2- 172.21.15.189:0/2084944581 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc3c14e3c0 0x7efc3c0b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:13.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.093+0000 7efc40df8700 1 --2- 172.21.15.189:0/2084944581 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc2c004850 0x7efc3c14dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:13.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.093+0000 7efc415f9700 1 --2- 172.21.15.189:0/2084944581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc3c0b3250 0x7efc3c0b3700 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7efc3c08c730 tx=0x7efc340104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:13.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.093+0000 7efc397fa700 1 -- 172.21.15.189:0/2084944581 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3401b070 con 0x7efc3c0b3250 2024-04-06T12:27:13.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.093+0000 7efc397fa700 1 -- 172.21.15.189:0/2084944581 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3400f040 con 0x7efc3c0b3250 2024-04-06T12:27:13.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.093+0000 7efc397fa700 1 -- 172.21.15.189:0/2084944581 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3400e050 con 0x7efc3c0b3250 2024-04-06T12:27:13.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.093+0000 7efc4305c700 1 -- 172.21.15.189:0/2084944581 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc3c0cb830 con 0x7efc3c0b3250 2024-04-06T12:27:13.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.093+0000 7efc4305c700 1 -- 172.21.15.189:0/2084944581 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc3c0cac40 con 0x7efc3c0b3250 2024-04-06T12:27:13.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.094+0000 7efc4305c700 1 -- 172.21.15.189:0/2084944581 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc3c074990 con 0x7efc3c0b3250 2024-04-06T12:27:13.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.095+0000 7efc397fa700 1 -- 172.21.15.189:0/2084944581 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 31) v1 ==== 89191+0+0 (secure 0 0 0) 0x7efc34003a20 con 0x7efc3c0b3250 2024-04-06T12:27:13.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.095+0000 7efc397fa700 1 --2- 172.21.15.189:0/2084944581 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efc2006ad20 0x7efc2006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.095+0000 7efc397fa700 1 -- 172.21.15.189:0/2084944581 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5865+0+0 (secure 0 0 0) 0x7efc34089760 con 0x7efc3c0b3250 2024-04-06T12:27:13.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.095+0000 7efc40df8700 1 --2- 172.21.15.189:0/2084944581 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efc2006ad20 0x7efc2006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.096+0000 7efc40df8700 1 --2- 172.21.15.189:0/2084944581 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efc2006ad20 0x7efc2006d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efc30009c60 tx=0x7efc30008040 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:13.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.100+0000 7efc397fa700 1 -- 172.21.15.189:0/2084944581 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efc340547a0 con 0x7efc3c0b3250 2024-04-06T12:27:13.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.294+0000 7efc4305c700 1 -- 172.21.15.189:0/2084944581 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7efc3c0cbce0 con 0x7efc3c0b3250 2024-04-06T12:27:13.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.298+0000 7efc397fa700 1 -- 172.21.15.189:0/2084944581 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v20) v1 ==== 115+0+0 (secure 0 0 0) 0x7efc34057dc0 con 0x7efc3c0b3250 2024-04-06T12:27:13.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.301+0000 7efc4305c700 1 -- 172.21.15.189:0/2084944581 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efc2006ad20 msgr2=0x7efc2006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:13.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.301+0000 7efc4305c700 1 --2- 172.21.15.189:0/2084944581 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efc2006ad20 0x7efc2006d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efc30009c60 tx=0x7efc30008040 comp rx=0 tx=0).stop 2024-04-06T12:27:13.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.301+0000 7efc4305c700 1 -- 172.21.15.189:0/2084944581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc3c0b3250 msgr2=0x7efc3c0b3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:13.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.301+0000 7efc4305c700 1 --2- 172.21.15.189:0/2084944581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc3c0b3250 0x7efc3c0b3700 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7efc3c08c730 tx=0x7efc340104a0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.301+0000 7efc4305c700 1 -- 172.21.15.189:0/2084944581 shutdown_connections 2024-04-06T12:27:13.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.301+0000 7efc4305c700 1 --2- 172.21.15.189:0/2084944581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc3c0b3250 0x7efc3c0b3700 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.301+0000 7efc4305c700 1 --2- 172.21.15.189:0/2084944581 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efc2006ad20 0x7efc2006d1e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.301+0000 7efc4305c700 1 --2- 172.21.15.189:0/2084944581 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc2c004850 0x7efc3c14dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.301+0000 7efc4305c700 1 --2- 172.21.15.189:0/2084944581 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc3c14e3c0 0x7efc3c0b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.301+0000 7efc4305c700 1 -- 172.21.15.189:0/2084944581 >> 172.21.15.189:0/2084944581 conn(0x7efc3c072250 msgr2=0x7efc3c0cde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:13.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.301+0000 7efc4305c700 1 -- 172.21.15.189:0/2084944581 shutdown_connections 2024-04-06T12:27:13.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.301+0000 7efc4305c700 1 -- 172.21.15.189:0/2084944581 wait complete. 2024-04-06T12:27:13.312 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-06T12:27:13.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.437+0000 7ff046f0e700 1 Processor -- start 2024-04-06T12:27:13.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.437+0000 7ff046f0e700 1 -- start start 2024-04-06T12:27:13.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.437+0000 7ff046f0e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff0400b4660 0x7ff0400b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.438+0000 7ff046f0e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0400cdbe0 con 0x7ff0400b2fc0 2024-04-06T12:27:13.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.438+0000 7ff046f0e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0400b5000 con 0x7ff04015a550 2024-04-06T12:27:13.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.438+0000 7ff046f0e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04015a940 con 0x7ff0400b4660 2024-04-06T12:27:13.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.438+0000 7ff0454ab700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff0400b2fc0 0x7ff0400b3430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37502/0 (socket says 172.21.15.189:37502) 2024-04-06T12:27:13.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.438+0000 7ff0454ab700 1 -- 172.21.15.189:0/4055826204 learned_addr learned my addr 172.21.15.189:0/4055826204 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:13.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.438+0000 7ff03ffff700 1 --2- 172.21.15.189:0/4055826204 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff0400b4660 0x7ff0400b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.438+0000 7ff03f7fe700 1 -- 172.21.15.189:0/4055826204 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372226720 0 0) 0x7ff0400cdbe0 con 0x7ff0400b2fc0 2024-04-06T12:27:13.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.438+0000 7ff03f7fe700 1 -- 172.21.15.189:0/4055826204 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0300035b0 con 0x7ff0400b2fc0 2024-04-06T12:27:13.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.438+0000 7ff03f7fe700 1 -- 172.21.15.189:0/4055826204 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622650468 0 0) 0x7ff0400b5000 con 0x7ff04015a550 2024-04-06T12:27:13.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.438+0000 7ff03f7fe700 1 -- 172.21.15.189:0/4055826204 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0400cdbe0 con 0x7ff04015a550 2024-04-06T12:27:13.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.438+0000 7ff03ffff700 1 -- 172.21.15.189:0/4055826204 >> v1:172.21.15.192:6789/0 conn(0x7ff04015a550 legacy=0x7ff0400cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:13.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.438+0000 7ff03f7fe700 1 -- 172.21.15.189:0/4055826204 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1967060338 0 0) 0x7ff0300035b0 con 0x7ff0400b2fc0 2024-04-06T12:27:13.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.438+0000 7ff03ffff700 1 -- 172.21.15.189:0/4055826204 >> v1:172.21.15.189:6789/0 conn(0x7ff0400b2fc0 legacy=0x7ff0400b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:13.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.438+0000 7ff03ffff700 1 -- 172.21.15.189:0/4055826204 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04015ac00 con 0x7ff0400b4660 2024-04-06T12:27:13.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.438+0000 7ff03ffff700 1 --2- 172.21.15.189:0/4055826204 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff0400b4660 0x7ff0400b4ac0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff02c009870 tx=0x7ff02c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53e81af986e6d549 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:13.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.439+0000 7ff03f7fe700 1 -- 172.21.15.189:0/4055826204 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0380019f0 con 0x7ff0400b4660 2024-04-06T12:27:13.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.439+0000 7ff03f7fe700 1 -- 172.21.15.189:0/4055826204 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff02c004030 con 0x7ff0400b4660 2024-04-06T12:27:13.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.439+0000 7ff03f7fe700 1 -- 172.21.15.189:0/4055826204 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c0043c0 con 0x7ff0400b4660 2024-04-06T12:27:13.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.439+0000 7ff046f0e700 1 -- 172.21.15.189:0/4055826204 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff0400b4660 msgr2=0x7ff0400b4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:13.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.439+0000 7ff046f0e700 1 --2- 172.21.15.189:0/4055826204 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff0400b4660 0x7ff0400b4ac0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff02c009870 tx=0x7ff02c009b80 comp rx=0 tx=0).stop 2024-04-06T12:27:13.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.439+0000 7ff046f0e700 1 -- 172.21.15.189:0/4055826204 shutdown_connections 2024-04-06T12:27:13.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.439+0000 7ff046f0e700 1 --2- 172.21.15.189:0/4055826204 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff0400b4660 0x7ff0400b4ac0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.439+0000 7ff046f0e700 1 -- 172.21.15.189:0/4055826204 >> 172.21.15.189:0/4055826204 conn(0x7ff0400d00b0 msgr2=0x7ff0400d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:13.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.439+0000 7ff046f0e700 1 -- 172.21.15.189:0/4055826204 shutdown_connections 2024-04-06T12:27:13.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.439+0000 7ff046f0e700 1 -- 172.21.15.189:0/4055826204 wait complete. 2024-04-06T12:27:13.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.440+0000 7ff046f0e700 1 Processor -- start 2024-04-06T12:27:13.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.440+0000 7ff046f0e700 1 -- start start 2024-04-06T12:27:13.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.441+0000 7ff046f0e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0400b2fc0 0x7ff0400aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.441+0000 7ff046f0e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff0400b4660 0x7ff040145240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.441+0000 7ff046f0e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff040145780 0x7ff0400b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.441+0000 7ff046f0e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0400a85e0 con 0x7ff0400b2fc0 2024-04-06T12:27:13.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.441+0000 7ff046f0e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0400a88c0 con 0x7ff0400b4660 2024-04-06T12:27:13.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.441+0000 7ff046f0e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0400a8ba0 con 0x7ff040145780 2024-04-06T12:27:13.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.441+0000 7ff044caa700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0400b2fc0 0x7ff0400aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.441+0000 7ff044caa700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0400b2fc0 0x7ff0400aec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37330/0 (socket says 172.21.15.189:37330) 2024-04-06T12:27:13.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.441+0000 7ff044caa700 1 -- 172.21.15.189:0/3330303495 learned_addr learned my addr 172.21.15.189:0/3330303495 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:13.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.441+0000 7ff0454ab700 1 --2- 172.21.15.189:0/3330303495 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff040145780 0x7ff0400b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.442+0000 7ff03ffff700 1 --2- 172.21.15.189:0/3330303495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff0400b4660 0x7ff040145240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.442+0000 7ff044caa700 1 -- 172.21.15.189:0/3330303495 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff040145780 msgr2=0x7ff0400b02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:13.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.442+0000 7ff044caa700 1 --2- 172.21.15.189:0/3330303495 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff040145780 0x7ff0400b02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.442+0000 7ff044caa700 1 -- 172.21.15.189:0/3330303495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff0400b4660 msgr2=0x7ff040145240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:13.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.442+0000 7ff044caa700 1 --2- 172.21.15.189:0/3330303495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff0400b4660 0x7ff040145240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.442+0000 7ff044caa700 1 -- 172.21.15.189:0/3330303495 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff02c0094a0 con 0x7ff0400b2fc0 2024-04-06T12:27:13.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.442+0000 7ff0454ab700 1 --2- 172.21.15.189:0/3330303495 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff040145780 0x7ff0400b02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:13.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.442+0000 7ff044caa700 1 --2- 172.21.15.189:0/3330303495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0400b2fc0 0x7ff0400aec60 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7ff03400c500 tx=0x7ff03400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:13.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.442+0000 7ff03d7fa700 1 -- 172.21.15.189:0/3330303495 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0340052d0 con 0x7ff0400b2fc0 2024-04-06T12:27:13.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.442+0000 7ff03d7fa700 1 -- 172.21.15.189:0/3330303495 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff034005450 con 0x7ff0400b2fc0 2024-04-06T12:27:13.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.442+0000 7ff03d7fa700 1 -- 172.21.15.189:0/3330303495 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0340038c0 con 0x7ff0400b2fc0 2024-04-06T12:27:13.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.442+0000 7ff046f0e700 1 -- 172.21.15.189:0/3330303495 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0400a8e80 con 0x7ff0400b2fc0 2024-04-06T12:27:13.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.442+0000 7ff046f0e700 1 -- 172.21.15.189:0/3330303495 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0400af1a0 con 0x7ff0400b2fc0 2024-04-06T12:27:13.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.443+0000 7ff046f0e700 1 -- 172.21.15.189:0/3330303495 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0400a93a0 con 0x7ff0400b2fc0 2024-04-06T12:27:13.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.443+0000 7ff03d7fa700 1 -- 172.21.15.189:0/3330303495 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 31) v1 ==== 89191+0+0 (secure 0 0 0) 0x7ff034003a20 con 0x7ff0400b2fc0 2024-04-06T12:27:13.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.444+0000 7ff03d7fa700 1 --2- 172.21.15.189:0/3330303495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff02406ad50 0x7ff02406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.444+0000 7ff03d7fa700 1 -- 172.21.15.189:0/3330303495 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5865+0+0 (secure 0 0 0) 0x7ff0340887e0 con 0x7ff0400b2fc0 2024-04-06T12:27:13.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.444+0000 7ff03ffff700 1 --2- 172.21.15.189:0/3330303495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff02406ad50 0x7ff02406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.444+0000 7ff03ffff700 1 --2- 172.21.15.189:0/3330303495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff02406ad50 0x7ff02406d210 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0090b0 tx=0x7ff02c008030 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:13.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.448+0000 7ff03d7fa700 1 -- 172.21.15.189:0/3330303495 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff0400a93a0 con 0x7ff0400b2fc0 2024-04-06T12:27:13.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.642+0000 7ff046f0e700 1 -- 172.21.15.189:0/3330303495 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7ff040067bb0 con 0x7ff0400b2fc0 2024-04-06T12:27:13.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.645+0000 7ff03d7fa700 1 -- 172.21.15.189:0/3330303495 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v21) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff0340539c0 con 0x7ff0400b2fc0 2024-04-06T12:27:13.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.648+0000 7ff046f0e700 1 -- 172.21.15.189:0/3330303495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff02406ad50 msgr2=0x7ff02406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:13.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.648+0000 7ff046f0e700 1 --2- 172.21.15.189:0/3330303495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff02406ad50 0x7ff02406d210 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0090b0 tx=0x7ff02c008030 comp rx=0 tx=0).stop 2024-04-06T12:27:13.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.648+0000 7ff046f0e700 1 -- 172.21.15.189:0/3330303495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0400b2fc0 msgr2=0x7ff0400aec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:13.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.648+0000 7ff046f0e700 1 --2- 172.21.15.189:0/3330303495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0400b2fc0 0x7ff0400aec60 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7ff03400c500 tx=0x7ff03400c8c0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.648+0000 7ff046f0e700 1 -- 172.21.15.189:0/3330303495 shutdown_connections 2024-04-06T12:27:13.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.648+0000 7ff046f0e700 1 --2- 172.21.15.189:0/3330303495 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff040145780 0x7ff0400b02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.648+0000 7ff046f0e700 1 --2- 172.21.15.189:0/3330303495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0400b2fc0 0x7ff0400aec60 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.648+0000 7ff046f0e700 1 --2- 172.21.15.189:0/3330303495 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff02406ad50 0x7ff02406d210 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.648+0000 7ff046f0e700 1 --2- 172.21.15.189:0/3330303495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff0400b4660 0x7ff040145240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.648+0000 7ff046f0e700 1 -- 172.21.15.189:0/3330303495 >> 172.21.15.189:0/3330303495 conn(0x7ff0400d00b0 msgr2=0x7ff04006d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:13.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.648+0000 7ff046f0e700 1 -- 172.21.15.189:0/3330303495 shutdown_connections 2024-04-06T12:27:13.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.648+0000 7ff046f0e700 1 -- 172.21.15.189:0/3330303495 wait complete. 2024-04-06T12:27:13.658 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2024-04-06T12:27:13.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.781+0000 7fdfb98c9700 1 Processor -- start 2024-04-06T12:27:13.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.782+0000 7fdfb98c9700 1 -- start start 2024-04-06T12:27:13.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.782+0000 7fdfb98c9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdfb40b4660 0x7fdfb40b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.782+0000 7fdfb98c9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdfb40b2fc0 0x7fdfb40b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.782+0000 7fdfb98c9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb40b5030 con 0x7fdfb415a550 2024-04-06T12:27:13.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.782+0000 7fdfb98c9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb40cdbe0 con 0x7fdfb40b4660 2024-04-06T12:27:13.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.782+0000 7fdfb98c9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb415a940 con 0x7fdfb40b2fc0 2024-04-06T12:27:13.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.782+0000 7fdfb27fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdfb40b4660 0x7fdfb40b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.782+0000 7fdfb27fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdfb40b4660 0x7fdfb40b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37334/0 (socket says 172.21.15.189:37334) 2024-04-06T12:27:13.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.782+0000 7fdfb27fc700 1 -- 172.21.15.189:0/3356910735 learned_addr learned my addr 172.21.15.189:0/3356910735 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:13.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.783+0000 7fdfb37fe700 1 --2- 172.21.15.189:0/3356910735 >> v2:172.21.15.192:3300/0 conn(0x7fdfb40b2fc0 0x7fdfb40b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.783+0000 7fdfb27fc700 1 -- 172.21.15.189:0/3356910735 >> v2:172.21.15.192:3300/0 conn(0x7fdfb40b2fc0 msgr2=0x7fdfb40b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:13.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.783+0000 7fdfb27fc700 1 --2- 172.21.15.189:0/3356910735 >> v2:172.21.15.192:3300/0 conn(0x7fdfb40b2fc0 0x7fdfb40b33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.783+0000 7fdfb27fc700 1 -- 172.21.15.189:0/3356910735 >> v1:172.21.15.192:6789/0 conn(0x7fdfb415a550 legacy=0x7fdfb40cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:13.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.783+0000 7fdfb27fc700 1 -- 172.21.15.189:0/3356910735 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb415ac00 con 0x7fdfb40b4660 2024-04-06T12:27:13.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.783+0000 7fdfb27fc700 1 --2- 172.21.15.189:0/3356910735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb40b4660 0x7fdfb40b4af0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4072830 tx=0x7fdfa400be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=472047be140c467a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:13.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.783+0000 7fdfb1ffb700 1 -- 172.21.15.189:0/3356910735 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa400dcf0 con 0x7fdfb40b4660 2024-04-06T12:27:13.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.783+0000 7fdfb1ffb700 1 -- 172.21.15.189:0/3356910735 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa400de50 con 0x7fdfb40b4660 2024-04-06T12:27:13.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.783+0000 7fdfb1ffb700 1 -- 172.21.15.189:0/3356910735 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4004250 con 0x7fdfb40b4660 2024-04-06T12:27:13.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.784+0000 7fdfb98c9700 1 -- 172.21.15.189:0/3356910735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb40b4660 msgr2=0x7fdfb40b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:13.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.784+0000 7fdfb98c9700 1 --2- 172.21.15.189:0/3356910735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb40b4660 0x7fdfb40b4af0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4072830 tx=0x7fdfa400be70 comp rx=0 tx=0).stop 2024-04-06T12:27:13.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.784+0000 7fdfb98c9700 1 -- 172.21.15.189:0/3356910735 shutdown_connections 2024-04-06T12:27:13.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.784+0000 7fdfb98c9700 1 --2- 172.21.15.189:0/3356910735 >> v2:172.21.15.192:3300/0 conn(0x7fdfb40b2fc0 0x7fdfb40b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.784+0000 7fdfb98c9700 1 --2- 172.21.15.189:0/3356910735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb40b4660 0x7fdfb40b4af0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.784+0000 7fdfb98c9700 1 -- 172.21.15.189:0/3356910735 >> 172.21.15.189:0/3356910735 conn(0x7fdfb40d00b0 msgr2=0x7fdfb40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:13.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.784+0000 7fdfb98c9700 1 -- 172.21.15.189:0/3356910735 shutdown_connections 2024-04-06T12:27:13.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.784+0000 7fdfb98c9700 1 -- 172.21.15.189:0/3356910735 wait complete. 2024-04-06T12:27:13.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.785+0000 7fdfb98c9700 1 Processor -- start 2024-04-06T12:27:13.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.785+0000 7fdfb98c9700 1 -- start start 2024-04-06T12:27:13.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdfb98c9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfb40b2fc0 0x7fdfb41f36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdfb98c9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb40b4660 0x7fdfb41f00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdfb98c9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfb415b8c0 0x7fdfb4153ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdfb98c9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb41457b0 con 0x7fdfb40b4660 2024-04-06T12:27:13.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdfb98c9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb4156f30 con 0x7fdfb40b2fc0 2024-04-06T12:27:13.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdfb98c9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb4155900 con 0x7fdfb415b8c0 2024-04-06T12:27:13.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdfb27fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb40b4660 0x7fdfb41f00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdfb27fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb40b4660 0x7fdfb41f00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37336/0 (socket says 172.21.15.189:37336) 2024-04-06T12:27:13.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdfb27fc700 1 -- 172.21.15.189:0/2430946408 learned_addr learned my addr 172.21.15.189:0/2430946408 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:13.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdfb2ffd700 1 --2- 172.21.15.189:0/2430946408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfb40b2fc0 0x7fdfb41f36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdfb37fe700 1 --2- 172.21.15.189:0/2430946408 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfb415b8c0 0x7fdfb4153ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdfb27fc700 1 -- 172.21.15.189:0/2430946408 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfb415b8c0 msgr2=0x7fdfb4153ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:13.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdfb27fc700 1 --2- 172.21.15.189:0/2430946408 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfb415b8c0 0x7fdfb4153ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdfb27fc700 1 -- 172.21.15.189:0/2430946408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfb40b2fc0 msgr2=0x7fdfb41f36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:13.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdfb27fc700 1 --2- 172.21.15.189:0/2430946408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfb40b2fc0 0x7fdfb41f36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:13.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdfb27fc700 1 -- 172.21.15.189:0/2430946408 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa400b7f0 con 0x7fdfb40b4660 2024-04-06T12:27:13.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdfb27fc700 1 --2- 172.21.15.189:0/2430946408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb40b4660 0x7fdfb41f00d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fdfa400beb0 tx=0x7fdfa4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:13.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdf9bfff700 1 -- 172.21.15.189:0/2430946408 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4004230 con 0x7fdfb40b4660 2024-04-06T12:27:13.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdf9bfff700 1 -- 172.21.15.189:0/2430946408 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa4010470 con 0x7fdfb40b4660 2024-04-06T12:27:13.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdfb98c9700 1 -- 172.21.15.189:0/2430946408 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb41f3bf0 con 0x7fdfb40b4660 2024-04-06T12:27:13.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.786+0000 7fdf9bfff700 1 -- 172.21.15.189:0/2430946408 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4010800 con 0x7fdfb40b4660 2024-04-06T12:27:13.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.787+0000 7fdfb98c9700 1 -- 172.21.15.189:0/2430946408 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb415c060 con 0x7fdfb40b4660 2024-04-06T12:27:13.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.787+0000 7fdfb98c9700 1 -- 172.21.15.189:0/2430946408 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb41570f0 con 0x7fdfb40b4660 2024-04-06T12:27:13.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.788+0000 7fdf9bfff700 1 -- 172.21.15.189:0/2430946408 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 31) v1 ==== 89191+0+0 (secure 0 0 0) 0x7fdfa40193f0 con 0x7fdfb40b4660 2024-04-06T12:27:13.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.788+0000 7fdf9bfff700 1 --2- 172.21.15.189:0/2430946408 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdfa006ad20 0x7fdfa006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:13.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.788+0000 7fdfb2ffd700 1 --2- 172.21.15.189:0/2430946408 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdfa006ad20 0x7fdfa006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:13.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.788+0000 7fdf9bfff700 1 -- 172.21.15.189:0/2430946408 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5865+0+0 (secure 0 0 0) 0x7fdfa408b360 con 0x7fdfb40b4660 2024-04-06T12:27:13.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.789+0000 7fdfb2ffd700 1 --2- 172.21.15.189:0/2430946408 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdfa006ad20 0x7fdfa006d1e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdfb415d2f0 tx=0x7fdfa8007ec0 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:13.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:13.793+0000 7fdf9bfff700 1 -- 172.21.15.189:0/2430946408 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdfb41570f0 con 0x7fdfb40b4660 2024-04-06T12:27:14.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.067+0000 7fdfb98c9700 1 -- 172.21.15.189:0/2430946408 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7fdfb41570f0 con 0x7fdfb40b4660 2024-04-06T12:27:14.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.306+0000 7fdf9bfff700 1 -- 172.21.15.189:0/2430946408 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fdfa40600d0 con 0x7fdfb40b4660 2024-04-06T12:27:14.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.597+0000 7fdf9bfff700 1 -- 172.21.15.189:0/2430946408 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v551) v1 ==== 154+0+0 (secure 0 0 0) 0x7fdfa4044b80 con 0x7fdfb40b4660 2024-04-06T12:27:14.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.599+0000 7fdfb98c9700 1 -- 172.21.15.189:0/2430946408 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdfa006ad20 msgr2=0x7fdfa006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:14.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.599+0000 7fdfb98c9700 1 --2- 172.21.15.189:0/2430946408 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdfa006ad20 0x7fdfa006d1e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdfb415d2f0 tx=0x7fdfa8007ec0 comp rx=0 tx=0).stop 2024-04-06T12:27:14.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.599+0000 7fdfb98c9700 1 -- 172.21.15.189:0/2430946408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb40b4660 msgr2=0x7fdfb41f00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:14.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.599+0000 7fdfb98c9700 1 --2- 172.21.15.189:0/2430946408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb40b4660 0x7fdfb41f00d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fdfa400beb0 tx=0x7fdfa4004750 comp rx=0 tx=0).stop 2024-04-06T12:27:14.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.599+0000 7fdfb98c9700 1 -- 172.21.15.189:0/2430946408 shutdown_connections 2024-04-06T12:27:14.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.599+0000 7fdfb98c9700 1 --2- 172.21.15.189:0/2430946408 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfb415b8c0 0x7fdfb4153ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:14.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.599+0000 7fdfb98c9700 1 --2- 172.21.15.189:0/2430946408 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdfa006ad20 0x7fdfa006d1e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:14.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.599+0000 7fdfb98c9700 1 --2- 172.21.15.189:0/2430946408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfb40b2fc0 0x7fdfb41f36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:14.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.599+0000 7fdfb98c9700 1 --2- 172.21.15.189:0/2430946408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb40b4660 0x7fdfb41f00d0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:14.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.599+0000 7fdfb98c9700 1 -- 172.21.15.189:0/2430946408 >> 172.21.15.189:0/2430946408 conn(0x7fdfb40d00b0 msgr2=0x7fdfb406d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:14.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.599+0000 7fdfb98c9700 1 -- 172.21.15.189:0/2430946408 shutdown_connections 2024-04-06T12:27:14.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.599+0000 7fdfb98c9700 1 -- 172.21.15.189:0/2430946408 wait complete. 2024-04-06T12:27:14.610 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2024-04-06T12:27:14.610 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2024-04-06T12:27:14.610 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-06T12:27:14.610 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2024-04-06T12:27:14.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.735+0000 7fbb4d43b700 1 Processor -- start 2024-04-06T12:27:14.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.735+0000 7fbb4d43b700 1 -- start start 2024-04-06T12:27:14.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.735+0000 7fbb4d43b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbb480ce730 0x7fbb4815dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.736+0000 7fbb4d43b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbb480b8d70 0x7fbb480d1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.736+0000 7fbb4d43b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb4815eac0 con 0x7fbb4815d6f0 2024-04-06T12:27:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.736+0000 7fbb4d43b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb4815e010 con 0x7fbb480ce730 2024-04-06T12:27:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.736+0000 7fbb4d43b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb4815e220 con 0x7fbb480b8d70 2024-04-06T12:27:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.736+0000 7fbb467fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbb480ce730 0x7fbb4815dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.736+0000 7fbb467fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbb480ce730 0x7fbb4815dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37338/0 (socket says 172.21.15.189:37338) 2024-04-06T12:27:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.736+0000 7fbb467fc700 1 -- 172.21.15.189:0/4029767665 learned_addr learned my addr 172.21.15.189:0/4029767665 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.736+0000 7fbb477fe700 1 --2- 172.21.15.189:0/4029767665 >> v2:172.21.15.192:3300/0 conn(0x7fbb480b8d70 0x7fbb480d1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.736+0000 7fbb467fc700 1 -- 172.21.15.189:0/4029767665 >> v2:172.21.15.192:3300/0 conn(0x7fbb480b8d70 msgr2=0x7fbb480d1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.736+0000 7fbb467fc700 1 --2- 172.21.15.189:0/4029767665 >> v2:172.21.15.192:3300/0 conn(0x7fbb480b8d70 0x7fbb480d1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.736+0000 7fbb467fc700 1 -- 172.21.15.189:0/4029767665 >> v1:172.21.15.192:6789/0 conn(0x7fbb4815d6f0 legacy=0x7fbb480ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.736+0000 7fbb467fc700 1 -- 172.21.15.189:0/4029767665 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4815e500 con 0x7fbb480ce730 2024-04-06T12:27:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.736+0000 7fbb467fc700 1 --2- 172.21.15.189:0/4029767665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb480ce730 0x7fbb4815dad0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fbb4806c180 tx=0x7fbb3c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b915d142aa669169 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.736+0000 7fbb45ffb700 1 -- 172.21.15.189:0/4029767665 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c00b870 con 0x7fbb480ce730 2024-04-06T12:27:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.736+0000 7fbb45ffb700 1 -- 172.21.15.189:0/4029767665 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3c00b9d0 con 0x7fbb480ce730 2024-04-06T12:27:14.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.736+0000 7fbb45ffb700 1 -- 172.21.15.189:0/4029767665 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c00bd60 con 0x7fbb480ce730 2024-04-06T12:27:14.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.737+0000 7fbb4d43b700 1 -- 172.21.15.189:0/4029767665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb480ce730 msgr2=0x7fbb4815dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:14.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.737+0000 7fbb4d43b700 1 --2- 172.21.15.189:0/4029767665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb480ce730 0x7fbb4815dad0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fbb4806c180 tx=0x7fbb3c009bd0 comp rx=0 tx=0).stop 2024-04-06T12:27:14.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.737+0000 7fbb4d43b700 1 -- 172.21.15.189:0/4029767665 shutdown_connections 2024-04-06T12:27:14.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.737+0000 7fbb4d43b700 1 --2- 172.21.15.189:0/4029767665 >> v2:172.21.15.192:3300/0 conn(0x7fbb480b8d70 0x7fbb480d1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:14.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.737+0000 7fbb4d43b700 1 --2- 172.21.15.189:0/4029767665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb480ce730 0x7fbb4815dad0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:14.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.737+0000 7fbb4d43b700 1 -- 172.21.15.189:0/4029767665 >> 172.21.15.189:0/4029767665 conn(0x7fbb48072490 msgr2=0x7fbb4806b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:14.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.737+0000 7fbb4d43b700 1 -- 172.21.15.189:0/4029767665 shutdown_connections 2024-04-06T12:27:14.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.737+0000 7fbb4d43b700 1 -- 172.21.15.189:0/4029767665 wait complete. 2024-04-06T12:27:14.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.739+0000 7fbb4d43b700 1 Processor -- start 2024-04-06T12:27:14.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.739+0000 7fbb4d43b700 1 -- start start 2024-04-06T12:27:14.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.739+0000 7fbb4d43b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb480b8d70 0x7fbb480c7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:14.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.739+0000 7fbb4d43b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb480ce730 0x7fbb480cc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:14.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb4d43b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb4815d6f0 0x7fbb480b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:14.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb4d43b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb480c8900 con 0x7fbb480b8d70 2024-04-06T12:27:14.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb4d43b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb480cb500 con 0x7fbb480ce730 2024-04-06T12:27:14.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb4d43b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb480caf40 con 0x7fbb4815d6f0 2024-04-06T12:27:14.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb46ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb480b8d70 0x7fbb480c7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:14.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb46ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb480b8d70 0x7fbb480c7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37348/0 (socket says 172.21.15.189:37348) 2024-04-06T12:27:14.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb46ffd700 1 -- 172.21.15.189:0/2850566580 learned_addr learned my addr 172.21.15.189:0/2850566580 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:14.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb467fc700 1 --2- 172.21.15.189:0/2850566580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb480ce730 0x7fbb480cc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:14.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb477fe700 1 --2- 172.21.15.189:0/2850566580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb4815d6f0 0x7fbb480b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:14.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb46ffd700 1 -- 172.21.15.189:0/2850566580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb4815d6f0 msgr2=0x7fbb480b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:14.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb46ffd700 1 --2- 172.21.15.189:0/2850566580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb4815d6f0 0x7fbb480b4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:14.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb46ffd700 1 -- 172.21.15.189:0/2850566580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb480ce730 msgr2=0x7fbb480cc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:14.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb46ffd700 1 --2- 172.21.15.189:0/2850566580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb480ce730 0x7fbb480cc9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:14.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb46ffd700 1 -- 172.21.15.189:0/2850566580 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3c009550 con 0x7fbb480b8d70 2024-04-06T12:27:14.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb467fc700 1 --2- 172.21.15.189:0/2850566580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb480ce730 0x7fbb480cc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:14.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb46ffd700 1 --2- 172.21.15.189:0/2850566580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb480b8d70 0x7fbb480c7920 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fbb3800c450 tx=0x7fbb3800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:14.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb2f7fe700 1 -- 172.21.15.189:0/2850566580 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb380052d0 con 0x7fbb480b8d70 2024-04-06T12:27:14.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb2f7fe700 1 -- 172.21.15.189:0/2850566580 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb38005430 con 0x7fbb480b8d70 2024-04-06T12:27:14.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb4d43b700 1 -- 172.21.15.189:0/2850566580 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb480cb220 con 0x7fbb480b8d70 2024-04-06T12:27:14.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb2f7fe700 1 -- 172.21.15.189:0/2850566580 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb380038c0 con 0x7fbb480b8d70 2024-04-06T12:27:14.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.740+0000 7fbb4d43b700 1 -- 172.21.15.189:0/2850566580 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb480c9bf0 con 0x7fbb480b8d70 2024-04-06T12:27:14.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.741+0000 7fbb4d43b700 1 -- 172.21.15.189:0/2850566580 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb48074990 con 0x7fbb480b8d70 2024-04-06T12:27:14.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.742+0000 7fbb2f7fe700 1 -- 172.21.15.189:0/2850566580 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fbb38003a20 con 0x7fbb480b8d70 2024-04-06T12:27:14.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.742+0000 7fbb2f7fe700 1 --2- 172.21.15.189:0/2850566580 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbb3406adb0 0x7fbb3406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:14.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.742+0000 7fbb2f7fe700 1 -- 172.21.15.189:0/2850566580 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5865+0+0 (secure 0 0 0) 0x7fbb380888e0 con 0x7fbb480b8d70 2024-04-06T12:27:14.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.742+0000 7fbb467fc700 1 --2- 172.21.15.189:0/2850566580 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbb3406adb0 0x7fbb3406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:14.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.743+0000 7fbb467fc700 1 --2- 172.21.15.189:0/2850566580 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbb3406adb0 0x7fbb3406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbb4806c180 tx=0x7fbb3c008030 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:14.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:14.747+0000 7fbb2f7fe700 1 -- 172.21.15.189:0/2850566580 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbb38053910 con 0x7fbb480b8d70 2024-04-06T12:27:15.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.020+0000 7fbb4d43b700 1 -- 172.21.15.189:0/2850566580 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7fbb480d2f50 con 0x7fbb480b8d70 2024-04-06T12:27:15.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.599+0000 7fbb2f7fe700 1 -- 172.21.15.189:0/2850566580 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v552) v1 ==== 159+0+0 (secure 0 0 0) 0x7fbb38056f30 con 0x7fbb480b8d70 2024-04-06T12:27:15.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.601+0000 7fbb4d43b700 1 -- 172.21.15.189:0/2850566580 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbb3406adb0 msgr2=0x7fbb3406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:15.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.601+0000 7fbb4d43b700 1 --2- 172.21.15.189:0/2850566580 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbb3406adb0 0x7fbb3406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbb4806c180 tx=0x7fbb3c008030 comp rx=0 tx=0).stop 2024-04-06T12:27:15.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.601+0000 7fbb4d43b700 1 -- 172.21.15.189:0/2850566580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb480b8d70 msgr2=0x7fbb480c7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:15.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.601+0000 7fbb4d43b700 1 --2- 172.21.15.189:0/2850566580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb480b8d70 0x7fbb480c7920 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fbb3800c450 tx=0x7fbb3800c810 comp rx=0 tx=0).stop 2024-04-06T12:27:15.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.601+0000 7fbb4d43b700 1 -- 172.21.15.189:0/2850566580 shutdown_connections 2024-04-06T12:27:15.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.601+0000 7fbb4d43b700 1 --2- 172.21.15.189:0/2850566580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb4815d6f0 0x7fbb480b4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:15.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.601+0000 7fbb4d43b700 1 --2- 172.21.15.189:0/2850566580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb480b8d70 0x7fbb480c7920 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:15.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.601+0000 7fbb4d43b700 1 --2- 172.21.15.189:0/2850566580 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbb3406adb0 0x7fbb3406d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:15.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.601+0000 7fbb4d43b700 1 --2- 172.21.15.189:0/2850566580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb480ce730 0x7fbb480cc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:15.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.601+0000 7fbb4d43b700 1 -- 172.21.15.189:0/2850566580 >> 172.21.15.189:0/2850566580 conn(0x7fbb48072490 msgr2=0x7fbb4806b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:15.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.601+0000 7fbb4d43b700 1 -- 172.21.15.189:0/2850566580 shutdown_connections 2024-04-06T12:27:15.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.601+0000 7fbb4d43b700 1 -- 172.21.15.189:0/2850566580 wait complete. 2024-04-06T12:27:15.613 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-06T12:27:15.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.742+0000 7f656eb95700 1 Processor -- start 2024-04-06T12:27:15.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.742+0000 7f656eb95700 1 -- start start 2024-04-06T12:27:15.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.743+0000 7f656eb95700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f656815d7b0 0x7f65680b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:15.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.743+0000 7f656eb95700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f656806a730 0x7f656806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:15.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.743+0000 7f656eb95700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f656815ddf0 0x7f656815e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:15.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.743+0000 7f656eb95700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f656806b0f0 con 0x7f656815d7b0 2024-04-06T12:27:15.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.743+0000 7f656eb95700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65680b3990 con 0x7f656815ddf0 2024-04-06T12:27:15.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.743+0000 7f656eb95700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65680b3060 con 0x7f656806a730 2024-04-06T12:27:15.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.743+0000 7f656c931700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f656815d7b0 0x7f65680b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:15.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.743+0000 7f656c931700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f656815d7b0 0x7f65680b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37356/0 (socket says 172.21.15.189:37356) 2024-04-06T12:27:15.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.743+0000 7f656c931700 1 -- 172.21.15.189:0/2197784756 learned_addr learned my addr 172.21.15.189:0/2197784756 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:15.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.743+0000 7f6567fff700 1 --2- 172.21.15.189:0/2197784756 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f656806a730 0x7f656806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:15.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.743+0000 7f656d132700 1 --2- 172.21.15.189:0/2197784756 >> v2:172.21.15.192:3300/0 conn(0x7f656815ddf0 0x7f656815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:15.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.743+0000 7f656c931700 1 -- 172.21.15.189:0/2197784756 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f656806a730 msgr2=0x7f656806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:15.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.743+0000 7f656c931700 1 --2- 172.21.15.189:0/2197784756 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f656806a730 0x7f656806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:15.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.743+0000 7f656c931700 1 -- 172.21.15.189:0/2197784756 >> v2:172.21.15.192:3300/0 conn(0x7f656815ddf0 msgr2=0x7f656815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:15.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.743+0000 7f656c931700 1 --2- 172.21.15.189:0/2197784756 >> v2:172.21.15.192:3300/0 conn(0x7f656815ddf0 0x7f656815e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:15.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.743+0000 7f656c931700 1 -- 172.21.15.189:0/2197784756 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65680b3340 con 0x7f656815d7b0 2024-04-06T12:27:15.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.744+0000 7f656c931700 1 --2- 172.21.15.189:0/2197784756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f656815d7b0 0x7f65680b3e20 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f65680b4360 tx=0x7f655c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b01e1b6dd0bff9a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:15.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.744+0000 7f65677fe700 1 -- 172.21.15.189:0/2197784756 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c00b870 con 0x7f656815d7b0 2024-04-06T12:27:15.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.744+0000 7f65677fe700 1 -- 172.21.15.189:0/2197784756 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655c00b9d0 con 0x7f656815d7b0 2024-04-06T12:27:15.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.744+0000 7f65677fe700 1 -- 172.21.15.189:0/2197784756 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c00bd60 con 0x7f656815d7b0 2024-04-06T12:27:15.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.744+0000 7f656eb95700 1 -- 172.21.15.189:0/2197784756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f656815d7b0 msgr2=0x7f65680b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:15.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.744+0000 7f656eb95700 1 --2- 172.21.15.189:0/2197784756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f656815d7b0 0x7f65680b3e20 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f65680b4360 tx=0x7f655c009bd0 comp rx=0 tx=0).stop 2024-04-06T12:27:15.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.744+0000 7f656eb95700 1 -- 172.21.15.189:0/2197784756 shutdown_connections 2024-04-06T12:27:15.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.744+0000 7f656eb95700 1 --2- 172.21.15.189:0/2197784756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f656815d7b0 0x7f65680b3e20 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:15.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.744+0000 7f656eb95700 1 --2- 172.21.15.189:0/2197784756 >> v2:172.21.15.192:3300/0 conn(0x7f656815ddf0 0x7f656815e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:15.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.744+0000 7f656eb95700 1 --2- 172.21.15.189:0/2197784756 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f656806a730 0x7f656806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:15.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.744+0000 7f656eb95700 1 -- 172.21.15.189:0/2197784756 >> 172.21.15.189:0/2197784756 conn(0x7f65680d00b0 msgr2=0x7f65680d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:15.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.744+0000 7f656eb95700 1 -- 172.21.15.189:0/2197784756 shutdown_connections 2024-04-06T12:27:15.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.744+0000 7f656eb95700 1 -- 172.21.15.189:0/2197784756 wait complete. 2024-04-06T12:27:15.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.745+0000 7f656eb95700 1 Processor -- start 2024-04-06T12:27:15.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.746+0000 7f656eb95700 1 -- start start 2024-04-06T12:27:15.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.746+0000 7f656eb95700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f656806a730 0x7f6568146780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:15.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.746+0000 7f656eb95700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f656815d7b0 0x7f65680aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:15.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.746+0000 7f656eb95700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f656815ddf0 0x7f6568148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:15.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.746+0000 7f656eb95700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6568148180 con 0x7f656815d7b0 2024-04-06T12:27:15.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.746+0000 7f656eb95700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6568147570 con 0x7f656815ddf0 2024-04-06T12:27:15.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.746+0000 7f656eb95700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6568147850 con 0x7f656806a730 2024-04-06T12:27:15.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.746+0000 7f6567fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f656815d7b0 0x7f65680aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:15.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.747+0000 7f6567fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f656815d7b0 0x7f65680aeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37372/0 (socket says 172.21.15.189:37372) 2024-04-06T12:27:15.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.747+0000 7f6567fff700 1 -- 172.21.15.189:0/2634821106 learned_addr learned my addr 172.21.15.189:0/2634821106 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:15.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.747+0000 7f656c931700 1 --2- 172.21.15.189:0/2634821106 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f656806a730 0x7f6568146780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:15.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.747+0000 7f656d132700 1 --2- 172.21.15.189:0/2634821106 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f656815ddf0 0x7f6568148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:15.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.747+0000 7f6567fff700 1 -- 172.21.15.189:0/2634821106 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f656806a730 msgr2=0x7f6568146780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:15.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.747+0000 7f6567fff700 1 --2- 172.21.15.189:0/2634821106 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f656806a730 0x7f6568146780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:15.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.747+0000 7f6567fff700 1 -- 172.21.15.189:0/2634821106 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f656815ddf0 msgr2=0x7f6568148610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:15.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.747+0000 7f6567fff700 1 --2- 172.21.15.189:0/2634821106 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f656815ddf0 0x7f6568148610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:15.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.747+0000 7f6567fff700 1 -- 172.21.15.189:0/2634821106 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655c009550 con 0x7f656815d7b0 2024-04-06T12:27:15.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.747+0000 7f6567fff700 1 --2- 172.21.15.189:0/2634821106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f656815d7b0 0x7f65680aeb80 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f655400e810 tx=0x7f655400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:15.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.747+0000 7f65657fa700 1 -- 172.21.15.189:0/2634821106 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6554013610 con 0x7f656815d7b0 2024-04-06T12:27:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.747+0000 7f65657fa700 1 -- 172.21.15.189:0/2634821106 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6554013770 con 0x7f656815d7b0 2024-04-06T12:27:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.747+0000 7f656eb95700 1 -- 172.21.15.189:0/2634821106 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6568147b30 con 0x7f656815d7b0 2024-04-06T12:27:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.747+0000 7f65657fa700 1 -- 172.21.15.189:0/2634821106 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6554013b20 con 0x7f656815d7b0 2024-04-06T12:27:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.747+0000 7f656eb95700 1 -- 172.21.15.189:0/2634821106 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6568146cc0 con 0x7f656815d7b0 2024-04-06T12:27:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.748+0000 7f656eb95700 1 -- 172.21.15.189:0/2634821106 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6568148330 con 0x7f656815d7b0 2024-04-06T12:27:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.749+0000 7f65657fa700 1 -- 172.21.15.189:0/2634821106 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f6554013c80 con 0x7f656815d7b0 2024-04-06T12:27:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.749+0000 7f65657fa700 1 --2- 172.21.15.189:0/2634821106 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f654c06adb0 0x7f654c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.749+0000 7f65657fa700 1 -- 172.21.15.189:0/2634821106 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f6554089270 con 0x7f656815d7b0 2024-04-06T12:27:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.749+0000 7f656c931700 1 --2- 172.21.15.189:0/2634821106 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f654c06adb0 0x7f654c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.749+0000 7f656c931700 1 --2- 172.21.15.189:0/2634821106 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f654c06adb0 0x7f654c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f655c000c00 tx=0x7f655c008030 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.753+0000 7f65657fa700 1 -- 172.21.15.189:0/2634821106 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6568148330 con 0x7f656815d7b0 2024-04-06T12:27:15.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.946+0000 7f656eb95700 1 -- 172.21.15.189:0/2634821106 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6568067bb0 con 0x7f656815d7b0 2024-04-06T12:27:15.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.950+0000 7f65657fa700 1 -- 172.21.15.189:0/2634821106 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f65540541a0 con 0x7f656815d7b0 2024-04-06T12:27:15.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.952+0000 7f656eb95700 1 -- 172.21.15.189:0/2634821106 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f654c06adb0 msgr2=0x7f654c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:15.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.952+0000 7f656eb95700 1 --2- 172.21.15.189:0/2634821106 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f654c06adb0 0x7f654c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f655c000c00 tx=0x7f655c008030 comp rx=0 tx=0).stop 2024-04-06T12:27:15.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.952+0000 7f656eb95700 1 -- 172.21.15.189:0/2634821106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f656815d7b0 msgr2=0x7f65680aeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:15.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.952+0000 7f656eb95700 1 --2- 172.21.15.189:0/2634821106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f656815d7b0 0x7f65680aeb80 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f655400e810 tx=0x7f655400ebd0 comp rx=0 tx=0).stop 2024-04-06T12:27:15.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.952+0000 7f656eb95700 1 -- 172.21.15.189:0/2634821106 shutdown_connections 2024-04-06T12:27:15.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.952+0000 7f656eb95700 1 --2- 172.21.15.189:0/2634821106 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f654c06adb0 0x7f654c06d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:15.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.952+0000 7f656eb95700 1 --2- 172.21.15.189:0/2634821106 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f656815ddf0 0x7f6568148610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:15.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.952+0000 7f656eb95700 1 --2- 172.21.15.189:0/2634821106 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f656806a730 0x7f6568146780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:15.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.952+0000 7f656eb95700 1 --2- 172.21.15.189:0/2634821106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f656815d7b0 0x7f65680aeb80 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:15.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.952+0000 7f656eb95700 1 -- 172.21.15.189:0/2634821106 >> 172.21.15.189:0/2634821106 conn(0x7f65680d00b0 msgr2=0x7f656815f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:15.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.952+0000 7f656eb95700 1 -- 172.21.15.189:0/2634821106 shutdown_connections 2024-04-06T12:27:15.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:15.952+0000 7f656eb95700 1 -- 172.21.15.189:0/2634821106 wait complete. 2024-04-06T12:27:15.962 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T12:27:15.963 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:27:16.008 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:27:16.060 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T12:27:16.060 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:27:16.060 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:27:16.078 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:27:16.078 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:27:16.078 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:27:16.098 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:27:16.098 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:27:16.098 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:27:16.135 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:27:16.136 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:27:16.136 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:27:16.156 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:27:16.157 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-06T12:27:16.157 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:27:16.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.299+0000 7f3e967fc700 1 Processor -- start 2024-04-06T12:27:16.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.299+0000 7f3e967fc700 1 -- start start 2024-04-06T12:27:16.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.300+0000 7f3e967fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3e9006ae40 0x7f3e900b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:16.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.300+0000 7f3e967fc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3e9015ddf0 0x7f3e9015e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:16.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.300+0000 7f3e967fc700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e900b3990 con 0x7f3e9015d7b0 2024-04-06T12:27:16.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.300+0000 7f3e967fc700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e900b3060 con 0x7f3e9006ae40 2024-04-06T12:27:16.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.300+0000 7f3e967fc700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e900b3340 con 0x7f3e9015ddf0 2024-04-06T12:27:16.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.300+0000 7f3e8f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3e9006ae40 0x7f3e900b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:16.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.300+0000 7f3e8f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3e9006ae40 0x7f3e900b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37380/0 (socket says 172.21.15.189:37380) 2024-04-06T12:27:16.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.300+0000 7f3e8f7fe700 1 -- 172.21.15.189:0/2786875220 learned_addr learned my addr 172.21.15.189:0/2786875220 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:16.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.300+0000 7f3e94d99700 1 --2- 172.21.15.189:0/2786875220 >> v2:172.21.15.192:3300/0 conn(0x7f3e9015ddf0 0x7f3e9015e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:16.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.300+0000 7f3e8f7fe700 1 -- 172.21.15.189:0/2786875220 >> v2:172.21.15.192:3300/0 conn(0x7f3e9015ddf0 msgr2=0x7f3e9015e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:16.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.300+0000 7f3e8f7fe700 1 --2- 172.21.15.189:0/2786875220 >> v2:172.21.15.192:3300/0 conn(0x7f3e9015ddf0 0x7f3e9015e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:16.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.300+0000 7f3e8f7fe700 1 -- 172.21.15.189:0/2786875220 >> v1:172.21.15.192:6789/0 conn(0x7f3e9015d7b0 legacy=0x7f3e9006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:16.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.300+0000 7f3e8f7fe700 1 -- 172.21.15.189:0/2786875220 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e900b3620 con 0x7f3e9006ae40 2024-04-06T12:27:16.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.300+0000 7f3e8f7fe700 1 --2- 172.21.15.189:0/2786875220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9006ae40 0x7f3e900b3e20 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f3e9006d2c0 tx=0x7f3e84009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e398a79de7e9c79e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:16.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.301+0000 7f3e8effd700 1 -- 172.21.15.189:0/2786875220 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e84004230 con 0x7f3e9006ae40 2024-04-06T12:27:16.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.301+0000 7f3e8effd700 1 -- 172.21.15.189:0/2786875220 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e84004390 con 0x7f3e9006ae40 2024-04-06T12:27:16.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.301+0000 7f3e8effd700 1 -- 172.21.15.189:0/2786875220 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e840036a0 con 0x7f3e9006ae40 2024-04-06T12:27:16.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.301+0000 7f3e967fc700 1 -- 172.21.15.189:0/2786875220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9006ae40 msgr2=0x7f3e900b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:16.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.301+0000 7f3e967fc700 1 --2- 172.21.15.189:0/2786875220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9006ae40 0x7f3e900b3e20 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f3e9006d2c0 tx=0x7f3e84009bd0 comp rx=0 tx=0).stop 2024-04-06T12:27:16.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.301+0000 7f3e967fc700 1 -- 172.21.15.189:0/2786875220 shutdown_connections 2024-04-06T12:27:16.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.301+0000 7f3e967fc700 1 --2- 172.21.15.189:0/2786875220 >> v2:172.21.15.192:3300/0 conn(0x7f3e9015ddf0 0x7f3e9015e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:16.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.301+0000 7f3e967fc700 1 --2- 172.21.15.189:0/2786875220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9006ae40 0x7f3e900b3e20 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:16.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.301+0000 7f3e967fc700 1 -- 172.21.15.189:0/2786875220 >> 172.21.15.189:0/2786875220 conn(0x7f3e900d00b0 msgr2=0x7f3e900d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:16.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.301+0000 7f3e967fc700 1 -- 172.21.15.189:0/2786875220 shutdown_connections 2024-04-06T12:27:16.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.301+0000 7f3e967fc700 1 -- 172.21.15.189:0/2786875220 wait complete. 2024-04-06T12:27:16.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.302+0000 7f3e967fc700 1 Processor -- start 2024-04-06T12:27:16.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.303+0000 7f3e967fc700 1 -- start start 2024-04-06T12:27:16.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.303+0000 7f3e967fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9006ae40 0x7f3e901519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:16.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.303+0000 7f3e967fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e9015d7b0 0x7f3e901511a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:16.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.303+0000 7f3e967fc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9015ddf0 0x7f3e900cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:16.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.303+0000 7f3e967fc700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e90157260 con 0x7f3e9006ae40 2024-04-06T12:27:16.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.303+0000 7f3e967fc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e900cc540 con 0x7f3e9015ddf0 2024-04-06T12:27:16.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.303+0000 7f3e967fc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e900cc800 con 0x7f3e9015d7b0 2024-04-06T12:27:16.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.303+0000 7f3e8ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9006ae40 0x7f3e901519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:16.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.303+0000 7f3e8ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9006ae40 0x7f3e901519e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37390/0 (socket says 172.21.15.189:37390) 2024-04-06T12:27:16.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.303+0000 7f3e8ffff700 1 -- 172.21.15.189:0/459122487 learned_addr learned my addr 172.21.15.189:0/459122487 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:16.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.304+0000 7f3e94d99700 1 --2- 172.21.15.189:0/459122487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9015ddf0 0x7f3e900cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:16.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.304+0000 7f3e8f7fe700 1 --2- 172.21.15.189:0/459122487 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e9015d7b0 0x7f3e901511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:16.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.304+0000 7f3e8ffff700 1 -- 172.21.15.189:0/459122487 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e9015d7b0 msgr2=0x7f3e901511a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:16.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.304+0000 7f3e8ffff700 1 --2- 172.21.15.189:0/459122487 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e9015d7b0 0x7f3e901511a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:16.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.304+0000 7f3e8ffff700 1 -- 172.21.15.189:0/459122487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9015ddf0 msgr2=0x7f3e900cd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:16.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.304+0000 7f3e8ffff700 1 --2- 172.21.15.189:0/459122487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9015ddf0 0x7f3e900cd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:16.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.304+0000 7f3e8ffff700 1 -- 172.21.15.189:0/459122487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e84009550 con 0x7f3e9006ae40 2024-04-06T12:27:16.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.304+0000 7f3e94d99700 1 --2- 172.21.15.189:0/459122487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9015ddf0 0x7f3e900cd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:16.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.304+0000 7f3e8f7fe700 1 --2- 172.21.15.189:0/459122487 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e9015d7b0 0x7f3e901511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:16.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.304+0000 7f3e8ffff700 1 --2- 172.21.15.189:0/459122487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9006ae40 0x7f3e901519e0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f3e8000c4d0 tx=0x7f3e8000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:16.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.304+0000 7f3e8cff9700 1 -- 172.21.15.189:0/459122487 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e80011790 con 0x7f3e9006ae40 2024-04-06T12:27:16.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.304+0000 7f3e8cff9700 1 -- 172.21.15.189:0/459122487 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e800118f0 con 0x7f3e9006ae40 2024-04-06T12:27:16.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.304+0000 7f3e967fc700 1 -- 172.21.15.189:0/459122487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e900ccae0 con 0x7f3e9006ae40 2024-04-06T12:27:16.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.304+0000 7f3e8cff9700 1 -- 172.21.15.189:0/459122487 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e80011ca0 con 0x7f3e9006ae40 2024-04-06T12:27:16.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.304+0000 7f3e967fc700 1 -- 172.21.15.189:0/459122487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e900cd0a0 con 0x7f3e9006ae40 2024-04-06T12:27:16.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.305+0000 7f3e967fc700 1 -- 172.21.15.189:0/459122487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e90067bb0 con 0x7f3e9006ae40 2024-04-06T12:27:16.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.306+0000 7f3e8cff9700 1 -- 172.21.15.189:0/459122487 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f3e8001c460 con 0x7f3e9006ae40 2024-04-06T12:27:16.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.306+0000 7f3e8cff9700 1 --2- 172.21.15.189:0/459122487 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3e7c06adb0 0x7f3e7c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:16.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.306+0000 7f3e8cff9700 1 -- 172.21.15.189:0/459122487 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f3e80089370 con 0x7f3e9006ae40 2024-04-06T12:27:16.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.306+0000 7f3e8f7fe700 1 --2- 172.21.15.189:0/459122487 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3e7c06adb0 0x7f3e7c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:16.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.306+0000 7f3e8f7fe700 1 --2- 172.21.15.189:0/459122487 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3e7c06adb0 0x7f3e7c06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3e9006d2c0 tx=0x7f3e84008030 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:16.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.310+0000 7f3e8cff9700 1 -- 172.21.15.189:0/459122487 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3e800543a0 con 0x7f3e9006ae40 2024-04-06T12:27:16.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.549+0000 7f3e967fc700 1 -- 172.21.15.189:0/459122487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e90074990 con 0x7f3e9006ae40 2024-04-06T12:27:16.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.550+0000 7f3e8cff9700 1 -- 172.21.15.189:0/459122487 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f3e800579c0 con 0x7f3e9006ae40 2024-04-06T12:27:16.548 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:16.548 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":26,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/527093473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":527093473},{"type":"v1","addr":"172.21.15.192:6835","nonce":527093473}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6105,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2142144473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2142144473},{"type":"v1","addr":"172.21.15.189:6836","nonce":2142144473}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6108,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2285762709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2285762709},{"type":"v1","addr":"172.21.15.189:6837","nonce":2285762709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6175,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/562788548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":562788548},{"type":"v1","addr":"172.21.15.192:6834","nonce":562788548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[]} 2024-04-06T12:27:16.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.552+0000 7f3e967fc700 1 -- 172.21.15.189:0/459122487 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3e7c06adb0 msgr2=0x7f3e7c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:16.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.552+0000 7f3e967fc700 1 --2- 172.21.15.189:0/459122487 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3e7c06adb0 0x7f3e7c06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3e9006d2c0 tx=0x7f3e84008030 comp rx=0 tx=0).stop 2024-04-06T12:27:16.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.552+0000 7f3e967fc700 1 -- 172.21.15.189:0/459122487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9006ae40 msgr2=0x7f3e901519e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:16.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.552+0000 7f3e967fc700 1 --2- 172.21.15.189:0/459122487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9006ae40 0x7f3e901519e0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f3e8000c4d0 tx=0x7f3e8000c890 comp rx=0 tx=0).stop 2024-04-06T12:27:16.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.552+0000 7f3e967fc700 1 -- 172.21.15.189:0/459122487 shutdown_connections 2024-04-06T12:27:16.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.552+0000 7f3e967fc700 1 --2- 172.21.15.189:0/459122487 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3e7c06adb0 0x7f3e7c06d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:16.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.552+0000 7f3e967fc700 1 --2- 172.21.15.189:0/459122487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9015ddf0 0x7f3e900cd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:16.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.552+0000 7f3e967fc700 1 --2- 172.21.15.189:0/459122487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9006ae40 0x7f3e901519e0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:16.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.552+0000 7f3e967fc700 1 --2- 172.21.15.189:0/459122487 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e9015d7b0 0x7f3e901511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:16.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.552+0000 7f3e967fc700 1 -- 172.21.15.189:0/459122487 >> 172.21.15.189:0/459122487 conn(0x7f3e900d00b0 msgr2=0x7f3e9006eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:16.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.552+0000 7f3e967fc700 1 -- 172.21.15.189:0/459122487 shutdown_connections 2024-04-06T12:27:16.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.552+0000 7f3e967fc700 1 -- 172.21.15.189:0/459122487 wait complete. 2024-04-06T12:27:16.551 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 26 2024-04-06T12:27:16.562 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-06T12:27:16.562 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-06T12:27:16.563 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-06T12:27:16.563 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:27:16.563 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-06T12:27:16.563 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-06T12:27:16.563 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-06T12:27:16.563 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:27:16.563 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-06T12:27:16.564 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-06T12:27:16.564 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-06T12:27:16.564 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:27:16.564 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-06T12:27:16.564 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-06T12:27:16.564 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-06T12:27:16.564 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:27:16.572 INFO:tasks.ceph.mds.a:Stopped 2024-04-06T12:27:16.572 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-06T12:27:16.573 INFO:tasks.ceph.mds.c:Stopped 2024-04-06T12:27:16.573 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-06T12:27:16.576 INFO:tasks.ceph.mds.d:Stopped 2024-04-06T12:27:16.576 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-06T12:27:16.576 INFO:tasks.ceph.mds.b:Stopped 2024-04-06T12:27:16.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-06T12:27:16.614 INFO:tasks.ceph.mds.a:Started 2024-04-06T12:27:16.615 DEBUG:teuthology.parallel:result is None 2024-04-06T12:27:16.615 INFO:tasks.ceph.mds.c:Started 2024-04-06T12:27:16.615 DEBUG:teuthology.parallel:result is None 2024-04-06T12:27:16.618 INFO:tasks.ceph.mds.d:Started 2024-04-06T12:27:16.618 DEBUG:teuthology.parallel:result is None 2024-04-06T12:27:16.619 INFO:tasks.ceph.mds.b:Started 2024-04-06T12:27:16.619 DEBUG:teuthology.parallel:result is None 2024-04-06T12:27:16.619 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-06T12:27:16.694 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2024-04-06T12:27:16.694 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T12:27:16.705 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T12:27:16.706 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2024-04-06T12:27:16.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.754+0000 7f06dec9d700 1 Processor -- start 2024-04-06T12:27:16.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.754+0000 7f06dec9d700 1 -- start start 2024-04-06T12:27:16.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.754+0000 7f06dec9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06d815d7b0 0x7f06d80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:16.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.755+0000 7f06dec9d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f06d806a730 0x7f06d806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:16.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.755+0000 7f06dec9d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d80b3990 con 0x7f06d815ddf0 2024-04-06T12:27:16.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.755+0000 7f06dec9d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d815e950 con 0x7f06d815d7b0 2024-04-06T12:27:16.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.755+0000 7f06dec9d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d806b0f0 con 0x7f06d806a730 2024-04-06T12:27:16.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.755+0000 7f06dca39700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06d815d7b0 0x7f06d80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:16.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.755+0000 7f06dd23a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f06d815ddf0 0x7f06d815e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37514/0 (socket says 172.21.15.189:37514) 2024-04-06T12:27:16.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.755+0000 7f06dd23a700 1 -- 172.21.15.189:0/2075127018 learned_addr learned my addr 172.21.15.189:0/2075127018 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:16.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.755+0000 7f06d7fff700 1 --2- 172.21.15.189:0/2075127018 >> v2:172.21.15.192:3300/0 conn(0x7f06d806a730 0x7f06d806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:16.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.755+0000 7f06dca39700 1 -- 172.21.15.189:0/2075127018 >> v2:172.21.15.192:3300/0 conn(0x7f06d806a730 msgr2=0x7f06d806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:16.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.755+0000 7f06dca39700 1 --2- 172.21.15.189:0/2075127018 >> v2:172.21.15.192:3300/0 conn(0x7f06d806a730 0x7f06d806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:16.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.755+0000 7f06dca39700 1 -- 172.21.15.189:0/2075127018 >> v1:172.21.15.189:6789/0 conn(0x7f06d815ddf0 legacy=0x7f06d815e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:16.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.755+0000 7f06d77fe700 1 -- 172.21.15.189:0/2075127018 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766355419 0 0) 0x7f06d80b3990 con 0x7f06d815ddf0 2024-04-06T12:27:16.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.755+0000 7f06dca39700 1 -- 172.21.15.189:0/2075127018 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d80b3060 con 0x7f06d815d7b0 2024-04-06T12:27:16.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.755+0000 7f06d7fff700 1 --2- 172.21.15.189:0/2075127018 >> v2:172.21.15.192:3300/0 conn(0x7f06d806a730 0x7f06d806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:16.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.755+0000 7f06dca39700 1 --2- 172.21.15.189:0/2075127018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06d815d7b0 0x7f06d80b3e20 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f06d80afc00 tx=0x7f06cc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a229fd3e5cd6465 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:16.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.755+0000 7f06d77fe700 1 -- 172.21.15.189:0/2075127018 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc011830 con 0x7f06d815d7b0 2024-04-06T12:27:16.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.755+0000 7f06d77fe700 1 -- 172.21.15.189:0/2075127018 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06cc011990 con 0x7f06d815d7b0 2024-04-06T12:27:16.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.755+0000 7f06d77fe700 1 -- 172.21.15.189:0/2075127018 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc011d20 con 0x7f06d815d7b0 2024-04-06T12:27:16.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.756+0000 7f06dec9d700 1 -- 172.21.15.189:0/2075127018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06d815d7b0 msgr2=0x7f06d80b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:16.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.756+0000 7f06dec9d700 1 --2- 172.21.15.189:0/2075127018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06d815d7b0 0x7f06d80b3e20 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f06d80afc00 tx=0x7f06cc009b20 comp rx=0 tx=0).stop 2024-04-06T12:27:16.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.756+0000 7f06dec9d700 1 -- 172.21.15.189:0/2075127018 shutdown_connections 2024-04-06T12:27:16.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.756+0000 7f06dec9d700 1 --2- 172.21.15.189:0/2075127018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06d815d7b0 0x7f06d80b3e20 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:16.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.756+0000 7f06dec9d700 1 --2- 172.21.15.189:0/2075127018 >> v2:172.21.15.192:3300/0 conn(0x7f06d806a730 0x7f06d806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:16.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.756+0000 7f06dec9d700 1 -- 172.21.15.189:0/2075127018 >> 172.21.15.189:0/2075127018 conn(0x7f06d80d00b0 msgr2=0x7f06d80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:16.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.756+0000 7f06dec9d700 1 -- 172.21.15.189:0/2075127018 shutdown_connections 2024-04-06T12:27:16.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.756+0000 7f06dec9d700 1 -- 172.21.15.189:0/2075127018 wait complete. 2024-04-06T12:27:16.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.757+0000 7f06dec9d700 1 Processor -- start 2024-04-06T12:27:16.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.757+0000 7f06dec9d700 1 -- start start 2024-04-06T12:27:16.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.758+0000 7f06dec9d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06d806a730 0x7f06d814fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:16.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.758+0000 7f06dec9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06d815d7b0 0x7f06d80cce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:16.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.758+0000 7f06dec9d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06d815ddf0 0x7f06d80b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:16.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.758+0000 7f06dec9d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d815b750 con 0x7f06d815d7b0 2024-04-06T12:27:16.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.758+0000 7f06dec9d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d8150110 con 0x7f06d815ddf0 2024-04-06T12:27:16.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.758+0000 7f06dec9d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d815aad0 con 0x7f06d806a730 2024-04-06T12:27:16.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.758+0000 7f06d7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06d815d7b0 0x7f06d80cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:16.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.758+0000 7f06d7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06d815d7b0 0x7f06d80cce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37442/0 (socket says 172.21.15.189:37442) 2024-04-06T12:27:16.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.758+0000 7f06d7fff700 1 -- 172.21.15.189:0/3645200344 learned_addr learned my addr 172.21.15.189:0/3645200344 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:16.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.758+0000 7f06d7fff700 1 -- 172.21.15.189:0/3645200344 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06d806a730 msgr2=0x7f06d814fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:16.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.758+0000 7f06dca39700 1 --2- 172.21.15.189:0/3645200344 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06d806a730 0x7f06d814fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:16.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.758+0000 7f06dd23a700 1 --2- 172.21.15.189:0/3645200344 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06d815ddf0 0x7f06d80b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:16.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.758+0000 7f06d7fff700 1 --2- 172.21.15.189:0/3645200344 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06d806a730 0x7f06d814fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:16.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.758+0000 7f06d7fff700 1 -- 172.21.15.189:0/3645200344 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06d815ddf0 msgr2=0x7f06d80b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:16.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.758+0000 7f06d7fff700 1 --2- 172.21.15.189:0/3645200344 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06d815ddf0 0x7f06d80b0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:16.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.758+0000 7f06d7fff700 1 -- 172.21.15.189:0/3645200344 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06cc0094a0 con 0x7f06d815d7b0 2024-04-06T12:27:16.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.759+0000 7f06d7fff700 1 --2- 172.21.15.189:0/3645200344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06d815d7b0 0x7f06d80cce80 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f06c400e810 tx=0x7f06c400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:16.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.759+0000 7f06d57fa700 1 -- 172.21.15.189:0/3645200344 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c4013610 con 0x7f06d815d7b0 2024-04-06T12:27:16.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.759+0000 7f06d57fa700 1 -- 172.21.15.189:0/3645200344 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c4013770 con 0x7f06d815d7b0 2024-04-06T12:27:16.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.759+0000 7f06dec9d700 1 -- 172.21.15.189:0/3645200344 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d815adb0 con 0x7f06d815d7b0 2024-04-06T12:27:16.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.759+0000 7f06d57fa700 1 -- 172.21.15.189:0/3645200344 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c4013b20 con 0x7f06d815d7b0 2024-04-06T12:27:16.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.759+0000 7f06dec9d700 1 -- 172.21.15.189:0/3645200344 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d815b370 con 0x7f06d815d7b0 2024-04-06T12:27:16.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.760+0000 7f06dec9d700 1 -- 172.21.15.189:0/3645200344 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d8067bb0 con 0x7f06d815d7b0 2024-04-06T12:27:16.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.760+0000 7f06d57fa700 1 -- 172.21.15.189:0/3645200344 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f06c4013c80 con 0x7f06d815d7b0 2024-04-06T12:27:16.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.760+0000 7f06d57fa700 1 --2- 172.21.15.189:0/3645200344 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06bc06adb0 0x7f06bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:16.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.760+0000 7f06d57fa700 1 -- 172.21.15.189:0/3645200344 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f06c4089390 con 0x7f06d815d7b0 2024-04-06T12:27:16.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.760+0000 7f06dca39700 1 --2- 172.21.15.189:0/3645200344 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06bc06adb0 0x7f06bc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:16.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.761+0000 7f06dca39700 1 --2- 172.21.15.189:0/3645200344 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06bc06adb0 0x7f06bc06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f06d80cfc50 tx=0x7f06cc008030 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:16.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.765+0000 7f06d57fa700 1 -- 172.21.15.189:0/3645200344 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f06c40542b0 con 0x7f06d815d7b0 2024-04-06T12:27:16.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:16.973+0000 7f06dec9d700 1 -- 172.21.15.189:0/3645200344 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f06d8074990 con 0x7f06d815d7b0 2024-04-06T12:27:17.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.740+0000 7f06d57fa700 1 -- 172.21.15.189:0/3645200344 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v80) v1 ==== 97+0+0 (secure 0 0 0) 0x7f06c40578d0 con 0x7f06d815d7b0 2024-04-06T12:27:17.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.744+0000 7f06dec9d700 1 -- 172.21.15.189:0/3645200344 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06bc06adb0 msgr2=0x7f06bc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:17.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.744+0000 7f06dec9d700 1 --2- 172.21.15.189:0/3645200344 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06bc06adb0 0x7f06bc06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f06d80cfc50 tx=0x7f06cc008030 comp rx=0 tx=0).stop 2024-04-06T12:27:17.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.744+0000 7f06dec9d700 1 -- 172.21.15.189:0/3645200344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06d815d7b0 msgr2=0x7f06d80cce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:17.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.744+0000 7f06dec9d700 1 --2- 172.21.15.189:0/3645200344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06d815d7b0 0x7f06d80cce80 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f06c400e810 tx=0x7f06c400ebd0 comp rx=0 tx=0).stop 2024-04-06T12:27:17.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.745+0000 7f06dec9d700 1 -- 172.21.15.189:0/3645200344 shutdown_connections 2024-04-06T12:27:17.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.745+0000 7f06dec9d700 1 --2- 172.21.15.189:0/3645200344 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06bc06adb0 0x7f06bc06d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:17.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.745+0000 7f06dec9d700 1 --2- 172.21.15.189:0/3645200344 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06d815ddf0 0x7f06d80b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:17.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.745+0000 7f06dec9d700 1 --2- 172.21.15.189:0/3645200344 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06d806a730 0x7f06d814fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:17.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.745+0000 7f06dec9d700 1 --2- 172.21.15.189:0/3645200344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06d815d7b0 0x7f06d80cce80 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:17.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.745+0000 7f06dec9d700 1 -- 172.21.15.189:0/3645200344 >> 172.21.15.189:0/3645200344 conn(0x7f06d80d00b0 msgr2=0x7f06d80ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:17.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.745+0000 7f06dec9d700 1 -- 172.21.15.189:0/3645200344 shutdown_connections 2024-04-06T12:27:17.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.745+0000 7f06dec9d700 1 -- 172.21.15.189:0/3645200344 wait complete. 2024-04-06T12:27:17.744 INFO:teuthology.orchestra.run.smithi189.stderr: removed all blocklist entries 2024-04-06T12:27:17.756 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-06T12:27:17.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.882+0000 7fd246b6f700 1 Processor -- start 2024-04-06T12:27:17.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.882+0000 7fd246b6f700 1 -- start start 2024-04-06T12:27:17.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.882+0000 7fd246b6f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd240146dd0 0x7fd24006d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:17.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.882+0000 7fd246b6f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2400cedf0 0x7fd2400cf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:17.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.882+0000 7fd246b6f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2400cf780 con 0x7fd2400d17a0 2024-04-06T12:27:17.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.882+0000 7fd246b6f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd24006beb0 con 0x7fd240146dd0 2024-04-06T12:27:17.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.882+0000 7fd246b6f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24006c0a0 con 0x7fd2400cedf0 2024-04-06T12:27:17.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.883+0000 7fd23ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd240146dd0 0x7fd24006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:17.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.883+0000 7fd23ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd240146dd0 0x7fd24006d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37448/0 (socket says 172.21.15.189:37448) 2024-04-06T12:27:17.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.883+0000 7fd23ffff700 1 -- 172.21.15.189:0/588261242 learned_addr learned my addr 172.21.15.189:0/588261242 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:17.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.883+0000 7fd24510c700 1 --2- 172.21.15.189:0/588261242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2400cedf0 0x7fd2400cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:17.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.883+0000 7fd23ffff700 1 -- 172.21.15.189:0/588261242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2400cedf0 msgr2=0x7fd2400cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:17.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.883+0000 7fd23ffff700 1 --2- 172.21.15.189:0/588261242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2400cedf0 0x7fd2400cf240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:17.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.883+0000 7fd23ffff700 1 -- 172.21.15.189:0/588261242 >> v1:172.21.15.192:6789/0 conn(0x7fd2400d17a0 legacy=0x7fd24006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:17.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.883+0000 7fd23ffff700 1 -- 172.21.15.189:0/588261242 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24006c380 con 0x7fd240146dd0 2024-04-06T12:27:17.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.883+0000 7fd23ffff700 1 --2- 172.21.15.189:0/588261242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd240146dd0 0x7fd24006d140 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0098f0 tx=0x7fd22c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246be0ad3b2821aa server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:17.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.883+0000 7fd23f7fe700 1 -- 172.21.15.189:0/588261242 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c00b8a0 con 0x7fd240146dd0 2024-04-06T12:27:17.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.883+0000 7fd23f7fe700 1 -- 172.21.15.189:0/588261242 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22c00ba00 con 0x7fd240146dd0 2024-04-06T12:27:17.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.883+0000 7fd23f7fe700 1 -- 172.21.15.189:0/588261242 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c00bd90 con 0x7fd240146dd0 2024-04-06T12:27:17.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.884+0000 7fd246b6f700 1 -- 172.21.15.189:0/588261242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd240146dd0 msgr2=0x7fd24006d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:17.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.884+0000 7fd246b6f700 1 --2- 172.21.15.189:0/588261242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd240146dd0 0x7fd24006d140 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0098f0 tx=0x7fd22c009c00 comp rx=0 tx=0).stop 2024-04-06T12:27:17.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.884+0000 7fd246b6f700 1 -- 172.21.15.189:0/588261242 shutdown_connections 2024-04-06T12:27:17.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.884+0000 7fd246b6f700 1 --2- 172.21.15.189:0/588261242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2400cedf0 0x7fd2400cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:17.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.884+0000 7fd246b6f700 1 --2- 172.21.15.189:0/588261242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd240146dd0 0x7fd24006d140 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:17.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.884+0000 7fd246b6f700 1 -- 172.21.15.189:0/588261242 >> 172.21.15.189:0/588261242 conn(0x7fd2400b2db0 msgr2=0x7fd2400b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:17.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.884+0000 7fd246b6f700 1 -- 172.21.15.189:0/588261242 shutdown_connections 2024-04-06T12:27:17.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.884+0000 7fd246b6f700 1 -- 172.21.15.189:0/588261242 wait complete. 2024-04-06T12:27:17.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.885+0000 7fd246b6f700 1 Processor -- start 2024-04-06T12:27:17.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.885+0000 7fd246b6f700 1 -- start start 2024-04-06T12:27:17.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.886+0000 7fd246b6f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2400cedf0 0x7fd24015c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:17.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.886+0000 7fd246b6f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2400d17a0 0x7fd2400b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:17.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.886+0000 7fd246b6f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd240146dd0 0x7fd2400aec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:17.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.886+0000 7fd246b6f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2400af180 con 0x7fd2400cedf0 2024-04-06T12:27:17.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.886+0000 7fd246b6f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2400b4f70 con 0x7fd2400d17a0 2024-04-06T12:27:17.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.886+0000 7fd246b6f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2400b5250 con 0x7fd240146dd0 2024-04-06T12:27:17.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.886+0000 7fd24490b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2400cedf0 0x7fd24015c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:17.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.886+0000 7fd24490b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2400cedf0 0x7fd24015c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37454/0 (socket says 172.21.15.189:37454) 2024-04-06T12:27:17.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.886+0000 7fd24490b700 1 -- 172.21.15.189:0/1541433500 learned_addr learned my addr 172.21.15.189:0/1541433500 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:17.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.886+0000 7fd24510c700 1 --2- 172.21.15.189:0/1541433500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd240146dd0 0x7fd2400aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:17.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.886+0000 7fd23ffff700 1 --2- 172.21.15.189:0/1541433500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2400d17a0 0x7fd2400b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:17.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.886+0000 7fd24490b700 1 -- 172.21.15.189:0/1541433500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd240146dd0 msgr2=0x7fd2400aec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:17.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.886+0000 7fd24490b700 1 --2- 172.21.15.189:0/1541433500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd240146dd0 0x7fd2400aec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:17.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.886+0000 7fd24490b700 1 -- 172.21.15.189:0/1541433500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2400d17a0 msgr2=0x7fd2400b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:17.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.886+0000 7fd24490b700 1 --2- 172.21.15.189:0/1541433500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2400d17a0 0x7fd2400b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:17.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.886+0000 7fd24490b700 1 -- 172.21.15.189:0/1541433500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22c009550 con 0x7fd2400cedf0 2024-04-06T12:27:17.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.886+0000 7fd24510c700 1 --2- 172.21.15.189:0/1541433500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd240146dd0 0x7fd2400aec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:17.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.887+0000 7fd24490b700 1 --2- 172.21.15.189:0/1541433500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2400cedf0 0x7fd24015c7a0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fd23400c450 tx=0x7fd23400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:17.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.887+0000 7fd23d7fa700 1 -- 172.21.15.189:0/1541433500 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2340052d0 con 0x7fd2400cedf0 2024-04-06T12:27:17.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.887+0000 7fd23d7fa700 1 -- 172.21.15.189:0/1541433500 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd234005430 con 0x7fd2400cedf0 2024-04-06T12:27:17.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.887+0000 7fd246b6f700 1 -- 172.21.15.189:0/1541433500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2400b5530 con 0x7fd2400cedf0 2024-04-06T12:27:17.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.887+0000 7fd23d7fa700 1 -- 172.21.15.189:0/1541433500 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2340038c0 con 0x7fd2400cedf0 2024-04-06T12:27:17.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.887+0000 7fd246b6f700 1 -- 172.21.15.189:0/1541433500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2401f2b40 con 0x7fd2400cedf0 2024-04-06T12:27:17.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.888+0000 7fd246b6f700 1 -- 172.21.15.189:0/1541433500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd240074990 con 0x7fd2400cedf0 2024-04-06T12:27:17.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.888+0000 7fd23d7fa700 1 -- 172.21.15.189:0/1541433500 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fd234003a20 con 0x7fd2400cedf0 2024-04-06T12:27:17.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.888+0000 7fd23d7fa700 1 --2- 172.21.15.189:0/1541433500 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd22406adb0 0x7fd22406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:17.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.889+0000 7fd23d7fa700 1 -- 172.21.15.189:0/1541433500 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fd234088420 con 0x7fd2400cedf0 2024-04-06T12:27:17.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.889+0000 7fd23ffff700 1 --2- 172.21.15.189:0/1541433500 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd22406adb0 0x7fd22406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:17.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.889+0000 7fd23ffff700 1 --2- 172.21.15.189:0/1541433500 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd22406adb0 0x7fd22406d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd2401471e0 tx=0x7fd22c008030 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:17.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:17.893+0000 7fd23d7fa700 1 -- 172.21.15.189:0/1541433500 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd234053790 con 0x7fd2400cedf0 2024-04-06T12:27:18.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.173+0000 7fd246b6f700 1 -- 172.21.15.189:0/1541433500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd2400cf780 con 0x7fd2400cedf0 2024-04-06T12:27:18.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.173+0000 7fd23d7fa700 1 -- 172.21.15.189:0/1541433500 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fd2400cf780 con 0x7fd2400cedf0 2024-04-06T12:27:18.172 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:18.172 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T12:27:18.172 INFO:teuthology.orchestra.run.smithi189.stdout: "auth_dump": [ 2024-04-06T12:27:18.172 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:18.172 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.a", 2024-04-06T12:27:18.172 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmTjYMNBAA5voWGV4Pa7pE27VogrdipQ==", 2024-04-06T12:27:18.172 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:27:18.172 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:27:18.172 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:27:18.173 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:27:18.173 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:27:18.173 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.173 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:27:18.173 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:18.173 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.b", 2024-04-06T12:27:18.173 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm9A1OBRAAnHzubCvRUX8UdW8hmDzKbQ==", 2024-04-06T12:27:18.173 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:27:18.173 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:27:18.173 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:27:18.173 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:27:18.173 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:27:18.173 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.173 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:27:18.173 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:18.174 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.c", 2024-04-06T12:27:18.174 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmz0hhOxAAD3/GINNmGoFk+oPyiHHJXg==", 2024-04-06T12:27:18.174 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:27:18.174 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:27:18.174 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:27:18.174 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:27:18.174 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:27:18.174 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.174 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:27:18.174 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:18.174 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.d", 2024-04-06T12:27:18.174 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmhoAeDRAAoQC0vicWumBRTwpi3PtYaw==", 2024-04-06T12:27:18.174 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:27:18.174 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:27:18.175 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:27:18.175 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:27:18.175 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:27:18.175 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.175 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:27:18.175 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:18.175 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.0", 2024-04-06T12:27:18.175 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDkOxFmkxLWJhAAFXewCNOFX7dhAZVgUmewpQ==", 2024-04-06T12:27:18.175 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:27:18.175 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:27:18.175 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:27:18.175 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:27:18.175 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.175 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:27:18.175 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:18.176 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.1", 2024-04-06T12:27:18.176 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDnOxFmHYqWJRAABlsVCiv3e7YJMbfodJTWag==", 2024-04-06T12:27:18.176 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:27:18.176 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:27:18.176 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:27:18.176 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:27:18.176 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.176 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:27:18.176 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:18.176 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.2", 2024-04-06T12:27:18.176 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDqOxFmkhcvJBAA8NMEpHVGbGALjV9RISKP8g==", 2024-04-06T12:27:18.176 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:27:18.176 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:27:18.176 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:27:18.176 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:27:18.177 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.177 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:27:18.177 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:18.177 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.3", 2024-04-06T12:27:18.177 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDtOxFmDGzhIxAAamBoAbE8+lAel4os67meyQ==", 2024-04-06T12:27:18.177 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:27:18.177 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:27:18.177 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:27:18.177 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:27:18.177 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.177 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:27:18.177 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:18.177 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.4", 2024-04-06T12:27:18.177 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDzOxFmobZDJxAA9l8wdS81Eyw6+17I5ac6bw==", 2024-04-06T12:27:18.177 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:27:18.177 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:27:18.178 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:27:18.178 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:27:18.178 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.178 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:27:18.178 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:18.178 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.5", 2024-04-06T12:27:18.178 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD2OxFmi+ruJRAASlQ0o9jX/wA7U5AosnL+Wg==", 2024-04-06T12:27:18.178 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:27:18.178 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:27:18.178 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:27:18.178 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:27:18.178 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.178 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:27:18.178 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:18.178 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.6", 2024-04-06T12:27:18.179 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD5OxFmbwuuJBAA0wqJRmnJyskQPZJ7/LnpvA==", 2024-04-06T12:27:18.179 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:27:18.179 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:27:18.179 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:27:18.179 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:27:18.179 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.179 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:27:18.179 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:18.179 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.7", 2024-04-06T12:27:18.179 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD8OxFmDmRgIxAAzyW9AGAbmb67ZvB5qunwLQ==", 2024-04-06T12:27:18.179 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:27:18.179 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:27:18.179 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:27:18.179 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:27:18.179 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.180 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:27:18.180 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:18.180 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.0", 2024-04-06T12:27:18.180 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm0ByJERAAoEV8Yfc09+Go6u95XafElQ==", 2024-04-06T12:27:18.180 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:27:18.180 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:27:18.180 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow r", 2024-04-06T12:27:18.180 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-06T12:27:18.180 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.180 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:27:18.180 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:18.180 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.1", 2024-04-06T12:27:18.180 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmSV5lGBAAavDEo2LGy0IHpWSAuitHXg==", 2024-04-06T12:27:18.180 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:27:18.180 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:27:18.180 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:27:18.181 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:27:18.181 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:27:18.181 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.181 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:27:18.181 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:18.181 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.2", 2024-04-06T12:27:18.181 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm/Gz5FBAA3xX/rsAyWZ1JoUVlQlBIvQ==", 2024-04-06T12:27:18.181 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:27:18.181 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:27:18.181 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:27:18.181 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:27:18.181 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:27:18.181 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.181 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:27:18.181 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:18.182 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.3", 2024-04-06T12:27:18.182 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmzra7GxAADbnzIvWnRTo65LvPwtxWrg==", 2024-04-06T12:27:18.182 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:27:18.182 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:27:18.182 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:27:18.182 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:27:18.182 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:27:18.182 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.182 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:27:18.182 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:18.182 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.admin", 2024-04-06T12:27:18.182 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmxG+wCBAANg5kt6TOUQdj3/X0Mqgx1g==", 2024-04-06T12:27:18.182 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:27:18.182 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:27:18.182 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:27:18.183 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:27:18.183 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:27:18.183 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.183 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:27:18.183 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:18.183 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.x", 2024-04-06T12:27:18.183 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmPLyBMBAAYfhkI9Z1CPlf1Tijm+L7xA==", 2024-04-06T12:27:18.183 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:27:18.183 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:27:18.183 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T12:27:18.183 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:27:18.183 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.183 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:27:18.183 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:18.184 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.y", 2024-04-06T12:27:18.184 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFm1nHJHhAAQWRe8cU8duWuUJiYaDeTaQ==", 2024-04-06T12:27:18.184 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:27:18.184 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:27:18.184 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T12:27:18.184 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:27:18.184 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.184 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:18.184 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-04-06T12:27:18.184 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T12:27:18.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.175+0000 7fd246b6f700 1 -- 172.21.15.189:0/1541433500 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd22406adb0 msgr2=0x7fd22406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:18.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.175+0000 7fd246b6f700 1 --2- 172.21.15.189:0/1541433500 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd22406adb0 0x7fd22406d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd2401471e0 tx=0x7fd22c008030 comp rx=0 tx=0).stop 2024-04-06T12:27:18.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.175+0000 7fd246b6f700 1 -- 172.21.15.189:0/1541433500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2400cedf0 msgr2=0x7fd24015c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:18.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.175+0000 7fd246b6f700 1 --2- 172.21.15.189:0/1541433500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2400cedf0 0x7fd24015c7a0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fd23400c450 tx=0x7fd23400c810 comp rx=0 tx=0).stop 2024-04-06T12:27:18.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.176+0000 7fd246b6f700 1 -- 172.21.15.189:0/1541433500 shutdown_connections 2024-04-06T12:27:18.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.176+0000 7fd246b6f700 1 --2- 172.21.15.189:0/1541433500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd240146dd0 0x7fd2400aec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.176+0000 7fd246b6f700 1 --2- 172.21.15.189:0/1541433500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2400cedf0 0x7fd24015c7a0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.176+0000 7fd246b6f700 1 --2- 172.21.15.189:0/1541433500 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd22406adb0 0x7fd22406d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.176+0000 7fd246b6f700 1 --2- 172.21.15.189:0/1541433500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2400d17a0 0x7fd2400b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.176+0000 7fd246b6f700 1 -- 172.21.15.189:0/1541433500 >> 172.21.15.189:0/1541433500 conn(0x7fd2400b2db0 msgr2=0x7fd24006b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:18.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.176+0000 7fd246b6f700 1 -- 172.21.15.189:0/1541433500 shutdown_connections 2024-04-06T12:27:18.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.176+0000 7fd246b6f700 1 -- 172.21.15.189:0/1541433500 wait complete. 2024-04-06T12:27:18.187 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:27:18.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.310+0000 7f6136c1b700 1 Processor -- start 2024-04-06T12:27:18.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.310+0000 7f6136c1b700 1 -- start start 2024-04-06T12:27:18.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.311+0000 7f6136c1b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f613015d7b0 0x7f61300b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:18.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.311+0000 7f6136c1b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f613006a730 0x7f613006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:18.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.311+0000 7f6136c1b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f613015ddf0 0x7f613015e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:18.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.311+0000 7f6136c1b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f613006b0f0 con 0x7f613006a730 2024-04-06T12:27:18.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.311+0000 7f6136c1b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61300b3990 con 0x7f613015d7b0 2024-04-06T12:27:18.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.311+0000 7f6136c1b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61300b3060 con 0x7f613015ddf0 2024-04-06T12:27:18.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.311+0000 7f612ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f613006a730 0x7f613006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:18.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.311+0000 7f612ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f613006a730 0x7f613006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37464/0 (socket says 172.21.15.189:37464) 2024-04-06T12:27:18.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.311+0000 7f612ffff700 1 -- 172.21.15.189:0/665290451 learned_addr learned my addr 172.21.15.189:0/665290451 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:18.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.312+0000 7f61351b8700 1 --2- 172.21.15.189:0/665290451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f613015ddf0 0x7f613015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:18.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.312+0000 7f61349b7700 1 --2- 172.21.15.189:0/665290451 >> v2:172.21.15.192:3300/0 conn(0x7f613015d7b0 0x7f61300b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:18.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.312+0000 7f612ffff700 1 -- 172.21.15.189:0/665290451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f613015ddf0 msgr2=0x7f613015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:18.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.312+0000 7f612ffff700 1 --2- 172.21.15.189:0/665290451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f613015ddf0 0x7f613015e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.312+0000 7f612ffff700 1 -- 172.21.15.189:0/665290451 >> v2:172.21.15.192:3300/0 conn(0x7f613015d7b0 msgr2=0x7f61300b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:18.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.312+0000 7f612ffff700 1 --2- 172.21.15.189:0/665290451 >> v2:172.21.15.192:3300/0 conn(0x7f613015d7b0 0x7f61300b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.312+0000 7f612ffff700 1 -- 172.21.15.189:0/665290451 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61300b3340 con 0x7f613006a730 2024-04-06T12:27:18.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.312+0000 7f612ffff700 1 --2- 172.21.15.189:0/665290451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f613006a730 0x7f613006abb0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f61300b6200 tx=0x7f611c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc404abe90d8caa0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:18.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.312+0000 7f612f7fe700 1 -- 172.21.15.189:0/665290451 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c00b870 con 0x7f613006a730 2024-04-06T12:27:18.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.312+0000 7f612f7fe700 1 -- 172.21.15.189:0/665290451 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f611c00b9d0 con 0x7f613006a730 2024-04-06T12:27:18.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.312+0000 7f612f7fe700 1 -- 172.21.15.189:0/665290451 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c00bd60 con 0x7f613006a730 2024-04-06T12:27:18.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.312+0000 7f6136c1b700 1 -- 172.21.15.189:0/665290451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f613006a730 msgr2=0x7f613006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:18.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.312+0000 7f6136c1b700 1 --2- 172.21.15.189:0/665290451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f613006a730 0x7f613006abb0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f61300b6200 tx=0x7f611c009bd0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.312+0000 7f6136c1b700 1 -- 172.21.15.189:0/665290451 shutdown_connections 2024-04-06T12:27:18.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.312+0000 7f6136c1b700 1 --2- 172.21.15.189:0/665290451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f613015ddf0 0x7f613015e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.312+0000 7f6136c1b700 1 --2- 172.21.15.189:0/665290451 >> v2:172.21.15.192:3300/0 conn(0x7f613015d7b0 0x7f61300b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.312+0000 7f6136c1b700 1 --2- 172.21.15.189:0/665290451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f613006a730 0x7f613006abb0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.313+0000 7f6136c1b700 1 -- 172.21.15.189:0/665290451 >> 172.21.15.189:0/665290451 conn(0x7f61300d00b0 msgr2=0x7f61300d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:18.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.313+0000 7f6136c1b700 1 -- 172.21.15.189:0/665290451 shutdown_connections 2024-04-06T12:27:18.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.313+0000 7f6136c1b700 1 -- 172.21.15.189:0/665290451 wait complete. 2024-04-06T12:27:18.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.314+0000 7f6136c1b700 1 Processor -- start 2024-04-06T12:27:18.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.314+0000 7f6136c1b700 1 -- start start 2024-04-06T12:27:18.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.315+0000 7f6136c1b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f613006a730 0x7f613014fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:18.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.315+0000 7f6136c1b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f613015d7b0 0x7f61300cec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:18.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.315+0000 7f6136c1b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f613015ddf0 0x7f61300cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:18.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.315+0000 7f6136c1b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61300cd6f0 con 0x7f613015ddf0 2024-04-06T12:27:18.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.315+0000 7f6136c1b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6130150210 con 0x7f613015d7b0 2024-04-06T12:27:18.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.315+0000 7f6136c1b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f613015fab0 con 0x7f613006a730 2024-04-06T12:27:18.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.315+0000 7f61351b8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f613015ddf0 0x7f61300cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:18.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.315+0000 7f61351b8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f613015ddf0 0x7f61300cc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37468/0 (socket says 172.21.15.189:37468) 2024-04-06T12:27:18.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.315+0000 7f61351b8700 1 -- 172.21.15.189:0/102009840 learned_addr learned my addr 172.21.15.189:0/102009840 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:18.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.316+0000 7f61349b7700 1 --2- 172.21.15.189:0/102009840 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f613006a730 0x7f613014fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:18.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.316+0000 7f61351b8700 1 -- 172.21.15.189:0/102009840 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f613006a730 msgr2=0x7f613014fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:18.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.316+0000 7f612ffff700 1 --2- 172.21.15.189:0/102009840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f613015d7b0 0x7f61300cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:18.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.316+0000 7f61351b8700 1 --2- 172.21.15.189:0/102009840 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f613006a730 0x7f613014fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.316+0000 7f61351b8700 1 -- 172.21.15.189:0/102009840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f613015d7b0 msgr2=0x7f61300cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:18.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.316+0000 7f61351b8700 1 --2- 172.21.15.189:0/102009840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f613015d7b0 0x7f61300cec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.316+0000 7f61351b8700 1 -- 172.21.15.189:0/102009840 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611c009550 con 0x7f613015ddf0 2024-04-06T12:27:18.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.316+0000 7f61351b8700 1 --2- 172.21.15.189:0/102009840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f613015ddf0 0x7f61300cc890 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f6128007e60 tx=0x7f612800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:18.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.316+0000 7f612d7fa700 1 -- 172.21.15.189:0/102009840 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612800ede0 con 0x7f613015ddf0 2024-04-06T12:27:18.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.316+0000 7f612d7fa700 1 -- 172.21.15.189:0/102009840 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6128014420 con 0x7f613015ddf0 2024-04-06T12:27:18.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.316+0000 7f6136c1b700 1 -- 172.21.15.189:0/102009840 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f613015fd90 con 0x7f613015ddf0 2024-04-06T12:27:18.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.316+0000 7f612d7fa700 1 -- 172.21.15.189:0/102009840 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61280147d0 con 0x7f613015ddf0 2024-04-06T12:27:18.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.316+0000 7f6136c1b700 1 -- 172.21.15.189:0/102009840 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6130160350 con 0x7f613015ddf0 2024-04-06T12:27:18.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.317+0000 7f6136c1b700 1 -- 172.21.15.189:0/102009840 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f613015ab60 con 0x7f613015ddf0 2024-04-06T12:27:18.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.317+0000 7f612d7fa700 1 -- 172.21.15.189:0/102009840 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f6128014930 con 0x7f613015ddf0 2024-04-06T12:27:18.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.318+0000 7f612d7fa700 1 --2- 172.21.15.189:0/102009840 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f611406adb0 0x7f611406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:18.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.318+0000 7f612d7fa700 1 -- 172.21.15.189:0/102009840 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f61280899c0 con 0x7f613015ddf0 2024-04-06T12:27:18.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.318+0000 7f61349b7700 1 --2- 172.21.15.189:0/102009840 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f611406adb0 0x7f611406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:18.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.318+0000 7f61349b7700 1 --2- 172.21.15.189:0/102009840 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f611406adb0 0x7f611406d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6124009950 tx=0x7f6124008040 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:18.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.322+0000 7f612d7fa700 1 -- 172.21.15.189:0/102009840 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f613015ab60 con 0x7f613015ddf0 2024-04-06T12:27:18.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.518+0000 7f6136c1b700 1 -- 172.21.15.189:0/102009840 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f613015ab60 con 0x7f613015ddf0 2024-04-06T12:27:18.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.519+0000 7f612d7fa700 1 -- 172.21.15.189:0/102009840 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+12642 (secure 0 0 0) 0x7f613015ab60 con 0x7f613015ddf0 2024-04-06T12:27:18.519 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:18.519 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":80,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:27:17.734538+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:27:18.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.521+0000 7f6136c1b700 1 -- 172.21.15.189:0/102009840 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f611406adb0 msgr2=0x7f611406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:18.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.521+0000 7f6136c1b700 1 --2- 172.21.15.189:0/102009840 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f611406adb0 0x7f611406d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6124009950 tx=0x7f6124008040 comp rx=0 tx=0).stop 2024-04-06T12:27:18.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.521+0000 7f6136c1b700 1 -- 172.21.15.189:0/102009840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f613015ddf0 msgr2=0x7f61300cc890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:18.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.521+0000 7f6136c1b700 1 --2- 172.21.15.189:0/102009840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f613015ddf0 0x7f61300cc890 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f6128007e60 tx=0x7f612800e3c0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.521+0000 7f6136c1b700 1 -- 172.21.15.189:0/102009840 shutdown_connections 2024-04-06T12:27:18.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.521+0000 7f6136c1b700 1 --2- 172.21.15.189:0/102009840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f613015ddf0 0x7f61300cc890 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.521+0000 7f6136c1b700 1 --2- 172.21.15.189:0/102009840 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f613006a730 0x7f613014fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.521+0000 7f6136c1b700 1 --2- 172.21.15.189:0/102009840 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f611406adb0 0x7f611406d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.521+0000 7f6136c1b700 1 --2- 172.21.15.189:0/102009840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f613015d7b0 0x7f61300cec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.521+0000 7f6136c1b700 1 -- 172.21.15.189:0/102009840 >> 172.21.15.189:0/102009840 conn(0x7f61300d00b0 msgr2=0x7f61300ae210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:18.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.521+0000 7f6136c1b700 1 -- 172.21.15.189:0/102009840 shutdown_connections 2024-04-06T12:27:18.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.521+0000 7f6136c1b700 1 -- 172.21.15.189:0/102009840 wait complete. 2024-04-06T12:27:18.532 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:27:18.532 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:27:18.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.661+0000 7fbb13d9b700 1 Processor -- start 2024-04-06T12:27:18.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.661+0000 7fbb13d9b700 1 -- start start 2024-04-06T12:27:18.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.662+0000 7fbb13d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbb0c15d7b0 0x7fbb0c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:18.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.662+0000 7fbb13d9b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbb0c15e500 0x7fbb0c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:18.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.662+0000 7fbb13d9b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0c0b3990 con 0x7fbb0c06a730 2024-04-06T12:27:18.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.662+0000 7fbb13d9b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb0c15e950 con 0x7fbb0c15d7b0 2024-04-06T12:27:18.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.662+0000 7fbb13d9b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb0c06b0f0 con 0x7fbb0c15e500 2024-04-06T12:27:18.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.662+0000 7fbb11b37700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbb0c15d7b0 0x7fbb0c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:18.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.662+0000 7fbb11336700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbb0c06a730 0x7fbb0c15ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37530/0 (socket says 172.21.15.189:37530) 2024-04-06T12:27:18.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.662+0000 7fbb11336700 1 -- 172.21.15.189:0/4031077360 learned_addr learned my addr 172.21.15.189:0/4031077360 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:18.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.662+0000 7fbb11b37700 1 -- 172.21.15.189:0/4031077360 >> v2:172.21.15.192:3300/0 conn(0x7fbb0c15e500 msgr2=0x7fbb0c06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:18.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.662+0000 7fbb11b37700 1 --2- 172.21.15.189:0/4031077360 >> v2:172.21.15.192:3300/0 conn(0x7fbb0c15e500 0x7fbb0c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.662+0000 7fbb11b37700 1 -- 172.21.15.189:0/4031077360 >> v1:172.21.15.189:6789/0 conn(0x7fbb0c06a730 legacy=0x7fbb0c15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:18.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.662+0000 7fbb11b37700 1 -- 172.21.15.189:0/4031077360 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0c0b3060 con 0x7fbb0c15d7b0 2024-04-06T12:27:18.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.662+0000 7fbb10b35700 1 -- 172.21.15.189:0/4031077360 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652339069 0 0) 0x7fbb0c0b3990 con 0x7fbb0c06a730 2024-04-06T12:27:18.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.662+0000 7fbb11b37700 1 --2- 172.21.15.189:0/4031077360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb0c15d7b0 0x7fbb0c0b3e20 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c0673e0 tx=0x7fbb08009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf58caf9bf8c726 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:18.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.662+0000 7fbb10b35700 1 -- 172.21.15.189:0/4031077360 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb08011830 con 0x7fbb0c15d7b0 2024-04-06T12:27:18.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.663+0000 7fbb10b35700 1 -- 172.21.15.189:0/4031077360 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb08011990 con 0x7fbb0c15d7b0 2024-04-06T12:27:18.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.663+0000 7fbb10b35700 1 -- 172.21.15.189:0/4031077360 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb08011d20 con 0x7fbb0c15d7b0 2024-04-06T12:27:18.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.663+0000 7fbb13d9b700 1 -- 172.21.15.189:0/4031077360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb0c15d7b0 msgr2=0x7fbb0c0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:18.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.663+0000 7fbb13d9b700 1 --2- 172.21.15.189:0/4031077360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb0c15d7b0 0x7fbb0c0b3e20 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c0673e0 tx=0x7fbb08009b20 comp rx=0 tx=0).stop 2024-04-06T12:27:18.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.663+0000 7fbb13d9b700 1 -- 172.21.15.189:0/4031077360 shutdown_connections 2024-04-06T12:27:18.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.663+0000 7fbb13d9b700 1 --2- 172.21.15.189:0/4031077360 >> v2:172.21.15.192:3300/0 conn(0x7fbb0c15e500 0x7fbb0c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.663+0000 7fbb13d9b700 1 --2- 172.21.15.189:0/4031077360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb0c15d7b0 0x7fbb0c0b3e20 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.663+0000 7fbb13d9b700 1 -- 172.21.15.189:0/4031077360 >> 172.21.15.189:0/4031077360 conn(0x7fbb0c0d00b0 msgr2=0x7fbb0c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:18.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.663+0000 7fbb13d9b700 1 -- 172.21.15.189:0/4031077360 shutdown_connections 2024-04-06T12:27:18.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.663+0000 7fbb13d9b700 1 -- 172.21.15.189:0/4031077360 wait complete. 2024-04-06T12:27:18.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.664+0000 7fbb13d9b700 1 Processor -- start 2024-04-06T12:27:18.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.664+0000 7fbb13d9b700 1 -- start start 2024-04-06T12:27:18.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.665+0000 7fbb13d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb0c06a730 0x7fbb0c1f5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:18.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.665+0000 7fbb13d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb0c15d7b0 0x7fbb0c15b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:18.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.665+0000 7fbb13d9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb0c15e500 0x7fbb0c0b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:18.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.665+0000 7fbb13d9b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c0cf4d0 con 0x7fbb0c15d7b0 2024-04-06T12:27:18.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.665+0000 7fbb13d9b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c15bd50 con 0x7fbb0c06a730 2024-04-06T12:27:18.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.665+0000 7fbb13d9b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c15c030 con 0x7fbb0c15e500 2024-04-06T12:27:18.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.665+0000 7fbb11336700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb0c15d7b0 0x7fbb0c15b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:18.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.665+0000 7fbb11336700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb0c15d7b0 0x7fbb0c15b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37488/0 (socket says 172.21.15.189:37488) 2024-04-06T12:27:18.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.665+0000 7fbb11336700 1 -- 172.21.15.189:0/2853232847 learned_addr learned my addr 172.21.15.189:0/2853232847 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:18.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.665+0000 7fbb11336700 1 -- 172.21.15.189:0/2853232847 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb0c15e500 msgr2=0x7fbb0c0b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:18.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.665+0000 7fbb11b37700 1 --2- 172.21.15.189:0/2853232847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb0c06a730 0x7fbb0c1f5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:18.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.665+0000 7fbb11336700 1 --2- 172.21.15.189:0/2853232847 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb0c15e500 0x7fbb0c0b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.665+0000 7fbb11336700 1 -- 172.21.15.189:0/2853232847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb0c06a730 msgr2=0x7fbb0c1f5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:18.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.665+0000 7fbb11336700 1 --2- 172.21.15.189:0/2853232847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb0c06a730 0x7fbb0c1f5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.665+0000 7fbb11336700 1 -- 172.21.15.189:0/2853232847 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb080094a0 con 0x7fbb0c15d7b0 2024-04-06T12:27:18.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.665+0000 7fbb11336700 1 --2- 172.21.15.189:0/2853232847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb0c15d7b0 0x7fbb0c15b150 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fbafc0011e0 tx=0x7fbafc00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:18.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.666+0000 7fbb027fc700 1 -- 172.21.15.189:0/2853232847 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc004c80 con 0x7fbb0c15d7b0 2024-04-06T12:27:18.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.666+0000 7fbb027fc700 1 -- 172.21.15.189:0/2853232847 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbafc004e00 con 0x7fbb0c15d7b0 2024-04-06T12:27:18.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.666+0000 7fbb13d9b700 1 -- 172.21.15.189:0/2853232847 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb0c15c310 con 0x7fbb0c15d7b0 2024-04-06T12:27:18.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.666+0000 7fbb027fc700 1 -- 172.21.15.189:0/2853232847 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc007310 con 0x7fbb0c15d7b0 2024-04-06T12:27:18.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.666+0000 7fbb13d9b700 1 -- 172.21.15.189:0/2853232847 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb0c15c8d0 con 0x7fbb0c15d7b0 2024-04-06T12:27:18.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.667+0000 7fbb13d9b700 1 -- 172.21.15.189:0/2853232847 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb0c15caf0 con 0x7fbb0c15d7b0 2024-04-06T12:27:18.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.667+0000 7fbb027fc700 1 -- 172.21.15.189:0/2853232847 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fbafc007470 con 0x7fbb0c15d7b0 2024-04-06T12:27:18.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.667+0000 7fbb027fc700 1 --2- 172.21.15.189:0/2853232847 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbaf006adb0 0x7fbaf006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:18.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.667+0000 7fbb027fc700 1 -- 172.21.15.189:0/2853232847 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fbafc089010 con 0x7fbb0c15d7b0 2024-04-06T12:27:18.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.667+0000 7fbb11b37700 1 --2- 172.21.15.189:0/2853232847 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbaf006adb0 0x7fbaf006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:18.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.668+0000 7fbb11b37700 1 --2- 172.21.15.189:0/2853232847 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbaf006adb0 0x7fbaf006d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbb080090b0 tx=0x7fbb08008030 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:18.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.672+0000 7fbb027fc700 1 -- 172.21.15.189:0/2853232847 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbafc0542f0 con 0x7fbb0c15d7b0 2024-04-06T12:27:18.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.864+0000 7fbb13d9b700 1 -- 172.21.15.189:0/2853232847 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbb0c067bb0 con 0x7fbb0c15d7b0 2024-04-06T12:27:18.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.864+0000 7fbb027fc700 1 -- 172.21.15.189:0/2853232847 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbafc057910 con 0x7fbb0c15d7b0 2024-04-06T12:27:18.863 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:27:18.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.867+0000 7fbb13d9b700 1 -- 172.21.15.189:0/2853232847 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbaf006adb0 msgr2=0x7fbaf006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:18.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.867+0000 7fbb13d9b700 1 --2- 172.21.15.189:0/2853232847 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbaf006adb0 0x7fbaf006d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbb080090b0 tx=0x7fbb08008030 comp rx=0 tx=0).stop 2024-04-06T12:27:18.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.867+0000 7fbb13d9b700 1 -- 172.21.15.189:0/2853232847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb0c15d7b0 msgr2=0x7fbb0c15b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:18.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.867+0000 7fbb13d9b700 1 --2- 172.21.15.189:0/2853232847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb0c15d7b0 0x7fbb0c15b150 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fbafc0011e0 tx=0x7fbafc00cae0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.867+0000 7fbb13d9b700 1 -- 172.21.15.189:0/2853232847 shutdown_connections 2024-04-06T12:27:18.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.867+0000 7fbb13d9b700 1 --2- 172.21.15.189:0/2853232847 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb0c15e500 0x7fbb0c0b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.867+0000 7fbb13d9b700 1 --2- 172.21.15.189:0/2853232847 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbaf006adb0 0x7fbaf006d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.867+0000 7fbb13d9b700 1 --2- 172.21.15.189:0/2853232847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb0c06a730 0x7fbb0c1f5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.867+0000 7fbb13d9b700 1 --2- 172.21.15.189:0/2853232847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb0c15d7b0 0x7fbb0c15b150 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:18.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.867+0000 7fbb13d9b700 1 -- 172.21.15.189:0/2853232847 >> 172.21.15.189:0/2853232847 conn(0x7fbb0c0d00b0 msgr2=0x7fbb0c071ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:18.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.867+0000 7fbb13d9b700 1 -- 172.21.15.189:0/2853232847 shutdown_connections 2024-04-06T12:27:18.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:18.867+0000 7fbb13d9b700 1 -- 172.21.15.189:0/2853232847 wait complete. 2024-04-06T12:27:18.878 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-06T12:27:19.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.002+0000 7fd1d2a2d700 1 Processor -- start 2024-04-06T12:27:19.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.002+0000 7fd1d2a2d700 1 -- start start 2024-04-06T12:27:19.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.003+0000 7fd1d2a2d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd1cc0cb870 0x7fd1cc15c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:19.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.003+0000 7fd1d2a2d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd1cc0cfa00 0x7fd1cc0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:19.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.003+0000 7fd1d2a2d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc0cbd80 con 0x7fd1cc0d11b0 2024-04-06T12:27:19.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.003+0000 7fd1d2a2d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1cc0ccdc0 con 0x7fd1cc0cb870 2024-04-06T12:27:19.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.003+0000 7fd1d2a2d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1cc0d0b50 con 0x7fd1cc0cfa00 2024-04-06T12:27:19.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.003+0000 7fd1cb7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd1cc0cb870 0x7fd1cc15c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:19.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.003+0000 7fd1cb7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd1cc0cb870 0x7fd1cc15c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37502/0 (socket says 172.21.15.189:37502) 2024-04-06T12:27:19.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.003+0000 7fd1cb7fe700 1 -- 172.21.15.189:0/2734594753 learned_addr learned my addr 172.21.15.189:0/2734594753 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:19.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.003+0000 7fd1d0fca700 1 --2- 172.21.15.189:0/2734594753 >> v2:172.21.15.192:3300/0 conn(0x7fd1cc0cfa00 0x7fd1cc0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:19.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.003+0000 7fd1cb7fe700 1 -- 172.21.15.189:0/2734594753 >> v2:172.21.15.192:3300/0 conn(0x7fd1cc0cfa00 msgr2=0x7fd1cc0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:19.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.003+0000 7fd1cb7fe700 1 --2- 172.21.15.189:0/2734594753 >> v2:172.21.15.192:3300/0 conn(0x7fd1cc0cfa00 0x7fd1cc0cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.003+0000 7fd1cb7fe700 1 -- 172.21.15.189:0/2734594753 >> v1:172.21.15.192:6789/0 conn(0x7fd1cc0d11b0 legacy=0x7fd1cc0d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:19.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.003+0000 7fd1cb7fe700 1 -- 172.21.15.189:0/2734594753 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1cc0cef20 con 0x7fd1cc0cb870 2024-04-06T12:27:19.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.003+0000 7fd1cb7fe700 1 --2- 172.21.15.189:0/2734594753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1cc0cb870 0x7fd1cc15c280 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc15ff70 tx=0x7fd1bc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f927ba1659b91fb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:19.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.004+0000 7fd1caffd700 1 -- 172.21.15.189:0/2734594753 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc011830 con 0x7fd1cc0cb870 2024-04-06T12:27:19.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.004+0000 7fd1caffd700 1 -- 172.21.15.189:0/2734594753 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1bc011990 con 0x7fd1cc0cb870 2024-04-06T12:27:19.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.004+0000 7fd1caffd700 1 -- 172.21.15.189:0/2734594753 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc011d20 con 0x7fd1cc0cb870 2024-04-06T12:27:19.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.004+0000 7fd1d2a2d700 1 -- 172.21.15.189:0/2734594753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1cc0cb870 msgr2=0x7fd1cc15c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:19.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.004+0000 7fd1d2a2d700 1 --2- 172.21.15.189:0/2734594753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1cc0cb870 0x7fd1cc15c280 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc15ff70 tx=0x7fd1bc009bd0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.004+0000 7fd1d2a2d700 1 -- 172.21.15.189:0/2734594753 shutdown_connections 2024-04-06T12:27:19.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.004+0000 7fd1d2a2d700 1 --2- 172.21.15.189:0/2734594753 >> v2:172.21.15.192:3300/0 conn(0x7fd1cc0cfa00 0x7fd1cc0cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.004+0000 7fd1d2a2d700 1 --2- 172.21.15.189:0/2734594753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1cc0cb870 0x7fd1cc15c280 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.004+0000 7fd1d2a2d700 1 -- 172.21.15.189:0/2734594753 >> 172.21.15.189:0/2734594753 conn(0x7fd1cc0b2db0 msgr2=0x7fd1cc0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:19.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.004+0000 7fd1d2a2d700 1 -- 172.21.15.189:0/2734594753 shutdown_connections 2024-04-06T12:27:19.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.004+0000 7fd1d2a2d700 1 -- 172.21.15.189:0/2734594753 wait complete. 2024-04-06T12:27:19.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.005+0000 7fd1d2a2d700 1 Processor -- start 2024-04-06T12:27:19.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.006+0000 7fd1d2a2d700 1 -- start start 2024-04-06T12:27:19.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.006+0000 7fd1d2a2d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1cc0cb870 0x7fd1cc14c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:19.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.006+0000 7fd1d2a2d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1cc0cfa00 0x7fd1cc0ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:19.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.006+0000 7fd1d2a2d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1cc0d11b0 0x7fd1cc0caa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:19.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.006+0000 7fd1d2a2d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc06b8e0 con 0x7fd1cc0cb870 2024-04-06T12:27:19.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.006+0000 7fd1d2a2d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc06bbc0 con 0x7fd1cc0cfa00 2024-04-06T12:27:19.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.006+0000 7fd1d2a2d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc06bea0 con 0x7fd1cc0d11b0 2024-04-06T12:27:19.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.006+0000 7fd1cbfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1cc0cb870 0x7fd1cc14c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:19.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.006+0000 7fd1cbfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1cc0cb870 0x7fd1cc14c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37512/0 (socket says 172.21.15.189:37512) 2024-04-06T12:27:19.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.006+0000 7fd1cbfff700 1 -- 172.21.15.189:0/2442083272 learned_addr learned my addr 172.21.15.189:0/2442083272 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:19.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.007+0000 7fd1cbfff700 1 -- 172.21.15.189:0/2442083272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1cc0d11b0 msgr2=0x7fd1cc0caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:19.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.007+0000 7fd1cbfff700 1 --2- 172.21.15.189:0/2442083272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1cc0d11b0 0x7fd1cc0caa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.007+0000 7fd1cbfff700 1 -- 172.21.15.189:0/2442083272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1cc0cfa00 msgr2=0x7fd1cc0ca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:19.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.007+0000 7fd1cbfff700 1 --2- 172.21.15.189:0/2442083272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1cc0cfa00 0x7fd1cc0ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.007+0000 7fd1cbfff700 1 -- 172.21.15.189:0/2442083272 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc009550 con 0x7fd1cc0cb870 2024-04-06T12:27:19.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.007+0000 7fd1cbfff700 1 --2- 172.21.15.189:0/2442083272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1cc0cb870 0x7fd1cc14c200 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd1c000c4e0 tx=0x7fd1c000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:19.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.007+0000 7fd1c8ff9700 1 -- 172.21.15.189:0/2442083272 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c00052d0 con 0x7fd1cc0cb870 2024-04-06T12:27:19.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.007+0000 7fd1c8ff9700 1 -- 172.21.15.189:0/2442083272 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c0005430 con 0x7fd1cc0cb870 2024-04-06T12:27:19.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.007+0000 7fd1d2a2d700 1 -- 172.21.15.189:0/2442083272 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1cc06c180 con 0x7fd1cc0cb870 2024-04-06T12:27:19.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.007+0000 7fd1c8ff9700 1 -- 172.21.15.189:0/2442083272 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c00038c0 con 0x7fd1cc0cb870 2024-04-06T12:27:19.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.007+0000 7fd1d2a2d700 1 -- 172.21.15.189:0/2442083272 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1cc06cd20 con 0x7fd1cc0cb870 2024-04-06T12:27:19.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.008+0000 7fd1d2a2d700 1 -- 172.21.15.189:0/2442083272 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1cc06c6a0 con 0x7fd1cc0cb870 2024-04-06T12:27:19.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.008+0000 7fd1c8ff9700 1 -- 172.21.15.189:0/2442083272 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fd1c0003a20 con 0x7fd1cc0cb870 2024-04-06T12:27:19.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.009+0000 7fd1c8ff9700 1 --2- 172.21.15.189:0/2442083272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1b806adb0 0x7fd1b806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:19.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.009+0000 7fd1cb7fe700 1 --2- 172.21.15.189:0/2442083272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1b806adb0 0x7fd1b806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:19.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.009+0000 7fd1c8ff9700 1 -- 172.21.15.189:0/2442083272 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fd1c0088680 con 0x7fd1cc0cb870 2024-04-06T12:27:19.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.009+0000 7fd1cb7fe700 1 --2- 172.21.15.189:0/2442083272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1b806adb0 0x7fd1b806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc0cbcb0 tx=0x7fd1bc010000 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:19.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.013+0000 7fd1c8ff9700 1 -- 172.21.15.189:0/2442083272 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd1cc06c6a0 con 0x7fd1cc0cb870 2024-04-06T12:27:19.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.208+0000 7fd1d2a2d700 1 -- 172.21.15.189:0/2442083272 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd1cc06c6a0 con 0x7fd1cc0cb870 2024-04-06T12:27:19.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.208+0000 7fd1c8ff9700 1 -- 172.21.15.189:0/2442083272 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v80) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd1c0017060 con 0x7fd1cc0cb870 2024-04-06T12:27:19.207 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:19.207 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T12:27:19.207 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:19.207 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 1, 2024-04-06T12:27:19.207 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": ".mgr" 2024-04-06T12:27:19.207 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:19.207 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T12:27:19.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.210+0000 7fd1d2a2d700 1 -- 172.21.15.189:0/2442083272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1b806adb0 msgr2=0x7fd1b806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:19.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.210+0000 7fd1d2a2d700 1 --2- 172.21.15.189:0/2442083272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1b806adb0 0x7fd1b806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc0cbcb0 tx=0x7fd1bc010000 comp rx=0 tx=0).stop 2024-04-06T12:27:19.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.210+0000 7fd1d2a2d700 1 -- 172.21.15.189:0/2442083272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1cc0cb870 msgr2=0x7fd1cc14c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:19.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.211+0000 7fd1d2a2d700 1 --2- 172.21.15.189:0/2442083272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1cc0cb870 0x7fd1cc14c200 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd1c000c4e0 tx=0x7fd1c000c8a0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.211+0000 7fd1d2a2d700 1 -- 172.21.15.189:0/2442083272 shutdown_connections 2024-04-06T12:27:19.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.211+0000 7fd1d2a2d700 1 --2- 172.21.15.189:0/2442083272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1cc0d11b0 0x7fd1cc0caa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.211+0000 7fd1d2a2d700 1 --2- 172.21.15.189:0/2442083272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1cc0cb870 0x7fd1cc14c200 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.211+0000 7fd1d2a2d700 1 --2- 172.21.15.189:0/2442083272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1b806adb0 0x7fd1b806d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.211+0000 7fd1d2a2d700 1 --2- 172.21.15.189:0/2442083272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1cc0cfa00 0x7fd1cc0ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.211+0000 7fd1d2a2d700 1 -- 172.21.15.189:0/2442083272 >> 172.21.15.189:0/2442083272 conn(0x7fd1cc0b2db0 msgr2=0x7fd1cc0b25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:19.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.211+0000 7fd1d2a2d700 1 -- 172.21.15.189:0/2442083272 shutdown_connections 2024-04-06T12:27:19.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.211+0000 7fd1d2a2d700 1 -- 172.21.15.189:0/2442083272 wait complete. 2024-04-06T12:27:19.221 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-06T12:27:19.221 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-06T12:27:19.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.349+0000 7f6cdfdcb700 1 Processor -- start 2024-04-06T12:27:19.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.350+0000 7f6cdfdcb700 1 -- start start 2024-04-06T12:27:19.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.350+0000 7f6cdfdcb700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6cd80b2ca0 0x7f6cd80b3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:19.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.350+0000 7f6cdfdcb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd80cfa50 con 0x7f6cd8161fa0 2024-04-06T12:27:19.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.350+0000 7f6cdfdcb700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd80b35c0 con 0x7f6cd815ea40 2024-04-06T12:27:19.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.350+0000 7f6cdfdcb700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd815dd70 con 0x7f6cd80b2ca0 2024-04-06T12:27:19.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.351+0000 7f6cde368700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6cd8161fa0 0x7f6cd80d0e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56848/0 (socket says 172.21.15.189:56848) 2024-04-06T12:27:19.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.351+0000 7f6cde368700 1 -- 172.21.15.189:0/608970177 learned_addr learned my addr 172.21.15.189:0/608970177 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:19.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.351+0000 7f6cddb67700 1 --2- 172.21.15.189:0/608970177 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6cd80b2ca0 0x7f6cd80b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:19.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.351+0000 7f6cdcb65700 1 -- 172.21.15.189:0/608970177 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744187245 0 0) 0x7f6cd80cfa50 con 0x7f6cd8161fa0 2024-04-06T12:27:19.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.351+0000 7f6cdcb65700 1 -- 172.21.15.189:0/608970177 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc40035b0 con 0x7f6cd8161fa0 2024-04-06T12:27:19.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.351+0000 7f6cdcb65700 1 -- 172.21.15.189:0/608970177 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671401764 0 0) 0x7f6cd80b35c0 con 0x7f6cd815ea40 2024-04-06T12:27:19.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.351+0000 7f6cdcb65700 1 -- 172.21.15.189:0/608970177 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cd80cfa50 con 0x7f6cd815ea40 2024-04-06T12:27:19.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.351+0000 7f6cddb67700 1 -- 172.21.15.189:0/608970177 >> v1:172.21.15.192:6789/0 conn(0x7f6cd815ea40 legacy=0x7f6cd80cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:19.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.351+0000 7f6cddb67700 1 -- 172.21.15.189:0/608970177 >> v1:172.21.15.189:6789/0 conn(0x7f6cd8161fa0 legacy=0x7f6cd80d0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:19.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.351+0000 7f6cddb67700 1 -- 172.21.15.189:0/608970177 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd80cf4c0 con 0x7f6cd80b2ca0 2024-04-06T12:27:19.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.351+0000 7f6cddb67700 1 --2- 172.21.15.189:0/608970177 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6cd80b2ca0 0x7f6cd80b3080 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4009870 tx=0x7f6cd4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d04d575dca27c1f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:19.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.351+0000 7f6cdcb65700 1 -- 172.21.15.189:0/608970177 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4011830 con 0x7f6cd80b2ca0 2024-04-06T12:27:19.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.351+0000 7f6cdcb65700 1 -- 172.21.15.189:0/608970177 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd4011990 con 0x7f6cd80b2ca0 2024-04-06T12:27:19.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.352+0000 7f6cdcb65700 1 -- 172.21.15.189:0/608970177 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4011d20 con 0x7f6cd80b2ca0 2024-04-06T12:27:19.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.352+0000 7f6cdfdcb700 1 -- 172.21.15.189:0/608970177 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6cd80b2ca0 msgr2=0x7f6cd80b3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:19.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.352+0000 7f6cdfdcb700 1 --2- 172.21.15.189:0/608970177 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6cd80b2ca0 0x7f6cd80b3080 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4009870 tx=0x7f6cd4009b80 comp rx=0 tx=0).stop 2024-04-06T12:27:19.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.352+0000 7f6cdfdcb700 1 -- 172.21.15.189:0/608970177 shutdown_connections 2024-04-06T12:27:19.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.352+0000 7f6cdfdcb700 1 --2- 172.21.15.189:0/608970177 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6cd80b2ca0 0x7f6cd80b3080 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.352+0000 7f6cdfdcb700 1 -- 172.21.15.189:0/608970177 >> 172.21.15.189:0/608970177 conn(0x7f6cd806eaa0 msgr2=0x7f6cd806ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:19.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.352+0000 7f6cdfdcb700 1 -- 172.21.15.189:0/608970177 shutdown_connections 2024-04-06T12:27:19.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.352+0000 7f6cdfdcb700 1 -- 172.21.15.189:0/608970177 wait complete. 2024-04-06T12:27:19.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.353+0000 7f6cdfdcb700 1 Processor -- start 2024-04-06T12:27:19.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.353+0000 7f6cdfdcb700 1 -- start start 2024-04-06T12:27:19.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.354+0000 7f6cdfdcb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cd80b2ca0 0x7f6cd8157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:19.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.354+0000 7f6cdfdcb700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6cd815ea40 0x7f6cd8070390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:19.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.354+0000 7f6cdfdcb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6cd8161fa0 0x7f6cd80708d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:19.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.354+0000 7f6cdfdcb700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd81f3410 con 0x7f6cd80b2ca0 2024-04-06T12:27:19.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.354+0000 7f6cdfdcb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd80c5d20 con 0x7f6cd8161fa0 2024-04-06T12:27:19.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.354+0000 7f6cdfdcb700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd80c5a40 con 0x7f6cd815ea40 2024-04-06T12:27:19.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.354+0000 7f6cddb67700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cd80b2ca0 0x7f6cd8157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:19.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.354+0000 7f6cddb67700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cd80b2ca0 0x7f6cd8157500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37456/0 (socket says 172.21.15.189:37456) 2024-04-06T12:27:19.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.354+0000 7f6cddb67700 1 -- 172.21.15.189:0/717093067 learned_addr learned my addr 172.21.15.189:0/717093067 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:19.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.354+0000 7f6cde368700 1 --2- 172.21.15.189:0/717093067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6cd8161fa0 0x7f6cd80708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:19.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.354+0000 7f6cdd366700 1 --2- 172.21.15.189:0/717093067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6cd815ea40 0x7f6cd8070390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:19.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.354+0000 7f6cddb67700 1 -- 172.21.15.189:0/717093067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6cd815ea40 msgr2=0x7f6cd8070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:19.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.354+0000 7f6cddb67700 1 --2- 172.21.15.189:0/717093067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6cd815ea40 0x7f6cd8070390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.354+0000 7f6cddb67700 1 -- 172.21.15.189:0/717093067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6cd8161fa0 msgr2=0x7f6cd80708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:19.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.354+0000 7f6cddb67700 1 --2- 172.21.15.189:0/717093067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6cd8161fa0 0x7f6cd80708d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.355+0000 7f6cddb67700 1 -- 172.21.15.189:0/717093067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd40094a0 con 0x7f6cd80b2ca0 2024-04-06T12:27:19.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.355+0000 7f6cddb67700 1 --2- 172.21.15.189:0/717093067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cd80b2ca0 0x7f6cd8157500 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4014040 tx=0x7f6cd4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:19.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.355+0000 7f6cce7fc700 1 -- 172.21.15.189:0/717093067 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd401a860 con 0x7f6cd80b2ca0 2024-04-06T12:27:19.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.355+0000 7f6cce7fc700 1 -- 172.21.15.189:0/717093067 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd401a9c0 con 0x7f6cd80b2ca0 2024-04-06T12:27:19.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.355+0000 7f6cdfdcb700 1 -- 172.21.15.189:0/717093067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd80c6240 con 0x7f6cd80b2ca0 2024-04-06T12:27:19.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.355+0000 7f6cce7fc700 1 -- 172.21.15.189:0/717093067 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd401ad30 con 0x7f6cd80b2ca0 2024-04-06T12:27:19.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.355+0000 7f6cdfdcb700 1 -- 172.21.15.189:0/717093067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd80c6800 con 0x7f6cd80b2ca0 2024-04-06T12:27:19.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.356+0000 7f6cdfdcb700 1 -- 172.21.15.189:0/717093067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd8067bf0 con 0x7f6cd80b2ca0 2024-04-06T12:27:19.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.356+0000 7f6cce7fc700 1 -- 172.21.15.189:0/717093067 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f6cd4022070 con 0x7f6cd80b2ca0 2024-04-06T12:27:19.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.357+0000 7f6cce7fc700 1 --2- 172.21.15.189:0/717093067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6cbc06adb0 0x7f6cbc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:19.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.357+0000 7f6cdd366700 1 --2- 172.21.15.189:0/717093067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6cbc06adb0 0x7f6cbc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:19.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.357+0000 7f6cce7fc700 1 -- 172.21.15.189:0/717093067 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f6cd408a3b0 con 0x7f6cd80b2ca0 2024-04-06T12:27:19.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.357+0000 7f6cdd366700 1 --2- 172.21.15.189:0/717093067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6cbc06adb0 0x7f6cbc06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6cc80010e0 tx=0x7f6cc8009040 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:19.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.361+0000 7f6cce7fc700 1 -- 172.21.15.189:0/717093067 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6cd4055680 con 0x7f6cd80b2ca0 2024-04-06T12:27:19.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.555+0000 7f6cdfdcb700 1 -- 172.21.15.189:0/717093067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f6cd80c60a0 con 0x7f6cd80b2ca0 2024-04-06T12:27:19.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.616+0000 7f6cce7fc700 1 -- 172.21.15.189:0/717093067 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v81) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6cd80c60a0 con 0x7f6cd80b2ca0 2024-04-06T12:27:19.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.618+0000 7f6cdfdcb700 1 -- 172.21.15.189:0/717093067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6cbc06adb0 msgr2=0x7f6cbc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:19.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.618+0000 7f6cdfdcb700 1 --2- 172.21.15.189:0/717093067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6cbc06adb0 0x7f6cbc06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6cc80010e0 tx=0x7f6cc8009040 comp rx=0 tx=0).stop 2024-04-06T12:27:19.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.618+0000 7f6cdfdcb700 1 -- 172.21.15.189:0/717093067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cd80b2ca0 msgr2=0x7f6cd8157500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:19.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.618+0000 7f6cdfdcb700 1 --2- 172.21.15.189:0/717093067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cd80b2ca0 0x7f6cd8157500 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4014040 tx=0x7f6cd4004750 comp rx=0 tx=0).stop 2024-04-06T12:27:19.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.618+0000 7f6cdfdcb700 1 -- 172.21.15.189:0/717093067 shutdown_connections 2024-04-06T12:27:19.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.618+0000 7f6cdfdcb700 1 --2- 172.21.15.189:0/717093067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6cbc06adb0 0x7f6cbc06d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.618+0000 7f6cdfdcb700 1 --2- 172.21.15.189:0/717093067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6cd8161fa0 0x7f6cd80708d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.618+0000 7f6cdfdcb700 1 --2- 172.21.15.189:0/717093067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cd80b2ca0 0x7f6cd8157500 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.618+0000 7f6cdfdcb700 1 --2- 172.21.15.189:0/717093067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6cd815ea40 0x7f6cd8070390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.618+0000 7f6cdfdcb700 1 -- 172.21.15.189:0/717093067 >> 172.21.15.189:0/717093067 conn(0x7f6cd806eaa0 msgr2=0x7f6cd80d0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:19.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.618+0000 7f6cdfdcb700 1 -- 172.21.15.189:0/717093067 shutdown_connections 2024-04-06T12:27:19.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.618+0000 7f6cdfdcb700 1 -- 172.21.15.189:0/717093067 wait complete. 2024-04-06T12:27:19.618 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' created 2024-04-06T12:27:19.632 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-06T12:27:19.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.763+0000 7f2cfd529700 1 Processor -- start 2024-04-06T12:27:19.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.763+0000 7f2cfd529700 1 -- start start 2024-04-06T12:27:19.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.764+0000 7f2cfd529700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2cf815d7b0 0x7f2cf80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:19.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.764+0000 7f2cfd529700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cf815e500 0x7f2cf806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:19.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.764+0000 7f2cfd529700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf80b3990 con 0x7f2cf806a730 2024-04-06T12:27:19.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.764+0000 7f2cfd529700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cf806b0f0 con 0x7f2cf815d7b0 2024-04-06T12:27:19.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.764+0000 7f2cfd529700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf80b3060 con 0x7f2cf815e500 2024-04-06T12:27:19.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.764+0000 7f2cf6ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2cf815d7b0 0x7f2cf80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:19.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.764+0000 7f2cf6ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2cf815d7b0 0x7f2cf80b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37470/0 (socket says 172.21.15.189:37470) 2024-04-06T12:27:19.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.764+0000 7f2cf6ffd700 1 -- 172.21.15.189:0/1901204608 learned_addr learned my addr 172.21.15.189:0/1901204608 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:19.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.764+0000 7f2cf77fe700 1 --2- 172.21.15.189:0/1901204608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cf815e500 0x7f2cf806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:19.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.764+0000 7f2cf6ffd700 1 -- 172.21.15.189:0/1901204608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cf815e500 msgr2=0x7f2cf806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:19.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.764+0000 7f2cf6ffd700 1 --2- 172.21.15.189:0/1901204608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cf815e500 0x7f2cf806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.764+0000 7f2cf6ffd700 1 -- 172.21.15.189:0/1901204608 >> v1:172.21.15.192:6789/0 conn(0x7f2cf806a730 legacy=0x7f2cf815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:19.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.765+0000 7f2cf6ffd700 1 -- 172.21.15.189:0/1901204608 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf80b3340 con 0x7f2cf815d7b0 2024-04-06T12:27:19.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.765+0000 7f2cf6ffd700 1 --2- 172.21.15.189:0/1901204608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cf815d7b0 0x7f2cf80b3e20 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009e50 tx=0x7f2cf000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff25a00c81ca80f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:19.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.765+0000 7f2cf5ffb700 1 -- 172.21.15.189:0/1901204608 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf000e070 con 0x7f2cf815d7b0 2024-04-06T12:27:19.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.765+0000 7f2cf5ffb700 1 -- 172.21.15.189:0/1901204608 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf0003a20 con 0x7f2cf815d7b0 2024-04-06T12:27:19.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.765+0000 7f2cf5ffb700 1 -- 172.21.15.189:0/1901204608 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0003dd0 con 0x7f2cf815d7b0 2024-04-06T12:27:19.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.765+0000 7f2cfd529700 1 -- 172.21.15.189:0/1901204608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cf815d7b0 msgr2=0x7f2cf80b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:19.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.765+0000 7f2cfd529700 1 --2- 172.21.15.189:0/1901204608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cf815d7b0 0x7f2cf80b3e20 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009e50 tx=0x7f2cf000b330 comp rx=0 tx=0).stop 2024-04-06T12:27:19.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.765+0000 7f2cfd529700 1 -- 172.21.15.189:0/1901204608 shutdown_connections 2024-04-06T12:27:19.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.765+0000 7f2cfd529700 1 --2- 172.21.15.189:0/1901204608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cf815e500 0x7f2cf806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.765+0000 7f2cfd529700 1 --2- 172.21.15.189:0/1901204608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cf815d7b0 0x7f2cf80b3e20 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.765+0000 7f2cfd529700 1 -- 172.21.15.189:0/1901204608 >> 172.21.15.189:0/1901204608 conn(0x7f2cf80d00b0 msgr2=0x7f2cf80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:19.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.765+0000 7f2cfd529700 1 -- 172.21.15.189:0/1901204608 shutdown_connections 2024-04-06T12:27:19.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.765+0000 7f2cfd529700 1 -- 172.21.15.189:0/1901204608 wait complete. 2024-04-06T12:27:19.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.767+0000 7f2cfd529700 1 Processor -- start 2024-04-06T12:27:19.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.767+0000 7f2cfd529700 1 -- start start 2024-04-06T12:27:19.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.767+0000 7f2cfd529700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cf806a730 0x7f2cf8148cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:19.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.767+0000 7f2cfd529700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cf815d7b0 0x7f2cf815c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:19.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.767+0000 7f2cfd529700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cf815e500 0x7f2cf81f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:19.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.767+0000 7f2cfd529700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf81f4250 con 0x7f2cf815e500 2024-04-06T12:27:19.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.767+0000 7f2cfd529700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf81f44d0 con 0x7f2cf815d7b0 2024-04-06T12:27:19.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.767+0000 7f2cfd529700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf81f47b0 con 0x7f2cf806a730 2024-04-06T12:27:19.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.768+0000 7f2cf77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cf815e500 0x7f2cf81f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:19.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.768+0000 7f2cf77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cf815e500 0x7f2cf81f3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37478/0 (socket says 172.21.15.189:37478) 2024-04-06T12:27:19.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.768+0000 7f2cf77fe700 1 -- 172.21.15.189:0/3979050803 learned_addr learned my addr 172.21.15.189:0/3979050803 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:19.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.768+0000 7f2cf6ffd700 1 --2- 172.21.15.189:0/3979050803 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cf806a730 0x7f2cf8148cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:19.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.768+0000 7f2cf67fc700 1 --2- 172.21.15.189:0/3979050803 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cf815d7b0 0x7f2cf815c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:19.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.768+0000 7f2cf77fe700 1 -- 172.21.15.189:0/3979050803 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cf806a730 msgr2=0x7f2cf8148cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:19.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.768+0000 7f2cf77fe700 1 --2- 172.21.15.189:0/3979050803 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cf806a730 0x7f2cf8148cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.768+0000 7f2cf77fe700 1 -- 172.21.15.189:0/3979050803 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cf815d7b0 msgr2=0x7f2cf815c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:19.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.768+0000 7f2cf77fe700 1 --2- 172.21.15.189:0/3979050803 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cf815d7b0 0x7f2cf815c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:19.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.768+0000 7f2cf77fe700 1 -- 172.21.15.189:0/3979050803 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf0009b30 con 0x7f2cf815e500 2024-04-06T12:27:19.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.768+0000 7f2cf67fc700 1 --2- 172.21.15.189:0/3979050803 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cf815d7b0 0x7f2cf815c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:19.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.768+0000 7f2cf77fe700 1 --2- 172.21.15.189:0/3979050803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cf815e500 0x7f2cf81f3d10 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f2ce800c7c0 tx=0x7f2ce800cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:19.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.768+0000 7f2cef7fe700 1 -- 172.21.15.189:0/3979050803 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce8004220 con 0x7f2cf815e500 2024-04-06T12:27:19.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.768+0000 7f2cef7fe700 1 -- 172.21.15.189:0/3979050803 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce8004380 con 0x7f2cf815e500 2024-04-06T12:27:19.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.768+0000 7f2cfd529700 1 -- 172.21.15.189:0/3979050803 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf81f4a90 con 0x7f2cf815e500 2024-04-06T12:27:19.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.768+0000 7f2cef7fe700 1 -- 172.21.15.189:0/3979050803 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce80036a0 con 0x7f2cf815e500 2024-04-06T12:27:19.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.768+0000 7f2cfd529700 1 -- 172.21.15.189:0/3979050803 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf81f5050 con 0x7f2cf815e500 2024-04-06T12:27:19.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.769+0000 7f2cfd529700 1 -- 172.21.15.189:0/3979050803 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cf8067bb0 con 0x7f2cf815e500 2024-04-06T12:27:19.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.770+0000 7f2cef7fe700 1 -- 172.21.15.189:0/3979050803 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f2ce8003800 con 0x7f2cf815e500 2024-04-06T12:27:19.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.770+0000 7f2cef7fe700 1 --2- 172.21.15.189:0/3979050803 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2cd806adb0 0x7f2cd806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:19.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.770+0000 7f2cef7fe700 1 -- 172.21.15.189:0/3979050803 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f2ce8089020 con 0x7f2cf815e500 2024-04-06T12:27:19.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.770+0000 7f2cf6ffd700 1 --2- 172.21.15.189:0/3979050803 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2cd806adb0 0x7f2cd806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:19.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.771+0000 7f2cf6ffd700 1 --2- 172.21.15.189:0/3979050803 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2cd806adb0 0x7f2cd806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0015040 tx=0x7f2cf0009000 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:19.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.775+0000 7f2cef7fe700 1 -- 172.21.15.189:0/3979050803 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2ce8054080 con 0x7f2cf815e500 2024-04-06T12:27:19.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:19.971+0000 7f2cfd529700 1 -- 172.21.15.189:0/3979050803 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2cf8074990 con 0x7f2cf815e500 2024-04-06T12:27:20.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.623+0000 7f2cef7fe700 1 -- 172.21.15.189:0/3979050803 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v82) v1 ==== 170+0+0 (secure 0 0 0) 0x7f2ce80576a0 con 0x7f2cf815e500 2024-04-06T12:27:20.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.627+0000 7f2cfd529700 1 -- 172.21.15.189:0/3979050803 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2cd806adb0 msgr2=0x7f2cd806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:20.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.627+0000 7f2cfd529700 1 --2- 172.21.15.189:0/3979050803 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2cd806adb0 0x7f2cd806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0015040 tx=0x7f2cf0009000 comp rx=0 tx=0).stop 2024-04-06T12:27:20.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.627+0000 7f2cfd529700 1 -- 172.21.15.189:0/3979050803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cf815e500 msgr2=0x7f2cf81f3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:20.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.627+0000 7f2cfd529700 1 --2- 172.21.15.189:0/3979050803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cf815e500 0x7f2cf81f3d10 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f2ce800c7c0 tx=0x7f2ce800cad0 comp rx=0 tx=0).stop 2024-04-06T12:27:20.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.627+0000 7f2cfd529700 1 -- 172.21.15.189:0/3979050803 shutdown_connections 2024-04-06T12:27:20.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.627+0000 7f2cfd529700 1 --2- 172.21.15.189:0/3979050803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cf815e500 0x7f2cf81f3d10 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:20.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.627+0000 7f2cfd529700 1 --2- 172.21.15.189:0/3979050803 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cf806a730 0x7f2cf8148cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:20.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.627+0000 7f2cfd529700 1 --2- 172.21.15.189:0/3979050803 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2cd806adb0 0x7f2cd806d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:20.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.627+0000 7f2cfd529700 1 --2- 172.21.15.189:0/3979050803 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cf815d7b0 0x7f2cf815c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:20.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.628+0000 7f2cfd529700 1 -- 172.21.15.189:0/3979050803 >> 172.21.15.189:0/3979050803 conn(0x7f2cf80d00b0 msgr2=0x7f2cf80afc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:20.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.628+0000 7f2cfd529700 1 -- 172.21.15.189:0/3979050803 shutdown_connections 2024-04-06T12:27:20.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.628+0000 7f2cfd529700 1 -- 172.21.15.189:0/3979050803 wait complete. 2024-04-06T12:27:20.628 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' created 2024-04-06T12:27:20.647 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-06T12:27:20.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.775+0000 7fbd724fb700 1 Processor -- start 2024-04-06T12:27:20.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.775+0000 7fbd724fb700 1 -- start start 2024-04-06T12:27:20.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.775+0000 7fbd724fb700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbd6c15a550 0x7fbd6c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:20.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.775+0000 7fbd724fb700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd6c0cf900 0x7fbd6c0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:20.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.775+0000 7fbd724fb700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbd6c0b4ba0 0x7fbd6c06eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:20.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.775+0000 7fbd724fb700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd6c0cdbe0 con 0x7fbd6c15a550 2024-04-06T12:27:20.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.775+0000 7fbd724fb700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd6c0b5010 con 0x7fbd6c0b4ba0 2024-04-06T12:27:20.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.775+0000 7fbd724fb700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c15a940 con 0x7fbd6c0cf900 2024-04-06T12:27:20.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.776+0000 7fbd6bfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbd6c15a550 0x7fbd6c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:20.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.776+0000 7fbd6bfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbd6c15a550 0x7fbd6c0cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37494/0 (socket says 172.21.15.189:37494) 2024-04-06T12:27:20.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.776+0000 7fbd6bfff700 1 -- 172.21.15.189:0/575207769 learned_addr learned my addr 172.21.15.189:0/575207769 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:20.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.776+0000 7fbd70a98700 1 --2- 172.21.15.189:0/575207769 >> v2:172.21.15.192:3300/0 conn(0x7fbd6c0b4ba0 0x7fbd6c06eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:20.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.776+0000 7fbd6b7fe700 1 --2- 172.21.15.189:0/575207769 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd6c0cf900 0x7fbd6c0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:20.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.776+0000 7fbd6bfff700 1 -- 172.21.15.189:0/575207769 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd6c0cf900 msgr2=0x7fbd6c0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:20.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.776+0000 7fbd6bfff700 1 --2- 172.21.15.189:0/575207769 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd6c0cf900 0x7fbd6c0b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:20.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.776+0000 7fbd6bfff700 1 -- 172.21.15.189:0/575207769 >> v2:172.21.15.192:3300/0 conn(0x7fbd6c0b4ba0 msgr2=0x7fbd6c06eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:20.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.776+0000 7fbd6bfff700 1 --2- 172.21.15.189:0/575207769 >> v2:172.21.15.192:3300/0 conn(0x7fbd6c0b4ba0 0x7fbd6c06eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:20.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.776+0000 7fbd6bfff700 1 -- 172.21.15.189:0/575207769 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6c15ac00 con 0x7fbd6c15a550 2024-04-06T12:27:20.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.776+0000 7fbd6bfff700 1 --2- 172.21.15.189:0/575207769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd6c15a550 0x7fbd6c0cf3c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0d1110 tx=0x7fbd5c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e118222c6baf21 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:20.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.776+0000 7fbd6affd700 1 -- 172.21.15.189:0/575207769 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c00b870 con 0x7fbd6c15a550 2024-04-06T12:27:20.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.776+0000 7fbd6affd700 1 -- 172.21.15.189:0/575207769 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5c00b9d0 con 0x7fbd6c15a550 2024-04-06T12:27:20.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.776+0000 7fbd6affd700 1 -- 172.21.15.189:0/575207769 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c00bd60 con 0x7fbd6c15a550 2024-04-06T12:27:20.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.777+0000 7fbd724fb700 1 -- 172.21.15.189:0/575207769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd6c15a550 msgr2=0x7fbd6c0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:20.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.777+0000 7fbd724fb700 1 --2- 172.21.15.189:0/575207769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd6c15a550 0x7fbd6c0cf3c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0d1110 tx=0x7fbd5c009bd0 comp rx=0 tx=0).stop 2024-04-06T12:27:20.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.777+0000 7fbd724fb700 1 -- 172.21.15.189:0/575207769 shutdown_connections 2024-04-06T12:27:20.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.777+0000 7fbd724fb700 1 --2- 172.21.15.189:0/575207769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd6c15a550 0x7fbd6c0cf3c0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:20.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.777+0000 7fbd724fb700 1 --2- 172.21.15.189:0/575207769 >> v2:172.21.15.192:3300/0 conn(0x7fbd6c0b4ba0 0x7fbd6c06eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:20.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.777+0000 7fbd724fb700 1 --2- 172.21.15.189:0/575207769 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd6c0cf900 0x7fbd6c0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:20.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.777+0000 7fbd724fb700 1 -- 172.21.15.189:0/575207769 >> 172.21.15.189:0/575207769 conn(0x7fbd6c0d00b0 msgr2=0x7fbd6c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:20.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.777+0000 7fbd724fb700 1 -- 172.21.15.189:0/575207769 shutdown_connections 2024-04-06T12:27:20.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.777+0000 7fbd724fb700 1 -- 172.21.15.189:0/575207769 wait complete. 2024-04-06T12:27:20.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.778+0000 7fbd724fb700 1 Processor -- start 2024-04-06T12:27:20.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.778+0000 7fbd724fb700 1 -- start start 2024-04-06T12:27:20.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.779+0000 7fbd724fb700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd6c0b4ba0 0x7fbd6c13fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:20.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.779+0000 7fbd724fb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd6c0cf900 0x7fbd6c0b2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:20.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.779+0000 7fbd724fb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd6c0b31f0 0x7fbd6c14dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:20.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.779+0000 7fbd724fb700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c0b3670 con 0x7fbd6c0cf900 2024-04-06T12:27:20.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.779+0000 7fbd724fb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c158af0 con 0x7fbd6c0b31f0 2024-04-06T12:27:20.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.779+0000 7fbd724fb700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c14e8b0 con 0x7fbd6c0b4ba0 2024-04-06T12:27:20.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.779+0000 7fbd6b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd6c0cf900 0x7fbd6c0b2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:20.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.779+0000 7fbd6b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd6c0cf900 0x7fbd6c0b2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37496/0 (socket says 172.21.15.189:37496) 2024-04-06T12:27:20.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.779+0000 7fbd6b7fe700 1 -- 172.21.15.189:0/3651741869 learned_addr learned my addr 172.21.15.189:0/3651741869 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:20.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.780+0000 7fbd6bfff700 1 --2- 172.21.15.189:0/3651741869 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd6c0b4ba0 0x7fbd6c13fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:20.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.780+0000 7fbd70a98700 1 --2- 172.21.15.189:0/3651741869 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd6c0b31f0 0x7fbd6c14dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:20.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.780+0000 7fbd6b7fe700 1 -- 172.21.15.189:0/3651741869 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd6c0b4ba0 msgr2=0x7fbd6c13fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:20.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.780+0000 7fbd6b7fe700 1 --2- 172.21.15.189:0/3651741869 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd6c0b4ba0 0x7fbd6c13fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:20.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.780+0000 7fbd6b7fe700 1 -- 172.21.15.189:0/3651741869 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd6c0b31f0 msgr2=0x7fbd6c14dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:20.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.780+0000 7fbd6b7fe700 1 --2- 172.21.15.189:0/3651741869 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd6c0b31f0 0x7fbd6c14dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:20.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.780+0000 7fbd6b7fe700 1 -- 172.21.15.189:0/3651741869 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c009550 con 0x7fbd6c0cf900 2024-04-06T12:27:20.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.780+0000 7fbd6b7fe700 1 --2- 172.21.15.189:0/3651741869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd6c0cf900 0x7fbd6c0b2cb0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fbd6000e840 tx=0x7fbd6000eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:20.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.780+0000 7fbd68ff9700 1 -- 172.21.15.189:0/3651741869 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd60013610 con 0x7fbd6c0cf900 2024-04-06T12:27:20.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.780+0000 7fbd68ff9700 1 -- 172.21.15.189:0/3651741869 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd60013770 con 0x7fbd6c0cf900 2024-04-06T12:27:20.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.780+0000 7fbd724fb700 1 -- 172.21.15.189:0/3651741869 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd6c14eb90 con 0x7fbd6c0cf900 2024-04-06T12:27:20.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.780+0000 7fbd68ff9700 1 -- 172.21.15.189:0/3651741869 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd60013b20 con 0x7fbd6c0cf900 2024-04-06T12:27:20.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.780+0000 7fbd724fb700 1 -- 172.21.15.189:0/3651741869 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd6c14f150 con 0x7fbd6c0cf900 2024-04-06T12:27:20.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.781+0000 7fbd724fb700 1 -- 172.21.15.189:0/3651741869 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd6c067bb0 con 0x7fbd6c0cf900 2024-04-06T12:27:20.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.782+0000 7fbd68ff9700 1 -- 172.21.15.189:0/3651741869 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fbd60013c80 con 0x7fbd6c0cf900 2024-04-06T12:27:20.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.782+0000 7fbd68ff9700 1 --2- 172.21.15.189:0/3651741869 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbd5806adb0 0x7fbd5806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:20.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.782+0000 7fbd68ff9700 1 -- 172.21.15.189:0/3651741869 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5929+0+0 (secure 0 0 0) 0x7fbd60089230 con 0x7fbd6c0cf900 2024-04-06T12:27:20.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.782+0000 7fbd6bfff700 1 --2- 172.21.15.189:0/3651741869 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbd5806adb0 0x7fbd5806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:20.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.783+0000 7fbd6bfff700 1 --2- 172.21.15.189:0/3651741869 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbd5806adb0 0x7fbd5806d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c000c00 tx=0x7fbd5c008030 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:20.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:20.787+0000 7fbd68ff9700 1 -- 172.21.15.189:0/3651741869 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbd60054120 con 0x7fbd6c0cf900 2024-04-06T12:27:21.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.028+0000 7fbd724fb700 1 -- 172.21.15.189:0/3651741869 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fbd6c074990 con 0x7fbd6c0cf900 2024-04-06T12:27:21.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.035+0000 7fbd68ff9700 1 -- 172.21.15.189:0/3651741869 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v28) v1 ==== 173+0+0 (secure 0 0 0) 0x7fbd60057740 con 0x7fbd6c0cf900 2024-04-06T12:27:21.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.040+0000 7fbd724fb700 1 -- 172.21.15.189:0/3651741869 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbd5806adb0 msgr2=0x7fbd5806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:21.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.040+0000 7fbd724fb700 1 --2- 172.21.15.189:0/3651741869 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbd5806adb0 0x7fbd5806d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c000c00 tx=0x7fbd5c008030 comp rx=0 tx=0).stop 2024-04-06T12:27:21.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.040+0000 7fbd724fb700 1 -- 172.21.15.189:0/3651741869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd6c0cf900 msgr2=0x7fbd6c0b2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:21.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.040+0000 7fbd724fb700 1 --2- 172.21.15.189:0/3651741869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd6c0cf900 0x7fbd6c0b2cb0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fbd6000e840 tx=0x7fbd6000eb50 comp rx=0 tx=0).stop 2024-04-06T12:27:21.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.040+0000 7fbd724fb700 1 -- 172.21.15.189:0/3651741869 shutdown_connections 2024-04-06T12:27:21.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.040+0000 7fbd724fb700 1 --2- 172.21.15.189:0/3651741869 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbd5806adb0 0x7fbd5806d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.040+0000 7fbd724fb700 1 --2- 172.21.15.189:0/3651741869 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd6c0b31f0 0x7fbd6c14dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.040+0000 7fbd724fb700 1 --2- 172.21.15.189:0/3651741869 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd6c0b4ba0 0x7fbd6c13fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.040+0000 7fbd724fb700 1 --2- 172.21.15.189:0/3651741869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd6c0cf900 0x7fbd6c0b2cb0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.040+0000 7fbd724fb700 1 -- 172.21.15.189:0/3651741869 >> 172.21.15.189:0/3651741869 conn(0x7fbd6c0d00b0 msgr2=0x7fbd6c0afc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:21.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.040+0000 7fbd724fb700 1 -- 172.21.15.189:0/3651741869 shutdown_connections 2024-04-06T12:27:21.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.040+0000 7fbd724fb700 1 -- 172.21.15.189:0/3651741869 wait complete. 2024-04-06T12:27:21.040 INFO:teuthology.orchestra.run.smithi189.stderr:new fs with metadata pool 9 and data pool 10 2024-04-06T12:27:21.060 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:27:21.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.184+0000 7fb8e7750700 1 Processor -- start 2024-04-06T12:27:21.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.184+0000 7fb8e7750700 1 -- start start 2024-04-06T12:27:21.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.185+0000 7fb8e7750700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb8e015a550 0x7fb8e00cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:21.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.185+0000 7fb8e7750700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8e00b4d70 0x7fb8e00b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:21.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.185+0000 7fb8e7750700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e00cdbe0 con 0x7fb8e00cf900 2024-04-06T12:27:21.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.185+0000 7fb8e7750700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e015a940 con 0x7fb8e015a550 2024-04-06T12:27:21.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.185+0000 7fb8e7750700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e015ac00 con 0x7fb8e00b4d70 2024-04-06T12:27:21.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.185+0000 7fb8e54ec700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb8e015a550 0x7fb8e00cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:21.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.185+0000 7fb8e5ced700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8e00b4d70 0x7fb8e00b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:21.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.185+0000 7fb8e54ec700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb8e015a550 0x7fb8e00cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:37406/0 (socket says 172.21.15.189:37406) 2024-04-06T12:27:21.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.185+0000 7fb8e54ec700 1 -- 172.21.15.189:0/2427691949 learned_addr learned my addr 172.21.15.189:0/2427691949 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:21.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.186+0000 7fb8e54ec700 1 -- 172.21.15.189:0/2427691949 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8e00b4d70 msgr2=0x7fb8e00b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:21.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.186+0000 7fb8e54ec700 1 --2- 172.21.15.189:0/2427691949 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8e00b4d70 0x7fb8e00b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.186+0000 7fb8e54ec700 1 -- 172.21.15.189:0/2427691949 >> v1:172.21.15.192:6789/0 conn(0x7fb8e00cf900 legacy=0x7fb8e00b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:21.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.186+0000 7fb8e54ec700 1 -- 172.21.15.189:0/2427691949 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e015aee0 con 0x7fb8e015a550 2024-04-06T12:27:21.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.186+0000 7fb8e5ced700 1 --2- 172.21.15.189:0/2427691949 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8e00b4d70 0x7fb8e00b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:21.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.186+0000 7fb8e54ec700 1 --2- 172.21.15.189:0/2427691949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8e015a550 0x7fb8e00cf3c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc0098f0 tx=0x7fb8dc009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2474094b6a4cb95e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:21.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.186+0000 7fb8d7fff700 1 -- 172.21.15.189:0/2427691949 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc011830 con 0x7fb8e015a550 2024-04-06T12:27:21.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.186+0000 7fb8d7fff700 1 -- 172.21.15.189:0/2427691949 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8dc011990 con 0x7fb8e015a550 2024-04-06T12:27:21.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.186+0000 7fb8d7fff700 1 -- 172.21.15.189:0/2427691949 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc011d20 con 0x7fb8e015a550 2024-04-06T12:27:21.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.186+0000 7fb8e7750700 1 -- 172.21.15.189:0/2427691949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8e015a550 msgr2=0x7fb8e00cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:21.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.186+0000 7fb8e7750700 1 --2- 172.21.15.189:0/2427691949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8e015a550 0x7fb8e00cf3c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc0098f0 tx=0x7fb8dc009c00 comp rx=0 tx=0).stop 2024-04-06T12:27:21.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.186+0000 7fb8e7750700 1 -- 172.21.15.189:0/2427691949 shutdown_connections 2024-04-06T12:27:21.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.186+0000 7fb8e7750700 1 --2- 172.21.15.189:0/2427691949 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8e00b4d70 0x7fb8e00b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.186+0000 7fb8e7750700 1 --2- 172.21.15.189:0/2427691949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8e015a550 0x7fb8e00cf3c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.186+0000 7fb8e7750700 1 -- 172.21.15.189:0/2427691949 >> 172.21.15.189:0/2427691949 conn(0x7fb8e00d00b0 msgr2=0x7fb8e00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:21.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.187+0000 7fb8e7750700 1 -- 172.21.15.189:0/2427691949 shutdown_connections 2024-04-06T12:27:21.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.187+0000 7fb8e7750700 1 -- 172.21.15.189:0/2427691949 wait complete. 2024-04-06T12:27:21.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.188+0000 7fb8e7750700 1 Processor -- start 2024-04-06T12:27:21.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.188+0000 7fb8e7750700 1 -- start start 2024-04-06T12:27:21.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.188+0000 7fb8e7750700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8e00b4d70 0x7fb8e0159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:21.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.188+0000 7fb8e7750700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8e00cf900 0x7fb8e0147260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:21.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.188+0000 7fb8e7750700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8e014db30 0x7fb8e01477a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:21.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.188+0000 7fb8e7750700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e01f6110 con 0x7fb8e00b4d70 2024-04-06T12:27:21.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.188+0000 7fb8e7750700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e01f6380 con 0x7fb8e00cf900 2024-04-06T12:27:21.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.188+0000 7fb8e7750700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e01f6660 con 0x7fb8e014db30 2024-04-06T12:27:21.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.189+0000 7fb8e54ec700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8e00b4d70 0x7fb8e0159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:21.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.189+0000 7fb8e54ec700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8e00b4d70 0x7fb8e0159790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37504/0 (socket says 172.21.15.189:37504) 2024-04-06T12:27:21.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.189+0000 7fb8e54ec700 1 -- 172.21.15.189:0/2635229054 learned_addr learned my addr 172.21.15.189:0/2635229054 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:21.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.189+0000 7fb8e5ced700 1 --2- 172.21.15.189:0/2635229054 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8e014db30 0x7fb8e01477a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:21.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.189+0000 7fb8e4ceb700 1 --2- 172.21.15.189:0/2635229054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8e00cf900 0x7fb8e0147260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:21.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.189+0000 7fb8e54ec700 1 -- 172.21.15.189:0/2635229054 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8e014db30 msgr2=0x7fb8e01477a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:21.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.189+0000 7fb8e54ec700 1 --2- 172.21.15.189:0/2635229054 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8e014db30 0x7fb8e01477a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.189+0000 7fb8e54ec700 1 -- 172.21.15.189:0/2635229054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8e00cf900 msgr2=0x7fb8e0147260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:21.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.189+0000 7fb8e54ec700 1 --2- 172.21.15.189:0/2635229054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8e00cf900 0x7fb8e0147260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.189+0000 7fb8e54ec700 1 -- 172.21.15.189:0/2635229054 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d8009a70 con 0x7fb8e00b4d70 2024-04-06T12:27:21.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.189+0000 7fb8e54ec700 1 --2- 172.21.15.189:0/2635229054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8e00b4d70 0x7fb8e0159790 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc009c40 tx=0x7fb8dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:21.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.189+0000 7fb8d5ffb700 1 -- 172.21.15.189:0/2635229054 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc0036a0 con 0x7fb8e00b4d70 2024-04-06T12:27:21.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.189+0000 7fb8d5ffb700 1 -- 172.21.15.189:0/2635229054 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8dc01e070 con 0x7fb8e00b4d70 2024-04-06T12:27:21.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.189+0000 7fb8e7750700 1 -- 172.21.15.189:0/2635229054 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8dc009550 con 0x7fb8e00b4d70 2024-04-06T12:27:21.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.189+0000 7fb8d5ffb700 1 -- 172.21.15.189:0/2635229054 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc003a60 con 0x7fb8e00b4d70 2024-04-06T12:27:21.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.189+0000 7fb8e7750700 1 -- 172.21.15.189:0/2635229054 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e01f6c20 con 0x7fb8e00b4d70 2024-04-06T12:27:21.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.190+0000 7fb8e7750700 1 -- 172.21.15.189:0/2635229054 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e0067bb0 con 0x7fb8e00b4d70 2024-04-06T12:27:21.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.191+0000 7fb8d5ffb700 1 -- 172.21.15.189:0/2635229054 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fb8dc003bc0 con 0x7fb8e00b4d70 2024-04-06T12:27:21.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.191+0000 7fb8d5ffb700 1 --2- 172.21.15.189:0/2635229054 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb8c406adb0 0x7fb8c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:21.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.191+0000 7fb8d5ffb700 1 -- 172.21.15.189:0/2635229054 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb8dc08a510 con 0x7fb8e00b4d70 2024-04-06T12:27:21.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.191+0000 7fb8e4ceb700 1 --2- 172.21.15.189:0/2635229054 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb8c406adb0 0x7fb8c406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:21.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.192+0000 7fb8e4ceb700 1 --2- 172.21.15.189:0/2635229054 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb8c406adb0 0x7fb8c406d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0000f60 tx=0x7fb8d0009040 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:21.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.196+0000 7fb8d5ffb700 1 -- 172.21.15.189:0/2635229054 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb8dc055530 con 0x7fb8e00b4d70 2024-04-06T12:27:21.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.392+0000 7fb8e7750700 1 -- 172.21.15.189:0/2635229054 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8e0074990 con 0x7fb8e00b4d70 2024-04-06T12:27:21.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.394+0000 7fb8d5ffb700 1 -- 172.21.15.189:0/2635229054 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16309 (secure 0 0 0) 0x7fb8dc058b50 con 0x7fb8e00b4d70 2024-04-06T12:27:21.392 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:21.392 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":83,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:27:21.030702+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:27:19.575213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-06T12:27:19.990714+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:27:21.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.396+0000 7fb8e7750700 1 -- 172.21.15.189:0/2635229054 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb8c406adb0 msgr2=0x7fb8c406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:21.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.396+0000 7fb8e7750700 1 --2- 172.21.15.189:0/2635229054 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb8c406adb0 0x7fb8c406d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0000f60 tx=0x7fb8d0009040 comp rx=0 tx=0).stop 2024-04-06T12:27:21.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.396+0000 7fb8e7750700 1 -- 172.21.15.189:0/2635229054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8e00b4d70 msgr2=0x7fb8e0159790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:21.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.396+0000 7fb8e7750700 1 --2- 172.21.15.189:0/2635229054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8e00b4d70 0x7fb8e0159790 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc009c40 tx=0x7fb8dc004750 comp rx=0 tx=0).stop 2024-04-06T12:27:21.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.396+0000 7fb8e7750700 1 -- 172.21.15.189:0/2635229054 shutdown_connections 2024-04-06T12:27:21.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.396+0000 7fb8e7750700 1 --2- 172.21.15.189:0/2635229054 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8e014db30 0x7fb8e01477a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.396+0000 7fb8e7750700 1 --2- 172.21.15.189:0/2635229054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8e00b4d70 0x7fb8e0159790 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.396+0000 7fb8e7750700 1 --2- 172.21.15.189:0/2635229054 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb8c406adb0 0x7fb8c406d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.396+0000 7fb8e7750700 1 --2- 172.21.15.189:0/2635229054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8e00cf900 0x7fb8e0147260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.396+0000 7fb8e7750700 1 -- 172.21.15.189:0/2635229054 >> 172.21.15.189:0/2635229054 conn(0x7fb8e00d00b0 msgr2=0x7fb8e006f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:21.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.396+0000 7fb8e7750700 1 -- 172.21.15.189:0/2635229054 shutdown_connections 2024-04-06T12:27:21.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.396+0000 7fb8e7750700 1 -- 172.21.15.189:0/2635229054 wait complete. 2024-04-06T12:27:21.407 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:27:21.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.532+0000 7fad8c57c700 1 Processor -- start 2024-04-06T12:27:21.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.532+0000 7fad8c57c700 1 -- start start 2024-04-06T12:27:21.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.533+0000 7fad8c57c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fad8406ae40 0x7fad8415ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.533+0000 7fad8c57c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8415c740 con 0x7fad84144f40 2024-04-06T12:27:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.533+0000 7fad8c57c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8415f1f0 con 0x7fad8415fcd0 2024-04-06T12:27:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.533+0000 7fad8c57c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad840cfcb0 con 0x7fad8406ae40 2024-04-06T12:27:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.533+0000 7fad89b17700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fad8406ae40 0x7fad8415ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.533+0000 7fad89b17700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fad8406ae40 0x7fad8415ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37518/0 (socket says 172.21.15.189:37518) 2024-04-06T12:27:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.533+0000 7fad89b17700 1 -- 172.21.15.189:0/2352434200 learned_addr learned my addr 172.21.15.189:0/2352434200 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.533+0000 7fad8a318700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fad84144f40 0x7fad8406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56862/0 (socket says 172.21.15.189:56862) 2024-04-06T12:27:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.533+0000 7fad89b17700 1 -- 172.21.15.189:0/2352434200 >> v1:172.21.15.192:6789/0 conn(0x7fad8415fcd0 legacy=0x7fad8415bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.533+0000 7fad89b17700 1 -- 172.21.15.189:0/2352434200 >> v1:172.21.15.189:6789/0 conn(0x7fad84144f40 legacy=0x7fad8406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.533+0000 7fad89b17700 1 -- 172.21.15.189:0/2352434200 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8415f4d0 con 0x7fad8406ae40 2024-04-06T12:27:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.533+0000 7fad89b17700 1 --2- 172.21.15.189:0/2352434200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad8406ae40 0x7fad8415ba40 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fad740098f0 tx=0x7fad74009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=261f3f2b025bedd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.533+0000 7fad89316700 1 -- 172.21.15.189:0/2352434200 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad74004230 con 0x7fad8406ae40 2024-04-06T12:27:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.533+0000 7fad89316700 1 -- 172.21.15.189:0/2352434200 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad74004390 con 0x7fad8406ae40 2024-04-06T12:27:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.533+0000 7fad89316700 1 -- 172.21.15.189:0/2352434200 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad740036a0 con 0x7fad8406ae40 2024-04-06T12:27:21.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.534+0000 7fad8c57c700 1 -- 172.21.15.189:0/2352434200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad8406ae40 msgr2=0x7fad8415ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:21.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.534+0000 7fad8c57c700 1 --2- 172.21.15.189:0/2352434200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad8406ae40 0x7fad8415ba40 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fad740098f0 tx=0x7fad74009c00 comp rx=0 tx=0).stop 2024-04-06T12:27:21.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.534+0000 7fad8c57c700 1 -- 172.21.15.189:0/2352434200 shutdown_connections 2024-04-06T12:27:21.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.534+0000 7fad8c57c700 1 --2- 172.21.15.189:0/2352434200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad8406ae40 0x7fad8415ba40 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.534+0000 7fad8c57c700 1 -- 172.21.15.189:0/2352434200 >> 172.21.15.189:0/2352434200 conn(0x7fad840d00b0 msgr2=0x7fad840d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:21.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.534+0000 7fad8c57c700 1 -- 172.21.15.189:0/2352434200 shutdown_connections 2024-04-06T12:27:21.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.534+0000 7fad8c57c700 1 -- 172.21.15.189:0/2352434200 wait complete. 2024-04-06T12:27:21.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.535+0000 7fad8c57c700 1 Processor -- start 2024-04-06T12:27:21.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.535+0000 7fad8c57c700 1 -- start start 2024-04-06T12:27:21.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.536+0000 7fad8c57c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad8406ae40 0x7fad84160900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:21.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.536+0000 7fad8c57c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad84144f40 0x7fad8413aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:21.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.536+0000 7fad8c57c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad8415fcd0 0x7fad84071a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:21.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.536+0000 7fad8c57c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8413bae0 con 0x7fad84144f40 2024-04-06T12:27:21.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.536+0000 7fad8c57c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8413c0b0 con 0x7fad8406ae40 2024-04-06T12:27:21.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.536+0000 7fad8c57c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad84160e40 con 0x7fad8415fcd0 2024-04-06T12:27:21.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.536+0000 7fad89b17700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad84144f40 0x7fad8413aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:21.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.536+0000 7fad89b17700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad84144f40 0x7fad8413aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37530/0 (socket says 172.21.15.189:37530) 2024-04-06T12:27:21.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.536+0000 7fad89b17700 1 -- 172.21.15.189:0/3436476355 learned_addr learned my addr 172.21.15.189:0/3436476355 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:21.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.536+0000 7fad89b17700 1 -- 172.21.15.189:0/3436476355 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad8415fcd0 msgr2=0x7fad84071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:21.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.536+0000 7fad8ab19700 1 --2- 172.21.15.189:0/3436476355 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad8415fcd0 0x7fad84071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:21.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.536+0000 7fad8a318700 1 --2- 172.21.15.189:0/3436476355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad8406ae40 0x7fad84160900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:21.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.536+0000 7fad89b17700 1 --2- 172.21.15.189:0/3436476355 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad8415fcd0 0x7fad84071a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.536+0000 7fad89b17700 1 -- 172.21.15.189:0/3436476355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad8406ae40 msgr2=0x7fad84160900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:21.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.536+0000 7fad89b17700 1 --2- 172.21.15.189:0/3436476355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad8406ae40 0x7fad84160900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.536+0000 7fad89b17700 1 -- 172.21.15.189:0/3436476355 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad74009550 con 0x7fad84144f40 2024-04-06T12:27:21.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.536+0000 7fad8ab19700 1 --2- 172.21.15.189:0/3436476355 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad8415fcd0 0x7fad84071a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:27:21.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.536+0000 7fad8a318700 1 --2- 172.21.15.189:0/3436476355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad8406ae40 0x7fad84160900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:27:21.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.537+0000 7fad89b17700 1 --2- 172.21.15.189:0/3436476355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad84144f40 0x7fad8413aa10 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fad74004000 tx=0x7fad74004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:21.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.537+0000 7fad7affd700 1 -- 172.21.15.189:0/3436476355 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad74003dc0 con 0x7fad84144f40 2024-04-06T12:27:21.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.537+0000 7fad7affd700 1 -- 172.21.15.189:0/3436476355 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7401f070 con 0x7fad84144f40 2024-04-06T12:27:21.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.537+0000 7fad8c57c700 1 -- 172.21.15.189:0/3436476355 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad8413b240 con 0x7fad84144f40 2024-04-06T12:27:21.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.537+0000 7fad7affd700 1 -- 172.21.15.189:0/3436476355 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7400b8a0 con 0x7fad84144f40 2024-04-06T12:27:21.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.537+0000 7fad8c57c700 1 -- 172.21.15.189:0/3436476355 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad8413b800 con 0x7fad84144f40 2024-04-06T12:27:21.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.538+0000 7fad8c57c700 1 -- 172.21.15.189:0/3436476355 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad84067bb0 con 0x7fad84144f40 2024-04-06T12:27:21.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.538+0000 7fad7affd700 1 -- 172.21.15.189:0/3436476355 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fad74021030 con 0x7fad84144f40 2024-04-06T12:27:21.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.538+0000 7fad7affd700 1 --2- 172.21.15.189:0/3436476355 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fad6806adb0 0x7fad6806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:21.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.539+0000 7fad7affd700 1 -- 172.21.15.189:0/3436476355 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fad7408ace0 con 0x7fad84144f40 2024-04-06T12:27:21.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.539+0000 7fad8a318700 1 --2- 172.21.15.189:0/3436476355 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fad6806adb0 0x7fad6806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:21.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.539+0000 7fad8a318700 1 --2- 172.21.15.189:0/3436476355 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fad6806adb0 0x7fad6806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fad80000f60 tx=0x7fad80009040 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:21.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.543+0000 7fad7affd700 1 -- 172.21.15.189:0/3436476355 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fad740be650 con 0x7fad84144f40 2024-04-06T12:27:21.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.740+0000 7fad8c57c700 1 -- 172.21.15.189:0/3436476355 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad84067bb0 con 0x7fad84144f40 2024-04-06T12:27:21.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.741+0000 7fad7affd700 1 -- 172.21.15.189:0/3436476355 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16309 (secure 0 0 0) 0x7fad74055d00 con 0x7fad84144f40 2024-04-06T12:27:21.740 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:21.740 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":83,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:27:21.030702+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:27:19.575213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-06T12:27:19.990714+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:27:21.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.743+0000 7fad8c57c700 1 -- 172.21.15.189:0/3436476355 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fad6806adb0 msgr2=0x7fad6806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:21.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.743+0000 7fad8c57c700 1 --2- 172.21.15.189:0/3436476355 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fad6806adb0 0x7fad6806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fad80000f60 tx=0x7fad80009040 comp rx=0 tx=0).stop 2024-04-06T12:27:21.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.743+0000 7fad8c57c700 1 -- 172.21.15.189:0/3436476355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad84144f40 msgr2=0x7fad8413aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:21.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.743+0000 7fad8c57c700 1 --2- 172.21.15.189:0/3436476355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad84144f40 0x7fad8413aa10 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fad74004000 tx=0x7fad74004750 comp rx=0 tx=0).stop 2024-04-06T12:27:21.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.743+0000 7fad8c57c700 1 -- 172.21.15.189:0/3436476355 shutdown_connections 2024-04-06T12:27:21.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.743+0000 7fad8c57c700 1 --2- 172.21.15.189:0/3436476355 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad8415fcd0 0x7fad84071a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.744+0000 7fad8c57c700 1 --2- 172.21.15.189:0/3436476355 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fad6806adb0 0x7fad6806d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.744+0000 7fad8c57c700 1 --2- 172.21.15.189:0/3436476355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad8406ae40 0x7fad84160900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.744+0000 7fad8c57c700 1 --2- 172.21.15.189:0/3436476355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad84144f40 0x7fad8413aa10 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.744+0000 7fad8c57c700 1 -- 172.21.15.189:0/3436476355 >> 172.21.15.189:0/3436476355 conn(0x7fad840d00b0 msgr2=0x7fad84165920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:21.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.744+0000 7fad8c57c700 1 -- 172.21.15.189:0/3436476355 shutdown_connections 2024-04-06T12:27:21.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.744+0000 7fad8c57c700 1 -- 172.21.15.189:0/3436476355 wait complete. 2024-04-06T12:27:21.754 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-06T12:27:21.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.881+0000 7f9ca8c12700 1 Processor -- start 2024-04-06T12:27:21.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.881+0000 7f9ca8c12700 1 -- start start 2024-04-06T12:27:21.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.882+0000 7f9ca8c12700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9ca40d17a0 0x7f9ca406b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:21.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.882+0000 7f9ca8c12700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9ca4146dd0 0x7f9ca406d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:21.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.882+0000 7f9ca8c12700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca406c0a0 con 0x7f9ca40cedf0 2024-04-06T12:27:21.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.882+0000 7f9ca8c12700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ca406c380 con 0x7f9ca4146dd0 2024-04-06T12:27:21.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.882+0000 7f9ca8c12700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ca406c660 con 0x7f9ca40d17a0 2024-04-06T12:27:21.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.882+0000 7f9ca1d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9ca4146dd0 0x7f9ca406d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:21.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.882+0000 7f9ca2d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9ca40cedf0 0x7f9ca40cf1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56878/0 (socket says 172.21.15.189:56878) 2024-04-06T12:27:21.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.882+0000 7f9ca2d9d700 1 -- 172.21.15.189:0/2397226526 learned_addr learned my addr 172.21.15.189:0/2397226526 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:21.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.882+0000 7f9ca259c700 1 --2- 172.21.15.189:0/2397226526 >> v2:172.21.15.192:3300/0 conn(0x7f9ca40d17a0 0x7f9ca406b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:21.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.882+0000 7f9ca1d9b700 1 -- 172.21.15.189:0/2397226526 >> v2:172.21.15.192:3300/0 conn(0x7f9ca40d17a0 msgr2=0x7f9ca406b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:21.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.882+0000 7f9ca1d9b700 1 --2- 172.21.15.189:0/2397226526 >> v2:172.21.15.192:3300/0 conn(0x7f9ca40d17a0 0x7f9ca406b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.882+0000 7f9ca1d9b700 1 -- 172.21.15.189:0/2397226526 >> v1:172.21.15.189:6789/0 conn(0x7f9ca40cedf0 legacy=0x7f9ca40cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:21.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.882+0000 7f9ca1d9b700 1 -- 172.21.15.189:0/2397226526 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca406c940 con 0x7f9ca4146dd0 2024-04-06T12:27:21.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.882+0000 7f9ca159a700 1 -- 172.21.15.189:0/2397226526 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910334861 0 0) 0x7f9ca406c0a0 con 0x7f9ca40cedf0 2024-04-06T12:27:21.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.882+0000 7f9ca259c700 1 --2- 172.21.15.189:0/2397226526 >> v2:172.21.15.192:3300/0 conn(0x7f9ca40d17a0 0x7f9ca406b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:27:21.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.882+0000 7f9ca1d9b700 1 --2- 172.21.15.189:0/2397226526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ca4146dd0 0x7f9ca406d140 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f9ca404fb40 tx=0x7f9c98009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41d95a3041fbec04 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:21.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.882+0000 7f9ca159a700 1 -- 172.21.15.189:0/2397226526 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98011830 con 0x7f9ca4146dd0 2024-04-06T12:27:21.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.883+0000 7f9ca159a700 1 -- 172.21.15.189:0/2397226526 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c98011990 con 0x7f9ca4146dd0 2024-04-06T12:27:21.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.883+0000 7f9ca159a700 1 -- 172.21.15.189:0/2397226526 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98011d20 con 0x7f9ca4146dd0 2024-04-06T12:27:21.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.883+0000 7f9ca8c12700 1 -- 172.21.15.189:0/2397226526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ca4146dd0 msgr2=0x7f9ca406d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:21.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.883+0000 7f9ca8c12700 1 --2- 172.21.15.189:0/2397226526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ca4146dd0 0x7f9ca406d140 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f9ca404fb40 tx=0x7f9c98009b20 comp rx=0 tx=0).stop 2024-04-06T12:27:21.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.883+0000 7f9ca8c12700 1 -- 172.21.15.189:0/2397226526 shutdown_connections 2024-04-06T12:27:21.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.883+0000 7f9ca8c12700 1 --2- 172.21.15.189:0/2397226526 >> v2:172.21.15.192:3300/0 conn(0x7f9ca40d17a0 0x7f9ca406b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.883+0000 7f9ca8c12700 1 --2- 172.21.15.189:0/2397226526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ca4146dd0 0x7f9ca406d140 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.883+0000 7f9ca8c12700 1 -- 172.21.15.189:0/2397226526 >> 172.21.15.189:0/2397226526 conn(0x7f9ca40b2db0 msgr2=0x7f9ca40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:21.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.883+0000 7f9ca8c12700 1 -- 172.21.15.189:0/2397226526 shutdown_connections 2024-04-06T12:27:21.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.883+0000 7f9ca8c12700 1 -- 172.21.15.189:0/2397226526 wait complete. 2024-04-06T12:27:21.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.885+0000 7f9ca8c12700 1 Processor -- start 2024-04-06T12:27:21.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.885+0000 7f9ca8c12700 1 -- start start 2024-04-06T12:27:21.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.885+0000 7f9ca8c12700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ca40cedf0 0x7f9ca40ae5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:21.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.885+0000 7f9ca8c12700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ca40d17a0 0x7f9ca40cdea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:21.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.885+0000 7f9ca8c12700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ca4146dd0 0x7f9ca41f2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:21.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.885+0000 7f9ca8c12700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40af150 con 0x7f9ca40cedf0 2024-04-06T12:27:21.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.885+0000 7f9ca8c12700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40aeb90 con 0x7f9ca40d17a0 2024-04-06T12:27:21.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.885+0000 7f9ca8c12700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40aee70 con 0x7f9ca4146dd0 2024-04-06T12:27:21.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.885+0000 7f9ca259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ca40cedf0 0x7f9ca40ae5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:21.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.885+0000 7f9ca259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ca40cedf0 0x7f9ca40ae5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37546/0 (socket says 172.21.15.189:37546) 2024-04-06T12:27:21.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.885+0000 7f9ca259c700 1 -- 172.21.15.189:0/469505143 learned_addr learned my addr 172.21.15.189:0/469505143 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:21.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.886+0000 7f9ca1d9b700 1 --2- 172.21.15.189:0/469505143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ca40d17a0 0x7f9ca40cdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:21.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.886+0000 7f9ca2d9d700 1 --2- 172.21.15.189:0/469505143 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ca4146dd0 0x7f9ca41f2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:21.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.886+0000 7f9ca259c700 1 -- 172.21.15.189:0/469505143 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ca4146dd0 msgr2=0x7f9ca41f2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:21.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.886+0000 7f9ca259c700 1 --2- 172.21.15.189:0/469505143 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ca4146dd0 0x7f9ca41f2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.886+0000 7f9ca259c700 1 -- 172.21.15.189:0/469505143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ca40d17a0 msgr2=0x7f9ca40cdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:21.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.886+0000 7f9ca259c700 1 --2- 172.21.15.189:0/469505143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ca40d17a0 0x7f9ca40cdea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:21.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.886+0000 7f9ca259c700 1 -- 172.21.15.189:0/469505143 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c980094a0 con 0x7f9ca40cedf0 2024-04-06T12:27:21.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.886+0000 7f9ca2d9d700 1 --2- 172.21.15.189:0/469505143 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ca4146dd0 0x7f9ca41f2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:27:21.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.886+0000 7f9ca1d9b700 1 --2- 172.21.15.189:0/469505143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ca40d17a0 0x7f9ca40cdea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:27:21.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.886+0000 7f9ca259c700 1 --2- 172.21.15.189:0/469505143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ca40cedf0 0x7f9ca40ae5b0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f9c9400c750 tx=0x7f9c9400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:21.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.886+0000 7f9c92ffd700 1 -- 172.21.15.189:0/469505143 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c94011800 con 0x7f9ca40cedf0 2024-04-06T12:27:21.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.886+0000 7f9c92ffd700 1 -- 172.21.15.189:0/469505143 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c94011960 con 0x7f9ca40cedf0 2024-04-06T12:27:21.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.886+0000 7f9ca8c12700 1 -- 172.21.15.189:0/469505143 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca40b4f90 con 0x7f9ca40cedf0 2024-04-06T12:27:21.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.886+0000 7f9c92ffd700 1 -- 172.21.15.189:0/469505143 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c94011cd0 con 0x7f9ca40cedf0 2024-04-06T12:27:21.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.886+0000 7f9ca8c12700 1 -- 172.21.15.189:0/469505143 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca40ce3e0 con 0x7f9ca40cedf0 2024-04-06T12:27:21.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.887+0000 7f9ca8c12700 1 -- 172.21.15.189:0/469505143 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca4074990 con 0x7f9ca40cedf0 2024-04-06T12:27:21.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.887+0000 7f9c92ffd700 1 -- 172.21.15.189:0/469505143 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f9c94021350 con 0x7f9ca40cedf0 2024-04-06T12:27:21.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.888+0000 7f9c92ffd700 1 --2- 172.21.15.189:0/469505143 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c8406adb0 0x7f9c8406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:21.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.888+0000 7f9c92ffd700 1 -- 172.21.15.189:0/469505143 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9c940896a0 con 0x7f9ca40cedf0 2024-04-06T12:27:21.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.888+0000 7f9ca1d9b700 1 --2- 172.21.15.189:0/469505143 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c8406adb0 0x7f9c8406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:21.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.888+0000 7f9ca1d9b700 1 --2- 172.21.15.189:0/469505143 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c8406adb0 0x7f9c8406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9ca41471e0 tx=0x7f9c98008030 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:21.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:21.892+0000 7f9c92ffd700 1 -- 172.21.15.189:0/469505143 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9c94054520 con 0x7f9ca40cedf0 2024-04-06T12:27:22.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:22.132+0000 7f9ca8c12700 1 -- 172.21.15.189:0/469505143 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9ca40d1e20 con 0x7f9ca40cedf0 2024-04-06T12:27:22.610 INFO:tasks.ceph.mds.c.smithi189.stderr:2024-04-06T12:27:22.611+0000 7f63badbb700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T12:27:23.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.038+0000 7f9c92ffd700 1 -- 172.21.15.189:0/469505143 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v30) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9c94057b40 con 0x7f9ca40cedf0 2024-04-06T12:27:23.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.041+0000 7f9ca8c12700 1 -- 172.21.15.189:0/469505143 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c8406adb0 msgr2=0x7f9c8406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:23.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.041+0000 7f9ca8c12700 1 --2- 172.21.15.189:0/469505143 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c8406adb0 0x7f9c8406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9ca41471e0 tx=0x7f9c98008030 comp rx=0 tx=0).stop 2024-04-06T12:27:23.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.041+0000 7f9ca8c12700 1 -- 172.21.15.189:0/469505143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ca40cedf0 msgr2=0x7f9ca40ae5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:23.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.041+0000 7f9ca8c12700 1 --2- 172.21.15.189:0/469505143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ca40cedf0 0x7f9ca40ae5b0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f9c9400c750 tx=0x7f9c9400cb10 comp rx=0 tx=0).stop 2024-04-06T12:27:23.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.041+0000 7f9ca8c12700 1 -- 172.21.15.189:0/469505143 shutdown_connections 2024-04-06T12:27:23.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.041+0000 7f9ca8c12700 1 --2- 172.21.15.189:0/469505143 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ca4146dd0 0x7f9ca41f2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:23.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.041+0000 7f9ca8c12700 1 --2- 172.21.15.189:0/469505143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ca40cedf0 0x7f9ca40ae5b0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:23.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.041+0000 7f9ca8c12700 1 --2- 172.21.15.189:0/469505143 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c8406adb0 0x7f9c8406d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:23.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.041+0000 7f9ca8c12700 1 --2- 172.21.15.189:0/469505143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ca40d17a0 0x7f9ca40cdea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:23.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.041+0000 7f9ca8c12700 1 -- 172.21.15.189:0/469505143 >> 172.21.15.189:0/469505143 conn(0x7f9ca40b2db0 msgr2=0x7f9ca41607c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:23.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.041+0000 7f9ca8c12700 1 -- 172.21.15.189:0/469505143 shutdown_connections 2024-04-06T12:27:23.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.041+0000 7f9ca8c12700 1 -- 172.21.15.189:0/469505143 wait complete. 2024-04-06T12:27:23.053 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-06T12:27:23.053 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-06T12:27:23.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.182+0000 7f83a3232700 1 Processor -- start 2024-04-06T12:27:23.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.182+0000 7f83a3232700 1 -- start start 2024-04-06T12:27:23.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.183+0000 7f83a3232700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f839c144f40 0x7f839c06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:23.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.183+0000 7f83a3232700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f839c15c150 0x7f839c06ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:23.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.183+0000 7f83a3232700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839c15c740 con 0x7f839c15fcd0 2024-04-06T12:27:23.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.183+0000 7f83a3232700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f839c0cfcb0 con 0x7f839c144f40 2024-04-06T12:27:23.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.183+0000 7f83a3232700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f839c15ca20 con 0x7f839c15c150 2024-04-06T12:27:23.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.183+0000 7f83a0fce700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f839c144f40 0x7f839c06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:23.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.183+0000 7f83a0fce700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f839c144f40 0x7f839c06a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37556/0 (socket says 172.21.15.189:37556) 2024-04-06T12:27:23.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.183+0000 7f83a0fce700 1 -- 172.21.15.189:0/4099115623 learned_addr learned my addr 172.21.15.189:0/4099115623 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:23.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.183+0000 7f83a17cf700 1 --2- 172.21.15.189:0/4099115623 >> v2:172.21.15.192:3300/0 conn(0x7f839c15c150 0x7f839c06ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:23.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.184+0000 7f83a0fce700 1 -- 172.21.15.189:0/4099115623 >> v2:172.21.15.192:3300/0 conn(0x7f839c15c150 msgr2=0x7f839c06ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:23.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.184+0000 7f83a0fce700 1 --2- 172.21.15.189:0/4099115623 >> v2:172.21.15.192:3300/0 conn(0x7f839c15c150 0x7f839c06ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:23.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.184+0000 7f83a0fce700 1 -- 172.21.15.189:0/4099115623 >> v1:172.21.15.192:6789/0 conn(0x7f839c15fcd0 legacy=0x7f839c15ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:23.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.184+0000 7f83a0fce700 1 -- 172.21.15.189:0/4099115623 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839c15f1f0 con 0x7f839c144f40 2024-04-06T12:27:23.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.184+0000 7f83a0fce700 1 --2- 172.21.15.189:0/4099115623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c144f40 0x7f839c06a730 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f839c0b5600 tx=0x7f8390009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be74ba3cc92e2cf server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:23.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.184+0000 7f839b7fe700 1 -- 172.21.15.189:0/4099115623 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8390004230 con 0x7f839c144f40 2024-04-06T12:27:23.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.184+0000 7f839b7fe700 1 -- 172.21.15.189:0/4099115623 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8390004390 con 0x7f839c144f40 2024-04-06T12:27:23.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.184+0000 7f839b7fe700 1 -- 172.21.15.189:0/4099115623 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83900036a0 con 0x7f839c144f40 2024-04-06T12:27:23.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.184+0000 7f83a3232700 1 -- 172.21.15.189:0/4099115623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c144f40 msgr2=0x7f839c06a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:23.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.184+0000 7f83a3232700 1 --2- 172.21.15.189:0/4099115623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c144f40 0x7f839c06a730 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f839c0b5600 tx=0x7f8390009bd0 comp rx=0 tx=0).stop 2024-04-06T12:27:23.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.184+0000 7f83a3232700 1 -- 172.21.15.189:0/4099115623 shutdown_connections 2024-04-06T12:27:23.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.184+0000 7f83a3232700 1 --2- 172.21.15.189:0/4099115623 >> v2:172.21.15.192:3300/0 conn(0x7f839c15c150 0x7f839c06ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:23.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.184+0000 7f83a3232700 1 --2- 172.21.15.189:0/4099115623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c144f40 0x7f839c06a730 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:23.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.184+0000 7f83a3232700 1 -- 172.21.15.189:0/4099115623 >> 172.21.15.189:0/4099115623 conn(0x7f839c0d00b0 msgr2=0x7f839c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:23.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.185+0000 7f83a3232700 1 -- 172.21.15.189:0/4099115623 shutdown_connections 2024-04-06T12:27:23.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.185+0000 7f83a3232700 1 -- 172.21.15.189:0/4099115623 wait complete. 2024-04-06T12:27:23.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.186+0000 7f83a3232700 1 Processor -- start 2024-04-06T12:27:23.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.186+0000 7f83a3232700 1 -- start start 2024-04-06T12:27:23.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.187+0000 7f83a3232700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f839c144f40 0x7f839c142300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:23.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.187+0000 7f83a3232700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f839c15c150 0x7f839c1430f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:23.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.187+0000 7f83a3232700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c15fcd0 0x7f839c1f0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:23.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.187+0000 7f83a3232700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c142840 con 0x7f839c15fcd0 2024-04-06T12:27:23.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.187+0000 7f83a3232700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c143630 con 0x7f839c15c150 2024-04-06T12:27:23.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.187+0000 7f83a3232700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839c0c54a0 con 0x7f839c144f40 2024-04-06T12:27:23.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.187+0000 7f83a17cf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c15fcd0 0x7f839c1f0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:23.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.187+0000 7f83a17cf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c15fcd0 0x7f839c1f0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37558/0 (socket says 172.21.15.189:37558) 2024-04-06T12:27:23.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.187+0000 7f83a17cf700 1 -- 172.21.15.189:0/3693566242 learned_addr learned my addr 172.21.15.189:0/3693566242 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:23.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.188+0000 7f83a0fce700 1 --2- 172.21.15.189:0/3693566242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f839c144f40 0x7f839c142300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:23.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.188+0000 7f839bfff700 1 --2- 172.21.15.189:0/3693566242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f839c15c150 0x7f839c1430f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:23.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.188+0000 7f83a17cf700 1 -- 172.21.15.189:0/3693566242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f839c144f40 msgr2=0x7f839c142300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:23.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.188+0000 7f83a17cf700 1 --2- 172.21.15.189:0/3693566242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f839c144f40 0x7f839c142300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:23.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.188+0000 7f83a17cf700 1 -- 172.21.15.189:0/3693566242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f839c15c150 msgr2=0x7f839c1430f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:23.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.188+0000 7f83a17cf700 1 --2- 172.21.15.189:0/3693566242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f839c15c150 0x7f839c1430f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:23.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.188+0000 7f83a17cf700 1 -- 172.21.15.189:0/3693566242 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8390009550 con 0x7f839c15fcd0 2024-04-06T12:27:23.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.188+0000 7f83a17cf700 1 --2- 172.21.15.189:0/3693566242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c15fcd0 0x7f839c1f0fc0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f8394007e40 tx=0x7f839400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:23.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.188+0000 7f83997fa700 1 -- 172.21.15.189:0/3693566242 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839400eea0 con 0x7f839c15fcd0 2024-04-06T12:27:23.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.188+0000 7f83997fa700 1 -- 172.21.15.189:0/3693566242 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8394014420 con 0x7f839c15fcd0 2024-04-06T12:27:23.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.188+0000 7f83a3232700 1 -- 172.21.15.189:0/3693566242 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839c0c5780 con 0x7f839c15fcd0 2024-04-06T12:27:23.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.188+0000 7f83997fa700 1 -- 172.21.15.189:0/3693566242 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83940147b0 con 0x7f839c15fcd0 2024-04-06T12:27:23.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.188+0000 7f83a3232700 1 -- 172.21.15.189:0/3693566242 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839c0c5d40 con 0x7f839c15fcd0 2024-04-06T12:27:23.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.189+0000 7f8386ffd700 1 -- 172.21.15.189:0/3693566242 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f839c067bb0 con 0x7f839c15fcd0 2024-04-06T12:27:23.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.189+0000 7f83997fa700 1 -- 172.21.15.189:0/3693566242 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f8394014910 con 0x7f839c15fcd0 2024-04-06T12:27:23.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.190+0000 7f83997fa700 1 --2- 172.21.15.189:0/3693566242 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f838006adb0 0x7f838006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:23.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.190+0000 7f83997fa700 1 -- 172.21.15.189:0/3693566242 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8394089cf0 con 0x7f839c15fcd0 2024-04-06T12:27:23.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.190+0000 7f83a0fce700 1 --2- 172.21.15.189:0/3693566242 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f838006adb0 0x7f838006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:23.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.190+0000 7f83a0fce700 1 --2- 172.21.15.189:0/3693566242 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f838006adb0 0x7f838006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8390000c00 tx=0x7f8390008030 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:23.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.194+0000 7f83997fa700 1 -- 172.21.15.189:0/3693566242 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f839408e070 con 0x7f839c15fcd0 2024-04-06T12:27:23.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:23.431+0000 7f8386ffd700 1 -- 172.21.15.189:0/3693566242 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f839c074990 con 0x7f839c15fcd0 2024-04-06T12:27:24.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.044+0000 7f83997fa700 1 -- 172.21.15.189:0/3693566242 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v31) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8394054d10 con 0x7f839c15fcd0 2024-04-06T12:27:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.046+0000 7f8386ffd700 1 -- 172.21.15.189:0/3693566242 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f838006adb0 msgr2=0x7f838006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.046+0000 7f8386ffd700 1 --2- 172.21.15.189:0/3693566242 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f838006adb0 0x7f838006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8390000c00 tx=0x7f8390008030 comp rx=0 tx=0).stop 2024-04-06T12:27:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.046+0000 7f8386ffd700 1 -- 172.21.15.189:0/3693566242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c15fcd0 msgr2=0x7f839c1f0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.047+0000 7f8386ffd700 1 --2- 172.21.15.189:0/3693566242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c15fcd0 0x7f839c1f0fc0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f8394007e40 tx=0x7f839400e3c0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.047+0000 7f8386ffd700 1 -- 172.21.15.189:0/3693566242 shutdown_connections 2024-04-06T12:27:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.047+0000 7f8386ffd700 1 --2- 172.21.15.189:0/3693566242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f839c15fcd0 0x7f839c1f0fc0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.047+0000 7f8386ffd700 1 --2- 172.21.15.189:0/3693566242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f839c144f40 0x7f839c142300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.047+0000 7f8386ffd700 1 --2- 172.21.15.189:0/3693566242 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f838006adb0 0x7f838006d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.047+0000 7f8386ffd700 1 --2- 172.21.15.189:0/3693566242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f839c15c150 0x7f839c1430f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.047+0000 7f8386ffd700 1 -- 172.21.15.189:0/3693566242 >> 172.21.15.189:0/3693566242 conn(0x7f839c0d00b0 msgr2=0x7f839c0cd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:24.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.047+0000 7f8386ffd700 1 -- 172.21.15.189:0/3693566242 shutdown_connections 2024-04-06T12:27:24.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.047+0000 7f8386ffd700 1 -- 172.21.15.189:0/3693566242 wait complete. 2024-04-06T12:27:24.059 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-06T12:27:24.059 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:27:24.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.185+0000 7fe37bd70700 1 Processor -- start 2024-04-06T12:27:24.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.185+0000 7fe37bd70700 1 -- start start 2024-04-06T12:27:24.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.186+0000 7fe37bd70700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe37415d7b0 0x7fe3740b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:24.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.186+0000 7fe37bd70700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe37406a730 0x7fe37406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:24.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.186+0000 7fe37bd70700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3740b3990 con 0x7fe37415ddf0 2024-04-06T12:27:24.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.186+0000 7fe37bd70700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe37406b0f0 con 0x7fe37415d7b0 2024-04-06T12:27:24.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.186+0000 7fe37bd70700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3740b3060 con 0x7fe37406a730 2024-04-06T12:27:24.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.186+0000 7fe379b0c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe37415d7b0 0x7fe3740b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:24.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.186+0000 7fe379b0c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe37415d7b0 0x7fe3740b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37568/0 (socket says 172.21.15.189:37568) 2024-04-06T12:27:24.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.186+0000 7fe379b0c700 1 -- 172.21.15.189:0/4192127068 learned_addr learned my addr 172.21.15.189:0/4192127068 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:24.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.186+0000 7fe379b0c700 1 -- 172.21.15.189:0/4192127068 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe37406a730 msgr2=0x7fe37406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:24.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.186+0000 7fe37930b700 1 --2- 172.21.15.189:0/4192127068 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe37406a730 0x7fe37406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.186+0000 7fe379b0c700 1 --2- 172.21.15.189:0/4192127068 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe37406a730 0x7fe37406abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.186+0000 7fe379b0c700 1 -- 172.21.15.189:0/4192127068 >> v1:172.21.15.192:6789/0 conn(0x7fe37415ddf0 legacy=0x7fe37415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.186+0000 7fe379b0c700 1 -- 172.21.15.189:0/4192127068 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3740b3340 con 0x7fe37415d7b0 2024-04-06T12:27:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.186+0000 7fe379b0c700 1 --2- 172.21.15.189:0/4192127068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe37415d7b0 0x7fe3740b3e20 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe3700098f0 tx=0x7fe370009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e7056f74916604 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.187+0000 7fe378b0a700 1 -- 172.21.15.189:0/4192127068 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe370004230 con 0x7fe37415d7b0 2024-04-06T12:27:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.187+0000 7fe378b0a700 1 -- 172.21.15.189:0/4192127068 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe370004390 con 0x7fe37415d7b0 2024-04-06T12:27:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.187+0000 7fe378b0a700 1 -- 172.21.15.189:0/4192127068 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3700036a0 con 0x7fe37415d7b0 2024-04-06T12:27:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.187+0000 7fe37bd70700 1 -- 172.21.15.189:0/4192127068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe37415d7b0 msgr2=0x7fe3740b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.187+0000 7fe37bd70700 1 --2- 172.21.15.189:0/4192127068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe37415d7b0 0x7fe3740b3e20 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe3700098f0 tx=0x7fe370009c00 comp rx=0 tx=0).stop 2024-04-06T12:27:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.187+0000 7fe37bd70700 1 -- 172.21.15.189:0/4192127068 shutdown_connections 2024-04-06T12:27:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.187+0000 7fe37bd70700 1 --2- 172.21.15.189:0/4192127068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe37415d7b0 0x7fe3740b3e20 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.187+0000 7fe37bd70700 1 --2- 172.21.15.189:0/4192127068 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe37406a730 0x7fe37406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.187+0000 7fe37bd70700 1 -- 172.21.15.189:0/4192127068 >> 172.21.15.189:0/4192127068 conn(0x7fe3740d00b0 msgr2=0x7fe3740d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:24.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.187+0000 7fe37bd70700 1 -- 172.21.15.189:0/4192127068 shutdown_connections 2024-04-06T12:27:24.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.187+0000 7fe37bd70700 1 -- 172.21.15.189:0/4192127068 wait complete. 2024-04-06T12:27:24.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.188+0000 7fe37bd70700 1 Processor -- start 2024-04-06T12:27:24.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.189+0000 7fe37bd70700 1 -- start start 2024-04-06T12:27:24.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.189+0000 7fe37bd70700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe37406a730 0x7fe37415c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:24.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.189+0000 7fe37bd70700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe37415d7b0 0x7fe3740b0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:24.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.189+0000 7fe37bd70700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe37415ddf0 0x7fe3740b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:24.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.189+0000 7fe37bd70700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3740aed00 con 0x7fe37415ddf0 2024-04-06T12:27:24.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.189+0000 7fe37bd70700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3740aefe0 con 0x7fe37415d7b0 2024-04-06T12:27:24.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.189+0000 7fe37bd70700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3740af2c0 con 0x7fe37406a730 2024-04-06T12:27:24.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.189+0000 7fe37a30d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe37415ddf0 0x7fe3740b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:24.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.189+0000 7fe37a30d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe37415ddf0 0x7fe3740b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37572/0 (socket says 172.21.15.189:37572) 2024-04-06T12:27:24.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.189+0000 7fe37a30d700 1 -- 172.21.15.189:0/1260575969 learned_addr learned my addr 172.21.15.189:0/1260575969 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:24.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.190+0000 7fe379b0c700 1 --2- 172.21.15.189:0/1260575969 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe37406a730 0x7fe37415c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:24.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.190+0000 7fe37a30d700 1 -- 172.21.15.189:0/1260575969 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe37406a730 msgr2=0x7fe37415c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:24.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.190+0000 7fe37a30d700 1 --2- 172.21.15.189:0/1260575969 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe37406a730 0x7fe37415c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.190+0000 7fe37930b700 1 --2- 172.21.15.189:0/1260575969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe37415d7b0 0x7fe3740b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:24.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.190+0000 7fe37a30d700 1 -- 172.21.15.189:0/1260575969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe37415d7b0 msgr2=0x7fe3740b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:24.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.190+0000 7fe37a30d700 1 --2- 172.21.15.189:0/1260575969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe37415d7b0 0x7fe3740b0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.190+0000 7fe37a30d700 1 -- 172.21.15.189:0/1260575969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe370009550 con 0x7fe37415ddf0 2024-04-06T12:27:24.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.190+0000 7fe37a30d700 1 --2- 172.21.15.189:0/1260575969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe37415ddf0 0x7fe3740b0b40 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fe36c00b970 tx=0x7fe36c00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:24.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.190+0000 7fe36a7fc700 1 -- 172.21.15.189:0/1260575969 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36c00c9b0 con 0x7fe37415ddf0 2024-04-06T12:27:24.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.190+0000 7fe36a7fc700 1 -- 172.21.15.189:0/1260575969 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36c00cb10 con 0x7fe37415ddf0 2024-04-06T12:27:24.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.190+0000 7fe36a7fc700 1 -- 172.21.15.189:0/1260575969 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36c014440 con 0x7fe37415ddf0 2024-04-06T12:27:24.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.190+0000 7fe37bd70700 1 -- 172.21.15.189:0/1260575969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3740af5a0 con 0x7fe37415ddf0 2024-04-06T12:27:24.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.190+0000 7fe37bd70700 1 -- 172.21.15.189:0/1260575969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe37415c640 con 0x7fe37415ddf0 2024-04-06T12:27:24.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.191+0000 7fe37bd70700 1 -- 172.21.15.189:0/1260575969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3740cc500 con 0x7fe37415ddf0 2024-04-06T12:27:24.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.191+0000 7fe36a7fc700 1 -- 172.21.15.189:0/1260575969 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fe36c01d3b0 con 0x7fe37415ddf0 2024-04-06T12:27:24.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.192+0000 7fe36a7fc700 1 --2- 172.21.15.189:0/1260575969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe35806adb0 0x7fe35806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:24.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.192+0000 7fe36a7fc700 1 -- 172.21.15.189:0/1260575969 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe36c089210 con 0x7fe37415ddf0 2024-04-06T12:27:24.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.192+0000 7fe379b0c700 1 --2- 172.21.15.189:0/1260575969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe35806adb0 0x7fe35806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:24.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.192+0000 7fe379b0c700 1 --2- 172.21.15.189:0/1260575969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe35806adb0 0x7fe35806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe370009c40 tx=0x7fe370008030 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:24.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.196+0000 7fe36a7fc700 1 -- 172.21.15.189:0/1260575969 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe36c054230 con 0x7fe37415ddf0 2024-04-06T12:27:24.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.436+0000 7fe37bd70700 1 -- 172.21.15.189:0/1260575969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe374067bb0 con 0x7fe37415ddf0 2024-04-06T12:27:24.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.437+0000 7fe36a7fc700 1 -- 172.21.15.189:0/1260575969 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4747 (secure 0 0 0) 0x7fe36c057850 con 0x7fe37415ddf0 2024-04-06T12:27:24.436 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:24.436 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5762,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/2827078766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2827078766},{"type":"v1","addr":"172.21.15.192:6833","nonce":2827078766}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5765,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1558111999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1558111999},{"type":"v1","addr":"172.21.15.192:6835","nonce":1558111999}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3448511259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3448511259},{"type":"v1","addr":"172.21.15.189:6837","nonce":3448511259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:27:21.031012+0000","modified":"2024-04-06T12:27:24.038778+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/1085319880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1085319880},{"type":"v1","addr":"172.21.15.189:6836","nonce":1085319880}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-06T12:27:24.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.439+0000 7fe37bd70700 1 -- 172.21.15.189:0/1260575969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe35806adb0 msgr2=0x7fe35806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:24.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.439+0000 7fe37bd70700 1 --2- 172.21.15.189:0/1260575969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe35806adb0 0x7fe35806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe370009c40 tx=0x7fe370008030 comp rx=0 tx=0).stop 2024-04-06T12:27:24.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.439+0000 7fe37bd70700 1 -- 172.21.15.189:0/1260575969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe37415ddf0 msgr2=0x7fe3740b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:24.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.439+0000 7fe37bd70700 1 --2- 172.21.15.189:0/1260575969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe37415ddf0 0x7fe3740b0b40 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fe36c00b970 tx=0x7fe36c00bd30 comp rx=0 tx=0).stop 2024-04-06T12:27:24.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.439+0000 7fe37bd70700 1 -- 172.21.15.189:0/1260575969 shutdown_connections 2024-04-06T12:27:24.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.439+0000 7fe37bd70700 1 --2- 172.21.15.189:0/1260575969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe37415ddf0 0x7fe3740b0b40 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.439+0000 7fe37bd70700 1 --2- 172.21.15.189:0/1260575969 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe37406a730 0x7fe37415c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.439+0000 7fe37bd70700 1 --2- 172.21.15.189:0/1260575969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe35806adb0 0x7fe35806d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.439+0000 7fe37bd70700 1 --2- 172.21.15.189:0/1260575969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe37415d7b0 0x7fe3740b0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.439+0000 7fe37bd70700 1 -- 172.21.15.189:0/1260575969 >> 172.21.15.189:0/1260575969 conn(0x7fe3740d00b0 msgr2=0x7fe3740adc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:24.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.439+0000 7fe37bd70700 1 -- 172.21.15.189:0/1260575969 shutdown_connections 2024-04-06T12:27:24.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.439+0000 7fe37bd70700 1 -- 172.21.15.189:0/1260575969 wait complete. 2024-04-06T12:27:24.439 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 31 2024-04-06T12:27:24.449 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:27:24.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.574+0000 7f7431832700 1 Processor -- start 2024-04-06T12:27:24.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.574+0000 7f7431832700 1 -- start start 2024-04-06T12:27:24.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.574+0000 7f7431832700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f742c15db10 0x7f742c0b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:24.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.574+0000 7f7431832700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f742c143470 0x7f742c0cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:24.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.574+0000 7f7431832700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f742c15eb40 con 0x7f742c160b60 2024-04-06T12:27:24.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.574+0000 7f7431832700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f742c160f40 con 0x7f742c143470 2024-04-06T12:27:24.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.574+0000 7f7431832700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f742c0d0bf0 con 0x7f742c15db10 2024-04-06T12:27:24.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.575+0000 7f742b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f742c143470 0x7f742c0cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:24.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.575+0000 7f742affd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f742c160b60 0x7f742c142d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56888/0 (socket says 172.21.15.189:56888) 2024-04-06T12:27:24.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.575+0000 7f742affd700 1 -- 172.21.15.189:0/369447203 learned_addr learned my addr 172.21.15.189:0/369447203 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:24.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.575+0000 7f742a7fc700 1 --2- 172.21.15.189:0/369447203 >> v2:172.21.15.192:3300/0 conn(0x7f742c15db10 0x7f742c0b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:24.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.575+0000 7f7429ffb700 1 -- 172.21.15.189:0/369447203 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198869614 0 0) 0x7f741c001110 con 0x7f742c160b60 2024-04-06T12:27:24.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.575+0000 7f7429ffb700 1 -- 172.21.15.189:0/369447203 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74180035b0 con 0x7f742c160b60 2024-04-06T12:27:24.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.575+0000 7f742b7fe700 1 -- 172.21.15.189:0/369447203 >> v2:172.21.15.192:3300/0 conn(0x7f742c15db10 msgr2=0x7f742c0b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:24.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.575+0000 7f742b7fe700 1 --2- 172.21.15.189:0/369447203 >> v2:172.21.15.192:3300/0 conn(0x7f742c15db10 0x7f742c0b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.575+0000 7f742b7fe700 1 -- 172.21.15.189:0/369447203 >> v1:172.21.15.189:6789/0 conn(0x7f742c160b60 legacy=0x7f742c142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:24.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.575+0000 7f742b7fe700 1 -- 172.21.15.189:0/369447203 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742c15dfc0 con 0x7f742c143470 2024-04-06T12:27:24.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.575+0000 7f742b7fe700 1 --2- 172.21.15.189:0/369447203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742c143470 0x7f742c0cd890 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f742c147320 tx=0x7f742400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f31a29e5a9045856 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:24.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.575+0000 7f7429ffb700 1 -- 172.21.15.189:0/369447203 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742400e070 con 0x7f742c143470 2024-04-06T12:27:24.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.575+0000 7f7429ffb700 1 -- 172.21.15.189:0/369447203 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742400bb20 con 0x7f742c143470 2024-04-06T12:27:24.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.575+0000 7f7429ffb700 1 -- 172.21.15.189:0/369447203 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7424012430 con 0x7f742c143470 2024-04-06T12:27:24.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.576+0000 7f7431832700 1 -- 172.21.15.189:0/369447203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742c143470 msgr2=0x7f742c0cd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:24.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.576+0000 7f7431832700 1 --2- 172.21.15.189:0/369447203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742c143470 0x7f742c0cd890 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f742c147320 tx=0x7f742400b330 comp rx=0 tx=0).stop 2024-04-06T12:27:24.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.576+0000 7f7431832700 1 -- 172.21.15.189:0/369447203 shutdown_connections 2024-04-06T12:27:24.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.576+0000 7f7431832700 1 --2- 172.21.15.189:0/369447203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742c143470 0x7f742c0cd890 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.576+0000 7f7431832700 1 --2- 172.21.15.189:0/369447203 >> v2:172.21.15.192:3300/0 conn(0x7f742c15db10 0x7f742c0b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.576+0000 7f7431832700 1 -- 172.21.15.189:0/369447203 >> 172.21.15.189:0/369447203 conn(0x7f742c06a730 msgr2=0x7f742c06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:24.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.576+0000 7f7431832700 1 -- 172.21.15.189:0/369447203 shutdown_connections 2024-04-06T12:27:24.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.576+0000 7f7431832700 1 -- 172.21.15.189:0/369447203 wait complete. 2024-04-06T12:27:24.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.577+0000 7f7431832700 1 Processor -- start 2024-04-06T12:27:24.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.578+0000 7f7431832700 1 -- start start 2024-04-06T12:27:24.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.578+0000 7f7431832700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f742c143470 0x7f742c14f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:24.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.578+0000 7f7431832700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742c15db10 0x7f742c0c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:24.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.578+0000 7f7431832700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f742c160b60 0x7f742c0b1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:24.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.578+0000 7f7431832700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742c0c7850 con 0x7f742c15db10 2024-04-06T12:27:24.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.578+0000 7f7431832700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742c0c6bd0 con 0x7f742c143470 2024-04-06T12:27:24.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.578+0000 7f7431832700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742c0c6eb0 con 0x7f742c160b60 2024-04-06T12:27:24.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.579+0000 7f742a7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742c15db10 0x7f742c0c4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:24.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.579+0000 7f742a7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742c15db10 0x7f742c0c4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37586/0 (socket says 172.21.15.189:37586) 2024-04-06T12:27:24.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.579+0000 7f742a7fc700 1 -- 172.21.15.189:0/597847297 learned_addr learned my addr 172.21.15.189:0/597847297 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:24.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.579+0000 7f742affd700 1 --2- 172.21.15.189:0/597847297 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f742c143470 0x7f742c14f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:24.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.579+0000 7f742b7fe700 1 --2- 172.21.15.189:0/597847297 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f742c160b60 0x7f742c0b1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:24.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.579+0000 7f742a7fc700 1 -- 172.21.15.189:0/597847297 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f742c160b60 msgr2=0x7f742c0b1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:24.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.579+0000 7f742a7fc700 1 --2- 172.21.15.189:0/597847297 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f742c160b60 0x7f742c0b1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.579+0000 7f742a7fc700 1 -- 172.21.15.189:0/597847297 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f742c143470 msgr2=0x7f742c14f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:24.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.579+0000 7f742a7fc700 1 --2- 172.21.15.189:0/597847297 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f742c143470 0x7f742c14f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.579+0000 7f742a7fc700 1 -- 172.21.15.189:0/597847297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7424009a70 con 0x7f742c15db10 2024-04-06T12:27:24.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.579+0000 7f742a7fc700 1 --2- 172.21.15.189:0/597847297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742c15db10 0x7f742c0c4d50 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f741400e730 tx=0x7f741400eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:24.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.579+0000 7f7423fff700 1 -- 172.21.15.189:0/597847297 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7414013610 con 0x7f742c15db10 2024-04-06T12:27:24.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.579+0000 7f7423fff700 1 -- 172.21.15.189:0/597847297 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7414013770 con 0x7f742c15db10 2024-04-06T12:27:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.579+0000 7f7423fff700 1 -- 172.21.15.189:0/597847297 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7414013b20 con 0x7f742c15db10 2024-04-06T12:27:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.579+0000 7f7431832700 1 -- 172.21.15.189:0/597847297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f742c0c7190 con 0x7f742c15db10 2024-04-06T12:27:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.579+0000 7f7431832700 1 -- 172.21.15.189:0/597847297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f742c15d0f0 con 0x7f742c15db10 2024-04-06T12:27:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.580+0000 7f7431832700 1 -- 172.21.15.189:0/597847297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f742c0c7ac0 con 0x7f742c15db10 2024-04-06T12:27:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.581+0000 7f7423fff700 1 -- 172.21.15.189:0/597847297 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7414013c80 con 0x7f742c15db10 2024-04-06T12:27:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.581+0000 7f7423fff700 1 --2- 172.21.15.189:0/597847297 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f740c06adb0 0x7f740c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.581+0000 7f7423fff700 1 -- 172.21.15.189:0/597847297 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7414088fc0 con 0x7f742c15db10 2024-04-06T12:27:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.581+0000 7f742affd700 1 --2- 172.21.15.189:0/597847297 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f740c06adb0 0x7f740c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.581+0000 7f742affd700 1 --2- 172.21.15.189:0/597847297 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f740c06adb0 0x7f740c06d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f741c001270 tx=0x7f741c009040 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.585+0000 7f7423fff700 1 -- 172.21.15.189:0/597847297 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7414053ed0 con 0x7f742c15db10 2024-04-06T12:27:24.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.780+0000 7f7431832700 1 -- 172.21.15.189:0/597847297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f742c074990 con 0x7f742c15db10 2024-04-06T12:27:24.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.781+0000 7f7423fff700 1 -- 172.21.15.189:0/597847297 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16296 (secure 0 0 0) 0x7f7414021090 con 0x7f742c15db10 2024-04-06T12:27:24.779 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:24.779 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":84,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:27:22.032887+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:27:19.575213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-06T12:27:19.990714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:27:24.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.783+0000 7f7431832700 1 -- 172.21.15.189:0/597847297 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f740c06adb0 msgr2=0x7f740c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:24.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.783+0000 7f7431832700 1 --2- 172.21.15.189:0/597847297 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f740c06adb0 0x7f740c06d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f741c001270 tx=0x7f741c009040 comp rx=0 tx=0).stop 2024-04-06T12:27:24.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.783+0000 7f7431832700 1 -- 172.21.15.189:0/597847297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742c15db10 msgr2=0x7f742c0c4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:24.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.783+0000 7f7431832700 1 --2- 172.21.15.189:0/597847297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742c15db10 0x7f742c0c4d50 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f741400e730 tx=0x7f741400eaf0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.783+0000 7f7431832700 1 -- 172.21.15.189:0/597847297 shutdown_connections 2024-04-06T12:27:24.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.783+0000 7f7431832700 1 --2- 172.21.15.189:0/597847297 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f742c160b60 0x7f742c0b1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.783+0000 7f7431832700 1 --2- 172.21.15.189:0/597847297 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f740c06adb0 0x7f740c06d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.783+0000 7f7431832700 1 --2- 172.21.15.189:0/597847297 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f742c143470 0x7f742c14f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.783+0000 7f7431832700 1 --2- 172.21.15.189:0/597847297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742c15db10 0x7f742c0c4d50 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.783+0000 7f7431832700 1 -- 172.21.15.189:0/597847297 >> 172.21.15.189:0/597847297 conn(0x7f742c06a730 msgr2=0x7f742c0b2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:24.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.783+0000 7f7431832700 1 -- 172.21.15.189:0/597847297 shutdown_connections 2024-04-06T12:27:24.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.783+0000 7f7431832700 1 -- 172.21.15.189:0/597847297 wait complete. 2024-04-06T12:27:24.794 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-06T12:27:24.794 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:27:24.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.918+0000 7f0add9df700 1 Processor -- start 2024-04-06T12:27:24.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.918+0000 7f0add9df700 1 -- start start 2024-04-06T12:27:24.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.919+0000 7f0add9df700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0ad80b4ed0 0x7f0ad80b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:24.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.919+0000 7f0add9df700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad80b5cc0 con 0x7f0ad80d2880 2024-04-06T12:27:24.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.919+0000 7f0add9df700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad80d1500 con 0x7f0ad815d590 2024-04-06T12:27:24.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.919+0000 7f0add9df700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad80b58c0 con 0x7f0ad80b4ed0 2024-04-06T12:27:24.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.919+0000 7f0ad67fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0ad80b4ed0 0x7f0ad80b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:24.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.919+0000 7f0ad77fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0ad80d2880 0x7f0ad80d2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56892/0 (socket says 172.21.15.189:56892) 2024-04-06T12:27:24.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.919+0000 7f0ad77fe700 1 -- 172.21.15.189:0/2635211431 learned_addr learned my addr 172.21.15.189:0/2635211431 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:24.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.919+0000 7f0ad5ffb700 1 -- 172.21.15.189:0/2635211431 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026732740 0 0) 0x7f0ad80b5cc0 con 0x7f0ad80d2880 2024-04-06T12:27:24.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.919+0000 7f0ad5ffb700 1 -- 172.21.15.189:0/2635211431 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ac00035b0 con 0x7f0ad80d2880 2024-04-06T12:27:24.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.919+0000 7f0ad67fc700 1 -- 172.21.15.189:0/2635211431 >> v1:172.21.15.192:6789/0 conn(0x7f0ad815d590 legacy=0x7f0ad80b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:24.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.919+0000 7f0ad67fc700 1 -- 172.21.15.189:0/2635211431 >> v1:172.21.15.189:6789/0 conn(0x7f0ad80d2880 legacy=0x7f0ad80d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:24.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.919+0000 7f0ad67fc700 1 -- 172.21.15.189:0/2635211431 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad80d17e0 con 0x7f0ad80b4ed0 2024-04-06T12:27:24.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.919+0000 7f0ad67fc700 1 --2- 172.21.15.189:0/2635211431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ad80b4ed0 0x7f0ad80b5380 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009840 tx=0x7f0ac8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a98c9342f2009579 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:24.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.920+0000 7f0ad5ffb700 1 -- 172.21.15.189:0/2635211431 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8011830 con 0x7f0ad80b4ed0 2024-04-06T12:27:24.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.920+0000 7f0ad5ffb700 1 -- 172.21.15.189:0/2635211431 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac8011990 con 0x7f0ad80b4ed0 2024-04-06T12:27:24.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.920+0000 7f0ad5ffb700 1 -- 172.21.15.189:0/2635211431 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8011d20 con 0x7f0ad80b4ed0 2024-04-06T12:27:24.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.920+0000 7f0add9df700 1 -- 172.21.15.189:0/2635211431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ad80b4ed0 msgr2=0x7f0ad80b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:24.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.920+0000 7f0add9df700 1 --2- 172.21.15.189:0/2635211431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ad80b4ed0 0x7f0ad80b5380 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009840 tx=0x7f0ac8009b50 comp rx=0 tx=0).stop 2024-04-06T12:27:24.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.920+0000 7f0add9df700 1 -- 172.21.15.189:0/2635211431 shutdown_connections 2024-04-06T12:27:24.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.920+0000 7f0add9df700 1 --2- 172.21.15.189:0/2635211431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ad80b4ed0 0x7f0ad80b5380 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.920+0000 7f0add9df700 1 -- 172.21.15.189:0/2635211431 >> 172.21.15.189:0/2635211431 conn(0x7f0ad806bbc0 msgr2=0x7f0ad806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:24.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.920+0000 7f0add9df700 1 -- 172.21.15.189:0/2635211431 shutdown_connections 2024-04-06T12:27:24.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.920+0000 7f0add9df700 1 -- 172.21.15.189:0/2635211431 wait complete. 2024-04-06T12:27:24.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.921+0000 7f0add9df700 1 Processor -- start 2024-04-06T12:27:24.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.922+0000 7f0add9df700 1 -- start start 2024-04-06T12:27:24.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.922+0000 7f0add9df700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ad80b4ed0 0x7f0ad81448e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:24.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.922+0000 7f0add9df700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ad80d2880 0x7f0ad80cc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:24.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.922+0000 7f0add9df700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ad815d590 0x7f0ad80ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:24.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.922+0000 7f0add9df700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad815de50 con 0x7f0ad80d2880 2024-04-06T12:27:24.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.922+0000 7f0add9df700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad815e130 con 0x7f0ad80b4ed0 2024-04-06T12:27:24.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.922+0000 7f0add9df700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad815e410 con 0x7f0ad815d590 2024-04-06T12:27:24.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.922+0000 7f0ad67fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ad80d2880 0x7f0ad80cc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:24.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.922+0000 7f0ad67fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ad80d2880 0x7f0ad80cc6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37602/0 (socket says 172.21.15.189:37602) 2024-04-06T12:27:24.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.922+0000 7f0ad67fc700 1 -- 172.21.15.189:0/2278761532 learned_addr learned my addr 172.21.15.189:0/2278761532 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:24.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.923+0000 7f0ad67fc700 1 -- 172.21.15.189:0/2278761532 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ad815d590 msgr2=0x7f0ad80ccc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:24.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.923+0000 7f0ad67fc700 1 --2- 172.21.15.189:0/2278761532 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ad815d590 0x7f0ad80ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.923+0000 7f0ad67fc700 1 -- 172.21.15.189:0/2278761532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ad80b4ed0 msgr2=0x7f0ad81448e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:24.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.923+0000 7f0ad67fc700 1 --2- 172.21.15.189:0/2278761532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ad80b4ed0 0x7f0ad81448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:24.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.923+0000 7f0ad67fc700 1 -- 172.21.15.189:0/2278761532 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac80094a0 con 0x7f0ad80d2880 2024-04-06T12:27:24.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.923+0000 7f0ad67fc700 1 --2- 172.21.15.189:0/2278761532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ad80d2880 0x7f0ad80cc6f0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009fd0 tx=0x7f0ac8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:24.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.923+0000 7f0abffff700 1 -- 172.21.15.189:0/2278761532 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac80038a0 con 0x7f0ad80d2880 2024-04-06T12:27:24.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.923+0000 7f0abffff700 1 -- 172.21.15.189:0/2278761532 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac801f070 con 0x7f0ad80d2880 2024-04-06T12:27:24.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.923+0000 7f0add9df700 1 -- 172.21.15.189:0/2278761532 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad815e6f0 con 0x7f0ad80d2880 2024-04-06T12:27:24.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.923+0000 7f0abffff700 1 -- 172.21.15.189:0/2278761532 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8003c40 con 0x7f0ad80d2880 2024-04-06T12:27:24.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.923+0000 7f0add9df700 1 -- 172.21.15.189:0/2278761532 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad815d970 con 0x7f0ad80d2880 2024-04-06T12:27:24.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.924+0000 7f0add9df700 1 -- 172.21.15.189:0/2278761532 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad815ec10 con 0x7f0ad80d2880 2024-04-06T12:27:24.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.925+0000 7f0abffff700 1 -- 172.21.15.189:0/2278761532 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0ac8026480 con 0x7f0ad80d2880 2024-04-06T12:27:24.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.925+0000 7f0abffff700 1 --2- 172.21.15.189:0/2278761532 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0ac406adb0 0x7f0ac406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:24.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.925+0000 7f0abffff700 1 -- 172.21.15.189:0/2278761532 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0ac8021070 con 0x7f0ad80d2880 2024-04-06T12:27:24.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.925+0000 7f0ad6ffd700 1 --2- 172.21.15.189:0/2278761532 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0ac406adb0 0x7f0ac406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:24.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.925+0000 7f0ad6ffd700 1 --2- 172.21.15.189:0/2278761532 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0ac406adb0 0x7f0ac406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0acc000f60 tx=0x7f0acc007a40 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:24.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:24.929+0000 7f0abffff700 1 -- 172.21.15.189:0/2278761532 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0ad815ec10 con 0x7f0ad80d2880 2024-04-06T12:27:25.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.121+0000 7f0add9df700 1 -- 172.21.15.189:0/2278761532 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0ad815e2a0 con 0x7f0ac406adb0 2024-04-06T12:27:25.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.129+0000 7f0abffff700 1 -- 172.21.15.189:0/2278761532 <== mgr.5582 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278503 (secure 0 0 0) 0x7f0ad815e2a0 con 0x7f0ac406adb0 2024-04-06T12:27:25.131 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:25.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.132+0000 7f0add9df700 1 -- 172.21.15.189:0/2278761532 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0ac406adb0 msgr2=0x7f0ac406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:25.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.132+0000 7f0add9df700 1 --2- 172.21.15.189:0/2278761532 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0ac406adb0 0x7f0ac406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0acc000f60 tx=0x7f0acc007a40 comp rx=0 tx=0).stop 2024-04-06T12:27:25.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.132+0000 7f0add9df700 1 -- 172.21.15.189:0/2278761532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ad80d2880 msgr2=0x7f0ad80cc6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:25.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.132+0000 7f0add9df700 1 --2- 172.21.15.189:0/2278761532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ad80d2880 0x7f0ad80cc6f0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009fd0 tx=0x7f0ac8004750 comp rx=0 tx=0).stop 2024-04-06T12:27:25.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.132+0000 7f0add9df700 1 -- 172.21.15.189:0/2278761532 shutdown_connections 2024-04-06T12:27:25.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.132+0000 7f0add9df700 1 --2- 172.21.15.189:0/2278761532 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ad815d590 0x7f0ad80ccc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.132+0000 7f0add9df700 1 --2- 172.21.15.189:0/2278761532 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0ac406adb0 0x7f0ac406d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.132+0000 7f0add9df700 1 --2- 172.21.15.189:0/2278761532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ad80b4ed0 0x7f0ad81448e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.132+0000 7f0add9df700 1 --2- 172.21.15.189:0/2278761532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ad80d2880 0x7f0ad80cc6f0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.132+0000 7f0add9df700 1 -- 172.21.15.189:0/2278761532 >> 172.21.15.189:0/2278761532 conn(0x7f0ad806bbc0 msgr2=0x7f0ad80b6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:25.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.132+0000 7f0add9df700 1 -- 172.21.15.189:0/2278761532 shutdown_connections 2024-04-06T12:27:25.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.132+0000 7f0add9df700 1 -- 172.21.15.189:0/2278761532 wait complete. 2024-04-06T12:27:25.134 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:27:25.143 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-04-06T12:27:23.309004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":315952,"kb_used_data":8748,"kb_used_omap":12,"kb_used_meta":307059,"kb_avail":754658768,"statfs":{"total":773094113280,"available":772770578432,"internally_reserved":0,"allocated":8957952,"data_stored":5013131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":314428984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000647"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.627611+0000","last_change":"2024-04-06T12:27:19.627611+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.627611+0000","last_undegraded":"2024-04-06T12:27:19.627611+0000","last_fullsized":"2024-04-06T12:27:19.627611+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:24.851744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.626305+0000","last_change":"2024-04-06T12:27:19.626305+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.626305+0000","last_undegraded":"2024-04-06T12:27:19.626305+0000","last_fullsized":"2024-04-06T12:27:19.626305+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:23:10.897604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.628420+0000","last_change":"2024-04-06T12:27:19.628420+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.628420+0000","last_undegraded":"2024-04-06T12:27:19.628420+0000","last_fullsized":"2024-04-06T12:27:19.628420+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:44:45.550106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.626356+0000","last_change":"2024-04-06T12:27:19.626356+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.626356+0000","last_undegraded":"2024-04-06T12:27:19.626356+0000","last_fullsized":"2024-04-06T12:27:19.626356+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:00:13.506742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623826+0000","last_change":"2024-04-06T12:27:19.623826+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623826+0000","last_undegraded":"2024-04-06T12:27:19.623826+0000","last_fullsized":"2024-04-06T12:27:19.623826+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:40:56.649109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.624561+0000","last_change":"2024-04-06T12:27:19.624561+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.624561+0000","last_undegraded":"2024-04-06T12:27:19.624561+0000","last_fullsized":"2024-04-06T12:27:19.624561+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:14:39.407591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.624252+0000","last_change":"2024-04-06T12:27:19.624252+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.624252+0000","last_undegraded":"2024-04-06T12:27:19.624252+0000","last_fullsized":"2024-04-06T12:27:19.624252+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:39:48.790443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.629210+0000","last_change":"2024-04-06T12:27:19.629210+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.629210+0000","last_undegraded":"2024-04-06T12:27:19.629210+0000","last_fullsized":"2024-04-06T12:27:19.629210+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:53:08.693479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623334+0000","last_change":"2024-04-06T12:27:19.623334+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623334+0000","last_undegraded":"2024-04-06T12:27:19.623334+0000","last_fullsized":"2024-04-06T12:27:19.623334+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:24.851744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.626869+0000","last_change":"2024-04-06T12:27:19.626869+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.626869+0000","last_undegraded":"2024-04-06T12:27:19.626869+0000","last_fullsized":"2024-04-06T12:27:19.626869+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:54:55.715540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.624131+0000","last_change":"2024-04-06T12:27:19.624131+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.624131+0000","last_undegraded":"2024-04-06T12:27:19.624131+0000","last_fullsized":"2024-04-06T12:27:19.624131+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:15.542425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.627800+0000","last_change":"2024-04-06T12:27:19.627800+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.627800+0000","last_undegraded":"2024-04-06T12:27:19.627800+0000","last_fullsized":"2024-04-06T12:27:19.627800+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:48:54.260905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623703+0000","last_change":"2024-04-06T12:27:19.623703+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623703+0000","last_undegraded":"2024-04-06T12:27:19.623703+0000","last_fullsized":"2024-04-06T12:27:19.623703+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:48:54.260905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623431+0000","last_change":"2024-04-06T12:27:19.623431+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623431+0000","last_undegraded":"2024-04-06T12:27:19.623431+0000","last_fullsized":"2024-04-06T12:27:19.623431+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:48.861736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.622973+0000","last_change":"2024-04-06T12:27:19.622973+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.622973+0000","last_undegraded":"2024-04-06T12:27:19.622973+0000","last_fullsized":"2024-04-06T12:27:19.622973+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:55.070413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.626419+0000","last_change":"2024-04-06T12:27:19.626419+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.626419+0000","last_undegraded":"2024-04-06T12:27:19.626419+0000","last_fullsized":"2024-04-06T12:27:19.626419+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:01:44.323843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623399+0000","last_change":"2024-04-06T12:27:19.623399+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623399+0000","last_undegraded":"2024-04-06T12:27:19.623399+0000","last_fullsized":"2024-04-06T12:27:19.623399+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:55.070413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.622989+0000","last_change":"2024-04-06T12:27:19.622989+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.622989+0000","last_undegraded":"2024-04-06T12:27:19.622989+0000","last_fullsized":"2024-04-06T12:27:19.622989+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:29:32.486815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623711+0000","last_change":"2024-04-06T12:27:19.623711+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623711+0000","last_undegraded":"2024-04-06T12:27:19.623711+0000","last_fullsized":"2024-04-06T12:27:19.623711+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:44.931614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623814+0000","last_change":"2024-04-06T12:27:19.623814+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623814+0000","last_undegraded":"2024-04-06T12:27:19.623814+0000","last_fullsized":"2024-04-06T12:27:19.623814+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:44.642507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.626259+0000","last_change":"2024-04-06T12:27:19.626259+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.626259+0000","last_undegraded":"2024-04-06T12:27:19.626259+0000","last_fullsized":"2024-04-06T12:27:19.626259+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:34:57.334069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.622770+0000","last_change":"2024-04-06T12:27:19.622770+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.622770+0000","last_undegraded":"2024-04-06T12:27:19.622770+0000","last_fullsized":"2024-04-06T12:27:19.622770+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:12:17.891288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.626381+0000","last_change":"2024-04-06T12:27:19.626381+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.626381+0000","last_undegraded":"2024-04-06T12:27:19.626381+0000","last_fullsized":"2024-04-06T12:27:19.626381+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:26:31.570422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.622822+0000","last_change":"2024-04-06T12:27:19.622822+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.622822+0000","last_undegraded":"2024-04-06T12:27:19.622822+0000","last_fullsized":"2024-04-06T12:27:19.622822+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:46:21.677982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.622770+0000","last_change":"2024-04-06T12:27:19.622770+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.622770+0000","last_undegraded":"2024-04-06T12:27:19.622770+0000","last_fullsized":"2024-04-06T12:27:19.622770+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:16:14.006637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.626396+0000","last_change":"2024-04-06T12:27:19.626396+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.626396+0000","last_undegraded":"2024-04-06T12:27:19.626396+0000","last_fullsized":"2024-04-06T12:27:19.626396+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:08:18.449996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.627417+0000","last_change":"2024-04-06T12:27:19.627417+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.627417+0000","last_undegraded":"2024-04-06T12:27:19.627417+0000","last_fullsized":"2024-04-06T12:27:19.627417+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:15.542425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623238+0000","last_change":"2024-04-06T12:27:19.623238+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623238+0000","last_undegraded":"2024-04-06T12:27:19.623238+0000","last_fullsized":"2024-04-06T12:27:19.623238+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:48.861736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623790+0000","last_change":"2024-04-06T12:27:19.623790+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623790+0000","last_undegraded":"2024-04-06T12:27:19.623790+0000","last_fullsized":"2024-04-06T12:27:19.623790+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:20:08.218629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623804+0000","last_change":"2024-04-06T12:27:19.623804+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623804+0000","last_undegraded":"2024-04-06T12:27:19.623804+0000","last_fullsized":"2024-04-06T12:27:19.623804+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:56:37.738753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.627296+0000","last_change":"2024-04-06T12:27:19.627296+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.627296+0000","last_undegraded":"2024-04-06T12:27:19.627296+0000","last_fullsized":"2024-04-06T12:27:19.627296+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:01.544743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623833+0000","last_change":"2024-04-06T12:27:19.623833+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623833+0000","last_undegraded":"2024-04-06T12:27:19.623833+0000","last_fullsized":"2024-04-06T12:27:19.623833+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:27:16.320148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'161","reported_seq":388,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-06T12:27:17.742656+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:27:17.742656+0000","last_peered":"2024-04-06T12:27:17.742656+0000","last_clean":"2024-04-06T12:27:17.742656+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:27:17.742656+0000","last_undegraded":"2024-04-06T12:27:17.742656+0000","last_fullsized":"2024-04-06T12:27:17.742656+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575034,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58900,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312940,"statfs":{"total":96636764160,"available":96576450560,"internally_reserved":0,"allocated":1564672,"data_stored":1068520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55800000000000005,"15min":0.54900000000000004},"min":{"1min":0.375,"5min":0.219,"15min":0.219},"max":{"1min":0.80100000000000005,"5min":1.143,"15min":1.143},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55700000000000005,"15min":0.55300000000000005},"min":{"1min":0.433,"5min":0.35399999999999998,"15min":0.25900000000000001},"max":{"1min":0.73299999999999998,"5min":0.77100000000000002,"15min":0.78200000000000003},"last":0.48399999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55400000000000005,"15min":0.55100000000000005},"min":{"1min":0.42299999999999999,"5min":0.33600000000000002,"15min":0.31900000000000001},"max":{"1min":0.748,"5min":0.77100000000000002,"15min":0.81399999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.41699999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.77400000000000002,"5min":0.79400000000000004,"15min":0.79600000000000004},"last":0.372}]},{"osd":2,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56999999999999995,"15min":0.55200000000000005},"min":{"1min":0.441,"5min":0.249,"15min":0.249},"max":{"1min":0.77100000000000002,"5min":0.78500000000000003,"15min":0.83299999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57099999999999995,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.24199999999999999},"max":{"1min":0.76600000000000001,"5min":0.78300000000000003,"15min":0.82299999999999995},"last":0.58999999999999997}]},{"osd":3,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58099999999999996,"15min":0.57199999999999995},"min":{"1min":0.45900000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.75800000000000001,"5min":1.357,"15min":1.357},"last":0.502},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59299999999999997,"15min":0.59599999999999997},"min":{"1min":0.44400000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.748,"5min":0.81799999999999995,"15min":0.92000000000000004},"last":0.626}]},{"osd":4,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.45300000000000001,"15min":0.45800000000000002},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.25},"max":{"1min":0.68300000000000005,"5min":0.70399999999999996,"15min":0.83599999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.53700000000000003},"min":{"1min":0.42899999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71399999999999997,"5min":0.77700000000000002,"15min":1.0780000000000001},"last":0.54400000000000004}]},{"osd":5,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.498,"15min":0.51300000000000001},"min":{"1min":0.33600000000000002,"5min":0.29199999999999998,"15min":0.23799999999999999},"max":{"1min":0.73299999999999998,"5min":0.80100000000000005,"15min":0.98199999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55900000000000005,"15min":0.56599999999999995},"min":{"1min":0.374,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.70699999999999996,"5min":0.81599999999999995,"15min":0.93600000000000005},"last":0.60599999999999998}]},{"osd":6,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53900000000000003,"15min":0.54900000000000004},"min":{"1min":0.39300000000000002,"5min":0.33700000000000002,"15min":0.28799999999999998},"max":{"1min":0.70699999999999996,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55800000000000005,"15min":0.56799999999999995},"min":{"1min":0.40200000000000002,"5min":0.30399999999999999,"15min":0.26900000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.88500000000000001},"last":0.58099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575033,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32072,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339768,"statfs":{"total":96636764160,"available":96603922432,"internally_reserved":0,"allocated":978944,"data_stored":483270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.35299999999999998,"5min":0.26200000000000001,"15min":0.23000000000000001},"max":{"1min":0.74399999999999999,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53100000000000003,"15min":0.52900000000000003},"min":{"1min":0.36599999999999999,"5min":0.30199999999999999,"15min":0.245},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.434}]},{"osd":1,"last update":"Sat Apr 6 12:26:20 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.58099999999999996,"15min":0.57599999999999996},"min":{"1min":0.52500000000000002,"5min":0.32600000000000001,"15min":0.24399999999999999},"max":{"1min":0.74299999999999999,"5min":0.78700000000000003,"15min":0.84999999999999998},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.58999999999999997,"15min":0.58799999999999997},"min":{"1min":0.497,"5min":0.221,"15min":0.221},"max":{"1min":0.75,"5min":0.76700000000000002,"15min":0.83099999999999996},"last":0.63600000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:26:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39900000000000002,"5min":0.30499999999999999,"15min":0.29899999999999999},"max":{"1min":0.70799999999999996,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.495,"5min":0.254,"15min":0.254},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.79800000000000004},"last":0.51800000000000002}]},{"osd":3,"last update":"Sat Apr 6 12:26:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.57099999999999995,"15min":0.57799999999999996},"min":{"1min":0.42299999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.72599999999999998,"5min":0.79000000000000004,"15min":0.81200000000000006},"last":0.502},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.61399999999999999,"15min":0.61499999999999999},"min":{"1min":0.498,"5min":0.41099999999999998,"15min":0.374},"max":{"1min":0.78300000000000003,"5min":0.82799999999999996,"15min":0.872},"last":0.626}]},{"osd":4,"last update":"Sat Apr 6 12:26:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48999999999999999,"15min":0.496},"min":{"1min":0.33000000000000002,"5min":0.27500000000000002,"15min":0.27300000000000002},"max":{"1min":0.66100000000000003,"5min":0.749,"15min":0.749},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.53200000000000003},"min":{"1min":0.29599999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64000000000000001,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.61499999999999999}]},{"osd":5,"last update":"Sat Apr 6 12:26:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48499999999999999,"15min":0.497},"min":{"1min":0.35199999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.60999999999999999,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.50700000000000001,"15min":0.51900000000000002},"min":{"1min":0.38400000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.67600000000000005,"5min":0.83699999999999997,"15min":0.92800000000000005},"last":0.45700000000000002}]},{"osd":7,"last update":"Sat Apr 6 12:26:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53300000000000003,"15min":0.54000000000000004},"min":{"1min":0.35499999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72299999999999998,"5min":0.80000000000000004,"15min":0.81000000000000005},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55700000000000005,"15min":0.56499999999999995},"min":{"1min":0.39100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73599999999999999,"5min":0.88900000000000001,"15min":0.96099999999999997},"last":0.59099999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575032,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32060,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339780,"statfs":{"total":96636764160,"available":96603934720,"internally_reserved":0,"allocated":966656,"data_stored":477133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.371,"15min":0.35999999999999999},"min":{"1min":0.27500000000000002,"5min":0.22600000000000001,"15min":0.224},"max":{"1min":0.53400000000000003,"5min":0.58999999999999997,"15min":0.71699999999999997},"last":0.216},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40100000000000002,"15min":0.39500000000000002},"min":{"1min":0.28499999999999998,"5min":0.215,"15min":0.215},"max":{"1min":0.54000000000000004,"5min":0.56499999999999995,"15min":0.61699999999999999},"last":0.36199999999999999}]},{"osd":2,"last update":"Sat Apr 6 12:26:24 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40999999999999998,"15min":0.39500000000000002},"min":{"1min":0.308,"5min":0.23899999999999999,"15min":0.19400000000000001},"max":{"1min":0.61599999999999999,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.42899999999999999,"15min":0.42999999999999999},"min":{"1min":0.35699999999999998,"5min":0.28299999999999997,"15min":0.249},"max":{"1min":0.624,"5min":0.63600000000000001,"15min":0.79000000000000004},"last":0.255}]},{"osd":3,"last update":"Sat Apr 6 12:26:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.45200000000000001,"15min":0.44500000000000001},"min":{"1min":0.34000000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.64000000000000001,"5min":0.67800000000000005,"15min":0.84299999999999997},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.42099999999999999,"15min":0.41599999999999998},"min":{"1min":0.33700000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.60299999999999998,"5min":0.69599999999999995,"15min":0.86299999999999999},"last":0.34300000000000003}]},{"osd":4,"last update":"Sat Apr 6 12:26:24 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63500000000000001,"15min":0.65000000000000002},"min":{"1min":0.46600000000000003,"5min":0.40100000000000002,"15min":0.34799999999999998},"max":{"1min":0.81799999999999995,"5min":0.86799999999999999,"15min":6.3739999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.63700000000000001,"15min":0.622},"min":{"1min":0.51300000000000001,"5min":0.441,"15min":0.30399999999999999},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":1.0269999999999999},"last":0.45400000000000001}]},{"osd":5,"last update":"Sat Apr 6 12:26:24 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.63100000000000001,"15min":0.61399999999999999},"min":{"1min":0.47999999999999998,"5min":0.42999999999999999,"15min":0.35999999999999999},"max":{"1min":0.85899999999999999,"5min":0.91300000000000003,"15min":0.97699999999999998},"last":0.501},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.64300000000000002,"15min":0.629},"min":{"1min":0.52200000000000002,"5min":0.43099999999999999,"15min":0.35099999999999998},"max":{"1min":0.999,"5min":0.999,"15min":1.0089999999999999},"last":0.442}]},{"osd":6,"last update":"Sat Apr 6 12:26:24 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.629,"15min":0.61399999999999999},"min":{"1min":0.57699999999999996,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.66200000000000003,"15min":0.65000000000000002},"min":{"1min":0.496,"5min":0.45600000000000002,"15min":0.41799999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":1.151},"last":0.54700000000000004}]},{"osd":7,"last update":"Sat Apr 6 12:26:24 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.64800000000000002,"15min":0.67300000000000004},"min":{"1min":0.499,"5min":0.41899999999999998,"15min":0.38800000000000001},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":6.3639999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.65000000000000002,"15min":0.63900000000000001},"min":{"1min":0.48699999999999999,"5min":0.40200000000000002,"15min":0.39300000000000002},"max":{"1min":0.871,"5min":0.95599999999999996,"15min":1.073},"last":0.51500000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575032,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32640,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1560576,"data_stored":1067501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.38700000000000001,"15min":0.38},"min":{"1min":0.312,"5min":0.19,"15min":0.19},"max":{"1min":0.53900000000000003,"5min":0.80000000000000004,"15min":0.89100000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.40200000000000002},"min":{"1min":0.30599999999999999,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":0.60599999999999998,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.27500000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.40899999999999997,"15min":0.41199999999999998},"min":{"1min":0.29899999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.57799999999999996,"5min":0.60299999999999998,"15min":1.8089999999999999},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.434,"15min":0.42899999999999999},"min":{"1min":0.32600000000000001,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":0.64700000000000002,"5min":1.4790000000000001,"15min":1.764},"last":0.34300000000000003}]},{"osd":3,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.41899999999999998,"15min":0.41099999999999998},"min":{"1min":0.34100000000000003,"5min":0.252,"15min":0.21299999999999999},"max":{"1min":0.63,"5min":1.25,"15min":1.25},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.32900000000000001,"5min":0.28499999999999998,"15min":0.249},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.92000000000000004},"last":0.42599999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.627,"15min":0.61799999999999999},"min":{"1min":0.46100000000000002,"5min":0.371,"15min":0.35699999999999998},"max":{"1min":0.77700000000000002,"5min":1.514,"15min":1.514},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.63900000000000001,"15min":0.628},"min":{"1min":0.47599999999999998,"5min":0.37,"15min":0.311},"max":{"1min":0.86799999999999999,"5min":1.202,"15min":1.202},"last":0.57199999999999995}]},{"osd":5,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63400000000000001,"15min":0.61899999999999999},"min":{"1min":0.46999999999999997,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.84899999999999998,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64400000000000002,"15min":0.64700000000000002},"min":{"1min":0.45200000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.88600000000000001,"5min":0.94499999999999995,"15min":0.97899999999999998},"last":0.69999999999999996}]},{"osd":6,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65300000000000002,"15min":0.64800000000000002},"min":{"1min":0.51700000000000002,"5min":0.39100000000000001,"15min":0.378},"max":{"1min":0.84999999999999998,"5min":0.91000000000000003,"15min":0.96699999999999997},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.66900000000000004,"15min":0.66400000000000003},"min":{"1min":0.499,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84999999999999998,"5min":1.613,"15min":1.613},"last":0.61299999999999999}]},{"osd":7,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67000000000000004,"15min":0.66300000000000003},"min":{"1min":0.51500000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90100000000000002,"5min":1.54,"15min":1.54},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.67700000000000005,"15min":0.65700000000000003},"min":{"1min":0.51400000000000001,"5min":0.38400000000000001,"15min":0.36699999999999999},"max":{"1min":0.88300000000000001,"5min":1.498,"15min":1.498},"last":0.69199999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50112,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321728,"statfs":{"total":96636764160,"available":96585449472,"internally_reserved":0,"allocated":966656,"data_stored":477133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38600000000000001,"15min":0.38100000000000001},"min":{"1min":0.26700000000000002,"5min":0.214,"15min":0.21099999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":1.159},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.41199999999999998,"15min":0.40899999999999997},"min":{"1min":0.27500000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.54600000000000004,"5min":0.60899999999999999,"15min":1.075},"last":0.31900000000000001}]},{"osd":1,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.41099999999999998,"15min":0.41899999999999998},"min":{"1min":0.29599999999999999,"5min":0.26400000000000001,"15min":0.24299999999999999},"max":{"1min":0.57599999999999996,"5min":0.78500000000000003,"15min":3.6179999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.42299999999999999,"15min":0.42999999999999999},"min":{"1min":0.309,"5min":0.27100000000000002,"15min":0.23599999999999999},"max":{"1min":0.58899999999999997,"5min":0.61399999999999999,"15min":1.125},"last":0.377}]},{"osd":3,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.46000000000000002,"15min":0.46500000000000002},"min":{"1min":0.34399999999999997,"5min":0.30499999999999999,"15min":0.29699999999999999},"max":{"1min":0.63100000000000001,"5min":0.79200000000000004,"15min":3.661},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.44400000000000001,"15min":0.438},"min":{"1min":0.376,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.55900000000000005,"5min":1.0549999999999999,"15min":1.1910000000000001},"last":0.33800000000000002}]},{"osd":4,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.61499999999999999,"15min":0.60699999999999998},"min":{"1min":0.45900000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81699999999999995,"5min":0.88200000000000001,"15min":1.2170000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63600000000000001,"15min":0.628},"min":{"1min":0.48799999999999999,"5min":0.316,"15min":0.30499999999999999},"max":{"1min":0.89900000000000002,"5min":0.90200000000000002,"15min":1.101},"last":0.41099999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.64800000000000002,"15min":0.65300000000000002},"min":{"1min":0.57599999999999996,"5min":0.42299999999999999,"15min":0.32900000000000001},"max":{"1min":0.90300000000000002,"5min":1.6870000000000001,"15min":3.6920000000000002},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.628,"15min":0.625},"min":{"1min":0.49399999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.81200000000000006,"5min":0.89500000000000002,"15min":1.244},"last":0.39900000000000002}]},{"osd":6,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64300000000000002,"15min":0.63400000000000001},"min":{"1min":0.56499999999999995,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.84899999999999998,"5min":0.93400000000000005,"15min":1.012},"last":0.436},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67100000000000004,"15min":0.66900000000000004},"min":{"1min":0.51300000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93100000000000005,"5min":1.728,"15min":3.7490000000000001},"last":0.47699999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65400000000000003,"15min":0.64900000000000002},"min":{"1min":0.52900000000000003,"5min":0.40699999999999997,"15min":0.35599999999999998},"max":{"1min":0.80500000000000005,"5min":0.86899999999999999,"15min":1.0469999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.67700000000000005,"15min":0.68200000000000005},"min":{"1min":0.54300000000000004,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.90900000000000003,"5min":1.758,"15min":3.7200000000000002},"last":0.497}]}]},{"osd":3,"up_from":13,"seq":55834575033,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32080,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339760,"statfs":{"total":96636764160,"available":96603914240,"internally_reserved":0,"allocated":983040,"data_stored":484289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.24199999999999999,"5min":0.21099999999999999,"15min":0.191},"max":{"1min":0.51700000000000002,"5min":0.59699999999999998,"15min":0.63900000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38100000000000001,"15min":0.377},"min":{"1min":0.308,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.38800000000000001}]},{"osd":1,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.372,"15min":0.372},"min":{"1min":0.248,"5min":0.22700000000000001,"15min":0.215},"max":{"1min":0.47099999999999997,"5min":0.59999999999999998,"15min":0.62},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40200000000000002,"15min":0.40799999999999997},"min":{"1min":0.34200000000000003,"5min":0.26200000000000001,"15min":0.247},"max":{"1min":0.55200000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.72599999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.44900000000000001,"15min":0.45000000000000001},"min":{"1min":0.32800000000000001,"5min":0.27200000000000002,"15min":0.25700000000000001},"max":{"1min":0.55300000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.495,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.39700000000000002,"5min":0.317,"15min":0.29899999999999999},"max":{"1min":0.622,"5min":0.64800000000000002,"15min":0.65700000000000003},"last":0.88600000000000001}]},{"osd":4,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.49299999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.95999999999999996},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60999999999999999,"15min":0.61699999999999999},"min":{"1min":0.41899999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.74099999999999999,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.90100000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.64400000000000002,"15min":0.65500000000000003},"min":{"1min":0.46000000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.91100000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65300000000000002,"15min":0.65200000000000002},"min":{"1min":0.42999999999999999,"5min":0.38200000000000001,"15min":0.36899999999999999},"max":{"1min":0.94499999999999995,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.85499999999999998}]},{"osd":6,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60799999999999998,"15min":0.61799999999999999},"min":{"1min":0.47799999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.96999999999999997},"last":1.026},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63500000000000001,"15min":0.64800000000000002},"min":{"1min":0.436,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91800000000000004,"5min":0.92600000000000005,"15min":0.94699999999999995},"last":1.716}]},{"osd":7,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.629,"15min":0.63700000000000001},"min":{"1min":0.46899999999999997,"5min":0.40400000000000003,"15min":0.33300000000000002},"max":{"1min":0.873,"5min":0.89000000000000001,"15min":0.93300000000000005},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65100000000000002,"15min":0.65700000000000003},"min":{"1min":0.52200000000000002,"5min":0.39500000000000002,"15min":0.373},"max":{"1min":0.93300000000000005,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":1.002}]}]},{"osd":4,"up_from":13,"seq":55834575033,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32072,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339768,"statfs":{"total":96636764160,"available":96603922432,"internally_reserved":0,"allocated":970752,"data_stored":478152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.47499999999999998,"5min":0.29399999999999998,"15min":0.23899999999999999},"max":{"1min":0.71099999999999997,"5min":0.80800000000000005,"15min":0.82999999999999996},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55200000000000005,"15min":0.55400000000000005},"min":{"1min":0.49099999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.82199999999999995},"last":0.54700000000000004}]},{"osd":1,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.59299999999999997,"15min":0.63500000000000001},"min":{"1min":0.48199999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74399999999999999,"5min":0.76900000000000002,"15min":4.3479999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58699999999999997,"15min":0.623},"min":{"1min":0.48399999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77100000000000002,"5min":0.78400000000000003,"15min":4.3319999999999999},"last":0.63800000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.56299999999999994,"15min":0.56599999999999995},"min":{"1min":0.47399999999999998,"5min":0.313,"15min":0.311},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.78400000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.622,"5min":0.58099999999999996,"15min":0.58199999999999996},"min":{"1min":0.50900000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72399999999999998,"5min":0.74299999999999999,"15min":0.82199999999999995},"last":0.60999999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55100000000000005,"15min":0.59199999999999997},"min":{"1min":0.432,"5min":0.36399999999999999,"15min":0.34899999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":4.1950000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.621,"5min":0.59699999999999998,"15min":0.60199999999999998},"min":{"1min":0.497,"5min":0.40500000000000003,"15min":0.36099999999999999},"max":{"1min":0.73699999999999999,"5min":0.75900000000000001,"15min":1.2330000000000001},"last":0.66900000000000004}]},{"osd":5,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.46500000000000002,"15min":0.48999999999999999},"min":{"1min":0.36199999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71099999999999997,"5min":0.76300000000000001,"15min":1.196},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.497,"15min":0.502},"min":{"1min":0.39400000000000002,"5min":0.33500000000000002,"15min":0.312},"max":{"1min":0.63900000000000001,"5min":0.74299999999999999,"15min":0.77000000000000002},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.498,"15min":0.51100000000000001},"min":{"1min":0.33200000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.64200000000000002,"5min":0.754,"15min":1.022},"last":0.625},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.52500000000000002,"15min":0.54400000000000004},"min":{"1min":0.39500000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67400000000000004,"5min":0.73699999999999999,"15min":0.872},"last":0.69399999999999995}]},{"osd":7,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51200000000000001,"15min":0.54100000000000004},"min":{"1min":0.377,"5min":0.311,"15min":0.311},"max":{"1min":0.68500000000000005,"5min":0.79300000000000004,"15min":0.91500000000000004},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55000000000000004,"15min":0.57699999999999996},"min":{"1min":0.39300000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73599999999999999,"5min":0.752,"15min":1.0589999999999999},"last":0.70699999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575033,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46016,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325824,"statfs":{"total":96636764160,"available":96589643776,"internally_reserved":0,"allocated":966656,"data_stored":477133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57299999999999995,"15min":0.56899999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.24199999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.433,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.45700000000000002}]},{"osd":1,"last update":"Sat Apr 6 12:26:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.44800000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73899999999999999,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.437,"5min":0.34499999999999997,"15min":0.30599999999999999},"max":{"1min":0.69199999999999995,"5min":0.74299999999999999,"15min":0.76200000000000001},"last":0.52300000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:26:22 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57599999999999996,"15min":0.56100000000000005},"min":{"1min":0.40899999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.81899999999999995,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57099999999999995,"15min":0.56799999999999995},"min":{"1min":0.40100000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.58599999999999997}]},{"osd":3,"last update":"Sat Apr 6 12:26:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57599999999999996,"15min":0.57099999999999995},"min":{"1min":0.45400000000000001,"5min":0.32500000000000001,"15min":0.31},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.60299999999999998,"15min":0.59799999999999998},"min":{"1min":0.44400000000000001,"5min":0.377,"15min":0.32300000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.54600000000000004}]},{"osd":4,"last update":"Sat Apr 6 12:26:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46800000000000003,"15min":0.47799999999999998},"min":{"1min":0.33200000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.63700000000000001,"5min":0.70299999999999996,"15min":0.81200000000000006},"last":0.433},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53700000000000003,"15min":0.53800000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.79800000000000004,"5min":0.80000000000000004,"15min":0.91800000000000004},"last":0.625}]},{"osd":6,"last update":"Sat Apr 6 12:26:22 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48199999999999998,"15min":0.49099999999999999},"min":{"1min":0.34000000000000002,"5min":0.316,"15min":0.28000000000000003},"max":{"1min":0.69899999999999995,"5min":0.73899999999999999,"15min":0.88700000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.497,"15min":0.5},"min":{"1min":0.35599999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.71899999999999997,"5min":0.73199999999999998,"15min":0.83299999999999996},"last":0.56000000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:26:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.52100000000000002,"15min":0.51700000000000002},"min":{"1min":0.376,"5min":0.314,"15min":0.252},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.79100000000000004},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.44400000000000001,"5min":0.41399999999999998,"15min":0.35199999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85599999999999998},"last":0.60399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:27:25.148 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:27:25.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.308+0000 7f3a47cff700 1 Processor -- start 2024-04-06T12:27:25.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.308+0000 7f3a47cff700 1 -- start start 2024-04-06T12:27:25.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.309+0000 7f3a47cff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a400b4af0 0x7f3a400b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:25.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.309+0000 7f3a47cff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a400cdbe0 con 0x7f3a400b4660 2024-04-06T12:27:25.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.309+0000 7f3a47cff700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4015a940 con 0x7f3a4015a550 2024-04-06T12:27:25.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.309+0000 7f3a47cff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4015ac00 con 0x7f3a400b4af0 2024-04-06T12:27:25.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.309+0000 7f3a4529a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3a400b4660 0x7f3a400b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56898/0 (socket says 172.21.15.189:56898) 2024-04-06T12:27:25.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.309+0000 7f3a4529a700 1 -- 172.21.15.189:0/468392897 learned_addr learned my addr 172.21.15.189:0/468392897 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:25.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.309+0000 7f3a44a99700 1 -- 172.21.15.189:0/468392897 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695370531 0 0) 0x7f3a400cdbe0 con 0x7f3a400b4660 2024-04-06T12:27:25.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.309+0000 7f3a4629c700 1 --2- 172.21.15.189:0/468392897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a400b4af0 0x7f3a400b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:25.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.309+0000 7f3a44a99700 1 -- 172.21.15.189:0/468392897 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a2c0035b0 con 0x7f3a400b4660 2024-04-06T12:27:25.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a44a99700 1 -- 172.21.15.189:0/468392897 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3381880876 0 0) 0x7f3a2c0035b0 con 0x7f3a400b4660 2024-04-06T12:27:25.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a44a99700 1 -- 172.21.15.189:0/468392897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a400b4af0 msgr2=0x7f3a400b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:25.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a44a99700 1 --2- 172.21.15.189:0/468392897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a400b4af0 0x7f3a400b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a44a99700 1 -- 172.21.15.189:0/468392897 >> v1:172.21.15.192:6789/0 conn(0x7f3a4015a550 legacy=0x7f3a400cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:25.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a44a99700 1 -- 172.21.15.189:0/468392897 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a4015aee0 con 0x7f3a400b4660 2024-04-06T12:27:25.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a44a99700 1 -- 172.21.15.189:0/468392897 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f3a30001670 con 0x7f3a400b4660 2024-04-06T12:27:25.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a44a99700 1 -- 172.21.15.189:0/468392897 >> v1:172.21.15.189:6789/0 conn(0x7f3a400b4660 legacy=0x7f3a400b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:25.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a44a99700 1 --2- 172.21.15.189:0/468392897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a2c003a30 0x7f3a2c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:25.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a44a99700 1 --2- 172.21.15.189:0/468392897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a2c004850 0x7f3a2c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:25.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a44a99700 1 --2- 172.21.15.189:0/468392897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a400b4af0 0x7f3a2c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:25.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a44a99700 1 -- 172.21.15.189:0/468392897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4015ac00 con 0x7f3a2c004850 2024-04-06T12:27:25.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a44a99700 1 -- 172.21.15.189:0/468392897 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2c00baf0 con 0x7f3a2c003a30 2024-04-06T12:27:25.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a44a99700 1 -- 172.21.15.189:0/468392897 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2c00bcb0 con 0x7f3a400b4af0 2024-04-06T12:27:25.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a4529a700 1 --2- 172.21.15.189:0/468392897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a2c004850 0x7f3a2c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:25.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a45a9b700 1 --2- 172.21.15.189:0/468392897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a2c003a30 0x7f3a2c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:25.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a4629c700 1 --2- 172.21.15.189:0/468392897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a400b4af0 0x7f3a2c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:25.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a4529a700 1 -- 172.21.15.189:0/468392897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a400b4af0 msgr2=0x7f3a2c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:25.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a4529a700 1 --2- 172.21.15.189:0/468392897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a400b4af0 0x7f3a2c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a4529a700 1 -- 172.21.15.189:0/468392897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a2c003a30 msgr2=0x7f3a2c003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:25.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a4529a700 1 --2- 172.21.15.189:0/468392897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a2c003a30 0x7f3a2c003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a4529a700 1 -- 172.21.15.189:0/468392897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2c0035b0 con 0x7f3a2c004850 2024-04-06T12:27:25.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a4529a700 1 --2- 172.21.15.189:0/468392897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a2c004850 0x7f3a2c004cd0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f3a30001eb0 tx=0x7f3a2c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7247104165ddf4e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:25.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a44a99700 1 -- 172.21.15.189:0/468392897 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a30007b70 con 0x7f3a2c004850 2024-04-06T12:27:25.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a44a99700 1 -- 172.21.15.189:0/468392897 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3000ccf0 con 0x7f3a2c004850 2024-04-06T12:27:25.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.310+0000 7f3a44a99700 1 -- 172.21.15.189:0/468392897 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a30005ba0 con 0x7f3a2c004850 2024-04-06T12:27:25.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.311+0000 7f3a47cff700 1 -- 172.21.15.189:0/468392897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a2c004850 msgr2=0x7f3a2c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:25.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.311+0000 7f3a47cff700 1 --2- 172.21.15.189:0/468392897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a2c004850 0x7f3a2c004cd0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f3a30001eb0 tx=0x7f3a2c005880 comp rx=0 tx=0).stop 2024-04-06T12:27:25.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.311+0000 7f3a4629c700 1 -- 172.21.15.189:0/468392897 reap_dead start 2024-04-06T12:27:25.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.311+0000 7f3a47cff700 1 -- 172.21.15.189:0/468392897 shutdown_connections 2024-04-06T12:27:25.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.311+0000 7f3a47cff700 1 -- 172.21.15.189:0/468392897 >> 172.21.15.189:0/468392897 conn(0x7f3a400d00b0 msgr2=0x7f3a400d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:25.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.311+0000 7f3a47cff700 1 -- 172.21.15.189:0/468392897 shutdown_connections 2024-04-06T12:27:25.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.311+0000 7f3a47cff700 1 -- 172.21.15.189:0/468392897 wait complete. 2024-04-06T12:27:25.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.312+0000 7f3a47cff700 1 Processor -- start 2024-04-06T12:27:25.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.312+0000 7f3a47cff700 1 -- start start 2024-04-06T12:27:25.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.313+0000 7f3a47cff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a2c004850 0x7f3a40145070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:25.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.313+0000 7f3a47cff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a400ae760 0x7f3a400aebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:25.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.313+0000 7f3a47cff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a4006a730 0x7f3a4006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:25.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.313+0000 7f3a47cff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4006b0f0 con 0x7f3a4006a730 2024-04-06T12:27:25.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.313+0000 7f3a47cff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a401460f0 con 0x7f3a2c004850 2024-04-06T12:27:25.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.313+0000 7f3a47cff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a401457c0 con 0x7f3a400ae760 2024-04-06T12:27:25.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.313+0000 7f3a4629c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a4006a730 0x7f3a4006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:25.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.313+0000 7f3a4629c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a4006a730 0x7f3a4006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37622/0 (socket says 172.21.15.189:37622) 2024-04-06T12:27:25.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.313+0000 7f3a4629c700 1 -- 172.21.15.189:0/643024046 learned_addr learned my addr 172.21.15.189:0/643024046 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:25.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.313+0000 7f3a4529a700 1 --2- 172.21.15.189:0/643024046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a400ae760 0x7f3a400aebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:25.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.313+0000 7f3a45a9b700 1 --2- 172.21.15.189:0/643024046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a2c004850 0x7f3a40145070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:25.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.313+0000 7f3a4629c700 1 -- 172.21.15.189:0/643024046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a400ae760 msgr2=0x7f3a400aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:25.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.313+0000 7f3a4629c700 1 --2- 172.21.15.189:0/643024046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a400ae760 0x7f3a400aebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.313+0000 7f3a4629c700 1 -- 172.21.15.189:0/643024046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a2c004850 msgr2=0x7f3a40145070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:25.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.314+0000 7f3a4629c700 1 --2- 172.21.15.189:0/643024046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a2c004850 0x7f3a40145070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.314+0000 7f3a4629c700 1 -- 172.21.15.189:0/643024046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a40145aa0 con 0x7f3a4006a730 2024-04-06T12:27:25.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.314+0000 7f3a4629c700 1 --2- 172.21.15.189:0/643024046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a4006a730 0x7f3a4006abb0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f3a40067b80 tx=0x7f3a380114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:25.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.314+0000 7f3a367fc700 1 -- 172.21.15.189:0/643024046 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3800e070 con 0x7f3a4006a730 2024-04-06T12:27:25.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.314+0000 7f3a367fc700 1 -- 172.21.15.189:0/643024046 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a38003cf0 con 0x7f3a4006a730 2024-04-06T12:27:25.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.314+0000 7f3a47cff700 1 -- 172.21.15.189:0/643024046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a40145d80 con 0x7f3a4006a730 2024-04-06T12:27:25.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.314+0000 7f3a367fc700 1 -- 172.21.15.189:0/643024046 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a38003ea0 con 0x7f3a4006a730 2024-04-06T12:27:25.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.314+0000 7f3a47cff700 1 -- 172.21.15.189:0/643024046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a4014b490 con 0x7f3a4006a730 2024-04-06T12:27:25.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.315+0000 7f3a47cff700 1 -- 172.21.15.189:0/643024046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a40067bb0 con 0x7f3a4006a730 2024-04-06T12:27:25.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.315+0000 7f3a367fc700 1 -- 172.21.15.189:0/643024046 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f3a38011900 con 0x7f3a4006a730 2024-04-06T12:27:25.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.316+0000 7f3a367fc700 1 --2- 172.21.15.189:0/643024046 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3a2406adb0 0x7f3a2406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:25.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.316+0000 7f3a367fc700 1 -- 172.21.15.189:0/643024046 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3a38089930 con 0x7f3a4006a730 2024-04-06T12:27:25.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.316+0000 7f3a45a9b700 1 --2- 172.21.15.189:0/643024046 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3a2406adb0 0x7f3a2406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:25.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.316+0000 7f3a45a9b700 1 --2- 172.21.15.189:0/643024046 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3a2406adb0 0x7f3a2406d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c007fd0 tx=0x7f3a3c009040 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:25.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.320+0000 7f3a367fc700 1 -- 172.21.15.189:0/643024046 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3a38054840 con 0x7f3a4006a730 2024-04-06T12:27:25.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.512+0000 7f3a47cff700 1 -- 172.21.15.189:0/643024046 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3a4006fdd0 con 0x7f3a2406adb0 2024-04-06T12:27:25.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.520+0000 7f3a367fc700 1 -- 172.21.15.189:0/643024046 <== mgr.5582 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287696 (secure 0 0 0) 0x7f3a3c001cc0 con 0x7f3a2406adb0 2024-04-06T12:27:25.521 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:25.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.523+0000 7f3a47cff700 1 -- 172.21.15.189:0/643024046 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3a2406adb0 msgr2=0x7f3a2406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:25.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.523+0000 7f3a47cff700 1 --2- 172.21.15.189:0/643024046 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3a2406adb0 0x7f3a2406d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c007fd0 tx=0x7f3a3c009040 comp rx=0 tx=0).stop 2024-04-06T12:27:25.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.523+0000 7f3a47cff700 1 -- 172.21.15.189:0/643024046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a4006a730 msgr2=0x7f3a4006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:25.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.523+0000 7f3a47cff700 1 --2- 172.21.15.189:0/643024046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a4006a730 0x7f3a4006abb0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f3a40067b80 tx=0x7f3a380114b0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.523+0000 7f3a47cff700 1 -- 172.21.15.189:0/643024046 shutdown_connections 2024-04-06T12:27:25.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.523+0000 7f3a47cff700 1 --2- 172.21.15.189:0/643024046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a4006a730 0x7f3a4006abb0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.523+0000 7f3a47cff700 1 --2- 172.21.15.189:0/643024046 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3a2406adb0 0x7f3a2406d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.523+0000 7f3a47cff700 1 --2- 172.21.15.189:0/643024046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a2c004850 0x7f3a40145070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.523+0000 7f3a47cff700 1 --2- 172.21.15.189:0/643024046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a400ae760 0x7f3a400aebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.523+0000 7f3a47cff700 1 -- 172.21.15.189:0/643024046 >> 172.21.15.189:0/643024046 conn(0x7f3a400d00b0 msgr2=0x7f3a4006d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:25.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.523+0000 7f3a47cff700 1 -- 172.21.15.189:0/643024046 shutdown_connections 2024-04-06T12:27:25.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.523+0000 7f3a47cff700 1 -- 172.21.15.189:0/643024046 wait complete. 2024-04-06T12:27:25.525 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:27:25.533 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-06T12:27:25.309302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592536,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":273,"num_write_kb":2421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":192,"ondisk_log_size":192,"up":176,"acting":176,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":137,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":316308,"kb_used_data":9108,"kb_used_omap":12,"kb_used_meta":307059,"kb_avail":754658412,"statfs":{"total":773094113280,"available":772770213888,"internally_reserved":0,"allocated":9326592,"data_stored":5189878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":314428984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2168,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.693015"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046604+0000","last_change":"2024-04-06T12:27:21.054450+0000","last_active":"2024-04-06T12:27:22.046604+0000","last_peered":"2024-04-06T12:27:22.046604+0000","last_clean":"2024-04-06T12:27:22.046604+0000","last_became_active":"2024-04-06T12:27:21.054202+0000","last_became_peered":"2024-04-06T12:27:21.054202+0000","last_unstale":"2024-04-06T12:27:22.046604+0000","last_undegraded":"2024-04-06T12:27:22.046604+0000","last_fullsized":"2024-04-06T12:27:22.046604+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:29:26.946143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046360+0000","last_change":"2024-04-06T12:27:21.053119+0000","last_active":"2024-04-06T12:27:22.046360+0000","last_peered":"2024-04-06T12:27:22.046360+0000","last_clean":"2024-04-06T12:27:22.046360+0000","last_became_active":"2024-04-06T12:27:21.052814+0000","last_became_peered":"2024-04-06T12:27:21.052814+0000","last_unstale":"2024-04-06T12:27:22.046360+0000","last_undegraded":"2024-04-06T12:27:22.046360+0000","last_fullsized":"2024-04-06T12:27:22.046360+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:14.746095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048036+0000","last_change":"2024-04-06T12:27:21.048815+0000","last_active":"2024-04-06T12:27:22.048036+0000","last_peered":"2024-04-06T12:27:22.048036+0000","last_clean":"2024-04-06T12:27:22.048036+0000","last_became_active":"2024-04-06T12:27:21.047390+0000","last_became_peered":"2024-04-06T12:27:21.047390+0000","last_unstale":"2024-04-06T12:27:22.048036+0000","last_undegraded":"2024-04-06T12:27:22.048036+0000","last_fullsized":"2024-04-06T12:27:22.048036+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:53:37.857841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047263+0000","last_change":"2024-04-06T12:27:21.064877+0000","last_active":"2024-04-06T12:27:22.047263+0000","last_peered":"2024-04-06T12:27:22.047263+0000","last_clean":"2024-04-06T12:27:22.047263+0000","last_became_active":"2024-04-06T12:27:21.064649+0000","last_became_peered":"2024-04-06T12:27:21.064649+0000","last_unstale":"2024-04-06T12:27:22.047263+0000","last_undegraded":"2024-04-06T12:27:22.047263+0000","last_fullsized":"2024-04-06T12:27:22.047263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:25.859663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.040075+0000","last_change":"2024-04-06T12:27:21.054089+0000","last_active":"2024-04-06T12:27:22.040075+0000","last_peered":"2024-04-06T12:27:22.040075+0000","last_clean":"2024-04-06T12:27:22.040075+0000","last_became_active":"2024-04-06T12:27:21.053844+0000","last_became_peered":"2024-04-06T12:27:21.053844+0000","last_unstale":"2024-04-06T12:27:22.040075+0000","last_undegraded":"2024-04-06T12:27:22.040075+0000","last_fullsized":"2024-04-06T12:27:22.040075+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:39:49.798363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046048+0000","last_change":"2024-04-06T12:27:21.060315+0000","last_active":"2024-04-06T12:27:22.046048+0000","last_peered":"2024-04-06T12:27:22.046048+0000","last_clean":"2024-04-06T12:27:22.046048+0000","last_became_active":"2024-04-06T12:27:21.060097+0000","last_became_peered":"2024-04-06T12:27:21.060097+0000","last_unstale":"2024-04-06T12:27:22.046048+0000","last_undegraded":"2024-04-06T12:27:22.046048+0000","last_fullsized":"2024-04-06T12:27:22.046048+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:53:09.701399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046794+0000","last_change":"2024-04-06T12:27:21.053746+0000","last_active":"2024-04-06T12:27:22.046794+0000","last_peered":"2024-04-06T12:27:22.046794+0000","last_clean":"2024-04-06T12:27:22.046794+0000","last_became_active":"2024-04-06T12:27:21.053476+0000","last_became_peered":"2024-04-06T12:27:21.053476+0000","last_unstale":"2024-04-06T12:27:22.046794+0000","last_undegraded":"2024-04-06T12:27:22.046794+0000","last_fullsized":"2024-04-06T12:27:22.046794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:59:57.937757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047029+0000","last_change":"2024-04-06T12:27:21.051431+0000","last_active":"2024-04-06T12:27:22.047029+0000","last_peered":"2024-04-06T12:27:22.047029+0000","last_clean":"2024-04-06T12:27:22.047029+0000","last_became_active":"2024-04-06T12:27:21.050731+0000","last_became_peered":"2024-04-06T12:27:21.050731+0000","last_unstale":"2024-04-06T12:27:22.047029+0000","last_undegraded":"2024-04-06T12:27:22.047029+0000","last_fullsized":"2024-04-06T12:27:22.047029+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:26:12.757657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047263+0000","last_change":"2024-04-06T12:27:21.056611+0000","last_active":"2024-04-06T12:27:22.047263+0000","last_peered":"2024-04-06T12:27:22.047263+0000","last_clean":"2024-04-06T12:27:22.047263+0000","last_became_active":"2024-04-06T12:27:21.056361+0000","last_became_peered":"2024-04-06T12:27:21.056361+0000","last_unstale":"2024-04-06T12:27:22.047263+0000","last_undegraded":"2024-04-06T12:27:22.047263+0000","last_fullsized":"2024-04-06T12:27:22.047263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:41:13.712569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047557+0000","last_change":"2024-04-06T12:27:21.053678+0000","last_active":"2024-04-06T12:27:22.047557+0000","last_peered":"2024-04-06T12:27:22.047557+0000","last_clean":"2024-04-06T12:27:22.047557+0000","last_became_active":"2024-04-06T12:27:21.053132+0000","last_became_peered":"2024-04-06T12:27:21.053132+0000","last_unstale":"2024-04-06T12:27:22.047557+0000","last_undegraded":"2024-04-06T12:27:22.047557+0000","last_fullsized":"2024-04-06T12:27:22.047557+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:57.645194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046311+0000","last_change":"2024-04-06T12:27:21.042215+0000","last_active":"2024-04-06T12:27:22.046311+0000","last_peered":"2024-04-06T12:27:22.046311+0000","last_clean":"2024-04-06T12:27:22.046311+0000","last_became_active":"2024-04-06T12:27:21.041921+0000","last_became_peered":"2024-04-06T12:27:21.041921+0000","last_unstale":"2024-04-06T12:27:22.046311+0000","last_undegraded":"2024-04-06T12:27:22.046311+0000","last_fullsized":"2024-04-06T12:27:22.046311+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:33:26.404172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046752+0000","last_change":"2024-04-06T12:27:21.056018+0000","last_active":"2024-04-06T12:27:22.046752+0000","last_peered":"2024-04-06T12:27:22.046752+0000","last_clean":"2024-04-06T12:27:22.046752+0000","last_became_active":"2024-04-06T12:27:21.055764+0000","last_became_peered":"2024-04-06T12:27:21.055764+0000","last_unstale":"2024-04-06T12:27:22.046752+0000","last_undegraded":"2024-04-06T12:27:22.046752+0000","last_fullsized":"2024-04-06T12:27:22.046752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:49:34.397313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047499+0000","last_change":"2024-04-06T12:27:21.052386+0000","last_active":"2024-04-06T12:27:22.047499+0000","last_peered":"2024-04-06T12:27:22.047499+0000","last_clean":"2024-04-06T12:27:22.047499+0000","last_became_active":"2024-04-06T12:27:21.052081+0000","last_became_peered":"2024-04-06T12:27:21.052081+0000","last_unstale":"2024-04-06T12:27:22.047499+0000","last_undegraded":"2024-04-06T12:27:22.047499+0000","last_fullsized":"2024-04-06T12:27:22.047499+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:41:13.712569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047792+0000","last_change":"2024-04-06T12:27:21.059175+0000","last_active":"2024-04-06T12:27:22.047792+0000","last_peered":"2024-04-06T12:27:22.047792+0000","last_clean":"2024-04-06T12:27:22.047792+0000","last_became_active":"2024-04-06T12:27:21.058651+0000","last_became_peered":"2024-04-06T12:27:21.058651+0000","last_unstale":"2024-04-06T12:27:22.047792+0000","last_undegraded":"2024-04-06T12:27:22.047792+0000","last_fullsized":"2024-04-06T12:27:22.047792+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:45.939534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039097+0000","last_change":"2024-04-06T12:27:21.062933+0000","last_active":"2024-04-06T12:27:22.039097+0000","last_peered":"2024-04-06T12:27:22.039097+0000","last_clean":"2024-04-06T12:27:22.039097+0000","last_became_active":"2024-04-06T12:27:21.062688+0000","last_became_peered":"2024-04-06T12:27:21.062688+0000","last_unstale":"2024-04-06T12:27:22.039097+0000","last_undegraded":"2024-04-06T12:27:22.039097+0000","last_fullsized":"2024-04-06T12:27:22.039097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:30:58.878304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047660+0000","last_change":"2024-04-06T12:27:21.059206+0000","last_active":"2024-04-06T12:27:22.047660+0000","last_peered":"2024-04-06T12:27:22.047660+0000","last_clean":"2024-04-06T12:27:22.047660+0000","last_became_active":"2024-04-06T12:27:21.058932+0000","last_became_peered":"2024-04-06T12:27:21.058932+0000","last_unstale":"2024-04-06T12:27:22.047660+0000","last_undegraded":"2024-04-06T12:27:22.047660+0000","last_fullsized":"2024-04-06T12:27:22.047660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:08:32.179469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.038878+0000","last_change":"2024-04-06T12:27:21.053594+0000","last_active":"2024-04-06T12:27:22.038878+0000","last_peered":"2024-04-06T12:27:22.038878+0000","last_clean":"2024-04-06T12:27:22.038878+0000","last_became_active":"2024-04-06T12:27:21.053330+0000","last_became_peered":"2024-04-06T12:27:21.053330+0000","last_unstale":"2024-04-06T12:27:22.038878+0000","last_undegraded":"2024-04-06T12:27:22.038878+0000","last_fullsized":"2024-04-06T12:27:22.038878+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:54:27.984787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.038730+0000","last_change":"2024-04-06T12:27:21.053574+0000","last_active":"2024-04-06T12:27:22.038730+0000","last_peered":"2024-04-06T12:27:22.038730+0000","last_clean":"2024-04-06T12:27:22.038730+0000","last_became_active":"2024-04-06T12:27:21.053289+0000","last_became_peered":"2024-04-06T12:27:21.053289+0000","last_unstale":"2024-04-06T12:27:22.038730+0000","last_undegraded":"2024-04-06T12:27:22.038730+0000","last_fullsized":"2024-04-06T12:27:22.038730+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:44.646361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048049+0000","last_change":"2024-04-06T12:27:21.049364+0000","last_active":"2024-04-06T12:27:22.048049+0000","last_peered":"2024-04-06T12:27:22.048049+0000","last_clean":"2024-04-06T12:27:22.048049+0000","last_became_active":"2024-04-06T12:27:21.048571+0000","last_became_peered":"2024-04-06T12:27:21.048571+0000","last_unstale":"2024-04-06T12:27:22.048049+0000","last_undegraded":"2024-04-06T12:27:22.048049+0000","last_fullsized":"2024-04-06T12:27:22.048049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:24:04.575202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048237+0000","last_change":"2024-04-06T12:27:21.048886+0000","last_active":"2024-04-06T12:27:22.048237+0000","last_peered":"2024-04-06T12:27:22.048237+0000","last_clean":"2024-04-06T12:27:22.048237+0000","last_became_active":"2024-04-06T12:27:21.048635+0000","last_became_peered":"2024-04-06T12:27:21.048635+0000","last_unstale":"2024-04-06T12:27:22.048237+0000","last_undegraded":"2024-04-06T12:27:22.048237+0000","last_fullsized":"2024-04-06T12:27:22.048237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:14.746095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047147+0000","last_change":"2024-04-06T12:27:21.045989+0000","last_active":"2024-04-06T12:27:22.047147+0000","last_peered":"2024-04-06T12:27:22.047147+0000","last_clean":"2024-04-06T12:27:22.047147+0000","last_became_active":"2024-04-06T12:27:21.045733+0000","last_became_peered":"2024-04-06T12:27:21.045733+0000","last_unstale":"2024-04-06T12:27:22.047147+0000","last_undegraded":"2024-04-06T12:27:22.047147+0000","last_fullsized":"2024-04-06T12:27:22.047147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:53:59.500416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046223+0000","last_change":"2024-04-06T12:27:21.063222+0000","last_active":"2024-04-06T12:27:22.046223+0000","last_peered":"2024-04-06T12:27:22.046223+0000","last_clean":"2024-04-06T12:27:22.046223+0000","last_became_active":"2024-04-06T12:27:21.062971+0000","last_became_peered":"2024-04-06T12:27:21.062971+0000","last_unstale":"2024-04-06T12:27:22.046223+0000","last_undegraded":"2024-04-06T12:27:22.046223+0000","last_fullsized":"2024-04-06T12:27:22.046223+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:41:35.413608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046965+0000","last_change":"2024-04-06T12:27:21.057755+0000","last_active":"2024-04-06T12:27:22.046965+0000","last_peered":"2024-04-06T12:27:22.046965+0000","last_clean":"2024-04-06T12:27:22.046965+0000","last_became_active":"2024-04-06T12:27:21.057491+0000","last_became_peered":"2024-04-06T12:27:21.057491+0000","last_unstale":"2024-04-06T12:27:22.046965+0000","last_undegraded":"2024-04-06T12:27:22.046965+0000","last_fullsized":"2024-04-06T12:27:22.046965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:54:23.429749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046895+0000","last_change":"2024-04-06T12:27:21.048114+0000","last_active":"2024-04-06T12:27:22.046895+0000","last_peered":"2024-04-06T12:27:22.046895+0000","last_clean":"2024-04-06T12:27:22.046895+0000","last_became_active":"2024-04-06T12:27:21.047854+0000","last_became_peered":"2024-04-06T12:27:21.047854+0000","last_unstale":"2024-04-06T12:27:22.046895+0000","last_undegraded":"2024-04-06T12:27:22.046895+0000","last_fullsized":"2024-04-06T12:27:22.046895+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:02.552663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039826+0000","last_change":"2024-04-06T12:27:21.054604+0000","last_active":"2024-04-06T12:27:22.039826+0000","last_peered":"2024-04-06T12:27:22.039826+0000","last_clean":"2024-04-06T12:27:22.039826+0000","last_became_active":"2024-04-06T12:27:21.049845+0000","last_became_peered":"2024-04-06T12:27:21.049845+0000","last_unstale":"2024-04-06T12:27:22.039826+0000","last_undegraded":"2024-04-06T12:27:22.039826+0000","last_fullsized":"2024-04-06T12:27:22.039826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:26.827213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047368+0000","last_change":"2024-04-06T12:27:21.064380+0000","last_active":"2024-04-06T12:27:22.047368+0000","last_peered":"2024-04-06T12:27:22.047368+0000","last_clean":"2024-04-06T12:27:22.047368+0000","last_became_active":"2024-04-06T12:27:21.064115+0000","last_became_peered":"2024-04-06T12:27:21.064115+0000","last_unstale":"2024-04-06T12:27:22.047368+0000","last_undegraded":"2024-04-06T12:27:22.047368+0000","last_fullsized":"2024-04-06T12:27:22.047368+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:12:18.899208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.065495+0000","last_change":"2024-04-06T12:27:20.639851+0000","last_active":"2024-04-06T12:27:21.065495+0000","last_peered":"2024-04-06T12:27:21.065495+0000","last_clean":"2024-04-06T12:27:21.065495+0000","last_became_active":"2024-04-06T12:27:20.639619+0000","last_became_peered":"2024-04-06T12:27:20.639619+0000","last_unstale":"2024-04-06T12:27:21.065495+0000","last_undegraded":"2024-04-06T12:27:21.065495+0000","last_fullsized":"2024-04-06T12:27:21.065495+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:24.851744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047260+0000","last_change":"2024-04-06T12:27:21.063852+0000","last_active":"2024-04-06T12:27:22.047260+0000","last_peered":"2024-04-06T12:27:22.047260+0000","last_clean":"2024-04-06T12:27:22.047260+0000","last_became_active":"2024-04-06T12:27:21.063593+0000","last_became_peered":"2024-04-06T12:27:21.063593+0000","last_unstale":"2024-04-06T12:27:22.047260+0000","last_undegraded":"2024-04-06T12:27:22.047260+0000","last_fullsized":"2024-04-06T12:27:22.047260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:35:07.924741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.065665+0000","last_change":"2024-04-06T12:27:20.640009+0000","last_active":"2024-04-06T12:27:21.065665+0000","last_peered":"2024-04-06T12:27:21.065665+0000","last_clean":"2024-04-06T12:27:21.065665+0000","last_became_active":"2024-04-06T12:27:20.639751+0000","last_became_peered":"2024-04-06T12:27:20.639751+0000","last_unstale":"2024-04-06T12:27:21.065665+0000","last_undegraded":"2024-04-06T12:27:21.065665+0000","last_fullsized":"2024-04-06T12:27:21.065665+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:23:10.897604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.040256+0000","last_change":"2024-04-06T12:27:21.050994+0000","last_active":"2024-04-06T12:27:22.040256+0000","last_peered":"2024-04-06T12:27:22.040256+0000","last_clean":"2024-04-06T12:27:22.040256+0000","last_became_active":"2024-04-06T12:27:21.049716+0000","last_became_peered":"2024-04-06T12:27:21.049716+0000","last_unstale":"2024-04-06T12:27:22.040256+0000","last_undegraded":"2024-04-06T12:27:22.040256+0000","last_fullsized":"2024-04-06T12:27:22.040256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:53:03.356300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.059811+0000","last_change":"2024-04-06T12:27:20.644294+0000","last_active":"2024-04-06T12:27:21.059811+0000","last_peered":"2024-04-06T12:27:21.059811+0000","last_clean":"2024-04-06T12:27:21.059811+0000","last_became_active":"2024-04-06T12:27:20.643994+0000","last_became_peered":"2024-04-06T12:27:20.643994+0000","last_unstale":"2024-04-06T12:27:21.059811+0000","last_undegraded":"2024-04-06T12:27:21.059811+0000","last_fullsized":"2024-04-06T12:27:21.059811+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:44:45.550106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.060664+0000","last_change":"2024-04-06T12:27:20.636807+0000","last_active":"2024-04-06T12:27:21.060664+0000","last_peered":"2024-04-06T12:27:21.060664+0000","last_clean":"2024-04-06T12:27:21.060664+0000","last_became_active":"2024-04-06T12:27:20.636516+0000","last_became_peered":"2024-04-06T12:27:20.636516+0000","last_unstale":"2024-04-06T12:27:21.060664+0000","last_undegraded":"2024-04-06T12:27:21.060664+0000","last_fullsized":"2024-04-06T12:27:21.060664+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:00:13.506742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.045921+0000","last_change":"2024-04-06T12:27:21.047046+0000","last_active":"2024-04-06T12:27:22.045921+0000","last_peered":"2024-04-06T12:27:22.045921+0000","last_clean":"2024-04-06T12:27:22.045921+0000","last_became_active":"2024-04-06T12:27:21.046798+0000","last_became_peered":"2024-04-06T12:27:21.046798+0000","last_unstale":"2024-04-06T12:27:22.045921+0000","last_undegraded":"2024-04-06T12:27:22.045921+0000","last_fullsized":"2024-04-06T12:27:22.045921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:23:44.000355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.048716+0000","last_change":"2024-04-06T12:27:20.642417+0000","last_active":"2024-04-06T12:27:21.048716+0000","last_peered":"2024-04-06T12:27:21.048716+0000","last_clean":"2024-04-06T12:27:21.048716+0000","last_became_active":"2024-04-06T12:27:20.642057+0000","last_became_peered":"2024-04-06T12:27:20.642057+0000","last_unstale":"2024-04-06T12:27:21.048716+0000","last_undegraded":"2024-04-06T12:27:21.048716+0000","last_fullsized":"2024-04-06T12:27:21.048716+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:40:56.649109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046707+0000","last_change":"2024-04-06T12:27:21.056061+0000","last_active":"2024-04-06T12:27:22.046707+0000","last_peered":"2024-04-06T12:27:22.046707+0000","last_clean":"2024-04-06T12:27:22.046707+0000","last_became_active":"2024-04-06T12:27:21.055761+0000","last_became_peered":"2024-04-06T12:27:21.055761+0000","last_unstale":"2024-04-06T12:27:22.046707+0000","last_undegraded":"2024-04-06T12:27:22.046707+0000","last_fullsized":"2024-04-06T12:27:22.046707+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:20:15.444097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.046311+0000","last_change":"2024-04-06T12:27:20.642798+0000","last_active":"2024-04-06T12:27:21.046311+0000","last_peered":"2024-04-06T12:27:21.046311+0000","last_clean":"2024-04-06T12:27:21.046311+0000","last_became_active":"2024-04-06T12:27:20.642006+0000","last_became_peered":"2024-04-06T12:27:20.642006+0000","last_unstale":"2024-04-06T12:27:21.046311+0000","last_undegraded":"2024-04-06T12:27:21.046311+0000","last_fullsized":"2024-04-06T12:27:21.046311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:14:39.407591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047517+0000","last_change":"2024-04-06T12:27:21.067188+0000","last_active":"2024-04-06T12:27:22.047517+0000","last_peered":"2024-04-06T12:27:22.047517+0000","last_clean":"2024-04-06T12:27:22.047517+0000","last_became_active":"2024-04-06T12:27:21.066981+0000","last_became_peered":"2024-04-06T12:27:21.066981+0000","last_unstale":"2024-04-06T12:27:22.047517+0000","last_undegraded":"2024-04-06T12:27:22.047517+0000","last_fullsized":"2024-04-06T12:27:22.047517+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:16:15.014557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035060+0000","last_change":"2024-04-06T12:27:20.642838+0000","last_active":"2024-04-06T12:27:21.035060+0000","last_peered":"2024-04-06T12:27:21.035060+0000","last_clean":"2024-04-06T12:27:21.035060+0000","last_became_active":"2024-04-06T12:27:20.642344+0000","last_became_peered":"2024-04-06T12:27:20.642344+0000","last_unstale":"2024-04-06T12:27:21.035060+0000","last_undegraded":"2024-04-06T12:27:21.035060+0000","last_fullsized":"2024-04-06T12:27:21.035060+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:39:48.790443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039044+0000","last_change":"2024-04-06T12:27:21.042767+0000","last_active":"2024-04-06T12:27:22.039044+0000","last_peered":"2024-04-06T12:27:22.039044+0000","last_clean":"2024-04-06T12:27:22.039044+0000","last_became_active":"2024-04-06T12:27:21.042541+0000","last_became_peered":"2024-04-06T12:27:21.042541+0000","last_unstale":"2024-04-06T12:27:22.039044+0000","last_undegraded":"2024-04-06T12:27:22.039044+0000","last_fullsized":"2024-04-06T12:27:22.039044+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:02:23.142749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.049488+0000","last_change":"2024-04-06T12:27:20.640874+0000","last_active":"2024-04-06T12:27:21.049488+0000","last_peered":"2024-04-06T12:27:21.049488+0000","last_clean":"2024-04-06T12:27:21.049488+0000","last_became_active":"2024-04-06T12:27:20.640592+0000","last_became_peered":"2024-04-06T12:27:20.640592+0000","last_unstale":"2024-04-06T12:27:21.049488+0000","last_undegraded":"2024-04-06T12:27:21.049488+0000","last_fullsized":"2024-04-06T12:27:21.049488+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:53:08.693479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046322+0000","last_change":"2024-04-06T12:27:21.050535+0000","last_active":"2024-04-06T12:27:22.046322+0000","last_peered":"2024-04-06T12:27:22.046322+0000","last_clean":"2024-04-06T12:27:22.046322+0000","last_became_active":"2024-04-06T12:27:21.050270+0000","last_became_peered":"2024-04-06T12:27:21.050270+0000","last_unstale":"2024-04-06T12:27:22.046322+0000","last_undegraded":"2024-04-06T12:27:22.046322+0000","last_fullsized":"2024-04-06T12:27:22.046322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:24:04.575202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623334+0000","last_change":"2024-04-06T12:27:19.623334+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623334+0000","last_undegraded":"2024-04-06T12:27:19.623334+0000","last_fullsized":"2024-04-06T12:27:19.623334+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:24.851744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039214+0000","last_change":"2024-04-06T12:27:21.042722+0000","last_active":"2024-04-06T12:27:22.039214+0000","last_peered":"2024-04-06T12:27:22.039214+0000","last_clean":"2024-04-06T12:27:22.039214+0000","last_became_active":"2024-04-06T12:27:21.042460+0000","last_became_peered":"2024-04-06T12:27:21.042460+0000","last_unstale":"2024-04-06T12:27:22.039214+0000","last_undegraded":"2024-04-06T12:27:22.039214+0000","last_fullsized":"2024-04-06T12:27:22.039214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:16:01.192728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.037118+0000","last_change":"2024-04-06T12:27:20.643086+0000","last_active":"2024-04-06T12:27:21.037118+0000","last_peered":"2024-04-06T12:27:21.037118+0000","last_clean":"2024-04-06T12:27:21.037118+0000","last_became_active":"2024-04-06T12:27:20.642767+0000","last_became_peered":"2024-04-06T12:27:20.642767+0000","last_unstale":"2024-04-06T12:27:21.037118+0000","last_undegraded":"2024-04-06T12:27:21.037118+0000","last_fullsized":"2024-04-06T12:27:21.037118+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:54:55.715540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047715+0000","last_change":"2024-04-06T12:27:21.058482+0000","last_active":"2024-04-06T12:27:22.047715+0000","last_peered":"2024-04-06T12:27:22.047715+0000","last_clean":"2024-04-06T12:27:22.047715+0000","last_became_active":"2024-04-06T12:27:21.058235+0000","last_became_peered":"2024-04-06T12:27:21.058235+0000","last_unstale":"2024-04-06T12:27:22.047715+0000","last_undegraded":"2024-04-06T12:27:22.047715+0000","last_fullsized":"2024-04-06T12:27:22.047715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:16.550344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.624131+0000","last_change":"2024-04-06T12:27:19.624131+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.624131+0000","last_undegraded":"2024-04-06T12:27:19.624131+0000","last_fullsized":"2024-04-06T12:27:19.624131+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:15.542425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047714+0000","last_change":"2024-04-06T12:27:21.050955+0000","last_active":"2024-04-06T12:27:22.047714+0000","last_peered":"2024-04-06T12:27:22.047714+0000","last_clean":"2024-04-06T12:27:22.047714+0000","last_became_active":"2024-04-06T12:27:21.050675+0000","last_became_peered":"2024-04-06T12:27:21.050675+0000","last_unstale":"2024-04-06T12:27:22.047714+0000","last_undegraded":"2024-04-06T12:27:22.047714+0000","last_fullsized":"2024-04-06T12:27:22.047714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:57.645194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"83'6","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.067970+0000","last_change":"2024-04-06T12:27:20.638258+0000","last_active":"2024-04-06T12:27:21.067970+0000","last_peered":"2024-04-06T12:27:21.067970+0000","last_clean":"2024-04-06T12:27:21.067970+0000","last_became_active":"2024-04-06T12:27:20.638017+0000","last_became_peered":"2024-04-06T12:27:20.638017+0000","last_unstale":"2024-04-06T12:27:21.067970+0000","last_undegraded":"2024-04-06T12:27:21.067970+0000","last_fullsized":"2024-04-06T12:27:21.067970+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:48:54.260905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047268+0000","last_change":"2024-04-06T12:27:21.048435+0000","last_active":"2024-04-06T12:27:22.047268+0000","last_peered":"2024-04-06T12:27:22.047268+0000","last_clean":"2024-04-06T12:27:22.047268+0000","last_became_active":"2024-04-06T12:27:21.048159+0000","last_became_peered":"2024-04-06T12:27:21.048159+0000","last_unstale":"2024-04-06T12:27:22.047268+0000","last_undegraded":"2024-04-06T12:27:22.047268+0000","last_fullsized":"2024-04-06T12:27:22.047268+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:59:57.937757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623703+0000","last_change":"2024-04-06T12:27:19.623703+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623703+0000","last_undegraded":"2024-04-06T12:27:19.623703+0000","last_fullsized":"2024-04-06T12:27:19.623703+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:48:54.260905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046846+0000","last_change":"2024-04-06T12:27:21.050116+0000","last_active":"2024-04-06T12:27:22.046846+0000","last_peered":"2024-04-06T12:27:22.046846+0000","last_clean":"2024-04-06T12:27:22.046846+0000","last_became_active":"2024-04-06T12:27:21.049828+0000","last_became_peered":"2024-04-06T12:27:21.049828+0000","last_unstale":"2024-04-06T12:27:22.046846+0000","last_undegraded":"2024-04-06T12:27:22.046846+0000","last_fullsized":"2024-04-06T12:27:22.046846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:41:35.413608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.043551+0000","last_change":"2024-04-06T12:27:20.643715+0000","last_active":"2024-04-06T12:27:21.043551+0000","last_peered":"2024-04-06T12:27:21.043551+0000","last_clean":"2024-04-06T12:27:21.043551+0000","last_became_active":"2024-04-06T12:27:20.643243+0000","last_became_peered":"2024-04-06T12:27:20.643243+0000","last_unstale":"2024-04-06T12:27:21.043551+0000","last_undegraded":"2024-04-06T12:27:21.043551+0000","last_fullsized":"2024-04-06T12:27:21.043551+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:48.861736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047578+0000","last_change":"2024-04-06T12:27:21.061425+0000","last_active":"2024-04-06T12:27:22.047578+0000","last_peered":"2024-04-06T12:27:22.047578+0000","last_clean":"2024-04-06T12:27:22.047578+0000","last_became_active":"2024-04-06T12:27:21.061137+0000","last_became_peered":"2024-04-06T12:27:21.061137+0000","last_unstale":"2024-04-06T12:27:22.047578+0000","last_undegraded":"2024-04-06T12:27:22.047578+0000","last_fullsized":"2024-04-06T12:27:22.047578+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:29:33.494735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.622973+0000","last_change":"2024-04-06T12:27:19.622973+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.622973+0000","last_undegraded":"2024-04-06T12:27:19.622973+0000","last_fullsized":"2024-04-06T12:27:19.622973+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:55.070413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046208+0000","last_change":"2024-04-06T12:27:21.051804+0000","last_active":"2024-04-06T12:27:22.046208+0000","last_peered":"2024-04-06T12:27:22.046208+0000","last_clean":"2024-04-06T12:27:22.046208+0000","last_became_active":"2024-04-06T12:27:21.051536+0000","last_became_peered":"2024-04-06T12:27:21.051536+0000","last_unstale":"2024-04-06T12:27:22.046208+0000","last_undegraded":"2024-04-06T12:27:22.046208+0000","last_fullsized":"2024-04-06T12:27:22.046208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:26:12.757657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"83'8","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.063908+0000","last_change":"2024-04-06T12:27:20.640204+0000","last_active":"2024-04-06T12:27:21.063908+0000","last_peered":"2024-04-06T12:27:21.063908+0000","last_clean":"2024-04-06T12:27:21.063908+0000","last_became_active":"2024-04-06T12:27:20.639913+0000","last_became_peered":"2024-04-06T12:27:20.639913+0000","last_unstale":"2024-04-06T12:27:21.063908+0000","last_undegraded":"2024-04-06T12:27:21.063908+0000","last_fullsized":"2024-04-06T12:27:21.063908+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:01:44.323843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.059785+0000","last_change":"2024-04-06T12:27:20.643933+0000","last_active":"2024-04-06T12:27:21.059785+0000","last_peered":"2024-04-06T12:27:21.059785+0000","last_clean":"2024-04-06T12:27:21.059785+0000","last_became_active":"2024-04-06T12:27:20.643644+0000","last_became_peered":"2024-04-06T12:27:20.643644+0000","last_unstale":"2024-04-06T12:27:21.059785+0000","last_undegraded":"2024-04-06T12:27:21.059785+0000","last_fullsized":"2024-04-06T12:27:21.059785+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:55.070413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047888+0000","last_change":"2024-04-06T12:27:21.059329+0000","last_active":"2024-04-06T12:27:22.047888+0000","last_peered":"2024-04-06T12:27:22.047888+0000","last_clean":"2024-04-06T12:27:22.047888+0000","last_became_active":"2024-04-06T12:27:21.058948+0000","last_became_peered":"2024-04-06T12:27:21.058948+0000","last_unstale":"2024-04-06T12:27:22.047888+0000","last_undegraded":"2024-04-06T12:27:22.047888+0000","last_fullsized":"2024-04-06T12:27:22.047888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:56.078333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.622989+0000","last_change":"2024-04-06T12:27:19.622989+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.622989+0000","last_undegraded":"2024-04-06T12:27:19.622989+0000","last_fullsized":"2024-04-06T12:27:19.622989+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:29:32.486815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047663+0000","last_change":"2024-04-06T12:27:21.059198+0000","last_active":"2024-04-06T12:27:22.047663+0000","last_peered":"2024-04-06T12:27:22.047663+0000","last_clean":"2024-04-06T12:27:22.047663+0000","last_became_active":"2024-04-06T12:27:21.058950+0000","last_became_peered":"2024-04-06T12:27:21.058950+0000","last_unstale":"2024-04-06T12:27:22.047663+0000","last_undegraded":"2024-04-06T12:27:22.047663+0000","last_fullsized":"2024-04-06T12:27:22.047663+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:20:12.514805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623711+0000","last_change":"2024-04-06T12:27:19.623711+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623711+0000","last_undegraded":"2024-04-06T12:27:19.623711+0000","last_fullsized":"2024-04-06T12:27:19.623711+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:44.931614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047541+0000","last_change":"2024-04-06T12:27:21.061929+0000","last_active":"2024-04-06T12:27:22.047541+0000","last_peered":"2024-04-06T12:27:22.047541+0000","last_clean":"2024-04-06T12:27:22.047541+0000","last_became_active":"2024-04-06T12:27:21.061627+0000","last_became_peered":"2024-04-06T12:27:21.061627+0000","last_unstale":"2024-04-06T12:27:22.047541+0000","last_undegraded":"2024-04-06T12:27:22.047541+0000","last_fullsized":"2024-04-06T12:27:22.047541+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:02.552663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035432+0000","last_change":"2024-04-06T12:27:20.642400+0000","last_active":"2024-04-06T12:27:21.035432+0000","last_peered":"2024-04-06T12:27:21.035432+0000","last_clean":"2024-04-06T12:27:21.035432+0000","last_became_active":"2024-04-06T12:27:20.642039+0000","last_became_peered":"2024-04-06T12:27:20.642039+0000","last_unstale":"2024-04-06T12:27:21.035432+0000","last_undegraded":"2024-04-06T12:27:21.035432+0000","last_fullsized":"2024-04-06T12:27:21.035432+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:44.642507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039877+0000","last_change":"2024-04-06T12:27:21.046054+0000","last_active":"2024-04-06T12:27:22.039877+0000","last_peered":"2024-04-06T12:27:22.039877+0000","last_clean":"2024-04-06T12:27:22.039877+0000","last_became_active":"2024-04-06T12:27:21.045799+0000","last_became_peered":"2024-04-06T12:27:21.045799+0000","last_unstale":"2024-04-06T12:27:22.039877+0000","last_undegraded":"2024-04-06T12:27:22.039877+0000","last_fullsized":"2024-04-06T12:27:22.039877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:29:16.068086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.059940+0000","last_change":"2024-04-06T12:27:20.635120+0000","last_active":"2024-04-06T12:27:21.059940+0000","last_peered":"2024-04-06T12:27:21.059940+0000","last_clean":"2024-04-06T12:27:21.059940+0000","last_became_active":"2024-04-06T12:27:20.634852+0000","last_became_peered":"2024-04-06T12:27:20.634852+0000","last_unstale":"2024-04-06T12:27:21.059940+0000","last_undegraded":"2024-04-06T12:27:21.059940+0000","last_fullsized":"2024-04-06T12:27:21.059940+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:34:57.334069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046898+0000","last_change":"2024-04-06T12:27:21.044875+0000","last_active":"2024-04-06T12:27:22.046898+0000","last_peered":"2024-04-06T12:27:22.046898+0000","last_clean":"2024-04-06T12:27:22.046898+0000","last_became_active":"2024-04-06T12:27:21.044609+0000","last_became_peered":"2024-04-06T12:27:21.044609+0000","last_unstale":"2024-04-06T12:27:22.046898+0000","last_undegraded":"2024-04-06T12:27:22.046898+0000","last_fullsized":"2024-04-06T12:27:22.046898+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:48:55.268825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.043598+0000","last_change":"2024-04-06T12:27:20.643509+0000","last_active":"2024-04-06T12:27:21.043598+0000","last_peered":"2024-04-06T12:27:21.043598+0000","last_clean":"2024-04-06T12:27:21.043598+0000","last_became_active":"2024-04-06T12:27:20.642877+0000","last_became_peered":"2024-04-06T12:27:20.642877+0000","last_unstale":"2024-04-06T12:27:21.043598+0000","last_undegraded":"2024-04-06T12:27:21.043598+0000","last_fullsized":"2024-04-06T12:27:21.043598+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:12:17.891288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048159+0000","last_change":"2024-04-06T12:27:21.063218+0000","last_active":"2024-04-06T12:27:22.048159+0000","last_peered":"2024-04-06T12:27:22.048159+0000","last_clean":"2024-04-06T12:27:22.048159+0000","last_became_active":"2024-04-06T12:27:21.062952+0000","last_became_peered":"2024-04-06T12:27:21.062952+0000","last_unstale":"2024-04-06T12:27:22.048159+0000","last_undegraded":"2024-04-06T12:27:22.048159+0000","last_fullsized":"2024-04-06T12:27:22.048159+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:54:56.723460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.038449+0000","last_change":"2024-04-06T12:27:20.640135+0000","last_active":"2024-04-06T12:27:21.038449+0000","last_peered":"2024-04-06T12:27:21.038449+0000","last_clean":"2024-04-06T12:27:21.038449+0000","last_became_active":"2024-04-06T12:27:20.639873+0000","last_became_peered":"2024-04-06T12:27:20.639873+0000","last_unstale":"2024-04-06T12:27:21.038449+0000","last_undegraded":"2024-04-06T12:27:21.038449+0000","last_fullsized":"2024-04-06T12:27:21.038449+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:26:31.570422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.040070+0000","last_change":"2024-04-06T12:27:21.053276+0000","last_active":"2024-04-06T12:27:22.040070+0000","last_peered":"2024-04-06T12:27:22.040070+0000","last_clean":"2024-04-06T12:27:22.040070+0000","last_became_active":"2024-04-06T12:27:21.053033+0000","last_became_peered":"2024-04-06T12:27:21.053033+0000","last_unstale":"2024-04-06T12:27:22.040070+0000","last_undegraded":"2024-04-06T12:27:22.040070+0000","last_fullsized":"2024-04-06T12:27:22.040070+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:49:47.315178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.061029+0000","last_change":"2024-04-06T12:27:20.642377+0000","last_active":"2024-04-06T12:27:21.061029+0000","last_peered":"2024-04-06T12:27:21.061029+0000","last_clean":"2024-04-06T12:27:21.061029+0000","last_became_active":"2024-04-06T12:27:20.642111+0000","last_became_peered":"2024-04-06T12:27:20.642111+0000","last_unstale":"2024-04-06T12:27:21.061029+0000","last_undegraded":"2024-04-06T12:27:21.061029+0000","last_fullsized":"2024-04-06T12:27:21.061029+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:46:21.677982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046699+0000","last_change":"2024-04-06T12:27:21.047728+0000","last_active":"2024-04-06T12:27:22.046699+0000","last_peered":"2024-04-06T12:27:22.046699+0000","last_clean":"2024-04-06T12:27:22.046699+0000","last_became_active":"2024-04-06T12:27:21.046317+0000","last_became_peered":"2024-04-06T12:27:21.046317+0000","last_unstale":"2024-04-06T12:27:22.046699+0000","last_undegraded":"2024-04-06T12:27:22.046699+0000","last_fullsized":"2024-04-06T12:27:22.046699+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:08:29.040668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.702984+0000","last_change":"2024-04-06T12:27:20.643133+0000","last_active":"2024-04-06T12:27:21.702984+0000","last_peered":"2024-04-06T12:27:21.702984+0000","last_clean":"2024-04-06T12:27:21.702984+0000","last_became_active":"2024-04-06T12:27:20.642858+0000","last_became_peered":"2024-04-06T12:27:20.642858+0000","last_unstale":"2024-04-06T12:27:21.702984+0000","last_undegraded":"2024-04-06T12:27:21.702984+0000","last_fullsized":"2024-04-06T12:27:21.702984+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:16:14.006637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048294+0000","last_change":"2024-04-06T12:27:21.059468+0000","last_active":"2024-04-06T12:27:22.048294+0000","last_peered":"2024-04-06T12:27:22.048294+0000","last_clean":"2024-04-06T12:27:22.048294+0000","last_became_active":"2024-04-06T12:27:21.059222+0000","last_became_peered":"2024-04-06T12:27:21.059222+0000","last_unstale":"2024-04-06T12:27:22.048294+0000","last_undegraded":"2024-04-06T12:27:22.048294+0000","last_fullsized":"2024-04-06T12:27:22.048294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:20:15.444097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.040662+0000","last_change":"2024-04-06T12:27:20.638922+0000","last_active":"2024-04-06T12:27:21.040662+0000","last_peered":"2024-04-06T12:27:21.040662+0000","last_clean":"2024-04-06T12:27:21.040662+0000","last_became_active":"2024-04-06T12:27:20.638666+0000","last_became_peered":"2024-04-06T12:27:20.638666+0000","last_unstale":"2024-04-06T12:27:21.040662+0000","last_undegraded":"2024-04-06T12:27:21.040662+0000","last_fullsized":"2024-04-06T12:27:21.040662+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:08:18.449996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047650+0000","last_change":"2024-04-06T12:27:21.060870+0000","last_active":"2024-04-06T12:27:22.047650+0000","last_peered":"2024-04-06T12:27:22.047650+0000","last_clean":"2024-04-06T12:27:22.047650+0000","last_became_active":"2024-04-06T12:27:21.060603+0000","last_became_peered":"2024-04-06T12:27:21.060603+0000","last_unstale":"2024-04-06T12:27:22.047650+0000","last_undegraded":"2024-04-06T12:27:22.047650+0000","last_fullsized":"2024-04-06T12:27:22.047650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:48:55.268825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.044529+0000","last_change":"2024-04-06T12:27:20.639042+0000","last_active":"2024-04-06T12:27:21.044529+0000","last_peered":"2024-04-06T12:27:21.044529+0000","last_clean":"2024-04-06T12:27:21.044529+0000","last_became_active":"2024-04-06T12:27:20.638475+0000","last_became_peered":"2024-04-06T12:27:20.638475+0000","last_unstale":"2024-04-06T12:27:21.044529+0000","last_undegraded":"2024-04-06T12:27:21.044529+0000","last_fullsized":"2024-04-06T12:27:21.044529+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:15.542425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047260+0000","last_change":"2024-04-06T12:27:21.060781+0000","last_active":"2024-04-06T12:27:22.047260+0000","last_peered":"2024-04-06T12:27:22.047260+0000","last_clean":"2024-04-06T12:27:22.047260+0000","last_became_active":"2024-04-06T12:27:21.060520+0000","last_became_peered":"2024-04-06T12:27:21.060520+0000","last_unstale":"2024-04-06T12:27:22.047260+0000","last_undegraded":"2024-04-06T12:27:22.047260+0000","last_fullsized":"2024-04-06T12:27:22.047260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:46:22.685902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623238+0000","last_change":"2024-04-06T12:27:19.623238+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623238+0000","last_undegraded":"2024-04-06T12:27:19.623238+0000","last_fullsized":"2024-04-06T12:27:19.623238+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:48.861736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047005+0000","last_change":"2024-04-06T12:27:21.055075+0000","last_active":"2024-04-06T12:27:22.047005+0000","last_peered":"2024-04-06T12:27:22.047005+0000","last_clean":"2024-04-06T12:27:22.047005+0000","last_became_active":"2024-04-06T12:27:21.054806+0000","last_became_peered":"2024-04-06T12:27:21.054806+0000","last_unstale":"2024-04-06T12:27:22.047005+0000","last_undegraded":"2024-04-06T12:27:22.047005+0000","last_fullsized":"2024-04-06T12:27:22.047005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:14.746095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035500+0000","last_change":"2024-04-06T12:27:20.641433+0000","last_active":"2024-04-06T12:27:21.035500+0000","last_peered":"2024-04-06T12:27:21.035500+0000","last_clean":"2024-04-06T12:27:21.035500+0000","last_became_active":"2024-04-06T12:27:20.641181+0000","last_became_peered":"2024-04-06T12:27:20.641181+0000","last_unstale":"2024-04-06T12:27:21.035500+0000","last_undegraded":"2024-04-06T12:27:21.035500+0000","last_fullsized":"2024-04-06T12:27:21.035500+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:20:08.218629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046219+0000","last_change":"2024-04-06T12:27:21.055928+0000","last_active":"2024-04-06T12:27:22.046219+0000","last_peered":"2024-04-06T12:27:22.046219+0000","last_clean":"2024-04-06T12:27:22.046219+0000","last_became_active":"2024-04-06T12:27:21.055641+0000","last_became_peered":"2024-04-06T12:27:21.055641+0000","last_unstale":"2024-04-06T12:27:22.046219+0000","last_undegraded":"2024-04-06T12:27:22.046219+0000","last_fullsized":"2024-04-06T12:27:22.046219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:08:32.179469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035357+0000","last_change":"2024-04-06T12:27:20.643620+0000","last_active":"2024-04-06T12:27:21.035357+0000","last_peered":"2024-04-06T12:27:21.035357+0000","last_clean":"2024-04-06T12:27:21.035357+0000","last_became_active":"2024-04-06T12:27:20.642671+0000","last_became_peered":"2024-04-06T12:27:20.642671+0000","last_unstale":"2024-04-06T12:27:21.035357+0000","last_undegraded":"2024-04-06T12:27:21.035357+0000","last_fullsized":"2024-04-06T12:27:21.035357+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:56:37.738753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046202+0000","last_change":"2024-04-06T12:27:21.046951+0000","last_active":"2024-04-06T12:27:22.046202+0000","last_peered":"2024-04-06T12:27:22.046202+0000","last_clean":"2024-04-06T12:27:22.046202+0000","last_became_active":"2024-04-06T12:27:21.046651+0000","last_became_peered":"2024-04-06T12:27:21.046651+0000","last_unstale":"2024-04-06T12:27:22.046202+0000","last_undegraded":"2024-04-06T12:27:22.046202+0000","last_fullsized":"2024-04-06T12:27:22.046202+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:59:55.516160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.044036+0000","last_change":"2024-04-06T12:27:20.635487+0000","last_active":"2024-04-06T12:27:21.044036+0000","last_peered":"2024-04-06T12:27:21.044036+0000","last_clean":"2024-04-06T12:27:21.044036+0000","last_became_active":"2024-04-06T12:27:20.635194+0000","last_became_peered":"2024-04-06T12:27:20.635194+0000","last_unstale":"2024-04-06T12:27:21.044036+0000","last_undegraded":"2024-04-06T12:27:21.044036+0000","last_fullsized":"2024-04-06T12:27:21.044036+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:01.544743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047357+0000","last_change":"2024-04-06T12:27:21.062962+0000","last_active":"2024-04-06T12:27:22.047357+0000","last_peered":"2024-04-06T12:27:22.047357+0000","last_clean":"2024-04-06T12:27:22.047357+0000","last_became_active":"2024-04-06T12:27:21.062683+0000","last_became_peered":"2024-04-06T12:27:21.062683+0000","last_unstale":"2024-04-06T12:27:22.047357+0000","last_undegraded":"2024-04-06T12:27:22.047357+0000","last_fullsized":"2024-04-06T12:27:22.047357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:49.869655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035100+0000","last_change":"2024-04-06T12:27:20.640825+0000","last_active":"2024-04-06T12:27:21.035100+0000","last_peered":"2024-04-06T12:27:21.035100+0000","last_clean":"2024-04-06T12:27:21.035100+0000","last_became_active":"2024-04-06T12:27:20.640568+0000","last_became_peered":"2024-04-06T12:27:20.640568+0000","last_unstale":"2024-04-06T12:27:21.035100+0000","last_undegraded":"2024-04-06T12:27:21.035100+0000","last_fullsized":"2024-04-06T12:27:21.035100+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:27:16.320148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'161","reported_seq":388,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-06T12:27:17.742656+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:27:17.742656+0000","last_peered":"2024-04-06T12:27:17.742656+0000","last_clean":"2024-04-06T12:27:17.742656+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:27:17.742656+0000","last_undegraded":"2024-04-06T12:27:17.742656+0000","last_fullsized":"2024-04-06T12:27:17.742656+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":110,"acting":110,"num_store_stats":7},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2168,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575034,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58900,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312940,"statfs":{"total":96636764160,"available":96576450560,"internally_reserved":0,"allocated":1564672,"data_stored":1068520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55800000000000005,"15min":0.54900000000000004},"min":{"1min":0.375,"5min":0.219,"15min":0.219},"max":{"1min":0.80100000000000005,"5min":1.143,"15min":1.143},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55700000000000005,"15min":0.55300000000000005},"min":{"1min":0.433,"5min":0.35399999999999998,"15min":0.25900000000000001},"max":{"1min":0.73299999999999998,"5min":0.77100000000000002,"15min":0.78200000000000003},"last":0.48399999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55400000000000005,"15min":0.55100000000000005},"min":{"1min":0.42299999999999999,"5min":0.33600000000000002,"15min":0.31900000000000001},"max":{"1min":0.748,"5min":0.77100000000000002,"15min":0.81399999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.41699999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.77400000000000002,"5min":0.79400000000000004,"15min":0.79600000000000004},"last":0.372}]},{"osd":2,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56999999999999995,"15min":0.55200000000000005},"min":{"1min":0.441,"5min":0.249,"15min":0.249},"max":{"1min":0.77100000000000002,"5min":0.78500000000000003,"15min":0.83299999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57099999999999995,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.24199999999999999},"max":{"1min":0.76600000000000001,"5min":0.78300000000000003,"15min":0.82299999999999995},"last":0.58999999999999997}]},{"osd":3,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58099999999999996,"15min":0.57199999999999995},"min":{"1min":0.45900000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.75800000000000001,"5min":1.357,"15min":1.357},"last":0.502},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59299999999999997,"15min":0.59599999999999997},"min":{"1min":0.44400000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.748,"5min":0.81799999999999995,"15min":0.92000000000000004},"last":0.626}]},{"osd":4,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.45300000000000001,"15min":0.45800000000000002},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.25},"max":{"1min":0.68300000000000005,"5min":0.70399999999999996,"15min":0.83599999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.53700000000000003},"min":{"1min":0.42899999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71399999999999997,"5min":0.77700000000000002,"15min":1.0780000000000001},"last":0.54400000000000004}]},{"osd":5,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.498,"15min":0.51300000000000001},"min":{"1min":0.33600000000000002,"5min":0.29199999999999998,"15min":0.23799999999999999},"max":{"1min":0.73299999999999998,"5min":0.80100000000000005,"15min":0.98199999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55900000000000005,"15min":0.56599999999999995},"min":{"1min":0.374,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.70699999999999996,"5min":0.81599999999999995,"15min":0.93600000000000005},"last":0.60599999999999998}]},{"osd":6,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53900000000000003,"15min":0.54900000000000004},"min":{"1min":0.39300000000000002,"5min":0.33700000000000002,"15min":0.28799999999999998},"max":{"1min":0.70699999999999996,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55800000000000005,"15min":0.56799999999999995},"min":{"1min":0.40200000000000002,"5min":0.30399999999999999,"15min":0.26900000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.88500000000000001},"last":0.58099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575034,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":1019904,"data_stored":503606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.36699999999999999,"5min":0.26200000000000001,"15min":0.23000000000000001},"max":{"1min":0.77500000000000002,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52500000000000002,"15min":0.53300000000000003},"min":{"1min":0.374,"5min":0.30199999999999999,"15min":0.245},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.53300000000000003}]},{"osd":1,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.32700000000000001,"5min":0.32600000000000001,"15min":0.24399999999999999},"max":{"1min":0.78200000000000003,"5min":0.78700000000000003,"15min":0.84999999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.38400000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.83099999999999996},"last":0.50700000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.39100000000000001,"5min":0.30499999999999999,"15min":0.29899999999999999},"max":{"1min":0.749,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.38800000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.56899999999999995,"15min":0.58099999999999996},"min":{"1min":0.35099999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.745,"5min":0.79000000000000004,"15min":0.81200000000000006},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.60899999999999999,"15min":0.61899999999999999},"min":{"1min":0.51700000000000002,"5min":0.41099999999999998,"15min":0.374},"max":{"1min":0.79200000000000004,"5min":0.82799999999999996,"15min":0.872},"last":0.52000000000000002}]},{"osd":4,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.49099999999999999,"15min":0.496},"min":{"1min":0.28599999999999998,"5min":0.27500000000000002,"15min":0.27300000000000002},"max":{"1min":0.747,"5min":0.749,"15min":0.749},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.53000000000000003},"min":{"1min":0.34899999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.73199999999999998,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.43099999999999999}]},{"osd":5,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.47999999999999998,"15min":0.49199999999999999},"min":{"1min":0.34699999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71099999999999997,"5min":0.755,"15min":0.75800000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.50600000000000001,"15min":0.51300000000000001},"min":{"1min":0.32800000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72599999999999998,"5min":0.83699999999999997,"15min":0.92800000000000005},"last":0.45400000000000001}]},{"osd":7,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53000000000000003,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70099999999999996,"5min":0.79800000000000004,"15min":0.81000000000000005},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55800000000000005,"15min":0.56100000000000005},"min":{"1min":0.41899999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80800000000000005,"5min":0.88900000000000001,"15min":0.96099999999999997},"last":0.54800000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575033,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":1044480,"data_stored":505900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.377,"15min":0.36199999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.47699999999999998,"5min":0.58999999999999997,"15min":0.71699999999999997},"last":0.308},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39900000000000002,"15min":0.39600000000000002},"min":{"1min":0.253,"5min":0.215,"15min":0.215},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.32600000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.40500000000000003,"15min":0.39600000000000002},"min":{"1min":0.27200000000000002,"5min":0.23899999999999999,"15min":0.19400000000000001},"max":{"1min":0.57599999999999996,"5min":0.624,"15min":0.67600000000000005},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.432,"15min":0.42999999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.249},"max":{"1min":0.60699999999999998,"5min":0.63600000000000001,"15min":0.79000000000000004},"last":0.28899999999999998}]},{"osd":3,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45600000000000002,"15min":0.44700000000000001},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67400000000000004,"5min":0.67800000000000005,"15min":0.84299999999999997},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.42599999999999999,"15min":0.41899999999999998},"min":{"1min":0.30499999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.56499999999999995,"5min":0.69599999999999995,"15min":0.86299999999999999},"last":0.375}]},{"osd":4,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.623,"15min":0.63200000000000001},"min":{"1min":0.42599999999999999,"5min":0.40100000000000002,"15min":0.34799999999999998},"max":{"1min":0.71499999999999997,"5min":0.86799999999999999,"15min":6.3739999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.63100000000000001,"15min":0.622},"min":{"1min":0.45400000000000001,"5min":0.441,"15min":0.30399999999999999},"max":{"1min":0.84299999999999997,"5min":0.91400000000000003,"15min":1.0269999999999999},"last":0.504}]},{"osd":5,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.624,"15min":0.61399999999999999},"min":{"1min":0.47599999999999998,"5min":0.42999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83999999999999997,"5min":0.91300000000000003,"15min":0.97699999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.628},"min":{"1min":0.442,"5min":0.43099999999999999,"15min":0.35099999999999998},"max":{"1min":0.78700000000000003,"5min":0.999,"15min":1.0089999999999999},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.623,"15min":0.61399999999999999},"min":{"1min":0.47699999999999998,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":0.76500000000000001,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.65300000000000002,"15min":0.64900000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.41799999999999998},"max":{"1min":0.83999999999999997,"5min":0.88700000000000001,"15min":1.151},"last":0.441}]},{"osd":7,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.64800000000000002,"15min":0.65500000000000003},"min":{"1min":0.44600000000000001,"5min":0.41899999999999998,"15min":0.38800000000000001},"max":{"1min":0.81699999999999995,"5min":1.0329999999999999,"15min":6.3639999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.47299999999999998,"5min":0.40200000000000002,"15min":0.39300000000000002},"max":{"1min":0.82499999999999996,"5min":0.95599999999999996,"15min":1.073},"last":0.55400000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575033,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32688,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339152,"statfs":{"total":96636764160,"available":96603291648,"internally_reserved":0,"allocated":1609728,"data_stored":1093940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.38700000000000001,"15min":0.38},"min":{"1min":0.312,"5min":0.19,"15min":0.19},"max":{"1min":0.53900000000000003,"5min":0.80000000000000004,"15min":0.89100000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.40200000000000002},"min":{"1min":0.30599999999999999,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":0.60599999999999998,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.32300000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.40899999999999997,"15min":0.41199999999999998},"min":{"1min":0.29899999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.57799999999999996,"5min":0.60299999999999998,"15min":1.8089999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.434,"15min":0.42899999999999999},"min":{"1min":0.32600000000000001,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":0.64700000000000002,"5min":1.4790000000000001,"15min":1.764},"last":0.35899999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.41899999999999998,"15min":0.41099999999999998},"min":{"1min":0.34100000000000003,"5min":0.252,"15min":0.21299999999999999},"max":{"1min":0.63,"5min":1.25,"15min":1.25},"last":0.309},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.32900000000000001,"5min":0.28499999999999998,"15min":0.249},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.92000000000000004},"last":0.41199999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.627,"15min":0.61799999999999999},"min":{"1min":0.46100000000000002,"5min":0.371,"15min":0.35699999999999998},"max":{"1min":0.77700000000000002,"5min":1.514,"15min":1.514},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.63900000000000001,"15min":0.628},"min":{"1min":0.47599999999999998,"5min":0.37,"15min":0.311},"max":{"1min":0.86799999999999999,"5min":1.202,"15min":1.202},"last":0.60299999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63400000000000001,"15min":0.61899999999999999},"min":{"1min":0.46999999999999997,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.84899999999999998,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64400000000000002,"15min":0.64700000000000002},"min":{"1min":0.45200000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.88600000000000001,"5min":0.94499999999999995,"15min":0.97899999999999998},"last":0.746}]},{"osd":6,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65300000000000002,"15min":0.64800000000000002},"min":{"1min":0.51700000000000002,"5min":0.39100000000000001,"15min":0.378},"max":{"1min":0.84999999999999998,"5min":0.91000000000000003,"15min":0.96699999999999997},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.66900000000000004,"15min":0.66400000000000003},"min":{"1min":0.499,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84999999999999998,"5min":1.613,"15min":1.613},"last":0.58799999999999997}]},{"osd":7,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67000000000000004,"15min":0.66300000000000003},"min":{"1min":0.51500000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90100000000000002,"5min":1.54,"15min":1.54},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.67700000000000005,"15min":0.65700000000000003},"min":{"1min":0.51400000000000001,"5min":0.38400000000000001,"15min":0.36699999999999999},"max":{"1min":0.88300000000000001,"5min":1.498,"15min":1.498},"last":0.57799999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575035,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50156,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321684,"statfs":{"total":96636764160,"available":96585404416,"internally_reserved":0,"allocated":1015808,"data_stored":503572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38600000000000001,"15min":0.38100000000000001},"min":{"1min":0.26700000000000002,"5min":0.214,"15min":0.21099999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":1.159},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.41199999999999998,"15min":0.40899999999999997},"min":{"1min":0.27500000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.54600000000000004,"5min":0.60899999999999999,"15min":1.075},"last":0.29599999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.41099999999999998,"15min":0.41899999999999998},"min":{"1min":0.29599999999999999,"5min":0.26400000000000001,"15min":0.24299999999999999},"max":{"1min":0.57599999999999996,"5min":0.78500000000000003,"15min":3.6179999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.42299999999999999,"15min":0.42999999999999999},"min":{"1min":0.309,"5min":0.27100000000000002,"15min":0.23599999999999999},"max":{"1min":0.58899999999999997,"5min":0.61399999999999999,"15min":1.125},"last":0.32700000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.46000000000000002,"15min":0.46500000000000002},"min":{"1min":0.34399999999999997,"5min":0.30499999999999999,"15min":0.29699999999999999},"max":{"1min":0.63100000000000001,"5min":0.79200000000000004,"15min":3.661},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.44400000000000001,"15min":0.438},"min":{"1min":0.376,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.55900000000000005,"5min":1.0549999999999999,"15min":1.1910000000000001},"last":0.313}]},{"osd":4,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.61499999999999999,"15min":0.60699999999999998},"min":{"1min":0.45900000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81699999999999995,"5min":0.88200000000000001,"15min":1.2170000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63600000000000001,"15min":0.628},"min":{"1min":0.48799999999999999,"5min":0.316,"15min":0.30499999999999999},"max":{"1min":0.89900000000000002,"5min":0.90200000000000002,"15min":1.101},"last":0.47599999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.64800000000000002,"15min":0.65300000000000002},"min":{"1min":0.57599999999999996,"5min":0.42299999999999999,"15min":0.32900000000000001},"max":{"1min":0.90300000000000002,"5min":1.6870000000000001,"15min":3.6920000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.628,"15min":0.625},"min":{"1min":0.49399999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.81200000000000006,"5min":0.89500000000000002,"15min":1.244},"last":0.49199999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64300000000000002,"15min":0.63400000000000001},"min":{"1min":0.56499999999999995,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.84899999999999998,"5min":0.93400000000000005,"15min":1.012},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67100000000000004,"15min":0.66900000000000004},"min":{"1min":0.51300000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93100000000000005,"5min":1.728,"15min":3.7490000000000001},"last":0.48299999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65400000000000003,"15min":0.64900000000000002},"min":{"1min":0.52900000000000003,"5min":0.40699999999999997,"15min":0.35599999999999998},"max":{"1min":0.80500000000000005,"5min":0.86899999999999999,"15min":1.0469999999999999},"last":0.501},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.67700000000000005,"15min":0.68200000000000005},"min":{"1min":0.54300000000000004,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.90900000000000003,"5min":1.758,"15min":3.7200000000000002},"last":0.56299999999999994}]}]},{"osd":3,"up_from":13,"seq":55834575034,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32120,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339720,"statfs":{"total":96636764160,"available":96603873280,"internally_reserved":0,"allocated":1019904,"data_stored":504591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.24199999999999999,"5min":0.21099999999999999,"15min":0.191},"max":{"1min":0.51700000000000002,"5min":0.59699999999999998,"15min":0.63900000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38100000000000001,"15min":0.377},"min":{"1min":0.308,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.21299999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.372,"15min":0.372},"min":{"1min":0.248,"5min":0.22700000000000001,"15min":0.215},"max":{"1min":0.47099999999999997,"5min":0.59999999999999998,"15min":0.62},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40200000000000002,"15min":0.40799999999999997},"min":{"1min":0.34200000000000003,"5min":0.26200000000000001,"15min":0.247},"max":{"1min":0.55200000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.39200000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.44900000000000001,"15min":0.45000000000000001},"min":{"1min":0.32800000000000001,"5min":0.27200000000000002,"15min":0.25700000000000001},"max":{"1min":0.55300000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.432},{"interface":"front","average":{"1min":0.495,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.39700000000000002,"5min":0.317,"15min":0.29899999999999999},"max":{"1min":0.622,"5min":0.64800000000000002,"15min":0.65700000000000003},"last":0.30199999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.49299999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.95999999999999996},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60999999999999999,"15min":0.61699999999999999},"min":{"1min":0.41899999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.74099999999999999,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.67100000000000004}]},{"osd":5,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.64400000000000002,"15min":0.65500000000000003},"min":{"1min":0.46000000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.91100000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.625},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65300000000000002,"15min":0.65200000000000002},"min":{"1min":0.42999999999999999,"5min":0.38200000000000001,"15min":0.36899999999999999},"max":{"1min":0.94499999999999995,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.65600000000000003}]},{"osd":6,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60799999999999998,"15min":0.61799999999999999},"min":{"1min":0.47799999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.96999999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63500000000000001,"15min":0.64800000000000002},"min":{"1min":0.436,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91800000000000004,"5min":0.92600000000000005,"15min":0.94699999999999995},"last":0.70899999999999996}]},{"osd":7,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.629,"15min":0.63700000000000001},"min":{"1min":0.46899999999999997,"5min":0.40400000000000003,"15min":0.33300000000000002},"max":{"1min":0.873,"5min":0.89000000000000001,"15min":0.93300000000000005},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65100000000000002,"15min":0.65700000000000003},"min":{"1min":0.52200000000000002,"5min":0.39500000000000002,"15min":0.373},"max":{"1min":0.93300000000000005,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.54600000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575034,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":1032192,"data_stored":506155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.47499999999999998,"5min":0.29399999999999998,"15min":0.23899999999999999},"max":{"1min":0.71099999999999997,"5min":0.80800000000000005,"15min":0.82999999999999996},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55200000000000005,"15min":0.55400000000000005},"min":{"1min":0.49099999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.82199999999999995},"last":0.58999999999999997}]},{"osd":1,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.59299999999999997,"15min":0.63500000000000001},"min":{"1min":0.48199999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74399999999999999,"5min":0.76900000000000002,"15min":4.3479999999999999},"last":0.629},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58699999999999997,"15min":0.623},"min":{"1min":0.48399999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77100000000000002,"5min":0.78400000000000003,"15min":4.3319999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.56299999999999994,"15min":0.56599999999999995},"min":{"1min":0.47399999999999998,"5min":0.313,"15min":0.311},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.78400000000000003},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.58099999999999996,"15min":0.58199999999999996},"min":{"1min":0.50900000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72399999999999998,"5min":0.74299999999999999,"15min":0.82199999999999995},"last":0.53000000000000003}]},{"osd":3,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55100000000000005,"15min":0.59199999999999997},"min":{"1min":0.432,"5min":0.36399999999999999,"15min":0.34899999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":4.1950000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.59699999999999998,"15min":0.60199999999999998},"min":{"1min":0.497,"5min":0.40500000000000003,"15min":0.36099999999999999},"max":{"1min":0.73699999999999999,"5min":0.75900000000000001,"15min":1.2330000000000001},"last":0.61299999999999999}]},{"osd":5,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.46500000000000002,"15min":0.48999999999999999},"min":{"1min":0.36199999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71099999999999997,"5min":0.76300000000000001,"15min":1.196},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.497,"15min":0.502},"min":{"1min":0.39400000000000002,"5min":0.33500000000000002,"15min":0.312},"max":{"1min":0.63900000000000001,"5min":0.74299999999999999,"15min":0.77000000000000002},"last":0.39800000000000002}]},{"osd":6,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.498,"15min":0.51100000000000001},"min":{"1min":0.33200000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.64200000000000002,"5min":0.754,"15min":1.022},"last":0.379},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.52500000000000002,"15min":0.54400000000000004},"min":{"1min":0.39500000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67400000000000004,"5min":0.73699999999999999,"15min":0.872},"last":0.49299999999999999}]},{"osd":7,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51200000000000001,"15min":0.54100000000000004},"min":{"1min":0.377,"5min":0.311,"15min":0.311},"max":{"1min":0.68500000000000005,"5min":0.79300000000000004,"15min":0.91500000000000004},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55000000000000004,"15min":0.57699999999999996},"min":{"1min":0.39300000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73599999999999999,"5min":0.752,"15min":1.0589999999999999},"last":0.44400000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575034,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46064,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325776,"statfs":{"total":96636764160,"available":96589594624,"internally_reserved":0,"allocated":1019904,"data_stored":503594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.36799999999999999,"5min":0.32700000000000001,"15min":0.24199999999999999},"max":{"1min":0.75900000000000001,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.29499999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.748,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.61199999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.30399999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55200000000000005,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.34499999999999997,"15min":0.30599999999999999},"max":{"1min":0.73599999999999999,"5min":0.74299999999999999,"15min":0.76200000000000001},"last":0.55800000000000005}]},{"osd":2,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56999999999999995,"15min":0.56200000000000006},"min":{"1min":0.33200000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.72199999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.629},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.40999999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72099999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.496}]},{"osd":3,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.33900000000000002,"5min":0.32500000000000001,"15min":0.31},"max":{"1min":0.77800000000000002,"5min":0.874,"15min":0.874},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.436,"5min":0.377,"15min":0.32300000000000001},"max":{"1min":0.73099999999999998,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.58899999999999997}]},{"osd":4,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.28299999999999997},"max":{"1min":0.755,"5min":0.755,"15min":0.81200000000000006},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54000000000000004,"15min":0.53300000000000003},"min":{"1min":0.42599999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73699999999999999,"5min":0.80000000000000004,"15min":0.91800000000000004},"last":0.503}]},{"osd":6,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49099999999999999,"15min":0.48599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.28000000000000003},"max":{"1min":0.63800000000000001,"5min":0.73899999999999999,"15min":0.88700000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.496,"15min":0.49299999999999999},"min":{"1min":0.33100000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64800000000000002,"5min":0.73199999999999998,"15min":0.83299999999999996},"last":0.47399999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.252},"max":{"1min":0.68999999999999995,"5min":0.78600000000000003,"15min":0.79100000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60999999999999999,"15min":0.60199999999999998},"min":{"1min":0.44800000000000001,"5min":0.41399999999999998,"15min":0.35199999999999998},"max":{"1min":0.72499999999999998,"5min":0.85299999999999998,"15min":0.85599999999999998},"last":0.54400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:27:25.538 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:27:25.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.698+0000 7fae44270700 1 Processor -- start 2024-04-06T12:27:25.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.698+0000 7fae44270700 1 -- start start 2024-04-06T12:27:25.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.699+0000 7fae44270700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fae3c0b3e20 0x7fae3c15e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:25.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.699+0000 7fae44270700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c0b3990 con 0x7fae3c15d7b0 2024-04-06T12:27:25.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.699+0000 7fae44270700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c0b3060 con 0x7fae3c06ae40 2024-04-06T12:27:25.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.699+0000 7fae44270700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae3c0b4230 con 0x7fae3c0b3e20 2024-04-06T12:27:25.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.699+0000 7fae4200c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fae3c15d7b0 0x7fae3c06a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56910/0 (socket says 172.21.15.189:56910) 2024-04-06T12:27:25.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.699+0000 7fae4200c700 1 -- 172.21.15.189:0/2189483047 learned_addr learned my addr 172.21.15.189:0/2189483047 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:25.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.699+0000 7fae4280d700 1 --2- 172.21.15.189:0/2189483047 >> v2:172.21.15.192:3300/0 conn(0x7fae3c0b3e20 0x7fae3c15e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:25.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.699+0000 7fae4100a700 1 -- 172.21.15.189:0/2189483047 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267921868 0 0) 0x7fae3c0b3990 con 0x7fae3c15d7b0 2024-04-06T12:27:25.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.699+0000 7fae4100a700 1 -- 172.21.15.189:0/2189483047 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae280035b0 con 0x7fae3c15d7b0 2024-04-06T12:27:25.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.699+0000 7fae4280d700 1 -- 172.21.15.189:0/2189483047 >> v1:172.21.15.192:6789/0 conn(0x7fae3c06ae40 legacy=0x7fae3c15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:25.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.699+0000 7fae4280d700 1 -- 172.21.15.189:0/2189483047 >> v1:172.21.15.189:6789/0 conn(0x7fae3c15d7b0 legacy=0x7fae3c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:25.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.699+0000 7fae4280d700 1 -- 172.21.15.189:0/2189483047 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c0b3340 con 0x7fae3c0b3e20 2024-04-06T12:27:25.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.700+0000 7fae4280d700 1 --2- 172.21.15.189:0/2189483047 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae3c0b3e20 0x7fae3c15e500 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fae34009d90 tx=0x7fae3400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f0f2c87fe2d3b4b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:25.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.700+0000 7fae4100a700 1 -- 172.21.15.189:0/2189483047 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3400e070 con 0x7fae3c0b3e20 2024-04-06T12:27:25.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.700+0000 7fae4100a700 1 -- 172.21.15.189:0/2189483047 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3400bb20 con 0x7fae3c0b3e20 2024-04-06T12:27:25.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.700+0000 7fae4100a700 1 -- 172.21.15.189:0/2189483047 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3400be90 con 0x7fae3c0b3e20 2024-04-06T12:27:25.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.700+0000 7fae44270700 1 -- 172.21.15.189:0/2189483047 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae3c0b3e20 msgr2=0x7fae3c15e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:25.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.700+0000 7fae44270700 1 --2- 172.21.15.189:0/2189483047 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae3c0b3e20 0x7fae3c15e500 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fae34009d90 tx=0x7fae3400b330 comp rx=0 tx=0).stop 2024-04-06T12:27:25.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.700+0000 7fae44270700 1 -- 172.21.15.189:0/2189483047 shutdown_connections 2024-04-06T12:27:25.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.700+0000 7fae44270700 1 --2- 172.21.15.189:0/2189483047 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae3c0b3e20 0x7fae3c15e500 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.700+0000 7fae44270700 1 -- 172.21.15.189:0/2189483047 >> 172.21.15.189:0/2189483047 conn(0x7fae3c0d00b0 msgr2=0x7fae3c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:25.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.700+0000 7fae44270700 1 -- 172.21.15.189:0/2189483047 shutdown_connections 2024-04-06T12:27:25.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.700+0000 7fae44270700 1 -- 172.21.15.189:0/2189483047 wait complete. 2024-04-06T12:27:25.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.702+0000 7fae44270700 1 Processor -- start 2024-04-06T12:27:25.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.702+0000 7fae44270700 1 -- start start 2024-04-06T12:27:25.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.702+0000 7fae44270700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae3c06ae40 0x7fae3c1543c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.702+0000 7fae44270700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae3c0b3e20 0x7fae3c157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.702+0000 7fae44270700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae3c15d7b0 0x7fae3c1526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.702+0000 7fae44270700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c0cc530 con 0x7fae3c0b3e20 2024-04-06T12:27:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.702+0000 7fae44270700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c154d50 con 0x7fae3c15d7b0 2024-04-06T12:27:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.702+0000 7fae44270700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c154ff0 con 0x7fae3c06ae40 2024-04-06T12:27:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.702+0000 7fae4180b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae3c0b3e20 0x7fae3c157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.703+0000 7fae4180b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae3c0b3e20 0x7fae3c157050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37624/0 (socket says 172.21.15.189:37624) 2024-04-06T12:27:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.703+0000 7fae4180b700 1 -- 172.21.15.189:0/3511922070 learned_addr learned my addr 172.21.15.189:0/3511922070 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.703+0000 7fae4280d700 1 --2- 172.21.15.189:0/3511922070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae3c15d7b0 0x7fae3c1526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.703+0000 7fae4200c700 1 --2- 172.21.15.189:0/3511922070 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae3c06ae40 0x7fae3c1543c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.703+0000 7fae4180b700 1 -- 172.21.15.189:0/3511922070 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae3c06ae40 msgr2=0x7fae3c1543c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.703+0000 7fae4180b700 1 --2- 172.21.15.189:0/3511922070 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae3c06ae40 0x7fae3c1543c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.703+0000 7fae4180b700 1 -- 172.21.15.189:0/3511922070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae3c15d7b0 msgr2=0x7fae3c1526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.703+0000 7fae4180b700 1 --2- 172.21.15.189:0/3511922070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae3c15d7b0 0x7fae3c1526d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.703+0000 7fae4180b700 1 -- 172.21.15.189:0/3511922070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae34009a70 con 0x7fae3c0b3e20 2024-04-06T12:27:25.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.703+0000 7fae4180b700 1 --2- 172.21.15.189:0/3511922070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae3c0b3e20 0x7fae3c157050 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fae2c00c490 tx=0x7fae2c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:25.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.703+0000 7fae32ffd700 1 -- 172.21.15.189:0/3511922070 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c0052d0 con 0x7fae3c0b3e20 2024-04-06T12:27:25.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.703+0000 7fae32ffd700 1 -- 172.21.15.189:0/3511922070 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2c005450 con 0x7fae3c0b3e20 2024-04-06T12:27:25.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.703+0000 7fae44270700 1 -- 172.21.15.189:0/3511922070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae3c1552d0 con 0x7fae3c0b3e20 2024-04-06T12:27:25.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.703+0000 7fae32ffd700 1 -- 172.21.15.189:0/3511922070 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c006aa0 con 0x7fae3c0b3e20 2024-04-06T12:27:25.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.703+0000 7fae44270700 1 -- 172.21.15.189:0/3511922070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae3c1534f0 con 0x7fae3c0b3e20 2024-04-06T12:27:25.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.704+0000 7fae44270700 1 -- 172.21.15.189:0/3511922070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae3c1557f0 con 0x7fae3c0b3e20 2024-04-06T12:27:25.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.704+0000 7fae32ffd700 1 -- 172.21.15.189:0/3511922070 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fae2c003680 con 0x7fae3c0b3e20 2024-04-06T12:27:25.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.705+0000 7fae32ffd700 1 --2- 172.21.15.189:0/3511922070 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fae2006adb0 0x7fae2006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:25.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.705+0000 7fae32ffd700 1 -- 172.21.15.189:0/3511922070 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fae2c088950 con 0x7fae3c0b3e20 2024-04-06T12:27:25.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.705+0000 7fae4200c700 1 --2- 172.21.15.189:0/3511922070 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fae2006adb0 0x7fae2006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:25.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.705+0000 7fae4200c700 1 --2- 172.21.15.189:0/3511922070 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fae2006adb0 0x7fae2006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fae380011e0 tx=0x7fae38009040 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:25.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.709+0000 7fae32ffd700 1 -- 172.21.15.189:0/3511922070 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fae3c1557f0 con 0x7fae3c0b3e20 2024-04-06T12:27:25.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.900+0000 7fae44270700 1 -- 172.21.15.189:0/3511922070 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fae3c0cc320 con 0x7fae2006adb0 2024-04-06T12:27:25.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.908+0000 7fae32ffd700 1 -- 172.21.15.189:0/3511922070 <== mgr.5582 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287696 (secure 0 0 0) 0x7fae38005920 con 0x7fae2006adb0 2024-04-06T12:27:25.909 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.911+0000 7fae44270700 1 -- 172.21.15.189:0/3511922070 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fae2006adb0 msgr2=0x7fae2006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.911+0000 7fae44270700 1 --2- 172.21.15.189:0/3511922070 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fae2006adb0 0x7fae2006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fae380011e0 tx=0x7fae38009040 comp rx=0 tx=0).stop 2024-04-06T12:27:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.911+0000 7fae44270700 1 -- 172.21.15.189:0/3511922070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae3c0b3e20 msgr2=0x7fae3c157050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.911+0000 7fae44270700 1 --2- 172.21.15.189:0/3511922070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae3c0b3e20 0x7fae3c157050 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fae2c00c490 tx=0x7fae2c00c850 comp rx=0 tx=0).stop 2024-04-06T12:27:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.911+0000 7fae44270700 1 -- 172.21.15.189:0/3511922070 shutdown_connections 2024-04-06T12:27:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.911+0000 7fae44270700 1 --2- 172.21.15.189:0/3511922070 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fae2006adb0 0x7fae2006d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.911+0000 7fae44270700 1 --2- 172.21.15.189:0/3511922070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae3c15d7b0 0x7fae3c1526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.911+0000 7fae44270700 1 --2- 172.21.15.189:0/3511922070 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae3c06ae40 0x7fae3c1543c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.911+0000 7fae44270700 1 --2- 172.21.15.189:0/3511922070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae3c0b3e20 0x7fae3c157050 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.911+0000 7fae44270700 1 -- 172.21.15.189:0/3511922070 >> 172.21.15.189:0/3511922070 conn(0x7fae3c0d00b0 msgr2=0x7fae3c0adbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:25.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.911+0000 7fae44270700 1 -- 172.21.15.189:0/3511922070 shutdown_connections 2024-04-06T12:27:25.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:25.911+0000 7fae44270700 1 -- 172.21.15.189:0/3511922070 wait complete. 2024-04-06T12:27:25.913 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:27:25.921 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-06T12:27:25.309302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592536,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":273,"num_write_kb":2421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":192,"ondisk_log_size":192,"up":176,"acting":176,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":137,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":316308,"kb_used_data":9108,"kb_used_omap":12,"kb_used_meta":307059,"kb_avail":754658412,"statfs":{"total":773094113280,"available":772770213888,"internally_reserved":0,"allocated":9326592,"data_stored":5189878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":314428984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2168,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.693015"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046604+0000","last_change":"2024-04-06T12:27:21.054450+0000","last_active":"2024-04-06T12:27:22.046604+0000","last_peered":"2024-04-06T12:27:22.046604+0000","last_clean":"2024-04-06T12:27:22.046604+0000","last_became_active":"2024-04-06T12:27:21.054202+0000","last_became_peered":"2024-04-06T12:27:21.054202+0000","last_unstale":"2024-04-06T12:27:22.046604+0000","last_undegraded":"2024-04-06T12:27:22.046604+0000","last_fullsized":"2024-04-06T12:27:22.046604+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:29:26.946143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046360+0000","last_change":"2024-04-06T12:27:21.053119+0000","last_active":"2024-04-06T12:27:22.046360+0000","last_peered":"2024-04-06T12:27:22.046360+0000","last_clean":"2024-04-06T12:27:22.046360+0000","last_became_active":"2024-04-06T12:27:21.052814+0000","last_became_peered":"2024-04-06T12:27:21.052814+0000","last_unstale":"2024-04-06T12:27:22.046360+0000","last_undegraded":"2024-04-06T12:27:22.046360+0000","last_fullsized":"2024-04-06T12:27:22.046360+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:14.746095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048036+0000","last_change":"2024-04-06T12:27:21.048815+0000","last_active":"2024-04-06T12:27:22.048036+0000","last_peered":"2024-04-06T12:27:22.048036+0000","last_clean":"2024-04-06T12:27:22.048036+0000","last_became_active":"2024-04-06T12:27:21.047390+0000","last_became_peered":"2024-04-06T12:27:21.047390+0000","last_unstale":"2024-04-06T12:27:22.048036+0000","last_undegraded":"2024-04-06T12:27:22.048036+0000","last_fullsized":"2024-04-06T12:27:22.048036+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:53:37.857841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047263+0000","last_change":"2024-04-06T12:27:21.064877+0000","last_active":"2024-04-06T12:27:22.047263+0000","last_peered":"2024-04-06T12:27:22.047263+0000","last_clean":"2024-04-06T12:27:22.047263+0000","last_became_active":"2024-04-06T12:27:21.064649+0000","last_became_peered":"2024-04-06T12:27:21.064649+0000","last_unstale":"2024-04-06T12:27:22.047263+0000","last_undegraded":"2024-04-06T12:27:22.047263+0000","last_fullsized":"2024-04-06T12:27:22.047263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:25.859663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.040075+0000","last_change":"2024-04-06T12:27:21.054089+0000","last_active":"2024-04-06T12:27:22.040075+0000","last_peered":"2024-04-06T12:27:22.040075+0000","last_clean":"2024-04-06T12:27:22.040075+0000","last_became_active":"2024-04-06T12:27:21.053844+0000","last_became_peered":"2024-04-06T12:27:21.053844+0000","last_unstale":"2024-04-06T12:27:22.040075+0000","last_undegraded":"2024-04-06T12:27:22.040075+0000","last_fullsized":"2024-04-06T12:27:22.040075+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:39:49.798363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046048+0000","last_change":"2024-04-06T12:27:21.060315+0000","last_active":"2024-04-06T12:27:22.046048+0000","last_peered":"2024-04-06T12:27:22.046048+0000","last_clean":"2024-04-06T12:27:22.046048+0000","last_became_active":"2024-04-06T12:27:21.060097+0000","last_became_peered":"2024-04-06T12:27:21.060097+0000","last_unstale":"2024-04-06T12:27:22.046048+0000","last_undegraded":"2024-04-06T12:27:22.046048+0000","last_fullsized":"2024-04-06T12:27:22.046048+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:53:09.701399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046794+0000","last_change":"2024-04-06T12:27:21.053746+0000","last_active":"2024-04-06T12:27:22.046794+0000","last_peered":"2024-04-06T12:27:22.046794+0000","last_clean":"2024-04-06T12:27:22.046794+0000","last_became_active":"2024-04-06T12:27:21.053476+0000","last_became_peered":"2024-04-06T12:27:21.053476+0000","last_unstale":"2024-04-06T12:27:22.046794+0000","last_undegraded":"2024-04-06T12:27:22.046794+0000","last_fullsized":"2024-04-06T12:27:22.046794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:59:57.937757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047029+0000","last_change":"2024-04-06T12:27:21.051431+0000","last_active":"2024-04-06T12:27:22.047029+0000","last_peered":"2024-04-06T12:27:22.047029+0000","last_clean":"2024-04-06T12:27:22.047029+0000","last_became_active":"2024-04-06T12:27:21.050731+0000","last_became_peered":"2024-04-06T12:27:21.050731+0000","last_unstale":"2024-04-06T12:27:22.047029+0000","last_undegraded":"2024-04-06T12:27:22.047029+0000","last_fullsized":"2024-04-06T12:27:22.047029+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:26:12.757657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047263+0000","last_change":"2024-04-06T12:27:21.056611+0000","last_active":"2024-04-06T12:27:22.047263+0000","last_peered":"2024-04-06T12:27:22.047263+0000","last_clean":"2024-04-06T12:27:22.047263+0000","last_became_active":"2024-04-06T12:27:21.056361+0000","last_became_peered":"2024-04-06T12:27:21.056361+0000","last_unstale":"2024-04-06T12:27:22.047263+0000","last_undegraded":"2024-04-06T12:27:22.047263+0000","last_fullsized":"2024-04-06T12:27:22.047263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:41:13.712569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047557+0000","last_change":"2024-04-06T12:27:21.053678+0000","last_active":"2024-04-06T12:27:22.047557+0000","last_peered":"2024-04-06T12:27:22.047557+0000","last_clean":"2024-04-06T12:27:22.047557+0000","last_became_active":"2024-04-06T12:27:21.053132+0000","last_became_peered":"2024-04-06T12:27:21.053132+0000","last_unstale":"2024-04-06T12:27:22.047557+0000","last_undegraded":"2024-04-06T12:27:22.047557+0000","last_fullsized":"2024-04-06T12:27:22.047557+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:57.645194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046311+0000","last_change":"2024-04-06T12:27:21.042215+0000","last_active":"2024-04-06T12:27:22.046311+0000","last_peered":"2024-04-06T12:27:22.046311+0000","last_clean":"2024-04-06T12:27:22.046311+0000","last_became_active":"2024-04-06T12:27:21.041921+0000","last_became_peered":"2024-04-06T12:27:21.041921+0000","last_unstale":"2024-04-06T12:27:22.046311+0000","last_undegraded":"2024-04-06T12:27:22.046311+0000","last_fullsized":"2024-04-06T12:27:22.046311+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:33:26.404172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046752+0000","last_change":"2024-04-06T12:27:21.056018+0000","last_active":"2024-04-06T12:27:22.046752+0000","last_peered":"2024-04-06T12:27:22.046752+0000","last_clean":"2024-04-06T12:27:22.046752+0000","last_became_active":"2024-04-06T12:27:21.055764+0000","last_became_peered":"2024-04-06T12:27:21.055764+0000","last_unstale":"2024-04-06T12:27:22.046752+0000","last_undegraded":"2024-04-06T12:27:22.046752+0000","last_fullsized":"2024-04-06T12:27:22.046752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:49:34.397313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047499+0000","last_change":"2024-04-06T12:27:21.052386+0000","last_active":"2024-04-06T12:27:22.047499+0000","last_peered":"2024-04-06T12:27:22.047499+0000","last_clean":"2024-04-06T12:27:22.047499+0000","last_became_active":"2024-04-06T12:27:21.052081+0000","last_became_peered":"2024-04-06T12:27:21.052081+0000","last_unstale":"2024-04-06T12:27:22.047499+0000","last_undegraded":"2024-04-06T12:27:22.047499+0000","last_fullsized":"2024-04-06T12:27:22.047499+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:41:13.712569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047792+0000","last_change":"2024-04-06T12:27:21.059175+0000","last_active":"2024-04-06T12:27:22.047792+0000","last_peered":"2024-04-06T12:27:22.047792+0000","last_clean":"2024-04-06T12:27:22.047792+0000","last_became_active":"2024-04-06T12:27:21.058651+0000","last_became_peered":"2024-04-06T12:27:21.058651+0000","last_unstale":"2024-04-06T12:27:22.047792+0000","last_undegraded":"2024-04-06T12:27:22.047792+0000","last_fullsized":"2024-04-06T12:27:22.047792+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:45.939534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039097+0000","last_change":"2024-04-06T12:27:21.062933+0000","last_active":"2024-04-06T12:27:22.039097+0000","last_peered":"2024-04-06T12:27:22.039097+0000","last_clean":"2024-04-06T12:27:22.039097+0000","last_became_active":"2024-04-06T12:27:21.062688+0000","last_became_peered":"2024-04-06T12:27:21.062688+0000","last_unstale":"2024-04-06T12:27:22.039097+0000","last_undegraded":"2024-04-06T12:27:22.039097+0000","last_fullsized":"2024-04-06T12:27:22.039097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:30:58.878304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047660+0000","last_change":"2024-04-06T12:27:21.059206+0000","last_active":"2024-04-06T12:27:22.047660+0000","last_peered":"2024-04-06T12:27:22.047660+0000","last_clean":"2024-04-06T12:27:22.047660+0000","last_became_active":"2024-04-06T12:27:21.058932+0000","last_became_peered":"2024-04-06T12:27:21.058932+0000","last_unstale":"2024-04-06T12:27:22.047660+0000","last_undegraded":"2024-04-06T12:27:22.047660+0000","last_fullsized":"2024-04-06T12:27:22.047660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:08:32.179469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.038878+0000","last_change":"2024-04-06T12:27:21.053594+0000","last_active":"2024-04-06T12:27:22.038878+0000","last_peered":"2024-04-06T12:27:22.038878+0000","last_clean":"2024-04-06T12:27:22.038878+0000","last_became_active":"2024-04-06T12:27:21.053330+0000","last_became_peered":"2024-04-06T12:27:21.053330+0000","last_unstale":"2024-04-06T12:27:22.038878+0000","last_undegraded":"2024-04-06T12:27:22.038878+0000","last_fullsized":"2024-04-06T12:27:22.038878+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:54:27.984787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.038730+0000","last_change":"2024-04-06T12:27:21.053574+0000","last_active":"2024-04-06T12:27:22.038730+0000","last_peered":"2024-04-06T12:27:22.038730+0000","last_clean":"2024-04-06T12:27:22.038730+0000","last_became_active":"2024-04-06T12:27:21.053289+0000","last_became_peered":"2024-04-06T12:27:21.053289+0000","last_unstale":"2024-04-06T12:27:22.038730+0000","last_undegraded":"2024-04-06T12:27:22.038730+0000","last_fullsized":"2024-04-06T12:27:22.038730+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:44.646361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048049+0000","last_change":"2024-04-06T12:27:21.049364+0000","last_active":"2024-04-06T12:27:22.048049+0000","last_peered":"2024-04-06T12:27:22.048049+0000","last_clean":"2024-04-06T12:27:22.048049+0000","last_became_active":"2024-04-06T12:27:21.048571+0000","last_became_peered":"2024-04-06T12:27:21.048571+0000","last_unstale":"2024-04-06T12:27:22.048049+0000","last_undegraded":"2024-04-06T12:27:22.048049+0000","last_fullsized":"2024-04-06T12:27:22.048049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:24:04.575202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048237+0000","last_change":"2024-04-06T12:27:21.048886+0000","last_active":"2024-04-06T12:27:22.048237+0000","last_peered":"2024-04-06T12:27:22.048237+0000","last_clean":"2024-04-06T12:27:22.048237+0000","last_became_active":"2024-04-06T12:27:21.048635+0000","last_became_peered":"2024-04-06T12:27:21.048635+0000","last_unstale":"2024-04-06T12:27:22.048237+0000","last_undegraded":"2024-04-06T12:27:22.048237+0000","last_fullsized":"2024-04-06T12:27:22.048237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:14.746095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047147+0000","last_change":"2024-04-06T12:27:21.045989+0000","last_active":"2024-04-06T12:27:22.047147+0000","last_peered":"2024-04-06T12:27:22.047147+0000","last_clean":"2024-04-06T12:27:22.047147+0000","last_became_active":"2024-04-06T12:27:21.045733+0000","last_became_peered":"2024-04-06T12:27:21.045733+0000","last_unstale":"2024-04-06T12:27:22.047147+0000","last_undegraded":"2024-04-06T12:27:22.047147+0000","last_fullsized":"2024-04-06T12:27:22.047147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:53:59.500416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046223+0000","last_change":"2024-04-06T12:27:21.063222+0000","last_active":"2024-04-06T12:27:22.046223+0000","last_peered":"2024-04-06T12:27:22.046223+0000","last_clean":"2024-04-06T12:27:22.046223+0000","last_became_active":"2024-04-06T12:27:21.062971+0000","last_became_peered":"2024-04-06T12:27:21.062971+0000","last_unstale":"2024-04-06T12:27:22.046223+0000","last_undegraded":"2024-04-06T12:27:22.046223+0000","last_fullsized":"2024-04-06T12:27:22.046223+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:41:35.413608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046965+0000","last_change":"2024-04-06T12:27:21.057755+0000","last_active":"2024-04-06T12:27:22.046965+0000","last_peered":"2024-04-06T12:27:22.046965+0000","last_clean":"2024-04-06T12:27:22.046965+0000","last_became_active":"2024-04-06T12:27:21.057491+0000","last_became_peered":"2024-04-06T12:27:21.057491+0000","last_unstale":"2024-04-06T12:27:22.046965+0000","last_undegraded":"2024-04-06T12:27:22.046965+0000","last_fullsized":"2024-04-06T12:27:22.046965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:54:23.429749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046895+0000","last_change":"2024-04-06T12:27:21.048114+0000","last_active":"2024-04-06T12:27:22.046895+0000","last_peered":"2024-04-06T12:27:22.046895+0000","last_clean":"2024-04-06T12:27:22.046895+0000","last_became_active":"2024-04-06T12:27:21.047854+0000","last_became_peered":"2024-04-06T12:27:21.047854+0000","last_unstale":"2024-04-06T12:27:22.046895+0000","last_undegraded":"2024-04-06T12:27:22.046895+0000","last_fullsized":"2024-04-06T12:27:22.046895+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:02.552663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039826+0000","last_change":"2024-04-06T12:27:21.054604+0000","last_active":"2024-04-06T12:27:22.039826+0000","last_peered":"2024-04-06T12:27:22.039826+0000","last_clean":"2024-04-06T12:27:22.039826+0000","last_became_active":"2024-04-06T12:27:21.049845+0000","last_became_peered":"2024-04-06T12:27:21.049845+0000","last_unstale":"2024-04-06T12:27:22.039826+0000","last_undegraded":"2024-04-06T12:27:22.039826+0000","last_fullsized":"2024-04-06T12:27:22.039826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:26.827213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047368+0000","last_change":"2024-04-06T12:27:21.064380+0000","last_active":"2024-04-06T12:27:22.047368+0000","last_peered":"2024-04-06T12:27:22.047368+0000","last_clean":"2024-04-06T12:27:22.047368+0000","last_became_active":"2024-04-06T12:27:21.064115+0000","last_became_peered":"2024-04-06T12:27:21.064115+0000","last_unstale":"2024-04-06T12:27:22.047368+0000","last_undegraded":"2024-04-06T12:27:22.047368+0000","last_fullsized":"2024-04-06T12:27:22.047368+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:12:18.899208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.065495+0000","last_change":"2024-04-06T12:27:20.639851+0000","last_active":"2024-04-06T12:27:21.065495+0000","last_peered":"2024-04-06T12:27:21.065495+0000","last_clean":"2024-04-06T12:27:21.065495+0000","last_became_active":"2024-04-06T12:27:20.639619+0000","last_became_peered":"2024-04-06T12:27:20.639619+0000","last_unstale":"2024-04-06T12:27:21.065495+0000","last_undegraded":"2024-04-06T12:27:21.065495+0000","last_fullsized":"2024-04-06T12:27:21.065495+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:24.851744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047260+0000","last_change":"2024-04-06T12:27:21.063852+0000","last_active":"2024-04-06T12:27:22.047260+0000","last_peered":"2024-04-06T12:27:22.047260+0000","last_clean":"2024-04-06T12:27:22.047260+0000","last_became_active":"2024-04-06T12:27:21.063593+0000","last_became_peered":"2024-04-06T12:27:21.063593+0000","last_unstale":"2024-04-06T12:27:22.047260+0000","last_undegraded":"2024-04-06T12:27:22.047260+0000","last_fullsized":"2024-04-06T12:27:22.047260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:35:07.924741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.065665+0000","last_change":"2024-04-06T12:27:20.640009+0000","last_active":"2024-04-06T12:27:21.065665+0000","last_peered":"2024-04-06T12:27:21.065665+0000","last_clean":"2024-04-06T12:27:21.065665+0000","last_became_active":"2024-04-06T12:27:20.639751+0000","last_became_peered":"2024-04-06T12:27:20.639751+0000","last_unstale":"2024-04-06T12:27:21.065665+0000","last_undegraded":"2024-04-06T12:27:21.065665+0000","last_fullsized":"2024-04-06T12:27:21.065665+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:23:10.897604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.040256+0000","last_change":"2024-04-06T12:27:21.050994+0000","last_active":"2024-04-06T12:27:22.040256+0000","last_peered":"2024-04-06T12:27:22.040256+0000","last_clean":"2024-04-06T12:27:22.040256+0000","last_became_active":"2024-04-06T12:27:21.049716+0000","last_became_peered":"2024-04-06T12:27:21.049716+0000","last_unstale":"2024-04-06T12:27:22.040256+0000","last_undegraded":"2024-04-06T12:27:22.040256+0000","last_fullsized":"2024-04-06T12:27:22.040256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:53:03.356300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.059811+0000","last_change":"2024-04-06T12:27:20.644294+0000","last_active":"2024-04-06T12:27:21.059811+0000","last_peered":"2024-04-06T12:27:21.059811+0000","last_clean":"2024-04-06T12:27:21.059811+0000","last_became_active":"2024-04-06T12:27:20.643994+0000","last_became_peered":"2024-04-06T12:27:20.643994+0000","last_unstale":"2024-04-06T12:27:21.059811+0000","last_undegraded":"2024-04-06T12:27:21.059811+0000","last_fullsized":"2024-04-06T12:27:21.059811+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:44:45.550106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.060664+0000","last_change":"2024-04-06T12:27:20.636807+0000","last_active":"2024-04-06T12:27:21.060664+0000","last_peered":"2024-04-06T12:27:21.060664+0000","last_clean":"2024-04-06T12:27:21.060664+0000","last_became_active":"2024-04-06T12:27:20.636516+0000","last_became_peered":"2024-04-06T12:27:20.636516+0000","last_unstale":"2024-04-06T12:27:21.060664+0000","last_undegraded":"2024-04-06T12:27:21.060664+0000","last_fullsized":"2024-04-06T12:27:21.060664+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:00:13.506742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.045921+0000","last_change":"2024-04-06T12:27:21.047046+0000","last_active":"2024-04-06T12:27:22.045921+0000","last_peered":"2024-04-06T12:27:22.045921+0000","last_clean":"2024-04-06T12:27:22.045921+0000","last_became_active":"2024-04-06T12:27:21.046798+0000","last_became_peered":"2024-04-06T12:27:21.046798+0000","last_unstale":"2024-04-06T12:27:22.045921+0000","last_undegraded":"2024-04-06T12:27:22.045921+0000","last_fullsized":"2024-04-06T12:27:22.045921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:23:44.000355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.048716+0000","last_change":"2024-04-06T12:27:20.642417+0000","last_active":"2024-04-06T12:27:21.048716+0000","last_peered":"2024-04-06T12:27:21.048716+0000","last_clean":"2024-04-06T12:27:21.048716+0000","last_became_active":"2024-04-06T12:27:20.642057+0000","last_became_peered":"2024-04-06T12:27:20.642057+0000","last_unstale":"2024-04-06T12:27:21.048716+0000","last_undegraded":"2024-04-06T12:27:21.048716+0000","last_fullsized":"2024-04-06T12:27:21.048716+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:40:56.649109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046707+0000","last_change":"2024-04-06T12:27:21.056061+0000","last_active":"2024-04-06T12:27:22.046707+0000","last_peered":"2024-04-06T12:27:22.046707+0000","last_clean":"2024-04-06T12:27:22.046707+0000","last_became_active":"2024-04-06T12:27:21.055761+0000","last_became_peered":"2024-04-06T12:27:21.055761+0000","last_unstale":"2024-04-06T12:27:22.046707+0000","last_undegraded":"2024-04-06T12:27:22.046707+0000","last_fullsized":"2024-04-06T12:27:22.046707+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:20:15.444097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.046311+0000","last_change":"2024-04-06T12:27:20.642798+0000","last_active":"2024-04-06T12:27:21.046311+0000","last_peered":"2024-04-06T12:27:21.046311+0000","last_clean":"2024-04-06T12:27:21.046311+0000","last_became_active":"2024-04-06T12:27:20.642006+0000","last_became_peered":"2024-04-06T12:27:20.642006+0000","last_unstale":"2024-04-06T12:27:21.046311+0000","last_undegraded":"2024-04-06T12:27:21.046311+0000","last_fullsized":"2024-04-06T12:27:21.046311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:14:39.407591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047517+0000","last_change":"2024-04-06T12:27:21.067188+0000","last_active":"2024-04-06T12:27:22.047517+0000","last_peered":"2024-04-06T12:27:22.047517+0000","last_clean":"2024-04-06T12:27:22.047517+0000","last_became_active":"2024-04-06T12:27:21.066981+0000","last_became_peered":"2024-04-06T12:27:21.066981+0000","last_unstale":"2024-04-06T12:27:22.047517+0000","last_undegraded":"2024-04-06T12:27:22.047517+0000","last_fullsized":"2024-04-06T12:27:22.047517+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:16:15.014557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035060+0000","last_change":"2024-04-06T12:27:20.642838+0000","last_active":"2024-04-06T12:27:21.035060+0000","last_peered":"2024-04-06T12:27:21.035060+0000","last_clean":"2024-04-06T12:27:21.035060+0000","last_became_active":"2024-04-06T12:27:20.642344+0000","last_became_peered":"2024-04-06T12:27:20.642344+0000","last_unstale":"2024-04-06T12:27:21.035060+0000","last_undegraded":"2024-04-06T12:27:21.035060+0000","last_fullsized":"2024-04-06T12:27:21.035060+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:39:48.790443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039044+0000","last_change":"2024-04-06T12:27:21.042767+0000","last_active":"2024-04-06T12:27:22.039044+0000","last_peered":"2024-04-06T12:27:22.039044+0000","last_clean":"2024-04-06T12:27:22.039044+0000","last_became_active":"2024-04-06T12:27:21.042541+0000","last_became_peered":"2024-04-06T12:27:21.042541+0000","last_unstale":"2024-04-06T12:27:22.039044+0000","last_undegraded":"2024-04-06T12:27:22.039044+0000","last_fullsized":"2024-04-06T12:27:22.039044+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:02:23.142749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.049488+0000","last_change":"2024-04-06T12:27:20.640874+0000","last_active":"2024-04-06T12:27:21.049488+0000","last_peered":"2024-04-06T12:27:21.049488+0000","last_clean":"2024-04-06T12:27:21.049488+0000","last_became_active":"2024-04-06T12:27:20.640592+0000","last_became_peered":"2024-04-06T12:27:20.640592+0000","last_unstale":"2024-04-06T12:27:21.049488+0000","last_undegraded":"2024-04-06T12:27:21.049488+0000","last_fullsized":"2024-04-06T12:27:21.049488+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:53:08.693479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046322+0000","last_change":"2024-04-06T12:27:21.050535+0000","last_active":"2024-04-06T12:27:22.046322+0000","last_peered":"2024-04-06T12:27:22.046322+0000","last_clean":"2024-04-06T12:27:22.046322+0000","last_became_active":"2024-04-06T12:27:21.050270+0000","last_became_peered":"2024-04-06T12:27:21.050270+0000","last_unstale":"2024-04-06T12:27:22.046322+0000","last_undegraded":"2024-04-06T12:27:22.046322+0000","last_fullsized":"2024-04-06T12:27:22.046322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:24:04.575202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623334+0000","last_change":"2024-04-06T12:27:19.623334+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623334+0000","last_undegraded":"2024-04-06T12:27:19.623334+0000","last_fullsized":"2024-04-06T12:27:19.623334+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:24.851744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039214+0000","last_change":"2024-04-06T12:27:21.042722+0000","last_active":"2024-04-06T12:27:22.039214+0000","last_peered":"2024-04-06T12:27:22.039214+0000","last_clean":"2024-04-06T12:27:22.039214+0000","last_became_active":"2024-04-06T12:27:21.042460+0000","last_became_peered":"2024-04-06T12:27:21.042460+0000","last_unstale":"2024-04-06T12:27:22.039214+0000","last_undegraded":"2024-04-06T12:27:22.039214+0000","last_fullsized":"2024-04-06T12:27:22.039214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:16:01.192728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.037118+0000","last_change":"2024-04-06T12:27:20.643086+0000","last_active":"2024-04-06T12:27:21.037118+0000","last_peered":"2024-04-06T12:27:21.037118+0000","last_clean":"2024-04-06T12:27:21.037118+0000","last_became_active":"2024-04-06T12:27:20.642767+0000","last_became_peered":"2024-04-06T12:27:20.642767+0000","last_unstale":"2024-04-06T12:27:21.037118+0000","last_undegraded":"2024-04-06T12:27:21.037118+0000","last_fullsized":"2024-04-06T12:27:21.037118+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:54:55.715540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047715+0000","last_change":"2024-04-06T12:27:21.058482+0000","last_active":"2024-04-06T12:27:22.047715+0000","last_peered":"2024-04-06T12:27:22.047715+0000","last_clean":"2024-04-06T12:27:22.047715+0000","last_became_active":"2024-04-06T12:27:21.058235+0000","last_became_peered":"2024-04-06T12:27:21.058235+0000","last_unstale":"2024-04-06T12:27:22.047715+0000","last_undegraded":"2024-04-06T12:27:22.047715+0000","last_fullsized":"2024-04-06T12:27:22.047715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:16.550344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.624131+0000","last_change":"2024-04-06T12:27:19.624131+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.624131+0000","last_undegraded":"2024-04-06T12:27:19.624131+0000","last_fullsized":"2024-04-06T12:27:19.624131+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:15.542425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047714+0000","last_change":"2024-04-06T12:27:21.050955+0000","last_active":"2024-04-06T12:27:22.047714+0000","last_peered":"2024-04-06T12:27:22.047714+0000","last_clean":"2024-04-06T12:27:22.047714+0000","last_became_active":"2024-04-06T12:27:21.050675+0000","last_became_peered":"2024-04-06T12:27:21.050675+0000","last_unstale":"2024-04-06T12:27:22.047714+0000","last_undegraded":"2024-04-06T12:27:22.047714+0000","last_fullsized":"2024-04-06T12:27:22.047714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:57.645194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"83'6","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.067970+0000","last_change":"2024-04-06T12:27:20.638258+0000","last_active":"2024-04-06T12:27:21.067970+0000","last_peered":"2024-04-06T12:27:21.067970+0000","last_clean":"2024-04-06T12:27:21.067970+0000","last_became_active":"2024-04-06T12:27:20.638017+0000","last_became_peered":"2024-04-06T12:27:20.638017+0000","last_unstale":"2024-04-06T12:27:21.067970+0000","last_undegraded":"2024-04-06T12:27:21.067970+0000","last_fullsized":"2024-04-06T12:27:21.067970+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:48:54.260905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047268+0000","last_change":"2024-04-06T12:27:21.048435+0000","last_active":"2024-04-06T12:27:22.047268+0000","last_peered":"2024-04-06T12:27:22.047268+0000","last_clean":"2024-04-06T12:27:22.047268+0000","last_became_active":"2024-04-06T12:27:21.048159+0000","last_became_peered":"2024-04-06T12:27:21.048159+0000","last_unstale":"2024-04-06T12:27:22.047268+0000","last_undegraded":"2024-04-06T12:27:22.047268+0000","last_fullsized":"2024-04-06T12:27:22.047268+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:59:57.937757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623703+0000","last_change":"2024-04-06T12:27:19.623703+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623703+0000","last_undegraded":"2024-04-06T12:27:19.623703+0000","last_fullsized":"2024-04-06T12:27:19.623703+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:48:54.260905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046846+0000","last_change":"2024-04-06T12:27:21.050116+0000","last_active":"2024-04-06T12:27:22.046846+0000","last_peered":"2024-04-06T12:27:22.046846+0000","last_clean":"2024-04-06T12:27:22.046846+0000","last_became_active":"2024-04-06T12:27:21.049828+0000","last_became_peered":"2024-04-06T12:27:21.049828+0000","last_unstale":"2024-04-06T12:27:22.046846+0000","last_undegraded":"2024-04-06T12:27:22.046846+0000","last_fullsized":"2024-04-06T12:27:22.046846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:41:35.413608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.043551+0000","last_change":"2024-04-06T12:27:20.643715+0000","last_active":"2024-04-06T12:27:21.043551+0000","last_peered":"2024-04-06T12:27:21.043551+0000","last_clean":"2024-04-06T12:27:21.043551+0000","last_became_active":"2024-04-06T12:27:20.643243+0000","last_became_peered":"2024-04-06T12:27:20.643243+0000","last_unstale":"2024-04-06T12:27:21.043551+0000","last_undegraded":"2024-04-06T12:27:21.043551+0000","last_fullsized":"2024-04-06T12:27:21.043551+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:48.861736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047578+0000","last_change":"2024-04-06T12:27:21.061425+0000","last_active":"2024-04-06T12:27:22.047578+0000","last_peered":"2024-04-06T12:27:22.047578+0000","last_clean":"2024-04-06T12:27:22.047578+0000","last_became_active":"2024-04-06T12:27:21.061137+0000","last_became_peered":"2024-04-06T12:27:21.061137+0000","last_unstale":"2024-04-06T12:27:22.047578+0000","last_undegraded":"2024-04-06T12:27:22.047578+0000","last_fullsized":"2024-04-06T12:27:22.047578+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:29:33.494735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.622973+0000","last_change":"2024-04-06T12:27:19.622973+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.622973+0000","last_undegraded":"2024-04-06T12:27:19.622973+0000","last_fullsized":"2024-04-06T12:27:19.622973+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:55.070413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046208+0000","last_change":"2024-04-06T12:27:21.051804+0000","last_active":"2024-04-06T12:27:22.046208+0000","last_peered":"2024-04-06T12:27:22.046208+0000","last_clean":"2024-04-06T12:27:22.046208+0000","last_became_active":"2024-04-06T12:27:21.051536+0000","last_became_peered":"2024-04-06T12:27:21.051536+0000","last_unstale":"2024-04-06T12:27:22.046208+0000","last_undegraded":"2024-04-06T12:27:22.046208+0000","last_fullsized":"2024-04-06T12:27:22.046208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:26:12.757657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"83'8","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.063908+0000","last_change":"2024-04-06T12:27:20.640204+0000","last_active":"2024-04-06T12:27:21.063908+0000","last_peered":"2024-04-06T12:27:21.063908+0000","last_clean":"2024-04-06T12:27:21.063908+0000","last_became_active":"2024-04-06T12:27:20.639913+0000","last_became_peered":"2024-04-06T12:27:20.639913+0000","last_unstale":"2024-04-06T12:27:21.063908+0000","last_undegraded":"2024-04-06T12:27:21.063908+0000","last_fullsized":"2024-04-06T12:27:21.063908+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:01:44.323843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.059785+0000","last_change":"2024-04-06T12:27:20.643933+0000","last_active":"2024-04-06T12:27:21.059785+0000","last_peered":"2024-04-06T12:27:21.059785+0000","last_clean":"2024-04-06T12:27:21.059785+0000","last_became_active":"2024-04-06T12:27:20.643644+0000","last_became_peered":"2024-04-06T12:27:20.643644+0000","last_unstale":"2024-04-06T12:27:21.059785+0000","last_undegraded":"2024-04-06T12:27:21.059785+0000","last_fullsized":"2024-04-06T12:27:21.059785+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:55.070413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047888+0000","last_change":"2024-04-06T12:27:21.059329+0000","last_active":"2024-04-06T12:27:22.047888+0000","last_peered":"2024-04-06T12:27:22.047888+0000","last_clean":"2024-04-06T12:27:22.047888+0000","last_became_active":"2024-04-06T12:27:21.058948+0000","last_became_peered":"2024-04-06T12:27:21.058948+0000","last_unstale":"2024-04-06T12:27:22.047888+0000","last_undegraded":"2024-04-06T12:27:22.047888+0000","last_fullsized":"2024-04-06T12:27:22.047888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:56.078333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.622989+0000","last_change":"2024-04-06T12:27:19.622989+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.622989+0000","last_undegraded":"2024-04-06T12:27:19.622989+0000","last_fullsized":"2024-04-06T12:27:19.622989+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:29:32.486815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047663+0000","last_change":"2024-04-06T12:27:21.059198+0000","last_active":"2024-04-06T12:27:22.047663+0000","last_peered":"2024-04-06T12:27:22.047663+0000","last_clean":"2024-04-06T12:27:22.047663+0000","last_became_active":"2024-04-06T12:27:21.058950+0000","last_became_peered":"2024-04-06T12:27:21.058950+0000","last_unstale":"2024-04-06T12:27:22.047663+0000","last_undegraded":"2024-04-06T12:27:22.047663+0000","last_fullsized":"2024-04-06T12:27:22.047663+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:20:12.514805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623711+0000","last_change":"2024-04-06T12:27:19.623711+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623711+0000","last_undegraded":"2024-04-06T12:27:19.623711+0000","last_fullsized":"2024-04-06T12:27:19.623711+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:44.931614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047541+0000","last_change":"2024-04-06T12:27:21.061929+0000","last_active":"2024-04-06T12:27:22.047541+0000","last_peered":"2024-04-06T12:27:22.047541+0000","last_clean":"2024-04-06T12:27:22.047541+0000","last_became_active":"2024-04-06T12:27:21.061627+0000","last_became_peered":"2024-04-06T12:27:21.061627+0000","last_unstale":"2024-04-06T12:27:22.047541+0000","last_undegraded":"2024-04-06T12:27:22.047541+0000","last_fullsized":"2024-04-06T12:27:22.047541+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:02.552663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035432+0000","last_change":"2024-04-06T12:27:20.642400+0000","last_active":"2024-04-06T12:27:21.035432+0000","last_peered":"2024-04-06T12:27:21.035432+0000","last_clean":"2024-04-06T12:27:21.035432+0000","last_became_active":"2024-04-06T12:27:20.642039+0000","last_became_peered":"2024-04-06T12:27:20.642039+0000","last_unstale":"2024-04-06T12:27:21.035432+0000","last_undegraded":"2024-04-06T12:27:21.035432+0000","last_fullsized":"2024-04-06T12:27:21.035432+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:44.642507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039877+0000","last_change":"2024-04-06T12:27:21.046054+0000","last_active":"2024-04-06T12:27:22.039877+0000","last_peered":"2024-04-06T12:27:22.039877+0000","last_clean":"2024-04-06T12:27:22.039877+0000","last_became_active":"2024-04-06T12:27:21.045799+0000","last_became_peered":"2024-04-06T12:27:21.045799+0000","last_unstale":"2024-04-06T12:27:22.039877+0000","last_undegraded":"2024-04-06T12:27:22.039877+0000","last_fullsized":"2024-04-06T12:27:22.039877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:29:16.068086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.059940+0000","last_change":"2024-04-06T12:27:20.635120+0000","last_active":"2024-04-06T12:27:21.059940+0000","last_peered":"2024-04-06T12:27:21.059940+0000","last_clean":"2024-04-06T12:27:21.059940+0000","last_became_active":"2024-04-06T12:27:20.634852+0000","last_became_peered":"2024-04-06T12:27:20.634852+0000","last_unstale":"2024-04-06T12:27:21.059940+0000","last_undegraded":"2024-04-06T12:27:21.059940+0000","last_fullsized":"2024-04-06T12:27:21.059940+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:34:57.334069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046898+0000","last_change":"2024-04-06T12:27:21.044875+0000","last_active":"2024-04-06T12:27:22.046898+0000","last_peered":"2024-04-06T12:27:22.046898+0000","last_clean":"2024-04-06T12:27:22.046898+0000","last_became_active":"2024-04-06T12:27:21.044609+0000","last_became_peered":"2024-04-06T12:27:21.044609+0000","last_unstale":"2024-04-06T12:27:22.046898+0000","last_undegraded":"2024-04-06T12:27:22.046898+0000","last_fullsized":"2024-04-06T12:27:22.046898+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:48:55.268825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.043598+0000","last_change":"2024-04-06T12:27:20.643509+0000","last_active":"2024-04-06T12:27:21.043598+0000","last_peered":"2024-04-06T12:27:21.043598+0000","last_clean":"2024-04-06T12:27:21.043598+0000","last_became_active":"2024-04-06T12:27:20.642877+0000","last_became_peered":"2024-04-06T12:27:20.642877+0000","last_unstale":"2024-04-06T12:27:21.043598+0000","last_undegraded":"2024-04-06T12:27:21.043598+0000","last_fullsized":"2024-04-06T12:27:21.043598+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:12:17.891288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048159+0000","last_change":"2024-04-06T12:27:21.063218+0000","last_active":"2024-04-06T12:27:22.048159+0000","last_peered":"2024-04-06T12:27:22.048159+0000","last_clean":"2024-04-06T12:27:22.048159+0000","last_became_active":"2024-04-06T12:27:21.062952+0000","last_became_peered":"2024-04-06T12:27:21.062952+0000","last_unstale":"2024-04-06T12:27:22.048159+0000","last_undegraded":"2024-04-06T12:27:22.048159+0000","last_fullsized":"2024-04-06T12:27:22.048159+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:54:56.723460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.038449+0000","last_change":"2024-04-06T12:27:20.640135+0000","last_active":"2024-04-06T12:27:21.038449+0000","last_peered":"2024-04-06T12:27:21.038449+0000","last_clean":"2024-04-06T12:27:21.038449+0000","last_became_active":"2024-04-06T12:27:20.639873+0000","last_became_peered":"2024-04-06T12:27:20.639873+0000","last_unstale":"2024-04-06T12:27:21.038449+0000","last_undegraded":"2024-04-06T12:27:21.038449+0000","last_fullsized":"2024-04-06T12:27:21.038449+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:26:31.570422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.040070+0000","last_change":"2024-04-06T12:27:21.053276+0000","last_active":"2024-04-06T12:27:22.040070+0000","last_peered":"2024-04-06T12:27:22.040070+0000","last_clean":"2024-04-06T12:27:22.040070+0000","last_became_active":"2024-04-06T12:27:21.053033+0000","last_became_peered":"2024-04-06T12:27:21.053033+0000","last_unstale":"2024-04-06T12:27:22.040070+0000","last_undegraded":"2024-04-06T12:27:22.040070+0000","last_fullsized":"2024-04-06T12:27:22.040070+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:49:47.315178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.061029+0000","last_change":"2024-04-06T12:27:20.642377+0000","last_active":"2024-04-06T12:27:21.061029+0000","last_peered":"2024-04-06T12:27:21.061029+0000","last_clean":"2024-04-06T12:27:21.061029+0000","last_became_active":"2024-04-06T12:27:20.642111+0000","last_became_peered":"2024-04-06T12:27:20.642111+0000","last_unstale":"2024-04-06T12:27:21.061029+0000","last_undegraded":"2024-04-06T12:27:21.061029+0000","last_fullsized":"2024-04-06T12:27:21.061029+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:46:21.677982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046699+0000","last_change":"2024-04-06T12:27:21.047728+0000","last_active":"2024-04-06T12:27:22.046699+0000","last_peered":"2024-04-06T12:27:22.046699+0000","last_clean":"2024-04-06T12:27:22.046699+0000","last_became_active":"2024-04-06T12:27:21.046317+0000","last_became_peered":"2024-04-06T12:27:21.046317+0000","last_unstale":"2024-04-06T12:27:22.046699+0000","last_undegraded":"2024-04-06T12:27:22.046699+0000","last_fullsized":"2024-04-06T12:27:22.046699+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:08:29.040668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.702984+0000","last_change":"2024-04-06T12:27:20.643133+0000","last_active":"2024-04-06T12:27:21.702984+0000","last_peered":"2024-04-06T12:27:21.702984+0000","last_clean":"2024-04-06T12:27:21.702984+0000","last_became_active":"2024-04-06T12:27:20.642858+0000","last_became_peered":"2024-04-06T12:27:20.642858+0000","last_unstale":"2024-04-06T12:27:21.702984+0000","last_undegraded":"2024-04-06T12:27:21.702984+0000","last_fullsized":"2024-04-06T12:27:21.702984+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:16:14.006637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048294+0000","last_change":"2024-04-06T12:27:21.059468+0000","last_active":"2024-04-06T12:27:22.048294+0000","last_peered":"2024-04-06T12:27:22.048294+0000","last_clean":"2024-04-06T12:27:22.048294+0000","last_became_active":"2024-04-06T12:27:21.059222+0000","last_became_peered":"2024-04-06T12:27:21.059222+0000","last_unstale":"2024-04-06T12:27:22.048294+0000","last_undegraded":"2024-04-06T12:27:22.048294+0000","last_fullsized":"2024-04-06T12:27:22.048294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:20:15.444097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.040662+0000","last_change":"2024-04-06T12:27:20.638922+0000","last_active":"2024-04-06T12:27:21.040662+0000","last_peered":"2024-04-06T12:27:21.040662+0000","last_clean":"2024-04-06T12:27:21.040662+0000","last_became_active":"2024-04-06T12:27:20.638666+0000","last_became_peered":"2024-04-06T12:27:20.638666+0000","last_unstale":"2024-04-06T12:27:21.040662+0000","last_undegraded":"2024-04-06T12:27:21.040662+0000","last_fullsized":"2024-04-06T12:27:21.040662+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:08:18.449996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047650+0000","last_change":"2024-04-06T12:27:21.060870+0000","last_active":"2024-04-06T12:27:22.047650+0000","last_peered":"2024-04-06T12:27:22.047650+0000","last_clean":"2024-04-06T12:27:22.047650+0000","last_became_active":"2024-04-06T12:27:21.060603+0000","last_became_peered":"2024-04-06T12:27:21.060603+0000","last_unstale":"2024-04-06T12:27:22.047650+0000","last_undegraded":"2024-04-06T12:27:22.047650+0000","last_fullsized":"2024-04-06T12:27:22.047650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:48:55.268825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.044529+0000","last_change":"2024-04-06T12:27:20.639042+0000","last_active":"2024-04-06T12:27:21.044529+0000","last_peered":"2024-04-06T12:27:21.044529+0000","last_clean":"2024-04-06T12:27:21.044529+0000","last_became_active":"2024-04-06T12:27:20.638475+0000","last_became_peered":"2024-04-06T12:27:20.638475+0000","last_unstale":"2024-04-06T12:27:21.044529+0000","last_undegraded":"2024-04-06T12:27:21.044529+0000","last_fullsized":"2024-04-06T12:27:21.044529+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:15.542425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047260+0000","last_change":"2024-04-06T12:27:21.060781+0000","last_active":"2024-04-06T12:27:22.047260+0000","last_peered":"2024-04-06T12:27:22.047260+0000","last_clean":"2024-04-06T12:27:22.047260+0000","last_became_active":"2024-04-06T12:27:21.060520+0000","last_became_peered":"2024-04-06T12:27:21.060520+0000","last_unstale":"2024-04-06T12:27:22.047260+0000","last_undegraded":"2024-04-06T12:27:22.047260+0000","last_fullsized":"2024-04-06T12:27:22.047260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:46:22.685902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623238+0000","last_change":"2024-04-06T12:27:19.623238+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623238+0000","last_undegraded":"2024-04-06T12:27:19.623238+0000","last_fullsized":"2024-04-06T12:27:19.623238+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:48.861736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047005+0000","last_change":"2024-04-06T12:27:21.055075+0000","last_active":"2024-04-06T12:27:22.047005+0000","last_peered":"2024-04-06T12:27:22.047005+0000","last_clean":"2024-04-06T12:27:22.047005+0000","last_became_active":"2024-04-06T12:27:21.054806+0000","last_became_peered":"2024-04-06T12:27:21.054806+0000","last_unstale":"2024-04-06T12:27:22.047005+0000","last_undegraded":"2024-04-06T12:27:22.047005+0000","last_fullsized":"2024-04-06T12:27:22.047005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:14.746095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035500+0000","last_change":"2024-04-06T12:27:20.641433+0000","last_active":"2024-04-06T12:27:21.035500+0000","last_peered":"2024-04-06T12:27:21.035500+0000","last_clean":"2024-04-06T12:27:21.035500+0000","last_became_active":"2024-04-06T12:27:20.641181+0000","last_became_peered":"2024-04-06T12:27:20.641181+0000","last_unstale":"2024-04-06T12:27:21.035500+0000","last_undegraded":"2024-04-06T12:27:21.035500+0000","last_fullsized":"2024-04-06T12:27:21.035500+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:20:08.218629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046219+0000","last_change":"2024-04-06T12:27:21.055928+0000","last_active":"2024-04-06T12:27:22.046219+0000","last_peered":"2024-04-06T12:27:22.046219+0000","last_clean":"2024-04-06T12:27:22.046219+0000","last_became_active":"2024-04-06T12:27:21.055641+0000","last_became_peered":"2024-04-06T12:27:21.055641+0000","last_unstale":"2024-04-06T12:27:22.046219+0000","last_undegraded":"2024-04-06T12:27:22.046219+0000","last_fullsized":"2024-04-06T12:27:22.046219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:08:32.179469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035357+0000","last_change":"2024-04-06T12:27:20.643620+0000","last_active":"2024-04-06T12:27:21.035357+0000","last_peered":"2024-04-06T12:27:21.035357+0000","last_clean":"2024-04-06T12:27:21.035357+0000","last_became_active":"2024-04-06T12:27:20.642671+0000","last_became_peered":"2024-04-06T12:27:20.642671+0000","last_unstale":"2024-04-06T12:27:21.035357+0000","last_undegraded":"2024-04-06T12:27:21.035357+0000","last_fullsized":"2024-04-06T12:27:21.035357+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:56:37.738753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046202+0000","last_change":"2024-04-06T12:27:21.046951+0000","last_active":"2024-04-06T12:27:22.046202+0000","last_peered":"2024-04-06T12:27:22.046202+0000","last_clean":"2024-04-06T12:27:22.046202+0000","last_became_active":"2024-04-06T12:27:21.046651+0000","last_became_peered":"2024-04-06T12:27:21.046651+0000","last_unstale":"2024-04-06T12:27:22.046202+0000","last_undegraded":"2024-04-06T12:27:22.046202+0000","last_fullsized":"2024-04-06T12:27:22.046202+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:59:55.516160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.044036+0000","last_change":"2024-04-06T12:27:20.635487+0000","last_active":"2024-04-06T12:27:21.044036+0000","last_peered":"2024-04-06T12:27:21.044036+0000","last_clean":"2024-04-06T12:27:21.044036+0000","last_became_active":"2024-04-06T12:27:20.635194+0000","last_became_peered":"2024-04-06T12:27:20.635194+0000","last_unstale":"2024-04-06T12:27:21.044036+0000","last_undegraded":"2024-04-06T12:27:21.044036+0000","last_fullsized":"2024-04-06T12:27:21.044036+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:01.544743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047357+0000","last_change":"2024-04-06T12:27:21.062962+0000","last_active":"2024-04-06T12:27:22.047357+0000","last_peered":"2024-04-06T12:27:22.047357+0000","last_clean":"2024-04-06T12:27:22.047357+0000","last_became_active":"2024-04-06T12:27:21.062683+0000","last_became_peered":"2024-04-06T12:27:21.062683+0000","last_unstale":"2024-04-06T12:27:22.047357+0000","last_undegraded":"2024-04-06T12:27:22.047357+0000","last_fullsized":"2024-04-06T12:27:22.047357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:49.869655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035100+0000","last_change":"2024-04-06T12:27:20.640825+0000","last_active":"2024-04-06T12:27:21.035100+0000","last_peered":"2024-04-06T12:27:21.035100+0000","last_clean":"2024-04-06T12:27:21.035100+0000","last_became_active":"2024-04-06T12:27:20.640568+0000","last_became_peered":"2024-04-06T12:27:20.640568+0000","last_unstale":"2024-04-06T12:27:21.035100+0000","last_undegraded":"2024-04-06T12:27:21.035100+0000","last_fullsized":"2024-04-06T12:27:21.035100+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:27:16.320148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'161","reported_seq":388,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-06T12:27:17.742656+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:27:17.742656+0000","last_peered":"2024-04-06T12:27:17.742656+0000","last_clean":"2024-04-06T12:27:17.742656+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:27:17.742656+0000","last_undegraded":"2024-04-06T12:27:17.742656+0000","last_fullsized":"2024-04-06T12:27:17.742656+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":110,"acting":110,"num_store_stats":7},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2168,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575034,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58900,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312940,"statfs":{"total":96636764160,"available":96576450560,"internally_reserved":0,"allocated":1564672,"data_stored":1068520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55800000000000005,"15min":0.54900000000000004},"min":{"1min":0.375,"5min":0.219,"15min":0.219},"max":{"1min":0.80100000000000005,"5min":1.143,"15min":1.143},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55700000000000005,"15min":0.55300000000000005},"min":{"1min":0.433,"5min":0.35399999999999998,"15min":0.25900000000000001},"max":{"1min":0.73299999999999998,"5min":0.77100000000000002,"15min":0.78200000000000003},"last":0.48399999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55400000000000005,"15min":0.55100000000000005},"min":{"1min":0.42299999999999999,"5min":0.33600000000000002,"15min":0.31900000000000001},"max":{"1min":0.748,"5min":0.77100000000000002,"15min":0.81399999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.41699999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.77400000000000002,"5min":0.79400000000000004,"15min":0.79600000000000004},"last":0.372}]},{"osd":2,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56999999999999995,"15min":0.55200000000000005},"min":{"1min":0.441,"5min":0.249,"15min":0.249},"max":{"1min":0.77100000000000002,"5min":0.78500000000000003,"15min":0.83299999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57099999999999995,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.24199999999999999},"max":{"1min":0.76600000000000001,"5min":0.78300000000000003,"15min":0.82299999999999995},"last":0.58999999999999997}]},{"osd":3,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58099999999999996,"15min":0.57199999999999995},"min":{"1min":0.45900000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.75800000000000001,"5min":1.357,"15min":1.357},"last":0.502},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59299999999999997,"15min":0.59599999999999997},"min":{"1min":0.44400000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.748,"5min":0.81799999999999995,"15min":0.92000000000000004},"last":0.626}]},{"osd":4,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.45300000000000001,"15min":0.45800000000000002},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.25},"max":{"1min":0.68300000000000005,"5min":0.70399999999999996,"15min":0.83599999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.53700000000000003},"min":{"1min":0.42899999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71399999999999997,"5min":0.77700000000000002,"15min":1.0780000000000001},"last":0.54400000000000004}]},{"osd":5,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.498,"15min":0.51300000000000001},"min":{"1min":0.33600000000000002,"5min":0.29199999999999998,"15min":0.23799999999999999},"max":{"1min":0.73299999999999998,"5min":0.80100000000000005,"15min":0.98199999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55900000000000005,"15min":0.56599999999999995},"min":{"1min":0.374,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.70699999999999996,"5min":0.81599999999999995,"15min":0.93600000000000005},"last":0.60599999999999998}]},{"osd":6,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53900000000000003,"15min":0.54900000000000004},"min":{"1min":0.39300000000000002,"5min":0.33700000000000002,"15min":0.28799999999999998},"max":{"1min":0.70699999999999996,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55800000000000005,"15min":0.56799999999999995},"min":{"1min":0.40200000000000002,"5min":0.30399999999999999,"15min":0.26900000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.88500000000000001},"last":0.58099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575034,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":1019904,"data_stored":503606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.36699999999999999,"5min":0.26200000000000001,"15min":0.23000000000000001},"max":{"1min":0.77500000000000002,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52500000000000002,"15min":0.53300000000000003},"min":{"1min":0.374,"5min":0.30199999999999999,"15min":0.245},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.53300000000000003}]},{"osd":1,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.32700000000000001,"5min":0.32600000000000001,"15min":0.24399999999999999},"max":{"1min":0.78200000000000003,"5min":0.78700000000000003,"15min":0.84999999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.38400000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.83099999999999996},"last":0.50700000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.39100000000000001,"5min":0.30499999999999999,"15min":0.29899999999999999},"max":{"1min":0.749,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.38800000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.56899999999999995,"15min":0.58099999999999996},"min":{"1min":0.35099999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.745,"5min":0.79000000000000004,"15min":0.81200000000000006},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.60899999999999999,"15min":0.61899999999999999},"min":{"1min":0.51700000000000002,"5min":0.41099999999999998,"15min":0.374},"max":{"1min":0.79200000000000004,"5min":0.82799999999999996,"15min":0.872},"last":0.52000000000000002}]},{"osd":4,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.49099999999999999,"15min":0.496},"min":{"1min":0.28599999999999998,"5min":0.27500000000000002,"15min":0.27300000000000002},"max":{"1min":0.747,"5min":0.749,"15min":0.749},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.53000000000000003},"min":{"1min":0.34899999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.73199999999999998,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.43099999999999999}]},{"osd":5,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.47999999999999998,"15min":0.49199999999999999},"min":{"1min":0.34699999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71099999999999997,"5min":0.755,"15min":0.75800000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.50600000000000001,"15min":0.51300000000000001},"min":{"1min":0.32800000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72599999999999998,"5min":0.83699999999999997,"15min":0.92800000000000005},"last":0.45400000000000001}]},{"osd":7,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53000000000000003,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70099999999999996,"5min":0.79800000000000004,"15min":0.81000000000000005},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55800000000000005,"15min":0.56100000000000005},"min":{"1min":0.41899999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80800000000000005,"5min":0.88900000000000001,"15min":0.96099999999999997},"last":0.54800000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575033,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":1044480,"data_stored":505900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.377,"15min":0.36199999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.47699999999999998,"5min":0.58999999999999997,"15min":0.71699999999999997},"last":0.308},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39900000000000002,"15min":0.39600000000000002},"min":{"1min":0.253,"5min":0.215,"15min":0.215},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.32600000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.40500000000000003,"15min":0.39600000000000002},"min":{"1min":0.27200000000000002,"5min":0.23899999999999999,"15min":0.19400000000000001},"max":{"1min":0.57599999999999996,"5min":0.624,"15min":0.67600000000000005},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.432,"15min":0.42999999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.249},"max":{"1min":0.60699999999999998,"5min":0.63600000000000001,"15min":0.79000000000000004},"last":0.28899999999999998}]},{"osd":3,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45600000000000002,"15min":0.44700000000000001},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67400000000000004,"5min":0.67800000000000005,"15min":0.84299999999999997},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.42599999999999999,"15min":0.41899999999999998},"min":{"1min":0.30499999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.56499999999999995,"5min":0.69599999999999995,"15min":0.86299999999999999},"last":0.375}]},{"osd":4,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.623,"15min":0.63200000000000001},"min":{"1min":0.42599999999999999,"5min":0.40100000000000002,"15min":0.34799999999999998},"max":{"1min":0.71499999999999997,"5min":0.86799999999999999,"15min":6.3739999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.63100000000000001,"15min":0.622},"min":{"1min":0.45400000000000001,"5min":0.441,"15min":0.30399999999999999},"max":{"1min":0.84299999999999997,"5min":0.91400000000000003,"15min":1.0269999999999999},"last":0.504}]},{"osd":5,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.624,"15min":0.61399999999999999},"min":{"1min":0.47599999999999998,"5min":0.42999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83999999999999997,"5min":0.91300000000000003,"15min":0.97699999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.628},"min":{"1min":0.442,"5min":0.43099999999999999,"15min":0.35099999999999998},"max":{"1min":0.78700000000000003,"5min":0.999,"15min":1.0089999999999999},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.623,"15min":0.61399999999999999},"min":{"1min":0.47699999999999998,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":0.76500000000000001,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.65300000000000002,"15min":0.64900000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.41799999999999998},"max":{"1min":0.83999999999999997,"5min":0.88700000000000001,"15min":1.151},"last":0.441}]},{"osd":7,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.64800000000000002,"15min":0.65500000000000003},"min":{"1min":0.44600000000000001,"5min":0.41899999999999998,"15min":0.38800000000000001},"max":{"1min":0.81699999999999995,"5min":1.0329999999999999,"15min":6.3639999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.47299999999999998,"5min":0.40200000000000002,"15min":0.39300000000000002},"max":{"1min":0.82499999999999996,"5min":0.95599999999999996,"15min":1.073},"last":0.55400000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575033,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32688,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339152,"statfs":{"total":96636764160,"available":96603291648,"internally_reserved":0,"allocated":1609728,"data_stored":1093940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.38700000000000001,"15min":0.38},"min":{"1min":0.312,"5min":0.19,"15min":0.19},"max":{"1min":0.53900000000000003,"5min":0.80000000000000004,"15min":0.89100000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.40200000000000002},"min":{"1min":0.30599999999999999,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":0.60599999999999998,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.32300000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.40899999999999997,"15min":0.41199999999999998},"min":{"1min":0.29899999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.57799999999999996,"5min":0.60299999999999998,"15min":1.8089999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.434,"15min":0.42899999999999999},"min":{"1min":0.32600000000000001,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":0.64700000000000002,"5min":1.4790000000000001,"15min":1.764},"last":0.35899999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.41899999999999998,"15min":0.41099999999999998},"min":{"1min":0.34100000000000003,"5min":0.252,"15min":0.21299999999999999},"max":{"1min":0.63,"5min":1.25,"15min":1.25},"last":0.309},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.32900000000000001,"5min":0.28499999999999998,"15min":0.249},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.92000000000000004},"last":0.41199999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.627,"15min":0.61799999999999999},"min":{"1min":0.46100000000000002,"5min":0.371,"15min":0.35699999999999998},"max":{"1min":0.77700000000000002,"5min":1.514,"15min":1.514},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.63900000000000001,"15min":0.628},"min":{"1min":0.47599999999999998,"5min":0.37,"15min":0.311},"max":{"1min":0.86799999999999999,"5min":1.202,"15min":1.202},"last":0.60299999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63400000000000001,"15min":0.61899999999999999},"min":{"1min":0.46999999999999997,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.84899999999999998,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64400000000000002,"15min":0.64700000000000002},"min":{"1min":0.45200000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.88600000000000001,"5min":0.94499999999999995,"15min":0.97899999999999998},"last":0.746}]},{"osd":6,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65300000000000002,"15min":0.64800000000000002},"min":{"1min":0.51700000000000002,"5min":0.39100000000000001,"15min":0.378},"max":{"1min":0.84999999999999998,"5min":0.91000000000000003,"15min":0.96699999999999997},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.66900000000000004,"15min":0.66400000000000003},"min":{"1min":0.499,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84999999999999998,"5min":1.613,"15min":1.613},"last":0.58799999999999997}]},{"osd":7,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67000000000000004,"15min":0.66300000000000003},"min":{"1min":0.51500000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90100000000000002,"5min":1.54,"15min":1.54},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.67700000000000005,"15min":0.65700000000000003},"min":{"1min":0.51400000000000001,"5min":0.38400000000000001,"15min":0.36699999999999999},"max":{"1min":0.88300000000000001,"5min":1.498,"15min":1.498},"last":0.57799999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575035,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50156,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321684,"statfs":{"total":96636764160,"available":96585404416,"internally_reserved":0,"allocated":1015808,"data_stored":503572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38600000000000001,"15min":0.38100000000000001},"min":{"1min":0.26700000000000002,"5min":0.214,"15min":0.21099999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":1.159},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.41199999999999998,"15min":0.40899999999999997},"min":{"1min":0.27500000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.54600000000000004,"5min":0.60899999999999999,"15min":1.075},"last":0.29599999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.41099999999999998,"15min":0.41899999999999998},"min":{"1min":0.29599999999999999,"5min":0.26400000000000001,"15min":0.24299999999999999},"max":{"1min":0.57599999999999996,"5min":0.78500000000000003,"15min":3.6179999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.42299999999999999,"15min":0.42999999999999999},"min":{"1min":0.309,"5min":0.27100000000000002,"15min":0.23599999999999999},"max":{"1min":0.58899999999999997,"5min":0.61399999999999999,"15min":1.125},"last":0.32700000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.46000000000000002,"15min":0.46500000000000002},"min":{"1min":0.34399999999999997,"5min":0.30499999999999999,"15min":0.29699999999999999},"max":{"1min":0.63100000000000001,"5min":0.79200000000000004,"15min":3.661},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.44400000000000001,"15min":0.438},"min":{"1min":0.376,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.55900000000000005,"5min":1.0549999999999999,"15min":1.1910000000000001},"last":0.313}]},{"osd":4,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.61499999999999999,"15min":0.60699999999999998},"min":{"1min":0.45900000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81699999999999995,"5min":0.88200000000000001,"15min":1.2170000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63600000000000001,"15min":0.628},"min":{"1min":0.48799999999999999,"5min":0.316,"15min":0.30499999999999999},"max":{"1min":0.89900000000000002,"5min":0.90200000000000002,"15min":1.101},"last":0.47599999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.64800000000000002,"15min":0.65300000000000002},"min":{"1min":0.57599999999999996,"5min":0.42299999999999999,"15min":0.32900000000000001},"max":{"1min":0.90300000000000002,"5min":1.6870000000000001,"15min":3.6920000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.628,"15min":0.625},"min":{"1min":0.49399999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.81200000000000006,"5min":0.89500000000000002,"15min":1.244},"last":0.49199999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64300000000000002,"15min":0.63400000000000001},"min":{"1min":0.56499999999999995,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.84899999999999998,"5min":0.93400000000000005,"15min":1.012},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67100000000000004,"15min":0.66900000000000004},"min":{"1min":0.51300000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93100000000000005,"5min":1.728,"15min":3.7490000000000001},"last":0.48299999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65400000000000003,"15min":0.64900000000000002},"min":{"1min":0.52900000000000003,"5min":0.40699999999999997,"15min":0.35599999999999998},"max":{"1min":0.80500000000000005,"5min":0.86899999999999999,"15min":1.0469999999999999},"last":0.501},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.67700000000000005,"15min":0.68200000000000005},"min":{"1min":0.54300000000000004,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.90900000000000003,"5min":1.758,"15min":3.7200000000000002},"last":0.56299999999999994}]}]},{"osd":3,"up_from":13,"seq":55834575034,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32120,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339720,"statfs":{"total":96636764160,"available":96603873280,"internally_reserved":0,"allocated":1019904,"data_stored":504591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.24199999999999999,"5min":0.21099999999999999,"15min":0.191},"max":{"1min":0.51700000000000002,"5min":0.59699999999999998,"15min":0.63900000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38100000000000001,"15min":0.377},"min":{"1min":0.308,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.21299999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.372,"15min":0.372},"min":{"1min":0.248,"5min":0.22700000000000001,"15min":0.215},"max":{"1min":0.47099999999999997,"5min":0.59999999999999998,"15min":0.62},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40200000000000002,"15min":0.40799999999999997},"min":{"1min":0.34200000000000003,"5min":0.26200000000000001,"15min":0.247},"max":{"1min":0.55200000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.39200000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.44900000000000001,"15min":0.45000000000000001},"min":{"1min":0.32800000000000001,"5min":0.27200000000000002,"15min":0.25700000000000001},"max":{"1min":0.55300000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.432},{"interface":"front","average":{"1min":0.495,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.39700000000000002,"5min":0.317,"15min":0.29899999999999999},"max":{"1min":0.622,"5min":0.64800000000000002,"15min":0.65700000000000003},"last":0.30199999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.49299999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.95999999999999996},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60999999999999999,"15min":0.61699999999999999},"min":{"1min":0.41899999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.74099999999999999,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.67100000000000004}]},{"osd":5,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.64400000000000002,"15min":0.65500000000000003},"min":{"1min":0.46000000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.91100000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.625},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65300000000000002,"15min":0.65200000000000002},"min":{"1min":0.42999999999999999,"5min":0.38200000000000001,"15min":0.36899999999999999},"max":{"1min":0.94499999999999995,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.65600000000000003}]},{"osd":6,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60799999999999998,"15min":0.61799999999999999},"min":{"1min":0.47799999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.96999999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63500000000000001,"15min":0.64800000000000002},"min":{"1min":0.436,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91800000000000004,"5min":0.92600000000000005,"15min":0.94699999999999995},"last":0.70899999999999996}]},{"osd":7,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.629,"15min":0.63700000000000001},"min":{"1min":0.46899999999999997,"5min":0.40400000000000003,"15min":0.33300000000000002},"max":{"1min":0.873,"5min":0.89000000000000001,"15min":0.93300000000000005},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65100000000000002,"15min":0.65700000000000003},"min":{"1min":0.52200000000000002,"5min":0.39500000000000002,"15min":0.373},"max":{"1min":0.93300000000000005,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.54600000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575034,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":1032192,"data_stored":506155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.47499999999999998,"5min":0.29399999999999998,"15min":0.23899999999999999},"max":{"1min":0.71099999999999997,"5min":0.80800000000000005,"15min":0.82999999999999996},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55200000000000005,"15min":0.55400000000000005},"min":{"1min":0.49099999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.82199999999999995},"last":0.58999999999999997}]},{"osd":1,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.59299999999999997,"15min":0.63500000000000001},"min":{"1min":0.48199999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74399999999999999,"5min":0.76900000000000002,"15min":4.3479999999999999},"last":0.629},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58699999999999997,"15min":0.623},"min":{"1min":0.48399999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77100000000000002,"5min":0.78400000000000003,"15min":4.3319999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.56299999999999994,"15min":0.56599999999999995},"min":{"1min":0.47399999999999998,"5min":0.313,"15min":0.311},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.78400000000000003},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.58099999999999996,"15min":0.58199999999999996},"min":{"1min":0.50900000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72399999999999998,"5min":0.74299999999999999,"15min":0.82199999999999995},"last":0.53000000000000003}]},{"osd":3,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55100000000000005,"15min":0.59199999999999997},"min":{"1min":0.432,"5min":0.36399999999999999,"15min":0.34899999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":4.1950000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.59699999999999998,"15min":0.60199999999999998},"min":{"1min":0.497,"5min":0.40500000000000003,"15min":0.36099999999999999},"max":{"1min":0.73699999999999999,"5min":0.75900000000000001,"15min":1.2330000000000001},"last":0.61299999999999999}]},{"osd":5,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.46500000000000002,"15min":0.48999999999999999},"min":{"1min":0.36199999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71099999999999997,"5min":0.76300000000000001,"15min":1.196},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.497,"15min":0.502},"min":{"1min":0.39400000000000002,"5min":0.33500000000000002,"15min":0.312},"max":{"1min":0.63900000000000001,"5min":0.74299999999999999,"15min":0.77000000000000002},"last":0.39800000000000002}]},{"osd":6,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.498,"15min":0.51100000000000001},"min":{"1min":0.33200000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.64200000000000002,"5min":0.754,"15min":1.022},"last":0.379},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.52500000000000002,"15min":0.54400000000000004},"min":{"1min":0.39500000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67400000000000004,"5min":0.73699999999999999,"15min":0.872},"last":0.49299999999999999}]},{"osd":7,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51200000000000001,"15min":0.54100000000000004},"min":{"1min":0.377,"5min":0.311,"15min":0.311},"max":{"1min":0.68500000000000005,"5min":0.79300000000000004,"15min":0.91500000000000004},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55000000000000004,"15min":0.57699999999999996},"min":{"1min":0.39300000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73599999999999999,"5min":0.752,"15min":1.0589999999999999},"last":0.44400000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575034,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46064,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325776,"statfs":{"total":96636764160,"available":96589594624,"internally_reserved":0,"allocated":1019904,"data_stored":503594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.36799999999999999,"5min":0.32700000000000001,"15min":0.24199999999999999},"max":{"1min":0.75900000000000001,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.29499999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.748,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.61199999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.30399999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55200000000000005,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.34499999999999997,"15min":0.30599999999999999},"max":{"1min":0.73599999999999999,"5min":0.74299999999999999,"15min":0.76200000000000001},"last":0.55800000000000005}]},{"osd":2,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56999999999999995,"15min":0.56200000000000006},"min":{"1min":0.33200000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.72199999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.629},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.40999999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72099999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.496}]},{"osd":3,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.33900000000000002,"5min":0.32500000000000001,"15min":0.31},"max":{"1min":0.77800000000000002,"5min":0.874,"15min":0.874},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.436,"5min":0.377,"15min":0.32300000000000001},"max":{"1min":0.73099999999999998,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.58899999999999997}]},{"osd":4,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.28299999999999997},"max":{"1min":0.755,"5min":0.755,"15min":0.81200000000000006},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54000000000000004,"15min":0.53300000000000003},"min":{"1min":0.42599999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73699999999999999,"5min":0.80000000000000004,"15min":0.91800000000000004},"last":0.503}]},{"osd":6,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49099999999999999,"15min":0.48599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.28000000000000003},"max":{"1min":0.63800000000000001,"5min":0.73899999999999999,"15min":0.88700000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.496,"15min":0.49299999999999999},"min":{"1min":0.33100000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64800000000000002,"5min":0.73199999999999998,"15min":0.83299999999999996},"last":0.47399999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.252},"max":{"1min":0.68999999999999995,"5min":0.78600000000000003,"15min":0.79100000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60999999999999999,"15min":0.60199999999999998},"min":{"1min":0.44800000000000001,"5min":0.41399999999999998,"15min":0.35199999999999998},"max":{"1min":0.72499999999999998,"5min":0.85299999999999998,"15min":0.85599999999999998},"last":0.54400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:27:25.926 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3d is not active+clean 2024-04-06T12:27:25.926 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T12:27:20.616323+0000', 'last_change': '2024-04-06T12:27:20.616323+0000', 'last_active': '2024-04-06T12:27:20.616323+0000', 'last_peered': '2024-04-06T12:27:20.616323+0000', 'last_clean': '2024-04-06T12:27:20.616323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T12:27:20.616323+0000', 'last_undegraded': '2024-04-06T12:27:20.616323+0000', 'last_fullsized': '2024-04-06T12:27:20.616323+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'last_clean_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T12:27:25.926 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.37 is not active+clean 2024-04-06T12:27:25.926 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T12:27:20.616323+0000', 'last_change': '2024-04-06T12:27:20.616323+0000', 'last_active': '2024-04-06T12:27:20.616323+0000', 'last_peered': '2024-04-06T12:27:20.616323+0000', 'last_clean': '2024-04-06T12:27:20.616323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T12:27:20.616323+0000', 'last_undegraded': '2024-04-06T12:27:20.616323+0000', 'last_fullsized': '2024-04-06T12:27:20.616323+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'last_clean_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T12:27:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.36 is not active+clean 2024-04-06T12:27:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T12:27:20.616323+0000', 'last_change': '2024-04-06T12:27:20.616323+0000', 'last_active': '2024-04-06T12:27:20.616323+0000', 'last_peered': '2024-04-06T12:27:20.616323+0000', 'last_clean': '2024-04-06T12:27:20.616323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T12:27:20.616323+0000', 'last_undegraded': '2024-04-06T12:27:20.616323+0000', 'last_fullsized': '2024-04-06T12:27:20.616323+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'last_clean_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T12:27:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.32 is not active+clean 2024-04-06T12:27:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T12:27:20.616323+0000', 'last_change': '2024-04-06T12:27:20.616323+0000', 'last_active': '2024-04-06T12:27:20.616323+0000', 'last_peered': '2024-04-06T12:27:20.616323+0000', 'last_clean': '2024-04-06T12:27:20.616323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T12:27:20.616323+0000', 'last_undegraded': '2024-04-06T12:27:20.616323+0000', 'last_fullsized': '2024-04-06T12:27:20.616323+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'last_clean_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T12:27:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.31 is not active+clean 2024-04-06T12:27:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T12:27:20.616323+0000', 'last_change': '2024-04-06T12:27:20.616323+0000', 'last_active': '2024-04-06T12:27:20.616323+0000', 'last_peered': '2024-04-06T12:27:20.616323+0000', 'last_clean': '2024-04-06T12:27:20.616323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T12:27:20.616323+0000', 'last_undegraded': '2024-04-06T12:27:20.616323+0000', 'last_fullsized': '2024-04-06T12:27:20.616323+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'last_clean_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T12:27:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.28 is not active+clean 2024-04-06T12:27:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T12:27:20.616323+0000', 'last_change': '2024-04-06T12:27:20.616323+0000', 'last_active': '2024-04-06T12:27:20.616323+0000', 'last_peered': '2024-04-06T12:27:20.616323+0000', 'last_clean': '2024-04-06T12:27:20.616323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T12:27:20.616323+0000', 'last_undegraded': '2024-04-06T12:27:20.616323+0000', 'last_fullsized': '2024-04-06T12:27:20.616323+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'last_clean_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T12:27:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.23 is not active+clean 2024-04-06T12:27:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T12:27:20.616323+0000', 'last_change': '2024-04-06T12:27:20.616323+0000', 'last_active': '2024-04-06T12:27:20.616323+0000', 'last_peered': '2024-04-06T12:27:20.616323+0000', 'last_clean': '2024-04-06T12:27:20.616323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T12:27:20.616323+0000', 'last_undegraded': '2024-04-06T12:27:20.616323+0000', 'last_fullsized': '2024-04-06T12:27:20.616323+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'last_clean_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T12:27:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1f is not active+clean 2024-04-06T12:27:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T12:27:20.616323+0000', 'last_change': '2024-04-06T12:27:20.616323+0000', 'last_active': '2024-04-06T12:27:20.616323+0000', 'last_peered': '2024-04-06T12:27:20.616323+0000', 'last_clean': '2024-04-06T12:27:20.616323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T12:27:20.616323+0000', 'last_undegraded': '2024-04-06T12:27:20.616323+0000', 'last_fullsized': '2024-04-06T12:27:20.616323+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'last_clean_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T12:27:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.17 is not active+clean 2024-04-06T12:27:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-04-06T12:27:19.623334+0000', 'last_change': '2024-04-06T12:27:19.623334+0000', 'last_active': '2024-04-06T12:27:19.608404+0000', 'last_peered': '2024-04-06T12:27:19.608404+0000', 'last_clean': '2024-04-06T12:27:19.608404+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T12:27:19.623334+0000', 'last_undegraded': '2024-04-06T12:27:19.623334+0000', 'last_fullsized': '2024-04-06T12:27:19.623334+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'last_clean_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:51:24.851744+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-06T12:27:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.15 is not active+clean 2024-04-06T12:27:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-04-06T12:27:19.624131+0000', 'last_change': '2024-04-06T12:27:19.624131+0000', 'last_active': '2024-04-06T12:27:19.608404+0000', 'last_peered': '2024-04-06T12:27:19.608404+0000', 'last_clean': '2024-04-06T12:27:19.608404+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T12:27:19.624131+0000', 'last_undegraded': '2024-04-06T12:27:19.624131+0000', 'last_fullsized': '2024-04-06T12:27:19.624131+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'last_clean_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:52:15.542425+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-06T12:27:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.13 is not active+clean 2024-04-06T12:27:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-04-06T12:27:19.623703+0000', 'last_change': '2024-04-06T12:27:19.623703+0000', 'last_active': '2024-04-06T12:27:19.608404+0000', 'last_peered': '2024-04-06T12:27:19.608404+0000', 'last_clean': '2024-04-06T12:27:19.608404+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T12:27:19.623703+0000', 'last_undegraded': '2024-04-06T12:27:19.623703+0000', 'last_fullsized': '2024-04-06T12:27:19.623703+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'last_clean_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:48:54.260905+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-06T12:27:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.11 is not active+clean 2024-04-06T12:27:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-04-06T12:27:19.622973+0000', 'last_change': '2024-04-06T12:27:19.622973+0000', 'last_active': '2024-04-06T12:27:19.608404+0000', 'last_peered': '2024-04-06T12:27:19.608404+0000', 'last_clean': '2024-04-06T12:27:19.608404+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T12:27:19.622973+0000', 'last_undegraded': '2024-04-06T12:27:19.622973+0000', 'last_fullsized': '2024-04-06T12:27:19.622973+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'last_clean_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:45:55.070413+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-06T12:27:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.c is not active+clean 2024-04-06T12:27:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T12:27:20.616323+0000', 'last_change': '2024-04-06T12:27:20.616323+0000', 'last_active': '2024-04-06T12:27:20.616323+0000', 'last_peered': '2024-04-06T12:27:20.616323+0000', 'last_clean': '2024-04-06T12:27:20.616323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T12:27:20.616323+0000', 'last_undegraded': '2024-04-06T12:27:20.616323+0000', 'last_fullsized': '2024-04-06T12:27:20.616323+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'last_clean_scrub_stamp': '2024-04-06T12:27:20.616323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T12:27:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.e is not active+clean 2024-04-06T12:27:25.929 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-04-06T12:27:19.622989+0000', 'last_change': '2024-04-06T12:27:19.622989+0000', 'last_active': '2024-04-06T12:27:19.608404+0000', 'last_peered': '2024-04-06T12:27:19.608404+0000', 'last_clean': '2024-04-06T12:27:19.608404+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T12:27:19.622989+0000', 'last_undegraded': '2024-04-06T12:27:19.622989+0000', 'last_fullsized': '2024-04-06T12:27:19.622989+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'last_clean_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T16:29:32.486815+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-06T12:27:25.929 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.d is not active+clean 2024-04-06T12:27:25.929 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-04-06T12:27:19.623711+0000', 'last_change': '2024-04-06T12:27:19.623711+0000', 'last_active': '2024-04-06T12:27:19.608404+0000', 'last_peered': '2024-04-06T12:27:19.608404+0000', 'last_clean': '2024-04-06T12:27:19.608404+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T12:27:19.623711+0000', 'last_undegraded': '2024-04-06T12:27:19.623711+0000', 'last_fullsized': '2024-04-06T12:27:19.623711+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'last_clean_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T16:33:44.931614+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-06T12:27:25.929 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3 is not active+clean 2024-04-06T12:27:25.929 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-04-06T12:27:19.623238+0000', 'last_change': '2024-04-06T12:27:19.623238+0000', 'last_active': '2024-04-06T12:27:19.608404+0000', 'last_peered': '2024-04-06T12:27:19.608404+0000', 'last_clean': '2024-04-06T12:27:19.608404+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T12:27:19.623238+0000', 'last_undegraded': '2024-04-06T12:27:19.623238+0000', 'last_fullsized': '2024-04-06T12:27:19.623238+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'last_clean_scrub_stamp': '2024-04-06T12:27:19.608404+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T16:42:48.861736+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-06T12:27:25.929 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-06T12:27:26.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.086+0000 7f94c3613700 1 Processor -- start 2024-04-06T12:27:26.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.087+0000 7f94c3613700 1 -- start start 2024-04-06T12:27:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.087+0000 7f94c3613700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f94bc15d550 0x7f94bc0b9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.087+0000 7f94c3613700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f94bc0d1810 0x7f94bc0cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.087+0000 7f94c3613700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94bc0d0800 con 0x7f94bc0d03b0 2024-04-06T12:27:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.087+0000 7f94c3613700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94bc0d2c00 con 0x7f94bc15d550 2024-04-06T12:27:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.087+0000 7f94c3613700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94bc0b75a0 con 0x7f94bc0d1810 2024-04-06T12:27:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.087+0000 7f94c13af700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f94bc15d550 0x7f94bc0b9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.087+0000 7f94c1bb0700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f94bc0d03b0 0x7f94bc0b4ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56916/0 (socket says 172.21.15.189:56916) 2024-04-06T12:27:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.088+0000 7f94c1bb0700 1 -- 172.21.15.189:0/2607650385 learned_addr learned my addr 172.21.15.189:0/2607650385 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.088+0000 7f94c0bae700 1 --2- 172.21.15.189:0/2607650385 >> v2:172.21.15.192:3300/0 conn(0x7f94bc0d1810 0x7f94bc0cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.088+0000 7f94c13af700 1 -- 172.21.15.189:0/2607650385 >> v2:172.21.15.192:3300/0 conn(0x7f94bc0d1810 msgr2=0x7f94bc0cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.088+0000 7f94c13af700 1 --2- 172.21.15.189:0/2607650385 >> v2:172.21.15.192:3300/0 conn(0x7f94bc0d1810 0x7f94bc0cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.088+0000 7f94c13af700 1 -- 172.21.15.189:0/2607650385 >> v1:172.21.15.189:6789/0 conn(0x7f94bc0d03b0 legacy=0x7f94bc0b4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.088+0000 7f94c13af700 1 -- 172.21.15.189:0/2607650385 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94bc0d1ff0 con 0x7f94bc15d550 2024-04-06T12:27:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.088+0000 7f94affff700 1 -- 172.21.15.189:0/2607650385 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627082118 0 0) 0x7f94b4001490 con 0x7f94bc0d03b0 2024-04-06T12:27:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.088+0000 7f94c13af700 1 --2- 172.21.15.189:0/2607650385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94bc15d550 0x7f94bc0b9010 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f94bc0b6360 tx=0x7f94b0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e78648d2a423cc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.088+0000 7f94affff700 1 -- 172.21.15.189:0/2607650385 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b0011830 con 0x7f94bc15d550 2024-04-06T12:27:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.088+0000 7f94affff700 1 -- 172.21.15.189:0/2607650385 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b0011990 con 0x7f94bc15d550 2024-04-06T12:27:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.088+0000 7f94affff700 1 -- 172.21.15.189:0/2607650385 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b0011d20 con 0x7f94bc15d550 2024-04-06T12:27:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.089+0000 7f94c3613700 1 -- 172.21.15.189:0/2607650385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94bc15d550 msgr2=0x7f94bc0b9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.089+0000 7f94c3613700 1 --2- 172.21.15.189:0/2607650385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94bc15d550 0x7f94bc0b9010 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f94bc0b6360 tx=0x7f94b0009b20 comp rx=0 tx=0).stop 2024-04-06T12:27:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.089+0000 7f94c3613700 1 -- 172.21.15.189:0/2607650385 shutdown_connections 2024-04-06T12:27:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.089+0000 7f94c3613700 1 --2- 172.21.15.189:0/2607650385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94bc15d550 0x7f94bc0b9010 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:26.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.089+0000 7f94c3613700 1 --2- 172.21.15.189:0/2607650385 >> v2:172.21.15.192:3300/0 conn(0x7f94bc0d1810 0x7f94bc0cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:26.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.089+0000 7f94c3613700 1 -- 172.21.15.189:0/2607650385 >> 172.21.15.189:0/2607650385 conn(0x7f94bc072490 msgr2=0x7f94bc06b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:26.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.089+0000 7f94c3613700 1 -- 172.21.15.189:0/2607650385 shutdown_connections 2024-04-06T12:27:26.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.089+0000 7f94c3613700 1 -- 172.21.15.189:0/2607650385 wait complete. 2024-04-06T12:27:26.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.090+0000 7f94c3613700 1 Processor -- start 2024-04-06T12:27:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.090+0000 7f94c3613700 1 -- start start 2024-04-06T12:27:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94c3613700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94bc0d03b0 0x7f94bc0cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94c3613700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94bc0d1810 0x7f94bc1e6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94c3613700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94bc15d550 0x7f94bc1e7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94c3613700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc1e3860 con 0x7f94bc15d550 2024-04-06T12:27:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94c3613700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc1e3580 con 0x7f94bc0d03b0 2024-04-06T12:27:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94c3613700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc149680 con 0x7f94bc0d1810 2024-04-06T12:27:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94c1bb0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94bc15d550 0x7f94bc1e7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94c1bb0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94bc15d550 0x7f94bc1e7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37652/0 (socket says 172.21.15.189:37652) 2024-04-06T12:27:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94c1bb0700 1 -- 172.21.15.189:0/2283014507 learned_addr learned my addr 172.21.15.189:0/2283014507 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94c13af700 1 --2- 172.21.15.189:0/2283014507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94bc0d03b0 0x7f94bc0cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94c0bae700 1 --2- 172.21.15.189:0/2283014507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94bc0d1810 0x7f94bc1e6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94c1bb0700 1 -- 172.21.15.189:0/2283014507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94bc0d1810 msgr2=0x7f94bc1e6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94c1bb0700 1 --2- 172.21.15.189:0/2283014507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94bc0d1810 0x7f94bc1e6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94c1bb0700 1 -- 172.21.15.189:0/2283014507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94bc0d03b0 msgr2=0x7f94bc0cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94c1bb0700 1 --2- 172.21.15.189:0/2283014507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94bc0d03b0 0x7f94bc0cc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94c1bb0700 1 -- 172.21.15.189:0/2283014507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b00094a0 con 0x7f94bc15d550 2024-04-06T12:27:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94c1bb0700 1 --2- 172.21.15.189:0/2283014507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94bc15d550 0x7f94bc1e7510 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f94b400cca0 tx=0x7f94b400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94adffb700 1 -- 172.21.15.189:0/2283014507 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b400ab70 con 0x7f94bc15d550 2024-04-06T12:27:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94adffb700 1 -- 172.21.15.189:0/2283014507 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b400acd0 con 0x7f94bc15d550 2024-04-06T12:27:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94c3613700 1 -- 172.21.15.189:0/2283014507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94bc149960 con 0x7f94bc15d550 2024-04-06T12:27:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.091+0000 7f94adffb700 1 -- 172.21.15.189:0/2283014507 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b4007310 con 0x7f94bc15d550 2024-04-06T12:27:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.092+0000 7f94c3613700 1 -- 172.21.15.189:0/2283014507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94bc0ccc20 con 0x7f94bc15d550 2024-04-06T12:27:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.092+0000 7f94c3613700 1 -- 172.21.15.189:0/2283014507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94bc149e80 con 0x7f94bc15d550 2024-04-06T12:27:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.093+0000 7f94adffb700 1 -- 172.21.15.189:0/2283014507 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f94b4007470 con 0x7f94bc15d550 2024-04-06T12:27:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.093+0000 7f94adffb700 1 --2- 172.21.15.189:0/2283014507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f94a006ade0 0x7f94a006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.093+0000 7f94adffb700 1 -- 172.21.15.189:0/2283014507 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f94b4088e60 con 0x7f94bc15d550 2024-04-06T12:27:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.093+0000 7f94c13af700 1 --2- 172.21.15.189:0/2283014507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f94a006ade0 0x7f94a006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:26.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.094+0000 7f94c13af700 1 --2- 172.21.15.189:0/2283014507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f94a006ade0 0x7f94a006d2a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f94b00090b0 tx=0x7f94b001c040 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:26.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.098+0000 7f94adffb700 1 -- 172.21.15.189:0/2283014507 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f94bc149e80 con 0x7f94bc15d550 2024-04-06T12:27:26.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.374+0000 7f94c3613700 1 -- 172.21.15.189:0/2283014507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f94bc0d0800 con 0x7f94bc15d550 2024-04-06T12:27:26.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.374+0000 7f94adffb700 1 -- 172.21.15.189:0/2283014507 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1173 (secure 0 0 0) 0x7f94bc0d0800 con 0x7f94bc15d550 2024-04-06T12:27:26.373 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:26.373 INFO:teuthology.orchestra.run.smithi189.stdout:{"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":939,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":84,"num_osds":8,"num_up_osds":8,"osd_up_since":1712405517,"num_in_osds":8,"osd_in_since":1712405509,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":81},{"state_name":"unknown","count":9},{"state_name":"creating+peering","count":7}],"num_pgs":97,"num_pools":3,"num_objects":18,"data_bytes":592536,"bytes_used":323899392,"bytes_avail":772770213888,"bytes_total":773094113280,"unknown_pgs_ratio":0.092783503234386444,"inactive_pgs_ratio":0.072164945304393768,"write_bytes_sec":899,"read_op_per_sec":0,"write_op_per_sec":5},"fsmap":{"epoch":31,"id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"c","status":"up:active","gid":6582}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-04-06T12:26:18.012503+0000","services":{}},"progress_events":{}} 2024-04-06T12:27:26.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.376+0000 7f94c3613700 1 -- 172.21.15.189:0/2283014507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f94a006ade0 msgr2=0x7f94a006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:26.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.376+0000 7f94c3613700 1 --2- 172.21.15.189:0/2283014507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f94a006ade0 0x7f94a006d2a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f94b00090b0 tx=0x7f94b001c040 comp rx=0 tx=0).stop 2024-04-06T12:27:26.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.376+0000 7f94c3613700 1 -- 172.21.15.189:0/2283014507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94bc15d550 msgr2=0x7f94bc1e7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:26.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.376+0000 7f94c3613700 1 --2- 172.21.15.189:0/2283014507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94bc15d550 0x7f94bc1e7510 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f94b400cca0 tx=0x7f94b400a060 comp rx=0 tx=0).stop 2024-04-06T12:27:26.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.376+0000 7f94c3613700 1 -- 172.21.15.189:0/2283014507 shutdown_connections 2024-04-06T12:27:26.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.376+0000 7f94c3613700 1 --2- 172.21.15.189:0/2283014507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94bc15d550 0x7f94bc1e7510 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:26.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.376+0000 7f94c3613700 1 --2- 172.21.15.189:0/2283014507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f94a006ade0 0x7f94a006d2a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:26.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.376+0000 7f94c3613700 1 --2- 172.21.15.189:0/2283014507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94bc0d03b0 0x7f94bc0cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:26.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.376+0000 7f94c3613700 1 --2- 172.21.15.189:0/2283014507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94bc0d1810 0x7f94bc1e6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:26.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.376+0000 7f94c3613700 1 -- 172.21.15.189:0/2283014507 >> 172.21.15.189:0/2283014507 conn(0x7f94bc072490 msgr2=0x7f94bc0b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:26.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.377+0000 7f94c3613700 1 -- 172.21.15.189:0/2283014507 shutdown_connections 2024-04-06T12:27:26.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.377+0000 7f94c3613700 1 -- 172.21.15.189:0/2283014507 wait complete. 2024-04-06T12:27:26.387 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-06T12:27:26.387 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:27:26.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.511+0000 7f4dd8385700 1 Processor -- start 2024-04-06T12:27:26.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.511+0000 7f4dd8385700 1 -- start start 2024-04-06T12:27:26.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.512+0000 7f4dd8385700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4dd006ae40 0x7f4dd00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:26.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.512+0000 7f4dd8385700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd00b3990 con 0x7f4dd015ddf0 2024-04-06T12:27:26.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.512+0000 7f4dd8385700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd00b3060 con 0x7f4dd015d7b0 2024-04-06T12:27:26.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.512+0000 7f4dd8385700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd015e950 con 0x7f4dd006ae40 2024-04-06T12:27:26.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.512+0000 7f4dd6922700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4dd015ddf0 0x7f4dd015e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56932/0 (socket says 172.21.15.189:56932) 2024-04-06T12:27:26.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.512+0000 7f4dd6922700 1 -- 172.21.15.189:0/1942463937 learned_addr learned my addr 172.21.15.189:0/1942463937 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:26.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.513+0000 7f4dd5920700 1 --2- 172.21.15.189:0/1942463937 >> v2:172.21.15.192:3300/0 conn(0x7f4dd006ae40 0x7f4dd00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:26.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.513+0000 7f4dd511f700 1 -- 172.21.15.189:0/1942463937 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679050360 0 0) 0x7f4dd00b3990 con 0x7f4dd015ddf0 2024-04-06T12:27:26.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.513+0000 7f4dd511f700 1 -- 172.21.15.189:0/1942463937 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dbc0035b0 con 0x7f4dd015ddf0 2024-04-06T12:27:26.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.513+0000 7f4dd511f700 1 -- 172.21.15.189:0/1942463937 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1222627858 0 0) 0x7f4dbc0035b0 con 0x7f4dd015ddf0 2024-04-06T12:27:26.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.513+0000 7f4dd5920700 1 -- 172.21.15.189:0/1942463937 >> v1:172.21.15.192:6789/0 conn(0x7f4dd015d7b0 legacy=0x7f4dd006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:26.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.513+0000 7f4dd5920700 1 -- 172.21.15.189:0/1942463937 >> v1:172.21.15.189:6789/0 conn(0x7f4dd015ddf0 legacy=0x7f4dd015e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:26.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.513+0000 7f4dd5920700 1 -- 172.21.15.189:0/1942463937 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd00b3340 con 0x7f4dd006ae40 2024-04-06T12:27:26.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.513+0000 7f4dd5920700 1 --2- 172.21.15.189:0/1942463937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd006ae40 0x7f4dd00b3e20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009840 tx=0x7f4dcc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=656feac8978027c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:26.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.513+0000 7f4dd511f700 1 -- 172.21.15.189:0/1942463937 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc80019f0 con 0x7f4dd006ae40 2024-04-06T12:27:26.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.513+0000 7f4dd511f700 1 -- 172.21.15.189:0/1942463937 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc004030 con 0x7f4dd006ae40 2024-04-06T12:27:26.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.513+0000 7f4dd511f700 1 -- 172.21.15.189:0/1942463937 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc0043c0 con 0x7f4dd006ae40 2024-04-06T12:27:26.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.514+0000 7f4dd8385700 1 -- 172.21.15.189:0/1942463937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd006ae40 msgr2=0x7f4dd00b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:26.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.514+0000 7f4dd8385700 1 --2- 172.21.15.189:0/1942463937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd006ae40 0x7f4dd00b3e20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009840 tx=0x7f4dcc009b50 comp rx=0 tx=0).stop 2024-04-06T12:27:26.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.514+0000 7f4dd8385700 1 -- 172.21.15.189:0/1942463937 shutdown_connections 2024-04-06T12:27:26.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.514+0000 7f4dd8385700 1 --2- 172.21.15.189:0/1942463937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd006ae40 0x7f4dd00b3e20 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:26.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.514+0000 7f4dd8385700 1 -- 172.21.15.189:0/1942463937 >> 172.21.15.189:0/1942463937 conn(0x7f4dd00d00b0 msgr2=0x7f4dd00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:26.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.514+0000 7f4dd8385700 1 -- 172.21.15.189:0/1942463937 shutdown_connections 2024-04-06T12:27:26.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.514+0000 7f4dd8385700 1 -- 172.21.15.189:0/1942463937 wait complete. 2024-04-06T12:27:26.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.515+0000 7f4dd8385700 1 Processor -- start 2024-04-06T12:27:26.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.515+0000 7f4dd8385700 1 -- start start 2024-04-06T12:27:26.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.516+0000 7f4dd8385700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd006ae40 0x7f4dd0151ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:26.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.516+0000 7f4dd8385700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd015d7b0 0x7f4dd00b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:26.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.516+0000 7f4dd8385700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd015ddf0 0x7f4dd00b1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:26.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.516+0000 7f4dd8385700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd0152530 con 0x7f4dd015d7b0 2024-04-06T12:27:26.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.516+0000 7f4dd8385700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd0157290 con 0x7f4dd006ae40 2024-04-06T12:27:26.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.516+0000 7f4dd8385700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd015c8e0 con 0x7f4dd015ddf0 2024-04-06T12:27:26.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.516+0000 7f4dd5920700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd015d7b0 0x7f4dd00b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:26.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.516+0000 7f4dd5920700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd015d7b0 0x7f4dd00b0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37656/0 (socket says 172.21.15.189:37656) 2024-04-06T12:27:26.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.516+0000 7f4dd5920700 1 -- 172.21.15.189:0/3655665190 learned_addr learned my addr 172.21.15.189:0/3655665190 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:26.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.516+0000 7f4dd6121700 1 --2- 172.21.15.189:0/3655665190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd006ae40 0x7f4dd0151ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:26.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.516+0000 7f4dd6922700 1 --2- 172.21.15.189:0/3655665190 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd015ddf0 0x7f4dd00b1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:26.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.516+0000 7f4dd5920700 1 -- 172.21.15.189:0/3655665190 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd015ddf0 msgr2=0x7f4dd00b1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:26.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.516+0000 7f4dd5920700 1 --2- 172.21.15.189:0/3655665190 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd015ddf0 0x7f4dd00b1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:26.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.516+0000 7f4dd5920700 1 -- 172.21.15.189:0/3655665190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd006ae40 msgr2=0x7f4dd0151ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:26.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.516+0000 7f4dd5920700 1 --2- 172.21.15.189:0/3655665190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd006ae40 0x7f4dd0151ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:26.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.516+0000 7f4dd5920700 1 -- 172.21.15.189:0/3655665190 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dcc0094a0 con 0x7f4dd015d7b0 2024-04-06T12:27:26.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.516+0000 7f4dd6922700 1 --2- 172.21.15.189:0/3655665190 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd015ddf0 0x7f4dd00b1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:26.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.516+0000 7f4dd5920700 1 --2- 172.21.15.189:0/3655665190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd015d7b0 0x7f4dd00b0b50 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009b90 tx=0x7f4dcc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:26.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.517+0000 7f4dc2ffd700 1 -- 172.21.15.189:0/3655665190 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc011830 con 0x7f4dd015d7b0 2024-04-06T12:27:26.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.517+0000 7f4dc2ffd700 1 -- 172.21.15.189:0/3655665190 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc01f070 con 0x7f4dd015d7b0 2024-04-06T12:27:26.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.517+0000 7f4dd8385700 1 -- 172.21.15.189:0/3655665190 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd0156710 con 0x7f4dd015d7b0 2024-04-06T12:27:26.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.517+0000 7f4dc2ffd700 1 -- 172.21.15.189:0/3655665190 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc011bd0 con 0x7f4dd015d7b0 2024-04-06T12:27:26.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.517+0000 7f4dd8385700 1 -- 172.21.15.189:0/3655665190 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd0156cd0 con 0x7f4dd015d7b0 2024-04-06T12:27:26.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.517+0000 7f4dd8385700 1 -- 172.21.15.189:0/3655665190 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd0067bb0 con 0x7f4dd015d7b0 2024-04-06T12:27:26.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.518+0000 7f4dc2ffd700 1 -- 172.21.15.189:0/3655665190 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f4dcc011d30 con 0x7f4dd015d7b0 2024-04-06T12:27:26.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.518+0000 7f4dc2ffd700 1 --2- 172.21.15.189:0/3655665190 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4db406ade0 0x7f4db406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:26.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.519+0000 7f4dc2ffd700 1 -- 172.21.15.189:0/3655665190 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4dcc021070 con 0x7f4dd015d7b0 2024-04-06T12:27:26.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.519+0000 7f4dd6121700 1 --2- 172.21.15.189:0/3655665190 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4db406ade0 0x7f4db406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:26.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.519+0000 7f4dd6121700 1 --2- 172.21.15.189:0/3655665190 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4db406ade0 0x7f4db406d2a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4007fd0 tx=0x7f4dc4009040 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:26.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.523+0000 7f4dc2ffd700 1 -- 172.21.15.189:0/3655665190 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4dcc055eb0 con 0x7f4dd015d7b0 2024-04-06T12:27:26.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.715+0000 7f4dd8385700 1 -- 172.21.15.189:0/3655665190 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4dd015eb40 con 0x7f4db406ade0 2024-04-06T12:27:26.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.722+0000 7f4dc2ffd700 1 -- 172.21.15.189:0/3655665190 <== mgr.5582 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287696 (secure 0 0 0) 0x7f4dc4001560 con 0x7f4db406ade0 2024-04-06T12:27:26.724 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:26.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.726+0000 7f4dd8385700 1 -- 172.21.15.189:0/3655665190 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4db406ade0 msgr2=0x7f4db406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:26.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.726+0000 7f4dd8385700 1 --2- 172.21.15.189:0/3655665190 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4db406ade0 0x7f4db406d2a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4007fd0 tx=0x7f4dc4009040 comp rx=0 tx=0).stop 2024-04-06T12:27:26.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.726+0000 7f4dd8385700 1 -- 172.21.15.189:0/3655665190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd015d7b0 msgr2=0x7f4dd00b0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:26.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.726+0000 7f4dd8385700 1 --2- 172.21.15.189:0/3655665190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd015d7b0 0x7f4dd00b0b50 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009b90 tx=0x7f4dcc004750 comp rx=0 tx=0).stop 2024-04-06T12:27:26.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.726+0000 7f4dd8385700 1 -- 172.21.15.189:0/3655665190 shutdown_connections 2024-04-06T12:27:26.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.726+0000 7f4dd8385700 1 --2- 172.21.15.189:0/3655665190 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd015ddf0 0x7f4dd00b1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:26.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.726+0000 7f4dd8385700 1 --2- 172.21.15.189:0/3655665190 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4db406ade0 0x7f4db406d2a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:26.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.726+0000 7f4dd8385700 1 --2- 172.21.15.189:0/3655665190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd006ae40 0x7f4dd0151ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:26.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.726+0000 7f4dd8385700 1 --2- 172.21.15.189:0/3655665190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd015d7b0 0x7f4dd00b0b50 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:26.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.726+0000 7f4dd8385700 1 -- 172.21.15.189:0/3655665190 >> 172.21.15.189:0/3655665190 conn(0x7f4dd00d00b0 msgr2=0x7f4dd00ade80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:26.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.726+0000 7f4dd8385700 1 -- 172.21.15.189:0/3655665190 shutdown_connections 2024-04-06T12:27:26.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:26.726+0000 7f4dd8385700 1 -- 172.21.15.189:0/3655665190 wait complete. 2024-04-06T12:27:26.727 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:27:26.737 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-06T12:27:25.309302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592536,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":273,"num_write_kb":2421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":192,"ondisk_log_size":192,"up":176,"acting":176,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":137,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":316308,"kb_used_data":9108,"kb_used_omap":12,"kb_used_meta":307059,"kb_avail":754658412,"statfs":{"total":773094113280,"available":772770213888,"internally_reserved":0,"allocated":9326592,"data_stored":5189878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":314428984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2168,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.693015"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046604+0000","last_change":"2024-04-06T12:27:21.054450+0000","last_active":"2024-04-06T12:27:22.046604+0000","last_peered":"2024-04-06T12:27:22.046604+0000","last_clean":"2024-04-06T12:27:22.046604+0000","last_became_active":"2024-04-06T12:27:21.054202+0000","last_became_peered":"2024-04-06T12:27:21.054202+0000","last_unstale":"2024-04-06T12:27:22.046604+0000","last_undegraded":"2024-04-06T12:27:22.046604+0000","last_fullsized":"2024-04-06T12:27:22.046604+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:29:26.946143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046360+0000","last_change":"2024-04-06T12:27:21.053119+0000","last_active":"2024-04-06T12:27:22.046360+0000","last_peered":"2024-04-06T12:27:22.046360+0000","last_clean":"2024-04-06T12:27:22.046360+0000","last_became_active":"2024-04-06T12:27:21.052814+0000","last_became_peered":"2024-04-06T12:27:21.052814+0000","last_unstale":"2024-04-06T12:27:22.046360+0000","last_undegraded":"2024-04-06T12:27:22.046360+0000","last_fullsized":"2024-04-06T12:27:22.046360+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:14.746095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048036+0000","last_change":"2024-04-06T12:27:21.048815+0000","last_active":"2024-04-06T12:27:22.048036+0000","last_peered":"2024-04-06T12:27:22.048036+0000","last_clean":"2024-04-06T12:27:22.048036+0000","last_became_active":"2024-04-06T12:27:21.047390+0000","last_became_peered":"2024-04-06T12:27:21.047390+0000","last_unstale":"2024-04-06T12:27:22.048036+0000","last_undegraded":"2024-04-06T12:27:22.048036+0000","last_fullsized":"2024-04-06T12:27:22.048036+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:53:37.857841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047263+0000","last_change":"2024-04-06T12:27:21.064877+0000","last_active":"2024-04-06T12:27:22.047263+0000","last_peered":"2024-04-06T12:27:22.047263+0000","last_clean":"2024-04-06T12:27:22.047263+0000","last_became_active":"2024-04-06T12:27:21.064649+0000","last_became_peered":"2024-04-06T12:27:21.064649+0000","last_unstale":"2024-04-06T12:27:22.047263+0000","last_undegraded":"2024-04-06T12:27:22.047263+0000","last_fullsized":"2024-04-06T12:27:22.047263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:25.859663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.040075+0000","last_change":"2024-04-06T12:27:21.054089+0000","last_active":"2024-04-06T12:27:22.040075+0000","last_peered":"2024-04-06T12:27:22.040075+0000","last_clean":"2024-04-06T12:27:22.040075+0000","last_became_active":"2024-04-06T12:27:21.053844+0000","last_became_peered":"2024-04-06T12:27:21.053844+0000","last_unstale":"2024-04-06T12:27:22.040075+0000","last_undegraded":"2024-04-06T12:27:22.040075+0000","last_fullsized":"2024-04-06T12:27:22.040075+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:39:49.798363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046048+0000","last_change":"2024-04-06T12:27:21.060315+0000","last_active":"2024-04-06T12:27:22.046048+0000","last_peered":"2024-04-06T12:27:22.046048+0000","last_clean":"2024-04-06T12:27:22.046048+0000","last_became_active":"2024-04-06T12:27:21.060097+0000","last_became_peered":"2024-04-06T12:27:21.060097+0000","last_unstale":"2024-04-06T12:27:22.046048+0000","last_undegraded":"2024-04-06T12:27:22.046048+0000","last_fullsized":"2024-04-06T12:27:22.046048+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:53:09.701399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046794+0000","last_change":"2024-04-06T12:27:21.053746+0000","last_active":"2024-04-06T12:27:22.046794+0000","last_peered":"2024-04-06T12:27:22.046794+0000","last_clean":"2024-04-06T12:27:22.046794+0000","last_became_active":"2024-04-06T12:27:21.053476+0000","last_became_peered":"2024-04-06T12:27:21.053476+0000","last_unstale":"2024-04-06T12:27:22.046794+0000","last_undegraded":"2024-04-06T12:27:22.046794+0000","last_fullsized":"2024-04-06T12:27:22.046794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:59:57.937757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047029+0000","last_change":"2024-04-06T12:27:21.051431+0000","last_active":"2024-04-06T12:27:22.047029+0000","last_peered":"2024-04-06T12:27:22.047029+0000","last_clean":"2024-04-06T12:27:22.047029+0000","last_became_active":"2024-04-06T12:27:21.050731+0000","last_became_peered":"2024-04-06T12:27:21.050731+0000","last_unstale":"2024-04-06T12:27:22.047029+0000","last_undegraded":"2024-04-06T12:27:22.047029+0000","last_fullsized":"2024-04-06T12:27:22.047029+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:26:12.757657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047263+0000","last_change":"2024-04-06T12:27:21.056611+0000","last_active":"2024-04-06T12:27:22.047263+0000","last_peered":"2024-04-06T12:27:22.047263+0000","last_clean":"2024-04-06T12:27:22.047263+0000","last_became_active":"2024-04-06T12:27:21.056361+0000","last_became_peered":"2024-04-06T12:27:21.056361+0000","last_unstale":"2024-04-06T12:27:22.047263+0000","last_undegraded":"2024-04-06T12:27:22.047263+0000","last_fullsized":"2024-04-06T12:27:22.047263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:41:13.712569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047557+0000","last_change":"2024-04-06T12:27:21.053678+0000","last_active":"2024-04-06T12:27:22.047557+0000","last_peered":"2024-04-06T12:27:22.047557+0000","last_clean":"2024-04-06T12:27:22.047557+0000","last_became_active":"2024-04-06T12:27:21.053132+0000","last_became_peered":"2024-04-06T12:27:21.053132+0000","last_unstale":"2024-04-06T12:27:22.047557+0000","last_undegraded":"2024-04-06T12:27:22.047557+0000","last_fullsized":"2024-04-06T12:27:22.047557+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:57.645194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046311+0000","last_change":"2024-04-06T12:27:21.042215+0000","last_active":"2024-04-06T12:27:22.046311+0000","last_peered":"2024-04-06T12:27:22.046311+0000","last_clean":"2024-04-06T12:27:22.046311+0000","last_became_active":"2024-04-06T12:27:21.041921+0000","last_became_peered":"2024-04-06T12:27:21.041921+0000","last_unstale":"2024-04-06T12:27:22.046311+0000","last_undegraded":"2024-04-06T12:27:22.046311+0000","last_fullsized":"2024-04-06T12:27:22.046311+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:33:26.404172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046752+0000","last_change":"2024-04-06T12:27:21.056018+0000","last_active":"2024-04-06T12:27:22.046752+0000","last_peered":"2024-04-06T12:27:22.046752+0000","last_clean":"2024-04-06T12:27:22.046752+0000","last_became_active":"2024-04-06T12:27:21.055764+0000","last_became_peered":"2024-04-06T12:27:21.055764+0000","last_unstale":"2024-04-06T12:27:22.046752+0000","last_undegraded":"2024-04-06T12:27:22.046752+0000","last_fullsized":"2024-04-06T12:27:22.046752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:49:34.397313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047499+0000","last_change":"2024-04-06T12:27:21.052386+0000","last_active":"2024-04-06T12:27:22.047499+0000","last_peered":"2024-04-06T12:27:22.047499+0000","last_clean":"2024-04-06T12:27:22.047499+0000","last_became_active":"2024-04-06T12:27:21.052081+0000","last_became_peered":"2024-04-06T12:27:21.052081+0000","last_unstale":"2024-04-06T12:27:22.047499+0000","last_undegraded":"2024-04-06T12:27:22.047499+0000","last_fullsized":"2024-04-06T12:27:22.047499+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:41:13.712569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047792+0000","last_change":"2024-04-06T12:27:21.059175+0000","last_active":"2024-04-06T12:27:22.047792+0000","last_peered":"2024-04-06T12:27:22.047792+0000","last_clean":"2024-04-06T12:27:22.047792+0000","last_became_active":"2024-04-06T12:27:21.058651+0000","last_became_peered":"2024-04-06T12:27:21.058651+0000","last_unstale":"2024-04-06T12:27:22.047792+0000","last_undegraded":"2024-04-06T12:27:22.047792+0000","last_fullsized":"2024-04-06T12:27:22.047792+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:45.939534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039097+0000","last_change":"2024-04-06T12:27:21.062933+0000","last_active":"2024-04-06T12:27:22.039097+0000","last_peered":"2024-04-06T12:27:22.039097+0000","last_clean":"2024-04-06T12:27:22.039097+0000","last_became_active":"2024-04-06T12:27:21.062688+0000","last_became_peered":"2024-04-06T12:27:21.062688+0000","last_unstale":"2024-04-06T12:27:22.039097+0000","last_undegraded":"2024-04-06T12:27:22.039097+0000","last_fullsized":"2024-04-06T12:27:22.039097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:30:58.878304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047660+0000","last_change":"2024-04-06T12:27:21.059206+0000","last_active":"2024-04-06T12:27:22.047660+0000","last_peered":"2024-04-06T12:27:22.047660+0000","last_clean":"2024-04-06T12:27:22.047660+0000","last_became_active":"2024-04-06T12:27:21.058932+0000","last_became_peered":"2024-04-06T12:27:21.058932+0000","last_unstale":"2024-04-06T12:27:22.047660+0000","last_undegraded":"2024-04-06T12:27:22.047660+0000","last_fullsized":"2024-04-06T12:27:22.047660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:08:32.179469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.038878+0000","last_change":"2024-04-06T12:27:21.053594+0000","last_active":"2024-04-06T12:27:22.038878+0000","last_peered":"2024-04-06T12:27:22.038878+0000","last_clean":"2024-04-06T12:27:22.038878+0000","last_became_active":"2024-04-06T12:27:21.053330+0000","last_became_peered":"2024-04-06T12:27:21.053330+0000","last_unstale":"2024-04-06T12:27:22.038878+0000","last_undegraded":"2024-04-06T12:27:22.038878+0000","last_fullsized":"2024-04-06T12:27:22.038878+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:54:27.984787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.038730+0000","last_change":"2024-04-06T12:27:21.053574+0000","last_active":"2024-04-06T12:27:22.038730+0000","last_peered":"2024-04-06T12:27:22.038730+0000","last_clean":"2024-04-06T12:27:22.038730+0000","last_became_active":"2024-04-06T12:27:21.053289+0000","last_became_peered":"2024-04-06T12:27:21.053289+0000","last_unstale":"2024-04-06T12:27:22.038730+0000","last_undegraded":"2024-04-06T12:27:22.038730+0000","last_fullsized":"2024-04-06T12:27:22.038730+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:44.646361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048049+0000","last_change":"2024-04-06T12:27:21.049364+0000","last_active":"2024-04-06T12:27:22.048049+0000","last_peered":"2024-04-06T12:27:22.048049+0000","last_clean":"2024-04-06T12:27:22.048049+0000","last_became_active":"2024-04-06T12:27:21.048571+0000","last_became_peered":"2024-04-06T12:27:21.048571+0000","last_unstale":"2024-04-06T12:27:22.048049+0000","last_undegraded":"2024-04-06T12:27:22.048049+0000","last_fullsized":"2024-04-06T12:27:22.048049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:24:04.575202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048237+0000","last_change":"2024-04-06T12:27:21.048886+0000","last_active":"2024-04-06T12:27:22.048237+0000","last_peered":"2024-04-06T12:27:22.048237+0000","last_clean":"2024-04-06T12:27:22.048237+0000","last_became_active":"2024-04-06T12:27:21.048635+0000","last_became_peered":"2024-04-06T12:27:21.048635+0000","last_unstale":"2024-04-06T12:27:22.048237+0000","last_undegraded":"2024-04-06T12:27:22.048237+0000","last_fullsized":"2024-04-06T12:27:22.048237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:14.746095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047147+0000","last_change":"2024-04-06T12:27:21.045989+0000","last_active":"2024-04-06T12:27:22.047147+0000","last_peered":"2024-04-06T12:27:22.047147+0000","last_clean":"2024-04-06T12:27:22.047147+0000","last_became_active":"2024-04-06T12:27:21.045733+0000","last_became_peered":"2024-04-06T12:27:21.045733+0000","last_unstale":"2024-04-06T12:27:22.047147+0000","last_undegraded":"2024-04-06T12:27:22.047147+0000","last_fullsized":"2024-04-06T12:27:22.047147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:53:59.500416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046223+0000","last_change":"2024-04-06T12:27:21.063222+0000","last_active":"2024-04-06T12:27:22.046223+0000","last_peered":"2024-04-06T12:27:22.046223+0000","last_clean":"2024-04-06T12:27:22.046223+0000","last_became_active":"2024-04-06T12:27:21.062971+0000","last_became_peered":"2024-04-06T12:27:21.062971+0000","last_unstale":"2024-04-06T12:27:22.046223+0000","last_undegraded":"2024-04-06T12:27:22.046223+0000","last_fullsized":"2024-04-06T12:27:22.046223+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:41:35.413608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046965+0000","last_change":"2024-04-06T12:27:21.057755+0000","last_active":"2024-04-06T12:27:22.046965+0000","last_peered":"2024-04-06T12:27:22.046965+0000","last_clean":"2024-04-06T12:27:22.046965+0000","last_became_active":"2024-04-06T12:27:21.057491+0000","last_became_peered":"2024-04-06T12:27:21.057491+0000","last_unstale":"2024-04-06T12:27:22.046965+0000","last_undegraded":"2024-04-06T12:27:22.046965+0000","last_fullsized":"2024-04-06T12:27:22.046965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:54:23.429749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046895+0000","last_change":"2024-04-06T12:27:21.048114+0000","last_active":"2024-04-06T12:27:22.046895+0000","last_peered":"2024-04-06T12:27:22.046895+0000","last_clean":"2024-04-06T12:27:22.046895+0000","last_became_active":"2024-04-06T12:27:21.047854+0000","last_became_peered":"2024-04-06T12:27:21.047854+0000","last_unstale":"2024-04-06T12:27:22.046895+0000","last_undegraded":"2024-04-06T12:27:22.046895+0000","last_fullsized":"2024-04-06T12:27:22.046895+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:02.552663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039826+0000","last_change":"2024-04-06T12:27:21.054604+0000","last_active":"2024-04-06T12:27:22.039826+0000","last_peered":"2024-04-06T12:27:22.039826+0000","last_clean":"2024-04-06T12:27:22.039826+0000","last_became_active":"2024-04-06T12:27:21.049845+0000","last_became_peered":"2024-04-06T12:27:21.049845+0000","last_unstale":"2024-04-06T12:27:22.039826+0000","last_undegraded":"2024-04-06T12:27:22.039826+0000","last_fullsized":"2024-04-06T12:27:22.039826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:26.827213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047368+0000","last_change":"2024-04-06T12:27:21.064380+0000","last_active":"2024-04-06T12:27:22.047368+0000","last_peered":"2024-04-06T12:27:22.047368+0000","last_clean":"2024-04-06T12:27:22.047368+0000","last_became_active":"2024-04-06T12:27:21.064115+0000","last_became_peered":"2024-04-06T12:27:21.064115+0000","last_unstale":"2024-04-06T12:27:22.047368+0000","last_undegraded":"2024-04-06T12:27:22.047368+0000","last_fullsized":"2024-04-06T12:27:22.047368+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:12:18.899208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.065495+0000","last_change":"2024-04-06T12:27:20.639851+0000","last_active":"2024-04-06T12:27:21.065495+0000","last_peered":"2024-04-06T12:27:21.065495+0000","last_clean":"2024-04-06T12:27:21.065495+0000","last_became_active":"2024-04-06T12:27:20.639619+0000","last_became_peered":"2024-04-06T12:27:20.639619+0000","last_unstale":"2024-04-06T12:27:21.065495+0000","last_undegraded":"2024-04-06T12:27:21.065495+0000","last_fullsized":"2024-04-06T12:27:21.065495+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:24.851744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047260+0000","last_change":"2024-04-06T12:27:21.063852+0000","last_active":"2024-04-06T12:27:22.047260+0000","last_peered":"2024-04-06T12:27:22.047260+0000","last_clean":"2024-04-06T12:27:22.047260+0000","last_became_active":"2024-04-06T12:27:21.063593+0000","last_became_peered":"2024-04-06T12:27:21.063593+0000","last_unstale":"2024-04-06T12:27:22.047260+0000","last_undegraded":"2024-04-06T12:27:22.047260+0000","last_fullsized":"2024-04-06T12:27:22.047260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:35:07.924741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.065665+0000","last_change":"2024-04-06T12:27:20.640009+0000","last_active":"2024-04-06T12:27:21.065665+0000","last_peered":"2024-04-06T12:27:21.065665+0000","last_clean":"2024-04-06T12:27:21.065665+0000","last_became_active":"2024-04-06T12:27:20.639751+0000","last_became_peered":"2024-04-06T12:27:20.639751+0000","last_unstale":"2024-04-06T12:27:21.065665+0000","last_undegraded":"2024-04-06T12:27:21.065665+0000","last_fullsized":"2024-04-06T12:27:21.065665+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:23:10.897604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.040256+0000","last_change":"2024-04-06T12:27:21.050994+0000","last_active":"2024-04-06T12:27:22.040256+0000","last_peered":"2024-04-06T12:27:22.040256+0000","last_clean":"2024-04-06T12:27:22.040256+0000","last_became_active":"2024-04-06T12:27:21.049716+0000","last_became_peered":"2024-04-06T12:27:21.049716+0000","last_unstale":"2024-04-06T12:27:22.040256+0000","last_undegraded":"2024-04-06T12:27:22.040256+0000","last_fullsized":"2024-04-06T12:27:22.040256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:53:03.356300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.059811+0000","last_change":"2024-04-06T12:27:20.644294+0000","last_active":"2024-04-06T12:27:21.059811+0000","last_peered":"2024-04-06T12:27:21.059811+0000","last_clean":"2024-04-06T12:27:21.059811+0000","last_became_active":"2024-04-06T12:27:20.643994+0000","last_became_peered":"2024-04-06T12:27:20.643994+0000","last_unstale":"2024-04-06T12:27:21.059811+0000","last_undegraded":"2024-04-06T12:27:21.059811+0000","last_fullsized":"2024-04-06T12:27:21.059811+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:44:45.550106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.060664+0000","last_change":"2024-04-06T12:27:20.636807+0000","last_active":"2024-04-06T12:27:21.060664+0000","last_peered":"2024-04-06T12:27:21.060664+0000","last_clean":"2024-04-06T12:27:21.060664+0000","last_became_active":"2024-04-06T12:27:20.636516+0000","last_became_peered":"2024-04-06T12:27:20.636516+0000","last_unstale":"2024-04-06T12:27:21.060664+0000","last_undegraded":"2024-04-06T12:27:21.060664+0000","last_fullsized":"2024-04-06T12:27:21.060664+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:00:13.506742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.045921+0000","last_change":"2024-04-06T12:27:21.047046+0000","last_active":"2024-04-06T12:27:22.045921+0000","last_peered":"2024-04-06T12:27:22.045921+0000","last_clean":"2024-04-06T12:27:22.045921+0000","last_became_active":"2024-04-06T12:27:21.046798+0000","last_became_peered":"2024-04-06T12:27:21.046798+0000","last_unstale":"2024-04-06T12:27:22.045921+0000","last_undegraded":"2024-04-06T12:27:22.045921+0000","last_fullsized":"2024-04-06T12:27:22.045921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:23:44.000355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.048716+0000","last_change":"2024-04-06T12:27:20.642417+0000","last_active":"2024-04-06T12:27:21.048716+0000","last_peered":"2024-04-06T12:27:21.048716+0000","last_clean":"2024-04-06T12:27:21.048716+0000","last_became_active":"2024-04-06T12:27:20.642057+0000","last_became_peered":"2024-04-06T12:27:20.642057+0000","last_unstale":"2024-04-06T12:27:21.048716+0000","last_undegraded":"2024-04-06T12:27:21.048716+0000","last_fullsized":"2024-04-06T12:27:21.048716+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:40:56.649109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046707+0000","last_change":"2024-04-06T12:27:21.056061+0000","last_active":"2024-04-06T12:27:22.046707+0000","last_peered":"2024-04-06T12:27:22.046707+0000","last_clean":"2024-04-06T12:27:22.046707+0000","last_became_active":"2024-04-06T12:27:21.055761+0000","last_became_peered":"2024-04-06T12:27:21.055761+0000","last_unstale":"2024-04-06T12:27:22.046707+0000","last_undegraded":"2024-04-06T12:27:22.046707+0000","last_fullsized":"2024-04-06T12:27:22.046707+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:20:15.444097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.046311+0000","last_change":"2024-04-06T12:27:20.642798+0000","last_active":"2024-04-06T12:27:21.046311+0000","last_peered":"2024-04-06T12:27:21.046311+0000","last_clean":"2024-04-06T12:27:21.046311+0000","last_became_active":"2024-04-06T12:27:20.642006+0000","last_became_peered":"2024-04-06T12:27:20.642006+0000","last_unstale":"2024-04-06T12:27:21.046311+0000","last_undegraded":"2024-04-06T12:27:21.046311+0000","last_fullsized":"2024-04-06T12:27:21.046311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:14:39.407591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047517+0000","last_change":"2024-04-06T12:27:21.067188+0000","last_active":"2024-04-06T12:27:22.047517+0000","last_peered":"2024-04-06T12:27:22.047517+0000","last_clean":"2024-04-06T12:27:22.047517+0000","last_became_active":"2024-04-06T12:27:21.066981+0000","last_became_peered":"2024-04-06T12:27:21.066981+0000","last_unstale":"2024-04-06T12:27:22.047517+0000","last_undegraded":"2024-04-06T12:27:22.047517+0000","last_fullsized":"2024-04-06T12:27:22.047517+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:16:15.014557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035060+0000","last_change":"2024-04-06T12:27:20.642838+0000","last_active":"2024-04-06T12:27:21.035060+0000","last_peered":"2024-04-06T12:27:21.035060+0000","last_clean":"2024-04-06T12:27:21.035060+0000","last_became_active":"2024-04-06T12:27:20.642344+0000","last_became_peered":"2024-04-06T12:27:20.642344+0000","last_unstale":"2024-04-06T12:27:21.035060+0000","last_undegraded":"2024-04-06T12:27:21.035060+0000","last_fullsized":"2024-04-06T12:27:21.035060+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:39:48.790443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039044+0000","last_change":"2024-04-06T12:27:21.042767+0000","last_active":"2024-04-06T12:27:22.039044+0000","last_peered":"2024-04-06T12:27:22.039044+0000","last_clean":"2024-04-06T12:27:22.039044+0000","last_became_active":"2024-04-06T12:27:21.042541+0000","last_became_peered":"2024-04-06T12:27:21.042541+0000","last_unstale":"2024-04-06T12:27:22.039044+0000","last_undegraded":"2024-04-06T12:27:22.039044+0000","last_fullsized":"2024-04-06T12:27:22.039044+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:02:23.142749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.049488+0000","last_change":"2024-04-06T12:27:20.640874+0000","last_active":"2024-04-06T12:27:21.049488+0000","last_peered":"2024-04-06T12:27:21.049488+0000","last_clean":"2024-04-06T12:27:21.049488+0000","last_became_active":"2024-04-06T12:27:20.640592+0000","last_became_peered":"2024-04-06T12:27:20.640592+0000","last_unstale":"2024-04-06T12:27:21.049488+0000","last_undegraded":"2024-04-06T12:27:21.049488+0000","last_fullsized":"2024-04-06T12:27:21.049488+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:53:08.693479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046322+0000","last_change":"2024-04-06T12:27:21.050535+0000","last_active":"2024-04-06T12:27:22.046322+0000","last_peered":"2024-04-06T12:27:22.046322+0000","last_clean":"2024-04-06T12:27:22.046322+0000","last_became_active":"2024-04-06T12:27:21.050270+0000","last_became_peered":"2024-04-06T12:27:21.050270+0000","last_unstale":"2024-04-06T12:27:22.046322+0000","last_undegraded":"2024-04-06T12:27:22.046322+0000","last_fullsized":"2024-04-06T12:27:22.046322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:24:04.575202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623334+0000","last_change":"2024-04-06T12:27:19.623334+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623334+0000","last_undegraded":"2024-04-06T12:27:19.623334+0000","last_fullsized":"2024-04-06T12:27:19.623334+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:24.851744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039214+0000","last_change":"2024-04-06T12:27:21.042722+0000","last_active":"2024-04-06T12:27:22.039214+0000","last_peered":"2024-04-06T12:27:22.039214+0000","last_clean":"2024-04-06T12:27:22.039214+0000","last_became_active":"2024-04-06T12:27:21.042460+0000","last_became_peered":"2024-04-06T12:27:21.042460+0000","last_unstale":"2024-04-06T12:27:22.039214+0000","last_undegraded":"2024-04-06T12:27:22.039214+0000","last_fullsized":"2024-04-06T12:27:22.039214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:16:01.192728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.037118+0000","last_change":"2024-04-06T12:27:20.643086+0000","last_active":"2024-04-06T12:27:21.037118+0000","last_peered":"2024-04-06T12:27:21.037118+0000","last_clean":"2024-04-06T12:27:21.037118+0000","last_became_active":"2024-04-06T12:27:20.642767+0000","last_became_peered":"2024-04-06T12:27:20.642767+0000","last_unstale":"2024-04-06T12:27:21.037118+0000","last_undegraded":"2024-04-06T12:27:21.037118+0000","last_fullsized":"2024-04-06T12:27:21.037118+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:54:55.715540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047715+0000","last_change":"2024-04-06T12:27:21.058482+0000","last_active":"2024-04-06T12:27:22.047715+0000","last_peered":"2024-04-06T12:27:22.047715+0000","last_clean":"2024-04-06T12:27:22.047715+0000","last_became_active":"2024-04-06T12:27:21.058235+0000","last_became_peered":"2024-04-06T12:27:21.058235+0000","last_unstale":"2024-04-06T12:27:22.047715+0000","last_undegraded":"2024-04-06T12:27:22.047715+0000","last_fullsized":"2024-04-06T12:27:22.047715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:16.550344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.624131+0000","last_change":"2024-04-06T12:27:19.624131+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.624131+0000","last_undegraded":"2024-04-06T12:27:19.624131+0000","last_fullsized":"2024-04-06T12:27:19.624131+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:15.542425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047714+0000","last_change":"2024-04-06T12:27:21.050955+0000","last_active":"2024-04-06T12:27:22.047714+0000","last_peered":"2024-04-06T12:27:22.047714+0000","last_clean":"2024-04-06T12:27:22.047714+0000","last_became_active":"2024-04-06T12:27:21.050675+0000","last_became_peered":"2024-04-06T12:27:21.050675+0000","last_unstale":"2024-04-06T12:27:22.047714+0000","last_undegraded":"2024-04-06T12:27:22.047714+0000","last_fullsized":"2024-04-06T12:27:22.047714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:57.645194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"83'6","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.067970+0000","last_change":"2024-04-06T12:27:20.638258+0000","last_active":"2024-04-06T12:27:21.067970+0000","last_peered":"2024-04-06T12:27:21.067970+0000","last_clean":"2024-04-06T12:27:21.067970+0000","last_became_active":"2024-04-06T12:27:20.638017+0000","last_became_peered":"2024-04-06T12:27:20.638017+0000","last_unstale":"2024-04-06T12:27:21.067970+0000","last_undegraded":"2024-04-06T12:27:21.067970+0000","last_fullsized":"2024-04-06T12:27:21.067970+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:48:54.260905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047268+0000","last_change":"2024-04-06T12:27:21.048435+0000","last_active":"2024-04-06T12:27:22.047268+0000","last_peered":"2024-04-06T12:27:22.047268+0000","last_clean":"2024-04-06T12:27:22.047268+0000","last_became_active":"2024-04-06T12:27:21.048159+0000","last_became_peered":"2024-04-06T12:27:21.048159+0000","last_unstale":"2024-04-06T12:27:22.047268+0000","last_undegraded":"2024-04-06T12:27:22.047268+0000","last_fullsized":"2024-04-06T12:27:22.047268+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:59:57.937757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623703+0000","last_change":"2024-04-06T12:27:19.623703+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623703+0000","last_undegraded":"2024-04-06T12:27:19.623703+0000","last_fullsized":"2024-04-06T12:27:19.623703+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:48:54.260905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046846+0000","last_change":"2024-04-06T12:27:21.050116+0000","last_active":"2024-04-06T12:27:22.046846+0000","last_peered":"2024-04-06T12:27:22.046846+0000","last_clean":"2024-04-06T12:27:22.046846+0000","last_became_active":"2024-04-06T12:27:21.049828+0000","last_became_peered":"2024-04-06T12:27:21.049828+0000","last_unstale":"2024-04-06T12:27:22.046846+0000","last_undegraded":"2024-04-06T12:27:22.046846+0000","last_fullsized":"2024-04-06T12:27:22.046846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:41:35.413608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.043551+0000","last_change":"2024-04-06T12:27:20.643715+0000","last_active":"2024-04-06T12:27:21.043551+0000","last_peered":"2024-04-06T12:27:21.043551+0000","last_clean":"2024-04-06T12:27:21.043551+0000","last_became_active":"2024-04-06T12:27:20.643243+0000","last_became_peered":"2024-04-06T12:27:20.643243+0000","last_unstale":"2024-04-06T12:27:21.043551+0000","last_undegraded":"2024-04-06T12:27:21.043551+0000","last_fullsized":"2024-04-06T12:27:21.043551+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:48.861736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047578+0000","last_change":"2024-04-06T12:27:21.061425+0000","last_active":"2024-04-06T12:27:22.047578+0000","last_peered":"2024-04-06T12:27:22.047578+0000","last_clean":"2024-04-06T12:27:22.047578+0000","last_became_active":"2024-04-06T12:27:21.061137+0000","last_became_peered":"2024-04-06T12:27:21.061137+0000","last_unstale":"2024-04-06T12:27:22.047578+0000","last_undegraded":"2024-04-06T12:27:22.047578+0000","last_fullsized":"2024-04-06T12:27:22.047578+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:29:33.494735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.622973+0000","last_change":"2024-04-06T12:27:19.622973+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.622973+0000","last_undegraded":"2024-04-06T12:27:19.622973+0000","last_fullsized":"2024-04-06T12:27:19.622973+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:55.070413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046208+0000","last_change":"2024-04-06T12:27:21.051804+0000","last_active":"2024-04-06T12:27:22.046208+0000","last_peered":"2024-04-06T12:27:22.046208+0000","last_clean":"2024-04-06T12:27:22.046208+0000","last_became_active":"2024-04-06T12:27:21.051536+0000","last_became_peered":"2024-04-06T12:27:21.051536+0000","last_unstale":"2024-04-06T12:27:22.046208+0000","last_undegraded":"2024-04-06T12:27:22.046208+0000","last_fullsized":"2024-04-06T12:27:22.046208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:26:12.757657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"83'8","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.063908+0000","last_change":"2024-04-06T12:27:20.640204+0000","last_active":"2024-04-06T12:27:21.063908+0000","last_peered":"2024-04-06T12:27:21.063908+0000","last_clean":"2024-04-06T12:27:21.063908+0000","last_became_active":"2024-04-06T12:27:20.639913+0000","last_became_peered":"2024-04-06T12:27:20.639913+0000","last_unstale":"2024-04-06T12:27:21.063908+0000","last_undegraded":"2024-04-06T12:27:21.063908+0000","last_fullsized":"2024-04-06T12:27:21.063908+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:01:44.323843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:27:20.616323+0000","last_change":"2024-04-06T12:27:20.616323+0000","last_active":"2024-04-06T12:27:20.616323+0000","last_peered":"2024-04-06T12:27:20.616323+0000","last_clean":"2024-04-06T12:27:20.616323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:20.616323+0000","last_undegraded":"2024-04-06T12:27:20.616323+0000","last_fullsized":"2024-04-06T12:27:20.616323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.059785+0000","last_change":"2024-04-06T12:27:20.643933+0000","last_active":"2024-04-06T12:27:21.059785+0000","last_peered":"2024-04-06T12:27:21.059785+0000","last_clean":"2024-04-06T12:27:21.059785+0000","last_became_active":"2024-04-06T12:27:20.643644+0000","last_became_peered":"2024-04-06T12:27:20.643644+0000","last_unstale":"2024-04-06T12:27:21.059785+0000","last_undegraded":"2024-04-06T12:27:21.059785+0000","last_fullsized":"2024-04-06T12:27:21.059785+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:55.070413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047888+0000","last_change":"2024-04-06T12:27:21.059329+0000","last_active":"2024-04-06T12:27:22.047888+0000","last_peered":"2024-04-06T12:27:22.047888+0000","last_clean":"2024-04-06T12:27:22.047888+0000","last_became_active":"2024-04-06T12:27:21.058948+0000","last_became_peered":"2024-04-06T12:27:21.058948+0000","last_unstale":"2024-04-06T12:27:22.047888+0000","last_undegraded":"2024-04-06T12:27:22.047888+0000","last_fullsized":"2024-04-06T12:27:22.047888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:56.078333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.622989+0000","last_change":"2024-04-06T12:27:19.622989+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.622989+0000","last_undegraded":"2024-04-06T12:27:19.622989+0000","last_fullsized":"2024-04-06T12:27:19.622989+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:29:32.486815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047663+0000","last_change":"2024-04-06T12:27:21.059198+0000","last_active":"2024-04-06T12:27:22.047663+0000","last_peered":"2024-04-06T12:27:22.047663+0000","last_clean":"2024-04-06T12:27:22.047663+0000","last_became_active":"2024-04-06T12:27:21.058950+0000","last_became_peered":"2024-04-06T12:27:21.058950+0000","last_unstale":"2024-04-06T12:27:22.047663+0000","last_undegraded":"2024-04-06T12:27:22.047663+0000","last_fullsized":"2024-04-06T12:27:22.047663+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:20:12.514805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623711+0000","last_change":"2024-04-06T12:27:19.623711+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623711+0000","last_undegraded":"2024-04-06T12:27:19.623711+0000","last_fullsized":"2024-04-06T12:27:19.623711+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:44.931614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047541+0000","last_change":"2024-04-06T12:27:21.061929+0000","last_active":"2024-04-06T12:27:22.047541+0000","last_peered":"2024-04-06T12:27:22.047541+0000","last_clean":"2024-04-06T12:27:22.047541+0000","last_became_active":"2024-04-06T12:27:21.061627+0000","last_became_peered":"2024-04-06T12:27:21.061627+0000","last_unstale":"2024-04-06T12:27:22.047541+0000","last_undegraded":"2024-04-06T12:27:22.047541+0000","last_fullsized":"2024-04-06T12:27:22.047541+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:02.552663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035432+0000","last_change":"2024-04-06T12:27:20.642400+0000","last_active":"2024-04-06T12:27:21.035432+0000","last_peered":"2024-04-06T12:27:21.035432+0000","last_clean":"2024-04-06T12:27:21.035432+0000","last_became_active":"2024-04-06T12:27:20.642039+0000","last_became_peered":"2024-04-06T12:27:20.642039+0000","last_unstale":"2024-04-06T12:27:21.035432+0000","last_undegraded":"2024-04-06T12:27:21.035432+0000","last_fullsized":"2024-04-06T12:27:21.035432+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:44.642507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039877+0000","last_change":"2024-04-06T12:27:21.046054+0000","last_active":"2024-04-06T12:27:22.039877+0000","last_peered":"2024-04-06T12:27:22.039877+0000","last_clean":"2024-04-06T12:27:22.039877+0000","last_became_active":"2024-04-06T12:27:21.045799+0000","last_became_peered":"2024-04-06T12:27:21.045799+0000","last_unstale":"2024-04-06T12:27:22.039877+0000","last_undegraded":"2024-04-06T12:27:22.039877+0000","last_fullsized":"2024-04-06T12:27:22.039877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:29:16.068086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.059940+0000","last_change":"2024-04-06T12:27:20.635120+0000","last_active":"2024-04-06T12:27:21.059940+0000","last_peered":"2024-04-06T12:27:21.059940+0000","last_clean":"2024-04-06T12:27:21.059940+0000","last_became_active":"2024-04-06T12:27:20.634852+0000","last_became_peered":"2024-04-06T12:27:20.634852+0000","last_unstale":"2024-04-06T12:27:21.059940+0000","last_undegraded":"2024-04-06T12:27:21.059940+0000","last_fullsized":"2024-04-06T12:27:21.059940+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:34:57.334069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046898+0000","last_change":"2024-04-06T12:27:21.044875+0000","last_active":"2024-04-06T12:27:22.046898+0000","last_peered":"2024-04-06T12:27:22.046898+0000","last_clean":"2024-04-06T12:27:22.046898+0000","last_became_active":"2024-04-06T12:27:21.044609+0000","last_became_peered":"2024-04-06T12:27:21.044609+0000","last_unstale":"2024-04-06T12:27:22.046898+0000","last_undegraded":"2024-04-06T12:27:22.046898+0000","last_fullsized":"2024-04-06T12:27:22.046898+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:48:55.268825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.043598+0000","last_change":"2024-04-06T12:27:20.643509+0000","last_active":"2024-04-06T12:27:21.043598+0000","last_peered":"2024-04-06T12:27:21.043598+0000","last_clean":"2024-04-06T12:27:21.043598+0000","last_became_active":"2024-04-06T12:27:20.642877+0000","last_became_peered":"2024-04-06T12:27:20.642877+0000","last_unstale":"2024-04-06T12:27:21.043598+0000","last_undegraded":"2024-04-06T12:27:21.043598+0000","last_fullsized":"2024-04-06T12:27:21.043598+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:12:17.891288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048159+0000","last_change":"2024-04-06T12:27:21.063218+0000","last_active":"2024-04-06T12:27:22.048159+0000","last_peered":"2024-04-06T12:27:22.048159+0000","last_clean":"2024-04-06T12:27:22.048159+0000","last_became_active":"2024-04-06T12:27:21.062952+0000","last_became_peered":"2024-04-06T12:27:21.062952+0000","last_unstale":"2024-04-06T12:27:22.048159+0000","last_undegraded":"2024-04-06T12:27:22.048159+0000","last_fullsized":"2024-04-06T12:27:22.048159+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:54:56.723460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.038449+0000","last_change":"2024-04-06T12:27:20.640135+0000","last_active":"2024-04-06T12:27:21.038449+0000","last_peered":"2024-04-06T12:27:21.038449+0000","last_clean":"2024-04-06T12:27:21.038449+0000","last_became_active":"2024-04-06T12:27:20.639873+0000","last_became_peered":"2024-04-06T12:27:20.639873+0000","last_unstale":"2024-04-06T12:27:21.038449+0000","last_undegraded":"2024-04-06T12:27:21.038449+0000","last_fullsized":"2024-04-06T12:27:21.038449+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:26:31.570422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.040070+0000","last_change":"2024-04-06T12:27:21.053276+0000","last_active":"2024-04-06T12:27:22.040070+0000","last_peered":"2024-04-06T12:27:22.040070+0000","last_clean":"2024-04-06T12:27:22.040070+0000","last_became_active":"2024-04-06T12:27:21.053033+0000","last_became_peered":"2024-04-06T12:27:21.053033+0000","last_unstale":"2024-04-06T12:27:22.040070+0000","last_undegraded":"2024-04-06T12:27:22.040070+0000","last_fullsized":"2024-04-06T12:27:22.040070+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:49:47.315178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.061029+0000","last_change":"2024-04-06T12:27:20.642377+0000","last_active":"2024-04-06T12:27:21.061029+0000","last_peered":"2024-04-06T12:27:21.061029+0000","last_clean":"2024-04-06T12:27:21.061029+0000","last_became_active":"2024-04-06T12:27:20.642111+0000","last_became_peered":"2024-04-06T12:27:20.642111+0000","last_unstale":"2024-04-06T12:27:21.061029+0000","last_undegraded":"2024-04-06T12:27:21.061029+0000","last_fullsized":"2024-04-06T12:27:21.061029+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:46:21.677982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046699+0000","last_change":"2024-04-06T12:27:21.047728+0000","last_active":"2024-04-06T12:27:22.046699+0000","last_peered":"2024-04-06T12:27:22.046699+0000","last_clean":"2024-04-06T12:27:22.046699+0000","last_became_active":"2024-04-06T12:27:21.046317+0000","last_became_peered":"2024-04-06T12:27:21.046317+0000","last_unstale":"2024-04-06T12:27:22.046699+0000","last_undegraded":"2024-04-06T12:27:22.046699+0000","last_fullsized":"2024-04-06T12:27:22.046699+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:08:29.040668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.702984+0000","last_change":"2024-04-06T12:27:20.643133+0000","last_active":"2024-04-06T12:27:21.702984+0000","last_peered":"2024-04-06T12:27:21.702984+0000","last_clean":"2024-04-06T12:27:21.702984+0000","last_became_active":"2024-04-06T12:27:20.642858+0000","last_became_peered":"2024-04-06T12:27:20.642858+0000","last_unstale":"2024-04-06T12:27:21.702984+0000","last_undegraded":"2024-04-06T12:27:21.702984+0000","last_fullsized":"2024-04-06T12:27:21.702984+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:16:14.006637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048294+0000","last_change":"2024-04-06T12:27:21.059468+0000","last_active":"2024-04-06T12:27:22.048294+0000","last_peered":"2024-04-06T12:27:22.048294+0000","last_clean":"2024-04-06T12:27:22.048294+0000","last_became_active":"2024-04-06T12:27:21.059222+0000","last_became_peered":"2024-04-06T12:27:21.059222+0000","last_unstale":"2024-04-06T12:27:22.048294+0000","last_undegraded":"2024-04-06T12:27:22.048294+0000","last_fullsized":"2024-04-06T12:27:22.048294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:20:15.444097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.040662+0000","last_change":"2024-04-06T12:27:20.638922+0000","last_active":"2024-04-06T12:27:21.040662+0000","last_peered":"2024-04-06T12:27:21.040662+0000","last_clean":"2024-04-06T12:27:21.040662+0000","last_became_active":"2024-04-06T12:27:20.638666+0000","last_became_peered":"2024-04-06T12:27:20.638666+0000","last_unstale":"2024-04-06T12:27:21.040662+0000","last_undegraded":"2024-04-06T12:27:21.040662+0000","last_fullsized":"2024-04-06T12:27:21.040662+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:08:18.449996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047650+0000","last_change":"2024-04-06T12:27:21.060870+0000","last_active":"2024-04-06T12:27:22.047650+0000","last_peered":"2024-04-06T12:27:22.047650+0000","last_clean":"2024-04-06T12:27:22.047650+0000","last_became_active":"2024-04-06T12:27:21.060603+0000","last_became_peered":"2024-04-06T12:27:21.060603+0000","last_unstale":"2024-04-06T12:27:22.047650+0000","last_undegraded":"2024-04-06T12:27:22.047650+0000","last_fullsized":"2024-04-06T12:27:22.047650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:48:55.268825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.044529+0000","last_change":"2024-04-06T12:27:20.639042+0000","last_active":"2024-04-06T12:27:21.044529+0000","last_peered":"2024-04-06T12:27:21.044529+0000","last_clean":"2024-04-06T12:27:21.044529+0000","last_became_active":"2024-04-06T12:27:20.638475+0000","last_became_peered":"2024-04-06T12:27:20.638475+0000","last_unstale":"2024-04-06T12:27:21.044529+0000","last_undegraded":"2024-04-06T12:27:21.044529+0000","last_fullsized":"2024-04-06T12:27:21.044529+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:15.542425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047260+0000","last_change":"2024-04-06T12:27:21.060781+0000","last_active":"2024-04-06T12:27:22.047260+0000","last_peered":"2024-04-06T12:27:22.047260+0000","last_clean":"2024-04-06T12:27:22.047260+0000","last_became_active":"2024-04-06T12:27:21.060520+0000","last_became_peered":"2024-04-06T12:27:21.060520+0000","last_unstale":"2024-04-06T12:27:22.047260+0000","last_undegraded":"2024-04-06T12:27:22.047260+0000","last_fullsized":"2024-04-06T12:27:22.047260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:46:22.685902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-04-06T12:27:19.623238+0000","last_change":"2024-04-06T12:27:19.623238+0000","last_active":"2024-04-06T12:27:19.608404+0000","last_peered":"2024-04-06T12:27:19.608404+0000","last_clean":"2024-04-06T12:27:19.608404+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:27:19.623238+0000","last_undegraded":"2024-04-06T12:27:19.623238+0000","last_fullsized":"2024-04-06T12:27:19.623238+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:48.861736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047005+0000","last_change":"2024-04-06T12:27:21.055075+0000","last_active":"2024-04-06T12:27:22.047005+0000","last_peered":"2024-04-06T12:27:22.047005+0000","last_clean":"2024-04-06T12:27:22.047005+0000","last_became_active":"2024-04-06T12:27:21.054806+0000","last_became_peered":"2024-04-06T12:27:21.054806+0000","last_unstale":"2024-04-06T12:27:22.047005+0000","last_undegraded":"2024-04-06T12:27:22.047005+0000","last_fullsized":"2024-04-06T12:27:22.047005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:14.746095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035500+0000","last_change":"2024-04-06T12:27:20.641433+0000","last_active":"2024-04-06T12:27:21.035500+0000","last_peered":"2024-04-06T12:27:21.035500+0000","last_clean":"2024-04-06T12:27:21.035500+0000","last_became_active":"2024-04-06T12:27:20.641181+0000","last_became_peered":"2024-04-06T12:27:20.641181+0000","last_unstale":"2024-04-06T12:27:21.035500+0000","last_undegraded":"2024-04-06T12:27:21.035500+0000","last_fullsized":"2024-04-06T12:27:21.035500+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:20:08.218629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046219+0000","last_change":"2024-04-06T12:27:21.055928+0000","last_active":"2024-04-06T12:27:22.046219+0000","last_peered":"2024-04-06T12:27:22.046219+0000","last_clean":"2024-04-06T12:27:22.046219+0000","last_became_active":"2024-04-06T12:27:21.055641+0000","last_became_peered":"2024-04-06T12:27:21.055641+0000","last_unstale":"2024-04-06T12:27:22.046219+0000","last_undegraded":"2024-04-06T12:27:22.046219+0000","last_fullsized":"2024-04-06T12:27:22.046219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:08:32.179469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035357+0000","last_change":"2024-04-06T12:27:20.643620+0000","last_active":"2024-04-06T12:27:21.035357+0000","last_peered":"2024-04-06T12:27:21.035357+0000","last_clean":"2024-04-06T12:27:21.035357+0000","last_became_active":"2024-04-06T12:27:20.642671+0000","last_became_peered":"2024-04-06T12:27:20.642671+0000","last_unstale":"2024-04-06T12:27:21.035357+0000","last_undegraded":"2024-04-06T12:27:21.035357+0000","last_fullsized":"2024-04-06T12:27:21.035357+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:56:37.738753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046202+0000","last_change":"2024-04-06T12:27:21.046951+0000","last_active":"2024-04-06T12:27:22.046202+0000","last_peered":"2024-04-06T12:27:22.046202+0000","last_clean":"2024-04-06T12:27:22.046202+0000","last_became_active":"2024-04-06T12:27:21.046651+0000","last_became_peered":"2024-04-06T12:27:21.046651+0000","last_unstale":"2024-04-06T12:27:22.046202+0000","last_undegraded":"2024-04-06T12:27:22.046202+0000","last_fullsized":"2024-04-06T12:27:22.046202+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:59:55.516160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.044036+0000","last_change":"2024-04-06T12:27:20.635487+0000","last_active":"2024-04-06T12:27:21.044036+0000","last_peered":"2024-04-06T12:27:21.044036+0000","last_clean":"2024-04-06T12:27:21.044036+0000","last_became_active":"2024-04-06T12:27:20.635194+0000","last_became_peered":"2024-04-06T12:27:20.635194+0000","last_unstale":"2024-04-06T12:27:21.044036+0000","last_undegraded":"2024-04-06T12:27:21.044036+0000","last_fullsized":"2024-04-06T12:27:21.044036+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:01.544743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047357+0000","last_change":"2024-04-06T12:27:21.062962+0000","last_active":"2024-04-06T12:27:22.047357+0000","last_peered":"2024-04-06T12:27:22.047357+0000","last_clean":"2024-04-06T12:27:22.047357+0000","last_became_active":"2024-04-06T12:27:21.062683+0000","last_became_peered":"2024-04-06T12:27:21.062683+0000","last_unstale":"2024-04-06T12:27:22.047357+0000","last_undegraded":"2024-04-06T12:27:22.047357+0000","last_fullsized":"2024-04-06T12:27:22.047357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:49.869655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035100+0000","last_change":"2024-04-06T12:27:20.640825+0000","last_active":"2024-04-06T12:27:21.035100+0000","last_peered":"2024-04-06T12:27:21.035100+0000","last_clean":"2024-04-06T12:27:21.035100+0000","last_became_active":"2024-04-06T12:27:20.640568+0000","last_became_peered":"2024-04-06T12:27:20.640568+0000","last_unstale":"2024-04-06T12:27:21.035100+0000","last_undegraded":"2024-04-06T12:27:21.035100+0000","last_fullsized":"2024-04-06T12:27:21.035100+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:27:16.320148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'161","reported_seq":388,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-06T12:27:17.742656+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:27:17.742656+0000","last_peered":"2024-04-06T12:27:17.742656+0000","last_clean":"2024-04-06T12:27:17.742656+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:27:17.742656+0000","last_undegraded":"2024-04-06T12:27:17.742656+0000","last_fullsized":"2024-04-06T12:27:17.742656+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":110,"acting":110,"num_store_stats":7},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2168,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575034,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58900,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312940,"statfs":{"total":96636764160,"available":96576450560,"internally_reserved":0,"allocated":1564672,"data_stored":1068520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55800000000000005,"15min":0.54900000000000004},"min":{"1min":0.375,"5min":0.219,"15min":0.219},"max":{"1min":0.80100000000000005,"5min":1.143,"15min":1.143},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55700000000000005,"15min":0.55300000000000005},"min":{"1min":0.433,"5min":0.35399999999999998,"15min":0.25900000000000001},"max":{"1min":0.73299999999999998,"5min":0.77100000000000002,"15min":0.78200000000000003},"last":0.48399999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55400000000000005,"15min":0.55100000000000005},"min":{"1min":0.42299999999999999,"5min":0.33600000000000002,"15min":0.31900000000000001},"max":{"1min":0.748,"5min":0.77100000000000002,"15min":0.81399999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.41699999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.77400000000000002,"5min":0.79400000000000004,"15min":0.79600000000000004},"last":0.372}]},{"osd":2,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56999999999999995,"15min":0.55200000000000005},"min":{"1min":0.441,"5min":0.249,"15min":0.249},"max":{"1min":0.77100000000000002,"5min":0.78500000000000003,"15min":0.83299999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57099999999999995,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.24199999999999999},"max":{"1min":0.76600000000000001,"5min":0.78300000000000003,"15min":0.82299999999999995},"last":0.58999999999999997}]},{"osd":3,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58099999999999996,"15min":0.57199999999999995},"min":{"1min":0.45900000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.75800000000000001,"5min":1.357,"15min":1.357},"last":0.502},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59299999999999997,"15min":0.59599999999999997},"min":{"1min":0.44400000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.748,"5min":0.81799999999999995,"15min":0.92000000000000004},"last":0.626}]},{"osd":4,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.45300000000000001,"15min":0.45800000000000002},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.25},"max":{"1min":0.68300000000000005,"5min":0.70399999999999996,"15min":0.83599999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.53700000000000003},"min":{"1min":0.42899999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71399999999999997,"5min":0.77700000000000002,"15min":1.0780000000000001},"last":0.54400000000000004}]},{"osd":5,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.498,"15min":0.51300000000000001},"min":{"1min":0.33600000000000002,"5min":0.29199999999999998,"15min":0.23799999999999999},"max":{"1min":0.73299999999999998,"5min":0.80100000000000005,"15min":0.98199999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55900000000000005,"15min":0.56599999999999995},"min":{"1min":0.374,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.70699999999999996,"5min":0.81599999999999995,"15min":0.93600000000000005},"last":0.60599999999999998}]},{"osd":6,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53900000000000003,"15min":0.54900000000000004},"min":{"1min":0.39300000000000002,"5min":0.33700000000000002,"15min":0.28799999999999998},"max":{"1min":0.70699999999999996,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55800000000000005,"15min":0.56799999999999995},"min":{"1min":0.40200000000000002,"5min":0.30399999999999999,"15min":0.26900000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.88500000000000001},"last":0.58099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575034,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":1019904,"data_stored":503606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.36699999999999999,"5min":0.26200000000000001,"15min":0.23000000000000001},"max":{"1min":0.77500000000000002,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52500000000000002,"15min":0.53300000000000003},"min":{"1min":0.374,"5min":0.30199999999999999,"15min":0.245},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.53300000000000003}]},{"osd":1,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.32700000000000001,"5min":0.32600000000000001,"15min":0.24399999999999999},"max":{"1min":0.78200000000000003,"5min":0.78700000000000003,"15min":0.84999999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.38400000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.83099999999999996},"last":0.50700000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.39100000000000001,"5min":0.30499999999999999,"15min":0.29899999999999999},"max":{"1min":0.749,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.38800000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.56899999999999995,"15min":0.58099999999999996},"min":{"1min":0.35099999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.745,"5min":0.79000000000000004,"15min":0.81200000000000006},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.60899999999999999,"15min":0.61899999999999999},"min":{"1min":0.51700000000000002,"5min":0.41099999999999998,"15min":0.374},"max":{"1min":0.79200000000000004,"5min":0.82799999999999996,"15min":0.872},"last":0.52000000000000002}]},{"osd":4,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.49099999999999999,"15min":0.496},"min":{"1min":0.28599999999999998,"5min":0.27500000000000002,"15min":0.27300000000000002},"max":{"1min":0.747,"5min":0.749,"15min":0.749},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.53000000000000003},"min":{"1min":0.34899999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.73199999999999998,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.43099999999999999}]},{"osd":5,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.47999999999999998,"15min":0.49199999999999999},"min":{"1min":0.34699999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71099999999999997,"5min":0.755,"15min":0.75800000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.50600000000000001,"15min":0.51300000000000001},"min":{"1min":0.32800000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72599999999999998,"5min":0.83699999999999997,"15min":0.92800000000000005},"last":0.45400000000000001}]},{"osd":7,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53000000000000003,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70099999999999996,"5min":0.79800000000000004,"15min":0.81000000000000005},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55800000000000005,"15min":0.56100000000000005},"min":{"1min":0.41899999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80800000000000005,"5min":0.88900000000000001,"15min":0.96099999999999997},"last":0.54800000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575033,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":1044480,"data_stored":505900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.377,"15min":0.36199999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.47699999999999998,"5min":0.58999999999999997,"15min":0.71699999999999997},"last":0.308},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39900000000000002,"15min":0.39600000000000002},"min":{"1min":0.253,"5min":0.215,"15min":0.215},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.32600000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.40500000000000003,"15min":0.39600000000000002},"min":{"1min":0.27200000000000002,"5min":0.23899999999999999,"15min":0.19400000000000001},"max":{"1min":0.57599999999999996,"5min":0.624,"15min":0.67600000000000005},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.432,"15min":0.42999999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.249},"max":{"1min":0.60699999999999998,"5min":0.63600000000000001,"15min":0.79000000000000004},"last":0.28899999999999998}]},{"osd":3,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45600000000000002,"15min":0.44700000000000001},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67400000000000004,"5min":0.67800000000000005,"15min":0.84299999999999997},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.42599999999999999,"15min":0.41899999999999998},"min":{"1min":0.30499999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.56499999999999995,"5min":0.69599999999999995,"15min":0.86299999999999999},"last":0.375}]},{"osd":4,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.623,"15min":0.63200000000000001},"min":{"1min":0.42599999999999999,"5min":0.40100000000000002,"15min":0.34799999999999998},"max":{"1min":0.71499999999999997,"5min":0.86799999999999999,"15min":6.3739999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.63100000000000001,"15min":0.622},"min":{"1min":0.45400000000000001,"5min":0.441,"15min":0.30399999999999999},"max":{"1min":0.84299999999999997,"5min":0.91400000000000003,"15min":1.0269999999999999},"last":0.504}]},{"osd":5,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.624,"15min":0.61399999999999999},"min":{"1min":0.47599999999999998,"5min":0.42999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83999999999999997,"5min":0.91300000000000003,"15min":0.97699999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.628},"min":{"1min":0.442,"5min":0.43099999999999999,"15min":0.35099999999999998},"max":{"1min":0.78700000000000003,"5min":0.999,"15min":1.0089999999999999},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.623,"15min":0.61399999999999999},"min":{"1min":0.47699999999999998,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":0.76500000000000001,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.65300000000000002,"15min":0.64900000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.41799999999999998},"max":{"1min":0.83999999999999997,"5min":0.88700000000000001,"15min":1.151},"last":0.441}]},{"osd":7,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.64800000000000002,"15min":0.65500000000000003},"min":{"1min":0.44600000000000001,"5min":0.41899999999999998,"15min":0.38800000000000001},"max":{"1min":0.81699999999999995,"5min":1.0329999999999999,"15min":6.3639999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.47299999999999998,"5min":0.40200000000000002,"15min":0.39300000000000002},"max":{"1min":0.82499999999999996,"5min":0.95599999999999996,"15min":1.073},"last":0.55400000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575033,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32688,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339152,"statfs":{"total":96636764160,"available":96603291648,"internally_reserved":0,"allocated":1609728,"data_stored":1093940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.38700000000000001,"15min":0.38},"min":{"1min":0.312,"5min":0.19,"15min":0.19},"max":{"1min":0.53900000000000003,"5min":0.80000000000000004,"15min":0.89100000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.40200000000000002},"min":{"1min":0.30599999999999999,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":0.60599999999999998,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.32300000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.40899999999999997,"15min":0.41199999999999998},"min":{"1min":0.29899999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.57799999999999996,"5min":0.60299999999999998,"15min":1.8089999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.434,"15min":0.42899999999999999},"min":{"1min":0.32600000000000001,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":0.64700000000000002,"5min":1.4790000000000001,"15min":1.764},"last":0.35899999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.41899999999999998,"15min":0.41099999999999998},"min":{"1min":0.34100000000000003,"5min":0.252,"15min":0.21299999999999999},"max":{"1min":0.63,"5min":1.25,"15min":1.25},"last":0.309},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.32900000000000001,"5min":0.28499999999999998,"15min":0.249},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.92000000000000004},"last":0.41199999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.627,"15min":0.61799999999999999},"min":{"1min":0.46100000000000002,"5min":0.371,"15min":0.35699999999999998},"max":{"1min":0.77700000000000002,"5min":1.514,"15min":1.514},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.63900000000000001,"15min":0.628},"min":{"1min":0.47599999999999998,"5min":0.37,"15min":0.311},"max":{"1min":0.86799999999999999,"5min":1.202,"15min":1.202},"last":0.60299999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63400000000000001,"15min":0.61899999999999999},"min":{"1min":0.46999999999999997,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.84899999999999998,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64400000000000002,"15min":0.64700000000000002},"min":{"1min":0.45200000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.88600000000000001,"5min":0.94499999999999995,"15min":0.97899999999999998},"last":0.746}]},{"osd":6,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65300000000000002,"15min":0.64800000000000002},"min":{"1min":0.51700000000000002,"5min":0.39100000000000001,"15min":0.378},"max":{"1min":0.84999999999999998,"5min":0.91000000000000003,"15min":0.96699999999999997},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.66900000000000004,"15min":0.66400000000000003},"min":{"1min":0.499,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84999999999999998,"5min":1.613,"15min":1.613},"last":0.58799999999999997}]},{"osd":7,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67000000000000004,"15min":0.66300000000000003},"min":{"1min":0.51500000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90100000000000002,"5min":1.54,"15min":1.54},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.67700000000000005,"15min":0.65700000000000003},"min":{"1min":0.51400000000000001,"5min":0.38400000000000001,"15min":0.36699999999999999},"max":{"1min":0.88300000000000001,"5min":1.498,"15min":1.498},"last":0.57799999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575035,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50156,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321684,"statfs":{"total":96636764160,"available":96585404416,"internally_reserved":0,"allocated":1015808,"data_stored":503572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38600000000000001,"15min":0.38100000000000001},"min":{"1min":0.26700000000000002,"5min":0.214,"15min":0.21099999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":1.159},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.41199999999999998,"15min":0.40899999999999997},"min":{"1min":0.27500000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.54600000000000004,"5min":0.60899999999999999,"15min":1.075},"last":0.29599999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.41099999999999998,"15min":0.41899999999999998},"min":{"1min":0.29599999999999999,"5min":0.26400000000000001,"15min":0.24299999999999999},"max":{"1min":0.57599999999999996,"5min":0.78500000000000003,"15min":3.6179999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.42299999999999999,"15min":0.42999999999999999},"min":{"1min":0.309,"5min":0.27100000000000002,"15min":0.23599999999999999},"max":{"1min":0.58899999999999997,"5min":0.61399999999999999,"15min":1.125},"last":0.32700000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.46000000000000002,"15min":0.46500000000000002},"min":{"1min":0.34399999999999997,"5min":0.30499999999999999,"15min":0.29699999999999999},"max":{"1min":0.63100000000000001,"5min":0.79200000000000004,"15min":3.661},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.44400000000000001,"15min":0.438},"min":{"1min":0.376,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.55900000000000005,"5min":1.0549999999999999,"15min":1.1910000000000001},"last":0.313}]},{"osd":4,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.61499999999999999,"15min":0.60699999999999998},"min":{"1min":0.45900000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81699999999999995,"5min":0.88200000000000001,"15min":1.2170000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63600000000000001,"15min":0.628},"min":{"1min":0.48799999999999999,"5min":0.316,"15min":0.30499999999999999},"max":{"1min":0.89900000000000002,"5min":0.90200000000000002,"15min":1.101},"last":0.47599999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.64800000000000002,"15min":0.65300000000000002},"min":{"1min":0.57599999999999996,"5min":0.42299999999999999,"15min":0.32900000000000001},"max":{"1min":0.90300000000000002,"5min":1.6870000000000001,"15min":3.6920000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.628,"15min":0.625},"min":{"1min":0.49399999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.81200000000000006,"5min":0.89500000000000002,"15min":1.244},"last":0.49199999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64300000000000002,"15min":0.63400000000000001},"min":{"1min":0.56499999999999995,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.84899999999999998,"5min":0.93400000000000005,"15min":1.012},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67100000000000004,"15min":0.66900000000000004},"min":{"1min":0.51300000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93100000000000005,"5min":1.728,"15min":3.7490000000000001},"last":0.48299999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65400000000000003,"15min":0.64900000000000002},"min":{"1min":0.52900000000000003,"5min":0.40699999999999997,"15min":0.35599999999999998},"max":{"1min":0.80500000000000005,"5min":0.86899999999999999,"15min":1.0469999999999999},"last":0.501},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.67700000000000005,"15min":0.68200000000000005},"min":{"1min":0.54300000000000004,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.90900000000000003,"5min":1.758,"15min":3.7200000000000002},"last":0.56299999999999994}]}]},{"osd":3,"up_from":13,"seq":55834575034,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32120,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339720,"statfs":{"total":96636764160,"available":96603873280,"internally_reserved":0,"allocated":1019904,"data_stored":504591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.24199999999999999,"5min":0.21099999999999999,"15min":0.191},"max":{"1min":0.51700000000000002,"5min":0.59699999999999998,"15min":0.63900000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38100000000000001,"15min":0.377},"min":{"1min":0.308,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.21299999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.372,"15min":0.372},"min":{"1min":0.248,"5min":0.22700000000000001,"15min":0.215},"max":{"1min":0.47099999999999997,"5min":0.59999999999999998,"15min":0.62},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40200000000000002,"15min":0.40799999999999997},"min":{"1min":0.34200000000000003,"5min":0.26200000000000001,"15min":0.247},"max":{"1min":0.55200000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.39200000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.44900000000000001,"15min":0.45000000000000001},"min":{"1min":0.32800000000000001,"5min":0.27200000000000002,"15min":0.25700000000000001},"max":{"1min":0.55300000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.432},{"interface":"front","average":{"1min":0.495,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.39700000000000002,"5min":0.317,"15min":0.29899999999999999},"max":{"1min":0.622,"5min":0.64800000000000002,"15min":0.65700000000000003},"last":0.30199999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.49299999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.95999999999999996},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60999999999999999,"15min":0.61699999999999999},"min":{"1min":0.41899999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.74099999999999999,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.67100000000000004}]},{"osd":5,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.64400000000000002,"15min":0.65500000000000003},"min":{"1min":0.46000000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.91100000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.625},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65300000000000002,"15min":0.65200000000000002},"min":{"1min":0.42999999999999999,"5min":0.38200000000000001,"15min":0.36899999999999999},"max":{"1min":0.94499999999999995,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.65600000000000003}]},{"osd":6,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60799999999999998,"15min":0.61799999999999999},"min":{"1min":0.47799999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.96999999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63500000000000001,"15min":0.64800000000000002},"min":{"1min":0.436,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91800000000000004,"5min":0.92600000000000005,"15min":0.94699999999999995},"last":0.70899999999999996}]},{"osd":7,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.629,"15min":0.63700000000000001},"min":{"1min":0.46899999999999997,"5min":0.40400000000000003,"15min":0.33300000000000002},"max":{"1min":0.873,"5min":0.89000000000000001,"15min":0.93300000000000005},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65100000000000002,"15min":0.65700000000000003},"min":{"1min":0.52200000000000002,"5min":0.39500000000000002,"15min":0.373},"max":{"1min":0.93300000000000005,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.54600000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575034,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":1032192,"data_stored":506155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.47499999999999998,"5min":0.29399999999999998,"15min":0.23899999999999999},"max":{"1min":0.71099999999999997,"5min":0.80800000000000005,"15min":0.82999999999999996},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55200000000000005,"15min":0.55400000000000005},"min":{"1min":0.49099999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.82199999999999995},"last":0.58999999999999997}]},{"osd":1,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.59299999999999997,"15min":0.63500000000000001},"min":{"1min":0.48199999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74399999999999999,"5min":0.76900000000000002,"15min":4.3479999999999999},"last":0.629},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58699999999999997,"15min":0.623},"min":{"1min":0.48399999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77100000000000002,"5min":0.78400000000000003,"15min":4.3319999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.56299999999999994,"15min":0.56599999999999995},"min":{"1min":0.47399999999999998,"5min":0.313,"15min":0.311},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.78400000000000003},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.58099999999999996,"15min":0.58199999999999996},"min":{"1min":0.50900000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72399999999999998,"5min":0.74299999999999999,"15min":0.82199999999999995},"last":0.53000000000000003}]},{"osd":3,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55100000000000005,"15min":0.59199999999999997},"min":{"1min":0.432,"5min":0.36399999999999999,"15min":0.34899999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":4.1950000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.59699999999999998,"15min":0.60199999999999998},"min":{"1min":0.497,"5min":0.40500000000000003,"15min":0.36099999999999999},"max":{"1min":0.73699999999999999,"5min":0.75900000000000001,"15min":1.2330000000000001},"last":0.61299999999999999}]},{"osd":5,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.46500000000000002,"15min":0.48999999999999999},"min":{"1min":0.36199999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71099999999999997,"5min":0.76300000000000001,"15min":1.196},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.497,"15min":0.502},"min":{"1min":0.39400000000000002,"5min":0.33500000000000002,"15min":0.312},"max":{"1min":0.63900000000000001,"5min":0.74299999999999999,"15min":0.77000000000000002},"last":0.39800000000000002}]},{"osd":6,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.498,"15min":0.51100000000000001},"min":{"1min":0.33200000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.64200000000000002,"5min":0.754,"15min":1.022},"last":0.379},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.52500000000000002,"15min":0.54400000000000004},"min":{"1min":0.39500000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67400000000000004,"5min":0.73699999999999999,"15min":0.872},"last":0.49299999999999999}]},{"osd":7,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51200000000000001,"15min":0.54100000000000004},"min":{"1min":0.377,"5min":0.311,"15min":0.311},"max":{"1min":0.68500000000000005,"5min":0.79300000000000004,"15min":0.91500000000000004},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55000000000000004,"15min":0.57699999999999996},"min":{"1min":0.39300000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73599999999999999,"5min":0.752,"15min":1.0589999999999999},"last":0.44400000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575034,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46064,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325776,"statfs":{"total":96636764160,"available":96589594624,"internally_reserved":0,"allocated":1019904,"data_stored":503594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.36799999999999999,"5min":0.32700000000000001,"15min":0.24199999999999999},"max":{"1min":0.75900000000000001,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.29499999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.748,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.61199999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.30399999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55200000000000005,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.34499999999999997,"15min":0.30599999999999999},"max":{"1min":0.73599999999999999,"5min":0.74299999999999999,"15min":0.76200000000000001},"last":0.55800000000000005}]},{"osd":2,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56999999999999995,"15min":0.56200000000000006},"min":{"1min":0.33200000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.72199999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.629},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.40999999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72099999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.496}]},{"osd":3,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.33900000000000002,"5min":0.32500000000000001,"15min":0.31},"max":{"1min":0.77800000000000002,"5min":0.874,"15min":0.874},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.436,"5min":0.377,"15min":0.32300000000000001},"max":{"1min":0.73099999999999998,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.58899999999999997}]},{"osd":4,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.28299999999999997},"max":{"1min":0.755,"5min":0.755,"15min":0.81200000000000006},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54000000000000004,"15min":0.53300000000000003},"min":{"1min":0.42599999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73699999999999999,"5min":0.80000000000000004,"15min":0.91800000000000004},"last":0.503}]},{"osd":6,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49099999999999999,"15min":0.48599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.28000000000000003},"max":{"1min":0.63800000000000001,"5min":0.73899999999999999,"15min":0.88700000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.496,"15min":0.49299999999999999},"min":{"1min":0.33100000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64800000000000002,"5min":0.73199999999999998,"15min":0.83299999999999996},"last":0.47399999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.252},"max":{"1min":0.68999999999999995,"5min":0.78600000000000003,"15min":0.79100000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60999999999999999,"15min":0.60199999999999998},"min":{"1min":0.44800000000000001,"5min":0.41399999999999998,"15min":0.35199999999999998},"max":{"1min":0.72499999999999998,"5min":0.85299999999999998,"15min":0.85599999999999998},"last":0.54400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:27:29.744 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:27:29.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.880+0000 7f3b2b75f700 1 Processor -- start 2024-04-06T12:27:29.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.880+0000 7f3b2b75f700 1 -- start start 2024-04-06T12:27:29.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.881+0000 7f3b2b75f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3b2415d7b0 0x7f3b240b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:29.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.881+0000 7f3b2b75f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b240b3990 con 0x7f3b2406a730 2024-04-06T12:27:29.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.881+0000 7f3b2b75f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b240b3060 con 0x7f3b2415e500 2024-04-06T12:27:29.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.881+0000 7f3b2b75f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2415e950 con 0x7f3b2415d7b0 2024-04-06T12:27:29.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b28cfa700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3b2406a730 0x7f3b2415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52604/0 (socket says 172.21.15.189:52604) 2024-04-06T12:27:29.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b28cfa700 1 -- 172.21.15.189:0/3941053044 learned_addr learned my addr 172.21.15.189:0/3941053044 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:29.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b294fb700 1 --2- 172.21.15.189:0/3941053044 >> v2:172.21.15.192:3300/0 conn(0x7f3b2415d7b0 0x7f3b240b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:29.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b1bfff700 1 -- 172.21.15.189:0/3941053044 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022467721 0 0) 0x7f3b240b3990 con 0x7f3b2406a730 2024-04-06T12:27:29.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b1bfff700 1 -- 172.21.15.189:0/3941053044 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b100035b0 con 0x7f3b2406a730 2024-04-06T12:27:29.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b1bfff700 1 -- 172.21.15.189:0/3941053044 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3063687829 0 0) 0x7f3b100035b0 con 0x7f3b2406a730 2024-04-06T12:27:29.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b1bfff700 1 -- 172.21.15.189:0/3941053044 >> v2:172.21.15.192:3300/0 conn(0x7f3b2415d7b0 msgr2=0x7f3b240b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:29.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b1bfff700 1 --2- 172.21.15.189:0/3941053044 >> v2:172.21.15.192:3300/0 conn(0x7f3b2415d7b0 0x7f3b240b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:29.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b1bfff700 1 -- 172.21.15.189:0/3941053044 >> v1:172.21.15.192:6789/0 conn(0x7f3b2415e500 legacy=0x7f3b240717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:29.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b1bfff700 1 -- 172.21.15.189:0/3941053044 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b240b3340 con 0x7f3b2406a730 2024-04-06T12:27:29.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b1bfff700 1 -- 172.21.15.189:0/3941053044 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f3b20001670 con 0x7f3b2406a730 2024-04-06T12:27:29.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b1bfff700 1 -- 172.21.15.189:0/3941053044 >> v1:172.21.15.189:6789/0 conn(0x7f3b2406a730 legacy=0x7f3b2415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:29.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b1bfff700 1 --2- 172.21.15.189:0/3941053044 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b10003a00 0x7f3b10003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:29.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b1bfff700 1 --2- 172.21.15.189:0/3941053044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b10004850 0x7f3b10004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:29.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b1bfff700 1 --2- 172.21.15.189:0/3941053044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b10005210 0x7f3b1000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:29.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b1bfff700 1 -- 172.21.15.189:0/3941053044 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1000c180 con 0x7f3b10004850 2024-04-06T12:27:29.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b1bfff700 1 -- 172.21.15.189:0/3941053044 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1000c2f0 con 0x7f3b10005210 2024-04-06T12:27:29.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b1bfff700 1 -- 172.21.15.189:0/3941053044 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1000c4b0 con 0x7f3b10003a00 2024-04-06T12:27:29.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b28cfa700 1 --2- 172.21.15.189:0/3941053044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b10004850 0x7f3b10004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:29.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b294fb700 1 --2- 172.21.15.189:0/3941053044 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b10003a00 0x7f3b10003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:29.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.882+0000 7f3b29cfc700 1 --2- 172.21.15.189:0/3941053044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b10005210 0x7f3b1000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:29.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.883+0000 7f3b28cfa700 1 -- 172.21.15.189:0/3941053044 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b10003a00 msgr2=0x7f3b10003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:29.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.883+0000 7f3b28cfa700 1 --2- 172.21.15.189:0/3941053044 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b10003a00 0x7f3b10003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:29.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.883+0000 7f3b28cfa700 1 -- 172.21.15.189:0/3941053044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b10005210 msgr2=0x7f3b1000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:29.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.883+0000 7f3b28cfa700 1 --2- 172.21.15.189:0/3941053044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b10005210 0x7f3b1000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:29.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.883+0000 7f3b28cfa700 1 -- 172.21.15.189:0/3941053044 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b100035b0 con 0x7f3b10004850 2024-04-06T12:27:29.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.883+0000 7f3b294fb700 1 --2- 172.21.15.189:0/3941053044 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b10003a00 0x7f3b10003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:29.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.883+0000 7f3b29cfc700 1 --2- 172.21.15.189:0/3941053044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b10005210 0x7f3b1000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:29.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.883+0000 7f3b29cfc700 1 -- 172.21.15.189:0/3941053044 reap_dead start 2024-04-06T12:27:29.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.883+0000 7f3b28cfa700 1 --2- 172.21.15.189:0/3941053044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b10004850 0x7f3b10004cd0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000c240 tx=0x7f3b10005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2367dccae20219a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:29.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.883+0000 7f3b1bfff700 1 -- 172.21.15.189:0/3941053044 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20007c10 con 0x7f3b10004850 2024-04-06T12:27:29.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.883+0000 7f3b1bfff700 1 -- 172.21.15.189:0/3941053044 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2000ce00 con 0x7f3b10004850 2024-04-06T12:27:29.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.883+0000 7f3b1bfff700 1 -- 172.21.15.189:0/3941053044 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20005ba0 con 0x7f3b10004850 2024-04-06T12:27:29.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.883+0000 7f3b2b75f700 1 -- 172.21.15.189:0/3941053044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b10004850 msgr2=0x7f3b10004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:29.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.883+0000 7f3b2b75f700 1 --2- 172.21.15.189:0/3941053044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b10004850 0x7f3b10004cd0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000c240 tx=0x7f3b10005880 comp rx=0 tx=0).stop 2024-04-06T12:27:29.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.883+0000 7f3b2b75f700 1 -- 172.21.15.189:0/3941053044 shutdown_connections 2024-04-06T12:27:29.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.883+0000 7f3b2b75f700 1 --2- 172.21.15.189:0/3941053044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b10004850 0x7f3b10004cd0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:29.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.883+0000 7f3b2b75f700 1 -- 172.21.15.189:0/3941053044 >> 172.21.15.189:0/3941053044 conn(0x7f3b240d00b0 msgr2=0x7f3b240d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:29.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.884+0000 7f3b2b75f700 1 -- 172.21.15.189:0/3941053044 shutdown_connections 2024-04-06T12:27:29.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.884+0000 7f3b2b75f700 1 -- 172.21.15.189:0/3941053044 wait complete. 2024-04-06T12:27:29.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.885+0000 7f3b2b75f700 1 Processor -- start 2024-04-06T12:27:29.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.885+0000 7f3b2b75f700 1 -- start start 2024-04-06T12:27:29.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.885+0000 7f3b2b75f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b10004850 0x7f3b24160930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:29.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.885+0000 7f3b2b75f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b2415a470 0x7f3b240ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:29.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.885+0000 7f3b2b75f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b24151bf0 0x7f3b241f6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:29.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.885+0000 7f3b2b75f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240cd440 con 0x7f3b10004850 2024-04-06T12:27:29.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.885+0000 7f3b2b75f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240afd70 con 0x7f3b2415a470 2024-04-06T12:27:29.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.885+0000 7f3b2b75f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2415a8f0 con 0x7f3b24151bf0 2024-04-06T12:27:29.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.886+0000 7f3b294fb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b10004850 0x7f3b24160930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:29.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.886+0000 7f3b294fb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b10004850 0x7f3b24160930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33020/0 (socket says 172.21.15.189:33020) 2024-04-06T12:27:29.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.886+0000 7f3b294fb700 1 -- 172.21.15.189:0/3887158456 learned_addr learned my addr 172.21.15.189:0/3887158456 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:29.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.886+0000 7f3b294fb700 1 -- 172.21.15.189:0/3887158456 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b24151bf0 msgr2=0x7f3b241f6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:29.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.886+0000 7f3b294fb700 1 --2- 172.21.15.189:0/3887158456 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b24151bf0 0x7f3b241f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:29.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.886+0000 7f3b28cfa700 1 --2- 172.21.15.189:0/3887158456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b2415a470 0x7f3b240ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:29.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.886+0000 7f3b294fb700 1 -- 172.21.15.189:0/3887158456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b2415a470 msgr2=0x7f3b240ccf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:29.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.886+0000 7f3b294fb700 1 --2- 172.21.15.189:0/3887158456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b2415a470 0x7f3b240ccf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:29.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.886+0000 7f3b294fb700 1 -- 172.21.15.189:0/3887158456 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b240cd680 con 0x7f3b10004850 2024-04-06T12:27:29.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.886+0000 7f3b28cfa700 1 --2- 172.21.15.189:0/3887158456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b2415a470 0x7f3b240ccf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:27:29.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.886+0000 7f3b294fb700 1 --2- 172.21.15.189:0/3887158456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b10004850 0x7f3b24160930 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f3b14011590 tx=0x7f3b14011950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:29.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.886+0000 7f3b19ffb700 1 -- 172.21.15.189:0/3887158456 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14004220 con 0x7f3b10004850 2024-04-06T12:27:29.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.886+0000 7f3b19ffb700 1 -- 172.21.15.189:0/3887158456 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b14004380 con 0x7f3b10004850 2024-04-06T12:27:29.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.886+0000 7f3b19ffb700 1 -- 172.21.15.189:0/3887158456 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b140036a0 con 0x7f3b10004850 2024-04-06T12:27:29.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.886+0000 7f3b2b75f700 1 -- 172.21.15.189:0/3887158456 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b240affd0 con 0x7f3b10004850 2024-04-06T12:27:29.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.886+0000 7f3b2b75f700 1 -- 172.21.15.189:0/3887158456 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b240b0590 con 0x7f3b10004850 2024-04-06T12:27:29.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.887+0000 7f3b2b75f700 1 -- 172.21.15.189:0/3887158456 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b24067bb0 con 0x7f3b10004850 2024-04-06T12:27:29.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.888+0000 7f3b19ffb700 1 -- 172.21.15.189:0/3887158456 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f3b14003800 con 0x7f3b10004850 2024-04-06T12:27:29.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.888+0000 7f3b19ffb700 1 --2- 172.21.15.189:0/3887158456 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3b0806adb0 0x7f3b0806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:29.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.888+0000 7f3b19ffb700 1 -- 172.21.15.189:0/3887158456 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3b14089930 con 0x7f3b10004850 2024-04-06T12:27:29.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.888+0000 7f3b28cfa700 1 --2- 172.21.15.189:0/3887158456 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3b0806adb0 0x7f3b0806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:29.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.889+0000 7f3b28cfa700 1 --2- 172.21.15.189:0/3887158456 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3b0806adb0 0x7f3b0806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3b20015040 tx=0x7f3b20007000 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:29.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:29.893+0000 7f3b19ffb700 1 -- 172.21.15.189:0/3887158456 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3b140548c0 con 0x7f3b10004850 2024-04-06T12:27:30.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.083+0000 7f3b2b75f700 1 -- 172.21.15.189:0/3887158456 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b2415eb40 con 0x7f3b0806adb0 2024-04-06T12:27:30.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.091+0000 7f3b19ffb700 1 -- 172.21.15.189:0/3887158456 <== mgr.5582 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289297 (secure 0 0 0) 0x7f3b20007bb0 con 0x7f3b0806adb0 2024-04-06T12:27:30.093 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:30.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.094+0000 7f3b2b75f700 1 -- 172.21.15.189:0/3887158456 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3b0806adb0 msgr2=0x7f3b0806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:30.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.094+0000 7f3b2b75f700 1 --2- 172.21.15.189:0/3887158456 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3b0806adb0 0x7f3b0806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3b20015040 tx=0x7f3b20007000 comp rx=0 tx=0).stop 2024-04-06T12:27:30.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.094+0000 7f3b2b75f700 1 -- 172.21.15.189:0/3887158456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b10004850 msgr2=0x7f3b24160930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:30.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.094+0000 7f3b2b75f700 1 --2- 172.21.15.189:0/3887158456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b10004850 0x7f3b24160930 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f3b14011590 tx=0x7f3b14011950 comp rx=0 tx=0).stop 2024-04-06T12:27:30.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.095+0000 7f3b2b75f700 1 -- 172.21.15.189:0/3887158456 shutdown_connections 2024-04-06T12:27:30.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.095+0000 7f3b2b75f700 1 --2- 172.21.15.189:0/3887158456 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b24151bf0 0x7f3b241f6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.095+0000 7f3b2b75f700 1 --2- 172.21.15.189:0/3887158456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b10004850 0x7f3b24160930 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.095+0000 7f3b2b75f700 1 --2- 172.21.15.189:0/3887158456 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3b0806adb0 0x7f3b0806d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.095+0000 7f3b2b75f700 1 --2- 172.21.15.189:0/3887158456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b2415a470 0x7f3b240ccf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.095+0000 7f3b2b75f700 1 -- 172.21.15.189:0/3887158456 >> 172.21.15.189:0/3887158456 conn(0x7f3b240d00b0 msgr2=0x7f3b240717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:30.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.095+0000 7f3b2b75f700 1 -- 172.21.15.189:0/3887158456 shutdown_connections 2024-04-06T12:27:30.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.095+0000 7f3b2b75f700 1 -- 172.21.15.189:0/3887158456 wait complete. 2024-04-06T12:27:30.096 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:27:30.105 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-04-06T12:27:29.310322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":284,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":200,"ondisk_log_size":200,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":159,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":316356,"kb_used_data":9172,"kb_used_omap":12,"kb_used_meta":307059,"kb_avail":754658364,"statfs":{"total":773094113280,"available":772770164736,"internally_reserved":0,"allocated":9392128,"data_stored":5217093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":314428984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.277130"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046604+0000","last_change":"2024-04-06T12:27:21.054450+0000","last_active":"2024-04-06T12:27:22.046604+0000","last_peered":"2024-04-06T12:27:22.046604+0000","last_clean":"2024-04-06T12:27:22.046604+0000","last_became_active":"2024-04-06T12:27:21.054202+0000","last_became_peered":"2024-04-06T12:27:21.054202+0000","last_unstale":"2024-04-06T12:27:22.046604+0000","last_undegraded":"2024-04-06T12:27:22.046604+0000","last_fullsized":"2024-04-06T12:27:22.046604+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:29:26.946143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046360+0000","last_change":"2024-04-06T12:27:21.053119+0000","last_active":"2024-04-06T12:27:22.046360+0000","last_peered":"2024-04-06T12:27:22.046360+0000","last_clean":"2024-04-06T12:27:22.046360+0000","last_became_active":"2024-04-06T12:27:21.052814+0000","last_became_peered":"2024-04-06T12:27:21.052814+0000","last_unstale":"2024-04-06T12:27:22.046360+0000","last_undegraded":"2024-04-06T12:27:22.046360+0000","last_fullsized":"2024-04-06T12:27:22.046360+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:14.746095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039737+0000","last_change":"2024-04-06T12:27:21.051098+0000","last_active":"2024-04-06T12:27:22.039737+0000","last_peered":"2024-04-06T12:27:22.039737+0000","last_clean":"2024-04-06T12:27:22.039737+0000","last_became_active":"2024-04-06T12:27:21.050787+0000","last_became_peered":"2024-04-06T12:27:21.050787+0000","last_unstale":"2024-04-06T12:27:22.039737+0000","last_undegraded":"2024-04-06T12:27:22.039737+0000","last_fullsized":"2024-04-06T12:27:22.039737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:57.645194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048036+0000","last_change":"2024-04-06T12:27:21.048815+0000","last_active":"2024-04-06T12:27:22.048036+0000","last_peered":"2024-04-06T12:27:22.048036+0000","last_clean":"2024-04-06T12:27:22.048036+0000","last_became_active":"2024-04-06T12:27:21.047390+0000","last_became_peered":"2024-04-06T12:27:21.047390+0000","last_unstale":"2024-04-06T12:27:22.048036+0000","last_undegraded":"2024-04-06T12:27:22.048036+0000","last_fullsized":"2024-04-06T12:27:22.048036+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:53:37.857841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047263+0000","last_change":"2024-04-06T12:27:21.064877+0000","last_active":"2024-04-06T12:27:22.047263+0000","last_peered":"2024-04-06T12:27:22.047263+0000","last_clean":"2024-04-06T12:27:22.047263+0000","last_became_active":"2024-04-06T12:27:21.064649+0000","last_became_peered":"2024-04-06T12:27:21.064649+0000","last_unstale":"2024-04-06T12:27:22.047263+0000","last_undegraded":"2024-04-06T12:27:22.047263+0000","last_fullsized":"2024-04-06T12:27:22.047263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:25.859663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.040075+0000","last_change":"2024-04-06T12:27:21.054089+0000","last_active":"2024-04-06T12:27:22.040075+0000","last_peered":"2024-04-06T12:27:22.040075+0000","last_clean":"2024-04-06T12:27:22.040075+0000","last_became_active":"2024-04-06T12:27:21.053844+0000","last_became_peered":"2024-04-06T12:27:21.053844+0000","last_unstale":"2024-04-06T12:27:22.040075+0000","last_undegraded":"2024-04-06T12:27:22.040075+0000","last_fullsized":"2024-04-06T12:27:22.040075+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:39:49.798363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046048+0000","last_change":"2024-04-06T12:27:21.060315+0000","last_active":"2024-04-06T12:27:22.046048+0000","last_peered":"2024-04-06T12:27:22.046048+0000","last_clean":"2024-04-06T12:27:22.046048+0000","last_became_active":"2024-04-06T12:27:21.060097+0000","last_became_peered":"2024-04-06T12:27:21.060097+0000","last_unstale":"2024-04-06T12:27:22.046048+0000","last_undegraded":"2024-04-06T12:27:22.046048+0000","last_fullsized":"2024-04-06T12:27:22.046048+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:53:09.701399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046794+0000","last_change":"2024-04-06T12:27:21.053746+0000","last_active":"2024-04-06T12:27:22.046794+0000","last_peered":"2024-04-06T12:27:22.046794+0000","last_clean":"2024-04-06T12:27:22.046794+0000","last_became_active":"2024-04-06T12:27:21.053476+0000","last_became_peered":"2024-04-06T12:27:21.053476+0000","last_unstale":"2024-04-06T12:27:22.046794+0000","last_undegraded":"2024-04-06T12:27:22.046794+0000","last_fullsized":"2024-04-06T12:27:22.046794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:59:57.937757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.040067+0000","last_change":"2024-04-06T12:27:21.055688+0000","last_active":"2024-04-06T12:27:22.040067+0000","last_peered":"2024-04-06T12:27:22.040067+0000","last_clean":"2024-04-06T12:27:22.040067+0000","last_became_active":"2024-04-06T12:27:21.055417+0000","last_became_peered":"2024-04-06T12:27:21.055417+0000","last_unstale":"2024-04-06T12:27:22.040067+0000","last_undegraded":"2024-04-06T12:27:22.040067+0000","last_fullsized":"2024-04-06T12:27:22.040067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:12:22.930768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039825+0000","last_change":"2024-04-06T12:27:21.056862+0000","last_active":"2024-04-06T12:27:22.039825+0000","last_peered":"2024-04-06T12:27:22.039825+0000","last_clean":"2024-04-06T12:27:22.039825+0000","last_became_active":"2024-04-06T12:27:21.056602+0000","last_became_peered":"2024-04-06T12:27:21.056602+0000","last_unstale":"2024-04-06T12:27:22.039825+0000","last_undegraded":"2024-04-06T12:27:22.039825+0000","last_fullsized":"2024-04-06T12:27:22.039825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:41:35.413608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047029+0000","last_change":"2024-04-06T12:27:21.051431+0000","last_active":"2024-04-06T12:27:22.047029+0000","last_peered":"2024-04-06T12:27:22.047029+0000","last_clean":"2024-04-06T12:27:22.047029+0000","last_became_active":"2024-04-06T12:27:21.050731+0000","last_became_peered":"2024-04-06T12:27:21.050731+0000","last_unstale":"2024-04-06T12:27:22.047029+0000","last_undegraded":"2024-04-06T12:27:22.047029+0000","last_fullsized":"2024-04-06T12:27:22.047029+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:26:12.757657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047263+0000","last_change":"2024-04-06T12:27:21.056611+0000","last_active":"2024-04-06T12:27:22.047263+0000","last_peered":"2024-04-06T12:27:22.047263+0000","last_clean":"2024-04-06T12:27:22.047263+0000","last_became_active":"2024-04-06T12:27:21.056361+0000","last_became_peered":"2024-04-06T12:27:21.056361+0000","last_unstale":"2024-04-06T12:27:22.047263+0000","last_undegraded":"2024-04-06T12:27:22.047263+0000","last_fullsized":"2024-04-06T12:27:22.047263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:41:13.712569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047557+0000","last_change":"2024-04-06T12:27:21.053678+0000","last_active":"2024-04-06T12:27:22.047557+0000","last_peered":"2024-04-06T12:27:22.047557+0000","last_clean":"2024-04-06T12:27:22.047557+0000","last_became_active":"2024-04-06T12:27:21.053132+0000","last_became_peered":"2024-04-06T12:27:21.053132+0000","last_unstale":"2024-04-06T12:27:22.047557+0000","last_undegraded":"2024-04-06T12:27:22.047557+0000","last_fullsized":"2024-04-06T12:27:22.047557+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:57.645194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039708+0000","last_change":"2024-04-06T12:27:21.052837+0000","last_active":"2024-04-06T12:27:22.039708+0000","last_peered":"2024-04-06T12:27:22.039708+0000","last_clean":"2024-04-06T12:27:22.039708+0000","last_became_active":"2024-04-06T12:27:21.052576+0000","last_became_peered":"2024-04-06T12:27:21.052576+0000","last_unstale":"2024-04-06T12:27:22.039708+0000","last_undegraded":"2024-04-06T12:27:22.039708+0000","last_fullsized":"2024-04-06T12:27:22.039708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:59:57.937757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039511+0000","last_change":"2024-04-06T12:27:21.058953+0000","last_active":"2024-04-06T12:27:22.039511+0000","last_peered":"2024-04-06T12:27:22.039511+0000","last_clean":"2024-04-06T12:27:22.039511+0000","last_became_active":"2024-04-06T12:27:21.058509+0000","last_became_peered":"2024-04-06T12:27:21.058509+0000","last_unstale":"2024-04-06T12:27:22.039511+0000","last_undegraded":"2024-04-06T12:27:22.039511+0000","last_fullsized":"2024-04-06T12:27:22.039511+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:08:32.179469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046311+0000","last_change":"2024-04-06T12:27:21.042215+0000","last_active":"2024-04-06T12:27:22.046311+0000","last_peered":"2024-04-06T12:27:22.046311+0000","last_clean":"2024-04-06T12:27:22.046311+0000","last_became_active":"2024-04-06T12:27:21.041921+0000","last_became_peered":"2024-04-06T12:27:21.041921+0000","last_unstale":"2024-04-06T12:27:22.046311+0000","last_undegraded":"2024-04-06T12:27:22.046311+0000","last_fullsized":"2024-04-06T12:27:22.046311+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:33:26.404172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046752+0000","last_change":"2024-04-06T12:27:21.056018+0000","last_active":"2024-04-06T12:27:22.046752+0000","last_peered":"2024-04-06T12:27:22.046752+0000","last_clean":"2024-04-06T12:27:22.046752+0000","last_became_active":"2024-04-06T12:27:21.055764+0000","last_became_peered":"2024-04-06T12:27:21.055764+0000","last_unstale":"2024-04-06T12:27:22.046752+0000","last_undegraded":"2024-04-06T12:27:22.046752+0000","last_fullsized":"2024-04-06T12:27:22.046752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:49:34.397313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047499+0000","last_change":"2024-04-06T12:27:21.052386+0000","last_active":"2024-04-06T12:27:22.047499+0000","last_peered":"2024-04-06T12:27:22.047499+0000","last_clean":"2024-04-06T12:27:22.047499+0000","last_became_active":"2024-04-06T12:27:21.052081+0000","last_became_peered":"2024-04-06T12:27:21.052081+0000","last_unstale":"2024-04-06T12:27:22.047499+0000","last_undegraded":"2024-04-06T12:27:22.047499+0000","last_fullsized":"2024-04-06T12:27:22.047499+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:41:13.712569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047792+0000","last_change":"2024-04-06T12:27:21.059175+0000","last_active":"2024-04-06T12:27:22.047792+0000","last_peered":"2024-04-06T12:27:22.047792+0000","last_clean":"2024-04-06T12:27:22.047792+0000","last_became_active":"2024-04-06T12:27:21.058651+0000","last_became_peered":"2024-04-06T12:27:21.058651+0000","last_unstale":"2024-04-06T12:27:22.047792+0000","last_undegraded":"2024-04-06T12:27:22.047792+0000","last_fullsized":"2024-04-06T12:27:22.047792+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:45.939534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039097+0000","last_change":"2024-04-06T12:27:21.062933+0000","last_active":"2024-04-06T12:27:22.039097+0000","last_peered":"2024-04-06T12:27:22.039097+0000","last_clean":"2024-04-06T12:27:22.039097+0000","last_became_active":"2024-04-06T12:27:21.062688+0000","last_became_peered":"2024-04-06T12:27:21.062688+0000","last_unstale":"2024-04-06T12:27:22.039097+0000","last_undegraded":"2024-04-06T12:27:22.039097+0000","last_fullsized":"2024-04-06T12:27:22.039097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:30:58.878304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047660+0000","last_change":"2024-04-06T12:27:21.059206+0000","last_active":"2024-04-06T12:27:22.047660+0000","last_peered":"2024-04-06T12:27:22.047660+0000","last_clean":"2024-04-06T12:27:22.047660+0000","last_became_active":"2024-04-06T12:27:21.058932+0000","last_became_peered":"2024-04-06T12:27:21.058932+0000","last_unstale":"2024-04-06T12:27:22.047660+0000","last_undegraded":"2024-04-06T12:27:22.047660+0000","last_fullsized":"2024-04-06T12:27:22.047660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:08:32.179469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.038878+0000","last_change":"2024-04-06T12:27:21.053594+0000","last_active":"2024-04-06T12:27:22.038878+0000","last_peered":"2024-04-06T12:27:22.038878+0000","last_clean":"2024-04-06T12:27:22.038878+0000","last_became_active":"2024-04-06T12:27:21.053330+0000","last_became_peered":"2024-04-06T12:27:21.053330+0000","last_unstale":"2024-04-06T12:27:22.038878+0000","last_undegraded":"2024-04-06T12:27:22.038878+0000","last_fullsized":"2024-04-06T12:27:22.038878+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:54:27.984787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.038730+0000","last_change":"2024-04-06T12:27:21.053574+0000","last_active":"2024-04-06T12:27:22.038730+0000","last_peered":"2024-04-06T12:27:22.038730+0000","last_clean":"2024-04-06T12:27:22.038730+0000","last_became_active":"2024-04-06T12:27:21.053289+0000","last_became_peered":"2024-04-06T12:27:21.053289+0000","last_unstale":"2024-04-06T12:27:22.038730+0000","last_undegraded":"2024-04-06T12:27:22.038730+0000","last_fullsized":"2024-04-06T12:27:22.038730+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:44.646361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039397+0000","last_change":"2024-04-06T12:27:21.053031+0000","last_active":"2024-04-06T12:27:22.039397+0000","last_peered":"2024-04-06T12:27:22.039397+0000","last_clean":"2024-04-06T12:27:22.039397+0000","last_became_active":"2024-04-06T12:27:21.051418+0000","last_became_peered":"2024-04-06T12:27:21.051418+0000","last_unstale":"2024-04-06T12:27:22.039397+0000","last_undegraded":"2024-04-06T12:27:22.039397+0000","last_fullsized":"2024-04-06T12:27:22.039397+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:24:04.575202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048049+0000","last_change":"2024-04-06T12:27:21.049364+0000","last_active":"2024-04-06T12:27:22.048049+0000","last_peered":"2024-04-06T12:27:22.048049+0000","last_clean":"2024-04-06T12:27:22.048049+0000","last_became_active":"2024-04-06T12:27:21.048571+0000","last_became_peered":"2024-04-06T12:27:21.048571+0000","last_unstale":"2024-04-06T12:27:22.048049+0000","last_undegraded":"2024-04-06T12:27:22.048049+0000","last_fullsized":"2024-04-06T12:27:22.048049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:24:04.575202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048237+0000","last_change":"2024-04-06T12:27:21.048886+0000","last_active":"2024-04-06T12:27:22.048237+0000","last_peered":"2024-04-06T12:27:22.048237+0000","last_clean":"2024-04-06T12:27:22.048237+0000","last_became_active":"2024-04-06T12:27:21.048635+0000","last_became_peered":"2024-04-06T12:27:21.048635+0000","last_unstale":"2024-04-06T12:27:22.048237+0000","last_undegraded":"2024-04-06T12:27:22.048237+0000","last_fullsized":"2024-04-06T12:27:22.048237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:14.746095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047147+0000","last_change":"2024-04-06T12:27:21.045989+0000","last_active":"2024-04-06T12:27:22.047147+0000","last_peered":"2024-04-06T12:27:22.047147+0000","last_clean":"2024-04-06T12:27:22.047147+0000","last_became_active":"2024-04-06T12:27:21.045733+0000","last_became_peered":"2024-04-06T12:27:21.045733+0000","last_unstale":"2024-04-06T12:27:22.047147+0000","last_undegraded":"2024-04-06T12:27:22.047147+0000","last_fullsized":"2024-04-06T12:27:22.047147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:53:59.500416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046223+0000","last_change":"2024-04-06T12:27:21.063222+0000","last_active":"2024-04-06T12:27:22.046223+0000","last_peered":"2024-04-06T12:27:22.046223+0000","last_clean":"2024-04-06T12:27:22.046223+0000","last_became_active":"2024-04-06T12:27:21.062971+0000","last_became_peered":"2024-04-06T12:27:21.062971+0000","last_unstale":"2024-04-06T12:27:22.046223+0000","last_undegraded":"2024-04-06T12:27:22.046223+0000","last_fullsized":"2024-04-06T12:27:22.046223+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:41:35.413608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039459+0000","last_change":"2024-04-06T12:27:21.053664+0000","last_active":"2024-04-06T12:27:22.039459+0000","last_peered":"2024-04-06T12:27:22.039459+0000","last_clean":"2024-04-06T12:27:22.039459+0000","last_became_active":"2024-04-06T12:27:21.053408+0000","last_became_peered":"2024-04-06T12:27:21.053408+0000","last_unstale":"2024-04-06T12:27:22.039459+0000","last_undegraded":"2024-04-06T12:27:22.039459+0000","last_fullsized":"2024-04-06T12:27:22.039459+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:49:34.397313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046965+0000","last_change":"2024-04-06T12:27:21.057755+0000","last_active":"2024-04-06T12:27:22.046965+0000","last_peered":"2024-04-06T12:27:22.046965+0000","last_clean":"2024-04-06T12:27:22.046965+0000","last_became_active":"2024-04-06T12:27:21.057491+0000","last_became_peered":"2024-04-06T12:27:21.057491+0000","last_unstale":"2024-04-06T12:27:22.046965+0000","last_undegraded":"2024-04-06T12:27:22.046965+0000","last_fullsized":"2024-04-06T12:27:22.046965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:54:23.429749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046895+0000","last_change":"2024-04-06T12:27:21.048114+0000","last_active":"2024-04-06T12:27:22.046895+0000","last_peered":"2024-04-06T12:27:22.046895+0000","last_clean":"2024-04-06T12:27:22.046895+0000","last_became_active":"2024-04-06T12:27:21.047854+0000","last_became_peered":"2024-04-06T12:27:21.047854+0000","last_unstale":"2024-04-06T12:27:22.046895+0000","last_undegraded":"2024-04-06T12:27:22.046895+0000","last_fullsized":"2024-04-06T12:27:22.046895+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:02.552663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039826+0000","last_change":"2024-04-06T12:27:21.054604+0000","last_active":"2024-04-06T12:27:22.039826+0000","last_peered":"2024-04-06T12:27:22.039826+0000","last_clean":"2024-04-06T12:27:22.039826+0000","last_became_active":"2024-04-06T12:27:21.049845+0000","last_became_peered":"2024-04-06T12:27:21.049845+0000","last_unstale":"2024-04-06T12:27:22.039826+0000","last_undegraded":"2024-04-06T12:27:22.039826+0000","last_fullsized":"2024-04-06T12:27:22.039826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:26.827213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047368+0000","last_change":"2024-04-06T12:27:21.064380+0000","last_active":"2024-04-06T12:27:22.047368+0000","last_peered":"2024-04-06T12:27:22.047368+0000","last_clean":"2024-04-06T12:27:22.047368+0000","last_became_active":"2024-04-06T12:27:21.064115+0000","last_became_peered":"2024-04-06T12:27:21.064115+0000","last_unstale":"2024-04-06T12:27:22.047368+0000","last_undegraded":"2024-04-06T12:27:22.047368+0000","last_fullsized":"2024-04-06T12:27:22.047368+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:12:18.899208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.065495+0000","last_change":"2024-04-06T12:27:20.639851+0000","last_active":"2024-04-06T12:27:21.065495+0000","last_peered":"2024-04-06T12:27:21.065495+0000","last_clean":"2024-04-06T12:27:21.065495+0000","last_became_active":"2024-04-06T12:27:20.639619+0000","last_became_peered":"2024-04-06T12:27:20.639619+0000","last_unstale":"2024-04-06T12:27:21.065495+0000","last_undegraded":"2024-04-06T12:27:21.065495+0000","last_fullsized":"2024-04-06T12:27:21.065495+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:24.851744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047260+0000","last_change":"2024-04-06T12:27:21.063852+0000","last_active":"2024-04-06T12:27:22.047260+0000","last_peered":"2024-04-06T12:27:22.047260+0000","last_clean":"2024-04-06T12:27:22.047260+0000","last_became_active":"2024-04-06T12:27:21.063593+0000","last_became_peered":"2024-04-06T12:27:21.063593+0000","last_unstale":"2024-04-06T12:27:22.047260+0000","last_undegraded":"2024-04-06T12:27:22.047260+0000","last_fullsized":"2024-04-06T12:27:22.047260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:35:07.924741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.065665+0000","last_change":"2024-04-06T12:27:20.640009+0000","last_active":"2024-04-06T12:27:21.065665+0000","last_peered":"2024-04-06T12:27:21.065665+0000","last_clean":"2024-04-06T12:27:21.065665+0000","last_became_active":"2024-04-06T12:27:20.639751+0000","last_became_peered":"2024-04-06T12:27:20.639751+0000","last_unstale":"2024-04-06T12:27:21.065665+0000","last_undegraded":"2024-04-06T12:27:21.065665+0000","last_fullsized":"2024-04-06T12:27:21.065665+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:23:10.897604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.040256+0000","last_change":"2024-04-06T12:27:21.050994+0000","last_active":"2024-04-06T12:27:22.040256+0000","last_peered":"2024-04-06T12:27:22.040256+0000","last_clean":"2024-04-06T12:27:22.040256+0000","last_became_active":"2024-04-06T12:27:21.049716+0000","last_became_peered":"2024-04-06T12:27:21.049716+0000","last_unstale":"2024-04-06T12:27:22.040256+0000","last_undegraded":"2024-04-06T12:27:22.040256+0000","last_fullsized":"2024-04-06T12:27:22.040256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:53:03.356300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.059811+0000","last_change":"2024-04-06T12:27:20.644294+0000","last_active":"2024-04-06T12:27:21.059811+0000","last_peered":"2024-04-06T12:27:21.059811+0000","last_clean":"2024-04-06T12:27:21.059811+0000","last_became_active":"2024-04-06T12:27:20.643994+0000","last_became_peered":"2024-04-06T12:27:20.643994+0000","last_unstale":"2024-04-06T12:27:21.059811+0000","last_undegraded":"2024-04-06T12:27:21.059811+0000","last_fullsized":"2024-04-06T12:27:21.059811+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:44:45.550106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.040155+0000","last_change":"2024-04-06T12:27:21.057123+0000","last_active":"2024-04-06T12:27:22.040155+0000","last_peered":"2024-04-06T12:27:22.040155+0000","last_clean":"2024-04-06T12:27:22.040155+0000","last_became_active":"2024-04-06T12:27:21.056905+0000","last_became_peered":"2024-04-06T12:27:21.056905+0000","last_unstale":"2024-04-06T12:27:22.040155+0000","last_undegraded":"2024-04-06T12:27:22.040155+0000","last_fullsized":"2024-04-06T12:27:22.040155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:29:26.946143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.060664+0000","last_change":"2024-04-06T12:27:20.636807+0000","last_active":"2024-04-06T12:27:21.060664+0000","last_peered":"2024-04-06T12:27:21.060664+0000","last_clean":"2024-04-06T12:27:21.060664+0000","last_became_active":"2024-04-06T12:27:20.636516+0000","last_became_peered":"2024-04-06T12:27:20.636516+0000","last_unstale":"2024-04-06T12:27:21.060664+0000","last_undegraded":"2024-04-06T12:27:21.060664+0000","last_fullsized":"2024-04-06T12:27:21.060664+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:00:13.506742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.045921+0000","last_change":"2024-04-06T12:27:21.047046+0000","last_active":"2024-04-06T12:27:22.045921+0000","last_peered":"2024-04-06T12:27:22.045921+0000","last_clean":"2024-04-06T12:27:22.045921+0000","last_became_active":"2024-04-06T12:27:21.046798+0000","last_became_peered":"2024-04-06T12:27:21.046798+0000","last_unstale":"2024-04-06T12:27:22.045921+0000","last_undegraded":"2024-04-06T12:27:22.045921+0000","last_fullsized":"2024-04-06T12:27:22.045921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:23:44.000355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.048716+0000","last_change":"2024-04-06T12:27:20.642417+0000","last_active":"2024-04-06T12:27:21.048716+0000","last_peered":"2024-04-06T12:27:21.048716+0000","last_clean":"2024-04-06T12:27:21.048716+0000","last_became_active":"2024-04-06T12:27:20.642057+0000","last_became_peered":"2024-04-06T12:27:20.642057+0000","last_unstale":"2024-04-06T12:27:21.048716+0000","last_undegraded":"2024-04-06T12:27:21.048716+0000","last_fullsized":"2024-04-06T12:27:21.048716+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:40:56.649109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046707+0000","last_change":"2024-04-06T12:27:21.056061+0000","last_active":"2024-04-06T12:27:22.046707+0000","last_peered":"2024-04-06T12:27:22.046707+0000","last_clean":"2024-04-06T12:27:22.046707+0000","last_became_active":"2024-04-06T12:27:21.055761+0000","last_became_peered":"2024-04-06T12:27:21.055761+0000","last_unstale":"2024-04-06T12:27:22.046707+0000","last_undegraded":"2024-04-06T12:27:22.046707+0000","last_fullsized":"2024-04-06T12:27:22.046707+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:20:15.444097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.046311+0000","last_change":"2024-04-06T12:27:20.642798+0000","last_active":"2024-04-06T12:27:21.046311+0000","last_peered":"2024-04-06T12:27:21.046311+0000","last_clean":"2024-04-06T12:27:21.046311+0000","last_became_active":"2024-04-06T12:27:20.642006+0000","last_became_peered":"2024-04-06T12:27:20.642006+0000","last_unstale":"2024-04-06T12:27:21.046311+0000","last_undegraded":"2024-04-06T12:27:21.046311+0000","last_fullsized":"2024-04-06T12:27:21.046311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:14:39.407591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047517+0000","last_change":"2024-04-06T12:27:21.067188+0000","last_active":"2024-04-06T12:27:22.047517+0000","last_peered":"2024-04-06T12:27:22.047517+0000","last_clean":"2024-04-06T12:27:22.047517+0000","last_became_active":"2024-04-06T12:27:21.066981+0000","last_became_peered":"2024-04-06T12:27:21.066981+0000","last_unstale":"2024-04-06T12:27:22.047517+0000","last_undegraded":"2024-04-06T12:27:22.047517+0000","last_fullsized":"2024-04-06T12:27:22.047517+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:16:15.014557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035060+0000","last_change":"2024-04-06T12:27:20.642838+0000","last_active":"2024-04-06T12:27:21.035060+0000","last_peered":"2024-04-06T12:27:21.035060+0000","last_clean":"2024-04-06T12:27:21.035060+0000","last_became_active":"2024-04-06T12:27:20.642344+0000","last_became_peered":"2024-04-06T12:27:20.642344+0000","last_unstale":"2024-04-06T12:27:21.035060+0000","last_undegraded":"2024-04-06T12:27:21.035060+0000","last_fullsized":"2024-04-06T12:27:21.035060+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:39:48.790443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039044+0000","last_change":"2024-04-06T12:27:21.042767+0000","last_active":"2024-04-06T12:27:22.039044+0000","last_peered":"2024-04-06T12:27:22.039044+0000","last_clean":"2024-04-06T12:27:22.039044+0000","last_became_active":"2024-04-06T12:27:21.042541+0000","last_became_peered":"2024-04-06T12:27:21.042541+0000","last_unstale":"2024-04-06T12:27:22.039044+0000","last_undegraded":"2024-04-06T12:27:22.039044+0000","last_fullsized":"2024-04-06T12:27:22.039044+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:02:23.142749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.049488+0000","last_change":"2024-04-06T12:27:20.640874+0000","last_active":"2024-04-06T12:27:21.049488+0000","last_peered":"2024-04-06T12:27:21.049488+0000","last_clean":"2024-04-06T12:27:21.049488+0000","last_became_active":"2024-04-06T12:27:20.640592+0000","last_became_peered":"2024-04-06T12:27:20.640592+0000","last_unstale":"2024-04-06T12:27:21.049488+0000","last_undegraded":"2024-04-06T12:27:21.049488+0000","last_fullsized":"2024-04-06T12:27:21.049488+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:53:08.693479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046322+0000","last_change":"2024-04-06T12:27:21.050535+0000","last_active":"2024-04-06T12:27:22.046322+0000","last_peered":"2024-04-06T12:27:22.046322+0000","last_clean":"2024-04-06T12:27:22.046322+0000","last_became_active":"2024-04-06T12:27:21.050270+0000","last_became_peered":"2024-04-06T12:27:21.050270+0000","last_unstale":"2024-04-06T12:27:22.046322+0000","last_undegraded":"2024-04-06T12:27:22.046322+0000","last_fullsized":"2024-04-06T12:27:22.046322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:24:04.575202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.044986+0000","last_change":"2024-04-06T12:27:20.642938+0000","last_active":"2024-04-06T12:27:21.044986+0000","last_peered":"2024-04-06T12:27:21.044986+0000","last_clean":"2024-04-06T12:27:21.044986+0000","last_became_active":"2024-04-06T12:27:20.642392+0000","last_became_peered":"2024-04-06T12:27:20.642392+0000","last_unstale":"2024-04-06T12:27:21.044986+0000","last_undegraded":"2024-04-06T12:27:21.044986+0000","last_fullsized":"2024-04-06T12:27:21.044986+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:24.851744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039214+0000","last_change":"2024-04-06T12:27:21.042722+0000","last_active":"2024-04-06T12:27:22.039214+0000","last_peered":"2024-04-06T12:27:22.039214+0000","last_clean":"2024-04-06T12:27:22.039214+0000","last_became_active":"2024-04-06T12:27:21.042460+0000","last_became_peered":"2024-04-06T12:27:21.042460+0000","last_unstale":"2024-04-06T12:27:22.039214+0000","last_undegraded":"2024-04-06T12:27:22.039214+0000","last_fullsized":"2024-04-06T12:27:22.039214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:16:01.192728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.037118+0000","last_change":"2024-04-06T12:27:20.643086+0000","last_active":"2024-04-06T12:27:21.037118+0000","last_peered":"2024-04-06T12:27:21.037118+0000","last_clean":"2024-04-06T12:27:21.037118+0000","last_became_active":"2024-04-06T12:27:20.642767+0000","last_became_peered":"2024-04-06T12:27:20.642767+0000","last_unstale":"2024-04-06T12:27:21.037118+0000","last_undegraded":"2024-04-06T12:27:21.037118+0000","last_fullsized":"2024-04-06T12:27:21.037118+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:54:55.715540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047715+0000","last_change":"2024-04-06T12:27:21.058482+0000","last_active":"2024-04-06T12:27:22.047715+0000","last_peered":"2024-04-06T12:27:22.047715+0000","last_clean":"2024-04-06T12:27:22.047715+0000","last_became_active":"2024-04-06T12:27:21.058235+0000","last_became_peered":"2024-04-06T12:27:21.058235+0000","last_unstale":"2024-04-06T12:27:22.047715+0000","last_undegraded":"2024-04-06T12:27:22.047715+0000","last_fullsized":"2024-04-06T12:27:22.047715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:16.550344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"83'3","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.065344+0000","last_change":"2024-04-06T12:27:20.643591+0000","last_active":"2024-04-06T12:27:21.065344+0000","last_peered":"2024-04-06T12:27:21.065344+0000","last_clean":"2024-04-06T12:27:21.065344+0000","last_became_active":"2024-04-06T12:27:20.643307+0000","last_became_peered":"2024-04-06T12:27:20.643307+0000","last_unstale":"2024-04-06T12:27:21.065344+0000","last_undegraded":"2024-04-06T12:27:21.065344+0000","last_fullsized":"2024-04-06T12:27:21.065344+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:15.542425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047714+0000","last_change":"2024-04-06T12:27:21.050955+0000","last_active":"2024-04-06T12:27:22.047714+0000","last_peered":"2024-04-06T12:27:22.047714+0000","last_clean":"2024-04-06T12:27:22.047714+0000","last_became_active":"2024-04-06T12:27:21.050675+0000","last_became_peered":"2024-04-06T12:27:21.050675+0000","last_unstale":"2024-04-06T12:27:22.047714+0000","last_undegraded":"2024-04-06T12:27:22.047714+0000","last_fullsized":"2024-04-06T12:27:22.047714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:57.645194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"83'6","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.067970+0000","last_change":"2024-04-06T12:27:20.638258+0000","last_active":"2024-04-06T12:27:21.067970+0000","last_peered":"2024-04-06T12:27:21.067970+0000","last_clean":"2024-04-06T12:27:21.067970+0000","last_became_active":"2024-04-06T12:27:20.638017+0000","last_became_peered":"2024-04-06T12:27:20.638017+0000","last_unstale":"2024-04-06T12:27:21.067970+0000","last_undegraded":"2024-04-06T12:27:21.067970+0000","last_fullsized":"2024-04-06T12:27:21.067970+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:48:54.260905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047268+0000","last_change":"2024-04-06T12:27:21.048435+0000","last_active":"2024-04-06T12:27:22.047268+0000","last_peered":"2024-04-06T12:27:22.047268+0000","last_clean":"2024-04-06T12:27:22.047268+0000","last_became_active":"2024-04-06T12:27:21.048159+0000","last_became_peered":"2024-04-06T12:27:21.048159+0000","last_unstale":"2024-04-06T12:27:22.047268+0000","last_undegraded":"2024-04-06T12:27:22.047268+0000","last_fullsized":"2024-04-06T12:27:22.047268+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:59:57.937757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.062993+0000","last_change":"2024-04-06T12:27:20.640097+0000","last_active":"2024-04-06T12:27:21.062993+0000","last_peered":"2024-04-06T12:27:21.062993+0000","last_clean":"2024-04-06T12:27:21.062993+0000","last_became_active":"2024-04-06T12:27:20.639654+0000","last_became_peered":"2024-04-06T12:27:20.639654+0000","last_unstale":"2024-04-06T12:27:21.062993+0000","last_undegraded":"2024-04-06T12:27:21.062993+0000","last_fullsized":"2024-04-06T12:27:21.062993+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:48:54.260905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046846+0000","last_change":"2024-04-06T12:27:21.050116+0000","last_active":"2024-04-06T12:27:22.046846+0000","last_peered":"2024-04-06T12:27:22.046846+0000","last_clean":"2024-04-06T12:27:22.046846+0000","last_became_active":"2024-04-06T12:27:21.049828+0000","last_became_peered":"2024-04-06T12:27:21.049828+0000","last_unstale":"2024-04-06T12:27:22.046846+0000","last_undegraded":"2024-04-06T12:27:22.046846+0000","last_fullsized":"2024-04-06T12:27:22.046846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:41:35.413608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.043551+0000","last_change":"2024-04-06T12:27:20.643715+0000","last_active":"2024-04-06T12:27:21.043551+0000","last_peered":"2024-04-06T12:27:21.043551+0000","last_clean":"2024-04-06T12:27:21.043551+0000","last_became_active":"2024-04-06T12:27:20.643243+0000","last_became_peered":"2024-04-06T12:27:20.643243+0000","last_unstale":"2024-04-06T12:27:21.043551+0000","last_undegraded":"2024-04-06T12:27:21.043551+0000","last_fullsized":"2024-04-06T12:27:21.043551+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:48.861736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047578+0000","last_change":"2024-04-06T12:27:21.061425+0000","last_active":"2024-04-06T12:27:22.047578+0000","last_peered":"2024-04-06T12:27:22.047578+0000","last_clean":"2024-04-06T12:27:22.047578+0000","last_became_active":"2024-04-06T12:27:21.061137+0000","last_became_peered":"2024-04-06T12:27:21.061137+0000","last_unstale":"2024-04-06T12:27:22.047578+0000","last_undegraded":"2024-04-06T12:27:22.047578+0000","last_fullsized":"2024-04-06T12:27:22.047578+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:29:33.494735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.047140+0000","last_change":"2024-04-06T12:27:20.641538+0000","last_active":"2024-04-06T12:27:21.047140+0000","last_peered":"2024-04-06T12:27:21.047140+0000","last_clean":"2024-04-06T12:27:21.047140+0000","last_became_active":"2024-04-06T12:27:20.641121+0000","last_became_peered":"2024-04-06T12:27:20.641121+0000","last_unstale":"2024-04-06T12:27:21.047140+0000","last_undegraded":"2024-04-06T12:27:21.047140+0000","last_fullsized":"2024-04-06T12:27:21.047140+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:55.070413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046208+0000","last_change":"2024-04-06T12:27:21.051804+0000","last_active":"2024-04-06T12:27:22.046208+0000","last_peered":"2024-04-06T12:27:22.046208+0000","last_clean":"2024-04-06T12:27:22.046208+0000","last_became_active":"2024-04-06T12:27:21.051536+0000","last_became_peered":"2024-04-06T12:27:21.051536+0000","last_unstale":"2024-04-06T12:27:22.046208+0000","last_undegraded":"2024-04-06T12:27:22.046208+0000","last_fullsized":"2024-04-06T12:27:22.046208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:26:12.757657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"83'8","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.063908+0000","last_change":"2024-04-06T12:27:20.640204+0000","last_active":"2024-04-06T12:27:21.063908+0000","last_peered":"2024-04-06T12:27:21.063908+0000","last_clean":"2024-04-06T12:27:21.063908+0000","last_became_active":"2024-04-06T12:27:20.639913+0000","last_became_peered":"2024-04-06T12:27:20.639913+0000","last_unstale":"2024-04-06T12:27:21.063908+0000","last_undegraded":"2024-04-06T12:27:21.063908+0000","last_fullsized":"2024-04-06T12:27:21.063908+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:01:44.323843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039710+0000","last_change":"2024-04-06T12:27:21.055924+0000","last_active":"2024-04-06T12:27:22.039710+0000","last_peered":"2024-04-06T12:27:22.039710+0000","last_clean":"2024-04-06T12:27:22.039710+0000","last_became_active":"2024-04-06T12:27:21.053732+0000","last_became_peered":"2024-04-06T12:27:21.053732+0000","last_unstale":"2024-04-06T12:27:22.039710+0000","last_undegraded":"2024-04-06T12:27:22.039710+0000","last_fullsized":"2024-04-06T12:27:22.039710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:20:15.444097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.059785+0000","last_change":"2024-04-06T12:27:20.643933+0000","last_active":"2024-04-06T12:27:21.059785+0000","last_peered":"2024-04-06T12:27:21.059785+0000","last_clean":"2024-04-06T12:27:21.059785+0000","last_became_active":"2024-04-06T12:27:20.643644+0000","last_became_peered":"2024-04-06T12:27:20.643644+0000","last_unstale":"2024-04-06T12:27:21.059785+0000","last_undegraded":"2024-04-06T12:27:21.059785+0000","last_fullsized":"2024-04-06T12:27:21.059785+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:55.070413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047888+0000","last_change":"2024-04-06T12:27:21.059329+0000","last_active":"2024-04-06T12:27:22.047888+0000","last_peered":"2024-04-06T12:27:22.047888+0000","last_clean":"2024-04-06T12:27:22.047888+0000","last_became_active":"2024-04-06T12:27:21.058948+0000","last_became_peered":"2024-04-06T12:27:21.058948+0000","last_unstale":"2024-04-06T12:27:22.047888+0000","last_undegraded":"2024-04-06T12:27:22.047888+0000","last_fullsized":"2024-04-06T12:27:22.047888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:56.078333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.062616+0000","last_change":"2024-04-06T12:27:20.644387+0000","last_active":"2024-04-06T12:27:21.062616+0000","last_peered":"2024-04-06T12:27:21.062616+0000","last_clean":"2024-04-06T12:27:21.062616+0000","last_became_active":"2024-04-06T12:27:20.644127+0000","last_became_peered":"2024-04-06T12:27:20.644127+0000","last_unstale":"2024-04-06T12:27:21.062616+0000","last_undegraded":"2024-04-06T12:27:21.062616+0000","last_fullsized":"2024-04-06T12:27:21.062616+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:29:32.486815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047663+0000","last_change":"2024-04-06T12:27:21.059198+0000","last_active":"2024-04-06T12:27:22.047663+0000","last_peered":"2024-04-06T12:27:22.047663+0000","last_clean":"2024-04-06T12:27:22.047663+0000","last_became_active":"2024-04-06T12:27:21.058950+0000","last_became_peered":"2024-04-06T12:27:21.058950+0000","last_unstale":"2024-04-06T12:27:22.047663+0000","last_undegraded":"2024-04-06T12:27:22.047663+0000","last_fullsized":"2024-04-06T12:27:22.047663+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:20:12.514805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.065347+0000","last_change":"2024-04-06T12:27:20.644199+0000","last_active":"2024-04-06T12:27:21.065347+0000","last_peered":"2024-04-06T12:27:21.065347+0000","last_clean":"2024-04-06T12:27:21.065347+0000","last_became_active":"2024-04-06T12:27:20.643870+0000","last_became_peered":"2024-04-06T12:27:20.643870+0000","last_unstale":"2024-04-06T12:27:21.065347+0000","last_undegraded":"2024-04-06T12:27:21.065347+0000","last_fullsized":"2024-04-06T12:27:21.065347+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:44.931614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047541+0000","last_change":"2024-04-06T12:27:21.061929+0000","last_active":"2024-04-06T12:27:22.047541+0000","last_peered":"2024-04-06T12:27:22.047541+0000","last_clean":"2024-04-06T12:27:22.047541+0000","last_became_active":"2024-04-06T12:27:21.061627+0000","last_became_peered":"2024-04-06T12:27:21.061627+0000","last_unstale":"2024-04-06T12:27:22.047541+0000","last_undegraded":"2024-04-06T12:27:22.047541+0000","last_fullsized":"2024-04-06T12:27:22.047541+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:02.552663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035432+0000","last_change":"2024-04-06T12:27:20.642400+0000","last_active":"2024-04-06T12:27:21.035432+0000","last_peered":"2024-04-06T12:27:21.035432+0000","last_clean":"2024-04-06T12:27:21.035432+0000","last_became_active":"2024-04-06T12:27:20.642039+0000","last_became_peered":"2024-04-06T12:27:20.642039+0000","last_unstale":"2024-04-06T12:27:21.035432+0000","last_undegraded":"2024-04-06T12:27:21.035432+0000","last_fullsized":"2024-04-06T12:27:21.035432+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:44.642507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.039877+0000","last_change":"2024-04-06T12:27:21.046054+0000","last_active":"2024-04-06T12:27:22.039877+0000","last_peered":"2024-04-06T12:27:22.039877+0000","last_clean":"2024-04-06T12:27:22.039877+0000","last_became_active":"2024-04-06T12:27:21.045799+0000","last_became_peered":"2024-04-06T12:27:21.045799+0000","last_unstale":"2024-04-06T12:27:22.039877+0000","last_undegraded":"2024-04-06T12:27:22.039877+0000","last_fullsized":"2024-04-06T12:27:22.039877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:29:16.068086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.059940+0000","last_change":"2024-04-06T12:27:20.635120+0000","last_active":"2024-04-06T12:27:21.059940+0000","last_peered":"2024-04-06T12:27:21.059940+0000","last_clean":"2024-04-06T12:27:21.059940+0000","last_became_active":"2024-04-06T12:27:20.634852+0000","last_became_peered":"2024-04-06T12:27:20.634852+0000","last_unstale":"2024-04-06T12:27:21.059940+0000","last_undegraded":"2024-04-06T12:27:21.059940+0000","last_fullsized":"2024-04-06T12:27:21.059940+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:34:57.334069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046898+0000","last_change":"2024-04-06T12:27:21.044875+0000","last_active":"2024-04-06T12:27:22.046898+0000","last_peered":"2024-04-06T12:27:22.046898+0000","last_clean":"2024-04-06T12:27:22.046898+0000","last_became_active":"2024-04-06T12:27:21.044609+0000","last_became_peered":"2024-04-06T12:27:21.044609+0000","last_unstale":"2024-04-06T12:27:22.046898+0000","last_undegraded":"2024-04-06T12:27:22.046898+0000","last_fullsized":"2024-04-06T12:27:22.046898+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:48:55.268825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.043598+0000","last_change":"2024-04-06T12:27:20.643509+0000","last_active":"2024-04-06T12:27:21.043598+0000","last_peered":"2024-04-06T12:27:21.043598+0000","last_clean":"2024-04-06T12:27:21.043598+0000","last_became_active":"2024-04-06T12:27:20.642877+0000","last_became_peered":"2024-04-06T12:27:20.642877+0000","last_unstale":"2024-04-06T12:27:21.043598+0000","last_undegraded":"2024-04-06T12:27:21.043598+0000","last_fullsized":"2024-04-06T12:27:21.043598+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:12:17.891288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048159+0000","last_change":"2024-04-06T12:27:21.063218+0000","last_active":"2024-04-06T12:27:22.048159+0000","last_peered":"2024-04-06T12:27:22.048159+0000","last_clean":"2024-04-06T12:27:22.048159+0000","last_became_active":"2024-04-06T12:27:21.062952+0000","last_became_peered":"2024-04-06T12:27:21.062952+0000","last_unstale":"2024-04-06T12:27:22.048159+0000","last_undegraded":"2024-04-06T12:27:22.048159+0000","last_fullsized":"2024-04-06T12:27:22.048159+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:54:56.723460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.038449+0000","last_change":"2024-04-06T12:27:20.640135+0000","last_active":"2024-04-06T12:27:21.038449+0000","last_peered":"2024-04-06T12:27:21.038449+0000","last_clean":"2024-04-06T12:27:21.038449+0000","last_became_active":"2024-04-06T12:27:20.639873+0000","last_became_peered":"2024-04-06T12:27:20.639873+0000","last_unstale":"2024-04-06T12:27:21.038449+0000","last_undegraded":"2024-04-06T12:27:21.038449+0000","last_fullsized":"2024-04-06T12:27:21.038449+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:26:31.570422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.040070+0000","last_change":"2024-04-06T12:27:21.053276+0000","last_active":"2024-04-06T12:27:22.040070+0000","last_peered":"2024-04-06T12:27:22.040070+0000","last_clean":"2024-04-06T12:27:22.040070+0000","last_became_active":"2024-04-06T12:27:21.053033+0000","last_became_peered":"2024-04-06T12:27:21.053033+0000","last_unstale":"2024-04-06T12:27:22.040070+0000","last_undegraded":"2024-04-06T12:27:22.040070+0000","last_fullsized":"2024-04-06T12:27:22.040070+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:49:47.315178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.061029+0000","last_change":"2024-04-06T12:27:20.642377+0000","last_active":"2024-04-06T12:27:21.061029+0000","last_peered":"2024-04-06T12:27:21.061029+0000","last_clean":"2024-04-06T12:27:21.061029+0000","last_became_active":"2024-04-06T12:27:20.642111+0000","last_became_peered":"2024-04-06T12:27:20.642111+0000","last_unstale":"2024-04-06T12:27:21.061029+0000","last_undegraded":"2024-04-06T12:27:21.061029+0000","last_fullsized":"2024-04-06T12:27:21.061029+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:46:21.677982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046699+0000","last_change":"2024-04-06T12:27:21.047728+0000","last_active":"2024-04-06T12:27:22.046699+0000","last_peered":"2024-04-06T12:27:22.046699+0000","last_clean":"2024-04-06T12:27:22.046699+0000","last_became_active":"2024-04-06T12:27:21.046317+0000","last_became_peered":"2024-04-06T12:27:21.046317+0000","last_unstale":"2024-04-06T12:27:22.046699+0000","last_undegraded":"2024-04-06T12:27:22.046699+0000","last_fullsized":"2024-04-06T12:27:22.046699+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:08:29.040668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.702984+0000","last_change":"2024-04-06T12:27:20.643133+0000","last_active":"2024-04-06T12:27:21.702984+0000","last_peered":"2024-04-06T12:27:21.702984+0000","last_clean":"2024-04-06T12:27:21.702984+0000","last_became_active":"2024-04-06T12:27:20.642858+0000","last_became_peered":"2024-04-06T12:27:20.642858+0000","last_unstale":"2024-04-06T12:27:21.702984+0000","last_undegraded":"2024-04-06T12:27:21.702984+0000","last_fullsized":"2024-04-06T12:27:21.702984+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:16:14.006637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.048294+0000","last_change":"2024-04-06T12:27:21.059468+0000","last_active":"2024-04-06T12:27:22.048294+0000","last_peered":"2024-04-06T12:27:22.048294+0000","last_clean":"2024-04-06T12:27:22.048294+0000","last_became_active":"2024-04-06T12:27:21.059222+0000","last_became_peered":"2024-04-06T12:27:21.059222+0000","last_unstale":"2024-04-06T12:27:22.048294+0000","last_undegraded":"2024-04-06T12:27:22.048294+0000","last_fullsized":"2024-04-06T12:27:22.048294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:20:15.444097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.040662+0000","last_change":"2024-04-06T12:27:20.638922+0000","last_active":"2024-04-06T12:27:21.040662+0000","last_peered":"2024-04-06T12:27:21.040662+0000","last_clean":"2024-04-06T12:27:21.040662+0000","last_became_active":"2024-04-06T12:27:20.638666+0000","last_became_peered":"2024-04-06T12:27:20.638666+0000","last_unstale":"2024-04-06T12:27:21.040662+0000","last_undegraded":"2024-04-06T12:27:21.040662+0000","last_fullsized":"2024-04-06T12:27:21.040662+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:08:18.449996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047650+0000","last_change":"2024-04-06T12:27:21.060870+0000","last_active":"2024-04-06T12:27:22.047650+0000","last_peered":"2024-04-06T12:27:22.047650+0000","last_clean":"2024-04-06T12:27:22.047650+0000","last_became_active":"2024-04-06T12:27:21.060603+0000","last_became_peered":"2024-04-06T12:27:21.060603+0000","last_unstale":"2024-04-06T12:27:22.047650+0000","last_undegraded":"2024-04-06T12:27:22.047650+0000","last_fullsized":"2024-04-06T12:27:22.047650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:48:55.268825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.044529+0000","last_change":"2024-04-06T12:27:20.639042+0000","last_active":"2024-04-06T12:27:21.044529+0000","last_peered":"2024-04-06T12:27:21.044529+0000","last_clean":"2024-04-06T12:27:21.044529+0000","last_became_active":"2024-04-06T12:27:20.638475+0000","last_became_peered":"2024-04-06T12:27:20.638475+0000","last_unstale":"2024-04-06T12:27:21.044529+0000","last_undegraded":"2024-04-06T12:27:21.044529+0000","last_fullsized":"2024-04-06T12:27:21.044529+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:52:15.542425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047260+0000","last_change":"2024-04-06T12:27:21.060781+0000","last_active":"2024-04-06T12:27:22.047260+0000","last_peered":"2024-04-06T12:27:22.047260+0000","last_clean":"2024-04-06T12:27:22.047260+0000","last_became_active":"2024-04-06T12:27:21.060520+0000","last_became_peered":"2024-04-06T12:27:21.060520+0000","last_unstale":"2024-04-06T12:27:22.047260+0000","last_undegraded":"2024-04-06T12:27:22.047260+0000","last_fullsized":"2024-04-06T12:27:22.047260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:46:22.685902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.041028+0000","last_change":"2024-04-06T12:27:20.641603+0000","last_active":"2024-04-06T12:27:21.041028+0000","last_peered":"2024-04-06T12:27:21.041028+0000","last_clean":"2024-04-06T12:27:21.041028+0000","last_became_active":"2024-04-06T12:27:20.641135+0000","last_became_peered":"2024-04-06T12:27:20.641135+0000","last_unstale":"2024-04-06T12:27:21.041028+0000","last_undegraded":"2024-04-06T12:27:21.041028+0000","last_fullsized":"2024-04-06T12:27:21.041028+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:48.861736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047005+0000","last_change":"2024-04-06T12:27:21.055075+0000","last_active":"2024-04-06T12:27:22.047005+0000","last_peered":"2024-04-06T12:27:22.047005+0000","last_clean":"2024-04-06T12:27:22.047005+0000","last_became_active":"2024-04-06T12:27:21.054806+0000","last_became_peered":"2024-04-06T12:27:21.054806+0000","last_unstale":"2024-04-06T12:27:22.047005+0000","last_undegraded":"2024-04-06T12:27:22.047005+0000","last_fullsized":"2024-04-06T12:27:22.047005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:14.746095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035500+0000","last_change":"2024-04-06T12:27:20.641433+0000","last_active":"2024-04-06T12:27:21.035500+0000","last_peered":"2024-04-06T12:27:21.035500+0000","last_clean":"2024-04-06T12:27:21.035500+0000","last_became_active":"2024-04-06T12:27:20.641181+0000","last_became_peered":"2024-04-06T12:27:20.641181+0000","last_unstale":"2024-04-06T12:27:21.035500+0000","last_undegraded":"2024-04-06T12:27:21.035500+0000","last_fullsized":"2024-04-06T12:27:21.035500+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:20:08.218629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046219+0000","last_change":"2024-04-06T12:27:21.055928+0000","last_active":"2024-04-06T12:27:22.046219+0000","last_peered":"2024-04-06T12:27:22.046219+0000","last_clean":"2024-04-06T12:27:22.046219+0000","last_became_active":"2024-04-06T12:27:21.055641+0000","last_became_peered":"2024-04-06T12:27:21.055641+0000","last_unstale":"2024-04-06T12:27:22.046219+0000","last_undegraded":"2024-04-06T12:27:22.046219+0000","last_fullsized":"2024-04-06T12:27:22.046219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:08:32.179469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035357+0000","last_change":"2024-04-06T12:27:20.643620+0000","last_active":"2024-04-06T12:27:21.035357+0000","last_peered":"2024-04-06T12:27:21.035357+0000","last_clean":"2024-04-06T12:27:21.035357+0000","last_became_active":"2024-04-06T12:27:20.642671+0000","last_became_peered":"2024-04-06T12:27:20.642671+0000","last_unstale":"2024-04-06T12:27:21.035357+0000","last_undegraded":"2024-04-06T12:27:21.035357+0000","last_fullsized":"2024-04-06T12:27:21.035357+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:56:37.738753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.046202+0000","last_change":"2024-04-06T12:27:21.046951+0000","last_active":"2024-04-06T12:27:22.046202+0000","last_peered":"2024-04-06T12:27:22.046202+0000","last_clean":"2024-04-06T12:27:22.046202+0000","last_became_active":"2024-04-06T12:27:21.046651+0000","last_became_peered":"2024-04-06T12:27:21.046651+0000","last_unstale":"2024-04-06T12:27:22.046202+0000","last_undegraded":"2024-04-06T12:27:22.046202+0000","last_fullsized":"2024-04-06T12:27:22.046202+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:59:55.516160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.044036+0000","last_change":"2024-04-06T12:27:20.635487+0000","last_active":"2024-04-06T12:27:21.044036+0000","last_peered":"2024-04-06T12:27:21.044036+0000","last_clean":"2024-04-06T12:27:21.044036+0000","last_became_active":"2024-04-06T12:27:20.635194+0000","last_became_peered":"2024-04-06T12:27:20.635194+0000","last_unstale":"2024-04-06T12:27:21.044036+0000","last_undegraded":"2024-04-06T12:27:21.044036+0000","last_fullsized":"2024-04-06T12:27:21.044036+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:01.544743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-06T12:27:22.047357+0000","last_change":"2024-04-06T12:27:21.062962+0000","last_active":"2024-04-06T12:27:22.047357+0000","last_peered":"2024-04-06T12:27:22.047357+0000","last_clean":"2024-04-06T12:27:22.047357+0000","last_became_active":"2024-04-06T12:27:21.062683+0000","last_became_peered":"2024-04-06T12:27:21.062683+0000","last_unstale":"2024-04-06T12:27:22.047357+0000","last_undegraded":"2024-04-06T12:27:22.047357+0000","last_fullsized":"2024-04-06T12:27:22.047357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:20.616323+0000","last_clean_scrub_stamp":"2024-04-06T12:27:20.616323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:49.869655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.035100+0000","last_change":"2024-04-06T12:27:20.640825+0000","last_active":"2024-04-06T12:27:21.035100+0000","last_peered":"2024-04-06T12:27:21.035100+0000","last_clean":"2024-04-06T12:27:21.035100+0000","last_became_active":"2024-04-06T12:27:20.640568+0000","last_became_peered":"2024-04-06T12:27:20.640568+0000","last_unstale":"2024-04-06T12:27:21.035100+0000","last_undegraded":"2024-04-06T12:27:21.035100+0000","last_fullsized":"2024-04-06T12:27:21.035100+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:27:19.608404+0000","last_clean_scrub_stamp":"2024-04-06T12:27:19.608404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:27:16.320148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'161","reported_seq":391,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-06T12:27:21.041570+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:27:21.041570+0000","last_peered":"2024-04-06T12:27:21.041570+0000","last_clean":"2024-04-06T12:27:21.041570+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:27:21.041570+0000","last_undegraded":"2024-04-06T12:27:21.041570+0000","last_fullsized":"2024-04-06T12:27:21.041570+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575035,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58948,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312892,"statfs":{"total":96636764160,"available":96576401408,"internally_reserved":0,"allocated":1630208,"data_stored":1095735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55800000000000005,"15min":0.54900000000000004},"min":{"1min":0.375,"5min":0.219,"15min":0.219},"max":{"1min":0.80100000000000005,"5min":1.143,"15min":1.143},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55700000000000005,"15min":0.55300000000000005},"min":{"1min":0.433,"5min":0.35399999999999998,"15min":0.25900000000000001},"max":{"1min":0.73299999999999998,"5min":0.77100000000000002,"15min":0.78200000000000003},"last":0.51400000000000001}]},{"osd":1,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55400000000000005,"15min":0.55100000000000005},"min":{"1min":0.42299999999999999,"5min":0.33600000000000002,"15min":0.31900000000000001},"max":{"1min":0.748,"5min":0.77100000000000002,"15min":0.81399999999999995},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.41699999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.77400000000000002,"5min":0.79400000000000004,"15min":0.79600000000000004},"last":0.41399999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56999999999999995,"15min":0.55200000000000005},"min":{"1min":0.441,"5min":0.249,"15min":0.249},"max":{"1min":0.77100000000000002,"5min":0.78500000000000003,"15min":0.83299999999999996},"last":0.374},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57099999999999995,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.24199999999999999},"max":{"1min":0.76600000000000001,"5min":0.78300000000000003,"15min":0.82299999999999995},"last":0.49199999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58099999999999996,"15min":0.57199999999999995},"min":{"1min":0.45900000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.75800000000000001,"5min":1.357,"15min":1.357},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59299999999999997,"15min":0.59599999999999997},"min":{"1min":0.44400000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.748,"5min":0.81799999999999995,"15min":0.92000000000000004},"last":0.44}]},{"osd":4,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.45300000000000001,"15min":0.45800000000000002},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.25},"max":{"1min":0.68300000000000005,"5min":0.70399999999999996,"15min":0.83599999999999997},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.53700000000000003},"min":{"1min":0.42899999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71399999999999997,"5min":0.77700000000000002,"15min":1.0780000000000001},"last":0.50700000000000001}]},{"osd":5,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.498,"15min":0.51300000000000001},"min":{"1min":0.33600000000000002,"5min":0.29199999999999998,"15min":0.23799999999999999},"max":{"1min":0.73299999999999998,"5min":0.80100000000000005,"15min":0.98199999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55900000000000005,"15min":0.56599999999999995},"min":{"1min":0.374,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.70699999999999996,"5min":0.81599999999999995,"15min":0.93600000000000005},"last":0.55800000000000005}]},{"osd":6,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53900000000000003,"15min":0.54900000000000004},"min":{"1min":0.39300000000000002,"5min":0.33700000000000002,"15min":0.28799999999999998},"max":{"1min":0.70699999999999996,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55800000000000005,"15min":0.56799999999999995},"min":{"1min":0.40200000000000002,"5min":0.30399999999999999,"15min":0.26900000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.88500000000000001},"last":0.54000000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575034,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":1019904,"data_stored":503606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.36699999999999999,"5min":0.26200000000000001,"15min":0.23000000000000001},"max":{"1min":0.77500000000000002,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52500000000000002,"15min":0.53300000000000003},"min":{"1min":0.374,"5min":0.30199999999999999,"15min":0.245},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.53300000000000003}]},{"osd":1,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.32700000000000001,"5min":0.32600000000000001,"15min":0.24399999999999999},"max":{"1min":0.78200000000000003,"5min":0.78700000000000003,"15min":0.84999999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.38400000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.83099999999999996},"last":0.50700000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.39100000000000001,"5min":0.30499999999999999,"15min":0.29899999999999999},"max":{"1min":0.749,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.38800000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.56899999999999995,"15min":0.58099999999999996},"min":{"1min":0.35099999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.745,"5min":0.79000000000000004,"15min":0.81200000000000006},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.60899999999999999,"15min":0.61899999999999999},"min":{"1min":0.51700000000000002,"5min":0.41099999999999998,"15min":0.374},"max":{"1min":0.79200000000000004,"5min":0.82799999999999996,"15min":0.872},"last":0.52000000000000002}]},{"osd":4,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.49099999999999999,"15min":0.496},"min":{"1min":0.28599999999999998,"5min":0.27500000000000002,"15min":0.27300000000000002},"max":{"1min":0.747,"5min":0.749,"15min":0.749},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.53000000000000003},"min":{"1min":0.34899999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.73199999999999998,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.43099999999999999}]},{"osd":5,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.47999999999999998,"15min":0.49199999999999999},"min":{"1min":0.34699999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71099999999999997,"5min":0.755,"15min":0.75800000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.50600000000000001,"15min":0.51300000000000001},"min":{"1min":0.32800000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72599999999999998,"5min":0.83699999999999997,"15min":0.92800000000000005},"last":0.45400000000000001}]},{"osd":7,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53000000000000003,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70099999999999996,"5min":0.79800000000000004,"15min":0.81000000000000005},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55800000000000005,"15min":0.56100000000000005},"min":{"1min":0.41899999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80800000000000005,"5min":0.88900000000000001,"15min":0.96099999999999997},"last":0.54800000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575033,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":1044480,"data_stored":505900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.377,"15min":0.36199999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.47699999999999998,"5min":0.58999999999999997,"15min":0.71699999999999997},"last":0.308},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39900000000000002,"15min":0.39600000000000002},"min":{"1min":0.253,"5min":0.215,"15min":0.215},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.32600000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.40500000000000003,"15min":0.39600000000000002},"min":{"1min":0.27200000000000002,"5min":0.23899999999999999,"15min":0.19400000000000001},"max":{"1min":0.57599999999999996,"5min":0.624,"15min":0.67600000000000005},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.432,"15min":0.42999999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.249},"max":{"1min":0.60699999999999998,"5min":0.63600000000000001,"15min":0.79000000000000004},"last":0.28899999999999998}]},{"osd":3,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45600000000000002,"15min":0.44700000000000001},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67400000000000004,"5min":0.67800000000000005,"15min":0.84299999999999997},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.42599999999999999,"15min":0.41899999999999998},"min":{"1min":0.30499999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.56499999999999995,"5min":0.69599999999999995,"15min":0.86299999999999999},"last":0.375}]},{"osd":4,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.623,"15min":0.63200000000000001},"min":{"1min":0.42599999999999999,"5min":0.40100000000000002,"15min":0.34799999999999998},"max":{"1min":0.71499999999999997,"5min":0.86799999999999999,"15min":6.3739999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.63100000000000001,"15min":0.622},"min":{"1min":0.45400000000000001,"5min":0.441,"15min":0.30399999999999999},"max":{"1min":0.84299999999999997,"5min":0.91400000000000003,"15min":1.0269999999999999},"last":0.504}]},{"osd":5,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.624,"15min":0.61399999999999999},"min":{"1min":0.47599999999999998,"5min":0.42999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83999999999999997,"5min":0.91300000000000003,"15min":0.97699999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.628},"min":{"1min":0.442,"5min":0.43099999999999999,"15min":0.35099999999999998},"max":{"1min":0.78700000000000003,"5min":0.999,"15min":1.0089999999999999},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.623,"15min":0.61399999999999999},"min":{"1min":0.47699999999999998,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":0.76500000000000001,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.65300000000000002,"15min":0.64900000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.41799999999999998},"max":{"1min":0.83999999999999997,"5min":0.88700000000000001,"15min":1.151},"last":0.441}]},{"osd":7,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.64800000000000002,"15min":0.65500000000000003},"min":{"1min":0.44600000000000001,"5min":0.41899999999999998,"15min":0.38800000000000001},"max":{"1min":0.81699999999999995,"5min":1.0329999999999999,"15min":6.3639999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.47299999999999998,"5min":0.40200000000000002,"15min":0.39300000000000002},"max":{"1min":0.82499999999999996,"5min":0.95599999999999996,"15min":1.073},"last":0.55400000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575033,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32688,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339152,"statfs":{"total":96636764160,"available":96603291648,"internally_reserved":0,"allocated":1609728,"data_stored":1093940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.38700000000000001,"15min":0.38},"min":{"1min":0.312,"5min":0.19,"15min":0.19},"max":{"1min":0.53900000000000003,"5min":0.80000000000000004,"15min":0.89100000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.40200000000000002},"min":{"1min":0.30599999999999999,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":0.60599999999999998,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.32300000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.40899999999999997,"15min":0.41199999999999998},"min":{"1min":0.29899999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.57799999999999996,"5min":0.60299999999999998,"15min":1.8089999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.434,"15min":0.42899999999999999},"min":{"1min":0.32600000000000001,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":0.64700000000000002,"5min":1.4790000000000001,"15min":1.764},"last":0.35899999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.41899999999999998,"15min":0.41099999999999998},"min":{"1min":0.34100000000000003,"5min":0.252,"15min":0.21299999999999999},"max":{"1min":0.63,"5min":1.25,"15min":1.25},"last":0.309},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.32900000000000001,"5min":0.28499999999999998,"15min":0.249},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.92000000000000004},"last":0.41199999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.627,"15min":0.61799999999999999},"min":{"1min":0.46100000000000002,"5min":0.371,"15min":0.35699999999999998},"max":{"1min":0.77700000000000002,"5min":1.514,"15min":1.514},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.63900000000000001,"15min":0.628},"min":{"1min":0.47599999999999998,"5min":0.37,"15min":0.311},"max":{"1min":0.86799999999999999,"5min":1.202,"15min":1.202},"last":0.60299999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63400000000000001,"15min":0.61899999999999999},"min":{"1min":0.46999999999999997,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.84899999999999998,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64400000000000002,"15min":0.64700000000000002},"min":{"1min":0.45200000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.88600000000000001,"5min":0.94499999999999995,"15min":0.97899999999999998},"last":0.746}]},{"osd":6,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65300000000000002,"15min":0.64800000000000002},"min":{"1min":0.51700000000000002,"5min":0.39100000000000001,"15min":0.378},"max":{"1min":0.84999999999999998,"5min":0.91000000000000003,"15min":0.96699999999999997},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.66900000000000004,"15min":0.66400000000000003},"min":{"1min":0.499,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84999999999999998,"5min":1.613,"15min":1.613},"last":0.58799999999999997}]},{"osd":7,"last update":"Sat Apr 6 12:26:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67000000000000004,"15min":0.66300000000000003},"min":{"1min":0.51500000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90100000000000002,"5min":1.54,"15min":1.54},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.67700000000000005,"15min":0.65700000000000003},"min":{"1min":0.51400000000000001,"5min":0.38400000000000001,"15min":0.36699999999999999},"max":{"1min":0.88300000000000001,"5min":1.498,"15min":1.498},"last":0.57799999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575035,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50156,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321684,"statfs":{"total":96636764160,"available":96585404416,"internally_reserved":0,"allocated":1015808,"data_stored":503572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38600000000000001,"15min":0.38100000000000001},"min":{"1min":0.26700000000000002,"5min":0.214,"15min":0.21099999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":1.159},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.41199999999999998,"15min":0.40899999999999997},"min":{"1min":0.27500000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.54600000000000004,"5min":0.60899999999999999,"15min":1.075},"last":0.29599999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.41099999999999998,"15min":0.41899999999999998},"min":{"1min":0.29599999999999999,"5min":0.26400000000000001,"15min":0.24299999999999999},"max":{"1min":0.57599999999999996,"5min":0.78500000000000003,"15min":3.6179999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.42299999999999999,"15min":0.42999999999999999},"min":{"1min":0.309,"5min":0.27100000000000002,"15min":0.23599999999999999},"max":{"1min":0.58899999999999997,"5min":0.61399999999999999,"15min":1.125},"last":0.32700000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.46000000000000002,"15min":0.46500000000000002},"min":{"1min":0.34399999999999997,"5min":0.30499999999999999,"15min":0.29699999999999999},"max":{"1min":0.63100000000000001,"5min":0.79200000000000004,"15min":3.661},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.44400000000000001,"15min":0.438},"min":{"1min":0.376,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.55900000000000005,"5min":1.0549999999999999,"15min":1.1910000000000001},"last":0.313}]},{"osd":4,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.61499999999999999,"15min":0.60699999999999998},"min":{"1min":0.45900000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81699999999999995,"5min":0.88200000000000001,"15min":1.2170000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63600000000000001,"15min":0.628},"min":{"1min":0.48799999999999999,"5min":0.316,"15min":0.30499999999999999},"max":{"1min":0.89900000000000002,"5min":0.90200000000000002,"15min":1.101},"last":0.47599999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.64800000000000002,"15min":0.65300000000000002},"min":{"1min":0.57599999999999996,"5min":0.42299999999999999,"15min":0.32900000000000001},"max":{"1min":0.90300000000000002,"5min":1.6870000000000001,"15min":3.6920000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.628,"15min":0.625},"min":{"1min":0.49399999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.81200000000000006,"5min":0.89500000000000002,"15min":1.244},"last":0.49199999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64300000000000002,"15min":0.63400000000000001},"min":{"1min":0.56499999999999995,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.84899999999999998,"5min":0.93400000000000005,"15min":1.012},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67100000000000004,"15min":0.66900000000000004},"min":{"1min":0.51300000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93100000000000005,"5min":1.728,"15min":3.7490000000000001},"last":0.48299999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65400000000000003,"15min":0.64900000000000002},"min":{"1min":0.52900000000000003,"5min":0.40699999999999997,"15min":0.35599999999999998},"max":{"1min":0.80500000000000005,"5min":0.86899999999999999,"15min":1.0469999999999999},"last":0.501},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.67700000000000005,"15min":0.68200000000000005},"min":{"1min":0.54300000000000004,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.90900000000000003,"5min":1.758,"15min":3.7200000000000002},"last":0.56299999999999994}]}]},{"osd":3,"up_from":13,"seq":55834575034,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32120,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339720,"statfs":{"total":96636764160,"available":96603873280,"internally_reserved":0,"allocated":1019904,"data_stored":504591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.24199999999999999,"5min":0.21099999999999999,"15min":0.191},"max":{"1min":0.51700000000000002,"5min":0.59699999999999998,"15min":0.63900000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38100000000000001,"15min":0.377},"min":{"1min":0.308,"5min":0.218,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.21299999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.372,"15min":0.372},"min":{"1min":0.248,"5min":0.22700000000000001,"15min":0.215},"max":{"1min":0.47099999999999997,"5min":0.59999999999999998,"15min":0.62},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40200000000000002,"15min":0.40799999999999997},"min":{"1min":0.34200000000000003,"5min":0.26200000000000001,"15min":0.247},"max":{"1min":0.55200000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.39200000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.44900000000000001,"15min":0.45000000000000001},"min":{"1min":0.32800000000000001,"5min":0.27200000000000002,"15min":0.25700000000000001},"max":{"1min":0.55300000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.432},{"interface":"front","average":{"1min":0.495,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.39700000000000002,"5min":0.317,"15min":0.29899999999999999},"max":{"1min":0.622,"5min":0.64800000000000002,"15min":0.65700000000000003},"last":0.30199999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.49299999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.95999999999999996},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60999999999999999,"15min":0.61699999999999999},"min":{"1min":0.41899999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.74099999999999999,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.67100000000000004}]},{"osd":5,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.64400000000000002,"15min":0.65500000000000003},"min":{"1min":0.46000000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.91100000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.625},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65300000000000002,"15min":0.65200000000000002},"min":{"1min":0.42999999999999999,"5min":0.38200000000000001,"15min":0.36899999999999999},"max":{"1min":0.94499999999999995,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.65600000000000003}]},{"osd":6,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60799999999999998,"15min":0.61799999999999999},"min":{"1min":0.47799999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.96999999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63500000000000001,"15min":0.64800000000000002},"min":{"1min":0.436,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91800000000000004,"5min":0.92600000000000005,"15min":0.94699999999999995},"last":0.70899999999999996}]},{"osd":7,"last update":"Sat Apr 6 12:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.629,"15min":0.63700000000000001},"min":{"1min":0.46899999999999997,"5min":0.40400000000000003,"15min":0.33300000000000002},"max":{"1min":0.873,"5min":0.89000000000000001,"15min":0.93300000000000005},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65100000000000002,"15min":0.65700000000000003},"min":{"1min":0.52200000000000002,"5min":0.39500000000000002,"15min":0.373},"max":{"1min":0.93300000000000005,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.54600000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575034,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":1032192,"data_stored":506155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.47499999999999998,"5min":0.29399999999999998,"15min":0.23899999999999999},"max":{"1min":0.71099999999999997,"5min":0.80800000000000005,"15min":0.82999999999999996},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55200000000000005,"15min":0.55400000000000005},"min":{"1min":0.49099999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.82199999999999995},"last":0.58999999999999997}]},{"osd":1,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.59299999999999997,"15min":0.63500000000000001},"min":{"1min":0.48199999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74399999999999999,"5min":0.76900000000000002,"15min":4.3479999999999999},"last":0.629},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58699999999999997,"15min":0.623},"min":{"1min":0.48399999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77100000000000002,"5min":0.78400000000000003,"15min":4.3319999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.56299999999999994,"15min":0.56599999999999995},"min":{"1min":0.47399999999999998,"5min":0.313,"15min":0.311},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.78400000000000003},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.58099999999999996,"15min":0.58199999999999996},"min":{"1min":0.50900000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72399999999999998,"5min":0.74299999999999999,"15min":0.82199999999999995},"last":0.53000000000000003}]},{"osd":3,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55100000000000005,"15min":0.59199999999999997},"min":{"1min":0.432,"5min":0.36399999999999999,"15min":0.34899999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":4.1950000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.59699999999999998,"15min":0.60199999999999998},"min":{"1min":0.497,"5min":0.40500000000000003,"15min":0.36099999999999999},"max":{"1min":0.73699999999999999,"5min":0.75900000000000001,"15min":1.2330000000000001},"last":0.61299999999999999}]},{"osd":5,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.46500000000000002,"15min":0.48999999999999999},"min":{"1min":0.36199999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71099999999999997,"5min":0.76300000000000001,"15min":1.196},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.497,"15min":0.502},"min":{"1min":0.39400000000000002,"5min":0.33500000000000002,"15min":0.312},"max":{"1min":0.63900000000000001,"5min":0.74299999999999999,"15min":0.77000000000000002},"last":0.39800000000000002}]},{"osd":6,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.498,"15min":0.51100000000000001},"min":{"1min":0.33200000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.64200000000000002,"5min":0.754,"15min":1.022},"last":0.379},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.52500000000000002,"15min":0.54400000000000004},"min":{"1min":0.39500000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67400000000000004,"5min":0.73699999999999999,"15min":0.872},"last":0.49299999999999999}]},{"osd":7,"last update":"Sat Apr 6 12:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51200000000000001,"15min":0.54100000000000004},"min":{"1min":0.377,"5min":0.311,"15min":0.311},"max":{"1min":0.68500000000000005,"5min":0.79300000000000004,"15min":0.91500000000000004},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55000000000000004,"15min":0.57699999999999996},"min":{"1min":0.39300000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73599999999999999,"5min":0.752,"15min":1.0589999999999999},"last":0.44400000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575034,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46064,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325776,"statfs":{"total":96636764160,"available":96589594624,"internally_reserved":0,"allocated":1019904,"data_stored":503594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.36799999999999999,"5min":0.32700000000000001,"15min":0.24199999999999999},"max":{"1min":0.75900000000000001,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.29499999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.748,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.61199999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.30399999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55200000000000005,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.34499999999999997,"15min":0.30599999999999999},"max":{"1min":0.73599999999999999,"5min":0.74299999999999999,"15min":0.76200000000000001},"last":0.55800000000000005}]},{"osd":2,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56999999999999995,"15min":0.56200000000000006},"min":{"1min":0.33200000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.72199999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.629},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.40999999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72099999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.496}]},{"osd":3,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.33900000000000002,"5min":0.32500000000000001,"15min":0.31},"max":{"1min":0.77800000000000002,"5min":0.874,"15min":0.874},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.436,"5min":0.377,"15min":0.32300000000000001},"max":{"1min":0.73099999999999998,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.58899999999999997}]},{"osd":4,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.28299999999999997},"max":{"1min":0.755,"5min":0.755,"15min":0.81200000000000006},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54000000000000004,"15min":0.53300000000000003},"min":{"1min":0.42599999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73699999999999999,"5min":0.80000000000000004,"15min":0.91800000000000004},"last":0.503}]},{"osd":6,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49099999999999999,"15min":0.48599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.28000000000000003},"max":{"1min":0.63800000000000001,"5min":0.73899999999999999,"15min":0.88700000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.496,"15min":0.49299999999999999},"min":{"1min":0.33100000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64800000000000002,"5min":0.73199999999999998,"15min":0.83299999999999996},"last":0.47399999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.252},"max":{"1min":0.68999999999999995,"5min":0.78600000000000003,"15min":0.79100000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60999999999999999,"15min":0.60199999999999998},"min":{"1min":0.44800000000000001,"5min":0.41399999999999998,"15min":0.35199999999999998},"max":{"1min":0.72499999999999998,"5min":0.85299999999999998,"15min":0.85599999999999998},"last":0.54400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:27:30.111 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-06T12:27:30.111 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-06T12:27:30.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.275+0000 7f0353f10700 1 Processor -- start 2024-04-06T12:27:30.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.276+0000 7f0353f10700 1 -- start start 2024-04-06T12:27:30.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.277+0000 7f0353f10700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f034c0b4660 0x7f034c0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:30.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.277+0000 7f0353f10700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034c0b5030 con 0x7f034c0b2fc0 2024-04-06T12:27:30.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.277+0000 7f0353f10700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034c0cdbe0 con 0x7f034c15a550 2024-04-06T12:27:30.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.277+0000 7f0353f10700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f034c15a940 con 0x7f034c0b4660 2024-04-06T12:27:30.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.277+0000 7f03524ad700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f034c0b2fc0 0x7f034c0b33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52606/0 (socket says 172.21.15.189:52606) 2024-04-06T12:27:30.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.277+0000 7f03524ad700 1 -- 172.21.15.189:0/681951349 learned_addr learned my addr 172.21.15.189:0/681951349 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:30.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.277+0000 7f03514ab700 1 --2- 172.21.15.189:0/681951349 >> v2:172.21.15.192:3300/0 conn(0x7f034c0b4660 0x7f034c0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:30.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.277+0000 7f0350caa700 1 -- 172.21.15.189:0/681951349 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218324418 0 0) 0x7f034c0b5030 con 0x7f034c0b2fc0 2024-04-06T12:27:30.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.277+0000 7f0350caa700 1 -- 172.21.15.189:0/681951349 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03380035b0 con 0x7f034c0b2fc0 2024-04-06T12:27:30.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.277+0000 7f0350caa700 1 -- 172.21.15.189:0/681951349 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4035277186 0 0) 0x7f03380035b0 con 0x7f034c0b2fc0 2024-04-06T12:27:30.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.277+0000 7f0350caa700 1 -- 172.21.15.189:0/681951349 >> v2:172.21.15.192:3300/0 conn(0x7f034c0b4660 msgr2=0x7f034c0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:30.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.277+0000 7f0350caa700 1 --2- 172.21.15.189:0/681951349 >> v2:172.21.15.192:3300/0 conn(0x7f034c0b4660 0x7f034c0b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.277+0000 7f0350caa700 1 -- 172.21.15.189:0/681951349 >> v1:172.21.15.192:6789/0 conn(0x7f034c15a550 legacy=0x7f034c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:30.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.277+0000 7f0350caa700 1 -- 172.21.15.189:0/681951349 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034c15ac00 con 0x7f034c0b2fc0 2024-04-06T12:27:30.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.277+0000 7f0350caa700 1 -- 172.21.15.189:0/681951349 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f03480019f0 con 0x7f034c0b2fc0 2024-04-06T12:27:30.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.277+0000 7f0350caa700 1 -- 172.21.15.189:0/681951349 >> v1:172.21.15.189:6789/0 conn(0x7f034c0b2fc0 legacy=0x7f034c0b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:30.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.277+0000 7f0350caa700 1 --2- 172.21.15.189:0/681951349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0338003a00 0x7f0338003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:30.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.278+0000 7f0350caa700 1 --2- 172.21.15.189:0/681951349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0338004850 0x7f0338004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:30.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.278+0000 7f0350caa700 1 --2- 172.21.15.189:0/681951349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0338005210 0x7f033800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:30.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.278+0000 7f0350caa700 1 -- 172.21.15.189:0/681951349 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c15a940 con 0x7f0338004850 2024-04-06T12:27:30.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.278+0000 7f0350caa700 1 -- 172.21.15.189:0/681951349 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033800c0b0 con 0x7f0338003a00 2024-04-06T12:27:30.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.278+0000 7f0350caa700 1 -- 172.21.15.189:0/681951349 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f033800c270 con 0x7f0338005210 2024-04-06T12:27:30.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.278+0000 7f03514ab700 1 --2- 172.21.15.189:0/681951349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0338004850 0x7f0338004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:30.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.278+0000 7f0351cac700 1 --2- 172.21.15.189:0/681951349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0338003a00 0x7f0338003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:30.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.278+0000 7f03524ad700 1 --2- 172.21.15.189:0/681951349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0338005210 0x7f033800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:30.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.278+0000 7f03514ab700 1 -- 172.21.15.189:0/681951349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0338005210 msgr2=0x7f033800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:30.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.278+0000 7f03514ab700 1 --2- 172.21.15.189:0/681951349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0338005210 0x7f033800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.278+0000 7f03514ab700 1 -- 172.21.15.189:0/681951349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0338003a00 msgr2=0x7f0338003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:30.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.278+0000 7f03514ab700 1 --2- 172.21.15.189:0/681951349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0338003a00 0x7f0338003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.278+0000 7f03514ab700 1 -- 172.21.15.189:0/681951349 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03380035b0 con 0x7f0338004850 2024-04-06T12:27:30.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.278+0000 7f03524ad700 1 -- 172.21.15.189:0/681951349 reap_dead start 2024-04-06T12:27:30.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.278+0000 7f03514ab700 1 --2- 172.21.15.189:0/681951349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0338004850 0x7f0338004cd0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f033c009b60 tx=0x7f0338005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3946b4b0bce78c37 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:30.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.278+0000 7f0350caa700 1 -- 172.21.15.189:0/681951349 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033c00e6c0 con 0x7f0338004850 2024-04-06T12:27:30.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.278+0000 7f0350caa700 1 -- 172.21.15.189:0/681951349 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f033c00e820 con 0x7f0338004850 2024-04-06T12:27:30.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.278+0000 7f0350caa700 1 -- 172.21.15.189:0/681951349 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033c00eb90 con 0x7f0338004850 2024-04-06T12:27:30.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.279+0000 7f0353f10700 1 -- 172.21.15.189:0/681951349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0338004850 msgr2=0x7f0338004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:30.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.279+0000 7f0353f10700 1 --2- 172.21.15.189:0/681951349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0338004850 0x7f0338004cd0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f033c009b60 tx=0x7f0338005880 comp rx=0 tx=0).stop 2024-04-06T12:27:30.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.279+0000 7f0353f10700 1 -- 172.21.15.189:0/681951349 shutdown_connections 2024-04-06T12:27:30.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.279+0000 7f0353f10700 1 --2- 172.21.15.189:0/681951349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0338004850 0x7f0338004cd0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.279+0000 7f0353f10700 1 -- 172.21.15.189:0/681951349 >> 172.21.15.189:0/681951349 conn(0x7f034c0d00b0 msgr2=0x7f034c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:30.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.279+0000 7f0353f10700 1 -- 172.21.15.189:0/681951349 shutdown_connections 2024-04-06T12:27:30.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.279+0000 7f0353f10700 1 -- 172.21.15.189:0/681951349 wait complete. 2024-04-06T12:27:30.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.280+0000 7f0353f10700 1 Processor -- start 2024-04-06T12:27:30.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.280+0000 7f0353f10700 1 -- start start 2024-04-06T12:27:30.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.281+0000 7f0353f10700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0338004850 0x7f034c1540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:30.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.281+0000 7f0353f10700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f034c151400 0x7f034c150aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:30.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.281+0000 7f0353f10700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f034c1f4fa0 0x7f034c06a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:30.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.281+0000 7f0353f10700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c150fe0 con 0x7f034c1f4fa0 2024-04-06T12:27:30.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.281+0000 7f0353f10700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c134360 con 0x7f034c151400 2024-04-06T12:27:30.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.281+0000 7f0353f10700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034c1f44c0 con 0x7f0338004850 2024-04-06T12:27:30.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.281+0000 7f03524ad700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f034c1f4fa0 0x7f034c06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:30.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.281+0000 7f03524ad700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f034c1f4fa0 0x7f034c06a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33038/0 (socket says 172.21.15.189:33038) 2024-04-06T12:27:30.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.282+0000 7f03524ad700 1 -- 172.21.15.189:0/582551987 learned_addr learned my addr 172.21.15.189:0/582551987 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:30.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.282+0000 7f03524ad700 1 -- 172.21.15.189:0/582551987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0338004850 msgr2=0x7f034c1540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:30.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.282+0000 7f03524ad700 1 --2- 172.21.15.189:0/582551987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0338004850 0x7f034c1540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.282+0000 7f03524ad700 1 -- 172.21.15.189:0/582551987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f034c151400 msgr2=0x7f034c150aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:30.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.282+0000 7f03514ab700 1 --2- 172.21.15.189:0/582551987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f034c151400 0x7f034c150aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:30.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.282+0000 7f03524ad700 1 --2- 172.21.15.189:0/582551987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f034c151400 0x7f034c150aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.282+0000 7f03524ad700 1 -- 172.21.15.189:0/582551987 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034c1f47a0 con 0x7f034c1f4fa0 2024-04-06T12:27:30.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.282+0000 7f03514ab700 1 --2- 172.21.15.189:0/582551987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f034c151400 0x7f034c150aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:27:30.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.282+0000 7f03524ad700 1 --2- 172.21.15.189:0/582551987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f034c1f4fa0 0x7f034c06a730 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f034c0d1110 tx=0x7f0348007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:30.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.282+0000 7f03427fc700 1 -- 172.21.15.189:0/582551987 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034800ec60 con 0x7f034c1f4fa0 2024-04-06T12:27:30.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.282+0000 7f03427fc700 1 -- 172.21.15.189:0/582551987 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034800edc0 con 0x7f034c1f4fa0 2024-04-06T12:27:30.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.282+0000 7f0353f10700 1 -- 172.21.15.189:0/582551987 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034c1f4a80 con 0x7f034c1f4fa0 2024-04-06T12:27:30.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.282+0000 7f03427fc700 1 -- 172.21.15.189:0/582551987 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0348013660 con 0x7f034c1f4fa0 2024-04-06T12:27:30.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.282+0000 7f0353f10700 1 -- 172.21.15.189:0/582551987 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034c1340f0 con 0x7f034c1f4fa0 2024-04-06T12:27:30.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.283+0000 7f0353f10700 1 -- 172.21.15.189:0/582551987 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f034c1f4320 con 0x7f034c1f4fa0 2024-04-06T12:27:30.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.284+0000 7f03427fc700 1 -- 172.21.15.189:0/582551987 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f03480137c0 con 0x7f034c1f4fa0 2024-04-06T12:27:30.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.284+0000 7f03427fc700 1 --2- 172.21.15.189:0/582551987 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f033006adb0 0x7f033006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:30.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.284+0000 7f03427fc700 1 -- 172.21.15.189:0/582551987 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0348089b40 con 0x7f034c1f4fa0 2024-04-06T12:27:30.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.284+0000 7f0351cac700 1 --2- 172.21.15.189:0/582551987 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f033006adb0 0x7f033006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:30.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.285+0000 7f0351cac700 1 --2- 172.21.15.189:0/582551987 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f033006adb0 0x7f033006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f034c068450 tx=0x7f0344009040 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:30.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.288+0000 7f03427fc700 1 -- 172.21.15.189:0/582551987 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f034c1f4320 con 0x7f034c1f4fa0 2024-04-06T12:27:30.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.564+0000 7f0353f10700 1 -- 172.21.15.189:0/582551987 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f034c1f4320 con 0x7f034c1f4fa0 2024-04-06T12:27:30.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.567+0000 7f03427fc700 1 -- 172.21.15.189:0/582551987 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+0 (secure 0 0 0) 0x7f034c1f4320 con 0x7f034c1f4fa0 2024-04-06T12:27:30.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.570+0000 7f0353f10700 1 -- 172.21.15.189:0/582551987 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f033006adb0 msgr2=0x7f033006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:30.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.570+0000 7f0353f10700 1 --2- 172.21.15.189:0/582551987 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f033006adb0 0x7f033006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f034c068450 tx=0x7f0344009040 comp rx=0 tx=0).stop 2024-04-06T12:27:30.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.570+0000 7f0353f10700 1 -- 172.21.15.189:0/582551987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f034c1f4fa0 msgr2=0x7f034c06a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:30.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.570+0000 7f0353f10700 1 --2- 172.21.15.189:0/582551987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f034c1f4fa0 0x7f034c06a730 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f034c0d1110 tx=0x7f0348007c90 comp rx=0 tx=0).stop 2024-04-06T12:27:30.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.570+0000 7f0353f10700 1 -- 172.21.15.189:0/582551987 shutdown_connections 2024-04-06T12:27:30.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.570+0000 7f0353f10700 1 --2- 172.21.15.189:0/582551987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f034c1f4fa0 0x7f034c06a730 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.570+0000 7f0353f10700 1 --2- 172.21.15.189:0/582551987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0338004850 0x7f034c1540f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.570+0000 7f0353f10700 1 --2- 172.21.15.189:0/582551987 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f033006adb0 0x7f033006d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.570+0000 7f0353f10700 1 --2- 172.21.15.189:0/582551987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f034c151400 0x7f034c150aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.570+0000 7f0353f10700 1 -- 172.21.15.189:0/582551987 >> 172.21.15.189:0/582551987 conn(0x7f034c0d00b0 msgr2=0x7f034c06f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:30.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.570+0000 7f0353f10700 1 -- 172.21.15.189:0/582551987 shutdown_connections 2024-04-06T12:27:30.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.570+0000 7f0353f10700 1 -- 172.21.15.189:0/582551987 wait complete. 2024-04-06T12:27:30.570 INFO:teuthology.orchestra.run.smithi189.stderr:updated caps for client.0 2024-04-06T12:27:30.580 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:27:30.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.703+0000 7f93000be700 1 Processor -- start 2024-04-06T12:27:30.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.703+0000 7f93000be700 1 -- start start 2024-04-06T12:27:30.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.704+0000 7f93000be700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f92f815d7b0 0x7f92f80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:30.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.704+0000 7f93000be700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f80b3990 con 0x7f92f815e500 2024-04-06T12:27:30.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.704+0000 7f93000be700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f80b3060 con 0x7f92f806a730 2024-04-06T12:27:30.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.704+0000 7f93000be700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f815e950 con 0x7f92f815d7b0 2024-04-06T12:27:30.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.704+0000 7f92fde5a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f92f815d7b0 0x7f92f80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:30.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.704+0000 7f92fe65b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f92f815e500 0x7f92f80717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52608/0 (socket says 172.21.15.189:52608) 2024-04-06T12:27:30.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.704+0000 7f92fe65b700 1 -- 172.21.15.189:0/1825285405 learned_addr learned my addr 172.21.15.189:0/1825285405 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:30.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.704+0000 7f92fce58700 1 -- 172.21.15.189:0/1825285405 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213954147 0 0) 0x7f92f80b3990 con 0x7f92f815e500 2024-04-06T12:27:30.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.704+0000 7f92fce58700 1 -- 172.21.15.189:0/1825285405 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e40035b0 con 0x7f92f815e500 2024-04-06T12:27:30.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.705+0000 7f92fde5a700 1 -- 172.21.15.189:0/1825285405 >> v1:172.21.15.192:6789/0 conn(0x7f92f806a730 legacy=0x7f92f815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:30.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.705+0000 7f92fde5a700 1 -- 172.21.15.189:0/1825285405 >> v1:172.21.15.189:6789/0 conn(0x7f92f815e500 legacy=0x7f92f80717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:30.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.705+0000 7f92fde5a700 1 -- 172.21.15.189:0/1825285405 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f80b3340 con 0x7f92f815d7b0 2024-04-06T12:27:30.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.705+0000 7f92fde5a700 1 --2- 172.21.15.189:0/1825285405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92f815d7b0 0x7f92f80b3e20 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f92e8009840 tx=0x7f92e8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=761cd194e3959578 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:30.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.705+0000 7f92fce58700 1 -- 172.21.15.189:0/1825285405 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e8004230 con 0x7f92f815d7b0 2024-04-06T12:27:30.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.705+0000 7f92fce58700 1 -- 172.21.15.189:0/1825285405 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e8004390 con 0x7f92f815d7b0 2024-04-06T12:27:30.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.705+0000 7f92fce58700 1 -- 172.21.15.189:0/1825285405 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e80036a0 con 0x7f92f815d7b0 2024-04-06T12:27:30.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.705+0000 7f93000be700 1 -- 172.21.15.189:0/1825285405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92f815d7b0 msgr2=0x7f92f80b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:30.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.705+0000 7f93000be700 1 --2- 172.21.15.189:0/1825285405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92f815d7b0 0x7f92f80b3e20 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f92e8009840 tx=0x7f92e8009b50 comp rx=0 tx=0).stop 2024-04-06T12:27:30.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.705+0000 7f93000be700 1 -- 172.21.15.189:0/1825285405 shutdown_connections 2024-04-06T12:27:30.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.705+0000 7f93000be700 1 --2- 172.21.15.189:0/1825285405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92f815d7b0 0x7f92f80b3e20 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.705+0000 7f93000be700 1 -- 172.21.15.189:0/1825285405 >> 172.21.15.189:0/1825285405 conn(0x7f92f80d00b0 msgr2=0x7f92f80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:30.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.705+0000 7f93000be700 1 -- 172.21.15.189:0/1825285405 shutdown_connections 2024-04-06T12:27:30.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.706+0000 7f93000be700 1 -- 172.21.15.189:0/1825285405 wait complete. 2024-04-06T12:27:30.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.707+0000 7f93000be700 1 Processor -- start 2024-04-06T12:27:30.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.707+0000 7f93000be700 1 -- start start 2024-04-06T12:27:30.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.708+0000 7f93000be700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92f806a730 0x7f92f814bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:30.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.708+0000 7f93000be700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92f815d7b0 0x7f92f80cef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:30.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.708+0000 7f93000be700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92f815e500 0x7f92f814b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:30.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.708+0000 7f93000be700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f80c50d0 con 0x7f92f806a730 2024-04-06T12:27:30.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.708+0000 7f93000be700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f8149220 con 0x7f92f815e500 2024-04-06T12:27:30.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.708+0000 7f93000be700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f8149500 con 0x7f92f815d7b0 2024-04-06T12:27:30.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.708+0000 7f92fde5a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92f806a730 0x7f92f814bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:30.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.708+0000 7f92fde5a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92f806a730 0x7f92f814bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33052/0 (socket says 172.21.15.189:33052) 2024-04-06T12:27:30.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.708+0000 7f92fde5a700 1 -- 172.21.15.189:0/2644106793 learned_addr learned my addr 172.21.15.189:0/2644106793 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:30.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.708+0000 7f92fe65b700 1 --2- 172.21.15.189:0/2644106793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92f815e500 0x7f92f814b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:30.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.708+0000 7f92fde5a700 1 -- 172.21.15.189:0/2644106793 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92f815d7b0 msgr2=0x7f92f80cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:30.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.708+0000 7f92fd659700 1 --2- 172.21.15.189:0/2644106793 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92f815d7b0 0x7f92f80cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:30.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.708+0000 7f92fde5a700 1 --2- 172.21.15.189:0/2644106793 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92f815d7b0 0x7f92f80cef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.708+0000 7f92fde5a700 1 -- 172.21.15.189:0/2644106793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92f815e500 msgr2=0x7f92f814b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:30.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.708+0000 7f92fde5a700 1 --2- 172.21.15.189:0/2644106793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92f815e500 0x7f92f814b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.708+0000 7f92fde5a700 1 -- 172.21.15.189:0/2644106793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e80094a0 con 0x7f92f806a730 2024-04-06T12:27:30.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.708+0000 7f92fde5a700 1 --2- 172.21.15.189:0/2644106793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92f806a730 0x7f92f814bdb0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f92e8004000 tx=0x7f92e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:30.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.709+0000 7f92ee7fc700 1 -- 172.21.15.189:0/2644106793 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e8011630 con 0x7f92f806a730 2024-04-06T12:27:30.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.709+0000 7f92ee7fc700 1 -- 172.21.15.189:0/2644106793 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e801e070 con 0x7f92f806a730 2024-04-06T12:27:30.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.709+0000 7f93000be700 1 -- 172.21.15.189:0/2644106793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f81497e0 con 0x7f92f806a730 2024-04-06T12:27:30.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.709+0000 7f92ee7fc700 1 -- 172.21.15.189:0/2644106793 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e80119b0 con 0x7f92f806a730 2024-04-06T12:27:30.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.709+0000 7f93000be700 1 -- 172.21.15.189:0/2644106793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f814b9c0 con 0x7f92f806a730 2024-04-06T12:27:30.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.709+0000 7f93000be700 1 -- 172.21.15.189:0/2644106793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f8067bb0 con 0x7f92f806a730 2024-04-06T12:27:30.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.710+0000 7f92ee7fc700 1 -- 172.21.15.189:0/2644106793 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f92e8011b10 con 0x7f92f806a730 2024-04-06T12:27:30.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.710+0000 7f92ee7fc700 1 --2- 172.21.15.189:0/2644106793 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f92dc06adb0 0x7f92dc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:30.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.710+0000 7f92ee7fc700 1 -- 172.21.15.189:0/2644106793 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f92e808a790 con 0x7f92f806a730 2024-04-06T12:27:30.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.711+0000 7f92fd659700 1 --2- 172.21.15.189:0/2644106793 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f92dc06adb0 0x7f92dc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:30.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.711+0000 7f92fd659700 1 --2- 172.21.15.189:0/2644106793 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f92dc06adb0 0x7f92dc06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f92f80cfc50 tx=0x7f92f4009040 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:30.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.715+0000 7f92ee7fc700 1 -- 172.21.15.189:0/2644106793 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f92e80557b0 con 0x7f92f806a730 2024-04-06T12:27:30.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.952+0000 7f93000be700 1 -- 172.21.15.189:0/2644106793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92f8074990 con 0x7f92f806a730 2024-04-06T12:27:30.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.953+0000 7f92ee7fc700 1 -- 172.21.15.189:0/2644106793 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4747 (secure 0 0 0) 0x7f92e8058dd0 con 0x7f92f806a730 2024-04-06T12:27:30.951 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:30.951 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5762,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/2827078766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2827078766},{"type":"v1","addr":"172.21.15.192:6833","nonce":2827078766}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5765,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1558111999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1558111999},{"type":"v1","addr":"172.21.15.192:6835","nonce":1558111999}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3448511259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3448511259},{"type":"v1","addr":"172.21.15.189:6837","nonce":3448511259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:27:21.031012+0000","modified":"2024-04-06T12:27:24.038778+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/1085319880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1085319880},{"type":"v1","addr":"172.21.15.189:6836","nonce":1085319880}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-06T12:27:30.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.955+0000 7f93000be700 1 -- 172.21.15.189:0/2644106793 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f92dc06adb0 msgr2=0x7f92dc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:30.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.955+0000 7f93000be700 1 --2- 172.21.15.189:0/2644106793 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f92dc06adb0 0x7f92dc06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f92f80cfc50 tx=0x7f92f4009040 comp rx=0 tx=0).stop 2024-04-06T12:27:30.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.955+0000 7f93000be700 1 -- 172.21.15.189:0/2644106793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92f806a730 msgr2=0x7f92f814bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:30.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.955+0000 7f93000be700 1 --2- 172.21.15.189:0/2644106793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92f806a730 0x7f92f814bdb0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f92e8004000 tx=0x7f92e8004750 comp rx=0 tx=0).stop 2024-04-06T12:27:30.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.955+0000 7f93000be700 1 -- 172.21.15.189:0/2644106793 shutdown_connections 2024-04-06T12:27:30.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.955+0000 7f93000be700 1 --2- 172.21.15.189:0/2644106793 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f92dc06adb0 0x7f92dc06d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.955+0000 7f93000be700 1 --2- 172.21.15.189:0/2644106793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92f815e500 0x7f92f814b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.955+0000 7f93000be700 1 --2- 172.21.15.189:0/2644106793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92f806a730 0x7f92f814bdb0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.955+0000 7f93000be700 1 --2- 172.21.15.189:0/2644106793 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92f815d7b0 0x7f92f80cef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:30.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.955+0000 7f93000be700 1 -- 172.21.15.189:0/2644106793 >> 172.21.15.189:0/2644106793 conn(0x7f92f80d00b0 msgr2=0x7f92f80adf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:30.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.955+0000 7f93000be700 1 -- 172.21.15.189:0/2644106793 shutdown_connections 2024-04-06T12:27:30.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:30.955+0000 7f93000be700 1 -- 172.21.15.189:0/2644106793 wait complete. 2024-04-06T12:27:30.954 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 31 2024-04-06T12:27:31.368 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 31, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-06T12:27:21.031012+0000', 'modified': '2024-04-06T12:27:24.038778+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6582}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6582': {'gid': 6582, 'name': 'c', 'rank': 0, 'incarnation': 28, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.189:6836/1085319880', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 1085319880}, {'type': 'v1', 'addr': '172.21.15.189:6836', 'nonce': 1085319880}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-06T12:27:31.368 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-06T12:27:31.368 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-06T12:27:31.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.494+0000 7f52c354a700 1 Processor -- start 2024-04-06T12:27:31.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.494+0000 7f52c354a700 1 -- start start 2024-04-06T12:27:31.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.495+0000 7f52c354a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52bc0b4af0 0x7f52bc0b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.495+0000 7f52c354a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52bc0cdbe0 con 0x7f52bc15a550 2024-04-06T12:27:31.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.495+0000 7f52c354a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52bc15a940 con 0x7f52bc0b4660 2024-04-06T12:27:31.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.495+0000 7f52c354a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc15ac00 con 0x7f52bc0b4af0 2024-04-06T12:27:31.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.495+0000 7f52c12e6700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f52bc15a550 0x7f52bc0cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52620/0 (socket says 172.21.15.189:52620) 2024-04-06T12:27:31.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.495+0000 7f52c12e6700 1 -- 172.21.15.189:0/2159015616 learned_addr learned my addr 172.21.15.189:0/2159015616 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:31.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.495+0000 7f52c1ae7700 1 --2- 172.21.15.189:0/2159015616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52bc0b4af0 0x7f52bc0b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.495+0000 7f52b3fff700 1 -- 172.21.15.189:0/2159015616 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584483082 0 0) 0x7f52bc0cdbe0 con 0x7f52bc15a550 2024-04-06T12:27:31.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.495+0000 7f52b3fff700 1 -- 172.21.15.189:0/2159015616 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52a80035b0 con 0x7f52bc15a550 2024-04-06T12:27:31.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.495+0000 7f52b3fff700 1 -- 172.21.15.189:0/2159015616 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584920313 0 0) 0x7f52bc15a940 con 0x7f52bc0b4660 2024-04-06T12:27:31.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.495+0000 7f52b3fff700 1 -- 172.21.15.189:0/2159015616 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52bc0cdbe0 con 0x7f52bc0b4660 2024-04-06T12:27:31.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.495+0000 7f52b3fff700 1 -- 172.21.15.189:0/2159015616 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2831900844 0 0) 0x7f52a80035b0 con 0x7f52bc15a550 2024-04-06T12:27:31.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.495+0000 7f52c1ae7700 1 -- 172.21.15.189:0/2159015616 >> v1:172.21.15.192:6789/0 conn(0x7f52bc0b4660 legacy=0x7f52bc0b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:31.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.495+0000 7f52c1ae7700 1 -- 172.21.15.189:0/2159015616 >> v1:172.21.15.189:6789/0 conn(0x7f52bc15a550 legacy=0x7f52bc0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:31.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.495+0000 7f52c1ae7700 1 -- 172.21.15.189:0/2159015616 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52bc15aee0 con 0x7f52bc0b4af0 2024-04-06T12:27:31.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.496+0000 7f52c1ae7700 1 --2- 172.21.15.189:0/2159015616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52bc0b4af0 0x7f52bc0b36d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f52b4009dc0 tx=0x7f52b400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a68cf3d245a7826 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:31.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.496+0000 7f52b3fff700 1 -- 172.21.15.189:0/2159015616 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b400e070 con 0x7f52bc0b4af0 2024-04-06T12:27:31.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.496+0000 7f52b3fff700 1 -- 172.21.15.189:0/2159015616 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b4003a20 con 0x7f52bc0b4af0 2024-04-06T12:27:31.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.496+0000 7f52b3fff700 1 -- 172.21.15.189:0/2159015616 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b8001670 con 0x7f52bc0b4af0 2024-04-06T12:27:31.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.496+0000 7f52c354a700 1 -- 172.21.15.189:0/2159015616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52bc0b4af0 msgr2=0x7f52bc0b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:31.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.496+0000 7f52c354a700 1 --2- 172.21.15.189:0/2159015616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52bc0b4af0 0x7f52bc0b36d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f52b4009dc0 tx=0x7f52b400b330 comp rx=0 tx=0).stop 2024-04-06T12:27:31.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.496+0000 7f52c354a700 1 -- 172.21.15.189:0/2159015616 shutdown_connections 2024-04-06T12:27:31.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.496+0000 7f52c354a700 1 --2- 172.21.15.189:0/2159015616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52bc0b4af0 0x7f52bc0b36d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.496+0000 7f52c354a700 1 -- 172.21.15.189:0/2159015616 >> 172.21.15.189:0/2159015616 conn(0x7f52bc0d00b0 msgr2=0x7f52bc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:31.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.496+0000 7f52c354a700 1 -- 172.21.15.189:0/2159015616 shutdown_connections 2024-04-06T12:27:31.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.496+0000 7f52c354a700 1 -- 172.21.15.189:0/2159015616 wait complete. 2024-04-06T12:27:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.498+0000 7f52c354a700 1 Processor -- start 2024-04-06T12:27:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.498+0000 7f52c354a700 1 -- start start 2024-04-06T12:27:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.499+0000 7f52c354a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52bc0b4660 0x7f52bc0ac4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.499+0000 7f52c354a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52bc0b4af0 0x7f52bc1537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.499+0000 7f52c354a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52bc0a9a90 0x7f52bc0b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.499+0000 7f52c354a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc15c670 con 0x7f52bc0b4660 2024-04-06T12:27:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.499+0000 7f52c354a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc15b9f0 con 0x7f52bc0a9a90 2024-04-06T12:27:31.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.499+0000 7f52c354a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc15bcd0 con 0x7f52bc0b4af0 2024-04-06T12:27:31.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.499+0000 7f52c12e6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52bc0b4660 0x7f52bc0ac4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.499+0000 7f52c12e6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52bc0b4660 0x7f52bc0ac4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33054/0 (socket says 172.21.15.189:33054) 2024-04-06T12:27:31.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.499+0000 7f52c12e6700 1 -- 172.21.15.189:0/1327480344 learned_addr learned my addr 172.21.15.189:0/1327480344 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:31.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.499+0000 7f52c1ae7700 1 --2- 172.21.15.189:0/1327480344 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52bc0a9a90 0x7f52bc0b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.499+0000 7f52c12e6700 1 -- 172.21.15.189:0/1327480344 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52bc0b4af0 msgr2=0x7f52bc1537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:31.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.499+0000 7f52c0ae5700 1 --2- 172.21.15.189:0/1327480344 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52bc0b4af0 0x7f52bc1537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.499+0000 7f52c12e6700 1 --2- 172.21.15.189:0/1327480344 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52bc0b4af0 0x7f52bc1537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.499+0000 7f52c12e6700 1 -- 172.21.15.189:0/1327480344 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52bc0a9a90 msgr2=0x7f52bc0b0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:31.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.499+0000 7f52c12e6700 1 --2- 172.21.15.189:0/1327480344 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52bc0a9a90 0x7f52bc0b0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.499+0000 7f52c12e6700 1 -- 172.21.15.189:0/1327480344 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b4009a70 con 0x7f52bc0b4660 2024-04-06T12:27:31.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.499+0000 7f52c1ae7700 1 --2- 172.21.15.189:0/1327480344 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52bc0a9a90 0x7f52bc0b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:27:31.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.499+0000 7f52c12e6700 1 --2- 172.21.15.189:0/1327480344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52bc0b4660 0x7f52bc0ac4c0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f52b800c6d0 tx=0x7f52b800ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:31.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.500+0000 7f52b1ffb700 1 -- 172.21.15.189:0/1327480344 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b8004c80 con 0x7f52bc0b4660 2024-04-06T12:27:31.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.500+0000 7f52b1ffb700 1 -- 172.21.15.189:0/1327480344 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b8004de0 con 0x7f52bc0b4660 2024-04-06T12:27:31.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.500+0000 7f52c354a700 1 -- 172.21.15.189:0/1327480344 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52bc15bfb0 con 0x7f52bc0b4660 2024-04-06T12:27:31.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.500+0000 7f52b1ffb700 1 -- 172.21.15.189:0/1327480344 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b8007310 con 0x7f52bc0b4660 2024-04-06T12:27:31.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.500+0000 7f52c354a700 1 -- 172.21.15.189:0/1327480344 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52bc153d00 con 0x7f52bc0b4660 2024-04-06T12:27:31.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.501+0000 7f52b1ffb700 1 -- 172.21.15.189:0/1327480344 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f52b8007470 con 0x7f52bc0b4660 2024-04-06T12:27:31.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.501+0000 7f52b1ffb700 1 --2- 172.21.15.189:0/1327480344 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f52a006adb0 0x7f52a006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.501+0000 7f52b1ffb700 1 -- 172.21.15.189:0/1327480344 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f52b8089400 con 0x7f52bc0b4660 2024-04-06T12:27:31.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.502+0000 7f52c0ae5700 1 --2- 172.21.15.189:0/1327480344 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f52a006adb0 0x7f52a006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.502+0000 7f52c0ae5700 1 --2- 172.21.15.189:0/1327480344 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f52a006adb0 0x7f52a006d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f52bc0b1c70 tx=0x7f52ac009040 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:31.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.504+0000 7f52cf6feb80 1 Processor -- start 2024-04-06T12:27:31.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.504+0000 7f52cf6feb80 1 -- start start 2024-04-06T12:27:31.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.504+0000 7f52cf6feb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938ee2570 0x55b938ee3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.504+0000 7f52cf6feb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938f0ad50 0x55b938ee2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.504+0000 7f52cf6feb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 0x55b938f29860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.504+0000 7f52cf6feb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b938e3a940 con 0x55b938ee41d0 2024-04-06T12:27:31.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.504+0000 7f52cf6feb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b938e38300 con 0x55b938ee2570 2024-04-06T12:27:31.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.504+0000 7f52cf6feb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b938e33bf0 con 0x55b938f0ad50 2024-04-06T12:27:31.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.504+0000 7f52c1ae7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 0x55b938f29860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.504+0000 7f52c1ae7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 0x55b938f29860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33064/0 (socket says 172.21.15.189:33064) 2024-04-06T12:27:31.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.504+0000 7f52c1ae7700 1 -- 172.21.15.189:0/3157693829 learned_addr learned my addr 172.21.15.189:0/3157693829 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:31.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.505+0000 7f52c0ae5700 1 --2- 172.21.15.189:0/3157693829 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938f0ad50 0x55b938ee2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.505+0000 7f52c1ae7700 1 -- 172.21.15.189:0/3157693829 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938f0ad50 msgr2=0x55b938ee2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:31.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.505+0000 7f52c12e6700 1 --2- 172.21.15.189:0/3157693829 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938ee2570 0x55b938ee3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.505+0000 7f52c1ae7700 1 --2- 172.21.15.189:0/3157693829 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938f0ad50 0x55b938ee2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.505+0000 7f52c1ae7700 1 -- 172.21.15.189:0/3157693829 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938ee2570 msgr2=0x55b938ee3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:31.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.505+0000 7f52c1ae7700 1 --2- 172.21.15.189:0/3157693829 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938ee2570 0x55b938ee3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.505+0000 7f52c1ae7700 1 -- 172.21.15.189:0/3157693829 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b938c61b10 con 0x55b938ee41d0 2024-04-06T12:27:31.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.505+0000 7f52c1ae7700 1 --2- 172.21.15.189:0/3157693829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 0x55b938f29860 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f52b4004000 tx=0x7f52b4010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a302e3559936a8a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:31.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.505+0000 7f52a6ffd700 1 -- 172.21.15.189:0/3157693829 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b400ba20 con 0x55b938ee41d0 2024-04-06T12:27:31.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.505+0000 7f52a6ffd700 1 -- 172.21.15.189:0/3157693829 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b400bb80 con 0x55b938ee41d0 2024-04-06T12:27:31.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.505+0000 7f52a6ffd700 1 -- 172.21.15.189:0/3157693829 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b4004030 con 0x55b938ee41d0 2024-04-06T12:27:31.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.506+0000 7f52cf6feb80 1 -- 172.21.15.189:0/3157693829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 msgr2=0x55b938f29860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:31.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.506+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/3157693829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 0x55b938f29860 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f52b4004000 tx=0x7f52b4010000 comp rx=0 tx=0).stop 2024-04-06T12:27:31.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.506+0000 7f52cf6feb80 1 -- 172.21.15.189:0/3157693829 shutdown_connections 2024-04-06T12:27:31.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.506+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/3157693829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 0x55b938f29860 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.506+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/3157693829 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938ee2570 0x55b938ee3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.506+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/3157693829 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938f0ad50 0x55b938ee2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.506+0000 7f52cf6feb80 1 -- 172.21.15.189:0/3157693829 >> 172.21.15.189:0/3157693829 conn(0x55b938e41650 msgr2=0x55b938f283e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:31.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.506+0000 7f52cf6feb80 1 -- 172.21.15.189:0/3157693829 shutdown_connections 2024-04-06T12:27:31.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.506+0000 7f52cf6feb80 1 -- 172.21.15.189:0/3157693829 wait complete. 2024-04-06T12:27:31.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.506+0000 7f52cf6feb80 1 -- start start 2024-04-06T12:27:31.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.506+0000 7f52cf6feb80 1 Processor -- start 2024-04-06T12:27:31.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.507+0000 7f52cf6feb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938ee2570 0x55b938f3cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.507+0000 7f52cf6feb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 0x55b938ee0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.507+0000 7f52cf6feb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938f0ad50 0x55b938ee0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.507+0000 7f52cf6feb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b938dcfd30 con 0x55b938ee41d0 2024-04-06T12:27:31.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.507+0000 7f52cf6feb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b938dcf9d0 con 0x55b938f0ad50 2024-04-06T12:27:31.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.507+0000 7f52cf6feb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b938cb7650 con 0x55b938ee2570 2024-04-06T12:27:31.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.507+0000 7f52cf6feb80 10 client.0 _ll_register_callbacks cb 0x55b938dd9420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:27:31.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.507+0000 7f52c0ae5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 0x55b938ee0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.507+0000 7f52c0ae5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 0x55b938ee0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33080/0 (socket says 172.21.15.189:33080) 2024-04-06T12:27:31.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.507+0000 7f52c0ae5700 1 -- 172.21.15.189:0/2730488877 learned_addr learned my addr 172.21.15.189:0/2730488877 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:31.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f52c1ae7700 1 --2- 172.21.15.189:0/2730488877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938f0ad50 0x55b938ee0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f52c12e6700 1 --2- 172.21.15.189:0/2730488877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938ee2570 0x55b938f3cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f52c0ae5700 1 -- 172.21.15.189:0/2730488877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938ee2570 msgr2=0x55b938f3cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:31.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f52c0ae5700 1 --2- 172.21.15.189:0/2730488877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938ee2570 0x55b938f3cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f52c0ae5700 1 -- 172.21.15.189:0/2730488877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938f0ad50 msgr2=0x55b938ee0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:31.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f52c0ae5700 1 --2- 172.21.15.189:0/2730488877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938f0ad50 0x55b938ee0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f52c0ae5700 1 -- 172.21.15.189:0/2730488877 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52bc15aee0 con 0x55b938ee41d0 2024-04-06T12:27:31.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f52c1ae7700 1 --2- 172.21.15.189:0/2730488877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938f0ad50 0x55b938ee0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:31.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f52c0ae5700 1 --2- 172.21.15.189:0/2730488877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 0x55b938ee0850 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f52ac0011e0 tx=0x7f52ac010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a35a786e851d56 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:31.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f529bfff700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:27:31.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f529bfff700 1 -- 172.21.15.189:0/2730488877 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac012dc0 con 0x55b938ee41d0 2024-04-06T12:27:31.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f529bfff700 1 -- 172.21.15.189:0/2730488877 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f52ac01f410 con 0x55b938ee41d0 2024-04-06T12:27:31.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f52cf6feb80 1 -- 172.21.15.189:0/2730488877 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b938df4330 con 0x55b938ee41d0 2024-04-06T12:27:31.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f529bfff700 1 -- 172.21.15.189:0/2730488877 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52ac01f5d0 con 0x55b938ee41d0 2024-04-06T12:27:31.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f52cf6feb80 10 client.6765 fetch_fsmap learned FSMap version 31 2024-04-06T12:27:31.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f52cf6feb80 1 -- 172.21.15.189:0/2730488877 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=2+}) v3 -- 0x55b938df9e50 con 0x55b938ee41d0 2024-04-06T12:27:31.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f529bfff700 1 -- 172.21.15.189:0/2730488877 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b938df4330 con 0x55b938ee41d0 2024-04-06T12:27:31.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f529bfff700 1 -- 172.21.15.189:0/2730488877 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f52ac01f730 con 0x55b938ee41d0 2024-04-06T12:27:31.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f52cf6feb80 10 client.6765 fetch_fsmap finished waiting for FSMap version 31 2024-04-06T12:27:31.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f52cf6feb80 10 client.6765 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-06T12:27:31.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f52cf6feb80 20 client.6765 populate_metadata read hostname 'smithi189' 2024-04-06T12:27:31.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/2730488877 >> [v2:172.21.15.189:6834/1085319880,v1:172.21.15.189:6836/1085319880] conn(0x55b938f23f40 0x55b938f214b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f52cf6feb80 4 client.6765 mds_command: new command op to 6582 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-06T12:27:31.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.508+0000 7f52cf6feb80 1 -- 172.21.15.189:0/2730488877 --> [v2:172.21.15.189:6834/1085319880,v1:172.21.15.189:6836/1085319880] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b938e2c6b0 con 0x55b938f23f40 2024-04-06T12:27:31.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.509+0000 7f52c12e6700 1 --2- 172.21.15.189:0/2730488877 >> [v2:172.21.15.189:6834/1085319880,v1:172.21.15.189:6836/1085319880] conn(0x55b938f23f40 0x55b938f214b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.509+0000 7f529bfff700 10 client.6765 ms_handle_connect on v2:172.21.15.189:6834/1085319880 2024-04-06T12:27:31.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.509+0000 7f52c12e6700 1 --2- 172.21.15.189:0/2730488877 >> [v2:172.21.15.189:6834/1085319880,v1:172.21.15.189:6836/1085319880] conn(0x55b938f23f40 0x55b938f214b0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86d1ef5ae62d1724 server_cookie=7db923521499368e in_seq=0 out_seq=0 2024-04-06T12:27:31.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.509+0000 7f529bfff700 1 -- 172.21.15.189:0/2730488877 <== mds.0 v2:172.21.15.189:6834/1085319880 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b938e2c6b0 con 0x55b938f23f40 2024-04-06T12:27:31.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.509+0000 7f529bfff700 10 client.6765 handle_command_reply: tid=0 2024-04-06T12:27:31.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.509+0000 7f52cf6feb80 1 client.6765 shutdown 2024-04-06T12:27:31.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.510+0000 7f52cf6feb80 1 -- 172.21.15.189:0/2730488877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 msgr2=0x55b938ee0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:31.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.510+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/2730488877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 0x55b938ee0850 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f52ac0011e0 tx=0x7f52ac010000 comp rx=0 tx=0).stop 2024-04-06T12:27:31.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.510+0000 7f52cf6feb80 1 -- 172.21.15.189:0/2730488877 shutdown_connections 2024-04-06T12:27:31.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.510+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/2730488877 >> [v2:172.21.15.189:6834/1085319880,v1:172.21.15.189:6836/1085319880] conn(0x55b938f23f40 0x55b938f214b0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.510+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/2730488877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938f0ad50 0x55b938ee0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.510+0000 7f529bfff700 0 client.6765 ms_handle_reset on v2:172.21.15.189:6834/1085319880 2024-04-06T12:27:31.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.510+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/2730488877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938ee2570 0x55b938f3cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.510+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/2730488877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 0x55b938ee0850 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.510+0000 7f52cf6feb80 1 -- 172.21.15.189:0/2730488877 >> 172.21.15.189:0/2730488877 conn(0x55b938e41650 msgr2=0x55b938f283e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:31.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.510+0000 7f52cf6feb80 1 -- 172.21.15.189:0/2730488877 shutdown_connections 2024-04-06T12:27:31.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.510+0000 7f52cf6feb80 1 -- 172.21.15.189:0/2730488877 wait complete. 2024-04-06T12:27:31.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.510+0000 7f52cf6feb80 20 client.6765 trim_cache size 0 max 16384 2024-04-06T12:27:31.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.527+0000 7f52cf6feb80 1 Processor -- start 2024-04-06T12:27:31.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.528+0000 7f52cf6feb80 1 -- start start 2024-04-06T12:27:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.528+0000 7f52cf6feb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938ee2570 0x55b938ee0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.528+0000 7f52cf6feb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 0x55b938efe430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.528+0000 7f52cf6feb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938f0ad50 0x55b938edd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.528+0000 7f52cf6feb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b938eddb90 con 0x55b938ee41d0 2024-04-06T12:27:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.528+0000 7f52cf6feb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b938f47610 con 0x55b938f0ad50 2024-04-06T12:27:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.528+0000 7f52cf6feb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b938f29f00 con 0x55b938ee2570 2024-04-06T12:27:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.528+0000 7f52c0ae5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 0x55b938efe430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.528+0000 7f52c0ae5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 0x55b938efe430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33092/0 (socket says 172.21.15.189:33092) 2024-04-06T12:27:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.528+0000 7f52c0ae5700 1 -- 172.21.15.189:0/309943762 learned_addr learned my addr 172.21.15.189:0/309943762 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.529+0000 7f52c12e6700 1 --2- 172.21.15.189:0/309943762 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938ee2570 0x55b938ee0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.529+0000 7f52c1ae7700 1 --2- 172.21.15.189:0/309943762 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938f0ad50 0x55b938edd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.529+0000 7f52c0ae5700 1 -- 172.21.15.189:0/309943762 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938ee2570 msgr2=0x55b938ee0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.529+0000 7f52c0ae5700 1 --2- 172.21.15.189:0/309943762 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938ee2570 0x55b938ee0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.529+0000 7f52c0ae5700 1 -- 172.21.15.189:0/309943762 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938f0ad50 msgr2=0x55b938edd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.529+0000 7f52c0ae5700 1 --2- 172.21.15.189:0/309943762 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938f0ad50 0x55b938edd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.529+0000 7f52c0ae5700 1 -- 172.21.15.189:0/309943762 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b938cb9c80 con 0x55b938ee41d0 2024-04-06T12:27:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.529+0000 7f52c0ae5700 1 --2- 172.21.15.189:0/309943762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 0x55b938efe430 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f52ac007bd0 tx=0x7f52ac014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd6f266a62cd0de9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.529+0000 7f52a77fe700 1 -- 172.21.15.189:0/309943762 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac012c40 con 0x55b938ee41d0 2024-04-06T12:27:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.529+0000 7f52a77fe700 1 -- 172.21.15.189:0/309943762 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ac01a050 con 0x55b938ee41d0 2024-04-06T12:27:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.529+0000 7f52a77fe700 1 -- 172.21.15.189:0/309943762 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac01fc90 con 0x55b938ee41d0 2024-04-06T12:27:31.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.530+0000 7f52cf6feb80 1 -- 172.21.15.189:0/309943762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 msgr2=0x55b938efe430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:31.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.530+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/309943762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 0x55b938efe430 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f52ac007bd0 tx=0x7f52ac014000 comp rx=0 tx=0).stop 2024-04-06T12:27:31.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.530+0000 7f52cf6feb80 1 -- 172.21.15.189:0/309943762 shutdown_connections 2024-04-06T12:27:31.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.530+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/309943762 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938f0ad50 0x55b938edd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.530+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/309943762 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938ee2570 0x55b938ee0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.530+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/309943762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee41d0 0x55b938efe430 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.530+0000 7f52cf6feb80 1 -- 172.21.15.189:0/309943762 >> 172.21.15.189:0/309943762 conn(0x55b938e41650 msgr2=0x55b938f28b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:31.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.530+0000 7f52cf6feb80 1 -- 172.21.15.189:0/309943762 shutdown_connections 2024-04-06T12:27:31.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.530+0000 7f52cf6feb80 1 -- 172.21.15.189:0/309943762 wait complete. 2024-04-06T12:27:31.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.530+0000 7f52cf6feb80 1 -- start start 2024-04-06T12:27:31.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.530+0000 7f52cf6feb80 1 Processor -- start 2024-04-06T12:27:31.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.531+0000 7f52cf6feb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee2570 0x55b938f2b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.531+0000 7f52cf6feb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938ee41d0 0x55b938f2b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.531+0000 7f52cf6feb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938f0ad50 0x55b938f214a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.531+0000 7f52cf6feb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b938edfc90 con 0x55b938ee2570 2024-04-06T12:27:31.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.531+0000 7f52cf6feb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b938f2c2c0 con 0x55b938ee41d0 2024-04-06T12:27:31.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.531+0000 7f52cf6feb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b938edf9b0 con 0x55b938f0ad50 2024-04-06T12:27:31.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.531+0000 7f52cf6feb80 10 client.0 _ll_register_callbacks cb 0x55b938dd9420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:27:31.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.531+0000 7f52c12e6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee2570 0x55b938f2b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.531+0000 7f52c12e6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee2570 0x55b938f2b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33106/0 (socket says 172.21.15.189:33106) 2024-04-06T12:27:31.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.531+0000 7f52c12e6700 1 -- 172.21.15.189:0/2894599220 learned_addr learned my addr 172.21.15.189:0/2894599220 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:31.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.531+0000 7f52c1ae7700 1 --2- 172.21.15.189:0/2894599220 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938f0ad50 0x55b938f214a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.531+0000 7f52c0ae5700 1 --2- 172.21.15.189:0/2894599220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938ee41d0 0x55b938f2b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f52c12e6700 1 -- 172.21.15.189:0/2894599220 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938f0ad50 msgr2=0x55b938f214a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:31.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f52c12e6700 1 --2- 172.21.15.189:0/2894599220 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938f0ad50 0x55b938f214a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f52c12e6700 1 -- 172.21.15.189:0/2894599220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938ee41d0 msgr2=0x55b938f2b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:31.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f52c12e6700 1 --2- 172.21.15.189:0/2894599220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938ee41d0 0x55b938f2b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f52c12e6700 1 -- 172.21.15.189:0/2894599220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b938df9e50 con 0x55b938ee2570 2024-04-06T12:27:31.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f52c0ae5700 1 --2- 172.21.15.189:0/2894599220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938ee41d0 0x55b938f2b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:31.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f52c12e6700 1 --2- 172.21.15.189:0/2894599220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee2570 0x55b938f2b2d0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f52b8057fd0 tx=0x7f52b8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc49d2ecb46ec03 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:31.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f529bfff700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:27:31.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f529bfff700 1 -- 172.21.15.189:0/2894599220 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b805be70 con 0x55b938ee2570 2024-04-06T12:27:31.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f529bfff700 1 -- 172.21.15.189:0/2894599220 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f52b8060430 con 0x55b938ee2570 2024-04-06T12:27:31.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f52cf6feb80 1 -- 172.21.15.189:0/2894599220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f52b400bb80 con 0x55b938ee2570 2024-04-06T12:27:31.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f529bfff700 1 -- 172.21.15.189:0/2894599220 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52b80563e0 con 0x55b938ee2570 2024-04-06T12:27:31.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f52cf6feb80 10 client.6771 fetch_fsmap learned FSMap version 31 2024-04-06T12:27:31.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f52cf6feb80 1 -- 172.21.15.189:0/2894599220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=2+}) v3 -- 0x55b938c5ffe0 con 0x55b938ee2570 2024-04-06T12:27:31.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f529bfff700 1 -- 172.21.15.189:0/2894599220 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b807a080 con 0x55b938ee2570 2024-04-06T12:27:31.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f529bfff700 1 -- 172.21.15.189:0/2894599220 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f52b8054750 con 0x55b938ee2570 2024-04-06T12:27:31.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f52cf6feb80 10 client.6771 fetch_fsmap finished waiting for FSMap version 31 2024-04-06T12:27:31.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f52cf6feb80 10 client.6771 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-06T12:27:31.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f52cf6feb80 20 client.6771 populate_metadata read hostname 'smithi189' 2024-04-06T12:27:31.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/2894599220 >> [v2:172.21.15.189:6834/1085319880,v1:172.21.15.189:6836/1085319880] conn(0x55b938f21d40 0x55b938f93dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f52cf6feb80 4 client.6771 mds_command: new command op to 6582 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-06T12:27:31.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f52cf6feb80 1 -- 172.21.15.189:0/2894599220 --> [v2:172.21.15.189:6834/1085319880,v1:172.21.15.189:6836/1085319880] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b938c82ca0 con 0x55b938f21d40 2024-04-06T12:27:31.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.532+0000 7f52c0ae5700 1 --2- 172.21.15.189:0/2894599220 >> [v2:172.21.15.189:6834/1085319880,v1:172.21.15.189:6836/1085319880] conn(0x55b938f21d40 0x55b938f93dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.533+0000 7f529bfff700 10 client.6771 ms_handle_connect on v2:172.21.15.189:6834/1085319880 2024-04-06T12:27:31.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.533+0000 7f52c0ae5700 1 --2- 172.21.15.189:0/2894599220 >> [v2:172.21.15.189:6834/1085319880,v1:172.21.15.189:6836/1085319880] conn(0x55b938f21d40 0x55b938f93dc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92101b41769cc6cd server_cookie=a85e40d0e66df34 in_seq=0 out_seq=0 2024-04-06T12:27:31.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.533+0000 7f529bfff700 1 -- 172.21.15.189:0/2894599220 <== mds.0 v2:172.21.15.189:6834/1085319880 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x55b938c82ca0 con 0x55b938f21d40 2024-04-06T12:27:31.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.533+0000 7f529bfff700 10 client.6771 handle_command_reply: tid=0 2024-04-06T12:27:31.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.533+0000 7f52cf6feb80 1 client.6771 shutdown 2024-04-06T12:27:31.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.533+0000 7f52cf6feb80 1 -- 172.21.15.189:0/2894599220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee2570 msgr2=0x55b938f2b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:31.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.533+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/2894599220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee2570 0x55b938f2b2d0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f52b8057fd0 tx=0x7f52b8000bc0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.533+0000 7f52cf6feb80 1 -- 172.21.15.189:0/2894599220 shutdown_connections 2024-04-06T12:27:31.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.533+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/2894599220 >> [v2:172.21.15.189:6834/1085319880,v1:172.21.15.189:6836/1085319880] conn(0x55b938f21d40 0x55b938f93dc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.533+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/2894599220 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b938f0ad50 0x55b938f214a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.533+0000 7f529bfff700 0 client.6771 ms_handle_reset on v2:172.21.15.189:6834/1085319880 2024-04-06T12:27:31.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.533+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/2894599220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b938ee2570 0x55b938f2b2d0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.533+0000 7f52cf6feb80 1 --2- 172.21.15.189:0/2894599220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b938ee41d0 0x55b938f2b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.533+0000 7f52cf6feb80 1 -- 172.21.15.189:0/2894599220 >> 172.21.15.189:0/2894599220 conn(0x55b938e41650 msgr2=0x55b938f25270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:31.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.533+0000 7f52cf6feb80 1 -- 172.21.15.189:0/2894599220 shutdown_connections 2024-04-06T12:27:31.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.533+0000 7f52cf6feb80 1 -- 172.21.15.189:0/2894599220 wait complete. 2024-04-06T12:27:31.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.533+0000 7f52cf6feb80 20 client.6771 trim_cache size 0 max 16384 2024-04-06T12:27:31.539 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T12:27:31.539 INFO:teuthology.orchestra.run.smithi189.stdout: "cluster_fsid": "e30041de-e9a3-4351-8ca7-29d7f3e3728c", 2024-04-06T12:27:31.539 INFO:teuthology.orchestra.run.smithi189.stdout: "whoami": 0, 2024-04-06T12:27:31.539 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 6582, 2024-04-06T12:27:31.539 INFO:teuthology.orchestra.run.smithi189.stdout: "want_state": "up:active", 2024-04-06T12:27:31.539 INFO:teuthology.orchestra.run.smithi189.stdout: "state": "up:active", 2024-04-06T12:27:31.539 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs", 2024-04-06T12:27:31.539 INFO:teuthology.orchestra.run.smithi189.stdout: "rank_uptime": 10.501734056, 2024-04-06T12:27:31.539 INFO:teuthology.orchestra.run.smithi189.stdout: "mdsmap_epoch": 31, 2024-04-06T12:27:31.539 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch": 84, 2024-04-06T12:27:31.539 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch_barrier": 84, 2024-04-06T12:27:31.539 INFO:teuthology.orchestra.run.smithi189.stdout: "uptime": 14.835328854 2024-04-06T12:27:31.539 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T12:27:31.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.534+0000 7f52c354a700 1 -- 172.21.15.189:0/1327480344 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f52a006adb0 msgr2=0x7f52a006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:31.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.534+0000 7f52c354a700 1 --2- 172.21.15.189:0/1327480344 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f52a006adb0 0x7f52a006d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f52bc0b1c70 tx=0x7f52ac009040 comp rx=0 tx=0).stop 2024-04-06T12:27:31.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.534+0000 7f52c354a700 1 -- 172.21.15.189:0/1327480344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52bc0b4660 msgr2=0x7f52bc0ac4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:31.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.534+0000 7f52c354a700 1 --2- 172.21.15.189:0/1327480344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52bc0b4660 0x7f52bc0ac4c0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f52b800c6d0 tx=0x7f52b800ca90 comp rx=0 tx=0).stop 2024-04-06T12:27:31.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.534+0000 7f52c354a700 1 -- 172.21.15.189:0/1327480344 shutdown_connections 2024-04-06T12:27:31.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.534+0000 7f52c354a700 1 --2- 172.21.15.189:0/1327480344 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f52a006adb0 0x7f52a006d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.534+0000 7f52c354a700 1 --2- 172.21.15.189:0/1327480344 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52bc0a9a90 0x7f52bc0b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.534+0000 7f52c354a700 1 --2- 172.21.15.189:0/1327480344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52bc0b4660 0x7f52bc0ac4c0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.534+0000 7f52c354a700 1 --2- 172.21.15.189:0/1327480344 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52bc0b4af0 0x7f52bc1537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.534+0000 7f52c354a700 1 -- 172.21.15.189:0/1327480344 >> 172.21.15.189:0/1327480344 conn(0x7f52bc0d00b0 msgr2=0x7f52bc073a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:31.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.534+0000 7f52c354a700 1 -- 172.21.15.189:0/1327480344 shutdown_connections 2024-04-06T12:27:31.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.534+0000 7f52c354a700 1 -- 172.21.15.189:0/1327480344 wait complete. 2024-04-06T12:27:31.544 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T12:27:31.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.668+0000 7f5b2d88b700 1 Processor -- start 2024-04-06T12:27:31.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.668+0000 7f5b2d88b700 1 -- start start 2024-04-06T12:27:31.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b2d88b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b2815d590 0x7f5b280b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b2d88b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b280b4ed0 0x7f5b280b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b2d88b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5b280b6580 0x7f5b280b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b2d88b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b280b5890 con 0x7f5b280b4ed0 2024-04-06T12:27:31.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b2d88b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b280b5cc0 con 0x7f5b280b6580 2024-04-06T12:27:31.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b2d88b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b280d1500 con 0x7f5b2815d590 2024-04-06T12:27:31.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b267fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b280b4ed0 0x7f5b280b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b267fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b280b4ed0 0x7f5b280b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33122/0 (socket says 172.21.15.189:33122) 2024-04-06T12:27:31.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b267fc700 1 -- 172.21.15.189:0/88892328 learned_addr learned my addr 172.21.15.189:0/88892328 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:31.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b277fe700 1 --2- 172.21.15.189:0/88892328 >> v2:172.21.15.192:3300/0 conn(0x7f5b280b6580 0x7f5b280b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b26ffd700 1 --2- 172.21.15.189:0/88892328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b2815d590 0x7f5b280b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b267fc700 1 -- 172.21.15.189:0/88892328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b2815d590 msgr2=0x7f5b280b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:31.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b267fc700 1 --2- 172.21.15.189:0/88892328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b2815d590 0x7f5b280b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b267fc700 1 -- 172.21.15.189:0/88892328 >> v2:172.21.15.192:3300/0 conn(0x7f5b280b6580 msgr2=0x7f5b280b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:31.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b267fc700 1 --2- 172.21.15.189:0/88892328 >> v2:172.21.15.192:3300/0 conn(0x7f5b280b6580 0x7f5b280b6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b267fc700 1 -- 172.21.15.189:0/88892328 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b280d17e0 con 0x7f5b280b4ed0 2024-04-06T12:27:31.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b277fe700 1 --2- 172.21.15.189:0/88892328 >> v2:172.21.15.192:3300/0 conn(0x7f5b280b6580 0x7f5b280b6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:31.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b267fc700 1 --2- 172.21.15.189:0/88892328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b280b4ed0 0x7f5b280b5350 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0098f0 tx=0x7f5b1c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deb2ecafeed9efcc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:31.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b25ffb700 1 -- 172.21.15.189:0/88892328 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c00b860 con 0x7f5b280b4ed0 2024-04-06T12:27:31.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b25ffb700 1 -- 172.21.15.189:0/88892328 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1c00b9c0 con 0x7f5b280b4ed0 2024-04-06T12:27:31.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.669+0000 7f5b25ffb700 1 -- 172.21.15.189:0/88892328 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c00bd50 con 0x7f5b280b4ed0 2024-04-06T12:27:31.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.670+0000 7f5b2d88b700 1 -- 172.21.15.189:0/88892328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b280b4ed0 msgr2=0x7f5b280b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:31.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.670+0000 7f5b2d88b700 1 --2- 172.21.15.189:0/88892328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b280b4ed0 0x7f5b280b5350 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0098f0 tx=0x7f5b1c009c00 comp rx=0 tx=0).stop 2024-04-06T12:27:31.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.670+0000 7f5b2d88b700 1 -- 172.21.15.189:0/88892328 shutdown_connections 2024-04-06T12:27:31.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.670+0000 7f5b2d88b700 1 --2- 172.21.15.189:0/88892328 >> v2:172.21.15.192:3300/0 conn(0x7f5b280b6580 0x7f5b280b6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.670+0000 7f5b2d88b700 1 --2- 172.21.15.189:0/88892328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b2815d590 0x7f5b280b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.670+0000 7f5b2d88b700 1 --2- 172.21.15.189:0/88892328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b280b4ed0 0x7f5b280b5350 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.670+0000 7f5b2d88b700 1 -- 172.21.15.189:0/88892328 >> 172.21.15.189:0/88892328 conn(0x7f5b2806bbc0 msgr2=0x7f5b2806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:31.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.670+0000 7f5b2d88b700 1 -- 172.21.15.189:0/88892328 shutdown_connections 2024-04-06T12:27:31.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.670+0000 7f5b2d88b700 1 -- 172.21.15.189:0/88892328 wait complete. 2024-04-06T12:27:31.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.671+0000 7f5b2d88b700 1 Processor -- start 2024-04-06T12:27:31.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.671+0000 7f5b2d88b700 1 -- start start 2024-04-06T12:27:31.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.672+0000 7f5b2d88b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b280b4ed0 0x7f5b281f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.672+0000 7f5b2d88b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b280b6580 0x7f5b280b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.672+0000 7f5b2d88b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b2815d590 0x7f5b280b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.672+0000 7f5b2d88b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2814b720 con 0x7f5b280b4ed0 2024-04-06T12:27:31.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.672+0000 7f5b2d88b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2814ba00 con 0x7f5b2815d590 2024-04-06T12:27:31.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.672+0000 7f5b2d88b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2814bce0 con 0x7f5b280b6580 2024-04-06T12:27:31.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.672+0000 7f5b26ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b280b4ed0 0x7f5b281f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.672+0000 7f5b26ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b280b4ed0 0x7f5b281f2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33128/0 (socket says 172.21.15.189:33128) 2024-04-06T12:27:31.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.672+0000 7f5b26ffd700 1 -- 172.21.15.189:0/823562680 learned_addr learned my addr 172.21.15.189:0/823562680 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:31.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.672+0000 7f5b267fc700 1 --2- 172.21.15.189:0/823562680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b280b6580 0x7f5b280b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.672+0000 7f5b277fe700 1 --2- 172.21.15.189:0/823562680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b2815d590 0x7f5b280b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.672+0000 7f5b26ffd700 1 -- 172.21.15.189:0/823562680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b280b6580 msgr2=0x7f5b280b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:31.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.672+0000 7f5b26ffd700 1 --2- 172.21.15.189:0/823562680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b280b6580 0x7f5b280b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.672+0000 7f5b26ffd700 1 -- 172.21.15.189:0/823562680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b2815d590 msgr2=0x7f5b280b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:31.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.672+0000 7f5b26ffd700 1 --2- 172.21.15.189:0/823562680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b2815d590 0x7f5b280b48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.672+0000 7f5b26ffd700 1 -- 172.21.15.189:0/823562680 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c009550 con 0x7f5b280b4ed0 2024-04-06T12:27:31.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.672+0000 7f5b267fc700 1 --2- 172.21.15.189:0/823562680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b280b6580 0x7f5b280b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:27:31.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.672+0000 7f5b277fe700 1 --2- 172.21.15.189:0/823562680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b2815d590 0x7f5b280b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:27:31.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.673+0000 7f5b26ffd700 1 --2- 172.21.15.189:0/823562680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b280b4ed0 0x7f5b281f2480 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800d800 tx=0x7f5b1800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:31.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.673+0000 7f5b0ffff700 1 -- 172.21.15.189:0/823562680 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b180125f0 con 0x7f5b280b4ed0 2024-04-06T12:27:31.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.673+0000 7f5b0ffff700 1 -- 172.21.15.189:0/823562680 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b18012750 con 0x7f5b280b4ed0 2024-04-06T12:27:31.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.673+0000 7f5b2d88b700 1 -- 172.21.15.189:0/823562680 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b2814bfc0 con 0x7f5b280b4ed0 2024-04-06T12:27:31.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.673+0000 7f5b0ffff700 1 -- 172.21.15.189:0/823562680 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18012b00 con 0x7f5b280b4ed0 2024-04-06T12:27:31.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.673+0000 7f5b2d88b700 1 -- 172.21.15.189:0/823562680 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b281f29c0 con 0x7f5b280b4ed0 2024-04-06T12:27:31.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.674+0000 7f5b2d88b700 1 -- 172.21.15.189:0/823562680 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b28074990 con 0x7f5b280b4ed0 2024-04-06T12:27:31.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.674+0000 7f5b0ffff700 1 -- 172.21.15.189:0/823562680 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f5b18012c60 con 0x7f5b280b4ed0 2024-04-06T12:27:31.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.675+0000 7f5b0ffff700 1 --2- 172.21.15.189:0/823562680 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b1406adb0 0x7f5b1406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:31.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.675+0000 7f5b267fc700 1 --2- 172.21.15.189:0/823562680 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b1406adb0 0x7f5b1406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:31.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.675+0000 7f5b0ffff700 1 -- 172.21.15.189:0/823562680 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5b18089980 con 0x7f5b280b4ed0 2024-04-06T12:27:31.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.675+0000 7f5b267fc700 1 --2- 172.21.15.189:0/823562680 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b1406adb0 0x7f5b1406d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5b2814fac0 tx=0x7f5b1c008030 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:31.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.679+0000 7f5b0ffff700 1 -- 172.21.15.189:0/823562680 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5b18054910 con 0x7f5b280b4ed0 2024-04-06T12:27:31.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.919+0000 7f5b2d88b700 1 -- 172.21.15.189:0/823562680 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5b280d3560 con 0x7f5b280b4ed0 2024-04-06T12:27:31.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.919+0000 7f5b0ffff700 1 -- 172.21.15.189:0/823562680 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v31) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5b18057f30 con 0x7f5b280b4ed0 2024-04-06T12:27:31.918 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-06T12:27:31.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.921+0000 7f5b2d88b700 1 -- 172.21.15.189:0/823562680 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b1406adb0 msgr2=0x7f5b1406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:31.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.921+0000 7f5b2d88b700 1 --2- 172.21.15.189:0/823562680 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b1406adb0 0x7f5b1406d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5b2814fac0 tx=0x7f5b1c008030 comp rx=0 tx=0).stop 2024-04-06T12:27:31.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.921+0000 7f5b2d88b700 1 -- 172.21.15.189:0/823562680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b280b4ed0 msgr2=0x7f5b281f2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:31.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.921+0000 7f5b2d88b700 1 --2- 172.21.15.189:0/823562680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b280b4ed0 0x7f5b281f2480 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800d800 tx=0x7f5b1800dbc0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.922+0000 7f5b2d88b700 1 -- 172.21.15.189:0/823562680 shutdown_connections 2024-04-06T12:27:31.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.922+0000 7f5b2d88b700 1 --2- 172.21.15.189:0/823562680 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b1406adb0 0x7f5b1406d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.922+0000 7f5b2d88b700 1 --2- 172.21.15.189:0/823562680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b2815d590 0x7f5b280b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.922+0000 7f5b2d88b700 1 --2- 172.21.15.189:0/823562680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b280b4ed0 0x7f5b281f2480 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.922+0000 7f5b2d88b700 1 --2- 172.21.15.189:0/823562680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b280b6580 0x7f5b280b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:31.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.922+0000 7f5b2d88b700 1 -- 172.21.15.189:0/823562680 >> 172.21.15.189:0/823562680 conn(0x7f5b2806bbc0 msgr2=0x7f5b280cd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:31.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.922+0000 7f5b2d88b700 1 -- 172.21.15.189:0/823562680 shutdown_connections 2024-04-06T12:27:31.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:31.922+0000 7f5b2d88b700 1 -- 172.21.15.189:0/823562680 wait complete. 2024-04-06T12:27:31.932 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-06T12:27:31.932 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T12:27:31.932 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-06T12:27:31.932 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-06T12:27:31.932 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T12:27:31.932 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-06T12:27:31.932 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-06T12:27:31.932 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T12:27:31.932 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:27:31.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:31.932 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-06T12:27:31.950 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T12:27:31.950 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T12:27:31.950 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T12:27:31.950 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:27:31.950 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T12:27:31.951 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:27:31.951 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:27:31.951 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:27:31.951 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-06T12:27:31.951 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:27:31.951 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:27:31.951 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-06T12:27:31.951 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:27:31.951 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:27:31.951 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T12:27:31.951 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:27:31.951 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T12:27:31.951 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 41437sec preferred_lft 41437sec 2024-04-06T12:27:31.951 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::8592:df2c:92e2:dace/64 scope link noprefixroute 2024-04-06T12:27:31.951 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:27:31.952 INFO:teuthology.orchestra.run.smithi189.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T12:27:31.952 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff 2024-04-06T12:27:31.952 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-06T12:27:31.952 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:27:31.952 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::88e:b6ff:fe6f:1cb0/64 scope link 2024-04-06T12:27:31.952 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:27:31.952 INFO:teuthology.orchestra.run.smithi189.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T12:27:31.952 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 62:a5:8d:c5:1a:9c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T12:27:31.952 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::60a5:8dff:fec5:1a9c/64 scope link 2024-04-06T12:27:31.952 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:27:31.952 INFO:teuthology.orchestra.run.smithi189.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T12:27:31.952 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-06T12:27:31.952 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::1c7e:c6ff:febb:1dab/64 scope link 2024-04-06T12:27:31.952 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:27:31.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:31.953 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T12:27:32.016 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-06T12:27:32.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:32.018 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-06T12:27:32.077 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-06T12:27:32.077 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T12:27:32.077 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T12:27:32.077 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T12:27:32.077 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T12:27:32.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:32.078 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:27:32.078 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:27:32.078 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T12:27:32.078 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T12:27:32.078 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T12:27:32.078 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:27:32.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:32.182 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T12:27:32.199 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-06T12:27:32.199 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-06T12:27:32.200 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:27:32.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:32.200 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-04-06T12:27:32.267 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:27:32.267 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-06T12:27:32.325 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:27:32.325 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:27:32.325 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-06T12:27:32.381 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T12:27:32.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:32.382 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:27:32.468 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:27:32.468 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:27:32.468 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:27:32.477 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:27:32.477 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:27:32.477 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:27:32.477 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:27:32.477 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:27:32.496 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:27:32.496 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:27:32.496 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:27:32.496 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:27:32.496 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:27:32.496 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790" 2024-04-06T12:27:32.496 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2024-04-06T12:27:32.496 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e30041de-e9a3-4351-8ca7-29d7f3e3728c" will pass to kernel 2024-04-06T12:27:32.498 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T12:27:32.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:32.499 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-06T12:27:32.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:32.527 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:27:32.609 INFO:teuthology.orchestra.run.smithi189.stdout:client6349 2024-04-06T12:27:32.609 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:27:32.609 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:27:32.667 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-06T12:27:32.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.832+0000 7f1525c8b700 1 Processor -- start 2024-04-06T12:27:32.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.832+0000 7f1525c8b700 1 -- start start 2024-04-06T12:27:32.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.832+0000 7f1525c8b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f15200d17a0 0x7f152006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:32.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.832+0000 7f1525c8b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1520146dd0 0x7f152006d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:32.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.832+0000 7f1525c8b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152006c0a0 con 0x7f15200cedf0 2024-04-06T12:27:32.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.832+0000 7f1525c8b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f152006c380 con 0x7f15200d17a0 2024-04-06T12:27:32.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.832+0000 7f1525c8b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f152006c660 con 0x7f1520146dd0 2024-04-06T12:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.833+0000 7f151f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f15200d17a0 0x7f152006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.833+0000 7f151ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f15200cedf0 0x7f15200cf1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52624/0 (socket says 172.21.15.189:52624) 2024-04-06T12:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.833+0000 7f151f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f15200d17a0 0x7f152006b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33142/0 (socket says 172.21.15.189:33142) 2024-04-06T12:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.833+0000 7f151ffff700 1 -- 172.21.15.189:0/2466938025 learned_addr learned my addr 172.21.15.189:0/2466938025 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.833+0000 7f151effd700 1 --2- 172.21.15.189:0/2466938025 >> v2:172.21.15.192:3300/0 conn(0x7f1520146dd0 0x7f152006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.833+0000 7f151f7fe700 1 -- 172.21.15.189:0/2466938025 >> v2:172.21.15.192:3300/0 conn(0x7f1520146dd0 msgr2=0x7f152006d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.833+0000 7f151f7fe700 1 --2- 172.21.15.189:0/2466938025 >> v2:172.21.15.192:3300/0 conn(0x7f1520146dd0 0x7f152006d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.833+0000 7f151f7fe700 1 -- 172.21.15.189:0/2466938025 >> v1:172.21.15.189:6789/0 conn(0x7f15200cedf0 legacy=0x7f15200cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.833+0000 7f151f7fe700 1 -- 172.21.15.189:0/2466938025 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152006c940 con 0x7f15200d17a0 2024-04-06T12:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.833+0000 7f151effd700 1 --2- 172.21.15.189:0/2466938025 >> v2:172.21.15.192:3300/0 conn(0x7f1520146dd0 0x7f152006d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.833+0000 7f151f7fe700 1 --2- 172.21.15.189:0/2466938025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15200d17a0 0x7f152006b7a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f15200d2300 tx=0x7f1510009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a39f7e6669cb7082 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.833+0000 7f151e7fc700 1 -- 172.21.15.189:0/2466938025 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1510011830 con 0x7f15200d17a0 2024-04-06T12:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.833+0000 7f151e7fc700 1 -- 172.21.15.189:0/2466938025 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1510011990 con 0x7f15200d17a0 2024-04-06T12:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.833+0000 7f151e7fc700 1 -- 172.21.15.189:0/2466938025 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1510011d20 con 0x7f15200d17a0 2024-04-06T12:27:32.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.834+0000 7f1525c8b700 1 -- 172.21.15.189:0/2466938025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15200d17a0 msgr2=0x7f152006b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:32.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.834+0000 7f1525c8b700 1 --2- 172.21.15.189:0/2466938025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15200d17a0 0x7f152006b7a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f15200d2300 tx=0x7f1510009b20 comp rx=0 tx=0).stop 2024-04-06T12:27:32.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.834+0000 7f1525c8b700 1 -- 172.21.15.189:0/2466938025 shutdown_connections 2024-04-06T12:27:32.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.834+0000 7f1525c8b700 1 --2- 172.21.15.189:0/2466938025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15200d17a0 0x7f152006b7a0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:32.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.834+0000 7f1525c8b700 1 --2- 172.21.15.189:0/2466938025 >> v2:172.21.15.192:3300/0 conn(0x7f1520146dd0 0x7f152006d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:32.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.834+0000 7f1525c8b700 1 -- 172.21.15.189:0/2466938025 >> 172.21.15.189:0/2466938025 conn(0x7f15200b2db0 msgr2=0x7f15200b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:32.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.834+0000 7f1525c8b700 1 -- 172.21.15.189:0/2466938025 shutdown_connections 2024-04-06T12:27:32.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.834+0000 7f1525c8b700 1 -- 172.21.15.189:0/2466938025 wait complete. 2024-04-06T12:27:32.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.835+0000 7f1525c8b700 1 Processor -- start 2024-04-06T12:27:32.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.835+0000 7f1525c8b700 1 -- start start 2024-04-06T12:27:32.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.836+0000 7f1525c8b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15200cedf0 0x7f15201f6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:32.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.836+0000 7f1525c8b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15200d17a0 0x7f15201f6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:32.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.836+0000 7f1525c8b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1520146dd0 0x7f15200b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:32.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.836+0000 7f1525c8b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15200b4c10 con 0x7f1520146dd0 2024-04-06T12:27:32.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.836+0000 7f1525c8b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15200af7b0 con 0x7f15200cedf0 2024-04-06T12:27:32.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.836+0000 7f1525c8b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15200b42e0 con 0x7f15200d17a0 2024-04-06T12:27:32.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.836+0000 7f151ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1520146dd0 0x7f15200b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:32.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.836+0000 7f151ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1520146dd0 0x7f15200b50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33152/0 (socket says 172.21.15.189:33152) 2024-04-06T12:27:32.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.836+0000 7f151ffff700 1 -- 172.21.15.189:0/1512028832 learned_addr learned my addr 172.21.15.189:0/1512028832 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:32.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.836+0000 7f151effd700 1 --2- 172.21.15.189:0/1512028832 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15200d17a0 0x7f15201f6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:32.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.836+0000 7f151f7fe700 1 --2- 172.21.15.189:0/1512028832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15200cedf0 0x7f15201f6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:32.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.836+0000 7f151ffff700 1 -- 172.21.15.189:0/1512028832 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15200d17a0 msgr2=0x7f15201f6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:32.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.836+0000 7f151ffff700 1 --2- 172.21.15.189:0/1512028832 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15200d17a0 0x7f15201f6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:32.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.836+0000 7f151ffff700 1 -- 172.21.15.189:0/1512028832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15200cedf0 msgr2=0x7f15201f6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:32.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.836+0000 7f151ffff700 1 --2- 172.21.15.189:0/1512028832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15200cedf0 0x7f15201f6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:32.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.836+0000 7f151ffff700 1 -- 172.21.15.189:0/1512028832 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15100094a0 con 0x7f1520146dd0 2024-04-06T12:27:32.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.836+0000 7f151ffff700 1 --2- 172.21.15.189:0/1512028832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1520146dd0 0x7f15200b50a0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f151800ca80 tx=0x7f151800ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:32.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.837+0000 7f1507fff700 1 -- 172.21.15.189:0/1512028832 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151800aaa0 con 0x7f1520146dd0 2024-04-06T12:27:32.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.837+0000 7f1507fff700 1 -- 172.21.15.189:0/1512028832 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151800ac00 con 0x7f1520146dd0 2024-04-06T12:27:32.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.837+0000 7f1525c8b700 1 -- 172.21.15.189:0/1512028832 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15200b45c0 con 0x7f1520146dd0 2024-04-06T12:27:32.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.837+0000 7f1507fff700 1 -- 172.21.15.189:0/1512028832 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1518012400 con 0x7f1520146dd0 2024-04-06T12:27:32.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.837+0000 7f1525c8b700 1 -- 172.21.15.189:0/1512028832 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f152015cb60 con 0x7f1520146dd0 2024-04-06T12:27:32.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.838+0000 7f1525c8b700 1 -- 172.21.15.189:0/1512028832 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1520074990 con 0x7f1520146dd0 2024-04-06T12:27:32.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.838+0000 7f1507fff700 1 -- 172.21.15.189:0/1512028832 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f151801b3b0 con 0x7f1520146dd0 2024-04-06T12:27:32.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.839+0000 7f1507fff700 1 --2- 172.21.15.189:0/1512028832 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f150c06adb0 0x7f150c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:32.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.839+0000 7f151f7fe700 1 --2- 172.21.15.189:0/1512028832 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f150c06adb0 0x7f150c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:32.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.839+0000 7f1507fff700 1 -- 172.21.15.189:0/1512028832 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1518089140 con 0x7f1520146dd0 2024-04-06T12:27:32.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.839+0000 7f151f7fe700 1 --2- 172.21.15.189:0/1512028832 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f150c06adb0 0x7f150c06d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f15100090b0 tx=0x7f1510008030 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:32.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:32.843+0000 7f1507fff700 1 -- 172.21.15.189:0/1512028832 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1518054180 con 0x7f1520146dd0 2024-04-06T12:27:33.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.069+0000 7f1525c8b700 1 -- 172.21.15.189:0/1512028832 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f15200b4450 con 0x7f150c06adb0 2024-04-06T12:27:33.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.070+0000 7f1507fff700 1 -- 172.21.15.189:0/1512028832 <== mgr.5582 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f15200b4450 con 0x7f150c06adb0 2024-04-06T12:27:33.068 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T12:27:33.068 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:27:33.069 INFO:teuthology.orchestra.run.smithi189.stdout: "name": "cephfs" 2024-04-06T12:27:33.069 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:27:33.069 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T12:27:33.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.072+0000 7f1525c8b700 1 -- 172.21.15.189:0/1512028832 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f150c06adb0 msgr2=0x7f150c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:33.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.072+0000 7f1525c8b700 1 --2- 172.21.15.189:0/1512028832 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f150c06adb0 0x7f150c06d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f15100090b0 tx=0x7f1510008030 comp rx=0 tx=0).stop 2024-04-06T12:27:33.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.072+0000 7f1525c8b700 1 -- 172.21.15.189:0/1512028832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1520146dd0 msgr2=0x7f15200b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:33.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.072+0000 7f1525c8b700 1 --2- 172.21.15.189:0/1512028832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1520146dd0 0x7f15200b50a0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f151800ca80 tx=0x7f151800ce40 comp rx=0 tx=0).stop 2024-04-06T12:27:33.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.072+0000 7f1525c8b700 1 -- 172.21.15.189:0/1512028832 shutdown_connections 2024-04-06T12:27:33.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.072+0000 7f1525c8b700 1 --2- 172.21.15.189:0/1512028832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1520146dd0 0x7f15200b50a0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.072+0000 7f1525c8b700 1 --2- 172.21.15.189:0/1512028832 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f150c06adb0 0x7f150c06d270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.072+0000 7f1525c8b700 1 --2- 172.21.15.189:0/1512028832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15200cedf0 0x7f15201f6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.072+0000 7f1525c8b700 1 --2- 172.21.15.189:0/1512028832 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15200d17a0 0x7f15201f6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.072+0000 7f1525c8b700 1 -- 172.21.15.189:0/1512028832 >> 172.21.15.189:0/1512028832 conn(0x7f15200b2db0 msgr2=0x7f152015e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:33.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.072+0000 7f1525c8b700 1 -- 172.21.15.189:0/1512028832 shutdown_connections 2024-04-06T12:27:33.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.072+0000 7f1525c8b700 1 -- 172.21.15.189:0/1512028832 wait complete. 2024-04-06T12:27:33.082 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-06T12:27:33.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.208+0000 7f039a3e3700 1 Processor -- start 2024-04-06T12:27:33.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.208+0000 7f039a3e3700 1 -- start start 2024-04-06T12:27:33.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.209+0000 7f039a3e3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f039415a550 0x7f03940cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:33.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.209+0000 7f039a3e3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03940cf900 0x7f03940b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:33.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.209+0000 7f039a3e3700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03940cdbe0 con 0x7f03940b4ba0 2024-04-06T12:27:33.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.209+0000 7f039a3e3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03940b5010 con 0x7f039415a550 2024-04-06T12:27:33.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.209+0000 7f039a3e3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f039415a940 con 0x7f03940cf900 2024-04-06T12:27:33.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.209+0000 7f0393fff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f039415a550 0x7f03940cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:33.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.209+0000 7f03937fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03940cf900 0x7f03940b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:33.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.209+0000 7f0398980700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f03940b4ba0 0x7f039406eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:44858/0 (socket says 172.21.15.189:44858) 2024-04-06T12:27:33.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.209+0000 7f0398980700 1 -- 172.21.15.189:0/1944720160 learned_addr learned my addr 172.21.15.189:0/1944720160 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:33.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.210+0000 7f0393fff700 1 -- 172.21.15.189:0/1944720160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03940cf900 msgr2=0x7f03940b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:33.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.210+0000 7f0393fff700 1 --2- 172.21.15.189:0/1944720160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03940cf900 0x7f03940b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.210+0000 7f0393fff700 1 -- 172.21.15.189:0/1944720160 >> v1:172.21.15.192:6789/0 conn(0x7f03940b4ba0 legacy=0x7f039406eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:33.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.210+0000 7f0392ffd700 1 -- 172.21.15.189:0/1944720160 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2090160778 0 0) 0x7f03940cdbe0 con 0x7f03940b4ba0 2024-04-06T12:27:33.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.210+0000 7f0393fff700 1 -- 172.21.15.189:0/1944720160 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039415ac00 con 0x7f039415a550 2024-04-06T12:27:33.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.210+0000 7f03937fe700 1 --2- 172.21.15.189:0/1944720160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03940cf900 0x7f03940b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:33.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.210+0000 7f0393fff700 1 --2- 172.21.15.189:0/1944720160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f039415a550 0x7f03940cf3c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0388009840 tx=0x7f0388009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dc80e7f1b40e690 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:33.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.210+0000 7f0392ffd700 1 -- 172.21.15.189:0/1944720160 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0388011830 con 0x7f039415a550 2024-04-06T12:27:33.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.210+0000 7f0392ffd700 1 -- 172.21.15.189:0/1944720160 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0388011990 con 0x7f039415a550 2024-04-06T12:27:33.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.210+0000 7f0392ffd700 1 -- 172.21.15.189:0/1944720160 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0388011d20 con 0x7f039415a550 2024-04-06T12:27:33.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.210+0000 7f039a3e3700 1 -- 172.21.15.189:0/1944720160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f039415a550 msgr2=0x7f03940cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:33.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.210+0000 7f039a3e3700 1 --2- 172.21.15.189:0/1944720160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f039415a550 0x7f03940cf3c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0388009840 tx=0x7f0388009b50 comp rx=0 tx=0).stop 2024-04-06T12:27:33.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.211+0000 7f039a3e3700 1 -- 172.21.15.189:0/1944720160 shutdown_connections 2024-04-06T12:27:33.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.211+0000 7f039a3e3700 1 --2- 172.21.15.189:0/1944720160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f039415a550 0x7f03940cf3c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.211+0000 7f039a3e3700 1 --2- 172.21.15.189:0/1944720160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03940cf900 0x7f03940b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.211+0000 7f039a3e3700 1 -- 172.21.15.189:0/1944720160 >> 172.21.15.189:0/1944720160 conn(0x7f03940d00b0 msgr2=0x7f03940d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:33.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.211+0000 7f039a3e3700 1 -- 172.21.15.189:0/1944720160 shutdown_connections 2024-04-06T12:27:33.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.211+0000 7f039a3e3700 1 -- 172.21.15.189:0/1944720160 wait complete. 2024-04-06T12:27:33.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.212+0000 7f039a3e3700 1 Processor -- start 2024-04-06T12:27:33.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.212+0000 7f039a3e3700 1 -- start start 2024-04-06T12:27:33.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.213+0000 7f039a3e3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03940b4ba0 0x7f039414cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:33.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.213+0000 7f039a3e3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03940cf900 0x7f0394151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:33.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.213+0000 7f039a3e3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03941582c0 0x7f0394158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:33.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.213+0000 7f039a3e3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039414df40 con 0x7f03940cf900 2024-04-06T12:27:33.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.213+0000 7f039a3e3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0394159340 con 0x7f03941582c0 2024-04-06T12:27:33.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.213+0000 7f039a3e3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0394158d80 con 0x7f03940b4ba0 2024-04-06T12:27:33.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.213+0000 7f03937fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03940cf900 0x7f0394151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:33.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.213+0000 7f03937fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03940cf900 0x7f0394151720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33156/0 (socket says 172.21.15.189:33156) 2024-04-06T12:27:33.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.213+0000 7f03937fe700 1 -- 172.21.15.189:0/1212825596 learned_addr learned my addr 172.21.15.189:0/1212825596 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:33.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.214+0000 7f0393fff700 1 --2- 172.21.15.189:0/1212825596 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03940b4ba0 0x7f039414cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:33.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.214+0000 7f0398980700 1 --2- 172.21.15.189:0/1212825596 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03941582c0 0x7f0394158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:33.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.214+0000 7f03937fe700 1 -- 172.21.15.189:0/1212825596 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03940b4ba0 msgr2=0x7f039414cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:33.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.214+0000 7f03937fe700 1 --2- 172.21.15.189:0/1212825596 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03940b4ba0 0x7f039414cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.214+0000 7f03937fe700 1 -- 172.21.15.189:0/1212825596 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03941582c0 msgr2=0x7f0394158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:33.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.214+0000 7f03937fe700 1 --2- 172.21.15.189:0/1212825596 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03941582c0 0x7f0394158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.214+0000 7f03937fe700 1 -- 172.21.15.189:0/1212825596 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03840094a0 con 0x7f03940cf900 2024-04-06T12:27:33.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.214+0000 7f03937fe700 1 --2- 172.21.15.189:0/1212825596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03940cf900 0x7f0394151720 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f038400ebe0 tx=0x7f0384009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:33.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.214+0000 7f0390ff9700 1 -- 172.21.15.189:0/1212825596 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038400c910 con 0x7f03940cf900 2024-04-06T12:27:33.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.214+0000 7f0390ff9700 1 -- 172.21.15.189:0/1212825596 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f038400ca70 con 0x7f03940cf900 2024-04-06T12:27:33.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.214+0000 7f039a3e3700 1 -- 172.21.15.189:0/1212825596 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03880094a0 con 0x7f03940cf900 2024-04-06T12:27:33.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.214+0000 7f0390ff9700 1 -- 172.21.15.189:0/1212825596 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038400cde0 con 0x7f03940cf900 2024-04-06T12:27:33.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.214+0000 7f039a3e3700 1 -- 172.21.15.189:0/1212825596 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f039414dbd0 con 0x7f03940cf900 2024-04-06T12:27:33.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.215+0000 7f039a3e3700 1 -- 172.21.15.189:0/1212825596 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0394067bb0 con 0x7f03940cf900 2024-04-06T12:27:33.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.215+0000 7f0390ff9700 1 -- 172.21.15.189:0/1212825596 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 32) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0384013470 con 0x7f03940cf900 2024-04-06T12:27:33.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.216+0000 7f0390ff9700 1 --2- 172.21.15.189:0/1212825596 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f038006adb0 0x7f038006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:33.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.216+0000 7f0390ff9700 1 -- 172.21.15.189:0/1212825596 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0384089780 con 0x7f03940cf900 2024-04-06T12:27:33.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.216+0000 7f0393fff700 1 --2- 172.21.15.189:0/1212825596 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f038006adb0 0x7f038006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:33.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.216+0000 7f0393fff700 1 --2- 172.21.15.189:0/1212825596 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f038006adb0 0x7f038006d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f03880090b0 tx=0x7f0388008030 comp rx=0 tx=0).ready entity=mgr.5582 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:33.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.220+0000 7f0390ff9700 1 -- 172.21.15.189:0/1212825596 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f03840547a0 con 0x7f03940cf900 2024-04-06T12:27:33.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.472+0000 7f039a3e3700 1 -- 172.21.15.189:0/1212825596 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f0394074990 con 0x7f03940cf900 2024-04-06T12:27:33.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.582+0000 7f0390ff9700 1 -- 172.21.15.189:0/1212825596 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 33) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f0384054b70 con 0x7f03940cf900 2024-04-06T12:27:33.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.583+0000 7f0390ff9700 1 -- 172.21.15.189:0/1212825596 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v33) v1 ==== 92+0+0 (secure 0 0 0) 0x7f0384087240 con 0x7f03940cf900 2024-04-06T12:27:33.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.585+0000 7f039a3e3700 1 -- 172.21.15.189:0/1212825596 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f038006adb0 msgr2=0x7f038006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:33.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.585+0000 7f039a3e3700 1 --2- 172.21.15.189:0/1212825596 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f038006adb0 0x7f038006d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f03880090b0 tx=0x7f0388008030 comp rx=0 tx=0).stop 2024-04-06T12:27:33.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.585+0000 7f039a3e3700 1 -- 172.21.15.189:0/1212825596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03940cf900 msgr2=0x7f0394151720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:33.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.585+0000 7f039a3e3700 1 --2- 172.21.15.189:0/1212825596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03940cf900 0x7f0394151720 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f038400ebe0 tx=0x7f0384009ec0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.586+0000 7f039a3e3700 1 -- 172.21.15.189:0/1212825596 shutdown_connections 2024-04-06T12:27:33.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.586+0000 7f039a3e3700 1 --2- 172.21.15.189:0/1212825596 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f038006adb0 0x7f038006d270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.586+0000 7f039a3e3700 1 --2- 172.21.15.189:0/1212825596 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03941582c0 0x7f0394158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.586+0000 7f039a3e3700 1 --2- 172.21.15.189:0/1212825596 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03940b4ba0 0x7f039414cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.586+0000 7f039a3e3700 1 --2- 172.21.15.189:0/1212825596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03940cf900 0x7f0394151720 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.586+0000 7f039a3e3700 1 -- 172.21.15.189:0/1212825596 >> 172.21.15.189:0/1212825596 conn(0x7f03940d00b0 msgr2=0x7f03940717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:33.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.586+0000 7f039a3e3700 1 -- 172.21.15.189:0/1212825596 shutdown_connections 2024-04-06T12:27:33.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.586+0000 7f039a3e3700 1 -- 172.21.15.189:0/1212825596 wait complete. 2024-04-06T12:27:33.597 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-06T12:27:33.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.728+0000 7f8ed3c73700 1 Processor -- start 2024-04-06T12:27:33.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.728+0000 7f8ed3c73700 1 -- start start 2024-04-06T12:27:33.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.729+0000 7f8ed3c73700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8ecc06ae40 0x7f8ecc0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:33.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.729+0000 7f8ed3c73700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8ecc15ddf0 0x7f8ecc15e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:33.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.729+0000 7f8ed3c73700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ecc0b3990 con 0x7f8ecc15d7b0 2024-04-06T12:27:33.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.729+0000 7f8ed3c73700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ecc0b3060 con 0x7f8ecc06ae40 2024-04-06T12:27:33.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.729+0000 7f8ed3c73700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ecc0b3340 con 0x7f8ecc15ddf0 2024-04-06T12:27:33.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.729+0000 7f8ed120e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8ecc06ae40 0x7f8ecc0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:33.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.729+0000 7f8ed120e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8ecc06ae40 0x7f8ecc0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33190/0 (socket says 172.21.15.189:33190) 2024-04-06T12:27:33.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.729+0000 7f8ed120e700 1 -- 172.21.15.189:0/950761152 learned_addr learned my addr 172.21.15.189:0/950761152 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:33.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.729+0000 7f8ed2210700 1 --2- 172.21.15.189:0/950761152 >> v2:172.21.15.192:3300/0 conn(0x7f8ecc15ddf0 0x7f8ecc15e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:33.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.729+0000 7f8ed120e700 1 -- 172.21.15.189:0/950761152 >> v2:172.21.15.192:3300/0 conn(0x7f8ecc15ddf0 msgr2=0x7f8ecc15e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:33.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.729+0000 7f8ed120e700 1 --2- 172.21.15.189:0/950761152 >> v2:172.21.15.192:3300/0 conn(0x7f8ecc15ddf0 0x7f8ecc15e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.729+0000 7f8ed120e700 1 -- 172.21.15.189:0/950761152 >> v1:172.21.15.189:6789/0 conn(0x7f8ecc15d7b0 legacy=0x7f8ecc06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:33.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.729+0000 7f8ed120e700 1 -- 172.21.15.189:0/950761152 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc0b3620 con 0x7f8ecc06ae40 2024-04-06T12:27:33.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.729+0000 7f8ed120e700 1 --2- 172.21.15.189:0/950761152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ecc06ae40 0x7f8ecc0b3e20 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc06d2c0 tx=0x7f8ebc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ccabf84b2b4b46 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:33.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.729+0000 7f8ed0a0d700 1 -- 172.21.15.189:0/950761152 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc011830 con 0x7f8ecc06ae40 2024-04-06T12:27:33.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.729+0000 7f8ed0a0d700 1 -- 172.21.15.189:0/950761152 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc011990 con 0x7f8ecc06ae40 2024-04-06T12:27:33.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.730+0000 7f8ed0a0d700 1 -- 172.21.15.189:0/950761152 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc011d20 con 0x7f8ecc06ae40 2024-04-06T12:27:33.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.730+0000 7f8ed3c73700 1 -- 172.21.15.189:0/950761152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ecc06ae40 msgr2=0x7f8ecc0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:33.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.730+0000 7f8ed3c73700 1 --2- 172.21.15.189:0/950761152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ecc06ae40 0x7f8ecc0b3e20 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc06d2c0 tx=0x7f8ebc009bd0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.730+0000 7f8ed3c73700 1 -- 172.21.15.189:0/950761152 shutdown_connections 2024-04-06T12:27:33.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.730+0000 7f8ed3c73700 1 --2- 172.21.15.189:0/950761152 >> v2:172.21.15.192:3300/0 conn(0x7f8ecc15ddf0 0x7f8ecc15e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.730+0000 7f8ed3c73700 1 --2- 172.21.15.189:0/950761152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ecc06ae40 0x7f8ecc0b3e20 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.730+0000 7f8ed3c73700 1 -- 172.21.15.189:0/950761152 >> 172.21.15.189:0/950761152 conn(0x7f8ecc0d00b0 msgr2=0x7f8ecc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:33.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.730+0000 7f8ed3c73700 1 -- 172.21.15.189:0/950761152 shutdown_connections 2024-04-06T12:27:33.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.730+0000 7f8ed3c73700 1 -- 172.21.15.189:0/950761152 wait complete. 2024-04-06T12:27:33.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.731+0000 7f8ed3c73700 1 Processor -- start 2024-04-06T12:27:33.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.731+0000 7f8ed3c73700 1 -- start start 2024-04-06T12:27:33.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.732+0000 7f8ed3c73700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ecc06ae40 0x7f8ecc143e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:33.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.732+0000 7f8ed3c73700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ecc15d7b0 0x7f8ecc1f3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:33.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.732+0000 7f8ed3c73700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ecc15ddf0 0x7f8ecc0ce960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:33.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.732+0000 7f8ed3c73700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc154da0 con 0x7f8ecc15d7b0 2024-04-06T12:27:33.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.732+0000 7f8ed3c73700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc0ce540 con 0x7f8ecc15ddf0 2024-04-06T12:27:33.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.732+0000 7f8ed3c73700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc144370 con 0x7f8ecc06ae40 2024-04-06T12:27:33.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.732+0000 7f8ed120e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ecc15d7b0 0x7f8ecc1f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:33.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.732+0000 7f8ed120e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ecc15d7b0 0x7f8ecc1f3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33194/0 (socket says 172.21.15.189:33194) 2024-04-06T12:27:33.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.732+0000 7f8ed120e700 1 -- 172.21.15.189:0/115639910 learned_addr learned my addr 172.21.15.189:0/115639910 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:33.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.732+0000 7f8ed1a0f700 1 --2- 172.21.15.189:0/115639910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ecc06ae40 0x7f8ecc143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:33.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.732+0000 7f8ed2210700 1 --2- 172.21.15.189:0/115639910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ecc15ddf0 0x7f8ecc0ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:33.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.733+0000 7f8ed120e700 1 -- 172.21.15.189:0/115639910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ecc06ae40 msgr2=0x7f8ecc143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:33.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.733+0000 7f8ed120e700 1 --2- 172.21.15.189:0/115639910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ecc06ae40 0x7f8ecc143e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.733+0000 7f8ed120e700 1 -- 172.21.15.189:0/115639910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ecc15ddf0 msgr2=0x7f8ecc0ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:33.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.733+0000 7f8ed120e700 1 --2- 172.21.15.189:0/115639910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ecc15ddf0 0x7f8ecc0ce960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.733+0000 7f8ed120e700 1 -- 172.21.15.189:0/115639910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc009550 con 0x7f8ecc15d7b0 2024-04-06T12:27:33.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.733+0000 7f8ed120e700 1 --2- 172.21.15.189:0/115639910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ecc15d7b0 0x7f8ecc1f3440 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009c10 tx=0x7f8ebc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:33.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.733+0000 7f8ec27fc700 1 -- 172.21.15.189:0/115639910 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc004380 con 0x7f8ecc15d7b0 2024-04-06T12:27:33.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.733+0000 7f8ec27fc700 1 -- 172.21.15.189:0/115639910 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc01f070 con 0x7f8ecc15d7b0 2024-04-06T12:27:33.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.733+0000 7f8ed3c73700 1 -- 172.21.15.189:0/115639910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ecc1f3980 con 0x7f8ecc15d7b0 2024-04-06T12:27:33.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.733+0000 7f8ec27fc700 1 -- 172.21.15.189:0/115639910 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc0036a0 con 0x7f8ecc15d7b0 2024-04-06T12:27:33.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.733+0000 7f8ed3c73700 1 -- 172.21.15.189:0/115639910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ecc0cdba0 con 0x7f8ecc15d7b0 2024-04-06T12:27:33.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.734+0000 7f8ed3c73700 1 -- 172.21.15.189:0/115639910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ecc0ce680 con 0x7f8ecc15d7b0 2024-04-06T12:27:33.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.735+0000 7f8ec27fc700 1 -- 172.21.15.189:0/115639910 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 33) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f8ebc021030 con 0x7f8ecc15d7b0 2024-04-06T12:27:33.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.735+0000 7f8ec27fc700 1 --2- 172.21.15.189:0/115639910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8eb006ae00 0x7f8eb006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:33.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.735+0000 7f8ed1a0f700 1 -- 172.21.15.189:0/115639910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8eb006ae00 msgr2=0x7f8eb006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:33.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.735+0000 7f8ec27fc700 1 -- 172.21.15.189:0/115639910 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8ebc08a8a0 con 0x7f8ecc15d7b0 2024-04-06T12:27:33.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.735+0000 7f8ed1a0f700 1 --2- 172.21.15.189:0/115639910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8eb006ae00 0x7f8eb006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:33.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.739+0000 7f8ec27fc700 1 -- 172.21.15.189:0/115639910 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8ecc0ce680 con 0x7f8ecc15d7b0 2024-04-06T12:27:33.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.936+0000 7f8ed1a0f700 1 -- 172.21.15.189:0/115639910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8eb006ae00 msgr2=0x7f8eb006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:33.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.936+0000 7f8ed1a0f700 1 --2- 172.21.15.189:0/115639910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8eb006ae00 0x7f8eb006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:33.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.944+0000 7f8ed3c73700 1 -- 172.21.15.189:0/115639910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f8ecc0ce680 con 0x7f8ecc15d7b0 2024-04-06T12:27:33.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.949+0000 7f8ec27fc700 1 -- 172.21.15.189:0/115639910 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8ecc0ce680 con 0x7f8ecc15d7b0 2024-04-06T12:27:33.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.952+0000 7f8ed3c73700 1 -- 172.21.15.189:0/115639910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8eb006ae00 msgr2=0x7f8eb006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:33.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.952+0000 7f8ed3c73700 1 --2- 172.21.15.189:0/115639910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8eb006ae00 0x7f8eb006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.952+0000 7f8ed3c73700 1 -- 172.21.15.189:0/115639910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ecc15d7b0 msgr2=0x7f8ecc1f3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:33.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.952+0000 7f8ed3c73700 1 --2- 172.21.15.189:0/115639910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ecc15d7b0 0x7f8ecc1f3440 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009c10 tx=0x7f8ebc004750 comp rx=0 tx=0).stop 2024-04-06T12:27:33.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.952+0000 7f8ed3c73700 1 -- 172.21.15.189:0/115639910 shutdown_connections 2024-04-06T12:27:33.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.952+0000 7f8ed3c73700 1 --2- 172.21.15.189:0/115639910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8eb006ae00 0x7f8eb006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.952+0000 7f8ed3c73700 1 --2- 172.21.15.189:0/115639910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ecc15ddf0 0x7f8ecc0ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.952+0000 7f8ed3c73700 1 --2- 172.21.15.189:0/115639910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ecc06ae40 0x7f8ecc143e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.952+0000 7f8ed3c73700 1 --2- 172.21.15.189:0/115639910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ecc15d7b0 0x7f8ecc1f3440 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:33.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.952+0000 7f8ed3c73700 1 -- 172.21.15.189:0/115639910 >> 172.21.15.189:0/115639910 conn(0x7f8ecc0d00b0 msgr2=0x7f8ecc15c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:33.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.952+0000 7f8ed3c73700 1 -- 172.21.15.189:0/115639910 shutdown_connections 2024-04-06T12:27:33.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:33.952+0000 7f8ed3c73700 1 -- 172.21.15.189:0/115639910 wait complete. 2024-04-06T12:27:33.961 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:33.963+0000 7fb0fbe5e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:27:33.963 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-06T12:27:33.978 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:33.980+0000 7f357afe6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:27:34.039 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:34.041+0000 7fb0fbe5e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:27:34.062 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:34.064+0000 7f357afe6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:27:34.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.092+0000 7f4267612700 1 Processor -- start 2024-04-06T12:27:34.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.092+0000 7f4267612700 1 -- start start 2024-04-06T12:27:34.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.093+0000 7f4267612700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600d17a0 0x7f426006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:34.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.093+0000 7f4267612700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4260146dd0 0x7f426006d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:34.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.093+0000 7f4267612700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f42600cedf0 0x7f42600cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:34.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.093+0000 7f4267612700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42600cf760 con 0x7f42600cedf0 2024-04-06T12:27:34.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.093+0000 7f4267612700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f426006c0a0 con 0x7f4260146dd0 2024-04-06T12:27:34.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.093+0000 7f4267612700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f426006c380 con 0x7f42600d17a0 2024-04-06T12:27:34.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.093+0000 7f4265baf700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f42600cedf0 0x7f42600cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:34.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.093+0000 7f4265baf700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f42600cedf0 0x7f42600cf220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33208/0 (socket says 172.21.15.189:33208) 2024-04-06T12:27:34.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.093+0000 7f4265baf700 1 -- 172.21.15.189:0/3406422412 learned_addr learned my addr 172.21.15.189:0/3406422412 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:34.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.093+0000 7f42653ae700 1 --2- 172.21.15.189:0/3406422412 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600d17a0 0x7f426006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:34.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.093+0000 7f4264bad700 1 --2- 172.21.15.189:0/3406422412 >> v2:172.21.15.192:3300/0 conn(0x7f4260146dd0 0x7f426006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:34.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.093+0000 7f4265baf700 1 -- 172.21.15.189:0/3406422412 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600d17a0 msgr2=0x7f426006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:34.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.093+0000 7f4265baf700 1 --2- 172.21.15.189:0/3406422412 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600d17a0 0x7f426006b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:34.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.093+0000 7f4265baf700 1 -- 172.21.15.189:0/3406422412 >> v2:172.21.15.192:3300/0 conn(0x7f4260146dd0 msgr2=0x7f426006d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:34.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.093+0000 7f4265baf700 1 --2- 172.21.15.189:0/3406422412 >> v2:172.21.15.192:3300/0 conn(0x7f4260146dd0 0x7f426006d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:34.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.093+0000 7f4265baf700 1 -- 172.21.15.189:0/3406422412 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f426006c660 con 0x7f42600cedf0 2024-04-06T12:27:34.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.094+0000 7f4265baf700 1 --2- 172.21.15.189:0/3406422412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600cedf0 0x7f42600cf220 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f42600ced30 tx=0x7f425800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e36a934749b713 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:34.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.094+0000 7f4257fff700 1 -- 172.21.15.189:0/3406422412 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425800e070 con 0x7f42600cedf0 2024-04-06T12:27:34.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.094+0000 7f4257fff700 1 -- 172.21.15.189:0/3406422412 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425800b910 con 0x7f42600cedf0 2024-04-06T12:27:34.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.094+0000 7f4257fff700 1 -- 172.21.15.189:0/3406422412 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425800bcc0 con 0x7f42600cedf0 2024-04-06T12:27:34.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.094+0000 7f4267612700 1 -- 172.21.15.189:0/3406422412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600cedf0 msgr2=0x7f42600cf220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:34.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.094+0000 7f4267612700 1 --2- 172.21.15.189:0/3406422412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600cedf0 0x7f42600cf220 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f42600ced30 tx=0x7f425800b330 comp rx=0 tx=0).stop 2024-04-06T12:27:34.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.094+0000 7f4267612700 1 -- 172.21.15.189:0/3406422412 shutdown_connections 2024-04-06T12:27:34.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.094+0000 7f4267612700 1 --2- 172.21.15.189:0/3406422412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600cedf0 0x7f42600cf220 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:34.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.094+0000 7f4267612700 1 --2- 172.21.15.189:0/3406422412 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600d17a0 0x7f426006b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:34.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.094+0000 7f4267612700 1 --2- 172.21.15.189:0/3406422412 >> v2:172.21.15.192:3300/0 conn(0x7f4260146dd0 0x7f426006d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:34.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.094+0000 7f4267612700 1 -- 172.21.15.189:0/3406422412 >> 172.21.15.189:0/3406422412 conn(0x7f42600b2db0 msgr2=0x7f42600b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:34.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.094+0000 7f4267612700 1 -- 172.21.15.189:0/3406422412 shutdown_connections 2024-04-06T12:27:34.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.094+0000 7f4267612700 1 -- 172.21.15.189:0/3406422412 wait complete. 2024-04-06T12:27:34.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.096+0000 7f4267612700 1 Processor -- start 2024-04-06T12:27:34.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.096+0000 7f4267612700 1 -- start start 2024-04-06T12:27:34.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.096+0000 7f4267612700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42600cedf0 0x7f426015c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:34.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.096+0000 7f4267612700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600d17a0 0x7f42600b6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:34.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.096+0000 7f4267612700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4260146dd0 0x7f42600c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:34.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.096+0000 7f4267612700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42600b5ba0 con 0x7f4260146dd0 2024-04-06T12:27:34.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.096+0000 7f4267612700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42600b4f90 con 0x7f42600cedf0 2024-04-06T12:27:34.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.096+0000 7f4267612700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42600b5270 con 0x7f42600d17a0 2024-04-06T12:27:34.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.096+0000 7f4265baf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4260146dd0 0x7f42600c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:34.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.096+0000 7f4265baf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4260146dd0 0x7f42600c6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33224/0 (socket says 172.21.15.189:33224) 2024-04-06T12:27:34.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.096+0000 7f4265baf700 1 -- 172.21.15.189:0/85251900 learned_addr learned my addr 172.21.15.189:0/85251900 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:34.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.097+0000 7f42653ae700 1 --2- 172.21.15.189:0/85251900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42600cedf0 0x7f426015c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:34.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.097+0000 7f4264bad700 1 --2- 172.21.15.189:0/85251900 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600d17a0 0x7f42600b6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:34.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.097+0000 7f4265baf700 1 -- 172.21.15.189:0/85251900 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600d17a0 msgr2=0x7f42600b6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:34.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.097+0000 7f4265baf700 1 --2- 172.21.15.189:0/85251900 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600d17a0 0x7f42600b6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:34.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.097+0000 7f4265baf700 1 -- 172.21.15.189:0/85251900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42600cedf0 msgr2=0x7f426015c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:34.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.097+0000 7f4265baf700 1 --2- 172.21.15.189:0/85251900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42600cedf0 0x7f426015c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:34.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.097+0000 7f4265baf700 1 -- 172.21.15.189:0/85251900 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4258009b30 con 0x7f4260146dd0 2024-04-06T12:27:34.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.097+0000 7f4265baf700 1 --2- 172.21.15.189:0/85251900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4260146dd0 0x7f42600c6d60 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f42600677e0 tx=0x7f4258004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:34.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.097+0000 7f4255ffb700 1 -- 172.21.15.189:0/85251900 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42580099b0 con 0x7f4260146dd0 2024-04-06T12:27:34.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.097+0000 7f4255ffb700 1 -- 172.21.15.189:0/85251900 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425800ce80 con 0x7f4260146dd0 2024-04-06T12:27:34.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.097+0000 7f4267612700 1 -- 172.21.15.189:0/85251900 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42600b5550 con 0x7f4260146dd0 2024-04-06T12:27:34.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.097+0000 7f4255ffb700 1 -- 172.21.15.189:0/85251900 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4258004490 con 0x7f4260146dd0 2024-04-06T12:27:34.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.097+0000 7f4267612700 1 -- 172.21.15.189:0/85251900 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42600c82f0 con 0x7f4260146dd0 2024-04-06T12:27:34.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.098+0000 7f4267612700 1 -- 172.21.15.189:0/85251900 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42600b5d50 con 0x7f4260146dd0 2024-04-06T12:27:34.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.099+0000 7f4255ffb700 1 -- 172.21.15.189:0/85251900 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 33) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f42580045f0 con 0x7f4260146dd0 2024-04-06T12:27:34.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.099+0000 7f4255ffb700 1 --2- 172.21.15.189:0/85251900 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f424406ae00 0x7f424406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:34.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.099+0000 7f42653ae700 1 -- 172.21.15.189:0/85251900 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f424406ae00 msgr2=0x7f424406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:34.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.099+0000 7f4255ffb700 1 -- 172.21.15.189:0/85251900 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f425808a590 con 0x7f4260146dd0 2024-04-06T12:27:34.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.099+0000 7f42653ae700 1 --2- 172.21.15.189:0/85251900 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f424406ae00 0x7f424406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:34.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.104+0000 7f4255ffb700 1 -- 172.21.15.189:0/85251900 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f42580554a0 con 0x7f4260146dd0 2024-04-06T12:27:34.111 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:34.113+0000 7fb0fbe5e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:27:34.141 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:34.143+0000 7f357afe6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:27:34.178 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:34.180+0000 7fb0fbe5e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:27:34.211 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:34.213+0000 7f357afe6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:27:34.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.300+0000 7f42653ae700 1 -- 172.21.15.189:0/85251900 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f424406ae00 msgr2=0x7f424406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:34.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.300+0000 7f42653ae700 1 --2- 172.21.15.189:0/85251900 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f424406ae00 0x7f424406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:34.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.306+0000 7f4267612700 1 -- 172.21.15.189:0/85251900 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f4260074990 con 0x7f4260146dd0 2024-04-06T12:27:34.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.310+0000 7f4255ffb700 1 -- 172.21.15.189:0/85251900 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7f4258058ac0 con 0x7f4260146dd0 2024-04-06T12:27:34.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.312+0000 7f4267612700 1 -- 172.21.15.189:0/85251900 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f424406ae00 msgr2=0x7f424406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:34.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.312+0000 7f4267612700 1 --2- 172.21.15.189:0/85251900 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f424406ae00 0x7f424406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:34.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.312+0000 7f4267612700 1 -- 172.21.15.189:0/85251900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4260146dd0 msgr2=0x7f42600c6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:34.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.312+0000 7f4267612700 1 --2- 172.21.15.189:0/85251900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4260146dd0 0x7f42600c6d60 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f42600677e0 tx=0x7f4258004ad0 comp rx=0 tx=0).stop 2024-04-06T12:27:34.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.312+0000 7f4267612700 1 -- 172.21.15.189:0/85251900 shutdown_connections 2024-04-06T12:27:34.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.312+0000 7f4267612700 1 --2- 172.21.15.189:0/85251900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4260146dd0 0x7f42600c6d60 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:34.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.312+0000 7f4267612700 1 --2- 172.21.15.189:0/85251900 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f424406ae00 0x7f424406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:34.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.312+0000 7f4267612700 1 --2- 172.21.15.189:0/85251900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42600cedf0 0x7f426015c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:34.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.312+0000 7f4267612700 1 --2- 172.21.15.189:0/85251900 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600d17a0 0x7f42600b6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:34.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.312+0000 7f4267612700 1 -- 172.21.15.189:0/85251900 >> 172.21.15.189:0/85251900 conn(0x7f42600b2db0 msgr2=0x7f426015e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:34.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.312+0000 7f4267612700 1 -- 172.21.15.189:0/85251900 shutdown_connections 2024-04-06T12:27:34.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.312+0000 7f4267612700 1 -- 172.21.15.189:0/85251900 wait complete. 2024-04-06T12:27:34.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:34.323 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-06T12:27:34.355 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:34.357+0000 7fb0fbe5e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:27:34.385 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-04-06T12:27:34.396 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:34.398+0000 7f357afe6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:27:34.510 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:34.512+0000 7fb0fbe5e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:27:34.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.525+0000 7f6679757700 1 Processor -- start 2024-04-06T12:27:34.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.525+0000 7f6679757700 1 -- start start 2024-04-06T12:27:34.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.525+0000 7f6679757700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f66740b5380 0x7f66740d2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:34.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.525+0000 7f6679757700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66740b5cc0 con 0x7f66740b4ed0 2024-04-06T12:27:34.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.525+0000 7f6679757700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66740b57d0 con 0x7f667415d590 2024-04-06T12:27:34.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.525+0000 7f6679757700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66740d1500 con 0x7f66740b5380 2024-04-06T12:27:34.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.526+0000 7f66737fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f66740b5380 0x7f66740d2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:34.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.526+0000 7f66737fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f66740b5380 0x7f66740d2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33234/0 (socket says 172.21.15.189:33234) 2024-04-06T12:27:34.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.526+0000 7f66737fe700 1 -- 172.21.15.189:0/2072976633 learned_addr learned my addr 172.21.15.189:0/2072976633 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:34.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.526+0000 7f66727fc700 1 --1- 172.21.15.189:0/2072976633 >> v1:172.21.15.189:6789/0 conn(0x7f66740b4ed0 0x7f66740d2880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52648/0 (socket says 172.21.15.189:52648) 2024-04-06T12:27:34.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.526+0000 7f66737fe700 1 -- 172.21.15.189:0/2072976633 >> v1:172.21.15.192:6789/0 conn(0x7f667415d590 legacy=0x7f66740b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:34.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.526+0000 7f66737fe700 1 -- 172.21.15.189:0/2072976633 >> v1:172.21.15.189:6789/0 conn(0x7f66740b4ed0 legacy=0x7f66740d2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:34.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.526+0000 7f66737fe700 1 -- 172.21.15.189:0/2072976633 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66740d17e0 con 0x7f66740b5380 2024-04-06T12:27:34.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.526+0000 7f66737fe700 1 --2- 172.21.15.189:0/2072976633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66740b5380 0x7f66740d2f90 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f666c009e50 tx=0x7f666c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93ef4446331c332b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:34.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.526+0000 7f6671ffb700 1 -- 172.21.15.189:0/2072976633 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c00e070 con 0x7f66740b5380 2024-04-06T12:27:34.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.526+0000 7f6671ffb700 1 -- 172.21.15.189:0/2072976633 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666c00b9c0 con 0x7f66740b5380 2024-04-06T12:27:34.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.526+0000 7f6671ffb700 1 -- 172.21.15.189:0/2072976633 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c00bd70 con 0x7f66740b5380 2024-04-06T12:27:34.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.527+0000 7f6679757700 1 -- 172.21.15.189:0/2072976633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66740b5380 msgr2=0x7f66740d2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:34.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.527+0000 7f6679757700 1 --2- 172.21.15.189:0/2072976633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66740b5380 0x7f66740d2f90 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f666c009e50 tx=0x7f666c00b330 comp rx=0 tx=0).stop 2024-04-06T12:27:34.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.527+0000 7f6679757700 1 -- 172.21.15.189:0/2072976633 shutdown_connections 2024-04-06T12:27:34.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.527+0000 7f6679757700 1 --2- 172.21.15.189:0/2072976633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66740b5380 0x7f66740d2f90 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:34.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.527+0000 7f6679757700 1 -- 172.21.15.189:0/2072976633 >> 172.21.15.189:0/2072976633 conn(0x7f667406bbc0 msgr2=0x7f667406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:34.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.527+0000 7f6679757700 1 -- 172.21.15.189:0/2072976633 shutdown_connections 2024-04-06T12:27:34.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.527+0000 7f6679757700 1 -- 172.21.15.189:0/2072976633 wait complete. 2024-04-06T12:27:34.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.528+0000 7f6679757700 1 Processor -- start 2024-04-06T12:27:34.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.528+0000 7f6679757700 1 -- start start 2024-04-06T12:27:34.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.529+0000 7f6679757700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66740b4ed0 0x7f66740c90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:34.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.529+0000 7f6679757700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66740b5380 0x7f6674153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:34.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.529+0000 7f6679757700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f667415d590 0x7f6674153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:34.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.529+0000 7f6679757700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6674154e40 con 0x7f66740b5380 2024-04-06T12:27:34.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.529+0000 7f6679757700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6674154230 con 0x7f667415d590 2024-04-06T12:27:34.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.529+0000 7f6679757700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6674154510 con 0x7f66740b4ed0 2024-04-06T12:27:34.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.529+0000 7f66727fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66740b5380 0x7f6674153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:34.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.529+0000 7f66727fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66740b5380 0x7f6674153700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33236/0 (socket says 172.21.15.189:33236) 2024-04-06T12:27:34.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.529+0000 7f66727fc700 1 -- 172.21.15.189:0/315912494 learned_addr learned my addr 172.21.15.189:0/315912494 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:34.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.529+0000 7f6672ffd700 1 --2- 172.21.15.189:0/315912494 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66740b4ed0 0x7f66740c90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:34.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.529+0000 7f66737fe700 1 --2- 172.21.15.189:0/315912494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f667415d590 0x7f6674153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:34.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.530+0000 7f66727fc700 1 -- 172.21.15.189:0/315912494 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66740b4ed0 msgr2=0x7f66740c90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:34.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.530+0000 7f66727fc700 1 --2- 172.21.15.189:0/315912494 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66740b4ed0 0x7f66740c90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:34.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.530+0000 7f66727fc700 1 -- 172.21.15.189:0/315912494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f667415d590 msgr2=0x7f6674153c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:34.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.530+0000 7f66727fc700 1 --2- 172.21.15.189:0/315912494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f667415d590 0x7f6674153c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:34.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.530+0000 7f66727fc700 1 -- 172.21.15.189:0/315912494 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f666c009b30 con 0x7f66740b5380 2024-04-06T12:27:34.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.530+0000 7f66727fc700 1 --2- 172.21.15.189:0/315912494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66740b5380 0x7f6674153700 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f666800c520 tx=0x7f666800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:34.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.530+0000 7f665b7fe700 1 -- 172.21.15.189:0/315912494 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6668005cb0 con 0x7f66740b5380 2024-04-06T12:27:34.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.530+0000 7f665b7fe700 1 -- 172.21.15.189:0/315912494 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6668005e10 con 0x7f66740b5380 2024-04-06T12:27:34.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.530+0000 7f6679757700 1 -- 172.21.15.189:0/315912494 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66741547f0 con 0x7f66740b5380 2024-04-06T12:27:34.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.530+0000 7f665b7fe700 1 -- 172.21.15.189:0/315912494 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66680072d0 con 0x7f66740b5380 2024-04-06T12:27:34.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.530+0000 7f6679757700 1 -- 172.21.15.189:0/315912494 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66741552d0 con 0x7f66740b5380 2024-04-06T12:27:34.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.531+0000 7f6679757700 1 -- 172.21.15.189:0/315912494 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6674074990 con 0x7f66740b5380 2024-04-06T12:27:34.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.531+0000 7f665b7fe700 1 -- 172.21.15.189:0/315912494 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 33) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f6668007430 con 0x7f66740b5380 2024-04-06T12:27:34.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.532+0000 7f665b7fe700 1 --2- 172.21.15.189:0/315912494 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006ae00 0x7f666006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:34.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.532+0000 7f6672ffd700 1 -- 172.21.15.189:0/315912494 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006ae00 msgr2=0x7f666006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:34.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.532+0000 7f665b7fe700 1 -- 172.21.15.189:0/315912494 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f66680888a0 con 0x7f66740b5380 2024-04-06T12:27:34.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.532+0000 7f6672ffd700 1 --2- 172.21.15.189:0/315912494 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006ae00 0x7f666006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:34.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.540+0000 7f665b7fe700 1 -- 172.21.15.189:0/315912494 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f66680538e0 con 0x7f66740b5380 2024-04-06T12:27:34.571 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:34.573+0000 7f357afe6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:27:34.620 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:34.621+0000 7fb0fbe5e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:27:34.685 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:34.687+0000 7f357afe6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:27:34.700 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:34.702+0000 7fb0fbe5e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:27:34.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.733+0000 7f6672ffd700 1 -- 172.21.15.189:0/315912494 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006ae00 msgr2=0x7f666006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:34.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.733+0000 7f6672ffd700 1 --2- 172.21.15.189:0/315912494 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006ae00 0x7f666006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:34.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:34.769+0000 7f6679757700 1 -- 172.21.15.189:0/315912494 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f66740d2080 con 0x7f666006ae00 2024-04-06T12:27:34.769 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:34.772+0000 7f357afe6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:27:34.774 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:34.776+0000 7fb0fbe5e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:27:34.845 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:34.847+0000 7f357afe6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:27:34.976 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:34.978+0000 7fb0fbe5e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:27:35.047 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:35.049+0000 7f357afe6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:27:35.054 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:35.054+0000 7fb0fbe5e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:27:35.123 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:35.125+0000 7f357afe6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:27:35.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:35.134+0000 7f6672ffd700 1 -- 172.21.15.189:0/315912494 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006ae00 msgr2=0x7f666006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:35.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:35.134+0000 7f6672ffd700 1 --2- 172.21.15.189:0/315912494 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006ae00 0x7f666006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-06T12:27:35.368 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:35.369+0000 7fb0fbe5e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:27:35.443 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:35.445+0000 7f357afe6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:27:35.681 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:35.683+0000 7fb0fbe5e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:27:35.763 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:35.765+0000 7f357afe6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:27:35.839 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:35.841+0000 7fb0fbe5e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:27:35.910 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:35.912+0000 7fb0fbe5e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:27:35.923 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:35.925+0000 7f357afe6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:27:35.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:35.935+0000 7f6672ffd700 1 -- 172.21.15.189:0/315912494 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006ae00 msgr2=0x7f666006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:35.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:35.935+0000 7f6672ffd700 1 --2- 172.21.15.189:0/315912494 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006ae00 0x7f666006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-06T12:27:35.998 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:36.000+0000 7f357afe6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:27:36.521 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:36.523+0000 7fb0fbe5e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:27:36.598 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:36.600+0000 7fb0fbe5e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:27:36.616 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:36.619+0000 7f357afe6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:27:36.698 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:36.699+0000 7f357afe6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:27:36.726 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:36.728+0000 7fb0fbe5e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:27:36.824 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:36.827+0000 7f357afe6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:27:37.235 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:37.238+0000 7fb0fbe5e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:27:37.339 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:37.342+0000 7f357afe6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:27:37.387 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:37.389+0000 7fb0fbe5e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:27:37.494 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:37.497+0000 7f357afe6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:27:37.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:37.537+0000 7f6672ffd700 1 -- 172.21.15.189:0/315912494 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006ae00 msgr2=0x7f666006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:37.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:37.537+0000 7f6672ffd700 1 --2- 172.21.15.189:0/315912494 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006ae00 0x7f666006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-06T12:27:37.539 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:37.542+0000 7fb0fbe5e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:27:37.651 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:37.654+0000 7f357afe6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:27:38.253 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:38.256+0000 7fb0fbe5e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:27:38.375 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:38.378+0000 7f357afe6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:27:38.491 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:38.493+0000 7fb0fbe5e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:27:38.565 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:38.567+0000 7fb0fbe5e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:27:38.618 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:38.621+0000 7f357afe6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:27:38.694 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:38.697+0000 7f357afe6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:27:38.935 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:38.938+0000 7fb0fbe5e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:27:39.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:39.010+0000 7f665b7fe700 1 -- 172.21.15.189:0/315912494 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 34) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f6668086c80 con 0x7f66740b5380 2024-04-06T12:27:39.071 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:39.074+0000 7f357afe6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:27:39.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:39.097+0000 7f665b7fe700 1 -- 172.21.15.189:0/315912494 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mgrmap(e 35) v1 ==== 88873+0+0 (secure 0 0 0) 0x7f6668053cb0 con 0x7f66740b5380 2024-04-06T12:27:39.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:39.098+0000 7f665b7fe700 1 -- 172.21.15.189:0/315912494 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006ae00 msgr2=0x7f666006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:39.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:39.098+0000 7f665b7fe700 1 --2- 172.21.15.189:0/315912494 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006ae00 0x7f666006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:39.235 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:39.237+0000 7f34efef8700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:39.235 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:39.237+0000 7f34efef8700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:39.235 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:39.237+0000 7f34efef8700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:39.235 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:39.237+0000 7f34efef8700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:39.236 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:39.237+0000 7f34efef8700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:39.236 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:39.238+0000 7f34edef4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:39.236 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:39.238+0000 7f34edef4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:39.236 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:39.238+0000 7f34edef4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:39.236 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:39.238+0000 7f34edef4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:39.236 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:39.238+0000 7f34edef4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:40.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.103+0000 7f665b7fe700 1 -- 172.21.15.189:0/315912494 <== mon.0 v2:172.21.15.189:3300/0 9 ==== mgrmap(e 36) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f666805afc0 con 0x7f66740b5380 2024-04-06T12:27:40.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.104+0000 7f665b7fe700 1 --2- 172.21.15.189:0/315912494 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006ae00 0x7f666006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:40.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.104+0000 7f665b7fe700 1 -- 172.21.15.189:0/315912494 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6660027a50 con 0x7f666006ae00 2024-04-06T12:27:40.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.105+0000 7f6672ffd700 1 --2- 172.21.15.189:0/315912494 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006ae00 0x7f666006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:40.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.109+0000 7f6672ffd700 1 --2- 172.21.15.189:0/315912494 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f666006ae00 0x7f666006d2c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6664007fd0 tx=0x7f6664007ec0 comp rx=0 tx=0).ready entity=mgr.6798 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:40.112 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:40.115+0000 7f34f7f08700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-04-06T12:27:40.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.116+0000 7f665b7fe700 1 -- 172.21.15.189:0/315912494 <== mgr.6798 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f6660027a50 con 0x7f666006ae00 2024-04-06T12:27:40.114 INFO:teuthology.orchestra.run.smithi189.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-04-06T12:27:40.118 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-06T12:27:40.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:40.119 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-06T12:27:40.139 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-06T12:27:40.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.307+0000 7f98f4cc2700 1 Processor -- start 2024-04-06T12:27:40.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.307+0000 7f98f4cc2700 1 -- start start 2024-04-06T12:27:40.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98f4cc2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f98f015a550 0x7f98f00cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:40.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98f4cc2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98f00b4d70 0x7f98f00b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:40.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98f4cc2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f00cdbe0 con 0x7f98f00cf900 2024-04-06T12:27:40.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98f4cc2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98f015a940 con 0x7f98f015a550 2024-04-06T12:27:40.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98f4cc2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f015ac00 con 0x7f98f00b4d70 2024-04-06T12:27:40.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98edd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f98f00cf900 0x7f98f00b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44320/0 (socket says 172.21.15.189:44320) 2024-04-06T12:27:40.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98edd9b700 1 -- 172.21.15.189:0/493843799 learned_addr learned my addr 172.21.15.189:0/493843799 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:40.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98ee59c700 1 --2- 172.21.15.189:0/493843799 >> v2:172.21.15.192:3300/0 conn(0x7f98f015a550 0x7f98f00cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:40.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98eed9d700 1 --2- 172.21.15.189:0/493843799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98f00b4d70 0x7f98f00b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:40.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98ed59a700 1 -- 172.21.15.189:0/493843799 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628469743 0 0) 0x7f98f00cdbe0 con 0x7f98f00cf900 2024-04-06T12:27:40.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98ed59a700 1 -- 172.21.15.189:0/493843799 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d80035b0 con 0x7f98f00cf900 2024-04-06T12:27:40.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98ed59a700 1 -- 172.21.15.189:0/493843799 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 686637718 0 0) 0x7f98d80035b0 con 0x7f98f00cf900 2024-04-06T12:27:40.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98ed59a700 1 -- 172.21.15.189:0/493843799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98f00b4d70 msgr2=0x7f98f00b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:40.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98ed59a700 1 --2- 172.21.15.189:0/493843799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98f00b4d70 0x7f98f00b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:40.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98ed59a700 1 -- 172.21.15.189:0/493843799 >> v2:172.21.15.192:3300/0 conn(0x7f98f015a550 msgr2=0x7f98f00cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:40.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98ed59a700 1 --2- 172.21.15.189:0/493843799 >> v2:172.21.15.192:3300/0 conn(0x7f98f015a550 0x7f98f00cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:40.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98ed59a700 1 -- 172.21.15.189:0/493843799 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f015aee0 con 0x7f98f00cf900 2024-04-06T12:27:40.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98ed59a700 1 -- 172.21.15.189:0/493843799 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f98e4001670 con 0x7f98f00cf900 2024-04-06T12:27:40.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98ed59a700 1 -- 172.21.15.189:0/493843799 >> v1:172.21.15.189:6789/0 conn(0x7f98f00cf900 legacy=0x7f98f00b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:40.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98ed59a700 1 --2- 172.21.15.189:0/493843799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98d8003a00 0x7f98d8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:40.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98ed59a700 1 --2- 172.21.15.189:0/493843799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98f00b4d70 0x7f98d8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:40.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98ed59a700 1 --2- 172.21.15.189:0/493843799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98d8004d90 0x7f98d8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:40.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98ed59a700 1 -- 172.21.15.189:0/493843799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f015a940 con 0x7f98d8004d90 2024-04-06T12:27:40.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98ed59a700 1 -- 172.21.15.189:0/493843799 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f015ac00 con 0x7f98d8003a00 2024-04-06T12:27:40.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.308+0000 7f98ed59a700 1 -- 172.21.15.189:0/493843799 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d800dc20 con 0x7f98f00b4d70 2024-04-06T12:27:40.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.309+0000 7f98ee59c700 1 --2- 172.21.15.189:0/493843799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98d8003a00 0x7f98d8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:40.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.309+0000 7f98eed9d700 1 --2- 172.21.15.189:0/493843799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98d8004d90 0x7f98d8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:40.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.309+0000 7f98edd9b700 1 --2- 172.21.15.189:0/493843799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98f00b4d70 0x7f98d8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:40.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.309+0000 7f98eed9d700 1 -- 172.21.15.189:0/493843799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98f00b4d70 msgr2=0x7f98d8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:40.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.309+0000 7f98eed9d700 1 --2- 172.21.15.189:0/493843799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98f00b4d70 0x7f98d8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:40.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.309+0000 7f98eed9d700 1 -- 172.21.15.189:0/493843799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98d8003a00 msgr2=0x7f98d8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:40.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.309+0000 7f98eed9d700 1 --2- 172.21.15.189:0/493843799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98d8003a00 0x7f98d8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:40.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.309+0000 7f98eed9d700 1 -- 172.21.15.189:0/493843799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d80035b0 con 0x7f98d8004d90 2024-04-06T12:27:40.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.309+0000 7f98ee59c700 1 --2- 172.21.15.189:0/493843799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98d8003a00 0x7f98d8003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:40.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.309+0000 7f98eed9d700 1 --2- 172.21.15.189:0/493843799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98d8004d90 0x7f98d8005210 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f98e8009f90 tx=0x7f98d8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=739395adee430d18 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:40.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.309+0000 7f98ed59a700 1 -- 172.21.15.189:0/493843799 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e800ebd0 con 0x7f98d8004d90 2024-04-06T12:27:40.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.309+0000 7f98ed59a700 1 -- 172.21.15.189:0/493843799 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e800f040 con 0x7f98d8004d90 2024-04-06T12:27:40.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.309+0000 7f98ed59a700 1 -- 172.21.15.189:0/493843799 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8014480 con 0x7f98d8004d90 2024-04-06T12:27:40.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.310+0000 7f98f4cc2700 1 -- 172.21.15.189:0/493843799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98d8004d90 msgr2=0x7f98d8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:40.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.310+0000 7f98f4cc2700 1 --2- 172.21.15.189:0/493843799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98d8004d90 0x7f98d8005210 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f98e8009f90 tx=0x7f98d8005880 comp rx=0 tx=0).stop 2024-04-06T12:27:40.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.310+0000 7f98eed9d700 1 -- 172.21.15.189:0/493843799 reap_dead start 2024-04-06T12:27:40.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.310+0000 7f98f4cc2700 1 -- 172.21.15.189:0/493843799 shutdown_connections 2024-04-06T12:27:40.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.310+0000 7f98f4cc2700 1 -- 172.21.15.189:0/493843799 >> 172.21.15.189:0/493843799 conn(0x7f98f00d00b0 msgr2=0x7f98f00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:40.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.310+0000 7f98f4cc2700 1 -- 172.21.15.189:0/493843799 shutdown_connections 2024-04-06T12:27:40.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.310+0000 7f98f4cc2700 1 -- 172.21.15.189:0/493843799 wait complete. 2024-04-06T12:27:40.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.311+0000 7f98f4cc2700 1 Processor -- start 2024-04-06T12:27:40.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.311+0000 7f98f4cc2700 1 -- start start 2024-04-06T12:27:40.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.312+0000 7f98f4cc2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98d8004d90 0x7f98f0141060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:40.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.312+0000 7f98f4cc2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98f01f2990 0x7f98f013df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:40.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.312+0000 7f98f4cc2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98f00cc210 0x7f98f00b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:40.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.312+0000 7f98f4cc2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f014d390 con 0x7f98f00cc210 2024-04-06T12:27:40.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.312+0000 7f98f4cc2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f0141950 con 0x7f98d8004d90 2024-04-06T12:27:40.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.312+0000 7f98f4cc2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f01f2df0 con 0x7f98f01f2990 2024-04-06T12:27:40.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.312+0000 7f98eed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98f00cc210 0x7f98f00b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:40.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.312+0000 7f98eed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98f00cc210 0x7f98f00b0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48404/0 (socket says 172.21.15.189:48404) 2024-04-06T12:27:40.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.312+0000 7f98eed9d700 1 -- 172.21.15.189:0/599018483 learned_addr learned my addr 172.21.15.189:0/599018483 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:40.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.312+0000 7f98edd9b700 1 --2- 172.21.15.189:0/599018483 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98f01f2990 0x7f98f013df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:40.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.312+0000 7f98ee59c700 1 --2- 172.21.15.189:0/599018483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98d8004d90 0x7f98f0141060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:40.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.312+0000 7f98eed9d700 1 -- 172.21.15.189:0/599018483 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98f01f2990 msgr2=0x7f98f013df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:40.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.312+0000 7f98eed9d700 1 --2- 172.21.15.189:0/599018483 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98f01f2990 0x7f98f013df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:40.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.312+0000 7f98eed9d700 1 -- 172.21.15.189:0/599018483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98d8004d90 msgr2=0x7f98f0141060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:40.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.312+0000 7f98eed9d700 1 --2- 172.21.15.189:0/599018483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98d8004d90 0x7f98f0141060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:40.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.312+0000 7f98eed9d700 1 -- 172.21.15.189:0/599018483 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f014cdd0 con 0x7f98f00cc210 2024-04-06T12:27:40.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.312+0000 7f98ee59c700 1 --2- 172.21.15.189:0/599018483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98d8004d90 0x7f98f0141060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:40.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.313+0000 7f98eed9d700 1 --2- 172.21.15.189:0/599018483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98f00cc210 0x7f98f00b0db0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f98d80046e0 tx=0x7f98e8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:40.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.313+0000 7f98deffd700 1 -- 172.21.15.189:0/599018483 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8007000 con 0x7f98f00cc210 2024-04-06T12:27:40.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.313+0000 7f98deffd700 1 -- 172.21.15.189:0/599018483 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e8014a70 con 0x7f98f00cc210 2024-04-06T12:27:40.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.313+0000 7f98f4cc2700 1 -- 172.21.15.189:0/599018483 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f014d0b0 con 0x7f98f00cc210 2024-04-06T12:27:40.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.313+0000 7f98deffd700 1 -- 172.21.15.189:0/599018483 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8014e40 con 0x7f98f00cc210 2024-04-06T12:27:40.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.313+0000 7f98f4cc2700 1 -- 172.21.15.189:0/599018483 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f013da00 con 0x7f98f00cc210 2024-04-06T12:27:40.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.314+0000 7f98f4cc2700 1 -- 172.21.15.189:0/599018483 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98f0067bb0 con 0x7f98f00cc210 2024-04-06T12:27:40.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.314+0000 7f98deffd700 1 -- 172.21.15.189:0/599018483 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 36) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f98e8023030 con 0x7f98f00cc210 2024-04-06T12:27:40.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.315+0000 7f98deffd700 1 --2- 172.21.15.189:0/599018483 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f98d006ab30 0x7f98d006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:40.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.315+0000 7f98ee59c700 1 --2- 172.21.15.189:0/599018483 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f98d006ab30 0x7f98d006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:40.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.315+0000 7f98deffd700 1 -- 172.21.15.189:0/599018483 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f98e808a130 con 0x7f98f00cc210 2024-04-06T12:27:40.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.315+0000 7f98ee59c700 1 --2- 172.21.15.189:0/599018483 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f98d006ab30 0x7f98d006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f98f01f1ee0 tx=0x7f98e0008040 comp rx=0 tx=0).ready entity=mgr.6798 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:40.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.319+0000 7f98deffd700 1 -- 172.21.15.189:0/599018483 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f98e80553b0 con 0x7f98f00cc210 2024-04-06T12:27:40.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:40.568+0000 7f98f4cc2700 1 -- 172.21.15.189:0/599018483 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f98f0074990 con 0x7f98f00cc210 2024-04-06T12:27:41.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.110+0000 7f98deffd700 1 -- 172.21.15.189:0/599018483 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 37) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f98e8055780 con 0x7f98f00cc210 2024-04-06T12:27:41.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.111+0000 7f98deffd700 1 -- 172.21.15.189:0/599018483 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v37) v1 ==== 93+0+0 (secure 0 0 0) 0x7f98e806ceb0 con 0x7f98f00cc210 2024-04-06T12:27:41.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.113+0000 7f98f4cc2700 1 -- 172.21.15.189:0/599018483 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f98d006ab30 msgr2=0x7f98d006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:41.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.113+0000 7f98f4cc2700 1 --2- 172.21.15.189:0/599018483 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f98d006ab30 0x7f98d006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f98f01f1ee0 tx=0x7f98e0008040 comp rx=0 tx=0).stop 2024-04-06T12:27:41.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.113+0000 7f98f4cc2700 1 -- 172.21.15.189:0/599018483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98f00cc210 msgr2=0x7f98f00b0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:41.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.113+0000 7f98f4cc2700 1 --2- 172.21.15.189:0/599018483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98f00cc210 0x7f98f00b0db0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f98d80046e0 tx=0x7f98e8009750 comp rx=0 tx=0).stop 2024-04-06T12:27:41.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.113+0000 7f98f4cc2700 1 -- 172.21.15.189:0/599018483 shutdown_connections 2024-04-06T12:27:41.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.113+0000 7f98f4cc2700 1 --2- 172.21.15.189:0/599018483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98f00cc210 0x7f98f00b0db0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:41.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.113+0000 7f98f4cc2700 1 --2- 172.21.15.189:0/599018483 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f98d006ab30 0x7f98d006cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:41.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.113+0000 7f98f4cc2700 1 --2- 172.21.15.189:0/599018483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98d8004d90 0x7f98f0141060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:41.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.113+0000 7f98f4cc2700 1 --2- 172.21.15.189:0/599018483 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98f01f2990 0x7f98f013df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:41.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.113+0000 7f98f4cc2700 1 -- 172.21.15.189:0/599018483 >> 172.21.15.189:0/599018483 conn(0x7f98f00d00b0 msgr2=0x7f98f0155c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:41.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.114+0000 7f98f4cc2700 1 -- 172.21.15.189:0/599018483 shutdown_connections 2024-04-06T12:27:41.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.114+0000 7f98f4cc2700 1 -- 172.21.15.189:0/599018483 wait complete. 2024-04-06T12:27:41.124 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T12:27:41.125 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:27:41.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:27:41.209 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T12:27:41.209 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:27:41.209 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:27:41.226 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:27:41.226 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:27:41.283 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:27:41.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.451+0000 7f688e6eb700 1 Processor -- start 2024-04-06T12:27:41.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.452+0000 7f688e6eb700 1 -- start start 2024-04-06T12:27:41.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.452+0000 7f688e6eb700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f68880d11b0 0x7f68880d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:41.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.452+0000 7f688e6eb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68880d1ad0 con 0x7f68880cfa00 2024-04-06T12:27:41.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.452+0000 7f688e6eb700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68880cbd80 con 0x7f68880cb870 2024-04-06T12:27:41.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.452+0000 7f688e6eb700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68880ccdc0 con 0x7f68880d11b0 2024-04-06T12:27:41.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.453+0000 7f6887fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f68880d11b0 0x7f68880d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:41.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.453+0000 7f688cc88700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f68880cfa00 0x7f68880cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44340/0 (socket says 172.21.15.189:44340) 2024-04-06T12:27:41.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.453+0000 7f688cc88700 1 -- 172.21.15.189:0/2763625197 learned_addr learned my addr 172.21.15.189:0/2763625197 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:41.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.453+0000 7f6887fff700 1 -- 172.21.15.189:0/2763625197 >> v1:172.21.15.192:6789/0 conn(0x7f68880cb870 legacy=0x7f688815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:41.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.453+0000 7f6887fff700 1 -- 172.21.15.189:0/2763625197 >> v1:172.21.15.189:6789/0 conn(0x7f68880cfa00 legacy=0x7f68880cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:41.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.453+0000 7f6886ffd700 1 -- 172.21.15.189:0/2763625197 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714840809 0 0) 0x7f68880d1ad0 con 0x7f68880cfa00 2024-04-06T12:27:41.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.453+0000 7f6887fff700 1 -- 172.21.15.189:0/2763625197 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68880cef20 con 0x7f68880d11b0 2024-04-06T12:27:41.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.453+0000 7f6887fff700 1 --2- 172.21.15.189:0/2763625197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68880d11b0 0x7f68880d1590 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6878009840 tx=0x7f6878009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf5d7d538851f1f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:41.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.453+0000 7f6886ffd700 1 -- 172.21.15.189:0/2763625197 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6878011830 con 0x7f68880d11b0 2024-04-06T12:27:41.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.453+0000 7f6886ffd700 1 -- 172.21.15.189:0/2763625197 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6878011990 con 0x7f68880d11b0 2024-04-06T12:27:41.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.453+0000 7f6886ffd700 1 -- 172.21.15.189:0/2763625197 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6878011d20 con 0x7f68880d11b0 2024-04-06T12:27:41.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.454+0000 7f688e6eb700 1 -- 172.21.15.189:0/2763625197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68880d11b0 msgr2=0x7f68880d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:41.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.454+0000 7f688e6eb700 1 --2- 172.21.15.189:0/2763625197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68880d11b0 0x7f68880d1590 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6878009840 tx=0x7f6878009b50 comp rx=0 tx=0).stop 2024-04-06T12:27:41.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.454+0000 7f688e6eb700 1 -- 172.21.15.189:0/2763625197 shutdown_connections 2024-04-06T12:27:41.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.454+0000 7f688e6eb700 1 --2- 172.21.15.189:0/2763625197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68880d11b0 0x7f68880d1590 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:41.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.454+0000 7f688e6eb700 1 -- 172.21.15.189:0/2763625197 >> 172.21.15.189:0/2763625197 conn(0x7f68880b2db0 msgr2=0x7f68880b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:41.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.454+0000 7f688e6eb700 1 -- 172.21.15.189:0/2763625197 shutdown_connections 2024-04-06T12:27:41.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.454+0000 7f688e6eb700 1 -- 172.21.15.189:0/2763625197 wait complete. 2024-04-06T12:27:41.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.455+0000 7f688e6eb700 1 Processor -- start 2024-04-06T12:27:41.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.455+0000 7f688e6eb700 1 -- start start 2024-04-06T12:27:41.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.456+0000 7f688e6eb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68880cb870 0x7f688815c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:41.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.456+0000 7f688e6eb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68880cfa00 0x7f68880c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:41.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.456+0000 7f688e6eb700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68880d11b0 0x7f688806b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:41.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.456+0000 7f688e6eb700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688806c8f0 con 0x7f68880cb870 2024-04-06T12:27:41.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.456+0000 7f688e6eb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688815e1b0 con 0x7f68880cfa00 2024-04-06T12:27:41.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.456+0000 7f688e6eb700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f688815de80 con 0x7f68880d11b0 2024-04-06T12:27:41.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.456+0000 7f6887fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68880cb870 0x7f688815c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:41.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.456+0000 7f6887fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68880cb870 0x7f688815c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48428/0 (socket says 172.21.15.189:48428) 2024-04-06T12:27:41.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.456+0000 7f6887fff700 1 -- 172.21.15.189:0/1872316351 learned_addr learned my addr 172.21.15.189:0/1872316351 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:41.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.456+0000 7f68877fe700 1 --2- 172.21.15.189:0/1872316351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68880cfa00 0x7f68880c4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:41.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.456+0000 7f688cc88700 1 --2- 172.21.15.189:0/1872316351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68880d11b0 0x7f688806b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:41.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.457+0000 7f6887fff700 1 -- 172.21.15.189:0/1872316351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68880d11b0 msgr2=0x7f688806b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:41.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.457+0000 7f6887fff700 1 --2- 172.21.15.189:0/1872316351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68880d11b0 0x7f688806b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:41.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.457+0000 7f6887fff700 1 -- 172.21.15.189:0/1872316351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68880cfa00 msgr2=0x7f68880c4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:41.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.457+0000 7f6887fff700 1 --2- 172.21.15.189:0/1872316351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68880cfa00 0x7f68880c4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:41.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.457+0000 7f6887fff700 1 -- 172.21.15.189:0/1872316351 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68780094a0 con 0x7f68880cb870 2024-04-06T12:27:41.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.457+0000 7f6887fff700 1 --2- 172.21.15.189:0/1872316351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68880cb870 0x7f688815c640 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f6878014040 tx=0x7f6878004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:41.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.457+0000 7f6884ff9700 1 -- 172.21.15.189:0/1872316351 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687801a860 con 0x7f68880cb870 2024-04-06T12:27:41.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.457+0000 7f6884ff9700 1 -- 172.21.15.189:0/1872316351 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687801a9c0 con 0x7f68880cb870 2024-04-06T12:27:41.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.457+0000 7f6884ff9700 1 -- 172.21.15.189:0/1872316351 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687801ad30 con 0x7f68880cb870 2024-04-06T12:27:41.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.457+0000 7f688e6eb700 1 -- 172.21.15.189:0/1872316351 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68880c5290 con 0x7f68880cb870 2024-04-06T12:27:41.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.457+0000 7f688e6eb700 1 -- 172.21.15.189:0/1872316351 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f688806c330 con 0x7f68880cb870 2024-04-06T12:27:41.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.458+0000 7f688e6eb700 1 -- 172.21.15.189:0/1872316351 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68880c4bc0 con 0x7f68880cb870 2024-04-06T12:27:41.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.459+0000 7f6884ff9700 1 -- 172.21.15.189:0/1872316351 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 37) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f6878022070 con 0x7f68880cb870 2024-04-06T12:27:41.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.459+0000 7f6884ff9700 1 --2- 172.21.15.189:0/1872316351 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f687406adb0 0x7f687406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:41.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.459+0000 7f68877fe700 1 -- 172.21.15.189:0/1872316351 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f687406adb0 msgr2=0x7f687406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:41.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.459+0000 7f6884ff9700 1 -- 172.21.15.189:0/1872316351 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f687808a3b0 con 0x7f68880cb870 2024-04-06T12:27:41.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.459+0000 7f68877fe700 1 --2- 172.21.15.189:0/1872316351 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f687406adb0 0x7f687406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:41.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.464+0000 7f6884ff9700 1 -- 172.21.15.189:0/1872316351 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6878055320 con 0x7f68880cb870 2024-04-06T12:27:41.490 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:41.493+0000 7fd720bf8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:27:41.509 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:41.512+0000 7fc1585b6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:27:41.569 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:41.572+0000 7fd720bf8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:27:41.592 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:41.595+0000 7fc1585b6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:27:41.641 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:41.644+0000 7fd720bf8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:27:41.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.660+0000 7f68877fe700 1 -- 172.21.15.189:0/1872316351 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f687406adb0 msgr2=0x7f687406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:41.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.660+0000 7f68877fe700 1 --2- 172.21.15.189:0/1872316351 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f687406adb0 0x7f687406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:41.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.667+0000 7f688e6eb700 1 -- 172.21.15.189:0/1872316351 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6888074990 con 0x7f68880cb870 2024-04-06T12:27:41.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.669+0000 7f6884ff9700 1 -- 172.21.15.189:0/1872316351 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16793 (secure 0 0 0) 0x7f6878058940 con 0x7f68880cb870 2024-04-06T12:27:41.666 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:41.666 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":85,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:27:39.087666+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:27:19.575213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-06T12:27:19.990714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/615870143":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/1180713047":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/2430686509":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/535246627":"2024-04-07T12:27:39.087614+0000","172.21.15.189:6832/47128":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/2829807377":"2024-04-07T12:27:39.087614+0000","172.21.15.189:6833/47128":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/4012548359":"2024-04-07T12:27:39.087614+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:27:41.667 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:41.671+0000 7fc1585b6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:27:41.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.671+0000 7f688e6eb700 1 -- 172.21.15.189:0/1872316351 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f687406adb0 msgr2=0x7f687406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:41.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.671+0000 7f688e6eb700 1 --2- 172.21.15.189:0/1872316351 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f687406adb0 0x7f687406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:41.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.671+0000 7f688e6eb700 1 -- 172.21.15.189:0/1872316351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68880cb870 msgr2=0x7f688815c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:41.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.671+0000 7f688e6eb700 1 --2- 172.21.15.189:0/1872316351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68880cb870 0x7f688815c640 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f6878014040 tx=0x7f6878004750 comp rx=0 tx=0).stop 2024-04-06T12:27:41.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.671+0000 7f688e6eb700 1 -- 172.21.15.189:0/1872316351 shutdown_connections 2024-04-06T12:27:41.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.671+0000 7f688e6eb700 1 --2- 172.21.15.189:0/1872316351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68880d11b0 0x7f688806b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:41.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.671+0000 7f688e6eb700 1 --2- 172.21.15.189:0/1872316351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68880cb870 0x7f688815c640 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:41.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.671+0000 7f688e6eb700 1 --2- 172.21.15.189:0/1872316351 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f687406adb0 0x7f687406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:41.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.671+0000 7f688e6eb700 1 --2- 172.21.15.189:0/1872316351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68880cfa00 0x7f68880c4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:41.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.671+0000 7f688e6eb700 1 -- 172.21.15.189:0/1872316351 >> 172.21.15.189:0/1872316351 conn(0x7f68880b2db0 msgr2=0x7f68880ca300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:41.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.671+0000 7f688e6eb700 1 -- 172.21.15.189:0/1872316351 shutdown_connections 2024-04-06T12:27:41.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.671+0000 7f688e6eb700 1 -- 172.21.15.189:0/1872316351 wait complete. 2024-04-06T12:27:41.681 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:27:19.575213+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:27:19.990714+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:27:41.682 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:27:41.709 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:41.712+0000 7fd720bf8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:27:41.745 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:41.748+0000 7fc1585b6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:27:41.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.814+0000 7f86c5d12700 1 Processor -- start 2024-04-06T12:27:41.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.814+0000 7f86c5d12700 1 -- start start 2024-04-06T12:27:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.815+0000 7f86c5d12700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f86c015d590 0x7f86c00b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.815+0000 7f86c5d12700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c00b5cc0 con 0x7f86c00d3890 2024-04-06T12:27:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.815+0000 7f86c5d12700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c00d1500 con 0x7f86c00b4ed0 2024-04-06T12:27:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.815+0000 7f86c5d12700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c00d17e0 con 0x7f86c015d590 2024-04-06T12:27:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.815+0000 7f86bf7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f86c015d590 0x7f86c00b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.815+0000 7f86bffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f86c00d3890 0x7f86c00b6c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44342/0 (socket says 172.21.15.189:44342) 2024-04-06T12:27:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.815+0000 7f86bffff700 1 -- 172.21.15.189:0/2133094545 learned_addr learned my addr 172.21.15.189:0/2133094545 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.815+0000 7f86bf7fe700 1 -- 172.21.15.189:0/2133094545 >> v1:172.21.15.192:6789/0 conn(0x7f86c00b4ed0 legacy=0x7f86c00b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.815+0000 7f86bf7fe700 1 -- 172.21.15.189:0/2133094545 >> v1:172.21.15.189:6789/0 conn(0x7f86c00d3890 legacy=0x7f86c00b6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.815+0000 7f86be7fc700 1 -- 172.21.15.189:0/2133094545 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746545087 0 0) 0x7f86c00d1500 con 0x7f86c00b4ed0 2024-04-06T12:27:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.816+0000 7f86bf7fe700 1 -- 172.21.15.189:0/2133094545 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c00d1ac0 con 0x7f86c015d590 2024-04-06T12:27:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.816+0000 7f86bf7fe700 1 --2- 172.21.15.189:0/2133094545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86c015d590 0x7f86c00b8d70 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f86b0009840 tx=0x7f86b0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=568cacee0129262c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.816+0000 7f86be7fc700 1 -- 172.21.15.189:0/2133094545 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b0004230 con 0x7f86c015d590 2024-04-06T12:27:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.816+0000 7f86be7fc700 1 -- 172.21.15.189:0/2133094545 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b0004390 con 0x7f86c015d590 2024-04-06T12:27:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.816+0000 7f86be7fc700 1 -- 172.21.15.189:0/2133094545 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b00036a0 con 0x7f86c015d590 2024-04-06T12:27:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.816+0000 7f86c5d12700 1 -- 172.21.15.189:0/2133094545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86c015d590 msgr2=0x7f86c00b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.816+0000 7f86c5d12700 1 --2- 172.21.15.189:0/2133094545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86c015d590 0x7f86c00b8d70 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f86b0009840 tx=0x7f86b0009b50 comp rx=0 tx=0).stop 2024-04-06T12:27:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.816+0000 7f86c5d12700 1 -- 172.21.15.189:0/2133094545 shutdown_connections 2024-04-06T12:27:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.816+0000 7f86c5d12700 1 --2- 172.21.15.189:0/2133094545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86c015d590 0x7f86c00b8d70 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.816+0000 7f86c5d12700 1 -- 172.21.15.189:0/2133094545 >> 172.21.15.189:0/2133094545 conn(0x7f86c006bbc0 msgr2=0x7f86c006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.817+0000 7f86c5d12700 1 -- 172.21.15.189:0/2133094545 shutdown_connections 2024-04-06T12:27:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.817+0000 7f86c5d12700 1 -- 172.21.15.189:0/2133094545 wait complete. 2024-04-06T12:27:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.818+0000 7f86c5d12700 1 Processor -- start 2024-04-06T12:27:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.818+0000 7f86c5d12700 1 -- start start 2024-04-06T12:27:41.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86c5d12700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f86c00b4ed0 0x7f86c00c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:41.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86c5d12700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86c00d3890 0x7f86c00cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:41.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86c5d12700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86c015d590 0x7f86c00b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:41.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86c5d12700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c00cc250 con 0x7f86c00d3890 2024-04-06T12:27:41.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86c5d12700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c00cc530 con 0x7f86c015d590 2024-04-06T12:27:41.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86c5d12700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c00cc810 con 0x7f86c00b4ed0 2024-04-06T12:27:41.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86beffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86c00d3890 0x7f86c00cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:41.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86beffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86c00d3890 0x7f86c00cb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48442/0 (socket says 172.21.15.189:48442) 2024-04-06T12:27:41.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86beffd700 1 -- 172.21.15.189:0/839965856 learned_addr learned my addr 172.21.15.189:0/839965856 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:41.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86bffff700 1 --2- 172.21.15.189:0/839965856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86c015d590 0x7f86c00b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:41.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86bf7fe700 1 --2- 172.21.15.189:0/839965856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f86c00b4ed0 0x7f86c00c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:41.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86beffd700 1 -- 172.21.15.189:0/839965856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f86c00b4ed0 msgr2=0x7f86c00c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:41.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86beffd700 1 --2- 172.21.15.189:0/839965856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f86c00b4ed0 0x7f86c00c8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:41.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86beffd700 1 -- 172.21.15.189:0/839965856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86c015d590 msgr2=0x7f86c00b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:41.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86beffd700 1 --2- 172.21.15.189:0/839965856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86c015d590 0x7f86c00b3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:41.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86beffd700 1 -- 172.21.15.189:0/839965856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b00094a0 con 0x7f86c00d3890 2024-04-06T12:27:41.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86beffd700 1 --2- 172.21.15.189:0/839965856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86c00d3890 0x7f86c00cb890 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f86a80011e0 tx=0x7f86a800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:41.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86b7fff700 1 -- 172.21.15.189:0/839965856 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a80054e0 con 0x7f86c00d3890 2024-04-06T12:27:41.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86b7fff700 1 -- 172.21.15.189:0/839965856 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a8003680 con 0x7f86c00d3890 2024-04-06T12:27:41.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86b7fff700 1 -- 172.21.15.189:0/839965856 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a8003a30 con 0x7f86c00d3890 2024-04-06T12:27:41.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86c5d12700 1 -- 172.21.15.189:0/839965856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c00ccaf0 con 0x7f86c00d3890 2024-04-06T12:27:41.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.819+0000 7f86c5d12700 1 -- 172.21.15.189:0/839965856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c00cd2f0 con 0x7f86c00d3890 2024-04-06T12:27:41.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.820+0000 7f86c5d12700 1 -- 172.21.15.189:0/839965856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c00cd010 con 0x7f86c00d3890 2024-04-06T12:27:41.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.821+0000 7f86b7fff700 1 -- 172.21.15.189:0/839965856 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 37) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f86a8003b90 con 0x7f86c00d3890 2024-04-06T12:27:41.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.821+0000 7f86b7fff700 1 --2- 172.21.15.189:0/839965856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86a006adb0 0x7f86a006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:41.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.821+0000 7f86bf7fe700 1 -- 172.21.15.189:0/839965856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86a006adb0 msgr2=0x7f86a006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:41.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.821+0000 7f86b7fff700 1 -- 172.21.15.189:0/839965856 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f86a8088b60 con 0x7f86c00d3890 2024-04-06T12:27:41.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.821+0000 7f86bf7fe700 1 --2- 172.21.15.189:0/839965856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86a006adb0 0x7f86a006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:41.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:41.826+0000 7f86b7fff700 1 -- 172.21.15.189:0/839965856 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f86c00cd010 con 0x7f86c00d3890 2024-04-06T12:27:41.884 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:41.886+0000 7fd720bf8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:27:41.931 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:41.934+0000 7fc1585b6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:27:42.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.022+0000 7f86bf7fe700 1 -- 172.21.15.189:0/839965856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86a006adb0 msgr2=0x7f86a006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:42.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.022+0000 7f86bf7fe700 1 --2- 172.21.15.189:0/839965856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86a006adb0 0x7f86a006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:42.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.028+0000 7f86c5d12700 1 -- 172.21.15.189:0/839965856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f86c00cd010 con 0x7f86c00d3890 2024-04-06T12:27:42.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.028+0000 7f86b7fff700 1 -- 172.21.15.189:0/839965856 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f86c00cd010 con 0x7f86c00d3890 2024-04-06T12:27:42.025 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:27:42.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.031+0000 7f86c5d12700 1 -- 172.21.15.189:0/839965856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86a006adb0 msgr2=0x7f86a006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:42.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.031+0000 7f86c5d12700 1 --2- 172.21.15.189:0/839965856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86a006adb0 0x7f86a006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:42.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.031+0000 7f86c5d12700 1 -- 172.21.15.189:0/839965856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86c00d3890 msgr2=0x7f86c00cb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:42.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.031+0000 7f86c5d12700 1 --2- 172.21.15.189:0/839965856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86c00d3890 0x7f86c00cb890 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f86a80011e0 tx=0x7f86a800ca30 comp rx=0 tx=0).stop 2024-04-06T12:27:42.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.031+0000 7f86c5d12700 1 -- 172.21.15.189:0/839965856 shutdown_connections 2024-04-06T12:27:42.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.031+0000 7f86c5d12700 1 --2- 172.21.15.189:0/839965856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86a006adb0 0x7f86a006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:42.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.031+0000 7f86c5d12700 1 --2- 172.21.15.189:0/839965856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86c015d590 0x7f86c00b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:42.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.031+0000 7f86c5d12700 1 --2- 172.21.15.189:0/839965856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f86c00b4ed0 0x7f86c00c8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:42.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.031+0000 7f86c5d12700 1 --2- 172.21.15.189:0/839965856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86c00d3890 0x7f86c00cb890 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:42.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.031+0000 7f86c5d12700 1 -- 172.21.15.189:0/839965856 >> 172.21.15.189:0/839965856 conn(0x7f86c006bbc0 msgr2=0x7f86c006bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:42.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.031+0000 7f86c5d12700 1 -- 172.21.15.189:0/839965856 shutdown_connections 2024-04-06T12:27:42.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.031+0000 7f86c5d12700 1 -- 172.21.15.189:0/839965856 wait complete. 2024-04-06T12:27:42.038 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:42.041+0000 7fd720bf8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:27:42.041 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:27:42.095 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:42.098+0000 7fc1585b6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:27:42.148 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:42.151+0000 7fd720bf8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:27:42.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.173+0000 7f239aeb5700 1 Processor -- start 2024-04-06T12:27:42.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.173+0000 7f239aeb5700 1 -- start start 2024-04-06T12:27:42.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.174+0000 7f239aeb5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23940b5380 0x7f23940d2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:42.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.174+0000 7f239aeb5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23940b5cc0 con 0x7f239415d590 2024-04-06T12:27:42.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.174+0000 7f239aeb5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23940b57f0 con 0x7f23940b4ed0 2024-04-06T12:27:42.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.174+0000 7f239aeb5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23940d1500 con 0x7f23940b5380 2024-04-06T12:27:42.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.174+0000 7f2398c51700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f239415d590 0x7f23940b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44358/0 (socket says 172.21.15.189:44358) 2024-04-06T12:27:42.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.174+0000 7f2398c51700 1 -- 172.21.15.189:0/313799831 learned_addr learned my addr 172.21.15.189:0/313799831 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:42.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.174+0000 7f2399452700 1 --2- 172.21.15.189:0/313799831 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23940b5380 0x7f23940d2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:42.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.175+0000 7f23937fe700 1 -- 172.21.15.189:0/313799831 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41396750 0 0) 0x7f23940b5cc0 con 0x7f239415d590 2024-04-06T12:27:42.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.175+0000 7f23937fe700 1 -- 172.21.15.189:0/313799831 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23840035b0 con 0x7f239415d590 2024-04-06T12:27:42.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.175+0000 7f23937fe700 1 -- 172.21.15.189:0/313799831 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466378111 0 0) 0x7f23940b57f0 con 0x7f23940b4ed0 2024-04-06T12:27:42.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.175+0000 7f23937fe700 1 -- 172.21.15.189:0/313799831 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23940b5cc0 con 0x7f23940b4ed0 2024-04-06T12:27:42.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.175+0000 7f2399452700 1 -- 172.21.15.189:0/313799831 >> v1:172.21.15.192:6789/0 conn(0x7f23940b4ed0 legacy=0x7f23940d2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:42.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.175+0000 7f2399452700 1 -- 172.21.15.189:0/313799831 >> v1:172.21.15.189:6789/0 conn(0x7f239415d590 legacy=0x7f23940b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:42.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.175+0000 7f2399452700 1 -- 172.21.15.189:0/313799831 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23940d17e0 con 0x7f23940b5380 2024-04-06T12:27:42.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.175+0000 7f2399452700 1 --2- 172.21.15.189:0/313799831 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23940b5380 0x7f23940d2f90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f238c009dc0 tx=0x7f238c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1ac25a5b309f9e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:42.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.175+0000 7f23937fe700 1 -- 172.21.15.189:0/313799831 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c00e070 con 0x7f23940b5380 2024-04-06T12:27:42.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.175+0000 7f23937fe700 1 -- 172.21.15.189:0/313799831 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238c00bb20 con 0x7f23940b5380 2024-04-06T12:27:42.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.175+0000 7f23937fe700 1 -- 172.21.15.189:0/313799831 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c00be90 con 0x7f23940b5380 2024-04-06T12:27:42.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.176+0000 7f239aeb5700 1 -- 172.21.15.189:0/313799831 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23940b5380 msgr2=0x7f23940d2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:42.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.176+0000 7f239aeb5700 1 --2- 172.21.15.189:0/313799831 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23940b5380 0x7f23940d2f90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f238c009dc0 tx=0x7f238c00b330 comp rx=0 tx=0).stop 2024-04-06T12:27:42.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.176+0000 7f239aeb5700 1 -- 172.21.15.189:0/313799831 shutdown_connections 2024-04-06T12:27:42.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.176+0000 7f239aeb5700 1 --2- 172.21.15.189:0/313799831 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23940b5380 0x7f23940d2f90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:42.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.176+0000 7f239aeb5700 1 -- 172.21.15.189:0/313799831 >> 172.21.15.189:0/313799831 conn(0x7f239406bbc0 msgr2=0x7f239406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:42.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.176+0000 7f239aeb5700 1 -- 172.21.15.189:0/313799831 shutdown_connections 2024-04-06T12:27:42.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.176+0000 7f239aeb5700 1 -- 172.21.15.189:0/313799831 wait complete. 2024-04-06T12:27:42.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.177+0000 7f239aeb5700 1 Processor -- start 2024-04-06T12:27:42.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.177+0000 7f239aeb5700 1 -- start start 2024-04-06T12:27:42.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.178+0000 7f239aeb5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23940b4ed0 0x7f2394142560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:42.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.178+0000 7f239aeb5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23940b5380 0x7f2394160310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:42.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.178+0000 7f239aeb5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f239415d590 0x7f239415bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:42.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.178+0000 7f239aeb5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2394159a70 con 0x7f239415d590 2024-04-06T12:27:42.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.178+0000 7f239aeb5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239415a870 con 0x7f23940b4ed0 2024-04-06T12:27:42.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.178+0000 7f239aeb5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2394159fe0 con 0x7f23940b5380 2024-04-06T12:27:42.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.179+0000 7f2399452700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f239415d590 0x7f239415bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:42.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.179+0000 7f2399452700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f239415d590 0x7f239415bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48454/0 (socket says 172.21.15.189:48454) 2024-04-06T12:27:42.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.179+0000 7f2399452700 1 -- 172.21.15.189:0/4006500445 learned_addr learned my addr 172.21.15.189:0/4006500445 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:42.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.179+0000 7f2398c51700 1 --2- 172.21.15.189:0/4006500445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23940b4ed0 0x7f2394142560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:42.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.179+0000 7f2393fff700 1 --2- 172.21.15.189:0/4006500445 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23940b5380 0x7f2394160310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:42.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.179+0000 7f2399452700 1 -- 172.21.15.189:0/4006500445 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23940b5380 msgr2=0x7f2394160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:42.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.179+0000 7f2399452700 1 --2- 172.21.15.189:0/4006500445 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23940b5380 0x7f2394160310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:42.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.179+0000 7f2399452700 1 -- 172.21.15.189:0/4006500445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23940b4ed0 msgr2=0x7f2394142560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:42.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.179+0000 7f2399452700 1 --2- 172.21.15.189:0/4006500445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23940b4ed0 0x7f2394142560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:42.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.179+0000 7f2399452700 1 -- 172.21.15.189:0/4006500445 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c009a70 con 0x7f239415d590 2024-04-06T12:27:42.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.179+0000 7f2399452700 1 --2- 172.21.15.189:0/4006500445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f239415d590 0x7f239415bb90 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f238c014040 tx=0x7f238c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:42.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.179+0000 7f23917fa700 1 -- 172.21.15.189:0/4006500445 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c0048c0 con 0x7f239415d590 2024-04-06T12:27:42.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.179+0000 7f23917fa700 1 -- 172.21.15.189:0/4006500445 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238c020070 con 0x7f239415d590 2024-04-06T12:27:42.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.179+0000 7f239aeb5700 1 -- 172.21.15.189:0/4006500445 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2394142aa0 con 0x7f239415d590 2024-04-06T12:27:42.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.179+0000 7f23917fa700 1 -- 172.21.15.189:0/4006500445 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c027680 con 0x7f239415d590 2024-04-06T12:27:42.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.179+0000 7f239aeb5700 1 -- 172.21.15.189:0/4006500445 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2394145320 con 0x7f239415d590 2024-04-06T12:27:42.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.180+0000 7f239aeb5700 1 -- 172.21.15.189:0/4006500445 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239415aa20 con 0x7f239415d590 2024-04-06T12:27:42.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.181+0000 7f23917fa700 1 -- 172.21.15.189:0/4006500445 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 37) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f238c012430 con 0x7f239415d590 2024-04-06T12:27:42.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.181+0000 7f23917fa700 1 --2- 172.21.15.189:0/4006500445 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f237806adb0 0x7f237806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:42.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.181+0000 7f2398c51700 1 -- 172.21.15.189:0/4006500445 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f237806adb0 msgr2=0x7f237806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:42.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.181+0000 7f23917fa700 1 -- 172.21.15.189:0/4006500445 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f238c022070 con 0x7f239415d590 2024-04-06T12:27:42.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.181+0000 7f2398c51700 1 --2- 172.21.15.189:0/4006500445 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f237806adb0 0x7f237806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:42.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.186+0000 7f23917fa700 1 -- 172.21.15.189:0/4006500445 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f239415aa20 con 0x7f239415d590 2024-04-06T12:27:42.214 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:42.217+0000 7fc1585b6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:27:42.230 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:42.233+0000 7fd720bf8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:27:42.299 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:42.302+0000 7fc1585b6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:27:42.303 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:42.306+0000 7fd720bf8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:27:42.377 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:42.380+0000 7fc1585b6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:27:42.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.382+0000 7f2398c51700 1 -- 172.21.15.189:0/4006500445 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f237806adb0 msgr2=0x7f237806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:42.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.382+0000 7f2398c51700 1 --2- 172.21.15.189:0/4006500445 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f237806adb0 0x7f237806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:42.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.388+0000 7f239aeb5700 1 -- 172.21.15.189:0/4006500445 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f239415aa20 con 0x7f239415d590 2024-04-06T12:27:42.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.388+0000 7f23917fa700 1 -- 172.21.15.189:0/4006500445 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f239415aa20 con 0x7f239415d590 2024-04-06T12:27:42.385 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:27:42.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.390+0000 7f239aeb5700 1 -- 172.21.15.189:0/4006500445 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f237806adb0 msgr2=0x7f237806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:42.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.390+0000 7f239aeb5700 1 --2- 172.21.15.189:0/4006500445 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f237806adb0 0x7f237806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:42.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.390+0000 7f239aeb5700 1 -- 172.21.15.189:0/4006500445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f239415d590 msgr2=0x7f239415bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:42.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.390+0000 7f239aeb5700 1 --2- 172.21.15.189:0/4006500445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f239415d590 0x7f239415bb90 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f238c014040 tx=0x7f238c004ad0 comp rx=0 tx=0).stop 2024-04-06T12:27:42.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.390+0000 7f239aeb5700 1 -- 172.21.15.189:0/4006500445 shutdown_connections 2024-04-06T12:27:42.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.390+0000 7f239aeb5700 1 --2- 172.21.15.189:0/4006500445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f239415d590 0x7f239415bb90 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:42.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.391+0000 7f239aeb5700 1 --2- 172.21.15.189:0/4006500445 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f237806adb0 0x7f237806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:42.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.391+0000 7f239aeb5700 1 --2- 172.21.15.189:0/4006500445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23940b4ed0 0x7f2394142560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:42.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.391+0000 7f239aeb5700 1 --2- 172.21.15.189:0/4006500445 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23940b5380 0x7f2394160310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:42.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.391+0000 7f239aeb5700 1 -- 172.21.15.189:0/4006500445 >> 172.21.15.189:0/4006500445 conn(0x7f239406bbc0 msgr2=0x7f23940d36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:42.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.391+0000 7f239aeb5700 1 -- 172.21.15.189:0/4006500445 shutdown_connections 2024-04-06T12:27:42.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.391+0000 7f239aeb5700 1 -- 172.21.15.189:0/4006500445 wait complete. 2024-04-06T12:27:42.400 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:27:42.503 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:42.506+0000 7fd720bf8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:27:42.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.532+0000 7f620ba33700 1 Processor -- start 2024-04-06T12:27:42.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.532+0000 7f620ba33700 1 -- start start 2024-04-06T12:27:42.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.533+0000 7f620ba33700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f62040b4ed0 0x7f62040b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:42.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.533+0000 7f620ba33700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62040d2880 0x7f62040d2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:42.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.533+0000 7f620ba33700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62040b5cc0 con 0x7f620415d590 2024-04-06T12:27:42.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.533+0000 7f620ba33700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62040b58c0 con 0x7f62040b4ed0 2024-04-06T12:27:42.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.533+0000 7f620ba33700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62040d1500 con 0x7f62040d2880 2024-04-06T12:27:42.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.533+0000 7f6208fce700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f62040b4ed0 0x7f62040b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:42.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.533+0000 7f6209fd0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62040d2880 0x7f62040d2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:42.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.533+0000 7f62097cf700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f620415d590 0x7f62040b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:34542/0 (socket says 172.21.15.189:34542) 2024-04-06T12:27:42.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.533+0000 7f6208fce700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f62040b4ed0 0x7f62040b5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:56114/0 (socket says 172.21.15.189:56114) 2024-04-06T12:27:42.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.533+0000 7f62097cf700 1 -- 172.21.15.189:0/4160159269 learned_addr learned my addr 172.21.15.189:0/4160159269 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:42.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.534+0000 7f6209fd0700 1 -- 172.21.15.189:0/4160159269 >> v2:172.21.15.192:3300/0 conn(0x7f62040b4ed0 msgr2=0x7f62040b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:42.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.534+0000 7f6209fd0700 1 --2- 172.21.15.189:0/4160159269 >> v2:172.21.15.192:3300/0 conn(0x7f62040b4ed0 0x7f62040b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:42.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.534+0000 7f6209fd0700 1 -- 172.21.15.189:0/4160159269 >> v1:172.21.15.192:6789/0 conn(0x7f620415d590 legacy=0x7f62040b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:42.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.534+0000 7f6209fd0700 1 -- 172.21.15.189:0/4160159269 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62040d17e0 con 0x7f62040d2880 2024-04-06T12:27:42.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.534+0000 7f6208fce700 1 --2- 172.21.15.189:0/4160159269 >> v2:172.21.15.192:3300/0 conn(0x7f62040b4ed0 0x7f62040b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:42.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.534+0000 7f6209fd0700 1 --2- 172.21.15.189:0/4160159269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62040d2880 0x7f62040d2cf0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f61fc009d90 tx=0x7f61fc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72ee1ca6917f39c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:42.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.534+0000 7f61fbfff700 1 -- 172.21.15.189:0/4160159269 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc00e070 con 0x7f62040d2880 2024-04-06T12:27:42.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.534+0000 7f61fbfff700 1 -- 172.21.15.189:0/4160159269 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61fc00bb20 con 0x7f62040d2880 2024-04-06T12:27:42.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.534+0000 7f61fbfff700 1 -- 172.21.15.189:0/4160159269 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc012430 con 0x7f62040d2880 2024-04-06T12:27:42.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.534+0000 7f620ba33700 1 -- 172.21.15.189:0/4160159269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62040d2880 msgr2=0x7f62040d2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:42.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.534+0000 7f620ba33700 1 --2- 172.21.15.189:0/4160159269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62040d2880 0x7f62040d2cf0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f61fc009d90 tx=0x7f61fc00b330 comp rx=0 tx=0).stop 2024-04-06T12:27:42.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.534+0000 7f620ba33700 1 -- 172.21.15.189:0/4160159269 shutdown_connections 2024-04-06T12:27:42.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.534+0000 7f620ba33700 1 --2- 172.21.15.189:0/4160159269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62040d2880 0x7f62040d2cf0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:42.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.534+0000 7f620ba33700 1 --2- 172.21.15.189:0/4160159269 >> v2:172.21.15.192:3300/0 conn(0x7f62040b4ed0 0x7f62040b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:42.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.534+0000 7f620ba33700 1 -- 172.21.15.189:0/4160159269 >> 172.21.15.189:0/4160159269 conn(0x7f620406bbc0 msgr2=0x7f620406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:42.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.535+0000 7f620ba33700 1 -- 172.21.15.189:0/4160159269 shutdown_connections 2024-04-06T12:27:42.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.535+0000 7f620ba33700 1 -- 172.21.15.189:0/4160159269 wait complete. 2024-04-06T12:27:42.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.536+0000 7f620ba33700 1 Processor -- start 2024-04-06T12:27:42.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.536+0000 7f620ba33700 1 -- start start 2024-04-06T12:27:42.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.537+0000 7f620ba33700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62040b4ed0 0x7f620415e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:42.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.537+0000 7f620ba33700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62040d2880 0x7f620415fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:42.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.537+0000 7f620ba33700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f620415d590 0x7f6204160340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:42.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.537+0000 7f620ba33700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62040cb410 con 0x7f62040b4ed0 2024-04-06T12:27:42.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.537+0000 7f620ba33700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6204160880 con 0x7f62040d2880 2024-04-06T12:27:42.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.537+0000 7f620ba33700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62040ca520 con 0x7f620415d590 2024-04-06T12:27:42.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.537+0000 7f62097cf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62040b4ed0 0x7f620415e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:42.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.537+0000 7f6208fce700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62040d2880 0x7f620415fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:42.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.537+0000 7f62097cf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62040b4ed0 0x7f620415e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48466/0 (socket says 172.21.15.189:48466) 2024-04-06T12:27:42.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.537+0000 7f62097cf700 1 -- 172.21.15.189:0/3845190448 learned_addr learned my addr 172.21.15.189:0/3845190448 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:42.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.537+0000 7f6209fd0700 1 --2- 172.21.15.189:0/3845190448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f620415d590 0x7f6204160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:42.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.537+0000 7f6208fce700 1 -- 172.21.15.189:0/3845190448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f620415d590 msgr2=0x7f6204160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:42.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.537+0000 7f6208fce700 1 --2- 172.21.15.189:0/3845190448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f620415d590 0x7f6204160340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:42.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.537+0000 7f6208fce700 1 -- 172.21.15.189:0/3845190448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62040b4ed0 msgr2=0x7f620415e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:42.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.537+0000 7f6208fce700 1 --2- 172.21.15.189:0/3845190448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62040b4ed0 0x7f620415e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:42.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.537+0000 7f6208fce700 1 -- 172.21.15.189:0/3845190448 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f40094a0 con 0x7f62040d2880 2024-04-06T12:27:42.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.538+0000 7f6208fce700 1 --2- 172.21.15.189:0/3845190448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62040d2880 0x7f620415fe00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f61f400e9b0 tx=0x7f61f400ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:42.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.538+0000 7f61f9ffb700 1 -- 172.21.15.189:0/3845190448 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f400ee50 con 0x7f62040d2880 2024-04-06T12:27:42.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.538+0000 7f61f9ffb700 1 -- 172.21.15.189:0/3845190448 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f400c850 con 0x7f62040d2880 2024-04-06T12:27:42.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.538+0000 7f61f9ffb700 1 -- 172.21.15.189:0/3845190448 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f400cbe0 con 0x7f62040d2880 2024-04-06T12:27:42.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.538+0000 7f620ba33700 1 -- 172.21.15.189:0/3845190448 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61fc009a70 con 0x7f62040d2880 2024-04-06T12:27:42.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.538+0000 7f620ba33700 1 -- 172.21.15.189:0/3845190448 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62040caae0 con 0x7f62040d2880 2024-04-06T12:27:42.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.539+0000 7f620ba33700 1 -- 172.21.15.189:0/3845190448 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6204074990 con 0x7f62040d2880 2024-04-06T12:27:42.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.540+0000 7f61f9ffb700 1 -- 172.21.15.189:0/3845190448 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 37) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f61f400cd40 con 0x7f62040d2880 2024-04-06T12:27:42.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.541+0000 7f61f9ffb700 1 --2- 172.21.15.189:0/3845190448 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f61e806adb0 0x7f61e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:42.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.541+0000 7f62097cf700 1 -- 172.21.15.189:0/3845190448 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f61e806adb0 msgr2=0x7f61e806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:42.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.541+0000 7f61f9ffb700 1 -- 172.21.15.189:0/3845190448 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f61f40897f0 con 0x7f62040d2880 2024-04-06T12:27:42.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.541+0000 7f62097cf700 1 --2- 172.21.15.189:0/3845190448 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f61e806adb0 0x7f61e806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:42.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.545+0000 7f61f9ffb700 1 -- 172.21.15.189:0/3845190448 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f61f4054580 con 0x7f62040d2880 2024-04-06T12:27:42.542 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:27:42.546 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:27:42.583 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:42.585+0000 7fd720bf8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:27:42.595 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:42.597+0000 7fc1585b6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:27:42.671 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:42.674+0000 7fc1585b6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:27:42.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.742+0000 7f62097cf700 1 -- 172.21.15.189:0/3845190448 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f61e806adb0 msgr2=0x7f61e806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:42.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.742+0000 7f62097cf700 1 --2- 172.21.15.189:0/3845190448 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f61e806adb0 0x7f61e806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:42.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.778+0000 7f620ba33700 1 -- 172.21.15.189:0/3845190448 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6204071c40 con 0x7f62040d2880 2024-04-06T12:27:43.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.778+0000 7f61f9ffb700 1 -- 172.21.15.189:0/3845190448 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7f61f4057ba0 con 0x7f62040d2880 2024-04-06T12:27:43.319 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:42.906+0000 7fd720bf8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:27:43.319 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:43.226+0000 7fd720bf8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:27:43.319 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:27:43.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.780+0000 7f620ba33700 1 -- 172.21.15.189:0/3845190448 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f61e806adb0 msgr2=0x7f61e806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:43.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.780+0000 7f620ba33700 1 --2- 172.21.15.189:0/3845190448 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f61e806adb0 0x7f61e806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:43.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.780+0000 7f620ba33700 1 -- 172.21.15.189:0/3845190448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62040d2880 msgr2=0x7f620415fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:43.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.780+0000 7f620ba33700 1 --2- 172.21.15.189:0/3845190448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62040d2880 0x7f620415fe00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f61f400e9b0 tx=0x7f61f400ecc0 comp rx=0 tx=0).stop 2024-04-06T12:27:43.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.780+0000 7f620ba33700 1 -- 172.21.15.189:0/3845190448 shutdown_connections 2024-04-06T12:27:43.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.780+0000 7f620ba33700 1 --2- 172.21.15.189:0/3845190448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f620415d590 0x7f6204160340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:43.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.780+0000 7f620ba33700 1 --2- 172.21.15.189:0/3845190448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62040b4ed0 0x7f620415e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:43.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.780+0000 7f620ba33700 1 --2- 172.21.15.189:0/3845190448 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f61e806adb0 0x7f61e806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:43.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.780+0000 7f620ba33700 1 --2- 172.21.15.189:0/3845190448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62040d2880 0x7f620415fe00 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:43.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.780+0000 7f620ba33700 1 -- 172.21.15.189:0/3845190448 >> 172.21.15.189:0/3845190448 conn(0x7f620406bbc0 msgr2=0x7f62040b8830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:43.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.780+0000 7f620ba33700 1 -- 172.21.15.189:0/3845190448 shutdown_connections 2024-04-06T12:27:43.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:42.781+0000 7f620ba33700 1 -- 172.21.15.189:0/3845190448 wait complete. 2024-04-06T12:27:43.322 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T12:27:43.323 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:43.000+0000 7fc1585b6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:27:43.323 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:43.325+0000 7fc1585b6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:27:43.382 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:43.385+0000 7fd720bf8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:27:43.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.452+0000 7fe780449700 1 Processor -- start 2024-04-06T12:27:43.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.452+0000 7fe780449700 1 -- start start 2024-04-06T12:27:43.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.453+0000 7fe780449700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe77815d7b0 0x7fe7780b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:43.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.453+0000 7fe780449700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe77815e500 0x7fe77806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:43.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.453+0000 7fe780449700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7780b3990 con 0x7fe77806a730 2024-04-06T12:27:43.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.453+0000 7fe780449700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe77815e950 con 0x7fe77815e500 2024-04-06T12:27:43.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.453+0000 7fe780449700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe77806b0f0 con 0x7fe77815d7b0 2024-04-06T12:27:43.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.453+0000 7fe77e9e6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe77815e500 0x7fe77806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:43.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.453+0000 7fe77e1e5700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe77815d7b0 0x7fe7780b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:43.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.453+0000 7fe77e9e6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe77815e500 0x7fe77806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48468/0 (socket says 172.21.15.189:48468) 2024-04-06T12:27:43.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.453+0000 7fe77e9e6700 1 -- 172.21.15.189:0/4012522847 learned_addr learned my addr 172.21.15.189:0/4012522847 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:43.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.453+0000 7fe77d9e4700 1 --1- 172.21.15.189:0/4012522847 >> v1:172.21.15.192:6789/0 conn(0x7fe77806a730 0x7fe77815ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:34556/0 (socket says 172.21.15.189:34556) 2024-04-06T12:27:43.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.454+0000 7fe77e9e6700 1 -- 172.21.15.189:0/4012522847 >> v2:172.21.15.192:3300/0 conn(0x7fe77815d7b0 msgr2=0x7fe7780b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:43.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.454+0000 7fe77e9e6700 1 --2- 172.21.15.189:0/4012522847 >> v2:172.21.15.192:3300/0 conn(0x7fe77815d7b0 0x7fe7780b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:43.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.454+0000 7fe77e9e6700 1 -- 172.21.15.189:0/4012522847 >> v1:172.21.15.192:6789/0 conn(0x7fe77806a730 legacy=0x7fe77815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:43.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.454+0000 7fe77e9e6700 1 -- 172.21.15.189:0/4012522847 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7780b3060 con 0x7fe77815e500 2024-04-06T12:27:43.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.454+0000 7fe77d1e3700 1 -- 172.21.15.189:0/4012522847 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126734311 0 0) 0x7fe7780b3990 con 0x7fe77806a730 2024-04-06T12:27:43.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.454+0000 7fe77e1e5700 1 --2- 172.21.15.189:0/4012522847 >> v2:172.21.15.192:3300/0 conn(0x7fe77815d7b0 0x7fe7780b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:43.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.454+0000 7fe77e9e6700 1 --2- 172.21.15.189:0/4012522847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77815e500 0x7fe77806abb0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fe7780b6200 tx=0x7fe77000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b8ea70856c1380 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:43.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.454+0000 7fe77d1e3700 1 -- 172.21.15.189:0/4012522847 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77000e070 con 0x7fe77815e500 2024-04-06T12:27:43.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.454+0000 7fe77d1e3700 1 -- 172.21.15.189:0/4012522847 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe77000b910 con 0x7fe77815e500 2024-04-06T12:27:43.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.454+0000 7fe77d1e3700 1 -- 172.21.15.189:0/4012522847 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77000bcc0 con 0x7fe77815e500 2024-04-06T12:27:43.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.455+0000 7fe780449700 1 -- 172.21.15.189:0/4012522847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77815e500 msgr2=0x7fe77806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:43.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.455+0000 7fe780449700 1 --2- 172.21.15.189:0/4012522847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77815e500 0x7fe77806abb0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fe7780b6200 tx=0x7fe77000b330 comp rx=0 tx=0).stop 2024-04-06T12:27:43.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.455+0000 7fe780449700 1 -- 172.21.15.189:0/4012522847 shutdown_connections 2024-04-06T12:27:43.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.455+0000 7fe780449700 1 --2- 172.21.15.189:0/4012522847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77815e500 0x7fe77806abb0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:43.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.455+0000 7fe780449700 1 --2- 172.21.15.189:0/4012522847 >> v2:172.21.15.192:3300/0 conn(0x7fe77815d7b0 0x7fe7780b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:43.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.455+0000 7fe780449700 1 -- 172.21.15.189:0/4012522847 >> 172.21.15.189:0/4012522847 conn(0x7fe7780d00b0 msgr2=0x7fe7780d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:43.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.455+0000 7fe780449700 1 -- 172.21.15.189:0/4012522847 shutdown_connections 2024-04-06T12:27:43.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.455+0000 7fe780449700 1 -- 172.21.15.189:0/4012522847 wait complete. 2024-04-06T12:27:43.456 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:43.457+0000 7fd720bf8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:27:43.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.456+0000 7fe780449700 1 Processor -- start 2024-04-06T12:27:43.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.456+0000 7fe780449700 1 -- start start 2024-04-06T12:27:43.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.457+0000 7fe780449700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77806a730 0x7fe77814fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:43.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.457+0000 7fe780449700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe77815d7b0 0x7fe7780cef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:43.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.457+0000 7fe780449700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe77815e500 0x7fe778158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:43.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.457+0000 7fe780449700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77815c7a0 con 0x7fe77806a730 2024-04-06T12:27:43.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.457+0000 7fe780449700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77815c4c0 con 0x7fe77815d7b0 2024-04-06T12:27:43.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.457+0000 7fe780449700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7781584d0 con 0x7fe77815e500 2024-04-06T12:27:43.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.457+0000 7fe77e1e5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77806a730 0x7fe77814fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:43.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.457+0000 7fe77e1e5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77806a730 0x7fe77814fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48472/0 (socket says 172.21.15.189:48472) 2024-04-06T12:27:43.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.457+0000 7fe77e1e5700 1 -- 172.21.15.189:0/3339558591 learned_addr learned my addr 172.21.15.189:0/3339558591 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:43.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.457+0000 7fe77d9e4700 1 --2- 172.21.15.189:0/3339558591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe77815d7b0 0x7fe7780cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:43.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.457+0000 7fe77e9e6700 1 --2- 172.21.15.189:0/3339558591 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe77815e500 0x7fe778158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:43.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.457+0000 7fe77e1e5700 1 -- 172.21.15.189:0/3339558591 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe77815e500 msgr2=0x7fe778158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:43.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.457+0000 7fe77e1e5700 1 --2- 172.21.15.189:0/3339558591 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe77815e500 0x7fe778158fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:43.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.457+0000 7fe77e1e5700 1 -- 172.21.15.189:0/3339558591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe77815d7b0 msgr2=0x7fe7780cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:43.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.457+0000 7fe77e1e5700 1 --2- 172.21.15.189:0/3339558591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe77815d7b0 0x7fe7780cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:43.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.457+0000 7fe77e1e5700 1 -- 172.21.15.189:0/3339558591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7740094a0 con 0x7fe77806a730 2024-04-06T12:27:43.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.457+0000 7fe77e1e5700 1 --2- 172.21.15.189:0/3339558591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77806a730 0x7fe77814fd60 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fe77400ebe0 tx=0x7fe77400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:43.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.458+0000 7fe76effd700 1 -- 172.21.15.189:0/3339558591 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77400c850 con 0x7fe77806a730 2024-04-06T12:27:43.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.458+0000 7fe76effd700 1 -- 172.21.15.189:0/3339558591 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe77400c9b0 con 0x7fe77806a730 2024-04-06T12:27:43.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.458+0000 7fe76effd700 1 -- 172.21.15.189:0/3339558591 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77400cd40 con 0x7fe77806a730 2024-04-06T12:27:43.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.458+0000 7fe780449700 1 -- 172.21.15.189:0/3339558591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe770009b30 con 0x7fe77806a730 2024-04-06T12:27:43.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.458+0000 7fe780449700 1 -- 172.21.15.189:0/3339558591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe778158a90 con 0x7fe77806a730 2024-04-06T12:27:43.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.459+0000 7fe780449700 1 -- 172.21.15.189:0/3339558591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe778158e00 con 0x7fe77806a730 2024-04-06T12:27:43.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.459+0000 7fe76effd700 1 -- 172.21.15.189:0/3339558591 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 37) v1 ==== 89248+0+0 (secure 0 0 0) 0x7fe774013470 con 0x7fe77806a730 2024-04-06T12:27:43.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.460+0000 7fe76effd700 1 --2- 172.21.15.189:0/3339558591 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe75c06adb0 0x7fe75c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:43.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.460+0000 7fe77d9e4700 1 -- 172.21.15.189:0/3339558591 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe75c06adb0 msgr2=0x7fe75c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:43.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.460+0000 7fe76effd700 1 -- 172.21.15.189:0/3339558591 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fe774089780 con 0x7fe77806a730 2024-04-06T12:27:43.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.460+0000 7fe77d9e4700 1 --2- 172.21.15.189:0/3339558591 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe75c06adb0 0x7fe75c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:43.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.464+0000 7fe76effd700 1 -- 172.21.15.189:0/3339558591 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe774054640 con 0x7fe77806a730 2024-04-06T12:27:43.492 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:43.495+0000 7fc1585b6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:27:43.566 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:43.569+0000 7fc1585b6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:27:43.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.661+0000 7fe77d9e4700 1 -- 172.21.15.189:0/3339558591 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe75c06adb0 msgr2=0x7fe75c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:43.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.661+0000 7fe77d9e4700 1 --2- 172.21.15.189:0/3339558591 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe75c06adb0 0x7fe75c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:43.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.679+0000 7fe780449700 1 -- 172.21.15.189:0/3339558591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe778158330 con 0x7fe77806a730 2024-04-06T12:27:43.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.680+0000 7fe76effd700 1 -- 172.21.15.189:0/3339558591 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v85) v1 ==== 80+0+466 (secure 0 0 0) 0x7fe778158330 con 0x7fe77806a730 2024-04-06T12:27:43.677 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/615870143 2024-04-07T12:27:39.087614+0000 2024-04-06T12:27:43.677 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1180713047 2024-04-07T12:27:39.087614+0000 2024-04-06T12:27:43.677 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2430686509 2024-04-07T12:27:39.087614+0000 2024-04-06T12:27:43.677 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/535246627 2024-04-07T12:27:39.087614+0000 2024-04-06T12:27:43.678 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6832/47128 2024-04-07T12:27:39.087614+0000 2024-04-06T12:27:43.678 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2829807377 2024-04-07T12:27:39.087614+0000 2024-04-06T12:27:43.678 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6833/47128 2024-04-07T12:27:39.087614+0000 2024-04-06T12:27:43.678 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/4012548359 2024-04-07T12:27:39.087614+0000 2024-04-06T12:27:43.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.682+0000 7fe780449700 1 -- 172.21.15.189:0/3339558591 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe75c06adb0 msgr2=0x7fe75c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:43.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.682+0000 7fe780449700 1 --2- 172.21.15.189:0/3339558591 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe75c06adb0 0x7fe75c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:43.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.682+0000 7fe780449700 1 -- 172.21.15.189:0/3339558591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77806a730 msgr2=0x7fe77814fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:43.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.682+0000 7fe780449700 1 --2- 172.21.15.189:0/3339558591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77806a730 0x7fe77814fd60 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fe77400ebe0 tx=0x7fe77400eef0 comp rx=0 tx=0).stop 2024-04-06T12:27:43.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.682+0000 7fe780449700 1 -- 172.21.15.189:0/3339558591 shutdown_connections 2024-04-06T12:27:43.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.682+0000 7fe780449700 1 --2- 172.21.15.189:0/3339558591 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe77815e500 0x7fe778158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:43.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.682+0000 7fe780449700 1 --2- 172.21.15.189:0/3339558591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe77806a730 0x7fe77814fd60 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:43.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.682+0000 7fe780449700 1 --2- 172.21.15.189:0/3339558591 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe75c06adb0 0x7fe75c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:43.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.682+0000 7fe780449700 1 --2- 172.21.15.189:0/3339558591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe77815d7b0 0x7fe7780cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:43.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.682+0000 7fe780449700 1 -- 172.21.15.189:0/3339558591 >> 172.21.15.189:0/3339558591 conn(0x7fe7780d00b0 msgr2=0x7fe778150580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:43.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.682+0000 7fe780449700 1 -- 172.21.15.189:0/3339558591 shutdown_connections 2024-04-06T12:27:43.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:43.682+0000 7fe780449700 1 -- 172.21.15.189:0/3339558591 wait complete. 2024-04-06T12:27:43.680 INFO:teuthology.orchestra.run.smithi189.stderr:listed 8 entries 2024-04-06T12:27:43.692 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-06T12:27:43.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:43.692 DEBUG:teuthology.orchestra.run.smithi189:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-06T12:27:43.732 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T12:27:43.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:27:43.732 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-06T12:27:43.790 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:27:43.791 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:27:43.808 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:27:43.808 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:27:43.808 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:27:43.850 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:27:43.850 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:27:43.850 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:27:43.867 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:27:43.867 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:27:44.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.016+0000 7f04d8fcc700 1 Processor -- start 2024-04-06T12:27:44.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.017+0000 7f04d8fcc700 1 -- start start 2024-04-06T12:27:44.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.017+0000 7f04d8fcc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04d40d11b0 0x7f04d40d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:44.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.017+0000 7f04d8fcc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f04d40cb870 0x7f04d415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:44.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.018+0000 7f04d8fcc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f04d40cfa00 0x7f04d40cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:44.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.018+0000 7f04d8fcc700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04d40ccdc0 con 0x7f04d40cfa00 2024-04-06T12:27:44.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.018+0000 7f04d8fcc700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04d40d0b50 con 0x7f04d40cb870 2024-04-06T12:27:44.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.018+0000 7f04d8fcc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d40d1ad0 con 0x7f04d40d11b0 2024-04-06T12:27:44.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.018+0000 7f04d2d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f04d40cfa00 0x7f04d40cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:44.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.018+0000 7f04d2d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f04d40cfa00 0x7f04d40cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48488/0 (socket says 172.21.15.189:48488) 2024-04-06T12:27:44.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.018+0000 7f04d2d9d700 1 -- 172.21.15.189:0/2799521528 learned_addr learned my addr 172.21.15.189:0/2799521528 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:44.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.018+0000 7f04d259c700 1 --2- 172.21.15.189:0/2799521528 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04d40d11b0 0x7f04d40d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:44.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.018+0000 7f04d1d9b700 1 --2- 172.21.15.189:0/2799521528 >> v2:172.21.15.192:3300/0 conn(0x7f04d40cb870 0x7f04d415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:44.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.018+0000 7f04d2d9d700 1 -- 172.21.15.189:0/2799521528 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04d40d11b0 msgr2=0x7f04d40d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:44.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.018+0000 7f04d2d9d700 1 --2- 172.21.15.189:0/2799521528 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04d40d11b0 0x7f04d40d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.018+0000 7f04d2d9d700 1 -- 172.21.15.189:0/2799521528 >> v2:172.21.15.192:3300/0 conn(0x7f04d40cb870 msgr2=0x7f04d415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:44.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.018+0000 7f04d2d9d700 1 --2- 172.21.15.189:0/2799521528 >> v2:172.21.15.192:3300/0 conn(0x7f04d40cb870 0x7f04d415c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.018+0000 7f04d2d9d700 1 -- 172.21.15.189:0/2799521528 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d40cbd80 con 0x7f04d40cfa00 2024-04-06T12:27:44.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.018+0000 7f04d1d9b700 1 --2- 172.21.15.189:0/2799521528 >> v2:172.21.15.192:3300/0 conn(0x7f04d40cb870 0x7f04d415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:44.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.018+0000 7f04d2d9d700 1 --2- 172.21.15.189:0/2799521528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04d40cfa00 0x7f04d40cd560 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f04d40ced30 tx=0x7f04cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33faa4ea48d863e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:44.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.018+0000 7f04d159a700 1 -- 172.21.15.189:0/2799521528 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc00e070 con 0x7f04d40cfa00 2024-04-06T12:27:44.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.018+0000 7f04d159a700 1 -- 172.21.15.189:0/2799521528 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04cc00b910 con 0x7f04d40cfa00 2024-04-06T12:27:44.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.018+0000 7f04d159a700 1 -- 172.21.15.189:0/2799521528 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc00bcc0 con 0x7f04d40cfa00 2024-04-06T12:27:44.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.019+0000 7f04d8fcc700 1 -- 172.21.15.189:0/2799521528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04d40cfa00 msgr2=0x7f04d40cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:44.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.019+0000 7f04d8fcc700 1 --2- 172.21.15.189:0/2799521528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04d40cfa00 0x7f04d40cd560 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f04d40ced30 tx=0x7f04cc00b330 comp rx=0 tx=0).stop 2024-04-06T12:27:44.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.019+0000 7f04d8fcc700 1 -- 172.21.15.189:0/2799521528 shutdown_connections 2024-04-06T12:27:44.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.019+0000 7f04d8fcc700 1 --2- 172.21.15.189:0/2799521528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04d40cfa00 0x7f04d40cd560 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.019+0000 7f04d8fcc700 1 --2- 172.21.15.189:0/2799521528 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04d40d11b0 0x7f04d40d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.019+0000 7f04d8fcc700 1 --2- 172.21.15.189:0/2799521528 >> v2:172.21.15.192:3300/0 conn(0x7f04d40cb870 0x7f04d415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.019+0000 7f04d8fcc700 1 -- 172.21.15.189:0/2799521528 >> 172.21.15.189:0/2799521528 conn(0x7f04d40b2db0 msgr2=0x7f04d40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:44.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.019+0000 7f04d8fcc700 1 -- 172.21.15.189:0/2799521528 shutdown_connections 2024-04-06T12:27:44.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.019+0000 7f04d8fcc700 1 -- 172.21.15.189:0/2799521528 wait complete. 2024-04-06T12:27:44.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.020+0000 7f04d8fcc700 1 Processor -- start 2024-04-06T12:27:44.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.020+0000 7f04d8fcc700 1 -- start start 2024-04-06T12:27:44.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.021+0000 7f04d8fcc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04d40cb870 0x7f04d4154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:44.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.021+0000 7f04d8fcc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04d40cfa00 0x7f04d4155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:44.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.021+0000 7f04d8fcc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04d40d11b0 0x7f04d415c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:44.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.021+0000 7f04d8fcc700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d4155a70 con 0x7f04d40cb870 2024-04-06T12:27:44.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.021+0000 7f04d8fcc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d406c2f0 con 0x7f04d40d11b0 2024-04-06T12:27:44.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.021+0000 7f04d8fcc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d406c590 con 0x7f04d40cfa00 2024-04-06T12:27:44.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.021+0000 7f04d259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04d40cb870 0x7f04d4154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:44.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.021+0000 7f04d259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04d40cb870 0x7f04d4154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48492/0 (socket says 172.21.15.189:48492) 2024-04-06T12:27:44.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.021+0000 7f04d2d9d700 1 --2- 172.21.15.189:0/4057790119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04d40d11b0 0x7f04d415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:44.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.021+0000 7f04d259c700 1 -- 172.21.15.189:0/4057790119 learned_addr learned my addr 172.21.15.189:0/4057790119 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:44.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.021+0000 7f04d1d9b700 1 --2- 172.21.15.189:0/4057790119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04d40cfa00 0x7f04d4155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:44.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.022+0000 7f04d259c700 1 -- 172.21.15.189:0/4057790119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04d40cfa00 msgr2=0x7f04d4155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:44.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.022+0000 7f04d259c700 1 --2- 172.21.15.189:0/4057790119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04d40cfa00 0x7f04d4155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.022+0000 7f04d259c700 1 -- 172.21.15.189:0/4057790119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04d40d11b0 msgr2=0x7f04d415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:44.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.022+0000 7f04d259c700 1 --2- 172.21.15.189:0/4057790119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04d40d11b0 0x7f04d415c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.022+0000 7f04d259c700 1 -- 172.21.15.189:0/4057790119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04cc009b30 con 0x7f04d40cb870 2024-04-06T12:27:44.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.022+0000 7f04d1d9b700 1 --2- 172.21.15.189:0/4057790119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04d40cfa00 0x7f04d4155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:44.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.022+0000 7f04d2d9d700 1 --2- 172.21.15.189:0/4057790119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04d40d11b0 0x7f04d415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:44.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.022+0000 7f04d259c700 1 --2- 172.21.15.189:0/4057790119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04d40cb870 0x7f04d4154ff0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f04c400e9b0 tx=0x7f04c400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:44.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.022+0000 7f04c2ffd700 1 -- 172.21.15.189:0/4057790119 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c4009710 con 0x7f04d40cb870 2024-04-06T12:27:44.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.022+0000 7f04c2ffd700 1 -- 172.21.15.189:0/4057790119 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c400ee50 con 0x7f04d40cb870 2024-04-06T12:27:44.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.022+0000 7f04d8fcc700 1 -- 172.21.15.189:0/4057790119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d406c870 con 0x7f04d40cb870 2024-04-06T12:27:44.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.022+0000 7f04c2ffd700 1 -- 172.21.15.189:0/4057790119 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c400ca50 con 0x7f04d40cb870 2024-04-06T12:27:44.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.022+0000 7f04d8fcc700 1 -- 172.21.15.189:0/4057790119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d406ce30 con 0x7f04d40cb870 2024-04-06T12:27:44.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.023+0000 7f04d8fcc700 1 -- 172.21.15.189:0/4057790119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d406d1a0 con 0x7f04d40cb870 2024-04-06T12:27:44.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.024+0000 7f04c2ffd700 1 -- 172.21.15.189:0/4057790119 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 37) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f04c400cbb0 con 0x7f04d40cb870 2024-04-06T12:27:44.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.024+0000 7f04c2ffd700 1 --2- 172.21.15.189:0/4057790119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f04b406adb0 0x7f04b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:44.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.024+0000 7f04d1d9b700 1 -- 172.21.15.189:0/4057790119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f04b406adb0 msgr2=0x7f04b406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:44.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.024+0000 7f04c2ffd700 1 -- 172.21.15.189:0/4057790119 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f04c4089750 con 0x7f04d40cb870 2024-04-06T12:27:44.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.024+0000 7f04d1d9b700 1 --2- 172.21.15.189:0/4057790119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f04b406adb0 0x7f04b406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:44.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.028+0000 7f04c2ffd700 1 -- 172.21.15.189:0/4057790119 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f04c4054580 con 0x7f04d40cb870 2024-04-06T12:27:44.076 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:44.079+0000 7fd720bf8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:27:44.156 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:44.158+0000 7fd720bf8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:27:44.220 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:44.223+0000 7fc1585b6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:27:44.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.225+0000 7f04d1d9b700 1 -- 172.21.15.189:0/4057790119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f04b406adb0 msgr2=0x7f04b406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:44.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.225+0000 7f04d1d9b700 1 --2- 172.21.15.189:0/4057790119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f04b406adb0 0x7f04b406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:44.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.278+0000 7f04d8fcc700 1 -- 172.21.15.189:0/4057790119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04d40670c0 con 0x7f04d40cb870 2024-04-06T12:27:44.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.278+0000 7f04c2ffd700 1 -- 172.21.15.189:0/4057790119 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4747 (secure 0 0 0) 0x7f04c400ce60 con 0x7f04d40cb870 2024-04-06T12:27:44.275 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:44.275 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5762,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/2827078766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2827078766},{"type":"v1","addr":"172.21.15.192:6833","nonce":2827078766}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5765,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1558111999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1558111999},{"type":"v1","addr":"172.21.15.192:6835","nonce":1558111999}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3448511259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3448511259},{"type":"v1","addr":"172.21.15.189:6837","nonce":3448511259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:27:21.031012+0000","modified":"2024-04-06T12:27:24.038778+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/1085319880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1085319880},{"type":"v1","addr":"172.21.15.189:6836","nonce":1085319880}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-06T12:27:44.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.280+0000 7f04d8fcc700 1 -- 172.21.15.189:0/4057790119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f04b406adb0 msgr2=0x7f04b406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:44.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.280+0000 7f04d8fcc700 1 --2- 172.21.15.189:0/4057790119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f04b406adb0 0x7f04b406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.280+0000 7f04d8fcc700 1 -- 172.21.15.189:0/4057790119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04d40cb870 msgr2=0x7f04d4154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:44.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.280+0000 7f04d8fcc700 1 --2- 172.21.15.189:0/4057790119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04d40cb870 0x7f04d4154ff0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f04c400e9b0 tx=0x7f04c400ecc0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.280+0000 7f04d8fcc700 1 -- 172.21.15.189:0/4057790119 shutdown_connections 2024-04-06T12:27:44.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.280+0000 7f04d8fcc700 1 --2- 172.21.15.189:0/4057790119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f04b406adb0 0x7f04b406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.280+0000 7f04d8fcc700 1 --2- 172.21.15.189:0/4057790119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04d40d11b0 0x7f04d415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.280+0000 7f04d8fcc700 1 --2- 172.21.15.189:0/4057790119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04d40cb870 0x7f04d4154ff0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.280+0000 7f04d8fcc700 1 --2- 172.21.15.189:0/4057790119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04d40cfa00 0x7f04d4155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.280+0000 7f04d8fcc700 1 -- 172.21.15.189:0/4057790119 >> 172.21.15.189:0/4057790119 conn(0x7f04d40b2db0 msgr2=0x7f04d415a550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:44.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.280+0000 7f04d8fcc700 1 -- 172.21.15.189:0/4057790119 shutdown_connections 2024-04-06T12:27:44.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.280+0000 7f04d8fcc700 1 -- 172.21.15.189:0/4057790119 wait complete. 2024-04-06T12:27:44.279 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:44.282+0000 7fd720bf8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:27:44.279 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 31 2024-04-06T12:27:44.290 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:27:44.302 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:44.305+0000 7fc1585b6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:27:44.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.417+0000 7f6cac6c9700 1 Processor -- start 2024-04-06T12:27:44.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.417+0000 7f6cac6c9700 1 -- start start 2024-04-06T12:27:44.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.418+0000 7f6cac6c9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6ca415a550 0x7f6ca40cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:44.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.418+0000 7f6cac6c9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca40cf900 0x7f6ca40b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:44.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.418+0000 7f6cac6c9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca40cdbe0 con 0x7f6ca40b4ba0 2024-04-06T12:27:44.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.418+0000 7f6cac6c9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca40b5010 con 0x7f6ca415a550 2024-04-06T12:27:44.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.418+0000 7f6cac6c9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca415a940 con 0x7f6ca40cf900 2024-04-06T12:27:44.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.418+0000 7f6ca9c64700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca40cf900 0x7f6ca40b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:44.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.418+0000 7f6caa465700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6ca415a550 0x7f6ca40cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:44.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.418+0000 7f6ca9c64700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca40cf900 0x7f6ca40b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:40750/0 (socket says 172.21.15.189:40750) 2024-04-06T12:27:44.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.418+0000 7f6ca9c64700 1 -- 172.21.15.189:0/1781351730 learned_addr learned my addr 172.21.15.189:0/1781351730 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:44.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.419+0000 7f6caa465700 1 -- 172.21.15.189:0/1781351730 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca40cf900 msgr2=0x7f6ca40b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:44.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.419+0000 7f6caa465700 1 --2- 172.21.15.189:0/1781351730 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca40cf900 0x7f6ca40b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.419+0000 7f6caa465700 1 -- 172.21.15.189:0/1781351730 >> v1:172.21.15.192:6789/0 conn(0x7f6ca40b4ba0 legacy=0x7f6ca406eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:44.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.419+0000 7f6caa465700 1 -- 172.21.15.189:0/1781351730 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca415ac00 con 0x7f6ca415a550 2024-04-06T12:27:44.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.419+0000 7f6ca9463700 1 -- 172.21.15.189:0/1781351730 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224279639 0 0) 0x7f6ca40cdbe0 con 0x7f6ca40b4ba0 2024-04-06T12:27:44.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.419+0000 7f6ca9c64700 1 --2- 172.21.15.189:0/1781351730 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca40cf900 0x7f6ca40b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:44.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.419+0000 7f6caa465700 1 --2- 172.21.15.189:0/1781351730 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca415a550 0x7f6ca40cf3c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009840 tx=0x7f6ca0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32b59cc617b95a8c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:44.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.419+0000 7f6ca9463700 1 -- 172.21.15.189:0/1781351730 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0011830 con 0x7f6ca415a550 2024-04-06T12:27:44.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.419+0000 7f6ca9463700 1 -- 172.21.15.189:0/1781351730 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca0011990 con 0x7f6ca415a550 2024-04-06T12:27:44.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.419+0000 7f6ca9463700 1 -- 172.21.15.189:0/1781351730 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0011d20 con 0x7f6ca415a550 2024-04-06T12:27:44.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.420+0000 7f6cac6c9700 1 -- 172.21.15.189:0/1781351730 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca415a550 msgr2=0x7f6ca40cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:44.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.420+0000 7f6cac6c9700 1 --2- 172.21.15.189:0/1781351730 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca415a550 0x7f6ca40cf3c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009840 tx=0x7f6ca0009b50 comp rx=0 tx=0).stop 2024-04-06T12:27:44.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.420+0000 7f6cac6c9700 1 -- 172.21.15.189:0/1781351730 shutdown_connections 2024-04-06T12:27:44.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.420+0000 7f6cac6c9700 1 --2- 172.21.15.189:0/1781351730 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca415a550 0x7f6ca40cf3c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.420+0000 7f6cac6c9700 1 --2- 172.21.15.189:0/1781351730 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca40cf900 0x7f6ca40b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.420+0000 7f6cac6c9700 1 -- 172.21.15.189:0/1781351730 >> 172.21.15.189:0/1781351730 conn(0x7f6ca40d00b0 msgr2=0x7f6ca40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:44.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.420+0000 7f6cac6c9700 1 -- 172.21.15.189:0/1781351730 shutdown_connections 2024-04-06T12:27:44.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.420+0000 7f6cac6c9700 1 -- 172.21.15.189:0/1781351730 wait complete. 2024-04-06T12:27:44.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.421+0000 7f6cac6c9700 1 Processor -- start 2024-04-06T12:27:44.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.421+0000 7f6cac6c9700 1 -- start start 2024-04-06T12:27:44.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.422+0000 7f6cac6c9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca40b4ba0 0x7f6ca41561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:44.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.422+0000 7f6cac6c9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca40cf900 0x7f6ca4150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:44.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.422+0000 7f6cac6c9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca4151360 0x7f6ca414ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:44.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.422+0000 7f6cac6c9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca4151b90 con 0x7f6ca40b4ba0 2024-04-06T12:27:44.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.422+0000 7f6cac6c9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca41529f0 con 0x7f6ca4151360 2024-04-06T12:27:44.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.422+0000 7f6cac6c9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca4152cd0 con 0x7f6ca40cf900 2024-04-06T12:27:44.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.422+0000 7f6caa465700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca40b4ba0 0x7f6ca41561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:44.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.422+0000 7f6caa465700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca40b4ba0 0x7f6ca41561c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48506/0 (socket says 172.21.15.189:48506) 2024-04-06T12:27:44.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.422+0000 7f6caa465700 1 -- 172.21.15.189:0/763855444 learned_addr learned my addr 172.21.15.189:0/763855444 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:44.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.422+0000 7f6ca9c64700 1 --2- 172.21.15.189:0/763855444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca40cf900 0x7f6ca4150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:44.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.422+0000 7f6caac66700 1 --2- 172.21.15.189:0/763855444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca4151360 0x7f6ca414ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:44.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.422+0000 7f6caa465700 1 -- 172.21.15.189:0/763855444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca40cf900 msgr2=0x7f6ca4150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:44.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.422+0000 7f6caa465700 1 --2- 172.21.15.189:0/763855444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca40cf900 0x7f6ca4150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.422+0000 7f6caa465700 1 -- 172.21.15.189:0/763855444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca4151360 msgr2=0x7f6ca414ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:44.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.422+0000 7f6caa465700 1 --2- 172.21.15.189:0/763855444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca4151360 0x7f6ca414ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.422+0000 7f6caa465700 1 -- 172.21.15.189:0/763855444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c94009550 con 0x7f6ca40b4ba0 2024-04-06T12:27:44.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.422+0000 7f6caa465700 1 --2- 172.21.15.189:0/763855444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca40b4ba0 0x7f6ca41561c0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009fd0 tx=0x7f6ca0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:44.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.422+0000 7f6c9affd700 1 -- 172.21.15.189:0/763855444 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0004550 con 0x7f6ca40b4ba0 2024-04-06T12:27:44.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.423+0000 7f6c9affd700 1 -- 172.21.15.189:0/763855444 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca001f070 con 0x7f6ca40b4ba0 2024-04-06T12:27:44.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.423+0000 7f6cac6c9700 1 -- 172.21.15.189:0/763855444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca00094a0 con 0x7f6ca40b4ba0 2024-04-06T12:27:44.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.423+0000 7f6c9affd700 1 -- 172.21.15.189:0/763855444 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca00038e0 con 0x7f6ca40b4ba0 2024-04-06T12:27:44.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.423+0000 7f6cac6c9700 1 -- 172.21.15.189:0/763855444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca4153290 con 0x7f6ca40b4ba0 2024-04-06T12:27:44.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.424+0000 7f6cac6c9700 1 -- 172.21.15.189:0/763855444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca4067bb0 con 0x7f6ca40b4ba0 2024-04-06T12:27:44.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.424+0000 7f6c9affd700 1 -- 172.21.15.189:0/763855444 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 37) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f6ca0003a40 con 0x7f6ca40b4ba0 2024-04-06T12:27:44.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.425+0000 7f6c9affd700 1 --2- 172.21.15.189:0/763855444 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c8806adb0 0x7f6c8806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:44.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.425+0000 7f6ca9c64700 1 -- 172.21.15.189:0/763855444 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c8806adb0 msgr2=0x7f6c8806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:44.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.425+0000 7f6c9affd700 1 -- 172.21.15.189:0/763855444 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f6ca0021070 con 0x7f6ca40b4ba0 2024-04-06T12:27:44.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.425+0000 7f6ca9c64700 1 --2- 172.21.15.189:0/763855444 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c8806adb0 0x7f6c8806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:44.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.429+0000 7f6c9affd700 1 -- 172.21.15.189:0/763855444 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6ca0056220 con 0x7f6ca40b4ba0 2024-04-06T12:27:44.437 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:44.440+0000 7fc1585b6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:27:44.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.626+0000 7f6ca9c64700 1 -- 172.21.15.189:0/763855444 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c8806adb0 msgr2=0x7f6c8806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:44.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.626+0000 7f6ca9c64700 1 --2- 172.21.15.189:0/763855444 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c8806adb0 0x7f6c8806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:44.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.631+0000 7f6cac6c9700 1 -- 172.21.15.189:0/763855444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ca4074990 con 0x7f6ca40b4ba0 2024-04-06T12:27:44.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.633+0000 7f6c9affd700 1 -- 172.21.15.189:0/763855444 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16793 (secure 0 0 0) 0x7f6ca0015090 con 0x7f6ca40b4ba0 2024-04-06T12:27:44.630 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:44.630 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":85,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:27:39.087666+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:27:19.575213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-06T12:27:19.990714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/615870143":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/1180713047":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/2430686509":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/535246627":"2024-04-07T12:27:39.087614+0000","172.21.15.189:6832/47128":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/2829807377":"2024-04-07T12:27:39.087614+0000","172.21.15.189:6833/47128":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/4012548359":"2024-04-07T12:27:39.087614+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:27:44.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.635+0000 7f6cac6c9700 1 -- 172.21.15.189:0/763855444 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c8806adb0 msgr2=0x7f6c8806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:44.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.635+0000 7f6cac6c9700 1 --2- 172.21.15.189:0/763855444 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c8806adb0 0x7f6c8806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.635+0000 7f6cac6c9700 1 -- 172.21.15.189:0/763855444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca40b4ba0 msgr2=0x7f6ca41561c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:44.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.635+0000 7f6cac6c9700 1 --2- 172.21.15.189:0/763855444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca40b4ba0 0x7f6ca41561c0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009fd0 tx=0x7f6ca0004750 comp rx=0 tx=0).stop 2024-04-06T12:27:44.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.635+0000 7f6cac6c9700 1 -- 172.21.15.189:0/763855444 shutdown_connections 2024-04-06T12:27:44.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.635+0000 7f6cac6c9700 1 --2- 172.21.15.189:0/763855444 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c8806adb0 0x7f6c8806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.635+0000 7f6cac6c9700 1 --2- 172.21.15.189:0/763855444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca4151360 0x7f6ca414ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.635+0000 7f6cac6c9700 1 --2- 172.21.15.189:0/763855444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca40b4ba0 0x7f6ca41561c0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.635+0000 7f6cac6c9700 1 --2- 172.21.15.189:0/763855444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca40cf900 0x7f6ca4150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.635+0000 7f6cac6c9700 1 -- 172.21.15.189:0/763855444 >> 172.21.15.189:0/763855444 conn(0x7f6ca40d00b0 msgr2=0x7f6ca406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:44.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.635+0000 7f6cac6c9700 1 -- 172.21.15.189:0/763855444 shutdown_connections 2024-04-06T12:27:44.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.635+0000 7f6cac6c9700 1 -- 172.21.15.189:0/763855444 wait complete. 2024-04-06T12:27:44.647 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:27:19.575213+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:27:19.990714+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:27:44.647 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:27:44.787 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:44.790+0000 7fd720bf8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:27:44.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.818+0000 7f7863c0e700 1 Processor -- start 2024-04-06T12:27:44.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.818+0000 7f7863c0e700 1 -- start start 2024-04-06T12:27:44.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.819+0000 7f7863c0e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f785c06ae40 0x7f785c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:44.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.819+0000 7f7863c0e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f785c15ddf0 0x7f785c15e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:44.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.819+0000 7f7863c0e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f785c0b3990 con 0x7f785c15d7b0 2024-04-06T12:27:44.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.819+0000 7f7863c0e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f785c0b3060 con 0x7f785c06ae40 2024-04-06T12:27:44.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.819+0000 7f7863c0e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785c0b3340 con 0x7f785c15ddf0 2024-04-06T12:27:44.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.819+0000 7f78619aa700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f785c15d7b0 0x7f785c06a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44370/0 (socket says 172.21.15.189:44370) 2024-04-06T12:27:44.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.819+0000 7f78619aa700 1 -- 172.21.15.189:0/2766032074 learned_addr learned my addr 172.21.15.189:0/2766032074 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:44.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.819+0000 7f78611a9700 1 --2- 172.21.15.189:0/2766032074 >> v2:172.21.15.192:3300/0 conn(0x7f785c06ae40 0x7f785c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:44.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.819+0000 7f78621ab700 1 --2- 172.21.15.189:0/2766032074 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f785c15ddf0 0x7f785c15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:44.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.819+0000 7f78609a8700 1 -- 172.21.15.189:0/2766032074 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454888651 0 0) 0x7f785c0b3990 con 0x7f785c15d7b0 2024-04-06T12:27:44.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.819+0000 7f78609a8700 1 -- 172.21.15.189:0/2766032074 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78480035b0 con 0x7f785c15d7b0 2024-04-06T12:27:44.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.819+0000 7f78621ab700 1 -- 172.21.15.189:0/2766032074 >> v2:172.21.15.192:3300/0 conn(0x7f785c06ae40 msgr2=0x7f785c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:44.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.819+0000 7f78621ab700 1 --2- 172.21.15.189:0/2766032074 >> v2:172.21.15.192:3300/0 conn(0x7f785c06ae40 0x7f785c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.819+0000 7f78621ab700 1 -- 172.21.15.189:0/2766032074 >> v1:172.21.15.189:6789/0 conn(0x7f785c15d7b0 legacy=0x7f785c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:44.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.819+0000 7f78621ab700 1 -- 172.21.15.189:0/2766032074 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785c0b3620 con 0x7f785c15ddf0 2024-04-06T12:27:44.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.819+0000 7f78611a9700 1 --2- 172.21.15.189:0/2766032074 >> v2:172.21.15.192:3300/0 conn(0x7f785c06ae40 0x7f785c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:44.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.820+0000 7f78621ab700 1 --2- 172.21.15.189:0/2766032074 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f785c15ddf0 0x7f785c15e280 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7854009d90 tx=0x7f785400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca95bc632eb27de8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:44.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.820+0000 7f78609a8700 1 -- 172.21.15.189:0/2766032074 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785400e070 con 0x7f785c15ddf0 2024-04-06T12:27:44.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.820+0000 7f78609a8700 1 -- 172.21.15.189:0/2766032074 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785400bb20 con 0x7f785c15ddf0 2024-04-06T12:27:44.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.820+0000 7f78609a8700 1 -- 172.21.15.189:0/2766032074 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7854012430 con 0x7f785c15ddf0 2024-04-06T12:27:44.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.820+0000 7f7863c0e700 1 -- 172.21.15.189:0/2766032074 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f785c15ddf0 msgr2=0x7f785c15e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:44.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.820+0000 7f7863c0e700 1 --2- 172.21.15.189:0/2766032074 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f785c15ddf0 0x7f785c15e280 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7854009d90 tx=0x7f785400b330 comp rx=0 tx=0).stop 2024-04-06T12:27:44.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.820+0000 7f7863c0e700 1 -- 172.21.15.189:0/2766032074 shutdown_connections 2024-04-06T12:27:44.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.820+0000 7f7863c0e700 1 --2- 172.21.15.189:0/2766032074 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f785c15ddf0 0x7f785c15e280 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.820+0000 7f7863c0e700 1 --2- 172.21.15.189:0/2766032074 >> v2:172.21.15.192:3300/0 conn(0x7f785c06ae40 0x7f785c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.820+0000 7f7863c0e700 1 -- 172.21.15.189:0/2766032074 >> 172.21.15.189:0/2766032074 conn(0x7f785c0d00b0 msgr2=0x7f785c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:44.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.820+0000 7f7863c0e700 1 -- 172.21.15.189:0/2766032074 shutdown_connections 2024-04-06T12:27:44.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.821+0000 7f7863c0e700 1 -- 172.21.15.189:0/2766032074 wait complete. 2024-04-06T12:27:44.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.822+0000 7f7863c0e700 1 Processor -- start 2024-04-06T12:27:44.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.822+0000 7f7863c0e700 1 -- start start 2024-04-06T12:27:44.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.822+0000 7f7863c0e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f785c06ae40 0x7f785c0aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:44.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.822+0000 7f7863c0e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f785c15d7b0 0x7f785c1f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:44.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.822+0000 7f7863c0e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f785c15ddf0 0x7f785c0cf470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:44.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.822+0000 7f7863c0e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c1f3710 con 0x7f785c06ae40 2024-04-06T12:27:44.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.822+0000 7f7863c0e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c0afb30 con 0x7f785c15d7b0 2024-04-06T12:27:44.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.822+0000 7f7863c0e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785c0af290 con 0x7f785c15ddf0 2024-04-06T12:27:44.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.823+0000 7f78619aa700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f785c06ae40 0x7f785c0aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:44.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.823+0000 7f78619aa700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f785c06ae40 0x7f785c0aeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48508/0 (socket says 172.21.15.189:48508) 2024-04-06T12:27:44.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.823+0000 7f78619aa700 1 -- 172.21.15.189:0/2597918395 learned_addr learned my addr 172.21.15.189:0/2597918395 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:44.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.823+0000 7f78621ab700 1 --2- 172.21.15.189:0/2597918395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f785c15ddf0 0x7f785c0cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:44.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.823+0000 7f78611a9700 1 --2- 172.21.15.189:0/2597918395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f785c15d7b0 0x7f785c1f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:44.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.823+0000 7f78621ab700 1 -- 172.21.15.189:0/2597918395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f785c15d7b0 msgr2=0x7f785c1f31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:44.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.823+0000 7f78621ab700 1 --2- 172.21.15.189:0/2597918395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f785c15d7b0 0x7f785c1f31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.823+0000 7f78621ab700 1 -- 172.21.15.189:0/2597918395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f785c06ae40 msgr2=0x7f785c0aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:44.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.823+0000 7f78621ab700 1 --2- 172.21.15.189:0/2597918395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f785c06ae40 0x7f785c0aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:44.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.823+0000 7f78621ab700 1 -- 172.21.15.189:0/2597918395 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784c0094a0 con 0x7f785c15ddf0 2024-04-06T12:27:44.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.823+0000 7f78619aa700 1 --2- 172.21.15.189:0/2597918395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f785c06ae40 0x7f785c0aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:44.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.823+0000 7f78611a9700 1 --2- 172.21.15.189:0/2597918395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f785c15d7b0 0x7f785c1f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:44.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.823+0000 7f78621ab700 1 --2- 172.21.15.189:0/2597918395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f785c15ddf0 0x7f785c0cf470 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7854016000 tx=0x7f7854004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:44.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.823+0000 7f78527fc700 1 -- 172.21.15.189:0/2597918395 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7854012430 con 0x7f785c15ddf0 2024-04-06T12:27:44.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.823+0000 7f78527fc700 1 -- 172.21.15.189:0/2597918395 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7854020070 con 0x7f785c15ddf0 2024-04-06T12:27:44.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.823+0000 7f7863c0e700 1 -- 172.21.15.189:0/2597918395 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7854009a70 con 0x7f785c15ddf0 2024-04-06T12:27:44.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.823+0000 7f78527fc700 1 -- 172.21.15.189:0/2597918395 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78540098f0 con 0x7f785c15ddf0 2024-04-06T12:27:44.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.823+0000 7f7863c0e700 1 -- 172.21.15.189:0/2597918395 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f785c0af850 con 0x7f785c15ddf0 2024-04-06T12:27:44.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.825+0000 7f7863c0e700 1 -- 172.21.15.189:0/2597918395 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f785c067bb0 con 0x7f785c15ddf0 2024-04-06T12:27:44.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.825+0000 7f78527fc700 1 -- 172.21.15.189:0/2597918395 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 37) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f7854004040 con 0x7f785c15ddf0 2024-04-06T12:27:44.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.825+0000 7f78527fc700 1 --2- 172.21.15.189:0/2597918395 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f784006adb0 0x7f784006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:44.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.825+0000 7f78619aa700 1 -- 172.21.15.189:0/2597918395 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f784006adb0 msgr2=0x7f784006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:44.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.825+0000 7f78527fc700 1 -- 172.21.15.189:0/2597918395 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f7854022070 con 0x7f785c15ddf0 2024-04-06T12:27:44.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.825+0000 7f78619aa700 1 --2- 172.21.15.189:0/2597918395 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f784006adb0 0x7f784006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:44.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:44.830+0000 7f78527fc700 1 -- 172.21.15.189:0/2597918395 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f78540be650 con 0x7f785c15ddf0 2024-04-06T12:27:44.939 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:44.942+0000 7fd720bf8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:27:44.988 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:44.991+0000 7fc1585b6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:27:45.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.026+0000 7f78619aa700 1 -- 172.21.15.189:0/2597918395 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f784006adb0 msgr2=0x7f784006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:45.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.026+0000 7f78619aa700 1 --2- 172.21.15.189:0/2597918395 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f784006adb0 0x7f784006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:45.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.043+0000 7f7863c0e700 1 -- 172.21.15.189:0/2597918395 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f785c074990 con 0x7f785c15ddf0 2024-04-06T12:27:45.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.044+0000 7f78527fc700 1 -- 172.21.15.189:0/2597918395 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f785402c0a0 con 0x7f785c15ddf0 2024-04-06T12:27:45.041 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:27:45.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.046+0000 7f7863c0e700 1 -- 172.21.15.189:0/2597918395 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f784006adb0 msgr2=0x7f784006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:45.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.046+0000 7f7863c0e700 1 --2- 172.21.15.189:0/2597918395 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f784006adb0 0x7f784006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.046+0000 7f7863c0e700 1 -- 172.21.15.189:0/2597918395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f785c15ddf0 msgr2=0x7f785c0cf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:45.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.046+0000 7f7863c0e700 1 --2- 172.21.15.189:0/2597918395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f785c15ddf0 0x7f785c0cf470 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7854016000 tx=0x7f7854004ad0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.047+0000 7f7863c0e700 1 -- 172.21.15.189:0/2597918395 shutdown_connections 2024-04-06T12:27:45.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.047+0000 7f7863c0e700 1 --2- 172.21.15.189:0/2597918395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f785c15ddf0 0x7f785c0cf470 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.047+0000 7f7863c0e700 1 --2- 172.21.15.189:0/2597918395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f785c06ae40 0x7f785c0aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.047+0000 7f7863c0e700 1 --2- 172.21.15.189:0/2597918395 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f784006adb0 0x7f784006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.047+0000 7f7863c0e700 1 --2- 172.21.15.189:0/2597918395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f785c15d7b0 0x7f785c1f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.047+0000 7f7863c0e700 1 -- 172.21.15.189:0/2597918395 >> 172.21.15.189:0/2597918395 conn(0x7f785c0d00b0 msgr2=0x7f785c15f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:45.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.047+0000 7f7863c0e700 1 -- 172.21.15.189:0/2597918395 shutdown_connections 2024-04-06T12:27:45.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.047+0000 7f7863c0e700 1 -- 172.21.15.189:0/2597918395 wait complete. 2024-04-06T12:27:45.057 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:27:45.096 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:45.098+0000 7fd720bf8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:27:45.156 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:45.158+0000 7fc1585b6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:27:45.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.188+0000 7f6b49887700 1 Processor -- start 2024-04-06T12:27:45.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.188+0000 7f6b49887700 1 -- start start 2024-04-06T12:27:45.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.189+0000 7f6b49887700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b440d0a60 0x7f6b440d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:45.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.189+0000 7f6b49887700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6b440d1af0 0x7f6b440b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:45.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.189+0000 7f6b49887700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b440cdf80 con 0x7f6b4415d7e0 2024-04-06T12:27:45.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.189+0000 7f6b49887700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b440d1f80 con 0x7f6b440d1af0 2024-04-06T12:27:45.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.189+0000 7f6b49887700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b440d3230 con 0x7f6b440d0a60 2024-04-06T12:27:45.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.189+0000 7f6b42ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6b4415d7e0 0x7f6b440d0350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44380/0 (socket says 172.21.15.189:44380) 2024-04-06T12:27:45.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.189+0000 7f6b42ffd700 1 -- 172.21.15.189:0/803900049 learned_addr learned my addr 172.21.15.189:0/803900049 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:45.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.189+0000 7f6b437fe700 1 --2- 172.21.15.189:0/803900049 >> v2:172.21.15.189:3300/0 conn(0x7f6b440d1af0 0x7f6b440b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:45.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.189+0000 7f6b427fc700 1 --2- 172.21.15.189:0/803900049 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b440d0a60 0x7f6b440d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:45.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.189+0000 7f6b41ffb700 1 -- 172.21.15.189:0/803900049 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945145684 0 0) 0x7f6b440cdf80 con 0x7f6b4415d7e0 2024-04-06T12:27:45.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.189+0000 7f6b41ffb700 1 -- 172.21.15.189:0/803900049 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b2c0035b0 con 0x7f6b4415d7e0 2024-04-06T12:27:45.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.189+0000 7f6b437fe700 1 -- 172.21.15.189:0/803900049 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b440d0a60 msgr2=0x7f6b440d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:45.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.189+0000 7f6b437fe700 1 --2- 172.21.15.189:0/803900049 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b440d0a60 0x7f6b440d15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.189+0000 7f6b437fe700 1 -- 172.21.15.189:0/803900049 >> v1:172.21.15.189:6789/0 conn(0x7f6b4415d7e0 legacy=0x7f6b440d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:45.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.189+0000 7f6b437fe700 1 -- 172.21.15.189:0/803900049 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b440d2da0 con 0x7f6b440d1af0 2024-04-06T12:27:45.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.189+0000 7f6b437fe700 1 --2- 172.21.15.189:0/803900049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b440d1af0 0x7f6b440b66b0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c009d90 tx=0x7f6b3c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54c6368bef36c606 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:45.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.189+0000 7f6b41ffb700 1 -- 172.21.15.189:0/803900049 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c00e070 con 0x7f6b440d1af0 2024-04-06T12:27:45.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.189+0000 7f6b41ffb700 1 -- 172.21.15.189:0/803900049 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3c00bb20 con 0x7f6b440d1af0 2024-04-06T12:27:45.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.189+0000 7f6b41ffb700 1 -- 172.21.15.189:0/803900049 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c012430 con 0x7f6b440d1af0 2024-04-06T12:27:45.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.190+0000 7f6b49887700 1 -- 172.21.15.189:0/803900049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b440d1af0 msgr2=0x7f6b440b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:45.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.190+0000 7f6b49887700 1 --2- 172.21.15.189:0/803900049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b440d1af0 0x7f6b440b66b0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c009d90 tx=0x7f6b3c00b330 comp rx=0 tx=0).stop 2024-04-06T12:27:45.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.190+0000 7f6b49887700 1 -- 172.21.15.189:0/803900049 shutdown_connections 2024-04-06T12:27:45.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.190+0000 7f6b49887700 1 --2- 172.21.15.189:0/803900049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b440d1af0 0x7f6b440b66b0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.190+0000 7f6b49887700 1 --2- 172.21.15.189:0/803900049 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b440d0a60 0x7f6b440d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.190+0000 7f6b49887700 1 -- 172.21.15.189:0/803900049 >> 172.21.15.189:0/803900049 conn(0x7f6b44072250 msgr2=0x7f6b44072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:45.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.190+0000 7f6b49887700 1 -- 172.21.15.189:0/803900049 shutdown_connections 2024-04-06T12:27:45.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.190+0000 7f6b49887700 1 -- 172.21.15.189:0/803900049 wait complete. 2024-04-06T12:27:45.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.191+0000 7f6b49887700 1 Processor -- start 2024-04-06T12:27:45.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.192+0000 7f6b49887700 1 -- start start 2024-04-06T12:27:45.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.192+0000 7f6b49887700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b440d0a60 0x7f6b440c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:45.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.192+0000 7f6b49887700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b440d1af0 0x7f6b440cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:45.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.192+0000 7f6b49887700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b4415d7e0 0x7f6b440ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:45.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.192+0000 7f6b49887700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4414c260 con 0x7f6b4415d7e0 2024-04-06T12:27:45.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.192+0000 7f6b49887700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4414c540 con 0x7f6b440d0a60 2024-04-06T12:27:45.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.192+0000 7f6b49887700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4414c820 con 0x7f6b440d1af0 2024-04-06T12:27:45.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.192+0000 7f6b437fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b4415d7e0 0x7f6b440ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:45.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.192+0000 7f6b437fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b4415d7e0 0x7f6b440ce3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48532/0 (socket says 172.21.15.189:48532) 2024-04-06T12:27:45.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.192+0000 7f6b437fe700 1 -- 172.21.15.189:0/484449670 learned_addr learned my addr 172.21.15.189:0/484449670 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:45.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.193+0000 7f6b42ffd700 1 --2- 172.21.15.189:0/484449670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b440d0a60 0x7f6b440c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:45.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.193+0000 7f6b427fc700 1 --2- 172.21.15.189:0/484449670 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b440d1af0 0x7f6b440cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:45.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.193+0000 7f6b437fe700 1 -- 172.21.15.189:0/484449670 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b440d1af0 msgr2=0x7f6b440cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:45.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.193+0000 7f6b437fe700 1 --2- 172.21.15.189:0/484449670 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b440d1af0 0x7f6b440cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.193+0000 7f6b437fe700 1 -- 172.21.15.189:0/484449670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b440d0a60 msgr2=0x7f6b440c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:45.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.193+0000 7f6b437fe700 1 --2- 172.21.15.189:0/484449670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b440d0a60 0x7f6b440c70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.193+0000 7f6b437fe700 1 -- 172.21.15.189:0/484449670 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c009a70 con 0x7f6b4415d7e0 2024-04-06T12:27:45.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.193+0000 7f6b437fe700 1 --2- 172.21.15.189:0/484449670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b4415d7e0 0x7f6b440ce3f0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c014040 tx=0x7f6b3c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:45.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.193+0000 7f6b2bfff700 1 -- 172.21.15.189:0/484449670 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c0129f0 con 0x7f6b4415d7e0 2024-04-06T12:27:45.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.193+0000 7f6b2bfff700 1 -- 172.21.15.189:0/484449670 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3c020070 con 0x7f6b4415d7e0 2024-04-06T12:27:45.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.193+0000 7f6b49887700 1 -- 172.21.15.189:0/484449670 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b4414cb00 con 0x7f6b4415d7e0 2024-04-06T12:27:45.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.193+0000 7f6b2bfff700 1 -- 172.21.15.189:0/484449670 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c012db0 con 0x7f6b4415d7e0 2024-04-06T12:27:45.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.193+0000 7f6b49887700 1 -- 172.21.15.189:0/484449670 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b440c75f0 con 0x7f6b4415d7e0 2024-04-06T12:27:45.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.194+0000 7f6b49887700 1 -- 172.21.15.189:0/484449670 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b4414cd40 con 0x7f6b4415d7e0 2024-04-06T12:27:45.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.195+0000 7f6b2bfff700 1 -- 172.21.15.189:0/484449670 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 37) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f6b3c004040 con 0x7f6b4415d7e0 2024-04-06T12:27:45.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.195+0000 7f6b2bfff700 1 --2- 172.21.15.189:0/484449670 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6b3006adb0 0x7f6b3006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:45.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.195+0000 7f6b42ffd700 1 -- 172.21.15.189:0/484449670 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6b3006adb0 msgr2=0x7f6b3006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:45.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.195+0000 7f6b2bfff700 1 -- 172.21.15.189:0/484449670 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f6b3c022070 con 0x7f6b4415d7e0 2024-04-06T12:27:45.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.195+0000 7f6b42ffd700 1 --2- 172.21.15.189:0/484449670 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6b3006adb0 0x7f6b3006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:45.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.199+0000 7f6b2bfff700 1 -- 172.21.15.189:0/484449670 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6b4414cd40 con 0x7f6b4415d7e0 2024-04-06T12:27:45.321 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:45.324+0000 7fc1585b6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:27:45.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.396+0000 7f6b42ffd700 1 -- 172.21.15.189:0/484449670 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6b3006adb0 msgr2=0x7f6b3006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:45.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.396+0000 7f6b42ffd700 1 --2- 172.21.15.189:0/484449670 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6b3006adb0 0x7f6b3006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:45.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.402+0000 7f6b49887700 1 -- 172.21.15.189:0/484449670 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6b4414c3a0 con 0x7f6b4415d7e0 2024-04-06T12:27:45.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.403+0000 7f6b2bfff700 1 -- 172.21.15.189:0/484449670 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6b4414c3a0 con 0x7f6b4415d7e0 2024-04-06T12:27:45.400 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:27:45.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.405+0000 7f6b49887700 1 -- 172.21.15.189:0/484449670 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6b3006adb0 msgr2=0x7f6b3006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:45.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.405+0000 7f6b49887700 1 --2- 172.21.15.189:0/484449670 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6b3006adb0 0x7f6b3006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.405+0000 7f6b49887700 1 -- 172.21.15.189:0/484449670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b4415d7e0 msgr2=0x7f6b440ce3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:45.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.405+0000 7f6b49887700 1 --2- 172.21.15.189:0/484449670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b4415d7e0 0x7f6b440ce3f0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c014040 tx=0x7f6b3c004ad0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.405+0000 7f6b49887700 1 -- 172.21.15.189:0/484449670 shutdown_connections 2024-04-06T12:27:45.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.405+0000 7f6b49887700 1 --2- 172.21.15.189:0/484449670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b4415d7e0 0x7f6b440ce3f0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.405+0000 7f6b49887700 1 --2- 172.21.15.189:0/484449670 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6b3006adb0 0x7f6b3006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.405+0000 7f6b49887700 1 --2- 172.21.15.189:0/484449670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b440d0a60 0x7f6b440c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.405+0000 7f6b49887700 1 --2- 172.21.15.189:0/484449670 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b440d1af0 0x7f6b440cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.405+0000 7f6b49887700 1 -- 172.21.15.189:0/484449670 >> 172.21.15.189:0/484449670 conn(0x7f6b44072250 msgr2=0x7f6b4415e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:45.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.405+0000 7f6b49887700 1 -- 172.21.15.189:0/484449670 shutdown_connections 2024-04-06T12:27:45.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.405+0000 7f6b49887700 1 -- 172.21.15.189:0/484449670 wait complete. 2024-04-06T12:27:45.415 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:27:45.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.549+0000 7f32931b1700 1 Processor -- start 2024-04-06T12:27:45.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.549+0000 7f32931b1700 1 -- start start 2024-04-06T12:27:45.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.550+0000 7f32931b1700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f328c15d590 0x7f328c0b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:45.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.550+0000 7f32931b1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328c0b4ed0 0x7f328c0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:45.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.550+0000 7f32931b1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f328c0b6580 0x7f328c0b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:45.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.550+0000 7f32931b1700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328c0b5890 con 0x7f328c0b6580 2024-04-06T12:27:45.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.550+0000 7f32931b1700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328c0b5cc0 con 0x7f328c15d590 2024-04-06T12:27:45.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.550+0000 7f32931b1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328c0d1500 con 0x7f328c0b4ed0 2024-04-06T12:27:45.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.550+0000 7f329174e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f328c0b6580 0x7f328c0b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:45.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.550+0000 7f329174e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f328c0b6580 0x7f328c0b6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48540/0 (socket says 172.21.15.189:48540) 2024-04-06T12:27:45.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.550+0000 7f329174e700 1 -- 172.21.15.189:0/3730903538 learned_addr learned my addr 172.21.15.189:0/3730903538 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:45.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.550+0000 7f328bfff700 1 --2- 172.21.15.189:0/3730903538 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328c0b4ed0 0x7f328c0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:45.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.550+0000 7f3290f4d700 1 --2- 172.21.15.189:0/3730903538 >> v2:172.21.15.192:3300/0 conn(0x7f328c15d590 0x7f328c0b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:45.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.550+0000 7f329174e700 1 -- 172.21.15.189:0/3730903538 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328c0b4ed0 msgr2=0x7f328c0b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:45.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.550+0000 7f329174e700 1 --2- 172.21.15.189:0/3730903538 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328c0b4ed0 0x7f328c0b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.550+0000 7f329174e700 1 -- 172.21.15.189:0/3730903538 >> v2:172.21.15.192:3300/0 conn(0x7f328c15d590 msgr2=0x7f328c0b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:45.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.550+0000 7f329174e700 1 --2- 172.21.15.189:0/3730903538 >> v2:172.21.15.192:3300/0 conn(0x7f328c15d590 0x7f328c0b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.550+0000 7f329174e700 1 -- 172.21.15.189:0/3730903538 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c0d17e0 con 0x7f328c0b6580 2024-04-06T12:27:45.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.551+0000 7f329174e700 1 --2- 172.21.15.189:0/3730903538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c0b6580 0x7f328c0b6a10 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f3284009e50 tx=0x7f328400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b2d45542590b5de server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:45.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.551+0000 7f328b7fe700 1 -- 172.21.15.189:0/3730903538 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328400e070 con 0x7f328c0b6580 2024-04-06T12:27:45.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.551+0000 7f328b7fe700 1 -- 172.21.15.189:0/3730903538 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328400b8d0 con 0x7f328c0b6580 2024-04-06T12:27:45.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.551+0000 7f328b7fe700 1 -- 172.21.15.189:0/3730903538 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328400bc80 con 0x7f328c0b6580 2024-04-06T12:27:45.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.551+0000 7f32931b1700 1 -- 172.21.15.189:0/3730903538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c0b6580 msgr2=0x7f328c0b6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:45.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.551+0000 7f32931b1700 1 --2- 172.21.15.189:0/3730903538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c0b6580 0x7f328c0b6a10 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f3284009e50 tx=0x7f328400b330 comp rx=0 tx=0).stop 2024-04-06T12:27:45.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.551+0000 7f32931b1700 1 -- 172.21.15.189:0/3730903538 shutdown_connections 2024-04-06T12:27:45.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.551+0000 7f32931b1700 1 --2- 172.21.15.189:0/3730903538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c0b6580 0x7f328c0b6a10 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.551+0000 7f32931b1700 1 --2- 172.21.15.189:0/3730903538 >> v2:172.21.15.192:3300/0 conn(0x7f328c15d590 0x7f328c0b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.551+0000 7f32931b1700 1 --2- 172.21.15.189:0/3730903538 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328c0b4ed0 0x7f328c0b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.551+0000 7f32931b1700 1 -- 172.21.15.189:0/3730903538 >> 172.21.15.189:0/3730903538 conn(0x7f328c06bbc0 msgr2=0x7f328c06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:45.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.551+0000 7f32931b1700 1 -- 172.21.15.189:0/3730903538 shutdown_connections 2024-04-06T12:27:45.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.551+0000 7f32931b1700 1 -- 172.21.15.189:0/3730903538 wait complete. 2024-04-06T12:27:45.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.552+0000 7f32931b1700 1 Processor -- start 2024-04-06T12:27:45.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.553+0000 7f32931b1700 1 -- start start 2024-04-06T12:27:45.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.553+0000 7f32931b1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f328c0b4ed0 0x7f328c159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:45.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.553+0000 7f32931b1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328c0b6580 0x7f328c15ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:45.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.553+0000 7f32931b1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c15d590 0x7f328c0c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:45.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.553+0000 7f32931b1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c0ca520 con 0x7f328c15d590 2024-04-06T12:27:45.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.553+0000 7f32931b1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c0cb380 con 0x7f328c0b4ed0 2024-04-06T12:27:45.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.553+0000 7f32931b1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328c0ca770 con 0x7f328c0b6580 2024-04-06T12:27:45.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.554+0000 7f329174e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c15d590 0x7f328c0c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:45.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.554+0000 7f329174e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c15d590 0x7f328c0c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48550/0 (socket says 172.21.15.189:48550) 2024-04-06T12:27:45.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.554+0000 7f329174e700 1 -- 172.21.15.189:0/1759635738 learned_addr learned my addr 172.21.15.189:0/1759635738 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:45.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.554+0000 7f3290f4d700 1 --2- 172.21.15.189:0/1759635738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f328c0b4ed0 0x7f328c159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:45.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.554+0000 7f328bfff700 1 --2- 172.21.15.189:0/1759635738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328c0b6580 0x7f328c15ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:45.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.554+0000 7f3290f4d700 1 -- 172.21.15.189:0/1759635738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328c0b6580 msgr2=0x7f328c15ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:45.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.554+0000 7f3290f4d700 1 --2- 172.21.15.189:0/1759635738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328c0b6580 0x7f328c15ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.554+0000 7f3290f4d700 1 -- 172.21.15.189:0/1759635738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c15d590 msgr2=0x7f328c0c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:45.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.554+0000 7f3290f4d700 1 --2- 172.21.15.189:0/1759635738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c15d590 0x7f328c0c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.554+0000 7f3290f4d700 1 -- 172.21.15.189:0/1759635738 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3284009b30 con 0x7f328c0b4ed0 2024-04-06T12:27:45.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.554+0000 7f329174e700 1 --2- 172.21.15.189:0/1759635738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c15d590 0x7f328c0c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:45.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.554+0000 7f3290f4d700 1 --2- 172.21.15.189:0/1759635738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f328c0b4ed0 0x7f328c159090 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f328000e870 tx=0x7f328000eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:45.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.554+0000 7f32897fa700 1 -- 172.21.15.189:0/1759635738 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3280013680 con 0x7f328c0b4ed0 2024-04-06T12:27:45.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.554+0000 7f32897fa700 1 -- 172.21.15.189:0/1759635738 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32800137e0 con 0x7f328c0b4ed0 2024-04-06T12:27:45.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.554+0000 7f32931b1700 1 -- 172.21.15.189:0/1759635738 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328c0caa50 con 0x7f328c0b4ed0 2024-04-06T12:27:45.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.554+0000 7f32897fa700 1 -- 172.21.15.189:0/1759635738 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3280013b90 con 0x7f328c0b4ed0 2024-04-06T12:27:45.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.554+0000 7f32931b1700 1 -- 172.21.15.189:0/1759635738 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328c0cb010 con 0x7f328c0b4ed0 2024-04-06T12:27:45.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.555+0000 7f32931b1700 1 -- 172.21.15.189:0/1759635738 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328c074990 con 0x7f328c0b4ed0 2024-04-06T12:27:45.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.556+0000 7f32897fa700 1 -- 172.21.15.189:0/1759635738 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 37) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f3280013cf0 con 0x7f328c0b4ed0 2024-04-06T12:27:45.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.556+0000 7f32897fa700 1 --2- 172.21.15.189:0/1759635738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f327006adb0 0x7f327006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:45.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.556+0000 7f328bfff700 1 -- 172.21.15.189:0/1759635738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f327006adb0 msgr2=0x7f327006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:45.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.556+0000 7f32897fa700 1 -- 172.21.15.189:0/1759635738 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f32800895b0 con 0x7f328c0b4ed0 2024-04-06T12:27:45.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.556+0000 7f328bfff700 1 --2- 172.21.15.189:0/1759635738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f327006adb0 0x7f327006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:45.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.561+0000 7f32897fa700 1 -- 172.21.15.189:0/1759635738 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3280054340 con 0x7f328c0b4ed0 2024-04-06T12:27:45.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.757+0000 7f328bfff700 1 -- 172.21.15.189:0/1759635738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f327006adb0 msgr2=0x7f327006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:45.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.757+0000 7f328bfff700 1 --2- 172.21.15.189:0/1759635738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f327006adb0 0x7f327006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:45.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.763+0000 7f32931b1700 1 -- 172.21.15.189:0/1759635738 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f328c0b5560 con 0x7f328c0b4ed0 2024-04-06T12:27:45.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.764+0000 7f32897fa700 1 -- 172.21.15.189:0/1759635738 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3280057960 con 0x7f328c0b4ed0 2024-04-06T12:27:45.761 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:27:45.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.766+0000 7f32931b1700 1 -- 172.21.15.189:0/1759635738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f327006adb0 msgr2=0x7f327006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:45.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.766+0000 7f32931b1700 1 --2- 172.21.15.189:0/1759635738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f327006adb0 0x7f327006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.766+0000 7f32931b1700 1 -- 172.21.15.189:0/1759635738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f328c0b4ed0 msgr2=0x7f328c159090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:45.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.766+0000 7f32931b1700 1 --2- 172.21.15.189:0/1759635738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f328c0b4ed0 0x7f328c159090 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f328000e870 tx=0x7f328000eb80 comp rx=0 tx=0).stop 2024-04-06T12:27:45.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.766+0000 7f32931b1700 1 -- 172.21.15.189:0/1759635738 shutdown_connections 2024-04-06T12:27:45.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.766+0000 7f32931b1700 1 --2- 172.21.15.189:0/1759635738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c15d590 0x7f328c0c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.766+0000 7f32931b1700 1 --2- 172.21.15.189:0/1759635738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f327006adb0 0x7f327006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.766+0000 7f32931b1700 1 --2- 172.21.15.189:0/1759635738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f328c0b4ed0 0x7f328c159090 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.766+0000 7f32931b1700 1 --2- 172.21.15.189:0/1759635738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328c0b6580 0x7f328c15ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.766+0000 7f32931b1700 1 -- 172.21.15.189:0/1759635738 >> 172.21.15.189:0/1759635738 conn(0x7f328c06bbc0 msgr2=0x7f328c15bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:45.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.766+0000 7f32931b1700 1 -- 172.21.15.189:0/1759635738 shutdown_connections 2024-04-06T12:27:45.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.766+0000 7f32931b1700 1 -- 172.21.15.189:0/1759635738 wait complete. 2024-04-06T12:27:45.776 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:27:45.816 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:45.819+0000 7fd720bf8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:27:45.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.903+0000 7fbb9cf36700 1 Processor -- start 2024-04-06T12:27:45.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.903+0000 7fbb9cf36700 1 -- start start 2024-04-06T12:27:45.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.905+0000 7fbb9cf36700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbb980b5380 0x7fbb980d2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:45.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.905+0000 7fbb9cf36700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb980b5cc0 con 0x7fbb9815d590 2024-04-06T12:27:45.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.905+0000 7fbb9cf36700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb980b57d0 con 0x7fbb980b4ed0 2024-04-06T12:27:45.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.905+0000 7fbb9cf36700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb980d1500 con 0x7fbb980b5380 2024-04-06T12:27:45.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.905+0000 7fbb9659c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbb9815d590 0x7fbb980b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44392/0 (socket says 172.21.15.189:44392) 2024-04-06T12:27:45.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.905+0000 7fbb9659c700 1 -- 172.21.15.189:0/82294315 learned_addr learned my addr 172.21.15.189:0/82294315 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:45.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.905+0000 7fbb96d9d700 1 --2- 172.21.15.189:0/82294315 >> v2:172.21.15.189:3300/0 conn(0x7fbb980b5380 0x7fbb980d2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:45.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.905+0000 7fbb9559a700 1 -- 172.21.15.189:0/82294315 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668208209 0 0) 0x7fbb980b5cc0 con 0x7fbb9815d590 2024-04-06T12:27:45.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.905+0000 7fbb9559a700 1 -- 172.21.15.189:0/82294315 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb800035b0 con 0x7fbb9815d590 2024-04-06T12:27:45.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.905+0000 7fbb96d9d700 1 -- 172.21.15.189:0/82294315 >> v1:172.21.15.192:6789/0 conn(0x7fbb980b4ed0 legacy=0x7fbb980d2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:45.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.905+0000 7fbb96d9d700 1 -- 172.21.15.189:0/82294315 >> v1:172.21.15.189:6789/0 conn(0x7fbb9815d590 legacy=0x7fbb980b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:45.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.905+0000 7fbb96d9d700 1 -- 172.21.15.189:0/82294315 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb980d17e0 con 0x7fbb980b5380 2024-04-06T12:27:45.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.905+0000 7fbb96d9d700 1 --2- 172.21.15.189:0/82294315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb980b5380 0x7fbb980d2f90 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fbb90009d90 tx=0x7fbb9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9719e969f019600 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:45.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.905+0000 7fbb9559a700 1 -- 172.21.15.189:0/82294315 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9000e070 con 0x7fbb980b5380 2024-04-06T12:27:45.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.905+0000 7fbb9559a700 1 -- 172.21.15.189:0/82294315 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb9000bb20 con 0x7fbb980b5380 2024-04-06T12:27:45.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.905+0000 7fbb9559a700 1 -- 172.21.15.189:0/82294315 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb90012430 con 0x7fbb980b5380 2024-04-06T12:27:45.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.906+0000 7fbb9cf36700 1 -- 172.21.15.189:0/82294315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb980b5380 msgr2=0x7fbb980d2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:45.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.906+0000 7fbb9cf36700 1 --2- 172.21.15.189:0/82294315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb980b5380 0x7fbb980d2f90 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fbb90009d90 tx=0x7fbb9000b330 comp rx=0 tx=0).stop 2024-04-06T12:27:45.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.906+0000 7fbb9cf36700 1 -- 172.21.15.189:0/82294315 shutdown_connections 2024-04-06T12:27:45.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.906+0000 7fbb9cf36700 1 --2- 172.21.15.189:0/82294315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb980b5380 0x7fbb980d2f90 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.906+0000 7fbb9cf36700 1 -- 172.21.15.189:0/82294315 >> 172.21.15.189:0/82294315 conn(0x7fbb9806bbc0 msgr2=0x7fbb9806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:45.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.906+0000 7fbb9cf36700 1 -- 172.21.15.189:0/82294315 shutdown_connections 2024-04-06T12:27:45.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.906+0000 7fbb9cf36700 1 -- 172.21.15.189:0/82294315 wait complete. 2024-04-06T12:27:45.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.908+0000 7fbb9cf36700 1 Processor -- start 2024-04-06T12:27:45.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.909+0000 7fbb9cf36700 1 -- start start 2024-04-06T12:27:45.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb9cf36700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb980b4ed0 0x7fbb9815e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:45.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb9cf36700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb980b5380 0x7fbb981f25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:45.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb9cf36700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb9815d590 0x7fbb9815f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:45.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb9cf36700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9815f000 con 0x7fbb9815d590 2024-04-06T12:27:45.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb9cf36700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9815ea40 con 0x7fbb980b5380 2024-04-06T12:27:45.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb9cf36700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9815ed20 con 0x7fbb980b4ed0 2024-04-06T12:27:45.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb96d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb9815d590 0x7fbb9815f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:45.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb96d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb9815d590 0x7fbb9815f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48568/0 (socket says 172.21.15.189:48568) 2024-04-06T12:27:45.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb96d9d700 1 -- 172.21.15.189:0/696879075 learned_addr learned my addr 172.21.15.189:0/696879075 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:45.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb9659c700 1 --2- 172.21.15.189:0/696879075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb980b4ed0 0x7fbb9815e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:45.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb95d9b700 1 --2- 172.21.15.189:0/696879075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb980b5380 0x7fbb981f25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:45.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb96d9d700 1 -- 172.21.15.189:0/696879075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb980b4ed0 msgr2=0x7fbb9815e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:45.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb96d9d700 1 --2- 172.21.15.189:0/696879075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb980b4ed0 0x7fbb9815e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb96d9d700 1 -- 172.21.15.189:0/696879075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb980b5380 msgr2=0x7fbb981f25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:45.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb96d9d700 1 --2- 172.21.15.189:0/696879075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb980b5380 0x7fbb981f25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:45.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb96d9d700 1 -- 172.21.15.189:0/696879075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb90009a70 con 0x7fbb9815d590 2024-04-06T12:27:45.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb96d9d700 1 --2- 172.21.15.189:0/696879075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb9815d590 0x7fbb9815f710 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fbb980d41b0 tx=0x7fbb90004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:45.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb86ffd700 1 -- 172.21.15.189:0/696879075 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb90012430 con 0x7fbb9815d590 2024-04-06T12:27:45.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb86ffd700 1 -- 172.21.15.189:0/696879075 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb90020070 con 0x7fbb9815d590 2024-04-06T12:27:45.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.910+0000 7fbb9cf36700 1 -- 172.21.15.189:0/696879075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb980cb710 con 0x7fbb9815d590 2024-04-06T12:27:45.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.911+0000 7fbb86ffd700 1 -- 172.21.15.189:0/696879075 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb90004280 con 0x7fbb9815d590 2024-04-06T12:27:45.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.911+0000 7fbb9cf36700 1 -- 172.21.15.189:0/696879075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb98160560 con 0x7fbb9815d590 2024-04-06T12:27:45.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.912+0000 7fbb9cf36700 1 -- 172.21.15.189:0/696879075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb980cb570 con 0x7fbb9815d590 2024-04-06T12:27:45.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.912+0000 7fbb86ffd700 1 -- 172.21.15.189:0/696879075 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 37) v1 ==== 89248+0+0 (secure 0 0 0) 0x7fbb900043e0 con 0x7fbb9815d590 2024-04-06T12:27:45.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.912+0000 7fbb86ffd700 1 --2- 172.21.15.189:0/696879075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbb7806adb0 0x7fbb7806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:45.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.913+0000 7fbb9659c700 1 -- 172.21.15.189:0/696879075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbb7806adb0 msgr2=0x7fbb7806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:45.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.913+0000 7fbb86ffd700 1 -- 172.21.15.189:0/696879075 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fbb90022070 con 0x7fbb9815d590 2024-04-06T12:27:45.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.913+0000 7fbb9659c700 1 --2- 172.21.15.189:0/696879075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbb7806adb0 0x7fbb7806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:45.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:45.917+0000 7fbb86ffd700 1 -- 172.21.15.189:0/696879075 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbb980cb570 con 0x7fbb9815d590 2024-04-06T12:27:46.055 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:46.058+0000 7fd720bf8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:27:46.090 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:46.093+0000 7fc1585b6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:27:46.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.114+0000 7fbb9659c700 1 -- 172.21.15.189:0/696879075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbb7806adb0 msgr2=0x7fbb7806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:46.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.114+0000 7fbb9659c700 1 --2- 172.21.15.189:0/696879075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbb7806adb0 0x7fbb7806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:46.130 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:46.133+0000 7fd720bf8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:27:46.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.168+0000 7fbb9cf36700 1 -- 172.21.15.189:0/696879075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb980cbc30 con 0x7fbb9815d590 2024-04-06T12:27:46.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.169+0000 7fbb86ffd700 1 -- 172.21.15.189:0/696879075 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4747 (secure 0 0 0) 0x7fbb980cbc30 con 0x7fbb9815d590 2024-04-06T12:27:46.166 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:46.166 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5762,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/2827078766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2827078766},{"type":"v1","addr":"172.21.15.192:6833","nonce":2827078766}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5765,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1558111999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1558111999},{"type":"v1","addr":"172.21.15.192:6835","nonce":1558111999}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3448511259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3448511259},{"type":"v1","addr":"172.21.15.189:6837","nonce":3448511259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:27:21.031012+0000","modified":"2024-04-06T12:27:24.038778+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/1085319880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1085319880},{"type":"v1","addr":"172.21.15.189:6836","nonce":1085319880}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-06T12:27:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.172+0000 7fbb9cf36700 1 -- 172.21.15.189:0/696879075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbb7806adb0 msgr2=0x7fbb7806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.172+0000 7fbb9cf36700 1 --2- 172.21.15.189:0/696879075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbb7806adb0 0x7fbb7806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.172+0000 7fbb9cf36700 1 -- 172.21.15.189:0/696879075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb9815d590 msgr2=0x7fbb9815f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.172+0000 7fbb9cf36700 1 --2- 172.21.15.189:0/696879075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb9815d590 0x7fbb9815f710 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fbb980d41b0 tx=0x7fbb90004ad0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.172+0000 7fbb9cf36700 1 -- 172.21.15.189:0/696879075 shutdown_connections 2024-04-06T12:27:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.172+0000 7fbb9cf36700 1 --2- 172.21.15.189:0/696879075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb9815d590 0x7fbb9815f710 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.172+0000 7fbb9cf36700 1 --2- 172.21.15.189:0/696879075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb980b4ed0 0x7fbb9815e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.172+0000 7fbb9cf36700 1 --2- 172.21.15.189:0/696879075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbb7806adb0 0x7fbb7806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.172+0000 7fbb9cf36700 1 --2- 172.21.15.189:0/696879075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb980b5380 0x7fbb981f25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.172+0000 7fbb9cf36700 1 -- 172.21.15.189:0/696879075 >> 172.21.15.189:0/696879075 conn(0x7fbb9806bbc0 msgr2=0x7fbb980b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:46.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.172+0000 7fbb9cf36700 1 -- 172.21.15.189:0/696879075 shutdown_connections 2024-04-06T12:27:46.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.172+0000 7fbb9cf36700 1 -- 172.21.15.189:0/696879075 wait complete. 2024-04-06T12:27:46.170 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 31 2024-04-06T12:27:46.181 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:27:46.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.310+0000 7fc01cb58700 1 Processor -- start 2024-04-06T12:27:46.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.311+0000 7fc01cb58700 1 -- start start 2024-04-06T12:27:46.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.311+0000 7fc01cb58700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0180cedf0 0x7fc0180cf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:46.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.311+0000 7fc01cb58700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0180cf780 con 0x7fc018146dd0 2024-04-06T12:27:46.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.312+0000 7fc01cb58700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc01806d850 con 0x7fc0180d17a0 2024-04-06T12:27:46.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.312+0000 7fc01cb58700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc01806beb0 con 0x7fc0180cedf0 2024-04-06T12:27:46.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.312+0000 7fc015d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc018146dd0 0x7fc01806d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44398/0 (socket says 172.21.15.189:44398) 2024-04-06T12:27:46.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.312+0000 7fc015d9b700 1 -- 172.21.15.189:0/907242577 learned_addr learned my addr 172.21.15.189:0/907242577 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:46.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.312+0000 7fc016d9d700 1 --2- 172.21.15.189:0/907242577 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0180cedf0 0x7fc0180cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:46.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.312+0000 7fc01559a700 1 -- 172.21.15.189:0/907242577 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369208636 0 0) 0x7fc00c001110 con 0x7fc018146dd0 2024-04-06T12:27:46.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.312+0000 7fc01559a700 1 -- 172.21.15.189:0/907242577 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0000035b0 con 0x7fc018146dd0 2024-04-06T12:27:46.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.312+0000 7fc01559a700 1 -- 172.21.15.189:0/907242577 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316153741 0 0) 0x7fc01806d850 con 0x7fc0180d17a0 2024-04-06T12:27:46.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.312+0000 7fc01559a700 1 -- 172.21.15.189:0/907242577 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc00c001110 con 0x7fc0180d17a0 2024-04-06T12:27:46.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.312+0000 7fc016d9d700 1 -- 172.21.15.189:0/907242577 >> v1:172.21.15.192:6789/0 conn(0x7fc0180d17a0 legacy=0x7fc01806b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:46.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.312+0000 7fc016d9d700 1 -- 172.21.15.189:0/907242577 >> v1:172.21.15.189:6789/0 conn(0x7fc018146dd0 legacy=0x7fc01806d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:46.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.312+0000 7fc016d9d700 1 -- 172.21.15.189:0/907242577 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01806c0a0 con 0x7fc0180cedf0 2024-04-06T12:27:46.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.312+0000 7fc016d9d700 1 --2- 172.21.15.189:0/907242577 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0180cedf0 0x7fc0180cf240 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc010009dc0 tx=0x7fc01000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27322983ffbf37a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:46.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.313+0000 7fc01559a700 1 -- 172.21.15.189:0/907242577 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01000e070 con 0x7fc0180cedf0 2024-04-06T12:27:46.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.313+0000 7fc01559a700 1 -- 172.21.15.189:0/907242577 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01000bb20 con 0x7fc0180cedf0 2024-04-06T12:27:46.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.313+0000 7fc01559a700 1 -- 172.21.15.189:0/907242577 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01000be90 con 0x7fc0180cedf0 2024-04-06T12:27:46.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.313+0000 7fc01cb58700 1 -- 172.21.15.189:0/907242577 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0180cedf0 msgr2=0x7fc0180cf240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:46.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.313+0000 7fc01cb58700 1 --2- 172.21.15.189:0/907242577 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0180cedf0 0x7fc0180cf240 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc010009dc0 tx=0x7fc01000b330 comp rx=0 tx=0).stop 2024-04-06T12:27:46.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.313+0000 7fc01cb58700 1 -- 172.21.15.189:0/907242577 shutdown_connections 2024-04-06T12:27:46.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.313+0000 7fc01cb58700 1 --2- 172.21.15.189:0/907242577 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0180cedf0 0x7fc0180cf240 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.313+0000 7fc01cb58700 1 -- 172.21.15.189:0/907242577 >> 172.21.15.189:0/907242577 conn(0x7fc0180b2db0 msgr2=0x7fc0180b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:46.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.313+0000 7fc01cb58700 1 -- 172.21.15.189:0/907242577 shutdown_connections 2024-04-06T12:27:46.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.313+0000 7fc01cb58700 1 -- 172.21.15.189:0/907242577 wait complete. 2024-04-06T12:27:46.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.315+0000 7fc01cb58700 1 Processor -- start 2024-04-06T12:27:46.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.315+0000 7fc01cb58700 1 -- start start 2024-04-06T12:27:46.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.315+0000 7fc01cb58700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0180cedf0 0x7fc018145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:46.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.315+0000 7fc01cb58700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0180d17a0 0x7fc01815f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:46.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.315+0000 7fc01cb58700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc018146dd0 0x7fc01815fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:46.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.315+0000 7fc01cb58700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0180c7690 con 0x7fc0180d17a0 2024-04-06T12:27:46.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.315+0000 7fc01cb58700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0180b49f0 con 0x7fc0180cedf0 2024-04-06T12:27:46.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.315+0000 7fc01cb58700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0180c7080 con 0x7fc018146dd0 2024-04-06T12:27:46.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.316+0000 7fc015d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0180d17a0 0x7fc01815f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:46.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.316+0000 7fc015d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0180d17a0 0x7fc01815f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48578/0 (socket says 172.21.15.189:48578) 2024-04-06T12:27:46.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.316+0000 7fc015d9b700 1 -- 172.21.15.189:0/888843453 learned_addr learned my addr 172.21.15.189:0/888843453 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:46.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.316+0000 7fc016d9d700 1 --2- 172.21.15.189:0/888843453 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc018146dd0 0x7fc01815fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:46.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.316+0000 7fc01659c700 1 --2- 172.21.15.189:0/888843453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0180cedf0 0x7fc018145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:46.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.316+0000 7fc015d9b700 1 -- 172.21.15.189:0/888843453 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc018146dd0 msgr2=0x7fc01815fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:46.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.316+0000 7fc015d9b700 1 --2- 172.21.15.189:0/888843453 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc018146dd0 0x7fc01815fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.316+0000 7fc015d9b700 1 -- 172.21.15.189:0/888843453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0180cedf0 msgr2=0x7fc018145790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:46.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.316+0000 7fc015d9b700 1 --2- 172.21.15.189:0/888843453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0180cedf0 0x7fc018145790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.316+0000 7fc015d9b700 1 -- 172.21.15.189:0/888843453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc010009a70 con 0x7fc0180d17a0 2024-04-06T12:27:46.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.316+0000 7fc015d9b700 1 --2- 172.21.15.189:0/888843453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0180d17a0 0x7fc01815f660 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fc00c005910 tx=0x7fc00c00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:46.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.316+0000 7fc006ffd700 1 -- 172.21.15.189:0/888843453 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c005d70 con 0x7fc0180d17a0 2024-04-06T12:27:46.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.316+0000 7fc006ffd700 1 -- 172.21.15.189:0/888843453 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00c0070d0 con 0x7fc0180d17a0 2024-04-06T12:27:46.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.316+0000 7fc006ffd700 1 -- 172.21.15.189:0/888843453 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c007440 con 0x7fc0180d17a0 2024-04-06T12:27:46.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.316+0000 7fc01cb58700 1 -- 172.21.15.189:0/888843453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0180b4c90 con 0x7fc0180d17a0 2024-04-06T12:27:46.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.316+0000 7fc01cb58700 1 -- 172.21.15.189:0/888843453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0180b63f0 con 0x7fc0180d17a0 2024-04-06T12:27:46.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.317+0000 7fc01cb58700 1 -- 172.21.15.189:0/888843453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc018074990 con 0x7fc0180d17a0 2024-04-06T12:27:46.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.318+0000 7fc006ffd700 1 -- 172.21.15.189:0/888843453 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 37) v1 ==== 89248+0+0 (secure 0 0 0) 0x7fc00c0075a0 con 0x7fc0180d17a0 2024-04-06T12:27:46.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.318+0000 7fc006ffd700 1 --2- 172.21.15.189:0/888843453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbff806adb0 0x7fbff806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:46.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.318+0000 7fc01659c700 1 -- 172.21.15.189:0/888843453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbff806adb0 msgr2=0x7fbff806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:46.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.318+0000 7fc006ffd700 1 -- 172.21.15.189:0/888843453 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fc00c088a90 con 0x7fc0180d17a0 2024-04-06T12:27:46.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.318+0000 7fc01659c700 1 --2- 172.21.15.189:0/888843453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbff806adb0 0x7fbff806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:46.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.323+0000 7fc006ffd700 1 -- 172.21.15.189:0/888843453 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc00c0538d0 con 0x7fc0180d17a0 2024-04-06T12:27:46.351 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:46.354+0000 7fc1585b6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:27:46.432 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:46.435+0000 7fc1585b6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:27:46.503 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:27:46.506+0000 7fd720bf8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:27:46.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.519+0000 7fc01659c700 1 -- 172.21.15.189:0/888843453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbff806adb0 msgr2=0x7fbff806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:46.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.519+0000 7fc01659c700 1 --2- 172.21.15.189:0/888843453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbff806adb0 0x7fbff806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:27:46.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.527+0000 7fc01cb58700 1 -- 172.21.15.189:0/888843453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0180cf780 con 0x7fc0180d17a0 2024-04-06T12:27:46.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.528+0000 7fc006ffd700 1 -- 172.21.15.189:0/888843453 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16793 (secure 0 0 0) 0x7fc0180cf780 con 0x7fc0180d17a0 2024-04-06T12:27:46.526 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:46.526 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":85,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:27:39.087666+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:27:19.575213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-06T12:27:19.990714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/615870143":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/1180713047":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/2430686509":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/535246627":"2024-04-07T12:27:39.087614+0000","172.21.15.189:6832/47128":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/2829807377":"2024-04-07T12:27:39.087614+0000","172.21.15.189:6833/47128":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/4012548359":"2024-04-07T12:27:39.087614+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:27:46.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.531+0000 7fc01cb58700 1 -- 172.21.15.189:0/888843453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbff806adb0 msgr2=0x7fbff806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:46.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.531+0000 7fc01cb58700 1 --2- 172.21.15.189:0/888843453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbff806adb0 0x7fbff806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.531+0000 7fc01cb58700 1 -- 172.21.15.189:0/888843453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0180d17a0 msgr2=0x7fc01815f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:46.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.531+0000 7fc01cb58700 1 --2- 172.21.15.189:0/888843453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0180d17a0 0x7fc01815f660 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fc00c005910 tx=0x7fc00c00c9e0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.531+0000 7fc01cb58700 1 -- 172.21.15.189:0/888843453 shutdown_connections 2024-04-06T12:27:46.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.531+0000 7fc01cb58700 1 --2- 172.21.15.189:0/888843453 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc018146dd0 0x7fc01815fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.531+0000 7fc01cb58700 1 --2- 172.21.15.189:0/888843453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbff806adb0 0x7fbff806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.531+0000 7fc01cb58700 1 --2- 172.21.15.189:0/888843453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0180cedf0 0x7fc018145790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.531+0000 7fc01cb58700 1 --2- 172.21.15.189:0/888843453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0180d17a0 0x7fc01815f660 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.531+0000 7fc01cb58700 1 -- 172.21.15.189:0/888843453 >> 172.21.15.189:0/888843453 conn(0x7fc0180b2db0 msgr2=0x7fc0180b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:46.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.531+0000 7fc01cb58700 1 -- 172.21.15.189:0/888843453 shutdown_connections 2024-04-06T12:27:46.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.531+0000 7fc01cb58700 1 -- 172.21.15.189:0/888843453 wait complete. 2024-04-06T12:27:46.541 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-06T12:27:46.541 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:27:46.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.673+0000 7f8ab15a3700 1 Processor -- start 2024-04-06T12:27:46.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.674+0000 7f8ab15a3700 1 -- start start 2024-04-06T12:27:46.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.674+0000 7f8ab15a3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8aac15a550 0x7f8aac0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:46.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.674+0000 7f8ab15a3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8aac0cf900 0x7f8aac0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:46.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.674+0000 7f8ab15a3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aac0cdbe0 con 0x7f8aac0b4ba0 2024-04-06T12:27:46.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.674+0000 7f8ab15a3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aac0b5010 con 0x7f8aac0cf900 2024-04-06T12:27:46.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.674+0000 7f8ab15a3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac15a940 con 0x7f8aac15a550 2024-04-06T12:27:46.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.675+0000 7f8aab7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8aac0b4ba0 0x7f8aac06eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44406/0 (socket says 172.21.15.189:44406) 2024-04-06T12:27:46.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.675+0000 7f8aab7fe700 1 -- 172.21.15.189:0/1485775657 learned_addr learned my addr 172.21.15.189:0/1485775657 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:46.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.675+0000 7f8aaaffd700 1 --2- 172.21.15.189:0/1485775657 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8aac15a550 0x7f8aac0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:46.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.675+0000 7f8aaa7fc700 1 --2- 172.21.15.189:0/1485775657 >> v2:172.21.15.192:3300/0 conn(0x7f8aac0cf900 0x7f8aac0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:46.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.675+0000 7f8aa9ffb700 1 -- 172.21.15.189:0/1485775657 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812769936 0 0) 0x7f8aac0cdbe0 con 0x7f8aac0b4ba0 2024-04-06T12:27:46.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.675+0000 7f8aa9ffb700 1 -- 172.21.15.189:0/1485775657 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a940035b0 con 0x7f8aac0b4ba0 2024-04-06T12:27:46.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.675+0000 7f8aaaffd700 1 -- 172.21.15.189:0/1485775657 >> v2:172.21.15.192:3300/0 conn(0x7f8aac0cf900 msgr2=0x7f8aac0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:46.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.675+0000 7f8aaaffd700 1 --2- 172.21.15.189:0/1485775657 >> v2:172.21.15.192:3300/0 conn(0x7f8aac0cf900 0x7f8aac0b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.675+0000 7f8aaaffd700 1 -- 172.21.15.189:0/1485775657 >> v1:172.21.15.189:6789/0 conn(0x7f8aac0b4ba0 legacy=0x7f8aac06eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:46.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.675+0000 7f8aaaffd700 1 -- 172.21.15.189:0/1485775657 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac15ac00 con 0x7f8aac15a550 2024-04-06T12:27:46.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.675+0000 7f8aaa7fc700 1 --2- 172.21.15.189:0/1485775657 >> v2:172.21.15.192:3300/0 conn(0x7f8aac0cf900 0x7f8aac0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:46.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.675+0000 7f8aaaffd700 1 --2- 172.21.15.189:0/1485775657 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8aac15a550 0x7f8aac0cf3c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c009840 tx=0x7f8a9c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e6c8692f43dd901 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:46.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.675+0000 7f8aa9ffb700 1 -- 172.21.15.189:0/1485775657 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c011830 con 0x7f8aac15a550 2024-04-06T12:27:46.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.675+0000 7f8aa9ffb700 1 -- 172.21.15.189:0/1485775657 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9c011990 con 0x7f8aac15a550 2024-04-06T12:27:46.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.675+0000 7f8aa9ffb700 1 -- 172.21.15.189:0/1485775657 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c011d20 con 0x7f8aac15a550 2024-04-06T12:27:46.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.676+0000 7f8ab15a3700 1 -- 172.21.15.189:0/1485775657 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8aac15a550 msgr2=0x7f8aac0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:46.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.676+0000 7f8ab15a3700 1 --2- 172.21.15.189:0/1485775657 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8aac15a550 0x7f8aac0cf3c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c009840 tx=0x7f8a9c009b50 comp rx=0 tx=0).stop 2024-04-06T12:27:46.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.676+0000 7f8ab15a3700 1 -- 172.21.15.189:0/1485775657 shutdown_connections 2024-04-06T12:27:46.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.676+0000 7f8ab15a3700 1 --2- 172.21.15.189:0/1485775657 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8aac15a550 0x7f8aac0cf3c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.676+0000 7f8ab15a3700 1 --2- 172.21.15.189:0/1485775657 >> v2:172.21.15.192:3300/0 conn(0x7f8aac0cf900 0x7f8aac0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.676+0000 7f8ab15a3700 1 -- 172.21.15.189:0/1485775657 >> 172.21.15.189:0/1485775657 conn(0x7f8aac0d00b0 msgr2=0x7f8aac0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:46.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.676+0000 7f8ab15a3700 1 -- 172.21.15.189:0/1485775657 shutdown_connections 2024-04-06T12:27:46.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.676+0000 7f8ab15a3700 1 -- 172.21.15.189:0/1485775657 wait complete. 2024-04-06T12:27:46.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.678+0000 7f8ab15a3700 1 Processor -- start 2024-04-06T12:27:46.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.678+0000 7f8ab15a3700 1 -- start start 2024-04-06T12:27:46.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.678+0000 7f8ab15a3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8aac0b4ba0 0x7f8aac14cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:46.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.678+0000 7f8ab15a3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8aac0cf900 0x7f8aac149000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:46.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.678+0000 7f8ab15a3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8aac149720 0x7f8aac06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:46.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.678+0000 7f8ab15a3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac149540 con 0x7f8aac149720 2024-04-06T12:27:46.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.678+0000 7f8ab15a3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac149fe0 con 0x7f8aac0cf900 2024-04-06T12:27:46.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.678+0000 7f8ab15a3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac14a2c0 con 0x7f8aac0b4ba0 2024-04-06T12:27:46.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.678+0000 7f8aab7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8aac149720 0x7f8aac06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:46.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.679+0000 7f8aab7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8aac149720 0x7f8aac06ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48584/0 (socket says 172.21.15.189:48584) 2024-04-06T12:27:46.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.679+0000 7f8aab7fe700 1 -- 172.21.15.189:0/989923303 learned_addr learned my addr 172.21.15.189:0/989923303 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:46.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.679+0000 7f8aaaffd700 1 --2- 172.21.15.189:0/989923303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8aac0b4ba0 0x7f8aac14cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:46.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.679+0000 7f8aaa7fc700 1 --2- 172.21.15.189:0/989923303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8aac0cf900 0x7f8aac149000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:46.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.679+0000 7f8aab7fe700 1 -- 172.21.15.189:0/989923303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8aac0b4ba0 msgr2=0x7f8aac14cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:46.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.679+0000 7f8aab7fe700 1 --2- 172.21.15.189:0/989923303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8aac0b4ba0 0x7f8aac14cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.679+0000 7f8aab7fe700 1 -- 172.21.15.189:0/989923303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8aac0cf900 msgr2=0x7f8aac149000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:46.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.679+0000 7f8aab7fe700 1 --2- 172.21.15.189:0/989923303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8aac0cf900 0x7f8aac149000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.679+0000 7f8aab7fe700 1 -- 172.21.15.189:0/989923303 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa00094a0 con 0x7f8aac149720 2024-04-06T12:27:46.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.679+0000 7f8aaaffd700 1 --2- 172.21.15.189:0/989923303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8aac0b4ba0 0x7f8aac14cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:46.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.679+0000 7f8aaa7fc700 1 --2- 172.21.15.189:0/989923303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8aac0cf900 0x7f8aac149000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:46.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.679+0000 7f8aab7fe700 1 --2- 172.21.15.189:0/989923303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8aac149720 0x7f8aac06ee80 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f8aa400caa0 tx=0x7f8aa400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:46.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.679+0000 7f8a937fe700 1 -- 172.21.15.189:0/989923303 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa400aa80 con 0x7f8aac149720 2024-04-06T12:27:46.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.679+0000 7f8a937fe700 1 -- 172.21.15.189:0/989923303 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa400abe0 con 0x7f8aac149720 2024-04-06T12:27:46.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.679+0000 7f8ab15a3700 1 -- 172.21.15.189:0/989923303 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a9c0094a0 con 0x7f8aac149720 2024-04-06T12:27:46.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.679+0000 7f8a937fe700 1 -- 172.21.15.189:0/989923303 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4012400 con 0x7f8aac149720 2024-04-06T12:27:46.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.679+0000 7f8ab15a3700 1 -- 172.21.15.189:0/989923303 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aac14a880 con 0x7f8aac149720 2024-04-06T12:27:46.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.680+0000 7f8ab15a3700 1 -- 172.21.15.189:0/989923303 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aac067bb0 con 0x7f8aac149720 2024-04-06T12:27:46.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.681+0000 7f8a937fe700 1 -- 172.21.15.189:0/989923303 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 37) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f8aa401b3b0 con 0x7f8aac149720 2024-04-06T12:27:46.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.681+0000 7f8a937fe700 1 --2- 172.21.15.189:0/989923303 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8a9806adb0 0x7f8a9806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:46.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.681+0000 7f8aaaffd700 1 -- 172.21.15.189:0/989923303 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8a9806adb0 msgr2=0x7f8a9806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:27:46.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.681+0000 7f8a937fe700 1 -- 172.21.15.189:0/989923303 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f8aa4089460 con 0x7f8aac149720 2024-04-06T12:27:46.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.681+0000 7f8aaaffd700 1 --2- 172.21.15.189:0/989923303 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8a9806adb0 0x7f8a9806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:27:46.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.686+0000 7f8a937fe700 1 -- 172.21.15.189:0/989923303 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8aa4054290 con 0x7f8aac149720 2024-04-06T12:27:46.823 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:46.826+0000 7fc1585b6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:27:46.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.849+0000 7f8a937fe700 1 -- 172.21.15.189:0/989923303 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 38) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f8aa40876e0 con 0x7f8aac149720 2024-04-06T12:27:46.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.850+0000 7f8a937fe700 1 -- 172.21.15.189:0/989923303 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8a9806adb0 msgr2=0x7f8a9806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:27:46.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.850+0000 7f8a937fe700 1 --2- 172.21.15.189:0/989923303 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8a9806adb0 0x7f8a9806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.911 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:46.914+0000 7fc0d2c93700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:46.912 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:46.914+0000 7fc0d2c93700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:46.912 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:46.914+0000 7fc0d2c93700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:46.912 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:46.914+0000 7fc0d2c93700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:46.912 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:46.914+0000 7fc0d2c93700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:46.912 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:46.915+0000 7fc0d0c8f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:46.912 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:46.915+0000 7fc0d0c8f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:46.913 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:46.915+0000 7fc0d0c8f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:46.913 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:46.915+0000 7fc0d0c8f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:46.913 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:27:46.915+0000 7fc0d0c8f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:27:46.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.952+0000 7f8ab15a3700 1 -- 172.21.15.189:0/989923303 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8aac074990 con 0x7f8aac149720 2024-04-06T12:27:46.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.952+0000 7f8a937fe700 1 -- 172.21.15.189:0/989923303 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4747 (secure 0 0 0) 0x7f8aa4073550 con 0x7f8aac149720 2024-04-06T12:27:46.949 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:46.950 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5762,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/2827078766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2827078766},{"type":"v1","addr":"172.21.15.192:6833","nonce":2827078766}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5765,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1558111999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1558111999},{"type":"v1","addr":"172.21.15.192:6835","nonce":1558111999}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3448511259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3448511259},{"type":"v1","addr":"172.21.15.189:6837","nonce":3448511259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:27:21.031012+0000","modified":"2024-04-06T12:27:24.038778+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/1085319880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1085319880},{"type":"v1","addr":"172.21.15.189:6836","nonce":1085319880}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-06T12:27:46.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.954+0000 7f8ab15a3700 1 -- 172.21.15.189:0/989923303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8aac149720 msgr2=0x7f8aac06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:46.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.954+0000 7f8ab15a3700 1 --2- 172.21.15.189:0/989923303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8aac149720 0x7f8aac06ee80 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f8aa400caa0 tx=0x7f8aa400ce60 comp rx=0 tx=0).stop 2024-04-06T12:27:46.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.954+0000 7f8ab15a3700 1 -- 172.21.15.189:0/989923303 shutdown_connections 2024-04-06T12:27:46.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.954+0000 7f8ab15a3700 1 --2- 172.21.15.189:0/989923303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8aac149720 0x7f8aac06ee80 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.954+0000 7f8ab15a3700 1 --2- 172.21.15.189:0/989923303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8aac0b4ba0 0x7f8aac14cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.954+0000 7f8ab15a3700 1 --2- 172.21.15.189:0/989923303 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8a9806adb0 0x7f8a9806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.954+0000 7f8ab15a3700 1 --2- 172.21.15.189:0/989923303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8aac0cf900 0x7f8aac149000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:46.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.954+0000 7f8ab15a3700 1 -- 172.21.15.189:0/989923303 >> 172.21.15.189:0/989923303 conn(0x7f8aac0d00b0 msgr2=0x7f8aac0b32c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:46.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.954+0000 7f8ab15a3700 1 -- 172.21.15.189:0/989923303 shutdown_connections 2024-04-06T12:27:46.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:46.954+0000 7f8ab15a3700 1 -- 172.21.15.189:0/989923303 wait complete. 2024-04-06T12:27:46.953 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 31 2024-04-06T12:27:46.964 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:27:47.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.089+0000 7f0dd763b700 1 Processor -- start 2024-04-06T12:27:47.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.089+0000 7f0dd763b700 1 -- start start 2024-04-06T12:27:47.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.090+0000 7f0dd763b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0dd006ae40 0x7f0dd00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:47.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.090+0000 7f0dd763b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd00b3990 con 0x7f0dd015d7b0 2024-04-06T12:27:47.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.090+0000 7f0dd763b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd00b3060 con 0x7f0dd015ddf0 2024-04-06T12:27:47.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.090+0000 7f0dd763b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd015e950 con 0x7f0dd006ae40 2024-04-06T12:27:47.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.090+0000 7f0dd53d7700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0dd015d7b0 0x7f0dd006a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44408/0 (socket says 172.21.15.189:44408) 2024-04-06T12:27:47.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.090+0000 7f0dd53d7700 1 -- 172.21.15.189:0/2250070065 learned_addr learned my addr 172.21.15.189:0/2250070065 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:47.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.090+0000 7f0dd4bd6700 1 --2- 172.21.15.189:0/2250070065 >> v2:172.21.15.192:3300/0 conn(0x7f0dd006ae40 0x7f0dd00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:47.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.090+0000 7f0dc7fff700 1 -- 172.21.15.189:0/2250070065 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886102001 0 0) 0x7f0dd00b3990 con 0x7f0dd015d7b0 2024-04-06T12:27:47.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.091+0000 7f0dc7fff700 1 -- 172.21.15.189:0/2250070065 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dbc0035b0 con 0x7f0dd015d7b0 2024-04-06T12:27:47.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.091+0000 7f0dc7fff700 1 -- 172.21.15.189:0/2250070065 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175164009 0 0) 0x7f0dd00b3060 con 0x7f0dd015ddf0 2024-04-06T12:27:47.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.091+0000 7f0dc7fff700 1 -- 172.21.15.189:0/2250070065 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dd00b3990 con 0x7f0dd015ddf0 2024-04-06T12:27:47.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.091+0000 7f0dd4bd6700 1 -- 172.21.15.189:0/2250070065 >> v1:172.21.15.192:6789/0 conn(0x7f0dd015ddf0 legacy=0x7f0dd015e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:47.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.091+0000 7f0dd4bd6700 1 -- 172.21.15.189:0/2250070065 >> v1:172.21.15.189:6789/0 conn(0x7f0dd015d7b0 legacy=0x7f0dd006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:47.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.091+0000 7f0dd4bd6700 1 -- 172.21.15.189:0/2250070065 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd00b3340 con 0x7f0dd006ae40 2024-04-06T12:27:47.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.091+0000 7f0dd4bd6700 1 --2- 172.21.15.189:0/2250070065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dd006ae40 0x7f0dd00b3e20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc009840 tx=0x7f0dcc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f26869b78053f86 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:47.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.091+0000 7f0dc7fff700 1 -- 172.21.15.189:0/2250070065 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc011830 con 0x7f0dd006ae40 2024-04-06T12:27:47.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.091+0000 7f0dc7fff700 1 -- 172.21.15.189:0/2250070065 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dcc011990 con 0x7f0dd006ae40 2024-04-06T12:27:47.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.091+0000 7f0dc7fff700 1 -- 172.21.15.189:0/2250070065 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc011d00 con 0x7f0dd006ae40 2024-04-06T12:27:47.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.091+0000 7f0dd763b700 1 -- 172.21.15.189:0/2250070065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dd006ae40 msgr2=0x7f0dd00b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:47.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.091+0000 7f0dd763b700 1 --2- 172.21.15.189:0/2250070065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dd006ae40 0x7f0dd00b3e20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc009840 tx=0x7f0dcc009b50 comp rx=0 tx=0).stop 2024-04-06T12:27:47.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.092+0000 7f0dd763b700 1 -- 172.21.15.189:0/2250070065 shutdown_connections 2024-04-06T12:27:47.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.092+0000 7f0dd763b700 1 --2- 172.21.15.189:0/2250070065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dd006ae40 0x7f0dd00b3e20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.092+0000 7f0dd763b700 1 -- 172.21.15.189:0/2250070065 >> 172.21.15.189:0/2250070065 conn(0x7f0dd00d00b0 msgr2=0x7f0dd00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:47.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.092+0000 7f0dd763b700 1 -- 172.21.15.189:0/2250070065 shutdown_connections 2024-04-06T12:27:47.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.092+0000 7f0dd763b700 1 -- 172.21.15.189:0/2250070065 wait complete. 2024-04-06T12:27:47.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.093+0000 7f0dd763b700 1 Processor -- start 2024-04-06T12:27:47.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.093+0000 7f0dd763b700 1 -- start start 2024-04-06T12:27:47.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dd763b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd006ae40 0x7f0dd0155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:47.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dd763b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dd015d7b0 0x7f0dd0152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:47.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dd763b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd015ddf0 0x7f0dd0144960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:47.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dd763b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd0154200 con 0x7f0dd006ae40 2024-04-06T12:27:47.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dd763b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd01532e0 con 0x7f0dd015d7b0 2024-04-06T12:27:47.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dd763b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd0153580 con 0x7f0dd015ddf0 2024-04-06T12:27:47.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dd53d7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd006ae40 0x7f0dd0155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:47.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dd53d7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd006ae40 0x7f0dd0155090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48660/0 (socket says 172.21.15.189:48660) 2024-04-06T12:27:47.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dd53d7700 1 -- 172.21.15.189:0/4111386899 learned_addr learned my addr 172.21.15.189:0/4111386899 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:47.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dd4bd6700 1 --2- 172.21.15.189:0/4111386899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dd015d7b0 0x7f0dd0152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:47.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dd5bd8700 1 --2- 172.21.15.189:0/4111386899 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd015ddf0 0x7f0dd0144960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:47.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dd53d7700 1 -- 172.21.15.189:0/4111386899 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd015ddf0 msgr2=0x7f0dd0144960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:47.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dd53d7700 1 --2- 172.21.15.189:0/4111386899 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd015ddf0 0x7f0dd0144960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dd53d7700 1 -- 172.21.15.189:0/4111386899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dd015d7b0 msgr2=0x7f0dd0152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:47.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dd53d7700 1 --2- 172.21.15.189:0/4111386899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dd015d7b0 0x7f0dd0152da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dd53d7700 1 -- 172.21.15.189:0/4111386899 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dcc0094a0 con 0x7f0dd006ae40 2024-04-06T12:27:47.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dd53d7700 1 --2- 172.21.15.189:0/4111386899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd006ae40 0x7f0dd0155090 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000c700 tx=0x7f0dc000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:47.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dc5ffb700 1 -- 172.21.15.189:0/4111386899 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0005d70 con 0x7f0dd006ae40 2024-04-06T12:27:47.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dc5ffb700 1 -- 172.21.15.189:0/4111386899 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc00070d0 con 0x7f0dd006ae40 2024-04-06T12:27:47.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dd763b700 1 -- 172.21.15.189:0/4111386899 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd0153860 con 0x7f0dd006ae40 2024-04-06T12:27:47.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.094+0000 7f0dc5ffb700 1 -- 172.21.15.189:0/4111386899 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0007480 con 0x7f0dd006ae40 2024-04-06T12:27:47.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.095+0000 7f0dd763b700 1 -- 172.21.15.189:0/4111386899 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd0153e20 con 0x7f0dd006ae40 2024-04-06T12:27:47.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.095+0000 7f0dd763b700 1 -- 172.21.15.189:0/4111386899 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd0154340 con 0x7f0dd006ae40 2024-04-06T12:27:47.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.096+0000 7f0dc5ffb700 1 -- 172.21.15.189:0/4111386899 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 38) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f0dc00075e0 con 0x7f0dd006ae40 2024-04-06T12:27:47.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.096+0000 7f0dc5ffb700 1 -- 172.21.15.189:0/4111386899 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6631+0+0 (secure 0 0 0) 0x7f0dc0088610 con 0x7f0dd006ae40 2024-04-06T12:27:47.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.101+0000 7f0dc5ffb700 1 -- 172.21.15.189:0/4111386899 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0dc0053820 con 0x7f0dd006ae40 2024-04-06T12:27:47.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.339+0000 7f0dd763b700 1 -- 172.21.15.189:0/4111386899 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0dd0067bb0 con 0x7f0dd006ae40 2024-04-06T12:27:47.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.339+0000 7f0dc5ffb700 1 -- 172.21.15.189:0/4111386899 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4747 (secure 0 0 0) 0x7f0dc0056e40 con 0x7f0dd006ae40 2024-04-06T12:27:47.337 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:47.337 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5762,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/2827078766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2827078766},{"type":"v1","addr":"172.21.15.192:6833","nonce":2827078766}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5765,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1558111999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1558111999},{"type":"v1","addr":"172.21.15.192:6835","nonce":1558111999}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3448511259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3448511259},{"type":"v1","addr":"172.21.15.189:6837","nonce":3448511259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:27:21.031012+0000","modified":"2024-04-06T12:27:24.038778+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/1085319880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1085319880},{"type":"v1","addr":"172.21.15.189:6836","nonce":1085319880}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-06T12:27:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.341+0000 7f0dd763b700 1 -- 172.21.15.189:0/4111386899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd006ae40 msgr2=0x7f0dd0155090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.341+0000 7f0dd763b700 1 --2- 172.21.15.189:0/4111386899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd006ae40 0x7f0dd0155090 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000c700 tx=0x7f0dc000ca10 comp rx=0 tx=0).stop 2024-04-06T12:27:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.341+0000 7f0dd763b700 1 -- 172.21.15.189:0/4111386899 shutdown_connections 2024-04-06T12:27:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.341+0000 7f0dd763b700 1 --2- 172.21.15.189:0/4111386899 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd015ddf0 0x7f0dd0144960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.341+0000 7f0dd763b700 1 --2- 172.21.15.189:0/4111386899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd006ae40 0x7f0dd0155090 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.341+0000 7f0dd763b700 1 --2- 172.21.15.189:0/4111386899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dd015d7b0 0x7f0dd0152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.341+0000 7f0dd763b700 1 -- 172.21.15.189:0/4111386899 >> 172.21.15.189:0/4111386899 conn(0x7f0dd00d00b0 msgr2=0x7f0dd00adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.341+0000 7f0dd763b700 1 -- 172.21.15.189:0/4111386899 shutdown_connections 2024-04-06T12:27:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.341+0000 7f0dd763b700 1 -- 172.21.15.189:0/4111386899 wait complete. 2024-04-06T12:27:47.340 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 31 2024-04-06T12:27:47.351 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:27:47.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.475+0000 7f3ed28ce700 1 Processor -- start 2024-04-06T12:27:47.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.475+0000 7f3ed28ce700 1 -- start start 2024-04-06T12:27:47.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.476+0000 7f3ed28ce700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc15a550 0x7f3ecc0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:47.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.476+0000 7f3ed28ce700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3ecc0b4d70 0x7f3ecc0b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:47.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.476+0000 7f3ed28ce700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ecc0cdbe0 con 0x7f3ecc0cf900 2024-04-06T12:27:47.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.476+0000 7f3ed28ce700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ecc15a940 con 0x7f3ecc0b4d70 2024-04-06T12:27:47.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.476+0000 7f3ed28ce700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc15ac00 con 0x7f3ecc15a550 2024-04-06T12:27:47.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.476+0000 7f3ed0e6b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3ecc0b4d70 0x7f3ecc0b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:47.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.476+0000 7f3ed0e6b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3ecc0b4d70 0x7f3ecc0b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48670/0 (socket says 172.21.15.189:48670) 2024-04-06T12:27:47.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.476+0000 7f3ecb7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3ecc0cf900 0x7f3ecc0b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44422/0 (socket says 172.21.15.189:44422) 2024-04-06T12:27:47.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.476+0000 7f3ed0e6b700 1 -- 172.21.15.189:0/1157099927 learned_addr learned my addr 172.21.15.189:0/1157099927 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:47.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.476+0000 7f3ecbfff700 1 --2- 172.21.15.189:0/1157099927 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc15a550 0x7f3ecc0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:47.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.476+0000 7f3ed0e6b700 1 -- 172.21.15.189:0/1157099927 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc15a550 msgr2=0x7f3ecc0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:47.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.476+0000 7f3ed0e6b700 1 --2- 172.21.15.189:0/1157099927 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc15a550 0x7f3ecc0cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.476+0000 7f3ed0e6b700 1 -- 172.21.15.189:0/1157099927 >> v1:172.21.15.189:6789/0 conn(0x7f3ecc0cf900 legacy=0x7f3ecc0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:47.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.476+0000 7f3ed0e6b700 1 -- 172.21.15.189:0/1157099927 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ecc15aee0 con 0x7f3ecc0b4d70 2024-04-06T12:27:47.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.476+0000 7f3ecaffd700 1 -- 172.21.15.189:0/1157099927 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923446922 0 0) 0x7f3ecc0cdbe0 con 0x7f3ecc0cf900 2024-04-06T12:27:47.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.476+0000 7f3ecbfff700 1 --2- 172.21.15.189:0/1157099927 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc15a550 0x7f3ecc0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:47.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.476+0000 7f3ed0e6b700 1 --2- 172.21.15.189:0/1157099927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ecc0b4d70 0x7f3ecc0b2fc0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f3ec400d150 tx=0x7f3ec400d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f33e19bd172cf39 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:47.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.477+0000 7f3ecaffd700 1 -- 172.21.15.189:0/1157099927 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4010070 con 0x7f3ecc0b4d70 2024-04-06T12:27:47.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.477+0000 7f3ecaffd700 1 -- 172.21.15.189:0/1157099927 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec400ddc0 con 0x7f3ecc0b4d70 2024-04-06T12:27:47.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.477+0000 7f3ecaffd700 1 -- 172.21.15.189:0/1157099927 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec400b310 con 0x7f3ecc0b4d70 2024-04-06T12:27:47.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.477+0000 7f3ed28ce700 1 -- 172.21.15.189:0/1157099927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ecc0b4d70 msgr2=0x7f3ecc0b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:47.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.477+0000 7f3ed28ce700 1 --2- 172.21.15.189:0/1157099927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ecc0b4d70 0x7f3ecc0b2fc0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f3ec400d150 tx=0x7f3ec400d460 comp rx=0 tx=0).stop 2024-04-06T12:27:47.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.477+0000 7f3ed28ce700 1 -- 172.21.15.189:0/1157099927 shutdown_connections 2024-04-06T12:27:47.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.477+0000 7f3ed28ce700 1 --2- 172.21.15.189:0/1157099927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ecc0b4d70 0x7f3ecc0b2fc0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.477+0000 7f3ed28ce700 1 --2- 172.21.15.189:0/1157099927 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc15a550 0x7f3ecc0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.477+0000 7f3ed28ce700 1 -- 172.21.15.189:0/1157099927 >> 172.21.15.189:0/1157099927 conn(0x7f3ecc0d00b0 msgr2=0x7f3ecc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:47.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.477+0000 7f3ed28ce700 1 -- 172.21.15.189:0/1157099927 shutdown_connections 2024-04-06T12:27:47.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.477+0000 7f3ed28ce700 1 -- 172.21.15.189:0/1157099927 wait complete. 2024-04-06T12:27:47.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.478+0000 7f3ed28ce700 1 Processor -- start 2024-04-06T12:27:47.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.479+0000 7f3ed28ce700 1 -- start start 2024-04-06T12:27:47.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.479+0000 7f3ed28ce700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc0b4d70 0x7f3ecc0af730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:47.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.479+0000 7f3ed28ce700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ecc0cf900 0x7f3ecc14fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:47.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.479+0000 7f3ed28ce700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ecc1f23b0 0x7f3ecc155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:47.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.479+0000 7f3ed28ce700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc0afc70 con 0x7f3ecc1f23b0 2024-04-06T12:27:47.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.479+0000 7f3ed28ce700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc150080 con 0x7f3ecc0cf900 2024-04-06T12:27:47.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.479+0000 7f3ed28ce700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc0afea0 con 0x7f3ecc0b4d70 2024-04-06T12:27:47.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.479+0000 7f3ed0e6b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ecc1f23b0 0x7f3ecc155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:47.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.479+0000 7f3ed0e6b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ecc1f23b0 0x7f3ecc155950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48674/0 (socket says 172.21.15.189:48674) 2024-04-06T12:27:47.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.479+0000 7f3ed0e6b700 1 -- 172.21.15.189:0/2555377341 learned_addr learned my addr 172.21.15.189:0/2555377341 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:47.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.480+0000 7f3ecbfff700 1 --2- 172.21.15.189:0/2555377341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc0b4d70 0x7f3ecc0af730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:47.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.480+0000 7f3ecb7fe700 1 --2- 172.21.15.189:0/2555377341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ecc0cf900 0x7f3ecc14fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:47.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.480+0000 7f3ed0e6b700 1 -- 172.21.15.189:0/2555377341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc0b4d70 msgr2=0x7f3ecc0af730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:47.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.480+0000 7f3ed0e6b700 1 --2- 172.21.15.189:0/2555377341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc0b4d70 0x7f3ecc0af730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.480+0000 7f3ed0e6b700 1 -- 172.21.15.189:0/2555377341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ecc0cf900 msgr2=0x7f3ecc14fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:47.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.480+0000 7f3ed0e6b700 1 --2- 172.21.15.189:0/2555377341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ecc0cf900 0x7f3ecc14fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.480+0000 7f3ed0e6b700 1 -- 172.21.15.189:0/2555377341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec400bd40 con 0x7f3ecc1f23b0 2024-04-06T12:27:47.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.480+0000 7f3ed0e6b700 1 --2- 172.21.15.189:0/2555377341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ecc1f23b0 0x7f3ecc155950 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc06c820 tx=0x7f3ec4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:47.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.480+0000 7f3ec8ff9700 1 -- 172.21.15.189:0/2555377341 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec400bbc0 con 0x7f3ecc1f23b0 2024-04-06T12:27:47.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.480+0000 7f3ec8ff9700 1 -- 172.21.15.189:0/2555377341 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec400b4c0 con 0x7f3ecc1f23b0 2024-04-06T12:27:47.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.480+0000 7f3ed28ce700 1 -- 172.21.15.189:0/2555377341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ecc0b0180 con 0x7f3ecc1f23b0 2024-04-06T12:27:47.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.480+0000 7f3ec8ff9700 1 -- 172.21.15.189:0/2555377341 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec400ae30 con 0x7f3ecc1f23b0 2024-04-06T12:27:47.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.480+0000 7f3ed28ce700 1 -- 172.21.15.189:0/2555377341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ecc0b0740 con 0x7f3ecc1f23b0 2024-04-06T12:27:47.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.481+0000 7f3ed28ce700 1 -- 172.21.15.189:0/2555377341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ecc067bb0 con 0x7f3ecc1f23b0 2024-04-06T12:27:47.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.481+0000 7f3ec8ff9700 1 -- 172.21.15.189:0/2555377341 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 38) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f3ec4004040 con 0x7f3ecc1f23b0 2024-04-06T12:27:47.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.482+0000 7f3ec8ff9700 1 -- 172.21.15.189:0/2555377341 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6631+0+0 (secure 0 0 0) 0x7f3ec408c8e0 con 0x7f3ecc1f23b0 2024-04-06T12:27:47.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.486+0000 7f3ec8ff9700 1 -- 172.21.15.189:0/2555377341 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3ec4057b70 con 0x7f3ecc1f23b0 2024-04-06T12:27:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.683+0000 7f3ed28ce700 1 -- 172.21.15.189:0/2555377341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ecc074990 con 0x7f3ecc1f23b0 2024-04-06T12:27:47.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.684+0000 7f3ec8ff9700 1 -- 172.21.15.189:0/2555377341 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+17171 (secure 0 0 0) 0x7f3ec405b190 con 0x7f3ecc1f23b0 2024-04-06T12:27:47.682 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:47.682 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":86,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:27:46.841130+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:27:19.575213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-06T12:27:19.990714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/3780401759":"2024-04-07T12:27:46.841097+0000","172.21.15.189:0/2438652477":"2024-04-07T12:27:46.841097+0000","172.21.15.189:0/3736136234":"2024-04-07T12:27:46.841097+0000","172.21.15.189:0/3208709895":"2024-04-07T12:27:46.841097+0000","172.21.15.189:0/615870143":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/1180713047":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/2789663450":"2024-04-07T12:27:46.841097+0000","172.21.15.189:0/2430686509":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/535246627":"2024-04-07T12:27:39.087614+0000","172.21.15.189:6832/47128":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/2829807377":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/1423746639":"2024-04-07T12:27:46.841097+0000","172.21.15.189:6833/47128":"2024-04-07T12:27:39.087614+0000","172.21.15.189:0/4012548359":"2024-04-07T12:27:39.087614+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:27:47.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.686+0000 7f3ed28ce700 1 -- 172.21.15.189:0/2555377341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ecc1f23b0 msgr2=0x7f3ecc155950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:47.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.687+0000 7f3ed28ce700 1 --2- 172.21.15.189:0/2555377341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ecc1f23b0 0x7f3ecc155950 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc06c820 tx=0x7f3ec4004ad0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.687+0000 7f3ed28ce700 1 -- 172.21.15.189:0/2555377341 shutdown_connections 2024-04-06T12:27:47.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.687+0000 7f3ed28ce700 1 --2- 172.21.15.189:0/2555377341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ecc1f23b0 0x7f3ecc155950 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.687+0000 7f3ed28ce700 1 --2- 172.21.15.189:0/2555377341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc0b4d70 0x7f3ecc0af730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.687+0000 7f3ed28ce700 1 --2- 172.21.15.189:0/2555377341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ecc0cf900 0x7f3ecc14fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.687+0000 7f3ed28ce700 1 -- 172.21.15.189:0/2555377341 >> 172.21.15.189:0/2555377341 conn(0x7f3ecc0d00b0 msgr2=0x7f3ecc152e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:47.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.687+0000 7f3ed28ce700 1 -- 172.21.15.189:0/2555377341 shutdown_connections 2024-04-06T12:27:47.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.687+0000 7f3ed28ce700 1 -- 172.21.15.189:0/2555377341 wait complete. 2024-04-06T12:27:47.696 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-06T12:27:47.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.823+0000 7f96896ea700 1 Processor -- start 2024-04-06T12:27:47.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.823+0000 7f96896ea700 1 -- start start 2024-04-06T12:27:47.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.824+0000 7f96896ea700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f968415d590 0x7f96840b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:47.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.824+0000 7f96896ea700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96840b5cc0 con 0x7f96840d3890 2024-04-06T12:27:47.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.824+0000 7f96896ea700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96840d1500 con 0x7f96840b4ed0 2024-04-06T12:27:47.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.824+0000 7f96896ea700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96840d17e0 con 0x7f968415d590 2024-04-06T12:27:47.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.824+0000 7f96837fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f96840d3890 0x7f96840b6c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44438/0 (socket says 172.21.15.189:44438) 2024-04-06T12:27:47.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.824+0000 7f96837fe700 1 -- 172.21.15.189:0/3490788475 learned_addr learned my addr 172.21.15.189:0/3490788475 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:47.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f9681ffb700 1 -- 172.21.15.189:0/3490788475 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514165391 0 0) 0x7f96840b5cc0 con 0x7f96840d3890 2024-04-06T12:27:47.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f9681ffb700 1 -- 172.21.15.189:0/3490788475 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f966c0035b0 con 0x7f96840d3890 2024-04-06T12:27:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f9682ffd700 1 --2- 172.21.15.189:0/3490788475 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f968415d590 0x7f96840b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f9681ffb700 1 -- 172.21.15.189:0/3490788475 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1340360329 0 0) 0x7f966c0035b0 con 0x7f96840d3890 2024-04-06T12:27:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f9681ffb700 1 -- 172.21.15.189:0/3490788475 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f968415d590 msgr2=0x7f96840b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f9681ffb700 1 --2- 172.21.15.189:0/3490788475 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f968415d590 0x7f96840b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f9681ffb700 1 -- 172.21.15.189:0/3490788475 >> v1:172.21.15.192:6789/0 conn(0x7f96840b4ed0 legacy=0x7f96840b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f9681ffb700 1 -- 172.21.15.189:0/3490788475 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96840d1ac0 con 0x7f96840d3890 2024-04-06T12:27:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f9681ffb700 1 -- 172.21.15.189:0/3490788475 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f967c0019f0 con 0x7f96840d3890 2024-04-06T12:27:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f9681ffb700 1 -- 172.21.15.189:0/3490788475 >> v1:172.21.15.189:6789/0 conn(0x7f96840d3890 legacy=0x7f96840b6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f9681ffb700 1 --2- 172.21.15.189:0/3490788475 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f968415d590 0x7f966c003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f9681ffb700 1 --2- 172.21.15.189:0/3490788475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f966c004060 0x7f966c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f9681ffb700 1 --2- 172.21.15.189:0/3490788475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f966c004d90 0x7f966c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f9681ffb700 1 -- 172.21.15.189:0/3490788475 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96840d17e0 con 0x7f966c004d90 2024-04-06T12:27:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f9681ffb700 1 -- 172.21.15.189:0/3490788475 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c00bbd0 con 0x7f966c004060 2024-04-06T12:27:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f9681ffb700 1 -- 172.21.15.189:0/3490788475 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966c00bd90 con 0x7f968415d590 2024-04-06T12:27:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f9682ffd700 1 --2- 172.21.15.189:0/3490788475 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f968415d590 0x7f966c003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:47.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f96837fe700 1 --2- 172.21.15.189:0/3490788475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f966c004d90 0x7f966c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:47.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f96827fc700 1 --2- 172.21.15.189:0/3490788475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f966c004060 0x7f966c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:47.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f96837fe700 1 -- 172.21.15.189:0/3490788475 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f968415d590 msgr2=0x7f966c003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:47.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f96837fe700 1 --2- 172.21.15.189:0/3490788475 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f968415d590 0x7f966c003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f96837fe700 1 -- 172.21.15.189:0/3490788475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f966c004060 msgr2=0x7f966c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:47.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f96837fe700 1 --2- 172.21.15.189:0/3490788475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f966c004060 0x7f966c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f96837fe700 1 -- 172.21.15.189:0/3490788475 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c0035b0 con 0x7f966c004d90 2024-04-06T12:27:47.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.825+0000 7f9682ffd700 1 --2- 172.21.15.189:0/3490788475 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f968415d590 0x7f966c003a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:47.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.826+0000 7f96837fe700 1 --2- 172.21.15.189:0/3490788475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f966c004d90 0x7f966c005210 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f967c00a830 tx=0x7f966c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d61b1b5942d8207b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:47.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.826+0000 7f9681ffb700 1 -- 172.21.15.189:0/3490788475 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c00a2f0 con 0x7f966c004d90 2024-04-06T12:27:47.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.826+0000 7f9681ffb700 1 -- 172.21.15.189:0/3490788475 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967c012620 con 0x7f966c004d90 2024-04-06T12:27:47.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.826+0000 7f9681ffb700 1 -- 172.21.15.189:0/3490788475 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c0129d0 con 0x7f966c004d90 2024-04-06T12:27:47.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.826+0000 7f96896ea700 1 -- 172.21.15.189:0/3490788475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f966c004d90 msgr2=0x7f966c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:47.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.826+0000 7f96896ea700 1 --2- 172.21.15.189:0/3490788475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f966c004d90 0x7f966c005210 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f967c00a830 tx=0x7f966c005880 comp rx=0 tx=0).stop 2024-04-06T12:27:47.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.826+0000 7f96837fe700 1 -- 172.21.15.189:0/3490788475 reap_dead start 2024-04-06T12:27:47.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.826+0000 7f96896ea700 1 -- 172.21.15.189:0/3490788475 shutdown_connections 2024-04-06T12:27:47.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.826+0000 7f96896ea700 1 -- 172.21.15.189:0/3490788475 >> 172.21.15.189:0/3490788475 conn(0x7f968406bbc0 msgr2=0x7f968406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:47.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.826+0000 7f96896ea700 1 -- 172.21.15.189:0/3490788475 shutdown_connections 2024-04-06T12:27:47.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.826+0000 7f96896ea700 1 -- 172.21.15.189:0/3490788475 wait complete. 2024-04-06T12:27:47.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.828+0000 7f96896ea700 1 Processor -- start 2024-04-06T12:27:47.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.828+0000 7f96896ea700 1 -- start start 2024-04-06T12:27:47.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.828+0000 7f96896ea700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96840c9290 0x7f9684154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:47.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.828+0000 7f96896ea700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96840cc2f0 0x7f96840b6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:47.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.828+0000 7f96896ea700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96841f6880 0x7f96840b6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:47.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.828+0000 7f96896ea700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96841f6e40 con 0x7f96841f6880 2024-04-06T12:27:47.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.828+0000 7f96896ea700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96841f7120 con 0x7f96840cc2f0 2024-04-06T12:27:47.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.828+0000 7f96896ea700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96841f7400 con 0x7f96840c9290 2024-04-06T12:27:47.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.829+0000 7f96837fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96841f6880 0x7f96840b6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:47.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.829+0000 7f96837fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96841f6880 0x7f96840b6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48682/0 (socket says 172.21.15.189:48682) 2024-04-06T12:27:47.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.829+0000 7f96837fe700 1 -- 172.21.15.189:0/2743192474 learned_addr learned my addr 172.21.15.189:0/2743192474 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:47.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.829+0000 7f9682ffd700 1 --2- 172.21.15.189:0/2743192474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96840c9290 0x7f9684154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:47.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.829+0000 7f96827fc700 1 --2- 172.21.15.189:0/2743192474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96840cc2f0 0x7f96840b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:47.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.829+0000 7f96837fe700 1 -- 172.21.15.189:0/2743192474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96840c9290 msgr2=0x7f9684154210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:47.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.829+0000 7f96837fe700 1 --2- 172.21.15.189:0/2743192474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96840c9290 0x7f9684154210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.829+0000 7f96837fe700 1 -- 172.21.15.189:0/2743192474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96840cc2f0 msgr2=0x7f96840b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:47.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.829+0000 7f96837fe700 1 --2- 172.21.15.189:0/2743192474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96840cc2f0 0x7f96840b6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:47.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.829+0000 7f96837fe700 1 -- 172.21.15.189:0/2743192474 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96841f33a0 con 0x7f96841f6880 2024-04-06T12:27:47.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.829+0000 7f96827fc700 1 --2- 172.21.15.189:0/2743192474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96840cc2f0 0x7f96840b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:27:47.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.829+0000 7f96837fe700 1 --2- 172.21.15.189:0/2743192474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96841f6880 0x7f96840b6ac0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f967c001cb0 tx=0x7f967c001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:47.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.829+0000 7f966b7fe700 1 -- 172.21.15.189:0/2743192474 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c0011c0 con 0x7f96841f6880 2024-04-06T12:27:47.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.829+0000 7f966b7fe700 1 -- 172.21.15.189:0/2743192474 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967c0017f0 con 0x7f96841f6880 2024-04-06T12:27:47.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.829+0000 7f966b7fe700 1 -- 172.21.15.189:0/2743192474 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c01e670 con 0x7f96841f6880 2024-04-06T12:27:47.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.829+0000 7f96896ea700 1 -- 172.21.15.189:0/2743192474 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96841609e0 con 0x7f96841f6880 2024-04-06T12:27:47.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.829+0000 7f96896ea700 1 -- 172.21.15.189:0/2743192474 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96841f76e0 con 0x7f96841f6880 2024-04-06T12:27:47.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.830+0000 7f96896ea700 1 -- 172.21.15.189:0/2743192474 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9684074990 con 0x7f96841f6880 2024-04-06T12:27:47.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.831+0000 7f966b7fe700 1 -- 172.21.15.189:0/2743192474 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 38) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f967c01e7d0 con 0x7f96841f6880 2024-04-06T12:27:47.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.831+0000 7f966b7fe700 1 -- 172.21.15.189:0/2743192474 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6631+0+0 (secure 0 0 0) 0x7f967c08af10 con 0x7f96841f6880 2024-04-06T12:27:47.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.835+0000 7f966b7fe700 1 -- 172.21.15.189:0/2743192474 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f967c0561a0 con 0x7f96841f6880 2024-04-06T12:27:47.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.853+0000 7f966b7fe700 1 -- 172.21.15.189:0/2743192474 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 39) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f967c00a410 con 0x7f96841f6880 2024-04-06T12:27:47.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.854+0000 7f966b7fe700 1 --2- 172.21.15.189:0/2743192474 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9670036150 0x7f9670072850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:47.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.855+0000 7f9682ffd700 1 --2- 172.21.15.189:0/2743192474 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9670036150 0x7f9670072850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:47.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:47.856+0000 7f9682ffd700 1 --2- 172.21.15.189:0/2743192474 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9670036150 0x7f9670072850 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9678009950 tx=0x7f9678008040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:48.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.082+0000 7f96896ea700 1 -- 172.21.15.189:0/2743192474 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f96840ce570 con 0x7f96841f6880 2024-04-06T12:27:48.080 INFO:tasks.ceph.mon.a.smithi189.stderr:2024-04-06T12:27:48.083+0000 7fa9fe1ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T12:27:48.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.096+0000 7f966b7fe700 1 -- 172.21.15.189:0/2743192474 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v32) v1 ==== 161+0+0 (secure 0 0 0) 0x7f967c06ffc0 con 0x7f96841f6880 2024-04-06T12:27:48.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.098+0000 7f96896ea700 1 -- 172.21.15.189:0/2743192474 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9670036150 msgr2=0x7f9670072850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:48.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.098+0000 7f96896ea700 1 --2- 172.21.15.189:0/2743192474 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9670036150 0x7f9670072850 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9678009950 tx=0x7f9678008040 comp rx=0 tx=0).stop 2024-04-06T12:27:48.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.098+0000 7f96896ea700 1 -- 172.21.15.189:0/2743192474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96841f6880 msgr2=0x7f96840b6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:48.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.098+0000 7f96896ea700 1 --2- 172.21.15.189:0/2743192474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96841f6880 0x7f96840b6ac0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f967c001cb0 tx=0x7f967c001d80 comp rx=0 tx=0).stop 2024-04-06T12:27:48.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.098+0000 7f96896ea700 1 -- 172.21.15.189:0/2743192474 shutdown_connections 2024-04-06T12:27:48.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.098+0000 7f96896ea700 1 --2- 172.21.15.189:0/2743192474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96841f6880 0x7f96840b6ac0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:48.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.098+0000 7f96896ea700 1 --2- 172.21.15.189:0/2743192474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96840c9290 0x7f9684154210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:48.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.098+0000 7f96896ea700 1 --2- 172.21.15.189:0/2743192474 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9670036150 0x7f9670072850 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:48.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.098+0000 7f96896ea700 1 --2- 172.21.15.189:0/2743192474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96840cc2f0 0x7f96840b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:48.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.098+0000 7f96896ea700 1 -- 172.21.15.189:0/2743192474 >> 172.21.15.189:0/2743192474 conn(0x7f968406bbc0 msgr2=0x7f968406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:48.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.098+0000 7f96896ea700 1 -- 172.21.15.189:0/2743192474 shutdown_connections 2024-04-06T12:27:48.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.098+0000 7f96896ea700 1 -- 172.21.15.189:0/2743192474 wait complete. 2024-04-06T12:27:48.096 INFO:tasks.ceph.mds.c.smithi189.stderr: -5321> 2024-04-06T12:27:22.611+0000 7f63badbb700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T12:27:48.097 INFO:teuthology.orchestra.run.smithi189.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-06T12:27:48.108 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-06T12:27:48.126 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T12:27:48.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.237+0000 7effd4ded700 1 Processor -- start 2024-04-06T12:27:48.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.237+0000 7effd4ded700 1 -- start start 2024-04-06T12:27:48.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effd4ded700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effd00b4ed0 0x7effd00b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:48.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effd4ded700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7effd00d2880 0x7effd00d2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:48.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effd4ded700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd00b5cc0 con 0x7effd015d590 2024-04-06T12:27:48.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effd4ded700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effd00b5890 con 0x7effd00d2880 2024-04-06T12:27:48.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effd4ded700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd00d1500 con 0x7effd00b4ed0 2024-04-06T12:27:48.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effce59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7effd015d590 0x7effd00b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44450/0 (socket says 172.21.15.189:44450) 2024-04-06T12:27:48.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effce59c700 1 -- 172.21.15.189:0/2635063870 learned_addr learned my addr 172.21.15.189:0/2635063870 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:48.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effcd59a700 1 -- 172.21.15.189:0/2635063870 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255227591 0 0) 0x7effd00b5cc0 con 0x7effd015d590 2024-04-06T12:27:48.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effcd59a700 1 -- 172.21.15.189:0/2635063870 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effb80035b0 con 0x7effd015d590 2024-04-06T12:27:48.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effcdd9b700 1 --2- 172.21.15.189:0/2635063870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effd00b4ed0 0x7effd00b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:48.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effced9d700 1 --2- 172.21.15.189:0/2635063870 >> v2:172.21.15.192:3300/0 conn(0x7effd00d2880 0x7effd00d2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:48.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effcd59a700 1 -- 172.21.15.189:0/2635063870 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3739892573 0 0) 0x7effb80035b0 con 0x7effd015d590 2024-04-06T12:27:48.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effcd59a700 1 -- 172.21.15.189:0/2635063870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effd00b4ed0 msgr2=0x7effd00b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:48.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effcd59a700 1 --2- 172.21.15.189:0/2635063870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effd00b4ed0 0x7effd00b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:48.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effcd59a700 1 -- 172.21.15.189:0/2635063870 >> v2:172.21.15.192:3300/0 conn(0x7effd00d2880 msgr2=0x7effd00d2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:48.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effcd59a700 1 --2- 172.21.15.189:0/2635063870 >> v2:172.21.15.192:3300/0 conn(0x7effd00d2880 0x7effd00d2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:48.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effcd59a700 1 -- 172.21.15.189:0/2635063870 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd00d17e0 con 0x7effd015d590 2024-04-06T12:27:48.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effcd59a700 1 -- 172.21.15.189:0/2635063870 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7effc0001670 con 0x7effd015d590 2024-04-06T12:27:48.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effcd59a700 1 -- 172.21.15.189:0/2635063870 >> v1:172.21.15.189:6789/0 conn(0x7effd015d590 legacy=0x7effd00b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:48.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effcd59a700 1 --2- 172.21.15.189:0/2635063870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effb8003a00 0x7effb8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:48.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effcd59a700 1 --2- 172.21.15.189:0/2635063870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effd00b4ed0 0x7effb8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:48.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effcd59a700 1 --2- 172.21.15.189:0/2635063870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effb8004d90 0x7effb8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:48.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effcd59a700 1 -- 172.21.15.189:0/2635063870 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd00b5890 con 0x7effb8004d90 2024-04-06T12:27:48.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effcd59a700 1 -- 172.21.15.189:0/2635063870 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd00d1500 con 0x7effb8003a00 2024-04-06T12:27:48.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.238+0000 7effcd59a700 1 -- 172.21.15.189:0/2635063870 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb800ba10 con 0x7effd00b4ed0 2024-04-06T12:27:48.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.239+0000 7effce59c700 1 --2- 172.21.15.189:0/2635063870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effb8003a00 0x7effb8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:48.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.239+0000 7effced9d700 1 --2- 172.21.15.189:0/2635063870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effb8004d90 0x7effb8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:48.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.239+0000 7effcdd9b700 1 --2- 172.21.15.189:0/2635063870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effd00b4ed0 0x7effb8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:48.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.239+0000 7effced9d700 1 -- 172.21.15.189:0/2635063870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effd00b4ed0 msgr2=0x7effb8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:48.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.239+0000 7effced9d700 1 --2- 172.21.15.189:0/2635063870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effd00b4ed0 0x7effb8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:48.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.239+0000 7effced9d700 1 -- 172.21.15.189:0/2635063870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effb8003a00 msgr2=0x7effb8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:48.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.239+0000 7effced9d700 1 --2- 172.21.15.189:0/2635063870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effb8003a00 0x7effb8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:48.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.239+0000 7effced9d700 1 -- 172.21.15.189:0/2635063870 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb80035b0 con 0x7effb8004d90 2024-04-06T12:27:48.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.239+0000 7effced9d700 1 --2- 172.21.15.189:0/2635063870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effb8004d90 0x7effb8005210 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7effc800e170 tx=0x7effb8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32f23e76b029c68 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:48.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.239+0000 7effcd59a700 1 -- 172.21.15.189:0/2635063870 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc800ec90 con 0x7effb8004d90 2024-04-06T12:27:48.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.239+0000 7effcd59a700 1 -- 172.21.15.189:0/2635063870 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc800f040 con 0x7effb8004d90 2024-04-06T12:27:48.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.239+0000 7effcd59a700 1 -- 172.21.15.189:0/2635063870 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8014480 con 0x7effb8004d90 2024-04-06T12:27:48.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.240+0000 7effd4ded700 1 -- 172.21.15.189:0/2635063870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effb8004d90 msgr2=0x7effb8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:48.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.240+0000 7effd4ded700 1 --2- 172.21.15.189:0/2635063870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effb8004d90 0x7effb8005210 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7effc800e170 tx=0x7effb8005880 comp rx=0 tx=0).stop 2024-04-06T12:27:48.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.240+0000 7effced9d700 1 -- 172.21.15.189:0/2635063870 reap_dead start 2024-04-06T12:27:48.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.240+0000 7effd4ded700 1 -- 172.21.15.189:0/2635063870 shutdown_connections 2024-04-06T12:27:48.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.240+0000 7effd4ded700 1 -- 172.21.15.189:0/2635063870 >> 172.21.15.189:0/2635063870 conn(0x7effd006bbc0 msgr2=0x7effd006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:48.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.240+0000 7effd4ded700 1 -- 172.21.15.189:0/2635063870 shutdown_connections 2024-04-06T12:27:48.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.240+0000 7effd4ded700 1 -- 172.21.15.189:0/2635063870 wait complete. 2024-04-06T12:27:48.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.241+0000 7effd4ded700 1 Processor -- start 2024-04-06T12:27:48.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.241+0000 7effd4ded700 1 -- start start 2024-04-06T12:27:48.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.242+0000 7effd4ded700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effb8004d90 0x7effd00cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:48.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.242+0000 7effd4ded700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effd015fd50 0x7effd015af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:48.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.242+0000 7effd4ded700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effd01602c0 0x7effd00cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:48.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.242+0000 7effd4ded700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd00cc190 con 0x7effd01602c0 2024-04-06T12:27:48.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.242+0000 7effd4ded700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd00cb8a0 con 0x7effb8004d90 2024-04-06T12:27:48.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.242+0000 7effd4ded700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd015bad0 con 0x7effd015fd50 2024-04-06T12:27:48.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.242+0000 7effced9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effd01602c0 0x7effd00cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:48.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.242+0000 7effced9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effd01602c0 0x7effd00cc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48702/0 (socket says 172.21.15.189:48702) 2024-04-06T12:27:48.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.242+0000 7effced9d700 1 -- 172.21.15.189:0/3067846359 learned_addr learned my addr 172.21.15.189:0/3067846359 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:48.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.242+0000 7effcdd9b700 1 --2- 172.21.15.189:0/3067846359 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effd015fd50 0x7effd015af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:48.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.242+0000 7effce59c700 1 --2- 172.21.15.189:0/3067846359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effb8004d90 0x7effd00cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:48.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.242+0000 7effced9d700 1 -- 172.21.15.189:0/3067846359 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effd015fd50 msgr2=0x7effd015af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:48.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.242+0000 7effced9d700 1 --2- 172.21.15.189:0/3067846359 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effd015fd50 0x7effd015af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:48.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.242+0000 7effced9d700 1 -- 172.21.15.189:0/3067846359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effb8004d90 msgr2=0x7effd00cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:48.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.242+0000 7effced9d700 1 --2- 172.21.15.189:0/3067846359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effb8004d90 0x7effd00cbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:48.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.242+0000 7effced9d700 1 -- 172.21.15.189:0/3067846359 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd015bdb0 con 0x7effd01602c0 2024-04-06T12:27:48.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.243+0000 7effced9d700 1 --2- 172.21.15.189:0/3067846359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effd01602c0 0x7effd00cc330 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7effc8016000 tx=0x7effc8004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:48.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.243+0000 7effbeffd700 1 -- 172.21.15.189:0/3067846359 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8007000 con 0x7effd01602c0 2024-04-06T12:27:48.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.243+0000 7effbeffd700 1 -- 172.21.15.189:0/3067846359 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc800ee80 con 0x7effd01602c0 2024-04-06T12:27:48.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.243+0000 7effd4ded700 1 -- 172.21.15.189:0/3067846359 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd015c090 con 0x7effd01602c0 2024-04-06T12:27:48.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.243+0000 7effbeffd700 1 -- 172.21.15.189:0/3067846359 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8003a20 con 0x7effd01602c0 2024-04-06T12:27:48.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.243+0000 7effd4ded700 1 -- 172.21.15.189:0/3067846359 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd015f830 con 0x7effd01602c0 2024-04-06T12:27:48.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.244+0000 7effd4ded700 1 -- 172.21.15.189:0/3067846359 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd0074990 con 0x7effd01602c0 2024-04-06T12:27:48.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.244+0000 7effbeffd700 1 -- 172.21.15.189:0/3067846359 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 39) v1 ==== 88983+0+0 (secure 0 0 0) 0x7effc8003c00 con 0x7effd01602c0 2024-04-06T12:27:48.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.245+0000 7effbeffd700 1 --2- 172.21.15.189:0/3067846359 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7effb006aae0 0x7effb006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:48.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.245+0000 7effbeffd700 1 -- 172.21.15.189:0/3067846359 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6717+0+0 (secure 0 0 0) 0x7effc808a9f0 con 0x7effd01602c0 2024-04-06T12:27:48.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.245+0000 7effce59c700 1 --2- 172.21.15.189:0/3067846359 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7effb006aae0 0x7effb006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:48.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.245+0000 7effce59c700 1 --2- 172.21.15.189:0/3067846359 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7effb006aae0 0x7effb006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7effc0000c00 tx=0x7effc0009040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:48.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.250+0000 7effbeffd700 1 -- 172.21.15.189:0/3067846359 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7effc8055a50 con 0x7effd01602c0 2024-04-06T12:27:48.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:48.490+0000 7effd4ded700 1 -- 172.21.15.189:0/3067846359 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7effd015caa0 con 0x7effd01602c0 2024-04-06T12:27:49.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.122+0000 7effbeffd700 1 -- 172.21.15.189:0/3067846359 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 40) v1 ==== 89191+0+0 (secure 0 0 0) 0x7effc808cd50 con 0x7effd01602c0 2024-04-06T12:27:49.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.127+0000 7effbeffd700 1 -- 172.21.15.189:0/3067846359 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7effc806f190 con 0x7effd01602c0 2024-04-06T12:27:49.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.129+0000 7effd4ded700 1 -- 172.21.15.189:0/3067846359 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7effb006aae0 msgr2=0x7effb006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:49.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.129+0000 7effd4ded700 1 --2- 172.21.15.189:0/3067846359 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7effb006aae0 0x7effb006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7effc0000c00 tx=0x7effc0009040 comp rx=0 tx=0).stop 2024-04-06T12:27:49.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.129+0000 7effd4ded700 1 -- 172.21.15.189:0/3067846359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effd01602c0 msgr2=0x7effd00cc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:49.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.129+0000 7effd4ded700 1 --2- 172.21.15.189:0/3067846359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effd01602c0 0x7effd00cc330 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7effc8016000 tx=0x7effc8004b40 comp rx=0 tx=0).stop 2024-04-06T12:27:49.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.129+0000 7effd4ded700 1 -- 172.21.15.189:0/3067846359 shutdown_connections 2024-04-06T12:27:49.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.129+0000 7effd4ded700 1 --2- 172.21.15.189:0/3067846359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effd01602c0 0x7effd00cc330 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:49.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.129+0000 7effd4ded700 1 --2- 172.21.15.189:0/3067846359 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7effb006aae0 0x7effb006cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:49.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.129+0000 7effd4ded700 1 --2- 172.21.15.189:0/3067846359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effb8004d90 0x7effd00cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:49.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.129+0000 7effd4ded700 1 --2- 172.21.15.189:0/3067846359 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effd015fd50 0x7effd015af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:49.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.129+0000 7effd4ded700 1 -- 172.21.15.189:0/3067846359 >> 172.21.15.189:0/3067846359 conn(0x7effd006bbc0 msgr2=0x7effd00b33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:49.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.130+0000 7effd4ded700 1 -- 172.21.15.189:0/3067846359 shutdown_connections 2024-04-06T12:27:49.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.130+0000 7effd4ded700 1 -- 172.21.15.189:0/3067846359 wait complete. 2024-04-06T12:27:49.139 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-06T12:27:49.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.309+0000 7fbeb0289700 1 Processor -- start 2024-04-06T12:27:49.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.309+0000 7fbeb0289700 1 -- start start 2024-04-06T12:27:49.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.310+0000 7fbeb0289700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbea80cfa00 0x7fbea80cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:49.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.310+0000 7fbeb0289700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea80cbd80 con 0x7fbea80cb870 2024-04-06T12:27:49.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.310+0000 7fbeb0289700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea80cef20 con 0x7fbea80d11b0 2024-04-06T12:27:49.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.310+0000 7fbeb0289700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbea80ccdc0 con 0x7fbea80cfa00 2024-04-06T12:27:49.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.310+0000 7fbead824700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbea80cb870 0x7fbea815c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33848/0 (socket says 172.21.15.189:33848) 2024-04-06T12:27:49.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.310+0000 7fbead824700 1 -- 172.21.15.189:0/1263252033 learned_addr learned my addr 172.21.15.189:0/1263252033 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:49.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbeae826700 1 --2- 172.21.15.189:0/1263252033 >> v2:172.21.15.192:3300/0 conn(0x7fbea80cfa00 0x7fbea80cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:49.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbead023700 1 -- 172.21.15.189:0/1263252033 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423974120 0 0) 0x7fbea80cbd80 con 0x7fbea80cb870 2024-04-06T12:27:49.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbead023700 1 -- 172.21.15.189:0/1263252033 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe940035b0 con 0x7fbea80cb870 2024-04-06T12:27:49.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbead023700 1 -- 172.21.15.189:0/1263252033 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 98115963 0 0) 0x7fbe940035b0 con 0x7fbea80cb870 2024-04-06T12:27:49.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbead023700 1 -- 172.21.15.189:0/1263252033 >> v2:172.21.15.192:3300/0 conn(0x7fbea80cfa00 msgr2=0x7fbea80cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:49.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbead023700 1 --2- 172.21.15.189:0/1263252033 >> v2:172.21.15.192:3300/0 conn(0x7fbea80cfa00 0x7fbea80cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:49.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbead023700 1 -- 172.21.15.189:0/1263252033 >> v1:172.21.15.192:6789/0 conn(0x7fbea80d11b0 legacy=0x7fbea80d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:49.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbead023700 1 -- 172.21.15.189:0/1263252033 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea80cf200 con 0x7fbea80cb870 2024-04-06T12:27:49.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbead023700 1 -- 172.21.15.189:0/1263252033 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fbea4001670 con 0x7fbea80cb870 2024-04-06T12:27:49.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbead023700 1 -- 172.21.15.189:0/1263252033 >> v1:172.21.15.189:6789/0 conn(0x7fbea80cb870 legacy=0x7fbea815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:49.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbead023700 1 --2- 172.21.15.189:0/1263252033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe94003a00 0x7fbe94003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:49.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbead023700 1 --2- 172.21.15.189:0/1263252033 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe94004850 0x7fbe94004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:49.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbead023700 1 --2- 172.21.15.189:0/1263252033 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe94005210 0x7fbe9400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:49.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbead023700 1 -- 172.21.15.189:0/1263252033 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9400c180 con 0x7fbe94003a00 2024-04-06T12:27:49.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbead023700 1 -- 172.21.15.189:0/1263252033 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9400c2f0 con 0x7fbe94005210 2024-04-06T12:27:49.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbead023700 1 -- 172.21.15.189:0/1263252033 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9400c4b0 con 0x7fbe94004850 2024-04-06T12:27:49.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbeae025700 1 --2- 172.21.15.189:0/1263252033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe94003a00 0x7fbe94003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:49.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbead824700 1 --2- 172.21.15.189:0/1263252033 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe94004850 0x7fbe94004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:49.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbeae826700 1 --2- 172.21.15.189:0/1263252033 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe94005210 0x7fbe9400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:49.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbeae025700 1 -- 172.21.15.189:0/1263252033 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe94004850 msgr2=0x7fbe94004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:49.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbeae025700 1 --2- 172.21.15.189:0/1263252033 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe94004850 0x7fbe94004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:49.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbeae025700 1 -- 172.21.15.189:0/1263252033 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe94005210 msgr2=0x7fbe9400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:49.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbeae025700 1 --2- 172.21.15.189:0/1263252033 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe94005210 0x7fbe9400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:49.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbeae826700 1 -- 172.21.15.189:0/1263252033 reap_dead start 2024-04-06T12:27:49.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.311+0000 7fbeae025700 1 -- 172.21.15.189:0/1263252033 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe940035b0 con 0x7fbe94003a00 2024-04-06T12:27:49.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.312+0000 7fbeae025700 1 --2- 172.21.15.189:0/1263252033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe94003a00 0x7fbe94003e20 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fbe980017c0 tx=0x7fbe94005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d226fbf3b235d6f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:49.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.312+0000 7fbead023700 1 -- 172.21.15.189:0/1263252033 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe980052d0 con 0x7fbe94003a00 2024-04-06T12:27:49.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.312+0000 7fbead023700 1 -- 172.21.15.189:0/1263252033 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe98005450 con 0x7fbe94003a00 2024-04-06T12:27:49.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.312+0000 7fbead023700 1 -- 172.21.15.189:0/1263252033 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe980038c0 con 0x7fbe94003a00 2024-04-06T12:27:49.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.312+0000 7fbeb0289700 1 -- 172.21.15.189:0/1263252033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe94003a00 msgr2=0x7fbe94003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:49.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.312+0000 7fbeb0289700 1 --2- 172.21.15.189:0/1263252033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe94003a00 0x7fbe94003e20 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fbe980017c0 tx=0x7fbe94005880 comp rx=0 tx=0).stop 2024-04-06T12:27:49.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.312+0000 7fbeb0289700 1 -- 172.21.15.189:0/1263252033 shutdown_connections 2024-04-06T12:27:49.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.312+0000 7fbeb0289700 1 --2- 172.21.15.189:0/1263252033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe94003a00 0x7fbe94003e20 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:49.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.312+0000 7fbeb0289700 1 -- 172.21.15.189:0/1263252033 >> 172.21.15.189:0/1263252033 conn(0x7fbea80b2db0 msgr2=0x7fbea80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:49.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.312+0000 7fbeb0289700 1 -- 172.21.15.189:0/1263252033 shutdown_connections 2024-04-06T12:27:49.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.312+0000 7fbeb0289700 1 -- 172.21.15.189:0/1263252033 wait complete. 2024-04-06T12:27:49.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.313+0000 7fbeb0289700 1 Processor -- start 2024-04-06T12:27:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.314+0000 7fbeb0289700 1 -- start start 2024-04-06T12:27:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.314+0000 7fbeb0289700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe94003a00 0x7fbea814ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.314+0000 7fbeb0289700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbea80ca9e0 0x7fbea815a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.314+0000 7fbeb0289700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbea80cae60 0x7fbea815a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.314+0000 7fbeb0289700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea80ceb50 con 0x7fbe94003a00 2024-04-06T12:27:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.314+0000 7fbeb0289700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea81534b0 con 0x7fbea80cae60 2024-04-06T12:27:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.314+0000 7fbeb0289700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbea80ce2f0 con 0x7fbea80ca9e0 2024-04-06T12:27:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.314+0000 7fbeae025700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe94003a00 0x7fbea814ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.314+0000 7fbeae025700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe94003a00 0x7fbea814ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38816/0 (socket says 172.21.15.189:38816) 2024-04-06T12:27:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.314+0000 7fbeae025700 1 -- 172.21.15.189:0/1830606384 learned_addr learned my addr 172.21.15.189:0/1830606384 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.314+0000 7fbead824700 1 --2- 172.21.15.189:0/1830606384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbea80ca9e0 0x7fbea815a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.314+0000 7fbeae826700 1 --2- 172.21.15.189:0/1830606384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbea80cae60 0x7fbea815a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.315+0000 7fbeae025700 1 -- 172.21.15.189:0/1830606384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbea80ca9e0 msgr2=0x7fbea815a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.315+0000 7fbeae025700 1 --2- 172.21.15.189:0/1830606384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbea80ca9e0 0x7fbea815a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.315+0000 7fbeae025700 1 -- 172.21.15.189:0/1830606384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbea80cae60 msgr2=0x7fbea815a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.315+0000 7fbeae025700 1 --2- 172.21.15.189:0/1830606384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbea80cae60 0x7fbea815a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.315+0000 7fbeae025700 1 -- 172.21.15.189:0/1830606384 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea80ce590 con 0x7fbe94003a00 2024-04-06T12:27:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.315+0000 7fbeae826700 1 --2- 172.21.15.189:0/1830606384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbea80cae60 0x7fbea815a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.315+0000 7fbeae025700 1 --2- 172.21.15.189:0/1830606384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe94003a00 0x7fbea814ea20 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fbe98000c00 tx=0x7fbe98001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.315+0000 7fbe9effd700 1 -- 172.21.15.189:0/1830606384 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe98001290 con 0x7fbe94003a00 2024-04-06T12:27:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.315+0000 7fbe9effd700 1 -- 172.21.15.189:0/1830606384 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe98007040 con 0x7fbe94003a00 2024-04-06T12:27:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.315+0000 7fbeb0289700 1 -- 172.21.15.189:0/1830606384 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea80ce870 con 0x7fbe94003a00 2024-04-06T12:27:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.315+0000 7fbe9effd700 1 -- 172.21.15.189:0/1830606384 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe980073d0 con 0x7fbe94003a00 2024-04-06T12:27:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.315+0000 7fbeb0289700 1 -- 172.21.15.189:0/1830606384 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea80ca190 con 0x7fbe94003a00 2024-04-06T12:27:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.316+0000 7fbeb0289700 1 -- 172.21.15.189:0/1830606384 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea80c9d10 con 0x7fbe94003a00 2024-04-06T12:27:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.316+0000 7fbe9effd700 1 -- 172.21.15.189:0/1830606384 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 40) v1 ==== 89191+0+0 (secure 0 0 0) 0x7fbe98007530 con 0x7fbe94003a00 2024-04-06T12:27:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.317+0000 7fbe9effd700 1 --2- 172.21.15.189:0/1830606384 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbe8c06ad20 0x7fbe8c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.317+0000 7fbe9effd700 1 -- 172.21.15.189:0/1830606384 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6717+0+0 (secure 0 0 0) 0x7fbe9808a1c0 con 0x7fbe94003a00 2024-04-06T12:27:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.317+0000 7fbead824700 1 --2- 172.21.15.189:0/1830606384 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbe8c06ad20 0x7fbe8c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:49.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.317+0000 7fbead824700 1 --2- 172.21.15.189:0/1830606384 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbe8c06ad20 0x7fbe8c06d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbea4001c90 tx=0x7fbea4001b20 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:49.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.321+0000 7fbe9effd700 1 -- 172.21.15.189:0/1830606384 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbea80c9d10 con 0x7fbe94003a00 2024-04-06T12:27:49.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:49.514+0000 7fbeb0289700 1 -- 172.21.15.189:0/1830606384 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbea80c9d10 con 0x7fbe94003a00 2024-04-06T12:27:50.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.132+0000 7fbe9effd700 1 -- 172.21.15.189:0/1830606384 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v88) v1 ==== 181+0+0 (secure 0 0 0) 0x7fbea80c9d10 con 0x7fbe94003a00 2024-04-06T12:27:50.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.136+0000 7fbeb0289700 1 -- 172.21.15.189:0/1830606384 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbe8c06ad20 msgr2=0x7fbe8c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:50.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.136+0000 7fbeb0289700 1 --2- 172.21.15.189:0/1830606384 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbe8c06ad20 0x7fbe8c06d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbea4001c90 tx=0x7fbea4001b20 comp rx=0 tx=0).stop 2024-04-06T12:27:50.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.136+0000 7fbeb0289700 1 -- 172.21.15.189:0/1830606384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe94003a00 msgr2=0x7fbea814ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:50.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.136+0000 7fbeb0289700 1 --2- 172.21.15.189:0/1830606384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe94003a00 0x7fbea814ea20 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fbe98000c00 tx=0x7fbe98001120 comp rx=0 tx=0).stop 2024-04-06T12:27:50.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.136+0000 7fbeb0289700 1 -- 172.21.15.189:0/1830606384 shutdown_connections 2024-04-06T12:27:50.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.136+0000 7fbeb0289700 1 --2- 172.21.15.189:0/1830606384 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbe8c06ad20 0x7fbe8c06d1e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:50.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.136+0000 7fbeb0289700 1 --2- 172.21.15.189:0/1830606384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbea80cae60 0x7fbea815a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:50.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.136+0000 7fbeb0289700 1 --2- 172.21.15.189:0/1830606384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe94003a00 0x7fbea814ea20 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:50.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.136+0000 7fbeb0289700 1 --2- 172.21.15.189:0/1830606384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbea80ca9e0 0x7fbea815a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:50.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.136+0000 7fbeb0289700 1 -- 172.21.15.189:0/1830606384 >> 172.21.15.189:0/1830606384 conn(0x7fbea80b2db0 msgr2=0x7fbea80b6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:50.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.136+0000 7fbeb0289700 1 -- 172.21.15.189:0/1830606384 shutdown_connections 2024-04-06T12:27:50.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.136+0000 7fbeb0289700 1 -- 172.21.15.189:0/1830606384 wait complete. 2024-04-06T12:27:50.134 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' removed 2024-04-06T12:27:50.148 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-06T12:27:50.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.283+0000 7f82907d1700 1 Processor -- start 2024-04-06T12:27:50.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.283+0000 7f82907d1700 1 -- start start 2024-04-06T12:27:50.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.284+0000 7f82907d1700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f828815d7b0 0x7f82880b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:50.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.284+0000 7f82907d1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f828806a730 0x7f828806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:50.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.284+0000 7f82907d1700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82880b3990 con 0x7f828815ddf0 2024-04-06T12:27:50.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.284+0000 7f82907d1700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f828806b0f0 con 0x7f828815d7b0 2024-04-06T12:27:50.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.284+0000 7f82907d1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82880b3060 con 0x7f828806a730 2024-04-06T12:27:50.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.284+0000 7f828e56d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f828815d7b0 0x7f82880b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:50.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.284+0000 7f828dd6c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f828806a730 0x7f828806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:50.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.284+0000 7f828e56d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f828815d7b0 0x7f82880b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:36340/0 (socket says 172.21.15.189:36340) 2024-04-06T12:27:50.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.284+0000 7f828dd6c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f828806a730 0x7f828806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:59350/0 (socket says 172.21.15.189:59350) 2024-04-06T12:27:50.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.284+0000 7f828e56d700 1 -- 172.21.15.189:0/2575831159 learned_addr learned my addr 172.21.15.189:0/2575831159 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:50.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.284+0000 7f828ed6e700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f828815ddf0 0x7f828815e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:57600/0 (socket says 172.21.15.189:57600) 2024-04-06T12:27:50.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.284+0000 7f828e56d700 1 -- 172.21.15.189:0/2575831159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f828806a730 msgr2=0x7f828806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:50.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.284+0000 7f828e56d700 1 --2- 172.21.15.189:0/2575831159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f828806a730 0x7f828806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:50.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.284+0000 7f828e56d700 1 -- 172.21.15.189:0/2575831159 >> v1:172.21.15.192:6789/0 conn(0x7f828815ddf0 legacy=0x7f828815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:50.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.284+0000 7f828e56d700 1 -- 172.21.15.189:0/2575831159 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82880b3340 con 0x7f828815d7b0 2024-04-06T12:27:50.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.284+0000 7f828d56b700 1 -- 172.21.15.189:0/2575831159 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835312505 0 0) 0x7f82880b3990 con 0x7f828815ddf0 2024-04-06T12:27:50.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.284+0000 7f828dd6c700 1 --2- 172.21.15.189:0/2575831159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f828806a730 0x7f828806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:50.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.285+0000 7f828e56d700 1 --2- 172.21.15.189:0/2575831159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f828815d7b0 0x7f82880b3e20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f82840098f0 tx=0x7f8284009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f21b30f1e5fc71a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:50.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.285+0000 7f828d56b700 1 -- 172.21.15.189:0/2575831159 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284011830 con 0x7f828815d7b0 2024-04-06T12:27:50.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.285+0000 7f828d56b700 1 -- 172.21.15.189:0/2575831159 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8284011990 con 0x7f828815d7b0 2024-04-06T12:27:50.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.285+0000 7f828d56b700 1 -- 172.21.15.189:0/2575831159 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284011d20 con 0x7f828815d7b0 2024-04-06T12:27:50.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.285+0000 7f82907d1700 1 -- 172.21.15.189:0/2575831159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f828815d7b0 msgr2=0x7f82880b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:50.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.285+0000 7f82907d1700 1 --2- 172.21.15.189:0/2575831159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f828815d7b0 0x7f82880b3e20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f82840098f0 tx=0x7f8284009c00 comp rx=0 tx=0).stop 2024-04-06T12:27:50.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.285+0000 7f82907d1700 1 -- 172.21.15.189:0/2575831159 shutdown_connections 2024-04-06T12:27:50.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.286+0000 7f82907d1700 1 --2- 172.21.15.189:0/2575831159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f828815d7b0 0x7f82880b3e20 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:50.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.286+0000 7f82907d1700 1 --2- 172.21.15.189:0/2575831159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f828806a730 0x7f828806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:50.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.286+0000 7f82907d1700 1 -- 172.21.15.189:0/2575831159 >> 172.21.15.189:0/2575831159 conn(0x7f82880d00b0 msgr2=0x7f82880d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:50.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.286+0000 7f82907d1700 1 -- 172.21.15.189:0/2575831159 shutdown_connections 2024-04-06T12:27:50.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.286+0000 7f82907d1700 1 -- 172.21.15.189:0/2575831159 wait complete. 2024-04-06T12:27:50.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.287+0000 7f82907d1700 1 Processor -- start 2024-04-06T12:27:50.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.287+0000 7f82907d1700 1 -- start start 2024-04-06T12:27:50.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.287+0000 7f82907d1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f828806a730 0x7f82880b1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:50.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.287+0000 7f82907d1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f828815d7b0 0x7f82880ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:50.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.287+0000 7f82907d1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f828815ddf0 0x7f82881f2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:50.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.287+0000 7f82907d1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828815a810 con 0x7f828815ddf0 2024-04-06T12:27:50.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.287+0000 7f82907d1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82880cedf0 con 0x7f828815d7b0 2024-04-06T12:27:50.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.287+0000 7f82907d1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82880cf630 con 0x7f828806a730 2024-04-06T12:27:50.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.288+0000 7f828ed6e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f828815ddf0 0x7f82881f2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:50.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.288+0000 7f828ed6e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f828815ddf0 0x7f82881f2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38830/0 (socket says 172.21.15.189:38830) 2024-04-06T12:27:50.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.288+0000 7f828ed6e700 1 -- 172.21.15.189:0/654961506 learned_addr learned my addr 172.21.15.189:0/654961506 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:50.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.288+0000 7f828e56d700 1 --2- 172.21.15.189:0/654961506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f828806a730 0x7f82880b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:50.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.288+0000 7f828dd6c700 1 --2- 172.21.15.189:0/654961506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f828815d7b0 0x7f82880ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:50.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.288+0000 7f828ed6e700 1 -- 172.21.15.189:0/654961506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f828806a730 msgr2=0x7f82880b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:50.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.288+0000 7f828ed6e700 1 --2- 172.21.15.189:0/654961506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f828806a730 0x7f82880b1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:50.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.288+0000 7f828ed6e700 1 -- 172.21.15.189:0/654961506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f828815d7b0 msgr2=0x7f82880ce8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:50.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.288+0000 7f828ed6e700 1 --2- 172.21.15.189:0/654961506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f828815d7b0 0x7f82880ce8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:50.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.288+0000 7f828ed6e700 1 -- 172.21.15.189:0/654961506 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82780094a0 con 0x7f828815ddf0 2024-04-06T12:27:50.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.288+0000 7f828dd6c700 1 --2- 172.21.15.189:0/654961506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f828815d7b0 0x7f82880ce8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:27:50.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.288+0000 7f828ed6e700 1 --2- 172.21.15.189:0/654961506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f828815ddf0 0x7f82881f2d00 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f828000cb90 tx=0x7f828000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:50.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.288+0000 7f827effd700 1 -- 172.21.15.189:0/654961506 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828000ab30 con 0x7f828815ddf0 2024-04-06T12:27:50.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.288+0000 7f827effd700 1 -- 172.21.15.189:0/654961506 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828000ac90 con 0x7f828815ddf0 2024-04-06T12:27:50.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.288+0000 7f82907d1700 1 -- 172.21.15.189:0/654961506 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8284009550 con 0x7f828815ddf0 2024-04-06T12:27:50.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.288+0000 7f827effd700 1 -- 172.21.15.189:0/654961506 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82800070d0 con 0x7f828815ddf0 2024-04-06T12:27:50.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.288+0000 7f82907d1700 1 -- 172.21.15.189:0/654961506 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82880cf350 con 0x7f828815ddf0 2024-04-06T12:27:50.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.289+0000 7f82907d1700 1 -- 172.21.15.189:0/654961506 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f828815a9c0 con 0x7f828815ddf0 2024-04-06T12:27:50.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.290+0000 7f827effd700 1 -- 172.21.15.189:0/654961506 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 40) v1 ==== 89191+0+0 (secure 0 0 0) 0x7f8280007230 con 0x7f828815ddf0 2024-04-06T12:27:50.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.290+0000 7f827effd700 1 --2- 172.21.15.189:0/654961506 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f826c06ad20 0x7f826c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:50.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.290+0000 7f827effd700 1 -- 172.21.15.189:0/654961506 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6279+0+0 (secure 0 0 0) 0x7f82800893b0 con 0x7f828815ddf0 2024-04-06T12:27:50.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.290+0000 7f828e56d700 1 --2- 172.21.15.189:0/654961506 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f826c06ad20 0x7f826c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:50.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.290+0000 7f828e56d700 1 --2- 172.21.15.189:0/654961506 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f826c06ad20 0x7f826c06d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8284009c40 tx=0x7f8284008030 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:50.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.294+0000 7f827effd700 1 -- 172.21.15.189:0/654961506 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f828815a9c0 con 0x7f828815ddf0 2024-04-06T12:27:50.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:50.488+0000 7f82907d1700 1 -- 172.21.15.189:0/654961506 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f828815a9c0 con 0x7f828815ddf0 2024-04-06T12:27:51.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.144+0000 7f827effd700 1 -- 172.21.15.189:0/654961506 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f8280054730 con 0x7f828815ddf0 2024-04-06T12:27:51.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.145+0000 7f827effd700 1 -- 172.21.15.189:0/654961506 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v89) v1 ==== 169+0+0 (secure 0 0 0) 0x7f82800505f0 con 0x7f828815ddf0 2024-04-06T12:27:51.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.149+0000 7f82907d1700 1 -- 172.21.15.189:0/654961506 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f826c06ad20 msgr2=0x7f826c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:51.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.149+0000 7f82907d1700 1 --2- 172.21.15.189:0/654961506 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f826c06ad20 0x7f826c06d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8284009c40 tx=0x7f8284008030 comp rx=0 tx=0).stop 2024-04-06T12:27:51.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.149+0000 7f82907d1700 1 -- 172.21.15.189:0/654961506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f828815ddf0 msgr2=0x7f82881f2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:51.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.149+0000 7f82907d1700 1 --2- 172.21.15.189:0/654961506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f828815ddf0 0x7f82881f2d00 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f828000cb90 tx=0x7f828000cf50 comp rx=0 tx=0).stop 2024-04-06T12:27:51.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.149+0000 7f82907d1700 1 -- 172.21.15.189:0/654961506 shutdown_connections 2024-04-06T12:27:51.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.149+0000 7f82907d1700 1 --2- 172.21.15.189:0/654961506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f828815ddf0 0x7f82881f2d00 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.149+0000 7f82907d1700 1 --2- 172.21.15.189:0/654961506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f828806a730 0x7f82880b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.149+0000 7f82907d1700 1 --2- 172.21.15.189:0/654961506 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f826c06ad20 0x7f826c06d1e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.149+0000 7f82907d1700 1 --2- 172.21.15.189:0/654961506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f828815d7b0 0x7f82880ce8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.149+0000 7f82907d1700 1 -- 172.21.15.189:0/654961506 >> 172.21.15.189:0/654961506 conn(0x7f82880d00b0 msgr2=0x7f82880acea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:51.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.149+0000 7f82907d1700 1 -- 172.21.15.189:0/654961506 shutdown_connections 2024-04-06T12:27:51.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.149+0000 7f82907d1700 1 -- 172.21.15.189:0/654961506 wait complete. 2024-04-06T12:27:51.147 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' removed 2024-04-06T12:27:51.161 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-06T12:27:51.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.289+0000 7fe1cdfce700 1 Processor -- start 2024-04-06T12:27:51.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.289+0000 7fe1cdfce700 1 -- start start 2024-04-06T12:27:51.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.290+0000 7fe1cdfce700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe1c80cb870 0x7fe1c815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:51.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.290+0000 7fe1cdfce700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe1c80cfa00 0x7fe1c80cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:51.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.290+0000 7fe1cdfce700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c80cbd80 con 0x7fe1c80d11b0 2024-04-06T12:27:51.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.290+0000 7fe1cdfce700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c80ccdc0 con 0x7fe1c80cb870 2024-04-06T12:27:51.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.290+0000 7fe1cdfce700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c80d0b50 con 0x7fe1c80cfa00 2024-04-06T12:27:51.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.290+0000 7fe1c6ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe1c80cb870 0x7fe1c815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:51.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.290+0000 7fe1c6ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe1c80cb870 0x7fe1c815c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38838/0 (socket says 172.21.15.189:38838) 2024-04-06T12:27:51.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.290+0000 7fe1c6ffd700 1 -- 172.21.15.189:0/1660734721 learned_addr learned my addr 172.21.15.189:0/1660734721 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:51.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.290+0000 7fe1c7fff700 1 --2- 172.21.15.189:0/1660734721 >> v2:172.21.15.192:3300/0 conn(0x7fe1c80cfa00 0x7fe1c80cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:51.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.290+0000 7fe1c6ffd700 1 -- 172.21.15.189:0/1660734721 >> v2:172.21.15.192:3300/0 conn(0x7fe1c80cfa00 msgr2=0x7fe1c80cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:51.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.290+0000 7fe1c6ffd700 1 --2- 172.21.15.189:0/1660734721 >> v2:172.21.15.192:3300/0 conn(0x7fe1c80cfa00 0x7fe1c80cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.290+0000 7fe1c6ffd700 1 -- 172.21.15.189:0/1660734721 >> v1:172.21.15.189:6789/0 conn(0x7fe1c80d11b0 legacy=0x7fe1c80d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:51.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.290+0000 7fe1c6ffd700 1 -- 172.21.15.189:0/1660734721 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c80cef20 con 0x7fe1c80cb870 2024-04-06T12:27:51.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.290+0000 7fe1c6ffd700 1 --2- 172.21.15.189:0/1660734721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1c80cb870 0x7fe1c815c280 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fe1c815ff70 tx=0x7fe1b8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbaf4bb7f46dd8e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:51.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.291+0000 7fe1c67fc700 1 -- 172.21.15.189:0/1660734721 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b800b870 con 0x7fe1c80cb870 2024-04-06T12:27:51.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.291+0000 7fe1c67fc700 1 -- 172.21.15.189:0/1660734721 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b800b9d0 con 0x7fe1c80cb870 2024-04-06T12:27:51.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.291+0000 7fe1c67fc700 1 -- 172.21.15.189:0/1660734721 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b800bd60 con 0x7fe1c80cb870 2024-04-06T12:27:51.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.291+0000 7fe1cdfce700 1 -- 172.21.15.189:0/1660734721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1c80cb870 msgr2=0x7fe1c815c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:51.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.291+0000 7fe1cdfce700 1 --2- 172.21.15.189:0/1660734721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1c80cb870 0x7fe1c815c280 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fe1c815ff70 tx=0x7fe1b8009bd0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.291+0000 7fe1cdfce700 1 -- 172.21.15.189:0/1660734721 shutdown_connections 2024-04-06T12:27:51.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.291+0000 7fe1cdfce700 1 --2- 172.21.15.189:0/1660734721 >> v2:172.21.15.192:3300/0 conn(0x7fe1c80cfa00 0x7fe1c80cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.291+0000 7fe1cdfce700 1 --2- 172.21.15.189:0/1660734721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1c80cb870 0x7fe1c815c280 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.291+0000 7fe1cdfce700 1 -- 172.21.15.189:0/1660734721 >> 172.21.15.189:0/1660734721 conn(0x7fe1c80b2db0 msgr2=0x7fe1c80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:51.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.291+0000 7fe1cdfce700 1 -- 172.21.15.189:0/1660734721 shutdown_connections 2024-04-06T12:27:51.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.291+0000 7fe1cdfce700 1 -- 172.21.15.189:0/1660734721 wait complete. 2024-04-06T12:27:51.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.292+0000 7fe1cdfce700 1 Processor -- start 2024-04-06T12:27:51.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.293+0000 7fe1cdfce700 1 -- start start 2024-04-06T12:27:51.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.293+0000 7fe1cdfce700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1c80cb870 0x7fe1c81f4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:51.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.293+0000 7fe1cdfce700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1c80cfa00 0x7fe1c814d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:51.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.293+0000 7fe1cdfce700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1c80d11b0 0x7fe1c8159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:51.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.293+0000 7fe1cdfce700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c81f5400 con 0x7fe1c80d11b0 2024-04-06T12:27:51.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.293+0000 7fe1cdfce700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c81573d0 con 0x7fe1c80cb870 2024-04-06T12:27:51.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.293+0000 7fe1cdfce700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c81567c0 con 0x7fe1c80cfa00 2024-04-06T12:27:51.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.293+0000 7fe1c7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1c80d11b0 0x7fe1c8159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:51.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.293+0000 7fe1c7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1c80d11b0 0x7fe1c8159cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38852/0 (socket says 172.21.15.189:38852) 2024-04-06T12:27:51.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.293+0000 7fe1c7fff700 1 -- 172.21.15.189:0/2390300769 learned_addr learned my addr 172.21.15.189:0/2390300769 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:51.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.294+0000 7fe1c77fe700 1 --2- 172.21.15.189:0/2390300769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1c80cb870 0x7fe1c81f4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:51.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.294+0000 7fe1c7fff700 1 -- 172.21.15.189:0/2390300769 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1c80cfa00 msgr2=0x7fe1c814d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:51.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.294+0000 7fe1c7fff700 1 --2- 172.21.15.189:0/2390300769 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1c80cfa00 0x7fe1c814d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.294+0000 7fe1c7fff700 1 -- 172.21.15.189:0/2390300769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1c80cb870 msgr2=0x7fe1c81f4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:51.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.294+0000 7fe1c7fff700 1 --2- 172.21.15.189:0/2390300769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1c80cb870 0x7fe1c81f4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.294+0000 7fe1c7fff700 1 -- 172.21.15.189:0/2390300769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b8009550 con 0x7fe1c80d11b0 2024-04-06T12:27:51.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.294+0000 7fe1c77fe700 1 --2- 172.21.15.189:0/2390300769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1c80cb870 0x7fe1c81f4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:27:51.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.294+0000 7fe1c7fff700 1 --2- 172.21.15.189:0/2390300769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1c80d11b0 0x7fe1c8159cb0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0007e40 tx=0x7fe1c000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:51.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.294+0000 7fe1bffff700 1 -- 172.21.15.189:0/2390300769 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c000eea0 con 0x7fe1c80d11b0 2024-04-06T12:27:51.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.294+0000 7fe1bffff700 1 -- 172.21.15.189:0/2390300769 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c0014420 con 0x7fe1c80d11b0 2024-04-06T12:27:51.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.294+0000 7fe1cdfce700 1 -- 172.21.15.189:0/2390300769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c8156aa0 con 0x7fe1c80d11b0 2024-04-06T12:27:51.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.294+0000 7fe1bffff700 1 -- 172.21.15.189:0/2390300769 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c00147b0 con 0x7fe1c80d11b0 2024-04-06T12:27:51.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.294+0000 7fe1cdfce700 1 -- 172.21.15.189:0/2390300769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c8157060 con 0x7fe1c80d11b0 2024-04-06T12:27:51.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.295+0000 7fe1cdfce700 1 -- 172.21.15.189:0/2390300769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c8074990 con 0x7fe1c80d11b0 2024-04-06T12:27:51.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.295+0000 7fe1bffff700 1 -- 172.21.15.189:0/2390300769 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fe1c0014910 con 0x7fe1c80d11b0 2024-04-06T12:27:51.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.296+0000 7fe1bffff700 1 --2- 172.21.15.189:0/2390300769 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe1a806adb0 0x7fe1a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:51.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.296+0000 7fe1bffff700 1 -- 172.21.15.189:0/2390300769 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5865+0+0 (secure 0 0 0) 0x7fe1c0089ce0 con 0x7fe1c80d11b0 2024-04-06T12:27:51.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.296+0000 7fe1c77fe700 1 --2- 172.21.15.189:0/2390300769 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe1a806adb0 0x7fe1a806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:51.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.296+0000 7fe1c77fe700 1 --2- 172.21.15.189:0/2390300769 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe1a806adb0 0x7fe1a806d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0000f60 tx=0x7fe1b0009040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:51.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.300+0000 7fe1bffff700 1 -- 172.21.15.189:0/2390300769 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe1c0054da0 con 0x7fe1c80d11b0 2024-04-06T12:27:51.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.495+0000 7fe1cdfce700 1 -- 172.21.15.189:0/2390300769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe1c80d1e20 con 0x7fe1c80d11b0 2024-04-06T12:27:51.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.498+0000 7fe1bffff700 1 -- 172.21.15.189:0/2390300769 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe1c00583c0 con 0x7fe1c80d11b0 2024-04-06T12:27:51.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.500+0000 7fe1cdfce700 1 -- 172.21.15.189:0/2390300769 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe1a806adb0 msgr2=0x7fe1a806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:51.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.500+0000 7fe1cdfce700 1 --2- 172.21.15.189:0/2390300769 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe1a806adb0 0x7fe1a806d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0000f60 tx=0x7fe1b0009040 comp rx=0 tx=0).stop 2024-04-06T12:27:51.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.501+0000 7fe1cdfce700 1 -- 172.21.15.189:0/2390300769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1c80d11b0 msgr2=0x7fe1c8159cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:51.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.501+0000 7fe1cdfce700 1 --2- 172.21.15.189:0/2390300769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1c80d11b0 0x7fe1c8159cb0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0007e40 tx=0x7fe1c000e3c0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.501+0000 7fe1cdfce700 1 -- 172.21.15.189:0/2390300769 shutdown_connections 2024-04-06T12:27:51.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.501+0000 7fe1cdfce700 1 --2- 172.21.15.189:0/2390300769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1c80d11b0 0x7fe1c8159cb0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.501+0000 7fe1cdfce700 1 --2- 172.21.15.189:0/2390300769 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe1a806adb0 0x7fe1a806d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.501+0000 7fe1cdfce700 1 --2- 172.21.15.189:0/2390300769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1c80cb870 0x7fe1c81f4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.501+0000 7fe1cdfce700 1 --2- 172.21.15.189:0/2390300769 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1c80cfa00 0x7fe1c814d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.501+0000 7fe1cdfce700 1 -- 172.21.15.189:0/2390300769 >> 172.21.15.189:0/2390300769 conn(0x7fe1c80b2db0 msgr2=0x7fe1c806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:51.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.501+0000 7fe1cdfce700 1 -- 172.21.15.189:0/2390300769 shutdown_connections 2024-04-06T12:27:51.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.501+0000 7fe1cdfce700 1 -- 172.21.15.189:0/2390300769 wait complete. 2024-04-06T12:27:51.510 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-06T12:27:51.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.637+0000 7fa3efc8c700 1 Processor -- start 2024-04-06T12:27:51.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.637+0000 7fa3efc8c700 1 -- start start 2024-04-06T12:27:51.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.638+0000 7fa3efc8c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa3e815e8a0 0x7fa3e80cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:51.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.638+0000 7fa3efc8c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa3e815fa70 0x7fa3e806aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:51.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.638+0000 7fa3efc8c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e80669d0 con 0x7fa3e815eee0 2024-04-06T12:27:51.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.638+0000 7fa3efc8c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3e806ede0 con 0x7fa3e815fa70 2024-04-06T12:27:51.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.638+0000 7fa3efc8c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3e815ec80 con 0x7fa3e815e8a0 2024-04-06T12:27:51.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.638+0000 7fa3ee229700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa3e815fa70 0x7fa3e806aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:51.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.638+0000 7fa3ee229700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa3e815fa70 0x7fa3e806aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38856/0 (socket says 172.21.15.189:38856) 2024-04-06T12:27:51.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.638+0000 7fa3ee229700 1 -- 172.21.15.189:0/2614130323 learned_addr learned my addr 172.21.15.189:0/2614130323 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:51.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.638+0000 7fa3eda28700 1 --2- 172.21.15.189:0/2614130323 >> v2:172.21.15.192:3300/0 conn(0x7fa3e815e8a0 0x7fa3e80cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:51.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.638+0000 7fa3ee229700 1 -- 172.21.15.189:0/2614130323 >> v2:172.21.15.192:3300/0 conn(0x7fa3e815e8a0 msgr2=0x7fa3e80cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:51.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.638+0000 7fa3ee229700 1 --2- 172.21.15.189:0/2614130323 >> v2:172.21.15.192:3300/0 conn(0x7fa3e815e8a0 0x7fa3e80cbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.638+0000 7fa3ee229700 1 -- 172.21.15.189:0/2614130323 >> v1:172.21.15.192:6789/0 conn(0x7fa3e815eee0 legacy=0x7fa3e815f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:51.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.638+0000 7fa3ee229700 1 -- 172.21.15.189:0/2614130323 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e80666f0 con 0x7fa3e815fa70 2024-04-06T12:27:51.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.638+0000 7fa3ee229700 1 --2- 172.21.15.189:0/2614130323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e815fa70 0x7fa3e806aa10 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fa3e80cf650 tx=0x7fa3e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3dc633fa49a05b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:51.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.638+0000 7fa3eca26700 1 -- 172.21.15.189:0/2614130323 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e000e070 con 0x7fa3e815fa70 2024-04-06T12:27:51.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.638+0000 7fa3eca26700 1 -- 172.21.15.189:0/2614130323 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e0003a20 con 0x7fa3e815fa70 2024-04-06T12:27:51.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.638+0000 7fa3eca26700 1 -- 172.21.15.189:0/2614130323 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e0003dd0 con 0x7fa3e815fa70 2024-04-06T12:27:51.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.639+0000 7fa3efc8c700 1 -- 172.21.15.189:0/2614130323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e815fa70 msgr2=0x7fa3e806aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:51.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.639+0000 7fa3efc8c700 1 --2- 172.21.15.189:0/2614130323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e815fa70 0x7fa3e806aa10 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fa3e80cf650 tx=0x7fa3e000b330 comp rx=0 tx=0).stop 2024-04-06T12:27:51.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.639+0000 7fa3efc8c700 1 -- 172.21.15.189:0/2614130323 shutdown_connections 2024-04-06T12:27:51.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.639+0000 7fa3efc8c700 1 --2- 172.21.15.189:0/2614130323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e815fa70 0x7fa3e806aa10 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.639+0000 7fa3efc8c700 1 --2- 172.21.15.189:0/2614130323 >> v2:172.21.15.192:3300/0 conn(0x7fa3e815e8a0 0x7fa3e80cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.639+0000 7fa3efc8c700 1 -- 172.21.15.189:0/2614130323 >> 172.21.15.189:0/2614130323 conn(0x7fa3e80cc290 msgr2=0x7fa3e80cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:51.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.639+0000 7fa3efc8c700 1 -- 172.21.15.189:0/2614130323 shutdown_connections 2024-04-06T12:27:51.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.639+0000 7fa3efc8c700 1 -- 172.21.15.189:0/2614130323 wait complete. 2024-04-06T12:27:51.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.641+0000 7fa3efc8c700 1 Processor -- start 2024-04-06T12:27:51.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.641+0000 7fa3efc8c700 1 -- start start 2024-04-06T12:27:51.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.641+0000 7fa3efc8c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e815e8a0 0x7fa3e80ade40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:51.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.641+0000 7fa3efc8c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3e815eee0 0x7fa3e8159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:51.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.641+0000 7fa3efc8c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3e815fa70 0x7fa3e80af210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:51.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.642+0000 7fa3efc8c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e80b5330 con 0x7fa3e815e8a0 2024-04-06T12:27:51.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.642+0000 7fa3efc8c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e80ae380 con 0x7fa3e815eee0 2024-04-06T12:27:51.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.642+0000 7fa3efc8c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e815a220 con 0x7fa3e815fa70 2024-04-06T12:27:51.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.642+0000 7fa3eda28700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e815e8a0 0x7fa3e80ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:51.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.642+0000 7fa3eda28700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e815e8a0 0x7fa3e80ade40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38866/0 (socket says 172.21.15.189:38866) 2024-04-06T12:27:51.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.642+0000 7fa3eda28700 1 -- 172.21.15.189:0/1987235634 learned_addr learned my addr 172.21.15.189:0/1987235634 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:51.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.642+0000 7fa3eda28700 1 -- 172.21.15.189:0/1987235634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3e815fa70 msgr2=0x7fa3e80af210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:51.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.642+0000 7fa3eda28700 1 --2- 172.21.15.189:0/1987235634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3e815fa70 0x7fa3e80af210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.642+0000 7fa3eda28700 1 -- 172.21.15.189:0/1987235634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3e815eee0 msgr2=0x7fa3e8159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:51.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.642+0000 7fa3eda28700 1 --2- 172.21.15.189:0/1987235634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3e815eee0 0x7fa3e8159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.642+0000 7fa3eda28700 1 -- 172.21.15.189:0/1987235634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e0009b30 con 0x7fa3e815e8a0 2024-04-06T12:27:51.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.642+0000 7fa3eda28700 1 --2- 172.21.15.189:0/1987235634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e815e8a0 0x7fa3e80ade40 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fa3e400e810 tx=0x7fa3e400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:51.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.642+0000 7fa3de7fc700 1 -- 172.21.15.189:0/1987235634 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4013610 con 0x7fa3e815e8a0 2024-04-06T12:27:51.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.642+0000 7fa3de7fc700 1 -- 172.21.15.189:0/1987235634 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e4013770 con 0x7fa3e815e8a0 2024-04-06T12:27:51.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.642+0000 7fa3efc8c700 1 -- 172.21.15.189:0/1987235634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e80b3920 con 0x7fa3e815e8a0 2024-04-06T12:27:51.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.642+0000 7fa3de7fc700 1 -- 172.21.15.189:0/1987235634 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4013b20 con 0x7fa3e815e8a0 2024-04-06T12:27:51.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.642+0000 7fa3efc8c700 1 -- 172.21.15.189:0/1987235634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e80b3ee0 con 0x7fa3e815e8a0 2024-04-06T12:27:51.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.643+0000 7fa3efc8c700 1 -- 172.21.15.189:0/1987235634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e8072140 con 0x7fa3e815e8a0 2024-04-06T12:27:51.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.644+0000 7fa3de7fc700 1 -- 172.21.15.189:0/1987235634 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fa3e4013c80 con 0x7fa3e815e8a0 2024-04-06T12:27:51.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.644+0000 7fa3de7fc700 1 --2- 172.21.15.189:0/1987235634 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa3cc06adb0 0x7fa3cc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:51.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.644+0000 7fa3de7fc700 1 -- 172.21.15.189:0/1987235634 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5865+0+0 (secure 0 0 0) 0x7fa3e4089330 con 0x7fa3e815e8a0 2024-04-06T12:27:51.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.644+0000 7fa3ed227700 1 --2- 172.21.15.189:0/1987235634 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa3cc06adb0 0x7fa3cc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:51.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.644+0000 7fa3ed227700 1 --2- 172.21.15.189:0/1987235634 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa3cc06adb0 0x7fa3cc06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa3e80b4b70 tx=0x7fa3d8007a40 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:51.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.648+0000 7fa3de7fc700 1 -- 172.21.15.189:0/1987235634 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa3e4054260 con 0x7fa3e815e8a0 2024-04-06T12:27:51.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.847+0000 7fa3efc8c700 1 -- 172.21.15.189:0/1987235634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fa3e8074990 con 0x7fa3e815e8a0 2024-04-06T12:27:51.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.853+0000 7fa3de7fc700 1 -- 172.21.15.189:0/1987235634 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v26) v1 ==== 115+0+0 (secure 0 0 0) 0x7fa3e4057880 con 0x7fa3e815e8a0 2024-04-06T12:27:51.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.856+0000 7fa3efc8c700 1 -- 172.21.15.189:0/1987235634 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa3cc06adb0 msgr2=0x7fa3cc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:51.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.856+0000 7fa3efc8c700 1 --2- 172.21.15.189:0/1987235634 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa3cc06adb0 0x7fa3cc06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa3e80b4b70 tx=0x7fa3d8007a40 comp rx=0 tx=0).stop 2024-04-06T12:27:51.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.856+0000 7fa3efc8c700 1 -- 172.21.15.189:0/1987235634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e815e8a0 msgr2=0x7fa3e80ade40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:51.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.856+0000 7fa3efc8c700 1 --2- 172.21.15.189:0/1987235634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e815e8a0 0x7fa3e80ade40 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fa3e400e810 tx=0x7fa3e400ebd0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.856+0000 7fa3efc8c700 1 -- 172.21.15.189:0/1987235634 shutdown_connections 2024-04-06T12:27:51.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.856+0000 7fa3efc8c700 1 --2- 172.21.15.189:0/1987235634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3e815fa70 0x7fa3e80af210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.856+0000 7fa3efc8c700 1 --2- 172.21.15.189:0/1987235634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e815e8a0 0x7fa3e80ade40 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.856+0000 7fa3efc8c700 1 --2- 172.21.15.189:0/1987235634 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa3cc06adb0 0x7fa3cc06d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.856+0000 7fa3efc8c700 1 --2- 172.21.15.189:0/1987235634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3e815eee0 0x7fa3e8159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.856+0000 7fa3efc8c700 1 -- 172.21.15.189:0/1987235634 >> 172.21.15.189:0/1987235634 conn(0x7fa3e80cc290 msgr2=0x7fa3e815c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:51.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.856+0000 7fa3efc8c700 1 -- 172.21.15.189:0/1987235634 shutdown_connections 2024-04-06T12:27:51.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.856+0000 7fa3efc8c700 1 -- 172.21.15.189:0/1987235634 wait complete. 2024-04-06T12:27:51.865 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-06T12:27:51.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.993+0000 7f34f703f700 1 Processor -- start 2024-04-06T12:27:51.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.993+0000 7f34f703f700 1 -- start start 2024-04-06T12:27:51.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.994+0000 7f34f703f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f34f015c2d0 0x7f34f0166230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:51.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.994+0000 7f34f703f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f015f9e0 con 0x7f34f006ae40 2024-04-06T12:27:51.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.994+0000 7f34f703f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f015f0d0 con 0x7f34f0158fe0 2024-04-06T12:27:51.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.994+0000 7f34f703f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f015c8f0 con 0x7f34f015c2d0 2024-04-06T12:27:51.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.994+0000 7f34effff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f34f006ae40 0x7f34f015bbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33870/0 (socket says 172.21.15.189:33870) 2024-04-06T12:27:51.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.994+0000 7f34effff700 1 -- 172.21.15.189:0/2308513599 learned_addr learned my addr 172.21.15.189:0/2308513599 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:51.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.994+0000 7f34ef7fe700 1 -- 172.21.15.189:0/2308513599 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375392911 0 0) 0x7f34f015f9e0 con 0x7f34f006ae40 2024-04-06T12:27:51.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.994+0000 7f34ef7fe700 1 -- 172.21.15.189:0/2308513599 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34e00035b0 con 0x7f34f006ae40 2024-04-06T12:27:51.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.994+0000 7f34f55dc700 1 --2- 172.21.15.189:0/2308513599 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f34f015c2d0 0x7f34f0166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:51.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.994+0000 7f34ef7fe700 1 -- 172.21.15.189:0/2308513599 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1586463754 0 0) 0x7f34e00035b0 con 0x7f34f006ae40 2024-04-06T12:27:51.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.994+0000 7f34ef7fe700 1 -- 172.21.15.189:0/2308513599 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f34f015c2d0 msgr2=0x7f34f0166230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:51.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.994+0000 7f34ef7fe700 1 --2- 172.21.15.189:0/2308513599 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f34f015c2d0 0x7f34f0166230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.994+0000 7f34ef7fe700 1 -- 172.21.15.189:0/2308513599 >> v1:172.21.15.192:6789/0 conn(0x7f34f0158fe0 legacy=0x7f34f006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:51.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.994+0000 7f34ef7fe700 1 -- 172.21.15.189:0/2308513599 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f015f390 con 0x7f34f006ae40 2024-04-06T12:27:51.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.994+0000 7f34ef7fe700 1 -- 172.21.15.189:0/2308513599 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f34dc001670 con 0x7f34f006ae40 2024-04-06T12:27:51.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.994+0000 7f34ef7fe700 1 -- 172.21.15.189:0/2308513599 >> v1:172.21.15.189:6789/0 conn(0x7f34f006ae40 legacy=0x7f34f015bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:51.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.994+0000 7f34ef7fe700 1 --2- 172.21.15.189:0/2308513599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34e0003a30 0x7f34e0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:51.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.995+0000 7f34ef7fe700 1 --2- 172.21.15.189:0/2308513599 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f34f015c2d0 0x7f34e0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:51.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.995+0000 7f34ef7fe700 1 --2- 172.21.15.189:0/2308513599 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34e0004d90 0x7f34e0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:51.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.995+0000 7f34ef7fe700 1 -- 172.21.15.189:0/2308513599 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f015c8f0 con 0x7f34e0003a30 2024-04-06T12:27:51.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.995+0000 7f34ef7fe700 1 -- 172.21.15.189:0/2308513599 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e000ff50 con 0x7f34e0004d90 2024-04-06T12:27:51.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.995+0000 7f34ef7fe700 1 -- 172.21.15.189:0/2308513599 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e0010110 con 0x7f34f015c2d0 2024-04-06T12:27:51.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.995+0000 7f34f4ddb700 1 --2- 172.21.15.189:0/2308513599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34e0003a30 0x7f34e0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:51.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.995+0000 7f34effff700 1 --2- 172.21.15.189:0/2308513599 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f34f015c2d0 0x7f34e0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:51.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.995+0000 7f34f55dc700 1 --2- 172.21.15.189:0/2308513599 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34e0004d90 0x7f34e0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:51.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.995+0000 7f34f4ddb700 1 -- 172.21.15.189:0/2308513599 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f34f015c2d0 msgr2=0x7f34e0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:51.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.995+0000 7f34f4ddb700 1 --2- 172.21.15.189:0/2308513599 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f34f015c2d0 0x7f34e0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.995+0000 7f34f4ddb700 1 -- 172.21.15.189:0/2308513599 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34e0004d90 msgr2=0x7f34e0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:51.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.995+0000 7f34f4ddb700 1 --2- 172.21.15.189:0/2308513599 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34e0004d90 0x7f34e0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:51.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.995+0000 7f34f4ddb700 1 -- 172.21.15.189:0/2308513599 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e00035b0 con 0x7f34e0003a30 2024-04-06T12:27:51.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.995+0000 7f34f4ddb700 1 --2- 172.21.15.189:0/2308513599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34e0003a30 0x7f34e0003e50 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f34e4012dd0 tx=0x7f34e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd8f04cb9aef7a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:51.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.995+0000 7f34ef7fe700 1 -- 172.21.15.189:0/2308513599 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e4017070 con 0x7f34e0003a30 2024-04-06T12:27:51.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.995+0000 7f34ef7fe700 1 -- 172.21.15.189:0/2308513599 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e4012e00 con 0x7f34e0003a30 2024-04-06T12:27:51.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.995+0000 7f34ef7fe700 1 -- 172.21.15.189:0/2308513599 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e4012910 con 0x7f34e0003a30 2024-04-06T12:27:51.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.996+0000 7f34f703f700 1 -- 172.21.15.189:0/2308513599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34e0003a30 msgr2=0x7f34e0003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:51.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.996+0000 7f34f703f700 1 --2- 172.21.15.189:0/2308513599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34e0003a30 0x7f34e0003e50 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f34e4012dd0 tx=0x7f34e0005880 comp rx=0 tx=0).stop 2024-04-06T12:27:51.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.996+0000 7f34f55dc700 1 -- 172.21.15.189:0/2308513599 reap_dead start 2024-04-06T12:27:51.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.996+0000 7f34f703f700 1 -- 172.21.15.189:0/2308513599 shutdown_connections 2024-04-06T12:27:51.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.996+0000 7f34f703f700 1 -- 172.21.15.189:0/2308513599 >> 172.21.15.189:0/2308513599 conn(0x7f34f00d00b0 msgr2=0x7f34f00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:51.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.996+0000 7f34f703f700 1 -- 172.21.15.189:0/2308513599 shutdown_connections 2024-04-06T12:27:51.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.996+0000 7f34f703f700 1 -- 172.21.15.189:0/2308513599 wait complete. 2024-04-06T12:27:51.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.997+0000 7f34f703f700 1 Processor -- start 2024-04-06T12:27:51.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.997+0000 7f34f703f700 1 -- start start 2024-04-06T12:27:51.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.998+0000 7f34f703f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34e0003a30 0x7f34f00b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:51.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.998+0000 7f34f703f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34f0141500 0x7f34f01ee360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:51.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.998+0000 7f34f703f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f34f00717b0 0x7f34f0071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:51.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.998+0000 7f34f703f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f00721a0 con 0x7f34e0003a30 2024-04-06T12:27:51.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.998+0000 7f34f703f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f013ef90 con 0x7f34f0141500 2024-04-06T12:27:51.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.998+0000 7f34f703f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f00b5400 con 0x7f34f00717b0 2024-04-06T12:27:51.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.998+0000 7f34f4ddb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34e0003a30 0x7f34f00b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:52.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.998+0000 7f34f4ddb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34e0003a30 0x7f34f00b4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38882/0 (socket says 172.21.15.189:38882) 2024-04-06T12:27:52.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.998+0000 7f34f4ddb700 1 -- 172.21.15.189:0/1660834534 learned_addr learned my addr 172.21.15.189:0/1660834534 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:52.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.998+0000 7f34effff700 1 --2- 172.21.15.189:0/1660834534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34f0141500 0x7f34f01ee360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:52.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.998+0000 7f34f55dc700 1 --2- 172.21.15.189:0/1660834534 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f34f00717b0 0x7f34f0071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:52.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.998+0000 7f34f4ddb700 1 -- 172.21.15.189:0/1660834534 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f34f00717b0 msgr2=0x7f34f0071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:52.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.998+0000 7f34f4ddb700 1 --2- 172.21.15.189:0/1660834534 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f34f00717b0 0x7f34f0071c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:52.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.998+0000 7f34f4ddb700 1 -- 172.21.15.189:0/1660834534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34f0141500 msgr2=0x7f34f01ee360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:52.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.998+0000 7f34f4ddb700 1 --2- 172.21.15.189:0/1660834534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34f0141500 0x7f34f01ee360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:52.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.998+0000 7f34f4ddb700 1 -- 172.21.15.189:0/1660834534 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f01419c0 con 0x7f34e0003a30 2024-04-06T12:27:52.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.998+0000 7f34f55dc700 1 --2- 172.21.15.189:0/1660834534 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f34f00717b0 0x7f34f0071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:27:52.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.998+0000 7f34f4ddb700 1 --2- 172.21.15.189:0/1660834534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34e0003a30 0x7f34f00b4ec0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f34e4004000 tx=0x7f34e4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:52.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.998+0000 7f34ed7fa700 1 -- 172.21.15.189:0/1660834534 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e4020070 con 0x7f34e0003a30 2024-04-06T12:27:52.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.999+0000 7f34ed7fa700 1 -- 172.21.15.189:0/1660834534 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e4003680 con 0x7f34e0003a30 2024-04-06T12:27:52.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.999+0000 7f34f703f700 1 -- 172.21.15.189:0/1660834534 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f0141ca0 con 0x7f34e0003a30 2024-04-06T12:27:52.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.999+0000 7f34ed7fa700 1 -- 172.21.15.189:0/1660834534 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e4003a10 con 0x7f34e0003a30 2024-04-06T12:27:52.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.999+0000 7f34f703f700 1 -- 172.21.15.189:0/1660834534 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f0142260 con 0x7f34e0003a30 2024-04-06T12:27:52.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:51.999+0000 7f34f703f700 1 -- 172.21.15.189:0/1660834534 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f0067bb0 con 0x7f34e0003a30 2024-04-06T12:27:52.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.000+0000 7f34ed7fa700 1 -- 172.21.15.189:0/1660834534 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f34e4003b70 con 0x7f34e0003a30 2024-04-06T12:27:52.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.000+0000 7f34ed7fa700 1 --2- 172.21.15.189:0/1660834534 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f34d406ade0 0x7f34d406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:52.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.000+0000 7f34ed7fa700 1 -- 172.21.15.189:0/1660834534 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f34e4023040 con 0x7f34e0003a30 2024-04-06T12:27:52.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.000+0000 7f34effff700 1 --2- 172.21.15.189:0/1660834534 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f34d406ade0 0x7f34d406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:52.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.001+0000 7f34effff700 1 --2- 172.21.15.189:0/1660834534 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f34d406ade0 0x7f34d406d2a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f34f00cd5c0 tx=0x7f34dc009040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:52.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.005+0000 7f34ed7fa700 1 -- 172.21.15.189:0/1660834534 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f34e405cfb0 con 0x7f34e0003a30 2024-04-06T12:27:52.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.203+0000 7f34f703f700 1 -- 172.21.15.189:0/1660834534 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f34f01423f0 con 0x7f34e0003a30 2024-04-06T12:27:52.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.206+0000 7f34ed7fa700 1 -- 172.21.15.189:0/1660834534 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v27) v1 ==== 120+0+0 (secure 0 0 0) 0x7f34e40605d0 con 0x7f34e0003a30 2024-04-06T12:27:52.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.209+0000 7f34f703f700 1 -- 172.21.15.189:0/1660834534 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f34d406ade0 msgr2=0x7f34d406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:52.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.209+0000 7f34f703f700 1 --2- 172.21.15.189:0/1660834534 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f34d406ade0 0x7f34d406d2a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f34f00cd5c0 tx=0x7f34dc009040 comp rx=0 tx=0).stop 2024-04-06T12:27:52.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.209+0000 7f34f703f700 1 -- 172.21.15.189:0/1660834534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34e0003a30 msgr2=0x7f34f00b4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:52.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.209+0000 7f34f703f700 1 --2- 172.21.15.189:0/1660834534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34e0003a30 0x7f34f00b4ec0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f34e4004000 tx=0x7f34e4001120 comp rx=0 tx=0).stop 2024-04-06T12:27:52.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.209+0000 7f34f703f700 1 -- 172.21.15.189:0/1660834534 shutdown_connections 2024-04-06T12:27:52.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.209+0000 7f34f703f700 1 --2- 172.21.15.189:0/1660834534 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f34f00717b0 0x7f34f0071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:52.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.209+0000 7f34f703f700 1 --2- 172.21.15.189:0/1660834534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34e0003a30 0x7f34f00b4ec0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:52.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.209+0000 7f34f703f700 1 --2- 172.21.15.189:0/1660834534 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f34d406ade0 0x7f34d406d2a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:52.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.209+0000 7f34f703f700 1 --2- 172.21.15.189:0/1660834534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34f0141500 0x7f34f01ee360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:52.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.209+0000 7f34f703f700 1 -- 172.21.15.189:0/1660834534 >> 172.21.15.189:0/1660834534 conn(0x7f34f00d00b0 msgr2=0x7f34f006fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:52.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.209+0000 7f34f703f700 1 -- 172.21.15.189:0/1660834534 shutdown_connections 2024-04-06T12:27:52.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.209+0000 7f34f703f700 1 -- 172.21.15.189:0/1660834534 wait complete. 2024-04-06T12:27:52.218 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2024-04-06T12:27:52.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.348+0000 7f88f9cae700 1 Processor -- start 2024-04-06T12:27:52.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.348+0000 7f88f9cae700 1 -- start start 2024-04-06T12:27:52.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.349+0000 7f88f9cae700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f415e8a0 0x7f88f40cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:52.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.349+0000 7f88f9cae700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f88f415fa70 0x7f88f406aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:52.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.349+0000 7f88f9cae700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f40669d0 con 0x7f88f415eee0 2024-04-06T12:27:52.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.349+0000 7f88f9cae700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88f406ede0 con 0x7f88f415fa70 2024-04-06T12:27:52.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.349+0000 7f88f9cae700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f415ec80 con 0x7f88f415e8a0 2024-04-06T12:27:52.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.349+0000 7f88f3fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f88f415fa70 0x7f88f406aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:52.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.349+0000 7f88f3fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f88f415fa70 0x7f88f406aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38884/0 (socket says 172.21.15.189:38884) 2024-04-06T12:27:52.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.349+0000 7f88f3fff700 1 -- 172.21.15.189:0/1797296543 learned_addr learned my addr 172.21.15.189:0/1797296543 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:52.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.350+0000 7f88f3fff700 1 -- 172.21.15.189:0/1797296543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f415e8a0 msgr2=0x7f88f40cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:52.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.350+0000 7f88f37fe700 1 --2- 172.21.15.189:0/1797296543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f415e8a0 0x7f88f40cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:52.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.350+0000 7f88f3fff700 1 --2- 172.21.15.189:0/1797296543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f415e8a0 0x7f88f40cbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:52.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.350+0000 7f88f3fff700 1 -- 172.21.15.189:0/1797296543 >> v1:172.21.15.192:6789/0 conn(0x7f88f415eee0 legacy=0x7f88f415f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:52.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.350+0000 7f88f3fff700 1 -- 172.21.15.189:0/1797296543 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f40666f0 con 0x7f88f415fa70 2024-04-06T12:27:52.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.350+0000 7f88f3fff700 1 --2- 172.21.15.189:0/1797296543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88f415fa70 0x7f88f406aa10 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f88ec009e80 tx=0x7f88ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=684f944edfe6ff14 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:52.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.350+0000 7f88f27fc700 1 -- 172.21.15.189:0/1797296543 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec00e070 con 0x7f88f415fa70 2024-04-06T12:27:52.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.350+0000 7f88f27fc700 1 -- 172.21.15.189:0/1797296543 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ec003a20 con 0x7f88f415fa70 2024-04-06T12:27:52.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.350+0000 7f88f27fc700 1 -- 172.21.15.189:0/1797296543 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec003dd0 con 0x7f88f415fa70 2024-04-06T12:27:52.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.350+0000 7f88f9cae700 1 -- 172.21.15.189:0/1797296543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88f415fa70 msgr2=0x7f88f406aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:52.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.350+0000 7f88f9cae700 1 --2- 172.21.15.189:0/1797296543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88f415fa70 0x7f88f406aa10 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f88ec009e80 tx=0x7f88ec00b330 comp rx=0 tx=0).stop 2024-04-06T12:27:52.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.350+0000 7f88f9cae700 1 -- 172.21.15.189:0/1797296543 shutdown_connections 2024-04-06T12:27:52.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.350+0000 7f88f9cae700 1 --2- 172.21.15.189:0/1797296543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88f415fa70 0x7f88f406aa10 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:52.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.350+0000 7f88f9cae700 1 --2- 172.21.15.189:0/1797296543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f415e8a0 0x7f88f40cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:52.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.350+0000 7f88f9cae700 1 -- 172.21.15.189:0/1797296543 >> 172.21.15.189:0/1797296543 conn(0x7f88f40cc290 msgr2=0x7f88f40cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:52.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.350+0000 7f88f9cae700 1 -- 172.21.15.189:0/1797296543 shutdown_connections 2024-04-06T12:27:52.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.350+0000 7f88f9cae700 1 -- 172.21.15.189:0/1797296543 wait complete. 2024-04-06T12:27:52.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.352+0000 7f88f9cae700 1 Processor -- start 2024-04-06T12:27:52.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.352+0000 7f88f9cae700 1 -- start start 2024-04-06T12:27:52.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.352+0000 7f88f9cae700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88f415e8a0 0x7f88f4148ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:52.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.352+0000 7f88f9cae700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88f415eee0 0x7f88f414c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:52.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.352+0000 7f88f9cae700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f415fa70 0x7f88f40b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:52.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.352+0000 7f88f9cae700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f40c8a30 con 0x7f88f415e8a0 2024-04-06T12:27:52.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.353+0000 7f88f9cae700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f40c8d10 con 0x7f88f415eee0 2024-04-06T12:27:52.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.353+0000 7f88f9cae700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f40c8ff0 con 0x7f88f415fa70 2024-04-06T12:27:52.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.353+0000 7f88f37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88f415e8a0 0x7f88f4148ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:52.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.353+0000 7f88f37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88f415e8a0 0x7f88f4148ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38898/0 (socket says 172.21.15.189:38898) 2024-04-06T12:27:52.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.353+0000 7f88f37fe700 1 -- 172.21.15.189:0/1975475500 learned_addr learned my addr 172.21.15.189:0/1975475500 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:52.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.353+0000 7f88f3fff700 1 --2- 172.21.15.189:0/1975475500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f415fa70 0x7f88f40b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:52.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.353+0000 7f88f2ffd700 1 --2- 172.21.15.189:0/1975475500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88f415eee0 0x7f88f414c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:52.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.353+0000 7f88f37fe700 1 -- 172.21.15.189:0/1975475500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f415fa70 msgr2=0x7f88f40b4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:52.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.353+0000 7f88f37fe700 1 --2- 172.21.15.189:0/1975475500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f415fa70 0x7f88f40b4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:52.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.353+0000 7f88f37fe700 1 -- 172.21.15.189:0/1975475500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88f415eee0 msgr2=0x7f88f414c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:52.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.353+0000 7f88f37fe700 1 --2- 172.21.15.189:0/1975475500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88f415eee0 0x7f88f414c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:52.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.353+0000 7f88f37fe700 1 -- 172.21.15.189:0/1975475500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec009b30 con 0x7f88f415e8a0 2024-04-06T12:27:52.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.353+0000 7f88f2ffd700 1 --2- 172.21.15.189:0/1975475500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88f415eee0 0x7f88f414c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:27:52.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.353+0000 7f88f37fe700 1 --2- 172.21.15.189:0/1975475500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88f415e8a0 0x7f88f4148ff0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f88dc00c7f0 tx=0x7f88dc00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:52.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.353+0000 7f88ebfff700 1 -- 172.21.15.189:0/1975475500 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc011800 con 0x7f88f415e8a0 2024-04-06T12:27:52.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.353+0000 7f88ebfff700 1 -- 172.21.15.189:0/1975475500 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88dc011960 con 0x7f88f415e8a0 2024-04-06T12:27:52.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.353+0000 7f88ebfff700 1 -- 172.21.15.189:0/1975475500 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc011cf0 con 0x7f88f415e8a0 2024-04-06T12:27:52.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.353+0000 7f88f9cae700 1 -- 172.21.15.189:0/1975475500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f40c92d0 con 0x7f88f415e8a0 2024-04-06T12:27:52.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.353+0000 7f88f9cae700 1 -- 172.21.15.189:0/1975475500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f4149530 con 0x7f88f415e8a0 2024-04-06T12:27:52.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.354+0000 7f88f9cae700 1 -- 172.21.15.189:0/1975475500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f4072140 con 0x7f88f415e8a0 2024-04-06T12:27:52.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.355+0000 7f88ebfff700 1 -- 172.21.15.189:0/1975475500 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f88dc018360 con 0x7f88f415e8a0 2024-04-06T12:27:52.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.355+0000 7f88ebfff700 1 --2- 172.21.15.189:0/1975475500 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f88d406adb0 0x7f88d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:52.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.355+0000 7f88ebfff700 1 -- 172.21.15.189:0/1975475500 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f88dc089630 con 0x7f88f415e8a0 2024-04-06T12:27:52.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.355+0000 7f88f2ffd700 1 --2- 172.21.15.189:0/1975475500 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f88d406adb0 0x7f88d406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:52.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.356+0000 7f88f2ffd700 1 --2- 172.21.15.189:0/1975475500 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f88d406adb0 0x7f88d406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f88f40cb060 tx=0x7f88e4009040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:52.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.359+0000 7f88ebfff700 1 -- 172.21.15.189:0/1975475500 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f88dc054560 con 0x7f88f415e8a0 2024-04-06T12:27:52.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:52.635+0000 7f88f9cae700 1 -- 172.21.15.189:0/1975475500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f88f40c97f0 con 0x7f88f415e8a0 2024-04-06T12:27:53.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.141+0000 7f88ebfff700 1 -- 172.21.15.189:0/1975475500 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v584) v1 ==== 156+0+0 (secure 0 0 0) 0x7f88f40c97f0 con 0x7f88f415e8a0 2024-04-06T12:27:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.144+0000 7f88f9cae700 1 -- 172.21.15.189:0/1975475500 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f88d406adb0 msgr2=0x7f88d406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:53.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.144+0000 7f88f9cae700 1 --2- 172.21.15.189:0/1975475500 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f88d406adb0 0x7f88d406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f88f40cb060 tx=0x7f88e4009040 comp rx=0 tx=0).stop 2024-04-06T12:27:53.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.144+0000 7f88f9cae700 1 -- 172.21.15.189:0/1975475500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88f415e8a0 msgr2=0x7f88f4148ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:53.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.144+0000 7f88f9cae700 1 --2- 172.21.15.189:0/1975475500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88f415e8a0 0x7f88f4148ff0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f88dc00c7f0 tx=0x7f88dc00cbb0 comp rx=0 tx=0).stop 2024-04-06T12:27:53.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.144+0000 7f88f9cae700 1 -- 172.21.15.189:0/1975475500 shutdown_connections 2024-04-06T12:27:53.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.144+0000 7f88f9cae700 1 --2- 172.21.15.189:0/1975475500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f415fa70 0x7f88f40b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:53.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.144+0000 7f88f9cae700 1 --2- 172.21.15.189:0/1975475500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88f415e8a0 0x7f88f4148ff0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:53.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.144+0000 7f88f9cae700 1 --2- 172.21.15.189:0/1975475500 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f88d406adb0 0x7f88d406d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:53.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.144+0000 7f88f9cae700 1 --2- 172.21.15.189:0/1975475500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88f415eee0 0x7f88f414c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:53.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.144+0000 7f88f9cae700 1 -- 172.21.15.189:0/1975475500 >> 172.21.15.189:0/1975475500 conn(0x7f88f40cc290 msgr2=0x7f88f415d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:53.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.144+0000 7f88f9cae700 1 -- 172.21.15.189:0/1975475500 shutdown_connections 2024-04-06T12:27:53.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.144+0000 7f88f9cae700 1 -- 172.21.15.189:0/1975475500 wait complete. 2024-04-06T12:27:53.154 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2024-04-06T12:27:53.154 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2024-04-06T12:27:53.155 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-06T12:27:53.155 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2024-04-06T12:27:53.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.282+0000 7fcfe5d18700 1 Processor -- start 2024-04-06T12:27:53.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.282+0000 7fcfe5d18700 1 -- start start 2024-04-06T12:27:53.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.283+0000 7fcfe5d18700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcfe00cdae0 0x7fcfe00cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:53.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.283+0000 7fcfe5d18700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfe015a830 con 0x7fcfe00cea80 2024-04-06T12:27:53.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.283+0000 7fcfe5d18700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfe00cc050 con 0x7fcfe00aec70 2024-04-06T12:27:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.283+0000 7fcfe5d18700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe00cd920 con 0x7fcfe00cdae0 2024-04-06T12:27:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.283+0000 7fcfdeffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcfe00cea80 0x7fcfe006ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33882/0 (socket says 172.21.15.189:33882) 2024-04-06T12:27:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.283+0000 7fcfdeffd700 1 -- 172.21.15.189:0/1300115457 learned_addr learned my addr 172.21.15.189:0/1300115457 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.283+0000 7fcfdffff700 1 --2- 172.21.15.189:0/1300115457 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcfe00cdae0 0x7fcfe00cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.283+0000 7fcfde7fc700 1 -- 172.21.15.189:0/1300115457 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942926604 0 0) 0x7fcfe015a830 con 0x7fcfe00cea80 2024-04-06T12:27:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.283+0000 7fcfde7fc700 1 -- 172.21.15.189:0/1300115457 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfcc0035b0 con 0x7fcfe00cea80 2024-04-06T12:27:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.283+0000 7fcfde7fc700 1 -- 172.21.15.189:0/1300115457 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171394330 0 0) 0x7fcfe00cc050 con 0x7fcfe00aec70 2024-04-06T12:27:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.283+0000 7fcfde7fc700 1 -- 172.21.15.189:0/1300115457 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfe015a830 con 0x7fcfe00aec70 2024-04-06T12:27:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.283+0000 7fcfde7fc700 1 -- 172.21.15.189:0/1300115457 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1659752629 0 0) 0x7fcfcc0035b0 con 0x7fcfe00cea80 2024-04-06T12:27:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.283+0000 7fcfde7fc700 1 -- 172.21.15.189:0/1300115457 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcfe00cdae0 msgr2=0x7fcfe00cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfde7fc700 1 --2- 172.21.15.189:0/1300115457 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcfe00cdae0 0x7fcfe00cd3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfde7fc700 1 -- 172.21.15.189:0/1300115457 >> v1:172.21.15.192:6789/0 conn(0x7fcfe00aec70 legacy=0x7fcfe00ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfde7fc700 1 -- 172.21.15.189:0/1300115457 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe00666f0 con 0x7fcfe00cea80 2024-04-06T12:27:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfde7fc700 1 -- 172.21.15.189:0/1300115457 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fcfd0001670 con 0x7fcfe00cea80 2024-04-06T12:27:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfdffff700 1 --2- 172.21.15.189:0/1300115457 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcfe00cdae0 0x7fcfe00cd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfde7fc700 1 -- 172.21.15.189:0/1300115457 >> v1:172.21.15.189:6789/0 conn(0x7fcfe00cea80 legacy=0x7fcfe006ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfde7fc700 1 --2- 172.21.15.189:0/1300115457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfcc003bc0 0x7fcfcc003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfde7fc700 1 --2- 172.21.15.189:0/1300115457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfcc004850 0x7fcfcc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfde7fc700 1 --2- 172.21.15.189:0/1300115457 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcfe00cdae0 0x7fcfcc005280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfde7fc700 1 -- 172.21.15.189:0/1300115457 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe00cd920 con 0x7fcfcc004850 2024-04-06T12:27:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfde7fc700 1 -- 172.21.15.189:0/1300115457 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc00bba0 con 0x7fcfcc003bc0 2024-04-06T12:27:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfde7fc700 1 -- 172.21.15.189:0/1300115457 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc00bd30 con 0x7fcfe00cdae0 2024-04-06T12:27:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfdeffd700 1 --2- 172.21.15.189:0/1300115457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfcc004850 0x7fcfcc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfdf7fe700 1 --2- 172.21.15.189:0/1300115457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfcc003bc0 0x7fcfcc003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfdffff700 1 --2- 172.21.15.189:0/1300115457 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcfe00cdae0 0x7fcfcc005280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfdeffd700 1 -- 172.21.15.189:0/1300115457 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcfe00cdae0 msgr2=0x7fcfcc005280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfdeffd700 1 --2- 172.21.15.189:0/1300115457 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcfe00cdae0 0x7fcfcc005280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfdeffd700 1 -- 172.21.15.189:0/1300115457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfcc003bc0 msgr2=0x7fcfcc003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:53.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfdeffd700 1 --2- 172.21.15.189:0/1300115457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfcc003bc0 0x7fcfcc003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:53.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfdeffd700 1 -- 172.21.15.189:0/1300115457 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd8009a70 con 0x7fcfcc004850 2024-04-06T12:27:53.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfdeffd700 1 --2- 172.21.15.189:0/1300115457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfcc004850 0x7fcfcc004cd0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0007f90 tx=0x7fcfcc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b37df6f757d024d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:53.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfde7fc700 1 -- 172.21.15.189:0/1300115457 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd000cd40 con 0x7fcfcc004850 2024-04-06T12:27:53.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfde7fc700 1 -- 172.21.15.189:0/1300115457 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd0005960 con 0x7fcfcc004850 2024-04-06T12:27:53.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.284+0000 7fcfde7fc700 1 -- 172.21.15.189:0/1300115457 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd000cea0 con 0x7fcfcc004850 2024-04-06T12:27:53.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.285+0000 7fcfe5d18700 1 -- 172.21.15.189:0/1300115457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfcc004850 msgr2=0x7fcfcc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:53.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.285+0000 7fcfe5d18700 1 --2- 172.21.15.189:0/1300115457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfcc004850 0x7fcfcc004cd0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0007f90 tx=0x7fcfcc005880 comp rx=0 tx=0).stop 2024-04-06T12:27:53.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.285+0000 7fcfdffff700 1 -- 172.21.15.189:0/1300115457 reap_dead start 2024-04-06T12:27:53.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.285+0000 7fcfe5d18700 1 -- 172.21.15.189:0/1300115457 shutdown_connections 2024-04-06T12:27:53.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.285+0000 7fcfe5d18700 1 -- 172.21.15.189:0/1300115457 >> 172.21.15.189:0/1300115457 conn(0x7fcfe00cc290 msgr2=0x7fcfe00cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:53.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.285+0000 7fcfe5d18700 1 -- 172.21.15.189:0/1300115457 shutdown_connections 2024-04-06T12:27:53.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.285+0000 7fcfe5d18700 1 -- 172.21.15.189:0/1300115457 wait complete. 2024-04-06T12:27:53.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.286+0000 7fcfe5d18700 1 Processor -- start 2024-04-06T12:27:53.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.286+0000 7fcfe5d18700 1 -- start start 2024-04-06T12:27:53.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.287+0000 7fcfe5d18700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfcc004850 0x7fcfe0156e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:53.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.287+0000 7fcfe5d18700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcfe01ee6b0 0x7fcfe00c2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:53.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.287+0000 7fcfe5d18700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfe00c3470 0x7fcfe00afe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:53.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.287+0000 7fcfe5d18700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe00c3f70 con 0x7fcfe00c3470 2024-04-06T12:27:53.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.287+0000 7fcfe5d18700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe00c4a50 con 0x7fcfcc004850 2024-04-06T12:27:53.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.287+0000 7fcfe5d18700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe0157360 con 0x7fcfe01ee6b0 2024-04-06T12:27:53.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.287+0000 7fcfdffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfe00c3470 0x7fcfe00afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:53.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.287+0000 7fcfdffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfe00c3470 0x7fcfe00afe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38914/0 (socket says 172.21.15.189:38914) 2024-04-06T12:27:53.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.287+0000 7fcfdffff700 1 -- 172.21.15.189:0/523118298 learned_addr learned my addr 172.21.15.189:0/523118298 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:53.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.287+0000 7fcfdeffd700 1 --2- 172.21.15.189:0/523118298 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcfe01ee6b0 0x7fcfe00c2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:53.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.287+0000 7fcfdf7fe700 1 --2- 172.21.15.189:0/523118298 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfcc004850 0x7fcfe0156e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:53.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.287+0000 7fcfdffff700 1 -- 172.21.15.189:0/523118298 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcfe01ee6b0 msgr2=0x7fcfe00c2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:53.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.287+0000 7fcfdffff700 1 --2- 172.21.15.189:0/523118298 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcfe01ee6b0 0x7fcfe00c2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:53.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.287+0000 7fcfdffff700 1 -- 172.21.15.189:0/523118298 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfcc004850 msgr2=0x7fcfe0156e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:53.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.287+0000 7fcfdffff700 1 --2- 172.21.15.189:0/523118298 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfcc004850 0x7fcfe0156e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:53.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.287+0000 7fcfdffff700 1 -- 172.21.15.189:0/523118298 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe00c39b0 con 0x7fcfe00c3470 2024-04-06T12:27:53.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.288+0000 7fcfdffff700 1 --2- 172.21.15.189:0/523118298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfe00c3470 0x7fcfe00afe40 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fcfe00c6870 tx=0x7fcfd800f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:53.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.288+0000 7fcfd7fff700 1 -- 172.21.15.189:0/523118298 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd801c070 con 0x7fcfe00c3470 2024-04-06T12:27:53.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.288+0000 7fcfd7fff700 1 -- 172.21.15.189:0/523118298 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd800f040 con 0x7fcfe00c3470 2024-04-06T12:27:53.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.288+0000 7fcfe5d18700 1 -- 172.21.15.189:0/523118298 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe00c3c90 con 0x7fcfe00c3470 2024-04-06T12:27:53.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.288+0000 7fcfd7fff700 1 -- 172.21.15.189:0/523118298 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd800e040 con 0x7fcfe00c3470 2024-04-06T12:27:53.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.288+0000 7fcfe5d18700 1 -- 172.21.15.189:0/523118298 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe00c75b0 con 0x7fcfe00c3470 2024-04-06T12:27:53.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.289+0000 7fcfe5d18700 1 -- 172.21.15.189:0/523118298 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe00cd390 con 0x7fcfe00c3470 2024-04-06T12:27:53.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.289+0000 7fcfd7fff700 1 -- 172.21.15.189:0/523118298 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fcfd800fd90 con 0x7fcfe00c3470 2024-04-06T12:27:53.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.289+0000 7fcfd7fff700 1 --2- 172.21.15.189:0/523118298 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcfc006adb0 0x7fcfc006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:53.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.290+0000 7fcfd7fff700 1 -- 172.21.15.189:0/523118298 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5865+0+0 (secure 0 0 0) 0x7fcfd8089330 con 0x7fcfe00c3470 2024-04-06T12:27:53.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.290+0000 7fcfdf7fe700 1 --2- 172.21.15.189:0/523118298 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcfc006adb0 0x7fcfc006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:53.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.290+0000 7fcfdf7fe700 1 --2- 172.21.15.189:0/523118298 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcfc006adb0 0x7fcfc006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcfe00b54e0 tx=0x7fcfc8009040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:53.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.294+0000 7fcfd7fff700 1 -- 172.21.15.189:0/523118298 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fcfd8054250 con 0x7fcfe00c3470 2024-04-06T12:27:53.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:53.572+0000 7fcfe5d18700 1 -- 172.21.15.189:0/523118298 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7fcfe0074990 con 0x7fcfe00c3470 2024-04-06T12:27:54.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.143+0000 7fcfd7fff700 1 -- 172.21.15.189:0/523118298 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v585) v1 ==== 169+0+0 (secure 0 0 0) 0x7fcfd8057870 con 0x7fcfe00c3470 2024-04-06T12:27:54.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.146+0000 7fcfe5d18700 1 -- 172.21.15.189:0/523118298 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcfc006adb0 msgr2=0x7fcfc006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:54.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.146+0000 7fcfe5d18700 1 --2- 172.21.15.189:0/523118298 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcfc006adb0 0x7fcfc006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcfe00b54e0 tx=0x7fcfc8009040 comp rx=0 tx=0).stop 2024-04-06T12:27:54.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.146+0000 7fcfe5d18700 1 -- 172.21.15.189:0/523118298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfe00c3470 msgr2=0x7fcfe00afe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:54.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.146+0000 7fcfe5d18700 1 --2- 172.21.15.189:0/523118298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfe00c3470 0x7fcfe00afe40 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fcfe00c6870 tx=0x7fcfd800f490 comp rx=0 tx=0).stop 2024-04-06T12:27:54.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.146+0000 7fcfe5d18700 1 -- 172.21.15.189:0/523118298 shutdown_connections 2024-04-06T12:27:54.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.146+0000 7fcfe5d18700 1 --2- 172.21.15.189:0/523118298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfe00c3470 0x7fcfe00afe40 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:54.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.146+0000 7fcfe5d18700 1 --2- 172.21.15.189:0/523118298 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcfc006adb0 0x7fcfc006d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:54.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.146+0000 7fcfe5d18700 1 --2- 172.21.15.189:0/523118298 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfcc004850 0x7fcfe0156e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:54.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.146+0000 7fcfe5d18700 1 --2- 172.21.15.189:0/523118298 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcfe01ee6b0 0x7fcfe00c2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:54.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.146+0000 7fcfe5d18700 1 -- 172.21.15.189:0/523118298 >> 172.21.15.189:0/523118298 conn(0x7fcfe00cc290 msgr2=0x7fcfe006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:54.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.146+0000 7fcfe5d18700 1 -- 172.21.15.189:0/523118298 shutdown_connections 2024-04-06T12:27:54.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.146+0000 7fcfe5d18700 1 -- 172.21.15.189:0/523118298 wait complete. 2024-04-06T12:27:54.156 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-06T12:27:54.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.285+0000 7fc552f75700 1 Processor -- start 2024-04-06T12:27:54.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.285+0000 7fc552f75700 1 -- start start 2024-04-06T12:27:54.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.285+0000 7fc552f75700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc54c15a550 0x7fc54c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:54.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.285+0000 7fc552f75700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc54c0b4d70 0x7fc54c0b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:54.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.285+0000 7fc552f75700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc54c0cdbe0 con 0x7fc54c0cf900 2024-04-06T12:27:54.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.285+0000 7fc552f75700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc54c15a940 con 0x7fc54c0b4d70 2024-04-06T12:27:54.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.285+0000 7fc552f75700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc54c15ac00 con 0x7fc54c15a550 2024-04-06T12:27:54.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.286+0000 7fc551512700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc54c0b4d70 0x7fc54c0b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:54.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.286+0000 7fc551512700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc54c0b4d70 0x7fc54c0b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38924/0 (socket says 172.21.15.189:38924) 2024-04-06T12:27:54.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.286+0000 7fc551512700 1 -- 172.21.15.189:0/1260685958 learned_addr learned my addr 172.21.15.189:0/1260685958 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:54.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.286+0000 7fc550d11700 1 --2- 172.21.15.189:0/1260685958 >> v2:172.21.15.192:3300/0 conn(0x7fc54c15a550 0x7fc54c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:54.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.286+0000 7fc551512700 1 -- 172.21.15.189:0/1260685958 >> v2:172.21.15.192:3300/0 conn(0x7fc54c15a550 msgr2=0x7fc54c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:54.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.286+0000 7fc551512700 1 --2- 172.21.15.189:0/1260685958 >> v2:172.21.15.192:3300/0 conn(0x7fc54c15a550 0x7fc54c0cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:54.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.286+0000 7fc551512700 1 -- 172.21.15.189:0/1260685958 >> v1:172.21.15.189:6789/0 conn(0x7fc54c0cf900 legacy=0x7fc54c0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:54.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.286+0000 7fc551512700 1 -- 172.21.15.189:0/1260685958 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54c15aee0 con 0x7fc54c0b4d70 2024-04-06T12:27:54.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.286+0000 7fc551512700 1 --2- 172.21.15.189:0/1260685958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc54c0b4d70 0x7fc54c0b2fc0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc54c073470 tx=0x7fc54400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2233c17e5f218cb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:54.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.286+0000 7fc54b7fe700 1 -- 172.21.15.189:0/1260685958 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54400e070 con 0x7fc54c0b4d70 2024-04-06T12:27:54.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.286+0000 7fc54b7fe700 1 -- 172.21.15.189:0/1260685958 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54400b9c0 con 0x7fc54c0b4d70 2024-04-06T12:27:54.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.286+0000 7fc54b7fe700 1 -- 172.21.15.189:0/1260685958 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54400bd70 con 0x7fc54c0b4d70 2024-04-06T12:27:54.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.287+0000 7fc552f75700 1 -- 172.21.15.189:0/1260685958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc54c0b4d70 msgr2=0x7fc54c0b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:54.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.287+0000 7fc552f75700 1 --2- 172.21.15.189:0/1260685958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc54c0b4d70 0x7fc54c0b2fc0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc54c073470 tx=0x7fc54400b330 comp rx=0 tx=0).stop 2024-04-06T12:27:54.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.287+0000 7fc552f75700 1 -- 172.21.15.189:0/1260685958 shutdown_connections 2024-04-06T12:27:54.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.287+0000 7fc552f75700 1 --2- 172.21.15.189:0/1260685958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc54c0b4d70 0x7fc54c0b2fc0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:54.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.287+0000 7fc552f75700 1 --2- 172.21.15.189:0/1260685958 >> v2:172.21.15.192:3300/0 conn(0x7fc54c15a550 0x7fc54c0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:54.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.287+0000 7fc552f75700 1 -- 172.21.15.189:0/1260685958 >> 172.21.15.189:0/1260685958 conn(0x7fc54c0d00b0 msgr2=0x7fc54c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:54.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.287+0000 7fc552f75700 1 -- 172.21.15.189:0/1260685958 shutdown_connections 2024-04-06T12:27:54.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.287+0000 7fc552f75700 1 -- 172.21.15.189:0/1260685958 wait complete. 2024-04-06T12:27:54.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.288+0000 7fc552f75700 1 Processor -- start 2024-04-06T12:27:54.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.288+0000 7fc552f75700 1 -- start start 2024-04-06T12:27:54.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.289+0000 7fc552f75700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc54c0b4d70 0x7fc54c154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:54.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.289+0000 7fc552f75700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc54c0cf900 0x7fc54c14b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:54.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.289+0000 7fc552f75700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc54c14bed0 0x7fc54c157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:54.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.289+0000 7fc552f75700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c14f5e0 con 0x7fc54c0b4d70 2024-04-06T12:27:54.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.289+0000 7fc552f75700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c14b7a0 con 0x7fc54c14bed0 2024-04-06T12:27:54.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.289+0000 7fc552f75700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c150930 con 0x7fc54c0cf900 2024-04-06T12:27:54.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.289+0000 7fc550d11700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc54c0b4d70 0x7fc54c154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:54.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.289+0000 7fc550d11700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc54c0b4d70 0x7fc54c154f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38936/0 (socket says 172.21.15.189:38936) 2024-04-06T12:27:54.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.289+0000 7fc550d11700 1 -- 172.21.15.189:0/4086381390 learned_addr learned my addr 172.21.15.189:0/4086381390 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:54.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.289+0000 7fc54bfff700 1 --2- 172.21.15.189:0/4086381390 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc54c0cf900 0x7fc54c14b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:54.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.289+0000 7fc551512700 1 --2- 172.21.15.189:0/4086381390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc54c14bed0 0x7fc54c157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:54.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.289+0000 7fc550d11700 1 -- 172.21.15.189:0/4086381390 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc54c0cf900 msgr2=0x7fc54c14b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:54.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.289+0000 7fc550d11700 1 --2- 172.21.15.189:0/4086381390 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc54c0cf900 0x7fc54c14b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:54.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.289+0000 7fc550d11700 1 -- 172.21.15.189:0/4086381390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc54c14bed0 msgr2=0x7fc54c157990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:54.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.289+0000 7fc550d11700 1 --2- 172.21.15.189:0/4086381390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc54c14bed0 0x7fc54c157990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:54.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.289+0000 7fc550d11700 1 -- 172.21.15.189:0/4086381390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc544009b30 con 0x7fc54c0b4d70 2024-04-06T12:27:54.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.289+0000 7fc551512700 1 --2- 172.21.15.189:0/4086381390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc54c14bed0 0x7fc54c157990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:27:54.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.289+0000 7fc550d11700 1 --2- 172.21.15.189:0/4086381390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc54c0b4d70 0x7fc54c154f30 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fc54000d830 tx=0x7fc54000dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:54.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.289+0000 7fc5497fa700 1 -- 172.21.15.189:0/4086381390 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5400125f0 con 0x7fc54c0b4d70 2024-04-06T12:27:54.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.290+0000 7fc5497fa700 1 -- 172.21.15.189:0/4086381390 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc540012750 con 0x7fc54c0b4d70 2024-04-06T12:27:54.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.290+0000 7fc552f75700 1 -- 172.21.15.189:0/4086381390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc54c150390 con 0x7fc54c0b4d70 2024-04-06T12:27:54.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.290+0000 7fc5497fa700 1 -- 172.21.15.189:0/4086381390 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc540012b00 con 0x7fc54c0b4d70 2024-04-06T12:27:54.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.290+0000 7fc552f75700 1 -- 172.21.15.189:0/4086381390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54c150650 con 0x7fc54c0b4d70 2024-04-06T12:27:54.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.291+0000 7fc552f75700 1 -- 172.21.15.189:0/4086381390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc54c067bb0 con 0x7fc54c0b4d70 2024-04-06T12:27:54.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.291+0000 7fc5497fa700 1 -- 172.21.15.189:0/4086381390 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fc540012c60 con 0x7fc54c0b4d70 2024-04-06T12:27:54.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.291+0000 7fc5497fa700 1 --2- 172.21.15.189:0/4086381390 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc53006adb0 0x7fc53006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:54.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.291+0000 7fc5497fa700 1 -- 172.21.15.189:0/4086381390 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5865+0+0 (secure 0 0 0) 0x7fc540089830 con 0x7fc54c0b4d70 2024-04-06T12:27:54.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.292+0000 7fc54bfff700 1 --2- 172.21.15.189:0/4086381390 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc53006adb0 0x7fc53006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:54.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.292+0000 7fc54bfff700 1 --2- 172.21.15.189:0/4086381390 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc53006adb0 0x7fc53006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc54c0720e0 tx=0x7fc538009040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:54.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.296+0000 7fc5497fa700 1 -- 172.21.15.189:0/4086381390 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc5400548f0 con 0x7fc54c0b4d70 2024-04-06T12:27:54.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.490+0000 7fc552f75700 1 -- 172.21.15.189:0/4086381390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc54c074990 con 0x7fc54c0b4d70 2024-04-06T12:27:54.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.494+0000 7fc5497fa700 1 -- 172.21.15.189:0/4086381390 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc540057f10 con 0x7fc54c0b4d70 2024-04-06T12:27:54.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.496+0000 7fc552f75700 1 -- 172.21.15.189:0/4086381390 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc53006adb0 msgr2=0x7fc53006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:54.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.496+0000 7fc552f75700 1 --2- 172.21.15.189:0/4086381390 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc53006adb0 0x7fc53006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc54c0720e0 tx=0x7fc538009040 comp rx=0 tx=0).stop 2024-04-06T12:27:54.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.496+0000 7fc552f75700 1 -- 172.21.15.189:0/4086381390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc54c0b4d70 msgr2=0x7fc54c154f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:54.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.496+0000 7fc552f75700 1 --2- 172.21.15.189:0/4086381390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc54c0b4d70 0x7fc54c154f30 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fc54000d830 tx=0x7fc54000dbf0 comp rx=0 tx=0).stop 2024-04-06T12:27:54.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.496+0000 7fc552f75700 1 -- 172.21.15.189:0/4086381390 shutdown_connections 2024-04-06T12:27:54.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.496+0000 7fc552f75700 1 --2- 172.21.15.189:0/4086381390 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc53006adb0 0x7fc53006d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:54.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.496+0000 7fc552f75700 1 --2- 172.21.15.189:0/4086381390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc54c14bed0 0x7fc54c157990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:54.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.496+0000 7fc552f75700 1 --2- 172.21.15.189:0/4086381390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc54c0b4d70 0x7fc54c154f30 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:54.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.496+0000 7fc552f75700 1 --2- 172.21.15.189:0/4086381390 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc54c0cf900 0x7fc54c14b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:54.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.496+0000 7fc552f75700 1 -- 172.21.15.189:0/4086381390 >> 172.21.15.189:0/4086381390 conn(0x7fc54c0d00b0 msgr2=0x7fc54c06dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:54.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.496+0000 7fc552f75700 1 -- 172.21.15.189:0/4086381390 shutdown_connections 2024-04-06T12:27:54.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.496+0000 7fc552f75700 1 -- 172.21.15.189:0/4086381390 wait complete. 2024-04-06T12:27:54.505 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T12:27:54.505 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:27:54.551 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:27:54.603 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T12:27:54.603 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:27:54.603 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:27:54.622 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:27:54.622 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:27:54.622 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:27:54.642 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:27:54.642 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:27:54.642 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:27:54.680 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:27:54.680 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:27:54.681 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:27:54.703 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:27:54.703 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-06T12:27:54.703 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:27:54.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.844+0000 7f2c0b8b5700 1 Processor -- start 2024-04-06T12:27:54.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.844+0000 7f2c0b8b5700 1 -- start start 2024-04-06T12:27:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.845+0000 7f2c0b8b5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c040b3e20 0x7f2c0415e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.845+0000 7f2c0b8b5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c040b3990 con 0x7f2c0406ae40 2024-04-06T12:27:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.845+0000 7f2c0b8b5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c040b3060 con 0x7f2c0415d7b0 2024-04-06T12:27:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.845+0000 7f2c0b8b5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c040b3340 con 0x7f2c040b3e20 2024-04-06T12:27:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.845+0000 7f2c08e50700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2c0406ae40 0x7f2c0415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33904/0 (socket says 172.21.15.189:33904) 2024-04-06T12:27:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.845+0000 7f2c08e50700 1 -- 172.21.15.189:0/2399598132 learned_addr learned my addr 172.21.15.189:0/2399598132 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:27:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.845+0000 7f2c09e52700 1 --2- 172.21.15.189:0/2399598132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c040b3e20 0x7f2c0415e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.845+0000 7f2bfbfff700 1 -- 172.21.15.189:0/2399598132 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638551918 0 0) 0x7f2c040b3990 con 0x7f2c0406ae40 2024-04-06T12:27:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.845+0000 7f2bfbfff700 1 -- 172.21.15.189:0/2399598132 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bf00035b0 con 0x7f2c0406ae40 2024-04-06T12:27:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.845+0000 7f2bfbfff700 1 -- 172.21.15.189:0/2399598132 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 585240835 0 0) 0x7f2bf00035b0 con 0x7f2c0406ae40 2024-04-06T12:27:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.845+0000 7f2bfbfff700 1 -- 172.21.15.189:0/2399598132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c040b3e20 msgr2=0x7f2c0415e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.845+0000 7f2bfbfff700 1 --2- 172.21.15.189:0/2399598132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c040b3e20 0x7f2c0415e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.845+0000 7f2bfbfff700 1 -- 172.21.15.189:0/2399598132 >> v1:172.21.15.192:6789/0 conn(0x7f2c0415d7b0 legacy=0x7f2c0406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.845+0000 7f2bfbfff700 1 -- 172.21.15.189:0/2399598132 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c040b3620 con 0x7f2c0406ae40 2024-04-06T12:27:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.845+0000 7f2bfbfff700 1 -- 172.21.15.189:0/2399598132 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f2bf4001670 con 0x7f2c0406ae40 2024-04-06T12:27:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.845+0000 7f2bfbfff700 1 -- 172.21.15.189:0/2399598132 >> v1:172.21.15.189:6789/0 conn(0x7f2c0406ae40 legacy=0x7f2c0415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.845+0000 7f2bfbfff700 1 --2- 172.21.15.189:0/2399598132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bf0003a30 0x7f2bf0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.846+0000 7f2bfbfff700 1 --2- 172.21.15.189:0/2399598132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c040b3e20 0x7f2bf0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.846+0000 7f2bfbfff700 1 --2- 172.21.15.189:0/2399598132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bf0004d90 0x7f2bf0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.846+0000 7f2bfbfff700 1 -- 172.21.15.189:0/2399598132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c040b3340 con 0x7f2bf0003a30 2024-04-06T12:27:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.846+0000 7f2bfbfff700 1 -- 172.21.15.189:0/2399598132 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf000bac0 con 0x7f2bf0004d90 2024-04-06T12:27:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.846+0000 7f2bfbfff700 1 -- 172.21.15.189:0/2399598132 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf000bc80 con 0x7f2c040b3e20 2024-04-06T12:27:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.846+0000 7f2c09651700 1 --2- 172.21.15.189:0/2399598132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bf0003a30 0x7f2bf0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.846+0000 7f2c08e50700 1 --2- 172.21.15.189:0/2399598132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c040b3e20 0x7f2bf0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.846+0000 7f2c09e52700 1 --2- 172.21.15.189:0/2399598132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bf0004d90 0x7f2bf0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.846+0000 7f2c09651700 1 -- 172.21.15.189:0/2399598132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c040b3e20 msgr2=0x7f2bf0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.846+0000 7f2c09651700 1 --2- 172.21.15.189:0/2399598132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c040b3e20 0x7f2bf0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.846+0000 7f2c09651700 1 -- 172.21.15.189:0/2399598132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bf0004d90 msgr2=0x7f2bf0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:27:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.846+0000 7f2c09651700 1 --2- 172.21.15.189:0/2399598132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bf0004d90 0x7f2bf0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.846+0000 7f2c09651700 1 -- 172.21.15.189:0/2399598132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf00035b0 con 0x7f2bf0003a30 2024-04-06T12:27:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.846+0000 7f2c09651700 1 --2- 172.21.15.189:0/2399598132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bf0003a30 0x7f2bf0003e10 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f2c000018b0 tx=0x7f2bf0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=942b26ae6946dd78 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.846+0000 7f2bfbfff700 1 -- 172.21.15.189:0/2399598132 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c00001370 con 0x7f2bf0003a30 2024-04-06T12:27:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.846+0000 7f2bfbfff700 1 -- 172.21.15.189:0/2399598132 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0000c6b0 con 0x7f2bf0003a30 2024-04-06T12:27:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.846+0000 7f2bfbfff700 1 -- 172.21.15.189:0/2399598132 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0000ca60 con 0x7f2bf0003a30 2024-04-06T12:27:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.847+0000 7f2c0b8b5700 1 -- 172.21.15.189:0/2399598132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bf0003a30 msgr2=0x7f2bf0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.847+0000 7f2c0b8b5700 1 --2- 172.21.15.189:0/2399598132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bf0003a30 0x7f2bf0003e10 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f2c000018b0 tx=0x7f2bf0005880 comp rx=0 tx=0).stop 2024-04-06T12:27:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.847+0000 7f2c09e52700 1 -- 172.21.15.189:0/2399598132 reap_dead start 2024-04-06T12:27:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.847+0000 7f2c0b8b5700 1 -- 172.21.15.189:0/2399598132 shutdown_connections 2024-04-06T12:27:54.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.847+0000 7f2c0b8b5700 1 -- 172.21.15.189:0/2399598132 >> 172.21.15.189:0/2399598132 conn(0x7f2c040d00b0 msgr2=0x7f2c040d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:54.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.847+0000 7f2c0b8b5700 1 -- 172.21.15.189:0/2399598132 shutdown_connections 2024-04-06T12:27:54.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.847+0000 7f2c0b8b5700 1 -- 172.21.15.189:0/2399598132 wait complete. 2024-04-06T12:27:54.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.848+0000 7f2c0b8b5700 1 Processor -- start 2024-04-06T12:27:54.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.848+0000 7f2c0b8b5700 1 -- start start 2024-04-06T12:27:54.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2c0b8b5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bf0003a30 0x7f2c04151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2c0b8b5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c041604e0 0x7f2c04155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2c0b8b5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c04160960 0x7f2c04156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2c0b8b5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c04152450 con 0x7f2c04160960 2024-04-06T12:27:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2c0b8b5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0415ae80 con 0x7f2bf0003a30 2024-04-06T12:27:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2c0b8b5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0415b120 con 0x7f2c041604e0 2024-04-06T12:27:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2c09e52700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c04160960 0x7f2c04156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2c09651700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bf0003a30 0x7f2c04151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2c09e52700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c04160960 0x7f2c04156520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38948/0 (socket says 172.21.15.189:38948) 2024-04-06T12:27:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2c09e52700 1 -- 172.21.15.189:0/241391452 learned_addr learned my addr 172.21.15.189:0/241391452 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:27:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2c08e50700 1 --2- 172.21.15.189:0/241391452 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c041604e0 0x7f2c04155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2c09e52700 1 -- 172.21.15.189:0/241391452 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c041604e0 msgr2=0x7f2c04155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2c09e52700 1 --2- 172.21.15.189:0/241391452 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c041604e0 0x7f2c04155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2c09e52700 1 -- 172.21.15.189:0/241391452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bf0003a30 msgr2=0x7f2c04151f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2c09e52700 1 --2- 172.21.15.189:0/241391452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bf0003a30 0x7f2c04151f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2c09e52700 1 -- 172.21.15.189:0/241391452 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0415b400 con 0x7f2c04160960 2024-04-06T12:27:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2c09651700 1 --2- 172.21.15.189:0/241391452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bf0003a30 0x7f2c04151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:27:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2c09e52700 1 --2- 172.21.15.189:0/241391452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c04160960 0x7f2c04156520 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f2c040b4800 tx=0x7f2bfc0111d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2bf9ffb700 1 -- 172.21.15.189:0/241391452 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bfc00e070 con 0x7f2c04160960 2024-04-06T12:27:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2bf9ffb700 1 -- 172.21.15.189:0/241391452 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bfc003cf0 con 0x7f2c04160960 2024-04-06T12:27:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2bf9ffb700 1 -- 172.21.15.189:0/241391452 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bfc0117a0 con 0x7f2c04160960 2024-04-06T12:27:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.849+0000 7f2c0b8b5700 1 -- 172.21.15.189:0/241391452 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c0415b6e0 con 0x7f2c04160960 2024-04-06T12:27:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.850+0000 7f2c0b8b5700 1 -- 172.21.15.189:0/241391452 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c0415bca0 con 0x7f2c04160960 2024-04-06T12:27:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.851+0000 7f2c0b8b5700 1 -- 172.21.15.189:0/241391452 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c04067bb0 con 0x7f2c04160960 2024-04-06T12:27:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.851+0000 7f2bf9ffb700 1 -- 172.21.15.189:0/241391452 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f2bfc011900 con 0x7f2c04160960 2024-04-06T12:27:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.851+0000 7f2bf9ffb700 1 --2- 172.21.15.189:0/241391452 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2be806adb0 0x7f2be806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:27:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.851+0000 7f2bf9ffb700 1 -- 172.21.15.189:0/241391452 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f2bfc089a40 con 0x7f2c04160960 2024-04-06T12:27:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.851+0000 7f2c09651700 1 --2- 172.21.15.189:0/241391452 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2be806adb0 0x7f2be806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:27:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.852+0000 7f2c09651700 1 --2- 172.21.15.189:0/241391452 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2be806adb0 0x7f2be806d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2c000018b0 tx=0x7f2c00005040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:27:54.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:54.856+0000 7f2bf9ffb700 1 -- 172.21.15.189:0/241391452 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2bfc08be60 con 0x7f2c04160960 2024-04-06T12:27:55.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:55.098+0000 7f2c0b8b5700 1 -- 172.21.15.189:0/241391452 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c040af9e0 con 0x7f2c04160960 2024-04-06T12:27:55.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:55.098+0000 7f2bf9ffb700 1 -- 172.21.15.189:0/241391452 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+3541 (secure 0 0 0) 0x7f2bfc054960 con 0x7f2c04160960 2024-04-06T12:27:55.095 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:27:55.096 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":33,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5762,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/2827078766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2827078766},{"type":"v1","addr":"172.21.15.192:6833","nonce":2827078766}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5765,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1558111999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1558111999},{"type":"v1","addr":"172.21.15.192:6835","nonce":1558111999}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3448511259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3448511259},{"type":"v1","addr":"172.21.15.189:6837","nonce":3448511259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":7008,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3183078766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3183078766},{"type":"v1","addr":"172.21.15.189:6836","nonce":3183078766}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[]} 2024-04-06T12:27:55.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:55.100+0000 7f2c0b8b5700 1 -- 172.21.15.189:0/241391452 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2be806adb0 msgr2=0x7f2be806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:55.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:55.100+0000 7f2c0b8b5700 1 --2- 172.21.15.189:0/241391452 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2be806adb0 0x7f2be806d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2c000018b0 tx=0x7f2c00005040 comp rx=0 tx=0).stop 2024-04-06T12:27:55.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:55.101+0000 7f2c0b8b5700 1 -- 172.21.15.189:0/241391452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c04160960 msgr2=0x7f2c04156520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:27:55.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:55.101+0000 7f2c0b8b5700 1 --2- 172.21.15.189:0/241391452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c04160960 0x7f2c04156520 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f2c040b4800 tx=0x7f2bfc0111d0 comp rx=0 tx=0).stop 2024-04-06T12:27:55.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:55.101+0000 7f2c0b8b5700 1 -- 172.21.15.189:0/241391452 shutdown_connections 2024-04-06T12:27:55.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:55.101+0000 7f2c0b8b5700 1 --2- 172.21.15.189:0/241391452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c04160960 0x7f2c04156520 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:55.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:55.101+0000 7f2c0b8b5700 1 --2- 172.21.15.189:0/241391452 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2be806adb0 0x7f2be806d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:55.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:55.101+0000 7f2c0b8b5700 1 --2- 172.21.15.189:0/241391452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bf0003a30 0x7f2c04151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:55.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:55.101+0000 7f2c0b8b5700 1 --2- 172.21.15.189:0/241391452 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c041604e0 0x7f2c04155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:27:55.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:55.101+0000 7f2c0b8b5700 1 -- 172.21.15.189:0/241391452 >> 172.21.15.189:0/241391452 conn(0x7f2c040d00b0 msgr2=0x7f2c040ae1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:27:55.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:55.101+0000 7f2c0b8b5700 1 -- 172.21.15.189:0/241391452 shutdown_connections 2024-04-06T12:27:55.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:27:55.101+0000 7f2c0b8b5700 1 -- 172.21.15.189:0/241391452 wait complete. 2024-04-06T12:27:55.099 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 33 2024-04-06T12:27:55.110 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-06T12:27:55.110 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-06T12:27:55.111 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-06T12:27:55.111 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:27:55.111 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-06T12:27:55.111 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-06T12:27:55.111 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-06T12:27:55.111 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:27:55.112 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-06T12:27:55.112 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-06T12:27:55.112 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-06T12:27:55.112 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:27:55.112 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-06T12:27:55.112 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-06T12:27:55.112 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-06T12:27:55.112 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:27:55.119 INFO:tasks.ceph.mds.a:Stopped 2024-04-06T12:27:55.120 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-06T12:27:55.120 INFO:tasks.ceph.mds.c:Stopped 2024-04-06T12:27:55.121 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-06T12:27:55.123 INFO:tasks.ceph.mds.b:Stopped 2024-04-06T12:27:55.123 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-06T12:27:55.162 INFO:tasks.ceph.mds.a:Started 2024-04-06T12:27:55.163 DEBUG:teuthology.parallel:result is None 2024-04-06T12:27:55.163 INFO:tasks.ceph.mds.c:Started 2024-04-06T12:27:55.163 DEBUG:teuthology.parallel:result is None 2024-04-06T12:27:55.166 INFO:tasks.ceph.mds.b:Started 2024-04-06T12:27:55.166 DEBUG:teuthology.parallel:result is None 2024-04-06T12:27:55.241 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T12:27:55.241 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2024-04-06T12:27:55.253 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T12:27:56.087 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-04-06T12:28:01.124 INFO:tasks.ceph.mds.d:Stopped 2024-04-06T12:28:01.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-06T12:28:01.127 INFO:tasks.ceph.mds.d:Started 2024-04-06T12:28:01.127 DEBUG:teuthology.parallel:result is None 2024-04-06T12:28:01.127 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-06T12:28:01.213 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2024-04-06T12:28:01.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.269+0000 7fc0d5540700 1 Processor -- start 2024-04-06T12:28:01.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.269+0000 7fc0d5540700 1 -- start start 2024-04-06T12:28:01.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.269+0000 7fc0d5540700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d015d6f0 0x7fc0d015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:01.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.269+0000 7fc0d5540700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc0d00b8d70 0x7fc0d00ce020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:01.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.270+0000 7fc0d5540700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d00ce9b0 con 0x7fc0d00ce560 2024-04-06T12:28:01.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.270+0000 7fc0d5540700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0d015e010 con 0x7fc0d00b8d70 2024-04-06T12:28:01.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.270+0000 7fc0d5540700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d015eac0 con 0x7fc0d015d6f0 2024-04-06T12:28:01.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.270+0000 7fc0ce7fc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc0d00b8d70 0x7fc0d00ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:01.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.270+0000 7fc0cf7fe700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fc0d00ce560 0x7fc0d00b4e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:50428/0 (socket says 172.21.15.189:50428) 2024-04-06T12:28:01.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.270+0000 7fc0cf7fe700 1 -- 172.21.15.189:0/3895468247 learned_addr learned my addr 172.21.15.189:0/3895468247 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:01.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.270+0000 7fc0ceffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d015d6f0 0x7fc0d015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:01.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.270+0000 7fc0ce7fc700 1 -- 172.21.15.189:0/3895468247 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d015d6f0 msgr2=0x7fc0d015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:01.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.270+0000 7fc0ce7fc700 1 --2- 172.21.15.189:0/3895468247 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d015d6f0 0x7fc0d015dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:01.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.270+0000 7fc0ce7fc700 1 -- 172.21.15.189:0/3895468247 >> v1:172.21.15.192:6789/0 conn(0x7fc0d00ce560 legacy=0x7fc0d00b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:01.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.270+0000 7fc0ce7fc700 1 -- 172.21.15.189:0/3895468247 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d015e220 con 0x7fc0d00b8d70 2024-04-06T12:28:01.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.270+0000 7fc0ceffd700 1 --2- 172.21.15.189:0/3895468247 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d015d6f0 0x7fc0d015dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:28:01.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.270+0000 7fc0cdffb700 1 -- 172.21.15.189:0/3895468247 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2077054118 0 0) 0x7fc0c8001490 con 0x7fc0d00ce560 2024-04-06T12:28:01.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.270+0000 7fc0ce7fc700 1 --2- 172.21.15.189:0/3895468247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0d00b8d70 0x7fc0d00ce020 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4009840 tx=0x7fc0c4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71a8d24c3e6bac72 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:01.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.270+0000 7fc0cdffb700 1 -- 172.21.15.189:0/3895468247 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4011830 con 0x7fc0d00b8d70 2024-04-06T12:28:01.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.271+0000 7fc0cdffb700 1 -- 172.21.15.189:0/3895468247 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c4011990 con 0x7fc0d00b8d70 2024-04-06T12:28:01.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.271+0000 7fc0cdffb700 1 -- 172.21.15.189:0/3895468247 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4011d20 con 0x7fc0d00b8d70 2024-04-06T12:28:01.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.271+0000 7fc0d5540700 1 -- 172.21.15.189:0/3895468247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0d00b8d70 msgr2=0x7fc0d00ce020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:01.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.271+0000 7fc0d5540700 1 --2- 172.21.15.189:0/3895468247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0d00b8d70 0x7fc0d00ce020 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4009840 tx=0x7fc0c4009b50 comp rx=0 tx=0).stop 2024-04-06T12:28:01.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.271+0000 7fc0d5540700 1 -- 172.21.15.189:0/3895468247 shutdown_connections 2024-04-06T12:28:01.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.271+0000 7fc0d5540700 1 --2- 172.21.15.189:0/3895468247 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d015d6f0 0x7fc0d015dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:01.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.271+0000 7fc0d5540700 1 --2- 172.21.15.189:0/3895468247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0d00b8d70 0x7fc0d00ce020 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:01.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.271+0000 7fc0d5540700 1 -- 172.21.15.189:0/3895468247 >> 172.21.15.189:0/3895468247 conn(0x7fc0d0072490 msgr2=0x7fc0d006b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:01.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.271+0000 7fc0d5540700 1 -- 172.21.15.189:0/3895468247 shutdown_connections 2024-04-06T12:28:01.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.271+0000 7fc0d5540700 1 -- 172.21.15.189:0/3895468247 wait complete. 2024-04-06T12:28:01.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.272+0000 7fc0d5540700 1 Processor -- start 2024-04-06T12:28:01.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.272+0000 7fc0d5540700 1 -- start start 2024-04-06T12:28:01.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.273+0000 7fc0d5540700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d00b8d70 0x7fc0d00b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:01.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.273+0000 7fc0d5540700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0d00ce560 0x7fc0d014ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:01.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.273+0000 7fc0d5540700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0d015d6f0 0x7fc0d0160450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:01.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.273+0000 7fc0d5540700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d014bc90 con 0x7fc0d00ce560 2024-04-06T12:28:01.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.273+0000 7fc0d5540700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d014bf70 con 0x7fc0d015d6f0 2024-04-06T12:28:01.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.273+0000 7fc0d5540700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d014c250 con 0x7fc0d00b8d70 2024-04-06T12:28:01.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.273+0000 7fc0ce7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0d00ce560 0x7fc0d014ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:01.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.273+0000 7fc0ce7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0d00ce560 0x7fc0d014ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57868/0 (socket says 172.21.15.189:57868) 2024-04-06T12:28:01.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.273+0000 7fc0ce7fc700 1 -- 172.21.15.189:0/1305971384 learned_addr learned my addr 172.21.15.189:0/1305971384 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:01.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.273+0000 7fc0cf7fe700 1 --2- 172.21.15.189:0/1305971384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0d015d6f0 0x7fc0d0160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:01.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.274+0000 7fc0ceffd700 1 --2- 172.21.15.189:0/1305971384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d00b8d70 0x7fc0d00b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:01.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.274+0000 7fc0ce7fc700 1 -- 172.21.15.189:0/1305971384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d00b8d70 msgr2=0x7fc0d00b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:01.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.274+0000 7fc0ce7fc700 1 --2- 172.21.15.189:0/1305971384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d00b8d70 0x7fc0d00b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:01.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.274+0000 7fc0ce7fc700 1 -- 172.21.15.189:0/1305971384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0d015d6f0 msgr2=0x7fc0d0160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:01.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.274+0000 7fc0ce7fc700 1 --2- 172.21.15.189:0/1305971384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0d015d6f0 0x7fc0d0160450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:01.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.274+0000 7fc0ce7fc700 1 -- 172.21.15.189:0/1305971384 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c00094a0 con 0x7fc0d00ce560 2024-04-06T12:28:01.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.274+0000 7fc0ceffd700 1 --2- 172.21.15.189:0/1305971384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d00b8d70 0x7fc0d00b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:01.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.274+0000 7fc0ce7fc700 1 --2- 172.21.15.189:0/1305971384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0d00ce560 0x7fc0d014ee10 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4009fd0 tx=0x7fc0c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:01.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.274+0000 7fc0b77fe700 1 -- 172.21.15.189:0/1305971384 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c40036a0 con 0x7fc0d00ce560 2024-04-06T12:28:01.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.274+0000 7fc0b77fe700 1 -- 172.21.15.189:0/1305971384 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c401f070 con 0x7fc0d00ce560 2024-04-06T12:28:01.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.274+0000 7fc0d5540700 1 -- 172.21.15.189:0/1305971384 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c40094a0 con 0x7fc0d00ce560 2024-04-06T12:28:01.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.274+0000 7fc0b77fe700 1 -- 172.21.15.189:0/1305971384 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4003a20 con 0x7fc0d00ce560 2024-04-06T12:28:01.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.274+0000 7fc0d5540700 1 -- 172.21.15.189:0/1305971384 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d014c810 con 0x7fc0d00ce560 2024-04-06T12:28:01.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.275+0000 7fc0d5540700 1 -- 172.21.15.189:0/1305971384 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d00d12c0 con 0x7fc0d00ce560 2024-04-06T12:28:01.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.276+0000 7fc0b77fe700 1 -- 172.21.15.189:0/1305971384 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fc0c4003b80 con 0x7fc0d00ce560 2024-04-06T12:28:01.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.276+0000 7fc0b77fe700 1 --2- 172.21.15.189:0/1305971384 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc0bc06adb0 0x7fc0bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:01.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.276+0000 7fc0b77fe700 1 -- 172.21.15.189:0/1305971384 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5865+0+0 (secure 0 0 0) 0x7fc0c4021070 con 0x7fc0d00ce560 2024-04-06T12:28:01.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.276+0000 7fc0ceffd700 1 --2- 172.21.15.189:0/1305971384 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc0bc06adb0 0x7fc0bc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:01.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.277+0000 7fc0ceffd700 1 --2- 172.21.15.189:0/1305971384 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc0bc06adb0 0x7fc0bc06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0009d20 tx=0x7fc0c0008040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:01.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.280+0000 7fc0b77fe700 1 -- 172.21.15.189:0/1305971384 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc0c4056260 con 0x7fc0d00ce560 2024-04-06T12:28:01.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:01.496+0000 7fc0d5540700 1 -- 172.21.15.189:0/1305971384 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc0d0074990 con 0x7fc0d00ce560 2024-04-06T12:28:02.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.239+0000 7fc0b77fe700 1 -- 172.21.15.189:0/1305971384 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v94) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc0c4059880 con 0x7fc0d00ce560 2024-04-06T12:28:02.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.242+0000 7fc0d5540700 1 -- 172.21.15.189:0/1305971384 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc0bc06adb0 msgr2=0x7fc0bc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:02.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.242+0000 7fc0d5540700 1 --2- 172.21.15.189:0/1305971384 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc0bc06adb0 0x7fc0bc06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0009d20 tx=0x7fc0c0008040 comp rx=0 tx=0).stop 2024-04-06T12:28:02.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.242+0000 7fc0d5540700 1 -- 172.21.15.189:0/1305971384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0d00ce560 msgr2=0x7fc0d014ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:02.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.242+0000 7fc0d5540700 1 --2- 172.21.15.189:0/1305971384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0d00ce560 0x7fc0d014ee10 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4009fd0 tx=0x7fc0c4004750 comp rx=0 tx=0).stop 2024-04-06T12:28:02.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.242+0000 7fc0d5540700 1 -- 172.21.15.189:0/1305971384 shutdown_connections 2024-04-06T12:28:02.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.242+0000 7fc0d5540700 1 --2- 172.21.15.189:0/1305971384 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc0bc06adb0 0x7fc0bc06d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:02.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.242+0000 7fc0d5540700 1 --2- 172.21.15.189:0/1305971384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0d015d6f0 0x7fc0d0160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:02.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.242+0000 7fc0d5540700 1 --2- 172.21.15.189:0/1305971384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d00b8d70 0x7fc0d00b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:02.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.242+0000 7fc0d5540700 1 --2- 172.21.15.189:0/1305971384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0d00ce560 0x7fc0d014ee10 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:02.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.242+0000 7fc0d5540700 1 -- 172.21.15.189:0/1305971384 >> 172.21.15.189:0/1305971384 conn(0x7fc0d0072490 msgr2=0x7fc0d00b65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:02.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.243+0000 7fc0d5540700 1 -- 172.21.15.189:0/1305971384 shutdown_connections 2024-04-06T12:28:02.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.243+0000 7fc0d5540700 1 -- 172.21.15.189:0/1305971384 wait complete. 2024-04-06T12:28:02.242 INFO:teuthology.orchestra.run.smithi189.stderr: removed all blocklist entries 2024-04-06T12:28:02.256 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-06T12:28:02.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.385+0000 7f8ffa3cd700 1 Processor -- start 2024-04-06T12:28:02.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.385+0000 7f8ffa3cd700 1 -- start start 2024-04-06T12:28:02.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.386+0000 7f8ffa3cd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff415d7b0 0x7f8ff40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:02.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.386+0000 7f8ffa3cd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff40b3990 con 0x7f8ff415e500 2024-04-06T12:28:02.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.386+0000 7f8ffa3cd700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff40b3060 con 0x7f8ff406a730 2024-04-06T12:28:02.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.386+0000 7f8ffa3cd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40b3340 con 0x7f8ff415d7b0 2024-04-06T12:28:02.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.386+0000 7f8ff896a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8ff415e500 0x7f8ff40717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55398/0 (socket says 172.21.15.189:55398) 2024-04-06T12:28:02.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.386+0000 7f8ff896a700 1 -- 172.21.15.189:0/1639984070 learned_addr learned my addr 172.21.15.189:0/1639984070 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:02.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.386+0000 7f8ff2ffd700 1 -- 172.21.15.189:0/1639984070 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383850550 0 0) 0x7f8ff40b3990 con 0x7f8ff415e500 2024-04-06T12:28:02.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.386+0000 7f8ff3fff700 1 --2- 172.21.15.189:0/1639984070 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff415d7b0 0x7f8ff40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:02.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.386+0000 7f8ff2ffd700 1 -- 172.21.15.189:0/1639984070 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fdc0035b0 con 0x7f8ff415e500 2024-04-06T12:28:02.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.386+0000 7f8ff2ffd700 1 -- 172.21.15.189:0/1639984070 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3542071242 0 0) 0x7f8fdc0035b0 con 0x7f8ff415e500 2024-04-06T12:28:02.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.386+0000 7f8ff2ffd700 1 -- 172.21.15.189:0/1639984070 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff415d7b0 msgr2=0x7f8ff40b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:02.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.386+0000 7f8ff2ffd700 1 --2- 172.21.15.189:0/1639984070 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff415d7b0 0x7f8ff40b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:02.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.386+0000 7f8ff2ffd700 1 -- 172.21.15.189:0/1639984070 >> v1:172.21.15.192:6789/0 conn(0x7f8ff406a730 legacy=0x7f8ff415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:02.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.386+0000 7f8ff2ffd700 1 -- 172.21.15.189:0/1639984070 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff40b3620 con 0x7f8ff415e500 2024-04-06T12:28:02.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff2ffd700 1 -- 172.21.15.189:0/1639984070 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f8fec0019f0 con 0x7f8ff415e500 2024-04-06T12:28:02.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff2ffd700 1 -- 172.21.15.189:0/1639984070 >> v1:172.21.15.189:6789/0 conn(0x7f8ff415e500 legacy=0x7f8ff40717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:02.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff2ffd700 1 --2- 172.21.15.189:0/1639984070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fdc003a30 0x7f8fdc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:02.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff2ffd700 1 --2- 172.21.15.189:0/1639984070 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff415d7b0 0x7f8fdc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:02.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff2ffd700 1 --2- 172.21.15.189:0/1639984070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fdc004d90 0x7f8fdc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:02.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff2ffd700 1 -- 172.21.15.189:0/1639984070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40b3340 con 0x7f8fdc003a30 2024-04-06T12:28:02.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff2ffd700 1 -- 172.21.15.189:0/1639984070 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc00bb40 con 0x7f8fdc004d90 2024-04-06T12:28:02.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff2ffd700 1 -- 172.21.15.189:0/1639984070 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc00bd00 con 0x7f8ff415d7b0 2024-04-06T12:28:02.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff3fff700 1 --2- 172.21.15.189:0/1639984070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fdc003a30 0x7f8fdc003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:02.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff37fe700 1 --2- 172.21.15.189:0/1639984070 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff415d7b0 0x7f8fdc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:02.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff896a700 1 --2- 172.21.15.189:0/1639984070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fdc004d90 0x7f8fdc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:02.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff3fff700 1 -- 172.21.15.189:0/1639984070 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff415d7b0 msgr2=0x7f8fdc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:02.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff3fff700 1 --2- 172.21.15.189:0/1639984070 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff415d7b0 0x7f8fdc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:02.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff3fff700 1 -- 172.21.15.189:0/1639984070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fdc004d90 msgr2=0x7f8fdc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:02.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff3fff700 1 --2- 172.21.15.189:0/1639984070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fdc004d90 0x7f8fdc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:02.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff3fff700 1 -- 172.21.15.189:0/1639984070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fdc0035b0 con 0x7f8fdc003a30 2024-04-06T12:28:02.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff3fff700 1 --2- 172.21.15.189:0/1639984070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fdc003a30 0x7f8fdc003e50 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009b60 tx=0x7f8fdc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25ddae60625031e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:02.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff2ffd700 1 -- 172.21.15.189:0/1639984070 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe800e6c0 con 0x7f8fdc003a30 2024-04-06T12:28:02.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff2ffd700 1 -- 172.21.15.189:0/1639984070 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe800e820 con 0x7f8fdc003a30 2024-04-06T12:28:02.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.387+0000 7f8ff2ffd700 1 -- 172.21.15.189:0/1639984070 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe800eb90 con 0x7f8fdc003a30 2024-04-06T12:28:02.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.388+0000 7f8ffa3cd700 1 -- 172.21.15.189:0/1639984070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fdc003a30 msgr2=0x7f8fdc003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:02.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.388+0000 7f8ffa3cd700 1 --2- 172.21.15.189:0/1639984070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fdc003a30 0x7f8fdc003e50 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009b60 tx=0x7f8fdc005880 comp rx=0 tx=0).stop 2024-04-06T12:28:02.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.388+0000 7f8ff896a700 1 -- 172.21.15.189:0/1639984070 reap_dead start 2024-04-06T12:28:02.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.388+0000 7f8ffa3cd700 1 -- 172.21.15.189:0/1639984070 shutdown_connections 2024-04-06T12:28:02.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.388+0000 7f8ffa3cd700 1 -- 172.21.15.189:0/1639984070 >> 172.21.15.189:0/1639984070 conn(0x7f8ff40d00b0 msgr2=0x7f8ff40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:02.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.388+0000 7f8ffa3cd700 1 -- 172.21.15.189:0/1639984070 shutdown_connections 2024-04-06T12:28:02.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.388+0000 7f8ffa3cd700 1 -- 172.21.15.189:0/1639984070 wait complete. 2024-04-06T12:28:02.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.389+0000 7f8ffa3cd700 1 Processor -- start 2024-04-06T12:28:02.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.389+0000 7f8ffa3cd700 1 -- start start 2024-04-06T12:28:02.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.390+0000 7f8ffa3cd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff40b0ef0 0x7f8ff40cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:02.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.390+0000 7f8ffa3cd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ff40b12d0 0x7f8ff413eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:02.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.390+0000 7f8ffa3cd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ff413f0b0 0x7f8ff4141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:02.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.390+0000 7f8ffa3cd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40cee20 con 0x7f8ff413f0b0 2024-04-06T12:28:02.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.390+0000 7f8ffa3cd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff413f530 con 0x7f8ff40b12d0 2024-04-06T12:28:02.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.390+0000 7f8ffa3cd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40ce1a0 con 0x7f8ff40b0ef0 2024-04-06T12:28:02.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.390+0000 7f8ff896a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ff413f0b0 0x7f8ff4141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:02.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.390+0000 7f8ff896a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ff413f0b0 0x7f8ff4141850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57882/0 (socket says 172.21.15.189:57882) 2024-04-06T12:28:02.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.390+0000 7f8ff896a700 1 -- 172.21.15.189:0/2305194845 learned_addr learned my addr 172.21.15.189:0/2305194845 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:02.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.390+0000 7f8ff37fe700 1 --2- 172.21.15.189:0/2305194845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ff40b12d0 0x7f8ff413eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:02.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.390+0000 7f8ff3fff700 1 --2- 172.21.15.189:0/2305194845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff40b0ef0 0x7f8ff40cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:02.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.390+0000 7f8ff896a700 1 -- 172.21.15.189:0/2305194845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff40b0ef0 msgr2=0x7f8ff40cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:02.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.390+0000 7f8ff896a700 1 --2- 172.21.15.189:0/2305194845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff40b0ef0 0x7f8ff40cc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:02.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.390+0000 7f8ff896a700 1 -- 172.21.15.189:0/2305194845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ff40b12d0 msgr2=0x7f8ff413eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:02.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.390+0000 7f8ff896a700 1 --2- 172.21.15.189:0/2305194845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ff40b12d0 0x7f8ff413eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:02.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.390+0000 7f8ff896a700 1 -- 172.21.15.189:0/2305194845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff40ce480 con 0x7f8ff413f0b0 2024-04-06T12:28:02.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.390+0000 7f8ff3fff700 1 --2- 172.21.15.189:0/2305194845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff40b0ef0 0x7f8ff40cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:02.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.391+0000 7f8ff896a700 1 --2- 172.21.15.189:0/2305194845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ff413f0b0 0x7f8ff4141850 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f8ff406d2c0 tx=0x7f8fec00a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:02.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.391+0000 7f8ff0ff9700 1 -- 172.21.15.189:0/2305194845 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec00ad60 con 0x7f8ff413f0b0 2024-04-06T12:28:02.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.391+0000 7f8ff0ff9700 1 -- 172.21.15.189:0/2305194845 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec0133f0 con 0x7f8ff413f0b0 2024-04-06T12:28:02.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.391+0000 7f8ffa3cd700 1 -- 172.21.15.189:0/2305194845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff40ce760 con 0x7f8ff413f0b0 2024-04-06T12:28:02.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.391+0000 7f8ff0ff9700 1 -- 172.21.15.189:0/2305194845 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec0137a0 con 0x7f8ff413f0b0 2024-04-06T12:28:02.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.391+0000 7f8ffa3cd700 1 -- 172.21.15.189:0/2305194845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff40cf240 con 0x7f8ff413f0b0 2024-04-06T12:28:02.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.392+0000 7f8ffa3cd700 1 -- 172.21.15.189:0/2305194845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff40cef60 con 0x7f8ff413f0b0 2024-04-06T12:28:02.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.392+0000 7f8ff0ff9700 1 -- 172.21.15.189:0/2305194845 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f8fec013900 con 0x7f8ff413f0b0 2024-04-06T12:28:02.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.393+0000 7f8ff0ff9700 1 --2- 172.21.15.189:0/2305194845 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8fe006adb0 0x7f8fe006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:02.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.393+0000 7f8ff3fff700 1 --2- 172.21.15.189:0/2305194845 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8fe006adb0 0x7f8fe006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:02.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.393+0000 7f8ff0ff9700 1 -- 172.21.15.189:0/2305194845 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f8fec089b00 con 0x7f8ff413f0b0 2024-04-06T12:28:02.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.393+0000 7f8ff3fff700 1 --2- 172.21.15.189:0/2305194845 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8fe006adb0 0x7f8fe006d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8ff406c7a0 tx=0x7f8fe801e040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:02.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.397+0000 7f8ff0ff9700 1 -- 172.21.15.189:0/2305194845 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8ff40cef60 con 0x7f8ff413f0b0 2024-04-06T12:28:02.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.672+0000 7f8ffa3cd700 1 -- 172.21.15.189:0/2305194845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8ff40cef60 con 0x7f8ff413f0b0 2024-04-06T12:28:02.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.672+0000 7f8ff0ff9700 1 -- 172.21.15.189:0/2305194845 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f8ff40cef60 con 0x7f8ff413f0b0 2024-04-06T12:28:02.673 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:02.674 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T12:28:02.674 INFO:teuthology.orchestra.run.smithi189.stdout: "auth_dump": [ 2024-04-06T12:28:02.674 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:02.674 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.a", 2024-04-06T12:28:02.674 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmTjYMNBAA5voWGV4Pa7pE27VogrdipQ==", 2024-04-06T12:28:02.674 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:02.674 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:28:02.674 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:28:02.674 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:28:02.674 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:02.674 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.675 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:02.675 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:02.675 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.b", 2024-04-06T12:28:02.675 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm9A1OBRAAnHzubCvRUX8UdW8hmDzKbQ==", 2024-04-06T12:28:02.675 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:02.675 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:28:02.675 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:28:02.675 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:28:02.675 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:02.675 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.675 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:02.675 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:02.675 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.c", 2024-04-06T12:28:02.675 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmz0hhOxAAD3/GINNmGoFk+oPyiHHJXg==", 2024-04-06T12:28:02.676 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:02.676 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:28:02.676 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:28:02.676 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:28:02.676 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:02.676 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.676 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:02.676 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:02.676 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.d", 2024-04-06T12:28:02.676 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmhoAeDRAAoQC0vicWumBRTwpi3PtYaw==", 2024-04-06T12:28:02.676 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:02.676 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:28:02.676 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:28:02.676 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:28:02.676 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:02.677 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.677 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:02.677 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:02.677 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.0", 2024-04-06T12:28:02.677 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDkOxFmkxLWJhAAFXewCNOFX7dhAZVgUmewpQ==", 2024-04-06T12:28:02.677 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:02.677 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:28:02.677 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:28:02.677 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:02.677 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.677 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:02.677 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:02.677 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.1", 2024-04-06T12:28:02.677 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDnOxFmHYqWJRAABlsVCiv3e7YJMbfodJTWag==", 2024-04-06T12:28:02.677 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:02.678 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:28:02.678 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:28:02.678 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:02.678 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.678 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:02.678 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:02.678 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.2", 2024-04-06T12:28:02.678 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDqOxFmkhcvJBAA8NMEpHVGbGALjV9RISKP8g==", 2024-04-06T12:28:02.678 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:02.678 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:28:02.678 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:28:02.678 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:02.678 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.678 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:02.679 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:02.679 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.3", 2024-04-06T12:28:02.679 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDtOxFmDGzhIxAAamBoAbE8+lAel4os67meyQ==", 2024-04-06T12:28:02.679 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:02.679 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:28:02.679 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:28:02.679 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:02.679 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.679 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:02.679 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:02.679 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.4", 2024-04-06T12:28:02.679 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDzOxFmobZDJxAA9l8wdS81Eyw6+17I5ac6bw==", 2024-04-06T12:28:02.679 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:02.680 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:28:02.680 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:28:02.680 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:02.680 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.680 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:02.680 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:02.680 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.5", 2024-04-06T12:28:02.680 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD2OxFmi+ruJRAASlQ0o9jX/wA7U5AosnL+Wg==", 2024-04-06T12:28:02.680 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:02.680 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:28:02.680 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:28:02.680 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:02.680 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.680 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:02.680 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:02.681 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.6", 2024-04-06T12:28:02.681 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD5OxFmbwuuJBAA0wqJRmnJyskQPZJ7/LnpvA==", 2024-04-06T12:28:02.681 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:02.681 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:28:02.681 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:28:02.681 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:02.681 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.681 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:02.681 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:02.681 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.7", 2024-04-06T12:28:02.681 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD8OxFmDmRgIxAAzyW9AGAbmb67ZvB5qunwLQ==", 2024-04-06T12:28:02.681 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:02.681 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:28:02.682 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:28:02.682 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:02.682 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.682 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:02.682 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:02.682 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.0", 2024-04-06T12:28:02.682 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm0ByJERAAoEV8Yfc09+Go6u95XafElQ==", 2024-04-06T12:28:02.682 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:02.682 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:28:02.682 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow r", 2024-04-06T12:28:02.682 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-06T12:28:02.682 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.682 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:02.682 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:02.682 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.1", 2024-04-06T12:28:02.683 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmSV5lGBAAavDEo2LGy0IHpWSAuitHXg==", 2024-04-06T12:28:02.683 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:02.683 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:28:02.683 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:28:02.683 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:28:02.683 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:28:02.683 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.683 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:02.683 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:02.683 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.2", 2024-04-06T12:28:02.683 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm/Gz5FBAA3xX/rsAyWZ1JoUVlQlBIvQ==", 2024-04-06T12:28:02.683 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:02.683 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:28:02.683 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:28:02.684 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:28:02.684 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:28:02.684 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.684 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:02.684 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:02.684 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.3", 2024-04-06T12:28:02.684 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmzra7GxAADbnzIvWnRTo65LvPwtxWrg==", 2024-04-06T12:28:02.684 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:02.684 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:28:02.684 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:28:02.684 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:28:02.684 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:28:02.684 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.684 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:02.684 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:02.685 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.admin", 2024-04-06T12:28:02.685 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmxG+wCBAANg5kt6TOUQdj3/X0Mqgx1g==", 2024-04-06T12:28:02.685 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:02.685 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:28:02.685 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:28:02.685 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:28:02.685 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:02.685 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.685 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:02.685 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:02.685 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.x", 2024-04-06T12:28:02.685 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmPLyBMBAAYfhkI9Z1CPlf1Tijm+L7xA==", 2024-04-06T12:28:02.685 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:02.685 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:28:02.686 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T12:28:02.686 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:02.686 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.686 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:02.686 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:02.686 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.y", 2024-04-06T12:28:02.686 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFm1nHJHhAAQWRe8cU8duWuUJiYaDeTaQ==", 2024-04-06T12:28:02.686 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:02.686 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:28:02.686 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T12:28:02.686 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:02.686 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.686 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:02.687 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-04-06T12:28:02.687 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T12:28:02.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.674+0000 7f8ffa3cd700 1 -- 172.21.15.189:0/2305194845 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8fe006adb0 msgr2=0x7f8fe006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:02.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.674+0000 7f8ffa3cd700 1 --2- 172.21.15.189:0/2305194845 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8fe006adb0 0x7f8fe006d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8ff406c7a0 tx=0x7f8fe801e040 comp rx=0 tx=0).stop 2024-04-06T12:28:02.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.674+0000 7f8ffa3cd700 1 -- 172.21.15.189:0/2305194845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ff413f0b0 msgr2=0x7f8ff4141850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:02.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.674+0000 7f8ffa3cd700 1 --2- 172.21.15.189:0/2305194845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ff413f0b0 0x7f8ff4141850 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f8ff406d2c0 tx=0x7f8fec00a5f0 comp rx=0 tx=0).stop 2024-04-06T12:28:02.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.674+0000 7f8ffa3cd700 1 -- 172.21.15.189:0/2305194845 shutdown_connections 2024-04-06T12:28:02.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.674+0000 7f8ffa3cd700 1 --2- 172.21.15.189:0/2305194845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ff413f0b0 0x7f8ff4141850 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:02.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.674+0000 7f8ffa3cd700 1 --2- 172.21.15.189:0/2305194845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff40b0ef0 0x7f8ff40cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:02.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.674+0000 7f8ffa3cd700 1 --2- 172.21.15.189:0/2305194845 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8fe006adb0 0x7f8fe006d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:02.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.674+0000 7f8ffa3cd700 1 --2- 172.21.15.189:0/2305194845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ff40b12d0 0x7f8ff413eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:02.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.674+0000 7f8ffa3cd700 1 -- 172.21.15.189:0/2305194845 >> 172.21.15.189:0/2305194845 conn(0x7f8ff40d00b0 msgr2=0x7f8ff406ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:02.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.675+0000 7f8ffa3cd700 1 -- 172.21.15.189:0/2305194845 shutdown_connections 2024-04-06T12:28:02.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.675+0000 7f8ffa3cd700 1 -- 172.21.15.189:0/2305194845 wait complete. 2024-04-06T12:28:02.689 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:28:02.800 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-04-06T12:28:02.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.814+0000 7fb883da6700 1 Processor -- start 2024-04-06T12:28:02.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.814+0000 7fb883da6700 1 -- start start 2024-04-06T12:28:02.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.815+0000 7fb883da6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb87c15d7b0 0x7fb87c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:02.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.815+0000 7fb883da6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb87c06a730 0x7fb87c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:02.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.815+0000 7fb883da6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87c0b3990 con 0x7fb87c15ddf0 2024-04-06T12:28:02.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.815+0000 7fb883da6700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb87c06b0f0 con 0x7fb87c06a730 2024-04-06T12:28:02.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.815+0000 7fb883da6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0b3060 con 0x7fb87c15d7b0 2024-04-06T12:28:02.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.815+0000 7fb881341700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb87c06a730 0x7fb87c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:02.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.815+0000 7fb882343700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb87c15ddf0 0x7fb87c15e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55400/0 (socket says 172.21.15.189:55400) 2024-04-06T12:28:02.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.815+0000 7fb882343700 1 -- 172.21.15.189:0/1716109605 learned_addr learned my addr 172.21.15.189:0/1716109605 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:02.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.816+0000 7fb881b42700 1 --2- 172.21.15.189:0/1716109605 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb87c15d7b0 0x7fb87c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.816+0000 7fb881341700 1 -- 172.21.15.189:0/1716109605 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb87c15d7b0 msgr2=0x7fb87c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.816+0000 7fb881341700 1 --2- 172.21.15.189:0/1716109605 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb87c15d7b0 0x7fb87c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.816+0000 7fb881341700 1 -- 172.21.15.189:0/1716109605 >> v1:172.21.15.189:6789/0 conn(0x7fb87c15ddf0 legacy=0x7fb87c15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.816+0000 7fb881341700 1 -- 172.21.15.189:0/1716109605 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c0b3340 con 0x7fb87c06a730 2024-04-06T12:28:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.816+0000 7fb880b40700 1 -- 172.21.15.189:0/1716109605 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 815091394 0 0) 0x7fb87c0b3990 con 0x7fb87c15ddf0 2024-04-06T12:28:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.816+0000 7fb881b42700 1 --2- 172.21.15.189:0/1716109605 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb87c15d7b0 0x7fb87c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:28:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.816+0000 7fb881341700 1 --2- 172.21.15.189:0/1716109605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb87c06a730 0x7fb87c06abb0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fb878009840 tx=0x7fb878009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a6e2dec8572054 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.816+0000 7fb880b40700 1 -- 172.21.15.189:0/1716109605 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb878011830 con 0x7fb87c06a730 2024-04-06T12:28:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.816+0000 7fb880b40700 1 -- 172.21.15.189:0/1716109605 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb878011990 con 0x7fb87c06a730 2024-04-06T12:28:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.816+0000 7fb880b40700 1 -- 172.21.15.189:0/1716109605 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb878011d20 con 0x7fb87c06a730 2024-04-06T12:28:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.817+0000 7fb883da6700 1 -- 172.21.15.189:0/1716109605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb87c06a730 msgr2=0x7fb87c06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.817+0000 7fb883da6700 1 --2- 172.21.15.189:0/1716109605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb87c06a730 0x7fb87c06abb0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fb878009840 tx=0x7fb878009b50 comp rx=0 tx=0).stop 2024-04-06T12:28:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.817+0000 7fb883da6700 1 -- 172.21.15.189:0/1716109605 shutdown_connections 2024-04-06T12:28:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.817+0000 7fb883da6700 1 --2- 172.21.15.189:0/1716109605 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb87c15d7b0 0x7fb87c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.817+0000 7fb883da6700 1 --2- 172.21.15.189:0/1716109605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb87c06a730 0x7fb87c06abb0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.817+0000 7fb883da6700 1 -- 172.21.15.189:0/1716109605 >> 172.21.15.189:0/1716109605 conn(0x7fb87c0d00b0 msgr2=0x7fb87c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.817+0000 7fb883da6700 1 -- 172.21.15.189:0/1716109605 shutdown_connections 2024-04-06T12:28:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.817+0000 7fb883da6700 1 -- 172.21.15.189:0/1716109605 wait complete. 2024-04-06T12:28:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.818+0000 7fb883da6700 1 Processor -- start 2024-04-06T12:28:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.818+0000 7fb883da6700 1 -- start start 2024-04-06T12:28:02.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb883da6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb87c06a730 0x7fb87c14ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:02.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb883da6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb87c15d7b0 0x7fb87c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:02.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb883da6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb87c15ddf0 0x7fb87c0c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:02.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb883da6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0c53f0 con 0x7fb87c15ddf0 2024-04-06T12:28:02.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb883da6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0c56d0 con 0x7fb87c06a730 2024-04-06T12:28:02.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb883da6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0c59b0 con 0x7fb87c15d7b0 2024-04-06T12:28:02.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb882343700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb87c15ddf0 0x7fb87c0c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:02.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb882343700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb87c15ddf0 0x7fb87c0c79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57902/0 (socket says 172.21.15.189:57902) 2024-04-06T12:28:02.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb882343700 1 -- 172.21.15.189:0/922433777 learned_addr learned my addr 172.21.15.189:0/922433777 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:02.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb881b42700 1 --2- 172.21.15.189:0/922433777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb87c06a730 0x7fb87c14ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:02.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb881341700 1 --2- 172.21.15.189:0/922433777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb87c15d7b0 0x7fb87c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:02.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb882343700 1 -- 172.21.15.189:0/922433777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb87c15d7b0 msgr2=0x7fb87c15b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:02.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb882343700 1 --2- 172.21.15.189:0/922433777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb87c15d7b0 0x7fb87c15b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:02.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb882343700 1 -- 172.21.15.189:0/922433777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb87c06a730 msgr2=0x7fb87c14ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:02.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb882343700 1 --2- 172.21.15.189:0/922433777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb87c06a730 0x7fb87c14ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:02.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb882343700 1 -- 172.21.15.189:0/922433777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8780094a0 con 0x7fb87c15ddf0 2024-04-06T12:28:02.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb882343700 1 --2- 172.21.15.189:0/922433777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb87c15ddf0 0x7fb87c0c79c0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fb87400cb60 tx=0x7fb87400ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:02.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb86e7fc700 1 -- 172.21.15.189:0/922433777 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87400a9e0 con 0x7fb87c15ddf0 2024-04-06T12:28:02.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb86e7fc700 1 -- 172.21.15.189:0/922433777 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87400ab40 con 0x7fb87c15ddf0 2024-04-06T12:28:02.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb883da6700 1 -- 172.21.15.189:0/922433777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87c0c5c90 con 0x7fb87c15ddf0 2024-04-06T12:28:02.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb86e7fc700 1 -- 172.21.15.189:0/922433777 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8740070d0 con 0x7fb87c15ddf0 2024-04-06T12:28:02.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.819+0000 7fb883da6700 1 -- 172.21.15.189:0/922433777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87c0c5050 con 0x7fb87c15ddf0 2024-04-06T12:28:02.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.820+0000 7fb883da6700 1 -- 172.21.15.189:0/922433777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb87c0c61b0 con 0x7fb87c15ddf0 2024-04-06T12:28:02.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.821+0000 7fb86e7fc700 1 -- 172.21.15.189:0/922433777 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fb874007230 con 0x7fb87c15ddf0 2024-04-06T12:28:02.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.821+0000 7fb86e7fc700 1 --2- 172.21.15.189:0/922433777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb86006adb0 0x7fb86006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:02.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.821+0000 7fb86e7fc700 1 -- 172.21.15.189:0/922433777 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fb874088c80 con 0x7fb87c15ddf0 2024-04-06T12:28:02.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.821+0000 7fb881b42700 1 --2- 172.21.15.189:0/922433777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb86006adb0 0x7fb86006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:02.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.822+0000 7fb881b42700 1 --2- 172.21.15.189:0/922433777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb86006adb0 0x7fb86006d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb870006fd0 tx=0x7fb870008040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:02.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:02.826+0000 7fb86e7fc700 1 -- 172.21.15.189:0/922433777 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb874053f60 con 0x7fb87c15ddf0 2024-04-06T12:28:03.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.021+0000 7fb883da6700 1 -- 172.21.15.189:0/922433777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb87c067bb0 con 0x7fb87c15ddf0 2024-04-06T12:28:03.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.022+0000 7fb86e7fc700 1 -- 172.21.15.189:0/922433777 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+12643 (secure 0 0 0) 0x7fb874018070 con 0x7fb87c15ddf0 2024-04-06T12:28:03.021 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:03.021 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":94,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:28:02.237291+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:28:03.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.024+0000 7fb883da6700 1 -- 172.21.15.189:0/922433777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb86006adb0 msgr2=0x7fb86006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:03.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.024+0000 7fb883da6700 1 --2- 172.21.15.189:0/922433777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb86006adb0 0x7fb86006d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb870006fd0 tx=0x7fb870008040 comp rx=0 tx=0).stop 2024-04-06T12:28:03.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.024+0000 7fb883da6700 1 -- 172.21.15.189:0/922433777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb87c15ddf0 msgr2=0x7fb87c0c79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:03.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.024+0000 7fb883da6700 1 --2- 172.21.15.189:0/922433777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb87c15ddf0 0x7fb87c0c79c0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fb87400cb60 tx=0x7fb87400ce70 comp rx=0 tx=0).stop 2024-04-06T12:28:03.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.024+0000 7fb883da6700 1 -- 172.21.15.189:0/922433777 shutdown_connections 2024-04-06T12:28:03.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.024+0000 7fb883da6700 1 --2- 172.21.15.189:0/922433777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb87c15ddf0 0x7fb87c0c79c0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.024+0000 7fb883da6700 1 --2- 172.21.15.189:0/922433777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb86006adb0 0x7fb86006d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.024+0000 7fb883da6700 1 --2- 172.21.15.189:0/922433777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb87c06a730 0x7fb87c14ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.024+0000 7fb883da6700 1 --2- 172.21.15.189:0/922433777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb87c15d7b0 0x7fb87c15b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.024+0000 7fb883da6700 1 -- 172.21.15.189:0/922433777 >> 172.21.15.189:0/922433777 conn(0x7fb87c0d00b0 msgr2=0x7fb87c0ae1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:03.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.025+0000 7fb883da6700 1 -- 172.21.15.189:0/922433777 shutdown_connections 2024-04-06T12:28:03.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.025+0000 7fb883da6700 1 -- 172.21.15.189:0/922433777 wait complete. 2024-04-06T12:28:03.035 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:28:03.036 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:28:03.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.162+0000 7f104829c700 1 Processor -- start 2024-04-06T12:28:03.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.162+0000 7f104829c700 1 -- start start 2024-04-06T12:28:03.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.163+0000 7f104829c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f10400d11b0 0x7f10400d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.163+0000 7f104829c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10400cb870 0x7f104015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.163+0000 7f104829c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f10400cfa00 0x7f10400cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.163+0000 7f104829c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10400ccdc0 con 0x7f10400d11b0 2024-04-06T12:28:03.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.163+0000 7f104829c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10400d0b50 con 0x7f10400cfa00 2024-04-06T12:28:03.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.163+0000 7f104829c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10400d1ad0 con 0x7f10400cb870 2024-04-06T12:28:03.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.163+0000 7f1046038700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f10400d11b0 0x7f10400d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.163+0000 7f1046038700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f10400d11b0 0x7f10400d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57912/0 (socket says 172.21.15.189:57912) 2024-04-06T12:28:03.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.163+0000 7f1046038700 1 -- 172.21.15.189:0/356056211 learned_addr learned my addr 172.21.15.189:0/356056211 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:03.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.163+0000 7f1046839700 1 --2- 172.21.15.189:0/356056211 >> v2:172.21.15.192:3300/0 conn(0x7f10400cfa00 0x7f10400cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.163+0000 7f1045837700 1 --2- 172.21.15.189:0/356056211 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10400cb870 0x7f104015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.164+0000 7f1046038700 1 -- 172.21.15.189:0/356056211 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10400cb870 msgr2=0x7f104015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:03.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.164+0000 7f1046038700 1 --2- 172.21.15.189:0/356056211 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10400cb870 0x7f104015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.164+0000 7f1046038700 1 -- 172.21.15.189:0/356056211 >> v2:172.21.15.192:3300/0 conn(0x7f10400cfa00 msgr2=0x7f10400cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:03.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.164+0000 7f1046038700 1 --2- 172.21.15.189:0/356056211 >> v2:172.21.15.192:3300/0 conn(0x7f10400cfa00 0x7f10400cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.164+0000 7f1046038700 1 -- 172.21.15.189:0/356056211 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10400cbd80 con 0x7f10400d11b0 2024-04-06T12:28:03.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.164+0000 7f1046038700 1 --2- 172.21.15.189:0/356056211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10400d11b0 0x7f10400d1590 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f10400ccd10 tx=0x7f103c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be2df7a87f07a079 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:03.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.164+0000 7f1045036700 1 -- 172.21.15.189:0/356056211 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c011830 con 0x7f10400d11b0 2024-04-06T12:28:03.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.164+0000 7f1045036700 1 -- 172.21.15.189:0/356056211 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f103c011990 con 0x7f10400d11b0 2024-04-06T12:28:03.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.164+0000 7f1045036700 1 -- 172.21.15.189:0/356056211 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c011d20 con 0x7f10400d11b0 2024-04-06T12:28:03.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.164+0000 7f104829c700 1 -- 172.21.15.189:0/356056211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10400d11b0 msgr2=0x7f10400d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:03.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.164+0000 7f104829c700 1 --2- 172.21.15.189:0/356056211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10400d11b0 0x7f10400d1590 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f10400ccd10 tx=0x7f103c009bd0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.164+0000 7f104829c700 1 -- 172.21.15.189:0/356056211 shutdown_connections 2024-04-06T12:28:03.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.164+0000 7f104829c700 1 --2- 172.21.15.189:0/356056211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10400d11b0 0x7f10400d1590 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.164+0000 7f104829c700 1 --2- 172.21.15.189:0/356056211 >> v2:172.21.15.192:3300/0 conn(0x7f10400cfa00 0x7f10400cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.165+0000 7f104829c700 1 --2- 172.21.15.189:0/356056211 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10400cb870 0x7f104015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.165+0000 7f104829c700 1 -- 172.21.15.189:0/356056211 >> 172.21.15.189:0/356056211 conn(0x7f10400b2db0 msgr2=0x7f10400b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:03.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.165+0000 7f104829c700 1 -- 172.21.15.189:0/356056211 shutdown_connections 2024-04-06T12:28:03.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.165+0000 7f104829c700 1 -- 172.21.15.189:0/356056211 wait complete. 2024-04-06T12:28:03.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.166+0000 7f104829c700 1 Processor -- start 2024-04-06T12:28:03.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.166+0000 7f104829c700 1 -- start start 2024-04-06T12:28:03.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.166+0000 7f104829c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10400cb870 0x7f1040155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.166+0000 7f104829c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10400cfa00 0x7f104006d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.166+0000 7f104829c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10400d11b0 0x7f10401e6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.166+0000 7f104829c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104006c810 con 0x7f10400cfa00 2024-04-06T12:28:03.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.166+0000 7f104829c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104006bb90 con 0x7f10400d11b0 2024-04-06T12:28:03.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.167+0000 7f104829c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f104006be70 con 0x7f10400cb870 2024-04-06T12:28:03.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.167+0000 7f1045837700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10400cfa00 0x7f104006d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.167+0000 7f1045837700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10400cfa00 0x7f104006d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57916/0 (socket says 172.21.15.189:57916) 2024-04-06T12:28:03.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.167+0000 7f1045837700 1 -- 172.21.15.189:0/3266294905 learned_addr learned my addr 172.21.15.189:0/3266294905 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:03.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.167+0000 7f1046038700 1 --2- 172.21.15.189:0/3266294905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10400cb870 0x7f1040155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.167+0000 7f1046839700 1 --2- 172.21.15.189:0/3266294905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10400d11b0 0x7f10401e6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.167+0000 7f1045837700 1 -- 172.21.15.189:0/3266294905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10400cb870 msgr2=0x7f1040155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:03.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.167+0000 7f1045837700 1 --2- 172.21.15.189:0/3266294905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10400cb870 0x7f1040155920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.167+0000 7f1045837700 1 -- 172.21.15.189:0/3266294905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10400d11b0 msgr2=0x7f10401e6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:03.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.167+0000 7f1045837700 1 --2- 172.21.15.189:0/3266294905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10400d11b0 0x7f10401e6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.167+0000 7f1045837700 1 -- 172.21.15.189:0/3266294905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103c009550 con 0x7f10400cfa00 2024-04-06T12:28:03.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.167+0000 7f1045837700 1 --2- 172.21.15.189:0/3266294905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10400cfa00 0x7f104006d0b0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f103000e840 tx=0x7f103000eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:03.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.168+0000 7f1036ffd700 1 -- 172.21.15.189:0/3266294905 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1030013680 con 0x7f10400cfa00 2024-04-06T12:28:03.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.168+0000 7f104829c700 1 -- 172.21.15.189:0/3266294905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f104006c150 con 0x7f10400cfa00 2024-04-06T12:28:03.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.168+0000 7f104829c700 1 -- 172.21.15.189:0/3266294905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f104006b840 con 0x7f10400cfa00 2024-04-06T12:28:03.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.168+0000 7f1036ffd700 1 -- 172.21.15.189:0/3266294905 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10300137e0 con 0x7f10400cfa00 2024-04-06T12:28:03.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.168+0000 7f1036ffd700 1 -- 172.21.15.189:0/3266294905 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1030013b50 con 0x7f10400cfa00 2024-04-06T12:28:03.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.169+0000 7f104829c700 1 -- 172.21.15.189:0/3266294905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1040074990 con 0x7f10400cfa00 2024-04-06T12:28:03.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.169+0000 7f1036ffd700 1 -- 172.21.15.189:0/3266294905 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f1030013cb0 con 0x7f10400cfa00 2024-04-06T12:28:03.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.169+0000 7f1036ffd700 1 --2- 172.21.15.189:0/3266294905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f102406adb0 0x7f102406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.169+0000 7f1036ffd700 1 -- 172.21.15.189:0/3266294905 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f1030089330 con 0x7f10400cfa00 2024-04-06T12:28:03.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.170+0000 7f1046038700 1 --2- 172.21.15.189:0/3266294905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f102406adb0 0x7f102406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.170+0000 7f1046038700 1 --2- 172.21.15.189:0/3266294905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f102406adb0 0x7f102406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f103c009c10 tx=0x7f103c008030 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:03.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.174+0000 7f1036ffd700 1 -- 172.21.15.189:0/3266294905 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1030054620 con 0x7f10400cfa00 2024-04-06T12:28:03.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.370+0000 7f104829c700 1 -- 172.21.15.189:0/3266294905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f10400b0a00 con 0x7f10400cfa00 2024-04-06T12:28:03.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.371+0000 7f1036ffd700 1 -- 172.21.15.189:0/3266294905 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1030057c40 con 0x7f10400cfa00 2024-04-06T12:28:03.369 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:28:03.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.372+0000 7f104829c700 1 -- 172.21.15.189:0/3266294905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f102406adb0 msgr2=0x7f102406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:03.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.372+0000 7f104829c700 1 --2- 172.21.15.189:0/3266294905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f102406adb0 0x7f102406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f103c009c10 tx=0x7f103c008030 comp rx=0 tx=0).stop 2024-04-06T12:28:03.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.373+0000 7f104829c700 1 -- 172.21.15.189:0/3266294905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10400cfa00 msgr2=0x7f104006d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:03.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.373+0000 7f104829c700 1 --2- 172.21.15.189:0/3266294905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10400cfa00 0x7f104006d0b0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f103000e840 tx=0x7f103000eb50 comp rx=0 tx=0).stop 2024-04-06T12:28:03.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.373+0000 7f104829c700 1 -- 172.21.15.189:0/3266294905 shutdown_connections 2024-04-06T12:28:03.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.373+0000 7f104829c700 1 --2- 172.21.15.189:0/3266294905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f102406adb0 0x7f102406d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.373+0000 7f104829c700 1 --2- 172.21.15.189:0/3266294905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10400d11b0 0x7f10401e6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.373+0000 7f104829c700 1 --2- 172.21.15.189:0/3266294905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10400cb870 0x7f1040155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.373+0000 7f104829c700 1 --2- 172.21.15.189:0/3266294905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10400cfa00 0x7f104006d0b0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.373+0000 7f104829c700 1 -- 172.21.15.189:0/3266294905 >> 172.21.15.189:0/3266294905 conn(0x7f10400b2db0 msgr2=0x7f104015a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:03.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.373+0000 7f104829c700 1 -- 172.21.15.189:0/3266294905 shutdown_connections 2024-04-06T12:28:03.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.373+0000 7f104829c700 1 -- 172.21.15.189:0/3266294905 wait complete. 2024-04-06T12:28:03.383 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-06T12:28:03.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.506+0000 7fb2cbdda700 1 Processor -- start 2024-04-06T12:28:03.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.506+0000 7fb2cbdda700 1 -- start start 2024-04-06T12:28:03.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2cbdda700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb2c406ae40 0x7fb2c40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2cbdda700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2c415ddf0 0x7fb2c415e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2cbdda700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c40b3990 con 0x7fb2c415d7b0 2024-04-06T12:28:03.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2cbdda700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2c40b3060 con 0x7fb2c406ae40 2024-04-06T12:28:03.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2cbdda700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c40b3340 con 0x7fb2c415ddf0 2024-04-06T12:28:03.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2c9b76700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb2c415d7b0 0x7fb2c406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55412/0 (socket says 172.21.15.189:55412) 2024-04-06T12:28:03.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2c9b76700 1 -- 172.21.15.189:0/3415423998 learned_addr learned my addr 172.21.15.189:0/3415423998 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:03.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2c9375700 1 --2- 172.21.15.189:0/3415423998 >> v2:172.21.15.192:3300/0 conn(0x7fb2c406ae40 0x7fb2c40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2ca377700 1 --2- 172.21.15.189:0/3415423998 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2c415ddf0 0x7fb2c415e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2c8b74700 1 -- 172.21.15.189:0/3415423998 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246421570 0 0) 0x7fb2c40b3990 con 0x7fb2c415d7b0 2024-04-06T12:28:03.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2c8b74700 1 -- 172.21.15.189:0/3415423998 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b00035b0 con 0x7fb2c415d7b0 2024-04-06T12:28:03.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2c8b74700 1 -- 172.21.15.189:0/3415423998 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3049022512 0 0) 0x7fb2b00035b0 con 0x7fb2c415d7b0 2024-04-06T12:28:03.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2c8b74700 1 -- 172.21.15.189:0/3415423998 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2c415ddf0 msgr2=0x7fb2c415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:03.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2c8b74700 1 --2- 172.21.15.189:0/3415423998 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2c415ddf0 0x7fb2c415e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2c8b74700 1 -- 172.21.15.189:0/3415423998 >> v2:172.21.15.192:3300/0 conn(0x7fb2c406ae40 msgr2=0x7fb2c40b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:03.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2c8b74700 1 --2- 172.21.15.189:0/3415423998 >> v2:172.21.15.192:3300/0 conn(0x7fb2c406ae40 0x7fb2c40b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2c8b74700 1 -- 172.21.15.189:0/3415423998 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c40b3620 con 0x7fb2c415d7b0 2024-04-06T12:28:03.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2c8b74700 1 -- 172.21.15.189:0/3415423998 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fb2c0001670 con 0x7fb2c415d7b0 2024-04-06T12:28:03.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2c8b74700 1 -- 172.21.15.189:0/3415423998 >> v1:172.21.15.189:6789/0 conn(0x7fb2c415d7b0 legacy=0x7fb2c406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:03.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2c8b74700 1 --2- 172.21.15.189:0/3415423998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2b0003a00 0x7fb2b0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2c8b74700 1 --2- 172.21.15.189:0/3415423998 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2c415ddf0 0x7fb2b0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2c8b74700 1 --2- 172.21.15.189:0/3415423998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2b0004d90 0x7fb2b0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2c8b74700 1 -- 172.21.15.189:0/3415423998 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c40b3060 con 0x7fb2b0004d90 2024-04-06T12:28:03.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2c8b74700 1 -- 172.21.15.189:0/3415423998 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c40b3340 con 0x7fb2b0003a00 2024-04-06T12:28:03.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.507+0000 7fb2c8b74700 1 -- 172.21.15.189:0/3415423998 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b000ba10 con 0x7fb2c415ddf0 2024-04-06T12:28:03.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.508+0000 7fb2c9b76700 1 --2- 172.21.15.189:0/3415423998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2b0003a00 0x7fb2b0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.508+0000 7fb2c9375700 1 --2- 172.21.15.189:0/3415423998 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2c415ddf0 0x7fb2b0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.508+0000 7fb2ca377700 1 --2- 172.21.15.189:0/3415423998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2b0004d90 0x7fb2b0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.508+0000 7fb2ca377700 1 -- 172.21.15.189:0/3415423998 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2c415ddf0 msgr2=0x7fb2b0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:03.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.508+0000 7fb2ca377700 1 --2- 172.21.15.189:0/3415423998 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2c415ddf0 0x7fb2b0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.508+0000 7fb2ca377700 1 -- 172.21.15.189:0/3415423998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2b0003a00 msgr2=0x7fb2b0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:03.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.508+0000 7fb2ca377700 1 --2- 172.21.15.189:0/3415423998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2b0003a00 0x7fb2b0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.508+0000 7fb2ca377700 1 -- 172.21.15.189:0/3415423998 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b00035b0 con 0x7fb2b0004d90 2024-04-06T12:28:03.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.508+0000 7fb2ca377700 1 --2- 172.21.15.189:0/3415423998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2b0004d90 0x7fb2b0005210 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc009fc0 tx=0x7fb2b0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=352a576b88fa0168 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:03.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.508+0000 7fb2c8b74700 1 -- 172.21.15.189:0/3415423998 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc00ec30 con 0x7fb2b0004d90 2024-04-06T12:28:03.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.508+0000 7fb2c8b74700 1 -- 172.21.15.189:0/3415423998 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2bc00f040 con 0x7fb2b0004d90 2024-04-06T12:28:03.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.508+0000 7fb2c8b74700 1 -- 172.21.15.189:0/3415423998 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc014480 con 0x7fb2b0004d90 2024-04-06T12:28:03.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.509+0000 7fb2cbdda700 1 -- 172.21.15.189:0/3415423998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2b0004d90 msgr2=0x7fb2b0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:03.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.509+0000 7fb2cbdda700 1 --2- 172.21.15.189:0/3415423998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2b0004d90 0x7fb2b0005210 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc009fc0 tx=0x7fb2b0005880 comp rx=0 tx=0).stop 2024-04-06T12:28:03.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.509+0000 7fb2ca377700 1 -- 172.21.15.189:0/3415423998 reap_dead start 2024-04-06T12:28:03.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.509+0000 7fb2cbdda700 1 -- 172.21.15.189:0/3415423998 shutdown_connections 2024-04-06T12:28:03.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.509+0000 7fb2cbdda700 1 -- 172.21.15.189:0/3415423998 >> 172.21.15.189:0/3415423998 conn(0x7fb2c40d00b0 msgr2=0x7fb2c40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:03.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.509+0000 7fb2cbdda700 1 -- 172.21.15.189:0/3415423998 shutdown_connections 2024-04-06T12:28:03.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.509+0000 7fb2cbdda700 1 -- 172.21.15.189:0/3415423998 wait complete. 2024-04-06T12:28:03.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.510+0000 7fb2cbdda700 1 Processor -- start 2024-04-06T12:28:03.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.510+0000 7fb2cbdda700 1 -- start start 2024-04-06T12:28:03.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2cbdda700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2c40b1070 0x7fb2c4159830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2cbdda700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2c40b1450 0x7fb2c40ce9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2cbdda700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2c40cd270 0x7fb2c41f5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2cbdda700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c40cef10 con 0x7fb2c40cd270 2024-04-06T12:28:03.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2cbdda700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c40cf140 con 0x7fb2c40b1450 2024-04-06T12:28:03.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2cbdda700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c40cc4b0 con 0x7fb2c40b1070 2024-04-06T12:28:03.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2ca377700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2c40cd270 0x7fb2c41f5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2ca377700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2c40cd270 0x7fb2c41f5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57922/0 (socket says 172.21.15.189:57922) 2024-04-06T12:28:03.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2ca377700 1 -- 172.21.15.189:0/278700380 learned_addr learned my addr 172.21.15.189:0/278700380 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:03.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2c9375700 1 --2- 172.21.15.189:0/278700380 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2c40b1450 0x7fb2c40ce9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2c9b76700 1 --2- 172.21.15.189:0/278700380 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2c40b1070 0x7fb2c4159830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2ca377700 1 -- 172.21.15.189:0/278700380 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2c40b1070 msgr2=0x7fb2c4159830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:03.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2ca377700 1 --2- 172.21.15.189:0/278700380 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2c40b1070 0x7fb2c4159830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2ca377700 1 -- 172.21.15.189:0/278700380 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2c40b1450 msgr2=0x7fb2c40ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:03.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2ca377700 1 --2- 172.21.15.189:0/278700380 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2c40b1450 0x7fb2c40ce9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2ca377700 1 -- 172.21.15.189:0/278700380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c40cc790 con 0x7fb2c40cd270 2024-04-06T12:28:03.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2c9375700 1 --2- 172.21.15.189:0/278700380 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2c40b1450 0x7fb2c40ce9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:03.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2ca377700 1 --2- 172.21.15.189:0/278700380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2c40cd270 0x7fb2c41f5ff0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc016000 tx=0x7fb2bc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:03.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2ba7fc700 1 -- 172.21.15.189:0/278700380 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc007000 con 0x7fb2c40cd270 2024-04-06T12:28:03.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2ba7fc700 1 -- 172.21.15.189:0/278700380 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2bc022070 con 0x7fb2c40cd270 2024-04-06T12:28:03.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2cbdda700 1 -- 172.21.15.189:0/278700380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c40cca70 con 0x7fb2c40cd270 2024-04-06T12:28:03.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.511+0000 7fb2ba7fc700 1 -- 172.21.15.189:0/278700380 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc029480 con 0x7fb2c40cd270 2024-04-06T12:28:03.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.512+0000 7fb2cbdda700 1 -- 172.21.15.189:0/278700380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c40cd700 con 0x7fb2c40cd270 2024-04-06T12:28:03.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.512+0000 7fb2cbdda700 1 -- 172.21.15.189:0/278700380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c40ccf90 con 0x7fb2c40cd270 2024-04-06T12:28:03.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.513+0000 7fb2ba7fc700 1 -- 172.21.15.189:0/278700380 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fb2bc01e430 con 0x7fb2c40cd270 2024-04-06T12:28:03.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.513+0000 7fb2ba7fc700 1 --2- 172.21.15.189:0/278700380 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2a806adb0 0x7fb2a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.513+0000 7fb2ba7fc700 1 -- 172.21.15.189:0/278700380 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fb2bc024070 con 0x7fb2c40cd270 2024-04-06T12:28:03.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.513+0000 7fb2c9b76700 1 --2- 172.21.15.189:0/278700380 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2a806adb0 0x7fb2a806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.514+0000 7fb2c9b76700 1 --2- 172.21.15.189:0/278700380 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2a806adb0 0x7fb2a806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0000c00 tx=0x7fb2c0009040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:03.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.518+0000 7fb2ba7fc700 1 -- 172.21.15.189:0/278700380 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb2c40ccf90 con 0x7fb2c40cd270 2024-04-06T12:28:03.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.713+0000 7fb2cbdda700 1 -- 172.21.15.189:0/278700380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb2c4074990 con 0x7fb2c40cd270 2024-04-06T12:28:03.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.713+0000 7fb2ba7fc700 1 -- 172.21.15.189:0/278700380 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v94) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb2bc018090 con 0x7fb2c40cd270 2024-04-06T12:28:03.712 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:03.712 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T12:28:03.712 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:03.712 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 1, 2024-04-06T12:28:03.712 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": ".mgr" 2024-04-06T12:28:03.712 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:03.712 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T12:28:03.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.716+0000 7fb2cbdda700 1 -- 172.21.15.189:0/278700380 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2a806adb0 msgr2=0x7fb2a806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:03.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.716+0000 7fb2cbdda700 1 --2- 172.21.15.189:0/278700380 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2a806adb0 0x7fb2a806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0000c00 tx=0x7fb2c0009040 comp rx=0 tx=0).stop 2024-04-06T12:28:03.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.716+0000 7fb2cbdda700 1 -- 172.21.15.189:0/278700380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2c40cd270 msgr2=0x7fb2c41f5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:03.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.716+0000 7fb2cbdda700 1 --2- 172.21.15.189:0/278700380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2c40cd270 0x7fb2c41f5ff0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc016000 tx=0x7fb2bc009750 comp rx=0 tx=0).stop 2024-04-06T12:28:03.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.716+0000 7fb2cbdda700 1 -- 172.21.15.189:0/278700380 shutdown_connections 2024-04-06T12:28:03.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.716+0000 7fb2cbdda700 1 --2- 172.21.15.189:0/278700380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2c40cd270 0x7fb2c41f5ff0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.716+0000 7fb2cbdda700 1 --2- 172.21.15.189:0/278700380 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2c40b1070 0x7fb2c4159830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.716+0000 7fb2cbdda700 1 --2- 172.21.15.189:0/278700380 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb2a806adb0 0x7fb2a806d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.716+0000 7fb2cbdda700 1 --2- 172.21.15.189:0/278700380 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2c40b1450 0x7fb2c40ce9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.716+0000 7fb2cbdda700 1 -- 172.21.15.189:0/278700380 >> 172.21.15.189:0/278700380 conn(0x7fb2c40d00b0 msgr2=0x7fb2c4072740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:03.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.716+0000 7fb2cbdda700 1 -- 172.21.15.189:0/278700380 shutdown_connections 2024-04-06T12:28:03.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.716+0000 7fb2cbdda700 1 -- 172.21.15.189:0/278700380 wait complete. 2024-04-06T12:28:03.726 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-06T12:28:03.726 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-06T12:28:03.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.853+0000 7ff57ac5b700 1 Processor -- start 2024-04-06T12:28:03.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.853+0000 7ff57ac5b700 1 -- start start 2024-04-06T12:28:03.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.854+0000 7ff57ac5b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff57415d7e0 0x7ff5740d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.854+0000 7ff57ac5b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff5740d0890 0x7ff5740d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.854+0000 7ff57ac5b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff5740d1af0 0x7ff5740b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.854+0000 7ff57ac5b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5740d0d10 con 0x7ff5740d1af0 2024-04-06T12:28:03.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.854+0000 7ff57ac5b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5740d1f80 con 0x7ff5740d0890 2024-04-06T12:28:03.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.854+0000 7ff57ac5b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5740cdf80 con 0x7ff57415d7e0 2024-04-06T12:28:03.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.854+0000 7ff5791f8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff5740d1af0 0x7ff5740b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.854+0000 7ff5791f8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff5740d1af0 0x7ff5740b4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57934/0 (socket says 172.21.15.189:57934) 2024-04-06T12:28:03.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.854+0000 7ff5791f8700 1 -- 172.21.15.189:0/258306620 learned_addr learned my addr 172.21.15.189:0/258306620 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:03.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.854+0000 7ff573fff700 1 --2- 172.21.15.189:0/258306620 >> v2:172.21.15.192:3300/0 conn(0x7ff5740d0890 0x7ff5740d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.854+0000 7ff5789f7700 1 --2- 172.21.15.189:0/258306620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff57415d7e0 0x7ff5740d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.854+0000 7ff5791f8700 1 -- 172.21.15.189:0/258306620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff57415d7e0 msgr2=0x7ff5740d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:03.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.854+0000 7ff5791f8700 1 --2- 172.21.15.189:0/258306620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff57415d7e0 0x7ff5740d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.854+0000 7ff5791f8700 1 -- 172.21.15.189:0/258306620 >> v2:172.21.15.192:3300/0 conn(0x7ff5740d0890 msgr2=0x7ff5740d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:03.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.854+0000 7ff5791f8700 1 --2- 172.21.15.189:0/258306620 >> v2:172.21.15.192:3300/0 conn(0x7ff5740d0890 0x7ff5740d15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.854+0000 7ff5791f8700 1 -- 172.21.15.189:0/258306620 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5740d3230 con 0x7ff5740d1af0 2024-04-06T12:28:03.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.855+0000 7ff5789f7700 1 --2- 172.21.15.189:0/258306620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff57415d7e0 0x7ff5740d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:28:03.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.855+0000 7ff5791f8700 1 --2- 172.21.15.189:0/258306620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5740d1af0 0x7ff5740b4e50 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7ff5740b5f20 tx=0x7ff56c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3dc97e23f7f4299 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:03.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.855+0000 7ff5737fe700 1 -- 172.21.15.189:0/258306620 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c00e070 con 0x7ff5740d1af0 2024-04-06T12:28:03.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.855+0000 7ff5737fe700 1 -- 172.21.15.189:0/258306620 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56c00b9c0 con 0x7ff5740d1af0 2024-04-06T12:28:03.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.855+0000 7ff5737fe700 1 -- 172.21.15.189:0/258306620 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c00bd70 con 0x7ff5740d1af0 2024-04-06T12:28:03.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.855+0000 7ff57ac5b700 1 -- 172.21.15.189:0/258306620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5740d1af0 msgr2=0x7ff5740b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:03.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.855+0000 7ff57ac5b700 1 --2- 172.21.15.189:0/258306620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5740d1af0 0x7ff5740b4e50 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7ff5740b5f20 tx=0x7ff56c00b330 comp rx=0 tx=0).stop 2024-04-06T12:28:03.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.855+0000 7ff57ac5b700 1 -- 172.21.15.189:0/258306620 shutdown_connections 2024-04-06T12:28:03.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.855+0000 7ff57ac5b700 1 --2- 172.21.15.189:0/258306620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5740d1af0 0x7ff5740b4e50 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.855+0000 7ff57ac5b700 1 --2- 172.21.15.189:0/258306620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff57415d7e0 0x7ff5740d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.855+0000 7ff57ac5b700 1 --2- 172.21.15.189:0/258306620 >> v2:172.21.15.192:3300/0 conn(0x7ff5740d0890 0x7ff5740d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.855+0000 7ff57ac5b700 1 -- 172.21.15.189:0/258306620 >> 172.21.15.189:0/258306620 conn(0x7ff574072250 msgr2=0x7ff574072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:03.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.855+0000 7ff57ac5b700 1 -- 172.21.15.189:0/258306620 shutdown_connections 2024-04-06T12:28:03.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.855+0000 7ff57ac5b700 1 -- 172.21.15.189:0/258306620 wait complete. 2024-04-06T12:28:03.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.857+0000 7ff57ac5b700 1 Processor -- start 2024-04-06T12:28:03.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.857+0000 7ff57ac5b700 1 -- start start 2024-04-06T12:28:03.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.857+0000 7ff57ac5b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5740d0890 0x7ff57415e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.857+0000 7ff57ac5b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5740d1af0 0x7ff5740abc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.857+0000 7ff57ac5b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff57415d7e0 0x7ff57414d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.857+0000 7ff57ac5b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5740cb4f0 con 0x7ff57415d7e0 2024-04-06T12:28:03.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.857+0000 7ff57ac5b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57415b160 con 0x7ff5740d1af0 2024-04-06T12:28:03.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.857+0000 7ff57ac5b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5740cb740 con 0x7ff5740d0890 2024-04-06T12:28:03.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.858+0000 7ff5791f8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff57415d7e0 0x7ff57414d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.858+0000 7ff5791f8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff57415d7e0 0x7ff57414d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57936/0 (socket says 172.21.15.189:57936) 2024-04-06T12:28:03.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.858+0000 7ff5791f8700 1 -- 172.21.15.189:0/2163247960 learned_addr learned my addr 172.21.15.189:0/2163247960 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:03.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.858+0000 7ff5789f7700 1 --2- 172.21.15.189:0/2163247960 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5740d0890 0x7ff57415e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.858+0000 7ff573fff700 1 --2- 172.21.15.189:0/2163247960 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5740d1af0 0x7ff5740abc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.858+0000 7ff5791f8700 1 -- 172.21.15.189:0/2163247960 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5740d0890 msgr2=0x7ff57415e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:03.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.858+0000 7ff5791f8700 1 --2- 172.21.15.189:0/2163247960 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5740d0890 0x7ff57415e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.858+0000 7ff5791f8700 1 -- 172.21.15.189:0/2163247960 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5740d1af0 msgr2=0x7ff5740abc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:03.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.858+0000 7ff5791f8700 1 --2- 172.21.15.189:0/2163247960 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5740d1af0 0x7ff5740abc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:03.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.858+0000 7ff5791f8700 1 -- 172.21.15.189:0/2163247960 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c009b30 con 0x7ff57415d7e0 2024-04-06T12:28:03.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.858+0000 7ff573fff700 1 --2- 172.21.15.189:0/2163247960 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5740d1af0 0x7ff5740abc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:03.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.858+0000 7ff5791f8700 1 --2- 172.21.15.189:0/2163247960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff57415d7e0 0x7ff57414d370 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7ff57406b8c0 tx=0x7ff56c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:03.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.858+0000 7ff5717fa700 1 -- 172.21.15.189:0/2163247960 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c004250 con 0x7ff57415d7e0 2024-04-06T12:28:03.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.858+0000 7ff5717fa700 1 -- 172.21.15.189:0/2163247960 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56c0043d0 con 0x7ff57415d7e0 2024-04-06T12:28:03.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.858+0000 7ff57ac5b700 1 -- 172.21.15.189:0/2163247960 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5740cba20 con 0x7ff57415d7e0 2024-04-06T12:28:03.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.858+0000 7ff5717fa700 1 -- 172.21.15.189:0/2163247960 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c004780 con 0x7ff57415d7e0 2024-04-06T12:28:03.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.858+0000 7ff57ac5b700 1 -- 172.21.15.189:0/2163247960 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5740cbfe0 con 0x7ff57415d7e0 2024-04-06T12:28:03.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.859+0000 7ff57ac5b700 1 -- 172.21.15.189:0/2163247960 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff574074990 con 0x7ff57415d7e0 2024-04-06T12:28:03.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.860+0000 7ff5717fa700 1 -- 172.21.15.189:0/2163247960 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7ff56c01c440 con 0x7ff57415d7e0 2024-04-06T12:28:03.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.860+0000 7ff5717fa700 1 --2- 172.21.15.189:0/2163247960 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff55806adb0 0x7ff55806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:03.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.860+0000 7ff5717fa700 1 -- 172.21.15.189:0/2163247960 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5177+0+0 (secure 0 0 0) 0x7ff56c08a700 con 0x7ff57415d7e0 2024-04-06T12:28:03.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.860+0000 7ff5789f7700 1 --2- 172.21.15.189:0/2163247960 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff55806adb0 0x7ff55806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:03.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.861+0000 7ff5789f7700 1 --2- 172.21.15.189:0/2163247960 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff55806adb0 0x7ff55806d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff568006fd0 tx=0x7ff568008040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:03.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:03.865+0000 7ff5717fa700 1 -- 172.21.15.189:0/2163247960 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff56c055960 con 0x7ff57415d7e0 2024-04-06T12:28:04.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.060+0000 7ff57ac5b700 1 -- 172.21.15.189:0/2163247960 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff5740d3230 con 0x7ff57415d7e0 2024-04-06T12:28:04.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.247+0000 7ff5717fa700 1 -- 172.21.15.189:0/2163247960 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v95) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff5740d3230 con 0x7ff57415d7e0 2024-04-06T12:28:04.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.251+0000 7ff57ac5b700 1 -- 172.21.15.189:0/2163247960 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff55806adb0 msgr2=0x7ff55806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:04.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.251+0000 7ff57ac5b700 1 --2- 172.21.15.189:0/2163247960 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff55806adb0 0x7ff55806d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff568006fd0 tx=0x7ff568008040 comp rx=0 tx=0).stop 2024-04-06T12:28:04.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.251+0000 7ff57ac5b700 1 -- 172.21.15.189:0/2163247960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff57415d7e0 msgr2=0x7ff57414d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:04.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.251+0000 7ff57ac5b700 1 --2- 172.21.15.189:0/2163247960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff57415d7e0 0x7ff57414d370 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7ff57406b8c0 tx=0x7ff56c004ad0 comp rx=0 tx=0).stop 2024-04-06T12:28:04.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.252+0000 7ff57ac5b700 1 -- 172.21.15.189:0/2163247960 shutdown_connections 2024-04-06T12:28:04.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.252+0000 7ff57ac5b700 1 --2- 172.21.15.189:0/2163247960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff57415d7e0 0x7ff57414d370 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:04.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.252+0000 7ff57ac5b700 1 --2- 172.21.15.189:0/2163247960 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5740d0890 0x7ff57415e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:04.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.252+0000 7ff57ac5b700 1 --2- 172.21.15.189:0/2163247960 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff55806adb0 0x7ff55806d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:04.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.252+0000 7ff57ac5b700 1 --2- 172.21.15.189:0/2163247960 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5740d1af0 0x7ff5740abc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:04.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.252+0000 7ff57ac5b700 1 -- 172.21.15.189:0/2163247960 >> 172.21.15.189:0/2163247960 conn(0x7ff574072250 msgr2=0x7ff57415f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:04.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.252+0000 7ff57ac5b700 1 -- 172.21.15.189:0/2163247960 shutdown_connections 2024-04-06T12:28:04.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.252+0000 7ff57ac5b700 1 -- 172.21.15.189:0/2163247960 wait complete. 2024-04-06T12:28:04.251 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' created 2024-04-06T12:28:04.264 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-06T12:28:04.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.394+0000 7fc4b0e12700 1 Processor -- start 2024-04-06T12:28:04.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.394+0000 7fc4b0e12700 1 -- start start 2024-04-06T12:28:04.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.395+0000 7fc4b0e12700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc4ac0b3e20 0x7fc4ac15e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:04.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.395+0000 7fc4b0e12700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ac0b3990 con 0x7fc4ac06ae40 2024-04-06T12:28:04.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.395+0000 7fc4b0e12700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ac0b3060 con 0x7fc4ac15d7b0 2024-04-06T12:28:04.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.395+0000 7fc4b0e12700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ac0b4230 con 0x7fc4ac0b3e20 2024-04-06T12:28:04.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.395+0000 7fc4a9d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc4ac06ae40 0x7fc4ac15ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55424/0 (socket says 172.21.15.189:55424) 2024-04-06T12:28:04.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.395+0000 7fc4a9d9b700 1 -- 172.21.15.189:0/3878080690 learned_addr learned my addr 172.21.15.189:0/3878080690 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:04.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.396+0000 7fc4aad9d700 1 --2- 172.21.15.189:0/3878080690 >> v2:172.21.15.192:3300/0 conn(0x7fc4ac0b3e20 0x7fc4ac15e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:04.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.396+0000 7fc4a959a700 1 -- 172.21.15.189:0/3878080690 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856358794 0 0) 0x7fc4ac0b3990 con 0x7fc4ac06ae40 2024-04-06T12:28:04.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.396+0000 7fc4a959a700 1 -- 172.21.15.189:0/3878080690 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4940035b0 con 0x7fc4ac06ae40 2024-04-06T12:28:04.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.396+0000 7fc4a959a700 1 -- 172.21.15.189:0/3878080690 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2015188195 0 0) 0x7fc4940035b0 con 0x7fc4ac06ae40 2024-04-06T12:28:04.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.396+0000 7fc4aad9d700 1 -- 172.21.15.189:0/3878080690 >> v1:172.21.15.192:6789/0 conn(0x7fc4ac15d7b0 legacy=0x7fc4ac06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:04.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.396+0000 7fc4aad9d700 1 -- 172.21.15.189:0/3878080690 >> v1:172.21.15.189:6789/0 conn(0x7fc4ac06ae40 legacy=0x7fc4ac15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:04.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.396+0000 7fc4aad9d700 1 -- 172.21.15.189:0/3878080690 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac0b3340 con 0x7fc4ac0b3e20 2024-04-06T12:28:04.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.396+0000 7fc4aad9d700 1 --2- 172.21.15.189:0/3878080690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4ac0b3e20 0x7fc4ac15e500 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4009d90 tx=0x7fc4a400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=630cd312b1b928b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:04.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.396+0000 7fc4a959a700 1 -- 172.21.15.189:0/3878080690 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a400e070 con 0x7fc4ac0b3e20 2024-04-06T12:28:04.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.396+0000 7fc4a959a700 1 -- 172.21.15.189:0/3878080690 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a4003a20 con 0x7fc4ac0b3e20 2024-04-06T12:28:04.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.396+0000 7fc4a959a700 1 -- 172.21.15.189:0/3878080690 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c001670 con 0x7fc4ac0b3e20 2024-04-06T12:28:04.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.397+0000 7fc4b0e12700 1 -- 172.21.15.189:0/3878080690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4ac0b3e20 msgr2=0x7fc4ac15e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:04.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.397+0000 7fc4b0e12700 1 --2- 172.21.15.189:0/3878080690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4ac0b3e20 0x7fc4ac15e500 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4009d90 tx=0x7fc4a400b330 comp rx=0 tx=0).stop 2024-04-06T12:28:04.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.397+0000 7fc4b0e12700 1 -- 172.21.15.189:0/3878080690 shutdown_connections 2024-04-06T12:28:04.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.397+0000 7fc4b0e12700 1 --2- 172.21.15.189:0/3878080690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4ac0b3e20 0x7fc4ac15e500 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:04.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.397+0000 7fc4b0e12700 1 -- 172.21.15.189:0/3878080690 >> 172.21.15.189:0/3878080690 conn(0x7fc4ac0d00b0 msgr2=0x7fc4ac0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:04.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.397+0000 7fc4b0e12700 1 -- 172.21.15.189:0/3878080690 shutdown_connections 2024-04-06T12:28:04.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.397+0000 7fc4b0e12700 1 -- 172.21.15.189:0/3878080690 wait complete. 2024-04-06T12:28:04.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.398+0000 7fc4b0e12700 1 Processor -- start 2024-04-06T12:28:04.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.398+0000 7fc4b0e12700 1 -- start start 2024-04-06T12:28:04.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.399+0000 7fc4b0e12700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4ac06ae40 0x7fc4ac15a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:04.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.399+0000 7fc4b0e12700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4ac0b3e20 0x7fc4ac14fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:04.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.399+0000 7fc4b0e12700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4ac15d7b0 0x7fc4ac0ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:04.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.399+0000 7fc4b0e12700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac15c420 con 0x7fc4ac0b3e20 2024-04-06T12:28:04.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.399+0000 7fc4b0e12700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac160180 con 0x7fc4ac15d7b0 2024-04-06T12:28:04.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.399+0000 7fc4b0e12700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac160610 con 0x7fc4ac06ae40 2024-04-06T12:28:04.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.399+0000 7fc4a9d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4ac0b3e20 0x7fc4ac14fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:04.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.399+0000 7fc4a9d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4ac0b3e20 0x7fc4ac14fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57950/0 (socket says 172.21.15.189:57950) 2024-04-06T12:28:04.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.399+0000 7fc4a9d9b700 1 -- 172.21.15.189:0/426324699 learned_addr learned my addr 172.21.15.189:0/426324699 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:04.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.399+0000 7fc4aad9d700 1 --2- 172.21.15.189:0/426324699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4ac15d7b0 0x7fc4ac0ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:04.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.399+0000 7fc4aa59c700 1 --2- 172.21.15.189:0/426324699 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4ac06ae40 0x7fc4ac15a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:04.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.400+0000 7fc4a9d9b700 1 -- 172.21.15.189:0/426324699 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4ac06ae40 msgr2=0x7fc4ac15a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:04.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.400+0000 7fc4a9d9b700 1 --2- 172.21.15.189:0/426324699 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4ac06ae40 0x7fc4ac15a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:04.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.400+0000 7fc4a9d9b700 1 -- 172.21.15.189:0/426324699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4ac15d7b0 msgr2=0x7fc4ac0ccdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:04.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.400+0000 7fc4a9d9b700 1 --2- 172.21.15.189:0/426324699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4ac15d7b0 0x7fc4ac0ccdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:04.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.400+0000 7fc4a9d9b700 1 -- 172.21.15.189:0/426324699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a4009a70 con 0x7fc4ac0b3e20 2024-04-06T12:28:04.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.400+0000 7fc4a9d9b700 1 --2- 172.21.15.189:0/426324699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4ac0b3e20 0x7fc4ac14fdd0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fc49c00c700 tx=0x7fc49c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:04.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.400+0000 7fc49affd700 1 -- 172.21.15.189:0/426324699 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c004c80 con 0x7fc4ac0b3e20 2024-04-06T12:28:04.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.400+0000 7fc49affd700 1 -- 172.21.15.189:0/426324699 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49c004e00 con 0x7fc4ac0b3e20 2024-04-06T12:28:04.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.400+0000 7fc4b0e12700 1 -- 172.21.15.189:0/426324699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ac15f850 con 0x7fc4ac0b3e20 2024-04-06T12:28:04.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.400+0000 7fc49affd700 1 -- 172.21.15.189:0/426324699 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c007310 con 0x7fc4ac0b3e20 2024-04-06T12:28:04.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.400+0000 7fc4b0e12700 1 -- 172.21.15.189:0/426324699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ac15fe10 con 0x7fc4ac0b3e20 2024-04-06T12:28:04.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.401+0000 7fc4b0e12700 1 -- 172.21.15.189:0/426324699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ac067bb0 con 0x7fc4ac0b3e20 2024-04-06T12:28:04.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.402+0000 7fc49affd700 1 -- 172.21.15.189:0/426324699 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fc49c007470 con 0x7fc4ac0b3e20 2024-04-06T12:28:04.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.403+0000 7fc49affd700 1 --2- 172.21.15.189:0/426324699 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc48c06adb0 0x7fc48c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:04.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.403+0000 7fc49affd700 1 -- 172.21.15.189:0/426324699 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5547+0+0 (secure 0 0 0) 0x7fc49c089830 con 0x7fc4ac0b3e20 2024-04-06T12:28:04.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.403+0000 7fc4aa59c700 1 --2- 172.21.15.189:0/426324699 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc48c06adb0 0x7fc48c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:04.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.403+0000 7fc4aa59c700 1 --2- 172.21.15.189:0/426324699 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc48c06adb0 0x7fc48c06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0000f60 tx=0x7fc4a0009040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:04.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.407+0000 7fc49affd700 1 -- 172.21.15.189:0/426324699 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc49c054a80 con 0x7fc4ac0b3e20 2024-04-06T12:28:04.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:04.600+0000 7fc4b0e12700 1 -- 172.21.15.189:0/426324699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc4ac1602f0 con 0x7fc4ac0b3e20 2024-04-06T12:28:05.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.259+0000 7fc49affd700 1 -- 172.21.15.189:0/426324699 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v96) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc49c0580a0 con 0x7fc4ac0b3e20 2024-04-06T12:28:05.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.268+0000 7fc4b0e12700 1 -- 172.21.15.189:0/426324699 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc48c06adb0 msgr2=0x7fc48c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:05.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.268+0000 7fc4b0e12700 1 --2- 172.21.15.189:0/426324699 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc48c06adb0 0x7fc48c06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0000f60 tx=0x7fc4a0009040 comp rx=0 tx=0).stop 2024-04-06T12:28:05.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.268+0000 7fc4b0e12700 1 -- 172.21.15.189:0/426324699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4ac0b3e20 msgr2=0x7fc4ac14fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:05.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.268+0000 7fc4b0e12700 1 --2- 172.21.15.189:0/426324699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4ac0b3e20 0x7fc4ac14fdd0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fc49c00c700 tx=0x7fc49c00ca10 comp rx=0 tx=0).stop 2024-04-06T12:28:05.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.269+0000 7fc4b0e12700 1 -- 172.21.15.189:0/426324699 shutdown_connections 2024-04-06T12:28:05.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.269+0000 7fc4b0e12700 1 --2- 172.21.15.189:0/426324699 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc48c06adb0 0x7fc48c06d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.269+0000 7fc4b0e12700 1 --2- 172.21.15.189:0/426324699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4ac15d7b0 0x7fc4ac0ccdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.269+0000 7fc4b0e12700 1 --2- 172.21.15.189:0/426324699 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4ac06ae40 0x7fc4ac15a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.269+0000 7fc4b0e12700 1 --2- 172.21.15.189:0/426324699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4ac0b3e20 0x7fc4ac14fdd0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.269+0000 7fc4b0e12700 1 -- 172.21.15.189:0/426324699 >> 172.21.15.189:0/426324699 conn(0x7fc4ac0d00b0 msgr2=0x7fc4ac0afc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:05.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.269+0000 7fc4b0e12700 1 -- 172.21.15.189:0/426324699 shutdown_connections 2024-04-06T12:28:05.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.269+0000 7fc4b0e12700 1 -- 172.21.15.189:0/426324699 wait complete. 2024-04-06T12:28:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' created 2024-04-06T12:28:05.289 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-06T12:28:05.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.420+0000 7f23175fd700 1 Processor -- start 2024-04-06T12:28:05.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.420+0000 7f23175fd700 1 -- start start 2024-04-06T12:28:05.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.421+0000 7f23175fd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f23100d11b0 0x7f23100d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:05.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.421+0000 7f23175fd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23100cb870 0x7f231015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:05.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.421+0000 7f23175fd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f23100cfa00 0x7f23100cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:05.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.421+0000 7f23175fd700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23100ccdc0 con 0x7f23100d11b0 2024-04-06T12:28:05.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.421+0000 7f23175fd700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23100d0b50 con 0x7f23100cfa00 2024-04-06T12:28:05.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.421+0000 7f23175fd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23100d1ad0 con 0x7f23100cb870 2024-04-06T12:28:05.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.421+0000 7f2315399700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f23100d11b0 0x7f23100d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:05.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.421+0000 7f2315399700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f23100d11b0 0x7f23100d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57952/0 (socket says 172.21.15.189:57952) 2024-04-06T12:28:05.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.421+0000 7f2315399700 1 -- 172.21.15.189:0/1194887842 learned_addr learned my addr 172.21.15.189:0/1194887842 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:05.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.421+0000 7f2315b9a700 1 --2- 172.21.15.189:0/1194887842 >> v2:172.21.15.192:3300/0 conn(0x7f23100cfa00 0x7f23100cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:05.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.421+0000 7f2314b98700 1 --2- 172.21.15.189:0/1194887842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23100cb870 0x7f231015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:05.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.421+0000 7f2315399700 1 -- 172.21.15.189:0/1194887842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23100cb870 msgr2=0x7f231015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:05.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.421+0000 7f2315399700 1 --2- 172.21.15.189:0/1194887842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23100cb870 0x7f231015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.422+0000 7f2315399700 1 -- 172.21.15.189:0/1194887842 >> v2:172.21.15.192:3300/0 conn(0x7f23100cfa00 msgr2=0x7f23100cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:05.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.422+0000 7f2315399700 1 --2- 172.21.15.189:0/1194887842 >> v2:172.21.15.192:3300/0 conn(0x7f23100cfa00 0x7f23100cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.422+0000 7f2315399700 1 -- 172.21.15.189:0/1194887842 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23100cbd80 con 0x7f23100d11b0 2024-04-06T12:28:05.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.422+0000 7f2315b9a700 1 --2- 172.21.15.189:0/1194887842 >> v2:172.21.15.192:3300/0 conn(0x7f23100cfa00 0x7f23100cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:05.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.422+0000 7f2314b98700 1 --2- 172.21.15.189:0/1194887842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23100cb870 0x7f231015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:05.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.422+0000 7f2315399700 1 --2- 172.21.15.189:0/1194887842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23100d11b0 0x7f23100d1590 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f23100ccd10 tx=0x7f2300009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ffeddd0eb2fdcf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:05.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.422+0000 7f2307fff700 1 -- 172.21.15.189:0/1194887842 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230000b870 con 0x7f23100d11b0 2024-04-06T12:28:05.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.422+0000 7f2307fff700 1 -- 172.21.15.189:0/1194887842 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f230000b9d0 con 0x7f23100d11b0 2024-04-06T12:28:05.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.422+0000 7f2307fff700 1 -- 172.21.15.189:0/1194887842 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230000bd60 con 0x7f23100d11b0 2024-04-06T12:28:05.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.422+0000 7f23175fd700 1 -- 172.21.15.189:0/1194887842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23100d11b0 msgr2=0x7f23100d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:05.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.422+0000 7f23175fd700 1 --2- 172.21.15.189:0/1194887842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23100d11b0 0x7f23100d1590 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f23100ccd10 tx=0x7f2300009bd0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.422+0000 7f23175fd700 1 -- 172.21.15.189:0/1194887842 shutdown_connections 2024-04-06T12:28:05.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.422+0000 7f23175fd700 1 --2- 172.21.15.189:0/1194887842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23100d11b0 0x7f23100d1590 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.422+0000 7f23175fd700 1 --2- 172.21.15.189:0/1194887842 >> v2:172.21.15.192:3300/0 conn(0x7f23100cfa00 0x7f23100cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.422+0000 7f23175fd700 1 --2- 172.21.15.189:0/1194887842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23100cb870 0x7f231015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.422+0000 7f23175fd700 1 -- 172.21.15.189:0/1194887842 >> 172.21.15.189:0/1194887842 conn(0x7f23100b2db0 msgr2=0x7f23100b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:05.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.423+0000 7f23175fd700 1 -- 172.21.15.189:0/1194887842 shutdown_connections 2024-04-06T12:28:05.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.423+0000 7f23175fd700 1 -- 172.21.15.189:0/1194887842 wait complete. 2024-04-06T12:28:05.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.424+0000 7f23175fd700 1 Processor -- start 2024-04-06T12:28:05.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.424+0000 7f23175fd700 1 -- start start 2024-04-06T12:28:05.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.424+0000 7f23175fd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23100cb870 0x7f23100c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:05.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.424+0000 7f23175fd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23100cfa00 0x7f2310144880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:05.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.424+0000 7f23175fd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23100d11b0 0x7f231015e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:05.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.424+0000 7f23175fd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2310144dc0 con 0x7f23100cfa00 2024-04-06T12:28:05.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.424+0000 7f23175fd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231015d2d0 con 0x7f23100cb870 2024-04-06T12:28:05.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.424+0000 7f23175fd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f231015d570 con 0x7f23100d11b0 2024-04-06T12:28:05.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.425+0000 7f2314b98700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23100cfa00 0x7f2310144880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:05.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.425+0000 7f2314b98700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23100cfa00 0x7f2310144880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57954/0 (socket says 172.21.15.189:57954) 2024-04-06T12:28:05.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.425+0000 7f2314b98700 1 -- 172.21.15.189:0/1587610897 learned_addr learned my addr 172.21.15.189:0/1587610897 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:05.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.425+0000 7f2315399700 1 --2- 172.21.15.189:0/1587610897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23100cb870 0x7f23100c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:05.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.425+0000 7f2315b9a700 1 --2- 172.21.15.189:0/1587610897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23100d11b0 0x7f231015e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:05.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.425+0000 7f2314b98700 1 -- 172.21.15.189:0/1587610897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23100d11b0 msgr2=0x7f231015e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:05.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.425+0000 7f2314b98700 1 --2- 172.21.15.189:0/1587610897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23100d11b0 0x7f231015e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.425+0000 7f2314b98700 1 -- 172.21.15.189:0/1587610897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23100cb870 msgr2=0x7f23100c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:05.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.425+0000 7f2314b98700 1 --2- 172.21.15.189:0/1587610897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23100cb870 0x7f23100c6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.425+0000 7f2314b98700 1 -- 172.21.15.189:0/1587610897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2300009550 con 0x7f23100cfa00 2024-04-06T12:28:05.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.425+0000 7f2315399700 1 --2- 172.21.15.189:0/1587610897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23100cb870 0x7f23100c6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:05.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.425+0000 7f2314b98700 1 --2- 172.21.15.189:0/1587610897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23100cfa00 0x7f2310144880 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f230c00d7d0 tx=0x7f230c00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:05.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.425+0000 7f2305ffb700 1 -- 172.21.15.189:0/1587610897 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c004030 con 0x7f23100cfa00 2024-04-06T12:28:05.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.425+0000 7f2305ffb700 1 -- 172.21.15.189:0/1587610897 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f230c004190 con 0x7f23100cfa00 2024-04-06T12:28:05.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.425+0000 7f2305ffb700 1 -- 172.21.15.189:0/1587610897 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c004500 con 0x7f23100cfa00 2024-04-06T12:28:05.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.425+0000 7f23175fd700 1 -- 172.21.15.189:0/1587610897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f231015d850 con 0x7f23100cfa00 2024-04-06T12:28:05.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.425+0000 7f23175fd700 1 -- 172.21.15.189:0/1587610897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f231015de10 con 0x7f23100cfa00 2024-04-06T12:28:05.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.426+0000 7f23175fd700 1 -- 172.21.15.189:0/1587610897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2310074990 con 0x7f23100cfa00 2024-04-06T12:28:05.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.427+0000 7f2305ffb700 1 -- 172.21.15.189:0/1587610897 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f230c0036a0 con 0x7f23100cfa00 2024-04-06T12:28:05.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.427+0000 7f2305ffb700 1 --2- 172.21.15.189:0/1587610897 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f22f406adb0 0x7f22f406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:05.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.427+0000 7f2305ffb700 1 -- 172.21.15.189:0/1587610897 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f230c088d70 con 0x7f23100cfa00 2024-04-06T12:28:05.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.427+0000 7f2315399700 1 --2- 172.21.15.189:0/1587610897 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f22f406adb0 0x7f22f406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:05.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.427+0000 7f2315399700 1 --2- 172.21.15.189:0/1587610897 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f22f406adb0 0x7f22f406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2300000c00 tx=0x7f2300008030 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:05.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.432+0000 7f2305ffb700 1 -- 172.21.15.189:0/1587610897 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f230c053ea0 con 0x7f23100cfa00 2024-04-06T12:28:05.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.671+0000 7f23175fd700 1 -- 172.21.15.189:0/1587610897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f23100b0a00 con 0x7f23100cfa00 2024-04-06T12:28:05.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.677+0000 7f2305ffb700 1 -- 172.21.15.189:0/1587610897 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v36) v1 ==== 174+0+0 (secure 0 0 0) 0x7f230c019020 con 0x7f23100cfa00 2024-04-06T12:28:05.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.684+0000 7f23175fd700 1 -- 172.21.15.189:0/1587610897 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f22f406adb0 msgr2=0x7f22f406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:05.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.684+0000 7f23175fd700 1 --2- 172.21.15.189:0/1587610897 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f22f406adb0 0x7f22f406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2300000c00 tx=0x7f2300008030 comp rx=0 tx=0).stop 2024-04-06T12:28:05.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.684+0000 7f23175fd700 1 -- 172.21.15.189:0/1587610897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23100cfa00 msgr2=0x7f2310144880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:05.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.684+0000 7f23175fd700 1 --2- 172.21.15.189:0/1587610897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23100cfa00 0x7f2310144880 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f230c00d7d0 tx=0x7f230c00db90 comp rx=0 tx=0).stop 2024-04-06T12:28:05.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.694+0000 7f23175fd700 1 -- 172.21.15.189:0/1587610897 shutdown_connections 2024-04-06T12:28:05.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.694+0000 7f23175fd700 1 --2- 172.21.15.189:0/1587610897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23100d11b0 0x7f231015e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.694+0000 7f23175fd700 1 --2- 172.21.15.189:0/1587610897 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f22f406adb0 0x7f22f406d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.694+0000 7f23175fd700 1 --2- 172.21.15.189:0/1587610897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23100cb870 0x7f23100c6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.694+0000 7f23175fd700 1 --2- 172.21.15.189:0/1587610897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23100cfa00 0x7f2310144880 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.695+0000 7f23175fd700 1 -- 172.21.15.189:0/1587610897 >> 172.21.15.189:0/1587610897 conn(0x7f23100b2db0 msgr2=0x7f23100b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:05.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.695+0000 7f23175fd700 1 -- 172.21.15.189:0/1587610897 shutdown_connections 2024-04-06T12:28:05.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.695+0000 7f23175fd700 1 -- 172.21.15.189:0/1587610897 wait complete. 2024-04-06T12:28:05.703 INFO:teuthology.orchestra.run.smithi189.stderr:new fs with metadata pool 11 and data pool 12 2024-04-06T12:28:05.711 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:28:05.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.835+0000 7f0dd4114700 1 Processor -- start 2024-04-06T12:28:05.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.835+0000 7f0dd4114700 1 -- start start 2024-04-06T12:28:05.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.836+0000 7f0dd4114700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0dcc06ae40 0x7f0dcc15ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:05.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.836+0000 7f0dd4114700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0dcc15fcd0 0x7f0dcc15bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:05.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.836+0000 7f0dd4114700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dcc15c4c0 con 0x7f0dcc144f40 2024-04-06T12:28:05.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.836+0000 7f0dd4114700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dcc0cfcb0 con 0x7f0dcc15fcd0 2024-04-06T12:28:05.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.836+0000 7f0dd4114700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dcc15ca20 con 0x7f0dcc06ae40 2024-04-06T12:28:05.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.836+0000 7f0dd26b1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0dcc15fcd0 0x7f0dcc15bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:05.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.836+0000 7f0dd26b1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0dcc15fcd0 0x7f0dcc15bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57964/0 (socket says 172.21.15.189:57964) 2024-04-06T12:28:05.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.836+0000 7f0dd26b1700 1 -- 172.21.15.189:0/4177953926 learned_addr learned my addr 172.21.15.189:0/4177953926 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:05.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.836+0000 7f0dd16af700 1 --2- 172.21.15.189:0/4177953926 >> v2:172.21.15.192:3300/0 conn(0x7f0dcc06ae40 0x7f0dcc15ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:05.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.837+0000 7f0dd26b1700 1 -- 172.21.15.189:0/4177953926 >> v2:172.21.15.192:3300/0 conn(0x7f0dcc06ae40 msgr2=0x7f0dcc15ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:05.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.837+0000 7f0dd26b1700 1 --2- 172.21.15.189:0/4177953926 >> v2:172.21.15.192:3300/0 conn(0x7f0dcc06ae40 0x7f0dcc15ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.837+0000 7f0dd26b1700 1 -- 172.21.15.189:0/4177953926 >> v1:172.21.15.189:6789/0 conn(0x7f0dcc144f40 legacy=0x7f0dcc06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:05.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.837+0000 7f0dd26b1700 1 -- 172.21.15.189:0/4177953926 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dcc15c740 con 0x7f0dcc15fcd0 2024-04-06T12:28:05.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.837+0000 7f0dd0eae700 1 -- 172.21.15.189:0/4177953926 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177857647 0 0) 0x7f0dcc15c4c0 con 0x7f0dcc144f40 2024-04-06T12:28:05.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.837+0000 7f0dd16af700 1 --2- 172.21.15.189:0/4177953926 >> v2:172.21.15.192:3300/0 conn(0x7f0dcc06ae40 0x7f0dcc15ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:05.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.837+0000 7f0dd26b1700 1 --2- 172.21.15.189:0/4177953926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dcc15fcd0 0x7f0dcc15bf80 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc06c7c0 tx=0x7f0dc400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=666a2f2cf0d75b4e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:05.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.837+0000 7f0dd0eae700 1 -- 172.21.15.189:0/4177953926 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc400e070 con 0x7f0dcc15fcd0 2024-04-06T12:28:05.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.837+0000 7f0dd0eae700 1 -- 172.21.15.189:0/4177953926 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc400b9c0 con 0x7f0dcc15fcd0 2024-04-06T12:28:05.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.837+0000 7f0dd0eae700 1 -- 172.21.15.189:0/4177953926 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc400bd70 con 0x7f0dcc15fcd0 2024-04-06T12:28:05.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.837+0000 7f0dd4114700 1 -- 172.21.15.189:0/4177953926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dcc15fcd0 msgr2=0x7f0dcc15bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:05.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.837+0000 7f0dd4114700 1 --2- 172.21.15.189:0/4177953926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dcc15fcd0 0x7f0dcc15bf80 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc06c7c0 tx=0x7f0dc400b330 comp rx=0 tx=0).stop 2024-04-06T12:28:05.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.837+0000 7f0dd4114700 1 -- 172.21.15.189:0/4177953926 shutdown_connections 2024-04-06T12:28:05.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.837+0000 7f0dd4114700 1 --2- 172.21.15.189:0/4177953926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dcc15fcd0 0x7f0dcc15bf80 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.837+0000 7f0dd4114700 1 --2- 172.21.15.189:0/4177953926 >> v2:172.21.15.192:3300/0 conn(0x7f0dcc06ae40 0x7f0dcc15ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.838+0000 7f0dd4114700 1 -- 172.21.15.189:0/4177953926 >> 172.21.15.189:0/4177953926 conn(0x7f0dcc0d00b0 msgr2=0x7f0dcc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:05.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.838+0000 7f0dd4114700 1 -- 172.21.15.189:0/4177953926 shutdown_connections 2024-04-06T12:28:05.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.838+0000 7f0dd4114700 1 -- 172.21.15.189:0/4177953926 wait complete. 2024-04-06T12:28:05.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.839+0000 7f0dd4114700 1 Processor -- start 2024-04-06T12:28:05.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.839+0000 7f0dd4114700 1 -- start start 2024-04-06T12:28:05.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.839+0000 7f0dd4114700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dcc06ae40 0x7f0dcc0b4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:05.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.839+0000 7f0dd4114700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dcc144f40 0x7f0dcc1f2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:05.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.839+0000 7f0dd4114700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dcc15fcd0 0x7f0dcc0c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:05.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.839+0000 7f0dd4114700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc0c9d50 con 0x7f0dcc15fcd0 2024-04-06T12:28:05.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.839+0000 7f0dd4114700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc130390 con 0x7f0dcc06ae40 2024-04-06T12:28:05.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.839+0000 7f0dd4114700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc130670 con 0x7f0dcc144f40 2024-04-06T12:28:05.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.840+0000 7f0dd26b1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dcc15fcd0 0x7f0dcc0c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:05.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.840+0000 7f0dd26b1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dcc15fcd0 0x7f0dcc0c9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57970/0 (socket says 172.21.15.189:57970) 2024-04-06T12:28:05.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.840+0000 7f0dd26b1700 1 -- 172.21.15.189:0/2274781170 learned_addr learned my addr 172.21.15.189:0/2274781170 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:05.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.840+0000 7f0dd1eb0700 1 --2- 172.21.15.189:0/2274781170 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dcc06ae40 0x7f0dcc0b4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:05.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.840+0000 7f0dd16af700 1 --2- 172.21.15.189:0/2274781170 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dcc144f40 0x7f0dcc1f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:05.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.840+0000 7f0dd26b1700 1 -- 172.21.15.189:0/2274781170 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dcc144f40 msgr2=0x7f0dcc1f2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:05.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.840+0000 7f0dd26b1700 1 --2- 172.21.15.189:0/2274781170 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dcc144f40 0x7f0dcc1f2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.840+0000 7f0dd26b1700 1 -- 172.21.15.189:0/2274781170 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dcc06ae40 msgr2=0x7f0dcc0b4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:05.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.840+0000 7f0dd26b1700 1 --2- 172.21.15.189:0/2274781170 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dcc06ae40 0x7f0dcc0b4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:05.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.840+0000 7f0dd26b1700 1 -- 172.21.15.189:0/2274781170 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc4009b30 con 0x7f0dcc15fcd0 2024-04-06T12:28:05.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.840+0000 7f0dd26b1700 1 --2- 172.21.15.189:0/2274781170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dcc15fcd0 0x7f0dcc0c9810 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4012000 tx=0x7f0dc4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:05.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.840+0000 7f0dc27fc700 1 -- 172.21.15.189:0/2274781170 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc40099b0 con 0x7f0dcc15fcd0 2024-04-06T12:28:05.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.840+0000 7f0dc27fc700 1 -- 172.21.15.189:0/2274781170 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc40092e0 con 0x7f0dcc15fcd0 2024-04-06T12:28:05.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.840+0000 7f0dc27fc700 1 -- 172.21.15.189:0/2274781170 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc401b650 con 0x7f0dcc15fcd0 2024-04-06T12:28:05.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.840+0000 7f0dd4114700 1 -- 172.21.15.189:0/2274781170 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dcc130950 con 0x7f0dcc15fcd0 2024-04-06T12:28:05.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.840+0000 7f0dd4114700 1 -- 172.21.15.189:0/2274781170 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dcc130f10 con 0x7f0dcc15fcd0 2024-04-06T12:28:05.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.841+0000 7f0dd4114700 1 -- 172.21.15.189:0/2274781170 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dcc067bb0 con 0x7f0dcc15fcd0 2024-04-06T12:28:05.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.842+0000 7f0dc27fc700 1 -- 172.21.15.189:0/2274781170 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0dc401b7b0 con 0x7f0dcc15fcd0 2024-04-06T12:28:05.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.842+0000 7f0dc27fc700 1 --2- 172.21.15.189:0/2274781170 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0db006adb0 0x7f0db006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:05.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.842+0000 7f0dc27fc700 1 -- 172.21.15.189:0/2274781170 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0dc408acb0 con 0x7f0dcc15fcd0 2024-04-06T12:28:05.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.842+0000 7f0dd1eb0700 1 --2- 172.21.15.189:0/2274781170 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0db006adb0 0x7f0db006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:05.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.842+0000 7f0dd1eb0700 1 --2- 172.21.15.189:0/2274781170 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0db006adb0 0x7f0db006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc000f60 tx=0x7f0dbc009040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:05.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:05.846+0000 7f0dc27fc700 1 -- 172.21.15.189:0/2274781170 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0dc4055bc0 con 0x7f0dcc15fcd0 2024-04-06T12:28:06.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.041+0000 7f0dd4114700 1 -- 172.21.15.189:0/2274781170 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0dcc074990 con 0x7f0dcc15fcd0 2024-04-06T12:28:06.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.043+0000 7f0dc27fc700 1 -- 172.21.15.189:0/2274781170 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16370 (secure 0 0 0) 0x7f0dc40591e0 con 0x7f0dcc15fcd0 2024-04-06T12:28:06.041 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:06.041 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":97,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:28:05.673014+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:28:04.080500+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-06T12:28:04.619802+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:28:06.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.045+0000 7f0dd4114700 1 -- 172.21.15.189:0/2274781170 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0db006adb0 msgr2=0x7f0db006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:06.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.045+0000 7f0dd4114700 1 --2- 172.21.15.189:0/2274781170 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0db006adb0 0x7f0db006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc000f60 tx=0x7f0dbc009040 comp rx=0 tx=0).stop 2024-04-06T12:28:06.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.045+0000 7f0dd4114700 1 -- 172.21.15.189:0/2274781170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dcc15fcd0 msgr2=0x7f0dcc0c9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:06.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.045+0000 7f0dd4114700 1 --2- 172.21.15.189:0/2274781170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dcc15fcd0 0x7f0dcc0c9810 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4012000 tx=0x7f0dc4004ad0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.045+0000 7f0dd4114700 1 -- 172.21.15.189:0/2274781170 shutdown_connections 2024-04-06T12:28:06.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.045+0000 7f0dd4114700 1 --2- 172.21.15.189:0/2274781170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dcc15fcd0 0x7f0dcc0c9810 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.045+0000 7f0dd4114700 1 --2- 172.21.15.189:0/2274781170 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0db006adb0 0x7f0db006d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.045+0000 7f0dd4114700 1 --2- 172.21.15.189:0/2274781170 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dcc06ae40 0x7f0dcc0b4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.045+0000 7f0dd4114700 1 --2- 172.21.15.189:0/2274781170 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dcc144f40 0x7f0dcc1f2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.045+0000 7f0dd4114700 1 -- 172.21.15.189:0/2274781170 >> 172.21.15.189:0/2274781170 conn(0x7f0dcc0d00b0 msgr2=0x7f0dcc06fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:06.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.045+0000 7f0dd4114700 1 -- 172.21.15.189:0/2274781170 shutdown_connections 2024-04-06T12:28:06.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.045+0000 7f0dd4114700 1 -- 172.21.15.189:0/2274781170 wait complete. 2024-04-06T12:28:06.056 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:28:06.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.179+0000 7f17ed628700 1 Processor -- start 2024-04-06T12:28:06.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.180+0000 7f17ed628700 1 -- start start 2024-04-06T12:28:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.180+0000 7f17ed628700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f17e815db10 0x7f17e80b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.180+0000 7f17ed628700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17e8143470 0x7f17e80cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.180+0000 7f17ed628700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e815eb40 con 0x7f17e8160b60 2024-04-06T12:28:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.180+0000 7f17ed628700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e8160f40 con 0x7f17e815db10 2024-04-06T12:28:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.180+0000 7f17ed628700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80d0bf0 con 0x7f17e8143470 2024-04-06T12:28:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.180+0000 7f17e67fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f17e815db10 0x7f17e80b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.181+0000 7f17e67fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f17e815db10 0x7f17e80b37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57984/0 (socket says 172.21.15.189:57984) 2024-04-06T12:28:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.181+0000 7f17e67fc700 1 -- 172.21.15.189:0/854597616 learned_addr learned my addr 172.21.15.189:0/854597616 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.181+0000 7f17e67fc700 1 -- 172.21.15.189:0/854597616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17e8143470 msgr2=0x7f17e80cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.181+0000 7f17e77fe700 1 --2- 172.21.15.189:0/854597616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17e8143470 0x7f17e80cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.181+0000 7f17e67fc700 1 --2- 172.21.15.189:0/854597616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17e8143470 0x7f17e80cd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.181+0000 7f17e67fc700 1 -- 172.21.15.189:0/854597616 >> v1:172.21.15.189:6789/0 conn(0x7f17e8160b60 legacy=0x7f17e8142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.181+0000 7f17e67fc700 1 -- 172.21.15.189:0/854597616 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e815dfc0 con 0x7f17e815db10 2024-04-06T12:28:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.181+0000 7f17e5ffb700 1 -- 172.21.15.189:0/854597616 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252821701 0 0) 0x7f17d8001110 con 0x7f17e8160b60 2024-04-06T12:28:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.181+0000 7f17e77fe700 1 --2- 172.21.15.189:0/854597616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17e8143470 0x7f17e80cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:28:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.181+0000 7f17e67fc700 1 --2- 172.21.15.189:0/854597616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17e815db10 0x7f17e80b37f0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f17d0006fd0 tx=0x7f17d0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe2d34bf9b13565c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.181+0000 7f17e5ffb700 1 -- 172.21.15.189:0/854597616 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0005e30 con 0x7f17e815db10 2024-04-06T12:28:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.181+0000 7f17e5ffb700 1 -- 172.21.15.189:0/854597616 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d0004030 con 0x7f17e815db10 2024-04-06T12:28:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.181+0000 7f17e5ffb700 1 -- 172.21.15.189:0/854597616 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d00043e0 con 0x7f17e815db10 2024-04-06T12:28:06.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.182+0000 7f17ed628700 1 -- 172.21.15.189:0/854597616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17e815db10 msgr2=0x7f17e80b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:06.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.182+0000 7f17ed628700 1 --2- 172.21.15.189:0/854597616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17e815db10 0x7f17e80b37f0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f17d0006fd0 tx=0x7f17d0009bd0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.182+0000 7f17ed628700 1 -- 172.21.15.189:0/854597616 shutdown_connections 2024-04-06T12:28:06.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.182+0000 7f17ed628700 1 --2- 172.21.15.189:0/854597616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17e8143470 0x7f17e80cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.182+0000 7f17ed628700 1 --2- 172.21.15.189:0/854597616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17e815db10 0x7f17e80b37f0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.182+0000 7f17ed628700 1 -- 172.21.15.189:0/854597616 >> 172.21.15.189:0/854597616 conn(0x7f17e806a730 msgr2=0x7f17e806ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:06.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.182+0000 7f17ed628700 1 -- 172.21.15.189:0/854597616 shutdown_connections 2024-04-06T12:28:06.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.182+0000 7f17ed628700 1 -- 172.21.15.189:0/854597616 wait complete. 2024-04-06T12:28:06.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.183+0000 7f17ed628700 1 Processor -- start 2024-04-06T12:28:06.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.183+0000 7f17ed628700 1 -- start start 2024-04-06T12:28:06.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17ed628700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17e8143470 0x7f17e80c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:06.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17ed628700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17e815db10 0x7f17e80c4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:06.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17ed628700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17e8160b60 0x7f17e80ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:06.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17ed628700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80c99c0 con 0x7f17e815db10 2024-04-06T12:28:06.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17ed628700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80c5900 con 0x7f17e8160b60 2024-04-06T12:28:06.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17ed628700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80c96e0 con 0x7f17e8143470 2024-04-06T12:28:06.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17e67fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17e815db10 0x7f17e80c4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:06.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17e67fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17e815db10 0x7f17e80c4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57990/0 (socket says 172.21.15.189:57990) 2024-04-06T12:28:06.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17e67fc700 1 -- 172.21.15.189:0/444015164 learned_addr learned my addr 172.21.15.189:0/444015164 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:06.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17e77fe700 1 --2- 172.21.15.189:0/444015164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17e8160b60 0x7f17e80ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:06.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17e6ffd700 1 --2- 172.21.15.189:0/444015164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17e8143470 0x7f17e80c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:06.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17e67fc700 1 -- 172.21.15.189:0/444015164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17e8143470 msgr2=0x7f17e80c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:06.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17e67fc700 1 --2- 172.21.15.189:0/444015164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17e8143470 0x7f17e80c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17e67fc700 1 -- 172.21.15.189:0/444015164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17e8160b60 msgr2=0x7f17e80ca4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:06.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17e67fc700 1 --2- 172.21.15.189:0/444015164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17e8160b60 0x7f17e80ca4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17e67fc700 1 -- 172.21.15.189:0/444015164 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d0009500 con 0x7f17e815db10 2024-04-06T12:28:06.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17e6ffd700 1 --2- 172.21.15.189:0/444015164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17e8143470 0x7f17e80c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:06.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17e67fc700 1 --2- 172.21.15.189:0/444015164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17e815db10 0x7f17e80c4550 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009c10 tx=0x7f17d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:06.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17df7fe700 1 -- 172.21.15.189:0/444015164 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0004560 con 0x7f17e815db10 2024-04-06T12:28:06.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17df7fe700 1 -- 172.21.15.189:0/444015164 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d0011420 con 0x7f17e815db10 2024-04-06T12:28:06.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.184+0000 7f17ed628700 1 -- 172.21.15.189:0/444015164 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e80c5ba0 con 0x7f17e815db10 2024-04-06T12:28:06.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.185+0000 7f17df7fe700 1 -- 172.21.15.189:0/444015164 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d00117d0 con 0x7f17e815db10 2024-04-06T12:28:06.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.185+0000 7f17ed628700 1 -- 172.21.15.189:0/444015164 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e80c7b30 con 0x7f17e815db10 2024-04-06T12:28:06.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.186+0000 7f17ed628700 1 -- 172.21.15.189:0/444015164 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e8067bf0 con 0x7f17e815db10 2024-04-06T12:28:06.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.186+0000 7f17df7fe700 1 -- 172.21.15.189:0/444015164 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f17d0011930 con 0x7f17e815db10 2024-04-06T12:28:06.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.186+0000 7f17df7fe700 1 --2- 172.21.15.189:0/444015164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f17c806adb0 0x7f17c806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:06.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.187+0000 7f17df7fe700 1 -- 172.21.15.189:0/444015164 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f17d008a640 con 0x7f17e815db10 2024-04-06T12:28:06.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.187+0000 7f17e6ffd700 1 --2- 172.21.15.189:0/444015164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f17c806adb0 0x7f17c806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:06.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.187+0000 7f17e6ffd700 1 --2- 172.21.15.189:0/444015164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f17c806adb0 0x7f17c806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f17d8000f60 tx=0x7f17d8009040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:06.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.191+0000 7f17df7fe700 1 -- 172.21.15.189:0/444015164 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f17d00555e0 con 0x7f17e815db10 2024-04-06T12:28:06.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.387+0000 7f17ed628700 1 -- 172.21.15.189:0/444015164 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17e80c5760 con 0x7f17e815db10 2024-04-06T12:28:06.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.388+0000 7f17df7fe700 1 -- 172.21.15.189:0/444015164 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16370 (secure 0 0 0) 0x7f17e80c5760 con 0x7f17e815db10 2024-04-06T12:28:06.386 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:06.386 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":97,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:28:05.673014+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:28:04.080500+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-06T12:28:04.619802+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:28:06.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.390+0000 7f17ed628700 1 -- 172.21.15.189:0/444015164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f17c806adb0 msgr2=0x7f17c806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.390+0000 7f17ed628700 1 --2- 172.21.15.189:0/444015164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f17c806adb0 0x7f17c806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f17d8000f60 tx=0x7f17d8009040 comp rx=0 tx=0).stop 2024-04-06T12:28:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.390+0000 7f17ed628700 1 -- 172.21.15.189:0/444015164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17e815db10 msgr2=0x7f17e80c4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.390+0000 7f17ed628700 1 --2- 172.21.15.189:0/444015164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17e815db10 0x7f17e80c4550 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009c10 tx=0x7f17d0004750 comp rx=0 tx=0).stop 2024-04-06T12:28:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.390+0000 7f17ed628700 1 -- 172.21.15.189:0/444015164 shutdown_connections 2024-04-06T12:28:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.390+0000 7f17ed628700 1 --2- 172.21.15.189:0/444015164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f17c806adb0 0x7f17c806d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.390+0000 7f17ed628700 1 --2- 172.21.15.189:0/444015164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17e8160b60 0x7f17e80ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.390+0000 7f17ed628700 1 --2- 172.21.15.189:0/444015164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17e8143470 0x7f17e80c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.390+0000 7f17ed628700 1 --2- 172.21.15.189:0/444015164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17e815db10 0x7f17e80c4550 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.390+0000 7f17ed628700 1 -- 172.21.15.189:0/444015164 >> 172.21.15.189:0/444015164 conn(0x7f17e806a730 msgr2=0x7f17e8161080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.390+0000 7f17ed628700 1 -- 172.21.15.189:0/444015164 shutdown_connections 2024-04-06T12:28:06.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.390+0000 7f17ed628700 1 -- 172.21.15.189:0/444015164 wait complete. 2024-04-06T12:28:06.401 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-06T12:28:06.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.528+0000 7fa60ccc2700 1 Processor -- start 2024-04-06T12:28:06.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.528+0000 7fa60ccc2700 1 -- start start 2024-04-06T12:28:06.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.529+0000 7fa60ccc2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa6080b4660 0x7fa6080b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:06.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.529+0000 7fa60ccc2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa6080b2fc0 0x7fa6080b3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:06.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.529+0000 7fa60ccc2700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6080cdbe0 con 0x7fa60815a550 2024-04-06T12:28:06.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.529+0000 7fa60ccc2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6080b5000 con 0x7fa6080b2fc0 2024-04-06T12:28:06.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.529+0000 7fa60ccc2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa60815a940 con 0x7fa6080b4660 2024-04-06T12:28:06.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.529+0000 7fa605d9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa6080b4660 0x7fa6080b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:06.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.529+0000 7fa606d9d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa6080b2fc0 0x7fa6080b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:06.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.529+0000 7fa60659c700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fa60815a550 0x7fa6080cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:50444/0 (socket says 172.21.15.189:50444) 2024-04-06T12:28:06.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.529+0000 7fa605d9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa6080b4660 0x7fa6080b4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:60978/0 (socket says 172.21.15.189:60978) 2024-04-06T12:28:06.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.529+0000 7fa606d9d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa6080b2fc0 0x7fa6080b3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:39930/0 (socket says 172.21.15.189:39930) 2024-04-06T12:28:06.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.529+0000 7fa60659c700 1 -- 172.21.15.189:0/86852315 learned_addr learned my addr 172.21.15.189:0/86852315 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:06.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.529+0000 7fa605d9b700 1 -- 172.21.15.189:0/86852315 >> v2:172.21.15.192:3300/0 conn(0x7fa6080b2fc0 msgr2=0x7fa6080b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:06.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.529+0000 7fa605d9b700 1 --2- 172.21.15.189:0/86852315 >> v2:172.21.15.192:3300/0 conn(0x7fa6080b2fc0 0x7fa6080b3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.529+0000 7fa605d9b700 1 -- 172.21.15.189:0/86852315 >> v1:172.21.15.192:6789/0 conn(0x7fa60815a550 legacy=0x7fa6080cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:06.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.529+0000 7fa605d9b700 1 -- 172.21.15.189:0/86852315 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60815ac00 con 0x7fa6080b4660 2024-04-06T12:28:06.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.530+0000 7fa606d9d700 1 --2- 172.21.15.189:0/86852315 >> v2:172.21.15.192:3300/0 conn(0x7fa6080b2fc0 0x7fa6080b3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:28:06.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.530+0000 7fa605d9b700 1 --2- 172.21.15.189:0/86852315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa6080b4660 0x7fa6080b4ac0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc009840 tx=0x7fa5fc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfd6e0e289344ea7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:06.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.532+0000 7fa60559a700 1 -- 172.21.15.189:0/86852315 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc011830 con 0x7fa6080b4660 2024-04-06T12:28:06.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.532+0000 7fa60559a700 1 -- 172.21.15.189:0/86852315 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5fc011990 con 0x7fa6080b4660 2024-04-06T12:28:06.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.532+0000 7fa60559a700 1 -- 172.21.15.189:0/86852315 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc011d20 con 0x7fa6080b4660 2024-04-06T12:28:06.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.533+0000 7fa60ccc2700 1 -- 172.21.15.189:0/86852315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa6080b4660 msgr2=0x7fa6080b4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:06.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.533+0000 7fa60ccc2700 1 --2- 172.21.15.189:0/86852315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa6080b4660 0x7fa6080b4ac0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc009840 tx=0x7fa5fc009b50 comp rx=0 tx=0).stop 2024-04-06T12:28:06.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.533+0000 7fa60ccc2700 1 -- 172.21.15.189:0/86852315 shutdown_connections 2024-04-06T12:28:06.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.533+0000 7fa60ccc2700 1 --2- 172.21.15.189:0/86852315 >> v2:172.21.15.192:3300/0 conn(0x7fa6080b2fc0 0x7fa6080b3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.533+0000 7fa60ccc2700 1 --2- 172.21.15.189:0/86852315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa6080b4660 0x7fa6080b4ac0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.533+0000 7fa60ccc2700 1 -- 172.21.15.189:0/86852315 >> 172.21.15.189:0/86852315 conn(0x7fa6080d00b0 msgr2=0x7fa6080d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:06.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.533+0000 7fa60ccc2700 1 -- 172.21.15.189:0/86852315 shutdown_connections 2024-04-06T12:28:06.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.533+0000 7fa60ccc2700 1 -- 172.21.15.189:0/86852315 wait complete. 2024-04-06T12:28:06.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.534+0000 7fa60ccc2700 1 Processor -- start 2024-04-06T12:28:06.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.534+0000 7fa60ccc2700 1 -- start start 2024-04-06T12:28:06.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.535+0000 7fa60ccc2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6080b2fc0 0x7fa6081f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:06.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.535+0000 7fa60ccc2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6080b4660 0x7fa608153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.535+0000 7fa60ccc2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa6080ac8c0 0x7fa6080cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.535+0000 7fa60ccc2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa608154250 con 0x7fa6080b2fc0 2024-04-06T12:28:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.535+0000 7fa60ccc2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa608157690 con 0x7fa6080b4660 2024-04-06T12:28:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.535+0000 7fa60ccc2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6081f3050 con 0x7fa6080ac8c0 2024-04-06T12:28:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.535+0000 7fa60659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6080b2fc0 0x7fa6081f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.535+0000 7fa60659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6080b2fc0 0x7fa6081f2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58004/0 (socket says 172.21.15.189:58004) 2024-04-06T12:28:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.535+0000 7fa60659c700 1 -- 172.21.15.189:0/1987125858 learned_addr learned my addr 172.21.15.189:0/1987125858 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.535+0000 7fa605d9b700 1 --2- 172.21.15.189:0/1987125858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6080b4660 0x7fa608153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.535+0000 7fa606d9d700 1 --2- 172.21.15.189:0/1987125858 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa6080ac8c0 0x7fa6080cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.535+0000 7fa60659c700 1 -- 172.21.15.189:0/1987125858 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa6080ac8c0 msgr2=0x7fa6080cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.535+0000 7fa60659c700 1 --2- 172.21.15.189:0/1987125858 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa6080ac8c0 0x7fa6080cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.535+0000 7fa60659c700 1 -- 172.21.15.189:0/1987125858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6080b4660 msgr2=0x7fa608153d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.535+0000 7fa60659c700 1 --2- 172.21.15.189:0/1987125858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6080b4660 0x7fa608153d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.535+0000 7fa60659c700 1 -- 172.21.15.189:0/1987125858 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa600009a70 con 0x7fa6080b2fc0 2024-04-06T12:28:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.536+0000 7fa60659c700 1 --2- 172.21.15.189:0/1987125858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6080b2fc0 0x7fa6081f2a70 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fa5f800c520 tx=0x7fa5f800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.536+0000 7fa5f6ffd700 1 -- 172.21.15.189:0/1987125858 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8005d70 con 0x7fa6080b2fc0 2024-04-06T12:28:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.536+0000 7fa5f6ffd700 1 -- 172.21.15.189:0/1987125858 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f80070d0 con 0x7fa6080b2fc0 2024-04-06T12:28:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.536+0000 7fa60ccc2700 1 -- 172.21.15.189:0/1987125858 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5fc0094a0 con 0x7fa6080b2fc0 2024-04-06T12:28:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.536+0000 7fa5f6ffd700 1 -- 172.21.15.189:0/1987125858 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8007440 con 0x7fa6080b2fc0 2024-04-06T12:28:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.536+0000 7fa60ccc2700 1 -- 172.21.15.189:0/1987125858 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6081f3610 con 0x7fa6080b2fc0 2024-04-06T12:28:06.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.537+0000 7fa60ccc2700 1 -- 172.21.15.189:0/1987125858 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa608067bb0 con 0x7fa6080b2fc0 2024-04-06T12:28:06.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.537+0000 7fa5f6ffd700 1 -- 172.21.15.189:0/1987125858 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fa5f80075a0 con 0x7fa6080b2fc0 2024-04-06T12:28:06.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.538+0000 7fa5f6ffd700 1 --2- 172.21.15.189:0/1987125858 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5e806adb0 0x7fa5e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:06.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.538+0000 7fa605d9b700 1 --2- 172.21.15.189:0/1987125858 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5e806adb0 0x7fa5e806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:06.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.538+0000 7fa5f6ffd700 1 -- 172.21.15.189:0/1987125858 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa5f8088890 con 0x7fa6080b2fc0 2024-04-06T12:28:06.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.538+0000 7fa605d9b700 1 --2- 172.21.15.189:0/1987125858 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5e806adb0 0x7fa5e806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc000c00 tx=0x7fa5fc008030 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:06.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.542+0000 7fa5f6ffd700 1 -- 172.21.15.189:0/1987125858 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa5f80538d0 con 0x7fa6080b2fc0 2024-04-06T12:28:06.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:06.788+0000 7fa60ccc2700 1 -- 172.21.15.189:0/1987125858 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa608074990 con 0x7fa6080b2fc0 2024-04-06T12:28:07.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.681+0000 7fa5f6ffd700 1 -- 172.21.15.189:0/1987125858 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v38) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa5f80170b0 con 0x7fa6080b2fc0 2024-04-06T12:28:07.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.684+0000 7fa60ccc2700 1 -- 172.21.15.189:0/1987125858 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5e806adb0 msgr2=0x7fa5e806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:07.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.684+0000 7fa60ccc2700 1 --2- 172.21.15.189:0/1987125858 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5e806adb0 0x7fa5e806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc000c00 tx=0x7fa5fc008030 comp rx=0 tx=0).stop 2024-04-06T12:28:07.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.684+0000 7fa60ccc2700 1 -- 172.21.15.189:0/1987125858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6080b2fc0 msgr2=0x7fa6081f2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:07.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.684+0000 7fa60ccc2700 1 --2- 172.21.15.189:0/1987125858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6080b2fc0 0x7fa6081f2a70 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fa5f800c520 tx=0x7fa5f800c830 comp rx=0 tx=0).stop 2024-04-06T12:28:07.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.684+0000 7fa60ccc2700 1 -- 172.21.15.189:0/1987125858 shutdown_connections 2024-04-06T12:28:07.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.684+0000 7fa60ccc2700 1 --2- 172.21.15.189:0/1987125858 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa6080ac8c0 0x7fa6080cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:07.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.684+0000 7fa60ccc2700 1 --2- 172.21.15.189:0/1987125858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6080b2fc0 0x7fa6081f2a70 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:07.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.684+0000 7fa60ccc2700 1 --2- 172.21.15.189:0/1987125858 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5e806adb0 0x7fa5e806d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:07.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.684+0000 7fa60ccc2700 1 --2- 172.21.15.189:0/1987125858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6080b4660 0x7fa608153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:07.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.685+0000 7fa60ccc2700 1 -- 172.21.15.189:0/1987125858 >> 172.21.15.189:0/1987125858 conn(0x7fa6080d00b0 msgr2=0x7fa60806f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:07.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.685+0000 7fa60ccc2700 1 -- 172.21.15.189:0/1987125858 shutdown_connections 2024-04-06T12:28:07.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.685+0000 7fa60ccc2700 1 -- 172.21.15.189:0/1987125858 wait complete. 2024-04-06T12:28:07.697 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-06T12:28:07.698 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-06T12:28:07.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.830+0000 7f7958c75700 1 Processor -- start 2024-04-06T12:28:07.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.831+0000 7f7958c75700 1 -- start start 2024-04-06T12:28:07.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f7958c75700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f795406ae40 0x7f79540b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:07.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f7958c75700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79540b3990 con 0x7f795415d7b0 2024-04-06T12:28:07.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f7958c75700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79540b3060 con 0x7f795415ddf0 2024-04-06T12:28:07.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f7958c75700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f795415e950 con 0x7f795406ae40 2024-04-06T12:28:07.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f795259c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f795415d7b0 0x7f795406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55444/0 (socket says 172.21.15.189:55444) 2024-04-06T12:28:07.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f795259c700 1 -- 172.21.15.189:0/3540998127 learned_addr learned my addr 172.21.15.189:0/3540998127 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:07.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f795159a700 1 -- 172.21.15.189:0/3540998127 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885390332 0 0) 0x7f79540b3990 con 0x7f795415d7b0 2024-04-06T12:28:07.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f7951d9b700 1 --2- 172.21.15.189:0/3540998127 >> v2:172.21.15.192:3300/0 conn(0x7f795406ae40 0x7f79540b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:07.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f795159a700 1 -- 172.21.15.189:0/3540998127 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79400035b0 con 0x7f795415d7b0 2024-04-06T12:28:07.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f795159a700 1 -- 172.21.15.189:0/3540998127 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 813574493 0 0) 0x7f79400035b0 con 0x7f795415d7b0 2024-04-06T12:28:07.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f795159a700 1 -- 172.21.15.189:0/3540998127 >> v2:172.21.15.192:3300/0 conn(0x7f795406ae40 msgr2=0x7f79540b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:07.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f795159a700 1 --2- 172.21.15.189:0/3540998127 >> v2:172.21.15.192:3300/0 conn(0x7f795406ae40 0x7f79540b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:07.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f795159a700 1 -- 172.21.15.189:0/3540998127 >> v1:172.21.15.192:6789/0 conn(0x7f795415ddf0 legacy=0x7f795415e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:07.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f795159a700 1 -- 172.21.15.189:0/3540998127 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79540b3340 con 0x7f795415d7b0 2024-04-06T12:28:07.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f795159a700 1 -- 172.21.15.189:0/3540998127 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f7944001670 con 0x7f795415d7b0 2024-04-06T12:28:07.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f795159a700 1 -- 172.21.15.189:0/3540998127 >> v1:172.21.15.189:6789/0 conn(0x7f795415d7b0 legacy=0x7f795406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:07.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f795159a700 1 --2- 172.21.15.189:0/3540998127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7940003a00 0x7f7940003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f795159a700 1 --2- 172.21.15.189:0/3540998127 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7940004850 0x7f7940004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f795159a700 1 --2- 172.21.15.189:0/3540998127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7940005210 0x7f794000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f795159a700 1 -- 172.21.15.189:0/3540998127 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794000c180 con 0x7f7940005210 2024-04-06T12:28:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f795159a700 1 -- 172.21.15.189:0/3540998127 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794000c2f0 con 0x7f7940003a00 2024-04-06T12:28:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f795159a700 1 -- 172.21.15.189:0/3540998127 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794000c4b0 con 0x7f7940004850 2024-04-06T12:28:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f795259c700 1 --2- 172.21.15.189:0/3540998127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7940003a00 0x7f7940003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.832+0000 7f7952d9d700 1 --2- 172.21.15.189:0/3540998127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7940005210 0x7f794000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.833+0000 7f7951d9b700 1 --2- 172.21.15.189:0/3540998127 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7940004850 0x7f7940004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.833+0000 7f7952d9d700 1 -- 172.21.15.189:0/3540998127 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7940004850 msgr2=0x7f7940004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.833+0000 7f7952d9d700 1 --2- 172.21.15.189:0/3540998127 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7940004850 0x7f7940004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.833+0000 7f7952d9d700 1 -- 172.21.15.189:0/3540998127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7940003a00 msgr2=0x7f7940003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.833+0000 7f7952d9d700 1 --2- 172.21.15.189:0/3540998127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7940003a00 0x7f7940003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.833+0000 7f7952d9d700 1 -- 172.21.15.189:0/3540998127 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79400035b0 con 0x7f7940005210 2024-04-06T12:28:07.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.833+0000 7f7952d9d700 1 -- 172.21.15.189:0/3540998127 reap_dead start 2024-04-06T12:28:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.833+0000 7f7952d9d700 1 --2- 172.21.15.189:0/3540998127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7940005210 0x7f794000ba10 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f794c00c190 tx=0x7f7940005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a85fcfabb4e30f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.833+0000 7f795159a700 1 -- 172.21.15.189:0/3540998127 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794c001db0 con 0x7f7940005210 2024-04-06T12:28:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.833+0000 7f795159a700 1 -- 172.21.15.189:0/3540998127 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f794c00ccd0 con 0x7f7940005210 2024-04-06T12:28:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.833+0000 7f795159a700 1 -- 172.21.15.189:0/3540998127 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794c003c40 con 0x7f7940005210 2024-04-06T12:28:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.833+0000 7f7958c75700 1 -- 172.21.15.189:0/3540998127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7940005210 msgr2=0x7f794000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.833+0000 7f7958c75700 1 --2- 172.21.15.189:0/3540998127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7940005210 0x7f794000ba10 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f794c00c190 tx=0x7f7940005880 comp rx=0 tx=0).stop 2024-04-06T12:28:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.834+0000 7f7958c75700 1 -- 172.21.15.189:0/3540998127 shutdown_connections 2024-04-06T12:28:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.834+0000 7f7958c75700 1 --2- 172.21.15.189:0/3540998127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7940005210 0x7f794000ba10 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.834+0000 7f7958c75700 1 -- 172.21.15.189:0/3540998127 >> 172.21.15.189:0/3540998127 conn(0x7f79540d00b0 msgr2=0x7f79540d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.834+0000 7f7958c75700 1 -- 172.21.15.189:0/3540998127 shutdown_connections 2024-04-06T12:28:07.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.834+0000 7f7958c75700 1 -- 172.21.15.189:0/3540998127 wait complete. 2024-04-06T12:28:07.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.835+0000 7f7958c75700 1 Processor -- start 2024-04-06T12:28:07.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.835+0000 7f7958c75700 1 -- start start 2024-04-06T12:28:07.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f7958c75700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7940005210 0x7f795414bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:07.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f7958c75700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f79541f5d40 0x7f79541f61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:07.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f7958c75700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79541f6700 0x7f7954153ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f7958c75700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7954159000 con 0x7f7940005210 2024-04-06T12:28:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f7958c75700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7954158100 con 0x7f79541f6700 2024-04-06T12:28:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f7958c75700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7954158380 con 0x7f79541f5d40 2024-04-06T12:28:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f795259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7940005210 0x7f795414bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f795259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7940005210 0x7f795414bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58022/0 (socket says 172.21.15.189:58022) 2024-04-06T12:28:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f795259c700 1 -- 172.21.15.189:0/809661937 learned_addr learned my addr 172.21.15.189:0/809661937 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f7951d9b700 1 --2- 172.21.15.189:0/809661937 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f79541f5d40 0x7f79541f61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f7952d9d700 1 --2- 172.21.15.189:0/809661937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79541f6700 0x7f7954153ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f795259c700 1 -- 172.21.15.189:0/809661937 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f79541f5d40 msgr2=0x7f79541f61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f795259c700 1 --2- 172.21.15.189:0/809661937 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f79541f5d40 0x7f79541f61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f795259c700 1 -- 172.21.15.189:0/809661937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79541f6700 msgr2=0x7f7954153ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f795259c700 1 --2- 172.21.15.189:0/809661937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79541f6700 0x7f7954153ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f795259c700 1 -- 172.21.15.189:0/809661937 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7954158660 con 0x7f7940005210 2024-04-06T12:28:07.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f795259c700 1 --2- 172.21.15.189:0/809661937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7940005210 0x7f795414bb20 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f79440072c0 tx=0x7f7944007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f794affd700 1 -- 172.21.15.189:0/809661937 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79440077a0 con 0x7f7940005210 2024-04-06T12:28:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f794affd700 1 -- 172.21.15.189:0/809661937 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f794400f8d0 con 0x7f7940005210 2024-04-06T12:28:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f7958c75700 1 -- 172.21.15.189:0/809661937 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7954158940 con 0x7f7940005210 2024-04-06T12:28:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f794affd700 1 -- 172.21.15.189:0/809661937 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794400fc80 con 0x7f7940005210 2024-04-06T12:28:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.836+0000 7f7958c75700 1 -- 172.21.15.189:0/809661937 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79541f6b80 con 0x7f7940005210 2024-04-06T12:28:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.837+0000 7f7958c75700 1 -- 172.21.15.189:0/809661937 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7954067bb0 con 0x7f7940005210 2024-04-06T12:28:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.838+0000 7f794affd700 1 -- 172.21.15.189:0/809661937 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f794401d460 con 0x7f7940005210 2024-04-06T12:28:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.838+0000 7f794affd700 1 --2- 172.21.15.189:0/809661937 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f793406adb0 0x7f793406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.838+0000 7f7951d9b700 1 --2- 172.21.15.189:0/809661937 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f793406adb0 0x7f793406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.839+0000 7f794affd700 1 -- 172.21.15.189:0/809661937 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7944089550 con 0x7f7940005210 2024-04-06T12:28:07.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.839+0000 7f7951d9b700 1 --2- 172.21.15.189:0/809661937 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f793406adb0 0x7f793406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f79541f5ca0 tx=0x7f793c008040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:07.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:07.843+0000 7f794affd700 1 -- 172.21.15.189:0/809661937 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7944054570 con 0x7f7940005210 2024-04-06T12:28:08.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.082+0000 7f7958c75700 1 -- 172.21.15.189:0/809661937 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7954074990 con 0x7f7940005210 2024-04-06T12:28:08.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.683+0000 7f794affd700 1 -- 172.21.15.189:0/809661937 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v39) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7944057b90 con 0x7f7940005210 2024-04-06T12:28:08.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.685+0000 7f7958c75700 1 -- 172.21.15.189:0/809661937 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f793406adb0 msgr2=0x7f793406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:08.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.685+0000 7f7958c75700 1 --2- 172.21.15.189:0/809661937 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f793406adb0 0x7f793406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f79541f5ca0 tx=0x7f793c008040 comp rx=0 tx=0).stop 2024-04-06T12:28:08.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.685+0000 7f7958c75700 1 -- 172.21.15.189:0/809661937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7940005210 msgr2=0x7f795414bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:08.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.685+0000 7f7958c75700 1 --2- 172.21.15.189:0/809661937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7940005210 0x7f795414bb20 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f79440072c0 tx=0x7f7944007680 comp rx=0 tx=0).stop 2024-04-06T12:28:08.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.685+0000 7f7958c75700 1 -- 172.21.15.189:0/809661937 shutdown_connections 2024-04-06T12:28:08.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.685+0000 7f7958c75700 1 --2- 172.21.15.189:0/809661937 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f793406adb0 0x7f793406d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:08.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.685+0000 7f7958c75700 1 --2- 172.21.15.189:0/809661937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79541f6700 0x7f7954153ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:08.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.685+0000 7f7958c75700 1 --2- 172.21.15.189:0/809661937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7940005210 0x7f795414bb20 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:08.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.685+0000 7f7958c75700 1 --2- 172.21.15.189:0/809661937 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f79541f5d40 0x7f79541f61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:08.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.686+0000 7f7958c75700 1 -- 172.21.15.189:0/809661937 >> 172.21.15.189:0/809661937 conn(0x7f79540d00b0 msgr2=0x7f7954156d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:08.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.686+0000 7f7958c75700 1 -- 172.21.15.189:0/809661937 shutdown_connections 2024-04-06T12:28:08.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.686+0000 7f7958c75700 1 -- 172.21.15.189:0/809661937 wait complete. 2024-04-06T12:28:08.698 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-06T12:28:08.698 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:28:08.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.825+0000 7f732612c700 1 Processor -- start 2024-04-06T12:28:08.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.825+0000 7f732612c700 1 -- start start 2024-04-06T12:28:08.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.826+0000 7f732612c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f732013e340 0x7f732013e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:08.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.826+0000 7f732612c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f732015c380 0x7f73200cbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:08.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.826+0000 7f732612c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73200cdeb0 con 0x7f73200cb9d0 2024-04-06T12:28:08.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.826+0000 7f732612c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73200cc3a0 con 0x7f732013e340 2024-04-06T12:28:08.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.826+0000 7f732612c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732015c810 con 0x7f732015c380 2024-04-06T12:28:08.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.826+0000 7f731f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f732013e340 0x7f732013e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:08.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.826+0000 7f731effd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f73200cb9d0 0x7f732015bc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55454/0 (socket says 172.21.15.189:55454) 2024-04-06T12:28:08.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.826+0000 7f731effd700 1 -- 172.21.15.189:0/1032232064 learned_addr learned my addr 172.21.15.189:0/1032232064 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:08.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.827+0000 7f731ffff700 1 --2- 172.21.15.189:0/1032232064 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f732015c380 0x7f73200cbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:08.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.827+0000 7f731f7fe700 1 -- 172.21.15.189:0/1032232064 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f732015c380 msgr2=0x7f73200cbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:08.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.827+0000 7f731f7fe700 1 --2- 172.21.15.189:0/1032232064 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f732015c380 0x7f73200cbe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:08.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.827+0000 7f731f7fe700 1 -- 172.21.15.189:0/1032232064 >> v1:172.21.15.189:6789/0 conn(0x7f73200cb9d0 legacy=0x7f732015bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:08.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.827+0000 7f731f7fe700 1 -- 172.21.15.189:0/1032232064 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73200ce190 con 0x7f732013e340 2024-04-06T12:28:08.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.827+0000 7f731e7fc700 1 -- 172.21.15.189:0/1032232064 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183271972 0 0) 0x7f73200cdeb0 con 0x7f73200cb9d0 2024-04-06T12:28:08.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.827+0000 7f731f7fe700 1 --2- 172.21.15.189:0/1032232064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732013e340 0x7f732013e8b0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f7308009870 tx=0x7f7308009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52619a61c238216e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:08.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.827+0000 7f731e7fc700 1 -- 172.21.15.189:0/1032232064 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7308011830 con 0x7f732013e340 2024-04-06T12:28:08.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.827+0000 7f731e7fc700 1 -- 172.21.15.189:0/1032232064 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7308011990 con 0x7f732013e340 2024-04-06T12:28:08.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.827+0000 7f731e7fc700 1 -- 172.21.15.189:0/1032232064 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7308011d20 con 0x7f732013e340 2024-04-06T12:28:08.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.828+0000 7f732612c700 1 -- 172.21.15.189:0/1032232064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732013e340 msgr2=0x7f732013e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:08.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.828+0000 7f732612c700 1 --2- 172.21.15.189:0/1032232064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732013e340 0x7f732013e8b0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f7308009870 tx=0x7f7308009b80 comp rx=0 tx=0).stop 2024-04-06T12:28:08.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.828+0000 7f732612c700 1 -- 172.21.15.189:0/1032232064 shutdown_connections 2024-04-06T12:28:08.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.828+0000 7f732612c700 1 --2- 172.21.15.189:0/1032232064 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f732015c380 0x7f73200cbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:08.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.828+0000 7f732612c700 1 --2- 172.21.15.189:0/1032232064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732013e340 0x7f732013e8b0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:08.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.828+0000 7f732612c700 1 -- 172.21.15.189:0/1032232064 >> 172.21.15.189:0/1032232064 conn(0x7f732006a730 msgr2=0x7f732006ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:08.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.828+0000 7f732612c700 1 -- 172.21.15.189:0/1032232064 shutdown_connections 2024-04-06T12:28:08.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.828+0000 7f732612c700 1 -- 172.21.15.189:0/1032232064 wait complete. 2024-04-06T12:28:08.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.829+0000 7f732612c700 1 Processor -- start 2024-04-06T12:28:08.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.829+0000 7f732612c700 1 -- start start 2024-04-06T12:28:08.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f732612c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73200cb9d0 0x7f732015b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:08.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f732612c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732013e340 0x7f73201595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:08.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f732612c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f732015c380 0x7f7320159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:08.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f732612c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73201f4bd0 con 0x7f732013e340 2024-04-06T12:28:08.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f732612c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73200cf8b0 con 0x7f732015c380 2024-04-06T12:28:08.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f732612c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73201f48f0 con 0x7f73200cb9d0 2024-04-06T12:28:08.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f731effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732013e340 0x7f73201595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:08.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f731effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732013e340 0x7f73201595e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58038/0 (socket says 172.21.15.189:58038) 2024-04-06T12:28:08.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f731effd700 1 -- 172.21.15.189:0/1021976679 learned_addr learned my addr 172.21.15.189:0/1021976679 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:08.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f731f7fe700 1 --2- 172.21.15.189:0/1021976679 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73200cb9d0 0x7f732015b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:08.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f731ffff700 1 --2- 172.21.15.189:0/1021976679 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f732015c380 0x7f7320159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:08.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f731effd700 1 -- 172.21.15.189:0/1021976679 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73200cb9d0 msgr2=0x7f732015b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:08.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f731effd700 1 --2- 172.21.15.189:0/1021976679 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73200cb9d0 0x7f732015b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:08.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f731effd700 1 -- 172.21.15.189:0/1021976679 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f732015c380 msgr2=0x7f7320159b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:08.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f731effd700 1 --2- 172.21.15.189:0/1021976679 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f732015c380 0x7f7320159b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:08.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f731effd700 1 -- 172.21.15.189:0/1021976679 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73080094a0 con 0x7f732013e340 2024-04-06T12:28:08.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f731effd700 1 --2- 172.21.15.189:0/1021976679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732013e340 0x7f73201595e0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f731000c550 tx=0x7f731000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:08.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f7317fff700 1 -- 172.21.15.189:0/1021976679 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7310004c80 con 0x7f732013e340 2024-04-06T12:28:08.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f7317fff700 1 -- 172.21.15.189:0/1021976679 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7310004e00 con 0x7f732013e340 2024-04-06T12:28:08.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.830+0000 7f7317fff700 1 -- 172.21.15.189:0/1021976679 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7310007310 con 0x7f732013e340 2024-04-06T12:28:08.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.831+0000 7f732612c700 1 -- 172.21.15.189:0/1021976679 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73201f1950 con 0x7f732013e340 2024-04-06T12:28:08.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.831+0000 7f732612c700 1 -- 172.21.15.189:0/1021976679 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73201f1f10 con 0x7f732013e340 2024-04-06T12:28:08.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.832+0000 7f732612c700 1 -- 172.21.15.189:0/1021976679 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73201f2130 con 0x7f732013e340 2024-04-06T12:28:08.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.832+0000 7f7317fff700 1 -- 172.21.15.189:0/1021976679 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7310007470 con 0x7f732013e340 2024-04-06T12:28:08.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.832+0000 7f7317fff700 1 --2- 172.21.15.189:0/1021976679 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f730006adb0 0x7f730006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:08.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.833+0000 7f7317fff700 1 -- 172.21.15.189:0/1021976679 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7310089270 con 0x7f732013e340 2024-04-06T12:28:08.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.833+0000 7f731f7fe700 1 --2- 172.21.15.189:0/1021976679 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f730006adb0 0x7f730006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:08.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.833+0000 7f731f7fe700 1 --2- 172.21.15.189:0/1021976679 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f730006adb0 0x7f730006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f73080090b0 tx=0x7f730801c040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:08.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:08.837+0000 7f7317fff700 1 -- 172.21.15.189:0/1021976679 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7310054160 con 0x7f732013e340 2024-04-06T12:28:09.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.077+0000 7f732612c700 1 -- 172.21.15.189:0/1021976679 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7320073270 con 0x7f732013e340 2024-04-06T12:28:09.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.077+0000 7f7317fff700 1 -- 172.21.15.189:0/1021976679 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f7310057780 con 0x7f732013e340 2024-04-06T12:28:09.075 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:09.075 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":39,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3065963154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3065963154},{"type":"v1","addr":"172.21.15.192:6833","nonce":3065963154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6071,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/831568588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":831568588},{"type":"v1","addr":"172.21.15.192:6835","nonce":831568588}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":7080,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1688936113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1688936113},{"type":"v1","addr":"172.21.15.189:6836","nonce":1688936113}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:28:05.673337+0000","modified":"2024-04-06T12:28:08.681505+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7083},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7083":{"gid":7083,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":4,"addr":"172.21.15.189:6837/3778396425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3778396425},{"type":"v1","addr":"172.21.15.189:6837","nonce":3778396425}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-04-06T12:28:09.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.080+0000 7f732612c700 1 -- 172.21.15.189:0/1021976679 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f730006adb0 msgr2=0x7f730006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:09.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.080+0000 7f732612c700 1 --2- 172.21.15.189:0/1021976679 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f730006adb0 0x7f730006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f73080090b0 tx=0x7f730801c040 comp rx=0 tx=0).stop 2024-04-06T12:28:09.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.080+0000 7f732612c700 1 -- 172.21.15.189:0/1021976679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732013e340 msgr2=0x7f73201595e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:09.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.080+0000 7f732612c700 1 --2- 172.21.15.189:0/1021976679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732013e340 0x7f73201595e0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f731000c550 tx=0x7f731000c860 comp rx=0 tx=0).stop 2024-04-06T12:28:09.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.080+0000 7f732612c700 1 -- 172.21.15.189:0/1021976679 shutdown_connections 2024-04-06T12:28:09.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.080+0000 7f732612c700 1 --2- 172.21.15.189:0/1021976679 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f730006adb0 0x7f730006d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.080+0000 7f732612c700 1 --2- 172.21.15.189:0/1021976679 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f732015c380 0x7f7320159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.080+0000 7f732612c700 1 --2- 172.21.15.189:0/1021976679 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73200cb9d0 0x7f732015b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.080+0000 7f732612c700 1 --2- 172.21.15.189:0/1021976679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732013e340 0x7f73201595e0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.080+0000 7f732612c700 1 -- 172.21.15.189:0/1021976679 >> 172.21.15.189:0/1021976679 conn(0x7f732006a730 msgr2=0x7f7320140a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:09.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.080+0000 7f732612c700 1 -- 172.21.15.189:0/1021976679 shutdown_connections 2024-04-06T12:28:09.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.080+0000 7f732612c700 1 -- 172.21.15.189:0/1021976679 wait complete. 2024-04-06T12:28:09.079 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 39 2024-04-06T12:28:09.091 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:28:09.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.214+0000 7ffbb783a700 1 Processor -- start 2024-04-06T12:28:09.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.214+0000 7ffbb783a700 1 -- start start 2024-04-06T12:28:09.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.215+0000 7ffbb783a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbb015d7b0 0x7ffbb00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.215+0000 7ffbb783a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ffbb006a730 0x7ffbb006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.215+0000 7ffbb783a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb00b3990 con 0x7ffbb015ddf0 2024-04-06T12:28:09.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.215+0000 7ffbb783a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbb006b0f0 con 0x7ffbb006a730 2024-04-06T12:28:09.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.215+0000 7ffbb783a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb00b3060 con 0x7ffbb015d7b0 2024-04-06T12:28:09.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.215+0000 7ffbb55d6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbb015d7b0 0x7ffbb00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.215+0000 7ffbb55d6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbb015d7b0 0x7ffbb00b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:32804/0 (socket says 172.21.15.189:32804) 2024-04-06T12:28:09.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.215+0000 7ffbb55d6700 1 -- 172.21.15.189:0/1655073955 learned_addr learned my addr 172.21.15.189:0/1655073955 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:09.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.215+0000 7ffbb4dd5700 1 --2- 172.21.15.189:0/1655073955 >> v2:172.21.15.192:3300/0 conn(0x7ffbb006a730 0x7ffbb006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.216+0000 7ffbb55d6700 1 -- 172.21.15.189:0/1655073955 >> v2:172.21.15.192:3300/0 conn(0x7ffbb006a730 msgr2=0x7ffbb006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:09.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.216+0000 7ffbb55d6700 1 --2- 172.21.15.189:0/1655073955 >> v2:172.21.15.192:3300/0 conn(0x7ffbb006a730 0x7ffbb006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.216+0000 7ffbb55d6700 1 -- 172.21.15.189:0/1655073955 >> v1:172.21.15.192:6789/0 conn(0x7ffbb015ddf0 legacy=0x7ffbb015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:09.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.216+0000 7ffbb55d6700 1 -- 172.21.15.189:0/1655073955 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb00b3340 con 0x7ffbb015d7b0 2024-04-06T12:28:09.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.216+0000 7ffbb55d6700 1 --2- 172.21.15.189:0/1655073955 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbb015d7b0 0x7ffbb00b3e20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffba00098f0 tx=0x7ffba0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37e32dd66820c021 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:09.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.216+0000 7ffba7fff700 1 -- 172.21.15.189:0/1655073955 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba0011830 con 0x7ffbb015d7b0 2024-04-06T12:28:09.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.216+0000 7ffba7fff700 1 -- 172.21.15.189:0/1655073955 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba0011990 con 0x7ffbb015d7b0 2024-04-06T12:28:09.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.216+0000 7ffba7fff700 1 -- 172.21.15.189:0/1655073955 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba0011d20 con 0x7ffbb015d7b0 2024-04-06T12:28:09.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.217+0000 7ffbb783a700 1 -- 172.21.15.189:0/1655073955 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbb015d7b0 msgr2=0x7ffbb00b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:09.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.217+0000 7ffbb783a700 1 --2- 172.21.15.189:0/1655073955 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbb015d7b0 0x7ffbb00b3e20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffba00098f0 tx=0x7ffba0009c00 comp rx=0 tx=0).stop 2024-04-06T12:28:09.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.217+0000 7ffbb783a700 1 -- 172.21.15.189:0/1655073955 shutdown_connections 2024-04-06T12:28:09.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.217+0000 7ffbb783a700 1 --2- 172.21.15.189:0/1655073955 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbb015d7b0 0x7ffbb00b3e20 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.217+0000 7ffbb783a700 1 --2- 172.21.15.189:0/1655073955 >> v2:172.21.15.192:3300/0 conn(0x7ffbb006a730 0x7ffbb006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.217+0000 7ffbb783a700 1 -- 172.21.15.189:0/1655073955 >> 172.21.15.189:0/1655073955 conn(0x7ffbb00d00b0 msgr2=0x7ffbb00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:09.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.217+0000 7ffbb783a700 1 -- 172.21.15.189:0/1655073955 shutdown_connections 2024-04-06T12:28:09.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.217+0000 7ffbb783a700 1 -- 172.21.15.189:0/1655073955 wait complete. 2024-04-06T12:28:09.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.218+0000 7ffbb783a700 1 Processor -- start 2024-04-06T12:28:09.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.218+0000 7ffbb783a700 1 -- start start 2024-04-06T12:28:09.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.219+0000 7ffbb783a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbb006a730 0x7ffbb014fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.219+0000 7ffbb783a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbb015d7b0 0x7ffbb00ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.219+0000 7ffbb783a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbb015ddf0 0x7ffbb00cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.219+0000 7ffbb783a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb0158370 con 0x7ffbb015ddf0 2024-04-06T12:28:09.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.219+0000 7ffbb783a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb0159600 con 0x7ffbb015d7b0 2024-04-06T12:28:09.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.219+0000 7ffbb783a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb0158560 con 0x7ffbb006a730 2024-04-06T12:28:09.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.219+0000 7ffbb5dd7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbb015ddf0 0x7ffbb00cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.219+0000 7ffbb4dd5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbb015d7b0 0x7ffbb00ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.219+0000 7ffbb5dd7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbb015ddf0 0x7ffbb00cd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58044/0 (socket says 172.21.15.189:58044) 2024-04-06T12:28:09.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.219+0000 7ffbb55d6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbb006a730 0x7ffbb014fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.219+0000 7ffbb5dd7700 1 -- 172.21.15.189:0/2628380453 learned_addr learned my addr 172.21.15.189:0/2628380453 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:09.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.219+0000 7ffbb5dd7700 1 -- 172.21.15.189:0/2628380453 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbb006a730 msgr2=0x7ffbb014fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:09.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.219+0000 7ffbb5dd7700 1 --2- 172.21.15.189:0/2628380453 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbb006a730 0x7ffbb014fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.219+0000 7ffbb5dd7700 1 -- 172.21.15.189:0/2628380453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbb015d7b0 msgr2=0x7ffbb00ccec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:09.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.219+0000 7ffbb5dd7700 1 --2- 172.21.15.189:0/2628380453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbb015d7b0 0x7ffbb00ccec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.219+0000 7ffbb5dd7700 1 -- 172.21.15.189:0/2628380453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba0009550 con 0x7ffbb015ddf0 2024-04-06T12:28:09.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.219+0000 7ffbb5dd7700 1 --2- 172.21.15.189:0/2628380453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbb015ddf0 0x7ffbb00cd400 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7ffba800cb80 tx=0x7ffba800ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:09.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.219+0000 7ffba5ffb700 1 -- 172.21.15.189:0/2628380453 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba8003c00 con 0x7ffbb015ddf0 2024-04-06T12:28:09.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.220+0000 7ffba5ffb700 1 -- 172.21.15.189:0/2628380453 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba8003d60 con 0x7ffbb015ddf0 2024-04-06T12:28:09.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.220+0000 7ffbb783a700 1 -- 172.21.15.189:0/2628380453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb0158840 con 0x7ffbb015ddf0 2024-04-06T12:28:09.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.220+0000 7ffba5ffb700 1 -- 172.21.15.189:0/2628380453 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba800aaa0 con 0x7ffbb015ddf0 2024-04-06T12:28:09.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.220+0000 7ffbb783a700 1 -- 172.21.15.189:0/2628380453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb0158e00 con 0x7ffbb015ddf0 2024-04-06T12:28:09.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.221+0000 7ffbb783a700 1 -- 172.21.15.189:0/2628380453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb0067bb0 con 0x7ffbb015ddf0 2024-04-06T12:28:09.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.221+0000 7ffba5ffb700 1 -- 172.21.15.189:0/2628380453 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7ffba800ac00 con 0x7ffbb015ddf0 2024-04-06T12:28:09.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.221+0000 7ffba5ffb700 1 --2- 172.21.15.189:0/2628380453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb9406adb0 0x7ffb9406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.222+0000 7ffbb55d6700 1 --2- 172.21.15.189:0/2628380453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb9406adb0 0x7ffb9406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.222+0000 7ffba5ffb700 1 -- 172.21.15.189:0/2628380453 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ffba8089540 con 0x7ffbb015ddf0 2024-04-06T12:28:09.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.222+0000 7ffbb55d6700 1 --2- 172.21.15.189:0/2628380453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb9406adb0 0x7ffb9406d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffba0009c40 tx=0x7ffba0008030 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:09.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.226+0000 7ffba5ffb700 1 -- 172.21.15.189:0/2628380453 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ffba80544d0 con 0x7ffbb015ddf0 2024-04-06T12:28:09.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.421+0000 7ffbb783a700 1 -- 172.21.15.189:0/2628380453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffbb0074990 con 0x7ffbb015ddf0 2024-04-06T12:28:09.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.423+0000 7ffba5ffb700 1 -- 172.21.15.189:0/2628380453 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16357 (secure 0 0 0) 0x7ffba8057af0 con 0x7ffbb015ddf0 2024-04-06T12:28:09.421 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:09.421 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":98,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:28:06.674829+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:28:04.080500+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-06T12:28:04.619802+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:28:09.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.425+0000 7ffbb783a700 1 -- 172.21.15.189:0/2628380453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb9406adb0 msgr2=0x7ffb9406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:09.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.425+0000 7ffbb783a700 1 --2- 172.21.15.189:0/2628380453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb9406adb0 0x7ffb9406d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffba0009c40 tx=0x7ffba0008030 comp rx=0 tx=0).stop 2024-04-06T12:28:09.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.425+0000 7ffbb783a700 1 -- 172.21.15.189:0/2628380453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbb015ddf0 msgr2=0x7ffbb00cd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:09.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.425+0000 7ffbb783a700 1 --2- 172.21.15.189:0/2628380453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbb015ddf0 0x7ffbb00cd400 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7ffba800cb80 tx=0x7ffba800ce90 comp rx=0 tx=0).stop 2024-04-06T12:28:09.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.425+0000 7ffbb783a700 1 -- 172.21.15.189:0/2628380453 shutdown_connections 2024-04-06T12:28:09.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.425+0000 7ffbb783a700 1 --2- 172.21.15.189:0/2628380453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbb015ddf0 0x7ffbb00cd400 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.425+0000 7ffbb783a700 1 --2- 172.21.15.189:0/2628380453 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbb006a730 0x7ffbb014fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.425+0000 7ffbb783a700 1 --2- 172.21.15.189:0/2628380453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb9406adb0 0x7ffb9406d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.425+0000 7ffbb783a700 1 --2- 172.21.15.189:0/2628380453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbb015d7b0 0x7ffbb00ccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.425+0000 7ffbb783a700 1 -- 172.21.15.189:0/2628380453 >> 172.21.15.189:0/2628380453 conn(0x7ffbb00d00b0 msgr2=0x7ffbb006fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:09.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.425+0000 7ffbb783a700 1 -- 172.21.15.189:0/2628380453 shutdown_connections 2024-04-06T12:28:09.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.425+0000 7ffbb783a700 1 -- 172.21.15.189:0/2628380453 wait complete. 2024-04-06T12:28:09.438 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-06T12:28:09.438 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:28:09.559 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.561+0000 7f231c64b700 1 Processor -- start 2024-04-06T12:28:09.559 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.561+0000 7f231c64b700 1 -- start start 2024-04-06T12:28:09.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.562+0000 7f231c64b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f23140d17a0 0x7f231406b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.562+0000 7f231c64b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2314146dd0 0x7f231406d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.562+0000 7f231c64b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231406c0a0 con 0x7f23140cedf0 2024-04-06T12:28:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.562+0000 7f231c64b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f231406c380 con 0x7f23140d17a0 2024-04-06T12:28:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.562+0000 7f231c64b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f231406c660 con 0x7f2314146dd0 2024-04-06T12:28:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.562+0000 7f231abe8700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f23140cedf0 0x7f23141441f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44592/0 (socket says 172.21.15.189:44592) 2024-04-06T12:28:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.562+0000 7f231abe8700 1 -- 172.21.15.189:0/1334475049 learned_addr learned my addr 172.21.15.189:0/1334475049 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.562+0000 7f231a3e7700 1 --2- 172.21.15.189:0/1334475049 >> v2:172.21.15.192:3300/0 conn(0x7f23140d17a0 0x7f231406b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.562+0000 7f2319be6700 1 --2- 172.21.15.189:0/1334475049 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2314146dd0 0x7f231406d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.562+0000 7f23193e5700 1 -- 172.21.15.189:0/1334475049 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376988663 0 0) 0x7f231406c0a0 con 0x7f23140cedf0 2024-04-06T12:28:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.562+0000 7f23193e5700 1 -- 172.21.15.189:0/1334475049 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23000035b0 con 0x7f23140cedf0 2024-04-06T12:28:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.562+0000 7f231a3e7700 1 -- 172.21.15.189:0/1334475049 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2314146dd0 msgr2=0x7f231406d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.562+0000 7f231a3e7700 1 --2- 172.21.15.189:0/1334475049 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2314146dd0 0x7f231406d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.562+0000 7f231a3e7700 1 -- 172.21.15.189:0/1334475049 >> v1:172.21.15.189:6789/0 conn(0x7f23140cedf0 legacy=0x7f23141441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.563+0000 7f231a3e7700 1 -- 172.21.15.189:0/1334475049 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231406c940 con 0x7f23140d17a0 2024-04-06T12:28:09.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.563+0000 7f231a3e7700 1 --2- 172.21.15.189:0/1334475049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23140d17a0 0x7f231406b7a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2304009840 tx=0x7f2304009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ade29516b5cc5b8e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:09.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.563+0000 7f23193e5700 1 -- 172.21.15.189:0/1334475049 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2304011830 con 0x7f23140d17a0 2024-04-06T12:28:09.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.563+0000 7f23193e5700 1 -- 172.21.15.189:0/1334475049 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2304011990 con 0x7f23140d17a0 2024-04-06T12:28:09.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.563+0000 7f23193e5700 1 -- 172.21.15.189:0/1334475049 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2304011d20 con 0x7f23140d17a0 2024-04-06T12:28:09.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.563+0000 7f231c64b700 1 -- 172.21.15.189:0/1334475049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23140d17a0 msgr2=0x7f231406b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:09.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.563+0000 7f231c64b700 1 --2- 172.21.15.189:0/1334475049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23140d17a0 0x7f231406b7a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2304009840 tx=0x7f2304009b50 comp rx=0 tx=0).stop 2024-04-06T12:28:09.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.564+0000 7f231c64b700 1 -- 172.21.15.189:0/1334475049 shutdown_connections 2024-04-06T12:28:09.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.564+0000 7f231c64b700 1 --2- 172.21.15.189:0/1334475049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23140d17a0 0x7f231406b7a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.564+0000 7f231c64b700 1 --2- 172.21.15.189:0/1334475049 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2314146dd0 0x7f231406d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.564+0000 7f231c64b700 1 -- 172.21.15.189:0/1334475049 >> 172.21.15.189:0/1334475049 conn(0x7f23140b2db0 msgr2=0x7f23140b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:09.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.564+0000 7f231c64b700 1 -- 172.21.15.189:0/1334475049 shutdown_connections 2024-04-06T12:28:09.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.564+0000 7f231c64b700 1 -- 172.21.15.189:0/1334475049 wait complete. 2024-04-06T12:28:09.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.565+0000 7f231c64b700 1 Processor -- start 2024-04-06T12:28:09.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.565+0000 7f231c64b700 1 -- start start 2024-04-06T12:28:09.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.565+0000 7f231c64b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23140cedf0 0x7f2314159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.565+0000 7f231c64b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23140d17a0 0x7f23140c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.565+0000 7f231c64b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2314146dd0 0x7f23140c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.566+0000 7f231c64b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2314158da0 con 0x7f2314146dd0 2024-04-06T12:28:09.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.566+0000 7f231c64b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23141584b0 con 0x7f23140d17a0 2024-04-06T12:28:09.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.566+0000 7f231c64b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23141f39c0 con 0x7f23140cedf0 2024-04-06T12:28:09.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.566+0000 7f231abe8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2314146dd0 0x7f23140c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.566+0000 7f231abe8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2314146dd0 0x7f23140c7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52178/0 (socket says 172.21.15.189:52178) 2024-04-06T12:28:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.566+0000 7f231abe8700 1 -- 172.21.15.189:0/3923980522 learned_addr learned my addr 172.21.15.189:0/3923980522 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.566+0000 7f2319be6700 1 --2- 172.21.15.189:0/3923980522 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23140d17a0 0x7f23140c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.566+0000 7f231a3e7700 1 --2- 172.21.15.189:0/3923980522 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23140cedf0 0x7f2314159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.566+0000 7f231abe8700 1 -- 172.21.15.189:0/3923980522 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23140cedf0 msgr2=0x7f2314159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.566+0000 7f231abe8700 1 --2- 172.21.15.189:0/3923980522 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23140cedf0 0x7f2314159230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.566+0000 7f231abe8700 1 -- 172.21.15.189:0/3923980522 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23140d17a0 msgr2=0x7f23140c35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.566+0000 7f231abe8700 1 --2- 172.21.15.189:0/3923980522 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23140d17a0 0x7f23140c35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.566+0000 7f231abe8700 1 -- 172.21.15.189:0/3923980522 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23040094a0 con 0x7f2314146dd0 2024-04-06T12:28:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.566+0000 7f231abe8700 1 --2- 172.21.15.189:0/3923980522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2314146dd0 0x7f23140c7080 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f230c00caa0 tx=0x7f230c00cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.566+0000 7f230affd700 1 -- 172.21.15.189:0/3923980522 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c00a9e0 con 0x7f2314146dd0 2024-04-06T12:28:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.566+0000 7f230affd700 1 -- 172.21.15.189:0/3923980522 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f230c00ab40 con 0x7f2314146dd0 2024-04-06T12:28:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.566+0000 7f230affd700 1 -- 172.21.15.189:0/3923980522 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c012400 con 0x7f2314146dd0 2024-04-06T12:28:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.566+0000 7f231c64b700 1 -- 172.21.15.189:0/3923980522 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2314158750 con 0x7f2314146dd0 2024-04-06T12:28:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.566+0000 7f231c64b700 1 -- 172.21.15.189:0/3923980522 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2314159770 con 0x7f2314146dd0 2024-04-06T12:28:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.567+0000 7f231c64b700 1 -- 172.21.15.189:0/3923980522 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2314074990 con 0x7f2314146dd0 2024-04-06T12:28:09.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.568+0000 7f230affd700 1 -- 172.21.15.189:0/3923980522 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f230c01b3b0 con 0x7f2314146dd0 2024-04-06T12:28:09.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.568+0000 7f230affd700 1 --2- 172.21.15.189:0/3923980522 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f22f806adb0 0x7f22f806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.568+0000 7f230affd700 1 -- 172.21.15.189:0/3923980522 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f230c089150 con 0x7f2314146dd0 2024-04-06T12:28:09.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.568+0000 7f231a3e7700 1 --2- 172.21.15.189:0/3923980522 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f22f806adb0 0x7f22f806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.569+0000 7f231a3e7700 1 --2- 172.21.15.189:0/3923980522 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f22f806adb0 0x7f22f806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2304000c00 tx=0x7f2304008030 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:09.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.573+0000 7f230affd700 1 -- 172.21.15.189:0/3923980522 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f230c054230 con 0x7f2314146dd0 2024-04-06T12:28:09.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.767+0000 7f231c64b700 1 -- 172.21.15.189:0/3923980522 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f23140d1b80 con 0x7f22f806adb0 2024-04-06T12:28:09.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.775+0000 7f230affd700 1 -- 172.21.15.189:0/3923980522 <== mgr.6442 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289226 (secure 0 0 0) 0x7f2304011630 con 0x7f22f806adb0 2024-04-06T12:28:09.778 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:09.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.779+0000 7f231c64b700 1 -- 172.21.15.189:0/3923980522 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f22f806adb0 msgr2=0x7f22f806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:09.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.779+0000 7f231c64b700 1 --2- 172.21.15.189:0/3923980522 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f22f806adb0 0x7f22f806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2304000c00 tx=0x7f2304008030 comp rx=0 tx=0).stop 2024-04-06T12:28:09.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.779+0000 7f231c64b700 1 -- 172.21.15.189:0/3923980522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2314146dd0 msgr2=0x7f23140c7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:09.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.779+0000 7f231c64b700 1 --2- 172.21.15.189:0/3923980522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2314146dd0 0x7f23140c7080 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f230c00caa0 tx=0x7f230c00cdb0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.779+0000 7f231c64b700 1 -- 172.21.15.189:0/3923980522 shutdown_connections 2024-04-06T12:28:09.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.779+0000 7f231c64b700 1 --2- 172.21.15.189:0/3923980522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2314146dd0 0x7f23140c7080 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.779+0000 7f231c64b700 1 --2- 172.21.15.189:0/3923980522 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23140cedf0 0x7f2314159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.779+0000 7f231c64b700 1 --2- 172.21.15.189:0/3923980522 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f22f806adb0 0x7f22f806d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.779+0000 7f231c64b700 1 --2- 172.21.15.189:0/3923980522 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23140d17a0 0x7f23140c35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.779+0000 7f231c64b700 1 -- 172.21.15.189:0/3923980522 >> 172.21.15.189:0/3923980522 conn(0x7f23140b2db0 msgr2=0x7f23140b2080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:09.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.779+0000 7f231c64b700 1 -- 172.21.15.189:0/3923980522 shutdown_connections 2024-04-06T12:28:09.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.779+0000 7f231c64b700 1 -- 172.21.15.189:0/3923980522 wait complete. 2024-04-06T12:28:09.781 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:28:09.789 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-06T12:28:08.863553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":303,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":317772,"kb_used_data":10556,"kb_used_omap":12,"kb_used_meta":307059,"kb_avail":754656948,"statfs":{"total":773094113280,"available":772768714752,"internally_reserved":0,"allocated":10809344,"data_stored":5933061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":314428984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.611382"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.680019+0000","last_change":"2024-04-06T12:28:05.691699+0000","last_active":"2024-04-06T12:28:06.680019+0000","last_peered":"2024-04-06T12:28:06.680019+0000","last_clean":"2024-04-06T12:28:06.680019+0000","last_became_active":"2024-04-06T12:28:05.691446+0000","last_became_peered":"2024-04-06T12:28:05.691446+0000","last_unstale":"2024-04-06T12:28:06.680019+0000","last_undegraded":"2024-04-06T12:28:06.680019+0000","last_fullsized":"2024-04-06T12:28:06.680019+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:06:40.208413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317327+0000","last_change":"2024-04-06T12:28:05.695965+0000","last_active":"2024-04-06T12:28:07.317327+0000","last_peered":"2024-04-06T12:28:07.317327+0000","last_clean":"2024-04-06T12:28:07.317327+0000","last_became_active":"2024-04-06T12:28:05.695730+0000","last_became_peered":"2024-04-06T12:28:05.695730+0000","last_unstale":"2024-04-06T12:28:07.317327+0000","last_undegraded":"2024-04-06T12:28:07.317327+0000","last_fullsized":"2024-04-06T12:28:07.317327+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:06:46.449407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.770547+0000","last_change":"2024-04-06T12:28:05.698830+0000","last_active":"2024-04-06T12:28:06.770547+0000","last_peered":"2024-04-06T12:28:06.770547+0000","last_clean":"2024-04-06T12:28:06.770547+0000","last_became_active":"2024-04-06T12:28:05.698509+0000","last_became_peered":"2024-04-06T12:28:05.698509+0000","last_unstale":"2024-04-06T12:28:06.770547+0000","last_undegraded":"2024-04-06T12:28:06.770547+0000","last_fullsized":"2024-04-06T12:28:06.770547+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:40:58.915981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.699172+0000","last_change":"2024-04-06T12:28:05.697459+0000","last_active":"2024-04-06T12:28:05.699172+0000","last_peered":"2024-04-06T12:28:05.699172+0000","last_clean":"2024-04-06T12:28:05.699172+0000","last_became_active":"2024-04-06T12:28:05.697204+0000","last_became_peered":"2024-04-06T12:28:05.697204+0000","last_unstale":"2024-04-06T12:28:05.699172+0000","last_undegraded":"2024-04-06T12:28:05.699172+0000","last_fullsized":"2024-04-06T12:28:05.699172+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:39:26.704304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553130+0000","last_change":"2024-04-06T12:28:05.693661+0000","last_active":"2024-04-06T12:28:07.553130+0000","last_peered":"2024-04-06T12:28:07.553130+0000","last_clean":"2024-04-06T12:28:07.553130+0000","last_became_active":"2024-04-06T12:28:05.693416+0000","last_became_peered":"2024-04-06T12:28:05.693416+0000","last_unstale":"2024-04-06T12:28:07.553130+0000","last_undegraded":"2024-04-06T12:28:07.553130+0000","last_fullsized":"2024-04-06T12:28:07.553130+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:45:05.938509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679084+0000","last_change":"2024-04-06T12:28:05.701612+0000","last_active":"2024-04-06T12:28:06.679084+0000","last_peered":"2024-04-06T12:28:06.679084+0000","last_clean":"2024-04-06T12:28:06.679084+0000","last_became_active":"2024-04-06T12:28:05.700750+0000","last_became_peered":"2024-04-06T12:28:05.700750+0000","last_unstale":"2024-04-06T12:28:06.679084+0000","last_undegraded":"2024-04-06T12:28:06.679084+0000","last_fullsized":"2024-04-06T12:28:06.679084+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:27:50.018142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.696254+0000","last_change":"2024-04-06T12:28:05.690758+0000","last_active":"2024-04-06T12:28:05.696254+0000","last_peered":"2024-04-06T12:28:05.696254+0000","last_clean":"2024-04-06T12:28:05.696254+0000","last_became_active":"2024-04-06T12:28:05.690110+0000","last_became_peered":"2024-04-06T12:28:05.690110+0000","last_unstale":"2024-04-06T12:28:05.696254+0000","last_undegraded":"2024-04-06T12:28:05.696254+0000","last_fullsized":"2024-04-06T12:28:05.696254+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:06:04.989226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.552985+0000","last_change":"2024-04-06T12:28:05.694144+0000","last_active":"2024-04-06T12:28:07.552985+0000","last_peered":"2024-04-06T12:28:07.552985+0000","last_clean":"2024-04-06T12:28:07.552985+0000","last_became_active":"2024-04-06T12:28:05.693690+0000","last_became_peered":"2024-04-06T12:28:05.693690+0000","last_unstale":"2024-04-06T12:28:07.552985+0000","last_undegraded":"2024-04-06T12:28:07.552985+0000","last_fullsized":"2024-04-06T12:28:07.552985+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:53:04.368148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.680385+0000","last_change":"2024-04-06T12:28:05.701364+0000","last_active":"2024-04-06T12:28:06.680385+0000","last_peered":"2024-04-06T12:28:06.680385+0000","last_clean":"2024-04-06T12:28:06.680385+0000","last_became_active":"2024-04-06T12:28:05.701072+0000","last_became_peered":"2024-04-06T12:28:05.701072+0000","last_unstale":"2024-04-06T12:28:06.680385+0000","last_undegraded":"2024-04-06T12:28:06.680385+0000","last_fullsized":"2024-04-06T12:28:06.680385+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:52:04.452630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.699043+0000","last_change":"2024-04-06T12:28:05.693611+0000","last_active":"2024-04-06T12:28:05.699043+0000","last_peered":"2024-04-06T12:28:05.699043+0000","last_clean":"2024-04-06T12:28:05.699043+0000","last_became_active":"2024-04-06T12:28:05.693133+0000","last_became_peered":"2024-04-06T12:28:05.693133+0000","last_unstale":"2024-04-06T12:28:05.699043+0000","last_undegraded":"2024-04-06T12:28:05.699043+0000","last_fullsized":"2024-04-06T12:28:05.699043+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:17:03.573001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317258+0000","last_change":"2024-04-06T12:28:05.691929+0000","last_active":"2024-04-06T12:28:07.317258+0000","last_peered":"2024-04-06T12:28:07.317258+0000","last_clean":"2024-04-06T12:28:07.317258+0000","last_became_active":"2024-04-06T12:28:05.691355+0000","last_became_peered":"2024-04-06T12:28:05.691355+0000","last_unstale":"2024-04-06T12:28:07.317258+0000","last_undegraded":"2024-04-06T12:28:07.317258+0000","last_fullsized":"2024-04-06T12:28:07.317258+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:30:39.282974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.706880+0000","last_change":"2024-04-06T12:28:05.702194+0000","last_active":"2024-04-06T12:28:05.706880+0000","last_peered":"2024-04-06T12:28:05.706880+0000","last_clean":"2024-04-06T12:28:05.706880+0000","last_became_active":"2024-04-06T12:28:05.701858+0000","last_became_peered":"2024-04-06T12:28:05.701858+0000","last_unstale":"2024-04-06T12:28:05.706880+0000","last_undegraded":"2024-04-06T12:28:05.706880+0000","last_fullsized":"2024-04-06T12:28:05.706880+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:40:58.915981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553057+0000","last_change":"2024-04-06T12:28:05.704635+0000","last_active":"2024-04-06T12:28:07.553057+0000","last_peered":"2024-04-06T12:28:07.553057+0000","last_clean":"2024-04-06T12:28:07.553057+0000","last_became_active":"2024-04-06T12:28:05.700942+0000","last_became_peered":"2024-04-06T12:28:05.700942+0000","last_unstale":"2024-04-06T12:28:07.553057+0000","last_undegraded":"2024-04-06T12:28:07.553057+0000","last_fullsized":"2024-04-06T12:28:07.553057+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:53:25.733752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679036+0000","last_change":"2024-04-06T12:28:05.701948+0000","last_active":"2024-04-06T12:28:06.679036+0000","last_peered":"2024-04-06T12:28:06.679036+0000","last_clean":"2024-04-06T12:28:06.679036+0000","last_became_active":"2024-04-06T12:28:05.701371+0000","last_became_peered":"2024-04-06T12:28:05.701371+0000","last_unstale":"2024-04-06T12:28:06.679036+0000","last_undegraded":"2024-04-06T12:28:06.679036+0000","last_fullsized":"2024-04-06T12:28:06.679036+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:38:41.832113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317014+0000","last_change":"2024-04-06T12:28:05.695922+0000","last_active":"2024-04-06T12:28:07.317014+0000","last_peered":"2024-04-06T12:28:07.317014+0000","last_clean":"2024-04-06T12:28:07.317014+0000","last_became_active":"2024-04-06T12:28:05.695668+0000","last_became_peered":"2024-04-06T12:28:05.695668+0000","last_unstale":"2024-04-06T12:28:07.317014+0000","last_undegraded":"2024-04-06T12:28:07.317014+0000","last_fullsized":"2024-04-06T12:28:07.317014+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:19:00.647934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679316+0000","last_change":"2024-04-06T12:28:05.692941+0000","last_active":"2024-04-06T12:28:06.679316+0000","last_peered":"2024-04-06T12:28:06.679316+0000","last_clean":"2024-04-06T12:28:06.679316+0000","last_became_active":"2024-04-06T12:28:05.692642+0000","last_became_peered":"2024-04-06T12:28:05.692642+0000","last_unstale":"2024-04-06T12:28:06.679316+0000","last_undegraded":"2024-04-06T12:28:06.679316+0000","last_fullsized":"2024-04-06T12:28:06.679316+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:35:21.328512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553173+0000","last_change":"2024-04-06T12:28:05.699666+0000","last_active":"2024-04-06T12:28:07.553173+0000","last_peered":"2024-04-06T12:28:07.553173+0000","last_clean":"2024-04-06T12:28:07.553173+0000","last_became_active":"2024-04-06T12:28:05.698732+0000","last_became_peered":"2024-04-06T12:28:05.698732+0000","last_unstale":"2024-04-06T12:28:07.553173+0000","last_undegraded":"2024-04-06T12:28:07.553173+0000","last_fullsized":"2024-04-06T12:28:07.553173+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:25:10.228465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.770690+0000","last_change":"2024-04-06T12:28:05.695478+0000","last_active":"2024-04-06T12:28:06.770690+0000","last_peered":"2024-04-06T12:28:06.770690+0000","last_clean":"2024-04-06T12:28:06.770690+0000","last_became_active":"2024-04-06T12:28:05.695232+0000","last_became_peered":"2024-04-06T12:28:05.695232+0000","last_unstale":"2024-04-06T12:28:06.770690+0000","last_undegraded":"2024-04-06T12:28:06.770690+0000","last_fullsized":"2024-04-06T12:28:06.770690+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:33:04.371728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.678868+0000","last_change":"2024-04-06T12:28:05.693475+0000","last_active":"2024-04-06T12:28:06.678868+0000","last_peered":"2024-04-06T12:28:06.678868+0000","last_clean":"2024-04-06T12:28:06.678868+0000","last_became_active":"2024-04-06T12:28:05.693181+0000","last_became_peered":"2024-04-06T12:28:05.693181+0000","last_unstale":"2024-04-06T12:28:06.678868+0000","last_undegraded":"2024-04-06T12:28:06.678868+0000","last_fullsized":"2024-04-06T12:28:06.678868+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:00:06.871498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553087+0000","last_change":"2024-04-06T12:28:05.700148+0000","last_active":"2024-04-06T12:28:07.553087+0000","last_peered":"2024-04-06T12:28:07.553087+0000","last_clean":"2024-04-06T12:28:07.553087+0000","last_became_active":"2024-04-06T12:28:05.698218+0000","last_became_peered":"2024-04-06T12:28:05.698218+0000","last_unstale":"2024-04-06T12:28:07.553087+0000","last_undegraded":"2024-04-06T12:28:07.553087+0000","last_fullsized":"2024-04-06T12:28:07.553087+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:35:59.792050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679684+0000","last_change":"2024-04-06T12:28:05.696033+0000","last_active":"2024-04-06T12:28:06.679684+0000","last_peered":"2024-04-06T12:28:06.679684+0000","last_clean":"2024-04-06T12:28:06.679684+0000","last_became_active":"2024-04-06T12:28:05.695383+0000","last_became_peered":"2024-04-06T12:28:05.695383+0000","last_unstale":"2024-04-06T12:28:06.679684+0000","last_undegraded":"2024-04-06T12:28:06.679684+0000","last_fullsized":"2024-04-06T12:28:06.679684+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:21:01.116675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553129+0000","last_change":"2024-04-06T12:28:05.704702+0000","last_active":"2024-04-06T12:28:07.553129+0000","last_peered":"2024-04-06T12:28:07.553129+0000","last_clean":"2024-04-06T12:28:07.553129+0000","last_became_active":"2024-04-06T12:28:05.700999+0000","last_became_peered":"2024-04-06T12:28:05.700999+0000","last_unstale":"2024-04-06T12:28:07.553129+0000","last_undegraded":"2024-04-06T12:28:07.553129+0000","last_fullsized":"2024-04-06T12:28:07.553129+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:52:16.330147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679389+0000","last_change":"2024-04-06T12:28:05.694511+0000","last_active":"2024-04-06T12:28:06.679389+0000","last_peered":"2024-04-06T12:28:06.679389+0000","last_clean":"2024-04-06T12:28:06.679389+0000","last_became_active":"2024-04-06T12:28:05.694256+0000","last_became_peered":"2024-04-06T12:28:05.694256+0000","last_unstale":"2024-04-06T12:28:06.679389+0000","last_undegraded":"2024-04-06T12:28:06.679389+0000","last_fullsized":"2024-04-06T12:28:06.679389+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:09:43.251187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.316979+0000","last_change":"2024-04-06T12:28:05.689643+0000","last_active":"2024-04-06T12:28:07.316979+0000","last_peered":"2024-04-06T12:28:07.316979+0000","last_clean":"2024-04-06T12:28:07.316979+0000","last_became_active":"2024-04-06T12:28:05.689380+0000","last_became_peered":"2024-04-06T12:28:05.689380+0000","last_unstale":"2024-04-06T12:28:07.316979+0000","last_undegraded":"2024-04-06T12:28:07.316979+0000","last_fullsized":"2024-04-06T12:28:07.316979+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:04:49.090630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317309+0000","last_change":"2024-04-06T12:28:05.701915+0000","last_active":"2024-04-06T12:28:07.317309+0000","last_peered":"2024-04-06T12:28:07.317309+0000","last_clean":"2024-04-06T12:28:07.317309+0000","last_became_active":"2024-04-06T12:28:05.701658+0000","last_became_peered":"2024-04-06T12:28:05.701658+0000","last_unstale":"2024-04-06T12:28:07.317309+0000","last_undegraded":"2024-04-06T12:28:07.317309+0000","last_fullsized":"2024-04-06T12:28:07.317309+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:41:29.659432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.706861+0000","last_change":"2024-04-06T12:28:05.702195+0000","last_active":"2024-04-06T12:28:05.706861+0000","last_peered":"2024-04-06T12:28:05.706861+0000","last_clean":"2024-04-06T12:28:05.706861+0000","last_became_active":"2024-04-06T12:28:05.701927+0000","last_became_peered":"2024-04-06T12:28:05.701927+0000","last_unstale":"2024-04-06T12:28:05.706861+0000","last_undegraded":"2024-04-06T12:28:05.706861+0000","last_fullsized":"2024-04-06T12:28:05.706861+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:41:14.655391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679838+0000","last_change":"2024-04-06T12:28:05.688824+0000","last_active":"2024-04-06T12:28:06.679838+0000","last_peered":"2024-04-06T12:28:06.679838+0000","last_clean":"2024-04-06T12:28:06.679838+0000","last_became_active":"2024-04-06T12:28:05.688538+0000","last_became_peered":"2024-04-06T12:28:05.688538+0000","last_unstale":"2024-04-06T12:28:06.679838+0000","last_undegraded":"2024-04-06T12:28:06.679838+0000","last_fullsized":"2024-04-06T12:28:06.679838+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:18:59.687118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553124+0000","last_change":"2024-04-06T12:28:05.700151+0000","last_active":"2024-04-06T12:28:07.553124+0000","last_peered":"2024-04-06T12:28:07.553124+0000","last_clean":"2024-04-06T12:28:07.553124+0000","last_became_active":"2024-04-06T12:28:05.699858+0000","last_became_peered":"2024-04-06T12:28:05.699858+0000","last_unstale":"2024-04-06T12:28:07.553124+0000","last_undegraded":"2024-04-06T12:28:07.553124+0000","last_fullsized":"2024-04-06T12:28:07.553124+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:56:29.371271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317016+0000","last_change":"2024-04-06T12:28:05.689846+0000","last_active":"2024-04-06T12:28:07.317016+0000","last_peered":"2024-04-06T12:28:07.317016+0000","last_clean":"2024-04-06T12:28:07.317016+0000","last_became_active":"2024-04-06T12:28:05.689545+0000","last_became_peered":"2024-04-06T12:28:05.689545+0000","last_unstale":"2024-04-06T12:28:07.317016+0000","last_undegraded":"2024-04-06T12:28:07.317016+0000","last_fullsized":"2024-04-06T12:28:07.317016+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:56:45.841002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679597+0000","last_change":"2024-04-06T12:28:05.696142+0000","last_active":"2024-04-06T12:28:06.679597+0000","last_peered":"2024-04-06T12:28:06.679597+0000","last_clean":"2024-04-06T12:28:06.679597+0000","last_became_active":"2024-04-06T12:28:05.695854+0000","last_became_peered":"2024-04-06T12:28:05.695854+0000","last_unstale":"2024-04-06T12:28:06.679597+0000","last_undegraded":"2024-04-06T12:28:06.679597+0000","last_fullsized":"2024-04-06T12:28:06.679597+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:45:17.012967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.770344+0000","last_change":"2024-04-06T12:28:05.692161+0000","last_active":"2024-04-06T12:28:06.770344+0000","last_peered":"2024-04-06T12:28:06.770344+0000","last_clean":"2024-04-06T12:28:06.770344+0000","last_became_active":"2024-04-06T12:28:05.691907+0000","last_became_peered":"2024-04-06T12:28:05.691907+0000","last_unstale":"2024-04-06T12:28:06.770344+0000","last_undegraded":"2024-04-06T12:28:06.770344+0000","last_fullsized":"2024-04-06T12:28:06.770344+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:21:01.116675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679392+0000","last_change":"2024-04-06T12:28:05.692715+0000","last_active":"2024-04-06T12:28:06.679392+0000","last_peered":"2024-04-06T12:28:06.679392+0000","last_clean":"2024-04-06T12:28:06.679392+0000","last_became_active":"2024-04-06T12:28:05.692307+0000","last_became_peered":"2024-04-06T12:28:05.692307+0000","last_unstale":"2024-04-06T12:28:06.679392+0000","last_undegraded":"2024-04-06T12:28:06.679392+0000","last_fullsized":"2024-04-06T12:28:06.679392+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:25:11.631130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.770353+0000","last_change":"2024-04-06T12:28:05.690158+0000","last_active":"2024-04-06T12:28:06.770353+0000","last_peered":"2024-04-06T12:28:06.770353+0000","last_clean":"2024-04-06T12:28:06.770353+0000","last_became_active":"2024-04-06T12:28:05.689912+0000","last_became_peered":"2024-04-06T12:28:05.689912+0000","last_unstale":"2024-04-06T12:28:06.770353+0000","last_undegraded":"2024-04-06T12:28:06.770353+0000","last_fullsized":"2024-04-06T12:28:06.770353+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:17:03.573001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"97'4","reported_seq":15,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553163+0000","last_change":"2024-04-06T12:28:05.271413+0000","last_active":"2024-04-06T12:28:07.553163+0000","last_peered":"2024-04-06T12:28:07.553163+0000","last_clean":"2024-04-06T12:28:07.553163+0000","last_became_active":"2024-04-06T12:28:05.270920+0000","last_became_peered":"2024-04-06T12:28:05.270920+0000","last_unstale":"2024-04-06T12:28:07.553163+0000","last_undegraded":"2024-04-06T12:28:07.553163+0000","last_fullsized":"2024-04-06T12:28:07.553163+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:29:59.694114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.316981+0000","last_change":"2024-04-06T12:28:05.690616+0000","last_active":"2024-04-06T12:28:07.316981+0000","last_peered":"2024-04-06T12:28:07.316981+0000","last_clean":"2024-04-06T12:28:07.316981+0000","last_became_active":"2024-04-06T12:28:05.690327+0000","last_became_peered":"2024-04-06T12:28:05.690327+0000","last_unstale":"2024-04-06T12:28:07.316981+0000","last_undegraded":"2024-04-06T12:28:07.316981+0000","last_fullsized":"2024-04-06T12:28:07.316981+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:38:41.832113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"97'1","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.680310+0000","last_change":"2024-04-06T12:28:05.267505+0000","last_active":"2024-04-06T12:28:06.680310+0000","last_peered":"2024-04-06T12:28:06.680310+0000","last_clean":"2024-04-06T12:28:06.680310+0000","last_became_active":"2024-04-06T12:28:05.267224+0000","last_became_peered":"2024-04-06T12:28:05.267224+0000","last_unstale":"2024-04-06T12:28:06.680310+0000","last_undegraded":"2024-04-06T12:28:06.680310+0000","last_fullsized":"2024-04-06T12:28:06.680310+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:39:25.698878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.680156+0000","last_change":"2024-04-06T12:28:05.701166+0000","last_active":"2024-04-06T12:28:06.680156+0000","last_peered":"2024-04-06T12:28:06.680156+0000","last_clean":"2024-04-06T12:28:06.680156+0000","last_became_active":"2024-04-06T12:28:05.700899+0000","last_became_peered":"2024-04-06T12:28:05.700899+0000","last_unstale":"2024-04-06T12:28:06.680156+0000","last_undegraded":"2024-04-06T12:28:06.680156+0000","last_fullsized":"2024-04-06T12:28:06.680156+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:01:21.360236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"97'2","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.680041+0000","last_change":"2024-04-06T12:28:05.282017+0000","last_active":"2024-04-06T12:28:06.680041+0000","last_peered":"2024-04-06T12:28:06.680041+0000","last_clean":"2024-04-06T12:28:06.680041+0000","last_became_active":"2024-04-06T12:28:05.281774+0000","last_became_peered":"2024-04-06T12:28:05.281774+0000","last_unstale":"2024-04-06T12:28:06.680041+0000","last_undegraded":"2024-04-06T12:28:06.680041+0000","last_fullsized":"2024-04-06T12:28:06.680041+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:33:03.366303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679786+0000","last_change":"2024-04-06T12:28:05.696197+0000","last_active":"2024-04-06T12:28:06.679786+0000","last_peered":"2024-04-06T12:28:06.679786+0000","last_clean":"2024-04-06T12:28:06.679786+0000","last_became_active":"2024-04-06T12:28:05.695772+0000","last_became_peered":"2024-04-06T12:28:05.695772+0000","last_unstale":"2024-04-06T12:28:06.679786+0000","last_undegraded":"2024-04-06T12:28:06.679786+0000","last_fullsized":"2024-04-06T12:28:06.679786+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:27:56.179062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"97'2","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.693681+0000","last_change":"2024-04-06T12:28:05.270215+0000","last_active":"2024-04-06T12:28:05.693681+0000","last_peered":"2024-04-06T12:28:05.693681+0000","last_clean":"2024-04-06T12:28:05.693681+0000","last_became_active":"2024-04-06T12:28:05.269744+0000","last_became_peered":"2024-04-06T12:28:05.269744+0000","last_unstale":"2024-04-06T12:28:05.693681+0000","last_undegraded":"2024-04-06T12:28:05.693681+0000","last_fullsized":"2024-04-06T12:28:05.693681+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:45:04.933084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317132+0000","last_change":"2024-04-06T12:28:05.689014+0000","last_active":"2024-04-06T12:28:07.317132+0000","last_peered":"2024-04-06T12:28:07.317132+0000","last_clean":"2024-04-06T12:28:07.317132+0000","last_became_active":"2024-04-06T12:28:05.688757+0000","last_became_peered":"2024-04-06T12:28:05.688757+0000","last_unstale":"2024-04-06T12:28:07.317132+0000","last_undegraded":"2024-04-06T12:28:07.317132+0000","last_fullsized":"2024-04-06T12:28:07.317132+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:02:58.770663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"97'1","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679885+0000","last_change":"2024-04-06T12:28:05.271076+0000","last_active":"2024-04-06T12:28:06.679885+0000","last_peered":"2024-04-06T12:28:06.679885+0000","last_clean":"2024-04-06T12:28:06.679885+0000","last_became_active":"2024-04-06T12:28:05.270832+0000","last_became_peered":"2024-04-06T12:28:05.270832+0000","last_unstale":"2024-04-06T12:28:06.679885+0000","last_undegraded":"2024-04-06T12:28:06.679885+0000","last_fullsized":"2024-04-06T12:28:06.679885+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:54:21.568200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317209+0000","last_change":"2024-04-06T12:28:05.691996+0000","last_active":"2024-04-06T12:28:07.317209+0000","last_peered":"2024-04-06T12:28:07.317209+0000","last_clean":"2024-04-06T12:28:07.317209+0000","last_became_active":"2024-04-06T12:28:05.691690+0000","last_became_peered":"2024-04-06T12:28:05.691690+0000","last_unstale":"2024-04-06T12:28:07.317209+0000","last_undegraded":"2024-04-06T12:28:07.317209+0000","last_fullsized":"2024-04-06T12:28:07.317209+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:47:25.008418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"97'1","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.694542+0000","last_change":"2024-04-06T12:28:05.274217+0000","last_active":"2024-04-06T12:28:05.694542+0000","last_peered":"2024-04-06T12:28:05.694542+0000","last_clean":"2024-04-06T12:28:05.694542+0000","last_became_active":"2024-04-06T12:28:05.273941+0000","last_became_peered":"2024-04-06T12:28:05.273941+0000","last_unstale":"2024-04-06T12:28:05.694542+0000","last_undegraded":"2024-04-06T12:28:05.694542+0000","last_fullsized":"2024-04-06T12:28:05.694542+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:06:03.983801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679797+0000","last_change":"2024-04-06T12:28:05.701755+0000","last_active":"2024-04-06T12:28:06.679797+0000","last_peered":"2024-04-06T12:28:06.679797+0000","last_clean":"2024-04-06T12:28:06.679797+0000","last_became_active":"2024-04-06T12:28:05.701410+0000","last_became_peered":"2024-04-06T12:28:05.701410+0000","last_unstale":"2024-04-06T12:28:06.679797+0000","last_undegraded":"2024-04-06T12:28:06.679797+0000","last_fullsized":"2024-04-06T12:28:06.679797+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:31:06.373662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317095+0000","last_change":"2024-04-06T12:28:05.281567+0000","last_active":"2024-04-06T12:28:07.317095+0000","last_peered":"2024-04-06T12:28:07.317095+0000","last_clean":"2024-04-06T12:28:07.317095+0000","last_became_active":"2024-04-06T12:28:05.281115+0000","last_became_peered":"2024-04-06T12:28:05.281115+0000","last_unstale":"2024-04-06T12:28:07.317095+0000","last_undegraded":"2024-04-06T12:28:07.317095+0000","last_fullsized":"2024-04-06T12:28:07.317095+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:30:05.507128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.700108+0000","last_change":"2024-04-06T12:28:05.697209+0000","last_active":"2024-04-06T12:28:05.700108+0000","last_peered":"2024-04-06T12:28:05.700108+0000","last_clean":"2024-04-06T12:28:05.700108+0000","last_became_active":"2024-04-06T12:28:05.696917+0000","last_became_peered":"2024-04-06T12:28:05.696917+0000","last_unstale":"2024-04-06T12:28:05.700108+0000","last_undegraded":"2024-04-06T12:28:05.700108+0000","last_fullsized":"2024-04-06T12:28:05.700108+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:03:59.005178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.552944+0000","last_change":"2024-04-06T12:28:05.271913+0000","last_active":"2024-04-06T12:28:07.552944+0000","last_peered":"2024-04-06T12:28:07.552944+0000","last_clean":"2024-04-06T12:28:07.552944+0000","last_became_active":"2024-04-06T12:28:05.271654+0000","last_became_peered":"2024-04-06T12:28:05.271654+0000","last_unstale":"2024-04-06T12:28:07.552944+0000","last_undegraded":"2024-04-06T12:28:07.552944+0000","last_fullsized":"2024-04-06T12:28:07.552944+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:44:14.491419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.316959+0000","last_change":"2024-04-06T12:28:05.683822+0000","last_active":"2024-04-06T12:28:07.316959+0000","last_peered":"2024-04-06T12:28:07.316959+0000","last_clean":"2024-04-06T12:28:07.316959+0000","last_became_active":"2024-04-06T12:28:05.683561+0000","last_became_peered":"2024-04-06T12:28:05.683561+0000","last_unstale":"2024-04-06T12:28:07.316959+0000","last_undegraded":"2024-04-06T12:28:07.316959+0000","last_fullsized":"2024-04-06T12:28:07.316959+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:29:29.562684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.680207+0000","last_change":"2024-04-06T12:28:05.272126+0000","last_active":"2024-04-06T12:28:06.680207+0000","last_peered":"2024-04-06T12:28:06.680207+0000","last_clean":"2024-04-06T12:28:06.680207+0000","last_became_active":"2024-04-06T12:28:05.271842+0000","last_became_peered":"2024-04-06T12:28:05.271842+0000","last_unstale":"2024-04-06T12:28:06.680207+0000","last_undegraded":"2024-04-06T12:28:06.680207+0000","last_fullsized":"2024-04-06T12:28:06.680207+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:52:03.447204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.702062+0000","last_change":"2024-04-06T12:28:05.700058+0000","last_active":"2024-04-06T12:28:05.702062+0000","last_peered":"2024-04-06T12:28:05.702062+0000","last_clean":"2024-04-06T12:28:05.702062+0000","last_became_active":"2024-04-06T12:28:05.699750+0000","last_became_peered":"2024-04-06T12:28:05.699750+0000","last_unstale":"2024-04-06T12:28:05.702062+0000","last_undegraded":"2024-04-06T12:28:05.702062+0000","last_fullsized":"2024-04-06T12:28:05.702062+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:33:04.371728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.682545+0000","last_change":"2024-04-06T12:28:05.274311+0000","last_active":"2024-04-06T12:28:05.682545+0000","last_peered":"2024-04-06T12:28:05.682545+0000","last_clean":"2024-04-06T12:28:05.682545+0000","last_became_active":"2024-04-06T12:28:05.273912+0000","last_became_peered":"2024-04-06T12:28:05.273912+0000","last_unstale":"2024-04-06T12:28:05.682545+0000","last_undegraded":"2024-04-06T12:28:05.682545+0000","last_fullsized":"2024-04-06T12:28:05.682545+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:47:53.798347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.678960+0000","last_change":"2024-04-06T12:28:05.701766+0000","last_active":"2024-04-06T12:28:06.678960+0000","last_peered":"2024-04-06T12:28:06.678960+0000","last_clean":"2024-04-06T12:28:06.678960+0000","last_became_active":"2024-04-06T12:28:05.701055+0000","last_became_peered":"2024-04-06T12:28:05.701055+0000","last_unstale":"2024-04-06T12:28:06.678960+0000","last_undegraded":"2024-04-06T12:28:06.678960+0000","last_fullsized":"2024-04-06T12:28:06.678960+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:39:43.950966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"97'3","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553340+0000","last_change":"2024-04-06T12:28:05.275722+0000","last_active":"2024-04-06T12:28:07.553340+0000","last_peered":"2024-04-06T12:28:07.553340+0000","last_clean":"2024-04-06T12:28:07.553340+0000","last_became_active":"2024-04-06T12:28:05.275329+0000","last_became_peered":"2024-04-06T12:28:05.275329+0000","last_unstale":"2024-04-06T12:28:07.553340+0000","last_undegraded":"2024-04-06T12:28:07.553340+0000","last_fullsized":"2024-04-06T12:28:07.553340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:50:30.941207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.770481+0000","last_change":"2024-04-06T12:28:05.696619+0000","last_active":"2024-04-06T12:28:06.770481+0000","last_peered":"2024-04-06T12:28:06.770481+0000","last_clean":"2024-04-06T12:28:06.770481+0000","last_became_active":"2024-04-06T12:28:05.695878+0000","last_became_peered":"2024-04-06T12:28:05.695878+0000","last_unstale":"2024-04-06T12:28:06.770481+0000","last_undegraded":"2024-04-06T12:28:06.770481+0000","last_fullsized":"2024-04-06T12:28:06.770481+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:07:50.827777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"97'6","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.707170+0000","last_change":"2024-04-06T12:28:05.271400+0000","last_active":"2024-04-06T12:28:05.707170+0000","last_peered":"2024-04-06T12:28:05.707170+0000","last_clean":"2024-04-06T12:28:05.707170+0000","last_became_active":"2024-04-06T12:28:05.271163+0000","last_became_peered":"2024-04-06T12:28:05.271163+0000","last_unstale":"2024-04-06T12:28:05.707170+0000","last_undegraded":"2024-04-06T12:28:05.707170+0000","last_fullsized":"2024-04-06T12:28:05.707170+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:52:15.324722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.702098+0000","last_change":"2024-04-06T12:28:05.699987+0000","last_active":"2024-04-06T12:28:05.702098+0000","last_peered":"2024-04-06T12:28:05.702098+0000","last_clean":"2024-04-06T12:28:05.702098+0000","last_became_active":"2024-04-06T12:28:05.699377+0000","last_became_peered":"2024-04-06T12:28:05.699377+0000","last_unstale":"2024-04-06T12:28:05.702098+0000","last_undegraded":"2024-04-06T12:28:05.702098+0000","last_fullsized":"2024-04-06T12:28:05.702098+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:07:50.827777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"97'2","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.699431+0000","last_change":"2024-04-06T12:28:05.272962+0000","last_active":"2024-04-06T12:28:05.699431+0000","last_peered":"2024-04-06T12:28:05.699431+0000","last_clean":"2024-04-06T12:28:05.699431+0000","last_became_active":"2024-04-06T12:28:05.272663+0000","last_became_peered":"2024-04-06T12:28:05.272663+0000","last_unstale":"2024-04-06T12:28:05.699431+0000","last_undegraded":"2024-04-06T12:28:05.699431+0000","last_fullsized":"2024-04-06T12:28:05.699431+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:09:42.245762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.701629+0000","last_change":"2024-04-06T12:28:05.698630+0000","last_active":"2024-04-06T12:28:05.701629+0000","last_peered":"2024-04-06T12:28:05.701629+0000","last_clean":"2024-04-06T12:28:05.701629+0000","last_became_active":"2024-04-06T12:28:05.696333+0000","last_became_peered":"2024-04-06T12:28:05.696333+0000","last_unstale":"2024-04-06T12:28:05.701629+0000","last_undegraded":"2024-04-06T12:28:05.701629+0000","last_fullsized":"2024-04-06T12:28:05.701629+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:09:43.251187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679702+0000","last_change":"2024-04-06T12:28:05.275895+0000","last_active":"2024-04-06T12:28:06.679702+0000","last_peered":"2024-04-06T12:28:06.679702+0000","last_clean":"2024-04-06T12:28:06.679702+0000","last_became_active":"2024-04-06T12:28:05.275601+0000","last_became_peered":"2024-04-06T12:28:05.275601+0000","last_unstale":"2024-04-06T12:28:06.679702+0000","last_undegraded":"2024-04-06T12:28:06.679702+0000","last_fullsized":"2024-04-06T12:28:06.679702+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:27:55.173637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.698732+0000","last_change":"2024-04-06T12:28:05.691274+0000","last_active":"2024-04-06T12:28:05.698732+0000","last_peered":"2024-04-06T12:28:05.698732+0000","last_clean":"2024-04-06T12:28:05.698732+0000","last_became_active":"2024-04-06T12:28:05.691000+0000","last_became_peered":"2024-04-06T12:28:05.691000+0000","last_unstale":"2024-04-06T12:28:05.698732+0000","last_undegraded":"2024-04-06T12:28:05.698732+0000","last_fullsized":"2024-04-06T12:28:05.698732+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:31:06.373662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.552947+0000","last_change":"2024-04-06T12:28:05.274168+0000","last_active":"2024-04-06T12:28:07.552947+0000","last_peered":"2024-04-06T12:28:07.552947+0000","last_clean":"2024-04-06T12:28:07.552947+0000","last_became_active":"2024-04-06T12:28:05.273912+0000","last_became_peered":"2024-04-06T12:28:05.273912+0000","last_unstale":"2024-04-06T12:28:07.552947+0000","last_undegraded":"2024-04-06T12:28:07.552947+0000","last_fullsized":"2024-04-06T12:28:07.552947+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:53:46.982328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.699480+0000","last_change":"2024-04-06T12:28:05.694938+0000","last_active":"2024-04-06T12:28:05.699480+0000","last_peered":"2024-04-06T12:28:05.699480+0000","last_clean":"2024-04-06T12:28:05.699480+0000","last_became_active":"2024-04-06T12:28:05.694673+0000","last_became_peered":"2024-04-06T12:28:05.694673+0000","last_unstale":"2024-04-06T12:28:05.699480+0000","last_undegraded":"2024-04-06T12:28:05.699480+0000","last_fullsized":"2024-04-06T12:28:05.699480+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:58:23.827592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"97'8","reported_seq":19,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317101+0000","last_change":"2024-04-06T12:28:05.281308+0000","last_active":"2024-04-06T12:28:07.317101+0000","last_peered":"2024-04-06T12:28:07.317101+0000","last_clean":"2024-04-06T12:28:07.317101+0000","last_became_active":"2024-04-06T12:28:05.279789+0000","last_became_peered":"2024-04-06T12:28:05.279789+0000","last_unstale":"2024-04-06T12:28:07.317101+0000","last_undegraded":"2024-04-06T12:28:07.317101+0000","last_fullsized":"2024-04-06T12:28:07.317101+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:00:21.413158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.678524+0000","last_change":"2024-04-06T12:28:05.693048+0000","last_active":"2024-04-06T12:28:06.678524+0000","last_peered":"2024-04-06T12:28:06.678524+0000","last_clean":"2024-04-06T12:28:06.678524+0000","last_became_active":"2024-04-06T12:28:05.692812+0000","last_became_peered":"2024-04-06T12:28:05.692812+0000","last_unstale":"2024-04-06T12:28:06.678524+0000","last_undegraded":"2024-04-06T12:28:06.678524+0000","last_fullsized":"2024-04-06T12:28:06.678524+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:47:00.031681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"97'1","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.680295+0000","last_change":"2024-04-06T12:28:05.274967+0000","last_active":"2024-04-06T12:28:06.680295+0000","last_peered":"2024-04-06T12:28:06.680295+0000","last_clean":"2024-04-06T12:28:06.680295+0000","last_became_active":"2024-04-06T12:28:05.274706+0000","last_became_peered":"2024-04-06T12:28:05.274706+0000","last_unstale":"2024-04-06T12:28:06.680295+0000","last_undegraded":"2024-04-06T12:28:06.680295+0000","last_fullsized":"2024-04-06T12:28:06.680295+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:01:20.354811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553060+0000","last_change":"2024-04-06T12:28:05.695403+0000","last_active":"2024-04-06T12:28:07.553060+0000","last_peered":"2024-04-06T12:28:07.553060+0000","last_clean":"2024-04-06T12:28:07.553060+0000","last_became_active":"2024-04-06T12:28:05.695149+0000","last_became_peered":"2024-04-06T12:28:05.695149+0000","last_unstale":"2024-04-06T12:28:07.553060+0000","last_undegraded":"2024-04-06T12:28:07.553060+0000","last_fullsized":"2024-04-06T12:28:07.553060+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:28.212844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"97'2","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679986+0000","last_change":"2024-04-06T12:28:05.275346+0000","last_active":"2024-04-06T12:28:06.679986+0000","last_peered":"2024-04-06T12:28:06.679986+0000","last_clean":"2024-04-06T12:28:06.679986+0000","last_became_active":"2024-04-06T12:28:05.275064+0000","last_became_peered":"2024-04-06T12:28:05.275064+0000","last_unstale":"2024-04-06T12:28:06.679986+0000","last_undegraded":"2024-04-06T12:28:06.679986+0000","last_fullsized":"2024-04-06T12:28:06.679986+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:29.967187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553079+0000","last_change":"2024-04-06T12:28:05.700626+0000","last_active":"2024-04-06T12:28:07.553079+0000","last_peered":"2024-04-06T12:28:07.553079+0000","last_clean":"2024-04-06T12:28:07.553079+0000","last_became_active":"2024-04-06T12:28:05.700312+0000","last_became_peered":"2024-04-06T12:28:05.700312+0000","last_unstale":"2024-04-06T12:28:07.553079+0000","last_undegraded":"2024-04-06T12:28:07.553079+0000","last_fullsized":"2024-04-06T12:28:07.553079+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:40:45.532919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"97'1","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553343+0000","last_change":"2024-04-06T12:28:05.272133+0000","last_active":"2024-04-06T12:28:07.553343+0000","last_peered":"2024-04-06T12:28:07.553343+0000","last_clean":"2024-04-06T12:28:07.553343+0000","last_became_active":"2024-04-06T12:28:05.271051+0000","last_became_peered":"2024-04-06T12:28:05.271051+0000","last_unstale":"2024-04-06T12:28:07.553343+0000","last_undegraded":"2024-04-06T12:28:07.553343+0000","last_fullsized":"2024-04-06T12:28:07.553343+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:05.280995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.678761+0000","last_change":"2024-04-06T12:28:05.695403+0000","last_active":"2024-04-06T12:28:06.678761+0000","last_peered":"2024-04-06T12:28:06.678761+0000","last_clean":"2024-04-06T12:28:06.678761+0000","last_became_active":"2024-04-06T12:28:05.695139+0000","last_became_peered":"2024-04-06T12:28:05.695139+0000","last_unstale":"2024-04-06T12:28:06.678761+0000","last_undegraded":"2024-04-06T12:28:06.678761+0000","last_fullsized":"2024-04-06T12:28:06.678761+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:10:38.252973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.770519+0000","last_change":"2024-04-06T12:28:05.274790+0000","last_active":"2024-04-06T12:28:06.770519+0000","last_peered":"2024-04-06T12:28:06.770519+0000","last_clean":"2024-04-06T12:28:06.770519+0000","last_became_active":"2024-04-06T12:28:05.274522+0000","last_became_peered":"2024-04-06T12:28:05.274522+0000","last_unstale":"2024-04-06T12:28:06.770519+0000","last_undegraded":"2024-04-06T12:28:06.770519+0000","last_fullsized":"2024-04-06T12:28:06.770519+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:47:53.798347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317233+0000","last_change":"2024-04-06T12:28:05.689552+0000","last_active":"2024-04-06T12:28:07.317233+0000","last_peered":"2024-04-06T12:28:07.317233+0000","last_clean":"2024-04-06T12:28:07.317233+0000","last_became_active":"2024-04-06T12:28:05.689302+0000","last_became_peered":"2024-04-06T12:28:05.689302+0000","last_unstale":"2024-04-06T12:28:07.317233+0000","last_undegraded":"2024-04-06T12:28:07.317233+0000","last_fullsized":"2024-04-06T12:28:07.317233+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:44:06.415054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"97'2","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.678844+0000","last_change":"2024-04-06T12:28:05.274888+0000","last_active":"2024-04-06T12:28:06.678844+0000","last_peered":"2024-04-06T12:28:06.678844+0000","last_clean":"2024-04-06T12:28:06.678844+0000","last_became_active":"2024-04-06T12:28:05.274629+0000","last_became_peered":"2024-04-06T12:28:05.274629+0000","last_unstale":"2024-04-06T12:28:06.678844+0000","last_undegraded":"2024-04-06T12:28:06.678844+0000","last_fullsized":"2024-04-06T12:28:06.678844+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:03:21.619567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317380+0000","last_change":"2024-04-06T12:28:05.695277+0000","last_active":"2024-04-06T12:28:07.317380+0000","last_peered":"2024-04-06T12:28:07.317380+0000","last_clean":"2024-04-06T12:28:07.317380+0000","last_became_active":"2024-04-06T12:28:05.695002+0000","last_became_peered":"2024-04-06T12:28:05.695002+0000","last_unstale":"2024-04-06T12:28:07.317380+0000","last_undegraded":"2024-04-06T12:28:07.317380+0000","last_fullsized":"2024-04-06T12:28:07.317380+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:23:48.291159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.678659+0000","last_change":"2024-04-06T12:28:05.272012+0000","last_active":"2024-04-06T12:28:06.678659+0000","last_peered":"2024-04-06T12:28:06.678659+0000","last_clean":"2024-04-06T12:28:06.678659+0000","last_became_active":"2024-04-06T12:28:05.271733+0000","last_became_peered":"2024-04-06T12:28:05.271733+0000","last_unstale":"2024-04-06T12:28:06.678659+0000","last_undegraded":"2024-04-06T12:28:06.678659+0000","last_fullsized":"2024-04-06T12:28:06.678659+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:27:20.824420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.695335+0000","last_change":"2024-04-06T12:28:05.688208+0000","last_active":"2024-04-06T12:28:05.695335+0000","last_peered":"2024-04-06T12:28:05.695335+0000","last_clean":"2024-04-06T12:28:05.695335+0000","last_became_active":"2024-04-06T12:28:05.687846+0000","last_became_peered":"2024-04-06T12:28:05.687846+0000","last_unstale":"2024-04-06T12:28:05.695335+0000","last_undegraded":"2024-04-06T12:28:05.695335+0000","last_fullsized":"2024-04-06T12:28:05.695335+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:26:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.678647+0000","last_change":"2024-04-06T12:28:05.272679+0000","last_active":"2024-04-06T12:28:06.678647+0000","last_peered":"2024-04-06T12:28:06.678647+0000","last_clean":"2024-04-06T12:28:06.678647+0000","last_became_active":"2024-04-06T12:28:05.272331+0000","last_became_peered":"2024-04-06T12:28:05.272331+0000","last_unstale":"2024-04-06T12:28:06.678647+0000","last_undegraded":"2024-04-06T12:28:06.678647+0000","last_fullsized":"2024-04-06T12:28:06.678647+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:13:41.048698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679189+0000","last_change":"2024-04-06T12:28:05.693228+0000","last_active":"2024-04-06T12:28:06.679189+0000","last_peered":"2024-04-06T12:28:06.679189+0000","last_clean":"2024-04-06T12:28:06.679189+0000","last_became_active":"2024-04-06T12:28:05.690502+0000","last_became_peered":"2024-04-06T12:28:05.690502+0000","last_unstale":"2024-04-06T12:28:06.679189+0000","last_undegraded":"2024-04-06T12:28:06.679189+0000","last_fullsized":"2024-04-06T12:28:06.679189+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:26:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"97'2","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.700399+0000","last_change":"2024-04-06T12:28:05.275974+0000","last_active":"2024-04-06T12:28:05.700399+0000","last_peered":"2024-04-06T12:28:05.700399+0000","last_clean":"2024-04-06T12:28:05.700399+0000","last_became_active":"2024-04-06T12:28:05.275692+0000","last_became_peered":"2024-04-06T12:28:05.275692+0000","last_unstale":"2024-04-06T12:28:05.700399+0000","last_undegraded":"2024-04-06T12:28:05.700399+0000","last_fullsized":"2024-04-06T12:28:05.700399+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:03:57.999753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.698649+0000","last_change":"2024-04-06T12:28:05.694333+0000","last_active":"2024-04-06T12:28:05.698649+0000","last_peered":"2024-04-06T12:28:05.698649+0000","last_clean":"2024-04-06T12:28:05.698649+0000","last_became_active":"2024-04-06T12:28:05.694053+0000","last_became_peered":"2024-04-06T12:28:05.694053+0000","last_unstale":"2024-04-06T12:28:05.698649+0000","last_undegraded":"2024-04-06T12:28:05.698649+0000","last_fullsized":"2024-04-06T12:28:05.698649+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:01:21.360236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317376+0000","last_change":"2024-04-06T12:28:05.281755+0000","last_active":"2024-04-06T12:28:07.317376+0000","last_peered":"2024-04-06T12:28:07.317376+0000","last_clean":"2024-04-06T12:28:07.317376+0000","last_became_active":"2024-04-06T12:28:05.281346+0000","last_became_peered":"2024-04-06T12:28:05.281346+0000","last_unstale":"2024-04-06T12:28:07.317376+0000","last_undegraded":"2024-04-06T12:28:07.317376+0000","last_fullsized":"2024-04-06T12:28:07.317376+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:13:50.117433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.680256+0000","last_change":"2024-04-06T12:28:05.701311+0000","last_active":"2024-04-06T12:28:06.680256+0000","last_peered":"2024-04-06T12:28:06.680256+0000","last_clean":"2024-04-06T12:28:06.680256+0000","last_became_active":"2024-04-06T12:28:05.700946+0000","last_became_peered":"2024-04-06T12:28:05.700946+0000","last_unstale":"2024-04-06T12:28:06.680256+0000","last_undegraded":"2024-04-06T12:28:06.680256+0000","last_fullsized":"2024-04-06T12:28:06.680256+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:58:23.827592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.680144+0000","last_change":"2024-04-06T12:28:05.270955+0000","last_active":"2024-04-06T12:28:06.680144+0000","last_peered":"2024-04-06T12:28:06.680144+0000","last_clean":"2024-04-06T12:28:06.680144+0000","last_became_active":"2024-04-06T12:28:05.270687+0000","last_became_peered":"2024-04-06T12:28:05.270687+0000","last_unstale":"2024-04-06T12:28:06.680144+0000","last_undegraded":"2024-04-06T12:28:06.680144+0000","last_fullsized":"2024-04-06T12:28:06.680144+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:17:02.567576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317125+0000","last_change":"2024-04-06T12:28:05.684328+0000","last_active":"2024-04-06T12:28:07.317125+0000","last_peered":"2024-04-06T12:28:07.317125+0000","last_clean":"2024-04-06T12:28:07.317125+0000","last_became_active":"2024-04-06T12:28:05.684086+0000","last_became_peered":"2024-04-06T12:28:05.684086+0000","last_unstale":"2024-04-06T12:28:07.317125+0000","last_undegraded":"2024-04-06T12:28:07.317125+0000","last_fullsized":"2024-04-06T12:28:07.317125+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:47:00.031681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679460+0000","last_change":"2024-04-06T12:28:05.272141+0000","last_active":"2024-04-06T12:28:06.679460+0000","last_peered":"2024-04-06T12:28:06.679460+0000","last_clean":"2024-04-06T12:28:06.679460+0000","last_became_active":"2024-04-06T12:28:05.271870+0000","last_became_peered":"2024-04-06T12:28:05.271870+0000","last_unstale":"2024-04-06T12:28:06.679460+0000","last_undegraded":"2024-04-06T12:28:06.679460+0000","last_fullsized":"2024-04-06T12:28:06.679460+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:21:00.111250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.698507+0000","last_change":"2024-04-06T12:28:05.694128+0000","last_active":"2024-04-06T12:28:05.698507+0000","last_peered":"2024-04-06T12:28:05.698507+0000","last_clean":"2024-04-06T12:28:05.698507+0000","last_became_active":"2024-04-06T12:28:05.693888+0000","last_became_peered":"2024-04-06T12:28:05.693888+0000","last_unstale":"2024-04-06T12:28:05.698507+0000","last_undegraded":"2024-04-06T12:28:05.698507+0000","last_fullsized":"2024-04-06T12:28:05.698507+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:27:56.179062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.678486+0000","last_change":"2024-04-06T12:28:05.266184+0000","last_active":"2024-04-06T12:28:05.678486+0000","last_peered":"2024-04-06T12:28:05.678486+0000","last_clean":"2024-04-06T12:28:05.678486+0000","last_became_active":"2024-04-06T12:28:05.265916+0000","last_became_peered":"2024-04-06T12:28:05.265916+0000","last_unstale":"2024-04-06T12:28:05.678486+0000","last_undegraded":"2024-04-06T12:28:05.678486+0000","last_fullsized":"2024-04-06T12:28:05.678486+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:25:09.223040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.695908+0000","last_change":"2024-04-06T12:28:05.690680+0000","last_active":"2024-04-06T12:28:05.695908+0000","last_peered":"2024-04-06T12:28:05.695908+0000","last_clean":"2024-04-06T12:28:05.695908+0000","last_became_active":"2024-04-06T12:28:05.690418+0000","last_became_peered":"2024-04-06T12:28:05.690418+0000","last_unstale":"2024-04-06T12:28:05.695908+0000","last_undegraded":"2024-04-06T12:28:05.695908+0000","last_fullsized":"2024-04-06T12:28:05.695908+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:47:54.803772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553272+0000","last_change":"2024-04-06T12:28:05.274049+0000","last_active":"2024-04-06T12:28:07.553272+0000","last_peered":"2024-04-06T12:28:07.553272+0000","last_clean":"2024-04-06T12:28:07.553272+0000","last_became_active":"2024-04-06T12:28:05.273801+0000","last_became_peered":"2024-04-06T12:28:05.273801+0000","last_unstale":"2024-04-06T12:28:07.553272+0000","last_undegraded":"2024-04-06T12:28:07.553272+0000","last_fullsized":"2024-04-06T12:28:07.553272+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:10.453242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.770653+0000","last_change":"2024-04-06T12:28:05.699523+0000","last_active":"2024-04-06T12:28:06.770653+0000","last_peered":"2024-04-06T12:28:06.770653+0000","last_clean":"2024-04-06T12:28:06.770653+0000","last_became_active":"2024-04-06T12:28:05.699255+0000","last_became_peered":"2024-04-06T12:28:05.699255+0000","last_unstale":"2024-04-06T12:28:06.770653+0000","last_undegraded":"2024-04-06T12:28:06.770653+0000","last_fullsized":"2024-04-06T12:28:06.770653+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:41:14.655391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.678761+0000","last_change":"2024-04-06T12:28:05.274836+0000","last_active":"2024-04-06T12:28:05.678761+0000","last_peered":"2024-04-06T12:28:05.678761+0000","last_clean":"2024-04-06T12:28:05.678761+0000","last_became_active":"2024-04-06T12:28:05.274546+0000","last_became_peered":"2024-04-06T12:28:05.274546+0000","last_unstale":"2024-04-06T12:28:05.678761+0000","last_undegraded":"2024-04-06T12:28:05.678761+0000","last_fullsized":"2024-04-06T12:28:05.678761+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:53:24.728327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.770656+0000","last_change":"2024-04-06T12:28:05.696466+0000","last_active":"2024-04-06T12:28:06.770656+0000","last_peered":"2024-04-06T12:28:06.770656+0000","last_clean":"2024-04-06T12:28:06.770656+0000","last_became_active":"2024-04-06T12:28:05.696189+0000","last_became_peered":"2024-04-06T12:28:05.696189+0000","last_unstale":"2024-04-06T12:28:06.770656+0000","last_undegraded":"2024-04-06T12:28:06.770656+0000","last_fullsized":"2024-04-06T12:28:06.770656+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:39:26.704304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.770323+0000","last_change":"2024-04-06T12:28:05.272062+0000","last_active":"2024-04-06T12:28:06.770323+0000","last_peered":"2024-04-06T12:28:06.770323+0000","last_clean":"2024-04-06T12:28:06.770323+0000","last_became_active":"2024-04-06T12:28:05.271804+0000","last_became_peered":"2024-04-06T12:28:05.271804+0000","last_unstale":"2024-04-06T12:28:06.770323+0000","last_undegraded":"2024-04-06T12:28:06.770323+0000","last_fullsized":"2024-04-06T12:28:06.770323+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:06:03.983801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679940+0000","last_change":"2024-04-06T12:28:05.697981+0000","last_active":"2024-04-06T12:28:06.679940+0000","last_peered":"2024-04-06T12:28:06.679940+0000","last_clean":"2024-04-06T12:28:06.679940+0000","last_became_active":"2024-04-06T12:28:05.697684+0000","last_became_peered":"2024-04-06T12:28:05.697684+0000","last_unstale":"2024-04-06T12:28:06.679940+0000","last_undegraded":"2024-04-06T12:28:06.679940+0000","last_fullsized":"2024-04-06T12:28:06.679940+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:14.483577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679044+0000","last_change":"2024-04-06T12:28:05.276729+0000","last_active":"2024-04-06T12:28:06.679044+0000","last_peered":"2024-04-06T12:28:06.679044+0000","last_clean":"2024-04-06T12:28:06.679044+0000","last_became_active":"2024-04-06T12:28:05.274128+0000","last_became_peered":"2024-04-06T12:28:05.274128+0000","last_unstale":"2024-04-06T12:28:06.679044+0000","last_undegraded":"2024-04-06T12:28:06.679044+0000","last_fullsized":"2024-04-06T12:28:06.679044+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:23:35.861872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"87'172","reported_seq":445,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679455+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:28:06.679455+0000","last_peered":"2024-04-06T12:28:06.679455+0000","last_clean":"2024-04-06T12:28:06.679455+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:28:06.679455+0000","last_undegraded":"2024-04-06T12:28:06.679455+0000","last_fullsized":"2024-04-06T12:28:06.679455+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172,"ondisk_log_size":172,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575042,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59124,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312716,"statfs":{"total":96636764160,"available":96576221184,"internally_reserved":0,"allocated":1798144,"data_stored":1186132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.34300000000000003,"5min":0.219,"15min":0.219},"max":{"1min":0.68100000000000005,"5min":1.143,"15min":1.143},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54300000000000004,"15min":0.55200000000000005},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.61499999999999999,"5min":0.77100000000000002,"15min":0.78200000000000003},"last":0.372}]},{"osd":1,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.63600000000000001,"5min":0.77100000000000002,"15min":0.81399999999999995},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.55500000000000005,"15min":0.56899999999999995},"min":{"1min":0.36599999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68899999999999995,"5min":0.79400000000000004,"15min":0.79600000000000004},"last":0.56299999999999994}]},{"osd":2,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55200000000000005,"15min":0.55400000000000005},"min":{"1min":0.316,"5min":0.249,"15min":0.249},"max":{"1min":0.65500000000000003,"5min":0.78500000000000003,"15min":0.83299999999999996},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55900000000000005,"15min":0.57799999999999996},"min":{"1min":0.36699999999999999,"5min":0.34499999999999997,"15min":0.24199999999999999},"max":{"1min":0.66500000000000004,"5min":0.77600000000000002,"15min":0.82299999999999995},"last":0.39400000000000002}]},{"osd":3,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.57099999999999995,"15min":0.57199999999999995},"min":{"1min":0.39300000000000002,"5min":0.34799999999999998,"15min":0.34399999999999997},"max":{"1min":0.72399999999999998,"5min":1.357,"15min":1.357},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.57899999999999996,"15min":0.59199999999999997},"min":{"1min":0.38500000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.65900000000000003,"5min":0.81799999999999995,"15min":0.92000000000000004},"last":0.42899999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.29999999999999999,"5min":0.27100000000000002,"15min":0.25},"max":{"1min":0.63900000000000001,"5min":0.70399999999999996,"15min":0.83599999999999997},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.41199999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.66500000000000004,"5min":0.77700000000000002,"15min":1.0780000000000001},"last":0.55100000000000005}]},{"osd":5,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.48399999999999999,"15min":0.502},"min":{"1min":0.34300000000000003,"5min":0.29199999999999998,"15min":0.23799999999999999},"max":{"1min":0.60499999999999998,"5min":0.80100000000000005,"15min":0.98199999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54300000000000004,"15min":0.55800000000000005},"min":{"1min":0.39900000000000002,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.625,"5min":0.81599999999999995,"15min":0.93600000000000005},"last":0.61599999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.52400000000000002,"15min":0.54100000000000004},"min":{"1min":0.37,"5min":0.33700000000000002,"15min":0.28799999999999998},"max":{"1min":0.67700000000000005,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54900000000000004,"15min":0.56200000000000006},"min":{"1min":0.373,"5min":0.30399999999999999,"15min":0.26900000000000002},"max":{"1min":0.68500000000000005,"5min":0.79700000000000004,"15min":0.88500000000000001},"last":0.58599999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575041,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1196032,"data_stored":589667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.36699999999999999,"5min":0.26200000000000001,"15min":0.23000000000000001},"max":{"1min":0.77500000000000002,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52500000000000002,"15min":0.53300000000000003},"min":{"1min":0.374,"5min":0.30199999999999999,"15min":0.245},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.60599999999999998}]},{"osd":1,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.32700000000000001,"5min":0.32600000000000001,"15min":0.24399999999999999},"max":{"1min":0.78200000000000003,"5min":0.78700000000000003,"15min":0.84999999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.38400000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.83099999999999996},"last":0.56799999999999995}]},{"osd":2,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.39100000000000001,"5min":0.30499999999999999,"15min":0.29899999999999999},"max":{"1min":0.749,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.68999999999999995}]},{"osd":3,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.56899999999999995,"15min":0.58099999999999996},"min":{"1min":0.35099999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.745,"5min":0.79000000000000004,"15min":0.81200000000000006},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.60899999999999999,"15min":0.61899999999999999},"min":{"1min":0.51700000000000002,"5min":0.41099999999999998,"15min":0.374},"max":{"1min":0.79200000000000004,"5min":0.82799999999999996,"15min":0.872},"last":0.77600000000000002}]},{"osd":4,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.49099999999999999,"15min":0.496},"min":{"1min":0.28599999999999998,"5min":0.27500000000000002,"15min":0.27300000000000002},"max":{"1min":0.747,"5min":0.749,"15min":0.749},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.53000000000000003},"min":{"1min":0.34899999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.73199999999999998,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.70799999999999996}]},{"osd":5,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.47999999999999998,"15min":0.49199999999999999},"min":{"1min":0.34699999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71099999999999997,"5min":0.755,"15min":0.75800000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.50600000000000001,"15min":0.51300000000000001},"min":{"1min":0.32800000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72599999999999998,"5min":0.83699999999999997,"15min":0.92800000000000005},"last":0.80200000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53000000000000003,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70099999999999996,"5min":0.79800000000000004,"15min":0.81000000000000005},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55800000000000005,"15min":0.56100000000000005},"min":{"1min":0.41899999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80800000000000005,"5min":0.88900000000000001,"15min":0.96099999999999997},"last":0.84799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575040,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32304,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339536,"statfs":{"total":96636764160,"available":96603684864,"internally_reserved":0,"allocated":1204224,"data_stored":594835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.377,"15min":0.36199999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.47699999999999998,"5min":0.58999999999999997,"15min":0.71699999999999997},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39900000000000002,"15min":0.39600000000000002},"min":{"1min":0.253,"5min":0.215,"15min":0.215},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.316}]},{"osd":2,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.40500000000000003,"15min":0.39600000000000002},"min":{"1min":0.27200000000000002,"5min":0.23899999999999999,"15min":0.19400000000000001},"max":{"1min":0.57599999999999996,"5min":0.624,"15min":0.67600000000000005},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.432,"15min":0.42999999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.249},"max":{"1min":0.60699999999999998,"5min":0.63600000000000001,"15min":0.79000000000000004},"last":0.437}]},{"osd":3,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45600000000000002,"15min":0.44700000000000001},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67400000000000004,"5min":0.67800000000000005,"15min":0.84299999999999997},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.435,"5min":0.42599999999999999,"15min":0.41899999999999998},"min":{"1min":0.30499999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.56499999999999995,"5min":0.69599999999999995,"15min":0.86299999999999999},"last":0.38400000000000001}]},{"osd":4,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.623,"15min":0.63200000000000001},"min":{"1min":0.42599999999999999,"5min":0.40100000000000002,"15min":0.34799999999999998},"max":{"1min":0.71499999999999997,"5min":0.86799999999999999,"15min":6.3739999999999997},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.63100000000000001,"15min":0.622},"min":{"1min":0.45400000000000001,"5min":0.441,"15min":0.30399999999999999},"max":{"1min":0.84299999999999997,"5min":0.91400000000000003,"15min":1.0269999999999999},"last":0.70999999999999996}]},{"osd":5,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.624,"15min":0.61399999999999999},"min":{"1min":0.47599999999999998,"5min":0.42999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83999999999999997,"5min":0.91300000000000003,"15min":0.97699999999999998},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.628},"min":{"1min":0.442,"5min":0.43099999999999999,"15min":0.35099999999999998},"max":{"1min":0.78700000000000003,"5min":0.999,"15min":1.0089999999999999},"last":0.61699999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.623,"15min":0.61399999999999999},"min":{"1min":0.47699999999999998,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":0.76500000000000001,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.623,"5min":0.65300000000000002,"15min":0.64900000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.41799999999999998},"max":{"1min":0.83999999999999997,"5min":0.88700000000000001,"15min":1.151},"last":0.59299999999999997}]},{"osd":7,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.64800000000000002,"15min":0.65500000000000003},"min":{"1min":0.44600000000000001,"5min":0.41899999999999998,"15min":0.38800000000000001},"max":{"1min":0.81699999999999995,"5min":1.0329999999999999,"15min":6.3639999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.47299999999999998,"5min":0.40200000000000002,"15min":0.39300000000000002},"max":{"1min":0.82499999999999996,"5min":0.95599999999999996,"15min":1.073},"last":0.75700000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575040,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32864,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338976,"statfs":{"total":96636764160,"available":96603111424,"internally_reserved":0,"allocated":1789952,"data_stored":1185091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 12:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.38200000000000001,"15min":0.38100000000000001},"min":{"1min":0.222,"5min":0.19,"15min":0.19},"max":{"1min":0.59899999999999998,"5min":0.80000000000000004,"15min":0.89100000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.40400000000000003,"15min":0.40300000000000002},"min":{"1min":0.25700000000000001,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":0.57299999999999995,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.40600000000000003}]},{"osd":2,"last update":"Sat Apr 6 12:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40699999999999997,"15min":0.41099999999999998},"min":{"1min":0.255,"5min":0.23599999999999999,"15min":0.19700000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":1.8089999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.30599999999999999,"5min":0.249,"15min":0.23200000000000001},"max":{"1min":0.58299999999999996,"5min":1.4790000000000001,"15min":1.764},"last":0.372}]},{"osd":3,"last update":"Sat Apr 6 12:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.41499999999999998,"15min":0.41399999999999998},"min":{"1min":0.29099999999999998,"5min":0.252,"15min":0.21299999999999999},"max":{"1min":0.624,"5min":1.25,"15min":1.25},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45900000000000002,"15min":0.46200000000000002},"min":{"1min":0.34200000000000003,"5min":0.28499999999999998,"15min":0.249},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.92000000000000004},"last":0.433}]},{"osd":4,"last update":"Sat Apr 6 12:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.498,"5min":0.371,"15min":0.35699999999999998},"max":{"1min":0.73199999999999998,"5min":1.514,"15min":1.514},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.621,"15min":0.627},"min":{"1min":0.42099999999999999,"5min":0.39900000000000002,"15min":0.311},"max":{"1min":0.73799999999999999,"5min":1.202,"15min":1.202},"last":0.625}]},{"osd":5,"last update":"Sat Apr 6 12:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.621,"15min":0.622},"min":{"1min":0.38900000000000001,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.78900000000000003,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63900000000000001,"15min":0.64800000000000002},"min":{"1min":0.495,"5min":0.38800000000000001,"15min":0.315},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.53800000000000003}]},{"osd":6,"last update":"Sat Apr 6 12:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.64500000000000002,"15min":0.65100000000000002},"min":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.378},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.65400000000000003,"15min":0.66200000000000003},"min":{"1min":0.46500000000000002,"5min":0.437,"15min":0.38100000000000001},"max":{"1min":0.77500000000000002,"5min":1.613,"15min":1.613},"last":0.48899999999999999}]},{"osd":7,"last update":"Sat Apr 6 12:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.66000000000000003,"15min":0.66300000000000003},"min":{"1min":0.45300000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.78600000000000003,"5min":1.54,"15min":1.54},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.65600000000000003,"15min":0.65500000000000003},"min":{"1min":0.42199999999999999,"5min":0.38400000000000001,"15min":0.36699999999999999},"max":{"1min":0.72899999999999998,"5min":1.498,"15min":1.498},"last":0.57899999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575042,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50344,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321496,"statfs":{"total":96636764160,"available":96585211904,"internally_reserved":0,"allocated":1208320,"data_stored":595465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:27 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38600000000000001,"15min":0.38200000000000001},"min":{"1min":0.26800000000000002,"5min":0.214,"15min":0.21099999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":1.159},"last":0.441},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40799999999999997,"15min":0.40999999999999998},"min":{"1min":0.26000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":1.075},"last":0.48699999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:27:27 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.28000000000000003,"5min":0.26400000000000001,"15min":0.24299999999999999},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":3.6179999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42499999999999999,"15min":0.43099999999999999},"min":{"1min":0.29399999999999998,"5min":0.27100000000000002,"15min":0.23599999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":1.125},"last":0.32900000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:27:27 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.45900000000000002},"min":{"1min":0.316,"5min":0.30499999999999999,"15min":0.29699999999999999},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":3.661},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.438,"5min":0.44700000000000001,"15min":0.438},"min":{"1min":0.313,"5min":0.29499999999999998,"15min":0.27600000000000002},"max":{"1min":0.97099999999999997,"5min":1.0549999999999999,"15min":1.1910000000000001},"last":0.34999999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:27:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60599999999999998,"15min":0.61199999999999999},"min":{"1min":0.38600000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.2170000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61899999999999999,"15min":0.626},"min":{"1min":0.375,"5min":0.316,"15min":0.30499999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":1.101},"last":0.65100000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:27:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.64400000000000002,"15min":0.64600000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.32900000000000001},"max":{"1min":1.0369999999999999,"5min":1.6870000000000001,"15min":3.6920000000000002},"last":0.628},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.624,"15min":0.63},"min":{"1min":0.39900000000000002,"5min":0.31,"15min":0.31},"max":{"1min":1.9570000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.47799999999999998}]},{"osd":6,"last update":"Sat Apr 6 12:27:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.63100000000000001,"15min":0.63500000000000001},"min":{"1min":0.436,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.66400000000000003,"15min":0.66300000000000003},"min":{"1min":0.42499999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0760000000000001,"5min":1.728,"15min":3.7490000000000001},"last":0.51000000000000001}]},{"osd":7,"last update":"Sat Apr 6 12:27:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.64000000000000001,"15min":0.64800000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.35599999999999998},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":1.0469999999999999},"last":0.62},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.46899999999999997,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0049999999999999,"5min":1.758,"15min":3.7200000000000002},"last":0.54600000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575041,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1204224,"data_stored":596324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.23599999999999999,"5min":0.21099999999999999,"15min":0.191},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.379,"15min":0.38100000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.19600000000000001},"max":{"1min":0.56100000000000005,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.30399999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.38100000000000001,"15min":0.38},"min":{"1min":0.32000000000000001,"5min":0.22700000000000001,"15min":0.215},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.432,"5min":0.40600000000000003,"15min":0.41299999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.247},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.33300000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.36199999999999999,"5min":0.27200000000000002,"15min":0.25700000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.379},{"interface":"front","average":{"1min":0.503,"5min":0.47399999999999998,"15min":0.47499999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.29899999999999999},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.36699999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.40300000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81100000000000005,"5min":0.88500000000000001,"15min":0.95999999999999996},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.60299999999999998,"15min":0.622},"min":{"1min":0.41099999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90100000000000002,"5min":0.90300000000000002,"15min":0.93500000000000005},"last":0.69799999999999995}]},{"osd":5,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.64200000000000002,"15min":0.65900000000000003},"min":{"1min":0.45600000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.96699999999999997},"last":0.746},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65300000000000002,"15min":0.65800000000000003},"min":{"1min":0.44400000000000001,"5min":0.38200000000000001,"15min":0.36899999999999999},"max":{"1min":0.86699999999999999,"5min":0.94499999999999995,"15min":0.95599999999999996},"last":0.70799999999999996}]},{"osd":6,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.60899999999999999,"15min":0.622},"min":{"1min":0.38,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.65100000000000002,"15min":0.65800000000000003},"min":{"1min":0.47599999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.60399999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.63500000000000001,"15min":0.64400000000000002},"min":{"1min":0.47299999999999998,"5min":0.40400000000000003,"15min":0.33300000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.65100000000000002,"15min":0.66300000000000003},"min":{"1min":0.47999999999999998,"5min":0.39500000000000002,"15min":0.373},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.82599999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575041,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32308,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339532,"statfs":{"total":96636764160,"available":96603680768,"internally_reserved":0,"allocated":1220608,"data_stored":597410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56399999999999995,"15min":0.56299999999999994},"min":{"1min":0.371,"5min":0.29399999999999998,"15min":0.23899999999999999},"max":{"1min":0.72499999999999998,"5min":0.80800000000000005,"15min":0.82999999999999996},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.55900000000000005},"min":{"1min":0.33800000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.82199999999999995},"last":0.40899999999999997}]},{"osd":1,"last update":"Sat Apr 6 12:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59299999999999997,"15min":0.622},"min":{"1min":0.45100000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.76200000000000001,"5min":0.76900000000000002,"15min":4.3479999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58899999999999997,"15min":0.60899999999999999},"min":{"1min":0.36799999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":4.3319999999999999},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55800000000000005,"15min":0.56699999999999995},"min":{"1min":0.32700000000000001,"5min":0.313,"15min":0.311},"max":{"1min":0.73499999999999999,"5min":0.76100000000000001,"15min":0.78400000000000003},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.40100000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.71699999999999997,"5min":0.74299999999999999,"15min":0.82199999999999995},"last":0.379}]},{"osd":3,"last update":"Sat Apr 6 12:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55600000000000005,"15min":0.57599999999999996},"min":{"1min":0.373,"5min":0.36399999999999999,"15min":0.34899999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":4.1950000000000003},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.59099999999999997,"15min":0.60099999999999998},"min":{"1min":0.434,"5min":0.40500000000000003,"15min":0.36099999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":1.2330000000000001},"last":0.42099999999999999}]},{"osd":5,"last update":"Sat Apr 6 12:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.45900000000000002,"15min":0.48299999999999998},"min":{"1min":0.315,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65500000000000003,"5min":0.76300000000000001,"15min":1.196},"last":0.315},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.49299999999999999,"15min":0.499},"min":{"1min":0.34399999999999997,"5min":0.33500000000000002,"15min":0.312},"max":{"1min":0.67500000000000004,"5min":0.74299999999999999,"15min":0.77000000000000002},"last":0.29499999999999998}]},{"osd":6,"last update":"Sat Apr 6 12:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.49299999999999999,"15min":0.504},"min":{"1min":0.34200000000000003,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.67200000000000004,"5min":0.754,"15min":1.022},"last":0.307},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52000000000000002,"15min":0.53800000000000003},"min":{"1min":0.39900000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69399999999999995,"5min":0.71799999999999997,"15min":0.872},"last":0.35299999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51700000000000002,"15min":0.53500000000000003},"min":{"1min":0.38400000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.73799999999999999,"5min":0.79300000000000004,"15min":0.91500000000000004},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54900000000000004,"15min":0.56999999999999995},"min":{"1min":0.38700000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.749,"5min":0.749,"15min":1.0589999999999999},"last":0.36699999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575041,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46236,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325604,"statfs":{"total":96636764160,"available":96589418496,"internally_reserved":0,"allocated":1187840,"data_stored":588137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.36799999999999999,"5min":0.32700000000000001,"15min":0.24199999999999999},"max":{"1min":0.75900000000000001,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.29499999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.748,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.41899999999999998}]},{"osd":1,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.30399999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55200000000000005,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.34499999999999997,"15min":0.30599999999999999},"max":{"1min":0.73599999999999999,"5min":0.74299999999999999,"15min":0.76200000000000001},"last":0.45700000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56999999999999995,"15min":0.56200000000000006},"min":{"1min":0.33200000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.72199999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.40999999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72099999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54100000000000004}]},{"osd":3,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.33900000000000002,"5min":0.32500000000000001,"15min":0.31},"max":{"1min":0.77800000000000002,"5min":0.874,"15min":0.874},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.436,"5min":0.377,"15min":0.32300000000000001},"max":{"1min":0.73099999999999998,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.58099999999999996}]},{"osd":4,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.28299999999999997},"max":{"1min":0.755,"5min":0.755,"15min":0.81200000000000006},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54000000000000004,"15min":0.53300000000000003},"min":{"1min":0.42599999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73699999999999999,"5min":0.80000000000000004,"15min":0.91800000000000004},"last":0.64900000000000002}]},{"osd":6,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49099999999999999,"15min":0.48599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.28000000000000003},"max":{"1min":0.63800000000000001,"5min":0.73899999999999999,"15min":0.88700000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.496,"15min":0.49299999999999999},"min":{"1min":0.33100000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64800000000000002,"5min":0.73199999999999998,"15min":0.83299999999999996},"last":0.627}]},{"osd":7,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.252},"max":{"1min":0.68999999999999995,"5min":0.78600000000000003,"15min":0.79100000000000004},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60999999999999999,"15min":0.60199999999999998},"min":{"1min":0.44800000000000001,"5min":0.41399999999999998,"15min":0.35199999999999998},"max":{"1min":0.72499999999999998,"5min":0.85299999999999998,"15min":0.85599999999999998},"last":0.72699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:28:09.795 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:28:09.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.954+0000 7fa9c55c3700 1 Processor -- start 2024-04-06T12:28:09.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.954+0000 7fa9c55c3700 1 -- start start 2024-04-06T12:28:09.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9c55c3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9c0160b60 0x7fa9c00b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9c55c3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa9c0143470 0x7fa9c00cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9c55c3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c015eb40 con 0x7fa9c015db10 2024-04-06T12:28:09.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9c55c3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9c0160f40 con 0x7fa9c0143470 2024-04-06T12:28:09.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9c55c3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c00d0bf0 con 0x7fa9c0160b60 2024-04-06T12:28:09.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9be7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa9c015db10 0x7fa9c0142d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44598/0 (socket says 172.21.15.189:44598) 2024-04-06T12:28:09.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9be7fc700 1 -- 172.21.15.189:0/3636602281 learned_addr learned my addr 172.21.15.189:0/3636602281 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:09.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9bf7fe700 1 --2- 172.21.15.189:0/3636602281 >> v2:172.21.15.192:3300/0 conn(0x7fa9c0143470 0x7fa9c00cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9bdffb700 1 -- 172.21.15.189:0/3636602281 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120591636 0 0) 0x7fa9b4001290 con 0x7fa9c015db10 2024-04-06T12:28:09.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9beffd700 1 --2- 172.21.15.189:0/3636602281 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9c0160b60 0x7fa9c00b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9bdffb700 1 -- 172.21.15.189:0/3636602281 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a80035b0 con 0x7fa9c015db10 2024-04-06T12:28:09.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9bdffb700 1 -- 172.21.15.189:0/3636602281 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 627613172 0 0) 0x7fa9a80035b0 con 0x7fa9c015db10 2024-04-06T12:28:09.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9bdffb700 1 -- 172.21.15.189:0/3636602281 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9c0160b60 msgr2=0x7fa9c00b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:09.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9bdffb700 1 --2- 172.21.15.189:0/3636602281 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9c0160b60 0x7fa9c00b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9bdffb700 1 -- 172.21.15.189:0/3636602281 >> v2:172.21.15.192:3300/0 conn(0x7fa9c0143470 msgr2=0x7fa9c00cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:09.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9bdffb700 1 --2- 172.21.15.189:0/3636602281 >> v2:172.21.15.192:3300/0 conn(0x7fa9c0143470 0x7fa9c00cd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9bdffb700 1 -- 172.21.15.189:0/3636602281 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c015dfc0 con 0x7fa9c015db10 2024-04-06T12:28:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9bdffb700 1 -- 172.21.15.189:0/3636602281 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fa9b4001980 con 0x7fa9c015db10 2024-04-06T12:28:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9bdffb700 1 -- 172.21.15.189:0/3636602281 >> v1:172.21.15.189:6789/0 conn(0x7fa9c015db10 legacy=0x7fa9c0142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9bdffb700 1 --2- 172.21.15.189:0/3636602281 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9c0160b60 0x7fa9a8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9bdffb700 1 --2- 172.21.15.189:0/3636602281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9a8003f80 0x7fa9a8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9bdffb700 1 --2- 172.21.15.189:0/3636602281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9a8004d90 0x7fa9a8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9bdffb700 1 -- 172.21.15.189:0/3636602281 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c0160f40 con 0x7fa9a8003f80 2024-04-06T12:28:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.955+0000 7fa9bdffb700 1 -- 172.21.15.189:0/3636602281 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c00d0bf0 con 0x7fa9a8004d90 2024-04-06T12:28:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.956+0000 7fa9bdffb700 1 -- 172.21.15.189:0/3636602281 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a800ba10 con 0x7fa9c0160b60 2024-04-06T12:28:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.956+0000 7fa9be7fc700 1 --2- 172.21.15.189:0/3636602281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9a8003f80 0x7fa9a8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.956+0000 7fa9beffd700 1 --2- 172.21.15.189:0/3636602281 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9c0160b60 0x7fa9a8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.956+0000 7fa9bf7fe700 1 --2- 172.21.15.189:0/3636602281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9a8004d90 0x7fa9a8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.956+0000 7fa9be7fc700 1 -- 172.21.15.189:0/3636602281 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9c0160b60 msgr2=0x7fa9a8003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.956+0000 7fa9be7fc700 1 --2- 172.21.15.189:0/3636602281 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9c0160b60 0x7fa9a8003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.956+0000 7fa9be7fc700 1 -- 172.21.15.189:0/3636602281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9a8004d90 msgr2=0x7fa9a8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:09.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.956+0000 7fa9be7fc700 1 --2- 172.21.15.189:0/3636602281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9a8004d90 0x7fa9a8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.956+0000 7fa9be7fc700 1 -- 172.21.15.189:0/3636602281 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a80035b0 con 0x7fa9a8003f80 2024-04-06T12:28:09.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.956+0000 7fa9be7fc700 1 --2- 172.21.15.189:0/3636602281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9a8003f80 0x7fa9a8004850 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40011d0 tx=0x7fa9a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dbfea3f8e5a3fbc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:09.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.956+0000 7fa9bdffb700 1 -- 172.21.15.189:0/3636602281 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b400cd70 con 0x7fa9a8003f80 2024-04-06T12:28:09.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.956+0000 7fa9bdffb700 1 -- 172.21.15.189:0/3636602281 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b4007d80 con 0x7fa9a8003f80 2024-04-06T12:28:09.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.956+0000 7fa9bdffb700 1 -- 172.21.15.189:0/3636602281 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b4007310 con 0x7fa9a8003f80 2024-04-06T12:28:09.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.957+0000 7fa9c55c3700 1 -- 172.21.15.189:0/3636602281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9a8003f80 msgr2=0x7fa9a8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:09.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.957+0000 7fa9c55c3700 1 --2- 172.21.15.189:0/3636602281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9a8003f80 0x7fa9a8004850 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40011d0 tx=0x7fa9a8005880 comp rx=0 tx=0).stop 2024-04-06T12:28:09.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.957+0000 7fa9bf7fe700 1 -- 172.21.15.189:0/3636602281 reap_dead start 2024-04-06T12:28:09.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.957+0000 7fa9c55c3700 1 -- 172.21.15.189:0/3636602281 shutdown_connections 2024-04-06T12:28:09.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.957+0000 7fa9c55c3700 1 -- 172.21.15.189:0/3636602281 >> 172.21.15.189:0/3636602281 conn(0x7fa9c006a730 msgr2=0x7fa9c006ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:09.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.957+0000 7fa9c55c3700 1 -- 172.21.15.189:0/3636602281 shutdown_connections 2024-04-06T12:28:09.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.957+0000 7fa9c55c3700 1 -- 172.21.15.189:0/3636602281 wait complete. 2024-04-06T12:28:09.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.958+0000 7fa9c55c3700 1 Processor -- start 2024-04-06T12:28:09.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.958+0000 7fa9c55c3700 1 -- start start 2024-04-06T12:28:09.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.959+0000 7fa9c55c3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c006cce0 0x7fa9c00c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.959+0000 7fa9c55c3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9c006d0c0 0x7fa9c015c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.959+0000 7fa9c55c3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9c00c9ab0 0x7fa9c015cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.959+0000 7fa9c55c3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c00ca070 con 0x7fa9c006cce0 2024-04-06T12:28:09.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.959+0000 7fa9c55c3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c00ca350 con 0x7fa9c006d0c0 2024-04-06T12:28:09.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.959+0000 7fa9c55c3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c00ca630 con 0x7fa9c00c9ab0 2024-04-06T12:28:09.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.959+0000 7fa9beffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c006cce0 0x7fa9c00c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.959+0000 7fa9beffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c006cce0 0x7fa9c00c90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52206/0 (socket says 172.21.15.189:52206) 2024-04-06T12:28:09.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.959+0000 7fa9beffd700 1 -- 172.21.15.189:0/1924398611 learned_addr learned my addr 172.21.15.189:0/1924398611 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:09.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.959+0000 7fa9be7fc700 1 --2- 172.21.15.189:0/1924398611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9c006d0c0 0x7fa9c015c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.959+0000 7fa9bf7fe700 1 --2- 172.21.15.189:0/1924398611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9c00c9ab0 0x7fa9c015cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.959+0000 7fa9beffd700 1 -- 172.21.15.189:0/1924398611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9c00c9ab0 msgr2=0x7fa9c015cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:09.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.959+0000 7fa9beffd700 1 --2- 172.21.15.189:0/1924398611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9c00c9ab0 0x7fa9c015cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.959+0000 7fa9beffd700 1 -- 172.21.15.189:0/1924398611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9c006d0c0 msgr2=0x7fa9c015c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:09.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.959+0000 7fa9beffd700 1 --2- 172.21.15.189:0/1924398611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9c006d0c0 0x7fa9c015c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:09.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.959+0000 7fa9beffd700 1 -- 172.21.15.189:0/1924398611 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c00ca910 con 0x7fa9c006cce0 2024-04-06T12:28:09.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.959+0000 7fa9be7fc700 1 --2- 172.21.15.189:0/1924398611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9c006d0c0 0x7fa9c015c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:09.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.959+0000 7fa9beffd700 1 --2- 172.21.15.189:0/1924398611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c006cce0 0x7fa9c00c90c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0011aa0 tx=0x7fa9b0011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:09.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.960+0000 7fa9a77fe700 1 -- 172.21.15.189:0/1924398611 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b000f980 con 0x7fa9c006cce0 2024-04-06T12:28:09.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.960+0000 7fa9a77fe700 1 -- 172.21.15.189:0/1924398611 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b000fae0 con 0x7fa9c006cce0 2024-04-06T12:28:09.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.960+0000 7fa9a77fe700 1 -- 172.21.15.189:0/1924398611 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b000fe90 con 0x7fa9c006cce0 2024-04-06T12:28:09.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.960+0000 7fa9c55c3700 1 -- 172.21.15.189:0/1924398611 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c00cabf0 con 0x7fa9c006cce0 2024-04-06T12:28:09.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.960+0000 7fa9c55c3700 1 -- 172.21.15.189:0/1924398611 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c00cb110 con 0x7fa9c006cce0 2024-04-06T12:28:09.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.960+0000 7fa9c55c3700 1 -- 172.21.15.189:0/1924398611 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c00ca490 con 0x7fa9c006cce0 2024-04-06T12:28:09.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.961+0000 7fa9a77fe700 1 -- 172.21.15.189:0/1924398611 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fa9b0014440 con 0x7fa9c006cce0 2024-04-06T12:28:09.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.961+0000 7fa9a77fe700 1 --2- 172.21.15.189:0/1924398611 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa9ac06adb0 0x7fa9ac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:09.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.962+0000 7fa9be7fc700 1 --2- 172.21.15.189:0/1924398611 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa9ac06adb0 0x7fa9ac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:09.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.962+0000 7fa9a77fe700 1 -- 172.21.15.189:0/1924398611 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa9b0089cd0 con 0x7fa9c006cce0 2024-04-06T12:28:09.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.962+0000 7fa9be7fc700 1 --2- 172.21.15.189:0/1924398611 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa9ac06adb0 0x7fa9ac06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa9c004fd90 tx=0x7fa9b400a000 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:09.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:09.966+0000 7fa9a77fe700 1 -- 172.21.15.189:0/1924398611 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa9c00ca490 con 0x7fa9c006cce0 2024-04-06T12:28:10.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.155+0000 7fa9c55c3700 1 -- 172.21.15.189:0/1924398611 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa9c015e860 con 0x7fa9ac06adb0 2024-04-06T12:28:10.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.162+0000 7fa9a77fe700 1 -- 172.21.15.189:0/1924398611 <== mgr.6442 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289226 (secure 0 0 0) 0x7fa9b40018c0 con 0x7fa9ac06adb0 2024-04-06T12:28:10.164 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:10.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.166+0000 7fa9c55c3700 1 -- 172.21.15.189:0/1924398611 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa9ac06adb0 msgr2=0x7fa9ac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:10.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.166+0000 7fa9c55c3700 1 --2- 172.21.15.189:0/1924398611 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa9ac06adb0 0x7fa9ac06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa9c004fd90 tx=0x7fa9b400a000 comp rx=0 tx=0).stop 2024-04-06T12:28:10.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.166+0000 7fa9c55c3700 1 -- 172.21.15.189:0/1924398611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c006cce0 msgr2=0x7fa9c00c90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.166+0000 7fa9c55c3700 1 --2- 172.21.15.189:0/1924398611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c006cce0 0x7fa9c00c90c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0011aa0 tx=0x7fa9b0011e60 comp rx=0 tx=0).stop 2024-04-06T12:28:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.166+0000 7fa9c55c3700 1 -- 172.21.15.189:0/1924398611 shutdown_connections 2024-04-06T12:28:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.166+0000 7fa9c55c3700 1 --2- 172.21.15.189:0/1924398611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9c00c9ab0 0x7fa9c015cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.166+0000 7fa9c55c3700 1 --2- 172.21.15.189:0/1924398611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c006cce0 0x7fa9c00c90c0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.166+0000 7fa9c55c3700 1 --2- 172.21.15.189:0/1924398611 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa9ac06adb0 0x7fa9ac06d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.166+0000 7fa9c55c3700 1 --2- 172.21.15.189:0/1924398611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9c006d0c0 0x7fa9c015c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.166+0000 7fa9c55c3700 1 -- 172.21.15.189:0/1924398611 >> 172.21.15.189:0/1924398611 conn(0x7fa9c006a730 msgr2=0x7fa9c0145f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.166+0000 7fa9c55c3700 1 -- 172.21.15.189:0/1924398611 shutdown_connections 2024-04-06T12:28:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.166+0000 7fa9c55c3700 1 -- 172.21.15.189:0/1924398611 wait complete. 2024-04-06T12:28:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:28:10.177 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-06T12:28:08.863553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":303,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":317772,"kb_used_data":10556,"kb_used_omap":12,"kb_used_meta":307059,"kb_avail":754656948,"statfs":{"total":773094113280,"available":772768714752,"internally_reserved":0,"allocated":10809344,"data_stored":5933061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":314428984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.611382"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.680019+0000","last_change":"2024-04-06T12:28:05.691699+0000","last_active":"2024-04-06T12:28:06.680019+0000","last_peered":"2024-04-06T12:28:06.680019+0000","last_clean":"2024-04-06T12:28:06.680019+0000","last_became_active":"2024-04-06T12:28:05.691446+0000","last_became_peered":"2024-04-06T12:28:05.691446+0000","last_unstale":"2024-04-06T12:28:06.680019+0000","last_undegraded":"2024-04-06T12:28:06.680019+0000","last_fullsized":"2024-04-06T12:28:06.680019+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:06:40.208413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317327+0000","last_change":"2024-04-06T12:28:05.695965+0000","last_active":"2024-04-06T12:28:07.317327+0000","last_peered":"2024-04-06T12:28:07.317327+0000","last_clean":"2024-04-06T12:28:07.317327+0000","last_became_active":"2024-04-06T12:28:05.695730+0000","last_became_peered":"2024-04-06T12:28:05.695730+0000","last_unstale":"2024-04-06T12:28:07.317327+0000","last_undegraded":"2024-04-06T12:28:07.317327+0000","last_fullsized":"2024-04-06T12:28:07.317327+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:06:46.449407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.770547+0000","last_change":"2024-04-06T12:28:05.698830+0000","last_active":"2024-04-06T12:28:06.770547+0000","last_peered":"2024-04-06T12:28:06.770547+0000","last_clean":"2024-04-06T12:28:06.770547+0000","last_became_active":"2024-04-06T12:28:05.698509+0000","last_became_peered":"2024-04-06T12:28:05.698509+0000","last_unstale":"2024-04-06T12:28:06.770547+0000","last_undegraded":"2024-04-06T12:28:06.770547+0000","last_fullsized":"2024-04-06T12:28:06.770547+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:40:58.915981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.699172+0000","last_change":"2024-04-06T12:28:05.697459+0000","last_active":"2024-04-06T12:28:05.699172+0000","last_peered":"2024-04-06T12:28:05.699172+0000","last_clean":"2024-04-06T12:28:05.699172+0000","last_became_active":"2024-04-06T12:28:05.697204+0000","last_became_peered":"2024-04-06T12:28:05.697204+0000","last_unstale":"2024-04-06T12:28:05.699172+0000","last_undegraded":"2024-04-06T12:28:05.699172+0000","last_fullsized":"2024-04-06T12:28:05.699172+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:39:26.704304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553130+0000","last_change":"2024-04-06T12:28:05.693661+0000","last_active":"2024-04-06T12:28:07.553130+0000","last_peered":"2024-04-06T12:28:07.553130+0000","last_clean":"2024-04-06T12:28:07.553130+0000","last_became_active":"2024-04-06T12:28:05.693416+0000","last_became_peered":"2024-04-06T12:28:05.693416+0000","last_unstale":"2024-04-06T12:28:07.553130+0000","last_undegraded":"2024-04-06T12:28:07.553130+0000","last_fullsized":"2024-04-06T12:28:07.553130+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:45:05.938509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679084+0000","last_change":"2024-04-06T12:28:05.701612+0000","last_active":"2024-04-06T12:28:06.679084+0000","last_peered":"2024-04-06T12:28:06.679084+0000","last_clean":"2024-04-06T12:28:06.679084+0000","last_became_active":"2024-04-06T12:28:05.700750+0000","last_became_peered":"2024-04-06T12:28:05.700750+0000","last_unstale":"2024-04-06T12:28:06.679084+0000","last_undegraded":"2024-04-06T12:28:06.679084+0000","last_fullsized":"2024-04-06T12:28:06.679084+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:27:50.018142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.696254+0000","last_change":"2024-04-06T12:28:05.690758+0000","last_active":"2024-04-06T12:28:05.696254+0000","last_peered":"2024-04-06T12:28:05.696254+0000","last_clean":"2024-04-06T12:28:05.696254+0000","last_became_active":"2024-04-06T12:28:05.690110+0000","last_became_peered":"2024-04-06T12:28:05.690110+0000","last_unstale":"2024-04-06T12:28:05.696254+0000","last_undegraded":"2024-04-06T12:28:05.696254+0000","last_fullsized":"2024-04-06T12:28:05.696254+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:06:04.989226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.552985+0000","last_change":"2024-04-06T12:28:05.694144+0000","last_active":"2024-04-06T12:28:07.552985+0000","last_peered":"2024-04-06T12:28:07.552985+0000","last_clean":"2024-04-06T12:28:07.552985+0000","last_became_active":"2024-04-06T12:28:05.693690+0000","last_became_peered":"2024-04-06T12:28:05.693690+0000","last_unstale":"2024-04-06T12:28:07.552985+0000","last_undegraded":"2024-04-06T12:28:07.552985+0000","last_fullsized":"2024-04-06T12:28:07.552985+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:53:04.368148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.680385+0000","last_change":"2024-04-06T12:28:05.701364+0000","last_active":"2024-04-06T12:28:06.680385+0000","last_peered":"2024-04-06T12:28:06.680385+0000","last_clean":"2024-04-06T12:28:06.680385+0000","last_became_active":"2024-04-06T12:28:05.701072+0000","last_became_peered":"2024-04-06T12:28:05.701072+0000","last_unstale":"2024-04-06T12:28:06.680385+0000","last_undegraded":"2024-04-06T12:28:06.680385+0000","last_fullsized":"2024-04-06T12:28:06.680385+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:52:04.452630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.699043+0000","last_change":"2024-04-06T12:28:05.693611+0000","last_active":"2024-04-06T12:28:05.699043+0000","last_peered":"2024-04-06T12:28:05.699043+0000","last_clean":"2024-04-06T12:28:05.699043+0000","last_became_active":"2024-04-06T12:28:05.693133+0000","last_became_peered":"2024-04-06T12:28:05.693133+0000","last_unstale":"2024-04-06T12:28:05.699043+0000","last_undegraded":"2024-04-06T12:28:05.699043+0000","last_fullsized":"2024-04-06T12:28:05.699043+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:17:03.573001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317258+0000","last_change":"2024-04-06T12:28:05.691929+0000","last_active":"2024-04-06T12:28:07.317258+0000","last_peered":"2024-04-06T12:28:07.317258+0000","last_clean":"2024-04-06T12:28:07.317258+0000","last_became_active":"2024-04-06T12:28:05.691355+0000","last_became_peered":"2024-04-06T12:28:05.691355+0000","last_unstale":"2024-04-06T12:28:07.317258+0000","last_undegraded":"2024-04-06T12:28:07.317258+0000","last_fullsized":"2024-04-06T12:28:07.317258+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:30:39.282974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.706880+0000","last_change":"2024-04-06T12:28:05.702194+0000","last_active":"2024-04-06T12:28:05.706880+0000","last_peered":"2024-04-06T12:28:05.706880+0000","last_clean":"2024-04-06T12:28:05.706880+0000","last_became_active":"2024-04-06T12:28:05.701858+0000","last_became_peered":"2024-04-06T12:28:05.701858+0000","last_unstale":"2024-04-06T12:28:05.706880+0000","last_undegraded":"2024-04-06T12:28:05.706880+0000","last_fullsized":"2024-04-06T12:28:05.706880+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:40:58.915981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553057+0000","last_change":"2024-04-06T12:28:05.704635+0000","last_active":"2024-04-06T12:28:07.553057+0000","last_peered":"2024-04-06T12:28:07.553057+0000","last_clean":"2024-04-06T12:28:07.553057+0000","last_became_active":"2024-04-06T12:28:05.700942+0000","last_became_peered":"2024-04-06T12:28:05.700942+0000","last_unstale":"2024-04-06T12:28:07.553057+0000","last_undegraded":"2024-04-06T12:28:07.553057+0000","last_fullsized":"2024-04-06T12:28:07.553057+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:53:25.733752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679036+0000","last_change":"2024-04-06T12:28:05.701948+0000","last_active":"2024-04-06T12:28:06.679036+0000","last_peered":"2024-04-06T12:28:06.679036+0000","last_clean":"2024-04-06T12:28:06.679036+0000","last_became_active":"2024-04-06T12:28:05.701371+0000","last_became_peered":"2024-04-06T12:28:05.701371+0000","last_unstale":"2024-04-06T12:28:06.679036+0000","last_undegraded":"2024-04-06T12:28:06.679036+0000","last_fullsized":"2024-04-06T12:28:06.679036+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:38:41.832113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317014+0000","last_change":"2024-04-06T12:28:05.695922+0000","last_active":"2024-04-06T12:28:07.317014+0000","last_peered":"2024-04-06T12:28:07.317014+0000","last_clean":"2024-04-06T12:28:07.317014+0000","last_became_active":"2024-04-06T12:28:05.695668+0000","last_became_peered":"2024-04-06T12:28:05.695668+0000","last_unstale":"2024-04-06T12:28:07.317014+0000","last_undegraded":"2024-04-06T12:28:07.317014+0000","last_fullsized":"2024-04-06T12:28:07.317014+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:19:00.647934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679316+0000","last_change":"2024-04-06T12:28:05.692941+0000","last_active":"2024-04-06T12:28:06.679316+0000","last_peered":"2024-04-06T12:28:06.679316+0000","last_clean":"2024-04-06T12:28:06.679316+0000","last_became_active":"2024-04-06T12:28:05.692642+0000","last_became_peered":"2024-04-06T12:28:05.692642+0000","last_unstale":"2024-04-06T12:28:06.679316+0000","last_undegraded":"2024-04-06T12:28:06.679316+0000","last_fullsized":"2024-04-06T12:28:06.679316+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:35:21.328512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553173+0000","last_change":"2024-04-06T12:28:05.699666+0000","last_active":"2024-04-06T12:28:07.553173+0000","last_peered":"2024-04-06T12:28:07.553173+0000","last_clean":"2024-04-06T12:28:07.553173+0000","last_became_active":"2024-04-06T12:28:05.698732+0000","last_became_peered":"2024-04-06T12:28:05.698732+0000","last_unstale":"2024-04-06T12:28:07.553173+0000","last_undegraded":"2024-04-06T12:28:07.553173+0000","last_fullsized":"2024-04-06T12:28:07.553173+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:25:10.228465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.770690+0000","last_change":"2024-04-06T12:28:05.695478+0000","last_active":"2024-04-06T12:28:06.770690+0000","last_peered":"2024-04-06T12:28:06.770690+0000","last_clean":"2024-04-06T12:28:06.770690+0000","last_became_active":"2024-04-06T12:28:05.695232+0000","last_became_peered":"2024-04-06T12:28:05.695232+0000","last_unstale":"2024-04-06T12:28:06.770690+0000","last_undegraded":"2024-04-06T12:28:06.770690+0000","last_fullsized":"2024-04-06T12:28:06.770690+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:33:04.371728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.678868+0000","last_change":"2024-04-06T12:28:05.693475+0000","last_active":"2024-04-06T12:28:06.678868+0000","last_peered":"2024-04-06T12:28:06.678868+0000","last_clean":"2024-04-06T12:28:06.678868+0000","last_became_active":"2024-04-06T12:28:05.693181+0000","last_became_peered":"2024-04-06T12:28:05.693181+0000","last_unstale":"2024-04-06T12:28:06.678868+0000","last_undegraded":"2024-04-06T12:28:06.678868+0000","last_fullsized":"2024-04-06T12:28:06.678868+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:00:06.871498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553087+0000","last_change":"2024-04-06T12:28:05.700148+0000","last_active":"2024-04-06T12:28:07.553087+0000","last_peered":"2024-04-06T12:28:07.553087+0000","last_clean":"2024-04-06T12:28:07.553087+0000","last_became_active":"2024-04-06T12:28:05.698218+0000","last_became_peered":"2024-04-06T12:28:05.698218+0000","last_unstale":"2024-04-06T12:28:07.553087+0000","last_undegraded":"2024-04-06T12:28:07.553087+0000","last_fullsized":"2024-04-06T12:28:07.553087+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:35:59.792050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679684+0000","last_change":"2024-04-06T12:28:05.696033+0000","last_active":"2024-04-06T12:28:06.679684+0000","last_peered":"2024-04-06T12:28:06.679684+0000","last_clean":"2024-04-06T12:28:06.679684+0000","last_became_active":"2024-04-06T12:28:05.695383+0000","last_became_peered":"2024-04-06T12:28:05.695383+0000","last_unstale":"2024-04-06T12:28:06.679684+0000","last_undegraded":"2024-04-06T12:28:06.679684+0000","last_fullsized":"2024-04-06T12:28:06.679684+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:21:01.116675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553129+0000","last_change":"2024-04-06T12:28:05.704702+0000","last_active":"2024-04-06T12:28:07.553129+0000","last_peered":"2024-04-06T12:28:07.553129+0000","last_clean":"2024-04-06T12:28:07.553129+0000","last_became_active":"2024-04-06T12:28:05.700999+0000","last_became_peered":"2024-04-06T12:28:05.700999+0000","last_unstale":"2024-04-06T12:28:07.553129+0000","last_undegraded":"2024-04-06T12:28:07.553129+0000","last_fullsized":"2024-04-06T12:28:07.553129+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:52:16.330147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679389+0000","last_change":"2024-04-06T12:28:05.694511+0000","last_active":"2024-04-06T12:28:06.679389+0000","last_peered":"2024-04-06T12:28:06.679389+0000","last_clean":"2024-04-06T12:28:06.679389+0000","last_became_active":"2024-04-06T12:28:05.694256+0000","last_became_peered":"2024-04-06T12:28:05.694256+0000","last_unstale":"2024-04-06T12:28:06.679389+0000","last_undegraded":"2024-04-06T12:28:06.679389+0000","last_fullsized":"2024-04-06T12:28:06.679389+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:09:43.251187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.316979+0000","last_change":"2024-04-06T12:28:05.689643+0000","last_active":"2024-04-06T12:28:07.316979+0000","last_peered":"2024-04-06T12:28:07.316979+0000","last_clean":"2024-04-06T12:28:07.316979+0000","last_became_active":"2024-04-06T12:28:05.689380+0000","last_became_peered":"2024-04-06T12:28:05.689380+0000","last_unstale":"2024-04-06T12:28:07.316979+0000","last_undegraded":"2024-04-06T12:28:07.316979+0000","last_fullsized":"2024-04-06T12:28:07.316979+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:04:49.090630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317309+0000","last_change":"2024-04-06T12:28:05.701915+0000","last_active":"2024-04-06T12:28:07.317309+0000","last_peered":"2024-04-06T12:28:07.317309+0000","last_clean":"2024-04-06T12:28:07.317309+0000","last_became_active":"2024-04-06T12:28:05.701658+0000","last_became_peered":"2024-04-06T12:28:05.701658+0000","last_unstale":"2024-04-06T12:28:07.317309+0000","last_undegraded":"2024-04-06T12:28:07.317309+0000","last_fullsized":"2024-04-06T12:28:07.317309+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:41:29.659432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.706861+0000","last_change":"2024-04-06T12:28:05.702195+0000","last_active":"2024-04-06T12:28:05.706861+0000","last_peered":"2024-04-06T12:28:05.706861+0000","last_clean":"2024-04-06T12:28:05.706861+0000","last_became_active":"2024-04-06T12:28:05.701927+0000","last_became_peered":"2024-04-06T12:28:05.701927+0000","last_unstale":"2024-04-06T12:28:05.706861+0000","last_undegraded":"2024-04-06T12:28:05.706861+0000","last_fullsized":"2024-04-06T12:28:05.706861+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:41:14.655391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679838+0000","last_change":"2024-04-06T12:28:05.688824+0000","last_active":"2024-04-06T12:28:06.679838+0000","last_peered":"2024-04-06T12:28:06.679838+0000","last_clean":"2024-04-06T12:28:06.679838+0000","last_became_active":"2024-04-06T12:28:05.688538+0000","last_became_peered":"2024-04-06T12:28:05.688538+0000","last_unstale":"2024-04-06T12:28:06.679838+0000","last_undegraded":"2024-04-06T12:28:06.679838+0000","last_fullsized":"2024-04-06T12:28:06.679838+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:18:59.687118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553124+0000","last_change":"2024-04-06T12:28:05.700151+0000","last_active":"2024-04-06T12:28:07.553124+0000","last_peered":"2024-04-06T12:28:07.553124+0000","last_clean":"2024-04-06T12:28:07.553124+0000","last_became_active":"2024-04-06T12:28:05.699858+0000","last_became_peered":"2024-04-06T12:28:05.699858+0000","last_unstale":"2024-04-06T12:28:07.553124+0000","last_undegraded":"2024-04-06T12:28:07.553124+0000","last_fullsized":"2024-04-06T12:28:07.553124+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:56:29.371271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317016+0000","last_change":"2024-04-06T12:28:05.689846+0000","last_active":"2024-04-06T12:28:07.317016+0000","last_peered":"2024-04-06T12:28:07.317016+0000","last_clean":"2024-04-06T12:28:07.317016+0000","last_became_active":"2024-04-06T12:28:05.689545+0000","last_became_peered":"2024-04-06T12:28:05.689545+0000","last_unstale":"2024-04-06T12:28:07.317016+0000","last_undegraded":"2024-04-06T12:28:07.317016+0000","last_fullsized":"2024-04-06T12:28:07.317016+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:56:45.841002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679597+0000","last_change":"2024-04-06T12:28:05.696142+0000","last_active":"2024-04-06T12:28:06.679597+0000","last_peered":"2024-04-06T12:28:06.679597+0000","last_clean":"2024-04-06T12:28:06.679597+0000","last_became_active":"2024-04-06T12:28:05.695854+0000","last_became_peered":"2024-04-06T12:28:05.695854+0000","last_unstale":"2024-04-06T12:28:06.679597+0000","last_undegraded":"2024-04-06T12:28:06.679597+0000","last_fullsized":"2024-04-06T12:28:06.679597+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:45:17.012967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.770344+0000","last_change":"2024-04-06T12:28:05.692161+0000","last_active":"2024-04-06T12:28:06.770344+0000","last_peered":"2024-04-06T12:28:06.770344+0000","last_clean":"2024-04-06T12:28:06.770344+0000","last_became_active":"2024-04-06T12:28:05.691907+0000","last_became_peered":"2024-04-06T12:28:05.691907+0000","last_unstale":"2024-04-06T12:28:06.770344+0000","last_undegraded":"2024-04-06T12:28:06.770344+0000","last_fullsized":"2024-04-06T12:28:06.770344+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:21:01.116675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679392+0000","last_change":"2024-04-06T12:28:05.692715+0000","last_active":"2024-04-06T12:28:06.679392+0000","last_peered":"2024-04-06T12:28:06.679392+0000","last_clean":"2024-04-06T12:28:06.679392+0000","last_became_active":"2024-04-06T12:28:05.692307+0000","last_became_peered":"2024-04-06T12:28:05.692307+0000","last_unstale":"2024-04-06T12:28:06.679392+0000","last_undegraded":"2024-04-06T12:28:06.679392+0000","last_fullsized":"2024-04-06T12:28:06.679392+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:25:11.631130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.770353+0000","last_change":"2024-04-06T12:28:05.690158+0000","last_active":"2024-04-06T12:28:06.770353+0000","last_peered":"2024-04-06T12:28:06.770353+0000","last_clean":"2024-04-06T12:28:06.770353+0000","last_became_active":"2024-04-06T12:28:05.689912+0000","last_became_peered":"2024-04-06T12:28:05.689912+0000","last_unstale":"2024-04-06T12:28:06.770353+0000","last_undegraded":"2024-04-06T12:28:06.770353+0000","last_fullsized":"2024-04-06T12:28:06.770353+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:17:03.573001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"97'4","reported_seq":15,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553163+0000","last_change":"2024-04-06T12:28:05.271413+0000","last_active":"2024-04-06T12:28:07.553163+0000","last_peered":"2024-04-06T12:28:07.553163+0000","last_clean":"2024-04-06T12:28:07.553163+0000","last_became_active":"2024-04-06T12:28:05.270920+0000","last_became_peered":"2024-04-06T12:28:05.270920+0000","last_unstale":"2024-04-06T12:28:07.553163+0000","last_undegraded":"2024-04-06T12:28:07.553163+0000","last_fullsized":"2024-04-06T12:28:07.553163+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:29:59.694114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.316981+0000","last_change":"2024-04-06T12:28:05.690616+0000","last_active":"2024-04-06T12:28:07.316981+0000","last_peered":"2024-04-06T12:28:07.316981+0000","last_clean":"2024-04-06T12:28:07.316981+0000","last_became_active":"2024-04-06T12:28:05.690327+0000","last_became_peered":"2024-04-06T12:28:05.690327+0000","last_unstale":"2024-04-06T12:28:07.316981+0000","last_undegraded":"2024-04-06T12:28:07.316981+0000","last_fullsized":"2024-04-06T12:28:07.316981+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:38:41.832113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"97'1","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.680310+0000","last_change":"2024-04-06T12:28:05.267505+0000","last_active":"2024-04-06T12:28:06.680310+0000","last_peered":"2024-04-06T12:28:06.680310+0000","last_clean":"2024-04-06T12:28:06.680310+0000","last_became_active":"2024-04-06T12:28:05.267224+0000","last_became_peered":"2024-04-06T12:28:05.267224+0000","last_unstale":"2024-04-06T12:28:06.680310+0000","last_undegraded":"2024-04-06T12:28:06.680310+0000","last_fullsized":"2024-04-06T12:28:06.680310+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:39:25.698878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.680156+0000","last_change":"2024-04-06T12:28:05.701166+0000","last_active":"2024-04-06T12:28:06.680156+0000","last_peered":"2024-04-06T12:28:06.680156+0000","last_clean":"2024-04-06T12:28:06.680156+0000","last_became_active":"2024-04-06T12:28:05.700899+0000","last_became_peered":"2024-04-06T12:28:05.700899+0000","last_unstale":"2024-04-06T12:28:06.680156+0000","last_undegraded":"2024-04-06T12:28:06.680156+0000","last_fullsized":"2024-04-06T12:28:06.680156+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:01:21.360236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"97'2","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.680041+0000","last_change":"2024-04-06T12:28:05.282017+0000","last_active":"2024-04-06T12:28:06.680041+0000","last_peered":"2024-04-06T12:28:06.680041+0000","last_clean":"2024-04-06T12:28:06.680041+0000","last_became_active":"2024-04-06T12:28:05.281774+0000","last_became_peered":"2024-04-06T12:28:05.281774+0000","last_unstale":"2024-04-06T12:28:06.680041+0000","last_undegraded":"2024-04-06T12:28:06.680041+0000","last_fullsized":"2024-04-06T12:28:06.680041+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:33:03.366303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679786+0000","last_change":"2024-04-06T12:28:05.696197+0000","last_active":"2024-04-06T12:28:06.679786+0000","last_peered":"2024-04-06T12:28:06.679786+0000","last_clean":"2024-04-06T12:28:06.679786+0000","last_became_active":"2024-04-06T12:28:05.695772+0000","last_became_peered":"2024-04-06T12:28:05.695772+0000","last_unstale":"2024-04-06T12:28:06.679786+0000","last_undegraded":"2024-04-06T12:28:06.679786+0000","last_fullsized":"2024-04-06T12:28:06.679786+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:27:56.179062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"97'2","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.693681+0000","last_change":"2024-04-06T12:28:05.270215+0000","last_active":"2024-04-06T12:28:05.693681+0000","last_peered":"2024-04-06T12:28:05.693681+0000","last_clean":"2024-04-06T12:28:05.693681+0000","last_became_active":"2024-04-06T12:28:05.269744+0000","last_became_peered":"2024-04-06T12:28:05.269744+0000","last_unstale":"2024-04-06T12:28:05.693681+0000","last_undegraded":"2024-04-06T12:28:05.693681+0000","last_fullsized":"2024-04-06T12:28:05.693681+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:45:04.933084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317132+0000","last_change":"2024-04-06T12:28:05.689014+0000","last_active":"2024-04-06T12:28:07.317132+0000","last_peered":"2024-04-06T12:28:07.317132+0000","last_clean":"2024-04-06T12:28:07.317132+0000","last_became_active":"2024-04-06T12:28:05.688757+0000","last_became_peered":"2024-04-06T12:28:05.688757+0000","last_unstale":"2024-04-06T12:28:07.317132+0000","last_undegraded":"2024-04-06T12:28:07.317132+0000","last_fullsized":"2024-04-06T12:28:07.317132+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:02:58.770663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"97'1","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679885+0000","last_change":"2024-04-06T12:28:05.271076+0000","last_active":"2024-04-06T12:28:06.679885+0000","last_peered":"2024-04-06T12:28:06.679885+0000","last_clean":"2024-04-06T12:28:06.679885+0000","last_became_active":"2024-04-06T12:28:05.270832+0000","last_became_peered":"2024-04-06T12:28:05.270832+0000","last_unstale":"2024-04-06T12:28:06.679885+0000","last_undegraded":"2024-04-06T12:28:06.679885+0000","last_fullsized":"2024-04-06T12:28:06.679885+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:54:21.568200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317209+0000","last_change":"2024-04-06T12:28:05.691996+0000","last_active":"2024-04-06T12:28:07.317209+0000","last_peered":"2024-04-06T12:28:07.317209+0000","last_clean":"2024-04-06T12:28:07.317209+0000","last_became_active":"2024-04-06T12:28:05.691690+0000","last_became_peered":"2024-04-06T12:28:05.691690+0000","last_unstale":"2024-04-06T12:28:07.317209+0000","last_undegraded":"2024-04-06T12:28:07.317209+0000","last_fullsized":"2024-04-06T12:28:07.317209+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:47:25.008418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"97'1","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.694542+0000","last_change":"2024-04-06T12:28:05.274217+0000","last_active":"2024-04-06T12:28:05.694542+0000","last_peered":"2024-04-06T12:28:05.694542+0000","last_clean":"2024-04-06T12:28:05.694542+0000","last_became_active":"2024-04-06T12:28:05.273941+0000","last_became_peered":"2024-04-06T12:28:05.273941+0000","last_unstale":"2024-04-06T12:28:05.694542+0000","last_undegraded":"2024-04-06T12:28:05.694542+0000","last_fullsized":"2024-04-06T12:28:05.694542+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:06:03.983801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679797+0000","last_change":"2024-04-06T12:28:05.701755+0000","last_active":"2024-04-06T12:28:06.679797+0000","last_peered":"2024-04-06T12:28:06.679797+0000","last_clean":"2024-04-06T12:28:06.679797+0000","last_became_active":"2024-04-06T12:28:05.701410+0000","last_became_peered":"2024-04-06T12:28:05.701410+0000","last_unstale":"2024-04-06T12:28:06.679797+0000","last_undegraded":"2024-04-06T12:28:06.679797+0000","last_fullsized":"2024-04-06T12:28:06.679797+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:31:06.373662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317095+0000","last_change":"2024-04-06T12:28:05.281567+0000","last_active":"2024-04-06T12:28:07.317095+0000","last_peered":"2024-04-06T12:28:07.317095+0000","last_clean":"2024-04-06T12:28:07.317095+0000","last_became_active":"2024-04-06T12:28:05.281115+0000","last_became_peered":"2024-04-06T12:28:05.281115+0000","last_unstale":"2024-04-06T12:28:07.317095+0000","last_undegraded":"2024-04-06T12:28:07.317095+0000","last_fullsized":"2024-04-06T12:28:07.317095+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:30:05.507128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.700108+0000","last_change":"2024-04-06T12:28:05.697209+0000","last_active":"2024-04-06T12:28:05.700108+0000","last_peered":"2024-04-06T12:28:05.700108+0000","last_clean":"2024-04-06T12:28:05.700108+0000","last_became_active":"2024-04-06T12:28:05.696917+0000","last_became_peered":"2024-04-06T12:28:05.696917+0000","last_unstale":"2024-04-06T12:28:05.700108+0000","last_undegraded":"2024-04-06T12:28:05.700108+0000","last_fullsized":"2024-04-06T12:28:05.700108+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:03:59.005178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.552944+0000","last_change":"2024-04-06T12:28:05.271913+0000","last_active":"2024-04-06T12:28:07.552944+0000","last_peered":"2024-04-06T12:28:07.552944+0000","last_clean":"2024-04-06T12:28:07.552944+0000","last_became_active":"2024-04-06T12:28:05.271654+0000","last_became_peered":"2024-04-06T12:28:05.271654+0000","last_unstale":"2024-04-06T12:28:07.552944+0000","last_undegraded":"2024-04-06T12:28:07.552944+0000","last_fullsized":"2024-04-06T12:28:07.552944+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:44:14.491419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.316959+0000","last_change":"2024-04-06T12:28:05.683822+0000","last_active":"2024-04-06T12:28:07.316959+0000","last_peered":"2024-04-06T12:28:07.316959+0000","last_clean":"2024-04-06T12:28:07.316959+0000","last_became_active":"2024-04-06T12:28:05.683561+0000","last_became_peered":"2024-04-06T12:28:05.683561+0000","last_unstale":"2024-04-06T12:28:07.316959+0000","last_undegraded":"2024-04-06T12:28:07.316959+0000","last_fullsized":"2024-04-06T12:28:07.316959+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:29:29.562684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.680207+0000","last_change":"2024-04-06T12:28:05.272126+0000","last_active":"2024-04-06T12:28:06.680207+0000","last_peered":"2024-04-06T12:28:06.680207+0000","last_clean":"2024-04-06T12:28:06.680207+0000","last_became_active":"2024-04-06T12:28:05.271842+0000","last_became_peered":"2024-04-06T12:28:05.271842+0000","last_unstale":"2024-04-06T12:28:06.680207+0000","last_undegraded":"2024-04-06T12:28:06.680207+0000","last_fullsized":"2024-04-06T12:28:06.680207+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:52:03.447204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.702062+0000","last_change":"2024-04-06T12:28:05.700058+0000","last_active":"2024-04-06T12:28:05.702062+0000","last_peered":"2024-04-06T12:28:05.702062+0000","last_clean":"2024-04-06T12:28:05.702062+0000","last_became_active":"2024-04-06T12:28:05.699750+0000","last_became_peered":"2024-04-06T12:28:05.699750+0000","last_unstale":"2024-04-06T12:28:05.702062+0000","last_undegraded":"2024-04-06T12:28:05.702062+0000","last_fullsized":"2024-04-06T12:28:05.702062+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:33:04.371728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.682545+0000","last_change":"2024-04-06T12:28:05.274311+0000","last_active":"2024-04-06T12:28:05.682545+0000","last_peered":"2024-04-06T12:28:05.682545+0000","last_clean":"2024-04-06T12:28:05.682545+0000","last_became_active":"2024-04-06T12:28:05.273912+0000","last_became_peered":"2024-04-06T12:28:05.273912+0000","last_unstale":"2024-04-06T12:28:05.682545+0000","last_undegraded":"2024-04-06T12:28:05.682545+0000","last_fullsized":"2024-04-06T12:28:05.682545+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:47:53.798347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.678960+0000","last_change":"2024-04-06T12:28:05.701766+0000","last_active":"2024-04-06T12:28:06.678960+0000","last_peered":"2024-04-06T12:28:06.678960+0000","last_clean":"2024-04-06T12:28:06.678960+0000","last_became_active":"2024-04-06T12:28:05.701055+0000","last_became_peered":"2024-04-06T12:28:05.701055+0000","last_unstale":"2024-04-06T12:28:06.678960+0000","last_undegraded":"2024-04-06T12:28:06.678960+0000","last_fullsized":"2024-04-06T12:28:06.678960+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:39:43.950966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"97'3","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553340+0000","last_change":"2024-04-06T12:28:05.275722+0000","last_active":"2024-04-06T12:28:07.553340+0000","last_peered":"2024-04-06T12:28:07.553340+0000","last_clean":"2024-04-06T12:28:07.553340+0000","last_became_active":"2024-04-06T12:28:05.275329+0000","last_became_peered":"2024-04-06T12:28:05.275329+0000","last_unstale":"2024-04-06T12:28:07.553340+0000","last_undegraded":"2024-04-06T12:28:07.553340+0000","last_fullsized":"2024-04-06T12:28:07.553340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:50:30.941207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.770481+0000","last_change":"2024-04-06T12:28:05.696619+0000","last_active":"2024-04-06T12:28:06.770481+0000","last_peered":"2024-04-06T12:28:06.770481+0000","last_clean":"2024-04-06T12:28:06.770481+0000","last_became_active":"2024-04-06T12:28:05.695878+0000","last_became_peered":"2024-04-06T12:28:05.695878+0000","last_unstale":"2024-04-06T12:28:06.770481+0000","last_undegraded":"2024-04-06T12:28:06.770481+0000","last_fullsized":"2024-04-06T12:28:06.770481+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:07:50.827777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"97'6","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.707170+0000","last_change":"2024-04-06T12:28:05.271400+0000","last_active":"2024-04-06T12:28:05.707170+0000","last_peered":"2024-04-06T12:28:05.707170+0000","last_clean":"2024-04-06T12:28:05.707170+0000","last_became_active":"2024-04-06T12:28:05.271163+0000","last_became_peered":"2024-04-06T12:28:05.271163+0000","last_unstale":"2024-04-06T12:28:05.707170+0000","last_undegraded":"2024-04-06T12:28:05.707170+0000","last_fullsized":"2024-04-06T12:28:05.707170+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:52:15.324722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.702098+0000","last_change":"2024-04-06T12:28:05.699987+0000","last_active":"2024-04-06T12:28:05.702098+0000","last_peered":"2024-04-06T12:28:05.702098+0000","last_clean":"2024-04-06T12:28:05.702098+0000","last_became_active":"2024-04-06T12:28:05.699377+0000","last_became_peered":"2024-04-06T12:28:05.699377+0000","last_unstale":"2024-04-06T12:28:05.702098+0000","last_undegraded":"2024-04-06T12:28:05.702098+0000","last_fullsized":"2024-04-06T12:28:05.702098+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:07:50.827777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"97'2","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.699431+0000","last_change":"2024-04-06T12:28:05.272962+0000","last_active":"2024-04-06T12:28:05.699431+0000","last_peered":"2024-04-06T12:28:05.699431+0000","last_clean":"2024-04-06T12:28:05.699431+0000","last_became_active":"2024-04-06T12:28:05.272663+0000","last_became_peered":"2024-04-06T12:28:05.272663+0000","last_unstale":"2024-04-06T12:28:05.699431+0000","last_undegraded":"2024-04-06T12:28:05.699431+0000","last_fullsized":"2024-04-06T12:28:05.699431+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:09:42.245762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.701629+0000","last_change":"2024-04-06T12:28:05.698630+0000","last_active":"2024-04-06T12:28:05.701629+0000","last_peered":"2024-04-06T12:28:05.701629+0000","last_clean":"2024-04-06T12:28:05.701629+0000","last_became_active":"2024-04-06T12:28:05.696333+0000","last_became_peered":"2024-04-06T12:28:05.696333+0000","last_unstale":"2024-04-06T12:28:05.701629+0000","last_undegraded":"2024-04-06T12:28:05.701629+0000","last_fullsized":"2024-04-06T12:28:05.701629+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:09:43.251187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679702+0000","last_change":"2024-04-06T12:28:05.275895+0000","last_active":"2024-04-06T12:28:06.679702+0000","last_peered":"2024-04-06T12:28:06.679702+0000","last_clean":"2024-04-06T12:28:06.679702+0000","last_became_active":"2024-04-06T12:28:05.275601+0000","last_became_peered":"2024-04-06T12:28:05.275601+0000","last_unstale":"2024-04-06T12:28:06.679702+0000","last_undegraded":"2024-04-06T12:28:06.679702+0000","last_fullsized":"2024-04-06T12:28:06.679702+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:27:55.173637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.698732+0000","last_change":"2024-04-06T12:28:05.691274+0000","last_active":"2024-04-06T12:28:05.698732+0000","last_peered":"2024-04-06T12:28:05.698732+0000","last_clean":"2024-04-06T12:28:05.698732+0000","last_became_active":"2024-04-06T12:28:05.691000+0000","last_became_peered":"2024-04-06T12:28:05.691000+0000","last_unstale":"2024-04-06T12:28:05.698732+0000","last_undegraded":"2024-04-06T12:28:05.698732+0000","last_fullsized":"2024-04-06T12:28:05.698732+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:31:06.373662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.552947+0000","last_change":"2024-04-06T12:28:05.274168+0000","last_active":"2024-04-06T12:28:07.552947+0000","last_peered":"2024-04-06T12:28:07.552947+0000","last_clean":"2024-04-06T12:28:07.552947+0000","last_became_active":"2024-04-06T12:28:05.273912+0000","last_became_peered":"2024-04-06T12:28:05.273912+0000","last_unstale":"2024-04-06T12:28:07.552947+0000","last_undegraded":"2024-04-06T12:28:07.552947+0000","last_fullsized":"2024-04-06T12:28:07.552947+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:53:46.982328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.699480+0000","last_change":"2024-04-06T12:28:05.694938+0000","last_active":"2024-04-06T12:28:05.699480+0000","last_peered":"2024-04-06T12:28:05.699480+0000","last_clean":"2024-04-06T12:28:05.699480+0000","last_became_active":"2024-04-06T12:28:05.694673+0000","last_became_peered":"2024-04-06T12:28:05.694673+0000","last_unstale":"2024-04-06T12:28:05.699480+0000","last_undegraded":"2024-04-06T12:28:05.699480+0000","last_fullsized":"2024-04-06T12:28:05.699480+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:58:23.827592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"97'8","reported_seq":19,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317101+0000","last_change":"2024-04-06T12:28:05.281308+0000","last_active":"2024-04-06T12:28:07.317101+0000","last_peered":"2024-04-06T12:28:07.317101+0000","last_clean":"2024-04-06T12:28:07.317101+0000","last_became_active":"2024-04-06T12:28:05.279789+0000","last_became_peered":"2024-04-06T12:28:05.279789+0000","last_unstale":"2024-04-06T12:28:07.317101+0000","last_undegraded":"2024-04-06T12:28:07.317101+0000","last_fullsized":"2024-04-06T12:28:07.317101+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:00:21.413158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.678524+0000","last_change":"2024-04-06T12:28:05.693048+0000","last_active":"2024-04-06T12:28:06.678524+0000","last_peered":"2024-04-06T12:28:06.678524+0000","last_clean":"2024-04-06T12:28:06.678524+0000","last_became_active":"2024-04-06T12:28:05.692812+0000","last_became_peered":"2024-04-06T12:28:05.692812+0000","last_unstale":"2024-04-06T12:28:06.678524+0000","last_undegraded":"2024-04-06T12:28:06.678524+0000","last_fullsized":"2024-04-06T12:28:06.678524+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:47:00.031681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"97'1","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.680295+0000","last_change":"2024-04-06T12:28:05.274967+0000","last_active":"2024-04-06T12:28:06.680295+0000","last_peered":"2024-04-06T12:28:06.680295+0000","last_clean":"2024-04-06T12:28:06.680295+0000","last_became_active":"2024-04-06T12:28:05.274706+0000","last_became_peered":"2024-04-06T12:28:05.274706+0000","last_unstale":"2024-04-06T12:28:06.680295+0000","last_undegraded":"2024-04-06T12:28:06.680295+0000","last_fullsized":"2024-04-06T12:28:06.680295+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:01:20.354811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553060+0000","last_change":"2024-04-06T12:28:05.695403+0000","last_active":"2024-04-06T12:28:07.553060+0000","last_peered":"2024-04-06T12:28:07.553060+0000","last_clean":"2024-04-06T12:28:07.553060+0000","last_became_active":"2024-04-06T12:28:05.695149+0000","last_became_peered":"2024-04-06T12:28:05.695149+0000","last_unstale":"2024-04-06T12:28:07.553060+0000","last_undegraded":"2024-04-06T12:28:07.553060+0000","last_fullsized":"2024-04-06T12:28:07.553060+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:28.212844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"97'2","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679986+0000","last_change":"2024-04-06T12:28:05.275346+0000","last_active":"2024-04-06T12:28:06.679986+0000","last_peered":"2024-04-06T12:28:06.679986+0000","last_clean":"2024-04-06T12:28:06.679986+0000","last_became_active":"2024-04-06T12:28:05.275064+0000","last_became_peered":"2024-04-06T12:28:05.275064+0000","last_unstale":"2024-04-06T12:28:06.679986+0000","last_undegraded":"2024-04-06T12:28:06.679986+0000","last_fullsized":"2024-04-06T12:28:06.679986+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:29.967187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553079+0000","last_change":"2024-04-06T12:28:05.700626+0000","last_active":"2024-04-06T12:28:07.553079+0000","last_peered":"2024-04-06T12:28:07.553079+0000","last_clean":"2024-04-06T12:28:07.553079+0000","last_became_active":"2024-04-06T12:28:05.700312+0000","last_became_peered":"2024-04-06T12:28:05.700312+0000","last_unstale":"2024-04-06T12:28:07.553079+0000","last_undegraded":"2024-04-06T12:28:07.553079+0000","last_fullsized":"2024-04-06T12:28:07.553079+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:40:45.532919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"97'1","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553343+0000","last_change":"2024-04-06T12:28:05.272133+0000","last_active":"2024-04-06T12:28:07.553343+0000","last_peered":"2024-04-06T12:28:07.553343+0000","last_clean":"2024-04-06T12:28:07.553343+0000","last_became_active":"2024-04-06T12:28:05.271051+0000","last_became_peered":"2024-04-06T12:28:05.271051+0000","last_unstale":"2024-04-06T12:28:07.553343+0000","last_undegraded":"2024-04-06T12:28:07.553343+0000","last_fullsized":"2024-04-06T12:28:07.553343+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:05.280995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.678761+0000","last_change":"2024-04-06T12:28:05.695403+0000","last_active":"2024-04-06T12:28:06.678761+0000","last_peered":"2024-04-06T12:28:06.678761+0000","last_clean":"2024-04-06T12:28:06.678761+0000","last_became_active":"2024-04-06T12:28:05.695139+0000","last_became_peered":"2024-04-06T12:28:05.695139+0000","last_unstale":"2024-04-06T12:28:06.678761+0000","last_undegraded":"2024-04-06T12:28:06.678761+0000","last_fullsized":"2024-04-06T12:28:06.678761+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:10:38.252973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.770519+0000","last_change":"2024-04-06T12:28:05.274790+0000","last_active":"2024-04-06T12:28:06.770519+0000","last_peered":"2024-04-06T12:28:06.770519+0000","last_clean":"2024-04-06T12:28:06.770519+0000","last_became_active":"2024-04-06T12:28:05.274522+0000","last_became_peered":"2024-04-06T12:28:05.274522+0000","last_unstale":"2024-04-06T12:28:06.770519+0000","last_undegraded":"2024-04-06T12:28:06.770519+0000","last_fullsized":"2024-04-06T12:28:06.770519+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:47:53.798347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317233+0000","last_change":"2024-04-06T12:28:05.689552+0000","last_active":"2024-04-06T12:28:07.317233+0000","last_peered":"2024-04-06T12:28:07.317233+0000","last_clean":"2024-04-06T12:28:07.317233+0000","last_became_active":"2024-04-06T12:28:05.689302+0000","last_became_peered":"2024-04-06T12:28:05.689302+0000","last_unstale":"2024-04-06T12:28:07.317233+0000","last_undegraded":"2024-04-06T12:28:07.317233+0000","last_fullsized":"2024-04-06T12:28:07.317233+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:44:06.415054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"97'2","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.678844+0000","last_change":"2024-04-06T12:28:05.274888+0000","last_active":"2024-04-06T12:28:06.678844+0000","last_peered":"2024-04-06T12:28:06.678844+0000","last_clean":"2024-04-06T12:28:06.678844+0000","last_became_active":"2024-04-06T12:28:05.274629+0000","last_became_peered":"2024-04-06T12:28:05.274629+0000","last_unstale":"2024-04-06T12:28:06.678844+0000","last_undegraded":"2024-04-06T12:28:06.678844+0000","last_fullsized":"2024-04-06T12:28:06.678844+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:03:21.619567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317380+0000","last_change":"2024-04-06T12:28:05.695277+0000","last_active":"2024-04-06T12:28:07.317380+0000","last_peered":"2024-04-06T12:28:07.317380+0000","last_clean":"2024-04-06T12:28:07.317380+0000","last_became_active":"2024-04-06T12:28:05.695002+0000","last_became_peered":"2024-04-06T12:28:05.695002+0000","last_unstale":"2024-04-06T12:28:07.317380+0000","last_undegraded":"2024-04-06T12:28:07.317380+0000","last_fullsized":"2024-04-06T12:28:07.317380+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:23:48.291159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.678659+0000","last_change":"2024-04-06T12:28:05.272012+0000","last_active":"2024-04-06T12:28:06.678659+0000","last_peered":"2024-04-06T12:28:06.678659+0000","last_clean":"2024-04-06T12:28:06.678659+0000","last_became_active":"2024-04-06T12:28:05.271733+0000","last_became_peered":"2024-04-06T12:28:05.271733+0000","last_unstale":"2024-04-06T12:28:06.678659+0000","last_undegraded":"2024-04-06T12:28:06.678659+0000","last_fullsized":"2024-04-06T12:28:06.678659+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:27:20.824420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.695335+0000","last_change":"2024-04-06T12:28:05.688208+0000","last_active":"2024-04-06T12:28:05.695335+0000","last_peered":"2024-04-06T12:28:05.695335+0000","last_clean":"2024-04-06T12:28:05.695335+0000","last_became_active":"2024-04-06T12:28:05.687846+0000","last_became_peered":"2024-04-06T12:28:05.687846+0000","last_unstale":"2024-04-06T12:28:05.695335+0000","last_undegraded":"2024-04-06T12:28:05.695335+0000","last_fullsized":"2024-04-06T12:28:05.695335+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:26:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.678647+0000","last_change":"2024-04-06T12:28:05.272679+0000","last_active":"2024-04-06T12:28:06.678647+0000","last_peered":"2024-04-06T12:28:06.678647+0000","last_clean":"2024-04-06T12:28:06.678647+0000","last_became_active":"2024-04-06T12:28:05.272331+0000","last_became_peered":"2024-04-06T12:28:05.272331+0000","last_unstale":"2024-04-06T12:28:06.678647+0000","last_undegraded":"2024-04-06T12:28:06.678647+0000","last_fullsized":"2024-04-06T12:28:06.678647+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:13:41.048698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679189+0000","last_change":"2024-04-06T12:28:05.693228+0000","last_active":"2024-04-06T12:28:06.679189+0000","last_peered":"2024-04-06T12:28:06.679189+0000","last_clean":"2024-04-06T12:28:06.679189+0000","last_became_active":"2024-04-06T12:28:05.690502+0000","last_became_peered":"2024-04-06T12:28:05.690502+0000","last_unstale":"2024-04-06T12:28:06.679189+0000","last_undegraded":"2024-04-06T12:28:06.679189+0000","last_fullsized":"2024-04-06T12:28:06.679189+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:26:18.833810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"97'2","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.700399+0000","last_change":"2024-04-06T12:28:05.275974+0000","last_active":"2024-04-06T12:28:05.700399+0000","last_peered":"2024-04-06T12:28:05.700399+0000","last_clean":"2024-04-06T12:28:05.700399+0000","last_became_active":"2024-04-06T12:28:05.275692+0000","last_became_peered":"2024-04-06T12:28:05.275692+0000","last_unstale":"2024-04-06T12:28:05.700399+0000","last_undegraded":"2024-04-06T12:28:05.700399+0000","last_fullsized":"2024-04-06T12:28:05.700399+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:03:57.999753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.698649+0000","last_change":"2024-04-06T12:28:05.694333+0000","last_active":"2024-04-06T12:28:05.698649+0000","last_peered":"2024-04-06T12:28:05.698649+0000","last_clean":"2024-04-06T12:28:05.698649+0000","last_became_active":"2024-04-06T12:28:05.694053+0000","last_became_peered":"2024-04-06T12:28:05.694053+0000","last_unstale":"2024-04-06T12:28:05.698649+0000","last_undegraded":"2024-04-06T12:28:05.698649+0000","last_fullsized":"2024-04-06T12:28:05.698649+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:01:21.360236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317376+0000","last_change":"2024-04-06T12:28:05.281755+0000","last_active":"2024-04-06T12:28:07.317376+0000","last_peered":"2024-04-06T12:28:07.317376+0000","last_clean":"2024-04-06T12:28:07.317376+0000","last_became_active":"2024-04-06T12:28:05.281346+0000","last_became_peered":"2024-04-06T12:28:05.281346+0000","last_unstale":"2024-04-06T12:28:07.317376+0000","last_undegraded":"2024-04-06T12:28:07.317376+0000","last_fullsized":"2024-04-06T12:28:07.317376+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:13:50.117433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.680256+0000","last_change":"2024-04-06T12:28:05.701311+0000","last_active":"2024-04-06T12:28:06.680256+0000","last_peered":"2024-04-06T12:28:06.680256+0000","last_clean":"2024-04-06T12:28:06.680256+0000","last_became_active":"2024-04-06T12:28:05.700946+0000","last_became_peered":"2024-04-06T12:28:05.700946+0000","last_unstale":"2024-04-06T12:28:06.680256+0000","last_undegraded":"2024-04-06T12:28:06.680256+0000","last_fullsized":"2024-04-06T12:28:06.680256+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:58:23.827592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.680144+0000","last_change":"2024-04-06T12:28:05.270955+0000","last_active":"2024-04-06T12:28:06.680144+0000","last_peered":"2024-04-06T12:28:06.680144+0000","last_clean":"2024-04-06T12:28:06.680144+0000","last_became_active":"2024-04-06T12:28:05.270687+0000","last_became_peered":"2024-04-06T12:28:05.270687+0000","last_unstale":"2024-04-06T12:28:06.680144+0000","last_undegraded":"2024-04-06T12:28:06.680144+0000","last_fullsized":"2024-04-06T12:28:06.680144+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:17:02.567576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.317125+0000","last_change":"2024-04-06T12:28:05.684328+0000","last_active":"2024-04-06T12:28:07.317125+0000","last_peered":"2024-04-06T12:28:07.317125+0000","last_clean":"2024-04-06T12:28:07.317125+0000","last_became_active":"2024-04-06T12:28:05.684086+0000","last_became_peered":"2024-04-06T12:28:05.684086+0000","last_unstale":"2024-04-06T12:28:07.317125+0000","last_undegraded":"2024-04-06T12:28:07.317125+0000","last_fullsized":"2024-04-06T12:28:07.317125+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:47:00.031681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679460+0000","last_change":"2024-04-06T12:28:05.272141+0000","last_active":"2024-04-06T12:28:06.679460+0000","last_peered":"2024-04-06T12:28:06.679460+0000","last_clean":"2024-04-06T12:28:06.679460+0000","last_became_active":"2024-04-06T12:28:05.271870+0000","last_became_peered":"2024-04-06T12:28:05.271870+0000","last_unstale":"2024-04-06T12:28:06.679460+0000","last_undegraded":"2024-04-06T12:28:06.679460+0000","last_fullsized":"2024-04-06T12:28:06.679460+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:21:00.111250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.698507+0000","last_change":"2024-04-06T12:28:05.694128+0000","last_active":"2024-04-06T12:28:05.698507+0000","last_peered":"2024-04-06T12:28:05.698507+0000","last_clean":"2024-04-06T12:28:05.698507+0000","last_became_active":"2024-04-06T12:28:05.693888+0000","last_became_peered":"2024-04-06T12:28:05.693888+0000","last_unstale":"2024-04-06T12:28:05.698507+0000","last_undegraded":"2024-04-06T12:28:05.698507+0000","last_fullsized":"2024-04-06T12:28:05.698507+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:27:56.179062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.678486+0000","last_change":"2024-04-06T12:28:05.266184+0000","last_active":"2024-04-06T12:28:05.678486+0000","last_peered":"2024-04-06T12:28:05.678486+0000","last_clean":"2024-04-06T12:28:05.678486+0000","last_became_active":"2024-04-06T12:28:05.265916+0000","last_became_peered":"2024-04-06T12:28:05.265916+0000","last_unstale":"2024-04-06T12:28:05.678486+0000","last_undegraded":"2024-04-06T12:28:05.678486+0000","last_fullsized":"2024-04-06T12:28:05.678486+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:25:09.223040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.695908+0000","last_change":"2024-04-06T12:28:05.690680+0000","last_active":"2024-04-06T12:28:05.695908+0000","last_peered":"2024-04-06T12:28:05.695908+0000","last_clean":"2024-04-06T12:28:05.695908+0000","last_became_active":"2024-04-06T12:28:05.690418+0000","last_became_peered":"2024-04-06T12:28:05.690418+0000","last_unstale":"2024-04-06T12:28:05.695908+0000","last_undegraded":"2024-04-06T12:28:05.695908+0000","last_fullsized":"2024-04-06T12:28:05.695908+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:47:54.803772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:07.553272+0000","last_change":"2024-04-06T12:28:05.274049+0000","last_active":"2024-04-06T12:28:07.553272+0000","last_peered":"2024-04-06T12:28:07.553272+0000","last_clean":"2024-04-06T12:28:07.553272+0000","last_became_active":"2024-04-06T12:28:05.273801+0000","last_became_peered":"2024-04-06T12:28:05.273801+0000","last_unstale":"2024-04-06T12:28:07.553272+0000","last_undegraded":"2024-04-06T12:28:07.553272+0000","last_fullsized":"2024-04-06T12:28:07.553272+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:10.453242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.770653+0000","last_change":"2024-04-06T12:28:05.699523+0000","last_active":"2024-04-06T12:28:06.770653+0000","last_peered":"2024-04-06T12:28:06.770653+0000","last_clean":"2024-04-06T12:28:06.770653+0000","last_became_active":"2024-04-06T12:28:05.699255+0000","last_became_peered":"2024-04-06T12:28:05.699255+0000","last_unstale":"2024-04-06T12:28:06.770653+0000","last_undegraded":"2024-04-06T12:28:06.770653+0000","last_fullsized":"2024-04-06T12:28:06.770653+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:41:14.655391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-06T12:28:05.678761+0000","last_change":"2024-04-06T12:28:05.274836+0000","last_active":"2024-04-06T12:28:05.678761+0000","last_peered":"2024-04-06T12:28:05.678761+0000","last_clean":"2024-04-06T12:28:05.678761+0000","last_became_active":"2024-04-06T12:28:05.274546+0000","last_became_peered":"2024-04-06T12:28:05.274546+0000","last_unstale":"2024-04-06T12:28:05.678761+0000","last_undegraded":"2024-04-06T12:28:05.678761+0000","last_fullsized":"2024-04-06T12:28:05.678761+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:53:24.728327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.770656+0000","last_change":"2024-04-06T12:28:05.696466+0000","last_active":"2024-04-06T12:28:06.770656+0000","last_peered":"2024-04-06T12:28:06.770656+0000","last_clean":"2024-04-06T12:28:06.770656+0000","last_became_active":"2024-04-06T12:28:05.696189+0000","last_became_peered":"2024-04-06T12:28:05.696189+0000","last_unstale":"2024-04-06T12:28:06.770656+0000","last_undegraded":"2024-04-06T12:28:06.770656+0000","last_fullsized":"2024-04-06T12:28:06.770656+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:39:26.704304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.770323+0000","last_change":"2024-04-06T12:28:05.272062+0000","last_active":"2024-04-06T12:28:06.770323+0000","last_peered":"2024-04-06T12:28:06.770323+0000","last_clean":"2024-04-06T12:28:06.770323+0000","last_became_active":"2024-04-06T12:28:05.271804+0000","last_became_peered":"2024-04-06T12:28:05.271804+0000","last_unstale":"2024-04-06T12:28:06.770323+0000","last_undegraded":"2024-04-06T12:28:06.770323+0000","last_fullsized":"2024-04-06T12:28:06.770323+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:06:03.983801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679940+0000","last_change":"2024-04-06T12:28:05.697981+0000","last_active":"2024-04-06T12:28:06.679940+0000","last_peered":"2024-04-06T12:28:06.679940+0000","last_clean":"2024-04-06T12:28:06.679940+0000","last_became_active":"2024-04-06T12:28:05.697684+0000","last_became_peered":"2024-04-06T12:28:05.697684+0000","last_unstale":"2024-04-06T12:28:06.679940+0000","last_undegraded":"2024-04-06T12:28:06.679940+0000","last_fullsized":"2024-04-06T12:28:06.679940+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:05.247777+0000","last_clean_scrub_stamp":"2024-04-06T12:28:05.247777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:14.483577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679044+0000","last_change":"2024-04-06T12:28:05.276729+0000","last_active":"2024-04-06T12:28:06.679044+0000","last_peered":"2024-04-06T12:28:06.679044+0000","last_clean":"2024-04-06T12:28:06.679044+0000","last_became_active":"2024-04-06T12:28:05.274128+0000","last_became_peered":"2024-04-06T12:28:05.274128+0000","last_unstale":"2024-04-06T12:28:06.679044+0000","last_undegraded":"2024-04-06T12:28:06.679044+0000","last_fullsized":"2024-04-06T12:28:06.679044+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:04.242352+0000","last_clean_scrub_stamp":"2024-04-06T12:28:04.242352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:23:35.861872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"87'172","reported_seq":445,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-06T12:28:06.679455+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:28:06.679455+0000","last_peered":"2024-04-06T12:28:06.679455+0000","last_clean":"2024-04-06T12:28:06.679455+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:28:06.679455+0000","last_undegraded":"2024-04-06T12:28:06.679455+0000","last_fullsized":"2024-04-06T12:28:06.679455+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172,"ondisk_log_size":172,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575042,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59124,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312716,"statfs":{"total":96636764160,"available":96576221184,"internally_reserved":0,"allocated":1798144,"data_stored":1186132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.34300000000000003,"5min":0.219,"15min":0.219},"max":{"1min":0.68100000000000005,"5min":1.143,"15min":1.143},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54300000000000004,"15min":0.55200000000000005},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.61499999999999999,"5min":0.77100000000000002,"15min":0.78200000000000003},"last":0.372}]},{"osd":1,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.63600000000000001,"5min":0.77100000000000002,"15min":0.81399999999999995},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.55500000000000005,"15min":0.56899999999999995},"min":{"1min":0.36599999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68899999999999995,"5min":0.79400000000000004,"15min":0.79600000000000004},"last":0.56299999999999994}]},{"osd":2,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55200000000000005,"15min":0.55400000000000005},"min":{"1min":0.316,"5min":0.249,"15min":0.249},"max":{"1min":0.65500000000000003,"5min":0.78500000000000003,"15min":0.83299999999999996},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55900000000000005,"15min":0.57799999999999996},"min":{"1min":0.36699999999999999,"5min":0.34499999999999997,"15min":0.24199999999999999},"max":{"1min":0.66500000000000004,"5min":0.77600000000000002,"15min":0.82299999999999995},"last":0.39400000000000002}]},{"osd":3,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.57099999999999995,"15min":0.57199999999999995},"min":{"1min":0.39300000000000002,"5min":0.34799999999999998,"15min":0.34399999999999997},"max":{"1min":0.72399999999999998,"5min":1.357,"15min":1.357},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.57899999999999996,"15min":0.59199999999999997},"min":{"1min":0.38500000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.65900000000000003,"5min":0.81799999999999995,"15min":0.92000000000000004},"last":0.42899999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.29999999999999999,"5min":0.27100000000000002,"15min":0.25},"max":{"1min":0.63900000000000001,"5min":0.70399999999999996,"15min":0.83599999999999997},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.41199999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.66500000000000004,"5min":0.77700000000000002,"15min":1.0780000000000001},"last":0.55100000000000005}]},{"osd":5,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.48399999999999999,"15min":0.502},"min":{"1min":0.34300000000000003,"5min":0.29199999999999998,"15min":0.23799999999999999},"max":{"1min":0.60499999999999998,"5min":0.80100000000000005,"15min":0.98199999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54300000000000004,"15min":0.55800000000000005},"min":{"1min":0.39900000000000002,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.625,"5min":0.81599999999999995,"15min":0.93600000000000005},"last":0.61599999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.52400000000000002,"15min":0.54100000000000004},"min":{"1min":0.37,"5min":0.33700000000000002,"15min":0.28799999999999998},"max":{"1min":0.67700000000000005,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54900000000000004,"15min":0.56200000000000006},"min":{"1min":0.373,"5min":0.30399999999999999,"15min":0.26900000000000002},"max":{"1min":0.68500000000000005,"5min":0.79700000000000004,"15min":0.88500000000000001},"last":0.58599999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575041,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1196032,"data_stored":589667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.36699999999999999,"5min":0.26200000000000001,"15min":0.23000000000000001},"max":{"1min":0.77500000000000002,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52500000000000002,"15min":0.53300000000000003},"min":{"1min":0.374,"5min":0.30199999999999999,"15min":0.245},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.60599999999999998}]},{"osd":1,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.32700000000000001,"5min":0.32600000000000001,"15min":0.24399999999999999},"max":{"1min":0.78200000000000003,"5min":0.78700000000000003,"15min":0.84999999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.38400000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.83099999999999996},"last":0.56799999999999995}]},{"osd":2,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.39100000000000001,"5min":0.30499999999999999,"15min":0.29899999999999999},"max":{"1min":0.749,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.68999999999999995}]},{"osd":3,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.56899999999999995,"15min":0.58099999999999996},"min":{"1min":0.35099999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.745,"5min":0.79000000000000004,"15min":0.81200000000000006},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.60899999999999999,"15min":0.61899999999999999},"min":{"1min":0.51700000000000002,"5min":0.41099999999999998,"15min":0.374},"max":{"1min":0.79200000000000004,"5min":0.82799999999999996,"15min":0.872},"last":0.77600000000000002}]},{"osd":4,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.49099999999999999,"15min":0.496},"min":{"1min":0.28599999999999998,"5min":0.27500000000000002,"15min":0.27300000000000002},"max":{"1min":0.747,"5min":0.749,"15min":0.749},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.53000000000000003},"min":{"1min":0.34899999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.73199999999999998,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.70799999999999996}]},{"osd":5,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.47999999999999998,"15min":0.49199999999999999},"min":{"1min":0.34699999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71099999999999997,"5min":0.755,"15min":0.75800000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.50600000000000001,"15min":0.51300000000000001},"min":{"1min":0.32800000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72599999999999998,"5min":0.83699999999999997,"15min":0.92800000000000005},"last":0.80200000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53000000000000003,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70099999999999996,"5min":0.79800000000000004,"15min":0.81000000000000005},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55800000000000005,"15min":0.56100000000000005},"min":{"1min":0.41899999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80800000000000005,"5min":0.88900000000000001,"15min":0.96099999999999997},"last":0.84799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575040,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32304,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339536,"statfs":{"total":96636764160,"available":96603684864,"internally_reserved":0,"allocated":1204224,"data_stored":594835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.377,"15min":0.36199999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.47699999999999998,"5min":0.58999999999999997,"15min":0.71699999999999997},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39900000000000002,"15min":0.39600000000000002},"min":{"1min":0.253,"5min":0.215,"15min":0.215},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.316}]},{"osd":2,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.40500000000000003,"15min":0.39600000000000002},"min":{"1min":0.27200000000000002,"5min":0.23899999999999999,"15min":0.19400000000000001},"max":{"1min":0.57599999999999996,"5min":0.624,"15min":0.67600000000000005},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.432,"15min":0.42999999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.249},"max":{"1min":0.60699999999999998,"5min":0.63600000000000001,"15min":0.79000000000000004},"last":0.437}]},{"osd":3,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45600000000000002,"15min":0.44700000000000001},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67400000000000004,"5min":0.67800000000000005,"15min":0.84299999999999997},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.435,"5min":0.42599999999999999,"15min":0.41899999999999998},"min":{"1min":0.30499999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.56499999999999995,"5min":0.69599999999999995,"15min":0.86299999999999999},"last":0.38400000000000001}]},{"osd":4,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.623,"15min":0.63200000000000001},"min":{"1min":0.42599999999999999,"5min":0.40100000000000002,"15min":0.34799999999999998},"max":{"1min":0.71499999999999997,"5min":0.86799999999999999,"15min":6.3739999999999997},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.63100000000000001,"15min":0.622},"min":{"1min":0.45400000000000001,"5min":0.441,"15min":0.30399999999999999},"max":{"1min":0.84299999999999997,"5min":0.91400000000000003,"15min":1.0269999999999999},"last":0.70999999999999996}]},{"osd":5,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.624,"15min":0.61399999999999999},"min":{"1min":0.47599999999999998,"5min":0.42999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83999999999999997,"5min":0.91300000000000003,"15min":0.97699999999999998},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.628},"min":{"1min":0.442,"5min":0.43099999999999999,"15min":0.35099999999999998},"max":{"1min":0.78700000000000003,"5min":0.999,"15min":1.0089999999999999},"last":0.61699999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.623,"15min":0.61399999999999999},"min":{"1min":0.47699999999999998,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":0.76500000000000001,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.623,"5min":0.65300000000000002,"15min":0.64900000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.41799999999999998},"max":{"1min":0.83999999999999997,"5min":0.88700000000000001,"15min":1.151},"last":0.59299999999999997}]},{"osd":7,"last update":"Sat Apr 6 12:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.64800000000000002,"15min":0.65500000000000003},"min":{"1min":0.44600000000000001,"5min":0.41899999999999998,"15min":0.38800000000000001},"max":{"1min":0.81699999999999995,"5min":1.0329999999999999,"15min":6.3639999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.47299999999999998,"5min":0.40200000000000002,"15min":0.39300000000000002},"max":{"1min":0.82499999999999996,"5min":0.95599999999999996,"15min":1.073},"last":0.75700000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575040,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32864,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338976,"statfs":{"total":96636764160,"available":96603111424,"internally_reserved":0,"allocated":1789952,"data_stored":1185091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 12:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.38200000000000001,"15min":0.38100000000000001},"min":{"1min":0.222,"5min":0.19,"15min":0.19},"max":{"1min":0.59899999999999998,"5min":0.80000000000000004,"15min":0.89100000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.40400000000000003,"15min":0.40300000000000002},"min":{"1min":0.25700000000000001,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":0.57299999999999995,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.40600000000000003}]},{"osd":2,"last update":"Sat Apr 6 12:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40699999999999997,"15min":0.41099999999999998},"min":{"1min":0.255,"5min":0.23599999999999999,"15min":0.19700000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":1.8089999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.30599999999999999,"5min":0.249,"15min":0.23200000000000001},"max":{"1min":0.58299999999999996,"5min":1.4790000000000001,"15min":1.764},"last":0.372}]},{"osd":3,"last update":"Sat Apr 6 12:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.41499999999999998,"15min":0.41399999999999998},"min":{"1min":0.29099999999999998,"5min":0.252,"15min":0.21299999999999999},"max":{"1min":0.624,"5min":1.25,"15min":1.25},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45900000000000002,"15min":0.46200000000000002},"min":{"1min":0.34200000000000003,"5min":0.28499999999999998,"15min":0.249},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.92000000000000004},"last":0.433}]},{"osd":4,"last update":"Sat Apr 6 12:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.498,"5min":0.371,"15min":0.35699999999999998},"max":{"1min":0.73199999999999998,"5min":1.514,"15min":1.514},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.621,"15min":0.627},"min":{"1min":0.42099999999999999,"5min":0.39900000000000002,"15min":0.311},"max":{"1min":0.73799999999999999,"5min":1.202,"15min":1.202},"last":0.625}]},{"osd":5,"last update":"Sat Apr 6 12:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.621,"15min":0.622},"min":{"1min":0.38900000000000001,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.78900000000000003,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63900000000000001,"15min":0.64800000000000002},"min":{"1min":0.495,"5min":0.38800000000000001,"15min":0.315},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.53800000000000003}]},{"osd":6,"last update":"Sat Apr 6 12:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.64500000000000002,"15min":0.65100000000000002},"min":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.378},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.65400000000000003,"15min":0.66200000000000003},"min":{"1min":0.46500000000000002,"5min":0.437,"15min":0.38100000000000001},"max":{"1min":0.77500000000000002,"5min":1.613,"15min":1.613},"last":0.48899999999999999}]},{"osd":7,"last update":"Sat Apr 6 12:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.66000000000000003,"15min":0.66300000000000003},"min":{"1min":0.45300000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.78600000000000003,"5min":1.54,"15min":1.54},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.65600000000000003,"15min":0.65500000000000003},"min":{"1min":0.42199999999999999,"5min":0.38400000000000001,"15min":0.36699999999999999},"max":{"1min":0.72899999999999998,"5min":1.498,"15min":1.498},"last":0.57899999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575042,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50344,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321496,"statfs":{"total":96636764160,"available":96585211904,"internally_reserved":0,"allocated":1208320,"data_stored":595465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:27 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38600000000000001,"15min":0.38200000000000001},"min":{"1min":0.26800000000000002,"5min":0.214,"15min":0.21099999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":1.159},"last":0.441},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40799999999999997,"15min":0.40999999999999998},"min":{"1min":0.26000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":1.075},"last":0.48699999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:27:27 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.28000000000000003,"5min":0.26400000000000001,"15min":0.24299999999999999},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":3.6179999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42499999999999999,"15min":0.43099999999999999},"min":{"1min":0.29399999999999998,"5min":0.27100000000000002,"15min":0.23599999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":1.125},"last":0.32900000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:27:27 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.45900000000000002},"min":{"1min":0.316,"5min":0.30499999999999999,"15min":0.29699999999999999},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":3.661},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.438,"5min":0.44700000000000001,"15min":0.438},"min":{"1min":0.313,"5min":0.29499999999999998,"15min":0.27600000000000002},"max":{"1min":0.97099999999999997,"5min":1.0549999999999999,"15min":1.1910000000000001},"last":0.34999999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:27:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60599999999999998,"15min":0.61199999999999999},"min":{"1min":0.38600000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.2170000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61899999999999999,"15min":0.626},"min":{"1min":0.375,"5min":0.316,"15min":0.30499999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":1.101},"last":0.65100000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:27:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.64400000000000002,"15min":0.64600000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.32900000000000001},"max":{"1min":1.0369999999999999,"5min":1.6870000000000001,"15min":3.6920000000000002},"last":0.628},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.624,"15min":0.63},"min":{"1min":0.39900000000000002,"5min":0.31,"15min":0.31},"max":{"1min":1.9570000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.47799999999999998}]},{"osd":6,"last update":"Sat Apr 6 12:27:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.63100000000000001,"15min":0.63500000000000001},"min":{"1min":0.436,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.66400000000000003,"15min":0.66300000000000003},"min":{"1min":0.42499999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0760000000000001,"5min":1.728,"15min":3.7490000000000001},"last":0.51000000000000001}]},{"osd":7,"last update":"Sat Apr 6 12:27:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.64000000000000001,"15min":0.64800000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.35599999999999998},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":1.0469999999999999},"last":0.62},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.46899999999999997,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0049999999999999,"5min":1.758,"15min":3.7200000000000002},"last":0.54600000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575041,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1204224,"data_stored":596324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.23599999999999999,"5min":0.21099999999999999,"15min":0.191},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.379,"15min":0.38100000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.19600000000000001},"max":{"1min":0.56100000000000005,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.30399999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.38100000000000001,"15min":0.38},"min":{"1min":0.32000000000000001,"5min":0.22700000000000001,"15min":0.215},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.432,"5min":0.40600000000000003,"15min":0.41299999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.247},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.33300000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.36199999999999999,"5min":0.27200000000000002,"15min":0.25700000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.379},{"interface":"front","average":{"1min":0.503,"5min":0.47399999999999998,"15min":0.47499999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.29899999999999999},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.36699999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.40300000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81100000000000005,"5min":0.88500000000000001,"15min":0.95999999999999996},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.60299999999999998,"15min":0.622},"min":{"1min":0.41099999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90100000000000002,"5min":0.90300000000000002,"15min":0.93500000000000005},"last":0.69799999999999995}]},{"osd":5,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.64200000000000002,"15min":0.65900000000000003},"min":{"1min":0.45600000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.96699999999999997},"last":0.746},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65300000000000002,"15min":0.65800000000000003},"min":{"1min":0.44400000000000001,"5min":0.38200000000000001,"15min":0.36899999999999999},"max":{"1min":0.86699999999999999,"5min":0.94499999999999995,"15min":0.95599999999999996},"last":0.70799999999999996}]},{"osd":6,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.60899999999999999,"15min":0.622},"min":{"1min":0.38,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.65100000000000002,"15min":0.65800000000000003},"min":{"1min":0.47599999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.60399999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.63500000000000001,"15min":0.64400000000000002},"min":{"1min":0.47299999999999998,"5min":0.40400000000000003,"15min":0.33300000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.65100000000000002,"15min":0.66300000000000003},"min":{"1min":0.47999999999999998,"5min":0.39500000000000002,"15min":0.373},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.82599999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575041,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32308,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339532,"statfs":{"total":96636764160,"available":96603680768,"internally_reserved":0,"allocated":1220608,"data_stored":597410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56399999999999995,"15min":0.56299999999999994},"min":{"1min":0.371,"5min":0.29399999999999998,"15min":0.23899999999999999},"max":{"1min":0.72499999999999998,"5min":0.80800000000000005,"15min":0.82999999999999996},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.55900000000000005},"min":{"1min":0.33800000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.82199999999999995},"last":0.40899999999999997}]},{"osd":1,"last update":"Sat Apr 6 12:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59299999999999997,"15min":0.622},"min":{"1min":0.45100000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.76200000000000001,"5min":0.76900000000000002,"15min":4.3479999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58899999999999997,"15min":0.60899999999999999},"min":{"1min":0.36799999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":4.3319999999999999},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55800000000000005,"15min":0.56699999999999995},"min":{"1min":0.32700000000000001,"5min":0.313,"15min":0.311},"max":{"1min":0.73499999999999999,"5min":0.76100000000000001,"15min":0.78400000000000003},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.40100000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.71699999999999997,"5min":0.74299999999999999,"15min":0.82199999999999995},"last":0.379}]},{"osd":3,"last update":"Sat Apr 6 12:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55600000000000005,"15min":0.57599999999999996},"min":{"1min":0.373,"5min":0.36399999999999999,"15min":0.34899999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":4.1950000000000003},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.59099999999999997,"15min":0.60099999999999998},"min":{"1min":0.434,"5min":0.40500000000000003,"15min":0.36099999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":1.2330000000000001},"last":0.42099999999999999}]},{"osd":5,"last update":"Sat Apr 6 12:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.45900000000000002,"15min":0.48299999999999998},"min":{"1min":0.315,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65500000000000003,"5min":0.76300000000000001,"15min":1.196},"last":0.315},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.49299999999999999,"15min":0.499},"min":{"1min":0.34399999999999997,"5min":0.33500000000000002,"15min":0.312},"max":{"1min":0.67500000000000004,"5min":0.74299999999999999,"15min":0.77000000000000002},"last":0.29499999999999998}]},{"osd":6,"last update":"Sat Apr 6 12:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.49299999999999999,"15min":0.504},"min":{"1min":0.34200000000000003,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.67200000000000004,"5min":0.754,"15min":1.022},"last":0.307},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52000000000000002,"15min":0.53800000000000003},"min":{"1min":0.39900000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69399999999999995,"5min":0.71799999999999997,"15min":0.872},"last":0.35299999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51700000000000002,"15min":0.53500000000000003},"min":{"1min":0.38400000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.73799999999999999,"5min":0.79300000000000004,"15min":0.91500000000000004},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54900000000000004,"15min":0.56999999999999995},"min":{"1min":0.38700000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.749,"5min":0.749,"15min":1.0589999999999999},"last":0.36699999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575041,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46236,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325604,"statfs":{"total":96636764160,"available":96589418496,"internally_reserved":0,"allocated":1187840,"data_stored":588137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.36799999999999999,"5min":0.32700000000000001,"15min":0.24199999999999999},"max":{"1min":0.75900000000000001,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.29499999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.748,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.41899999999999998}]},{"osd":1,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.30399999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55200000000000005,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.34499999999999997,"15min":0.30599999999999999},"max":{"1min":0.73599999999999999,"5min":0.74299999999999999,"15min":0.76200000000000001},"last":0.45700000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56999999999999995,"15min":0.56200000000000006},"min":{"1min":0.33200000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.72199999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.40999999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72099999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54100000000000004}]},{"osd":3,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.33900000000000002,"5min":0.32500000000000001,"15min":0.31},"max":{"1min":0.77800000000000002,"5min":0.874,"15min":0.874},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.436,"5min":0.377,"15min":0.32300000000000001},"max":{"1min":0.73099999999999998,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.58099999999999996}]},{"osd":4,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.46800000000000003,"15min":0.47099999999999997},"min":{"1min":0.32100000000000001,"5min":0.315,"15min":0.28299999999999997},"max":{"1min":0.755,"5min":0.755,"15min":0.81200000000000006},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54000000000000004,"15min":0.53300000000000003},"min":{"1min":0.42599999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73699999999999999,"5min":0.80000000000000004,"15min":0.91800000000000004},"last":0.64900000000000002}]},{"osd":6,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49099999999999999,"15min":0.48599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.28000000000000003},"max":{"1min":0.63800000000000001,"5min":0.73899999999999999,"15min":0.88700000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.496,"15min":0.49299999999999999},"min":{"1min":0.33100000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64800000000000002,"5min":0.73199999999999998,"15min":0.83299999999999996},"last":0.627}]},{"osd":7,"last update":"Sat Apr 6 12:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.252},"max":{"1min":0.68999999999999995,"5min":0.78600000000000003,"15min":0.79100000000000004},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60999999999999999,"15min":0.60199999999999998},"min":{"1min":0.44800000000000001,"5min":0.41399999999999998,"15min":0.35199999999999998},"max":{"1min":0.72499999999999998,"5min":0.85299999999999998,"15min":0.85599999999999998},"last":0.72699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:28:10.182 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-06T12:28:10.182 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-06T12:28:10.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.348+0000 7fa71e737700 1 Processor -- start 2024-04-06T12:28:10.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.348+0000 7fa71e737700 1 -- start start 2024-04-06T12:28:10.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.349+0000 7fa71e737700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa7180cea80 0x7fa7180cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:10.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.349+0000 7fa71e737700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7180cdae0 0x7fa71806ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:10.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.349+0000 7fa71e737700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa71815a830 con 0x7fa7180aec70 2024-04-06T12:28:10.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.349+0000 7fa71e737700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7180cd920 con 0x7fa7180cea80 2024-04-06T12:28:10.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.349+0000 7fa71e737700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7180cc050 con 0x7fa7180cdae0 2024-04-06T12:28:10.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.349+0000 7fa7177fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa7180cea80 0x7fa7180cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:10.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.349+0000 7fa717fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa7180aec70 0x7fa7180ce370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44606/0 (socket says 172.21.15.189:44606) 2024-04-06T12:28:10.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.349+0000 7fa717fff700 1 -- 172.21.15.189:0/720788374 learned_addr learned my addr 172.21.15.189:0/720788374 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:10.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.349+0000 7fa71ccd4700 1 --2- 172.21.15.189:0/720788374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7180cdae0 0x7fa71806ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:10.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.350+0000 7fa71ccd4700 1 -- 172.21.15.189:0/720788374 >> v2:172.21.15.189:3300/0 conn(0x7fa7180cea80 msgr2=0x7fa7180cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:10.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.350+0000 7fa71ccd4700 1 --2- 172.21.15.189:0/720788374 >> v2:172.21.15.189:3300/0 conn(0x7fa7180cea80 0x7fa7180cd3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:10.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.350+0000 7fa71ccd4700 1 -- 172.21.15.189:0/720788374 >> v1:172.21.15.189:6789/0 conn(0x7fa7180aec70 legacy=0x7fa7180ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:10.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.350+0000 7fa71ccd4700 1 -- 172.21.15.189:0/720788374 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7180666f0 con 0x7fa7180cdae0 2024-04-06T12:28:10.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.350+0000 7fa7177fe700 1 --2- 172.21.15.189:0/720788374 >> v2:172.21.15.189:3300/0 conn(0x7fa7180cea80 0x7fa7180cd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:28:10.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.350+0000 7fa716ffd700 1 -- 172.21.15.189:0/720788374 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993465866 0 0) 0x7fa71815a830 con 0x7fa7180aec70 2024-04-06T12:28:10.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.350+0000 7fa71ccd4700 1 --2- 172.21.15.189:0/720788374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7180cdae0 0x7fa71806ceb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa710009d90 tx=0x7fa71000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78bec16f04217b2f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:10.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.350+0000 7fa716ffd700 1 -- 172.21.15.189:0/720788374 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71000e070 con 0x7fa7180cdae0 2024-04-06T12:28:10.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.350+0000 7fa716ffd700 1 -- 172.21.15.189:0/720788374 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa71000bb20 con 0x7fa7180cdae0 2024-04-06T12:28:10.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.350+0000 7fa716ffd700 1 -- 172.21.15.189:0/720788374 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa710012430 con 0x7fa7180cdae0 2024-04-06T12:28:10.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.351+0000 7fa71e737700 1 -- 172.21.15.189:0/720788374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7180cdae0 msgr2=0x7fa71806ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:10.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.351+0000 7fa71e737700 1 --2- 172.21.15.189:0/720788374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7180cdae0 0x7fa71806ceb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa710009d90 tx=0x7fa71000b330 comp rx=0 tx=0).stop 2024-04-06T12:28:10.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.351+0000 7fa71e737700 1 -- 172.21.15.189:0/720788374 shutdown_connections 2024-04-06T12:28:10.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.351+0000 7fa71e737700 1 --2- 172.21.15.189:0/720788374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7180cdae0 0x7fa71806ceb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:10.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.351+0000 7fa71e737700 1 --2- 172.21.15.189:0/720788374 >> v2:172.21.15.189:3300/0 conn(0x7fa7180cea80 0x7fa7180cd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:10.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.351+0000 7fa71e737700 1 -- 172.21.15.189:0/720788374 >> 172.21.15.189:0/720788374 conn(0x7fa7180cc290 msgr2=0x7fa7180cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:10.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.351+0000 7fa71e737700 1 -- 172.21.15.189:0/720788374 shutdown_connections 2024-04-06T12:28:10.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.351+0000 7fa71e737700 1 -- 172.21.15.189:0/720788374 wait complete. 2024-04-06T12:28:10.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.352+0000 7fa71e737700 1 Processor -- start 2024-04-06T12:28:10.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.352+0000 7fa71e737700 1 -- start start 2024-04-06T12:28:10.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.353+0000 7fa71e737700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7180aec70 0x7fa7180c5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:10.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.353+0000 7fa71e737700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7180cdae0 0x7fa7180c8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:10.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.353+0000 7fa71e737700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7180cea80 0x7fa7180c9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:10.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.353+0000 7fa71e737700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7180c5860 con 0x7fa7180cea80 2024-04-06T12:28:10.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.353+0000 7fa71e737700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7180c5b40 con 0x7fa7180cdae0 2024-04-06T12:28:10.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.353+0000 7fa71e737700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7180c5e20 con 0x7fa7180aec70 2024-04-06T12:28:10.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.353+0000 7fa71ccd4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7180cea80 0x7fa7180c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:10.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.353+0000 7fa71ccd4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7180cea80 0x7fa7180c9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52230/0 (socket says 172.21.15.189:52230) 2024-04-06T12:28:10.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.353+0000 7fa71ccd4700 1 -- 172.21.15.189:0/3970300700 learned_addr learned my addr 172.21.15.189:0/3970300700 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:10.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.354+0000 7fa7177fe700 1 --2- 172.21.15.189:0/3970300700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7180cdae0 0x7fa7180c8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:10.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.354+0000 7fa717fff700 1 --2- 172.21.15.189:0/3970300700 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7180aec70 0x7fa7180c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:10.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.354+0000 7fa71ccd4700 1 -- 172.21.15.189:0/3970300700 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7180aec70 msgr2=0x7fa7180c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:10.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.354+0000 7fa71ccd4700 1 --2- 172.21.15.189:0/3970300700 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7180aec70 0x7fa7180c5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:10.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.354+0000 7fa71ccd4700 1 -- 172.21.15.189:0/3970300700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7180cdae0 msgr2=0x7fa7180c8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:10.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.354+0000 7fa71ccd4700 1 --2- 172.21.15.189:0/3970300700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7180cdae0 0x7fa7180c8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:10.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.354+0000 7fa71ccd4700 1 -- 172.21.15.189:0/3970300700 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7080094a0 con 0x7fa7180cea80 2024-04-06T12:28:10.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.354+0000 7fa7177fe700 1 --2- 172.21.15.189:0/3970300700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7180cdae0 0x7fa7180c8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:10.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.354+0000 7fa71ccd4700 1 --2- 172.21.15.189:0/3970300700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7180cea80 0x7fa7180c9060 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fa710014040 tx=0x7fa710004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:10.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.354+0000 7fa714ff9700 1 -- 172.21.15.189:0/3970300700 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa710020070 con 0x7fa7180cea80 2024-04-06T12:28:10.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.354+0000 7fa714ff9700 1 -- 172.21.15.189:0/3970300700 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7100098f0 con 0x7fa7180cea80 2024-04-06T12:28:10.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.354+0000 7fa714ff9700 1 -- 172.21.15.189:0/3970300700 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa710012430 con 0x7fa7180cea80 2024-04-06T12:28:10.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.354+0000 7fa71e737700 1 -- 172.21.15.189:0/3970300700 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa710009a70 con 0x7fa7180cea80 2024-04-06T12:28:10.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.354+0000 7fa71e737700 1 -- 172.21.15.189:0/3970300700 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7180c63e0 con 0x7fa7180cea80 2024-04-06T12:28:10.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.355+0000 7fa71e737700 1 -- 172.21.15.189:0/3970300700 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa718074990 con 0x7fa7180cea80 2024-04-06T12:28:10.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.355+0000 7fa714ff9700 1 -- 172.21.15.189:0/3970300700 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fa710024070 con 0x7fa7180cea80 2024-04-06T12:28:10.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.356+0000 7fa714ff9700 1 --2- 172.21.15.189:0/3970300700 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa70406adb0 0x7fa70406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:10.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.356+0000 7fa714ff9700 1 -- 172.21.15.189:0/3970300700 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa71008c6b0 con 0x7fa7180cea80 2024-04-06T12:28:10.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.356+0000 7fa717fff700 1 --2- 172.21.15.189:0/3970300700 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa70406adb0 0x7fa70406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:10.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.356+0000 7fa717fff700 1 --2- 172.21.15.189:0/3970300700 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa70406adb0 0x7fa70406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa70c000f60 tx=0x7fa70c007c00 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:10.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.360+0000 7fa714ff9700 1 -- 172.21.15.189:0/3970300700 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa7100576e0 con 0x7fa7180cea80 2024-04-06T12:28:10.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.636+0000 7fa71e737700 1 -- 172.21.15.189:0/3970300700 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa7180cd2e0 con 0x7fa7180cea80 2024-04-06T12:28:10.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.640+0000 7fa714ff9700 1 -- 172.21.15.189:0/3970300700 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+0 (secure 0 0 0) 0x7fa71005ad00 con 0x7fa7180cea80 2024-04-06T12:28:10.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.642+0000 7fa71e737700 1 -- 172.21.15.189:0/3970300700 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa70406adb0 msgr2=0x7fa70406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:10.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.642+0000 7fa71e737700 1 --2- 172.21.15.189:0/3970300700 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa70406adb0 0x7fa70406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa70c000f60 tx=0x7fa70c007c00 comp rx=0 tx=0).stop 2024-04-06T12:28:10.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.642+0000 7fa71e737700 1 -- 172.21.15.189:0/3970300700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7180cea80 msgr2=0x7fa7180c9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:10.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.642+0000 7fa71e737700 1 --2- 172.21.15.189:0/3970300700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7180cea80 0x7fa7180c9060 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fa710014040 tx=0x7fa710004ad0 comp rx=0 tx=0).stop 2024-04-06T12:28:10.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.642+0000 7fa71e737700 1 -- 172.21.15.189:0/3970300700 shutdown_connections 2024-04-06T12:28:10.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.642+0000 7fa71e737700 1 --2- 172.21.15.189:0/3970300700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7180cea80 0x7fa7180c9060 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:10.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.642+0000 7fa71e737700 1 --2- 172.21.15.189:0/3970300700 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7180aec70 0x7fa7180c5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:10.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.642+0000 7fa71e737700 1 --2- 172.21.15.189:0/3970300700 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa70406adb0 0x7fa70406d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:10.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.642+0000 7fa71e737700 1 --2- 172.21.15.189:0/3970300700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7180cdae0 0x7fa7180c8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:10.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.642+0000 7fa71e737700 1 -- 172.21.15.189:0/3970300700 >> 172.21.15.189:0/3970300700 conn(0x7fa7180cc290 msgr2=0x7fa7180b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:10.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.643+0000 7fa71e737700 1 -- 172.21.15.189:0/3970300700 shutdown_connections 2024-04-06T12:28:10.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.643+0000 7fa71e737700 1 -- 172.21.15.189:0/3970300700 wait complete. 2024-04-06T12:28:10.642 INFO:teuthology.orchestra.run.smithi189.stderr:updated caps for client.0 2024-04-06T12:28:10.653 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:28:10.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.777+0000 7f0330574700 1 Processor -- start 2024-04-06T12:28:10.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.778+0000 7f0330574700 1 -- start start 2024-04-06T12:28:10.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.778+0000 7f0330574700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f03280b4660 0x7f03280b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:10.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.778+0000 7f0330574700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f03280b2fc0 0x7f03280b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:10.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.778+0000 7f0330574700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03280b5030 con 0x7f032815a550 2024-04-06T12:28:10.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.779+0000 7f0330574700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03280cdbe0 con 0x7f03280b2fc0 2024-04-06T12:28:10.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.779+0000 7f0330574700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f032815a940 con 0x7f03280b4660 2024-04-06T12:28:10.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.779+0000 7f032eb11700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f03280b2fc0 0x7f03280b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:10.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.779+0000 7f032e310700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f032815a550 0x7f03280cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44612/0 (socket says 172.21.15.189:44612) 2024-04-06T12:28:10.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.779+0000 7f032eb11700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f03280b2fc0 0x7f03280b33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52232/0 (socket says 172.21.15.189:52232) 2024-04-06T12:28:10.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.779+0000 7f032e310700 1 -- 172.21.15.189:0/722509782 learned_addr learned my addr 172.21.15.189:0/722509782 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:10.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.779+0000 7f032db0f700 1 --2- 172.21.15.189:0/722509782 >> v2:172.21.15.192:3300/0 conn(0x7f03280b4660 0x7f03280b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:10.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.779+0000 7f032eb11700 1 -- 172.21.15.189:0/722509782 >> v2:172.21.15.192:3300/0 conn(0x7f03280b4660 msgr2=0x7f03280b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:10.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.779+0000 7f032eb11700 1 --2- 172.21.15.189:0/722509782 >> v2:172.21.15.192:3300/0 conn(0x7f03280b4660 0x7f03280b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:10.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.779+0000 7f032eb11700 1 -- 172.21.15.189:0/722509782 >> v1:172.21.15.189:6789/0 conn(0x7f032815a550 legacy=0x7f03280cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:10.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.779+0000 7f032eb11700 1 -- 172.21.15.189:0/722509782 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032815ac00 con 0x7f03280b2fc0 2024-04-06T12:28:10.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.779+0000 7f032d30e700 1 -- 172.21.15.189:0/722509782 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998423764 0 0) 0x7f03280b5030 con 0x7f032815a550 2024-04-06T12:28:10.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.779+0000 7f032eb11700 1 --2- 172.21.15.189:0/722509782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03280b2fc0 0x7f03280b33d0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f03280b2d40 tx=0x7f032000d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef68d81f5aea5f51 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:10.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.779+0000 7f032d30e700 1 -- 172.21.15.189:0/722509782 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0320010070 con 0x7f03280b2fc0 2024-04-06T12:28:10.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.779+0000 7f032d30e700 1 -- 172.21.15.189:0/722509782 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032000b2e0 con 0x7f03280b2fc0 2024-04-06T12:28:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.779+0000 7f032d30e700 1 -- 172.21.15.189:0/722509782 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032000dcc0 con 0x7f03280b2fc0 2024-04-06T12:28:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.780+0000 7f0330574700 1 -- 172.21.15.189:0/722509782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03280b2fc0 msgr2=0x7f03280b33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.780+0000 7f0330574700 1 --2- 172.21.15.189:0/722509782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03280b2fc0 0x7f03280b33d0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f03280b2d40 tx=0x7f032000d330 comp rx=0 tx=0).stop 2024-04-06T12:28:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.780+0000 7f0330574700 1 -- 172.21.15.189:0/722509782 shutdown_connections 2024-04-06T12:28:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.780+0000 7f0330574700 1 --2- 172.21.15.189:0/722509782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03280b2fc0 0x7f03280b33d0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.780+0000 7f0330574700 1 --2- 172.21.15.189:0/722509782 >> v2:172.21.15.192:3300/0 conn(0x7f03280b4660 0x7f03280b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.780+0000 7f0330574700 1 -- 172.21.15.189:0/722509782 >> 172.21.15.189:0/722509782 conn(0x7f03280d00b0 msgr2=0x7f03280d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.780+0000 7f0330574700 1 -- 172.21.15.189:0/722509782 shutdown_connections 2024-04-06T12:28:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.780+0000 7f0330574700 1 -- 172.21.15.189:0/722509782 wait complete. 2024-04-06T12:28:10.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.781+0000 7f0330574700 1 Processor -- start 2024-04-06T12:28:10.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.781+0000 7f0330574700 1 -- start start 2024-04-06T12:28:10.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.782+0000 7f0330574700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03280b2fc0 0x7f03281f2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:10.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.782+0000 7f0330574700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03280b4660 0x7f03280ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:10.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.782+0000 7f0330574700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03280aebe0 0x7f0328158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:10.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.782+0000 7f0330574700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03280af070 con 0x7f03280aebe0 2024-04-06T12:28:10.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.782+0000 7f0330574700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0328155fa0 con 0x7f03280b4660 2024-04-06T12:28:10.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.782+0000 7f0330574700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03280af260 con 0x7f03280b2fc0 2024-04-06T12:28:10.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.782+0000 7f032eb11700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03280aebe0 0x7f0328158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:10.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.782+0000 7f032eb11700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03280aebe0 0x7f0328158b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52244/0 (socket says 172.21.15.189:52244) 2024-04-06T12:28:10.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.782+0000 7f032eb11700 1 -- 172.21.15.189:0/2363665038 learned_addr learned my addr 172.21.15.189:0/2363665038 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:10.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.782+0000 7f032db0f700 1 --2- 172.21.15.189:0/2363665038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03280b4660 0x7f03280ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:10.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.782+0000 7f032e310700 1 --2- 172.21.15.189:0/2363665038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03280b2fc0 0x7f03281f2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:10.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.782+0000 7f032eb11700 1 -- 172.21.15.189:0/2363665038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03280b2fc0 msgr2=0x7f03281f2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:10.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.782+0000 7f032eb11700 1 --2- 172.21.15.189:0/2363665038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03280b2fc0 0x7f03281f2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:10.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.782+0000 7f032eb11700 1 -- 172.21.15.189:0/2363665038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03280b4660 msgr2=0x7f03280ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:10.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.782+0000 7f032eb11700 1 --2- 172.21.15.189:0/2363665038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03280b4660 0x7f03280ae6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:10.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.782+0000 7f032eb11700 1 -- 172.21.15.189:0/2363665038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032000bc80 con 0x7f03280aebe0 2024-04-06T12:28:10.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.782+0000 7f032db0f700 1 --2- 172.21.15.189:0/2363665038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03280b4660 0x7f03280ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:10.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.782+0000 7f032eb11700 1 --2- 172.21.15.189:0/2363665038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03280aebe0 0x7f0328158b50 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f032806c820 tx=0x7f0320004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:10.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.782+0000 7f031affd700 1 -- 172.21.15.189:0/2363665038 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032000b450 con 0x7f03280aebe0 2024-04-06T12:28:10.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.783+0000 7f031affd700 1 -- 172.21.15.189:0/2363665038 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032000bb00 con 0x7f03280aebe0 2024-04-06T12:28:10.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.783+0000 7f0330574700 1 -- 172.21.15.189:0/2363665038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0328155670 con 0x7f03280aebe0 2024-04-06T12:28:10.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.783+0000 7f031affd700 1 -- 172.21.15.189:0/2363665038 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0320009760 con 0x7f03280aebe0 2024-04-06T12:28:10.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.783+0000 7f0330574700 1 -- 172.21.15.189:0/2363665038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0328155c30 con 0x7f03280aebe0 2024-04-06T12:28:10.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.783+0000 7f0330574700 1 -- 172.21.15.189:0/2363665038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0328156130 con 0x7f03280aebe0 2024-04-06T12:28:10.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.784+0000 7f031affd700 1 -- 172.21.15.189:0/2363665038 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0320009070 con 0x7f03280aebe0 2024-04-06T12:28:10.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.784+0000 7f031affd700 1 --2- 172.21.15.189:0/2363665038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f030c06adb0 0x7f030c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:10.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.784+0000 7f031affd700 1 -- 172.21.15.189:0/2363665038 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f032008a6a0 con 0x7f03280aebe0 2024-04-06T12:28:10.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.785+0000 7f032e310700 1 --2- 172.21.15.189:0/2363665038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f030c06adb0 0x7f030c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:10.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.785+0000 7f032e310700 1 --2- 172.21.15.189:0/2363665038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f030c06adb0 0x7f030c06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f031c000f60 tx=0x7f031c007be0 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:10.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:10.789+0000 7f031affd700 1 -- 172.21.15.189:0/2363665038 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f03200555b0 con 0x7f03280aebe0 2024-04-06T12:28:11.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.028+0000 7f0330574700 1 -- 172.21.15.189:0/2363665038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0328067bb0 con 0x7f03280aebe0 2024-04-06T12:28:11.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.028+0000 7f031affd700 1 -- 172.21.15.189:0/2363665038 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f0320058bd0 con 0x7f03280aebe0 2024-04-06T12:28:11.026 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:11.026 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":39,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3065963154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3065963154},{"type":"v1","addr":"172.21.15.192:6833","nonce":3065963154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6071,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/831568588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":831568588},{"type":"v1","addr":"172.21.15.192:6835","nonce":831568588}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":7080,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1688936113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1688936113},{"type":"v1","addr":"172.21.15.189:6836","nonce":1688936113}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:28:05.673337+0000","modified":"2024-04-06T12:28:08.681505+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7083},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7083":{"gid":7083,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":4,"addr":"172.21.15.189:6837/3778396425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3778396425},{"type":"v1","addr":"172.21.15.189:6837","nonce":3778396425}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-04-06T12:28:11.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.030+0000 7f0330574700 1 -- 172.21.15.189:0/2363665038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f030c06adb0 msgr2=0x7f030c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:11.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.030+0000 7f0330574700 1 --2- 172.21.15.189:0/2363665038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f030c06adb0 0x7f030c06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f031c000f60 tx=0x7f031c007be0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.030+0000 7f0330574700 1 -- 172.21.15.189:0/2363665038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03280aebe0 msgr2=0x7f0328158b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:11.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.030+0000 7f0330574700 1 --2- 172.21.15.189:0/2363665038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03280aebe0 0x7f0328158b50 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f032806c820 tx=0x7f0320004ad0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.030+0000 7f0330574700 1 -- 172.21.15.189:0/2363665038 shutdown_connections 2024-04-06T12:28:11.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.030+0000 7f0330574700 1 --2- 172.21.15.189:0/2363665038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03280aebe0 0x7f0328158b50 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.030+0000 7f0330574700 1 --2- 172.21.15.189:0/2363665038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03280b2fc0 0x7f03281f2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.030+0000 7f0330574700 1 --2- 172.21.15.189:0/2363665038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f030c06adb0 0x7f030c06d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.030+0000 7f0330574700 1 --2- 172.21.15.189:0/2363665038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03280b4660 0x7f03280ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.030+0000 7f0330574700 1 -- 172.21.15.189:0/2363665038 >> 172.21.15.189:0/2363665038 conn(0x7f03280d00b0 msgr2=0x7f0328157a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:11.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.030+0000 7f0330574700 1 -- 172.21.15.189:0/2363665038 shutdown_connections 2024-04-06T12:28:11.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.030+0000 7f0330574700 1 -- 172.21.15.189:0/2363665038 wait complete. 2024-04-06T12:28:11.030 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 39 2024-04-06T12:28:11.151 INFO:tasks.ceph.mds.a.smithi189.stderr:2024-04-06T12:28:11.153+0000 7f7a76c2d700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T12:28:11.444 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 39, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-06T12:28:05.673337+0000', 'modified': '2024-04-06T12:28:08.681505+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7083}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7083': {'gid': 7083, 'name': 'a', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.189:6837/3778396425', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6835', 'nonce': 3778396425}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 3778396425}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-06T12:28:11.444 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-06T12:28:11.444 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-06T12:28:11.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.573+0000 7f668d879700 1 Processor -- start 2024-04-06T12:28:11.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.573+0000 7f668d879700 1 -- start start 2024-04-06T12:28:11.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.573+0000 7f668d879700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f668815d6f0 0x7f668815dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.573+0000 7f668d879700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66880b8d70 0x7f66880ce020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.573+0000 7f668d879700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f668815eac0 con 0x7f66880ce560 2024-04-06T12:28:11.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.574+0000 7f668d879700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66880cea10 con 0x7f668815d6f0 2024-04-06T12:28:11.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.574+0000 7f668d879700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f668815e010 con 0x7f66880b8d70 2024-04-06T12:28:11.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.574+0000 7f6686ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f668815d6f0 0x7f668815dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.574+0000 7f6686ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f668815d6f0 0x7f668815dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52256/0 (socket says 172.21.15.189:52256) 2024-04-06T12:28:11.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.574+0000 7f66877fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f66880ce560 0x7f66880b4e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44614/0 (socket says 172.21.15.189:44614) 2024-04-06T12:28:11.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.574+0000 7f6686ffd700 1 -- 172.21.15.189:0/98267904 learned_addr learned my addr 172.21.15.189:0/98267904 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:11.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.574+0000 7f66867fc700 1 --2- 172.21.15.189:0/98267904 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66880b8d70 0x7f66880ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.574+0000 7f6686ffd700 1 -- 172.21.15.189:0/98267904 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66880b8d70 msgr2=0x7f66880ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:11.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.574+0000 7f6686ffd700 1 --2- 172.21.15.189:0/98267904 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66880b8d70 0x7f66880ce020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.574+0000 7f6686ffd700 1 -- 172.21.15.189:0/98267904 >> v1:172.21.15.189:6789/0 conn(0x7f66880ce560 legacy=0x7f66880b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:11.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.574+0000 7f6686ffd700 1 -- 172.21.15.189:0/98267904 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668815e220 con 0x7f668815d6f0 2024-04-06T12:28:11.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.574+0000 7f6685ffb700 1 -- 172.21.15.189:0/98267904 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813224203 0 0) 0x7f668815eac0 con 0x7f66880ce560 2024-04-06T12:28:11.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.574+0000 7f6686ffd700 1 --2- 172.21.15.189:0/98267904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f668815d6f0 0x7f668815dad0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f66780098f0 tx=0x7f6678009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=919b03cce9c36128 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:11.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.574+0000 7f6685ffb700 1 -- 172.21.15.189:0/98267904 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667800b8a0 con 0x7f668815d6f0 2024-04-06T12:28:11.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.574+0000 7f6685ffb700 1 -- 172.21.15.189:0/98267904 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667800ba00 con 0x7f668815d6f0 2024-04-06T12:28:11.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.574+0000 7f6685ffb700 1 -- 172.21.15.189:0/98267904 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667800bd90 con 0x7f668815d6f0 2024-04-06T12:28:11.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.575+0000 7f668d879700 1 -- 172.21.15.189:0/98267904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f668815d6f0 msgr2=0x7f668815dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:11.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.575+0000 7f668d879700 1 --2- 172.21.15.189:0/98267904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f668815d6f0 0x7f668815dad0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f66780098f0 tx=0x7f6678009c00 comp rx=0 tx=0).stop 2024-04-06T12:28:11.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.575+0000 7f668d879700 1 -- 172.21.15.189:0/98267904 shutdown_connections 2024-04-06T12:28:11.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.575+0000 7f668d879700 1 --2- 172.21.15.189:0/98267904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f668815d6f0 0x7f668815dad0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.575+0000 7f668d879700 1 --2- 172.21.15.189:0/98267904 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66880b8d70 0x7f66880ce020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.575+0000 7f668d879700 1 -- 172.21.15.189:0/98267904 >> 172.21.15.189:0/98267904 conn(0x7f6688072490 msgr2=0x7f668806b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:11.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.575+0000 7f668d879700 1 -- 172.21.15.189:0/98267904 shutdown_connections 2024-04-06T12:28:11.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.575+0000 7f668d879700 1 -- 172.21.15.189:0/98267904 wait complete. 2024-04-06T12:28:11.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.576+0000 7f668d879700 1 Processor -- start 2024-04-06T12:28:11.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.576+0000 7f668d879700 1 -- start start 2024-04-06T12:28:11.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.577+0000 7f668d879700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66880b8d70 0x7f6688158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.577+0000 7f668d879700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66880ce560 0x7f66880c8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.577+0000 7f668d879700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f668815d6f0 0x7f66881f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.577+0000 7f668d879700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66880c6f10 con 0x7f66880b8d70 2024-04-06T12:28:11.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.577+0000 7f668d879700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66880c7160 con 0x7f66880ce560 2024-04-06T12:28:11.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.577+0000 7f668d879700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66880c7440 con 0x7f668815d6f0 2024-04-06T12:28:11.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.577+0000 7f6686ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66880b8d70 0x7f6688158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.577+0000 7f6686ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66880b8d70 0x7f6688158ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52268/0 (socket says 172.21.15.189:52268) 2024-04-06T12:28:11.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.577+0000 7f6686ffd700 1 -- 172.21.15.189:0/945926625 learned_addr learned my addr 172.21.15.189:0/945926625 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:11.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.577+0000 7f6686ffd700 1 -- 172.21.15.189:0/945926625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f668815d6f0 msgr2=0x7f66881f3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:11.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.577+0000 7f66867fc700 1 --2- 172.21.15.189:0/945926625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66880ce560 0x7f66880c8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.577+0000 7f6686ffd700 1 --2- 172.21.15.189:0/945926625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f668815d6f0 0x7f66881f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.577+0000 7f6686ffd700 1 -- 172.21.15.189:0/945926625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66880ce560 msgr2=0x7f66880c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:11.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.577+0000 7f6686ffd700 1 --2- 172.21.15.189:0/945926625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66880ce560 0x7f66880c8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.577+0000 7f6686ffd700 1 -- 172.21.15.189:0/945926625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6678009550 con 0x7f66880b8d70 2024-04-06T12:28:11.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.577+0000 7f6686ffd700 1 --2- 172.21.15.189:0/945926625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66880b8d70 0x7f6688158ae0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f6678014040 tx=0x7f6678004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:11.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.578+0000 7f666ffff700 1 -- 172.21.15.189:0/945926625 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66780036a0 con 0x7f66880b8d70 2024-04-06T12:28:11.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.578+0000 7f666ffff700 1 -- 172.21.15.189:0/945926625 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667801e070 con 0x7f66880b8d70 2024-04-06T12:28:11.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.578+0000 7f668d879700 1 -- 172.21.15.189:0/945926625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66880c7720 con 0x7f66880b8d70 2024-04-06T12:28:11.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.578+0000 7f666ffff700 1 -- 172.21.15.189:0/945926625 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6678003a40 con 0x7f66880b8d70 2024-04-06T12:28:11.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.578+0000 7f668d879700 1 -- 172.21.15.189:0/945926625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66880c7ce0 con 0x7f66880b8d70 2024-04-06T12:28:11.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.579+0000 7f666ffff700 1 -- 172.21.15.189:0/945926625 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f6678003ba0 con 0x7f66880b8d70 2024-04-06T12:28:11.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.580+0000 7f666ffff700 1 --2- 172.21.15.189:0/945926625 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f667406adb0 0x7f667406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.580+0000 7f666ffff700 1 -- 172.21.15.189:0/945926625 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6678020070 con 0x7f66880b8d70 2024-04-06T12:28:11.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.580+0000 7f66867fc700 1 --2- 172.21.15.189:0/945926625 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f667406adb0 0x7f667406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.580+0000 7f66867fc700 1 --2- 172.21.15.189:0/945926625 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f667406adb0 0x7f667406d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f66880b9220 tx=0x7f667c008040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:11.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.581+0000 7f6699a2db80 1 Processor -- start 2024-04-06T12:28:11.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.582+0000 7f6699a2db80 1 -- start start 2024-04-06T12:28:11.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.582+0000 7f6699a2db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 0x55cab7838a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.582+0000 7f6699a2db80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab7839fc0 0x55cab78105f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.582+0000 7f6699a2db80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab780f980 0x55cab7810b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.582+0000 7f6699a2db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cab7769940 con 0x55cab7834c20 2024-04-06T12:28:11.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.582+0000 7f6699a2db80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cab7767300 con 0x55cab780f980 2024-04-06T12:28:11.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.582+0000 7f6699a2db80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cab7762bf0 con 0x55cab7839fc0 2024-04-06T12:28:11.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.583+0000 7f6686ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 0x55cab7838a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.583+0000 7f6686ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 0x55cab7838a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52270/0 (socket says 172.21.15.189:52270) 2024-04-06T12:28:11.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.583+0000 7f6686ffd700 1 -- 172.21.15.189:0/1509209461 learned_addr learned my addr 172.21.15.189:0/1509209461 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:11.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.583+0000 7f66877fe700 1 --2- 172.21.15.189:0/1509209461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab780f980 0x55cab7810b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.583+0000 7f66867fc700 1 --2- 172.21.15.189:0/1509209461 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab7839fc0 0x55cab78105f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.583+0000 7f6686ffd700 1 -- 172.21.15.189:0/1509209461 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab7839fc0 msgr2=0x55cab78105f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:11.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.583+0000 7f6686ffd700 1 --2- 172.21.15.189:0/1509209461 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab7839fc0 0x55cab78105f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.583+0000 7f6686ffd700 1 -- 172.21.15.189:0/1509209461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab780f980 msgr2=0x55cab7810b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:11.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.583+0000 7f6686ffd700 1 --2- 172.21.15.189:0/1509209461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab780f980 0x55cab7810b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.583+0000 7f6686ffd700 1 -- 172.21.15.189:0/1509209461 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cab7590b10 con 0x55cab7834c20 2024-04-06T12:28:11.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.583+0000 7f6686ffd700 1 --2- 172.21.15.189:0/1509209461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 0x55cab7838a60 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f667800b0e0 tx=0x7f6678055bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2901b765bb8581b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:11.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.583+0000 7f666d7fa700 1 -- 172.21.15.189:0/1509209461 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667805d630 con 0x55cab7834c20 2024-04-06T12:28:11.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.583+0000 7f666d7fa700 1 -- 172.21.15.189:0/1509209461 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6678061bf0 con 0x55cab7834c20 2024-04-06T12:28:11.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.583+0000 7f666d7fa700 1 -- 172.21.15.189:0/1509209461 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6678059a60 con 0x55cab7834c20 2024-04-06T12:28:11.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.584+0000 7f6699a2db80 1 -- 172.21.15.189:0/1509209461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 msgr2=0x55cab7838a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:11.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.584+0000 7f6699a2db80 1 --2- 172.21.15.189:0/1509209461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 0x55cab7838a60 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f667800b0e0 tx=0x7f6678055bc0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.584+0000 7f6699a2db80 1 -- 172.21.15.189:0/1509209461 shutdown_connections 2024-04-06T12:28:11.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.584+0000 7f6699a2db80 1 --2- 172.21.15.189:0/1509209461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab780f980 0x55cab7810b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.584+0000 7f6699a2db80 1 --2- 172.21.15.189:0/1509209461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 0x55cab7838a60 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.584+0000 7f6699a2db80 1 --2- 172.21.15.189:0/1509209461 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab7839fc0 0x55cab78105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.584+0000 7f6699a2db80 1 -- 172.21.15.189:0/1509209461 >> 172.21.15.189:0/1509209461 conn(0x55cab7770650 msgr2=0x55cab78573e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:11.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.584+0000 7f6699a2db80 1 -- 172.21.15.189:0/1509209461 shutdown_connections 2024-04-06T12:28:11.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.584+0000 7f6699a2db80 1 -- 172.21.15.189:0/1509209461 wait complete. 2024-04-06T12:28:11.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.584+0000 7f6699a2db80 1 -- start start 2024-04-06T12:28:11.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.585+0000 7f6699a2db80 1 Processor -- start 2024-04-06T12:28:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.585+0000 7f6699a2db80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab780f980 0x55cab78372e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.585+0000 7f6699a2db80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab7834c20 0x55cab786a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.585+0000 7f6699a2db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7839fc0 0x55cab783b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.585+0000 7f6699a2db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cab76fed30 con 0x55cab7839fc0 2024-04-06T12:28:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.585+0000 7f6699a2db80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cab76fe9d0 con 0x55cab7834c20 2024-04-06T12:28:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.585+0000 7f6699a2db80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cab75e6650 con 0x55cab780f980 2024-04-06T12:28:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.585+0000 7f6699a2db80 10 client.0 _ll_register_callbacks cb 0x55cab7708420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:28:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.585+0000 7f66877fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7839fc0 0x55cab783b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.585+0000 7f66877fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7839fc0 0x55cab783b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52278/0 (socket says 172.21.15.189:52278) 2024-04-06T12:28:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.585+0000 7f66877fe700 1 -- 172.21.15.189:0/988321214 learned_addr learned my addr 172.21.15.189:0/988321214 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.585+0000 7f6686ffd700 1 --2- 172.21.15.189:0/988321214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab780f980 0x55cab78372e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.585+0000 7f66867fc700 1 --2- 172.21.15.189:0/988321214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab7834c20 0x55cab786a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f66877fe700 1 -- 172.21.15.189:0/988321214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab780f980 msgr2=0x55cab78372e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:11.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f66877fe700 1 --2- 172.21.15.189:0/988321214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab780f980 0x55cab78372e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f66877fe700 1 -- 172.21.15.189:0/988321214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab7834c20 msgr2=0x55cab786a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:11.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f66877fe700 1 --2- 172.21.15.189:0/988321214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab7834c20 0x55cab786a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f66877fe700 1 -- 172.21.15.189:0/988321214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66880c7ce0 con 0x55cab7839fc0 2024-04-06T12:28:11.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f66867fc700 1 --2- 172.21.15.189:0/988321214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab7834c20 0x55cab786a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:11.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f66877fe700 1 --2- 172.21.15.189:0/988321214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7839fc0 0x55cab783b4b0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f6680007810 tx=0x7f6680007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35affcfa1353c8a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:11.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f66667fc700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:28:11.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f66667fc700 1 -- 172.21.15.189:0/988321214 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668000e9d0 con 0x55cab7839fc0 2024-04-06T12:28:11.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f66667fc700 1 -- 172.21.15.189:0/988321214 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f668000eb30 con 0x55cab7839fc0 2024-04-06T12:28:11.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f6699a2db80 1 -- 172.21.15.189:0/988321214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cab7723330 con 0x55cab7839fc0 2024-04-06T12:28:11.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f66667fc700 1 -- 172.21.15.189:0/988321214 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6680001cb0 con 0x55cab7839fc0 2024-04-06T12:28:11.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f6699a2db80 10 client.7227 fetch_fsmap learned FSMap version 39 2024-04-06T12:28:11.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f6699a2db80 1 -- 172.21.15.189:0/988321214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=2+}) v3 -- 0x55cab7728e50 con 0x55cab7839fc0 2024-04-06T12:28:11.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f66667fc700 1 -- 172.21.15.189:0/988321214 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55cab7723330 con 0x55cab7839fc0 2024-04-06T12:28:11.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f66667fc700 1 -- 172.21.15.189:0/988321214 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 39) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f668000ecf0 con 0x55cab7839fc0 2024-04-06T12:28:11.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f6699a2db80 10 client.7227 fetch_fsmap finished waiting for FSMap version 39 2024-04-06T12:28:11.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f6699a2db80 10 client.7227 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-06T12:28:11.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f6699a2db80 20 client.7227 populate_metadata read hostname 'smithi189' 2024-04-06T12:28:11.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f6699a2db80 1 --2- 172.21.15.189:0/988321214 >> [v2:172.21.15.189:6835/3778396425,v1:172.21.15.189:6837/3778396425] conn(0x55cab783fd90 0x55cab783a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f6699a2db80 4 client.7227 mds_command: new command op to 7083 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-06T12:28:11.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.586+0000 7f6699a2db80 1 -- 172.21.15.189:0/988321214 --> [v2:172.21.15.189:6835/3778396425,v1:172.21.15.189:6837/3778396425] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cab775b6b0 con 0x55cab783fd90 2024-04-06T12:28:11.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.587+0000 7f6686ffd700 1 --2- 172.21.15.189:0/988321214 >> [v2:172.21.15.189:6835/3778396425,v1:172.21.15.189:6837/3778396425] conn(0x55cab783fd90 0x55cab783a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.587+0000 7f66667fc700 10 client.7227 ms_handle_connect on v2:172.21.15.189:6835/3778396425 2024-04-06T12:28:11.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.587+0000 7f6686ffd700 1 --2- 172.21.15.189:0/988321214 >> [v2:172.21.15.189:6835/3778396425,v1:172.21.15.189:6837/3778396425] conn(0x55cab783fd90 0x55cab783a3a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30bf684289aee006 server_cookie=93082c1faef5d28 in_seq=0 out_seq=0 2024-04-06T12:28:11.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.587+0000 7f66667fc700 1 -- 172.21.15.189:0/988321214 <== mds.0 v2:172.21.15.189:6835/3778396425 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55cab775b6b0 con 0x55cab783fd90 2024-04-06T12:28:11.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.587+0000 7f66667fc700 10 client.7227 handle_command_reply: tid=0 2024-04-06T12:28:11.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.587+0000 7f6699a2db80 1 client.7227 shutdown 2024-04-06T12:28:11.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.587+0000 7f6699a2db80 1 -- 172.21.15.189:0/988321214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7839fc0 msgr2=0x55cab783b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:11.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.587+0000 7f6699a2db80 1 --2- 172.21.15.189:0/988321214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7839fc0 0x55cab783b4b0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f6680007810 tx=0x7f6680007b20 comp rx=0 tx=0).stop 2024-04-06T12:28:11.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.588+0000 7f6699a2db80 1 -- 172.21.15.189:0/988321214 shutdown_connections 2024-04-06T12:28:11.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.588+0000 7f6699a2db80 1 --2- 172.21.15.189:0/988321214 >> [v2:172.21.15.189:6835/3778396425,v1:172.21.15.189:6837/3778396425] conn(0x55cab783fd90 0x55cab783a3a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.588+0000 7f6699a2db80 1 --2- 172.21.15.189:0/988321214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7839fc0 0x55cab783b4b0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.588+0000 7f6699a2db80 1 --2- 172.21.15.189:0/988321214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab780f980 0x55cab78372e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.588+0000 7f66667fc700 0 client.7227 ms_handle_reset on v2:172.21.15.189:6835/3778396425 2024-04-06T12:28:11.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.588+0000 7f6699a2db80 1 --2- 172.21.15.189:0/988321214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab7834c20 0x55cab786a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.588+0000 7f6699a2db80 1 -- 172.21.15.189:0/988321214 >> 172.21.15.189:0/988321214 conn(0x55cab7770650 msgr2=0x55cab78105f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:11.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.588+0000 7f6699a2db80 1 -- 172.21.15.189:0/988321214 shutdown_connections 2024-04-06T12:28:11.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.588+0000 7f6699a2db80 1 -- 172.21.15.189:0/988321214 wait complete. 2024-04-06T12:28:11.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.588+0000 7f6699a2db80 20 client.7227 trim_cache size 0 max 16384 2024-04-06T12:28:11.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.606+0000 7f6699a2db80 1 Processor -- start 2024-04-06T12:28:11.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.606+0000 7f6699a2db80 1 -- start start 2024-04-06T12:28:11.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.606+0000 7f6699a2db80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab780f980 0x55cab7836630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.606+0000 7f6699a2db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 0x55cab783c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.606+0000 7f6699a2db80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab7839fc0 0x55cab783ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.606+0000 7f6699a2db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cab780c900 con 0x55cab7834c20 2024-04-06T12:28:11.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.606+0000 7f6699a2db80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cab780d590 con 0x55cab780f980 2024-04-06T12:28:11.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.606+0000 7f6699a2db80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cab786abd0 con 0x55cab7839fc0 2024-04-06T12:28:11.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.607+0000 7f66867fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 0x55cab783c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.607+0000 7f66867fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 0x55cab783c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52288/0 (socket says 172.21.15.189:52288) 2024-04-06T12:28:11.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.607+0000 7f66867fc700 1 -- 172.21.15.189:0/3431002416 learned_addr learned my addr 172.21.15.189:0/3431002416 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:11.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.607+0000 7f6686ffd700 1 --2- 172.21.15.189:0/3431002416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab780f980 0x55cab7836630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.607+0000 7f66877fe700 1 --2- 172.21.15.189:0/3431002416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab7839fc0 0x55cab783ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.607+0000 7f66867fc700 1 -- 172.21.15.189:0/3431002416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab7839fc0 msgr2=0x55cab783ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:11.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.607+0000 7f66867fc700 1 --2- 172.21.15.189:0/3431002416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab7839fc0 0x55cab783ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.607+0000 7f66867fc700 1 -- 172.21.15.189:0/3431002416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab780f980 msgr2=0x55cab7836630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:11.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.607+0000 7f66867fc700 1 --2- 172.21.15.189:0/3431002416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab780f980 0x55cab7836630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.607+0000 7f66867fc700 1 -- 172.21.15.189:0/3431002416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cab75e8c80 con 0x55cab7834c20 2024-04-06T12:28:11.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.607+0000 7f6686ffd700 1 --2- 172.21.15.189:0/3431002416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab780f980 0x55cab7836630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:28:11.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.607+0000 7f66877fe700 1 --2- 172.21.15.189:0/3431002416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab7839fc0 0x55cab783ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:28:11.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.607+0000 7f66867fc700 1 --2- 172.21.15.189:0/3431002416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 0x55cab783c8d0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f667c0041b0 tx=0x7f667c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3967e98bc48ce4a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:11.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.607+0000 7f666dffb700 1 -- 172.21.15.189:0/3431002416 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c004570 con 0x55cab7834c20 2024-04-06T12:28:11.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.607+0000 7f666dffb700 1 -- 172.21.15.189:0/3431002416 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667c003b30 con 0x55cab7834c20 2024-04-06T12:28:11.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.607+0000 7f666dffb700 1 -- 172.21.15.189:0/3431002416 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c00f6d0 con 0x55cab7834c20 2024-04-06T12:28:11.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.608+0000 7f6699a2db80 1 -- 172.21.15.189:0/3431002416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 msgr2=0x55cab783c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:11.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.608+0000 7f6699a2db80 1 --2- 172.21.15.189:0/3431002416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 0x55cab783c8d0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f667c0041b0 tx=0x7f667c015000 comp rx=0 tx=0).stop 2024-04-06T12:28:11.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.608+0000 7f6699a2db80 1 -- 172.21.15.189:0/3431002416 shutdown_connections 2024-04-06T12:28:11.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.608+0000 7f6699a2db80 1 --2- 172.21.15.189:0/3431002416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab7839fc0 0x55cab783ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.608+0000 7f6699a2db80 1 --2- 172.21.15.189:0/3431002416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab780f980 0x55cab7836630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.608+0000 7f6699a2db80 1 --2- 172.21.15.189:0/3431002416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 0x55cab783c8d0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.608+0000 7f6699a2db80 1 -- 172.21.15.189:0/3431002416 >> 172.21.15.189:0/3431002416 conn(0x55cab7770650 msgr2=0x55cab783c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:11.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.608+0000 7f6699a2db80 1 -- 172.21.15.189:0/3431002416 shutdown_connections 2024-04-06T12:28:11.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.608+0000 7f6699a2db80 1 -- 172.21.15.189:0/3431002416 wait complete. 2024-04-06T12:28:11.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.608+0000 7f6699a2db80 1 -- start start 2024-04-06T12:28:11.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.608+0000 7f6699a2db80 1 Processor -- start 2024-04-06T12:28:11.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.609+0000 7f6699a2db80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab780f980 0x55cab78bdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.609+0000 7f6699a2db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 0x55cab786f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.609+0000 7f6699a2db80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab7839fc0 0x55cab786f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.609+0000 7f6699a2db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cab78399d0 con 0x55cab7834c20 2024-04-06T12:28:11.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.609+0000 7f6699a2db80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cab78378a0 con 0x55cab7839fc0 2024-04-06T12:28:11.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.609+0000 7f6699a2db80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cab783ab80 con 0x55cab780f980 2024-04-06T12:28:11.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.609+0000 7f6699a2db80 10 client.0 _ll_register_callbacks cb 0x55cab7708420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:28:11.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.609+0000 7f66867fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 0x55cab786f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.609+0000 7f66867fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 0x55cab786f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52296/0 (socket says 172.21.15.189:52296) 2024-04-06T12:28:11.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.609+0000 7f66867fc700 1 -- 172.21.15.189:0/1420053004 learned_addr learned my addr 172.21.15.189:0/1420053004 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:11.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.609+0000 7f6686ffd700 1 --2- 172.21.15.189:0/1420053004 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab780f980 0x55cab78bdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.609+0000 7f66877fe700 1 --2- 172.21.15.189:0/1420053004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab7839fc0 0x55cab786f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f66867fc700 1 -- 172.21.15.189:0/1420053004 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab780f980 msgr2=0x55cab78bdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:11.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f66867fc700 1 --2- 172.21.15.189:0/1420053004 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab780f980 0x55cab78bdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f66867fc700 1 -- 172.21.15.189:0/1420053004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab7839fc0 msgr2=0x55cab786f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:11.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f66867fc700 1 --2- 172.21.15.189:0/1420053004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab7839fc0 0x55cab786f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f66867fc700 1 -- 172.21.15.189:0/1420053004 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f667c00e040 con 0x55cab7834c20 2024-04-06T12:28:11.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f66877fe700 1 --2- 172.21.15.189:0/1420053004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab7839fc0 0x55cab786f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:11.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f66867fc700 1 --2- 172.21.15.189:0/1420053004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 0x55cab786f240 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x55cab780ca40 tx=0x7f667c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e39d994f72246f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:11.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f66667fc700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:28:11.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f66667fc700 1 -- 172.21.15.189:0/1420053004 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c036460 con 0x55cab7834c20 2024-04-06T12:28:11.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f66667fc700 1 -- 172.21.15.189:0/1420053004 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f667c0365c0 con 0x55cab7834c20 2024-04-06T12:28:11.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f6699a2db80 1 -- 172.21.15.189:0/1420053004 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cab780cef0 con 0x55cab7834c20 2024-04-06T12:28:11.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f66667fc700 1 -- 172.21.15.189:0/1420053004 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f667c036780 con 0x55cab7834c20 2024-04-06T12:28:11.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f6699a2db80 10 client.7233 fetch_fsmap learned FSMap version 39 2024-04-06T12:28:11.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f6699a2db80 1 -- 172.21.15.189:0/1420053004 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=2+}) v3 -- 0x55cab758efe0 con 0x55cab7834c20 2024-04-06T12:28:11.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f66667fc700 1 -- 172.21.15.189:0/1420053004 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55cab780cef0 con 0x55cab7834c20 2024-04-06T12:28:11.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f66667fc700 1 -- 172.21.15.189:0/1420053004 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 39) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f667c021070 con 0x55cab7834c20 2024-04-06T12:28:11.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f6699a2db80 10 client.7233 fetch_fsmap finished waiting for FSMap version 39 2024-04-06T12:28:11.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f6699a2db80 10 client.7233 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-06T12:28:11.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f6699a2db80 20 client.7233 populate_metadata read hostname 'smithi189' 2024-04-06T12:28:11.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f6699a2db80 1 --2- 172.21.15.189:0/1420053004 >> [v2:172.21.15.189:6835/3778396425,v1:172.21.15.189:6837/3778396425] conn(0x55cab783c8d0 0x55cab783ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f6699a2db80 4 client.7233 mds_command: new command op to 7083 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-06T12:28:11.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f6699a2db80 1 -- 172.21.15.189:0/1420053004 --> [v2:172.21.15.189:6835/3778396425,v1:172.21.15.189:6837/3778396425] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f6678061bf0 con 0x55cab783c8d0 2024-04-06T12:28:11.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.610+0000 7f6686ffd700 1 --2- 172.21.15.189:0/1420053004 >> [v2:172.21.15.189:6835/3778396425,v1:172.21.15.189:6837/3778396425] conn(0x55cab783c8d0 0x55cab783ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.611+0000 7f66667fc700 10 client.7233 ms_handle_connect on v2:172.21.15.189:6835/3778396425 2024-04-06T12:28:11.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.611+0000 7f6686ffd700 1 --2- 172.21.15.189:0/1420053004 >> [v2:172.21.15.189:6835/3778396425,v1:172.21.15.189:6837/3778396425] conn(0x55cab783c8d0 0x55cab783ccb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72946ad29192561f server_cookie=91f496af59d1be7d in_seq=0 out_seq=0 2024-04-06T12:28:11.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.611+0000 7f66667fc700 1 -- 172.21.15.189:0/1420053004 <== mds.0 v2:172.21.15.189:6835/3778396425 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x7f6678061bf0 con 0x55cab783c8d0 2024-04-06T12:28:11.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.611+0000 7f66667fc700 10 client.7233 handle_command_reply: tid=0 2024-04-06T12:28:11.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.611+0000 7f6699a2db80 1 client.7233 shutdown 2024-04-06T12:28:11.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.611+0000 7f6699a2db80 1 -- 172.21.15.189:0/1420053004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 msgr2=0x55cab786f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:11.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.611+0000 7f6699a2db80 1 --2- 172.21.15.189:0/1420053004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 0x55cab786f240 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x55cab780ca40 tx=0x7f667c0047c0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.611+0000 7f6699a2db80 1 -- 172.21.15.189:0/1420053004 shutdown_connections 2024-04-06T12:28:11.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.611+0000 7f6699a2db80 1 --2- 172.21.15.189:0/1420053004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cab7839fc0 0x55cab786f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.611+0000 7f6699a2db80 1 --2- 172.21.15.189:0/1420053004 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55cab780f980 0x55cab78bdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.611+0000 7f6699a2db80 1 --2- 172.21.15.189:0/1420053004 >> [v2:172.21.15.189:6835/3778396425,v1:172.21.15.189:6837/3778396425] conn(0x55cab783c8d0 0x55cab783ccb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.611+0000 7f6699a2db80 1 --2- 172.21.15.189:0/1420053004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cab7834c20 0x55cab786f240 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.611+0000 7f66667fc700 0 client.7233 ms_handle_reset on v2:172.21.15.189:6835/3778396425 2024-04-06T12:28:11.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.611+0000 7f6699a2db80 1 -- 172.21.15.189:0/1420053004 >> 172.21.15.189:0/1420053004 conn(0x55cab7770650 msgr2=0x55cab78085c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:11.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.611+0000 7f6699a2db80 1 -- 172.21.15.189:0/1420053004 shutdown_connections 2024-04-06T12:28:11.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.611+0000 7f6699a2db80 1 -- 172.21.15.189:0/1420053004 wait complete. 2024-04-06T12:28:11.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.611+0000 7f6699a2db80 20 client.7233 trim_cache size 0 max 16384 2024-04-06T12:28:11.617 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T12:28:11.617 INFO:teuthology.orchestra.run.smithi189.stdout: "cluster_fsid": "e30041de-e9a3-4351-8ca7-29d7f3e3728c", 2024-04-06T12:28:11.617 INFO:teuthology.orchestra.run.smithi189.stdout: "whoami": 0, 2024-04-06T12:28:11.617 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 7083, 2024-04-06T12:28:11.617 INFO:teuthology.orchestra.run.smithi189.stdout: "want_state": "up:active", 2024-04-06T12:28:11.617 INFO:teuthology.orchestra.run.smithi189.stdout: "state": "up:active", 2024-04-06T12:28:11.617 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs", 2024-04-06T12:28:11.617 INFO:teuthology.orchestra.run.smithi189.stdout: "rank_uptime": 5.9374933639999998, 2024-04-06T12:28:11.617 INFO:teuthology.orchestra.run.smithi189.stdout: "mdsmap_epoch": 39, 2024-04-06T12:28:11.617 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch": 98, 2024-04-06T12:28:11.617 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch_barrier": 98, 2024-04-06T12:28:11.617 INFO:teuthology.orchestra.run.smithi189.stdout: "uptime": 16.365870774000001 2024-04-06T12:28:11.617 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T12:28:11.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.612+0000 7f668d879700 1 -- 172.21.15.189:0/945926625 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f667406adb0 msgr2=0x7f667406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:11.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.612+0000 7f668d879700 1 --2- 172.21.15.189:0/945926625 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f667406adb0 0x7f667406d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f66880b9220 tx=0x7f667c008040 comp rx=0 tx=0).stop 2024-04-06T12:28:11.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.612+0000 7f668d879700 1 -- 172.21.15.189:0/945926625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66880b8d70 msgr2=0x7f6688158ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:11.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.612+0000 7f668d879700 1 --2- 172.21.15.189:0/945926625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66880b8d70 0x7f6688158ae0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f6678014040 tx=0x7f6678004750 comp rx=0 tx=0).stop 2024-04-06T12:28:11.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.612+0000 7f668d879700 1 -- 172.21.15.189:0/945926625 shutdown_connections 2024-04-06T12:28:11.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.612+0000 7f668d879700 1 --2- 172.21.15.189:0/945926625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f668815d6f0 0x7f66881f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.612+0000 7f668d879700 1 --2- 172.21.15.189:0/945926625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66880b8d70 0x7f6688158ae0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.612+0000 7f668d879700 1 --2- 172.21.15.189:0/945926625 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f667406adb0 0x7f667406d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.612+0000 7f668d879700 1 --2- 172.21.15.189:0/945926625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66880ce560 0x7f66880c8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.612+0000 7f668d879700 1 -- 172.21.15.189:0/945926625 >> 172.21.15.189:0/945926625 conn(0x7f6688072490 msgr2=0x7f66880d2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:11.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.612+0000 7f668d879700 1 -- 172.21.15.189:0/945926625 shutdown_connections 2024-04-06T12:28:11.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.612+0000 7f668d879700 1 -- 172.21.15.189:0/945926625 wait complete. 2024-04-06T12:28:11.622 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T12:28:11.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.747+0000 7f85dfaa5700 1 Processor -- start 2024-04-06T12:28:11.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.747+0000 7f85dfaa5700 1 -- start start 2024-04-06T12:28:11.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dfaa5700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f85d80cb870 0x7f85d815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dfaa5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d80cbd80 con 0x7f85d80cfa00 2024-04-06T12:28:11.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dfaa5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d80cef20 con 0x7f85d80d11b0 2024-04-06T12:28:11.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dfaa5700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d80ccdc0 con 0x7f85d80cb870 2024-04-06T12:28:11.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85de042700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f85d80cfa00 0x7f85d80cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44616/0 (socket says 172.21.15.189:44616) 2024-04-06T12:28:11.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85de042700 1 -- 172.21.15.189:0/747010009 learned_addr learned my addr 172.21.15.189:0/747010009 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:11.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dd040700 1 --2- 172.21.15.189:0/747010009 >> v2:172.21.15.192:3300/0 conn(0x7f85d80cb870 0x7f85d815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dc83f700 1 -- 172.21.15.189:0/747010009 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400566394 0 0) 0x7f85d80cbd80 con 0x7f85d80cfa00 2024-04-06T12:28:11.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dc83f700 1 -- 172.21.15.189:0/747010009 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85c40035b0 con 0x7f85d80cfa00 2024-04-06T12:28:11.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dc83f700 1 -- 172.21.15.189:0/747010009 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2390186705 0 0) 0x7f85c40035b0 con 0x7f85d80cfa00 2024-04-06T12:28:11.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dc83f700 1 -- 172.21.15.189:0/747010009 >> v2:172.21.15.192:3300/0 conn(0x7f85d80cb870 msgr2=0x7f85d815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:11.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dc83f700 1 --2- 172.21.15.189:0/747010009 >> v2:172.21.15.192:3300/0 conn(0x7f85d80cb870 0x7f85d815c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dc83f700 1 -- 172.21.15.189:0/747010009 >> v1:172.21.15.192:6789/0 conn(0x7f85d80d11b0 legacy=0x7f85d80d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:11.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dc83f700 1 -- 172.21.15.189:0/747010009 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d80cf200 con 0x7f85d80cfa00 2024-04-06T12:28:11.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dc83f700 1 -- 172.21.15.189:0/747010009 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f85d00019f0 con 0x7f85d80cfa00 2024-04-06T12:28:11.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dc83f700 1 -- 172.21.15.189:0/747010009 >> v1:172.21.15.189:6789/0 conn(0x7f85d80cfa00 legacy=0x7f85d80cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:11.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dc83f700 1 --2- 172.21.15.189:0/747010009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85c4003a00 0x7f85c4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dc83f700 1 --2- 172.21.15.189:0/747010009 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85c4004850 0x7f85c4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dc83f700 1 --2- 172.21.15.189:0/747010009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c4005210 0x7f85c400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dc83f700 1 -- 172.21.15.189:0/747010009 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c400c180 con 0x7f85c4005210 2024-04-06T12:28:11.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dc83f700 1 -- 172.21.15.189:0/747010009 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c400c2f0 con 0x7f85c4003a00 2024-04-06T12:28:11.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.748+0000 7f85dc83f700 1 -- 172.21.15.189:0/747010009 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c400c4b0 con 0x7f85c4004850 2024-04-06T12:28:11.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.749+0000 7f85dd841700 1 --2- 172.21.15.189:0/747010009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85c4003a00 0x7f85c4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.749+0000 7f85de042700 1 --2- 172.21.15.189:0/747010009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c4005210 0x7f85c400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.749+0000 7f85dd040700 1 --2- 172.21.15.189:0/747010009 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85c4004850 0x7f85c4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.749+0000 7f85de042700 1 -- 172.21.15.189:0/747010009 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85c4004850 msgr2=0x7f85c4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:11.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.749+0000 7f85de042700 1 --2- 172.21.15.189:0/747010009 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85c4004850 0x7f85c4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.749+0000 7f85de042700 1 -- 172.21.15.189:0/747010009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85c4003a00 msgr2=0x7f85c4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:11.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.749+0000 7f85de042700 1 --2- 172.21.15.189:0/747010009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85c4003a00 0x7f85c4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.749+0000 7f85de042700 1 -- 172.21.15.189:0/747010009 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c40035b0 con 0x7f85c4005210 2024-04-06T12:28:11.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.749+0000 7f85de042700 1 -- 172.21.15.189:0/747010009 reap_dead start 2024-04-06T12:28:11.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.749+0000 7f85de042700 1 --2- 172.21.15.189:0/747010009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c4005210 0x7f85c400ba10 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f85d000a830 tx=0x7f85c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1484372716713cf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:11.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.749+0000 7f85dc83f700 1 -- 172.21.15.189:0/747010009 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d000a2f0 con 0x7f85c4005210 2024-04-06T12:28:11.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.749+0000 7f85dc83f700 1 -- 172.21.15.189:0/747010009 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d0012620 con 0x7f85c4005210 2024-04-06T12:28:11.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.749+0000 7f85dc83f700 1 -- 172.21.15.189:0/747010009 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d00129d0 con 0x7f85c4005210 2024-04-06T12:28:11.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.750+0000 7f85dfaa5700 1 -- 172.21.15.189:0/747010009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c4005210 msgr2=0x7f85c400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:11.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.750+0000 7f85dfaa5700 1 --2- 172.21.15.189:0/747010009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c4005210 0x7f85c400ba10 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f85d000a830 tx=0x7f85c4005880 comp rx=0 tx=0).stop 2024-04-06T12:28:11.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.750+0000 7f85dfaa5700 1 -- 172.21.15.189:0/747010009 shutdown_connections 2024-04-06T12:28:11.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.750+0000 7f85dfaa5700 1 --2- 172.21.15.189:0/747010009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c4005210 0x7f85c400ba10 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.750+0000 7f85dfaa5700 1 -- 172.21.15.189:0/747010009 >> 172.21.15.189:0/747010009 conn(0x7f85d80b2db0 msgr2=0x7f85d80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:11.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.750+0000 7f85dfaa5700 1 -- 172.21.15.189:0/747010009 shutdown_connections 2024-04-06T12:28:11.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.750+0000 7f85dfaa5700 1 -- 172.21.15.189:0/747010009 wait complete. 2024-04-06T12:28:11.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.751+0000 7f85dfaa5700 1 Processor -- start 2024-04-06T12:28:11.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.751+0000 7f85dfaa5700 1 -- start start 2024-04-06T12:28:11.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85dfaa5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85c4005210 0x7f85d814ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85dfaa5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85d8153a30 0x7f85d815a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85dfaa5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85d815a8d0 0x7f85d815eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85dfaa5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d815ad80 con 0x7f85d8153a30 2024-04-06T12:28:11.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85dfaa5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d815c5f0 con 0x7f85c4005210 2024-04-06T12:28:11.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85dfaa5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d814cf40 con 0x7f85d815a8d0 2024-04-06T12:28:11.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85dd040700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85d8153a30 0x7f85d815a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85dd040700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85d8153a30 0x7f85d815a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52318/0 (socket says 172.21.15.189:52318) 2024-04-06T12:28:11.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85dd040700 1 -- 172.21.15.189:0/524248905 learned_addr learned my addr 172.21.15.189:0/524248905 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:11.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85dd841700 1 --2- 172.21.15.189:0/524248905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85c4005210 0x7f85d814ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85de042700 1 --2- 172.21.15.189:0/524248905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85d815a8d0 0x7f85d815eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85dd040700 1 -- 172.21.15.189:0/524248905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85d815a8d0 msgr2=0x7f85d815eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:11.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85dd040700 1 --2- 172.21.15.189:0/524248905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85d815a8d0 0x7f85d815eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85dd040700 1 -- 172.21.15.189:0/524248905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85c4005210 msgr2=0x7f85d814ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:11.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85dd040700 1 --2- 172.21.15.189:0/524248905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85c4005210 0x7f85d814ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:11.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85dd040700 1 -- 172.21.15.189:0/524248905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d8152990 con 0x7f85d8153a30 2024-04-06T12:28:11.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85de042700 1 --2- 172.21.15.189:0/524248905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85d815a8d0 0x7f85d815eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:11.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85dd040700 1 --2- 172.21.15.189:0/524248905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85d8153a30 0x7f85d815a390 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f85c8011940 tx=0x7f85c8011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:11.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85ce7fc700 1 -- 172.21.15.189:0/524248905 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8011e90 con 0x7f85d8153a30 2024-04-06T12:28:11.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85ce7fc700 1 -- 172.21.15.189:0/524248905 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c8004240 con 0x7f85d8153a30 2024-04-06T12:28:11.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85ce7fc700 1 -- 172.21.15.189:0/524248905 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c800f730 con 0x7f85d8153a30 2024-04-06T12:28:11.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85dfaa5700 1 -- 172.21.15.189:0/524248905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d8152c70 con 0x7f85d8153a30 2024-04-06T12:28:11.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.752+0000 7f85dfaa5700 1 -- 172.21.15.189:0/524248905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d8153230 con 0x7f85d8153a30 2024-04-06T12:28:11.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.753+0000 7f85dfaa5700 1 -- 172.21.15.189:0/524248905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d8074990 con 0x7f85d8153a30 2024-04-06T12:28:11.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.754+0000 7f85ce7fc700 1 -- 172.21.15.189:0/524248905 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f85c800f890 con 0x7f85d8153a30 2024-04-06T12:28:11.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.754+0000 7f85ce7fc700 1 --2- 172.21.15.189:0/524248905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f85bc06adb0 0x7f85bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:11.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.754+0000 7f85ce7fc700 1 -- 172.21.15.189:0/524248905 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f85c8089e40 con 0x7f85d8153a30 2024-04-06T12:28:11.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.754+0000 7f85dd841700 1 --2- 172.21.15.189:0/524248905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f85bc06adb0 0x7f85bc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:11.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.755+0000 7f85dd841700 1 --2- 172.21.15.189:0/524248905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f85bc06adb0 0x7f85bc06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f85d4000c00 tx=0x7f85d4009040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:11.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.759+0000 7f85ce7fc700 1 -- 172.21.15.189:0/524248905 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f85c8054d30 con 0x7f85d8153a30 2024-04-06T12:28:11.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.999+0000 7f85dfaa5700 1 -- 172.21.15.189:0/524248905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f85d80d1e20 con 0x7f85d8153a30 2024-04-06T12:28:11.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:11.999+0000 7f85ce7fc700 1 -- 172.21.15.189:0/524248905 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v39) v1 ==== 53+0+73 (secure 0 0 0) 0x7f85c8023070 con 0x7f85d8153a30 2024-04-06T12:28:11.998 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-06T12:28:12.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.001+0000 7f85dfaa5700 1 -- 172.21.15.189:0/524248905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f85bc06adb0 msgr2=0x7f85bc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:12.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.001+0000 7f85dfaa5700 1 --2- 172.21.15.189:0/524248905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f85bc06adb0 0x7f85bc06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f85d4000c00 tx=0x7f85d4009040 comp rx=0 tx=0).stop 2024-04-06T12:28:12.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.001+0000 7f85dfaa5700 1 -- 172.21.15.189:0/524248905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85d8153a30 msgr2=0x7f85d815a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:12.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.002+0000 7f85dfaa5700 1 --2- 172.21.15.189:0/524248905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85d8153a30 0x7f85d815a390 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f85c8011940 tx=0x7f85c8011c50 comp rx=0 tx=0).stop 2024-04-06T12:28:12.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.002+0000 7f85dfaa5700 1 -- 172.21.15.189:0/524248905 shutdown_connections 2024-04-06T12:28:12.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.002+0000 7f85dfaa5700 1 --2- 172.21.15.189:0/524248905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85d815a8d0 0x7f85d815eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:12.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.002+0000 7f85dfaa5700 1 --2- 172.21.15.189:0/524248905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f85bc06adb0 0x7f85bc06d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:12.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.002+0000 7f85dfaa5700 1 --2- 172.21.15.189:0/524248905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85c4005210 0x7f85d814ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:12.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.002+0000 7f85dfaa5700 1 --2- 172.21.15.189:0/524248905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85d8153a30 0x7f85d815a390 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:12.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.002+0000 7f85dfaa5700 1 -- 172.21.15.189:0/524248905 >> 172.21.15.189:0/524248905 conn(0x7f85d80b2db0 msgr2=0x7f85d815f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:12.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.002+0000 7f85dfaa5700 1 -- 172.21.15.189:0/524248905 shutdown_connections 2024-04-06T12:28:12.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.002+0000 7f85dfaa5700 1 -- 172.21.15.189:0/524248905 wait complete. 2024-04-06T12:28:12.012 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-06T12:28:12.012 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T12:28:12.012 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-06T12:28:12.012 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-06T12:28:12.012 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T12:28:12.012 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-06T12:28:12.012 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-06T12:28:12.012 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T12:28:12.012 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:28:12.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:12.013 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-06T12:28:12.030 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T12:28:12.030 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T12:28:12.030 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T12:28:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:28:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T12:28:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:28:12.031 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:28:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:28:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-06T12:28:12.031 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:28:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:28:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-06T12:28:12.031 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:28:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:28:12.031 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T12:28:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:28:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T12:28:12.031 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 41398sec preferred_lft 41398sec 2024-04-06T12:28:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::8592:df2c:92e2:dace/64 scope link noprefixroute 2024-04-06T12:28:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:28:12.032 INFO:teuthology.orchestra.run.smithi189.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T12:28:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff 2024-04-06T12:28:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-06T12:28:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:28:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::88e:b6ff:fe6f:1cb0/64 scope link 2024-04-06T12:28:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:28:12.032 INFO:teuthology.orchestra.run.smithi189.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T12:28:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 62:a5:8d:c5:1a:9c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T12:28:12.032 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::60a5:8dff:fec5:1a9c/64 scope link 2024-04-06T12:28:12.033 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:28:12.033 INFO:teuthology.orchestra.run.smithi189.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T12:28:12.033 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-06T12:28:12.033 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::1c7e:c6ff:febb:1dab/64 scope link 2024-04-06T12:28:12.033 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:28:12.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:12.033 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T12:28:12.097 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-06T12:28:12.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:12.098 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-06T12:28:12.159 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-06T12:28:12.159 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T12:28:12.159 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T12:28:12.159 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T12:28:12.159 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T12:28:12.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:12.160 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:28:12.160 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:28:12.160 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T12:28:12.160 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T12:28:12.160 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T12:28:12.160 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:28:12.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:12.265 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T12:28:12.282 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-06T12:28:12.282 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-06T12:28:12.282 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:28:12.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:12.283 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-04-06T12:28:12.350 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:28:12.350 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-06T12:28:12.408 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:28:12.408 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:28:12.409 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-06T12:28:12.466 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T12:28:12.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:12.466 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:28:12.552 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:28:12.552 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:28:12.553 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:28:12.561 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:28:12.561 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:28:12.561 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:28:12.561 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:28:12.561 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:28:12.580 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:28:12.580 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:28:12.580 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:28:12.580 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:28:12.580 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:28:12.580 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790" 2024-04-06T12:28:12.580 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2024-04-06T12:28:12.580 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e30041de-e9a3-4351-8ca7-29d7f3e3728c" will pass to kernel 2024-04-06T12:28:12.582 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T12:28:12.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:12.583 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-06T12:28:12.608 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:28:12.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:12.610 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:28:12.652 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:28:12.694 INFO:teuthology.orchestra.run.smithi189.stdout:client6778 2024-04-06T12:28:12.695 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:28:12.695 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:28:12.759 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-06T12:28:12.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.925+0000 7f41bd62d700 1 Processor -- start 2024-04-06T12:28:12.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.925+0000 7f41bd62d700 1 -- start start 2024-04-06T12:28:12.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.926+0000 7f41bd62d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f41b80b5380 0x7f41b80d2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:12.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.926+0000 7f41bd62d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b80b5cc0 con 0x7f41b80b4ed0 2024-04-06T12:28:12.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.926+0000 7f41bd62d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b80b57d0 con 0x7f41b815d590 2024-04-06T12:28:12.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.926+0000 7f41bd62d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b80d1500 con 0x7f41b80b5380 2024-04-06T12:28:12.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.926+0000 7f41b67fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f41b80b4ed0 0x7f41b80d2880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44620/0 (socket says 172.21.15.189:44620) 2024-04-06T12:28:12.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.926+0000 7f41b67fc700 1 -- 172.21.15.189:0/813229101 learned_addr learned my addr 172.21.15.189:0/813229101 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:12.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.926+0000 7f41b77fe700 1 --2- 172.21.15.189:0/813229101 >> v2:172.21.15.192:3300/0 conn(0x7f41b80b5380 0x7f41b80d2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:12.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.926+0000 7f41b5ffb700 1 -- 172.21.15.189:0/813229101 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331098555 0 0) 0x7f41b80b5cc0 con 0x7f41b80b4ed0 2024-04-06T12:28:12.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.926+0000 7f41b5ffb700 1 -- 172.21.15.189:0/813229101 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41a00035b0 con 0x7f41b80b4ed0 2024-04-06T12:28:12.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.926+0000 7f41b5ffb700 1 -- 172.21.15.189:0/813229101 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1909198573 0 0) 0x7f41a00035b0 con 0x7f41b80b4ed0 2024-04-06T12:28:12.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b5ffb700 1 -- 172.21.15.189:0/813229101 >> v2:172.21.15.192:3300/0 conn(0x7f41b80b5380 msgr2=0x7f41b80d2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:12.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b5ffb700 1 --2- 172.21.15.189:0/813229101 >> v2:172.21.15.192:3300/0 conn(0x7f41b80b5380 0x7f41b80d2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:12.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b5ffb700 1 -- 172.21.15.189:0/813229101 >> v1:172.21.15.192:6789/0 conn(0x7f41b815d590 legacy=0x7f41b80b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:12.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b5ffb700 1 -- 172.21.15.189:0/813229101 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b80d17e0 con 0x7f41b80b4ed0 2024-04-06T12:28:12.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b5ffb700 1 -- 172.21.15.189:0/813229101 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f41a8001670 con 0x7f41b80b4ed0 2024-04-06T12:28:12.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b5ffb700 1 -- 172.21.15.189:0/813229101 >> v1:172.21.15.189:6789/0 conn(0x7f41b80b4ed0 legacy=0x7f41b80d2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:12.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b5ffb700 1 --2- 172.21.15.189:0/813229101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a0003a00 0x7f41a0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:12.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b5ffb700 1 --2- 172.21.15.189:0/813229101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41a0004850 0x7f41a0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:12.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b5ffb700 1 --2- 172.21.15.189:0/813229101 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41a0005210 0x7f41a000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:12.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b5ffb700 1 -- 172.21.15.189:0/813229101 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b80d1500 con 0x7f41a0003a00 2024-04-06T12:28:12.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b5ffb700 1 -- 172.21.15.189:0/813229101 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a000c0b0 con 0x7f41a0004850 2024-04-06T12:28:12.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b5ffb700 1 -- 172.21.15.189:0/813229101 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a000c270 con 0x7f41a0005210 2024-04-06T12:28:12.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b6ffd700 1 --2- 172.21.15.189:0/813229101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a0003a00 0x7f41a0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:12.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b67fc700 1 --2- 172.21.15.189:0/813229101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41a0004850 0x7f41a0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:12.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b77fe700 1 --2- 172.21.15.189:0/813229101 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41a0005210 0x7f41a000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:12.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b6ffd700 1 -- 172.21.15.189:0/813229101 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41a0005210 msgr2=0x7f41a000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:12.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b6ffd700 1 --2- 172.21.15.189:0/813229101 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41a0005210 0x7f41a000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:12.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b6ffd700 1 -- 172.21.15.189:0/813229101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41a0004850 msgr2=0x7f41a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:12.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b6ffd700 1 --2- 172.21.15.189:0/813229101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41a0004850 0x7f41a0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:12.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b6ffd700 1 -- 172.21.15.189:0/813229101 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a00035b0 con 0x7f41a0003a00 2024-04-06T12:28:12.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b77fe700 1 --2- 172.21.15.189:0/813229101 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41a0005210 0x7f41a000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:12.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b77fe700 1 -- 172.21.15.189:0/813229101 reap_dead start 2024-04-06T12:28:12.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b6ffd700 1 --2- 172.21.15.189:0/813229101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a0003a00 0x7f41a0003e20 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f41ac0017c0 tx=0x7f41a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f96272df7adf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:12.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b5ffb700 1 -- 172.21.15.189:0/813229101 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac0052d0 con 0x7f41a0003a00 2024-04-06T12:28:12.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b5ffb700 1 -- 172.21.15.189:0/813229101 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41ac005430 con 0x7f41a0003a00 2024-04-06T12:28:12.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.927+0000 7f41b5ffb700 1 -- 172.21.15.189:0/813229101 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac0038c0 con 0x7f41a0003a00 2024-04-06T12:28:12.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.928+0000 7f41bd62d700 1 -- 172.21.15.189:0/813229101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a0003a00 msgr2=0x7f41a0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:12.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.928+0000 7f41bd62d700 1 --2- 172.21.15.189:0/813229101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a0003a00 0x7f41a0003e20 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f41ac0017c0 tx=0x7f41a0005880 comp rx=0 tx=0).stop 2024-04-06T12:28:12.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.928+0000 7f41bd62d700 1 -- 172.21.15.189:0/813229101 shutdown_connections 2024-04-06T12:28:12.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.928+0000 7f41bd62d700 1 --2- 172.21.15.189:0/813229101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a0003a00 0x7f41a0003e20 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:12.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.928+0000 7f41bd62d700 1 -- 172.21.15.189:0/813229101 >> 172.21.15.189:0/813229101 conn(0x7f41b806bbc0 msgr2=0x7f41b806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:12.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.928+0000 7f41bd62d700 1 -- 172.21.15.189:0/813229101 shutdown_connections 2024-04-06T12:28:12.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.928+0000 7f41bd62d700 1 -- 172.21.15.189:0/813229101 wait complete. 2024-04-06T12:28:12.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.929+0000 7f41bd62d700 1 Processor -- start 2024-04-06T12:28:12.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.929+0000 7f41bd62d700 1 -- start start 2024-04-06T12:28:12.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.930+0000 7f41bd62d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a0003a00 0x7f41b81f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:12.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.930+0000 7f41bd62d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41b80ca520 0x7f41b80ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:12.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.930+0000 7f41bd62d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b80cc8c0 0x7f41b80ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.930+0000 7f41bd62d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b80cd280 con 0x7f41a0003a00 2024-04-06T12:28:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.930+0000 7f41bd62d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b80cb090 con 0x7f41b80ca520 2024-04-06T12:28:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.930+0000 7f41bd62d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b80cb370 con 0x7f41b80cc8c0 2024-04-06T12:28:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.930+0000 7f41b6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a0003a00 0x7f41b81f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.930+0000 7f41b6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a0003a00 0x7f41b81f26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52338/0 (socket says 172.21.15.189:52338) 2024-04-06T12:28:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.930+0000 7f41b6ffd700 1 -- 172.21.15.189:0/2269059205 learned_addr learned my addr 172.21.15.189:0/2269059205 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.930+0000 7f41b6ffd700 1 -- 172.21.15.189:0/2269059205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b80cc8c0 msgr2=0x7f41b80ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.930+0000 7f41b6ffd700 1 --2- 172.21.15.189:0/2269059205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b80cc8c0 0x7f41b80ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.930+0000 7f41b6ffd700 1 -- 172.21.15.189:0/2269059205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41b80ca520 msgr2=0x7f41b80ca9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.930+0000 7f41b6ffd700 1 --2- 172.21.15.189:0/2269059205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41b80ca520 0x7f41b80ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.930+0000 7f41b6ffd700 1 -- 172.21.15.189:0/2269059205 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b80cb650 con 0x7f41a0003a00 2024-04-06T12:28:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.931+0000 7f41b6ffd700 1 --2- 172.21.15.189:0/2269059205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a0003a00 0x7f41b81f26e0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f41ac005040 tx=0x7f41ac001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.931+0000 7f419f7fe700 1 -- 172.21.15.189:0/2269059205 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac001290 con 0x7f41a0003a00 2024-04-06T12:28:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.931+0000 7f419f7fe700 1 -- 172.21.15.189:0/2269059205 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41ac01d480 con 0x7f41a0003a00 2024-04-06T12:28:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.931+0000 7f41bd62d700 1 -- 172.21.15.189:0/2269059205 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b80cb930 con 0x7f41a0003a00 2024-04-06T12:28:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.931+0000 7f419f7fe700 1 -- 172.21.15.189:0/2269059205 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac01d810 con 0x7f41a0003a00 2024-04-06T12:28:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.931+0000 7f41bd62d700 1 -- 172.21.15.189:0/2269059205 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b8146600 con 0x7f41a0003a00 2024-04-06T12:28:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.932+0000 7f41bd62d700 1 -- 172.21.15.189:0/2269059205 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b80cbf80 con 0x7f41a0003a00 2024-04-06T12:28:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.932+0000 7f419f7fe700 1 -- 172.21.15.189:0/2269059205 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f41ac019040 con 0x7f41a0003a00 2024-04-06T12:28:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.932+0000 7f419f7fe700 1 --2- 172.21.15.189:0/2269059205 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f41a406adb0 0x7f41a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.933+0000 7f419f7fe700 1 -- 172.21.15.189:0/2269059205 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f41ac08a080 con 0x7f41a0003a00 2024-04-06T12:28:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.933+0000 7f41b67fc700 1 --2- 172.21.15.189:0/2269059205 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f41a406adb0 0x7f41a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.933+0000 7f41b67fc700 1 --2- 172.21.15.189:0/2269059205 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f41a406adb0 0x7f41a406d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f41a8001050 tx=0x7f41a8000fe0 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:12.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:12.937+0000 7f419f7fe700 1 -- 172.21.15.189:0/2269059205 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f41ac0551c0 con 0x7f41a0003a00 2024-04-06T12:28:13.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.159+0000 7f41bd62d700 1 -- 172.21.15.189:0/2269059205 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f41b80cd280 con 0x7f41a406adb0 2024-04-06T12:28:13.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.160+0000 7f419f7fe700 1 -- 172.21.15.189:0/2269059205 <== mgr.6442 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f41b80cd280 con 0x7f41a406adb0 2024-04-06T12:28:13.158 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T12:28:13.158 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:13.158 INFO:teuthology.orchestra.run.smithi189.stdout: "name": "cephfs" 2024-04-06T12:28:13.159 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:13.159 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T12:28:13.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.162+0000 7f41bd62d700 1 -- 172.21.15.189:0/2269059205 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f41a406adb0 msgr2=0x7f41a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:13.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.162+0000 7f41bd62d700 1 --2- 172.21.15.189:0/2269059205 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f41a406adb0 0x7f41a406d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f41a8001050 tx=0x7f41a8000fe0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.162+0000 7f41bd62d700 1 -- 172.21.15.189:0/2269059205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a0003a00 msgr2=0x7f41b81f26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:13.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.162+0000 7f41bd62d700 1 --2- 172.21.15.189:0/2269059205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a0003a00 0x7f41b81f26e0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f41ac005040 tx=0x7f41ac001120 comp rx=0 tx=0).stop 2024-04-06T12:28:13.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.162+0000 7f41bd62d700 1 -- 172.21.15.189:0/2269059205 shutdown_connections 2024-04-06T12:28:13.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.162+0000 7f41bd62d700 1 --2- 172.21.15.189:0/2269059205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b80cc8c0 0x7f41b80ccd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.162+0000 7f41bd62d700 1 --2- 172.21.15.189:0/2269059205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a0003a00 0x7f41b81f26e0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.162+0000 7f41bd62d700 1 --2- 172.21.15.189:0/2269059205 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f41a406adb0 0x7f41a406d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.162+0000 7f41bd62d700 1 --2- 172.21.15.189:0/2269059205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41b80ca520 0x7f41b80ca9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.162+0000 7f41bd62d700 1 -- 172.21.15.189:0/2269059205 >> 172.21.15.189:0/2269059205 conn(0x7f41b806bbc0 msgr2=0x7f41b80b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:13.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.162+0000 7f41bd62d700 1 -- 172.21.15.189:0/2269059205 shutdown_connections 2024-04-06T12:28:13.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.162+0000 7f41bd62d700 1 -- 172.21.15.189:0/2269059205 wait complete. 2024-04-06T12:28:13.173 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-06T12:28:13.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.299+0000 7f09621d8700 1 Processor -- start 2024-04-06T12:28:13.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.299+0000 7f09621d8700 1 -- start start 2024-04-06T12:28:13.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.300+0000 7f09621d8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f095c15d7b0 0x7f095c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:13.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.300+0000 7f09621d8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095c0b3990 con 0x7f095c06a730 2024-04-06T12:28:13.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.300+0000 7f09621d8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095c0b3060 con 0x7f095c15e500 2024-04-06T12:28:13.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.300+0000 7f09621d8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095c0b3340 con 0x7f095c15d7b0 2024-04-06T12:28:13.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.300+0000 7f095affd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f095c06a730 0x7f095c15ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44624/0 (socket says 172.21.15.189:44624) 2024-04-06T12:28:13.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.300+0000 7f095affd700 1 -- 172.21.15.189:0/687278678 learned_addr learned my addr 172.21.15.189:0/687278678 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:13.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095a7fc700 1 -- 172.21.15.189:0/687278678 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144747800 0 0) 0x7f095c0b3990 con 0x7f095c06a730 2024-04-06T12:28:13.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095b7fe700 1 --2- 172.21.15.189:0/687278678 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f095c15d7b0 0x7f095c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:13.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095a7fc700 1 -- 172.21.15.189:0/687278678 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09480035b0 con 0x7f095c06a730 2024-04-06T12:28:13.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095a7fc700 1 -- 172.21.15.189:0/687278678 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 53816155 0 0) 0x7f09480035b0 con 0x7f095c06a730 2024-04-06T12:28:13.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095a7fc700 1 -- 172.21.15.189:0/687278678 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f095c15d7b0 msgr2=0x7f095c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:13.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095a7fc700 1 --2- 172.21.15.189:0/687278678 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f095c15d7b0 0x7f095c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095a7fc700 1 -- 172.21.15.189:0/687278678 >> v1:172.21.15.192:6789/0 conn(0x7f095c15e500 legacy=0x7f095c0717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:13.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095a7fc700 1 -- 172.21.15.189:0/687278678 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095c0b3620 con 0x7f095c06a730 2024-04-06T12:28:13.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095a7fc700 1 -- 172.21.15.189:0/687278678 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f094c001670 con 0x7f095c06a730 2024-04-06T12:28:13.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095a7fc700 1 -- 172.21.15.189:0/687278678 >> v1:172.21.15.189:6789/0 conn(0x7f095c06a730 legacy=0x7f095c15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:13.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095a7fc700 1 --2- 172.21.15.189:0/687278678 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f095c15d7b0 0x7f0948003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:13.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095a7fc700 1 --2- 172.21.15.189:0/687278678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0948003fb0 0x7f0948004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:13.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095a7fc700 1 --2- 172.21.15.189:0/687278678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0948004d90 0x7f0948005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:13.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095a7fc700 1 -- 172.21.15.189:0/687278678 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c0b3340 con 0x7f0948004d90 2024-04-06T12:28:13.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095a7fc700 1 -- 172.21.15.189:0/687278678 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094800bba0 con 0x7f0948003fb0 2024-04-06T12:28:13.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095a7fc700 1 -- 172.21.15.189:0/687278678 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f094800bd60 con 0x7f095c15d7b0 2024-04-06T12:28:13.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095bfff700 1 --2- 172.21.15.189:0/687278678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0948004d90 0x7f0948005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:13.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095affd700 1 --2- 172.21.15.189:0/687278678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0948003fb0 0x7f0948004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:13.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095b7fe700 1 --2- 172.21.15.189:0/687278678 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f095c15d7b0 0x7f0948003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:13.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095bfff700 1 -- 172.21.15.189:0/687278678 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f095c15d7b0 msgr2=0x7f0948003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:13.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095bfff700 1 --2- 172.21.15.189:0/687278678 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f095c15d7b0 0x7f0948003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095bfff700 1 -- 172.21.15.189:0/687278678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0948003fb0 msgr2=0x7f0948004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:13.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095bfff700 1 --2- 172.21.15.189:0/687278678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0948003fb0 0x7f0948004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095bfff700 1 -- 172.21.15.189:0/687278678 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09480035b0 con 0x7f0948004d90 2024-04-06T12:28:13.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.301+0000 7f095bfff700 1 --2- 172.21.15.189:0/687278678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0948004d90 0x7f0948005210 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f095400c190 tx=0x7f0948005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d7d809b1f39ef3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:13.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.302+0000 7f095a7fc700 1 -- 172.21.15.189:0/687278678 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0954001db0 con 0x7f0948004d90 2024-04-06T12:28:13.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.302+0000 7f095a7fc700 1 -- 172.21.15.189:0/687278678 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095400ccd0 con 0x7f0948004d90 2024-04-06T12:28:13.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.302+0000 7f095a7fc700 1 -- 172.21.15.189:0/687278678 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0954003c40 con 0x7f0948004d90 2024-04-06T12:28:13.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.302+0000 7f09621d8700 1 -- 172.21.15.189:0/687278678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0948004d90 msgr2=0x7f0948005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:13.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.302+0000 7f09621d8700 1 --2- 172.21.15.189:0/687278678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0948004d90 0x7f0948005210 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f095400c190 tx=0x7f0948005880 comp rx=0 tx=0).stop 2024-04-06T12:28:13.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.302+0000 7f095bfff700 1 -- 172.21.15.189:0/687278678 reap_dead start 2024-04-06T12:28:13.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.302+0000 7f09621d8700 1 -- 172.21.15.189:0/687278678 shutdown_connections 2024-04-06T12:28:13.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.302+0000 7f09621d8700 1 -- 172.21.15.189:0/687278678 >> 172.21.15.189:0/687278678 conn(0x7f095c0d00b0 msgr2=0x7f095c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:13.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.302+0000 7f09621d8700 1 -- 172.21.15.189:0/687278678 shutdown_connections 2024-04-06T12:28:13.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.302+0000 7f09621d8700 1 -- 172.21.15.189:0/687278678 wait complete. 2024-04-06T12:28:13.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.303+0000 7f09621d8700 1 Processor -- start 2024-04-06T12:28:13.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.304+0000 7f09621d8700 1 -- start start 2024-04-06T12:28:13.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.304+0000 7f09621d8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f095c159820 0x7f095c0b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:13.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.304+0000 7f09621d8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f095c0b12f0 0x7f095c155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:13.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.304+0000 7f09621d8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f095c155690 0x7f095c15fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:13.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.304+0000 7f09621d8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c1f6d70 con 0x7f095c155690 2024-04-06T12:28:13.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.304+0000 7f09621d8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c0b1770 con 0x7f095c0b12f0 2024-04-06T12:28:13.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.304+0000 7f09621d8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095c1f60f0 con 0x7f095c159820 2024-04-06T12:28:13.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.304+0000 7f095bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f095c155690 0x7f095c15fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:13.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.304+0000 7f095bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f095c155690 0x7f095c15fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52356/0 (socket says 172.21.15.189:52356) 2024-04-06T12:28:13.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.304+0000 7f095bfff700 1 -- 172.21.15.189:0/894976374 learned_addr learned my addr 172.21.15.189:0/894976374 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:13.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.305+0000 7f095b7fe700 1 --2- 172.21.15.189:0/894976374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f095c159820 0x7f095c0b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:13.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.305+0000 7f095affd700 1 --2- 172.21.15.189:0/894976374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f095c0b12f0 0x7f095c155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:13.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.305+0000 7f095bfff700 1 -- 172.21.15.189:0/894976374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f095c159820 msgr2=0x7f095c0b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:13.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.305+0000 7f095bfff700 1 --2- 172.21.15.189:0/894976374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f095c159820 0x7f095c0b0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.305+0000 7f095bfff700 1 -- 172.21.15.189:0/894976374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f095c0b12f0 msgr2=0x7f095c155150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:13.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.305+0000 7f095bfff700 1 --2- 172.21.15.189:0/894976374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f095c0b12f0 0x7f095c155150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.305+0000 7f095bfff700 1 -- 172.21.15.189:0/894976374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095c1f63d0 con 0x7f095c155690 2024-04-06T12:28:13.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.305+0000 7f095b7fe700 1 --2- 172.21.15.189:0/894976374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f095c159820 0x7f095c0b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:13.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.305+0000 7f095affd700 1 --2- 172.21.15.189:0/894976374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f095c0b12f0 0x7f095c155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:13.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.305+0000 7f095bfff700 1 --2- 172.21.15.189:0/894976374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f095c155690 0x7f095c15fda0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f0954005040 tx=0x7f09540014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:13.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.305+0000 7f0953fff700 1 -- 172.21.15.189:0/894976374 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09540050d0 con 0x7f095c155690 2024-04-06T12:28:13.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.305+0000 7f0953fff700 1 -- 172.21.15.189:0/894976374 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0954005230 con 0x7f095c155690 2024-04-06T12:28:13.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.305+0000 7f09621d8700 1 -- 172.21.15.189:0/894976374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095c1f66b0 con 0x7f095c155690 2024-04-06T12:28:13.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.305+0000 7f0953fff700 1 -- 172.21.15.189:0/894976374 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09540053a0 con 0x7f095c155690 2024-04-06T12:28:13.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.305+0000 7f09621d8700 1 -- 172.21.15.189:0/894976374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f095c155b40 con 0x7f095c155690 2024-04-06T12:28:13.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.306+0000 7f09621d8700 1 -- 172.21.15.189:0/894976374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f095c1f6eb0 con 0x7f095c155690 2024-04-06T12:28:13.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.307+0000 7f0953fff700 1 -- 172.21.15.189:0/894976374 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 41) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0954019040 con 0x7f095c155690 2024-04-06T12:28:13.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.307+0000 7f0953fff700 1 --2- 172.21.15.189:0/894976374 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f093c06adb0 0x7f093c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:13.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.307+0000 7f095b7fe700 1 --2- 172.21.15.189:0/894976374 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f093c06adb0 0x7f093c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:13.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.307+0000 7f0953fff700 1 -- 172.21.15.189:0/894976374 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0954089bf0 con 0x7f095c155690 2024-04-06T12:28:13.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.307+0000 7f095b7fe700 1 --2- 172.21.15.189:0/894976374 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f093c06adb0 0x7f093c06d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f095c158200 tx=0x7f0944008040 comp rx=0 tx=0).ready entity=mgr.6442 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:13.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.312+0000 7f0953fff700 1 -- 172.21.15.189:0/894976374 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f095c1f6eb0 con 0x7f095c155690 2024-04-06T12:28:13.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.562+0000 7f09621d8700 1 -- 172.21.15.189:0/894976374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f095c1f6eb0 con 0x7f095c155690 2024-04-06T12:28:13.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.706+0000 7f0953fff700 1 -- 172.21.15.189:0/894976374 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 42) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f0954055090 con 0x7f095c155690 2024-04-06T12:28:13.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.708+0000 7f0953fff700 1 -- 172.21.15.189:0/894976374 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v42) v1 ==== 92+0+0 (secure 0 0 0) 0x7f095406e060 con 0x7f095c155690 2024-04-06T12:28:13.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.710+0000 7f09621d8700 1 -- 172.21.15.189:0/894976374 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f093c06adb0 msgr2=0x7f093c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:13.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.710+0000 7f09621d8700 1 --2- 172.21.15.189:0/894976374 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f093c06adb0 0x7f093c06d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f095c158200 tx=0x7f0944008040 comp rx=0 tx=0).stop 2024-04-06T12:28:13.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.710+0000 7f09621d8700 1 -- 172.21.15.189:0/894976374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f095c155690 msgr2=0x7f095c15fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:13.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.710+0000 7f09621d8700 1 --2- 172.21.15.189:0/894976374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f095c155690 0x7f095c15fda0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f0954005040 tx=0x7f09540014a0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.710+0000 7f09621d8700 1 -- 172.21.15.189:0/894976374 shutdown_connections 2024-04-06T12:28:13.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.710+0000 7f09621d8700 1 --2- 172.21.15.189:0/894976374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f095c155690 0x7f095c15fda0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.710+0000 7f09621d8700 1 --2- 172.21.15.189:0/894976374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f095c159820 0x7f095c0b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.710+0000 7f09621d8700 1 --2- 172.21.15.189:0/894976374 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f093c06adb0 0x7f093c06d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.710+0000 7f09621d8700 1 --2- 172.21.15.189:0/894976374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f095c0b12f0 0x7f095c155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.710+0000 7f09621d8700 1 -- 172.21.15.189:0/894976374 >> 172.21.15.189:0/894976374 conn(0x7f095c0d00b0 msgr2=0x7f095c0ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:13.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.710+0000 7f09621d8700 1 -- 172.21.15.189:0/894976374 shutdown_connections 2024-04-06T12:28:13.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.711+0000 7f09621d8700 1 -- 172.21.15.189:0/894976374 wait complete. 2024-04-06T12:28:13.721 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-06T12:28:13.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.852+0000 7fadff9a7700 1 Processor -- start 2024-04-06T12:28:13.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.853+0000 7fadff9a7700 1 -- start start 2024-04-06T12:28:13.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.854+0000 7fadff9a7700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fadf815d7e0 0x7fadf80d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:13.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.854+0000 7fadff9a7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadf80d0890 0x7fadf80d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:13.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.854+0000 7fadff9a7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf80cdf80 con 0x7fadf80d1af0 2024-04-06T12:28:13.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.854+0000 7fadff9a7700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf80d1f80 con 0x7fadf815d7e0 2024-04-06T12:28:13.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.854+0000 7fadff9a7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf80d3230 con 0x7fadf80d0890 2024-04-06T12:28:13.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.854+0000 7fadfdf44700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fadf80d1af0 0x7fadf80b4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44632/0 (socket says 172.21.15.189:44632) 2024-04-06T12:28:13.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.854+0000 7fadfdf44700 1 -- 172.21.15.189:0/1839502257 learned_addr learned my addr 172.21.15.189:0/1839502257 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:13.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.854+0000 7fadfd743700 1 --2- 172.21.15.189:0/1839502257 >> v2:172.21.15.192:3300/0 conn(0x7fadf815d7e0 0x7fadf80d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:13.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.854+0000 7fadfcf42700 1 --2- 172.21.15.189:0/1839502257 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadf80d0890 0x7fadf80d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:13.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.854+0000 7fadeffff700 1 -- 172.21.15.189:0/1839502257 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721116680 0 0) 0x7fadf80cdf80 con 0x7fadf80d1af0 2024-04-06T12:28:13.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.854+0000 7fadeffff700 1 -- 172.21.15.189:0/1839502257 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade40035b0 con 0x7fadf80d1af0 2024-04-06T12:28:13.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.854+0000 7fadfcf42700 1 -- 172.21.15.189:0/1839502257 >> v2:172.21.15.192:3300/0 conn(0x7fadf815d7e0 msgr2=0x7fadf80d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:13.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.854+0000 7fadfcf42700 1 --2- 172.21.15.189:0/1839502257 >> v2:172.21.15.192:3300/0 conn(0x7fadf815d7e0 0x7fadf80d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.854+0000 7fadfcf42700 1 -- 172.21.15.189:0/1839502257 >> v1:172.21.15.189:6789/0 conn(0x7fadf80d1af0 legacy=0x7fadf80b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:13.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.854+0000 7fadfcf42700 1 -- 172.21.15.189:0/1839502257 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf80d0cf0 con 0x7fadf80d0890 2024-04-06T12:28:13.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.855+0000 7fadfd743700 1 --2- 172.21.15.189:0/1839502257 >> v2:172.21.15.192:3300/0 conn(0x7fadf815d7e0 0x7fadf80d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:28:13.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.855+0000 7fadfcf42700 1 --2- 172.21.15.189:0/1839502257 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadf80d0890 0x7fadf80d15b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fade8009840 tx=0x7fade8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84527efdf6d702b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:13.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.855+0000 7fadeffff700 1 -- 172.21.15.189:0/1839502257 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade8011830 con 0x7fadf80d0890 2024-04-06T12:28:13.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.855+0000 7fadeffff700 1 -- 172.21.15.189:0/1839502257 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade8011990 con 0x7fadf80d0890 2024-04-06T12:28:13.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.855+0000 7fadeffff700 1 -- 172.21.15.189:0/1839502257 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade8011d20 con 0x7fadf80d0890 2024-04-06T12:28:13.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.856+0000 7fadff9a7700 1 -- 172.21.15.189:0/1839502257 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadf80d0890 msgr2=0x7fadf80d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:13.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.856+0000 7fadff9a7700 1 --2- 172.21.15.189:0/1839502257 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadf80d0890 0x7fadf80d15b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fade8009840 tx=0x7fade8009b50 comp rx=0 tx=0).stop 2024-04-06T12:28:13.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.856+0000 7fadff9a7700 1 -- 172.21.15.189:0/1839502257 shutdown_connections 2024-04-06T12:28:13.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.856+0000 7fadff9a7700 1 --2- 172.21.15.189:0/1839502257 >> v2:172.21.15.192:3300/0 conn(0x7fadf815d7e0 0x7fadf80d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.856+0000 7fadff9a7700 1 --2- 172.21.15.189:0/1839502257 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadf80d0890 0x7fadf80d15b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.856+0000 7fadff9a7700 1 -- 172.21.15.189:0/1839502257 >> 172.21.15.189:0/1839502257 conn(0x7fadf8072250 msgr2=0x7fadf8072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:13.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.856+0000 7fadff9a7700 1 -- 172.21.15.189:0/1839502257 shutdown_connections 2024-04-06T12:28:13.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.856+0000 7fadff9a7700 1 -- 172.21.15.189:0/1839502257 wait complete. 2024-04-06T12:28:13.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.858+0000 7fadff9a7700 1 Processor -- start 2024-04-06T12:28:13.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.858+0000 7fadff9a7700 1 -- start start 2024-04-06T12:28:13.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.859+0000 7fadff9a7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadf80d0890 0x7fadf81f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:13.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.859+0000 7fadff9a7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadf80d1af0 0x7fadf814be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:13.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.859+0000 7fadff9a7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fadf815d7e0 0x7fadf80c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:13.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.859+0000 7fadff9a7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf81f64a0 con 0x7fadf80d1af0 2024-04-06T12:28:13.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.859+0000 7fadff9a7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf80c86b0 con 0x7fadf815d7e0 2024-04-06T12:28:13.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.859+0000 7fadff9a7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf8148d00 con 0x7fadf80d0890 2024-04-06T12:28:13.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.859+0000 7fadfcf42700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadf80d1af0 0x7fadf814be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:13.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.859+0000 7fadfcf42700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadf80d1af0 0x7fadf814be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52382/0 (socket says 172.21.15.189:52382) 2024-04-06T12:28:13.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.859+0000 7fadfcf42700 1 -- 172.21.15.189:0/574059474 learned_addr learned my addr 172.21.15.189:0/574059474 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:13.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.859+0000 7fadfd743700 1 --2- 172.21.15.189:0/574059474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadf80d0890 0x7fadf81f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:13.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.859+0000 7fadfdf44700 1 --2- 172.21.15.189:0/574059474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fadf815d7e0 0x7fadf80c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:13.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.859+0000 7fadfcf42700 1 -- 172.21.15.189:0/574059474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadf80d0890 msgr2=0x7fadf81f5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:13.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.859+0000 7fadfcf42700 1 --2- 172.21.15.189:0/574059474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadf80d0890 0x7fadf81f5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.859+0000 7fadfcf42700 1 -- 172.21.15.189:0/574059474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fadf815d7e0 msgr2=0x7fadf80c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:13.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.859+0000 7fadfcf42700 1 --2- 172.21.15.189:0/574059474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fadf815d7e0 0x7fadf80c6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:13.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.859+0000 7fadfcf42700 1 -- 172.21.15.189:0/574059474 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf40094a0 con 0x7fadf80d1af0 2024-04-06T12:28:13.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.859+0000 7fadfdf44700 1 --2- 172.21.15.189:0/574059474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fadf815d7e0 0x7fadf80c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:13.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.860+0000 7fadfcf42700 1 --2- 172.21.15.189:0/574059474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadf80d1af0 0x7fadf814be80 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fade8009fd0 tx=0x7fade8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:13.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.860+0000 7fadedffb700 1 -- 172.21.15.189:0/574059474 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade80036a0 con 0x7fadf80d1af0 2024-04-06T12:28:13.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.860+0000 7fadedffb700 1 -- 172.21.15.189:0/574059474 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade801e070 con 0x7fadf80d1af0 2024-04-06T12:28:13.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.860+0000 7fadedffb700 1 -- 172.21.15.189:0/574059474 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade8003a60 con 0x7fadf80d1af0 2024-04-06T12:28:13.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.860+0000 7fadff9a7700 1 -- 172.21.15.189:0/574059474 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade80094a0 con 0x7fadf80d1af0 2024-04-06T12:28:13.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.860+0000 7fadff9a7700 1 -- 172.21.15.189:0/574059474 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf81492c0 con 0x7fadf80d1af0 2024-04-06T12:28:13.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.861+0000 7fadff9a7700 1 -- 172.21.15.189:0/574059474 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf806be20 con 0x7fadf80d1af0 2024-04-06T12:28:13.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.862+0000 7fadedffb700 1 -- 172.21.15.189:0/574059474 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 42) v1 ==== 89264+0+0 (secure 0 0 0) 0x7fade8003bc0 con 0x7fadf80d1af0 2024-04-06T12:28:13.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.862+0000 7fadedffb700 1 --2- 172.21.15.189:0/574059474 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faddc06ae00 0x7faddc06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:13.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.862+0000 7fadfd743700 1 -- 172.21.15.189:0/574059474 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faddc06ae00 msgr2=0x7faddc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:13.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.863+0000 7fadedffb700 1 -- 172.21.15.189:0/574059474 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fade808a910 con 0x7fadf80d1af0 2024-04-06T12:28:13.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.863+0000 7fadfd743700 1 --2- 172.21.15.189:0/574059474 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faddc06ae00 0x7faddc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:13.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:13.866+0000 7fadedffb700 1 -- 172.21.15.189:0/574059474 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fade8055910 con 0x7fadf80d1af0 2024-04-06T12:28:14.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.064+0000 7fadfd743700 1 -- 172.21.15.189:0/574059474 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faddc06ae00 msgr2=0x7faddc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:14.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.064+0000 7fadfd743700 1 --2- 172.21.15.189:0/574059474 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faddc06ae00 0x7faddc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.071+0000 7fadff9a7700 1 -- 172.21.15.189:0/574059474 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fadf8074990 con 0x7fadf80d1af0 2024-04-06T12:28:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.075+0000 7fadedffb700 1 -- 172.21.15.189:0/574059474 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7fade8058f30 con 0x7fadf80d1af0 2024-04-06T12:28:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.077+0000 7fadff9a7700 1 -- 172.21.15.189:0/574059474 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faddc06ae00 msgr2=0x7faddc06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.077+0000 7fadff9a7700 1 --2- 172.21.15.189:0/574059474 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faddc06ae00 0x7faddc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.077+0000 7fadff9a7700 1 -- 172.21.15.189:0/574059474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadf80d1af0 msgr2=0x7fadf814be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.077+0000 7fadff9a7700 1 --2- 172.21.15.189:0/574059474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadf80d1af0 0x7fadf814be80 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fade8009fd0 tx=0x7fade8004750 comp rx=0 tx=0).stop 2024-04-06T12:28:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.077+0000 7fadff9a7700 1 -- 172.21.15.189:0/574059474 shutdown_connections 2024-04-06T12:28:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.077+0000 7fadff9a7700 1 --2- 172.21.15.189:0/574059474 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faddc06ae00 0x7faddc06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.077+0000 7fadff9a7700 1 --2- 172.21.15.189:0/574059474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fadf815d7e0 0x7fadf80c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.077+0000 7fadff9a7700 1 --2- 172.21.15.189:0/574059474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadf80d0890 0x7fadf81f5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.077+0000 7fadff9a7700 1 --2- 172.21.15.189:0/574059474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadf80d1af0 0x7fadf814be80 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.077+0000 7fadff9a7700 1 -- 172.21.15.189:0/574059474 >> 172.21.15.189:0/574059474 conn(0x7fadf8072250 msgr2=0x7fadf80b8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:14.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.077+0000 7fadff9a7700 1 -- 172.21.15.189:0/574059474 shutdown_connections 2024-04-06T12:28:14.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.077+0000 7fadff9a7700 1 -- 172.21.15.189:0/574059474 wait complete. 2024-04-06T12:28:14.087 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:14.089+0000 7fe5e5b66200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:28:14.088 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-06T12:28:14.106 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:14.108+0000 7fd7c0918200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:28:14.167 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:14.168+0000 7fe5e5b66200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:28:14.190 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:14.191+0000 7fd7c0918200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:28:14.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.225+0000 7f3810c4c700 1 Processor -- start 2024-04-06T12:28:14.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.225+0000 7f3810c4c700 1 -- start start 2024-04-06T12:28:14.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.226+0000 7f3810c4c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f380c15d590 0x7f380c0b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:14.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.226+0000 7f3810c4c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f380c0b4ed0 0x7f380c0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:14.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.226+0000 7f3810c4c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380c0b5cc0 con 0x7f380c0b6580 2024-04-06T12:28:14.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.226+0000 7f3810c4c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f380c0b5890 con 0x7f380c15d590 2024-04-06T12:28:14.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.226+0000 7f3810c4c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380c0d1500 con 0x7f380c0b4ed0 2024-04-06T12:28:14.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.226+0000 7f380ad9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f380c0b6580 0x7f380c0b6a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44638/0 (socket says 172.21.15.189:44638) 2024-04-06T12:28:14.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.226+0000 7f380ad9d700 1 -- 172.21.15.189:0/2183669897 learned_addr learned my addr 172.21.15.189:0/2183669897 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:14.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.226+0000 7f3809d9b700 1 --2- 172.21.15.189:0/2183669897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f380c0b4ed0 0x7f380c0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:14.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.226+0000 7f380a59c700 1 --2- 172.21.15.189:0/2183669897 >> v2:172.21.15.192:3300/0 conn(0x7f380c15d590 0x7f380c0b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:14.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.227+0000 7f380959a700 1 -- 172.21.15.189:0/2183669897 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871216042 0 0) 0x7f380c0b5cc0 con 0x7f380c0b6580 2024-04-06T12:28:14.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.227+0000 7f380959a700 1 -- 172.21.15.189:0/2183669897 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f40035b0 con 0x7f380c0b6580 2024-04-06T12:28:14.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.227+0000 7f3809d9b700 1 -- 172.21.15.189:0/2183669897 >> v2:172.21.15.192:3300/0 conn(0x7f380c15d590 msgr2=0x7f380c0b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:14.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.227+0000 7f3809d9b700 1 --2- 172.21.15.189:0/2183669897 >> v2:172.21.15.192:3300/0 conn(0x7f380c15d590 0x7f380c0b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.227+0000 7f3809d9b700 1 -- 172.21.15.189:0/2183669897 >> v1:172.21.15.189:6789/0 conn(0x7f380c0b6580 legacy=0x7f380c0b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.227+0000 7f3809d9b700 1 -- 172.21.15.189:0/2183669897 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380c0d17e0 con 0x7f380c0b4ed0 2024-04-06T12:28:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.227+0000 7f380a59c700 1 --2- 172.21.15.189:0/2183669897 >> v2:172.21.15.192:3300/0 conn(0x7f380c15d590 0x7f380c0b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:28:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.227+0000 7f3809d9b700 1 --2- 172.21.15.189:0/2183669897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f380c0b4ed0 0x7f380c0b5350 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3800009870 tx=0x7f3800009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=522c12514e1e5af2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.227+0000 7f380959a700 1 -- 172.21.15.189:0/2183669897 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3800011830 con 0x7f380c0b4ed0 2024-04-06T12:28:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.227+0000 7f380959a700 1 -- 172.21.15.189:0/2183669897 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3800011990 con 0x7f380c0b4ed0 2024-04-06T12:28:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.227+0000 7f380959a700 1 -- 172.21.15.189:0/2183669897 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3800011d20 con 0x7f380c0b4ed0 2024-04-06T12:28:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.228+0000 7f3810c4c700 1 -- 172.21.15.189:0/2183669897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f380c0b4ed0 msgr2=0x7f380c0b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.228+0000 7f3810c4c700 1 --2- 172.21.15.189:0/2183669897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f380c0b4ed0 0x7f380c0b5350 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3800009870 tx=0x7f3800009b80 comp rx=0 tx=0).stop 2024-04-06T12:28:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.228+0000 7f3810c4c700 1 -- 172.21.15.189:0/2183669897 shutdown_connections 2024-04-06T12:28:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.228+0000 7f3810c4c700 1 --2- 172.21.15.189:0/2183669897 >> v2:172.21.15.192:3300/0 conn(0x7f380c15d590 0x7f380c0b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.228+0000 7f3810c4c700 1 --2- 172.21.15.189:0/2183669897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f380c0b4ed0 0x7f380c0b5350 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.228+0000 7f3810c4c700 1 -- 172.21.15.189:0/2183669897 >> 172.21.15.189:0/2183669897 conn(0x7f380c06bbc0 msgr2=0x7f380c06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.228+0000 7f3810c4c700 1 -- 172.21.15.189:0/2183669897 shutdown_connections 2024-04-06T12:28:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.228+0000 7f3810c4c700 1 -- 172.21.15.189:0/2183669897 wait complete. 2024-04-06T12:28:14.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.229+0000 7f3810c4c700 1 Processor -- start 2024-04-06T12:28:14.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.229+0000 7f3810c4c700 1 -- start start 2024-04-06T12:28:14.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.230+0000 7f3810c4c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f380c0b4ed0 0x7f380c158610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:14.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.230+0000 7f3810c4c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f380c0b6580 0x7f380c0b71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:14.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.230+0000 7f3810c4c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f380c15d590 0x7f380c0b7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:14.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.230+0000 7f3810c4c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c15e4c0 con 0x7f380c0b6580 2024-04-06T12:28:14.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.230+0000 7f3810c4c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c158ff0 con 0x7f380c0b4ed0 2024-04-06T12:28:14.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.230+0000 7f3810c4c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380c1592d0 con 0x7f380c15d590 2024-04-06T12:28:14.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.230+0000 7f3809d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f380c0b6580 0x7f380c0b71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:14.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.230+0000 7f3809d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f380c0b6580 0x7f380c0b71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52398/0 (socket says 172.21.15.189:52398) 2024-04-06T12:28:14.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.230+0000 7f380a59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f380c0b4ed0 0x7f380c158610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:14.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.230+0000 7f3809d9b700 1 -- 172.21.15.189:0/3342876327 learned_addr learned my addr 172.21.15.189:0/3342876327 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:14.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.230+0000 7f380ad9d700 1 --2- 172.21.15.189:0/3342876327 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f380c15d590 0x7f380c0b7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:14.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.231+0000 7f3809d9b700 1 -- 172.21.15.189:0/3342876327 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f380c15d590 msgr2=0x7f380c0b7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:14.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.231+0000 7f3809d9b700 1 --2- 172.21.15.189:0/3342876327 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f380c15d590 0x7f380c0b7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.231+0000 7f3809d9b700 1 -- 172.21.15.189:0/3342876327 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f380c0b4ed0 msgr2=0x7f380c158610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:14.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.231+0000 7f3809d9b700 1 --2- 172.21.15.189:0/3342876327 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f380c0b4ed0 0x7f380c158610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.231+0000 7f3809d9b700 1 -- 172.21.15.189:0/3342876327 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37fc0094a0 con 0x7f380c0b6580 2024-04-06T12:28:14.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.231+0000 7f3809d9b700 1 --2- 172.21.15.189:0/3342876327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f380c0b6580 0x7f380c0b71f0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f3800000c00 tx=0x7f3800004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:14.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.231+0000 7f37faffd700 1 -- 172.21.15.189:0/3342876327 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38000036a0 con 0x7f380c0b6580 2024-04-06T12:28:14.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.231+0000 7f37faffd700 1 -- 172.21.15.189:0/3342876327 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380001f070 con 0x7f380c0b6580 2024-04-06T12:28:14.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.231+0000 7f3810c4c700 1 -- 172.21.15.189:0/3342876327 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38000094a0 con 0x7f380c0b6580 2024-04-06T12:28:14.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.231+0000 7f37faffd700 1 -- 172.21.15.189:0/3342876327 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3800003a20 con 0x7f380c0b6580 2024-04-06T12:28:14.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.231+0000 7f3810c4c700 1 -- 172.21.15.189:0/3342876327 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f380c15e070 con 0x7f380c0b6580 2024-04-06T12:28:14.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.232+0000 7f3810c4c700 1 -- 172.21.15.189:0/3342876327 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f380c15e200 con 0x7f380c0b6580 2024-04-06T12:28:14.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.232+0000 7f37faffd700 1 -- 172.21.15.189:0/3342876327 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 42) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f3800003b80 con 0x7f380c0b6580 2024-04-06T12:28:14.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.233+0000 7f37faffd700 1 --2- 172.21.15.189:0/3342876327 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f37ec06ae00 0x7f37ec06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:14.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.233+0000 7f380a59c700 1 -- 172.21.15.189:0/3342876327 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f37ec06ae00 msgr2=0x7f37ec06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:14.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.233+0000 7f37faffd700 1 -- 172.21.15.189:0/3342876327 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3800021070 con 0x7f380c0b6580 2024-04-06T12:28:14.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.233+0000 7f380a59c700 1 --2- 172.21.15.189:0/3342876327 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f37ec06ae00 0x7f37ec06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:14.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.238+0000 7f37faffd700 1 -- 172.21.15.189:0/3342876327 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3800055d60 con 0x7f380c0b6580 2024-04-06T12:28:14.239 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:14.241+0000 7fe5e5b66200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:28:14.267 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:14.269+0000 7fd7c0918200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:28:14.307 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:14.308+0000 7fe5e5b66200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:28:14.339 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:14.340+0000 7fd7c0918200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:28:14.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.434+0000 7f380a59c700 1 -- 172.21.15.189:0/3342876327 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f37ec06ae00 msgr2=0x7f37ec06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:14.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.434+0000 7f380a59c700 1 --2- 172.21.15.189:0/3342876327 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f37ec06ae00 0x7f37ec06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:14.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.439+0000 7f3810c4c700 1 -- 172.21.15.189:0/3342876327 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f380c074990 con 0x7f380c0b6580 2024-04-06T12:28:14.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.442+0000 7f37faffd700 1 -- 172.21.15.189:0/3342876327 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7f380002b020 con 0x7f380c0b6580 2024-04-06T12:28:14.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.444+0000 7f3810c4c700 1 -- 172.21.15.189:0/3342876327 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f37ec06ae00 msgr2=0x7f37ec06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:14.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.444+0000 7f3810c4c700 1 --2- 172.21.15.189:0/3342876327 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f37ec06ae00 0x7f37ec06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.445+0000 7f3810c4c700 1 -- 172.21.15.189:0/3342876327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f380c0b6580 msgr2=0x7f380c0b71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:14.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.445+0000 7f3810c4c700 1 --2- 172.21.15.189:0/3342876327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f380c0b6580 0x7f380c0b71f0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f3800000c00 tx=0x7f3800004750 comp rx=0 tx=0).stop 2024-04-06T12:28:14.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.445+0000 7f3810c4c700 1 -- 172.21.15.189:0/3342876327 shutdown_connections 2024-04-06T12:28:14.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.445+0000 7f3810c4c700 1 --2- 172.21.15.189:0/3342876327 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f380c15d590 0x7f380c0b7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.445+0000 7f3810c4c700 1 --2- 172.21.15.189:0/3342876327 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f37ec06ae00 0x7f37ec06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.445+0000 7f3810c4c700 1 --2- 172.21.15.189:0/3342876327 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f380c0b4ed0 0x7f380c158610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.445+0000 7f3810c4c700 1 --2- 172.21.15.189:0/3342876327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f380c0b6580 0x7f380c0b71f0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.445+0000 7f3810c4c700 1 -- 172.21.15.189:0/3342876327 >> 172.21.15.189:0/3342876327 conn(0x7f380c06bbc0 msgr2=0x7f380c154940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:14.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.445+0000 7f3810c4c700 1 -- 172.21.15.189:0/3342876327 shutdown_connections 2024-04-06T12:28:14.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.445+0000 7f3810c4c700 1 -- 172.21.15.189:0/3342876327 wait complete. 2024-04-06T12:28:14.455 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-04-06T12:28:14.482 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:14.484+0000 7fe5e5b66200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:28:14.522 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:14.524+0000 7fd7c0918200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:28:14.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.629+0000 7f3ed3978700 1 Processor -- start 2024-04-06T12:28:14.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.629+0000 7f3ed3978700 1 -- start start 2024-04-06T12:28:14.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.630+0000 7f3ed3978700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc15d7b0 0x7f3ecc0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:14.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.630+0000 7f3ed3978700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3ecc06a730 0x7f3ecc06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:14.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.630+0000 7f3ed3978700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ecc0b3990 con 0x7f3ecc15ddf0 2024-04-06T12:28:14.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.630+0000 7f3ed3978700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ecc06b0f0 con 0x7f3ecc06a730 2024-04-06T12:28:14.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.630+0000 7f3ed3978700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc0b3060 con 0x7f3ecc15d7b0 2024-04-06T12:28:14.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.630+0000 7f3ed0f13700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3ecc06a730 0x7f3ecc06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:14.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.630+0000 7f3ed0f13700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3ecc06a730 0x7f3ecc06abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52402/0 (socket says 172.21.15.189:52402) 2024-04-06T12:28:14.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.630+0000 7f3ed0f13700 1 -- 172.21.15.189:0/3595792472 learned_addr learned my addr 172.21.15.189:0/3595792472 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:14.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.630+0000 7f3ed1714700 1 --2- 172.21.15.189:0/3595792472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc15d7b0 0x7f3ecc0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:14.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.630+0000 7f3ed0f13700 1 -- 172.21.15.189:0/3595792472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc15d7b0 msgr2=0x7f3ecc0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:14.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.630+0000 7f3ed0f13700 1 --2- 172.21.15.189:0/3595792472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc15d7b0 0x7f3ecc0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.630+0000 7f3ed0f13700 1 -- 172.21.15.189:0/3595792472 >> v1:172.21.15.189:6789/0 conn(0x7f3ecc15ddf0 legacy=0x7f3ecc15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:14.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.630+0000 7f3ed0f13700 1 -- 172.21.15.189:0/3595792472 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ecc0b3340 con 0x7f3ecc06a730 2024-04-06T12:28:14.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.630+0000 7f3ed0f13700 1 --2- 172.21.15.189:0/3595792472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ecc06a730 0x7f3ecc06abb0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc0098f0 tx=0x7f3ebc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30503b9f05b44de server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:14.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.631+0000 7f3ec3fff700 1 -- 172.21.15.189:0/3595792472 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc011830 con 0x7f3ecc06a730 2024-04-06T12:28:14.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.631+0000 7f3ec3fff700 1 -- 172.21.15.189:0/3595792472 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ebc011990 con 0x7f3ecc06a730 2024-04-06T12:28:14.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.631+0000 7f3ec3fff700 1 -- 172.21.15.189:0/3595792472 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc011d20 con 0x7f3ecc06a730 2024-04-06T12:28:14.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.631+0000 7f3ed3978700 1 -- 172.21.15.189:0/3595792472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ecc06a730 msgr2=0x7f3ecc06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:14.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.631+0000 7f3ed3978700 1 --2- 172.21.15.189:0/3595792472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ecc06a730 0x7f3ecc06abb0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc0098f0 tx=0x7f3ebc009c00 comp rx=0 tx=0).stop 2024-04-06T12:28:14.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.631+0000 7f3ed3978700 1 -- 172.21.15.189:0/3595792472 shutdown_connections 2024-04-06T12:28:14.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.631+0000 7f3ed3978700 1 --2- 172.21.15.189:0/3595792472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc15d7b0 0x7f3ecc0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.631+0000 7f3ed3978700 1 --2- 172.21.15.189:0/3595792472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ecc06a730 0x7f3ecc06abb0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.631+0000 7f3ed3978700 1 -- 172.21.15.189:0/3595792472 >> 172.21.15.189:0/3595792472 conn(0x7f3ecc0d00b0 msgr2=0x7f3ecc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:14.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.631+0000 7f3ed3978700 1 -- 172.21.15.189:0/3595792472 shutdown_connections 2024-04-06T12:28:14.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.631+0000 7f3ed3978700 1 -- 172.21.15.189:0/3595792472 wait complete. 2024-04-06T12:28:14.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.632+0000 7f3ed3978700 1 Processor -- start 2024-04-06T12:28:14.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.633+0000 7f3ed3978700 1 -- start start 2024-04-06T12:28:14.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.633+0000 7f3ed3978700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ecc06a730 0x7f3ecc0cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:14.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.633+0000 7f3ed3978700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc15d7b0 0x7f3ecc14f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:14.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.633+0000 7f3ed3978700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ecc15ddf0 0x7f3ecc0ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:14.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.633+0000 7f3ed3978700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc157ea0 con 0x7f3ecc06a730 2024-04-06T12:28:14.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.633+0000 7f3ed3978700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc150e20 con 0x7f3ecc15ddf0 2024-04-06T12:28:14.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.633+0000 7f3ed3978700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc0cee40 con 0x7f3ecc15d7b0 2024-04-06T12:28:14.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.633+0000 7f3ed1714700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ecc06a730 0x7f3ecc0cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:14.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.633+0000 7f3ed1714700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ecc06a730 0x7f3ecc0cc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52410/0 (socket says 172.21.15.189:52410) 2024-04-06T12:28:14.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.633+0000 7f3ed1714700 1 -- 172.21.15.189:0/2482863736 learned_addr learned my addr 172.21.15.189:0/2482863736 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:14.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.633+0000 7f3ed0f13700 1 --2- 172.21.15.189:0/2482863736 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc15d7b0 0x7f3ecc14f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:14.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.634+0000 7f3ed1f15700 1 --2- 172.21.15.189:0/2482863736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ecc15ddf0 0x7f3ecc0ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:14.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.634+0000 7f3ed1714700 1 -- 172.21.15.189:0/2482863736 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc15d7b0 msgr2=0x7f3ecc14f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:14.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.634+0000 7f3ed1714700 1 --2- 172.21.15.189:0/2482863736 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ecc15d7b0 0x7f3ecc14f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.634+0000 7f3ed1714700 1 -- 172.21.15.189:0/2482863736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ecc15ddf0 msgr2=0x7f3ecc0ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:14.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.634+0000 7f3ed1714700 1 --2- 172.21.15.189:0/2482863736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ecc15ddf0 0x7f3ecc0ccb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:14.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.634+0000 7f3ed1714700 1 -- 172.21.15.189:0/2482863736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ebc009550 con 0x7f3ecc06a730 2024-04-06T12:28:14.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.634+0000 7f3ed1f15700 1 --2- 172.21.15.189:0/2482863736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ecc15ddf0 0x7f3ecc0ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:14.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.634+0000 7f3ed1714700 1 --2- 172.21.15.189:0/2482863736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ecc06a730 0x7f3ecc0cc370 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f3ec800e9b0 tx=0x7f3ec800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:14.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.634+0000 7f3ec1ffb700 1 -- 172.21.15.189:0/2482863736 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8009710 con 0x7f3ecc06a730 2024-04-06T12:28:14.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.634+0000 7f3ec1ffb700 1 -- 172.21.15.189:0/2482863736 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec800c850 con 0x7f3ecc06a730 2024-04-06T12:28:14.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.634+0000 7f3ec1ffb700 1 -- 172.21.15.189:0/2482863736 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec800cbc0 con 0x7f3ecc06a730 2024-04-06T12:28:14.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.634+0000 7f3ed3978700 1 -- 172.21.15.189:0/2482863736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ecc0cf120 con 0x7f3ecc06a730 2024-04-06T12:28:14.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.634+0000 7f3ed3978700 1 -- 172.21.15.189:0/2482863736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ecc14fe70 con 0x7f3ecc06a730 2024-04-06T12:28:14.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.635+0000 7f3ed3978700 1 -- 172.21.15.189:0/2482863736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ecc157920 con 0x7f3ecc06a730 2024-04-06T12:28:14.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.636+0000 7f3ec1ffb700 1 -- 172.21.15.189:0/2482863736 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 42) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f3ec800cd20 con 0x7f3ecc06a730 2024-04-06T12:28:14.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.636+0000 7f3ec1ffb700 1 --2- 172.21.15.189:0/2482863736 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3eb006ae00 0x7f3eb006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:14.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.636+0000 7f3ed0f13700 1 -- 172.21.15.189:0/2482863736 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3eb006ae00 msgr2=0x7f3eb006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:14.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.636+0000 7f3ec1ffb700 1 -- 172.21.15.189:0/2482863736 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3ec80896d0 con 0x7f3ecc06a730 2024-04-06T12:28:14.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.636+0000 7f3ed0f13700 1 --2- 172.21.15.189:0/2482863736 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3eb006ae00 0x7f3eb006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:14.638 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:14.640+0000 7fe5e5b66200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:28:14.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.640+0000 7f3ec1ffb700 1 -- 172.21.15.189:0/2482863736 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3ecc157920 con 0x7f3ecc06a730 2024-04-06T12:28:14.688 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:14.690+0000 7fd7c0918200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:28:14.750 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:14.751+0000 7fe5e5b66200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:28:14.801 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:14.803+0000 7fd7c0918200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:28:14.830 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:14.832+0000 7fe5e5b66200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:28:14.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.837+0000 7f3ed0f13700 1 -- 172.21.15.189:0/2482863736 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3eb006ae00 msgr2=0x7f3eb006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:14.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.837+0000 7f3ed0f13700 1 --2- 172.21.15.189:0/2482863736 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3eb006ae00 0x7f3eb006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:14.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:14.872+0000 7f3ed3978700 1 -- 172.21.15.189:0/2482863736 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ecc15eb40 con 0x7f3eb006ae00 2024-04-06T12:28:14.885 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:14.887+0000 7fd7c0918200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:28:14.904 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:14.906+0000 7fe5e5b66200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:28:14.960 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:14.962+0000 7fd7c0918200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:28:15.106 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:15.108+0000 7fe5e5b66200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:28:15.162 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:15.164+0000 7fd7c0918200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:28:15.182 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:15.184+0000 7fe5e5b66200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:28:15.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:15.238+0000 7f3ed0f13700 1 -- 172.21.15.189:0/2482863736 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3eb006ae00 msgr2=0x7f3eb006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:15.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:15.238+0000 7f3ed0f13700 1 --2- 172.21.15.189:0/2482863736 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3eb006ae00 0x7f3eb006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-06T12:28:15.237 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:15.239+0000 7fd7c0918200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:28:15.499 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:15.501+0000 7fe5e5b66200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:28:15.558 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:15.559+0000 7fd7c0918200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:28:15.818 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:15.819+0000 7fe5e5b66200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:28:15.877 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:15.879+0000 7fd7c0918200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:28:15.976 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:15.978+0000 7fe5e5b66200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:28:16.036 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:16.038+0000 7fd7c0918200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:28:16.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:16.039+0000 7f3ed0f13700 1 -- 172.21.15.189:0/2482863736 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3eb006ae00 msgr2=0x7f3eb006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:16.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:16.039+0000 7f3ed0f13700 1 --2- 172.21.15.189:0/2482863736 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3eb006ae00 0x7f3eb006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-06T12:28:16.049 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:16.050+0000 7fe5e5b66200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:28:16.110 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:16.112+0000 7fd7c0918200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:28:16.662 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:16.664+0000 7fe5e5b66200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:28:16.727 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:16.729+0000 7fd7c0918200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:28:16.741 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:16.743+0000 7fe5e5b66200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:28:16.806 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:16.808+0000 7fd7c0918200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:28:16.866 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:16.868+0000 7fe5e5b66200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:28:16.931 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:16.933+0000 7fd7c0918200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:28:17.374 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:17.376+0000 7fe5e5b66200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:28:17.441 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:17.442+0000 7fd7c0918200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:28:17.526 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:17.528+0000 7fe5e5b66200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:28:17.594 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:17.596+0000 7fd7c0918200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:28:17.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:17.640+0000 7f3ed0f13700 1 -- 172.21.15.189:0/2482863736 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3eb006ae00 msgr2=0x7f3eb006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:17.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:17.640+0000 7f3ed0f13700 1 --2- 172.21.15.189:0/2482863736 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3eb006ae00 0x7f3eb006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-06T12:28:17.679 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:17.681+0000 7fe5e5b66200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:28:17.750 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:17.752+0000 7fd7c0918200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:28:18.400 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:18.402+0000 7fe5e5b66200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:28:18.469 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:18.471+0000 7fd7c0918200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:28:18.637 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:18.639+0000 7fe5e5b66200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:28:18.710 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:18.712+0000 7fd7c0918200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:28:18.711 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:18.713+0000 7fe5e5b66200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:28:18.787 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:18.789+0000 7fd7c0918200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:28:19.083 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:19.085+0000 7fe5e5b66200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:28:19.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:19.156+0000 7f3ec1ffb700 1 -- 172.21.15.189:0/2482863736 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 43) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f3ec8087ab0 con 0x7f3ecc06a730 2024-04-06T12:28:19.162 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:19.164+0000 7fd7c0918200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:28:19.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:19.185+0000 7f3ec1ffb700 1 -- 172.21.15.189:0/2482863736 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mgrmap(e 44) v1 ==== 88873+0+0 (secure 0 0 0) 0x7f3ec806ff30 con 0x7f3ecc06a730 2024-04-06T12:28:19.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:19.186+0000 7f3ec1ffb700 1 -- 172.21.15.189:0/2482863736 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3eb006ae00 msgr2=0x7f3eb006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:19.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:19.186+0000 7f3ec1ffb700 1 --2- 172.21.15.189:0/2482863736 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3eb006ae00 0x7f3eb006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:19.341 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:19.343+0000 7fd733826700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:19.342 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:19.343+0000 7fd73682c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:19.342 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:19.343+0000 7fd733826700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:19.342 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:19.343+0000 7fd73682c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:19.342 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:19.343+0000 7fd733826700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:19.342 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:19.343+0000 7fd73682c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:19.342 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:19.343+0000 7fd73682c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:19.342 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:19.343+0000 7fd733826700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:19.342 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:19.343+0000 7fd73682c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:19.342 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:19.343+0000 7fd733826700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.191+0000 7f3ec1ffb700 1 -- 172.21.15.189:0/2482863736 <== mon.0 v2:172.21.15.189:3300/0 9 ==== mgrmap(e 45) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f3ec803b830 con 0x7f3ecc06a730 2024-04-06T12:28:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.191+0000 7f3ec1ffb700 1 --2- 172.21.15.189:0/2482863736 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3eb006ae00 0x7f3eb006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.191+0000 7f3ec1ffb700 1 -- 172.21.15.189:0/2482863736 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3eb001cd40 con 0x7f3eb006ae00 2024-04-06T12:28:20.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.192+0000 7f3ed0f13700 1 --2- 172.21.15.189:0/2482863736 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3eb006ae00 0x7f3eb006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:20.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.194+0000 7f3ed0f13700 1 --2- 172.21.15.189:0/2482863736 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3eb006ae00 0x7f3eb006d2c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009c40 tx=0x7f3ebc008030 comp rx=0 tx=0).ready entity=mgr.6799 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:20.200 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:20.202+0000 7fd73d83a700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-04-06T12:28:20.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.202+0000 7f3ec1ffb700 1 -- 172.21.15.189:0/2482863736 <== mgr.6799 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f3eb001cd40 con 0x7f3eb006ae00 2024-04-06T12:28:20.202 INFO:teuthology.orchestra.run.smithi189.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-04-06T12:28:20.206 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-06T12:28:20.207 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-06T12:28:20.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.380+0000 7f97d6a05700 1 Processor -- start 2024-04-06T12:28:20.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.380+0000 7f97d6a05700 1 -- start start 2024-04-06T12:28:20.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.381+0000 7f97d6a05700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f97d0158fe0 0x7f97d006a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:20.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.381+0000 7f97d6a05700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d015f9e0 con 0x7f97d015c2d0 2024-04-06T12:28:20.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.381+0000 7f97d6a05700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d015f0d0 con 0x7f97d006ac70 2024-04-06T12:28:20.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.381+0000 7f97d6a05700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d006b0f0 con 0x7f97d0158fe0 2024-04-06T12:28:20.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.381+0000 7f97cffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f97d0158fe0 0x7f97d006a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:20.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.381+0000 7f97cffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f97d0158fe0 0x7f97d006a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58968/0 (socket says 172.21.15.189:58968) 2024-04-06T12:28:20.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.381+0000 7f97cffff700 1 -- 172.21.15.189:0/4083030776 learned_addr learned my addr 172.21.15.189:0/4083030776 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:20.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.381+0000 7f97cffff700 1 -- 172.21.15.189:0/4083030776 >> v1:172.21.15.192:6789/0 conn(0x7f97d006ac70 legacy=0x7f97d015bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:20.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.381+0000 7f97cffff700 1 -- 172.21.15.189:0/4083030776 >> v1:172.21.15.189:6789/0 conn(0x7f97d015c2d0 legacy=0x7f97d00ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:20.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.381+0000 7f97cffff700 1 -- 172.21.15.189:0/4083030776 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d015c8f0 con 0x7f97d0158fe0 2024-04-06T12:28:20.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.382+0000 7f97cffff700 1 --2- 172.21.15.189:0/4083030776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d0158fe0 0x7f97d006a730 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f97c40098f0 tx=0x7f97c4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155a8da582a8f9e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:20.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.382+0000 7f97ceffd700 1 -- 172.21.15.189:0/4083030776 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4004230 con 0x7f97d0158fe0 2024-04-06T12:28:20.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.382+0000 7f97ceffd700 1 -- 172.21.15.189:0/4083030776 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c4004390 con 0x7f97d0158fe0 2024-04-06T12:28:20.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.382+0000 7f97ceffd700 1 -- 172.21.15.189:0/4083030776 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c40036a0 con 0x7f97d0158fe0 2024-04-06T12:28:20.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.382+0000 7f97d6a05700 1 -- 172.21.15.189:0/4083030776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d0158fe0 msgr2=0x7f97d006a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:20.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.382+0000 7f97d6a05700 1 --2- 172.21.15.189:0/4083030776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d0158fe0 0x7f97d006a730 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f97c40098f0 tx=0x7f97c4009c00 comp rx=0 tx=0).stop 2024-04-06T12:28:20.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.382+0000 7f97d6a05700 1 -- 172.21.15.189:0/4083030776 shutdown_connections 2024-04-06T12:28:20.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.382+0000 7f97d6a05700 1 --2- 172.21.15.189:0/4083030776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d0158fe0 0x7f97d006a730 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:20.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.382+0000 7f97d6a05700 1 -- 172.21.15.189:0/4083030776 >> 172.21.15.189:0/4083030776 conn(0x7f97d00d00b0 msgr2=0x7f97d00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:20.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.382+0000 7f97d6a05700 1 -- 172.21.15.189:0/4083030776 shutdown_connections 2024-04-06T12:28:20.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.382+0000 7f97d6a05700 1 -- 172.21.15.189:0/4083030776 wait complete. 2024-04-06T12:28:20.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.383+0000 7f97d6a05700 1 Processor -- start 2024-04-06T12:28:20.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.384+0000 7f97d6a05700 1 -- start start 2024-04-06T12:28:20.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.384+0000 7f97d6a05700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97d006ac70 0x7f97d01f0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:20.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.384+0000 7f97d6a05700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d0158fe0 0x7f97d013a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:20.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.384+0000 7f97d6a05700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97d015c2d0 0x7f97d0157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:20.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.384+0000 7f97d6a05700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d01f1eb0 con 0x7f97d0158fe0 2024-04-06T12:28:20.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.384+0000 7f97d6a05700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d013aaa0 con 0x7f97d015c2d0 2024-04-06T12:28:20.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.384+0000 7f97d6a05700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d01577a0 con 0x7f97d006ac70 2024-04-06T12:28:20.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.385+0000 7f97cf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d0158fe0 0x7f97d013a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:20.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.385+0000 7f97cf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d0158fe0 0x7f97d013a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58970/0 (socket says 172.21.15.189:58970) 2024-04-06T12:28:20.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.385+0000 7f97cf7fe700 1 -- 172.21.15.189:0/1069628552 learned_addr learned my addr 172.21.15.189:0/1069628552 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:20.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.385+0000 7f97d4fa2700 1 --2- 172.21.15.189:0/1069628552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97d015c2d0 0x7f97d0157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:20.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.385+0000 7f97cffff700 1 --2- 172.21.15.189:0/1069628552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97d006ac70 0x7f97d01f0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:20.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.385+0000 7f97cf7fe700 1 -- 172.21.15.189:0/1069628552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97d006ac70 msgr2=0x7f97d01f0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:20.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.385+0000 7f97cf7fe700 1 --2- 172.21.15.189:0/1069628552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97d006ac70 0x7f97d01f0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:20.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.385+0000 7f97cf7fe700 1 -- 172.21.15.189:0/1069628552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97d015c2d0 msgr2=0x7f97d0157260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:20.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.385+0000 7f97cf7fe700 1 --2- 172.21.15.189:0/1069628552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97d015c2d0 0x7f97d0157260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:20.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.385+0000 7f97cf7fe700 1 -- 172.21.15.189:0/1069628552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c4009550 con 0x7f97d0158fe0 2024-04-06T12:28:20.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.385+0000 7f97d4fa2700 1 --2- 172.21.15.189:0/1069628552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97d015c2d0 0x7f97d0157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:20.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.385+0000 7f97cf7fe700 1 --2- 172.21.15.189:0/1069628552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d0158fe0 0x7f97d013a560 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f97c000c4a0 tx=0x7f97c000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:20.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.385+0000 7f97ccff9700 1 -- 172.21.15.189:0/1069628552 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c00054e0 con 0x7f97d0158fe0 2024-04-06T12:28:20.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.385+0000 7f97ccff9700 1 -- 172.21.15.189:0/1069628552 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c0003680 con 0x7f97d0158fe0 2024-04-06T12:28:20.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.385+0000 7f97ccff9700 1 -- 172.21.15.189:0/1069628552 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c0003a30 con 0x7f97d0158fe0 2024-04-06T12:28:20.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.385+0000 7f97d6a05700 1 -- 172.21.15.189:0/1069628552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d01f1610 con 0x7f97d0158fe0 2024-04-06T12:28:20.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.385+0000 7f97d6a05700 1 -- 172.21.15.189:0/1069628552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d01f1bd0 con 0x7f97d0158fe0 2024-04-06T12:28:20.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.386+0000 7f97d6a05700 1 -- 172.21.15.189:0/1069628552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d015fe50 con 0x7f97d0158fe0 2024-04-06T12:28:20.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.387+0000 7f97ccff9700 1 -- 172.21.15.189:0/1069628552 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 45) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f97c0003b90 con 0x7f97d0158fe0 2024-04-06T12:28:20.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.387+0000 7f97ccff9700 1 --2- 172.21.15.189:0/1069628552 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f97bc06ab30 0x7f97bc06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:20.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.387+0000 7f97ccff9700 1 -- 172.21.15.189:0/1069628552 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f97c0088320 con 0x7f97d0158fe0 2024-04-06T12:28:20.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.387+0000 7f97cffff700 1 --2- 172.21.15.189:0/1069628552 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f97bc06ab30 0x7f97bc06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:20.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.387+0000 7f97cffff700 1 --2- 172.21.15.189:0/1069628552 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f97bc06ab30 0x7f97bc06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f97c4009130 tx=0x7f97c4008030 comp rx=0 tx=0).ready entity=mgr.6799 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:20.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.391+0000 7f97ccff9700 1 -- 172.21.15.189:0/1069628552 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f97d015fe50 con 0x7f97d0158fe0 2024-04-06T12:28:20.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:20.640+0000 7f97d6a05700 1 -- 172.21.15.189:0/1069628552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f97d015fe50 con 0x7f97d0158fe0 2024-04-06T12:28:21.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.199+0000 7f97ccff9700 1 -- 172.21.15.189:0/1069628552 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 46) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f97c0053970 con 0x7f97d0158fe0 2024-04-06T12:28:21.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.199+0000 7f97ccff9700 1 -- 172.21.15.189:0/1069628552 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v46) v1 ==== 93+0+0 (secure 0 0 0) 0x7f97c0074120 con 0x7f97d0158fe0 2024-04-06T12:28:21.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.202+0000 7f97d6a05700 1 -- 172.21.15.189:0/1069628552 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f97bc06ab30 msgr2=0x7f97bc06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:21.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.202+0000 7f97d6a05700 1 --2- 172.21.15.189:0/1069628552 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f97bc06ab30 0x7f97bc06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f97c4009130 tx=0x7f97c4008030 comp rx=0 tx=0).stop 2024-04-06T12:28:21.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.202+0000 7f97d6a05700 1 -- 172.21.15.189:0/1069628552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d0158fe0 msgr2=0x7f97d013a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:21.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.202+0000 7f97d6a05700 1 --2- 172.21.15.189:0/1069628552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d0158fe0 0x7f97d013a560 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f97c000c4a0 tx=0x7f97c000c860 comp rx=0 tx=0).stop 2024-04-06T12:28:21.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.202+0000 7f97d6a05700 1 -- 172.21.15.189:0/1069628552 shutdown_connections 2024-04-06T12:28:21.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.202+0000 7f97d6a05700 1 --2- 172.21.15.189:0/1069628552 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f97bc06ab30 0x7f97bc06cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:21.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.202+0000 7f97d6a05700 1 --2- 172.21.15.189:0/1069628552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97d015c2d0 0x7f97d0157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:21.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.202+0000 7f97d6a05700 1 --2- 172.21.15.189:0/1069628552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97d006ac70 0x7f97d01f0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:21.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.202+0000 7f97d6a05700 1 --2- 172.21.15.189:0/1069628552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d0158fe0 0x7f97d013a560 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:21.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.202+0000 7f97d6a05700 1 -- 172.21.15.189:0/1069628552 >> 172.21.15.189:0/1069628552 conn(0x7f97d00d00b0 msgr2=0x7f97d01583e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:21.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.203+0000 7f97d6a05700 1 -- 172.21.15.189:0/1069628552 shutdown_connections 2024-04-06T12:28:21.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.203+0000 7f97d6a05700 1 -- 172.21.15.189:0/1069628552 wait complete. 2024-04-06T12:28:21.216 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T12:28:21.217 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:28:21.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:28:21.307 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T12:28:21.307 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:28:21.307 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:28:21.325 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:28:21.325 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:28:21.384 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:28:21.558 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.559+0000 7f430eb7c700 1 Processor -- start 2024-04-06T12:28:21.558 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.560+0000 7f430eb7c700 1 -- start start 2024-04-06T12:28:21.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.560+0000 7f430eb7c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f430806ae40 0x7f43080b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:21.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.560+0000 7f430eb7c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43080b3990 con 0x7f430815d7b0 2024-04-06T12:28:21.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.560+0000 7f430eb7c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43080b3060 con 0x7f430815ddf0 2024-04-06T12:28:21.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.560+0000 7f430eb7c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43080b3340 con 0x7f430806ae40 2024-04-06T12:28:21.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.560+0000 7f430c918700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f430815d7b0 0x7f430806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33612/0 (socket says 172.21.15.189:33612) 2024-04-06T12:28:21.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.560+0000 7f430c918700 1 -- 172.21.15.189:0/816988977 learned_addr learned my addr 172.21.15.189:0/816988977 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:21.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f4307fff700 1 --2- 172.21.15.189:0/816988977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f430806ae40 0x7f43080b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:21.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f43077fe700 1 -- 172.21.15.189:0/816988977 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249213253 0 0) 0x7f43080b3990 con 0x7f430815d7b0 2024-04-06T12:28:21.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f43077fe700 1 -- 172.21.15.189:0/816988977 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42f80035b0 con 0x7f430815d7b0 2024-04-06T12:28:21.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f43077fe700 1 -- 172.21.15.189:0/816988977 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516664829 0 0) 0x7f43080b3060 con 0x7f430815ddf0 2024-04-06T12:28:21.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f43077fe700 1 -- 172.21.15.189:0/816988977 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43080b3990 con 0x7f430815ddf0 2024-04-06T12:28:21.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f43077fe700 1 -- 172.21.15.189:0/816988977 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2207648538 0 0) 0x7f42f80035b0 con 0x7f430815d7b0 2024-04-06T12:28:21.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f43077fe700 1 -- 172.21.15.189:0/816988977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f430806ae40 msgr2=0x7f43080b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:21.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f43077fe700 1 --2- 172.21.15.189:0/816988977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f430806ae40 0x7f43080b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:21.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f43077fe700 1 -- 172.21.15.189:0/816988977 >> v1:172.21.15.192:6789/0 conn(0x7f430815ddf0 legacy=0x7f430815e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:21.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f43077fe700 1 -- 172.21.15.189:0/816988977 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43080b3620 con 0x7f430815d7b0 2024-04-06T12:28:21.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f4307fff700 1 --2- 172.21.15.189:0/816988977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f430806ae40 0x7f43080b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:28:21.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f43077fe700 1 -- 172.21.15.189:0/816988977 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f42fc001670 con 0x7f430815d7b0 2024-04-06T12:28:21.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f43077fe700 1 -- 172.21.15.189:0/816988977 >> v1:172.21.15.189:6789/0 conn(0x7f430815d7b0 legacy=0x7f430806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:21.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f43077fe700 1 --2- 172.21.15.189:0/816988977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42f8003bc0 0x7f42f8003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:21.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f43077fe700 1 --2- 172.21.15.189:0/816988977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42f8004850 0x7f42f8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:21.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f43077fe700 1 --2- 172.21.15.189:0/816988977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f430806ae40 0x7f42f8005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:21.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f43077fe700 1 -- 172.21.15.189:0/816988977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43080b3340 con 0x7f42f8003bc0 2024-04-06T12:28:21.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f43077fe700 1 -- 172.21.15.189:0/816988977 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f800bb20 con 0x7f42f8004850 2024-04-06T12:28:21.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f43077fe700 1 -- 172.21.15.189:0/816988977 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f800bce0 con 0x7f430806ae40 2024-04-06T12:28:21.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f430c918700 1 --2- 172.21.15.189:0/816988977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42f8003bc0 0x7f42f8003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:21.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f430d119700 1 --2- 172.21.15.189:0/816988977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f430806ae40 0x7f42f8005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:21.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f4307fff700 1 --2- 172.21.15.189:0/816988977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42f8004850 0x7f42f8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:21.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f430c918700 1 -- 172.21.15.189:0/816988977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f430806ae40 msgr2=0x7f42f8005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:21.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f430c918700 1 --2- 172.21.15.189:0/816988977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f430806ae40 0x7f42f8005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:21.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f430c918700 1 -- 172.21.15.189:0/816988977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42f8004850 msgr2=0x7f42f8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:21.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f430c918700 1 --2- 172.21.15.189:0/816988977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42f8004850 0x7f42f8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:21.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f430c918700 1 -- 172.21.15.189:0/816988977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f40094a0 con 0x7f42f8003bc0 2024-04-06T12:28:21.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.561+0000 7f4307fff700 1 --2- 172.21.15.189:0/816988977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42f8004850 0x7f42f8004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:21.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.562+0000 7f430c918700 1 --2- 172.21.15.189:0/816988977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42f8003bc0 0x7f42f8003fc0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f42fc00c160 tx=0x7f42f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906e292d9eec8f58 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:21.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.562+0000 7f43077fe700 1 -- 172.21.15.189:0/816988977 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42fc00cca0 con 0x7f42f8003bc0 2024-04-06T12:28:21.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.562+0000 7f43077fe700 1 -- 172.21.15.189:0/816988977 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42fc00ce20 con 0x7f42f8003bc0 2024-04-06T12:28:21.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.562+0000 7f43077fe700 1 -- 172.21.15.189:0/816988977 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42fc005ba0 con 0x7f42f8003bc0 2024-04-06T12:28:21.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.562+0000 7f430eb7c700 1 -- 172.21.15.189:0/816988977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42f8003bc0 msgr2=0x7f42f8003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:21.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.562+0000 7f430eb7c700 1 --2- 172.21.15.189:0/816988977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42f8003bc0 0x7f42f8003fc0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f42fc00c160 tx=0x7f42f8005880 comp rx=0 tx=0).stop 2024-04-06T12:28:21.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.562+0000 7f430d119700 1 -- 172.21.15.189:0/816988977 reap_dead start 2024-04-06T12:28:21.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.562+0000 7f430eb7c700 1 -- 172.21.15.189:0/816988977 shutdown_connections 2024-04-06T12:28:21.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.562+0000 7f430eb7c700 1 -- 172.21.15.189:0/816988977 >> 172.21.15.189:0/816988977 conn(0x7f43080d00b0 msgr2=0x7f43080d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:21.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.562+0000 7f430eb7c700 1 -- 172.21.15.189:0/816988977 shutdown_connections 2024-04-06T12:28:21.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.562+0000 7f430eb7c700 1 -- 172.21.15.189:0/816988977 wait complete. 2024-04-06T12:28:21.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.564+0000 7f430eb7c700 1 Processor -- start 2024-04-06T12:28:21.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.564+0000 7f430eb7c700 1 -- start start 2024-04-06T12:28:21.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f430eb7c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42f8003bc0 0x7f43081420b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:21.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f430eb7c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43081f6fb0 0x7f4308155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:21.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f430eb7c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43081561a0 0x7f43081f6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:21.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f430eb7c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43081566d0 con 0x7f43081561a0 2024-04-06T12:28:21.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f430eb7c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43081569b0 con 0x7f43081f6fb0 2024-04-06T12:28:21.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f430eb7c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4308156c90 con 0x7f42f8003bc0 2024-04-06T12:28:21.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f430d119700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43081561a0 0x7f43081f6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:21.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f430d119700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43081561a0 0x7f43081f6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58982/0 (socket says 172.21.15.189:58982) 2024-04-06T12:28:21.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f430d119700 1 -- 172.21.15.189:0/1470306786 learned_addr learned my addr 172.21.15.189:0/1470306786 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:21.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f4307fff700 1 --2- 172.21.15.189:0/1470306786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43081f6fb0 0x7f4308155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:21.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f430c918700 1 --2- 172.21.15.189:0/1470306786 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42f8003bc0 0x7f43081420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:21.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f430d119700 1 -- 172.21.15.189:0/1470306786 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42f8003bc0 msgr2=0x7f43081420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:21.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f430d119700 1 --2- 172.21.15.189:0/1470306786 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42f8003bc0 0x7f43081420b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:21.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f430d119700 1 -- 172.21.15.189:0/1470306786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43081f6fb0 msgr2=0x7f4308155c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:21.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f430d119700 1 --2- 172.21.15.189:0/1470306786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43081f6fb0 0x7f4308155c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:21.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f430d119700 1 -- 172.21.15.189:0/1470306786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4308156f70 con 0x7f43081561a0 2024-04-06T12:28:21.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f430d119700 1 --2- 172.21.15.189:0/1470306786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43081561a0 0x7f43081f6360 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f430806d2c0 tx=0x7f430000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:21.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f43057fa700 1 -- 172.21.15.189:0/1470306786 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4300003c10 con 0x7f43081561a0 2024-04-06T12:28:21.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f43057fa700 1 -- 172.21.15.189:0/1470306786 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4300003d70 con 0x7f43081561a0 2024-04-06T12:28:21.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f430eb7c700 1 -- 172.21.15.189:0/1470306786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43081f68a0 con 0x7f43081561a0 2024-04-06T12:28:21.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f43057fa700 1 -- 172.21.15.189:0/1470306786 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430000db20 con 0x7f43081561a0 2024-04-06T12:28:21.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.565+0000 7f430eb7c700 1 -- 172.21.15.189:0/1470306786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4308144810 con 0x7f43081561a0 2024-04-06T12:28:21.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.566+0000 7f430eb7c700 1 -- 172.21.15.189:0/1470306786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4308157490 con 0x7f43081561a0 2024-04-06T12:28:21.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.567+0000 7f43057fa700 1 -- 172.21.15.189:0/1470306786 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 46) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f430000dc80 con 0x7f43081561a0 2024-04-06T12:28:21.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.567+0000 7f43057fa700 1 --2- 172.21.15.189:0/1470306786 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42ec06aae0 0x7f42ec06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:21.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.567+0000 7f430c918700 1 -- 172.21.15.189:0/1470306786 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42ec06aae0 msgr2=0x7f42ec06cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:21.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.567+0000 7f43057fa700 1 -- 172.21.15.189:0/1470306786 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f4300089cd0 con 0x7f43081561a0 2024-04-06T12:28:21.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.567+0000 7f430c918700 1 --2- 172.21.15.189:0/1470306786 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42ec06aae0 0x7f42ec06cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:21.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.571+0000 7f43057fa700 1 -- 172.21.15.189:0/1470306786 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4300055010 con 0x7f43081561a0 2024-04-06T12:28:21.584 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:21.586+0000 7ffa50309200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:28:21.607 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:21.609+0000 7ffb7e96d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:28:21.663 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:21.665+0000 7ffa50309200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:28:21.689 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:21.691+0000 7ffb7e96d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:28:21.738 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:21.740+0000 7ffa50309200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:28:21.766 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:21.768+0000 7ffb7e96d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:28:21.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.768+0000 7f430c918700 1 -- 172.21.15.189:0/1470306786 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42ec06aae0 msgr2=0x7f42ec06cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:21.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.768+0000 7f430c918700 1 --2- 172.21.15.189:0/1470306786 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42ec06aae0 0x7f42ec06cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:21.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.781+0000 7f430eb7c700 1 -- 172.21.15.189:0/1470306786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4308157490 con 0x7f43081561a0 2024-04-06T12:28:21.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.782+0000 7f43057fa700 1 -- 172.21.15.189:0/1470306786 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16855 (secure 0 0 0) 0x7f4308157490 con 0x7f43081561a0 2024-04-06T12:28:21.781 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:21.781 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":99,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:28:19.179240+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:28:04.080500+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-06T12:28:04.619802+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/530172869":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/4174795471":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/2207020726":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/2070668132":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/1706294115":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/4285218889":"2024-04-07T12:28:19.179156+0000","172.21.15.189:6833/47128":"2024-04-07T12:28:19.179156+0000","172.21.15.189:6832/47128":"2024-04-07T12:28:19.179156+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:28:21.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.785+0000 7f430eb7c700 1 -- 172.21.15.189:0/1470306786 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42ec06aae0 msgr2=0x7f42ec06cfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:21.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.785+0000 7f430eb7c700 1 --2- 172.21.15.189:0/1470306786 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42ec06aae0 0x7f42ec06cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:21.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.785+0000 7f430eb7c700 1 -- 172.21.15.189:0/1470306786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43081561a0 msgr2=0x7f43081f6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:21.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.785+0000 7f430eb7c700 1 --2- 172.21.15.189:0/1470306786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43081561a0 0x7f43081f6360 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f430806d2c0 tx=0x7f430000d430 comp rx=0 tx=0).stop 2024-04-06T12:28:21.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.785+0000 7f430eb7c700 1 -- 172.21.15.189:0/1470306786 shutdown_connections 2024-04-06T12:28:21.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.785+0000 7f430eb7c700 1 --2- 172.21.15.189:0/1470306786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43081561a0 0x7f43081f6360 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:21.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.785+0000 7f430eb7c700 1 --2- 172.21.15.189:0/1470306786 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42f8003bc0 0x7f43081420b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:21.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.785+0000 7f430eb7c700 1 --2- 172.21.15.189:0/1470306786 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42ec06aae0 0x7f42ec06cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:21.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.785+0000 7f430eb7c700 1 --2- 172.21.15.189:0/1470306786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43081f6fb0 0x7f4308155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:21.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.785+0000 7f430eb7c700 1 -- 172.21.15.189:0/1470306786 >> 172.21.15.189:0/1470306786 conn(0x7f43080d00b0 msgr2=0x7f4308072040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:21.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.785+0000 7f430eb7c700 1 -- 172.21.15.189:0/1470306786 shutdown_connections 2024-04-06T12:28:21.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.785+0000 7f430eb7c700 1 -- 172.21.15.189:0/1470306786 wait complete. 2024-04-06T12:28:21.796 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:28:04.080500+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:28:04.619802+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:28:21.797 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:28:21.805 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:21.807+0000 7ffa50309200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:28:21.839 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:21.841+0000 7ffb7e96d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:28:21.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.929+0000 7fe2e1523700 1 Processor -- start 2024-04-06T12:28:21.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.929+0000 7fe2e1523700 1 -- start start 2024-04-06T12:28:21.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.929+0000 7fe2e1523700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe2dc0b5380 0x7fe2dc0d2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:21.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.929+0000 7fe2e1523700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2dc0b5cc0 con 0x7fe2dc0b4ed0 2024-04-06T12:28:21.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.930+0000 7fe2e1523700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2dc0b57d0 con 0x7fe2dc15d590 2024-04-06T12:28:21.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.930+0000 7fe2e1523700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2dc0d1500 con 0x7fe2dc0b5380 2024-04-06T12:28:21.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.930+0000 7fe2db7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe2dc0b5380 0x7fe2dc0d2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:21.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.930+0000 7fe2da7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe2dc0b4ed0 0x7fe2dc0d2880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33624/0 (socket says 172.21.15.189:33624) 2024-04-06T12:28:21.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.930+0000 7fe2da7fc700 1 -- 172.21.15.189:0/2675443984 learned_addr learned my addr 172.21.15.189:0/2675443984 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:21.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.930+0000 7fe2db7fe700 1 -- 172.21.15.189:0/2675443984 >> v1:172.21.15.192:6789/0 conn(0x7fe2dc15d590 legacy=0x7fe2dc0b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:21.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.930+0000 7fe2db7fe700 1 -- 172.21.15.189:0/2675443984 >> v1:172.21.15.189:6789/0 conn(0x7fe2dc0b4ed0 legacy=0x7fe2dc0d2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:21.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.930+0000 7fe2db7fe700 1 -- 172.21.15.189:0/2675443984 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc0d17e0 con 0x7fe2dc0b5380 2024-04-06T12:28:21.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.930+0000 7fe2d9ffb700 1 -- 172.21.15.189:0/2675443984 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 736554704 0 0) 0x7fe2dc0b5cc0 con 0x7fe2dc0b4ed0 2024-04-06T12:28:21.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.930+0000 7fe2db7fe700 1 --2- 172.21.15.189:0/2675443984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc0b5380 0x7fe2dc0d2f90 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4009d90 tx=0x7fe2d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82037e4fdea78262 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:21.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.930+0000 7fe2d9ffb700 1 -- 172.21.15.189:0/2675443984 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d400e070 con 0x7fe2dc0b5380 2024-04-06T12:28:21.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.930+0000 7fe2d9ffb700 1 -- 172.21.15.189:0/2675443984 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d400bbd0 con 0x7fe2dc0b5380 2024-04-06T12:28:21.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.930+0000 7fe2d9ffb700 1 -- 172.21.15.189:0/2675443984 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4012430 con 0x7fe2dc0b5380 2024-04-06T12:28:21.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.931+0000 7fe2e1523700 1 -- 172.21.15.189:0/2675443984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc0b5380 msgr2=0x7fe2dc0d2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:21.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.931+0000 7fe2e1523700 1 --2- 172.21.15.189:0/2675443984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc0b5380 0x7fe2dc0d2f90 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4009d90 tx=0x7fe2d400b330 comp rx=0 tx=0).stop 2024-04-06T12:28:21.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.931+0000 7fe2e1523700 1 -- 172.21.15.189:0/2675443984 shutdown_connections 2024-04-06T12:28:21.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.931+0000 7fe2e1523700 1 --2- 172.21.15.189:0/2675443984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc0b5380 0x7fe2dc0d2f90 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:21.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.931+0000 7fe2e1523700 1 -- 172.21.15.189:0/2675443984 >> 172.21.15.189:0/2675443984 conn(0x7fe2dc06bbc0 msgr2=0x7fe2dc06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:21.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.931+0000 7fe2e1523700 1 -- 172.21.15.189:0/2675443984 shutdown_connections 2024-04-06T12:28:21.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.931+0000 7fe2e1523700 1 -- 172.21.15.189:0/2675443984 wait complete. 2024-04-06T12:28:21.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.932+0000 7fe2e1523700 1 Processor -- start 2024-04-06T12:28:21.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.932+0000 7fe2e1523700 1 -- start start 2024-04-06T12:28:21.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.933+0000 7fe2e1523700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2dc0b4ed0 0x7fe2dc15e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:21.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.933+0000 7fe2e1523700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2dc0b5380 0x7fe2dc15e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:21.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.933+0000 7fe2e1523700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc15d590 0x7fe2dc159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:21.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.933+0000 7fe2e1523700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc1549d0 con 0x7fe2dc15d590 2024-04-06T12:28:21.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.933+0000 7fe2e1523700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc156cd0 con 0x7fe2dc0b4ed0 2024-04-06T12:28:21.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.933+0000 7fe2e1523700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc0cc5b0 con 0x7fe2dc0b5380 2024-04-06T12:28:21.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.933+0000 7fe2db7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc15d590 0x7fe2dc159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:21.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.933+0000 7fe2daffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2dc0b4ed0 0x7fe2dc15e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:21.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.933+0000 7fe2db7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc15d590 0x7fe2dc159660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59010/0 (socket says 172.21.15.189:59010) 2024-04-06T12:28:21.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.933+0000 7fe2da7fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2dc0b5380 0x7fe2dc15e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:21.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.933+0000 7fe2daffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2dc0b4ed0 0x7fe2dc15e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:54508/0 (socket says 172.21.15.189:54508) 2024-04-06T12:28:21.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.933+0000 7fe2db7fe700 1 -- 172.21.15.189:0/977960120 learned_addr learned my addr 172.21.15.189:0/977960120 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:21.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.934+0000 7fe2db7fe700 1 -- 172.21.15.189:0/977960120 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2dc0b5380 msgr2=0x7fe2dc15e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:21.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.934+0000 7fe2db7fe700 1 --2- 172.21.15.189:0/977960120 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2dc0b5380 0x7fe2dc15e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:21.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.934+0000 7fe2db7fe700 1 -- 172.21.15.189:0/977960120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2dc0b4ed0 msgr2=0x7fe2dc15e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:21.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.934+0000 7fe2db7fe700 1 --2- 172.21.15.189:0/977960120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2dc0b4ed0 0x7fe2dc15e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:21.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.934+0000 7fe2db7fe700 1 -- 172.21.15.189:0/977960120 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d4009a70 con 0x7fe2dc15d590 2024-04-06T12:28:21.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.934+0000 7fe2db7fe700 1 --2- 172.21.15.189:0/977960120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc15d590 0x7fe2dc159660 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4014040 tx=0x7fe2d4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:21.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.934+0000 7fe2c37fe700 1 -- 172.21.15.189:0/977960120 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d401b810 con 0x7fe2dc15d590 2024-04-06T12:28:21.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.934+0000 7fe2c37fe700 1 -- 172.21.15.189:0/977960120 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d4020070 con 0x7fe2dc15d590 2024-04-06T12:28:21.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.934+0000 7fe2c37fe700 1 -- 172.21.15.189:0/977960120 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d401bbb0 con 0x7fe2dc15d590 2024-04-06T12:28:21.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.934+0000 7fe2e1523700 1 -- 172.21.15.189:0/977960120 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2dc15eef0 con 0x7fe2dc15d590 2024-04-06T12:28:21.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.934+0000 7fe2e1523700 1 -- 172.21.15.189:0/977960120 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2dc15f4b0 con 0x7fe2dc15d590 2024-04-06T12:28:21.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.935+0000 7fe2e1523700 1 -- 172.21.15.189:0/977960120 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2dc154b80 con 0x7fe2dc15d590 2024-04-06T12:28:21.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.936+0000 7fe2c37fe700 1 -- 172.21.15.189:0/977960120 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 46) v1 ==== 88983+0+0 (secure 0 0 0) 0x7fe2d401bd10 con 0x7fe2dc15d590 2024-04-06T12:28:21.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.936+0000 7fe2c37fe700 1 --2- 172.21.15.189:0/977960120 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2c806aae0 0x7fe2c806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:21.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.936+0000 7fe2daffd700 1 -- 172.21.15.189:0/977960120 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2c806aae0 msgr2=0x7fe2c806cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:21.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.936+0000 7fe2c37fe700 1 -- 172.21.15.189:0/977960120 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fe2d4022070 con 0x7fe2dc15d590 2024-04-06T12:28:21.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.936+0000 7fe2daffd700 1 --2- 172.21.15.189:0/977960120 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2c806aae0 0x7fe2c806cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:21.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:21.940+0000 7fe2c37fe700 1 -- 172.21.15.189:0/977960120 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe2d40564c0 con 0x7fe2dc15d590 2024-04-06T12:28:21.980 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:21.982+0000 7ffa50309200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:28:22.022 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:22.024+0000 7ffb7e96d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:28:22.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.137+0000 7fe2daffd700 1 -- 172.21.15.189:0/977960120 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2c806aae0 msgr2=0x7fe2c806cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:22.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.137+0000 7fe2daffd700 1 --2- 172.21.15.189:0/977960120 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2c806aae0 0x7fe2c806cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:22.137 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:22.139+0000 7ffa50309200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:28:22.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.143+0000 7fe2e1523700 1 -- 172.21.15.189:0/977960120 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe2dc074990 con 0x7fe2dc15d590 2024-04-06T12:28:22.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.144+0000 7fe2c37fe700 1 -- 172.21.15.189:0/977960120 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe2d402c020 con 0x7fe2dc15d590 2024-04-06T12:28:22.142 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:28:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.146+0000 7fe2e1523700 1 -- 172.21.15.189:0/977960120 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2c806aae0 msgr2=0x7fe2c806cfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.146+0000 7fe2e1523700 1 --2- 172.21.15.189:0/977960120 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2c806aae0 0x7fe2c806cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.146+0000 7fe2e1523700 1 -- 172.21.15.189:0/977960120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc15d590 msgr2=0x7fe2dc159660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.146+0000 7fe2e1523700 1 --2- 172.21.15.189:0/977960120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc15d590 0x7fe2dc159660 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4014040 tx=0x7fe2d4004ad0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.146+0000 7fe2e1523700 1 -- 172.21.15.189:0/977960120 shutdown_connections 2024-04-06T12:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.146+0000 7fe2e1523700 1 --2- 172.21.15.189:0/977960120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2dc15d590 0x7fe2dc159660 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.146+0000 7fe2e1523700 1 --2- 172.21.15.189:0/977960120 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2c806aae0 0x7fe2c806cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.146+0000 7fe2e1523700 1 --2- 172.21.15.189:0/977960120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2dc0b4ed0 0x7fe2dc15e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.146+0000 7fe2e1523700 1 --2- 172.21.15.189:0/977960120 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2dc0b5380 0x7fe2dc15e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.146+0000 7fe2e1523700 1 -- 172.21.15.189:0/977960120 >> 172.21.15.189:0/977960120 conn(0x7fe2dc06bbc0 msgr2=0x7fe2dc0d3680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.146+0000 7fe2e1523700 1 -- 172.21.15.189:0/977960120 shutdown_connections 2024-04-06T12:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.146+0000 7fe2e1523700 1 -- 172.21.15.189:0/977960120 wait complete. 2024-04-06T12:28:22.157 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:28:22.185 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:22.187+0000 7ffb7e96d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:28:22.249 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:22.251+0000 7ffa50309200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:28:22.304 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:22.306+0000 7ffb7e96d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:28:22.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.319+0000 7f0b76c94700 1 Processor -- start 2024-04-06T12:28:22.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.319+0000 7f0b76c94700 1 -- start start 2024-04-06T12:28:22.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.320+0000 7f0b76c94700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0b7015d7e0 0x7f0b700d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:22.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.320+0000 7f0b76c94700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0b700d1cc0 0x7f0b700b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:22.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.320+0000 7f0b76c94700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b700cdf80 con 0x7f0b700d0890 2024-04-06T12:28:22.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.320+0000 7f0b76c94700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b700d3230 con 0x7f0b700d1cc0 2024-04-06T12:28:22.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.320+0000 7f0b76c94700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b700d2da0 con 0x7f0b7015d7e0 2024-04-06T12:28:22.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.320+0000 7f0b75231700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0b700d1cc0 0x7f0b700b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:22.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.320+0000 7f0b75231700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0b700d1cc0 0x7f0b700b66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59020/0 (socket says 172.21.15.189:59020) 2024-04-06T12:28:22.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.320+0000 7f0b75231700 1 -- 172.21.15.189:0/627047242 learned_addr learned my addr 172.21.15.189:0/627047242 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:22.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.320+0000 7f0b74a30700 1 --2- 172.21.15.189:0/627047242 >> v2:172.21.15.192:3300/0 conn(0x7f0b7015d7e0 0x7f0b700d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:22.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.321+0000 7f0b75231700 1 -- 172.21.15.189:0/627047242 >> v2:172.21.15.192:3300/0 conn(0x7f0b7015d7e0 msgr2=0x7f0b700d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:22.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.321+0000 7f0b75231700 1 --2- 172.21.15.189:0/627047242 >> v2:172.21.15.192:3300/0 conn(0x7f0b7015d7e0 0x7f0b700d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.321+0000 7f0b75231700 1 -- 172.21.15.189:0/627047242 >> v1:172.21.15.192:6789/0 conn(0x7f0b700d0890 legacy=0x7f0b700d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:22.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.321+0000 7f0b75231700 1 -- 172.21.15.189:0/627047242 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b700d0cf0 con 0x7f0b700d1cc0 2024-04-06T12:28:22.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.321+0000 7f0b75231700 1 --2- 172.21.15.189:0/627047242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b700d1cc0 0x7f0b700b66b0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f0b7008c190 tx=0x7f0b6800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e2e580890727ea0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:22.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.321+0000 7f0b6f7fe700 1 -- 172.21.15.189:0/627047242 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6800e070 con 0x7f0b700d1cc0 2024-04-06T12:28:22.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.321+0000 7f0b6f7fe700 1 -- 172.21.15.189:0/627047242 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6800b910 con 0x7f0b700d1cc0 2024-04-06T12:28:22.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.321+0000 7f0b6f7fe700 1 -- 172.21.15.189:0/627047242 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6800bcc0 con 0x7f0b700d1cc0 2024-04-06T12:28:22.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.322+0000 7f0b76c94700 1 -- 172.21.15.189:0/627047242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b700d1cc0 msgr2=0x7f0b700b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:22.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.322+0000 7f0b76c94700 1 --2- 172.21.15.189:0/627047242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b700d1cc0 0x7f0b700b66b0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f0b7008c190 tx=0x7f0b6800b330 comp rx=0 tx=0).stop 2024-04-06T12:28:22.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.322+0000 7f0b76c94700 1 -- 172.21.15.189:0/627047242 shutdown_connections 2024-04-06T12:28:22.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.322+0000 7f0b76c94700 1 --2- 172.21.15.189:0/627047242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b700d1cc0 0x7f0b700b66b0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.322+0000 7f0b76c94700 1 --2- 172.21.15.189:0/627047242 >> v2:172.21.15.192:3300/0 conn(0x7f0b7015d7e0 0x7f0b700d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.322+0000 7f0b76c94700 1 -- 172.21.15.189:0/627047242 >> 172.21.15.189:0/627047242 conn(0x7f0b70072250 msgr2=0x7f0b70072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:22.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.322+0000 7f0b76c94700 1 -- 172.21.15.189:0/627047242 shutdown_connections 2024-04-06T12:28:22.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.322+0000 7f0b76c94700 1 -- 172.21.15.189:0/627047242 wait complete. 2024-04-06T12:28:22.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.324+0000 7f0b76c94700 1 Processor -- start 2024-04-06T12:28:22.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.324+0000 7f0b76c94700 1 -- start start 2024-04-06T12:28:22.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.325+0000 7f0b76c94700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b700d0890 0x7f0b700b37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:22.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.325+0000 7f0b76c94700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b700d1cc0 0x7f0b700c9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:22.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.325+0000 7f0b76c94700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b7015d7e0 0x7f0b70158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:22.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.325+0000 7f0b76c94700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b700b4810 con 0x7f0b700d1cc0 2024-04-06T12:28:22.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.325+0000 7f0b76c94700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b700cb830 con 0x7f0b700d0890 2024-04-06T12:28:22.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.325+0000 7f0b76c94700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b700b4a10 con 0x7f0b7015d7e0 2024-04-06T12:28:22.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.325+0000 7f0b6ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b700d1cc0 0x7f0b700c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:22.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.325+0000 7f0b6ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b700d1cc0 0x7f0b700c9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59034/0 (socket says 172.21.15.189:59034) 2024-04-06T12:28:22.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.325+0000 7f0b6ffff700 1 -- 172.21.15.189:0/3221328087 learned_addr learned my addr 172.21.15.189:0/3221328087 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:22.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.325+0000 7f0b74a30700 1 --2- 172.21.15.189:0/3221328087 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b700d0890 0x7f0b700b37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:22.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.325+0000 7f0b75231700 1 --2- 172.21.15.189:0/3221328087 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b7015d7e0 0x7f0b70158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:22.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.325+0000 7f0b74a30700 1 -- 172.21.15.189:0/3221328087 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b7015d7e0 msgr2=0x7f0b70158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:22.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.325+0000 7f0b74a30700 1 --2- 172.21.15.189:0/3221328087 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b7015d7e0 0x7f0b70158f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.325+0000 7f0b74a30700 1 -- 172.21.15.189:0/3221328087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b700d1cc0 msgr2=0x7f0b700c9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:22.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.326+0000 7f0b74a30700 1 --2- 172.21.15.189:0/3221328087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b700d1cc0 0x7f0b700c9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.326+0000 7f0b74a30700 1 -- 172.21.15.189:0/3221328087 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b68009b30 con 0x7f0b700d0890 2024-04-06T12:28:22.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.326+0000 7f0b6ffff700 1 --2- 172.21.15.189:0/3221328087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b700d1cc0 0x7f0b700c9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:28:22.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.326+0000 7f0b74a30700 1 --2- 172.21.15.189:0/3221328087 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b700d0890 0x7f0b700b37e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0b6400e870 tx=0x7f0b6400eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:22.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.326+0000 7f0b6d7fa700 1 -- 172.21.15.189:0/3221328087 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b64013680 con 0x7f0b700d0890 2024-04-06T12:28:22.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.326+0000 7f0b6d7fa700 1 -- 172.21.15.189:0/3221328087 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b640137e0 con 0x7f0b700d0890 2024-04-06T12:28:22.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.326+0000 7f0b6d7fa700 1 -- 172.21.15.189:0/3221328087 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b64013b90 con 0x7f0b700d0890 2024-04-06T12:28:22.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.326+0000 7f0b76c94700 1 -- 172.21.15.189:0/3221328087 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b700b4cf0 con 0x7f0b700d0890 2024-04-06T12:28:22.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.326+0000 7f0b76c94700 1 -- 172.21.15.189:0/3221328087 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b700b52b0 con 0x7f0b700d0890 2024-04-06T12:28:22.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.327+0000 7f0b76c94700 1 -- 172.21.15.189:0/3221328087 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b700cb9c0 con 0x7f0b700d0890 2024-04-06T12:28:22.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.327+0000 7f0b6d7fa700 1 -- 172.21.15.189:0/3221328087 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 47) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f0b64013cf0 con 0x7f0b700d0890 2024-04-06T12:28:22.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.328+0000 7f0b6d7fa700 1 --2- 172.21.15.189:0/3221328087 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b5406adb0 0x7f0b5406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:22.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.328+0000 7f0b6ffff700 1 -- 172.21.15.189:0/3221328087 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b5406adb0 msgr2=0x7f0b5406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:22.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.328+0000 7f0b6d7fa700 1 -- 172.21.15.189:0/3221328087 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f0b64089450 con 0x7f0b700d0890 2024-04-06T12:28:22.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.328+0000 7f0b6ffff700 1 --2- 172.21.15.189:0/3221328087 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b5406adb0 0x7f0b5406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:22.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.332+0000 7f0b6d7fa700 1 -- 172.21.15.189:0/3221328087 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0b64054280 con 0x7f0b700d0890 2024-04-06T12:28:22.331 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:22.333+0000 7ffa50309200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:28:22.391 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:22.393+0000 7ffb7e96d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:28:22.406 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:22.408+0000 7ffa50309200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:28:22.468 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:22.470+0000 7ffb7e96d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:28:22.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.529+0000 7f0b6ffff700 1 -- 172.21.15.189:0/3221328087 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b5406adb0 msgr2=0x7f0b5406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:22.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.529+0000 7f0b6ffff700 1 --2- 172.21.15.189:0/3221328087 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b5406adb0 0x7f0b5406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:22.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.535+0000 7f0b76c94700 1 -- 172.21.15.189:0/3221328087 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0b70074990 con 0x7f0b700d0890 2024-04-06T12:28:22.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.536+0000 7f0b6d7fa700 1 -- 172.21.15.189:0/3221328087 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0b640578a0 con 0x7f0b700d0890 2024-04-06T12:28:22.534 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:28:22.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.538+0000 7f0b76c94700 1 -- 172.21.15.189:0/3221328087 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b5406adb0 msgr2=0x7f0b5406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:22.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.538+0000 7f0b76c94700 1 --2- 172.21.15.189:0/3221328087 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b5406adb0 0x7f0b5406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.538+0000 7f0b76c94700 1 -- 172.21.15.189:0/3221328087 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b700d0890 msgr2=0x7f0b700b37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:22.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.538+0000 7f0b76c94700 1 --2- 172.21.15.189:0/3221328087 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b700d0890 0x7f0b700b37e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0b6400e870 tx=0x7f0b6400eb80 comp rx=0 tx=0).stop 2024-04-06T12:28:22.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.539+0000 7f0b76c94700 1 -- 172.21.15.189:0/3221328087 shutdown_connections 2024-04-06T12:28:22.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.539+0000 7f0b76c94700 1 --2- 172.21.15.189:0/3221328087 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b7015d7e0 0x7f0b70158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.539+0000 7f0b76c94700 1 --2- 172.21.15.189:0/3221328087 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b5406adb0 0x7f0b5406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.539+0000 7f0b76c94700 1 --2- 172.21.15.189:0/3221328087 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b700d0890 0x7f0b700b37e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.539+0000 7f0b76c94700 1 --2- 172.21.15.189:0/3221328087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b700d1cc0 0x7f0b700c9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.539+0000 7f0b76c94700 1 -- 172.21.15.189:0/3221328087 >> 172.21.15.189:0/3221328087 conn(0x7f0b70072250 msgr2=0x7f0b7015bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:22.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.539+0000 7f0b76c94700 1 -- 172.21.15.189:0/3221328087 shutdown_connections 2024-04-06T12:28:22.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.539+0000 7f0b76c94700 1 -- 172.21.15.189:0/3221328087 wait complete. 2024-04-06T12:28:22.551 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:28:22.610 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:22.611+0000 7ffa50309200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:28:22.685 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:22.687+0000 7ffa50309200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:28:22.688 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:22.683+0000 7ffb7e96d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:28:22.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.705+0000 7f1a37057700 1 Processor -- start 2024-04-06T12:28:22.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.705+0000 7f1a37057700 1 -- start start 2024-04-06T12:28:22.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.705+0000 7f1a37057700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a300d0a60 0x7f1a300d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:22.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.705+0000 7f1a37057700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1a300d1af0 0x7f1a300b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:22.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.705+0000 7f1a37057700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a300cdf80 con 0x7f1a3015d7e0 2024-04-06T12:28:22.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.705+0000 7f1a37057700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a300d1f80 con 0x7f1a300d1af0 2024-04-06T12:28:22.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.706+0000 7f1a37057700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a300d3230 con 0x7f1a300d0a60 2024-04-06T12:28:22.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.706+0000 7f1a355f4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1a300d1af0 0x7f1a300b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:22.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.706+0000 7f1a2ffff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a300d0a60 0x7f1a300d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:22.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.706+0000 7f1a34df3700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f1a3015d7e0 0x7f1a300d0350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:51372/0 (socket says 172.21.15.189:51372) 2024-04-06T12:28:22.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.706+0000 7f1a34df3700 1 -- 172.21.15.189:0/3669262452 learned_addr learned my addr 172.21.15.189:0/3669262452 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:22.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.706+0000 7f1a355f4700 1 -- 172.21.15.189:0/3669262452 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a300d0a60 msgr2=0x7f1a300d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:22.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.706+0000 7f1a355f4700 1 --2- 172.21.15.189:0/3669262452 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a300d0a60 0x7f1a300d15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.706+0000 7f1a2f7fe700 1 -- 172.21.15.189:0/3669262452 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698787362 0 0) 0x7f1a300cdf80 con 0x7f1a3015d7e0 2024-04-06T12:28:22.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.706+0000 7f1a355f4700 1 -- 172.21.15.189:0/3669262452 >> v1:172.21.15.192:6789/0 conn(0x7f1a3015d7e0 legacy=0x7f1a300d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:22.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.706+0000 7f1a355f4700 1 -- 172.21.15.189:0/3669262452 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a300d2da0 con 0x7f1a300d1af0 2024-04-06T12:28:22.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.706+0000 7f1a355f4700 1 --2- 172.21.15.189:0/3669262452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a300d1af0 0x7f1a300b66b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1a28009d90 tx=0x7f1a2800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da542ace2e3f7ba2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:22.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.706+0000 7f1a2f7fe700 1 -- 172.21.15.189:0/3669262452 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2800e070 con 0x7f1a300d1af0 2024-04-06T12:28:22.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.707+0000 7f1a2f7fe700 1 -- 172.21.15.189:0/3669262452 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2800bb20 con 0x7f1a300d1af0 2024-04-06T12:28:22.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.707+0000 7f1a2f7fe700 1 -- 172.21.15.189:0/3669262452 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a28012430 con 0x7f1a300d1af0 2024-04-06T12:28:22.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.707+0000 7f1a37057700 1 -- 172.21.15.189:0/3669262452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a300d1af0 msgr2=0x7f1a300b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:22.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.707+0000 7f1a37057700 1 --2- 172.21.15.189:0/3669262452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a300d1af0 0x7f1a300b66b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1a28009d90 tx=0x7f1a2800b330 comp rx=0 tx=0).stop 2024-04-06T12:28:22.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.707+0000 7f1a37057700 1 -- 172.21.15.189:0/3669262452 shutdown_connections 2024-04-06T12:28:22.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.707+0000 7f1a37057700 1 --2- 172.21.15.189:0/3669262452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a300d1af0 0x7f1a300b66b0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.707+0000 7f1a37057700 1 --2- 172.21.15.189:0/3669262452 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a300d0a60 0x7f1a300d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.707+0000 7f1a37057700 1 -- 172.21.15.189:0/3669262452 >> 172.21.15.189:0/3669262452 conn(0x7f1a30072250 msgr2=0x7f1a30072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:22.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.707+0000 7f1a37057700 1 -- 172.21.15.189:0/3669262452 shutdown_connections 2024-04-06T12:28:22.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.707+0000 7f1a37057700 1 -- 172.21.15.189:0/3669262452 wait complete. 2024-04-06T12:28:22.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.708+0000 7f1a37057700 1 Processor -- start 2024-04-06T12:28:22.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.708+0000 7f1a37057700 1 -- start start 2024-04-06T12:28:22.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.709+0000 7f1a37057700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a300d0a60 0x7f1a300cb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:22.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.709+0000 7f1a37057700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a300d1af0 0x7f1a300ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:22.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.709+0000 7f1a37057700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a3015d7e0 0x7f1a301f67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:22.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.709+0000 7f1a37057700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a301f63c0 con 0x7f1a300d0a60 2024-04-06T12:28:22.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.709+0000 7f1a37057700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a300cbde0 con 0x7f1a300d1af0 2024-04-06T12:28:22.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.709+0000 7f1a37057700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a301f5db0 con 0x7f1a3015d7e0 2024-04-06T12:28:22.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.710+0000 7f1a34df3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a300d0a60 0x7f1a300cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:22.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.710+0000 7f1a355f4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a3015d7e0 0x7f1a301f67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:22.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.710+0000 7f1a34df3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a300d0a60 0x7f1a300cb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59050/0 (socket says 172.21.15.189:59050) 2024-04-06T12:28:22.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.710+0000 7f1a355f4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a3015d7e0 0x7f1a301f67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:60042/0 (socket says 172.21.15.189:60042) 2024-04-06T12:28:22.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.710+0000 7f1a34df3700 1 -- 172.21.15.189:0/2516662910 learned_addr learned my addr 172.21.15.189:0/2516662910 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:22.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.710+0000 7f1a2ffff700 1 --2- 172.21.15.189:0/2516662910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a300d1af0 0x7f1a300ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:22.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.710+0000 7f1a34df3700 1 -- 172.21.15.189:0/2516662910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a3015d7e0 msgr2=0x7f1a301f67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:22.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.710+0000 7f1a34df3700 1 --2- 172.21.15.189:0/2516662910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a3015d7e0 0x7f1a301f67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.710+0000 7f1a34df3700 1 -- 172.21.15.189:0/2516662910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a300d1af0 msgr2=0x7f1a300ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:22.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.710+0000 7f1a34df3700 1 --2- 172.21.15.189:0/2516662910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a300d1af0 0x7f1a300ca970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.710+0000 7f1a34df3700 1 -- 172.21.15.189:0/2516662910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a28009a70 con 0x7f1a300d0a60 2024-04-06T12:28:22.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.710+0000 7f1a34df3700 1 --2- 172.21.15.189:0/2516662910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a300d0a60 0x7f1a300cb140 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400c750 tx=0x7f1a2400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:22.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.710+0000 7f1a2d7fa700 1 -- 172.21.15.189:0/2516662910 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a24004c80 con 0x7f1a300d0a60 2024-04-06T12:28:22.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.710+0000 7f1a2d7fa700 1 -- 172.21.15.189:0/2516662910 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a24004e00 con 0x7f1a300d0a60 2024-04-06T12:28:22.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.710+0000 7f1a37057700 1 -- 172.21.15.189:0/2516662910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a300caeb0 con 0x7f1a300d0a60 2024-04-06T12:28:22.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.710+0000 7f1a2d7fa700 1 -- 172.21.15.189:0/2516662910 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a24007310 con 0x7f1a300d0a60 2024-04-06T12:28:22.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.710+0000 7f1a37057700 1 -- 172.21.15.189:0/2516662910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a301f6d20 con 0x7f1a300d0a60 2024-04-06T12:28:22.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.711+0000 7f1a37057700 1 -- 172.21.15.189:0/2516662910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a30074990 con 0x7f1a300d0a60 2024-04-06T12:28:22.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.712+0000 7f1a2d7fa700 1 -- 172.21.15.189:0/2516662910 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 47) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f1a24007470 con 0x7f1a300d0a60 2024-04-06T12:28:22.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.712+0000 7f1a2d7fa700 1 --2- 172.21.15.189:0/2516662910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a1406ae00 0x7f1a1406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:22.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.712+0000 7f1a2ffff700 1 -- 172.21.15.189:0/2516662910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a1406ae00 msgr2=0x7f1a1406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:22.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.712+0000 7f1a2d7fa700 1 -- 172.21.15.189:0/2516662910 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f1a24089040 con 0x7f1a300d0a60 2024-04-06T12:28:22.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.712+0000 7f1a2ffff700 1 --2- 172.21.15.189:0/2516662910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a1406ae00 0x7f1a1406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:22.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.716+0000 7f1a2d7fa700 1 -- 172.21.15.189:0/2516662910 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1a24053fe0 con 0x7f1a300d0a60 2024-04-06T12:28:22.760 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:22.761+0000 7ffb7e96d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:28:22.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.913+0000 7f1a2ffff700 1 -- 172.21.15.189:0/2516662910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a1406ae00 msgr2=0x7f1a1406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:22.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.913+0000 7f1a2ffff700 1 --2- 172.21.15.189:0/2516662910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a1406ae00 0x7f1a1406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:22.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.917+0000 7f1a37057700 1 -- 172.21.15.189:0/2516662910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1a301f6d20 con 0x7f1a300d0a60 2024-04-06T12:28:22.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.917+0000 7f1a2d7fa700 1 -- 172.21.15.189:0/2516662910 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1a301f6d20 con 0x7f1a300d0a60 2024-04-06T12:28:22.915 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:28:22.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.920+0000 7f1a37057700 1 -- 172.21.15.189:0/2516662910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a1406ae00 msgr2=0x7f1a1406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:22.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.920+0000 7f1a37057700 1 --2- 172.21.15.189:0/2516662910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a1406ae00 0x7f1a1406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.920+0000 7f1a37057700 1 -- 172.21.15.189:0/2516662910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a300d0a60 msgr2=0x7f1a300cb140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:22.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.920+0000 7f1a37057700 1 --2- 172.21.15.189:0/2516662910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a300d0a60 0x7f1a300cb140 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400c750 tx=0x7f1a2400ca60 comp rx=0 tx=0).stop 2024-04-06T12:28:22.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.920+0000 7f1a37057700 1 -- 172.21.15.189:0/2516662910 shutdown_connections 2024-04-06T12:28:22.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.920+0000 7f1a37057700 1 --2- 172.21.15.189:0/2516662910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a3015d7e0 0x7f1a301f67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.920+0000 7f1a37057700 1 --2- 172.21.15.189:0/2516662910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a300d0a60 0x7f1a300cb140 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.920+0000 7f1a37057700 1 --2- 172.21.15.189:0/2516662910 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a1406ae00 0x7f1a1406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.920+0000 7f1a37057700 1 --2- 172.21.15.189:0/2516662910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a300d1af0 0x7f1a300ca970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:22.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.920+0000 7f1a37057700 1 -- 172.21.15.189:0/2516662910 >> 172.21.15.189:0/2516662910 conn(0x7f1a30072250 msgr2=0x7f1a3015ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:22.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.920+0000 7f1a37057700 1 -- 172.21.15.189:0/2516662910 shutdown_connections 2024-04-06T12:28:22.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:22.920+0000 7f1a37057700 1 -- 172.21.15.189:0/2516662910 wait complete. 2024-04-06T12:28:22.932 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T12:28:23.005 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:23.007+0000 7ffa50309200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:28:23.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.060+0000 7f8def26b700 1 Processor -- start 2024-04-06T12:28:23.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.060+0000 7f8def26b700 1 -- start start 2024-04-06T12:28:23.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.061+0000 7f8def26b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8de815d7b0 0x7f8de80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:23.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.061+0000 7f8def26b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de80b3990 con 0x7f8de815e500 2024-04-06T12:28:23.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.061+0000 7f8def26b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de80b3060 con 0x7f8de806a730 2024-04-06T12:28:23.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.061+0000 7f8def26b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de815e950 con 0x7f8de815d7b0 2024-04-06T12:28:23.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.061+0000 7f8ded007700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8de815d7b0 0x7f8de80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:23.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.061+0000 7f8ded808700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8de815e500 0x7f8de80717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33636/0 (socket says 172.21.15.189:33636) 2024-04-06T12:28:23.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.061+0000 7f8ded808700 1 -- 172.21.15.189:0/1538754159 learned_addr learned my addr 172.21.15.189:0/1538754159 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:23.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.061+0000 7f8ded007700 1 -- 172.21.15.189:0/1538754159 >> v1:172.21.15.192:6789/0 conn(0x7f8de806a730 legacy=0x7f8de815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:23.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.061+0000 7f8ded007700 1 -- 172.21.15.189:0/1538754159 >> v1:172.21.15.189:6789/0 conn(0x7f8de815e500 legacy=0x7f8de80717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:23.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.062+0000 7f8ded007700 1 -- 172.21.15.189:0/1538754159 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de80b3340 con 0x7f8de815d7b0 2024-04-06T12:28:23.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.061+0000 7f8ddffff700 1 -- 172.21.15.189:0/1538754159 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500639780 0 0) 0x7f8de80b3990 con 0x7f8de815e500 2024-04-06T12:28:23.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.062+0000 7f8ded007700 1 --2- 172.21.15.189:0/1538754159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8de815d7b0 0x7f8de80b3e20 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f8de4009840 tx=0x7f8de4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f81083fe2b0ba4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:23.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.062+0000 7f8ddffff700 1 -- 172.21.15.189:0/1538754159 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de4011830 con 0x7f8de815d7b0 2024-04-06T12:28:23.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.062+0000 7f8ddffff700 1 -- 172.21.15.189:0/1538754159 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de4011990 con 0x7f8de815d7b0 2024-04-06T12:28:23.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.062+0000 7f8ddffff700 1 -- 172.21.15.189:0/1538754159 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de4011d20 con 0x7f8de815d7b0 2024-04-06T12:28:23.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.062+0000 7f8def26b700 1 -- 172.21.15.189:0/1538754159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8de815d7b0 msgr2=0x7f8de80b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:23.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.062+0000 7f8def26b700 1 --2- 172.21.15.189:0/1538754159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8de815d7b0 0x7f8de80b3e20 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f8de4009840 tx=0x7f8de4009b50 comp rx=0 tx=0).stop 2024-04-06T12:28:23.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.062+0000 7f8def26b700 1 -- 172.21.15.189:0/1538754159 shutdown_connections 2024-04-06T12:28:23.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.062+0000 7f8def26b700 1 --2- 172.21.15.189:0/1538754159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8de815d7b0 0x7f8de80b3e20 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.062+0000 7f8def26b700 1 -- 172.21.15.189:0/1538754159 >> 172.21.15.189:0/1538754159 conn(0x7f8de80d00b0 msgr2=0x7f8de80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:23.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.063+0000 7f8def26b700 1 -- 172.21.15.189:0/1538754159 shutdown_connections 2024-04-06T12:28:23.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.063+0000 7f8def26b700 1 -- 172.21.15.189:0/1538754159 wait complete. 2024-04-06T12:28:23.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.064+0000 7f8def26b700 1 Processor -- start 2024-04-06T12:28:23.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.064+0000 7f8def26b700 1 -- start start 2024-04-06T12:28:23.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8def26b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8de806a730 0x7f8de8153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:23.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8def26b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8de815d7b0 0x7f8de80ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:23.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8def26b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8de815e500 0x7f8de815c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:23.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8def26b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de80af270 con 0x7f8de806a730 2024-04-06T12:28:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8def26b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de80cee50 con 0x7f8de815e500 2024-04-06T12:28:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8def26b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de80cf130 con 0x7f8de815d7b0 2024-04-06T12:28:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8ded007700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8de806a730 0x7f8de8153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8ded007700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8de806a730 0x7f8de8153ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59070/0 (socket says 172.21.15.189:59070) 2024-04-06T12:28:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8ded007700 1 -- 172.21.15.189:0/3294431411 learned_addr learned my addr 172.21.15.189:0/3294431411 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8ded808700 1 --2- 172.21.15.189:0/3294431411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8de815e500 0x7f8de815c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8dec806700 1 --2- 172.21.15.189:0/3294431411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8de815d7b0 0x7f8de80ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8ded007700 1 -- 172.21.15.189:0/3294431411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8de815d7b0 msgr2=0x7f8de80ce3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8ded007700 1 --2- 172.21.15.189:0/3294431411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8de815d7b0 0x7f8de80ce3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8ded007700 1 -- 172.21.15.189:0/3294431411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8de815e500 msgr2=0x7f8de815c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8ded007700 1 --2- 172.21.15.189:0/3294431411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8de815e500 0x7f8de815c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8ded007700 1 -- 172.21.15.189:0/3294431411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de40094a0 con 0x7f8de806a730 2024-04-06T12:28:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8ded007700 1 --2- 172.21.15.189:0/3294431411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8de806a730 0x7f8de8153ae0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f8de40090b0 tx=0x7f8de4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8dddffb700 1 -- 172.21.15.189:0/3294431411 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de401a7e0 con 0x7f8de806a730 2024-04-06T12:28:23.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8dddffb700 1 -- 172.21.15.189:0/3294431411 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de401a940 con 0x7f8de806a730 2024-04-06T12:28:23.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8def26b700 1 -- 172.21.15.189:0/3294431411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de80aec20 con 0x7f8de806a730 2024-04-06T12:28:23.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.066+0000 7f8dddffb700 1 -- 172.21.15.189:0/3294431411 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de401acb0 con 0x7f8de806a730 2024-04-06T12:28:23.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.067+0000 7f8def26b700 1 -- 172.21.15.189:0/3294431411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de80cd6d0 con 0x7f8de806a730 2024-04-06T12:28:23.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.068+0000 7f8def26b700 1 -- 172.21.15.189:0/3294431411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de8067bb0 con 0x7f8de806a730 2024-04-06T12:28:23.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.068+0000 7f8dddffb700 1 -- 172.21.15.189:0/3294431411 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 47) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f8de4022070 con 0x7f8de806a730 2024-04-06T12:28:23.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.068+0000 7f8dddffb700 1 --2- 172.21.15.189:0/3294431411 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8dcc06adb0 0x7f8dcc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:23.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.069+0000 7f8dec806700 1 -- 172.21.15.189:0/3294431411 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8dcc06adb0 msgr2=0x7f8dcc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:23.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.069+0000 7f8dddffb700 1 -- 172.21.15.189:0/3294431411 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f8de408a8c0 con 0x7f8de806a730 2024-04-06T12:28:23.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.069+0000 7f8dec806700 1 --2- 172.21.15.189:0/3294431411 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8dcc06adb0 0x7f8dcc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:23.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.073+0000 7f8dddffb700 1 -- 172.21.15.189:0/3294431411 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8de4055950 con 0x7f8de806a730 2024-04-06T12:28:23.110 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:23.112+0000 7ffb7e96d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:28:23.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.270+0000 7f8dec806700 1 -- 172.21.15.189:0/3294431411 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8dcc06adb0 msgr2=0x7f8dcc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:23.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.270+0000 7f8dec806700 1 --2- 172.21.15.189:0/3294431411 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8dcc06adb0 0x7f8dcc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:23.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.313+0000 7f8def26b700 1 -- 172.21.15.189:0/3294431411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8de8074990 con 0x7f8de806a730 2024-04-06T12:28:23.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.313+0000 7f8dddffb700 1 -- 172.21.15.189:0/3294431411 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v99) v1 ==== 80+0+467 (secure 0 0 0) 0x7f8de40291f0 con 0x7f8de806a730 2024-04-06T12:28:23.312 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/530172869 2024-04-07T12:28:19.179156+0000 2024-04-06T12:28:23.312 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/4174795471 2024-04-07T12:28:19.179156+0000 2024-04-06T12:28:23.312 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2207020726 2024-04-07T12:28:19.179156+0000 2024-04-06T12:28:23.312 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2070668132 2024-04-07T12:28:19.179156+0000 2024-04-06T12:28:23.312 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1706294115 2024-04-07T12:28:19.179156+0000 2024-04-06T12:28:23.312 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/4285218889 2024-04-07T12:28:19.179156+0000 2024-04-06T12:28:23.312 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6833/47128 2024-04-07T12:28:19.179156+0000 2024-04-06T12:28:23.312 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6832/47128 2024-04-07T12:28:19.179156+0000 2024-04-06T12:28:23.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.316+0000 7f8def26b700 1 -- 172.21.15.189:0/3294431411 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8dcc06adb0 msgr2=0x7f8dcc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:23.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.316+0000 7f8def26b700 1 --2- 172.21.15.189:0/3294431411 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8dcc06adb0 0x7f8dcc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.316+0000 7f8def26b700 1 -- 172.21.15.189:0/3294431411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8de806a730 msgr2=0x7f8de8153ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:23.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.316+0000 7f8def26b700 1 --2- 172.21.15.189:0/3294431411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8de806a730 0x7f8de8153ae0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f8de40090b0 tx=0x7f8de4004750 comp rx=0 tx=0).stop 2024-04-06T12:28:23.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.316+0000 7f8def26b700 1 -- 172.21.15.189:0/3294431411 shutdown_connections 2024-04-06T12:28:23.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.316+0000 7f8def26b700 1 --2- 172.21.15.189:0/3294431411 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8dcc06adb0 0x7f8dcc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.316+0000 7f8def26b700 1 --2- 172.21.15.189:0/3294431411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8de815e500 0x7f8de815c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.316+0000 7f8def26b700 1 --2- 172.21.15.189:0/3294431411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8de806a730 0x7f8de8153ae0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.316+0000 7f8def26b700 1 --2- 172.21.15.189:0/3294431411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8de815d7b0 0x7f8de80ce3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.316+0000 7f8def26b700 1 -- 172.21.15.189:0/3294431411 >> 172.21.15.189:0/3294431411 conn(0x7f8de80d00b0 msgr2=0x7f8de815f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:23.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.316+0000 7f8def26b700 1 -- 172.21.15.189:0/3294431411 shutdown_connections 2024-04-06T12:28:23.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.316+0000 7f8def26b700 1 -- 172.21.15.189:0/3294431411 wait complete. 2024-04-06T12:28:23.316 INFO:teuthology.orchestra.run.smithi189.stderr:listed 8 entries 2024-04-06T12:28:23.324 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:23.326+0000 7ffa50309200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:28:23.327 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-06T12:28:23.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:23.327 DEBUG:teuthology.orchestra.run.smithi189:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-06T12:28:23.367 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T12:28:23.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:23.367 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-06T12:28:23.424 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:28:23.424 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:28:23.442 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:28:23.442 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:28:23.443 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:28:23.448 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:23.450+0000 7ffb7e96d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:28:23.466 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:28:23.466 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:28:23.466 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:28:23.486 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:23.488+0000 7ffa50309200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:28:23.500 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:28:23.500 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:28:23.562 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:23.564+0000 7ffa50309200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:28:23.622 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:23.624+0000 7ffb7e96d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:28:23.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.636+0000 7fc8bd5e6700 1 Processor -- start 2024-04-06T12:28:23.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.636+0000 7fc8bd5e6700 1 -- start start 2024-04-06T12:28:23.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.636+0000 7fc8bd5e6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc8b815d7b0 0x7fc8b80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:23.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.636+0000 7fc8bd5e6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b806a730 0x7fc8b806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:23.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.636+0000 7fc8bd5e6700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc8b815ddf0 0x7fc8b815e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:23.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.636+0000 7fc8bd5e6700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b806b0f0 con 0x7fc8b815d7b0 2024-04-06T12:28:23.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.636+0000 7fc8bd5e6700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b80b3990 con 0x7fc8b815ddf0 2024-04-06T12:28:23.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.636+0000 7fc8bd5e6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b80b3060 con 0x7fc8b806a730 2024-04-06T12:28:23.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.637+0000 7fc8b6ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc8b815d7b0 0x7fc8b80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:23.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.637+0000 7fc8b6ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc8b815d7b0 0x7fc8b80b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59080/0 (socket says 172.21.15.189:59080) 2024-04-06T12:28:23.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.637+0000 7fc8b6ffd700 1 -- 172.21.15.189:0/1846018241 learned_addr learned my addr 172.21.15.189:0/1846018241 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:23.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.637+0000 7fc8b77fe700 1 --2- 172.21.15.189:0/1846018241 >> v2:172.21.15.192:3300/0 conn(0x7fc8b815ddf0 0x7fc8b815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:23.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.637+0000 7fc8b67fc700 1 --2- 172.21.15.189:0/1846018241 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b806a730 0x7fc8b806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:23.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.637+0000 7fc8b6ffd700 1 -- 172.21.15.189:0/1846018241 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b806a730 msgr2=0x7fc8b806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:23.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.637+0000 7fc8b6ffd700 1 --2- 172.21.15.189:0/1846018241 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b806a730 0x7fc8b806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.637+0000 7fc8b6ffd700 1 -- 172.21.15.189:0/1846018241 >> v2:172.21.15.192:3300/0 conn(0x7fc8b815ddf0 msgr2=0x7fc8b815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:23.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.637+0000 7fc8b6ffd700 1 --2- 172.21.15.189:0/1846018241 >> v2:172.21.15.192:3300/0 conn(0x7fc8b815ddf0 0x7fc8b815e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.637+0000 7fc8b6ffd700 1 -- 172.21.15.189:0/1846018241 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b80b3340 con 0x7fc8b815d7b0 2024-04-06T12:28:23.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.637+0000 7fc8b6ffd700 1 --2- 172.21.15.189:0/1846018241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8b815d7b0 0x7fc8b80b3e20 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fc8b80b4360 tx=0x7fc8b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=749eae4ce9118920 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:23.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.637+0000 7fc8b5ffb700 1 -- 172.21.15.189:0/1846018241 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b000e070 con 0x7fc8b815d7b0 2024-04-06T12:28:23.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.637+0000 7fc8b5ffb700 1 -- 172.21.15.189:0/1846018241 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b000b910 con 0x7fc8b815d7b0 2024-04-06T12:28:23.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.637+0000 7fc8b5ffb700 1 -- 172.21.15.189:0/1846018241 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b000bcc0 con 0x7fc8b815d7b0 2024-04-06T12:28:23.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.638+0000 7fc8bd5e6700 1 -- 172.21.15.189:0/1846018241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8b815d7b0 msgr2=0x7fc8b80b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:23.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.638+0000 7fc8bd5e6700 1 --2- 172.21.15.189:0/1846018241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8b815d7b0 0x7fc8b80b3e20 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fc8b80b4360 tx=0x7fc8b000b330 comp rx=0 tx=0).stop 2024-04-06T12:28:23.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.638+0000 7fc8bd5e6700 1 -- 172.21.15.189:0/1846018241 shutdown_connections 2024-04-06T12:28:23.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.638+0000 7fc8bd5e6700 1 --2- 172.21.15.189:0/1846018241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8b815d7b0 0x7fc8b80b3e20 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.638+0000 7fc8bd5e6700 1 --2- 172.21.15.189:0/1846018241 >> v2:172.21.15.192:3300/0 conn(0x7fc8b815ddf0 0x7fc8b815e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.638+0000 7fc8bd5e6700 1 --2- 172.21.15.189:0/1846018241 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b806a730 0x7fc8b806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.638+0000 7fc8bd5e6700 1 -- 172.21.15.189:0/1846018241 >> 172.21.15.189:0/1846018241 conn(0x7fc8b80d00b0 msgr2=0x7fc8b80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:23.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.638+0000 7fc8bd5e6700 1 -- 172.21.15.189:0/1846018241 shutdown_connections 2024-04-06T12:28:23.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.638+0000 7fc8bd5e6700 1 -- 172.21.15.189:0/1846018241 wait complete. 2024-04-06T12:28:23.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.639+0000 7fc8bd5e6700 1 Processor -- start 2024-04-06T12:28:23.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.639+0000 7fc8bd5e6700 1 -- start start 2024-04-06T12:28:23.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.640+0000 7fc8bd5e6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8b806a730 0x7fc8b81492d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:23.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.640+0000 7fc8bd5e6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8b815d7b0 0x7fc8b80cf570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:23.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.640+0000 7fc8bd5e6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b815ddf0 0x7fc8b8146a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:23.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.640+0000 7fc8bd5e6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b8147420 con 0x7fc8b806a730 2024-04-06T12:28:23.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.640+0000 7fc8bd5e6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b80cfab0 con 0x7fc8b815d7b0 2024-04-06T12:28:23.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.640+0000 7fc8bd5e6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b814a220 con 0x7fc8b815ddf0 2024-04-06T12:28:23.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.640+0000 7fc8b6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8b806a730 0x7fc8b81492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:23.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.640+0000 7fc8b6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8b806a730 0x7fc8b81492d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59096/0 (socket says 172.21.15.189:59096) 2024-04-06T12:28:23.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.640+0000 7fc8b6ffd700 1 -- 172.21.15.189:0/2556018765 learned_addr learned my addr 172.21.15.189:0/2556018765 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:23.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.640+0000 7fc8b67fc700 1 --2- 172.21.15.189:0/2556018765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8b815d7b0 0x7fc8b80cf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:23.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.640+0000 7fc8b77fe700 1 --2- 172.21.15.189:0/2556018765 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b815ddf0 0x7fc8b8146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:23.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.640+0000 7fc8b6ffd700 1 -- 172.21.15.189:0/2556018765 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b815ddf0 msgr2=0x7fc8b8146a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:23.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.640+0000 7fc8b6ffd700 1 --2- 172.21.15.189:0/2556018765 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b815ddf0 0x7fc8b8146a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.640+0000 7fc8b6ffd700 1 -- 172.21.15.189:0/2556018765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8b815d7b0 msgr2=0x7fc8b80cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:23.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.640+0000 7fc8b6ffd700 1 --2- 172.21.15.189:0/2556018765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8b815d7b0 0x7fc8b80cf570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.641+0000 7fc8b6ffd700 1 -- 172.21.15.189:0/2556018765 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b0009b30 con 0x7fc8b806a730 2024-04-06T12:28:23.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.641+0000 7fc8b67fc700 1 --2- 172.21.15.189:0/2556018765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8b815d7b0 0x7fc8b80cf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:28:23.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.641+0000 7fc8b6ffd700 1 --2- 172.21.15.189:0/2556018765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8b806a730 0x7fc8b81492d0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fc8b806c820 tx=0x7fc8b0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:23.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.641+0000 7fc89f7fe700 1 -- 172.21.15.189:0/2556018765 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b00099b0 con 0x7fc8b806a730 2024-04-06T12:28:23.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.641+0000 7fc89f7fe700 1 -- 172.21.15.189:0/2556018765 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b000ce80 con 0x7fc8b806a730 2024-04-06T12:28:23.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.641+0000 7fc8bd5e6700 1 -- 172.21.15.189:0/2556018765 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b8146f50 con 0x7fc8b806a730 2024-04-06T12:28:23.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.641+0000 7fc89f7fe700 1 -- 172.21.15.189:0/2556018765 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b0004470 con 0x7fc8b806a730 2024-04-06T12:28:23.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.641+0000 7fc8bd5e6700 1 -- 172.21.15.189:0/2556018765 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b8149c60 con 0x7fc8b806a730 2024-04-06T12:28:23.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.642+0000 7fc8bd5e6700 1 -- 172.21.15.189:0/2556018765 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b8067bb0 con 0x7fc8b806a730 2024-04-06T12:28:23.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.642+0000 7fc89f7fe700 1 -- 172.21.15.189:0/2556018765 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 47) v1 ==== 89248+0+0 (secure 0 0 0) 0x7fc8b00045d0 con 0x7fc8b806a730 2024-04-06T12:28:23.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.643+0000 7fc89f7fe700 1 --2- 172.21.15.189:0/2556018765 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc8a406adb0 0x7fc8a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:23.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.643+0000 7fc8b67fc700 1 -- 172.21.15.189:0/2556018765 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc8a406adb0 msgr2=0x7fc8a406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:23.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.643+0000 7fc89f7fe700 1 -- 172.21.15.189:0/2556018765 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fc8b008ac30 con 0x7fc8b806a730 2024-04-06T12:28:23.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.643+0000 7fc8b67fc700 1 --2- 172.21.15.189:0/2556018765 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc8a406adb0 0x7fc8a406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:23.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.651+0000 7fc89f7fe700 1 -- 172.21.15.189:0/2556018765 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc8b0055a10 con 0x7fc8b806a730 2024-04-06T12:28:23.703 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:23.705+0000 7ffb7e96d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:28:23.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.844+0000 7fc8b67fc700 1 -- 172.21.15.189:0/2556018765 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc8a406adb0 msgr2=0x7fc8a406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:23.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.844+0000 7fc8b67fc700 1 --2- 172.21.15.189:0/2556018765 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc8a406adb0 0x7fc8a406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:23.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.898+0000 7fc8bd5e6700 1 -- 172.21.15.189:0/2556018765 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8b815e200 con 0x7fc8b806a730 2024-04-06T12:28:23.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.899+0000 7fc89f7fe700 1 -- 172.21.15.189:0/2556018765 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4745 (secure 0 0 0) 0x7fc8b0004880 con 0x7fc8b806a730 2024-04-06T12:28:23.897 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:23.897 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":39,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3065963154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3065963154},{"type":"v1","addr":"172.21.15.192:6833","nonce":3065963154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6071,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/831568588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":831568588},{"type":"v1","addr":"172.21.15.192:6835","nonce":831568588}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":7080,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1688936113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1688936113},{"type":"v1","addr":"172.21.15.189:6836","nonce":1688936113}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:28:05.673337+0000","modified":"2024-04-06T12:28:08.681505+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7083},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7083":{"gid":7083,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":4,"addr":"172.21.15.189:6837/3778396425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3778396425},{"type":"v1","addr":"172.21.15.189:6837","nonce":3778396425}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-04-06T12:28:23.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.901+0000 7fc8bd5e6700 1 -- 172.21.15.189:0/2556018765 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc8a406adb0 msgr2=0x7fc8a406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:23.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.901+0000 7fc8bd5e6700 1 --2- 172.21.15.189:0/2556018765 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc8a406adb0 0x7fc8a406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.901+0000 7fc8bd5e6700 1 -- 172.21.15.189:0/2556018765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8b806a730 msgr2=0x7fc8b81492d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:23.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.901+0000 7fc8bd5e6700 1 --2- 172.21.15.189:0/2556018765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8b806a730 0x7fc8b81492d0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fc8b806c820 tx=0x7fc8b0004ad0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.901+0000 7fc8bd5e6700 1 -- 172.21.15.189:0/2556018765 shutdown_connections 2024-04-06T12:28:23.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.901+0000 7fc8bd5e6700 1 --2- 172.21.15.189:0/2556018765 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b815ddf0 0x7fc8b8146a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.901+0000 7fc8bd5e6700 1 --2- 172.21.15.189:0/2556018765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8b806a730 0x7fc8b81492d0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.901+0000 7fc8bd5e6700 1 --2- 172.21.15.189:0/2556018765 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc8a406adb0 0x7fc8a406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.901+0000 7fc8bd5e6700 1 --2- 172.21.15.189:0/2556018765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8b815d7b0 0x7fc8b80cf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:23.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.901+0000 7fc8bd5e6700 1 -- 172.21.15.189:0/2556018765 >> 172.21.15.189:0/2556018765 conn(0x7fc8b80d00b0 msgr2=0x7fc8b815f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:23.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.901+0000 7fc8bd5e6700 1 -- 172.21.15.189:0/2556018765 shutdown_connections 2024-04-06T12:28:23.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:23.901+0000 7fc8bd5e6700 1 -- 172.21.15.189:0/2556018765 wait complete. 2024-04-06T12:28:23.901 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 39 2024-04-06T12:28:23.913 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:28:24.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.044+0000 7f1546295700 1 Processor -- start 2024-04-06T12:28:24.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.044+0000 7f1546295700 1 -- start start 2024-04-06T12:28:24.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.044+0000 7f1546295700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f154015d590 0x7f15400b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:24.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.044+0000 7f1546295700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15400b4ed0 0x7f15400b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.044+0000 7f1546295700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15400b5cc0 con 0x7f15400b6580 2024-04-06T12:28:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.044+0000 7f1546295700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15400b5890 con 0x7f154015d590 2024-04-06T12:28:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.044+0000 7f1546295700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15400d1500 con 0x7f15400b4ed0 2024-04-06T12:28:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.045+0000 7f153ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f154015d590 0x7f15400b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.045+0000 7f1544832700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f15400b6580 0x7f15400b6a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33646/0 (socket says 172.21.15.189:33646) 2024-04-06T12:28:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.045+0000 7f1544832700 1 -- 172.21.15.189:0/4004129872 learned_addr learned my addr 172.21.15.189:0/4004129872 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.045+0000 7f153f7fe700 1 --2- 172.21.15.189:0/4004129872 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15400b4ed0 0x7f15400b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.045+0000 7f153ffff700 1 -- 172.21.15.189:0/4004129872 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15400b4ed0 msgr2=0x7f15400b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.045+0000 7f153ffff700 1 --2- 172.21.15.189:0/4004129872 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15400b4ed0 0x7f15400b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.045+0000 7f153ffff700 1 -- 172.21.15.189:0/4004129872 >> v1:172.21.15.189:6789/0 conn(0x7f15400b6580 legacy=0x7f15400b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.045+0000 7f153ffff700 1 -- 172.21.15.189:0/4004129872 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15400d17e0 con 0x7f154015d590 2024-04-06T12:28:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.045+0000 7f153effd700 1 -- 172.21.15.189:0/4004129872 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728307898 0 0) 0x7f15400b5cc0 con 0x7f15400b6580 2024-04-06T12:28:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.045+0000 7f153ffff700 1 --2- 172.21.15.189:0/4004129872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f154015d590 0x7f15400b8d70 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f1530009870 tx=0x7f1530009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2bbcb7d236d83d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.045+0000 7f153effd700 1 -- 172.21.15.189:0/4004129872 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1530011830 con 0x7f154015d590 2024-04-06T12:28:24.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.045+0000 7f153effd700 1 -- 172.21.15.189:0/4004129872 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1530011990 con 0x7f154015d590 2024-04-06T12:28:24.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.045+0000 7f153effd700 1 -- 172.21.15.189:0/4004129872 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1530011d20 con 0x7f154015d590 2024-04-06T12:28:24.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.046+0000 7f1546295700 1 -- 172.21.15.189:0/4004129872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f154015d590 msgr2=0x7f15400b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:24.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.046+0000 7f1546295700 1 --2- 172.21.15.189:0/4004129872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f154015d590 0x7f15400b8d70 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f1530009870 tx=0x7f1530009b80 comp rx=0 tx=0).stop 2024-04-06T12:28:24.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.046+0000 7f1546295700 1 -- 172.21.15.189:0/4004129872 shutdown_connections 2024-04-06T12:28:24.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.046+0000 7f1546295700 1 --2- 172.21.15.189:0/4004129872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f154015d590 0x7f15400b8d70 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.046+0000 7f1546295700 1 --2- 172.21.15.189:0/4004129872 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15400b4ed0 0x7f15400b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.046+0000 7f1546295700 1 -- 172.21.15.189:0/4004129872 >> 172.21.15.189:0/4004129872 conn(0x7f154006bbc0 msgr2=0x7f154006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:24.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.046+0000 7f1546295700 1 -- 172.21.15.189:0/4004129872 shutdown_connections 2024-04-06T12:28:24.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.046+0000 7f1546295700 1 -- 172.21.15.189:0/4004129872 wait complete. 2024-04-06T12:28:24.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.047+0000 7f1546295700 1 Processor -- start 2024-04-06T12:28:24.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.048+0000 7f1546295700 1 -- start start 2024-04-06T12:28:24.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.048+0000 7f1546295700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15400b4ed0 0x7f154015c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:24.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.048+0000 7f1546295700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15400b6580 0x7f154014a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:24.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.048+0000 7f1546295700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f154015d590 0x7f154014a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:24.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.048+0000 7f1546295700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154015e770 con 0x7f15400b6580 2024-04-06T12:28:24.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.048+0000 7f1546295700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154015eb90 con 0x7f154015d590 2024-04-06T12:28:24.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.048+0000 7f1546295700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154015daf0 con 0x7f15400b4ed0 2024-04-06T12:28:24.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.048+0000 7f153f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15400b6580 0x7f154014a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:24.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.048+0000 7f153f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15400b6580 0x7f154014a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59120/0 (socket says 172.21.15.189:59120) 2024-04-06T12:28:24.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.048+0000 7f153f7fe700 1 -- 172.21.15.189:0/1173988608 learned_addr learned my addr 172.21.15.189:0/1173988608 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:24.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.048+0000 7f153ffff700 1 --2- 172.21.15.189:0/1173988608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15400b4ed0 0x7f154015c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:24.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.048+0000 7f1544832700 1 --2- 172.21.15.189:0/1173988608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f154015d590 0x7f154014a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:24.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.049+0000 7f153f7fe700 1 -- 172.21.15.189:0/1173988608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15400b4ed0 msgr2=0x7f154015c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:24.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.049+0000 7f153f7fe700 1 --2- 172.21.15.189:0/1173988608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15400b4ed0 0x7f154015c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.049+0000 7f153f7fe700 1 -- 172.21.15.189:0/1173988608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f154015d590 msgr2=0x7f154014a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:24.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.049+0000 7f153f7fe700 1 --2- 172.21.15.189:0/1173988608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f154015d590 0x7f154014a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.049+0000 7f153f7fe700 1 -- 172.21.15.189:0/1173988608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15300094a0 con 0x7f15400b6580 2024-04-06T12:28:24.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.049+0000 7f153f7fe700 1 --2- 172.21.15.189:0/1173988608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15400b6580 0x7f154014a430 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f153400e9b0 tx=0x7f153400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:24.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.049+0000 7f153cff9700 1 -- 172.21.15.189:0/1173988608 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1534009710 con 0x7f15400b6580 2024-04-06T12:28:24.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.049+0000 7f153cff9700 1 -- 172.21.15.189:0/1173988608 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153400ee50 con 0x7f15400b6580 2024-04-06T12:28:24.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.049+0000 7f1546295700 1 -- 172.21.15.189:0/1173988608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f154015ddd0 con 0x7f15400b6580 2024-04-06T12:28:24.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.049+0000 7f153cff9700 1 -- 172.21.15.189:0/1173988608 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153400ca50 con 0x7f15400b6580 2024-04-06T12:28:24.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.049+0000 7f1546295700 1 -- 172.21.15.189:0/1173988608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f154015e390 con 0x7f15400b6580 2024-04-06T12:28:24.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.050+0000 7f1546295700 1 -- 172.21.15.189:0/1173988608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1540074990 con 0x7f15400b6580 2024-04-06T12:28:24.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.051+0000 7f153cff9700 1 -- 172.21.15.189:0/1173988608 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 47) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f153400cbb0 con 0x7f15400b6580 2024-04-06T12:28:24.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.052+0000 7f153cff9700 1 --2- 172.21.15.189:0/1173988608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f152c06adb0 0x7f152c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:24.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.052+0000 7f153ffff700 1 -- 172.21.15.189:0/1173988608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f152c06adb0 msgr2=0x7f152c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:24.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.052+0000 7f153cff9700 1 -- 172.21.15.189:0/1173988608 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f1534089660 con 0x7f15400b6580 2024-04-06T12:28:24.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.052+0000 7f153ffff700 1 --2- 172.21.15.189:0/1173988608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f152c06adb0 0x7f152c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:24.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.056+0000 7f153cff9700 1 -- 172.21.15.189:0/1173988608 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1534054520 con 0x7f15400b6580 2024-04-06T12:28:24.179 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:24.181+0000 7ffa50309200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:28:24.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.253+0000 7f153ffff700 1 -- 172.21.15.189:0/1173988608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f152c06adb0 msgr2=0x7f152c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:24.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.253+0000 7f153ffff700 1 --2- 172.21.15.189:0/1173988608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f152c06adb0 0x7f152c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:24.258 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:24.260+0000 7ffa50309200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:28:24.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.264+0000 7f1546295700 1 -- 172.21.15.189:0/1173988608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15400c9900 con 0x7f15400b6580 2024-04-06T12:28:24.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.265+0000 7f153cff9700 1 -- 172.21.15.189:0/1173988608 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16855 (secure 0 0 0) 0x7f153400ce60 con 0x7f15400b6580 2024-04-06T12:28:24.263 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:24.264 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":99,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:28:19.179240+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:28:04.080500+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-06T12:28:04.619802+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/530172869":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/4174795471":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/2207020726":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/2070668132":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/1706294115":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/4285218889":"2024-04-07T12:28:19.179156+0000","172.21.15.189:6833/47128":"2024-04-07T12:28:19.179156+0000","172.21.15.189:6832/47128":"2024-04-07T12:28:19.179156+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:28:24.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.268+0000 7f1546295700 1 -- 172.21.15.189:0/1173988608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f152c06adb0 msgr2=0x7f152c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:24.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.268+0000 7f1546295700 1 --2- 172.21.15.189:0/1173988608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f152c06adb0 0x7f152c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.268+0000 7f1546295700 1 -- 172.21.15.189:0/1173988608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15400b6580 msgr2=0x7f154014a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:24.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.268+0000 7f1546295700 1 --2- 172.21.15.189:0/1173988608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15400b6580 0x7f154014a430 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f153400e9b0 tx=0x7f153400ecc0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.268+0000 7f1546295700 1 -- 172.21.15.189:0/1173988608 shutdown_connections 2024-04-06T12:28:24.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.268+0000 7f1546295700 1 --2- 172.21.15.189:0/1173988608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f152c06adb0 0x7f152c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.268+0000 7f1546295700 1 --2- 172.21.15.189:0/1173988608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f154015d590 0x7f154014a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.268+0000 7f1546295700 1 --2- 172.21.15.189:0/1173988608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15400b4ed0 0x7f154015c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.268+0000 7f1546295700 1 --2- 172.21.15.189:0/1173988608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15400b6580 0x7f154014a430 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.268+0000 7f1546295700 1 -- 172.21.15.189:0/1173988608 >> 172.21.15.189:0/1173988608 conn(0x7f154006bbc0 msgr2=0x7f1540159040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:24.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.268+0000 7f1546295700 1 -- 172.21.15.189:0/1173988608 shutdown_connections 2024-04-06T12:28:24.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.268+0000 7f1546295700 1 -- 172.21.15.189:0/1173988608 wait complete. 2024-04-06T12:28:24.279 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:28:04.080500+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:28:04.619802+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:28:24.279 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:28:24.359 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:24.361+0000 7ffb7e96d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:28:24.381 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:24.383+0000 7ffa50309200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:28:24.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.409+0000 7f442b092700 1 Processor -- start 2024-04-06T12:28:24.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.409+0000 7f442b092700 1 -- start start 2024-04-06T12:28:24.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.410+0000 7f442b092700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44240d17a0 0x7f442406b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:24.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.410+0000 7f442b092700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442406d850 con 0x7f44240cedf0 2024-04-06T12:28:24.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.410+0000 7f442b092700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442406c0a0 con 0x7f4424146dd0 2024-04-06T12:28:24.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.410+0000 7f442b092700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f442406c380 con 0x7f44240d17a0 2024-04-06T12:28:24.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.410+0000 7f442962f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f44240cedf0 0x7f44241441f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33660/0 (socket says 172.21.15.189:33660) 2024-04-06T12:28:24.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.410+0000 7f442962f700 1 -- 172.21.15.189:0/753558808 learned_addr learned my addr 172.21.15.189:0/753558808 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:24.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.410+0000 7f4428e2e700 1 --2- 172.21.15.189:0/753558808 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44240d17a0 0x7f442406b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:24.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.410+0000 7f44237fe700 1 -- 172.21.15.189:0/753558808 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249891182 0 0) 0x7f442406c0a0 con 0x7f4424146dd0 2024-04-06T12:28:24.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.410+0000 7f44237fe700 1 -- 172.21.15.189:0/753558808 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44140035b0 con 0x7f4424146dd0 2024-04-06T12:28:24.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.410+0000 7f44237fe700 1 -- 172.21.15.189:0/753558808 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655582656 0 0) 0x7f442406d850 con 0x7f44240cedf0 2024-04-06T12:28:24.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.410+0000 7f44237fe700 1 -- 172.21.15.189:0/753558808 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f442406c0a0 con 0x7f44240cedf0 2024-04-06T12:28:24.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.410+0000 7f4428e2e700 1 -- 172.21.15.189:0/753558808 >> v1:172.21.15.192:6789/0 conn(0x7f4424146dd0 legacy=0x7f442406d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:24.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.410+0000 7f4428e2e700 1 -- 172.21.15.189:0/753558808 >> v1:172.21.15.189:6789/0 conn(0x7f44240cedf0 legacy=0x7f44241441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:24.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.410+0000 7f4428e2e700 1 -- 172.21.15.189:0/753558808 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442406c660 con 0x7f44240d17a0 2024-04-06T12:28:24.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.411+0000 7f4428e2e700 1 --2- 172.21.15.189:0/753558808 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44240d17a0 0x7f442406b7a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4418009870 tx=0x7f4418009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7de1b0dce0ec5d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:24.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.411+0000 7f44237fe700 1 -- 172.21.15.189:0/753558808 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4418011830 con 0x7f44240d17a0 2024-04-06T12:28:24.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.411+0000 7f44237fe700 1 -- 172.21.15.189:0/753558808 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4418011990 con 0x7f44240d17a0 2024-04-06T12:28:24.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.411+0000 7f44237fe700 1 -- 172.21.15.189:0/753558808 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4418011d20 con 0x7f44240d17a0 2024-04-06T12:28:24.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.411+0000 7f442b092700 1 -- 172.21.15.189:0/753558808 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44240d17a0 msgr2=0x7f442406b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:24.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.411+0000 7f442b092700 1 --2- 172.21.15.189:0/753558808 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44240d17a0 0x7f442406b7a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4418009870 tx=0x7f4418009b80 comp rx=0 tx=0).stop 2024-04-06T12:28:24.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.411+0000 7f442b092700 1 -- 172.21.15.189:0/753558808 shutdown_connections 2024-04-06T12:28:24.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.411+0000 7f442b092700 1 --2- 172.21.15.189:0/753558808 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44240d17a0 0x7f442406b7a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.411+0000 7f442b092700 1 -- 172.21.15.189:0/753558808 >> 172.21.15.189:0/753558808 conn(0x7f44240b2db0 msgr2=0x7f44240b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:24.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.412+0000 7f442b092700 1 -- 172.21.15.189:0/753558808 shutdown_connections 2024-04-06T12:28:24.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.412+0000 7f442b092700 1 -- 172.21.15.189:0/753558808 wait complete. 2024-04-06T12:28:24.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.413+0000 7f442b092700 1 Processor -- start 2024-04-06T12:28:24.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.413+0000 7f442b092700 1 -- start start 2024-04-06T12:28:24.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f442b092700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44240cedf0 0x7f442415f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:24.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f442b092700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44240d17a0 0x7f44241f30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:24.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f442b092700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4424146dd0 0x7f44240c5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:24.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f442b092700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44240c6860 con 0x7f4424146dd0 2024-04-06T12:28:24.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f442b092700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44240c6b40 con 0x7f44240cedf0 2024-04-06T12:28:24.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f442b092700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44240aca70 con 0x7f44240d17a0 2024-04-06T12:28:24.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f442962f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4424146dd0 0x7f44240c5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:24.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f4428e2e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44240cedf0 0x7f442415f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:24.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f4423fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44240d17a0 0x7f44241f30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:24.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f442962f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4424146dd0 0x7f44240c5da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59130/0 (socket says 172.21.15.189:59130) 2024-04-06T12:28:24.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f4428e2e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44240cedf0 0x7f442415f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:54606/0 (socket says 172.21.15.189:54606) 2024-04-06T12:28:24.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f442962f700 1 -- 172.21.15.189:0/3388010553 learned_addr learned my addr 172.21.15.189:0/3388010553 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:24.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f442962f700 1 -- 172.21.15.189:0/3388010553 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44240d17a0 msgr2=0x7f44241f30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:24.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f442962f700 1 --2- 172.21.15.189:0/3388010553 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44240d17a0 0x7f44241f30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f442962f700 1 -- 172.21.15.189:0/3388010553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44240cedf0 msgr2=0x7f442415f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:24.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f442962f700 1 --2- 172.21.15.189:0/3388010553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44240cedf0 0x7f442415f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f442962f700 1 -- 172.21.15.189:0/3388010553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44180094a0 con 0x7f4424146dd0 2024-04-06T12:28:24.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f442962f700 1 --2- 172.21.15.189:0/3388010553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4424146dd0 0x7f44240c5da0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f441c00caa0 tx=0x7f441c00cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:24.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f44217fa700 1 -- 172.21.15.189:0/3388010553 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441c00a9e0 con 0x7f4424146dd0 2024-04-06T12:28:24.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f44217fa700 1 -- 172.21.15.189:0/3388010553 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f441c00ab40 con 0x7f4424146dd0 2024-04-06T12:28:24.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.414+0000 7f442b092700 1 -- 172.21.15.189:0/3388010553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44240acd50 con 0x7f4424146dd0 2024-04-06T12:28:24.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.415+0000 7f44217fa700 1 -- 172.21.15.189:0/3388010553 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441c012400 con 0x7f4424146dd0 2024-04-06T12:28:24.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.415+0000 7f442b092700 1 -- 172.21.15.189:0/3388010553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f442415f990 con 0x7f4424146dd0 2024-04-06T12:28:24.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.416+0000 7f442b092700 1 -- 172.21.15.189:0/3388010553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44240ac8d0 con 0x7f4424146dd0 2024-04-06T12:28:24.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.416+0000 7f44217fa700 1 -- 172.21.15.189:0/3388010553 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 47) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f441c01b3b0 con 0x7f4424146dd0 2024-04-06T12:28:24.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.417+0000 7f44217fa700 1 --2- 172.21.15.189:0/3388010553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f440806adb0 0x7f440806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:24.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.417+0000 7f4428e2e700 1 -- 172.21.15.189:0/3388010553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f440806adb0 msgr2=0x7f440806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:24.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.417+0000 7f44217fa700 1 -- 172.21.15.189:0/3388010553 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f441c0893b0 con 0x7f4424146dd0 2024-04-06T12:28:24.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.417+0000 7f4428e2e700 1 --2- 172.21.15.189:0/3388010553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f440806adb0 0x7f440806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:24.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.422+0000 7f44217fa700 1 -- 172.21.15.189:0/3388010553 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f44240ac8d0 con 0x7f4424146dd0 2024-04-06T12:28:24.443 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:24.445+0000 7ffb7e96d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:28:24.573 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:24.575+0000 7ffb7e96d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:28:24.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.618+0000 7f4428e2e700 1 -- 172.21.15.189:0/3388010553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f440806adb0 msgr2=0x7f440806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:24.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.618+0000 7f4428e2e700 1 --2- 172.21.15.189:0/3388010553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f440806adb0 0x7f440806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:24.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.624+0000 7f442b092700 1 -- 172.21.15.189:0/3388010553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f44240ac8d0 con 0x7f4424146dd0 2024-04-06T12:28:24.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.625+0000 7f44217fa700 1 -- 172.21.15.189:0/3388010553 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7f44240ac8d0 con 0x7f4424146dd0 2024-04-06T12:28:24.623 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:28:24.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.627+0000 7f442b092700 1 -- 172.21.15.189:0/3388010553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f440806adb0 msgr2=0x7f440806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:24.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.627+0000 7f442b092700 1 --2- 172.21.15.189:0/3388010553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f440806adb0 0x7f440806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.627+0000 7f442b092700 1 -- 172.21.15.189:0/3388010553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4424146dd0 msgr2=0x7f44240c5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:24.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.627+0000 7f442b092700 1 --2- 172.21.15.189:0/3388010553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4424146dd0 0x7f44240c5da0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f441c00caa0 tx=0x7f441c00cdb0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.627+0000 7f442b092700 1 -- 172.21.15.189:0/3388010553 shutdown_connections 2024-04-06T12:28:24.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.627+0000 7f442b092700 1 --2- 172.21.15.189:0/3388010553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4424146dd0 0x7f44240c5da0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.627+0000 7f442b092700 1 --2- 172.21.15.189:0/3388010553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f440806adb0 0x7f440806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.627+0000 7f442b092700 1 --2- 172.21.15.189:0/3388010553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44240cedf0 0x7f442415f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.627+0000 7f442b092700 1 --2- 172.21.15.189:0/3388010553 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44240d17a0 0x7f44241f30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.627+0000 7f442b092700 1 -- 172.21.15.189:0/3388010553 >> 172.21.15.189:0/3388010553 conn(0x7f44240b2db0 msgr2=0x7f442406d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:24.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.627+0000 7f442b092700 1 -- 172.21.15.189:0/3388010553 shutdown_connections 2024-04-06T12:28:24.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.627+0000 7f442b092700 1 -- 172.21.15.189:0/3388010553 wait complete. 2024-04-06T12:28:24.639 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:28:24.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.773+0000 7fa3ad94c700 1 Processor -- start 2024-04-06T12:28:24.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.773+0000 7fa3ad94c700 1 -- start start 2024-04-06T12:28:24.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.773+0000 7fa3ad94c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa3a80d11b0 0x7fa3a80d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:24.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.774+0000 7fa3ad94c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a80d1ad0 con 0x7fa3a80cfa00 2024-04-06T12:28:24.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.774+0000 7fa3ad94c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a80cbd80 con 0x7fa3a80cb870 2024-04-06T12:28:24.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.774+0000 7fa3ad94c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a80ccdc0 con 0x7fa3a80d11b0 2024-04-06T12:28:24.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.774+0000 7fa3a77fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa3a80cfa00 0x7fa3a80cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33670/0 (socket says 172.21.15.189:33670) 2024-04-06T12:28:24.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.774+0000 7fa3a77fe700 1 -- 172.21.15.189:0/3531694143 learned_addr learned my addr 172.21.15.189:0/3531694143 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:24.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.774+0000 7fa3a6ffd700 1 --2- 172.21.15.189:0/3531694143 >> v2:172.21.15.192:3300/0 conn(0x7fa3a80d11b0 0x7fa3a80d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:24.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.774+0000 7fa3a5ffb700 1 -- 172.21.15.189:0/3531694143 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208239479 0 0) 0x7fa3a80d1ad0 con 0x7fa3a80cfa00 2024-04-06T12:28:24.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.774+0000 7fa3a5ffb700 1 -- 172.21.15.189:0/3531694143 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3900035b0 con 0x7fa3a80cfa00 2024-04-06T12:28:24.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.774+0000 7fa3a6ffd700 1 -- 172.21.15.189:0/3531694143 >> v1:172.21.15.192:6789/0 conn(0x7fa3a80cb870 legacy=0x7fa3a815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:24.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.774+0000 7fa3a6ffd700 1 -- 172.21.15.189:0/3531694143 >> v1:172.21.15.189:6789/0 conn(0x7fa3a80cfa00 legacy=0x7fa3a80cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:24.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.774+0000 7fa3a6ffd700 1 -- 172.21.15.189:0/3531694143 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a80cef20 con 0x7fa3a80d11b0 2024-04-06T12:28:24.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.774+0000 7fa3a6ffd700 1 --2- 172.21.15.189:0/3531694143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3a80d11b0 0x7fa3a80d1590 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa398009840 tx=0x7fa398009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f48a1e5b5d9ed6a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:24.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.774+0000 7fa3a5ffb700 1 -- 172.21.15.189:0/3531694143 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa398011830 con 0x7fa3a80d11b0 2024-04-06T12:28:24.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.775+0000 7fa3a5ffb700 1 -- 172.21.15.189:0/3531694143 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa398011990 con 0x7fa3a80d11b0 2024-04-06T12:28:24.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.775+0000 7fa3a5ffb700 1 -- 172.21.15.189:0/3531694143 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa398011d20 con 0x7fa3a80d11b0 2024-04-06T12:28:24.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.775+0000 7fa3ad94c700 1 -- 172.21.15.189:0/3531694143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3a80d11b0 msgr2=0x7fa3a80d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:24.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.775+0000 7fa3ad94c700 1 --2- 172.21.15.189:0/3531694143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3a80d11b0 0x7fa3a80d1590 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa398009840 tx=0x7fa398009b50 comp rx=0 tx=0).stop 2024-04-06T12:28:24.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.775+0000 7fa3ad94c700 1 -- 172.21.15.189:0/3531694143 shutdown_connections 2024-04-06T12:28:24.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.775+0000 7fa3ad94c700 1 --2- 172.21.15.189:0/3531694143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3a80d11b0 0x7fa3a80d1590 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.775+0000 7fa3ad94c700 1 -- 172.21.15.189:0/3531694143 >> 172.21.15.189:0/3531694143 conn(0x7fa3a80b2db0 msgr2=0x7fa3a80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:24.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.775+0000 7fa3ad94c700 1 -- 172.21.15.189:0/3531694143 shutdown_connections 2024-04-06T12:28:24.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.775+0000 7fa3ad94c700 1 -- 172.21.15.189:0/3531694143 wait complete. 2024-04-06T12:28:24.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.776+0000 7fa3ad94c700 1 Processor -- start 2024-04-06T12:28:24.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.777+0000 7fa3ad94c700 1 -- start start 2024-04-06T12:28:24.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.777+0000 7fa3ad94c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3a80cb870 0x7fa3a815a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:24.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.777+0000 7fa3ad94c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3a80cfa00 0x7fa3a8158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:24.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.777+0000 7fa3ad94c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3a80d11b0 0x7fa3a815eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:24.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.777+0000 7fa3ad94c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a8159dd0 con 0x7fa3a80d11b0 2024-04-06T12:28:24.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.777+0000 7fa3ad94c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a81557f0 con 0x7fa3a80cfa00 2024-04-06T12:28:24.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.777+0000 7fa3ad94c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a8159150 con 0x7fa3a80cb870 2024-04-06T12:28:24.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.778+0000 7fa3a77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3a80d11b0 0x7fa3a815eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:24.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.778+0000 7fa3a67fc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3a80cfa00 0x7fa3a8158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:24.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.778+0000 7fa3a77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3a80d11b0 0x7fa3a815eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59146/0 (socket says 172.21.15.189:59146) 2024-04-06T12:28:24.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.778+0000 7fa3a6ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3a80cb870 0x7fa3a815a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:24.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.778+0000 7fa3a67fc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3a80cfa00 0x7fa3a8158900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:54622/0 (socket says 172.21.15.189:54622) 2024-04-06T12:28:24.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.778+0000 7fa3a67fc700 1 -- 172.21.15.189:0/3851308919 learned_addr learned my addr 172.21.15.189:0/3851308919 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:24.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.778+0000 7fa3a67fc700 1 -- 172.21.15.189:0/3851308919 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3a80cb870 msgr2=0x7fa3a815a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:24.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.778+0000 7fa3a67fc700 1 --2- 172.21.15.189:0/3851308919 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3a80cb870 0x7fa3a815a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.778+0000 7fa3a67fc700 1 -- 172.21.15.189:0/3851308919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3a80d11b0 msgr2=0x7fa3a815eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:24.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.778+0000 7fa3a67fc700 1 --2- 172.21.15.189:0/3851308919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3a80d11b0 0x7fa3a815eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:24.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.778+0000 7fa3a67fc700 1 -- 172.21.15.189:0/3851308919 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3980094a0 con 0x7fa3a80cfa00 2024-04-06T12:28:24.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.778+0000 7fa3a77fe700 1 --2- 172.21.15.189:0/3851308919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3a80d11b0 0x7fa3a815eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:28:24.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.778+0000 7fa3a6ffd700 1 --2- 172.21.15.189:0/3851308919 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3a80cb870 0x7fa3a815a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:28:24.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.778+0000 7fa3a67fc700 1 --2- 172.21.15.189:0/3851308919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3a80cfa00 0x7fa3a8158900 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa39c00c4b0 tx=0x7fa39c00c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:24.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.778+0000 7fa38ffff700 1 -- 172.21.15.189:0/3851308919 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39c0052d0 con 0x7fa3a80cfa00 2024-04-06T12:28:24.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.778+0000 7fa38ffff700 1 -- 172.21.15.189:0/3851308919 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa39c005430 con 0x7fa3a80cfa00 2024-04-06T12:28:24.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.778+0000 7fa38ffff700 1 -- 172.21.15.189:0/3851308919 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39c0038c0 con 0x7fa3a80cfa00 2024-04-06T12:28:24.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.778+0000 7fa3ad94c700 1 -- 172.21.15.189:0/3851308919 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a8159430 con 0x7fa3a80cfa00 2024-04-06T12:28:24.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.778+0000 7fa3ad94c700 1 -- 172.21.15.189:0/3851308919 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a81599f0 con 0x7fa3a80cfa00 2024-04-06T12:28:24.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.779+0000 7fa3ad94c700 1 -- 172.21.15.189:0/3851308919 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a8159f10 con 0x7fa3a80cfa00 2024-04-06T12:28:24.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.780+0000 7fa38ffff700 1 -- 172.21.15.189:0/3851308919 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 47) v1 ==== 89248+0+0 (secure 0 0 0) 0x7fa39c003a20 con 0x7fa3a80cfa00 2024-04-06T12:28:24.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.780+0000 7fa38ffff700 1 --2- 172.21.15.189:0/3851308919 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa39406adb0 0x7fa39406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:24.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.780+0000 7fa3a6ffd700 1 -- 172.21.15.189:0/3851308919 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa39406adb0 msgr2=0x7fa39406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:24.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.780+0000 7fa38ffff700 1 -- 172.21.15.189:0/3851308919 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fa39c0889b0 con 0x7fa3a80cfa00 2024-04-06T12:28:24.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.780+0000 7fa3a6ffd700 1 --2- 172.21.15.189:0/3851308919 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa39406adb0 0x7fa39406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:24.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.785+0000 7fa38ffff700 1 -- 172.21.15.189:0/3851308919 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa3a8159f10 con 0x7fa3a80cfa00 2024-04-06T12:28:24.890 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:24.893+0000 7ffa50309200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:28:24.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.981+0000 7fa3a6ffd700 1 -- 172.21.15.189:0/3851308919 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa39406adb0 msgr2=0x7fa39406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:24.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:24.981+0000 7fa3a6ffd700 1 --2- 172.21.15.189:0/3851308919 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa39406adb0 0x7fa39406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:24.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.001+0000 7fa3ad94c700 1 -- 172.21.15.189:0/3851308919 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa3a8159f10 con 0x7fa3a80cfa00 2024-04-06T12:28:25.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.002+0000 7fa38ffff700 1 -- 172.21.15.189:0/3851308919 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa39c017020 con 0x7fa3a80cfa00 2024-04-06T12:28:25.000 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:28:25.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.004+0000 7fa3ad94c700 1 -- 172.21.15.189:0/3851308919 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa39406adb0 msgr2=0x7fa39406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:25.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.004+0000 7fa3ad94c700 1 --2- 172.21.15.189:0/3851308919 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa39406adb0 0x7fa39406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.004+0000 7fa3ad94c700 1 -- 172.21.15.189:0/3851308919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3a80cfa00 msgr2=0x7fa3a8158900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:25.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.004+0000 7fa3ad94c700 1 --2- 172.21.15.189:0/3851308919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3a80cfa00 0x7fa3a8158900 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa39c00c4b0 tx=0x7fa39c00c7c0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.004+0000 7fa3ad94c700 1 -- 172.21.15.189:0/3851308919 shutdown_connections 2024-04-06T12:28:25.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.004+0000 7fa3ad94c700 1 --2- 172.21.15.189:0/3851308919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3a80d11b0 0x7fa3a815eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.004+0000 7fa3ad94c700 1 --2- 172.21.15.189:0/3851308919 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3a80cb870 0x7fa3a815a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.004+0000 7fa3ad94c700 1 --2- 172.21.15.189:0/3851308919 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa39406adb0 0x7fa39406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.004+0000 7fa3ad94c700 1 --2- 172.21.15.189:0/3851308919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3a80cfa00 0x7fa3a8158900 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.004+0000 7fa3ad94c700 1 -- 172.21.15.189:0/3851308919 >> 172.21.15.189:0/3851308919 conn(0x7fa3a80b2db0 msgr2=0x7fa3a80c6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:25.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.004+0000 7fa3ad94c700 1 -- 172.21.15.189:0/3851308919 shutdown_connections 2024-04-06T12:28:25.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.004+0000 7fa3ad94c700 1 -- 172.21.15.189:0/3851308919 wait complete. 2024-04-06T12:28:25.015 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:28:25.042 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:25.045+0000 7ffa50309200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:28:25.124 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:25.126+0000 7ffb7e96d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:28:25.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.146+0000 7fec8a8b9700 1 Processor -- start 2024-04-06T12:28:25.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.146+0000 7fec8a8b9700 1 -- start start 2024-04-06T12:28:25.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.147+0000 7fec8a8b9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fec840b4af0 0x7fec840b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:25.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.147+0000 7fec8a8b9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec840cdbe0 con 0x7fec840b4660 2024-04-06T12:28:25.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.147+0000 7fec8a8b9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8415a940 con 0x7fec8415a550 2024-04-06T12:28:25.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.147+0000 7fec8a8b9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec8415ac00 con 0x7fec840b4af0 2024-04-06T12:28:25.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.147+0000 7fec837fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fec840b4660 0x7fec840b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33672/0 (socket says 172.21.15.189:33672) 2024-04-06T12:28:25.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.147+0000 7fec837fe700 1 -- 172.21.15.189:0/1158713906 learned_addr learned my addr 172.21.15.189:0/1158713906 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:25.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.148+0000 7fec88e56700 1 --2- 172.21.15.189:0/1158713906 >> v2:172.21.15.192:3300/0 conn(0x7fec840b4af0 0x7fec840b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:25.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.148+0000 7fec82ffd700 1 -- 172.21.15.189:0/1158713906 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720275230 0 0) 0x7fec840cdbe0 con 0x7fec840b4660 2024-04-06T12:28:25.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.148+0000 7fec82ffd700 1 -- 172.21.15.189:0/1158713906 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec6c0035b0 con 0x7fec840b4660 2024-04-06T12:28:25.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.148+0000 7fec82ffd700 1 -- 172.21.15.189:0/1158713906 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675686782 0 0) 0x7fec8415a940 con 0x7fec8415a550 2024-04-06T12:28:25.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.148+0000 7fec82ffd700 1 -- 172.21.15.189:0/1158713906 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec840cdbe0 con 0x7fec8415a550 2024-04-06T12:28:25.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.148+0000 7fec88e56700 1 -- 172.21.15.189:0/1158713906 >> v1:172.21.15.192:6789/0 conn(0x7fec8415a550 legacy=0x7fec840cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:25.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.148+0000 7fec88e56700 1 -- 172.21.15.189:0/1158713906 >> v1:172.21.15.189:6789/0 conn(0x7fec840b4660 legacy=0x7fec840b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:25.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.148+0000 7fec82ffd700 1 -- 172.21.15.189:0/1158713906 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4230261495 0 0) 0x7fec6c0035b0 con 0x7fec840b4660 2024-04-06T12:28:25.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.148+0000 7fec88e56700 1 -- 172.21.15.189:0/1158713906 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8415aee0 con 0x7fec840b4af0 2024-04-06T12:28:25.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.148+0000 7fec88e56700 1 --2- 172.21.15.189:0/1158713906 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec840b4af0 0x7fec840b36d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fec7c009d90 tx=0x7fec7c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bcdaa2cc788962e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:25.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.148+0000 7fec82ffd700 1 -- 172.21.15.189:0/1158713906 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c00e070 con 0x7fec840b4af0 2024-04-06T12:28:25.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.148+0000 7fec82ffd700 1 -- 172.21.15.189:0/1158713906 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7c003a20 con 0x7fec840b4af0 2024-04-06T12:28:25.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.148+0000 7fec82ffd700 1 -- 172.21.15.189:0/1158713906 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78001670 con 0x7fec840b4af0 2024-04-06T12:28:25.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.149+0000 7fec8a8b9700 1 -- 172.21.15.189:0/1158713906 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec840b4af0 msgr2=0x7fec840b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:25.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.149+0000 7fec8a8b9700 1 --2- 172.21.15.189:0/1158713906 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec840b4af0 0x7fec840b36d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fec7c009d90 tx=0x7fec7c00b330 comp rx=0 tx=0).stop 2024-04-06T12:28:25.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.149+0000 7fec8a8b9700 1 -- 172.21.15.189:0/1158713906 shutdown_connections 2024-04-06T12:28:25.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.149+0000 7fec8a8b9700 1 --2- 172.21.15.189:0/1158713906 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec840b4af0 0x7fec840b36d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.149+0000 7fec8a8b9700 1 -- 172.21.15.189:0/1158713906 >> 172.21.15.189:0/1158713906 conn(0x7fec840d00b0 msgr2=0x7fec840d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:25.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.149+0000 7fec8a8b9700 1 -- 172.21.15.189:0/1158713906 shutdown_connections 2024-04-06T12:28:25.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.149+0000 7fec8a8b9700 1 -- 172.21.15.189:0/1158713906 wait complete. 2024-04-06T12:28:25.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.150+0000 7fec8a8b9700 1 Processor -- start 2024-04-06T12:28:25.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.150+0000 7fec8a8b9700 1 -- start start 2024-04-06T12:28:25.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.151+0000 7fec8a8b9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec840b4660 0x7fec84148a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:25.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.151+0000 7fec8a8b9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec840b4af0 0x7fec841456e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:25.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.151+0000 7fec8a8b9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec841555d0 0x7fec8406a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:25.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.151+0000 7fec8a8b9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8414c340 con 0x7fec840b4660 2024-04-06T12:28:25.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.151+0000 7fec8a8b9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec84148fa0 con 0x7fec840b4af0 2024-04-06T12:28:25.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.151+0000 7fec8a8b9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec841f6400 con 0x7fec841555d0 2024-04-06T12:28:25.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.151+0000 7fec83fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec840b4660 0x7fec84148a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:25.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.151+0000 7fec83fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec840b4660 0x7fec84148a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59156/0 (socket says 172.21.15.189:59156) 2024-04-06T12:28:25.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.151+0000 7fec83fff700 1 -- 172.21.15.189:0/2559677260 learned_addr learned my addr 172.21.15.189:0/2559677260 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:25.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.151+0000 7fec837fe700 1 --2- 172.21.15.189:0/2559677260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec840b4af0 0x7fec841456e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:25.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.151+0000 7fec88e56700 1 --2- 172.21.15.189:0/2559677260 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec841555d0 0x7fec8406a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:25.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.152+0000 7fec837fe700 1 -- 172.21.15.189:0/2559677260 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec841555d0 msgr2=0x7fec8406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:25.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.152+0000 7fec837fe700 1 --2- 172.21.15.189:0/2559677260 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec841555d0 0x7fec8406a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.152+0000 7fec837fe700 1 -- 172.21.15.189:0/2559677260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec840b4660 msgr2=0x7fec84148a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:25.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.152+0000 7fec837fe700 1 --2- 172.21.15.189:0/2559677260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec840b4660 0x7fec84148a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.152+0000 7fec837fe700 1 -- 172.21.15.189:0/2559677260 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7c009a70 con 0x7fec840b4af0 2024-04-06T12:28:25.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.152+0000 7fec83fff700 1 --2- 172.21.15.189:0/2559677260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec840b4660 0x7fec84148a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:28:25.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.152+0000 7fec837fe700 1 --2- 172.21.15.189:0/2559677260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec840b4af0 0x7fec841456e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fec7800c520 tx=0x7fec7800c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:25.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.152+0000 7fec80ff9700 1 -- 172.21.15.189:0/2559677260 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78004c80 con 0x7fec840b4af0 2024-04-06T12:28:25.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.152+0000 7fec80ff9700 1 -- 172.21.15.189:0/2559677260 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec78004de0 con 0x7fec840b4af0 2024-04-06T12:28:25.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.152+0000 7fec8a8b9700 1 -- 172.21.15.189:0/2559677260 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec841f66e0 con 0x7fec840b4af0 2024-04-06T12:28:25.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.152+0000 7fec80ff9700 1 -- 172.21.15.189:0/2559677260 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78007310 con 0x7fec840b4af0 2024-04-06T12:28:25.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.152+0000 7fec8a8b9700 1 -- 172.21.15.189:0/2559677260 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec841f6ca0 con 0x7fec840b4af0 2024-04-06T12:28:25.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.153+0000 7fec8a8b9700 1 -- 172.21.15.189:0/2559677260 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec8415b480 con 0x7fec840b4af0 2024-04-06T12:28:25.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.153+0000 7fec80ff9700 1 -- 172.21.15.189:0/2559677260 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 47) v1 ==== 89248+0+0 (secure 0 0 0) 0x7fec78007470 con 0x7fec840b4af0 2024-04-06T12:28:25.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.154+0000 7fec80ff9700 1 --2- 172.21.15.189:0/2559677260 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec7006adb0 0x7fec7006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:25.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.154+0000 7fec83fff700 1 -- 172.21.15.189:0/2559677260 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec7006adb0 msgr2=0x7fec7006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:25.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.154+0000 7fec80ff9700 1 -- 172.21.15.189:0/2559677260 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fec78089390 con 0x7fec840b4af0 2024-04-06T12:28:25.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.154+0000 7fec83fff700 1 --2- 172.21.15.189:0/2559677260 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec7006adb0 0x7fec7006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:25.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.158+0000 7fec80ff9700 1 -- 172.21.15.189:0/2559677260 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fec8415b480 con 0x7fec840b4af0 2024-04-06T12:28:25.197 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:25.200+0000 7ffa50309200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:28:25.287 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:25.289+0000 7ffb7e96d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:28:25.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.355+0000 7fec83fff700 1 -- 172.21.15.189:0/2559677260 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec7006adb0 msgr2=0x7fec7006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:25.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.355+0000 7fec83fff700 1 --2- 172.21.15.189:0/2559677260 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec7006adb0 0x7fec7006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:25.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.359+0000 7fec8a8b9700 1 -- 172.21.15.189:0/2559677260 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fec8415b480 con 0x7fec840b4af0 2024-04-06T12:28:25.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.360+0000 7fec80ff9700 1 -- 172.21.15.189:0/2559677260 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) v1 ==== 100+0+11 (secure 0 0 0) 0x7fec8415b480 con 0x7fec840b4af0 2024-04-06T12:28:25.358 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:28:25.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.362+0000 7fec8a8b9700 1 -- 172.21.15.189:0/2559677260 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec7006adb0 msgr2=0x7fec7006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:25.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.362+0000 7fec8a8b9700 1 --2- 172.21.15.189:0/2559677260 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec7006adb0 0x7fec7006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.362+0000 7fec8a8b9700 1 -- 172.21.15.189:0/2559677260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec840b4af0 msgr2=0x7fec841456e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:25.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.362+0000 7fec8a8b9700 1 --2- 172.21.15.189:0/2559677260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec840b4af0 0x7fec841456e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fec7800c520 tx=0x7fec7800c830 comp rx=0 tx=0).stop 2024-04-06T12:28:25.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.362+0000 7fec8a8b9700 1 -- 172.21.15.189:0/2559677260 shutdown_connections 2024-04-06T12:28:25.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.362+0000 7fec8a8b9700 1 --2- 172.21.15.189:0/2559677260 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec841555d0 0x7fec8406a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.362+0000 7fec8a8b9700 1 --2- 172.21.15.189:0/2559677260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec840b4660 0x7fec84148a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.362+0000 7fec8a8b9700 1 --2- 172.21.15.189:0/2559677260 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec7006adb0 0x7fec7006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.362+0000 7fec8a8b9700 1 --2- 172.21.15.189:0/2559677260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec840b4af0 0x7fec841456e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.362+0000 7fec8a8b9700 1 -- 172.21.15.189:0/2559677260 >> 172.21.15.189:0/2559677260 conn(0x7fec840d00b0 msgr2=0x7fec840739d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:25.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.362+0000 7fec8a8b9700 1 -- 172.21.15.189:0/2559677260 shutdown_connections 2024-04-06T12:28:25.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.362+0000 7fec8a8b9700 1 -- 172.21.15.189:0/2559677260 wait complete. 2024-04-06T12:28:25.373 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:28:25.453 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:25.455+0000 7ffb7e96d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:28:25.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.501+0000 7f227f3f0700 1 Processor -- start 2024-04-06T12:28:25.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.502+0000 7f227f3f0700 1 -- start start 2024-04-06T12:28:25.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.502+0000 7f227f3f0700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f22780d17a0 0x7f227806b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:25.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.502+0000 7f227f3f0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f227806d850 con 0x7f2278146dd0 2024-04-06T12:28:25.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.502+0000 7f227f3f0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f227806c0a0 con 0x7f22780cedf0 2024-04-06T12:28:25.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.502+0000 7f227f3f0700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f227806c380 con 0x7f22780d17a0 2024-04-06T12:28:25.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.504+0000 7f227c98b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2278146dd0 0x7f227806d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33686/0 (socket says 172.21.15.189:33686) 2024-04-06T12:28:25.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.504+0000 7f227c98b700 1 -- 172.21.15.189:0/164864612 learned_addr learned my addr 172.21.15.189:0/164864612 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:25.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.504+0000 7f227d18c700 1 --2- 172.21.15.189:0/164864612 >> v2:172.21.15.192:3300/0 conn(0x7f22780d17a0 0x7f227806b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:25.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.504+0000 7f226ffff700 1 -- 172.21.15.189:0/164864612 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649762885 0 0) 0x7f227806d850 con 0x7f2278146dd0 2024-04-06T12:28:25.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.504+0000 7f226ffff700 1 -- 172.21.15.189:0/164864612 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22640035b0 con 0x7f2278146dd0 2024-04-06T12:28:25.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.504+0000 7f227d18c700 1 -- 172.21.15.189:0/164864612 >> v1:172.21.15.192:6789/0 conn(0x7f22780cedf0 legacy=0x7f22780cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:25.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.504+0000 7f227d18c700 1 -- 172.21.15.189:0/164864612 >> v1:172.21.15.189:6789/0 conn(0x7f2278146dd0 legacy=0x7f227806d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:25.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.504+0000 7f227d18c700 1 -- 172.21.15.189:0/164864612 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227806c660 con 0x7f22780d17a0 2024-04-06T12:28:25.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.504+0000 7f227d18c700 1 --2- 172.21.15.189:0/164864612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22780d17a0 0x7f227806b7a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2274009840 tx=0x7f2274009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4135748134158a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:25.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.504+0000 7f226ffff700 1 -- 172.21.15.189:0/164864612 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2274011830 con 0x7f22780d17a0 2024-04-06T12:28:25.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.504+0000 7f226ffff700 1 -- 172.21.15.189:0/164864612 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2274011990 con 0x7f22780d17a0 2024-04-06T12:28:25.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.504+0000 7f226ffff700 1 -- 172.21.15.189:0/164864612 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2274011d00 con 0x7f22780d17a0 2024-04-06T12:28:25.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.505+0000 7f227f3f0700 1 -- 172.21.15.189:0/164864612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22780d17a0 msgr2=0x7f227806b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:25.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.505+0000 7f227f3f0700 1 --2- 172.21.15.189:0/164864612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22780d17a0 0x7f227806b7a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2274009840 tx=0x7f2274009b50 comp rx=0 tx=0).stop 2024-04-06T12:28:25.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.505+0000 7f227f3f0700 1 -- 172.21.15.189:0/164864612 shutdown_connections 2024-04-06T12:28:25.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.505+0000 7f227f3f0700 1 --2- 172.21.15.189:0/164864612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22780d17a0 0x7f227806b7a0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.505+0000 7f227f3f0700 1 -- 172.21.15.189:0/164864612 >> 172.21.15.189:0/164864612 conn(0x7f22780b2db0 msgr2=0x7f22780b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:25.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.505+0000 7f227f3f0700 1 -- 172.21.15.189:0/164864612 shutdown_connections 2024-04-06T12:28:25.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.505+0000 7f227f3f0700 1 -- 172.21.15.189:0/164864612 wait complete. 2024-04-06T12:28:25.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.506+0000 7f227f3f0700 1 Processor -- start 2024-04-06T12:28:25.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.506+0000 7f227f3f0700 1 -- start start 2024-04-06T12:28:25.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.507+0000 7f227f3f0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22780cedf0 0x7f227815a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:25.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.507+0000 7f227f3f0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22780d17a0 0x7f22780b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:25.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.507+0000 7f227f3f0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2278146dd0 0x7f22780b4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:25.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.507+0000 7f227f3f0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22780b71c0 con 0x7f2278146dd0 2024-04-06T12:28:25.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.507+0000 7f227f3f0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227815e6a0 con 0x7f22780cedf0 2024-04-06T12:28:25.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.507+0000 7f227f3f0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f227815e940 con 0x7f22780d17a0 2024-04-06T12:28:25.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.507+0000 7f227d98d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2278146dd0 0x7f22780b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:25.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.507+0000 7f227d18c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22780cedf0 0x7f227815a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:25.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.507+0000 7f227d98d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2278146dd0 0x7f22780b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59170/0 (socket says 172.21.15.189:59170) 2024-04-06T12:28:25.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.507+0000 7f227d98d700 1 -- 172.21.15.189:0/1578802510 learned_addr learned my addr 172.21.15.189:0/1578802510 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:25.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.507+0000 7f227c98b700 1 --2- 172.21.15.189:0/1578802510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22780d17a0 0x7f22780b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:25.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.508+0000 7f227d18c700 1 -- 172.21.15.189:0/1578802510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22780d17a0 msgr2=0x7f22780b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:25.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.508+0000 7f227d18c700 1 --2- 172.21.15.189:0/1578802510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22780d17a0 0x7f22780b6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.508+0000 7f227d18c700 1 -- 172.21.15.189:0/1578802510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2278146dd0 msgr2=0x7f22780b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:25.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.508+0000 7f227d18c700 1 --2- 172.21.15.189:0/1578802510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2278146dd0 0x7f22780b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.508+0000 7f227d18c700 1 -- 172.21.15.189:0/1578802510 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22740094a0 con 0x7f22780cedf0 2024-04-06T12:28:25.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.508+0000 7f227d98d700 1 --2- 172.21.15.189:0/1578802510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2278146dd0 0x7f22780b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:28:25.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.508+0000 7f227d18c700 1 --2- 172.21.15.189:0/1578802510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22780cedf0 0x7f227815a180 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2274016000 tx=0x7f2274004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:25.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.508+0000 7f226dffb700 1 -- 172.21.15.189:0/1578802510 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2274004550 con 0x7f22780cedf0 2024-04-06T12:28:25.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.508+0000 7f226dffb700 1 -- 172.21.15.189:0/1578802510 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227401f070 con 0x7f22780cedf0 2024-04-06T12:28:25.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.508+0000 7f227f3f0700 1 -- 172.21.15.189:0/1578802510 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f227815ec20 con 0x7f22780cedf0 2024-04-06T12:28:25.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.508+0000 7f226dffb700 1 -- 172.21.15.189:0/1578802510 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22740038c0 con 0x7f22780cedf0 2024-04-06T12:28:25.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.508+0000 7f227f3f0700 1 -- 172.21.15.189:0/1578802510 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f227815f1e0 con 0x7f22780cedf0 2024-04-06T12:28:25.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.509+0000 7f227f3f0700 1 -- 172.21.15.189:0/1578802510 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2278074990 con 0x7f22780cedf0 2024-04-06T12:28:25.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.510+0000 7f226dffb700 1 -- 172.21.15.189:0/1578802510 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 47) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f2274003a20 con 0x7f22780cedf0 2024-04-06T12:28:25.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.510+0000 7f226dffb700 1 --2- 172.21.15.189:0/1578802510 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f225c06adb0 0x7f225c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:25.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.510+0000 7f227c98b700 1 -- 172.21.15.189:0/1578802510 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f225c06adb0 msgr2=0x7f225c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:25.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.510+0000 7f226dffb700 1 -- 172.21.15.189:0/1578802510 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f2274021070 con 0x7f22780cedf0 2024-04-06T12:28:25.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.510+0000 7f227c98b700 1 --2- 172.21.15.189:0/1578802510 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f225c06adb0 0x7f225c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:25.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.519+0000 7f226dffb700 1 -- 172.21.15.189:0/1578802510 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2274055d20 con 0x7f22780cedf0 2024-04-06T12:28:25.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.711+0000 7f227c98b700 1 -- 172.21.15.189:0/1578802510 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f225c06adb0 msgr2=0x7f225c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:25.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.711+0000 7f227c98b700 1 --2- 172.21.15.189:0/1578802510 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f225c06adb0 0x7f225c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:25.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.771+0000 7f227f3f0700 1 -- 172.21.15.189:0/1578802510 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22780d1e20 con 0x7f22780cedf0 2024-04-06T12:28:25.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.771+0000 7f226dffb700 1 -- 172.21.15.189:0/1578802510 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f2274015070 con 0x7f22780cedf0 2024-04-06T12:28:25.769 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:25.769 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":39,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3065963154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3065963154},{"type":"v1","addr":"172.21.15.192:6833","nonce":3065963154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6071,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/831568588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":831568588},{"type":"v1","addr":"172.21.15.192:6835","nonce":831568588}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":7080,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1688936113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1688936113},{"type":"v1","addr":"172.21.15.189:6836","nonce":1688936113}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:28:05.673337+0000","modified":"2024-04-06T12:28:08.681505+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7083},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7083":{"gid":7083,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":4,"addr":"172.21.15.189:6837/3778396425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3778396425},{"type":"v1","addr":"172.21.15.189:6837","nonce":3778396425}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-04-06T12:28:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.774+0000 7f227f3f0700 1 -- 172.21.15.189:0/1578802510 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f225c06adb0 msgr2=0x7f225c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.774+0000 7f227f3f0700 1 --2- 172.21.15.189:0/1578802510 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f225c06adb0 0x7f225c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.774+0000 7f227f3f0700 1 -- 172.21.15.189:0/1578802510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22780cedf0 msgr2=0x7f227815a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.774+0000 7f227f3f0700 1 --2- 172.21.15.189:0/1578802510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22780cedf0 0x7f227815a180 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2274016000 tx=0x7f2274004750 comp rx=0 tx=0).stop 2024-04-06T12:28:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.774+0000 7f227f3f0700 1 -- 172.21.15.189:0/1578802510 shutdown_connections 2024-04-06T12:28:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.774+0000 7f227f3f0700 1 --2- 172.21.15.189:0/1578802510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2278146dd0 0x7f22780b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.774+0000 7f227f3f0700 1 --2- 172.21.15.189:0/1578802510 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f225c06adb0 0x7f225c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.774+0000 7f227f3f0700 1 --2- 172.21.15.189:0/1578802510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22780cedf0 0x7f227815a180 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.774+0000 7f227f3f0700 1 --2- 172.21.15.189:0/1578802510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22780d17a0 0x7f22780b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.774+0000 7f227f3f0700 1 -- 172.21.15.189:0/1578802510 >> 172.21.15.189:0/1578802510 conn(0x7f22780b2db0 msgr2=0x7f22780c6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:25.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.774+0000 7f227f3f0700 1 -- 172.21.15.189:0/1578802510 shutdown_connections 2024-04-06T12:28:25.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.774+0000 7f227f3f0700 1 -- 172.21.15.189:0/1578802510 wait complete. 2024-04-06T12:28:25.773 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 39 2024-04-06T12:28:25.784 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:28:25.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.911+0000 7f4897f0d700 1 Processor -- start 2024-04-06T12:28:25.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.911+0000 7f4897f0d700 1 -- start start 2024-04-06T12:28:25.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.912+0000 7f4897f0d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f489015d590 0x7f48900b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:25.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.912+0000 7f4897f0d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48900b5cc0 con 0x7f48900b4ed0 2024-04-06T12:28:25.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.912+0000 7f4897f0d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48900d1500 con 0x7f48900d3890 2024-04-06T12:28:25.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.912+0000 7f4897f0d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48900d17e0 con 0x7f489015d590 2024-04-06T12:28:25.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.912+0000 7f4895ca9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f489015d590 0x7f48900b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:25.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.912+0000 7f4895ca9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f489015d590 0x7f48900b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59174/0 (socket says 172.21.15.189:59174) 2024-04-06T12:28:25.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.912+0000 7f4895ca9700 1 -- 172.21.15.189:0/1307192539 learned_addr learned my addr 172.21.15.189:0/1307192539 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:25.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.912+0000 7f48954a8700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f48900b4ed0 0x7f48900b6580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33696/0 (socket says 172.21.15.189:33696) 2024-04-06T12:28:25.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.913+0000 7f4895ca9700 1 -- 172.21.15.189:0/1307192539 >> v1:172.21.15.192:6789/0 conn(0x7f48900d3890 legacy=0x7f48900b6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:25.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.913+0000 7f4895ca9700 1 -- 172.21.15.189:0/1307192539 >> v1:172.21.15.189:6789/0 conn(0x7f48900b4ed0 legacy=0x7f48900b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:25.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.913+0000 7f4895ca9700 1 -- 172.21.15.189:0/1307192539 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48900d1ac0 con 0x7f489015d590 2024-04-06T12:28:25.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.913+0000 7f4894ca7700 1 -- 172.21.15.189:0/1307192539 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655265997 0 0) 0x7f48900b5cc0 con 0x7f48900b4ed0 2024-04-06T12:28:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.913+0000 7f4895ca9700 1 --2- 172.21.15.189:0/1307192539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f489015d590 0x7f48900b8d70 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f488c0098f0 tx=0x7f488c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ac5dded5dda104 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.913+0000 7f4894ca7700 1 -- 172.21.15.189:0/1307192539 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488c00b8a0 con 0x7f489015d590 2024-04-06T12:28:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.913+0000 7f4894ca7700 1 -- 172.21.15.189:0/1307192539 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488c00ba00 con 0x7f489015d590 2024-04-06T12:28:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.913+0000 7f4894ca7700 1 -- 172.21.15.189:0/1307192539 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488c00bd90 con 0x7f489015d590 2024-04-06T12:28:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.914+0000 7f4897f0d700 1 -- 172.21.15.189:0/1307192539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f489015d590 msgr2=0x7f48900b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.914+0000 7f4897f0d700 1 --2- 172.21.15.189:0/1307192539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f489015d590 0x7f48900b8d70 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f488c0098f0 tx=0x7f488c009c00 comp rx=0 tx=0).stop 2024-04-06T12:28:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.914+0000 7f4897f0d700 1 -- 172.21.15.189:0/1307192539 shutdown_connections 2024-04-06T12:28:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.914+0000 7f4897f0d700 1 --2- 172.21.15.189:0/1307192539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f489015d590 0x7f48900b8d70 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.914+0000 7f4897f0d700 1 -- 172.21.15.189:0/1307192539 >> 172.21.15.189:0/1307192539 conn(0x7f489006bbc0 msgr2=0x7f489006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.914+0000 7f4897f0d700 1 -- 172.21.15.189:0/1307192539 shutdown_connections 2024-04-06T12:28:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.914+0000 7f4897f0d700 1 -- 172.21.15.189:0/1307192539 wait complete. 2024-04-06T12:28:25.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.915+0000 7f4897f0d700 1 Processor -- start 2024-04-06T12:28:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.915+0000 7f4897f0d700 1 -- start start 2024-04-06T12:28:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.916+0000 7f4897f0d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48900b4ed0 0x7f48900cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.916+0000 7f4897f0d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48900d3890 0x7f48900cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.916+0000 7f4897f0d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f489015d590 0x7f48900cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.916+0000 7f4897f0d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4890160630 con 0x7f48900d3890 2024-04-06T12:28:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.916+0000 7f4897f0d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48900cbeb0 con 0x7f489015d590 2024-04-06T12:28:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.916+0000 7f4897f0d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48900cc190 con 0x7f48900b4ed0 2024-04-06T12:28:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.916+0000 7f48954a8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48900d3890 0x7f48900cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.916+0000 7f4895ca9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48900b4ed0 0x7f48900cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.916+0000 7f48964aa700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f489015d590 0x7f48900cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.916+0000 7f48954a8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48900d3890 0x7f48900cb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59182/0 (socket says 172.21.15.189:59182) 2024-04-06T12:28:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.916+0000 7f48954a8700 1 -- 172.21.15.189:0/3376329311 learned_addr learned my addr 172.21.15.189:0/3376329311 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.916+0000 7f48954a8700 1 -- 172.21.15.189:0/3376329311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48900b4ed0 msgr2=0x7f48900cd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.916+0000 7f48954a8700 1 --2- 172.21.15.189:0/3376329311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48900b4ed0 0x7f48900cd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.916+0000 7f48954a8700 1 -- 172.21.15.189:0/3376329311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f489015d590 msgr2=0x7f48900cb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.916+0000 7f48954a8700 1 --2- 172.21.15.189:0/3376329311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f489015d590 0x7f48900cb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.916+0000 7f48954a8700 1 -- 172.21.15.189:0/3376329311 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c009550 con 0x7f48900d3890 2024-04-06T12:28:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.916+0000 7f48954a8700 1 --2- 172.21.15.189:0/3376329311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48900d3890 0x7f48900cb2a0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f488000c520 tx=0x7f488000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.917+0000 7f48867fc700 1 -- 172.21.15.189:0/3376329311 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4880004c80 con 0x7f48900d3890 2024-04-06T12:28:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.917+0000 7f48867fc700 1 -- 172.21.15.189:0/3376329311 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4880004e00 con 0x7f48900d3890 2024-04-06T12:28:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.917+0000 7f4897f0d700 1 -- 172.21.15.189:0/3376329311 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48900cc470 con 0x7f48900d3890 2024-04-06T12:28:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.917+0000 7f48867fc700 1 -- 172.21.15.189:0/3376329311 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4880007310 con 0x7f48900d3890 2024-04-06T12:28:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.917+0000 7f4897f0d700 1 -- 172.21.15.189:0/3376329311 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48900cca30 con 0x7f48900d3890 2024-04-06T12:28:25.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.918+0000 7f4897f0d700 1 -- 172.21.15.189:0/3376329311 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48900ccc50 con 0x7f48900d3890 2024-04-06T12:28:25.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.919+0000 7f48867fc700 1 -- 172.21.15.189:0/3376329311 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 47) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f4880007470 con 0x7f48900d3890 2024-04-06T12:28:25.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.919+0000 7f48867fc700 1 --2- 172.21.15.189:0/3376329311 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f487406adb0 0x7f487406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:25.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.919+0000 7f4895ca9700 1 -- 172.21.15.189:0/3376329311 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f487406adb0 msgr2=0x7f487406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:25.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.919+0000 7f48867fc700 1 -- 172.21.15.189:0/3376329311 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f4880089810 con 0x7f48900d3890 2024-04-06T12:28:25.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.919+0000 7f4895ca9700 1 --2- 172.21.15.189:0/3376329311 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f487406adb0 0x7f487406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:25.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:25.923+0000 7f48867fc700 1 -- 172.21.15.189:0/3376329311 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f48800547b0 con 0x7f48900d3890 2024-04-06T12:28:25.924 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:25.926+0000 7ffa50309200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:28:26.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.120+0000 7f4895ca9700 1 -- 172.21.15.189:0/3376329311 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f487406adb0 msgr2=0x7f487406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:26.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.120+0000 7f4895ca9700 1 --2- 172.21.15.189:0/3376329311 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f487406adb0 0x7f487406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:26.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.126+0000 7f4897f0d700 1 -- 172.21.15.189:0/3376329311 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4890074990 con 0x7f48900d3890 2024-04-06T12:28:26.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.127+0000 7f48867fc700 1 -- 172.21.15.189:0/3376329311 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16855 (secure 0 0 0) 0x7f4880057dd0 con 0x7f48900d3890 2024-04-06T12:28:26.125 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:26.125 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":99,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:28:19.179240+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:28:04.080500+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-06T12:28:04.619802+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/530172869":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/4174795471":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/2207020726":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/2070668132":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/1706294115":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/4285218889":"2024-04-07T12:28:19.179156+0000","172.21.15.189:6833/47128":"2024-04-07T12:28:19.179156+0000","172.21.15.189:6832/47128":"2024-04-07T12:28:19.179156+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:28:26.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.131+0000 7f4897f0d700 1 -- 172.21.15.189:0/3376329311 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f487406adb0 msgr2=0x7f487406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:26.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.131+0000 7f4897f0d700 1 --2- 172.21.15.189:0/3376329311 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f487406adb0 0x7f487406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.131+0000 7f4897f0d700 1 -- 172.21.15.189:0/3376329311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48900d3890 msgr2=0x7f48900cb2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:26.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.131+0000 7f4897f0d700 1 --2- 172.21.15.189:0/3376329311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48900d3890 0x7f48900cb2a0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f488000c520 tx=0x7f488000c830 comp rx=0 tx=0).stop 2024-04-06T12:28:26.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.131+0000 7f4897f0d700 1 -- 172.21.15.189:0/3376329311 shutdown_connections 2024-04-06T12:28:26.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.131+0000 7f4897f0d700 1 --2- 172.21.15.189:0/3376329311 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f487406adb0 0x7f487406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.131+0000 7f4897f0d700 1 --2- 172.21.15.189:0/3376329311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f489015d590 0x7f48900cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.131+0000 7f4897f0d700 1 --2- 172.21.15.189:0/3376329311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48900b4ed0 0x7f48900cd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.131+0000 7f4897f0d700 1 --2- 172.21.15.189:0/3376329311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48900d3890 0x7f48900cb2a0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.131+0000 7f4897f0d700 1 -- 172.21.15.189:0/3376329311 >> 172.21.15.189:0/3376329311 conn(0x7f489006bbc0 msgr2=0x7f489006bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:26.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.131+0000 7f4897f0d700 1 -- 172.21.15.189:0/3376329311 shutdown_connections 2024-04-06T12:28:26.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.131+0000 7f4897f0d700 1 -- 172.21.15.189:0/3376329311 wait complete. 2024-04-06T12:28:26.141 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-06T12:28:26.141 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:28:26.162 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:26.165+0000 7ffa50309200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:28:26.218 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:26.221+0000 7ffb7e96d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:28:26.240 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:26.242+0000 7ffa50309200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:28:26.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.270+0000 7fc558277700 1 Processor -- start 2024-04-06T12:28:26.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.271+0000 7fc558277700 1 -- start start 2024-04-06T12:28:26.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.272+0000 7fc558277700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc55015d7e0 0x7fc5500d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:26.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.272+0000 7fc558277700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc5500d0890 0x7fc5500d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:26.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.272+0000 7fc558277700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5500cdf80 con 0x7fc5500d1af0 2024-04-06T12:28:26.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.272+0000 7fc558277700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5500d1f80 con 0x7fc55015d7e0 2024-04-06T12:28:26.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.272+0000 7fc558277700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5500d3230 con 0x7fc5500d0890 2024-04-06T12:28:26.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.272+0000 7fc556013700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc55015d7e0 0x7fc5500d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:26.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.272+0000 7fc556013700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc55015d7e0 0x7fc5500d0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59192/0 (socket says 172.21.15.189:59192) 2024-04-06T12:28:26.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.272+0000 7fc556814700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc5500d1af0 0x7fc5500b4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33710/0 (socket says 172.21.15.189:33710) 2024-04-06T12:28:26.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.272+0000 7fc556013700 1 -- 172.21.15.189:0/3644577894 learned_addr learned my addr 172.21.15.189:0/3644577894 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:26.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.272+0000 7fc555812700 1 --2- 172.21.15.189:0/3644577894 >> v2:172.21.15.192:3300/0 conn(0x7fc5500d0890 0x7fc5500d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:26.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.272+0000 7fc556013700 1 -- 172.21.15.189:0/3644577894 >> v2:172.21.15.192:3300/0 conn(0x7fc5500d0890 msgr2=0x7fc5500d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:26.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.272+0000 7fc556013700 1 --2- 172.21.15.189:0/3644577894 >> v2:172.21.15.192:3300/0 conn(0x7fc5500d0890 0x7fc5500d15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.272+0000 7fc556013700 1 -- 172.21.15.189:0/3644577894 >> v1:172.21.15.189:6789/0 conn(0x7fc5500d1af0 legacy=0x7fc5500b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:26.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.272+0000 7fc556013700 1 -- 172.21.15.189:0/3644577894 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5500d0cf0 con 0x7fc55015d7e0 2024-04-06T12:28:26.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.272+0000 7fc555011700 1 -- 172.21.15.189:0/3644577894 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1467417824 0 0) 0x7fc5500cdf80 con 0x7fc5500d1af0 2024-04-06T12:28:26.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.272+0000 7fc556013700 1 --2- 172.21.15.189:0/3644577894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55015d7e0 0x7fc5500d0350 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fc5500cfda0 tx=0x7fc54c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=200a4e2388f764ed server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:26.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.273+0000 7fc555011700 1 -- 172.21.15.189:0/3644577894 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c011830 con 0x7fc55015d7e0 2024-04-06T12:28:26.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.273+0000 7fc555011700 1 -- 172.21.15.189:0/3644577894 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54c011990 con 0x7fc55015d7e0 2024-04-06T12:28:26.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.273+0000 7fc555011700 1 -- 172.21.15.189:0/3644577894 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c011d20 con 0x7fc55015d7e0 2024-04-06T12:28:26.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.273+0000 7fc558277700 1 -- 172.21.15.189:0/3644577894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55015d7e0 msgr2=0x7fc5500d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:26.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.273+0000 7fc558277700 1 --2- 172.21.15.189:0/3644577894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55015d7e0 0x7fc5500d0350 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fc5500cfda0 tx=0x7fc54c009bd0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.273+0000 7fc558277700 1 -- 172.21.15.189:0/3644577894 shutdown_connections 2024-04-06T12:28:26.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.273+0000 7fc558277700 1 --2- 172.21.15.189:0/3644577894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55015d7e0 0x7fc5500d0350 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.273+0000 7fc558277700 1 --2- 172.21.15.189:0/3644577894 >> v2:172.21.15.192:3300/0 conn(0x7fc5500d0890 0x7fc5500d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.273+0000 7fc558277700 1 -- 172.21.15.189:0/3644577894 >> 172.21.15.189:0/3644577894 conn(0x7fc550072250 msgr2=0x7fc550072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:26.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.273+0000 7fc558277700 1 -- 172.21.15.189:0/3644577894 shutdown_connections 2024-04-06T12:28:26.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.273+0000 7fc558277700 1 -- 172.21.15.189:0/3644577894 wait complete. 2024-04-06T12:28:26.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.275+0000 7fc558277700 1 Processor -- start 2024-04-06T12:28:26.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.275+0000 7fc558277700 1 -- start start 2024-04-06T12:28:26.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.275+0000 7fc558277700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5500d0890 0x7fc5500c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:26.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.275+0000 7fc558277700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5500d1af0 0x7fc5501599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:26.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.275+0000 7fc558277700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc55015d7e0 0x7fc5500c6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:26.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.275+0000 7fc558277700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5500ca4a0 con 0x7fc5500d0890 2024-04-06T12:28:26.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.275+0000 7fc558277700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc550159f10 con 0x7fc55015d7e0 2024-04-06T12:28:26.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.275+0000 7fc558277700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5500c7080 con 0x7fc5500d1af0 2024-04-06T12:28:26.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.276+0000 7fc556013700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5500d0890 0x7fc5500c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:26.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.276+0000 7fc556013700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5500d0890 0x7fc5500c73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59208/0 (socket says 172.21.15.189:59208) 2024-04-06T12:28:26.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.276+0000 7fc556013700 1 -- 172.21.15.189:0/1227347896 learned_addr learned my addr 172.21.15.189:0/1227347896 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:26.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.276+0000 7fc555812700 1 --2- 172.21.15.189:0/1227347896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5500d1af0 0x7fc5501599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:26.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.276+0000 7fc556814700 1 --2- 172.21.15.189:0/1227347896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc55015d7e0 0x7fc5500c6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:26.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.276+0000 7fc556013700 1 -- 172.21.15.189:0/1227347896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5500d1af0 msgr2=0x7fc5501599d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:26.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.276+0000 7fc556013700 1 --2- 172.21.15.189:0/1227347896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5500d1af0 0x7fc5501599d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.276+0000 7fc556013700 1 -- 172.21.15.189:0/1227347896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc55015d7e0 msgr2=0x7fc5500c6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:26.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.276+0000 7fc556013700 1 --2- 172.21.15.189:0/1227347896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc55015d7e0 0x7fc5500c6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.276+0000 7fc556013700 1 -- 172.21.15.189:0/1227347896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54c009550 con 0x7fc5500d0890 2024-04-06T12:28:26.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.276+0000 7fc556013700 1 --2- 172.21.15.189:0/1227347896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5500d0890 0x7fc5500c73f0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009c10 tx=0x7fc54c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:26.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.276+0000 7fc546ffd700 1 -- 172.21.15.189:0/1227347896 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c004510 con 0x7fc5500d0890 2024-04-06T12:28:26.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.276+0000 7fc546ffd700 1 -- 172.21.15.189:0/1227347896 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54c0036a0 con 0x7fc5500d0890 2024-04-06T12:28:26.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.276+0000 7fc546ffd700 1 -- 172.21.15.189:0/1227347896 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c003a10 con 0x7fc5500d0890 2024-04-06T12:28:26.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.276+0000 7fc558277700 1 -- 172.21.15.189:0/1227347896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5500c7a60 con 0x7fc5500d0890 2024-04-06T12:28:26.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.276+0000 7fc558277700 1 -- 172.21.15.189:0/1227347896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5500ca9d0 con 0x7fc5500d0890 2024-04-06T12:28:26.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.277+0000 7fc558277700 1 -- 172.21.15.189:0/1227347896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc550074990 con 0x7fc5500d0890 2024-04-06T12:28:26.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.278+0000 7fc546ffd700 1 -- 172.21.15.189:0/1227347896 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 47) v1 ==== 89248+0+0 (secure 0 0 0) 0x7fc54c003b70 con 0x7fc5500d0890 2024-04-06T12:28:26.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.278+0000 7fc546ffd700 1 --2- 172.21.15.189:0/1227347896 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc53406adb0 0x7fc53406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:26.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.278+0000 7fc555812700 1 -- 172.21.15.189:0/1227347896 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc53406adb0 msgr2=0x7fc53406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:26.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.278+0000 7fc546ffd700 1 -- 172.21.15.189:0/1227347896 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fc54c021030 con 0x7fc5500d0890 2024-04-06T12:28:26.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.278+0000 7fc555812700 1 --2- 172.21.15.189:0/1227347896 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc53406adb0 0x7fc53406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:26.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.283+0000 7fc546ffd700 1 -- 172.21.15.189:0/1227347896 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc54c055be0 con 0x7fc5500d0890 2024-04-06T12:28:26.473 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:26.476+0000 7ffb7e96d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:28:26.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.479+0000 7fc555812700 1 -- 172.21.15.189:0/1227347896 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc53406adb0 msgr2=0x7fc53406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:26.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.479+0000 7fc555812700 1 --2- 172.21.15.189:0/1227347896 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc53406adb0 0x7fc53406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:26.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.529+0000 7fc558277700 1 -- 172.21.15.189:0/1227347896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc550160b40 con 0x7fc5500d0890 2024-04-06T12:28:26.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.530+0000 7fc546ffd700 1 -- 172.21.15.189:0/1227347896 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4745 (secure 0 0 0) 0x7fc54c015090 con 0x7fc5500d0890 2024-04-06T12:28:26.527 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:26.527 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":39,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3065963154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3065963154},{"type":"v1","addr":"172.21.15.192:6833","nonce":3065963154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6071,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/831568588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":831568588},{"type":"v1","addr":"172.21.15.192:6835","nonce":831568588}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":7080,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1688936113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1688936113},{"type":"v1","addr":"172.21.15.189:6836","nonce":1688936113}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:28:05.673337+0000","modified":"2024-04-06T12:28:08.681505+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7083},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7083":{"gid":7083,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":4,"addr":"172.21.15.189:6837/3778396425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3778396425},{"type":"v1","addr":"172.21.15.189:6837","nonce":3778396425}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-04-06T12:28:26.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.533+0000 7fc558277700 1 -- 172.21.15.189:0/1227347896 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc53406adb0 msgr2=0x7fc53406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:26.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.533+0000 7fc558277700 1 --2- 172.21.15.189:0/1227347896 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc53406adb0 0x7fc53406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.533+0000 7fc558277700 1 -- 172.21.15.189:0/1227347896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5500d0890 msgr2=0x7fc5500c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:26.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.533+0000 7fc558277700 1 --2- 172.21.15.189:0/1227347896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5500d0890 0x7fc5500c73f0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009c10 tx=0x7fc54c004750 comp rx=0 tx=0).stop 2024-04-06T12:28:26.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.533+0000 7fc558277700 1 -- 172.21.15.189:0/1227347896 shutdown_connections 2024-04-06T12:28:26.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.533+0000 7fc558277700 1 --2- 172.21.15.189:0/1227347896 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc53406adb0 0x7fc53406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.533+0000 7fc558277700 1 --2- 172.21.15.189:0/1227347896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc55015d7e0 0x7fc5500c6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.533+0000 7fc558277700 1 --2- 172.21.15.189:0/1227347896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5500d0890 0x7fc5500c73f0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.533+0000 7fc558277700 1 --2- 172.21.15.189:0/1227347896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5500d1af0 0x7fc5501599d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.533+0000 7fc558277700 1 -- 172.21.15.189:0/1227347896 >> 172.21.15.189:0/1227347896 conn(0x7fc550072250 msgr2=0x7fc55015ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:26.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.533+0000 7fc558277700 1 -- 172.21.15.189:0/1227347896 shutdown_connections 2024-04-06T12:28:26.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.533+0000 7fc558277700 1 -- 172.21.15.189:0/1227347896 wait complete. 2024-04-06T12:28:26.531 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 39 2024-04-06T12:28:26.543 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:28:26.556 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:26.559+0000 7ffb7e96d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:28:26.614 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:26.617+0000 7ffa50309200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:28:26.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.671+0000 7fda8bcee700 1 Processor -- start 2024-04-06T12:28:26.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.671+0000 7fda8bcee700 1 -- start start 2024-04-06T12:28:26.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.672+0000 7fda8bcee700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fda8415b130 0x7fda840b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:26.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.672+0000 7fda8bcee700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda840724f0 0x7fda840b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:26.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.672+0000 7fda8bcee700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda840b1fb0 con 0x7fda8415b510 2024-04-06T12:28:26.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.672+0000 7fda8bcee700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda840b2200 con 0x7fda8415b130 2024-04-06T12:28:26.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.672+0000 7fda8bcee700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda840b24e0 con 0x7fda840724f0 2024-04-06T12:28:26.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.672+0000 7fda89289700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fda8415b510 0x7fda84071de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33716/0 (socket says 172.21.15.189:33716) 2024-04-06T12:28:26.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.672+0000 7fda89289700 1 -- 172.21.15.189:0/2626619887 learned_addr learned my addr 172.21.15.189:0/2626619887 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:26.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.672+0000 7fda89a8a700 1 --2- 172.21.15.189:0/2626619887 >> v2:172.21.15.192:3300/0 conn(0x7fda8415b130 0x7fda840b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:26.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.672+0000 7fda8a28b700 1 --2- 172.21.15.189:0/2626619887 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda840724f0 0x7fda840b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:26.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.673+0000 7fda88a88700 1 -- 172.21.15.189:0/2626619887 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152270110 0 0) 0x7fda840b1fb0 con 0x7fda8415b510 2024-04-06T12:28:26.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.673+0000 7fda88a88700 1 -- 172.21.15.189:0/2626619887 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda700035b0 con 0x7fda8415b510 2024-04-06T12:28:26.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.673+0000 7fda89a8a700 1 -- 172.21.15.189:0/2626619887 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda840724f0 msgr2=0x7fda840b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:26.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.673+0000 7fda89a8a700 1 --2- 172.21.15.189:0/2626619887 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda840724f0 0x7fda840b39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.673+0000 7fda89a8a700 1 -- 172.21.15.189:0/2626619887 >> v1:172.21.15.189:6789/0 conn(0x7fda8415b510 legacy=0x7fda84071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:26.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.673+0000 7fda89a8a700 1 -- 172.21.15.189:0/2626619887 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda840b27c0 con 0x7fda8415b130 2024-04-06T12:28:26.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.673+0000 7fda8a28b700 1 --2- 172.21.15.189:0/2626619887 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda840724f0 0x7fda840b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:28:26.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.673+0000 7fda89a8a700 1 --2- 172.21.15.189:0/2626619887 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda8415b130 0x7fda840b3470 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fda74009840 tx=0x7fda74009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1af43f1ac6e7bb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:26.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.673+0000 7fda88a88700 1 -- 172.21.15.189:0/2626619887 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74011830 con 0x7fda8415b130 2024-04-06T12:28:26.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.673+0000 7fda88a88700 1 -- 172.21.15.189:0/2626619887 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda74011990 con 0x7fda8415b130 2024-04-06T12:28:26.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.673+0000 7fda88a88700 1 -- 172.21.15.189:0/2626619887 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74011d20 con 0x7fda8415b130 2024-04-06T12:28:26.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.674+0000 7fda8bcee700 1 -- 172.21.15.189:0/2626619887 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda8415b130 msgr2=0x7fda840b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:26.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.674+0000 7fda8bcee700 1 --2- 172.21.15.189:0/2626619887 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda8415b130 0x7fda840b3470 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fda74009840 tx=0x7fda74009b50 comp rx=0 tx=0).stop 2024-04-06T12:28:26.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.674+0000 7fda8bcee700 1 -- 172.21.15.189:0/2626619887 shutdown_connections 2024-04-06T12:28:26.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.674+0000 7fda8bcee700 1 --2- 172.21.15.189:0/2626619887 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda840724f0 0x7fda840b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.674+0000 7fda8bcee700 1 --2- 172.21.15.189:0/2626619887 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda8415b130 0x7fda840b3470 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.674+0000 7fda8bcee700 1 -- 172.21.15.189:0/2626619887 >> 172.21.15.189:0/2626619887 conn(0x7fda8406a730 msgr2=0x7fda8406ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:26.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.674+0000 7fda8bcee700 1 -- 172.21.15.189:0/2626619887 shutdown_connections 2024-04-06T12:28:26.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.674+0000 7fda8bcee700 1 -- 172.21.15.189:0/2626619887 wait complete. 2024-04-06T12:28:26.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.675+0000 7fda8bcee700 1 Processor -- start 2024-04-06T12:28:26.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.675+0000 7fda8bcee700 1 -- start start 2024-04-06T12:28:26.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.676+0000 7fda8bcee700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda840724f0 0x7fda840cf370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:26.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.676+0000 7fda8bcee700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda8415b130 0x7fda84154eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:26.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.676+0000 7fda8bcee700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda8415b510 0x7fda8412e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:26.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.676+0000 7fda8bcee700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda841f48f0 con 0x7fda8415b510 2024-04-06T12:28:26.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.676+0000 7fda8bcee700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda841f4b00 con 0x7fda840724f0 2024-04-06T12:28:26.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.676+0000 7fda8bcee700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda841f4de0 con 0x7fda8415b130 2024-04-06T12:28:26.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.676+0000 7fda8a28b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda8415b510 0x7fda8412e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:26.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.676+0000 7fda8a28b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda8415b510 0x7fda8412e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59210/0 (socket says 172.21.15.189:59210) 2024-04-06T12:28:26.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.676+0000 7fda8a28b700 1 -- 172.21.15.189:0/1575610477 learned_addr learned my addr 172.21.15.189:0/1575610477 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:26.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.676+0000 7fda89289700 1 --2- 172.21.15.189:0/1575610477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda8415b130 0x7fda84154eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:26.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.676+0000 7fda89a8a700 1 --2- 172.21.15.189:0/1575610477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda840724f0 0x7fda840cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:26.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.676+0000 7fda8a28b700 1 -- 172.21.15.189:0/1575610477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda8415b130 msgr2=0x7fda84154eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:26.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.676+0000 7fda8a28b700 1 --2- 172.21.15.189:0/1575610477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda8415b130 0x7fda84154eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.676+0000 7fda8a28b700 1 -- 172.21.15.189:0/1575610477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda840724f0 msgr2=0x7fda840cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:26.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.676+0000 7fda8a28b700 1 --2- 172.21.15.189:0/1575610477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda840724f0 0x7fda840cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.676+0000 7fda8a28b700 1 -- 172.21.15.189:0/1575610477 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7c009a70 con 0x7fda8415b510 2024-04-06T12:28:26.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.676+0000 7fda89289700 1 --2- 172.21.15.189:0/1575610477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda8415b130 0x7fda84154eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:28:26.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.676+0000 7fda8a28b700 1 --2- 172.21.15.189:0/1575610477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda8415b510 0x7fda8412e3e0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00efc0 tx=0x7fda7c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:26.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.676+0000 7fda7a7fc700 1 -- 172.21.15.189:0/1575610477 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c00ca90 con 0x7fda8415b510 2024-04-06T12:28:26.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.677+0000 7fda7a7fc700 1 -- 172.21.15.189:0/1575610477 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7c00f040 con 0x7fda8415b510 2024-04-06T12:28:26.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.677+0000 7fda7a7fc700 1 -- 172.21.15.189:0/1575610477 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c00ce10 con 0x7fda8415b510 2024-04-06T12:28:26.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.677+0000 7fda8bcee700 1 -- 172.21.15.189:0/1575610477 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda740094a0 con 0x7fda8415b510 2024-04-06T12:28:26.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.677+0000 7fda8bcee700 1 -- 172.21.15.189:0/1575610477 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda84156e70 con 0x7fda8415b510 2024-04-06T12:28:26.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.677+0000 7fda8bcee700 1 -- 172.21.15.189:0/1575610477 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda84157090 con 0x7fda8415b510 2024-04-06T12:28:26.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.678+0000 7fda7a7fc700 1 -- 172.21.15.189:0/1575610477 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 47) v1 ==== 89248+0+0 (secure 0 0 0) 0x7fda7c014480 con 0x7fda8415b510 2024-04-06T12:28:26.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.679+0000 7fda7a7fc700 1 --2- 172.21.15.189:0/1575610477 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fda6806adb0 0x7fda6806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:26.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.679+0000 7fda89a8a700 1 -- 172.21.15.189:0/1575610477 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fda6806adb0 msgr2=0x7fda6806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:26.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.679+0000 7fda7a7fc700 1 -- 172.21.15.189:0/1575610477 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fda7c089df0 con 0x7fda8415b510 2024-04-06T12:28:26.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.679+0000 7fda89a8a700 1 --2- 172.21.15.189:0/1575610477 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fda6806adb0 0x7fda6806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:26.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.683+0000 7fda7a7fc700 1 -- 172.21.15.189:0/1575610477 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fda7c054ba0 con 0x7fda8415b510 2024-04-06T12:28:26.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.880+0000 7fda89a8a700 1 -- 172.21.15.189:0/1575610477 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fda6806adb0 msgr2=0x7fda6806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:26.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.880+0000 7fda89a8a700 1 --2- 172.21.15.189:0/1575610477 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fda6806adb0 0x7fda6806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:26.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.935+0000 7fda8bcee700 1 -- 172.21.15.189:0/1575610477 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda84074990 con 0x7fda8415b510 2024-04-06T12:28:26.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.935+0000 7fda7a7fc700 1 -- 172.21.15.189:0/1575610477 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4745 (secure 0 0 0) 0x7fda7c009d70 con 0x7fda8415b510 2024-04-06T12:28:26.933 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:26.933 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":39,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3065963154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3065963154},{"type":"v1","addr":"172.21.15.192:6833","nonce":3065963154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6071,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/831568588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":831568588},{"type":"v1","addr":"172.21.15.192:6835","nonce":831568588}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":7080,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1688936113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1688936113},{"type":"v1","addr":"172.21.15.189:6836","nonce":1688936113}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:28:05.673337+0000","modified":"2024-04-06T12:28:08.681505+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7083},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7083":{"gid":7083,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":4,"addr":"172.21.15.189:6837/3778396425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3778396425},{"type":"v1","addr":"172.21.15.189:6837","nonce":3778396425}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-04-06T12:28:26.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.938+0000 7fda8bcee700 1 -- 172.21.15.189:0/1575610477 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fda6806adb0 msgr2=0x7fda6806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:26.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.938+0000 7fda8bcee700 1 --2- 172.21.15.189:0/1575610477 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fda6806adb0 0x7fda6806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.938+0000 7fda8bcee700 1 -- 172.21.15.189:0/1575610477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda8415b510 msgr2=0x7fda8412e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:26.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.938+0000 7fda8bcee700 1 --2- 172.21.15.189:0/1575610477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda8415b510 0x7fda8412e3e0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00efc0 tx=0x7fda7c00c3a0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.938+0000 7fda8bcee700 1 -- 172.21.15.189:0/1575610477 shutdown_connections 2024-04-06T12:28:26.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.938+0000 7fda8bcee700 1 --2- 172.21.15.189:0/1575610477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda8415b510 0x7fda8412e3e0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.938+0000 7fda8bcee700 1 --2- 172.21.15.189:0/1575610477 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fda6806adb0 0x7fda6806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.938+0000 7fda8bcee700 1 --2- 172.21.15.189:0/1575610477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda840724f0 0x7fda840cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.938+0000 7fda8bcee700 1 --2- 172.21.15.189:0/1575610477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda8415b130 0x7fda84154eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:26.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.938+0000 7fda8bcee700 1 -- 172.21.15.189:0/1575610477 >> 172.21.15.189:0/1575610477 conn(0x7fda8406a730 msgr2=0x7fda84159690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:26.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.938+0000 7fda8bcee700 1 -- 172.21.15.189:0/1575610477 shutdown_connections 2024-04-06T12:28:26.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:26.938+0000 7fda8bcee700 1 -- 172.21.15.189:0/1575610477 wait complete. 2024-04-06T12:28:26.936 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 39 2024-04-06T12:28:26.947 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:28:26.951 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:26.954+0000 7ffb7e96d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:28:27.035 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:27.038+0000 7ffafa08c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:27.036 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:27.038+0000 7ffafa08c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:27.036 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:27.038+0000 7ffafa08c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:27.036 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:27.038+0000 7ffafa08c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:27.036 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:27.038+0000 7ffafa08c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:27.036 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:27.039+0000 7ffaf7086700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:27.036 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:27.039+0000 7ffaf7086700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:27.036 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:27.039+0000 7ffaf7086700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:27.036 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:27.039+0000 7ffaf7086700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:27.036 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:27.039+0000 7ffaf7086700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:27.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.090+0000 7f96a0985700 1 Processor -- start 2024-04-06T12:28:27.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.090+0000 7f96a0985700 1 -- start start 2024-04-06T12:28:27.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.091+0000 7f96a0985700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c15d7b0 0x7f969c0af040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:27.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.091+0000 7f96a0985700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f969c0723f0 0x7f969c0728a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:27.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.091+0000 7f96a0985700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c071950 con 0x7f969c0b39d0 2024-04-06T12:28:27.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.091+0000 7f96a0985700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f969c0b4170 con 0x7f969c0723f0 2024-04-06T12:28:27.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.091+0000 7f96a0985700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969c0b3e90 con 0x7f969c15d7b0 2024-04-06T12:28:27.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.091+0000 7f9699d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f969c0b39d0 0x7f969c0ac8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33720/0 (socket says 172.21.15.189:33720) 2024-04-06T12:28:27.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.091+0000 7f9699d9b700 1 -- 172.21.15.189:0/2548644167 learned_addr learned my addr 172.21.15.189:0/2548644167 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:27.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.091+0000 7f969a59c700 1 --2- 172.21.15.189:0/2548644167 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c15d7b0 0x7f969c0af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:27.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.091+0000 7f969ad9d700 1 --2- 172.21.15.189:0/2548644167 >> v2:172.21.15.192:3300/0 conn(0x7f969c0723f0 0x7f969c0728a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:27.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.091+0000 7f969959a700 1 -- 172.21.15.189:0/2548644167 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842906528 0 0) 0x7f969c071950 con 0x7f969c0b39d0 2024-04-06T12:28:27.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.091+0000 7f969959a700 1 -- 172.21.15.189:0/2548644167 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96840035b0 con 0x7f969c0b39d0 2024-04-06T12:28:27.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.092+0000 7f969a59c700 1 -- 172.21.15.189:0/2548644167 >> v2:172.21.15.192:3300/0 conn(0x7f969c0723f0 msgr2=0x7f969c0728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:27.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.092+0000 7f969a59c700 1 --2- 172.21.15.189:0/2548644167 >> v2:172.21.15.192:3300/0 conn(0x7f969c0723f0 0x7f969c0728a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.092+0000 7f969a59c700 1 -- 172.21.15.189:0/2548644167 >> v1:172.21.15.189:6789/0 conn(0x7f969c0b39d0 legacy=0x7f969c0ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:27.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.092+0000 7f969959a700 1 -- 172.21.15.189:0/2548644167 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 383562930 0 0) 0x7f96840035b0 con 0x7f969c0b39d0 2024-04-06T12:28:27.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.092+0000 7f969a59c700 1 -- 172.21.15.189:0/2548644167 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c071bf0 con 0x7f969c15d7b0 2024-04-06T12:28:27.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.092+0000 7f969a59c700 1 --2- 172.21.15.189:0/2548644167 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c15d7b0 0x7f969c0af040 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9690009840 tx=0x7f9690009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c3de8ee5818841 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:27.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.092+0000 7f969959a700 1 -- 172.21.15.189:0/2548644167 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c001670 con 0x7f969c15d7b0 2024-04-06T12:28:27.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.092+0000 7f969959a700 1 -- 172.21.15.189:0/2548644167 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9690004030 con 0x7f969c15d7b0 2024-04-06T12:28:27.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.092+0000 7f969959a700 1 -- 172.21.15.189:0/2548644167 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96900043c0 con 0x7f969c15d7b0 2024-04-06T12:28:27.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.092+0000 7f96a0985700 1 -- 172.21.15.189:0/2548644167 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c15d7b0 msgr2=0x7f969c0af040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:27.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.092+0000 7f96a0985700 1 --2- 172.21.15.189:0/2548644167 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c15d7b0 0x7f969c0af040 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9690009840 tx=0x7f9690009b50 comp rx=0 tx=0).stop 2024-04-06T12:28:27.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.093+0000 7f96a0985700 1 -- 172.21.15.189:0/2548644167 shutdown_connections 2024-04-06T12:28:27.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.093+0000 7f96a0985700 1 --2- 172.21.15.189:0/2548644167 >> v2:172.21.15.192:3300/0 conn(0x7f969c0723f0 0x7f969c0728a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.093+0000 7f96a0985700 1 --2- 172.21.15.189:0/2548644167 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c15d7b0 0x7f969c0af040 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.093+0000 7f96a0985700 1 -- 172.21.15.189:0/2548644167 >> 172.21.15.189:0/2548644167 conn(0x7f969c0d00b0 msgr2=0x7f969c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:27.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.093+0000 7f96a0985700 1 -- 172.21.15.189:0/2548644167 shutdown_connections 2024-04-06T12:28:27.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.093+0000 7f96a0985700 1 -- 172.21.15.189:0/2548644167 wait complete. 2024-04-06T12:28:27.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.094+0000 7f96a0985700 1 Processor -- start 2024-04-06T12:28:27.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.094+0000 7f96a0985700 1 -- start start 2024-04-06T12:28:27.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f96a0985700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f969c0723f0 0x7f969c1539f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:27.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f96a0985700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c0b39d0 0x7f969c154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:27.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f96a0985700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f969c153250 0x7f969c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:27.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f96a0985700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c1556f0 con 0x7f969c0723f0 2024-04-06T12:28:27.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f96a0985700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c154ae0 con 0x7f969c153250 2024-04-06T12:28:27.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f96a0985700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969c154dc0 con 0x7f969c0b39d0 2024-04-06T12:28:27.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f969a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f969c0723f0 0x7f969c1539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:27.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f969a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f969c0723f0 0x7f969c1539f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59310/0 (socket says 172.21.15.189:59310) 2024-04-06T12:28:27.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f969a59c700 1 -- 172.21.15.189:0/3077118883 learned_addr learned my addr 172.21.15.189:0/3077118883 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:27.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f969ad9d700 1 --2- 172.21.15.189:0/3077118883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f969c153250 0x7f969c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:27.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f9699d9b700 1 --2- 172.21.15.189:0/3077118883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c0b39d0 0x7f969c154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:27.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f969a59c700 1 -- 172.21.15.189:0/3077118883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c0b39d0 msgr2=0x7f969c154450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:27.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f969a59c700 1 --2- 172.21.15.189:0/3077118883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c0b39d0 0x7f969c154450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f969a59c700 1 -- 172.21.15.189:0/3077118883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f969c153250 msgr2=0x7f969c0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:27.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f969a59c700 1 --2- 172.21.15.189:0/3077118883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f969c153250 0x7f969c0cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f969a59c700 1 -- 172.21.15.189:0/3077118883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96900094a0 con 0x7f969c0723f0 2024-04-06T12:28:27.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f969ad9d700 1 --2- 172.21.15.189:0/3077118883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f969c153250 0x7f969c0cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:27.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f969a59c700 1 --2- 172.21.15.189:0/3077118883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f969c0723f0 0x7f969c1539f0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f969000a010 tx=0x7f9690004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:27.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f968affd700 1 -- 172.21.15.189:0/3077118883 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9690003e10 con 0x7f969c0723f0 2024-04-06T12:28:27.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f968affd700 1 -- 172.21.15.189:0/3077118883 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969001f070 con 0x7f969c0723f0 2024-04-06T12:28:27.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f96a0985700 1 -- 172.21.15.189:0/3077118883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f969c1550a0 con 0x7f969c0723f0 2024-04-06T12:28:27.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.095+0000 7f968affd700 1 -- 172.21.15.189:0/3077118883 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9690011870 con 0x7f969c0723f0 2024-04-06T12:28:27.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.096+0000 7f96a0985700 1 -- 172.21.15.189:0/3077118883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f969c1541d0 con 0x7f969c0723f0 2024-04-06T12:28:27.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.096+0000 7f96a0985700 1 -- 172.21.15.189:0/3077118883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f969c067bb0 con 0x7f969c0723f0 2024-04-06T12:28:27.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.097+0000 7f968affd700 1 -- 172.21.15.189:0/3077118883 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 48) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f96900119d0 con 0x7f969c0723f0 2024-04-06T12:28:27.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.097+0000 7f968affd700 1 -- 172.21.15.189:0/3077118883 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6631+0+0 (secure 0 0 0) 0x7f9690021070 con 0x7f969c0723f0 2024-04-06T12:28:27.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.102+0000 7f968affd700 1 -- 172.21.15.189:0/3077118883 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9690055d50 con 0x7f969c0723f0 2024-04-06T12:28:27.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.299+0000 7f96a0985700 1 -- 172.21.15.189:0/3077118883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f969c0d0e00 con 0x7f969c0723f0 2024-04-06T12:28:27.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.300+0000 7f968affd700 1 -- 172.21.15.189:0/3077118883 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+17233 (secure 0 0 0) 0x7f969002b020 con 0x7f969c0723f0 2024-04-06T12:28:27.297 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:27.297 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":100,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:28:26.968341+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:28:04.080500+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-06T12:28:04.619802+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/613690502":"2024-04-07T12:28:26.968304+0000","172.21.15.189:0/2338058300":"2024-04-07T12:28:26.968304+0000","172.21.15.189:0/3832086055":"2024-04-07T12:28:26.968304+0000","172.21.15.189:0/2317885199":"2024-04-07T12:28:26.968304+0000","172.21.15.189:0/530172869":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/4174795471":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/2207020726":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/2070668132":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/1706294115":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/3663637545":"2024-04-07T12:28:26.968304+0000","172.21.15.189:0/4285218889":"2024-04-07T12:28:19.179156+0000","172.21.15.189:0/3308171451":"2024-04-07T12:28:26.968304+0000","172.21.15.189:6833/47128":"2024-04-07T12:28:19.179156+0000","172.21.15.189:6832/47128":"2024-04-07T12:28:19.179156+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:28:27.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.302+0000 7f96a0985700 1 -- 172.21.15.189:0/3077118883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f969c0723f0 msgr2=0x7f969c1539f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:27.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.302+0000 7f96a0985700 1 --2- 172.21.15.189:0/3077118883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f969c0723f0 0x7f969c1539f0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f969000a010 tx=0x7f9690004750 comp rx=0 tx=0).stop 2024-04-06T12:28:27.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.302+0000 7f96a0985700 1 -- 172.21.15.189:0/3077118883 shutdown_connections 2024-04-06T12:28:27.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.302+0000 7f96a0985700 1 --2- 172.21.15.189:0/3077118883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f969c153250 0x7f969c0cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.302+0000 7f96a0985700 1 --2- 172.21.15.189:0/3077118883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f969c0723f0 0x7f969c1539f0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.302+0000 7f96a0985700 1 --2- 172.21.15.189:0/3077118883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c0b39d0 0x7f969c154450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.302+0000 7f96a0985700 1 -- 172.21.15.189:0/3077118883 >> 172.21.15.189:0/3077118883 conn(0x7f969c0d00b0 msgr2=0x7f969c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:27.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.302+0000 7f96a0985700 1 -- 172.21.15.189:0/3077118883 shutdown_connections 2024-04-06T12:28:27.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.302+0000 7f96a0985700 1 -- 172.21.15.189:0/3077118883 wait complete. 2024-04-06T12:28:27.312 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-06T12:28:27.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.437+0000 7f4c16f49700 1 Processor -- start 2024-04-06T12:28:27.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.437+0000 7f4c16f49700 1 -- start start 2024-04-06T12:28:27.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c16f49700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4c1015d7b0 0x7f4c100b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:27.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c16f49700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c1006a730 0x7f4c1006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:27.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c16f49700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4c1015ddf0 0x7f4c1015e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:27.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c16f49700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c1006b0f0 con 0x7f4c1015ddf0 2024-04-06T12:28:27.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c16f49700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c100b3990 con 0x7f4c1015d7b0 2024-04-06T12:28:27.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c16f49700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c100b3060 con 0x7f4c1006a730 2024-04-06T12:28:27.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c154e6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4c1015ddf0 0x7f4c1015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:27.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c154e6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4c1015ddf0 0x7f4c1015e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59326/0 (socket says 172.21.15.189:59326) 2024-04-06T12:28:27.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c154e6700 1 -- 172.21.15.189:0/71459837 learned_addr learned my addr 172.21.15.189:0/71459837 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:27.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c14ce5700 1 --2- 172.21.15.189:0/71459837 >> v2:172.21.15.192:3300/0 conn(0x7f4c1015d7b0 0x7f4c100b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:27.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c0ffff700 1 --2- 172.21.15.189:0/71459837 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c1006a730 0x7f4c1006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:27.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c154e6700 1 -- 172.21.15.189:0/71459837 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c1006a730 msgr2=0x7f4c1006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:27.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c154e6700 1 --2- 172.21.15.189:0/71459837 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c1006a730 0x7f4c1006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c154e6700 1 -- 172.21.15.189:0/71459837 >> v2:172.21.15.192:3300/0 conn(0x7f4c1015d7b0 msgr2=0x7f4c100b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:27.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c154e6700 1 --2- 172.21.15.189:0/71459837 >> v2:172.21.15.192:3300/0 conn(0x7f4c1015d7b0 0x7f4c100b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c154e6700 1 -- 172.21.15.189:0/71459837 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c100b3340 con 0x7f4c1015ddf0 2024-04-06T12:28:27.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c154e6700 1 --2- 172.21.15.189:0/71459837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c1015ddf0 0x7f4c1015e280 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f4c100b6200 tx=0x7f4c0800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c39e9ebb3716ec33 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:27.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c0f7fe700 1 -- 172.21.15.189:0/71459837 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0800e070 con 0x7f4c1015ddf0 2024-04-06T12:28:27.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c0f7fe700 1 -- 172.21.15.189:0/71459837 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0800b9c0 con 0x7f4c1015ddf0 2024-04-06T12:28:27.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.438+0000 7f4c0f7fe700 1 -- 172.21.15.189:0/71459837 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0800bd70 con 0x7f4c1015ddf0 2024-04-06T12:28:27.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.439+0000 7f4c16f49700 1 -- 172.21.15.189:0/71459837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c1015ddf0 msgr2=0x7f4c1015e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:27.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.439+0000 7f4c16f49700 1 --2- 172.21.15.189:0/71459837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c1015ddf0 0x7f4c1015e280 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f4c100b6200 tx=0x7f4c0800b330 comp rx=0 tx=0).stop 2024-04-06T12:28:27.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.439+0000 7f4c16f49700 1 -- 172.21.15.189:0/71459837 shutdown_connections 2024-04-06T12:28:27.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.439+0000 7f4c16f49700 1 --2- 172.21.15.189:0/71459837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c1015ddf0 0x7f4c1015e280 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.439+0000 7f4c16f49700 1 --2- 172.21.15.189:0/71459837 >> v2:172.21.15.192:3300/0 conn(0x7f4c1015d7b0 0x7f4c100b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.439+0000 7f4c16f49700 1 --2- 172.21.15.189:0/71459837 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c1006a730 0x7f4c1006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.439+0000 7f4c16f49700 1 -- 172.21.15.189:0/71459837 >> 172.21.15.189:0/71459837 conn(0x7f4c100d00b0 msgr2=0x7f4c100d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:27.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.439+0000 7f4c16f49700 1 -- 172.21.15.189:0/71459837 shutdown_connections 2024-04-06T12:28:27.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.439+0000 7f4c16f49700 1 -- 172.21.15.189:0/71459837 wait complete. 2024-04-06T12:28:27.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.440+0000 7f4c16f49700 1 Processor -- start 2024-04-06T12:28:27.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.440+0000 7f4c16f49700 1 -- start start 2024-04-06T12:28:27.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.441+0000 7f4c16f49700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c1006a730 0x7f4c10158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:27.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.441+0000 7f4c16f49700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c1015d7b0 0x7f4c10149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:27.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.441+0000 7f4c16f49700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c1015ddf0 0x7f4c100af490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:27.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.441+0000 7f4c16f49700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c10159270 con 0x7f4c1015ddf0 2024-04-06T12:28:27.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.441+0000 7f4c16f49700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c100b1ac0 con 0x7f4c1006a730 2024-04-06T12:28:27.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.441+0000 7f4c16f49700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c100af060 con 0x7f4c1015d7b0 2024-04-06T12:28:27.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.441+0000 7f4c154e6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c1015ddf0 0x7f4c100af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:27.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.441+0000 7f4c154e6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c1015ddf0 0x7f4c100af490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59342/0 (socket says 172.21.15.189:59342) 2024-04-06T12:28:27.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.441+0000 7f4c154e6700 1 -- 172.21.15.189:0/3541584757 learned_addr learned my addr 172.21.15.189:0/3541584757 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:27.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.441+0000 7f4c14ce5700 1 --2- 172.21.15.189:0/3541584757 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c1006a730 0x7f4c10158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:27.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.441+0000 7f4c0ffff700 1 --2- 172.21.15.189:0/3541584757 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c1015d7b0 0x7f4c10149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:27.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.442+0000 7f4c154e6700 1 -- 172.21.15.189:0/3541584757 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c1015d7b0 msgr2=0x7f4c10149ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:27.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.442+0000 7f4c154e6700 1 --2- 172.21.15.189:0/3541584757 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c1015d7b0 0x7f4c10149ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.442+0000 7f4c154e6700 1 -- 172.21.15.189:0/3541584757 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c1006a730 msgr2=0x7f4c10158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:27.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.442+0000 7f4c154e6700 1 --2- 172.21.15.189:0/3541584757 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c1006a730 0x7f4c10158d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.442+0000 7f4c154e6700 1 -- 172.21.15.189:0/3541584757 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c08009b30 con 0x7f4c1015ddf0 2024-04-06T12:28:27.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.442+0000 7f4c0ffff700 1 --2- 172.21.15.189:0/3541584757 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c1015d7b0 0x7f4c10149ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:27.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.442+0000 7f4c154e6700 1 --2- 172.21.15.189:0/3541584757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c1015ddf0 0x7f4c100af490 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f4c0800b730 tx=0x7f4c08004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:27.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.442+0000 7f4c0d7fa700 1 -- 172.21.15.189:0/3541584757 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c080099b0 con 0x7f4c1015ddf0 2024-04-06T12:28:27.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.442+0000 7f4c0d7fa700 1 -- 172.21.15.189:0/3541584757 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c08004250 con 0x7f4c1015ddf0 2024-04-06T12:28:27.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.442+0000 7f4c16f49700 1 -- 172.21.15.189:0/3541584757 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c100ccff0 con 0x7f4c1015ddf0 2024-04-06T12:28:27.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.442+0000 7f4c0d7fa700 1 -- 172.21.15.189:0/3541584757 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c08004600 con 0x7f4c1015ddf0 2024-04-06T12:28:27.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.442+0000 7f4c16f49700 1 -- 172.21.15.189:0/3541584757 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c100cd5b0 con 0x7f4c1015ddf0 2024-04-06T12:28:27.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.443+0000 7f4c16f49700 1 -- 172.21.15.189:0/3541584757 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c10067bb0 con 0x7f4c1015ddf0 2024-04-06T12:28:27.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.443+0000 7f4c0d7fa700 1 -- 172.21.15.189:0/3541584757 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 48) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f4c08004760 con 0x7f4c1015ddf0 2024-04-06T12:28:27.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.444+0000 7f4c0d7fa700 1 -- 172.21.15.189:0/3541584757 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6631+0+0 (secure 0 0 0) 0x7f4c0808a910 con 0x7f4c1015ddf0 2024-04-06T12:28:27.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.448+0000 7f4c0d7fa700 1 -- 172.21.15.189:0/3541584757 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4c08055ba0 con 0x7f4c1015ddf0 2024-04-06T12:28:27.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.688+0000 7f4c16f49700 1 -- 172.21.15.189:0/3541584757 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4c10074990 con 0x7f4c1015ddf0 2024-04-06T12:28:27.686 INFO:tasks.ceph.mon.a.smithi189.stderr:2024-04-06T12:28:27.689+0000 7fa9fe1ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T12:28:27.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.696+0000 7f4c0d7fa700 1 -- 172.21.15.189:0/3541584757 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v40) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4c08020090 con 0x7f4c1015ddf0 2024-04-06T12:28:27.695 INFO:tasks.ceph.mds.a.smithi189.stderr: -3707> 2024-04-06T12:28:11.153+0000 7f7a76c2d700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T12:28:27.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.698+0000 7f4c16f49700 1 -- 172.21.15.189:0/3541584757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c1015ddf0 msgr2=0x7f4c100af490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:27.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.698+0000 7f4c16f49700 1 --2- 172.21.15.189:0/3541584757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c1015ddf0 0x7f4c100af490 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f4c0800b730 tx=0x7f4c08004ad0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.699+0000 7f4c16f49700 1 -- 172.21.15.189:0/3541584757 shutdown_connections 2024-04-06T12:28:27.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.699+0000 7f4c16f49700 1 --2- 172.21.15.189:0/3541584757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c1015ddf0 0x7f4c100af490 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.699+0000 7f4c16f49700 1 --2- 172.21.15.189:0/3541584757 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c1006a730 0x7f4c10158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.699+0000 7f4c16f49700 1 --2- 172.21.15.189:0/3541584757 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c1015d7b0 0x7f4c10149ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.699+0000 7f4c16f49700 1 -- 172.21.15.189:0/3541584757 >> 172.21.15.189:0/3541584757 conn(0x7f4c100d00b0 msgr2=0x7f4c100adf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:27.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.699+0000 7f4c16f49700 1 -- 172.21.15.189:0/3541584757 shutdown_connections 2024-04-06T12:28:27.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.699+0000 7f4c16f49700 1 -- 172.21.15.189:0/3541584757 wait complete. 2024-04-06T12:28:27.697 INFO:teuthology.orchestra.run.smithi189.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-06T12:28:27.709 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-06T12:28:27.724 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2024-04-06T12:28:27.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.835+0000 7fd445423700 1 Processor -- start 2024-04-06T12:28:27.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.835+0000 7fd445423700 1 -- start start 2024-04-06T12:28:27.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.836+0000 7fd445423700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd44006ae40 0x7fd4400b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:27.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.836+0000 7fd445423700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd44015ddf0 0x7fd44015e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:27.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.836+0000 7fd445423700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4400b3990 con 0x7fd44015d7b0 2024-04-06T12:28:27.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.836+0000 7fd445423700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4400b3060 con 0x7fd44015ddf0 2024-04-06T12:28:27.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.836+0000 7fd445423700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4400b3340 con 0x7fd44006ae40 2024-04-06T12:28:27.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.836+0000 7fd43f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd44015ddf0 0x7fd44015e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:27.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.836+0000 7fd43f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd44015ddf0 0x7fd44015e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59386/0 (socket says 172.21.15.189:59386) 2024-04-06T12:28:27.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.836+0000 7fd43f7fe700 1 -- 172.21.15.189:0/566734252 learned_addr learned my addr 172.21.15.189:0/566734252 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:27.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.837+0000 7fd43e7fc700 1 --2- 172.21.15.189:0/566734252 >> v2:172.21.15.192:3300/0 conn(0x7fd44006ae40 0x7fd4400b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:27.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.837+0000 7fd43f7fe700 1 -- 172.21.15.189:0/566734252 >> v2:172.21.15.192:3300/0 conn(0x7fd44006ae40 msgr2=0x7fd4400b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:27.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.837+0000 7fd43f7fe700 1 --2- 172.21.15.189:0/566734252 >> v2:172.21.15.192:3300/0 conn(0x7fd44006ae40 0x7fd4400b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.837+0000 7fd43f7fe700 1 -- 172.21.15.189:0/566734252 >> v1:172.21.15.192:6789/0 conn(0x7fd44015d7b0 legacy=0x7fd44006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:27.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.837+0000 7fd43f7fe700 1 -- 172.21.15.189:0/566734252 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4400b3620 con 0x7fd44015ddf0 2024-04-06T12:28:27.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.837+0000 7fd43f7fe700 1 --2- 172.21.15.189:0/566734252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44015ddf0 0x7fd44015e240 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fd4400afc00 tx=0x7fd43800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92cb851093ee4a63 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:27.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.837+0000 7fd43dffb700 1 -- 172.21.15.189:0/566734252 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43800e070 con 0x7fd44015ddf0 2024-04-06T12:28:27.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.837+0000 7fd43dffb700 1 -- 172.21.15.189:0/566734252 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd438003a20 con 0x7fd44015ddf0 2024-04-06T12:28:27.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.837+0000 7fd43dffb700 1 -- 172.21.15.189:0/566734252 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd438003dd0 con 0x7fd44015ddf0 2024-04-06T12:28:27.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.837+0000 7fd445423700 1 -- 172.21.15.189:0/566734252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44015ddf0 msgr2=0x7fd44015e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:27.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.837+0000 7fd445423700 1 --2- 172.21.15.189:0/566734252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44015ddf0 0x7fd44015e240 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fd4400afc00 tx=0x7fd43800b330 comp rx=0 tx=0).stop 2024-04-06T12:28:27.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.837+0000 7fd445423700 1 -- 172.21.15.189:0/566734252 shutdown_connections 2024-04-06T12:28:27.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.838+0000 7fd445423700 1 --2- 172.21.15.189:0/566734252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44015ddf0 0x7fd44015e240 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.838+0000 7fd445423700 1 --2- 172.21.15.189:0/566734252 >> v2:172.21.15.192:3300/0 conn(0x7fd44006ae40 0x7fd4400b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.838+0000 7fd445423700 1 -- 172.21.15.189:0/566734252 >> 172.21.15.189:0/566734252 conn(0x7fd4400d00b0 msgr2=0x7fd4400d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:27.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.838+0000 7fd445423700 1 -- 172.21.15.189:0/566734252 shutdown_connections 2024-04-06T12:28:27.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.838+0000 7fd445423700 1 -- 172.21.15.189:0/566734252 wait complete. 2024-04-06T12:28:27.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.839+0000 7fd445423700 1 Processor -- start 2024-04-06T12:28:27.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.839+0000 7fd445423700 1 -- start start 2024-04-06T12:28:27.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd445423700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd44006ae40 0x7fd440153a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:27.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd445423700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd44015d7b0 0x7fd4400cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:27.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd445423700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44015ddf0 0x7fd44015fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:27.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd445423700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd440154550 con 0x7fd44015ddf0 2024-04-06T12:28:27.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd445423700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4401547a0 con 0x7fd44006ae40 2024-04-06T12:28:27.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd445423700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd440154a80 con 0x7fd44015d7b0 2024-04-06T12:28:27.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd43f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44015ddf0 0x7fd44015fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:27.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd43f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44015ddf0 0x7fd44015fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59388/0 (socket says 172.21.15.189:59388) 2024-04-06T12:28:27.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd43f7fe700 1 -- 172.21.15.189:0/2125980905 learned_addr learned my addr 172.21.15.189:0/2125980905 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:27.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd43e7fc700 1 --2- 172.21.15.189:0/2125980905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd44015d7b0 0x7fd4400cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:27.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd43effd700 1 --2- 172.21.15.189:0/2125980905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd44006ae40 0x7fd440153a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:27.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd43f7fe700 1 -- 172.21.15.189:0/2125980905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd44015d7b0 msgr2=0x7fd4400cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:27.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd43f7fe700 1 --2- 172.21.15.189:0/2125980905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd44015d7b0 0x7fd4400cc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd43f7fe700 1 -- 172.21.15.189:0/2125980905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd44006ae40 msgr2=0x7fd440153a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:27.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd43f7fe700 1 --2- 172.21.15.189:0/2125980905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd44006ae40 0x7fd440153a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:27.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd43f7fe700 1 -- 172.21.15.189:0/2125980905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd438009b30 con 0x7fd44015ddf0 2024-04-06T12:28:27.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd43effd700 1 --2- 172.21.15.189:0/2125980905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd44006ae40 0x7fd440153a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:27.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd43f7fe700 1 --2- 172.21.15.189:0/2125980905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44015ddf0 0x7fd44015fdc0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fd43800b370 tx=0x7fd438004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:27.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd4277fe700 1 -- 172.21.15.189:0/2125980905 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd438004440 con 0x7fd44015ddf0 2024-04-06T12:28:27.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd4277fe700 1 -- 172.21.15.189:0/2125980905 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4380045c0 con 0x7fd44015ddf0 2024-04-06T12:28:27.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd445423700 1 -- 172.21.15.189:0/2125980905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd440154d60 con 0x7fd44015ddf0 2024-04-06T12:28:27.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd4277fe700 1 -- 172.21.15.189:0/2125980905 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd438004970 con 0x7fd44015ddf0 2024-04-06T12:28:27.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.840+0000 7fd445423700 1 -- 172.21.15.189:0/2125980905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4400adcd0 con 0x7fd44015ddf0 2024-04-06T12:28:27.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.841+0000 7fd445423700 1 -- 172.21.15.189:0/2125980905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd440155280 con 0x7fd44015ddf0 2024-04-06T12:28:27.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.842+0000 7fd4277fe700 1 -- 172.21.15.189:0/2125980905 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 48) v1 ==== 88856+0+0 (secure 0 0 0) 0x7fd438012430 con 0x7fd44015ddf0 2024-04-06T12:28:27.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.842+0000 7fd4277fe700 1 -- 172.21.15.189:0/2125980905 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6717+0+0 (secure 0 0 0) 0x7fd43808a4e0 con 0x7fd44015ddf0 2024-04-06T12:28:27.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.846+0000 7fd4277fe700 1 -- 172.21.15.189:0/2125980905 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd440155280 con 0x7fd44015ddf0 2024-04-06T12:28:27.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.979+0000 7fd4277fe700 1 -- 172.21.15.189:0/2125980905 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 49) v1 ==== 88983+0+0 (secure 0 0 0) 0x7fd4380551d0 con 0x7fd44015ddf0 2024-04-06T12:28:27.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.979+0000 7fd4277fe700 1 --2- 172.21.15.189:0/2125980905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd42c036150 0x7fd42c0728f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:27.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.980+0000 7fd43effd700 1 --2- 172.21.15.189:0/2125980905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd42c036150 0x7fd42c0728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:27.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:27.981+0000 7fd43effd700 1 --2- 172.21.15.189:0/2125980905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd42c036150 0x7fd42c0728f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd434003fc0 tx=0x7fd434009040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:28.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.094+0000 7fd445423700 1 -- 172.21.15.189:0/2125980905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd440155280 con 0x7fd44015ddf0 2024-04-06T12:28:28.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.698+0000 7fd4277fe700 1 -- 172.21.15.189:0/2125980905 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v41) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd440155280 con 0x7fd44015ddf0 2024-04-06T12:28:28.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.701+0000 7fd445423700 1 -- 172.21.15.189:0/2125980905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd42c036150 msgr2=0x7fd42c0728f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:28.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.701+0000 7fd445423700 1 --2- 172.21.15.189:0/2125980905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd42c036150 0x7fd42c0728f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd434003fc0 tx=0x7fd434009040 comp rx=0 tx=0).stop 2024-04-06T12:28:28.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.701+0000 7fd445423700 1 -- 172.21.15.189:0/2125980905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44015ddf0 msgr2=0x7fd44015fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:28.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.701+0000 7fd445423700 1 --2- 172.21.15.189:0/2125980905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44015ddf0 0x7fd44015fdc0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fd43800b370 tx=0x7fd438004ad0 comp rx=0 tx=0).stop 2024-04-06T12:28:28.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.701+0000 7fd445423700 1 -- 172.21.15.189:0/2125980905 shutdown_connections 2024-04-06T12:28:28.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.701+0000 7fd445423700 1 --2- 172.21.15.189:0/2125980905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44015ddf0 0x7fd44015fdc0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:28.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.701+0000 7fd445423700 1 --2- 172.21.15.189:0/2125980905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd42c036150 0x7fd42c0728f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:28.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.701+0000 7fd445423700 1 --2- 172.21.15.189:0/2125980905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd44006ae40 0x7fd440153a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:28.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.701+0000 7fd445423700 1 --2- 172.21.15.189:0/2125980905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd44015d7b0 0x7fd4400cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:28.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.701+0000 7fd445423700 1 -- 172.21.15.189:0/2125980905 >> 172.21.15.189:0/2125980905 conn(0x7fd4400d00b0 msgr2=0x7fd440072400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:28.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.701+0000 7fd445423700 1 -- 172.21.15.189:0/2125980905 shutdown_connections 2024-04-06T12:28:28.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.701+0000 7fd445423700 1 -- 172.21.15.189:0/2125980905 wait complete. 2024-04-06T12:28:28.713 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-06T12:28:28.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.889+0000 7fbfb181c700 1 Processor -- start 2024-04-06T12:28:28.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.889+0000 7fbfb181c700 1 -- start start 2024-04-06T12:28:28.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.890+0000 7fbfb181c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbfac0b4ed0 0x7fbfac0b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:28.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.890+0000 7fbfb181c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfac0d2880 0x7fbfac0d2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:28.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.890+0000 7fbfb181c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfac0b5cc0 con 0x7fbfac15d590 2024-04-06T12:28:28.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.890+0000 7fbfb181c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfac0b58c0 con 0x7fbfac0b4ed0 2024-04-06T12:28:28.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.890+0000 7fbfb181c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac0d1500 con 0x7fbfac0d2880 2024-04-06T12:28:28.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.890+0000 7fbfaaffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbfac15d590 0x7fbfac0b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33728/0 (socket says 172.21.15.189:33728) 2024-04-06T12:28:28.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.890+0000 7fbfaaffd700 1 -- 172.21.15.189:0/1214239312 learned_addr learned my addr 172.21.15.189:0/1214239312 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:28.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.890+0000 7fbfab7fe700 1 --2- 172.21.15.189:0/1214239312 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfac0d2880 0x7fbfac0d2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:28.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.890+0000 7fbfaa7fc700 1 --2- 172.21.15.189:0/1214239312 >> v2:172.21.15.192:3300/0 conn(0x7fbfac0b4ed0 0x7fbfac0b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:28.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.890+0000 7fbfa9ffb700 1 -- 172.21.15.189:0/1214239312 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1210393233 0 0) 0x7fbfac0b5cc0 con 0x7fbfac15d590 2024-04-06T12:28:28.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.890+0000 7fbfa9ffb700 1 -- 172.21.15.189:0/1214239312 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf940035b0 con 0x7fbfac15d590 2024-04-06T12:28:28.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfa9ffb700 1 -- 172.21.15.189:0/1214239312 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 851522535 0 0) 0x7fbf940035b0 con 0x7fbfac15d590 2024-04-06T12:28:28.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfa9ffb700 1 -- 172.21.15.189:0/1214239312 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfac0d2880 msgr2=0x7fbfac0d2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:28.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfa9ffb700 1 --2- 172.21.15.189:0/1214239312 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfac0d2880 0x7fbfac0d2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:28.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfa9ffb700 1 -- 172.21.15.189:0/1214239312 >> v2:172.21.15.192:3300/0 conn(0x7fbfac0b4ed0 msgr2=0x7fbfac0b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:28.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfa9ffb700 1 --2- 172.21.15.189:0/1214239312 >> v2:172.21.15.192:3300/0 conn(0x7fbfac0b4ed0 0x7fbfac0b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:28.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfa9ffb700 1 -- 172.21.15.189:0/1214239312 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfac0d17e0 con 0x7fbfac15d590 2024-04-06T12:28:28.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfa9ffb700 1 -- 172.21.15.189:0/1214239312 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fbf9c001670 con 0x7fbfac15d590 2024-04-06T12:28:28.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfa9ffb700 1 -- 172.21.15.189:0/1214239312 >> v1:172.21.15.189:6789/0 conn(0x7fbfac15d590 legacy=0x7fbfac0b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:28.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfa9ffb700 1 --2- 172.21.15.189:0/1214239312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf94003a00 0x7fbf94003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:28.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfa9ffb700 1 --2- 172.21.15.189:0/1214239312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf94004850 0x7fbf94004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:28.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfa9ffb700 1 --2- 172.21.15.189:0/1214239312 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfac0d2880 0x7fbf94005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:28.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfa9ffb700 1 -- 172.21.15.189:0/1214239312 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac0b58c0 con 0x7fbf94004850 2024-04-06T12:28:28.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfa9ffb700 1 -- 172.21.15.189:0/1214239312 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac0d1500 con 0x7fbf94003a00 2024-04-06T12:28:28.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfa9ffb700 1 -- 172.21.15.189:0/1214239312 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9400ba10 con 0x7fbfac0d2880 2024-04-06T12:28:28.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfaa7fc700 1 --2- 172.21.15.189:0/1214239312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf94004850 0x7fbf94004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:28.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfaaffd700 1 --2- 172.21.15.189:0/1214239312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf94003a00 0x7fbf94003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:28.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfab7fe700 1 --2- 172.21.15.189:0/1214239312 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfac0d2880 0x7fbf94005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:28.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfaa7fc700 1 -- 172.21.15.189:0/1214239312 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfac0d2880 msgr2=0x7fbf94005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:28.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfaa7fc700 1 --2- 172.21.15.189:0/1214239312 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfac0d2880 0x7fbf94005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:28.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfaa7fc700 1 -- 172.21.15.189:0/1214239312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf94003a00 msgr2=0x7fbf94003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:28.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfaa7fc700 1 --2- 172.21.15.189:0/1214239312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf94003a00 0x7fbf94003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:28.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfaa7fc700 1 -- 172.21.15.189:0/1214239312 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf940035b0 con 0x7fbf94004850 2024-04-06T12:28:28.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfaa7fc700 1 --2- 172.21.15.189:0/1214239312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf94004850 0x7fbf94004cd0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0009b60 tx=0x7fbf94005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2661d03425d175b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:28.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfa9ffb700 1 -- 172.21.15.189:0/1214239312 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa000e6c0 con 0x7fbf94004850 2024-04-06T12:28:28.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfa9ffb700 1 -- 172.21.15.189:0/1214239312 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa000e820 con 0x7fbf94004850 2024-04-06T12:28:28.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.891+0000 7fbfa9ffb700 1 -- 172.21.15.189:0/1214239312 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa000eb90 con 0x7fbf94004850 2024-04-06T12:28:28.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.892+0000 7fbfb181c700 1 -- 172.21.15.189:0/1214239312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf94004850 msgr2=0x7fbf94004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:28.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.892+0000 7fbfb181c700 1 --2- 172.21.15.189:0/1214239312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf94004850 0x7fbf94004cd0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0009b60 tx=0x7fbf94005880 comp rx=0 tx=0).stop 2024-04-06T12:28:28.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.892+0000 7fbfab7fe700 1 -- 172.21.15.189:0/1214239312 reap_dead start 2024-04-06T12:28:28.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.892+0000 7fbfb181c700 1 -- 172.21.15.189:0/1214239312 shutdown_connections 2024-04-06T12:28:28.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.892+0000 7fbfb181c700 1 -- 172.21.15.189:0/1214239312 >> 172.21.15.189:0/1214239312 conn(0x7fbfac06bbc0 msgr2=0x7fbfac06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:28.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.892+0000 7fbfb181c700 1 -- 172.21.15.189:0/1214239312 shutdown_connections 2024-04-06T12:28:28.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.892+0000 7fbfb181c700 1 -- 172.21.15.189:0/1214239312 wait complete. 2024-04-06T12:28:28.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.893+0000 7fbfb181c700 1 Processor -- start 2024-04-06T12:28:28.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.893+0000 7fbfb181c700 1 -- start start 2024-04-06T12:28:28.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.894+0000 7fbfb181c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf94004850 0x7fbfac1f3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:28.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.894+0000 7fbfb181c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfac15f500 0x7fbfac15f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:28.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.894+0000 7fbfb181c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbfac15e330 0x7fbfac15e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:28.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.894+0000 7fbfb181c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac15fec0 con 0x7fbf94004850 2024-04-06T12:28:28.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.894+0000 7fbfb181c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac0cb1d0 con 0x7fbfac15e330 2024-04-06T12:28:28.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.894+0000 7fbfb181c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac15ecf0 con 0x7fbfac15f500 2024-04-06T12:28:28.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.894+0000 7fbfaaffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf94004850 0x7fbfac1f3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:28.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.894+0000 7fbfaaffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf94004850 0x7fbfac1f3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59406/0 (socket says 172.21.15.189:59406) 2024-04-06T12:28:28.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.894+0000 7fbfaaffd700 1 -- 172.21.15.189:0/2216210746 learned_addr learned my addr 172.21.15.189:0/2216210746 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:28.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.894+0000 7fbfab7fe700 1 --2- 172.21.15.189:0/2216210746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbfac15e330 0x7fbfac15e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:28.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.895+0000 7fbfaaffd700 1 -- 172.21.15.189:0/2216210746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfac15f500 msgr2=0x7fbfac15f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:28.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.895+0000 7fbfaaffd700 1 --2- 172.21.15.189:0/2216210746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfac15f500 0x7fbfac15f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:28.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.895+0000 7fbfaaffd700 1 -- 172.21.15.189:0/2216210746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbfac15e330 msgr2=0x7fbfac15e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:28.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.895+0000 7fbfaaffd700 1 --2- 172.21.15.189:0/2216210746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbfac15e330 0x7fbfac15e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:28.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.895+0000 7fbfaaffd700 1 -- 172.21.15.189:0/2216210746 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfac0cc970 con 0x7fbf94004850 2024-04-06T12:28:28.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.895+0000 7fbfaaffd700 1 --2- 172.21.15.189:0/2216210746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf94004850 0x7fbfac1f3e60 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c000fb0 tx=0x7fbf9c0075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:28.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.895+0000 7fbf93fff700 1 -- 172.21.15.189:0/2216210746 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c0077d0 con 0x7fbf94004850 2024-04-06T12:28:28.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.895+0000 7fbf93fff700 1 -- 172.21.15.189:0/2216210746 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9c00f8d0 con 0x7fbf94004850 2024-04-06T12:28:28.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.895+0000 7fbfb181c700 1 -- 172.21.15.189:0/2216210746 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfac0ca5c0 con 0x7fbf94004850 2024-04-06T12:28:28.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.895+0000 7fbf93fff700 1 -- 172.21.15.189:0/2216210746 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c00fc80 con 0x7fbf94004850 2024-04-06T12:28:28.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.895+0000 7fbfb181c700 1 -- 172.21.15.189:0/2216210746 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfac0cab80 con 0x7fbf94004850 2024-04-06T12:28:28.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.896+0000 7fbfb181c700 1 -- 172.21.15.189:0/2216210746 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfac074990 con 0x7fbf94004850 2024-04-06T12:28:28.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.896+0000 7fbf93fff700 1 -- 172.21.15.189:0/2216210746 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 49) v1 ==== 88983+0+0 (secure 0 0 0) 0x7fbf9c01d460 con 0x7fbf94004850 2024-04-06T12:28:28.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.897+0000 7fbf93fff700 1 --2- 172.21.15.189:0/2216210746 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf9806aae0 0x7fbf9806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:28.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.897+0000 7fbf93fff700 1 -- 172.21.15.189:0/2216210746 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6717+0+0 (secure 0 0 0) 0x7fbf9c089470 con 0x7fbf94004850 2024-04-06T12:28:28.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.897+0000 7fbfaa7fc700 1 --2- 172.21.15.189:0/2216210746 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf9806aae0 0x7fbf9806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:28.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.897+0000 7fbfaa7fc700 1 --2- 172.21.15.189:0/2216210746 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf9806aae0 0x7fbf9806cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbfac160920 tx=0x7fbfa0013000 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:28.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:28.901+0000 7fbf93fff700 1 -- 172.21.15.189:0/2216210746 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbf9c0544d0 con 0x7fbf94004850 2024-04-06T12:28:29.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:29.103+0000 7fbfb181c700 1 -- 172.21.15.189:0/2216210746 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbfac0b49b0 con 0x7fbf94004850 2024-04-06T12:28:29.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:29.993+0000 7fbf93fff700 1 -- 172.21.15.189:0/2216210746 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 50) v1 ==== 89191+0+0 (secure 0 0 0) 0x7fbf9c0548a0 con 0x7fbf94004850 2024-04-06T12:28:29.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:29.993+0000 7fbf93fff700 1 -- 172.21.15.189:0/2216210746 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v102) v1 ==== 181+0+0 (secure 0 0 0) 0x7fbf9c050ad0 con 0x7fbf94004850 2024-04-06T12:28:29.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:29.996+0000 7fbfb181c700 1 -- 172.21.15.189:0/2216210746 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf9806aae0 msgr2=0x7fbf9806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:29.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:29.996+0000 7fbfb181c700 1 --2- 172.21.15.189:0/2216210746 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf9806aae0 0x7fbf9806cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbfac160920 tx=0x7fbfa0013000 comp rx=0 tx=0).stop 2024-04-06T12:28:29.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:29.996+0000 7fbfb181c700 1 -- 172.21.15.189:0/2216210746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf94004850 msgr2=0x7fbfac1f3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:29.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:29.996+0000 7fbfb181c700 1 --2- 172.21.15.189:0/2216210746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf94004850 0x7fbfac1f3e60 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c000fb0 tx=0x7fbf9c0075f0 comp rx=0 tx=0).stop 2024-04-06T12:28:29.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:29.996+0000 7fbfb181c700 1 -- 172.21.15.189:0/2216210746 shutdown_connections 2024-04-06T12:28:29.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:29.996+0000 7fbfb181c700 1 --2- 172.21.15.189:0/2216210746 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbf9806aae0 0x7fbf9806cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:29.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:29.996+0000 7fbfb181c700 1 --2- 172.21.15.189:0/2216210746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbfac15e330 0x7fbfac15e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:29.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:29.996+0000 7fbfb181c700 1 --2- 172.21.15.189:0/2216210746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf94004850 0x7fbfac1f3e60 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:29.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:29.996+0000 7fbfb181c700 1 --2- 172.21.15.189:0/2216210746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbfac15f500 0x7fbfac15f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:29.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:29.996+0000 7fbfb181c700 1 -- 172.21.15.189:0/2216210746 >> 172.21.15.189:0/2216210746 conn(0x7fbfac06bbc0 msgr2=0x7fbfac0b6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:29.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:29.996+0000 7fbfb181c700 1 -- 172.21.15.189:0/2216210746 shutdown_connections 2024-04-06T12:28:29.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:29.996+0000 7fbfb181c700 1 -- 172.21.15.189:0/2216210746 wait complete. 2024-04-06T12:28:29.994 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' removed 2024-04-06T12:28:30.008 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-06T12:28:30.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.141+0000 7f1b986ed700 1 Processor -- start 2024-04-06T12:28:30.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.141+0000 7f1b986ed700 1 -- start start 2024-04-06T12:28:30.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.142+0000 7f1b986ed700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b900d17a0 0x7f1b9006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:30.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.142+0000 7f1b986ed700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1b900cedf0 0x7f1b900cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:30.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.142+0000 7f1b986ed700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b900cf760 con 0x7f1b90146dd0 2024-04-06T12:28:30.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.142+0000 7f1b986ed700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b9006d850 con 0x7f1b900cedf0 2024-04-06T12:28:30.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.142+0000 7f1b986ed700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9006c0a0 con 0x7f1b900d17a0 2024-04-06T12:28:30.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.142+0000 7f1b96c8a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1b900cedf0 0x7f1b900cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:30.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.142+0000 7f1b96c8a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1b900cedf0 0x7f1b900cf220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36024/0 (socket says 172.21.15.189:36024) 2024-04-06T12:28:30.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.142+0000 7f1b96c8a700 1 -- 172.21.15.189:0/3763206534 learned_addr learned my addr 172.21.15.189:0/3763206534 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:30.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.142+0000 7f1b96489700 1 --2- 172.21.15.189:0/3763206534 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b900d17a0 0x7f1b9006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:30.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.142+0000 7f1b96c8a700 1 -- 172.21.15.189:0/3763206534 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b900d17a0 msgr2=0x7f1b9006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:30.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.142+0000 7f1b96c8a700 1 --2- 172.21.15.189:0/3763206534 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b900d17a0 0x7f1b9006b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:30.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.142+0000 7f1b96c8a700 1 -- 172.21.15.189:0/3763206534 >> v1:172.21.15.189:6789/0 conn(0x7f1b90146dd0 legacy=0x7f1b9006d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:30.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.142+0000 7f1b96c8a700 1 -- 172.21.15.189:0/3763206534 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9006c380 con 0x7f1b900cedf0 2024-04-06T12:28:30.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.143+0000 7f1b96c8a700 1 --2- 172.21.15.189:0/3763206534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b900cedf0 0x7f1b900cf220 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f1b88009e50 tx=0x7f1b8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc1ea4b4af52b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:30.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.143+0000 7f1b95487700 1 -- 172.21.15.189:0/3763206534 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8800e070 con 0x7f1b900cedf0 2024-04-06T12:28:30.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.143+0000 7f1b95487700 1 -- 172.21.15.189:0/3763206534 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8800b9c0 con 0x7f1b900cedf0 2024-04-06T12:28:30.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.143+0000 7f1b95487700 1 -- 172.21.15.189:0/3763206534 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8800bd70 con 0x7f1b900cedf0 2024-04-06T12:28:30.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.143+0000 7f1b986ed700 1 -- 172.21.15.189:0/3763206534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b900cedf0 msgr2=0x7f1b900cf220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:30.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.143+0000 7f1b986ed700 1 --2- 172.21.15.189:0/3763206534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b900cedf0 0x7f1b900cf220 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f1b88009e50 tx=0x7f1b8800b330 comp rx=0 tx=0).stop 2024-04-06T12:28:30.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.143+0000 7f1b986ed700 1 -- 172.21.15.189:0/3763206534 shutdown_connections 2024-04-06T12:28:30.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.143+0000 7f1b986ed700 1 --2- 172.21.15.189:0/3763206534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b900cedf0 0x7f1b900cf220 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:30.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.143+0000 7f1b986ed700 1 --2- 172.21.15.189:0/3763206534 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b900d17a0 0x7f1b9006b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:30.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.143+0000 7f1b986ed700 1 -- 172.21.15.189:0/3763206534 >> 172.21.15.189:0/3763206534 conn(0x7f1b900b2db0 msgr2=0x7f1b900b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:30.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.143+0000 7f1b986ed700 1 -- 172.21.15.189:0/3763206534 shutdown_connections 2024-04-06T12:28:30.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.143+0000 7f1b986ed700 1 -- 172.21.15.189:0/3763206534 wait complete. 2024-04-06T12:28:30.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.144+0000 7f1b986ed700 1 Processor -- start 2024-04-06T12:28:30.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.145+0000 7f1b986ed700 1 -- start start 2024-04-06T12:28:30.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.145+0000 7f1b986ed700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b900cedf0 0x7f1b9015fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:30.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.145+0000 7f1b986ed700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b900d17a0 0x7f1b90159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:30.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.145+0000 7f1b986ed700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b90146dd0 0x7f1b900b6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:30.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.145+0000 7f1b986ed700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b900c6f50 con 0x7f1b90146dd0 2024-04-06T12:28:30.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.145+0000 7f1b986ed700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9015f940 con 0x7f1b900cedf0 2024-04-06T12:28:30.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.145+0000 7f1b986ed700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b90160280 con 0x7f1b900d17a0 2024-04-06T12:28:30.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.145+0000 7f1b96c8a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b90146dd0 0x7f1b900b6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:30.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.145+0000 7f1b96c8a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b90146dd0 0x7f1b900b6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36032/0 (socket says 172.21.15.189:36032) 2024-04-06T12:28:30.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.145+0000 7f1b96c8a700 1 -- 172.21.15.189:0/1164495343 learned_addr learned my addr 172.21.15.189:0/1164495343 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:30.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.146+0000 7f1b96489700 1 --2- 172.21.15.189:0/1164495343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b900cedf0 0x7f1b9015fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:30.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.146+0000 7f1b95c88700 1 --2- 172.21.15.189:0/1164495343 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b900d17a0 0x7f1b90159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:30.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.146+0000 7f1b96c8a700 1 -- 172.21.15.189:0/1164495343 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b900d17a0 msgr2=0x7f1b90159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:30.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.146+0000 7f1b96c8a700 1 --2- 172.21.15.189:0/1164495343 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b900d17a0 0x7f1b90159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:30.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.146+0000 7f1b96c8a700 1 -- 172.21.15.189:0/1164495343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b900cedf0 msgr2=0x7f1b9015fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:30.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.146+0000 7f1b96c8a700 1 --2- 172.21.15.189:0/1164495343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b900cedf0 0x7f1b9015fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:30.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.146+0000 7f1b96c8a700 1 -- 172.21.15.189:0/1164495343 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b88009b30 con 0x7f1b90146dd0 2024-04-06T12:28:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.146+0000 7f1b96c8a700 1 --2- 172.21.15.189:0/1164495343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b90146dd0 0x7f1b900b6070 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f1b900677e0 tx=0x7f1b88004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.146+0000 7f1b86ffd700 1 -- 172.21.15.189:0/1164495343 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b88004960 con 0x7f1b90146dd0 2024-04-06T12:28:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.146+0000 7f1b86ffd700 1 -- 172.21.15.189:0/1164495343 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b880099b0 con 0x7f1b90146dd0 2024-04-06T12:28:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.146+0000 7f1b986ed700 1 -- 172.21.15.189:0/1164495343 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b900c62d0 con 0x7f1b90146dd0 2024-04-06T12:28:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.146+0000 7f1b86ffd700 1 -- 172.21.15.189:0/1164495343 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8801b670 con 0x7f1b90146dd0 2024-04-06T12:28:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.146+0000 7f1b986ed700 1 -- 172.21.15.189:0/1164495343 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b900c6890 con 0x7f1b90146dd0 2024-04-06T12:28:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.147+0000 7f1b986ed700 1 -- 172.21.15.189:0/1164495343 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b900c7090 con 0x7f1b90146dd0 2024-04-06T12:28:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.147+0000 7f1b86ffd700 1 -- 172.21.15.189:0/1164495343 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 50) v1 ==== 89191+0+0 (secure 0 0 0) 0x7f1b8801b7d0 con 0x7f1b90146dd0 2024-04-06T12:28:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.148+0000 7f1b86ffd700 1 --2- 172.21.15.189:0/1164495343 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1b7406ad20 0x7f1b7406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.148+0000 7f1b86ffd700 1 -- 172.21.15.189:0/1164495343 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6279+0+0 (secure 0 0 0) 0x7f1b8808afb0 con 0x7f1b90146dd0 2024-04-06T12:28:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.148+0000 7f1b96489700 1 --2- 172.21.15.189:0/1164495343 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1b7406ad20 0x7f1b7406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.148+0000 7f1b96489700 1 --2- 172.21.15.189:0/1164495343 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1b7406ad20 0x7f1b7406d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1b8000b9b0 tx=0x7f1b80008040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:30.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.152+0000 7f1b86ffd700 1 -- 172.21.15.189:0/1164495343 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1b900c7090 con 0x7f1b90146dd0 2024-04-06T12:28:30.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:30.354+0000 7f1b986ed700 1 -- 172.21.15.189:0/1164495343 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1b900c7090 con 0x7f1b90146dd0 2024-04-06T12:28:31.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.228+0000 7f1b86ffd700 1 -- 172.21.15.189:0/1164495343 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f1b880562b0 con 0x7f1b90146dd0 2024-04-06T12:28:31.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.228+0000 7f1b86ffd700 1 -- 172.21.15.189:0/1164495343 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v103) v1 ==== 169+0+0 (secure 0 0 0) 0x7f1b88052580 con 0x7f1b90146dd0 2024-04-06T12:28:31.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.232+0000 7f1b986ed700 1 -- 172.21.15.189:0/1164495343 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1b7406ad20 msgr2=0x7f1b7406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:31.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.232+0000 7f1b986ed700 1 --2- 172.21.15.189:0/1164495343 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1b7406ad20 0x7f1b7406d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1b8000b9b0 tx=0x7f1b80008040 comp rx=0 tx=0).stop 2024-04-06T12:28:31.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.232+0000 7f1b986ed700 1 -- 172.21.15.189:0/1164495343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b90146dd0 msgr2=0x7f1b900b6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:31.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.232+0000 7f1b986ed700 1 --2- 172.21.15.189:0/1164495343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b90146dd0 0x7f1b900b6070 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f1b900677e0 tx=0x7f1b88004ad0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.232+0000 7f1b986ed700 1 -- 172.21.15.189:0/1164495343 shutdown_connections 2024-04-06T12:28:31.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.232+0000 7f1b986ed700 1 --2- 172.21.15.189:0/1164495343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b90146dd0 0x7f1b900b6070 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.232+0000 7f1b986ed700 1 --2- 172.21.15.189:0/1164495343 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1b7406ad20 0x7f1b7406d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.232+0000 7f1b986ed700 1 --2- 172.21.15.189:0/1164495343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b900cedf0 0x7f1b9015fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.232+0000 7f1b986ed700 1 --2- 172.21.15.189:0/1164495343 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b900d17a0 0x7f1b90159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.232+0000 7f1b986ed700 1 -- 172.21.15.189:0/1164495343 >> 172.21.15.189:0/1164495343 conn(0x7f1b900b2db0 msgr2=0x7f1b9006d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:31.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.232+0000 7f1b986ed700 1 -- 172.21.15.189:0/1164495343 shutdown_connections 2024-04-06T12:28:31.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.232+0000 7f1b986ed700 1 -- 172.21.15.189:0/1164495343 wait complete. 2024-04-06T12:28:31.230 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' removed 2024-04-06T12:28:31.244 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-06T12:28:31.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.376+0000 7fc27ab6b700 1 Processor -- start 2024-04-06T12:28:31.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.376+0000 7fc27ab6b700 1 -- start start 2024-04-06T12:28:31.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.377+0000 7fc27ab6b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc27415d7b0 0x7fc2740b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:31.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.377+0000 7fc27ab6b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc27406a730 0x7fc27406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:31.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.377+0000 7fc27ab6b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2740b3990 con 0x7fc27415ddf0 2024-04-06T12:28:31.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.377+0000 7fc27ab6b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc27415e950 con 0x7fc27415d7b0 2024-04-06T12:28:31.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.377+0000 7fc27ab6b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc27406b0f0 con 0x7fc27406a730 2024-04-06T12:28:31.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.377+0000 7fc278907700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc27415d7b0 0x7fc2740b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:31.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.377+0000 7fc278907700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc27415d7b0 0x7fc2740b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36046/0 (socket says 172.21.15.189:36046) 2024-04-06T12:28:31.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.377+0000 7fc278907700 1 -- 172.21.15.189:0/2519833955 learned_addr learned my addr 172.21.15.189:0/2519833955 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:31.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.377+0000 7fc273fff700 1 --2- 172.21.15.189:0/2519833955 >> v2:172.21.15.192:3300/0 conn(0x7fc27406a730 0x7fc27406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:31.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.377+0000 7fc278907700 1 -- 172.21.15.189:0/2519833955 >> v2:172.21.15.192:3300/0 conn(0x7fc27406a730 msgr2=0x7fc27406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:31.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.377+0000 7fc278907700 1 --2- 172.21.15.189:0/2519833955 >> v2:172.21.15.192:3300/0 conn(0x7fc27406a730 0x7fc27406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.377+0000 7fc278907700 1 -- 172.21.15.189:0/2519833955 >> v1:172.21.15.192:6789/0 conn(0x7fc27415ddf0 legacy=0x7fc27415e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:31.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.377+0000 7fc278907700 1 -- 172.21.15.189:0/2519833955 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2740b3060 con 0x7fc27415d7b0 2024-04-06T12:28:31.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.377+0000 7fc278907700 1 --2- 172.21.15.189:0/2519833955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc27415d7b0 0x7fc2740b3e20 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fc2740afc00 tx=0x7fc268009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7119275691d401 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:31.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.377+0000 7fc2737fe700 1 -- 172.21.15.189:0/2519833955 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26800b870 con 0x7fc27415d7b0 2024-04-06T12:28:31.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.378+0000 7fc2737fe700 1 -- 172.21.15.189:0/2519833955 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26800b9d0 con 0x7fc27415d7b0 2024-04-06T12:28:31.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.378+0000 7fc2737fe700 1 -- 172.21.15.189:0/2519833955 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26800bd60 con 0x7fc27415d7b0 2024-04-06T12:28:31.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.378+0000 7fc27ab6b700 1 -- 172.21.15.189:0/2519833955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc27415d7b0 msgr2=0x7fc2740b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:31.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.378+0000 7fc27ab6b700 1 --2- 172.21.15.189:0/2519833955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc27415d7b0 0x7fc2740b3e20 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fc2740afc00 tx=0x7fc268009bd0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.378+0000 7fc27ab6b700 1 -- 172.21.15.189:0/2519833955 shutdown_connections 2024-04-06T12:28:31.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.378+0000 7fc27ab6b700 1 --2- 172.21.15.189:0/2519833955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc27415d7b0 0x7fc2740b3e20 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.378+0000 7fc27ab6b700 1 --2- 172.21.15.189:0/2519833955 >> v2:172.21.15.192:3300/0 conn(0x7fc27406a730 0x7fc27406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.378+0000 7fc27ab6b700 1 -- 172.21.15.189:0/2519833955 >> 172.21.15.189:0/2519833955 conn(0x7fc2740d00b0 msgr2=0x7fc2740d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:31.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.378+0000 7fc27ab6b700 1 -- 172.21.15.189:0/2519833955 shutdown_connections 2024-04-06T12:28:31.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.378+0000 7fc27ab6b700 1 -- 172.21.15.189:0/2519833955 wait complete. 2024-04-06T12:28:31.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.379+0000 7fc27ab6b700 1 Processor -- start 2024-04-06T12:28:31.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.379+0000 7fc27ab6b700 1 -- start start 2024-04-06T12:28:31.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.380+0000 7fc27ab6b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc27406a730 0x7fc2740cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:31.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.380+0000 7fc27ab6b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc27415d7b0 0x7fc274146910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:31.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.380+0000 7fc27ab6b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc27415ddf0 0x7fc27415b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:31.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.380+0000 7fc27ab6b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27415a610 con 0x7fc27415ddf0 2024-04-06T12:28:31.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.380+0000 7fc27ab6b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27415a8b0 con 0x7fc27406a730 2024-04-06T12:28:31.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.380+0000 7fc27ab6b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc27415ab90 con 0x7fc27415d7b0 2024-04-06T12:28:31.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.380+0000 7fc279108700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc27415ddf0 0x7fc27415b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:31.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.380+0000 7fc279108700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc27415ddf0 0x7fc27415b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36056/0 (socket says 172.21.15.189:36056) 2024-04-06T12:28:31.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.380+0000 7fc279108700 1 -- 172.21.15.189:0/2483613392 learned_addr learned my addr 172.21.15.189:0/2483613392 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:31.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.381+0000 7fc278907700 1 --2- 172.21.15.189:0/2483613392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc27406a730 0x7fc2740cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:31.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.381+0000 7fc273fff700 1 --2- 172.21.15.189:0/2483613392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc27415d7b0 0x7fc274146910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:31.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.381+0000 7fc279108700 1 -- 172.21.15.189:0/2483613392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc27415d7b0 msgr2=0x7fc274146910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:31.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.381+0000 7fc279108700 1 --2- 172.21.15.189:0/2483613392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc27415d7b0 0x7fc274146910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.381+0000 7fc279108700 1 -- 172.21.15.189:0/2483613392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc27406a730 msgr2=0x7fc2740cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:31.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.381+0000 7fc279108700 1 --2- 172.21.15.189:0/2483613392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc27406a730 0x7fc2740cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.381+0000 7fc279108700 1 -- 172.21.15.189:0/2483613392 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc268009550 con 0x7fc27415ddf0 2024-04-06T12:28:31.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.381+0000 7fc279108700 1 --2- 172.21.15.189:0/2483613392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc27415ddf0 0x7fc27415b670 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fc26c00cb50 tx=0x7fc26c00cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:31.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.381+0000 7fc2717fa700 1 -- 172.21.15.189:0/2483613392 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c003c00 con 0x7fc27415ddf0 2024-04-06T12:28:31.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.381+0000 7fc2717fa700 1 -- 172.21.15.189:0/2483613392 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26c003d60 con 0x7fc27415ddf0 2024-04-06T12:28:31.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.381+0000 7fc27ab6b700 1 -- 172.21.15.189:0/2483613392 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc27415ae70 con 0x7fc27415ddf0 2024-04-06T12:28:31.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.381+0000 7fc2717fa700 1 -- 172.21.15.189:0/2483613392 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c00ab70 con 0x7fc27415ddf0 2024-04-06T12:28:31.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.381+0000 7fc27ab6b700 1 -- 172.21.15.189:0/2483613392 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2740cc9f0 con 0x7fc27415ddf0 2024-04-06T12:28:31.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.382+0000 7fc27ab6b700 1 -- 172.21.15.189:0/2483613392 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc274067bb0 con 0x7fc27415ddf0 2024-04-06T12:28:31.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.382+0000 7fc2717fa700 1 -- 172.21.15.189:0/2483613392 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fc26c00acd0 con 0x7fc27415ddf0 2024-04-06T12:28:31.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.383+0000 7fc2717fa700 1 --2- 172.21.15.189:0/2483613392 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc25806adb0 0x7fc25806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:31.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.383+0000 7fc2717fa700 1 -- 172.21.15.189:0/2483613392 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5865+0+0 (secure 0 0 0) 0x7fc26c089620 con 0x7fc27415ddf0 2024-04-06T12:28:31.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.383+0000 7fc278907700 1 --2- 172.21.15.189:0/2483613392 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc25806adb0 0x7fc25806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:31.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.383+0000 7fc278907700 1 --2- 172.21.15.189:0/2483613392 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc25806adb0 0x7fc25806d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc268000c00 tx=0x7fc268008030 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:31.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.387+0000 7fc2717fa700 1 -- 172.21.15.189:0/2483613392 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc26c0545d0 con 0x7fc27415ddf0 2024-04-06T12:28:31.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.582+0000 7fc27ab6b700 1 -- 172.21.15.189:0/2483613392 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc274074990 con 0x7fc27415ddf0 2024-04-06T12:28:31.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.585+0000 7fc2717fa700 1 -- 172.21.15.189:0/2483613392 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc26c057bf0 con 0x7fc27415ddf0 2024-04-06T12:28:31.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.587+0000 7fc27ab6b700 1 -- 172.21.15.189:0/2483613392 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc25806adb0 msgr2=0x7fc25806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:31.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.587+0000 7fc27ab6b700 1 --2- 172.21.15.189:0/2483613392 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc25806adb0 0x7fc25806d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc268000c00 tx=0x7fc268008030 comp rx=0 tx=0).stop 2024-04-06T12:28:31.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.587+0000 7fc27ab6b700 1 -- 172.21.15.189:0/2483613392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc27415ddf0 msgr2=0x7fc27415b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:31.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.587+0000 7fc27ab6b700 1 --2- 172.21.15.189:0/2483613392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc27415ddf0 0x7fc27415b670 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fc26c00cb50 tx=0x7fc26c00cf10 comp rx=0 tx=0).stop 2024-04-06T12:28:31.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.587+0000 7fc27ab6b700 1 -- 172.21.15.189:0/2483613392 shutdown_connections 2024-04-06T12:28:31.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.587+0000 7fc27ab6b700 1 --2- 172.21.15.189:0/2483613392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc27415ddf0 0x7fc27415b670 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.587+0000 7fc27ab6b700 1 --2- 172.21.15.189:0/2483613392 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc25806adb0 0x7fc25806d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.587+0000 7fc27ab6b700 1 --2- 172.21.15.189:0/2483613392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc27406a730 0x7fc2740cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.587+0000 7fc27ab6b700 1 --2- 172.21.15.189:0/2483613392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc27415d7b0 0x7fc274146910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.587+0000 7fc27ab6b700 1 -- 172.21.15.189:0/2483613392 >> 172.21.15.189:0/2483613392 conn(0x7fc2740d00b0 msgr2=0x7fc2740ad390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:31.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.587+0000 7fc27ab6b700 1 -- 172.21.15.189:0/2483613392 shutdown_connections 2024-04-06T12:28:31.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.587+0000 7fc27ab6b700 1 -- 172.21.15.189:0/2483613392 wait complete. 2024-04-06T12:28:31.597 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-06T12:28:31.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.732+0000 7f13e934a700 1 Processor -- start 2024-04-06T12:28:31.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.732+0000 7f13e934a700 1 -- start start 2024-04-06T12:28:31.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.733+0000 7f13e934a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f13e415d590 0x7f13e40b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:31.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.733+0000 7f13e934a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f13e40d3890 0x7f13e40b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:31.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.733+0000 7f13e934a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e40b5cc0 con 0x7f13e40b4ed0 2024-04-06T12:28:31.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.733+0000 7f13e934a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13e40b58c0 con 0x7f13e415d590 2024-04-06T12:28:31.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.733+0000 7f13e934a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13e40d1500 con 0x7f13e40d3890 2024-04-06T12:28:31.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.733+0000 7f13e2ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f13e415d590 0x7f13e40b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:31.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.733+0000 7f13e27fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f13e40b4ed0 0x7f13e40b6580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58396/0 (socket says 172.21.15.189:58396) 2024-04-06T12:28:31.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.733+0000 7f13e2ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f13e415d590 0x7f13e40b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36058/0 (socket says 172.21.15.189:36058) 2024-04-06T12:28:31.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.733+0000 7f13e27fc700 1 -- 172.21.15.189:0/3588232913 learned_addr learned my addr 172.21.15.189:0/3588232913 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:31.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.733+0000 7f13e37fe700 1 --2- 172.21.15.189:0/3588232913 >> v2:172.21.15.192:3300/0 conn(0x7f13e40d3890 0x7f13e40b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:31.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.734+0000 7f13e2ffd700 1 -- 172.21.15.189:0/3588232913 >> v2:172.21.15.192:3300/0 conn(0x7f13e40d3890 msgr2=0x7f13e40b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:31.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.734+0000 7f13e2ffd700 1 --2- 172.21.15.189:0/3588232913 >> v2:172.21.15.192:3300/0 conn(0x7f13e40d3890 0x7f13e40b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.734+0000 7f13e2ffd700 1 -- 172.21.15.189:0/3588232913 >> v1:172.21.15.189:6789/0 conn(0x7f13e40b4ed0 legacy=0x7f13e40b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:31.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.734+0000 7f13e1ffb700 1 -- 172.21.15.189:0/3588232913 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448941815 0 0) 0x7f13e40b5cc0 con 0x7f13e40b4ed0 2024-04-06T12:28:31.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.734+0000 7f13e2ffd700 1 -- 172.21.15.189:0/3588232913 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e40d17e0 con 0x7f13e415d590 2024-04-06T12:28:31.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.734+0000 7f13e2ffd700 1 --2- 172.21.15.189:0/3588232913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13e415d590 0x7f13e40b8d70 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f13e40cf880 tx=0x7f13d8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a922b89da888063 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:31.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.734+0000 7f13e1ffb700 1 -- 172.21.15.189:0/3588232913 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d8011830 con 0x7f13e415d590 2024-04-06T12:28:31.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.734+0000 7f13e1ffb700 1 -- 172.21.15.189:0/3588232913 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d8011990 con 0x7f13e415d590 2024-04-06T12:28:31.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.734+0000 7f13e1ffb700 1 -- 172.21.15.189:0/3588232913 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d8011d20 con 0x7f13e415d590 2024-04-06T12:28:31.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.734+0000 7f13e934a700 1 -- 172.21.15.189:0/3588232913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13e415d590 msgr2=0x7f13e40b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:31.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.734+0000 7f13e934a700 1 --2- 172.21.15.189:0/3588232913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13e415d590 0x7f13e40b8d70 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f13e40cf880 tx=0x7f13d8009b20 comp rx=0 tx=0).stop 2024-04-06T12:28:31.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.734+0000 7f13e934a700 1 -- 172.21.15.189:0/3588232913 shutdown_connections 2024-04-06T12:28:31.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.734+0000 7f13e934a700 1 --2- 172.21.15.189:0/3588232913 >> v2:172.21.15.192:3300/0 conn(0x7f13e40d3890 0x7f13e40b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.734+0000 7f13e934a700 1 --2- 172.21.15.189:0/3588232913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13e415d590 0x7f13e40b8d70 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.734+0000 7f13e934a700 1 -- 172.21.15.189:0/3588232913 >> 172.21.15.189:0/3588232913 conn(0x7f13e406bbc0 msgr2=0x7f13e406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:31.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.734+0000 7f13e934a700 1 -- 172.21.15.189:0/3588232913 shutdown_connections 2024-04-06T12:28:31.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.734+0000 7f13e934a700 1 -- 172.21.15.189:0/3588232913 wait complete. 2024-04-06T12:28:31.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.736+0000 7f13e934a700 1 Processor -- start 2024-04-06T12:28:31.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.736+0000 7f13e934a700 1 -- start start 2024-04-06T12:28:31.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.736+0000 7f13e934a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13e40b4ed0 0x7f13e414fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:31.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.736+0000 7f13e934a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13e40d3890 0x7f13e415e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:31.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.736+0000 7f13e934a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13e415d590 0x7f13e4152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:31.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.736+0000 7f13e934a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e4152a50 con 0x7f13e40d3890 2024-04-06T12:28:31.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.736+0000 7f13e934a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e415fea0 con 0x7f13e415d590 2024-04-06T12:28:31.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.736+0000 7f13e934a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e4160140 con 0x7f13e40b4ed0 2024-04-06T12:28:31.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.737+0000 7f13e27fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13e40d3890 0x7f13e415e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:31.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.737+0000 7f13e27fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13e40d3890 0x7f13e415e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36072/0 (socket says 172.21.15.189:36072) 2024-04-06T12:28:31.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.737+0000 7f13e27fc700 1 -- 172.21.15.189:0/56970794 learned_addr learned my addr 172.21.15.189:0/56970794 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:31.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.737+0000 7f13e37fe700 1 --2- 172.21.15.189:0/56970794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13e415d590 0x7f13e4152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:31.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.737+0000 7f13e2ffd700 1 --2- 172.21.15.189:0/56970794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13e40b4ed0 0x7f13e414fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:31.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.737+0000 7f13e27fc700 1 -- 172.21.15.189:0/56970794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13e40b4ed0 msgr2=0x7f13e414fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:31.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.737+0000 7f13e27fc700 1 --2- 172.21.15.189:0/56970794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13e40b4ed0 0x7f13e414fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.737+0000 7f13e27fc700 1 -- 172.21.15.189:0/56970794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13e415d590 msgr2=0x7f13e4152ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:31.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.737+0000 7f13e27fc700 1 --2- 172.21.15.189:0/56970794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13e415d590 0x7f13e4152ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.737+0000 7f13e27fc700 1 -- 172.21.15.189:0/56970794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d80094a0 con 0x7f13e40d3890 2024-04-06T12:28:31.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.737+0000 7f13e37fe700 1 --2- 172.21.15.189:0/56970794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13e415d590 0x7f13e4152ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:31.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.737+0000 7f13e27fc700 1 --2- 172.21.15.189:0/56970794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13e40d3890 0x7f13e415e380 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f13d40011e0 tx=0x7f13d400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:31.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.737+0000 7f13cb7fe700 1 -- 172.21.15.189:0/56970794 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d4004c80 con 0x7f13e40d3890 2024-04-06T12:28:31.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.737+0000 7f13cb7fe700 1 -- 172.21.15.189:0/56970794 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d4004e00 con 0x7f13e40d3890 2024-04-06T12:28:31.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.737+0000 7f13e934a700 1 -- 172.21.15.189:0/56970794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e4160420 con 0x7f13e40d3890 2024-04-06T12:28:31.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.737+0000 7f13cb7fe700 1 -- 172.21.15.189:0/56970794 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d4007310 con 0x7f13e40d3890 2024-04-06T12:28:31.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.737+0000 7f13e934a700 1 -- 172.21.15.189:0/56970794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e41526e0 con 0x7f13e40d3890 2024-04-06T12:28:31.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.738+0000 7f13e934a700 1 -- 172.21.15.189:0/56970794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e4074990 con 0x7f13e40d3890 2024-04-06T12:28:31.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.739+0000 7f13cb7fe700 1 -- 172.21.15.189:0/56970794 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f13d4007470 con 0x7f13e40d3890 2024-04-06T12:28:31.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.739+0000 7f13cb7fe700 1 --2- 172.21.15.189:0/56970794 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13d006adb0 0x7f13d006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:31.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.739+0000 7f13cb7fe700 1 -- 172.21.15.189:0/56970794 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f13d40892c0 con 0x7f13e40d3890 2024-04-06T12:28:31.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.739+0000 7f13e2ffd700 1 --2- 172.21.15.189:0/56970794 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13d006adb0 0x7f13d006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:31.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.739+0000 7f13e2ffd700 1 --2- 172.21.15.189:0/56970794 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13d006adb0 0x7f13d006d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f13d80090b0 tx=0x7f13d801c040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:31.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.743+0000 7f13cb7fe700 1 -- 172.21.15.189:0/56970794 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f13d40542f0 con 0x7f13e40d3890 2024-04-06T12:28:31.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.940+0000 7f13e934a700 1 -- 172.21.15.189:0/56970794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f13e4152b90 con 0x7f13e40d3890 2024-04-06T12:28:31.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.944+0000 7f13cb7fe700 1 -- 172.21.15.189:0/56970794 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v32) v1 ==== 115+0+0 (secure 0 0 0) 0x7f13d4057910 con 0x7f13e40d3890 2024-04-06T12:28:31.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.946+0000 7f13e934a700 1 -- 172.21.15.189:0/56970794 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13d006adb0 msgr2=0x7f13d006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:31.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.946+0000 7f13e934a700 1 --2- 172.21.15.189:0/56970794 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13d006adb0 0x7f13d006d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f13d80090b0 tx=0x7f13d801c040 comp rx=0 tx=0).stop 2024-04-06T12:28:31.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.946+0000 7f13e934a700 1 -- 172.21.15.189:0/56970794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13e40d3890 msgr2=0x7f13e415e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:31.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.946+0000 7f13e934a700 1 --2- 172.21.15.189:0/56970794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13e40d3890 0x7f13e415e380 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f13d40011e0 tx=0x7f13d400cae0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.946+0000 7f13e934a700 1 -- 172.21.15.189:0/56970794 shutdown_connections 2024-04-06T12:28:31.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.946+0000 7f13e934a700 1 --2- 172.21.15.189:0/56970794 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13d006adb0 0x7f13d006d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.946+0000 7f13e934a700 1 --2- 172.21.15.189:0/56970794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13e415d590 0x7f13e4152ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.946+0000 7f13e934a700 1 --2- 172.21.15.189:0/56970794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13e40b4ed0 0x7f13e414fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.946+0000 7f13e934a700 1 --2- 172.21.15.189:0/56970794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13e40d3890 0x7f13e415e380 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:31.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.946+0000 7f13e934a700 1 -- 172.21.15.189:0/56970794 >> 172.21.15.189:0/56970794 conn(0x7f13e406bbc0 msgr2=0x7f13e40cd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:31.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.946+0000 7f13e934a700 1 -- 172.21.15.189:0/56970794 shutdown_connections 2024-04-06T12:28:31.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:31.946+0000 7f13e934a700 1 -- 172.21.15.189:0/56970794 wait complete. 2024-04-06T12:28:31.956 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-06T12:28:32.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.083+0000 7ff050903700 1 Processor -- start 2024-04-06T12:28:32.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.083+0000 7ff050903700 1 -- start start 2024-04-06T12:28:32.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.084+0000 7ff050903700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff04c15a550 0x7ff04c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:32.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.084+0000 7ff050903700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff04c0cf900 0x7ff04c0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:32.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.084+0000 7ff050903700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04c0cdbe0 con 0x7ff04c0b4ba0 2024-04-06T12:28:32.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.084+0000 7ff050903700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff04c0b5010 con 0x7ff04c0cf900 2024-04-06T12:28:32.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.084+0000 7ff050903700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c15a940 con 0x7ff04c15a550 2024-04-06T12:28:32.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.084+0000 7ff049d9b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff04c0cf900 0x7ff04c0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:32.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.084+0000 7ff04a59c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff04c15a550 0x7ff04c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:32.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.084+0000 7ff049d9b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff04c0cf900 0x7ff04c0b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:33142/0 (socket says 172.21.15.189:33142) 2024-04-06T12:28:32.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.084+0000 7ff049d9b700 1 -- 172.21.15.189:0/2775476810 learned_addr learned my addr 172.21.15.189:0/2775476810 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:32.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.084+0000 7ff04a59c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff04c15a550 0x7ff04c0cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:36690/0 (socket says 172.21.15.189:36690) 2024-04-06T12:28:32.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.084+0000 7ff04ad9d700 1 --1- 172.21.15.189:0/2775476810 >> v1:172.21.15.192:6789/0 conn(0x7ff04c0b4ba0 0x7ff04c06eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:58758/0 (socket says 172.21.15.189:58758) 2024-04-06T12:28:32.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.085+0000 7ff04959a700 1 -- 172.21.15.189:0/2775476810 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413998534 0 0) 0x7ff04c0cdbe0 con 0x7ff04c0b4ba0 2024-04-06T12:28:32.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.085+0000 7ff04959a700 1 -- 172.21.15.189:0/2775476810 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0340035b0 con 0x7ff04c0b4ba0 2024-04-06T12:28:32.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.085+0000 7ff04a59c700 1 -- 172.21.15.189:0/2775476810 >> v2:172.21.15.192:3300/0 conn(0x7ff04c0cf900 msgr2=0x7ff04c0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:32.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.085+0000 7ff04a59c700 1 --2- 172.21.15.189:0/2775476810 >> v2:172.21.15.192:3300/0 conn(0x7ff04c0cf900 0x7ff04c0b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:32.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.085+0000 7ff04a59c700 1 -- 172.21.15.189:0/2775476810 >> v1:172.21.15.192:6789/0 conn(0x7ff04c0b4ba0 legacy=0x7ff04c06eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:32.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.085+0000 7ff04a59c700 1 -- 172.21.15.189:0/2775476810 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04c15ac00 con 0x7ff04c15a550 2024-04-06T12:28:32.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.085+0000 7ff049d9b700 1 --2- 172.21.15.189:0/2775476810 >> v2:172.21.15.192:3300/0 conn(0x7ff04c0cf900 0x7ff04c0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:28:32.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.085+0000 7ff04a59c700 1 --2- 172.21.15.189:0/2775476810 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff04c15a550 0x7ff04c0cf3c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff040009840 tx=0x7ff040009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfb808a4b7a2129b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:32.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.085+0000 7ff04959a700 1 -- 172.21.15.189:0/2775476810 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff040011830 con 0x7ff04c15a550 2024-04-06T12:28:32.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.085+0000 7ff04959a700 1 -- 172.21.15.189:0/2775476810 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff040011990 con 0x7ff04c15a550 2024-04-06T12:28:32.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.085+0000 7ff04959a700 1 -- 172.21.15.189:0/2775476810 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff040011d20 con 0x7ff04c15a550 2024-04-06T12:28:32.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.086+0000 7ff050903700 1 -- 172.21.15.189:0/2775476810 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff04c15a550 msgr2=0x7ff04c0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:32.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.086+0000 7ff050903700 1 --2- 172.21.15.189:0/2775476810 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff04c15a550 0x7ff04c0cf3c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff040009840 tx=0x7ff040009b50 comp rx=0 tx=0).stop 2024-04-06T12:28:32.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.086+0000 7ff050903700 1 -- 172.21.15.189:0/2775476810 shutdown_connections 2024-04-06T12:28:32.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.086+0000 7ff050903700 1 --2- 172.21.15.189:0/2775476810 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff04c15a550 0x7ff04c0cf3c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:32.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.086+0000 7ff050903700 1 --2- 172.21.15.189:0/2775476810 >> v2:172.21.15.192:3300/0 conn(0x7ff04c0cf900 0x7ff04c0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:32.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.086+0000 7ff050903700 1 -- 172.21.15.189:0/2775476810 >> 172.21.15.189:0/2775476810 conn(0x7ff04c0d00b0 msgr2=0x7ff04c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:32.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.086+0000 7ff050903700 1 -- 172.21.15.189:0/2775476810 shutdown_connections 2024-04-06T12:28:32.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.086+0000 7ff050903700 1 -- 172.21.15.189:0/2775476810 wait complete. 2024-04-06T12:28:32.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.087+0000 7ff050903700 1 Processor -- start 2024-04-06T12:28:32.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.087+0000 7ff050903700 1 -- start start 2024-04-06T12:28:32.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.088+0000 7ff050903700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff04c0b4ba0 0x7ff04c1502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:32.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.088+0000 7ff050903700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff04c0cf900 0x7ff04c14e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:32.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.088+0000 7ff050903700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff04c142ad0 0x7ff04c142260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:32.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.088+0000 7ff050903700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c14eff0 con 0x7ff04c0b4ba0 2024-04-06T12:28:32.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.088+0000 7ff050903700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c14edf0 con 0x7ff04c142ad0 2024-04-06T12:28:32.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.088+0000 7ff050903700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c14f240 con 0x7ff04c0cf900 2024-04-06T12:28:32.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.088+0000 7ff04a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff04c0b4ba0 0x7ff04c1502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:32.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.088+0000 7ff04a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff04c0b4ba0 0x7ff04c1502e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36082/0 (socket says 172.21.15.189:36082) 2024-04-06T12:28:32.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.088+0000 7ff04a59c700 1 -- 172.21.15.189:0/2657656433 learned_addr learned my addr 172.21.15.189:0/2657656433 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:32.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.088+0000 7ff049d9b700 1 --2- 172.21.15.189:0/2657656433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff04c0cf900 0x7ff04c14e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:32.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.088+0000 7ff04ad9d700 1 --2- 172.21.15.189:0/2657656433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff04c142ad0 0x7ff04c142260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:32.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.088+0000 7ff04a59c700 1 -- 172.21.15.189:0/2657656433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff04c0cf900 msgr2=0x7ff04c14e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:32.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.088+0000 7ff04a59c700 1 --2- 172.21.15.189:0/2657656433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff04c0cf900 0x7ff04c14e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:32.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.088+0000 7ff04a59c700 1 -- 172.21.15.189:0/2657656433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff04c142ad0 msgr2=0x7ff04c142260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:32.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.088+0000 7ff04a59c700 1 --2- 172.21.15.189:0/2657656433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff04c142ad0 0x7ff04c142260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:32.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.088+0000 7ff04a59c700 1 -- 172.21.15.189:0/2657656433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03c009550 con 0x7ff04c0b4ba0 2024-04-06T12:28:32.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.088+0000 7ff049d9b700 1 --2- 172.21.15.189:0/2657656433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff04c0cf900 0x7ff04c14e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:28:32.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.089+0000 7ff04a59c700 1 --2- 172.21.15.189:0/2657656433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff04c0b4ba0 0x7ff04c1502e0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7ff040009b90 tx=0x7ff040004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:32.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.089+0000 7ff03affd700 1 -- 172.21.15.189:0/2657656433 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0400036a0 con 0x7ff04c0b4ba0 2024-04-06T12:28:32.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.089+0000 7ff03affd700 1 -- 172.21.15.189:0/2657656433 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff040003800 con 0x7ff04c0b4ba0 2024-04-06T12:28:32.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.089+0000 7ff03affd700 1 -- 172.21.15.189:0/2657656433 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff040003b90 con 0x7ff04c0b4ba0 2024-04-06T12:28:32.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.089+0000 7ff050903700 1 -- 172.21.15.189:0/2657656433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0400094a0 con 0x7ff04c0b4ba0 2024-04-06T12:28:32.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.089+0000 7ff050903700 1 -- 172.21.15.189:0/2657656433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04c14f800 con 0x7ff04c0b4ba0 2024-04-06T12:28:32.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.090+0000 7ff050903700 1 -- 172.21.15.189:0/2657656433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff04c067bb0 con 0x7ff04c0b4ba0 2024-04-06T12:28:32.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.090+0000 7ff03affd700 1 -- 172.21.15.189:0/2657656433 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7ff040003cf0 con 0x7ff04c0b4ba0 2024-04-06T12:28:32.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.091+0000 7ff03affd700 1 --2- 172.21.15.189:0/2657656433 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff02c06adb0 0x7ff02c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:32.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.091+0000 7ff049d9b700 1 --2- 172.21.15.189:0/2657656433 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff02c06adb0 0x7ff02c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:32.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.091+0000 7ff03affd700 1 -- 172.21.15.189:0/2657656433 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5865+0+0 (secure 0 0 0) 0x7ff04008a910 con 0x7ff04c0b4ba0 2024-04-06T12:28:32.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.091+0000 7ff049d9b700 1 --2- 172.21.15.189:0/2657656433 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff02c06adb0 0x7ff02c06d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff04c15d8e0 tx=0x7ff03c008040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:32.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.095+0000 7ff03affd700 1 -- 172.21.15.189:0/2657656433 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff040055930 con 0x7ff04c0b4ba0 2024-04-06T12:28:32.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.293+0000 7ff050903700 1 -- 172.21.15.189:0/2657656433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7ff04c0671d0 con 0x7ff04c0b4ba0 2024-04-06T12:28:32.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.296+0000 7ff03affd700 1 -- 172.21.15.189:0/2657656433 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v33) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff0400150b0 con 0x7ff04c0b4ba0 2024-04-06T12:28:32.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.299+0000 7ff050903700 1 -- 172.21.15.189:0/2657656433 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff02c06adb0 msgr2=0x7ff02c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:32.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.299+0000 7ff050903700 1 --2- 172.21.15.189:0/2657656433 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff02c06adb0 0x7ff02c06d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff04c15d8e0 tx=0x7ff03c008040 comp rx=0 tx=0).stop 2024-04-06T12:28:32.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.299+0000 7ff050903700 1 -- 172.21.15.189:0/2657656433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff04c0b4ba0 msgr2=0x7ff04c1502e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:32.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.299+0000 7ff050903700 1 --2- 172.21.15.189:0/2657656433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff04c0b4ba0 0x7ff04c1502e0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7ff040009b90 tx=0x7ff040004750 comp rx=0 tx=0).stop 2024-04-06T12:28:32.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.299+0000 7ff050903700 1 -- 172.21.15.189:0/2657656433 shutdown_connections 2024-04-06T12:28:32.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.299+0000 7ff050903700 1 --2- 172.21.15.189:0/2657656433 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff02c06adb0 0x7ff02c06d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:32.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.299+0000 7ff050903700 1 --2- 172.21.15.189:0/2657656433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff04c142ad0 0x7ff04c142260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:32.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.299+0000 7ff050903700 1 --2- 172.21.15.189:0/2657656433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff04c0b4ba0 0x7ff04c1502e0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:32.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.299+0000 7ff050903700 1 --2- 172.21.15.189:0/2657656433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff04c0cf900 0x7ff04c14e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:32.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.299+0000 7ff050903700 1 -- 172.21.15.189:0/2657656433 >> 172.21.15.189:0/2657656433 conn(0x7ff04c0d00b0 msgr2=0x7ff04c06ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:32.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.299+0000 7ff050903700 1 -- 172.21.15.189:0/2657656433 shutdown_connections 2024-04-06T12:28:32.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.299+0000 7ff050903700 1 -- 172.21.15.189:0/2657656433 wait complete. 2024-04-06T12:28:32.309 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2024-04-06T12:28:32.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.434+0000 7f5b997c6700 1 Processor -- start 2024-04-06T12:28:32.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.434+0000 7f5b997c6700 1 -- start start 2024-04-06T12:28:32.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.435+0000 7f5b997c6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b9415d610 0x7f5b940b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:32.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.435+0000 7f5b997c6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b9415d9f0 con 0x7f5b9406a730 2024-04-06T12:28:32.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.435+0000 7f5b997c6700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b940d1040 con 0x7f5b9406abb0 2024-04-06T12:28:32.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.435+0000 7f5b997c6700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b94161f20 con 0x7f5b9415d610 2024-04-06T12:28:32.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.435+0000 7f5b92ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b9415d610 0x7f5b940b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:32.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.435+0000 7f5b92ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b9415d610 0x7f5b940b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36092/0 (socket says 172.21.15.189:36092) 2024-04-06T12:28:32.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.435+0000 7f5b92ffd700 1 -- 172.21.15.189:0/47160295 learned_addr learned my addr 172.21.15.189:0/47160295 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:32.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.435+0000 7f5b927fc700 1 --1- 172.21.15.189:0/47160295 >> v1:172.21.15.189:6789/0 conn(0x7f5b9406a730 0x7f5b94161100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58408/0 (socket says 172.21.15.189:58408) 2024-04-06T12:28:32.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.435+0000 7f5b92ffd700 1 -- 172.21.15.189:0/47160295 >> v1:172.21.15.192:6789/0 conn(0x7f5b9406abb0 legacy=0x7f5b94161810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:32.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.435+0000 7f5b92ffd700 1 -- 172.21.15.189:0/47160295 >> v1:172.21.15.189:6789/0 conn(0x7f5b9406a730 legacy=0x7f5b94161100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:32.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.435+0000 7f5b92ffd700 1 -- 172.21.15.189:0/47160295 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b94072760 con 0x7f5b9415d610 2024-04-06T12:28:32.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.435+0000 7f5b92ffd700 1 --2- 172.21.15.189:0/47160295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9415d610 0x7f5b940b3de0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f5b880098f0 tx=0x7f5b88009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=947a44e97c9dcbd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:32.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.435+0000 7f5b91ffb700 1 -- 172.21.15.189:0/47160295 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b88004230 con 0x7f5b9415d610 2024-04-06T12:28:32.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.436+0000 7f5b91ffb700 1 -- 172.21.15.189:0/47160295 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b88004390 con 0x7f5b9415d610 2024-04-06T12:28:32.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.436+0000 7f5b91ffb700 1 -- 172.21.15.189:0/47160295 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b880036a0 con 0x7f5b9415d610 2024-04-06T12:28:32.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.436+0000 7f5b997c6700 1 -- 172.21.15.189:0/47160295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9415d610 msgr2=0x7f5b940b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:32.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.436+0000 7f5b997c6700 1 --2- 172.21.15.189:0/47160295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9415d610 0x7f5b940b3de0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f5b880098f0 tx=0x7f5b88009c00 comp rx=0 tx=0).stop 2024-04-06T12:28:32.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.436+0000 7f5b997c6700 1 -- 172.21.15.189:0/47160295 shutdown_connections 2024-04-06T12:28:32.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.436+0000 7f5b997c6700 1 --2- 172.21.15.189:0/47160295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9415d610 0x7f5b940b3de0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:32.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.436+0000 7f5b997c6700 1 -- 172.21.15.189:0/47160295 >> 172.21.15.189:0/47160295 conn(0x7f5b940d00b0 msgr2=0x7f5b940d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:32.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.436+0000 7f5b997c6700 1 -- 172.21.15.189:0/47160295 shutdown_connections 2024-04-06T12:28:32.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.436+0000 7f5b997c6700 1 -- 172.21.15.189:0/47160295 wait complete. 2024-04-06T12:28:32.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.437+0000 7f5b997c6700 1 Processor -- start 2024-04-06T12:28:32.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.437+0000 7f5b997c6700 1 -- start start 2024-04-06T12:28:32.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.438+0000 7f5b997c6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9406a730 0x7f5b94149a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:32.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.438+0000 7f5b997c6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b9406abb0 0x7f5b940ce730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:32.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.438+0000 7f5b997c6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b9415d610 0x7f5b9415af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:32.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.438+0000 7f5b997c6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b94149f40 con 0x7f5b9406a730 2024-04-06T12:28:32.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.438+0000 7f5b997c6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b940cec70 con 0x7f5b9406abb0 2024-04-06T12:28:32.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.438+0000 7f5b997c6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b940cd690 con 0x7f5b9415d610 2024-04-06T12:28:32.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.438+0000 7f5b92ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9406a730 0x7f5b94149a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:32.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.438+0000 7f5b92ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9406a730 0x7f5b94149a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36106/0 (socket says 172.21.15.189:36106) 2024-04-06T12:28:32.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.438+0000 7f5b92ffd700 1 -- 172.21.15.189:0/2453891977 learned_addr learned my addr 172.21.15.189:0/2453891977 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:32.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.439+0000 7f5b927fc700 1 --2- 172.21.15.189:0/2453891977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b9406abb0 0x7f5b940ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:32.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.439+0000 7f5b92ffd700 1 -- 172.21.15.189:0/2453891977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b9415d610 msgr2=0x7f5b9415af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:32.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.439+0000 7f5b92ffd700 1 --2- 172.21.15.189:0/2453891977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b9415d610 0x7f5b9415af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:32.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.439+0000 7f5b92ffd700 1 -- 172.21.15.189:0/2453891977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b9406abb0 msgr2=0x7f5b940ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:32.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.439+0000 7f5b92ffd700 1 --2- 172.21.15.189:0/2453891977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b9406abb0 0x7f5b940ce730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:32.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.439+0000 7f5b92ffd700 1 -- 172.21.15.189:0/2453891977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b88009550 con 0x7f5b9406a730 2024-04-06T12:28:32.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.439+0000 7f5b92ffd700 1 --2- 172.21.15.189:0/2453891977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9406a730 0x7f5b94149a00 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f5b94071460 tx=0x7f5b88004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:32.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.439+0000 7f5b7b7fe700 1 -- 172.21.15.189:0/2453891977 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8800b660 con 0x7f5b9406a730 2024-04-06T12:28:32.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.439+0000 7f5b7b7fe700 1 -- 172.21.15.189:0/2453891977 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8800b7e0 con 0x7f5b9406a730 2024-04-06T12:28:32.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.439+0000 7f5b997c6700 1 -- 172.21.15.189:0/2453891977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b940cd970 con 0x7f5b9406a730 2024-04-06T12:28:32.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.439+0000 7f5b7b7fe700 1 -- 172.21.15.189:0/2453891977 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8800bb90 con 0x7f5b9406a730 2024-04-06T12:28:32.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.439+0000 7f5b997c6700 1 -- 172.21.15.189:0/2453891977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b940cdf30 con 0x7f5b9406a730 2024-04-06T12:28:32.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.440+0000 7f5b997c6700 1 -- 172.21.15.189:0/2453891977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b94067bf0 con 0x7f5b9406a730 2024-04-06T12:28:32.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.440+0000 7f5b7b7fe700 1 -- 172.21.15.189:0/2453891977 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f5b8800bcf0 con 0x7f5b9406a730 2024-04-06T12:28:32.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.441+0000 7f5b7b7fe700 1 --2- 172.21.15.189:0/2453891977 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b8006adb0 0x7f5b8006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:32.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.441+0000 7f5b7b7fe700 1 -- 172.21.15.189:0/2453891977 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f5b8808ade0 con 0x7f5b9406a730 2024-04-06T12:28:32.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.441+0000 7f5b927fc700 1 --2- 172.21.15.189:0/2453891977 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b8006adb0 0x7f5b8006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:32.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.441+0000 7f5b927fc700 1 --2- 172.21.15.189:0/2453891977 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b8006adb0 0x7f5b8006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5b9414a7b0 tx=0x7f5b84009040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:32.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.445+0000 7f5b7b7fe700 1 -- 172.21.15.189:0/2453891977 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5b88055e00 con 0x7f5b9406a730 2024-04-06T12:28:32.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:32.721+0000 7f5b997c6700 1 -- 172.21.15.189:0/2453891977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7f5b940d1040 con 0x7f5b9406a730 2024-04-06T12:28:33.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.222+0000 7f5b7b7fe700 1 -- 172.21.15.189:0/2453891977 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v618) v1 ==== 166+0+0 (secure 0 0 0) 0x7f5b940d1040 con 0x7f5b9406a730 2024-04-06T12:28:33.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.225+0000 7f5b997c6700 1 -- 172.21.15.189:0/2453891977 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b8006adb0 msgr2=0x7f5b8006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:33.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.225+0000 7f5b997c6700 1 --2- 172.21.15.189:0/2453891977 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b8006adb0 0x7f5b8006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5b9414a7b0 tx=0x7f5b84009040 comp rx=0 tx=0).stop 2024-04-06T12:28:33.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.225+0000 7f5b997c6700 1 -- 172.21.15.189:0/2453891977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9406a730 msgr2=0x7f5b94149a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:33.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.225+0000 7f5b997c6700 1 --2- 172.21.15.189:0/2453891977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9406a730 0x7f5b94149a00 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f5b94071460 tx=0x7f5b88004750 comp rx=0 tx=0).stop 2024-04-06T12:28:33.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.225+0000 7f5b997c6700 1 -- 172.21.15.189:0/2453891977 shutdown_connections 2024-04-06T12:28:33.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.225+0000 7f5b997c6700 1 --2- 172.21.15.189:0/2453891977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b9415d610 0x7f5b9415af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:33.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.225+0000 7f5b997c6700 1 --2- 172.21.15.189:0/2453891977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9406a730 0x7f5b94149a00 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:33.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.225+0000 7f5b997c6700 1 --2- 172.21.15.189:0/2453891977 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5b8006adb0 0x7f5b8006d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:33.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.225+0000 7f5b997c6700 1 --2- 172.21.15.189:0/2453891977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b9406abb0 0x7f5b940ce730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:33.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.225+0000 7f5b997c6700 1 -- 172.21.15.189:0/2453891977 >> 172.21.15.189:0/2453891977 conn(0x7f5b940d00b0 msgr2=0x7f5b940ae8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:33.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.225+0000 7f5b997c6700 1 -- 172.21.15.189:0/2453891977 shutdown_connections 2024-04-06T12:28:33.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.225+0000 7f5b997c6700 1 -- 172.21.15.189:0/2453891977 wait complete. 2024-04-06T12:28:33.236 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2024-04-06T12:28:33.236 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that a non-existent path leads to schedule deactivation after a few retries. 2024-04-06T12:28:33.236 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-06T12:28:33.236 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2024-04-06T12:28:33.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.365+0000 7efea8015700 1 Processor -- start 2024-04-06T12:28:33.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.366+0000 7efea8015700 1 -- start start 2024-04-06T12:28:33.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.367+0000 7efea8015700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efea015a550 0x7efea00cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:33.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.367+0000 7efea8015700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efea00cf900 0x7efea00b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:33.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.367+0000 7efea8015700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efea00cdbe0 con 0x7efea00b4ba0 2024-04-06T12:28:33.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.367+0000 7efea8015700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efea015a940 con 0x7efea015a550 2024-04-06T12:28:33.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.367+0000 7efea8015700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efea015ac00 con 0x7efea00cf900 2024-04-06T12:28:33.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.367+0000 7efea5db1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efea015a550 0x7efea00cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:33.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.367+0000 7efea5db1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efea015a550 0x7efea00cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36116/0 (socket says 172.21.15.189:36116) 2024-04-06T12:28:33.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.367+0000 7efea65b2700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efea00b4ba0 0x7efea006eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58422/0 (socket says 172.21.15.189:58422) 2024-04-06T12:28:33.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.367+0000 7efea5db1700 1 -- 172.21.15.189:0/2589091769 learned_addr learned my addr 172.21.15.189:0/2589091769 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:33.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.367+0000 7efea55b0700 1 --2- 172.21.15.189:0/2589091769 >> v2:172.21.15.192:3300/0 conn(0x7efea00cf900 0x7efea00b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:33.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.367+0000 7efea5db1700 1 -- 172.21.15.189:0/2589091769 >> v2:172.21.15.192:3300/0 conn(0x7efea00cf900 msgr2=0x7efea00b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:33.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.367+0000 7efea5db1700 1 --2- 172.21.15.189:0/2589091769 >> v2:172.21.15.192:3300/0 conn(0x7efea00cf900 0x7efea00b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:33.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.367+0000 7efea5db1700 1 -- 172.21.15.189:0/2589091769 >> v1:172.21.15.189:6789/0 conn(0x7efea00b4ba0 legacy=0x7efea006eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:33.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.367+0000 7efea5db1700 1 -- 172.21.15.189:0/2589091769 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea015aee0 con 0x7efea015a550 2024-04-06T12:28:33.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.367+0000 7efea5db1700 1 --2- 172.21.15.189:0/2589091769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea015a550 0x7efea00cf3c0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7efea00b2d40 tx=0x7efe90009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3cb522463d0df9b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:33.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.367+0000 7efea4daf700 1 -- 172.21.15.189:0/2589091769 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9000b870 con 0x7efea015a550 2024-04-06T12:28:33.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.367+0000 7efea4daf700 1 -- 172.21.15.189:0/2589091769 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9000b9d0 con 0x7efea015a550 2024-04-06T12:28:33.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.367+0000 7efea4daf700 1 -- 172.21.15.189:0/2589091769 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9000bd60 con 0x7efea015a550 2024-04-06T12:28:33.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.368+0000 7efea8015700 1 -- 172.21.15.189:0/2589091769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea015a550 msgr2=0x7efea00cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:33.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.368+0000 7efea8015700 1 --2- 172.21.15.189:0/2589091769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea015a550 0x7efea00cf3c0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7efea00b2d40 tx=0x7efe90009bd0 comp rx=0 tx=0).stop 2024-04-06T12:28:33.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.368+0000 7efea8015700 1 -- 172.21.15.189:0/2589091769 shutdown_connections 2024-04-06T12:28:33.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.368+0000 7efea8015700 1 --2- 172.21.15.189:0/2589091769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea015a550 0x7efea00cf3c0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:33.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.368+0000 7efea8015700 1 --2- 172.21.15.189:0/2589091769 >> v2:172.21.15.192:3300/0 conn(0x7efea00cf900 0x7efea00b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:33.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.368+0000 7efea8015700 1 -- 172.21.15.189:0/2589091769 >> 172.21.15.189:0/2589091769 conn(0x7efea00d00b0 msgr2=0x7efea00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:33.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.368+0000 7efea8015700 1 -- 172.21.15.189:0/2589091769 shutdown_connections 2024-04-06T12:28:33.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.368+0000 7efea8015700 1 -- 172.21.15.189:0/2589091769 wait complete. 2024-04-06T12:28:33.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.369+0000 7efea8015700 1 Processor -- start 2024-04-06T12:28:33.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.369+0000 7efea8015700 1 -- start start 2024-04-06T12:28:33.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.370+0000 7efea8015700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea00b4ba0 0x7efea01f6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:33.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.370+0000 7efea8015700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efea00cf900 0x7efea015b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:33.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.370+0000 7efea8015700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efea01462a0 0x7efea01583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:33.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.370+0000 7efea8015700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea015c220 con 0x7efea00b4ba0 2024-04-06T12:28:33.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.370+0000 7efea8015700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea00adb50 con 0x7efea00cf900 2024-04-06T12:28:33.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.370+0000 7efea8015700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea00ad2b0 con 0x7efea01462a0 2024-04-06T12:28:33.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.370+0000 7efea5db1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea00b4ba0 0x7efea01f6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:33.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.370+0000 7efea5db1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea00b4ba0 0x7efea01f6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36118/0 (socket says 172.21.15.189:36118) 2024-04-06T12:28:33.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.370+0000 7efea5db1700 1 -- 172.21.15.189:0/1566114798 learned_addr learned my addr 172.21.15.189:0/1566114798 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:33.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.370+0000 7efea5db1700 1 -- 172.21.15.189:0/1566114798 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efea01462a0 msgr2=0x7efea01583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:33.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.370+0000 7efea5db1700 1 --2- 172.21.15.189:0/1566114798 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efea01462a0 0x7efea01583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:33.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.370+0000 7efea5db1700 1 -- 172.21.15.189:0/1566114798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efea00cf900 msgr2=0x7efea015b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:33.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.370+0000 7efea5db1700 1 --2- 172.21.15.189:0/1566114798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efea00cf900 0x7efea015b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:33.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.370+0000 7efea5db1700 1 -- 172.21.15.189:0/1566114798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe90009550 con 0x7efea00b4ba0 2024-04-06T12:28:33.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.371+0000 7efea5db1700 1 --2- 172.21.15.189:0/1566114798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea00b4ba0 0x7efea01f6c00 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7efe90000c00 tx=0x7efe90004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:33.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.371+0000 7efe967fc700 1 -- 172.21.15.189:0/1566114798 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe90004520 con 0x7efea00b4ba0 2024-04-06T12:28:33.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.371+0000 7efe967fc700 1 -- 172.21.15.189:0/1566114798 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9001e070 con 0x7efea00b4ba0 2024-04-06T12:28:33.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.371+0000 7efea8015700 1 -- 172.21.15.189:0/1566114798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea00ad590 con 0x7efea00b4ba0 2024-04-06T12:28:33.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.371+0000 7efe967fc700 1 -- 172.21.15.189:0/1566114798 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe900038a0 con 0x7efea00b4ba0 2024-04-06T12:28:33.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.371+0000 7efea8015700 1 -- 172.21.15.189:0/1566114798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea015be40 con 0x7efea00b4ba0 2024-04-06T12:28:33.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.372+0000 7efea8015700 1 -- 172.21.15.189:0/1566114798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea015c360 con 0x7efea00b4ba0 2024-04-06T12:28:33.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.372+0000 7efe967fc700 1 -- 172.21.15.189:0/1566114798 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7efe90003a00 con 0x7efea00b4ba0 2024-04-06T12:28:33.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.373+0000 7efe967fc700 1 --2- 172.21.15.189:0/1566114798 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efe8406adb0 0x7efe8406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:33.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.373+0000 7efe967fc700 1 -- 172.21.15.189:0/1566114798 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5865+0+0 (secure 0 0 0) 0x7efe90020070 con 0x7efea00b4ba0 2024-04-06T12:28:33.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.373+0000 7efea55b0700 1 --2- 172.21.15.189:0/1566114798 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efe8406adb0 0x7efe8406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:33.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.373+0000 7efea55b0700 1 --2- 172.21.15.189:0/1566114798 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efe8406adb0 0x7efe8406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efea00b0ec0 tx=0x7efe9c008040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:33.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.377+0000 7efe967fc700 1 -- 172.21.15.189:0/1566114798 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efea015c360 con 0x7efea00b4ba0 2024-04-06T12:28:33.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:33.651+0000 7efea8015700 1 -- 172.21.15.189:0/1566114798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7efea015c360 con 0x7efea00b4ba0 2024-04-06T12:28:34.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.226+0000 7efe967fc700 1 -- 172.21.15.189:0/1566114798 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v619) v1 ==== 186+0+0 (secure 0 0 0) 0x7efea015c360 con 0x7efea00b4ba0 2024-04-06T12:28:34.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.228+0000 7efea8015700 1 -- 172.21.15.189:0/1566114798 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efe8406adb0 msgr2=0x7efe8406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:34.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.228+0000 7efea8015700 1 --2- 172.21.15.189:0/1566114798 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efe8406adb0 0x7efe8406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efea00b0ec0 tx=0x7efe9c008040 comp rx=0 tx=0).stop 2024-04-06T12:28:34.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.228+0000 7efea8015700 1 -- 172.21.15.189:0/1566114798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea00b4ba0 msgr2=0x7efea01f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:34.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.228+0000 7efea8015700 1 --2- 172.21.15.189:0/1566114798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea00b4ba0 0x7efea01f6c00 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7efe90000c00 tx=0x7efe90004750 comp rx=0 tx=0).stop 2024-04-06T12:28:34.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.228+0000 7efea8015700 1 -- 172.21.15.189:0/1566114798 shutdown_connections 2024-04-06T12:28:34.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.229+0000 7efea8015700 1 --2- 172.21.15.189:0/1566114798 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efea01462a0 0x7efea01583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:34.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.229+0000 7efea8015700 1 --2- 172.21.15.189:0/1566114798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea00b4ba0 0x7efea01f6c00 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:34.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.229+0000 7efea8015700 1 --2- 172.21.15.189:0/1566114798 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efe8406adb0 0x7efe8406d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:34.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.229+0000 7efea8015700 1 --2- 172.21.15.189:0/1566114798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efea00cf900 0x7efea015b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:34.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.229+0000 7efea8015700 1 -- 172.21.15.189:0/1566114798 >> 172.21.15.189:0/1566114798 conn(0x7efea00d00b0 msgr2=0x7efea00b1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:34.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.229+0000 7efea8015700 1 -- 172.21.15.189:0/1566114798 shutdown_connections 2024-04-06T12:28:34.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.229+0000 7efea8015700 1 -- 172.21.15.189:0/1566114798 wait complete. 2024-04-06T12:28:34.239 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-06T12:28:34.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.366+0000 7f60a44ca700 1 Processor -- start 2024-04-06T12:28:34.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.366+0000 7f60a44ca700 1 -- start start 2024-04-06T12:28:34.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.367+0000 7f60a44ca700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f609c0b4660 0x7f609c0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:34.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.367+0000 7f60a44ca700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f609c0b2fc0 0x7f609c0b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:34.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.367+0000 7f60a44ca700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f609c0b5030 con 0x7f609c15a550 2024-04-06T12:28:34.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.367+0000 7f60a44ca700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f609c0cdbe0 con 0x7f609c0b2fc0 2024-04-06T12:28:34.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.367+0000 7f60a44ca700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f609c15a940 con 0x7f609c0b4660 2024-04-06T12:28:34.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.367+0000 7f60a2a67700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f609c0b2fc0 0x7f609c0b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:34.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.367+0000 7f60a2a67700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f609c0b2fc0 0x7f609c0b33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36120/0 (socket says 172.21.15.189:36120) 2024-04-06T12:28:34.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.367+0000 7f60a2a67700 1 -- 172.21.15.189:0/2046648316 learned_addr learned my addr 172.21.15.189:0/2046648316 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:34.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.368+0000 7f60a1a65700 1 --2- 172.21.15.189:0/2046648316 >> v2:172.21.15.192:3300/0 conn(0x7f609c0b4660 0x7f609c0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:34.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.368+0000 7f60a2a67700 1 -- 172.21.15.189:0/2046648316 >> v2:172.21.15.192:3300/0 conn(0x7f609c0b4660 msgr2=0x7f609c0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:34.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.368+0000 7f60a2a67700 1 --2- 172.21.15.189:0/2046648316 >> v2:172.21.15.192:3300/0 conn(0x7f609c0b4660 0x7f609c0b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:34.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.368+0000 7f60a2a67700 1 -- 172.21.15.189:0/2046648316 >> v1:172.21.15.189:6789/0 conn(0x7f609c15a550 legacy=0x7f609c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:34.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.368+0000 7f60a2a67700 1 -- 172.21.15.189:0/2046648316 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609c15ac00 con 0x7f609c0b2fc0 2024-04-06T12:28:34.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.368+0000 7f60a2a67700 1 --2- 172.21.15.189:0/2046648316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f609c0b2fc0 0x7f609c0b33d0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f609c0b2d40 tx=0x7f609400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc520990eab61b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:34.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.368+0000 7f60a1264700 1 -- 172.21.15.189:0/2046648316 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609400e070 con 0x7f609c0b2fc0 2024-04-06T12:28:34.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.368+0000 7f60a1264700 1 -- 172.21.15.189:0/2046648316 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609400b9c0 con 0x7f609c0b2fc0 2024-04-06T12:28:34.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.368+0000 7f60a1264700 1 -- 172.21.15.189:0/2046648316 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609400bd70 con 0x7f609c0b2fc0 2024-04-06T12:28:34.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.369+0000 7f60a44ca700 1 -- 172.21.15.189:0/2046648316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f609c0b2fc0 msgr2=0x7f609c0b33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:34.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.369+0000 7f60a44ca700 1 --2- 172.21.15.189:0/2046648316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f609c0b2fc0 0x7f609c0b33d0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f609c0b2d40 tx=0x7f609400b330 comp rx=0 tx=0).stop 2024-04-06T12:28:34.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.369+0000 7f60a44ca700 1 -- 172.21.15.189:0/2046648316 shutdown_connections 2024-04-06T12:28:34.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.369+0000 7f60a44ca700 1 --2- 172.21.15.189:0/2046648316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f609c0b2fc0 0x7f609c0b33d0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:34.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.369+0000 7f60a44ca700 1 --2- 172.21.15.189:0/2046648316 >> v2:172.21.15.192:3300/0 conn(0x7f609c0b4660 0x7f609c0b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:34.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.369+0000 7f60a44ca700 1 -- 172.21.15.189:0/2046648316 >> 172.21.15.189:0/2046648316 conn(0x7f609c0d00b0 msgr2=0x7f609c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:34.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.369+0000 7f60a44ca700 1 -- 172.21.15.189:0/2046648316 shutdown_connections 2024-04-06T12:28:34.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.369+0000 7f60a44ca700 1 -- 172.21.15.189:0/2046648316 wait complete. 2024-04-06T12:28:34.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.370+0000 7f60a44ca700 1 Processor -- start 2024-04-06T12:28:34.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.370+0000 7f60a44ca700 1 -- start start 2024-04-06T12:28:34.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.371+0000 7f60a44ca700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f609c0b2fc0 0x7f609c0a8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:34.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.371+0000 7f60a44ca700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f609c0b4660 0x7f609c152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:34.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.371+0000 7f60a44ca700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f609c0ce8f0 0x7f609c0a9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:34.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.371+0000 7f60a44ca700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c15c020 con 0x7f609c0ce8f0 2024-04-06T12:28:34.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.371+0000 7f60a44ca700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c152ff0 con 0x7f609c0b2fc0 2024-04-06T12:28:34.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.371+0000 7f60a44ca700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f609c0a9780 con 0x7f609c0b4660 2024-04-06T12:28:34.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.371+0000 7f60a2a67700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f609c0ce8f0 0x7f609c0a9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:34.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.371+0000 7f60a2a67700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f609c0ce8f0 0x7f609c0a9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36128/0 (socket says 172.21.15.189:36128) 2024-04-06T12:28:34.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.371+0000 7f60a2a67700 1 -- 172.21.15.189:0/1497805951 learned_addr learned my addr 172.21.15.189:0/1497805951 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:34.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.372+0000 7f60a2a67700 1 -- 172.21.15.189:0/1497805951 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f609c0b4660 msgr2=0x7f609c152ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:34.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.372+0000 7f60a2266700 1 --2- 172.21.15.189:0/1497805951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f609c0b2fc0 0x7f609c0a8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:34.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.372+0000 7f60a2a67700 1 --2- 172.21.15.189:0/1497805951 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f609c0b4660 0x7f609c152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:34.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.372+0000 7f60a2a67700 1 -- 172.21.15.189:0/1497805951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f609c0b2fc0 msgr2=0x7f609c0a8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:34.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.372+0000 7f60a2a67700 1 --2- 172.21.15.189:0/1497805951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f609c0b2fc0 0x7f609c0a8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:34.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.372+0000 7f60a2a67700 1 -- 172.21.15.189:0/1497805951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6094009b30 con 0x7f609c0ce8f0 2024-04-06T12:28:34.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.372+0000 7f60a2a67700 1 --2- 172.21.15.189:0/1497805951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f609c0ce8f0 0x7f609c0a9ba0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f6094004270 tx=0x7f6094004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:34.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.372+0000 7f6092ffd700 1 -- 172.21.15.189:0/1497805951 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60940099b0 con 0x7f609c0ce8f0 2024-04-06T12:28:34.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.372+0000 7f6092ffd700 1 -- 172.21.15.189:0/1497805951 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60940092e0 con 0x7f609c0ce8f0 2024-04-06T12:28:34.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.372+0000 7f60a44ca700 1 -- 172.21.15.189:0/1497805951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f609c0a94a0 con 0x7f609c0ce8f0 2024-04-06T12:28:34.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.372+0000 7f6092ffd700 1 -- 172.21.15.189:0/1497805951 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609401b630 con 0x7f609c0ce8f0 2024-04-06T12:28:34.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.372+0000 7f60a44ca700 1 -- 172.21.15.189:0/1497805951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609c15bc40 con 0x7f609c0ce8f0 2024-04-06T12:28:34.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.373+0000 7f60a44ca700 1 -- 172.21.15.189:0/1497805951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f609c15c160 con 0x7f609c0ce8f0 2024-04-06T12:28:34.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.373+0000 7f6092ffd700 1 -- 172.21.15.189:0/1497805951 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f609401b790 con 0x7f609c0ce8f0 2024-04-06T12:28:34.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.374+0000 7f6092ffd700 1 --2- 172.21.15.189:0/1497805951 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f608006adb0 0x7f608006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:34.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.374+0000 7f6092ffd700 1 -- 172.21.15.189:0/1497805951 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f609408acf0 con 0x7f609c0ce8f0 2024-04-06T12:28:34.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.374+0000 7f60a2266700 1 --2- 172.21.15.189:0/1497805951 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f608006adb0 0x7f608006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:34.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.374+0000 7f60a2266700 1 --2- 172.21.15.189:0/1497805951 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f608006adb0 0x7f608006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f608c000f60 tx=0x7f608c009040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:34.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.378+0000 7f6092ffd700 1 -- 172.21.15.189:0/1497805951 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f609c15c160 con 0x7f609c0ce8f0 2024-04-06T12:28:34.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.572+0000 7f60a44ca700 1 -- 172.21.15.189:0/1497805951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f609c15c160 con 0x7f609c0ce8f0 2024-04-06T12:28:34.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.576+0000 7f6092ffd700 1 -- 172.21.15.189:0/1497805951 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f609c15c160 con 0x7f609c0ce8f0 2024-04-06T12:28:34.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.578+0000 7f60a44ca700 1 -- 172.21.15.189:0/1497805951 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f608006adb0 msgr2=0x7f608006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:34.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.578+0000 7f60a44ca700 1 --2- 172.21.15.189:0/1497805951 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f608006adb0 0x7f608006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f608c000f60 tx=0x7f608c009040 comp rx=0 tx=0).stop 2024-04-06T12:28:34.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.578+0000 7f60a44ca700 1 -- 172.21.15.189:0/1497805951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f609c0ce8f0 msgr2=0x7f609c0a9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:34.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.578+0000 7f60a44ca700 1 --2- 172.21.15.189:0/1497805951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f609c0ce8f0 0x7f609c0a9ba0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f6094004270 tx=0x7f6094004ad0 comp rx=0 tx=0).stop 2024-04-06T12:28:34.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.578+0000 7f60a44ca700 1 -- 172.21.15.189:0/1497805951 shutdown_connections 2024-04-06T12:28:34.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.578+0000 7f60a44ca700 1 --2- 172.21.15.189:0/1497805951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f609c0ce8f0 0x7f609c0a9ba0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:34.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.578+0000 7f60a44ca700 1 --2- 172.21.15.189:0/1497805951 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f608006adb0 0x7f608006d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:34.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.578+0000 7f60a44ca700 1 --2- 172.21.15.189:0/1497805951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f609c0b2fc0 0x7f609c0a8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:34.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.578+0000 7f60a44ca700 1 --2- 172.21.15.189:0/1497805951 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f609c0b4660 0x7f609c152ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:34.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.578+0000 7f60a44ca700 1 -- 172.21.15.189:0/1497805951 >> 172.21.15.189:0/1497805951 conn(0x7f609c0d00b0 msgr2=0x7f609c158350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:34.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.578+0000 7f60a44ca700 1 -- 172.21.15.189:0/1497805951 shutdown_connections 2024-04-06T12:28:34.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.578+0000 7f60a44ca700 1 -- 172.21.15.189:0/1497805951 wait complete. 2024-04-06T12:28:34.587 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T12:28:34.587 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:28:34.631 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:28:34.680 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T12:28:34.680 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:28:34.680 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:28:34.698 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:28:34.698 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:28:34.698 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:28:34.718 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:28:34.718 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:28:34.718 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:28:34.755 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:28:34.755 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:28:34.756 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:28:34.777 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:28:34.777 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-06T12:28:34.778 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:28:34.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.927+0000 7f2a1d5a0700 1 Processor -- start 2024-04-06T12:28:34.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.927+0000 7f2a1d5a0700 1 -- start start 2024-04-06T12:28:34.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.928+0000 7f2a1d5a0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a1806ae40 0x7f2a180b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:34.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.928+0000 7f2a1d5a0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2a1815ddf0 0x7f2a1815e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:34.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.928+0000 7f2a1d5a0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a180b3990 con 0x7f2a1815d7b0 2024-04-06T12:28:34.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.928+0000 7f2a1d5a0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a180b3060 con 0x7f2a1815ddf0 2024-04-06T12:28:34.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.928+0000 7f2a1d5a0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a180b3340 con 0x7f2a1806ae40 2024-04-06T12:28:34.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.928+0000 7f2a177fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2a1815ddf0 0x7f2a1815e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:34.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.928+0000 7f2a177fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2a1815ddf0 0x7f2a1815e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36144/0 (socket says 172.21.15.189:36144) 2024-04-06T12:28:34.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.928+0000 7f2a177fe700 1 -- 172.21.15.189:0/2676317483 learned_addr learned my addr 172.21.15.189:0/2676317483 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:34.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.928+0000 7f2a177fe700 1 -- 172.21.15.189:0/2676317483 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a1806ae40 msgr2=0x7f2a180b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:34.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.928+0000 7f2a167fc700 1 --2- 172.21.15.189:0/2676317483 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a1806ae40 0x7f2a180b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:34.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.928+0000 7f2a177fe700 1 --2- 172.21.15.189:0/2676317483 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a1806ae40 0x7f2a180b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:34.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.928+0000 7f2a177fe700 1 -- 172.21.15.189:0/2676317483 >> v1:172.21.15.192:6789/0 conn(0x7f2a1815d7b0 legacy=0x7f2a1806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:34.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.929+0000 7f2a177fe700 1 -- 172.21.15.189:0/2676317483 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a180b3620 con 0x7f2a1815ddf0 2024-04-06T12:28:34.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.929+0000 7f2a177fe700 1 --2- 172.21.15.189:0/2676317483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a1815ddf0 0x7f2a1815e2a0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f2a10009e50 tx=0x7f2a1000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b0deee7136d225 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:34.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.929+0000 7f2a15ffb700 1 -- 172.21.15.189:0/2676317483 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1000e070 con 0x7f2a1815ddf0 2024-04-06T12:28:34.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.929+0000 7f2a15ffb700 1 -- 172.21.15.189:0/2676317483 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1000b9c0 con 0x7f2a1815ddf0 2024-04-06T12:28:34.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.929+0000 7f2a15ffb700 1 -- 172.21.15.189:0/2676317483 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1000bd70 con 0x7f2a1815ddf0 2024-04-06T12:28:34.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.929+0000 7f2a1d5a0700 1 -- 172.21.15.189:0/2676317483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a1815ddf0 msgr2=0x7f2a1815e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:34.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.929+0000 7f2a1d5a0700 1 --2- 172.21.15.189:0/2676317483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a1815ddf0 0x7f2a1815e2a0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f2a10009e50 tx=0x7f2a1000b330 comp rx=0 tx=0).stop 2024-04-06T12:28:34.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.929+0000 7f2a1d5a0700 1 -- 172.21.15.189:0/2676317483 shutdown_connections 2024-04-06T12:28:34.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.929+0000 7f2a1d5a0700 1 --2- 172.21.15.189:0/2676317483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a1815ddf0 0x7f2a1815e2a0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:34.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.929+0000 7f2a1d5a0700 1 --2- 172.21.15.189:0/2676317483 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a1806ae40 0x7f2a180b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:34.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.929+0000 7f2a1d5a0700 1 -- 172.21.15.189:0/2676317483 >> 172.21.15.189:0/2676317483 conn(0x7f2a180d00b0 msgr2=0x7f2a180d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:34.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.929+0000 7f2a1d5a0700 1 -- 172.21.15.189:0/2676317483 shutdown_connections 2024-04-06T12:28:34.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.929+0000 7f2a1d5a0700 1 -- 172.21.15.189:0/2676317483 wait complete. 2024-04-06T12:28:34.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.931+0000 7f2a1d5a0700 1 Processor -- start 2024-04-06T12:28:34.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.931+0000 7f2a1d5a0700 1 -- start start 2024-04-06T12:28:34.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a1d5a0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a1806ae40 0x7f2a180c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:34.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a1d5a0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a1815d7b0 0x7f2a181f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:34.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a1d5a0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a1815ddf0 0x7f2a181f5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:34.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a1d5a0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a180b0b80 con 0x7f2a1806ae40 2024-04-06T12:28:34.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a1d5a0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a180c5470 con 0x7f2a1815ddf0 2024-04-06T12:28:34.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a1d5a0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a180b0d90 con 0x7f2a1815d7b0 2024-04-06T12:28:34.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a16ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a1806ae40 0x7f2a180c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:34.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a16ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a1806ae40 0x7f2a180c4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36150/0 (socket says 172.21.15.189:36150) 2024-04-06T12:28:34.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a16ffd700 1 -- 172.21.15.189:0/3756331382 learned_addr learned my addr 172.21.15.189:0/3756331382 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:34.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a167fc700 1 --2- 172.21.15.189:0/3756331382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a1815d7b0 0x7f2a181f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:34.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a177fe700 1 --2- 172.21.15.189:0/3756331382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a1815ddf0 0x7f2a181f5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:34.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a16ffd700 1 -- 172.21.15.189:0/3756331382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a1815d7b0 msgr2=0x7f2a181f2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:34.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a16ffd700 1 --2- 172.21.15.189:0/3756331382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a1815d7b0 0x7f2a181f2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:34.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a16ffd700 1 -- 172.21.15.189:0/3756331382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a1815ddf0 msgr2=0x7f2a181f5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:34.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a16ffd700 1 --2- 172.21.15.189:0/3756331382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a1815ddf0 0x7f2a181f5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:34.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a16ffd700 1 -- 172.21.15.189:0/3756331382 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a10009b30 con 0x7f2a1806ae40 2024-04-06T12:28:34.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a167fc700 1 --2- 172.21.15.189:0/3756331382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a1815d7b0 0x7f2a181f2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:34.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a177fe700 1 --2- 172.21.15.189:0/3756331382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a1815ddf0 0x7f2a181f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:34.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a16ffd700 1 --2- 172.21.15.189:0/3756331382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a1806ae40 0x7f2a180c4f30 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c00c4e0 tx=0x7f2a0c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:34.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f29ff7fe700 1 -- 172.21.15.189:0/3756331382 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c0052d0 con 0x7f2a1806ae40 2024-04-06T12:28:34.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f29ff7fe700 1 -- 172.21.15.189:0/3756331382 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0c005430 con 0x7f2a1806ae40 2024-04-06T12:28:34.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a1d5a0700 1 -- 172.21.15.189:0/3756331382 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a180b1070 con 0x7f2a1806ae40 2024-04-06T12:28:34.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f29ff7fe700 1 -- 172.21.15.189:0/3756331382 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c0038c0 con 0x7f2a1806ae40 2024-04-06T12:28:34.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.932+0000 7f2a1d5a0700 1 -- 172.21.15.189:0/3756331382 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a180b1400 con 0x7f2a1806ae40 2024-04-06T12:28:34.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.933+0000 7f2a1d5a0700 1 -- 172.21.15.189:0/3756331382 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a18067bb0 con 0x7f2a1806ae40 2024-04-06T12:28:34.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.934+0000 7f29ff7fe700 1 -- 172.21.15.189:0/3756331382 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f2a0c003a20 con 0x7f2a1806ae40 2024-04-06T12:28:34.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.934+0000 7f29ff7fe700 1 --2- 172.21.15.189:0/3756331382 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2a0406adb0 0x7f2a0406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:34.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.934+0000 7f29ff7fe700 1 -- 172.21.15.189:0/3756331382 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f2a0c088970 con 0x7f2a1806ae40 2024-04-06T12:28:34.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.934+0000 7f2a167fc700 1 --2- 172.21.15.189:0/3756331382 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2a0406adb0 0x7f2a0406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:34.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.934+0000 7f2a167fc700 1 --2- 172.21.15.189:0/3756331382 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2a0406adb0 0x7f2a0406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2a1814f700 tx=0x7f2a08008040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:34.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:34.938+0000 7f29ff7fe700 1 -- 172.21.15.189:0/3756331382 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2a0c053a30 con 0x7f2a1806ae40 2024-04-06T12:28:35.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.178+0000 7f2a1d5a0700 1 -- 172.21.15.189:0/3756331382 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a18074990 con 0x7f2a1806ae40 2024-04-06T12:28:35.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.178+0000 7f29ff7fe700 1 -- 172.21.15.189:0/3756331382 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+3538 (secure 0 0 0) 0x7f2a0c017090 con 0x7f2a1806ae40 2024-04-06T12:28:35.176 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:35.176 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":41,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3065963154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3065963154},{"type":"v1","addr":"172.21.15.192:6833","nonce":3065963154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":6071,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/831568588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":831568588},{"type":"v1","addr":"172.21.15.192:6835","nonce":831568588}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":7080,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1688936113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1688936113},{"type":"v1","addr":"172.21.15.189:6836","nonce":1688936113}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34},{"gid":7485,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1879104934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1879104934},{"type":"v1","addr":"172.21.15.189:6837","nonce":1879104934}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[]} 2024-04-06T12:28:35.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.181+0000 7f2a1d5a0700 1 -- 172.21.15.189:0/3756331382 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2a0406adb0 msgr2=0x7f2a0406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:35.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.181+0000 7f2a1d5a0700 1 --2- 172.21.15.189:0/3756331382 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2a0406adb0 0x7f2a0406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2a1814f700 tx=0x7f2a08008040 comp rx=0 tx=0).stop 2024-04-06T12:28:35.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.181+0000 7f2a1d5a0700 1 -- 172.21.15.189:0/3756331382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a1806ae40 msgr2=0x7f2a180c4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:35.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.181+0000 7f2a1d5a0700 1 --2- 172.21.15.189:0/3756331382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a1806ae40 0x7f2a180c4f30 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c00c4e0 tx=0x7f2a0c00c8a0 comp rx=0 tx=0).stop 2024-04-06T12:28:35.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.181+0000 7f2a1d5a0700 1 -- 172.21.15.189:0/3756331382 shutdown_connections 2024-04-06T12:28:35.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.181+0000 7f2a1d5a0700 1 --2- 172.21.15.189:0/3756331382 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2a0406adb0 0x7f2a0406d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:35.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.181+0000 7f2a1d5a0700 1 --2- 172.21.15.189:0/3756331382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a1815ddf0 0x7f2a181f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:35.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.181+0000 7f2a1d5a0700 1 --2- 172.21.15.189:0/3756331382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a1806ae40 0x7f2a180c4f30 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.181+0000 7f2a1d5a0700 1 --2- 172.21.15.189:0/3756331382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a1815d7b0 0x7f2a181f2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.181+0000 7f2a1d5a0700 1 -- 172.21.15.189:0/3756331382 >> 172.21.15.189:0/3756331382 conn(0x7f2a180d00b0 msgr2=0x7f2a1815c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.181+0000 7f2a1d5a0700 1 -- 172.21.15.189:0/3756331382 shutdown_connections 2024-04-06T12:28:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.181+0000 7f2a1d5a0700 1 -- 172.21.15.189:0/3756331382 wait complete. 2024-04-06T12:28:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 41 2024-04-06T12:28:35.190 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-06T12:28:35.190 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-06T12:28:35.190 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-06T12:28:35.191 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:28:35.191 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-06T12:28:35.191 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-06T12:28:35.191 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-06T12:28:35.191 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:28:35.191 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-06T12:28:35.192 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-06T12:28:35.192 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-06T12:28:35.192 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:28:35.192 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-06T12:28:35.192 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-06T12:28:35.192 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-06T12:28:35.192 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:28:35.200 INFO:tasks.ceph.mds.a:Stopped 2024-04-06T12:28:35.200 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-06T12:28:35.201 INFO:tasks.ceph.mds.c:Stopped 2024-04-06T12:28:35.201 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-06T12:28:35.203 INFO:tasks.ceph.mds.b:Stopped 2024-04-06T12:28:35.203 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-06T12:28:35.204 INFO:tasks.ceph.mds.d:Stopped 2024-04-06T12:28:35.204 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-06T12:28:35.242 INFO:tasks.ceph.mds.a:Started 2024-04-06T12:28:35.243 DEBUG:teuthology.parallel:result is None 2024-04-06T12:28:35.243 INFO:tasks.ceph.mds.c:Started 2024-04-06T12:28:35.243 DEBUG:teuthology.parallel:result is None 2024-04-06T12:28:35.245 INFO:tasks.ceph.mds.b:Started 2024-04-06T12:28:35.245 DEBUG:teuthology.parallel:result is None 2024-04-06T12:28:35.246 INFO:tasks.ceph.mds.d:Started 2024-04-06T12:28:35.246 DEBUG:teuthology.parallel:result is None 2024-04-06T12:28:35.246 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-06T12:28:35.323 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2024-04-06T12:28:35.323 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T12:28:35.333 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T12:28:35.334 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2024-04-06T12:28:35.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.380+0000 7f371cb0f700 1 Processor -- start 2024-04-06T12:28:35.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.380+0000 7f371cb0f700 1 -- start start 2024-04-06T12:28:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.381+0000 7f371cb0f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37180b5380 0x7f37180d2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.381+0000 7f371cb0f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37180b5cc0 con 0x7f371815d590 2024-04-06T12:28:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.381+0000 7f371cb0f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37180b57f0 con 0x7f37180b4ed0 2024-04-06T12:28:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.381+0000 7f371cb0f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37180d1500 con 0x7f37180b5380 2024-04-06T12:28:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f371659c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f371815d590 0x7f37180b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58456/0 (socket says 172.21.15.189:58456) 2024-04-06T12:28:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f371659c700 1 -- 172.21.15.189:0/740476414 learned_addr learned my addr 172.21.15.189:0/740476414 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f371559a700 1 -- 172.21.15.189:0/740476414 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831402956 0 0) 0x7f37180b5cc0 con 0x7f371815d590 2024-04-06T12:28:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f371559a700 1 -- 172.21.15.189:0/740476414 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37040035b0 con 0x7f371815d590 2024-04-06T12:28:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f3716d9d700 1 --2- 172.21.15.189:0/740476414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37180b5380 0x7f37180d2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f371559a700 1 -- 172.21.15.189:0/740476414 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3467357862 0 0) 0x7f37040035b0 con 0x7f371815d590 2024-04-06T12:28:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f371559a700 1 -- 172.21.15.189:0/740476414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37180b5380 msgr2=0x7f37180d2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f371559a700 1 --2- 172.21.15.189:0/740476414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37180b5380 0x7f37180d2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f371559a700 1 -- 172.21.15.189:0/740476414 >> v1:172.21.15.192:6789/0 conn(0x7f37180b4ed0 legacy=0x7f37180d2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f371559a700 1 -- 172.21.15.189:0/740476414 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37180d17e0 con 0x7f371815d590 2024-04-06T12:28:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f371559a700 1 -- 172.21.15.189:0/740476414 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f37080019f0 con 0x7f371815d590 2024-04-06T12:28:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f371559a700 1 -- 172.21.15.189:0/740476414 >> v1:172.21.15.189:6789/0 conn(0x7f371815d590 legacy=0x7f37180b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f371559a700 1 --2- 172.21.15.189:0/740476414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3704003a30 0x7f3704003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f371559a700 1 --2- 172.21.15.189:0/740476414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3704004850 0x7f3704004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f371559a700 1 --2- 172.21.15.189:0/740476414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3704005210 0x7f370400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f371559a700 1 -- 172.21.15.189:0/740476414 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37180d1500 con 0x7f3704004850 2024-04-06T12:28:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f371559a700 1 -- 172.21.15.189:0/740476414 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370400c0b0 con 0x7f3704003a30 2024-04-06T12:28:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f371559a700 1 -- 172.21.15.189:0/740476414 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370400c270 con 0x7f3704005210 2024-04-06T12:28:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f3716d9d700 1 --2- 172.21.15.189:0/740476414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37180b5380 0x7f37180d2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f3715d9b700 1 --2- 172.21.15.189:0/740476414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3704004850 0x7f3704004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f371659c700 1 --2- 172.21.15.189:0/740476414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3704003a30 0x7f3704003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.382+0000 7f3716d9d700 1 --2- 172.21.15.189:0/740476414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3704005210 0x7f370400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.383+0000 7f3715d9b700 1 -- 172.21.15.189:0/740476414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3704005210 msgr2=0x7f370400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.383+0000 7f3715d9b700 1 --2- 172.21.15.189:0/740476414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3704005210 0x7f370400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.383+0000 7f3715d9b700 1 -- 172.21.15.189:0/740476414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3704003a30 msgr2=0x7f3704003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.383+0000 7f3715d9b700 1 --2- 172.21.15.189:0/740476414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3704003a30 0x7f3704003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.383+0000 7f3715d9b700 1 -- 172.21.15.189:0/740476414 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37040035b0 con 0x7f3704004850 2024-04-06T12:28:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.383+0000 7f3715d9b700 1 --2- 172.21.15.189:0/740476414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3704004850 0x7f3704004cd0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f37000017c0 tx=0x7f3704005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=429910ee93ea3b5a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.383+0000 7f371559a700 1 -- 172.21.15.189:0/740476414 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37000052d0 con 0x7f3704004850 2024-04-06T12:28:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.383+0000 7f371559a700 1 -- 172.21.15.189:0/740476414 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3700005430 con 0x7f3704004850 2024-04-06T12:28:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.383+0000 7f371559a700 1 -- 172.21.15.189:0/740476414 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37000038c0 con 0x7f3704004850 2024-04-06T12:28:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.383+0000 7f371cb0f700 1 -- 172.21.15.189:0/740476414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3704004850 msgr2=0x7f3704004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.383+0000 7f371cb0f700 1 --2- 172.21.15.189:0/740476414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3704004850 0x7f3704004cd0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f37000017c0 tx=0x7f3704005880 comp rx=0 tx=0).stop 2024-04-06T12:28:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.383+0000 7f3716d9d700 1 -- 172.21.15.189:0/740476414 reap_dead start 2024-04-06T12:28:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.383+0000 7f371cb0f700 1 -- 172.21.15.189:0/740476414 shutdown_connections 2024-04-06T12:28:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.383+0000 7f371cb0f700 1 -- 172.21.15.189:0/740476414 >> 172.21.15.189:0/740476414 conn(0x7f371806bbc0 msgr2=0x7f371806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.383+0000 7f371cb0f700 1 -- 172.21.15.189:0/740476414 shutdown_connections 2024-04-06T12:28:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.383+0000 7f371cb0f700 1 -- 172.21.15.189:0/740476414 wait complete. 2024-04-06T12:28:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.385+0000 7f371cb0f700 1 Processor -- start 2024-04-06T12:28:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.385+0000 7f371cb0f700 1 -- start start 2024-04-06T12:28:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f371cb0f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3704004850 0x7f37180b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f371cb0f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f371815ff50 0x7f37181603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f371cb0f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37180c8dd0 0x7f37180c9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f371cb0f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37181f7300 con 0x7f3704004850 2024-04-06T12:28:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f371cb0f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37181f6680 con 0x7f371815ff50 2024-04-06T12:28:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f371cb0f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37181f6960 con 0x7f37180c8dd0 2024-04-06T12:28:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f371659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3704004850 0x7f37180b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:35.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f371659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3704004850 0x7f37180b3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36172/0 (socket says 172.21.15.189:36172) 2024-04-06T12:28:35.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f371659c700 1 -- 172.21.15.189:0/3913229729 learned_addr learned my addr 172.21.15.189:0/3913229729 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:35.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f3716d9d700 1 --2- 172.21.15.189:0/3913229729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37180c8dd0 0x7f37180c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f3715d9b700 1 --2- 172.21.15.189:0/3913229729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f371815ff50 0x7f37181603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f371659c700 1 -- 172.21.15.189:0/3913229729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37180c8dd0 msgr2=0x7f37180c9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f371659c700 1 --2- 172.21.15.189:0/3913229729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37180c8dd0 0x7f37180c9240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f371659c700 1 -- 172.21.15.189:0/3913229729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f371815ff50 msgr2=0x7f37181603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f371659c700 1 --2- 172.21.15.189:0/3913229729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f371815ff50 0x7f37181603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f371659c700 1 -- 172.21.15.189:0/3913229729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37181f6c40 con 0x7f3704004850 2024-04-06T12:28:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f371659c700 1 --2- 172.21.15.189:0/3913229729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3704004850 0x7f37180b3120 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f370800a540 tx=0x7f370800a850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f370effd700 1 -- 172.21.15.189:0/3913229729 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37080133f0 con 0x7f3704004850 2024-04-06T12:28:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f370effd700 1 -- 172.21.15.189:0/3913229729 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3708013550 con 0x7f3704004850 2024-04-06T12:28:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f371cb0f700 1 -- 172.21.15.189:0/3913229729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37181f6f20 con 0x7f3704004850 2024-04-06T12:28:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f370effd700 1 -- 172.21.15.189:0/3913229729 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3708013900 con 0x7f3704004850 2024-04-06T12:28:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.386+0000 7f371cb0f700 1 -- 172.21.15.189:0/3913229729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37181f3e30 con 0x7f3704004850 2024-04-06T12:28:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.387+0000 7f371cb0f700 1 -- 172.21.15.189:0/3913229729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3718074990 con 0x7f3704004850 2024-04-06T12:28:35.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.388+0000 7f370effd700 1 -- 172.21.15.189:0/3913229729 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f3708013a60 con 0x7f3704004850 2024-04-06T12:28:35.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.388+0000 7f370effd700 1 --2- 172.21.15.189:0/3913229729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f36f806adb0 0x7f36f806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:35.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.388+0000 7f370effd700 1 -- 172.21.15.189:0/3913229729 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f370808a2d0 con 0x7f3704004850 2024-04-06T12:28:35.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.388+0000 7f3715d9b700 1 --2- 172.21.15.189:0/3913229729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f36f806adb0 0x7f36f806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:35.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.388+0000 7f3715d9b700 1 --2- 172.21.15.189:0/3913229729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f36f806adb0 0x7f36f806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3700000ff0 tx=0x7f3700009030 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:35.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.392+0000 7f370effd700 1 -- 172.21.15.189:0/3913229729 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3708055390 con 0x7f3704004850 2024-04-06T12:28:35.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:35.602+0000 7f371cb0f700 1 -- 172.21.15.189:0/3913229729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f371815c680 con 0x7f3704004850 2024-04-06T12:28:36.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.578+0000 7f370effd700 1 -- 172.21.15.189:0/3913229729 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v108) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3708001840 con 0x7f3704004850 2024-04-06T12:28:36.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.581+0000 7f371cb0f700 1 -- 172.21.15.189:0/3913229729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f36f806adb0 msgr2=0x7f36f806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:36.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.581+0000 7f371cb0f700 1 --2- 172.21.15.189:0/3913229729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f36f806adb0 0x7f36f806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3700000ff0 tx=0x7f3700009030 comp rx=0 tx=0).stop 2024-04-06T12:28:36.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.581+0000 7f371cb0f700 1 -- 172.21.15.189:0/3913229729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3704004850 msgr2=0x7f37180b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:36.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.581+0000 7f371cb0f700 1 --2- 172.21.15.189:0/3913229729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3704004850 0x7f37180b3120 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f370800a540 tx=0x7f370800a850 comp rx=0 tx=0).stop 2024-04-06T12:28:36.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.581+0000 7f371cb0f700 1 -- 172.21.15.189:0/3913229729 shutdown_connections 2024-04-06T12:28:36.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.581+0000 7f371cb0f700 1 --2- 172.21.15.189:0/3913229729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37180c8dd0 0x7f37180c9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:36.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.581+0000 7f371cb0f700 1 --2- 172.21.15.189:0/3913229729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3704004850 0x7f37180b3120 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:36.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.581+0000 7f371cb0f700 1 --2- 172.21.15.189:0/3913229729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f36f806adb0 0x7f36f806d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:36.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.581+0000 7f371cb0f700 1 --2- 172.21.15.189:0/3913229729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f371815ff50 0x7f37181603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:36.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.581+0000 7f371cb0f700 1 -- 172.21.15.189:0/3913229729 >> 172.21.15.189:0/3913229729 conn(0x7f371806bbc0 msgr2=0x7f371806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:36.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.581+0000 7f371cb0f700 1 -- 172.21.15.189:0/3913229729 shutdown_connections 2024-04-06T12:28:36.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.581+0000 7f371cb0f700 1 -- 172.21.15.189:0/3913229729 wait complete. 2024-04-06T12:28:36.580 INFO:teuthology.orchestra.run.smithi189.stderr: removed all blocklist entries 2024-04-06T12:28:36.593 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-06T12:28:36.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.723+0000 7f3f9bba4700 1 Processor -- start 2024-04-06T12:28:36.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.724+0000 7f3f9bba4700 1 -- start start 2024-04-06T12:28:36.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.724+0000 7f3f9bba4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3f940d11b0 0x7f3f940d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:36.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.724+0000 7f3f9bba4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f940cb870 0x7f3f9415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:36.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.724+0000 7f3f9bba4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f940d1ad0 con 0x7f3f940cfa00 2024-04-06T12:28:36.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.724+0000 7f3f9bba4700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f940ccdc0 con 0x7f3f940d11b0 2024-04-06T12:28:36.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.724+0000 7f3f9bba4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f940d0b50 con 0x7f3f940cb870 2024-04-06T12:28:36.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.724+0000 7f3f99940700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3f940d11b0 0x7f3f940d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:36.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.725+0000 7f3f99940700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3f940d11b0 0x7f3f940d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36186/0 (socket says 172.21.15.189:36186) 2024-04-06T12:28:36.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.725+0000 7f3f99940700 1 -- 172.21.15.189:0/2427171474 learned_addr learned my addr 172.21.15.189:0/2427171474 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:36.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.725+0000 7f3f99940700 1 -- 172.21.15.189:0/2427171474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f940cb870 msgr2=0x7f3f9415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:36.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.725+0000 7f3f99940700 1 --2- 172.21.15.189:0/2427171474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f940cb870 0x7f3f9415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:36.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.725+0000 7f3f99940700 1 -- 172.21.15.189:0/2427171474 >> v1:172.21.15.192:6789/0 conn(0x7f3f940cfa00 legacy=0x7f3f940cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:36.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.725+0000 7f3f99940700 1 -- 172.21.15.189:0/2427171474 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f940cbd80 con 0x7f3f940d11b0 2024-04-06T12:28:36.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.725+0000 7f3f99940700 1 --2- 172.21.15.189:0/2427171474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f940d11b0 0x7f3f940d1590 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f3f840098f0 tx=0x7f3f84009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79971ac3f861cd9f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:36.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.725+0000 7f3f9893e700 1 -- 172.21.15.189:0/2427171474 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f84004230 con 0x7f3f940d11b0 2024-04-06T12:28:36.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.725+0000 7f3f9893e700 1 -- 172.21.15.189:0/2427171474 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f84004390 con 0x7f3f940d11b0 2024-04-06T12:28:36.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.725+0000 7f3f9893e700 1 -- 172.21.15.189:0/2427171474 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f840036a0 con 0x7f3f940d11b0 2024-04-06T12:28:36.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.726+0000 7f3f9bba4700 1 -- 172.21.15.189:0/2427171474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f940d11b0 msgr2=0x7f3f940d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:36.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.726+0000 7f3f9bba4700 1 --2- 172.21.15.189:0/2427171474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f940d11b0 0x7f3f940d1590 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f3f840098f0 tx=0x7f3f84009c00 comp rx=0 tx=0).stop 2024-04-06T12:28:36.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.726+0000 7f3f9bba4700 1 -- 172.21.15.189:0/2427171474 shutdown_connections 2024-04-06T12:28:36.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.726+0000 7f3f9bba4700 1 --2- 172.21.15.189:0/2427171474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f940d11b0 0x7f3f940d1590 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:36.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.726+0000 7f3f9bba4700 1 --2- 172.21.15.189:0/2427171474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f940cb870 0x7f3f9415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:36.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.726+0000 7f3f9bba4700 1 -- 172.21.15.189:0/2427171474 >> 172.21.15.189:0/2427171474 conn(0x7f3f940b2db0 msgr2=0x7f3f940b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:36.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.726+0000 7f3f9bba4700 1 -- 172.21.15.189:0/2427171474 shutdown_connections 2024-04-06T12:28:36.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.726+0000 7f3f9bba4700 1 -- 172.21.15.189:0/2427171474 wait complete. 2024-04-06T12:28:36.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.727+0000 7f3f9bba4700 1 Processor -- start 2024-04-06T12:28:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.727+0000 7f3f9bba4700 1 -- start start 2024-04-06T12:28:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f9bba4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f940cb870 0x7f3f94155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f9bba4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f940cfa00 0x7f3f9415c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f9bba4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f940d11b0 0x7f3f94158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f9bba4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f94155c70 con 0x7f3f940cb870 2024-04-06T12:28:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f9bba4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f940c9730 con 0x7f3f940cfa00 2024-04-06T12:28:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f9bba4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f940c9a10 con 0x7f3f940d11b0 2024-04-06T12:28:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f99940700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f940cb870 0x7f3f94155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f99940700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f940cb870 0x7f3f94155730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36188/0 (socket says 172.21.15.189:36188) 2024-04-06T12:28:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f99940700 1 -- 172.21.15.189:0/2575331224 learned_addr learned my addr 172.21.15.189:0/2575331224 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f9a141700 1 --2- 172.21.15.189:0/2575331224 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f940d11b0 0x7f3f94158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f99940700 1 -- 172.21.15.189:0/2575331224 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f940d11b0 msgr2=0x7f3f94158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f9913f700 1 --2- 172.21.15.189:0/2575331224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f940cfa00 0x7f3f9415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f99940700 1 --2- 172.21.15.189:0/2575331224 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f940d11b0 0x7f3f94158ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f99940700 1 -- 172.21.15.189:0/2575331224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f940cfa00 msgr2=0x7f3f9415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f99940700 1 --2- 172.21.15.189:0/2575331224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f940cfa00 0x7f3f9415c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f99940700 1 -- 172.21.15.189:0/2575331224 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f84009550 con 0x7f3f940cb870 2024-04-06T12:28:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f9a141700 1 --2- 172.21.15.189:0/2575331224 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f940d11b0 0x7f3f94158ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:28:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f9913f700 1 --2- 172.21.15.189:0/2575331224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f940cfa00 0x7f3f9415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:28:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f99940700 1 --2- 172.21.15.189:0/2575331224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f940cb870 0x7f3f94155730 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f3f940677e0 tx=0x7f3f84004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f8a7fc700 1 -- 172.21.15.189:0/2575331224 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f84003e70 con 0x7f3f940cb870 2024-04-06T12:28:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f8a7fc700 1 -- 172.21.15.189:0/2575331224 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f8401f070 con 0x7f3f940cb870 2024-04-06T12:28:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f9bba4700 1 -- 172.21.15.189:0/2575331224 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f940c9cf0 con 0x7f3f940cb870 2024-04-06T12:28:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f8a7fc700 1 -- 172.21.15.189:0/2575331224 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f8400b8c0 con 0x7f3f940cb870 2024-04-06T12:28:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.728+0000 7f3f9bba4700 1 -- 172.21.15.189:0/2575331224 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f940ca2b0 con 0x7f3f940cb870 2024-04-06T12:28:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.729+0000 7f3f9bba4700 1 -- 172.21.15.189:0/2575331224 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f94074990 con 0x7f3f940cb870 2024-04-06T12:28:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.730+0000 7f3f8a7fc700 1 -- 172.21.15.189:0/2575331224 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f3f84021030 con 0x7f3f940cb870 2024-04-06T12:28:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.730+0000 7f3f8a7fc700 1 --2- 172.21.15.189:0/2575331224 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3f7806adb0 0x7f3f7806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.730+0000 7f3f8a7fc700 1 -- 172.21.15.189:0/2575331224 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f3f8408a940 con 0x7f3f940cb870 2024-04-06T12:28:36.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.730+0000 7f3f9913f700 1 --2- 172.21.15.189:0/2575331224 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3f7806adb0 0x7f3f7806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:36.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.731+0000 7f3f9913f700 1 --2- 172.21.15.189:0/2575331224 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3f7806adb0 0x7f3f7806d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3f9415f640 tx=0x7f3f90008040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:36.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:36.735+0000 7f3f8a7fc700 1 -- 172.21.15.189:0/2575331224 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3f84055dd0 con 0x7f3f940cb870 2024-04-06T12:28:37.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.008+0000 7f3f9bba4700 1 -- 172.21.15.189:0/2575331224 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3f94074990 con 0x7f3f940cb870 2024-04-06T12:28:37.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.009+0000 7f3f8a7fc700 1 -- 172.21.15.189:0/2575331224 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f3f840593f0 con 0x7f3f940cb870 2024-04-06T12:28:37.006 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:37.006 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T12:28:37.006 INFO:teuthology.orchestra.run.smithi189.stdout: "auth_dump": [ 2024-04-06T12:28:37.006 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:37.006 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.a", 2024-04-06T12:28:37.006 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmTjYMNBAA5voWGV4Pa7pE27VogrdipQ==", 2024-04-06T12:28:37.006 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:37.006 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:28:37.007 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:28:37.007 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:28:37.007 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:37.007 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.007 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:37.007 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:37.007 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.b", 2024-04-06T12:28:37.007 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm9A1OBRAAnHzubCvRUX8UdW8hmDzKbQ==", 2024-04-06T12:28:37.007 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:37.007 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:28:37.007 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:28:37.007 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:28:37.007 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:37.007 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.007 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:37.008 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:37.008 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.c", 2024-04-06T12:28:37.008 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmz0hhOxAAD3/GINNmGoFk+oPyiHHJXg==", 2024-04-06T12:28:37.008 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:37.008 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:28:37.008 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:28:37.008 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:28:37.008 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:37.008 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.008 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:37.008 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:37.008 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.d", 2024-04-06T12:28:37.008 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmhoAeDRAAoQC0vicWumBRTwpi3PtYaw==", 2024-04-06T12:28:37.008 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:37.009 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:28:37.009 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:28:37.009 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:28:37.009 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:37.009 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.009 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:37.009 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:37.009 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.0", 2024-04-06T12:28:37.009 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDkOxFmkxLWJhAAFXewCNOFX7dhAZVgUmewpQ==", 2024-04-06T12:28:37.009 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:37.009 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:28:37.009 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:28:37.009 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:37.009 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.009 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:37.009 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:37.010 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.1", 2024-04-06T12:28:37.010 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDnOxFmHYqWJRAABlsVCiv3e7YJMbfodJTWag==", 2024-04-06T12:28:37.010 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:37.010 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:28:37.010 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:28:37.010 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:37.010 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.010 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:37.010 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:37.010 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.2", 2024-04-06T12:28:37.010 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDqOxFmkhcvJBAA8NMEpHVGbGALjV9RISKP8g==", 2024-04-06T12:28:37.010 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:37.010 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:28:37.010 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:28:37.011 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:37.011 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.011 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:37.011 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:37.011 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.3", 2024-04-06T12:28:37.011 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDtOxFmDGzhIxAAamBoAbE8+lAel4os67meyQ==", 2024-04-06T12:28:37.011 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:37.011 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:28:37.011 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:28:37.011 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:37.011 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.011 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:37.011 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:37.011 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.4", 2024-04-06T12:28:37.011 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDzOxFmobZDJxAA9l8wdS81Eyw6+17I5ac6bw==", 2024-04-06T12:28:37.011 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:37.012 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:28:37.012 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:28:37.012 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:37.012 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.012 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:37.012 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:37.012 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.5", 2024-04-06T12:28:37.012 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD2OxFmi+ruJRAASlQ0o9jX/wA7U5AosnL+Wg==", 2024-04-06T12:28:37.012 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:37.012 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:28:37.012 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:28:37.012 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:37.013 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.013 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:37.013 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:37.013 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.6", 2024-04-06T12:28:37.013 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD5OxFmbwuuJBAA0wqJRmnJyskQPZJ7/LnpvA==", 2024-04-06T12:28:37.013 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:37.013 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:28:37.013 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:28:37.013 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:37.013 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.013 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:37.013 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:37.013 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.7", 2024-04-06T12:28:37.014 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD8OxFmDmRgIxAAzyW9AGAbmb67ZvB5qunwLQ==", 2024-04-06T12:28:37.014 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:37.014 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:28:37.014 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:28:37.014 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:37.014 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.014 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:37.014 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:37.014 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.0", 2024-04-06T12:28:37.014 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm0ByJERAAoEV8Yfc09+Go6u95XafElQ==", 2024-04-06T12:28:37.014 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:37.014 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:28:37.014 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow r", 2024-04-06T12:28:37.014 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-06T12:28:37.015 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.015 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:37.015 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:37.015 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.1", 2024-04-06T12:28:37.015 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmSV5lGBAAavDEo2LGy0IHpWSAuitHXg==", 2024-04-06T12:28:37.015 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:37.015 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:28:37.015 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:28:37.015 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:28:37.015 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:28:37.015 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.015 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:37.015 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:37.015 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.2", 2024-04-06T12:28:37.016 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm/Gz5FBAA3xX/rsAyWZ1JoUVlQlBIvQ==", 2024-04-06T12:28:37.016 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:37.016 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:28:37.016 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:28:37.016 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:28:37.016 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:28:37.016 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.016 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:37.016 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:37.016 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.3", 2024-04-06T12:28:37.016 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmzra7GxAADbnzIvWnRTo65LvPwtxWrg==", 2024-04-06T12:28:37.016 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:37.016 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:28:37.016 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:28:37.016 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:28:37.017 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:28:37.017 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.017 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:37.017 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:37.017 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.admin", 2024-04-06T12:28:37.017 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmxG+wCBAANg5kt6TOUQdj3/X0Mqgx1g==", 2024-04-06T12:28:37.017 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:37.017 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:28:37.017 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:28:37.017 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:28:37.017 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:37.017 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.017 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:37.017 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:37.017 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.x", 2024-04-06T12:28:37.017 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmPLyBMBAAYfhkI9Z1CPlf1Tijm+L7xA==", 2024-04-06T12:28:37.018 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:37.018 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:28:37.018 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T12:28:37.018 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:37.018 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.018 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:28:37.018 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:37.018 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.y", 2024-04-06T12:28:37.018 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFm1nHJHhAAQWRe8cU8duWuUJiYaDeTaQ==", 2024-04-06T12:28:37.018 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:28:37.018 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:28:37.018 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T12:28:37.018 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:28:37.018 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.019 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:37.019 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-04-06T12:28:37.019 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T12:28:37.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.011+0000 7f3f9bba4700 1 -- 172.21.15.189:0/2575331224 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3f7806adb0 msgr2=0x7f3f7806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.011+0000 7f3f9bba4700 1 --2- 172.21.15.189:0/2575331224 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3f7806adb0 0x7f3f7806d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3f9415f640 tx=0x7f3f90008040 comp rx=0 tx=0).stop 2024-04-06T12:28:37.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.011+0000 7f3f9bba4700 1 -- 172.21.15.189:0/2575331224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f940cb870 msgr2=0x7f3f94155730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.011+0000 7f3f9bba4700 1 --2- 172.21.15.189:0/2575331224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f940cb870 0x7f3f94155730 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f3f940677e0 tx=0x7f3f84004750 comp rx=0 tx=0).stop 2024-04-06T12:28:37.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.011+0000 7f3f9bba4700 1 -- 172.21.15.189:0/2575331224 shutdown_connections 2024-04-06T12:28:37.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.011+0000 7f3f9bba4700 1 --2- 172.21.15.189:0/2575331224 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f940d11b0 0x7f3f94158ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.011+0000 7f3f9bba4700 1 --2- 172.21.15.189:0/2575331224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f940cb870 0x7f3f94155730 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.011+0000 7f3f9bba4700 1 --2- 172.21.15.189:0/2575331224 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3f7806adb0 0x7f3f7806d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.011+0000 7f3f9bba4700 1 --2- 172.21.15.189:0/2575331224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f940cfa00 0x7f3f9415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.011+0000 7f3f9bba4700 1 -- 172.21.15.189:0/2575331224 >> 172.21.15.189:0/2575331224 conn(0x7f3f940b2db0 msgr2=0x7f3f940b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:37.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.011+0000 7f3f9bba4700 1 -- 172.21.15.189:0/2575331224 shutdown_connections 2024-04-06T12:28:37.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.011+0000 7f3f9bba4700 1 -- 172.21.15.189:0/2575331224 wait complete. 2024-04-06T12:28:37.021 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:28:37.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.146+0000 7ff87731b700 1 Processor -- start 2024-04-06T12:28:37.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.147+0000 7ff87731b700 1 -- start start 2024-04-06T12:28:37.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.147+0000 7ff87731b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff87015d7b0 0x7ff8700b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.147+0000 7ff87731b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8700b3990 con 0x7ff87015e500 2024-04-06T12:28:37.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.147+0000 7ff87731b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8700b3060 con 0x7ff87006a730 2024-04-06T12:28:37.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.147+0000 7ff87731b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff87015e950 con 0x7ff87015d7b0 2024-04-06T12:28:37.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff8758b8700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff87015e500 0x7ff8700717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58464/0 (socket says 172.21.15.189:58464) 2024-04-06T12:28:37.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff8758b8700 1 -- 172.21.15.189:0/2571998044 learned_addr learned my addr 172.21.15.189:0/2571998044 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:37.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff863fff700 1 -- 172.21.15.189:0/2571998044 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1129713028 0 0) 0x7ff8700b3990 con 0x7ff87015e500 2024-04-06T12:28:37.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff8750b7700 1 --2- 172.21.15.189:0/2571998044 >> v2:172.21.15.192:3300/0 conn(0x7ff87015d7b0 0x7ff8700b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff863fff700 1 -- 172.21.15.189:0/2571998044 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff85c0035b0 con 0x7ff87015e500 2024-04-06T12:28:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff863fff700 1 -- 172.21.15.189:0/2571998044 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 432278515 0 0) 0x7ff85c0035b0 con 0x7ff87015e500 2024-04-06T12:28:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff863fff700 1 -- 172.21.15.189:0/2571998044 >> v2:172.21.15.192:3300/0 conn(0x7ff87015d7b0 msgr2=0x7ff8700b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff863fff700 1 --2- 172.21.15.189:0/2571998044 >> v2:172.21.15.192:3300/0 conn(0x7ff87015d7b0 0x7ff8700b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff863fff700 1 -- 172.21.15.189:0/2571998044 >> v1:172.21.15.192:6789/0 conn(0x7ff87006a730 legacy=0x7ff87015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff863fff700 1 -- 172.21.15.189:0/2571998044 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8700b3340 con 0x7ff87015e500 2024-04-06T12:28:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff863fff700 1 -- 172.21.15.189:0/2571998044 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7ff8680019f0 con 0x7ff87015e500 2024-04-06T12:28:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff863fff700 1 -- 172.21.15.189:0/2571998044 >> v1:172.21.15.189:6789/0 conn(0x7ff87015e500 legacy=0x7ff8700717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff863fff700 1 --2- 172.21.15.189:0/2571998044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff85c003a00 0x7ff85c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff863fff700 1 --2- 172.21.15.189:0/2571998044 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff85c004850 0x7ff85c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff863fff700 1 --2- 172.21.15.189:0/2571998044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff85c005330 0x7ff85c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff863fff700 1 -- 172.21.15.189:0/2571998044 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c00c1d0 con 0x7ff85c003a00 2024-04-06T12:28:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff863fff700 1 -- 172.21.15.189:0/2571998044 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c00c340 con 0x7ff85c005330 2024-04-06T12:28:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff863fff700 1 -- 172.21.15.189:0/2571998044 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c00c500 con 0x7ff85c004850 2024-04-06T12:28:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff8750b7700 1 --2- 172.21.15.189:0/2571998044 >> v2:172.21.15.192:3300/0 conn(0x7ff87015d7b0 0x7ff8700b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff8750b7700 1 --2- 172.21.15.189:0/2571998044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff85c003a00 0x7ff85c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff8748b6700 1 --2- 172.21.15.189:0/2571998044 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff85c004850 0x7ff85c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.148+0000 7ff8758b8700 1 --2- 172.21.15.189:0/2571998044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff85c005330 0x7ff85c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.149+0000 7ff8750b7700 1 -- 172.21.15.189:0/2571998044 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff85c004850 msgr2=0x7ff85c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.149+0000 7ff8750b7700 1 --2- 172.21.15.189:0/2571998044 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff85c004850 0x7ff85c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.149+0000 7ff8750b7700 1 -- 172.21.15.189:0/2571998044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff85c005330 msgr2=0x7ff85c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.149+0000 7ff8750b7700 1 --2- 172.21.15.189:0/2571998044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff85c005330 0x7ff85c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.149+0000 7ff8758b8700 1 -- 172.21.15.189:0/2571998044 reap_dead start 2024-04-06T12:28:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.149+0000 7ff8750b7700 1 -- 172.21.15.189:0/2571998044 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85c0035b0 con 0x7ff85c003a00 2024-04-06T12:28:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.149+0000 7ff8750b7700 1 --2- 172.21.15.189:0/2571998044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff85c003a00 0x7ff85c003e20 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7ff86c009900 tx=0x7ff85c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b863965842cdbe8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.149+0000 7ff863fff700 1 -- 172.21.15.189:0/2571998044 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c00c070 con 0x7ff85c003a00 2024-04-06T12:28:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.149+0000 7ff863fff700 1 -- 172.21.15.189:0/2571998044 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86c004230 con 0x7ff85c003a00 2024-04-06T12:28:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.149+0000 7ff863fff700 1 -- 172.21.15.189:0/2571998044 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c0045a0 con 0x7ff85c003a00 2024-04-06T12:28:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.149+0000 7ff87731b700 1 -- 172.21.15.189:0/2571998044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff85c003a00 msgr2=0x7ff85c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.149+0000 7ff87731b700 1 --2- 172.21.15.189:0/2571998044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff85c003a00 0x7ff85c003e20 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7ff86c009900 tx=0x7ff85c005880 comp rx=0 tx=0).stop 2024-04-06T12:28:37.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.149+0000 7ff87731b700 1 -- 172.21.15.189:0/2571998044 shutdown_connections 2024-04-06T12:28:37.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.149+0000 7ff87731b700 1 --2- 172.21.15.189:0/2571998044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff85c003a00 0x7ff85c003e20 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.149+0000 7ff87731b700 1 -- 172.21.15.189:0/2571998044 >> 172.21.15.189:0/2571998044 conn(0x7ff8700d00b0 msgr2=0x7ff8700d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:37.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.150+0000 7ff87731b700 1 -- 172.21.15.189:0/2571998044 shutdown_connections 2024-04-06T12:28:37.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.150+0000 7ff87731b700 1 -- 172.21.15.189:0/2571998044 wait complete. 2024-04-06T12:28:37.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.151+0000 7ff87731b700 1 Processor -- start 2024-04-06T12:28:37.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.151+0000 7ff87731b700 1 -- start start 2024-04-06T12:28:37.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.151+0000 7ff87731b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff85c003a00 0x7ff8700b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.151+0000 7ff87731b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8700b1470 0x7ff8700aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.151+0000 7ff87731b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8700af0c0 0x7ff8700af550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.151+0000 7ff87731b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff870146940 con 0x7ff8700b1470 2024-04-06T12:28:37.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.151+0000 7ff87731b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff870145f60 con 0x7ff8700af0c0 2024-04-06T12:28:37.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.151+0000 7ff87731b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff870146240 con 0x7ff85c003a00 2024-04-06T12:28:37.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.152+0000 7ff8748b6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8700b1470 0x7ff8700aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.152+0000 7ff8748b6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8700b1470 0x7ff8700aeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36208/0 (socket says 172.21.15.189:36208) 2024-04-06T12:28:37.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.152+0000 7ff8748b6700 1 -- 172.21.15.189:0/355541743 learned_addr learned my addr 172.21.15.189:0/355541743 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:37.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.152+0000 7ff8750b7700 1 --2- 172.21.15.189:0/355541743 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff85c003a00 0x7ff8700b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.152+0000 7ff8758b8700 1 --2- 172.21.15.189:0/355541743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8700af0c0 0x7ff8700af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.152+0000 7ff8748b6700 1 -- 172.21.15.189:0/355541743 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff85c003a00 msgr2=0x7ff8700b0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.152+0000 7ff8748b6700 1 --2- 172.21.15.189:0/355541743 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff85c003a00 0x7ff8700b0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.152+0000 7ff8748b6700 1 -- 172.21.15.189:0/355541743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8700af0c0 msgr2=0x7ff8700af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.152+0000 7ff8748b6700 1 --2- 172.21.15.189:0/355541743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8700af0c0 0x7ff8700af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.152+0000 7ff8748b6700 1 -- 172.21.15.189:0/355541743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff870146b90 con 0x7ff8700b1470 2024-04-06T12:28:37.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.152+0000 7ff8748b6700 1 --2- 172.21.15.189:0/355541743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8700b1470 0x7ff8700aeb80 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7ff86400f7f0 tx=0x7ff86400fb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:37.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.152+0000 7ff861ffb700 1 -- 172.21.15.189:0/355541743 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff864005320 con 0x7ff8700b1470 2024-04-06T12:28:37.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.152+0000 7ff861ffb700 1 -- 172.21.15.189:0/355541743 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff864005480 con 0x7ff8700b1470 2024-04-06T12:28:37.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.152+0000 7ff87731b700 1 -- 172.21.15.189:0/355541743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff870146e70 con 0x7ff8700b1470 2024-04-06T12:28:37.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.152+0000 7ff861ffb700 1 -- 172.21.15.189:0/355541743 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff864006a80 con 0x7ff8700b1470 2024-04-06T12:28:37.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.152+0000 7ff87731b700 1 -- 172.21.15.189:0/355541743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff870149710 con 0x7ff8700b1470 2024-04-06T12:28:37.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.153+0000 7ff87731b700 1 -- 172.21.15.189:0/355541743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8701471e0 con 0x7ff8700b1470 2024-04-06T12:28:37.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.154+0000 7ff861ffb700 1 -- 172.21.15.189:0/355541743 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7ff864006be0 con 0x7ff8700b1470 2024-04-06T12:28:37.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.154+0000 7ff861ffb700 1 --2- 172.21.15.189:0/355541743 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff85406adb0 0x7ff85406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.154+0000 7ff861ffb700 1 -- 172.21.15.189:0/355541743 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5177+0+0 (secure 0 0 0) 0x7ff864088650 con 0x7ff8700b1470 2024-04-06T12:28:37.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.154+0000 7ff8750b7700 1 --2- 172.21.15.189:0/355541743 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff85406adb0 0x7ff85406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.155+0000 7ff8750b7700 1 --2- 172.21.15.189:0/355541743 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff85406adb0 0x7ff85406d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff86c008010 tx=0x7ff86c00c000 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:37.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.159+0000 7ff861ffb700 1 -- 172.21.15.189:0/355541743 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff864053940 con 0x7ff8700b1470 2024-04-06T12:28:37.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.356+0000 7ff87731b700 1 -- 172.21.15.189:0/355541743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff870067bb0 con 0x7ff8700b1470 2024-04-06T12:28:37.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.357+0000 7ff861ffb700 1 -- 172.21.15.189:0/355541743 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+12644 (secure 0 0 0) 0x7ff864056f60 con 0x7ff8700b1470 2024-04-06T12:28:37.354 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:37.354 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":108,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:28:36.575324+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:28:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.359+0000 7ff87731b700 1 -- 172.21.15.189:0/355541743 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff85406adb0 msgr2=0x7ff85406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.359+0000 7ff87731b700 1 --2- 172.21.15.189:0/355541743 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff85406adb0 0x7ff85406d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff86c008010 tx=0x7ff86c00c000 comp rx=0 tx=0).stop 2024-04-06T12:28:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.359+0000 7ff87731b700 1 -- 172.21.15.189:0/355541743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8700b1470 msgr2=0x7ff8700aeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.359+0000 7ff87731b700 1 --2- 172.21.15.189:0/355541743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8700b1470 0x7ff8700aeb80 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7ff86400f7f0 tx=0x7ff86400fb00 comp rx=0 tx=0).stop 2024-04-06T12:28:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.359+0000 7ff87731b700 1 -- 172.21.15.189:0/355541743 shutdown_connections 2024-04-06T12:28:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.359+0000 7ff87731b700 1 --2- 172.21.15.189:0/355541743 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff85406adb0 0x7ff85406d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.359+0000 7ff87731b700 1 --2- 172.21.15.189:0/355541743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8700af0c0 0x7ff8700af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.359+0000 7ff87731b700 1 --2- 172.21.15.189:0/355541743 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff85c003a00 0x7ff8700b0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.359+0000 7ff87731b700 1 --2- 172.21.15.189:0/355541743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8700b1470 0x7ff8700aeb80 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.359+0000 7ff87731b700 1 -- 172.21.15.189:0/355541743 >> 172.21.15.189:0/355541743 conn(0x7ff8700d00b0 msgr2=0x7ff87006abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.359+0000 7ff87731b700 1 -- 172.21.15.189:0/355541743 shutdown_connections 2024-04-06T12:28:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.359+0000 7ff87731b700 1 -- 172.21.15.189:0/355541743 wait complete. 2024-04-06T12:28:37.368 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:28:37.369 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:28:37.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.496+0000 7f0a21fe9700 1 Processor -- start 2024-04-06T12:28:37.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.497+0000 7f0a21fe9700 1 -- start start 2024-04-06T12:28:37.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.497+0000 7f0a21fe9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0a1c0cd3e0 0x7f0a1c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.497+0000 7f0a21fe9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a1c06b8c0 0x7f0a1c06dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.497+0000 7f0a21fe9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a1c06ceb0 con 0x7f0a1c15d7c0 2024-04-06T12:28:37.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.497+0000 7f0a21fe9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a1c0cdfa0 con 0x7f0a1c0cd3e0 2024-04-06T12:28:37.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.497+0000 7f0a21fe9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c06bcf0 con 0x7f0a1c06b8c0 2024-04-06T12:28:37.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.497+0000 7f0a1b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0a1c0cd3e0 0x7f0a1c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.497+0000 7f0a1b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0a1c0cd3e0 0x7f0a1c0cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36214/0 (socket says 172.21.15.189:36214) 2024-04-06T12:28:37.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.497+0000 7f0a1b7fe700 1 -- 172.21.15.189:0/293700545 learned_addr learned my addr 172.21.15.189:0/293700545 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:37.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.498+0000 7f0a1bfff700 1 --2- 172.21.15.189:0/293700545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a1c06b8c0 0x7f0a1c06dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.498+0000 7f0a1b7fe700 1 -- 172.21.15.189:0/293700545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a1c06b8c0 msgr2=0x7f0a1c06dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:37.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.498+0000 7f0a1b7fe700 1 --2- 172.21.15.189:0/293700545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a1c06b8c0 0x7f0a1c06dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.498+0000 7f0a1b7fe700 1 -- 172.21.15.189:0/293700545 >> v1:172.21.15.192:6789/0 conn(0x7f0a1c15d7c0 legacy=0x7f0a1c06b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.498+0000 7f0a1b7fe700 1 -- 172.21.15.189:0/293700545 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1c06d750 con 0x7f0a1c0cd3e0 2024-04-06T12:28:37.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.498+0000 7f0a1b7fe700 1 --2- 172.21.15.189:0/293700545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0cd3e0 0x7f0a1c0cd7c0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f0a040098f0 tx=0x7f0a04009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f82f7d3e0ade3ba server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:37.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.498+0000 7f0a1a7fc700 1 -- 172.21.15.189:0/293700545 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a04011830 con 0x7f0a1c0cd3e0 2024-04-06T12:28:37.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.498+0000 7f0a1a7fc700 1 -- 172.21.15.189:0/293700545 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a04011990 con 0x7f0a1c0cd3e0 2024-04-06T12:28:37.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.498+0000 7f0a1a7fc700 1 -- 172.21.15.189:0/293700545 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a04011d20 con 0x7f0a1c0cd3e0 2024-04-06T12:28:37.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.499+0000 7f0a21fe9700 1 -- 172.21.15.189:0/293700545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0cd3e0 msgr2=0x7f0a1c0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.499+0000 7f0a21fe9700 1 --2- 172.21.15.189:0/293700545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0cd3e0 0x7f0a1c0cd7c0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f0a040098f0 tx=0x7f0a04009c00 comp rx=0 tx=0).stop 2024-04-06T12:28:37.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.499+0000 7f0a21fe9700 1 -- 172.21.15.189:0/293700545 shutdown_connections 2024-04-06T12:28:37.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.499+0000 7f0a21fe9700 1 --2- 172.21.15.189:0/293700545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a1c06b8c0 0x7f0a1c06dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.499+0000 7f0a21fe9700 1 --2- 172.21.15.189:0/293700545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0cd3e0 0x7f0a1c0cd7c0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.499+0000 7f0a21fe9700 1 -- 172.21.15.189:0/293700545 >> 172.21.15.189:0/293700545 conn(0x7f0a1c0cc290 msgr2=0x7f0a1c0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:37.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.499+0000 7f0a21fe9700 1 -- 172.21.15.189:0/293700545 shutdown_connections 2024-04-06T12:28:37.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.499+0000 7f0a21fe9700 1 -- 172.21.15.189:0/293700545 wait complete. 2024-04-06T12:28:37.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.500+0000 7f0a21fe9700 1 Processor -- start 2024-04-06T12:28:37.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.500+0000 7f0a21fe9700 1 -- start start 2024-04-06T12:28:37.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.501+0000 7f0a21fe9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a1c06b8c0 0x7f0a1c0b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.501+0000 7f0a21fe9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0cd3e0 0x7f0a1c0c4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.501+0000 7f0a21fe9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a1c15d7c0 0x7f0a1c0c4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.501+0000 7f0a21fe9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c0b4e70 con 0x7f0a1c0cd3e0 2024-04-06T12:28:37.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.501+0000 7f0a21fe9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c0b49b0 con 0x7f0a1c15d7c0 2024-04-06T12:28:37.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.501+0000 7f0a21fe9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c0b4b90 con 0x7f0a1c06b8c0 2024-04-06T12:28:37.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.501+0000 7f0a1affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0cd3e0 0x7f0a1c0c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.501+0000 7f0a1affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0cd3e0 0x7f0a1c0c4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36228/0 (socket says 172.21.15.189:36228) 2024-04-06T12:28:37.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.501+0000 7f0a1affd700 1 -- 172.21.15.189:0/858885919 learned_addr learned my addr 172.21.15.189:0/858885919 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:37.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.501+0000 7f0a1b7fe700 1 --2- 172.21.15.189:0/858885919 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a1c06b8c0 0x7f0a1c0b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.501+0000 7f0a1bfff700 1 --2- 172.21.15.189:0/858885919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a1c15d7c0 0x7f0a1c0c4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.501+0000 7f0a1affd700 1 -- 172.21.15.189:0/858885919 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a1c06b8c0 msgr2=0x7f0a1c0b22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.501+0000 7f0a1affd700 1 --2- 172.21.15.189:0/858885919 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a1c06b8c0 0x7f0a1c0b22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.501+0000 7f0a1affd700 1 -- 172.21.15.189:0/858885919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a1c15d7c0 msgr2=0x7f0a1c0c4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.501+0000 7f0a1affd700 1 --2- 172.21.15.189:0/858885919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a1c15d7c0 0x7f0a1c0c4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.501+0000 7f0a1affd700 1 -- 172.21.15.189:0/858885919 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a04009550 con 0x7f0a1c0cd3e0 2024-04-06T12:28:37.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.501+0000 7f0a1affd700 1 --2- 172.21.15.189:0/858885919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0cd3e0 0x7f0a1c0c4110 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c00c4e0 tx=0x7f0a0c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:37.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.501+0000 7f0a13fff700 1 -- 172.21.15.189:0/858885919 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c0052d0 con 0x7f0a1c0cd3e0 2024-04-06T12:28:37.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.502+0000 7f0a13fff700 1 -- 172.21.15.189:0/858885919 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a0c005430 con 0x7f0a1c0cd3e0 2024-04-06T12:28:37.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.502+0000 7f0a21fe9700 1 -- 172.21.15.189:0/858885919 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a1c0b46d0 con 0x7f0a1c0cd3e0 2024-04-06T12:28:37.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.502+0000 7f0a13fff700 1 -- 172.21.15.189:0/858885919 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c0038c0 con 0x7f0a1c0cd3e0 2024-04-06T12:28:37.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.502+0000 7f0a21fe9700 1 -- 172.21.15.189:0/858885919 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a1c0c3cd0 con 0x7f0a1c0cd3e0 2024-04-06T12:28:37.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.502+0000 7f0a21fe9700 1 -- 172.21.15.189:0/858885919 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a1c1f4fb0 con 0x7f0a1c0cd3e0 2024-04-06T12:28:37.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.503+0000 7f0a13fff700 1 -- 172.21.15.189:0/858885919 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0a0c003a20 con 0x7f0a1c0cd3e0 2024-04-06T12:28:37.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.503+0000 7f0a13fff700 1 --2- 172.21.15.189:0/858885919 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f09fc06adb0 0x7f09fc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.503+0000 7f0a13fff700 1 -- 172.21.15.189:0/858885919 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f0a0c088590 con 0x7f0a1c0cd3e0 2024-04-06T12:28:37.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.503+0000 7f0a1b7fe700 1 --2- 172.21.15.189:0/858885919 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f09fc06adb0 0x7f09fc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.504+0000 7f0a1b7fe700 1 --2- 172.21.15.189:0/858885919 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f09fc06adb0 0x7f09fc06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0a04009c40 tx=0x7f0a04008030 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:37.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.508+0000 7f0a13fff700 1 -- 172.21.15.189:0/858885919 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0a1c1f4fb0 con 0x7f0a1c0cd3e0 2024-04-06T12:28:37.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.701+0000 7f0a21fe9700 1 -- 172.21.15.189:0/858885919 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0a1c1f4fb0 con 0x7f0a1c0cd3e0 2024-04-06T12:28:37.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.702+0000 7f0a13fff700 1 -- 172.21.15.189:0/858885919 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v108) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0a0c017020 con 0x7f0a1c0cd3e0 2024-04-06T12:28:37.699 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:28:37.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.704+0000 7f0a21fe9700 1 -- 172.21.15.189:0/858885919 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f09fc06adb0 msgr2=0x7f09fc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.704+0000 7f0a21fe9700 1 --2- 172.21.15.189:0/858885919 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f09fc06adb0 0x7f09fc06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0a04009c40 tx=0x7f0a04008030 comp rx=0 tx=0).stop 2024-04-06T12:28:37.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.704+0000 7f0a21fe9700 1 -- 172.21.15.189:0/858885919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0cd3e0 msgr2=0x7f0a1c0c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.704+0000 7f0a21fe9700 1 --2- 172.21.15.189:0/858885919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0cd3e0 0x7f0a1c0c4110 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c00c4e0 tx=0x7f0a0c00c7f0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.704+0000 7f0a21fe9700 1 -- 172.21.15.189:0/858885919 shutdown_connections 2024-04-06T12:28:37.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.704+0000 7f0a21fe9700 1 --2- 172.21.15.189:0/858885919 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f09fc06adb0 0x7f09fc06d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.704+0000 7f0a21fe9700 1 --2- 172.21.15.189:0/858885919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a1c15d7c0 0x7f0a1c0c4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.704+0000 7f0a21fe9700 1 --2- 172.21.15.189:0/858885919 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a1c06b8c0 0x7f0a1c0b22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.704+0000 7f0a21fe9700 1 --2- 172.21.15.189:0/858885919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0cd3e0 0x7f0a1c0c4110 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.704+0000 7f0a21fe9700 1 -- 172.21.15.189:0/858885919 >> 172.21.15.189:0/858885919 conn(0x7f0a1c0cc290 msgr2=0x7f0a1c0af2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:37.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.704+0000 7f0a21fe9700 1 -- 172.21.15.189:0/858885919 shutdown_connections 2024-04-06T12:28:37.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.704+0000 7f0a21fe9700 1 -- 172.21.15.189:0/858885919 wait complete. 2024-04-06T12:28:37.713 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-06T12:28:37.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.838+0000 7fcb56ef8700 1 Processor -- start 2024-04-06T12:28:37.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.838+0000 7fcb56ef8700 1 -- start start 2024-04-06T12:28:37.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.839+0000 7fcb56ef8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb500b3e20 0x7fcb5015e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.839+0000 7fcb56ef8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb500b3990 con 0x7fcb5015d7b0 2024-04-06T12:28:37.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.839+0000 7fcb56ef8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb500b3060 con 0x7fcb5006ae40 2024-04-06T12:28:37.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.839+0000 7fcb56ef8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb500b3340 con 0x7fcb500b3e20 2024-04-06T12:28:37.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.839+0000 7fcb54c94700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcb5015d7b0 0x7fcb5006a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58480/0 (socket says 172.21.15.189:58480) 2024-04-06T12:28:37.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.839+0000 7fcb54c94700 1 -- 172.21.15.189:0/1271174820 learned_addr learned my addr 172.21.15.189:0/1271174820 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:37.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb55495700 1 --2- 172.21.15.189:0/1271174820 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb500b3e20 0x7fcb5015e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4f7fe700 1 -- 172.21.15.189:0/1271174820 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3698000953 0 0) 0x7fcb500b3990 con 0x7fcb5015d7b0 2024-04-06T12:28:37.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4f7fe700 1 -- 172.21.15.189:0/1271174820 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb400035b0 con 0x7fcb5015d7b0 2024-04-06T12:28:37.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4f7fe700 1 -- 172.21.15.189:0/1271174820 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1067906926 0 0) 0x7fcb400035b0 con 0x7fcb5015d7b0 2024-04-06T12:28:37.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4f7fe700 1 -- 172.21.15.189:0/1271174820 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb500b3e20 msgr2=0x7fcb5015e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:37.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4f7fe700 1 --2- 172.21.15.189:0/1271174820 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb500b3e20 0x7fcb5015e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4f7fe700 1 -- 172.21.15.189:0/1271174820 >> v1:172.21.15.192:6789/0 conn(0x7fcb5006ae40 legacy=0x7fcb5015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4f7fe700 1 -- 172.21.15.189:0/1271174820 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb500b3620 con 0x7fcb5015d7b0 2024-04-06T12:28:37.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4f7fe700 1 -- 172.21.15.189:0/1271174820 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fcb44001670 con 0x7fcb5015d7b0 2024-04-06T12:28:37.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4f7fe700 1 -- 172.21.15.189:0/1271174820 >> v1:172.21.15.189:6789/0 conn(0x7fcb5015d7b0 legacy=0x7fcb5006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4f7fe700 1 --2- 172.21.15.189:0/1271174820 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb40003a30 0x7fcb40003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4f7fe700 1 --2- 172.21.15.189:0/1271174820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb400043e0 0x7fcb400048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4f7fe700 1 --2- 172.21.15.189:0/1271174820 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb500b3e20 0x7fcb400051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4f7fe700 1 -- 172.21.15.189:0/1271174820 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb500b3340 con 0x7fcb400043e0 2024-04-06T12:28:37.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4f7fe700 1 -- 172.21.15.189:0/1271174820 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4000bac0 con 0x7fcb40003a30 2024-04-06T12:28:37.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4f7fe700 1 -- 172.21.15.189:0/1271174820 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4000bc80 con 0x7fcb500b3e20 2024-04-06T12:28:37.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4ffff700 1 --2- 172.21.15.189:0/1271174820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb400043e0 0x7fcb400048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb54c94700 1 --2- 172.21.15.189:0/1271174820 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb40003a30 0x7fcb40003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb55495700 1 --2- 172.21.15.189:0/1271174820 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb500b3e20 0x7fcb400051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4ffff700 1 -- 172.21.15.189:0/1271174820 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb500b3e20 msgr2=0x7fcb400051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:37.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4ffff700 1 --2- 172.21.15.189:0/1271174820 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb500b3e20 0x7fcb400051b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4ffff700 1 -- 172.21.15.189:0/1271174820 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb40003a30 msgr2=0x7fcb40003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:37.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4ffff700 1 --2- 172.21.15.189:0/1271174820 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb40003a30 0x7fcb40003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.840+0000 7fcb4ffff700 1 -- 172.21.15.189:0/1271174820 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb400035b0 con 0x7fcb400043e0 2024-04-06T12:28:37.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.841+0000 7fcb4ffff700 1 --2- 172.21.15.189:0/1271174820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb400043e0 0x7fcb400048b0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c0018b0 tx=0x7fcb40005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c598938c319ea5a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:37.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.841+0000 7fcb4f7fe700 1 -- 172.21.15.189:0/1271174820 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c001370 con 0x7fcb400043e0 2024-04-06T12:28:37.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.841+0000 7fcb4f7fe700 1 -- 172.21.15.189:0/1271174820 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3c00c6b0 con 0x7fcb400043e0 2024-04-06T12:28:37.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.841+0000 7fcb4f7fe700 1 -- 172.21.15.189:0/1271174820 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c00ca60 con 0x7fcb400043e0 2024-04-06T12:28:37.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.841+0000 7fcb56ef8700 1 -- 172.21.15.189:0/1271174820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb400043e0 msgr2=0x7fcb400048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.841+0000 7fcb56ef8700 1 --2- 172.21.15.189:0/1271174820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb400043e0 0x7fcb400048b0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c0018b0 tx=0x7fcb40005880 comp rx=0 tx=0).stop 2024-04-06T12:28:37.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.841+0000 7fcb55495700 1 -- 172.21.15.189:0/1271174820 reap_dead start 2024-04-06T12:28:37.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.841+0000 7fcb56ef8700 1 -- 172.21.15.189:0/1271174820 shutdown_connections 2024-04-06T12:28:37.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.841+0000 7fcb56ef8700 1 -- 172.21.15.189:0/1271174820 >> 172.21.15.189:0/1271174820 conn(0x7fcb500d00b0 msgr2=0x7fcb500d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:37.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.841+0000 7fcb56ef8700 1 -- 172.21.15.189:0/1271174820 shutdown_connections 2024-04-06T12:28:37.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.841+0000 7fcb56ef8700 1 -- 172.21.15.189:0/1271174820 wait complete. 2024-04-06T12:28:37.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.842+0000 7fcb56ef8700 1 Processor -- start 2024-04-06T12:28:37.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.842+0000 7fcb56ef8700 1 -- start start 2024-04-06T12:28:37.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.843+0000 7fcb56ef8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb400043e0 0x7fcb500cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.843+0000 7fcb56ef8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb50160400 0x7fcb5014b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.843+0000 7fcb56ef8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb50160880 0x7fcb5014bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.843+0000 7fcb56ef8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5015c750 con 0x7fcb50160400 2024-04-06T12:28:37.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.843+0000 7fcb56ef8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb501f70c0 con 0x7fcb400043e0 2024-04-06T12:28:37.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.843+0000 7fcb56ef8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb501f73a0 con 0x7fcb50160880 2024-04-06T12:28:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.843+0000 7fcb4ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb50160400 0x7fcb5014b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.843+0000 7fcb4ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb50160400 0x7fcb5014b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36240/0 (socket says 172.21.15.189:36240) 2024-04-06T12:28:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.843+0000 7fcb4ffff700 1 -- 172.21.15.189:0/4029237987 learned_addr learned my addr 172.21.15.189:0/4029237987 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.844+0000 7fcb4ffff700 1 -- 172.21.15.189:0/4029237987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb50160880 msgr2=0x7fcb5014bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.844+0000 7fcb55495700 1 --2- 172.21.15.189:0/4029237987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb50160880 0x7fcb5014bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.844+0000 7fcb4ffff700 1 --2- 172.21.15.189:0/4029237987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb50160880 0x7fcb5014bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.844+0000 7fcb4ffff700 1 -- 172.21.15.189:0/4029237987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb400043e0 msgr2=0x7fcb500cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.844+0000 7fcb54c94700 1 --2- 172.21.15.189:0/4029237987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb400043e0 0x7fcb500cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.844+0000 7fcb4ffff700 1 --2- 172.21.15.189:0/4029237987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb400043e0 0x7fcb500cd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.844+0000 7fcb4ffff700 1 -- 172.21.15.189:0/4029237987 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb500aec20 con 0x7fcb50160400 2024-04-06T12:28:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.844+0000 7fcb4ffff700 1 --2- 172.21.15.189:0/4029237987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb50160400 0x7fcb5014b780 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fcb40005810 tx=0x7fcb3c001110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.844+0000 7fcb4d7fa700 1 -- 172.21.15.189:0/4029237987 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c001370 con 0x7fcb50160400 2024-04-06T12:28:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.844+0000 7fcb4d7fa700 1 -- 172.21.15.189:0/4029237987 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3c00cc50 con 0x7fcb50160400 2024-04-06T12:28:37.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.844+0000 7fcb56ef8700 1 -- 172.21.15.189:0/4029237987 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb500aef00 con 0x7fcb50160400 2024-04-06T12:28:37.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.844+0000 7fcb4d7fa700 1 -- 172.21.15.189:0/4029237987 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c006860 con 0x7fcb50160400 2024-04-06T12:28:37.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.844+0000 7fcb56ef8700 1 -- 172.21.15.189:0/4029237987 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb500cdc70 con 0x7fcb50160400 2024-04-06T12:28:37.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.845+0000 7fcb56ef8700 1 -- 172.21.15.189:0/4029237987 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb500af550 con 0x7fcb50160400 2024-04-06T12:28:37.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.846+0000 7fcb4d7fa700 1 -- 172.21.15.189:0/4029237987 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fcb3c023070 con 0x7fcb50160400 2024-04-06T12:28:37.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.846+0000 7fcb4d7fa700 1 --2- 172.21.15.189:0/4029237987 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcb3406adb0 0x7fcb3406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:37.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.846+0000 7fcb4d7fa700 1 -- 172.21.15.189:0/4029237987 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fcb3c08a6c0 con 0x7fcb50160400 2024-04-06T12:28:37.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.846+0000 7fcb54c94700 1 --2- 172.21.15.189:0/4029237987 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcb3406adb0 0x7fcb3406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:37.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.846+0000 7fcb54c94700 1 --2- 172.21.15.189:0/4029237987 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcb3406adb0 0x7fcb3406d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcb44001eb0 tx=0x7fcb44009040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:37.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:37.850+0000 7fcb4d7fa700 1 -- 172.21.15.189:0/4029237987 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fcb3c055b50 con 0x7fcb50160400 2024-04-06T12:28:38.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.047+0000 7fcb56ef8700 1 -- 172.21.15.189:0/4029237987 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fcb50067bb0 con 0x7fcb50160400 2024-04-06T12:28:38.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.047+0000 7fcb4d7fa700 1 -- 172.21.15.189:0/4029237987 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v108) v1 ==== 84+0+65 (secure 0 0 0) 0x7fcb3c02a020 con 0x7fcb50160400 2024-04-06T12:28:38.044 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:38.045 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T12:28:38.045 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:38.045 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 1, 2024-04-06T12:28:38.045 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": ".mgr" 2024-04-06T12:28:38.045 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:38.045 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T12:28:38.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.049+0000 7fcb56ef8700 1 -- 172.21.15.189:0/4029237987 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcb3406adb0 msgr2=0x7fcb3406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:38.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.049+0000 7fcb56ef8700 1 --2- 172.21.15.189:0/4029237987 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcb3406adb0 0x7fcb3406d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcb44001eb0 tx=0x7fcb44009040 comp rx=0 tx=0).stop 2024-04-06T12:28:38.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.049+0000 7fcb56ef8700 1 -- 172.21.15.189:0/4029237987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb50160400 msgr2=0x7fcb5014b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:38.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.049+0000 7fcb56ef8700 1 --2- 172.21.15.189:0/4029237987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb50160400 0x7fcb5014b780 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fcb40005810 tx=0x7fcb3c001110 comp rx=0 tx=0).stop 2024-04-06T12:28:38.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.049+0000 7fcb56ef8700 1 -- 172.21.15.189:0/4029237987 shutdown_connections 2024-04-06T12:28:38.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.049+0000 7fcb56ef8700 1 --2- 172.21.15.189:0/4029237987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb50160880 0x7fcb5014bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:38.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.049+0000 7fcb56ef8700 1 --2- 172.21.15.189:0/4029237987 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcb3406adb0 0x7fcb3406d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:38.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.049+0000 7fcb56ef8700 1 --2- 172.21.15.189:0/4029237987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb400043e0 0x7fcb500cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:38.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.049+0000 7fcb56ef8700 1 --2- 172.21.15.189:0/4029237987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb50160400 0x7fcb5014b780 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:38.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.049+0000 7fcb56ef8700 1 -- 172.21.15.189:0/4029237987 >> 172.21.15.189:0/4029237987 conn(0x7fcb500d00b0 msgr2=0x7fcb50072640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:38.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.049+0000 7fcb56ef8700 1 -- 172.21.15.189:0/4029237987 shutdown_connections 2024-04-06T12:28:38.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.049+0000 7fcb56ef8700 1 -- 172.21.15.189:0/4029237987 wait complete. 2024-04-06T12:28:38.059 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-06T12:28:38.059 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-06T12:28:38.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.189+0000 7f8b7b2e8700 1 Processor -- start 2024-04-06T12:28:38.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.189+0000 7f8b7b2e8700 1 -- start start 2024-04-06T12:28:38.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.190+0000 7f8b7b2e8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8b7406ae40 0x7f8b740b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:38.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.190+0000 7f8b7b2e8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b740b3990 con 0x7f8b7415d7b0 2024-04-06T12:28:38.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.190+0000 7f8b7b2e8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b740b3060 con 0x7f8b7415ddf0 2024-04-06T12:28:38.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.190+0000 7f8b7b2e8700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b7415e950 con 0x7f8b7406ae40 2024-04-06T12:28:38.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.190+0000 7f8b78883700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8b7406ae40 0x7f8b740b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:38.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.190+0000 7f8b78883700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8b7406ae40 0x7f8b740b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36252/0 (socket says 172.21.15.189:36252) 2024-04-06T12:28:38.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.190+0000 7f8b78883700 1 -- 172.21.15.189:0/1851959241 learned_addr learned my addr 172.21.15.189:0/1851959241 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:38.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.190+0000 7f8b79084700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8b7415d7b0 0x7f8b7406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58490/0 (socket says 172.21.15.189:58490) 2024-04-06T12:28:38.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.190+0000 7f8b78883700 1 -- 172.21.15.189:0/1851959241 >> v1:172.21.15.192:6789/0 conn(0x7f8b7415ddf0 legacy=0x7f8b7415e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:38.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.190+0000 7f8b78883700 1 -- 172.21.15.189:0/1851959241 >> v1:172.21.15.189:6789/0 conn(0x7f8b7415d7b0 legacy=0x7f8b7406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:38.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.190+0000 7f8b6bfff700 1 -- 172.21.15.189:0/1851959241 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166988612 0 0) 0x7f8b740b3990 con 0x7f8b7415d7b0 2024-04-06T12:28:38.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.190+0000 7f8b78883700 1 -- 172.21.15.189:0/1851959241 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b740b3340 con 0x7f8b7406ae40 2024-04-06T12:28:38.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.191+0000 7f8b78883700 1 --2- 172.21.15.189:0/1851959241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b7406ae40 0x7f8b740b3e20 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f8b700098f0 tx=0x7f8b70009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea1b6bc3b42c011 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:38.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.191+0000 7f8b6bfff700 1 -- 172.21.15.189:0/1851959241 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7000b8a0 con 0x7f8b7406ae40 2024-04-06T12:28:38.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.191+0000 7f8b6bfff700 1 -- 172.21.15.189:0/1851959241 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7000ba00 con 0x7f8b7406ae40 2024-04-06T12:28:38.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.191+0000 7f8b6bfff700 1 -- 172.21.15.189:0/1851959241 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7000bd90 con 0x7f8b7406ae40 2024-04-06T12:28:38.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.191+0000 7f8b7b2e8700 1 -- 172.21.15.189:0/1851959241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b7406ae40 msgr2=0x7f8b740b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:38.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.191+0000 7f8b7b2e8700 1 --2- 172.21.15.189:0/1851959241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b7406ae40 0x7f8b740b3e20 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f8b700098f0 tx=0x7f8b70009c00 comp rx=0 tx=0).stop 2024-04-06T12:28:38.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.191+0000 7f8b7b2e8700 1 -- 172.21.15.189:0/1851959241 shutdown_connections 2024-04-06T12:28:38.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.191+0000 7f8b7b2e8700 1 --2- 172.21.15.189:0/1851959241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b7406ae40 0x7f8b740b3e20 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:38.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.191+0000 7f8b7b2e8700 1 -- 172.21.15.189:0/1851959241 >> 172.21.15.189:0/1851959241 conn(0x7f8b740d00b0 msgr2=0x7f8b740d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:38.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.191+0000 7f8b7b2e8700 1 -- 172.21.15.189:0/1851959241 shutdown_connections 2024-04-06T12:28:38.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.191+0000 7f8b7b2e8700 1 -- 172.21.15.189:0/1851959241 wait complete. 2024-04-06T12:28:38.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.193+0000 7f8b7b2e8700 1 Processor -- start 2024-04-06T12:28:38.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.193+0000 7f8b7b2e8700 1 -- start start 2024-04-06T12:28:38.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.193+0000 7f8b7b2e8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b7406ae40 0x7f8b741545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:38.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.193+0000 7f8b7b2e8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8b7415d7b0 0x7f8b740cee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:38.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.193+0000 7f8b7b2e8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b7415ddf0 0x7f8b74145ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:38.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.193+0000 7f8b7b2e8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b740cf370 con 0x7f8b7415ddf0 2024-04-06T12:28:38.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.193+0000 7f8b7b2e8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7415bf00 con 0x7f8b7406ae40 2024-04-06T12:28:38.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.193+0000 7f8b7b2e8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7415b140 con 0x7f8b7415d7b0 2024-04-06T12:28:38.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.193+0000 7f8b79885700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b7415ddf0 0x7f8b74145ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:38.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.194+0000 7f8b79885700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b7415ddf0 0x7f8b74145ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36264/0 (socket says 172.21.15.189:36264) 2024-04-06T12:28:38.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.194+0000 7f8b79885700 1 -- 172.21.15.189:0/2210203065 learned_addr learned my addr 172.21.15.189:0/2210203065 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:38.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.194+0000 7f8b78883700 1 --2- 172.21.15.189:0/2210203065 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8b7415d7b0 0x7f8b740cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:38.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.194+0000 7f8b79084700 1 --2- 172.21.15.189:0/2210203065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b7406ae40 0x7f8b741545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:38.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.194+0000 7f8b79885700 1 -- 172.21.15.189:0/2210203065 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8b7415d7b0 msgr2=0x7f8b740cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:38.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.194+0000 7f8b79885700 1 --2- 172.21.15.189:0/2210203065 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8b7415d7b0 0x7f8b740cee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:38.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.194+0000 7f8b79885700 1 -- 172.21.15.189:0/2210203065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b7406ae40 msgr2=0x7f8b741545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:38.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.194+0000 7f8b79885700 1 --2- 172.21.15.189:0/2210203065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b7406ae40 0x7f8b741545c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:38.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.194+0000 7f8b79885700 1 -- 172.21.15.189:0/2210203065 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b70009550 con 0x7f8b7415ddf0 2024-04-06T12:28:38.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.194+0000 7f8b78883700 1 --2- 172.21.15.189:0/2210203065 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8b7415d7b0 0x7f8b740cee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:28:38.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.194+0000 7f8b79885700 1 --2- 172.21.15.189:0/2210203065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b7415ddf0 0x7f8b74145ac0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c00cb50 tx=0x7f8b6c00cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:38.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.194+0000 7f8b69ffb700 1 -- 172.21.15.189:0/2210203065 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6c003c00 con 0x7f8b7415ddf0 2024-04-06T12:28:38.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.194+0000 7f8b69ffb700 1 -- 172.21.15.189:0/2210203065 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b6c003d60 con 0x7f8b7415ddf0 2024-04-06T12:28:38.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.194+0000 7f8b7b2e8700 1 -- 172.21.15.189:0/2210203065 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b7415b420 con 0x7f8b7415ddf0 2024-04-06T12:28:38.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.194+0000 7f8b69ffb700 1 -- 172.21.15.189:0/2210203065 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6c00ab70 con 0x7f8b7415ddf0 2024-04-06T12:28:38.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.194+0000 7f8b7b2e8700 1 -- 172.21.15.189:0/2210203065 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b7415b9e0 con 0x7f8b7415ddf0 2024-04-06T12:28:38.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.195+0000 7f8b7b2e8700 1 -- 172.21.15.189:0/2210203065 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b74067bb0 con 0x7f8b7415ddf0 2024-04-06T12:28:38.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.196+0000 7f8b69ffb700 1 -- 172.21.15.189:0/2210203065 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f8b6c00acd0 con 0x7f8b7415ddf0 2024-04-06T12:28:38.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.196+0000 7f8b69ffb700 1 --2- 172.21.15.189:0/2210203065 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8b5806adb0 0x7f8b5806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:38.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.196+0000 7f8b69ffb700 1 -- 172.21.15.189:0/2210203065 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f8b6c089240 con 0x7f8b7415ddf0 2024-04-06T12:28:38.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.196+0000 7f8b79084700 1 --2- 172.21.15.189:0/2210203065 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8b5806adb0 0x7f8b5806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:38.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.196+0000 7f8b79084700 1 --2- 172.21.15.189:0/2210203065 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8b5806adb0 0x7f8b5806d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8b64000f60 tx=0x7f8b64009040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:38.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.200+0000 7f8b69ffb700 1 -- 172.21.15.189:0/2210203065 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8b6c0545b0 con 0x7f8b7415ddf0 2024-04-06T12:28:38.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:38.396+0000 7f8b7b2e8700 1 -- 172.21.15.189:0/2210203065 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8b74074990 con 0x7f8b7415ddf0 2024-04-06T12:28:39.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.247+0000 7f8b69ffb700 1 -- 172.21.15.189:0/2210203065 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v109) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8b6c057bd0 con 0x7f8b7415ddf0 2024-04-06T12:28:39.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.251+0000 7f8b7b2e8700 1 -- 172.21.15.189:0/2210203065 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8b5806adb0 msgr2=0x7f8b5806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:39.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.251+0000 7f8b7b2e8700 1 --2- 172.21.15.189:0/2210203065 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8b5806adb0 0x7f8b5806d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8b64000f60 tx=0x7f8b64009040 comp rx=0 tx=0).stop 2024-04-06T12:28:39.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.251+0000 7f8b7b2e8700 1 -- 172.21.15.189:0/2210203065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b7415ddf0 msgr2=0x7f8b74145ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:39.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.251+0000 7f8b7b2e8700 1 --2- 172.21.15.189:0/2210203065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b7415ddf0 0x7f8b74145ac0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c00cb50 tx=0x7f8b6c00cf10 comp rx=0 tx=0).stop 2024-04-06T12:28:39.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.252+0000 7f8b7b2e8700 1 -- 172.21.15.189:0/2210203065 shutdown_connections 2024-04-06T12:28:39.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.252+0000 7f8b7b2e8700 1 --2- 172.21.15.189:0/2210203065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b7415ddf0 0x7f8b74145ac0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:39.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.252+0000 7f8b7b2e8700 1 --2- 172.21.15.189:0/2210203065 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8b5806adb0 0x7f8b5806d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:39.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.252+0000 7f8b7b2e8700 1 --2- 172.21.15.189:0/2210203065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b7406ae40 0x7f8b741545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:39.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.252+0000 7f8b7b2e8700 1 --2- 172.21.15.189:0/2210203065 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8b7415d7b0 0x7f8b740cee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:39.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.252+0000 7f8b7b2e8700 1 -- 172.21.15.189:0/2210203065 >> 172.21.15.189:0/2210203065 conn(0x7f8b740d00b0 msgr2=0x7f8b740ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:39.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.252+0000 7f8b7b2e8700 1 -- 172.21.15.189:0/2210203065 shutdown_connections 2024-04-06T12:28:39.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.252+0000 7f8b7b2e8700 1 -- 172.21.15.189:0/2210203065 wait complete. 2024-04-06T12:28:39.251 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' created 2024-04-06T12:28:39.266 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-06T12:28:39.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.403+0000 7f4318c66700 1 Processor -- start 2024-04-06T12:28:39.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.403+0000 7f4318c66700 1 -- start start 2024-04-06T12:28:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.404+0000 7f4318c66700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f431415a550 0x7f43140cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.404+0000 7f4318c66700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43140cf900 0x7f43140b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.404+0000 7f4318c66700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43140cdbe0 con 0x7f43140b4ba0 2024-04-06T12:28:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.404+0000 7f4318c66700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43140b5010 con 0x7f431415a550 2024-04-06T12:28:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.404+0000 7f4318c66700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f431415a940 con 0x7f43140cf900 2024-04-06T12:28:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.404+0000 7f431259c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f431415a550 0x7f43140cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.404+0000 7f431259c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f431415a550 0x7f43140cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53306/0 (socket says 172.21.15.189:53306) 2024-04-06T12:28:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.404+0000 7f431259c700 1 -- 172.21.15.189:0/2057314039 learned_addr learned my addr 172.21.15.189:0/2057314039 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.404+0000 7f4311d9b700 1 --2- 172.21.15.189:0/2057314039 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43140cf900 0x7f43140b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.404+0000 7f431259c700 1 -- 172.21.15.189:0/2057314039 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43140cf900 msgr2=0x7f43140b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.404+0000 7f431259c700 1 --2- 172.21.15.189:0/2057314039 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43140cf900 0x7f43140b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.404+0000 7f431259c700 1 -- 172.21.15.189:0/2057314039 >> v1:172.21.15.192:6789/0 conn(0x7f43140b4ba0 legacy=0x7f431406eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.404+0000 7f431259c700 1 -- 172.21.15.189:0/2057314039 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431415ac00 con 0x7f431415a550 2024-04-06T12:28:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.404+0000 7f4311d9b700 1 --2- 172.21.15.189:0/2057314039 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43140cf900 0x7f43140b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.405+0000 7f431259c700 1 --2- 172.21.15.189:0/2057314039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f431415a550 0x7f43140cf3c0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f42fc0098f0 tx=0x7f42fc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1f0647852e925fc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.405+0000 7f431159a700 1 -- 172.21.15.189:0/2057314039 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42fc00b8a0 con 0x7f431415a550 2024-04-06T12:28:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.405+0000 7f431159a700 1 -- 172.21.15.189:0/2057314039 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42fc00ba00 con 0x7f431415a550 2024-04-06T12:28:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.405+0000 7f431159a700 1 -- 172.21.15.189:0/2057314039 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42fc00bd90 con 0x7f431415a550 2024-04-06T12:28:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.405+0000 7f4318c66700 1 -- 172.21.15.189:0/2057314039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f431415a550 msgr2=0x7f43140cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.405+0000 7f4318c66700 1 --2- 172.21.15.189:0/2057314039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f431415a550 0x7f43140cf3c0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f42fc0098f0 tx=0x7f42fc009c00 comp rx=0 tx=0).stop 2024-04-06T12:28:39.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.405+0000 7f4318c66700 1 -- 172.21.15.189:0/2057314039 shutdown_connections 2024-04-06T12:28:39.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.405+0000 7f4318c66700 1 --2- 172.21.15.189:0/2057314039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f431415a550 0x7f43140cf3c0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:39.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.405+0000 7f4318c66700 1 --2- 172.21.15.189:0/2057314039 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43140cf900 0x7f43140b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:39.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.405+0000 7f4318c66700 1 -- 172.21.15.189:0/2057314039 >> 172.21.15.189:0/2057314039 conn(0x7f43140d00b0 msgr2=0x7f43140d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:39.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.405+0000 7f4318c66700 1 -- 172.21.15.189:0/2057314039 shutdown_connections 2024-04-06T12:28:39.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.405+0000 7f4318c66700 1 -- 172.21.15.189:0/2057314039 wait complete. 2024-04-06T12:28:39.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.407+0000 7f4318c66700 1 Processor -- start 2024-04-06T12:28:39.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.407+0000 7f4318c66700 1 -- start start 2024-04-06T12:28:39.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.407+0000 7f4318c66700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43140b4ba0 0x7f43141f2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:39.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.407+0000 7f4318c66700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43140cf900 0x7f43141596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:39.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.407+0000 7f4318c66700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43141570d0 0x7f431406d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:39.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.407+0000 7f4318c66700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43140ad870 con 0x7f43140cf900 2024-04-06T12:28:39.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.407+0000 7f4318c66700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43140acbf0 con 0x7f43140b4ba0 2024-04-06T12:28:39.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.407+0000 7f4318c66700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43140aced0 con 0x7f43141570d0 2024-04-06T12:28:39.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.408+0000 7f4311d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43140cf900 0x7f43141596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:39.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.408+0000 7f4311d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43140cf900 0x7f43141596b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53312/0 (socket says 172.21.15.189:53312) 2024-04-06T12:28:39.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.408+0000 7f4311d9b700 1 -- 172.21.15.189:0/2063474779 learned_addr learned my addr 172.21.15.189:0/2063474779 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:39.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.408+0000 7f4312d9d700 1 --2- 172.21.15.189:0/2063474779 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43141570d0 0x7f431406d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:39.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.408+0000 7f431259c700 1 --2- 172.21.15.189:0/2063474779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43140b4ba0 0x7f43141f2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:39.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.408+0000 7f4311d9b700 1 -- 172.21.15.189:0/2063474779 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43141570d0 msgr2=0x7f431406d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:39.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.408+0000 7f4311d9b700 1 --2- 172.21.15.189:0/2063474779 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43141570d0 0x7f431406d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:39.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.408+0000 7f4311d9b700 1 -- 172.21.15.189:0/2063474779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43140b4ba0 msgr2=0x7f43141f2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:39.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.408+0000 7f4311d9b700 1 --2- 172.21.15.189:0/2063474779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43140b4ba0 0x7f43141f2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:39.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.408+0000 7f4311d9b700 1 -- 172.21.15.189:0/2063474779 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42fc009550 con 0x7f43140cf900 2024-04-06T12:28:39.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.408+0000 7f4311d9b700 1 --2- 172.21.15.189:0/2063474779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43140cf900 0x7f43141596b0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f430400e810 tx=0x7f430400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:39.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.408+0000 7f430affd700 1 -- 172.21.15.189:0/2063474779 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430400c070 con 0x7f43140cf900 2024-04-06T12:28:39.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.408+0000 7f430affd700 1 -- 172.21.15.189:0/2063474779 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4304004230 con 0x7f43140cf900 2024-04-06T12:28:39.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.408+0000 7f4318c66700 1 -- 172.21.15.189:0/2063474779 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43140ad1b0 con 0x7f43140cf900 2024-04-06T12:28:39.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.408+0000 7f430affd700 1 -- 172.21.15.189:0/2063474779 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43040036a0 con 0x7f43140cf900 2024-04-06T12:28:39.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.408+0000 7f4318c66700 1 -- 172.21.15.189:0/2063474779 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43141f2680 con 0x7f43140cf900 2024-04-06T12:28:39.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.409+0000 7f4318c66700 1 -- 172.21.15.189:0/2063474779 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43140ad9b0 con 0x7f43140cf900 2024-04-06T12:28:39.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.410+0000 7f430affd700 1 -- 172.21.15.189:0/2063474779 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f4304003800 con 0x7f43140cf900 2024-04-06T12:28:39.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.411+0000 7f430affd700 1 --2- 172.21.15.189:0/2063474779 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42f406adb0 0x7f42f406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:39.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.411+0000 7f430affd700 1 -- 172.21.15.189:0/2063474779 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f4304089540 con 0x7f43140cf900 2024-04-06T12:28:39.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.411+0000 7f431259c700 1 --2- 172.21.15.189:0/2063474779 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42f406adb0 0x7f42f406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:39.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.411+0000 7f431259c700 1 --2- 172.21.15.189:0/2063474779 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42f406adb0 0x7f42f406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f42fc014040 tx=0x7f42fc008030 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:39.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.415+0000 7f430affd700 1 -- 172.21.15.189:0/2063474779 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4304054610 con 0x7f43140cf900 2024-04-06T12:28:39.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:39.610+0000 7f4318c66700 1 -- 172.21.15.189:0/2063474779 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4314067bb0 con 0x7f43140cf900 2024-04-06T12:28:40.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.251+0000 7f430affd700 1 -- 172.21.15.189:0/2063474779 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v110) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4304057c30 con 0x7f43140cf900 2024-04-06T12:28:40.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.260+0000 7f4318c66700 1 -- 172.21.15.189:0/2063474779 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42f406adb0 msgr2=0x7f42f406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:40.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.260+0000 7f4318c66700 1 --2- 172.21.15.189:0/2063474779 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42f406adb0 0x7f42f406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f42fc014040 tx=0x7f42fc008030 comp rx=0 tx=0).stop 2024-04-06T12:28:40.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.260+0000 7f4318c66700 1 -- 172.21.15.189:0/2063474779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43140cf900 msgr2=0x7f43141596b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:40.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.260+0000 7f4318c66700 1 --2- 172.21.15.189:0/2063474779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43140cf900 0x7f43141596b0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f430400e810 tx=0x7f430400eb20 comp rx=0 tx=0).stop 2024-04-06T12:28:40.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.261+0000 7f4318c66700 1 -- 172.21.15.189:0/2063474779 shutdown_connections 2024-04-06T12:28:40.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.261+0000 7f4318c66700 1 --2- 172.21.15.189:0/2063474779 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43141570d0 0x7f431406d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.261+0000 7f4318c66700 1 --2- 172.21.15.189:0/2063474779 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f42f406adb0 0x7f42f406d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.261+0000 7f4318c66700 1 --2- 172.21.15.189:0/2063474779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43140b4ba0 0x7f43141f2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.261+0000 7f4318c66700 1 --2- 172.21.15.189:0/2063474779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43140cf900 0x7f43141596b0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.261+0000 7f4318c66700 1 -- 172.21.15.189:0/2063474779 >> 172.21.15.189:0/2063474779 conn(0x7f43140d00b0 msgr2=0x7f431406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:40.264 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.267+0000 7f4318c66700 1 -- 172.21.15.189:0/2063474779 shutdown_connections 2024-04-06T12:28:40.264 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.267+0000 7f4318c66700 1 -- 172.21.15.189:0/2063474779 wait complete. 2024-04-06T12:28:40.266 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' created 2024-04-06T12:28:40.279 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-06T12:28:40.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.410+0000 7fd40e030700 1 Processor -- start 2024-04-06T12:28:40.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.410+0000 7fd40e030700 1 -- start start 2024-04-06T12:28:40.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.411+0000 7fd40e030700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd4080d11b0 0x7fd4080d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:40.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.411+0000 7fd40e030700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd4080cfa00 0x7fd4080cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:40.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.411+0000 7fd40e030700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4080d1ad0 con 0x7fd4080cb870 2024-04-06T12:28:40.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.411+0000 7fd40e030700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4080ccdc0 con 0x7fd4080cfa00 2024-04-06T12:28:40.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.411+0000 7fd40e030700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4080d0b50 con 0x7fd4080d11b0 2024-04-06T12:28:40.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.411+0000 7fd406ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd4080cb870 0x7fd40815c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51422/0 (socket says 172.21.15.189:51422) 2024-04-06T12:28:40.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.411+0000 7fd407fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd4080cfa00 0x7fd4080cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:40.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.411+0000 7fd406ffd700 1 -- 172.21.15.189:0/910277524 learned_addr learned my addr 172.21.15.189:0/910277524 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:40.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.411+0000 7fd4077fe700 1 --2- 172.21.15.189:0/910277524 >> v2:172.21.15.192:3300/0 conn(0x7fd4080d11b0 0x7fd4080d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:40.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.411+0000 7fd4067fc700 1 -- 172.21.15.189:0/910277524 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225248642 0 0) 0x7fd4080d1ad0 con 0x7fd4080cb870 2024-04-06T12:28:40.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.411+0000 7fd4067fc700 1 -- 172.21.15.189:0/910277524 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f00035b0 con 0x7fd4080cb870 2024-04-06T12:28:40.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.411+0000 7fd407fff700 1 -- 172.21.15.189:0/910277524 >> v2:172.21.15.192:3300/0 conn(0x7fd4080d11b0 msgr2=0x7fd4080d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:40.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.411+0000 7fd407fff700 1 --2- 172.21.15.189:0/910277524 >> v2:172.21.15.192:3300/0 conn(0x7fd4080d11b0 0x7fd4080d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.411+0000 7fd407fff700 1 -- 172.21.15.189:0/910277524 >> v1:172.21.15.189:6789/0 conn(0x7fd4080cb870 legacy=0x7fd40815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:40.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.411+0000 7fd407fff700 1 -- 172.21.15.189:0/910277524 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4080cbd80 con 0x7fd4080cfa00 2024-04-06T12:28:40.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.412+0000 7fd407fff700 1 --2- 172.21.15.189:0/910277524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4080cfa00 0x7fd4080cd560 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fd4080ced00 tx=0x7fd40000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d64fb712bf6ce9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:40.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.412+0000 7fd4067fc700 1 -- 172.21.15.189:0/910277524 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40000e070 con 0x7fd4080cfa00 2024-04-06T12:28:40.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.412+0000 7fd4067fc700 1 -- 172.21.15.189:0/910277524 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40000bb20 con 0x7fd4080cfa00 2024-04-06T12:28:40.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.412+0000 7fd4067fc700 1 -- 172.21.15.189:0/910277524 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40000be90 con 0x7fd4080cfa00 2024-04-06T12:28:40.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.412+0000 7fd40e030700 1 -- 172.21.15.189:0/910277524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4080cfa00 msgr2=0x7fd4080cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:40.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.412+0000 7fd40e030700 1 --2- 172.21.15.189:0/910277524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4080cfa00 0x7fd4080cd560 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fd4080ced00 tx=0x7fd40000b330 comp rx=0 tx=0).stop 2024-04-06T12:28:40.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.412+0000 7fd40e030700 1 -- 172.21.15.189:0/910277524 shutdown_connections 2024-04-06T12:28:40.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.412+0000 7fd40e030700 1 --2- 172.21.15.189:0/910277524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4080cfa00 0x7fd4080cd560 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.412+0000 7fd40e030700 1 --2- 172.21.15.189:0/910277524 >> v2:172.21.15.192:3300/0 conn(0x7fd4080d11b0 0x7fd4080d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.412+0000 7fd40e030700 1 -- 172.21.15.189:0/910277524 >> 172.21.15.189:0/910277524 conn(0x7fd4080b2db0 msgr2=0x7fd4080b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:40.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.412+0000 7fd40e030700 1 -- 172.21.15.189:0/910277524 shutdown_connections 2024-04-06T12:28:40.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.412+0000 7fd40e030700 1 -- 172.21.15.189:0/910277524 wait complete. 2024-04-06T12:28:40.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.413+0000 7fd40e030700 1 Processor -- start 2024-04-06T12:28:40.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.414+0000 7fd40e030700 1 -- start start 2024-04-06T12:28:40.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.414+0000 7fd40e030700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4080cb870 0x7fd4080c3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:40.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.414+0000 7fd40e030700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4080cfa00 0x7fd408147b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:40.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.414+0000 7fd40e030700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4080d11b0 0x7fd4080b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:40.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.414+0000 7fd40e030700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080c42c0 con 0x7fd4080cfa00 2024-04-06T12:28:40.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.414+0000 7fd40e030700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080b4040 con 0x7fd4080d11b0 2024-04-06T12:28:40.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.414+0000 7fd40e030700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080b4320 con 0x7fd4080cb870 2024-04-06T12:28:40.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.414+0000 7fd406ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4080cfa00 0x7fd408147b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:40.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.415+0000 7fd406ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4080cfa00 0x7fd408147b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53324/0 (socket says 172.21.15.189:53324) 2024-04-06T12:28:40.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.415+0000 7fd406ffd700 1 -- 172.21.15.189:0/2170029878 learned_addr learned my addr 172.21.15.189:0/2170029878 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:40.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.415+0000 7fd4077fe700 1 --2- 172.21.15.189:0/2170029878 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4080cb870 0x7fd4080c3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:40.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.415+0000 7fd407fff700 1 --2- 172.21.15.189:0/2170029878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4080d11b0 0x7fd4080b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:40.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.415+0000 7fd406ffd700 1 -- 172.21.15.189:0/2170029878 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4080cb870 msgr2=0x7fd4080c3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:40.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.415+0000 7fd406ffd700 1 --2- 172.21.15.189:0/2170029878 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4080cb870 0x7fd4080c3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.415+0000 7fd406ffd700 1 -- 172.21.15.189:0/2170029878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4080d11b0 msgr2=0x7fd4080b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:40.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.415+0000 7fd406ffd700 1 --2- 172.21.15.189:0/2170029878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4080d11b0 0x7fd4080b50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.415+0000 7fd406ffd700 1 -- 172.21.15.189:0/2170029878 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd400009a70 con 0x7fd4080cfa00 2024-04-06T12:28:40.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.415+0000 7fd406ffd700 1 --2- 172.21.15.189:0/2170029878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4080cfa00 0x7fd408147b00 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800c400 tx=0x7fd3f800c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:40.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.415+0000 7fd3effff700 1 -- 172.21.15.189:0/2170029878 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f8005d70 con 0x7fd4080cfa00 2024-04-06T12:28:40.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.415+0000 7fd3effff700 1 -- 172.21.15.189:0/2170029878 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f800ce10 con 0x7fd4080cfa00 2024-04-06T12:28:40.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.415+0000 7fd3effff700 1 -- 172.21.15.189:0/2170029878 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f8007310 con 0x7fd4080cfa00 2024-04-06T12:28:40.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.415+0000 7fd40e030700 1 -- 172.21.15.189:0/2170029878 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4080b4600 con 0x7fd4080cfa00 2024-04-06T12:28:40.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.415+0000 7fd40e030700 1 -- 172.21.15.189:0/2170029878 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4080b4bc0 con 0x7fd4080cfa00 2024-04-06T12:28:40.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.416+0000 7fd40e030700 1 -- 172.21.15.189:0/2170029878 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd408074990 con 0x7fd4080cfa00 2024-04-06T12:28:40.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.417+0000 7fd3effff700 1 -- 172.21.15.189:0/2170029878 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fd3f8007470 con 0x7fd4080cfa00 2024-04-06T12:28:40.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.417+0000 7fd3effff700 1 --2- 172.21.15.189:0/2170029878 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3f406adb0 0x7fd3f406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:40.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.417+0000 7fd3effff700 1 -- 172.21.15.189:0/2170029878 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5929+0+0 (secure 0 0 0) 0x7fd3f8088910 con 0x7fd4080cfa00 2024-04-06T12:28:40.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.417+0000 7fd4077fe700 1 --2- 172.21.15.189:0/2170029878 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3f406adb0 0x7fd3f406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:40.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.418+0000 7fd4077fe700 1 --2- 172.21.15.189:0/2170029878 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3f406adb0 0x7fd3f406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc009be0 tx=0x7fd3fc009210 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:40.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.422+0000 7fd3effff700 1 -- 172.21.15.189:0/2170029878 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd3f80539c0 con 0x7fd4080cfa00 2024-04-06T12:28:40.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.658+0000 7fd40e030700 1 -- 172.21.15.189:0/2170029878 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd40815fd50 con 0x7fd4080cfa00 2024-04-06T12:28:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.664+0000 7fd3effff700 1 -- 172.21.15.189:0/2170029878 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 13 and data pool 14 v43) v1 ==== 174+0+0 (secure 0 0 0) 0x7fd3f8017020 con 0x7fd4080cfa00 2024-04-06T12:28:40.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.671+0000 7fd40e030700 1 -- 172.21.15.189:0/2170029878 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3f406adb0 msgr2=0x7fd3f406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:40.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.671+0000 7fd40e030700 1 --2- 172.21.15.189:0/2170029878 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3f406adb0 0x7fd3f406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc009be0 tx=0x7fd3fc009210 comp rx=0 tx=0).stop 2024-04-06T12:28:40.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.671+0000 7fd40e030700 1 -- 172.21.15.189:0/2170029878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4080cfa00 msgr2=0x7fd408147b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:40.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.671+0000 7fd40e030700 1 --2- 172.21.15.189:0/2170029878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4080cfa00 0x7fd408147b00 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800c400 tx=0x7fd3f800c710 comp rx=0 tx=0).stop 2024-04-06T12:28:40.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.671+0000 7fd40e030700 1 -- 172.21.15.189:0/2170029878 shutdown_connections 2024-04-06T12:28:40.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.671+0000 7fd40e030700 1 --2- 172.21.15.189:0/2170029878 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd3f406adb0 0x7fd3f406d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.671+0000 7fd40e030700 1 --2- 172.21.15.189:0/2170029878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4080d11b0 0x7fd4080b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.671+0000 7fd40e030700 1 --2- 172.21.15.189:0/2170029878 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4080cb870 0x7fd4080c3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.671+0000 7fd40e030700 1 --2- 172.21.15.189:0/2170029878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4080cfa00 0x7fd408147b00 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.671+0000 7fd40e030700 1 -- 172.21.15.189:0/2170029878 >> 172.21.15.189:0/2170029878 conn(0x7fd4080b2db0 msgr2=0x7fd4080b2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:40.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.672+0000 7fd40e030700 1 -- 172.21.15.189:0/2170029878 shutdown_connections 2024-04-06T12:28:40.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.672+0000 7fd40e030700 1 -- 172.21.15.189:0/2170029878 wait complete. 2024-04-06T12:28:40.690 INFO:teuthology.orchestra.run.smithi189.stderr:new fs with metadata pool 13 and data pool 14 2024-04-06T12:28:40.700 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:28:40.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.824+0000 7f09df9a6700 1 Processor -- start 2024-04-06T12:28:40.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.825+0000 7f09df9a6700 1 -- start start 2024-04-06T12:28:40.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.825+0000 7f09df9a6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f09d815a550 0x7f09d80cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:40.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.825+0000 7f09df9a6700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f09d80cf900 0x7f09d80b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:40.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.825+0000 7f09df9a6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09d80b4ba0 0x7f09d806eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:40.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.825+0000 7f09df9a6700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d80cdbe0 con 0x7f09d815a550 2024-04-06T12:28:40.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.825+0000 7f09df9a6700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d80b4ff0 con 0x7f09d80cf900 2024-04-06T12:28:40.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.825+0000 7f09df9a6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d815a940 con 0x7f09d80b4ba0 2024-04-06T12:28:40.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.825+0000 7f09dd742700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f09d815a550 0x7f09d80cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:40.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.825+0000 7f09dd742700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f09d815a550 0x7f09d80cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53330/0 (socket says 172.21.15.189:53330) 2024-04-06T12:28:40.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.825+0000 7f09dd742700 1 -- 172.21.15.189:0/1630815503 learned_addr learned my addr 172.21.15.189:0/1630815503 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:40.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.826+0000 7f09dcf41700 1 --2- 172.21.15.189:0/1630815503 >> v2:172.21.15.192:3300/0 conn(0x7f09d80cf900 0x7f09d80b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:40.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.826+0000 7f09ddf43700 1 --2- 172.21.15.189:0/1630815503 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09d80b4ba0 0x7f09d806eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:40.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.826+0000 7f09dd742700 1 -- 172.21.15.189:0/1630815503 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09d80b4ba0 msgr2=0x7f09d806eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:40.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.826+0000 7f09dd742700 1 --2- 172.21.15.189:0/1630815503 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09d80b4ba0 0x7f09d806eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.826+0000 7f09dd742700 1 -- 172.21.15.189:0/1630815503 >> v2:172.21.15.192:3300/0 conn(0x7f09d80cf900 msgr2=0x7f09d80b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:40.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.826+0000 7f09dd742700 1 --2- 172.21.15.189:0/1630815503 >> v2:172.21.15.192:3300/0 conn(0x7f09d80cf900 0x7f09d80b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.826+0000 7f09dd742700 1 -- 172.21.15.189:0/1630815503 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d815ac00 con 0x7f09d815a550 2024-04-06T12:28:40.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.826+0000 7f09dd742700 1 --2- 172.21.15.189:0/1630815503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09d815a550 0x7f09d80cf3c0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f09d80b2d40 tx=0x7f09d4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=642e7f474093ce87 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:40.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.826+0000 7f09cbfff700 1 -- 172.21.15.189:0/1630815503 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d400b870 con 0x7f09d815a550 2024-04-06T12:28:40.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.826+0000 7f09cbfff700 1 -- 172.21.15.189:0/1630815503 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d400b9d0 con 0x7f09d815a550 2024-04-06T12:28:40.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.826+0000 7f09cbfff700 1 -- 172.21.15.189:0/1630815503 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d400bd60 con 0x7f09d815a550 2024-04-06T12:28:40.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.826+0000 7f09df9a6700 1 -- 172.21.15.189:0/1630815503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09d815a550 msgr2=0x7f09d80cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:40.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.827+0000 7f09df9a6700 1 --2- 172.21.15.189:0/1630815503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09d815a550 0x7f09d80cf3c0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f09d80b2d40 tx=0x7f09d4009bd0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.827+0000 7f09df9a6700 1 -- 172.21.15.189:0/1630815503 shutdown_connections 2024-04-06T12:28:40.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.827+0000 7f09df9a6700 1 --2- 172.21.15.189:0/1630815503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09d815a550 0x7f09d80cf3c0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.827+0000 7f09df9a6700 1 --2- 172.21.15.189:0/1630815503 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09d80b4ba0 0x7f09d806eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.827+0000 7f09df9a6700 1 --2- 172.21.15.189:0/1630815503 >> v2:172.21.15.192:3300/0 conn(0x7f09d80cf900 0x7f09d80b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.827+0000 7f09df9a6700 1 -- 172.21.15.189:0/1630815503 >> 172.21.15.189:0/1630815503 conn(0x7f09d80d00b0 msgr2=0x7f09d80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:40.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.827+0000 7f09df9a6700 1 -- 172.21.15.189:0/1630815503 shutdown_connections 2024-04-06T12:28:40.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.827+0000 7f09df9a6700 1 -- 172.21.15.189:0/1630815503 wait complete. 2024-04-06T12:28:40.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.828+0000 7f09df9a6700 1 Processor -- start 2024-04-06T12:28:40.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.828+0000 7f09df9a6700 1 -- start start 2024-04-06T12:28:40.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09df9a6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09d80b4ba0 0x7f09d8151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:40.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09df9a6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09d80cf900 0x7f09d814e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:40.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09df9a6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09d814f780 0x7f09d814fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:40.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09df9a6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d8150350 con 0x7f09d814f780 2024-04-06T12:28:40.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09df9a6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d814fc10 con 0x7f09d80cf900 2024-04-06T12:28:40.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09df9a6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d8150560 con 0x7f09d80b4ba0 2024-04-06T12:28:40.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09ddf43700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09d814f780 0x7f09d814fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:40.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09dd742700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09d80b4ba0 0x7f09d8151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:40.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09ddf43700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09d814f780 0x7f09d814fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53338/0 (socket says 172.21.15.189:53338) 2024-04-06T12:28:40.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09dd742700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09d80b4ba0 0x7f09d8151600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:52438/0 (socket says 172.21.15.189:52438) 2024-04-06T12:28:40.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09ddf43700 1 -- 172.21.15.189:0/2734415922 learned_addr learned my addr 172.21.15.189:0/2734415922 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:40.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09dcf41700 1 --2- 172.21.15.189:0/2734415922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09d80cf900 0x7f09d814e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:40.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09ddf43700 1 -- 172.21.15.189:0/2734415922 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09d80b4ba0 msgr2=0x7f09d8151600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:40.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09ddf43700 1 --2- 172.21.15.189:0/2734415922 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09d80b4ba0 0x7f09d8151600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09ddf43700 1 -- 172.21.15.189:0/2734415922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09d80cf900 msgr2=0x7f09d814e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:40.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09ddf43700 1 --2- 172.21.15.189:0/2734415922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09d80cf900 0x7f09d814e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:40.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09ddf43700 1 -- 172.21.15.189:0/2734415922 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d4009550 con 0x7f09d814f780 2024-04-06T12:28:40.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09ddf43700 1 --2- 172.21.15.189:0/2734415922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09d814f780 0x7f09d814fe10 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f09d0007d20 tx=0x7f09d000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:40.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09c9ffb700 1 -- 172.21.15.189:0/2734415922 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d000ed30 con 0x7f09d814f780 2024-04-06T12:28:40.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09c9ffb700 1 -- 172.21.15.189:0/2734415922 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d0007e70 con 0x7f09d814f780 2024-04-06T12:28:40.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09df9a6700 1 -- 172.21.15.189:0/2734415922 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d8150840 con 0x7f09d814f780 2024-04-06T12:28:40.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.829+0000 7f09c9ffb700 1 -- 172.21.15.189:0/2734415922 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d0014640 con 0x7f09d814f780 2024-04-06T12:28:40.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.830+0000 7f09df9a6700 1 -- 172.21.15.189:0/2734415922 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d8150e00 con 0x7f09d814f780 2024-04-06T12:28:40.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.830+0000 7f09df9a6700 1 -- 172.21.15.189:0/2734415922 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d8067bb0 con 0x7f09d814f780 2024-04-06T12:28:40.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.831+0000 7f09c9ffb700 1 -- 172.21.15.189:0/2734415922 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f09d00147a0 con 0x7f09d814f780 2024-04-06T12:28:40.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.831+0000 7f09c9ffb700 1 --2- 172.21.15.189:0/2734415922 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f09bc06adb0 0x7f09bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:40.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.831+0000 7f09c9ffb700 1 -- 172.21.15.189:0/2734415922 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f09d0089b80 con 0x7f09d814f780 2024-04-06T12:28:40.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.831+0000 7f09dd742700 1 --2- 172.21.15.189:0/2734415922 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f09bc06adb0 0x7f09bc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:40.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.832+0000 7f09dd742700 1 --2- 172.21.15.189:0/2734415922 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f09bc06adb0 0x7f09bc06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f09d4000c00 tx=0x7f09d4008030 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:40.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:40.836+0000 7f09c9ffb700 1 -- 172.21.15.189:0/2734415922 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f09d0054b00 con 0x7f09d814f780 2024-04-06T12:28:41.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.033+0000 7f09df9a6700 1 -- 172.21.15.189:0/2734415922 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09d8074990 con 0x7f09d814f780 2024-04-06T12:28:41.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.034+0000 7f09c9ffb700 1 -- 172.21.15.189:0/2734415922 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16321 (secure 0 0 0) 0x7f09d0089e60 con 0x7f09d814f780 2024-04-06T12:28:41.031 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:41.031 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":111,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:28:40.659648+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:28:38.415021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-06T12:28:39.629378+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:28:41.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.036+0000 7f09df9a6700 1 -- 172.21.15.189:0/2734415922 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f09bc06adb0 msgr2=0x7f09bc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:41.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.036+0000 7f09df9a6700 1 --2- 172.21.15.189:0/2734415922 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f09bc06adb0 0x7f09bc06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f09d4000c00 tx=0x7f09d4008030 comp rx=0 tx=0).stop 2024-04-06T12:28:41.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.036+0000 7f09df9a6700 1 -- 172.21.15.189:0/2734415922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09d814f780 msgr2=0x7f09d814fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:41.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.036+0000 7f09df9a6700 1 --2- 172.21.15.189:0/2734415922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09d814f780 0x7f09d814fe10 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f09d0007d20 tx=0x7f09d000e3c0 comp rx=0 tx=0).stop 2024-04-06T12:28:41.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.036+0000 7f09df9a6700 1 -- 172.21.15.189:0/2734415922 shutdown_connections 2024-04-06T12:28:41.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.036+0000 7f09df9a6700 1 --2- 172.21.15.189:0/2734415922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09d814f780 0x7f09d814fe10 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:41.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.036+0000 7f09df9a6700 1 --2- 172.21.15.189:0/2734415922 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09d80b4ba0 0x7f09d8151600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:41.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.036+0000 7f09df9a6700 1 --2- 172.21.15.189:0/2734415922 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f09bc06adb0 0x7f09bc06d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:41.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.036+0000 7f09df9a6700 1 --2- 172.21.15.189:0/2734415922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09d80cf900 0x7f09d814e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:41.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.036+0000 7f09df9a6700 1 -- 172.21.15.189:0/2734415922 >> 172.21.15.189:0/2734415922 conn(0x7f09d80d00b0 msgr2=0x7f09d806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:41.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.036+0000 7f09df9a6700 1 -- 172.21.15.189:0/2734415922 shutdown_connections 2024-04-06T12:28:41.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.036+0000 7f09df9a6700 1 -- 172.21.15.189:0/2734415922 wait complete. 2024-04-06T12:28:41.046 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:28:41.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.171+0000 7f067b09f700 1 Processor -- start 2024-04-06T12:28:41.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.171+0000 7f067b09f700 1 -- start start 2024-04-06T12:28:41.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.172+0000 7f067b09f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f067406ae40 0x7f06740b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:41.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.172+0000 7f067b09f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f067415ddf0 0x7f067415e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:41.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.172+0000 7f067b09f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06740b3990 con 0x7f067415d7b0 2024-04-06T12:28:41.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.172+0000 7f067b09f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06740b3060 con 0x7f067415ddf0 2024-04-06T12:28:41.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.172+0000 7f067b09f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06740b3340 con 0x7f067406ae40 2024-04-06T12:28:41.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.172+0000 7f0678e3b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f067415d7b0 0x7f067406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51430/0 (socket says 172.21.15.189:51430) 2024-04-06T12:28:41.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.172+0000 7f0678e3b700 1 -- 172.21.15.189:0/674188160 learned_addr learned my addr 172.21.15.189:0/674188160 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:41.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.172+0000 7f067963c700 1 --2- 172.21.15.189:0/674188160 >> v2:172.21.15.192:3300/0 conn(0x7f067415ddf0 0x7f067415e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:41.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.172+0000 7f0673fff700 1 --2- 172.21.15.189:0/674188160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f067406ae40 0x7f06740b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:41.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.172+0000 7f06737fe700 1 -- 172.21.15.189:0/674188160 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684959321 0 0) 0x7f06740b3990 con 0x7f067415d7b0 2024-04-06T12:28:41.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.172+0000 7f06737fe700 1 -- 172.21.15.189:0/674188160 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06640035b0 con 0x7f067415d7b0 2024-04-06T12:28:41.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.173+0000 7f06737fe700 1 -- 172.21.15.189:0/674188160 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2656489271 0 0) 0x7f06640035b0 con 0x7f067415d7b0 2024-04-06T12:28:41.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.173+0000 7f067963c700 1 -- 172.21.15.189:0/674188160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f067406ae40 msgr2=0x7f06740b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:41.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.173+0000 7f067963c700 1 --2- 172.21.15.189:0/674188160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f067406ae40 0x7f06740b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:41.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.173+0000 7f067963c700 1 -- 172.21.15.189:0/674188160 >> v1:172.21.15.189:6789/0 conn(0x7f067415d7b0 legacy=0x7f067406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:41.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.173+0000 7f067963c700 1 -- 172.21.15.189:0/674188160 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06740b3620 con 0x7f067415ddf0 2024-04-06T12:28:41.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.173+0000 7f0673fff700 1 --2- 172.21.15.189:0/674188160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f067406ae40 0x7f06740b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:28:41.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.173+0000 7f067963c700 1 --2- 172.21.15.189:0/674188160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f067415ddf0 0x7f067415e2a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f066c009d90 tx=0x7f066c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=280c0946bc3a6e41 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:41.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.173+0000 7f06737fe700 1 -- 172.21.15.189:0/674188160 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c00e070 con 0x7f067415ddf0 2024-04-06T12:28:41.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.173+0000 7f06737fe700 1 -- 172.21.15.189:0/674188160 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066c003a20 con 0x7f067415ddf0 2024-04-06T12:28:41.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.173+0000 7f06737fe700 1 -- 172.21.15.189:0/674188160 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0668001670 con 0x7f067415ddf0 2024-04-06T12:28:41.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.174+0000 7f067b09f700 1 -- 172.21.15.189:0/674188160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f067415ddf0 msgr2=0x7f067415e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:41.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.174+0000 7f067b09f700 1 --2- 172.21.15.189:0/674188160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f067415ddf0 0x7f067415e2a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f066c009d90 tx=0x7f066c00b330 comp rx=0 tx=0).stop 2024-04-06T12:28:41.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.174+0000 7f067b09f700 1 -- 172.21.15.189:0/674188160 shutdown_connections 2024-04-06T12:28:41.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.174+0000 7f067b09f700 1 --2- 172.21.15.189:0/674188160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f067415ddf0 0x7f067415e2a0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:41.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.174+0000 7f067b09f700 1 --2- 172.21.15.189:0/674188160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f067406ae40 0x7f06740b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:41.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.174+0000 7f067b09f700 1 -- 172.21.15.189:0/674188160 >> 172.21.15.189:0/674188160 conn(0x7f06740d00b0 msgr2=0x7f06740d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:41.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.174+0000 7f067b09f700 1 -- 172.21.15.189:0/674188160 shutdown_connections 2024-04-06T12:28:41.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.174+0000 7f067b09f700 1 -- 172.21.15.189:0/674188160 wait complete. 2024-04-06T12:28:41.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.175+0000 7f067b09f700 1 Processor -- start 2024-04-06T12:28:41.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.175+0000 7f067b09f700 1 -- start start 2024-04-06T12:28:41.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.176+0000 7f067b09f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f067406ae40 0x7f067414a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:41.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.176+0000 7f067b09f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f067415d7b0 0x7f06741589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:41.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.176+0000 7f067b09f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f067415ddf0 0x7f0674149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:41.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.176+0000 7f067b09f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06740cd2f0 con 0x7f067415d7b0 2024-04-06T12:28:41.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.176+0000 7f067b09f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06740cd5d0 con 0x7f067406ae40 2024-04-06T12:28:41.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.176+0000 7f067b09f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06740cd8b0 con 0x7f067415ddf0 2024-04-06T12:28:41.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.176+0000 7f0673fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f067415d7b0 0x7f06741589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:41.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.176+0000 7f0673fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f067415d7b0 0x7f06741589a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53352/0 (socket says 172.21.15.189:53352) 2024-04-06T12:28:41.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.176+0000 7f0673fff700 1 -- 172.21.15.189:0/646013318 learned_addr learned my addr 172.21.15.189:0/646013318 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:41.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.176+0000 7f0678e3b700 1 --2- 172.21.15.189:0/646013318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f067406ae40 0x7f067414a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:41.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.176+0000 7f067963c700 1 --2- 172.21.15.189:0/646013318 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f067415ddf0 0x7f0674149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:41.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.176+0000 7f0673fff700 1 -- 172.21.15.189:0/646013318 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f067415ddf0 msgr2=0x7f0674149c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:41.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.176+0000 7f0673fff700 1 --2- 172.21.15.189:0/646013318 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f067415ddf0 0x7f0674149c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:41.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.176+0000 7f0673fff700 1 -- 172.21.15.189:0/646013318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f067406ae40 msgr2=0x7f067414a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:41.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.176+0000 7f0673fff700 1 --2- 172.21.15.189:0/646013318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f067406ae40 0x7f067414a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:41.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.176+0000 7f0673fff700 1 -- 172.21.15.189:0/646013318 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06600094a0 con 0x7f067415d7b0 2024-04-06T12:28:41.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.176+0000 7f0678e3b700 1 --2- 172.21.15.189:0/646013318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f067406ae40 0x7f067414a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:28:41.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.176+0000 7f0673fff700 1 --2- 172.21.15.189:0/646013318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f067415d7b0 0x7f06741589a0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f066000ebb0 tx=0x7f066000ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:41.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.176+0000 7f06717fa700 1 -- 172.21.15.189:0/646013318 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066000c910 con 0x7f067415d7b0 2024-04-06T12:28:41.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.177+0000 7f06717fa700 1 -- 172.21.15.189:0/646013318 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066000ca70 con 0x7f067415d7b0 2024-04-06T12:28:41.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.177+0000 7f067b09f700 1 -- 172.21.15.189:0/646013318 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f066c009a70 con 0x7f067415d7b0 2024-04-06T12:28:41.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.177+0000 7f06717fa700 1 -- 172.21.15.189:0/646013318 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066000ce00 con 0x7f067415d7b0 2024-04-06T12:28:41.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.177+0000 7f067b09f700 1 -- 172.21.15.189:0/646013318 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0674159030 con 0x7f067415d7b0 2024-04-06T12:28:41.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.178+0000 7f067b09f700 1 -- 172.21.15.189:0/646013318 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06740cddd0 con 0x7f067415d7b0 2024-04-06T12:28:41.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.178+0000 7f06717fa700 1 -- 172.21.15.189:0/646013318 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0660013470 con 0x7f067415d7b0 2024-04-06T12:28:41.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.178+0000 7f06717fa700 1 --2- 172.21.15.189:0/646013318 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f065806adb0 0x7f065806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:41.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.179+0000 7f06717fa700 1 -- 172.21.15.189:0/646013318 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0660089ca0 con 0x7f067415d7b0 2024-04-06T12:28:41.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.179+0000 7f0678e3b700 1 --2- 172.21.15.189:0/646013318 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f065806adb0 0x7f065806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:41.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.179+0000 7f0678e3b700 1 --2- 172.21.15.189:0/646013318 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f065806adb0 0x7f065806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0668007960 tx=0x7f0668009040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:41.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.183+0000 7f06717fa700 1 -- 172.21.15.189:0/646013318 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f06740cddd0 con 0x7f067415d7b0 2024-04-06T12:28:41.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.377+0000 7f067b09f700 1 -- 172.21.15.189:0/646013318 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06740cddd0 con 0x7f067415d7b0 2024-04-06T12:28:41.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.379+0000 7f06717fa700 1 -- 172.21.15.189:0/646013318 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16321 (secure 0 0 0) 0x7f06740cddd0 con 0x7f067415d7b0 2024-04-06T12:28:41.376 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:41.376 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":111,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:28:40.659648+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:28:38.415021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-06T12:28:39.629378+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:28:41.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.381+0000 7f067b09f700 1 -- 172.21.15.189:0/646013318 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f065806adb0 msgr2=0x7f065806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:41.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.381+0000 7f067b09f700 1 --2- 172.21.15.189:0/646013318 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f065806adb0 0x7f065806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0668007960 tx=0x7f0668009040 comp rx=0 tx=0).stop 2024-04-06T12:28:41.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.381+0000 7f067b09f700 1 -- 172.21.15.189:0/646013318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f067415d7b0 msgr2=0x7f06741589a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:41.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.381+0000 7f067b09f700 1 --2- 172.21.15.189:0/646013318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f067415d7b0 0x7f06741589a0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f066000ebb0 tx=0x7f066000ef70 comp rx=0 tx=0).stop 2024-04-06T12:28:41.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.381+0000 7f067b09f700 1 -- 172.21.15.189:0/646013318 shutdown_connections 2024-04-06T12:28:41.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.381+0000 7f067b09f700 1 --2- 172.21.15.189:0/646013318 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f067415ddf0 0x7f0674149c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:41.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.381+0000 7f067b09f700 1 --2- 172.21.15.189:0/646013318 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f065806adb0 0x7f065806d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:41.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.381+0000 7f067b09f700 1 --2- 172.21.15.189:0/646013318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f067406ae40 0x7f067414a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:41.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.381+0000 7f067b09f700 1 --2- 172.21.15.189:0/646013318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f067415d7b0 0x7f06741589a0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:41.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.381+0000 7f067b09f700 1 -- 172.21.15.189:0/646013318 >> 172.21.15.189:0/646013318 conn(0x7f06740d00b0 msgr2=0x7f067415f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:41.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.381+0000 7f067b09f700 1 -- 172.21.15.189:0/646013318 shutdown_connections 2024-04-06T12:28:41.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.381+0000 7f067b09f700 1 -- 172.21.15.189:0/646013318 wait complete. 2024-04-06T12:28:41.390 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-06T12:28:41.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.518+0000 7f62dbf4a700 1 Processor -- start 2024-04-06T12:28:41.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.518+0000 7f62dbf4a700 1 -- start start 2024-04-06T12:28:41.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.519+0000 7f62dbf4a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62d415a550 0x7f62d40cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.519+0000 7f62dbf4a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f62d40b4d70 0x7f62d40b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.519+0000 7f62dbf4a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d40cdbe0 con 0x7f62d40cf900 2024-04-06T12:28:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.519+0000 7f62dbf4a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62d415a940 con 0x7f62d40b4d70 2024-04-06T12:28:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.519+0000 7f62dbf4a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d415ac00 con 0x7f62d415a550 2024-04-06T12:28:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.520+0000 7f62da4e7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f62d40b4d70 0x7f62d40b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.520+0000 7f62d94e5700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f62d40cf900 0x7f62d40b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51444/0 (socket says 172.21.15.189:51444) 2024-04-06T12:28:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.520+0000 7f62d94e5700 1 -- 172.21.15.189:0/3349688353 learned_addr learned my addr 172.21.15.189:0/3349688353 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.520+0000 7f62d9ce6700 1 --2- 172.21.15.189:0/3349688353 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62d415a550 0x7f62d40cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.520+0000 7f62da4e7700 1 -- 172.21.15.189:0/3349688353 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62d415a550 msgr2=0x7f62d40cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.520+0000 7f62da4e7700 1 --2- 172.21.15.189:0/3349688353 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62d415a550 0x7f62d40cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.520+0000 7f62da4e7700 1 -- 172.21.15.189:0/3349688353 >> v1:172.21.15.189:6789/0 conn(0x7f62d40cf900 legacy=0x7f62d40b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.520+0000 7f62da4e7700 1 -- 172.21.15.189:0/3349688353 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d415aee0 con 0x7f62d40b4d70 2024-04-06T12:28:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.520+0000 7f62da4e7700 1 --2- 172.21.15.189:0/3349688353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62d40b4d70 0x7f62d40b2fc0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00bfa0 tx=0x7f62cc00d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea62d7dc552bca4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.520+0000 7f62d8ce4700 1 -- 172.21.15.189:0/3349688353 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc010070 con 0x7f62d40b4d70 2024-04-06T12:28:41.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.520+0000 7f62d8ce4700 1 -- 172.21.15.189:0/3349688353 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62cc00b2e0 con 0x7f62d40b4d70 2024-04-06T12:28:41.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.520+0000 7f62d8ce4700 1 -- 172.21.15.189:0/3349688353 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc00dcf0 con 0x7f62d40b4d70 2024-04-06T12:28:41.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.521+0000 7f62dbf4a700 1 -- 172.21.15.189:0/3349688353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62d40b4d70 msgr2=0x7f62d40b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:41.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.521+0000 7f62dbf4a700 1 --2- 172.21.15.189:0/3349688353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62d40b4d70 0x7f62d40b2fc0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00bfa0 tx=0x7f62cc00d330 comp rx=0 tx=0).stop 2024-04-06T12:28:41.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.521+0000 7f62dbf4a700 1 -- 172.21.15.189:0/3349688353 shutdown_connections 2024-04-06T12:28:41.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.521+0000 7f62dbf4a700 1 --2- 172.21.15.189:0/3349688353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62d40b4d70 0x7f62d40b2fc0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:41.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.521+0000 7f62dbf4a700 1 --2- 172.21.15.189:0/3349688353 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62d415a550 0x7f62d40cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:41.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.521+0000 7f62dbf4a700 1 -- 172.21.15.189:0/3349688353 >> 172.21.15.189:0/3349688353 conn(0x7f62d40d00b0 msgr2=0x7f62d40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:41.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.521+0000 7f62dbf4a700 1 -- 172.21.15.189:0/3349688353 shutdown_connections 2024-04-06T12:28:41.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.521+0000 7f62dbf4a700 1 -- 172.21.15.189:0/3349688353 wait complete. 2024-04-06T12:28:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.522+0000 7f62dbf4a700 1 Processor -- start 2024-04-06T12:28:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.522+0000 7f62dbf4a700 1 -- start start 2024-04-06T12:28:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.523+0000 7f62dbf4a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62d40b4d70 0x7f62d414fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.523+0000 7f62dbf4a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62d40cf900 0x7f62d415d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.523+0000 7f62dbf4a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62d40a84b0 0x7f62d40a8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.523+0000 7f62dbf4a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d41f2920 con 0x7f62d40cf900 2024-04-06T12:28:41.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.523+0000 7f62dbf4a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d415d6e0 con 0x7f62d40a84b0 2024-04-06T12:28:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.523+0000 7f62dbf4a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d406d610 con 0x7f62d40b4d70 2024-04-06T12:28:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.523+0000 7f62d94e5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62d40cf900 0x7f62d415d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.523+0000 7f62d94e5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62d40cf900 0x7f62d415d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53366/0 (socket says 172.21.15.189:53366) 2024-04-06T12:28:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.523+0000 7f62d94e5700 1 -- 172.21.15.189:0/3672851067 learned_addr learned my addr 172.21.15.189:0/3672851067 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.523+0000 7f62d9ce6700 1 --2- 172.21.15.189:0/3672851067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62d40b4d70 0x7f62d414fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.523+0000 7f62da4e7700 1 --2- 172.21.15.189:0/3672851067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62d40a84b0 0x7f62d40a8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.523+0000 7f62d94e5700 1 -- 172.21.15.189:0/3672851067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62d40b4d70 msgr2=0x7f62d414fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.523+0000 7f62d94e5700 1 --2- 172.21.15.189:0/3672851067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62d40b4d70 0x7f62d414fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.523+0000 7f62d94e5700 1 -- 172.21.15.189:0/3672851067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62d40a84b0 msgr2=0x7f62d40a8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.523+0000 7f62d94e5700 1 --2- 172.21.15.189:0/3672851067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62d40a84b0 0x7f62d40a8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.523+0000 7f62d94e5700 1 -- 172.21.15.189:0/3672851067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62cc00bc80 con 0x7f62d40cf900 2024-04-06T12:28:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.523+0000 7f62d9ce6700 1 --2- 172.21.15.189:0/3672851067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62d40b4d70 0x7f62d414fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.523+0000 7f62d94e5700 1 --2- 172.21.15.189:0/3672851067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62d40cf900 0x7f62d415d1a0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f62d000c520 tx=0x7f62d000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.524+0000 7f62ca7fc700 1 -- 172.21.15.189:0/3672851067 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d0005d70 con 0x7f62d40cf900 2024-04-06T12:28:41.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.524+0000 7f62ca7fc700 1 -- 172.21.15.189:0/3672851067 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d00070d0 con 0x7f62d40cf900 2024-04-06T12:28:41.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.524+0000 7f62dbf4a700 1 -- 172.21.15.189:0/3672851067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d406d8f0 con 0x7f62d40cf900 2024-04-06T12:28:41.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.524+0000 7f62ca7fc700 1 -- 172.21.15.189:0/3672851067 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d0007440 con 0x7f62d40cf900 2024-04-06T12:28:41.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.524+0000 7f62dbf4a700 1 -- 172.21.15.189:0/3672851067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d406deb0 con 0x7f62d40cf900 2024-04-06T12:28:41.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.524+0000 7f62dbf4a700 1 -- 172.21.15.189:0/3672851067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d41f2a60 con 0x7f62d40cf900 2024-04-06T12:28:41.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.525+0000 7f62ca7fc700 1 -- 172.21.15.189:0/3672851067 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f62d00075a0 con 0x7f62d40cf900 2024-04-06T12:28:41.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.525+0000 7f62ca7fc700 1 --2- 172.21.15.189:0/3672851067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f62b806adb0 0x7f62b806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:41.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.526+0000 7f62ca7fc700 1 -- 172.21.15.189:0/3672851067 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f62d0088980 con 0x7f62d40cf900 2024-04-06T12:28:41.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.526+0000 7f62d9ce6700 1 --2- 172.21.15.189:0/3672851067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f62b806adb0 0x7f62b806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:41.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.526+0000 7f62d9ce6700 1 --2- 172.21.15.189:0/3672851067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f62b806adb0 0x7f62b806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f62d40b3420 tx=0x7f62c4008040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:41.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.530+0000 7f62ca7fc700 1 -- 172.21.15.189:0/3672851067 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f62d41f2a60 con 0x7f62d40cf900 2024-04-06T12:28:41.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:41.774+0000 7f62dbf4a700 1 -- 172.21.15.189:0/3672851067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f62d41f2780 con 0x7f62d40cf900 2024-04-06T12:28:42.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.670+0000 7f62ca7fc700 1 -- 172.21.15.189:0/3672851067 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v45) v1 ==== 118+0+0 (secure 0 0 0) 0x7f62d41f2780 con 0x7f62d40cf900 2024-04-06T12:28:42.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.673+0000 7f62dbf4a700 1 -- 172.21.15.189:0/3672851067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f62b806adb0 msgr2=0x7f62b806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:42.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.673+0000 7f62dbf4a700 1 --2- 172.21.15.189:0/3672851067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f62b806adb0 0x7f62b806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f62d40b3420 tx=0x7f62c4008040 comp rx=0 tx=0).stop 2024-04-06T12:28:42.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.673+0000 7f62dbf4a700 1 -- 172.21.15.189:0/3672851067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62d40cf900 msgr2=0x7f62d415d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:42.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.673+0000 7f62dbf4a700 1 --2- 172.21.15.189:0/3672851067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62d40cf900 0x7f62d415d1a0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f62d000c520 tx=0x7f62d000c8e0 comp rx=0 tx=0).stop 2024-04-06T12:28:42.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.673+0000 7f62dbf4a700 1 -- 172.21.15.189:0/3672851067 shutdown_connections 2024-04-06T12:28:42.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.673+0000 7f62dbf4a700 1 --2- 172.21.15.189:0/3672851067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f62b806adb0 0x7f62b806d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:42.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.673+0000 7f62dbf4a700 1 --2- 172.21.15.189:0/3672851067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62d40a84b0 0x7f62d40a8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:42.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.673+0000 7f62dbf4a700 1 --2- 172.21.15.189:0/3672851067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62d40b4d70 0x7f62d414fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:42.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.673+0000 7f62dbf4a700 1 --2- 172.21.15.189:0/3672851067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62d40cf900 0x7f62d415d1a0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:42.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.673+0000 7f62dbf4a700 1 -- 172.21.15.189:0/3672851067 >> 172.21.15.189:0/3672851067 conn(0x7f62d40d00b0 msgr2=0x7f62d41536c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:42.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.673+0000 7f62dbf4a700 1 -- 172.21.15.189:0/3672851067 shutdown_connections 2024-04-06T12:28:42.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.673+0000 7f62dbf4a700 1 -- 172.21.15.189:0/3672851067 wait complete. 2024-04-06T12:28:42.685 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-06T12:28:42.685 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-06T12:28:42.722 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:28:42.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:28:42.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.822+0000 7f3ad95ff700 1 Processor -- start 2024-04-06T12:28:42.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.822+0000 7f3ad95ff700 1 -- start start 2024-04-06T12:28:42.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.822+0000 7f3ad95ff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ad415d6f0 0x7f3ad415dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:42.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.822+0000 7f3ad95ff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad415eac0 con 0x7f3ad40b8d70 2024-04-06T12:28:42.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad95ff700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad415e220 con 0x7f3ad40ce730 2024-04-06T12:28:42.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad95ff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad415e010 con 0x7f3ad415d6f0 2024-04-06T12:28:42.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad27fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3ad40b8d70 0x7f3ad40ce020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51456/0 (socket says 172.21.15.189:51456) 2024-04-06T12:28:42.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad27fc700 1 -- 172.21.15.189:0/873503463 learned_addr learned my addr 172.21.15.189:0/873503463 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:42.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad2ffd700 1 --2- 172.21.15.189:0/873503463 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ad415d6f0 0x7f3ad415dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:42.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad1ffb700 1 -- 172.21.15.189:0/873503463 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906860405 0 0) 0x7f3ad415eac0 con 0x7f3ad40b8d70 2024-04-06T12:28:42.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad1ffb700 1 -- 172.21.15.189:0/873503463 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3abc0035b0 con 0x7f3ad40b8d70 2024-04-06T12:28:42.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad1ffb700 1 -- 172.21.15.189:0/873503463 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 643014609 0 0) 0x7f3abc0035b0 con 0x7f3ad40b8d70 2024-04-06T12:28:42.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad1ffb700 1 -- 172.21.15.189:0/873503463 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ad415d6f0 msgr2=0x7f3ad415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:42.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad1ffb700 1 --2- 172.21.15.189:0/873503463 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ad415d6f0 0x7f3ad415dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:42.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad1ffb700 1 -- 172.21.15.189:0/873503463 >> v1:172.21.15.192:6789/0 conn(0x7f3ad40ce730 legacy=0x7f3ad40d1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:42.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad1ffb700 1 -- 172.21.15.189:0/873503463 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad415e500 con 0x7f3ad40b8d70 2024-04-06T12:28:42.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad1ffb700 1 -- 172.21.15.189:0/873503463 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f3ac8001670 con 0x7f3ad40b8d70 2024-04-06T12:28:42.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad1ffb700 1 -- 172.21.15.189:0/873503463 >> v1:172.21.15.189:6789/0 conn(0x7f3ad40b8d70 legacy=0x7f3ad40ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:42.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad1ffb700 1 --2- 172.21.15.189:0/873503463 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3abc003a30 0x7f3abc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:42.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad1ffb700 1 --2- 172.21.15.189:0/873503463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3abc004850 0x7f3abc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:42.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad1ffb700 1 --2- 172.21.15.189:0/873503463 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ad415d6f0 0x7f3abc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:42.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad1ffb700 1 -- 172.21.15.189:0/873503463 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad415e010 con 0x7f3abc004850 2024-04-06T12:28:42.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad1ffb700 1 -- 172.21.15.189:0/873503463 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc00bac0 con 0x7f3abc003a30 2024-04-06T12:28:42.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad1ffb700 1 -- 172.21.15.189:0/873503463 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc00bc80 con 0x7f3ad415d6f0 2024-04-06T12:28:42.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad27fc700 1 --2- 172.21.15.189:0/873503463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3abc004850 0x7f3abc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:42.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad2ffd700 1 --2- 172.21.15.189:0/873503463 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3abc003a30 0x7f3abc003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:42.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.823+0000 7f3ad37fe700 1 --2- 172.21.15.189:0/873503463 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ad415d6f0 0x7f3abc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:42.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.824+0000 7f3ad27fc700 1 -- 172.21.15.189:0/873503463 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ad415d6f0 msgr2=0x7f3abc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:42.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.824+0000 7f3ad27fc700 1 --2- 172.21.15.189:0/873503463 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ad415d6f0 0x7f3abc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:42.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.824+0000 7f3ad27fc700 1 -- 172.21.15.189:0/873503463 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3abc003a30 msgr2=0x7f3abc003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:42.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.824+0000 7f3ad27fc700 1 --2- 172.21.15.189:0/873503463 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3abc003a30 0x7f3abc003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:42.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.824+0000 7f3ad27fc700 1 -- 172.21.15.189:0/873503463 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3abc0035b0 con 0x7f3abc004850 2024-04-06T12:28:42.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.824+0000 7f3ad27fc700 1 --2- 172.21.15.189:0/873503463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3abc004850 0x7f3abc004cd0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800c210 tx=0x7f3abc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=145b4e899b9566c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:42.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.824+0000 7f3ad1ffb700 1 -- 172.21.15.189:0/873503463 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8007c10 con 0x7f3abc004850 2024-04-06T12:28:42.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.824+0000 7f3ad1ffb700 1 -- 172.21.15.189:0/873503463 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac800cdb0 con 0x7f3abc004850 2024-04-06T12:28:42.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.824+0000 7f3ad1ffb700 1 -- 172.21.15.189:0/873503463 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8005ba0 con 0x7f3abc004850 2024-04-06T12:28:42.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.824+0000 7f3ad95ff700 1 -- 172.21.15.189:0/873503463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3abc004850 msgr2=0x7f3abc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:42.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.824+0000 7f3ad95ff700 1 --2- 172.21.15.189:0/873503463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3abc004850 0x7f3abc004cd0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800c210 tx=0x7f3abc005880 comp rx=0 tx=0).stop 2024-04-06T12:28:42.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.824+0000 7f3ad37fe700 1 -- 172.21.15.189:0/873503463 reap_dead start 2024-04-06T12:28:42.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.824+0000 7f3ad95ff700 1 -- 172.21.15.189:0/873503463 shutdown_connections 2024-04-06T12:28:42.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.824+0000 7f3ad95ff700 1 -- 172.21.15.189:0/873503463 >> 172.21.15.189:0/873503463 conn(0x7f3ad4072490 msgr2=0x7f3ad406b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:42.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.825+0000 7f3ad95ff700 1 -- 172.21.15.189:0/873503463 shutdown_connections 2024-04-06T12:28:42.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.825+0000 7f3ad95ff700 1 -- 172.21.15.189:0/873503463 wait complete. 2024-04-06T12:28:42.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.826+0000 7f3ad95ff700 1 Processor -- start 2024-04-06T12:28:42.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.826+0000 7f3ad95ff700 1 -- start start 2024-04-06T12:28:42.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.827+0000 7f3ad95ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3abc004850 0x7f3ad415ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:42.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.827+0000 7f3ad95ff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ad4160340 0x7f3ad40ca6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:42.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.827+0000 7f3ad95ff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ad4159850 0x7f3ad40cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:42.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.827+0000 7f3ad95ff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad415ff20 con 0x7f3ad4160340 2024-04-06T12:28:42.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.827+0000 7f3ad95ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad40cabf0 con 0x7f3abc004850 2024-04-06T12:28:42.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.827+0000 7f3ad95ff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad415f2a0 con 0x7f3ad4159850 2024-04-06T12:28:42.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.827+0000 7f3ad27fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ad4160340 0x7f3ad40ca6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:42.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.827+0000 7f3ad27fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ad4160340 0x7f3ad40ca6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53384/0 (socket says 172.21.15.189:53384) 2024-04-06T12:28:42.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.827+0000 7f3ad27fc700 1 -- 172.21.15.189:0/712484233 learned_addr learned my addr 172.21.15.189:0/712484233 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:42.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.827+0000 7f3ad37fe700 1 --2- 172.21.15.189:0/712484233 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ad4159850 0x7f3ad40cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:42.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.827+0000 7f3ad2ffd700 1 --2- 172.21.15.189:0/712484233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3abc004850 0x7f3ad415ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:42.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.828+0000 7f3ad27fc700 1 -- 172.21.15.189:0/712484233 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ad4159850 msgr2=0x7f3ad40cae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:42.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.828+0000 7f3ad27fc700 1 --2- 172.21.15.189:0/712484233 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ad4159850 0x7f3ad40cae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:42.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.828+0000 7f3ad27fc700 1 -- 172.21.15.189:0/712484233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3abc004850 msgr2=0x7f3ad415ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:42.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.828+0000 7f3ad27fc700 1 --2- 172.21.15.189:0/712484233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3abc004850 0x7f3ad415ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:42.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.828+0000 7f3ad27fc700 1 -- 172.21.15.189:0/712484233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad415f580 con 0x7f3ad4160340 2024-04-06T12:28:42.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.828+0000 7f3ad27fc700 1 --2- 172.21.15.189:0/712484233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ad4160340 0x7f3ad40ca6b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f3abc0047b0 tx=0x7f3ac8001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:42.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.828+0000 7f3abb7fe700 1 -- 172.21.15.189:0/712484233 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8001640 con 0x7f3ad4160340 2024-04-06T12:28:42.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.828+0000 7f3abb7fe700 1 -- 172.21.15.189:0/712484233 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac80070d0 con 0x7f3ad4160340 2024-04-06T12:28:42.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.828+0000 7f3abb7fe700 1 -- 172.21.15.189:0/712484233 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8007440 con 0x7f3ad4160340 2024-04-06T12:28:42.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.828+0000 7f3ad95ff700 1 -- 172.21.15.189:0/712484233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad415f860 con 0x7f3ad4160340 2024-04-06T12:28:42.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.828+0000 7f3ad95ff700 1 -- 172.21.15.189:0/712484233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad4159ce0 con 0x7f3ad4160340 2024-04-06T12:28:42.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.829+0000 7f3ad95ff700 1 -- 172.21.15.189:0/712484233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad40cdd80 con 0x7f3ad4160340 2024-04-06T12:28:42.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.829+0000 7f3abb7fe700 1 -- 172.21.15.189:0/712484233 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f3ac8019030 con 0x7f3ad4160340 2024-04-06T12:28:42.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.830+0000 7f3abb7fe700 1 --2- 172.21.15.189:0/712484233 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3ac006adb0 0x7f3ac006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:42.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.830+0000 7f3abb7fe700 1 -- 172.21.15.189:0/712484233 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3ac808a4c0 con 0x7f3ad4160340 2024-04-06T12:28:42.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.830+0000 7f3ad2ffd700 1 --2- 172.21.15.189:0/712484233 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3ac006adb0 0x7f3ac006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:42.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.830+0000 7f3ad2ffd700 1 --2- 172.21.15.189:0/712484233 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3ac006adb0 0x7f3ac006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009ba0 tx=0x7f3ac4008040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:42.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:42.834+0000 7f3abb7fe700 1 -- 172.21.15.189:0/712484233 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3ac80554e0 con 0x7f3ad4160340 2024-04-06T12:28:43.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.075+0000 7f3ad95ff700 1 -- 172.21.15.189:0/712484233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f3ad4074990 con 0x7f3ad4160340 2024-04-06T12:28:43.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.674+0000 7f3abb7fe700 1 -- 172.21.15.189:0/712484233 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3ac8058b00 con 0x7f3ad4160340 2024-04-06T12:28:43.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.677+0000 7f3ad95ff700 1 -- 172.21.15.189:0/712484233 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3ac006adb0 msgr2=0x7f3ac006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:43.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.677+0000 7f3ad95ff700 1 --2- 172.21.15.189:0/712484233 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3ac006adb0 0x7f3ac006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009ba0 tx=0x7f3ac4008040 comp rx=0 tx=0).stop 2024-04-06T12:28:43.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.677+0000 7f3ad95ff700 1 -- 172.21.15.189:0/712484233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ad4160340 msgr2=0x7f3ad40ca6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:43.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.677+0000 7f3ad95ff700 1 --2- 172.21.15.189:0/712484233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ad4160340 0x7f3ad40ca6b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f3abc0047b0 tx=0x7f3ac8001a00 comp rx=0 tx=0).stop 2024-04-06T12:28:43.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.677+0000 7f3ad95ff700 1 -- 172.21.15.189:0/712484233 shutdown_connections 2024-04-06T12:28:43.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.677+0000 7f3ad95ff700 1 --2- 172.21.15.189:0/712484233 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ad4159850 0x7f3ad40cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:43.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.677+0000 7f3ad95ff700 1 --2- 172.21.15.189:0/712484233 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3ac006adb0 0x7f3ac006d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:43.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.677+0000 7f3ad95ff700 1 --2- 172.21.15.189:0/712484233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3abc004850 0x7f3ad415ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:43.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.677+0000 7f3ad95ff700 1 --2- 172.21.15.189:0/712484233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ad4160340 0x7f3ad40ca6b0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:43.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.677+0000 7f3ad95ff700 1 -- 172.21.15.189:0/712484233 >> 172.21.15.189:0/712484233 conn(0x7f3ad4072490 msgr2=0x7f3ad40b8570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:43.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.677+0000 7f3ad95ff700 1 -- 172.21.15.189:0/712484233 shutdown_connections 2024-04-06T12:28:43.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.677+0000 7f3ad95ff700 1 -- 172.21.15.189:0/712484233 wait complete. 2024-04-06T12:28:43.688 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-06T12:28:43.689 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:28:43.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.856+0000 7fa2f0b20700 1 Processor -- start 2024-04-06T12:28:43.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.856+0000 7fa2f0b20700 1 -- start start 2024-04-06T12:28:43.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2f0b20700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa2ec0cb870 0x7fa2ec15c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:43.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2f0b20700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ec0cbd80 con 0x7fa2ec0cfa00 2024-04-06T12:28:43.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2f0b20700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ec0cef20 con 0x7fa2ec0d11b0 2024-04-06T12:28:43.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2f0b20700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2ec0ccdc0 con 0x7fa2ec0cb870 2024-04-06T12:28:43.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2ead9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa2ec0cfa00 0x7fa2ec0cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51472/0 (socket says 172.21.15.189:51472) 2024-04-06T12:28:43.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2ead9d700 1 -- 172.21.15.189:0/2174666111 learned_addr learned my addr 172.21.15.189:0/2174666111 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:43.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2e9d9b700 1 --2- 172.21.15.189:0/2174666111 >> v2:172.21.15.192:3300/0 conn(0x7fa2ec0cb870 0x7fa2ec15c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:43.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2e959a700 1 -- 172.21.15.189:0/2174666111 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1289480362 0 0) 0x7fa2ec0cbd80 con 0x7fa2ec0cfa00 2024-04-06T12:28:43.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2e959a700 1 -- 172.21.15.189:0/2174666111 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2d40035b0 con 0x7fa2ec0cfa00 2024-04-06T12:28:43.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2e959a700 1 -- 172.21.15.189:0/2174666111 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538515400 0 0) 0x7fa2ec0cef20 con 0x7fa2ec0d11b0 2024-04-06T12:28:43.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2e959a700 1 -- 172.21.15.189:0/2174666111 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2ec0cbd80 con 0x7fa2ec0d11b0 2024-04-06T12:28:43.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2e959a700 1 -- 172.21.15.189:0/2174666111 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 59143313 0 0) 0x7fa2d40035b0 con 0x7fa2ec0cfa00 2024-04-06T12:28:43.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2e959a700 1 -- 172.21.15.189:0/2174666111 >> v2:172.21.15.192:3300/0 conn(0x7fa2ec0cb870 msgr2=0x7fa2ec15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:43.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2e959a700 1 --2- 172.21.15.189:0/2174666111 >> v2:172.21.15.192:3300/0 conn(0x7fa2ec0cb870 0x7fa2ec15c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:43.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2e959a700 1 -- 172.21.15.189:0/2174666111 >> v1:172.21.15.192:6789/0 conn(0x7fa2ec0d11b0 legacy=0x7fa2ec0d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:43.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2e959a700 1 -- 172.21.15.189:0/2174666111 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ec0cf200 con 0x7fa2ec0cfa00 2024-04-06T12:28:43.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2e959a700 1 -- 172.21.15.189:0/2174666111 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fa2e40019f0 con 0x7fa2ec0cfa00 2024-04-06T12:28:43.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2e959a700 1 -- 172.21.15.189:0/2174666111 >> v1:172.21.15.189:6789/0 conn(0x7fa2ec0cfa00 legacy=0x7fa2ec0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:43.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2e959a700 1 --2- 172.21.15.189:0/2174666111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d4003be0 0x7fa2d4003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:43.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2e959a700 1 --2- 172.21.15.189:0/2174666111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2d4004850 0x7fa2d4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:43.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2e959a700 1 --2- 172.21.15.189:0/2174666111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d4005260 0x7fa2d400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:43.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2e959a700 1 -- 172.21.15.189:0/2174666111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d400c160 con 0x7fa2d4003be0 2024-04-06T12:28:43.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2e959a700 1 -- 172.21.15.189:0/2174666111 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d400c2d0 con 0x7fa2d4004850 2024-04-06T12:28:43.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.857+0000 7fa2e959a700 1 -- 172.21.15.189:0/2174666111 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d400c490 con 0x7fa2d4005260 2024-04-06T12:28:43.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.858+0000 7fa2ea59c700 1 --2- 172.21.15.189:0/2174666111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d4003be0 0x7fa2d4003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:43.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.858+0000 7fa2e9d9b700 1 --2- 172.21.15.189:0/2174666111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2d4004850 0x7fa2d4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:43.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.858+0000 7fa2ead9d700 1 --2- 172.21.15.189:0/2174666111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d4005260 0x7fa2d400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:43.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.858+0000 7fa2ea59c700 1 -- 172.21.15.189:0/2174666111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d4005260 msgr2=0x7fa2d400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:43.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.858+0000 7fa2ea59c700 1 --2- 172.21.15.189:0/2174666111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d4005260 0x7fa2d400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:43.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.858+0000 7fa2ea59c700 1 -- 172.21.15.189:0/2174666111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2d4004850 msgr2=0x7fa2d4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:43.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.858+0000 7fa2ea59c700 1 --2- 172.21.15.189:0/2174666111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2d4004850 0x7fa2d4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:43.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.858+0000 7fa2ea59c700 1 -- 172.21.15.189:0/2174666111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d40035b0 con 0x7fa2d4003be0 2024-04-06T12:28:43.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.858+0000 7fa2ead9d700 1 -- 172.21.15.189:0/2174666111 reap_dead start 2024-04-06T12:28:43.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.858+0000 7fa2ea59c700 1 --2- 172.21.15.189:0/2174666111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d4003be0 0x7fa2d4003fe0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc001940 tx=0x7fa2d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd99a4b90dc91c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:43.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.858+0000 7fa2e959a700 1 -- 172.21.15.189:0/2174666111 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc001430 con 0x7fa2d4003be0 2024-04-06T12:28:43.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.858+0000 7fa2e959a700 1 -- 172.21.15.189:0/2174666111 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2dc001d70 con 0x7fa2d4003be0 2024-04-06T12:28:43.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.858+0000 7fa2e959a700 1 -- 172.21.15.189:0/2174666111 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc005510 con 0x7fa2d4003be0 2024-04-06T12:28:43.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.859+0000 7fa2f0b20700 1 -- 172.21.15.189:0/2174666111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d4003be0 msgr2=0x7fa2d4003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:43.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.859+0000 7fa2f0b20700 1 --2- 172.21.15.189:0/2174666111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d4003be0 0x7fa2d4003fe0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc001940 tx=0x7fa2d4005880 comp rx=0 tx=0).stop 2024-04-06T12:28:43.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.859+0000 7fa2f0b20700 1 -- 172.21.15.189:0/2174666111 shutdown_connections 2024-04-06T12:28:43.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.859+0000 7fa2f0b20700 1 --2- 172.21.15.189:0/2174666111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d4003be0 0x7fa2d4003fe0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:43.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.859+0000 7fa2f0b20700 1 -- 172.21.15.189:0/2174666111 >> 172.21.15.189:0/2174666111 conn(0x7fa2ec0b2db0 msgr2=0x7fa2ec0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:43.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.859+0000 7fa2f0b20700 1 -- 172.21.15.189:0/2174666111 shutdown_connections 2024-04-06T12:28:43.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.859+0000 7fa2f0b20700 1 -- 172.21.15.189:0/2174666111 wait complete. 2024-04-06T12:28:43.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.860+0000 7fa2f0b20700 1 Processor -- start 2024-04-06T12:28:43.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.860+0000 7fa2f0b20700 1 -- start start 2024-04-06T12:28:43.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.861+0000 7fa2f0b20700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d4003be0 0x7fa2ec141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:43.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.861+0000 7fa2f0b20700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2ec1f2030 0x7fa2ec154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:43.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.861+0000 7fa2f0b20700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2ec141860 0x7fa2ec15e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:43.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.861+0000 7fa2f0b20700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec151840 con 0x7fa2ec141860 2024-04-06T12:28:43.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.861+0000 7fa2f0b20700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec1f19b0 con 0x7fa2ec1f2030 2024-04-06T12:28:43.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.861+0000 7fa2f0b20700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec1511f0 con 0x7fa2d4003be0 2024-04-06T12:28:43.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.861+0000 7fa2ead9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2ec141860 0x7fa2ec15e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:43.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.861+0000 7fa2ead9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2ec141860 0x7fa2ec15e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53406/0 (socket says 172.21.15.189:53406) 2024-04-06T12:28:43.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.861+0000 7fa2ead9d700 1 -- 172.21.15.189:0/2986926703 learned_addr learned my addr 172.21.15.189:0/2986926703 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:43.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.862+0000 7fa2ea59c700 1 --2- 172.21.15.189:0/2986926703 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d4003be0 0x7fa2ec141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:43.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.862+0000 7fa2e9d9b700 1 --2- 172.21.15.189:0/2986926703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2ec1f2030 0x7fa2ec154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:43.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.862+0000 7fa2ead9d700 1 -- 172.21.15.189:0/2986926703 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d4003be0 msgr2=0x7fa2ec141290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:43.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.862+0000 7fa2ead9d700 1 --2- 172.21.15.189:0/2986926703 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d4003be0 0x7fa2ec141290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:43.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.862+0000 7fa2ead9d700 1 -- 172.21.15.189:0/2986926703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2ec1f2030 msgr2=0x7fa2ec154bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:43.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.862+0000 7fa2ead9d700 1 --2- 172.21.15.189:0/2986926703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2ec1f2030 0x7fa2ec154bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:43.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.862+0000 7fa2ead9d700 1 -- 172.21.15.189:0/2986926703 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ec1514d0 con 0x7fa2ec141860 2024-04-06T12:28:43.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.862+0000 7fa2e9d9b700 1 --2- 172.21.15.189:0/2986926703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2ec1f2030 0x7fa2ec154bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:28:43.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.862+0000 7fa2ead9d700 1 --2- 172.21.15.189:0/2986926703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2ec141860 0x7fa2ec15e980 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec0b3df0 tx=0x7fa2e400e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:43.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.862+0000 7fa2daffd700 1 -- 172.21.15.189:0/2986926703 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e400ebf0 con 0x7fa2ec141860 2024-04-06T12:28:43.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.862+0000 7fa2daffd700 1 -- 172.21.15.189:0/2986926703 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e400ed50 con 0x7fa2ec141860 2024-04-06T12:28:43.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.862+0000 7fa2f0b20700 1 -- 172.21.15.189:0/2986926703 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2ec1f1c90 con 0x7fa2ec141860 2024-04-06T12:28:43.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.862+0000 7fa2daffd700 1 -- 172.21.15.189:0/2986926703 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4013620 con 0x7fa2ec141860 2024-04-06T12:28:43.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.862+0000 7fa2f0b20700 1 -- 172.21.15.189:0/2986926703 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2ec1550f0 con 0x7fa2ec141860 2024-04-06T12:28:43.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.863+0000 7fa2f0b20700 1 -- 172.21.15.189:0/2986926703 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2ec1552e0 con 0x7fa2ec141860 2024-04-06T12:28:43.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.863+0000 7fa2daffd700 1 -- 172.21.15.189:0/2986926703 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fa2e4013780 con 0x7fa2ec141860 2024-04-06T12:28:43.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.864+0000 7fa2daffd700 1 --2- 172.21.15.189:0/2986926703 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2cc06adb0 0x7fa2cc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:43.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.864+0000 7fa2daffd700 1 -- 172.21.15.189:0/2986926703 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa2e4089b80 con 0x7fa2ec141860 2024-04-06T12:28:43.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.864+0000 7fa2ea59c700 1 --2- 172.21.15.189:0/2986926703 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2cc06adb0 0x7fa2cc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:43.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.864+0000 7fa2ea59c700 1 --2- 172.21.15.189:0/2986926703 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2cc06adb0 0x7fa2cc06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc005040 tx=0x7fa2dc00a000 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:43.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:43.868+0000 7fa2daffd700 1 -- 172.21.15.189:0/2986926703 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa2e4054a70 con 0x7fa2ec141860 2024-04-06T12:28:44.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.106+0000 7fa2f0b20700 1 -- 172.21.15.189:0/2986926703 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2ec151050 con 0x7fa2ec141860 2024-04-06T12:28:44.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.107+0000 7fa2daffd700 1 -- 172.21.15.189:0/2986926703 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4739 (secure 0 0 0) 0x7fa2ec151050 con 0x7fa2ec141860 2024-04-06T12:28:44.104 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:44.104 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":46,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6389,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/17345288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":17345288},{"type":"v1","addr":"172.21.15.192:6835","nonce":17345288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":7078,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/440618958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":440618958},{"type":"v1","addr":"172.21.15.192:6834","nonce":440618958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":7569,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3391245238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3391245238},{"type":"v1","addr":"172.21.15.189:6836","nonce":3391245238}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:28:40.659968+0000","modified":"2024-04-06T12:28:43.668129+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7572":{"gid":7572,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/2225868923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2225868923},{"type":"v1","addr":"172.21.15.189:6837","nonce":2225868923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-04-06T12:28:44.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.109+0000 7fa2f0b20700 1 -- 172.21.15.189:0/2986926703 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2cc06adb0 msgr2=0x7fa2cc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:44.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.109+0000 7fa2f0b20700 1 --2- 172.21.15.189:0/2986926703 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2cc06adb0 0x7fa2cc06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc005040 tx=0x7fa2dc00a000 comp rx=0 tx=0).stop 2024-04-06T12:28:44.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.109+0000 7fa2f0b20700 1 -- 172.21.15.189:0/2986926703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2ec141860 msgr2=0x7fa2ec15e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:44.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.109+0000 7fa2f0b20700 1 --2- 172.21.15.189:0/2986926703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2ec141860 0x7fa2ec15e980 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec0b3df0 tx=0x7fa2e400e440 comp rx=0 tx=0).stop 2024-04-06T12:28:44.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.109+0000 7fa2f0b20700 1 -- 172.21.15.189:0/2986926703 shutdown_connections 2024-04-06T12:28:44.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.109+0000 7fa2f0b20700 1 --2- 172.21.15.189:0/2986926703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2ec141860 0x7fa2ec15e980 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.109+0000 7fa2f0b20700 1 --2- 172.21.15.189:0/2986926703 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d4003be0 0x7fa2ec141290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.109+0000 7fa2f0b20700 1 --2- 172.21.15.189:0/2986926703 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2cc06adb0 0x7fa2cc06d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.109+0000 7fa2f0b20700 1 --2- 172.21.15.189:0/2986926703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2ec1f2030 0x7fa2ec154bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.109+0000 7fa2f0b20700 1 -- 172.21.15.189:0/2986926703 >> 172.21.15.189:0/2986926703 conn(0x7fa2ec0b2db0 msgr2=0x7fa2ec0b6090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:44.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.109+0000 7fa2f0b20700 1 -- 172.21.15.189:0/2986926703 shutdown_connections 2024-04-06T12:28:44.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.109+0000 7fa2f0b20700 1 -- 172.21.15.189:0/2986926703 wait complete. 2024-04-06T12:28:44.108 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 46 2024-04-06T12:28:44.119 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:28:44.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.243+0000 7f74047da700 1 Processor -- start 2024-04-06T12:28:44.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.243+0000 7f74047da700 1 -- start start 2024-04-06T12:28:44.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.244+0000 7f74047da700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73fc0b4660 0x7f73fc0b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:44.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.244+0000 7f74047da700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73fc0cdbe0 con 0x7f73fc0b2fc0 2024-04-06T12:28:44.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.244+0000 7f74047da700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73fc0b5000 con 0x7f73fc15a550 2024-04-06T12:28:44.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.244+0000 7f74047da700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc15a940 con 0x7f73fc0b4660 2024-04-06T12:28:44.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.245+0000 7f7402d77700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f73fc0b2fc0 0x7f73fc0b3430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51484/0 (socket says 172.21.15.189:51484) 2024-04-06T12:28:44.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.245+0000 7f7402d77700 1 -- 172.21.15.189:0/4294307424 learned_addr learned my addr 172.21.15.189:0/4294307424 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:44.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.245+0000 7f7401d75700 1 --2- 172.21.15.189:0/4294307424 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73fc0b4660 0x7f73fc0b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:44.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.245+0000 7f7401574700 1 -- 172.21.15.189:0/4294307424 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341900411 0 0) 0x7f73fc0cdbe0 con 0x7f73fc0b2fc0 2024-04-06T12:28:44.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.245+0000 7f7401574700 1 -- 172.21.15.189:0/4294307424 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e80035b0 con 0x7f73fc0b2fc0 2024-04-06T12:28:44.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.245+0000 7f7401d75700 1 -- 172.21.15.189:0/4294307424 >> v1:172.21.15.192:6789/0 conn(0x7f73fc15a550 legacy=0x7f73fc0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:44.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.245+0000 7f7401d75700 1 -- 172.21.15.189:0/4294307424 >> v1:172.21.15.189:6789/0 conn(0x7f73fc0b2fc0 legacy=0x7f73fc0b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:44.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.245+0000 7f7401574700 1 -- 172.21.15.189:0/4294307424 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978804653 0 0) 0x7f73fc0b5000 con 0x7f73fc15a550 2024-04-06T12:28:44.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.245+0000 7f7401d75700 1 -- 172.21.15.189:0/4294307424 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73fc15ac00 con 0x7f73fc0b4660 2024-04-06T12:28:44.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.245+0000 7f7401d75700 1 --2- 172.21.15.189:0/4294307424 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73fc0b4660 0x7f73fc0b4ac0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f73f8009870 tx=0x7f73f8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f32496b881b7c65f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:44.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.245+0000 7f7401574700 1 -- 172.21.15.189:0/4294307424 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f8011830 con 0x7f73fc0b4660 2024-04-06T12:28:44.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.245+0000 7f7401574700 1 -- 172.21.15.189:0/4294307424 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f8011990 con 0x7f73fc0b4660 2024-04-06T12:28:44.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.246+0000 7f7401574700 1 -- 172.21.15.189:0/4294307424 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f8011d00 con 0x7f73fc0b4660 2024-04-06T12:28:44.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.246+0000 7f74047da700 1 -- 172.21.15.189:0/4294307424 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73fc0b4660 msgr2=0x7f73fc0b4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:44.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.246+0000 7f74047da700 1 --2- 172.21.15.189:0/4294307424 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73fc0b4660 0x7f73fc0b4ac0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f73f8009870 tx=0x7f73f8009b80 comp rx=0 tx=0).stop 2024-04-06T12:28:44.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.246+0000 7f74047da700 1 -- 172.21.15.189:0/4294307424 shutdown_connections 2024-04-06T12:28:44.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.246+0000 7f74047da700 1 --2- 172.21.15.189:0/4294307424 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73fc0b4660 0x7f73fc0b4ac0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.246+0000 7f74047da700 1 -- 172.21.15.189:0/4294307424 >> 172.21.15.189:0/4294307424 conn(0x7f73fc0d00b0 msgr2=0x7f73fc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:44.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.246+0000 7f74047da700 1 -- 172.21.15.189:0/4294307424 shutdown_connections 2024-04-06T12:28:44.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.246+0000 7f74047da700 1 -- 172.21.15.189:0/4294307424 wait complete. 2024-04-06T12:28:44.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.247+0000 7f74047da700 1 Processor -- start 2024-04-06T12:28:44.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.247+0000 7f74047da700 1 -- start start 2024-04-06T12:28:44.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.248+0000 7f74047da700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73fc0b2fc0 0x7f73fc0c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:44.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.248+0000 7f74047da700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73fc0b4660 0x7f73fc0b1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:44.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.248+0000 7f74047da700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73fc0cadf0 0x7f73fc157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:44.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.248+0000 7f74047da700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0ca6a0 con 0x7f73fc0cadf0 2024-04-06T12:28:44.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.248+0000 7f74047da700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc1f2a60 con 0x7f73fc0b2fc0 2024-04-06T12:28:44.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.248+0000 7f74047da700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0c9880 con 0x7f73fc0b4660 2024-04-06T12:28:44.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.248+0000 7f7402d77700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73fc0cadf0 0x7f73fc157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:44.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.248+0000 7f7402d77700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73fc0cadf0 0x7f73fc157c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53412/0 (socket says 172.21.15.189:53412) 2024-04-06T12:28:44.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.248+0000 7f7402d77700 1 -- 172.21.15.189:0/611489790 learned_addr learned my addr 172.21.15.189:0/611489790 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:44.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.248+0000 7f7402576700 1 --2- 172.21.15.189:0/611489790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73fc0b2fc0 0x7f73fc0c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:44.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.248+0000 7f7401d75700 1 --2- 172.21.15.189:0/611489790 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73fc0b4660 0x7f73fc0b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:44.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.248+0000 7f7402d77700 1 -- 172.21.15.189:0/611489790 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73fc0b4660 msgr2=0x7f73fc0b1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:44.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.248+0000 7f7402d77700 1 --2- 172.21.15.189:0/611489790 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73fc0b4660 0x7f73fc0b1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.248+0000 7f7402d77700 1 -- 172.21.15.189:0/611489790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73fc0b2fc0 msgr2=0x7f73fc0c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:44.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.248+0000 7f7402d77700 1 --2- 172.21.15.189:0/611489790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73fc0b2fc0 0x7f73fc0c9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.248+0000 7f7402d77700 1 -- 172.21.15.189:0/611489790 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f80094a0 con 0x7f73fc0cadf0 2024-04-06T12:28:44.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.249+0000 7f7402d77700 1 --2- 172.21.15.189:0/611489790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73fc0cadf0 0x7f73fc157c90 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f73f400cb40 tx=0x7f73f400cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:44.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.249+0000 7f73eeffd700 1 -- 172.21.15.189:0/611489790 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f400ab50 con 0x7f73fc0cadf0 2024-04-06T12:28:44.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.249+0000 7f73eeffd700 1 -- 172.21.15.189:0/611489790 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f400acb0 con 0x7f73fc0cadf0 2024-04-06T12:28:44.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.249+0000 7f74047da700 1 -- 172.21.15.189:0/611489790 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73fc0b1aa0 con 0x7f73fc0cadf0 2024-04-06T12:28:44.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.249+0000 7f73eeffd700 1 -- 172.21.15.189:0/611489790 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f4012620 con 0x7f73fc0cadf0 2024-04-06T12:28:44.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.249+0000 7f74047da700 1 -- 172.21.15.189:0/611489790 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73fc0c9e00 con 0x7f73fc0cadf0 2024-04-06T12:28:44.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.250+0000 7f74047da700 1 -- 172.21.15.189:0/611489790 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73fc067bb0 con 0x7f73fc0cadf0 2024-04-06T12:28:44.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.250+0000 7f73eeffd700 1 -- 172.21.15.189:0/611489790 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f73f401b3b0 con 0x7f73fc0cadf0 2024-04-06T12:28:44.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.250+0000 7f73eeffd700 1 --2- 172.21.15.189:0/611489790 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f73e006adb0 0x7f73e006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:44.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.251+0000 7f73eeffd700 1 -- 172.21.15.189:0/611489790 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f73f4089450 con 0x7f73fc0cadf0 2024-04-06T12:28:44.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.251+0000 7f7402576700 1 --2- 172.21.15.189:0/611489790 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f73e006adb0 0x7f73e006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:44.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.251+0000 7f7402576700 1 --2- 172.21.15.189:0/611489790 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f73e006adb0 0x7f73e006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f73f0000f60 tx=0x7f73f0009040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:44.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.255+0000 7f73eeffd700 1 -- 172.21.15.189:0/611489790 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f73f4054360 con 0x7f73fc0cadf0 2024-04-06T12:28:44.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.451+0000 7f74047da700 1 -- 172.21.15.189:0/611489790 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73fc1f2bd0 con 0x7f73fc0cadf0 2024-04-06T12:28:44.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.453+0000 7f73eeffd700 1 -- 172.21.15.189:0/611489790 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+16308 (secure 0 0 0) 0x7f73f4057980 con 0x7f73fc0cadf0 2024-04-06T12:28:44.450 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:44.450 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":112,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:28:41.661702+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:28:38.415021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-06T12:28:39.629378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:28:44.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.455+0000 7f74047da700 1 -- 172.21.15.189:0/611489790 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f73e006adb0 msgr2=0x7f73e006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:44.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.455+0000 7f74047da700 1 --2- 172.21.15.189:0/611489790 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f73e006adb0 0x7f73e006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f73f0000f60 tx=0x7f73f0009040 comp rx=0 tx=0).stop 2024-04-06T12:28:44.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.455+0000 7f74047da700 1 -- 172.21.15.189:0/611489790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73fc0cadf0 msgr2=0x7f73fc157c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:44.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.455+0000 7f74047da700 1 --2- 172.21.15.189:0/611489790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73fc0cadf0 0x7f73fc157c90 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f73f400cb40 tx=0x7f73f400cf00 comp rx=0 tx=0).stop 2024-04-06T12:28:44.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.455+0000 7f74047da700 1 -- 172.21.15.189:0/611489790 shutdown_connections 2024-04-06T12:28:44.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.455+0000 7f74047da700 1 --2- 172.21.15.189:0/611489790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73fc0cadf0 0x7f73fc157c90 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.455+0000 7f74047da700 1 --2- 172.21.15.189:0/611489790 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f73e006adb0 0x7f73e006d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.455+0000 7f74047da700 1 --2- 172.21.15.189:0/611489790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73fc0b2fc0 0x7f73fc0c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.455+0000 7f74047da700 1 --2- 172.21.15.189:0/611489790 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73fc0b4660 0x7f73fc0b1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.455+0000 7f74047da700 1 -- 172.21.15.189:0/611489790 >> 172.21.15.189:0/611489790 conn(0x7f73fc0d00b0 msgr2=0x7f73fc06f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:44.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.455+0000 7f74047da700 1 -- 172.21.15.189:0/611489790 shutdown_connections 2024-04-06T12:28:44.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.455+0000 7f74047da700 1 -- 172.21.15.189:0/611489790 wait complete. 2024-04-06T12:28:44.465 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-06T12:28:44.465 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:28:44.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.590+0000 7f54a4b4b700 1 Processor -- start 2024-04-06T12:28:44.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.590+0000 7f54a4b4b700 1 -- start start 2024-04-06T12:28:44.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.591+0000 7f54a4b4b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a00cd3e0 0x7f54a00cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:44.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.591+0000 7f54a4b4b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a006ceb0 con 0x7f54a015d7c0 2024-04-06T12:28:44.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.591+0000 7f54a4b4b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a006d750 con 0x7f54a006b8c0 2024-04-06T12:28:44.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.591+0000 7f54a4b4b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a00cdfa0 con 0x7f54a00cd3e0 2024-04-06T12:28:44.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.591+0000 7f549dd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f54a015d7c0 0x7f54a006b1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51498/0 (socket says 172.21.15.189:51498) 2024-04-06T12:28:44.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.591+0000 7f549dd9b700 1 -- 172.21.15.189:0/3059855237 learned_addr learned my addr 172.21.15.189:0/3059855237 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:44.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.591+0000 7f549e59c700 1 --2- 172.21.15.189:0/3059855237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a00cd3e0 0x7f54a00cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:44.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.591+0000 7f549d59a700 1 -- 172.21.15.189:0/3059855237 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033291077 0 0) 0x7f54a006ceb0 con 0x7f54a015d7c0 2024-04-06T12:28:44.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.591+0000 7f549d59a700 1 -- 172.21.15.189:0/3059855237 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54880035b0 con 0x7f54a015d7c0 2024-04-06T12:28:44.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549d59a700 1 -- 172.21.15.189:0/3059855237 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2851059646 0 0) 0x7f54880035b0 con 0x7f54a015d7c0 2024-04-06T12:28:44.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549d59a700 1 -- 172.21.15.189:0/3059855237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a00cd3e0 msgr2=0x7f54a00cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:44.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549d59a700 1 --2- 172.21.15.189:0/3059855237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a00cd3e0 0x7f54a00cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549d59a700 1 -- 172.21.15.189:0/3059855237 >> v1:172.21.15.192:6789/0 conn(0x7f54a006b8c0 legacy=0x7f54a006dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:44.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549d59a700 1 -- 172.21.15.189:0/3059855237 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a00cdd00 con 0x7f54a015d7c0 2024-04-06T12:28:44.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549d59a700 1 -- 172.21.15.189:0/3059855237 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f5490001670 con 0x7f54a015d7c0 2024-04-06T12:28:44.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549d59a700 1 -- 172.21.15.189:0/3059855237 >> v1:172.21.15.189:6789/0 conn(0x7f54a015d7c0 legacy=0x7f54a006b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:44.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549d59a700 1 --2- 172.21.15.189:0/3059855237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5488003a30 0x7f5488003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:44.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549d59a700 1 --2- 172.21.15.189:0/3059855237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54880043e0 0x7f54880048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:44.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549d59a700 1 --2- 172.21.15.189:0/3059855237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5488005100 0x7f548800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:44.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549d59a700 1 -- 172.21.15.189:0/3059855237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a00cdfa0 con 0x7f5488005100 2024-04-06T12:28:44.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549d59a700 1 -- 172.21.15.189:0/3059855237 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548800c030 con 0x7f5488003a30 2024-04-06T12:28:44.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549d59a700 1 -- 172.21.15.189:0/3059855237 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548800c1f0 con 0x7f54880043e0 2024-04-06T12:28:44.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549e59c700 1 --2- 172.21.15.189:0/3059855237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a00cd3e0 0x7f54a00cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:28:44.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549ed9d700 1 --2- 172.21.15.189:0/3059855237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5488005100 0x7f548800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:44.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549dd9b700 1 --2- 172.21.15.189:0/3059855237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54880043e0 0x7f54880048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:44.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549e59c700 1 --2- 172.21.15.189:0/3059855237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5488003a30 0x7f5488003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:44.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549ed9d700 1 -- 172.21.15.189:0/3059855237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54880043e0 msgr2=0x7f54880048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:44.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549ed9d700 1 --2- 172.21.15.189:0/3059855237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54880043e0 0x7f54880048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549ed9d700 1 -- 172.21.15.189:0/3059855237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5488003a30 msgr2=0x7f5488003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:44.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549ed9d700 1 --2- 172.21.15.189:0/3059855237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5488003a30 0x7f5488003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549ed9d700 1 -- 172.21.15.189:0/3059855237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54880035b0 con 0x7f5488005100 2024-04-06T12:28:44.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549e59c700 1 --2- 172.21.15.189:0/3059855237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5488003a30 0x7f5488003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:44.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549ed9d700 1 --2- 172.21.15.189:0/3059855237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5488005100 0x7f548800ba10 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f549800c290 tx=0x7f5488005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a90e899b524cee server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:44.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549d59a700 1 -- 172.21.15.189:0/3059855237 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498001e70 con 0x7f5488005100 2024-04-06T12:28:44.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.592+0000 7f549d59a700 1 -- 172.21.15.189:0/3059855237 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549800ce50 con 0x7f5488005100 2024-04-06T12:28:44.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.593+0000 7f549d59a700 1 -- 172.21.15.189:0/3059855237 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498012660 con 0x7f5488005100 2024-04-06T12:28:44.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.593+0000 7f54a4b4b700 1 -- 172.21.15.189:0/3059855237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5488005100 msgr2=0x7f548800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:44.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.593+0000 7f54a4b4b700 1 --2- 172.21.15.189:0/3059855237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5488005100 0x7f548800ba10 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f549800c290 tx=0x7f5488005880 comp rx=0 tx=0).stop 2024-04-06T12:28:44.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.593+0000 7f549ed9d700 1 -- 172.21.15.189:0/3059855237 reap_dead start 2024-04-06T12:28:44.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.593+0000 7f54a4b4b700 1 -- 172.21.15.189:0/3059855237 shutdown_connections 2024-04-06T12:28:44.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.593+0000 7f54a4b4b700 1 -- 172.21.15.189:0/3059855237 >> 172.21.15.189:0/3059855237 conn(0x7f54a00cc290 msgr2=0x7f54a00cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:44.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.593+0000 7f54a4b4b700 1 -- 172.21.15.189:0/3059855237 shutdown_connections 2024-04-06T12:28:44.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.593+0000 7f54a4b4b700 1 -- 172.21.15.189:0/3059855237 wait complete. 2024-04-06T12:28:44.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.595+0000 7f54a4b4b700 1 Processor -- start 2024-04-06T12:28:44.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.595+0000 7f54a4b4b700 1 -- start start 2024-04-06T12:28:44.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.596+0000 7f54a4b4b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5488005100 0x7f54a015f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:44.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.596+0000 7f54a4b4b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54a00c5760 0x7f54a015c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:44.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.596+0000 7f54a4b4b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a0159e80 0x7f54a015a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:44.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.596+0000 7f54a4b4b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a00b3920 con 0x7f54a00c5760 2024-04-06T12:28:44.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.596+0000 7f54a4b4b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a015a840 con 0x7f5488005100 2024-04-06T12:28:44.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.596+0000 7f54a4b4b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a0159510 con 0x7f54a0159e80 2024-04-06T12:28:44.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.596+0000 7f549dd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54a00c5760 0x7f54a015c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:44.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.596+0000 7f549dd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54a00c5760 0x7f54a015c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53426/0 (socket says 172.21.15.189:53426) 2024-04-06T12:28:44.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.596+0000 7f549dd9b700 1 -- 172.21.15.189:0/1481576506 learned_addr learned my addr 172.21.15.189:0/1481576506 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:44.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.596+0000 7f549ed9d700 1 --2- 172.21.15.189:0/1481576506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a0159e80 0x7f54a015a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:44.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.596+0000 7f549e59c700 1 --2- 172.21.15.189:0/1481576506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5488005100 0x7f54a015f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:44.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.596+0000 7f549dd9b700 1 -- 172.21.15.189:0/1481576506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a0159e80 msgr2=0x7f54a015a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:44.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.596+0000 7f549dd9b700 1 --2- 172.21.15.189:0/1481576506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a0159e80 0x7f54a015a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.596+0000 7f549dd9b700 1 -- 172.21.15.189:0/1481576506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5488005100 msgr2=0x7f54a015f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:44.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.596+0000 7f549dd9b700 1 --2- 172.21.15.189:0/1481576506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5488005100 0x7f54a015f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.596+0000 7f549dd9b700 1 -- 172.21.15.189:0/1481576506 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a015cae0 con 0x7f54a00c5760 2024-04-06T12:28:44.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.596+0000 7f549dd9b700 1 --2- 172.21.15.189:0/1481576506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54a00c5760 0x7f54a015c5a0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f54900072c0 tx=0x7f54900075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:44.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.596+0000 7f548effd700 1 -- 172.21.15.189:0/1481576506 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5490012560 con 0x7f54a00c5760 2024-04-06T12:28:44.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.597+0000 7f548effd700 1 -- 172.21.15.189:0/1481576506 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54900126c0 con 0x7f54a00c5760 2024-04-06T12:28:44.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.597+0000 7f54a4b4b700 1 -- 172.21.15.189:0/1481576506 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a00cb9a0 con 0x7f54a00c5760 2024-04-06T12:28:44.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.597+0000 7f548effd700 1 -- 172.21.15.189:0/1481576506 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5490012a70 con 0x7f54a00c5760 2024-04-06T12:28:44.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.597+0000 7f54a4b4b700 1 -- 172.21.15.189:0/1481576506 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a00b3540 con 0x7f54a00c5760 2024-04-06T12:28:44.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.598+0000 7f54a4b4b700 1 -- 172.21.15.189:0/1481576506 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a0074990 con 0x7f54a00c5760 2024-04-06T12:28:44.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.598+0000 7f548effd700 1 -- 172.21.15.189:0/1481576506 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f5490012bd0 con 0x7f54a00c5760 2024-04-06T12:28:44.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.598+0000 7f548effd700 1 --2- 172.21.15.189:0/1481576506 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f548006adb0 0x7f548006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:44.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.598+0000 7f548effd700 1 -- 172.21.15.189:0/1481576506 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5490089360 con 0x7f54a00c5760 2024-04-06T12:28:44.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.598+0000 7f549e59c700 1 --2- 172.21.15.189:0/1481576506 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f548006adb0 0x7f548006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:44.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.599+0000 7f549e59c700 1 --2- 172.21.15.189:0/1481576506 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f548006adb0 0x7f548006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f54a00723a0 tx=0x7f5494008040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:44.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.603+0000 7f548effd700 1 -- 172.21.15.189:0/1481576506 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5490054380 con 0x7f54a00c5760 2024-04-06T12:28:44.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.796+0000 7f54a4b4b700 1 -- 172.21.15.189:0/1481576506 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f54a006d3e0 con 0x7f548006adb0 2024-04-06T12:28:44.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.804+0000 7f548effd700 1 -- 172.21.15.189:0/1481576506 <== mgr.7338 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289943 (secure 0 0 0) 0x7f54940099d0 con 0x7f548006adb0 2024-04-06T12:28:44.806 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:44.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.808+0000 7f54a4b4b700 1 -- 172.21.15.189:0/1481576506 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f548006adb0 msgr2=0x7f548006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:44.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.808+0000 7f54a4b4b700 1 --2- 172.21.15.189:0/1481576506 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f548006adb0 0x7f548006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f54a00723a0 tx=0x7f5494008040 comp rx=0 tx=0).stop 2024-04-06T12:28:44.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.808+0000 7f54a4b4b700 1 -- 172.21.15.189:0/1481576506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54a00c5760 msgr2=0x7f54a015c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:44.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.808+0000 7f54a4b4b700 1 --2- 172.21.15.189:0/1481576506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54a00c5760 0x7f54a015c5a0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f54900072c0 tx=0x7f54900075d0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.808+0000 7f54a4b4b700 1 -- 172.21.15.189:0/1481576506 shutdown_connections 2024-04-06T12:28:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.808+0000 7f54a4b4b700 1 --2- 172.21.15.189:0/1481576506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a0159e80 0x7f54a015a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.808+0000 7f54a4b4b700 1 --2- 172.21.15.189:0/1481576506 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f548006adb0 0x7f548006d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.808+0000 7f54a4b4b700 1 --2- 172.21.15.189:0/1481576506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5488005100 0x7f54a015f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.808+0000 7f54a4b4b700 1 --2- 172.21.15.189:0/1481576506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54a00c5760 0x7f54a015c5a0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.808+0000 7f54a4b4b700 1 -- 172.21.15.189:0/1481576506 >> 172.21.15.189:0/1481576506 conn(0x7f54a00cc290 msgr2=0x7f54a015fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.808+0000 7f54a4b4b700 1 -- 172.21.15.189:0/1481576506 shutdown_connections 2024-04-06T12:28:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.808+0000 7f54a4b4b700 1 -- 172.21.15.189:0/1481576506 wait complete. 2024-04-06T12:28:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:28:44.817 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-06T12:28:42.988375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":97,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":327372,"kb_used_data":11976,"kb_used_omap":12,"kb_used_meta":315251,"kb_avail":754647348,"statfs":{"total":773094113280,"available":772758884352,"internally_reserved":0,"allocated":12263424,"data_stored":6668889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":322817592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000765"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406806+0000","last_change":"2024-04-06T12:28:40.681154+0000","last_active":"2024-04-06T12:28:42.406806+0000","last_peered":"2024-04-06T12:28:42.406806+0000","last_clean":"2024-04-06T12:28:42.406806+0000","last_became_active":"2024-04-06T12:28:40.680729+0000","last_became_peered":"2024-04-06T12:28:40.680729+0000","last_unstale":"2024-04-06T12:28:42.406806+0000","last_undegraded":"2024-04-06T12:28:42.406806+0000","last_fullsized":"2024-04-06T12:28:42.406806+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:44:33.737630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405646+0000","last_change":"2024-04-06T12:28:40.675174+0000","last_active":"2024-04-06T12:28:42.405646+0000","last_peered":"2024-04-06T12:28:42.405646+0000","last_clean":"2024-04-06T12:28:42.405646+0000","last_became_active":"2024-04-06T12:28:40.674898+0000","last_became_peered":"2024-04-06T12:28:40.674898+0000","last_unstale":"2024-04-06T12:28:42.405646+0000","last_undegraded":"2024-04-06T12:28:42.405646+0000","last_fullsized":"2024-04-06T12:28:42.405646+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:16:57.826003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406458+0000","last_change":"2024-04-06T12:28:40.688392+0000","last_active":"2024-04-06T12:28:42.406458+0000","last_peered":"2024-04-06T12:28:42.406458+0000","last_clean":"2024-04-06T12:28:42.406458+0000","last_became_active":"2024-04-06T12:28:40.688139+0000","last_became_peered":"2024-04-06T12:28:40.688139+0000","last_unstale":"2024-04-06T12:28:42.406458+0000","last_undegraded":"2024-04-06T12:28:42.406458+0000","last_fullsized":"2024-04-06T12:28:42.406458+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:48:24.163860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406244+0000","last_change":"2024-04-06T12:28:40.689417+0000","last_active":"2024-04-06T12:28:42.406244+0000","last_peered":"2024-04-06T12:28:42.406244+0000","last_clean":"2024-04-06T12:28:42.406244+0000","last_became_active":"2024-04-06T12:28:40.689130+0000","last_became_peered":"2024-04-06T12:28:40.689130+0000","last_unstale":"2024-04-06T12:28:42.406244+0000","last_undegraded":"2024-04-06T12:28:42.406244+0000","last_fullsized":"2024-04-06T12:28:42.406244+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:48:51.138321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669983+0000","last_change":"2024-04-06T12:28:40.675409+0000","last_active":"2024-04-06T12:28:41.669983+0000","last_peered":"2024-04-06T12:28:41.669983+0000","last_clean":"2024-04-06T12:28:41.669983+0000","last_became_active":"2024-04-06T12:28:40.674857+0000","last_became_peered":"2024-04-06T12:28:40.674857+0000","last_unstale":"2024-04-06T12:28:41.669983+0000","last_undegraded":"2024-04-06T12:28:41.669983+0000","last_fullsized":"2024-04-06T12:28:41.669983+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:48:51.138321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406885+0000","last_change":"2024-04-06T12:28:40.692157+0000","last_active":"2024-04-06T12:28:42.406885+0000","last_peered":"2024-04-06T12:28:42.406885+0000","last_clean":"2024-04-06T12:28:42.406885+0000","last_became_active":"2024-04-06T12:28:40.691757+0000","last_became_peered":"2024-04-06T12:28:40.691757+0000","last_unstale":"2024-04-06T12:28:42.406885+0000","last_undegraded":"2024-04-06T12:28:42.406885+0000","last_fullsized":"2024-04-06T12:28:42.406885+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:55:40.250566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.668955+0000","last_change":"2024-04-06T12:28:40.674320+0000","last_active":"2024-04-06T12:28:41.668955+0000","last_peered":"2024-04-06T12:28:41.668955+0000","last_clean":"2024-04-06T12:28:41.668955+0000","last_became_active":"2024-04-06T12:28:40.674049+0000","last_became_peered":"2024-04-06T12:28:40.674049+0000","last_unstale":"2024-04-06T12:28:41.668955+0000","last_undegraded":"2024-04-06T12:28:41.668955+0000","last_fullsized":"2024-04-06T12:28:41.668955+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:32:04.534993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405629+0000","last_change":"2024-04-06T12:28:40.687356+0000","last_active":"2024-04-06T12:28:42.405629+0000","last_peered":"2024-04-06T12:28:42.405629+0000","last_clean":"2024-04-06T12:28:42.405629+0000","last_became_active":"2024-04-06T12:28:40.687106+0000","last_became_peered":"2024-04-06T12:28:40.687106+0000","last_unstale":"2024-04-06T12:28:42.405629+0000","last_undegraded":"2024-04-06T12:28:42.405629+0000","last_fullsized":"2024-04-06T12:28:42.405629+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:04:33.119466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670288+0000","last_change":"2024-04-06T12:28:40.683601+0000","last_active":"2024-04-06T12:28:41.670288+0000","last_peered":"2024-04-06T12:28:41.670288+0000","last_clean":"2024-04-06T12:28:41.670288+0000","last_became_active":"2024-04-06T12:28:40.683168+0000","last_became_peered":"2024-04-06T12:28:40.683168+0000","last_unstale":"2024-04-06T12:28:41.670288+0000","last_undegraded":"2024-04-06T12:28:41.670288+0000","last_fullsized":"2024-04-06T12:28:41.670288+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:20:10.545116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.056358+0000","last_change":"2024-04-06T12:28:40.691330+0000","last_active":"2024-04-06T12:28:42.056358+0000","last_peered":"2024-04-06T12:28:42.056358+0000","last_clean":"2024-04-06T12:28:42.056358+0000","last_became_active":"2024-04-06T12:28:40.691078+0000","last_became_peered":"2024-04-06T12:28:40.691078+0000","last_unstale":"2024-04-06T12:28:42.056358+0000","last_undegraded":"2024-04-06T12:28:42.056358+0000","last_fullsized":"2024-04-06T12:28:42.056358+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:54:32.630776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669445+0000","last_change":"2024-04-06T12:28:40.683258+0000","last_active":"2024-04-06T12:28:41.669445+0000","last_peered":"2024-04-06T12:28:41.669445+0000","last_clean":"2024-04-06T12:28:41.669445+0000","last_became_active":"2024-04-06T12:28:40.682878+0000","last_became_peered":"2024-04-06T12:28:40.682878+0000","last_unstale":"2024-04-06T12:28:41.669445+0000","last_undegraded":"2024-04-06T12:28:41.669445+0000","last_fullsized":"2024-04-06T12:28:41.669445+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:04:33.119466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.056283+0000","last_change":"2024-04-06T12:28:40.678892+0000","last_active":"2024-04-06T12:28:42.056283+0000","last_peered":"2024-04-06T12:28:42.056283+0000","last_clean":"2024-04-06T12:28:42.056283+0000","last_became_active":"2024-04-06T12:28:40.678639+0000","last_became_peered":"2024-04-06T12:28:40.678639+0000","last_unstale":"2024-04-06T12:28:42.056283+0000","last_undegraded":"2024-04-06T12:28:42.056283+0000","last_fullsized":"2024-04-06T12:28:42.056283+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:42:04.654510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406347+0000","last_change":"2024-04-06T12:28:40.689835+0000","last_active":"2024-04-06T12:28:42.406347+0000","last_peered":"2024-04-06T12:28:42.406347+0000","last_clean":"2024-04-06T12:28:42.406347+0000","last_became_active":"2024-04-06T12:28:40.689623+0000","last_became_peered":"2024-04-06T12:28:40.689623+0000","last_unstale":"2024-04-06T12:28:42.406347+0000","last_undegraded":"2024-04-06T12:28:42.406347+0000","last_fullsized":"2024-04-06T12:28:42.406347+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:08:10.898942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669703+0000","last_change":"2024-04-06T12:28:40.690544+0000","last_active":"2024-04-06T12:28:41.669703+0000","last_peered":"2024-04-06T12:28:41.669703+0000","last_clean":"2024-04-06T12:28:41.669703+0000","last_became_active":"2024-04-06T12:28:40.690338+0000","last_became_peered":"2024-04-06T12:28:40.690338+0000","last_unstale":"2024-04-06T12:28:41.669703+0000","last_undegraded":"2024-04-06T12:28:41.669703+0000","last_fullsized":"2024-04-06T12:28:41.669703+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:04:09.377839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405817+0000","last_change":"2024-04-06T12:28:40.693135+0000","last_active":"2024-04-06T12:28:42.405817+0000","last_peered":"2024-04-06T12:28:42.405817+0000","last_clean":"2024-04-06T12:28:42.405817+0000","last_became_active":"2024-04-06T12:28:40.692913+0000","last_became_peered":"2024-04-06T12:28:40.692913+0000","last_unstale":"2024-04-06T12:28:42.405817+0000","last_undegraded":"2024-04-06T12:28:42.405817+0000","last_fullsized":"2024-04-06T12:28:42.405817+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:20:10.545116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405858+0000","last_change":"2024-04-06T12:28:40.683406+0000","last_active":"2024-04-06T12:28:42.405858+0000","last_peered":"2024-04-06T12:28:42.405858+0000","last_clean":"2024-04-06T12:28:42.405858+0000","last_became_active":"2024-04-06T12:28:40.682518+0000","last_became_peered":"2024-04-06T12:28:40.682518+0000","last_unstale":"2024-04-06T12:28:42.405858+0000","last_undegraded":"2024-04-06T12:28:42.405858+0000","last_fullsized":"2024-04-06T12:28:42.405858+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:05:53.803843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406833+0000","last_change":"2024-04-06T12:28:40.680009+0000","last_active":"2024-04-06T12:28:42.406833+0000","last_peered":"2024-04-06T12:28:42.406833+0000","last_clean":"2024-04-06T12:28:42.406833+0000","last_became_active":"2024-04-06T12:28:40.679780+0000","last_became_peered":"2024-04-06T12:28:40.679780+0000","last_unstale":"2024-04-06T12:28:42.406833+0000","last_undegraded":"2024-04-06T12:28:42.406833+0000","last_fullsized":"2024-04-06T12:28:42.406833+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:01:36.708557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.407216+0000","last_change":"2024-04-06T12:28:40.673426+0000","last_active":"2024-04-06T12:28:42.407216+0000","last_peered":"2024-04-06T12:28:42.407216+0000","last_clean":"2024-04-06T12:28:42.407216+0000","last_became_active":"2024-04-06T12:28:40.673180+0000","last_became_peered":"2024-04-06T12:28:40.673180+0000","last_unstale":"2024-04-06T12:28:42.407216+0000","last_undegraded":"2024-04-06T12:28:42.407216+0000","last_fullsized":"2024-04-06T12:28:42.407216+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:45:09.039210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669676+0000","last_change":"2024-04-06T12:28:40.688439+0000","last_active":"2024-04-06T12:28:41.669676+0000","last_peered":"2024-04-06T12:28:41.669676+0000","last_clean":"2024-04-06T12:28:41.669676+0000","last_became_active":"2024-04-06T12:28:40.688173+0000","last_became_peered":"2024-04-06T12:28:40.688173+0000","last_unstale":"2024-04-06T12:28:41.669676+0000","last_undegraded":"2024-04-06T12:28:41.669676+0000","last_fullsized":"2024-04-06T12:28:41.669676+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:53:23.333343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405961+0000","last_change":"2024-04-06T12:28:40.693077+0000","last_active":"2024-04-06T12:28:42.405961+0000","last_peered":"2024-04-06T12:28:42.405961+0000","last_clean":"2024-04-06T12:28:42.405961+0000","last_became_active":"2024-04-06T12:28:40.692802+0000","last_became_peered":"2024-04-06T12:28:40.692802+0000","last_unstale":"2024-04-06T12:28:42.405961+0000","last_undegraded":"2024-04-06T12:28:42.405961+0000","last_fullsized":"2024-04-06T12:28:42.405961+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:18:29.810477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670070+0000","last_change":"2024-04-06T12:28:40.673389+0000","last_active":"2024-04-06T12:28:41.670070+0000","last_peered":"2024-04-06T12:28:41.670070+0000","last_clean":"2024-04-06T12:28:41.670070+0000","last_became_active":"2024-04-06T12:28:40.673115+0000","last_became_peered":"2024-04-06T12:28:40.673115+0000","last_unstale":"2024-04-06T12:28:41.670070+0000","last_undegraded":"2024-04-06T12:28:41.670070+0000","last_fullsized":"2024-04-06T12:28:41.670070+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:44:56.202282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669861+0000","last_change":"2024-04-06T12:28:40.690264+0000","last_active":"2024-04-06T12:28:41.669861+0000","last_peered":"2024-04-06T12:28:41.669861+0000","last_clean":"2024-04-06T12:28:41.669861+0000","last_became_active":"2024-04-06T12:28:40.689951+0000","last_became_peered":"2024-04-06T12:28:40.689951+0000","last_unstale":"2024-04-06T12:28:41.669861+0000","last_undegraded":"2024-04-06T12:28:41.669861+0000","last_fullsized":"2024-04-06T12:28:41.669861+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:06:31.270541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669421+0000","last_change":"2024-04-06T12:28:40.688680+0000","last_active":"2024-04-06T12:28:41.669421+0000","last_peered":"2024-04-06T12:28:41.669421+0000","last_clean":"2024-04-06T12:28:41.669421+0000","last_became_active":"2024-04-06T12:28:40.688378+0000","last_became_peered":"2024-04-06T12:28:40.688378+0000","last_unstale":"2024-04-06T12:28:41.669421+0000","last_undegraded":"2024-04-06T12:28:41.669421+0000","last_fullsized":"2024-04-06T12:28:41.669421+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:42:19.971508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406205+0000","last_change":"2024-04-06T12:28:40.684948+0000","last_active":"2024-04-06T12:28:42.406205+0000","last_peered":"2024-04-06T12:28:42.406205+0000","last_clean":"2024-04-06T12:28:42.406205+0000","last_became_active":"2024-04-06T12:28:40.684671+0000","last_became_peered":"2024-04-06T12:28:40.684671+0000","last_unstale":"2024-04-06T12:28:42.406205+0000","last_undegraded":"2024-04-06T12:28:42.406205+0000","last_fullsized":"2024-04-06T12:28:42.406205+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:36:04.039871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405757+0000","last_change":"2024-04-06T12:28:40.679074+0000","last_active":"2024-04-06T12:28:42.405757+0000","last_peered":"2024-04-06T12:28:42.405757+0000","last_clean":"2024-04-06T12:28:42.405757+0000","last_became_active":"2024-04-06T12:28:40.678816+0000","last_became_peered":"2024-04-06T12:28:40.678816+0000","last_unstale":"2024-04-06T12:28:42.405757+0000","last_undegraded":"2024-04-06T12:28:42.405757+0000","last_fullsized":"2024-04-06T12:28:42.405757+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:33:35.890745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670306+0000","last_change":"2024-04-06T12:28:40.690156+0000","last_active":"2024-04-06T12:28:41.670306+0000","last_peered":"2024-04-06T12:28:41.670306+0000","last_clean":"2024-04-06T12:28:41.670306+0000","last_became_active":"2024-04-06T12:28:40.689888+0000","last_became_peered":"2024-04-06T12:28:40.689888+0000","last_unstale":"2024-04-06T12:28:41.670306+0000","last_undegraded":"2024-04-06T12:28:41.670306+0000","last_fullsized":"2024-04-06T12:28:41.670306+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:08:10.898942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.056246+0000","last_change":"2024-04-06T12:28:40.674619+0000","last_active":"2024-04-06T12:28:42.056246+0000","last_peered":"2024-04-06T12:28:42.056246+0000","last_clean":"2024-04-06T12:28:42.056246+0000","last_became_active":"2024-04-06T12:28:40.674371+0000","last_became_peered":"2024-04-06T12:28:40.674371+0000","last_unstale":"2024-04-06T12:28:42.056246+0000","last_undegraded":"2024-04-06T12:28:42.056246+0000","last_fullsized":"2024-04-06T12:28:42.056246+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:05:24.085708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406067+0000","last_change":"2024-04-06T12:28:40.689355+0000","last_active":"2024-04-06T12:28:42.406067+0000","last_peered":"2024-04-06T12:28:42.406067+0000","last_clean":"2024-04-06T12:28:42.406067+0000","last_became_active":"2024-04-06T12:28:40.689116+0000","last_became_peered":"2024-04-06T12:28:40.689116+0000","last_unstale":"2024-04-06T12:28:42.406067+0000","last_undegraded":"2024-04-06T12:28:42.406067+0000","last_fullsized":"2024-04-06T12:28:42.406067+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:59.084289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669548+0000","last_change":"2024-04-06T12:28:40.680709+0000","last_active":"2024-04-06T12:28:41.669548+0000","last_peered":"2024-04-06T12:28:41.669548+0000","last_clean":"2024-04-06T12:28:41.669548+0000","last_became_active":"2024-04-06T12:28:40.680441+0000","last_became_peered":"2024-04-06T12:28:40.680441+0000","last_unstale":"2024-04-06T12:28:41.669548+0000","last_undegraded":"2024-04-06T12:28:41.669548+0000","last_fullsized":"2024-04-06T12:28:41.669548+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:04:33.119466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669964+0000","last_change":"2024-04-06T12:28:40.680796+0000","last_active":"2024-04-06T12:28:41.669964+0000","last_peered":"2024-04-06T12:28:41.669964+0000","last_clean":"2024-04-06T12:28:41.669964+0000","last_became_active":"2024-04-06T12:28:40.680549+0000","last_became_peered":"2024-04-06T12:28:40.680549+0000","last_unstale":"2024-04-06T12:28:41.669964+0000","last_undegraded":"2024-04-06T12:28:41.669964+0000","last_fullsized":"2024-04-06T12:28:41.669964+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:06:16.040887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405510+0000","last_change":"2024-04-06T12:28:40.694012+0000","last_active":"2024-04-06T12:28:42.405510+0000","last_peered":"2024-04-06T12:28:42.405510+0000","last_clean":"2024-04-06T12:28:42.405510+0000","last_became_active":"2024-04-06T12:28:40.693754+0000","last_became_peered":"2024-04-06T12:28:40.693754+0000","last_unstale":"2024-04-06T12:28:42.405510+0000","last_undegraded":"2024-04-06T12:28:42.405510+0000","last_fullsized":"2024-04-06T12:28:42.405510+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:14.593914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405962+0000","last_change":"2024-04-06T12:28:40.681220+0000","last_active":"2024-04-06T12:28:42.405962+0000","last_peered":"2024-04-06T12:28:42.405962+0000","last_clean":"2024-04-06T12:28:42.405962+0000","last_became_active":"2024-04-06T12:28:40.680608+0000","last_became_peered":"2024-04-06T12:28:40.680608+0000","last_unstale":"2024-04-06T12:28:42.405962+0000","last_undegraded":"2024-04-06T12:28:42.405962+0000","last_fullsized":"2024-04-06T12:28:42.405962+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:42:39.164250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406624+0000","last_change":"2024-04-06T12:28:40.684269+0000","last_active":"2024-04-06T12:28:42.406624+0000","last_peered":"2024-04-06T12:28:42.406624+0000","last_clean":"2024-04-06T12:28:42.406624+0000","last_became_active":"2024-04-06T12:28:40.684027+0000","last_became_peered":"2024-04-06T12:28:40.684027+0000","last_unstale":"2024-04-06T12:28:42.406624+0000","last_undegraded":"2024-04-06T12:28:42.406624+0000","last_fullsized":"2024-04-06T12:28:42.406624+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:42:31.925200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"111'4","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670046+0000","last_change":"2024-04-06T12:28:40.268591+0000","last_active":"2024-04-06T12:28:41.670046+0000","last_peered":"2024-04-06T12:28:41.670046+0000","last_clean":"2024-04-06T12:28:41.670046+0000","last_became_active":"2024-04-06T12:28:40.265662+0000","last_became_peered":"2024-04-06T12:28:40.265662+0000","last_unstale":"2024-04-06T12:28:41.670046+0000","last_undegraded":"2024-04-06T12:28:41.670046+0000","last_fullsized":"2024-04-06T12:28:41.670046+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:47:58.993684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.407204+0000","last_change":"2024-04-06T12:28:40.675397+0000","last_active":"2024-04-06T12:28:42.407204+0000","last_peered":"2024-04-06T12:28:42.407204+0000","last_clean":"2024-04-06T12:28:42.407204+0000","last_became_active":"2024-04-06T12:28:40.675133+0000","last_became_peered":"2024-04-06T12:28:40.675133+0000","last_unstale":"2024-04-06T12:28:42.407204+0000","last_undegraded":"2024-04-06T12:28:42.407204+0000","last_fullsized":"2024-04-06T12:28:42.407204+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:47:12.747983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"111'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670362+0000","last_change":"2024-04-06T12:28:40.266851+0000","last_active":"2024-04-06T12:28:41.670362+0000","last_peered":"2024-04-06T12:28:41.670362+0000","last_clean":"2024-04-06T12:28:41.670362+0000","last_became_active":"2024-04-06T12:28:40.266567+0000","last_became_peered":"2024-04-06T12:28:40.266567+0000","last_unstale":"2024-04-06T12:28:41.670362+0000","last_undegraded":"2024-04-06T12:28:41.670362+0000","last_fullsized":"2024-04-06T12:28:41.670362+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:13:58.017435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.668962+0000","last_change":"2024-04-06T12:28:40.687281+0000","last_active":"2024-04-06T12:28:41.668962+0000","last_peered":"2024-04-06T12:28:41.668962+0000","last_clean":"2024-04-06T12:28:41.668962+0000","last_became_active":"2024-04-06T12:28:40.686940+0000","last_became_peered":"2024-04-06T12:28:40.686940+0000","last_unstale":"2024-04-06T12:28:41.668962+0000","last_undegraded":"2024-04-06T12:28:41.668962+0000","last_fullsized":"2024-04-06T12:28:41.668962+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:49:51.903838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"111'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669512+0000","last_change":"2024-04-06T12:28:40.260409+0000","last_active":"2024-04-06T12:28:41.669512+0000","last_peered":"2024-04-06T12:28:41.669512+0000","last_clean":"2024-04-06T12:28:41.669512+0000","last_became_active":"2024-04-06T12:28:40.260065+0000","last_became_peered":"2024-04-06T12:28:40.260065+0000","last_unstale":"2024-04-06T12:28:41.669512+0000","last_undegraded":"2024-04-06T12:28:41.669512+0000","last_fullsized":"2024-04-06T12:28:41.669512+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:16:56.816191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406981+0000","last_change":"2024-04-06T12:28:40.681978+0000","last_active":"2024-04-06T12:28:42.406981+0000","last_peered":"2024-04-06T12:28:42.406981+0000","last_clean":"2024-04-06T12:28:42.406981+0000","last_became_active":"2024-04-06T12:28:40.681614+0000","last_became_peered":"2024-04-06T12:28:40.681614+0000","last_unstale":"2024-04-06T12:28:42.406981+0000","last_undegraded":"2024-04-06T12:28:42.406981+0000","last_fullsized":"2024-04-06T12:28:42.406981+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:47:18.019643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"111'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669554+0000","last_change":"2024-04-06T12:28:40.264333+0000","last_active":"2024-04-06T12:28:41.669554+0000","last_peered":"2024-04-06T12:28:41.669554+0000","last_clean":"2024-04-06T12:28:41.669554+0000","last_became_active":"2024-04-06T12:28:40.264107+0000","last_became_peered":"2024-04-06T12:28:40.264107+0000","last_unstale":"2024-04-06T12:28:41.669554+0000","last_undegraded":"2024-04-06T12:28:41.669554+0000","last_fullsized":"2024-04-06T12:28:41.669554+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:07:20.434672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405742+0000","last_change":"2024-04-06T12:28:40.693029+0000","last_active":"2024-04-06T12:28:42.405742+0000","last_peered":"2024-04-06T12:28:42.405742+0000","last_clean":"2024-04-06T12:28:42.405742+0000","last_became_active":"2024-04-06T12:28:40.692780+0000","last_became_peered":"2024-04-06T12:28:40.692780+0000","last_unstale":"2024-04-06T12:28:42.405742+0000","last_undegraded":"2024-04-06T12:28:42.405742+0000","last_fullsized":"2024-04-06T12:28:42.405742+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:53:23.333343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"111'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670241+0000","last_change":"2024-04-06T12:28:40.268542+0000","last_active":"2024-04-06T12:28:41.670241+0000","last_peered":"2024-04-06T12:28:41.670241+0000","last_clean":"2024-04-06T12:28:41.670241+0000","last_became_active":"2024-04-06T12:28:40.268275+0000","last_became_peered":"2024-04-06T12:28:40.268275+0000","last_unstale":"2024-04-06T12:28:41.670241+0000","last_undegraded":"2024-04-06T12:28:41.670241+0000","last_fullsized":"2024-04-06T12:28:41.670241+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:31:13.268239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669524+0000","last_change":"2024-04-06T12:28:40.681615+0000","last_active":"2024-04-06T12:28:41.669524+0000","last_peered":"2024-04-06T12:28:41.669524+0000","last_clean":"2024-04-06T12:28:41.669524+0000","last_became_active":"2024-04-06T12:28:40.681282+0000","last_became_peered":"2024-04-06T12:28:40.681282+0000","last_unstale":"2024-04-06T12:28:41.669524+0000","last_undegraded":"2024-04-06T12:28:41.669524+0000","last_fullsized":"2024-04-06T12:28:41.669524+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:45:21.708635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"111'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406452+0000","last_change":"2024-04-06T12:28:40.263780+0000","last_active":"2024-04-06T12:28:42.406452+0000","last_peered":"2024-04-06T12:28:42.406452+0000","last_clean":"2024-04-06T12:28:42.406452+0000","last_became_active":"2024-04-06T12:28:40.263416+0000","last_became_peered":"2024-04-06T12:28:40.263416+0000","last_unstale":"2024-04-06T12:28:42.406452+0000","last_undegraded":"2024-04-06T12:28:42.406452+0000","last_fullsized":"2024-04-06T12:28:42.406452+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:44:39.076665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406905+0000","last_change":"2024-04-06T12:28:40.692001+0000","last_active":"2024-04-06T12:28:42.406905+0000","last_peered":"2024-04-06T12:28:42.406905+0000","last_clean":"2024-04-06T12:28:42.406905+0000","last_became_active":"2024-04-06T12:28:40.691403+0000","last_became_peered":"2024-04-06T12:28:40.691403+0000","last_unstale":"2024-04-06T12:28:42.406905+0000","last_undegraded":"2024-04-06T12:28:42.406905+0000","last_fullsized":"2024-04-06T12:28:42.406905+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:37.635782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406705+0000","last_change":"2024-04-06T12:28:40.261521+0000","last_active":"2024-04-06T12:28:42.406705+0000","last_peered":"2024-04-06T12:28:42.406705+0000","last_clean":"2024-04-06T12:28:42.406705+0000","last_became_active":"2024-04-06T12:28:40.261256+0000","last_became_peered":"2024-04-06T12:28:40.261256+0000","last_unstale":"2024-04-06T12:28:42.406705+0000","last_undegraded":"2024-04-06T12:28:42.406705+0000","last_fullsized":"2024-04-06T12:28:42.406705+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:51:16.747872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406464+0000","last_change":"2024-04-06T12:28:40.687255+0000","last_active":"2024-04-06T12:28:42.406464+0000","last_peered":"2024-04-06T12:28:42.406464+0000","last_clean":"2024-04-06T12:28:42.406464+0000","last_became_active":"2024-04-06T12:28:40.686989+0000","last_became_peered":"2024-04-06T12:28:40.686989+0000","last_unstale":"2024-04-06T12:28:42.406464+0000","last_undegraded":"2024-04-06T12:28:42.406464+0000","last_fullsized":"2024-04-06T12:28:42.406464+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:04:09.377839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669022+0000","last_change":"2024-04-06T12:28:40.255069+0000","last_active":"2024-04-06T12:28:41.669022+0000","last_peered":"2024-04-06T12:28:41.669022+0000","last_clean":"2024-04-06T12:28:41.669022+0000","last_became_active":"2024-04-06T12:28:40.254778+0000","last_became_peered":"2024-04-06T12:28:40.254778+0000","last_unstale":"2024-04-06T12:28:41.669022+0000","last_undegraded":"2024-04-06T12:28:41.669022+0000","last_fullsized":"2024-04-06T12:28:41.669022+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:23:57.255553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669989+0000","last_change":"2024-04-06T12:28:40.677839+0000","last_active":"2024-04-06T12:28:41.669989+0000","last_peered":"2024-04-06T12:28:41.669989+0000","last_clean":"2024-04-06T12:28:41.669989+0000","last_became_active":"2024-04-06T12:28:40.676824+0000","last_became_peered":"2024-04-06T12:28:40.676824+0000","last_unstale":"2024-04-06T12:28:41.669989+0000","last_undegraded":"2024-04-06T12:28:41.669989+0000","last_fullsized":"2024-04-06T12:28:41.669989+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:59.084289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405849+0000","last_change":"2024-04-06T12:28:40.265329+0000","last_active":"2024-04-06T12:28:42.405849+0000","last_peered":"2024-04-06T12:28:42.405849+0000","last_clean":"2024-04-06T12:28:42.405849+0000","last_became_active":"2024-04-06T12:28:40.264509+0000","last_became_peered":"2024-04-06T12:28:40.264509+0000","last_unstale":"2024-04-06T12:28:42.405849+0000","last_undegraded":"2024-04-06T12:28:42.405849+0000","last_fullsized":"2024-04-06T12:28:42.405849+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:09:21.699469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669418+0000","last_change":"2024-04-06T12:28:40.675175+0000","last_active":"2024-04-06T12:28:41.669418+0000","last_peered":"2024-04-06T12:28:41.669418+0000","last_clean":"2024-04-06T12:28:41.669418+0000","last_became_active":"2024-04-06T12:28:40.674899+0000","last_became_peered":"2024-04-06T12:28:40.674899+0000","last_unstale":"2024-04-06T12:28:41.669418+0000","last_undegraded":"2024-04-06T12:28:41.669418+0000","last_fullsized":"2024-04-06T12:28:41.669418+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:59:47.830391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406842+0000","last_change":"2024-04-06T12:28:40.263044+0000","last_active":"2024-04-06T12:28:42.406842+0000","last_peered":"2024-04-06T12:28:42.406842+0000","last_clean":"2024-04-06T12:28:42.406842+0000","last_became_active":"2024-04-06T12:28:40.262786+0000","last_became_peered":"2024-04-06T12:28:40.262786+0000","last_unstale":"2024-04-06T12:28:42.406842+0000","last_undegraded":"2024-04-06T12:28:42.406842+0000","last_fullsized":"2024-04-06T12:28:42.406842+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:33:58.849161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406646+0000","last_change":"2024-04-06T12:28:40.687500+0000","last_active":"2024-04-06T12:28:42.406646+0000","last_peered":"2024-04-06T12:28:42.406646+0000","last_clean":"2024-04-06T12:28:42.406646+0000","last_became_active":"2024-04-06T12:28:40.681387+0000","last_became_peered":"2024-04-06T12:28:40.681387+0000","last_unstale":"2024-04-06T12:28:42.406646+0000","last_undegraded":"2024-04-06T12:28:42.406646+0000","last_fullsized":"2024-04-06T12:28:42.406646+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:10:48.365828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"111'3","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669819+0000","last_change":"2024-04-06T12:28:40.269680+0000","last_active":"2024-04-06T12:28:41.669819+0000","last_peered":"2024-04-06T12:28:41.669819+0000","last_clean":"2024-04-06T12:28:41.669819+0000","last_became_active":"2024-04-06T12:28:40.269382+0000","last_became_peered":"2024-04-06T12:28:40.269382+0000","last_unstale":"2024-04-06T12:28:41.669819+0000","last_undegraded":"2024-04-06T12:28:41.669819+0000","last_fullsized":"2024-04-06T12:28:41.669819+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:22:51.915093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406607+0000","last_change":"2024-04-06T12:28:40.687087+0000","last_active":"2024-04-06T12:28:42.406607+0000","last_peered":"2024-04-06T12:28:42.406607+0000","last_clean":"2024-04-06T12:28:42.406607+0000","last_became_active":"2024-04-06T12:28:40.686841+0000","last_became_peered":"2024-04-06T12:28:40.686841+0000","last_unstale":"2024-04-06T12:28:42.406607+0000","last_undegraded":"2024-04-06T12:28:42.406607+0000","last_fullsized":"2024-04-06T12:28:42.406607+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:27:34.410340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"111'6","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406718+0000","last_change":"2024-04-06T12:28:40.266700+0000","last_active":"2024-04-06T12:28:42.406718+0000","last_peered":"2024-04-06T12:28:42.406718+0000","last_clean":"2024-04-06T12:28:42.406718+0000","last_became_active":"2024-04-06T12:28:40.266424+0000","last_became_peered":"2024-04-06T12:28:40.266424+0000","last_unstale":"2024-04-06T12:28:42.406718+0000","last_undegraded":"2024-04-06T12:28:42.406718+0000","last_fullsized":"2024-04-06T12:28:42.406718+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:13.861933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405850+0000","last_change":"2024-04-06T12:28:40.689348+0000","last_active":"2024-04-06T12:28:42.405850+0000","last_peered":"2024-04-06T12:28:42.405850+0000","last_clean":"2024-04-06T12:28:42.405850+0000","last_became_active":"2024-04-06T12:28:40.689089+0000","last_became_peered":"2024-04-06T12:28:40.689089+0000","last_unstale":"2024-04-06T12:28:42.405850+0000","last_undegraded":"2024-04-06T12:28:42.405850+0000","last_fullsized":"2024-04-06T12:28:42.405850+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:16:57.826003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"111'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669737+0000","last_change":"2024-04-06T12:28:40.263978+0000","last_active":"2024-04-06T12:28:41.669737+0000","last_peered":"2024-04-06T12:28:41.669737+0000","last_clean":"2024-04-06T12:28:41.669737+0000","last_became_active":"2024-04-06T12:28:40.263711+0000","last_became_peered":"2024-04-06T12:28:40.263711+0000","last_unstale":"2024-04-06T12:28:41.669737+0000","last_undegraded":"2024-04-06T12:28:41.669737+0000","last_fullsized":"2024-04-06T12:28:41.669737+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:44:55.192470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669357+0000","last_change":"2024-04-06T12:28:40.676784+0000","last_active":"2024-04-06T12:28:41.669357+0000","last_peered":"2024-04-06T12:28:41.669357+0000","last_clean":"2024-04-06T12:28:41.669357+0000","last_became_active":"2024-04-06T12:28:40.676509+0000","last_became_peered":"2024-04-06T12:28:40.676509+0000","last_unstale":"2024-04-06T12:28:41.669357+0000","last_undegraded":"2024-04-06T12:28:41.669357+0000","last_fullsized":"2024-04-06T12:28:41.669357+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:42:19.971508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406993+0000","last_change":"2024-04-06T12:28:40.267877+0000","last_active":"2024-04-06T12:28:42.406993+0000","last_peered":"2024-04-06T12:28:42.406993+0000","last_clean":"2024-04-06T12:28:42.406993+0000","last_became_active":"2024-04-06T12:28:40.267602+0000","last_became_peered":"2024-04-06T12:28:40.267602+0000","last_unstale":"2024-04-06T12:28:42.406993+0000","last_undegraded":"2024-04-06T12:28:42.406993+0000","last_fullsized":"2024-04-06T12:28:42.406993+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:10:47.356015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.056302+0000","last_change":"2024-04-06T12:28:40.688068+0000","last_active":"2024-04-06T12:28:42.056302+0000","last_peered":"2024-04-06T12:28:42.056302+0000","last_clean":"2024-04-06T12:28:42.056302+0000","last_became_active":"2024-04-06T12:28:40.687785+0000","last_became_peered":"2024-04-06T12:28:40.687785+0000","last_unstale":"2024-04-06T12:28:42.056302+0000","last_undegraded":"2024-04-06T12:28:42.056302+0000","last_fullsized":"2024-04-06T12:28:42.056302+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:20:18.808689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405925+0000","last_change":"2024-04-06T12:28:40.270013+0000","last_active":"2024-04-06T12:28:42.405925+0000","last_peered":"2024-04-06T12:28:42.405925+0000","last_clean":"2024-04-06T12:28:42.405925+0000","last_became_active":"2024-04-06T12:28:40.269562+0000","last_became_peered":"2024-04-06T12:28:40.269562+0000","last_unstale":"2024-04-06T12:28:42.405925+0000","last_undegraded":"2024-04-06T12:28:42.405925+0000","last_fullsized":"2024-04-06T12:28:42.405925+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:45:20.698822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.056410+0000","last_change":"2024-04-06T12:28:40.683998+0000","last_active":"2024-04-06T12:28:42.056410+0000","last_peered":"2024-04-06T12:28:42.056410+0000","last_clean":"2024-04-06T12:28:42.056410+0000","last_became_active":"2024-04-06T12:28:40.683721+0000","last_became_peered":"2024-04-06T12:28:40.683721+0000","last_unstale":"2024-04-06T12:28:42.056410+0000","last_undegraded":"2024-04-06T12:28:42.056410+0000","last_fullsized":"2024-04-06T12:28:42.056410+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:40:18.946044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"111'8","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.407301+0000","last_change":"2024-04-06T12:28:40.264869+0000","last_active":"2024-04-06T12:28:42.407301+0000","last_peered":"2024-04-06T12:28:42.407301+0000","last_clean":"2024-04-06T12:28:42.407301+0000","last_became_active":"2024-04-06T12:28:40.264614+0000","last_became_peered":"2024-04-06T12:28:40.264614+0000","last_unstale":"2024-04-06T12:28:42.407301+0000","last_undegraded":"2024-04-06T12:28:42.407301+0000","last_fullsized":"2024-04-06T12:28:42.407301+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:18:23.534593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406519+0000","last_change":"2024-04-06T12:28:40.685374+0000","last_active":"2024-04-06T12:28:42.406519+0000","last_peered":"2024-04-06T12:28:42.406519+0000","last_clean":"2024-04-06T12:28:42.406519+0000","last_became_active":"2024-04-06T12:28:40.684181+0000","last_became_peered":"2024-04-06T12:28:40.684181+0000","last_unstale":"2024-04-06T12:28:42.406519+0000","last_undegraded":"2024-04-06T12:28:42.406519+0000","last_fullsized":"2024-04-06T12:28:42.406519+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:25:51.107371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"111'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670134+0000","last_change":"2024-04-06T12:28:40.265372+0000","last_active":"2024-04-06T12:28:41.670134+0000","last_peered":"2024-04-06T12:28:41.670134+0000","last_clean":"2024-04-06T12:28:41.670134+0000","last_became_active":"2024-04-06T12:28:40.265098+0000","last_became_peered":"2024-04-06T12:28:40.265098+0000","last_unstale":"2024-04-06T12:28:41.670134+0000","last_undegraded":"2024-04-06T12:28:41.670134+0000","last_fullsized":"2024-04-06T12:28:41.670134+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:31:13.268239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670225+0000","last_change":"2024-04-06T12:28:40.684162+0000","last_active":"2024-04-06T12:28:41.670225+0000","last_peered":"2024-04-06T12:28:41.670225+0000","last_clean":"2024-04-06T12:28:41.670225+0000","last_became_active":"2024-04-06T12:28:40.683748+0000","last_became_peered":"2024-04-06T12:28:40.683748+0000","last_unstale":"2024-04-06T12:28:41.670225+0000","last_undegraded":"2024-04-06T12:28:41.670225+0000","last_fullsized":"2024-04-06T12:28:41.670225+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:18:55.287857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"111'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669064+0000","last_change":"2024-04-06T12:28:40.264845+0000","last_active":"2024-04-06T12:28:41.669064+0000","last_peered":"2024-04-06T12:28:41.669064+0000","last_clean":"2024-04-06T12:28:41.669064+0000","last_became_active":"2024-04-06T12:28:40.264607+0000","last_became_peered":"2024-04-06T12:28:40.264607+0000","last_unstale":"2024-04-06T12:28:41.669064+0000","last_undegraded":"2024-04-06T12:28:41.669064+0000","last_fullsized":"2024-04-06T12:28:41.669064+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:06:30.260729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670970+0000","last_change":"2024-04-06T12:28:40.677641+0000","last_active":"2024-04-06T12:28:41.670970+0000","last_peered":"2024-04-06T12:28:41.670970+0000","last_clean":"2024-04-06T12:28:41.670970+0000","last_became_active":"2024-04-06T12:28:40.677361+0000","last_became_peered":"2024-04-06T12:28:40.677361+0000","last_unstale":"2024-04-06T12:28:41.670970+0000","last_undegraded":"2024-04-06T12:28:41.670970+0000","last_fullsized":"2024-04-06T12:28:41.670970+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:18:29.810477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"111'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670273+0000","last_change":"2024-04-06T12:28:40.270490+0000","last_active":"2024-04-06T12:28:41.670273+0000","last_peered":"2024-04-06T12:28:41.670273+0000","last_clean":"2024-04-06T12:28:41.670273+0000","last_became_active":"2024-04-06T12:28:40.270201+0000","last_became_peered":"2024-04-06T12:28:40.270201+0000","last_unstale":"2024-04-06T12:28:41.670273+0000","last_undegraded":"2024-04-06T12:28:41.670273+0000","last_fullsized":"2024-04-06T12:28:41.670273+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:59:43.892496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406070+0000","last_change":"2024-04-06T12:28:40.691581+0000","last_active":"2024-04-06T12:28:42.406070+0000","last_peered":"2024-04-06T12:28:42.406070+0000","last_clean":"2024-04-06T12:28:42.406070+0000","last_became_active":"2024-04-06T12:28:40.691306+0000","last_became_peered":"2024-04-06T12:28:40.691306+0000","last_unstale":"2024-04-06T12:28:42.406070+0000","last_undegraded":"2024-04-06T12:28:42.406070+0000","last_fullsized":"2024-04-06T12:28:42.406070+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:26.747580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"111'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405630+0000","last_change":"2024-04-06T12:28:40.263453+0000","last_active":"2024-04-06T12:28:42.405630+0000","last_peered":"2024-04-06T12:28:42.405630+0000","last_clean":"2024-04-06T12:28:42.405630+0000","last_became_active":"2024-04-06T12:28:40.263153+0000","last_became_peered":"2024-04-06T12:28:40.263153+0000","last_unstale":"2024-04-06T12:28:42.405630+0000","last_undegraded":"2024-04-06T12:28:42.405630+0000","last_fullsized":"2024-04-06T12:28:42.405630+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:22:51.915093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405640+0000","last_change":"2024-04-06T12:28:40.674977+0000","last_active":"2024-04-06T12:28:42.405640+0000","last_peered":"2024-04-06T12:28:42.405640+0000","last_clean":"2024-04-06T12:28:42.405640+0000","last_became_active":"2024-04-06T12:28:40.674677+0000","last_became_peered":"2024-04-06T12:28:40.674677+0000","last_unstale":"2024-04-06T12:28:42.405640+0000","last_undegraded":"2024-04-06T12:28:42.405640+0000","last_fullsized":"2024-04-06T12:28:42.405640+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:32:19.699532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.056204+0000","last_change":"2024-04-06T12:28:40.262648+0000","last_active":"2024-04-06T12:28:42.056204+0000","last_peered":"2024-04-06T12:28:42.056204+0000","last_clean":"2024-04-06T12:28:42.056204+0000","last_became_active":"2024-04-06T12:28:40.262363+0000","last_became_peered":"2024-04-06T12:28:40.262363+0000","last_unstale":"2024-04-06T12:28:42.056204+0000","last_undegraded":"2024-04-06T12:28:42.056204+0000","last_fullsized":"2024-04-06T12:28:42.056204+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:31:14.983665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405474+0000","last_change":"2024-04-06T12:28:40.688650+0000","last_active":"2024-04-06T12:28:42.405474+0000","last_peered":"2024-04-06T12:28:42.405474+0000","last_clean":"2024-04-06T12:28:42.405474+0000","last_became_active":"2024-04-06T12:28:40.687839+0000","last_became_peered":"2024-04-06T12:28:40.687839+0000","last_unstale":"2024-04-06T12:28:42.405474+0000","last_undegraded":"2024-04-06T12:28:42.405474+0000","last_fullsized":"2024-04-06T12:28:42.405474+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:44:56.202282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405668+0000","last_change":"2024-04-06T12:28:40.269996+0000","last_active":"2024-04-06T12:28:42.405668+0000","last_peered":"2024-04-06T12:28:42.405668+0000","last_clean":"2024-04-06T12:28:42.405668+0000","last_became_active":"2024-04-06T12:28:40.269522+0000","last_became_peered":"2024-04-06T12:28:40.269522+0000","last_unstale":"2024-04-06T12:28:42.405668+0000","last_undegraded":"2024-04-06T12:28:42.405668+0000","last_fullsized":"2024-04-06T12:28:42.405668+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:20:16.164471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406030+0000","last_change":"2024-04-06T12:28:40.681010+0000","last_active":"2024-04-06T12:28:42.406030+0000","last_peered":"2024-04-06T12:28:42.406030+0000","last_clean":"2024-04-06T12:28:42.406030+0000","last_became_active":"2024-04-06T12:28:40.680002+0000","last_became_peered":"2024-04-06T12:28:40.680002+0000","last_unstale":"2024-04-06T12:28:42.406030+0000","last_undegraded":"2024-04-06T12:28:42.406030+0000","last_fullsized":"2024-04-06T12:28:42.406030+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:21:36.738869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.056355+0000","last_change":"2024-04-06T12:28:40.260606+0000","last_active":"2024-04-06T12:28:42.056355+0000","last_peered":"2024-04-06T12:28:42.056355+0000","last_clean":"2024-04-06T12:28:42.056355+0000","last_became_active":"2024-04-06T12:28:40.260317+0000","last_became_peered":"2024-04-06T12:28:40.260317+0000","last_unstale":"2024-04-06T12:28:42.056355+0000","last_undegraded":"2024-04-06T12:28:42.056355+0000","last_fullsized":"2024-04-06T12:28:42.056355+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:14:25.108123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669521+0000","last_change":"2024-04-06T12:28:40.680189+0000","last_active":"2024-04-06T12:28:41.669521+0000","last_peered":"2024-04-06T12:28:41.669521+0000","last_clean":"2024-04-06T12:28:41.669521+0000","last_became_active":"2024-04-06T12:28:40.679236+0000","last_became_peered":"2024-04-06T12:28:40.679236+0000","last_unstale":"2024-04-06T12:28:41.669521+0000","last_undegraded":"2024-04-06T12:28:41.669521+0000","last_fullsized":"2024-04-06T12:28:41.669521+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:18:24.544405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"111'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669183+0000","last_change":"2024-04-06T12:28:40.267091+0000","last_active":"2024-04-06T12:28:41.669183+0000","last_peered":"2024-04-06T12:28:41.669183+0000","last_clean":"2024-04-06T12:28:41.669183+0000","last_became_active":"2024-04-06T12:28:40.266679+0000","last_became_peered":"2024-04-06T12:28:40.266679+0000","last_unstale":"2024-04-06T12:28:41.669183+0000","last_undegraded":"2024-04-06T12:28:41.669183+0000","last_fullsized":"2024-04-06T12:28:41.669183+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:13.584102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.407301+0000","last_change":"2024-04-06T12:28:40.684385+0000","last_active":"2024-04-06T12:28:42.407301+0000","last_peered":"2024-04-06T12:28:42.407301+0000","last_clean":"2024-04-06T12:28:42.407301+0000","last_became_active":"2024-04-06T12:28:40.684108+0000","last_became_peered":"2024-04-06T12:28:40.684108+0000","last_unstale":"2024-04-06T12:28:42.407301+0000","last_undegraded":"2024-04-06T12:28:42.407301+0000","last_fullsized":"2024-04-06T12:28:42.407301+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:24:57.296309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670215+0000","last_change":"2024-04-06T12:28:40.270102+0000","last_active":"2024-04-06T12:28:41.670215+0000","last_peered":"2024-04-06T12:28:41.670215+0000","last_clean":"2024-04-06T12:28:41.670215+0000","last_became_active":"2024-04-06T12:28:40.269825+0000","last_became_peered":"2024-04-06T12:28:40.269825+0000","last_unstale":"2024-04-06T12:28:41.670215+0000","last_undegraded":"2024-04-06T12:28:41.670215+0000","last_fullsized":"2024-04-06T12:28:41.670215+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:20:16.164471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406953+0000","last_change":"2024-04-06T12:28:40.680919+0000","last_active":"2024-04-06T12:28:42.406953+0000","last_peered":"2024-04-06T12:28:42.406953+0000","last_clean":"2024-04-06T12:28:42.406953+0000","last_became_active":"2024-04-06T12:28:40.680665+0000","last_became_peered":"2024-04-06T12:28:40.680665+0000","last_unstale":"2024-04-06T12:28:42.406953+0000","last_undegraded":"2024-04-06T12:28:42.406953+0000","last_fullsized":"2024-04-06T12:28:42.406953+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:32:13.448506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669545+0000","last_change":"2024-04-06T12:28:40.269837+0000","last_active":"2024-04-06T12:28:41.669545+0000","last_peered":"2024-04-06T12:28:41.669545+0000","last_clean":"2024-04-06T12:28:41.669545+0000","last_became_active":"2024-04-06T12:28:40.269481+0000","last_became_peered":"2024-04-06T12:28:40.269481+0000","last_unstale":"2024-04-06T12:28:41.669545+0000","last_undegraded":"2024-04-06T12:28:41.669545+0000","last_fullsized":"2024-04-06T12:28:41.669545+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:09:21.699469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405850+0000","last_change":"2024-04-06T12:28:40.683273+0000","last_active":"2024-04-06T12:28:42.405850+0000","last_peered":"2024-04-06T12:28:42.405850+0000","last_clean":"2024-04-06T12:28:42.405850+0000","last_became_active":"2024-04-06T12:28:40.683003+0000","last_became_peered":"2024-04-06T12:28:40.683003+0000","last_unstale":"2024-04-06T12:28:42.405850+0000","last_undegraded":"2024-04-06T12:28:42.405850+0000","last_fullsized":"2024-04-06T12:28:42.405850+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:23:58.265366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669627+0000","last_change":"2024-04-06T12:28:40.263840+0000","last_active":"2024-04-06T12:28:41.669627+0000","last_peered":"2024-04-06T12:28:41.669627+0000","last_clean":"2024-04-06T12:28:41.669627+0000","last_became_active":"2024-04-06T12:28:40.262653+0000","last_became_peered":"2024-04-06T12:28:40.262653+0000","last_unstale":"2024-04-06T12:28:41.669627+0000","last_undegraded":"2024-04-06T12:28:41.669627+0000","last_fullsized":"2024-04-06T12:28:41.669627+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:03:32.755928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.056136+0000","last_change":"2024-04-06T12:28:40.682380+0000","last_active":"2024-04-06T12:28:42.056136+0000","last_peered":"2024-04-06T12:28:42.056136+0000","last_clean":"2024-04-06T12:28:42.056136+0000","last_became_active":"2024-04-06T12:28:40.682116+0000","last_became_peered":"2024-04-06T12:28:40.682116+0000","last_unstale":"2024-04-06T12:28:42.056136+0000","last_undegraded":"2024-04-06T12:28:42.056136+0000","last_fullsized":"2024-04-06T12:28:42.056136+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:30:04.557762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406292+0000","last_change":"2024-04-06T12:28:40.267887+0000","last_active":"2024-04-06T12:28:42.406292+0000","last_peered":"2024-04-06T12:28:42.406292+0000","last_clean":"2024-04-06T12:28:42.406292+0000","last_became_active":"2024-04-06T12:28:40.267615+0000","last_became_peered":"2024-04-06T12:28:40.267615+0000","last_unstale":"2024-04-06T12:28:42.406292+0000","last_undegraded":"2024-04-06T12:28:42.406292+0000","last_fullsized":"2024-04-06T12:28:42.406292+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:44:32.727817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406602+0000","last_change":"2024-04-06T12:28:40.688761+0000","last_active":"2024-04-06T12:28:42.406602+0000","last_peered":"2024-04-06T12:28:42.406602+0000","last_clean":"2024-04-06T12:28:42.406602+0000","last_became_active":"2024-04-06T12:28:40.688502+0000","last_became_peered":"2024-04-06T12:28:40.688502+0000","last_unstale":"2024-04-06T12:28:42.406602+0000","last_undegraded":"2024-04-06T12:28:42.406602+0000","last_fullsized":"2024-04-06T12:28:42.406602+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:14.871746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405555+0000","last_change":"2024-04-06T12:28:40.263982+0000","last_active":"2024-04-06T12:28:42.405555+0000","last_peered":"2024-04-06T12:28:42.405555+0000","last_clean":"2024-04-06T12:28:42.405555+0000","last_became_active":"2024-04-06T12:28:40.263315+0000","last_became_peered":"2024-04-06T12:28:40.263315+0000","last_unstale":"2024-04-06T12:28:42.405555+0000","last_undegraded":"2024-04-06T12:28:42.405555+0000","last_fullsized":"2024-04-06T12:28:42.405555+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:59:43.892496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669669+0000","last_change":"2024-04-06T12:28:40.688444+0000","last_active":"2024-04-06T12:28:41.669669+0000","last_peered":"2024-04-06T12:28:41.669669+0000","last_clean":"2024-04-06T12:28:41.669669+0000","last_became_active":"2024-04-06T12:28:40.688175+0000","last_became_peered":"2024-04-06T12:28:40.688175+0000","last_unstale":"2024-04-06T12:28:41.669669+0000","last_undegraded":"2024-04-06T12:28:41.669669+0000","last_fullsized":"2024-04-06T12:28:41.669669+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:43:56.668566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670121+0000","last_change":"2024-04-06T12:28:40.258231+0000","last_active":"2024-04-06T12:28:41.670121+0000","last_peered":"2024-04-06T12:28:41.670121+0000","last_clean":"2024-04-06T12:28:41.670121+0000","last_became_active":"2024-04-06T12:28:40.257935+0000","last_became_peered":"2024-04-06T12:28:40.257935+0000","last_unstale":"2024-04-06T12:28:41.670121+0000","last_undegraded":"2024-04-06T12:28:41.670121+0000","last_fullsized":"2024-04-06T12:28:41.670121+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:19.826267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406138+0000","last_change":"2024-04-06T12:28:40.681058+0000","last_active":"2024-04-06T12:28:42.406138+0000","last_peered":"2024-04-06T12:28:42.406138+0000","last_clean":"2024-04-06T12:28:42.406138+0000","last_became_active":"2024-04-06T12:28:40.680282+0000","last_became_peered":"2024-04-06T12:28:40.680282+0000","last_unstale":"2024-04-06T12:28:42.406138+0000","last_undegraded":"2024-04-06T12:28:42.406138+0000","last_fullsized":"2024-04-06T12:28:42.406138+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:02:42.378007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669118+0000","last_change":"2024-04-06T12:28:40.263384+0000","last_active":"2024-04-06T12:28:41.669118+0000","last_peered":"2024-04-06T12:28:41.669118+0000","last_clean":"2024-04-06T12:28:41.669118+0000","last_became_active":"2024-04-06T12:28:40.263089+0000","last_became_peered":"2024-04-06T12:28:40.263089+0000","last_unstale":"2024-04-06T12:28:41.669118+0000","last_undegraded":"2024-04-06T12:28:41.669118+0000","last_fullsized":"2024-04-06T12:28:41.669118+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:04:08.368026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669098+0000","last_change":"2024-04-06T12:28:40.677430+0000","last_active":"2024-04-06T12:28:41.669098+0000","last_peered":"2024-04-06T12:28:41.669098+0000","last_clean":"2024-04-06T12:28:41.669098+0000","last_became_active":"2024-04-06T12:28:40.677175+0000","last_became_peered":"2024-04-06T12:28:40.677175+0000","last_unstale":"2024-04-06T12:28:41.669098+0000","last_undegraded":"2024-04-06T12:28:41.669098+0000","last_fullsized":"2024-04-06T12:28:41.669098+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:08:10.898942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406332+0000","last_change":"2024-04-06T12:28:40.263615+0000","last_active":"2024-04-06T12:28:42.406332+0000","last_peered":"2024-04-06T12:28:42.406332+0000","last_clean":"2024-04-06T12:28:42.406332+0000","last_became_active":"2024-04-06T12:28:40.263135+0000","last_became_peered":"2024-04-06T12:28:40.263135+0000","last_unstale":"2024-04-06T12:28:42.406332+0000","last_undegraded":"2024-04-06T12:28:42.406332+0000","last_fullsized":"2024-04-06T12:28:42.406332+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:22:51.915093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"101'183","reported_seq":498,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-06T12:28:40.665623+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:28:40.665623+0000","last_peered":"2024-04-06T12:28:40.665623+0000","last_clean":"2024-04-06T12:28:40.665623+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:28:40.665623+0000","last_undegraded":"2024-04-06T12:28:40.665623+0000","last_fullsized":"2024-04-06T12:28:40.665623+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575048,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59296,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312544,"statfs":{"total":96636764160,"available":96576045056,"internally_reserved":0,"allocated":1982464,"data_stored":1277317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.53200000000000003,"15min":0.55500000000000005},"min":{"1min":0.32000000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.78200000000000003,"5min":1.143,"15min":1.143},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.496,"5min":0.52900000000000003,"15min":0.55000000000000004},"min":{"1min":0.372,"5min":0.246,"15min":0.246},"max":{"1min":0.66100000000000003,"5min":0.77100000000000002,"15min":0.78200000000000003},"last":0.79100000000000004}]},{"osd":1,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.53200000000000003,"15min":0.55000000000000004},"min":{"1min":0.315,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64200000000000002,"5min":0.77100000000000002,"15min":0.81399999999999995},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.54300000000000004,"15min":0.56899999999999995},"min":{"1min":0.379,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.74199999999999999,"5min":0.79400000000000004,"15min":0.79600000000000004},"last":0.60799999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53600000000000003,"15min":0.55500000000000005},"min":{"1min":0.39400000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":0.83299999999999996},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.55400000000000005,"15min":0.57599999999999996},"min":{"1min":0.39400000000000002,"5min":0.34499999999999997,"15min":0.24199999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.82299999999999995},"last":0.84799999999999998}]},{"osd":3,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55800000000000005,"15min":0.57199999999999995},"min":{"1min":0.36299999999999999,"5min":0.34799999999999998,"15min":0.34399999999999997},"max":{"1min":0.73999999999999999,"5min":1.357,"15min":1.357},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57099999999999995,"15min":0.58799999999999997},"min":{"1min":0.35599999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.63400000000000001}]},{"osd":4,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.45100000000000001,"15min":0.44800000000000001},"min":{"1min":0.28399999999999997,"5min":0.27100000000000002,"15min":0.25},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.83599999999999997},"last":0.443},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53200000000000003,"15min":0.52900000000000003},"min":{"1min":0.35399999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.58299999999999996}]},{"osd":5,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.49399999999999999,"15min":0.497},"min":{"1min":0.33800000000000002,"5min":0.29299999999999998,"15min":0.23799999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.86799999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54500000000000004,"15min":0.55400000000000005},"min":{"1min":0.39600000000000002,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.79900000000000004,"5min":0.81599999999999995,"15min":0.83499999999999996},"last":0.73799999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.28799999999999998},"max":{"1min":0.84499999999999997,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.55100000000000005,"15min":0.55900000000000005},"min":{"1min":0.312,"5min":0.30399999999999999,"15min":0.26900000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.88500000000000001},"last":0.82199999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575047,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32488,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339352,"statfs":{"total":96636764160,"available":96603496448,"internally_reserved":0,"allocated":1396736,"data_stored":688042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52200000000000002,"15min":0.53300000000000003},"min":{"1min":0.34000000000000002,"5min":0.33700000000000002,"15min":0.23000000000000001},"max":{"1min":0.64400000000000002,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.52200000000000002,"15min":0.53400000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.245},"max":{"1min":0.65400000000000003,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Apr 6 12:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.56399999999999995,"15min":0.58499999999999996},"min":{"1min":0.39600000000000002,"5min":0.32700000000000001,"15min":0.24399999999999999},"max":{"1min":0.68000000000000005,"5min":0.78700000000000003,"15min":0.84999999999999998},"last":0.502},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58199999999999996,"15min":0.59599999999999997},"min":{"1min":0.42799999999999999,"5min":0.30199999999999999,"15min":0.221},"max":{"1min":0.72499999999999998,"5min":0.81999999999999995,"15min":0.83099999999999996},"last":0.58899999999999997}]},{"osd":2,"last update":"Sat Apr 6 12:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52900000000000003,"15min":0.54700000000000004},"min":{"1min":0.374,"5min":0.30499999999999999,"15min":0.29899999999999999},"max":{"1min":0.61799999999999999,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.55100000000000005,"15min":0.56699999999999995},"min":{"1min":0.35399999999999998,"5min":0.32300000000000001,"15min":0.254},"max":{"1min":0.68999999999999995,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.63400000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55900000000000005,"15min":0.57999999999999996},"min":{"1min":0.435,"5min":0.317,"15min":0.23300000000000001},"max":{"1min":0.66400000000000003,"5min":0.77300000000000002,"15min":0.81200000000000006},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.42599999999999999,"5min":0.41099999999999998,"15min":0.374},"max":{"1min":0.77600000000000002,"5min":0.82799999999999996,"15min":0.872},"last":0.69399999999999995}]},{"osd":4,"last update":"Sat Apr 6 12:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.315,"5min":0.28599999999999998,"15min":0.27300000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52400000000000002,"15min":0.52500000000000002},"min":{"1min":0.29799999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.71299999999999997,"5min":0.73999999999999999,"15min":0.82199999999999995},"last":0.71199999999999997}]},{"osd":5,"last update":"Sat Apr 6 12:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51600000000000001,"15min":0.50800000000000001},"min":{"1min":0.31,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.83699999999999997},"last":0.68300000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.53800000000000003,"15min":0.53200000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56299999999999994,"15min":0.55800000000000005},"min":{"1min":0.41599999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.88900000000000001},"last":0.73799999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575046,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1380352,"data_stored":685944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.371,"15min":0.35999999999999999},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.42399999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.38400000000000001,"15min":0.39400000000000002},"min":{"1min":0.245,"5min":0.215,"15min":0.215},"max":{"1min":0.55600000000000005,"5min":0.628,"15min":0.628},"last":0.255}]},{"osd":2,"last update":"Sat Apr 6 12:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.39700000000000002,"15min":0.39400000000000002},"min":{"1min":0.26100000000000001,"5min":0.23899999999999999,"15min":0.19400000000000001},"max":{"1min":0.47499999999999998,"5min":0.624,"15min":0.67600000000000005},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.42399999999999999,"15min":0.42799999999999999},"min":{"1min":0.29999999999999999,"5min":0.255,"15min":0.249},"max":{"1min":0.61699999999999999,"5min":0.63600000000000001,"15min":0.66100000000000003},"last":0.29899999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.442,"15min":0.44500000000000001},"min":{"1min":0.29899999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.51800000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.318},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.41299999999999998,"15min":0.41699999999999998},"min":{"1min":0.29099999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.59399999999999997,"5min":0.69599999999999995,"15min":0.86299999999999999},"last":0.34000000000000002}]},{"osd":4,"last update":"Sat Apr 6 12:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.44500000000000001,"5min":0.40100000000000002,"15min":0.34799999999999998},"max":{"1min":0.75600000000000001,"5min":0.86799999999999999,"15min":0.91100000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.621,"15min":0.621},"min":{"1min":0.47699999999999998,"5min":0.441,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":0.91400000000000003,"15min":0.94899999999999995},"last":0.47999999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.45200000000000001,"5min":0.42999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76200000000000001,"5min":0.91300000000000003,"15min":0.93500000000000005},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.629,"15min":0.626},"min":{"1min":0.48299999999999998,"5min":0.43099999999999999,"15min":0.35099999999999998},"max":{"1min":0.76700000000000002,"5min":0.999,"15min":0.999},"last":0.71499999999999997}]},{"osd":6,"last update":"Sat Apr 6 12:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.74099999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.64200000000000002,"15min":0.64600000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.41799999999999998},"max":{"1min":0.78300000000000003,"5min":0.88700000000000001,"15min":0.97899999999999998},"last":0.65300000000000002}]},{"osd":7,"last update":"Sat Apr 6 12:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.64100000000000001,"15min":0.63600000000000001},"min":{"1min":0.48799999999999999,"5min":0.41899999999999998,"15min":0.38800000000000001},"max":{"1min":0.79600000000000004,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.63500000000000001,"15min":0.63600000000000001},"min":{"1min":0.40999999999999998,"5min":0.40200000000000002,"15min":0.39300000000000002},"max":{"1min":0.75700000000000001,"5min":0.95599999999999996,"15min":0.96799999999999997},"last":0.53500000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575046,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33072,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338768,"statfs":{"total":96636764160,"available":96602898432,"internally_reserved":0,"allocated":1994752,"data_stored":1278512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 12:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38200000000000001,"15min":0.38300000000000001},"min":{"1min":0.23599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.374,"5min":0.39700000000000002,"15min":0.40300000000000002},"min":{"1min":0.25900000000000001,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":0.625,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.40999999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.39900000000000002,"15min":0.40699999999999997},"min":{"1min":0.25700000000000001,"5min":0.23599999999999999,"15min":0.19700000000000001},"max":{"1min":0.59799999999999998,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.26800000000000002,"5min":0.249,"15min":0.24299999999999999},"max":{"1min":0.89900000000000002,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.47099999999999997}]},{"osd":3,"last update":"Sat Apr 6 12:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.40500000000000003,"15min":0.41499999999999998},"min":{"1min":0.28799999999999998,"5min":0.252,"15min":0.222},"max":{"1min":0.56299999999999994,"5min":1.25,"15min":1.25},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.44600000000000001,"15min":0.46000000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.249},"max":{"1min":0.68600000000000005,"5min":0.74099999999999999,"15min":0.748},"last":0.35999999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.60299999999999998,"15min":0.61499999999999999},"min":{"1min":0.38400000000000001,"5min":0.371,"15min":0.35699999999999998},"max":{"1min":1.1100000000000001,"5min":1.514,"15min":1.514},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61599999999999999,"15min":0.626},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.311},"max":{"1min":0.93999999999999995,"5min":1.202,"15min":1.202},"last":0.84699999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.61299999999999999,"15min":0.621},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.40899999999999997,"5min":0.38800000000000001,"15min":0.315},"max":{"1min":0.97399999999999998,"5min":1.006,"15min":1.006},"last":0.82499999999999996}]},{"osd":6,"last update":"Sat Apr 6 12:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.63500000000000001,"15min":0.64700000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.64500000000000002,"15min":0.65700000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.38100000000000001},"max":{"1min":1.052,"5min":1.613,"15min":1.613},"last":0.79500000000000004}]},{"osd":7,"last update":"Sat Apr 6 12:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.64800000000000002,"15min":0.66000000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0820000000000001,"5min":1.54,"15min":1.54},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.64900000000000002,"15min":0.65400000000000003},"min":{"1min":0.42099999999999999,"5min":0.38400000000000001,"15min":0.36699999999999999},"max":{"1min":1.014,"5min":1.498,"15min":1.498},"last":0.54600000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575048,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54604,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317236,"statfs":{"total":96636764160,"available":96580849664,"internally_reserved":0,"allocated":1376256,"data_stored":685874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.374,"15min":0.38100000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51700000000000002,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.39900000000000002,"15min":0.40899999999999997},"min":{"1min":0.246,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.497,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.51000000000000001}]},{"osd":1,"last update":"Sat Apr 6 12:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.40300000000000002,"15min":0.40200000000000002},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.51700000000000002,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.40899999999999997,"15min":0.42899999999999999},"min":{"1min":0.29799999999999999,"5min":0.27100000000000002,"15min":0.23599999999999999},"max":{"1min":0.59799999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.45600000000000002}]},{"osd":3,"last update":"Sat Apr 6 12:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.44400000000000001,"15min":0.44800000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29699999999999999},"max":{"1min":0.55800000000000005,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.495},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.44,"15min":0.436},"min":{"1min":0.33000000000000002,"5min":0.29499999999999998,"15min":0.28100000000000003},"max":{"1min":0.58399999999999996,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.433}]},{"osd":4,"last update":"Sat Apr 6 12:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58399999999999996,"15min":0.61199999999999999},"min":{"1min":0.40200000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.67800000000000005,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.59699999999999998,"15min":0.623},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.753,"5min":0.91200000000000003,"15min":0.92600000000000005},"last":0.80400000000000005}]},{"osd":5,"last update":"Sat Apr 6 12:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.61699999999999999,"15min":0.63300000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.32900000000000001},"max":{"1min":0.73199999999999998,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.60299999999999998,"15min":0.628},"min":{"1min":0.35399999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.83099999999999996,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.71899999999999997}]},{"osd":6,"last update":"Sat Apr 6 12:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60599999999999998,"15min":0.63200000000000001},"min":{"1min":0.46000000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.69399999999999995,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.63700000000000001,"15min":0.65100000000000002},"min":{"1min":0.443,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.77000000000000002,"5min":1.728,"15min":1.728},"last":0.90200000000000002}]},{"osd":7,"last update":"Sat Apr 6 12:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.621,"15min":0.64500000000000002},"min":{"1min":0.42399999999999999,"5min":0.40699999999999997,"15min":0.35599999999999998},"max":{"1min":0.80300000000000005,"5min":0.95699999999999996,"15min":0.95799999999999996},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.64500000000000002,"15min":0.66100000000000003},"min":{"1min":0.41499999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83199999999999996,"5min":1.758,"15min":1.758},"last":0.78000000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575047,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1368064,"data_stored":680273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.35699999999999998,"15min":0.36499999999999999},"min":{"1min":0.248,"5min":0.21099999999999999,"15min":0.191},"max":{"1min":0.44500000000000001,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.375},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.36899999999999999,"15min":0.38200000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.501,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.38800000000000001}]},{"osd":1,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.377,"15min":0.38200000000000001},"min":{"1min":0.23699999999999999,"5min":0.22700000000000001,"15min":0.215},"max":{"1min":0.55800000000000005,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.314},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.40500000000000003,"15min":0.41499999999999998},"min":{"1min":0.28499999999999998,"5min":0.26100000000000001,"15min":0.25700000000000001},"max":{"1min":0.57299999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.33000000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.44400000000000001,"15min":0.45900000000000002},"min":{"1min":0.309,"5min":0.27200000000000002,"15min":0.25700000000000001},"max":{"1min":0.78700000000000003,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.46800000000000003,"15min":0.47599999999999998},"min":{"1min":0.32800000000000001,"5min":0.30199999999999999,"15min":0.29899999999999999},"max":{"1min":0.68799999999999994,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.40600000000000003}]},{"osd":4,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.59499999999999997,"15min":0.60699999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.81299999999999994,"5min":0.88500000000000001,"15min":0.95999999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59199999999999997,"15min":0.621},"min":{"1min":0.39600000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.77600000000000002,"5min":0.90300000000000002,"15min":0.93500000000000005},"last":0.63700000000000001}]},{"osd":5,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.63300000000000001,"15min":0.65700000000000003},"min":{"1min":0.39500000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.82199999999999995,"5min":0.92200000000000004,"15min":0.96699999999999997},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.64600000000000002,"15min":0.66000000000000003},"min":{"1min":0.42799999999999999,"5min":0.38200000000000001,"15min":0.36899999999999999},"max":{"1min":0.83799999999999997,"5min":0.94499999999999995,"15min":0.95599999999999996},"last":0.67100000000000004}]},{"osd":6,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60199999999999998,"15min":0.621},"min":{"1min":0.44,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.72499999999999998,"5min":1.026,"15min":1.026},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.073,"5min":1.716,"15min":1.716},"last":0.53100000000000003}]},{"osd":7,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.628,"15min":0.64400000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.33300000000000002},"max":{"1min":0.83599999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.64000000000000001,"15min":0.66200000000000003},"min":{"1min":0.40100000000000002,"5min":0.39500000000000002,"15min":0.373},"max":{"1min":0.82599999999999996,"5min":1.002,"15min":1.002},"last":0.59699999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575047,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1388544,"data_stored":687053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54800000000000004,"15min":0.56100000000000005},"min":{"1min":0.33200000000000002,"5min":0.29399999999999998,"15min":0.23899999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.53100000000000003,"15min":0.55900000000000005},"min":{"1min":0.36799999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.753,"5min":0.80000000000000004,"15min":0.82199999999999995},"last":0.59199999999999997}]},{"osd":1,"last update":"Sat Apr 6 12:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.57399999999999995,"15min":0.60499999999999998},"min":{"1min":0.41199999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67000000000000004,"5min":0.76200000000000001,"15min":0.96499999999999997},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.57299999999999995,"15min":0.59199999999999997},"min":{"1min":0.33400000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.754,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.68200000000000005}]},{"osd":2,"last update":"Sat Apr 6 12:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54800000000000004,"15min":0.56599999999999995},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.753,"5min":0.76100000000000001,"15min":0.78400000000000003},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.56499999999999995,"15min":0.58199999999999996},"min":{"1min":0.34999999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.80900000000000005},"last":0.625}]},{"osd":3,"last update":"Sat Apr 6 12:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.54600000000000004,"15min":0.55800000000000005},"min":{"1min":0.39600000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57899999999999996,"15min":0.59699999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.36099999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.80900000000000005},"last":0.67000000000000004}]},{"osd":5,"last update":"Sat Apr 6 12:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.46200000000000002,"15min":0.47799999999999998},"min":{"1min":0.315,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66300000000000003,"5min":0.76300000000000001,"15min":1.196},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.48599999999999999,"15min":0.495},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.64500000000000002,"5min":0.73099999999999998,"15min":0.77000000000000002},"last":0.47499999999999998}]},{"osd":6,"last update":"Sat Apr 6 12:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.497},"min":{"1min":0.307,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.435},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51600000000000001,"15min":0.53200000000000003},"min":{"1min":0.34100000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70499999999999996,"5min":0.71799999999999997,"15min":0.872},"last":0.44800000000000001}]},{"osd":7,"last update":"Sat Apr 6 12:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51500000000000001,"15min":0.53000000000000003},"min":{"1min":0.34100000000000003,"5min":0.311,"15min":0.311},"max":{"1min":0.71599999999999997,"5min":0.79300000000000004,"15min":0.82299999999999995},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54700000000000004,"15min":0.56399999999999995},"min":{"1min":0.36699999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.84799999999999998},"last":0.49299999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575047,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50508,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321332,"statfs":{"total":96636764160,"available":96585043968,"internally_reserved":0,"allocated":1376256,"data_stored":685874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.54400000000000004,"15min":0.56699999999999995},"min":{"1min":0.35199999999999998,"5min":0.32700000000000001,"15min":0.24199999999999999},"max":{"1min":0.68700000000000006,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.53100000000000003,"15min":0.54700000000000004},"min":{"1min":0.39200000000000002,"5min":0.29499999999999998,"15min":0.22500000000000001},"max":{"1min":0.64900000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.36099999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53500000000000003,"15min":0.55300000000000005},"min":{"1min":0.38400000000000001,"5min":0.30099999999999999,"15min":0.26000000000000001},"max":{"1min":0.58399999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.53900000000000003,"15min":0.55200000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.30599999999999999},"max":{"1min":0.70699999999999996,"5min":0.73599999999999999,"15min":0.76200000000000001},"last":0.38100000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.54600000000000004,"15min":0.56000000000000005},"min":{"1min":0.34000000000000002,"5min":0.32500000000000001,"15min":0.21099999999999999},"max":{"1min":0.67200000000000004,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.55200000000000005,"15min":0.56399999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.27400000000000002},"max":{"1min":0.67400000000000004,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.51900000000000002}]},{"osd":3,"last update":"Sat Apr 6 12:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55600000000000005,"15min":0.56699999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.31},"max":{"1min":0.623,"5min":0.874,"15min":0.874},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.57699999999999996,"15min":0.59399999999999997},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.32300000000000001},"max":{"1min":0.71699999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53500000000000003}]},{"osd":4,"last update":"Sat Apr 6 12:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.46899999999999997,"15min":0.46700000000000003},"min":{"1min":0.371,"5min":0.315,"15min":0.28299999999999997},"max":{"1min":0.61399999999999999,"5min":0.755,"15min":0.81200000000000006},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53900000000000003,"15min":0.52700000000000002},"min":{"1min":0.437,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70999999999999996,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.45800000000000002}]},{"osd":6,"last update":"Sat Apr 6 12:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.314,"5min":0.313,"15min":0.28000000000000003},"max":{"1min":0.73499999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.48899999999999999,"15min":0.48699999999999999},"min":{"1min":0.34000000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.68600000000000005,"5min":0.73199999999999998,"15min":0.80500000000000005},"last":0.505}]},{"osd":7,"last update":"Sat Apr 6 12:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.33200000000000002,"5min":0.29799999999999999,"15min":0.252},"max":{"1min":0.69699999999999995,"5min":0.78600000000000003,"15min":0.79100000000000004},"last":0.433},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.60099999999999998,"15min":0.59699999999999998},"min":{"1min":0.48199999999999998,"5min":0.41399999999999998,"15min":0.35199999999999998},"max":{"1min":0.76000000000000001,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.58499999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:28:44.823 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:28:44.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.985+0000 7f990883d700 1 Processor -- start 2024-04-06T12:28:44.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.985+0000 7f990883d700 1 -- start start 2024-04-06T12:28:44.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.986+0000 7f990883d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f990406ae40 0x7f99040b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:44.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.986+0000 7f990883d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f990415ddf0 0x7f990415e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:44.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.986+0000 7f990883d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99040b3990 con 0x7f990415d7b0 2024-04-06T12:28:44.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.986+0000 7f990883d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99040b3060 con 0x7f990415ddf0 2024-04-06T12:28:44.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.986+0000 7f990883d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99040b3340 con 0x7f990406ae40 2024-04-06T12:28:44.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.986+0000 7f9902d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f990415ddf0 0x7f990415e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:44.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.986+0000 7f9902d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f990415ddf0 0x7f990415e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53440/0 (socket says 172.21.15.189:53440) 2024-04-06T12:28:44.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.986+0000 7f9902d9d700 1 -- 172.21.15.189:0/3535380116 learned_addr learned my addr 172.21.15.189:0/3535380116 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:44.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.987+0000 7f9902d9d700 1 -- 172.21.15.189:0/3535380116 >> v2:172.21.15.192:3300/0 conn(0x7f990406ae40 msgr2=0x7f99040b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:44.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.987+0000 7f9902d9d700 1 --2- 172.21.15.189:0/3535380116 >> v2:172.21.15.192:3300/0 conn(0x7f990406ae40 0x7f99040b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.987+0000 7f9902d9d700 1 -- 172.21.15.189:0/3535380116 >> v1:172.21.15.192:6789/0 conn(0x7f990415d7b0 legacy=0x7f990406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:44.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.987+0000 7f9902d9d700 1 -- 172.21.15.189:0/3535380116 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99040b3620 con 0x7f990415ddf0 2024-04-06T12:28:44.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.987+0000 7f9902d9d700 1 --2- 172.21.15.189:0/3535380116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f990415ddf0 0x7f990415e240 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f99040afc00 tx=0x7f98fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd3d5c2b99eafe9e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:44.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.987+0000 7f990159a700 1 -- 172.21.15.189:0/3535380116 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc00e070 con 0x7f990415ddf0 2024-04-06T12:28:44.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.987+0000 7f990159a700 1 -- 172.21.15.189:0/3535380116 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98fc003a20 con 0x7f990415ddf0 2024-04-06T12:28:44.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.987+0000 7f990159a700 1 -- 172.21.15.189:0/3535380116 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc003dd0 con 0x7f990415ddf0 2024-04-06T12:28:44.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.988+0000 7f990883d700 1 -- 172.21.15.189:0/3535380116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f990415ddf0 msgr2=0x7f990415e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:44.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.988+0000 7f990883d700 1 --2- 172.21.15.189:0/3535380116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f990415ddf0 0x7f990415e240 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f99040afc00 tx=0x7f98fc00b330 comp rx=0 tx=0).stop 2024-04-06T12:28:44.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.988+0000 7f990883d700 1 -- 172.21.15.189:0/3535380116 shutdown_connections 2024-04-06T12:28:44.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.988+0000 7f990883d700 1 --2- 172.21.15.189:0/3535380116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f990415ddf0 0x7f990415e240 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.988+0000 7f990883d700 1 --2- 172.21.15.189:0/3535380116 >> v2:172.21.15.192:3300/0 conn(0x7f990406ae40 0x7f99040b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.988+0000 7f990883d700 1 -- 172.21.15.189:0/3535380116 >> 172.21.15.189:0/3535380116 conn(0x7f99040d00b0 msgr2=0x7f99040d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:44.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.988+0000 7f990883d700 1 -- 172.21.15.189:0/3535380116 shutdown_connections 2024-04-06T12:28:44.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.988+0000 7f990883d700 1 -- 172.21.15.189:0/3535380116 wait complete. 2024-04-06T12:28:44.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.989+0000 7f990883d700 1 Processor -- start 2024-04-06T12:28:44.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.989+0000 7f990883d700 1 -- start start 2024-04-06T12:28:44.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.990+0000 7f990883d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f990406ae40 0x7f99041517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:44.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.990+0000 7f990883d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f990415d7b0 0x7f9904160680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:44.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.990+0000 7f990883d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f990415ddf0 0x7f990415b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:44.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.990+0000 7f990883d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9904155720 con 0x7f990406ae40 2024-04-06T12:28:44.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.990+0000 7f990883d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9904154ba0 con 0x7f990415d7b0 2024-04-06T12:28:44.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.990+0000 7f990883d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9904154e80 con 0x7f990415ddf0 2024-04-06T12:28:44.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.990+0000 7f990259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f990406ae40 0x7f99041517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:44.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.990+0000 7f990259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f990406ae40 0x7f99041517a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53454/0 (socket says 172.21.15.189:53454) 2024-04-06T12:28:44.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.990+0000 7f990259c700 1 -- 172.21.15.189:0/2931912143 learned_addr learned my addr 172.21.15.189:0/2931912143 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:44.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.991+0000 7f990259c700 1 -- 172.21.15.189:0/2931912143 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f990415ddf0 msgr2=0x7f990415b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:44.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.991+0000 7f9902d9d700 1 --2- 172.21.15.189:0/2931912143 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f990415ddf0 0x7f990415b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:44.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.991+0000 7f9901d9b700 1 --2- 172.21.15.189:0/2931912143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f990415d7b0 0x7f9904160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:44.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.991+0000 7f990259c700 1 --2- 172.21.15.189:0/2931912143 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f990415ddf0 0x7f990415b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.991+0000 7f990259c700 1 -- 172.21.15.189:0/2931912143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f990415d7b0 msgr2=0x7f9904160680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:44.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.991+0000 7f990259c700 1 --2- 172.21.15.189:0/2931912143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f990415d7b0 0x7f9904160680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:44.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.991+0000 7f990259c700 1 -- 172.21.15.189:0/2931912143 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc009b30 con 0x7f990406ae40 2024-04-06T12:28:44.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.991+0000 7f990259c700 1 --2- 172.21.15.189:0/2931912143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f990406ae40 0x7f99041517a0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f98f400c4a0 tx=0x7f98f400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:44.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.991+0000 7f98faffd700 1 -- 172.21.15.189:0/2931912143 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f40054e0 con 0x7f990406ae40 2024-04-06T12:28:44.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.991+0000 7f98faffd700 1 -- 172.21.15.189:0/2931912143 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f4003680 con 0x7f990406ae40 2024-04-06T12:28:44.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.991+0000 7f990883d700 1 -- 172.21.15.189:0/2931912143 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9904155160 con 0x7f990406ae40 2024-04-06T12:28:44.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.991+0000 7f98faffd700 1 -- 172.21.15.189:0/2931912143 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f4003a30 con 0x7f990406ae40 2024-04-06T12:28:44.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.991+0000 7f990883d700 1 -- 172.21.15.189:0/2931912143 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9904160bc0 con 0x7f990406ae40 2024-04-06T12:28:44.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.992+0000 7f990883d700 1 -- 172.21.15.189:0/2931912143 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9904151e10 con 0x7f990406ae40 2024-04-06T12:28:44.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.993+0000 7f98faffd700 1 -- 172.21.15.189:0/2931912143 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f98f4003b90 con 0x7f990406ae40 2024-04-06T12:28:44.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.993+0000 7f98faffd700 1 --2- 172.21.15.189:0/2931912143 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f98e406adb0 0x7f98e406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:44.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.993+0000 7f98faffd700 1 -- 172.21.15.189:0/2931912143 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f98f4088b10 con 0x7f990406ae40 2024-04-06T12:28:44.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.993+0000 7f9901d9b700 1 --2- 172.21.15.189:0/2931912143 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f98e406adb0 0x7f98e406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:44.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.993+0000 7f9901d9b700 1 --2- 172.21.15.189:0/2931912143 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f98e406adb0 0x7f98e406d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f98ec006fd0 tx=0x7f98ec008040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:44.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:44.998+0000 7f98faffd700 1 -- 172.21.15.189:0/2931912143 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9904151e10 con 0x7f990406ae40 2024-04-06T12:28:45.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.188+0000 7f990883d700 1 -- 172.21.15.189:0/2931912143 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9904154ff0 con 0x7f98e406adb0 2024-04-06T12:28:45.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.196+0000 7f98faffd700 1 -- 172.21.15.189:0/2931912143 <== mgr.7338 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289943 (secure 0 0 0) 0x7f9904154ff0 con 0x7f98e406adb0 2024-04-06T12:28:45.197 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.199+0000 7f990883d700 1 -- 172.21.15.189:0/2931912143 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f98e406adb0 msgr2=0x7f98e406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.199+0000 7f990883d700 1 --2- 172.21.15.189:0/2931912143 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f98e406adb0 0x7f98e406d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f98ec006fd0 tx=0x7f98ec008040 comp rx=0 tx=0).stop 2024-04-06T12:28:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.199+0000 7f990883d700 1 -- 172.21.15.189:0/2931912143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f990406ae40 msgr2=0x7f99041517a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.199+0000 7f990883d700 1 --2- 172.21.15.189:0/2931912143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f990406ae40 0x7f99041517a0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f98f400c4a0 tx=0x7f98f400c860 comp rx=0 tx=0).stop 2024-04-06T12:28:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.199+0000 7f990883d700 1 -- 172.21.15.189:0/2931912143 shutdown_connections 2024-04-06T12:28:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.199+0000 7f990883d700 1 --2- 172.21.15.189:0/2931912143 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f990415ddf0 0x7f990415b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.199+0000 7f990883d700 1 --2- 172.21.15.189:0/2931912143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f990406ae40 0x7f99041517a0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.199+0000 7f990883d700 1 --2- 172.21.15.189:0/2931912143 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f98e406adb0 0x7f98e406d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.199+0000 7f990883d700 1 --2- 172.21.15.189:0/2931912143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f990415d7b0 0x7f9904160680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.199+0000 7f990883d700 1 -- 172.21.15.189:0/2931912143 >> 172.21.15.189:0/2931912143 conn(0x7f99040d00b0 msgr2=0x7f990406f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.199+0000 7f990883d700 1 -- 172.21.15.189:0/2931912143 shutdown_connections 2024-04-06T12:28:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.199+0000 7f990883d700 1 -- 172.21.15.189:0/2931912143 wait complete. 2024-04-06T12:28:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:28:45.208 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-06T12:28:44.988742+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":97,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":327372,"kb_used_data":11976,"kb_used_omap":12,"kb_used_meta":315251,"kb_avail":754647348,"statfs":{"total":773094113280,"available":772758884352,"internally_reserved":0,"allocated":12263424,"data_stored":6668889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":322817592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.742980"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406806+0000","last_change":"2024-04-06T12:28:40.681154+0000","last_active":"2024-04-06T12:28:42.406806+0000","last_peered":"2024-04-06T12:28:42.406806+0000","last_clean":"2024-04-06T12:28:42.406806+0000","last_became_active":"2024-04-06T12:28:40.680729+0000","last_became_peered":"2024-04-06T12:28:40.680729+0000","last_unstale":"2024-04-06T12:28:42.406806+0000","last_undegraded":"2024-04-06T12:28:42.406806+0000","last_fullsized":"2024-04-06T12:28:42.406806+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:44:33.737630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405646+0000","last_change":"2024-04-06T12:28:40.675174+0000","last_active":"2024-04-06T12:28:42.405646+0000","last_peered":"2024-04-06T12:28:42.405646+0000","last_clean":"2024-04-06T12:28:42.405646+0000","last_became_active":"2024-04-06T12:28:40.674898+0000","last_became_peered":"2024-04-06T12:28:40.674898+0000","last_unstale":"2024-04-06T12:28:42.405646+0000","last_undegraded":"2024-04-06T12:28:42.405646+0000","last_fullsized":"2024-04-06T12:28:42.405646+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:16:57.826003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406458+0000","last_change":"2024-04-06T12:28:40.688392+0000","last_active":"2024-04-06T12:28:42.406458+0000","last_peered":"2024-04-06T12:28:42.406458+0000","last_clean":"2024-04-06T12:28:42.406458+0000","last_became_active":"2024-04-06T12:28:40.688139+0000","last_became_peered":"2024-04-06T12:28:40.688139+0000","last_unstale":"2024-04-06T12:28:42.406458+0000","last_undegraded":"2024-04-06T12:28:42.406458+0000","last_fullsized":"2024-04-06T12:28:42.406458+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:48:24.163860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406244+0000","last_change":"2024-04-06T12:28:40.689417+0000","last_active":"2024-04-06T12:28:42.406244+0000","last_peered":"2024-04-06T12:28:42.406244+0000","last_clean":"2024-04-06T12:28:42.406244+0000","last_became_active":"2024-04-06T12:28:40.689130+0000","last_became_peered":"2024-04-06T12:28:40.689130+0000","last_unstale":"2024-04-06T12:28:42.406244+0000","last_undegraded":"2024-04-06T12:28:42.406244+0000","last_fullsized":"2024-04-06T12:28:42.406244+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:48:51.138321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669983+0000","last_change":"2024-04-06T12:28:40.675409+0000","last_active":"2024-04-06T12:28:41.669983+0000","last_peered":"2024-04-06T12:28:41.669983+0000","last_clean":"2024-04-06T12:28:41.669983+0000","last_became_active":"2024-04-06T12:28:40.674857+0000","last_became_peered":"2024-04-06T12:28:40.674857+0000","last_unstale":"2024-04-06T12:28:41.669983+0000","last_undegraded":"2024-04-06T12:28:41.669983+0000","last_fullsized":"2024-04-06T12:28:41.669983+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:48:51.138321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406885+0000","last_change":"2024-04-06T12:28:40.692157+0000","last_active":"2024-04-06T12:28:42.406885+0000","last_peered":"2024-04-06T12:28:42.406885+0000","last_clean":"2024-04-06T12:28:42.406885+0000","last_became_active":"2024-04-06T12:28:40.691757+0000","last_became_peered":"2024-04-06T12:28:40.691757+0000","last_unstale":"2024-04-06T12:28:42.406885+0000","last_undegraded":"2024-04-06T12:28:42.406885+0000","last_fullsized":"2024-04-06T12:28:42.406885+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:55:40.250566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.668955+0000","last_change":"2024-04-06T12:28:40.674320+0000","last_active":"2024-04-06T12:28:41.668955+0000","last_peered":"2024-04-06T12:28:41.668955+0000","last_clean":"2024-04-06T12:28:41.668955+0000","last_became_active":"2024-04-06T12:28:40.674049+0000","last_became_peered":"2024-04-06T12:28:40.674049+0000","last_unstale":"2024-04-06T12:28:41.668955+0000","last_undegraded":"2024-04-06T12:28:41.668955+0000","last_fullsized":"2024-04-06T12:28:41.668955+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:32:04.534993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405629+0000","last_change":"2024-04-06T12:28:40.687356+0000","last_active":"2024-04-06T12:28:42.405629+0000","last_peered":"2024-04-06T12:28:42.405629+0000","last_clean":"2024-04-06T12:28:42.405629+0000","last_became_active":"2024-04-06T12:28:40.687106+0000","last_became_peered":"2024-04-06T12:28:40.687106+0000","last_unstale":"2024-04-06T12:28:42.405629+0000","last_undegraded":"2024-04-06T12:28:42.405629+0000","last_fullsized":"2024-04-06T12:28:42.405629+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:04:33.119466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670288+0000","last_change":"2024-04-06T12:28:40.683601+0000","last_active":"2024-04-06T12:28:41.670288+0000","last_peered":"2024-04-06T12:28:41.670288+0000","last_clean":"2024-04-06T12:28:41.670288+0000","last_became_active":"2024-04-06T12:28:40.683168+0000","last_became_peered":"2024-04-06T12:28:40.683168+0000","last_unstale":"2024-04-06T12:28:41.670288+0000","last_undegraded":"2024-04-06T12:28:41.670288+0000","last_fullsized":"2024-04-06T12:28:41.670288+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:20:10.545116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.056358+0000","last_change":"2024-04-06T12:28:40.691330+0000","last_active":"2024-04-06T12:28:42.056358+0000","last_peered":"2024-04-06T12:28:42.056358+0000","last_clean":"2024-04-06T12:28:42.056358+0000","last_became_active":"2024-04-06T12:28:40.691078+0000","last_became_peered":"2024-04-06T12:28:40.691078+0000","last_unstale":"2024-04-06T12:28:42.056358+0000","last_undegraded":"2024-04-06T12:28:42.056358+0000","last_fullsized":"2024-04-06T12:28:42.056358+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:54:32.630776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669445+0000","last_change":"2024-04-06T12:28:40.683258+0000","last_active":"2024-04-06T12:28:41.669445+0000","last_peered":"2024-04-06T12:28:41.669445+0000","last_clean":"2024-04-06T12:28:41.669445+0000","last_became_active":"2024-04-06T12:28:40.682878+0000","last_became_peered":"2024-04-06T12:28:40.682878+0000","last_unstale":"2024-04-06T12:28:41.669445+0000","last_undegraded":"2024-04-06T12:28:41.669445+0000","last_fullsized":"2024-04-06T12:28:41.669445+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:04:33.119466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.056283+0000","last_change":"2024-04-06T12:28:40.678892+0000","last_active":"2024-04-06T12:28:42.056283+0000","last_peered":"2024-04-06T12:28:42.056283+0000","last_clean":"2024-04-06T12:28:42.056283+0000","last_became_active":"2024-04-06T12:28:40.678639+0000","last_became_peered":"2024-04-06T12:28:40.678639+0000","last_unstale":"2024-04-06T12:28:42.056283+0000","last_undegraded":"2024-04-06T12:28:42.056283+0000","last_fullsized":"2024-04-06T12:28:42.056283+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:42:04.654510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406347+0000","last_change":"2024-04-06T12:28:40.689835+0000","last_active":"2024-04-06T12:28:42.406347+0000","last_peered":"2024-04-06T12:28:42.406347+0000","last_clean":"2024-04-06T12:28:42.406347+0000","last_became_active":"2024-04-06T12:28:40.689623+0000","last_became_peered":"2024-04-06T12:28:40.689623+0000","last_unstale":"2024-04-06T12:28:42.406347+0000","last_undegraded":"2024-04-06T12:28:42.406347+0000","last_fullsized":"2024-04-06T12:28:42.406347+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:08:10.898942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669703+0000","last_change":"2024-04-06T12:28:40.690544+0000","last_active":"2024-04-06T12:28:41.669703+0000","last_peered":"2024-04-06T12:28:41.669703+0000","last_clean":"2024-04-06T12:28:41.669703+0000","last_became_active":"2024-04-06T12:28:40.690338+0000","last_became_peered":"2024-04-06T12:28:40.690338+0000","last_unstale":"2024-04-06T12:28:41.669703+0000","last_undegraded":"2024-04-06T12:28:41.669703+0000","last_fullsized":"2024-04-06T12:28:41.669703+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:04:09.377839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405817+0000","last_change":"2024-04-06T12:28:40.693135+0000","last_active":"2024-04-06T12:28:42.405817+0000","last_peered":"2024-04-06T12:28:42.405817+0000","last_clean":"2024-04-06T12:28:42.405817+0000","last_became_active":"2024-04-06T12:28:40.692913+0000","last_became_peered":"2024-04-06T12:28:40.692913+0000","last_unstale":"2024-04-06T12:28:42.405817+0000","last_undegraded":"2024-04-06T12:28:42.405817+0000","last_fullsized":"2024-04-06T12:28:42.405817+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:20:10.545116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405858+0000","last_change":"2024-04-06T12:28:40.683406+0000","last_active":"2024-04-06T12:28:42.405858+0000","last_peered":"2024-04-06T12:28:42.405858+0000","last_clean":"2024-04-06T12:28:42.405858+0000","last_became_active":"2024-04-06T12:28:40.682518+0000","last_became_peered":"2024-04-06T12:28:40.682518+0000","last_unstale":"2024-04-06T12:28:42.405858+0000","last_undegraded":"2024-04-06T12:28:42.405858+0000","last_fullsized":"2024-04-06T12:28:42.405858+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:05:53.803843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406833+0000","last_change":"2024-04-06T12:28:40.680009+0000","last_active":"2024-04-06T12:28:42.406833+0000","last_peered":"2024-04-06T12:28:42.406833+0000","last_clean":"2024-04-06T12:28:42.406833+0000","last_became_active":"2024-04-06T12:28:40.679780+0000","last_became_peered":"2024-04-06T12:28:40.679780+0000","last_unstale":"2024-04-06T12:28:42.406833+0000","last_undegraded":"2024-04-06T12:28:42.406833+0000","last_fullsized":"2024-04-06T12:28:42.406833+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:01:36.708557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.407216+0000","last_change":"2024-04-06T12:28:40.673426+0000","last_active":"2024-04-06T12:28:42.407216+0000","last_peered":"2024-04-06T12:28:42.407216+0000","last_clean":"2024-04-06T12:28:42.407216+0000","last_became_active":"2024-04-06T12:28:40.673180+0000","last_became_peered":"2024-04-06T12:28:40.673180+0000","last_unstale":"2024-04-06T12:28:42.407216+0000","last_undegraded":"2024-04-06T12:28:42.407216+0000","last_fullsized":"2024-04-06T12:28:42.407216+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:45:09.039210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669676+0000","last_change":"2024-04-06T12:28:40.688439+0000","last_active":"2024-04-06T12:28:41.669676+0000","last_peered":"2024-04-06T12:28:41.669676+0000","last_clean":"2024-04-06T12:28:41.669676+0000","last_became_active":"2024-04-06T12:28:40.688173+0000","last_became_peered":"2024-04-06T12:28:40.688173+0000","last_unstale":"2024-04-06T12:28:41.669676+0000","last_undegraded":"2024-04-06T12:28:41.669676+0000","last_fullsized":"2024-04-06T12:28:41.669676+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:53:23.333343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405961+0000","last_change":"2024-04-06T12:28:40.693077+0000","last_active":"2024-04-06T12:28:42.405961+0000","last_peered":"2024-04-06T12:28:42.405961+0000","last_clean":"2024-04-06T12:28:42.405961+0000","last_became_active":"2024-04-06T12:28:40.692802+0000","last_became_peered":"2024-04-06T12:28:40.692802+0000","last_unstale":"2024-04-06T12:28:42.405961+0000","last_undegraded":"2024-04-06T12:28:42.405961+0000","last_fullsized":"2024-04-06T12:28:42.405961+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:18:29.810477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670070+0000","last_change":"2024-04-06T12:28:40.673389+0000","last_active":"2024-04-06T12:28:41.670070+0000","last_peered":"2024-04-06T12:28:41.670070+0000","last_clean":"2024-04-06T12:28:41.670070+0000","last_became_active":"2024-04-06T12:28:40.673115+0000","last_became_peered":"2024-04-06T12:28:40.673115+0000","last_unstale":"2024-04-06T12:28:41.670070+0000","last_undegraded":"2024-04-06T12:28:41.670070+0000","last_fullsized":"2024-04-06T12:28:41.670070+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:44:56.202282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669861+0000","last_change":"2024-04-06T12:28:40.690264+0000","last_active":"2024-04-06T12:28:41.669861+0000","last_peered":"2024-04-06T12:28:41.669861+0000","last_clean":"2024-04-06T12:28:41.669861+0000","last_became_active":"2024-04-06T12:28:40.689951+0000","last_became_peered":"2024-04-06T12:28:40.689951+0000","last_unstale":"2024-04-06T12:28:41.669861+0000","last_undegraded":"2024-04-06T12:28:41.669861+0000","last_fullsized":"2024-04-06T12:28:41.669861+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:06:31.270541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669421+0000","last_change":"2024-04-06T12:28:40.688680+0000","last_active":"2024-04-06T12:28:41.669421+0000","last_peered":"2024-04-06T12:28:41.669421+0000","last_clean":"2024-04-06T12:28:41.669421+0000","last_became_active":"2024-04-06T12:28:40.688378+0000","last_became_peered":"2024-04-06T12:28:40.688378+0000","last_unstale":"2024-04-06T12:28:41.669421+0000","last_undegraded":"2024-04-06T12:28:41.669421+0000","last_fullsized":"2024-04-06T12:28:41.669421+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:42:19.971508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406205+0000","last_change":"2024-04-06T12:28:40.684948+0000","last_active":"2024-04-06T12:28:42.406205+0000","last_peered":"2024-04-06T12:28:42.406205+0000","last_clean":"2024-04-06T12:28:42.406205+0000","last_became_active":"2024-04-06T12:28:40.684671+0000","last_became_peered":"2024-04-06T12:28:40.684671+0000","last_unstale":"2024-04-06T12:28:42.406205+0000","last_undegraded":"2024-04-06T12:28:42.406205+0000","last_fullsized":"2024-04-06T12:28:42.406205+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:36:04.039871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405757+0000","last_change":"2024-04-06T12:28:40.679074+0000","last_active":"2024-04-06T12:28:42.405757+0000","last_peered":"2024-04-06T12:28:42.405757+0000","last_clean":"2024-04-06T12:28:42.405757+0000","last_became_active":"2024-04-06T12:28:40.678816+0000","last_became_peered":"2024-04-06T12:28:40.678816+0000","last_unstale":"2024-04-06T12:28:42.405757+0000","last_undegraded":"2024-04-06T12:28:42.405757+0000","last_fullsized":"2024-04-06T12:28:42.405757+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:33:35.890745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670306+0000","last_change":"2024-04-06T12:28:40.690156+0000","last_active":"2024-04-06T12:28:41.670306+0000","last_peered":"2024-04-06T12:28:41.670306+0000","last_clean":"2024-04-06T12:28:41.670306+0000","last_became_active":"2024-04-06T12:28:40.689888+0000","last_became_peered":"2024-04-06T12:28:40.689888+0000","last_unstale":"2024-04-06T12:28:41.670306+0000","last_undegraded":"2024-04-06T12:28:41.670306+0000","last_fullsized":"2024-04-06T12:28:41.670306+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:08:10.898942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.056246+0000","last_change":"2024-04-06T12:28:40.674619+0000","last_active":"2024-04-06T12:28:42.056246+0000","last_peered":"2024-04-06T12:28:42.056246+0000","last_clean":"2024-04-06T12:28:42.056246+0000","last_became_active":"2024-04-06T12:28:40.674371+0000","last_became_peered":"2024-04-06T12:28:40.674371+0000","last_unstale":"2024-04-06T12:28:42.056246+0000","last_undegraded":"2024-04-06T12:28:42.056246+0000","last_fullsized":"2024-04-06T12:28:42.056246+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:05:24.085708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406067+0000","last_change":"2024-04-06T12:28:40.689355+0000","last_active":"2024-04-06T12:28:42.406067+0000","last_peered":"2024-04-06T12:28:42.406067+0000","last_clean":"2024-04-06T12:28:42.406067+0000","last_became_active":"2024-04-06T12:28:40.689116+0000","last_became_peered":"2024-04-06T12:28:40.689116+0000","last_unstale":"2024-04-06T12:28:42.406067+0000","last_undegraded":"2024-04-06T12:28:42.406067+0000","last_fullsized":"2024-04-06T12:28:42.406067+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:59.084289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669548+0000","last_change":"2024-04-06T12:28:40.680709+0000","last_active":"2024-04-06T12:28:41.669548+0000","last_peered":"2024-04-06T12:28:41.669548+0000","last_clean":"2024-04-06T12:28:41.669548+0000","last_became_active":"2024-04-06T12:28:40.680441+0000","last_became_peered":"2024-04-06T12:28:40.680441+0000","last_unstale":"2024-04-06T12:28:41.669548+0000","last_undegraded":"2024-04-06T12:28:41.669548+0000","last_fullsized":"2024-04-06T12:28:41.669548+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:04:33.119466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669964+0000","last_change":"2024-04-06T12:28:40.680796+0000","last_active":"2024-04-06T12:28:41.669964+0000","last_peered":"2024-04-06T12:28:41.669964+0000","last_clean":"2024-04-06T12:28:41.669964+0000","last_became_active":"2024-04-06T12:28:40.680549+0000","last_became_peered":"2024-04-06T12:28:40.680549+0000","last_unstale":"2024-04-06T12:28:41.669964+0000","last_undegraded":"2024-04-06T12:28:41.669964+0000","last_fullsized":"2024-04-06T12:28:41.669964+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:06:16.040887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405510+0000","last_change":"2024-04-06T12:28:40.694012+0000","last_active":"2024-04-06T12:28:42.405510+0000","last_peered":"2024-04-06T12:28:42.405510+0000","last_clean":"2024-04-06T12:28:42.405510+0000","last_became_active":"2024-04-06T12:28:40.693754+0000","last_became_peered":"2024-04-06T12:28:40.693754+0000","last_unstale":"2024-04-06T12:28:42.405510+0000","last_undegraded":"2024-04-06T12:28:42.405510+0000","last_fullsized":"2024-04-06T12:28:42.405510+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:14.593914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405962+0000","last_change":"2024-04-06T12:28:40.681220+0000","last_active":"2024-04-06T12:28:42.405962+0000","last_peered":"2024-04-06T12:28:42.405962+0000","last_clean":"2024-04-06T12:28:42.405962+0000","last_became_active":"2024-04-06T12:28:40.680608+0000","last_became_peered":"2024-04-06T12:28:40.680608+0000","last_unstale":"2024-04-06T12:28:42.405962+0000","last_undegraded":"2024-04-06T12:28:42.405962+0000","last_fullsized":"2024-04-06T12:28:42.405962+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:42:39.164250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406624+0000","last_change":"2024-04-06T12:28:40.684269+0000","last_active":"2024-04-06T12:28:42.406624+0000","last_peered":"2024-04-06T12:28:42.406624+0000","last_clean":"2024-04-06T12:28:42.406624+0000","last_became_active":"2024-04-06T12:28:40.684027+0000","last_became_peered":"2024-04-06T12:28:40.684027+0000","last_unstale":"2024-04-06T12:28:42.406624+0000","last_undegraded":"2024-04-06T12:28:42.406624+0000","last_fullsized":"2024-04-06T12:28:42.406624+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:42:31.925200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"111'4","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670046+0000","last_change":"2024-04-06T12:28:40.268591+0000","last_active":"2024-04-06T12:28:41.670046+0000","last_peered":"2024-04-06T12:28:41.670046+0000","last_clean":"2024-04-06T12:28:41.670046+0000","last_became_active":"2024-04-06T12:28:40.265662+0000","last_became_peered":"2024-04-06T12:28:40.265662+0000","last_unstale":"2024-04-06T12:28:41.670046+0000","last_undegraded":"2024-04-06T12:28:41.670046+0000","last_fullsized":"2024-04-06T12:28:41.670046+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:47:58.993684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.407204+0000","last_change":"2024-04-06T12:28:40.675397+0000","last_active":"2024-04-06T12:28:42.407204+0000","last_peered":"2024-04-06T12:28:42.407204+0000","last_clean":"2024-04-06T12:28:42.407204+0000","last_became_active":"2024-04-06T12:28:40.675133+0000","last_became_peered":"2024-04-06T12:28:40.675133+0000","last_unstale":"2024-04-06T12:28:42.407204+0000","last_undegraded":"2024-04-06T12:28:42.407204+0000","last_fullsized":"2024-04-06T12:28:42.407204+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:47:12.747983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"111'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670362+0000","last_change":"2024-04-06T12:28:40.266851+0000","last_active":"2024-04-06T12:28:41.670362+0000","last_peered":"2024-04-06T12:28:41.670362+0000","last_clean":"2024-04-06T12:28:41.670362+0000","last_became_active":"2024-04-06T12:28:40.266567+0000","last_became_peered":"2024-04-06T12:28:40.266567+0000","last_unstale":"2024-04-06T12:28:41.670362+0000","last_undegraded":"2024-04-06T12:28:41.670362+0000","last_fullsized":"2024-04-06T12:28:41.670362+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:13:58.017435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.668962+0000","last_change":"2024-04-06T12:28:40.687281+0000","last_active":"2024-04-06T12:28:41.668962+0000","last_peered":"2024-04-06T12:28:41.668962+0000","last_clean":"2024-04-06T12:28:41.668962+0000","last_became_active":"2024-04-06T12:28:40.686940+0000","last_became_peered":"2024-04-06T12:28:40.686940+0000","last_unstale":"2024-04-06T12:28:41.668962+0000","last_undegraded":"2024-04-06T12:28:41.668962+0000","last_fullsized":"2024-04-06T12:28:41.668962+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:49:51.903838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"111'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669512+0000","last_change":"2024-04-06T12:28:40.260409+0000","last_active":"2024-04-06T12:28:41.669512+0000","last_peered":"2024-04-06T12:28:41.669512+0000","last_clean":"2024-04-06T12:28:41.669512+0000","last_became_active":"2024-04-06T12:28:40.260065+0000","last_became_peered":"2024-04-06T12:28:40.260065+0000","last_unstale":"2024-04-06T12:28:41.669512+0000","last_undegraded":"2024-04-06T12:28:41.669512+0000","last_fullsized":"2024-04-06T12:28:41.669512+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:16:56.816191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406981+0000","last_change":"2024-04-06T12:28:40.681978+0000","last_active":"2024-04-06T12:28:42.406981+0000","last_peered":"2024-04-06T12:28:42.406981+0000","last_clean":"2024-04-06T12:28:42.406981+0000","last_became_active":"2024-04-06T12:28:40.681614+0000","last_became_peered":"2024-04-06T12:28:40.681614+0000","last_unstale":"2024-04-06T12:28:42.406981+0000","last_undegraded":"2024-04-06T12:28:42.406981+0000","last_fullsized":"2024-04-06T12:28:42.406981+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:47:18.019643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"111'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669554+0000","last_change":"2024-04-06T12:28:40.264333+0000","last_active":"2024-04-06T12:28:41.669554+0000","last_peered":"2024-04-06T12:28:41.669554+0000","last_clean":"2024-04-06T12:28:41.669554+0000","last_became_active":"2024-04-06T12:28:40.264107+0000","last_became_peered":"2024-04-06T12:28:40.264107+0000","last_unstale":"2024-04-06T12:28:41.669554+0000","last_undegraded":"2024-04-06T12:28:41.669554+0000","last_fullsized":"2024-04-06T12:28:41.669554+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:07:20.434672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405742+0000","last_change":"2024-04-06T12:28:40.693029+0000","last_active":"2024-04-06T12:28:42.405742+0000","last_peered":"2024-04-06T12:28:42.405742+0000","last_clean":"2024-04-06T12:28:42.405742+0000","last_became_active":"2024-04-06T12:28:40.692780+0000","last_became_peered":"2024-04-06T12:28:40.692780+0000","last_unstale":"2024-04-06T12:28:42.405742+0000","last_undegraded":"2024-04-06T12:28:42.405742+0000","last_fullsized":"2024-04-06T12:28:42.405742+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:53:23.333343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"111'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670241+0000","last_change":"2024-04-06T12:28:40.268542+0000","last_active":"2024-04-06T12:28:41.670241+0000","last_peered":"2024-04-06T12:28:41.670241+0000","last_clean":"2024-04-06T12:28:41.670241+0000","last_became_active":"2024-04-06T12:28:40.268275+0000","last_became_peered":"2024-04-06T12:28:40.268275+0000","last_unstale":"2024-04-06T12:28:41.670241+0000","last_undegraded":"2024-04-06T12:28:41.670241+0000","last_fullsized":"2024-04-06T12:28:41.670241+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:31:13.268239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669524+0000","last_change":"2024-04-06T12:28:40.681615+0000","last_active":"2024-04-06T12:28:41.669524+0000","last_peered":"2024-04-06T12:28:41.669524+0000","last_clean":"2024-04-06T12:28:41.669524+0000","last_became_active":"2024-04-06T12:28:40.681282+0000","last_became_peered":"2024-04-06T12:28:40.681282+0000","last_unstale":"2024-04-06T12:28:41.669524+0000","last_undegraded":"2024-04-06T12:28:41.669524+0000","last_fullsized":"2024-04-06T12:28:41.669524+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:45:21.708635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"111'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406452+0000","last_change":"2024-04-06T12:28:40.263780+0000","last_active":"2024-04-06T12:28:42.406452+0000","last_peered":"2024-04-06T12:28:42.406452+0000","last_clean":"2024-04-06T12:28:42.406452+0000","last_became_active":"2024-04-06T12:28:40.263416+0000","last_became_peered":"2024-04-06T12:28:40.263416+0000","last_unstale":"2024-04-06T12:28:42.406452+0000","last_undegraded":"2024-04-06T12:28:42.406452+0000","last_fullsized":"2024-04-06T12:28:42.406452+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:44:39.076665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406905+0000","last_change":"2024-04-06T12:28:40.692001+0000","last_active":"2024-04-06T12:28:42.406905+0000","last_peered":"2024-04-06T12:28:42.406905+0000","last_clean":"2024-04-06T12:28:42.406905+0000","last_became_active":"2024-04-06T12:28:40.691403+0000","last_became_peered":"2024-04-06T12:28:40.691403+0000","last_unstale":"2024-04-06T12:28:42.406905+0000","last_undegraded":"2024-04-06T12:28:42.406905+0000","last_fullsized":"2024-04-06T12:28:42.406905+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:37.635782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406705+0000","last_change":"2024-04-06T12:28:40.261521+0000","last_active":"2024-04-06T12:28:42.406705+0000","last_peered":"2024-04-06T12:28:42.406705+0000","last_clean":"2024-04-06T12:28:42.406705+0000","last_became_active":"2024-04-06T12:28:40.261256+0000","last_became_peered":"2024-04-06T12:28:40.261256+0000","last_unstale":"2024-04-06T12:28:42.406705+0000","last_undegraded":"2024-04-06T12:28:42.406705+0000","last_fullsized":"2024-04-06T12:28:42.406705+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:51:16.747872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406464+0000","last_change":"2024-04-06T12:28:40.687255+0000","last_active":"2024-04-06T12:28:42.406464+0000","last_peered":"2024-04-06T12:28:42.406464+0000","last_clean":"2024-04-06T12:28:42.406464+0000","last_became_active":"2024-04-06T12:28:40.686989+0000","last_became_peered":"2024-04-06T12:28:40.686989+0000","last_unstale":"2024-04-06T12:28:42.406464+0000","last_undegraded":"2024-04-06T12:28:42.406464+0000","last_fullsized":"2024-04-06T12:28:42.406464+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:04:09.377839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669022+0000","last_change":"2024-04-06T12:28:40.255069+0000","last_active":"2024-04-06T12:28:41.669022+0000","last_peered":"2024-04-06T12:28:41.669022+0000","last_clean":"2024-04-06T12:28:41.669022+0000","last_became_active":"2024-04-06T12:28:40.254778+0000","last_became_peered":"2024-04-06T12:28:40.254778+0000","last_unstale":"2024-04-06T12:28:41.669022+0000","last_undegraded":"2024-04-06T12:28:41.669022+0000","last_fullsized":"2024-04-06T12:28:41.669022+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:23:57.255553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669989+0000","last_change":"2024-04-06T12:28:40.677839+0000","last_active":"2024-04-06T12:28:41.669989+0000","last_peered":"2024-04-06T12:28:41.669989+0000","last_clean":"2024-04-06T12:28:41.669989+0000","last_became_active":"2024-04-06T12:28:40.676824+0000","last_became_peered":"2024-04-06T12:28:40.676824+0000","last_unstale":"2024-04-06T12:28:41.669989+0000","last_undegraded":"2024-04-06T12:28:41.669989+0000","last_fullsized":"2024-04-06T12:28:41.669989+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:59.084289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405849+0000","last_change":"2024-04-06T12:28:40.265329+0000","last_active":"2024-04-06T12:28:42.405849+0000","last_peered":"2024-04-06T12:28:42.405849+0000","last_clean":"2024-04-06T12:28:42.405849+0000","last_became_active":"2024-04-06T12:28:40.264509+0000","last_became_peered":"2024-04-06T12:28:40.264509+0000","last_unstale":"2024-04-06T12:28:42.405849+0000","last_undegraded":"2024-04-06T12:28:42.405849+0000","last_fullsized":"2024-04-06T12:28:42.405849+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:09:21.699469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669418+0000","last_change":"2024-04-06T12:28:40.675175+0000","last_active":"2024-04-06T12:28:41.669418+0000","last_peered":"2024-04-06T12:28:41.669418+0000","last_clean":"2024-04-06T12:28:41.669418+0000","last_became_active":"2024-04-06T12:28:40.674899+0000","last_became_peered":"2024-04-06T12:28:40.674899+0000","last_unstale":"2024-04-06T12:28:41.669418+0000","last_undegraded":"2024-04-06T12:28:41.669418+0000","last_fullsized":"2024-04-06T12:28:41.669418+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:59:47.830391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406842+0000","last_change":"2024-04-06T12:28:40.263044+0000","last_active":"2024-04-06T12:28:42.406842+0000","last_peered":"2024-04-06T12:28:42.406842+0000","last_clean":"2024-04-06T12:28:42.406842+0000","last_became_active":"2024-04-06T12:28:40.262786+0000","last_became_peered":"2024-04-06T12:28:40.262786+0000","last_unstale":"2024-04-06T12:28:42.406842+0000","last_undegraded":"2024-04-06T12:28:42.406842+0000","last_fullsized":"2024-04-06T12:28:42.406842+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:33:58.849161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406646+0000","last_change":"2024-04-06T12:28:40.687500+0000","last_active":"2024-04-06T12:28:42.406646+0000","last_peered":"2024-04-06T12:28:42.406646+0000","last_clean":"2024-04-06T12:28:42.406646+0000","last_became_active":"2024-04-06T12:28:40.681387+0000","last_became_peered":"2024-04-06T12:28:40.681387+0000","last_unstale":"2024-04-06T12:28:42.406646+0000","last_undegraded":"2024-04-06T12:28:42.406646+0000","last_fullsized":"2024-04-06T12:28:42.406646+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:10:48.365828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"111'3","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669819+0000","last_change":"2024-04-06T12:28:40.269680+0000","last_active":"2024-04-06T12:28:41.669819+0000","last_peered":"2024-04-06T12:28:41.669819+0000","last_clean":"2024-04-06T12:28:41.669819+0000","last_became_active":"2024-04-06T12:28:40.269382+0000","last_became_peered":"2024-04-06T12:28:40.269382+0000","last_unstale":"2024-04-06T12:28:41.669819+0000","last_undegraded":"2024-04-06T12:28:41.669819+0000","last_fullsized":"2024-04-06T12:28:41.669819+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:22:51.915093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406607+0000","last_change":"2024-04-06T12:28:40.687087+0000","last_active":"2024-04-06T12:28:42.406607+0000","last_peered":"2024-04-06T12:28:42.406607+0000","last_clean":"2024-04-06T12:28:42.406607+0000","last_became_active":"2024-04-06T12:28:40.686841+0000","last_became_peered":"2024-04-06T12:28:40.686841+0000","last_unstale":"2024-04-06T12:28:42.406607+0000","last_undegraded":"2024-04-06T12:28:42.406607+0000","last_fullsized":"2024-04-06T12:28:42.406607+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:27:34.410340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"111'6","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406718+0000","last_change":"2024-04-06T12:28:40.266700+0000","last_active":"2024-04-06T12:28:42.406718+0000","last_peered":"2024-04-06T12:28:42.406718+0000","last_clean":"2024-04-06T12:28:42.406718+0000","last_became_active":"2024-04-06T12:28:40.266424+0000","last_became_peered":"2024-04-06T12:28:40.266424+0000","last_unstale":"2024-04-06T12:28:42.406718+0000","last_undegraded":"2024-04-06T12:28:42.406718+0000","last_fullsized":"2024-04-06T12:28:42.406718+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:13.861933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405850+0000","last_change":"2024-04-06T12:28:40.689348+0000","last_active":"2024-04-06T12:28:42.405850+0000","last_peered":"2024-04-06T12:28:42.405850+0000","last_clean":"2024-04-06T12:28:42.405850+0000","last_became_active":"2024-04-06T12:28:40.689089+0000","last_became_peered":"2024-04-06T12:28:40.689089+0000","last_unstale":"2024-04-06T12:28:42.405850+0000","last_undegraded":"2024-04-06T12:28:42.405850+0000","last_fullsized":"2024-04-06T12:28:42.405850+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:16:57.826003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"111'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669737+0000","last_change":"2024-04-06T12:28:40.263978+0000","last_active":"2024-04-06T12:28:41.669737+0000","last_peered":"2024-04-06T12:28:41.669737+0000","last_clean":"2024-04-06T12:28:41.669737+0000","last_became_active":"2024-04-06T12:28:40.263711+0000","last_became_peered":"2024-04-06T12:28:40.263711+0000","last_unstale":"2024-04-06T12:28:41.669737+0000","last_undegraded":"2024-04-06T12:28:41.669737+0000","last_fullsized":"2024-04-06T12:28:41.669737+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:44:55.192470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669357+0000","last_change":"2024-04-06T12:28:40.676784+0000","last_active":"2024-04-06T12:28:41.669357+0000","last_peered":"2024-04-06T12:28:41.669357+0000","last_clean":"2024-04-06T12:28:41.669357+0000","last_became_active":"2024-04-06T12:28:40.676509+0000","last_became_peered":"2024-04-06T12:28:40.676509+0000","last_unstale":"2024-04-06T12:28:41.669357+0000","last_undegraded":"2024-04-06T12:28:41.669357+0000","last_fullsized":"2024-04-06T12:28:41.669357+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:42:19.971508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406993+0000","last_change":"2024-04-06T12:28:40.267877+0000","last_active":"2024-04-06T12:28:42.406993+0000","last_peered":"2024-04-06T12:28:42.406993+0000","last_clean":"2024-04-06T12:28:42.406993+0000","last_became_active":"2024-04-06T12:28:40.267602+0000","last_became_peered":"2024-04-06T12:28:40.267602+0000","last_unstale":"2024-04-06T12:28:42.406993+0000","last_undegraded":"2024-04-06T12:28:42.406993+0000","last_fullsized":"2024-04-06T12:28:42.406993+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:10:47.356015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.056302+0000","last_change":"2024-04-06T12:28:40.688068+0000","last_active":"2024-04-06T12:28:42.056302+0000","last_peered":"2024-04-06T12:28:42.056302+0000","last_clean":"2024-04-06T12:28:42.056302+0000","last_became_active":"2024-04-06T12:28:40.687785+0000","last_became_peered":"2024-04-06T12:28:40.687785+0000","last_unstale":"2024-04-06T12:28:42.056302+0000","last_undegraded":"2024-04-06T12:28:42.056302+0000","last_fullsized":"2024-04-06T12:28:42.056302+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:20:18.808689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405925+0000","last_change":"2024-04-06T12:28:40.270013+0000","last_active":"2024-04-06T12:28:42.405925+0000","last_peered":"2024-04-06T12:28:42.405925+0000","last_clean":"2024-04-06T12:28:42.405925+0000","last_became_active":"2024-04-06T12:28:40.269562+0000","last_became_peered":"2024-04-06T12:28:40.269562+0000","last_unstale":"2024-04-06T12:28:42.405925+0000","last_undegraded":"2024-04-06T12:28:42.405925+0000","last_fullsized":"2024-04-06T12:28:42.405925+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:45:20.698822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.056410+0000","last_change":"2024-04-06T12:28:40.683998+0000","last_active":"2024-04-06T12:28:42.056410+0000","last_peered":"2024-04-06T12:28:42.056410+0000","last_clean":"2024-04-06T12:28:42.056410+0000","last_became_active":"2024-04-06T12:28:40.683721+0000","last_became_peered":"2024-04-06T12:28:40.683721+0000","last_unstale":"2024-04-06T12:28:42.056410+0000","last_undegraded":"2024-04-06T12:28:42.056410+0000","last_fullsized":"2024-04-06T12:28:42.056410+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:40:18.946044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"111'8","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.407301+0000","last_change":"2024-04-06T12:28:40.264869+0000","last_active":"2024-04-06T12:28:42.407301+0000","last_peered":"2024-04-06T12:28:42.407301+0000","last_clean":"2024-04-06T12:28:42.407301+0000","last_became_active":"2024-04-06T12:28:40.264614+0000","last_became_peered":"2024-04-06T12:28:40.264614+0000","last_unstale":"2024-04-06T12:28:42.407301+0000","last_undegraded":"2024-04-06T12:28:42.407301+0000","last_fullsized":"2024-04-06T12:28:42.407301+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:18:23.534593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406519+0000","last_change":"2024-04-06T12:28:40.685374+0000","last_active":"2024-04-06T12:28:42.406519+0000","last_peered":"2024-04-06T12:28:42.406519+0000","last_clean":"2024-04-06T12:28:42.406519+0000","last_became_active":"2024-04-06T12:28:40.684181+0000","last_became_peered":"2024-04-06T12:28:40.684181+0000","last_unstale":"2024-04-06T12:28:42.406519+0000","last_undegraded":"2024-04-06T12:28:42.406519+0000","last_fullsized":"2024-04-06T12:28:42.406519+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:25:51.107371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"111'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670134+0000","last_change":"2024-04-06T12:28:40.265372+0000","last_active":"2024-04-06T12:28:41.670134+0000","last_peered":"2024-04-06T12:28:41.670134+0000","last_clean":"2024-04-06T12:28:41.670134+0000","last_became_active":"2024-04-06T12:28:40.265098+0000","last_became_peered":"2024-04-06T12:28:40.265098+0000","last_unstale":"2024-04-06T12:28:41.670134+0000","last_undegraded":"2024-04-06T12:28:41.670134+0000","last_fullsized":"2024-04-06T12:28:41.670134+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:31:13.268239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670225+0000","last_change":"2024-04-06T12:28:40.684162+0000","last_active":"2024-04-06T12:28:41.670225+0000","last_peered":"2024-04-06T12:28:41.670225+0000","last_clean":"2024-04-06T12:28:41.670225+0000","last_became_active":"2024-04-06T12:28:40.683748+0000","last_became_peered":"2024-04-06T12:28:40.683748+0000","last_unstale":"2024-04-06T12:28:41.670225+0000","last_undegraded":"2024-04-06T12:28:41.670225+0000","last_fullsized":"2024-04-06T12:28:41.670225+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:18:55.287857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"111'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669064+0000","last_change":"2024-04-06T12:28:40.264845+0000","last_active":"2024-04-06T12:28:41.669064+0000","last_peered":"2024-04-06T12:28:41.669064+0000","last_clean":"2024-04-06T12:28:41.669064+0000","last_became_active":"2024-04-06T12:28:40.264607+0000","last_became_peered":"2024-04-06T12:28:40.264607+0000","last_unstale":"2024-04-06T12:28:41.669064+0000","last_undegraded":"2024-04-06T12:28:41.669064+0000","last_fullsized":"2024-04-06T12:28:41.669064+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:06:30.260729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670970+0000","last_change":"2024-04-06T12:28:40.677641+0000","last_active":"2024-04-06T12:28:41.670970+0000","last_peered":"2024-04-06T12:28:41.670970+0000","last_clean":"2024-04-06T12:28:41.670970+0000","last_became_active":"2024-04-06T12:28:40.677361+0000","last_became_peered":"2024-04-06T12:28:40.677361+0000","last_unstale":"2024-04-06T12:28:41.670970+0000","last_undegraded":"2024-04-06T12:28:41.670970+0000","last_fullsized":"2024-04-06T12:28:41.670970+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:18:29.810477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"111'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670273+0000","last_change":"2024-04-06T12:28:40.270490+0000","last_active":"2024-04-06T12:28:41.670273+0000","last_peered":"2024-04-06T12:28:41.670273+0000","last_clean":"2024-04-06T12:28:41.670273+0000","last_became_active":"2024-04-06T12:28:40.270201+0000","last_became_peered":"2024-04-06T12:28:40.270201+0000","last_unstale":"2024-04-06T12:28:41.670273+0000","last_undegraded":"2024-04-06T12:28:41.670273+0000","last_fullsized":"2024-04-06T12:28:41.670273+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:59:43.892496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406070+0000","last_change":"2024-04-06T12:28:40.691581+0000","last_active":"2024-04-06T12:28:42.406070+0000","last_peered":"2024-04-06T12:28:42.406070+0000","last_clean":"2024-04-06T12:28:42.406070+0000","last_became_active":"2024-04-06T12:28:40.691306+0000","last_became_peered":"2024-04-06T12:28:40.691306+0000","last_unstale":"2024-04-06T12:28:42.406070+0000","last_undegraded":"2024-04-06T12:28:42.406070+0000","last_fullsized":"2024-04-06T12:28:42.406070+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:26.747580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"111'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405630+0000","last_change":"2024-04-06T12:28:40.263453+0000","last_active":"2024-04-06T12:28:42.405630+0000","last_peered":"2024-04-06T12:28:42.405630+0000","last_clean":"2024-04-06T12:28:42.405630+0000","last_became_active":"2024-04-06T12:28:40.263153+0000","last_became_peered":"2024-04-06T12:28:40.263153+0000","last_unstale":"2024-04-06T12:28:42.405630+0000","last_undegraded":"2024-04-06T12:28:42.405630+0000","last_fullsized":"2024-04-06T12:28:42.405630+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:22:51.915093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405640+0000","last_change":"2024-04-06T12:28:40.674977+0000","last_active":"2024-04-06T12:28:42.405640+0000","last_peered":"2024-04-06T12:28:42.405640+0000","last_clean":"2024-04-06T12:28:42.405640+0000","last_became_active":"2024-04-06T12:28:40.674677+0000","last_became_peered":"2024-04-06T12:28:40.674677+0000","last_unstale":"2024-04-06T12:28:42.405640+0000","last_undegraded":"2024-04-06T12:28:42.405640+0000","last_fullsized":"2024-04-06T12:28:42.405640+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:32:19.699532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.056204+0000","last_change":"2024-04-06T12:28:40.262648+0000","last_active":"2024-04-06T12:28:42.056204+0000","last_peered":"2024-04-06T12:28:42.056204+0000","last_clean":"2024-04-06T12:28:42.056204+0000","last_became_active":"2024-04-06T12:28:40.262363+0000","last_became_peered":"2024-04-06T12:28:40.262363+0000","last_unstale":"2024-04-06T12:28:42.056204+0000","last_undegraded":"2024-04-06T12:28:42.056204+0000","last_fullsized":"2024-04-06T12:28:42.056204+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:31:14.983665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405474+0000","last_change":"2024-04-06T12:28:40.688650+0000","last_active":"2024-04-06T12:28:42.405474+0000","last_peered":"2024-04-06T12:28:42.405474+0000","last_clean":"2024-04-06T12:28:42.405474+0000","last_became_active":"2024-04-06T12:28:40.687839+0000","last_became_peered":"2024-04-06T12:28:40.687839+0000","last_unstale":"2024-04-06T12:28:42.405474+0000","last_undegraded":"2024-04-06T12:28:42.405474+0000","last_fullsized":"2024-04-06T12:28:42.405474+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:44:56.202282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405668+0000","last_change":"2024-04-06T12:28:40.269996+0000","last_active":"2024-04-06T12:28:42.405668+0000","last_peered":"2024-04-06T12:28:42.405668+0000","last_clean":"2024-04-06T12:28:42.405668+0000","last_became_active":"2024-04-06T12:28:40.269522+0000","last_became_peered":"2024-04-06T12:28:40.269522+0000","last_unstale":"2024-04-06T12:28:42.405668+0000","last_undegraded":"2024-04-06T12:28:42.405668+0000","last_fullsized":"2024-04-06T12:28:42.405668+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:20:16.164471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406030+0000","last_change":"2024-04-06T12:28:40.681010+0000","last_active":"2024-04-06T12:28:42.406030+0000","last_peered":"2024-04-06T12:28:42.406030+0000","last_clean":"2024-04-06T12:28:42.406030+0000","last_became_active":"2024-04-06T12:28:40.680002+0000","last_became_peered":"2024-04-06T12:28:40.680002+0000","last_unstale":"2024-04-06T12:28:42.406030+0000","last_undegraded":"2024-04-06T12:28:42.406030+0000","last_fullsized":"2024-04-06T12:28:42.406030+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:21:36.738869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.056355+0000","last_change":"2024-04-06T12:28:40.260606+0000","last_active":"2024-04-06T12:28:42.056355+0000","last_peered":"2024-04-06T12:28:42.056355+0000","last_clean":"2024-04-06T12:28:42.056355+0000","last_became_active":"2024-04-06T12:28:40.260317+0000","last_became_peered":"2024-04-06T12:28:40.260317+0000","last_unstale":"2024-04-06T12:28:42.056355+0000","last_undegraded":"2024-04-06T12:28:42.056355+0000","last_fullsized":"2024-04-06T12:28:42.056355+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:14:25.108123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669521+0000","last_change":"2024-04-06T12:28:40.680189+0000","last_active":"2024-04-06T12:28:41.669521+0000","last_peered":"2024-04-06T12:28:41.669521+0000","last_clean":"2024-04-06T12:28:41.669521+0000","last_became_active":"2024-04-06T12:28:40.679236+0000","last_became_peered":"2024-04-06T12:28:40.679236+0000","last_unstale":"2024-04-06T12:28:41.669521+0000","last_undegraded":"2024-04-06T12:28:41.669521+0000","last_fullsized":"2024-04-06T12:28:41.669521+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:18:24.544405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"111'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669183+0000","last_change":"2024-04-06T12:28:40.267091+0000","last_active":"2024-04-06T12:28:41.669183+0000","last_peered":"2024-04-06T12:28:41.669183+0000","last_clean":"2024-04-06T12:28:41.669183+0000","last_became_active":"2024-04-06T12:28:40.266679+0000","last_became_peered":"2024-04-06T12:28:40.266679+0000","last_unstale":"2024-04-06T12:28:41.669183+0000","last_undegraded":"2024-04-06T12:28:41.669183+0000","last_fullsized":"2024-04-06T12:28:41.669183+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:13.584102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.407301+0000","last_change":"2024-04-06T12:28:40.684385+0000","last_active":"2024-04-06T12:28:42.407301+0000","last_peered":"2024-04-06T12:28:42.407301+0000","last_clean":"2024-04-06T12:28:42.407301+0000","last_became_active":"2024-04-06T12:28:40.684108+0000","last_became_peered":"2024-04-06T12:28:40.684108+0000","last_unstale":"2024-04-06T12:28:42.407301+0000","last_undegraded":"2024-04-06T12:28:42.407301+0000","last_fullsized":"2024-04-06T12:28:42.407301+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:24:57.296309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670215+0000","last_change":"2024-04-06T12:28:40.270102+0000","last_active":"2024-04-06T12:28:41.670215+0000","last_peered":"2024-04-06T12:28:41.670215+0000","last_clean":"2024-04-06T12:28:41.670215+0000","last_became_active":"2024-04-06T12:28:40.269825+0000","last_became_peered":"2024-04-06T12:28:40.269825+0000","last_unstale":"2024-04-06T12:28:41.670215+0000","last_undegraded":"2024-04-06T12:28:41.670215+0000","last_fullsized":"2024-04-06T12:28:41.670215+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:20:16.164471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406953+0000","last_change":"2024-04-06T12:28:40.680919+0000","last_active":"2024-04-06T12:28:42.406953+0000","last_peered":"2024-04-06T12:28:42.406953+0000","last_clean":"2024-04-06T12:28:42.406953+0000","last_became_active":"2024-04-06T12:28:40.680665+0000","last_became_peered":"2024-04-06T12:28:40.680665+0000","last_unstale":"2024-04-06T12:28:42.406953+0000","last_undegraded":"2024-04-06T12:28:42.406953+0000","last_fullsized":"2024-04-06T12:28:42.406953+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:32:13.448506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669545+0000","last_change":"2024-04-06T12:28:40.269837+0000","last_active":"2024-04-06T12:28:41.669545+0000","last_peered":"2024-04-06T12:28:41.669545+0000","last_clean":"2024-04-06T12:28:41.669545+0000","last_became_active":"2024-04-06T12:28:40.269481+0000","last_became_peered":"2024-04-06T12:28:40.269481+0000","last_unstale":"2024-04-06T12:28:41.669545+0000","last_undegraded":"2024-04-06T12:28:41.669545+0000","last_fullsized":"2024-04-06T12:28:41.669545+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:09:21.699469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405850+0000","last_change":"2024-04-06T12:28:40.683273+0000","last_active":"2024-04-06T12:28:42.405850+0000","last_peered":"2024-04-06T12:28:42.405850+0000","last_clean":"2024-04-06T12:28:42.405850+0000","last_became_active":"2024-04-06T12:28:40.683003+0000","last_became_peered":"2024-04-06T12:28:40.683003+0000","last_unstale":"2024-04-06T12:28:42.405850+0000","last_undegraded":"2024-04-06T12:28:42.405850+0000","last_fullsized":"2024-04-06T12:28:42.405850+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:23:58.265366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669627+0000","last_change":"2024-04-06T12:28:40.263840+0000","last_active":"2024-04-06T12:28:41.669627+0000","last_peered":"2024-04-06T12:28:41.669627+0000","last_clean":"2024-04-06T12:28:41.669627+0000","last_became_active":"2024-04-06T12:28:40.262653+0000","last_became_peered":"2024-04-06T12:28:40.262653+0000","last_unstale":"2024-04-06T12:28:41.669627+0000","last_undegraded":"2024-04-06T12:28:41.669627+0000","last_fullsized":"2024-04-06T12:28:41.669627+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:03:32.755928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.056136+0000","last_change":"2024-04-06T12:28:40.682380+0000","last_active":"2024-04-06T12:28:42.056136+0000","last_peered":"2024-04-06T12:28:42.056136+0000","last_clean":"2024-04-06T12:28:42.056136+0000","last_became_active":"2024-04-06T12:28:40.682116+0000","last_became_peered":"2024-04-06T12:28:40.682116+0000","last_unstale":"2024-04-06T12:28:42.056136+0000","last_undegraded":"2024-04-06T12:28:42.056136+0000","last_fullsized":"2024-04-06T12:28:42.056136+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:30:04.557762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406292+0000","last_change":"2024-04-06T12:28:40.267887+0000","last_active":"2024-04-06T12:28:42.406292+0000","last_peered":"2024-04-06T12:28:42.406292+0000","last_clean":"2024-04-06T12:28:42.406292+0000","last_became_active":"2024-04-06T12:28:40.267615+0000","last_became_peered":"2024-04-06T12:28:40.267615+0000","last_unstale":"2024-04-06T12:28:42.406292+0000","last_undegraded":"2024-04-06T12:28:42.406292+0000","last_fullsized":"2024-04-06T12:28:42.406292+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:44:32.727817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406602+0000","last_change":"2024-04-06T12:28:40.688761+0000","last_active":"2024-04-06T12:28:42.406602+0000","last_peered":"2024-04-06T12:28:42.406602+0000","last_clean":"2024-04-06T12:28:42.406602+0000","last_became_active":"2024-04-06T12:28:40.688502+0000","last_became_peered":"2024-04-06T12:28:40.688502+0000","last_unstale":"2024-04-06T12:28:42.406602+0000","last_undegraded":"2024-04-06T12:28:42.406602+0000","last_fullsized":"2024-04-06T12:28:42.406602+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:14.871746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.405555+0000","last_change":"2024-04-06T12:28:40.263982+0000","last_active":"2024-04-06T12:28:42.405555+0000","last_peered":"2024-04-06T12:28:42.405555+0000","last_clean":"2024-04-06T12:28:42.405555+0000","last_became_active":"2024-04-06T12:28:40.263315+0000","last_became_peered":"2024-04-06T12:28:40.263315+0000","last_unstale":"2024-04-06T12:28:42.405555+0000","last_undegraded":"2024-04-06T12:28:42.405555+0000","last_fullsized":"2024-04-06T12:28:42.405555+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:59:43.892496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669669+0000","last_change":"2024-04-06T12:28:40.688444+0000","last_active":"2024-04-06T12:28:41.669669+0000","last_peered":"2024-04-06T12:28:41.669669+0000","last_clean":"2024-04-06T12:28:41.669669+0000","last_became_active":"2024-04-06T12:28:40.688175+0000","last_became_peered":"2024-04-06T12:28:40.688175+0000","last_unstale":"2024-04-06T12:28:41.669669+0000","last_undegraded":"2024-04-06T12:28:41.669669+0000","last_fullsized":"2024-04-06T12:28:41.669669+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:43:56.668566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.670121+0000","last_change":"2024-04-06T12:28:40.258231+0000","last_active":"2024-04-06T12:28:41.670121+0000","last_peered":"2024-04-06T12:28:41.670121+0000","last_clean":"2024-04-06T12:28:41.670121+0000","last_became_active":"2024-04-06T12:28:40.257935+0000","last_became_peered":"2024-04-06T12:28:40.257935+0000","last_unstale":"2024-04-06T12:28:41.670121+0000","last_undegraded":"2024-04-06T12:28:41.670121+0000","last_fullsized":"2024-04-06T12:28:41.670121+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:19.826267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406138+0000","last_change":"2024-04-06T12:28:40.681058+0000","last_active":"2024-04-06T12:28:42.406138+0000","last_peered":"2024-04-06T12:28:42.406138+0000","last_clean":"2024-04-06T12:28:42.406138+0000","last_became_active":"2024-04-06T12:28:40.680282+0000","last_became_peered":"2024-04-06T12:28:40.680282+0000","last_unstale":"2024-04-06T12:28:42.406138+0000","last_undegraded":"2024-04-06T12:28:42.406138+0000","last_fullsized":"2024-04-06T12:28:42.406138+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:02:42.378007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669118+0000","last_change":"2024-04-06T12:28:40.263384+0000","last_active":"2024-04-06T12:28:41.669118+0000","last_peered":"2024-04-06T12:28:41.669118+0000","last_clean":"2024-04-06T12:28:41.669118+0000","last_became_active":"2024-04-06T12:28:40.263089+0000","last_became_peered":"2024-04-06T12:28:40.263089+0000","last_unstale":"2024-04-06T12:28:41.669118+0000","last_undegraded":"2024-04-06T12:28:41.669118+0000","last_fullsized":"2024-04-06T12:28:41.669118+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:04:08.368026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:41.669098+0000","last_change":"2024-04-06T12:28:40.677430+0000","last_active":"2024-04-06T12:28:41.669098+0000","last_peered":"2024-04-06T12:28:41.669098+0000","last_clean":"2024-04-06T12:28:41.669098+0000","last_became_active":"2024-04-06T12:28:40.677175+0000","last_became_peered":"2024-04-06T12:28:40.677175+0000","last_unstale":"2024-04-06T12:28:41.669098+0000","last_undegraded":"2024-04-06T12:28:41.669098+0000","last_fullsized":"2024-04-06T12:28:41.669098+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:40.242855+0000","last_clean_scrub_stamp":"2024-04-06T12:28:40.242855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:08:10.898942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-06T12:28:42.406332+0000","last_change":"2024-04-06T12:28:40.263615+0000","last_active":"2024-04-06T12:28:42.406332+0000","last_peered":"2024-04-06T12:28:42.406332+0000","last_clean":"2024-04-06T12:28:42.406332+0000","last_became_active":"2024-04-06T12:28:40.263135+0000","last_became_peered":"2024-04-06T12:28:40.263135+0000","last_unstale":"2024-04-06T12:28:42.406332+0000","last_undegraded":"2024-04-06T12:28:42.406332+0000","last_fullsized":"2024-04-06T12:28:42.406332+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:28:39.233042+0000","last_clean_scrub_stamp":"2024-04-06T12:28:39.233042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:22:51.915093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"101'183","reported_seq":498,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-06T12:28:40.665623+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:28:40.665623+0000","last_peered":"2024-04-06T12:28:40.665623+0000","last_clean":"2024-04-06T12:28:40.665623+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:28:40.665623+0000","last_undegraded":"2024-04-06T12:28:40.665623+0000","last_fullsized":"2024-04-06T12:28:40.665623+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575048,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59296,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312544,"statfs":{"total":96636764160,"available":96576045056,"internally_reserved":0,"allocated":1982464,"data_stored":1277317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.53200000000000003,"15min":0.55500000000000005},"min":{"1min":0.32000000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.78200000000000003,"5min":1.143,"15min":1.143},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.496,"5min":0.52900000000000003,"15min":0.55000000000000004},"min":{"1min":0.372,"5min":0.246,"15min":0.246},"max":{"1min":0.66100000000000003,"5min":0.77100000000000002,"15min":0.78200000000000003},"last":0.79100000000000004}]},{"osd":1,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.53200000000000003,"15min":0.55000000000000004},"min":{"1min":0.315,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64200000000000002,"5min":0.77100000000000002,"15min":0.81399999999999995},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.54300000000000004,"15min":0.56899999999999995},"min":{"1min":0.379,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.74199999999999999,"5min":0.79400000000000004,"15min":0.79600000000000004},"last":0.60799999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53600000000000003,"15min":0.55500000000000005},"min":{"1min":0.39400000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":0.83299999999999996},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.55400000000000005,"15min":0.57599999999999996},"min":{"1min":0.39400000000000002,"5min":0.34499999999999997,"15min":0.24199999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.82299999999999995},"last":0.84799999999999998}]},{"osd":3,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55800000000000005,"15min":0.57199999999999995},"min":{"1min":0.36299999999999999,"5min":0.34799999999999998,"15min":0.34399999999999997},"max":{"1min":0.73999999999999999,"5min":1.357,"15min":1.357},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57099999999999995,"15min":0.58799999999999997},"min":{"1min":0.35599999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.63400000000000001}]},{"osd":4,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.45100000000000001,"15min":0.44800000000000001},"min":{"1min":0.28399999999999997,"5min":0.27100000000000002,"15min":0.25},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.83599999999999997},"last":0.443},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53200000000000003,"15min":0.52900000000000003},"min":{"1min":0.35399999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.58299999999999996}]},{"osd":5,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.49399999999999999,"15min":0.497},"min":{"1min":0.33800000000000002,"5min":0.29299999999999998,"15min":0.23799999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.86799999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54500000000000004,"15min":0.55400000000000005},"min":{"1min":0.39600000000000002,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.79900000000000004,"5min":0.81599999999999995,"15min":0.83499999999999996},"last":0.73799999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.28799999999999998},"max":{"1min":0.84499999999999997,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.55100000000000005,"15min":0.55900000000000005},"min":{"1min":0.312,"5min":0.30399999999999999,"15min":0.26900000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.88500000000000001},"last":0.82199999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575047,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32488,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339352,"statfs":{"total":96636764160,"available":96603496448,"internally_reserved":0,"allocated":1396736,"data_stored":688042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52200000000000002,"15min":0.53300000000000003},"min":{"1min":0.34000000000000002,"5min":0.33700000000000002,"15min":0.23000000000000001},"max":{"1min":0.64400000000000002,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.52200000000000002,"15min":0.53400000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.245},"max":{"1min":0.65400000000000003,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Apr 6 12:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.56399999999999995,"15min":0.58499999999999996},"min":{"1min":0.39600000000000002,"5min":0.32700000000000001,"15min":0.24399999999999999},"max":{"1min":0.68000000000000005,"5min":0.78700000000000003,"15min":0.84999999999999998},"last":0.502},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58199999999999996,"15min":0.59599999999999997},"min":{"1min":0.42799999999999999,"5min":0.30199999999999999,"15min":0.221},"max":{"1min":0.72499999999999998,"5min":0.81999999999999995,"15min":0.83099999999999996},"last":0.58899999999999997}]},{"osd":2,"last update":"Sat Apr 6 12:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52900000000000003,"15min":0.54700000000000004},"min":{"1min":0.374,"5min":0.30499999999999999,"15min":0.29899999999999999},"max":{"1min":0.61799999999999999,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.55100000000000005,"15min":0.56699999999999995},"min":{"1min":0.35399999999999998,"5min":0.32300000000000001,"15min":0.254},"max":{"1min":0.68999999999999995,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.63400000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55900000000000005,"15min":0.57999999999999996},"min":{"1min":0.435,"5min":0.317,"15min":0.23300000000000001},"max":{"1min":0.66400000000000003,"5min":0.77300000000000002,"15min":0.81200000000000006},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.42599999999999999,"5min":0.41099999999999998,"15min":0.374},"max":{"1min":0.77600000000000002,"5min":0.82799999999999996,"15min":0.872},"last":0.69399999999999995}]},{"osd":4,"last update":"Sat Apr 6 12:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.315,"5min":0.28599999999999998,"15min":0.27300000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52400000000000002,"15min":0.52500000000000002},"min":{"1min":0.29799999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.71299999999999997,"5min":0.73999999999999999,"15min":0.82199999999999995},"last":0.71199999999999997}]},{"osd":5,"last update":"Sat Apr 6 12:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51600000000000001,"15min":0.50800000000000001},"min":{"1min":0.31,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.83699999999999997},"last":0.68300000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.53800000000000003,"15min":0.53200000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56299999999999994,"15min":0.55800000000000005},"min":{"1min":0.41599999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.88900000000000001},"last":0.73799999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575046,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1380352,"data_stored":685944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.371,"15min":0.35999999999999999},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.42399999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.38400000000000001,"15min":0.39400000000000002},"min":{"1min":0.245,"5min":0.215,"15min":0.215},"max":{"1min":0.55600000000000005,"5min":0.628,"15min":0.628},"last":0.255}]},{"osd":2,"last update":"Sat Apr 6 12:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.39700000000000002,"15min":0.39400000000000002},"min":{"1min":0.26100000000000001,"5min":0.23899999999999999,"15min":0.19400000000000001},"max":{"1min":0.47499999999999998,"5min":0.624,"15min":0.67600000000000005},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.42399999999999999,"15min":0.42799999999999999},"min":{"1min":0.29999999999999999,"5min":0.255,"15min":0.249},"max":{"1min":0.61699999999999999,"5min":0.63600000000000001,"15min":0.66100000000000003},"last":0.29899999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.442,"15min":0.44500000000000001},"min":{"1min":0.29899999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.51800000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.318},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.41299999999999998,"15min":0.41699999999999998},"min":{"1min":0.29099999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.59399999999999997,"5min":0.69599999999999995,"15min":0.86299999999999999},"last":0.34000000000000002}]},{"osd":4,"last update":"Sat Apr 6 12:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.44500000000000001,"5min":0.40100000000000002,"15min":0.34799999999999998},"max":{"1min":0.75600000000000001,"5min":0.86799999999999999,"15min":0.91100000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.621,"15min":0.621},"min":{"1min":0.47699999999999998,"5min":0.441,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":0.91400000000000003,"15min":0.94899999999999995},"last":0.47999999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.45200000000000001,"5min":0.42999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76200000000000001,"5min":0.91300000000000003,"15min":0.93500000000000005},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.629,"15min":0.626},"min":{"1min":0.48299999999999998,"5min":0.43099999999999999,"15min":0.35099999999999998},"max":{"1min":0.76700000000000002,"5min":0.999,"15min":0.999},"last":0.71499999999999997}]},{"osd":6,"last update":"Sat Apr 6 12:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.74099999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.64200000000000002,"15min":0.64600000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.41799999999999998},"max":{"1min":0.78300000000000003,"5min":0.88700000000000001,"15min":0.97899999999999998},"last":0.65300000000000002}]},{"osd":7,"last update":"Sat Apr 6 12:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.64100000000000001,"15min":0.63600000000000001},"min":{"1min":0.48799999999999999,"5min":0.41899999999999998,"15min":0.38800000000000001},"max":{"1min":0.79600000000000004,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.63500000000000001,"15min":0.63600000000000001},"min":{"1min":0.40999999999999998,"5min":0.40200000000000002,"15min":0.39300000000000002},"max":{"1min":0.75700000000000001,"5min":0.95599999999999996,"15min":0.96799999999999997},"last":0.53500000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575046,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33072,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338768,"statfs":{"total":96636764160,"available":96602898432,"internally_reserved":0,"allocated":1994752,"data_stored":1278512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 12:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38200000000000001,"15min":0.38300000000000001},"min":{"1min":0.23599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.374,"5min":0.39700000000000002,"15min":0.40300000000000002},"min":{"1min":0.25900000000000001,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":0.625,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.40999999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.39900000000000002,"15min":0.40699999999999997},"min":{"1min":0.25700000000000001,"5min":0.23599999999999999,"15min":0.19700000000000001},"max":{"1min":0.59799999999999998,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.26800000000000002,"5min":0.249,"15min":0.24299999999999999},"max":{"1min":0.89900000000000002,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.47099999999999997}]},{"osd":3,"last update":"Sat Apr 6 12:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.40500000000000003,"15min":0.41499999999999998},"min":{"1min":0.28799999999999998,"5min":0.252,"15min":0.222},"max":{"1min":0.56299999999999994,"5min":1.25,"15min":1.25},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.44600000000000001,"15min":0.46000000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.249},"max":{"1min":0.68600000000000005,"5min":0.74099999999999999,"15min":0.748},"last":0.35999999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.60299999999999998,"15min":0.61499999999999999},"min":{"1min":0.38400000000000001,"5min":0.371,"15min":0.35699999999999998},"max":{"1min":1.1100000000000001,"5min":1.514,"15min":1.514},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61599999999999999,"15min":0.626},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.311},"max":{"1min":0.93999999999999995,"5min":1.202,"15min":1.202},"last":0.84699999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.61299999999999999,"15min":0.621},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.40899999999999997,"5min":0.38800000000000001,"15min":0.315},"max":{"1min":0.97399999999999998,"5min":1.006,"15min":1.006},"last":0.82499999999999996}]},{"osd":6,"last update":"Sat Apr 6 12:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.63500000000000001,"15min":0.64700000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.64500000000000002,"15min":0.65700000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.38100000000000001},"max":{"1min":1.052,"5min":1.613,"15min":1.613},"last":0.79500000000000004}]},{"osd":7,"last update":"Sat Apr 6 12:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.64800000000000002,"15min":0.66000000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0820000000000001,"5min":1.54,"15min":1.54},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.64900000000000002,"15min":0.65400000000000003},"min":{"1min":0.42099999999999999,"5min":0.38400000000000001,"15min":0.36699999999999999},"max":{"1min":1.014,"5min":1.498,"15min":1.498},"last":0.54600000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575048,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54604,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317236,"statfs":{"total":96636764160,"available":96580849664,"internally_reserved":0,"allocated":1376256,"data_stored":685874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.374,"15min":0.38100000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51700000000000002,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.39900000000000002,"15min":0.40899999999999997},"min":{"1min":0.246,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.497,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.51000000000000001}]},{"osd":1,"last update":"Sat Apr 6 12:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.40300000000000002,"15min":0.40200000000000002},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.51700000000000002,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.40899999999999997,"15min":0.42899999999999999},"min":{"1min":0.29799999999999999,"5min":0.27100000000000002,"15min":0.23599999999999999},"max":{"1min":0.59799999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.45600000000000002}]},{"osd":3,"last update":"Sat Apr 6 12:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.44400000000000001,"15min":0.44800000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29699999999999999},"max":{"1min":0.55800000000000005,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.495},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.44,"15min":0.436},"min":{"1min":0.33000000000000002,"5min":0.29499999999999998,"15min":0.28100000000000003},"max":{"1min":0.58399999999999996,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.433}]},{"osd":4,"last update":"Sat Apr 6 12:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58399999999999996,"15min":0.61199999999999999},"min":{"1min":0.40200000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.67800000000000005,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.59699999999999998,"15min":0.623},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.753,"5min":0.91200000000000003,"15min":0.92600000000000005},"last":0.80400000000000005}]},{"osd":5,"last update":"Sat Apr 6 12:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.61699999999999999,"15min":0.63300000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.32900000000000001},"max":{"1min":0.73199999999999998,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.60299999999999998,"15min":0.628},"min":{"1min":0.35399999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.83099999999999996,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.71899999999999997}]},{"osd":6,"last update":"Sat Apr 6 12:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60599999999999998,"15min":0.63200000000000001},"min":{"1min":0.46000000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.69399999999999995,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.63700000000000001,"15min":0.65100000000000002},"min":{"1min":0.443,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.77000000000000002,"5min":1.728,"15min":1.728},"last":0.90200000000000002}]},{"osd":7,"last update":"Sat Apr 6 12:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.621,"15min":0.64500000000000002},"min":{"1min":0.42399999999999999,"5min":0.40699999999999997,"15min":0.35599999999999998},"max":{"1min":0.80300000000000005,"5min":0.95699999999999996,"15min":0.95799999999999996},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.64500000000000002,"15min":0.66100000000000003},"min":{"1min":0.41499999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83199999999999996,"5min":1.758,"15min":1.758},"last":0.78000000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575047,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1368064,"data_stored":680273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.35699999999999998,"15min":0.36499999999999999},"min":{"1min":0.248,"5min":0.21099999999999999,"15min":0.191},"max":{"1min":0.44500000000000001,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.375},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.36899999999999999,"15min":0.38200000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.501,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.38800000000000001}]},{"osd":1,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.377,"15min":0.38200000000000001},"min":{"1min":0.23699999999999999,"5min":0.22700000000000001,"15min":0.215},"max":{"1min":0.55800000000000005,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.314},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.40500000000000003,"15min":0.41499999999999998},"min":{"1min":0.28499999999999998,"5min":0.26100000000000001,"15min":0.25700000000000001},"max":{"1min":0.57299999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.33000000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.44400000000000001,"15min":0.45900000000000002},"min":{"1min":0.309,"5min":0.27200000000000002,"15min":0.25700000000000001},"max":{"1min":0.78700000000000003,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.46800000000000003,"15min":0.47599999999999998},"min":{"1min":0.32800000000000001,"5min":0.30199999999999999,"15min":0.29899999999999999},"max":{"1min":0.68799999999999994,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.40600000000000003}]},{"osd":4,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.59499999999999997,"15min":0.60699999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.81299999999999994,"5min":0.88500000000000001,"15min":0.95999999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59199999999999997,"15min":0.621},"min":{"1min":0.39600000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.77600000000000002,"5min":0.90300000000000002,"15min":0.93500000000000005},"last":0.63700000000000001}]},{"osd":5,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.63300000000000001,"15min":0.65700000000000003},"min":{"1min":0.39500000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.82199999999999995,"5min":0.92200000000000004,"15min":0.96699999999999997},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.64600000000000002,"15min":0.66000000000000003},"min":{"1min":0.42799999999999999,"5min":0.38200000000000001,"15min":0.36899999999999999},"max":{"1min":0.83799999999999997,"5min":0.94499999999999995,"15min":0.95599999999999996},"last":0.67100000000000004}]},{"osd":6,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60199999999999998,"15min":0.621},"min":{"1min":0.44,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.72499999999999998,"5min":1.026,"15min":1.026},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.073,"5min":1.716,"15min":1.716},"last":0.53100000000000003}]},{"osd":7,"last update":"Sat Apr 6 12:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.628,"15min":0.64400000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.33300000000000002},"max":{"1min":0.83599999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.64000000000000001,"15min":0.66200000000000003},"min":{"1min":0.40100000000000002,"5min":0.39500000000000002,"15min":0.373},"max":{"1min":0.82599999999999996,"5min":1.002,"15min":1.002},"last":0.59699999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575047,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1388544,"data_stored":687053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54800000000000004,"15min":0.56100000000000005},"min":{"1min":0.33200000000000002,"5min":0.29399999999999998,"15min":0.23899999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.53100000000000003,"15min":0.55900000000000005},"min":{"1min":0.36799999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.753,"5min":0.80000000000000004,"15min":0.82199999999999995},"last":0.59199999999999997}]},{"osd":1,"last update":"Sat Apr 6 12:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.57399999999999995,"15min":0.60499999999999998},"min":{"1min":0.41199999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67000000000000004,"5min":0.76200000000000001,"15min":0.96499999999999997},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.57299999999999995,"15min":0.59199999999999997},"min":{"1min":0.33400000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.754,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.68200000000000005}]},{"osd":2,"last update":"Sat Apr 6 12:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54800000000000004,"15min":0.56599999999999995},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.753,"5min":0.76100000000000001,"15min":0.78400000000000003},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.56499999999999995,"15min":0.58199999999999996},"min":{"1min":0.34999999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.80900000000000005},"last":0.625}]},{"osd":3,"last update":"Sat Apr 6 12:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.54600000000000004,"15min":0.55800000000000005},"min":{"1min":0.39600000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57899999999999996,"15min":0.59699999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.36099999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.80900000000000005},"last":0.67000000000000004}]},{"osd":5,"last update":"Sat Apr 6 12:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.46200000000000002,"15min":0.47799999999999998},"min":{"1min":0.315,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66300000000000003,"5min":0.76300000000000001,"15min":1.196},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.48599999999999999,"15min":0.495},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.64500000000000002,"5min":0.73099999999999998,"15min":0.77000000000000002},"last":0.47499999999999998}]},{"osd":6,"last update":"Sat Apr 6 12:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.497},"min":{"1min":0.307,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.435},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51600000000000001,"15min":0.53200000000000003},"min":{"1min":0.34100000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70499999999999996,"5min":0.71799999999999997,"15min":0.872},"last":0.44800000000000001}]},{"osd":7,"last update":"Sat Apr 6 12:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51500000000000001,"15min":0.53000000000000003},"min":{"1min":0.34100000000000003,"5min":0.311,"15min":0.311},"max":{"1min":0.71599999999999997,"5min":0.79300000000000004,"15min":0.82299999999999995},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54700000000000004,"15min":0.56399999999999995},"min":{"1min":0.36699999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.84799999999999998},"last":0.49299999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575047,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50508,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321332,"statfs":{"total":96636764160,"available":96585043968,"internally_reserved":0,"allocated":1376256,"data_stored":685874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.54400000000000004,"15min":0.56699999999999995},"min":{"1min":0.35199999999999998,"5min":0.32700000000000001,"15min":0.24199999999999999},"max":{"1min":0.68700000000000006,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.53100000000000003,"15min":0.54700000000000004},"min":{"1min":0.39200000000000002,"5min":0.29499999999999998,"15min":0.22500000000000001},"max":{"1min":0.64900000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.36099999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53500000000000003,"15min":0.55300000000000005},"min":{"1min":0.38400000000000001,"5min":0.30099999999999999,"15min":0.26000000000000001},"max":{"1min":0.58399999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.53900000000000003,"15min":0.55200000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.30599999999999999},"max":{"1min":0.70699999999999996,"5min":0.73599999999999999,"15min":0.76200000000000001},"last":0.38100000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.54600000000000004,"15min":0.56000000000000005},"min":{"1min":0.34000000000000002,"5min":0.32500000000000001,"15min":0.21099999999999999},"max":{"1min":0.67200000000000004,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.55200000000000005,"15min":0.56399999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.27400000000000002},"max":{"1min":0.67400000000000004,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.51900000000000002}]},{"osd":3,"last update":"Sat Apr 6 12:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55600000000000005,"15min":0.56699999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.31},"max":{"1min":0.623,"5min":0.874,"15min":0.874},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.57699999999999996,"15min":0.59399999999999997},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.32300000000000001},"max":{"1min":0.71699999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53500000000000003}]},{"osd":4,"last update":"Sat Apr 6 12:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.46899999999999997,"15min":0.46700000000000003},"min":{"1min":0.371,"5min":0.315,"15min":0.28299999999999997},"max":{"1min":0.61399999999999999,"5min":0.755,"15min":0.81200000000000006},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53900000000000003,"15min":0.52700000000000002},"min":{"1min":0.437,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70999999999999996,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.45800000000000002}]},{"osd":6,"last update":"Sat Apr 6 12:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.314,"5min":0.313,"15min":0.28000000000000003},"max":{"1min":0.73499999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.48899999999999999,"15min":0.48699999999999999},"min":{"1min":0.34000000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.68600000000000005,"5min":0.73199999999999998,"15min":0.80500000000000005},"last":0.505}]},{"osd":7,"last update":"Sat Apr 6 12:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.33200000000000002,"5min":0.29799999999999999,"15min":0.252},"max":{"1min":0.69699999999999995,"5min":0.78600000000000003,"15min":0.79100000000000004},"last":0.433},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.60099999999999998,"15min":0.59699999999999998},"min":{"1min":0.48199999999999998,"5min":0.41399999999999998,"15min":0.35199999999999998},"max":{"1min":0.76000000000000001,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.58499999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:28:45.213 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-06T12:28:45.213 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-06T12:28:45.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.380+0000 7f4ca9220700 1 Processor -- start 2024-04-06T12:28:45.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.381+0000 7f4ca9220700 1 -- start start 2024-04-06T12:28:45.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.382+0000 7f4ca9220700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4ca406ae40 0x7f4ca415bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:45.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.382+0000 7f4ca9220700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca415f9e0 con 0x7f4ca415c100 2024-04-06T12:28:45.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.382+0000 7f4ca9220700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca415f0d0 con 0x7f4ca4158fe0 2024-04-06T12:28:45.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.382+0000 7f4ca9220700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca415c8f0 con 0x7f4ca406ae40 2024-04-06T12:28:45.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.382+0000 7f4ca259c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4ca406ae40 0x7f4ca415bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:45.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.382+0000 7f4ca359e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4ca415c100 0x7f4ca40ceca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51514/0 (socket says 172.21.15.189:51514) 2024-04-06T12:28:45.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.382+0000 7f4ca359e700 1 -- 172.21.15.189:0/78116913 learned_addr learned my addr 172.21.15.189:0/78116913 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:45.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.382+0000 7f4ca1d9b700 1 -- 172.21.15.189:0/78116913 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770236334 0 0) 0x7f4ca415f9e0 con 0x7f4ca415c100 2024-04-06T12:28:45.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.382+0000 7f4ca1d9b700 1 -- 172.21.15.189:0/78116913 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c8c0035b0 con 0x7f4ca415c100 2024-04-06T12:28:45.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.382+0000 7f4ca259c700 1 -- 172.21.15.189:0/78116913 >> v1:172.21.15.192:6789/0 conn(0x7f4ca4158fe0 legacy=0x7f4ca406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:45.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.382+0000 7f4ca259c700 1 -- 172.21.15.189:0/78116913 >> v1:172.21.15.189:6789/0 conn(0x7f4ca415c100 legacy=0x7f4ca40ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:45.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.382+0000 7f4ca259c700 1 -- 172.21.15.189:0/78116913 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca415f390 con 0x7f4ca406ae40 2024-04-06T12:28:45.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.382+0000 7f4ca259c700 1 --2- 172.21.15.189:0/78116913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca406ae40 0x7f4ca415bbc0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f4c94009840 tx=0x7f4c94009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=158a9f37298b82a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:45.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.383+0000 7f4ca1d9b700 1 -- 172.21.15.189:0/78116913 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c94011830 con 0x7f4ca406ae40 2024-04-06T12:28:45.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.383+0000 7f4ca1d9b700 1 -- 172.21.15.189:0/78116913 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c94011990 con 0x7f4ca406ae40 2024-04-06T12:28:45.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.383+0000 7f4ca1d9b700 1 -- 172.21.15.189:0/78116913 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c94011d20 con 0x7f4ca406ae40 2024-04-06T12:28:45.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.383+0000 7f4ca9220700 1 -- 172.21.15.189:0/78116913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca406ae40 msgr2=0x7f4ca415bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:45.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.383+0000 7f4ca9220700 1 --2- 172.21.15.189:0/78116913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca406ae40 0x7f4ca415bbc0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f4c94009840 tx=0x7f4c94009b50 comp rx=0 tx=0).stop 2024-04-06T12:28:45.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.383+0000 7f4ca9220700 1 -- 172.21.15.189:0/78116913 shutdown_connections 2024-04-06T12:28:45.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.383+0000 7f4ca9220700 1 --2- 172.21.15.189:0/78116913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca406ae40 0x7f4ca415bbc0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:45.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.383+0000 7f4ca9220700 1 -- 172.21.15.189:0/78116913 >> 172.21.15.189:0/78116913 conn(0x7f4ca40d00b0 msgr2=0x7f4ca40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:45.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.383+0000 7f4ca9220700 1 -- 172.21.15.189:0/78116913 shutdown_connections 2024-04-06T12:28:45.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.383+0000 7f4ca9220700 1 -- 172.21.15.189:0/78116913 wait complete. 2024-04-06T12:28:45.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.385+0000 7f4ca9220700 1 Processor -- start 2024-04-06T12:28:45.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.385+0000 7f4ca9220700 1 -- start start 2024-04-06T12:28:45.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.386+0000 7f4ca9220700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ca406ae40 0x7f4ca40c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:45.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.386+0000 7f4ca9220700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ca4158fe0 0x7f4ca40cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:45.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.386+0000 7f4ca9220700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca415c100 0x7f4ca412e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:45.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.386+0000 7f4ca9220700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca412c3e0 con 0x7f4ca415c100 2024-04-06T12:28:45.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.386+0000 7f4ca9220700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca412f2c0 con 0x7f4ca4158fe0 2024-04-06T12:28:45.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.386+0000 7f4ca9220700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca40c5470 con 0x7f4ca406ae40 2024-04-06T12:28:45.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.386+0000 7f4ca359e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca415c100 0x7f4ca412e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:45.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.386+0000 7f4ca359e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca415c100 0x7f4ca412e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53470/0 (socket says 172.21.15.189:53470) 2024-04-06T12:28:45.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.386+0000 7f4ca359e700 1 -- 172.21.15.189:0/3136702800 learned_addr learned my addr 172.21.15.189:0/3136702800 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:45.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.386+0000 7f4ca2d9d700 1 --2- 172.21.15.189:0/3136702800 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ca406ae40 0x7f4ca40c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:45.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.386+0000 7f4ca259c700 1 --2- 172.21.15.189:0/3136702800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ca4158fe0 0x7f4ca40cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:45.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.386+0000 7f4ca359e700 1 -- 172.21.15.189:0/3136702800 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ca406ae40 msgr2=0x7f4ca40c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:45.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.386+0000 7f4ca359e700 1 --2- 172.21.15.189:0/3136702800 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ca406ae40 0x7f4ca40c4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:45.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.386+0000 7f4ca359e700 1 -- 172.21.15.189:0/3136702800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ca4158fe0 msgr2=0x7f4ca40cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:45.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.386+0000 7f4ca359e700 1 --2- 172.21.15.189:0/3136702800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ca4158fe0 0x7f4ca40cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:45.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.386+0000 7f4ca359e700 1 -- 172.21.15.189:0/3136702800 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c940094a0 con 0x7f4ca415c100 2024-04-06T12:28:45.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.387+0000 7f4ca359e700 1 --2- 172.21.15.189:0/3136702800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca415c100 0x7f4ca412e440 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00caa0 tx=0x7f4c9c00cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:45.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.387+0000 7f4c8b7fe700 1 -- 172.21.15.189:0/3136702800 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c00a980 con 0x7f4ca415c100 2024-04-06T12:28:45.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.387+0000 7f4c8b7fe700 1 -- 172.21.15.189:0/3136702800 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c00aae0 con 0x7f4ca415c100 2024-04-06T12:28:45.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.387+0000 7f4ca9220700 1 -- 172.21.15.189:0/3136702800 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca4141cb0 con 0x7f4ca415c100 2024-04-06T12:28:45.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.387+0000 7f4c8b7fe700 1 -- 172.21.15.189:0/3136702800 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c00ae70 con 0x7f4ca415c100 2024-04-06T12:28:45.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.387+0000 7f4ca9220700 1 -- 172.21.15.189:0/3136702800 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca4142270 con 0x7f4ca415c100 2024-04-06T12:28:45.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.388+0000 7f4ca9220700 1 -- 172.21.15.189:0/3136702800 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca4067bb0 con 0x7f4ca415c100 2024-04-06T12:28:45.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.388+0000 7f4c8b7fe700 1 -- 172.21.15.189:0/3136702800 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f4c9c01b3b0 con 0x7f4ca415c100 2024-04-06T12:28:45.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.389+0000 7f4c8b7fe700 1 --2- 172.21.15.189:0/3136702800 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4c9006adb0 0x7f4c9006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:45.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.389+0000 7f4c8b7fe700 1 -- 172.21.15.189:0/3136702800 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4c9c089110 con 0x7f4ca415c100 2024-04-06T12:28:45.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.389+0000 7f4ca2d9d700 1 --2- 172.21.15.189:0/3136702800 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4c9006adb0 0x7f4c9006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:45.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.389+0000 7f4ca2d9d700 1 --2- 172.21.15.189:0/3136702800 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4c9006adb0 0x7f4c9006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4ca41f4480 tx=0x7f4c98009040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:45.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.393+0000 7f4c8b7fe700 1 -- 172.21.15.189:0/3136702800 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4c9c0541f0 con 0x7f4ca415c100 2024-04-06T12:28:45.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.667+0000 7f4ca9220700 1 -- 172.21.15.189:0/3136702800 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f4ca4074990 con 0x7f4ca415c100 2024-04-06T12:28:45.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.671+0000 7f4c8b7fe700 1 -- 172.21.15.189:0/3136702800 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+0 (secure 0 0 0) 0x7f4c9c057810 con 0x7f4ca415c100 2024-04-06T12:28:45.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.674+0000 7f4ca9220700 1 -- 172.21.15.189:0/3136702800 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4c9006adb0 msgr2=0x7f4c9006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:45.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.674+0000 7f4ca9220700 1 --2- 172.21.15.189:0/3136702800 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4c9006adb0 0x7f4c9006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4ca41f4480 tx=0x7f4c98009040 comp rx=0 tx=0).stop 2024-04-06T12:28:45.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.674+0000 7f4ca9220700 1 -- 172.21.15.189:0/3136702800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca415c100 msgr2=0x7f4ca412e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:45.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.674+0000 7f4ca9220700 1 --2- 172.21.15.189:0/3136702800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca415c100 0x7f4ca412e440 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00caa0 tx=0x7f4c9c00cdb0 comp rx=0 tx=0).stop 2024-04-06T12:28:45.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.674+0000 7f4ca9220700 1 -- 172.21.15.189:0/3136702800 shutdown_connections 2024-04-06T12:28:45.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.674+0000 7f4ca9220700 1 --2- 172.21.15.189:0/3136702800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca415c100 0x7f4ca412e440 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:45.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.674+0000 7f4ca9220700 1 --2- 172.21.15.189:0/3136702800 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ca406ae40 0x7f4ca40c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:45.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.674+0000 7f4ca9220700 1 --2- 172.21.15.189:0/3136702800 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4c9006adb0 0x7f4c9006d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:45.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.674+0000 7f4ca9220700 1 --2- 172.21.15.189:0/3136702800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ca4158fe0 0x7f4ca40cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:45.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.674+0000 7f4ca9220700 1 -- 172.21.15.189:0/3136702800 >> 172.21.15.189:0/3136702800 conn(0x7f4ca40d00b0 msgr2=0x7f4ca406f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:45.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.674+0000 7f4ca9220700 1 -- 172.21.15.189:0/3136702800 shutdown_connections 2024-04-06T12:28:45.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.674+0000 7f4ca9220700 1 -- 172.21.15.189:0/3136702800 wait complete. 2024-04-06T12:28:45.672 INFO:teuthology.orchestra.run.smithi189.stderr:updated caps for client.0 2024-04-06T12:28:45.684 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:28:45.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.808+0000 7f3fc6917700 1 Processor -- start 2024-04-06T12:28:45.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.808+0000 7f3fc6917700 1 -- start start 2024-04-06T12:28:45.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.809+0000 7f3fc6917700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3fc00b3e20 0x7f3fc015e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:45.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.809+0000 7f3fc6917700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc00b3990 con 0x7f3fc015d7b0 2024-04-06T12:28:45.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.809+0000 7f3fc6917700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc00b3060 con 0x7f3fc006ae40 2024-04-06T12:28:45.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.809+0000 7f3fc6917700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fc00b4230 con 0x7f3fc00b3e20 2024-04-06T12:28:45.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.809+0000 7f3fc4eb4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3fc00b3e20 0x7f3fc015e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:45.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.809+0000 7f3fbffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3fc015d7b0 0x7f3fc006a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51520/0 (socket says 172.21.15.189:51520) 2024-04-06T12:28:45.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.809+0000 7f3fbffff700 1 -- 172.21.15.189:0/752946751 learned_addr learned my addr 172.21.15.189:0/752946751 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:45.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.809+0000 7f3fc4eb4700 1 --2- 172.21.15.189:0/752946751 >> v2:172.21.15.189:3300/0 conn(0x7f3fc00b3e20 0x7f3fc015e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53478/0 (socket says 172.21.15.189:53478) 2024-04-06T12:28:45.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.810+0000 7f3fc4eb4700 1 -- 172.21.15.189:0/752946751 >> v1:172.21.15.192:6789/0 conn(0x7f3fc006ae40 legacy=0x7f3fc015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:45.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.810+0000 7f3fc4eb4700 1 -- 172.21.15.189:0/752946751 >> v1:172.21.15.189:6789/0 conn(0x7f3fc015d7b0 legacy=0x7f3fc006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:45.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.810+0000 7f3fc4eb4700 1 -- 172.21.15.189:0/752946751 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc00b3340 con 0x7f3fc00b3e20 2024-04-06T12:28:45.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.810+0000 7f3fbeffd700 1 -- 172.21.15.189:0/752946751 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308566454 0 0) 0x7f3fc00b3990 con 0x7f3fc015d7b0 2024-04-06T12:28:45.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.810+0000 7f3fc4eb4700 1 --2- 172.21.15.189:0/752946751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fc00b3e20 0x7f3fc015e500 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8009d90 tx=0x7f3fb800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826fa9e158378f60 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:45.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.810+0000 7f3fbeffd700 1 -- 172.21.15.189:0/752946751 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb800e070 con 0x7f3fc00b3e20 2024-04-06T12:28:45.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.810+0000 7f3fbeffd700 1 -- 172.21.15.189:0/752946751 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb800bb90 con 0x7f3fc00b3e20 2024-04-06T12:28:45.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.810+0000 7f3fbeffd700 1 -- 172.21.15.189:0/752946751 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8003a20 con 0x7f3fc00b3e20 2024-04-06T12:28:45.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.810+0000 7f3fc6917700 1 -- 172.21.15.189:0/752946751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fc00b3e20 msgr2=0x7f3fc015e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:45.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.810+0000 7f3fc6917700 1 --2- 172.21.15.189:0/752946751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fc00b3e20 0x7f3fc015e500 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8009d90 tx=0x7f3fb800b330 comp rx=0 tx=0).stop 2024-04-06T12:28:45.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.811+0000 7f3fc6917700 1 -- 172.21.15.189:0/752946751 shutdown_connections 2024-04-06T12:28:45.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.811+0000 7f3fc6917700 1 --2- 172.21.15.189:0/752946751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fc00b3e20 0x7f3fc015e500 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:45.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.811+0000 7f3fc6917700 1 -- 172.21.15.189:0/752946751 >> 172.21.15.189:0/752946751 conn(0x7f3fc00d00b0 msgr2=0x7f3fc00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:45.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.811+0000 7f3fc6917700 1 -- 172.21.15.189:0/752946751 shutdown_connections 2024-04-06T12:28:45.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.811+0000 7f3fc6917700 1 -- 172.21.15.189:0/752946751 wait complete. 2024-04-06T12:28:45.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.812+0000 7f3fc6917700 1 Processor -- start 2024-04-06T12:28:45.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.812+0000 7f3fc6917700 1 -- start start 2024-04-06T12:28:45.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.813+0000 7f3fc6917700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fc006ae40 0x7f3fc00cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:45.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.813+0000 7f3fc6917700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fc00b3e20 0x7f3fc00ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:45.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.813+0000 7f3fc6917700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fc015d7b0 0x7f3fc01f63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:45.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.813+0000 7f3fc6917700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc00cd310 con 0x7f3fc00b3e20 2024-04-06T12:28:45.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.813+0000 7f3fc6917700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc00cd5e0 con 0x7f3fc015d7b0 2024-04-06T12:28:45.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.813+0000 7f3fc6917700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc00cd8c0 con 0x7f3fc006ae40 2024-04-06T12:28:45.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.813+0000 7f3fbf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fc00b3e20 0x7f3fc00ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:45.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.813+0000 7f3fbf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fc00b3e20 0x7f3fc00ccdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53482/0 (socket says 172.21.15.189:53482) 2024-04-06T12:28:45.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.813+0000 7f3fbf7fe700 1 -- 172.21.15.189:0/3927758844 learned_addr learned my addr 172.21.15.189:0/3927758844 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:45.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.814+0000 7f3fbffff700 1 --2- 172.21.15.189:0/3927758844 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fc006ae40 0x7f3fc00cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:45.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.814+0000 7f3fc4eb4700 1 --2- 172.21.15.189:0/3927758844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fc015d7b0 0x7f3fc01f63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:45.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.814+0000 7f3fbf7fe700 1 -- 172.21.15.189:0/3927758844 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fc006ae40 msgr2=0x7f3fc00cc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:45.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.814+0000 7f3fbf7fe700 1 --2- 172.21.15.189:0/3927758844 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fc006ae40 0x7f3fc00cc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:45.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.814+0000 7f3fbf7fe700 1 -- 172.21.15.189:0/3927758844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fc015d7b0 msgr2=0x7f3fc01f63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:45.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.814+0000 7f3fbf7fe700 1 --2- 172.21.15.189:0/3927758844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fc015d7b0 0x7f3fc01f63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:45.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.814+0000 7f3fbf7fe700 1 -- 172.21.15.189:0/3927758844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb8009a70 con 0x7f3fc00b3e20 2024-04-06T12:28:45.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.814+0000 7f3fbf7fe700 1 --2- 172.21.15.189:0/3927758844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fc00b3e20 0x7f3fc00ccdd0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f3fb400c4a0 tx=0x7f3fb400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:45.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.814+0000 7f3fbcff9700 1 -- 172.21.15.189:0/3927758844 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb40054e0 con 0x7f3fc00b3e20 2024-04-06T12:28:45.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.814+0000 7f3fbcff9700 1 -- 172.21.15.189:0/3927758844 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb4003680 con 0x7f3fc00b3e20 2024-04-06T12:28:45.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.814+0000 7f3fbcff9700 1 -- 172.21.15.189:0/3927758844 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4003a30 con 0x7f3fc00b3e20 2024-04-06T12:28:45.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.814+0000 7f3fc6917700 1 -- 172.21.15.189:0/3927758844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc00cdba0 con 0x7f3fc00b3e20 2024-04-06T12:28:45.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.814+0000 7f3fc6917700 1 -- 172.21.15.189:0/3927758844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc00ce160 con 0x7f3fc00b3e20 2024-04-06T12:28:45.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.815+0000 7f3fc6917700 1 -- 172.21.15.189:0/3927758844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc00ce380 con 0x7f3fc00b3e20 2024-04-06T12:28:45.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.816+0000 7f3fbcff9700 1 -- 172.21.15.189:0/3927758844 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f3fb4003b90 con 0x7f3fc00b3e20 2024-04-06T12:28:45.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.816+0000 7f3fbcff9700 1 --2- 172.21.15.189:0/3927758844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3fac06adb0 0x7f3fac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:45.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.816+0000 7f3fbcff9700 1 -- 172.21.15.189:0/3927758844 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3fb40886b0 con 0x7f3fc00b3e20 2024-04-06T12:28:45.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.816+0000 7f3fbffff700 1 --2- 172.21.15.189:0/3927758844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3fac06adb0 0x7f3fac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:45.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.816+0000 7f3fbffff700 1 --2- 172.21.15.189:0/3927758844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3fac06adb0 0x7f3fac06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0000f60 tx=0x7f3fb0009040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:45.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:45.820+0000 7f3fbcff9700 1 -- 172.21.15.189:0/3927758844 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3fb4053640 con 0x7f3fc00b3e20 2024-04-06T12:28:46.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.058+0000 7f3fc6917700 1 -- 172.21.15.189:0/3927758844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fc0067bb0 con 0x7f3fc00b3e20 2024-04-06T12:28:46.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.059+0000 7f3fbcff9700 1 -- 172.21.15.189:0/3927758844 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f3fb4056c60 con 0x7f3fc00b3e20 2024-04-06T12:28:46.056 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:46.056 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":46,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6389,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/17345288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":17345288},{"type":"v1","addr":"172.21.15.192:6835","nonce":17345288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":7078,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/440618958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":440618958},{"type":"v1","addr":"172.21.15.192:6834","nonce":440618958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":7569,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3391245238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3391245238},{"type":"v1","addr":"172.21.15.189:6836","nonce":3391245238}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:28:40.659968+0000","modified":"2024-04-06T12:28:43.668129+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7572":{"gid":7572,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/2225868923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2225868923},{"type":"v1","addr":"172.21.15.189:6837","nonce":2225868923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-04-06T12:28:46.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.061+0000 7f3fc6917700 1 -- 172.21.15.189:0/3927758844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3fac06adb0 msgr2=0x7f3fac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:46.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.061+0000 7f3fc6917700 1 --2- 172.21.15.189:0/3927758844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3fac06adb0 0x7f3fac06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0000f60 tx=0x7f3fb0009040 comp rx=0 tx=0).stop 2024-04-06T12:28:46.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.061+0000 7f3fc6917700 1 -- 172.21.15.189:0/3927758844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fc00b3e20 msgr2=0x7f3fc00ccdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:46.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.061+0000 7f3fc6917700 1 --2- 172.21.15.189:0/3927758844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fc00b3e20 0x7f3fc00ccdd0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f3fb400c4a0 tx=0x7f3fb400c860 comp rx=0 tx=0).stop 2024-04-06T12:28:46.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.061+0000 7f3fc6917700 1 -- 172.21.15.189:0/3927758844 shutdown_connections 2024-04-06T12:28:46.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.061+0000 7f3fc6917700 1 --2- 172.21.15.189:0/3927758844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3fac06adb0 0x7f3fac06d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.061+0000 7f3fc6917700 1 --2- 172.21.15.189:0/3927758844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fc015d7b0 0x7f3fc01f63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.061+0000 7f3fc6917700 1 --2- 172.21.15.189:0/3927758844 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fc006ae40 0x7f3fc00cc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.061+0000 7f3fc6917700 1 --2- 172.21.15.189:0/3927758844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fc00b3e20 0x7f3fc00ccdd0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.061+0000 7f3fc6917700 1 -- 172.21.15.189:0/3927758844 >> 172.21.15.189:0/3927758844 conn(0x7f3fc00d00b0 msgr2=0x7f3fc006eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:46.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.061+0000 7f3fc6917700 1 -- 172.21.15.189:0/3927758844 shutdown_connections 2024-04-06T12:28:46.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.061+0000 7f3fc6917700 1 -- 172.21.15.189:0/3927758844 wait complete. 2024-04-06T12:28:46.059 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 46 2024-04-06T12:28:46.230 INFO:tasks.ceph.mds.c.smithi189.stderr:2024-04-06T12:28:46.233+0000 7fbfbbd71700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T12:28:46.473 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 46, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-06T12:28:40.659968+0000', 'modified': '2024-04-06T12:28:43.668129+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7572}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7572': {'gid': 7572, 'name': 'c', 'rank': 0, 'incarnation': 43, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.189:6837/2225868923', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6835', 'nonce': 2225868923}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 2225868923}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-06T12:28:46.473 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-06T12:28:46.473 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-06T12:28:46.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.602+0000 7ff345371700 1 Processor -- start 2024-04-06T12:28:46.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.602+0000 7ff345371700 1 -- start start 2024-04-06T12:28:46.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.603+0000 7ff345371700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff340146dd0 0x7ff34006d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.603+0000 7ff345371700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff3400cedf0 0x7ff3400cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.603+0000 7ff345371700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3400cf760 con 0x7ff3400d17a0 2024-04-06T12:28:46.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.603+0000 7ff345371700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff34006beb0 con 0x7ff3400cedf0 2024-04-06T12:28:46.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.603+0000 7ff345371700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34006c0a0 con 0x7ff340146dd0 2024-04-06T12:28:46.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.603+0000 7ff33effd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff3400d17a0 0x7ff34006b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51524/0 (socket says 172.21.15.189:51524) 2024-04-06T12:28:46.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.603+0000 7ff33effd700 1 -- 172.21.15.189:0/2218947798 learned_addr learned my addr 172.21.15.189:0/2218947798 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:46.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33dffb700 1 -- 172.21.15.189:0/2218947798 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277284185 0 0) 0x7ff3400cf760 con 0x7ff3400d17a0 2024-04-06T12:28:46.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33dffb700 1 -- 172.21.15.189:0/2218947798 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3280035b0 con 0x7ff3400d17a0 2024-04-06T12:28:46.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33e7fc700 1 --2- 172.21.15.189:0/2218947798 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff340146dd0 0x7ff34006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33f7fe700 1 --2- 172.21.15.189:0/2218947798 >> v2:172.21.15.192:3300/0 conn(0x7ff3400cedf0 0x7ff3400cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33dffb700 1 -- 172.21.15.189:0/2218947798 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2636387718 0 0) 0x7ff3280035b0 con 0x7ff3400d17a0 2024-04-06T12:28:46.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33dffb700 1 -- 172.21.15.189:0/2218947798 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff340146dd0 msgr2=0x7ff34006d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:46.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33dffb700 1 --2- 172.21.15.189:0/2218947798 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff340146dd0 0x7ff34006d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33dffb700 1 -- 172.21.15.189:0/2218947798 >> v2:172.21.15.192:3300/0 conn(0x7ff3400cedf0 msgr2=0x7ff3400cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:46.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33dffb700 1 --2- 172.21.15.189:0/2218947798 >> v2:172.21.15.192:3300/0 conn(0x7ff3400cedf0 0x7ff3400cf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33dffb700 1 -- 172.21.15.189:0/2218947798 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34006c380 con 0x7ff3400d17a0 2024-04-06T12:28:46.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33dffb700 1 -- 172.21.15.189:0/2218947798 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7ff334001670 con 0x7ff3400d17a0 2024-04-06T12:28:46.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33dffb700 1 -- 172.21.15.189:0/2218947798 >> v1:172.21.15.189:6789/0 conn(0x7ff3400d17a0 legacy=0x7ff34006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:46.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33dffb700 1 --2- 172.21.15.189:0/2218947798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff328003a00 0x7ff328003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33dffb700 1 --2- 172.21.15.189:0/2218947798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff328004850 0x7ff328004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33dffb700 1 --2- 172.21.15.189:0/2218947798 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff340146dd0 0x7ff328005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33dffb700 1 -- 172.21.15.189:0/2218947798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34006c0a0 con 0x7ff328003a00 2024-04-06T12:28:46.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33dffb700 1 -- 172.21.15.189:0/2218947798 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32800ba10 con 0x7ff328004850 2024-04-06T12:28:46.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33dffb700 1 -- 172.21.15.189:0/2218947798 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff32800bb80 con 0x7ff340146dd0 2024-04-06T12:28:46.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33f7fe700 1 --2- 172.21.15.189:0/2218947798 >> v2:172.21.15.192:3300/0 conn(0x7ff3400cedf0 0x7ff3400cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:46.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33effd700 1 --2- 172.21.15.189:0/2218947798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff328003a00 0x7ff328003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33e7fc700 1 --2- 172.21.15.189:0/2218947798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff328004850 0x7ff328004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33f7fe700 1 --2- 172.21.15.189:0/2218947798 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff340146dd0 0x7ff328005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33effd700 1 -- 172.21.15.189:0/2218947798 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff340146dd0 msgr2=0x7ff328005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:46.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33effd700 1 --2- 172.21.15.189:0/2218947798 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff340146dd0 0x7ff328005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33effd700 1 -- 172.21.15.189:0/2218947798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff328004850 msgr2=0x7ff328004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:46.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33effd700 1 --2- 172.21.15.189:0/2218947798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff328004850 0x7ff328004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33effd700 1 -- 172.21.15.189:0/2218947798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3280035b0 con 0x7ff328003a00 2024-04-06T12:28:46.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33e7fc700 1 --2- 172.21.15.189:0/2218947798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff328004850 0x7ff328004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:46.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.604+0000 7ff33effd700 1 --2- 172.21.15.189:0/2218947798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff328003a00 0x7ff328003e20 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7ff33400c2f0 tx=0x7ff328005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1cee949a76022b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:46.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.605+0000 7ff33dffb700 1 -- 172.21.15.189:0/2218947798 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33400ce30 con 0x7ff328003a00 2024-04-06T12:28:46.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.605+0000 7ff33dffb700 1 -- 172.21.15.189:0/2218947798 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff334005960 con 0x7ff328003a00 2024-04-06T12:28:46.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.605+0000 7ff33dffb700 1 -- 172.21.15.189:0/2218947798 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff334005d10 con 0x7ff328003a00 2024-04-06T12:28:46.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.605+0000 7ff345371700 1 -- 172.21.15.189:0/2218947798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff328003a00 msgr2=0x7ff328003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:46.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.605+0000 7ff345371700 1 --2- 172.21.15.189:0/2218947798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff328003a00 0x7ff328003e20 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7ff33400c2f0 tx=0x7ff328005880 comp rx=0 tx=0).stop 2024-04-06T12:28:46.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.605+0000 7ff33f7fe700 1 -- 172.21.15.189:0/2218947798 reap_dead start 2024-04-06T12:28:46.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.605+0000 7ff345371700 1 -- 172.21.15.189:0/2218947798 shutdown_connections 2024-04-06T12:28:46.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.605+0000 7ff345371700 1 -- 172.21.15.189:0/2218947798 >> 172.21.15.189:0/2218947798 conn(0x7ff3400b2db0 msgr2=0x7ff3400b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:46.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.605+0000 7ff345371700 1 -- 172.21.15.189:0/2218947798 shutdown_connections 2024-04-06T12:28:46.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.605+0000 7ff345371700 1 -- 172.21.15.189:0/2218947798 wait complete. 2024-04-06T12:28:46.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.607+0000 7ff345371700 1 Processor -- start 2024-04-06T12:28:46.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.607+0000 7ff345371700 1 -- start start 2024-04-06T12:28:46.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.608+0000 7ff345371700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff328003a00 0x7ff340157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.608+0000 7ff345371700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff34015a190 0x7ff3401f6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.608+0000 7ff345371700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3400c65d0 0x7ff3400ad030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.608+0000 7ff345371700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3400af850 con 0x7ff328003a00 2024-04-06T12:28:46.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.608+0000 7ff345371700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3401f6eb0 con 0x7ff3400c65d0 2024-04-06T12:28:46.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.608+0000 7ff345371700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3401f7190 con 0x7ff34015a190 2024-04-06T12:28:46.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.608+0000 7ff33effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff328003a00 0x7ff340157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.608+0000 7ff33effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff328003a00 0x7ff340157ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53504/0 (socket says 172.21.15.189:53504) 2024-04-06T12:28:46.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.608+0000 7ff33effd700 1 -- 172.21.15.189:0/1090212326 learned_addr learned my addr 172.21.15.189:0/1090212326 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:46.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.608+0000 7ff33e7fc700 1 --2- 172.21.15.189:0/1090212326 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff34015a190 0x7ff3401f6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.608+0000 7ff33f7fe700 1 --2- 172.21.15.189:0/1090212326 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3400c65d0 0x7ff3400ad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.608+0000 7ff33effd700 1 -- 172.21.15.189:0/1090212326 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff34015a190 msgr2=0x7ff3401f6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:46.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.608+0000 7ff33effd700 1 --2- 172.21.15.189:0/1090212326 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff34015a190 0x7ff3401f6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.608+0000 7ff33effd700 1 -- 172.21.15.189:0/1090212326 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3400c65d0 msgr2=0x7ff3400ad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:46.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.608+0000 7ff33effd700 1 --2- 172.21.15.189:0/1090212326 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3400c65d0 0x7ff3400ad030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.608+0000 7ff33effd700 1 -- 172.21.15.189:0/1090212326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3400af4e0 con 0x7ff328003a00 2024-04-06T12:28:46.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.608+0000 7ff33e7fc700 1 --2- 172.21.15.189:0/1090212326 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff34015a190 0x7ff3401f6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:46.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.608+0000 7ff33effd700 1 --2- 172.21.15.189:0/1090212326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff328003a00 0x7ff340157ff0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7ff33400c8e0 tx=0x7ff334001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:46.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.608+0000 7ff3277fe700 1 -- 172.21.15.189:0/1090212326 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff334004870 con 0x7ff328003a00 2024-04-06T12:28:46.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.609+0000 7ff3277fe700 1 -- 172.21.15.189:0/1090212326 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3340049d0 con 0x7ff328003a00 2024-04-06T12:28:46.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.609+0000 7ff345371700 1 -- 172.21.15.189:0/1090212326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3401f6a80 con 0x7ff328003a00 2024-04-06T12:28:46.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.609+0000 7ff3277fe700 1 -- 172.21.15.189:0/1090212326 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff334004d80 con 0x7ff328003a00 2024-04-06T12:28:46.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.609+0000 7ff345371700 1 -- 172.21.15.189:0/1090212326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3400c6c60 con 0x7ff328003a00 2024-04-06T12:28:46.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.610+0000 7ff3277fe700 1 -- 172.21.15.189:0/1090212326 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7ff3340070d0 con 0x7ff328003a00 2024-04-06T12:28:46.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.610+0000 7ff3277fe700 1 --2- 172.21.15.189:0/1090212326 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff32c06adb0 0x7ff32c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.610+0000 7ff3277fe700 1 -- 172.21.15.189:0/1090212326 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff33408ac00 con 0x7ff328003a00 2024-04-06T12:28:46.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.610+0000 7ff33e7fc700 1 --2- 172.21.15.189:0/1090212326 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff32c06adb0 0x7ff32c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.611+0000 7ff33e7fc700 1 --2- 172.21.15.189:0/1090212326 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff32c06adb0 0x7ff32c06d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff34015a790 tx=0x7ff330008040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:46.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.612+0000 7ff351525b80 1 Processor -- start 2024-04-06T12:28:46.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.613+0000 7ff351525b80 1 -- start start 2024-04-06T12:28:46.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.613+0000 7ff351525b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b4fc20 0x562221b53a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.613+0000 7ff351525b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b54fc0 0x562221b2b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.613+0000 7ff351525b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b2a980 0x562221b2bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.613+0000 7ff351525b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562221a84940 con 0x562221b2a980 2024-04-06T12:28:46.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.613+0000 7ff351525b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x562221a82300 con 0x562221b54fc0 2024-04-06T12:28:46.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.613+0000 7ff351525b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x562221a7dbf0 con 0x562221b4fc20 2024-04-06T12:28:46.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.613+0000 7ff33f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b2a980 0x562221b2bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.613+0000 7ff33f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b2a980 0x562221b2bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53514/0 (socket says 172.21.15.189:53514) 2024-04-06T12:28:46.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.613+0000 7ff33f7fe700 1 -- 172.21.15.189:0/3115293822 learned_addr learned my addr 172.21.15.189:0/3115293822 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:46.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.614+0000 7ff33effd700 1 --2- 172.21.15.189:0/3115293822 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b4fc20 0x562221b53a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.614+0000 7ff33e7fc700 1 --2- 172.21.15.189:0/3115293822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b54fc0 0x562221b2b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.614+0000 7ff33f7fe700 1 -- 172.21.15.189:0/3115293822 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b4fc20 msgr2=0x562221b53a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:46.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.614+0000 7ff33f7fe700 1 --2- 172.21.15.189:0/3115293822 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b4fc20 0x562221b53a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.614+0000 7ff33f7fe700 1 -- 172.21.15.189:0/3115293822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b54fc0 msgr2=0x562221b2b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:46.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.614+0000 7ff33f7fe700 1 --2- 172.21.15.189:0/3115293822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b54fc0 0x562221b2b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.614+0000 7ff33f7fe700 1 -- 172.21.15.189:0/3115293822 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622218abb10 con 0x562221b2a980 2024-04-06T12:28:46.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.614+0000 7ff33f7fe700 1 --2- 172.21.15.189:0/3115293822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b2a980 0x562221b2bb30 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7ff3400ced00 tx=0x7ff338009c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6ad6157a6f14613 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:46.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.614+0000 7ff324ff9700 1 -- 172.21.15.189:0/3115293822 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338004310 con 0x562221b2a980 2024-04-06T12:28:46.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.614+0000 7ff324ff9700 1 -- 172.21.15.189:0/3115293822 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33800f070 con 0x562221b2a980 2024-04-06T12:28:46.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.614+0000 7ff324ff9700 1 -- 172.21.15.189:0/3115293822 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338004690 con 0x562221b2a980 2024-04-06T12:28:46.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.615+0000 7ff351525b80 1 -- 172.21.15.189:0/3115293822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b2a980 msgr2=0x562221b2bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:46.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.615+0000 7ff351525b80 1 --2- 172.21.15.189:0/3115293822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b2a980 0x562221b2bb30 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7ff3400ced00 tx=0x7ff338009c40 comp rx=0 tx=0).stop 2024-04-06T12:28:46.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.615+0000 7ff351525b80 1 -- 172.21.15.189:0/3115293822 shutdown_connections 2024-04-06T12:28:46.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.615+0000 7ff351525b80 1 --2- 172.21.15.189:0/3115293822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b2a980 0x562221b2bb30 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.615+0000 7ff351525b80 1 --2- 172.21.15.189:0/3115293822 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b4fc20 0x562221b53a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.615+0000 7ff351525b80 1 --2- 172.21.15.189:0/3115293822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b54fc0 0x562221b2b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.615+0000 7ff351525b80 1 -- 172.21.15.189:0/3115293822 >> 172.21.15.189:0/3115293822 conn(0x562221a8b650 msgr2=0x562221b723e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:46.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.615+0000 7ff351525b80 1 -- 172.21.15.189:0/3115293822 shutdown_connections 2024-04-06T12:28:46.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.615+0000 7ff351525b80 1 -- 172.21.15.189:0/3115293822 wait complete. 2024-04-06T12:28:46.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.615+0000 7ff351525b80 1 -- start start 2024-04-06T12:28:46.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.616+0000 7ff351525b80 1 Processor -- start 2024-04-06T12:28:46.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.616+0000 7ff351525b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b2a980 0x562221b3fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.616+0000 7ff351525b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b4fc20 0x562221b74e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.616+0000 7ff351525b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b54fc0 0x562221b75390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.616+0000 7ff351525b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562221a19d30 con 0x562221b54fc0 2024-04-06T12:28:46.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.616+0000 7ff351525b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x562221a199d0 con 0x562221b2a980 2024-04-06T12:28:46.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.616+0000 7ff351525b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x562221901650 con 0x562221b4fc20 2024-04-06T12:28:46.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.616+0000 7ff351525b80 10 client.0 _ll_register_callbacks cb 0x562221a23420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:28:46.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.616+0000 7ff33f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b54fc0 0x562221b75390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.616+0000 7ff33f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b54fc0 0x562221b75390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53526/0 (socket says 172.21.15.189:53526) 2024-04-06T12:28:46.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.616+0000 7ff33f7fe700 1 -- 172.21.15.189:0/3325278081 learned_addr learned my addr 172.21.15.189:0/3325278081 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:46.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.616+0000 7ff33e7fc700 1 --2- 172.21.15.189:0/3325278081 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b4fc20 0x562221b74e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff33effd700 1 --2- 172.21.15.189:0/3325278081 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b2a980 0x562221b3fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff33f7fe700 1 -- 172.21.15.189:0/3325278081 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b4fc20 msgr2=0x562221b74e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:46.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff33f7fe700 1 --2- 172.21.15.189:0/3325278081 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b4fc20 0x562221b74e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff33f7fe700 1 -- 172.21.15.189:0/3325278081 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b2a980 msgr2=0x562221b3fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:46.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff33f7fe700 1 --2- 172.21.15.189:0/3325278081 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b2a980 0x562221b3fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff33f7fe700 1 -- 172.21.15.189:0/3325278081 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff338009690 con 0x562221b54fc0 2024-04-06T12:28:46.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff33effd700 1 --2- 172.21.15.189:0/3325278081 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b2a980 0x562221b3fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:46.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff33f7fe700 1 --2- 172.21.15.189:0/3325278081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b54fc0 0x562221b75390 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7ff338009c80 tx=0x7ff338004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee08663f9ac3d327 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:46.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff31dffb700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:28:46.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff31dffb700 1 -- 172.21.15.189:0/3325278081 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33801e840 con 0x562221b54fc0 2024-04-06T12:28:46.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff31dffb700 1 -- 172.21.15.189:0/3325278081 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff33801e9a0 con 0x562221b54fc0 2024-04-06T12:28:46.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff351525b80 1 -- 172.21.15.189:0/3325278081 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562221a3e330 con 0x562221b54fc0 2024-04-06T12:28:46.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff31dffb700 1 -- 172.21.15.189:0/3325278081 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff33801eb60 con 0x562221b54fc0 2024-04-06T12:28:46.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff351525b80 10 client.7722 fetch_fsmap learned FSMap version 46 2024-04-06T12:28:46.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff351525b80 1 -- 172.21.15.189:0/3325278081 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) v3 -- 0x562221a43e50 con 0x562221b54fc0 2024-04-06T12:28:46.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff31dffb700 1 -- 172.21.15.189:0/3325278081 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562221a3e330 con 0x562221b54fc0 2024-04-06T12:28:46.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff31dffb700 1 -- 172.21.15.189:0/3325278081 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 46) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff338014070 con 0x562221b54fc0 2024-04-06T12:28:46.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff351525b80 10 client.7722 fetch_fsmap finished waiting for FSMap version 46 2024-04-06T12:28:46.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff351525b80 10 client.7722 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-06T12:28:46.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff351525b80 20 client.7722 populate_metadata read hostname 'smithi189' 2024-04-06T12:28:46.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff351525b80 1 --2- 172.21.15.189:0/3325278081 >> [v2:172.21.15.189:6835/2225868923,v1:172.21.15.189:6837/2225868923] conn(0x562221b69720 0x562221b6ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff351525b80 4 client.7722 mds_command: new command op to 7572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-06T12:28:46.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.617+0000 7ff351525b80 1 -- 172.21.15.189:0/3325278081 --> [v2:172.21.15.189:6835/2225868923,v1:172.21.15.189:6837/2225868923] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562221a766b0 con 0x562221b69720 2024-04-06T12:28:46.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.618+0000 7ff33effd700 1 --2- 172.21.15.189:0/3325278081 >> [v2:172.21.15.189:6835/2225868923,v1:172.21.15.189:6837/2225868923] conn(0x562221b69720 0x562221b6ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.618+0000 7ff33effd700 1 --2- 172.21.15.189:0/3325278081 >> [v2:172.21.15.189:6835/2225868923,v1:172.21.15.189:6837/2225868923] conn(0x562221b69720 0x562221b6ba80 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=250890fe96acce5a server_cookie=7a8d841c57848bb1 in_seq=0 out_seq=0 2024-04-06T12:28:46.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.618+0000 7ff31dffb700 10 client.7722 ms_handle_connect on v2:172.21.15.189:6835/2225868923 2024-04-06T12:28:46.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.618+0000 7ff31dffb700 1 -- 172.21.15.189:0/3325278081 <== mds.0 v2:172.21.15.189:6835/2225868923 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x562221a766b0 con 0x562221b69720 2024-04-06T12:28:46.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.618+0000 7ff31dffb700 10 client.7722 handle_command_reply: tid=0 2024-04-06T12:28:46.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.618+0000 7ff351525b80 1 client.7722 shutdown 2024-04-06T12:28:46.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.618+0000 7ff351525b80 1 -- 172.21.15.189:0/3325278081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b54fc0 msgr2=0x562221b75390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:46.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.618+0000 7ff351525b80 1 --2- 172.21.15.189:0/3325278081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b54fc0 0x562221b75390 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7ff338009c80 tx=0x7ff338004ad0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.619+0000 7ff351525b80 1 -- 172.21.15.189:0/3325278081 shutdown_connections 2024-04-06T12:28:46.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.619+0000 7ff351525b80 1 --2- 172.21.15.189:0/3325278081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b54fc0 0x562221b75390 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.619+0000 7ff351525b80 1 --2- 172.21.15.189:0/3325278081 >> [v2:172.21.15.189:6835/2225868923,v1:172.21.15.189:6837/2225868923] conn(0x562221b69720 0x562221b6ba80 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.619+0000 7ff351525b80 1 --2- 172.21.15.189:0/3325278081 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b2a980 0x562221b3fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.619+0000 7ff351525b80 1 --2- 172.21.15.189:0/3325278081 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b4fc20 0x562221b74e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.619+0000 7ff31dffb700 0 client.7722 ms_handle_reset on v2:172.21.15.189:6835/2225868923 2024-04-06T12:28:46.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.619+0000 7ff351525b80 1 -- 172.21.15.189:0/3325278081 >> 172.21.15.189:0/3325278081 conn(0x562221a8b650 msgr2=0x562221b72370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:46.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.619+0000 7ff351525b80 1 -- 172.21.15.189:0/3325278081 shutdown_connections 2024-04-06T12:28:46.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.619+0000 7ff351525b80 1 -- 172.21.15.189:0/3325278081 wait complete. 2024-04-06T12:28:46.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.619+0000 7ff351525b80 20 client.7722 trim_cache size 0 max 16384 2024-04-06T12:28:46.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.636+0000 7ff351525b80 1 Processor -- start 2024-04-06T12:28:46.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.636+0000 7ff351525b80 1 -- start start 2024-04-06T12:28:46.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff351525b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b2a980 0x562221b28680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff351525b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b4fc20 0x562221b2b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff351525b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b54fc0 0x562221b235c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff351525b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562221b58340 con 0x562221b2a980 2024-04-06T12:28:46.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff351525b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x562221b578c0 con 0x562221b4fc20 2024-04-06T12:28:46.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff351525b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x562221b82600 con 0x562221b54fc0 2024-04-06T12:28:46.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff33effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b2a980 0x562221b28680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff33effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b2a980 0x562221b28680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53528/0 (socket says 172.21.15.189:53528) 2024-04-06T12:28:46.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff33effd700 1 -- 172.21.15.189:0/533552764 learned_addr learned my addr 172.21.15.189:0/533552764 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:46.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff33f7fe700 1 --2- 172.21.15.189:0/533552764 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b54fc0 0x562221b235c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff33e7fc700 1 --2- 172.21.15.189:0/533552764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b4fc20 0x562221b2b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff33effd700 1 -- 172.21.15.189:0/533552764 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b54fc0 msgr2=0x562221b235c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:46.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff33effd700 1 --2- 172.21.15.189:0/533552764 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b54fc0 0x562221b235c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff33effd700 1 -- 172.21.15.189:0/533552764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b4fc20 msgr2=0x562221b2b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:46.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff33effd700 1 --2- 172.21.15.189:0/533552764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b4fc20 0x562221b2b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff33effd700 1 -- 172.21.15.189:0/533552764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562221903c80 con 0x562221b2a980 2024-04-06T12:28:46.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff33f7fe700 1 --2- 172.21.15.189:0/533552764 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b54fc0 0x562221b235c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:28:46.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff33effd700 1 --2- 172.21.15.189:0/533552764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b2a980 0x562221b28680 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff334001710 tx=0x7ff334001010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a43b6023db84e01 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:46.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff3257fa700 1 -- 172.21.15.189:0/533552764 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff334061df0 con 0x562221b2a980 2024-04-06T12:28:46.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff3257fa700 1 -- 172.21.15.189:0/533552764 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff334057da0 con 0x562221b2a980 2024-04-06T12:28:46.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.637+0000 7ff3257fa700 1 -- 172.21.15.189:0/533552764 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff334088fb0 con 0x562221b2a980 2024-04-06T12:28:46.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.638+0000 7ff351525b80 1 -- 172.21.15.189:0/533552764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b2a980 msgr2=0x562221b28680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:46.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.638+0000 7ff351525b80 1 --2- 172.21.15.189:0/533552764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b2a980 0x562221b28680 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff334001710 tx=0x7ff334001010 comp rx=0 tx=0).stop 2024-04-06T12:28:46.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.638+0000 7ff351525b80 1 -- 172.21.15.189:0/533552764 shutdown_connections 2024-04-06T12:28:46.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.638+0000 7ff351525b80 1 --2- 172.21.15.189:0/533552764 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b54fc0 0x562221b235c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.638+0000 7ff351525b80 1 --2- 172.21.15.189:0/533552764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b2a980 0x562221b28680 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.638+0000 7ff351525b80 1 --2- 172.21.15.189:0/533552764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b4fc20 0x562221b2b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.638+0000 7ff351525b80 1 -- 172.21.15.189:0/533552764 >> 172.21.15.189:0/533552764 conn(0x562221a8b650 msgr2=0x562221b93310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:46.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.638+0000 7ff351525b80 1 -- 172.21.15.189:0/533552764 shutdown_connections 2024-04-06T12:28:46.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.638+0000 7ff351525b80 1 -- 172.21.15.189:0/533552764 wait complete. 2024-04-06T12:28:46.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.638+0000 7ff351525b80 1 -- start start 2024-04-06T12:28:46.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.639+0000 7ff351525b80 1 Processor -- start 2024-04-06T12:28:46.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.640+0000 7ff351525b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b2a980 0x562221bdbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.640+0000 7ff351525b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b4fc20 0x562221b72080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.640+0000 7ff351525b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b54fc0 0x562221b725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.640+0000 7ff351525b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562221b52b40 con 0x562221b4fc20 2024-04-06T12:28:46.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.640+0000 7ff351525b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x562221b98200 con 0x562221b2a980 2024-04-06T12:28:46.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.640+0000 7ff351525b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x562221b2ced0 con 0x562221b54fc0 2024-04-06T12:28:46.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.640+0000 7ff351525b80 10 client.0 _ll_register_callbacks cb 0x562221a23420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:28:46.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.640+0000 7ff33e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b4fc20 0x562221b72080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.640+0000 7ff33e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b4fc20 0x562221b72080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53532/0 (socket says 172.21.15.189:53532) 2024-04-06T12:28:46.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.640+0000 7ff33e7fc700 1 -- 172.21.15.189:0/1324221296 learned_addr learned my addr 172.21.15.189:0/1324221296 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:46.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.640+0000 7ff33effd700 1 --2- 172.21.15.189:0/1324221296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b2a980 0x562221bdbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.640+0000 7ff33f7fe700 1 --2- 172.21.15.189:0/1324221296 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b54fc0 0x562221b725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.640+0000 7ff33e7fc700 1 -- 172.21.15.189:0/1324221296 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b54fc0 msgr2=0x562221b725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:46.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.640+0000 7ff33e7fc700 1 --2- 172.21.15.189:0/1324221296 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b54fc0 0x562221b725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.640+0000 7ff33e7fc700 1 -- 172.21.15.189:0/1324221296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b2a980 msgr2=0x562221bdbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:46.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.640+0000 7ff33e7fc700 1 --2- 172.21.15.189:0/1324221296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b2a980 0x562221bdbe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.640+0000 7ff33e7fc700 1 -- 172.21.15.189:0/1324221296 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff334082e70 con 0x562221b4fc20 2024-04-06T12:28:46.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.640+0000 7ff33e7fc700 1 --2- 172.21.15.189:0/1324221296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b4fc20 0x562221b72080 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff34015b0f0 tx=0x7ff3300047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3fdafe7f85bea97 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:46.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.640+0000 7ff31dffb700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:28:46.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.641+0000 7ff31dffb700 1 -- 172.21.15.189:0/1324221296 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff330016dd0 con 0x562221b4fc20 2024-04-06T12:28:46.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.641+0000 7ff31dffb700 1 -- 172.21.15.189:0/1324221296 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff33001d040 con 0x562221b4fc20 2024-04-06T12:28:46.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.641+0000 7ff351525b80 1 -- 172.21.15.189:0/1324221296 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff33800f070 con 0x562221b4fc20 2024-04-06T12:28:46.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.641+0000 7ff31dffb700 1 -- 172.21.15.189:0/1324221296 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff330004020 con 0x562221b4fc20 2024-04-06T12:28:46.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.641+0000 7ff351525b80 10 client.7728 fetch_fsmap learned FSMap version 46 2024-04-06T12:28:46.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.641+0000 7ff351525b80 1 -- 172.21.15.189:0/1324221296 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) v3 -- 0x5622218a9fe0 con 0x562221b4fc20 2024-04-06T12:28:46.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.641+0000 7ff31dffb700 1 -- 172.21.15.189:0/1324221296 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33800f070 con 0x562221b4fc20 2024-04-06T12:28:46.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.641+0000 7ff31dffb700 1 -- 172.21.15.189:0/1324221296 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 46) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff330004180 con 0x562221b4fc20 2024-04-06T12:28:46.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.641+0000 7ff351525b80 10 client.7728 fetch_fsmap finished waiting for FSMap version 46 2024-04-06T12:28:46.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.641+0000 7ff351525b80 10 client.7728 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-06T12:28:46.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.641+0000 7ff351525b80 20 client.7728 populate_metadata read hostname 'smithi189' 2024-04-06T12:28:46.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.641+0000 7ff351525b80 1 --2- 172.21.15.189:0/1324221296 >> [v2:172.21.15.189:6835/2225868923,v1:172.21.15.189:6837/2225868923] conn(0x562221b70ff0 0x562221b713d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.641+0000 7ff351525b80 4 client.7728 mds_command: new command op to 7572 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-06T12:28:46.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.641+0000 7ff351525b80 1 -- 172.21.15.189:0/1324221296 --> [v2:172.21.15.189:6835/2225868923,v1:172.21.15.189:6837/2225868923] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5622218ccca0 con 0x562221b70ff0 2024-04-06T12:28:46.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.641+0000 7ff33effd700 1 --2- 172.21.15.189:0/1324221296 >> [v2:172.21.15.189:6835/2225868923,v1:172.21.15.189:6837/2225868923] conn(0x562221b70ff0 0x562221b713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.641+0000 7ff31dffb700 10 client.7728 ms_handle_connect on v2:172.21.15.189:6835/2225868923 2024-04-06T12:28:46.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.641+0000 7ff33effd700 1 --2- 172.21.15.189:0/1324221296 >> [v2:172.21.15.189:6835/2225868923,v1:172.21.15.189:6837/2225868923] conn(0x562221b70ff0 0x562221b713d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0ac5767c28f4512 server_cookie=a57cd12ab51fecb9 in_seq=0 out_seq=0 2024-04-06T12:28:46.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff31dffb700 1 -- 172.21.15.189:0/1324221296 <== mds.0 v2:172.21.15.189:6835/2225868923 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x5622218ccca0 con 0x562221b70ff0 2024-04-06T12:28:46.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff31dffb700 10 client.7728 handle_command_reply: tid=0 2024-04-06T12:28:46.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff351525b80 1 client.7728 shutdown 2024-04-06T12:28:46.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff351525b80 1 -- 172.21.15.189:0/1324221296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b4fc20 msgr2=0x562221b72080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:46.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff351525b80 1 --2- 172.21.15.189:0/1324221296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b4fc20 0x562221b72080 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff34015b0f0 tx=0x7ff3300047c0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff351525b80 1 -- 172.21.15.189:0/1324221296 shutdown_connections 2024-04-06T12:28:46.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff351525b80 1 --2- 172.21.15.189:0/1324221296 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x562221b54fc0 0x562221b725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff351525b80 1 --2- 172.21.15.189:0/1324221296 >> [v2:172.21.15.189:6835/2225868923,v1:172.21.15.189:6837/2225868923] conn(0x562221b70ff0 0x562221b713d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff351525b80 1 --2- 172.21.15.189:0/1324221296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562221b2a980 0x562221bdbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff351525b80 1 --2- 172.21.15.189:0/1324221296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562221b4fc20 0x562221b72080 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff31dffb700 0 client.7728 ms_handle_reset on v2:172.21.15.189:6835/2225868923 2024-04-06T12:28:46.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff351525b80 1 -- 172.21.15.189:0/1324221296 >> 172.21.15.189:0/1324221296 conn(0x562221a8b650 msgr2=0x562221b75870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:46.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff351525b80 1 -- 172.21.15.189:0/1324221296 shutdown_connections 2024-04-06T12:28:46.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff351525b80 1 -- 172.21.15.189:0/1324221296 wait complete. 2024-04-06T12:28:46.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff351525b80 20 client.7728 trim_cache size 0 max 16384 2024-04-06T12:28:46.646 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T12:28:46.646 INFO:teuthology.orchestra.run.smithi189.stdout: "cluster_fsid": "e30041de-e9a3-4351-8ca7-29d7f3e3728c", 2024-04-06T12:28:46.646 INFO:teuthology.orchestra.run.smithi189.stdout: "whoami": 0, 2024-04-06T12:28:46.646 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 7572, 2024-04-06T12:28:46.646 INFO:teuthology.orchestra.run.smithi189.stdout: "want_state": "up:active", 2024-04-06T12:28:46.646 INFO:teuthology.orchestra.run.smithi189.stdout: "state": "up:active", 2024-04-06T12:28:46.646 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs", 2024-04-06T12:28:46.646 INFO:teuthology.orchestra.run.smithi189.stdout: "rank_uptime": 5.9805670849999997, 2024-04-06T12:28:46.646 INFO:teuthology.orchestra.run.smithi189.stdout: "mdsmap_epoch": 46, 2024-04-06T12:28:46.646 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch": 112, 2024-04-06T12:28:46.646 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch_barrier": 112, 2024-04-06T12:28:46.646 INFO:teuthology.orchestra.run.smithi189.stdout: "uptime": 11.314467455999999 2024-04-06T12:28:46.646 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T12:28:46.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff345371700 1 -- 172.21.15.189:0/1090212326 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff32c06adb0 msgr2=0x7ff32c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:46.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff345371700 1 --2- 172.21.15.189:0/1090212326 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff32c06adb0 0x7ff32c06d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff34015a790 tx=0x7ff330008040 comp rx=0 tx=0).stop 2024-04-06T12:28:46.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff345371700 1 -- 172.21.15.189:0/1090212326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff328003a00 msgr2=0x7ff340157ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:46.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff345371700 1 --2- 172.21.15.189:0/1090212326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff328003a00 0x7ff340157ff0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7ff33400c8e0 tx=0x7ff334001a00 comp rx=0 tx=0).stop 2024-04-06T12:28:46.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff345371700 1 -- 172.21.15.189:0/1090212326 shutdown_connections 2024-04-06T12:28:46.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff345371700 1 --2- 172.21.15.189:0/1090212326 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff32c06adb0 0x7ff32c06d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff345371700 1 --2- 172.21.15.189:0/1090212326 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3400c65d0 0x7ff3400ad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff345371700 1 --2- 172.21.15.189:0/1090212326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff328003a00 0x7ff340157ff0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff345371700 1 --2- 172.21.15.189:0/1090212326 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff34015a190 0x7ff3401f6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff345371700 1 -- 172.21.15.189:0/1090212326 >> 172.21.15.189:0/1090212326 conn(0x7ff3400b2db0 msgr2=0x7ff34015dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:46.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff345371700 1 -- 172.21.15.189:0/1090212326 shutdown_connections 2024-04-06T12:28:46.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.642+0000 7ff345371700 1 -- 172.21.15.189:0/1090212326 wait complete. 2024-04-06T12:28:46.652 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T12:28:46.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.777+0000 7f0c3eb3a700 1 Processor -- start 2024-04-06T12:28:46.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.778+0000 7f0c3eb3a700 1 -- start start 2024-04-06T12:28:46.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.778+0000 7f0c3eb3a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c3806ae40 0x7f0c380b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.778+0000 7f0c3eb3a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0c3815ddf0 0x7f0c3815e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.778+0000 7f0c3eb3a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c380b3990 con 0x7f0c3815d7b0 2024-04-06T12:28:46.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.778+0000 7f0c3eb3a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c380b3060 con 0x7f0c3815ddf0 2024-04-06T12:28:46.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.778+0000 7f0c3eb3a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c380b3340 con 0x7f0c3806ae40 2024-04-06T12:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.778+0000 7f0c3d0d7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0c3815ddf0 0x7f0c3815e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.779+0000 7f0c3d0d7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0c3815ddf0 0x7f0c3815e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53534/0 (socket says 172.21.15.189:53534) 2024-04-06T12:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.779+0000 7f0c3d0d7700 1 -- 172.21.15.189:0/1787405402 learned_addr learned my addr 172.21.15.189:0/1787405402 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.779+0000 7f0c37fff700 1 --2- 172.21.15.189:0/1787405402 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c3806ae40 0x7f0c380b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.779+0000 7f0c3d0d7700 1 -- 172.21.15.189:0/1787405402 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c3806ae40 msgr2=0x7f0c380b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.779+0000 7f0c3d0d7700 1 --2- 172.21.15.189:0/1787405402 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c3806ae40 0x7f0c380b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.779+0000 7f0c3d0d7700 1 -- 172.21.15.189:0/1787405402 >> v1:172.21.15.192:6789/0 conn(0x7f0c3815d7b0 legacy=0x7f0c3806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.779+0000 7f0c3d0d7700 1 -- 172.21.15.189:0/1787405402 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c380b3620 con 0x7f0c3815ddf0 2024-04-06T12:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.779+0000 7f0c37fff700 1 --2- 172.21.15.189:0/1787405402 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c3806ae40 0x7f0c380b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.779+0000 7f0c3d0d7700 1 --2- 172.21.15.189:0/1787405402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3815ddf0 0x7f0c3815e2a0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f0c30009e50 tx=0x7f0c3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97433f6422e2e59b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.779+0000 7f0c377fe700 1 -- 172.21.15.189:0/1787405402 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3000e070 con 0x7f0c3815ddf0 2024-04-06T12:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.779+0000 7f0c377fe700 1 -- 172.21.15.189:0/1787405402 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c30003a20 con 0x7f0c3815ddf0 2024-04-06T12:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.779+0000 7f0c377fe700 1 -- 172.21.15.189:0/1787405402 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c30003dd0 con 0x7f0c3815ddf0 2024-04-06T12:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.780+0000 7f0c3eb3a700 1 -- 172.21.15.189:0/1787405402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3815ddf0 msgr2=0x7f0c3815e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:46.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.780+0000 7f0c3eb3a700 1 --2- 172.21.15.189:0/1787405402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3815ddf0 0x7f0c3815e2a0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f0c30009e50 tx=0x7f0c3000b330 comp rx=0 tx=0).stop 2024-04-06T12:28:46.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.780+0000 7f0c3eb3a700 1 -- 172.21.15.189:0/1787405402 shutdown_connections 2024-04-06T12:28:46.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.780+0000 7f0c3eb3a700 1 --2- 172.21.15.189:0/1787405402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3815ddf0 0x7f0c3815e2a0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.780+0000 7f0c3eb3a700 1 --2- 172.21.15.189:0/1787405402 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c3806ae40 0x7f0c380b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.780+0000 7f0c3eb3a700 1 -- 172.21.15.189:0/1787405402 >> 172.21.15.189:0/1787405402 conn(0x7f0c380d00b0 msgr2=0x7f0c380d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:46.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.780+0000 7f0c3eb3a700 1 -- 172.21.15.189:0/1787405402 shutdown_connections 2024-04-06T12:28:46.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.780+0000 7f0c3eb3a700 1 -- 172.21.15.189:0/1787405402 wait complete. 2024-04-06T12:28:46.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.781+0000 7f0c3eb3a700 1 Processor -- start 2024-04-06T12:28:46.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.781+0000 7f0c3eb3a700 1 -- start start 2024-04-06T12:28:46.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.782+0000 7f0c3eb3a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3806ae40 0x7f0c381519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.782+0000 7f0c3eb3a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c3815d7b0 0x7f0c38152b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.782+0000 7f0c3eb3a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c3815ddf0 0x7f0c38153080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.782+0000 7f0c3eb3a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c381f6ab0 con 0x7f0c3806ae40 2024-04-06T12:28:46.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.782+0000 7f0c3eb3a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c38151ee0 con 0x7f0c3815ddf0 2024-04-06T12:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.782+0000 7f0c3eb3a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c381f5e30 con 0x7f0c3815d7b0 2024-04-06T12:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.782+0000 7f0c3c8d6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3806ae40 0x7f0c381519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.782+0000 7f0c3c8d6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3806ae40 0x7f0c381519a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53536/0 (socket says 172.21.15.189:53536) 2024-04-06T12:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.782+0000 7f0c3c8d6700 1 -- 172.21.15.189:0/820587073 learned_addr learned my addr 172.21.15.189:0/820587073 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.782+0000 7f0c37fff700 1 --2- 172.21.15.189:0/820587073 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c3815d7b0 0x7f0c38152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.782+0000 7f0c3d0d7700 1 --2- 172.21.15.189:0/820587073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c3815ddf0 0x7f0c38153080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.782+0000 7f0c3c8d6700 1 -- 172.21.15.189:0/820587073 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c3815d7b0 msgr2=0x7f0c38152b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.782+0000 7f0c3c8d6700 1 --2- 172.21.15.189:0/820587073 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c3815d7b0 0x7f0c38152b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.782+0000 7f0c3c8d6700 1 -- 172.21.15.189:0/820587073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c3815ddf0 msgr2=0x7f0c38153080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.782+0000 7f0c3c8d6700 1 --2- 172.21.15.189:0/820587073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c3815ddf0 0x7f0c38153080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.782+0000 7f0c3c8d6700 1 -- 172.21.15.189:0/820587073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c30009b30 con 0x7f0c3806ae40 2024-04-06T12:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.782+0000 7f0c37fff700 1 --2- 172.21.15.189:0/820587073 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c3815d7b0 0x7f0c38152b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.782+0000 7f0c3d0d7700 1 --2- 172.21.15.189:0/820587073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c3815ddf0 0x7f0c38153080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.782+0000 7f0c3c8d6700 1 --2- 172.21.15.189:0/820587073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3806ae40 0x7f0c381519a0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c00c4d0 tx=0x7f0c2c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.783+0000 7f0c357fa700 1 -- 172.21.15.189:0/820587073 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c011790 con 0x7f0c3806ae40 2024-04-06T12:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.783+0000 7f0c357fa700 1 -- 172.21.15.189:0/820587073 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2c0118f0 con 0x7f0c3806ae40 2024-04-06T12:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.783+0000 7f0c3eb3a700 1 -- 172.21.15.189:0/820587073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c381f6110 con 0x7f0c3806ae40 2024-04-06T12:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.783+0000 7f0c357fa700 1 -- 172.21.15.189:0/820587073 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c011ca0 con 0x7f0c3806ae40 2024-04-06T12:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.783+0000 7f0c3eb3a700 1 -- 172.21.15.189:0/820587073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c381f66d0 con 0x7f0c3806ae40 2024-04-06T12:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.784+0000 7f0c3eb3a700 1 -- 172.21.15.189:0/820587073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c381f6bf0 con 0x7f0c3806ae40 2024-04-06T12:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.784+0000 7f0c357fa700 1 -- 172.21.15.189:0/820587073 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0c2c01c460 con 0x7f0c3806ae40 2024-04-06T12:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.784+0000 7f0c357fa700 1 --2- 172.21.15.189:0/820587073 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c1c06adb0 0x7f0c1c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.784+0000 7f0c357fa700 1 -- 172.21.15.189:0/820587073 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0c2c089440 con 0x7f0c3806ae40 2024-04-06T12:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.784+0000 7f0c37fff700 1 --2- 172.21.15.189:0/820587073 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c1c06adb0 0x7f0c1c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.785+0000 7f0c37fff700 1 --2- 172.21.15.189:0/820587073 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c1c06adb0 0x7f0c1c06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0c380cc2f0 tx=0x7f0c24008040 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:46.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:46.789+0000 7f0c357fa700 1 -- 172.21.15.189:0/820587073 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0c381f6bf0 con 0x7f0c3806ae40 2024-04-06T12:28:47.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.028+0000 7f0c3eb3a700 1 -- 172.21.15.189:0/820587073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0c381f6bf0 con 0x7f0c3806ae40 2024-04-06T12:28:47.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.028+0000 7f0c357fa700 1 -- 172.21.15.189:0/820587073 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v46) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0c381f6bf0 con 0x7f0c3806ae40 2024-04-06T12:28:47.025 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-06T12:28:47.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.030+0000 7f0c3eb3a700 1 -- 172.21.15.189:0/820587073 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c1c06adb0 msgr2=0x7f0c1c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:47.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.030+0000 7f0c3eb3a700 1 --2- 172.21.15.189:0/820587073 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c1c06adb0 0x7f0c1c06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0c380cc2f0 tx=0x7f0c24008040 comp rx=0 tx=0).stop 2024-04-06T12:28:47.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.030+0000 7f0c3eb3a700 1 -- 172.21.15.189:0/820587073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3806ae40 msgr2=0x7f0c381519a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:47.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.030+0000 7f0c3eb3a700 1 --2- 172.21.15.189:0/820587073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3806ae40 0x7f0c381519a0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c00c4d0 tx=0x7f0c2c00c890 comp rx=0 tx=0).stop 2024-04-06T12:28:47.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.030+0000 7f0c3eb3a700 1 -- 172.21.15.189:0/820587073 shutdown_connections 2024-04-06T12:28:47.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.030+0000 7f0c3eb3a700 1 --2- 172.21.15.189:0/820587073 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c1c06adb0 0x7f0c1c06d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:47.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.030+0000 7f0c3eb3a700 1 --2- 172.21.15.189:0/820587073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c3815ddf0 0x7f0c38153080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:47.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.030+0000 7f0c3eb3a700 1 --2- 172.21.15.189:0/820587073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3806ae40 0x7f0c381519a0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:47.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.030+0000 7f0c3eb3a700 1 --2- 172.21.15.189:0/820587073 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c3815d7b0 0x7f0c38152b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:47.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.030+0000 7f0c3eb3a700 1 -- 172.21.15.189:0/820587073 >> 172.21.15.189:0/820587073 conn(0x7f0c380d00b0 msgr2=0x7f0c3806eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:47.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.030+0000 7f0c3eb3a700 1 -- 172.21.15.189:0/820587073 shutdown_connections 2024-04-06T12:28:47.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.030+0000 7f0c3eb3a700 1 -- 172.21.15.189:0/820587073 wait complete. 2024-04-06T12:28:47.039 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-06T12:28:47.040 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T12:28:47.040 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-06T12:28:47.040 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-06T12:28:47.040 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T12:28:47.040 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-06T12:28:47.040 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-06T12:28:47.040 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T12:28:47.040 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:28:47.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:47.040 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-06T12:28:47.058 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T12:28:47.058 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T12:28:47.058 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T12:28:47.058 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:28:47.058 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T12:28:47.058 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:28:47.058 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:28:47.059 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:28:47.059 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-06T12:28:47.059 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:28:47.059 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:28:47.059 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-06T12:28:47.059 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:28:47.059 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:28:47.059 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T12:28:47.059 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:28:47.059 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T12:28:47.059 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 41363sec preferred_lft 41363sec 2024-04-06T12:28:47.059 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::8592:df2c:92e2:dace/64 scope link noprefixroute 2024-04-06T12:28:47.059 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:28:47.059 INFO:teuthology.orchestra.run.smithi189.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T12:28:47.059 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff 2024-04-06T12:28:47.060 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-06T12:28:47.060 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:28:47.060 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::88e:b6ff:fe6f:1cb0/64 scope link 2024-04-06T12:28:47.060 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:28:47.060 INFO:teuthology.orchestra.run.smithi189.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T12:28:47.060 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 62:a5:8d:c5:1a:9c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T12:28:47.060 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::60a5:8dff:fec5:1a9c/64 scope link 2024-04-06T12:28:47.060 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:28:47.060 INFO:teuthology.orchestra.run.smithi189.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T12:28:47.060 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-06T12:28:47.060 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::1c7e:c6ff:febb:1dab/64 scope link 2024-04-06T12:28:47.060 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:28:47.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:47.061 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T12:28:47.125 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-06T12:28:47.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:47.127 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-06T12:28:47.187 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-06T12:28:47.187 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T12:28:47.187 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T12:28:47.187 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T12:28:47.187 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T12:28:47.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:47.188 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:28:47.188 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:28:47.188 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T12:28:47.188 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T12:28:47.188 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T12:28:47.188 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:28:47.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:47.293 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T12:28:47.310 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-06T12:28:47.310 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-06T12:28:47.310 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:28:47.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:47.310 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-04-06T12:28:47.377 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:28:47.378 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-06T12:28:47.435 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:28:47.435 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:28:47.435 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-06T12:28:47.491 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T12:28:47.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:47.492 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:28:47.578 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:28:47.578 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:28:47.578 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:28:47.587 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:28:47.587 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:28:47.587 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:28:47.587 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:28:47.587 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:28:47.606 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:28:47.606 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:28:47.606 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:28:47.606 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:28:47.606 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:28:47.606 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790" 2024-04-06T12:28:47.606 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2024-04-06T12:28:47.606 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e30041de-e9a3-4351-8ca7-29d7f3e3728c" will pass to kernel 2024-04-06T12:28:47.608 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T12:28:47.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:47.608 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-06T12:28:47.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:28:47.636 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:28:47.720 INFO:teuthology.orchestra.run.smithi189.stdout:client6473 2024-04-06T12:28:47.720 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:28:47.720 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:28:47.778 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-06T12:28:47.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.943+0000 7f5c5bc8e700 1 Processor -- start 2024-04-06T12:28:47.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.944+0000 7f5c5bc8e700 1 -- start start 2024-04-06T12:28:47.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.944+0000 7f5c5bc8e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c540d0a60 0x7f5c540d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:47.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.944+0000 7f5c5bc8e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5c540d1af0 0x7f5c540b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:47.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.944+0000 7f5c5bc8e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c540cdf80 con 0x7f5c5415d7e0 2024-04-06T12:28:47.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.944+0000 7f5c5bc8e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c540d1f80 con 0x7f5c540d1af0 2024-04-06T12:28:47.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.944+0000 7f5c5bc8e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c540d3230 con 0x7f5c540d0a60 2024-04-06T12:28:47.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.944+0000 7f5c5a22b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5c540d1af0 0x7f5c540b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:47.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.944+0000 7f5c59a2a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5c5415d7e0 0x7f5c540d0350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51530/0 (socket says 172.21.15.189:51530) 2024-04-06T12:28:47.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.945+0000 7f5c59a2a700 1 -- 172.21.15.189:0/400714301 learned_addr learned my addr 172.21.15.189:0/400714301 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:47.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.945+0000 7f5c59229700 1 --2- 172.21.15.189:0/400714301 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c540d0a60 0x7f5c540d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:47.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.945+0000 7f5c5a22b700 1 -- 172.21.15.189:0/400714301 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c540d0a60 msgr2=0x7f5c540d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:47.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.945+0000 7f5c5a22b700 1 --2- 172.21.15.189:0/400714301 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c540d0a60 0x7f5c540d15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:47.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.945+0000 7f5c5a22b700 1 -- 172.21.15.189:0/400714301 >> v1:172.21.15.189:6789/0 conn(0x7f5c5415d7e0 legacy=0x7f5c540d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:47.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.945+0000 7f5c5a22b700 1 -- 172.21.15.189:0/400714301 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c540d2da0 con 0x7f5c540d1af0 2024-04-06T12:28:47.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.945+0000 7f5c58a28700 1 -- 172.21.15.189:0/400714301 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509764218 0 0) 0x7f5c540cdf80 con 0x7f5c5415d7e0 2024-04-06T12:28:47.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.945+0000 7f5c5a22b700 1 --2- 172.21.15.189:0/400714301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c540d1af0 0x7f5c540b66b0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009d90 tx=0x7f5c4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f56e6e687475593 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:47.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.945+0000 7f5c58a28700 1 -- 172.21.15.189:0/400714301 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c00e070 con 0x7f5c540d1af0 2024-04-06T12:28:47.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.945+0000 7f5c58a28700 1 -- 172.21.15.189:0/400714301 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4c00bbd0 con 0x7f5c540d1af0 2024-04-06T12:28:47.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.945+0000 7f5c58a28700 1 -- 172.21.15.189:0/400714301 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c012430 con 0x7f5c540d1af0 2024-04-06T12:28:47.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.946+0000 7f5c5bc8e700 1 -- 172.21.15.189:0/400714301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c540d1af0 msgr2=0x7f5c540b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:47.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.946+0000 7f5c5bc8e700 1 --2- 172.21.15.189:0/400714301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c540d1af0 0x7f5c540b66b0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009d90 tx=0x7f5c4c00b330 comp rx=0 tx=0).stop 2024-04-06T12:28:47.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.946+0000 7f5c5bc8e700 1 -- 172.21.15.189:0/400714301 shutdown_connections 2024-04-06T12:28:47.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.946+0000 7f5c5bc8e700 1 --2- 172.21.15.189:0/400714301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c540d1af0 0x7f5c540b66b0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:47.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.946+0000 7f5c5bc8e700 1 --2- 172.21.15.189:0/400714301 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c540d0a60 0x7f5c540d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:47.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.946+0000 7f5c5bc8e700 1 -- 172.21.15.189:0/400714301 >> 172.21.15.189:0/400714301 conn(0x7f5c54072250 msgr2=0x7f5c54072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:47.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.946+0000 7f5c5bc8e700 1 -- 172.21.15.189:0/400714301 shutdown_connections 2024-04-06T12:28:47.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.946+0000 7f5c5bc8e700 1 -- 172.21.15.189:0/400714301 wait complete. 2024-04-06T12:28:47.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.947+0000 7f5c5bc8e700 1 Processor -- start 2024-04-06T12:28:47.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.947+0000 7f5c5bc8e700 1 -- start start 2024-04-06T12:28:47.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c5bc8e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c540d0a60 0x7f5c540b1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:47.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c5bc8e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c540d1af0 0x7f5c540b4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:47.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c5bc8e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c5415d7e0 0x7f5c540b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:47.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c5bc8e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c540c8930 con 0x7f5c5415d7e0 2024-04-06T12:28:47.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c5bc8e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c540b1ab0 con 0x7f5c540d0a60 2024-04-06T12:28:47.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c5bc8e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c540c8bd0 con 0x7f5c540d1af0 2024-04-06T12:28:47.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c5a22b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c5415d7e0 0x7f5c540b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:47.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c5a22b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c5415d7e0 0x7f5c540b4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53560/0 (socket says 172.21.15.189:53560) 2024-04-06T12:28:47.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c5a22b700 1 -- 172.21.15.189:0/3634387018 learned_addr learned my addr 172.21.15.189:0/3634387018 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:47.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c59229700 1 --2- 172.21.15.189:0/3634387018 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c540d1af0 0x7f5c540b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:47.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c59a2a700 1 --2- 172.21.15.189:0/3634387018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c540d0a60 0x7f5c540b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:47.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c5a22b700 1 -- 172.21.15.189:0/3634387018 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c540d1af0 msgr2=0x7f5c540b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:47.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c5a22b700 1 --2- 172.21.15.189:0/3634387018 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c540d1af0 0x7f5c540b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:47.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c5a22b700 1 -- 172.21.15.189:0/3634387018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c540d0a60 msgr2=0x7f5c540b1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:47.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c5a22b700 1 --2- 172.21.15.189:0/3634387018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c540d0a60 0x7f5c540b1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:47.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c5a22b700 1 -- 172.21.15.189:0/3634387018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c009a70 con 0x7f5c5415d7e0 2024-04-06T12:28:47.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c5a22b700 1 --2- 172.21.15.189:0/3634387018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c5415d7e0 0x7f5c540b4e40 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f5c5406b8c0 tx=0x7f5c4c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:47.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c4a7fc700 1 -- 172.21.15.189:0/3634387018 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c012430 con 0x7f5c5415d7e0 2024-04-06T12:28:47.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c4a7fc700 1 -- 172.21.15.189:0/3634387018 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4c020070 con 0x7f5c5415d7e0 2024-04-06T12:28:47.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c5bc8e700 1 -- 172.21.15.189:0/3634387018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c540c8eb0 con 0x7f5c5415d7e0 2024-04-06T12:28:47.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c4a7fc700 1 -- 172.21.15.189:0/3634387018 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c004280 con 0x7f5c5415d7e0 2024-04-06T12:28:47.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.948+0000 7f5c5bc8e700 1 -- 172.21.15.189:0/3634387018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c540c9470 con 0x7f5c5415d7e0 2024-04-06T12:28:47.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.949+0000 7f5c5bc8e700 1 -- 172.21.15.189:0/3634387018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c54074990 con 0x7f5c5415d7e0 2024-04-06T12:28:47.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.950+0000 7f5c4a7fc700 1 -- 172.21.15.189:0/3634387018 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f5c4c0043e0 con 0x7f5c5415d7e0 2024-04-06T12:28:47.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.950+0000 7f5c4a7fc700 1 --2- 172.21.15.189:0/3634387018 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c3806adb0 0x7f5c3806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:47.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.950+0000 7f5c4a7fc700 1 -- 172.21.15.189:0/3634387018 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5c4c022070 con 0x7f5c5415d7e0 2024-04-06T12:28:47.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.950+0000 7f5c59a2a700 1 --2- 172.21.15.189:0/3634387018 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c3806adb0 0x7f5c3806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:47.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.951+0000 7f5c59a2a700 1 --2- 172.21.15.189:0/3634387018 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c3806adb0 0x7f5c3806d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5c44007fd0 tx=0x7f5c44007de0 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:47.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:47.955+0000 7f5c4a7fc700 1 -- 172.21.15.189:0/3634387018 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5c4c0564c0 con 0x7f5c5415d7e0 2024-04-06T12:28:48.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.181+0000 7f5c5bc8e700 1 -- 172.21.15.189:0/3634387018 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c540d2470 con 0x7f5c3806adb0 2024-04-06T12:28:48.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.183+0000 7f5c4a7fc700 1 -- 172.21.15.189:0/3634387018 <== mgr.7338 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f5c44001370 con 0x7f5c3806adb0 2024-04-06T12:28:48.180 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T12:28:48.180 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:28:48.180 INFO:teuthology.orchestra.run.smithi189.stdout: "name": "cephfs" 2024-04-06T12:28:48.180 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:28:48.180 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T12:28:48.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.185+0000 7f5c5bc8e700 1 -- 172.21.15.189:0/3634387018 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c3806adb0 msgr2=0x7f5c3806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:48.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.185+0000 7f5c5bc8e700 1 --2- 172.21.15.189:0/3634387018 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c3806adb0 0x7f5c3806d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5c44007fd0 tx=0x7f5c44007de0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.185+0000 7f5c5bc8e700 1 -- 172.21.15.189:0/3634387018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c5415d7e0 msgr2=0x7f5c540b4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:48.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.185+0000 7f5c5bc8e700 1 --2- 172.21.15.189:0/3634387018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c5415d7e0 0x7f5c540b4e40 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f5c5406b8c0 tx=0x7f5c4c004ad0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.185+0000 7f5c5bc8e700 1 -- 172.21.15.189:0/3634387018 shutdown_connections 2024-04-06T12:28:48.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.185+0000 7f5c5bc8e700 1 --2- 172.21.15.189:0/3634387018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c5415d7e0 0x7f5c540b4e40 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.185+0000 7f5c5bc8e700 1 --2- 172.21.15.189:0/3634387018 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c3806adb0 0x7f5c3806d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.185+0000 7f5c5bc8e700 1 --2- 172.21.15.189:0/3634387018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c540d0a60 0x7f5c540b1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.185+0000 7f5c5bc8e700 1 --2- 172.21.15.189:0/3634387018 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c540d1af0 0x7f5c540b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.185+0000 7f5c5bc8e700 1 -- 172.21.15.189:0/3634387018 >> 172.21.15.189:0/3634387018 conn(0x7f5c54072250 msgr2=0x7f5c540cd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:48.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.185+0000 7f5c5bc8e700 1 -- 172.21.15.189:0/3634387018 shutdown_connections 2024-04-06T12:28:48.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.185+0000 7f5c5bc8e700 1 -- 172.21.15.189:0/3634387018 wait complete. 2024-04-06T12:28:48.194 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-06T12:28:48.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.323+0000 7f7ec6517700 1 Processor -- start 2024-04-06T12:28:48.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.323+0000 7f7ec6517700 1 -- start start 2024-04-06T12:28:48.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.324+0000 7f7ec6517700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7ec00b4660 0x7f7ec00b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:48.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.324+0000 7f7ec6517700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7ec00b2fc0 0x7f7ec00b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:48.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.324+0000 7f7ec6517700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec00b5030 con 0x7f7ec015a550 2024-04-06T12:28:48.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.324+0000 7f7ec6517700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec00cdbe0 con 0x7f7ec00b4660 2024-04-06T12:28:48.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.324+0000 7f7ec6517700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec015a940 con 0x7f7ec00b2fc0 2024-04-06T12:28:48.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.324+0000 7f7ebf7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7ec00b4660 0x7f7ec00b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:48.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.324+0000 7f7ebffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7ec015a550 0x7f7ec00cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51542/0 (socket says 172.21.15.189:51542) 2024-04-06T12:28:48.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.324+0000 7f7ebffff700 1 -- 172.21.15.189:0/1298863413 learned_addr learned my addr 172.21.15.189:0/1298863413 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:48.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.324+0000 7f7ec4ab4700 1 --2- 172.21.15.189:0/1298863413 >> v2:172.21.15.192:3300/0 conn(0x7f7ec00b2fc0 0x7f7ec00b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:48.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.324+0000 7f7ebf7fe700 1 -- 172.21.15.189:0/1298863413 >> v2:172.21.15.192:3300/0 conn(0x7f7ec00b2fc0 msgr2=0x7f7ec00b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:48.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.324+0000 7f7ebf7fe700 1 --2- 172.21.15.189:0/1298863413 >> v2:172.21.15.192:3300/0 conn(0x7f7ec00b2fc0 0x7f7ec00b33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.324+0000 7f7ebf7fe700 1 -- 172.21.15.189:0/1298863413 >> v1:172.21.15.189:6789/0 conn(0x7f7ec015a550 legacy=0x7f7ec00cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:48.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.324+0000 7f7ebf7fe700 1 -- 172.21.15.189:0/1298863413 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec015ac00 con 0x7f7ec00b4660 2024-04-06T12:28:48.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.324+0000 7f7ebeffd700 1 -- 172.21.15.189:0/1298863413 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17055304 0 0) 0x7f7ec00b5030 con 0x7f7ec015a550 2024-04-06T12:28:48.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.324+0000 7f7ec4ab4700 1 --2- 172.21.15.189:0/1298863413 >> v2:172.21.15.192:3300/0 conn(0x7f7ec00b2fc0 0x7f7ec00b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:28:48.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.324+0000 7f7ebf7fe700 1 --2- 172.21.15.189:0/1298863413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ec00b4660 0x7f7ec00b4af0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0072830 tx=0x7f7eb000bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24a4baa2631f81b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:48.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.324+0000 7f7ebeffd700 1 -- 172.21.15.189:0/1298863413 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb000dc70 con 0x7f7ec00b4660 2024-04-06T12:28:48.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.324+0000 7f7ebeffd700 1 -- 172.21.15.189:0/1298863413 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb000ddd0 con 0x7f7ec00b4660 2024-04-06T12:28:48.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.324+0000 7f7ebeffd700 1 -- 172.21.15.189:0/1298863413 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb0010690 con 0x7f7ec00b4660 2024-04-06T12:28:48.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.325+0000 7f7ec6517700 1 -- 172.21.15.189:0/1298863413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ec00b4660 msgr2=0x7f7ec00b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:48.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.325+0000 7f7ec6517700 1 --2- 172.21.15.189:0/1298863413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ec00b4660 0x7f7ec00b4af0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0072830 tx=0x7f7eb000bdc0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.325+0000 7f7ec6517700 1 -- 172.21.15.189:0/1298863413 shutdown_connections 2024-04-06T12:28:48.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.325+0000 7f7ec6517700 1 --2- 172.21.15.189:0/1298863413 >> v2:172.21.15.192:3300/0 conn(0x7f7ec00b2fc0 0x7f7ec00b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.325+0000 7f7ec6517700 1 --2- 172.21.15.189:0/1298863413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ec00b4660 0x7f7ec00b4af0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.325+0000 7f7ec6517700 1 -- 172.21.15.189:0/1298863413 >> 172.21.15.189:0/1298863413 conn(0x7f7ec00d00b0 msgr2=0x7f7ec00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:48.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.325+0000 7f7ec6517700 1 -- 172.21.15.189:0/1298863413 shutdown_connections 2024-04-06T12:28:48.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.325+0000 7f7ec6517700 1 -- 172.21.15.189:0/1298863413 wait complete. 2024-04-06T12:28:48.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.326+0000 7f7ec6517700 1 Processor -- start 2024-04-06T12:28:48.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.326+0000 7f7ec6517700 1 -- start start 2024-04-06T12:28:48.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.327+0000 7f7ec6517700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ec00b2fc0 0x7f7ec014cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:48.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.327+0000 7f7ec6517700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ec00b4660 0x7f7ec0145aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:48.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.327+0000 7f7ec6517700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ec00aeb80 0x7f7ec00ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:48.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.327+0000 7f7ec6517700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec0154670 con 0x7f7ec00aeb80 2024-04-06T12:28:48.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.327+0000 7f7ec6517700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec0153a60 con 0x7f7ec00b4660 2024-04-06T12:28:48.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.327+0000 7f7ec6517700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec0153d40 con 0x7f7ec00b2fc0 2024-04-06T12:28:48.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.327+0000 7f7ec4ab4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ec00aeb80 0x7f7ec00ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:48.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.327+0000 7f7ec4ab4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ec00aeb80 0x7f7ec00ac8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53580/0 (socket says 172.21.15.189:53580) 2024-04-06T12:28:48.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.327+0000 7f7ec4ab4700 1 -- 172.21.15.189:0/2386181378 learned_addr learned my addr 172.21.15.189:0/2386181378 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:48.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.327+0000 7f7ebf7fe700 1 --2- 172.21.15.189:0/2386181378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ec00b4660 0x7f7ec0145aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:48.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.327+0000 7f7ebffff700 1 --2- 172.21.15.189:0/2386181378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ec00b2fc0 0x7f7ec014cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:48.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.328+0000 7f7ec4ab4700 1 -- 172.21.15.189:0/2386181378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ec00b2fc0 msgr2=0x7f7ec014cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:48.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.328+0000 7f7ec4ab4700 1 --2- 172.21.15.189:0/2386181378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ec00b2fc0 0x7f7ec014cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.328+0000 7f7ec4ab4700 1 -- 172.21.15.189:0/2386181378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ec00b4660 msgr2=0x7f7ec0145aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:48.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.328+0000 7f7ec4ab4700 1 --2- 172.21.15.189:0/2386181378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ec00b4660 0x7f7ec0145aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.328+0000 7f7ec4ab4700 1 -- 172.21.15.189:0/2386181378 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb000b740 con 0x7f7ec00aeb80 2024-04-06T12:28:48.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.328+0000 7f7ec4ab4700 1 --2- 172.21.15.189:0/2386181378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ec00aeb80 0x7f7ec00ac8c0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f7eb800cd50 tx=0x7f7eb800a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:48.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.328+0000 7f7ebcff9700 1 -- 172.21.15.189:0/2386181378 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb800cea0 con 0x7f7ec00aeb80 2024-04-06T12:28:48.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.328+0000 7f7ebcff9700 1 -- 172.21.15.189:0/2386181378 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb800f480 con 0x7f7ec00aeb80 2024-04-06T12:28:48.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.328+0000 7f7ebcff9700 1 -- 172.21.15.189:0/2386181378 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb800f830 con 0x7f7ec00aeb80 2024-04-06T12:28:48.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.328+0000 7f7ec6517700 1 -- 172.21.15.189:0/2386181378 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec0154020 con 0x7f7ec00aeb80 2024-04-06T12:28:48.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.328+0000 7f7ec6517700 1 -- 172.21.15.189:0/2386181378 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec0145fe0 con 0x7f7ec00aeb80 2024-04-06T12:28:48.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.329+0000 7f7ec6517700 1 -- 172.21.15.189:0/2386181378 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec0067bb0 con 0x7f7ec00aeb80 2024-04-06T12:28:48.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.329+0000 7f7ebcff9700 1 -- 172.21.15.189:0/2386181378 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 51) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7eb800b040 con 0x7f7ec00aeb80 2024-04-06T12:28:48.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.330+0000 7f7ebcff9700 1 --2- 172.21.15.189:0/2386181378 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7eac06adb0 0x7f7eac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:48.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.330+0000 7f7ebcff9700 1 -- 172.21.15.189:0/2386181378 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7eb8089d70 con 0x7f7ec00aeb80 2024-04-06T12:28:48.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.330+0000 7f7ebffff700 1 --2- 172.21.15.189:0/2386181378 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7eac06adb0 0x7f7eac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:48.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.330+0000 7f7ebffff700 1 --2- 172.21.15.189:0/2386181378 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7eac06adb0 0x7f7eac06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4007fd0 tx=0x7f7eb4007de0 comp rx=0 tx=0).ready entity=mgr.7338 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:48.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.334+0000 7f7ebcff9700 1 -- 172.21.15.189:0/2386181378 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7eb8054c80 con 0x7f7ec00aeb80 2024-04-06T12:28:48.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.586+0000 7f7ec6517700 1 -- 172.21.15.189:0/2386181378 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f7ec0074990 con 0x7f7ec00aeb80 2024-04-06T12:28:48.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.691+0000 7f7ebcff9700 1 -- 172.21.15.189:0/2386181378 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 52) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f7eb8088150 con 0x7f7ec00aeb80 2024-04-06T12:28:48.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.692+0000 7f7ebcff9700 1 -- 172.21.15.189:0/2386181378 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v52) v1 ==== 92+0+0 (secure 0 0 0) 0x7f7eb8050e40 con 0x7f7ec00aeb80 2024-04-06T12:28:48.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.696+0000 7f7ec6517700 1 -- 172.21.15.189:0/2386181378 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7eac06adb0 msgr2=0x7f7eac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:48.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.696+0000 7f7ec6517700 1 --2- 172.21.15.189:0/2386181378 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7eac06adb0 0x7f7eac06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4007fd0 tx=0x7f7eb4007de0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.696+0000 7f7ec6517700 1 -- 172.21.15.189:0/2386181378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ec00aeb80 msgr2=0x7f7ec00ac8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:48.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.696+0000 7f7ec6517700 1 --2- 172.21.15.189:0/2386181378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ec00aeb80 0x7f7ec00ac8c0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f7eb800cd50 tx=0x7f7eb800a560 comp rx=0 tx=0).stop 2024-04-06T12:28:48.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.696+0000 7f7ec6517700 1 -- 172.21.15.189:0/2386181378 shutdown_connections 2024-04-06T12:28:48.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.696+0000 7f7ec6517700 1 --2- 172.21.15.189:0/2386181378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ec00aeb80 0x7f7ec00ac8c0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.696+0000 7f7ec6517700 1 --2- 172.21.15.189:0/2386181378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ec00b2fc0 0x7f7ec014cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.696+0000 7f7ec6517700 1 --2- 172.21.15.189:0/2386181378 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7eac06adb0 0x7f7eac06d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.696+0000 7f7ec6517700 1 --2- 172.21.15.189:0/2386181378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ec00b4660 0x7f7ec0145aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.696+0000 7f7ec6517700 1 -- 172.21.15.189:0/2386181378 >> 172.21.15.189:0/2386181378 conn(0x7f7ec00d00b0 msgr2=0x7f7ec00cddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:48.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.696+0000 7f7ec6517700 1 -- 172.21.15.189:0/2386181378 shutdown_connections 2024-04-06T12:28:48.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.696+0000 7f7ec6517700 1 -- 172.21.15.189:0/2386181378 wait complete. 2024-04-06T12:28:48.711 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-06T12:28:48.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.852+0000 7f9c40a05700 1 Processor -- start 2024-04-06T12:28:48.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.853+0000 7f9c40a05700 1 -- start start 2024-04-06T12:28:48.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.853+0000 7f9c40a05700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c3c0d0a60 0x7f9c3c0d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:48.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.853+0000 7f9c40a05700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9c3c0d1af0 0x7f9c3c0b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:48.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.853+0000 7f9c40a05700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3c0cdf80 con 0x7f9c3c15d7e0 2024-04-06T12:28:48.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.853+0000 7f9c40a05700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c3c0d1f80 con 0x7f9c3c0d1af0 2024-04-06T12:28:48.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.853+0000 7f9c40a05700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c0d3230 con 0x7f9c3c0d0a60 2024-04-06T12:28:48.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.854+0000 7f9c3a59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9c3c15d7e0 0x7f9c3c0d0350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51548/0 (socket says 172.21.15.189:51548) 2024-04-06T12:28:48.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.854+0000 7f9c3a59c700 1 -- 172.21.15.189:0/1277926718 learned_addr learned my addr 172.21.15.189:0/1277926718 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:48.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.854+0000 7f9c39d9b700 1 --2- 172.21.15.189:0/1277926718 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c3c0d0a60 0x7f9c3c0d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:48.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.854+0000 7f9c3ad9d700 1 --2- 172.21.15.189:0/1277926718 >> v2:172.21.15.192:3300/0 conn(0x7f9c3c0d1af0 0x7f9c3c0b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:48.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.854+0000 7f9c3959a700 1 -- 172.21.15.189:0/1277926718 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870525612 0 0) 0x7f9c3c0cdf80 con 0x7f9c3c15d7e0 2024-04-06T12:28:48.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.854+0000 7f9c3959a700 1 -- 172.21.15.189:0/1277926718 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c240035b0 con 0x7f9c3c15d7e0 2024-04-06T12:28:48.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.854+0000 7f9c39d9b700 1 -- 172.21.15.189:0/1277926718 >> v2:172.21.15.192:3300/0 conn(0x7f9c3c0d1af0 msgr2=0x7f9c3c0b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:48.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.854+0000 7f9c39d9b700 1 --2- 172.21.15.189:0/1277926718 >> v2:172.21.15.192:3300/0 conn(0x7f9c3c0d1af0 0x7f9c3c0b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.854+0000 7f9c39d9b700 1 -- 172.21.15.189:0/1277926718 >> v1:172.21.15.189:6789/0 conn(0x7f9c3c15d7e0 legacy=0x7f9c3c0d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:48.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.854+0000 7f9c39d9b700 1 -- 172.21.15.189:0/1277926718 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c3c0d2da0 con 0x7f9c3c0d0a60 2024-04-06T12:28:48.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.854+0000 7f9c3ad9d700 1 --2- 172.21.15.189:0/1277926718 >> v2:172.21.15.192:3300/0 conn(0x7f9c3c0d1af0 0x7f9c3c0b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:28:48.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.854+0000 7f9c39d9b700 1 --2- 172.21.15.189:0/1277926718 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c3c0d0a60 0x7f9c3c0d15b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009840 tx=0x7f9c30009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaa9326be9b89377 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:48.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.855+0000 7f9c3959a700 1 -- 172.21.15.189:0/1277926718 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c30011830 con 0x7f9c3c0d0a60 2024-04-06T12:28:48.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.855+0000 7f9c3959a700 1 -- 172.21.15.189:0/1277926718 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c30011990 con 0x7f9c3c0d0a60 2024-04-06T12:28:48.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.855+0000 7f9c3959a700 1 -- 172.21.15.189:0/1277926718 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c30011d20 con 0x7f9c3c0d0a60 2024-04-06T12:28:48.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.855+0000 7f9c40a05700 1 -- 172.21.15.189:0/1277926718 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c3c0d0a60 msgr2=0x7f9c3c0d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:48.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.855+0000 7f9c40a05700 1 --2- 172.21.15.189:0/1277926718 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c3c0d0a60 0x7f9c3c0d15b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009840 tx=0x7f9c30009b50 comp rx=0 tx=0).stop 2024-04-06T12:28:48.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.855+0000 7f9c40a05700 1 -- 172.21.15.189:0/1277926718 shutdown_connections 2024-04-06T12:28:48.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.855+0000 7f9c40a05700 1 --2- 172.21.15.189:0/1277926718 >> v2:172.21.15.192:3300/0 conn(0x7f9c3c0d1af0 0x7f9c3c0b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.855+0000 7f9c40a05700 1 --2- 172.21.15.189:0/1277926718 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c3c0d0a60 0x7f9c3c0d15b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.855+0000 7f9c40a05700 1 -- 172.21.15.189:0/1277926718 >> 172.21.15.189:0/1277926718 conn(0x7f9c3c072250 msgr2=0x7f9c3c072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:48.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.855+0000 7f9c40a05700 1 -- 172.21.15.189:0/1277926718 shutdown_connections 2024-04-06T12:28:48.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.855+0000 7f9c40a05700 1 -- 172.21.15.189:0/1277926718 wait complete. 2024-04-06T12:28:48.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.857+0000 7f9c40a05700 1 Processor -- start 2024-04-06T12:28:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.857+0000 7f9c40a05700 1 -- start start 2024-04-06T12:28:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.857+0000 7f9c40a05700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c3c0d0a60 0x7f9c3c0b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.857+0000 7f9c40a05700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c3c0d1af0 0x7f9c3c1491a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.857+0000 7f9c40a05700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c3c15d7e0 0x7f9c3c1496e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.857+0000 7f9c40a05700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c149e70 con 0x7f9c3c15d7e0 2024-04-06T12:28:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.857+0000 7f9c40a05700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c0c9550 con 0x7f9c3c0d1af0 2024-04-06T12:28:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.857+0000 7f9c40a05700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c14a110 con 0x7f9c3c0d0a60 2024-04-06T12:28:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.857+0000 7f9c3ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c3c15d7e0 0x7f9c3c1496e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.858+0000 7f9c3ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c3c15d7e0 0x7f9c3c1496e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53594/0 (socket says 172.21.15.189:53594) 2024-04-06T12:28:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.858+0000 7f9c3ad9d700 1 -- 172.21.15.189:0/1538199898 learned_addr learned my addr 172.21.15.189:0/1538199898 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.858+0000 7f9c39d9b700 1 --2- 172.21.15.189:0/1538199898 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c3c0d1af0 0x7f9c3c1491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:48.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.858+0000 7f9c3a59c700 1 --2- 172.21.15.189:0/1538199898 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c3c0d0a60 0x7f9c3c0b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:48.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.858+0000 7f9c3ad9d700 1 -- 172.21.15.189:0/1538199898 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c3c0d0a60 msgr2=0x7f9c3c0b3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:48.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.858+0000 7f9c3ad9d700 1 --2- 172.21.15.189:0/1538199898 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c3c0d0a60 0x7f9c3c0b3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.858+0000 7f9c3ad9d700 1 -- 172.21.15.189:0/1538199898 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c3c0d1af0 msgr2=0x7f9c3c1491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:48.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.858+0000 7f9c3ad9d700 1 --2- 172.21.15.189:0/1538199898 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c3c0d1af0 0x7f9c3c1491a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:48.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.858+0000 7f9c3ad9d700 1 -- 172.21.15.189:0/1538199898 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c34009a70 con 0x7f9c3c15d7e0 2024-04-06T12:28:48.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.858+0000 7f9c3ad9d700 1 --2- 172.21.15.189:0/1538199898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c3c15d7e0 0x7f9c3c1496e0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400efc0 tx=0x7f9c3400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:48.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.858+0000 7f9c2affd700 1 -- 172.21.15.189:0/1538199898 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3400ca90 con 0x7f9c3c15d7e0 2024-04-06T12:28:48.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.858+0000 7f9c2affd700 1 -- 172.21.15.189:0/1538199898 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c3400f040 con 0x7f9c3c15d7e0 2024-04-06T12:28:48.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.858+0000 7f9c40a05700 1 -- 172.21.15.189:0/1538199898 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c300094a0 con 0x7f9c3c15d7e0 2024-04-06T12:28:48.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.858+0000 7f9c2affd700 1 -- 172.21.15.189:0/1538199898 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3400ce30 con 0x7f9c3c15d7e0 2024-04-06T12:28:48.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.858+0000 7f9c40a05700 1 -- 172.21.15.189:0/1538199898 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c3c0c9830 con 0x7f9c3c15d7e0 2024-04-06T12:28:48.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.859+0000 7f9c40a05700 1 -- 172.21.15.189:0/1538199898 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c3c074990 con 0x7f9c3c15d7e0 2024-04-06T12:28:48.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.860+0000 7f9c2affd700 1 -- 172.21.15.189:0/1538199898 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 52) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f9c34014480 con 0x7f9c3c15d7e0 2024-04-06T12:28:48.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.860+0000 7f9c2affd700 1 --2- 172.21.15.189:0/1538199898 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c1c06ae00 0x7f9c1c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:48.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.860+0000 7f9c3a59c700 1 -- 172.21.15.189:0/1538199898 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c1c06ae00 msgr2=0x7f9c1c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:48.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.860+0000 7f9c2affd700 1 -- 172.21.15.189:0/1538199898 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9c34089d70 con 0x7f9c3c15d7e0 2024-04-06T12:28:48.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.860+0000 7f9c3a59c700 1 --2- 172.21.15.189:0/1538199898 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c1c06ae00 0x7f9c1c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:48.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:48.864+0000 7f9c2affd700 1 -- 172.21.15.189:0/1538199898 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9c34054bf0 con 0x7f9c3c15d7e0 2024-04-06T12:28:49.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.061+0000 7f9c3a59c700 1 -- 172.21.15.189:0/1538199898 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c1c06ae00 msgr2=0x7f9c1c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:49.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.061+0000 7f9c3a59c700 1 --2- 172.21.15.189:0/1538199898 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c1c06ae00 0x7f9c1c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:49.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.067+0000 7f9c40a05700 1 -- 172.21.15.189:0/1538199898 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f9c3c06a730 con 0x7f9c3c15d7e0 2024-04-06T12:28:49.068 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:49.071+0000 7fe62397a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:28:49.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.072+0000 7f9c2affd700 1 -- 172.21.15.189:0/1538199898 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v35)=0 v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9c34058210 con 0x7f9c3c15d7e0 2024-04-06T12:28:49.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.074+0000 7f9c40a05700 1 -- 172.21.15.189:0/1538199898 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c1c06ae00 msgr2=0x7f9c1c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:49.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.074+0000 7f9c40a05700 1 --2- 172.21.15.189:0/1538199898 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c1c06ae00 0x7f9c1c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.074+0000 7f9c40a05700 1 -- 172.21.15.189:0/1538199898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c3c15d7e0 msgr2=0x7f9c3c1496e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:49.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.074+0000 7f9c40a05700 1 --2- 172.21.15.189:0/1538199898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c3c15d7e0 0x7f9c3c1496e0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400efc0 tx=0x7f9c3400c3a0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.074+0000 7f9c40a05700 1 -- 172.21.15.189:0/1538199898 shutdown_connections 2024-04-06T12:28:49.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.074+0000 7f9c40a05700 1 --2- 172.21.15.189:0/1538199898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c3c15d7e0 0x7f9c3c1496e0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.074+0000 7f9c40a05700 1 --2- 172.21.15.189:0/1538199898 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c3c0d0a60 0x7f9c3c0b3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.074+0000 7f9c40a05700 1 --2- 172.21.15.189:0/1538199898 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9c1c06ae00 0x7f9c1c06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.074+0000 7f9c40a05700 1 --2- 172.21.15.189:0/1538199898 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c3c0d1af0 0x7f9c3c1491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.074+0000 7f9c40a05700 1 -- 172.21.15.189:0/1538199898 >> 172.21.15.189:0/1538199898 conn(0x7f9c3c072250 msgr2=0x7f9c3c15ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:49.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.074+0000 7f9c40a05700 1 -- 172.21.15.189:0/1538199898 shutdown_connections 2024-04-06T12:28:49.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.074+0000 7f9c40a05700 1 -- 172.21.15.189:0/1538199898 wait complete. 2024-04-06T12:28:49.085 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-06T12:28:49.103 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:49.106+0000 7f7357dbd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:28:49.147 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:49.150+0000 7fe62397a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:28:49.186 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:49.188+0000 7f7357dbd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:28:49.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.214+0000 7fbc6b693700 1 Processor -- start 2024-04-06T12:28:49.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.214+0000 7fbc6b693700 1 -- start start 2024-04-06T12:28:49.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.215+0000 7fbc6b693700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc6415d7e0 0x7fbc640d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:49.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.215+0000 7fbc6b693700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbc640d1cc0 0x7fbc640b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:49.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.215+0000 7fbc6b693700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc640cdf80 con 0x7fbc640d0890 2024-04-06T12:28:49.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.215+0000 7fbc6b693700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc640d0d10 con 0x7fbc640d1cc0 2024-04-06T12:28:49.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.215+0000 7fbc6b693700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc640d3230 con 0x7fbc6415d7e0 2024-04-06T12:28:49.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.215+0000 7fbc68c2e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbc640d0890 0x7fbc640d15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51556/0 (socket says 172.21.15.189:51556) 2024-04-06T12:28:49.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.215+0000 7fbc69c30700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbc640d1cc0 0x7fbc640b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:49.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.215+0000 7fbc68c2e700 1 -- 172.21.15.189:0/197225445 learned_addr learned my addr 172.21.15.189:0/197225445 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:49.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.215+0000 7fbc6942f700 1 --2- 172.21.15.189:0/197225445 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc6415d7e0 0x7fbc640d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:49.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.216+0000 7fbc69c30700 1 -- 172.21.15.189:0/197225445 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc6415d7e0 msgr2=0x7fbc640d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:49.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.216+0000 7fbc69c30700 1 --2- 172.21.15.189:0/197225445 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc6415d7e0 0x7fbc640d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.216+0000 7fbc69c30700 1 -- 172.21.15.189:0/197225445 >> v1:172.21.15.189:6789/0 conn(0x7fbc640d0890 legacy=0x7fbc640d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:49.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.216+0000 7fbc5bfff700 1 -- 172.21.15.189:0/197225445 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379523637 0 0) 0x7fbc640cdf80 con 0x7fbc640d0890 2024-04-06T12:28:49.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.216+0000 7fbc69c30700 1 -- 172.21.15.189:0/197225445 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc640d2da0 con 0x7fbc640d1cc0 2024-04-06T12:28:49.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.216+0000 7fbc69c30700 1 --2- 172.21.15.189:0/197225445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc640d1cc0 0x7fbc640b66b0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009d90 tx=0x7fbc5c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1311e033a8e57e99 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:49.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.216+0000 7fbc5bfff700 1 -- 172.21.15.189:0/197225445 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c00e070 con 0x7fbc640d1cc0 2024-04-06T12:28:49.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.216+0000 7fbc5bfff700 1 -- 172.21.15.189:0/197225445 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5c00bb20 con 0x7fbc640d1cc0 2024-04-06T12:28:49.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.216+0000 7fbc5bfff700 1 -- 172.21.15.189:0/197225445 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c012430 con 0x7fbc640d1cc0 2024-04-06T12:28:49.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.217+0000 7fbc6b693700 1 -- 172.21.15.189:0/197225445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc640d1cc0 msgr2=0x7fbc640b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:49.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.217+0000 7fbc6b693700 1 --2- 172.21.15.189:0/197225445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc640d1cc0 0x7fbc640b66b0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009d90 tx=0x7fbc5c00b330 comp rx=0 tx=0).stop 2024-04-06T12:28:49.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.217+0000 7fbc6b693700 1 -- 172.21.15.189:0/197225445 shutdown_connections 2024-04-06T12:28:49.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.217+0000 7fbc6b693700 1 --2- 172.21.15.189:0/197225445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc640d1cc0 0x7fbc640b66b0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.217+0000 7fbc6b693700 1 --2- 172.21.15.189:0/197225445 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc6415d7e0 0x7fbc640d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.217+0000 7fbc6b693700 1 -- 172.21.15.189:0/197225445 >> 172.21.15.189:0/197225445 conn(0x7fbc64072250 msgr2=0x7fbc64072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:49.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.217+0000 7fbc6b693700 1 -- 172.21.15.189:0/197225445 shutdown_connections 2024-04-06T12:28:49.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.217+0000 7fbc6b693700 1 -- 172.21.15.189:0/197225445 wait complete. 2024-04-06T12:28:49.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.218+0000 7fbc6b693700 1 Processor -- start 2024-04-06T12:28:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.218+0000 7fbc6b693700 1 -- start start 2024-04-06T12:28:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc6b693700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc640d0890 0x7fbc640ca710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc6b693700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc640d1cc0 0x7fbc640b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc6b693700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc6415d7e0 0x7fbc640b4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc6b693700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc640b55f0 con 0x7fbc6415d7e0 2024-04-06T12:28:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc6b693700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc640b47a0 con 0x7fbc640d0890 2024-04-06T12:28:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc6b693700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc640b49e0 con 0x7fbc640d1cc0 2024-04-06T12:28:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc69c30700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc6415d7e0 0x7fbc640b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc69c30700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc6415d7e0 0x7fbc640b4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53614/0 (socket says 172.21.15.189:53614) 2024-04-06T12:28:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc69c30700 1 -- 172.21.15.189:0/196311408 learned_addr learned my addr 172.21.15.189:0/196311408 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc68c2e700 1 --2- 172.21.15.189:0/196311408 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc640d1cc0 0x7fbc640b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc6942f700 1 --2- 172.21.15.189:0/196311408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc640d0890 0x7fbc640ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc69c30700 1 -- 172.21.15.189:0/196311408 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc640d1cc0 msgr2=0x7fbc640b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc69c30700 1 --2- 172.21.15.189:0/196311408 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc640d1cc0 0x7fbc640b3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc69c30700 1 -- 172.21.15.189:0/196311408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc640d0890 msgr2=0x7fbc640ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc69c30700 1 --2- 172.21.15.189:0/196311408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc640d0890 0x7fbc640ca710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc69c30700 1 -- 172.21.15.189:0/196311408 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c009a70 con 0x7fbc6415d7e0 2024-04-06T12:28:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc69c30700 1 --2- 172.21.15.189:0/196311408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc6415d7e0 0x7fbc640b4260 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fbc6406b8c0 tx=0x7fbc5c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc59ffb700 1 -- 172.21.15.189:0/196311408 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c012430 con 0x7fbc6415d7e0 2024-04-06T12:28:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc59ffb700 1 -- 172.21.15.189:0/196311408 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5c020070 con 0x7fbc6415d7e0 2024-04-06T12:28:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc6b693700 1 -- 172.21.15.189:0/196311408 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc640b4cc0 con 0x7fbc6415d7e0 2024-04-06T12:28:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.219+0000 7fbc59ffb700 1 -- 172.21.15.189:0/196311408 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c004280 con 0x7fbc6415d7e0 2024-04-06T12:28:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.220+0000 7fbc6b693700 1 -- 172.21.15.189:0/196311408 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc640b5280 con 0x7fbc6415d7e0 2024-04-06T12:28:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.221+0000 7fbc6b693700 1 -- 172.21.15.189:0/196311408 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc64074990 con 0x7fbc6415d7e0 2024-04-06T12:28:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.221+0000 7fbc59ffb700 1 -- 172.21.15.189:0/196311408 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 52) v1 ==== 89264+0+0 (secure 0 0 0) 0x7fbc5c0043e0 con 0x7fbc6415d7e0 2024-04-06T12:28:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.221+0000 7fbc59ffb700 1 --2- 172.21.15.189:0/196311408 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc4806ae30 0x7fbc4806d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:49.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.221+0000 7fbc6942f700 1 -- 172.21.15.189:0/196311408 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc4806ae30 msgr2=0x7fbc4806d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:49.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.221+0000 7fbc59ffb700 1 -- 172.21.15.189:0/196311408 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fbc5c022070 con 0x7fbc6415d7e0 2024-04-06T12:28:49.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.221+0000 7fbc6942f700 1 --2- 172.21.15.189:0/196311408 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc4806ae30 0x7fbc4806d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:49.222 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:49.222+0000 7fe62397a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:28:49.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.226+0000 7fbc59ffb700 1 -- 172.21.15.189:0/196311408 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbc5c0565e0 con 0x7fbc6415d7e0 2024-04-06T12:28:49.265 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:49.267+0000 7f7357dbd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:28:49.288 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:49.291+0000 7fe62397a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:28:49.335 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:49.337+0000 7f7357dbd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:28:49.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.422+0000 7fbc6942f700 1 -- 172.21.15.189:0/196311408 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc4806ae30 msgr2=0x7fbc4806d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:49.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.422+0000 7fbc6942f700 1 --2- 172.21.15.189:0/196311408 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc4806ae30 0x7fbc4806d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:49.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.428+0000 7fbc6b693700 1 -- 172.21.15.189:0/196311408 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fbc640cdc40 con 0x7fbc6415d7e0 2024-04-06T12:28:49.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.433+0000 7fbc59ffb700 1 -- 172.21.15.189:0/196311408 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v36)=0 v36) v1 ==== 133+0+0 (secure 0 0 0) 0x7fbc5c089e60 con 0x7fbc6415d7e0 2024-04-06T12:28:49.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.436+0000 7fbc6b693700 1 -- 172.21.15.189:0/196311408 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc4806ae30 msgr2=0x7fbc4806d2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:49.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.436+0000 7fbc6b693700 1 --2- 172.21.15.189:0/196311408 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc4806ae30 0x7fbc4806d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.436+0000 7fbc6b693700 1 -- 172.21.15.189:0/196311408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc6415d7e0 msgr2=0x7fbc640b4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:49.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.436+0000 7fbc6b693700 1 --2- 172.21.15.189:0/196311408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc6415d7e0 0x7fbc640b4260 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fbc6406b8c0 tx=0x7fbc5c004ad0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.436+0000 7fbc6b693700 1 -- 172.21.15.189:0/196311408 shutdown_connections 2024-04-06T12:28:49.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.436+0000 7fbc6b693700 1 --2- 172.21.15.189:0/196311408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc6415d7e0 0x7fbc640b4260 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.436+0000 7fbc6b693700 1 --2- 172.21.15.189:0/196311408 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbc4806ae30 0x7fbc4806d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.436+0000 7fbc6b693700 1 --2- 172.21.15.189:0/196311408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc640d0890 0x7fbc640ca710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.436+0000 7fbc6b693700 1 --2- 172.21.15.189:0/196311408 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc640d1cc0 0x7fbc640b3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.436+0000 7fbc6b693700 1 -- 172.21.15.189:0/196311408 >> 172.21.15.189:0/196311408 conn(0x7fbc64072250 msgr2=0x7fbc6415ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:49.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.436+0000 7fbc6b693700 1 -- 172.21.15.189:0/196311408 shutdown_connections 2024-04-06T12:28:49.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.436+0000 7fbc6b693700 1 -- 172.21.15.189:0/196311408 wait complete. 2024-04-06T12:28:49.447 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /bad-path 1m 2024-04-06T12:28:49.464 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:49.467+0000 7fe62397a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:28:49.519 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:49.521+0000 7f7357dbd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:28:49.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.617+0000 7fa8fe87a700 1 Processor -- start 2024-04-06T12:28:49.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.617+0000 7fa8fe87a700 1 -- start start 2024-04-06T12:28:49.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.618+0000 7fa8fe87a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa8f815d590 0x7fa8f80b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:49.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.618+0000 7fa8fe87a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8f80b4ed0 0x7fa8f80b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:49.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.618+0000 7fa8fe87a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f80b5cc0 con 0x7fa8f80b6580 2024-04-06T12:28:49.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.618+0000 7fa8fe87a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8f80b5890 con 0x7fa8f815d590 2024-04-06T12:28:49.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.618+0000 7fa8fe87a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f80d1500 con 0x7fa8f80b4ed0 2024-04-06T12:28:49.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.618+0000 7fa8f7fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa8f815d590 0x7fa8f80b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:49.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.618+0000 7fa8f7fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa8f815d590 0x7fa8f80b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45040/0 (socket says 172.21.15.189:45040) 2024-04-06T12:28:49.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.618+0000 7fa8f7fff700 1 -- 172.21.15.189:0/2595868692 learned_addr learned my addr 172.21.15.189:0/2595868692 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:49.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.618+0000 7fa8fce17700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa8f80b6580 0x7fa8f80b6a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40554/0 (socket says 172.21.15.189:40554) 2024-04-06T12:28:49.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.619+0000 7fa8f77fe700 1 --2- 172.21.15.189:0/2595868692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8f80b4ed0 0x7fa8f80b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:49.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.619+0000 7fa8f7fff700 1 -- 172.21.15.189:0/2595868692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8f80b4ed0 msgr2=0x7fa8f80b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:28:49.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.619+0000 7fa8f7fff700 1 --2- 172.21.15.189:0/2595868692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8f80b4ed0 0x7fa8f80b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.619+0000 7fa8f7fff700 1 -- 172.21.15.189:0/2595868692 >> v1:172.21.15.189:6789/0 conn(0x7fa8f80b6580 legacy=0x7fa8f80b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:49.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.619+0000 7fa8f7fff700 1 -- 172.21.15.189:0/2595868692 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f80d17e0 con 0x7fa8f815d590 2024-04-06T12:28:49.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.619+0000 7fa8f6ffd700 1 -- 172.21.15.189:0/2595868692 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329295615 0 0) 0x7fa8f80b5cc0 con 0x7fa8f80b6580 2024-04-06T12:28:49.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.619+0000 7fa8f7fff700 1 --2- 172.21.15.189:0/2595868692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8f815d590 0x7fa8f80b8d70 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8009920 tx=0x7fa8e8009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e18de496cf144dd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:49.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.619+0000 7fa8f6ffd700 1 -- 172.21.15.189:0/2595868692 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8011830 con 0x7fa8f815d590 2024-04-06T12:28:49.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.619+0000 7fa8f6ffd700 1 -- 172.21.15.189:0/2595868692 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e8011990 con 0x7fa8f815d590 2024-04-06T12:28:49.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.619+0000 7fa8f6ffd700 1 -- 172.21.15.189:0/2595868692 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8011d20 con 0x7fa8f815d590 2024-04-06T12:28:49.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.620+0000 7fa8fe87a700 1 -- 172.21.15.189:0/2595868692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8f815d590 msgr2=0x7fa8f80b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:49.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.620+0000 7fa8fe87a700 1 --2- 172.21.15.189:0/2595868692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8f815d590 0x7fa8f80b8d70 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8009920 tx=0x7fa8e8009c30 comp rx=0 tx=0).stop 2024-04-06T12:28:49.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.620+0000 7fa8fe87a700 1 -- 172.21.15.189:0/2595868692 shutdown_connections 2024-04-06T12:28:49.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.620+0000 7fa8fe87a700 1 --2- 172.21.15.189:0/2595868692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8f815d590 0x7fa8f80b8d70 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.620+0000 7fa8fe87a700 1 --2- 172.21.15.189:0/2595868692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8f80b4ed0 0x7fa8f80b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.620+0000 7fa8fe87a700 1 -- 172.21.15.189:0/2595868692 >> 172.21.15.189:0/2595868692 conn(0x7fa8f806bbc0 msgr2=0x7fa8f806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:49.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.620+0000 7fa8fe87a700 1 -- 172.21.15.189:0/2595868692 shutdown_connections 2024-04-06T12:28:49.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.620+0000 7fa8fe87a700 1 -- 172.21.15.189:0/2595868692 wait complete. 2024-04-06T12:28:49.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.622+0000 7fa8fe87a700 1 Processor -- start 2024-04-06T12:28:49.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.622+0000 7fa8fe87a700 1 -- start start 2024-04-06T12:28:49.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.623+0000 7fa8fe87a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8f80b4ed0 0x7fa8f8153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:49.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.623+0000 7fa8fe87a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8f80b6580 0x7fa8f80b8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:49.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.623+0000 7fa8fe87a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8f815d590 0x7fa8f80b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:49.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.623+0000 7fa8fe87a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f80cb4a0 con 0x7fa8f80b4ed0 2024-04-06T12:28:49.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.623+0000 7fa8fe87a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f80cb1c0 con 0x7fa8f815d590 2024-04-06T12:28:49.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.623+0000 7fa8fe87a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f815da10 con 0x7fa8f80b6580 2024-04-06T12:28:49.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.623+0000 7fa8f7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8f80b4ed0 0x7fa8f8153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:49.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.623+0000 7fa8f7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8f80b4ed0 0x7fa8f8153580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45056/0 (socket says 172.21.15.189:45056) 2024-04-06T12:28:49.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.623+0000 7fa8f7fff700 1 -- 172.21.15.189:0/2277577439 learned_addr learned my addr 172.21.15.189:0/2277577439 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:49.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.623+0000 7fa8fce17700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8f815d590 0x7fa8f80b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:49.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.623+0000 7fa8f77fe700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8f80b6580 0x7fa8f80b8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:49.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.624+0000 7fa8f7fff700 1 -- 172.21.15.189:0/2277577439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8f80b6580 msgr2=0x7fa8f80b8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:49.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.624+0000 7fa8f7fff700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8f80b6580 0x7fa8f80b8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.624+0000 7fa8f7fff700 1 -- 172.21.15.189:0/2277577439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8f815d590 msgr2=0x7fa8f80b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:49.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.624+0000 7fa8f7fff700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8f815d590 0x7fa8f80b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:49.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.624+0000 7fa8f7fff700 1 -- 172.21.15.189:0/2277577439 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e8009550 con 0x7fa8f80b4ed0 2024-04-06T12:28:49.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.624+0000 7fa8f7fff700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8f80b4ed0 0x7fa8f8153580 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8000c00 tx=0x7fa8e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:49.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.624+0000 7fa8f4ff9700 1 -- 172.21.15.189:0/2277577439 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8004510 con 0x7fa8f80b4ed0 2024-04-06T12:28:49.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.624+0000 7fa8f4ff9700 1 -- 172.21.15.189:0/2277577439 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e801e070 con 0x7fa8f80b4ed0 2024-04-06T12:28:49.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.624+0000 7fa8fe87a700 1 -- 172.21.15.189:0/2277577439 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f815dcf0 con 0x7fa8f80b4ed0 2024-04-06T12:28:49.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.624+0000 7fa8f4ff9700 1 -- 172.21.15.189:0/2277577439 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e80038e0 con 0x7fa8f80b4ed0 2024-04-06T12:28:49.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.624+0000 7fa8fe87a700 1 -- 172.21.15.189:0/2277577439 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f81544e0 con 0x7fa8f80b4ed0 2024-04-06T12:28:49.625 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:49.624+0000 7fe62397a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:28:49.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.625+0000 7fa8fe87a700 1 -- 172.21.15.189:0/2277577439 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f8074990 con 0x7fa8f80b4ed0 2024-04-06T12:28:49.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.625+0000 7fa8f4ff9700 1 -- 172.21.15.189:0/2277577439 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 52) v1 ==== 89264+0+0 (secure 0 0 0) 0x7fa8e8003a40 con 0x7fa8f80b4ed0 2024-04-06T12:28:49.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.626+0000 7fa8f4ff9700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8e406ae00 0x7fa8e406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:49.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.626+0000 7fa8f77fe700 1 -- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8e406ae00 msgr2=0x7fa8e406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:49.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.626+0000 7fa8f4ff9700 1 -- 172.21.15.189:0/2277577439 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa8e808a460 con 0x7fa8f80b4ed0 2024-04-06T12:28:49.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.626+0000 7fa8f77fe700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8e406ae00 0x7fa8e406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:28:49.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.630+0000 7fa8f4ff9700 1 -- 172.21.15.189:0/2277577439 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa8e80553e0 con 0x7fa8f80b4ed0 2024-04-06T12:28:49.686 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:49.687+0000 7f7357dbd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:28:49.733 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:49.736+0000 7fe62397a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:28:49.799 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:49.801+0000 7f7357dbd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:28:49.814 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:49.817+0000 7fe62397a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:28:49.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.827+0000 7fa8f77fe700 1 -- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8e406ae00 msgr2=0x7fa8e406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:49.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.827+0000 7fa8f77fe700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8e406ae00 0x7fa8e406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:28:49.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:49.863+0000 7fa8fe87a700 1 -- 172.21.15.189:0/2277577439 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8f80d2080 con 0x7fa8e406ae00 2024-04-06T12:28:49.882 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:49.884+0000 7f7357dbd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:28:49.888 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:49.891+0000 7fe62397a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:28:49.957 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:49.959+0000 7f7357dbd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:28:50.091 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:50.094+0000 7fe62397a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:28:50.161 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:50.162+0000 7f7357dbd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:28:50.168 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:50.171+0000 7fe62397a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:28:50.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:50.228+0000 7fa8f77fe700 1 -- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8e406ae00 msgr2=0x7fa8e406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:50.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:50.228+0000 7fa8f77fe700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8e406ae00 0x7fa8e406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-06T12:28:50.241 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:50.237+0000 7f7357dbd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:28:50.486 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:50.489+0000 7fe62397a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:28:50.553 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:50.555+0000 7f7357dbd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:28:50.804 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:50.807+0000 7fe62397a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:28:50.873 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:50.875+0000 7f7357dbd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:28:50.963 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:50.966+0000 7fe62397a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:28:51.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:51.029+0000 7fa8f77fe700 1 -- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8e406ae00 msgr2=0x7fa8e406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:51.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:51.029+0000 7fa8f77fe700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8e406ae00 0x7fa8e406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-06T12:28:51.031 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:51.033+0000 7f7357dbd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:28:51.035 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:51.038+0000 7fe62397a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:28:51.103 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:51.105+0000 7f7357dbd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:28:51.650 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:51.653+0000 7fe62397a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:28:51.718 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:51.720+0000 7f7357dbd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:28:51.727 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:51.730+0000 7fe62397a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:28:51.797 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:51.799+0000 7f7357dbd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:28:51.853 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:51.856+0000 7fe62397a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:28:51.921 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:51.923+0000 7f7357dbd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:28:52.360 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:52.362+0000 7fe62397a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:28:52.428 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:52.430+0000 7f7357dbd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:28:52.513 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:52.515+0000 7fe62397a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:28:52.582 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:52.584+0000 7f7357dbd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:28:52.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:52.630+0000 7fa8f77fe700 1 -- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8e406ae00 msgr2=0x7fa8e406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:28:52.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:52.630+0000 7fa8f77fe700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8e406ae00 0x7fa8e406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-06T12:28:52.667 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:52.669+0000 7fe62397a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:28:52.739 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:52.741+0000 7f7357dbd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:28:53.386 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:53.388+0000 7fe62397a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:28:53.462 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:53.464+0000 7f7357dbd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:28:53.624 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:53.626+0000 7fe62397a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:28:53.699 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:53.701+0000 7fe62397a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:28:53.706 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:53.708+0000 7f7357dbd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:28:53.783 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:53.785+0000 7f7357dbd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:28:54.069 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:28:54.071+0000 7fe62397a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:28:54.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:54.140+0000 7fa8f4ff9700 1 -- 172.21.15.189:0/2277577439 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 53) v1 ==== 89264+0+0 (secure 0 0 0) 0x7fa8e8003cf0 con 0x7fa8f80b4ed0 2024-04-06T12:28:54.159 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:54.161+0000 7f7357dbd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:28:54.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:54.182+0000 7fa8f4ff9700 1 -- 172.21.15.189:0/2277577439 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mgrmap(e 54) v1 ==== 88873+0+0 (secure 0 0 0) 0x7fa8e80557b0 con 0x7fa8f80b4ed0 2024-04-06T12:28:54.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:54.183+0000 7fa8f4ff9700 1 -- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8e406ae00 msgr2=0x7fa8e406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:28:54.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:54.183+0000 7fa8f4ff9700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8e406ae00 0x7fa8e406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:54.343 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:54.345+0000 7f72caccb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:54.343 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:54.345+0000 7f72caccb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:54.344 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:54.345+0000 7f72caccb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:54.344 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:54.345+0000 7f72caccb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:54.344 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:54.345+0000 7f72caccb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:54.344 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:54.345+0000 7f72cdcd1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:54.344 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:54.346+0000 7f72cdcd1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:54.344 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:54.346+0000 7f72cdcd1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:54.344 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:54.346+0000 7f72cdcd1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:54.344 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:28:54.346+0000 7f72cdcd1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:28:55.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.186+0000 7fa8f4ff9700 1 -- 172.21.15.189:0/2277577439 <== mon.0 v2:172.21.15.189:3300/0 9 ==== mgrmap(e 55) v1 ==== 89000+0+0 (secure 0 0 0) 0x7fa8e8058e70 con 0x7fa8f80b4ed0 2024-04-06T12:28:55.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.186+0000 7fa8f4ff9700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8e406ae00 0x7fa8e406d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:55.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.186+0000 7fa8f4ff9700 1 -- 172.21.15.189:0/2277577439 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8e4069a70 con 0x7fa8e406ae00 2024-04-06T12:28:55.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.188+0000 7fa8f77fe700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8e406ae00 0x7fa8e406d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:55.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.191+0000 7fa8f77fe700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8e406ae00 0x7fa8e406d240 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa8f80d3fa0 tx=0x7fa8ec009210 comp rx=0 tx=0).ready entity=mgr.7195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:55.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.253+0000 7fa8f4ff9700 1 -- 172.21.15.189:0/2277577439 <== mgr.7195 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fa8ec009c00 con 0x7fa8e406ae00 2024-04-06T12:28:55.252 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path /bad-path 2024-04-06T12:28:55.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.256+0000 7fa8fe87a700 1 -- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8e406ae00 msgr2=0x7fa8e406d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:55.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.256+0000 7fa8fe87a700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8e406ae00 0x7fa8e406d240 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa8f80d3fa0 tx=0x7fa8ec009210 comp rx=0 tx=0).stop 2024-04-06T12:28:55.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.256+0000 7fa8fe87a700 1 -- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8f80b4ed0 msgr2=0x7fa8f8153580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:55.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.256+0000 7fa8fe87a700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8f80b4ed0 0x7fa8f8153580 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8000c00 tx=0x7fa8e8004750 comp rx=0 tx=0).stop 2024-04-06T12:28:55.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.256+0000 7fa8fe87a700 1 -- 172.21.15.189:0/2277577439 shutdown_connections 2024-04-06T12:28:55.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.256+0000 7fa8fe87a700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8e406ae00 0x7fa8e406d240 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:55.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.256+0000 7fa8fe87a700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8f815d590 0x7fa8f80b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:55.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.256+0000 7fa8fe87a700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8f80b4ed0 0x7fa8f8153580 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:55.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.256+0000 7fa8fe87a700 1 --2- 172.21.15.189:0/2277577439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8f80b6580 0x7fa8f80b8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:55.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.256+0000 7fa8fe87a700 1 -- 172.21.15.189:0/2277577439 >> 172.21.15.189:0/2277577439 conn(0x7fa8f806bbc0 msgr2=0x7fa8f806bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:55.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.256+0000 7fa8fe87a700 1 -- 172.21.15.189:0/2277577439 shutdown_connections 2024-04-06T12:28:55.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.256+0000 7fa8fe87a700 1 -- 172.21.15.189:0/2277577439 wait complete. 2024-04-06T12:28:55.268 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2024-04-06T12:28:55.268 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2024-04-06T12:28:55.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.438+0000 7fa4bf98d700 1 Processor -- start 2024-04-06T12:28:55.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.438+0000 7fa4bf98d700 1 -- start start 2024-04-06T12:28:55.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.439+0000 7fa4bf98d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa4b80b4660 0x7fa4b80b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:55.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.439+0000 7fa4bf98d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b80b5030 con 0x7fa4b80b2fc0 2024-04-06T12:28:55.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.439+0000 7fa4bf98d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b80cdbe0 con 0x7fa4b815a550 2024-04-06T12:28:55.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.439+0000 7fa4bf98d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4b815a940 con 0x7fa4b80b4660 2024-04-06T12:28:55.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.439+0000 7fa4bdf2a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa4b80b2fc0 0x7fa4b80b33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40558/0 (socket says 172.21.15.189:40558) 2024-04-06T12:28:55.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.439+0000 7fa4bdf2a700 1 -- 172.21.15.189:0/4224169973 learned_addr learned my addr 172.21.15.189:0/4224169973 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:28:55.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.439+0000 7fa4bcf28700 1 --2- 172.21.15.189:0/4224169973 >> v2:172.21.15.192:3300/0 conn(0x7fa4b80b4660 0x7fa4b80b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:55.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.439+0000 7fa4affff700 1 -- 172.21.15.189:0/4224169973 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635233073 0 0) 0x7fa4b80b5030 con 0x7fa4b80b2fc0 2024-04-06T12:28:55.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.439+0000 7fa4affff700 1 -- 172.21.15.189:0/4224169973 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a40035b0 con 0x7fa4b80b2fc0 2024-04-06T12:28:55.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.439+0000 7fa4bcf28700 1 -- 172.21.15.189:0/4224169973 >> v1:172.21.15.192:6789/0 conn(0x7fa4b815a550 legacy=0x7fa4b80cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:55.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.439+0000 7fa4bcf28700 1 -- 172.21.15.189:0/4224169973 >> v1:172.21.15.189:6789/0 conn(0x7fa4b80b2fc0 legacy=0x7fa4b80b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:55.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.439+0000 7fa4affff700 1 -- 172.21.15.189:0/4224169973 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492724192 0 0) 0x7fa4b80cdbe0 con 0x7fa4b815a550 2024-04-06T12:28:55.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.439+0000 7fa4bcf28700 1 -- 172.21.15.189:0/4224169973 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b815ac00 con 0x7fa4b80b4660 2024-04-06T12:28:55.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.439+0000 7fa4bcf28700 1 --2- 172.21.15.189:0/4224169973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4b80b4660 0x7fa4b80b4af0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009840 tx=0x7fa4a8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c8105bfc33e4efc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:55.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.440+0000 7fa4affff700 1 -- 172.21.15.189:0/4224169973 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b00019f0 con 0x7fa4b80b4660 2024-04-06T12:28:55.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.440+0000 7fa4affff700 1 -- 172.21.15.189:0/4224169973 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a8004030 con 0x7fa4b80b4660 2024-04-06T12:28:55.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.440+0000 7fa4affff700 1 -- 172.21.15.189:0/4224169973 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a80043a0 con 0x7fa4b80b4660 2024-04-06T12:28:55.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.440+0000 7fa4bf98d700 1 -- 172.21.15.189:0/4224169973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4b80b4660 msgr2=0x7fa4b80b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:55.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.440+0000 7fa4bf98d700 1 --2- 172.21.15.189:0/4224169973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4b80b4660 0x7fa4b80b4af0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009840 tx=0x7fa4a8009b50 comp rx=0 tx=0).stop 2024-04-06T12:28:55.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.440+0000 7fa4bf98d700 1 -- 172.21.15.189:0/4224169973 shutdown_connections 2024-04-06T12:28:55.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.440+0000 7fa4bf98d700 1 --2- 172.21.15.189:0/4224169973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4b80b4660 0x7fa4b80b4af0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:55.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.440+0000 7fa4bf98d700 1 -- 172.21.15.189:0/4224169973 >> 172.21.15.189:0/4224169973 conn(0x7fa4b80d00b0 msgr2=0x7fa4b80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:55.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.440+0000 7fa4bf98d700 1 -- 172.21.15.189:0/4224169973 shutdown_connections 2024-04-06T12:28:55.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.440+0000 7fa4bf98d700 1 -- 172.21.15.189:0/4224169973 wait complete. 2024-04-06T12:28:55.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.441+0000 7fa4bf98d700 1 Processor -- start 2024-04-06T12:28:55.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.441+0000 7fa4bf98d700 1 -- start start 2024-04-06T12:28:55.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.442+0000 7fa4bf98d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4b80b2fc0 0x7fa4b81561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:55.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.442+0000 7fa4bf98d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4b80b4660 0x7fa4b80aba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:55.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.442+0000 7fa4bf98d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4b81582c0 0x7fa4b806eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:55.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.442+0000 7fa4bf98d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b80af130 con 0x7fa4b80b2fc0 2024-04-06T12:28:55.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.442+0000 7fa4bf98d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b80aa6c0 con 0x7fa4b81582c0 2024-04-06T12:28:55.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.442+0000 7fa4bf98d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b80aa9a0 con 0x7fa4b80b4660 2024-04-06T12:28:55.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.442+0000 7fa4bd729700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4b80b2fc0 0x7fa4b81561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:55.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.442+0000 7fa4bd729700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4b80b2fc0 0x7fa4b81561f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45192/0 (socket says 172.21.15.189:45192) 2024-04-06T12:28:55.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.442+0000 7fa4bd729700 1 -- 172.21.15.189:0/260210541 learned_addr learned my addr 172.21.15.189:0/260210541 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:28:55.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.443+0000 7fa4bdf2a700 1 --2- 172.21.15.189:0/260210541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4b81582c0 0x7fa4b806eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:55.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.442+0000 7fa4bcf28700 1 --2- 172.21.15.189:0/260210541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4b80b4660 0x7fa4b80aba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:55.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.443+0000 7fa4bd729700 1 -- 172.21.15.189:0/260210541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4b80b4660 msgr2=0x7fa4b80aba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:55.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.443+0000 7fa4bd729700 1 --2- 172.21.15.189:0/260210541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4b80b4660 0x7fa4b80aba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:55.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.443+0000 7fa4bd729700 1 -- 172.21.15.189:0/260210541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4b81582c0 msgr2=0x7fa4b806eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:55.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.443+0000 7fa4bd729700 1 --2- 172.21.15.189:0/260210541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4b81582c0 0x7fa4b806eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:55.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.443+0000 7fa4bd729700 1 -- 172.21.15.189:0/260210541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a80094a0 con 0x7fa4b80b2fc0 2024-04-06T12:28:55.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.443+0000 7fa4bcf28700 1 --2- 172.21.15.189:0/260210541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4b80b4660 0x7fa4b80aba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:55.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.443+0000 7fa4bdf2a700 1 --2- 172.21.15.189:0/260210541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4b81582c0 0x7fa4b806eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:28:55.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.443+0000 7fa4bd729700 1 --2- 172.21.15.189:0/260210541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4b80b2fc0 0x7fa4b81561f0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400c450 tx=0x7fa4b400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:55.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.443+0000 7fa4adffb700 1 -- 172.21.15.189:0/260210541 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b40054e0 con 0x7fa4b80b2fc0 2024-04-06T12:28:55.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.443+0000 7fa4adffb700 1 -- 172.21.15.189:0/260210541 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b4003680 con 0x7fa4b80b2fc0 2024-04-06T12:28:55.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.443+0000 7fa4bf98d700 1 -- 172.21.15.189:0/260210541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b80aac80 con 0x7fa4b80b2fc0 2024-04-06T12:28:55.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.443+0000 7fa4adffb700 1 -- 172.21.15.189:0/260210541 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4003a30 con 0x7fa4b80b2fc0 2024-04-06T12:28:55.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.443+0000 7fa4bf98d700 1 -- 172.21.15.189:0/260210541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b80ab240 con 0x7fa4b80b2fc0 2024-04-06T12:28:55.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.444+0000 7fa4bf98d700 1 -- 172.21.15.189:0/260210541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b8067bb0 con 0x7fa4b80b2fc0 2024-04-06T12:28:55.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.445+0000 7fa4adffb700 1 -- 172.21.15.189:0/260210541 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 55) v1 ==== 89000+0+0 (secure 0 0 0) 0x7fa4b4003b90 con 0x7fa4b80b2fc0 2024-04-06T12:28:55.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.445+0000 7fa4adffb700 1 --2- 172.21.15.189:0/260210541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa49c06ab30 0x7fa49c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:28:55.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.445+0000 7fa4bcf28700 1 --2- 172.21.15.189:0/260210541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa49c06ab30 0x7fa49c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:28:55.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.445+0000 7fa4adffb700 1 -- 172.21.15.189:0/260210541 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fa4b4088710 con 0x7fa4b80b2fc0 2024-04-06T12:28:55.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.446+0000 7fa4bcf28700 1 --2- 172.21.15.189:0/260210541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa49c06ab30 0x7fa49c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8000c00 tx=0x7fa4a8008000 comp rx=0 tx=0).ready entity=mgr.7195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:28:55.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.450+0000 7fa4adffb700 1 -- 172.21.15.189:0/260210541 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa4b4053990 con 0x7fa4b80b2fc0 2024-04-06T12:28:55.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.669+0000 7fa4bf98d700 1 -- 172.21.15.189:0/260210541 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa4b815b4a0 con 0x7fa49c06ab30 2024-04-06T12:28:55.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.679+0000 7fa4adffb700 1 -- 172.21.15.189:0/260210541 <== mgr.7195 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+302 (secure 0 0 0) 0x7fa4a8021070 con 0x7fa49c06ab30 2024-04-06T12:28:55.677 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:28:55.677 INFO:teuthology.orchestra.run.smithi189.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:28:55", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-04-06T12:28:55.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.681+0000 7fa4bf98d700 1 -- 172.21.15.189:0/260210541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa49c06ab30 msgr2=0x7fa49c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:55.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.681+0000 7fa4bf98d700 1 --2- 172.21.15.189:0/260210541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa49c06ab30 0x7fa49c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8000c00 tx=0x7fa4a8008000 comp rx=0 tx=0).stop 2024-04-06T12:28:55.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.681+0000 7fa4bf98d700 1 -- 172.21.15.189:0/260210541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4b80b2fc0 msgr2=0x7fa4b81561f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:28:55.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.681+0000 7fa4bf98d700 1 --2- 172.21.15.189:0/260210541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4b80b2fc0 0x7fa4b81561f0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400c450 tx=0x7fa4b400c810 comp rx=0 tx=0).stop 2024-04-06T12:28:55.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.681+0000 7fa4bf98d700 1 -- 172.21.15.189:0/260210541 shutdown_connections 2024-04-06T12:28:55.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.681+0000 7fa4bf98d700 1 --2- 172.21.15.189:0/260210541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa49c06ab30 0x7fa49c06cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:55.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.681+0000 7fa4bf98d700 1 --2- 172.21.15.189:0/260210541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4b81582c0 0x7fa4b806eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:55.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.681+0000 7fa4bf98d700 1 --2- 172.21.15.189:0/260210541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4b80b2fc0 0x7fa4b81561f0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:55.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.681+0000 7fa4bf98d700 1 --2- 172.21.15.189:0/260210541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4b80b4660 0x7fa4b80aba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:28:55.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.681+0000 7fa4bf98d700 1 -- 172.21.15.189:0/260210541 >> 172.21.15.189:0/260210541 conn(0x7fa4b80d00b0 msgr2=0x7fa4b80ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:28:55.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.681+0000 7fa4bf98d700 1 -- 172.21.15.189:0/260210541 shutdown_connections 2024-04-06T12:28:55.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:28:55.681+0000 7fa4bf98d700 1 -- 172.21.15.189:0/260210541 wait complete. 2024-04-06T12:28:55.692 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:28:55", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-04-06T12:29:00.262 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:29:00.264+0000 7f72bd4b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:29:00.263 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:29:00.264+0000 7f72bd4b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:29:00.263 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:29:00.264+0000 7f72bd4b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:29:00.263 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:29:00.264+0000 7f72bd4b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:29:00.263 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:29:00.264+0000 7f72bd4b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:29:12.803 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:29:12.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:29:42.871 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:29:42.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:29:55.693 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2024-04-06T12:29:55.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.830+0000 7fdf48cba700 1 Processor -- start 2024-04-06T12:29:55.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.830+0000 7fdf48cba700 1 -- start start 2024-04-06T12:29:55.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.831+0000 7fdf48cba700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdf4415d7b0 0x7fdf440b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:55.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.831+0000 7fdf48cba700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdf4415e500 0x7fdf4406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:55.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.831+0000 7fdf48cba700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf440b3990 con 0x7fdf4406a730 2024-04-06T12:29:55.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.831+0000 7fdf48cba700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4415e950 con 0x7fdf4415d7b0 2024-04-06T12:29:55.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.831+0000 7fdf48cba700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4406b0f0 con 0x7fdf4415e500 2024-04-06T12:29:55.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.831+0000 7fdf4259c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdf4415d7b0 0x7fdf440b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:55.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.831+0000 7fdf41d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdf4406a730 0x7fdf4415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47500/0 (socket says 172.21.15.189:47500) 2024-04-06T12:29:55.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.831+0000 7fdf41d9b700 1 -- 172.21.15.189:0/955721767 learned_addr learned my addr 172.21.15.189:0/955721767 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:29:55.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.831+0000 7fdf4259c700 1 --2- 172.21.15.189:0/955721767 >> v2:172.21.15.189:3300/0 conn(0x7fdf4415d7b0 0x7fdf440b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41836/0 (socket says 172.21.15.189:41836) 2024-04-06T12:29:55.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.831+0000 7fdf42d9d700 1 --2- 172.21.15.189:0/955721767 >> v2:172.21.15.192:3300/0 conn(0x7fdf4415e500 0x7fdf4406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:55.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.831+0000 7fdf4259c700 1 -- 172.21.15.189:0/955721767 >> v2:172.21.15.192:3300/0 conn(0x7fdf4415e500 msgr2=0x7fdf4406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:29:55.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.831+0000 7fdf4259c700 1 --2- 172.21.15.189:0/955721767 >> v2:172.21.15.192:3300/0 conn(0x7fdf4415e500 0x7fdf4406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:55.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.831+0000 7fdf4259c700 1 -- 172.21.15.189:0/955721767 >> v1:172.21.15.189:6789/0 conn(0x7fdf4406a730 legacy=0x7fdf4415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:55.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.831+0000 7fdf4259c700 1 -- 172.21.15.189:0/955721767 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf440b3060 con 0x7fdf4415d7b0 2024-04-06T12:29:55.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.831+0000 7fdf4159a700 1 -- 172.21.15.189:0/955721767 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691696013 0 0) 0x7fdf440b3990 con 0x7fdf4406a730 2024-04-06T12:29:55.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.832+0000 7fdf4259c700 1 --2- 172.21.15.189:0/955721767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf4415d7b0 0x7fdf440b3e20 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fdf440673e0 tx=0x7fdf34009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a2ce9672c878d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:29:55.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.832+0000 7fdf4159a700 1 -- 172.21.15.189:0/955721767 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34011830 con 0x7fdf4415d7b0 2024-04-06T12:29:55.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.832+0000 7fdf4159a700 1 -- 172.21.15.189:0/955721767 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf34011990 con 0x7fdf4415d7b0 2024-04-06T12:29:55.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.832+0000 7fdf4159a700 1 -- 172.21.15.189:0/955721767 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34011d20 con 0x7fdf4415d7b0 2024-04-06T12:29:55.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.832+0000 7fdf48cba700 1 -- 172.21.15.189:0/955721767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf4415d7b0 msgr2=0x7fdf440b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:55.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.832+0000 7fdf48cba700 1 --2- 172.21.15.189:0/955721767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf4415d7b0 0x7fdf440b3e20 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fdf440673e0 tx=0x7fdf34009b20 comp rx=0 tx=0).stop 2024-04-06T12:29:55.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.832+0000 7fdf48cba700 1 -- 172.21.15.189:0/955721767 shutdown_connections 2024-04-06T12:29:55.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.832+0000 7fdf48cba700 1 --2- 172.21.15.189:0/955721767 >> v2:172.21.15.192:3300/0 conn(0x7fdf4415e500 0x7fdf4406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:55.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.832+0000 7fdf48cba700 1 --2- 172.21.15.189:0/955721767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf4415d7b0 0x7fdf440b3e20 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:55.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.832+0000 7fdf48cba700 1 -- 172.21.15.189:0/955721767 >> 172.21.15.189:0/955721767 conn(0x7fdf440d00b0 msgr2=0x7fdf440d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:29:55.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.832+0000 7fdf48cba700 1 -- 172.21.15.189:0/955721767 shutdown_connections 2024-04-06T12:29:55.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.832+0000 7fdf48cba700 1 -- 172.21.15.189:0/955721767 wait complete. 2024-04-06T12:29:55.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.834+0000 7fdf48cba700 1 Processor -- start 2024-04-06T12:29:55.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.834+0000 7fdf48cba700 1 -- start start 2024-04-06T12:29:55.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.834+0000 7fdf48cba700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf4406a730 0x7fdf4415adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:55.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.834+0000 7fdf48cba700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf4415d7b0 0x7fdf440ad090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:55.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.834+0000 7fdf48cba700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf4415e500 0x7fdf440cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:55.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.834+0000 7fdf48cba700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf441f2a30 con 0x7fdf4406a730 2024-04-06T12:29:55.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.834+0000 7fdf48cba700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf441f64b0 con 0x7fdf4415e500 2024-04-06T12:29:55.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.834+0000 7fdf48cba700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf441f6790 con 0x7fdf4415d7b0 2024-04-06T12:29:55.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.835+0000 7fdf4259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf4406a730 0x7fdf4415adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:55.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.835+0000 7fdf4259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf4406a730 0x7fdf4415adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41848/0 (socket says 172.21.15.189:41848) 2024-04-06T12:29:55.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.835+0000 7fdf4259c700 1 -- 172.21.15.189:0/2718498209 learned_addr learned my addr 172.21.15.189:0/2718498209 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:29:55.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.835+0000 7fdf42d9d700 1 --2- 172.21.15.189:0/2718498209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf4415e500 0x7fdf440cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:55.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.835+0000 7fdf41d9b700 1 --2- 172.21.15.189:0/2718498209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf4415d7b0 0x7fdf440ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:55.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.835+0000 7fdf4259c700 1 -- 172.21.15.189:0/2718498209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf4415d7b0 msgr2=0x7fdf440ad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:55.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.835+0000 7fdf4259c700 1 --2- 172.21.15.189:0/2718498209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf4415d7b0 0x7fdf440ad090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:55.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.835+0000 7fdf4259c700 1 -- 172.21.15.189:0/2718498209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf4415e500 msgr2=0x7fdf440cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:55.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.835+0000 7fdf4259c700 1 --2- 172.21.15.189:0/2718498209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf4415e500 0x7fdf440cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:55.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.835+0000 7fdf4259c700 1 -- 172.21.15.189:0/2718498209 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf340094a0 con 0x7fdf4406a730 2024-04-06T12:29:55.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.835+0000 7fdf42d9d700 1 --2- 172.21.15.189:0/2718498209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf4415e500 0x7fdf440cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:29:55.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.835+0000 7fdf4259c700 1 --2- 172.21.15.189:0/2718498209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf4406a730 0x7fdf4415adb0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fdf34000c00 tx=0x7fdf34004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:29:55.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.835+0000 7fdf32ffd700 1 -- 172.21.15.189:0/2718498209 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf340036a0 con 0x7fdf4406a730 2024-04-06T12:29:55.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.835+0000 7fdf32ffd700 1 -- 172.21.15.189:0/2718498209 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3401f070 con 0x7fdf4406a730 2024-04-06T12:29:55.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.835+0000 7fdf32ffd700 1 -- 172.21.15.189:0/2718498209 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34003a60 con 0x7fdf4406a730 2024-04-06T12:29:55.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.835+0000 7fdf48cba700 1 -- 172.21.15.189:0/2718498209 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf441f6a70 con 0x7fdf4406a730 2024-04-06T12:29:55.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.835+0000 7fdf48cba700 1 -- 172.21.15.189:0/2718498209 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf4415b2f0 con 0x7fdf4406a730 2024-04-06T12:29:55.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.836+0000 7fdf48cba700 1 -- 172.21.15.189:0/2718498209 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf441f6f90 con 0x7fdf4406a730 2024-04-06T12:29:55.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.837+0000 7fdf32ffd700 1 -- 172.21.15.189:0/2718498209 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 58) v1 ==== 89378+0+0 (secure 0 0 0) 0x7fdf34003bc0 con 0x7fdf4406a730 2024-04-06T12:29:55.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.837+0000 7fdf32ffd700 1 --2- 172.21.15.189:0/2718498209 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdf2406af20 0x7fdf2406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:55.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.837+0000 7fdf32ffd700 1 -- 172.21.15.189:0/2718498209 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fdf34021070 con 0x7fdf4406a730 2024-04-06T12:29:55.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.837+0000 7fdf41d9b700 1 --2- 172.21.15.189:0/2718498209 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdf2406af20 0x7fdf2406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:55.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.838+0000 7fdf41d9b700 1 --2- 172.21.15.189:0/2718498209 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdf2406af20 0x7fdf2406d3e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdf440add70 tx=0x7fdf38007be0 comp rx=0 tx=0).ready entity=mgr.7195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:29:55.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:55.842+0000 7fdf32ffd700 1 -- 172.21.15.189:0/2718498209 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdf34056000 con 0x7fdf4406a730 2024-04-06T12:29:56.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.061+0000 7fdf48cba700 1 -- 172.21.15.189:0/2718498209 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdf4415eb40 con 0x7fdf2406af20 2024-04-06T12:29:56.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.070+0000 7fdf32ffd700 1 -- 172.21.15.189:0/2718498209 <== mgr.7195 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+303 (secure 0 0 0) 0x7fdf38005aa0 con 0x7fdf2406af20 2024-04-06T12:29:56.068 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:29:56.069 INFO:teuthology.orchestra.run.smithi189.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:28:55", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2024-04-06T12:29:56.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.073+0000 7fdf48cba700 1 -- 172.21.15.189:0/2718498209 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdf2406af20 msgr2=0x7fdf2406d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:56.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.073+0000 7fdf48cba700 1 --2- 172.21.15.189:0/2718498209 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdf2406af20 0x7fdf2406d3e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdf440add70 tx=0x7fdf38007be0 comp rx=0 tx=0).stop 2024-04-06T12:29:56.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.073+0000 7fdf48cba700 1 -- 172.21.15.189:0/2718498209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf4406a730 msgr2=0x7fdf4415adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:56.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.073+0000 7fdf48cba700 1 --2- 172.21.15.189:0/2718498209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf4406a730 0x7fdf4415adb0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fdf34000c00 tx=0x7fdf34004750 comp rx=0 tx=0).stop 2024-04-06T12:29:56.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.073+0000 7fdf48cba700 1 -- 172.21.15.189:0/2718498209 shutdown_connections 2024-04-06T12:29:56.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.073+0000 7fdf48cba700 1 --2- 172.21.15.189:0/2718498209 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdf2406af20 0x7fdf2406d3e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:56.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.073+0000 7fdf48cba700 1 --2- 172.21.15.189:0/2718498209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf4415e500 0x7fdf440cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:56.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.073+0000 7fdf48cba700 1 --2- 172.21.15.189:0/2718498209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf4406a730 0x7fdf4415adb0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:56.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.073+0000 7fdf48cba700 1 --2- 172.21.15.189:0/2718498209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf4415d7b0 0x7fdf440ad090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:56.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.073+0000 7fdf48cba700 1 -- 172.21.15.189:0/2718498209 >> 172.21.15.189:0/2718498209 conn(0x7fdf440d00b0 msgr2=0x7fdf4406eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:29:56.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.073+0000 7fdf48cba700 1 -- 172.21.15.189:0/2718498209 shutdown_connections 2024-04-06T12:29:56.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.073+0000 7fdf48cba700 1 -- 172.21.15.189:0/2718498209 wait complete. 2024-04-06T12:29:56.083 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:28:55", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2024-04-06T12:29:56.083 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /bad-path 2024-04-06T12:29:56.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.208+0000 7fc367cf2700 1 Processor -- start 2024-04-06T12:29:56.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.208+0000 7fc367cf2700 1 -- start start 2024-04-06T12:29:56.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.209+0000 7fc367cf2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc36015d7b0 0x7fc3600b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:56.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.209+0000 7fc367cf2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc36006a730 0x7fc36006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:56.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.209+0000 7fc367cf2700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3600b3990 con 0x7fc36015ddf0 2024-04-06T12:29:56.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.209+0000 7fc367cf2700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc36006b0f0 con 0x7fc36006a730 2024-04-06T12:29:56.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.209+0000 7fc367cf2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3600b3060 con 0x7fc36015d7b0 2024-04-06T12:29:56.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.209+0000 7fc36528d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc36006a730 0x7fc36006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:56.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.209+0000 7fc36528d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc36006a730 0x7fc36006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41864/0 (socket says 172.21.15.189:41864) 2024-04-06T12:29:56.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.209+0000 7fc36528d700 1 -- 172.21.15.189:0/1733859203 learned_addr learned my addr 172.21.15.189:0/1733859203 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:29:56.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.209+0000 7fc365a8e700 1 --2- 172.21.15.189:0/1733859203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc36015d7b0 0x7fc3600b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:56.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.209+0000 7fc36528d700 1 -- 172.21.15.189:0/1733859203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc36015d7b0 msgr2=0x7fc3600b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:29:56.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.209+0000 7fc36528d700 1 --2- 172.21.15.189:0/1733859203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc36015d7b0 0x7fc3600b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:56.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.209+0000 7fc36528d700 1 -- 172.21.15.189:0/1733859203 >> v1:172.21.15.192:6789/0 conn(0x7fc36015ddf0 legacy=0x7fc36015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:56.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.209+0000 7fc36528d700 1 -- 172.21.15.189:0/1733859203 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3600b3340 con 0x7fc36006a730 2024-04-06T12:29:56.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.209+0000 7fc36528d700 1 --2- 172.21.15.189:0/1733859203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc36006a730 0x7fc36006abb0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fc3500098f0 tx=0x7fc350009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30270f51e8221865 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:29:56.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.210+0000 7fc364a8c700 1 -- 172.21.15.189:0/1733859203 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35000b8a0 con 0x7fc36006a730 2024-04-06T12:29:56.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.210+0000 7fc364a8c700 1 -- 172.21.15.189:0/1733859203 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35000ba00 con 0x7fc36006a730 2024-04-06T12:29:56.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.210+0000 7fc364a8c700 1 -- 172.21.15.189:0/1733859203 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35000bd90 con 0x7fc36006a730 2024-04-06T12:29:56.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.210+0000 7fc367cf2700 1 -- 172.21.15.189:0/1733859203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc36006a730 msgr2=0x7fc36006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:56.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.210+0000 7fc367cf2700 1 --2- 172.21.15.189:0/1733859203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc36006a730 0x7fc36006abb0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fc3500098f0 tx=0x7fc350009c00 comp rx=0 tx=0).stop 2024-04-06T12:29:56.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.210+0000 7fc367cf2700 1 -- 172.21.15.189:0/1733859203 shutdown_connections 2024-04-06T12:29:56.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.210+0000 7fc367cf2700 1 --2- 172.21.15.189:0/1733859203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc36015d7b0 0x7fc3600b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:56.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.210+0000 7fc367cf2700 1 --2- 172.21.15.189:0/1733859203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc36006a730 0x7fc36006abb0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:56.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.210+0000 7fc367cf2700 1 -- 172.21.15.189:0/1733859203 >> 172.21.15.189:0/1733859203 conn(0x7fc3600d00b0 msgr2=0x7fc3600d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:29:56.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.210+0000 7fc367cf2700 1 -- 172.21.15.189:0/1733859203 shutdown_connections 2024-04-06T12:29:56.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.210+0000 7fc367cf2700 1 -- 172.21.15.189:0/1733859203 wait complete. 2024-04-06T12:29:56.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.212+0000 7fc367cf2700 1 Processor -- start 2024-04-06T12:29:56.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.212+0000 7fc367cf2700 1 -- start start 2024-04-06T12:29:56.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.213+0000 7fc367cf2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc36006a730 0x7fc3600ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:56.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.213+0000 7fc367cf2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc36015d7b0 0x7fc3600cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:56.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.213+0000 7fc367cf2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc36015ddf0 0x7fc3601f2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:56.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.213+0000 7fc367cf2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3601f3d40 con 0x7fc36015d7b0 2024-04-06T12:29:56.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.213+0000 7fc367cf2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3601f2f80 con 0x7fc36015ddf0 2024-04-06T12:29:56.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.213+0000 7fc367cf2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3601f3260 con 0x7fc36006a730 2024-04-06T12:29:56.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.213+0000 7fc36528d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc36015d7b0 0x7fc3600cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:56.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.213+0000 7fc36528d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc36015d7b0 0x7fc3600cd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41866/0 (socket says 172.21.15.189:41866) 2024-04-06T12:29:56.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.213+0000 7fc36528d700 1 -- 172.21.15.189:0/4178169777 learned_addr learned my addr 172.21.15.189:0/4178169777 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:29:56.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.213+0000 7fc365a8e700 1 --2- 172.21.15.189:0/4178169777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc36006a730 0x7fc3600ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:56.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.213+0000 7fc36628f700 1 --2- 172.21.15.189:0/4178169777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc36015ddf0 0x7fc3601f2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:56.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.213+0000 7fc36528d700 1 -- 172.21.15.189:0/4178169777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc36006a730 msgr2=0x7fc3600ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:56.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.213+0000 7fc36528d700 1 --2- 172.21.15.189:0/4178169777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc36006a730 0x7fc3600ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:56.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.213+0000 7fc36528d700 1 -- 172.21.15.189:0/4178169777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc36015ddf0 msgr2=0x7fc3601f2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:56.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.213+0000 7fc36528d700 1 --2- 172.21.15.189:0/4178169777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc36015ddf0 0x7fc3601f2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:56.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.213+0000 7fc36528d700 1 -- 172.21.15.189:0/4178169777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc350009550 con 0x7fc36015d7b0 2024-04-06T12:29:56.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.213+0000 7fc36528d700 1 --2- 172.21.15.189:0/4178169777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc36015d7b0 0x7fc3600cd1c0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fc350000c00 tx=0x7fc350004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:29:56.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.213+0000 7fc3567fc700 1 -- 172.21.15.189:0/4178169777 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc350004380 con 0x7fc36015d7b0 2024-04-06T12:29:56.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.214+0000 7fc3567fc700 1 -- 172.21.15.189:0/4178169777 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35001f070 con 0x7fc36015d7b0 2024-04-06T12:29:56.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.214+0000 7fc367cf2700 1 -- 172.21.15.189:0/4178169777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3601f3540 con 0x7fc36015d7b0 2024-04-06T12:29:56.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.214+0000 7fc3567fc700 1 -- 172.21.15.189:0/4178169777 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3500036a0 con 0x7fc36015d7b0 2024-04-06T12:29:56.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.214+0000 7fc367cf2700 1 -- 172.21.15.189:0/4178169777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3600cdbf0 con 0x7fc36015d7b0 2024-04-06T12:29:56.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.215+0000 7fc367cf2700 1 -- 172.21.15.189:0/4178169777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3601f3a60 con 0x7fc36015d7b0 2024-04-06T12:29:56.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.215+0000 7fc3567fc700 1 -- 172.21.15.189:0/4178169777 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 58) v1 ==== 89378+0+0 (secure 0 0 0) 0x7fc350021030 con 0x7fc36015d7b0 2024-04-06T12:29:56.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.215+0000 7fc3567fc700 1 --2- 172.21.15.189:0/4178169777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc34406af20 0x7fc34406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:56.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.215+0000 7fc3567fc700 1 -- 172.21.15.189:0/4178169777 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fc35008af50 con 0x7fc36015d7b0 2024-04-06T12:29:56.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.215+0000 7fc365a8e700 1 --2- 172.21.15.189:0/4178169777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc34406af20 0x7fc34406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:56.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.216+0000 7fc365a8e700 1 --2- 172.21.15.189:0/4178169777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc34406af20 0x7fc34406d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0099d0 tx=0x7fc35c008040 comp rx=0 tx=0).ready entity=mgr.7195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:29:56.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.220+0000 7fc3567fc700 1 -- 172.21.15.189:0/4178169777 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc3601f3a60 con 0x7fc36015d7b0 2024-04-06T12:29:56.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.439+0000 7fc367cf2700 1 -- 172.21.15.189:0/4178169777 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/bad-path", "target": ["mon-mgr", ""]}) v1 -- 0x7fc36015eb40 con 0x7fc34406af20 2024-04-06T12:29:56.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.506+0000 7fc3567fc700 1 -- 172.21.15.189:0/4178169777 <== mgr.7195 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+36 (secure 0 0 0) 0x7fc35c009e40 con 0x7fc34406af20 2024-04-06T12:29:56.504 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule removed for path /bad-path 2024-04-06T12:29:56.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.508+0000 7fc367cf2700 1 -- 172.21.15.189:0/4178169777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc34406af20 msgr2=0x7fc34406d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:56.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.508+0000 7fc367cf2700 1 --2- 172.21.15.189:0/4178169777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc34406af20 0x7fc34406d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0099d0 tx=0x7fc35c008040 comp rx=0 tx=0).stop 2024-04-06T12:29:56.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.508+0000 7fc367cf2700 1 -- 172.21.15.189:0/4178169777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc36015d7b0 msgr2=0x7fc3600cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:56.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.508+0000 7fc367cf2700 1 --2- 172.21.15.189:0/4178169777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc36015d7b0 0x7fc3600cd1c0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fc350000c00 tx=0x7fc350004750 comp rx=0 tx=0).stop 2024-04-06T12:29:56.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.508+0000 7fc367cf2700 1 -- 172.21.15.189:0/4178169777 shutdown_connections 2024-04-06T12:29:56.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.508+0000 7fc367cf2700 1 --2- 172.21.15.189:0/4178169777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc34406af20 0x7fc34406d3e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:56.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.508+0000 7fc367cf2700 1 --2- 172.21.15.189:0/4178169777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc36015ddf0 0x7fc3601f2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:56.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.508+0000 7fc367cf2700 1 --2- 172.21.15.189:0/4178169777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc36006a730 0x7fc3600ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:56.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.508+0000 7fc367cf2700 1 --2- 172.21.15.189:0/4178169777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc36015d7b0 0x7fc3600cd1c0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:56.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.508+0000 7fc367cf2700 1 -- 172.21.15.189:0/4178169777 >> 172.21.15.189:0/4178169777 conn(0x7fc3600d00b0 msgr2=0x7fc36006eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:29:56.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.508+0000 7fc367cf2700 1 -- 172.21.15.189:0/4178169777 shutdown_connections 2024-04-06T12:29:56.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.508+0000 7fc367cf2700 1 -- 172.21.15.189:0/4178169777 wait complete. 2024-04-06T12:29:56.518 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2024-04-06T12:29:56.518 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-06T12:29:56.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.644+0000 7ff300023700 1 Processor -- start 2024-04-06T12:29:56.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.644+0000 7ff300023700 1 -- start start 2024-04-06T12:29:56.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.645+0000 7ff300023700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2f806ae40 0x7ff2f80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:56.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.645+0000 7ff300023700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f80b3990 con 0x7ff2f815d7b0 2024-04-06T12:29:56.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.645+0000 7ff300023700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f80b3060 con 0x7ff2f815ddf0 2024-04-06T12:29:56.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.645+0000 7ff300023700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f80b3340 con 0x7ff2f806ae40 2024-04-06T12:29:56.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.645+0000 7ff2fddbf700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff2f815d7b0 0x7ff2f806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47512/0 (socket says 172.21.15.189:47512) 2024-04-06T12:29:56.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.645+0000 7ff2fddbf700 1 -- 172.21.15.189:0/369536838 learned_addr learned my addr 172.21.15.189:0/369536838 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:29:56.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.645+0000 7ff2fd5be700 1 --2- 172.21.15.189:0/369536838 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2f806ae40 0x7ff2f80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:56.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.645+0000 7ff2fcdbd700 1 -- 172.21.15.189:0/369536838 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 905735115 0 0) 0x7ff2f80b3990 con 0x7ff2f815d7b0 2024-04-06T12:29:56.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.645+0000 7ff2fcdbd700 1 -- 172.21.15.189:0/369536838 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2e40035b0 con 0x7ff2f815d7b0 2024-04-06T12:29:56.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.645+0000 7ff2fcdbd700 1 -- 172.21.15.189:0/369536838 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4037491242 0 0) 0x7ff2e40035b0 con 0x7ff2f815d7b0 2024-04-06T12:29:56.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fcdbd700 1 -- 172.21.15.189:0/369536838 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2f806ae40 msgr2=0x7ff2f80b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:29:56.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fcdbd700 1 --2- 172.21.15.189:0/369536838 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2f806ae40 0x7ff2f80b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:56.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fcdbd700 1 -- 172.21.15.189:0/369536838 >> v1:172.21.15.192:6789/0 conn(0x7ff2f815ddf0 legacy=0x7ff2f815e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:56.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fcdbd700 1 -- 172.21.15.189:0/369536838 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f80b3620 con 0x7ff2f815d7b0 2024-04-06T12:29:56.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fcdbd700 1 -- 172.21.15.189:0/369536838 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7ff2e8001670 con 0x7ff2f815d7b0 2024-04-06T12:29:56.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fcdbd700 1 -- 172.21.15.189:0/369536838 >> v1:172.21.15.189:6789/0 conn(0x7ff2f815d7b0 legacy=0x7ff2f806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:56.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fcdbd700 1 --2- 172.21.15.189:0/369536838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2e4003a30 0x7ff2e4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:56.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fcdbd700 1 --2- 172.21.15.189:0/369536838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2e40043e0 0x7ff2e40048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:56.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fcdbd700 1 --2- 172.21.15.189:0/369536838 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2f806ae40 0x7ff2e40051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:56.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fcdbd700 1 -- 172.21.15.189:0/369536838 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f80b3340 con 0x7ff2e4003a30 2024-04-06T12:29:56.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fcdbd700 1 -- 172.21.15.189:0/369536838 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e400bac0 con 0x7ff2e40043e0 2024-04-06T12:29:56.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fcdbd700 1 -- 172.21.15.189:0/369536838 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e400bc80 con 0x7ff2f806ae40 2024-04-06T12:29:56.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fddbf700 1 --2- 172.21.15.189:0/369536838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2e4003a30 0x7ff2e4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:56.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fd5be700 1 --2- 172.21.15.189:0/369536838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2e40043e0 0x7ff2e40048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:56.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fe5c0700 1 --2- 172.21.15.189:0/369536838 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2f806ae40 0x7ff2e40051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:56.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fddbf700 1 -- 172.21.15.189:0/369536838 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2f806ae40 msgr2=0x7ff2e40051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:29:56.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fddbf700 1 --2- 172.21.15.189:0/369536838 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2f806ae40 0x7ff2e40051b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:56.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fddbf700 1 -- 172.21.15.189:0/369536838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2e40043e0 msgr2=0x7ff2e40048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:29:56.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fddbf700 1 --2- 172.21.15.189:0/369536838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2e40043e0 0x7ff2e40048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:56.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fddbf700 1 -- 172.21.15.189:0/369536838 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e40035b0 con 0x7ff2e4003a30 2024-04-06T12:29:56.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fddbf700 1 --2- 172.21.15.189:0/369536838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2e4003a30 0x7ff2e4003e10 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7ff2e800c2f0 tx=0x7ff2e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c8bd1118c7ba8e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:29:56.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fcdbd700 1 -- 172.21.15.189:0/369536838 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e800ce30 con 0x7ff2e4003a30 2024-04-06T12:29:56.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fcdbd700 1 -- 172.21.15.189:0/369536838 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e8005960 con 0x7ff2e4003a30 2024-04-06T12:29:56.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.646+0000 7ff2fcdbd700 1 -- 172.21.15.189:0/369536838 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8005cf0 con 0x7ff2e4003a30 2024-04-06T12:29:56.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.647+0000 7ff300023700 1 -- 172.21.15.189:0/369536838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2e4003a30 msgr2=0x7ff2e4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:56.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.647+0000 7ff300023700 1 --2- 172.21.15.189:0/369536838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2e4003a30 0x7ff2e4003e10 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7ff2e800c2f0 tx=0x7ff2e4005880 comp rx=0 tx=0).stop 2024-04-06T12:29:56.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.647+0000 7ff2fe5c0700 1 -- 172.21.15.189:0/369536838 reap_dead start 2024-04-06T12:29:56.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.647+0000 7ff300023700 1 -- 172.21.15.189:0/369536838 shutdown_connections 2024-04-06T12:29:56.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.647+0000 7ff300023700 1 -- 172.21.15.189:0/369536838 >> 172.21.15.189:0/369536838 conn(0x7ff2f80d00b0 msgr2=0x7ff2f80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:29:56.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.647+0000 7ff300023700 1 -- 172.21.15.189:0/369536838 shutdown_connections 2024-04-06T12:29:56.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.647+0000 7ff300023700 1 -- 172.21.15.189:0/369536838 wait complete. 2024-04-06T12:29:56.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.648+0000 7ff300023700 1 Processor -- start 2024-04-06T12:29:56.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.649+0000 7ff300023700 1 -- start start 2024-04-06T12:29:56.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.649+0000 7ff300023700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2e4003a30 0x7ff2f8153020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:56.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.649+0000 7ff300023700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2f815b450 0x7ff2f815b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:56.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.649+0000 7ff300023700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2f81f2e20 0x7ff2f81f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:56.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.649+0000 7ff300023700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f8153560 con 0x7ff2f81f2e20 2024-04-06T12:29:56.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.649+0000 7ff300023700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f815be10 con 0x7ff2f815b450 2024-04-06T12:29:56.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.649+0000 7ff300023700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f80ce0c0 con 0x7ff2e4003a30 2024-04-06T12:29:56.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.650+0000 7ff2fe5c0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2f81f2e20 0x7ff2f81f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:56.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.650+0000 7ff2fe5c0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2f81f2e20 0x7ff2f81f32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41880/0 (socket says 172.21.15.189:41880) 2024-04-06T12:29:56.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.650+0000 7ff2fe5c0700 1 -- 172.21.15.189:0/1539863061 learned_addr learned my addr 172.21.15.189:0/1539863061 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:29:56.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.650+0000 7ff2fddbf700 1 --2- 172.21.15.189:0/1539863061 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2e4003a30 0x7ff2f8153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:56.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.650+0000 7ff2fd5be700 1 --2- 172.21.15.189:0/1539863061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2f815b450 0x7ff2f815b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:56.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.650+0000 7ff2fe5c0700 1 -- 172.21.15.189:0/1539863061 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2e4003a30 msgr2=0x7ff2f8153020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:56.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.650+0000 7ff2fe5c0700 1 --2- 172.21.15.189:0/1539863061 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2e4003a30 0x7ff2f8153020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:56.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.650+0000 7ff2fe5c0700 1 -- 172.21.15.189:0/1539863061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2f815b450 msgr2=0x7ff2f815b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:56.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.650+0000 7ff2fe5c0700 1 --2- 172.21.15.189:0/1539863061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2f815b450 0x7ff2f815b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:56.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.650+0000 7ff2fe5c0700 1 -- 172.21.15.189:0/1539863061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f80ce3a0 con 0x7ff2f81f2e20 2024-04-06T12:29:56.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.650+0000 7ff2fe5c0700 1 --2- 172.21.15.189:0/1539863061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2f81f2e20 0x7ff2f81f32b0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7ff2f806d2c0 tx=0x7ff2f000d660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:29:56.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.650+0000 7ff2ee7fc700 1 -- 172.21.15.189:0/1539863061 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f000dbc0 con 0x7ff2f81f2e20 2024-04-06T12:29:56.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.650+0000 7ff2ee7fc700 1 -- 172.21.15.189:0/1539863061 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f000dd20 con 0x7ff2f81f2e20 2024-04-06T12:29:56.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.650+0000 7ff300023700 1 -- 172.21.15.189:0/1539863061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f80ce680 con 0x7ff2f81f2e20 2024-04-06T12:29:56.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.650+0000 7ff2ee7fc700 1 -- 172.21.15.189:0/1539863061 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f0003c40 con 0x7ff2f81f2e20 2024-04-06T12:29:56.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.650+0000 7ff300023700 1 -- 172.21.15.189:0/1539863061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f80cec40 con 0x7ff2f81f2e20 2024-04-06T12:29:56.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.651+0000 7ff300023700 1 -- 172.21.15.189:0/1539863061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f8067bb0 con 0x7ff2f81f2e20 2024-04-06T12:29:56.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.652+0000 7ff2ee7fc700 1 -- 172.21.15.189:0/1539863061 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 58) v1 ==== 89378+0+0 (secure 0 0 0) 0x7ff2f00050d0 con 0x7ff2f81f2e20 2024-04-06T12:29:56.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.652+0000 7ff2ee7fc700 1 --2- 172.21.15.189:0/1539863061 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff2dc06af20 0x7ff2dc06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:56.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.652+0000 7ff2ee7fc700 1 -- 172.21.15.189:0/1539863061 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6489+0+0 (secure 0 0 0) 0x7ff2f0089e00 con 0x7ff2f81f2e20 2024-04-06T12:29:56.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.652+0000 7ff2fddbf700 1 --2- 172.21.15.189:0/1539863061 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff2dc06af20 0x7ff2dc06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:56.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.652+0000 7ff2fddbf700 1 --2- 172.21.15.189:0/1539863061 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff2dc06af20 0x7ff2dc06d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8001930 tx=0x7ff2e8007000 comp rx=0 tx=0).ready entity=mgr.7195 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:29:56.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.656+0000 7ff2ee7fc700 1 -- 172.21.15.189:0/1539863061 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff2f0054970 con 0x7ff2f81f2e20 2024-04-06T12:29:56.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:56.908+0000 7ff300023700 1 -- 172.21.15.189:0/1539863061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7ff2f8074990 con 0x7ff2f81f2e20 2024-04-06T12:29:57.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:57.680+0000 7ff2ee7fc700 1 -- 172.21.15.189:0/1539863061 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 59) v1 ==== 89361+0+0 (secure 0 0 0) 0x7ff2f008e070 con 0x7ff2f81f2e20 2024-04-06T12:29:57.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:57.682+0000 7ff2ee7fc700 1 -- 172.21.15.189:0/1539863061 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v59) v1 ==== 93+0+0 (secure 0 0 0) 0x7ff2f00875b0 con 0x7ff2f81f2e20 2024-04-06T12:29:57.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:57.684+0000 7ff300023700 1 -- 172.21.15.189:0/1539863061 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff2dc06af20 msgr2=0x7ff2dc06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:57.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:57.684+0000 7ff300023700 1 --2- 172.21.15.189:0/1539863061 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff2dc06af20 0x7ff2dc06d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8001930 tx=0x7ff2e8007000 comp rx=0 tx=0).stop 2024-04-06T12:29:57.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:57.684+0000 7ff300023700 1 -- 172.21.15.189:0/1539863061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2f81f2e20 msgr2=0x7ff2f81f32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:57.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:57.684+0000 7ff300023700 1 --2- 172.21.15.189:0/1539863061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2f81f2e20 0x7ff2f81f32b0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7ff2f806d2c0 tx=0x7ff2f000d660 comp rx=0 tx=0).stop 2024-04-06T12:29:57.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:57.684+0000 7ff300023700 1 -- 172.21.15.189:0/1539863061 shutdown_connections 2024-04-06T12:29:57.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:57.684+0000 7ff300023700 1 --2- 172.21.15.189:0/1539863061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2f81f2e20 0x7ff2f81f32b0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:57.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:57.684+0000 7ff300023700 1 --2- 172.21.15.189:0/1539863061 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2e4003a30 0x7ff2f8153020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:57.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:57.684+0000 7ff300023700 1 --2- 172.21.15.189:0/1539863061 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff2dc06af20 0x7ff2dc06d3e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:57.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:57.684+0000 7ff300023700 1 --2- 172.21.15.189:0/1539863061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2f815b450 0x7ff2f815b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:57.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:57.684+0000 7ff300023700 1 -- 172.21.15.189:0/1539863061 >> 172.21.15.189:0/1539863061 conn(0x7ff2f80d00b0 msgr2=0x7ff2f806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:29:57.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:57.684+0000 7ff300023700 1 -- 172.21.15.189:0/1539863061 shutdown_connections 2024-04-06T12:29:57.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:57.684+0000 7ff300023700 1 -- 172.21.15.189:0/1539863061 wait complete. 2024-04-06T12:29:57.696 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T12:29:57.696 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:29:57.746 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:29:57.795 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T12:29:57.795 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:29:57.795 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:29:57.813 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:29:57.813 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:29:57.870 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:29:58.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.038+0000 7f2f32e95700 1 Processor -- start 2024-04-06T12:29:58.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.038+0000 7f2f32e95700 1 -- start start 2024-04-06T12:29:58.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.039+0000 7f2f32e95700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2f2c0b4ed0 0x7f2f2c0b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:58.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.039+0000 7f2f32e95700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f2c0b5cc0 con 0x7f2f2c15d590 2024-04-06T12:29:58.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.039+0000 7f2f32e95700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f2c0d1500 con 0x7f2f2c0d2880 2024-04-06T12:29:58.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.039+0000 7f2f32e95700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f2c0b58c0 con 0x7f2f2c0b4ed0 2024-04-06T12:29:58.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.039+0000 7f2f30c31700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2f2c15d590 0x7f2f2c0b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47532/0 (socket says 172.21.15.189:47532) 2024-04-06T12:29:58.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.039+0000 7f2f30c31700 1 -- 172.21.15.189:0/3672732955 learned_addr learned my addr 172.21.15.189:0/3672732955 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:29:58.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.039+0000 7f2f2bfff700 1 --2- 172.21.15.189:0/3672732955 >> v2:172.21.15.192:3300/0 conn(0x7f2f2c0b4ed0 0x7f2f2c0b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:58.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.040+0000 7f2f2b7fe700 1 -- 172.21.15.189:0/3672732955 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 365273982 0 0) 0x7f2f2c0b5cc0 con 0x7f2f2c15d590 2024-04-06T12:29:58.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.040+0000 7f2f2b7fe700 1 -- 172.21.15.189:0/3672732955 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f1c0035b0 con 0x7f2f2c15d590 2024-04-06T12:29:58.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.040+0000 7f2f2bfff700 1 -- 172.21.15.189:0/3672732955 >> v1:172.21.15.192:6789/0 conn(0x7f2f2c0d2880 legacy=0x7f2f2c0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:58.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.040+0000 7f2f2bfff700 1 -- 172.21.15.189:0/3672732955 >> v1:172.21.15.189:6789/0 conn(0x7f2f2c15d590 legacy=0x7f2f2c0b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:58.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.040+0000 7f2f2bfff700 1 -- 172.21.15.189:0/3672732955 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2c0d17e0 con 0x7f2f2c0b4ed0 2024-04-06T12:29:58.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.040+0000 7f2f2bfff700 1 --2- 172.21.15.189:0/3672732955 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f2c0b4ed0 0x7f2f2c0b5380 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2f18009840 tx=0x7f2f18009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9cbc76a3bf6942e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:29:58.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.040+0000 7f2f2b7fe700 1 -- 172.21.15.189:0/3672732955 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f18011830 con 0x7f2f2c0b4ed0 2024-04-06T12:29:58.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.040+0000 7f2f2b7fe700 1 -- 172.21.15.189:0/3672732955 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f18011990 con 0x7f2f2c0b4ed0 2024-04-06T12:29:58.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.040+0000 7f2f2b7fe700 1 -- 172.21.15.189:0/3672732955 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f18011d20 con 0x7f2f2c0b4ed0 2024-04-06T12:29:58.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.041+0000 7f2f32e95700 1 -- 172.21.15.189:0/3672732955 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f2c0b4ed0 msgr2=0x7f2f2c0b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:58.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.041+0000 7f2f32e95700 1 --2- 172.21.15.189:0/3672732955 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f2c0b4ed0 0x7f2f2c0b5380 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2f18009840 tx=0x7f2f18009b50 comp rx=0 tx=0).stop 2024-04-06T12:29:58.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.041+0000 7f2f32e95700 1 -- 172.21.15.189:0/3672732955 shutdown_connections 2024-04-06T12:29:58.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.041+0000 7f2f32e95700 1 --2- 172.21.15.189:0/3672732955 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f2c0b4ed0 0x7f2f2c0b5380 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.041+0000 7f2f32e95700 1 -- 172.21.15.189:0/3672732955 >> 172.21.15.189:0/3672732955 conn(0x7f2f2c06bbc0 msgr2=0x7f2f2c06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:29:58.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.041+0000 7f2f32e95700 1 -- 172.21.15.189:0/3672732955 shutdown_connections 2024-04-06T12:29:58.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.041+0000 7f2f32e95700 1 -- 172.21.15.189:0/3672732955 wait complete. 2024-04-06T12:29:58.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.042+0000 7f2f32e95700 1 Processor -- start 2024-04-06T12:29:58.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.042+0000 7f2f32e95700 1 -- start start 2024-04-06T12:29:58.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.043+0000 7f2f32e95700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f2c0b4ed0 0x7f2f2c1f6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:58.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.043+0000 7f2f32e95700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f2c0d2880 0x7f2f2c1f25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:58.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.043+0000 7f2f32e95700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f2c15d590 0x7f2f2c15d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:58.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.043+0000 7f2f32e95700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2c1f2ae0 con 0x7f2f2c0b4ed0 2024-04-06T12:29:58.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.043+0000 7f2f32e95700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2c15deb0 con 0x7f2f2c15d590 2024-04-06T12:29:58.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.043+0000 7f2f32e95700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2c15e160 con 0x7f2f2c0d2880 2024-04-06T12:29:58.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.043+0000 7f2f30c31700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f2c0b4ed0 0x7f2f2c1f6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:58.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.043+0000 7f2f30c31700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f2c0b4ed0 0x7f2f2c1f6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41896/0 (socket says 172.21.15.189:41896) 2024-04-06T12:29:58.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.043+0000 7f2f30c31700 1 -- 172.21.15.189:0/1667886528 learned_addr learned my addr 172.21.15.189:0/1667886528 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:29:58.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.043+0000 7f2f31432700 1 --2- 172.21.15.189:0/1667886528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f2c15d590 0x7f2f2c15d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:58.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.043+0000 7f2f2bfff700 1 --2- 172.21.15.189:0/1667886528 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f2c0d2880 0x7f2f2c1f25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:58.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.043+0000 7f2f30c31700 1 -- 172.21.15.189:0/1667886528 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f2c0d2880 msgr2=0x7f2f2c1f25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:58.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.043+0000 7f2f30c31700 1 --2- 172.21.15.189:0/1667886528 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f2c0d2880 0x7f2f2c1f25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.043+0000 7f2f30c31700 1 -- 172.21.15.189:0/1667886528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f2c15d590 msgr2=0x7f2f2c15d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:58.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.043+0000 7f2f30c31700 1 --2- 172.21.15.189:0/1667886528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f2c15d590 0x7f2f2c15d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.043+0000 7f2f30c31700 1 -- 172.21.15.189:0/1667886528 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f180094a0 con 0x7f2f2c0b4ed0 2024-04-06T12:29:58.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.044+0000 7f2f30c31700 1 --2- 172.21.15.189:0/1667886528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f2c0b4ed0 0x7f2f2c1f6d90 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f2f20001eb0 tx=0x7f2f2000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:29:58.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.044+0000 7f2f297fa700 1 -- 172.21.15.189:0/1667886528 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f20005d70 con 0x7f2f2c0b4ed0 2024-04-06T12:29:58.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.044+0000 7f2f297fa700 1 -- 172.21.15.189:0/1667886528 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f200070d0 con 0x7f2f2c0b4ed0 2024-04-06T12:29:58.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.044+0000 7f2f297fa700 1 -- 172.21.15.189:0/1667886528 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f20007460 con 0x7f2f2c0b4ed0 2024-04-06T12:29:58.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.044+0000 7f2f32e95700 1 -- 172.21.15.189:0/1667886528 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f2c15e440 con 0x7f2f2c0b4ed0 2024-04-06T12:29:58.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.044+0000 7f2f32e95700 1 -- 172.21.15.189:0/1667886528 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f2c15ea00 con 0x7f2f2c0b4ed0 2024-04-06T12:29:58.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.045+0000 7f2f32e95700 1 -- 172.21.15.189:0/1667886528 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f2c074990 con 0x7f2f2c0b4ed0 2024-04-06T12:29:58.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.045+0000 7f2f297fa700 1 -- 172.21.15.189:0/1667886528 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 59) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f2f200075c0 con 0x7f2f2c0b4ed0 2024-04-06T12:29:58.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.046+0000 7f2f297fa700 1 --2- 172.21.15.189:0/1667886528 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2f1006aed0 0x7f2f1006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:58.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.046+0000 7f2f2bfff700 1 -- 172.21.15.189:0/1667886528 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2f1006aed0 msgr2=0x7f2f1006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:29:58.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.046+0000 7f2f297fa700 1 -- 172.21.15.189:0/1667886528 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f2f20088f00 con 0x7f2f2c0b4ed0 2024-04-06T12:29:58.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.046+0000 7f2f2bfff700 1 --2- 172.21.15.189:0/1667886528 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2f1006aed0 0x7f2f1006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:29:58.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.050+0000 7f2f297fa700 1 -- 172.21.15.189:0/1667886528 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2f200539d0 con 0x7f2f2c0b4ed0 2024-04-06T12:29:58.068 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:29:58.070+0000 7fdd05b24200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:29:58.080 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:29:58.082+0000 7f4a4ce9c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:29:58.146 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:29:58.148+0000 7fdd05b24200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:29:58.162 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:29:58.163+0000 7f4a4ce9c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:29:58.219 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:29:58.221+0000 7fdd05b24200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:29:58.237 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:29:58.239+0000 7f4a4ce9c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:29:58.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.247+0000 7f2f2bfff700 1 -- 172.21.15.189:0/1667886528 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2f1006aed0 msgr2=0x7f2f1006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:29:58.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.247+0000 7f2f2bfff700 1 --2- 172.21.15.189:0/1667886528 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2f1006aed0 0x7f2f1006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:29:58.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.255+0000 7f2f32e95700 1 -- 172.21.15.189:0/1667886528 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f2c0b9190 con 0x7f2f2c0b4ed0 2024-04-06T12:29:58.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.257+0000 7f2f297fa700 1 -- 172.21.15.189:0/1667886528 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+17000 (secure 0 0 0) 0x7f2f20017090 con 0x7f2f2c0b4ed0 2024-04-06T12:29:58.255 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:29:58.255 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":115,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:29:55.659401+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:28:38.415021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-06T12:28:39.629378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.8","mappings":[{"from":1,"to":7}]},{"pgid":"14.1b","mappings":[{"from":1,"to":2}]},{"pgid":"14.31","mappings":[{"from":1,"to":7}]},{"pgid":"14.36","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/1207971310":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/3711853723":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/1381299583":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/1611293259":"2024-04-07T12:28:54.176092+0000","172.21.15.189:6833/47128":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/1131169404":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/2194712360":"2024-04-07T12:28:54.176092+0000","172.21.15.189:6832/47128":"2024-04-07T12:28:54.176092+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:29:58.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.259+0000 7f2f32e95700 1 -- 172.21.15.189:0/1667886528 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2f1006aed0 msgr2=0x7f2f1006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:29:58.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.259+0000 7f2f32e95700 1 --2- 172.21.15.189:0/1667886528 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2f1006aed0 0x7f2f1006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.259+0000 7f2f32e95700 1 -- 172.21.15.189:0/1667886528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f2c0b4ed0 msgr2=0x7f2f2c1f6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:58.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.259+0000 7f2f32e95700 1 --2- 172.21.15.189:0/1667886528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f2c0b4ed0 0x7f2f2c1f6d90 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f2f20001eb0 tx=0x7f2f2000c7b0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.259+0000 7f2f32e95700 1 -- 172.21.15.189:0/1667886528 shutdown_connections 2024-04-06T12:29:58.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.259+0000 7f2f32e95700 1 --2- 172.21.15.189:0/1667886528 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2f1006aed0 0x7f2f1006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.259+0000 7f2f32e95700 1 --2- 172.21.15.189:0/1667886528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f2c15d590 0x7f2f2c15d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.259+0000 7f2f32e95700 1 --2- 172.21.15.189:0/1667886528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f2c0b4ed0 0x7f2f2c1f6d90 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.259+0000 7f2f32e95700 1 --2- 172.21.15.189:0/1667886528 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f2c0d2880 0x7f2f2c1f25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.259+0000 7f2f32e95700 1 -- 172.21.15.189:0/1667886528 >> 172.21.15.189:0/1667886528 conn(0x7f2f2c06bbc0 msgr2=0x7f2f2c0d3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:29:58.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.259+0000 7f2f32e95700 1 -- 172.21.15.189:0/1667886528 shutdown_connections 2024-04-06T12:29:58.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.259+0000 7f2f32e95700 1 -- 172.21.15.189:0/1667886528 wait complete. 2024-04-06T12:29:58.270 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:28:38.415021+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:28:39.629378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:29:58.270 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:29:58.286 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:29:58.288+0000 7fdd05b24200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:29:58.313 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:29:58.315+0000 7f4a4ce9c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:29:58.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.403+0000 7fd1daca3700 1 Processor -- start 2024-04-06T12:29:58.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.403+0000 7fd1daca3700 1 -- start start 2024-04-06T12:29:58.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.404+0000 7fd1daca3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd1d40d11b0 0x7fd1d40d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:58.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.404+0000 7fd1daca3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1d40cfa00 0x7fd1d40cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:58.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.404+0000 7fd1daca3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d40d1ad0 con 0x7fd1d40cb870 2024-04-06T12:29:58.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.404+0000 7fd1daca3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d40ccdc0 con 0x7fd1d40d11b0 2024-04-06T12:29:58.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.404+0000 7fd1daca3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d40d0b50 con 0x7fd1d40cfa00 2024-04-06T12:29:58.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.404+0000 7fd1d3fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd1d40cb870 0x7fd1d415c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47536/0 (socket says 172.21.15.189:47536) 2024-04-06T12:29:58.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.404+0000 7fd1d3fff700 1 -- 172.21.15.189:0/2739281021 learned_addr learned my addr 172.21.15.189:0/2739281021 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:29:58.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.404+0000 7fd1d9240700 1 --2- 172.21.15.189:0/2739281021 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1d40cfa00 0x7fd1d40cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:58.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.404+0000 7fd1d8a3f700 1 --2- 172.21.15.189:0/2739281021 >> v2:172.21.15.192:3300/0 conn(0x7fd1d40d11b0 0x7fd1d40d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:58.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.404+0000 7fd1d37fe700 1 -- 172.21.15.189:0/2739281021 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018708264 0 0) 0x7fd1d40d1ad0 con 0x7fd1d40cb870 2024-04-06T12:29:58.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.404+0000 7fd1d37fe700 1 -- 172.21.15.189:0/2739281021 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1c40035b0 con 0x7fd1d40cb870 2024-04-06T12:29:58.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.404+0000 7fd1d8a3f700 1 -- 172.21.15.189:0/2739281021 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1d40cfa00 msgr2=0x7fd1d40cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:29:58.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.404+0000 7fd1d8a3f700 1 --2- 172.21.15.189:0/2739281021 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1d40cfa00 0x7fd1d40cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.404+0000 7fd1d8a3f700 1 -- 172.21.15.189:0/2739281021 >> v1:172.21.15.189:6789/0 conn(0x7fd1d40cb870 legacy=0x7fd1d415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:58.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.404+0000 7fd1d8a3f700 1 -- 172.21.15.189:0/2739281021 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d40cbd80 con 0x7fd1d40d11b0 2024-04-06T12:29:58.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.404+0000 7fd1d9240700 1 --2- 172.21.15.189:0/2739281021 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1d40cfa00 0x7fd1d40cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:29:58.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.405+0000 7fd1d8a3f700 1 --2- 172.21.15.189:0/2739281021 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1d40d11b0 0x7fd1d40d1590 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8009840 tx=0x7fd1c8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d77826c4c4957174 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:29:58.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.405+0000 7fd1d37fe700 1 -- 172.21.15.189:0/2739281021 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8011830 con 0x7fd1d40d11b0 2024-04-06T12:29:58.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.405+0000 7fd1d37fe700 1 -- 172.21.15.189:0/2739281021 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c8011990 con 0x7fd1d40d11b0 2024-04-06T12:29:58.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.405+0000 7fd1d37fe700 1 -- 172.21.15.189:0/2739281021 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8011d20 con 0x7fd1d40d11b0 2024-04-06T12:29:58.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.406+0000 7fd1daca3700 1 -- 172.21.15.189:0/2739281021 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1d40d11b0 msgr2=0x7fd1d40d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:58.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.406+0000 7fd1daca3700 1 --2- 172.21.15.189:0/2739281021 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1d40d11b0 0x7fd1d40d1590 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8009840 tx=0x7fd1c8009b50 comp rx=0 tx=0).stop 2024-04-06T12:29:58.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.406+0000 7fd1daca3700 1 -- 172.21.15.189:0/2739281021 shutdown_connections 2024-04-06T12:29:58.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.406+0000 7fd1daca3700 1 --2- 172.21.15.189:0/2739281021 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1d40cfa00 0x7fd1d40cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.406+0000 7fd1daca3700 1 --2- 172.21.15.189:0/2739281021 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1d40d11b0 0x7fd1d40d1590 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.406+0000 7fd1daca3700 1 -- 172.21.15.189:0/2739281021 >> 172.21.15.189:0/2739281021 conn(0x7fd1d40b2db0 msgr2=0x7fd1d40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:29:58.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.406+0000 7fd1daca3700 1 -- 172.21.15.189:0/2739281021 shutdown_connections 2024-04-06T12:29:58.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.406+0000 7fd1daca3700 1 -- 172.21.15.189:0/2739281021 wait complete. 2024-04-06T12:29:58.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.408+0000 7fd1daca3700 1 Processor -- start 2024-04-06T12:29:58.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.408+0000 7fd1daca3700 1 -- start start 2024-04-06T12:29:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.409+0000 7fd1daca3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1d40cb870 0x7fd1d41e5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.409+0000 7fd1daca3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1d40cfa00 0x7fd1d4154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.409+0000 7fd1daca3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1d40d11b0 0x7fd1d415e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.409+0000 7fd1daca3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d41e58e0 con 0x7fd1d40cb870 2024-04-06T12:29:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.409+0000 7fd1daca3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d41e5bc0 con 0x7fd1d40d11b0 2024-04-06T12:29:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.409+0000 7fd1daca3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d41571e0 con 0x7fd1d40cfa00 2024-04-06T12:29:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.409+0000 7fd1d8a3f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1d40cb870 0x7fd1d41e5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.409+0000 7fd1d8a3f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1d40cb870 0x7fd1d41e5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41908/0 (socket says 172.21.15.189:41908) 2024-04-06T12:29:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.409+0000 7fd1d8a3f700 1 -- 172.21.15.189:0/2552560511 learned_addr learned my addr 172.21.15.189:0/2552560511 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:29:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.409+0000 7fd1d9240700 1 --2- 172.21.15.189:0/2552560511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1d40d11b0 0x7fd1d415e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.409+0000 7fd1d3fff700 1 --2- 172.21.15.189:0/2552560511 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1d40cfa00 0x7fd1d4154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.409+0000 7fd1d8a3f700 1 -- 172.21.15.189:0/2552560511 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1d40cfa00 msgr2=0x7fd1d4154860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.409+0000 7fd1d8a3f700 1 --2- 172.21.15.189:0/2552560511 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1d40cfa00 0x7fd1d4154860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.409+0000 7fd1d8a3f700 1 -- 172.21.15.189:0/2552560511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1d40d11b0 msgr2=0x7fd1d415e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:58.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.409+0000 7fd1d8a3f700 1 --2- 172.21.15.189:0/2552560511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1d40d11b0 0x7fd1d415e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.409+0000 7fd1d8a3f700 1 -- 172.21.15.189:0/2552560511 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1cc009a70 con 0x7fd1d40cb870 2024-04-06T12:29:58.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.410+0000 7fd1d8a3f700 1 --2- 172.21.15.189:0/2552560511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1d40cb870 0x7fd1d41e5300 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fd1c80090b0 tx=0x7fd1c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:29:58.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.410+0000 7fd1d17fa700 1 -- 172.21.15.189:0/2552560511 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8004550 con 0x7fd1d40cb870 2024-04-06T12:29:58.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.410+0000 7fd1d17fa700 1 -- 172.21.15.189:0/2552560511 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c801f070 con 0x7fd1d40cb870 2024-04-06T12:29:58.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.410+0000 7fd1daca3700 1 -- 172.21.15.189:0/2552560511 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c80094a0 con 0x7fd1d40cb870 2024-04-06T12:29:58.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.410+0000 7fd1d17fa700 1 -- 172.21.15.189:0/2552560511 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c80038e0 con 0x7fd1d40cb870 2024-04-06T12:29:58.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.410+0000 7fd1daca3700 1 -- 172.21.15.189:0/2552560511 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d415b0c0 con 0x7fd1d40cb870 2024-04-06T12:29:58.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.411+0000 7fd1daca3700 1 -- 172.21.15.189:0/2552560511 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d4074990 con 0x7fd1d40cb870 2024-04-06T12:29:58.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.411+0000 7fd1d17fa700 1 -- 172.21.15.189:0/2552560511 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 59) v1 ==== 89361+0+0 (secure 0 0 0) 0x7fd1c8003a40 con 0x7fd1d40cb870 2024-04-06T12:29:58.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.412+0000 7fd1d17fa700 1 --2- 172.21.15.189:0/2552560511 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1b806aed0 0x7fd1b806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:58.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.412+0000 7fd1d3fff700 1 -- 172.21.15.189:0/2552560511 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1b806aed0 msgr2=0x7fd1b806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:29:58.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.412+0000 7fd1d17fa700 1 -- 172.21.15.189:0/2552560511 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fd1c8021070 con 0x7fd1d40cb870 2024-04-06T12:29:58.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.412+0000 7fd1d3fff700 1 --2- 172.21.15.189:0/2552560511 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1b806aed0 0x7fd1b806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:29:58.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.416+0000 7fd1d17fa700 1 -- 172.21.15.189:0/2552560511 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd1c8055d90 con 0x7fd1d40cb870 2024-04-06T12:29:58.461 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:29:58.463+0000 7fdd05b24200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:29:58.499 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:29:58.501+0000 7f4a4ce9c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:29:58.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.613+0000 7fd1d3fff700 1 -- 172.21.15.189:0/2552560511 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1b806aed0 msgr2=0x7fd1b806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:29:58.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.613+0000 7fd1d3fff700 1 --2- 172.21.15.189:0/2552560511 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1b806aed0 0x7fd1b806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:29:58.615 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:29:58.618+0000 7fdd05b24200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:29:58.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.618+0000 7fd1daca3700 1 -- 172.21.15.189:0/2552560511 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd1d415fd50 con 0x7fd1d40cb870 2024-04-06T12:29:58.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.619+0000 7fd1d17fa700 1 -- 172.21.15.189:0/2552560511 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd1c80593b0 con 0x7fd1d40cb870 2024-04-06T12:29:58.617 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:29:58.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.621+0000 7fd1daca3700 1 -- 172.21.15.189:0/2552560511 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1b806aed0 msgr2=0x7fd1b806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:29:58.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.621+0000 7fd1daca3700 1 --2- 172.21.15.189:0/2552560511 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1b806aed0 0x7fd1b806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.621+0000 7fd1daca3700 1 -- 172.21.15.189:0/2552560511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1d40cb870 msgr2=0x7fd1d41e5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:58.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.621+0000 7fd1daca3700 1 --2- 172.21.15.189:0/2552560511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1d40cb870 0x7fd1d41e5300 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fd1c80090b0 tx=0x7fd1c8004750 comp rx=0 tx=0).stop 2024-04-06T12:29:58.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.621+0000 7fd1daca3700 1 -- 172.21.15.189:0/2552560511 shutdown_connections 2024-04-06T12:29:58.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.621+0000 7fd1daca3700 1 --2- 172.21.15.189:0/2552560511 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1b806aed0 0x7fd1b806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.621+0000 7fd1daca3700 1 --2- 172.21.15.189:0/2552560511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1d40d11b0 0x7fd1d415e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.621+0000 7fd1daca3700 1 --2- 172.21.15.189:0/2552560511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1d40cb870 0x7fd1d41e5300 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.621+0000 7fd1daca3700 1 --2- 172.21.15.189:0/2552560511 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1d40cfa00 0x7fd1d4154860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.621+0000 7fd1daca3700 1 -- 172.21.15.189:0/2552560511 >> 172.21.15.189:0/2552560511 conn(0x7fd1d40b2db0 msgr2=0x7fd1d40cd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:29:58.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.621+0000 7fd1daca3700 1 -- 172.21.15.189:0/2552560511 shutdown_connections 2024-04-06T12:29:58.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.621+0000 7fd1daca3700 1 -- 172.21.15.189:0/2552560511 wait complete. 2024-04-06T12:29:58.632 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:29:58.662 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:29:58.664+0000 7f4a4ce9c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:29:58.725 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:29:58.727+0000 7fdd05b24200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:29:58.781 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:29:58.783+0000 7f4a4ce9c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:29:58.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.795+0000 7f111dd72700 1 Processor -- start 2024-04-06T12:29:58.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.795+0000 7f111dd72700 1 -- start start 2024-04-06T12:29:58.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.796+0000 7f111dd72700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1118144f40 0x7f111806a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:58.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.796+0000 7f111dd72700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f111815fcd0 0x7f111806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:58.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.796+0000 7f111dd72700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f111815ba40 0x7f111815be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:58.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.796+0000 7f111dd72700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f111815ca20 con 0x7f111815fcd0 2024-04-06T12:29:58.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.796+0000 7f111dd72700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11180d0f40 con 0x7f1118144f40 2024-04-06T12:29:58.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.796+0000 7f111dd72700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f111815c740 con 0x7f111815ba40 2024-04-06T12:29:58.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.796+0000 7f1116ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f111815fcd0 0x7f111806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:58.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.796+0000 7f1116ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f111815fcd0 0x7f111806ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41922/0 (socket says 172.21.15.189:41922) 2024-04-06T12:29:58.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.796+0000 7f1116ffd700 1 -- 172.21.15.189:0/1554715899 learned_addr learned my addr 172.21.15.189:0/1554715899 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:29:58.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.796+0000 7f11177fe700 1 --2- 172.21.15.189:0/1554715899 >> v2:172.21.15.192:3300/0 conn(0x7f1118144f40 0x7f111806a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:58.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.796+0000 7f1117fff700 1 --2- 172.21.15.189:0/1554715899 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f111815ba40 0x7f111815be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:58.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.796+0000 7f1116ffd700 1 -- 172.21.15.189:0/1554715899 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f111815ba40 msgr2=0x7f111815be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:29:58.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.796+0000 7f1116ffd700 1 --2- 172.21.15.189:0/1554715899 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f111815ba40 0x7f111815be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.796+0000 7f1116ffd700 1 -- 172.21.15.189:0/1554715899 >> v2:172.21.15.192:3300/0 conn(0x7f1118144f40 msgr2=0x7f111806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:29:58.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.796+0000 7f1116ffd700 1 --2- 172.21.15.189:0/1554715899 >> v2:172.21.15.192:3300/0 conn(0x7f1118144f40 0x7f111806a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.796+0000 7f1116ffd700 1 -- 172.21.15.189:0/1554715899 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111815f1f0 con 0x7f111815fcd0 2024-04-06T12:29:58.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.797+0000 7f1116ffd700 1 --2- 172.21.15.189:0/1554715899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f111815fcd0 0x7f111806ac70 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f1118073470 tx=0x7f110c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=148a815fd03ddc79 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:29:58.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.797+0000 7f11167fc700 1 -- 172.21.15.189:0/1554715899 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c011830 con 0x7f111815fcd0 2024-04-06T12:29:58.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.797+0000 7f11167fc700 1 -- 172.21.15.189:0/1554715899 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110c011990 con 0x7f111815fcd0 2024-04-06T12:29:58.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.797+0000 7f11167fc700 1 -- 172.21.15.189:0/1554715899 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c011d20 con 0x7f111815fcd0 2024-04-06T12:29:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.797+0000 7f111dd72700 1 -- 172.21.15.189:0/1554715899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f111815fcd0 msgr2=0x7f111806ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.797+0000 7f111dd72700 1 --2- 172.21.15.189:0/1554715899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f111815fcd0 0x7f111806ac70 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f1118073470 tx=0x7f110c009bd0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.797+0000 7f111dd72700 1 -- 172.21.15.189:0/1554715899 shutdown_connections 2024-04-06T12:29:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.797+0000 7f111dd72700 1 --2- 172.21.15.189:0/1554715899 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f111815ba40 0x7f111815be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.797+0000 7f111dd72700 1 --2- 172.21.15.189:0/1554715899 >> v2:172.21.15.192:3300/0 conn(0x7f1118144f40 0x7f111806a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.797+0000 7f111dd72700 1 --2- 172.21.15.189:0/1554715899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f111815fcd0 0x7f111806ac70 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.797+0000 7f111dd72700 1 -- 172.21.15.189:0/1554715899 >> 172.21.15.189:0/1554715899 conn(0x7f11180d00b0 msgr2=0x7f11180d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:29:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.797+0000 7f111dd72700 1 -- 172.21.15.189:0/1554715899 shutdown_connections 2024-04-06T12:29:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.797+0000 7f111dd72700 1 -- 172.21.15.189:0/1554715899 wait complete. 2024-04-06T12:29:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.799+0000 7f111dd72700 1 Processor -- start 2024-04-06T12:29:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.799+0000 7f111dd72700 1 -- start start 2024-04-06T12:29:58.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.800+0000 7f111dd72700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1118144f40 0x7f111813f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:58.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.800+0000 7f111dd72700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f111815ba40 0x7f11180c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:58.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.800+0000 7f111dd72700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f111815fcd0 0x7f11180b4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:58.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.800+0000 7f111dd72700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11180b40b0 con 0x7f1118144f40 2024-04-06T12:29:58.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.800+0000 7f111dd72700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111813f5e0 con 0x7f111815ba40 2024-04-06T12:29:58.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.800+0000 7f111dd72700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11180c9250 con 0x7f111815fcd0 2024-04-06T12:29:58.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.800+0000 7f11177fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1118144f40 0x7f111813f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:58.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.800+0000 7f11177fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1118144f40 0x7f111813f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41932/0 (socket says 172.21.15.189:41932) 2024-04-06T12:29:58.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.800+0000 7f11177fe700 1 -- 172.21.15.189:0/589323413 learned_addr learned my addr 172.21.15.189:0/589323413 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:29:58.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.800+0000 7f1116ffd700 1 --2- 172.21.15.189:0/589323413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f111815ba40 0x7f11180c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:58.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.800+0000 7f1117fff700 1 --2- 172.21.15.189:0/589323413 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f111815fcd0 0x7f11180b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:58.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.801+0000 7f11177fe700 1 -- 172.21.15.189:0/589323413 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f111815fcd0 msgr2=0x7f11180b4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:58.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.801+0000 7f11177fe700 1 --2- 172.21.15.189:0/589323413 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f111815fcd0 0x7f11180b4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.801+0000 7f11177fe700 1 -- 172.21.15.189:0/589323413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f111815ba40 msgr2=0x7f11180c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:58.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.801+0000 7f11177fe700 1 --2- 172.21.15.189:0/589323413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f111815ba40 0x7f11180c8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:58.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.801+0000 7f11177fe700 1 -- 172.21.15.189:0/589323413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110c009550 con 0x7f1118144f40 2024-04-06T12:29:58.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.801+0000 7f1116ffd700 1 --2- 172.21.15.189:0/589323413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f111815ba40 0x7f11180c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:29:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.801+0000 7f11177fe700 1 --2- 172.21.15.189:0/589323413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1118144f40 0x7f111813f0a0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f110800eed0 tx=0x7f110800b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:29:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.801+0000 7f10fffff700 1 -- 172.21.15.189:0/589323413 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1108010040 con 0x7f1118144f40 2024-04-06T12:29:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.801+0000 7f10fffff700 1 -- 172.21.15.189:0/589323413 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110800bd00 con 0x7f1118144f40 2024-04-06T12:29:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.801+0000 7f111dd72700 1 -- 172.21.15.189:0/589323413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11180b3530 con 0x7f1118144f40 2024-04-06T12:29:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.801+0000 7f10fffff700 1 -- 172.21.15.189:0/589323413 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1108014600 con 0x7f1118144f40 2024-04-06T12:29:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.801+0000 7f111dd72700 1 -- 172.21.15.189:0/589323413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11180b3af0 con 0x7f1118144f40 2024-04-06T12:29:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.802+0000 7f111dd72700 1 -- 172.21.15.189:0/589323413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1118067bb0 con 0x7f1118144f40 2024-04-06T12:29:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.803+0000 7f10fffff700 1 -- 172.21.15.189:0/589323413 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 59) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f1108014760 con 0x7f1118144f40 2024-04-06T12:29:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.803+0000 7f10fffff700 1 --2- 172.21.15.189:0/589323413 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f110406aed0 0x7f110406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.803+0000 7f1116ffd700 1 -- 172.21.15.189:0/589323413 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f110406aed0 msgr2=0x7f110406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:29:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.803+0000 7f10fffff700 1 -- 172.21.15.189:0/589323413 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f1108089f90 con 0x7f1118144f40 2024-04-06T12:29:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.803+0000 7f1116ffd700 1 --2- 172.21.15.189:0/589323413 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f110406aed0 0x7f110406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:29:58.806 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:29:58.808+0000 7fdd05b24200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:29:58.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:58.808+0000 7f10fffff700 1 -- 172.21.15.189:0/589323413 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f11080bd650 con 0x7f1118144f40 2024-04-06T12:29:58.868 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:29:58.871+0000 7f4a4ce9c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:29:58.880 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:29:58.882+0000 7fdd05b24200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:29:58.946 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:29:58.948+0000 7f4a4ce9c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:29:59.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.004+0000 7f1116ffd700 1 -- 172.21.15.189:0/589323413 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f110406aed0 msgr2=0x7f110406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:29:59.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.004+0000 7f1116ffd700 1 --2- 172.21.15.189:0/589323413 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f110406aed0 0x7f110406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:29:59.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.013+0000 7f111dd72700 1 -- 172.21.15.189:0/589323413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f11180cc830 con 0x7f1118144f40 2024-04-06T12:29:59.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.013+0000 7f10fffff700 1 -- 172.21.15.189:0/589323413 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v115) v1 ==== 104+0+11 (secure 0 0 0) 0x7f11080549f0 con 0x7f1118144f40 2024-04-06T12:29:59.011 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:29:59.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.016+0000 7f111dd72700 1 -- 172.21.15.189:0/589323413 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f110406aed0 msgr2=0x7f110406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:29:59.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.016+0000 7f111dd72700 1 --2- 172.21.15.189:0/589323413 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f110406aed0 0x7f110406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.016+0000 7f111dd72700 1 -- 172.21.15.189:0/589323413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1118144f40 msgr2=0x7f111813f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:59.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.016+0000 7f111dd72700 1 --2- 172.21.15.189:0/589323413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1118144f40 0x7f111813f0a0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f110800eed0 tx=0x7f110800b390 comp rx=0 tx=0).stop 2024-04-06T12:29:59.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.016+0000 7f111dd72700 1 -- 172.21.15.189:0/589323413 shutdown_connections 2024-04-06T12:29:59.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.016+0000 7f111dd72700 1 --2- 172.21.15.189:0/589323413 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f111815fcd0 0x7f11180b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.016+0000 7f111dd72700 1 --2- 172.21.15.189:0/589323413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1118144f40 0x7f111813f0a0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.016+0000 7f111dd72700 1 --2- 172.21.15.189:0/589323413 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f110406aed0 0x7f110406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.016+0000 7f111dd72700 1 --2- 172.21.15.189:0/589323413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f111815ba40 0x7f11180c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.016+0000 7f111dd72700 1 -- 172.21.15.189:0/589323413 >> 172.21.15.189:0/589323413 conn(0x7f11180d00b0 msgr2=0x7f11181402d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:29:59.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.016+0000 7f111dd72700 1 -- 172.21.15.189:0/589323413 shutdown_connections 2024-04-06T12:29:59.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.016+0000 7f111dd72700 1 -- 172.21.15.189:0/589323413 wait complete. 2024-04-06T12:29:59.028 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:29:59.082 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:29:59.084+0000 7fdd05b24200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:29:59.159 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:29:59.162+0000 7fdd05b24200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:29:59.159 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:29:59.161+0000 7f4a4ce9c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:29:59.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.181+0000 7f77b73f0700 1 Processor -- start 2024-04-06T12:29:59.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.182+0000 7f77b73f0700 1 -- start start 2024-04-06T12:29:59.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.182+0000 7f77b73f0700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f77b006ae40 0x7f77b00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:59.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.182+0000 7f77b73f0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77b015ddf0 0x7f77b015e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:59.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.182+0000 7f77b73f0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b00b3990 con 0x7f77b015d7b0 2024-04-06T12:29:59.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.182+0000 7f77b73f0700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77b00b3060 con 0x7f77b006ae40 2024-04-06T12:29:59.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.182+0000 7f77b73f0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b00b3340 con 0x7f77b015ddf0 2024-04-06T12:29:59.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.183+0000 7f77b518c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f77b015d7b0 0x7f77b006a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47538/0 (socket says 172.21.15.189:47538) 2024-04-06T12:29:59.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.183+0000 7f77b518c700 1 -- 172.21.15.189:0/1459960197 learned_addr learned my addr 172.21.15.189:0/1459960197 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:29:59.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.183+0000 7f77b498b700 1 --2- 172.21.15.189:0/1459960197 >> v2:172.21.15.192:3300/0 conn(0x7f77b006ae40 0x7f77b00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:59.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.183+0000 7f77b598d700 1 --2- 172.21.15.189:0/1459960197 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77b015ddf0 0x7f77b015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:59.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.183+0000 7f77a7fff700 1 -- 172.21.15.189:0/1459960197 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975862337 0 0) 0x7f77b00b3990 con 0x7f77b015d7b0 2024-04-06T12:29:59.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.183+0000 7f77a7fff700 1 -- 172.21.15.189:0/1459960197 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f779c0035b0 con 0x7f77b015d7b0 2024-04-06T12:29:59.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.183+0000 7f77b498b700 1 -- 172.21.15.189:0/1459960197 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77b015ddf0 msgr2=0x7f77b015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:29:59.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.183+0000 7f77b498b700 1 --2- 172.21.15.189:0/1459960197 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77b015ddf0 0x7f77b015e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.183+0000 7f77b498b700 1 -- 172.21.15.189:0/1459960197 >> v1:172.21.15.189:6789/0 conn(0x7f77b015d7b0 legacy=0x7f77b006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:59.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.183+0000 7f77b498b700 1 -- 172.21.15.189:0/1459960197 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b00b3620 con 0x7f77b006ae40 2024-04-06T12:29:59.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.183+0000 7f77b498b700 1 --2- 172.21.15.189:0/1459960197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77b006ae40 0x7f77b00b3e20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f77a0009840 tx=0x7f77a0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24bab085264bbe6b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:29:59.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.183+0000 7f77a7fff700 1 -- 172.21.15.189:0/1459960197 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a0011830 con 0x7f77b006ae40 2024-04-06T12:29:59.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.183+0000 7f77a7fff700 1 -- 172.21.15.189:0/1459960197 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a0011990 con 0x7f77b006ae40 2024-04-06T12:29:59.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.183+0000 7f77a7fff700 1 -- 172.21.15.189:0/1459960197 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a0011d20 con 0x7f77b006ae40 2024-04-06T12:29:59.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.184+0000 7f77b73f0700 1 -- 172.21.15.189:0/1459960197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77b006ae40 msgr2=0x7f77b00b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:59.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.184+0000 7f77b73f0700 1 --2- 172.21.15.189:0/1459960197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77b006ae40 0x7f77b00b3e20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f77a0009840 tx=0x7f77a0009b50 comp rx=0 tx=0).stop 2024-04-06T12:29:59.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.184+0000 7f77b73f0700 1 -- 172.21.15.189:0/1459960197 shutdown_connections 2024-04-06T12:29:59.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.184+0000 7f77b73f0700 1 --2- 172.21.15.189:0/1459960197 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77b015ddf0 0x7f77b015e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.184+0000 7f77b73f0700 1 --2- 172.21.15.189:0/1459960197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77b006ae40 0x7f77b00b3e20 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.184+0000 7f77b73f0700 1 -- 172.21.15.189:0/1459960197 >> 172.21.15.189:0/1459960197 conn(0x7f77b00d00b0 msgr2=0x7f77b00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:29:59.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.184+0000 7f77b73f0700 1 -- 172.21.15.189:0/1459960197 shutdown_connections 2024-04-06T12:29:59.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.184+0000 7f77b73f0700 1 -- 172.21.15.189:0/1459960197 wait complete. 2024-04-06T12:29:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.185+0000 7f77b73f0700 1 Processor -- start 2024-04-06T12:29:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.185+0000 7f77b73f0700 1 -- start start 2024-04-06T12:29:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.186+0000 7f77b73f0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77b006ae40 0x7f77b00ac950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.186+0000 7f77b73f0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77b015d7b0 0x7f77b00ace90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.186+0000 7f77b73f0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77b015ddf0 0x7f77b014b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.186+0000 7f77b73f0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b0149a90 con 0x7f77b015d7b0 2024-04-06T12:29:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.186+0000 7f77b73f0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b014f780 con 0x7f77b015ddf0 2024-04-06T12:29:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.186+0000 7f77b73f0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b014bfd0 con 0x7f77b006ae40 2024-04-06T12:29:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.186+0000 7f77b498b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77b015d7b0 0x7f77b00ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.186+0000 7f77b498b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77b015d7b0 0x7f77b00ace90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41938/0 (socket says 172.21.15.189:41938) 2024-04-06T12:29:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.186+0000 7f77b518c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77b006ae40 0x7f77b00ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.186+0000 7f77b498b700 1 -- 172.21.15.189:0/1068729453 learned_addr learned my addr 172.21.15.189:0/1068729453 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:29:59.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.186+0000 7f77b598d700 1 --2- 172.21.15.189:0/1068729453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77b015ddf0 0x7f77b014b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.187+0000 7f77b498b700 1 -- 172.21.15.189:0/1068729453 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77b006ae40 msgr2=0x7f77b00ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.187+0000 7f77b498b700 1 --2- 172.21.15.189:0/1068729453 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77b006ae40 0x7f77b00ac950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.187+0000 7f77b498b700 1 -- 172.21.15.189:0/1068729453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77b015ddf0 msgr2=0x7f77b014b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.187+0000 7f77b498b700 1 --2- 172.21.15.189:0/1068729453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77b015ddf0 0x7f77b014b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.187+0000 7f77b498b700 1 -- 172.21.15.189:0/1068729453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a00094a0 con 0x7f77b015d7b0 2024-04-06T12:29:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.187+0000 7f77b498b700 1 --2- 172.21.15.189:0/1068729453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77b015d7b0 0x7f77b00ace90 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f77a0000c00 tx=0x7f77a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:29:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.187+0000 7f77a5ffb700 1 -- 172.21.15.189:0/1068729453 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a0004590 con 0x7f77b015d7b0 2024-04-06T12:29:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.187+0000 7f77a5ffb700 1 -- 172.21.15.189:0/1068729453 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a001f070 con 0x7f77b015d7b0 2024-04-06T12:29:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.187+0000 7f77a5ffb700 1 -- 172.21.15.189:0/1068729453 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a00038e0 con 0x7f77b015d7b0 2024-04-06T12:29:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.188+0000 7f77b73f0700 1 -- 172.21.15.189:0/1068729453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b014c2b0 con 0x7f77b015d7b0 2024-04-06T12:29:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.188+0000 7f77b73f0700 1 -- 172.21.15.189:0/1068729453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b01496b0 con 0x7f77b015d7b0 2024-04-06T12:29:59.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.189+0000 7f77b73f0700 1 -- 172.21.15.189:0/1068729453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b0067bb0 con 0x7f77b015d7b0 2024-04-06T12:29:59.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.189+0000 7f77a5ffb700 1 -- 172.21.15.189:0/1068729453 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 59) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f77a0003a40 con 0x7f77b015d7b0 2024-04-06T12:29:59.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.190+0000 7f77a5ffb700 1 --2- 172.21.15.189:0/1068729453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f779406aed0 0x7f779406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:59.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.190+0000 7f77b518c700 1 -- 172.21.15.189:0/1068729453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f779406aed0 msgr2=0x7f779406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:29:59.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.190+0000 7f77a5ffb700 1 -- 172.21.15.189:0/1068729453 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f77a0021070 con 0x7f77b015d7b0 2024-04-06T12:29:59.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.190+0000 7f77b518c700 1 --2- 172.21.15.189:0/1068729453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f779406aed0 0x7f779406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:29:59.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.194+0000 7f77a5ffb700 1 -- 172.21.15.189:0/1068729453 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f77a00be650 con 0x7f77b015d7b0 2024-04-06T12:29:59.239 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:29:59.241+0000 7f4a4ce9c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:29:59.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.398+0000 7f77b73f0700 1 -- 172.21.15.189:0/1068729453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f77b0074990 con 0x7f77b015d7b0 2024-04-06T12:29:59.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.398+0000 7f77a5ffb700 1 -- 172.21.15.189:0/1068729453 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v115) v1 ==== 100+0+11 (secure 0 0 0) 0x7f77a00150a0 con 0x7f77b015d7b0 2024-04-06T12:29:59.397 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:29:59.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.400+0000 7f77b73f0700 1 -- 172.21.15.189:0/1068729453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f779406aed0 msgr2=0x7f779406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-06T12:29:59.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.400+0000 7f77b73f0700 1 --2- 172.21.15.189:0/1068729453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f779406aed0 0x7f779406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.400+0000 7f77b73f0700 1 -- 172.21.15.189:0/1068729453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77b015d7b0 msgr2=0x7f77b00ace90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:59.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.400+0000 7f77b73f0700 1 --2- 172.21.15.189:0/1068729453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77b015d7b0 0x7f77b00ace90 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f77a0000c00 tx=0x7f77a0004750 comp rx=0 tx=0).stop 2024-04-06T12:29:59.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.401+0000 7f77b73f0700 1 -- 172.21.15.189:0/1068729453 shutdown_connections 2024-04-06T12:29:59.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.401+0000 7f77b73f0700 1 --2- 172.21.15.189:0/1068729453 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f779406aed0 0x7f779406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.401+0000 7f77b73f0700 1 --2- 172.21.15.189:0/1068729453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77b015ddf0 0x7f77b014b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.401+0000 7f77b73f0700 1 --2- 172.21.15.189:0/1068729453 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77b006ae40 0x7f77b00ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.401+0000 7f77b73f0700 1 --2- 172.21.15.189:0/1068729453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77b015d7b0 0x7f77b00ace90 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.401+0000 7f77b73f0700 1 -- 172.21.15.189:0/1068729453 >> 172.21.15.189:0/1068729453 conn(0x7f77b00d00b0 msgr2=0x7f77b015f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:29:59.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.401+0000 7f77b73f0700 1 -- 172.21.15.189:0/1068729453 shutdown_connections 2024-04-06T12:29:59.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.401+0000 7f77b73f0700 1 -- 172.21.15.189:0/1068729453 wait complete. 2024-04-06T12:29:59.412 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T12:29:59.478 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:29:59.481+0000 7fdd05b24200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:29:59.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.540+0000 7f911abd1700 1 Processor -- start 2024-04-06T12:29:59.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.540+0000 7f911abd1700 1 -- start start 2024-04-06T12:29:59.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.541+0000 7f911abd1700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f91140b39d0 0x7f91140af040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:59.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.541+0000 7f911abd1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f91140723f0 0x7f9114072880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:59.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.541+0000 7f911abd1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9114071950 con 0x7f911415d7b0 2024-04-06T12:29:59.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.541+0000 7f911abd1700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9114072dc0 con 0x7f91140723f0 2024-04-06T12:29:59.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.541+0000 7f911abd1700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91140b4170 con 0x7f91140b39d0 2024-04-06T12:29:59.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.541+0000 7f911916e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f91140723f0 0x7f9114072880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:59.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.541+0000 7f911896d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f911415d7b0 0x7f91140ac8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40272/0 (socket says 172.21.15.189:40272) 2024-04-06T12:29:59.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.541+0000 7f911896d700 1 -- 172.21.15.189:0/3779739394 learned_addr learned my addr 172.21.15.189:0/3779739394 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:29:59.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.541+0000 7f9113fff700 1 --2- 172.21.15.189:0/3779739394 >> v2:172.21.15.192:3300/0 conn(0x7f91140b39d0 0x7f91140af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:59.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.542+0000 7f911916e700 1 -- 172.21.15.189:0/3779739394 >> v2:172.21.15.192:3300/0 conn(0x7f91140b39d0 msgr2=0x7f91140af040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:29:59.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.542+0000 7f911916e700 1 --2- 172.21.15.189:0/3779739394 >> v2:172.21.15.192:3300/0 conn(0x7f91140b39d0 0x7f91140af040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.542+0000 7f911916e700 1 -- 172.21.15.189:0/3779739394 >> v1:172.21.15.189:6789/0 conn(0x7f911415d7b0 legacy=0x7f91140ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:59.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.542+0000 7f91137fe700 1 -- 172.21.15.189:0/3779739394 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327581851 0 0) 0x7f9114071950 con 0x7f911415d7b0 2024-04-06T12:29:59.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.542+0000 7f911916e700 1 -- 172.21.15.189:0/3779739394 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91140b3e90 con 0x7f91140723f0 2024-04-06T12:29:59.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.542+0000 7f911916e700 1 --2- 172.21.15.189:0/3779739394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91140723f0 0x7f9114072880 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f9114067b80 tx=0x7f910c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dc74422937901e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:29:59.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.542+0000 7f91137fe700 1 -- 172.21.15.189:0/3779739394 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910c00e070 con 0x7f91140723f0 2024-04-06T12:29:59.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.542+0000 7f91137fe700 1 -- 172.21.15.189:0/3779739394 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f910c00bb20 con 0x7f91140723f0 2024-04-06T12:29:59.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.542+0000 7f91137fe700 1 -- 172.21.15.189:0/3779739394 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910c012430 con 0x7f91140723f0 2024-04-06T12:29:59.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.543+0000 7f911abd1700 1 -- 172.21.15.189:0/3779739394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91140723f0 msgr2=0x7f9114072880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:59.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.543+0000 7f911abd1700 1 --2- 172.21.15.189:0/3779739394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91140723f0 0x7f9114072880 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f9114067b80 tx=0x7f910c00b330 comp rx=0 tx=0).stop 2024-04-06T12:29:59.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.543+0000 7f911abd1700 1 -- 172.21.15.189:0/3779739394 shutdown_connections 2024-04-06T12:29:59.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.543+0000 7f911abd1700 1 --2- 172.21.15.189:0/3779739394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91140723f0 0x7f9114072880 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.543+0000 7f911abd1700 1 --2- 172.21.15.189:0/3779739394 >> v2:172.21.15.192:3300/0 conn(0x7f91140b39d0 0x7f91140af040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.543+0000 7f911abd1700 1 -- 172.21.15.189:0/3779739394 >> 172.21.15.189:0/3779739394 conn(0x7f91140d00b0 msgr2=0x7f91140d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:29:59.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.543+0000 7f911abd1700 1 -- 172.21.15.189:0/3779739394 shutdown_connections 2024-04-06T12:29:59.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.543+0000 7f911abd1700 1 -- 172.21.15.189:0/3779739394 wait complete. 2024-04-06T12:29:59.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.544+0000 7f911abd1700 1 Processor -- start 2024-04-06T12:29:59.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.544+0000 7f911abd1700 1 -- start start 2024-04-06T12:29:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.545+0000 7f911abd1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91140723f0 0x7f9114146ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.545+0000 7f911abd1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91140b39d0 0x7f911415b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.545+0000 7f911abd1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9114154b00 0x7f911406eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.545+0000 7f911abd1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91141f5f40 con 0x7f9114154b00 2024-04-06T12:29:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.545+0000 7f911abd1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91141f6220 con 0x7f91140b39d0 2024-04-06T12:29:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.545+0000 7f911abd1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91141f6500 con 0x7f91140723f0 2024-04-06T12:29:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.545+0000 7f911916e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9114154b00 0x7f911406eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.545+0000 7f911916e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9114154b00 0x7f911406eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58028/0 (socket says 172.21.15.189:58028) 2024-04-06T12:29:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.545+0000 7f911916e700 1 -- 172.21.15.189:0/2036367842 learned_addr learned my addr 172.21.15.189:0/2036367842 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:29:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.545+0000 7f9113fff700 1 --2- 172.21.15.189:0/2036367842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91140b39d0 0x7f911415b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.545+0000 7f911896d700 1 --2- 172.21.15.189:0/2036367842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91140723f0 0x7f9114146ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:29:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.545+0000 7f911916e700 1 -- 172.21.15.189:0/2036367842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91140723f0 msgr2=0x7f9114146ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.545+0000 7f911916e700 1 --2- 172.21.15.189:0/2036367842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91140723f0 0x7f9114146ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.545+0000 7f911916e700 1 -- 172.21.15.189:0/2036367842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91140b39d0 msgr2=0x7f911415b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:59.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.545+0000 7f911916e700 1 --2- 172.21.15.189:0/2036367842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91140b39d0 0x7f911415b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.545+0000 7f911916e700 1 -- 172.21.15.189:0/2036367842 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910c009a70 con 0x7f9114154b00 2024-04-06T12:29:59.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.545+0000 7f911916e700 1 --2- 172.21.15.189:0/2036367842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9114154b00 0x7f911406eb50 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f911406c820 tx=0x7f910c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:29:59.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.546+0000 7f91117fa700 1 -- 172.21.15.189:0/2036367842 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910c012430 con 0x7f9114154b00 2024-04-06T12:29:59.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.546+0000 7f91117fa700 1 -- 172.21.15.189:0/2036367842 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f910c020070 con 0x7f9114154b00 2024-04-06T12:29:59.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.546+0000 7f911abd1700 1 -- 172.21.15.189:0/2036367842 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91141f67e0 con 0x7f9114154b00 2024-04-06T12:29:59.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.546+0000 7f91117fa700 1 -- 172.21.15.189:0/2036367842 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910c004280 con 0x7f9114154b00 2024-04-06T12:29:59.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.546+0000 7f911abd1700 1 -- 172.21.15.189:0/2036367842 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9114147010 con 0x7f9114154b00 2024-04-06T12:29:59.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.547+0000 7f911abd1700 1 -- 172.21.15.189:0/2036367842 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91141f6d00 con 0x7f9114154b00 2024-04-06T12:29:59.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.547+0000 7f91117fa700 1 -- 172.21.15.189:0/2036367842 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 59) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f910c0043e0 con 0x7f9114154b00 2024-04-06T12:29:59.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.547+0000 7f91117fa700 1 --2- 172.21.15.189:0/2036367842 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f90f806aed0 0x7f90f806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:29:59.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.547+0000 7f91117fa700 1 -- 172.21.15.189:0/2036367842 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f910c022070 con 0x7f9114154b00 2024-04-06T12:29:59.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.552+0000 7f91117fa700 1 -- 172.21.15.189:0/2036367842 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f910c056200 con 0x7f9114154b00 2024-04-06T12:29:59.577 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:29:59.579+0000 7f4a4ce9c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:29:59.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.773+0000 7f911abd1700 1 -- 172.21.15.189:0/2036367842 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9114067bb0 con 0x7f9114154b00 2024-04-06T12:29:59.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.773+0000 7f91117fa700 1 -- 172.21.15.189:0/2036367842 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v115) v1 ==== 80+0+468 (secure 0 0 0) 0x7f910c059820 con 0x7f9114154b00 2024-04-06T12:29:59.772 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1207971310 2024-04-07T12:28:54.176092+0000 2024-04-06T12:29:59.772 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/3711853723 2024-04-07T12:28:54.176092+0000 2024-04-06T12:29:59.772 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1381299583 2024-04-07T12:28:54.176092+0000 2024-04-06T12:29:59.772 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1611293259 2024-04-07T12:28:54.176092+0000 2024-04-06T12:29:59.772 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6833/47128 2024-04-07T12:28:54.176092+0000 2024-04-06T12:29:59.772 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1131169404 2024-04-07T12:28:54.176092+0000 2024-04-06T12:29:59.772 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2194712360 2024-04-07T12:28:54.176092+0000 2024-04-06T12:29:59.772 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6832/47128 2024-04-07T12:28:54.176092+0000 2024-04-06T12:29:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.776+0000 7f911abd1700 1 -- 172.21.15.189:0/2036367842 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f90f806aed0 msgr2=0x7f90f806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-06T12:29:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.776+0000 7f911abd1700 1 --2- 172.21.15.189:0/2036367842 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f90f806aed0 0x7f90f806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.776+0000 7f911abd1700 1 -- 172.21.15.189:0/2036367842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9114154b00 msgr2=0x7f911406eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:29:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.776+0000 7f911abd1700 1 --2- 172.21.15.189:0/2036367842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9114154b00 0x7f911406eb50 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f911406c820 tx=0x7f910c004ad0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.776+0000 7f911abd1700 1 -- 172.21.15.189:0/2036367842 shutdown_connections 2024-04-06T12:29:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.776+0000 7f911abd1700 1 --2- 172.21.15.189:0/2036367842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9114154b00 0x7f911406eb50 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.776+0000 7f911abd1700 1 --2- 172.21.15.189:0/2036367842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91140723f0 0x7f9114146ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.776+0000 7f911abd1700 1 --2- 172.21.15.189:0/2036367842 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f90f806aed0 0x7f90f806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.776+0000 7f911abd1700 1 --2- 172.21.15.189:0/2036367842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91140b39d0 0x7f911415b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:29:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.776+0000 7f911abd1700 1 -- 172.21.15.189:0/2036367842 >> 172.21.15.189:0/2036367842 conn(0x7f91140d00b0 msgr2=0x7f91140ad2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:29:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.776+0000 7f911abd1700 1 -- 172.21.15.189:0/2036367842 shutdown_connections 2024-04-06T12:29:59.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:29:59.776+0000 7f911abd1700 1 -- 172.21.15.189:0/2036367842 wait complete. 2024-04-06T12:29:59.776 INFO:teuthology.orchestra.run.smithi189.stderr:listed 8 entries 2024-04-06T12:29:59.787 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-06T12:29:59.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:29:59.787 DEBUG:teuthology.orchestra.run.smithi189:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-06T12:29:59.798 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:29:59.801+0000 7fdd05b24200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:29:59.828 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T12:29:59.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:29:59.828 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-06T12:29:59.885 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:29:59.885 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:29:59.902 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:29:59.902 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:29:59.902 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:29:59.912 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:29:59.914+0000 7f4a4ce9c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:29:59.929 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:29:59.929 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:29:59.929 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:29:59.959 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:29:59.959 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:29:59.960 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:29:59.963+0000 7fdd05b24200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:30:00.035 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:00.038+0000 7fdd05b24200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:30:00.100 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:00.102+0000 7f4a4ce9c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:30:00.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.115+0000 7fa60ed3c700 1 Processor -- start 2024-04-06T12:30:00.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.115+0000 7fa60ed3c700 1 -- start start 2024-04-06T12:30:00.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.115+0000 7fa60ed3c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa60815a550 0x7fa6080cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:00.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.115+0000 7fa60ed3c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa6080cf900 0x7fa6080b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:00.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.116+0000 7fa60ed3c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6080cdbe0 con 0x7fa6080b4ba0 2024-04-06T12:30:00.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.116+0000 7fa60ed3c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6080b5010 con 0x7fa6080cf900 2024-04-06T12:30:00.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.116+0000 7fa60ed3c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa60815a940 con 0x7fa60815a550 2024-04-06T12:30:00.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.116+0000 7fa607fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa6080cf900 0x7fa6080b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:00.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.116+0000 7fa607fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa6080cf900 0x7fa6080b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58040/0 (socket says 172.21.15.189:58040) 2024-04-06T12:30:00.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.116+0000 7fa607fff700 1 -- 172.21.15.189:0/2212038482 learned_addr learned my addr 172.21.15.189:0/2212038482 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:00.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.116+0000 7fa60cad8700 1 --2- 172.21.15.189:0/2212038482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa60815a550 0x7fa6080cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:00.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.116+0000 7fa607fff700 1 -- 172.21.15.189:0/2212038482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa60815a550 msgr2=0x7fa6080cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:00.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.116+0000 7fa607fff700 1 --2- 172.21.15.189:0/2212038482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa60815a550 0x7fa6080cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.116+0000 7fa607fff700 1 -- 172.21.15.189:0/2212038482 >> v1:172.21.15.192:6789/0 conn(0x7fa6080b4ba0 legacy=0x7fa60806eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:00.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.116+0000 7fa607fff700 1 -- 172.21.15.189:0/2212038482 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60815ac00 con 0x7fa6080cf900 2024-04-06T12:30:00.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.116+0000 7fa607fff700 1 --2- 172.21.15.189:0/2212038482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6080cf900 0x7fa6080b4660 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fa5f40098f0 tx=0x7fa5f4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82efa47bfa7351b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:00.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.116+0000 7fa6077fe700 1 -- 172.21.15.189:0/2212038482 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f4011830 con 0x7fa6080cf900 2024-04-06T12:30:00.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.116+0000 7fa6077fe700 1 -- 172.21.15.189:0/2212038482 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f4011990 con 0x7fa6080cf900 2024-04-06T12:30:00.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.116+0000 7fa6077fe700 1 -- 172.21.15.189:0/2212038482 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f4011d20 con 0x7fa6080cf900 2024-04-06T12:30:00.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.117+0000 7fa60ed3c700 1 -- 172.21.15.189:0/2212038482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6080cf900 msgr2=0x7fa6080b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:00.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.117+0000 7fa60ed3c700 1 --2- 172.21.15.189:0/2212038482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6080cf900 0x7fa6080b4660 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fa5f40098f0 tx=0x7fa5f4009c00 comp rx=0 tx=0).stop 2024-04-06T12:30:00.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.117+0000 7fa60ed3c700 1 -- 172.21.15.189:0/2212038482 shutdown_connections 2024-04-06T12:30:00.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.117+0000 7fa60ed3c700 1 --2- 172.21.15.189:0/2212038482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa60815a550 0x7fa6080cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.117+0000 7fa60ed3c700 1 --2- 172.21.15.189:0/2212038482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6080cf900 0x7fa6080b4660 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.117+0000 7fa60ed3c700 1 -- 172.21.15.189:0/2212038482 >> 172.21.15.189:0/2212038482 conn(0x7fa6080d00b0 msgr2=0x7fa6080d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:00.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.117+0000 7fa60ed3c700 1 -- 172.21.15.189:0/2212038482 shutdown_connections 2024-04-06T12:30:00.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.117+0000 7fa60ed3c700 1 -- 172.21.15.189:0/2212038482 wait complete. 2024-04-06T12:30:00.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.118+0000 7fa60ed3c700 1 Processor -- start 2024-04-06T12:30:00.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.118+0000 7fa60ed3c700 1 -- start start 2024-04-06T12:30:00.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.119+0000 7fa60ed3c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6080b4ba0 0x7fa6081487c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:00.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.119+0000 7fa60ed3c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6080cf900 0x7fa6081f69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:00.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.119+0000 7fa60ed3c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa6080ad340 0x7fa60814c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:00.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.119+0000 7fa60ed3c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6081f6520 con 0x7fa6080b4ba0 2024-04-06T12:30:00.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.119+0000 7fa60ed3c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6081490c0 con 0x7fa6080cf900 2024-04-06T12:30:00.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.119+0000 7fa60ed3c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6081493a0 con 0x7fa6080ad340 2024-04-06T12:30:00.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.119+0000 7fa60cad8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6080b4ba0 0x7fa6081487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:00.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.119+0000 7fa60cad8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6080b4ba0 0x7fa6081487c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58054/0 (socket says 172.21.15.189:58054) 2024-04-06T12:30:00.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.119+0000 7fa60cad8700 1 -- 172.21.15.189:0/1214063507 learned_addr learned my addr 172.21.15.189:0/1214063507 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:00.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.119+0000 7fa607fff700 1 --2- 172.21.15.189:0/1214063507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6080cf900 0x7fa6081f69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:00.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.119+0000 7fa60d2d9700 1 --2- 172.21.15.189:0/1214063507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa6080ad340 0x7fa60814c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:00.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.119+0000 7fa60cad8700 1 -- 172.21.15.189:0/1214063507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa6080ad340 msgr2=0x7fa60814c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:00.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.119+0000 7fa60cad8700 1 --2- 172.21.15.189:0/1214063507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa6080ad340 0x7fa60814c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.119+0000 7fa60cad8700 1 -- 172.21.15.189:0/1214063507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6080cf900 msgr2=0x7fa6081f69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:00.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.119+0000 7fa60cad8700 1 --2- 172.21.15.189:0/1214063507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6080cf900 0x7fa6081f69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.119+0000 7fa60cad8700 1 -- 172.21.15.189:0/1214063507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f4009550 con 0x7fa6080b4ba0 2024-04-06T12:30:00.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.120+0000 7fa60cad8700 1 --2- 172.21.15.189:0/1214063507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6080b4ba0 0x7fa6081487c0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc00e870 tx=0x7fa5fc00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:00.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.120+0000 7fa6057fa700 1 -- 172.21.15.189:0/1214063507 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc009590 con 0x7fa6080b4ba0 2024-04-06T12:30:00.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.120+0000 7fa6057fa700 1 -- 172.21.15.189:0/1214063507 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5fc013680 con 0x7fa6080b4ba0 2024-04-06T12:30:00.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.120+0000 7fa60ed3c700 1 -- 172.21.15.189:0/1214063507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa608149680 con 0x7fa6080b4ba0 2024-04-06T12:30:00.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.120+0000 7fa6057fa700 1 -- 172.21.15.189:0/1214063507 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc0139f0 con 0x7fa6080b4ba0 2024-04-06T12:30:00.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.120+0000 7fa60ed3c700 1 -- 172.21.15.189:0/1214063507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa608148d00 con 0x7fa6080b4ba0 2024-04-06T12:30:00.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.121+0000 7fa60ed3c700 1 -- 172.21.15.189:0/1214063507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa608067bb0 con 0x7fa6080b4ba0 2024-04-06T12:30:00.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.121+0000 7fa6057fa700 1 -- 172.21.15.189:0/1214063507 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 59) v1 ==== 89361+0+0 (secure 0 0 0) 0x7fa5fc013b50 con 0x7fa6080b4ba0 2024-04-06T12:30:00.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.122+0000 7fa6057fa700 1 --2- 172.21.15.189:0/1214063507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5ec06aed0 0x7fa5ec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:00.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.122+0000 7fa607fff700 1 -- 172.21.15.189:0/1214063507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5ec06aed0 msgr2=0x7fa5ec06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:00.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.122+0000 7fa6057fa700 1 -- 172.21.15.189:0/1214063507 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fa5fc0898d0 con 0x7fa6080b4ba0 2024-04-06T12:30:00.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.122+0000 7fa607fff700 1 --2- 172.21.15.189:0/1214063507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5ec06aed0 0x7fa5ec06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:30:00.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.126+0000 7fa6057fa700 1 -- 172.21.15.189:0/1214063507 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa5fc054440 con 0x7fa6080b4ba0 2024-04-06T12:30:00.179 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:00.181+0000 7f4a4ce9c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:30:00.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.323+0000 7fa607fff700 1 -- 172.21.15.189:0/1214063507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5ec06aed0 msgr2=0x7fa5ec06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:00.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.323+0000 7fa607fff700 1 --2- 172.21.15.189:0/1214063507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5ec06aed0 0x7fa5ec06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:30:00.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.374+0000 7fa60ed3c700 1 -- 172.21.15.189:0/1214063507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa608074990 con 0x7fa6080b4ba0 2024-04-06T12:30:00.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.375+0000 7fa6057fa700 1 -- 172.21.15.189:0/1214063507 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4739 (secure 0 0 0) 0x7fa5fc057a60 con 0x7fa6080b4ba0 2024-04-06T12:30:00.373 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:30:00.373 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":46,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6389,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/17345288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":17345288},{"type":"v1","addr":"172.21.15.192:6835","nonce":17345288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":7078,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/440618958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":440618958},{"type":"v1","addr":"172.21.15.192:6834","nonce":440618958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":7569,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3391245238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3391245238},{"type":"v1","addr":"172.21.15.189:6836","nonce":3391245238}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:28:40.659968+0000","modified":"2024-04-06T12:28:43.668129+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7572":{"gid":7572,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/2225868923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2225868923},{"type":"v1","addr":"172.21.15.189:6837","nonce":2225868923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-04-06T12:30:00.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.377+0000 7fa60ed3c700 1 -- 172.21.15.189:0/1214063507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5ec06aed0 msgr2=0x7fa5ec06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:30:00.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.377+0000 7fa60ed3c700 1 --2- 172.21.15.189:0/1214063507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5ec06aed0 0x7fa5ec06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.377+0000 7fa60ed3c700 1 -- 172.21.15.189:0/1214063507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6080b4ba0 msgr2=0x7fa6081487c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:00.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.377+0000 7fa60ed3c700 1 --2- 172.21.15.189:0/1214063507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6080b4ba0 0x7fa6081487c0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc00e870 tx=0x7fa5fc00eb80 comp rx=0 tx=0).stop 2024-04-06T12:30:00.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.377+0000 7fa60ed3c700 1 -- 172.21.15.189:0/1214063507 shutdown_connections 2024-04-06T12:30:00.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.377+0000 7fa60ed3c700 1 --2- 172.21.15.189:0/1214063507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa6080ad340 0x7fa60814c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.377+0000 7fa60ed3c700 1 --2- 172.21.15.189:0/1214063507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6080b4ba0 0x7fa6081487c0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.377+0000 7fa60ed3c700 1 --2- 172.21.15.189:0/1214063507 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5ec06aed0 0x7fa5ec06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.377+0000 7fa60ed3c700 1 --2- 172.21.15.189:0/1214063507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6080cf900 0x7fa6081f69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.377+0000 7fa60ed3c700 1 -- 172.21.15.189:0/1214063507 >> 172.21.15.189:0/1214063507 conn(0x7fa6080d00b0 msgr2=0x7fa60815c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:00.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.377+0000 7fa60ed3c700 1 -- 172.21.15.189:0/1214063507 shutdown_connections 2024-04-06T12:30:00.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.377+0000 7fa60ed3c700 1 -- 172.21.15.189:0/1214063507 wait complete. 2024-04-06T12:30:00.377 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 46 2024-04-06T12:30:00.388 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:30:00.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.520+0000 7f74eddb7700 1 Processor -- start 2024-04-06T12:30:00.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.520+0000 7f74eddb7700 1 -- start start 2024-04-06T12:30:00.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.521+0000 7f74eddb7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f74e80b3b80 0x7f74e8071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:00.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.521+0000 7f74eddb7700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f74e815b510 0x7f74e8072320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:00.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.521+0000 7f74eddb7700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e80b1fb0 con 0x7f74e815b130 2024-04-06T12:30:00.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.521+0000 7f74eddb7700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74e8072860 con 0x7f74e80b3b80 2024-04-06T12:30:00.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.521+0000 7f74eddb7700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74e80b2200 con 0x7f74e815b510 2024-04-06T12:30:00.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.521+0000 7f74dffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f74e80b3b80 0x7f74e8071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:00.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.522+0000 7f74dffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f74e80b3b80 0x7f74e8071de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58056/0 (socket says 172.21.15.189:58056) 2024-04-06T12:30:00.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.522+0000 7f74dffff700 1 -- 172.21.15.189:0/1811874206 learned_addr learned my addr 172.21.15.189:0/1811874206 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:00.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.522+0000 7f74e7fff700 1 --2- 172.21.15.189:0/1811874206 >> v2:172.21.15.192:3300/0 conn(0x7f74e815b510 0x7f74e8072320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:00.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.522+0000 7f74dffff700 1 -- 172.21.15.189:0/1811874206 >> v2:172.21.15.192:3300/0 conn(0x7f74e815b510 msgr2=0x7f74e8072320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:00.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.522+0000 7f74dffff700 1 --2- 172.21.15.189:0/1811874206 >> v2:172.21.15.192:3300/0 conn(0x7f74e815b510 0x7f74e8072320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.522+0000 7f74dffff700 1 -- 172.21.15.189:0/1811874206 >> v1:172.21.15.192:6789/0 conn(0x7f74e815b130 legacy=0x7f74e80b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:00.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.522+0000 7f74dffff700 1 -- 172.21.15.189:0/1811874206 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e80b24e0 con 0x7f74e80b3b80 2024-04-06T12:30:00.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.522+0000 7f74dffff700 1 --2- 172.21.15.189:0/1811874206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e80b3b80 0x7f74e8071de0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f74d00098f0 tx=0x7f74d0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dcd836ee7ab83f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:00.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.522+0000 7f74e6ffd700 1 -- 172.21.15.189:0/1811874206 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d0011830 con 0x7f74e80b3b80 2024-04-06T12:30:00.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.522+0000 7f74e6ffd700 1 -- 172.21.15.189:0/1811874206 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d0011990 con 0x7f74e80b3b80 2024-04-06T12:30:00.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.522+0000 7f74e6ffd700 1 -- 172.21.15.189:0/1811874206 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d0011d20 con 0x7f74e80b3b80 2024-04-06T12:30:00.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.523+0000 7f74eddb7700 1 -- 172.21.15.189:0/1811874206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e80b3b80 msgr2=0x7f74e8071de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:00.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.523+0000 7f74eddb7700 1 --2- 172.21.15.189:0/1811874206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e80b3b80 0x7f74e8071de0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f74d00098f0 tx=0x7f74d0009c00 comp rx=0 tx=0).stop 2024-04-06T12:30:00.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.523+0000 7f74eddb7700 1 -- 172.21.15.189:0/1811874206 shutdown_connections 2024-04-06T12:30:00.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.523+0000 7f74eddb7700 1 --2- 172.21.15.189:0/1811874206 >> v2:172.21.15.192:3300/0 conn(0x7f74e815b510 0x7f74e8072320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.523+0000 7f74eddb7700 1 --2- 172.21.15.189:0/1811874206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e80b3b80 0x7f74e8071de0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.523+0000 7f74eddb7700 1 -- 172.21.15.189:0/1811874206 >> 172.21.15.189:0/1811874206 conn(0x7f74e806a730 msgr2=0x7f74e806ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:00.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.523+0000 7f74eddb7700 1 -- 172.21.15.189:0/1811874206 shutdown_connections 2024-04-06T12:30:00.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.523+0000 7f74eddb7700 1 -- 172.21.15.189:0/1811874206 wait complete. 2024-04-06T12:30:00.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.524+0000 7f74eddb7700 1 Processor -- start 2024-04-06T12:30:00.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.524+0000 7f74eddb7700 1 -- start start 2024-04-06T12:30:00.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.525+0000 7f74eddb7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e80b3b80 0x7f74e81f3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:00.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.525+0000 7f74eddb7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74e815b130 0x7f74e80ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:00.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.525+0000 7f74eddb7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74e815b510 0x7f74e80cf190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:00.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.525+0000 7f74eddb7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e80ae950 con 0x7f74e80b3b80 2024-04-06T12:30:00.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.525+0000 7f74eddb7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e80ace00 con 0x7f74e815b510 2024-04-06T12:30:00.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.525+0000 7f74eddb7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e80ad0a0 con 0x7f74e815b130 2024-04-06T12:30:00.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.525+0000 7f74e77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e80b3b80 0x7f74e81f3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:00.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.525+0000 7f74e77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e80b3b80 0x7f74e81f3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58060/0 (socket says 172.21.15.189:58060) 2024-04-06T12:30:00.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.525+0000 7f74e77fe700 1 -- 172.21.15.189:0/3581719418 learned_addr learned my addr 172.21.15.189:0/3581719418 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:00.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.525+0000 7f74dffff700 1 --2- 172.21.15.189:0/3581719418 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74e815b130 0x7f74e80ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:00.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.525+0000 7f74e7fff700 1 --2- 172.21.15.189:0/3581719418 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74e815b510 0x7f74e80cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:00.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.525+0000 7f74e77fe700 1 -- 172.21.15.189:0/3581719418 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74e815b130 msgr2=0x7f74e80ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:00.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.525+0000 7f74e77fe700 1 --2- 172.21.15.189:0/3581719418 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74e815b130 0x7f74e80ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.525+0000 7f74e77fe700 1 -- 172.21.15.189:0/3581719418 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74e815b510 msgr2=0x7f74e80cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:00.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.525+0000 7f74e77fe700 1 --2- 172.21.15.189:0/3581719418 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74e815b510 0x7f74e80cf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.525+0000 7f74e77fe700 1 -- 172.21.15.189:0/3581719418 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d0009550 con 0x7f74e80b3b80 2024-04-06T12:30:00.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.526+0000 7f74e77fe700 1 --2- 172.21.15.189:0/3581719418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e80b3b80 0x7f74e81f3b00 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f74d800c4e0 tx=0x7f74d800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:00.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.526+0000 7f74e4ff9700 1 -- 172.21.15.189:0/3581719418 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d80052d0 con 0x7f74e80b3b80 2024-04-06T12:30:00.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.526+0000 7f74e4ff9700 1 -- 172.21.15.189:0/3581719418 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d8005430 con 0x7f74e80b3b80 2024-04-06T12:30:00.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.526+0000 7f74eddb7700 1 -- 172.21.15.189:0/3581719418 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e80ad380 con 0x7f74e80b3b80 2024-04-06T12:30:00.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.526+0000 7f74e4ff9700 1 -- 172.21.15.189:0/3581719418 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d80038c0 con 0x7f74e80b3b80 2024-04-06T12:30:00.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.526+0000 7f74eddb7700 1 -- 172.21.15.189:0/3581719418 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e80cf6d0 con 0x7f74e80b3b80 2024-04-06T12:30:00.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.527+0000 7f74eddb7700 1 -- 172.21.15.189:0/3581719418 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e80ad8a0 con 0x7f74e80b3b80 2024-04-06T12:30:00.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.527+0000 7f74e4ff9700 1 -- 172.21.15.189:0/3581719418 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 59) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f74d8003a20 con 0x7f74e80b3b80 2024-04-06T12:30:00.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.528+0000 7f74e4ff9700 1 --2- 172.21.15.189:0/3581719418 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f74c806aed0 0x7f74c806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:00.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.528+0000 7f74dffff700 1 -- 172.21.15.189:0/3581719418 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f74c806aed0 msgr2=0x7f74c806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:00.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.528+0000 7f74e4ff9700 1 -- 172.21.15.189:0/3581719418 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f74d8088d80 con 0x7f74e80b3b80 2024-04-06T12:30:00.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.528+0000 7f74dffff700 1 --2- 172.21.15.189:0/3581719418 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f74c806aed0 0x7f74c806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:30:00.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.532+0000 7f74e4ff9700 1 -- 172.21.15.189:0/3581719418 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f74e80ad8a0 con 0x7f74e80b3b80 2024-04-06T12:30:00.650 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:00.653+0000 7fdd05b24200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:30:00.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.729+0000 7f74dffff700 1 -- 172.21.15.189:0/3581719418 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f74c806aed0 msgr2=0x7f74c806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:00.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.729+0000 7f74dffff700 1 --2- 172.21.15.189:0/3581719418 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f74c806aed0 0x7f74c806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:30:00.731 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:00.734+0000 7fdd05b24200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:30:00.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.742+0000 7f74eddb7700 1 -- 172.21.15.189:0/3581719418 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74e80ae720 con 0x7f74e80b3b80 2024-04-06T12:30:00.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.743+0000 7f74e4ff9700 1 -- 172.21.15.189:0/3581719418 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+17000 (secure 0 0 0) 0x7f74e80ae720 con 0x7f74e80b3b80 2024-04-06T12:30:00.742 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:30:00.742 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":115,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:29:55.659401+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:28:38.415021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-06T12:28:39.629378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.8","mappings":[{"from":1,"to":7}]},{"pgid":"14.1b","mappings":[{"from":1,"to":2}]},{"pgid":"14.31","mappings":[{"from":1,"to":7}]},{"pgid":"14.36","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/1207971310":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/3711853723":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/1381299583":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/1611293259":"2024-04-07T12:28:54.176092+0000","172.21.15.189:6833/47128":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/1131169404":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/2194712360":"2024-04-07T12:28:54.176092+0000","172.21.15.189:6832/47128":"2024-04-07T12:28:54.176092+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:30:00.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.746+0000 7f74eddb7700 1 -- 172.21.15.189:0/3581719418 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f74c806aed0 msgr2=0x7f74c806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:30:00.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.746+0000 7f74eddb7700 1 --2- 172.21.15.189:0/3581719418 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f74c806aed0 0x7f74c806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.746+0000 7f74eddb7700 1 -- 172.21.15.189:0/3581719418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e80b3b80 msgr2=0x7f74e81f3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:00.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.746+0000 7f74eddb7700 1 --2- 172.21.15.189:0/3581719418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e80b3b80 0x7f74e81f3b00 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f74d800c4e0 tx=0x7f74d800c7f0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.746+0000 7f74eddb7700 1 -- 172.21.15.189:0/3581719418 shutdown_connections 2024-04-06T12:30:00.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.746+0000 7f74eddb7700 1 --2- 172.21.15.189:0/3581719418 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f74c806aed0 0x7f74c806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.746+0000 7f74eddb7700 1 --2- 172.21.15.189:0/3581719418 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74e815b510 0x7f74e80cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.746+0000 7f74eddb7700 1 --2- 172.21.15.189:0/3581719418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e80b3b80 0x7f74e81f3b00 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.746+0000 7f74eddb7700 1 --2- 172.21.15.189:0/3581719418 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74e815b130 0x7f74e80ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.746+0000 7f74eddb7700 1 -- 172.21.15.189:0/3581719418 >> 172.21.15.189:0/3581719418 conn(0x7f74e806a730 msgr2=0x7f74e813cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:00.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.746+0000 7f74eddb7700 1 -- 172.21.15.189:0/3581719418 shutdown_connections 2024-04-06T12:30:00.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.746+0000 7f74eddb7700 1 -- 172.21.15.189:0/3581719418 wait complete. 2024-04-06T12:30:00.757 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:28:38.415021+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:28:39.629378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:30:00.758 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:30:00.838 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:00.840+0000 7f4a4ce9c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:30:00.856 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:00.859+0000 7fdd05b24200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:30:00.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.888+0000 7f5facd36700 1 Processor -- start 2024-04-06T12:30:00.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.888+0000 7f5facd36700 1 -- start start 2024-04-06T12:30:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.890+0000 7f5facd36700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5fa815d7e0 0x7f5fa80d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.890+0000 7f5facd36700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fa80d1cc0 0x7f5fa80b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.890+0000 7f5facd36700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa80cdf80 con 0x7f5fa80d0890 2024-04-06T12:30:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.890+0000 7f5facd36700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa80d3230 con 0x7f5fa815d7e0 2024-04-06T12:30:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.890+0000 7f5facd36700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80d2da0 con 0x7f5fa80d1cc0 2024-04-06T12:30:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.890+0000 7f5fa659c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5fa815d7e0 0x7f5fa80d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.890+0000 7f5fa659c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5fa815d7e0 0x7f5fa80d0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58064/0 (socket says 172.21.15.189:58064) 2024-04-06T12:30:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.890+0000 7f5fa659c700 1 -- 172.21.15.189:0/1614458657 learned_addr learned my addr 172.21.15.189:0/1614458657 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.890+0000 7f5fa6d9d700 1 --2- 172.21.15.189:0/1614458657 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fa80d1cc0 0x7f5fa80b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.890+0000 7f5fa659c700 1 -- 172.21.15.189:0/1614458657 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fa80d1cc0 msgr2=0x7f5fa80b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:00.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.890+0000 7f5fa659c700 1 --2- 172.21.15.189:0/1614458657 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fa80d1cc0 0x7f5fa80b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.890+0000 7f5fa659c700 1 -- 172.21.15.189:0/1614458657 >> v1:172.21.15.192:6789/0 conn(0x7f5fa80d0890 legacy=0x7f5fa80d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.890+0000 7f5fa659c700 1 -- 172.21.15.189:0/1614458657 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa80d0cf0 con 0x7f5fa815d7e0 2024-04-06T12:30:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.890+0000 7f5fa659c700 1 --2- 172.21.15.189:0/1614458657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fa815d7e0 0x7f5fa80d0350 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f5f980098f0 tx=0x7f5f98009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1686ac9c6d42537f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.890+0000 7f5fa559a700 1 -- 172.21.15.189:0/1614458657 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98011830 con 0x7f5fa815d7e0 2024-04-06T12:30:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.891+0000 7f5fa559a700 1 -- 172.21.15.189:0/1614458657 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f98011990 con 0x7f5fa815d7e0 2024-04-06T12:30:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.891+0000 7f5fa559a700 1 -- 172.21.15.189:0/1614458657 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98011d20 con 0x7f5fa815d7e0 2024-04-06T12:30:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.891+0000 7f5facd36700 1 -- 172.21.15.189:0/1614458657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fa815d7e0 msgr2=0x7f5fa80d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.891+0000 7f5facd36700 1 --2- 172.21.15.189:0/1614458657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fa815d7e0 0x7f5fa80d0350 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f5f980098f0 tx=0x7f5f98009c00 comp rx=0 tx=0).stop 2024-04-06T12:30:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.891+0000 7f5facd36700 1 -- 172.21.15.189:0/1614458657 shutdown_connections 2024-04-06T12:30:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.891+0000 7f5facd36700 1 --2- 172.21.15.189:0/1614458657 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fa80d1cc0 0x7f5fa80b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.891+0000 7f5facd36700 1 --2- 172.21.15.189:0/1614458657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fa815d7e0 0x7f5fa80d0350 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.891+0000 7f5facd36700 1 -- 172.21.15.189:0/1614458657 >> 172.21.15.189:0/1614458657 conn(0x7f5fa8072250 msgr2=0x7f5fa8072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.891+0000 7f5facd36700 1 -- 172.21.15.189:0/1614458657 shutdown_connections 2024-04-06T12:30:00.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.891+0000 7f5facd36700 1 -- 172.21.15.189:0/1614458657 wait complete. 2024-04-06T12:30:00.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.893+0000 7f5facd36700 1 Processor -- start 2024-04-06T12:30:00.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.893+0000 7f5facd36700 1 -- start start 2024-04-06T12:30:00.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.894+0000 7f5facd36700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fa80d0890 0x7f5fa8159b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:00.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.894+0000 7f5facd36700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5fa80d1cc0 0x7f5fa80cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:00.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.894+0000 7f5facd36700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fa815d7e0 0x7f5fa80c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:00.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.894+0000 7f5facd36700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80cbd60 con 0x7f5fa815d7e0 2024-04-06T12:30:00.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.894+0000 7f5facd36700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80c7990 con 0x7f5fa80d1cc0 2024-04-06T12:30:00.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.894+0000 7f5facd36700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80cb770 con 0x7f5fa80d0890 2024-04-06T12:30:00.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.894+0000 7f5fa6d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fa815d7e0 0x7f5fa80c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:00.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.894+0000 7f5fa659c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fa80d0890 0x7f5fa8159b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:00.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.894+0000 7f5fa6d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fa815d7e0 0x7f5fa80c6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58080/0 (socket says 172.21.15.189:58080) 2024-04-06T12:30:00.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.894+0000 7f5fa6d9d700 1 -- 172.21.15.189:0/770873206 learned_addr learned my addr 172.21.15.189:0/770873206 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:00.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.894+0000 7f5fa5d9b700 1 --2- 172.21.15.189:0/770873206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5fa80d1cc0 0x7f5fa80cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:00.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.895+0000 7f5fa659c700 1 -- 172.21.15.189:0/770873206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5fa80d1cc0 msgr2=0x7f5fa80cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:00.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.895+0000 7f5fa659c700 1 --2- 172.21.15.189:0/770873206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5fa80d1cc0 0x7f5fa80cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.895+0000 7f5fa659c700 1 -- 172.21.15.189:0/770873206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fa815d7e0 msgr2=0x7f5fa80c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:00.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.895+0000 7f5fa659c700 1 --2- 172.21.15.189:0/770873206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fa815d7e0 0x7f5fa80c6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.895+0000 7f5fa659c700 1 -- 172.21.15.189:0/770873206 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f98009550 con 0x7f5fa80d0890 2024-04-06T12:30:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.895+0000 7f5fa659c700 1 --2- 172.21.15.189:0/770873206 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fa80d0890 0x7f5fa8159b80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5f98016000 tx=0x7f5f98004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.895+0000 7f5f96ffd700 1 -- 172.21.15.189:0/770873206 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98004400 con 0x7f5fa80d0890 2024-04-06T12:30:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.895+0000 7f5f96ffd700 1 -- 172.21.15.189:0/770873206 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9801f070 con 0x7f5fa80d0890 2024-04-06T12:30:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.895+0000 7f5facd36700 1 -- 172.21.15.189:0/770873206 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa815a0c0 con 0x7f5fa80d0890 2024-04-06T12:30:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.895+0000 7f5f96ffd700 1 -- 172.21.15.189:0/770873206 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f980038e0 con 0x7f5fa80d0890 2024-04-06T12:30:00.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.895+0000 7f5facd36700 1 -- 172.21.15.189:0/770873206 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa80c76b0 con 0x7f5fa80d0890 2024-04-06T12:30:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.896+0000 7f5facd36700 1 -- 172.21.15.189:0/770873206 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa80cbea0 con 0x7f5fa80d0890 2024-04-06T12:30:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.896+0000 7f5f96ffd700 1 -- 172.21.15.189:0/770873206 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 59) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f5f98021030 con 0x7f5fa80d0890 2024-04-06T12:30:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.897+0000 7f5f96ffd700 1 --2- 172.21.15.189:0/770873206 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5f8806aed0 0x7f5f8806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.897+0000 7f5fa5d9b700 1 -- 172.21.15.189:0/770873206 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5f8806aed0 msgr2=0x7f5f8806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.897+0000 7f5f96ffd700 1 -- 172.21.15.189:0/770873206 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f5f9808ae00 con 0x7f5fa80d0890 2024-04-06T12:30:00.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.897+0000 7f5fa5d9b700 1 --2- 172.21.15.189:0/770873206 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5f8806aed0 0x7f5f8806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:30:00.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:00.902+0000 7f5f96ffd700 1 -- 172.21.15.189:0/770873206 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5fa80cbea0 con 0x7f5fa80d0890 2024-04-06T12:30:00.922 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:00.925+0000 7f4a4ce9c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:30:01.051 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:01.053+0000 7f4a4ce9c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:30:01.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.098+0000 7f5fa5d9b700 1 -- 172.21.15.189:0/770873206 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5f8806aed0 msgr2=0x7f5f8806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:01.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.098+0000 7f5fa5d9b700 1 --2- 172.21.15.189:0/770873206 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5f8806aed0 0x7f5f8806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:30:01.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.102+0000 7f5facd36700 1 -- 172.21.15.189:0/770873206 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5fa80cbea0 con 0x7f5fa80d0890 2024-04-06T12:30:01.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.102+0000 7f5f96ffd700 1 -- 172.21.15.189:0/770873206 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5fa80cbea0 con 0x7f5fa80d0890 2024-04-06T12:30:01.100 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:30:01.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.105+0000 7f5facd36700 1 -- 172.21.15.189:0/770873206 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5f8806aed0 msgr2=0x7f5f8806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:30:01.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.105+0000 7f5facd36700 1 --2- 172.21.15.189:0/770873206 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5f8806aed0 0x7f5f8806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.105+0000 7f5facd36700 1 -- 172.21.15.189:0/770873206 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fa80d0890 msgr2=0x7f5fa8159b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:01.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.105+0000 7f5facd36700 1 --2- 172.21.15.189:0/770873206 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fa80d0890 0x7f5fa8159b80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5f98016000 tx=0x7f5f98004750 comp rx=0 tx=0).stop 2024-04-06T12:30:01.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.105+0000 7f5facd36700 1 -- 172.21.15.189:0/770873206 shutdown_connections 2024-04-06T12:30:01.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.105+0000 7f5facd36700 1 --2- 172.21.15.189:0/770873206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fa815d7e0 0x7f5fa80c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.105+0000 7f5facd36700 1 --2- 172.21.15.189:0/770873206 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fa80d0890 0x7f5fa8159b80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.105+0000 7f5facd36700 1 --2- 172.21.15.189:0/770873206 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5f8806aed0 0x7f5f8806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.105+0000 7f5facd36700 1 --2- 172.21.15.189:0/770873206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5fa80d1cc0 0x7f5fa80cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.105+0000 7f5facd36700 1 -- 172.21.15.189:0/770873206 >> 172.21.15.189:0/770873206 conn(0x7f5fa8072250 msgr2=0x7f5fa80b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:01.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.105+0000 7f5facd36700 1 -- 172.21.15.189:0/770873206 shutdown_connections 2024-04-06T12:30:01.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.105+0000 7f5facd36700 1 -- 172.21.15.189:0/770873206 wait complete. 2024-04-06T12:30:01.116 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:30:01.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.250+0000 7fab0bf4a700 1 Processor -- start 2024-04-06T12:30:01.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.250+0000 7fab0bf4a700 1 -- start start 2024-04-06T12:30:01.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.251+0000 7fab0bf4a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab040d11b0 0x7fab040d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:01.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.251+0000 7fab0bf4a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fab040cb870 0x7fab0415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:01.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.251+0000 7fab0bf4a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab040d1ad0 con 0x7fab040cfa00 2024-04-06T12:30:01.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.251+0000 7fab0bf4a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab040ccdc0 con 0x7fab040cb870 2024-04-06T12:30:01.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.251+0000 7fab0bf4a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab040d0b50 con 0x7fab040d11b0 2024-04-06T12:30:01.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.251+0000 7fab094e5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fab040cb870 0x7fab0415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:01.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.251+0000 7fab094e5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fab040cb870 0x7fab0415c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58096/0 (socket says 172.21.15.189:58096) 2024-04-06T12:30:01.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.251+0000 7fab094e5700 1 -- 172.21.15.189:0/3211699256 learned_addr learned my addr 172.21.15.189:0/3211699256 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:01.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.251+0000 7fab09ce6700 1 --2- 172.21.15.189:0/3211699256 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab040d11b0 0x7fab040d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:01.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.251+0000 7fab094e5700 1 -- 172.21.15.189:0/3211699256 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab040d11b0 msgr2=0x7fab040d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:01.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.251+0000 7fab094e5700 1 --2- 172.21.15.189:0/3211699256 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab040d11b0 0x7fab040d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.251+0000 7fab094e5700 1 -- 172.21.15.189:0/3211699256 >> v1:172.21.15.192:6789/0 conn(0x7fab040cfa00 legacy=0x7fab040cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:01.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.251+0000 7fab094e5700 1 -- 172.21.15.189:0/3211699256 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab040cbd80 con 0x7fab040cb870 2024-04-06T12:30:01.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.252+0000 7fab094e5700 1 --2- 172.21.15.189:0/3211699256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab040cb870 0x7fab0415c280 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7faaf40098f0 tx=0x7faaf4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c63c4a4027784a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:01.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.252+0000 7fab08ce4700 1 -- 172.21.15.189:0/3211699256 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf4011830 con 0x7fab040cb870 2024-04-06T12:30:01.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.252+0000 7fab08ce4700 1 -- 172.21.15.189:0/3211699256 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf4011990 con 0x7fab040cb870 2024-04-06T12:30:01.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.252+0000 7fab08ce4700 1 -- 172.21.15.189:0/3211699256 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf4011d20 con 0x7fab040cb870 2024-04-06T12:30:01.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.252+0000 7fab0bf4a700 1 -- 172.21.15.189:0/3211699256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab040cb870 msgr2=0x7fab0415c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:01.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.252+0000 7fab0bf4a700 1 --2- 172.21.15.189:0/3211699256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab040cb870 0x7fab0415c280 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7faaf40098f0 tx=0x7faaf4009c00 comp rx=0 tx=0).stop 2024-04-06T12:30:01.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.253+0000 7fab0bf4a700 1 -- 172.21.15.189:0/3211699256 shutdown_connections 2024-04-06T12:30:01.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.253+0000 7fab0bf4a700 1 --2- 172.21.15.189:0/3211699256 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab040d11b0 0x7fab040d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.253+0000 7fab0bf4a700 1 --2- 172.21.15.189:0/3211699256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab040cb870 0x7fab0415c280 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.253+0000 7fab0bf4a700 1 -- 172.21.15.189:0/3211699256 >> 172.21.15.189:0/3211699256 conn(0x7fab040b2db0 msgr2=0x7fab040b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:01.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.253+0000 7fab0bf4a700 1 -- 172.21.15.189:0/3211699256 shutdown_connections 2024-04-06T12:30:01.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.253+0000 7fab0bf4a700 1 -- 172.21.15.189:0/3211699256 wait complete. 2024-04-06T12:30:01.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.254+0000 7fab0bf4a700 1 Processor -- start 2024-04-06T12:30:01.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.254+0000 7fab0bf4a700 1 -- start start 2024-04-06T12:30:01.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.255+0000 7fab0bf4a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab040cb870 0x7fab0414f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:01.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.255+0000 7fab0bf4a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab040cfa00 0x7fab040c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:01.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.255+0000 7fab0bf4a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab040d11b0 0x7fab0415c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:01.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.255+0000 7fab0bf4a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab04156120 con 0x7fab040cfa00 2024-04-06T12:30:01.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.255+0000 7fab0bf4a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab040c6ea0 con 0x7fab040d11b0 2024-04-06T12:30:01.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.255+0000 7fab0bf4a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab040c5b60 con 0x7fab040cb870 2024-04-06T12:30:01.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.255+0000 7fab094e5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab040cfa00 0x7fab040c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:01.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.255+0000 7fab094e5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab040cfa00 0x7fab040c7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58100/0 (socket says 172.21.15.189:58100) 2024-04-06T12:30:01.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.255+0000 7fab094e5700 1 -- 172.21.15.189:0/2268096296 learned_addr learned my addr 172.21.15.189:0/2268096296 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:01.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.255+0000 7fab09ce6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab040cb870 0x7fab0414f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:01.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.255+0000 7fab0a4e7700 1 --2- 172.21.15.189:0/2268096296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab040d11b0 0x7fab0415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:01.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.255+0000 7fab09ce6700 1 -- 172.21.15.189:0/2268096296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab040d11b0 msgr2=0x7fab0415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:01.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.255+0000 7fab09ce6700 1 --2- 172.21.15.189:0/2268096296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab040d11b0 0x7fab0415c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.255+0000 7fab09ce6700 1 -- 172.21.15.189:0/2268096296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab040cfa00 msgr2=0x7fab040c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:01.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.255+0000 7fab09ce6700 1 --2- 172.21.15.189:0/2268096296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab040cfa00 0x7fab040c7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.255+0000 7fab09ce6700 1 -- 172.21.15.189:0/2268096296 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf4009550 con 0x7fab040cb870 2024-04-06T12:30:01.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.255+0000 7fab094e5700 1 --2- 172.21.15.189:0/2268096296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab040cfa00 0x7fab040c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:30:01.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.255+0000 7fab09ce6700 1 --2- 172.21.15.189:0/2268096296 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab040cb870 0x7fab0414f610 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fab0000e870 tx=0x7fab0000eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:01.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.255+0000 7faafa7fc700 1 -- 172.21.15.189:0/2268096296 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab00013680 con 0x7fab040cb870 2024-04-06T12:30:01.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.256+0000 7faafa7fc700 1 -- 172.21.15.189:0/2268096296 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab000137e0 con 0x7fab040cb870 2024-04-06T12:30:01.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.256+0000 7fab0bf4a700 1 -- 172.21.15.189:0/2268096296 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab040c6570 con 0x7fab040cb870 2024-04-06T12:30:01.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.256+0000 7faafa7fc700 1 -- 172.21.15.189:0/2268096296 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab00013b90 con 0x7fab040cb870 2024-04-06T12:30:01.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.256+0000 7fab0bf4a700 1 -- 172.21.15.189:0/2268096296 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab040c6b30 con 0x7fab040cb870 2024-04-06T12:30:01.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.256+0000 7fab0bf4a700 1 -- 172.21.15.189:0/2268096296 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab040c7180 con 0x7fab040cb870 2024-04-06T12:30:01.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.257+0000 7faafa7fc700 1 -- 172.21.15.189:0/2268096296 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 59) v1 ==== 89361+0+0 (secure 0 0 0) 0x7fab00013cf0 con 0x7fab040cb870 2024-04-06T12:30:01.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.257+0000 7faafa7fc700 1 --2- 172.21.15.189:0/2268096296 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faae806aed0 0x7faae806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:01.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.257+0000 7fab094e5700 1 -- 172.21.15.189:0/2268096296 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faae806aed0 msgr2=0x7faae806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:01.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.257+0000 7faafa7fc700 1 -- 172.21.15.189:0/2268096296 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fab00089940 con 0x7fab040cb870 2024-04-06T12:30:01.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.257+0000 7fab094e5700 1 --2- 172.21.15.189:0/2268096296 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faae806aed0 0x7faae806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:30:01.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.262+0000 7faafa7fc700 1 -- 172.21.15.189:0/2268096296 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fab00054390 con 0x7fab040cb870 2024-04-06T12:30:01.362 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:01.365+0000 7fdd05b24200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:30:01.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.458+0000 7fab094e5700 1 -- 172.21.15.189:0/2268096296 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faae806aed0 msgr2=0x7faae806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:01.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.458+0000 7fab094e5700 1 --2- 172.21.15.189:0/2268096296 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faae806aed0 0x7faae806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:30:01.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.465+0000 7fab0bf4a700 1 -- 172.21.15.189:0/2268096296 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fab04074990 con 0x7fab040cb870 2024-04-06T12:30:01.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.465+0000 7faafa7fc700 1 -- 172.21.15.189:0/2268096296 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v115) v1 ==== 104+0+11 (secure 0 0 0) 0x7fab000579b0 con 0x7fab040cb870 2024-04-06T12:30:01.463 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:30:01.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.467+0000 7fab0bf4a700 1 -- 172.21.15.189:0/2268096296 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faae806aed0 msgr2=0x7faae806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:30:01.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.467+0000 7fab0bf4a700 1 --2- 172.21.15.189:0/2268096296 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faae806aed0 0x7faae806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.467+0000 7fab0bf4a700 1 -- 172.21.15.189:0/2268096296 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab040cb870 msgr2=0x7fab0414f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:01.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.467+0000 7fab0bf4a700 1 --2- 172.21.15.189:0/2268096296 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab040cb870 0x7fab0414f610 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fab0000e870 tx=0x7fab0000eb80 comp rx=0 tx=0).stop 2024-04-06T12:30:01.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.467+0000 7fab0bf4a700 1 -- 172.21.15.189:0/2268096296 shutdown_connections 2024-04-06T12:30:01.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.467+0000 7fab0bf4a700 1 --2- 172.21.15.189:0/2268096296 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7faae806aed0 0x7faae806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.467+0000 7fab0bf4a700 1 --2- 172.21.15.189:0/2268096296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab040d11b0 0x7fab0415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.467+0000 7fab0bf4a700 1 --2- 172.21.15.189:0/2268096296 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab040cb870 0x7fab0414f610 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.467+0000 7fab0bf4a700 1 --2- 172.21.15.189:0/2268096296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab040cfa00 0x7fab040c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.468+0000 7fab0bf4a700 1 -- 172.21.15.189:0/2268096296 >> 172.21.15.189:0/2268096296 conn(0x7fab040b2db0 msgr2=0x7fab0415a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:01.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.468+0000 7fab0bf4a700 1 -- 172.21.15.189:0/2268096296 shutdown_connections 2024-04-06T12:30:01.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.468+0000 7fab0bf4a700 1 -- 172.21.15.189:0/2268096296 wait complete. 2024-04-06T12:30:01.478 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:30:01.515 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:01.518+0000 7fdd05b24200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:30:01.594 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:01.596+0000 7f4a4ce9c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:30:01.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.609+0000 7f6432ab7700 1 Processor -- start 2024-04-06T12:30:01.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.609+0000 7f6432ab7700 1 -- start start 2024-04-06T12:30:01.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.609+0000 7f6432ab7700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f642c0cd3e0 0x7f642c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:01.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.609+0000 7f6432ab7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642c06bcd0 con 0x7f642c15d7c0 2024-04-06T12:30:01.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.610+0000 7f6432ab7700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642c06ceb0 con 0x7f642c06b8c0 2024-04-06T12:30:01.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.610+0000 7f6432ab7700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f642c0cdfa0 con 0x7f642c0cd3e0 2024-04-06T12:30:01.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.610+0000 7f642bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f642c15d7c0 0x7f642c06b1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40274/0 (socket says 172.21.15.189:40274) 2024-04-06T12:30:01.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.610+0000 7f642bfff700 1 -- 172.21.15.189:0/2507540357 learned_addr learned my addr 172.21.15.189:0/2507540357 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:01.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.610+0000 7f6430853700 1 --2- 172.21.15.189:0/2507540357 >> v2:172.21.15.192:3300/0 conn(0x7f642c0cd3e0 0x7f642c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:01.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.610+0000 7f642b7fe700 1 -- 172.21.15.189:0/2507540357 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734238548 0 0) 0x7f642c06bcd0 con 0x7f642c15d7c0 2024-04-06T12:30:01.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.610+0000 7f642b7fe700 1 -- 172.21.15.189:0/2507540357 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f641c0035b0 con 0x7f642c15d7c0 2024-04-06T12:30:01.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.610+0000 7f6430853700 1 -- 172.21.15.189:0/2507540357 >> v1:172.21.15.192:6789/0 conn(0x7f642c06b8c0 legacy=0x7f642c06dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:01.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.610+0000 7f6430853700 1 -- 172.21.15.189:0/2507540357 >> v1:172.21.15.189:6789/0 conn(0x7f642c15d7c0 legacy=0x7f642c06b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:01.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.610+0000 7f6430853700 1 -- 172.21.15.189:0/2507540357 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c06d750 con 0x7f642c0cd3e0 2024-04-06T12:30:01.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.610+0000 7f6430853700 1 --2- 172.21.15.189:0/2507540357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f642c0cd3e0 0x7f642c0cd7c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6420009840 tx=0x7f6420009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9699841b96fca878 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:01.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.610+0000 7f642b7fe700 1 -- 172.21.15.189:0/2507540357 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6420011830 con 0x7f642c0cd3e0 2024-04-06T12:30:01.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.610+0000 7f642b7fe700 1 -- 172.21.15.189:0/2507540357 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6420011990 con 0x7f642c0cd3e0 2024-04-06T12:30:01.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.611+0000 7f642b7fe700 1 -- 172.21.15.189:0/2507540357 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6420011d20 con 0x7f642c0cd3e0 2024-04-06T12:30:01.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.611+0000 7f6432ab7700 1 -- 172.21.15.189:0/2507540357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f642c0cd3e0 msgr2=0x7f642c0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:01.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.611+0000 7f6432ab7700 1 --2- 172.21.15.189:0/2507540357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f642c0cd3e0 0x7f642c0cd7c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6420009840 tx=0x7f6420009b50 comp rx=0 tx=0).stop 2024-04-06T12:30:01.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.611+0000 7f6432ab7700 1 -- 172.21.15.189:0/2507540357 shutdown_connections 2024-04-06T12:30:01.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.611+0000 7f6432ab7700 1 --2- 172.21.15.189:0/2507540357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f642c0cd3e0 0x7f642c0cd7c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.611+0000 7f6432ab7700 1 -- 172.21.15.189:0/2507540357 >> 172.21.15.189:0/2507540357 conn(0x7f642c0cc290 msgr2=0x7f642c0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:01.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.611+0000 7f6432ab7700 1 -- 172.21.15.189:0/2507540357 shutdown_connections 2024-04-06T12:30:01.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.611+0000 7f6432ab7700 1 -- 172.21.15.189:0/2507540357 wait complete. 2024-04-06T12:30:01.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.612+0000 7f6432ab7700 1 Processor -- start 2024-04-06T12:30:01.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.612+0000 7f6432ab7700 1 -- start start 2024-04-06T12:30:01.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.613+0000 7f6432ab7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f642c06b8c0 0x7f642c0c4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:01.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.613+0000 7f6432ab7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f642c0cd3e0 0x7f642c15e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:01.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.613+0000 7f6432ab7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f642c15d7c0 0x7f642c0b3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:01.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.613+0000 7f6432ab7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c0b29b0 con 0x7f642c15d7c0 2024-04-06T12:30:01.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.613+0000 7f6432ab7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c1f0b90 con 0x7f642c0cd3e0 2024-04-06T12:30:01.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.613+0000 7f6432ab7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642c0c37b0 con 0x7f642c06b8c0 2024-04-06T12:30:01.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.613+0000 7f6431054700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f642c15d7c0 0x7f642c0b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:01.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.613+0000 7f642bfff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f642c0cd3e0 0x7f642c15e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:01.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.613+0000 7f6431054700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f642c15d7c0 0x7f642c0b3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58116/0 (socket says 172.21.15.189:58116) 2024-04-06T12:30:01.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.613+0000 7f6431054700 1 -- 172.21.15.189:0/3947021164 learned_addr learned my addr 172.21.15.189:0/3947021164 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:01.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.613+0000 7f6430853700 1 --2- 172.21.15.189:0/3947021164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f642c06b8c0 0x7f642c0c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:01.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.614+0000 7f6431054700 1 -- 172.21.15.189:0/3947021164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f642c06b8c0 msgr2=0x7f642c0c4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:01.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.614+0000 7f6431054700 1 --2- 172.21.15.189:0/3947021164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f642c06b8c0 0x7f642c0c4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.614+0000 7f6431054700 1 -- 172.21.15.189:0/3947021164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f642c0cd3e0 msgr2=0x7f642c15e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:01.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.614+0000 7f6431054700 1 --2- 172.21.15.189:0/3947021164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f642c0cd3e0 0x7f642c15e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.614+0000 7f6431054700 1 -- 172.21.15.189:0/3947021164 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64200094a0 con 0x7f642c15d7c0 2024-04-06T12:30:01.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.614+0000 7f642bfff700 1 --2- 172.21.15.189:0/3947021164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f642c0cd3e0 0x7f642c15e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:30:01.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.614+0000 7f6431054700 1 --2- 172.21.15.189:0/3947021164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f642c15d7c0 0x7f642c0b3290 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f642400cc50 tx=0x7f642400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:01.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.614+0000 7f64297fa700 1 -- 172.21.15.189:0/3947021164 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424003c00 con 0x7f642c15d7c0 2024-04-06T12:30:01.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.614+0000 7f64297fa700 1 -- 172.21.15.189:0/3947021164 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6424003d60 con 0x7f642c15d7c0 2024-04-06T12:30:01.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.614+0000 7f6432ab7700 1 -- 172.21.15.189:0/3947021164 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642c0c3a90 con 0x7f642c15d7c0 2024-04-06T12:30:01.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.614+0000 7f64297fa700 1 -- 172.21.15.189:0/3947021164 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642400aa50 con 0x7f642c15d7c0 2024-04-06T12:30:01.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.614+0000 7f6432ab7700 1 -- 172.21.15.189:0/3947021164 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642c0c4050 con 0x7f642c15d7c0 2024-04-06T12:30:01.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.615+0000 7f6432ab7700 1 -- 172.21.15.189:0/3947021164 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642c074990 con 0x7f642c15d7c0 2024-04-06T12:30:01.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.615+0000 7f64297fa700 1 -- 172.21.15.189:0/3947021164 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 59) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f642400abb0 con 0x7f642c15d7c0 2024-04-06T12:30:01.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.616+0000 7f64297fa700 1 --2- 172.21.15.189:0/3947021164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f641006aed0 0x7f641006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:01.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.616+0000 7f6430853700 1 -- 172.21.15.189:0/3947021164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f641006aed0 msgr2=0x7f641006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:01.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.616+0000 7f64297fa700 1 -- 172.21.15.189:0/3947021164 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f64240899c0 con 0x7f642c15d7c0 2024-04-06T12:30:01.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.616+0000 7f6430853700 1 --2- 172.21.15.189:0/3947021164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f641006aed0 0x7f641006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:30:01.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.620+0000 7f64297fa700 1 -- 172.21.15.189:0/3947021164 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6424054530 con 0x7f642c15d7c0 2024-04-06T12:30:01.673 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:01.676+0000 7fdd05b24200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:30:01.759 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:01.761+0000 7f4a4ce9c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:30:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.817+0000 7f6430853700 1 -- 172.21.15.189:0/3947021164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f641006aed0 msgr2=0x7f641006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.817+0000 7f6430853700 1 --2- 172.21.15.189:0/3947021164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f641006aed0 0x7f641006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:30:01.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.823+0000 7f6432ab7700 1 -- 172.21.15.189:0/3947021164 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f642c0cbcf0 con 0x7f642c15d7c0 2024-04-06T12:30:01.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.824+0000 7f64297fa700 1 -- 172.21.15.189:0/3947021164 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v115) v1 ==== 100+0+11 (secure 0 0 0) 0x7f642400ae60 con 0x7f642c15d7c0 2024-04-06T12:30:01.822 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:30:01.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.826+0000 7f6432ab7700 1 -- 172.21.15.189:0/3947021164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f641006aed0 msgr2=0x7f641006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:30:01.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.826+0000 7f6432ab7700 1 --2- 172.21.15.189:0/3947021164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f641006aed0 0x7f641006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.826+0000 7f6432ab7700 1 -- 172.21.15.189:0/3947021164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f642c15d7c0 msgr2=0x7f642c0b3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:01.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.826+0000 7f6432ab7700 1 --2- 172.21.15.189:0/3947021164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f642c15d7c0 0x7f642c0b3290 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f642400cc50 tx=0x7f642400cf60 comp rx=0 tx=0).stop 2024-04-06T12:30:01.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.826+0000 7f6432ab7700 1 -- 172.21.15.189:0/3947021164 shutdown_connections 2024-04-06T12:30:01.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.826+0000 7f6432ab7700 1 --2- 172.21.15.189:0/3947021164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f642c15d7c0 0x7f642c0b3290 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.826+0000 7f6432ab7700 1 --2- 172.21.15.189:0/3947021164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f642c06b8c0 0x7f642c0c4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.826+0000 7f6432ab7700 1 --2- 172.21.15.189:0/3947021164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f641006aed0 0x7f641006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.826+0000 7f6432ab7700 1 --2- 172.21.15.189:0/3947021164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f642c0cd3e0 0x7f642c15e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.826+0000 7f6432ab7700 1 -- 172.21.15.189:0/3947021164 >> 172.21.15.189:0/3947021164 conn(0x7f642c0cc290 msgr2=0x7f642c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:01.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.826+0000 7f6432ab7700 1 -- 172.21.15.189:0/3947021164 shutdown_connections 2024-04-06T12:30:01.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.826+0000 7f6432ab7700 1 -- 172.21.15.189:0/3947021164 wait complete. 2024-04-06T12:30:01.837 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:30:01.928 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:01.930+0000 7f4a4ce9c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:30:01.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.965+0000 7f7321df7700 1 Processor -- start 2024-04-06T12:30:01.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.965+0000 7f7321df7700 1 -- start start 2024-04-06T12:30:01.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.966+0000 7f7321df7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f731c158fe0 0x7f731c06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:01.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.966+0000 7f7321df7700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f731c15c2d0 0x7f731c0ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:01.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.966+0000 7f7321df7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c15f9e0 con 0x7f731c06ac70 2024-04-06T12:30:01.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.966+0000 7f7321df7700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f731c06b0f0 con 0x7f731c158fe0 2024-04-06T12:30:01.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.966+0000 7f7321df7700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f731c15c6e0 con 0x7f731c15c2d0 2024-04-06T12:30:01.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.966+0000 7f731b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f731c158fe0 0x7f731c06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:01.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.966+0000 7f731affd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f731c06ac70 0x7f731c15bbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40288/0 (socket says 172.21.15.189:40288) 2024-04-06T12:30:01.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.966+0000 7f731affd700 1 -- 172.21.15.189:0/1689246023 learned_addr learned my addr 172.21.15.189:0/1689246023 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:01.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.966+0000 7f731b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f731c158fe0 0x7f731c06a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58118/0 (socket says 172.21.15.189:58118) 2024-04-06T12:30:01.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.966+0000 7f731bfff700 1 --2- 172.21.15.189:0/1689246023 >> v2:172.21.15.192:3300/0 conn(0x7f731c15c2d0 0x7f731c0ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:01.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.966+0000 7f731b7fe700 1 -- 172.21.15.189:0/1689246023 >> v2:172.21.15.192:3300/0 conn(0x7f731c15c2d0 msgr2=0x7f731c0ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:01.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.966+0000 7f731b7fe700 1 --2- 172.21.15.189:0/1689246023 >> v2:172.21.15.192:3300/0 conn(0x7f731c15c2d0 0x7f731c0ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.966+0000 7f731b7fe700 1 -- 172.21.15.189:0/1689246023 >> v1:172.21.15.189:6789/0 conn(0x7f731c06ac70 legacy=0x7f731c15bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:01.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.966+0000 7f731b7fe700 1 -- 172.21.15.189:0/1689246023 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c15f0d0 con 0x7f731c158fe0 2024-04-06T12:30:01.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.966+0000 7f731a7fc700 1 -- 172.21.15.189:0/1689246023 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681106660 0 0) 0x7f731c15f9e0 con 0x7f731c06ac70 2024-04-06T12:30:01.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.966+0000 7f731b7fe700 1 --2- 172.21.15.189:0/1689246023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f731c158fe0 0x7f731c06a730 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f731c0afbe0 tx=0x7f730c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af40af96c299864 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:01.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.966+0000 7f731a7fc700 1 -- 172.21.15.189:0/1689246023 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c011830 con 0x7f731c158fe0 2024-04-06T12:30:01.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.966+0000 7f731a7fc700 1 -- 172.21.15.189:0/1689246023 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730c011990 con 0x7f731c158fe0 2024-04-06T12:30:01.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.967+0000 7f731a7fc700 1 -- 172.21.15.189:0/1689246023 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c011d20 con 0x7f731c158fe0 2024-04-06T12:30:01.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.967+0000 7f7321df7700 1 -- 172.21.15.189:0/1689246023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f731c158fe0 msgr2=0x7f731c06a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:01.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.967+0000 7f7321df7700 1 --2- 172.21.15.189:0/1689246023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f731c158fe0 0x7f731c06a730 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f731c0afbe0 tx=0x7f730c009b20 comp rx=0 tx=0).stop 2024-04-06T12:30:01.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.967+0000 7f7321df7700 1 -- 172.21.15.189:0/1689246023 shutdown_connections 2024-04-06T12:30:01.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.967+0000 7f7321df7700 1 --2- 172.21.15.189:0/1689246023 >> v2:172.21.15.192:3300/0 conn(0x7f731c15c2d0 0x7f731c0ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.967+0000 7f7321df7700 1 --2- 172.21.15.189:0/1689246023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f731c158fe0 0x7f731c06a730 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.967+0000 7f7321df7700 1 -- 172.21.15.189:0/1689246023 >> 172.21.15.189:0/1689246023 conn(0x7f731c0d00b0 msgr2=0x7f731c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:01.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.967+0000 7f7321df7700 1 -- 172.21.15.189:0/1689246023 shutdown_connections 2024-04-06T12:30:01.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.967+0000 7f7321df7700 1 -- 172.21.15.189:0/1689246023 wait complete. 2024-04-06T12:30:01.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.968+0000 7f7321df7700 1 Processor -- start 2024-04-06T12:30:01.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.969+0000 7f7321df7700 1 -- start start 2024-04-06T12:30:01.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.969+0000 7f7321df7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f731c06ac70 0x7f731c157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:01.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.969+0000 7f7321df7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f731c158fe0 0x7f731c0ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:01.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.969+0000 7f7321df7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f731c15c2d0 0x7f731c141dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:01.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.969+0000 7f7321df7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731c140d20 con 0x7f731c15c2d0 2024-04-06T12:30:01.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.969+0000 7f7321df7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731c141000 con 0x7f731c158fe0 2024-04-06T12:30:01.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.969+0000 7f7321df7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731c1412e0 con 0x7f731c06ac70 2024-04-06T12:30:01.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.969+0000 7f731bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f731c15c2d0 0x7f731c141dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:01.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.970+0000 7f731bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f731c15c2d0 0x7f731c141dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58134/0 (socket says 172.21.15.189:58134) 2024-04-06T12:30:01.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.970+0000 7f731bfff700 1 -- 172.21.15.189:0/2451381969 learned_addr learned my addr 172.21.15.189:0/2451381969 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:01.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.970+0000 7f731affd700 1 --2- 172.21.15.189:0/2451381969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f731c158fe0 0x7f731c0ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:01.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.970+0000 7f731b7fe700 1 --2- 172.21.15.189:0/2451381969 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f731c06ac70 0x7f731c157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:01.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.970+0000 7f731bfff700 1 -- 172.21.15.189:0/2451381969 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f731c06ac70 msgr2=0x7f731c157750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:01.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.970+0000 7f731bfff700 1 --2- 172.21.15.189:0/2451381969 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f731c06ac70 0x7f731c157750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.970+0000 7f731bfff700 1 -- 172.21.15.189:0/2451381969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f731c158fe0 msgr2=0x7f731c0ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:01.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.970+0000 7f731bfff700 1 --2- 172.21.15.189:0/2451381969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f731c158fe0 0x7f731c0ad010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:01.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.970+0000 7f731bfff700 1 -- 172.21.15.189:0/2451381969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c0094a0 con 0x7f731c15c2d0 2024-04-06T12:30:01.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.970+0000 7f731bfff700 1 --2- 172.21.15.189:0/2451381969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f731c15c2d0 0x7f731c141dc0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f731400efc0 tx=0x7f731400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.970+0000 7f7303fff700 1 -- 172.21.15.189:0/2451381969 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314009ea0 con 0x7f731c15c2d0 2024-04-06T12:30:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.970+0000 7f7321df7700 1 -- 172.21.15.189:0/2451381969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731c1415c0 con 0x7f731c15c2d0 2024-04-06T12:30:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.970+0000 7f7303fff700 1 -- 172.21.15.189:0/2451381969 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f731400f040 con 0x7f731c15c2d0 2024-04-06T12:30:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.970+0000 7f7303fff700 1 -- 172.21.15.189:0/2451381969 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731400ced0 con 0x7f731c15c2d0 2024-04-06T12:30:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.970+0000 7f7321df7700 1 -- 172.21.15.189:0/2451381969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731c0ad550 con 0x7f731c15c2d0 2024-04-06T12:30:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.971+0000 7f7321df7700 1 -- 172.21.15.189:0/2451381969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f731c141ae0 con 0x7f731c15c2d0 2024-04-06T12:30:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.971+0000 7f7303fff700 1 -- 172.21.15.189:0/2451381969 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 59) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f7314014480 con 0x7f731c15c2d0 2024-04-06T12:30:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.972+0000 7f7303fff700 1 --2- 172.21.15.189:0/2451381969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f730806aed0 0x7f730806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.972+0000 7f731b7fe700 1 -- 172.21.15.189:0/2451381969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f730806aed0 msgr2=0x7f730806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.972+0000 7f7303fff700 1 -- 172.21.15.189:0/2451381969 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f731408a370 con 0x7f731c15c2d0 2024-04-06T12:30:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.972+0000 7f731b7fe700 1 --2- 172.21.15.189:0/2451381969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f730806aed0 0x7f730806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:30:01.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:01.976+0000 7f7303fff700 1 -- 172.21.15.189:0/2451381969 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f731c141ae0 con 0x7f731c15c2d0 2024-04-06T12:30:02.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.173+0000 7f731b7fe700 1 -- 172.21.15.189:0/2451381969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f730806aed0 msgr2=0x7f730806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:02.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.173+0000 7f731b7fe700 1 --2- 172.21.15.189:0/2451381969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f730806aed0 0x7f730806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:30:02.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.225+0000 7f7321df7700 1 -- 172.21.15.189:0/2451381969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f731c141ae0 con 0x7f731c15c2d0 2024-04-06T12:30:02.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.226+0000 7f7303fff700 1 -- 172.21.15.189:0/2451381969 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f731c141ae0 con 0x7f731c15c2d0 2024-04-06T12:30:02.224 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:30:02.224 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":46,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6389,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/17345288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":17345288},{"type":"v1","addr":"172.21.15.192:6835","nonce":17345288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":7078,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/440618958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":440618958},{"type":"v1","addr":"172.21.15.192:6834","nonce":440618958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":7569,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3391245238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3391245238},{"type":"v1","addr":"172.21.15.189:6836","nonce":3391245238}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:28:40.659968+0000","modified":"2024-04-06T12:28:43.668129+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7572":{"gid":7572,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/2225868923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2225868923},{"type":"v1","addr":"172.21.15.189:6837","nonce":2225868923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-04-06T12:30:02.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.228+0000 7f7321df7700 1 -- 172.21.15.189:0/2451381969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f730806aed0 msgr2=0x7f730806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:30:02.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.228+0000 7f7321df7700 1 --2- 172.21.15.189:0/2451381969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f730806aed0 0x7f730806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.228+0000 7f7321df7700 1 -- 172.21.15.189:0/2451381969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f731c15c2d0 msgr2=0x7f731c141dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:02.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.228+0000 7f7321df7700 1 --2- 172.21.15.189:0/2451381969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f731c15c2d0 0x7f731c141dc0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f731400efc0 tx=0x7f731400c3a0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.228+0000 7f7321df7700 1 -- 172.21.15.189:0/2451381969 shutdown_connections 2024-04-06T12:30:02.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.228+0000 7f7321df7700 1 --2- 172.21.15.189:0/2451381969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f731c15c2d0 0x7f731c141dc0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.228+0000 7f7321df7700 1 --2- 172.21.15.189:0/2451381969 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f731c06ac70 0x7f731c157750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.228+0000 7f7321df7700 1 --2- 172.21.15.189:0/2451381969 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f730806aed0 0x7f730806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.228+0000 7f7321df7700 1 --2- 172.21.15.189:0/2451381969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f731c158fe0 0x7f731c0ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.228+0000 7f7321df7700 1 -- 172.21.15.189:0/2451381969 >> 172.21.15.189:0/2451381969 conn(0x7f731c0d00b0 msgr2=0x7f731c158350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:02.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.228+0000 7f7321df7700 1 -- 172.21.15.189:0/2451381969 shutdown_connections 2024-04-06T12:30:02.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.228+0000 7f7321df7700 1 -- 172.21.15.189:0/2451381969 wait complete. 2024-04-06T12:30:02.228 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 46 2024-04-06T12:30:02.239 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:30:02.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.365+0000 7effd4146700 1 Processor -- start 2024-04-06T12:30:02.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.365+0000 7effd4146700 1 -- start start 2024-04-06T12:30:02.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.366+0000 7effd4146700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7effcc15d590 0x7effcc0b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:02.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.366+0000 7effd4146700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7effcc0b4ed0 0x7effcc0b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:02.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.366+0000 7effd4146700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effcc0b5cc0 con 0x7effcc0b6580 2024-04-06T12:30:02.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.366+0000 7effd4146700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effcc0b58c0 con 0x7effcc15d590 2024-04-06T12:30:02.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.366+0000 7effd4146700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effcc0d1500 con 0x7effcc0b4ed0 2024-04-06T12:30:02.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.366+0000 7effd1ee2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7effcc15d590 0x7effcc0b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:02.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.366+0000 7effd1ee2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7effcc15d590 0x7effcc0b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58144/0 (socket says 172.21.15.189:58144) 2024-04-06T12:30:02.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.366+0000 7effd1ee2700 1 -- 172.21.15.189:0/1139541706 learned_addr learned my addr 172.21.15.189:0/1139541706 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:02.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.366+0000 7effd16e1700 1 --2- 172.21.15.189:0/1139541706 >> v2:172.21.15.192:3300/0 conn(0x7effcc0b4ed0 0x7effcc0b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:02.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.366+0000 7effd1ee2700 1 -- 172.21.15.189:0/1139541706 >> v2:172.21.15.192:3300/0 conn(0x7effcc0b4ed0 msgr2=0x7effcc0b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:02.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.366+0000 7effd1ee2700 1 --2- 172.21.15.189:0/1139541706 >> v2:172.21.15.192:3300/0 conn(0x7effcc0b4ed0 0x7effcc0b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.366+0000 7effd1ee2700 1 -- 172.21.15.189:0/1139541706 >> v1:172.21.15.192:6789/0 conn(0x7effcc0b6580 legacy=0x7effcc0b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:02.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.366+0000 7effd1ee2700 1 -- 172.21.15.189:0/1139541706 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effcc0d17e0 con 0x7effcc15d590 2024-04-06T12:30:02.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.367+0000 7effd1ee2700 1 --2- 172.21.15.189:0/1139541706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effcc15d590 0x7effcc0b8d70 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7effcc0b6330 tx=0x7effc8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c56147255589daa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:02.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.367+0000 7effd0ee0700 1 -- 172.21.15.189:0/1139541706 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8011830 con 0x7effcc15d590 2024-04-06T12:30:02.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.367+0000 7effd0ee0700 1 -- 172.21.15.189:0/1139541706 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc8011990 con 0x7effcc15d590 2024-04-06T12:30:02.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.367+0000 7effd0ee0700 1 -- 172.21.15.189:0/1139541706 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8011d20 con 0x7effcc15d590 2024-04-06T12:30:02.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.367+0000 7effd4146700 1 -- 172.21.15.189:0/1139541706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effcc15d590 msgr2=0x7effcc0b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:02.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.367+0000 7effd4146700 1 --2- 172.21.15.189:0/1139541706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effcc15d590 0x7effcc0b8d70 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7effcc0b6330 tx=0x7effc8009bd0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.367+0000 7effd4146700 1 -- 172.21.15.189:0/1139541706 shutdown_connections 2024-04-06T12:30:02.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.367+0000 7effd4146700 1 --2- 172.21.15.189:0/1139541706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effcc15d590 0x7effcc0b8d70 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.367+0000 7effd4146700 1 --2- 172.21.15.189:0/1139541706 >> v2:172.21.15.192:3300/0 conn(0x7effcc0b4ed0 0x7effcc0b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.367+0000 7effd4146700 1 -- 172.21.15.189:0/1139541706 >> 172.21.15.189:0/1139541706 conn(0x7effcc06bbc0 msgr2=0x7effcc06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:02.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.368+0000 7effd4146700 1 -- 172.21.15.189:0/1139541706 shutdown_connections 2024-04-06T12:30:02.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.368+0000 7effd4146700 1 -- 172.21.15.189:0/1139541706 wait complete. 2024-04-06T12:30:02.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.369+0000 7effd4146700 1 Processor -- start 2024-04-06T12:30:02.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.369+0000 7effd4146700 1 -- start start 2024-04-06T12:30:02.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.370+0000 7effd4146700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effcc0b4ed0 0x7effcc15d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:02.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.370+0000 7effd4146700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effcc0b6580 0x7effcc14f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:02.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.370+0000 7effd4146700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effcc15d590 0x7effcc0cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:02.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.370+0000 7effd4146700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effcc15e8c0 con 0x7effcc0b4ed0 2024-04-06T12:30:02.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.370+0000 7effd4146700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effcc15c5e0 con 0x7effcc15d590 2024-04-06T12:30:02.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.370+0000 7effd4146700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effcc15c8c0 con 0x7effcc0b6580 2024-04-06T12:30:02.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.370+0000 7effd1ee2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effcc0b4ed0 0x7effcc15d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.370+0000 7effd1ee2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effcc0b4ed0 0x7effcc15d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58160/0 (socket says 172.21.15.189:58160) 2024-04-06T12:30:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.370+0000 7effd1ee2700 1 -- 172.21.15.189:0/2497156304 learned_addr learned my addr 172.21.15.189:0/2497156304 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.370+0000 7effd16e1700 1 --2- 172.21.15.189:0/2497156304 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effcc0b6580 0x7effcc14f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.370+0000 7effd26e3700 1 --2- 172.21.15.189:0/2497156304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effcc15d590 0x7effcc0cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.370+0000 7effd1ee2700 1 -- 172.21.15.189:0/2497156304 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effcc0b6580 msgr2=0x7effcc14f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.370+0000 7effd1ee2700 1 --2- 172.21.15.189:0/2497156304 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effcc0b6580 0x7effcc14f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.370+0000 7effd1ee2700 1 -- 172.21.15.189:0/2497156304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effcc15d590 msgr2=0x7effcc0cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.370+0000 7effd1ee2700 1 --2- 172.21.15.189:0/2497156304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effcc15d590 0x7effcc0cbec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.370+0000 7effd1ee2700 1 -- 172.21.15.189:0/2497156304 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc8009550 con 0x7effcc0b4ed0 2024-04-06T12:30:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.371+0000 7effd1ee2700 1 --2- 172.21.15.189:0/2497156304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effcc0b4ed0 0x7effcc15d9e0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7effc8009c10 tx=0x7effc8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.371+0000 7effc27fc700 1 -- 172.21.15.189:0/2497156304 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8004510 con 0x7effcc0b4ed0 2024-04-06T12:30:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.371+0000 7effc27fc700 1 -- 172.21.15.189:0/2497156304 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc801e070 con 0x7effcc0b4ed0 2024-04-06T12:30:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.371+0000 7effd4146700 1 -- 172.21.15.189:0/2497156304 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effcc15cba0 con 0x7effcc0b4ed0 2024-04-06T12:30:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.371+0000 7effc27fc700 1 -- 172.21.15.189:0/2497156304 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc80038e0 con 0x7effcc0b4ed0 2024-04-06T12:30:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.371+0000 7effd4146700 1 -- 172.21.15.189:0/2497156304 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effcc14f860 con 0x7effcc0b4ed0 2024-04-06T12:30:02.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.372+0000 7effd4146700 1 -- 172.21.15.189:0/2497156304 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effcc074990 con 0x7effcc0b4ed0 2024-04-06T12:30:02.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.372+0000 7effc27fc700 1 -- 172.21.15.189:0/2497156304 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 59) v1 ==== 89361+0+0 (secure 0 0 0) 0x7effc8003a40 con 0x7effcc0b4ed0 2024-04-06T12:30:02.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.373+0000 7effc27fc700 1 --2- 172.21.15.189:0/2497156304 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7effb006aed0 0x7effb006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:02.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.373+0000 7effd16e1700 1 -- 172.21.15.189:0/2497156304 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7effb006aed0 msgr2=0x7effb006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:02.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.373+0000 7effc27fc700 1 -- 172.21.15.189:0/2497156304 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6489+0+0 (secure 0 0 0) 0x7effc808a710 con 0x7effcc0b4ed0 2024-04-06T12:30:02.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.373+0000 7effd16e1700 1 --2- 172.21.15.189:0/2497156304 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7effb006aed0 0x7effb006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:30:02.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.377+0000 7effc27fc700 1 -- 172.21.15.189:0/2497156304 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7effc80553e0 con 0x7effcc0b4ed0 2024-04-06T12:30:02.392 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:02.395+0000 7fdd05b24200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:30:02.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.574+0000 7effd16e1700 1 -- 172.21.15.189:0/2497156304 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7effb006aed0 msgr2=0x7effb006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:02.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.574+0000 7effd16e1700 1 --2- 172.21.15.189:0/2497156304 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7effb006aed0 0x7effb006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:30:02.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.581+0000 7effd4146700 1 -- 172.21.15.189:0/2497156304 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effcc0d2820 con 0x7effcc0b4ed0 2024-04-06T12:30:02.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.582+0000 7effc27fc700 1 -- 172.21.15.189:0/2497156304 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+17000 (secure 0 0 0) 0x7effc8058a00 con 0x7effcc0b4ed0 2024-04-06T12:30:02.581 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:30:02.581 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":115,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:29:55.659401+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:28:38.415021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-06T12:28:39.629378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.8","mappings":[{"from":1,"to":7}]},{"pgid":"14.1b","mappings":[{"from":1,"to":2}]},{"pgid":"14.31","mappings":[{"from":1,"to":7}]},{"pgid":"14.36","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/1207971310":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/3711853723":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/1381299583":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/1611293259":"2024-04-07T12:28:54.176092+0000","172.21.15.189:6833/47128":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/1131169404":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/2194712360":"2024-04-07T12:28:54.176092+0000","172.21.15.189:6832/47128":"2024-04-07T12:28:54.176092+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:30:02.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.584+0000 7effd4146700 1 -- 172.21.15.189:0/2497156304 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7effb006aed0 msgr2=0x7effb006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:30:02.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.584+0000 7effd4146700 1 --2- 172.21.15.189:0/2497156304 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7effb006aed0 0x7effb006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.584+0000 7effd4146700 1 -- 172.21.15.189:0/2497156304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effcc0b4ed0 msgr2=0x7effcc15d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:02.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.584+0000 7effd4146700 1 --2- 172.21.15.189:0/2497156304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effcc0b4ed0 0x7effcc15d9e0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7effc8009c10 tx=0x7effc8004750 comp rx=0 tx=0).stop 2024-04-06T12:30:02.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.585+0000 7effd4146700 1 -- 172.21.15.189:0/2497156304 shutdown_connections 2024-04-06T12:30:02.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.585+0000 7effd4146700 1 --2- 172.21.15.189:0/2497156304 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7effb006aed0 0x7effb006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.585+0000 7effd4146700 1 --2- 172.21.15.189:0/2497156304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effcc15d590 0x7effcc0cbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.585+0000 7effd4146700 1 --2- 172.21.15.189:0/2497156304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effcc0b4ed0 0x7effcc15d9e0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.585+0000 7effd4146700 1 --2- 172.21.15.189:0/2497156304 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effcc0b6580 0x7effcc14f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.585+0000 7effd4146700 1 -- 172.21.15.189:0/2497156304 >> 172.21.15.189:0/2497156304 conn(0x7effcc06bbc0 msgr2=0x7effcc0b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:02.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.585+0000 7effd4146700 1 -- 172.21.15.189:0/2497156304 shutdown_connections 2024-04-06T12:30:02.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.585+0000 7effd4146700 1 -- 172.21.15.189:0/2497156304 wait complete. 2024-04-06T12:30:02.596 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-06T12:30:02.596 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:30:02.630 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:02.633+0000 7fdd05b24200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:30:02.696 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:02.698+0000 7f4a4ce9c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:30:02.706 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:02.709+0000 7fdd05b24200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:30:02.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.730+0000 7f10464c5700 1 Processor -- start 2024-04-06T12:30:02.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.730+0000 7f10464c5700 1 -- start start 2024-04-06T12:30:02.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.731+0000 7f10464c5700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f104015b130 0x7f10400b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:02.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.731+0000 7f10464c5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f104015b510 0x7f10400b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:02.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.731+0000 7f10464c5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10400b1fb0 con 0x7f1040071de0 2024-04-06T12:30:02.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.731+0000 7f10464c5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10400b2200 con 0x7f104015b510 2024-04-06T12:30:02.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.731+0000 7f10464c5700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10400b24e0 con 0x7f104015b130 2024-04-06T12:30:02.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.731+0000 7f103f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f104015b510 0x7f10400b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:02.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.731+0000 7f103f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f104015b510 0x7f10400b39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58172/0 (socket says 172.21.15.189:58172) 2024-04-06T12:30:02.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.731+0000 7f103f7fe700 1 -- 172.21.15.189:0/3473565533 learned_addr learned my addr 172.21.15.189:0/3473565533 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:02.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.731+0000 7f103ffff700 1 --2- 172.21.15.189:0/3473565533 >> v2:172.21.15.192:3300/0 conn(0x7f104015b130 0x7f10400b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:02.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.732+0000 7f103f7fe700 1 -- 172.21.15.189:0/3473565533 >> v2:172.21.15.192:3300/0 conn(0x7f104015b130 msgr2=0x7f10400b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:02.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.732+0000 7f103f7fe700 1 --2- 172.21.15.189:0/3473565533 >> v2:172.21.15.192:3300/0 conn(0x7f104015b130 0x7f10400b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.732+0000 7f103f7fe700 1 -- 172.21.15.189:0/3473565533 >> v1:172.21.15.192:6789/0 conn(0x7f1040071de0 legacy=0x7f1040072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:02.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.732+0000 7f103f7fe700 1 -- 172.21.15.189:0/3473565533 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10400b27c0 con 0x7f104015b510 2024-04-06T12:30:02.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.732+0000 7f103effd700 1 -- 172.21.15.189:0/3473565533 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3517946341 0 0) 0x7f10400b1fb0 con 0x7f1040071de0 2024-04-06T12:30:02.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.732+0000 7f103f7fe700 1 --2- 172.21.15.189:0/3473565533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f104015b510 0x7f10400b39b0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f10400681c0 tx=0x7f1034009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e8234aed29b254f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:02.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.732+0000 7f103effd700 1 -- 172.21.15.189:0/3473565533 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1034011830 con 0x7f104015b510 2024-04-06T12:30:02.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.732+0000 7f103effd700 1 -- 172.21.15.189:0/3473565533 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1034011990 con 0x7f104015b510 2024-04-06T12:30:02.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.732+0000 7f103effd700 1 -- 172.21.15.189:0/3473565533 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1034011d20 con 0x7f104015b510 2024-04-06T12:30:02.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.733+0000 7f10464c5700 1 -- 172.21.15.189:0/3473565533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f104015b510 msgr2=0x7f10400b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:02.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.733+0000 7f10464c5700 1 --2- 172.21.15.189:0/3473565533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f104015b510 0x7f10400b39b0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f10400681c0 tx=0x7f1034009bd0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.733+0000 7f10464c5700 1 -- 172.21.15.189:0/3473565533 shutdown_connections 2024-04-06T12:30:02.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.733+0000 7f10464c5700 1 --2- 172.21.15.189:0/3473565533 >> v2:172.21.15.192:3300/0 conn(0x7f104015b130 0x7f10400b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.733+0000 7f10464c5700 1 --2- 172.21.15.189:0/3473565533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f104015b510 0x7f10400b39b0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.733+0000 7f10464c5700 1 -- 172.21.15.189:0/3473565533 >> 172.21.15.189:0/3473565533 conn(0x7f104006a730 msgr2=0x7f104006ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:02.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.733+0000 7f10464c5700 1 -- 172.21.15.189:0/3473565533 shutdown_connections 2024-04-06T12:30:02.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.733+0000 7f10464c5700 1 -- 172.21.15.189:0/3473565533 wait complete. 2024-04-06T12:30:02.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.734+0000 7f10464c5700 1 Processor -- start 2024-04-06T12:30:02.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.734+0000 7f10464c5700 1 -- start start 2024-04-06T12:30:02.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.735+0000 7f10464c5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1040071de0 0x7f104013eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:02.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.735+0000 7f10464c5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f104015b130 0x7f10400cd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:02.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.735+0000 7f10464c5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f104015b510 0x7f10400cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:02.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.735+0000 7f10464c5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10400cf770 con 0x7f104015b510 2024-04-06T12:30:02.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.735+0000 7f10464c5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10400cca20 con 0x7f1040071de0 2024-04-06T12:30:02.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.735+0000 7f10464c5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10400ccd00 con 0x7f104015b130 2024-04-06T12:30:02.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.735+0000 7f103ffff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1040071de0 0x7f104013eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:02.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.735+0000 7f1044a62700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f104015b510 0x7f10400cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:02.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.735+0000 7f103ffff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1040071de0 0x7f104013eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:57444/0 (socket says 172.21.15.189:57444) 2024-04-06T12:30:02.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.735+0000 7f103ffff700 1 -- 172.21.15.189:0/1418978729 learned_addr learned my addr 172.21.15.189:0/1418978729 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:02.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.735+0000 7f103f7fe700 1 --2- 172.21.15.189:0/1418978729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f104015b130 0x7f10400cd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:02.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.735+0000 7f1044a62700 1 -- 172.21.15.189:0/1418978729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f104015b130 msgr2=0x7f10400cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:02.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.735+0000 7f1044a62700 1 --2- 172.21.15.189:0/1418978729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f104015b130 0x7f10400cd7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.735+0000 7f1044a62700 1 -- 172.21.15.189:0/1418978729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1040071de0 msgr2=0x7f104013eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:02.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.735+0000 7f1044a62700 1 --2- 172.21.15.189:0/1418978729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1040071de0 0x7f104013eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.735+0000 7f1044a62700 1 -- 172.21.15.189:0/1418978729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1034009550 con 0x7f104015b510 2024-04-06T12:30:02.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.735+0000 7f103ffff700 1 --2- 172.21.15.189:0/1418978729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1040071de0 0x7f104013eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:30:02.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.735+0000 7f103f7fe700 1 --2- 172.21.15.189:0/1418978729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f104015b130 0x7f10400cd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:30:02.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.736+0000 7f1044a62700 1 --2- 172.21.15.189:0/1418978729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f104015b510 0x7f10400cecf0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f103800c7c0 tx=0x7f103800cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:02.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.736+0000 7f103cff9700 1 -- 172.21.15.189:0/1418978729 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038003e30 con 0x7f104015b510 2024-04-06T12:30:02.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.736+0000 7f103cff9700 1 -- 172.21.15.189:0/1418978729 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1038005070 con 0x7f104015b510 2024-04-06T12:30:02.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.736+0000 7f10464c5700 1 -- 172.21.15.189:0/1418978729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10400ccfe0 con 0x7f104015b510 2024-04-06T12:30:02.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.736+0000 7f103cff9700 1 -- 172.21.15.189:0/1418978729 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038005420 con 0x7f104015b510 2024-04-06T12:30:02.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.736+0000 7f10464c5700 1 -- 172.21.15.189:0/1418978729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10401f3f10 con 0x7f104015b510 2024-04-06T12:30:02.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.737+0000 7f10464c5700 1 -- 172.21.15.189:0/1418978729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10400cd500 con 0x7f104015b510 2024-04-06T12:30:02.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.737+0000 7f103cff9700 1 -- 172.21.15.189:0/1418978729 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 59) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f1038005580 con 0x7f104015b510 2024-04-06T12:30:02.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.738+0000 7f103cff9700 1 --2- 172.21.15.189:0/1418978729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f102c06aed0 0x7f102c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:02.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.738+0000 7f103ffff700 1 -- 172.21.15.189:0/1418978729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f102c06aed0 msgr2=0x7f102c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:02.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.738+0000 7f103cff9700 1 -- 172.21.15.189:0/1418978729 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f1038089010 con 0x7f104015b510 2024-04-06T12:30:02.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.738+0000 7f103ffff700 1 --2- 172.21.15.189:0/1418978729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f102c06aed0 0x7f102c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:30:02.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.742+0000 7f103cff9700 1 -- 172.21.15.189:0/1418978729 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1038053b80 con 0x7f104015b510 2024-04-06T12:30:02.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.939+0000 7f103ffff700 1 -- 172.21.15.189:0/1418978729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f102c06aed0 msgr2=0x7f102c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:02.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.939+0000 7f103ffff700 1 --2- 172.21.15.189:0/1418978729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f102c06aed0 0x7f102c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:30:02.953 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:02.955+0000 7f4a4ce9c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:30:02.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.991+0000 7f10464c5700 1 -- 172.21.15.189:0/1418978729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1040073270 con 0x7f104015b510 2024-04-06T12:30:02.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.992+0000 7f103cff9700 1 -- 172.21.15.189:0/1418978729 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f1038017020 con 0x7f104015b510 2024-04-06T12:30:02.990 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:30:02.990 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":46,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6389,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/17345288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":17345288},{"type":"v1","addr":"172.21.15.192:6835","nonce":17345288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":7078,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/440618958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":440618958},{"type":"v1","addr":"172.21.15.192:6834","nonce":440618958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":7569,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3391245238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3391245238},{"type":"v1","addr":"172.21.15.189:6836","nonce":3391245238}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:28:40.659968+0000","modified":"2024-04-06T12:28:43.668129+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7572":{"gid":7572,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/2225868923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2225868923},{"type":"v1","addr":"172.21.15.189:6837","nonce":2225868923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-04-06T12:30:02.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.994+0000 7f10464c5700 1 -- 172.21.15.189:0/1418978729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f102c06aed0 msgr2=0x7f102c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:30:02.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.994+0000 7f10464c5700 1 --2- 172.21.15.189:0/1418978729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f102c06aed0 0x7f102c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.994+0000 7f10464c5700 1 -- 172.21.15.189:0/1418978729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f104015b510 msgr2=0x7f10400cecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:02.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.994+0000 7f10464c5700 1 --2- 172.21.15.189:0/1418978729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f104015b510 0x7f10400cecf0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f103800c7c0 tx=0x7f103800cad0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.994+0000 7f10464c5700 1 -- 172.21.15.189:0/1418978729 shutdown_connections 2024-04-06T12:30:02.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.994+0000 7f10464c5700 1 --2- 172.21.15.189:0/1418978729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f104015b510 0x7f10400cecf0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.994+0000 7f10464c5700 1 --2- 172.21.15.189:0/1418978729 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f102c06aed0 0x7f102c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.994+0000 7f10464c5700 1 --2- 172.21.15.189:0/1418978729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1040071de0 0x7f104013eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.994+0000 7f10464c5700 1 --2- 172.21.15.189:0/1418978729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f104015b130 0x7f10400cd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:02.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.994+0000 7f10464c5700 1 -- 172.21.15.189:0/1418978729 >> 172.21.15.189:0/1418978729 conn(0x7f104006a730 msgr2=0x7f10400adb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:02.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.994+0000 7f10464c5700 1 -- 172.21.15.189:0/1418978729 shutdown_connections 2024-04-06T12:30:02.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:02.994+0000 7f10464c5700 1 -- 172.21.15.189:0/1418978729 wait complete. 2024-04-06T12:30:02.993 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 46 2024-04-06T12:30:03.005 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:30:03.036 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:03.038+0000 7f4a4ce9c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:30:03.080 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:03.083+0000 7fdd05b24200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:30:03.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.132+0000 7f5598e86700 1 Processor -- start 2024-04-06T12:30:03.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.132+0000 7f5598e86700 1 -- start start 2024-04-06T12:30:03.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.133+0000 7f5598e86700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f559415d590 0x7f55940b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:03.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.133+0000 7f5598e86700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f55940b4ed0 0x7f55940b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:03.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.133+0000 7f5598e86700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f55940b6580 0x7f55940b69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:03.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.133+0000 7f5598e86700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55940b58c0 con 0x7f55940b4ed0 2024-04-06T12:30:03.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.133+0000 7f5598e86700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55940b5cc0 con 0x7f559415d590 2024-04-06T12:30:03.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.133+0000 7f5598e86700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55940d1500 con 0x7f55940b6580 2024-04-06T12:30:03.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.133+0000 7f5591d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f55940b4ed0 0x7f55940b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:03.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.133+0000 7f5591d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f55940b4ed0 0x7f55940b5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58188/0 (socket says 172.21.15.189:58188) 2024-04-06T12:30:03.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.133+0000 7f5591d9b700 1 -- 172.21.15.189:0/2125756590 learned_addr learned my addr 172.21.15.189:0/2125756590 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:03.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.133+0000 7f5592d9d700 1 --2- 172.21.15.189:0/2125756590 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f55940b6580 0x7f55940b69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:03.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.133+0000 7f559259c700 1 --2- 172.21.15.189:0/2125756590 >> v2:172.21.15.192:3300/0 conn(0x7f559415d590 0x7f55940b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:03.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.133+0000 7f5591d9b700 1 -- 172.21.15.189:0/2125756590 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f55940b6580 msgr2=0x7f55940b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:03.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.133+0000 7f5591d9b700 1 --2- 172.21.15.189:0/2125756590 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f55940b6580 0x7f55940b69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.133+0000 7f5591d9b700 1 -- 172.21.15.189:0/2125756590 >> v2:172.21.15.192:3300/0 conn(0x7f559415d590 msgr2=0x7f55940b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:03.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.133+0000 7f5591d9b700 1 --2- 172.21.15.189:0/2125756590 >> v2:172.21.15.192:3300/0 conn(0x7f559415d590 0x7f55940b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.133+0000 7f5591d9b700 1 -- 172.21.15.189:0/2125756590 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55940d17e0 con 0x7f55940b4ed0 2024-04-06T12:30:03.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.134+0000 7f5591d9b700 1 --2- 172.21.15.189:0/2125756590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55940b4ed0 0x7f55940b5380 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f55940b5fe0 tx=0x7f5588009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4938f5ba50368778 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:03.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.134+0000 7f559159a700 1 -- 172.21.15.189:0/2125756590 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558800b870 con 0x7f55940b4ed0 2024-04-06T12:30:03.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.134+0000 7f559159a700 1 -- 172.21.15.189:0/2125756590 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558800b9d0 con 0x7f55940b4ed0 2024-04-06T12:30:03.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.134+0000 7f559159a700 1 -- 172.21.15.189:0/2125756590 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558800bd60 con 0x7f55940b4ed0 2024-04-06T12:30:03.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.134+0000 7f5598e86700 1 -- 172.21.15.189:0/2125756590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55940b4ed0 msgr2=0x7f55940b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:03.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.134+0000 7f5598e86700 1 --2- 172.21.15.189:0/2125756590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55940b4ed0 0x7f55940b5380 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f55940b5fe0 tx=0x7f5588009bd0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.134+0000 7f5598e86700 1 -- 172.21.15.189:0/2125756590 shutdown_connections 2024-04-06T12:30:03.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.134+0000 7f5598e86700 1 --2- 172.21.15.189:0/2125756590 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f55940b6580 0x7f55940b69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.134+0000 7f5598e86700 1 --2- 172.21.15.189:0/2125756590 >> v2:172.21.15.192:3300/0 conn(0x7f559415d590 0x7f55940b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.134+0000 7f5598e86700 1 --2- 172.21.15.189:0/2125756590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55940b4ed0 0x7f55940b5380 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.134+0000 7f5598e86700 1 -- 172.21.15.189:0/2125756590 >> 172.21.15.189:0/2125756590 conn(0x7f559406bbc0 msgr2=0x7f559406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:03.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.135+0000 7f5598e86700 1 -- 172.21.15.189:0/2125756590 shutdown_connections 2024-04-06T12:30:03.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.135+0000 7f5598e86700 1 -- 172.21.15.189:0/2125756590 wait complete. 2024-04-06T12:30:03.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.136+0000 7f5598e86700 1 Processor -- start 2024-04-06T12:30:03.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.136+0000 7f5598e86700 1 -- start start 2024-04-06T12:30:03.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.137+0000 7f5598e86700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55940b4ed0 0x7f5594156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:03.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.137+0000 7f5598e86700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55940b6580 0x7f5594158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:03.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.137+0000 7f5598e86700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f559415d590 0x7f55940caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.137+0000 7f5598e86700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5594158610 con 0x7f55940b4ed0 2024-04-06T12:30:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.137+0000 7f5598e86700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55940ca520 con 0x7f55940b6580 2024-04-06T12:30:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.137+0000 7f5598e86700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5594159530 con 0x7f559415d590 2024-04-06T12:30:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.137+0000 7f559259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55940b4ed0 0x7f5594156da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.137+0000 7f559259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55940b4ed0 0x7f5594156da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58202/0 (socket says 172.21.15.189:58202) 2024-04-06T12:30:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.137+0000 7f559259c700 1 -- 172.21.15.189:0/1254032753 learned_addr learned my addr 172.21.15.189:0/1254032753 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.137+0000 7f5592d9d700 1 --2- 172.21.15.189:0/1254032753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f559415d590 0x7f55940caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.137+0000 7f5591d9b700 1 --2- 172.21.15.189:0/1254032753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55940b6580 0x7f5594158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.138+0000 7f559259c700 1 -- 172.21.15.189:0/1254032753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f559415d590 msgr2=0x7f55940caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.138+0000 7f559259c700 1 --2- 172.21.15.189:0/1254032753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f559415d590 0x7f55940caeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.138+0000 7f559259c700 1 -- 172.21.15.189:0/1254032753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55940b6580 msgr2=0x7f5594158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.138+0000 7f559259c700 1 --2- 172.21.15.189:0/1254032753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55940b6580 0x7f5594158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.138+0000 7f559259c700 1 -- 172.21.15.189:0/1254032753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5588009550 con 0x7f55940b4ed0 2024-04-06T12:30:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.138+0000 7f559259c700 1 --2- 172.21.15.189:0/1254032753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55940b4ed0 0x7f5594156da0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f558400d800 tx=0x7f558400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:03.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.138+0000 7f5582ffd700 1 -- 172.21.15.189:0/1254032753 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55840125f0 con 0x7f55940b4ed0 2024-04-06T12:30:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.138+0000 7f5582ffd700 1 -- 172.21.15.189:0/1254032753 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5584012750 con 0x7f55940b4ed0 2024-04-06T12:30:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.138+0000 7f5598e86700 1 -- 172.21.15.189:0/1254032753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55940cb3f0 con 0x7f55940b4ed0 2024-04-06T12:30:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.138+0000 7f5582ffd700 1 -- 172.21.15.189:0/1254032753 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5584012b00 con 0x7f55940b4ed0 2024-04-06T12:30:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.138+0000 7f5598e86700 1 -- 172.21.15.189:0/1254032753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55940caa50 con 0x7f55940b4ed0 2024-04-06T12:30:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.139+0000 7f5598e86700 1 -- 172.21.15.189:0/1254032753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55940cac40 con 0x7f55940b4ed0 2024-04-06T12:30:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.140+0000 7f5582ffd700 1 -- 172.21.15.189:0/1254032753 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 59) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f5584012c60 con 0x7f55940b4ed0 2024-04-06T12:30:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.140+0000 7f5582ffd700 1 --2- 172.21.15.189:0/1254032753 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f557406aed0 0x7f557406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.140+0000 7f5591d9b700 1 -- 172.21.15.189:0/1254032753 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f557406aed0 msgr2=0x7f557406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.140+0000 7f5582ffd700 1 -- 172.21.15.189:0/1254032753 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f5584089a10 con 0x7f55940b4ed0 2024-04-06T12:30:03.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.140+0000 7f5591d9b700 1 --2- 172.21.15.189:0/1254032753 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f557406aed0 0x7f557406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:30:03.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.144+0000 7f5582ffd700 1 -- 172.21.15.189:0/1254032753 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f55840546c0 con 0x7f55940b4ed0 2024-04-06T12:30:03.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.341+0000 7f5591d9b700 1 -- 172.21.15.189:0/1254032753 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f557406aed0 msgr2=0x7f557406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:03.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.341+0000 7f5591d9b700 1 --2- 172.21.15.189:0/1254032753 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f557406aed0 0x7f557406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:30:03.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.394+0000 7f5598e86700 1 -- 172.21.15.189:0/1254032753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5594074990 con 0x7f55940b4ed0 2024-04-06T12:30:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.395+0000 7f5582ffd700 1 -- 172.21.15.189:0/1254032753 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f5584057ce0 con 0x7f55940b4ed0 2024-04-06T12:30:03.393 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:30:03.393 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":46,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6389,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/17345288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":17345288},{"type":"v1","addr":"172.21.15.192:6835","nonce":17345288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":7078,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/440618958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":440618958},{"type":"v1","addr":"172.21.15.192:6834","nonce":440618958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":7569,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3391245238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3391245238},{"type":"v1","addr":"172.21.15.189:6836","nonce":3391245238}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:28:40.659968+0000","modified":"2024-04-06T12:28:43.668129+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7572":{"gid":7572,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/2225868923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2225868923},{"type":"v1","addr":"172.21.15.189:6837","nonce":2225868923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-04-06T12:30:03.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.397+0000 7f5598e86700 1 -- 172.21.15.189:0/1254032753 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f557406aed0 msgr2=0x7f557406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:30:03.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.397+0000 7f5598e86700 1 --2- 172.21.15.189:0/1254032753 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f557406aed0 0x7f557406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.397+0000 7f5598e86700 1 -- 172.21.15.189:0/1254032753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55940b4ed0 msgr2=0x7f5594156da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:03.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.397+0000 7f5598e86700 1 --2- 172.21.15.189:0/1254032753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55940b4ed0 0x7f5594156da0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f558400d800 tx=0x7f558400db10 comp rx=0 tx=0).stop 2024-04-06T12:30:03.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.398+0000 7f5598e86700 1 -- 172.21.15.189:0/1254032753 shutdown_connections 2024-04-06T12:30:03.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.398+0000 7f5598e86700 1 --2- 172.21.15.189:0/1254032753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f559415d590 0x7f55940caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.398+0000 7f5598e86700 1 --2- 172.21.15.189:0/1254032753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55940b4ed0 0x7f5594156da0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.398+0000 7f5598e86700 1 --2- 172.21.15.189:0/1254032753 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f557406aed0 0x7f557406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.398+0000 7f5598e86700 1 --2- 172.21.15.189:0/1254032753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55940b6580 0x7f5594158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.398+0000 7f5598e86700 1 -- 172.21.15.189:0/1254032753 >> 172.21.15.189:0/1254032753 conn(0x7f559406bbc0 msgr2=0x7f55940ce450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.398+0000 7f5598e86700 1 -- 172.21.15.189:0/1254032753 shutdown_connections 2024-04-06T12:30:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.398+0000 7f5598e86700 1 -- 172.21.15.189:0/1254032753 wait complete. 2024-04-06T12:30:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 46 2024-04-06T12:30:03.409 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:30:03.431 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:03.432+0000 7f4a4ce9c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:30:03.513 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:03.515+0000 7f49c55b5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:03.513 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:03.515+0000 7f49c55b5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:03.513 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:03.515+0000 7f49c85bb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:03.513 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:03.515+0000 7f49c85bb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:03.513 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:03.515+0000 7f49c85bb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:03.514 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:03.515+0000 7f49c85bb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:03.514 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:03.515+0000 7f49c85bb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:03.514 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:03.515+0000 7f49c55b5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:03.514 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:03.515+0000 7f49c55b5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:03.514 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:03.515+0000 7f49c55b5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:03.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.567+0000 7f3b64aee700 1 Processor -- start 2024-04-06T12:30:03.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.567+0000 7f3b64aee700 1 -- start start 2024-04-06T12:30:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.568+0000 7f3b64aee700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3b6015d7b0 0x7f3b600b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.568+0000 7f3b64aee700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3b6006a730 0x7f3b6006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.568+0000 7f3b64aee700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b6015ddf0 0x7f3b6015e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.568+0000 7f3b64aee700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6006b0f0 con 0x7f3b6006a730 2024-04-06T12:30:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.568+0000 7f3b64aee700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b600b3990 con 0x7f3b6015d7b0 2024-04-06T12:30:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.568+0000 7f3b64aee700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b600b3060 con 0x7f3b6015ddf0 2024-04-06T12:30:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.568+0000 7f3b5dd9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3b6006a730 0x7f3b6006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.568+0000 7f3b5dd9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3b6006a730 0x7f3b6006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58296/0 (socket says 172.21.15.189:58296) 2024-04-06T12:30:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.568+0000 7f3b5dd9b700 1 -- 172.21.15.189:0/331035529 learned_addr learned my addr 172.21.15.189:0/331035529 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.569+0000 7f3b5ed9d700 1 --2- 172.21.15.189:0/331035529 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b6015ddf0 0x7f3b6015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.569+0000 7f3b5e59c700 1 --2- 172.21.15.189:0/331035529 >> v2:172.21.15.192:3300/0 conn(0x7f3b6015d7b0 0x7f3b600b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.569+0000 7f3b5dd9b700 1 -- 172.21.15.189:0/331035529 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b6015ddf0 msgr2=0x7f3b6015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.569+0000 7f3b5dd9b700 1 --2- 172.21.15.189:0/331035529 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b6015ddf0 0x7f3b6015e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.569+0000 7f3b5dd9b700 1 -- 172.21.15.189:0/331035529 >> v2:172.21.15.192:3300/0 conn(0x7f3b6015d7b0 msgr2=0x7f3b600b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.569+0000 7f3b5dd9b700 1 --2- 172.21.15.189:0/331035529 >> v2:172.21.15.192:3300/0 conn(0x7f3b6015d7b0 0x7f3b600b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.569+0000 7f3b5dd9b700 1 -- 172.21.15.189:0/331035529 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b600b3340 con 0x7f3b6006a730 2024-04-06T12:30:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.569+0000 7f3b5dd9b700 1 --2- 172.21.15.189:0/331035529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b6006a730 0x7f3b6006abb0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f3b600b6200 tx=0x7f3b50009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ddc7dbfd640bcb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.569+0000 7f3b5d59a700 1 -- 172.21.15.189:0/331035529 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5000b870 con 0x7f3b6006a730 2024-04-06T12:30:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.569+0000 7f3b5d59a700 1 -- 172.21.15.189:0/331035529 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5000b9d0 con 0x7f3b6006a730 2024-04-06T12:30:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.569+0000 7f3b5d59a700 1 -- 172.21.15.189:0/331035529 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5000bd60 con 0x7f3b6006a730 2024-04-06T12:30:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.569+0000 7f3b64aee700 1 -- 172.21.15.189:0/331035529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b6006a730 msgr2=0x7f3b6006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.569+0000 7f3b64aee700 1 --2- 172.21.15.189:0/331035529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b6006a730 0x7f3b6006abb0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f3b600b6200 tx=0x7f3b50009bd0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.570+0000 7f3b64aee700 1 -- 172.21.15.189:0/331035529 shutdown_connections 2024-04-06T12:30:03.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.570+0000 7f3b64aee700 1 --2- 172.21.15.189:0/331035529 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b6015ddf0 0x7f3b6015e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.570+0000 7f3b64aee700 1 --2- 172.21.15.189:0/331035529 >> v2:172.21.15.192:3300/0 conn(0x7f3b6015d7b0 0x7f3b600b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.570+0000 7f3b64aee700 1 --2- 172.21.15.189:0/331035529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b6006a730 0x7f3b6006abb0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.570+0000 7f3b64aee700 1 -- 172.21.15.189:0/331035529 >> 172.21.15.189:0/331035529 conn(0x7f3b600d00b0 msgr2=0x7f3b600d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:03.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.570+0000 7f3b64aee700 1 -- 172.21.15.189:0/331035529 shutdown_connections 2024-04-06T12:30:03.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.570+0000 7f3b64aee700 1 -- 172.21.15.189:0/331035529 wait complete. 2024-04-06T12:30:03.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.571+0000 7f3b64aee700 1 Processor -- start 2024-04-06T12:30:03.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.571+0000 7f3b64aee700 1 -- start start 2024-04-06T12:30:03.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b64aee700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b6006a730 0x7f3b60150100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:03.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b64aee700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b6015d7b0 0x7f3b6015ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:03.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b64aee700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b6015ddf0 0x7f3b600cc930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:03.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b64aee700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6015b6c0 con 0x7f3b6015d7b0 2024-04-06T12:30:03.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b64aee700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6015b9a0 con 0x7f3b6006a730 2024-04-06T12:30:03.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b64aee700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6015bc80 con 0x7f3b6015ddf0 2024-04-06T12:30:03.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b5dd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b6015d7b0 0x7f3b6015ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:03.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b5e59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b6006a730 0x7f3b60150100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:03.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b5dd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b6015d7b0 0x7f3b6015ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58300/0 (socket says 172.21.15.189:58300) 2024-04-06T12:30:03.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b5dd9b700 1 -- 172.21.15.189:0/3752943215 learned_addr learned my addr 172.21.15.189:0/3752943215 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:03.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b5ed9d700 1 --2- 172.21.15.189:0/3752943215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b6015ddf0 0x7f3b600cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:03.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b5dd9b700 1 -- 172.21.15.189:0/3752943215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b6015ddf0 msgr2=0x7f3b600cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:03.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b5dd9b700 1 --2- 172.21.15.189:0/3752943215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b6015ddf0 0x7f3b600cc930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b5dd9b700 1 -- 172.21.15.189:0/3752943215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b6006a730 msgr2=0x7f3b60150100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:03.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b5dd9b700 1 --2- 172.21.15.189:0/3752943215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b6006a730 0x7f3b60150100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b5dd9b700 1 -- 172.21.15.189:0/3752943215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b50009550 con 0x7f3b6015d7b0 2024-04-06T12:30:03.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b5e59c700 1 --2- 172.21.15.189:0/3752943215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b6006a730 0x7f3b60150100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:30:03.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b5dd9b700 1 --2- 172.21.15.189:0/3752943215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b6015d7b0 0x7f3b6015ac70 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f3b6006c820 tx=0x7f3b50004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:03.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b4effd700 1 -- 172.21.15.189:0/3752943215 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b50004310 con 0x7f3b6015d7b0 2024-04-06T12:30:03.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b4effd700 1 -- 172.21.15.189:0/3752943215 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5001f070 con 0x7f3b6015d7b0 2024-04-06T12:30:03.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b4effd700 1 -- 172.21.15.189:0/3752943215 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b500036a0 con 0x7f3b6015d7b0 2024-04-06T12:30:03.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b64aee700 1 -- 172.21.15.189:0/3752943215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6015bf60 con 0x7f3b6015d7b0 2024-04-06T12:30:03.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.572+0000 7f3b64aee700 1 -- 172.21.15.189:0/3752943215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b600cf100 con 0x7f3b6015d7b0 2024-04-06T12:30:03.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.573+0000 7f3b64aee700 1 -- 172.21.15.189:0/3752943215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b6015c480 con 0x7f3b6015d7b0 2024-04-06T12:30:03.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.574+0000 7f3b4effd700 1 -- 172.21.15.189:0/3752943215 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 60) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f3b50021030 con 0x7f3b6015d7b0 2024-04-06T12:30:03.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.574+0000 7f3b4effd700 1 -- 172.21.15.189:0/3752943215 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6833+0+0 (secure 0 0 0) 0x7f3b5008a4f0 con 0x7f3b6015d7b0 2024-04-06T12:30:03.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.579+0000 7f3b4effd700 1 -- 172.21.15.189:0/3752943215 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3b6015c480 con 0x7f3b6015d7b0 2024-04-06T12:30:03.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.777+0000 7f3b64aee700 1 -- 172.21.15.189:0/3752943215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b6015c480 con 0x7f3b6015d7b0 2024-04-06T12:30:03.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.779+0000 7f3b4effd700 1 -- 172.21.15.189:0/3752943215 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+17504 (secure 0 0 0) 0x7f3b6015c480 con 0x7f3b6015d7b0 2024-04-06T12:30:03.777 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:30:03.777 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":116,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:30:03.447414+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:28:38.415021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-06T12:28:39.629378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.8","mappings":[{"from":1,"to":7}]},{"pgid":"14.1b","mappings":[{"from":1,"to":2}]},{"pgid":"14.31","mappings":[{"from":1,"to":7}]},{"pgid":"14.36","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/3777232029":"2024-04-07T12:30:03.447356+0000","172.21.15.189:0/1508079759":"2024-04-07T12:30:03.447356+0000","172.21.15.189:0/1558804048":"2024-04-07T12:30:03.447356+0000","172.21.15.189:0/1957136355":"2024-04-07T12:30:03.447356+0000","172.21.15.189:0/1710431875":"2024-04-07T12:30:03.447356+0000","172.21.15.189:0/2199841057":"2024-04-07T12:30:03.447356+0000","172.21.15.189:0/1207971310":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/3222083601":"2024-04-07T12:30:03.447356+0000","172.21.15.189:0/3711853723":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/1381299583":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/1611293259":"2024-04-07T12:28:54.176092+0000","172.21.15.189:6833/47128":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/3248281201":"2024-04-07T12:30:03.447356+0000","172.21.15.189:0/1131169404":"2024-04-07T12:28:54.176092+0000","172.21.15.189:0/2194712360":"2024-04-07T12:28:54.176092+0000","172.21.15.189:6832/47128":"2024-04-07T12:28:54.176092+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:30:03.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.781+0000 7f3b64aee700 1 -- 172.21.15.189:0/3752943215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b6015d7b0 msgr2=0x7f3b6015ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:03.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.781+0000 7f3b64aee700 1 --2- 172.21.15.189:0/3752943215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b6015d7b0 0x7f3b6015ac70 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f3b6006c820 tx=0x7f3b50004750 comp rx=0 tx=0).stop 2024-04-06T12:30:03.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.781+0000 7f3b64aee700 1 -- 172.21.15.189:0/3752943215 shutdown_connections 2024-04-06T12:30:03.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.781+0000 7f3b64aee700 1 --2- 172.21.15.189:0/3752943215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b6015ddf0 0x7f3b600cc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.781+0000 7f3b64aee700 1 --2- 172.21.15.189:0/3752943215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b6006a730 0x7f3b60150100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.781+0000 7f3b64aee700 1 --2- 172.21.15.189:0/3752943215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b6015d7b0 0x7f3b6015ac70 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.781+0000 7f3b64aee700 1 -- 172.21.15.189:0/3752943215 >> 172.21.15.189:0/3752943215 conn(0x7f3b600d00b0 msgr2=0x7f3b600ae210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:03.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.781+0000 7f3b64aee700 1 -- 172.21.15.189:0/3752943215 shutdown_connections 2024-04-06T12:30:03.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.781+0000 7f3b64aee700 1 -- 172.21.15.189:0/3752943215 wait complete. 2024-04-06T12:30:03.791 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-06T12:30:03.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.917+0000 7fefc714c700 1 Processor -- start 2024-04-06T12:30:03.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.917+0000 7fefc714c700 1 -- start start 2024-04-06T12:30:03.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.918+0000 7fefc714c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefc015d590 0x7fefc00b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:03.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.918+0000 7fefc714c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fefc00d3890 0x7fefc00b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:03.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.918+0000 7fefc714c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc00b5cc0 con 0x7fefc00b4ed0 2024-04-06T12:30:03.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.918+0000 7fefc714c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefc00b5890 con 0x7fefc00d3890 2024-04-06T12:30:03.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.918+0000 7fefc714c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc00d1500 con 0x7fefc015d590 2024-04-06T12:30:03.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.918+0000 7fefbffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fefc00b4ed0 0x7fefc00b6580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40294/0 (socket says 172.21.15.189:40294) 2024-04-06T12:30:03.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.918+0000 7fefbffff700 1 -- 172.21.15.189:0/2675700874 learned_addr learned my addr 172.21.15.189:0/2675700874 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:03.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.918+0000 7fefc56e9700 1 --2- 172.21.15.189:0/2675700874 >> v2:172.21.15.192:3300/0 conn(0x7fefc00d3890 0x7fefc00b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:03.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.918+0000 7fefc4ee8700 1 --2- 172.21.15.189:0/2675700874 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefc015d590 0x7fefc00b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:03.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.918+0000 7fefbf7fe700 1 -- 172.21.15.189:0/2675700874 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3348812397 0 0) 0x7fefc00b5cc0 con 0x7fefc00b4ed0 2024-04-06T12:30:03.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.918+0000 7fefbf7fe700 1 -- 172.21.15.189:0/2675700874 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefb00035b0 con 0x7fefc00b4ed0 2024-04-06T12:30:03.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.918+0000 7fefc4ee8700 1 -- 172.21.15.189:0/2675700874 >> v2:172.21.15.192:3300/0 conn(0x7fefc00d3890 msgr2=0x7fefc00b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:03.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.918+0000 7fefc4ee8700 1 --2- 172.21.15.189:0/2675700874 >> v2:172.21.15.192:3300/0 conn(0x7fefc00d3890 0x7fefc00b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.918+0000 7fefc4ee8700 1 -- 172.21.15.189:0/2675700874 >> v1:172.21.15.189:6789/0 conn(0x7fefc00b4ed0 legacy=0x7fefc00b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:03.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.918+0000 7fefc4ee8700 1 -- 172.21.15.189:0/2675700874 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc00d17e0 con 0x7fefc015d590 2024-04-06T12:30:03.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.919+0000 7fefc56e9700 1 --2- 172.21.15.189:0/2675700874 >> v2:172.21.15.192:3300/0 conn(0x7fefc00d3890 0x7fefc00b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:30:03.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.919+0000 7fefc4ee8700 1 --2- 172.21.15.189:0/2675700874 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefc015d590 0x7fefc00b8d70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009870 tx=0x7fefb4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34cd43464cc33f57 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:03.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.919+0000 7fefbf7fe700 1 -- 172.21.15.189:0/2675700874 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb4011830 con 0x7fefc015d590 2024-04-06T12:30:03.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.919+0000 7fefbf7fe700 1 -- 172.21.15.189:0/2675700874 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb4011990 con 0x7fefc015d590 2024-04-06T12:30:03.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.919+0000 7fefbf7fe700 1 -- 172.21.15.189:0/2675700874 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb4011d00 con 0x7fefc015d590 2024-04-06T12:30:03.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.919+0000 7fefc714c700 1 -- 172.21.15.189:0/2675700874 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefc015d590 msgr2=0x7fefc00b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:03.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.919+0000 7fefc714c700 1 --2- 172.21.15.189:0/2675700874 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefc015d590 0x7fefc00b8d70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009870 tx=0x7fefb4009b80 comp rx=0 tx=0).stop 2024-04-06T12:30:03.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.919+0000 7fefc714c700 1 -- 172.21.15.189:0/2675700874 shutdown_connections 2024-04-06T12:30:03.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.919+0000 7fefc714c700 1 --2- 172.21.15.189:0/2675700874 >> v2:172.21.15.192:3300/0 conn(0x7fefc00d3890 0x7fefc00b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.919+0000 7fefc714c700 1 --2- 172.21.15.189:0/2675700874 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefc015d590 0x7fefc00b8d70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.919+0000 7fefc714c700 1 -- 172.21.15.189:0/2675700874 >> 172.21.15.189:0/2675700874 conn(0x7fefc006bbc0 msgr2=0x7fefc006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:03.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.919+0000 7fefc714c700 1 -- 172.21.15.189:0/2675700874 shutdown_connections 2024-04-06T12:30:03.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.919+0000 7fefc714c700 1 -- 172.21.15.189:0/2675700874 wait complete. 2024-04-06T12:30:03.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.921+0000 7fefc714c700 1 Processor -- start 2024-04-06T12:30:03.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.921+0000 7fefc714c700 1 -- start start 2024-04-06T12:30:03.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.921+0000 7fefc714c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fefc00b4ed0 0x7fefc0150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:03.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.921+0000 7fefc714c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefc00d3890 0x7fefc014fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:03.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefc714c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefc015d590 0x7fefc015b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:03.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefc714c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc015eeb0 con 0x7fefc015d590 2024-04-06T12:30:03.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefc714c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc015e910 con 0x7fefc00b4ed0 2024-04-06T12:30:03.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefc714c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc015ebd0 con 0x7fefc00d3890 2024-04-06T12:30:03.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefc56e9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefc015d590 0x7fefc015b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:03.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefc56e9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefc015d590 0x7fefc015b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58312/0 (socket says 172.21.15.189:58312) 2024-04-06T12:30:03.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefc56e9700 1 -- 172.21.15.189:0/945427075 learned_addr learned my addr 172.21.15.189:0/945427075 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:03.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefbffff700 1 --2- 172.21.15.189:0/945427075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefc00d3890 0x7fefc014fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:03.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefc4ee8700 1 --2- 172.21.15.189:0/945427075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fefc00b4ed0 0x7fefc0150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:03.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefc56e9700 1 -- 172.21.15.189:0/945427075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefc00d3890 msgr2=0x7fefc014fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:03.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefc56e9700 1 --2- 172.21.15.189:0/945427075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefc00d3890 0x7fefc014fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefc56e9700 1 -- 172.21.15.189:0/945427075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fefc00b4ed0 msgr2=0x7fefc0150b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:03.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefc56e9700 1 --2- 172.21.15.189:0/945427075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fefc00b4ed0 0x7fefc0150b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:03.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefc56e9700 1 -- 172.21.15.189:0/945427075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb8009a70 con 0x7fefc015d590 2024-04-06T12:30:03.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefc56e9700 1 --2- 172.21.15.189:0/945427075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefc015d590 0x7fefc015b110 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fefb800efc0 tx=0x7fefb800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:03.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefbd7fa700 1 -- 172.21.15.189:0/945427075 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb800ca90 con 0x7fefc015d590 2024-04-06T12:30:03.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefbd7fa700 1 -- 172.21.15.189:0/945427075 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb800f040 con 0x7fefc015d590 2024-04-06T12:30:03.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefbd7fa700 1 -- 172.21.15.189:0/945427075 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb800ce10 con 0x7fefc015d590 2024-04-06T12:30:03.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefc714c700 1 -- 172.21.15.189:0/945427075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb40094a0 con 0x7fefc015d590 2024-04-06T12:30:03.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.922+0000 7fefc714c700 1 -- 172.21.15.189:0/945427075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc00ca980 con 0x7fefc015d590 2024-04-06T12:30:03.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.923+0000 7fefc714c700 1 -- 172.21.15.189:0/945427075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc00ca500 con 0x7fefc015d590 2024-04-06T12:30:03.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.924+0000 7fefbd7fa700 1 -- 172.21.15.189:0/945427075 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 60) v1 ==== 88856+0+0 (secure 0 0 0) 0x7fefb8014480 con 0x7fefc015d590 2024-04-06T12:30:03.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.924+0000 7fefbd7fa700 1 -- 172.21.15.189:0/945427075 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6833+0+0 (secure 0 0 0) 0x7fefb8089940 con 0x7fefc015d590 2024-04-06T12:30:03.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:03.929+0000 7fefbd7fa700 1 -- 172.21.15.189:0/945427075 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fefc00ca500 con 0x7fefc015d590 2024-04-06T12:30:04.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.167+0000 7fefc714c700 1 -- 172.21.15.189:0/945427075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fefc00ca500 con 0x7fefc015d590 2024-04-06T12:30:04.166 INFO:tasks.ceph.mon.a.smithi189.stderr:2024-04-06T12:30:04.168+0000 7fa9fe1ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T12:30:04.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.176+0000 7fefbd7fa700 1 -- 172.21.15.189:0/945427075 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v47) v1 ==== 161+0+0 (secure 0 0 0) 0x7fefc00ca500 con 0x7fefc015d590 2024-04-06T12:30:04.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.178+0000 7fefc714c700 1 -- 172.21.15.189:0/945427075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefc015d590 msgr2=0x7fefc015b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:04.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.178+0000 7fefc714c700 1 --2- 172.21.15.189:0/945427075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefc015d590 0x7fefc015b110 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fefb800efc0 tx=0x7fefb800c3a0 comp rx=0 tx=0).stop 2024-04-06T12:30:04.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.178+0000 7fefc714c700 1 -- 172.21.15.189:0/945427075 shutdown_connections 2024-04-06T12:30:04.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.178+0000 7fefc714c700 1 --2- 172.21.15.189:0/945427075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefc015d590 0x7fefc015b110 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:04.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.178+0000 7fefc714c700 1 --2- 172.21.15.189:0/945427075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fefc00b4ed0 0x7fefc0150b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:04.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.178+0000 7fefc714c700 1 --2- 172.21.15.189:0/945427075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefc00d3890 0x7fefc014fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:04.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.178+0000 7fefc714c700 1 -- 172.21.15.189:0/945427075 >> 172.21.15.189:0/945427075 conn(0x7fefc006bbc0 msgr2=0x7fefc00cd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:04.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.178+0000 7fefc714c700 1 -- 172.21.15.189:0/945427075 shutdown_connections 2024-04-06T12:30:04.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.178+0000 7fefc714c700 1 -- 172.21.15.189:0/945427075 wait complete. 2024-04-06T12:30:04.177 INFO:teuthology.orchestra.run.smithi189.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-06T12:30:04.178 INFO:tasks.ceph.mds.c.smithi189.stderr: -8743> 2024-04-06T12:28:46.233+0000 7fbfbbd71700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T12:30:04.189 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-06T12:30:04.211 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T12:30:04.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.317+0000 7f4477e72700 1 Processor -- start 2024-04-06T12:30:04.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.317+0000 7f4477e72700 1 -- start start 2024-04-06T12:30:04.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.317+0000 7f4477e72700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f447015e8a0 0x7f44700cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:04.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.317+0000 7f4477e72700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f447015eee0 0x7f447015f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:04.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.317+0000 7f4477e72700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44700669d0 con 0x7f447015f8a0 2024-04-06T12:30:04.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.317+0000 7f4477e72700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f447015fd30 con 0x7f447015eee0 2024-04-06T12:30:04.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.317+0000 7f4477e72700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f447006ede0 con 0x7f447015e8a0 2024-04-06T12:30:04.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.317+0000 7f447640f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f447015f8a0 0x7f447006ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40308/0 (socket says 172.21.15.189:40308) 2024-04-06T12:30:04.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.317+0000 7f447640f700 1 -- 172.21.15.189:0/479400748 learned_addr learned my addr 172.21.15.189:0/479400748 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:04.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.317+0000 7f447540d700 1 --2- 172.21.15.189:0/479400748 >> v2:172.21.15.192:3300/0 conn(0x7f447015eee0 0x7f447015f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:04.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.317+0000 7f4475c0e700 1 --2- 172.21.15.189:0/479400748 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f447015e8a0 0x7f44700cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:04.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.317+0000 7f4474c0c700 1 -- 172.21.15.189:0/479400748 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279802433 0 0) 0x7f44700669d0 con 0x7f447015f8a0 2024-04-06T12:30:04.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.317+0000 7f4474c0c700 1 -- 172.21.15.189:0/479400748 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f445c0035b0 con 0x7f447015f8a0 2024-04-06T12:30:04.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.317+0000 7f447540d700 1 -- 172.21.15.189:0/479400748 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f447015e8a0 msgr2=0x7f44700cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:04.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.317+0000 7f447540d700 1 --2- 172.21.15.189:0/479400748 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f447015e8a0 0x7f44700cbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:04.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.317+0000 7f447540d700 1 -- 172.21.15.189:0/479400748 >> v1:172.21.15.189:6789/0 conn(0x7f447015f8a0 legacy=0x7f447006ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:04.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.317+0000 7f447540d700 1 -- 172.21.15.189:0/479400748 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447015ec80 con 0x7f447015eee0 2024-04-06T12:30:04.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.317+0000 7f4475c0e700 1 --2- 172.21.15.189:0/479400748 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f447015e8a0 0x7f44700cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:30:04.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.318+0000 7f447540d700 1 --2- 172.21.15.189:0/479400748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f447015eee0 0x7f447015f360 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f446c009840 tx=0x7f446c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb70b798bc8bdf38 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:04.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.318+0000 7f4474c0c700 1 -- 172.21.15.189:0/479400748 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446c011830 con 0x7f447015eee0 2024-04-06T12:30:04.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.318+0000 7f4474c0c700 1 -- 172.21.15.189:0/479400748 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446c011990 con 0x7f447015eee0 2024-04-06T12:30:04.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.318+0000 7f4474c0c700 1 -- 172.21.15.189:0/479400748 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446c011d00 con 0x7f447015eee0 2024-04-06T12:30:04.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.318+0000 7f4477e72700 1 -- 172.21.15.189:0/479400748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f447015eee0 msgr2=0x7f447015f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:04.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.318+0000 7f4477e72700 1 --2- 172.21.15.189:0/479400748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f447015eee0 0x7f447015f360 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f446c009840 tx=0x7f446c009b50 comp rx=0 tx=0).stop 2024-04-06T12:30:04.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.318+0000 7f4477e72700 1 -- 172.21.15.189:0/479400748 shutdown_connections 2024-04-06T12:30:04.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.318+0000 7f4477e72700 1 --2- 172.21.15.189:0/479400748 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f447015e8a0 0x7f44700cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:04.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.318+0000 7f4477e72700 1 --2- 172.21.15.189:0/479400748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f447015eee0 0x7f447015f360 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:04.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.318+0000 7f4477e72700 1 -- 172.21.15.189:0/479400748 >> 172.21.15.189:0/479400748 conn(0x7f44700cc290 msgr2=0x7f44700cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:04.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.319+0000 7f4477e72700 1 -- 172.21.15.189:0/479400748 shutdown_connections 2024-04-06T12:30:04.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.319+0000 7f4477e72700 1 -- 172.21.15.189:0/479400748 wait complete. 2024-04-06T12:30:04.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.320+0000 7f4477e72700 1 Processor -- start 2024-04-06T12:30:04.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.320+0000 7f4477e72700 1 -- start start 2024-04-06T12:30:04.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.320+0000 7f4477e72700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f447015e8a0 0x7f44700c9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:04.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.320+0000 7f4477e72700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f447015eee0 0x7f4470144b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:04.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.320+0000 7f4477e72700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f447015f8a0 0x7f44700b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:04.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.320+0000 7f4477e72700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4470145040 con 0x7f447015e8a0 2024-04-06T12:30:04.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.320+0000 7f4477e72700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44700b3bb0 con 0x7f447015eee0 2024-04-06T12:30:04.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.320+0000 7f4477e72700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44700b3e90 con 0x7f447015f8a0 2024-04-06T12:30:04.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.321+0000 7f4475c0e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f447015e8a0 0x7f44700c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:04.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.321+0000 7f4475c0e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f447015e8a0 0x7f44700c9780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58346/0 (socket says 172.21.15.189:58346) 2024-04-06T12:30:04.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.321+0000 7f4475c0e700 1 -- 172.21.15.189:0/3930347656 learned_addr learned my addr 172.21.15.189:0/3930347656 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:04.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.321+0000 7f447640f700 1 --2- 172.21.15.189:0/3930347656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f447015f8a0 0x7f44700b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:04.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.321+0000 7f447540d700 1 --2- 172.21.15.189:0/3930347656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f447015eee0 0x7f4470144b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:04.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.321+0000 7f4475c0e700 1 -- 172.21.15.189:0/3930347656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f447015f8a0 msgr2=0x7f44700b4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:04.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.321+0000 7f4475c0e700 1 --2- 172.21.15.189:0/3930347656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f447015f8a0 0x7f44700b4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:04.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.321+0000 7f4475c0e700 1 -- 172.21.15.189:0/3930347656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f447015eee0 msgr2=0x7f4470144b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:04.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.321+0000 7f4475c0e700 1 --2- 172.21.15.189:0/3930347656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f447015eee0 0x7f4470144b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:04.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.321+0000 7f4475c0e700 1 -- 172.21.15.189:0/3930347656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44600094a0 con 0x7f447015e8a0 2024-04-06T12:30:04.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.321+0000 7f4475c0e700 1 --2- 172.21.15.189:0/3930347656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f447015e8a0 0x7f44700c9780 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f446000ebe0 tx=0x7f4460009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:04.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.321+0000 7f44667fc700 1 -- 172.21.15.189:0/3930347656 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446000c8b0 con 0x7f447015e8a0 2024-04-06T12:30:04.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.321+0000 7f44667fc700 1 -- 172.21.15.189:0/3930347656 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446000ca10 con 0x7f447015e8a0 2024-04-06T12:30:04.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.321+0000 7f44667fc700 1 -- 172.21.15.189:0/3930347656 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446000cda0 con 0x7f447015e8a0 2024-04-06T12:30:04.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.321+0000 7f4477e72700 1 -- 172.21.15.189:0/3930347656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446c0094a0 con 0x7f447015e8a0 2024-04-06T12:30:04.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.321+0000 7f4477e72700 1 -- 172.21.15.189:0/3930347656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44700b4450 con 0x7f447015e8a0 2024-04-06T12:30:04.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.322+0000 7f4477e72700 1 -- 172.21.15.189:0/3930347656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44700b4aa0 con 0x7f447015e8a0 2024-04-06T12:30:04.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.323+0000 7f44667fc700 1 -- 172.21.15.189:0/3930347656 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 60) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f4460013470 con 0x7f447015e8a0 2024-04-06T12:30:04.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.323+0000 7f44667fc700 1 -- 172.21.15.189:0/3930347656 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6919+0+0 (secure 0 0 0) 0x7f44600893b0 con 0x7f447015e8a0 2024-04-06T12:30:04.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.328+0000 7f44667fc700 1 -- 172.21.15.189:0/3930347656 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4460054520 con 0x7f447015e8a0 2024-04-06T12:30:04.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.456+0000 7f44667fc700 1 -- 172.21.15.189:0/3930347656 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 61) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f4460053fd0 con 0x7f447015e8a0 2024-04-06T12:30:04.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.456+0000 7f44667fc700 1 --2- 172.21.15.189:0/3930347656 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4454036150 0x7f4454072c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:04.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.457+0000 7f447540d700 1 --2- 172.21.15.189:0/3930347656 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4454036150 0x7f4454072c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:04.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.458+0000 7f447540d700 1 --2- 172.21.15.189:0/3930347656 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4454036150 0x7f4454072c10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f446c000c00 tx=0x7f446c008030 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:04.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:04.571+0000 7f4477e72700 1 -- 172.21.15.189:0/3930347656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f447006e790 con 0x7f447015e8a0 2024-04-06T12:30:05.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.179+0000 7f44667fc700 1 -- 172.21.15.189:0/3930347656 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v48) v1 ==== 104+0+0 (secure 0 0 0) 0x7f44600874f0 con 0x7f447015e8a0 2024-04-06T12:30:05.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.181+0000 7f4477e72700 1 -- 172.21.15.189:0/3930347656 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4454036150 msgr2=0x7f4454072c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:05.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.181+0000 7f4477e72700 1 --2- 172.21.15.189:0/3930347656 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4454036150 0x7f4454072c10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f446c000c00 tx=0x7f446c008030 comp rx=0 tx=0).stop 2024-04-06T12:30:05.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.181+0000 7f4477e72700 1 -- 172.21.15.189:0/3930347656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f447015e8a0 msgr2=0x7f44700c9780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:05.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.181+0000 7f4477e72700 1 --2- 172.21.15.189:0/3930347656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f447015e8a0 0x7f44700c9780 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f446000ebe0 tx=0x7f4460009e20 comp rx=0 tx=0).stop 2024-04-06T12:30:05.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.182+0000 7f4477e72700 1 -- 172.21.15.189:0/3930347656 shutdown_connections 2024-04-06T12:30:05.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.182+0000 7f4477e72700 1 --2- 172.21.15.189:0/3930347656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f447015f8a0 0x7f44700b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:05.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.182+0000 7f4477e72700 1 --2- 172.21.15.189:0/3930347656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f447015e8a0 0x7f44700c9780 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:05.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.182+0000 7f4477e72700 1 --2- 172.21.15.189:0/3930347656 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4454036150 0x7f4454072c10 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:05.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.182+0000 7f4477e72700 1 --2- 172.21.15.189:0/3930347656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f447015eee0 0x7f4470144b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:05.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.182+0000 7f4477e72700 1 -- 172.21.15.189:0/3930347656 >> 172.21.15.189:0/3930347656 conn(0x7f44700cc290 msgr2=0x7f447015c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:05.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.182+0000 7f4477e72700 1 -- 172.21.15.189:0/3930347656 shutdown_connections 2024-04-06T12:30:05.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.182+0000 7f4477e72700 1 -- 172.21.15.189:0/3930347656 wait complete. 2024-04-06T12:30:05.193 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-06T12:30:05.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.364+0000 7fbe3e729700 1 Processor -- start 2024-04-06T12:30:05.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.365+0000 7fbe3e729700 1 -- start start 2024-04-06T12:30:05.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.365+0000 7fbe3e729700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbe380cb870 0x7fbe3815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:05.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.365+0000 7fbe3e729700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbe380cfa00 0x7fbe380cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:05.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.365+0000 7fbe3e729700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe380cbd80 con 0x7fbe380d11b0 2024-04-06T12:30:05.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.365+0000 7fbe3e729700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe380ccdc0 con 0x7fbe380cfa00 2024-04-06T12:30:05.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.365+0000 7fbe3e729700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe380d0b50 con 0x7fbe380cb870 2024-04-06T12:30:05.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.365+0000 7fbe3ccc6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbe380cfa00 0x7fbe380cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:05.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.366+0000 7fbe3ccc6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbe380cfa00 0x7fbe380cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58356/0 (socket says 172.21.15.189:58356) 2024-04-06T12:30:05.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.366+0000 7fbe3ccc6700 1 -- 172.21.15.189:0/4236878332 learned_addr learned my addr 172.21.15.189:0/4236878332 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:05.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.366+0000 7fbe37fff700 1 --1- 172.21.15.189:0/4236878332 >> v1:172.21.15.189:6789/0 conn(0x7fbe380d11b0 0x7fbe380d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40322/0 (socket says 172.21.15.189:40322) 2024-04-06T12:30:05.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.366+0000 7fbe3ccc6700 1 -- 172.21.15.189:0/4236878332 >> v2:172.21.15.192:3300/0 conn(0x7fbe380cb870 msgr2=0x7fbe3815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:05.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.366+0000 7fbe377fe700 1 --2- 172.21.15.189:0/4236878332 >> v2:172.21.15.192:3300/0 conn(0x7fbe380cb870 0x7fbe3815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:05.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.366+0000 7fbe3ccc6700 1 --2- 172.21.15.189:0/4236878332 >> v2:172.21.15.192:3300/0 conn(0x7fbe380cb870 0x7fbe3815c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:05.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.366+0000 7fbe3ccc6700 1 -- 172.21.15.189:0/4236878332 >> v1:172.21.15.189:6789/0 conn(0x7fbe380d11b0 legacy=0x7fbe380d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:05.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.366+0000 7fbe3ccc6700 1 -- 172.21.15.189:0/4236878332 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe380cef20 con 0x7fbe380cfa00 2024-04-06T12:30:05.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.366+0000 7fbe3ccc6700 1 --2- 172.21.15.189:0/4236878332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe380cfa00 0x7fbe380cd560 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fbe3804fb40 tx=0x7fbe3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=471f8ad33329698b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:05.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.366+0000 7fbe36ffd700 1 -- 172.21.15.189:0/4236878332 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3000e070 con 0x7fbe380cfa00 2024-04-06T12:30:05.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.366+0000 7fbe36ffd700 1 -- 172.21.15.189:0/4236878332 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe3000b9c0 con 0x7fbe380cfa00 2024-04-06T12:30:05.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.366+0000 7fbe36ffd700 1 -- 172.21.15.189:0/4236878332 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3000bd70 con 0x7fbe380cfa00 2024-04-06T12:30:05.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.367+0000 7fbe3e729700 1 -- 172.21.15.189:0/4236878332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe380cfa00 msgr2=0x7fbe380cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.367+0000 7fbe3e729700 1 --2- 172.21.15.189:0/4236878332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe380cfa00 0x7fbe380cd560 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fbe3804fb40 tx=0x7fbe3000b330 comp rx=0 tx=0).stop 2024-04-06T12:30:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.367+0000 7fbe3e729700 1 -- 172.21.15.189:0/4236878332 shutdown_connections 2024-04-06T12:30:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.367+0000 7fbe3e729700 1 --2- 172.21.15.189:0/4236878332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe380cfa00 0x7fbe380cd560 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.367+0000 7fbe3e729700 1 --2- 172.21.15.189:0/4236878332 >> v2:172.21.15.192:3300/0 conn(0x7fbe380cb870 0x7fbe3815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.367+0000 7fbe3e729700 1 -- 172.21.15.189:0/4236878332 >> 172.21.15.189:0/4236878332 conn(0x7fbe380b2db0 msgr2=0x7fbe380b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.367+0000 7fbe3e729700 1 -- 172.21.15.189:0/4236878332 shutdown_connections 2024-04-06T12:30:05.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.367+0000 7fbe3e729700 1 -- 172.21.15.189:0/4236878332 wait complete. 2024-04-06T12:30:05.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.368+0000 7fbe3e729700 1 Processor -- start 2024-04-06T12:30:05.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.368+0000 7fbe3e729700 1 -- start start 2024-04-06T12:30:05.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.369+0000 7fbe3e729700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe380cb870 0x7fbe381f0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:05.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.369+0000 7fbe3e729700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe380cfa00 0x7fbe3815ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:05.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.369+0000 7fbe3e729700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe380d11b0 0x7fbe3815cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:05.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.369+0000 7fbe3e729700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe380c76a0 con 0x7fbe380cfa00 2024-04-06T12:30:05.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.369+0000 7fbe3e729700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe381f0b80 con 0x7fbe380cb870 2024-04-06T12:30:05.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.369+0000 7fbe3e729700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe380c6b20 con 0x7fbe380d11b0 2024-04-06T12:30:05.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.369+0000 7fbe377fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe380cfa00 0x7fbe3815ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:05.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.369+0000 7fbe377fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe380cfa00 0x7fbe3815ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58362/0 (socket says 172.21.15.189:58362) 2024-04-06T12:30:05.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.369+0000 7fbe377fe700 1 -- 172.21.15.189:0/3903985956 learned_addr learned my addr 172.21.15.189:0/3903985956 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:05.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.370+0000 7fbe3ccc6700 1 --2- 172.21.15.189:0/3903985956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe380d11b0 0x7fbe3815cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:05.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.370+0000 7fbe37fff700 1 --2- 172.21.15.189:0/3903985956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe380cb870 0x7fbe381f0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:05.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.370+0000 7fbe377fe700 1 -- 172.21.15.189:0/3903985956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe380d11b0 msgr2=0x7fbe3815cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:05.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.370+0000 7fbe377fe700 1 --2- 172.21.15.189:0/3903985956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe380d11b0 0x7fbe3815cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:05.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.370+0000 7fbe377fe700 1 -- 172.21.15.189:0/3903985956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe380cb870 msgr2=0x7fbe381f0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:05.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.370+0000 7fbe377fe700 1 --2- 172.21.15.189:0/3903985956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe380cb870 0x7fbe381f0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:05.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.370+0000 7fbe377fe700 1 -- 172.21.15.189:0/3903985956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe30009b30 con 0x7fbe380cfa00 2024-04-06T12:30:05.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.370+0000 7fbe3ccc6700 1 --2- 172.21.15.189:0/3903985956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe380d11b0 0x7fbe3815cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:30:05.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.370+0000 7fbe37fff700 1 --2- 172.21.15.189:0/3903985956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe380cb870 0x7fbe381f0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:30:05.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.370+0000 7fbe377fe700 1 --2- 172.21.15.189:0/3903985956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe380cfa00 0x7fbe3815ca60 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800c7f0 tx=0x7fbe2800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:05.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.370+0000 7fbe34ff9700 1 -- 172.21.15.189:0/3903985956 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe28011800 con 0x7fbe380cfa00 2024-04-06T12:30:05.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.370+0000 7fbe34ff9700 1 -- 172.21.15.189:0/3903985956 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe28011960 con 0x7fbe380cfa00 2024-04-06T12:30:05.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.370+0000 7fbe3e729700 1 -- 172.21.15.189:0/3903985956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe380c6e00 con 0x7fbe380cfa00 2024-04-06T12:30:05.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.370+0000 7fbe34ff9700 1 -- 172.21.15.189:0/3903985956 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe28011d10 con 0x7fbe380cfa00 2024-04-06T12:30:05.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.370+0000 7fbe3e729700 1 -- 172.21.15.189:0/3903985956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe380c73c0 con 0x7fbe380cfa00 2024-04-06T12:30:05.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.371+0000 7fbe3e729700 1 -- 172.21.15.189:0/3903985956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe380b49b0 con 0x7fbe380cfa00 2024-04-06T12:30:05.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.371+0000 7fbe34ff9700 1 -- 172.21.15.189:0/3903985956 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 61) v1 ==== 88983+0+0 (secure 0 0 0) 0x7fbe28018360 con 0x7fbe380cfa00 2024-04-06T12:30:05.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.372+0000 7fbe34ff9700 1 --2- 172.21.15.189:0/3903985956 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbe1806aae0 0x7fbe1806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:05.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.372+0000 7fbe34ff9700 1 -- 172.21.15.189:0/3903985956 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6919+0+0 (secure 0 0 0) 0x7fbe28089320 con 0x7fbe380cfa00 2024-04-06T12:30:05.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.372+0000 7fbe37fff700 1 --2- 172.21.15.189:0/3903985956 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbe1806aae0 0x7fbe1806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:05.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.372+0000 7fbe37fff700 1 --2- 172.21.15.189:0/3903985956 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbe1806aae0 0x7fbe1806cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbe20000f60 tx=0x7fbe20009040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:05.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.376+0000 7fbe34ff9700 1 -- 172.21.15.189:0/3903985956 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbe380b49b0 con 0x7fbe380cfa00 2024-04-06T12:30:05.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:05.577+0000 7fbe3e729700 1 -- 172.21.15.189:0/3903985956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbe380b49b0 con 0x7fbe380cfa00 2024-04-06T12:30:06.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.474+0000 7fbe34ff9700 1 -- 172.21.15.189:0/3903985956 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 62) v1 ==== 89191+0+0 (secure 0 0 0) 0x7fbe28054750 con 0x7fbe380cfa00 2024-04-06T12:30:06.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.475+0000 7fbe34ff9700 1 -- 172.21.15.189:0/3903985956 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v118) v1 ==== 181+0+0 (secure 0 0 0) 0x7fbe280563e0 con 0x7fbe380cfa00 2024-04-06T12:30:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.477+0000 7fbe3e729700 1 -- 172.21.15.189:0/3903985956 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbe1806aae0 msgr2=0x7fbe1806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.477+0000 7fbe3e729700 1 --2- 172.21.15.189:0/3903985956 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbe1806aae0 0x7fbe1806cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbe20000f60 tx=0x7fbe20009040 comp rx=0 tx=0).stop 2024-04-06T12:30:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.477+0000 7fbe3e729700 1 -- 172.21.15.189:0/3903985956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe380cfa00 msgr2=0x7fbe3815ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.477+0000 7fbe3e729700 1 --2- 172.21.15.189:0/3903985956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe380cfa00 0x7fbe3815ca60 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800c7f0 tx=0x7fbe2800cbb0 comp rx=0 tx=0).stop 2024-04-06T12:30:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.477+0000 7fbe3e729700 1 -- 172.21.15.189:0/3903985956 shutdown_connections 2024-04-06T12:30:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.477+0000 7fbe3e729700 1 --2- 172.21.15.189:0/3903985956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe380d11b0 0x7fbe3815cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.477+0000 7fbe3e729700 1 --2- 172.21.15.189:0/3903985956 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbe1806aae0 0x7fbe1806cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.477+0000 7fbe3e729700 1 --2- 172.21.15.189:0/3903985956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe380cb870 0x7fbe381f0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.477+0000 7fbe3e729700 1 --2- 172.21.15.189:0/3903985956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe380cfa00 0x7fbe3815ca60 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.477+0000 7fbe3e729700 1 -- 172.21.15.189:0/3903985956 >> 172.21.15.189:0/3903985956 conn(0x7fbe380b2db0 msgr2=0x7fbe380cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.477+0000 7fbe3e729700 1 -- 172.21.15.189:0/3903985956 shutdown_connections 2024-04-06T12:30:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.477+0000 7fbe3e729700 1 -- 172.21.15.189:0/3903985956 wait complete. 2024-04-06T12:30:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' removed 2024-04-06T12:30:06.488 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-06T12:30:06.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.620+0000 7f64de4dc700 1 Processor -- start 2024-04-06T12:30:06.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.620+0000 7f64de4dc700 1 -- start start 2024-04-06T12:30:06.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.621+0000 7f64de4dc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64d806ae40 0x7f64d80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:06.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.621+0000 7f64de4dc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f64d815ddf0 0x7f64d815e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:06.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.621+0000 7f64de4dc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64d80b3990 con 0x7f64d815d7b0 2024-04-06T12:30:06.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.621+0000 7f64de4dc700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64d80b3060 con 0x7f64d815ddf0 2024-04-06T12:30:06.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.621+0000 7f64de4dc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d80b3340 con 0x7f64d806ae40 2024-04-06T12:30:06.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.621+0000 7f64dca79700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f64d815ddf0 0x7f64d815e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:06.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.621+0000 7f64d7fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f64d815d7b0 0x7f64d806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40332/0 (socket says 172.21.15.189:40332) 2024-04-06T12:30:06.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.621+0000 7f64d7fff700 1 -- 172.21.15.189:0/3031611135 learned_addr learned my addr 172.21.15.189:0/3031611135 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:06.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.621+0000 7f64d77fe700 1 --2- 172.21.15.189:0/3031611135 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64d806ae40 0x7f64d80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:06.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.621+0000 7f64dca79700 1 -- 172.21.15.189:0/3031611135 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64d806ae40 msgr2=0x7f64d80b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:06.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.621+0000 7f64dca79700 1 --2- 172.21.15.189:0/3031611135 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64d806ae40 0x7f64d80b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:06.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.621+0000 7f64dca79700 1 -- 172.21.15.189:0/3031611135 >> v1:172.21.15.189:6789/0 conn(0x7f64d815d7b0 legacy=0x7f64d806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:06.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.621+0000 7f64d6ffd700 1 -- 172.21.15.189:0/3031611135 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160898701 0 0) 0x7f64d80b3990 con 0x7f64d815d7b0 2024-04-06T12:30:06.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.621+0000 7f64dca79700 1 -- 172.21.15.189:0/3031611135 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d80b3620 con 0x7f64d815ddf0 2024-04-06T12:30:06.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.621+0000 7f64dca79700 1 --2- 172.21.15.189:0/3031611135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64d815ddf0 0x7f64d815e2a0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f64d0009d90 tx=0x7f64d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2327d3c38b1a180 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:06.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.621+0000 7f64d6ffd700 1 -- 172.21.15.189:0/3031611135 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d000e070 con 0x7f64d815ddf0 2024-04-06T12:30:06.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.621+0000 7f64d6ffd700 1 -- 172.21.15.189:0/3031611135 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d000bb20 con 0x7f64d815ddf0 2024-04-06T12:30:06.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.622+0000 7f64d6ffd700 1 -- 172.21.15.189:0/3031611135 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d0012430 con 0x7f64d815ddf0 2024-04-06T12:30:06.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.622+0000 7f64de4dc700 1 -- 172.21.15.189:0/3031611135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64d815ddf0 msgr2=0x7f64d815e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:06.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.622+0000 7f64de4dc700 1 --2- 172.21.15.189:0/3031611135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64d815ddf0 0x7f64d815e2a0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f64d0009d90 tx=0x7f64d000b330 comp rx=0 tx=0).stop 2024-04-06T12:30:06.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.622+0000 7f64de4dc700 1 -- 172.21.15.189:0/3031611135 shutdown_connections 2024-04-06T12:30:06.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.622+0000 7f64de4dc700 1 --2- 172.21.15.189:0/3031611135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64d815ddf0 0x7f64d815e2a0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:06.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.622+0000 7f64de4dc700 1 --2- 172.21.15.189:0/3031611135 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64d806ae40 0x7f64d80b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:06.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.622+0000 7f64de4dc700 1 -- 172.21.15.189:0/3031611135 >> 172.21.15.189:0/3031611135 conn(0x7f64d80d00b0 msgr2=0x7f64d80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:06.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.622+0000 7f64de4dc700 1 -- 172.21.15.189:0/3031611135 shutdown_connections 2024-04-06T12:30:06.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.622+0000 7f64de4dc700 1 -- 172.21.15.189:0/3031611135 wait complete. 2024-04-06T12:30:06.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.623+0000 7f64de4dc700 1 Processor -- start 2024-04-06T12:30:06.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.623+0000 7f64de4dc700 1 -- start start 2024-04-06T12:30:06.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.624+0000 7f64de4dc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64d806ae40 0x7f64d8154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:06.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.624+0000 7f64de4dc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64d815d7b0 0x7f64d81f5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:06.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.624+0000 7f64de4dc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64d815ddf0 0x7f64d81f6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:06.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.624+0000 7f64de4dc700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d80cd1e0 con 0x7f64d806ae40 2024-04-06T12:30:06.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.624+0000 7f64de4dc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d815a7f0 con 0x7f64d815ddf0 2024-04-06T12:30:06.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.624+0000 7f64de4dc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d80af7d0 con 0x7f64d815d7b0 2024-04-06T12:30:06.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.624+0000 7f64d7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64d806ae40 0x7f64d8154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:06.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.624+0000 7f64d7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64d806ae40 0x7f64d8154740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58400/0 (socket says 172.21.15.189:58400) 2024-04-06T12:30:06.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.624+0000 7f64d7fff700 1 -- 172.21.15.189:0/354140255 learned_addr learned my addr 172.21.15.189:0/354140255 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:06.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.624+0000 7f64d77fe700 1 --2- 172.21.15.189:0/354140255 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64d815d7b0 0x7f64d81f5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:06.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.624+0000 7f64dca79700 1 --2- 172.21.15.189:0/354140255 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64d815ddf0 0x7f64d81f6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:06.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.625+0000 7f64d7fff700 1 -- 172.21.15.189:0/354140255 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64d815d7b0 msgr2=0x7f64d81f5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:06.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.625+0000 7f64d7fff700 1 --2- 172.21.15.189:0/354140255 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64d815d7b0 0x7f64d81f5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:06.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.625+0000 7f64d7fff700 1 -- 172.21.15.189:0/354140255 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64d815ddf0 msgr2=0x7f64d81f6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:06.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.625+0000 7f64d7fff700 1 --2- 172.21.15.189:0/354140255 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64d815ddf0 0x7f64d81f6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:06.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.625+0000 7f64d7fff700 1 -- 172.21.15.189:0/354140255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d0009a70 con 0x7f64d806ae40 2024-04-06T12:30:06.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.625+0000 7f64d7fff700 1 --2- 172.21.15.189:0/354140255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64d806ae40 0x7f64d8154740 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f64c800c750 tx=0x7f64c800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:06.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.625+0000 7f64d4ff9700 1 -- 172.21.15.189:0/354140255 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c8004c80 con 0x7f64d806ae40 2024-04-06T12:30:06.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.625+0000 7f64d4ff9700 1 -- 172.21.15.189:0/354140255 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c8004e00 con 0x7f64d806ae40 2024-04-06T12:30:06.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.625+0000 7f64d4ff9700 1 -- 172.21.15.189:0/354140255 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c8007310 con 0x7f64d806ae40 2024-04-06T12:30:06.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.625+0000 7f64de4dc700 1 -- 172.21.15.189:0/354140255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d80afab0 con 0x7f64d806ae40 2024-04-06T12:30:06.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.625+0000 7f64de4dc700 1 -- 172.21.15.189:0/354140255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d8154c80 con 0x7f64d806ae40 2024-04-06T12:30:06.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.626+0000 7f64de4dc700 1 -- 172.21.15.189:0/354140255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d80afcf0 con 0x7f64d806ae40 2024-04-06T12:30:06.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.626+0000 7f64d4ff9700 1 -- 172.21.15.189:0/354140255 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 62) v1 ==== 89191+0+0 (secure 0 0 0) 0x7f64c8007470 con 0x7f64d806ae40 2024-04-06T12:30:06.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.627+0000 7f64d4ff9700 1 --2- 172.21.15.189:0/354140255 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f64c406ad20 0x7f64c406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:06.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.627+0000 7f64d4ff9700 1 -- 172.21.15.189:0/354140255 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f64c8089400 con 0x7f64d806ae40 2024-04-06T12:30:06.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.627+0000 7f64d77fe700 1 --2- 172.21.15.189:0/354140255 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f64c406ad20 0x7f64c406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:06.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.627+0000 7f64d77fe700 1 --2- 172.21.15.189:0/354140255 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f64c406ad20 0x7f64c406d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f64d81f5e50 tx=0x7f64cc008040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:06.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.632+0000 7f64d4ff9700 1 -- 172.21.15.189:0/354140255 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f64d80afcf0 con 0x7f64d806ae40 2024-04-06T12:30:06.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:06.825+0000 7f64de4dc700 1 -- 172.21.15.189:0/354140255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f64d815a2e0 con 0x7f64d806ae40 2024-04-06T12:30:07.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.703+0000 7f64d4ff9700 1 -- 172.21.15.189:0/354140255 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f64c80546b0 con 0x7f64d806ae40 2024-04-06T12:30:07.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.704+0000 7f64d4ff9700 1 -- 172.21.15.189:0/354140255 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v119) v1 ==== 169+0+0 (secure 0 0 0) 0x7f64c8073ac0 con 0x7f64d806ae40 2024-04-06T12:30:07.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.707+0000 7f64de4dc700 1 -- 172.21.15.189:0/354140255 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f64c406ad20 msgr2=0x7f64c406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:07.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.707+0000 7f64de4dc700 1 --2- 172.21.15.189:0/354140255 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f64c406ad20 0x7f64c406d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f64d81f5e50 tx=0x7f64cc008040 comp rx=0 tx=0).stop 2024-04-06T12:30:07.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.707+0000 7f64de4dc700 1 -- 172.21.15.189:0/354140255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64d806ae40 msgr2=0x7f64d8154740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:07.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.707+0000 7f64de4dc700 1 --2- 172.21.15.189:0/354140255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64d806ae40 0x7f64d8154740 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f64c800c750 tx=0x7f64c800cb10 comp rx=0 tx=0).stop 2024-04-06T12:30:07.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.707+0000 7f64de4dc700 1 -- 172.21.15.189:0/354140255 shutdown_connections 2024-04-06T12:30:07.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.707+0000 7f64de4dc700 1 --2- 172.21.15.189:0/354140255 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f64c406ad20 0x7f64c406d1e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:07.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.707+0000 7f64de4dc700 1 --2- 172.21.15.189:0/354140255 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64d815ddf0 0x7f64d81f6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:07.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.707+0000 7f64de4dc700 1 --2- 172.21.15.189:0/354140255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64d806ae40 0x7f64d8154740 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:07.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.707+0000 7f64de4dc700 1 --2- 172.21.15.189:0/354140255 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64d815d7b0 0x7f64d81f5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:07.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.707+0000 7f64de4dc700 1 -- 172.21.15.189:0/354140255 >> 172.21.15.189:0/354140255 conn(0x7f64d80d00b0 msgr2=0x7f64d8071cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:07.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.708+0000 7f64de4dc700 1 -- 172.21.15.189:0/354140255 shutdown_connections 2024-04-06T12:30:07.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.708+0000 7f64de4dc700 1 -- 172.21.15.189:0/354140255 wait complete. 2024-04-06T12:30:07.707 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' removed 2024-04-06T12:30:07.721 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-06T12:30:07.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.854+0000 7f0b1962d700 1 Processor -- start 2024-04-06T12:30:07.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.854+0000 7f0b1962d700 1 -- start start 2024-04-06T12:30:07.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.855+0000 7f0b1962d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0b140b4660 0x7f0b140b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:07.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.855+0000 7f0b1962d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b140b5030 con 0x7f0b140b2fc0 2024-04-06T12:30:07.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.855+0000 7f0b1962d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b140cdbe0 con 0x7f0b1415a550 2024-04-06T12:30:07.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.855+0000 7f0b1962d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b1415a940 con 0x7f0b140b4660 2024-04-06T12:30:07.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.855+0000 7f0b137fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0b140b2fc0 0x7f0b140b33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40340/0 (socket says 172.21.15.189:40340) 2024-04-06T12:30:07.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.855+0000 7f0b137fe700 1 -- 172.21.15.189:0/4214021407 learned_addr learned my addr 172.21.15.189:0/4214021407 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:07.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b127fc700 1 --2- 172.21.15.189:0/4214021407 >> v2:172.21.15.192:3300/0 conn(0x7f0b140b4660 0x7f0b140b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:07.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b11ffb700 1 -- 172.21.15.189:0/4214021407 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83933403 0 0) 0x7f0b140b5030 con 0x7f0b140b2fc0 2024-04-06T12:30:07.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b11ffb700 1 -- 172.21.15.189:0/4214021407 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0afc0035b0 con 0x7f0b140b2fc0 2024-04-06T12:30:07.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b11ffb700 1 -- 172.21.15.189:0/4214021407 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2678487077 0 0) 0x7f0afc0035b0 con 0x7f0b140b2fc0 2024-04-06T12:30:07.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b11ffb700 1 -- 172.21.15.189:0/4214021407 >> v2:172.21.15.192:3300/0 conn(0x7f0b140b4660 msgr2=0x7f0b140b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:07.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b11ffb700 1 --2- 172.21.15.189:0/4214021407 >> v2:172.21.15.192:3300/0 conn(0x7f0b140b4660 0x7f0b140b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:07.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b11ffb700 1 -- 172.21.15.189:0/4214021407 >> v1:172.21.15.192:6789/0 conn(0x7f0b1415a550 legacy=0x7f0b140cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:07.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b11ffb700 1 -- 172.21.15.189:0/4214021407 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1415ac00 con 0x7f0b140b2fc0 2024-04-06T12:30:07.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b11ffb700 1 -- 172.21.15.189:0/4214021407 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f0b0c0019f0 con 0x7f0b140b2fc0 2024-04-06T12:30:07.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b11ffb700 1 -- 172.21.15.189:0/4214021407 >> v1:172.21.15.189:6789/0 conn(0x7f0b140b2fc0 legacy=0x7f0b140b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:07.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b11ffb700 1 --2- 172.21.15.189:0/4214021407 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0afc003a00 0x7f0afc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:07.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b11ffb700 1 --2- 172.21.15.189:0/4214021407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0afc004850 0x7f0afc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:07.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b11ffb700 1 --2- 172.21.15.189:0/4214021407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0afc005210 0x7f0afc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:07.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b11ffb700 1 -- 172.21.15.189:0/4214021407 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1415a940 con 0x7f0afc004850 2024-04-06T12:30:07.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b11ffb700 1 -- 172.21.15.189:0/4214021407 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc00c0b0 con 0x7f0afc005210 2024-04-06T12:30:07.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b11ffb700 1 -- 172.21.15.189:0/4214021407 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc00c270 con 0x7f0afc003a00 2024-04-06T12:30:07.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b127fc700 1 --2- 172.21.15.189:0/4214021407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0afc004850 0x7f0afc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:07.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b12ffd700 1 --2- 172.21.15.189:0/4214021407 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0afc003a00 0x7f0afc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:07.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b137fe700 1 --2- 172.21.15.189:0/4214021407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0afc005210 0x7f0afc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:07.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b127fc700 1 -- 172.21.15.189:0/4214021407 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0afc003a00 msgr2=0x7f0afc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:07.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b127fc700 1 --2- 172.21.15.189:0/4214021407 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0afc003a00 0x7f0afc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:07.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b127fc700 1 -- 172.21.15.189:0/4214021407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0afc005210 msgr2=0x7f0afc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:07.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b127fc700 1 --2- 172.21.15.189:0/4214021407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0afc005210 0x7f0afc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:07.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b127fc700 1 -- 172.21.15.189:0/4214021407 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0afc0035b0 con 0x7f0afc004850 2024-04-06T12:30:07.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b137fe700 1 --2- 172.21.15.189:0/4214021407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0afc005210 0x7f0afc00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:07.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b137fe700 1 -- 172.21.15.189:0/4214021407 reap_dead start 2024-04-06T12:30:07.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.856+0000 7f0b127fc700 1 --2- 172.21.15.189:0/4214021407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0afc004850 0x7f0afc004cd0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f0b04009b60 tx=0x7f0afc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=461186e0b9172cdd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:07.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.857+0000 7f0b11ffb700 1 -- 172.21.15.189:0/4214021407 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0400e6c0 con 0x7f0afc004850 2024-04-06T12:30:07.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.857+0000 7f0b11ffb700 1 -- 172.21.15.189:0/4214021407 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0400e820 con 0x7f0afc004850 2024-04-06T12:30:07.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.857+0000 7f0b11ffb700 1 -- 172.21.15.189:0/4214021407 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0400ebb0 con 0x7f0afc004850 2024-04-06T12:30:07.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.857+0000 7f0b1962d700 1 -- 172.21.15.189:0/4214021407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0afc004850 msgr2=0x7f0afc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:07.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.857+0000 7f0b1962d700 1 --2- 172.21.15.189:0/4214021407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0afc004850 0x7f0afc004cd0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f0b04009b60 tx=0x7f0afc005880 comp rx=0 tx=0).stop 2024-04-06T12:30:07.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.857+0000 7f0b1962d700 1 -- 172.21.15.189:0/4214021407 shutdown_connections 2024-04-06T12:30:07.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.857+0000 7f0b1962d700 1 --2- 172.21.15.189:0/4214021407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0afc004850 0x7f0afc004cd0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:07.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.857+0000 7f0b1962d700 1 -- 172.21.15.189:0/4214021407 >> 172.21.15.189:0/4214021407 conn(0x7f0b140d00b0 msgr2=0x7f0b140d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:07.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.857+0000 7f0b1962d700 1 -- 172.21.15.189:0/4214021407 shutdown_connections 2024-04-06T12:30:07.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.857+0000 7f0b1962d700 1 -- 172.21.15.189:0/4214021407 wait complete. 2024-04-06T12:30:07.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.858+0000 7f0b1962d700 1 Processor -- start 2024-04-06T12:30:07.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.859+0000 7f0b1962d700 1 -- start start 2024-04-06T12:30:07.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.859+0000 7f0b1962d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0afc004850 0x7f0b14150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:07.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.859+0000 7f0b1962d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b140cdfe0 0x7f0b1414e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:07.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.859+0000 7f0b1962d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b140ce440 0x7f0b1414ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:07.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.859+0000 7f0b1962d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b14150960 con 0x7f0afc004850 2024-04-06T12:30:07.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.859+0000 7f0b1962d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b141f2150 con 0x7f0b140cdfe0 2024-04-06T12:30:07.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.859+0000 7f0b1962d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b141f23f0 con 0x7f0b140ce440 2024-04-06T12:30:07.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.859+0000 7f0b12ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0afc004850 0x7f0b14150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:07.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.859+0000 7f0b12ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0afc004850 0x7f0b14150420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58418/0 (socket says 172.21.15.189:58418) 2024-04-06T12:30:07.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.859+0000 7f0b12ffd700 1 -- 172.21.15.189:0/1945058379 learned_addr learned my addr 172.21.15.189:0/1945058379 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:07.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.860+0000 7f0b127fc700 1 --2- 172.21.15.189:0/1945058379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b140cdfe0 0x7f0b1414e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:07.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.860+0000 7f0b137fe700 1 --2- 172.21.15.189:0/1945058379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b140ce440 0x7f0b1414ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:07.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.860+0000 7f0b12ffd700 1 -- 172.21.15.189:0/1945058379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b140ce440 msgr2=0x7f0b1414ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:07.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.860+0000 7f0b12ffd700 1 --2- 172.21.15.189:0/1945058379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b140ce440 0x7f0b1414ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:07.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.860+0000 7f0b12ffd700 1 -- 172.21.15.189:0/1945058379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b140cdfe0 msgr2=0x7f0b1414e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:07.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.860+0000 7f0b12ffd700 1 --2- 172.21.15.189:0/1945058379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b140cdfe0 0x7f0b1414e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:07.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.860+0000 7f0b12ffd700 1 -- 172.21.15.189:0/1945058379 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b141f26d0 con 0x7f0afc004850 2024-04-06T12:30:07.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.860+0000 7f0b12ffd700 1 --2- 172.21.15.189:0/1945058379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0afc004850 0x7f0b14150420 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f0b0800f820 tx=0x7f0b0800fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:07.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.860+0000 7f0afb7fe700 1 -- 172.21.15.189:0/1945058379 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b080052e0 con 0x7f0afc004850 2024-04-06T12:30:07.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.860+0000 7f0afb7fe700 1 -- 172.21.15.189:0/1945058379 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b08005440 con 0x7f0afc004850 2024-04-06T12:30:07.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.860+0000 7f0b1962d700 1 -- 172.21.15.189:0/1945058379 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b141f29b0 con 0x7f0afc004850 2024-04-06T12:30:07.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.860+0000 7f0afb7fe700 1 -- 172.21.15.189:0/1945058379 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b080038c0 con 0x7f0afc004850 2024-04-06T12:30:07.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.860+0000 7f0b1962d700 1 -- 172.21.15.189:0/1945058379 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b141f2f70 con 0x7f0afc004850 2024-04-06T12:30:07.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.861+0000 7f0b1962d700 1 -- 172.21.15.189:0/1945058379 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b14067bb0 con 0x7f0afc004850 2024-04-06T12:30:07.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.861+0000 7f0afb7fe700 1 -- 172.21.15.189:0/1945058379 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0b08003a20 con 0x7f0afc004850 2024-04-06T12:30:07.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.862+0000 7f0afb7fe700 1 --2- 172.21.15.189:0/1945058379 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b0006adb0 0x7f0b0006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:07.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.862+0000 7f0afb7fe700 1 -- 172.21.15.189:0/1945058379 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f0b080893d0 con 0x7f0afc004850 2024-04-06T12:30:07.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.862+0000 7f0b127fc700 1 --2- 172.21.15.189:0/1945058379 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b0006adb0 0x7f0b0006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:07.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.862+0000 7f0b127fc700 1 --2- 172.21.15.189:0/1945058379 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b0006adb0 0x7f0b0006d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0b04000c00 tx=0x7f0b04015000 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:07.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:07.866+0000 7f0afb7fe700 1 -- 172.21.15.189:0/1945058379 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0b080543b0 con 0x7f0afc004850 2024-04-06T12:30:08.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.058+0000 7f0b1962d700 1 -- 172.21.15.189:0/1945058379 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0b140b2de0 con 0x7f0afc004850 2024-04-06T12:30:08.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.062+0000 7f0afb7fe700 1 -- 172.21.15.189:0/1945058379 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v37) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0b080579d0 con 0x7f0afc004850 2024-04-06T12:30:08.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.065+0000 7f0b1962d700 1 -- 172.21.15.189:0/1945058379 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b0006adb0 msgr2=0x7f0b0006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:08.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.065+0000 7f0b1962d700 1 --2- 172.21.15.189:0/1945058379 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b0006adb0 0x7f0b0006d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0b04000c00 tx=0x7f0b04015000 comp rx=0 tx=0).stop 2024-04-06T12:30:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.065+0000 7f0b1962d700 1 -- 172.21.15.189:0/1945058379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0afc004850 msgr2=0x7f0b14150420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.065+0000 7f0b1962d700 1 --2- 172.21.15.189:0/1945058379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0afc004850 0x7f0b14150420 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f0b0800f820 tx=0x7f0b0800fbe0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.065+0000 7f0b1962d700 1 -- 172.21.15.189:0/1945058379 shutdown_connections 2024-04-06T12:30:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.065+0000 7f0b1962d700 1 --2- 172.21.15.189:0/1945058379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b140ce440 0x7f0b1414ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.065+0000 7f0b1962d700 1 --2- 172.21.15.189:0/1945058379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0afc004850 0x7f0b14150420 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.065+0000 7f0b1962d700 1 --2- 172.21.15.189:0/1945058379 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0b0006adb0 0x7f0b0006d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.065+0000 7f0b1962d700 1 --2- 172.21.15.189:0/1945058379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b140cdfe0 0x7f0b1414e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.065+0000 7f0b1962d700 1 -- 172.21.15.189:0/1945058379 >> 172.21.15.189:0/1945058379 conn(0x7f0b140d00b0 msgr2=0x7f0b140cf1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.065+0000 7f0b1962d700 1 -- 172.21.15.189:0/1945058379 shutdown_connections 2024-04-06T12:30:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.065+0000 7f0b1962d700 1 -- 172.21.15.189:0/1945058379 wait complete. 2024-04-06T12:30:08.074 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-06T12:30:08.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.201+0000 7f7c6aa0b700 1 Processor -- start 2024-04-06T12:30:08.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.201+0000 7f7c6aa0b700 1 -- start start 2024-04-06T12:30:08.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.201+0000 7f7c6aa0b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7c6415d7b0 0x7f7c640b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:08.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.201+0000 7f7c6aa0b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c6415e500 0x7f7c6406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:08.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.201+0000 7f7c6aa0b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c640b3990 con 0x7f7c6406a730 2024-04-06T12:30:08.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.201+0000 7f7c6aa0b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c6406b0f0 con 0x7f7c6415d7b0 2024-04-06T12:30:08.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.202+0000 7f7c6aa0b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c640b3060 con 0x7f7c6415e500 2024-04-06T12:30:08.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.202+0000 7f7c63fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7c6415d7b0 0x7f7c640b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:08.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.202+0000 7f7c63fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7c6415d7b0 0x7f7c640b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58430/0 (socket says 172.21.15.189:58430) 2024-04-06T12:30:08.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.202+0000 7f7c63fff700 1 -- 172.21.15.189:0/2226995234 learned_addr learned my addr 172.21.15.189:0/2226995234 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:08.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.202+0000 7f7c68fa8700 1 --2- 172.21.15.189:0/2226995234 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c6415e500 0x7f7c6406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:08.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.202+0000 7f7c63fff700 1 -- 172.21.15.189:0/2226995234 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c6415e500 msgr2=0x7f7c6406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:08.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.202+0000 7f7c63fff700 1 --2- 172.21.15.189:0/2226995234 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c6415e500 0x7f7c6406abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.202+0000 7f7c63fff700 1 -- 172.21.15.189:0/2226995234 >> v1:172.21.15.192:6789/0 conn(0x7f7c6406a730 legacy=0x7f7c6415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:08.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.202+0000 7f7c63fff700 1 -- 172.21.15.189:0/2226995234 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c640b3340 con 0x7f7c6415d7b0 2024-04-06T12:30:08.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.202+0000 7f7c63fff700 1 --2- 172.21.15.189:0/2226995234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c6415d7b0 0x7f7c640b3e20 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f7c540098f0 tx=0x7f7c54009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c26699eab26ed52e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:08.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.202+0000 7f7c62ffd700 1 -- 172.21.15.189:0/2226995234 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c54004230 con 0x7f7c6415d7b0 2024-04-06T12:30:08.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.202+0000 7f7c62ffd700 1 -- 172.21.15.189:0/2226995234 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c54004390 con 0x7f7c6415d7b0 2024-04-06T12:30:08.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.202+0000 7f7c62ffd700 1 -- 172.21.15.189:0/2226995234 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c540036a0 con 0x7f7c6415d7b0 2024-04-06T12:30:08.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.203+0000 7f7c6aa0b700 1 -- 172.21.15.189:0/2226995234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c6415d7b0 msgr2=0x7f7c640b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:08.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.203+0000 7f7c6aa0b700 1 --2- 172.21.15.189:0/2226995234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c6415d7b0 0x7f7c640b3e20 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f7c540098f0 tx=0x7f7c54009c00 comp rx=0 tx=0).stop 2024-04-06T12:30:08.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.203+0000 7f7c6aa0b700 1 -- 172.21.15.189:0/2226995234 shutdown_connections 2024-04-06T12:30:08.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.203+0000 7f7c6aa0b700 1 --2- 172.21.15.189:0/2226995234 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c6415e500 0x7f7c6406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.203+0000 7f7c6aa0b700 1 --2- 172.21.15.189:0/2226995234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c6415d7b0 0x7f7c640b3e20 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.203+0000 7f7c6aa0b700 1 -- 172.21.15.189:0/2226995234 >> 172.21.15.189:0/2226995234 conn(0x7f7c640d00b0 msgr2=0x7f7c640d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:08.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.203+0000 7f7c6aa0b700 1 -- 172.21.15.189:0/2226995234 shutdown_connections 2024-04-06T12:30:08.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.203+0000 7f7c6aa0b700 1 -- 172.21.15.189:0/2226995234 wait complete. 2024-04-06T12:30:08.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.204+0000 7f7c6aa0b700 1 Processor -- start 2024-04-06T12:30:08.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.204+0000 7f7c6aa0b700 1 -- start start 2024-04-06T12:30:08.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.205+0000 7f7c6aa0b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c6406a730 0x7f7c6413f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:08.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.205+0000 7f7c6aa0b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c6415d7b0 0x7f7c641511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:08.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.205+0000 7f7c6aa0b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c6415e500 0x7f7c641516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:08.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.205+0000 7f7c6aa0b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c64142490 con 0x7f7c6406a730 2024-04-06T12:30:08.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.205+0000 7f7c6aa0b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c64142770 con 0x7f7c6415e500 2024-04-06T12:30:08.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.205+0000 7f7c6aa0b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c64142a50 con 0x7f7c6415d7b0 2024-04-06T12:30:08.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.205+0000 7f7c63fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c6406a730 0x7f7c6413f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:08.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.205+0000 7f7c63fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c6406a730 0x7f7c6413f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58434/0 (socket says 172.21.15.189:58434) 2024-04-06T12:30:08.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.205+0000 7f7c63fff700 1 -- 172.21.15.189:0/1626332733 learned_addr learned my addr 172.21.15.189:0/1626332733 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:08.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.205+0000 7f7c637fe700 1 --2- 172.21.15.189:0/1626332733 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c6415d7b0 0x7f7c641511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:08.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.205+0000 7f7c68fa8700 1 --2- 172.21.15.189:0/1626332733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c6415e500 0x7f7c641516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:08.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.205+0000 7f7c63fff700 1 -- 172.21.15.189:0/1626332733 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c6415d7b0 msgr2=0x7f7c641511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:08.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.205+0000 7f7c63fff700 1 --2- 172.21.15.189:0/1626332733 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c6415d7b0 0x7f7c641511b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.205+0000 7f7c63fff700 1 -- 172.21.15.189:0/1626332733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c6415e500 msgr2=0x7f7c641516f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:08.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.205+0000 7f7c63fff700 1 --2- 172.21.15.189:0/1626332733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c6415e500 0x7f7c641516f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.205+0000 7f7c63fff700 1 -- 172.21.15.189:0/1626332733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c54009550 con 0x7f7c6406a730 2024-04-06T12:30:08.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.205+0000 7f7c63fff700 1 --2- 172.21.15.189:0/1626332733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c6406a730 0x7f7c6413f830 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f7c6406c820 tx=0x7f7c54004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:08.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.205+0000 7f7c60ff9700 1 -- 172.21.15.189:0/1626332733 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c54003e70 con 0x7f7c6406a730 2024-04-06T12:30:08.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.206+0000 7f7c60ff9700 1 -- 172.21.15.189:0/1626332733 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5401f070 con 0x7f7c6406a730 2024-04-06T12:30:08.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.206+0000 7f7c6aa0b700 1 -- 172.21.15.189:0/1626332733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c64142d30 con 0x7f7c6406a730 2024-04-06T12:30:08.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.206+0000 7f7c60ff9700 1 -- 172.21.15.189:0/1626332733 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5400b8c0 con 0x7f7c6406a730 2024-04-06T12:30:08.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.206+0000 7f7c6aa0b700 1 -- 172.21.15.189:0/1626332733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c64143010 con 0x7f7c6406a730 2024-04-06T12:30:08.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.207+0000 7f7c6aa0b700 1 -- 172.21.15.189:0/1626332733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c64067bb0 con 0x7f7c6406a730 2024-04-06T12:30:08.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.207+0000 7f7c60ff9700 1 -- 172.21.15.189:0/1626332733 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7c54021030 con 0x7f7c6406a730 2024-04-06T12:30:08.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.207+0000 7f7c60ff9700 1 --2- 172.21.15.189:0/1626332733 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7c5006adb0 0x7f7c5006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:08.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.208+0000 7f7c60ff9700 1 -- 172.21.15.189:0/1626332733 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f7c5408ac40 con 0x7f7c6406a730 2024-04-06T12:30:08.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.208+0000 7f7c637fe700 1 --2- 172.21.15.189:0/1626332733 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7c5006adb0 0x7f7c5006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:08.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.208+0000 7f7c637fe700 1 --2- 172.21.15.189:0/1626332733 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7c5006adb0 0x7f7c5006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7c64150760 tx=0x7f7c58009040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:08.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.212+0000 7f7c60ff9700 1 -- 172.21.15.189:0/1626332733 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7c54055dd0 con 0x7f7c6406a730 2024-04-06T12:30:08.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.405+0000 7f7c6aa0b700 1 -- 172.21.15.189:0/1626332733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f7c64067bb0 con 0x7f7c6406a730 2024-04-06T12:30:08.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.409+0000 7f7c60ff9700 1 -- 172.21.15.189:0/1626332733 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v38) v1 ==== 115+0+0 (secure 0 0 0) 0x7f7c540593f0 con 0x7f7c6406a730 2024-04-06T12:30:08.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.411+0000 7f7c6aa0b700 1 -- 172.21.15.189:0/1626332733 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7c5006adb0 msgr2=0x7f7c5006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:08.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.411+0000 7f7c6aa0b700 1 --2- 172.21.15.189:0/1626332733 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7c5006adb0 0x7f7c5006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7c64150760 tx=0x7f7c58009040 comp rx=0 tx=0).stop 2024-04-06T12:30:08.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.411+0000 7f7c6aa0b700 1 -- 172.21.15.189:0/1626332733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c6406a730 msgr2=0x7f7c6413f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:08.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.411+0000 7f7c6aa0b700 1 --2- 172.21.15.189:0/1626332733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c6406a730 0x7f7c6413f830 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f7c6406c820 tx=0x7f7c54004750 comp rx=0 tx=0).stop 2024-04-06T12:30:08.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.412+0000 7f7c6aa0b700 1 -- 172.21.15.189:0/1626332733 shutdown_connections 2024-04-06T12:30:08.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.412+0000 7f7c6aa0b700 1 --2- 172.21.15.189:0/1626332733 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7c5006adb0 0x7f7c5006d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.412+0000 7f7c6aa0b700 1 --2- 172.21.15.189:0/1626332733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c6415e500 0x7f7c641516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.412+0000 7f7c6aa0b700 1 --2- 172.21.15.189:0/1626332733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c6406a730 0x7f7c6413f830 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.412+0000 7f7c6aa0b700 1 --2- 172.21.15.189:0/1626332733 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c6415d7b0 0x7f7c641511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.412+0000 7f7c6aa0b700 1 -- 172.21.15.189:0/1626332733 >> 172.21.15.189:0/1626332733 conn(0x7f7c640d00b0 msgr2=0x7f7c640ad850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:08.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.412+0000 7f7c6aa0b700 1 -- 172.21.15.189:0/1626332733 shutdown_connections 2024-04-06T12:30:08.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.412+0000 7f7c6aa0b700 1 -- 172.21.15.189:0/1626332733 wait complete. 2024-04-06T12:30:08.421 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-06T12:30:08.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.549+0000 7f7573e61700 1 Processor -- start 2024-04-06T12:30:08.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.549+0000 7f7573e61700 1 -- start start 2024-04-06T12:30:08.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.550+0000 7f7573e61700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f756c0b2ca0 0x7f756c0b3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:08.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.550+0000 7f7573e61700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f756c15ea40 0x7f756c0cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:08.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.550+0000 7f7573e61700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f756c0cf2f0 0x7f756c06bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:08.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.550+0000 7f7573e61700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f756c15dd70 con 0x7f756c0b2ca0 2024-04-06T12:30:08.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.550+0000 7f7573e61700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f756c0cfa50 con 0x7f756c0cf2f0 2024-04-06T12:30:08.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.550+0000 7f7573e61700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756c0b35c0 con 0x7f756c15ea40 2024-04-06T12:30:08.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.550+0000 7f7571bfd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f756c0b2ca0 0x7f756c0b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:08.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.550+0000 7f7571bfd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f756c0b2ca0 0x7f756c0b3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58450/0 (socket says 172.21.15.189:58450) 2024-04-06T12:30:08.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.550+0000 7f7571bfd700 1 -- 172.21.15.189:0/1646729935 learned_addr learned my addr 172.21.15.189:0/1646729935 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:08.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.550+0000 7f75723fe700 1 --2- 172.21.15.189:0/1646729935 >> v2:172.21.15.192:3300/0 conn(0x7f756c0cf2f0 0x7f756c06bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:08.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.550+0000 7f75713fc700 1 --2- 172.21.15.189:0/1646729935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f756c15ea40 0x7f756c0cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:08.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.550+0000 7f7571bfd700 1 -- 172.21.15.189:0/1646729935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f756c15ea40 msgr2=0x7f756c0cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:08.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.550+0000 7f7571bfd700 1 --2- 172.21.15.189:0/1646729935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f756c15ea40 0x7f756c0cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.550+0000 7f7571bfd700 1 -- 172.21.15.189:0/1646729935 >> v2:172.21.15.192:3300/0 conn(0x7f756c0cf2f0 msgr2=0x7f756c06bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:08.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.550+0000 7f7571bfd700 1 --2- 172.21.15.189:0/1646729935 >> v2:172.21.15.192:3300/0 conn(0x7f756c0cf2f0 0x7f756c06bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.551+0000 7f7571bfd700 1 -- 172.21.15.189:0/1646729935 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756c0cf770 con 0x7f756c0b2ca0 2024-04-06T12:30:08.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.551+0000 7f7571bfd700 1 --2- 172.21.15.189:0/1646729935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f756c0b2ca0 0x7f756c0b3080 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f756c15eea0 tx=0x7f755c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee52733714d7333 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:08.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.551+0000 7f7570bfb700 1 -- 172.21.15.189:0/1646729935 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755c00b870 con 0x7f756c0b2ca0 2024-04-06T12:30:08.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.551+0000 7f7570bfb700 1 -- 172.21.15.189:0/1646729935 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f755c00b9d0 con 0x7f756c0b2ca0 2024-04-06T12:30:08.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.551+0000 7f7570bfb700 1 -- 172.21.15.189:0/1646729935 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755c00bd60 con 0x7f756c0b2ca0 2024-04-06T12:30:08.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.551+0000 7f7573e61700 1 -- 172.21.15.189:0/1646729935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f756c0b2ca0 msgr2=0x7f756c0b3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:08.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.551+0000 7f7573e61700 1 --2- 172.21.15.189:0/1646729935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f756c0b2ca0 0x7f756c0b3080 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f756c15eea0 tx=0x7f755c009bd0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.551+0000 7f7573e61700 1 -- 172.21.15.189:0/1646729935 shutdown_connections 2024-04-06T12:30:08.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.551+0000 7f7573e61700 1 --2- 172.21.15.189:0/1646729935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f756c0b2ca0 0x7f756c0b3080 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.551+0000 7f7573e61700 1 --2- 172.21.15.189:0/1646729935 >> v2:172.21.15.192:3300/0 conn(0x7f756c0cf2f0 0x7f756c06bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.551+0000 7f7573e61700 1 --2- 172.21.15.189:0/1646729935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f756c15ea40 0x7f756c0cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.551+0000 7f7573e61700 1 -- 172.21.15.189:0/1646729935 >> 172.21.15.189:0/1646729935 conn(0x7f756c06eaa0 msgr2=0x7f756c06ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:08.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.551+0000 7f7573e61700 1 -- 172.21.15.189:0/1646729935 shutdown_connections 2024-04-06T12:30:08.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.552+0000 7f7573e61700 1 -- 172.21.15.189:0/1646729935 wait complete. 2024-04-06T12:30:08.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.553+0000 7f7573e61700 1 Processor -- start 2024-04-06T12:30:08.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.553+0000 7f7573e61700 1 -- start start 2024-04-06T12:30:08.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.553+0000 7f7573e61700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f756c0b2ca0 0x7f756c1f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:08.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.553+0000 7f7573e61700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f756c0cf2f0 0x7f756c0c6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:08.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.553+0000 7f7573e61700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f756c15ea40 0x7f756c0c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:08.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.553+0000 7f7573e61700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756c0c7430 con 0x7f756c0b2ca0 2024-04-06T12:30:08.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.553+0000 7f7573e61700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756c1f6490 con 0x7f756c15ea40 2024-04-06T12:30:08.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.553+0000 7f7573e61700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756c0b5cf0 con 0x7f756c0cf2f0 2024-04-06T12:30:08.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.554+0000 7f7571bfd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f756c0b2ca0 0x7f756c1f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:08.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.554+0000 7f7571bfd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f756c0b2ca0 0x7f756c1f5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58460/0 (socket says 172.21.15.189:58460) 2024-04-06T12:30:08.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.554+0000 7f7571bfd700 1 -- 172.21.15.189:0/1272750164 learned_addr learned my addr 172.21.15.189:0/1272750164 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:08.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.554+0000 7f75723fe700 1 --2- 172.21.15.189:0/1272750164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f756c15ea40 0x7f756c0c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:08.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.554+0000 7f75713fc700 1 --2- 172.21.15.189:0/1272750164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f756c0cf2f0 0x7f756c0c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:08.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.554+0000 7f7571bfd700 1 -- 172.21.15.189:0/1272750164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f756c0cf2f0 msgr2=0x7f756c0c6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:08.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.554+0000 7f7571bfd700 1 --2- 172.21.15.189:0/1272750164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f756c0cf2f0 0x7f756c0c6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.554+0000 7f7571bfd700 1 -- 172.21.15.189:0/1272750164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f756c15ea40 msgr2=0x7f756c0c6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:08.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.554+0000 7f7571bfd700 1 --2- 172.21.15.189:0/1272750164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f756c15ea40 0x7f756c0c6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.554+0000 7f7571bfd700 1 -- 172.21.15.189:0/1272750164 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c009550 con 0x7f756c0b2ca0 2024-04-06T12:30:08.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.554+0000 7f7571bfd700 1 --2- 172.21.15.189:0/1272750164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f756c0b2ca0 0x7f756c1f5f50 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f755c009c10 tx=0x7f755c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:08.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.554+0000 7f75627fc700 1 -- 172.21.15.189:0/1272750164 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755c004250 con 0x7f756c0b2ca0 2024-04-06T12:30:08.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.554+0000 7f75627fc700 1 -- 172.21.15.189:0/1272750164 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f755c01f070 con 0x7f756c0b2ca0 2024-04-06T12:30:08.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.554+0000 7f7573e61700 1 -- 172.21.15.189:0/1272750164 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f756c0b5fd0 con 0x7f756c0b2ca0 2024-04-06T12:30:08.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.554+0000 7f75627fc700 1 -- 172.21.15.189:0/1272750164 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755c0045d0 con 0x7f756c0b2ca0 2024-04-06T12:30:08.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.554+0000 7f7573e61700 1 -- 172.21.15.189:0/1272750164 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f756c0b6590 con 0x7f756c0b2ca0 2024-04-06T12:30:08.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.555+0000 7f7573e61700 1 -- 172.21.15.189:0/1272750164 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f756c0d1fc0 con 0x7f756c0b2ca0 2024-04-06T12:30:08.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.556+0000 7f75627fc700 1 -- 172.21.15.189:0/1272750164 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f755c021030 con 0x7f756c0b2ca0 2024-04-06T12:30:08.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.556+0000 7f75627fc700 1 --2- 172.21.15.189:0/1272750164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f755006adb0 0x7f755006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:08.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.556+0000 7f75627fc700 1 -- 172.21.15.189:0/1272750164 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f755c08a850 con 0x7f756c0b2ca0 2024-04-06T12:30:08.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.556+0000 7f75713fc700 1 --2- 172.21.15.189:0/1272750164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f755006adb0 0x7f755006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:08.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.557+0000 7f75713fc700 1 --2- 172.21.15.189:0/1272750164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f755006adb0 0x7f755006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f756c157580 tx=0x7f7568008040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:08.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.560+0000 7f75627fc700 1 -- 172.21.15.189:0/1272750164 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f755c055840 con 0x7f756c0b2ca0 2024-04-06T12:30:08.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.757+0000 7f7573e61700 1 -- 172.21.15.189:0/1272750164 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f756c074990 con 0x7f756c0b2ca0 2024-04-06T12:30:08.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.761+0000 7f75627fc700 1 -- 172.21.15.189:0/1272750164 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v39) v1 ==== 120+0+0 (secure 0 0 0) 0x7f755c058e60 con 0x7f756c0b2ca0 2024-04-06T12:30:08.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.763+0000 7f7573e61700 1 -- 172.21.15.189:0/1272750164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f755006adb0 msgr2=0x7f755006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:08.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.763+0000 7f7573e61700 1 --2- 172.21.15.189:0/1272750164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f755006adb0 0x7f755006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f756c157580 tx=0x7f7568008040 comp rx=0 tx=0).stop 2024-04-06T12:30:08.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.763+0000 7f7573e61700 1 -- 172.21.15.189:0/1272750164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f756c0b2ca0 msgr2=0x7f756c1f5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:08.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.763+0000 7f7573e61700 1 --2- 172.21.15.189:0/1272750164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f756c0b2ca0 0x7f756c1f5f50 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f755c009c10 tx=0x7f755c004750 comp rx=0 tx=0).stop 2024-04-06T12:30:08.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.763+0000 7f7573e61700 1 -- 172.21.15.189:0/1272750164 shutdown_connections 2024-04-06T12:30:08.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.763+0000 7f7573e61700 1 --2- 172.21.15.189:0/1272750164 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f755006adb0 0x7f755006d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.763+0000 7f7573e61700 1 --2- 172.21.15.189:0/1272750164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f756c15ea40 0x7f756c0c6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.763+0000 7f7573e61700 1 --2- 172.21.15.189:0/1272750164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f756c0b2ca0 0x7f756c1f5f50 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.763+0000 7f7573e61700 1 --2- 172.21.15.189:0/1272750164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f756c0cf2f0 0x7f756c0c6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.763+0000 7f7573e61700 1 -- 172.21.15.189:0/1272750164 >> 172.21.15.189:0/1272750164 conn(0x7f756c06eaa0 msgr2=0x7f756c15d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:08.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.763+0000 7f7573e61700 1 -- 172.21.15.189:0/1272750164 shutdown_connections 2024-04-06T12:30:08.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.763+0000 7f7573e61700 1 -- 172.21.15.189:0/1272750164 wait complete. 2024-04-06T12:30:08.773 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2024-04-06T12:30:08.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.899+0000 7f45a06a1700 1 Processor -- start 2024-04-06T12:30:08.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.899+0000 7f45a06a1700 1 -- start start 2024-04-06T12:30:08.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.900+0000 7f45a06a1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f45980cd3e0 0x7f45980cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:08.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.900+0000 7f45a06a1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f459815d7c0 0x7f459806b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:08.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.900+0000 7f45a06a1700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459806ceb0 con 0x7f459806b6f0 2024-04-06T12:30:08.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.900+0000 7f45a06a1700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45980cdfa0 con 0x7f45980cd3e0 2024-04-06T12:30:08.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.900+0000 7f45a06a1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459806d750 con 0x7f459815d7c0 2024-04-06T12:30:08.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.900+0000 7f459e43d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f45980cd3e0 0x7f45980cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:08.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.900+0000 7f459e43d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f45980cd3e0 0x7f45980cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58466/0 (socket says 172.21.15.189:58466) 2024-04-06T12:30:08.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.900+0000 7f459e43d700 1 -- 172.21.15.189:0/612880254 learned_addr learned my addr 172.21.15.189:0/612880254 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:08.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.900+0000 7f459dc3c700 1 --2- 172.21.15.189:0/612880254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f459815d7c0 0x7f459806b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:08.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.901+0000 7f459e43d700 1 -- 172.21.15.189:0/612880254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f459815d7c0 msgr2=0x7f459806b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:08.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.901+0000 7f459e43d700 1 --2- 172.21.15.189:0/612880254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f459815d7c0 0x7f459806b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.901+0000 7f459e43d700 1 -- 172.21.15.189:0/612880254 >> v1:172.21.15.192:6789/0 conn(0x7f459806b6f0 legacy=0x7f45981610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:08.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.901+0000 7f459e43d700 1 -- 172.21.15.189:0/612880254 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45980cdd00 con 0x7f45980cd3e0 2024-04-06T12:30:08.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.901+0000 7f459dc3c700 1 --2- 172.21.15.189:0/612880254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f459815d7c0 0x7f459806b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:08.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.901+0000 7f459e43d700 1 --2- 172.21.15.189:0/612880254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45980cd3e0 0x7f45980cd7c0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f45880098f0 tx=0x7f4588009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9d91ae3118c7f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:08.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.901+0000 7f459d43b700 1 -- 172.21.15.189:0/612880254 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458800b8a0 con 0x7f45980cd3e0 2024-04-06T12:30:08.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.901+0000 7f459d43b700 1 -- 172.21.15.189:0/612880254 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458800ba00 con 0x7f45980cd3e0 2024-04-06T12:30:08.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.901+0000 7f459d43b700 1 -- 172.21.15.189:0/612880254 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458800bd90 con 0x7f45980cd3e0 2024-04-06T12:30:08.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.901+0000 7f45a06a1700 1 -- 172.21.15.189:0/612880254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45980cd3e0 msgr2=0x7f45980cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:08.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.901+0000 7f45a06a1700 1 --2- 172.21.15.189:0/612880254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45980cd3e0 0x7f45980cd7c0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f45880098f0 tx=0x7f4588009c00 comp rx=0 tx=0).stop 2024-04-06T12:30:08.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.902+0000 7f45a06a1700 1 -- 172.21.15.189:0/612880254 shutdown_connections 2024-04-06T12:30:08.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.902+0000 7f45a06a1700 1 --2- 172.21.15.189:0/612880254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45980cd3e0 0x7f45980cd7c0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.902+0000 7f45a06a1700 1 --2- 172.21.15.189:0/612880254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f459815d7c0 0x7f459806b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.902+0000 7f45a06a1700 1 -- 172.21.15.189:0/612880254 >> 172.21.15.189:0/612880254 conn(0x7f45980cc290 msgr2=0x7f45980cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:08.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.902+0000 7f45a06a1700 1 -- 172.21.15.189:0/612880254 shutdown_connections 2024-04-06T12:30:08.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.902+0000 7f45a06a1700 1 -- 172.21.15.189:0/612880254 wait complete. 2024-04-06T12:30:08.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.903+0000 7f45a06a1700 1 Processor -- start 2024-04-06T12:30:08.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.903+0000 7f45a06a1700 1 -- start start 2024-04-06T12:30:08.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.903+0000 7f45a06a1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f459806b6f0 0x7f45981f6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:08.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.903+0000 7f45a06a1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45980cd3e0 0x7f45980b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:08.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.903+0000 7f45a06a1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f459815d7c0 0x7f4598156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:08.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.903+0000 7f45a06a1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45981f5e80 con 0x7f459806b6f0 2024-04-06T12:30:08.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.904+0000 7f45a06a1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459815ceb0 con 0x7f459815d7c0 2024-04-06T12:30:08.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.904+0000 7f45a06a1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4598153ae0 con 0x7f45980cd3e0 2024-04-06T12:30:08.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.904+0000 7f459e43d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f459806b6f0 0x7f45981f6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:08.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.904+0000 7f459e43d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f459806b6f0 0x7f45981f6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58478/0 (socket says 172.21.15.189:58478) 2024-04-06T12:30:08.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.904+0000 7f459e43d700 1 -- 172.21.15.189:0/2699412434 learned_addr learned my addr 172.21.15.189:0/2699412434 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:08.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.904+0000 7f459dc3c700 1 --2- 172.21.15.189:0/2699412434 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45980cd3e0 0x7f45980b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:08.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.904+0000 7f459ec3e700 1 --2- 172.21.15.189:0/2699412434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f459815d7c0 0x7f4598156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:08.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.904+0000 7f459e43d700 1 -- 172.21.15.189:0/2699412434 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45980cd3e0 msgr2=0x7f45980b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:08.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.904+0000 7f459e43d700 1 --2- 172.21.15.189:0/2699412434 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45980cd3e0 0x7f45980b4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.904+0000 7f459e43d700 1 -- 172.21.15.189:0/2699412434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f459815d7c0 msgr2=0x7f4598156330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:08.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.904+0000 7f459e43d700 1 --2- 172.21.15.189:0/2699412434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f459815d7c0 0x7f4598156330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:08.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.904+0000 7f459e43d700 1 -- 172.21.15.189:0/2699412434 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4588009550 con 0x7f459806b6f0 2024-04-06T12:30:08.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.904+0000 7f459ec3e700 1 --2- 172.21.15.189:0/2699412434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f459815d7c0 0x7f4598156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:08.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.904+0000 7f459e43d700 1 --2- 172.21.15.189:0/2699412434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f459806b6f0 0x7f45981f6940 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f4588009c40 tx=0x7f4588004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:08.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.904+0000 7f458effd700 1 -- 172.21.15.189:0/2699412434 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4588004590 con 0x7f459806b6f0 2024-04-06T12:30:08.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.904+0000 7f458effd700 1 -- 172.21.15.189:0/2699412434 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458801e070 con 0x7f459806b6f0 2024-04-06T12:30:08.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.904+0000 7f45a06a1700 1 -- 172.21.15.189:0/2699412434 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4598153dc0 con 0x7f459806b6f0 2024-04-06T12:30:08.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.904+0000 7f458effd700 1 -- 172.21.15.189:0/2699412434 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45880038e0 con 0x7f459806b6f0 2024-04-06T12:30:08.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.904+0000 7f45a06a1700 1 -- 172.21.15.189:0/2699412434 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45981f6420 con 0x7f459806b6f0 2024-04-06T12:30:08.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.905+0000 7f45a06a1700 1 -- 172.21.15.189:0/2699412434 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45981f6610 con 0x7f459806b6f0 2024-04-06T12:30:08.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.906+0000 7f458effd700 1 -- 172.21.15.189:0/2699412434 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f4588003a40 con 0x7f459806b6f0 2024-04-06T12:30:08.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.906+0000 7f458effd700 1 --2- 172.21.15.189:0/2699412434 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f457c06adb0 0x7f457c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:08.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.906+0000 7f458effd700 1 -- 172.21.15.189:0/2699412434 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f4588020070 con 0x7f459806b6f0 2024-04-06T12:30:08.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.906+0000 7f459dc3c700 1 --2- 172.21.15.189:0/2699412434 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f457c06adb0 0x7f457c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:08.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.906+0000 7f459dc3c700 1 --2- 172.21.15.189:0/2699412434 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f457c06adb0 0x7f457c06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f45980c5f40 tx=0x7f4594008040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:08.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:08.910+0000 7f458effd700 1 -- 172.21.15.189:0/2699412434 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4588055ac0 con 0x7f459806b6f0 2024-04-06T12:30:09.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.186+0000 7f45a06a1700 1 -- 172.21.15.189:0/2699412434 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f4598074990 con 0x7f459806b6f0 2024-04-06T12:30:09.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.699+0000 7f458effd700 1 -- 172.21.15.189:0/2699412434 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v682) v1 ==== 183+0+0 (secure 0 0 0) 0x7f45880590e0 con 0x7f459806b6f0 2024-04-06T12:30:09.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.701+0000 7f45a06a1700 1 -- 172.21.15.189:0/2699412434 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f457c06adb0 msgr2=0x7f457c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:09.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.701+0000 7f45a06a1700 1 --2- 172.21.15.189:0/2699412434 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f457c06adb0 0x7f457c06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f45980c5f40 tx=0x7f4594008040 comp rx=0 tx=0).stop 2024-04-06T12:30:09.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.701+0000 7f45a06a1700 1 -- 172.21.15.189:0/2699412434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f459806b6f0 msgr2=0x7f45981f6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:09.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.701+0000 7f45a06a1700 1 --2- 172.21.15.189:0/2699412434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f459806b6f0 0x7f45981f6940 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f4588009c40 tx=0x7f4588004750 comp rx=0 tx=0).stop 2024-04-06T12:30:09.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.702+0000 7f45a06a1700 1 -- 172.21.15.189:0/2699412434 shutdown_connections 2024-04-06T12:30:09.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.702+0000 7f45a06a1700 1 --2- 172.21.15.189:0/2699412434 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f457c06adb0 0x7f457c06d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:09.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.702+0000 7f45a06a1700 1 --2- 172.21.15.189:0/2699412434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f459815d7c0 0x7f4598156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:09.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.702+0000 7f45a06a1700 1 --2- 172.21.15.189:0/2699412434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f459806b6f0 0x7f45981f6940 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:09.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.702+0000 7f45a06a1700 1 --2- 172.21.15.189:0/2699412434 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f45980cd3e0 0x7f45980b4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:09.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.702+0000 7f45a06a1700 1 -- 172.21.15.189:0/2699412434 >> 172.21.15.189:0/2699412434 conn(0x7f45980cc290 msgr2=0x7f45980afc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:09.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.702+0000 7f45a06a1700 1 -- 172.21.15.189:0/2699412434 shutdown_connections 2024-04-06T12:30:09.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.702+0000 7f45a06a1700 1 -- 172.21.15.189:0/2699412434 wait complete. 2024-04-06T12:30:09.712 INFO:tasks.cephfs_test_runner:Test that a non-existent path leads to schedule deactivation after a few retries. ... ok 2024-04-06T12:30:09.712 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2024-04-06T12:30:09.712 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-06T12:30:09.712 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2024-04-06T12:30:09.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.839+0000 7fcea8145700 1 Processor -- start 2024-04-06T12:30:09.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.839+0000 7fcea8145700 1 -- start start 2024-04-06T12:30:09.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.840+0000 7fcea8145700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcea0158fe0 0x7fcea006a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:09.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.840+0000 7fcea8145700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea015f9e0 con 0x7fcea006ac70 2024-04-06T12:30:09.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.840+0000 7fcea8145700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea015f0d0 con 0x7fcea015c2d0 2024-04-06T12:30:09.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.840+0000 7fcea8145700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea006b0f0 con 0x7fcea0158fe0 2024-04-06T12:30:09.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.840+0000 7fcea56e0700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcea006ac70 0x7fcea015bbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51132/0 (socket says 172.21.15.189:51132) 2024-04-06T12:30:09.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.840+0000 7fcea56e0700 1 -- 172.21.15.189:0/873422161 learned_addr learned my addr 172.21.15.189:0/873422161 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:09.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.840+0000 7fcea4edf700 1 -- 172.21.15.189:0/873422161 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158078270 0 0) 0x7fcea015f9e0 con 0x7fcea006ac70 2024-04-06T12:30:09.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.840+0000 7fcea4edf700 1 -- 172.21.15.189:0/873422161 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce8c0035b0 con 0x7fcea006ac70 2024-04-06T12:30:09.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.840+0000 7fcea5ee1700 1 --2- 172.21.15.189:0/873422161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcea0158fe0 0x7fcea006a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:09.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea4edf700 1 -- 172.21.15.189:0/873422161 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2130233326 0 0) 0x7fce8c0035b0 con 0x7fcea006ac70 2024-04-06T12:30:09.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea4edf700 1 -- 172.21.15.189:0/873422161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcea0158fe0 msgr2=0x7fcea006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:09.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea4edf700 1 --2- 172.21.15.189:0/873422161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcea0158fe0 0x7fcea006a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:09.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea4edf700 1 -- 172.21.15.189:0/873422161 >> v1:172.21.15.192:6789/0 conn(0x7fcea015c2d0 legacy=0x7fcea00ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:09.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea4edf700 1 -- 172.21.15.189:0/873422161 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea015c8f0 con 0x7fcea006ac70 2024-04-06T12:30:09.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea4edf700 1 -- 172.21.15.189:0/873422161 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fce90001670 con 0x7fcea006ac70 2024-04-06T12:30:09.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea4edf700 1 -- 172.21.15.189:0/873422161 >> v1:172.21.15.189:6789/0 conn(0x7fcea006ac70 legacy=0x7fcea015bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:09.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea4edf700 1 --2- 172.21.15.189:0/873422161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce8c003a30 0x7fce8c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:09.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea4edf700 1 --2- 172.21.15.189:0/873422161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce8c004850 0x7fce8c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:09.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea4edf700 1 --2- 172.21.15.189:0/873422161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcea0158fe0 0x7fce8c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:09.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea4edf700 1 -- 172.21.15.189:0/873422161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea006b0f0 con 0x7fce8c003a30 2024-04-06T12:30:09.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea4edf700 1 -- 172.21.15.189:0/873422161 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c005540 con 0x7fce8c004850 2024-04-06T12:30:09.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea4edf700 1 -- 172.21.15.189:0/873422161 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c005700 con 0x7fcea0158fe0 2024-04-06T12:30:09.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea5ee1700 1 --2- 172.21.15.189:0/873422161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce8c003a30 0x7fce8c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:09.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea66e2700 1 --2- 172.21.15.189:0/873422161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcea0158fe0 0x7fce8c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:09.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea56e0700 1 --2- 172.21.15.189:0/873422161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce8c004850 0x7fce8c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:09.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea5ee1700 1 -- 172.21.15.189:0/873422161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcea0158fe0 msgr2=0x7fce8c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:09.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea5ee1700 1 --2- 172.21.15.189:0/873422161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcea0158fe0 0x7fce8c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:09.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea5ee1700 1 -- 172.21.15.189:0/873422161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce8c004850 msgr2=0x7fce8c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:09.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea5ee1700 1 --2- 172.21.15.189:0/873422161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce8c004850 0x7fce8c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:09.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea5ee1700 1 -- 172.21.15.189:0/873422161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8c0035b0 con 0x7fce8c003a30 2024-04-06T12:30:09.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea56e0700 1 --2- 172.21.15.189:0/873422161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce8c004850 0x7fce8c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:09.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea5ee1700 1 --2- 172.21.15.189:0/873422161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce8c003a30 0x7fce8c003e50 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fce9c009890 tx=0x7fce8c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74392130d4abbaa server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:09.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea4edf700 1 -- 172.21.15.189:0/873422161 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c00c070 con 0x7fce8c003a30 2024-04-06T12:30:09.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea4edf700 1 -- 172.21.15.189:0/873422161 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9c013610 con 0x7fce8c003a30 2024-04-06T12:30:09.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.841+0000 7fcea4edf700 1 -- 172.21.15.189:0/873422161 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c0139a0 con 0x7fce8c003a30 2024-04-06T12:30:09.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.842+0000 7fcea8145700 1 -- 172.21.15.189:0/873422161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce8c003a30 msgr2=0x7fce8c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:09.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.842+0000 7fcea8145700 1 --2- 172.21.15.189:0/873422161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce8c003a30 0x7fce8c003e50 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fce9c009890 tx=0x7fce8c005880 comp rx=0 tx=0).stop 2024-04-06T12:30:09.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.842+0000 7fcea66e2700 1 -- 172.21.15.189:0/873422161 reap_dead start 2024-04-06T12:30:09.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.842+0000 7fcea8145700 1 -- 172.21.15.189:0/873422161 shutdown_connections 2024-04-06T12:30:09.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.842+0000 7fcea8145700 1 -- 172.21.15.189:0/873422161 >> 172.21.15.189:0/873422161 conn(0x7fcea00d00b0 msgr2=0x7fcea00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:09.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.842+0000 7fcea8145700 1 -- 172.21.15.189:0/873422161 shutdown_connections 2024-04-06T12:30:09.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.842+0000 7fcea8145700 1 -- 172.21.15.189:0/873422161 wait complete. 2024-04-06T12:30:09.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.843+0000 7fcea8145700 1 Processor -- start 2024-04-06T12:30:09.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.843+0000 7fcea8145700 1 -- start start 2024-04-06T12:30:09.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.844+0000 7fcea8145700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce8c003a30 0x7fcea013eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:09.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.844+0000 7fcea8145700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcea013db10 0x7fcea01422d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:09.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.844+0000 7fcea8145700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcea0139870 0x7fcea0139cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:09.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.844+0000 7fcea8145700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea013a230 con 0x7fce8c003a30 2024-04-06T12:30:09.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.844+0000 7fcea8145700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea0142c40 con 0x7fcea013db10 2024-04-06T12:30:09.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.844+0000 7fcea8145700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea0142f20 con 0x7fcea0139870 2024-04-06T12:30:09.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.844+0000 7fcea5ee1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce8c003a30 0x7fcea013eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:09.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.844+0000 7fcea5ee1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce8c003a30 0x7fcea013eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49562/0 (socket says 172.21.15.189:49562) 2024-04-06T12:30:09.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.844+0000 7fcea5ee1700 1 -- 172.21.15.189:0/1806152409 learned_addr learned my addr 172.21.15.189:0/1806152409 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:09.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.844+0000 7fcea66e2700 1 --2- 172.21.15.189:0/1806152409 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcea0139870 0x7fcea0139cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:09.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.844+0000 7fcea56e0700 1 --2- 172.21.15.189:0/1806152409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcea013db10 0x7fcea01422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:09.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.844+0000 7fcea5ee1700 1 -- 172.21.15.189:0/1806152409 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcea0139870 msgr2=0x7fcea0139cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:09.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.844+0000 7fcea5ee1700 1 --2- 172.21.15.189:0/1806152409 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcea0139870 0x7fcea0139cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:09.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.844+0000 7fcea5ee1700 1 -- 172.21.15.189:0/1806152409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcea013db10 msgr2=0x7fcea01422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:09.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.844+0000 7fcea5ee1700 1 --2- 172.21.15.189:0/1806152409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcea013db10 0x7fcea01422d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:09.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.844+0000 7fcea5ee1700 1 -- 172.21.15.189:0/1806152409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea01596c0 con 0x7fce8c003a30 2024-04-06T12:30:09.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.845+0000 7fcea5ee1700 1 --2- 172.21.15.189:0/1806152409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce8c003a30 0x7fcea013eb80 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fce9c016040 tx=0x7fce9c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:09.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.845+0000 7fce967fc700 1 -- 172.21.15.189:0/1806152409 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c013990 con 0x7fce8c003a30 2024-04-06T12:30:09.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.845+0000 7fce967fc700 1 -- 172.21.15.189:0/1806152409 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9c013b10 con 0x7fce8c003a30 2024-04-06T12:30:09.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.845+0000 7fcea8145700 1 -- 172.21.15.189:0/1806152409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea01599a0 con 0x7fce8c003a30 2024-04-06T12:30:09.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.845+0000 7fce967fc700 1 -- 172.21.15.189:0/1806152409 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c0036a0 con 0x7fce8c003a30 2024-04-06T12:30:09.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.845+0000 7fcea8145700 1 -- 172.21.15.189:0/1806152409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea013f2a0 con 0x7fce8c003a30 2024-04-06T12:30:09.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.846+0000 7fcea8145700 1 -- 172.21.15.189:0/1806152409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea0159d10 con 0x7fce8c003a30 2024-04-06T12:30:09.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.846+0000 7fce967fc700 1 -- 172.21.15.189:0/1806152409 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fce9c003800 con 0x7fce8c003a30 2024-04-06T12:30:09.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.846+0000 7fce967fc700 1 --2- 172.21.15.189:0/1806152409 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fce8406adb0 0x7fce8406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:09.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.846+0000 7fce967fc700 1 -- 172.21.15.189:0/1806152409 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5951+0+0 (secure 0 0 0) 0x7fce9c089df0 con 0x7fce8c003a30 2024-04-06T12:30:09.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.847+0000 7fcea56e0700 1 --2- 172.21.15.189:0/1806152409 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fce8406adb0 0x7fce8406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:09.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.847+0000 7fcea56e0700 1 --2- 172.21.15.189:0/1806152409 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fce8406adb0 0x7fce8406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcea013def0 tx=0x7fce90009040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:09.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:09.851+0000 7fce967fc700 1 -- 172.21.15.189:0/1806152409 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fce9c054d50 con 0x7fce8c003a30 2024-04-06T12:30:10.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.124+0000 7fcea8145700 1 -- 172.21.15.189:0/1806152409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7fcea013f2a0 con 0x7fce8c003a30 2024-04-06T12:30:10.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.701+0000 7fce967fc700 1 -- 172.21.15.189:0/1806152409 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v683) v1 ==== 151+0+0 (secure 0 0 0) 0x7fcea013f2a0 con 0x7fce8c003a30 2024-04-06T12:30:10.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.704+0000 7fcea8145700 1 -- 172.21.15.189:0/1806152409 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fce8406adb0 msgr2=0x7fce8406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:10.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.704+0000 7fcea8145700 1 --2- 172.21.15.189:0/1806152409 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fce8406adb0 0x7fce8406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcea013def0 tx=0x7fce90009040 comp rx=0 tx=0).stop 2024-04-06T12:30:10.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.704+0000 7fcea8145700 1 -- 172.21.15.189:0/1806152409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce8c003a30 msgr2=0x7fcea013eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:10.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.704+0000 7fcea8145700 1 --2- 172.21.15.189:0/1806152409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce8c003a30 0x7fcea013eb80 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fce9c016040 tx=0x7fce9c0047c0 comp rx=0 tx=0).stop 2024-04-06T12:30:10.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.704+0000 7fcea8145700 1 -- 172.21.15.189:0/1806152409 shutdown_connections 2024-04-06T12:30:10.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.704+0000 7fcea8145700 1 --2- 172.21.15.189:0/1806152409 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcea0139870 0x7fcea0139cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:10.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.704+0000 7fcea8145700 1 --2- 172.21.15.189:0/1806152409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce8c003a30 0x7fcea013eb80 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:10.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.704+0000 7fcea8145700 1 --2- 172.21.15.189:0/1806152409 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fce8406adb0 0x7fce8406d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:10.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.704+0000 7fcea8145700 1 --2- 172.21.15.189:0/1806152409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcea013db10 0x7fcea01422d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:10.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.704+0000 7fcea8145700 1 -- 172.21.15.189:0/1806152409 >> 172.21.15.189:0/1806152409 conn(0x7fcea00d00b0 msgr2=0x7fcea00717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:10.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.704+0000 7fcea8145700 1 -- 172.21.15.189:0/1806152409 shutdown_connections 2024-04-06T12:30:10.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.704+0000 7fcea8145700 1 -- 172.21.15.189:0/1806152409 wait complete. 2024-04-06T12:30:10.716 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-06T12:30:10.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.850+0000 7f091133a700 1 Processor -- start 2024-04-06T12:30:10.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.850+0000 7f091133a700 1 -- start start 2024-04-06T12:30:10.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.851+0000 7f091133a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f090c153c80 0x7f090c15d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:10.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.851+0000 7f091133a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f090c0b31d0 0x7f090c071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:10.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.851+0000 7f091133a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090c071130 con 0x7f090c15a530 2024-04-06T12:30:10.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.851+0000 7f091133a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f090c15d680 con 0x7f090c153c80 2024-04-06T12:30:10.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.851+0000 7f091133a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f090c15d960 con 0x7f090c0b31d0 2024-04-06T12:30:10.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.851+0000 7f090affd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f090c153c80 0x7f090c15d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:10.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.851+0000 7f090affd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f090c153c80 0x7f090c15d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49570/0 (socket says 172.21.15.189:49570) 2024-04-06T12:30:10.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.851+0000 7f090affd700 1 -- 172.21.15.189:0/3718563459 learned_addr learned my addr 172.21.15.189:0/3718563459 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:10.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.852+0000 7f090affd700 1 -- 172.21.15.189:0/3718563459 >> v2:172.21.15.192:3300/0 conn(0x7f090c0b31d0 msgr2=0x7f090c071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:10.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.852+0000 7f090affd700 1 --2- 172.21.15.189:0/3718563459 >> v2:172.21.15.192:3300/0 conn(0x7f090c0b31d0 0x7f090c071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:10.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.852+0000 7f090affd700 1 -- 172.21.15.189:0/3718563459 >> v1:172.21.15.192:6789/0 conn(0x7f090c15a530 legacy=0x7f090c0b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:10.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.852+0000 7f090affd700 1 -- 172.21.15.189:0/3718563459 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090c15dc40 con 0x7f090c153c80 2024-04-06T12:30:10.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.852+0000 7f090affd700 1 --2- 172.21.15.189:0/3718563459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f090c153c80 0x7f090c15d0a0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f090c0cbad0 tx=0x7f0900009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=667ed8a10986b2eb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:10.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.852+0000 7f0909ffb700 1 -- 172.21.15.189:0/3718563459 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0900004230 con 0x7f090c153c80 2024-04-06T12:30:10.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.852+0000 7f0909ffb700 1 -- 172.21.15.189:0/3718563459 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0900004390 con 0x7f090c153c80 2024-04-06T12:30:10.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.852+0000 7f0909ffb700 1 -- 172.21.15.189:0/3718563459 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09000036a0 con 0x7f090c153c80 2024-04-06T12:30:10.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.852+0000 7f091133a700 1 -- 172.21.15.189:0/3718563459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f090c153c80 msgr2=0x7f090c15d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:10.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.852+0000 7f091133a700 1 --2- 172.21.15.189:0/3718563459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f090c153c80 0x7f090c15d0a0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f090c0cbad0 tx=0x7f0900009bd0 comp rx=0 tx=0).stop 2024-04-06T12:30:10.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.853+0000 7f091133a700 1 -- 172.21.15.189:0/3718563459 shutdown_connections 2024-04-06T12:30:10.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.853+0000 7f091133a700 1 --2- 172.21.15.189:0/3718563459 >> v2:172.21.15.192:3300/0 conn(0x7f090c0b31d0 0x7f090c071e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:10.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.853+0000 7f091133a700 1 --2- 172.21.15.189:0/3718563459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f090c153c80 0x7f090c15d0a0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:10.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.853+0000 7f091133a700 1 -- 172.21.15.189:0/3718563459 >> 172.21.15.189:0/3718563459 conn(0x7f090c06a730 msgr2=0x7f090c06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:10.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.853+0000 7f091133a700 1 -- 172.21.15.189:0/3718563459 shutdown_connections 2024-04-06T12:30:10.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.853+0000 7f091133a700 1 -- 172.21.15.189:0/3718563459 wait complete. 2024-04-06T12:30:10.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.854+0000 7f091133a700 1 Processor -- start 2024-04-06T12:30:10.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.854+0000 7f091133a700 1 -- start start 2024-04-06T12:30:10.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f091133a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f090c0b31d0 0x7f090c14f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:10.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f091133a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f090c153c80 0x7f090c152250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:10.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f091133a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f090c15a530 0x7f090c0cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:10.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f091133a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090c0cbcc0 con 0x7f090c15a530 2024-04-06T12:30:10.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f091133a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090c0cbf60 con 0x7f090c0b31d0 2024-04-06T12:30:10.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f091133a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f090c0cc240 con 0x7f090c153c80 2024-04-06T12:30:10.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f090b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f090c15a530 0x7f090c0cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:10.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f090b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f090c15a530 0x7f090c0cdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49572/0 (socket says 172.21.15.189:49572) 2024-04-06T12:30:10.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f090b7fe700 1 -- 172.21.15.189:0/967843133 learned_addr learned my addr 172.21.15.189:0/967843133 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:10.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f090a7fc700 1 --2- 172.21.15.189:0/967843133 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f090c153c80 0x7f090c152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:10.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f090affd700 1 --2- 172.21.15.189:0/967843133 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f090c0b31d0 0x7f090c14f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:10.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f090b7fe700 1 -- 172.21.15.189:0/967843133 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f090c153c80 msgr2=0x7f090c152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:10.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f090b7fe700 1 --2- 172.21.15.189:0/967843133 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f090c153c80 0x7f090c152250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:10.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f090b7fe700 1 -- 172.21.15.189:0/967843133 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f090c0b31d0 msgr2=0x7f090c14f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:10.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f090b7fe700 1 --2- 172.21.15.189:0/967843133 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f090c0b31d0 0x7f090c14f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:10.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f090b7fe700 1 -- 172.21.15.189:0/967843133 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0900009550 con 0x7f090c15a530 2024-04-06T12:30:10.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f090b7fe700 1 --2- 172.21.15.189:0/967843133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f090c15a530 0x7f090c0cdaf0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f090400b970 tx=0x7f090400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:10.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f08f37fe700 1 -- 172.21.15.189:0/967843133 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090400c9b0 con 0x7f090c15a530 2024-04-06T12:30:10.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f08f37fe700 1 -- 172.21.15.189:0/967843133 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090400cb30 con 0x7f090c15a530 2024-04-06T12:30:10.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f091133a700 1 -- 172.21.15.189:0/967843133 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f090c0cc520 con 0x7f090c15a530 2024-04-06T12:30:10.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f08f37fe700 1 -- 172.21.15.189:0/967843133 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0904014440 con 0x7f090c15a530 2024-04-06T12:30:10.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.855+0000 7f091133a700 1 -- 172.21.15.189:0/967843133 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f090c14f730 con 0x7f090c15a530 2024-04-06T12:30:10.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.856+0000 7f091133a700 1 -- 172.21.15.189:0/967843133 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f090c067bf0 con 0x7f090c15a530 2024-04-06T12:30:10.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.857+0000 7f08f37fe700 1 -- 172.21.15.189:0/967843133 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f090401d3b0 con 0x7f090c15a530 2024-04-06T12:30:10.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.857+0000 7f08f37fe700 1 --2- 172.21.15.189:0/967843133 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f08f806adb0 0x7f08f806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:10.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.857+0000 7f08f37fe700 1 -- 172.21.15.189:0/967843133 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f0904006e40 con 0x7f090c15a530 2024-04-06T12:30:10.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.857+0000 7f090affd700 1 --2- 172.21.15.189:0/967843133 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f08f806adb0 0x7f08f806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:10.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.858+0000 7f090affd700 1 --2- 172.21.15.189:0/967843133 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f08f806adb0 0x7f08f806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0900009130 tx=0x7f0900008030 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:10.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:10.862+0000 7f08f37fe700 1 -- 172.21.15.189:0/967843133 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0904054310 con 0x7f090c15a530 2024-04-06T12:30:11.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.058+0000 7f091133a700 1 -- 172.21.15.189:0/967843133 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f090c0cca40 con 0x7f090c15a530 2024-04-06T12:30:11.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.062+0000 7f08f37fe700 1 -- 172.21.15.189:0/967843133 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0904057930 con 0x7f090c15a530 2024-04-06T12:30:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.064+0000 7f091133a700 1 -- 172.21.15.189:0/967843133 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f08f806adb0 msgr2=0x7f08f806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.064+0000 7f091133a700 1 --2- 172.21.15.189:0/967843133 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f08f806adb0 0x7f08f806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0900009130 tx=0x7f0900008030 comp rx=0 tx=0).stop 2024-04-06T12:30:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.064+0000 7f091133a700 1 -- 172.21.15.189:0/967843133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f090c15a530 msgr2=0x7f090c0cdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.064+0000 7f091133a700 1 --2- 172.21.15.189:0/967843133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f090c15a530 0x7f090c0cdaf0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f090400b970 tx=0x7f090400bd30 comp rx=0 tx=0).stop 2024-04-06T12:30:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.064+0000 7f091133a700 1 -- 172.21.15.189:0/967843133 shutdown_connections 2024-04-06T12:30:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.064+0000 7f091133a700 1 --2- 172.21.15.189:0/967843133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f090c15a530 0x7f090c0cdaf0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.064+0000 7f091133a700 1 --2- 172.21.15.189:0/967843133 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f08f806adb0 0x7f08f806d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.064+0000 7f091133a700 1 --2- 172.21.15.189:0/967843133 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f090c0b31d0 0x7f090c14f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.064+0000 7f091133a700 1 --2- 172.21.15.189:0/967843133 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f090c153c80 0x7f090c152250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.064+0000 7f091133a700 1 -- 172.21.15.189:0/967843133 >> 172.21.15.189:0/967843133 conn(0x7f090c06a730 msgr2=0x7f090c0b0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:11.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.065+0000 7f091133a700 1 -- 172.21.15.189:0/967843133 shutdown_connections 2024-04-06T12:30:11.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.065+0000 7f091133a700 1 -- 172.21.15.189:0/967843133 wait complete. 2024-04-06T12:30:11.074 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T12:30:11.074 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:30:11.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:30:11.170 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T12:30:11.170 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:30:11.170 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:30:11.188 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:30:11.188 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:30:11.188 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:30:11.208 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:30:11.208 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:30:11.208 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:30:11.245 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:30:11.245 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:30:11.245 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:30:11.267 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:30:11.267 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-06T12:30:11.268 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:30:11.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.412+0000 7f058792c700 1 Processor -- start 2024-04-06T12:30:11.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.412+0000 7f058792c700 1 -- start start 2024-04-06T12:30:11.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.413+0000 7f058792c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f05800cedf0 0x7f05800cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:11.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.413+0000 7f058792c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05800cf730 con 0x7f0580146dd0 2024-04-06T12:30:11.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.413+0000 7f058792c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058006d850 con 0x7f05800d17a0 2024-04-06T12:30:11.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.413+0000 7f058792c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f058006beb0 con 0x7f05800cedf0 2024-04-06T12:30:11.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.413+0000 7f0584ec7700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0580146dd0 0x7f058006d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51136/0 (socket says 172.21.15.189:51136) 2024-04-06T12:30:11.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.413+0000 7f0584ec7700 1 -- 172.21.15.189:0/2889292584 learned_addr learned my addr 172.21.15.189:0/2889292584 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:11.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.413+0000 7f0585ec9700 1 --2- 172.21.15.189:0/2889292584 >> v2:172.21.15.192:3300/0 conn(0x7f05800cedf0 0x7f05800cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:11.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0577fff700 1 -- 172.21.15.189:0/2889292584 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989132587 0 0) 0x7f05800cf730 con 0x7f0580146dd0 2024-04-06T12:30:11.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0577fff700 1 -- 172.21.15.189:0/2889292584 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f056c0035b0 con 0x7f0580146dd0 2024-04-06T12:30:11.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0577fff700 1 -- 172.21.15.189:0/2889292584 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207929401 0 0) 0x7f058006d850 con 0x7f05800d17a0 2024-04-06T12:30:11.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0577fff700 1 -- 172.21.15.189:0/2889292584 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05800cf730 con 0x7f05800d17a0 2024-04-06T12:30:11.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0577fff700 1 -- 172.21.15.189:0/2889292584 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 628822735 0 0) 0x7f056c0035b0 con 0x7f0580146dd0 2024-04-06T12:30:11.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0577fff700 1 -- 172.21.15.189:0/2889292584 >> v2:172.21.15.192:3300/0 conn(0x7f05800cedf0 msgr2=0x7f05800cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:11.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0577fff700 1 --2- 172.21.15.189:0/2889292584 >> v2:172.21.15.192:3300/0 conn(0x7f05800cedf0 0x7f05800cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0577fff700 1 -- 172.21.15.189:0/2889292584 >> v1:172.21.15.192:6789/0 conn(0x7f05800d17a0 legacy=0x7f058006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:11.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0577fff700 1 -- 172.21.15.189:0/2889292584 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058006c0a0 con 0x7f0580146dd0 2024-04-06T12:30:11.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0577fff700 1 -- 172.21.15.189:0/2889292584 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f057c001670 con 0x7f0580146dd0 2024-04-06T12:30:11.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0577fff700 1 -- 172.21.15.189:0/2889292584 >> v1:172.21.15.189:6789/0 conn(0x7f0580146dd0 legacy=0x7f058006d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:11.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0577fff700 1 --2- 172.21.15.189:0/2889292584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f056c003be0 0x7f056c003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:11.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0577fff700 1 --2- 172.21.15.189:0/2889292584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f056c004850 0x7f056c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:11.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0577fff700 1 --2- 172.21.15.189:0/2889292584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f056c005210 0x7f056c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:11.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0577fff700 1 -- 172.21.15.189:0/2889292584 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c00c0e0 con 0x7f056c004850 2024-04-06T12:30:11.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0577fff700 1 -- 172.21.15.189:0/2889292584 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c00c220 con 0x7f056c003be0 2024-04-06T12:30:11.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0577fff700 1 -- 172.21.15.189:0/2889292584 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f056c00c3e0 con 0x7f056c005210 2024-04-06T12:30:11.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0584ec7700 1 --2- 172.21.15.189:0/2889292584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f056c004850 0x7f056c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:11.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f05856c8700 1 --2- 172.21.15.189:0/2889292584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f056c003be0 0x7f056c003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:11.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0585ec9700 1 --2- 172.21.15.189:0/2889292584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f056c005210 0x7f056c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:11.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0584ec7700 1 -- 172.21.15.189:0/2889292584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f056c005210 msgr2=0x7f056c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:11.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0584ec7700 1 --2- 172.21.15.189:0/2889292584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f056c005210 0x7f056c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0584ec7700 1 -- 172.21.15.189:0/2889292584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f056c003be0 msgr2=0x7f056c003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:11.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0584ec7700 1 --2- 172.21.15.189:0/2889292584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f056c003be0 0x7f056c003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0584ec7700 1 -- 172.21.15.189:0/2889292584 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c0035b0 con 0x7f056c004850 2024-04-06T12:30:11.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0585ec9700 1 --2- 172.21.15.189:0/2889292584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f056c005210 0x7f056c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:11.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0585ec9700 1 -- 172.21.15.189:0/2889292584 reap_dead start 2024-04-06T12:30:11.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.414+0000 7f0584ec7700 1 --2- 172.21.15.189:0/2889292584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f056c004850 0x7f056c004cd0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f057c00c0e0 tx=0x7f056c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b67a88df7378b779 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:11.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.415+0000 7f0577fff700 1 -- 172.21.15.189:0/2889292584 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057c00cc80 con 0x7f056c004850 2024-04-06T12:30:11.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.415+0000 7f0577fff700 1 -- 172.21.15.189:0/2889292584 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057c00cde0 con 0x7f056c004850 2024-04-06T12:30:11.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.415+0000 7f0577fff700 1 -- 172.21.15.189:0/2889292584 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057c005ba0 con 0x7f056c004850 2024-04-06T12:30:11.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.415+0000 7f058792c700 1 -- 172.21.15.189:0/2889292584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f056c004850 msgr2=0x7f056c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:11.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.415+0000 7f058792c700 1 --2- 172.21.15.189:0/2889292584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f056c004850 0x7f056c004cd0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f057c00c0e0 tx=0x7f056c005880 comp rx=0 tx=0).stop 2024-04-06T12:30:11.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.415+0000 7f058792c700 1 -- 172.21.15.189:0/2889292584 shutdown_connections 2024-04-06T12:30:11.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.415+0000 7f058792c700 1 --2- 172.21.15.189:0/2889292584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f056c004850 0x7f056c004cd0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.415+0000 7f058792c700 1 -- 172.21.15.189:0/2889292584 >> 172.21.15.189:0/2889292584 conn(0x7f05800b2db0 msgr2=0x7f05800b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:11.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.415+0000 7f058792c700 1 -- 172.21.15.189:0/2889292584 shutdown_connections 2024-04-06T12:30:11.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.415+0000 7f058792c700 1 -- 172.21.15.189:0/2889292584 wait complete. 2024-04-06T12:30:11.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.417+0000 7f058792c700 1 Processor -- start 2024-04-06T12:30:11.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.417+0000 7f058792c700 1 -- start start 2024-04-06T12:30:11.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f058792c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f056c004850 0x7f05800adcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:11.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f058792c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05800c3dd0 0x7f05800c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:11.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f058792c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f058014eab0 0x7f058015de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:11.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f058792c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05800c8b40 con 0x7f05800c3dd0 2024-04-06T12:30:11.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f058792c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05800aee60 con 0x7f056c004850 2024-04-06T12:30:11.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f058792c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05800af140 con 0x7f058014eab0 2024-04-06T12:30:11.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f0584ec7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05800c3dd0 0x7f05800c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:11.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f0584ec7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05800c3dd0 0x7f05800c82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49590/0 (socket says 172.21.15.189:49590) 2024-04-06T12:30:11.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f0584ec7700 1 -- 172.21.15.189:0/2776668198 learned_addr learned my addr 172.21.15.189:0/2776668198 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:11.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f05856c8700 1 --2- 172.21.15.189:0/2776668198 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f056c004850 0x7f05800adcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:11.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f0585ec9700 1 --2- 172.21.15.189:0/2776668198 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f058014eab0 0x7f058015de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:11.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f0584ec7700 1 -- 172.21.15.189:0/2776668198 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f058014eab0 msgr2=0x7f058015de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:11.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f0584ec7700 1 --2- 172.21.15.189:0/2776668198 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f058014eab0 0x7f058015de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f0584ec7700 1 -- 172.21.15.189:0/2776668198 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f056c004850 msgr2=0x7f05800adcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:11.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f0584ec7700 1 --2- 172.21.15.189:0/2776668198 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f056c004850 0x7f05800adcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f0584ec7700 1 -- 172.21.15.189:0/2776668198 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05800af420 con 0x7f05800c3dd0 2024-04-06T12:30:11.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f0584ec7700 1 --2- 172.21.15.189:0/2776668198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05800c3dd0 0x7f05800c82f0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f056c0047d0 tx=0x7f057c001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:11.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f0575ffb700 1 -- 172.21.15.189:0/2776668198 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057c0072d0 con 0x7f05800c3dd0 2024-04-06T12:30:11.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f0575ffb700 1 -- 172.21.15.189:0/2776668198 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057c007450 con 0x7f05800c3dd0 2024-04-06T12:30:11.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f0575ffb700 1 -- 172.21.15.189:0/2776668198 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057c007800 con 0x7f05800c3dd0 2024-04-06T12:30:11.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.418+0000 7f058792c700 1 -- 172.21.15.189:0/2776668198 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05800af700 con 0x7f05800c3dd0 2024-04-06T12:30:11.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.419+0000 7f058792c700 1 -- 172.21.15.189:0/2776668198 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f058014ef30 con 0x7f05800c3dd0 2024-04-06T12:30:11.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.420+0000 7f058792c700 1 -- 172.21.15.189:0/2776668198 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05800afa70 con 0x7f05800c3dd0 2024-04-06T12:30:11.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.420+0000 7f0575ffb700 1 -- 172.21.15.189:0/2776668198 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f057c01e430 con 0x7f05800c3dd0 2024-04-06T12:30:11.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.421+0000 7f0575ffb700 1 --2- 172.21.15.189:0/2776668198 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f056406adb0 0x7f056406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:11.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.421+0000 7f05856c8700 1 --2- 172.21.15.189:0/2776668198 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f056406adb0 0x7f056406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:11.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.421+0000 7f0575ffb700 1 -- 172.21.15.189:0/2776668198 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f057c08a610 con 0x7f05800c3dd0 2024-04-06T12:30:11.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.421+0000 7f05856c8700 1 --2- 172.21.15.189:0/2776668198 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f056406adb0 0x7f056406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0570000c00 tx=0x7f0570009040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:11.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.425+0000 7f0575ffb700 1 -- 172.21.15.189:0/2776668198 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f057c0557a0 con 0x7f05800c3dd0 2024-04-06T12:30:11.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.663+0000 7f058792c700 1 -- 172.21.15.189:0/2776668198 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f058014f0c0 con 0x7f05800c3dd0 2024-04-06T12:30:11.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.664+0000 7f0575ffb700 1 -- 172.21.15.189:0/2776668198 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+3532 (secure 0 0 0) 0x7f057c017de0 con 0x7f05800c3dd0 2024-04-06T12:30:11.661 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:30:11.661 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":48,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6389,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/17345288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":17345288},{"type":"v1","addr":"172.21.15.192:6835","nonce":17345288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":7078,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/440618958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":440618958},{"type":"v1","addr":"172.21.15.192:6834","nonce":440618958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":7569,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3391245238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3391245238},{"type":"v1","addr":"172.21.15.189:6836","nonce":3391245238}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":8016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3454305305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3454305305},{"type":"v1","addr":"172.21.15.189:6837","nonce":3454305305}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":48}],"filesystems":[]} 2024-04-06T12:30:11.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.666+0000 7f058792c700 1 -- 172.21.15.189:0/2776668198 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f056406adb0 msgr2=0x7f056406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:11.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.666+0000 7f058792c700 1 --2- 172.21.15.189:0/2776668198 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f056406adb0 0x7f056406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0570000c00 tx=0x7f0570009040 comp rx=0 tx=0).stop 2024-04-06T12:30:11.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.666+0000 7f058792c700 1 -- 172.21.15.189:0/2776668198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05800c3dd0 msgr2=0x7f05800c82f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:11.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.666+0000 7f058792c700 1 --2- 172.21.15.189:0/2776668198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05800c3dd0 0x7f05800c82f0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f056c0047d0 tx=0x7f057c001930 comp rx=0 tx=0).stop 2024-04-06T12:30:11.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.666+0000 7f058792c700 1 -- 172.21.15.189:0/2776668198 shutdown_connections 2024-04-06T12:30:11.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.666+0000 7f058792c700 1 --2- 172.21.15.189:0/2776668198 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f058014eab0 0x7f058015de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.666+0000 7f058792c700 1 --2- 172.21.15.189:0/2776668198 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f056406adb0 0x7f056406d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.666+0000 7f058792c700 1 --2- 172.21.15.189:0/2776668198 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f056c004850 0x7f05800adcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.666+0000 7f058792c700 1 --2- 172.21.15.189:0/2776668198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05800c3dd0 0x7f05800c82f0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.666+0000 7f058792c700 1 -- 172.21.15.189:0/2776668198 >> 172.21.15.189:0/2776668198 conn(0x7f05800b2db0 msgr2=0x7f05800b3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:11.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.666+0000 7f058792c700 1 -- 172.21.15.189:0/2776668198 shutdown_connections 2024-04-06T12:30:11.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.666+0000 7f058792c700 1 -- 172.21.15.189:0/2776668198 wait complete. 2024-04-06T12:30:11.664 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 48 2024-04-06T12:30:11.676 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-06T12:30:11.676 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-06T12:30:11.676 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-06T12:30:11.676 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:30:11.677 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-06T12:30:11.677 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-06T12:30:11.677 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-06T12:30:11.677 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:30:11.677 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-06T12:30:11.677 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-06T12:30:11.678 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-06T12:30:11.678 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:30:11.679 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-06T12:30:11.679 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-06T12:30:11.679 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-06T12:30:11.679 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:30:11.687 INFO:tasks.ceph.mds.a:Stopped 2024-04-06T12:30:11.687 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-06T12:30:11.688 INFO:tasks.ceph.mds.c:Stopped 2024-04-06T12:30:11.688 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-06T12:30:11.690 INFO:tasks.ceph.mds.b:Stopped 2024-04-06T12:30:11.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-06T12:30:11.691 INFO:tasks.ceph.mds.d:Stopped 2024-04-06T12:30:11.691 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-06T12:30:11.730 INFO:tasks.ceph.mds.a:Started 2024-04-06T12:30:11.731 DEBUG:teuthology.parallel:result is None 2024-04-06T12:30:11.732 INFO:tasks.ceph.mds.c:Started 2024-04-06T12:30:11.732 DEBUG:teuthology.parallel:result is None 2024-04-06T12:30:11.733 INFO:tasks.ceph.mds.b:Started 2024-04-06T12:30:11.733 DEBUG:teuthology.parallel:result is None 2024-04-06T12:30:11.734 INFO:tasks.ceph.mds.d:Started 2024-04-06T12:30:11.734 DEBUG:teuthology.parallel:result is None 2024-04-06T12:30:11.734 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-06T12:30:11.810 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T12:30:11.811 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2024-04-06T12:30:11.825 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T12:30:11.826 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2024-04-06T12:30:11.826 INFO:tasks.ceph.mds.b.smithi192.stderr:2024-04-06T12:30:11.829+0000 7f97950c3700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-04-06T12:30:11.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.870+0000 7f0f1cf1a700 1 Processor -- start 2024-04-06T12:30:11.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.870+0000 7f0f1cf1a700 1 -- start start 2024-04-06T12:30:11.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.871+0000 7f0f1cf1a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f180d17a0 0x7f0f1806b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:11.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.871+0000 7f0f1cf1a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0f18146dd0 0x7f0f1806d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:11.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.871+0000 7f0f1cf1a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0f180cedf0 0x7f0f180cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:11.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.871+0000 7f0f1cf1a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f180cf760 con 0x7f0f18146dd0 2024-04-06T12:30:11.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.871+0000 7f0f1cf1a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f1806c0a0 con 0x7f0f180cedf0 2024-04-06T12:30:11.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.871+0000 7f0f1cf1a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1806c380 con 0x7f0f180d17a0 2024-04-06T12:30:11.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.871+0000 7f0f15d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0f18146dd0 0x7f0f1806d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:11.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.871+0000 7f0f15d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0f18146dd0 0x7f0f1806d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49612/0 (socket says 172.21.15.189:49612) 2024-04-06T12:30:11.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.871+0000 7f0f15d9b700 1 -- 172.21.15.189:0/1035355465 learned_addr learned my addr 172.21.15.189:0/1035355465 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:11.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.871+0000 7f0f1659c700 1 --2- 172.21.15.189:0/1035355465 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f180d17a0 0x7f0f1806b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:11.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.871+0000 7f0f16d9d700 1 --2- 172.21.15.189:0/1035355465 >> v2:172.21.15.192:3300/0 conn(0x7f0f180cedf0 0x7f0f180cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:11.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.871+0000 7f0f15d9b700 1 -- 172.21.15.189:0/1035355465 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f180d17a0 msgr2=0x7f0f1806b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:11.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.871+0000 7f0f15d9b700 1 --2- 172.21.15.189:0/1035355465 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f180d17a0 0x7f0f1806b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.871+0000 7f0f15d9b700 1 -- 172.21.15.189:0/1035355465 >> v2:172.21.15.192:3300/0 conn(0x7f0f180cedf0 msgr2=0x7f0f180cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:11.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.871+0000 7f0f15d9b700 1 --2- 172.21.15.189:0/1035355465 >> v2:172.21.15.192:3300/0 conn(0x7f0f180cedf0 0x7f0f180cf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.871+0000 7f0f15d9b700 1 -- 172.21.15.189:0/1035355465 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1806c660 con 0x7f0f18146dd0 2024-04-06T12:30:11.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.872+0000 7f0f15d9b700 1 --2- 172.21.15.189:0/1035355465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f18146dd0 0x7f0f1806d140 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f0f18162120 tx=0x7f0f08009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ce4a1ebb8fa50f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:11.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.872+0000 7f0f1559a700 1 -- 172.21.15.189:0/1035355465 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0800b870 con 0x7f0f18146dd0 2024-04-06T12:30:11.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.872+0000 7f0f1559a700 1 -- 172.21.15.189:0/1035355465 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0800b9d0 con 0x7f0f18146dd0 2024-04-06T12:30:11.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.872+0000 7f0f1559a700 1 -- 172.21.15.189:0/1035355465 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0800bd60 con 0x7f0f18146dd0 2024-04-06T12:30:11.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.872+0000 7f0f1cf1a700 1 -- 172.21.15.189:0/1035355465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f18146dd0 msgr2=0x7f0f1806d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:11.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.872+0000 7f0f1cf1a700 1 --2- 172.21.15.189:0/1035355465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f18146dd0 0x7f0f1806d140 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f0f18162120 tx=0x7f0f08009bd0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.872+0000 7f0f1cf1a700 1 -- 172.21.15.189:0/1035355465 shutdown_connections 2024-04-06T12:30:11.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.872+0000 7f0f1cf1a700 1 --2- 172.21.15.189:0/1035355465 >> v2:172.21.15.192:3300/0 conn(0x7f0f180cedf0 0x7f0f180cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.872+0000 7f0f1cf1a700 1 --2- 172.21.15.189:0/1035355465 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f180d17a0 0x7f0f1806b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.872+0000 7f0f1cf1a700 1 --2- 172.21.15.189:0/1035355465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f18146dd0 0x7f0f1806d140 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.872+0000 7f0f1cf1a700 1 -- 172.21.15.189:0/1035355465 >> 172.21.15.189:0/1035355465 conn(0x7f0f180b2db0 msgr2=0x7f0f180b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:11.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.872+0000 7f0f1cf1a700 1 -- 172.21.15.189:0/1035355465 shutdown_connections 2024-04-06T12:30:11.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.873+0000 7f0f1cf1a700 1 -- 172.21.15.189:0/1035355465 wait complete. 2024-04-06T12:30:11.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.874+0000 7f0f1cf1a700 1 Processor -- start 2024-04-06T12:30:11.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.874+0000 7f0f1cf1a700 1 -- start start 2024-04-06T12:30:11.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f1cf1a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f180cedf0 0x7f0f1815af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:11.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f1cf1a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f180d17a0 0x7f0f180b0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:11.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f1cf1a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f18146dd0 0x7f0f180b5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:11.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f1cf1a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f180ade10 con 0x7f0f18146dd0 2024-04-06T12:30:11.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f1cf1a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f180ad200 con 0x7f0f180d17a0 2024-04-06T12:30:11.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f1cf1a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f180ad4e0 con 0x7f0f180cedf0 2024-04-06T12:30:11.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f16d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f18146dd0 0x7f0f180b5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:11.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f16d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f18146dd0 0x7f0f180b5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49624/0 (socket says 172.21.15.189:49624) 2024-04-06T12:30:11.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f16d9d700 1 -- 172.21.15.189:0/3067629277 learned_addr learned my addr 172.21.15.189:0/3067629277 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:11.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f15d9b700 1 --2- 172.21.15.189:0/3067629277 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f180d17a0 0x7f0f180b0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:11.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f1659c700 1 --2- 172.21.15.189:0/3067629277 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f180cedf0 0x7f0f1815af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:11.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f16d9d700 1 -- 172.21.15.189:0/3067629277 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f180cedf0 msgr2=0x7f0f1815af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:11.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f16d9d700 1 --2- 172.21.15.189:0/3067629277 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f180cedf0 0x7f0f1815af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f16d9d700 1 -- 172.21.15.189:0/3067629277 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f180d17a0 msgr2=0x7f0f180b0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:11.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f16d9d700 1 --2- 172.21.15.189:0/3067629277 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f180d17a0 0x7f0f180b0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:11.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f16d9d700 1 -- 172.21.15.189:0/3067629277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f08009550 con 0x7f0f18146dd0 2024-04-06T12:30:11.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f16d9d700 1 --2- 172.21.15.189:0/3067629277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f18146dd0 0x7f0f180b5eb0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f0f10007e60 tx=0x7f0f1000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:11.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f06ffd700 1 -- 172.21.15.189:0/3067629277 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1000ed30 con 0x7f0f18146dd0 2024-04-06T12:30:11.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f06ffd700 1 -- 172.21.15.189:0/3067629277 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f10014420 con 0x7f0f18146dd0 2024-04-06T12:30:11.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f1cf1a700 1 -- 172.21.15.189:0/3067629277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f180ad7c0 con 0x7f0f18146dd0 2024-04-06T12:30:11.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f06ffd700 1 -- 172.21.15.189:0/3067629277 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f100147d0 con 0x7f0f18146dd0 2024-04-06T12:30:11.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.875+0000 7f0f1cf1a700 1 -- 172.21.15.189:0/3067629277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f181f2c30 con 0x7f0f18146dd0 2024-04-06T12:30:11.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.876+0000 7f0f1cf1a700 1 -- 172.21.15.189:0/3067629277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f18074990 con 0x7f0f18146dd0 2024-04-06T12:30:11.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.877+0000 7f0f06ffd700 1 -- 172.21.15.189:0/3067629277 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0f10014930 con 0x7f0f18146dd0 2024-04-06T12:30:11.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.877+0000 7f0f06ffd700 1 --2- 172.21.15.189:0/3067629277 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0ef806adb0 0x7f0ef806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:11.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.877+0000 7f0f06ffd700 1 -- 172.21.15.189:0/3067629277 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f0f10089ba0 con 0x7f0f18146dd0 2024-04-06T12:30:11.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.877+0000 7f0f1659c700 1 --2- 172.21.15.189:0/3067629277 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0ef806adb0 0x7f0ef806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:11.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.877+0000 7f0f1659c700 1 --2- 172.21.15.189:0/3067629277 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0ef806adb0 0x7f0ef806d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0f18146040 tx=0x7f0f0c008040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:11.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:11.882+0000 7f0f06ffd700 1 -- 172.21.15.189:0/3067629277 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0f10054c30 con 0x7f0f18146dd0 2024-04-06T12:30:12.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:12.091+0000 7f0f1cf1a700 1 -- 172.21.15.189:0/3067629277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0f180d1e20 con 0x7f0f18146dd0 2024-04-06T12:30:12.938 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:30:12.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:30:13.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.065+0000 7f0f06ffd700 1 -- 172.21.15.189:0/3067629277 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v124) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0f10058250 con 0x7f0f18146dd0 2024-04-06T12:30:13.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.068+0000 7f0f1cf1a700 1 -- 172.21.15.189:0/3067629277 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0ef806adb0 msgr2=0x7f0ef806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:13.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.068+0000 7f0f1cf1a700 1 --2- 172.21.15.189:0/3067629277 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0ef806adb0 0x7f0ef806d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0f18146040 tx=0x7f0f0c008040 comp rx=0 tx=0).stop 2024-04-06T12:30:13.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.069+0000 7f0f1cf1a700 1 -- 172.21.15.189:0/3067629277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f18146dd0 msgr2=0x7f0f180b5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:13.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.069+0000 7f0f1cf1a700 1 --2- 172.21.15.189:0/3067629277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f18146dd0 0x7f0f180b5eb0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f0f10007e60 tx=0x7f0f1000e3c0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.069+0000 7f0f1cf1a700 1 -- 172.21.15.189:0/3067629277 shutdown_connections 2024-04-06T12:30:13.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.069+0000 7f0f1cf1a700 1 --2- 172.21.15.189:0/3067629277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f18146dd0 0x7f0f180b5eb0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.069+0000 7f0f1cf1a700 1 --2- 172.21.15.189:0/3067629277 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f180cedf0 0x7f0f1815af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.069+0000 7f0f1cf1a700 1 --2- 172.21.15.189:0/3067629277 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0ef806adb0 0x7f0ef806d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.069+0000 7f0f1cf1a700 1 --2- 172.21.15.189:0/3067629277 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f180d17a0 0x7f0f180b0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.069+0000 7f0f1cf1a700 1 -- 172.21.15.189:0/3067629277 >> 172.21.15.189:0/3067629277 conn(0x7f0f180b2db0 msgr2=0x7f0f1815e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:13.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.069+0000 7f0f1cf1a700 1 -- 172.21.15.189:0/3067629277 shutdown_connections 2024-04-06T12:30:13.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.069+0000 7f0f1cf1a700 1 -- 172.21.15.189:0/3067629277 wait complete. 2024-04-06T12:30:13.067 INFO:teuthology.orchestra.run.smithi189.stderr: removed all blocklist entries 2024-04-06T12:30:13.081 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-06T12:30:13.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.252+0000 7f1dee76d700 1 Processor -- start 2024-04-06T12:30:13.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.253+0000 7f1dee76d700 1 -- start start 2024-04-06T12:30:13.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.253+0000 7f1dee76d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1de815d7b0 0x7f1de80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:13.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.253+0000 7f1dee76d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1de815e500 0x7f1de806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:13.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.253+0000 7f1dee76d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de80b3990 con 0x7f1de806a730 2024-04-06T12:30:13.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.253+0000 7f1dee76d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de815e950 con 0x7f1de815e500 2024-04-06T12:30:13.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.253+0000 7f1dee76d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de806b0f0 con 0x7f1de815d7b0 2024-04-06T12:30:13.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.253+0000 7f1decd0a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1de815e500 0x7f1de806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:13.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.254+0000 7f1decd0a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1de815e500 0x7f1de806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49634/0 (socket says 172.21.15.189:49634) 2024-04-06T12:30:13.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.254+0000 7f1decd0a700 1 -- 172.21.15.189:0/2386822097 learned_addr learned my addr 172.21.15.189:0/2386822097 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:13.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.254+0000 7f1de7fff700 1 --2- 172.21.15.189:0/2386822097 >> v2:172.21.15.192:3300/0 conn(0x7f1de815d7b0 0x7f1de80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:13.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.254+0000 7f1decd0a700 1 -- 172.21.15.189:0/2386822097 >> v2:172.21.15.192:3300/0 conn(0x7f1de815d7b0 msgr2=0x7f1de80b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:13.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.254+0000 7f1decd0a700 1 --2- 172.21.15.189:0/2386822097 >> v2:172.21.15.192:3300/0 conn(0x7f1de815d7b0 0x7f1de80b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.254+0000 7f1decd0a700 1 -- 172.21.15.189:0/2386822097 >> v1:172.21.15.192:6789/0 conn(0x7f1de806a730 legacy=0x7f1de815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:13.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.254+0000 7f1decd0a700 1 -- 172.21.15.189:0/2386822097 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de80b3060 con 0x7f1de815e500 2024-04-06T12:30:13.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.254+0000 7f1de7fff700 1 --2- 172.21.15.189:0/2386822097 >> v2:172.21.15.192:3300/0 conn(0x7f1de815d7b0 0x7f1de80b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:30:13.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.254+0000 7f1decd0a700 1 --2- 172.21.15.189:0/2386822097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1de815e500 0x7f1de806abb0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f1de80b6200 tx=0x7f1de000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebe1fc1ed35a820e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:13.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.254+0000 7f1de6ffd700 1 -- 172.21.15.189:0/2386822097 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de000e070 con 0x7f1de815e500 2024-04-06T12:30:13.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.254+0000 7f1de6ffd700 1 -- 172.21.15.189:0/2386822097 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de0003a20 con 0x7f1de815e500 2024-04-06T12:30:13.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.254+0000 7f1de6ffd700 1 -- 172.21.15.189:0/2386822097 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de0003dd0 con 0x7f1de815e500 2024-04-06T12:30:13.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.255+0000 7f1dee76d700 1 -- 172.21.15.189:0/2386822097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1de815e500 msgr2=0x7f1de806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:13.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.255+0000 7f1dee76d700 1 --2- 172.21.15.189:0/2386822097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1de815e500 0x7f1de806abb0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f1de80b6200 tx=0x7f1de000b330 comp rx=0 tx=0).stop 2024-04-06T12:30:13.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.255+0000 7f1dee76d700 1 -- 172.21.15.189:0/2386822097 shutdown_connections 2024-04-06T12:30:13.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.255+0000 7f1dee76d700 1 --2- 172.21.15.189:0/2386822097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1de815e500 0x7f1de806abb0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.255+0000 7f1dee76d700 1 --2- 172.21.15.189:0/2386822097 >> v2:172.21.15.192:3300/0 conn(0x7f1de815d7b0 0x7f1de80b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.255+0000 7f1dee76d700 1 -- 172.21.15.189:0/2386822097 >> 172.21.15.189:0/2386822097 conn(0x7f1de80d00b0 msgr2=0x7f1de80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:13.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.255+0000 7f1dee76d700 1 -- 172.21.15.189:0/2386822097 shutdown_connections 2024-04-06T12:30:13.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.255+0000 7f1dee76d700 1 -- 172.21.15.189:0/2386822097 wait complete. 2024-04-06T12:30:13.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.256+0000 7f1dee76d700 1 Processor -- start 2024-04-06T12:30:13.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.256+0000 7f1dee76d700 1 -- start start 2024-04-06T12:30:13.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.257+0000 7f1dee76d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1de806a730 0x7f1de8150090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:13.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.257+0000 7f1dee76d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1de815d7b0 0x7f1de815ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:13.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.257+0000 7f1dee76d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1de815e500 0x7f1de80ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:13.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.257+0000 7f1dee76d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de815c4f0 con 0x7f1de815e500 2024-04-06T12:30:13.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.257+0000 7f1dee76d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de815b8e0 con 0x7f1de815d7b0 2024-04-06T12:30:13.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.257+0000 7f1dee76d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de815bbc0 con 0x7f1de806a730 2024-04-06T12:30:13.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.257+0000 7f1decd0a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1de815e500 0x7f1de80ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:13.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.257+0000 7f1decd0a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1de815e500 0x7f1de80ccf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49636/0 (socket says 172.21.15.189:49636) 2024-04-06T12:30:13.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.257+0000 7f1decd0a700 1 -- 172.21.15.189:0/760787471 learned_addr learned my addr 172.21.15.189:0/760787471 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:13.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.257+0000 7f1de77fe700 1 --2- 172.21.15.189:0/760787471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1de815d7b0 0x7f1de815ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:13.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.257+0000 7f1decd0a700 1 -- 172.21.15.189:0/760787471 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1de806a730 msgr2=0x7f1de8150090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:13.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.257+0000 7f1decd0a700 1 --2- 172.21.15.189:0/760787471 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1de806a730 0x7f1de8150090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.257+0000 7f1decd0a700 1 -- 172.21.15.189:0/760787471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1de815d7b0 msgr2=0x7f1de815ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:13.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.257+0000 7f1decd0a700 1 --2- 172.21.15.189:0/760787471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1de815d7b0 0x7f1de815ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.257+0000 7f1decd0a700 1 -- 172.21.15.189:0/760787471 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de0009b30 con 0x7f1de815e500 2024-04-06T12:30:13.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.257+0000 7f1decd0a700 1 --2- 172.21.15.189:0/760787471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1de815e500 0x7f1de80ccf50 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f1de0015040 tx=0x7f1de0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:13.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.258+0000 7f1de4ff9700 1 -- 172.21.15.189:0/760787471 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de0004440 con 0x7f1de815e500 2024-04-06T12:30:13.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.258+0000 7f1de4ff9700 1 -- 172.21.15.189:0/760787471 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de00045a0 con 0x7f1de815e500 2024-04-06T12:30:13.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.258+0000 7f1dee76d700 1 -- 172.21.15.189:0/760787471 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de815bea0 con 0x7f1de815e500 2024-04-06T12:30:13.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.258+0000 7f1de4ff9700 1 -- 172.21.15.189:0/760787471 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de0004950 con 0x7f1de815e500 2024-04-06T12:30:13.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.258+0000 7f1dee76d700 1 -- 172.21.15.189:0/760787471 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de81505d0 con 0x7f1de815e500 2024-04-06T12:30:13.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.258+0000 7f1dee76d700 1 -- 172.21.15.189:0/760787471 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de8067bb0 con 0x7f1de815e500 2024-04-06T12:30:13.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.259+0000 7f1de4ff9700 1 -- 172.21.15.189:0/760787471 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f1de0012430 con 0x7f1de815e500 2024-04-06T12:30:13.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.259+0000 7f1de4ff9700 1 --2- 172.21.15.189:0/760787471 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dd406adb0 0x7f1dd406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:13.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.260+0000 7f1de4ff9700 1 -- 172.21.15.189:0/760787471 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f1de008a8e0 con 0x7f1de815e500 2024-04-06T12:30:13.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.260+0000 7f1de7fff700 1 --2- 172.21.15.189:0/760787471 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dd406adb0 0x7f1dd406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:13.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.260+0000 7f1de7fff700 1 --2- 172.21.15.189:0/760787471 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dd406adb0 0x7f1dd406d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8007920 tx=0x7f1dd8008040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:13.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.264+0000 7f1de4ff9700 1 -- 172.21.15.189:0/760787471 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1de0055ac0 con 0x7f1de815e500 2024-04-06T12:30:13.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.544+0000 7f1dee76d700 1 -- 172.21.15.189:0/760787471 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f1de8074990 con 0x7f1de815e500 2024-04-06T12:30:13.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.544+0000 7f1de4ff9700 1 -- 172.21.15.189:0/760787471 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f1de00590e0 con 0x7f1de815e500 2024-04-06T12:30:13.542 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:30:13.542 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T12:30:13.542 INFO:teuthology.orchestra.run.smithi189.stdout: "auth_dump": [ 2024-04-06T12:30:13.542 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:13.543 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.a", 2024-04-06T12:30:13.543 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmTjYMNBAA5voWGV4Pa7pE27VogrdipQ==", 2024-04-06T12:30:13.543 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:30:13.543 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:30:13.543 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:30:13.543 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:30:13.544 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:30:13.544 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.544 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:30:13.544 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:13.544 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.b", 2024-04-06T12:30:13.544 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm9A1OBRAAnHzubCvRUX8UdW8hmDzKbQ==", 2024-04-06T12:30:13.544 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:30:13.545 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:30:13.545 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:30:13.545 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:30:13.545 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:30:13.545 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.545 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:30:13.546 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:13.546 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.c", 2024-04-06T12:30:13.546 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmz0hhOxAAD3/GINNmGoFk+oPyiHHJXg==", 2024-04-06T12:30:13.546 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:30:13.546 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:30:13.546 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:30:13.547 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:30:13.547 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:30:13.547 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.547 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:30:13.547 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:13.547 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.d", 2024-04-06T12:30:13.547 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmhoAeDRAAoQC0vicWumBRTwpi3PtYaw==", 2024-04-06T12:30:13.548 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:30:13.548 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:30:13.548 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:30:13.548 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:30:13.548 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:30:13.548 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.548 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:30:13.549 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:13.549 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.0", 2024-04-06T12:30:13.549 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDkOxFmkxLWJhAAFXewCNOFX7dhAZVgUmewpQ==", 2024-04-06T12:30:13.549 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:30:13.549 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:30:13.549 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:30:13.550 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:30:13.550 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.550 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:30:13.550 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:13.550 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.1", 2024-04-06T12:30:13.550 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDnOxFmHYqWJRAABlsVCiv3e7YJMbfodJTWag==", 2024-04-06T12:30:13.551 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:30:13.551 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:30:13.551 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:30:13.551 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:30:13.551 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.551 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:30:13.551 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:13.551 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.2", 2024-04-06T12:30:13.552 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDqOxFmkhcvJBAA8NMEpHVGbGALjV9RISKP8g==", 2024-04-06T12:30:13.552 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:30:13.552 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:30:13.552 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:30:13.552 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:30:13.552 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.552 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:30:13.552 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:13.552 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.3", 2024-04-06T12:30:13.552 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDtOxFmDGzhIxAAamBoAbE8+lAel4os67meyQ==", 2024-04-06T12:30:13.552 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:30:13.552 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:30:13.552 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:30:13.552 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:30:13.552 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.553 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:30:13.553 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:13.553 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.4", 2024-04-06T12:30:13.553 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDzOxFmobZDJxAA9l8wdS81Eyw6+17I5ac6bw==", 2024-04-06T12:30:13.553 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:30:13.553 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:30:13.553 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:30:13.553 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:30:13.553 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.553 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:30:13.553 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:13.553 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.5", 2024-04-06T12:30:13.553 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD2OxFmi+ruJRAASlQ0o9jX/wA7U5AosnL+Wg==", 2024-04-06T12:30:13.553 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:30:13.554 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:30:13.554 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:30:13.554 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:30:13.554 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.554 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:30:13.554 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:13.554 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.6", 2024-04-06T12:30:13.554 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD5OxFmbwuuJBAA0wqJRmnJyskQPZJ7/LnpvA==", 2024-04-06T12:30:13.554 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:30:13.554 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:30:13.554 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:30:13.554 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:30:13.554 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.555 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:30:13.555 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:13.555 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.7", 2024-04-06T12:30:13.555 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD8OxFmDmRgIxAAzyW9AGAbmb67ZvB5qunwLQ==", 2024-04-06T12:30:13.555 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:30:13.555 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:30:13.555 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:30:13.555 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:30:13.555 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.555 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:30:13.555 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:13.555 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.0", 2024-04-06T12:30:13.555 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm0ByJERAAoEV8Yfc09+Go6u95XafElQ==", 2024-04-06T12:30:13.555 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:30:13.555 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:30:13.556 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow r", 2024-04-06T12:30:13.556 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-06T12:30:13.556 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.556 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:30:13.556 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:13.556 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.1", 2024-04-06T12:30:13.556 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmSV5lGBAAavDEo2LGy0IHpWSAuitHXg==", 2024-04-06T12:30:13.556 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:30:13.556 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:30:13.556 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:30:13.556 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:30:13.556 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:30:13.556 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.556 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:30:13.556 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:13.557 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.2", 2024-04-06T12:30:13.557 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm/Gz5FBAA3xX/rsAyWZ1JoUVlQlBIvQ==", 2024-04-06T12:30:13.557 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:30:13.557 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:30:13.557 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:30:13.557 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:30:13.557 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:30:13.557 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.557 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:30:13.557 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:13.557 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.3", 2024-04-06T12:30:13.557 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmzra7GxAADbnzIvWnRTo65LvPwtxWrg==", 2024-04-06T12:30:13.557 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:30:13.557 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:30:13.557 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:30:13.557 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:30:13.558 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:30:13.558 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.558 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:30:13.558 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:13.558 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.admin", 2024-04-06T12:30:13.558 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmxG+wCBAANg5kt6TOUQdj3/X0Mqgx1g==", 2024-04-06T12:30:13.558 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:30:13.558 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:30:13.558 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:30:13.558 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:30:13.558 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:30:13.558 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.558 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:30:13.558 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:13.558 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.x", 2024-04-06T12:30:13.559 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmPLyBMBAAYfhkI9Z1CPlf1Tijm+L7xA==", 2024-04-06T12:30:13.559 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:30:13.559 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:30:13.559 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T12:30:13.559 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:30:13.559 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.559 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:30:13.562 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:13.562 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.y", 2024-04-06T12:30:13.562 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFm1nHJHhAAQWRe8cU8duWuUJiYaDeTaQ==", 2024-04-06T12:30:13.562 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:30:13.563 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:30:13.563 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T12:30:13.563 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:30:13.563 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.563 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:13.563 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-04-06T12:30:13.563 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T12:30:13.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.546+0000 7f1dee76d700 1 -- 172.21.15.189:0/760787471 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dd406adb0 msgr2=0x7f1dd406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:13.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.546+0000 7f1dee76d700 1 --2- 172.21.15.189:0/760787471 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dd406adb0 0x7f1dd406d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8007920 tx=0x7f1dd8008040 comp rx=0 tx=0).stop 2024-04-06T12:30:13.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.546+0000 7f1dee76d700 1 -- 172.21.15.189:0/760787471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1de815e500 msgr2=0x7f1de80ccf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:13.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.546+0000 7f1dee76d700 1 --2- 172.21.15.189:0/760787471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1de815e500 0x7f1de80ccf50 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f1de0015040 tx=0x7f1de0004ad0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.546+0000 7f1dee76d700 1 -- 172.21.15.189:0/760787471 shutdown_connections 2024-04-06T12:30:13.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.546+0000 7f1dee76d700 1 --2- 172.21.15.189:0/760787471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1de815e500 0x7f1de80ccf50 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.546+0000 7f1dee76d700 1 --2- 172.21.15.189:0/760787471 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1de806a730 0x7f1de8150090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.546+0000 7f1dee76d700 1 --2- 172.21.15.189:0/760787471 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dd406adb0 0x7f1dd406d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.546+0000 7f1dee76d700 1 --2- 172.21.15.189:0/760787471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1de815d7b0 0x7f1de815ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.546+0000 7f1dee76d700 1 -- 172.21.15.189:0/760787471 >> 172.21.15.189:0/760787471 conn(0x7f1de80d00b0 msgr2=0x7f1de806eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:13.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.546+0000 7f1dee76d700 1 -- 172.21.15.189:0/760787471 shutdown_connections 2024-04-06T12:30:13.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.546+0000 7f1dee76d700 1 -- 172.21.15.189:0/760787471 wait complete. 2024-04-06T12:30:13.566 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:30:13.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.691+0000 7f8e29680700 1 Processor -- start 2024-04-06T12:30:13.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.691+0000 7f8e29680700 1 -- start start 2024-04-06T12:30:13.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.691+0000 7f8e29680700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8e240b3b80 0x7f8e24071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:13.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.691+0000 7f8e29680700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2415b510 0x7f8e24072320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:13.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.691+0000 7f8e29680700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e240b1fb0 con 0x7f8e2415b130 2024-04-06T12:30:13.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.691+0000 7f8e29680700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e24072860 con 0x7f8e240b3b80 2024-04-06T12:30:13.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.691+0000 7f8e29680700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e240b2200 con 0x7f8e2415b510 2024-04-06T12:30:13.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.692+0000 7f8e227fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8e240b3b80 0x7f8e24071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:13.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.692+0000 7f8e227fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8e240b3b80 0x7f8e24071de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49650/0 (socket says 172.21.15.189:49650) 2024-04-06T12:30:13.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.692+0000 7f8e227fc700 1 -- 172.21.15.189:0/4269825159 learned_addr learned my addr 172.21.15.189:0/4269825159 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:13.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.692+0000 7f8e227fc700 1 -- 172.21.15.189:0/4269825159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2415b510 msgr2=0x7f8e24072320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:13.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.692+0000 7f8e237fe700 1 --2- 172.21.15.189:0/4269825159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2415b510 0x7f8e24072320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:13.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.692+0000 7f8e227fc700 1 --2- 172.21.15.189:0/4269825159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2415b510 0x7f8e24072320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.692+0000 7f8e227fc700 1 -- 172.21.15.189:0/4269825159 >> v1:172.21.15.192:6789/0 conn(0x7f8e2415b130 legacy=0x7f8e240b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:13.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.692+0000 7f8e227fc700 1 -- 172.21.15.189:0/4269825159 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e240b24e0 con 0x7f8e240b3b80 2024-04-06T12:30:13.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.692+0000 7f8e227fc700 1 --2- 172.21.15.189:0/4269825159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e240b3b80 0x7f8e24071de0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00bbc0 tx=0x7f8e0c00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56db416ccddd1033 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:13.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.692+0000 7f8e21ffb700 1 -- 172.21.15.189:0/4269825159 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c00dd10 con 0x7f8e240b3b80 2024-04-06T12:30:13.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.692+0000 7f8e21ffb700 1 -- 172.21.15.189:0/4269825159 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0c00de70 con 0x7f8e240b3b80 2024-04-06T12:30:13.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.692+0000 7f8e21ffb700 1 -- 172.21.15.189:0/4269825159 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c004230 con 0x7f8e240b3b80 2024-04-06T12:30:13.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.693+0000 7f8e29680700 1 -- 172.21.15.189:0/4269825159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e240b3b80 msgr2=0x7f8e24071de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:13.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.693+0000 7f8e29680700 1 --2- 172.21.15.189:0/4269825159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e240b3b80 0x7f8e24071de0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00bbc0 tx=0x7f8e0c00bed0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.693+0000 7f8e29680700 1 -- 172.21.15.189:0/4269825159 shutdown_connections 2024-04-06T12:30:13.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.693+0000 7f8e29680700 1 --2- 172.21.15.189:0/4269825159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2415b510 0x7f8e24072320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.693+0000 7f8e29680700 1 --2- 172.21.15.189:0/4269825159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e240b3b80 0x7f8e24071de0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.693+0000 7f8e29680700 1 -- 172.21.15.189:0/4269825159 >> 172.21.15.189:0/4269825159 conn(0x7f8e2406a730 msgr2=0x7f8e2406ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:13.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.693+0000 7f8e29680700 1 -- 172.21.15.189:0/4269825159 shutdown_connections 2024-04-06T12:30:13.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.693+0000 7f8e29680700 1 -- 172.21.15.189:0/4269825159 wait complete. 2024-04-06T12:30:13.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.694+0000 7f8e29680700 1 Processor -- start 2024-04-06T12:30:13.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.694+0000 7f8e29680700 1 -- start start 2024-04-06T12:30:13.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.695+0000 7f8e29680700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e240b3b80 0x7f8e241521f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:13.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.695+0000 7f8e29680700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2415b130 0x7f8e240af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:13.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.695+0000 7f8e29680700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e2415b510 0x7f8e240afc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:13.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.695+0000 7f8e29680700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e240b0140 con 0x7f8e2415b510 2024-04-06T12:30:13.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.695+0000 7f8e29680700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e240ad400 con 0x7f8e240b3b80 2024-04-06T12:30:13.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.695+0000 7f8e29680700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e240ad6e0 con 0x7f8e2415b130 2024-04-06T12:30:13.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.695+0000 7f8e237fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e2415b510 0x7f8e240afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:13.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.695+0000 7f8e237fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e2415b510 0x7f8e240afc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49656/0 (socket says 172.21.15.189:49656) 2024-04-06T12:30:13.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.695+0000 7f8e237fe700 1 -- 172.21.15.189:0/1489101075 learned_addr learned my addr 172.21.15.189:0/1489101075 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:13.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.695+0000 7f8e22ffd700 1 --2- 172.21.15.189:0/1489101075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e240b3b80 0x7f8e241521f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:13.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.695+0000 7f8e227fc700 1 --2- 172.21.15.189:0/1489101075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2415b130 0x7f8e240af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:13.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.695+0000 7f8e237fe700 1 -- 172.21.15.189:0/1489101075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2415b130 msgr2=0x7f8e240af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:13.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.695+0000 7f8e237fe700 1 --2- 172.21.15.189:0/1489101075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2415b130 0x7f8e240af6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.695+0000 7f8e237fe700 1 -- 172.21.15.189:0/1489101075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e240b3b80 msgr2=0x7f8e241521f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:13.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.695+0000 7f8e237fe700 1 --2- 172.21.15.189:0/1489101075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e240b3b80 0x7f8e241521f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.695+0000 7f8e237fe700 1 -- 172.21.15.189:0/1489101075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0c00b7f0 con 0x7f8e2415b510 2024-04-06T12:30:13.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.695+0000 7f8e227fc700 1 --2- 172.21.15.189:0/1489101075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2415b130 0x7f8e240af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:13.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.695+0000 7f8e237fe700 1 --2- 172.21.15.189:0/1489101075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e2415b510 0x7f8e240afc00 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00cc00 tx=0x7f8e1c00cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:13.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.696+0000 7f8e1b7fe700 1 -- 172.21.15.189:0/1489101075 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c00aa80 con 0x7f8e2415b510 2024-04-06T12:30:13.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.696+0000 7f8e1b7fe700 1 -- 172.21.15.189:0/1489101075 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1c00abe0 con 0x7f8e2415b510 2024-04-06T12:30:13.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.696+0000 7f8e29680700 1 -- 172.21.15.189:0/1489101075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e240ad9c0 con 0x7f8e2415b510 2024-04-06T12:30:13.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.696+0000 7f8e1b7fe700 1 -- 172.21.15.189:0/1489101075 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c003a20 con 0x7f8e2415b510 2024-04-06T12:30:13.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.696+0000 7f8e29680700 1 -- 172.21.15.189:0/1489101075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e240adf80 con 0x7f8e2415b510 2024-04-06T12:30:13.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.697+0000 7f8e29680700 1 -- 172.21.15.189:0/1489101075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e24073270 con 0x7f8e2415b510 2024-04-06T12:30:13.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.697+0000 7f8e1b7fe700 1 -- 172.21.15.189:0/1489101075 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f8e1c00f3f0 con 0x7f8e2415b510 2024-04-06T12:30:13.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.698+0000 7f8e1b7fe700 1 --2- 172.21.15.189:0/1489101075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8e0406adb0 0x7f8e0406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:13.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.698+0000 7f8e22ffd700 1 --2- 172.21.15.189:0/1489101075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8e0406adb0 0x7f8e0406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:13.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.698+0000 7f8e1b7fe700 1 -- 172.21.15.189:0/1489101075 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f8e1c0894b0 con 0x7f8e2415b510 2024-04-06T12:30:13.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.698+0000 7f8e22ffd700 1 --2- 172.21.15.189:0/1489101075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8e0406adb0 0x7f8e0406d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8e14007fd0 tx=0x7f8e14007ec0 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:13.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.702+0000 7f8e1b7fe700 1 -- 172.21.15.189:0/1489101075 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8e1c054680 con 0x7f8e2415b510 2024-04-06T12:30:13.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.900+0000 7f8e29680700 1 -- 172.21.15.189:0/1489101075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e24074990 con 0x7f8e2415b510 2024-04-06T12:30:13.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.901+0000 7f8e1b7fe700 1 -- 172.21.15.189:0/1489101075 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+12652 (secure 0 0 0) 0x7f8e1c01d020 con 0x7f8e2415b510 2024-04-06T12:30:13.898 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:30:13.898 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":124,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:30:13.062093+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:30:13.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.903+0000 7f8e29680700 1 -- 172.21.15.189:0/1489101075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8e0406adb0 msgr2=0x7f8e0406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:13.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.903+0000 7f8e29680700 1 --2- 172.21.15.189:0/1489101075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8e0406adb0 0x7f8e0406d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8e14007fd0 tx=0x7f8e14007ec0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.903+0000 7f8e29680700 1 -- 172.21.15.189:0/1489101075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e2415b510 msgr2=0x7f8e240afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:13.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.903+0000 7f8e29680700 1 --2- 172.21.15.189:0/1489101075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e2415b510 0x7f8e240afc00 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00cc00 tx=0x7f8e1c00cf10 comp rx=0 tx=0).stop 2024-04-06T12:30:13.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.903+0000 7f8e29680700 1 -- 172.21.15.189:0/1489101075 shutdown_connections 2024-04-06T12:30:13.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.903+0000 7f8e29680700 1 --2- 172.21.15.189:0/1489101075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e2415b510 0x7f8e240afc00 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.903+0000 7f8e29680700 1 --2- 172.21.15.189:0/1489101075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8e0406adb0 0x7f8e0406d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.903+0000 7f8e29680700 1 --2- 172.21.15.189:0/1489101075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e240b3b80 0x7f8e241521f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.903+0000 7f8e29680700 1 --2- 172.21.15.189:0/1489101075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2415b130 0x7f8e240af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:13.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.903+0000 7f8e29680700 1 -- 172.21.15.189:0/1489101075 >> 172.21.15.189:0/1489101075 conn(0x7f8e2406a730 msgr2=0x7f8e2415de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:13.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.903+0000 7f8e29680700 1 -- 172.21.15.189:0/1489101075 shutdown_connections 2024-04-06T12:30:13.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:13.903+0000 7f8e29680700 1 -- 172.21.15.189:0/1489101075 wait complete. 2024-04-06T12:30:13.913 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:30:13.913 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:30:14.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.041+0000 7f10db908700 1 Processor -- start 2024-04-06T12:30:14.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.041+0000 7f10db908700 1 -- start start 2024-04-06T12:30:14.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.042+0000 7f10db908700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10d40ce730 0x7f10d415dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.042+0000 7f10db908700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f10d40b8d70 0x7f10d40d1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.042+0000 7f10db908700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d415eac0 con 0x7f10d415d6f0 2024-04-06T12:30:14.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.042+0000 7f10db908700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d415e010 con 0x7f10d40b8d70 2024-04-06T12:30:14.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.042+0000 7f10db908700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d415e220 con 0x7f10d40ce730 2024-04-06T12:30:14.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.042+0000 7f10d96a4700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f10d415d6f0 0x7f10d40ce020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51152/0 (socket says 172.21.15.189:51152) 2024-04-06T12:30:14.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.042+0000 7f10d96a4700 1 -- 172.21.15.189:0/2412582723 learned_addr learned my addr 172.21.15.189:0/2412582723 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:14.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.042+0000 7f10d8ea3700 1 --2- 172.21.15.189:0/2412582723 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10d40ce730 0x7f10d415dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:14.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.042+0000 7f10d9ea5700 1 --2- 172.21.15.189:0/2412582723 >> v2:172.21.15.192:3300/0 conn(0x7f10d40b8d70 0x7f10d40d1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:14.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.042+0000 7f10cbfff700 1 -- 172.21.15.189:0/2412582723 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66234963 0 0) 0x7f10d415eac0 con 0x7f10d415d6f0 2024-04-06T12:30:14.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.042+0000 7f10cbfff700 1 -- 172.21.15.189:0/2412582723 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c00035b0 con 0x7f10d415d6f0 2024-04-06T12:30:14.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10cbfff700 1 -- 172.21.15.189:0/2412582723 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2783119327 0 0) 0x7f10c00035b0 con 0x7f10d415d6f0 2024-04-06T12:30:14.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10cbfff700 1 -- 172.21.15.189:0/2412582723 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10d40ce730 msgr2=0x7f10d415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:14.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10cbfff700 1 --2- 172.21.15.189:0/2412582723 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10d40ce730 0x7f10d415dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10cbfff700 1 -- 172.21.15.189:0/2412582723 >> v2:172.21.15.192:3300/0 conn(0x7f10d40b8d70 msgr2=0x7f10d40d1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:14.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10cbfff700 1 --2- 172.21.15.189:0/2412582723 >> v2:172.21.15.192:3300/0 conn(0x7f10d40b8d70 0x7f10d40d1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10cbfff700 1 -- 172.21.15.189:0/2412582723 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d415e500 con 0x7f10d415d6f0 2024-04-06T12:30:14.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10cbfff700 1 -- 172.21.15.189:0/2412582723 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f10d0001670 con 0x7f10d415d6f0 2024-04-06T12:30:14.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10cbfff700 1 -- 172.21.15.189:0/2412582723 >> v1:172.21.15.189:6789/0 conn(0x7f10d415d6f0 legacy=0x7f10d40ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:14.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10cbfff700 1 --2- 172.21.15.189:0/2412582723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10c0003a00 0x7f10c0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10cbfff700 1 --2- 172.21.15.189:0/2412582723 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10d40ce730 0x7f10c0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10cbfff700 1 --2- 172.21.15.189:0/2412582723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10c0004d90 0x7f10c0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10cbfff700 1 -- 172.21.15.189:0/2412582723 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d415e010 con 0x7f10c0004d90 2024-04-06T12:30:14.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10cbfff700 1 -- 172.21.15.189:0/2412582723 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d415e220 con 0x7f10c0003a00 2024-04-06T12:30:14.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10cbfff700 1 -- 172.21.15.189:0/2412582723 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c000ba60 con 0x7f10d40ce730 2024-04-06T12:30:14.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10d9ea5700 1 --2- 172.21.15.189:0/2412582723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10c0004d90 0x7f10c0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:14.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10d8ea3700 1 --2- 172.21.15.189:0/2412582723 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10d40ce730 0x7f10c0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:14.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10d96a4700 1 --2- 172.21.15.189:0/2412582723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10c0003a00 0x7f10c0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:14.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10d9ea5700 1 -- 172.21.15.189:0/2412582723 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10d40ce730 msgr2=0x7f10c0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:14.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10d9ea5700 1 --2- 172.21.15.189:0/2412582723 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10d40ce730 0x7f10c0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10d9ea5700 1 -- 172.21.15.189:0/2412582723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10c0003a00 msgr2=0x7f10c0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:14.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10d9ea5700 1 --2- 172.21.15.189:0/2412582723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10c0003a00 0x7f10c0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10d9ea5700 1 -- 172.21.15.189:0/2412582723 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c00035b0 con 0x7f10c0004d90 2024-04-06T12:30:14.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10d96a4700 1 --2- 172.21.15.189:0/2412582723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10c0003a00 0x7f10c0003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:14.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10d9ea5700 1 --2- 172.21.15.189:0/2412582723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10c0004d90 0x7f10c0005210 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f10cc00e170 tx=0x7f10c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58c8a4a75642b096 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:14.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10cbfff700 1 -- 172.21.15.189:0/2412582723 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc00ec90 con 0x7f10c0004d90 2024-04-06T12:30:14.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10cbfff700 1 -- 172.21.15.189:0/2412582723 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10cc00f040 con 0x7f10c0004d90 2024-04-06T12:30:14.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.043+0000 7f10cbfff700 1 -- 172.21.15.189:0/2412582723 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc014480 con 0x7f10c0004d90 2024-04-06T12:30:14.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.044+0000 7f10db908700 1 -- 172.21.15.189:0/2412582723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10c0004d90 msgr2=0x7f10c0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:14.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.044+0000 7f10db908700 1 --2- 172.21.15.189:0/2412582723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10c0004d90 0x7f10c0005210 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f10cc00e170 tx=0x7f10c0005880 comp rx=0 tx=0).stop 2024-04-06T12:30:14.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.044+0000 7f10d9ea5700 1 -- 172.21.15.189:0/2412582723 reap_dead start 2024-04-06T12:30:14.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.044+0000 7f10db908700 1 -- 172.21.15.189:0/2412582723 shutdown_connections 2024-04-06T12:30:14.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.044+0000 7f10db908700 1 -- 172.21.15.189:0/2412582723 >> 172.21.15.189:0/2412582723 conn(0x7f10d4072490 msgr2=0x7f10d406b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:14.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.044+0000 7f10db908700 1 -- 172.21.15.189:0/2412582723 shutdown_connections 2024-04-06T12:30:14.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.044+0000 7f10db908700 1 -- 172.21.15.189:0/2412582723 wait complete. 2024-04-06T12:30:14.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.045+0000 7f10db908700 1 Processor -- start 2024-04-06T12:30:14.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.045+0000 7f10db908700 1 -- start start 2024-04-06T12:30:14.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.046+0000 7f10db908700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10c0004d90 0x7f10d41f5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.046+0000 7f10db908700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10d40b7b50 0x7f10d40b7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.046+0000 7f10db908700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10d40cbb90 0x7f10d40cc020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.046+0000 7f10db908700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d40b84f0 con 0x7f10d40cbb90 2024-04-06T12:30:14.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.046+0000 7f10db908700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d41543f0 con 0x7f10c0004d90 2024-04-06T12:30:14.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.046+0000 7f10db908700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d40cc8c0 con 0x7f10d40b7b50 2024-04-06T12:30:14.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.046+0000 7f10d9ea5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10d40cbb90 0x7f10d40cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:14.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.046+0000 7f10d9ea5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10d40cbb90 0x7f10d40cc020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49668/0 (socket says 172.21.15.189:49668) 2024-04-06T12:30:14.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.046+0000 7f10d9ea5700 1 -- 172.21.15.189:0/431544724 learned_addr learned my addr 172.21.15.189:0/431544724 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:14.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.046+0000 7f10d96a4700 1 --2- 172.21.15.189:0/431544724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10c0004d90 0x7f10d41f5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:14.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.046+0000 7f10d8ea3700 1 --2- 172.21.15.189:0/431544724 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10d40b7b50 0x7f10d40b7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:14.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.046+0000 7f10d9ea5700 1 -- 172.21.15.189:0/431544724 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10d40b7b50 msgr2=0x7f10d40b7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:14.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.046+0000 7f10d9ea5700 1 --2- 172.21.15.189:0/431544724 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10d40b7b50 0x7f10d40b7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.046+0000 7f10d9ea5700 1 -- 172.21.15.189:0/431544724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10c0004d90 msgr2=0x7f10d41f5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:14.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.046+0000 7f10d9ea5700 1 --2- 172.21.15.189:0/431544724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10c0004d90 0x7f10d41f5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.046+0000 7f10d9ea5700 1 -- 172.21.15.189:0/431544724 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d40ccba0 con 0x7f10d40cbb90 2024-04-06T12:30:14.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.047+0000 7f10d9ea5700 1 --2- 172.21.15.189:0/431544724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10d40cbb90 0x7f10d40cc020 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f10cc016000 tx=0x7f10cc004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:14.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.047+0000 7f10c9ffb700 1 -- 172.21.15.189:0/431544724 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc00ee80 con 0x7f10d40cbb90 2024-04-06T12:30:14.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.047+0000 7f10c9ffb700 1 -- 172.21.15.189:0/431544724 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10cc003a20 con 0x7f10d40cbb90 2024-04-06T12:30:14.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.047+0000 7f10db908700 1 -- 172.21.15.189:0/431544724 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d40cce80 con 0x7f10d40cbb90 2024-04-06T12:30:14.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.047+0000 7f10c9ffb700 1 -- 172.21.15.189:0/431544724 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc003dd0 con 0x7f10d40cbb90 2024-04-06T12:30:14.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.047+0000 7f10db908700 1 -- 172.21.15.189:0/431544724 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d40cc560 con 0x7f10d40cbb90 2024-04-06T12:30:14.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.048+0000 7f10db908700 1 -- 172.21.15.189:0/431544724 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d4074990 con 0x7f10d40cbb90 2024-04-06T12:30:14.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.048+0000 7f10c9ffb700 1 -- 172.21.15.189:0/431544724 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f10cc024020 con 0x7f10d40cbb90 2024-04-06T12:30:14.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.049+0000 7f10c9ffb700 1 --2- 172.21.15.189:0/431544724 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10b806ade0 0x7f10b806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.049+0000 7f10c9ffb700 1 -- 172.21.15.189:0/431544724 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f10cc08ab60 con 0x7f10d40cbb90 2024-04-06T12:30:14.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.049+0000 7f10d96a4700 1 --2- 172.21.15.189:0/431544724 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10b806ade0 0x7f10b806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:14.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.049+0000 7f10d96a4700 1 --2- 172.21.15.189:0/431544724 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10b806ade0 0x7f10b806d2a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f10d0000c00 tx=0x7f10d0009040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:14.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.053+0000 7f10c9ffb700 1 -- 172.21.15.189:0/431544724 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f10cc055f50 con 0x7f10d40cbb90 2024-04-06T12:30:14.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.244+0000 7f10db908700 1 -- 172.21.15.189:0/431544724 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f10d40b7630 con 0x7f10d40cbb90 2024-04-06T12:30:14.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.244+0000 7f10c9ffb700 1 -- 172.21.15.189:0/431544724 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v124) v1 ==== 93+0+10 (secure 0 0 0) 0x7f10cc059570 con 0x7f10d40cbb90 2024-04-06T12:30:14.242 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:30:14.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.246+0000 7f10db908700 1 -- 172.21.15.189:0/431544724 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10b806ade0 msgr2=0x7f10b806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:14.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.246+0000 7f10db908700 1 --2- 172.21.15.189:0/431544724 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10b806ade0 0x7f10b806d2a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f10d0000c00 tx=0x7f10d0009040 comp rx=0 tx=0).stop 2024-04-06T12:30:14.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.246+0000 7f10db908700 1 -- 172.21.15.189:0/431544724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10d40cbb90 msgr2=0x7f10d40cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:14.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.246+0000 7f10db908700 1 --2- 172.21.15.189:0/431544724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10d40cbb90 0x7f10d40cc020 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f10cc016000 tx=0x7f10cc004b40 comp rx=0 tx=0).stop 2024-04-06T12:30:14.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.246+0000 7f10db908700 1 -- 172.21.15.189:0/431544724 shutdown_connections 2024-04-06T12:30:14.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.247+0000 7f10db908700 1 --2- 172.21.15.189:0/431544724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10d40cbb90 0x7f10d40cc020 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.247+0000 7f10db908700 1 --2- 172.21.15.189:0/431544724 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10b806ade0 0x7f10b806d2a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.247+0000 7f10db908700 1 --2- 172.21.15.189:0/431544724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10c0004d90 0x7f10d41f5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.247+0000 7f10db908700 1 --2- 172.21.15.189:0/431544724 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10d40b7b50 0x7f10d40b7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.247+0000 7f10db908700 1 -- 172.21.15.189:0/431544724 >> 172.21.15.189:0/431544724 conn(0x7f10d4072490 msgr2=0x7f10d40b6730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:14.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.247+0000 7f10db908700 1 -- 172.21.15.189:0/431544724 shutdown_connections 2024-04-06T12:30:14.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.247+0000 7f10db908700 1 -- 172.21.15.189:0/431544724 wait complete. 2024-04-06T12:30:14.256 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-06T12:30:14.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.381+0000 7fd459a68700 1 Processor -- start 2024-04-06T12:30:14.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.381+0000 7fd459a68700 1 -- start start 2024-04-06T12:30:14.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.381+0000 7fd459a68700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd45415a550 0x7fd4540cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.381+0000 7fd459a68700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4540cdbe0 con 0x7fd4540b4d70 2024-04-06T12:30:14.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.381+0000 7fd459a68700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd45415a940 con 0x7fd4540cf900 2024-04-06T12:30:14.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.381+0000 7fd459a68700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45415ac00 con 0x7fd45415a550 2024-04-06T12:30:14.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.382+0000 7fd4537fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd45415a550 0x7fd4540cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:14.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.382+0000 7fd453fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd4540b4d70 0x7fd4540b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51162/0 (socket says 172.21.15.189:51162) 2024-04-06T12:30:14.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.382+0000 7fd453fff700 1 -- 172.21.15.189:0/2201204674 learned_addr learned my addr 172.21.15.189:0/2201204674 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:14.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.382+0000 7fd4537fe700 1 -- 172.21.15.189:0/2201204674 >> v1:172.21.15.192:6789/0 conn(0x7fd4540cf900 legacy=0x7fd4540b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:14.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.382+0000 7fd4537fe700 1 -- 172.21.15.189:0/2201204674 >> v1:172.21.15.189:6789/0 conn(0x7fd4540b4d70 legacy=0x7fd4540b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:14.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.382+0000 7fd4527fc700 1 -- 172.21.15.189:0/2201204674 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3933263707 0 0) 0x7fd4540cdbe0 con 0x7fd4540b4d70 2024-04-06T12:30:14.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.382+0000 7fd4537fe700 1 -- 172.21.15.189:0/2201204674 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45415aee0 con 0x7fd45415a550 2024-04-06T12:30:14.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.382+0000 7fd4537fe700 1 --2- 172.21.15.189:0/2201204674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45415a550 0x7fd4540cf3c0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fd444009840 tx=0x7fd444009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfba702a134da1c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:14.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.382+0000 7fd4527fc700 1 -- 172.21.15.189:0/2201204674 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd444004230 con 0x7fd45415a550 2024-04-06T12:30:14.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.382+0000 7fd4527fc700 1 -- 172.21.15.189:0/2201204674 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd444004390 con 0x7fd45415a550 2024-04-06T12:30:14.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.382+0000 7fd4527fc700 1 -- 172.21.15.189:0/2201204674 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4440036a0 con 0x7fd45415a550 2024-04-06T12:30:14.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.383+0000 7fd459a68700 1 -- 172.21.15.189:0/2201204674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45415a550 msgr2=0x7fd4540cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:14.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.383+0000 7fd459a68700 1 --2- 172.21.15.189:0/2201204674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45415a550 0x7fd4540cf3c0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fd444009840 tx=0x7fd444009b50 comp rx=0 tx=0).stop 2024-04-06T12:30:14.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.383+0000 7fd459a68700 1 -- 172.21.15.189:0/2201204674 shutdown_connections 2024-04-06T12:30:14.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.383+0000 7fd459a68700 1 --2- 172.21.15.189:0/2201204674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45415a550 0x7fd4540cf3c0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.383+0000 7fd459a68700 1 -- 172.21.15.189:0/2201204674 >> 172.21.15.189:0/2201204674 conn(0x7fd4540d00b0 msgr2=0x7fd4540d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:14.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.383+0000 7fd459a68700 1 -- 172.21.15.189:0/2201204674 shutdown_connections 2024-04-06T12:30:14.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.383+0000 7fd459a68700 1 -- 172.21.15.189:0/2201204674 wait complete. 2024-04-06T12:30:14.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.384+0000 7fd459a68700 1 Processor -- start 2024-04-06T12:30:14.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.384+0000 7fd459a68700 1 -- start start 2024-04-06T12:30:14.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.385+0000 7fd459a68700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4540b4d70 0x7fd4541f6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.385+0000 7fd459a68700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4540cf900 0x7fd4540ad250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.385+0000 7fd459a68700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4540b1020 0x7fd454153a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.385+0000 7fd459a68700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4540c5040 con 0x7fd4540b1020 2024-04-06T12:30:14.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.385+0000 7fd459a68700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4540ad790 con 0x7fd4540b4d70 2024-04-06T12:30:14.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.385+0000 7fd459a68700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4541f3710 con 0x7fd4540cf900 2024-04-06T12:30:14.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.385+0000 7fd453fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4540b1020 0x7fd454153a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:14.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.385+0000 7fd453fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4540b1020 0x7fd454153a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49692/0 (socket says 172.21.15.189:49692) 2024-04-06T12:30:14.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.385+0000 7fd453fff700 1 -- 172.21.15.189:0/1247480229 learned_addr learned my addr 172.21.15.189:0/1247480229 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:14.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.385+0000 7fd4537fe700 1 --2- 172.21.15.189:0/1247480229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4540b4d70 0x7fd4541f6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:14.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.385+0000 7fd452ffd700 1 --2- 172.21.15.189:0/1247480229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4540cf900 0x7fd4540ad250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:14.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.385+0000 7fd453fff700 1 -- 172.21.15.189:0/1247480229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4540cf900 msgr2=0x7fd4540ad250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:14.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.385+0000 7fd453fff700 1 --2- 172.21.15.189:0/1247480229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4540cf900 0x7fd4540ad250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.385+0000 7fd453fff700 1 -- 172.21.15.189:0/1247480229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4540b4d70 msgr2=0x7fd4541f6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:14.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.385+0000 7fd453fff700 1 --2- 172.21.15.189:0/1247480229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4540b4d70 0x7fd4541f6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.385+0000 7fd453fff700 1 -- 172.21.15.189:0/1247480229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4440094a0 con 0x7fd4540b1020 2024-04-06T12:30:14.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.385+0000 7fd4537fe700 1 --2- 172.21.15.189:0/1247480229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4540b4d70 0x7fd4541f6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:30:14.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.385+0000 7fd452ffd700 1 --2- 172.21.15.189:0/1247480229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4540cf900 0x7fd4540ad250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:30:14.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.385+0000 7fd453fff700 1 --2- 172.21.15.189:0/1247480229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4540b1020 0x7fd454153a10 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fd44c00cca0 tx=0x7fd44c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:14.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.386+0000 7fd43bfff700 1 -- 172.21.15.189:0/1247480229 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44c00cea0 con 0x7fd4540b1020 2024-04-06T12:30:14.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.386+0000 7fd43bfff700 1 -- 172.21.15.189:0/1247480229 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd44c00ade0 con 0x7fd4540b1020 2024-04-06T12:30:14.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.386+0000 7fd43bfff700 1 -- 172.21.15.189:0/1247480229 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44c007310 con 0x7fd4540b1020 2024-04-06T12:30:14.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.386+0000 7fd459a68700 1 -- 172.21.15.189:0/1247480229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4541f2e20 con 0x7fd4540b1020 2024-04-06T12:30:14.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.386+0000 7fd459a68700 1 -- 172.21.15.189:0/1247480229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4540ad9c0 con 0x7fd4540b1020 2024-04-06T12:30:14.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.387+0000 7fd459a68700 1 -- 172.21.15.189:0/1247480229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4541f31d0 con 0x7fd4540b1020 2024-04-06T12:30:14.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.387+0000 7fd43bfff700 1 -- 172.21.15.189:0/1247480229 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fd44c007470 con 0x7fd4540b1020 2024-04-06T12:30:14.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.387+0000 7fd43bfff700 1 --2- 172.21.15.189:0/1247480229 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd44006adb0 0x7fd44006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.388+0000 7fd4537fe700 1 --2- 172.21.15.189:0/1247480229 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd44006adb0 0x7fd44006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:14.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.388+0000 7fd43bfff700 1 -- 172.21.15.189:0/1247480229 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fd44c088f20 con 0x7fd4540b1020 2024-04-06T12:30:14.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.388+0000 7fd4537fe700 1 --2- 172.21.15.189:0/1247480229 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd44006adb0 0x7fd44006d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd4440090b0 tx=0x7fd444013040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:14.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.392+0000 7fd43bfff700 1 -- 172.21.15.189:0/1247480229 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd44c054200 con 0x7fd4540b1020 2024-04-06T12:30:14.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.590+0000 7fd459a68700 1 -- 172.21.15.189:0/1247480229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd454067bb0 con 0x7fd4540b1020 2024-04-06T12:30:14.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.590+0000 7fd43bfff700 1 -- 172.21.15.189:0/1247480229 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v124) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd44c018020 con 0x7fd4540b1020 2024-04-06T12:30:14.587 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:30:14.588 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T12:30:14.588 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:14.588 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 1, 2024-04-06T12:30:14.588 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": ".mgr" 2024-04-06T12:30:14.588 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:14.588 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T12:30:14.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.592+0000 7fd459a68700 1 -- 172.21.15.189:0/1247480229 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd44006adb0 msgr2=0x7fd44006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:14.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.592+0000 7fd459a68700 1 --2- 172.21.15.189:0/1247480229 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd44006adb0 0x7fd44006d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd4440090b0 tx=0x7fd444013040 comp rx=0 tx=0).stop 2024-04-06T12:30:14.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.592+0000 7fd459a68700 1 -- 172.21.15.189:0/1247480229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4540b1020 msgr2=0x7fd454153a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:14.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.592+0000 7fd459a68700 1 --2- 172.21.15.189:0/1247480229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4540b1020 0x7fd454153a10 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fd44c00cca0 tx=0x7fd44c00a340 comp rx=0 tx=0).stop 2024-04-06T12:30:14.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.592+0000 7fd459a68700 1 -- 172.21.15.189:0/1247480229 shutdown_connections 2024-04-06T12:30:14.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.592+0000 7fd459a68700 1 --2- 172.21.15.189:0/1247480229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4540b1020 0x7fd454153a10 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.592+0000 7fd459a68700 1 --2- 172.21.15.189:0/1247480229 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd44006adb0 0x7fd44006d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.592+0000 7fd459a68700 1 --2- 172.21.15.189:0/1247480229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4540b4d70 0x7fd4541f6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.592+0000 7fd459a68700 1 --2- 172.21.15.189:0/1247480229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4540cf900 0x7fd4540ad250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.592+0000 7fd459a68700 1 -- 172.21.15.189:0/1247480229 >> 172.21.15.189:0/1247480229 conn(0x7fd4540d00b0 msgr2=0x7fd4540b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:14.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.592+0000 7fd459a68700 1 -- 172.21.15.189:0/1247480229 shutdown_connections 2024-04-06T12:30:14.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.592+0000 7fd459a68700 1 -- 172.21.15.189:0/1247480229 wait complete. 2024-04-06T12:30:14.602 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-06T12:30:14.602 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-06T12:30:14.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.731+0000 7f0a9b7bc700 1 Processor -- start 2024-04-06T12:30:14.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.731+0000 7f0a9b7bc700 1 -- start start 2024-04-06T12:30:14.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.732+0000 7f0a9b7bc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0a9415d7b0 0x7f0a940b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.732+0000 7f0a9b7bc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0a9406a730 0x7f0a9406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.732+0000 7f0a9b7bc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a940b3990 con 0x7f0a9415ddf0 2024-04-06T12:30:14.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.732+0000 7f0a9b7bc700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a9415e950 con 0x7f0a9415d7b0 2024-04-06T12:30:14.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.732+0000 7f0a9b7bc700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a9406b0f0 con 0x7f0a9406a730 2024-04-06T12:30:14.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.732+0000 7f0a99558700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0a9415d7b0 0x7f0a940b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:14.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.732+0000 7f0a99d59700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0a9415ddf0 0x7f0a9415e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51170/0 (socket says 172.21.15.189:51170) 2024-04-06T12:30:14.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.732+0000 7f0a99d59700 1 -- 172.21.15.189:0/508808952 learned_addr learned my addr 172.21.15.189:0/508808952 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:14.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.732+0000 7f0a99558700 1 -- 172.21.15.189:0/508808952 >> v2:172.21.15.192:3300/0 conn(0x7f0a9406a730 msgr2=0x7f0a9406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:14.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.732+0000 7f0a99558700 1 --2- 172.21.15.189:0/508808952 >> v2:172.21.15.192:3300/0 conn(0x7f0a9406a730 0x7f0a9406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.732+0000 7f0a99558700 1 -- 172.21.15.189:0/508808952 >> v1:172.21.15.189:6789/0 conn(0x7f0a9415ddf0 legacy=0x7f0a9415e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:14.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.732+0000 7f0a99558700 1 -- 172.21.15.189:0/508808952 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a940b3060 con 0x7f0a9415d7b0 2024-04-06T12:30:14.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.732+0000 7f0a8bfff700 1 -- 172.21.15.189:0/508808952 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240897356 0 0) 0x7f0a940b3990 con 0x7f0a9415ddf0 2024-04-06T12:30:14.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.732+0000 7f0a99558700 1 --2- 172.21.15.189:0/508808952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a9415d7b0 0x7f0a940b3e20 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f0a940afc00 tx=0x7f0a84009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a92d6fc1a7550d28 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:14.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.733+0000 7f0a8bfff700 1 -- 172.21.15.189:0/508808952 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a84011830 con 0x7f0a9415d7b0 2024-04-06T12:30:14.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.733+0000 7f0a8bfff700 1 -- 172.21.15.189:0/508808952 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a84011990 con 0x7f0a9415d7b0 2024-04-06T12:30:14.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.733+0000 7f0a8bfff700 1 -- 172.21.15.189:0/508808952 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a84011d20 con 0x7f0a9415d7b0 2024-04-06T12:30:14.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.733+0000 7f0a9b7bc700 1 -- 172.21.15.189:0/508808952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a9415d7b0 msgr2=0x7f0a940b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:14.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.733+0000 7f0a9b7bc700 1 --2- 172.21.15.189:0/508808952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a9415d7b0 0x7f0a940b3e20 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f0a940afc00 tx=0x7f0a84009b20 comp rx=0 tx=0).stop 2024-04-06T12:30:14.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.733+0000 7f0a9b7bc700 1 -- 172.21.15.189:0/508808952 shutdown_connections 2024-04-06T12:30:14.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.733+0000 7f0a9b7bc700 1 --2- 172.21.15.189:0/508808952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a9415d7b0 0x7f0a940b3e20 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.733+0000 7f0a9b7bc700 1 --2- 172.21.15.189:0/508808952 >> v2:172.21.15.192:3300/0 conn(0x7f0a9406a730 0x7f0a9406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.733+0000 7f0a9b7bc700 1 -- 172.21.15.189:0/508808952 >> 172.21.15.189:0/508808952 conn(0x7f0a940d00b0 msgr2=0x7f0a940d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:14.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.733+0000 7f0a9b7bc700 1 -- 172.21.15.189:0/508808952 shutdown_connections 2024-04-06T12:30:14.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.733+0000 7f0a9b7bc700 1 -- 172.21.15.189:0/508808952 wait complete. 2024-04-06T12:30:14.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.734+0000 7f0a9b7bc700 1 Processor -- start 2024-04-06T12:30:14.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.735+0000 7f0a9b7bc700 1 -- start start 2024-04-06T12:30:14.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.735+0000 7f0a9b7bc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a9406a730 0x7f0a9415ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.735+0000 7f0a9b7bc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a9415d7b0 0x7f0a94149c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.735+0000 7f0a9b7bc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a9415ddf0 0x7f0a9415c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.735+0000 7f0a9b7bc700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9415bfe0 con 0x7f0a9406a730 2024-04-06T12:30:14.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.735+0000 7f0a9b7bc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9414a1d0 con 0x7f0a9415ddf0 2024-04-06T12:30:14.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.735+0000 7f0a9b7bc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a940cf350 con 0x7f0a9415d7b0 2024-04-06T12:30:14.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.735+0000 7f0a99558700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a9406a730 0x7f0a9415ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:14.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.735+0000 7f0a99558700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a9406a730 0x7f0a9415ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49710/0 (socket says 172.21.15.189:49710) 2024-04-06T12:30:14.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.735+0000 7f0a99558700 1 -- 172.21.15.189:0/1669877628 learned_addr learned my addr 172.21.15.189:0/1669877628 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.736+0000 7f0a99558700 1 -- 172.21.15.189:0/1669877628 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a9415d7b0 msgr2=0x7f0a94149c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.736+0000 7f0a98d57700 1 --2- 172.21.15.189:0/1669877628 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a9415d7b0 0x7f0a94149c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.736+0000 7f0a99d59700 1 --2- 172.21.15.189:0/1669877628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a9415ddf0 0x7f0a9415c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.736+0000 7f0a99558700 1 --2- 172.21.15.189:0/1669877628 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a9415d7b0 0x7f0a94149c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.736+0000 7f0a99558700 1 -- 172.21.15.189:0/1669877628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a9415ddf0 msgr2=0x7f0a9415c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.736+0000 7f0a99558700 1 --2- 172.21.15.189:0/1669877628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a9415ddf0 0x7f0a9415c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.736+0000 7f0a99558700 1 -- 172.21.15.189:0/1669877628 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a840094a0 con 0x7f0a9406a730 2024-04-06T12:30:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.736+0000 7f0a99d59700 1 --2- 172.21.15.189:0/1669877628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a9415ddf0 0x7f0a9415c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:30:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.736+0000 7f0a98d57700 1 --2- 172.21.15.189:0/1669877628 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a9415d7b0 0x7f0a94149c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:30:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.736+0000 7f0a99558700 1 --2- 172.21.15.189:0/1669877628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a9406a730 0x7f0a9415ac80 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f0a840090b0 tx=0x7f0a84004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.736+0000 7f0a89ffb700 1 -- 172.21.15.189:0/1669877628 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a840038a0 con 0x7f0a9406a730 2024-04-06T12:30:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.736+0000 7f0a89ffb700 1 -- 172.21.15.189:0/1669877628 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8401f070 con 0x7f0a9406a730 2024-04-06T12:30:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.736+0000 7f0a9b7bc700 1 -- 172.21.15.189:0/1669877628 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a9415b360 con 0x7f0a9406a730 2024-04-06T12:30:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.736+0000 7f0a89ffb700 1 -- 172.21.15.189:0/1669877628 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a84003c60 con 0x7f0a9406a730 2024-04-06T12:30:14.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.736+0000 7f0a9b7bc700 1 -- 172.21.15.189:0/1669877628 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a9415b920 con 0x7f0a9406a730 2024-04-06T12:30:14.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.737+0000 7f0a9b7bc700 1 -- 172.21.15.189:0/1669877628 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a94067bb0 con 0x7f0a9406a730 2024-04-06T12:30:14.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.738+0000 7f0a89ffb700 1 -- 172.21.15.189:0/1669877628 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0a84026480 con 0x7f0a9406a730 2024-04-06T12:30:14.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.738+0000 7f0a89ffb700 1 --2- 172.21.15.189:0/1669877628 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0a7806adb0 0x7f0a7806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:14.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.738+0000 7f0a89ffb700 1 -- 172.21.15.189:0/1669877628 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f0a84021070 con 0x7f0a9406a730 2024-04-06T12:30:14.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.738+0000 7f0a98d57700 1 --2- 172.21.15.189:0/1669877628 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0a7806adb0 0x7f0a7806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:14.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.739+0000 7f0a98d57700 1 --2- 172.21.15.189:0/1669877628 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0a7806adb0 0x7f0a7806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0a940b52b0 tx=0x7f0a90008040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:14.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.743+0000 7f0a89ffb700 1 -- 172.21.15.189:0/1669877628 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0a84056050 con 0x7f0a9406a730 2024-04-06T12:30:14.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:14.939+0000 7f0a9b7bc700 1 -- 172.21.15.189:0/1669877628 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0a9415c250 con 0x7f0a9406a730 2024-04-06T12:30:15.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.718+0000 7f0a89ffb700 1 -- 172.21.15.189:0/1669877628 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v125) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0a84015020 con 0x7f0a9406a730 2024-04-06T12:30:15.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.723+0000 7f0a9b7bc700 1 -- 172.21.15.189:0/1669877628 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0a7806adb0 msgr2=0x7f0a7806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:15.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.723+0000 7f0a9b7bc700 1 --2- 172.21.15.189:0/1669877628 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0a7806adb0 0x7f0a7806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0a940b52b0 tx=0x7f0a90008040 comp rx=0 tx=0).stop 2024-04-06T12:30:15.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.723+0000 7f0a9b7bc700 1 -- 172.21.15.189:0/1669877628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a9406a730 msgr2=0x7f0a9415ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:15.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.723+0000 7f0a9b7bc700 1 --2- 172.21.15.189:0/1669877628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a9406a730 0x7f0a9415ac80 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f0a840090b0 tx=0x7f0a84004750 comp rx=0 tx=0).stop 2024-04-06T12:30:15.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.724+0000 7f0a9b7bc700 1 -- 172.21.15.189:0/1669877628 shutdown_connections 2024-04-06T12:30:15.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.724+0000 7f0a9b7bc700 1 --2- 172.21.15.189:0/1669877628 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0a7806adb0 0x7f0a7806d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:15.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.724+0000 7f0a9b7bc700 1 --2- 172.21.15.189:0/1669877628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a9415ddf0 0x7f0a9415c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:15.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.724+0000 7f0a9b7bc700 1 --2- 172.21.15.189:0/1669877628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a9406a730 0x7f0a9415ac80 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:15.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.724+0000 7f0a9b7bc700 1 --2- 172.21.15.189:0/1669877628 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a9415d7b0 0x7f0a94149c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:15.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.724+0000 7f0a9b7bc700 1 -- 172.21.15.189:0/1669877628 >> 172.21.15.189:0/1669877628 conn(0x7f0a940d00b0 msgr2=0x7f0a940ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:15.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.724+0000 7f0a9b7bc700 1 -- 172.21.15.189:0/1669877628 shutdown_connections 2024-04-06T12:30:15.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.725+0000 7f0a9b7bc700 1 -- 172.21.15.189:0/1669877628 wait complete. 2024-04-06T12:30:15.723 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' created 2024-04-06T12:30:15.738 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-06T12:30:15.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.877+0000 7f82b41bf700 1 Processor -- start 2024-04-06T12:30:15.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.877+0000 7f82b41bf700 1 -- start start 2024-04-06T12:30:15.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.878+0000 7f82b41bf700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f82ac06ae40 0x7f82ac0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:15.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.878+0000 7f82b41bf700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f82ac15ddf0 0x7f82ac15e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:15.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.878+0000 7f82b41bf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ac0b3990 con 0x7f82ac15d7b0 2024-04-06T12:30:15.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.878+0000 7f82b41bf700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82ac0b3060 con 0x7f82ac15ddf0 2024-04-06T12:30:15.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.878+0000 7f82b41bf700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82ac0b3340 con 0x7f82ac06ae40 2024-04-06T12:30:15.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.878+0000 7f82b275c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f82ac15ddf0 0x7f82ac15e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:15.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.878+0000 7f82b1f5b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f82ac15d7b0 0x7f82ac06a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51180/0 (socket says 172.21.15.189:51180) 2024-04-06T12:30:15.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.878+0000 7f82b1f5b700 1 -- 172.21.15.189:0/4098629189 learned_addr learned my addr 172.21.15.189:0/4098629189 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:15.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.878+0000 7f82b175a700 1 --2- 172.21.15.189:0/4098629189 >> v2:172.21.15.192:3300/0 conn(0x7f82ac06ae40 0x7f82ac0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:15.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.878+0000 7f82b275c700 1 -- 172.21.15.189:0/4098629189 >> v2:172.21.15.192:3300/0 conn(0x7f82ac06ae40 msgr2=0x7f82ac0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:15.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.878+0000 7f82b275c700 1 --2- 172.21.15.189:0/4098629189 >> v2:172.21.15.192:3300/0 conn(0x7f82ac06ae40 0x7f82ac0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:15.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.878+0000 7f82b275c700 1 -- 172.21.15.189:0/4098629189 >> v1:172.21.15.189:6789/0 conn(0x7f82ac15d7b0 legacy=0x7f82ac06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:15.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.878+0000 7f82b275c700 1 -- 172.21.15.189:0/4098629189 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ac0b3620 con 0x7f82ac15ddf0 2024-04-06T12:30:15.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.878+0000 7f82b0f59700 1 -- 172.21.15.189:0/4098629189 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244858606 0 0) 0x7f82ac0b3990 con 0x7f82ac15d7b0 2024-04-06T12:30:15.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.878+0000 7f82b175a700 1 --2- 172.21.15.189:0/4098629189 >> v2:172.21.15.192:3300/0 conn(0x7f82ac06ae40 0x7f82ac0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:15.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.878+0000 7f82b275c700 1 --2- 172.21.15.189:0/4098629189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82ac15ddf0 0x7f82ac15e240 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f82ac0afc00 tx=0x7f82a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2b321f8922d63f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:15.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.879+0000 7f82b0f59700 1 -- 172.21.15.189:0/4098629189 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a400e070 con 0x7f82ac15ddf0 2024-04-06T12:30:15.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.879+0000 7f82b0f59700 1 -- 172.21.15.189:0/4098629189 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a400bbd0 con 0x7f82ac15ddf0 2024-04-06T12:30:15.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.879+0000 7f82b0f59700 1 -- 172.21.15.189:0/4098629189 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a4012430 con 0x7f82ac15ddf0 2024-04-06T12:30:15.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.879+0000 7f82b41bf700 1 -- 172.21.15.189:0/4098629189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82ac15ddf0 msgr2=0x7f82ac15e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:15.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.879+0000 7f82b41bf700 1 --2- 172.21.15.189:0/4098629189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82ac15ddf0 0x7f82ac15e240 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f82ac0afc00 tx=0x7f82a400b330 comp rx=0 tx=0).stop 2024-04-06T12:30:15.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.879+0000 7f82b41bf700 1 -- 172.21.15.189:0/4098629189 shutdown_connections 2024-04-06T12:30:15.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.879+0000 7f82b41bf700 1 --2- 172.21.15.189:0/4098629189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82ac15ddf0 0x7f82ac15e240 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:15.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.879+0000 7f82b41bf700 1 --2- 172.21.15.189:0/4098629189 >> v2:172.21.15.192:3300/0 conn(0x7f82ac06ae40 0x7f82ac0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:15.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.879+0000 7f82b41bf700 1 -- 172.21.15.189:0/4098629189 >> 172.21.15.189:0/4098629189 conn(0x7f82ac0d00b0 msgr2=0x7f82ac0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:15.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.879+0000 7f82b41bf700 1 -- 172.21.15.189:0/4098629189 shutdown_connections 2024-04-06T12:30:15.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.879+0000 7f82b41bf700 1 -- 172.21.15.189:0/4098629189 wait complete. 2024-04-06T12:30:15.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.880+0000 7f82b41bf700 1 Processor -- start 2024-04-06T12:30:15.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.880+0000 7f82b41bf700 1 -- start start 2024-04-06T12:30:15.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.881+0000 7f82b41bf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82ac06ae40 0x7f82ac149410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:15.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.881+0000 7f82b41bf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82ac15d7b0 0x7f82ac0cef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:15.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.881+0000 7f82b41bf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82ac15ddf0 0x7f82ac1f24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:15.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.881+0000 7f82b41bf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac14c380 con 0x7f82ac06ae40 2024-04-06T12:30:15.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.881+0000 7f82b41bf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac14a600 con 0x7f82ac15d7b0 2024-04-06T12:30:15.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.881+0000 7f82b41bf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac0cf460 con 0x7f82ac15ddf0 2024-04-06T12:30:15.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.881+0000 7f82b1f5b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82ac06ae40 0x7f82ac149410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:15.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.881+0000 7f82b1f5b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82ac06ae40 0x7f82ac149410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49734/0 (socket says 172.21.15.189:49734) 2024-04-06T12:30:15.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.881+0000 7f82b1f5b700 1 -- 172.21.15.189:0/1814728635 learned_addr learned my addr 172.21.15.189:0/1814728635 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:15.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.882+0000 7f82b175a700 1 --2- 172.21.15.189:0/1814728635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82ac15d7b0 0x7f82ac0cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:15.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.882+0000 7f82b275c700 1 --2- 172.21.15.189:0/1814728635 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82ac15ddf0 0x7f82ac1f24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:15.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.882+0000 7f82b1f5b700 1 -- 172.21.15.189:0/1814728635 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82ac15ddf0 msgr2=0x7f82ac1f24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:15.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.882+0000 7f82b1f5b700 1 --2- 172.21.15.189:0/1814728635 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82ac15ddf0 0x7f82ac1f24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:15.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.882+0000 7f82b1f5b700 1 -- 172.21.15.189:0/1814728635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82ac15d7b0 msgr2=0x7f82ac0cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:15.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.882+0000 7f82b1f5b700 1 --2- 172.21.15.189:0/1814728635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82ac15d7b0 0x7f82ac0cef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:15.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.882+0000 7f82b1f5b700 1 -- 172.21.15.189:0/1814728635 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a4009a70 con 0x7f82ac06ae40 2024-04-06T12:30:15.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.882+0000 7f82b1f5b700 1 --2- 172.21.15.189:0/1814728635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82ac06ae40 0x7f82ac149410 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f82a800c750 tx=0x7f82a800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:15.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.882+0000 7f82a27fc700 1 -- 172.21.15.189:0/1814728635 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a8004c80 con 0x7f82ac06ae40 2024-04-06T12:30:15.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.882+0000 7f82a27fc700 1 -- 172.21.15.189:0/1814728635 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a8004e00 con 0x7f82ac06ae40 2024-04-06T12:30:15.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.882+0000 7f82b41bf700 1 -- 172.21.15.189:0/1814728635 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ac1499f0 con 0x7f82ac06ae40 2024-04-06T12:30:15.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.882+0000 7f82a27fc700 1 -- 172.21.15.189:0/1814728635 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a8007310 con 0x7f82ac06ae40 2024-04-06T12:30:15.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.882+0000 7f82b41bf700 1 -- 172.21.15.189:0/1814728635 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ac149fb0 con 0x7f82ac06ae40 2024-04-06T12:30:15.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.883+0000 7f82b41bf700 1 -- 172.21.15.189:0/1814728635 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ac067bb0 con 0x7f82ac06ae40 2024-04-06T12:30:15.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.883+0000 7f82a27fc700 1 -- 172.21.15.189:0/1814728635 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f82a8007470 con 0x7f82ac06ae40 2024-04-06T12:30:15.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.884+0000 7f82a27fc700 1 --2- 172.21.15.189:0/1814728635 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f829006adb0 0x7f829006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:15.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.884+0000 7f82a27fc700 1 -- 172.21.15.189:0/1814728635 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f82a8089180 con 0x7f82ac06ae40 2024-04-06T12:30:15.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.884+0000 7f82b175a700 1 --2- 172.21.15.189:0/1814728635 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f829006adb0 0x7f829006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:15.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.884+0000 7f82b175a700 1 --2- 172.21.15.189:0/1814728635 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f829006adb0 0x7f829006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f82ac0728c0 tx=0x7f829c008040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:15.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:15.889+0000 7f82a27fc700 1 -- 172.21.15.189:0/1814728635 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f82a80542f0 con 0x7f82ac06ae40 2024-04-06T12:30:16.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.085+0000 7f82b41bf700 1 -- 172.21.15.189:0/1814728635 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f82ac0671d0 con 0x7f82ac06ae40 2024-04-06T12:30:16.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.720+0000 7f82a27fc700 1 -- 172.21.15.189:0/1814728635 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v126) v1 ==== 170+0+0 (secure 0 0 0) 0x7f82a8057910 con 0x7f82ac06ae40 2024-04-06T12:30:16.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.725+0000 7f82b41bf700 1 -- 172.21.15.189:0/1814728635 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f829006adb0 msgr2=0x7f829006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:16.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.725+0000 7f82b41bf700 1 --2- 172.21.15.189:0/1814728635 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f829006adb0 0x7f829006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f82ac0728c0 tx=0x7f829c008040 comp rx=0 tx=0).stop 2024-04-06T12:30:16.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.725+0000 7f82b41bf700 1 -- 172.21.15.189:0/1814728635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82ac06ae40 msgr2=0x7f82ac149410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:16.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.725+0000 7f82b41bf700 1 --2- 172.21.15.189:0/1814728635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82ac06ae40 0x7f82ac149410 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f82a800c750 tx=0x7f82a800cb10 comp rx=0 tx=0).stop 2024-04-06T12:30:16.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.726+0000 7f82b41bf700 1 -- 172.21.15.189:0/1814728635 shutdown_connections 2024-04-06T12:30:16.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.726+0000 7f82b41bf700 1 --2- 172.21.15.189:0/1814728635 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82ac15ddf0 0x7f82ac1f24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:16.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.726+0000 7f82b41bf700 1 --2- 172.21.15.189:0/1814728635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82ac06ae40 0x7f82ac149410 secure :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f82a800c750 tx=0x7f82a800cb10 comp rx=0 tx=0).stop 2024-04-06T12:30:16.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.726+0000 7f82b41bf700 1 --2- 172.21.15.189:0/1814728635 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f829006adb0 0x7f829006d270 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f82ac0728c0 tx=0x7f829c008040 comp rx=0 tx=0).stop 2024-04-06T12:30:16.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.726+0000 7f82b41bf700 1 --2- 172.21.15.189:0/1814728635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82ac15d7b0 0x7f82ac0cef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:16.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.726+0000 7f82b41bf700 1 -- 172.21.15.189:0/1814728635 >> 172.21.15.189:0/1814728635 conn(0x7f82ac0d00b0 msgr2=0x7f82ac071b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:16.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.728+0000 7f82b41bf700 1 -- 172.21.15.189:0/1814728635 shutdown_connections 2024-04-06T12:30:16.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.728+0000 7f82b41bf700 1 -- 172.21.15.189:0/1814728635 wait complete. 2024-04-06T12:30:16.727 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' created 2024-04-06T12:30:16.748 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-06T12:30:16.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.877+0000 7f1e3d75b700 1 Processor -- start 2024-04-06T12:30:16.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.877+0000 7f1e3d75b700 1 -- start start 2024-04-06T12:30:16.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.878+0000 7f1e3d75b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1e380b4ed0 0x7f1e380b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:16.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.878+0000 7f1e3d75b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1e380d2880 0x7f1e380d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:16.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.878+0000 7f1e3d75b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e380b5cc0 con 0x7f1e3815d590 2024-04-06T12:30:16.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.878+0000 7f1e3d75b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e380b58c0 con 0x7f1e380d2880 2024-04-06T12:30:16.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.878+0000 7f1e3d75b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e380d1500 con 0x7f1e380b4ed0 2024-04-06T12:30:16.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.878+0000 7f1e377fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1e380d2880 0x7f1e380d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:16.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.878+0000 7f1e377fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1e380d2880 0x7f1e380d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49744/0 (socket says 172.21.15.189:49744) 2024-04-06T12:30:16.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.878+0000 7f1e377fe700 1 -- 172.21.15.189:0/1529574867 learned_addr learned my addr 172.21.15.189:0/1529574867 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:16.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.878+0000 7f1e367fc700 1 --2- 172.21.15.189:0/1529574867 >> v2:172.21.15.192:3300/0 conn(0x7f1e380b4ed0 0x7f1e380b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:16.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.878+0000 7f1e377fe700 1 -- 172.21.15.189:0/1529574867 >> v2:172.21.15.192:3300/0 conn(0x7f1e380b4ed0 msgr2=0x7f1e380b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:16.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.878+0000 7f1e377fe700 1 --2- 172.21.15.189:0/1529574867 >> v2:172.21.15.192:3300/0 conn(0x7f1e380b4ed0 0x7f1e380b5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:16.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.878+0000 7f1e377fe700 1 -- 172.21.15.189:0/1529574867 >> v1:172.21.15.192:6789/0 conn(0x7f1e3815d590 legacy=0x7f1e380b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:16.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.878+0000 7f1e377fe700 1 -- 172.21.15.189:0/1529574867 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e380d17e0 con 0x7f1e380d2880 2024-04-06T12:30:16.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.879+0000 7f1e377fe700 1 --2- 172.21.15.189:0/1529574867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e380d2880 0x7f1e380d2cd0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f1e380b6330 tx=0x7f1e3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b3c94c97fa7d4fd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:16.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.879+0000 7f1e35ffb700 1 -- 172.21.15.189:0/1529574867 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3000e070 con 0x7f1e380d2880 2024-04-06T12:30:16.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.879+0000 7f1e35ffb700 1 -- 172.21.15.189:0/1529574867 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e30003a20 con 0x7f1e380d2880 2024-04-06T12:30:16.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.879+0000 7f1e35ffb700 1 -- 172.21.15.189:0/1529574867 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e30003dd0 con 0x7f1e380d2880 2024-04-06T12:30:16.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.879+0000 7f1e3d75b700 1 -- 172.21.15.189:0/1529574867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e380d2880 msgr2=0x7f1e380d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:16.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.879+0000 7f1e3d75b700 1 --2- 172.21.15.189:0/1529574867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e380d2880 0x7f1e380d2cd0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f1e380b6330 tx=0x7f1e3000b330 comp rx=0 tx=0).stop 2024-04-06T12:30:16.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.879+0000 7f1e3d75b700 1 -- 172.21.15.189:0/1529574867 shutdown_connections 2024-04-06T12:30:16.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.879+0000 7f1e3d75b700 1 --2- 172.21.15.189:0/1529574867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e380d2880 0x7f1e380d2cd0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:16.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.879+0000 7f1e3d75b700 1 --2- 172.21.15.189:0/1529574867 >> v2:172.21.15.192:3300/0 conn(0x7f1e380b4ed0 0x7f1e380b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:16.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.879+0000 7f1e3d75b700 1 -- 172.21.15.189:0/1529574867 >> 172.21.15.189:0/1529574867 conn(0x7f1e3806bbc0 msgr2=0x7f1e3806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:16.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.879+0000 7f1e3d75b700 1 -- 172.21.15.189:0/1529574867 shutdown_connections 2024-04-06T12:30:16.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.879+0000 7f1e3d75b700 1 -- 172.21.15.189:0/1529574867 wait complete. 2024-04-06T12:30:16.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.881+0000 7f1e3d75b700 1 Processor -- start 2024-04-06T12:30:16.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.881+0000 7f1e3d75b700 1 -- start start 2024-04-06T12:30:16.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.882+0000 7f1e3d75b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e380b4ed0 0x7f1e380cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:16.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.882+0000 7f1e3d75b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e380d2880 0x7f1e380c9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.882+0000 7f1e3d75b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e3815d590 0x7f1e3815f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.882+0000 7f1e3d75b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e380ccb50 con 0x7f1e380d2880 2024-04-06T12:30:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.882+0000 7f1e3d75b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e381f6740 con 0x7f1e3815d590 2024-04-06T12:30:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.882+0000 7f1e3d75b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e381f6a20 con 0x7f1e380b4ed0 2024-04-06T12:30:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.882+0000 7f1e367fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e380d2880 0x7f1e380c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.882+0000 7f1e367fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e380d2880 0x7f1e380c9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49758/0 (socket says 172.21.15.189:49758) 2024-04-06T12:30:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.882+0000 7f1e367fc700 1 -- 172.21.15.189:0/3055726406 learned_addr learned my addr 172.21.15.189:0/3055726406 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.882+0000 7f1e377fe700 1 --2- 172.21.15.189:0/3055726406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e3815d590 0x7f1e3815f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.882+0000 7f1e36ffd700 1 --2- 172.21.15.189:0/3055726406 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e380b4ed0 0x7f1e380cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.882+0000 7f1e367fc700 1 -- 172.21.15.189:0/3055726406 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e380b4ed0 msgr2=0x7f1e380cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.882+0000 7f1e367fc700 1 --2- 172.21.15.189:0/3055726406 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e380b4ed0 0x7f1e380cc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.882+0000 7f1e367fc700 1 -- 172.21.15.189:0/3055726406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e3815d590 msgr2=0x7f1e3815f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.882+0000 7f1e367fc700 1 --2- 172.21.15.189:0/3055726406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e3815d590 0x7f1e3815f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:16.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.882+0000 7f1e367fc700 1 -- 172.21.15.189:0/3055726406 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e30009b30 con 0x7f1e380d2880 2024-04-06T12:30:16.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.883+0000 7f1e367fc700 1 --2- 172.21.15.189:0/3055726406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e380d2880 0x7f1e380c9040 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f1e2800c7f0 tx=0x7f1e2800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:16.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.883+0000 7f1e1f7fe700 1 -- 172.21.15.189:0/3055726406 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e28011800 con 0x7f1e380d2880 2024-04-06T12:30:16.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.883+0000 7f1e1f7fe700 1 -- 172.21.15.189:0/3055726406 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e28011960 con 0x7f1e380d2880 2024-04-06T12:30:16.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.883+0000 7f1e3d75b700 1 -- 172.21.15.189:0/3055726406 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e381f6d00 con 0x7f1e380d2880 2024-04-06T12:30:16.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.883+0000 7f1e1f7fe700 1 -- 172.21.15.189:0/3055726406 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e28011cf0 con 0x7f1e380d2880 2024-04-06T12:30:16.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.883+0000 7f1e3d75b700 1 -- 172.21.15.189:0/3055726406 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e380c9580 con 0x7f1e380d2880 2024-04-06T12:30:16.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.884+0000 7f1e3d75b700 1 -- 172.21.15.189:0/3055726406 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e38074990 con 0x7f1e380d2880 2024-04-06T12:30:16.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.884+0000 7f1e1f7fe700 1 -- 172.21.15.189:0/3055726406 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f1e28024480 con 0x7f1e380d2880 2024-04-06T12:30:16.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.885+0000 7f1e1f7fe700 1 --2- 172.21.15.189:0/3055726406 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e2406adb0 0x7f1e2406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:16.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.885+0000 7f1e1f7fe700 1 -- 172.21.15.189:0/3055726406 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f1e28089a70 con 0x7f1e380d2880 2024-04-06T12:30:16.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.885+0000 7f1e36ffd700 1 --2- 172.21.15.189:0/3055726406 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e2406adb0 0x7f1e2406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:16.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.885+0000 7f1e36ffd700 1 --2- 172.21.15.189:0/3055726406 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e2406adb0 0x7f1e2406d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c009870 tx=0x7f1e2c008040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:16.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:16.889+0000 7f1e1f7fe700 1 -- 172.21.15.189:0/3055726406 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1e280549e0 con 0x7f1e380d2880 2024-04-06T12:30:17.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.126+0000 7f1e3d75b700 1 -- 172.21.15.189:0/3055726406 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1e380cda70 con 0x7f1e380d2880 2024-04-06T12:30:17.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.131+0000 7f1e1f7fe700 1 -- 172.21.15.189:0/3055726406 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v50) v1 ==== 174+0+0 (secure 0 0 0) 0x7f1e2801f0c0 con 0x7f1e380d2880 2024-04-06T12:30:17.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.137+0000 7f1e3d75b700 1 -- 172.21.15.189:0/3055726406 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e2406adb0 msgr2=0x7f1e2406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:17.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.137+0000 7f1e3d75b700 1 --2- 172.21.15.189:0/3055726406 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e2406adb0 0x7f1e2406d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c009870 tx=0x7f1e2c008040 comp rx=0 tx=0).stop 2024-04-06T12:30:17.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.137+0000 7f1e3d75b700 1 -- 172.21.15.189:0/3055726406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e380d2880 msgr2=0x7f1e380c9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:17.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.137+0000 7f1e3d75b700 1 --2- 172.21.15.189:0/3055726406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e380d2880 0x7f1e380c9040 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f1e2800c7f0 tx=0x7f1e2800cbb0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.137+0000 7f1e3d75b700 1 -- 172.21.15.189:0/3055726406 shutdown_connections 2024-04-06T12:30:17.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.137+0000 7f1e3d75b700 1 --2- 172.21.15.189:0/3055726406 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e2406adb0 0x7f1e2406d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.137+0000 7f1e3d75b700 1 --2- 172.21.15.189:0/3055726406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e3815d590 0x7f1e3815f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.137+0000 7f1e3d75b700 1 --2- 172.21.15.189:0/3055726406 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e380b4ed0 0x7f1e380cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.137+0000 7f1e3d75b700 1 --2- 172.21.15.189:0/3055726406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e380d2880 0x7f1e380c9040 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.137+0000 7f1e3d75b700 1 -- 172.21.15.189:0/3055726406 >> 172.21.15.189:0/3055726406 conn(0x7f1e3806bbc0 msgr2=0x7f1e3815bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:17.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.138+0000 7f1e3d75b700 1 -- 172.21.15.189:0/3055726406 shutdown_connections 2024-04-06T12:30:17.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.138+0000 7f1e3d75b700 1 -- 172.21.15.189:0/3055726406 wait complete. 2024-04-06T12:30:17.139 INFO:teuthology.orchestra.run.smithi189.stderr:new fs with metadata pool 15 and data pool 16 2024-04-06T12:30:17.156 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:30:17.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.281+0000 7f0705d31700 1 Processor -- start 2024-04-06T12:30:17.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.281+0000 7f0705d31700 1 -- start start 2024-04-06T12:30:17.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.282+0000 7f0705d31700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f07000b3e20 0x7f070015e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:17.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.282+0000 7f0705d31700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07000b3990 con 0x7f070006ae40 2024-04-06T12:30:17.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.282+0000 7f0705d31700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07000b3060 con 0x7f070015d7b0 2024-04-06T12:30:17.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.282+0000 7f0705d31700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07000b4230 con 0x7f07000b3e20 2024-04-06T12:30:17.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.282+0000 7f06feffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f070006ae40 0x7f070015ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51192/0 (socket says 172.21.15.189:51192) 2024-04-06T12:30:17.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.282+0000 7f06fffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f07000b3e20 0x7f070015e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:17.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.282+0000 7f06feffd700 1 -- 172.21.15.189:0/3972737213 learned_addr learned my addr 172.21.15.189:0/3972737213 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:17.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.282+0000 7f06fffff700 1 -- 172.21.15.189:0/3972737213 >> v1:172.21.15.192:6789/0 conn(0x7f070015d7b0 legacy=0x7f070006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:17.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.282+0000 7f06fffff700 1 -- 172.21.15.189:0/3972737213 >> v1:172.21.15.189:6789/0 conn(0x7f070006ae40 legacy=0x7f070015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:17.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.282+0000 7f06fffff700 1 -- 172.21.15.189:0/3972737213 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07000b3340 con 0x7f07000b3e20 2024-04-06T12:30:17.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.282+0000 7f06fe7fc700 1 -- 172.21.15.189:0/3972737213 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258683384 0 0) 0x7f07000b3990 con 0x7f070006ae40 2024-04-06T12:30:17.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.282+0000 7f06fffff700 1 --2- 172.21.15.189:0/3972737213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07000b3e20 0x7f070015e500 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f06f8009d90 tx=0x7f06f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591cc3b8fa79a07b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:17.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.282+0000 7f06fe7fc700 1 -- 172.21.15.189:0/3972737213 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f800e070 con 0x7f07000b3e20 2024-04-06T12:30:17.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.282+0000 7f06fe7fc700 1 -- 172.21.15.189:0/3972737213 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f800bbd0 con 0x7f07000b3e20 2024-04-06T12:30:17.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.282+0000 7f06fe7fc700 1 -- 172.21.15.189:0/3972737213 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f8012430 con 0x7f07000b3e20 2024-04-06T12:30:17.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.283+0000 7f0705d31700 1 -- 172.21.15.189:0/3972737213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07000b3e20 msgr2=0x7f070015e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:17.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.283+0000 7f0705d31700 1 --2- 172.21.15.189:0/3972737213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07000b3e20 0x7f070015e500 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f06f8009d90 tx=0x7f06f800b330 comp rx=0 tx=0).stop 2024-04-06T12:30:17.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.283+0000 7f0705d31700 1 -- 172.21.15.189:0/3972737213 shutdown_connections 2024-04-06T12:30:17.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.283+0000 7f0705d31700 1 --2- 172.21.15.189:0/3972737213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07000b3e20 0x7f070015e500 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.283+0000 7f0705d31700 1 -- 172.21.15.189:0/3972737213 >> 172.21.15.189:0/3972737213 conn(0x7f07000d00b0 msgr2=0x7f07000d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:17.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.283+0000 7f0705d31700 1 -- 172.21.15.189:0/3972737213 shutdown_connections 2024-04-06T12:30:17.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.283+0000 7f0705d31700 1 -- 172.21.15.189:0/3972737213 wait complete. 2024-04-06T12:30:17.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.284+0000 7f0705d31700 1 Processor -- start 2024-04-06T12:30:17.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.284+0000 7f0705d31700 1 -- start start 2024-04-06T12:30:17.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.285+0000 7f0705d31700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f070006ae40 0x7f07001f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:17.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.285+0000 7f0705d31700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07000b3e20 0x7f07001f5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:17.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.285+0000 7f0705d31700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f070015d7b0 0x7f07000cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:17.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.285+0000 7f0705d31700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07001f60f0 con 0x7f070015d7b0 2024-04-06T12:30:17.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.285+0000 7f0705d31700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07000cdbd0 con 0x7f070006ae40 2024-04-06T12:30:17.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.285+0000 7f0705d31700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07000cdeb0 con 0x7f07000b3e20 2024-04-06T12:30:17.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.285+0000 7f06fffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f070015d7b0 0x7f07000cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:17.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.285+0000 7f06fffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f070015d7b0 0x7f07000cd300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49776/0 (socket says 172.21.15.189:49776) 2024-04-06T12:30:17.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.285+0000 7f06fffff700 1 -- 172.21.15.189:0/1723660706 learned_addr learned my addr 172.21.15.189:0/1723660706 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:17.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.285+0000 7f06fffff700 1 -- 172.21.15.189:0/1723660706 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07000b3e20 msgr2=0x7f07001f5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:17.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.285+0000 7f06fffff700 1 --2- 172.21.15.189:0/1723660706 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07000b3e20 0x7f07001f5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.285+0000 7f06fffff700 1 -- 172.21.15.189:0/1723660706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f070006ae40 msgr2=0x7f07001f3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:17.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.285+0000 7f06fffff700 1 --2- 172.21.15.189:0/1723660706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f070006ae40 0x7f07001f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.285+0000 7f06fffff700 1 -- 172.21.15.189:0/1723660706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f8009a70 con 0x7f070015d7b0 2024-04-06T12:30:17.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.286+0000 7f06fffff700 1 --2- 172.21.15.189:0/1723660706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f070015d7b0 0x7f07000cd300 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f06f8014040 tx=0x7f06f8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:17.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.286+0000 7f06e7fff700 1 -- 172.21.15.189:0/1723660706 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f801b810 con 0x7f070015d7b0 2024-04-06T12:30:17.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.286+0000 7f06e7fff700 1 -- 172.21.15.189:0/1723660706 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f8020070 con 0x7f070015d7b0 2024-04-06T12:30:17.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.286+0000 7f0705d31700 1 -- 172.21.15.189:0/1723660706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f070015b7b0 con 0x7f070015d7b0 2024-04-06T12:30:17.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.286+0000 7f06e7fff700 1 -- 172.21.15.189:0/1723660706 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f801bbb0 con 0x7f070015d7b0 2024-04-06T12:30:17.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.286+0000 7f0705d31700 1 -- 172.21.15.189:0/1723660706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f070015bd70 con 0x7f070015d7b0 2024-04-06T12:30:17.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.287+0000 7f0705d31700 1 -- 172.21.15.189:0/1723660706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f070015b610 con 0x7f070015d7b0 2024-04-06T12:30:17.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.287+0000 7f06e7fff700 1 -- 172.21.15.189:0/1723660706 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f06f801bd10 con 0x7f070015d7b0 2024-04-06T12:30:17.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.288+0000 7f06e7fff700 1 --2- 172.21.15.189:0/1723660706 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06ec06adb0 0x7f06ec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:17.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.288+0000 7f06e7fff700 1 -- 172.21.15.189:0/1723660706 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f06f8022070 con 0x7f070015d7b0 2024-04-06T12:30:17.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.288+0000 7f06ff7fe700 1 --2- 172.21.15.189:0/1723660706 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06ec06adb0 0x7f06ec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:17.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.288+0000 7f06ff7fe700 1 --2- 172.21.15.189:0/1723660706 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06ec06adb0 0x7f06ec06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f06f0000f60 tx=0x7f06f0007a40 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:17.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.292+0000 7f06e7fff700 1 -- 172.21.15.189:0/1723660706 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f070015b610 con 0x7f070015d7b0 2024-04-06T12:30:17.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.487+0000 7f0705d31700 1 -- 172.21.15.189:0/1723660706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f070015b610 con 0x7f070015d7b0 2024-04-06T12:30:17.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.488+0000 7f06e7fff700 1 -- 172.21.15.189:0/1723660706 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16309 (secure 0 0 0) 0x7f070015b610 con 0x7f070015d7b0 2024-04-06T12:30:17.486 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:30:17.486 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":127,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:30:17.126982+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:30:14.958335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-06T12:30:16.104585+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:30:17.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.491+0000 7f0705d31700 1 -- 172.21.15.189:0/1723660706 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06ec06adb0 msgr2=0x7f06ec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:17.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.491+0000 7f0705d31700 1 --2- 172.21.15.189:0/1723660706 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06ec06adb0 0x7f06ec06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f06f0000f60 tx=0x7f06f0007a40 comp rx=0 tx=0).stop 2024-04-06T12:30:17.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.491+0000 7f0705d31700 1 -- 172.21.15.189:0/1723660706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f070015d7b0 msgr2=0x7f07000cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:17.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.491+0000 7f0705d31700 1 --2- 172.21.15.189:0/1723660706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f070015d7b0 0x7f07000cd300 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f06f8014040 tx=0x7f06f8004ad0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.491+0000 7f0705d31700 1 -- 172.21.15.189:0/1723660706 shutdown_connections 2024-04-06T12:30:17.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.491+0000 7f0705d31700 1 --2- 172.21.15.189:0/1723660706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f070015d7b0 0x7f07000cd300 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.491+0000 7f0705d31700 1 --2- 172.21.15.189:0/1723660706 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f06ec06adb0 0x7f06ec06d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.491+0000 7f0705d31700 1 --2- 172.21.15.189:0/1723660706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f070006ae40 0x7f07001f3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.491+0000 7f0705d31700 1 --2- 172.21.15.189:0/1723660706 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07000b3e20 0x7f07001f5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.491+0000 7f0705d31700 1 -- 172.21.15.189:0/1723660706 >> 172.21.15.189:0/1723660706 conn(0x7f07000d00b0 msgr2=0x7f07000ae0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:17.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.491+0000 7f0705d31700 1 -- 172.21.15.189:0/1723660706 shutdown_connections 2024-04-06T12:30:17.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.491+0000 7f0705d31700 1 -- 172.21.15.189:0/1723660706 wait complete. 2024-04-06T12:30:17.501 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:30:17.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.626+0000 7f7594b23700 1 Processor -- start 2024-04-06T12:30:17.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.626+0000 7f7594b23700 1 -- start start 2024-04-06T12:30:17.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.627+0000 7f7594b23700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f759015a550 0x7f75900cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:17.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.627+0000 7f7594b23700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75900b4d70 0x7f75900b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:17.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.627+0000 7f7594b23700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75900cdbe0 con 0x7f75900cf900 2024-04-06T12:30:17.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.627+0000 7f7594b23700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f759015a940 con 0x7f759015a550 2024-04-06T12:30:17.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.627+0000 7f7594b23700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759015ac00 con 0x7f75900b4d70 2024-04-06T12:30:17.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.627+0000 7f758e59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f759015a550 0x7f75900cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:17.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.627+0000 7f758dd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f75900cf900 0x7f75900b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51206/0 (socket says 172.21.15.189:51206) 2024-04-06T12:30:17.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.627+0000 7f758dd9b700 1 -- 172.21.15.189:0/969367559 learned_addr learned my addr 172.21.15.189:0/969367559 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:17.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.628+0000 7f758ed9d700 1 --2- 172.21.15.189:0/969367559 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75900b4d70 0x7f75900b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:17.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.628+0000 7f758e59c700 1 -- 172.21.15.189:0/969367559 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75900b4d70 msgr2=0x7f75900b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:17.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.628+0000 7f758e59c700 1 --2- 172.21.15.189:0/969367559 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75900b4d70 0x7f75900b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.628+0000 7f758e59c700 1 -- 172.21.15.189:0/969367559 >> v1:172.21.15.189:6789/0 conn(0x7f75900cf900 legacy=0x7f75900b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:17.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.628+0000 7f758e59c700 1 -- 172.21.15.189:0/969367559 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759015aee0 con 0x7f759015a550 2024-04-06T12:30:17.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.628+0000 7f758d59a700 1 -- 172.21.15.189:0/969367559 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310422774 0 0) 0x7f75900cdbe0 con 0x7f75900cf900 2024-04-06T12:30:17.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.628+0000 7f758e59c700 1 --2- 172.21.15.189:0/969367559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f759015a550 0x7f75900cf3c0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f7580009840 tx=0x7f7580009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c86ca8378999c7d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:17.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.628+0000 7f758d59a700 1 -- 172.21.15.189:0/969367559 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7580011830 con 0x7f759015a550 2024-04-06T12:30:17.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.628+0000 7f758d59a700 1 -- 172.21.15.189:0/969367559 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7580011990 con 0x7f759015a550 2024-04-06T12:30:17.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.628+0000 7f758d59a700 1 -- 172.21.15.189:0/969367559 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7580011d20 con 0x7f759015a550 2024-04-06T12:30:17.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.628+0000 7f7594b23700 1 -- 172.21.15.189:0/969367559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f759015a550 msgr2=0x7f75900cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:17.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.628+0000 7f7594b23700 1 --2- 172.21.15.189:0/969367559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f759015a550 0x7f75900cf3c0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f7580009840 tx=0x7f7580009b50 comp rx=0 tx=0).stop 2024-04-06T12:30:17.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.629+0000 7f7594b23700 1 -- 172.21.15.189:0/969367559 shutdown_connections 2024-04-06T12:30:17.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.629+0000 7f7594b23700 1 --2- 172.21.15.189:0/969367559 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75900b4d70 0x7f75900b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.629+0000 7f7594b23700 1 --2- 172.21.15.189:0/969367559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f759015a550 0x7f75900cf3c0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.629+0000 7f7594b23700 1 -- 172.21.15.189:0/969367559 >> 172.21.15.189:0/969367559 conn(0x7f75900d00b0 msgr2=0x7f75900d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:17.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.629+0000 7f7594b23700 1 -- 172.21.15.189:0/969367559 shutdown_connections 2024-04-06T12:30:17.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.629+0000 7f7594b23700 1 -- 172.21.15.189:0/969367559 wait complete. 2024-04-06T12:30:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.630+0000 7f7594b23700 1 Processor -- start 2024-04-06T12:30:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.630+0000 7f7594b23700 1 -- start start 2024-04-06T12:30:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.631+0000 7f7594b23700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75900b4d70 0x7f75901f0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.631+0000 7f7594b23700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75900cf900 0x7f759006f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.631+0000 7f7594b23700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f759006f5a0 0x7f75901468a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.631+0000 7f7594b23700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759006fa30 con 0x7f759006f5a0 2024-04-06T12:30:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.631+0000 7f7594b23700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75901475d0 con 0x7f75900cf900 2024-04-06T12:30:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.631+0000 7f7594b23700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7590144990 con 0x7f75900b4d70 2024-04-06T12:30:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.631+0000 7f758ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f759006f5a0 0x7f75901468a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.631+0000 7f758ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f759006f5a0 0x7f75901468a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49792/0 (socket says 172.21.15.189:49792) 2024-04-06T12:30:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.631+0000 7f758ed9d700 1 -- 172.21.15.189:0/3026938448 learned_addr learned my addr 172.21.15.189:0/3026938448 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.632+0000 7f758e59c700 1 --2- 172.21.15.189:0/3026938448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75900b4d70 0x7f75901f0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.632+0000 7f758dd9b700 1 --2- 172.21.15.189:0/3026938448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75900cf900 0x7f759006f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.632+0000 7f758ed9d700 1 -- 172.21.15.189:0/3026938448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75900b4d70 msgr2=0x7f75901f0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.632+0000 7f758ed9d700 1 --2- 172.21.15.189:0/3026938448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75900b4d70 0x7f75901f0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.632+0000 7f758ed9d700 1 -- 172.21.15.189:0/3026938448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75900cf900 msgr2=0x7f759006f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.632+0000 7f758ed9d700 1 --2- 172.21.15.189:0/3026938448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75900cf900 0x7f759006f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.632+0000 7f758ed9d700 1 -- 172.21.15.189:0/3026938448 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75800094a0 con 0x7f759006f5a0 2024-04-06T12:30:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.632+0000 7f758dd9b700 1 --2- 172.21.15.189:0/3026938448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75900cf900 0x7f759006f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:30:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.632+0000 7f758ed9d700 1 --2- 172.21.15.189:0/3026938448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f759006f5a0 0x7f75901468a0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f7588007e30 tx=0x7f758800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.632+0000 7f757effd700 1 -- 172.21.15.189:0/3026938448 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758800eea0 con 0x7f759006f5a0 2024-04-06T12:30:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.632+0000 7f757effd700 1 -- 172.21.15.189:0/3026938448 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7588014420 con 0x7f759006f5a0 2024-04-06T12:30:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.632+0000 7f7594b23700 1 -- 172.21.15.189:0/3026938448 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7590144c70 con 0x7f759006f5a0 2024-04-06T12:30:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.632+0000 7f757effd700 1 -- 172.21.15.189:0/3026938448 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75880147d0 con 0x7f759006f5a0 2024-04-06T12:30:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.632+0000 7f7594b23700 1 -- 172.21.15.189:0/3026938448 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75901f0980 con 0x7f759006f5a0 2024-04-06T12:30:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.633+0000 7f7594b23700 1 -- 172.21.15.189:0/3026938448 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7590145190 con 0x7f759006f5a0 2024-04-06T12:30:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.634+0000 7f757effd700 1 -- 172.21.15.189:0/3026938448 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7588014930 con 0x7f759006f5a0 2024-04-06T12:30:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.634+0000 7f757effd700 1 --2- 172.21.15.189:0/3026938448 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f757006adb0 0x7f757006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.634+0000 7f757effd700 1 -- 172.21.15.189:0/3026938448 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7588089cf0 con 0x7f759006f5a0 2024-04-06T12:30:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.634+0000 7f758e59c700 1 --2- 172.21.15.189:0/3026938448 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f757006adb0 0x7f757006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.635+0000 7f758e59c700 1 --2- 172.21.15.189:0/3026938448 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f757006adb0 0x7f757006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f75800090b0 tx=0x7f758001c040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.639+0000 7f757effd700 1 -- 172.21.15.189:0/3026938448 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7590145190 con 0x7f759006f5a0 2024-04-06T12:30:17.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.838+0000 7f7594b23700 1 -- 172.21.15.189:0/3026938448 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7590145190 con 0x7f759006f5a0 2024-04-06T12:30:17.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.839+0000 7f757effd700 1 -- 172.21.15.189:0/3026938448 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16309 (secure 0 0 0) 0x7f7590145190 con 0x7f759006f5a0 2024-04-06T12:30:17.836 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:30:17.836 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":127,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:30:17.126982+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:30:14.958335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-06T12:30:16.104585+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:30:17.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.841+0000 7f7594b23700 1 -- 172.21.15.189:0/3026938448 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f757006adb0 msgr2=0x7f757006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:17.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.841+0000 7f7594b23700 1 --2- 172.21.15.189:0/3026938448 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f757006adb0 0x7f757006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f75800090b0 tx=0x7f758001c040 comp rx=0 tx=0).stop 2024-04-06T12:30:17.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.841+0000 7f7594b23700 1 -- 172.21.15.189:0/3026938448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f759006f5a0 msgr2=0x7f75901468a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:17.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.841+0000 7f7594b23700 1 --2- 172.21.15.189:0/3026938448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f759006f5a0 0x7f75901468a0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f7588007e30 tx=0x7f758800e3c0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.841+0000 7f7594b23700 1 -- 172.21.15.189:0/3026938448 shutdown_connections 2024-04-06T12:30:17.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.841+0000 7f7594b23700 1 --2- 172.21.15.189:0/3026938448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f759006f5a0 0x7f75901468a0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.841+0000 7f7594b23700 1 --2- 172.21.15.189:0/3026938448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75900b4d70 0x7f75901f0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.841+0000 7f7594b23700 1 --2- 172.21.15.189:0/3026938448 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f757006adb0 0x7f757006d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.841+0000 7f7594b23700 1 --2- 172.21.15.189:0/3026938448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75900cf900 0x7f759006f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:17.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.841+0000 7f7594b23700 1 -- 172.21.15.189:0/3026938448 >> 172.21.15.189:0/3026938448 conn(0x7f75900d00b0 msgr2=0x7f7590153dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:17.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.841+0000 7f7594b23700 1 -- 172.21.15.189:0/3026938448 shutdown_connections 2024-04-06T12:30:17.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.841+0000 7f7594b23700 1 -- 172.21.15.189:0/3026938448 wait complete. 2024-04-06T12:30:17.851 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-06T12:30:17.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.979+0000 7fcce58f0700 1 Processor -- start 2024-04-06T12:30:18.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.979+0000 7fcce58f0700 1 -- start start 2024-04-06T12:30:18.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.980+0000 7fcce58f0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcce015db10 0x7fcce00b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:18.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.980+0000 7fcce58f0700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcce0143470 0x7fcce00cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:18.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.980+0000 7fcce58f0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce015eb40 con 0x7fcce0160b60 2024-04-06T12:30:18.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.980+0000 7fcce58f0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce0160f40 con 0x7fcce015db10 2024-04-06T12:30:18.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.980+0000 7fcce58f0700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce00d0bf0 con 0x7fcce0143470 2024-04-06T12:30:18.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.980+0000 7fccde7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcce015db10 0x7fcce00b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:18.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.980+0000 7fccde7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcce015db10 0x7fcce00b37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49808/0 (socket says 172.21.15.189:49808) 2024-04-06T12:30:18.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.980+0000 7fccde7fc700 1 -- 172.21.15.189:0/5575000 learned_addr learned my addr 172.21.15.189:0/5575000 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:18.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.980+0000 7fccdf7fe700 1 --2- 172.21.15.189:0/5575000 >> v2:172.21.15.192:3300/0 conn(0x7fcce0143470 0x7fcce00cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:18.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.980+0000 7fccde7fc700 1 -- 172.21.15.189:0/5575000 >> v2:172.21.15.192:3300/0 conn(0x7fcce0143470 msgr2=0x7fcce00cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:18.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.980+0000 7fccde7fc700 1 --2- 172.21.15.189:0/5575000 >> v2:172.21.15.192:3300/0 conn(0x7fcce0143470 0x7fcce00cd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:18.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.980+0000 7fccde7fc700 1 -- 172.21.15.189:0/5575000 >> v1:172.21.15.192:6789/0 conn(0x7fcce0160b60 legacy=0x7fcce0142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:18.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.980+0000 7fccde7fc700 1 -- 172.21.15.189:0/5575000 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce015dfc0 con 0x7fcce015db10 2024-04-06T12:30:18.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.980+0000 7fccde7fc700 1 --2- 172.21.15.189:0/5575000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcce015db10 0x7fcce00b37f0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fcce0146700 tx=0x7fccd4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0b5369c4f6c2bb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:18.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.980+0000 7fccddffb700 1 -- 172.21.15.189:0/5575000 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd4005e30 con 0x7fcce015db10 2024-04-06T12:30:18.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.980+0000 7fccddffb700 1 -- 172.21.15.189:0/5575000 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd4004030 con 0x7fcce015db10 2024-04-06T12:30:18.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.980+0000 7fccddffb700 1 -- 172.21.15.189:0/5575000 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd40043e0 con 0x7fcce015db10 2024-04-06T12:30:18.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.981+0000 7fcce58f0700 1 -- 172.21.15.189:0/5575000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcce015db10 msgr2=0x7fcce00b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:18.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.981+0000 7fcce58f0700 1 --2- 172.21.15.189:0/5575000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcce015db10 0x7fcce00b37f0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fcce0146700 tx=0x7fccd4009bd0 comp rx=0 tx=0).stop 2024-04-06T12:30:18.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.981+0000 7fcce58f0700 1 -- 172.21.15.189:0/5575000 shutdown_connections 2024-04-06T12:30:18.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.981+0000 7fcce58f0700 1 --2- 172.21.15.189:0/5575000 >> v2:172.21.15.192:3300/0 conn(0x7fcce0143470 0x7fcce00cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:18.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.981+0000 7fcce58f0700 1 --2- 172.21.15.189:0/5575000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcce015db10 0x7fcce00b37f0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:18.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.981+0000 7fcce58f0700 1 -- 172.21.15.189:0/5575000 >> 172.21.15.189:0/5575000 conn(0x7fcce006a730 msgr2=0x7fcce006ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:18.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.981+0000 7fcce58f0700 1 -- 172.21.15.189:0/5575000 shutdown_connections 2024-04-06T12:30:18.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.981+0000 7fcce58f0700 1 -- 172.21.15.189:0/5575000 wait complete. 2024-04-06T12:30:18.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.982+0000 7fcce58f0700 1 Processor -- start 2024-04-06T12:30:18.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.982+0000 7fcce58f0700 1 -- start start 2024-04-06T12:30:18.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.983+0000 7fcce58f0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcce0143470 0x7fcce00c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:18.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.983+0000 7fcce58f0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcce015db10 0x7fcce00b68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:18.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.983+0000 7fcce58f0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcce0160b60 0x7fcce006cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:18.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.983+0000 7fcce58f0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce00c9a10 con 0x7fcce0143470 2024-04-06T12:30:18.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.983+0000 7fcce58f0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce00c9cf0 con 0x7fcce0160b60 2024-04-06T12:30:18.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.983+0000 7fcce58f0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce00c9fd0 con 0x7fcce015db10 2024-04-06T12:30:18.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.983+0000 7fccdeffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcce0143470 0x7fcce00c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:18.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.983+0000 7fccdeffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcce0143470 0x7fcce00c93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49824/0 (socket says 172.21.15.189:49824) 2024-04-06T12:30:18.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.983+0000 7fccdeffd700 1 -- 172.21.15.189:0/247355952 learned_addr learned my addr 172.21.15.189:0/247355952 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:18.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.983+0000 7fccde7fc700 1 --2- 172.21.15.189:0/247355952 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcce015db10 0x7fcce00b68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:18.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.983+0000 7fccdf7fe700 1 --2- 172.21.15.189:0/247355952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcce0160b60 0x7fcce006cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:18.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.983+0000 7fccdeffd700 1 -- 172.21.15.189:0/247355952 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcce015db10 msgr2=0x7fcce00b68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:18.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.983+0000 7fccdeffd700 1 --2- 172.21.15.189:0/247355952 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcce015db10 0x7fcce00b68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:18.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.983+0000 7fccdeffd700 1 -- 172.21.15.189:0/247355952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcce0160b60 msgr2=0x7fcce006cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:18.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.983+0000 7fccdeffd700 1 --2- 172.21.15.189:0/247355952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcce0160b60 0x7fcce006cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:18.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.983+0000 7fccdeffd700 1 -- 172.21.15.189:0/247355952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd4009500 con 0x7fcce0143470 2024-04-06T12:30:18.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.984+0000 7fccdeffd700 1 --2- 172.21.15.189:0/247355952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcce0143470 0x7fcce00c93c0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fccd000c450 tx=0x7fccd000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:18.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.984+0000 7fccc7fff700 1 -- 172.21.15.189:0/247355952 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd00052d0 con 0x7fcce0143470 2024-04-06T12:30:18.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.984+0000 7fccc7fff700 1 -- 172.21.15.189:0/247355952 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd0005430 con 0x7fcce0143470 2024-04-06T12:30:18.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.984+0000 7fcce58f0700 1 -- 172.21.15.189:0/247355952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce00ca2b0 con 0x7fcce0143470 2024-04-06T12:30:18.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.984+0000 7fccc7fff700 1 -- 172.21.15.189:0/247355952 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd00038c0 con 0x7fcce0143470 2024-04-06T12:30:18.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.984+0000 7fcce58f0700 1 -- 172.21.15.189:0/247355952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce00b6df0 con 0x7fcce0143470 2024-04-06T12:30:18.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.985+0000 7fcce58f0700 1 -- 172.21.15.189:0/247355952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce00ca7d0 con 0x7fcce0143470 2024-04-06T12:30:18.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.985+0000 7fccc7fff700 1 -- 172.21.15.189:0/247355952 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fccd0003a20 con 0x7fcce0143470 2024-04-06T12:30:18.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.986+0000 7fccc7fff700 1 --2- 172.21.15.189:0/247355952 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcccc06adb0 0x7fcccc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:18.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.986+0000 7fccc7fff700 1 -- 172.21.15.189:0/247355952 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fccd0088810 con 0x7fcce0143470 2024-04-06T12:30:18.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.986+0000 7fccde7fc700 1 --2- 172.21.15.189:0/247355952 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcccc06adb0 0x7fcccc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:18.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.987+0000 7fccde7fc700 1 --2- 172.21.15.189:0/247355952 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcccc06adb0 0x7fcccc06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcce0146700 tx=0x7fccd4008070 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:18.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:17.990+0000 7fccc7fff700 1 -- 172.21.15.189:0/247355952 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fccd0053780 con 0x7fcce0143470 2024-04-06T12:30:18.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:18.230+0000 7fcce58f0700 1 -- 172.21.15.189:0/247355952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fcce015eb40 con 0x7fcce0143470 2024-04-06T12:30:19.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.145+0000 7fccc7fff700 1 -- 172.21.15.189:0/247355952 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v53) v1 ==== 118+0+0 (secure 0 0 0) 0x7fcce015eb40 con 0x7fcce0143470 2024-04-06T12:30:19.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.148+0000 7fcce58f0700 1 -- 172.21.15.189:0/247355952 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcccc06adb0 msgr2=0x7fcccc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:19.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.148+0000 7fcce58f0700 1 --2- 172.21.15.189:0/247355952 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcccc06adb0 0x7fcccc06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcce0146700 tx=0x7fccd4008070 comp rx=0 tx=0).stop 2024-04-06T12:30:19.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.148+0000 7fcce58f0700 1 -- 172.21.15.189:0/247355952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcce0143470 msgr2=0x7fcce00c93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:19.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.148+0000 7fcce58f0700 1 --2- 172.21.15.189:0/247355952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcce0143470 0x7fcce00c93c0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fccd000c450 tx=0x7fccd000c810 comp rx=0 tx=0).stop 2024-04-06T12:30:19.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.149+0000 7fcce58f0700 1 -- 172.21.15.189:0/247355952 shutdown_connections 2024-04-06T12:30:19.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.149+0000 7fcce58f0700 1 --2- 172.21.15.189:0/247355952 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcccc06adb0 0x7fcccc06d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:19.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.149+0000 7fcce58f0700 1 --2- 172.21.15.189:0/247355952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcce0160b60 0x7fcce006cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:19.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.149+0000 7fcce58f0700 1 --2- 172.21.15.189:0/247355952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcce0143470 0x7fcce00c93c0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:19.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.149+0000 7fcce58f0700 1 --2- 172.21.15.189:0/247355952 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcce015db10 0x7fcce00b68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:19.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.149+0000 7fcce58f0700 1 -- 172.21.15.189:0/247355952 >> 172.21.15.189:0/247355952 conn(0x7fcce006a730 msgr2=0x7fcce015fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:19.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.149+0000 7fcce58f0700 1 -- 172.21.15.189:0/247355952 shutdown_connections 2024-04-06T12:30:19.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.149+0000 7fcce58f0700 1 -- 172.21.15.189:0/247355952 wait complete. 2024-04-06T12:30:19.160 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-06T12:30:19.161 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-06T12:30:19.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.294+0000 7f6e16445700 1 Processor -- start 2024-04-06T12:30:19.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.294+0000 7f6e16445700 1 -- start start 2024-04-06T12:30:19.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.294+0000 7f6e16445700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6e1015d7b0 0x7f6e100b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:19.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.294+0000 7f6e16445700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6e1006a730 0x7f6e1006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:19.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.294+0000 7f6e16445700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e1015ddf0 0x7f6e1015e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:19.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.294+0000 7f6e16445700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e1006b0f0 con 0x7f6e1006a730 2024-04-06T12:30:19.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.294+0000 7f6e16445700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e100b3990 con 0x7f6e1015d7b0 2024-04-06T12:30:19.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.294+0000 7f6e16445700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e100b3060 con 0x7f6e1015ddf0 2024-04-06T12:30:19.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.294+0000 7f6e0f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6e1006a730 0x7f6e1006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:19.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.295+0000 7f6e0f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6e1006a730 0x7f6e1006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40668/0 (socket says 172.21.15.189:40668) 2024-04-06T12:30:19.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.295+0000 7f6e0f7fe700 1 -- 172.21.15.189:0/3375198173 learned_addr learned my addr 172.21.15.189:0/3375198173 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:19.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.295+0000 7f6e0ffff700 1 --2- 172.21.15.189:0/3375198173 >> v2:172.21.15.192:3300/0 conn(0x7f6e1015d7b0 0x7f6e100b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:19.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.295+0000 7f6e149e2700 1 --2- 172.21.15.189:0/3375198173 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e1015ddf0 0x7f6e1015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:19.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.295+0000 7f6e0f7fe700 1 -- 172.21.15.189:0/3375198173 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e1015ddf0 msgr2=0x7f6e1015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:19.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.295+0000 7f6e0f7fe700 1 --2- 172.21.15.189:0/3375198173 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e1015ddf0 0x7f6e1015e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:19.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.295+0000 7f6e0f7fe700 1 -- 172.21.15.189:0/3375198173 >> v2:172.21.15.192:3300/0 conn(0x7f6e1015d7b0 msgr2=0x7f6e100b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:19.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.295+0000 7f6e0f7fe700 1 --2- 172.21.15.189:0/3375198173 >> v2:172.21.15.192:3300/0 conn(0x7f6e1015d7b0 0x7f6e100b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:19.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.295+0000 7f6e0f7fe700 1 -- 172.21.15.189:0/3375198173 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e100b3340 con 0x7f6e1006a730 2024-04-06T12:30:19.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.295+0000 7f6e0ffff700 1 --2- 172.21.15.189:0/3375198173 >> v2:172.21.15.192:3300/0 conn(0x7f6e1015d7b0 0x7f6e100b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:30:19.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.295+0000 7f6e0f7fe700 1 --2- 172.21.15.189:0/3375198173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e1006a730 0x7f6e1006abb0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f6e100b6200 tx=0x7f6df8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de5f5b90136edddc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:19.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.295+0000 7f6e0effd700 1 -- 172.21.15.189:0/3375198173 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df800b870 con 0x7f6e1006a730 2024-04-06T12:30:19.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.295+0000 7f6e0effd700 1 -- 172.21.15.189:0/3375198173 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df800b9d0 con 0x7f6e1006a730 2024-04-06T12:30:19.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.295+0000 7f6e0effd700 1 -- 172.21.15.189:0/3375198173 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df800bd60 con 0x7f6e1006a730 2024-04-06T12:30:19.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.296+0000 7f6e16445700 1 -- 172.21.15.189:0/3375198173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e1006a730 msgr2=0x7f6e1006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:19.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.296+0000 7f6e16445700 1 --2- 172.21.15.189:0/3375198173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e1006a730 0x7f6e1006abb0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f6e100b6200 tx=0x7f6df8009bd0 comp rx=0 tx=0).stop 2024-04-06T12:30:19.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.296+0000 7f6e16445700 1 -- 172.21.15.189:0/3375198173 shutdown_connections 2024-04-06T12:30:19.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.296+0000 7f6e16445700 1 --2- 172.21.15.189:0/3375198173 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e1015ddf0 0x7f6e1015e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:19.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.296+0000 7f6e16445700 1 --2- 172.21.15.189:0/3375198173 >> v2:172.21.15.192:3300/0 conn(0x7f6e1015d7b0 0x7f6e100b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:19.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.296+0000 7f6e16445700 1 --2- 172.21.15.189:0/3375198173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e1006a730 0x7f6e1006abb0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:19.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.296+0000 7f6e16445700 1 -- 172.21.15.189:0/3375198173 >> 172.21.15.189:0/3375198173 conn(0x7f6e100d00b0 msgr2=0x7f6e100d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:19.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.296+0000 7f6e16445700 1 -- 172.21.15.189:0/3375198173 shutdown_connections 2024-04-06T12:30:19.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.296+0000 7f6e16445700 1 -- 172.21.15.189:0/3375198173 wait complete. 2024-04-06T12:30:19.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.297+0000 7f6e16445700 1 Processor -- start 2024-04-06T12:30:19.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.297+0000 7f6e16445700 1 -- start start 2024-04-06T12:30:19.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.298+0000 7f6e16445700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e1006a730 0x7f6e101508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:19.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.298+0000 7f6e16445700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e1015d7b0 0x7f6e100cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:19.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.298+0000 7f6e16445700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e1015ddf0 0x7f6e100cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:19.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.298+0000 7f6e16445700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e10150df0 con 0x7f6e1015ddf0 2024-04-06T12:30:19.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.298+0000 7f6e16445700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e100ce1b0 con 0x7f6e1015d7b0 2024-04-06T12:30:19.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.298+0000 7f6e16445700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1015bbe0 con 0x7f6e1006a730 2024-04-06T12:30:19.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.298+0000 7f6e149e2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e1015ddf0 0x7f6e100cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:19.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.298+0000 7f6e149e2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e1015ddf0 0x7f6e100cdb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40678/0 (socket says 172.21.15.189:40678) 2024-04-06T12:30:19.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.298+0000 7f6e149e2700 1 -- 172.21.15.189:0/3538350119 learned_addr learned my addr 172.21.15.189:0/3538350119 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:19.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.298+0000 7f6e0f7fe700 1 --2- 172.21.15.189:0/3538350119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e1015d7b0 0x7f6e100cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:19.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.298+0000 7f6e0ffff700 1 --2- 172.21.15.189:0/3538350119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e1006a730 0x7f6e101508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:19.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.298+0000 7f6e149e2700 1 -- 172.21.15.189:0/3538350119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e1006a730 msgr2=0x7f6e101508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:19.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.298+0000 7f6e149e2700 1 --2- 172.21.15.189:0/3538350119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e1006a730 0x7f6e101508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:19.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.298+0000 7f6e149e2700 1 -- 172.21.15.189:0/3538350119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e1015d7b0 msgr2=0x7f6e100cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:19.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.298+0000 7f6e149e2700 1 --2- 172.21.15.189:0/3538350119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e1015d7b0 0x7f6e100cd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:19.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.298+0000 7f6e149e2700 1 -- 172.21.15.189:0/3538350119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df8009550 con 0x7f6e1015ddf0 2024-04-06T12:30:19.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.298+0000 7f6e0ffff700 1 --2- 172.21.15.189:0/3538350119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e1006a730 0x7f6e101508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:19.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.298+0000 7f6e0f7fe700 1 --2- 172.21.15.189:0/3538350119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e1015d7b0 0x7f6e100cd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:19.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.299+0000 7f6e149e2700 1 --2- 172.21.15.189:0/3538350119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e1015ddf0 0x7f6e100cdb80 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f6e0800cc30 tx=0x7f6e0800a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:19.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.299+0000 7f6e0cff9700 1 -- 172.21.15.189:0/3538350119 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0800ab70 con 0x7f6e1015ddf0 2024-04-06T12:30:19.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.299+0000 7f6e0cff9700 1 -- 172.21.15.189:0/3538350119 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0800acd0 con 0x7f6e1015ddf0 2024-04-06T12:30:19.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.299+0000 7f6e16445700 1 -- 172.21.15.189:0/3538350119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e1015a4f0 con 0x7f6e1015ddf0 2024-04-06T12:30:19.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.299+0000 7f6e0cff9700 1 -- 172.21.15.189:0/3538350119 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e08003c40 con 0x7f6e1015ddf0 2024-04-06T12:30:19.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.299+0000 7f6e16445700 1 -- 172.21.15.189:0/3538350119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1015aab0 con 0x7f6e1015ddf0 2024-04-06T12:30:19.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.300+0000 7f6e16445700 1 -- 172.21.15.189:0/3538350119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e10067bb0 con 0x7f6e1015ddf0 2024-04-06T12:30:19.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.300+0000 7f6e0cff9700 1 -- 172.21.15.189:0/3538350119 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f6e0800f3f0 con 0x7f6e1015ddf0 2024-04-06T12:30:19.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.300+0000 7f6e0cff9700 1 --2- 172.21.15.189:0/3538350119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6df006adb0 0x7f6df006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:19.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.301+0000 7f6e0cff9700 1 -- 172.21.15.189:0/3538350119 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6e08089880 con 0x7f6e1015ddf0 2024-04-06T12:30:19.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.301+0000 7f6e0ffff700 1 --2- 172.21.15.189:0/3538350119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6df006adb0 0x7f6df006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:19.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.301+0000 7f6e0ffff700 1 --2- 172.21.15.189:0/3538350119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6df006adb0 0x7f6df006d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6e00006fd0 tx=0x7f6e00008040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:19.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.305+0000 7f6e0cff9700 1 -- 172.21.15.189:0/3538350119 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6e08054700 con 0x7f6e1015ddf0 2024-04-06T12:30:19.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:19.546+0000 7f6e16445700 1 -- 172.21.15.189:0/3538350119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6e10074990 con 0x7f6e1015ddf0 2024-04-06T12:30:20.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.151+0000 7f6e0cff9700 1 -- 172.21.15.189:0/3538350119 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v54) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6e0801d070 con 0x7f6e1015ddf0 2024-04-06T12:30:20.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.154+0000 7f6e16445700 1 -- 172.21.15.189:0/3538350119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6df006adb0 msgr2=0x7f6df006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:20.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.154+0000 7f6e16445700 1 --2- 172.21.15.189:0/3538350119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6df006adb0 0x7f6df006d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6e00006fd0 tx=0x7f6e00008040 comp rx=0 tx=0).stop 2024-04-06T12:30:20.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.154+0000 7f6e16445700 1 -- 172.21.15.189:0/3538350119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e1015ddf0 msgr2=0x7f6e100cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:20.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.154+0000 7f6e16445700 1 --2- 172.21.15.189:0/3538350119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e1015ddf0 0x7f6e100cdb80 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f6e0800cc30 tx=0x7f6e0800a1f0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.154+0000 7f6e16445700 1 -- 172.21.15.189:0/3538350119 shutdown_connections 2024-04-06T12:30:20.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.154+0000 7f6e16445700 1 --2- 172.21.15.189:0/3538350119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e1015ddf0 0x7f6e100cdb80 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.154+0000 7f6e16445700 1 --2- 172.21.15.189:0/3538350119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e1006a730 0x7f6e101508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.154+0000 7f6e16445700 1 --2- 172.21.15.189:0/3538350119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6df006adb0 0x7f6df006d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.154+0000 7f6e16445700 1 --2- 172.21.15.189:0/3538350119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e1015d7b0 0x7f6e100cd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.154+0000 7f6e16445700 1 -- 172.21.15.189:0/3538350119 >> 172.21.15.189:0/3538350119 conn(0x7f6e100d00b0 msgr2=0x7f6e100d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:20.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.154+0000 7f6e16445700 1 -- 172.21.15.189:0/3538350119 shutdown_connections 2024-04-06T12:30:20.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.154+0000 7f6e16445700 1 -- 172.21.15.189:0/3538350119 wait complete. 2024-04-06T12:30:20.166 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-06T12:30:20.166 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:30:20.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.296+0000 7fb02a6f3700 1 Processor -- start 2024-04-06T12:30:20.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.296+0000 7fb02a6f3700 1 -- start start 2024-04-06T12:30:20.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.297+0000 7fb02a6f3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb02406ae40 0x7fb0240b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:20.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.297+0000 7fb02a6f3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb02415ddf0 0x7fb02415e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:20.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.297+0000 7fb02a6f3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0240b3990 con 0x7fb02415d7b0 2024-04-06T12:30:20.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.297+0000 7fb02a6f3700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0240b3060 con 0x7fb02415ddf0 2024-04-06T12:30:20.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.297+0000 7fb02a6f3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0240b3340 con 0x7fb02406ae40 2024-04-06T12:30:20.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.297+0000 7fb028c90700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb02415ddf0 0x7fb02415e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:20.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.297+0000 7fb023fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb02415d7b0 0x7fb02406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45532/0 (socket says 172.21.15.189:45532) 2024-04-06T12:30:20.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.297+0000 7fb023fff700 1 -- 172.21.15.189:0/4209543779 learned_addr learned my addr 172.21.15.189:0/4209543779 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:20.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.297+0000 7fb0237fe700 1 --2- 172.21.15.189:0/4209543779 >> v2:172.21.15.192:3300/0 conn(0x7fb02406ae40 0x7fb0240b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:20.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.297+0000 7fb028c90700 1 -- 172.21.15.189:0/4209543779 >> v2:172.21.15.192:3300/0 conn(0x7fb02406ae40 msgr2=0x7fb0240b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:20.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.297+0000 7fb028c90700 1 --2- 172.21.15.189:0/4209543779 >> v2:172.21.15.192:3300/0 conn(0x7fb02406ae40 0x7fb0240b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.297+0000 7fb028c90700 1 -- 172.21.15.189:0/4209543779 >> v1:172.21.15.189:6789/0 conn(0x7fb02415d7b0 legacy=0x7fb02406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:20.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.297+0000 7fb028c90700 1 -- 172.21.15.189:0/4209543779 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0240b3620 con 0x7fb02415ddf0 2024-04-06T12:30:20.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.297+0000 7fb0237fe700 1 --2- 172.21.15.189:0/4209543779 >> v2:172.21.15.192:3300/0 conn(0x7fb02406ae40 0x7fb0240b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:30:20.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.297+0000 7fb028c90700 1 --2- 172.21.15.189:0/4209543779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb02415ddf0 0x7fb02415e240 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fb0240afc00 tx=0x7fb01c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda4b13dd85169c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:20.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.298+0000 7fb022ffd700 1 -- 172.21.15.189:0/4209543779 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c00e070 con 0x7fb02415ddf0 2024-04-06T12:30:20.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.298+0000 7fb022ffd700 1 -- 172.21.15.189:0/4209543779 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01c00bbd0 con 0x7fb02415ddf0 2024-04-06T12:30:20.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.298+0000 7fb022ffd700 1 -- 172.21.15.189:0/4209543779 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c012430 con 0x7fb02415ddf0 2024-04-06T12:30:20.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.298+0000 7fb02a6f3700 1 -- 172.21.15.189:0/4209543779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb02415ddf0 msgr2=0x7fb02415e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:20.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.298+0000 7fb02a6f3700 1 --2- 172.21.15.189:0/4209543779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb02415ddf0 0x7fb02415e240 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fb0240afc00 tx=0x7fb01c00b330 comp rx=0 tx=0).stop 2024-04-06T12:30:20.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.298+0000 7fb02a6f3700 1 -- 172.21.15.189:0/4209543779 shutdown_connections 2024-04-06T12:30:20.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.298+0000 7fb02a6f3700 1 --2- 172.21.15.189:0/4209543779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb02415ddf0 0x7fb02415e240 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.298+0000 7fb02a6f3700 1 --2- 172.21.15.189:0/4209543779 >> v2:172.21.15.192:3300/0 conn(0x7fb02406ae40 0x7fb0240b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.298+0000 7fb02a6f3700 1 -- 172.21.15.189:0/4209543779 >> 172.21.15.189:0/4209543779 conn(0x7fb0240d00b0 msgr2=0x7fb0240d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:20.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.298+0000 7fb02a6f3700 1 -- 172.21.15.189:0/4209543779 shutdown_connections 2024-04-06T12:30:20.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.298+0000 7fb02a6f3700 1 -- 172.21.15.189:0/4209543779 wait complete. 2024-04-06T12:30:20.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.300+0000 7fb02a6f3700 1 Processor -- start 2024-04-06T12:30:20.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.300+0000 7fb02a6f3700 1 -- start start 2024-04-06T12:30:20.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb02a6f3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb02406ae40 0x7fb0241f3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:20.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb02a6f3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb02415d7b0 0x7fb02415b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:20.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb02a6f3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb02415ddf0 0x7fb0241f6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:20.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb02a6f3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02415b550 con 0x7fb02415d7b0 2024-04-06T12:30:20.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb02a6f3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0241f39d0 con 0x7fb02415ddf0 2024-04-06T12:30:20.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb02a6f3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02415a6a0 con 0x7fb02406ae40 2024-04-06T12:30:20.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb0237fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb02415d7b0 0x7fb02415b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:20.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb0237fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb02415d7b0 0x7fb02415b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40704/0 (socket says 172.21.15.189:40704) 2024-04-06T12:30:20.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb0237fe700 1 -- 172.21.15.189:0/244242348 learned_addr learned my addr 172.21.15.189:0/244242348 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:20.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb028c90700 1 --2- 172.21.15.189:0/244242348 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb02415ddf0 0x7fb0241f6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:20.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb023fff700 1 --2- 172.21.15.189:0/244242348 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb02406ae40 0x7fb0241f3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:20.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb0237fe700 1 -- 172.21.15.189:0/244242348 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb02406ae40 msgr2=0x7fb0241f3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:20.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb0237fe700 1 --2- 172.21.15.189:0/244242348 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb02406ae40 0x7fb0241f3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb0237fe700 1 -- 172.21.15.189:0/244242348 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb02415ddf0 msgr2=0x7fb0241f6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:20.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb0237fe700 1 --2- 172.21.15.189:0/244242348 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb02415ddf0 0x7fb0241f6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb0237fe700 1 -- 172.21.15.189:0/244242348 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c009a70 con 0x7fb02415d7b0 2024-04-06T12:30:20.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb028c90700 1 --2- 172.21.15.189:0/244242348 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb02415ddf0 0x7fb0241f6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:20.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb0237fe700 1 --2- 172.21.15.189:0/244242348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb02415d7b0 0x7fb02415b9e0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fb01400c740 tx=0x7fb01400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:20.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb020ff9700 1 -- 172.21.15.189:0/244242348 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb014004220 con 0x7fb02415d7b0 2024-04-06T12:30:20.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb020ff9700 1 -- 172.21.15.189:0/244242348 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb014004380 con 0x7fb02415d7b0 2024-04-06T12:30:20.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb020ff9700 1 -- 172.21.15.189:0/244242348 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0140036a0 con 0x7fb02415d7b0 2024-04-06T12:30:20.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.301+0000 7fb02a6f3700 1 -- 172.21.15.189:0/244242348 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb02415a940 con 0x7fb02415d7b0 2024-04-06T12:30:20.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.302+0000 7fb02a6f3700 1 -- 172.21.15.189:0/244242348 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb02415af00 con 0x7fb02415d7b0 2024-04-06T12:30:20.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.303+0000 7fb02a6f3700 1 -- 172.21.15.189:0/244242348 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb024067bb0 con 0x7fb02415d7b0 2024-04-06T12:30:20.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.303+0000 7fb020ff9700 1 -- 172.21.15.189:0/244242348 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fb014003800 con 0x7fb02415d7b0 2024-04-06T12:30:20.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.303+0000 7fb020ff9700 1 --2- 172.21.15.189:0/244242348 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb01006adb0 0x7fb01006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:20.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.303+0000 7fb020ff9700 1 -- 172.21.15.189:0/244242348 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb014089a20 con 0x7fb02415d7b0 2024-04-06T12:30:20.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.303+0000 7fb023fff700 1 --2- 172.21.15.189:0/244242348 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb01006adb0 0x7fb01006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:20.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.304+0000 7fb023fff700 1 --2- 172.21.15.189:0/244242348 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb01006adb0 0x7fb01006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb018005910 tx=0x7fb018009040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:20.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.308+0000 7fb020ff9700 1 -- 172.21.15.189:0/244242348 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb0140548b0 con 0x7fb02415d7b0 2024-04-06T12:30:20.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.547+0000 7fb02a6f3700 1 -- 172.21.15.189:0/244242348 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb024074990 con 0x7fb02415d7b0 2024-04-06T12:30:20.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.547+0000 7fb020ff9700 1 -- 172.21.15.189:0/244242348 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4739 (secure 0 0 0) 0x7fb014057ed0 con 0x7fb02415d7b0 2024-04-06T12:30:20.545 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:30:20.545 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/749266547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":749266547},{"type":"v1","addr":"172.21.15.192:6835","nonce":749266547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":6764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/947183051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":947183051},{"type":"v1","addr":"172.21.15.192:6834","nonce":947183051}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7492,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3990051444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3990051444},{"type":"v1","addr":"172.21.15.189:6837","nonce":3990051444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:30:17.127295+0000","modified":"2024-04-06T12:30:20.144458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8088},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8088":{"gid":8088,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/393232044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":393232044},{"type":"v1","addr":"172.21.15.189:6836","nonce":393232044}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-06T12:30:20.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.549+0000 7fb02a6f3700 1 -- 172.21.15.189:0/244242348 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb01006adb0 msgr2=0x7fb01006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:20.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.549+0000 7fb02a6f3700 1 --2- 172.21.15.189:0/244242348 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb01006adb0 0x7fb01006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb018005910 tx=0x7fb018009040 comp rx=0 tx=0).stop 2024-04-06T12:30:20.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.549+0000 7fb02a6f3700 1 -- 172.21.15.189:0/244242348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb02415d7b0 msgr2=0x7fb02415b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:20.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.549+0000 7fb02a6f3700 1 --2- 172.21.15.189:0/244242348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb02415d7b0 0x7fb02415b9e0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fb01400c740 tx=0x7fb01400cb00 comp rx=0 tx=0).stop 2024-04-06T12:30:20.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.549+0000 7fb02a6f3700 1 -- 172.21.15.189:0/244242348 shutdown_connections 2024-04-06T12:30:20.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.549+0000 7fb02a6f3700 1 --2- 172.21.15.189:0/244242348 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb01006adb0 0x7fb01006d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.549+0000 7fb02a6f3700 1 --2- 172.21.15.189:0/244242348 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb02415ddf0 0x7fb0241f6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.549+0000 7fb02a6f3700 1 --2- 172.21.15.189:0/244242348 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb02406ae40 0x7fb0241f3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.549+0000 7fb02a6f3700 1 --2- 172.21.15.189:0/244242348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb02415d7b0 0x7fb02415b9e0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.549+0000 7fb02a6f3700 1 -- 172.21.15.189:0/244242348 >> 172.21.15.189:0/244242348 conn(0x7fb0240d00b0 msgr2=0x7fb02406dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:20.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.550+0000 7fb02a6f3700 1 -- 172.21.15.189:0/244242348 shutdown_connections 2024-04-06T12:30:20.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.550+0000 7fb02a6f3700 1 -- 172.21.15.189:0/244242348 wait complete. 2024-04-06T12:30:20.548 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 54 2024-04-06T12:30:20.559 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:30:20.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.684+0000 7fea25b1e700 1 Processor -- start 2024-04-06T12:30:20.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.684+0000 7fea25b1e700 1 -- start start 2024-04-06T12:30:20.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea25b1e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fea2015d7b0 0x7fea200b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:20.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea25b1e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fea2006a730 0x7fea2006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:20.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea25b1e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea2015ddf0 0x7fea2015e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:20.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea25b1e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea2006b0f0 con 0x7fea2015d7b0 2024-04-06T12:30:20.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea25b1e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea200b3990 con 0x7fea2006a730 2024-04-06T12:30:20.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea25b1e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea200b3060 con 0x7fea2015ddf0 2024-04-06T12:30:20.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea1f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fea2015d7b0 0x7fea200b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:20.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea1f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fea2015d7b0 0x7fea200b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40710/0 (socket says 172.21.15.189:40710) 2024-04-06T12:30:20.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea1f7fe700 1 -- 172.21.15.189:0/1188512370 learned_addr learned my addr 172.21.15.189:0/1188512370 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:20.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea1effd700 1 --2- 172.21.15.189:0/1188512370 >> v2:172.21.15.192:3300/0 conn(0x7fea2006a730 0x7fea2006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:20.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea1ffff700 1 --2- 172.21.15.189:0/1188512370 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea2015ddf0 0x7fea2015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:20.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea1f7fe700 1 -- 172.21.15.189:0/1188512370 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea2015ddf0 msgr2=0x7fea2015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:20.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea1f7fe700 1 --2- 172.21.15.189:0/1188512370 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea2015ddf0 0x7fea2015e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea1f7fe700 1 -- 172.21.15.189:0/1188512370 >> v2:172.21.15.192:3300/0 conn(0x7fea2006a730 msgr2=0x7fea2006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:20.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea1f7fe700 1 --2- 172.21.15.189:0/1188512370 >> v2:172.21.15.192:3300/0 conn(0x7fea2006a730 0x7fea2006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea1f7fe700 1 -- 172.21.15.189:0/1188512370 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea200b3340 con 0x7fea2015d7b0 2024-04-06T12:30:20.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea1ffff700 1 --2- 172.21.15.189:0/1188512370 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea2015ddf0 0x7fea2015e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:30:20.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea1effd700 1 --2- 172.21.15.189:0/1188512370 >> v2:172.21.15.192:3300/0 conn(0x7fea2006a730 0x7fea2006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:30:20.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea1f7fe700 1 --2- 172.21.15.189:0/1188512370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea2015d7b0 0x7fea200b3e20 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fea200afc00 tx=0x7fea10009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c43766d191a446d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:20.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea1e7fc700 1 -- 172.21.15.189:0/1188512370 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1000b870 con 0x7fea2015d7b0 2024-04-06T12:30:20.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea1e7fc700 1 -- 172.21.15.189:0/1188512370 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea1000b9d0 con 0x7fea2015d7b0 2024-04-06T12:30:20.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.685+0000 7fea1e7fc700 1 -- 172.21.15.189:0/1188512370 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1000bd60 con 0x7fea2015d7b0 2024-04-06T12:30:20.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.686+0000 7fea25b1e700 1 -- 172.21.15.189:0/1188512370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea2015d7b0 msgr2=0x7fea200b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:20.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.686+0000 7fea25b1e700 1 --2- 172.21.15.189:0/1188512370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea2015d7b0 0x7fea200b3e20 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fea200afc00 tx=0x7fea10009bd0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.686+0000 7fea25b1e700 1 -- 172.21.15.189:0/1188512370 shutdown_connections 2024-04-06T12:30:20.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.686+0000 7fea25b1e700 1 --2- 172.21.15.189:0/1188512370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea2015d7b0 0x7fea200b3e20 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.686+0000 7fea25b1e700 1 --2- 172.21.15.189:0/1188512370 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea2015ddf0 0x7fea2015e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.686+0000 7fea25b1e700 1 --2- 172.21.15.189:0/1188512370 >> v2:172.21.15.192:3300/0 conn(0x7fea2006a730 0x7fea2006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.686+0000 7fea25b1e700 1 -- 172.21.15.189:0/1188512370 >> 172.21.15.189:0/1188512370 conn(0x7fea200d00b0 msgr2=0x7fea200d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:20.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.686+0000 7fea25b1e700 1 -- 172.21.15.189:0/1188512370 shutdown_connections 2024-04-06T12:30:20.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.686+0000 7fea25b1e700 1 -- 172.21.15.189:0/1188512370 wait complete. 2024-04-06T12:30:20.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.687+0000 7fea25b1e700 1 Processor -- start 2024-04-06T12:30:20.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.687+0000 7fea25b1e700 1 -- start start 2024-04-06T12:30:20.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.688+0000 7fea25b1e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea2006a730 0x7fea20148270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:20.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.688+0000 7fea25b1e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea2015d7b0 0x7fea201f6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:20.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.688+0000 7fea25b1e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea2015ddf0 0x7fea201f65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:20.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.688+0000 7fea25b1e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea20148d30 con 0x7fea2015d7b0 2024-04-06T12:30:20.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.688+0000 7fea25b1e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea201492f0 con 0x7fea2006a730 2024-04-06T12:30:20.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.688+0000 7fea25b1e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea2015b9b0 con 0x7fea2015ddf0 2024-04-06T12:30:20.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.688+0000 7fea1effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea2015d7b0 0x7fea201f6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:20.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.688+0000 7fea1effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea2015d7b0 0x7fea201f6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40716/0 (socket says 172.21.15.189:40716) 2024-04-06T12:30:20.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.688+0000 7fea1effd700 1 -- 172.21.15.189:0/1622256176 learned_addr learned my addr 172.21.15.189:0/1622256176 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:20.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.688+0000 7fea1f7fe700 1 --2- 172.21.15.189:0/1622256176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea2006a730 0x7fea20148270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:20.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.688+0000 7fea1ffff700 1 --2- 172.21.15.189:0/1622256176 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea2015ddf0 0x7fea201f65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:20.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.688+0000 7fea1effd700 1 -- 172.21.15.189:0/1622256176 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea2015ddf0 msgr2=0x7fea201f65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:20.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.688+0000 7fea1effd700 1 --2- 172.21.15.189:0/1622256176 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea2015ddf0 0x7fea201f65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.688+0000 7fea1effd700 1 -- 172.21.15.189:0/1622256176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea2006a730 msgr2=0x7fea20148270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:20.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.688+0000 7fea1effd700 1 --2- 172.21.15.189:0/1622256176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea2006a730 0x7fea20148270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.689+0000 7fea1effd700 1 -- 172.21.15.189:0/1622256176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea10009550 con 0x7fea2015d7b0 2024-04-06T12:30:20.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.689+0000 7fea1f7fe700 1 --2- 172.21.15.189:0/1622256176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea2006a730 0x7fea20148270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:20.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.689+0000 7fea1effd700 1 --2- 172.21.15.189:0/1622256176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea2015d7b0 0x7fea201f6070 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fea1400c750 tx=0x7fea1400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:20.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.689+0000 7fea07fff700 1 -- 172.21.15.189:0/1622256176 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea14011800 con 0x7fea2015d7b0 2024-04-06T12:30:20.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.689+0000 7fea07fff700 1 -- 172.21.15.189:0/1622256176 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea14011960 con 0x7fea2015d7b0 2024-04-06T12:30:20.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.689+0000 7fea25b1e700 1 -- 172.21.15.189:0/1622256176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea2015bc90 con 0x7fea2015d7b0 2024-04-06T12:30:20.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.689+0000 7fea07fff700 1 -- 172.21.15.189:0/1622256176 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea14011cd0 con 0x7fea2015d7b0 2024-04-06T12:30:20.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.689+0000 7fea25b1e700 1 -- 172.21.15.189:0/1622256176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea201487b0 con 0x7fea2015d7b0 2024-04-06T12:30:20.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.690+0000 7fea25b1e700 1 -- 172.21.15.189:0/1622256176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea20067bb0 con 0x7fea2015d7b0 2024-04-06T12:30:20.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.690+0000 7fea07fff700 1 -- 172.21.15.189:0/1622256176 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fea14021350 con 0x7fea2015d7b0 2024-04-06T12:30:20.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.691+0000 7fea07fff700 1 --2- 172.21.15.189:0/1622256176 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fea0c06adb0 0x7fea0c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:20.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.691+0000 7fea07fff700 1 -- 172.21.15.189:0/1622256176 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fea14089670 con 0x7fea2015d7b0 2024-04-06T12:30:20.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.691+0000 7fea1f7fe700 1 --2- 172.21.15.189:0/1622256176 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fea0c06adb0 0x7fea0c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:20.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.691+0000 7fea1f7fe700 1 --2- 172.21.15.189:0/1622256176 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fea0c06adb0 0x7fea0c06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fea10000c00 tx=0x7fea10008030 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:20.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.695+0000 7fea07fff700 1 -- 172.21.15.189:0/1622256176 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fea14054580 con 0x7fea2015d7b0 2024-04-06T12:30:20.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.892+0000 7fea25b1e700 1 -- 172.21.15.189:0/1622256176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea20149460 con 0x7fea2015d7b0 2024-04-06T12:30:20.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.893+0000 7fea07fff700 1 -- 172.21.15.189:0/1622256176 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+16296 (secure 0 0 0) 0x7fea14057ba0 con 0x7fea2015d7b0 2024-04-06T12:30:20.890 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:30:20.890 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":128,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:30:18.128990+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:30:14.958335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-06T12:30:16.104585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:30:20.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.895+0000 7fea25b1e700 1 -- 172.21.15.189:0/1622256176 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fea0c06adb0 msgr2=0x7fea0c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:20.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.895+0000 7fea25b1e700 1 --2- 172.21.15.189:0/1622256176 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fea0c06adb0 0x7fea0c06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fea10000c00 tx=0x7fea10008030 comp rx=0 tx=0).stop 2024-04-06T12:30:20.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.895+0000 7fea25b1e700 1 -- 172.21.15.189:0/1622256176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea2015d7b0 msgr2=0x7fea201f6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:20.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.895+0000 7fea25b1e700 1 --2- 172.21.15.189:0/1622256176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea2015d7b0 0x7fea201f6070 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fea1400c750 tx=0x7fea1400cb10 comp rx=0 tx=0).stop 2024-04-06T12:30:20.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.895+0000 7fea25b1e700 1 -- 172.21.15.189:0/1622256176 shutdown_connections 2024-04-06T12:30:20.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.895+0000 7fea25b1e700 1 --2- 172.21.15.189:0/1622256176 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea2015ddf0 0x7fea201f65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.895+0000 7fea25b1e700 1 --2- 172.21.15.189:0/1622256176 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fea0c06adb0 0x7fea0c06d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.895+0000 7fea25b1e700 1 --2- 172.21.15.189:0/1622256176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea2006a730 0x7fea20148270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.895+0000 7fea25b1e700 1 --2- 172.21.15.189:0/1622256176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea2015d7b0 0x7fea201f6070 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:20.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.895+0000 7fea25b1e700 1 -- 172.21.15.189:0/1622256176 >> 172.21.15.189:0/1622256176 conn(0x7fea200d00b0 msgr2=0x7fea200723d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:20.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.895+0000 7fea25b1e700 1 -- 172.21.15.189:0/1622256176 shutdown_connections 2024-04-06T12:30:20.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:20.895+0000 7fea25b1e700 1 -- 172.21.15.189:0/1622256176 wait complete. 2024-04-06T12:30:20.905 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-06T12:30:20.905 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:30:21.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.031+0000 7f9bbbfa0700 1 Processor -- start 2024-04-06T12:30:21.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.031+0000 7f9bbbfa0700 1 -- start start 2024-04-06T12:30:21.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.032+0000 7f9bbbfa0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9bb40d11b0 0x7f9bb40d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.032+0000 7f9bbbfa0700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9bb40cfa00 0x7f9bb40cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.032+0000 7f9bbbfa0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb40d1ad0 con 0x7f9bb40cb870 2024-04-06T12:30:21.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.032+0000 7f9bbbfa0700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb40ccdc0 con 0x7f9bb40cfa00 2024-04-06T12:30:21.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.032+0000 7f9bbbfa0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb40d0b50 con 0x7f9bb40d11b0 2024-04-06T12:30:21.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.032+0000 7f9bb9d3c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9bb40d11b0 0x7f9bb40d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.032+0000 7f9bba53d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9bb40cfa00 0x7f9bb40cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.032+0000 7f9bb953b700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f9bb40cb870 0x7f9bb415c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:56508/0 (socket says 172.21.15.189:56508) 2024-04-06T12:30:21.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.032+0000 7f9bb953b700 1 -- 172.21.15.189:0/3269112245 learned_addr learned my addr 172.21.15.189:0/3269112245 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:21.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.033+0000 7f9bb9d3c700 1 -- 172.21.15.189:0/3269112245 >> v2:172.21.15.192:3300/0 conn(0x7f9bb40cfa00 msgr2=0x7f9bb40cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:21.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.033+0000 7f9bb9d3c700 1 --2- 172.21.15.189:0/3269112245 >> v2:172.21.15.192:3300/0 conn(0x7f9bb40cfa00 0x7f9bb40cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.033+0000 7f9bb9d3c700 1 -- 172.21.15.189:0/3269112245 >> v1:172.21.15.192:6789/0 conn(0x7f9bb40cb870 legacy=0x7f9bb415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:21.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.033+0000 7f9bb9d3c700 1 -- 172.21.15.189:0/3269112245 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb40cbd80 con 0x7f9bb40d11b0 2024-04-06T12:30:21.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.033+0000 7f9bba53d700 1 --2- 172.21.15.189:0/3269112245 >> v2:172.21.15.192:3300/0 conn(0x7f9bb40cfa00 0x7f9bb40cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:30:21.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.033+0000 7f9bb9d3c700 1 --2- 172.21.15.189:0/3269112245 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9bb40d11b0 0x7f9bb40d1590 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0009840 tx=0x7f9bb0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7647594ab38a980 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:21.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.033+0000 7f9bb8d3a700 1 -- 172.21.15.189:0/3269112245 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb0011830 con 0x7f9bb40d11b0 2024-04-06T12:30:21.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.033+0000 7f9bb8d3a700 1 -- 172.21.15.189:0/3269112245 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb0011990 con 0x7f9bb40d11b0 2024-04-06T12:30:21.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.033+0000 7f9bb8d3a700 1 -- 172.21.15.189:0/3269112245 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb0011d20 con 0x7f9bb40d11b0 2024-04-06T12:30:21.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.033+0000 7f9bbbfa0700 1 -- 172.21.15.189:0/3269112245 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9bb40d11b0 msgr2=0x7f9bb40d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:21.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.033+0000 7f9bbbfa0700 1 --2- 172.21.15.189:0/3269112245 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9bb40d11b0 0x7f9bb40d1590 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0009840 tx=0x7f9bb0009b50 comp rx=0 tx=0).stop 2024-04-06T12:30:21.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.033+0000 7f9bbbfa0700 1 -- 172.21.15.189:0/3269112245 shutdown_connections 2024-04-06T12:30:21.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.033+0000 7f9bbbfa0700 1 --2- 172.21.15.189:0/3269112245 >> v2:172.21.15.192:3300/0 conn(0x7f9bb40cfa00 0x7f9bb40cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.033+0000 7f9bbbfa0700 1 --2- 172.21.15.189:0/3269112245 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9bb40d11b0 0x7f9bb40d1590 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.033+0000 7f9bbbfa0700 1 -- 172.21.15.189:0/3269112245 >> 172.21.15.189:0/3269112245 conn(0x7f9bb40b2db0 msgr2=0x7f9bb40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:21.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.034+0000 7f9bbbfa0700 1 -- 172.21.15.189:0/3269112245 shutdown_connections 2024-04-06T12:30:21.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.034+0000 7f9bbbfa0700 1 -- 172.21.15.189:0/3269112245 wait complete. 2024-04-06T12:30:21.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.035+0000 7f9bbbfa0700 1 Processor -- start 2024-04-06T12:30:21.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.035+0000 7f9bbbfa0700 1 -- start start 2024-04-06T12:30:21.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.035+0000 7f9bbbfa0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9bb40cb870 0x7f9bb4154cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.035+0000 7f9bbbfa0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9bb40cfa00 0x7f9bb40c9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.035+0000 7f9bbbfa0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bb40d11b0 0x7f9bb40ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.035+0000 7f9bbbfa0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb41551f0 con 0x7f9bb40d11b0 2024-04-06T12:30:21.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.035+0000 7f9bbbfa0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb40cb1b0 con 0x7f9bb40cfa00 2024-04-06T12:30:21.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.036+0000 7f9bbbfa0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb4159030 con 0x7f9bb40cb870 2024-04-06T12:30:21.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.036+0000 7f9bba53d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bb40d11b0 0x7f9bb40ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.036+0000 7f9bba53d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bb40d11b0 0x7f9bb40ca080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40718/0 (socket says 172.21.15.189:40718) 2024-04-06T12:30:21.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.036+0000 7f9bba53d700 1 -- 172.21.15.189:0/1853911603 learned_addr learned my addr 172.21.15.189:0/1853911603 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:21.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.036+0000 7f9bb953b700 1 --2- 172.21.15.189:0/1853911603 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9bb40cfa00 0x7f9bb40c9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.036+0000 7f9bb9d3c700 1 --2- 172.21.15.189:0/1853911603 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9bb40cb870 0x7f9bb4154cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.036+0000 7f9bba53d700 1 -- 172.21.15.189:0/1853911603 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9bb40cb870 msgr2=0x7f9bb4154cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:21.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.036+0000 7f9bba53d700 1 --2- 172.21.15.189:0/1853911603 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9bb40cb870 0x7f9bb4154cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.036+0000 7f9bba53d700 1 -- 172.21.15.189:0/1853911603 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9bb40cfa00 msgr2=0x7f9bb40c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:21.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.036+0000 7f9bba53d700 1 --2- 172.21.15.189:0/1853911603 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9bb40cfa00 0x7f9bb40c9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.036+0000 7f9bba53d700 1 -- 172.21.15.189:0/1853911603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac009a70 con 0x7f9bb40d11b0 2024-04-06T12:30:21.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.036+0000 7f9bba53d700 1 --2- 172.21.15.189:0/1853911603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bb40d11b0 0x7f9bb40ca080 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f9bac00efc0 tx=0x7f9bac00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:21.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.036+0000 7f9baa7fc700 1 -- 172.21.15.189:0/1853911603 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bac00ca90 con 0x7f9bb40d11b0 2024-04-06T12:30:21.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.036+0000 7f9baa7fc700 1 -- 172.21.15.189:0/1853911603 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bac00f040 con 0x7f9bb40d11b0 2024-04-06T12:30:21.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.036+0000 7f9bbbfa0700 1 -- 172.21.15.189:0/1853911603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb00094a0 con 0x7f9bb40d11b0 2024-04-06T12:30:21.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.036+0000 7f9baa7fc700 1 -- 172.21.15.189:0/1853911603 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bac00ce10 con 0x7f9bb40d11b0 2024-04-06T12:30:21.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.036+0000 7f9bbbfa0700 1 -- 172.21.15.189:0/1853911603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb40ca880 con 0x7f9bb40d11b0 2024-04-06T12:30:21.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.037+0000 7f9bbbfa0700 1 -- 172.21.15.189:0/1853911603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb4074990 con 0x7f9bb40d11b0 2024-04-06T12:30:21.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.038+0000 7f9baa7fc700 1 -- 172.21.15.189:0/1853911603 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f9bac014480 con 0x7f9bb40d11b0 2024-04-06T12:30:21.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.038+0000 7f9baa7fc700 1 --2- 172.21.15.189:0/1853911603 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b9806adb0 0x7f9b9806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.038+0000 7f9baa7fc700 1 -- 172.21.15.189:0/1853911603 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9bac089e20 con 0x7f9bb40d11b0 2024-04-06T12:30:21.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.038+0000 7f9bb9d3c700 1 --2- 172.21.15.189:0/1853911603 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b9806adb0 0x7f9b9806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.039+0000 7f9bb9d3c700 1 --2- 172.21.15.189:0/1853911603 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b9806adb0 0x7f9b9806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0000c00 tx=0x7f9bb0008030 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:21.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.043+0000 7f9baa7fc700 1 -- 172.21.15.189:0/1853911603 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9bac054c20 con 0x7f9bb40d11b0 2024-04-06T12:30:21.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.232+0000 7f9bbbfa0700 1 -- 172.21.15.189:0/1853911603 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9bb40cf4e0 con 0x7f9b9806adb0 2024-04-06T12:30:21.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.240+0000 7f9baa7fc700 1 -- 172.21.15.189:0/1853911603 <== mgr.7875 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+285980 (secure 0 0 0) 0x7f9bb0011c00 con 0x7f9b9806adb0 2024-04-06T12:30:21.241 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:30:21.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.243+0000 7f9bbbfa0700 1 -- 172.21.15.189:0/1853911603 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b9806adb0 msgr2=0x7f9b9806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:21.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.243+0000 7f9bbbfa0700 1 --2- 172.21.15.189:0/1853911603 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b9806adb0 0x7f9b9806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0000c00 tx=0x7f9bb0008030 comp rx=0 tx=0).stop 2024-04-06T12:30:21.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.243+0000 7f9bbbfa0700 1 -- 172.21.15.189:0/1853911603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bb40d11b0 msgr2=0x7f9bb40ca080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:21.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.243+0000 7f9bbbfa0700 1 --2- 172.21.15.189:0/1853911603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bb40d11b0 0x7f9bb40ca080 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f9bac00efc0 tx=0x7f9bac00c3a0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.243+0000 7f9bbbfa0700 1 -- 172.21.15.189:0/1853911603 shutdown_connections 2024-04-06T12:30:21.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.243+0000 7f9bbbfa0700 1 --2- 172.21.15.189:0/1853911603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bb40d11b0 0x7f9bb40ca080 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.243+0000 7f9bbbfa0700 1 --2- 172.21.15.189:0/1853911603 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9bb40cb870 0x7f9bb4154cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.243+0000 7f9bbbfa0700 1 --2- 172.21.15.189:0/1853911603 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9b9806adb0 0x7f9b9806d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.243+0000 7f9bbbfa0700 1 --2- 172.21.15.189:0/1853911603 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9bb40cfa00 0x7f9bb40c9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.243+0000 7f9bbbfa0700 1 -- 172.21.15.189:0/1853911603 >> 172.21.15.189:0/1853911603 conn(0x7f9bb40b2db0 msgr2=0x7f9bb415b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:21.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.243+0000 7f9bbbfa0700 1 -- 172.21.15.189:0/1853911603 shutdown_connections 2024-04-06T12:30:21.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.243+0000 7f9bbbfa0700 1 -- 172.21.15.189:0/1853911603 wait complete. 2024-04-06T12:30:21.244 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:30:21.252 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-06T12:30:19.464511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590562,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":352,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":232,"ondisk_log_size":232,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":328848,"kb_used_data":13432,"kb_used_omap":12,"kb_used_meta":315251,"kb_avail":754645872,"statfs":{"total":773094113280,"available":772757372928,"internally_reserved":0,"allocated":13754368,"data_stored":7447437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":322817592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":194,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000694"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434489+0000","last_change":"2024-04-06T12:30:17.150881+0000","last_active":"2024-04-06T12:30:18.434489+0000","last_peered":"2024-04-06T12:30:18.434489+0000","last_clean":"2024-04-06T12:30:18.434489+0000","last_became_active":"2024-04-06T12:30:17.149992+0000","last_became_peered":"2024-04-06T12:30:17.149992+0000","last_unstale":"2024-04-06T12:30:18.434489+0000","last_undegraded":"2024-04-06T12:30:18.434489+0000","last_fullsized":"2024-04-06T12:30:18.434489+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:21:43.683412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:16.716721+0000","last_change":"2024-04-06T12:30:16.716721+0000","last_active":"2024-04-06T12:30:16.716721+0000","last_peered":"2024-04-06T12:30:16.716721+0000","last_clean":"2024-04-06T12:30:16.716721+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:16.716721+0000","last_undegraded":"2024-04-06T12:30:16.716721+0000","last_fullsized":"2024-04-06T12:30:16.716721+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434601+0000","last_change":"2024-04-06T12:30:17.154869+0000","last_active":"2024-04-06T12:30:18.434601+0000","last_peered":"2024-04-06T12:30:18.434601+0000","last_clean":"2024-04-06T12:30:18.434601+0000","last_became_active":"2024-04-06T12:30:17.154616+0000","last_became_peered":"2024-04-06T12:30:17.154616+0000","last_unstale":"2024-04-06T12:30:18.434601+0000","last_undegraded":"2024-04-06T12:30:18.434601+0000","last_fullsized":"2024-04-06T12:30:18.434601+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:14:49.657999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:16.716721+0000","last_change":"2024-04-06T12:30:16.716721+0000","last_active":"2024-04-06T12:30:16.716721+0000","last_peered":"2024-04-06T12:30:16.716721+0000","last_clean":"2024-04-06T12:30:16.716721+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:16.716721+0000","last_undegraded":"2024-04-06T12:30:16.716721+0000","last_fullsized":"2024-04-06T12:30:16.716721+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.161440+0000","last_change":"2024-04-06T12:30:17.142421+0000","last_active":"2024-04-06T12:30:17.161440+0000","last_peered":"2024-04-06T12:30:17.161440+0000","last_clean":"2024-04-06T12:30:17.161440+0000","last_became_active":"2024-04-06T12:30:17.142166+0000","last_became_peered":"2024-04-06T12:30:17.142166+0000","last_unstale":"2024-04-06T12:30:17.161440+0000","last_undegraded":"2024-04-06T12:30:17.161440+0000","last_fullsized":"2024-04-06T12:30:17.161440+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:49:47.788561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:16.716721+0000","last_change":"2024-04-06T12:30:16.716721+0000","last_active":"2024-04-06T12:30:16.716721+0000","last_peered":"2024-04-06T12:30:16.716721+0000","last_clean":"2024-04-06T12:30:16.716721+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:16.716721+0000","last_undegraded":"2024-04-06T12:30:16.716721+0000","last_fullsized":"2024-04-06T12:30:16.716721+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:16.716721+0000","last_change":"2024-04-06T12:30:16.716721+0000","last_active":"2024-04-06T12:30:16.716721+0000","last_peered":"2024-04-06T12:30:16.716721+0000","last_clean":"2024-04-06T12:30:16.716721+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:16.716721+0000","last_undegraded":"2024-04-06T12:30:16.716721+0000","last_fullsized":"2024-04-06T12:30:16.716721+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371725+0000","last_change":"2024-04-06T12:30:17.161842+0000","last_active":"2024-04-06T12:30:19.371725+0000","last_peered":"2024-04-06T12:30:19.371725+0000","last_clean":"2024-04-06T12:30:19.371725+0000","last_became_active":"2024-04-06T12:30:17.161572+0000","last_became_peered":"2024-04-06T12:30:17.161572+0000","last_unstale":"2024-04-06T12:30:19.371725+0000","last_undegraded":"2024-04-06T12:30:19.371725+0000","last_fullsized":"2024-04-06T12:30:19.371725+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:29:16.578210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:16.716721+0000","last_change":"2024-04-06T12:30:16.716721+0000","last_active":"2024-04-06T12:30:16.716721+0000","last_peered":"2024-04-06T12:30:16.716721+0000","last_clean":"2024-04-06T12:30:16.716721+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:16.716721+0000","last_undegraded":"2024-04-06T12:30:16.716721+0000","last_fullsized":"2024-04-06T12:30:16.716721+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.161296+0000","last_change":"2024-04-06T12:30:17.150373+0000","last_active":"2024-04-06T12:30:17.161296+0000","last_peered":"2024-04-06T12:30:17.161296+0000","last_clean":"2024-04-06T12:30:17.161296+0000","last_became_active":"2024-04-06T12:30:17.150122+0000","last_became_peered":"2024-04-06T12:30:17.150122+0000","last_unstale":"2024-04-06T12:30:17.161296+0000","last_undegraded":"2024-04-06T12:30:17.161296+0000","last_fullsized":"2024-04-06T12:30:17.161296+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:21:32.239013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.155505+0000","last_change":"2024-04-06T12:30:17.151989+0000","last_active":"2024-04-06T12:30:17.155505+0000","last_peered":"2024-04-06T12:30:17.155505+0000","last_clean":"2024-04-06T12:30:17.155505+0000","last_became_active":"2024-04-06T12:30:17.151739+0000","last_became_peered":"2024-04-06T12:30:17.151739+0000","last_unstale":"2024-04-06T12:30:17.155505+0000","last_undegraded":"2024-04-06T12:30:17.155505+0000","last_fullsized":"2024-04-06T12:30:17.155505+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:11:54.910628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136116+0000","last_change":"2024-04-06T12:30:17.149805+0000","last_active":"2024-04-06T12:30:18.136116+0000","last_peered":"2024-04-06T12:30:18.136116+0000","last_clean":"2024-04-06T12:30:18.136116+0000","last_became_active":"2024-04-06T12:30:17.149520+0000","last_became_peered":"2024-04-06T12:30:17.149520+0000","last_unstale":"2024-04-06T12:30:18.136116+0000","last_undegraded":"2024-04-06T12:30:18.136116+0000","last_fullsized":"2024-04-06T12:30:18.136116+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:49:47.788561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:16.716721+0000","last_change":"2024-04-06T12:30:16.716721+0000","last_active":"2024-04-06T12:30:16.716721+0000","last_peered":"2024-04-06T12:30:16.716721+0000","last_clean":"2024-04-06T12:30:16.716721+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:16.716721+0000","last_undegraded":"2024-04-06T12:30:16.716721+0000","last_fullsized":"2024-04-06T12:30:16.716721+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.140775+0000","last_change":"2024-04-06T12:30:17.146818+0000","last_active":"2024-04-06T12:30:18.140775+0000","last_peered":"2024-04-06T12:30:18.140775+0000","last_clean":"2024-04-06T12:30:18.140775+0000","last_became_active":"2024-04-06T12:30:17.146541+0000","last_became_peered":"2024-04-06T12:30:17.146541+0000","last_unstale":"2024-04-06T12:30:18.140775+0000","last_undegraded":"2024-04-06T12:30:18.140775+0000","last_fullsized":"2024-04-06T12:30:18.140775+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:42:07.595684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.140671+0000","last_change":"2024-04-06T12:30:17.140598+0000","last_active":"2024-04-06T12:30:18.140671+0000","last_peered":"2024-04-06T12:30:18.140671+0000","last_clean":"2024-04-06T12:30:18.140671+0000","last_became_active":"2024-04-06T12:30:17.140322+0000","last_became_peered":"2024-04-06T12:30:17.140322+0000","last_unstale":"2024-04-06T12:30:18.140671+0000","last_undegraded":"2024-04-06T12:30:18.140671+0000","last_fullsized":"2024-04-06T12:30:18.140671+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:40:59.167127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.140648+0000","last_change":"2024-04-06T12:30:17.147645+0000","last_active":"2024-04-06T12:30:18.140648+0000","last_peered":"2024-04-06T12:30:18.140648+0000","last_clean":"2024-04-06T12:30:18.140648+0000","last_became_active":"2024-04-06T12:30:17.147389+0000","last_became_peered":"2024-04-06T12:30:17.147389+0000","last_unstale":"2024-04-06T12:30:18.140648+0000","last_undegraded":"2024-04-06T12:30:18.140648+0000","last_fullsized":"2024-04-06T12:30:18.140648+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:13:11.187491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434608+0000","last_change":"2024-04-06T12:30:17.155373+0000","last_active":"2024-04-06T12:30:18.434608+0000","last_peered":"2024-04-06T12:30:18.434608+0000","last_clean":"2024-04-06T12:30:18.434608+0000","last_became_active":"2024-04-06T12:30:17.150651+0000","last_became_peered":"2024-04-06T12:30:17.150651+0000","last_unstale":"2024-04-06T12:30:18.434608+0000","last_undegraded":"2024-04-06T12:30:18.434608+0000","last_fullsized":"2024-04-06T12:30:18.434608+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:02:38.211510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:16.716721+0000","last_change":"2024-04-06T12:30:16.716721+0000","last_active":"2024-04-06T12:30:16.716721+0000","last_peered":"2024-04-06T12:30:16.716721+0000","last_clean":"2024-04-06T12:30:16.716721+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:16.716721+0000","last_undegraded":"2024-04-06T12:30:16.716721+0000","last_fullsized":"2024-04-06T12:30:16.716721+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434495+0000","last_change":"2024-04-06T12:30:17.157461+0000","last_active":"2024-04-06T12:30:18.434495+0000","last_peered":"2024-04-06T12:30:18.434495+0000","last_clean":"2024-04-06T12:30:18.434495+0000","last_became_active":"2024-04-06T12:30:17.155996+0000","last_became_peered":"2024-04-06T12:30:17.155996+0000","last_unstale":"2024-04-06T12:30:18.434495+0000","last_undegraded":"2024-04-06T12:30:18.434495+0000","last_fullsized":"2024-04-06T12:30:18.434495+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:56:33.370573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136216+0000","last_change":"2024-04-06T12:30:17.144941+0000","last_active":"2024-04-06T12:30:18.136216+0000","last_peered":"2024-04-06T12:30:18.136216+0000","last_clean":"2024-04-06T12:30:18.136216+0000","last_became_active":"2024-04-06T12:30:17.144669+0000","last_became_peered":"2024-04-06T12:30:17.144669+0000","last_unstale":"2024-04-06T12:30:18.136216+0000","last_undegraded":"2024-04-06T12:30:18.136216+0000","last_fullsized":"2024-04-06T12:30:18.136216+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:13:11.187491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.372078+0000","last_change":"2024-04-06T12:30:17.149388+0000","last_active":"2024-04-06T12:30:19.372078+0000","last_peered":"2024-04-06T12:30:19.372078+0000","last_clean":"2024-04-06T12:30:19.372078+0000","last_became_active":"2024-04-06T12:30:17.149094+0000","last_became_peered":"2024-04-06T12:30:17.149094+0000","last_unstale":"2024-04-06T12:30:19.372078+0000","last_undegraded":"2024-04-06T12:30:19.372078+0000","last_fullsized":"2024-04-06T12:30:19.372078+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:49.479601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.160924+0000","last_change":"2024-04-06T12:30:17.140583+0000","last_active":"2024-04-06T12:30:17.160924+0000","last_peered":"2024-04-06T12:30:17.160924+0000","last_clean":"2024-04-06T12:30:17.160924+0000","last_became_active":"2024-04-06T12:30:17.140314+0000","last_became_peered":"2024-04-06T12:30:17.140314+0000","last_unstale":"2024-04-06T12:30:17.160924+0000","last_undegraded":"2024-04-06T12:30:17.160924+0000","last_fullsized":"2024-04-06T12:30:17.160924+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:13:15.814692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434421+0000","last_change":"2024-04-06T12:30:17.139552+0000","last_active":"2024-04-06T12:30:18.434421+0000","last_peered":"2024-04-06T12:30:18.434421+0000","last_clean":"2024-04-06T12:30:18.434421+0000","last_became_active":"2024-04-06T12:30:17.139264+0000","last_became_peered":"2024-04-06T12:30:17.139264+0000","last_unstale":"2024-04-06T12:30:18.434421+0000","last_undegraded":"2024-04-06T12:30:18.434421+0000","last_fullsized":"2024-04-06T12:30:18.434421+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:22:06.600431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.161007+0000","last_change":"2024-04-06T12:30:17.147290+0000","last_active":"2024-04-06T12:30:17.161007+0000","last_peered":"2024-04-06T12:30:17.161007+0000","last_clean":"2024-04-06T12:30:17.161007+0000","last_became_active":"2024-04-06T12:30:17.147009+0000","last_became_peered":"2024-04-06T12:30:17.147009+0000","last_unstale":"2024-04-06T12:30:17.161007+0000","last_undegraded":"2024-04-06T12:30:17.161007+0000","last_fullsized":"2024-04-06T12:30:17.161007+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:25:08.630322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136007+0000","last_change":"2024-04-06T12:30:17.144462+0000","last_active":"2024-04-06T12:30:18.136007+0000","last_peered":"2024-04-06T12:30:18.136007+0000","last_clean":"2024-04-06T12:30:18.136007+0000","last_became_active":"2024-04-06T12:30:17.144166+0000","last_became_peered":"2024-04-06T12:30:17.144166+0000","last_unstale":"2024-04-06T12:30:18.136007+0000","last_undegraded":"2024-04-06T12:30:18.136007+0000","last_fullsized":"2024-04-06T12:30:18.136007+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:28:04.728040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.137772+0000","last_change":"2024-04-06T12:30:17.154176+0000","last_active":"2024-04-06T12:30:18.137772+0000","last_peered":"2024-04-06T12:30:18.137772+0000","last_clean":"2024-04-06T12:30:18.137772+0000","last_became_active":"2024-04-06T12:30:17.153918+0000","last_became_peered":"2024-04-06T12:30:17.153918+0000","last_unstale":"2024-04-06T12:30:18.137772+0000","last_undegraded":"2024-04-06T12:30:18.137772+0000","last_fullsized":"2024-04-06T12:30:18.137772+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:44:50.175971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.155713+0000","last_change":"2024-04-06T12:30:17.152888+0000","last_active":"2024-04-06T12:30:17.155713+0000","last_peered":"2024-04-06T12:30:17.155713+0000","last_clean":"2024-04-06T12:30:17.155713+0000","last_became_active":"2024-04-06T12:30:17.152616+0000","last_became_peered":"2024-04-06T12:30:17.152616+0000","last_unstale":"2024-04-06T12:30:17.155713+0000","last_undegraded":"2024-04-06T12:30:17.155713+0000","last_fullsized":"2024-04-06T12:30:17.155713+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:29:16.578210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:16.716721+0000","last_change":"2024-04-06T12:30:16.716721+0000","last_active":"2024-04-06T12:30:16.716721+0000","last_peered":"2024-04-06T12:30:16.716721+0000","last_clean":"2024-04-06T12:30:16.716721+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:16.716721+0000","last_undegraded":"2024-04-06T12:30:16.716721+0000","last_fullsized":"2024-04-06T12:30:16.716721+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.372001+0000","last_change":"2024-04-06T12:30:17.150886+0000","last_active":"2024-04-06T12:30:19.372001+0000","last_peered":"2024-04-06T12:30:19.372001+0000","last_clean":"2024-04-06T12:30:19.372001+0000","last_became_active":"2024-04-06T12:30:17.150616+0000","last_became_peered":"2024-04-06T12:30:17.150616+0000","last_unstale":"2024-04-06T12:30:19.372001+0000","last_undegraded":"2024-04-06T12:30:19.372001+0000","last_fullsized":"2024-04-06T12:30:19.372001+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:29:46.115525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.135858+0000","last_change":"2024-04-06T12:30:17.147614+0000","last_active":"2024-04-06T12:30:18.135858+0000","last_peered":"2024-04-06T12:30:18.135858+0000","last_clean":"2024-04-06T12:30:18.135858+0000","last_became_active":"2024-04-06T12:30:17.147313+0000","last_became_peered":"2024-04-06T12:30:17.147313+0000","last_unstale":"2024-04-06T12:30:18.135858+0000","last_undegraded":"2024-04-06T12:30:18.135858+0000","last_fullsized":"2024-04-06T12:30:18.135858+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:12:50.047089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:16.716721+0000","last_change":"2024-04-06T12:30:16.716721+0000","last_active":"2024-04-06T12:30:16.716721+0000","last_peered":"2024-04-06T12:30:16.716721+0000","last_clean":"2024-04-06T12:30:16.716721+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:16.716721+0000","last_undegraded":"2024-04-06T12:30:16.716721+0000","last_fullsized":"2024-04-06T12:30:16.716721+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371721+0000","last_change":"2024-04-06T12:30:17.153878+0000","last_active":"2024-04-06T12:30:19.371721+0000","last_peered":"2024-04-06T12:30:19.371721+0000","last_clean":"2024-04-06T12:30:19.371721+0000","last_became_active":"2024-04-06T12:30:17.153627+0000","last_became_peered":"2024-04-06T12:30:17.153627+0000","last_unstale":"2024-04-06T12:30:19.371721+0000","last_undegraded":"2024-04-06T12:30:19.371721+0000","last_fullsized":"2024-04-06T12:30:19.371721+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:03:40.459030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434601+0000","last_change":"2024-04-06T12:30:17.157737+0000","last_active":"2024-04-06T12:30:18.434601+0000","last_peered":"2024-04-06T12:30:18.434601+0000","last_clean":"2024-04-06T12:30:18.434601+0000","last_became_active":"2024-04-06T12:30:17.155307+0000","last_became_peered":"2024-04-06T12:30:17.155307+0000","last_unstale":"2024-04-06T12:30:18.434601+0000","last_undegraded":"2024-04-06T12:30:18.434601+0000","last_fullsized":"2024-04-06T12:30:18.434601+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:43:58.535609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:15.709785+0000","last_change":"2024-04-06T12:30:15.709785+0000","last_active":"2024-04-06T12:30:15.709785+0000","last_peered":"2024-04-06T12:30:15.709785+0000","last_clean":"2024-04-06T12:30:15.709785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:15.709785+0000","last_undegraded":"2024-04-06T12:30:15.709785+0000","last_fullsized":"2024-04-06T12:30:15.709785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371778+0000","last_change":"2024-04-06T12:30:17.162532+0000","last_active":"2024-04-06T12:30:19.371778+0000","last_peered":"2024-04-06T12:30:19.371778+0000","last_clean":"2024-04-06T12:30:19.371778+0000","last_became_active":"2024-04-06T12:30:17.162281+0000","last_became_peered":"2024-04-06T12:30:17.162281+0000","last_unstale":"2024-04-06T12:30:19.371778+0000","last_undegraded":"2024-04-06T12:30:19.371778+0000","last_fullsized":"2024-04-06T12:30:19.371778+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:30:54.593855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"127'1","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.372098+0000","last_change":"2024-04-06T12:30:16.729681+0000","last_active":"2024-04-06T12:30:19.372098+0000","last_peered":"2024-04-06T12:30:19.372098+0000","last_clean":"2024-04-06T12:30:19.372098+0000","last_became_active":"2024-04-06T12:30:16.729420+0000","last_became_peered":"2024-04-06T12:30:16.729420+0000","last_unstale":"2024-04-06T12:30:19.372098+0000","last_undegraded":"2024-04-06T12:30:19.372098+0000","last_fullsized":"2024-04-06T12:30:19.372098+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:08:20.306799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136005+0000","last_change":"2024-04-06T12:30:17.157150+0000","last_active":"2024-04-06T12:30:18.136005+0000","last_peered":"2024-04-06T12:30:18.136005+0000","last_clean":"2024-04-06T12:30:18.136005+0000","last_became_active":"2024-04-06T12:30:17.156889+0000","last_became_peered":"2024-04-06T12:30:17.156889+0000","last_unstale":"2024-04-06T12:30:18.136005+0000","last_undegraded":"2024-04-06T12:30:18.136005+0000","last_fullsized":"2024-04-06T12:30:18.136005+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:09:21.777123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"127'2","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136241+0000","last_change":"2024-04-06T12:30:16.738561+0000","last_active":"2024-04-06T12:30:18.136241+0000","last_peered":"2024-04-06T12:30:18.136241+0000","last_clean":"2024-04-06T12:30:18.136241+0000","last_became_active":"2024-04-06T12:30:16.738213+0000","last_became_peered":"2024-04-06T12:30:16.738213+0000","last_unstale":"2024-04-06T12:30:18.136241+0000","last_undegraded":"2024-04-06T12:30:18.136241+0000","last_fullsized":"2024-04-06T12:30:18.136241+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:34:14.692966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.143840+0000","last_change":"2024-04-06T12:30:17.142648+0000","last_active":"2024-04-06T12:30:17.143840+0000","last_peered":"2024-04-06T12:30:17.143840+0000","last_clean":"2024-04-06T12:30:17.143840+0000","last_became_active":"2024-04-06T12:30:17.142100+0000","last_became_peered":"2024-04-06T12:30:17.142100+0000","last_unstale":"2024-04-06T12:30:17.143840+0000","last_undegraded":"2024-04-06T12:30:17.143840+0000","last_fullsized":"2024-04-06T12:30:17.143840+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:03:40.459030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"127'2","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.154167+0000","last_change":"2024-04-06T12:30:16.739257+0000","last_active":"2024-04-06T12:30:17.154167+0000","last_peered":"2024-04-06T12:30:17.154167+0000","last_clean":"2024-04-06T12:30:17.154167+0000","last_became_active":"2024-04-06T12:30:16.738995+0000","last_became_peered":"2024-04-06T12:30:16.738995+0000","last_unstale":"2024-04-06T12:30:17.154167+0000","last_undegraded":"2024-04-06T12:30:17.154167+0000","last_fullsized":"2024-04-06T12:30:17.154167+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:34:14.692966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:16.716721+0000","last_change":"2024-04-06T12:30:16.716721+0000","last_active":"2024-04-06T12:30:16.716721+0000","last_peered":"2024-04-06T12:30:16.716721+0000","last_clean":"2024-04-06T12:30:16.716721+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:16.716721+0000","last_undegraded":"2024-04-06T12:30:16.716721+0000","last_fullsized":"2024-04-06T12:30:16.716721+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1b","version":"127'1","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.140804+0000","last_change":"2024-04-06T12:30:16.737577+0000","last_active":"2024-04-06T12:30:18.140804+0000","last_peered":"2024-04-06T12:30:18.140804+0000","last_clean":"2024-04-06T12:30:18.140804+0000","last_became_active":"2024-04-06T12:30:16.737305+0000","last_became_peered":"2024-04-06T12:30:16.737305+0000","last_unstale":"2024-04-06T12:30:18.140804+0000","last_undegraded":"2024-04-06T12:30:18.140804+0000","last_fullsized":"2024-04-06T12:30:18.140804+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:32:52.686846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434615+0000","last_change":"2024-04-06T12:30:17.156240+0000","last_active":"2024-04-06T12:30:18.434615+0000","last_peered":"2024-04-06T12:30:18.434615+0000","last_clean":"2024-04-06T12:30:18.434615+0000","last_became_active":"2024-04-06T12:30:17.155684+0000","last_became_peered":"2024-04-06T12:30:17.155684+0000","last_unstale":"2024-04-06T12:30:18.434615+0000","last_undegraded":"2024-04-06T12:30:18.434615+0000","last_fullsized":"2024-04-06T12:30:18.434615+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:54:02.871601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"127'1","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.153013+0000","last_change":"2024-04-06T12:30:16.740797+0000","last_active":"2024-04-06T12:30:17.153013+0000","last_peered":"2024-04-06T12:30:17.153013+0000","last_clean":"2024-04-06T12:30:17.153013+0000","last_became_active":"2024-04-06T12:30:16.740512+0000","last_became_peered":"2024-04-06T12:30:16.740512+0000","last_unstale":"2024-04-06T12:30:17.153013+0000","last_undegraded":"2024-04-06T12:30:17.153013+0000","last_fullsized":"2024-04-06T12:30:17.153013+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:33:52.101095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:16.716721+0000","last_change":"2024-04-06T12:30:16.716721+0000","last_active":"2024-04-06T12:30:16.716721+0000","last_peered":"2024-04-06T12:30:16.716721+0000","last_clean":"2024-04-06T12:30:16.716721+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:16.716721+0000","last_undegraded":"2024-04-06T12:30:16.716721+0000","last_fullsized":"2024-04-06T12:30:16.716721+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136040+0000","last_change":"2024-04-06T12:30:16.737571+0000","last_active":"2024-04-06T12:30:18.136040+0000","last_peered":"2024-04-06T12:30:18.136040+0000","last_clean":"2024-04-06T12:30:18.136040+0000","last_became_active":"2024-04-06T12:30:16.737232+0000","last_became_peered":"2024-04-06T12:30:16.737232+0000","last_unstale":"2024-04-06T12:30:18.136040+0000","last_undegraded":"2024-04-06T12:30:18.136040+0000","last_fullsized":"2024-04-06T12:30:18.136040+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:32:52.686846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:16.716721+0000","last_change":"2024-04-06T12:30:16.716721+0000","last_active":"2024-04-06T12:30:16.716721+0000","last_peered":"2024-04-06T12:30:16.716721+0000","last_clean":"2024-04-06T12:30:16.716721+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:16.716721+0000","last_undegraded":"2024-04-06T12:30:16.716721+0000","last_fullsized":"2024-04-06T12:30:16.716721+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:15.709785+0000","last_change":"2024-04-06T12:30:15.709785+0000","last_active":"2024-04-06T12:30:15.709785+0000","last_peered":"2024-04-06T12:30:15.709785+0000","last_clean":"2024-04-06T12:30:15.709785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:15.709785+0000","last_undegraded":"2024-04-06T12:30:15.709785+0000","last_fullsized":"2024-04-06T12:30:15.709785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434498+0000","last_change":"2024-04-06T12:30:17.157952+0000","last_active":"2024-04-06T12:30:18.434498+0000","last_peered":"2024-04-06T12:30:18.434498+0000","last_clean":"2024-04-06T12:30:18.434498+0000","last_became_active":"2024-04-06T12:30:17.157452+0000","last_became_peered":"2024-04-06T12:30:17.157452+0000","last_unstale":"2024-04-06T12:30:18.434498+0000","last_undegraded":"2024-04-06T12:30:18.434498+0000","last_fullsized":"2024-04-06T12:30:18.434498+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:17:50.436516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136112+0000","last_change":"2024-04-06T12:30:16.738077+0000","last_active":"2024-04-06T12:30:18.136112+0000","last_peered":"2024-04-06T12:30:18.136112+0000","last_clean":"2024-04-06T12:30:18.136112+0000","last_became_active":"2024-04-06T12:30:16.737826+0000","last_became_peered":"2024-04-06T12:30:16.737826+0000","last_unstale":"2024-04-06T12:30:18.136112+0000","last_undegraded":"2024-04-06T12:30:18.136112+0000","last_fullsized":"2024-04-06T12:30:18.136112+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:08:44.968098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.161269+0000","last_change":"2024-04-06T12:30:17.151460+0000","last_active":"2024-04-06T12:30:17.161269+0000","last_peered":"2024-04-06T12:30:17.161269+0000","last_clean":"2024-04-06T12:30:17.161269+0000","last_became_active":"2024-04-06T12:30:17.151231+0000","last_became_peered":"2024-04-06T12:30:17.151231+0000","last_unstale":"2024-04-06T12:30:17.161269+0000","last_undegraded":"2024-04-06T12:30:17.161269+0000","last_fullsized":"2024-04-06T12:30:17.161269+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:44:50.175971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434415+0000","last_change":"2024-04-06T12:30:16.739301+0000","last_active":"2024-04-06T12:30:18.434415+0000","last_peered":"2024-04-06T12:30:18.434415+0000","last_clean":"2024-04-06T12:30:18.434415+0000","last_became_active":"2024-04-06T12:30:16.738988+0000","last_became_peered":"2024-04-06T12:30:16.738988+0000","last_unstale":"2024-04-06T12:30:18.434415+0000","last_undegraded":"2024-04-06T12:30:18.434415+0000","last_fullsized":"2024-04-06T12:30:18.434415+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:47:39.607805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371807+0000","last_change":"2024-04-06T12:30:17.162362+0000","last_active":"2024-04-06T12:30:19.371807+0000","last_peered":"2024-04-06T12:30:19.371807+0000","last_clean":"2024-04-06T12:30:19.371807+0000","last_became_active":"2024-04-06T12:30:17.162106+0000","last_became_peered":"2024-04-06T12:30:17.162106+0000","last_unstale":"2024-04-06T12:30:19.371807+0000","last_undegraded":"2024-04-06T12:30:19.371807+0000","last_fullsized":"2024-04-06T12:30:19.371807+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:02:03.887605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"127'3","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.140828+0000","last_change":"2024-04-06T12:30:16.740347+0000","last_active":"2024-04-06T12:30:18.140828+0000","last_peered":"2024-04-06T12:30:18.140828+0000","last_clean":"2024-04-06T12:30:18.140828+0000","last_became_active":"2024-04-06T12:30:16.740008+0000","last_became_peered":"2024-04-06T12:30:16.740008+0000","last_unstale":"2024-04-06T12:30:18.140828+0000","last_undegraded":"2024-04-06T12:30:18.140828+0000","last_fullsized":"2024-04-06T12:30:18.140828+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:42:07.052380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.161458+0000","last_change":"2024-04-06T12:30:17.151748+0000","last_active":"2024-04-06T12:30:17.161458+0000","last_peered":"2024-04-06T12:30:17.161458+0000","last_clean":"2024-04-06T12:30:17.161458+0000","last_became_active":"2024-04-06T12:30:17.151427+0000","last_became_peered":"2024-04-06T12:30:17.151427+0000","last_unstale":"2024-04-06T12:30:17.161458+0000","last_undegraded":"2024-04-06T12:30:17.161458+0000","last_fullsized":"2024-04-06T12:30:17.161458+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:09:21.777123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:15.709785+0000","last_change":"2024-04-06T12:30:15.709785+0000","last_active":"2024-04-06T12:30:15.709785+0000","last_peered":"2024-04-06T12:30:15.709785+0000","last_clean":"2024-04-06T12:30:15.709785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:15.709785+0000","last_undegraded":"2024-04-06T12:30:15.709785+0000","last_fullsized":"2024-04-06T12:30:15.709785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.154473+0000","last_change":"2024-04-06T12:30:17.146501+0000","last_active":"2024-04-06T12:30:17.154473+0000","last_peered":"2024-04-06T12:30:17.154473+0000","last_clean":"2024-04-06T12:30:17.154473+0000","last_became_active":"2024-04-06T12:30:17.146211+0000","last_became_peered":"2024-04-06T12:30:17.146211+0000","last_unstale":"2024-04-06T12:30:17.154473+0000","last_undegraded":"2024-04-06T12:30:17.154473+0000","last_fullsized":"2024-04-06T12:30:17.154473+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:52:01.384680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"127'2","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.161390+0000","last_change":"2024-04-06T12:30:16.738690+0000","last_active":"2024-04-06T12:30:17.161390+0000","last_peered":"2024-04-06T12:30:17.161390+0000","last_clean":"2024-04-06T12:30:17.161390+0000","last_became_active":"2024-04-06T12:30:16.738430+0000","last_became_peered":"2024-04-06T12:30:16.738430+0000","last_unstale":"2024-04-06T12:30:17.161390+0000","last_undegraded":"2024-04-06T12:30:17.161390+0000","last_fullsized":"2024-04-06T12:30:17.161390+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:52:00.377745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:16.716721+0000","last_change":"2024-04-06T12:30:16.716721+0000","last_active":"2024-04-06T12:30:16.716721+0000","last_peered":"2024-04-06T12:30:16.716721+0000","last_clean":"2024-04-06T12:30:16.716721+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:16.716721+0000","last_undegraded":"2024-04-06T12:30:16.716721+0000","last_fullsized":"2024-04-06T12:30:16.716721+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:15.709785+0000","last_change":"2024-04-06T12:30:15.709785+0000","last_active":"2024-04-06T12:30:15.709785+0000","last_peered":"2024-04-06T12:30:15.709785+0000","last_clean":"2024-04-06T12:30:15.709785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:15.709785+0000","last_undegraded":"2024-04-06T12:30:15.709785+0000","last_fullsized":"2024-04-06T12:30:15.709785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.372034+0000","last_change":"2024-04-06T12:30:17.153701+0000","last_active":"2024-04-06T12:30:19.372034+0000","last_peered":"2024-04-06T12:30:19.372034+0000","last_clean":"2024-04-06T12:30:19.372034+0000","last_became_active":"2024-04-06T12:30:17.153454+0000","last_became_peered":"2024-04-06T12:30:17.153454+0000","last_unstale":"2024-04-06T12:30:19.372034+0000","last_undegraded":"2024-04-06T12:30:19.372034+0000","last_fullsized":"2024-04-06T12:30:19.372034+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:17:26.384789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371825+0000","last_change":"2024-04-06T12:30:16.739584+0000","last_active":"2024-04-06T12:30:19.371825+0000","last_peered":"2024-04-06T12:30:19.371825+0000","last_clean":"2024-04-06T12:30:19.371825+0000","last_became_active":"2024-04-06T12:30:16.739233+0000","last_became_peered":"2024-04-06T12:30:16.739233+0000","last_unstale":"2024-04-06T12:30:19.371825+0000","last_undegraded":"2024-04-06T12:30:19.371825+0000","last_fullsized":"2024-04-06T12:30:19.371825+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:45:40.445595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434614+0000","last_change":"2024-04-06T12:30:17.155200+0000","last_active":"2024-04-06T12:30:18.434614+0000","last_peered":"2024-04-06T12:30:18.434614+0000","last_clean":"2024-04-06T12:30:18.434614+0000","last_became_active":"2024-04-06T12:30:17.150303+0000","last_became_peered":"2024-04-06T12:30:17.150303+0000","last_unstale":"2024-04-06T12:30:18.434614+0000","last_undegraded":"2024-04-06T12:30:18.434614+0000","last_fullsized":"2024-04-06T12:30:18.434614+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:50:28.054904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"127'8","reported_seq":19,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.135833+0000","last_change":"2024-04-06T12:30:16.736060+0000","last_active":"2024-04-06T12:30:18.135833+0000","last_peered":"2024-04-06T12:30:18.135833+0000","last_clean":"2024-04-06T12:30:18.135833+0000","last_became_active":"2024-04-06T12:30:16.733937+0000","last_became_peered":"2024-04-06T12:30:16.733937+0000","last_unstale":"2024-04-06T12:30:18.135833+0000","last_undegraded":"2024-04-06T12:30:18.135833+0000","last_fullsized":"2024-04-06T12:30:18.135833+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:42:07.052380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371854+0000","last_change":"2024-04-06T12:30:17.158840+0000","last_active":"2024-04-06T12:30:19.371854+0000","last_peered":"2024-04-06T12:30:19.371854+0000","last_clean":"2024-04-06T12:30:19.371854+0000","last_became_active":"2024-04-06T12:30:17.158386+0000","last_became_peered":"2024-04-06T12:30:17.158386+0000","last_unstale":"2024-04-06T12:30:19.371854+0000","last_undegraded":"2024-04-06T12:30:19.371854+0000","last_fullsized":"2024-04-06T12:30:19.371854+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:32:53.693781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:15.709785+0000","last_change":"2024-04-06T12:30:15.709785+0000","last_active":"2024-04-06T12:30:15.709785+0000","last_peered":"2024-04-06T12:30:15.709785+0000","last_clean":"2024-04-06T12:30:15.709785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:15.709785+0000","last_undegraded":"2024-04-06T12:30:15.709785+0000","last_fullsized":"2024-04-06T12:30:15.709785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:16.716721+0000","last_change":"2024-04-06T12:30:16.716721+0000","last_active":"2024-04-06T12:30:16.716721+0000","last_peered":"2024-04-06T12:30:16.716721+0000","last_clean":"2024-04-06T12:30:16.716721+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:16.716721+0000","last_undegraded":"2024-04-06T12:30:16.716721+0000","last_fullsized":"2024-04-06T12:30:16.716721+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.d","version":"127'1","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.140949+0000","last_change":"2024-04-06T12:30:16.738834+0000","last_active":"2024-04-06T12:30:18.140949+0000","last_peered":"2024-04-06T12:30:18.140949+0000","last_clean":"2024-04-06T12:30:18.140949+0000","last_became_active":"2024-04-06T12:30:16.738548+0000","last_became_peered":"2024-04-06T12:30:16.738548+0000","last_unstale":"2024-04-06T12:30:18.140949+0000","last_undegraded":"2024-04-06T12:30:18.140949+0000","last_fullsized":"2024-04-06T12:30:18.140949+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:34:14.692966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.155787+0000","last_change":"2024-04-06T12:30:17.141293+0000","last_active":"2024-04-06T12:30:17.155787+0000","last_peered":"2024-04-06T12:30:17.155787+0000","last_clean":"2024-04-06T12:30:17.155787+0000","last_became_active":"2024-04-06T12:30:17.141041+0000","last_became_peered":"2024-04-06T12:30:17.141041+0000","last_unstale":"2024-04-06T12:30:17.155787+0000","last_undegraded":"2024-04-06T12:30:17.155787+0000","last_fullsized":"2024-04-06T12:30:17.155787+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:28:04.728040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":10,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.139118+0000","last_change":"2024-04-06T12:30:16.733035+0000","last_active":"2024-04-06T12:30:17.139118+0000","last_peered":"2024-04-06T12:30:17.139118+0000","last_clean":"2024-04-06T12:30:17.139118+0000","last_became_active":"2024-04-06T12:30:16.732648+0000","last_became_peered":"2024-04-06T12:30:16.732648+0000","last_unstale":"2024-04-06T12:30:17.139118+0000","last_undegraded":"2024-04-06T12:30:17.139118+0000","last_fullsized":"2024-04-06T12:30:17.139118+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:32:58.290093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434555+0000","last_change":"2024-04-06T12:30:17.150419+0000","last_active":"2024-04-06T12:30:18.434555+0000","last_peered":"2024-04-06T12:30:18.434555+0000","last_clean":"2024-04-06T12:30:18.434555+0000","last_became_active":"2024-04-06T12:30:17.149955+0000","last_became_peered":"2024-04-06T12:30:17.149955+0000","last_unstale":"2024-04-06T12:30:18.434555+0000","last_undegraded":"2024-04-06T12:30:18.434555+0000","last_fullsized":"2024-04-06T12:30:18.434555+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:05:02.037200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"127'2","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.372079+0000","last_change":"2024-04-06T12:30:16.740234+0000","last_active":"2024-04-06T12:30:19.372079+0000","last_peered":"2024-04-06T12:30:19.372079+0000","last_clean":"2024-04-06T12:30:19.372079+0000","last_became_active":"2024-04-06T12:30:16.739973+0000","last_became_peered":"2024-04-06T12:30:16.739973+0000","last_unstale":"2024-04-06T12:30:19.372079+0000","last_undegraded":"2024-04-06T12:30:19.372079+0000","last_fullsized":"2024-04-06T12:30:19.372079+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:45:54.102151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434487+0000","last_change":"2024-04-06T12:30:17.161559+0000","last_active":"2024-04-06T12:30:18.434487+0000","last_peered":"2024-04-06T12:30:18.434487+0000","last_clean":"2024-04-06T12:30:18.434487+0000","last_became_active":"2024-04-06T12:30:17.161296+0000","last_became_peered":"2024-04-06T12:30:17.161296+0000","last_unstale":"2024-04-06T12:30:18.434487+0000","last_undegraded":"2024-04-06T12:30:18.434487+0000","last_fullsized":"2024-04-06T12:30:18.434487+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:50:43.712066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371778+0000","last_change":"2024-04-06T12:30:16.736815+0000","last_active":"2024-04-06T12:30:19.371778+0000","last_peered":"2024-04-06T12:30:19.371778+0000","last_clean":"2024-04-06T12:30:19.371778+0000","last_became_active":"2024-04-06T12:30:16.736563+0000","last_became_peered":"2024-04-06T12:30:16.736563+0000","last_unstale":"2024-04-06T12:30:19.371778+0000","last_undegraded":"2024-04-06T12:30:19.371778+0000","last_fullsized":"2024-04-06T12:30:19.371778+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:42:37.653575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:16.716721+0000","last_change":"2024-04-06T12:30:16.716721+0000","last_active":"2024-04-06T12:30:16.716721+0000","last_peered":"2024-04-06T12:30:16.716721+0000","last_clean":"2024-04-06T12:30:16.716721+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:16.716721+0000","last_undegraded":"2024-04-06T12:30:16.716721+0000","last_fullsized":"2024-04-06T12:30:16.716721+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:15.709785+0000","last_change":"2024-04-06T12:30:15.709785+0000","last_active":"2024-04-06T12:30:15.709785+0000","last_peered":"2024-04-06T12:30:15.709785+0000","last_clean":"2024-04-06T12:30:15.709785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:15.709785+0000","last_undegraded":"2024-04-06T12:30:15.709785+0000","last_fullsized":"2024-04-06T12:30:15.709785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.372205+0000","last_change":"2024-04-06T12:30:17.161742+0000","last_active":"2024-04-06T12:30:19.372205+0000","last_peered":"2024-04-06T12:30:19.372205+0000","last_clean":"2024-04-06T12:30:19.372205+0000","last_became_active":"2024-04-06T12:30:17.161515+0000","last_became_peered":"2024-04-06T12:30:17.161515+0000","last_unstale":"2024-04-06T12:30:19.372205+0000","last_undegraded":"2024-04-06T12:30:19.372205+0000","last_fullsized":"2024-04-06T12:30:19.372205+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:52:01.384680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371895+0000","last_change":"2024-04-06T12:30:16.741843+0000","last_active":"2024-04-06T12:30:19.371895+0000","last_peered":"2024-04-06T12:30:19.371895+0000","last_clean":"2024-04-06T12:30:19.371895+0000","last_became_active":"2024-04-06T12:30:16.741604+0000","last_became_peered":"2024-04-06T12:30:16.741604+0000","last_unstale":"2024-04-06T12:30:19.371895+0000","last_undegraded":"2024-04-06T12:30:19.371895+0000","last_fullsized":"2024-04-06T12:30:19.371895+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:12:30.025007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434679+0000","last_change":"2024-04-06T12:30:17.150538+0000","last_active":"2024-04-06T12:30:18.434679+0000","last_peered":"2024-04-06T12:30:18.434679+0000","last_clean":"2024-04-06T12:30:18.434679+0000","last_became_active":"2024-04-06T12:30:17.150275+0000","last_became_peered":"2024-04-06T12:30:17.150275+0000","last_unstale":"2024-04-06T12:30:18.434679+0000","last_undegraded":"2024-04-06T12:30:18.434679+0000","last_fullsized":"2024-04-06T12:30:18.434679+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:11:36.447101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"127'2","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.154685+0000","last_change":"2024-04-06T12:30:16.736237+0000","last_active":"2024-04-06T12:30:17.154685+0000","last_peered":"2024-04-06T12:30:17.154685+0000","last_clean":"2024-04-06T12:30:17.154685+0000","last_became_active":"2024-04-06T12:30:16.735966+0000","last_became_peered":"2024-04-06T12:30:16.735966+0000","last_unstale":"2024-04-06T12:30:17.154685+0000","last_undegraded":"2024-04-06T12:30:17.154685+0000","last_fullsized":"2024-04-06T12:30:17.154685+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:32:52.686846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.135875+0000","last_change":"2024-04-06T12:30:17.146535+0000","last_active":"2024-04-06T12:30:18.135875+0000","last_peered":"2024-04-06T12:30:18.135875+0000","last_clean":"2024-04-06T12:30:18.135875+0000","last_became_active":"2024-04-06T12:30:17.146260+0000","last_became_peered":"2024-04-06T12:30:17.146260+0000","last_unstale":"2024-04-06T12:30:18.135875+0000","last_undegraded":"2024-04-06T12:30:18.135875+0000","last_fullsized":"2024-04-06T12:30:18.135875+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:59:28.638138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434421+0000","last_change":"2024-04-06T12:30:16.741338+0000","last_active":"2024-04-06T12:30:18.434421+0000","last_peered":"2024-04-06T12:30:18.434421+0000","last_clean":"2024-04-06T12:30:18.434421+0000","last_became_active":"2024-04-06T12:30:16.740513+0000","last_became_peered":"2024-04-06T12:30:16.740513+0000","last_unstale":"2024-04-06T12:30:18.434421+0000","last_undegraded":"2024-04-06T12:30:18.434421+0000","last_fullsized":"2024-04-06T12:30:18.434421+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:02:37.869473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:16.716721+0000","last_change":"2024-04-06T12:30:16.716721+0000","last_active":"2024-04-06T12:30:16.716721+0000","last_peered":"2024-04-06T12:30:16.716721+0000","last_clean":"2024-04-06T12:30:16.716721+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:16.716721+0000","last_undegraded":"2024-04-06T12:30:16.716721+0000","last_fullsized":"2024-04-06T12:30:16.716721+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:15.709785+0000","last_change":"2024-04-06T12:30:15.709785+0000","last_active":"2024-04-06T12:30:15.709785+0000","last_peered":"2024-04-06T12:30:15.709785+0000","last_clean":"2024-04-06T12:30:15.709785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:15.709785+0000","last_undegraded":"2024-04-06T12:30:15.709785+0000","last_fullsized":"2024-04-06T12:30:15.709785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.157532+0000","last_change":"2024-04-06T12:30:17.156559+0000","last_active":"2024-04-06T12:30:17.157532+0000","last_peered":"2024-04-06T12:30:17.157532+0000","last_clean":"2024-04-06T12:30:17.157532+0000","last_became_active":"2024-04-06T12:30:17.156304+0000","last_became_peered":"2024-04-06T12:30:17.156304+0000","last_unstale":"2024-04-06T12:30:17.157532+0000","last_undegraded":"2024-04-06T12:30:17.157532+0000","last_fullsized":"2024-04-06T12:30:17.157532+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:29:46.115525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":10,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.131174+0000","last_change":"2024-04-06T12:30:16.738521+0000","last_active":"2024-04-06T12:30:17.131174+0000","last_peered":"2024-04-06T12:30:17.131174+0000","last_clean":"2024-04-06T12:30:17.131174+0000","last_became_active":"2024-04-06T12:30:16.738245+0000","last_became_peered":"2024-04-06T12:30:16.738245+0000","last_unstale":"2024-04-06T12:30:17.131174+0000","last_undegraded":"2024-04-06T12:30:17.131174+0000","last_fullsized":"2024-04-06T12:30:17.131174+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:29:45.108590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.372011+0000","last_change":"2024-04-06T12:30:17.161738+0000","last_active":"2024-04-06T12:30:19.372011+0000","last_peered":"2024-04-06T12:30:19.372011+0000","last_clean":"2024-04-06T12:30:19.372011+0000","last_became_active":"2024-04-06T12:30:17.161484+0000","last_became_peered":"2024-04-06T12:30:17.161484+0000","last_unstale":"2024-04-06T12:30:19.372011+0000","last_undegraded":"2024-04-06T12:30:19.372011+0000","last_fullsized":"2024-04-06T12:30:19.372011+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:02:56.674896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136113+0000","last_change":"2024-04-06T12:30:16.738491+0000","last_active":"2024-04-06T12:30:18.136113+0000","last_peered":"2024-04-06T12:30:18.136113+0000","last_clean":"2024-04-06T12:30:18.136113+0000","last_became_active":"2024-04-06T12:30:16.738227+0000","last_became_peered":"2024-04-06T12:30:16.738227+0000","last_unstale":"2024-04-06T12:30:18.136113+0000","last_undegraded":"2024-04-06T12:30:18.136113+0000","last_fullsized":"2024-04-06T12:30:18.136113+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:29:45.108590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.372011+0000","last_change":"2024-04-06T12:30:17.162535+0000","last_active":"2024-04-06T12:30:19.372011+0000","last_peered":"2024-04-06T12:30:19.372011+0000","last_clean":"2024-04-06T12:30:19.372011+0000","last_became_active":"2024-04-06T12:30:17.161329+0000","last_became_peered":"2024-04-06T12:30:17.161329+0000","last_unstale":"2024-04-06T12:30:19.372011+0000","last_undegraded":"2024-04-06T12:30:19.372011+0000","last_fullsized":"2024-04-06T12:30:19.372011+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:11:54.910628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371888+0000","last_change":"2024-04-06T12:30:16.739555+0000","last_active":"2024-04-06T12:30:19.371888+0000","last_peered":"2024-04-06T12:30:19.371888+0000","last_clean":"2024-04-06T12:30:19.371888+0000","last_became_active":"2024-04-06T12:30:16.739275+0000","last_became_peered":"2024-04-06T12:30:16.739275+0000","last_unstale":"2024-04-06T12:30:19.371888+0000","last_undegraded":"2024-04-06T12:30:19.371888+0000","last_fullsized":"2024-04-06T12:30:19.371888+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:04:55.999480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.140925+0000","last_change":"2024-04-06T12:30:17.140057+0000","last_active":"2024-04-06T12:30:18.140925+0000","last_peered":"2024-04-06T12:30:18.140925+0000","last_clean":"2024-04-06T12:30:18.140925+0000","last_became_active":"2024-04-06T12:30:17.139574+0000","last_became_peered":"2024-04-06T12:30:17.139574+0000","last_unstale":"2024-04-06T12:30:18.140925+0000","last_undegraded":"2024-04-06T12:30:18.140925+0000","last_fullsized":"2024-04-06T12:30:18.140925+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:06:36.859191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:30:15.709785+0000","last_change":"2024-04-06T12:30:15.709785+0000","last_active":"2024-04-06T12:30:15.709785+0000","last_peered":"2024-04-06T12:30:15.709785+0000","last_clean":"2024-04-06T12:30:15.709785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:30:15.709785+0000","last_undegraded":"2024-04-06T12:30:15.709785+0000","last_fullsized":"2024-04-06T12:30:15.709785+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.154910+0000","last_change":"2024-04-06T12:30:17.147343+0000","last_active":"2024-04-06T12:30:17.154910+0000","last_peered":"2024-04-06T12:30:17.154910+0000","last_clean":"2024-04-06T12:30:17.154910+0000","last_became_active":"2024-04-06T12:30:17.147093+0000","last_became_peered":"2024-04-06T12:30:17.147093+0000","last_unstale":"2024-04-06T12:30:17.154910+0000","last_undegraded":"2024-04-06T12:30:17.154910+0000","last_fullsized":"2024-04-06T12:30:17.154910+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:30:54.593855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371951+0000","last_change":"2024-04-06T12:30:16.737778+0000","last_active":"2024-04-06T12:30:19.371951+0000","last_peered":"2024-04-06T12:30:19.371951+0000","last_clean":"2024-04-06T12:30:19.371951+0000","last_became_active":"2024-04-06T12:30:16.737562+0000","last_became_peered":"2024-04-06T12:30:16.737562+0000","last_unstale":"2024-04-06T12:30:19.371951+0000","last_undegraded":"2024-04-06T12:30:19.371951+0000","last_fullsized":"2024-04-06T12:30:19.371951+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:35.534922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.154148+0000","last_change":"2024-04-06T12:30:17.146320+0000","last_active":"2024-04-06T12:30:17.154148+0000","last_peered":"2024-04-06T12:30:17.154148+0000","last_clean":"2024-04-06T12:30:17.154148+0000","last_became_active":"2024-04-06T12:30:17.146053+0000","last_became_peered":"2024-04-06T12:30:17.146053+0000","last_unstale":"2024-04-06T12:30:17.154148+0000","last_undegraded":"2024-04-06T12:30:17.154148+0000","last_fullsized":"2024-04-06T12:30:17.154148+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:02:56.674896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"127'2","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.372128+0000","last_change":"2024-04-06T12:30:16.735892+0000","last_active":"2024-04-06T12:30:19.372128+0000","last_peered":"2024-04-06T12:30:19.372128+0000","last_clean":"2024-04-06T12:30:19.372128+0000","last_became_active":"2024-04-06T12:30:16.735628+0000","last_became_peered":"2024-04-06T12:30:16.735628+0000","last_unstale":"2024-04-06T12:30:19.372128+0000","last_undegraded":"2024-04-06T12:30:19.372128+0000","last_fullsized":"2024-04-06T12:30:19.372128+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:06:07.991018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"117'205","reported_seq":593,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.131557+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:30:17.131557+0000","last_peered":"2024-04-06T12:30:17.131557+0000","last_clean":"2024-04-06T12:30:17.131557+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:30:17.131557+0000","last_undegraded":"2024-04-06T12:30:17.131557+0000","last_fullsized":"2024-04-06T12:30:17.131557+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":6},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":194,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":48,"acting":48,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575066,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59500,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312340,"statfs":{"total":96636764160,"available":96575836160,"internally_reserved":0,"allocated":2187264,"data_stored":1383350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53200000000000003,"15min":0.55800000000000005},"min":{"1min":0.311,"5min":0.28899999999999998,"15min":0.219},"max":{"1min":0.75600000000000001,"5min":1.143,"15min":1.143},"last":0.628},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.52500000000000002,"15min":0.55400000000000005},"min":{"1min":0.37,"5min":0.246,"15min":0.246},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.501}]},{"osd":1,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53800000000000003,"15min":0.55700000000000005},"min":{"1min":0.40000000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.81399999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.57199999999999995},"min":{"1min":0.41199999999999998,"5min":0.34300000000000003,"15min":0.27900000000000003},"max":{"1min":0.70499999999999996,"5min":0.77400000000000002,"15min":0.79600000000000004},"last":0.54800000000000004}]},{"osd":2,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52600000000000002,"15min":0.55700000000000005},"min":{"1min":0.38600000000000001,"5min":0.30599999999999999,"15min":0.249},"max":{"1min":0.69899999999999995,"5min":0.77100000000000002,"15min":0.83299999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.55800000000000005,"15min":0.58399999999999996},"min":{"1min":0.42899999999999999,"5min":0.34499999999999997,"15min":0.29599999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.60099999999999998}]},{"osd":3,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55800000000000005,"15min":0.57599999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.34399999999999997},"max":{"1min":0.69299999999999995,"5min":1.357,"15min":1.357},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.57399999999999995,"15min":0.59299999999999997},"min":{"1min":0.46200000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.72999999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.53800000000000003}]},{"osd":4,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45400000000000001,"15min":0.44400000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.25},"max":{"1min":0.63600000000000001,"5min":0.71399999999999997,"15min":0.72999999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53700000000000003,"15min":0.52600000000000002},"min":{"1min":0.373,"5min":0.34300000000000003,"15min":0.32800000000000001},"max":{"1min":0.74199999999999999,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.57399999999999995}]},{"osd":5,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.50700000000000001,"15min":0.496},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.23799999999999999},"max":{"1min":0.80500000000000005,"5min":0.81899999999999995,"15min":0.86799999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.29299999999999998},"max":{"1min":0.73799999999999999,"5min":0.81599999999999995,"15min":0.83499999999999996},"last":0.58999999999999997}]},{"osd":6,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54900000000000004,"15min":0.53800000000000003},"min":{"1min":0.36799999999999999,"5min":0.33300000000000002,"15min":0.28799999999999998},"max":{"1min":0.76400000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.33100000000000002,"5min":0.312,"15min":0.30399999999999999},"max":{"1min":0.82199999999999995,"5min":0.86299999999999999,"15min":0.88500000000000001},"last":0.51700000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575064,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32616,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339224,"statfs":{"total":96636764160,"available":96603365376,"internally_reserved":0,"allocated":1531904,"data_stored":765356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.52500000000000002,"15min":0.54000000000000004},"min":{"1min":0.40999999999999998,"5min":0.33700000000000002,"15min":0.26200000000000001},"max":{"1min":0.73199999999999998,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52500000000000002,"15min":0.53800000000000003},"min":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.245},"max":{"1min":0.754,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.48399999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56699999999999995,"15min":0.59099999999999997},"min":{"1min":0.46300000000000002,"5min":0.32700000000000001,"15min":0.24399999999999999},"max":{"1min":0.73899999999999999,"5min":0.78700000000000003,"15min":0.84999999999999998},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58299999999999996,"15min":0.60099999999999998},"min":{"1min":0.41599999999999998,"5min":0.30199999999999999,"15min":0.221},"max":{"1min":0.72699999999999998,"5min":0.81999999999999995,"15min":0.83099999999999996},"last":0.58599999999999997}]},{"osd":2,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52800000000000002,"15min":0.55400000000000005},"min":{"1min":0.38100000000000001,"5min":0.30499999999999999,"15min":0.29899999999999999},"max":{"1min":0.71399999999999997,"5min":0.749,"15min":0.82399999999999995},"last":0.624},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55200000000000005,"15min":0.57399999999999995},"min":{"1min":0.43099999999999999,"5min":0.32300000000000001,"15min":0.254},"max":{"1min":0.76900000000000002,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.51900000000000002}]},{"osd":3,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.56399999999999995,"15min":0.58299999999999996},"min":{"1min":0.41399999999999998,"5min":0.317,"15min":0.23300000000000001},"max":{"1min":0.73199999999999998,"5min":0.77300000000000002,"15min":0.81200000000000006},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59899999999999998,"15min":0.621},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.374},"max":{"1min":0.76200000000000001,"5min":0.82799999999999996,"15min":0.872},"last":0.66300000000000003}]},{"osd":4,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.502,"15min":0.496},"min":{"1min":0.35999999999999999,"5min":0.28599999999999998,"15min":0.27500000000000002},"max":{"1min":0.66900000000000004,"5min":0.754,"15min":0.754},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.52900000000000003,"15min":0.52800000000000002},"min":{"1min":0.317,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72799999999999998,"5min":0.73199999999999998,"15min":0.82199999999999995},"last":0.60899999999999999}]},{"osd":5,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.49199999999999999,"15min":0.48899999999999999},"min":{"1min":0.30099999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.74099999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.32400000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.80200000000000005,"15min":0.83699999999999997},"last":0.64400000000000002}]},{"osd":7,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54000000000000004,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76500000000000001,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56699999999999995,"15min":0.55800000000000005},"min":{"1min":0.38900000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80700000000000005,"5min":0.84799999999999998,"15min":0.88900000000000001},"last":0.67500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575064,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1572864,"data_stored":785217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:29:32 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.36399999999999999,"15min":0.36099999999999999},"min":{"1min":0.25,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.52800000000000002,"5min":0.53400000000000003,"15min":0.58999999999999997},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.37,"15min":0.39300000000000002},"min":{"1min":0.251,"5min":0.215,"15min":0.215},"max":{"1min":0.56399999999999995,"5min":0.628,"15min":0.628},"last":0.32700000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:29:32 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38700000000000001,"15min":0.39600000000000002},"min":{"1min":0.27000000000000002,"5min":0.23899999999999999,"15min":0.19400000000000001},"max":{"1min":0.58099999999999996,"5min":0.61799999999999999,"15min":0.67600000000000005},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.41999999999999998,"15min":0.432},"min":{"1min":0.29899999999999999,"5min":0.255,"15min":0.255},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.66100000000000003},"last":0.36399999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:29:32 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44,"15min":0.44800000000000001},"min":{"1min":0.318,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67500000000000004,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40500000000000003,"15min":0.41899999999999998},"min":{"1min":0.29699999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.65900000000000003,"5min":0.69599999999999995,"15min":0.86299999999999999},"last":0.35199999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:29:32 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59599999999999997,"15min":0.61499999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.34799999999999998},"max":{"1min":0.78700000000000003,"5min":0.86799999999999999,"15min":0.91100000000000003},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.60499999999999998,"15min":0.61799999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.30399999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.94899999999999995},"last":0.624}]},{"osd":5,"last update":"Sat Apr 6 12:29:32 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.61199999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.38700000000000001},"max":{"1min":0.83699999999999997,"5min":0.85899999999999999,"15min":0.93500000000000005},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.622,"15min":0.628},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.38300000000000001},"max":{"1min":0.747,"5min":0.999,"15min":0.999},"last":0.54000000000000004}]},{"osd":6,"last update":"Sat Apr 6 12:29:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60199999999999998,"15min":0.61399999999999999},"min":{"1min":0.46899999999999997,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78300000000000003,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.628,"15min":0.64700000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78700000000000003,"5min":0.88700000000000001,"15min":0.97899999999999998},"last":0.52000000000000002}]},{"osd":7,"last update":"Sat Apr 6 12:29:32 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63300000000000001,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.41899999999999998,"15min":0.38800000000000001},"max":{"1min":0.81200000000000006,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.62,"15min":0.63600000000000001},"min":{"1min":0.42399999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.91400000000000003,"5min":0.95599999999999996,"15min":0.96799999999999997},"last":0.55000000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575064,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33264,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338576,"statfs":{"total":96636764160,"available":96602701824,"internally_reserved":0,"allocated":2191360,"data_stored":1383825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.376,"15min":0.38600000000000001},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":0.72699999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38900000000000001,"15min":0.40500000000000003},"min":{"1min":0.23499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.505,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.26700000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.38900000000000001,"15min":0.40899999999999997},"min":{"1min":0.28100000000000003,"5min":0.23599999999999999,"15min":0.19700000000000001},"max":{"1min":0.49099999999999999,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41999999999999998,"15min":0.432},"min":{"1min":0.28100000000000003,"5min":0.249,"15min":0.24299999999999999},"max":{"1min":0.56299999999999994,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.33900000000000002}]},{"osd":3,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.40400000000000003,"15min":0.41899999999999998},"min":{"1min":0.28799999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.53300000000000003,"5min":1.25,"15min":1.25},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.436,"15min":0.46300000000000002},"min":{"1min":0.26700000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.64200000000000002,"5min":0.74099999999999999,"15min":0.748},"last":0.39400000000000002}]},{"osd":4,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.59899999999999998,"15min":0.621},"min":{"1min":0.38500000000000001,"5min":0.371,"15min":0.35699999999999998},"max":{"1min":0.79500000000000004,"5min":1.514,"15min":1.514},"last":0.755},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.60999999999999999,"15min":0.63},"min":{"1min":0.42199999999999999,"5min":0.36699999999999999,"15min":0.35199999999999998},"max":{"1min":0.84699999999999998,"5min":1.202,"15min":1.202},"last":0.628}]},{"osd":5,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.624},"min":{"1min":0.38100000000000001,"5min":0.34399999999999997,"15min":0.33700000000000002},"max":{"1min":0.75700000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.46200000000000002,"5min":0.38800000000000001,"15min":0.315},"max":{"1min":0.82499999999999996,"5min":1.006,"15min":1.006},"last":0.61499999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.625,"15min":0.65200000000000002},"min":{"1min":0.48699999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.79000000000000004,"5min":1.038,"15min":1.038},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63600000000000001,"15min":0.66100000000000003},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.38100000000000001},"max":{"1min":0.82699999999999996,"5min":1.613,"15min":1.613},"last":0.72399999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.63900000000000001,"15min":0.66400000000000003},"min":{"1min":0.50800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80600000000000005,"5min":1.54,"15min":1.54},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.63800000000000001,"15min":0.65700000000000003},"min":{"1min":0.39300000000000002,"5min":0.38400000000000001,"15min":0.375},"max":{"1min":0.77400000000000002,"5min":1.498,"15min":1.498},"last":0.70699999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575066,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54804,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317036,"statfs":{"total":96636764160,"available":96580644864,"internally_reserved":0,"allocated":1572864,"data_stored":785217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:29:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.36599999999999999,"15min":0.38300000000000001},"min":{"1min":0.23899999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.54000000000000004,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.39200000000000002,"15min":0.41199999999999998},"min":{"1min":0.248,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.65600000000000003,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.33700000000000002}]},{"osd":1,"last update":"Sat Apr 6 12:29:29 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.38700000000000001,"15min":0.40300000000000002},"min":{"1min":0.252,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.64600000000000002,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.315},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40000000000000002,"15min":0.43099999999999999},"min":{"1min":0.28299999999999997,"5min":0.27100000000000002,"15min":0.23599999999999999},"max":{"1min":0.61899999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.28000000000000003}]},{"osd":3,"last update":"Sat Apr 6 12:29:29 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.433,"15min":0.45000000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64200000000000002,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.373},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.434,"15min":0.439},"min":{"1min":0.308,"5min":0.29499999999999998,"15min":0.29099999999999998},"max":{"1min":0.63200000000000001,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.30099999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:29:29 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.58299999999999996,"15min":0.61599999999999999},"min":{"1min":0.40000000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.93300000000000005,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.58699999999999997,"15min":0.626},"min":{"1min":0.38900000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.80400000000000005,"5min":0.91200000000000003,"15min":0.92600000000000005},"last":0.52800000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:29:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.61299999999999999,"15min":0.63800000000000001},"min":{"1min":0.39700000000000002,"5min":0.374,"15min":0.36499999999999999},"max":{"1min":0.83899999999999997,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.59799999999999998,"15min":0.629},"min":{"1min":0.41399999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.98099999999999998,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.502}]},{"osd":6,"last update":"Sat Apr 6 12:29:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.60199999999999998,"15min":0.63400000000000001},"min":{"1min":0.41799999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.91800000000000004,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.627,"15min":0.65400000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":1.728,"15min":1.728},"last":0.45300000000000001}]},{"osd":7,"last update":"Sat Apr 6 12:29:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.60999999999999999,"15min":0.64800000000000002},"min":{"1min":0.46800000000000003,"5min":0.40699999999999997,"15min":0.35599999999999998},"max":{"1min":0.88900000000000001,"5min":0.95699999999999996,"15min":0.95799999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.64100000000000001,"15min":0.66400000000000003},"min":{"1min":0.42699999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.95299999999999996,"5min":1.758,"15min":1.758},"last":0.56499999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575064,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32628,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339212,"statfs":{"total":96636764160,"available":96603353088,"internally_reserved":0,"allocated":1536000,"data_stored":766375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:29:33 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36099999999999999,"15min":0.36799999999999999},"min":{"1min":0.29299999999999998,"5min":0.21099999999999999,"15min":0.191},"max":{"1min":0.46000000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.247},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36399999999999999,"15min":0.38300000000000001},"min":{"1min":0.28000000000000003,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.48899999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.27500000000000002}]},{"osd":1,"last update":"Sat Apr 6 12:29:33 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.38400000000000001},"min":{"1min":0.26600000000000001,"5min":0.22700000000000001,"15min":0.217},"max":{"1min":0.44600000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.37},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.40300000000000002,"15min":0.41599999999999998},"min":{"1min":0.27500000000000002,"5min":0.26100000000000001,"15min":0.25700000000000001},"max":{"1min":0.49299999999999999,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.30499999999999999}]},{"osd":2,"last update":"Sat Apr 6 12:29:33 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.442,"15min":0.46200000000000002},"min":{"1min":0.28799999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59699999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.46500000000000002,"15min":0.47799999999999998},"min":{"1min":0.35599999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.55700000000000005,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.34999999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:29:33 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57799999999999996,"15min":0.60699999999999998},"min":{"1min":0.38200000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.76100000000000001,"5min":0.88500000000000001,"15min":0.95999999999999996},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58999999999999997,"15min":0.624},"min":{"1min":0.41099999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.79500000000000004,"5min":0.90100000000000002,"15min":0.93500000000000005},"last":0.46400000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:29:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.625,"15min":0.65900000000000003},"min":{"1min":0.45400000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.80300000000000005,"5min":0.92200000000000004,"15min":0.96699999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.63500000000000001,"15min":0.66200000000000003},"min":{"1min":0.44800000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.79600000000000004,"5min":0.94499999999999995,"15min":0.95599999999999996},"last":0.45500000000000002}]},{"osd":6,"last update":"Sat Apr 6 12:29:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.59699999999999998,"15min":0.624},"min":{"1min":0.40300000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77000000000000002,"5min":1.026,"15min":1.026},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.64000000000000001,"15min":0.65800000000000003},"min":{"1min":0.39200000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.83699999999999997,"5min":1.716,"15min":1.716},"last":0.48699999999999999}]},{"osd":7,"last update":"Sat Apr 6 12:29:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.64400000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.33300000000000002},"max":{"1min":0.85799999999999998,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.63,"15min":0.66300000000000003},"min":{"1min":0.41899999999999998,"5min":0.39600000000000002,"15min":0.373},"max":{"1min":0.81399999999999995,"5min":1.002,"15min":1.002},"last":0.502}]}]},{"osd":4,"up_from":13,"seq":55834575065,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32672,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":1593344,"data_stored":792914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:29:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54700000000000004,"15min":0.56499999999999995},"min":{"1min":0.41099999999999998,"5min":0.29399999999999998,"15min":0.23899999999999999},"max":{"1min":0.80500000000000005,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53900000000000003,"15min":0.56100000000000005},"min":{"1min":0.35499999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.45500000000000002}]},{"osd":1,"last update":"Sat Apr 6 12:29:43 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.58099999999999996,"15min":0.60899999999999999},"min":{"1min":0.42399999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.83399999999999996},"last":0.315},{"interface":"front","average":{"1min":0.62,"5min":0.57799999999999996,"15min":0.59499999999999997},"min":{"1min":0.38400000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.751,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.38700000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:29:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.55200000000000005,"15min":0.56999999999999995},"min":{"1min":0.41799999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.56299999999999994,"15min":0.58499999999999996},"min":{"1min":0.39700000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.753,"5min":0.76900000000000002,"15min":0.80900000000000005},"last":0.503}]},{"osd":3,"last update":"Sat Apr 6 12:29:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.55300000000000005,"15min":0.56299999999999994},"min":{"1min":0.38500000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76800000000000002,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.443},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58199999999999996,"15min":0.60099999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.77500000000000002,"5min":0.79600000000000004,"15min":0.80900000000000005},"last":0.47099999999999997}]},{"osd":5,"last update":"Sat Apr 6 12:29:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.46899999999999997,"15min":0.47699999999999998},"min":{"1min":0.308,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.73599999999999999,"5min":0.76300000000000001,"15min":0.83599999999999997},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.49099999999999999,"15min":0.497},"min":{"1min":0.33600000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.77000000000000002},"last":0.49299999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:29:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.37,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.70499999999999996,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52000000000000002,"15min":0.53300000000000003},"min":{"1min":0.41399999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.748,"5min":0.748,"15min":0.872},"last":0.56999999999999995}]},{"osd":7,"last update":"Sat Apr 6 12:29:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.43099999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.77600000000000002,"5min":0.79300000000000004,"15min":0.82299999999999995},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.55900000000000005,"15min":0.56699999999999995},"min":{"1min":0.47499999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.84799999999999998},"last":0.55500000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575065,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50704,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321136,"statfs":{"total":96636764160,"available":96584843264,"internally_reserved":0,"allocated":1568768,"data_stored":785183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.54000000000000004,"15min":0.57199999999999995},"min":{"1min":0.38800000000000001,"5min":0.32700000000000001,"15min":0.24199999999999999},"max":{"1min":0.81999999999999995,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52600000000000002,"15min":0.55000000000000004},"min":{"1min":0.36099999999999999,"5min":0.29499999999999998,"15min":0.22500000000000001},"max":{"1min":0.72399999999999998,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.36699999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52100000000000002,"15min":0.55600000000000005},"min":{"1min":0.36099999999999999,"5min":0.30099999999999999,"15min":0.26000000000000001},"max":{"1min":0.68600000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53800000000000003,"15min":0.55600000000000005},"min":{"1min":0.379,"5min":0.33400000000000002,"15min":0.30599999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.41799999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53600000000000003,"15min":0.56399999999999995},"min":{"1min":0.38500000000000001,"5min":0.32500000000000001,"15min":0.21099999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.62},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.54600000000000004,"15min":0.56699999999999995},"min":{"1min":0.44800000000000001,"5min":0.35999999999999999,"15min":0.27400000000000002},"max":{"1min":0.77100000000000002,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.49099999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.54900000000000004,"15min":0.57099999999999995},"min":{"1min":0.37,"5min":0.32000000000000001,"15min":0.31},"max":{"1min":0.83799999999999997,"5min":0.874,"15min":0.874},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56699999999999995,"15min":0.59499999999999997},"min":{"1min":0.41899999999999998,"5min":0.35699999999999998,"15min":0.32300000000000001},"max":{"1min":0.71199999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.51700000000000002}]},{"osd":4,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46400000000000002,"15min":0.46700000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28299999999999997},"max":{"1min":0.71799999999999997,"5min":0.755,"15min":0.81200000000000006},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53400000000000003,"15min":0.52600000000000002},"min":{"1min":0.29499999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.79900000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.437}]},{"osd":6,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47299999999999998,"15min":0.47599999999999998},"min":{"1min":0.35099999999999998,"5min":0.313,"15min":0.28000000000000003},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47999999999999998,"15min":0.48399999999999999},"min":{"1min":0.33200000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.77000000000000002},"last":0.504}]},{"osd":7,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.496,"15min":0.5},"min":{"1min":0.318,"5min":0.29799999999999999,"15min":0.252},"max":{"1min":0.753,"5min":0.78600000000000003,"15min":0.79100000000000004},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58899999999999997,"15min":0.59599999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.35199999999999998},"max":{"1min":0.83399999999999996,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.57199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:30:21.258 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:30:21.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.418+0000 7f3e6018d700 1 Processor -- start 2024-04-06T12:30:21.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.418+0000 7f3e6018d700 1 -- start start 2024-04-06T12:30:21.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.419+0000 7f3e6018d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e580b3e20 0x7f3e5815e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.419+0000 7f3e6018d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e580b3990 con 0x7f3e5806ae40 2024-04-06T12:30:21.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.419+0000 7f3e6018d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e580b3060 con 0x7f3e5815d7b0 2024-04-06T12:30:21.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.419+0000 7f3e6018d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e580b3340 con 0x7f3e580b3e20 2024-04-06T12:30:21.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.419+0000 7f3e5d728700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3e5806ae40 0x7f3e5815ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45546/0 (socket says 172.21.15.189:45546) 2024-04-06T12:30:21.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.419+0000 7f3e5d728700 1 -- 172.21.15.189:0/2100382802 learned_addr learned my addr 172.21.15.189:0/2100382802 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:21.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5cf27700 1 -- 172.21.15.189:0/2100382802 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915531707 0 0) 0x7f3e580b3990 con 0x7f3e5806ae40 2024-04-06T12:30:21.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5e72a700 1 --2- 172.21.15.189:0/2100382802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e580b3e20 0x7f3e5815e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5cf27700 1 -- 172.21.15.189:0/2100382802 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e440035b0 con 0x7f3e5806ae40 2024-04-06T12:30:21.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5cf27700 1 -- 172.21.15.189:0/2100382802 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 805420072 0 0) 0x7f3e440035b0 con 0x7f3e5806ae40 2024-04-06T12:30:21.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5cf27700 1 -- 172.21.15.189:0/2100382802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e580b3e20 msgr2=0x7f3e5815e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:21.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5cf27700 1 --2- 172.21.15.189:0/2100382802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e580b3e20 0x7f3e5815e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5cf27700 1 -- 172.21.15.189:0/2100382802 >> v1:172.21.15.192:6789/0 conn(0x7f3e5815d7b0 legacy=0x7f3e5806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:21.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5cf27700 1 -- 172.21.15.189:0/2100382802 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e580b3620 con 0x7f3e5806ae40 2024-04-06T12:30:21.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5cf27700 1 -- 172.21.15.189:0/2100382802 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f3e54001670 con 0x7f3e5806ae40 2024-04-06T12:30:21.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5cf27700 1 -- 172.21.15.189:0/2100382802 >> v1:172.21.15.189:6789/0 conn(0x7f3e5806ae40 legacy=0x7f3e5815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:21.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5cf27700 1 --2- 172.21.15.189:0/2100382802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e580b3e20 0x7f3e44003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5cf27700 1 --2- 172.21.15.189:0/2100382802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e44003fb0 0x7f3e44004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5cf27700 1 --2- 172.21.15.189:0/2100382802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e44004d90 0x7f3e44005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5cf27700 1 -- 172.21.15.189:0/2100382802 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e580b3340 con 0x7f3e44003fb0 2024-04-06T12:30:21.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5cf27700 1 -- 172.21.15.189:0/2100382802 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4400bba0 con 0x7f3e44004d90 2024-04-06T12:30:21.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5cf27700 1 -- 172.21.15.189:0/2100382802 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4400bd60 con 0x7f3e580b3e20 2024-04-06T12:30:21.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5d728700 1 --2- 172.21.15.189:0/2100382802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e44003fb0 0x7f3e44004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5df29700 1 --2- 172.21.15.189:0/2100382802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e580b3e20 0x7f3e44003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5e72a700 1 --2- 172.21.15.189:0/2100382802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e44004d90 0x7f3e44005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5d728700 1 -- 172.21.15.189:0/2100382802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e580b3e20 msgr2=0x7f3e44003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:21.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5d728700 1 --2- 172.21.15.189:0/2100382802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e580b3e20 0x7f3e44003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5d728700 1 -- 172.21.15.189:0/2100382802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e44004d90 msgr2=0x7f3e44005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:21.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5d728700 1 --2- 172.21.15.189:0/2100382802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e44004d90 0x7f3e44005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.420+0000 7f3e5d728700 1 -- 172.21.15.189:0/2100382802 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e440035b0 con 0x7f3e44003fb0 2024-04-06T12:30:21.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.421+0000 7f3e5d728700 1 --2- 172.21.15.189:0/2100382802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e44003fb0 0x7f3e44004850 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f3e5400c240 tx=0x7f3e44005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=962d619f2214fa8c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:21.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.421+0000 7f3e5cf27700 1 -- 172.21.15.189:0/2100382802 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e54007c10 con 0x7f3e44003fb0 2024-04-06T12:30:21.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.421+0000 7f3e5cf27700 1 -- 172.21.15.189:0/2100382802 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5400cde0 con 0x7f3e44003fb0 2024-04-06T12:30:21.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.421+0000 7f3e5cf27700 1 -- 172.21.15.189:0/2100382802 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e54005ba0 con 0x7f3e44003fb0 2024-04-06T12:30:21.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.421+0000 7f3e6018d700 1 -- 172.21.15.189:0/2100382802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e44003fb0 msgr2=0x7f3e44004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:21.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.421+0000 7f3e6018d700 1 --2- 172.21.15.189:0/2100382802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e44003fb0 0x7f3e44004850 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f3e5400c240 tx=0x7f3e44005880 comp rx=0 tx=0).stop 2024-04-06T12:30:21.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.421+0000 7f3e5e72a700 1 -- 172.21.15.189:0/2100382802 reap_dead start 2024-04-06T12:30:21.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.421+0000 7f3e6018d700 1 -- 172.21.15.189:0/2100382802 shutdown_connections 2024-04-06T12:30:21.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.421+0000 7f3e6018d700 1 -- 172.21.15.189:0/2100382802 >> 172.21.15.189:0/2100382802 conn(0x7f3e580d00b0 msgr2=0x7f3e580d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:21.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.421+0000 7f3e6018d700 1 -- 172.21.15.189:0/2100382802 shutdown_connections 2024-04-06T12:30:21.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.421+0000 7f3e6018d700 1 -- 172.21.15.189:0/2100382802 wait complete. 2024-04-06T12:30:21.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.423+0000 7f3e6018d700 1 Processor -- start 2024-04-06T12:30:21.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.423+0000 7f3e6018d700 1 -- start start 2024-04-06T12:30:21.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.423+0000 7f3e6018d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e44003fb0 0x7f3e580afb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.423+0000 7f3e6018d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e580cda40 0x7f3e5815a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.423+0000 7f3e6018d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e580cf070 0x7f3e5815aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.423+0000 7f3e6018d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e580cdfd0 con 0x7f3e44003fb0 2024-04-06T12:30:21.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.423+0000 7f3e6018d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e580ce2b0 con 0x7f3e580cf070 2024-04-06T12:30:21.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.423+0000 7f3e6018d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e580ce590 con 0x7f3e580cda40 2024-04-06T12:30:21.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.423+0000 7f3e5df29700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e44003fb0 0x7f3e580afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.423+0000 7f3e5df29700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e44003fb0 0x7f3e580afb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40738/0 (socket says 172.21.15.189:40738) 2024-04-06T12:30:21.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.423+0000 7f3e5df29700 1 -- 172.21.15.189:0/3374342591 learned_addr learned my addr 172.21.15.189:0/3374342591 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:21.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.423+0000 7f3e5d728700 1 --2- 172.21.15.189:0/3374342591 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e580cda40 0x7f3e5815a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.423+0000 7f3e5e72a700 1 --2- 172.21.15.189:0/3374342591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e580cf070 0x7f3e5815aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.424+0000 7f3e5df29700 1 -- 172.21.15.189:0/3374342591 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e580cda40 msgr2=0x7f3e5815a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:21.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.424+0000 7f3e5df29700 1 --2- 172.21.15.189:0/3374342591 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e580cda40 0x7f3e5815a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.424+0000 7f3e5df29700 1 -- 172.21.15.189:0/3374342591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e580cf070 msgr2=0x7f3e5815aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:21.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.424+0000 7f3e5df29700 1 --2- 172.21.15.189:0/3374342591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e580cf070 0x7f3e5815aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.424+0000 7f3e5df29700 1 -- 172.21.15.189:0/3374342591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e580ce870 con 0x7f3e44003fb0 2024-04-06T12:30:21.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.424+0000 7f3e5df29700 1 --2- 172.21.15.189:0/3374342591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e44003fb0 0x7f3e580afb70 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f3e4800f820 tx=0x7f3e4800fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:21.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.424+0000 7f3e4e7fc700 1 -- 172.21.15.189:0/3374342591 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e480052e0 con 0x7f3e44003fb0 2024-04-06T12:30:21.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.424+0000 7f3e4e7fc700 1 -- 172.21.15.189:0/3374342591 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e48005440 con 0x7f3e44003fb0 2024-04-06T12:30:21.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.424+0000 7f3e4e7fc700 1 -- 172.21.15.189:0/3374342591 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e480038c0 con 0x7f3e44003fb0 2024-04-06T12:30:21.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.424+0000 7f3e6018d700 1 -- 172.21.15.189:0/3374342591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e580ceb50 con 0x7f3e44003fb0 2024-04-06T12:30:21.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.424+0000 7f3e6018d700 1 -- 172.21.15.189:0/3374342591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e580b00b0 con 0x7f3e44003fb0 2024-04-06T12:30:21.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.425+0000 7f3e6018d700 1 -- 172.21.15.189:0/3374342591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e580ced90 con 0x7f3e44003fb0 2024-04-06T12:30:21.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.425+0000 7f3e4e7fc700 1 -- 172.21.15.189:0/3374342591 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f3e48003a20 con 0x7f3e44003fb0 2024-04-06T12:30:21.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.426+0000 7f3e4e7fc700 1 --2- 172.21.15.189:0/3374342591 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3e3c06adb0 0x7f3e3c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.426+0000 7f3e4e7fc700 1 -- 172.21.15.189:0/3374342591 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3e48089270 con 0x7f3e44003fb0 2024-04-06T12:30:21.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.426+0000 7f3e5d728700 1 --2- 172.21.15.189:0/3374342591 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3e3c06adb0 0x7f3e3c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.426+0000 7f3e5d728700 1 --2- 172.21.15.189:0/3374342591 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3e3c06adb0 0x7f3e3c06d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3e54015040 tx=0x7f3e54007000 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:21.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.430+0000 7f3e4e7fc700 1 -- 172.21.15.189:0/3374342591 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3e580ced90 con 0x7f3e44003fb0 2024-04-06T12:30:21.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.621+0000 7f3e6018d700 1 -- 172.21.15.189:0/3374342591 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3e58067bb0 con 0x7f3e3c06adb0 2024-04-06T12:30:21.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.629+0000 7f3e4e7fc700 1 -- 172.21.15.189:0/3374342591 <== mgr.7875 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289803 (secure 0 0 0) 0x7f3e58067bb0 con 0x7f3e3c06adb0 2024-04-06T12:30:21.629 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:30:21.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.632+0000 7f3e6018d700 1 -- 172.21.15.189:0/3374342591 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3e3c06adb0 msgr2=0x7f3e3c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:21.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.632+0000 7f3e6018d700 1 --2- 172.21.15.189:0/3374342591 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3e3c06adb0 0x7f3e3c06d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3e54015040 tx=0x7f3e54007000 comp rx=0 tx=0).stop 2024-04-06T12:30:21.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.632+0000 7f3e6018d700 1 -- 172.21.15.189:0/3374342591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e44003fb0 msgr2=0x7f3e580afb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:21.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.632+0000 7f3e6018d700 1 --2- 172.21.15.189:0/3374342591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e44003fb0 0x7f3e580afb70 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f3e4800f820 tx=0x7f3e4800fb30 comp rx=0 tx=0).stop 2024-04-06T12:30:21.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.632+0000 7f3e6018d700 1 -- 172.21.15.189:0/3374342591 shutdown_connections 2024-04-06T12:30:21.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.632+0000 7f3e6018d700 1 --2- 172.21.15.189:0/3374342591 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3e3c06adb0 0x7f3e3c06d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.632+0000 7f3e6018d700 1 --2- 172.21.15.189:0/3374342591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e580cf070 0x7f3e5815aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.632+0000 7f3e6018d700 1 --2- 172.21.15.189:0/3374342591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e44003fb0 0x7f3e580afb70 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.632+0000 7f3e6018d700 1 --2- 172.21.15.189:0/3374342591 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e580cda40 0x7f3e5815a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.632+0000 7f3e6018d700 1 -- 172.21.15.189:0/3374342591 >> 172.21.15.189:0/3374342591 conn(0x7f3e580d00b0 msgr2=0x7f3e580ae240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:21.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.632+0000 7f3e6018d700 1 -- 172.21.15.189:0/3374342591 shutdown_connections 2024-04-06T12:30:21.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.632+0000 7f3e6018d700 1 -- 172.21.15.189:0/3374342591 wait complete. 2024-04-06T12:30:21.633 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:30:21.642 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-06T12:30:21.464901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":363,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":139,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":328948,"kb_used_data":13540,"kb_used_omap":12,"kb_used_meta":315251,"kb_avail":754645772,"statfs":{"total":773094113280,"available":772757270528,"internally_reserved":0,"allocated":13864960,"data_stored":7496461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":322817592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.745540"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434489+0000","last_change":"2024-04-06T12:30:17.150881+0000","last_active":"2024-04-06T12:30:18.434489+0000","last_peered":"2024-04-06T12:30:18.434489+0000","last_clean":"2024-04-06T12:30:18.434489+0000","last_became_active":"2024-04-06T12:30:17.149992+0000","last_became_peered":"2024-04-06T12:30:17.149992+0000","last_unstale":"2024-04-06T12:30:18.434489+0000","last_undegraded":"2024-04-06T12:30:18.434489+0000","last_fullsized":"2024-04-06T12:30:18.434489+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:21:43.683412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.155686+0000","last_change":"2024-04-06T12:30:17.153710+0000","last_active":"2024-04-06T12:30:17.155686+0000","last_peered":"2024-04-06T12:30:17.155686+0000","last_clean":"2024-04-06T12:30:17.155686+0000","last_became_active":"2024-04-06T12:30:17.153427+0000","last_became_peered":"2024-04-06T12:30:17.153427+0000","last_unstale":"2024-04-06T12:30:17.155686+0000","last_undegraded":"2024-04-06T12:30:17.155686+0000","last_fullsized":"2024-04-06T12:30:17.155686+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:29:16.578210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434601+0000","last_change":"2024-04-06T12:30:17.154869+0000","last_active":"2024-04-06T12:30:18.434601+0000","last_peered":"2024-04-06T12:30:18.434601+0000","last_clean":"2024-04-06T12:30:18.434601+0000","last_became_active":"2024-04-06T12:30:17.154616+0000","last_became_peered":"2024-04-06T12:30:17.154616+0000","last_unstale":"2024-04-06T12:30:18.434601+0000","last_undegraded":"2024-04-06T12:30:18.434601+0000","last_fullsized":"2024-04-06T12:30:18.434601+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:14:49.657999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136985+0000","last_change":"2024-04-06T12:30:17.143623+0000","last_active":"2024-04-06T12:30:18.136985+0000","last_peered":"2024-04-06T12:30:18.136985+0000","last_clean":"2024-04-06T12:30:18.136985+0000","last_became_active":"2024-04-06T12:30:17.143385+0000","last_became_peered":"2024-04-06T12:30:17.143385+0000","last_unstale":"2024-04-06T12:30:18.136985+0000","last_undegraded":"2024-04-06T12:30:18.136985+0000","last_fullsized":"2024-04-06T12:30:18.136985+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:42:40.962052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.161440+0000","last_change":"2024-04-06T12:30:17.142421+0000","last_active":"2024-04-06T12:30:17.161440+0000","last_peered":"2024-04-06T12:30:17.161440+0000","last_clean":"2024-04-06T12:30:17.161440+0000","last_became_active":"2024-04-06T12:30:17.142166+0000","last_became_peered":"2024-04-06T12:30:17.142166+0000","last_unstale":"2024-04-06T12:30:17.161440+0000","last_undegraded":"2024-04-06T12:30:17.161440+0000","last_fullsized":"2024-04-06T12:30:17.161440+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:49:47.788561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.155130+0000","last_change":"2024-04-06T12:30:17.149507+0000","last_active":"2024-04-06T12:30:17.155130+0000","last_peered":"2024-04-06T12:30:17.155130+0000","last_clean":"2024-04-06T12:30:17.155130+0000","last_became_active":"2024-04-06T12:30:17.149223+0000","last_became_peered":"2024-04-06T12:30:17.149223+0000","last_unstale":"2024-04-06T12:30:17.155130+0000","last_undegraded":"2024-04-06T12:30:17.155130+0000","last_fullsized":"2024-04-06T12:30:17.155130+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:32:53.693781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.155431+0000","last_change":"2024-04-06T12:30:17.149327+0000","last_active":"2024-04-06T12:30:17.155431+0000","last_peered":"2024-04-06T12:30:17.155431+0000","last_clean":"2024-04-06T12:30:17.155431+0000","last_became_active":"2024-04-06T12:30:17.148625+0000","last_became_peered":"2024-04-06T12:30:17.148625+0000","last_unstale":"2024-04-06T12:30:17.155431+0000","last_undegraded":"2024-04-06T12:30:17.155431+0000","last_fullsized":"2024-04-06T12:30:17.155431+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:34:15.699901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371725+0000","last_change":"2024-04-06T12:30:17.161842+0000","last_active":"2024-04-06T12:30:19.371725+0000","last_peered":"2024-04-06T12:30:19.371725+0000","last_clean":"2024-04-06T12:30:19.371725+0000","last_became_active":"2024-04-06T12:30:17.161572+0000","last_became_peered":"2024-04-06T12:30:17.161572+0000","last_unstale":"2024-04-06T12:30:19.371725+0000","last_undegraded":"2024-04-06T12:30:19.371725+0000","last_fullsized":"2024-04-06T12:30:19.371725+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:29:16.578210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.155047+0000","last_change":"2024-04-06T12:30:17.148538+0000","last_active":"2024-04-06T12:30:17.155047+0000","last_peered":"2024-04-06T12:30:17.155047+0000","last_clean":"2024-04-06T12:30:17.155047+0000","last_became_active":"2024-04-06T12:30:17.148277+0000","last_became_peered":"2024-04-06T12:30:17.148277+0000","last_unstale":"2024-04-06T12:30:17.155047+0000","last_undegraded":"2024-04-06T12:30:17.155047+0000","last_fullsized":"2024-04-06T12:30:17.155047+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:08:45.975034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.161296+0000","last_change":"2024-04-06T12:30:17.150373+0000","last_active":"2024-04-06T12:30:17.161296+0000","last_peered":"2024-04-06T12:30:17.161296+0000","last_clean":"2024-04-06T12:30:17.161296+0000","last_became_active":"2024-04-06T12:30:17.150122+0000","last_became_peered":"2024-04-06T12:30:17.150122+0000","last_unstale":"2024-04-06T12:30:17.161296+0000","last_undegraded":"2024-04-06T12:30:17.161296+0000","last_fullsized":"2024-04-06T12:30:17.161296+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:21:32.239013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.155505+0000","last_change":"2024-04-06T12:30:17.151989+0000","last_active":"2024-04-06T12:30:17.155505+0000","last_peered":"2024-04-06T12:30:17.155505+0000","last_clean":"2024-04-06T12:30:17.155505+0000","last_became_active":"2024-04-06T12:30:17.151739+0000","last_became_peered":"2024-04-06T12:30:17.151739+0000","last_unstale":"2024-04-06T12:30:17.155505+0000","last_undegraded":"2024-04-06T12:30:17.155505+0000","last_fullsized":"2024-04-06T12:30:17.155505+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:11:54.910628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136116+0000","last_change":"2024-04-06T12:30:17.149805+0000","last_active":"2024-04-06T12:30:18.136116+0000","last_peered":"2024-04-06T12:30:18.136116+0000","last_clean":"2024-04-06T12:30:18.136116+0000","last_became_active":"2024-04-06T12:30:17.149520+0000","last_became_peered":"2024-04-06T12:30:17.149520+0000","last_unstale":"2024-04-06T12:30:18.136116+0000","last_undegraded":"2024-04-06T12:30:18.136116+0000","last_fullsized":"2024-04-06T12:30:18.136116+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:49:47.788561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136831+0000","last_change":"2024-04-06T12:30:17.146422+0000","last_active":"2024-04-06T12:30:18.136831+0000","last_peered":"2024-04-06T12:30:18.136831+0000","last_clean":"2024-04-06T12:30:18.136831+0000","last_became_active":"2024-04-06T12:30:17.146180+0000","last_became_peered":"2024-04-06T12:30:17.146180+0000","last_unstale":"2024-04-06T12:30:18.136831+0000","last_undegraded":"2024-04-06T12:30:18.136831+0000","last_fullsized":"2024-04-06T12:30:18.136831+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:21:37.628811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.140775+0000","last_change":"2024-04-06T12:30:17.146818+0000","last_active":"2024-04-06T12:30:18.140775+0000","last_peered":"2024-04-06T12:30:18.140775+0000","last_clean":"2024-04-06T12:30:18.140775+0000","last_became_active":"2024-04-06T12:30:17.146541+0000","last_became_peered":"2024-04-06T12:30:17.146541+0000","last_unstale":"2024-04-06T12:30:18.140775+0000","last_undegraded":"2024-04-06T12:30:18.140775+0000","last_fullsized":"2024-04-06T12:30:18.140775+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:42:07.595684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.140671+0000","last_change":"2024-04-06T12:30:17.140598+0000","last_active":"2024-04-06T12:30:18.140671+0000","last_peered":"2024-04-06T12:30:18.140671+0000","last_clean":"2024-04-06T12:30:18.140671+0000","last_became_active":"2024-04-06T12:30:17.140322+0000","last_became_peered":"2024-04-06T12:30:17.140322+0000","last_unstale":"2024-04-06T12:30:18.140671+0000","last_undegraded":"2024-04-06T12:30:18.140671+0000","last_fullsized":"2024-04-06T12:30:18.140671+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:40:59.167127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.140648+0000","last_change":"2024-04-06T12:30:17.147645+0000","last_active":"2024-04-06T12:30:18.140648+0000","last_peered":"2024-04-06T12:30:18.140648+0000","last_clean":"2024-04-06T12:30:18.140648+0000","last_became_active":"2024-04-06T12:30:17.147389+0000","last_became_peered":"2024-04-06T12:30:17.147389+0000","last_unstale":"2024-04-06T12:30:18.140648+0000","last_undegraded":"2024-04-06T12:30:18.140648+0000","last_fullsized":"2024-04-06T12:30:18.140648+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:13:11.187491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434608+0000","last_change":"2024-04-06T12:30:17.155373+0000","last_active":"2024-04-06T12:30:18.434608+0000","last_peered":"2024-04-06T12:30:18.434608+0000","last_clean":"2024-04-06T12:30:18.434608+0000","last_became_active":"2024-04-06T12:30:17.150651+0000","last_became_peered":"2024-04-06T12:30:17.150651+0000","last_unstale":"2024-04-06T12:30:18.434608+0000","last_undegraded":"2024-04-06T12:30:18.434608+0000","last_fullsized":"2024-04-06T12:30:18.434608+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:02:38.211510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.158155+0000","last_change":"2024-04-06T12:30:17.156612+0000","last_active":"2024-04-06T12:30:17.158155+0000","last_peered":"2024-04-06T12:30:17.158155+0000","last_clean":"2024-04-06T12:30:17.158155+0000","last_became_active":"2024-04-06T12:30:17.156205+0000","last_became_peered":"2024-04-06T12:30:17.156205+0000","last_unstale":"2024-04-06T12:30:17.158155+0000","last_undegraded":"2024-04-06T12:30:17.158155+0000","last_fullsized":"2024-04-06T12:30:17.158155+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:42:08.059315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434495+0000","last_change":"2024-04-06T12:30:17.157461+0000","last_active":"2024-04-06T12:30:18.434495+0000","last_peered":"2024-04-06T12:30:18.434495+0000","last_clean":"2024-04-06T12:30:18.434495+0000","last_became_active":"2024-04-06T12:30:17.155996+0000","last_became_peered":"2024-04-06T12:30:17.155996+0000","last_unstale":"2024-04-06T12:30:18.434495+0000","last_undegraded":"2024-04-06T12:30:18.434495+0000","last_fullsized":"2024-04-06T12:30:18.434495+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:56:33.370573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136216+0000","last_change":"2024-04-06T12:30:17.144941+0000","last_active":"2024-04-06T12:30:18.136216+0000","last_peered":"2024-04-06T12:30:18.136216+0000","last_clean":"2024-04-06T12:30:18.136216+0000","last_became_active":"2024-04-06T12:30:17.144669+0000","last_became_peered":"2024-04-06T12:30:17.144669+0000","last_unstale":"2024-04-06T12:30:18.136216+0000","last_undegraded":"2024-04-06T12:30:18.136216+0000","last_fullsized":"2024-04-06T12:30:18.136216+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:13:11.187491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.372078+0000","last_change":"2024-04-06T12:30:17.149388+0000","last_active":"2024-04-06T12:30:19.372078+0000","last_peered":"2024-04-06T12:30:19.372078+0000","last_clean":"2024-04-06T12:30:19.372078+0000","last_became_active":"2024-04-06T12:30:17.149094+0000","last_became_peered":"2024-04-06T12:30:17.149094+0000","last_unstale":"2024-04-06T12:30:19.372078+0000","last_undegraded":"2024-04-06T12:30:19.372078+0000","last_fullsized":"2024-04-06T12:30:19.372078+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:45:49.479601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.160924+0000","last_change":"2024-04-06T12:30:17.140583+0000","last_active":"2024-04-06T12:30:17.160924+0000","last_peered":"2024-04-06T12:30:17.160924+0000","last_clean":"2024-04-06T12:30:17.160924+0000","last_became_active":"2024-04-06T12:30:17.140314+0000","last_became_peered":"2024-04-06T12:30:17.140314+0000","last_unstale":"2024-04-06T12:30:17.160924+0000","last_undegraded":"2024-04-06T12:30:17.160924+0000","last_fullsized":"2024-04-06T12:30:17.160924+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:13:15.814692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434421+0000","last_change":"2024-04-06T12:30:17.139552+0000","last_active":"2024-04-06T12:30:18.434421+0000","last_peered":"2024-04-06T12:30:18.434421+0000","last_clean":"2024-04-06T12:30:18.434421+0000","last_became_active":"2024-04-06T12:30:17.139264+0000","last_became_peered":"2024-04-06T12:30:17.139264+0000","last_unstale":"2024-04-06T12:30:18.434421+0000","last_undegraded":"2024-04-06T12:30:18.434421+0000","last_fullsized":"2024-04-06T12:30:18.434421+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:22:06.600431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.161007+0000","last_change":"2024-04-06T12:30:17.147290+0000","last_active":"2024-04-06T12:30:17.161007+0000","last_peered":"2024-04-06T12:30:17.161007+0000","last_clean":"2024-04-06T12:30:17.161007+0000","last_became_active":"2024-04-06T12:30:17.147009+0000","last_became_peered":"2024-04-06T12:30:17.147009+0000","last_unstale":"2024-04-06T12:30:17.161007+0000","last_undegraded":"2024-04-06T12:30:17.161007+0000","last_fullsized":"2024-04-06T12:30:17.161007+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:25:08.630322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136007+0000","last_change":"2024-04-06T12:30:17.144462+0000","last_active":"2024-04-06T12:30:18.136007+0000","last_peered":"2024-04-06T12:30:18.136007+0000","last_clean":"2024-04-06T12:30:18.136007+0000","last_became_active":"2024-04-06T12:30:17.144166+0000","last_became_peered":"2024-04-06T12:30:17.144166+0000","last_unstale":"2024-04-06T12:30:18.136007+0000","last_undegraded":"2024-04-06T12:30:18.136007+0000","last_fullsized":"2024-04-06T12:30:18.136007+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:28:04.728040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.137772+0000","last_change":"2024-04-06T12:30:17.154176+0000","last_active":"2024-04-06T12:30:18.137772+0000","last_peered":"2024-04-06T12:30:18.137772+0000","last_clean":"2024-04-06T12:30:18.137772+0000","last_became_active":"2024-04-06T12:30:17.153918+0000","last_became_peered":"2024-04-06T12:30:17.153918+0000","last_unstale":"2024-04-06T12:30:18.137772+0000","last_undegraded":"2024-04-06T12:30:18.137772+0000","last_fullsized":"2024-04-06T12:30:18.137772+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:44:50.175971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.155713+0000","last_change":"2024-04-06T12:30:17.152888+0000","last_active":"2024-04-06T12:30:17.155713+0000","last_peered":"2024-04-06T12:30:17.155713+0000","last_clean":"2024-04-06T12:30:17.155713+0000","last_became_active":"2024-04-06T12:30:17.152616+0000","last_became_peered":"2024-04-06T12:30:17.152616+0000","last_unstale":"2024-04-06T12:30:17.155713+0000","last_undegraded":"2024-04-06T12:30:17.155713+0000","last_fullsized":"2024-04-06T12:30:17.155713+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:29:16.578210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.155781+0000","last_change":"2024-04-06T12:30:17.148375+0000","last_active":"2024-04-06T12:30:17.155781+0000","last_peered":"2024-04-06T12:30:17.155781+0000","last_clean":"2024-04-06T12:30:17.155781+0000","last_became_active":"2024-04-06T12:30:17.147516+0000","last_became_peered":"2024-04-06T12:30:17.147516+0000","last_unstale":"2024-04-06T12:30:17.155781+0000","last_undegraded":"2024-04-06T12:30:17.155781+0000","last_fullsized":"2024-04-06T12:30:17.155781+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:03:40.459030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.372001+0000","last_change":"2024-04-06T12:30:17.150886+0000","last_active":"2024-04-06T12:30:19.372001+0000","last_peered":"2024-04-06T12:30:19.372001+0000","last_clean":"2024-04-06T12:30:19.372001+0000","last_became_active":"2024-04-06T12:30:17.150616+0000","last_became_peered":"2024-04-06T12:30:17.150616+0000","last_unstale":"2024-04-06T12:30:19.372001+0000","last_undegraded":"2024-04-06T12:30:19.372001+0000","last_fullsized":"2024-04-06T12:30:19.372001+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:29:46.115525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.135858+0000","last_change":"2024-04-06T12:30:17.147614+0000","last_active":"2024-04-06T12:30:18.135858+0000","last_peered":"2024-04-06T12:30:18.135858+0000","last_clean":"2024-04-06T12:30:18.135858+0000","last_became_active":"2024-04-06T12:30:17.147313+0000","last_became_peered":"2024-04-06T12:30:17.147313+0000","last_unstale":"2024-04-06T12:30:18.135858+0000","last_undegraded":"2024-04-06T12:30:18.135858+0000","last_fullsized":"2024-04-06T12:30:18.135858+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:12:50.047089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.154833+0000","last_change":"2024-04-06T12:30:17.147735+0000","last_active":"2024-04-06T12:30:17.154833+0000","last_peered":"2024-04-06T12:30:17.154833+0000","last_clean":"2024-04-06T12:30:17.154833+0000","last_became_active":"2024-04-06T12:30:17.147396+0000","last_became_peered":"2024-04-06T12:30:17.147396+0000","last_unstale":"2024-04-06T12:30:17.154833+0000","last_undegraded":"2024-04-06T12:30:17.154833+0000","last_fullsized":"2024-04-06T12:30:17.154833+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:29:46.115525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371721+0000","last_change":"2024-04-06T12:30:17.153878+0000","last_active":"2024-04-06T12:30:19.371721+0000","last_peered":"2024-04-06T12:30:19.371721+0000","last_clean":"2024-04-06T12:30:19.371721+0000","last_became_active":"2024-04-06T12:30:17.153627+0000","last_became_peered":"2024-04-06T12:30:17.153627+0000","last_unstale":"2024-04-06T12:30:19.371721+0000","last_undegraded":"2024-04-06T12:30:19.371721+0000","last_fullsized":"2024-04-06T12:30:19.371721+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:03:40.459030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434601+0000","last_change":"2024-04-06T12:30:17.157737+0000","last_active":"2024-04-06T12:30:18.434601+0000","last_peered":"2024-04-06T12:30:18.434601+0000","last_clean":"2024-04-06T12:30:18.434601+0000","last_became_active":"2024-04-06T12:30:17.155307+0000","last_became_peered":"2024-04-06T12:30:17.155307+0000","last_unstale":"2024-04-06T12:30:18.434601+0000","last_undegraded":"2024-04-06T12:30:18.434601+0000","last_fullsized":"2024-04-06T12:30:18.434601+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:43:58.535609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"127'4","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.137218+0000","last_change":"2024-04-06T12:30:16.740767+0000","last_active":"2024-04-06T12:30:18.137218+0000","last_peered":"2024-04-06T12:30:18.137218+0000","last_clean":"2024-04-06T12:30:18.137218+0000","last_became_active":"2024-04-06T12:30:16.740438+0000","last_became_peered":"2024-04-06T12:30:16.740438+0000","last_unstale":"2024-04-06T12:30:18.137218+0000","last_undegraded":"2024-04-06T12:30:18.137218+0000","last_fullsized":"2024-04-06T12:30:18.137218+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:49:46.781626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371778+0000","last_change":"2024-04-06T12:30:17.162532+0000","last_active":"2024-04-06T12:30:19.371778+0000","last_peered":"2024-04-06T12:30:19.371778+0000","last_clean":"2024-04-06T12:30:19.371778+0000","last_became_active":"2024-04-06T12:30:17.162281+0000","last_became_peered":"2024-04-06T12:30:17.162281+0000","last_unstale":"2024-04-06T12:30:19.371778+0000","last_undegraded":"2024-04-06T12:30:19.371778+0000","last_fullsized":"2024-04-06T12:30:19.371778+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:30:54.593855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"127'1","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.372098+0000","last_change":"2024-04-06T12:30:16.729681+0000","last_active":"2024-04-06T12:30:19.372098+0000","last_peered":"2024-04-06T12:30:19.372098+0000","last_clean":"2024-04-06T12:30:19.372098+0000","last_became_active":"2024-04-06T12:30:16.729420+0000","last_became_peered":"2024-04-06T12:30:16.729420+0000","last_unstale":"2024-04-06T12:30:19.372098+0000","last_undegraded":"2024-04-06T12:30:19.372098+0000","last_fullsized":"2024-04-06T12:30:19.372098+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:08:20.306799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136005+0000","last_change":"2024-04-06T12:30:17.157150+0000","last_active":"2024-04-06T12:30:18.136005+0000","last_peered":"2024-04-06T12:30:18.136005+0000","last_clean":"2024-04-06T12:30:18.136005+0000","last_became_active":"2024-04-06T12:30:17.156889+0000","last_became_peered":"2024-04-06T12:30:17.156889+0000","last_unstale":"2024-04-06T12:30:18.136005+0000","last_undegraded":"2024-04-06T12:30:18.136005+0000","last_fullsized":"2024-04-06T12:30:18.136005+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:09:21.777123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"127'2","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136241+0000","last_change":"2024-04-06T12:30:16.738561+0000","last_active":"2024-04-06T12:30:18.136241+0000","last_peered":"2024-04-06T12:30:18.136241+0000","last_clean":"2024-04-06T12:30:18.136241+0000","last_became_active":"2024-04-06T12:30:16.738213+0000","last_became_peered":"2024-04-06T12:30:16.738213+0000","last_unstale":"2024-04-06T12:30:18.136241+0000","last_undegraded":"2024-04-06T12:30:18.136241+0000","last_fullsized":"2024-04-06T12:30:18.136241+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:34:14.692966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.143840+0000","last_change":"2024-04-06T12:30:17.142648+0000","last_active":"2024-04-06T12:30:17.143840+0000","last_peered":"2024-04-06T12:30:17.143840+0000","last_clean":"2024-04-06T12:30:17.143840+0000","last_became_active":"2024-04-06T12:30:17.142100+0000","last_became_peered":"2024-04-06T12:30:17.142100+0000","last_unstale":"2024-04-06T12:30:17.143840+0000","last_undegraded":"2024-04-06T12:30:17.143840+0000","last_fullsized":"2024-04-06T12:30:17.143840+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:03:40.459030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"127'2","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.154167+0000","last_change":"2024-04-06T12:30:16.739257+0000","last_active":"2024-04-06T12:30:17.154167+0000","last_peered":"2024-04-06T12:30:17.154167+0000","last_clean":"2024-04-06T12:30:17.154167+0000","last_became_active":"2024-04-06T12:30:16.738995+0000","last_became_peered":"2024-04-06T12:30:16.738995+0000","last_unstale":"2024-04-06T12:30:17.154167+0000","last_undegraded":"2024-04-06T12:30:17.154167+0000","last_fullsized":"2024-04-06T12:30:17.154167+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:34:14.692966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.137043+0000","last_change":"2024-04-06T12:30:17.148019+0000","last_active":"2024-04-06T12:30:18.137043+0000","last_peered":"2024-04-06T12:30:18.137043+0000","last_clean":"2024-04-06T12:30:18.137043+0000","last_became_active":"2024-04-06T12:30:17.147699+0000","last_became_peered":"2024-04-06T12:30:17.147699+0000","last_unstale":"2024-04-06T12:30:18.137043+0000","last_undegraded":"2024-04-06T12:30:18.137043+0000","last_fullsized":"2024-04-06T12:30:18.137043+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:20:39.925631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"127'1","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.140804+0000","last_change":"2024-04-06T12:30:16.737577+0000","last_active":"2024-04-06T12:30:18.140804+0000","last_peered":"2024-04-06T12:30:18.140804+0000","last_clean":"2024-04-06T12:30:18.140804+0000","last_became_active":"2024-04-06T12:30:16.737305+0000","last_became_peered":"2024-04-06T12:30:16.737305+0000","last_unstale":"2024-04-06T12:30:18.140804+0000","last_undegraded":"2024-04-06T12:30:18.140804+0000","last_fullsized":"2024-04-06T12:30:18.140804+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:32:52.686846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434615+0000","last_change":"2024-04-06T12:30:17.156240+0000","last_active":"2024-04-06T12:30:18.434615+0000","last_peered":"2024-04-06T12:30:18.434615+0000","last_clean":"2024-04-06T12:30:18.434615+0000","last_became_active":"2024-04-06T12:30:17.155684+0000","last_became_peered":"2024-04-06T12:30:17.155684+0000","last_unstale":"2024-04-06T12:30:18.434615+0000","last_undegraded":"2024-04-06T12:30:18.434615+0000","last_fullsized":"2024-04-06T12:30:18.434615+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:54:02.871601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"127'1","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.153013+0000","last_change":"2024-04-06T12:30:16.740797+0000","last_active":"2024-04-06T12:30:17.153013+0000","last_peered":"2024-04-06T12:30:17.153013+0000","last_clean":"2024-04-06T12:30:17.153013+0000","last_became_active":"2024-04-06T12:30:16.740512+0000","last_became_peered":"2024-04-06T12:30:16.740512+0000","last_unstale":"2024-04-06T12:30:17.153013+0000","last_undegraded":"2024-04-06T12:30:17.153013+0000","last_fullsized":"2024-04-06T12:30:17.153013+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:33:52.101095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.154933+0000","last_change":"2024-04-06T12:30:17.148187+0000","last_active":"2024-04-06T12:30:17.154933+0000","last_peered":"2024-04-06T12:30:17.154933+0000","last_clean":"2024-04-06T12:30:17.154933+0000","last_became_active":"2024-04-06T12:30:17.147906+0000","last_became_peered":"2024-04-06T12:30:17.147906+0000","last_unstale":"2024-04-06T12:30:17.154933+0000","last_undegraded":"2024-04-06T12:30:17.154933+0000","last_fullsized":"2024-04-06T12:30:17.154933+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:30:54.593855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136040+0000","last_change":"2024-04-06T12:30:16.737571+0000","last_active":"2024-04-06T12:30:18.136040+0000","last_peered":"2024-04-06T12:30:18.136040+0000","last_clean":"2024-04-06T12:30:18.136040+0000","last_became_active":"2024-04-06T12:30:16.737232+0000","last_became_peered":"2024-04-06T12:30:16.737232+0000","last_unstale":"2024-04-06T12:30:18.136040+0000","last_undegraded":"2024-04-06T12:30:18.136040+0000","last_fullsized":"2024-04-06T12:30:18.136040+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:32:52.686846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.137174+0000","last_change":"2024-04-06T12:30:17.144655+0000","last_active":"2024-04-06T12:30:18.137174+0000","last_peered":"2024-04-06T12:30:18.137174+0000","last_clean":"2024-04-06T12:30:18.137174+0000","last_became_active":"2024-04-06T12:30:17.144365+0000","last_became_peered":"2024-04-06T12:30:17.144365+0000","last_unstale":"2024-04-06T12:30:18.137174+0000","last_undegraded":"2024-04-06T12:30:18.137174+0000","last_fullsized":"2024-04-06T12:30:18.137174+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:10:01.003876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136794+0000","last_change":"2024-04-06T12:30:16.729844+0000","last_active":"2024-04-06T12:30:18.136794+0000","last_peered":"2024-04-06T12:30:18.136794+0000","last_clean":"2024-04-06T12:30:18.136794+0000","last_became_active":"2024-04-06T12:30:16.729596+0000","last_became_peered":"2024-04-06T12:30:16.729596+0000","last_unstale":"2024-04-06T12:30:18.136794+0000","last_undegraded":"2024-04-06T12:30:18.136794+0000","last_fullsized":"2024-04-06T12:30:18.136794+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:09:20.770188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434498+0000","last_change":"2024-04-06T12:30:17.157952+0000","last_active":"2024-04-06T12:30:18.434498+0000","last_peered":"2024-04-06T12:30:18.434498+0000","last_clean":"2024-04-06T12:30:18.434498+0000","last_became_active":"2024-04-06T12:30:17.157452+0000","last_became_peered":"2024-04-06T12:30:17.157452+0000","last_unstale":"2024-04-06T12:30:18.434498+0000","last_undegraded":"2024-04-06T12:30:18.434498+0000","last_fullsized":"2024-04-06T12:30:18.434498+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:17:50.436516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136112+0000","last_change":"2024-04-06T12:30:16.738077+0000","last_active":"2024-04-06T12:30:18.136112+0000","last_peered":"2024-04-06T12:30:18.136112+0000","last_clean":"2024-04-06T12:30:18.136112+0000","last_became_active":"2024-04-06T12:30:16.737826+0000","last_became_peered":"2024-04-06T12:30:16.737826+0000","last_unstale":"2024-04-06T12:30:18.136112+0000","last_undegraded":"2024-04-06T12:30:18.136112+0000","last_fullsized":"2024-04-06T12:30:18.136112+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:08:44.968098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.161269+0000","last_change":"2024-04-06T12:30:17.151460+0000","last_active":"2024-04-06T12:30:17.161269+0000","last_peered":"2024-04-06T12:30:17.161269+0000","last_clean":"2024-04-06T12:30:17.161269+0000","last_became_active":"2024-04-06T12:30:17.151231+0000","last_became_peered":"2024-04-06T12:30:17.151231+0000","last_unstale":"2024-04-06T12:30:17.161269+0000","last_undegraded":"2024-04-06T12:30:17.161269+0000","last_fullsized":"2024-04-06T12:30:17.161269+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:44:50.175971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434415+0000","last_change":"2024-04-06T12:30:16.739301+0000","last_active":"2024-04-06T12:30:18.434415+0000","last_peered":"2024-04-06T12:30:18.434415+0000","last_clean":"2024-04-06T12:30:18.434415+0000","last_became_active":"2024-04-06T12:30:16.738988+0000","last_became_peered":"2024-04-06T12:30:16.738988+0000","last_unstale":"2024-04-06T12:30:18.434415+0000","last_undegraded":"2024-04-06T12:30:18.434415+0000","last_fullsized":"2024-04-06T12:30:18.434415+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:47:39.607805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371807+0000","last_change":"2024-04-06T12:30:17.162362+0000","last_active":"2024-04-06T12:30:19.371807+0000","last_peered":"2024-04-06T12:30:19.371807+0000","last_clean":"2024-04-06T12:30:19.371807+0000","last_became_active":"2024-04-06T12:30:17.162106+0000","last_became_peered":"2024-04-06T12:30:17.162106+0000","last_unstale":"2024-04-06T12:30:19.371807+0000","last_undegraded":"2024-04-06T12:30:19.371807+0000","last_fullsized":"2024-04-06T12:30:19.371807+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:02:03.887605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"127'3","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.140828+0000","last_change":"2024-04-06T12:30:16.740347+0000","last_active":"2024-04-06T12:30:18.140828+0000","last_peered":"2024-04-06T12:30:18.140828+0000","last_clean":"2024-04-06T12:30:18.140828+0000","last_became_active":"2024-04-06T12:30:16.740008+0000","last_became_peered":"2024-04-06T12:30:16.740008+0000","last_unstale":"2024-04-06T12:30:18.140828+0000","last_undegraded":"2024-04-06T12:30:18.140828+0000","last_fullsized":"2024-04-06T12:30:18.140828+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:42:07.052380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.161458+0000","last_change":"2024-04-06T12:30:17.151748+0000","last_active":"2024-04-06T12:30:17.161458+0000","last_peered":"2024-04-06T12:30:17.161458+0000","last_clean":"2024-04-06T12:30:17.161458+0000","last_became_active":"2024-04-06T12:30:17.151427+0000","last_became_peered":"2024-04-06T12:30:17.151427+0000","last_unstale":"2024-04-06T12:30:17.161458+0000","last_undegraded":"2024-04-06T12:30:17.161458+0000","last_fullsized":"2024-04-06T12:30:17.161458+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:09:21.777123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"127'6","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.161089+0000","last_change":"2024-04-06T12:30:16.731358+0000","last_active":"2024-04-06T12:30:17.161089+0000","last_peered":"2024-04-06T12:30:17.161089+0000","last_clean":"2024-04-06T12:30:17.161089+0000","last_became_active":"2024-04-06T12:30:16.731050+0000","last_became_peered":"2024-04-06T12:30:16.731050+0000","last_unstale":"2024-04-06T12:30:17.161089+0000","last_undegraded":"2024-04-06T12:30:17.161089+0000","last_fullsized":"2024-04-06T12:30:17.161089+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:59:12.117978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.154473+0000","last_change":"2024-04-06T12:30:17.146501+0000","last_active":"2024-04-06T12:30:17.154473+0000","last_peered":"2024-04-06T12:30:17.154473+0000","last_clean":"2024-04-06T12:30:17.154473+0000","last_became_active":"2024-04-06T12:30:17.146211+0000","last_became_peered":"2024-04-06T12:30:17.146211+0000","last_unstale":"2024-04-06T12:30:17.154473+0000","last_undegraded":"2024-04-06T12:30:17.154473+0000","last_fullsized":"2024-04-06T12:30:17.154473+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:52:01.384680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"127'2","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.161390+0000","last_change":"2024-04-06T12:30:16.738690+0000","last_active":"2024-04-06T12:30:17.161390+0000","last_peered":"2024-04-06T12:30:17.161390+0000","last_clean":"2024-04-06T12:30:17.161390+0000","last_became_active":"2024-04-06T12:30:16.738430+0000","last_became_peered":"2024-04-06T12:30:16.738430+0000","last_unstale":"2024-04-06T12:30:17.161390+0000","last_undegraded":"2024-04-06T12:30:17.161390+0000","last_fullsized":"2024-04-06T12:30:17.161390+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:52:00.377745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.155531+0000","last_change":"2024-04-06T12:30:17.148161+0000","last_active":"2024-04-06T12:30:17.155531+0000","last_peered":"2024-04-06T12:30:17.155531+0000","last_clean":"2024-04-06T12:30:17.155531+0000","last_became_active":"2024-04-06T12:30:17.147812+0000","last_became_peered":"2024-04-06T12:30:17.147812+0000","last_unstale":"2024-04-06T12:30:17.155531+0000","last_undegraded":"2024-04-06T12:30:17.155531+0000","last_fullsized":"2024-04-06T12:30:17.155531+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:02:56.674896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136913+0000","last_change":"2024-04-06T12:30:16.733345+0000","last_active":"2024-04-06T12:30:18.136913+0000","last_peered":"2024-04-06T12:30:18.136913+0000","last_clean":"2024-04-06T12:30:18.136913+0000","last_became_active":"2024-04-06T12:30:16.733088+0000","last_became_peered":"2024-04-06T12:30:16.733088+0000","last_unstale":"2024-04-06T12:30:18.136913+0000","last_undegraded":"2024-04-06T12:30:18.136913+0000","last_fullsized":"2024-04-06T12:30:18.136913+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:25:07.623387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.372034+0000","last_change":"2024-04-06T12:30:17.153701+0000","last_active":"2024-04-06T12:30:19.372034+0000","last_peered":"2024-04-06T12:30:19.372034+0000","last_clean":"2024-04-06T12:30:19.372034+0000","last_became_active":"2024-04-06T12:30:17.153454+0000","last_became_peered":"2024-04-06T12:30:17.153454+0000","last_unstale":"2024-04-06T12:30:19.372034+0000","last_undegraded":"2024-04-06T12:30:19.372034+0000","last_fullsized":"2024-04-06T12:30:19.372034+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:17:26.384789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371825+0000","last_change":"2024-04-06T12:30:16.739584+0000","last_active":"2024-04-06T12:30:19.371825+0000","last_peered":"2024-04-06T12:30:19.371825+0000","last_clean":"2024-04-06T12:30:19.371825+0000","last_became_active":"2024-04-06T12:30:16.739233+0000","last_became_peered":"2024-04-06T12:30:16.739233+0000","last_unstale":"2024-04-06T12:30:19.371825+0000","last_undegraded":"2024-04-06T12:30:19.371825+0000","last_fullsized":"2024-04-06T12:30:19.371825+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:45:40.445595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434614+0000","last_change":"2024-04-06T12:30:17.155200+0000","last_active":"2024-04-06T12:30:18.434614+0000","last_peered":"2024-04-06T12:30:18.434614+0000","last_clean":"2024-04-06T12:30:18.434614+0000","last_became_active":"2024-04-06T12:30:17.150303+0000","last_became_peered":"2024-04-06T12:30:17.150303+0000","last_unstale":"2024-04-06T12:30:18.434614+0000","last_undegraded":"2024-04-06T12:30:18.434614+0000","last_fullsized":"2024-04-06T12:30:18.434614+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:50:28.054904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"127'8","reported_seq":19,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.135833+0000","last_change":"2024-04-06T12:30:16.736060+0000","last_active":"2024-04-06T12:30:18.135833+0000","last_peered":"2024-04-06T12:30:18.135833+0000","last_clean":"2024-04-06T12:30:18.135833+0000","last_became_active":"2024-04-06T12:30:16.733937+0000","last_became_peered":"2024-04-06T12:30:16.733937+0000","last_unstale":"2024-04-06T12:30:18.135833+0000","last_undegraded":"2024-04-06T12:30:18.135833+0000","last_fullsized":"2024-04-06T12:30:18.135833+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:42:07.052380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371854+0000","last_change":"2024-04-06T12:30:17.158840+0000","last_active":"2024-04-06T12:30:19.371854+0000","last_peered":"2024-04-06T12:30:19.371854+0000","last_clean":"2024-04-06T12:30:19.371854+0000","last_became_active":"2024-04-06T12:30:17.158386+0000","last_became_peered":"2024-04-06T12:30:17.158386+0000","last_unstale":"2024-04-06T12:30:19.371854+0000","last_undegraded":"2024-04-06T12:30:19.371854+0000","last_fullsized":"2024-04-06T12:30:19.371854+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:32:53.693781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"127'1","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.153178+0000","last_change":"2024-04-06T12:30:16.738809+0000","last_active":"2024-04-06T12:30:17.153178+0000","last_peered":"2024-04-06T12:30:17.153178+0000","last_clean":"2024-04-06T12:30:17.153178+0000","last_became_active":"2024-04-06T12:30:16.738547+0000","last_became_peered":"2024-04-06T12:30:16.738547+0000","last_unstale":"2024-04-06T12:30:17.153178+0000","last_undegraded":"2024-04-06T12:30:17.153178+0000","last_fullsized":"2024-04-06T12:30:17.153178+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:10:48.781671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.155152+0000","last_change":"2024-04-06T12:30:17.149891+0000","last_active":"2024-04-06T12:30:17.155152+0000","last_peered":"2024-04-06T12:30:17.155152+0000","last_clean":"2024-04-06T12:30:17.155152+0000","last_became_active":"2024-04-06T12:30:17.147638+0000","last_became_peered":"2024-04-06T12:30:17.147638+0000","last_unstale":"2024-04-06T12:30:17.155152+0000","last_undegraded":"2024-04-06T12:30:17.155152+0000","last_fullsized":"2024-04-06T12:30:17.155152+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:52:01.384680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"127'1","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.140949+0000","last_change":"2024-04-06T12:30:16.738834+0000","last_active":"2024-04-06T12:30:18.140949+0000","last_peered":"2024-04-06T12:30:18.140949+0000","last_clean":"2024-04-06T12:30:18.140949+0000","last_became_active":"2024-04-06T12:30:16.738548+0000","last_became_peered":"2024-04-06T12:30:16.738548+0000","last_unstale":"2024-04-06T12:30:18.140949+0000","last_undegraded":"2024-04-06T12:30:18.140949+0000","last_fullsized":"2024-04-06T12:30:18.140949+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:34:14.692966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.155787+0000","last_change":"2024-04-06T12:30:17.141293+0000","last_active":"2024-04-06T12:30:17.155787+0000","last_peered":"2024-04-06T12:30:17.155787+0000","last_clean":"2024-04-06T12:30:17.155787+0000","last_became_active":"2024-04-06T12:30:17.141041+0000","last_became_peered":"2024-04-06T12:30:17.141041+0000","last_unstale":"2024-04-06T12:30:17.155787+0000","last_undegraded":"2024-04-06T12:30:17.155787+0000","last_fullsized":"2024-04-06T12:30:17.155787+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:28:04.728040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":10,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.139118+0000","last_change":"2024-04-06T12:30:16.733035+0000","last_active":"2024-04-06T12:30:17.139118+0000","last_peered":"2024-04-06T12:30:17.139118+0000","last_clean":"2024-04-06T12:30:17.139118+0000","last_became_active":"2024-04-06T12:30:16.732648+0000","last_became_peered":"2024-04-06T12:30:16.732648+0000","last_unstale":"2024-04-06T12:30:17.139118+0000","last_undegraded":"2024-04-06T12:30:17.139118+0000","last_fullsized":"2024-04-06T12:30:17.139118+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:32:58.290093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434555+0000","last_change":"2024-04-06T12:30:17.150419+0000","last_active":"2024-04-06T12:30:18.434555+0000","last_peered":"2024-04-06T12:30:18.434555+0000","last_clean":"2024-04-06T12:30:18.434555+0000","last_became_active":"2024-04-06T12:30:17.149955+0000","last_became_peered":"2024-04-06T12:30:17.149955+0000","last_unstale":"2024-04-06T12:30:18.434555+0000","last_undegraded":"2024-04-06T12:30:18.434555+0000","last_fullsized":"2024-04-06T12:30:18.434555+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:05:02.037200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"127'2","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.372079+0000","last_change":"2024-04-06T12:30:16.740234+0000","last_active":"2024-04-06T12:30:19.372079+0000","last_peered":"2024-04-06T12:30:19.372079+0000","last_clean":"2024-04-06T12:30:19.372079+0000","last_became_active":"2024-04-06T12:30:16.739973+0000","last_became_peered":"2024-04-06T12:30:16.739973+0000","last_unstale":"2024-04-06T12:30:19.372079+0000","last_undegraded":"2024-04-06T12:30:19.372079+0000","last_fullsized":"2024-04-06T12:30:19.372079+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:45:54.102151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434487+0000","last_change":"2024-04-06T12:30:17.161559+0000","last_active":"2024-04-06T12:30:18.434487+0000","last_peered":"2024-04-06T12:30:18.434487+0000","last_clean":"2024-04-06T12:30:18.434487+0000","last_became_active":"2024-04-06T12:30:17.161296+0000","last_became_peered":"2024-04-06T12:30:17.161296+0000","last_unstale":"2024-04-06T12:30:18.434487+0000","last_undegraded":"2024-04-06T12:30:18.434487+0000","last_fullsized":"2024-04-06T12:30:18.434487+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:50:43.712066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371778+0000","last_change":"2024-04-06T12:30:16.736815+0000","last_active":"2024-04-06T12:30:19.371778+0000","last_peered":"2024-04-06T12:30:19.371778+0000","last_clean":"2024-04-06T12:30:19.371778+0000","last_became_active":"2024-04-06T12:30:16.736563+0000","last_became_peered":"2024-04-06T12:30:16.736563+0000","last_unstale":"2024-04-06T12:30:19.371778+0000","last_undegraded":"2024-04-06T12:30:19.371778+0000","last_fullsized":"2024-04-06T12:30:19.371778+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:42:37.653575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136913+0000","last_change":"2024-04-06T12:30:17.155123+0000","last_active":"2024-04-06T12:30:18.136913+0000","last_peered":"2024-04-06T12:30:18.136913+0000","last_clean":"2024-04-06T12:30:18.136913+0000","last_became_active":"2024-04-06T12:30:17.154910+0000","last_became_peered":"2024-04-06T12:30:17.154910+0000","last_unstale":"2024-04-06T12:30:18.136913+0000","last_undegraded":"2024-04-06T12:30:18.136913+0000","last_fullsized":"2024-04-06T12:30:18.136913+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:43:22.296771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":10,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.134252+0000","last_change":"2024-04-06T12:30:16.737460+0000","last_active":"2024-04-06T12:30:17.134252+0000","last_peered":"2024-04-06T12:30:17.134252+0000","last_clean":"2024-04-06T12:30:17.134252+0000","last_became_active":"2024-04-06T12:30:16.737210+0000","last_became_peered":"2024-04-06T12:30:16.737210+0000","last_unstale":"2024-04-06T12:30:17.134252+0000","last_undegraded":"2024-04-06T12:30:17.134252+0000","last_fullsized":"2024-04-06T12:30:17.134252+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:51:43.578634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.372205+0000","last_change":"2024-04-06T12:30:17.161742+0000","last_active":"2024-04-06T12:30:19.372205+0000","last_peered":"2024-04-06T12:30:19.372205+0000","last_clean":"2024-04-06T12:30:19.372205+0000","last_became_active":"2024-04-06T12:30:17.161515+0000","last_became_peered":"2024-04-06T12:30:17.161515+0000","last_unstale":"2024-04-06T12:30:19.372205+0000","last_undegraded":"2024-04-06T12:30:19.372205+0000","last_fullsized":"2024-04-06T12:30:19.372205+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:52:01.384680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371895+0000","last_change":"2024-04-06T12:30:16.741843+0000","last_active":"2024-04-06T12:30:19.371895+0000","last_peered":"2024-04-06T12:30:19.371895+0000","last_clean":"2024-04-06T12:30:19.371895+0000","last_became_active":"2024-04-06T12:30:16.741604+0000","last_became_peered":"2024-04-06T12:30:16.741604+0000","last_unstale":"2024-04-06T12:30:19.371895+0000","last_undegraded":"2024-04-06T12:30:19.371895+0000","last_fullsized":"2024-04-06T12:30:19.371895+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:12:30.025007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434679+0000","last_change":"2024-04-06T12:30:17.150538+0000","last_active":"2024-04-06T12:30:18.434679+0000","last_peered":"2024-04-06T12:30:18.434679+0000","last_clean":"2024-04-06T12:30:18.434679+0000","last_became_active":"2024-04-06T12:30:17.150275+0000","last_became_peered":"2024-04-06T12:30:17.150275+0000","last_unstale":"2024-04-06T12:30:18.434679+0000","last_undegraded":"2024-04-06T12:30:18.434679+0000","last_fullsized":"2024-04-06T12:30:18.434679+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:11:36.447101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"127'2","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.154685+0000","last_change":"2024-04-06T12:30:16.736237+0000","last_active":"2024-04-06T12:30:17.154685+0000","last_peered":"2024-04-06T12:30:17.154685+0000","last_clean":"2024-04-06T12:30:17.154685+0000","last_became_active":"2024-04-06T12:30:16.735966+0000","last_became_peered":"2024-04-06T12:30:16.735966+0000","last_unstale":"2024-04-06T12:30:17.154685+0000","last_undegraded":"2024-04-06T12:30:17.154685+0000","last_fullsized":"2024-04-06T12:30:17.154685+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:32:52.686846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.135875+0000","last_change":"2024-04-06T12:30:17.146535+0000","last_active":"2024-04-06T12:30:18.135875+0000","last_peered":"2024-04-06T12:30:18.135875+0000","last_clean":"2024-04-06T12:30:18.135875+0000","last_became_active":"2024-04-06T12:30:17.146260+0000","last_became_peered":"2024-04-06T12:30:17.146260+0000","last_unstale":"2024-04-06T12:30:18.135875+0000","last_undegraded":"2024-04-06T12:30:18.135875+0000","last_fullsized":"2024-04-06T12:30:18.135875+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:59:28.638138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.434421+0000","last_change":"2024-04-06T12:30:16.741338+0000","last_active":"2024-04-06T12:30:18.434421+0000","last_peered":"2024-04-06T12:30:18.434421+0000","last_clean":"2024-04-06T12:30:18.434421+0000","last_became_active":"2024-04-06T12:30:16.740513+0000","last_became_peered":"2024-04-06T12:30:16.740513+0000","last_unstale":"2024-04-06T12:30:18.434421+0000","last_undegraded":"2024-04-06T12:30:18.434421+0000","last_fullsized":"2024-04-06T12:30:18.434421+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:02:37.869473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136805+0000","last_change":"2024-04-06T12:30:17.153799+0000","last_active":"2024-04-06T12:30:18.136805+0000","last_peered":"2024-04-06T12:30:18.136805+0000","last_clean":"2024-04-06T12:30:18.136805+0000","last_became_active":"2024-04-06T12:30:17.153509+0000","last_became_peered":"2024-04-06T12:30:17.153509+0000","last_unstale":"2024-04-06T12:30:18.136805+0000","last_undegraded":"2024-04-06T12:30:18.136805+0000","last_fullsized":"2024-04-06T12:30:18.136805+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:20:09.324436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":10,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.137735+0000","last_change":"2024-04-06T12:30:16.734086+0000","last_active":"2024-04-06T12:30:17.137735+0000","last_peered":"2024-04-06T12:30:17.137735+0000","last_clean":"2024-04-06T12:30:17.137735+0000","last_became_active":"2024-04-06T12:30:16.733829+0000","last_became_peered":"2024-04-06T12:30:16.733829+0000","last_unstale":"2024-04-06T12:30:17.137735+0000","last_undegraded":"2024-04-06T12:30:17.137735+0000","last_fullsized":"2024-04-06T12:30:17.137735+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:24:24.304885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.157532+0000","last_change":"2024-04-06T12:30:17.156559+0000","last_active":"2024-04-06T12:30:17.157532+0000","last_peered":"2024-04-06T12:30:17.157532+0000","last_clean":"2024-04-06T12:30:17.157532+0000","last_became_active":"2024-04-06T12:30:17.156304+0000","last_became_peered":"2024-04-06T12:30:17.156304+0000","last_unstale":"2024-04-06T12:30:17.157532+0000","last_undegraded":"2024-04-06T12:30:17.157532+0000","last_fullsized":"2024-04-06T12:30:17.157532+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:29:46.115525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":10,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.131174+0000","last_change":"2024-04-06T12:30:16.738521+0000","last_active":"2024-04-06T12:30:17.131174+0000","last_peered":"2024-04-06T12:30:17.131174+0000","last_clean":"2024-04-06T12:30:17.131174+0000","last_became_active":"2024-04-06T12:30:16.738245+0000","last_became_peered":"2024-04-06T12:30:16.738245+0000","last_unstale":"2024-04-06T12:30:17.131174+0000","last_undegraded":"2024-04-06T12:30:17.131174+0000","last_fullsized":"2024-04-06T12:30:17.131174+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:29:45.108590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.372011+0000","last_change":"2024-04-06T12:30:17.161738+0000","last_active":"2024-04-06T12:30:19.372011+0000","last_peered":"2024-04-06T12:30:19.372011+0000","last_clean":"2024-04-06T12:30:19.372011+0000","last_became_active":"2024-04-06T12:30:17.161484+0000","last_became_peered":"2024-04-06T12:30:17.161484+0000","last_unstale":"2024-04-06T12:30:19.372011+0000","last_undegraded":"2024-04-06T12:30:19.372011+0000","last_fullsized":"2024-04-06T12:30:19.372011+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:02:56.674896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.136113+0000","last_change":"2024-04-06T12:30:16.738491+0000","last_active":"2024-04-06T12:30:18.136113+0000","last_peered":"2024-04-06T12:30:18.136113+0000","last_clean":"2024-04-06T12:30:18.136113+0000","last_became_active":"2024-04-06T12:30:16.738227+0000","last_became_peered":"2024-04-06T12:30:16.738227+0000","last_unstale":"2024-04-06T12:30:18.136113+0000","last_undegraded":"2024-04-06T12:30:18.136113+0000","last_fullsized":"2024-04-06T12:30:18.136113+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:29:45.108590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.372011+0000","last_change":"2024-04-06T12:30:17.162535+0000","last_active":"2024-04-06T12:30:19.372011+0000","last_peered":"2024-04-06T12:30:19.372011+0000","last_clean":"2024-04-06T12:30:19.372011+0000","last_became_active":"2024-04-06T12:30:17.161329+0000","last_became_peered":"2024-04-06T12:30:17.161329+0000","last_unstale":"2024-04-06T12:30:19.372011+0000","last_undegraded":"2024-04-06T12:30:19.372011+0000","last_fullsized":"2024-04-06T12:30:19.372011+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:11:54.910628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371888+0000","last_change":"2024-04-06T12:30:16.739555+0000","last_active":"2024-04-06T12:30:19.371888+0000","last_peered":"2024-04-06T12:30:19.371888+0000","last_clean":"2024-04-06T12:30:19.371888+0000","last_became_active":"2024-04-06T12:30:16.739275+0000","last_became_peered":"2024-04-06T12:30:16.739275+0000","last_unstale":"2024-04-06T12:30:19.371888+0000","last_undegraded":"2024-04-06T12:30:19.371888+0000","last_fullsized":"2024-04-06T12:30:19.371888+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:04:55.999480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:18.140925+0000","last_change":"2024-04-06T12:30:17.140057+0000","last_active":"2024-04-06T12:30:18.140925+0000","last_peered":"2024-04-06T12:30:18.140925+0000","last_clean":"2024-04-06T12:30:18.140925+0000","last_became_active":"2024-04-06T12:30:17.139574+0000","last_became_peered":"2024-04-06T12:30:17.139574+0000","last_unstale":"2024-04-06T12:30:18.140925+0000","last_undegraded":"2024-04-06T12:30:18.140925+0000","last_fullsized":"2024-04-06T12:30:18.140925+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:06:36.859191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":10,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.134365+0000","last_change":"2024-04-06T12:30:16.742015+0000","last_active":"2024-04-06T12:30:17.134365+0000","last_peered":"2024-04-06T12:30:17.134365+0000","last_clean":"2024-04-06T12:30:17.134365+0000","last_became_active":"2024-04-06T12:30:16.741717+0000","last_became_peered":"2024-04-06T12:30:16.741717+0000","last_unstale":"2024-04-06T12:30:17.134365+0000","last_undegraded":"2024-04-06T12:30:17.134365+0000","last_fullsized":"2024-04-06T12:30:17.134365+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:23:36.754305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.154910+0000","last_change":"2024-04-06T12:30:17.147343+0000","last_active":"2024-04-06T12:30:17.154910+0000","last_peered":"2024-04-06T12:30:17.154910+0000","last_clean":"2024-04-06T12:30:17.154910+0000","last_became_active":"2024-04-06T12:30:17.147093+0000","last_became_peered":"2024-04-06T12:30:17.147093+0000","last_unstale":"2024-04-06T12:30:17.154910+0000","last_undegraded":"2024-04-06T12:30:17.154910+0000","last_fullsized":"2024-04-06T12:30:17.154910+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:30:54.593855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.371951+0000","last_change":"2024-04-06T12:30:16.737778+0000","last_active":"2024-04-06T12:30:19.371951+0000","last_peered":"2024-04-06T12:30:19.371951+0000","last_clean":"2024-04-06T12:30:19.371951+0000","last_became_active":"2024-04-06T12:30:16.737562+0000","last_became_peered":"2024-04-06T12:30:16.737562+0000","last_unstale":"2024-04-06T12:30:19.371951+0000","last_undegraded":"2024-04-06T12:30:19.371951+0000","last_fullsized":"2024-04-06T12:30:19.371951+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:35.534922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.154148+0000","last_change":"2024-04-06T12:30:17.146320+0000","last_active":"2024-04-06T12:30:17.154148+0000","last_peered":"2024-04-06T12:30:17.154148+0000","last_clean":"2024-04-06T12:30:17.154148+0000","last_became_active":"2024-04-06T12:30:17.146053+0000","last_became_peered":"2024-04-06T12:30:17.146053+0000","last_unstale":"2024-04-06T12:30:17.154148+0000","last_undegraded":"2024-04-06T12:30:17.154148+0000","last_fullsized":"2024-04-06T12:30:17.154148+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:16.716721+0000","last_clean_scrub_stamp":"2024-04-06T12:30:16.716721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:02:56.674896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"127'2","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-06T12:30:19.372128+0000","last_change":"2024-04-06T12:30:16.735892+0000","last_active":"2024-04-06T12:30:19.372128+0000","last_peered":"2024-04-06T12:30:19.372128+0000","last_clean":"2024-04-06T12:30:19.372128+0000","last_became_active":"2024-04-06T12:30:16.735628+0000","last_became_peered":"2024-04-06T12:30:16.735628+0000","last_unstale":"2024-04-06T12:30:19.372128+0000","last_undegraded":"2024-04-06T12:30:19.372128+0000","last_fullsized":"2024-04-06T12:30:19.372128+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:30:15.709785+0000","last_clean_scrub_stamp":"2024-04-06T12:30:15.709785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:06:07.991018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"117'205","reported_seq":593,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-06T12:30:17.131557+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:30:17.131557+0000","last_peered":"2024-04-06T12:30:17.131557+0000","last_clean":"2024-04-06T12:30:17.131557+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:30:17.131557+0000","last_undegraded":"2024-04-06T12:30:17.131557+0000","last_fullsized":"2024-04-06T12:30:17.131557+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575066,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59500,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312340,"statfs":{"total":96636764160,"available":96575836160,"internally_reserved":0,"allocated":2187264,"data_stored":1383350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53200000000000003,"15min":0.55800000000000005},"min":{"1min":0.311,"5min":0.28899999999999998,"15min":0.219},"max":{"1min":0.75600000000000001,"5min":1.143,"15min":1.143},"last":0.628},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.52500000000000002,"15min":0.55400000000000005},"min":{"1min":0.37,"5min":0.246,"15min":0.246},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.501}]},{"osd":1,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53800000000000003,"15min":0.55700000000000005},"min":{"1min":0.40000000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.81399999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.57199999999999995},"min":{"1min":0.41199999999999998,"5min":0.34300000000000003,"15min":0.27900000000000003},"max":{"1min":0.70499999999999996,"5min":0.77400000000000002,"15min":0.79600000000000004},"last":0.54800000000000004}]},{"osd":2,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52600000000000002,"15min":0.55700000000000005},"min":{"1min":0.38600000000000001,"5min":0.30599999999999999,"15min":0.249},"max":{"1min":0.69899999999999995,"5min":0.77100000000000002,"15min":0.83299999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.55800000000000005,"15min":0.58399999999999996},"min":{"1min":0.42899999999999999,"5min":0.34499999999999997,"15min":0.29599999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.60099999999999998}]},{"osd":3,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55800000000000005,"15min":0.57599999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.34399999999999997},"max":{"1min":0.69299999999999995,"5min":1.357,"15min":1.357},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.57399999999999995,"15min":0.59299999999999997},"min":{"1min":0.46200000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.72999999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.53800000000000003}]},{"osd":4,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45400000000000001,"15min":0.44400000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.25},"max":{"1min":0.63600000000000001,"5min":0.71399999999999997,"15min":0.72999999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53700000000000003,"15min":0.52600000000000002},"min":{"1min":0.373,"5min":0.34300000000000003,"15min":0.32800000000000001},"max":{"1min":0.74199999999999999,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.57399999999999995}]},{"osd":5,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.50700000000000001,"15min":0.496},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.23799999999999999},"max":{"1min":0.80500000000000005,"5min":0.81899999999999995,"15min":0.86799999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.29299999999999998},"max":{"1min":0.73799999999999999,"5min":0.81599999999999995,"15min":0.83499999999999996},"last":0.58999999999999997}]},{"osd":6,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54900000000000004,"15min":0.53800000000000003},"min":{"1min":0.36799999999999999,"5min":0.33300000000000002,"15min":0.28799999999999998},"max":{"1min":0.76400000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.33100000000000002,"5min":0.312,"15min":0.30399999999999999},"max":{"1min":0.82199999999999995,"5min":0.86299999999999999,"15min":0.88500000000000001},"last":0.51700000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575065,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32676,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339164,"statfs":{"total":96636764160,"available":96603303936,"internally_reserved":0,"allocated":1589248,"data_stored":787351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.52500000000000002,"15min":0.54000000000000004},"min":{"1min":0.40999999999999998,"5min":0.33700000000000002,"15min":0.26200000000000001},"max":{"1min":0.73199999999999998,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52500000000000002,"15min":0.53800000000000003},"min":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.245},"max":{"1min":0.754,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.47599999999999998}]},{"osd":1,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56699999999999995,"15min":0.59099999999999997},"min":{"1min":0.46300000000000002,"5min":0.32700000000000001,"15min":0.24399999999999999},"max":{"1min":0.73899999999999999,"5min":0.78700000000000003,"15min":0.84999999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58299999999999996,"15min":0.60099999999999998},"min":{"1min":0.41599999999999998,"5min":0.30199999999999999,"15min":0.221},"max":{"1min":0.72699999999999998,"5min":0.81999999999999995,"15min":0.83099999999999996},"last":0.55500000000000005}]},{"osd":2,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52800000000000002,"15min":0.55400000000000005},"min":{"1min":0.38100000000000001,"5min":0.30499999999999999,"15min":0.29899999999999999},"max":{"1min":0.71399999999999997,"5min":0.749,"15min":0.82399999999999995},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55200000000000005,"15min":0.57399999999999995},"min":{"1min":0.43099999999999999,"5min":0.32300000000000001,"15min":0.254},"max":{"1min":0.76900000000000002,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.40999999999999998}]},{"osd":3,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.56399999999999995,"15min":0.58299999999999996},"min":{"1min":0.41399999999999998,"5min":0.317,"15min":0.23300000000000001},"max":{"1min":0.73199999999999998,"5min":0.77300000000000002,"15min":0.81200000000000006},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59899999999999998,"15min":0.621},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.374},"max":{"1min":0.76200000000000001,"5min":0.82799999999999996,"15min":0.872},"last":0.61799999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.502,"15min":0.496},"min":{"1min":0.35999999999999999,"5min":0.28599999999999998,"15min":0.27500000000000002},"max":{"1min":0.66900000000000004,"5min":0.754,"15min":0.754},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.52900000000000003,"15min":0.52800000000000002},"min":{"1min":0.317,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72799999999999998,"5min":0.73199999999999998,"15min":0.82199999999999995},"last":0.56799999999999995}]},{"osd":5,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.49199999999999999,"15min":0.48899999999999999},"min":{"1min":0.30099999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.74099999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.32400000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.80200000000000005,"15min":0.83699999999999997},"last":0.46100000000000002}]},{"osd":7,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54000000000000004,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76500000000000001,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56699999999999995,"15min":0.55800000000000005},"min":{"1min":0.38900000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80700000000000005,"5min":0.84799999999999998,"15min":0.88900000000000001},"last":0.51000000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575064,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1572864,"data_stored":785217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:29:32 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.36399999999999999,"15min":0.36099999999999999},"min":{"1min":0.25,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.52800000000000002,"5min":0.53400000000000003,"15min":0.58999999999999997},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.37,"15min":0.39300000000000002},"min":{"1min":0.251,"5min":0.215,"15min":0.215},"max":{"1min":0.56399999999999995,"5min":0.628,"15min":0.628},"last":0.32700000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:29:32 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38700000000000001,"15min":0.39600000000000002},"min":{"1min":0.27000000000000002,"5min":0.23899999999999999,"15min":0.19400000000000001},"max":{"1min":0.58099999999999996,"5min":0.61799999999999999,"15min":0.67600000000000005},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.41999999999999998,"15min":0.432},"min":{"1min":0.29899999999999999,"5min":0.255,"15min":0.255},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.66100000000000003},"last":0.36399999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:29:32 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44,"15min":0.44800000000000001},"min":{"1min":0.318,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67500000000000004,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40500000000000003,"15min":0.41899999999999998},"min":{"1min":0.29699999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.65900000000000003,"5min":0.69599999999999995,"15min":0.86299999999999999},"last":0.35199999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:29:32 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59599999999999997,"15min":0.61499999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.34799999999999998},"max":{"1min":0.78700000000000003,"5min":0.86799999999999999,"15min":0.91100000000000003},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.60499999999999998,"15min":0.61799999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.30399999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.94899999999999995},"last":0.624}]},{"osd":5,"last update":"Sat Apr 6 12:29:32 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.61199999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.38700000000000001},"max":{"1min":0.83699999999999997,"5min":0.85899999999999999,"15min":0.93500000000000005},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.622,"15min":0.628},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.38300000000000001},"max":{"1min":0.747,"5min":0.999,"15min":0.999},"last":0.54000000000000004}]},{"osd":6,"last update":"Sat Apr 6 12:29:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60199999999999998,"15min":0.61399999999999999},"min":{"1min":0.46899999999999997,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78300000000000003,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.628,"15min":0.64700000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78700000000000003,"5min":0.88700000000000001,"15min":0.97899999999999998},"last":0.52000000000000002}]},{"osd":7,"last update":"Sat Apr 6 12:29:32 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63300000000000001,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.41899999999999998,"15min":0.38800000000000001},"max":{"1min":0.81200000000000006,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.62,"15min":0.63600000000000001},"min":{"1min":0.42399999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.91400000000000003,"5min":0.95599999999999996,"15min":0.96799999999999997},"last":0.55000000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575064,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33264,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338576,"statfs":{"total":96636764160,"available":96602701824,"internally_reserved":0,"allocated":2191360,"data_stored":1383825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.376,"15min":0.38600000000000001},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":0.72699999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38900000000000001,"15min":0.40500000000000003},"min":{"1min":0.23499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.505,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.26700000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.38900000000000001,"15min":0.40899999999999997},"min":{"1min":0.28100000000000003,"5min":0.23599999999999999,"15min":0.19700000000000001},"max":{"1min":0.49099999999999999,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41999999999999998,"15min":0.432},"min":{"1min":0.28100000000000003,"5min":0.249,"15min":0.24299999999999999},"max":{"1min":0.56299999999999994,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.33900000000000002}]},{"osd":3,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.40400000000000003,"15min":0.41899999999999998},"min":{"1min":0.28799999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.53300000000000003,"5min":1.25,"15min":1.25},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.436,"15min":0.46300000000000002},"min":{"1min":0.26700000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.64200000000000002,"5min":0.74099999999999999,"15min":0.748},"last":0.39400000000000002}]},{"osd":4,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.59899999999999998,"15min":0.621},"min":{"1min":0.38500000000000001,"5min":0.371,"15min":0.35699999999999998},"max":{"1min":0.79500000000000004,"5min":1.514,"15min":1.514},"last":0.755},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.60999999999999999,"15min":0.63},"min":{"1min":0.42199999999999999,"5min":0.36699999999999999,"15min":0.35199999999999998},"max":{"1min":0.84699999999999998,"5min":1.202,"15min":1.202},"last":0.628}]},{"osd":5,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.624},"min":{"1min":0.38100000000000001,"5min":0.34399999999999997,"15min":0.33700000000000002},"max":{"1min":0.75700000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.46200000000000002,"5min":0.38800000000000001,"15min":0.315},"max":{"1min":0.82499999999999996,"5min":1.006,"15min":1.006},"last":0.61499999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.625,"15min":0.65200000000000002},"min":{"1min":0.48699999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.79000000000000004,"5min":1.038,"15min":1.038},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63600000000000001,"15min":0.66100000000000003},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.38100000000000001},"max":{"1min":0.82699999999999996,"5min":1.613,"15min":1.613},"last":0.72399999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:29:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.63900000000000001,"15min":0.66400000000000003},"min":{"1min":0.50800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80600000000000005,"5min":1.54,"15min":1.54},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.63800000000000001,"15min":0.65700000000000003},"min":{"1min":0.39300000000000002,"5min":0.38400000000000001,"15min":0.375},"max":{"1min":0.77400000000000002,"5min":1.498,"15min":1.498},"last":0.70699999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575066,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54804,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317036,"statfs":{"total":96636764160,"available":96580644864,"internally_reserved":0,"allocated":1572864,"data_stored":785217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:29:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.36599999999999999,"15min":0.38300000000000001},"min":{"1min":0.23899999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.54000000000000004,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.39200000000000002,"15min":0.41199999999999998},"min":{"1min":0.248,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.65600000000000003,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.33700000000000002}]},{"osd":1,"last update":"Sat Apr 6 12:29:29 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.38700000000000001,"15min":0.40300000000000002},"min":{"1min":0.252,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.64600000000000002,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.315},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40000000000000002,"15min":0.43099999999999999},"min":{"1min":0.28299999999999997,"5min":0.27100000000000002,"15min":0.23599999999999999},"max":{"1min":0.61899999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.28000000000000003}]},{"osd":3,"last update":"Sat Apr 6 12:29:29 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.433,"15min":0.45000000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64200000000000002,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.373},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.434,"15min":0.439},"min":{"1min":0.308,"5min":0.29499999999999998,"15min":0.29099999999999998},"max":{"1min":0.63200000000000001,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.30099999999999999}]},{"osd":4,"last update":"Sat Apr 6 12:29:29 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.58299999999999996,"15min":0.61599999999999999},"min":{"1min":0.40000000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.93300000000000005,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.58699999999999997,"15min":0.626},"min":{"1min":0.38900000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.80400000000000005,"5min":0.91200000000000003,"15min":0.92600000000000005},"last":0.52800000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:29:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.61299999999999999,"15min":0.63800000000000001},"min":{"1min":0.39700000000000002,"5min":0.374,"15min":0.36499999999999999},"max":{"1min":0.83899999999999997,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.59799999999999998,"15min":0.629},"min":{"1min":0.41399999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.98099999999999998,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.502}]},{"osd":6,"last update":"Sat Apr 6 12:29:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.60199999999999998,"15min":0.63400000000000001},"min":{"1min":0.41799999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.91800000000000004,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.627,"15min":0.65400000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":1.728,"15min":1.728},"last":0.45300000000000001}]},{"osd":7,"last update":"Sat Apr 6 12:29:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.60999999999999999,"15min":0.64800000000000002},"min":{"1min":0.46800000000000003,"5min":0.40699999999999997,"15min":0.35599999999999998},"max":{"1min":0.88900000000000001,"5min":0.95699999999999996,"15min":0.95799999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.64100000000000001,"15min":0.66400000000000003},"min":{"1min":0.42699999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.95299999999999996,"5min":1.758,"15min":1.758},"last":0.56499999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575065,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1589248,"data_stored":793404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:29:33 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36099999999999999,"15min":0.36799999999999999},"min":{"1min":0.29299999999999998,"5min":0.21099999999999999,"15min":0.191},"max":{"1min":0.46000000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36399999999999999,"15min":0.38300000000000001},"min":{"1min":0.28000000000000003,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.48899999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.35899999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:29:33 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.38400000000000001},"min":{"1min":0.26600000000000001,"5min":0.22700000000000001,"15min":0.217},"max":{"1min":0.44600000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.40300000000000002,"15min":0.41599999999999998},"min":{"1min":0.27500000000000002,"5min":0.26100000000000001,"15min":0.25700000000000001},"max":{"1min":0.49299999999999999,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.34100000000000003}]},{"osd":2,"last update":"Sat Apr 6 12:29:33 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.442,"15min":0.46200000000000002},"min":{"1min":0.28799999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59699999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.318},{"interface":"front","average":{"1min":0.442,"5min":0.46500000000000002,"15min":0.47799999999999998},"min":{"1min":0.35599999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.55700000000000005,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.38}]},{"osd":4,"last update":"Sat Apr 6 12:29:33 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57799999999999996,"15min":0.60699999999999998},"min":{"1min":0.38200000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.76100000000000001,"5min":0.88500000000000001,"15min":0.95999999999999996},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58999999999999997,"15min":0.624},"min":{"1min":0.41099999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.79500000000000004,"5min":0.90100000000000002,"15min":0.93500000000000005},"last":0.42999999999999999}]},{"osd":5,"last update":"Sat Apr 6 12:29:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.625,"15min":0.65900000000000003},"min":{"1min":0.45400000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.80300000000000005,"5min":0.92200000000000004,"15min":0.96699999999999997},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.63500000000000001,"15min":0.66200000000000003},"min":{"1min":0.44800000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.79600000000000004,"5min":0.94499999999999995,"15min":0.95599999999999996},"last":0.41499999999999998}]},{"osd":6,"last update":"Sat Apr 6 12:29:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.59699999999999998,"15min":0.624},"min":{"1min":0.40300000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77000000000000002,"5min":1.026,"15min":1.026},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.64000000000000001,"15min":0.65800000000000003},"min":{"1min":0.39200000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.83699999999999997,"5min":1.716,"15min":1.716},"last":0.58399999999999996}]},{"osd":7,"last update":"Sat Apr 6 12:29:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.64400000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.33300000000000002},"max":{"1min":0.85799999999999998,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.63,"15min":0.66300000000000003},"min":{"1min":0.41899999999999998,"5min":0.39600000000000002,"15min":0.373},"max":{"1min":0.81399999999999995,"5min":1.002,"15min":1.002},"last":0.55700000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575065,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32672,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":1593344,"data_stored":792914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:29:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54700000000000004,"15min":0.56499999999999995},"min":{"1min":0.41099999999999998,"5min":0.29399999999999998,"15min":0.23899999999999999},"max":{"1min":0.80500000000000005,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53900000000000003,"15min":0.56100000000000005},"min":{"1min":0.35499999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.45500000000000002}]},{"osd":1,"last update":"Sat Apr 6 12:29:43 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.58099999999999996,"15min":0.60899999999999999},"min":{"1min":0.42399999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.83399999999999996},"last":0.315},{"interface":"front","average":{"1min":0.62,"5min":0.57799999999999996,"15min":0.59499999999999997},"min":{"1min":0.38400000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.751,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.38700000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:29:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.55200000000000005,"15min":0.56999999999999995},"min":{"1min":0.41799999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.56299999999999994,"15min":0.58499999999999996},"min":{"1min":0.39700000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.753,"5min":0.76900000000000002,"15min":0.80900000000000005},"last":0.503}]},{"osd":3,"last update":"Sat Apr 6 12:29:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.55300000000000005,"15min":0.56299999999999994},"min":{"1min":0.38500000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76800000000000002,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.443},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58199999999999996,"15min":0.60099999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.77500000000000002,"5min":0.79600000000000004,"15min":0.80900000000000005},"last":0.47099999999999997}]},{"osd":5,"last update":"Sat Apr 6 12:29:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.46899999999999997,"15min":0.47699999999999998},"min":{"1min":0.308,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.73599999999999999,"5min":0.76300000000000001,"15min":0.83599999999999997},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.49099999999999999,"15min":0.497},"min":{"1min":0.33600000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.77000000000000002},"last":0.49299999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:29:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.37,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.70499999999999996,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52000000000000002,"15min":0.53300000000000003},"min":{"1min":0.41399999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.748,"5min":0.748,"15min":0.872},"last":0.56999999999999995}]},{"osd":7,"last update":"Sat Apr 6 12:29:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.43099999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.77600000000000002,"5min":0.79300000000000004,"15min":0.82299999999999995},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.55900000000000005,"15min":0.56699999999999995},"min":{"1min":0.47499999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.84799999999999998},"last":0.55500000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575065,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50704,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321136,"statfs":{"total":96636764160,"available":96584843264,"internally_reserved":0,"allocated":1568768,"data_stored":785183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.54000000000000004,"15min":0.57199999999999995},"min":{"1min":0.38800000000000001,"5min":0.32700000000000001,"15min":0.24199999999999999},"max":{"1min":0.81999999999999995,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52600000000000002,"15min":0.55000000000000004},"min":{"1min":0.36099999999999999,"5min":0.29499999999999998,"15min":0.22500000000000001},"max":{"1min":0.72399999999999998,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.36699999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52100000000000002,"15min":0.55600000000000005},"min":{"1min":0.36099999999999999,"5min":0.30099999999999999,"15min":0.26000000000000001},"max":{"1min":0.68600000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53800000000000003,"15min":0.55600000000000005},"min":{"1min":0.379,"5min":0.33400000000000002,"15min":0.30599999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.41799999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53600000000000003,"15min":0.56399999999999995},"min":{"1min":0.38500000000000001,"5min":0.32500000000000001,"15min":0.21099999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.62},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.54600000000000004,"15min":0.56699999999999995},"min":{"1min":0.44800000000000001,"5min":0.35999999999999999,"15min":0.27400000000000002},"max":{"1min":0.77100000000000002,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.49099999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.54900000000000004,"15min":0.57099999999999995},"min":{"1min":0.37,"5min":0.32000000000000001,"15min":0.31},"max":{"1min":0.83799999999999997,"5min":0.874,"15min":0.874},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56699999999999995,"15min":0.59499999999999997},"min":{"1min":0.41899999999999998,"5min":0.35699999999999998,"15min":0.32300000000000001},"max":{"1min":0.71199999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.51700000000000002}]},{"osd":4,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46400000000000002,"15min":0.46700000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28299999999999997},"max":{"1min":0.71799999999999997,"5min":0.755,"15min":0.81200000000000006},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53400000000000003,"15min":0.52600000000000002},"min":{"1min":0.29499999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.79900000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.437}]},{"osd":6,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47299999999999998,"15min":0.47599999999999998},"min":{"1min":0.35099999999999998,"5min":0.313,"15min":0.28000000000000003},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47999999999999998,"15min":0.48399999999999999},"min":{"1min":0.33200000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.77000000000000002},"last":0.504}]},{"osd":7,"last update":"Sat Apr 6 12:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.496,"15min":0.5},"min":{"1min":0.318,"5min":0.29799999999999999,"15min":0.252},"max":{"1min":0.753,"5min":0.78600000000000003,"15min":0.79100000000000004},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58899999999999997,"15min":0.59599999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.35199999999999998},"max":{"1min":0.83399999999999996,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.57199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:30:21.646 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-06T12:30:21.646 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-06T12:30:21.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.814+0000 7f9d3b698700 1 Processor -- start 2024-04-06T12:30:21.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.814+0000 7f9d3b698700 1 -- start start 2024-04-06T12:30:21.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.815+0000 7f9d3b698700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d34158fe0 0x7f9d3406a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.815+0000 7f9d3b698700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9d3406ac70 0x7f9d3415bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.815+0000 7f9d3b698700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d3415f9e0 con 0x7f9d3415c100 2024-04-06T12:30:21.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.815+0000 7f9d3b698700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d3406b0f0 con 0x7f9d3406ac70 2024-04-06T12:30:21.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.815+0000 7f9d3b698700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3415f0d0 con 0x7f9d34158fe0 2024-04-06T12:30:21.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.815+0000 7f9d38c33700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9d3406ac70 0x7f9d3415bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.815+0000 7f9d39c35700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9d3415c100 0x7f9d340717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45554/0 (socket says 172.21.15.189:45554) 2024-04-06T12:30:21.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.815+0000 7f9d39c35700 1 -- 172.21.15.189:0/3108267891 learned_addr learned my addr 172.21.15.189:0/3108267891 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:21.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.815+0000 7f9d39434700 1 --2- 172.21.15.189:0/3108267891 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d34158fe0 0x7f9d3406a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.815+0000 7f9d38c33700 1 -- 172.21.15.189:0/3108267891 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d34158fe0 msgr2=0x7f9d3406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:21.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.815+0000 7f9d38c33700 1 --2- 172.21.15.189:0/3108267891 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d34158fe0 0x7f9d3406a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.815+0000 7f9d38c33700 1 -- 172.21.15.189:0/3108267891 >> v1:172.21.15.189:6789/0 conn(0x7f9d3415c100 legacy=0x7f9d340717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:21.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.815+0000 7f9d2bfff700 1 -- 172.21.15.189:0/3108267891 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766225533 0 0) 0x7f9d3415f9e0 con 0x7f9d3415c100 2024-04-06T12:30:21.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.815+0000 7f9d38c33700 1 -- 172.21.15.189:0/3108267891 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3415c8f0 con 0x7f9d3406ac70 2024-04-06T12:30:21.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.815+0000 7f9d38c33700 1 --2- 172.21.15.189:0/3108267891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d3406ac70 0x7f9d3415bbc0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f9d24009840 tx=0x7f9d24009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=611e5175167416af server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:21.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.815+0000 7f9d2bfff700 1 -- 172.21.15.189:0/3108267891 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d24011830 con 0x7f9d3406ac70 2024-04-06T12:30:21.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.815+0000 7f9d2bfff700 1 -- 172.21.15.189:0/3108267891 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d24011990 con 0x7f9d3406ac70 2024-04-06T12:30:21.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.815+0000 7f9d2bfff700 1 -- 172.21.15.189:0/3108267891 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d24011d20 con 0x7f9d3406ac70 2024-04-06T12:30:21.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.816+0000 7f9d3b698700 1 -- 172.21.15.189:0/3108267891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d3406ac70 msgr2=0x7f9d3415bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:21.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.816+0000 7f9d3b698700 1 --2- 172.21.15.189:0/3108267891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d3406ac70 0x7f9d3415bbc0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f9d24009840 tx=0x7f9d24009b50 comp rx=0 tx=0).stop 2024-04-06T12:30:21.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.816+0000 7f9d3b698700 1 -- 172.21.15.189:0/3108267891 shutdown_connections 2024-04-06T12:30:21.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.816+0000 7f9d3b698700 1 --2- 172.21.15.189:0/3108267891 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d34158fe0 0x7f9d3406a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.816+0000 7f9d3b698700 1 --2- 172.21.15.189:0/3108267891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d3406ac70 0x7f9d3415bbc0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.816+0000 7f9d3b698700 1 -- 172.21.15.189:0/3108267891 >> 172.21.15.189:0/3108267891 conn(0x7f9d340d00b0 msgr2=0x7f9d340d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:21.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.816+0000 7f9d3b698700 1 -- 172.21.15.189:0/3108267891 shutdown_connections 2024-04-06T12:30:21.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.816+0000 7f9d3b698700 1 -- 172.21.15.189:0/3108267891 wait complete. 2024-04-06T12:30:21.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.817+0000 7f9d3b698700 1 Processor -- start 2024-04-06T12:30:21.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.817+0000 7f9d3b698700 1 -- start start 2024-04-06T12:30:21.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.818+0000 7f9d3b698700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d3406ac70 0x7f9d3413d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.818+0000 7f9d3b698700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d34158fe0 0x7f9d341440a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.818+0000 7f9d3b698700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d3415c100 0x7f9d34157650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.818+0000 7f9d3b698700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3406fc90 con 0x7f9d34158fe0 2024-04-06T12:30:21.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.818+0000 7f9d3b698700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d34157b90 con 0x7f9d3415c100 2024-04-06T12:30:21.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.818+0000 7f9d3b698700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3415d6c0 con 0x7f9d3406ac70 2024-04-06T12:30:21.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.818+0000 7f9d38c33700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d34158fe0 0x7f9d341440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.818+0000 7f9d38c33700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d34158fe0 0x7f9d341440a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40756/0 (socket says 172.21.15.189:40756) 2024-04-06T12:30:21.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.818+0000 7f9d38c33700 1 -- 172.21.15.189:0/490519842 learned_addr learned my addr 172.21.15.189:0/490519842 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:21.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.818+0000 7f9d39434700 1 --2- 172.21.15.189:0/490519842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d3406ac70 0x7f9d3413d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.818+0000 7f9d39c35700 1 --2- 172.21.15.189:0/490519842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d3415c100 0x7f9d34157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.818+0000 7f9d38c33700 1 -- 172.21.15.189:0/490519842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d3406ac70 msgr2=0x7f9d3413d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:21.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.818+0000 7f9d38c33700 1 --2- 172.21.15.189:0/490519842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d3406ac70 0x7f9d3413d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.818+0000 7f9d38c33700 1 -- 172.21.15.189:0/490519842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d3415c100 msgr2=0x7f9d34157650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:21.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.818+0000 7f9d38c33700 1 --2- 172.21.15.189:0/490519842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d3415c100 0x7f9d34157650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:21.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.818+0000 7f9d38c33700 1 -- 172.21.15.189:0/490519842 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d240094a0 con 0x7f9d34158fe0 2024-04-06T12:30:21.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.819+0000 7f9d38c33700 1 --2- 172.21.15.189:0/490519842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d34158fe0 0x7f9d341440a0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f9d24014040 tx=0x7f9d24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:21.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.819+0000 7f9d29ffb700 1 -- 172.21.15.189:0/490519842 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d240038a0 con 0x7f9d34158fe0 2024-04-06T12:30:21.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.819+0000 7f9d29ffb700 1 -- 172.21.15.189:0/490519842 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2401e070 con 0x7f9d34158fe0 2024-04-06T12:30:21.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.819+0000 7f9d3b698700 1 -- 172.21.15.189:0/490519842 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3415d9a0 con 0x7f9d34158fe0 2024-04-06T12:30:21.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.819+0000 7f9d29ffb700 1 -- 172.21.15.189:0/490519842 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d24003c40 con 0x7f9d34158fe0 2024-04-06T12:30:21.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.819+0000 7f9d3b698700 1 -- 172.21.15.189:0/490519842 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d3415df60 con 0x7f9d34158fe0 2024-04-06T12:30:21.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.820+0000 7f9d3b698700 1 -- 172.21.15.189:0/490519842 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d34067bb0 con 0x7f9d34158fe0 2024-04-06T12:30:21.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.821+0000 7f9d29ffb700 1 -- 172.21.15.189:0/490519842 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f9d2401a420 con 0x7f9d34158fe0 2024-04-06T12:30:21.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.821+0000 7f9d29ffb700 1 --2- 172.21.15.189:0/490519842 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9d1806adb0 0x7f9d1806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:21.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.821+0000 7f9d29ffb700 1 -- 172.21.15.189:0/490519842 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9d2408a7d0 con 0x7f9d34158fe0 2024-04-06T12:30:21.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.821+0000 7f9d39434700 1 --2- 172.21.15.189:0/490519842 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9d1806adb0 0x7f9d1806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:21.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.822+0000 7f9d39434700 1 --2- 172.21.15.189:0/490519842 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9d1806adb0 0x7f9d1806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9d3415b120 tx=0x7f9d30008040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:21.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:21.826+0000 7f9d29ffb700 1 -- 172.21.15.189:0/490519842 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9d240557f0 con 0x7f9d34158fe0 2024-04-06T12:30:22.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.098+0000 7f9d3b698700 1 -- 172.21.15.189:0/490519842 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f9d340cfcb0 con 0x7f9d34158fe0 2024-04-06T12:30:22.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.102+0000 7f9d29ffb700 1 -- 172.21.15.189:0/490519842 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+0 (secure 0 0 0) 0x7f9d240200e0 con 0x7f9d34158fe0 2024-04-06T12:30:22.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.104+0000 7f9d3b698700 1 -- 172.21.15.189:0/490519842 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9d1806adb0 msgr2=0x7f9d1806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:22.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.104+0000 7f9d3b698700 1 --2- 172.21.15.189:0/490519842 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9d1806adb0 0x7f9d1806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9d3415b120 tx=0x7f9d30008040 comp rx=0 tx=0).stop 2024-04-06T12:30:22.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.104+0000 7f9d3b698700 1 -- 172.21.15.189:0/490519842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d34158fe0 msgr2=0x7f9d341440a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:22.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.104+0000 7f9d3b698700 1 --2- 172.21.15.189:0/490519842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d34158fe0 0x7f9d341440a0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f9d24014040 tx=0x7f9d24004750 comp rx=0 tx=0).stop 2024-04-06T12:30:22.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.104+0000 7f9d3b698700 1 -- 172.21.15.189:0/490519842 shutdown_connections 2024-04-06T12:30:22.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.104+0000 7f9d3b698700 1 --2- 172.21.15.189:0/490519842 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9d1806adb0 0x7f9d1806d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:22.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.104+0000 7f9d3b698700 1 --2- 172.21.15.189:0/490519842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d3415c100 0x7f9d34157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:22.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.104+0000 7f9d3b698700 1 --2- 172.21.15.189:0/490519842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d3406ac70 0x7f9d3413d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:22.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.104+0000 7f9d3b698700 1 --2- 172.21.15.189:0/490519842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d34158fe0 0x7f9d341440a0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:22.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.104+0000 7f9d3b698700 1 -- 172.21.15.189:0/490519842 >> 172.21.15.189:0/490519842 conn(0x7f9d340d00b0 msgr2=0x7f9d3406eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:22.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.104+0000 7f9d3b698700 1 -- 172.21.15.189:0/490519842 shutdown_connections 2024-04-06T12:30:22.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.104+0000 7f9d3b698700 1 -- 172.21.15.189:0/490519842 wait complete. 2024-04-06T12:30:22.104 INFO:teuthology.orchestra.run.smithi189.stderr:updated caps for client.0 2024-04-06T12:30:22.113 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:30:22.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.239+0000 7fdfa3826700 1 Processor -- start 2024-04-06T12:30:22.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.239+0000 7fdfa3826700 1 -- start start 2024-04-06T12:30:22.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.240+0000 7fdfa3826700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdf9c0b3e20 0x7fdf9c15e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:22.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.240+0000 7fdfa3826700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9c0b3990 con 0x7fdf9c15d7b0 2024-04-06T12:30:22.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.240+0000 7fdfa3826700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9c0b3060 con 0x7fdf9c06ae40 2024-04-06T12:30:22.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.240+0000 7fdfa3826700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf9c0b4230 con 0x7fdf9c0b3e20 2024-04-06T12:30:22.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.240+0000 7fdfa15c2700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdf9c15d7b0 0x7fdf9c06a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45566/0 (socket says 172.21.15.189:45566) 2024-04-06T12:30:22.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.240+0000 7fdfa15c2700 1 -- 172.21.15.189:0/2205778073 learned_addr learned my addr 172.21.15.189:0/2205778073 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:22.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.240+0000 7fdfa1dc3700 1 --2- 172.21.15.189:0/2205778073 >> v2:172.21.15.192:3300/0 conn(0x7fdf9c0b3e20 0x7fdf9c15e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:22.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.240+0000 7fdf93fff700 1 -- 172.21.15.189:0/2205778073 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338841859 0 0) 0x7fdf9c0b3990 con 0x7fdf9c15d7b0 2024-04-06T12:30:22.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.240+0000 7fdf93fff700 1 -- 172.21.15.189:0/2205778073 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf880035b0 con 0x7fdf9c15d7b0 2024-04-06T12:30:22.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.240+0000 7fdf93fff700 1 -- 172.21.15.189:0/2205778073 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460774925 0 0) 0x7fdf9c0b3060 con 0x7fdf9c06ae40 2024-04-06T12:30:22.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.240+0000 7fdf93fff700 1 -- 172.21.15.189:0/2205778073 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf9c0b3990 con 0x7fdf9c06ae40 2024-04-06T12:30:22.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.240+0000 7fdf93fff700 1 -- 172.21.15.189:0/2205778073 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1884395252 0 0) 0x7fdf880035b0 con 0x7fdf9c15d7b0 2024-04-06T12:30:22.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.240+0000 7fdf93fff700 1 -- 172.21.15.189:0/2205778073 >> v2:172.21.15.192:3300/0 conn(0x7fdf9c0b3e20 msgr2=0x7fdf9c15e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:22.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdf93fff700 1 --2- 172.21.15.189:0/2205778073 >> v2:172.21.15.192:3300/0 conn(0x7fdf9c0b3e20 0x7fdf9c15e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:22.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdf93fff700 1 -- 172.21.15.189:0/2205778073 >> v1:172.21.15.192:6789/0 conn(0x7fdf9c06ae40 legacy=0x7fdf9c15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:22.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdf93fff700 1 -- 172.21.15.189:0/2205778073 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9c0b3340 con 0x7fdf9c15d7b0 2024-04-06T12:30:22.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdf93fff700 1 -- 172.21.15.189:0/2205778073 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fdf98001670 con 0x7fdf9c15d7b0 2024-04-06T12:30:22.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdf93fff700 1 -- 172.21.15.189:0/2205778073 >> v1:172.21.15.189:6789/0 conn(0x7fdf9c15d7b0 legacy=0x7fdf9c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:22.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdf93fff700 1 --2- 172.21.15.189:0/2205778073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf88003be0 0x7fdf88003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:22.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdf93fff700 1 --2- 172.21.15.189:0/2205778073 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf88004850 0x7fdf88004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:22.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdf93fff700 1 --2- 172.21.15.189:0/2205778073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf88005260 0x7fdf8800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:22.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdf93fff700 1 -- 172.21.15.189:0/2205778073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c0b4230 con 0x7fdf88003be0 2024-04-06T12:30:22.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdf93fff700 1 -- 172.21.15.189:0/2205778073 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8800c190 con 0x7fdf88005260 2024-04-06T12:30:22.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdf93fff700 1 -- 172.21.15.189:0/2205778073 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8800c350 con 0x7fdf88004850 2024-04-06T12:30:22.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdfa1dc3700 1 --2- 172.21.15.189:0/2205778073 >> v2:172.21.15.192:3300/0 conn(0x7fdf9c0b3e20 0x7fdf9c15e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:30:22.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdfa15c2700 1 --2- 172.21.15.189:0/2205778073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf88003be0 0x7fdf88003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:22.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdfa0dc1700 1 --2- 172.21.15.189:0/2205778073 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf88004850 0x7fdf88004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:22.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdfa1dc3700 1 --2- 172.21.15.189:0/2205778073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf88005260 0x7fdf8800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:22.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdfa15c2700 1 -- 172.21.15.189:0/2205778073 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf88004850 msgr2=0x7fdf88004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:22.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdfa15c2700 1 --2- 172.21.15.189:0/2205778073 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf88004850 0x7fdf88004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:22.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdfa15c2700 1 -- 172.21.15.189:0/2205778073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf88005260 msgr2=0x7fdf8800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:22.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdfa15c2700 1 --2- 172.21.15.189:0/2205778073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf88005260 0x7fdf8800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:22.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdfa15c2700 1 -- 172.21.15.189:0/2205778073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf880035b0 con 0x7fdf88003be0 2024-04-06T12:30:22.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdfa1dc3700 1 --2- 172.21.15.189:0/2205778073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf88005260 0x7fdf8800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:30:22.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdfa0dc1700 1 --2- 172.21.15.189:0/2205778073 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf88004850 0x7fdf88004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:22.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdfa1dc3700 1 -- 172.21.15.189:0/2205778073 reap_dead start 2024-04-06T12:30:22.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdfa15c2700 1 --2- 172.21.15.189:0/2205778073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf88003be0 0x7fdf88003fe0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fdf9800c160 tx=0x7fdf88005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=612bcce900862245 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:22.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdf93fff700 1 -- 172.21.15.189:0/2205778073 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9800cca0 con 0x7fdf88003be0 2024-04-06T12:30:22.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdf93fff700 1 -- 172.21.15.189:0/2205778073 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9800ce20 con 0x7fdf88003be0 2024-04-06T12:30:22.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.241+0000 7fdf93fff700 1 -- 172.21.15.189:0/2205778073 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf98005ba0 con 0x7fdf88003be0 2024-04-06T12:30:22.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.242+0000 7fdfa3826700 1 -- 172.21.15.189:0/2205778073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf88003be0 msgr2=0x7fdf88003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:22.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.242+0000 7fdfa3826700 1 --2- 172.21.15.189:0/2205778073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf88003be0 0x7fdf88003fe0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fdf9800c160 tx=0x7fdf88005880 comp rx=0 tx=0).stop 2024-04-06T12:30:22.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.242+0000 7fdfa3826700 1 -- 172.21.15.189:0/2205778073 shutdown_connections 2024-04-06T12:30:22.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.242+0000 7fdfa3826700 1 --2- 172.21.15.189:0/2205778073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf88003be0 0x7fdf88003fe0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:22.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.242+0000 7fdfa3826700 1 -- 172.21.15.189:0/2205778073 >> 172.21.15.189:0/2205778073 conn(0x7fdf9c0d00b0 msgr2=0x7fdf9c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:22.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.242+0000 7fdfa3826700 1 -- 172.21.15.189:0/2205778073 shutdown_connections 2024-04-06T12:30:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.242+0000 7fdfa3826700 1 -- 172.21.15.189:0/2205778073 wait complete. 2024-04-06T12:30:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.243+0000 7fdfa3826700 1 Processor -- start 2024-04-06T12:30:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.243+0000 7fdfa3826700 1 -- start start 2024-04-06T12:30:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.244+0000 7fdfa3826700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf88003be0 0x7fdf9c0cc480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.244+0000 7fdfa3826700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf9c0b16c0 0x7fdf9c14e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.244+0000 7fdfa3826700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf9c0ce600 0x7fdf9c0cea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.244+0000 7fdfa3826700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c0cefc0 con 0x7fdf9c0ce600 2024-04-06T12:30:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.244+0000 7fdfa3826700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c0cde00 con 0x7fdf88003be0 2024-04-06T12:30:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.244+0000 7fdfa3826700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c0ccd60 con 0x7fdf9c0b16c0 2024-04-06T12:30:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.244+0000 7fdfa1dc3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf9c0ce600 0x7fdf9c0cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.244+0000 7fdfa1dc3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf9c0ce600 0x7fdf9c0cea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40772/0 (socket says 172.21.15.189:40772) 2024-04-06T12:30:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.244+0000 7fdfa1dc3700 1 -- 172.21.15.189:0/2064655229 learned_addr learned my addr 172.21.15.189:0/2064655229 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.244+0000 7fdfa0dc1700 1 --2- 172.21.15.189:0/2064655229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf9c0b16c0 0x7fdf9c14e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.244+0000 7fdfa15c2700 1 --2- 172.21.15.189:0/2064655229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf88003be0 0x7fdf9c0cc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.244+0000 7fdfa1dc3700 1 -- 172.21.15.189:0/2064655229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf9c0b16c0 msgr2=0x7fdf9c14e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.244+0000 7fdfa1dc3700 1 --2- 172.21.15.189:0/2064655229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf9c0b16c0 0x7fdf9c14e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.244+0000 7fdfa1dc3700 1 -- 172.21.15.189:0/2064655229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf88003be0 msgr2=0x7fdf9c0cc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.244+0000 7fdfa1dc3700 1 --2- 172.21.15.189:0/2064655229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf88003be0 0x7fdf9c0cc480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.244+0000 7fdfa1dc3700 1 -- 172.21.15.189:0/2064655229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9c0cd040 con 0x7fdf9c0ce600 2024-04-06T12:30:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.244+0000 7fdfa15c2700 1 --2- 172.21.15.189:0/2064655229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf88003be0 0x7fdf9c0cc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.244+0000 7fdfa1dc3700 1 --2- 172.21.15.189:0/2064655229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf9c0ce600 0x7fdf9c0cea80 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0b4800 tx=0x7fdf9400f660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.244+0000 7fdf91ffb700 1 -- 172.21.15.189:0/2064655229 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf94009ea0 con 0x7fdf9c0ce600 2024-04-06T12:30:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.245+0000 7fdf91ffb700 1 -- 172.21.15.189:0/2064655229 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf94003a20 con 0x7fdf9c0ce600 2024-04-06T12:30:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.245+0000 7fdf91ffb700 1 -- 172.21.15.189:0/2064655229 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf94003dd0 con 0x7fdf9c0ce600 2024-04-06T12:30:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.245+0000 7fdfa3826700 1 -- 172.21.15.189:0/2064655229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf9c0cd320 con 0x7fdf9c0ce600 2024-04-06T12:30:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.245+0000 7fdfa3826700 1 -- 172.21.15.189:0/2064655229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf9c0cd8e0 con 0x7fdf9c0ce600 2024-04-06T12:30:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.246+0000 7fdfa3826700 1 -- 172.21.15.189:0/2064655229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf9c067bb0 con 0x7fdf9c0ce600 2024-04-06T12:30:22.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.246+0000 7fdf91ffb700 1 -- 172.21.15.189:0/2064655229 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fdf94004030 con 0x7fdf9c0ce600 2024-04-06T12:30:22.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.246+0000 7fdf91ffb700 1 --2- 172.21.15.189:0/2064655229 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdf8006ade0 0x7fdf8006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:22.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.247+0000 7fdf91ffb700 1 -- 172.21.15.189:0/2064655229 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fdf9408a020 con 0x7fdf9c0ce600 2024-04-06T12:30:22.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.247+0000 7fdfa15c2700 1 --2- 172.21.15.189:0/2064655229 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdf8006ade0 0x7fdf8006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:22.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.247+0000 7fdfa15c2700 1 --2- 172.21.15.189:0/2064655229 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdf8006ade0 0x7fdf8006d2a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdf98000c00 tx=0x7fdf98007000 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:22.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.251+0000 7fdf91ffb700 1 -- 172.21.15.189:0/2064655229 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdf94054e10 con 0x7fdf9c0ce600 2024-04-06T12:30:22.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.487+0000 7fdfa3826700 1 -- 172.21.15.189:0/2064655229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf9c074990 con 0x7fdf9c0ce600 2024-04-06T12:30:22.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.488+0000 7fdf91ffb700 1 -- 172.21.15.189:0/2064655229 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4739 (secure 0 0 0) 0x7fdf9401c020 con 0x7fdf9c0ce600 2024-04-06T12:30:22.485 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:30:22.485 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/749266547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":749266547},{"type":"v1","addr":"172.21.15.192:6835","nonce":749266547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":6764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/947183051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":947183051},{"type":"v1","addr":"172.21.15.192:6834","nonce":947183051}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7492,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3990051444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3990051444},{"type":"v1","addr":"172.21.15.189:6837","nonce":3990051444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:30:17.127295+0000","modified":"2024-04-06T12:30:20.144458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8088},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8088":{"gid":8088,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/393232044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":393232044},{"type":"v1","addr":"172.21.15.189:6836","nonce":393232044}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-06T12:30:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.490+0000 7fdfa3826700 1 -- 172.21.15.189:0/2064655229 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdf8006ade0 msgr2=0x7fdf8006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.490+0000 7fdfa3826700 1 --2- 172.21.15.189:0/2064655229 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdf8006ade0 0x7fdf8006d2a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdf98000c00 tx=0x7fdf98007000 comp rx=0 tx=0).stop 2024-04-06T12:30:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.490+0000 7fdfa3826700 1 -- 172.21.15.189:0/2064655229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf9c0ce600 msgr2=0x7fdf9c0cea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.490+0000 7fdfa3826700 1 --2- 172.21.15.189:0/2064655229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf9c0ce600 0x7fdf9c0cea80 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0b4800 tx=0x7fdf9400f660 comp rx=0 tx=0).stop 2024-04-06T12:30:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.490+0000 7fdfa3826700 1 -- 172.21.15.189:0/2064655229 shutdown_connections 2024-04-06T12:30:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.490+0000 7fdfa3826700 1 --2- 172.21.15.189:0/2064655229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf9c0ce600 0x7fdf9c0cea80 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.490+0000 7fdfa3826700 1 --2- 172.21.15.189:0/2064655229 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdf8006ade0 0x7fdf8006d2a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.490+0000 7fdfa3826700 1 --2- 172.21.15.189:0/2064655229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf88003be0 0x7fdf9c0cc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.490+0000 7fdfa3826700 1 --2- 172.21.15.189:0/2064655229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf9c0b16c0 0x7fdf9c14e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.490+0000 7fdfa3826700 1 -- 172.21.15.189:0/2064655229 >> 172.21.15.189:0/2064655229 conn(0x7fdf9c0d00b0 msgr2=0x7fdf9c15e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.490+0000 7fdfa3826700 1 -- 172.21.15.189:0/2064655229 shutdown_connections 2024-04-06T12:30:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:22.490+0000 7fdfa3826700 1 -- 172.21.15.189:0/2064655229 wait complete. 2024-04-06T12:30:22.489 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 54 2024-04-06T12:30:22.705 INFO:tasks.ceph.mds.c.smithi189.stderr:2024-04-06T12:30:22.708+0000 7f6a24e1c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T12:30:22.903 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 54, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-06T12:30:17.127295+0000', 'modified': '2024-04-06T12:30:20.144458+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8088}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8088': {'gid': 8088, 'name': 'c', 'rank': 0, 'incarnation': 50, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.189:6836/393232044', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 393232044}, {'type': 'v1', 'addr': '172.21.15.189:6836', 'nonce': 393232044}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-06T12:30:22.903 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-06T12:30:22.903 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-06T12:30:23.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.030+0000 7f9989c42700 1 Processor -- start 2024-04-06T12:30:23.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.030+0000 7f9989c42700 1 -- start start 2024-04-06T12:30:23.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.031+0000 7f9989c42700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f998415ef60 0x7f99840cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.031+0000 7f9989c42700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99840cb850 con 0x7f9984070370 2024-04-06T12:30:23.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.031+0000 7f9989c42700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9984158220 con 0x7f998406b840 2024-04-06T12:30:23.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.031+0000 7f9989c42700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998406bfb0 con 0x7f998415ef60 2024-04-06T12:30:23.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.031+0000 7f99837fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9984070370 0x7f9984070750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45568/0 (socket says 172.21.15.189:45568) 2024-04-06T12:30:23.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.031+0000 7f99837fe700 1 -- 172.21.15.189:0/2101993626 learned_addr learned my addr 172.21.15.189:0/2101993626 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:23.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.031+0000 7f9983fff700 1 --2- 172.21.15.189:0/2101993626 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f998415ef60 0x7f99840cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.031+0000 7f99827fc700 1 -- 172.21.15.189:0/2101993626 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156421399 0 0) 0x7f99840cb850 con 0x7f9984070370 2024-04-06T12:30:23.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.031+0000 7f99827fc700 1 -- 172.21.15.189:0/2101993626 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99700035b0 con 0x7f9984070370 2024-04-06T12:30:23.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99827fc700 1 -- 172.21.15.189:0/2101993626 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147035273 0 0) 0x7f9984158220 con 0x7f998406b840 2024-04-06T12:30:23.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99827fc700 1 -- 172.21.15.189:0/2101993626 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99840cb850 con 0x7f998406b840 2024-04-06T12:30:23.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99827fc700 1 -- 172.21.15.189:0/2101993626 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1917966726 0 0) 0x7f99700035b0 con 0x7f9984070370 2024-04-06T12:30:23.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99827fc700 1 -- 172.21.15.189:0/2101993626 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f998415ef60 msgr2=0x7f99840cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:23.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99827fc700 1 --2- 172.21.15.189:0/2101993626 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f998415ef60 0x7f99840cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99827fc700 1 -- 172.21.15.189:0/2101993626 >> v1:172.21.15.192:6789/0 conn(0x7f998406b840 legacy=0x7f998415e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:23.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99827fc700 1 -- 172.21.15.189:0/2101993626 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99840cbaf0 con 0x7f9984070370 2024-04-06T12:30:23.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99827fc700 1 -- 172.21.15.189:0/2101993626 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f996c001670 con 0x7f9984070370 2024-04-06T12:30:23.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99827fc700 1 -- 172.21.15.189:0/2101993626 >> v1:172.21.15.189:6789/0 conn(0x7f9984070370 legacy=0x7f9984070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:23.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99827fc700 1 --2- 172.21.15.189:0/2101993626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9970003bc0 0x7f9970003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99827fc700 1 --2- 172.21.15.189:0/2101993626 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f998415ef60 0x7f9970004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99827fc700 1 --2- 172.21.15.189:0/2101993626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9970004d90 0x7f9970005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99827fc700 1 -- 172.21.15.189:0/2101993626 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998406bfb0 con 0x7f9970004d90 2024-04-06T12:30:23.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99827fc700 1 -- 172.21.15.189:0/2101993626 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997000bba0 con 0x7f9970003bc0 2024-04-06T12:30:23.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99827fc700 1 -- 172.21.15.189:0/2101993626 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997000bd30 con 0x7f998415ef60 2024-04-06T12:30:23.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f9983fff700 1 --2- 172.21.15.189:0/2101993626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9970004d90 0x7f9970005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99837fe700 1 --2- 172.21.15.189:0/2101993626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9970003bc0 0x7f9970003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f9982ffd700 1 --2- 172.21.15.189:0/2101993626 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f998415ef60 0x7f9970004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f9983fff700 1 -- 172.21.15.189:0/2101993626 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f998415ef60 msgr2=0x7f9970004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:23.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f9983fff700 1 --2- 172.21.15.189:0/2101993626 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f998415ef60 0x7f9970004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f9983fff700 1 -- 172.21.15.189:0/2101993626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9970003bc0 msgr2=0x7f9970003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:23.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f9983fff700 1 --2- 172.21.15.189:0/2101993626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9970003bc0 0x7f9970003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f9983fff700 1 -- 172.21.15.189:0/2101993626 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99700035b0 con 0x7f9970004d90 2024-04-06T12:30:23.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99837fe700 1 --2- 172.21.15.189:0/2101993626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9970003bc0 0x7f9970003fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:23.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f9982ffd700 1 --2- 172.21.15.189:0/2101993626 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f998415ef60 0x7f9970004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:23.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f9983fff700 1 --2- 172.21.15.189:0/2101993626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9970004d90 0x7f9970005210 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f997c00e170 tx=0x7f9970005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e690ccd0c020f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:23.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99827fc700 1 -- 172.21.15.189:0/2101993626 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c00ec90 con 0x7f9970004d90 2024-04-06T12:30:23.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99827fc700 1 -- 172.21.15.189:0/2101993626 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997c00f040 con 0x7f9970004d90 2024-04-06T12:30:23.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.032+0000 7f99827fc700 1 -- 172.21.15.189:0/2101993626 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c014480 con 0x7f9970004d90 2024-04-06T12:30:23.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.033+0000 7f9989c42700 1 -- 172.21.15.189:0/2101993626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9970004d90 msgr2=0x7f9970005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:23.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.033+0000 7f9989c42700 1 --2- 172.21.15.189:0/2101993626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9970004d90 0x7f9970005210 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f997c00e170 tx=0x7f9970005880 comp rx=0 tx=0).stop 2024-04-06T12:30:23.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.033+0000 7f9983fff700 1 -- 172.21.15.189:0/2101993626 reap_dead start 2024-04-06T12:30:23.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.033+0000 7f9989c42700 1 -- 172.21.15.189:0/2101993626 shutdown_connections 2024-04-06T12:30:23.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.033+0000 7f9989c42700 1 -- 172.21.15.189:0/2101993626 >> 172.21.15.189:0/2101993626 conn(0x7f998406ea80 msgr2=0x7f998406ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:23.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.033+0000 7f9989c42700 1 -- 172.21.15.189:0/2101993626 shutdown_connections 2024-04-06T12:30:23.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.033+0000 7f9989c42700 1 -- 172.21.15.189:0/2101993626 wait complete. 2024-04-06T12:30:23.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.034+0000 7f9989c42700 1 Processor -- start 2024-04-06T12:30:23.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.034+0000 7f9989c42700 1 -- start start 2024-04-06T12:30:23.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.035+0000 7f9989c42700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99840cf690 0x7f99841f4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.035+0000 7f9989c42700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f998415b040 0x7f99840ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.035+0000 7f9989c42700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99841f4960 0x7f99840ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.035+0000 7f9989c42700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99841f3470 con 0x7f998415b040 2024-04-06T12:30:23.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.035+0000 7f9989c42700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99840d1610 con 0x7f99841f4960 2024-04-06T12:30:23.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.035+0000 7f9989c42700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99840d18f0 con 0x7f99840cf690 2024-04-06T12:30:23.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.035+0000 7f9982ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f998415b040 0x7f99840ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.035+0000 7f9982ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f998415b040 0x7f99840ca400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40792/0 (socket says 172.21.15.189:40792) 2024-04-06T12:30:23.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.035+0000 7f9982ffd700 1 -- 172.21.15.189:0/2343234893 learned_addr learned my addr 172.21.15.189:0/2343234893 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:23.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.035+0000 7f99837fe700 1 --2- 172.21.15.189:0/2343234893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99840cf690 0x7f99841f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.035+0000 7f9983fff700 1 --2- 172.21.15.189:0/2343234893 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99841f4960 0x7f99840ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.035+0000 7f9982ffd700 1 -- 172.21.15.189:0/2343234893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99840cf690 msgr2=0x7f99841f4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:23.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.035+0000 7f9982ffd700 1 --2- 172.21.15.189:0/2343234893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99840cf690 0x7f99841f4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.035+0000 7f9982ffd700 1 -- 172.21.15.189:0/2343234893 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99841f4960 msgr2=0x7f99840ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:23.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.035+0000 7f9982ffd700 1 --2- 172.21.15.189:0/2343234893 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99841f4960 0x7f99840ca940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.035+0000 7f9982ffd700 1 -- 172.21.15.189:0/2343234893 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99840d1bd0 con 0x7f998415b040 2024-04-06T12:30:23.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.035+0000 7f99837fe700 1 --2- 172.21.15.189:0/2343234893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99840cf690 0x7f99841f4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:23.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.036+0000 7f9982ffd700 1 --2- 172.21.15.189:0/2343234893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f998415b040 0x7f99840ca400 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f997400e7b0 tx=0x7f997400eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:23.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.036+0000 7f997bfff700 1 -- 172.21.15.189:0/2343234893 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9974017070 con 0x7f998415b040 2024-04-06T12:30:23.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.036+0000 7f997bfff700 1 -- 172.21.15.189:0/2343234893 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99740052e0 con 0x7f998415b040 2024-04-06T12:30:23.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.036+0000 7f9989c42700 1 -- 172.21.15.189:0/2343234893 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99840d1eb0 con 0x7f998415b040 2024-04-06T12:30:23.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.036+0000 7f997bfff700 1 -- 172.21.15.189:0/2343234893 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9974005490 con 0x7f998415b040 2024-04-06T12:30:23.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.036+0000 7f9989c42700 1 -- 172.21.15.189:0/2343234893 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99840cfc50 con 0x7f998415b040 2024-04-06T12:30:23.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.037+0000 7f997bfff700 1 -- 172.21.15.189:0/2343234893 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f9974006860 con 0x7f998415b040 2024-04-06T12:30:23.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.038+0000 7f997bfff700 1 --2- 172.21.15.189:0/2343234893 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f996406adb0 0x7f996406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.038+0000 7f997bfff700 1 -- 172.21.15.189:0/2343234893 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9974088d60 con 0x7f998415b040 2024-04-06T12:30:23.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.038+0000 7f99837fe700 1 --2- 172.21.15.189:0/2343234893 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f996406adb0 0x7f996406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.038+0000 7f99837fe700 1 --2- 172.21.15.189:0/2343234893 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f996406adb0 0x7f996406d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f996c005b00 tx=0x7f996c009040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:23.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.040+0000 7f9995df6b80 1 Processor -- start 2024-04-06T12:30:23.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.040+0000 7f9995df6b80 1 -- start start 2024-04-06T12:30:23.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.041+0000 7f9995df6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 0x55dd76c3da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.041+0000 7f9995df6b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55dd76c3efc0 0x55dd76c155f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.041+0000 7f9995df6b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c14980 0x55dd76c15b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.041+0000 7f9995df6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd76b6e940 con 0x55dd76c39c20 2024-04-06T12:30:23.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.041+0000 7f9995df6b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd76b6c300 con 0x55dd76c14980 2024-04-06T12:30:23.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.041+0000 7f9995df6b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dd76b67bf0 con 0x55dd76c3efc0 2024-04-06T12:30:23.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.041+0000 7f99837fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 0x55dd76c3da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.041+0000 7f99837fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 0x55dd76c3da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40800/0 (socket says 172.21.15.189:40800) 2024-04-06T12:30:23.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.041+0000 7f99837fe700 1 -- 172.21.15.189:0/3167657813 learned_addr learned my addr 172.21.15.189:0/3167657813 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:23.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.041+0000 7f99837fe700 1 -- 172.21.15.189:0/3167657813 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55dd76c3efc0 msgr2=0x55dd76c155f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:23.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.041+0000 7f99837fe700 1 --2- 172.21.15.189:0/3167657813 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55dd76c3efc0 0x55dd76c155f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.041+0000 7f9983fff700 1 --2- 172.21.15.189:0/3167657813 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c14980 0x55dd76c15b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.041+0000 7f99837fe700 1 -- 172.21.15.189:0/3167657813 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c14980 msgr2=0x55dd76c15b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:23.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.041+0000 7f99837fe700 1 --2- 172.21.15.189:0/3167657813 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c14980 0x55dd76c15b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.041+0000 7f99837fe700 1 -- 172.21.15.189:0/3167657813 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd76995b10 con 0x55dd76c39c20 2024-04-06T12:30:23.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.042+0000 7f99837fe700 1 --2- 172.21.15.189:0/3167657813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 0x55dd76c3da60 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f996c005fd0 tx=0x7f996c00c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a56908f18ff2e12b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:23.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.042+0000 7f99797fa700 1 -- 172.21.15.189:0/3167657813 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c014780 con 0x55dd76c39c20 2024-04-06T12:30:23.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.042+0000 7f99797fa700 1 -- 172.21.15.189:0/3167657813 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996c0148e0 con 0x55dd76c39c20 2024-04-06T12:30:23.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.042+0000 7f99797fa700 1 -- 172.21.15.189:0/3167657813 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c014c50 con 0x55dd76c39c20 2024-04-06T12:30:23.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.042+0000 7f9995df6b80 1 -- 172.21.15.189:0/3167657813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 msgr2=0x55dd76c3da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:23.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.042+0000 7f9995df6b80 1 --2- 172.21.15.189:0/3167657813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 0x55dd76c3da60 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f996c005fd0 tx=0x7f996c00c000 comp rx=0 tx=0).stop 2024-04-06T12:30:23.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.042+0000 7f9995df6b80 1 -- 172.21.15.189:0/3167657813 shutdown_connections 2024-04-06T12:30:23.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.042+0000 7f9995df6b80 1 --2- 172.21.15.189:0/3167657813 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c14980 0x55dd76c15b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.042+0000 7f9995df6b80 1 --2- 172.21.15.189:0/3167657813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 0x55dd76c3da60 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.042+0000 7f9995df6b80 1 --2- 172.21.15.189:0/3167657813 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55dd76c3efc0 0x55dd76c155f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.042+0000 7f9995df6b80 1 -- 172.21.15.189:0/3167657813 >> 172.21.15.189:0/3167657813 conn(0x55dd76b75650 msgr2=0x55dd76c5c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:23.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.043+0000 7f9995df6b80 1 -- 172.21.15.189:0/3167657813 shutdown_connections 2024-04-06T12:30:23.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.043+0000 7f9995df6b80 1 -- 172.21.15.189:0/3167657813 wait complete. 2024-04-06T12:30:23.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.043+0000 7f9995df6b80 1 -- start start 2024-04-06T12:30:23.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.043+0000 7f9995df6b80 1 Processor -- start 2024-04-06T12:30:23.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.044+0000 7f9995df6b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c14980 0x55dd76c388d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.044+0000 7f9995df6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 0x55dd76c5f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.044+0000 7f9995df6b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55dd76c3efc0 0x55dd76c5fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.044+0000 7f9995df6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd76b03d30 con 0x55dd76c39c20 2024-04-06T12:30:23.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.044+0000 7f9995df6b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd76b039d0 con 0x55dd76c14980 2024-04-06T12:30:23.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.044+0000 7f9995df6b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dd769eb650 con 0x55dd76c3efc0 2024-04-06T12:30:23.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.044+0000 7f9995df6b80 10 client.0 _ll_register_callbacks cb 0x55dd76b0d420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:30:23.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.044+0000 7f9982ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 0x55dd76c5f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.044+0000 7f9982ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 0x55dd76c5f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40812/0 (socket says 172.21.15.189:40812) 2024-04-06T12:30:23.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.044+0000 7f9982ffd700 1 -- 172.21.15.189:0/3663342720 learned_addr learned my addr 172.21.15.189:0/3663342720 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:23.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.044+0000 7f99837fe700 1 --2- 172.21.15.189:0/3663342720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c14980 0x55dd76c388d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.044+0000 7f9983fff700 1 --2- 172.21.15.189:0/3663342720 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55dd76c3efc0 0x55dd76c5fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.044+0000 7f9982ffd700 1 -- 172.21.15.189:0/3663342720 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55dd76c3efc0 msgr2=0x55dd76c5fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:23.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.044+0000 7f9982ffd700 1 --2- 172.21.15.189:0/3663342720 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55dd76c3efc0 0x55dd76c5fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.044+0000 7f9982ffd700 1 -- 172.21.15.189:0/3663342720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c14980 msgr2=0x55dd76c388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:23.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.044+0000 7f9982ffd700 1 --2- 172.21.15.189:0/3663342720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c14980 0x55dd76c388d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.044+0000 7f9982ffd700 1 -- 172.21.15.189:0/3663342720 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f996c00f040 con 0x55dd76c39c20 2024-04-06T12:30:23.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.045+0000 7f9982ffd700 1 --2- 172.21.15.189:0/3663342720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 0x55dd76c5f960 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f9974053ea0 tx=0x7f9974058120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15cbf0c4a928bff5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:23.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.045+0000 7f99627fc700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:30:23.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.045+0000 7f99627fc700 1 -- 172.21.15.189:0/3663342720 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997401a510 con 0x55dd76c39c20 2024-04-06T12:30:23.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.045+0000 7f99627fc700 1 -- 172.21.15.189:0/3663342720 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(128..128 src has 1..128) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f997405b7d0 con 0x55dd76c39c20 2024-04-06T12:30:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.045+0000 7f9995df6b80 1 -- 172.21.15.189:0/3663342720 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dd76b28330 con 0x55dd76c39c20 2024-04-06T12:30:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.045+0000 7f99627fc700 1 -- 172.21.15.189:0/3663342720 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) v2 ==== 24+0+0 (secure 0 0 0) 0x7f997405fd90 con 0x55dd76c39c20 2024-04-06T12:30:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.045+0000 7f9995df6b80 10 client.8238 fetch_fsmap learned FSMap version 54 2024-04-06T12:30:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.045+0000 7f9995df6b80 1 -- 172.21.15.189:0/3663342720 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=2+}) v3 -- 0x55dd76b2de50 con 0x55dd76c39c20 2024-04-06T12:30:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.045+0000 7f99627fc700 1 -- 172.21.15.189:0/3663342720 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9974073020 con 0x55dd76c39c20 2024-04-06T12:30:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.045+0000 7f99627fc700 1 -- 172.21.15.189:0/3663342720 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 54) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f997406f180 con 0x55dd76c39c20 2024-04-06T12:30:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.045+0000 7f9995df6b80 10 client.8238 fetch_fsmap finished waiting for FSMap version 54 2024-04-06T12:30:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.045+0000 7f9995df6b80 10 client.8238 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-06T12:30:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.045+0000 7f9995df6b80 20 client.8238 populate_metadata read hostname 'smithi189' 2024-04-06T12:30:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.045+0000 7f9995df6b80 1 --2- 172.21.15.189:0/3663342720 >> [v2:172.21.15.189:6834/393232044,v1:172.21.15.189:6836/393232044] conn(0x55dd76c41590 0x55dd76c6e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.045+0000 7f9995df6b80 4 client.8238 mds_command: new command op to 8088 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-06T12:30:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.045+0000 7f9995df6b80 1 -- 172.21.15.189:0/3663342720 --> [v2:172.21.15.189:6834/393232044,v1:172.21.15.189:6836/393232044] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dd76b606b0 con 0x55dd76c41590 2024-04-06T12:30:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.045+0000 7f99837fe700 1 --2- 172.21.15.189:0/3663342720 >> [v2:172.21.15.189:6834/393232044,v1:172.21.15.189:6836/393232044] conn(0x55dd76c41590 0x55dd76c6e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.046+0000 7f99627fc700 10 client.8238 ms_handle_connect on v2:172.21.15.189:6834/393232044 2024-04-06T12:30:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.046+0000 7f99837fe700 1 --2- 172.21.15.189:0/3663342720 >> [v2:172.21.15.189:6834/393232044,v1:172.21.15.189:6836/393232044] conn(0x55dd76c41590 0x55dd76c6e350 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fa8e46293b31cef server_cookie=8e1729026cb07f84 in_seq=0 out_seq=0 2024-04-06T12:30:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.046+0000 7f99627fc700 1 -- 172.21.15.189:0/3663342720 <== mds.0 v2:172.21.15.189:6834/393232044 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55dd76b606b0 con 0x55dd76c41590 2024-04-06T12:30:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.046+0000 7f99627fc700 10 client.8238 handle_command_reply: tid=0 2024-04-06T12:30:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.046+0000 7f9995df6b80 1 client.8238 shutdown 2024-04-06T12:30:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.046+0000 7f9995df6b80 1 -- 172.21.15.189:0/3663342720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 msgr2=0x55dd76c5f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.046+0000 7f9995df6b80 1 --2- 172.21.15.189:0/3663342720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 0x55dd76c5f960 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f9974053ea0 tx=0x7f9974058120 comp rx=0 tx=0).stop 2024-04-06T12:30:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.046+0000 7f9995df6b80 1 -- 172.21.15.189:0/3663342720 shutdown_connections 2024-04-06T12:30:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.046+0000 7f9995df6b80 1 --2- 172.21.15.189:0/3663342720 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55dd76c3efc0 0x55dd76c5fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.046+0000 7f9995df6b80 1 --2- 172.21.15.189:0/3663342720 >> [v2:172.21.15.189:6834/393232044,v1:172.21.15.189:6836/393232044] conn(0x55dd76c41590 0x55dd76c6e350 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.046+0000 7f9995df6b80 1 --2- 172.21.15.189:0/3663342720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c14980 0x55dd76c388d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.046+0000 7f9995df6b80 1 --2- 172.21.15.189:0/3663342720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 0x55dd76c5f960 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.046+0000 7f99627fc700 0 client.8238 ms_handle_reset on v2:172.21.15.189:6834/393232044 2024-04-06T12:30:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.046+0000 7f9995df6b80 1 -- 172.21.15.189:0/3663342720 >> 172.21.15.189:0/3663342720 conn(0x55dd76b75650 msgr2=0x55dd76c155f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.046+0000 7f9995df6b80 1 -- 172.21.15.189:0/3663342720 shutdown_connections 2024-04-06T12:30:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.046+0000 7f9995df6b80 1 -- 172.21.15.189:0/3663342720 wait complete. 2024-04-06T12:30:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.046+0000 7f9995df6b80 20 client.8238 trim_cache size 0 max 16384 2024-04-06T12:30:23.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.064+0000 7f9995df6b80 1 Processor -- start 2024-04-06T12:30:23.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.064+0000 7f9995df6b80 1 -- start start 2024-04-06T12:30:23.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9995df6b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55dd76c14980 0x55dd76c5c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9995df6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 0x55dd76c13760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9995df6b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c3efc0 0x55dd76c10820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9995df6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd76c11eb0 con 0x55dd76c39c20 2024-04-06T12:30:23.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9995df6b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd76c39430 con 0x55dd76c3efc0 2024-04-06T12:30:23.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9995df6b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dd76c39120 con 0x55dd76c14980 2024-04-06T12:30:23.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9982ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 0x55dd76c13760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9982ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 0x55dd76c13760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40822/0 (socket says 172.21.15.189:40822) 2024-04-06T12:30:23.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9982ffd700 1 -- 172.21.15.189:0/2601862876 learned_addr learned my addr 172.21.15.189:0/2601862876 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:23.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9983fff700 1 --2- 172.21.15.189:0/2601862876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c3efc0 0x55dd76c10820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f99837fe700 1 --2- 172.21.15.189:0/2601862876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55dd76c14980 0x55dd76c5c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9982ffd700 1 -- 172.21.15.189:0/2601862876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55dd76c14980 msgr2=0x55dd76c5c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:23.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9982ffd700 1 --2- 172.21.15.189:0/2601862876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55dd76c14980 0x55dd76c5c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9982ffd700 1 -- 172.21.15.189:0/2601862876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c3efc0 msgr2=0x55dd76c10820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:23.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9982ffd700 1 --2- 172.21.15.189:0/2601862876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c3efc0 0x55dd76c10820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9982ffd700 1 -- 172.21.15.189:0/2601862876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd769edc80 con 0x55dd76c39c20 2024-04-06T12:30:23.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9983fff700 1 --2- 172.21.15.189:0/2601862876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c3efc0 0x55dd76c10820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:23.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9982ffd700 1 --2- 172.21.15.189:0/2601862876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 0x55dd76c13760 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f99740873a0 tx=0x7f997405a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c310c05bf9275f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:23.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9979ffb700 1 -- 172.21.15.189:0/2601862876 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997405b7d0 con 0x55dd76c39c20 2024-04-06T12:30:23.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9979ffb700 1 -- 172.21.15.189:0/2601862876 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9974073040 con 0x55dd76c39c20 2024-04-06T12:30:23.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.065+0000 7f9979ffb700 1 -- 172.21.15.189:0/2601862876 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997405fd90 con 0x55dd76c39c20 2024-04-06T12:30:23.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.066+0000 7f9995df6b80 1 -- 172.21.15.189:0/2601862876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 msgr2=0x55dd76c13760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:23.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.066+0000 7f9995df6b80 1 --2- 172.21.15.189:0/2601862876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 0x55dd76c13760 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f99740873a0 tx=0x7f997405a000 comp rx=0 tx=0).stop 2024-04-06T12:30:23.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.066+0000 7f9995df6b80 1 -- 172.21.15.189:0/2601862876 shutdown_connections 2024-04-06T12:30:23.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.066+0000 7f9995df6b80 1 --2- 172.21.15.189:0/2601862876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c3efc0 0x55dd76c10820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.066+0000 7f9995df6b80 1 --2- 172.21.15.189:0/2601862876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55dd76c14980 0x55dd76c5c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.066+0000 7f9995df6b80 1 --2- 172.21.15.189:0/2601862876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c39c20 0x55dd76c13760 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.066+0000 7f9995df6b80 1 -- 172.21.15.189:0/2601862876 >> 172.21.15.189:0/2601862876 conn(0x55dd76b75650 msgr2=0x55dd76c7d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.066+0000 7f9995df6b80 1 -- 172.21.15.189:0/2601862876 shutdown_connections 2024-04-06T12:30:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.066+0000 7f9995df6b80 1 -- 172.21.15.189:0/2601862876 wait complete. 2024-04-06T12:30:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.066+0000 7f9995df6b80 1 -- start start 2024-04-06T12:30:23.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.067+0000 7f9995df6b80 1 Processor -- start 2024-04-06T12:30:23.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f9995df6b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55dd76c14980 0x55dd76cbd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f9995df6b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c39c20 0x55dd76c41970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f9995df6b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c3efc0 0x55dd76c41eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f9995df6b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd76c15ea0 con 0x55dd76c3efc0 2024-04-06T12:30:23.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f9995df6b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd76c187f0 con 0x55dd76c39c20 2024-04-06T12:30:23.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f9995df6b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dd76c447a0 con 0x55dd76c14980 2024-04-06T12:30:23.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f9995df6b80 10 client.0 _ll_register_callbacks cb 0x55dd76b0d420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:30:23.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f9983fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c3efc0 0x55dd76c41eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f9983fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c3efc0 0x55dd76c41eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40826/0 (socket says 172.21.15.189:40826) 2024-04-06T12:30:23.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f9983fff700 1 -- 172.21.15.189:0/650698846 learned_addr learned my addr 172.21.15.189:0/650698846 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:23.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f99837fe700 1 --2- 172.21.15.189:0/650698846 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55dd76c14980 0x55dd76cbd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f9982ffd700 1 --2- 172.21.15.189:0/650698846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c39c20 0x55dd76c41970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f9983fff700 1 -- 172.21.15.189:0/650698846 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55dd76c14980 msgr2=0x55dd76cbd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:23.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f9983fff700 1 --2- 172.21.15.189:0/650698846 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55dd76c14980 0x55dd76cbd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f9983fff700 1 -- 172.21.15.189:0/650698846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c39c20 msgr2=0x55dd76c41970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:23.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f9983fff700 1 --2- 172.21.15.189:0/650698846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c39c20 0x55dd76c41970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f9983fff700 1 -- 172.21.15.189:0/650698846 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f996c00f040 con 0x55dd76c3efc0 2024-04-06T12:30:23.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f9982ffd700 1 --2- 172.21.15.189:0/650698846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c39c20 0x55dd76c41970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:23.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f9983fff700 1 --2- 172.21.15.189:0/650698846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c3efc0 0x55dd76c41eb0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f997c016000 tx=0x7f997c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c503e2d9f25ece server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:23.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f99627fc700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:30:23.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f99627fc700 1 -- 172.21.15.189:0/650698846 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c022040 con 0x55dd76c3efc0 2024-04-06T12:30:23.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f99627fc700 1 -- 172.21.15.189:0/650698846 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(128..128 src has 1..128) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f997c003e30 con 0x55dd76c3efc0 2024-04-06T12:30:23.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.068+0000 7f9995df6b80 1 -- 172.21.15.189:0/650698846 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f996c0148e0 con 0x55dd76c3efc0 2024-04-06T12:30:23.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.069+0000 7f99627fc700 1 -- 172.21.15.189:0/650698846 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) v2 ==== 24+0+0 (secure 0 0 0) 0x7f997c007000 con 0x55dd76c3efc0 2024-04-06T12:30:23.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.069+0000 7f9995df6b80 10 client.8244 fetch_fsmap learned FSMap version 54 2024-04-06T12:30:23.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.069+0000 7f9995df6b80 1 -- 172.21.15.189:0/650698846 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=2+}) v3 -- 0x55dd76993fe0 con 0x55dd76c3efc0 2024-04-06T12:30:23.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.069+0000 7f99627fc700 1 -- 172.21.15.189:0/650698846 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996c0148e0 con 0x55dd76c3efc0 2024-04-06T12:30:23.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.069+0000 7f99627fc700 1 -- 172.21.15.189:0/650698846 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 54) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f997c004030 con 0x55dd76c3efc0 2024-04-06T12:30:23.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.069+0000 7f9995df6b80 10 client.8244 fetch_fsmap finished waiting for FSMap version 54 2024-04-06T12:30:23.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.069+0000 7f9995df6b80 10 client.8244 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-06T12:30:23.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.069+0000 7f9995df6b80 20 client.8244 populate_metadata read hostname 'smithi189' 2024-04-06T12:30:23.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.069+0000 7f9995df6b80 1 --2- 172.21.15.189:0/650698846 >> [v2:172.21.15.189:6834/393232044,v1:172.21.15.189:6836/393232044] conn(0x55dd76c54410 0x55dd76c547f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.069+0000 7f9995df6b80 4 client.8244 mds_command: new command op to 8088 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-06T12:30:23.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.069+0000 7f9995df6b80 1 -- 172.21.15.189:0/650698846 --> [v2:172.21.15.189:6834/393232044,v1:172.21.15.189:6836/393232044] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55dd769b6ca0 con 0x55dd76c54410 2024-04-06T12:30:23.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.069+0000 7f99837fe700 1 --2- 172.21.15.189:0/650698846 >> [v2:172.21.15.189:6834/393232044,v1:172.21.15.189:6836/393232044] conn(0x55dd76c54410 0x55dd76c547f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.069+0000 7f99627fc700 10 client.8244 ms_handle_connect on v2:172.21.15.189:6834/393232044 2024-04-06T12:30:23.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.069+0000 7f99837fe700 1 --2- 172.21.15.189:0/650698846 >> [v2:172.21.15.189:6834/393232044,v1:172.21.15.189:6836/393232044] conn(0x55dd76c54410 0x55dd76c547f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ade247d50e459e server_cookie=3fd911d608e0b04c in_seq=0 out_seq=0 2024-04-06T12:30:23.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f99627fc700 1 -- 172.21.15.189:0/650698846 <== mds.0 v2:172.21.15.189:6834/393232044 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x55dd769b6ca0 con 0x55dd76c54410 2024-04-06T12:30:23.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f99627fc700 10 client.8244 handle_command_reply: tid=0 2024-04-06T12:30:23.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9995df6b80 1 client.8244 shutdown 2024-04-06T12:30:23.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9995df6b80 1 -- 172.21.15.189:0/650698846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c3efc0 msgr2=0x55dd76c41eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:23.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9995df6b80 1 --2- 172.21.15.189:0/650698846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c3efc0 0x55dd76c41eb0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f997c016000 tx=0x7f997c004ad0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9995df6b80 1 -- 172.21.15.189:0/650698846 shutdown_connections 2024-04-06T12:30:23.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9995df6b80 1 --2- 172.21.15.189:0/650698846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd76c3efc0 0x55dd76c41eb0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9995df6b80 1 --2- 172.21.15.189:0/650698846 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55dd76c14980 0x55dd76cbd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9995df6b80 1 --2- 172.21.15.189:0/650698846 >> [v2:172.21.15.189:6834/393232044,v1:172.21.15.189:6836/393232044] conn(0x55dd76c54410 0x55dd76c547f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9995df6b80 1 --2- 172.21.15.189:0/650698846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dd76c39c20 0x55dd76c41970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f99627fc700 0 client.8244 ms_handle_reset on v2:172.21.15.189:6834/393232044 2024-04-06T12:30:23.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9995df6b80 1 -- 172.21.15.189:0/650698846 >> 172.21.15.189:0/650698846 conn(0x55dd76b75650 msgr2=0x55dd76c0d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:23.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9995df6b80 1 -- 172.21.15.189:0/650698846 shutdown_connections 2024-04-06T12:30:23.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9995df6b80 1 -- 172.21.15.189:0/650698846 wait complete. 2024-04-06T12:30:23.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9995df6b80 20 client.8244 trim_cache size 0 max 16384 2024-04-06T12:30:23.074 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T12:30:23.074 INFO:teuthology.orchestra.run.smithi189.stdout: "cluster_fsid": "e30041de-e9a3-4351-8ca7-29d7f3e3728c", 2024-04-06T12:30:23.074 INFO:teuthology.orchestra.run.smithi189.stdout: "whoami": 0, 2024-04-06T12:30:23.074 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 8088, 2024-04-06T12:30:23.074 INFO:teuthology.orchestra.run.smithi189.stdout: "want_state": "up:active", 2024-04-06T12:30:23.074 INFO:teuthology.orchestra.run.smithi189.stdout: "state": "up:active", 2024-04-06T12:30:23.075 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs", 2024-04-06T12:30:23.075 INFO:teuthology.orchestra.run.smithi189.stdout: "rank_uptime": 5.9413420009999998, 2024-04-06T12:30:23.075 INFO:teuthology.orchestra.run.smithi189.stdout: "mdsmap_epoch": 54, 2024-04-06T12:30:23.075 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch": 128, 2024-04-06T12:30:23.075 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch_barrier": 128, 2024-04-06T12:30:23.075 INFO:teuthology.orchestra.run.smithi189.stdout: "uptime": 11.25496875 2024-04-06T12:30:23.075 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T12:30:23.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9989c42700 1 -- 172.21.15.189:0/2343234893 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f996406adb0 msgr2=0x7f996406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:23.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9989c42700 1 --2- 172.21.15.189:0/2343234893 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f996406adb0 0x7f996406d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f996c005b00 tx=0x7f996c009040 comp rx=0 tx=0).stop 2024-04-06T12:30:23.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9989c42700 1 -- 172.21.15.189:0/2343234893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f998415b040 msgr2=0x7f99840ca400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:23.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9989c42700 1 --2- 172.21.15.189:0/2343234893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f998415b040 0x7f99840ca400 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f997400e7b0 tx=0x7f997400eb70 comp rx=0 tx=0).stop 2024-04-06T12:30:23.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9989c42700 1 -- 172.21.15.189:0/2343234893 shutdown_connections 2024-04-06T12:30:23.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9989c42700 1 --2- 172.21.15.189:0/2343234893 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f996406adb0 0x7f996406d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9989c42700 1 --2- 172.21.15.189:0/2343234893 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99841f4960 0x7f99840ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9989c42700 1 --2- 172.21.15.189:0/2343234893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99840cf690 0x7f99841f4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9989c42700 1 --2- 172.21.15.189:0/2343234893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f998415b040 0x7f99840ca400 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9989c42700 1 -- 172.21.15.189:0/2343234893 >> 172.21.15.189:0/2343234893 conn(0x7f998406ea80 msgr2=0x7f998406ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:23.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.070+0000 7f9989c42700 1 -- 172.21.15.189:0/2343234893 shutdown_connections 2024-04-06T12:30:23.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.071+0000 7f9989c42700 1 -- 172.21.15.189:0/2343234893 wait complete. 2024-04-06T12:30:23.082 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T12:30:23.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.206+0000 7f4633c37700 1 Processor -- start 2024-04-06T12:30:23.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.207+0000 7f4633c37700 1 -- start start 2024-04-06T12:30:23.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.207+0000 7f4633c37700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f462c15d7b0 0x7f462c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.207+0000 7f4633c37700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f462c06a730 0x7f462c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.207+0000 7f4633c37700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f462c0b3990 con 0x7f462c15ddf0 2024-04-06T12:30:23.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.207+0000 7f4633c37700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f462c06b0f0 con 0x7f462c06a730 2024-04-06T12:30:23.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.207+0000 7f4633c37700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f462c0b3060 con 0x7f462c15d7b0 2024-04-06T12:30:23.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.208+0000 7f46321d4700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f462c15ddf0 0x7f462c15e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45570/0 (socket says 172.21.15.189:45570) 2024-04-06T12:30:23.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.208+0000 7f46321d4700 1 -- 172.21.15.189:0/3000851266 learned_addr learned my addr 172.21.15.189:0/3000851266 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:23.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.208+0000 7f46311d2700 1 --2- 172.21.15.189:0/3000851266 >> v2:172.21.15.192:3300/0 conn(0x7f462c06a730 0x7f462c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.208+0000 7f46319d3700 1 --2- 172.21.15.189:0/3000851266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f462c15d7b0 0x7f462c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.208+0000 7f46309d1700 1 -- 172.21.15.189:0/3000851266 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227884501 0 0) 0x7f462c0b3990 con 0x7f462c15ddf0 2024-04-06T12:30:23.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.208+0000 7f46309d1700 1 -- 172.21.15.189:0/3000851266 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46180035b0 con 0x7f462c15ddf0 2024-04-06T12:30:23.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.208+0000 7f46319d3700 1 -- 172.21.15.189:0/3000851266 >> v2:172.21.15.192:3300/0 conn(0x7f462c06a730 msgr2=0x7f462c06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:23.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.208+0000 7f46319d3700 1 --2- 172.21.15.189:0/3000851266 >> v2:172.21.15.192:3300/0 conn(0x7f462c06a730 0x7f462c06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.208+0000 7f46319d3700 1 -- 172.21.15.189:0/3000851266 >> v1:172.21.15.189:6789/0 conn(0x7f462c15ddf0 legacy=0x7f462c15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:23.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.208+0000 7f46319d3700 1 -- 172.21.15.189:0/3000851266 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f462c0b3340 con 0x7f462c15d7b0 2024-04-06T12:30:23.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.208+0000 7f46311d2700 1 --2- 172.21.15.189:0/3000851266 >> v2:172.21.15.192:3300/0 conn(0x7f462c06a730 0x7f462c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:30:23.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.208+0000 7f46319d3700 1 --2- 172.21.15.189:0/3000851266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f462c15d7b0 0x7f462c0b3e20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4628009840 tx=0x7f4628009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9891d4ee3a20007c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:23.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.208+0000 7f46309d1700 1 -- 172.21.15.189:0/3000851266 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4628011830 con 0x7f462c15d7b0 2024-04-06T12:30:23.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.208+0000 7f46309d1700 1 -- 172.21.15.189:0/3000851266 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4628011990 con 0x7f462c15d7b0 2024-04-06T12:30:23.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.208+0000 7f46309d1700 1 -- 172.21.15.189:0/3000851266 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4628011d00 con 0x7f462c15d7b0 2024-04-06T12:30:23.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.209+0000 7f4633c37700 1 -- 172.21.15.189:0/3000851266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f462c15d7b0 msgr2=0x7f462c0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:23.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.209+0000 7f4633c37700 1 --2- 172.21.15.189:0/3000851266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f462c15d7b0 0x7f462c0b3e20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4628009840 tx=0x7f4628009b50 comp rx=0 tx=0).stop 2024-04-06T12:30:23.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.209+0000 7f4633c37700 1 -- 172.21.15.189:0/3000851266 shutdown_connections 2024-04-06T12:30:23.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.209+0000 7f4633c37700 1 --2- 172.21.15.189:0/3000851266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f462c15d7b0 0x7f462c0b3e20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.209+0000 7f4633c37700 1 --2- 172.21.15.189:0/3000851266 >> v2:172.21.15.192:3300/0 conn(0x7f462c06a730 0x7f462c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.209+0000 7f4633c37700 1 -- 172.21.15.189:0/3000851266 >> 172.21.15.189:0/3000851266 conn(0x7f462c0d00b0 msgr2=0x7f462c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:23.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.209+0000 7f4633c37700 1 -- 172.21.15.189:0/3000851266 shutdown_connections 2024-04-06T12:30:23.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.209+0000 7f4633c37700 1 -- 172.21.15.189:0/3000851266 wait complete. 2024-04-06T12:30:23.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.210+0000 7f4633c37700 1 Processor -- start 2024-04-06T12:30:23.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.210+0000 7f4633c37700 1 -- start start 2024-04-06T12:30:23.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.211+0000 7f4633c37700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f462c06a730 0x7f462c149a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.211+0000 7f4633c37700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f462c15d7b0 0x7f462c15b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.211+0000 7f4633c37700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f462c15ddf0 0x7f462c0cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.211+0000 7f4633c37700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462c15a5c0 con 0x7f462c06a730 2024-04-06T12:30:23.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.211+0000 7f4633c37700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462c0cd5c0 con 0x7f462c15ddf0 2024-04-06T12:30:23.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.211+0000 7f4633c37700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f462c0cd8a0 con 0x7f462c15d7b0 2024-04-06T12:30:23.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.211+0000 7f46319d3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f462c06a730 0x7f462c149a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.211+0000 7f46319d3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f462c06a730 0x7f462c149a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40842/0 (socket says 172.21.15.189:40842) 2024-04-06T12:30:23.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.211+0000 7f46319d3700 1 -- 172.21.15.189:0/4037531275 learned_addr learned my addr 172.21.15.189:0/4037531275 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:23.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.211+0000 7f46311d2700 1 --2- 172.21.15.189:0/4037531275 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f462c15d7b0 0x7f462c15b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.211+0000 7f46321d4700 1 --2- 172.21.15.189:0/4037531275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f462c15ddf0 0x7f462c0cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.211+0000 7f46319d3700 1 -- 172.21.15.189:0/4037531275 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f462c15d7b0 msgr2=0x7f462c15b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:23.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.211+0000 7f46319d3700 1 --2- 172.21.15.189:0/4037531275 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f462c15d7b0 0x7f462c15b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.211+0000 7f46319d3700 1 -- 172.21.15.189:0/4037531275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f462c15ddf0 msgr2=0x7f462c0cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:23.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.211+0000 7f46319d3700 1 --2- 172.21.15.189:0/4037531275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f462c15ddf0 0x7f462c0cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.211+0000 7f46319d3700 1 -- 172.21.15.189:0/4037531275 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461c0094a0 con 0x7f462c06a730 2024-04-06T12:30:23.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.212+0000 7f46319d3700 1 --2- 172.21.15.189:0/4037531275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f462c06a730 0x7f462c149a50 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f46280090b0 tx=0x7f4628004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:23.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.212+0000 7f46227fc700 1 -- 172.21.15.189:0/4037531275 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46280036a0 con 0x7f462c06a730 2024-04-06T12:30:23.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.212+0000 7f46227fc700 1 -- 172.21.15.189:0/4037531275 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f462801f070 con 0x7f462c06a730 2024-04-06T12:30:23.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.212+0000 7f4633c37700 1 -- 172.21.15.189:0/4037531275 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46280094a0 con 0x7f462c06a730 2024-04-06T12:30:23.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.212+0000 7f46227fc700 1 -- 172.21.15.189:0/4037531275 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4628003a40 con 0x7f462c06a730 2024-04-06T12:30:23.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.212+0000 7f4633c37700 1 -- 172.21.15.189:0/4037531275 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f462c0cde60 con 0x7f462c06a730 2024-04-06T12:30:23.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.213+0000 7f4633c37700 1 -- 172.21.15.189:0/4037531275 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f462c067bb0 con 0x7f462c06a730 2024-04-06T12:30:23.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.213+0000 7f46227fc700 1 -- 172.21.15.189:0/4037531275 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f4628003ba0 con 0x7f462c06a730 2024-04-06T12:30:23.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.214+0000 7f46227fc700 1 --2- 172.21.15.189:0/4037531275 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f461006adb0 0x7f461006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:23.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.214+0000 7f46227fc700 1 -- 172.21.15.189:0/4037531275 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4628021070 con 0x7f462c06a730 2024-04-06T12:30:23.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.214+0000 7f46311d2700 1 --2- 172.21.15.189:0/4037531275 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f461006adb0 0x7f461006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:23.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.214+0000 7f46311d2700 1 --2- 172.21.15.189:0/4037531275 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f461006adb0 0x7f461006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f462c1f6d60 tx=0x7f461c008040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:23.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.218+0000 7f46227fc700 1 -- 172.21.15.189:0/4037531275 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4628055df0 con 0x7f462c06a730 2024-04-06T12:30:23.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.459+0000 7f4633c37700 1 -- 172.21.15.189:0/4037531275 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f462c074990 con 0x7f462c06a730 2024-04-06T12:30:23.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.460+0000 7f46227fc700 1 -- 172.21.15.189:0/4037531275 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v54) v1 ==== 53+0+73 (secure 0 0 0) 0x7f462802b070 con 0x7f462c06a730 2024-04-06T12:30:23.457 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-06T12:30:23.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.462+0000 7f4633c37700 1 -- 172.21.15.189:0/4037531275 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f461006adb0 msgr2=0x7f461006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:23.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.462+0000 7f4633c37700 1 --2- 172.21.15.189:0/4037531275 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f461006adb0 0x7f461006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f462c1f6d60 tx=0x7f461c008040 comp rx=0 tx=0).stop 2024-04-06T12:30:23.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.462+0000 7f4633c37700 1 -- 172.21.15.189:0/4037531275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f462c06a730 msgr2=0x7f462c149a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.462+0000 7f4633c37700 1 --2- 172.21.15.189:0/4037531275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f462c06a730 0x7f462c149a50 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f46280090b0 tx=0x7f4628004750 comp rx=0 tx=0).stop 2024-04-06T12:30:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.462+0000 7f4633c37700 1 -- 172.21.15.189:0/4037531275 shutdown_connections 2024-04-06T12:30:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.462+0000 7f4633c37700 1 --2- 172.21.15.189:0/4037531275 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f461006adb0 0x7f461006d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.462+0000 7f4633c37700 1 --2- 172.21.15.189:0/4037531275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f462c15ddf0 0x7f462c0cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.462+0000 7f4633c37700 1 --2- 172.21.15.189:0/4037531275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f462c06a730 0x7f462c149a50 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.462+0000 7f4633c37700 1 --2- 172.21.15.189:0/4037531275 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f462c15d7b0 0x7f462c15b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.462+0000 7f4633c37700 1 -- 172.21.15.189:0/4037531275 >> 172.21.15.189:0/4037531275 conn(0x7f462c0d00b0 msgr2=0x7f462c159580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.462+0000 7f4633c37700 1 -- 172.21.15.189:0/4037531275 shutdown_connections 2024-04-06T12:30:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:23.462+0000 7f4633c37700 1 -- 172.21.15.189:0/4037531275 wait complete. 2024-04-06T12:30:23.472 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-06T12:30:23.472 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T12:30:23.472 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-06T12:30:23.472 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-06T12:30:23.472 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T12:30:23.472 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-06T12:30:23.472 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-06T12:30:23.472 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T12:30:23.472 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:30:23.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:23.473 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-06T12:30:23.490 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T12:30:23.490 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T12:30:23.491 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T12:30:23.491 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:30:23.491 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T12:30:23.491 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:30:23.491 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:30:23.491 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:30:23.491 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-06T12:30:23.491 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:30:23.491 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:30:23.491 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-06T12:30:23.491 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:30:23.491 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:30:23.491 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T12:30:23.492 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:30:23.492 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T12:30:23.492 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 41266sec preferred_lft 41266sec 2024-04-06T12:30:23.492 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::8592:df2c:92e2:dace/64 scope link noprefixroute 2024-04-06T12:30:23.492 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:30:23.492 INFO:teuthology.orchestra.run.smithi189.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T12:30:23.492 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff 2024-04-06T12:30:23.492 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-06T12:30:23.492 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:30:23.492 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::88e:b6ff:fe6f:1cb0/64 scope link 2024-04-06T12:30:23.492 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:30:23.492 INFO:teuthology.orchestra.run.smithi189.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T12:30:23.493 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 62:a5:8d:c5:1a:9c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T12:30:23.493 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::60a5:8dff:fec5:1a9c/64 scope link 2024-04-06T12:30:23.493 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:30:23.493 INFO:teuthology.orchestra.run.smithi189.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T12:30:23.493 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-06T12:30:23.493 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::1c7e:c6ff:febb:1dab/64 scope link 2024-04-06T12:30:23.493 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:30:23.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:23.493 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T12:30:23.556 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-06T12:30:23.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:23.558 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-06T12:30:23.617 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-06T12:30:23.617 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T12:30:23.617 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T12:30:23.618 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T12:30:23.618 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T12:30:23.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:23.618 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:30:23.618 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:30:23.618 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T12:30:23.618 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T12:30:23.618 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T12:30:23.618 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:30:23.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:23.724 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T12:30:23.741 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-06T12:30:23.741 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-06T12:30:23.741 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:30:23.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:23.741 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-04-06T12:30:23.808 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:30:23.808 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-06T12:30:23.865 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:30:23.865 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:30:23.865 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-06T12:30:23.921 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T12:30:23.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:23.922 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:30:24.008 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:30:24.008 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:30:24.008 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:30:24.017 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:30:24.017 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:30:24.017 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:30:24.017 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:30:24.017 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:30:24.036 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:30:24.036 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:30:24.036 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:30:24.036 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:30:24.036 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:30:24.036 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790" 2024-04-06T12:30:24.036 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2024-04-06T12:30:24.036 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e30041de-e9a3-4351-8ca7-29d7f3e3728c" will pass to kernel 2024-04-06T12:30:24.038 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T12:30:24.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:24.038 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-06T12:30:24.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:24.066 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:30:24.149 INFO:teuthology.orchestra.run.smithi189.stdout:client6845 2024-04-06T12:30:24.149 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:30:24.150 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:30:24.208 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-06T12:30:24.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.374+0000 7f564eea1700 1 Processor -- start 2024-04-06T12:30:24.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.375+0000 7f564eea1700 1 -- start start 2024-04-06T12:30:24.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.375+0000 7f564eea1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f564815a550 0x7f56480cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:24.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.375+0000 7f564eea1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56480cdbe0 con 0x7f56480b4d70 2024-04-06T12:30:24.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.375+0000 7f564eea1700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f564815a940 con 0x7f56480cf900 2024-04-06T12:30:24.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.375+0000 7f564eea1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564815ac00 con 0x7f564815a550 2024-04-06T12:30:24.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.375+0000 7f564d43e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f56480b4d70 0x7f56480b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45580/0 (socket says 172.21.15.189:45580) 2024-04-06T12:30:24.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.375+0000 7f564d43e700 1 -- 172.21.15.189:0/918149855 learned_addr learned my addr 172.21.15.189:0/918149855 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:24.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f56477fe700 1 -- 172.21.15.189:0/918149855 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306830130 0 0) 0x7f56480cdbe0 con 0x7f56480b4d70 2024-04-06T12:30:24.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f56477fe700 1 -- 172.21.15.189:0/918149855 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56380035b0 con 0x7f56480b4d70 2024-04-06T12:30:24.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f564cc3d700 1 --2- 172.21.15.189:0/918149855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f564815a550 0x7f56480cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:24.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f56477fe700 1 -- 172.21.15.189:0/918149855 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4081647422 0 0) 0x7f56380035b0 con 0x7f56480b4d70 2024-04-06T12:30:24.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f56477fe700 1 -- 172.21.15.189:0/918149855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f564815a550 msgr2=0x7f56480cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:24.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f56477fe700 1 --2- 172.21.15.189:0/918149855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f564815a550 0x7f56480cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:24.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f56477fe700 1 -- 172.21.15.189:0/918149855 >> v1:172.21.15.192:6789/0 conn(0x7f56480cf900 legacy=0x7f56480b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:24.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f56477fe700 1 -- 172.21.15.189:0/918149855 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564815aee0 con 0x7f56480b4d70 2024-04-06T12:30:24.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f56477fe700 1 -- 172.21.15.189:0/918149855 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f56400019f0 con 0x7f56480b4d70 2024-04-06T12:30:24.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f56477fe700 1 -- 172.21.15.189:0/918149855 >> v1:172.21.15.189:6789/0 conn(0x7f56480b4d70 legacy=0x7f56480b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:24.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f56477fe700 1 --2- 172.21.15.189:0/918149855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5638003a30 0x7f5638003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:24.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f56477fe700 1 --2- 172.21.15.189:0/918149855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5638004850 0x7f5638004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:24.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f56477fe700 1 --2- 172.21.15.189:0/918149855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56380051d0 0x7f563800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:24.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f56477fe700 1 -- 172.21.15.189:0/918149855 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564815ac00 con 0x7f5638003a30 2024-04-06T12:30:24.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f56477fe700 1 -- 172.21.15.189:0/918149855 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563800bf50 con 0x7f56380051d0 2024-04-06T12:30:24.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f56477fe700 1 -- 172.21.15.189:0/918149855 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563800c090 con 0x7f5638004850 2024-04-06T12:30:24.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f564cc3d700 1 --2- 172.21.15.189:0/918149855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5638003a30 0x7f5638003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:24.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f5647fff700 1 --2- 172.21.15.189:0/918149855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5638004850 0x7f5638004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:24.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f564d43e700 1 --2- 172.21.15.189:0/918149855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56380051d0 0x7f563800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:24.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f564cc3d700 1 -- 172.21.15.189:0/918149855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5638004850 msgr2=0x7f5638004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:24.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f564cc3d700 1 --2- 172.21.15.189:0/918149855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5638004850 0x7f5638004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:24.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f564cc3d700 1 -- 172.21.15.189:0/918149855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56380051d0 msgr2=0x7f563800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:24.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f564cc3d700 1 --2- 172.21.15.189:0/918149855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56380051d0 0x7f563800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:24.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f564cc3d700 1 -- 172.21.15.189:0/918149855 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56380035b0 con 0x7f5638003a30 2024-04-06T12:30:24.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.376+0000 7f564d43e700 1 --2- 172.21.15.189:0/918149855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56380051d0 0x7f563800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:24.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.377+0000 7f564cc3d700 1 --2- 172.21.15.189:0/918149855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5638003a30 0x7f5638003e50 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f563c009b60 tx=0x7f5638005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e33914d7e44831 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:24.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.377+0000 7f56477fe700 1 -- 172.21.15.189:0/918149855 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c00e660 con 0x7f5638003a30 2024-04-06T12:30:24.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.377+0000 7f56477fe700 1 -- 172.21.15.189:0/918149855 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563c00e7c0 con 0x7f5638003a30 2024-04-06T12:30:24.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.377+0000 7f56477fe700 1 -- 172.21.15.189:0/918149855 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c00eb30 con 0x7f5638003a30 2024-04-06T12:30:24.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.377+0000 7f564eea1700 1 -- 172.21.15.189:0/918149855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5638003a30 msgr2=0x7f5638003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:24.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.377+0000 7f564eea1700 1 --2- 172.21.15.189:0/918149855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5638003a30 0x7f5638003e50 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f563c009b60 tx=0x7f5638005880 comp rx=0 tx=0).stop 2024-04-06T12:30:24.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.377+0000 7f564d43e700 1 -- 172.21.15.189:0/918149855 reap_dead start 2024-04-06T12:30:24.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.377+0000 7f564eea1700 1 -- 172.21.15.189:0/918149855 shutdown_connections 2024-04-06T12:30:24.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.377+0000 7f564eea1700 1 -- 172.21.15.189:0/918149855 >> 172.21.15.189:0/918149855 conn(0x7f56480d00b0 msgr2=0x7f56480d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:24.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.377+0000 7f564eea1700 1 -- 172.21.15.189:0/918149855 shutdown_connections 2024-04-06T12:30:24.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.377+0000 7f564eea1700 1 -- 172.21.15.189:0/918149855 wait complete. 2024-04-06T12:30:24.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.378+0000 7f564eea1700 1 Processor -- start 2024-04-06T12:30:24.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.379+0000 7f564eea1700 1 -- start start 2024-04-06T12:30:24.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.379+0000 7f564eea1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5638003a30 0x7f564814da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:24.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.379+0000 7f564eea1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5648155150 0x7f564814f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:24.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.379+0000 7f564eea1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f564814eab0 0x7f564814f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:24.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.379+0000 7f564eea1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5648154d30 con 0x7f5638003a30 2024-04-06T12:30:24.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.379+0000 7f564eea1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564814f000 con 0x7f564814eab0 2024-04-06T12:30:24.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.379+0000 7f564eea1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5648154390 con 0x7f5648155150 2024-04-06T12:30:24.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.379+0000 7f564cc3d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5638003a30 0x7f564814da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:24.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.380+0000 7f564cc3d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5638003a30 0x7f564814da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40848/0 (socket says 172.21.15.189:40848) 2024-04-06T12:30:24.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.380+0000 7f564cc3d700 1 -- 172.21.15.189:0/3648444799 learned_addr learned my addr 172.21.15.189:0/3648444799 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:24.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.380+0000 7f5647fff700 1 --2- 172.21.15.189:0/3648444799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5648155150 0x7f564814f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:24.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.380+0000 7f564d43e700 1 --2- 172.21.15.189:0/3648444799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f564814eab0 0x7f564814f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:24.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.380+0000 7f564cc3d700 1 -- 172.21.15.189:0/3648444799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5648155150 msgr2=0x7f564814f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:24.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.380+0000 7f564cc3d700 1 --2- 172.21.15.189:0/3648444799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5648155150 0x7f564814f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:24.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.380+0000 7f564cc3d700 1 -- 172.21.15.189:0/3648444799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f564814eab0 msgr2=0x7f564814f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:24.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.380+0000 7f564cc3d700 1 --2- 172.21.15.189:0/3648444799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f564814eab0 0x7f564814f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:24.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.380+0000 7f564cc3d700 1 -- 172.21.15.189:0/3648444799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5648154670 con 0x7f5638003a30 2024-04-06T12:30:24.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.380+0000 7f5647fff700 1 --2- 172.21.15.189:0/3648444799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5648155150 0x7f564814f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:24.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.380+0000 7f564d43e700 1 --2- 172.21.15.189:0/3648444799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f564814eab0 0x7f564814f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:24.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.380+0000 7f564cc3d700 1 --2- 172.21.15.189:0/3648444799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5638003a30 0x7f564814da80 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f5638005800 tx=0x7f563c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:24.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.380+0000 7f56457fa700 1 -- 172.21.15.189:0/3648444799 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c00e660 con 0x7f5638003a30 2024-04-06T12:30:24.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.380+0000 7f56457fa700 1 -- 172.21.15.189:0/3648444799 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563c003ab0 con 0x7f5638003a30 2024-04-06T12:30:24.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.380+0000 7f564eea1700 1 -- 172.21.15.189:0/3648444799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5648154950 con 0x7f5638003a30 2024-04-06T12:30:24.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.380+0000 7f56457fa700 1 -- 172.21.15.189:0/3648444799 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c003e60 con 0x7f5638003a30 2024-04-06T12:30:24.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.380+0000 7f564eea1700 1 -- 172.21.15.189:0/3648444799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56481501e0 con 0x7f5638003a30 2024-04-06T12:30:24.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.381+0000 7f564eea1700 1 -- 172.21.15.189:0/3648444799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5648154fa0 con 0x7f5638003a30 2024-04-06T12:30:24.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.382+0000 7f56457fa700 1 -- 172.21.15.189:0/3648444799 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f563c013470 con 0x7f5638003a30 2024-04-06T12:30:24.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.382+0000 7f56457fa700 1 --2- 172.21.15.189:0/3648444799 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f562c06adb0 0x7f562c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:24.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.382+0000 7f56457fa700 1 -- 172.21.15.189:0/3648444799 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f563c08ad90 con 0x7f5638003a30 2024-04-06T12:30:24.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.382+0000 7f5647fff700 1 --2- 172.21.15.189:0/3648444799 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f562c06adb0 0x7f562c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:24.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.383+0000 7f5647fff700 1 --2- 172.21.15.189:0/3648444799 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f562c06adb0 0x7f562c06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f56481f5b60 tx=0x7f5634009040 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:24.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.387+0000 7f56457fa700 1 -- 172.21.15.189:0/3648444799 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f563c055ed0 con 0x7f5638003a30 2024-04-06T12:30:24.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.610+0000 7f564eea1700 1 -- 172.21.15.189:0/3648444799 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f5648154500 con 0x7f562c06adb0 2024-04-06T12:30:24.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.612+0000 7f56457fa700 1 -- 172.21.15.189:0/3648444799 <== mgr.7875 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f5648154500 con 0x7f562c06adb0 2024-04-06T12:30:24.609 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T12:30:24.609 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:30:24.609 INFO:teuthology.orchestra.run.smithi189.stdout: "name": "cephfs" 2024-04-06T12:30:24.609 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:30:24.609 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T12:30:24.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.614+0000 7f564eea1700 1 -- 172.21.15.189:0/3648444799 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f562c06adb0 msgr2=0x7f562c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:24.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.614+0000 7f564eea1700 1 --2- 172.21.15.189:0/3648444799 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f562c06adb0 0x7f562c06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f56481f5b60 tx=0x7f5634009040 comp rx=0 tx=0).stop 2024-04-06T12:30:24.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.614+0000 7f564eea1700 1 -- 172.21.15.189:0/3648444799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5638003a30 msgr2=0x7f564814da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:24.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.614+0000 7f564eea1700 1 --2- 172.21.15.189:0/3648444799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5638003a30 0x7f564814da80 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f5638005800 tx=0x7f563c0047c0 comp rx=0 tx=0).stop 2024-04-06T12:30:24.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.614+0000 7f564eea1700 1 -- 172.21.15.189:0/3648444799 shutdown_connections 2024-04-06T12:30:24.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.614+0000 7f564eea1700 1 --2- 172.21.15.189:0/3648444799 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f562c06adb0 0x7f562c06d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:24.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.614+0000 7f564eea1700 1 --2- 172.21.15.189:0/3648444799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f564814eab0 0x7f564814f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:24.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.614+0000 7f564eea1700 1 --2- 172.21.15.189:0/3648444799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5638003a30 0x7f564814da80 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:24.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.614+0000 7f564eea1700 1 --2- 172.21.15.189:0/3648444799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5648155150 0x7f564814f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:24.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.614+0000 7f564eea1700 1 -- 172.21.15.189:0/3648444799 >> 172.21.15.189:0/3648444799 conn(0x7f56480d00b0 msgr2=0x7f56480b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:24.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.614+0000 7f564eea1700 1 -- 172.21.15.189:0/3648444799 shutdown_connections 2024-04-06T12:30:24.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.614+0000 7f564eea1700 1 -- 172.21.15.189:0/3648444799 wait complete. 2024-04-06T12:30:24.623 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-06T12:30:24.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.750+0000 7f922ff5e700 1 Processor -- start 2024-04-06T12:30:24.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.750+0000 7f922ff5e700 1 -- start start 2024-04-06T12:30:24.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.751+0000 7f922ff5e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f92280cdc60 0x7f92280ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:24.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.751+0000 7f922ff5e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f922806e470 0x7f922806e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:24.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.751+0000 7f922ff5e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922806c080 con 0x7f9228159b20 2024-04-06T12:30:24.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.751+0000 7f922ff5e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92280ce650 con 0x7f92280cdc60 2024-04-06T12:30:24.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.751+0000 7f922ff5e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9228159f00 con 0x7f922806e470 2024-04-06T12:30:24.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.751+0000 7f922d4f9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f92280cdc60 0x7f92280ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:24.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.752+0000 7f922dcfa700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f9228159b20 0x7f922806b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:56536/0 (socket says 172.21.15.189:56536) 2024-04-06T12:30:24.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.752+0000 7f922dcfa700 1 -- 172.21.15.189:0/1268501654 learned_addr learned my addr 172.21.15.189:0/1268501654 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:24.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.752+0000 7f922d4f9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f92280cdc60 0x7f92280ce110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:44614/0 (socket says 172.21.15.189:44614) 2024-04-06T12:30:24.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.752+0000 7f922e4fb700 1 --2- 172.21.15.189:0/1268501654 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f922806e470 0x7f922806e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:24.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.752+0000 7f922ccf8700 1 -- 172.21.15.189:0/1268501654 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732505888 0 0) 0x7f922806c080 con 0x7f9228159b20 2024-04-06T12:30:24.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.752+0000 7f922ccf8700 1 -- 172.21.15.189:0/1268501654 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92140035b0 con 0x7f9228159b20 2024-04-06T12:30:24.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.752+0000 7f922d4f9700 1 -- 172.21.15.189:0/1268501654 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f922806e470 msgr2=0x7f922806e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:24.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.752+0000 7f922d4f9700 1 --2- 172.21.15.189:0/1268501654 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f922806e470 0x7f922806e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:24.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.752+0000 7f922d4f9700 1 -- 172.21.15.189:0/1268501654 >> v1:172.21.15.192:6789/0 conn(0x7f9228159b20 legacy=0x7f922806b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:24.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.752+0000 7f922d4f9700 1 -- 172.21.15.189:0/1268501654 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922806edc0 con 0x7f92280cdc60 2024-04-06T12:30:24.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.752+0000 7f922e4fb700 1 --2- 172.21.15.189:0/1268501654 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f922806e470 0x7f922806e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:30:24.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.752+0000 7f922d4f9700 1 --2- 172.21.15.189:0/1268501654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92280cdc60 0x7f92280ce110 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f921800aa10 tx=0x7f921800ad20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5216fbe57dc4416 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:24.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.752+0000 7f922ccf8700 1 -- 172.21.15.189:0/1268501654 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921800cb30 con 0x7f92280cdc60 2024-04-06T12:30:24.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.752+0000 7f922ccf8700 1 -- 172.21.15.189:0/1268501654 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f921800cc90 con 0x7f92280cdc60 2024-04-06T12:30:24.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.752+0000 7f922ccf8700 1 -- 172.21.15.189:0/1268501654 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9218018400 con 0x7f92280cdc60 2024-04-06T12:30:24.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.753+0000 7f922ff5e700 1 -- 172.21.15.189:0/1268501654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92280cdc60 msgr2=0x7f92280ce110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:24.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.753+0000 7f922ff5e700 1 --2- 172.21.15.189:0/1268501654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92280cdc60 0x7f92280ce110 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f921800aa10 tx=0x7f921800ad20 comp rx=0 tx=0).stop 2024-04-06T12:30:24.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.753+0000 7f922ff5e700 1 -- 172.21.15.189:0/1268501654 shutdown_connections 2024-04-06T12:30:24.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.753+0000 7f922ff5e700 1 --2- 172.21.15.189:0/1268501654 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f922806e470 0x7f922806e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:24.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.753+0000 7f922ff5e700 1 --2- 172.21.15.189:0/1268501654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92280cdc60 0x7f92280ce110 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:24.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.753+0000 7f922ff5e700 1 -- 172.21.15.189:0/1268501654 >> 172.21.15.189:0/1268501654 conn(0x7f92280cc290 msgr2=0x7f92280cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:24.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.753+0000 7f922ff5e700 1 -- 172.21.15.189:0/1268501654 shutdown_connections 2024-04-06T12:30:24.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.753+0000 7f922ff5e700 1 -- 172.21.15.189:0/1268501654 wait complete. 2024-04-06T12:30:24.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.754+0000 7f922ff5e700 1 Processor -- start 2024-04-06T12:30:24.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.755+0000 7f922ff5e700 1 -- start start 2024-04-06T12:30:24.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.755+0000 7f922ff5e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f922806e470 0x7f922814bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:24.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.755+0000 7f922ff5e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92280cdc60 0x7f92280c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:24.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.755+0000 7f922ff5e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228159b20 0x7f92280c57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:24.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.755+0000 7f922ff5e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92280c9b60 con 0x7f9228159b20 2024-04-06T12:30:24.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.755+0000 7f922ff5e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92280c5330 con 0x7f92280cdc60 2024-04-06T12:30:24.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.755+0000 7f922ff5e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92280c9880 con 0x7f922806e470 2024-04-06T12:30:24.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.756+0000 7f922e4fb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228159b20 0x7f92280c57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:24.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.756+0000 7f922e4fb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228159b20 0x7f92280c57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40850/0 (socket says 172.21.15.189:40850) 2024-04-06T12:30:24.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.756+0000 7f922e4fb700 1 -- 172.21.15.189:0/3525143221 learned_addr learned my addr 172.21.15.189:0/3525143221 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:24.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.756+0000 7f922dcfa700 1 --2- 172.21.15.189:0/3525143221 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f922806e470 0x7f922814bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:24.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.756+0000 7f922d4f9700 1 --2- 172.21.15.189:0/3525143221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92280cdc60 0x7f92280c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:24.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.756+0000 7f922e4fb700 1 -- 172.21.15.189:0/3525143221 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f922806e470 msgr2=0x7f922814bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:24.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.756+0000 7f922e4fb700 1 --2- 172.21.15.189:0/3525143221 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f922806e470 0x7f922814bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.756+0000 7f922e4fb700 1 -- 172.21.15.189:0/3525143221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92280cdc60 msgr2=0x7f92280c43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.756+0000 7f922e4fb700 1 --2- 172.21.15.189:0/3525143221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92280cdc60 0x7f92280c43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.756+0000 7f922e4fb700 1 -- 172.21.15.189:0/3525143221 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9220009a70 con 0x7f9228159b20 2024-04-06T12:30:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.756+0000 7f922e4fb700 1 --2- 172.21.15.189:0/3525143221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228159b20 0x7f92280c57c0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f922000e580 tx=0x7f922000e890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.756+0000 7f921e7fc700 1 -- 172.21.15.189:0/3525143221 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9220003c20 con 0x7f9228159b20 2024-04-06T12:30:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.756+0000 7f921e7fc700 1 -- 172.21.15.189:0/3525143221 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9220003d80 con 0x7f9228159b20 2024-04-06T12:30:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.756+0000 7f922ff5e700 1 -- 172.21.15.189:0/3525143221 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f921800a640 con 0x7f9228159b20 2024-04-06T12:30:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.756+0000 7f921e7fc700 1 -- 172.21.15.189:0/3525143221 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9220004260 con 0x7f9228159b20 2024-04-06T12:30:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.756+0000 7f922ff5e700 1 -- 172.21.15.189:0/3525143221 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92280c4fc0 con 0x7f9228159b20 2024-04-06T12:30:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.757+0000 7f922ff5e700 1 -- 172.21.15.189:0/3525143221 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92280c54c0 con 0x7f9228159b20 2024-04-06T12:30:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.758+0000 7f921e7fc700 1 -- 172.21.15.189:0/3525143221 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 63) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f92200043c0 con 0x7f9228159b20 2024-04-06T12:30:24.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.758+0000 7f921e7fc700 1 --2- 172.21.15.189:0/3525143221 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f920c06adb0 0x7f920c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:24.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.758+0000 7f921e7fc700 1 -- 172.21.15.189:0/3525143221 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9220016070 con 0x7f9228159b20 2024-04-06T12:30:24.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.758+0000 7f922dcfa700 1 --2- 172.21.15.189:0/3525143221 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f920c06adb0 0x7f920c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:24.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.758+0000 7f922dcfa700 1 --2- 172.21.15.189:0/3525143221 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f920c06adb0 0x7f920c06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9224007fd0 tx=0x7f9224007dc0 comp rx=0 tx=0).ready entity=mgr.7875 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:24.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:24.762+0000 7f921e7fc700 1 -- 172.21.15.189:0/3525143221 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9220057270 con 0x7f9228159b20 2024-04-06T12:30:25.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.015+0000 7f922ff5e700 1 -- 172.21.15.189:0/3525143221 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f9228074990 con 0x7f9228159b20 2024-04-06T12:30:25.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.580+0000 7f921e7fc700 1 -- 172.21.15.189:0/3525143221 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 64) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f922008c0c0 con 0x7f9228159b20 2024-04-06T12:30:25.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.581+0000 7f921e7fc700 1 -- 172.21.15.189:0/3525143221 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v64) v1 ==== 92+0+0 (secure 0 0 0) 0x7f9220076010 con 0x7f9228159b20 2024-04-06T12:30:25.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.584+0000 7f922ff5e700 1 -- 172.21.15.189:0/3525143221 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f920c06adb0 msgr2=0x7f920c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:25.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.584+0000 7f922ff5e700 1 --2- 172.21.15.189:0/3525143221 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f920c06adb0 0x7f920c06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9224007fd0 tx=0x7f9224007dc0 comp rx=0 tx=0).stop 2024-04-06T12:30:25.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.584+0000 7f922ff5e700 1 -- 172.21.15.189:0/3525143221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228159b20 msgr2=0x7f92280c57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:25.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.584+0000 7f922ff5e700 1 --2- 172.21.15.189:0/3525143221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228159b20 0x7f92280c57c0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f922000e580 tx=0x7f922000e890 comp rx=0 tx=0).stop 2024-04-06T12:30:25.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.584+0000 7f922ff5e700 1 -- 172.21.15.189:0/3525143221 shutdown_connections 2024-04-06T12:30:25.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.584+0000 7f922ff5e700 1 --2- 172.21.15.189:0/3525143221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228159b20 0x7f92280c57c0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:25.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.584+0000 7f922ff5e700 1 --2- 172.21.15.189:0/3525143221 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f922806e470 0x7f922814bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:25.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.584+0000 7f922ff5e700 1 --2- 172.21.15.189:0/3525143221 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f920c06adb0 0x7f920c06d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:25.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.584+0000 7f922ff5e700 1 --2- 172.21.15.189:0/3525143221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92280cdc60 0x7f92280c43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:25.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.584+0000 7f922ff5e700 1 -- 172.21.15.189:0/3525143221 >> 172.21.15.189:0/3525143221 conn(0x7f92280cc290 msgr2=0x7f92280ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:25.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.584+0000 7f922ff5e700 1 -- 172.21.15.189:0/3525143221 shutdown_connections 2024-04-06T12:30:25.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.584+0000 7f922ff5e700 1 -- 172.21.15.189:0/3525143221 wait complete. 2024-04-06T12:30:25.595 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-06T12:30:25.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.728+0000 7f1b13a15700 1 Processor -- start 2024-04-06T12:30:25.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.728+0000 7f1b13a15700 1 -- start start 2024-04-06T12:30:25.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.729+0000 7f1b13a15700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1b0c15d590 0x7f1b0c0b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:25.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.729+0000 7f1b13a15700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1b0c0b4ed0 0x7f1b0c0b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:25.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.729+0000 7f1b13a15700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b0c0b6580 0x7f1b0c0b69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:25.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.729+0000 7f1b13a15700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b0c0b58c0 con 0x7f1b0c0b4ed0 2024-04-06T12:30:25.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.729+0000 7f1b13a15700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b0c0b5cc0 con 0x7f1b0c15d590 2024-04-06T12:30:25.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.729+0000 7f1b13a15700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c0d1500 con 0x7f1b0c0b6580 2024-04-06T12:30:25.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.729+0000 7f1b10fb0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1b0c0b4ed0 0x7f1b0c0b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:25.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.729+0000 7f1b117b1700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1b0c15d590 0x7f1b0c0b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:25.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.729+0000 7f1b10fb0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1b0c0b4ed0 0x7f1b0c0b5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40856/0 (socket says 172.21.15.189:40856) 2024-04-06T12:30:25.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.729+0000 7f1b10fb0700 1 -- 172.21.15.189:0/1603370982 learned_addr learned my addr 172.21.15.189:0/1603370982 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:25.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.729+0000 7f1b11fb2700 1 --2- 172.21.15.189:0/1603370982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b0c0b6580 0x7f1b0c0b69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:25.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.729+0000 7f1b10fb0700 1 -- 172.21.15.189:0/1603370982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b0c0b6580 msgr2=0x7f1b0c0b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:25.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.729+0000 7f1b10fb0700 1 --2- 172.21.15.189:0/1603370982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b0c0b6580 0x7f1b0c0b69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:25.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.729+0000 7f1b10fb0700 1 -- 172.21.15.189:0/1603370982 >> v2:172.21.15.192:3300/0 conn(0x7f1b0c15d590 msgr2=0x7f1b0c0b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:25.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.729+0000 7f1b10fb0700 1 --2- 172.21.15.189:0/1603370982 >> v2:172.21.15.192:3300/0 conn(0x7f1b0c15d590 0x7f1b0c0b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:25.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.729+0000 7f1b10fb0700 1 -- 172.21.15.189:0/1603370982 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0c0d17e0 con 0x7f1b0c0b4ed0 2024-04-06T12:30:25.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.730+0000 7f1b10fb0700 1 --2- 172.21.15.189:0/1603370982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b0c0b4ed0 0x7f1b0c0b5380 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c0b5fe0 tx=0x7f1afc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d05661eb6d1736c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:25.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.730+0000 7f1b03fff700 1 -- 172.21.15.189:0/1603370982 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1afc011830 con 0x7f1b0c0b4ed0 2024-04-06T12:30:25.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.730+0000 7f1b03fff700 1 -- 172.21.15.189:0/1603370982 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1afc011990 con 0x7f1b0c0b4ed0 2024-04-06T12:30:25.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.730+0000 7f1b03fff700 1 -- 172.21.15.189:0/1603370982 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1afc011d20 con 0x7f1b0c0b4ed0 2024-04-06T12:30:25.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.730+0000 7f1b13a15700 1 -- 172.21.15.189:0/1603370982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b0c0b4ed0 msgr2=0x7f1b0c0b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:25.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.730+0000 7f1b13a15700 1 --2- 172.21.15.189:0/1603370982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b0c0b4ed0 0x7f1b0c0b5380 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c0b5fe0 tx=0x7f1afc009bd0 comp rx=0 tx=0).stop 2024-04-06T12:30:25.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.730+0000 7f1b13a15700 1 -- 172.21.15.189:0/1603370982 shutdown_connections 2024-04-06T12:30:25.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.730+0000 7f1b13a15700 1 --2- 172.21.15.189:0/1603370982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b0c0b6580 0x7f1b0c0b69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:25.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.730+0000 7f1b13a15700 1 --2- 172.21.15.189:0/1603370982 >> v2:172.21.15.192:3300/0 conn(0x7f1b0c15d590 0x7f1b0c0b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:25.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.730+0000 7f1b13a15700 1 --2- 172.21.15.189:0/1603370982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b0c0b4ed0 0x7f1b0c0b5380 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:25.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.730+0000 7f1b13a15700 1 -- 172.21.15.189:0/1603370982 >> 172.21.15.189:0/1603370982 conn(0x7f1b0c06bbc0 msgr2=0x7f1b0c06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:25.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.731+0000 7f1b13a15700 1 -- 172.21.15.189:0/1603370982 shutdown_connections 2024-04-06T12:30:25.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.731+0000 7f1b13a15700 1 -- 172.21.15.189:0/1603370982 wait complete. 2024-04-06T12:30:25.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.732+0000 7f1b13a15700 1 Processor -- start 2024-04-06T12:30:25.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.732+0000 7f1b13a15700 1 -- start start 2024-04-06T12:30:25.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.733+0000 7f1b13a15700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b0c0b4ed0 0x7f1b0c1f30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:25.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.733+0000 7f1b13a15700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b0c0b6580 0x7f1b0c1f7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:25.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.733+0000 7f1b13a15700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b0c15d590 0x7f1b0c153610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:25.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.733+0000 7f1b13a15700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c1f7640 con 0x7f1b0c0b4ed0 2024-04-06T12:30:25.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.733+0000 7f1b13a15700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c1f4060 con 0x7f1b0c0b6580 2024-04-06T12:30:25.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.733+0000 7f1b13a15700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c153b50 con 0x7f1b0c15d590 2024-04-06T12:30:25.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.733+0000 7f1b117b1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b0c0b4ed0 0x7f1b0c1f30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:25.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.733+0000 7f1b117b1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b0c0b4ed0 0x7f1b0c1f30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40866/0 (socket says 172.21.15.189:40866) 2024-04-06T12:30:25.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.733+0000 7f1b117b1700 1 -- 172.21.15.189:0/4168407876 learned_addr learned my addr 172.21.15.189:0/4168407876 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:25.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.733+0000 7f1b10fb0700 1 --2- 172.21.15.189:0/4168407876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b0c0b6580 0x7f1b0c1f7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:25.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.733+0000 7f1b11fb2700 1 --2- 172.21.15.189:0/4168407876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b0c15d590 0x7f1b0c153610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:25.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.733+0000 7f1b117b1700 1 -- 172.21.15.189:0/4168407876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b0c15d590 msgr2=0x7f1b0c153610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:25.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.733+0000 7f1b117b1700 1 --2- 172.21.15.189:0/4168407876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b0c15d590 0x7f1b0c153610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:25.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.733+0000 7f1b117b1700 1 -- 172.21.15.189:0/4168407876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b0c0b6580 msgr2=0x7f1b0c1f7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:25.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.733+0000 7f1b117b1700 1 --2- 172.21.15.189:0/4168407876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b0c0b6580 0x7f1b0c1f7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:25.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.733+0000 7f1b117b1700 1 -- 172.21.15.189:0/4168407876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1afc009550 con 0x7f1b0c0b4ed0 2024-04-06T12:30:25.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.734+0000 7f1b117b1700 1 --2- 172.21.15.189:0/4168407876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b0c0b4ed0 0x7f1b0c1f30d0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f1b0800e8a0 tx=0x7f1b0800ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:25.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.734+0000 7f1b01ffb700 1 -- 172.21.15.189:0/4168407876 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b08009590 con 0x7f1b0c0b4ed0 2024-04-06T12:30:25.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.734+0000 7f1b01ffb700 1 -- 172.21.15.189:0/4168407876 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b08013680 con 0x7f1b0c0b4ed0 2024-04-06T12:30:25.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.734+0000 7f1b13a15700 1 -- 172.21.15.189:0/4168407876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b0c1f37c0 con 0x7f1b0c0b4ed0 2024-04-06T12:30:25.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.734+0000 7f1b01ffb700 1 -- 172.21.15.189:0/4168407876 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b080139f0 con 0x7f1b0c0b4ed0 2024-04-06T12:30:25.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.734+0000 7f1b13a15700 1 -- 172.21.15.189:0/4168407876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b0c1f3d80 con 0x7f1b0c0b4ed0 2024-04-06T12:30:25.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.735+0000 7f1b13a15700 1 -- 172.21.15.189:0/4168407876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b0c074990 con 0x7f1b0c0b4ed0 2024-04-06T12:30:25.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.736+0000 7f1b01ffb700 1 -- 172.21.15.189:0/4168407876 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 64) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f1b08013b50 con 0x7f1b0c0b4ed0 2024-04-06T12:30:25.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.736+0000 7f1b01ffb700 1 --2- 172.21.15.189:0/4168407876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1af006ae00 0x7f1af006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:25.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.736+0000 7f1b10fb0700 1 -- 172.21.15.189:0/4168407876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1af006ae00 msgr2=0x7f1af006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:25.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.736+0000 7f1b01ffb700 1 -- 172.21.15.189:0/4168407876 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1b08089470 con 0x7f1b0c0b4ed0 2024-04-06T12:30:25.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.736+0000 7f1b10fb0700 1 --2- 172.21.15.189:0/4168407876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1af006ae00 0x7f1af006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:30:25.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.740+0000 7f1b01ffb700 1 -- 172.21.15.189:0/4168407876 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1b08054400 con 0x7f1b0c0b4ed0 2024-04-06T12:30:25.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.937+0000 7f1b10fb0700 1 -- 172.21.15.189:0/4168407876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1af006ae00 msgr2=0x7f1af006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:25.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.937+0000 7f1b10fb0700 1 --2- 172.21.15.189:0/4168407876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1af006ae00 0x7f1af006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:30:25.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.944+0000 7f1b13a15700 1 -- 172.21.15.189:0/4168407876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f1b0c0c9db0 con 0x7f1b0c0b4ed0 2024-04-06T12:30:25.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.948+0000 7f1b01ffb700 1 -- 172.21.15.189:0/4168407876 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v41)=0 v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1b08057a20 con 0x7f1b0c0b4ed0 2024-04-06T12:30:25.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.951+0000 7f1b13a15700 1 -- 172.21.15.189:0/4168407876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1af006ae00 msgr2=0x7f1af006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:30:25.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.951+0000 7f1b13a15700 1 --2- 172.21.15.189:0/4168407876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1af006ae00 0x7f1af006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:25.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.951+0000 7f1b13a15700 1 -- 172.21.15.189:0/4168407876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b0c0b4ed0 msgr2=0x7f1b0c1f30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:25.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.951+0000 7f1b13a15700 1 --2- 172.21.15.189:0/4168407876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b0c0b4ed0 0x7f1b0c1f30d0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f1b0800e8a0 tx=0x7f1b0800ebb0 comp rx=0 tx=0).stop 2024-04-06T12:30:25.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.951+0000 7f1b13a15700 1 -- 172.21.15.189:0/4168407876 shutdown_connections 2024-04-06T12:30:25.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.951+0000 7f1b13a15700 1 --2- 172.21.15.189:0/4168407876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b0c15d590 0x7f1b0c153610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:25.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.951+0000 7f1b13a15700 1 --2- 172.21.15.189:0/4168407876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b0c0b4ed0 0x7f1b0c1f30d0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:25.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.951+0000 7f1b13a15700 1 --2- 172.21.15.189:0/4168407876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1af006ae00 0x7f1af006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:25.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.951+0000 7f1b13a15700 1 --2- 172.21.15.189:0/4168407876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b0c0b6580 0x7f1b0c1f7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:25.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.951+0000 7f1b13a15700 1 -- 172.21.15.189:0/4168407876 >> 172.21.15.189:0/4168407876 conn(0x7f1b0c06bbc0 msgr2=0x7f1b0c0b7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:25.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.951+0000 7f1b13a15700 1 -- 172.21.15.189:0/4168407876 shutdown_connections 2024-04-06T12:30:25.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:25.951+0000 7f1b13a15700 1 -- 172.21.15.189:0/4168407876 wait complete. 2024-04-06T12:30:25.961 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-06T12:30:25.964 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:25.967+0000 7fa9a4dba200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:30:25.981 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:25.984+0000 7fba67a4e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:30:26.043 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:26.046+0000 7fa9a4dba200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:30:26.065 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:26.068+0000 7fba67a4e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:30:26.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.092+0000 7f62d0aa1700 1 Processor -- start 2024-04-06T12:30:26.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.092+0000 7f62d0aa1700 1 -- start start 2024-04-06T12:30:26.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.093+0000 7f62d0aa1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62cc15d7b0 0x7f62cc0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:26.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.093+0000 7f62d0aa1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f62cc06a730 0x7f62cc06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:26.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.093+0000 7f62d0aa1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc0b3990 con 0x7f62cc15ddf0 2024-04-06T12:30:26.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.093+0000 7f62d0aa1700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62cc06b0f0 con 0x7f62cc06a730 2024-04-06T12:30:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.093+0000 7f62d0aa1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62cc0b3060 con 0x7f62cc15d7b0 2024-04-06T12:30:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.093+0000 7f62c9d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f62cc06a730 0x7f62cc06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.093+0000 7f62cad9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f62cc15ddf0 0x7f62cc15e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45588/0 (socket says 172.21.15.189:45588) 2024-04-06T12:30:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.093+0000 7f62cad9d700 1 -- 172.21.15.189:0/3224061269 learned_addr learned my addr 172.21.15.189:0/3224061269 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.093+0000 7f62ca59c700 1 --2- 172.21.15.189:0/3224061269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62cc15d7b0 0x7f62cc0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.093+0000 7f62c9d9b700 1 -- 172.21.15.189:0/3224061269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62cc15d7b0 msgr2=0x7f62cc0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:30:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.093+0000 7f62c9d9b700 1 --2- 172.21.15.189:0/3224061269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62cc15d7b0 0x7f62cc0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.093+0000 7f62c9d9b700 1 -- 172.21.15.189:0/3224061269 >> v1:172.21.15.189:6789/0 conn(0x7f62cc15ddf0 legacy=0x7f62cc15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.093+0000 7f62c9d9b700 1 -- 172.21.15.189:0/3224061269 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62cc0b3340 con 0x7f62cc06a730 2024-04-06T12:30:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.093+0000 7f62c9d9b700 1 --2- 172.21.15.189:0/3224061269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62cc06a730 0x7f62cc06abb0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f62c0009840 tx=0x7f62c0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5493934c770a16f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.094+0000 7f62c959a700 1 -- 172.21.15.189:0/3224061269 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c0011830 con 0x7f62cc06a730 2024-04-06T12:30:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.094+0000 7f62c959a700 1 -- 172.21.15.189:0/3224061269 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c0011990 con 0x7f62cc06a730 2024-04-06T12:30:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.094+0000 7f62c959a700 1 -- 172.21.15.189:0/3224061269 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c0011d20 con 0x7f62cc06a730 2024-04-06T12:30:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.094+0000 7f62d0aa1700 1 -- 172.21.15.189:0/3224061269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62cc06a730 msgr2=0x7f62cc06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:26.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.094+0000 7f62d0aa1700 1 --2- 172.21.15.189:0/3224061269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62cc06a730 0x7f62cc06abb0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f62c0009840 tx=0x7f62c0009b50 comp rx=0 tx=0).stop 2024-04-06T12:30:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.094+0000 7f62d0aa1700 1 -- 172.21.15.189:0/3224061269 shutdown_connections 2024-04-06T12:30:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.094+0000 7f62d0aa1700 1 --2- 172.21.15.189:0/3224061269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62cc15d7b0 0x7f62cc0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.094+0000 7f62d0aa1700 1 --2- 172.21.15.189:0/3224061269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62cc06a730 0x7f62cc06abb0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.094+0000 7f62d0aa1700 1 -- 172.21.15.189:0/3224061269 >> 172.21.15.189:0/3224061269 conn(0x7f62cc0d00b0 msgr2=0x7f62cc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.094+0000 7f62d0aa1700 1 -- 172.21.15.189:0/3224061269 shutdown_connections 2024-04-06T12:30:26.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.094+0000 7f62d0aa1700 1 -- 172.21.15.189:0/3224061269 wait complete. 2024-04-06T12:30:26.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.096+0000 7f62d0aa1700 1 Processor -- start 2024-04-06T12:30:26.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.096+0000 7f62d0aa1700 1 -- start start 2024-04-06T12:30:26.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.096+0000 7f62d0aa1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62cc06a730 0x7f62cc0cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:26.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.096+0000 7f62d0aa1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62cc15d7b0 0x7f62cc14f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:26.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.096+0000 7f62d0aa1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62cc15ddf0 0x7f62cc0ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:26.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.096+0000 7f62d0aa1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62cc157ea0 con 0x7f62cc15d7b0 2024-04-06T12:30:26.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.096+0000 7f62d0aa1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62cc150e20 con 0x7f62cc06a730 2024-04-06T12:30:26.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.096+0000 7f62d0aa1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62cc0cee40 con 0x7f62cc15ddf0 2024-04-06T12:30:26.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.097+0000 7f62c9d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62cc15d7b0 0x7f62cc14f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:26.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.097+0000 7f62c9d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62cc15d7b0 0x7f62cc14f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40884/0 (socket says 172.21.15.189:40884) 2024-04-06T12:30:26.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.097+0000 7f62c9d9b700 1 -- 172.21.15.189:0/3495274777 learned_addr learned my addr 172.21.15.189:0/3495274777 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:26.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.097+0000 7f62ca59c700 1 --2- 172.21.15.189:0/3495274777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62cc06a730 0x7f62cc0cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:26.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.097+0000 7f62cad9d700 1 --2- 172.21.15.189:0/3495274777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62cc15ddf0 0x7f62cc0ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:26.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.097+0000 7f62c9d9b700 1 -- 172.21.15.189:0/3495274777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62cc15ddf0 msgr2=0x7f62cc0ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:26.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.097+0000 7f62c9d9b700 1 --2- 172.21.15.189:0/3495274777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62cc15ddf0 0x7f62cc0ccb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:26.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.097+0000 7f62c9d9b700 1 -- 172.21.15.189:0/3495274777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62cc06a730 msgr2=0x7f62cc0cc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:26.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.097+0000 7f62c9d9b700 1 --2- 172.21.15.189:0/3495274777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62cc06a730 0x7f62cc0cc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:26.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.097+0000 7f62c9d9b700 1 -- 172.21.15.189:0/3495274777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c00094a0 con 0x7f62cc15d7b0 2024-04-06T12:30:26.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.097+0000 7f62c9d9b700 1 --2- 172.21.15.189:0/3495274777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62cc15d7b0 0x7f62cc14f930 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f62c0009fd0 tx=0x7f62c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:26.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.097+0000 7f62baffd700 1 -- 172.21.15.189:0/3495274777 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c00036a0 con 0x7f62cc15d7b0 2024-04-06T12:30:26.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.097+0000 7f62baffd700 1 -- 172.21.15.189:0/3495274777 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c001f070 con 0x7f62cc15d7b0 2024-04-06T12:30:26.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.097+0000 7f62d0aa1700 1 -- 172.21.15.189:0/3495274777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62cc0cf120 con 0x7f62cc15d7b0 2024-04-06T12:30:26.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.097+0000 7f62baffd700 1 -- 172.21.15.189:0/3495274777 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c0003a40 con 0x7f62cc15d7b0 2024-04-06T12:30:26.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.097+0000 7f62d0aa1700 1 -- 172.21.15.189:0/3495274777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62cc14fe70 con 0x7f62cc15d7b0 2024-04-06T12:30:26.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.098+0000 7f62d0aa1700 1 -- 172.21.15.189:0/3495274777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62cc0cf360 con 0x7f62cc15d7b0 2024-04-06T12:30:26.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.099+0000 7f62baffd700 1 -- 172.21.15.189:0/3495274777 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 64) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f62c0003ba0 con 0x7f62cc15d7b0 2024-04-06T12:30:26.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.099+0000 7f62baffd700 1 --2- 172.21.15.189:0/3495274777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f62ac06ae00 0x7f62ac06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:26.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.099+0000 7f62ca59c700 1 -- 172.21.15.189:0/3495274777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f62ac06ae00 msgr2=0x7f62ac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:26.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.099+0000 7f62baffd700 1 -- 172.21.15.189:0/3495274777 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f62c0021070 con 0x7f62cc15d7b0 2024-04-06T12:30:26.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.099+0000 7f62ca59c700 1 --2- 172.21.15.189:0/3495274777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f62ac06ae00 0x7f62ac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:30:26.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.104+0000 7f62baffd700 1 -- 172.21.15.189:0/3495274777 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f62cc0cf360 con 0x7f62cc15d7b0 2024-04-06T12:30:26.116 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:26.119+0000 7fa9a4dba200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:30:26.143 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:26.146+0000 7fba67a4e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:30:26.183 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:26.186+0000 7fa9a4dba200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:30:26.214 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:26.217+0000 7fba67a4e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:30:26.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.300+0000 7f62ca59c700 1 -- 172.21.15.189:0/3495274777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f62ac06ae00 msgr2=0x7f62ac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:26.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.300+0000 7f62ca59c700 1 --2- 172.21.15.189:0/3495274777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f62ac06ae00 0x7f62ac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:30:26.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.308+0000 7f62d0aa1700 1 -- 172.21.15.189:0/3495274777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f62cc157920 con 0x7f62cc15d7b0 2024-04-06T12:30:26.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.313+0000 7f62baffd700 1 -- 172.21.15.189:0/3495274777 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v42)=0 v42) v1 ==== 133+0+0 (secure 0 0 0) 0x7f62cc157920 con 0x7f62cc15d7b0 2024-04-06T12:30:26.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.315+0000 7f62d0aa1700 1 -- 172.21.15.189:0/3495274777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f62ac06ae00 msgr2=0x7f62ac06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:30:26.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.315+0000 7f62d0aa1700 1 --2- 172.21.15.189:0/3495274777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f62ac06ae00 0x7f62ac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:26.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.315+0000 7f62d0aa1700 1 -- 172.21.15.189:0/3495274777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62cc15d7b0 msgr2=0x7f62cc14f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:26.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.315+0000 7f62d0aa1700 1 --2- 172.21.15.189:0/3495274777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62cc15d7b0 0x7f62cc14f930 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f62c0009fd0 tx=0x7f62c0004750 comp rx=0 tx=0).stop 2024-04-06T12:30:26.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.315+0000 7f62d0aa1700 1 -- 172.21.15.189:0/3495274777 shutdown_connections 2024-04-06T12:30:26.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.315+0000 7f62d0aa1700 1 --2- 172.21.15.189:0/3495274777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62cc15ddf0 0x7f62cc0ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:26.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.315+0000 7f62d0aa1700 1 --2- 172.21.15.189:0/3495274777 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f62ac06ae00 0x7f62ac06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:26.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.315+0000 7f62d0aa1700 1 --2- 172.21.15.189:0/3495274777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62cc06a730 0x7f62cc0cc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:26.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.315+0000 7f62d0aa1700 1 --2- 172.21.15.189:0/3495274777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62cc15d7b0 0x7f62cc14f930 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:26.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.315+0000 7f62d0aa1700 1 -- 172.21.15.189:0/3495274777 >> 172.21.15.189:0/3495274777 conn(0x7f62cc0d00b0 msgr2=0x7f62cc06fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:26.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.315+0000 7f62d0aa1700 1 -- 172.21.15.189:0/3495274777 shutdown_connections 2024-04-06T12:30:26.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.315+0000 7f62d0aa1700 1 -- 172.21.15.189:0/3495274777 wait complete. 2024-04-06T12:30:26.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:26.325 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-06T12:30:26.358 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:26.361+0000 7fa9a4dba200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:30:26.386 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-04-06T12:30:26.399 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:26.402+0000 7fba67a4e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:30:26.514 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:26.517+0000 7fa9a4dba200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:30:26.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.529+0000 7f73478a4700 1 Processor -- start 2024-04-06T12:30:26.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.529+0000 7f73478a4700 1 -- start start 2024-04-06T12:30:26.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.530+0000 7f73478a4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f73400cd3e0 0x7f73400cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:26.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.530+0000 7f73478a4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734006bcd0 con 0x7f734006b8c0 2024-04-06T12:30:26.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.530+0000 7f73478a4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734006ceb0 con 0x7f734015d7c0 2024-04-06T12:30:26.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.530+0000 7f73478a4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73400cdfa0 con 0x7f73400cd3e0 2024-04-06T12:30:26.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.530+0000 7f7345e41700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f734006b8c0 0x7f734006dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45604/0 (socket says 172.21.15.189:45604) 2024-04-06T12:30:26.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.530+0000 7f7345e41700 1 -- 172.21.15.189:0/1138238275 learned_addr learned my addr 172.21.15.189:0/1138238275 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:26.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.530+0000 7f7345640700 1 --2- 172.21.15.189:0/1138238275 >> v2:172.21.15.192:3300/0 conn(0x7f73400cd3e0 0x7f73400cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:26.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.530+0000 7f7337fff700 1 -- 172.21.15.189:0/1138238275 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391097995 0 0) 0x7f734006bcd0 con 0x7f734006b8c0 2024-04-06T12:30:26.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.531+0000 7f7337fff700 1 -- 172.21.15.189:0/1138238275 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f732c0035b0 con 0x7f734006b8c0 2024-04-06T12:30:26.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.531+0000 7f7345640700 1 -- 172.21.15.189:0/1138238275 >> v1:172.21.15.192:6789/0 conn(0x7f734015d7c0 legacy=0x7f734006b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:26.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.531+0000 7f7345640700 1 -- 172.21.15.189:0/1138238275 >> v1:172.21.15.189:6789/0 conn(0x7f734006b8c0 legacy=0x7f734006dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:26.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.531+0000 7f7345640700 1 -- 172.21.15.189:0/1138238275 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734006d750 con 0x7f73400cd3e0 2024-04-06T12:30:26.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.531+0000 7f7345640700 1 --2- 172.21.15.189:0/1138238275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73400cd3e0 0x7f73400cd7c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f7330009840 tx=0x7f7330009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a71d8e566e64211d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:26.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.531+0000 7f7337fff700 1 -- 172.21.15.189:0/1138238275 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7330011830 con 0x7f73400cd3e0 2024-04-06T12:30:26.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.531+0000 7f7337fff700 1 -- 172.21.15.189:0/1138238275 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7330011990 con 0x7f73400cd3e0 2024-04-06T12:30:26.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.531+0000 7f7337fff700 1 -- 172.21.15.189:0/1138238275 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7330011d00 con 0x7f73400cd3e0 2024-04-06T12:30:26.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.532+0000 7f73478a4700 1 -- 172.21.15.189:0/1138238275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73400cd3e0 msgr2=0x7f73400cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:26.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.532+0000 7f73478a4700 1 --2- 172.21.15.189:0/1138238275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73400cd3e0 0x7f73400cd7c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f7330009840 tx=0x7f7330009b50 comp rx=0 tx=0).stop 2024-04-06T12:30:26.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.532+0000 7f73478a4700 1 -- 172.21.15.189:0/1138238275 shutdown_connections 2024-04-06T12:30:26.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.532+0000 7f73478a4700 1 --2- 172.21.15.189:0/1138238275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73400cd3e0 0x7f73400cd7c0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:26.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.532+0000 7f73478a4700 1 -- 172.21.15.189:0/1138238275 >> 172.21.15.189:0/1138238275 conn(0x7f73400cc290 msgr2=0x7f73400cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:26.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.532+0000 7f73478a4700 1 -- 172.21.15.189:0/1138238275 shutdown_connections 2024-04-06T12:30:26.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.532+0000 7f73478a4700 1 -- 172.21.15.189:0/1138238275 wait complete. 2024-04-06T12:30:26.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.533+0000 7f73478a4700 1 Processor -- start 2024-04-06T12:30:26.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.533+0000 7f73478a4700 1 -- start start 2024-04-06T12:30:26.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f73478a4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f734006b8c0 0x7f7340146b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:26.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f73478a4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73400cd3e0 0x7f73400b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:26.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f73478a4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f734015d7c0 0x7f734015e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:26.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f73478a4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73400b3bf0 con 0x7f734015d7c0 2024-04-06T12:30:26.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f73478a4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73400b3ed0 con 0x7f73400cd3e0 2024-04-06T12:30:26.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f73478a4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73400c33c0 con 0x7f734006b8c0 2024-04-06T12:30:26.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f7345e41700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f734015d7c0 0x7f734015e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:26.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f7345e41700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f734015d7c0 0x7f734015e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40900/0 (socket says 172.21.15.189:40900) 2024-04-06T12:30:26.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f7345e41700 1 -- 172.21.15.189:0/4209155169 learned_addr learned my addr 172.21.15.189:0/4209155169 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:26.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f7344e3f700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73400cd3e0 0x7f73400b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:26.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f7345640700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f734006b8c0 0x7f7340146b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:26.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f7344e3f700 1 -- 172.21.15.189:0/4209155169 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f734006b8c0 msgr2=0x7f7340146b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:26.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f7344e3f700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f734006b8c0 0x7f7340146b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:26.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f7344e3f700 1 -- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f734015d7c0 msgr2=0x7f734015e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:26.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f7344e3f700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f734015d7c0 0x7f734015e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:26.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f7344e3f700 1 -- 172.21.15.189:0/4209155169 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73300094a0 con 0x7f73400cd3e0 2024-04-06T12:30:26.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f7345e41700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f734015d7c0 0x7f734015e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:30:26.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f7345640700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f734006b8c0 0x7f7340146b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:30:26.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f7344e3f700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73400cd3e0 0x7f73400b32f0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f733c00c440 tx=0x7f733c00c750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:26.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.534+0000 7f7335ffb700 1 -- 172.21.15.189:0/4209155169 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c0052d0 con 0x7f73400cd3e0 2024-04-06T12:30:26.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.535+0000 7f7335ffb700 1 -- 172.21.15.189:0/4209155169 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733c005430 con 0x7f73400cd3e0 2024-04-06T12:30:26.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.535+0000 7f73478a4700 1 -- 172.21.15.189:0/4209155169 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73400c36a0 con 0x7f73400cd3e0 2024-04-06T12:30:26.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.535+0000 7f7335ffb700 1 -- 172.21.15.189:0/4209155169 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c006aa0 con 0x7f73400cd3e0 2024-04-06T12:30:26.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.535+0000 7f73478a4700 1 -- 172.21.15.189:0/4209155169 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73401470d0 con 0x7f73400cd3e0 2024-04-06T12:30:26.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.536+0000 7f73478a4700 1 -- 172.21.15.189:0/4209155169 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73400c38c0 con 0x7f73400cd3e0 2024-04-06T12:30:26.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.536+0000 7f7335ffb700 1 -- 172.21.15.189:0/4209155169 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 64) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f733c003680 con 0x7f73400cd3e0 2024-04-06T12:30:26.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.536+0000 7f7335ffb700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f732406ae00 0x7f732406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:26.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.536+0000 7f7345640700 1 -- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f732406ae00 msgr2=0x7f732406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:26.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.536+0000 7f7335ffb700 1 -- 172.21.15.189:0/4209155169 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f733c0886d0 con 0x7f73400cd3e0 2024-04-06T12:30:26.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.536+0000 7f7345640700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f732406ae00 0x7f732406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:30:26.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.541+0000 7f7335ffb700 1 -- 172.21.15.189:0/4209155169 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f733c0535e0 con 0x7f73400cd3e0 2024-04-06T12:30:26.567 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:26.570+0000 7fba67a4e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:30:26.626 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:26.629+0000 7fa9a4dba200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:30:26.681 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:26.684+0000 7fba67a4e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:30:26.706 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:26.709+0000 7fa9a4dba200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:30:26.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.737+0000 7f7345640700 1 -- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f732406ae00 msgr2=0x7f732406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:26.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.737+0000 7f7345640700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f732406ae00 0x7f732406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:30:26.766 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:26.769+0000 7fba67a4e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:30:26.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:26.771+0000 7f73478a4700 1 -- 172.21.15.189:0/4209155169 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f734006d100 con 0x7f732406ae00 2024-04-06T12:30:26.780 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:26.784+0000 7fa9a4dba200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:30:26.841 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:26.844+0000 7fba67a4e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:30:26.982 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:26.984+0000 7fa9a4dba200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:30:27.045 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:27.048+0000 7fba67a4e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:30:27.059 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:27.061+0000 7fa9a4dba200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:30:27.121 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:27.124+0000 7fba67a4e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:30:27.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:27.138+0000 7f7345640700 1 -- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f732406ae00 msgr2=0x7f732406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:27.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:27.138+0000 7f7345640700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f732406ae00 0x7f732406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-06T12:30:27.375 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:27.377+0000 7fa9a4dba200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:30:27.442 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:27.445+0000 7fba67a4e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:30:27.692 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:27.694+0000 7fa9a4dba200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:30:27.762 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:27.765+0000 7fba67a4e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:30:27.851 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:27.853+0000 7fa9a4dba200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:30:27.921 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:27.923+0000 7fba67a4e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:30:27.922 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:27.924+0000 7fa9a4dba200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:30:27.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:27.939+0000 7f7345640700 1 -- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f732406ae00 msgr2=0x7f732406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:27.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:27.939+0000 7f7345640700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f732406ae00 0x7f732406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-06T12:30:27.994 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:27.997+0000 7fba67a4e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:30:28.533 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:28.535+0000 7fa9a4dba200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:30:28.610 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:28.612+0000 7fa9a4dba200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:30:28.611 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:28.613+0000 7fba67a4e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:30:28.691 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:28.693+0000 7fba67a4e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:30:28.736 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:28.738+0000 7fa9a4dba200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:30:28.817 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:28.819+0000 7fba67a4e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:30:29.241 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:29.243+0000 7fa9a4dba200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:30:29.328 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:29.330+0000 7fba67a4e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:30:29.391 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:29.393+0000 7fa9a4dba200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:30:29.483 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:29.485+0000 7fba67a4e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:30:29.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:29.539+0000 7f7345640700 1 -- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f732406ae00 msgr2=0x7f732406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:30:29.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:29.539+0000 7f7345640700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f732406ae00 0x7f732406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-06T12:30:29.545 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:29.547+0000 7fa9a4dba200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:30:29.640 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:29.642+0000 7fba67a4e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:30:30.261 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:30.263+0000 7fa9a4dba200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:30:30.368 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:30.370+0000 7fba67a4e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:30:30.501 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:30.503+0000 7fa9a4dba200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:30:30.575 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:30.577+0000 7fa9a4dba200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:30:30.613 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:30.615+0000 7fba67a4e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:30:30.690 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:30.692+0000 7fba67a4e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:30:30.943 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:30:30.945+0000 7fa9a4dba200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:30:31.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:31.015+0000 7f7335ffb700 1 -- 172.21.15.189:0/4209155169 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mgrmap(e 65) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f733c086ab0 con 0x7f73400cd3e0 2024-04-06T12:30:31.068 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:31.070+0000 7fba67a4e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:30:31.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:31.091+0000 7f7335ffb700 1 -- 172.21.15.189:0/4209155169 <== mon.1 v2:172.21.15.192:3300/0 8 ==== mgrmap(e 66) v1 ==== 88873+0+0 (secure 0 0 0) 0x7f733c0539b0 con 0x7f73400cd3e0 2024-04-06T12:30:31.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:31.091+0000 7f7335ffb700 1 -- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f732406ae00 msgr2=0x7f732406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:30:31.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:31.091+0000 7f7335ffb700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f732406ae00 0x7f732406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:31.233 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:31.235+0000 7fb9dd962700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:31.233 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:31.235+0000 7fb9dd962700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:31.233 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:31.235+0000 7fb9dd962700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:31.233 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:31.235+0000 7fb9dd962700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:31.233 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:31.235+0000 7fb9dd962700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:31.234 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:31.235+0000 7fb9da95c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:31.234 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:31.235+0000 7fb9da95c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:31.234 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:31.235+0000 7fb9da95c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:31.234 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:31.235+0000 7fb9da95c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:31.234 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:30:31.235+0000 7fb9da95c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:30:32.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.096+0000 7f7335ffb700 1 -- 172.21.15.189:0/4209155169 <== mon.1 v2:172.21.15.192:3300/0 9 ==== mgrmap(e 67) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f733c054280 con 0x7f73400cd3e0 2024-04-06T12:30:32.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.096+0000 7f7335ffb700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f732406ae00 0x7f732406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:32.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.096+0000 7f7335ffb700 1 -- 172.21.15.189:0/4209155169 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f732401cd40 con 0x7f732406ae00 2024-04-06T12:30:32.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.097+0000 7f7345640700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f732406ae00 0x7f732406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:32.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.099+0000 7f7345640700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f732406ae00 0x7f732406d2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7330000c00 tx=0x7f7330008030 comp rx=0 tx=0).ready entity=mgr.8271 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:32.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.158+0000 7f7335ffb700 1 -- 172.21.15.189:0/4209155169 <== mgr.8271 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f732401cd40 con 0x7f732406ae00 2024-04-06T12:30:32.156 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path snap_test_dir1 2024-04-06T12:30:32.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.160+0000 7f73478a4700 1 -- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f732406ae00 msgr2=0x7f732406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:32.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.160+0000 7f73478a4700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f732406ae00 0x7f732406d2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7330000c00 tx=0x7f7330008030 comp rx=0 tx=0).stop 2024-04-06T12:30:32.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.160+0000 7f73478a4700 1 -- 172.21.15.189:0/4209155169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73400cd3e0 msgr2=0x7f73400b32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:32.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.160+0000 7f73478a4700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73400cd3e0 0x7f73400b32f0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f733c00c440 tx=0x7f733c00c750 comp rx=0 tx=0).stop 2024-04-06T12:30:32.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.161+0000 7f73478a4700 1 -- 172.21.15.189:0/4209155169 shutdown_connections 2024-04-06T12:30:32.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.161+0000 7f73478a4700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f734015d7c0 0x7f734015e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:32.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.161+0000 7f73478a4700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f734006b8c0 0x7f7340146b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:32.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.161+0000 7f73478a4700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f732406ae00 0x7f732406d2c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:32.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.161+0000 7f73478a4700 1 --2- 172.21.15.189:0/4209155169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73400cd3e0 0x7f73400b32f0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:32.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.161+0000 7f73478a4700 1 -- 172.21.15.189:0/4209155169 >> 172.21.15.189:0/4209155169 conn(0x7f73400cc290 msgr2=0x7f73400af2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:32.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.161+0000 7f73478a4700 1 -- 172.21.15.189:0/4209155169 shutdown_connections 2024-04-06T12:30:32.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.161+0000 7f73478a4700 1 -- 172.21.15.189:0/4209155169 wait complete. 2024-04-06T12:30:32.173 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-04-06T12:30:32.173 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-04-06-12_31_00 in ~28s... 2024-04-06T12:30:32.173 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2024-04-06T12:30:32.173 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-04-06T12:30:32.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.300+0000 7f51e7758700 1 Processor -- start 2024-04-06T12:30:32.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.300+0000 7f51e7758700 1 -- start start 2024-04-06T12:30:32.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.301+0000 7f51e7758700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f51e00b4660 0x7f51e00b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:32.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.301+0000 7f51e7758700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e00b5030 con 0x7f51e00b2fc0 2024-04-06T12:30:32.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.301+0000 7f51e7758700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e00cdbe0 con 0x7f51e015a550 2024-04-06T12:30:32.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.301+0000 7f51e7758700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e015a940 con 0x7f51e00b4660 2024-04-06T12:30:32.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.301+0000 7f51e5cf5700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f51e00b2fc0 0x7f51e00b33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55098/0 (socket says 172.21.15.189:55098) 2024-04-06T12:30:32.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.301+0000 7f51e5cf5700 1 -- 172.21.15.189:0/1673643381 learned_addr learned my addr 172.21.15.189:0/1673643381 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:30:32.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.301+0000 7f51e4cf3700 1 --2- 172.21.15.189:0/1673643381 >> v2:172.21.15.192:3300/0 conn(0x7f51e00b4660 0x7f51e00b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:32.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.301+0000 7f51d7fff700 1 -- 172.21.15.189:0/1673643381 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791805802 0 0) 0x7f51e00b5030 con 0x7f51e00b2fc0 2024-04-06T12:30:32.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.301+0000 7f51d7fff700 1 -- 172.21.15.189:0/1673643381 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51cc0035b0 con 0x7f51e00b2fc0 2024-04-06T12:30:32.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.301+0000 7f51e4cf3700 1 -- 172.21.15.189:0/1673643381 >> v1:172.21.15.192:6789/0 conn(0x7f51e015a550 legacy=0x7f51e00cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:32.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.301+0000 7f51e4cf3700 1 -- 172.21.15.189:0/1673643381 >> v1:172.21.15.189:6789/0 conn(0x7f51e00b2fc0 legacy=0x7f51e00b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:32.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.301+0000 7f51d7fff700 1 -- 172.21.15.189:0/1673643381 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1740169349 0 0) 0x7f51cc0035b0 con 0x7f51e00b2fc0 2024-04-06T12:30:32.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.301+0000 7f51e4cf3700 1 -- 172.21.15.189:0/1673643381 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e015ac00 con 0x7f51e00b4660 2024-04-06T12:30:32.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.302+0000 7f51e4cf3700 1 --2- 172.21.15.189:0/1673643381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51e00b4660 0x7f51e00b4af0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f51d0009840 tx=0x7f51d0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d1d724ddc024a3f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:32.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.302+0000 7f51d7fff700 1 -- 172.21.15.189:0/1673643381 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d80019f0 con 0x7f51e00b4660 2024-04-06T12:30:32.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.302+0000 7f51d7fff700 1 -- 172.21.15.189:0/1673643381 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d0004030 con 0x7f51e00b4660 2024-04-06T12:30:32.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.302+0000 7f51d7fff700 1 -- 172.21.15.189:0/1673643381 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d00043a0 con 0x7f51e00b4660 2024-04-06T12:30:32.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.302+0000 7f51e7758700 1 -- 172.21.15.189:0/1673643381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51e00b4660 msgr2=0x7f51e00b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:32.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.302+0000 7f51e7758700 1 --2- 172.21.15.189:0/1673643381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51e00b4660 0x7f51e00b4af0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f51d0009840 tx=0x7f51d0009b50 comp rx=0 tx=0).stop 2024-04-06T12:30:32.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.302+0000 7f51e7758700 1 -- 172.21.15.189:0/1673643381 shutdown_connections 2024-04-06T12:30:32.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.302+0000 7f51e7758700 1 --2- 172.21.15.189:0/1673643381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51e00b4660 0x7f51e00b4af0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:32.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.302+0000 7f51e7758700 1 -- 172.21.15.189:0/1673643381 >> 172.21.15.189:0/1673643381 conn(0x7f51e00d00b0 msgr2=0x7f51e00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:32.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.302+0000 7f51e7758700 1 -- 172.21.15.189:0/1673643381 shutdown_connections 2024-04-06T12:30:32.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.302+0000 7f51e7758700 1 -- 172.21.15.189:0/1673643381 wait complete. 2024-04-06T12:30:32.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.304+0000 7f51e7758700 1 Processor -- start 2024-04-06T12:30:32.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.304+0000 7f51e7758700 1 -- start start 2024-04-06T12:30:32.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.304+0000 7f51e7758700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51e00b2fc0 0x7f51e0145240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:32.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.304+0000 7f51e7758700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51e00b4660 0x7f51e01eff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:32.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.304+0000 7f51e7758700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51e01475c0 0x7f51e00cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:32.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.304+0000 7f51e7758700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e00accf0 con 0x7f51e00b2fc0 2024-04-06T12:30:32.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.304+0000 7f51e7758700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e0145b40 con 0x7f51e01475c0 2024-04-06T12:30:32.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.304+0000 7f51e7758700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e0145e20 con 0x7f51e00b4660 2024-04-06T12:30:32.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.304+0000 7f51e54f4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51e00b2fc0 0x7f51e0145240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:32.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.304+0000 7f51e54f4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51e00b2fc0 0x7f51e0145240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34064/0 (socket says 172.21.15.189:34064) 2024-04-06T12:30:32.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.304+0000 7f51e54f4700 1 -- 172.21.15.189:0/2698940893 learned_addr learned my addr 172.21.15.189:0/2698940893 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:30:32.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.305+0000 7f51e4cf3700 1 --2- 172.21.15.189:0/2698940893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51e00b4660 0x7f51e01eff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:32.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.305+0000 7f51e5cf5700 1 --2- 172.21.15.189:0/2698940893 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51e01475c0 0x7f51e00cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:32.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.305+0000 7f51e54f4700 1 -- 172.21.15.189:0/2698940893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51e00b4660 msgr2=0x7f51e01eff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:32.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.305+0000 7f51e54f4700 1 --2- 172.21.15.189:0/2698940893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51e00b4660 0x7f51e01eff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:32.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.305+0000 7f51e54f4700 1 -- 172.21.15.189:0/2698940893 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51e01475c0 msgr2=0x7f51e00cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:32.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.305+0000 7f51e54f4700 1 --2- 172.21.15.189:0/2698940893 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51e01475c0 0x7f51e00cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:32.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.305+0000 7f51e54f4700 1 -- 172.21.15.189:0/2698940893 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d00094a0 con 0x7f51e00b2fc0 2024-04-06T12:30:32.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.305+0000 7f51e4cf3700 1 --2- 172.21.15.189:0/2698940893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51e00b4660 0x7f51e01eff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:30:32.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.305+0000 7f51e54f4700 1 --2- 172.21.15.189:0/2698940893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51e00b2fc0 0x7f51e0145240 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00c430 tx=0x7f51dc00c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:32.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.305+0000 7f51d5ffb700 1 -- 172.21.15.189:0/2698940893 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc0052d0 con 0x7f51e00b2fc0 2024-04-06T12:30:32.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.305+0000 7f51d5ffb700 1 -- 172.21.15.189:0/2698940893 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc005450 con 0x7f51e00b2fc0 2024-04-06T12:30:32.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.305+0000 7f51d5ffb700 1 -- 172.21.15.189:0/2698940893 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc006aa0 con 0x7f51e00b2fc0 2024-04-06T12:30:32.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.305+0000 7f51e7758700 1 -- 172.21.15.189:0/2698940893 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e0146100 con 0x7f51e00b2fc0 2024-04-06T12:30:32.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.305+0000 7f51e7758700 1 -- 172.21.15.189:0/2698940893 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e0145780 con 0x7f51e00b2fc0 2024-04-06T12:30:32.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.306+0000 7f51e7758700 1 -- 172.21.15.189:0/2698940893 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e0145910 con 0x7f51e00b2fc0 2024-04-06T12:30:32.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.306+0000 7f51d5ffb700 1 -- 172.21.15.189:0/2698940893 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 67) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f51dc003680 con 0x7f51e00b2fc0 2024-04-06T12:30:32.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.307+0000 7f51d5ffb700 1 --2- 172.21.15.189:0/2698940893 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f51c406ab30 0x7f51c406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:30:32.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.307+0000 7f51d5ffb700 1 -- 172.21.15.189:0/2698940893 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f51dc088400 con 0x7f51e00b2fc0 2024-04-06T12:30:32.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.307+0000 7f51e4cf3700 1 --2- 172.21.15.189:0/2698940893 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f51c406ab30 0x7f51c406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:30:32.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.307+0000 7f51e4cf3700 1 --2- 172.21.15.189:0/2698940893 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f51c406ab30 0x7f51c406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f51d00090b0 tx=0x7f51d0008030 comp rx=0 tx=0).ready entity=mgr.8271 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:30:32.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.311+0000 7f51d5ffb700 1 -- 172.21.15.189:0/2698940893 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f51dc053680 con 0x7f51e00b2fc0 2024-04-06T12:30:32.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.528+0000 7f51e7758700 1 -- 172.21.15.189:0/2698940893 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f51e015b4a0 con 0x7f51c406ab30 2024-04-06T12:30:32.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.538+0000 7f51d5ffb700 1 -- 172.21.15.189:0/2698940893 <== mgr.8271 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f51d00036a0 con 0x7f51c406ab30 2024-04-06T12:30:32.536 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:30:32.536 INFO:teuthology.orchestra.run.smithi189.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2024-04-06T12:30:32.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.540+0000 7f51e7758700 1 -- 172.21.15.189:0/2698940893 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f51c406ab30 msgr2=0x7f51c406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:32.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.540+0000 7f51e7758700 1 --2- 172.21.15.189:0/2698940893 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f51c406ab30 0x7f51c406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f51d00090b0 tx=0x7f51d0008030 comp rx=0 tx=0).stop 2024-04-06T12:30:32.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.540+0000 7f51e7758700 1 -- 172.21.15.189:0/2698940893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51e00b2fc0 msgr2=0x7f51e0145240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:30:32.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.540+0000 7f51e7758700 1 --2- 172.21.15.189:0/2698940893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51e00b2fc0 0x7f51e0145240 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00c430 tx=0x7f51dc00c740 comp rx=0 tx=0).stop 2024-04-06T12:30:32.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.540+0000 7f51e7758700 1 -- 172.21.15.189:0/2698940893 shutdown_connections 2024-04-06T12:30:32.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.540+0000 7f51e7758700 1 --2- 172.21.15.189:0/2698940893 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f51c406ab30 0x7f51c406cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:32.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.540+0000 7f51e7758700 1 --2- 172.21.15.189:0/2698940893 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51e01475c0 0x7f51e00cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:32.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.540+0000 7f51e7758700 1 --2- 172.21.15.189:0/2698940893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51e00b2fc0 0x7f51e0145240 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:32.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.540+0000 7f51e7758700 1 --2- 172.21.15.189:0/2698940893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51e00b4660 0x7f51e01eff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:30:32.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.540+0000 7f51e7758700 1 -- 172.21.15.189:0/2698940893 >> 172.21.15.189:0/2698940893 conn(0x7f51e00d00b0 msgr2=0x7f51e0159850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:30:32.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.540+0000 7f51e7758700 1 -- 172.21.15.189:0/2698940893 shutdown_connections 2024-04-06T12:30:32.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:30:32.540+0000 7f51e7758700 1 -- 172.21.15.189:0/2698940893 wait complete. 2024-04-06T12:30:32.550 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2024-04-06T12:30:32.550 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{}]} 2024-04-06T12:30:32.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:32.551 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:32.569 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:32.569 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:32.569 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:33.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:33.570 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:33.591 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:33.591 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:33.591 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:34.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:34.592 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:34.615 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:34.615 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:34.615 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:35.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:35.616 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:35.638 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:35.638 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:35.638 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:36.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:36.639 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:36.661 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:36.661 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:36.661 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:37.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:37.663 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:37.685 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:37.685 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:37.685 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:38.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:38.687 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:38.709 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:38.709 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:38.709 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:39.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:39.710 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:39.733 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:39.733 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:39.733 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:40.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:40.733 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:40.755 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:40.755 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:40.755 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:41.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:41.756 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:41.779 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:41.779 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:41.779 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:42.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:42.780 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:42.803 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:42.803 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:42.803 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:43.036 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:30:43.039 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:30:43.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:43.805 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:43.828 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:43.828 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:43.828 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:44.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:44.829 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:44.851 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:44.851 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:44.851 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:45.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:45.852 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:45.874 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:45.874 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:45.874 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:46.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:46.875 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:46.897 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:46.897 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:46.897 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:47.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:47.899 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:47.921 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:47.922 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:47.922 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:48.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:48.923 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:48.945 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:48.945 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:48.945 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:49.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:49.946 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:49.968 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:49.968 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:49.968 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:50.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:50.969 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:50.991 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:50.991 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:50.991 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:51.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:51.992 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:52.015 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:52.015 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:52.015 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:53.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:53.016 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:53.039 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:53.039 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:53.039 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:54.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:54.040 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:54.062 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:54.062 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:54.063 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:55.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:55.063 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:55.086 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:55.087 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:55.087 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:56.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:56.088 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:56.110 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:56.110 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:56.110 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:57.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:57.112 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:57.134 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:57.134 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:57.134 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:58.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:58.135 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:58.157 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:58.157 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:58.157 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:30:59.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:30:59.158 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:30:59.180 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T12:30:59.180 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T12:30:59.180 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:31:00.166 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:00.169+0000 7fb9cd141700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:00.166 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:00.169+0000 7fb9cd141700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:00.166 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:00.169+0000 7fb9cd141700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:00.166 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:00.169+0000 7fb9cd141700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:00.166 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:00.169+0000 7fb9cd141700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:00.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:00.181 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:31:00.202 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_31_00_UTC 2024-04-06T12:31:00.202 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-12_31_00_UTC'} 2024-04-06T12:31:00.202 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-04-06-12_31_00_UTC'} 2024-04-06T12:31:00.202 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T12:31:00.202 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-04-06-12_31_00_UTC'] 2024-04-06T12:31:00.202 DEBUG:tasks.cephfs.test_snap_schedules:exec=1712406632.173125, now = 1712406660.202735, timo = 28 2024-04-06T12:31:01.204 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-04-06T12:31:01.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.345+0000 7f7980eb9700 1 Processor -- start 2024-04-06T12:31:01.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.345+0000 7f7980eb9700 1 -- start start 2024-04-06T12:31:01.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.346+0000 7f7980eb9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f797c15d040 0x7f797c15dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:01.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.346+0000 7f7980eb9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f797c15e2a0 0x7f797c06c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:01.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.346+0000 7f7980eb9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f797c15d8d0 con 0x7f797c0b2cd0 2024-04-06T12:31:01.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.346+0000 7f7980eb9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f797c0cba40 con 0x7f797c15e2a0 2024-04-06T12:31:01.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.346+0000 7f7980eb9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f797c15d580 con 0x7f797c15d040 2024-04-06T12:31:01.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.346+0000 7f7979d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f797c15e2a0 0x7f797c06c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:01.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.346+0000 7f7979d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f797c15e2a0 0x7f797c06c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36650/0 (socket says 172.21.15.189:36650) 2024-04-06T12:31:01.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.346+0000 7f7979d9b700 1 -- 172.21.15.189:0/1993964027 learned_addr learned my addr 172.21.15.189:0/1993964027 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:01.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.346+0000 7f797a59c700 1 --2- 172.21.15.189:0/1993964027 >> v2:172.21.15.192:3300/0 conn(0x7f797c15d040 0x7f797c15dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:01.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.346+0000 7f7979d9b700 1 -- 172.21.15.189:0/1993964027 >> v2:172.21.15.192:3300/0 conn(0x7f797c15d040 msgr2=0x7f797c15dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:01.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.346+0000 7f7979d9b700 1 --2- 172.21.15.189:0/1993964027 >> v2:172.21.15.192:3300/0 conn(0x7f797c15d040 0x7f797c15dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:01.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.346+0000 7f7979d9b700 1 -- 172.21.15.189:0/1993964027 >> v1:172.21.15.192:6789/0 conn(0x7f797c0b2cd0 legacy=0x7f797c0cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:01.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.346+0000 7f7979d9b700 1 -- 172.21.15.189:0/1993964027 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797c0b3240 con 0x7f797c15e2a0 2024-04-06T12:31:01.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.346+0000 7f7979d9b700 1 --2- 172.21.15.189:0/1993964027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797c15e2a0 0x7f797c06c4e0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f797c06ea40 tx=0x7f7970009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8dfae2f06cdc51f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:01.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.347+0000 7f797959a700 1 -- 172.21.15.189:0/1993964027 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7970004230 con 0x7f797c15e2a0 2024-04-06T12:31:01.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.347+0000 7f797959a700 1 -- 172.21.15.189:0/1993964027 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7970004390 con 0x7f797c15e2a0 2024-04-06T12:31:01.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.347+0000 7f797959a700 1 -- 172.21.15.189:0/1993964027 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79700036a0 con 0x7f797c15e2a0 2024-04-06T12:31:01.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.347+0000 7f7980eb9700 1 -- 172.21.15.189:0/1993964027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797c15e2a0 msgr2=0x7f797c06c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:01.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.347+0000 7f7980eb9700 1 --2- 172.21.15.189:0/1993964027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797c15e2a0 0x7f797c06c4e0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f797c06ea40 tx=0x7f7970009bd0 comp rx=0 tx=0).stop 2024-04-06T12:31:01.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.347+0000 7f7980eb9700 1 -- 172.21.15.189:0/1993964027 shutdown_connections 2024-04-06T12:31:01.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.347+0000 7f7980eb9700 1 --2- 172.21.15.189:0/1993964027 >> v2:172.21.15.192:3300/0 conn(0x7f797c15d040 0x7f797c15dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:01.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.347+0000 7f7980eb9700 1 --2- 172.21.15.189:0/1993964027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797c15e2a0 0x7f797c06c4e0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:01.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.347+0000 7f7980eb9700 1 -- 172.21.15.189:0/1993964027 >> 172.21.15.189:0/1993964027 conn(0x7f797c06a730 msgr2=0x7f797c06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:01.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.347+0000 7f7980eb9700 1 -- 172.21.15.189:0/1993964027 shutdown_connections 2024-04-06T12:31:01.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.347+0000 7f7980eb9700 1 -- 172.21.15.189:0/1993964027 wait complete. 2024-04-06T12:31:01.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.349+0000 7f7980eb9700 1 Processor -- start 2024-04-06T12:31:01.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.349+0000 7f7980eb9700 1 -- start start 2024-04-06T12:31:01.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.349+0000 7f7980eb9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f797c0b2cd0 0x7f797c151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:01.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.349+0000 7f7980eb9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f797c15d040 0x7f797c140970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:01.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.349+0000 7f7980eb9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797c0ce130 0x7f797c0affe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:01.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.349+0000 7f7980eb9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797c1523b0 con 0x7f797c0ce130 2024-04-06T12:31:01.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.349+0000 7f7980eb9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797c140eb0 con 0x7f797c15d040 2024-04-06T12:31:01.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.349+0000 7f7980eb9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f797c150dd0 con 0x7f797c0b2cd0 2024-04-06T12:31:01.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.350+0000 7f797ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797c0ce130 0x7f797c0affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:01.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.350+0000 7f797ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797c0ce130 0x7f797c0affe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36658/0 (socket says 172.21.15.189:36658) 2024-04-06T12:31:01.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.350+0000 7f797ad9d700 1 -- 172.21.15.189:0/3195813063 learned_addr learned my addr 172.21.15.189:0/3195813063 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:01.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.350+0000 7f7979d9b700 1 --2- 172.21.15.189:0/3195813063 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f797c15d040 0x7f797c140970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:01.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.350+0000 7f797a59c700 1 --2- 172.21.15.189:0/3195813063 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f797c0b2cd0 0x7f797c151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:01.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.350+0000 7f797ad9d700 1 -- 172.21.15.189:0/3195813063 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f797c0b2cd0 msgr2=0x7f797c151e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:01.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.350+0000 7f797ad9d700 1 --2- 172.21.15.189:0/3195813063 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f797c0b2cd0 0x7f797c151e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:01.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.350+0000 7f797ad9d700 1 -- 172.21.15.189:0/3195813063 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f797c15d040 msgr2=0x7f797c140970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:01.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.350+0000 7f797ad9d700 1 --2- 172.21.15.189:0/3195813063 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f797c15d040 0x7f797c140970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:01.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.350+0000 7f797ad9d700 1 -- 172.21.15.189:0/3195813063 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7970009550 con 0x7f797c0ce130 2024-04-06T12:31:01.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.350+0000 7f797ad9d700 1 --2- 172.21.15.189:0/3195813063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797c0ce130 0x7f797c0affe0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f797400c7a0 tx=0x7f797400cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:01.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.350+0000 7f796affd700 1 -- 172.21.15.189:0/3195813063 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974003e10 con 0x7f797c0ce130 2024-04-06T12:31:01.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.350+0000 7f796affd700 1 -- 172.21.15.189:0/3195813063 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7974005070 con 0x7f797c0ce130 2024-04-06T12:31:01.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.350+0000 7f7980eb9700 1 -- 172.21.15.189:0/3195813063 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f797c1510b0 con 0x7f797c0ce130 2024-04-06T12:31:01.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.350+0000 7f796affd700 1 -- 172.21.15.189:0/3195813063 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974005420 con 0x7f797c0ce130 2024-04-06T12:31:01.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.350+0000 7f7980eb9700 1 -- 172.21.15.189:0/3195813063 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f797c151670 con 0x7f797c0ce130 2024-04-06T12:31:01.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.351+0000 7f7980eb9700 1 -- 172.21.15.189:0/3195813063 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f797c073270 con 0x7f797c0ce130 2024-04-06T12:31:01.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.352+0000 7f796affd700 1 -- 172.21.15.189:0/3195813063 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 69) v1 ==== 89321+0+0 (secure 0 0 0) 0x7f7974005580 con 0x7f797c0ce130 2024-04-06T12:31:01.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.352+0000 7f796affd700 1 --2- 172.21.15.189:0/3195813063 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f795c06ae90 0x7f795c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:01.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.352+0000 7f796affd700 1 -- 172.21.15.189:0/3195813063 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f7974088ef0 con 0x7f797c0ce130 2024-04-06T12:31:01.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.352+0000 7f797a59c700 1 --2- 172.21.15.189:0/3195813063 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f795c06ae90 0x7f795c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:01.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.353+0000 7f797a59c700 1 --2- 172.21.15.189:0/3195813063 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f795c06ae90 0x7f795c06d350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f796c00ab60 tx=0x7f796c008040 comp rx=0 tx=0).ready entity=mgr.8271 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:01.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.357+0000 7f796affd700 1 -- 172.21.15.189:0/3195813063 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7974053bb0 con 0x7f797c0ce130 2024-04-06T12:31:01.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.579+0000 7f7980eb9700 1 -- 172.21.15.189:0/3195813063 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f797c06d2c0 con 0x7f795c06ae90 2024-04-06T12:31:01.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.619+0000 7f796affd700 1 -- 172.21.15.189:0/3195813063 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 70) v1 ==== 89378+0+0 (secure 0 0 0) 0x7f7974053f80 con 0x7f797c0ce130 2024-04-06T12:31:01.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.653+0000 7f796affd700 1 -- 172.21.15.189:0/3195813063 <== mgr.8271 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f797c06d2c0 con 0x7f795c06ae90 2024-04-06T12:31:01.650 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule removed for path snap_test_dir1 2024-04-06T12:31:01.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.655+0000 7f7980eb9700 1 -- 172.21.15.189:0/3195813063 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f795c06ae90 msgr2=0x7f795c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:01.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.655+0000 7f7980eb9700 1 --2- 172.21.15.189:0/3195813063 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f795c06ae90 0x7f795c06d350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f796c00ab60 tx=0x7f796c008040 comp rx=0 tx=0).stop 2024-04-06T12:31:01.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.655+0000 7f7980eb9700 1 -- 172.21.15.189:0/3195813063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797c0ce130 msgr2=0x7f797c0affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:01.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.655+0000 7f7980eb9700 1 --2- 172.21.15.189:0/3195813063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797c0ce130 0x7f797c0affe0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f797400c7a0 tx=0x7f797400cab0 comp rx=0 tx=0).stop 2024-04-06T12:31:01.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.655+0000 7f7980eb9700 1 -- 172.21.15.189:0/3195813063 shutdown_connections 2024-04-06T12:31:01.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.655+0000 7f7980eb9700 1 --2- 172.21.15.189:0/3195813063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f797c0ce130 0x7f797c0affe0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:01.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.655+0000 7f7980eb9700 1 --2- 172.21.15.189:0/3195813063 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f797c0b2cd0 0x7f797c151e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:01.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.655+0000 7f7980eb9700 1 --2- 172.21.15.189:0/3195813063 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f795c06ae90 0x7f795c06d350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:01.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.655+0000 7f7980eb9700 1 --2- 172.21.15.189:0/3195813063 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f797c15d040 0x7f797c140970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:01.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.655+0000 7f7980eb9700 1 -- 172.21.15.189:0/3195813063 >> 172.21.15.189:0/3195813063 conn(0x7f797c06a730 msgr2=0x7f797c0cefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:01.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.655+0000 7f7980eb9700 1 -- 172.21.15.189:0/3195813063 shutdown_connections 2024-04-06T12:31:01.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.655+0000 7f7980eb9700 1 -- 172.21.15.189:0/3195813063 wait complete. 2024-04-06T12:31:01.664 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-04-06T12:31:01.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:01.665 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T12:31:01.682 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_31_00_UTC 2024-04-06T12:31:01.682 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_31_00_UTC 2024-04-06T12:31:01.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:01.683 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_31_00_UTC) 2024-04-06T12:31:01.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:01.744 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-06T12:31:01.803 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-06T12:31:01.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.969+0000 7fa4a96b1700 1 Processor -- start 2024-04-06T12:31:01.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.969+0000 7fa4a96b1700 1 -- start start 2024-04-06T12:31:01.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.970+0000 7fa4a96b1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa4a415d6f0 0x7fa4a415dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:01.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.970+0000 7fa4a96b1700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa4a40ce730 0x7fa4a40d1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:01.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.970+0000 7fa4a96b1700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a415eac0 con 0x7fa4a40b8d70 2024-04-06T12:31:01.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.970+0000 7fa4a96b1700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a415e010 con 0x7fa4a415d6f0 2024-04-06T12:31:01.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.970+0000 7fa4a96b1700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a415e220 con 0x7fa4a40ce730 2024-04-06T12:31:01.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.970+0000 7fa4a2ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa4a415d6f0 0x7fa4a415dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:01.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.970+0000 7fa4a2ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa4a415d6f0 0x7fa4a415dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36662/0 (socket says 172.21.15.189:36662) 2024-04-06T12:31:01.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.970+0000 7fa4a2ffd700 1 -- 172.21.15.189:0/1121202985 learned_addr learned my addr 172.21.15.189:0/1121202985 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:01.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.970+0000 7fa4a37fe700 1 --2- 172.21.15.189:0/1121202985 >> v2:172.21.15.192:3300/0 conn(0x7fa4a40ce730 0x7fa4a40d1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:01.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.970+0000 7fa4a2ffd700 1 -- 172.21.15.189:0/1121202985 >> v2:172.21.15.192:3300/0 conn(0x7fa4a40ce730 msgr2=0x7fa4a40d1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:01.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.970+0000 7fa4a2ffd700 1 --2- 172.21.15.189:0/1121202985 >> v2:172.21.15.192:3300/0 conn(0x7fa4a40ce730 0x7fa4a40d1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:01.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.970+0000 7fa4a2ffd700 1 -- 172.21.15.189:0/1121202985 >> v1:172.21.15.192:6789/0 conn(0x7fa4a40b8d70 legacy=0x7fa4a40ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:01.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.970+0000 7fa4a2ffd700 1 -- 172.21.15.189:0/1121202985 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a415e500 con 0x7fa4a415d6f0 2024-04-06T12:31:01.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.970+0000 7fa4a37fe700 1 --2- 172.21.15.189:0/1121202985 >> v2:172.21.15.192:3300/0 conn(0x7fa4a40ce730 0x7fa4a40d1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:01.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.970+0000 7fa4a2ffd700 1 --2- 172.21.15.189:0/1121202985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a415d6f0 0x7fa4a415dad0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fa4a40b6350 tx=0x7fa498009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbda568dc76b210d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:01.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.971+0000 7fa4a1ffb700 1 -- 172.21.15.189:0/1121202985 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa498004230 con 0x7fa4a415d6f0 2024-04-06T12:31:01.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.971+0000 7fa4a1ffb700 1 -- 172.21.15.189:0/1121202985 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa498004390 con 0x7fa4a415d6f0 2024-04-06T12:31:01.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.971+0000 7fa4a1ffb700 1 -- 172.21.15.189:0/1121202985 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4980036a0 con 0x7fa4a415d6f0 2024-04-06T12:31:01.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.971+0000 7fa4a96b1700 1 -- 172.21.15.189:0/1121202985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a415d6f0 msgr2=0x7fa4a415dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:01.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.971+0000 7fa4a96b1700 1 --2- 172.21.15.189:0/1121202985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a415d6f0 0x7fa4a415dad0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fa4a40b6350 tx=0x7fa498009bd0 comp rx=0 tx=0).stop 2024-04-06T12:31:01.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.971+0000 7fa4a96b1700 1 -- 172.21.15.189:0/1121202985 shutdown_connections 2024-04-06T12:31:01.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.971+0000 7fa4a96b1700 1 --2- 172.21.15.189:0/1121202985 >> v2:172.21.15.192:3300/0 conn(0x7fa4a40ce730 0x7fa4a40d1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:01.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.971+0000 7fa4a96b1700 1 --2- 172.21.15.189:0/1121202985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a415d6f0 0x7fa4a415dad0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:01.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.971+0000 7fa4a96b1700 1 -- 172.21.15.189:0/1121202985 >> 172.21.15.189:0/1121202985 conn(0x7fa4a4072490 msgr2=0x7fa4a406b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:01.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.971+0000 7fa4a96b1700 1 -- 172.21.15.189:0/1121202985 shutdown_connections 2024-04-06T12:31:01.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.971+0000 7fa4a96b1700 1 -- 172.21.15.189:0/1121202985 wait complete. 2024-04-06T12:31:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.972+0000 7fa4a96b1700 1 Processor -- start 2024-04-06T12:31:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.973+0000 7fa4a96b1700 1 -- start start 2024-04-06T12:31:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.973+0000 7fa4a96b1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4a40b8d70 0x7fa4a41576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.973+0000 7fa4a96b1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4a40ce730 0x7fa4a4156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.973+0000 7fa4a96b1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a415d6f0 0x7fa4a41541c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.973+0000 7fa4a96b1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a40b5c90 con 0x7fa4a415d6f0 2024-04-06T12:31:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.973+0000 7fa4a96b1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a4154700 con 0x7fa4a40ce730 2024-04-06T12:31:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.973+0000 7fa4a96b1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a4153770 con 0x7fa4a40b8d70 2024-04-06T12:31:01.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.973+0000 7fa4a37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a415d6f0 0x7fa4a41541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:01.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.973+0000 7fa4a37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a415d6f0 0x7fa4a41541c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36664/0 (socket says 172.21.15.189:36664) 2024-04-06T12:31:01.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.973+0000 7fa4a37fe700 1 -- 172.21.15.189:0/367015235 learned_addr learned my addr 172.21.15.189:0/367015235 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:01.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.974+0000 7fa4a27fc700 1 --2- 172.21.15.189:0/367015235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4a40ce730 0x7fa4a4156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:01.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.974+0000 7fa4a2ffd700 1 --2- 172.21.15.189:0/367015235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4a40b8d70 0x7fa4a41576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:01.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.974+0000 7fa4a37fe700 1 -- 172.21.15.189:0/367015235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4a40b8d70 msgr2=0x7fa4a41576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:01.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.974+0000 7fa4a37fe700 1 --2- 172.21.15.189:0/367015235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4a40b8d70 0x7fa4a41576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:01.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.974+0000 7fa4a37fe700 1 -- 172.21.15.189:0/367015235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4a40ce730 msgr2=0x7fa4a4156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:01.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.974+0000 7fa4a37fe700 1 --2- 172.21.15.189:0/367015235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4a40ce730 0x7fa4a4156cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:01.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.974+0000 7fa4a37fe700 1 -- 172.21.15.189:0/367015235 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa498009550 con 0x7fa4a415d6f0 2024-04-06T12:31:01.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.974+0000 7fa4a27fc700 1 --2- 172.21.15.189:0/367015235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4a40ce730 0x7fa4a4156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:01.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.974+0000 7fa4a37fe700 1 --2- 172.21.15.189:0/367015235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a415d6f0 0x7fa4a41541c0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00cf20 tx=0x7fa49c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:01.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.974+0000 7fa48b7fe700 1 -- 172.21.15.189:0/367015235 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c00ae80 con 0x7fa4a415d6f0 2024-04-06T12:31:01.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.974+0000 7fa48b7fe700 1 -- 172.21.15.189:0/367015235 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49c015440 con 0x7fa4a415d6f0 2024-04-06T12:31:01.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.974+0000 7fa4a96b1700 1 -- 172.21.15.189:0/367015235 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a4157c90 con 0x7fa4a415d6f0 2024-04-06T12:31:01.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.974+0000 7fa48b7fe700 1 -- 172.21.15.189:0/367015235 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c0157d0 con 0x7fa4a415d6f0 2024-04-06T12:31:01.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.974+0000 7fa4a96b1700 1 -- 172.21.15.189:0/367015235 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a41539c0 con 0x7fa4a415d6f0 2024-04-06T12:31:01.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.975+0000 7fa4a96b1700 1 -- 172.21.15.189:0/367015235 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a4074990 con 0x7fa4a415d6f0 2024-04-06T12:31:01.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.975+0000 7fa48b7fe700 1 -- 172.21.15.189:0/367015235 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 70) v1 ==== 89378+0+0 (secure 0 0 0) 0x7fa49c00d430 con 0x7fa4a415d6f0 2024-04-06T12:31:01.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.976+0000 7fa48b7fe700 1 --2- 172.21.15.189:0/367015235 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa49006af20 0x7fa49006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:01.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.976+0000 7fa48b7fe700 1 -- 172.21.15.189:0/367015235 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fa49c08a0b0 con 0x7fa4a415d6f0 2024-04-06T12:31:01.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.976+0000 7fa4a2ffd700 1 --2- 172.21.15.189:0/367015235 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa49006af20 0x7fa49006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:01.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.976+0000 7fa4a2ffd700 1 --2- 172.21.15.189:0/367015235 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa49006af20 0x7fa49006d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa498009130 tx=0x7fa498008030 comp rx=0 tx=0).ready entity=mgr.8271 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:01.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:01.980+0000 7fa48b7fe700 1 -- 172.21.15.189:0/367015235 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa49c054c90 con 0x7fa4a415d6f0 2024-04-06T12:31:02.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.231+0000 7fa4a96b1700 1 -- 172.21.15.189:0/367015235 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fa4a40b6670 con 0x7fa4a415d6f0 2024-04-06T12:31:02.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.625+0000 7fa48b7fe700 1 -- 172.21.15.189:0/367015235 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 71) v1 ==== 89361+0+0 (secure 0 0 0) 0x7fa49c055060 con 0x7fa4a415d6f0 2024-04-06T12:31:02.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.626+0000 7fa48b7fe700 1 -- 172.21.15.189:0/367015235 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v71) v1 ==== 93+0+0 (secure 0 0 0) 0x7fa49c0716b0 con 0x7fa4a415d6f0 2024-04-06T12:31:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.628+0000 7fa4a96b1700 1 -- 172.21.15.189:0/367015235 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa49006af20 msgr2=0x7fa49006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.628+0000 7fa4a96b1700 1 --2- 172.21.15.189:0/367015235 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa49006af20 0x7fa49006d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa498009130 tx=0x7fa498008030 comp rx=0 tx=0).stop 2024-04-06T12:31:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.628+0000 7fa4a96b1700 1 -- 172.21.15.189:0/367015235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a415d6f0 msgr2=0x7fa4a41541c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.628+0000 7fa4a96b1700 1 --2- 172.21.15.189:0/367015235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a415d6f0 0x7fa4a41541c0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00cf20 tx=0x7fa49c00a340 comp rx=0 tx=0).stop 2024-04-06T12:31:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.628+0000 7fa4a96b1700 1 -- 172.21.15.189:0/367015235 shutdown_connections 2024-04-06T12:31:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.628+0000 7fa4a96b1700 1 --2- 172.21.15.189:0/367015235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a415d6f0 0x7fa4a41541c0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.628+0000 7fa4a96b1700 1 --2- 172.21.15.189:0/367015235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4a40b8d70 0x7fa4a41576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:02.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.628+0000 7fa4a96b1700 1 --2- 172.21.15.189:0/367015235 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa49006af20 0x7fa49006d3e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:02.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.628+0000 7fa4a96b1700 1 --2- 172.21.15.189:0/367015235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4a40ce730 0x7fa4a4156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:02.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.628+0000 7fa4a96b1700 1 -- 172.21.15.189:0/367015235 >> 172.21.15.189:0/367015235 conn(0x7fa4a4072490 msgr2=0x7fa4a40cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:02.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.629+0000 7fa4a96b1700 1 -- 172.21.15.189:0/367015235 shutdown_connections 2024-04-06T12:31:02.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.629+0000 7fa4a96b1700 1 -- 172.21.15.189:0/367015235 wait complete. 2024-04-06T12:31:02.640 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T12:31:02.640 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:31:02.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:31:02.725 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T12:31:02.726 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:31:02.726 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:31:02.744 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:31:02.745 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:31:02.804 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:31:02.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.974+0000 7f982574a700 1 Processor -- start 2024-04-06T12:31:02.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.974+0000 7f982574a700 1 -- start start 2024-04-06T12:31:02.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.975+0000 7f982574a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98200d11b0 0x7f98200d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:02.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.975+0000 7f982574a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f98200cb870 0x7f982015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:02.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.975+0000 7f982574a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98200d1ad0 con 0x7f98200cfa00 2024-04-06T12:31:02.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.975+0000 7f982574a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98200ccdc0 con 0x7f98200cb870 2024-04-06T12:31:02.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.975+0000 7f982574a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98200d0b50 con 0x7f98200d11b0 2024-04-06T12:31:02.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.975+0000 7f981effd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98200d11b0 0x7f98200d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:02.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.975+0000 7f981e7fc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f98200cb870 0x7f982015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:02.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.975+0000 7f981f7fe700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f98200cfa00 0x7f98200cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:46932/0 (socket says 172.21.15.189:46932) 2024-04-06T12:31:02.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.975+0000 7f981f7fe700 1 -- 172.21.15.189:0/1202296500 learned_addr learned my addr 172.21.15.189:0/1202296500 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:02.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.975+0000 7f981dffb700 1 -- 172.21.15.189:0/1202296500 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656467576 0 0) 0x7f98200d1ad0 con 0x7f98200cfa00 2024-04-06T12:31:02.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.975+0000 7f981effd700 1 -- 172.21.15.189:0/1202296500 >> v2:172.21.15.192:3300/0 conn(0x7f98200cb870 msgr2=0x7f982015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:02.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.975+0000 7f981effd700 1 --2- 172.21.15.189:0/1202296500 >> v2:172.21.15.192:3300/0 conn(0x7f98200cb870 0x7f982015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:02.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.975+0000 7f981effd700 1 -- 172.21.15.189:0/1202296500 >> v1:172.21.15.192:6789/0 conn(0x7f98200cfa00 legacy=0x7f98200cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:02.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.975+0000 7f981effd700 1 -- 172.21.15.189:0/1202296500 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98200cbd80 con 0x7f98200d11b0 2024-04-06T12:31:02.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.976+0000 7f981effd700 1 --2- 172.21.15.189:0/1202296500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98200d11b0 0x7f98200d1590 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9810009840 tx=0x7f9810009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d27ecfa3bd0ca99 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:02.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.976+0000 7f981dffb700 1 -- 172.21.15.189:0/1202296500 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810011830 con 0x7f98200d11b0 2024-04-06T12:31:02.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.976+0000 7f981dffb700 1 -- 172.21.15.189:0/1202296500 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9810011990 con 0x7f98200d11b0 2024-04-06T12:31:02.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.976+0000 7f981dffb700 1 -- 172.21.15.189:0/1202296500 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810011d20 con 0x7f98200d11b0 2024-04-06T12:31:02.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.976+0000 7f982574a700 1 -- 172.21.15.189:0/1202296500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98200d11b0 msgr2=0x7f98200d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:02.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.976+0000 7f982574a700 1 --2- 172.21.15.189:0/1202296500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98200d11b0 0x7f98200d1590 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9810009840 tx=0x7f9810009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:02.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.977+0000 7f982574a700 1 -- 172.21.15.189:0/1202296500 shutdown_connections 2024-04-06T12:31:02.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.977+0000 7f982574a700 1 --2- 172.21.15.189:0/1202296500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98200d11b0 0x7f98200d1590 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:02.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.977+0000 7f982574a700 1 --2- 172.21.15.189:0/1202296500 >> v2:172.21.15.192:3300/0 conn(0x7f98200cb870 0x7f982015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:02.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.977+0000 7f982574a700 1 -- 172.21.15.189:0/1202296500 >> 172.21.15.189:0/1202296500 conn(0x7f98200b2db0 msgr2=0x7f98200b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:02.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.977+0000 7f982574a700 1 -- 172.21.15.189:0/1202296500 shutdown_connections 2024-04-06T12:31:02.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.977+0000 7f982574a700 1 -- 172.21.15.189:0/1202296500 wait complete. 2024-04-06T12:31:02.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.978+0000 7f982574a700 1 Processor -- start 2024-04-06T12:31:02.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.978+0000 7f982574a700 1 -- start start 2024-04-06T12:31:02.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.979+0000 7f982574a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200cb870 0x7f9820152050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:02.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.979+0000 7f982574a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200cfa00 0x7f982015c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:02.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.979+0000 7f982574a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98200d11b0 0x7f982015c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:02.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.979+0000 7f982574a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98201586e0 con 0x7f98200cb870 2024-04-06T12:31:02.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.979+0000 7f982574a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982015d200 con 0x7f98200cfa00 2024-04-06T12:31:02.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.979+0000 7f982574a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9820157c00 con 0x7f98200d11b0 2024-04-06T12:31:02.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.979+0000 7f981effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200cb870 0x7f9820152050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:02.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.979+0000 7f981effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200cb870 0x7f9820152050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36692/0 (socket says 172.21.15.189:36692) 2024-04-06T12:31:02.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.979+0000 7f981e7fc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200cfa00 0x7f982015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:02.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.979+0000 7f981effd700 1 -- 172.21.15.189:0/931130826 learned_addr learned my addr 172.21.15.189:0/931130826 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:02.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.979+0000 7f981f7fe700 1 --2- 172.21.15.189:0/931130826 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98200d11b0 0x7f982015c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:02.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.979+0000 7f981effd700 1 -- 172.21.15.189:0/931130826 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98200d11b0 msgr2=0x7f982015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:02.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.979+0000 7f981effd700 1 --2- 172.21.15.189:0/931130826 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98200d11b0 0x7f982015c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:02.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.979+0000 7f981effd700 1 -- 172.21.15.189:0/931130826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200cfa00 msgr2=0x7f982015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:02.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.979+0000 7f981effd700 1 --2- 172.21.15.189:0/931130826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200cfa00 0x7f982015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:02.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.979+0000 7f981effd700 1 -- 172.21.15.189:0/931130826 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98100094a0 con 0x7f98200cb870 2024-04-06T12:31:02.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.979+0000 7f981f7fe700 1 --2- 172.21.15.189:0/931130826 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98200d11b0 0x7f982015c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:02.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.979+0000 7f981effd700 1 --2- 172.21.15.189:0/931130826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200cb870 0x7f9820152050 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f9810009fd0 tx=0x7f9810004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:02.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.979+0000 7f98077fe700 1 -- 172.21.15.189:0/931130826 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98100036a0 con 0x7f98200cb870 2024-04-06T12:31:02.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.980+0000 7f98077fe700 1 -- 172.21.15.189:0/931130826 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981001f070 con 0x7f98200cb870 2024-04-06T12:31:02.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.980+0000 7f982574a700 1 -- 172.21.15.189:0/931130826 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9820157ee0 con 0x7f98200cb870 2024-04-06T12:31:02.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.980+0000 7f98077fe700 1 -- 172.21.15.189:0/931130826 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810003a60 con 0x7f98200cb870 2024-04-06T12:31:02.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.980+0000 7f982574a700 1 -- 172.21.15.189:0/931130826 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982015d4e0 con 0x7f98200cb870 2024-04-06T12:31:02.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.981+0000 7f982574a700 1 -- 172.21.15.189:0/931130826 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982015b060 con 0x7f98200cb870 2024-04-06T12:31:02.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.981+0000 7f98077fe700 1 -- 172.21.15.189:0/931130826 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 71) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f9810003bc0 con 0x7f98200cb870 2024-04-06T12:31:02.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.981+0000 7f98077fe700 1 --2- 172.21.15.189:0/931130826 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f980c06aed0 0x7f980c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:02.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.981+0000 7f981e7fc700 1 -- 172.21.15.189:0/931130826 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f980c06aed0 msgr2=0x7f980c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:02.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.981+0000 7f98077fe700 1 -- 172.21.15.189:0/931130826 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f9810021070 con 0x7f98200cb870 2024-04-06T12:31:02.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.981+0000 7f981e7fc700 1 --2- 172.21.15.189:0/931130826 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f980c06aed0 0x7f980c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:02.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:02.986+0000 7f98077fe700 1 -- 172.21.15.189:0/931130826 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f98100be650 con 0x7f98200cb870 2024-04-06T12:31:03.010 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:03.013+0000 7f3500729200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:31:03.030 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:03.033+0000 7f6920c15200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:31:03.088 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:03.091+0000 7f3500729200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:31:03.111 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:03.114+0000 7f6920c15200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:31:03.161 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:03.163+0000 7f3500729200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:31:03.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.182+0000 7f981e7fc700 1 -- 172.21.15.189:0/931130826 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f980c06aed0 msgr2=0x7f980c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:03.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.182+0000 7f981e7fc700 1 --2- 172.21.15.189:0/931130826 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f980c06aed0 0x7f980c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:03.186 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:03.189+0000 7f6920c15200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:31:03.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.190+0000 7f982574a700 1 -- 172.21.15.189:0/931130826 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98200d0ff0 con 0x7f98200cb870 2024-04-06T12:31:03.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.191+0000 7f98077fe700 1 -- 172.21.15.189:0/931130826 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16792 (secure 0 0 0) 0x7f9810015090 con 0x7f98200cb870 2024-04-06T12:31:03.188 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:03.188 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":129,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:30:31.084655+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:30:14.958335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-06T12:30:16.104585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/4170508796":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/169651662":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/29183104":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/3994101871":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/1300227922":"2024-04-07T12:30:31.084606+0000","172.21.15.189:6833/47128":"2024-04-07T12:30:31.084606+0000","172.21.15.189:6832/47128":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/1086259216":"2024-04-07T12:30:31.084606+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:31:03.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.193+0000 7f982574a700 1 -- 172.21.15.189:0/931130826 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f980c06aed0 msgr2=0x7f980c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:03.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.193+0000 7f982574a700 1 --2- 172.21.15.189:0/931130826 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f980c06aed0 0x7f980c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.193+0000 7f982574a700 1 -- 172.21.15.189:0/931130826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200cb870 msgr2=0x7f9820152050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:03.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.193+0000 7f982574a700 1 --2- 172.21.15.189:0/931130826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200cb870 0x7f9820152050 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f9810009fd0 tx=0x7f9810004750 comp rx=0 tx=0).stop 2024-04-06T12:31:03.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.193+0000 7f982574a700 1 -- 172.21.15.189:0/931130826 shutdown_connections 2024-04-06T12:31:03.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.193+0000 7f982574a700 1 --2- 172.21.15.189:0/931130826 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98200d11b0 0x7f982015c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.193+0000 7f982574a700 1 --2- 172.21.15.189:0/931130826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200cb870 0x7f9820152050 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.193+0000 7f982574a700 1 --2- 172.21.15.189:0/931130826 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f980c06aed0 0x7f980c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.193+0000 7f982574a700 1 --2- 172.21.15.189:0/931130826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200cfa00 0x7f982015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.193+0000 7f982574a700 1 -- 172.21.15.189:0/931130826 >> 172.21.15.189:0/931130826 conn(0x7f98200b2db0 msgr2=0x7f98200b2380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:03.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.193+0000 7f982574a700 1 -- 172.21.15.189:0/931130826 shutdown_connections 2024-04-06T12:31:03.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.193+0000 7f982574a700 1 -- 172.21.15.189:0/931130826 wait complete. 2024-04-06T12:31:03.203 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:30:14.958335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:30:16.104585+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:31:03.204 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:31:03.228 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:03.231+0000 7f3500729200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:31:03.263 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:03.265+0000 7f6920c15200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:31:03.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.336+0000 7f8cba613700 1 Processor -- start 2024-04-06T12:31:03.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.336+0000 7f8cba613700 1 -- start start 2024-04-06T12:31:03.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.337+0000 7f8cba613700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8cb40d0a60 0x7f8cb40d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:03.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.337+0000 7f8cba613700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8cb40d1af0 0x7f8cb40b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:03.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.337+0000 7f8cba613700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb40cdf80 con 0x7f8cb415d7e0 2024-04-06T12:31:03.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.337+0000 7f8cba613700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb40d1f80 con 0x7f8cb40d0a60 2024-04-06T12:31:03.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.337+0000 7f8cba613700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb40d3230 con 0x7f8cb40d1af0 2024-04-06T12:31:03.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.337+0000 7f8cb3fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8cb415d7e0 0x7f8cb40d0350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45310/0 (socket says 172.21.15.189:45310) 2024-04-06T12:31:03.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.337+0000 7f8cb3fff700 1 -- 172.21.15.189:0/2420749332 learned_addr learned my addr 172.21.15.189:0/2420749332 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:03.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.337+0000 7f8cb37fe700 1 --2- 172.21.15.189:0/2420749332 >> v2:172.21.15.192:3300/0 conn(0x7f8cb40d0a60 0x7f8cb40d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:03.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.337+0000 7f8cb8bb0700 1 --2- 172.21.15.189:0/2420749332 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8cb40d1af0 0x7f8cb40b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:03.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.337+0000 7f8cb2ffd700 1 -- 172.21.15.189:0/2420749332 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568372083 0 0) 0x7f8cb40cdf80 con 0x7f8cb415d7e0 2024-04-06T12:31:03.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.337+0000 7f8cb2ffd700 1 -- 172.21.15.189:0/2420749332 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c9c0035b0 con 0x7f8cb415d7e0 2024-04-06T12:31:03.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.338+0000 7f8cb8bb0700 1 -- 172.21.15.189:0/2420749332 >> v2:172.21.15.192:3300/0 conn(0x7f8cb40d0a60 msgr2=0x7f8cb40d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:03.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.338+0000 7f8cb8bb0700 1 --2- 172.21.15.189:0/2420749332 >> v2:172.21.15.192:3300/0 conn(0x7f8cb40d0a60 0x7f8cb40d15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.338+0000 7f8cb8bb0700 1 -- 172.21.15.189:0/2420749332 >> v1:172.21.15.189:6789/0 conn(0x7f8cb415d7e0 legacy=0x7f8cb40d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:03.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.338+0000 7f8cb8bb0700 1 -- 172.21.15.189:0/2420749332 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb40d2da0 con 0x7f8cb40d1af0 2024-04-06T12:31:03.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.338+0000 7f8cb37fe700 1 --2- 172.21.15.189:0/2420749332 >> v2:172.21.15.192:3300/0 conn(0x7f8cb40d0a60 0x7f8cb40d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:03.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.338+0000 7f8cb8bb0700 1 --2- 172.21.15.189:0/2420749332 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8cb40d1af0 0x7f8cb40b66b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8cac009d90 tx=0x7f8cac00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=292574760f6cb99d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:03.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.338+0000 7f8cb2ffd700 1 -- 172.21.15.189:0/2420749332 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac00e070 con 0x7f8cb40d1af0 2024-04-06T12:31:03.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.338+0000 7f8cb2ffd700 1 -- 172.21.15.189:0/2420749332 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cac00bb20 con 0x7f8cb40d1af0 2024-04-06T12:31:03.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.338+0000 7f8cb2ffd700 1 -- 172.21.15.189:0/2420749332 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac012430 con 0x7f8cb40d1af0 2024-04-06T12:31:03.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.338+0000 7f8cba613700 1 -- 172.21.15.189:0/2420749332 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8cb40d1af0 msgr2=0x7f8cb40b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:03.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.338+0000 7f8cba613700 1 --2- 172.21.15.189:0/2420749332 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8cb40d1af0 0x7f8cb40b66b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8cac009d90 tx=0x7f8cac00b330 comp rx=0 tx=0).stop 2024-04-06T12:31:03.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.339+0000 7f8cba613700 1 -- 172.21.15.189:0/2420749332 shutdown_connections 2024-04-06T12:31:03.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.339+0000 7f8cba613700 1 --2- 172.21.15.189:0/2420749332 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8cb40d1af0 0x7f8cb40b66b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.339+0000 7f8cba613700 1 --2- 172.21.15.189:0/2420749332 >> v2:172.21.15.192:3300/0 conn(0x7f8cb40d0a60 0x7f8cb40d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.339+0000 7f8cba613700 1 -- 172.21.15.189:0/2420749332 >> 172.21.15.189:0/2420749332 conn(0x7f8cb4072250 msgr2=0x7f8cb4072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:03.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.339+0000 7f8cba613700 1 -- 172.21.15.189:0/2420749332 shutdown_connections 2024-04-06T12:31:03.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.339+0000 7f8cba613700 1 -- 172.21.15.189:0/2420749332 wait complete. 2024-04-06T12:31:03.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.340+0000 7f8cba613700 1 Processor -- start 2024-04-06T12:31:03.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.340+0000 7f8cba613700 1 -- start start 2024-04-06T12:31:03.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.341+0000 7f8cba613700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cb40d0a60 0x7f8cb40c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:03.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.341+0000 7f8cba613700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8cb40d1af0 0x7f8cb40c84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:03.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.341+0000 7f8cba613700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cb415d7e0 0x7f8cb414d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:03.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.341+0000 7f8cba613700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb40c94b0 con 0x7f8cb40d0a60 2024-04-06T12:31:03.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.341+0000 7f8cba613700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb40c8a00 con 0x7f8cb415d7e0 2024-04-06T12:31:03.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.341+0000 7f8cba613700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb40c8c10 con 0x7f8cb40d1af0 2024-04-06T12:31:03.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.341+0000 7f8cb37fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8cb40d1af0 0x7f8cb40c84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:03.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.341+0000 7f8cb3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cb40d0a60 0x7f8cb40c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:03.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.341+0000 7f8cb37fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8cb40d1af0 0x7f8cb40c84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:43210/0 (socket says 172.21.15.189:43210) 2024-04-06T12:31:03.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.341+0000 7f8cb37fe700 1 -- 172.21.15.189:0/3634520508 learned_addr learned my addr 172.21.15.189:0/3634520508 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:03.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.341+0000 7f8cb8bb0700 1 --2- 172.21.15.189:0/3634520508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cb415d7e0 0x7f8cb414d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:03.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.341+0000 7f8cb37fe700 1 -- 172.21.15.189:0/3634520508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cb415d7e0 msgr2=0x7f8cb414d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:03.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.341+0000 7f8cb37fe700 1 --2- 172.21.15.189:0/3634520508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cb415d7e0 0x7f8cb414d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.341+0000 7f8cb37fe700 1 -- 172.21.15.189:0/3634520508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cb40d0a60 msgr2=0x7f8cb40c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:03.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.341+0000 7f8cb37fe700 1 --2- 172.21.15.189:0/3634520508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cb40d0a60 0x7f8cb40c98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.341+0000 7f8cb37fe700 1 -- 172.21.15.189:0/3634520508 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca80094a0 con 0x7f8cb40d1af0 2024-04-06T12:31:03.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.341+0000 7f8cb37fe700 1 --2- 172.21.15.189:0/3634520508 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8cb40d1af0 0x7f8cb40c84c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800ebe0 tx=0x7f8ca800eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:03.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.341+0000 7f8cb0ff9700 1 -- 172.21.15.189:0/3634520508 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca8009760 con 0x7f8cb40d1af0 2024-04-06T12:31:03.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.342+0000 7f8cb0ff9700 1 -- 172.21.15.189:0/3634520508 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca800c850 con 0x7f8cb40d1af0 2024-04-06T12:31:03.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.342+0000 7f8cb0ff9700 1 -- 172.21.15.189:0/3634520508 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca800cbe0 con 0x7f8cb40d1af0 2024-04-06T12:31:03.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.342+0000 7f8cba613700 1 -- 172.21.15.189:0/3634520508 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cac009a70 con 0x7f8cb40d1af0 2024-04-06T12:31:03.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.342+0000 7f8cba613700 1 -- 172.21.15.189:0/3634520508 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb40c91d0 con 0x7f8cb40d1af0 2024-04-06T12:31:03.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.343+0000 7f8cba613700 1 -- 172.21.15.189:0/3634520508 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb4074990 con 0x7f8cb40d1af0 2024-04-06T12:31:03.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.343+0000 7f8cb0ff9700 1 -- 172.21.15.189:0/3634520508 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 71) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f8ca800cd40 con 0x7f8cb40d1af0 2024-04-06T12:31:03.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.343+0000 7f8cb0ff9700 1 --2- 172.21.15.189:0/3634520508 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8ca006aed0 0x7f8ca006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:03.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.344+0000 7f8cb3fff700 1 -- 172.21.15.189:0/3634520508 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8ca006aed0 msgr2=0x7f8ca006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:03.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.344+0000 7f8cb0ff9700 1 -- 172.21.15.189:0/3634520508 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f8ca8089aa0 con 0x7f8cb40d1af0 2024-04-06T12:31:03.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.344+0000 7f8cb3fff700 1 --2- 172.21.15.189:0/3634520508 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8ca006aed0 0x7f8ca006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:03.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.348+0000 7f8cb0ff9700 1 -- 172.21.15.189:0/3634520508 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8ca80545e0 con 0x7f8cb40d1af0 2024-04-06T12:31:03.403 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:03.406+0000 7f3500729200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:31:03.447 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:03.450+0000 7f6920c15200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:31:03.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.545+0000 7f8cb3fff700 1 -- 172.21.15.189:0/3634520508 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8ca006aed0 msgr2=0x7f8ca006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:03.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.545+0000 7f8cb3fff700 1 --2- 172.21.15.189:0/3634520508 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8ca006aed0 0x7f8ca006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:03.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.549+0000 7f8cba613700 1 -- 172.21.15.189:0/3634520508 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8cb40d2a20 con 0x7f8cb40d1af0 2024-04-06T12:31:03.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.550+0000 7f8cb0ff9700 1 -- 172.21.15.189:0/3634520508 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8ca80170a0 con 0x7f8cb40d1af0 2024-04-06T12:31:03.547 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:31:03.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.552+0000 7f8cba613700 1 -- 172.21.15.189:0/3634520508 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8ca006aed0 msgr2=0x7f8ca006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:03.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.552+0000 7f8cba613700 1 --2- 172.21.15.189:0/3634520508 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8ca006aed0 0x7f8ca006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.552+0000 7f8cba613700 1 -- 172.21.15.189:0/3634520508 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8cb40d1af0 msgr2=0x7f8cb40c84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:03.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.552+0000 7f8cba613700 1 --2- 172.21.15.189:0/3634520508 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8cb40d1af0 0x7f8cb40c84c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800ebe0 tx=0x7f8ca800eef0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.552+0000 7f8cba613700 1 -- 172.21.15.189:0/3634520508 shutdown_connections 2024-04-06T12:31:03.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.552+0000 7f8cba613700 1 --2- 172.21.15.189:0/3634520508 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8ca006aed0 0x7f8ca006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.552+0000 7f8cba613700 1 --2- 172.21.15.189:0/3634520508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cb415d7e0 0x7f8cb414d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.552+0000 7f8cba613700 1 --2- 172.21.15.189:0/3634520508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cb40d0a60 0x7f8cb40c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.552+0000 7f8cba613700 1 --2- 172.21.15.189:0/3634520508 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8cb40d1af0 0x7f8cb40c84c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.552+0000 7f8cba613700 1 -- 172.21.15.189:0/3634520508 >> 172.21.15.189:0/3634520508 conn(0x7f8cb4072250 msgr2=0x7f8cb40b6670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:03.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.552+0000 7f8cba613700 1 -- 172.21.15.189:0/3634520508 shutdown_connections 2024-04-06T12:31:03.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.552+0000 7f8cba613700 1 -- 172.21.15.189:0/3634520508 wait complete. 2024-04-06T12:31:03.559 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:03.562+0000 7f3500729200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:31:03.562 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:31:03.611 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:03.614+0000 7f6920c15200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:31:03.668 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:03.671+0000 7f3500729200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:31:03.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.695+0000 7f3930e7b700 1 Processor -- start 2024-04-06T12:31:03.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.695+0000 7f3930e7b700 1 -- start start 2024-04-06T12:31:03.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.696+0000 7f3930e7b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f392c15d7b0 0x7f392c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:03.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.696+0000 7f3930e7b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f392c15e500 0x7f392c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:03.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.696+0000 7f3930e7b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f392c0b3990 con 0x7f392c06a730 2024-04-06T12:31:03.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.696+0000 7f3930e7b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f392c06b0f0 con 0x7f392c15d7b0 2024-04-06T12:31:03.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.696+0000 7f3930e7b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f392c0b3060 con 0x7f392c15e500 2024-04-06T12:31:03.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.696+0000 7f392a59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f392c15d7b0 0x7f392c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:03.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.696+0000 7f392a59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f392c15d7b0 0x7f392c0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36702/0 (socket says 172.21.15.189:36702) 2024-04-06T12:31:03.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.696+0000 7f392a59c700 1 -- 172.21.15.189:0/3827547418 learned_addr learned my addr 172.21.15.189:0/3827547418 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:03.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.696+0000 7f392ad9d700 1 --2- 172.21.15.189:0/3827547418 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f392c15e500 0x7f392c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:03.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.696+0000 7f392a59c700 1 -- 172.21.15.189:0/3827547418 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f392c15e500 msgr2=0x7f392c06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:03.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.696+0000 7f392a59c700 1 --2- 172.21.15.189:0/3827547418 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f392c15e500 0x7f392c06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.696+0000 7f392a59c700 1 -- 172.21.15.189:0/3827547418 >> v1:172.21.15.192:6789/0 conn(0x7f392c06a730 legacy=0x7f392c15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:03.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.696+0000 7f392a59c700 1 -- 172.21.15.189:0/3827547418 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392c0b3340 con 0x7f392c15d7b0 2024-04-06T12:31:03.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.696+0000 7f392a59c700 1 --2- 172.21.15.189:0/3827547418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c15d7b0 0x7f392c0b3e20 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f391c0098f0 tx=0x7f391c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376494f17b4b5e6d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:03.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.697+0000 7f392959a700 1 -- 172.21.15.189:0/3827547418 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c011830 con 0x7f392c15d7b0 2024-04-06T12:31:03.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.697+0000 7f392959a700 1 -- 172.21.15.189:0/3827547418 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391c011990 con 0x7f392c15d7b0 2024-04-06T12:31:03.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.697+0000 7f392959a700 1 -- 172.21.15.189:0/3827547418 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c011d20 con 0x7f392c15d7b0 2024-04-06T12:31:03.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.697+0000 7f3930e7b700 1 -- 172.21.15.189:0/3827547418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c15d7b0 msgr2=0x7f392c0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:03.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.697+0000 7f3930e7b700 1 --2- 172.21.15.189:0/3827547418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c15d7b0 0x7f392c0b3e20 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f391c0098f0 tx=0x7f391c009c00 comp rx=0 tx=0).stop 2024-04-06T12:31:03.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.697+0000 7f3930e7b700 1 -- 172.21.15.189:0/3827547418 shutdown_connections 2024-04-06T12:31:03.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.697+0000 7f3930e7b700 1 --2- 172.21.15.189:0/3827547418 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f392c15e500 0x7f392c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.697+0000 7f3930e7b700 1 --2- 172.21.15.189:0/3827547418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c15d7b0 0x7f392c0b3e20 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.697+0000 7f3930e7b700 1 -- 172.21.15.189:0/3827547418 >> 172.21.15.189:0/3827547418 conn(0x7f392c0d00b0 msgr2=0x7f392c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:03.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.697+0000 7f3930e7b700 1 -- 172.21.15.189:0/3827547418 shutdown_connections 2024-04-06T12:31:03.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.697+0000 7f3930e7b700 1 -- 172.21.15.189:0/3827547418 wait complete. 2024-04-06T12:31:03.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.699+0000 7f3930e7b700 1 Processor -- start 2024-04-06T12:31:03.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.699+0000 7f3930e7b700 1 -- start start 2024-04-06T12:31:03.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.699+0000 7f3930e7b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f392c06a730 0x7f392c15c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:03.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.699+0000 7f3930e7b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c15d7b0 0x7f392c146a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:03.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.699+0000 7f3930e7b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f392c15e500 0x7f392c158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:03.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.699+0000 7f3930e7b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c15bf50 con 0x7f392c15d7b0 2024-04-06T12:31:03.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.699+0000 7f3930e7b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c15c920 con 0x7f392c15e500 2024-04-06T12:31:03.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.699+0000 7f3930e7b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f392c15b340 con 0x7f392c06a730 2024-04-06T12:31:03.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.700+0000 7f3929d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c15d7b0 0x7f392c146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:03.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.700+0000 7f3929d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c15d7b0 0x7f392c146a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36716/0 (socket says 172.21.15.189:36716) 2024-04-06T12:31:03.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.700+0000 7f3929d9b700 1 -- 172.21.15.189:0/3662362148 learned_addr learned my addr 172.21.15.189:0/3662362148 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:03.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.700+0000 7f392ad9d700 1 --2- 172.21.15.189:0/3662362148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f392c15e500 0x7f392c158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:03.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.700+0000 7f392a59c700 1 --2- 172.21.15.189:0/3662362148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f392c06a730 0x7f392c15c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:03.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.700+0000 7f3929d9b700 1 -- 172.21.15.189:0/3662362148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f392c06a730 msgr2=0x7f392c15c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:03.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.700+0000 7f3929d9b700 1 --2- 172.21.15.189:0/3662362148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f392c06a730 0x7f392c15c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.700+0000 7f3929d9b700 1 -- 172.21.15.189:0/3662362148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f392c15e500 msgr2=0x7f392c158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:03.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.700+0000 7f3929d9b700 1 --2- 172.21.15.189:0/3662362148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f392c15e500 0x7f392c158b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.700+0000 7f3929d9b700 1 -- 172.21.15.189:0/3662362148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f391c009550 con 0x7f392c15d7b0 2024-04-06T12:31:03.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.700+0000 7f392a59c700 1 --2- 172.21.15.189:0/3662362148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f392c06a730 0x7f392c15c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:03.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.700+0000 7f3929d9b700 1 --2- 172.21.15.189:0/3662362148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c15d7b0 0x7f392c146a10 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f392000c4e0 tx=0x7f392000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:03.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.700+0000 7f391affd700 1 -- 172.21.15.189:0/3662362148 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39200052d0 con 0x7f392c15d7b0 2024-04-06T12:31:03.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.700+0000 7f391affd700 1 -- 172.21.15.189:0/3662362148 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3920005430 con 0x7f392c15d7b0 2024-04-06T12:31:03.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.700+0000 7f391affd700 1 -- 172.21.15.189:0/3662362148 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39200038c0 con 0x7f392c15d7b0 2024-04-06T12:31:03.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.700+0000 7f3930e7b700 1 -- 172.21.15.189:0/3662362148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392c15b620 con 0x7f392c15d7b0 2024-04-06T12:31:03.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.700+0000 7f3930e7b700 1 -- 172.21.15.189:0/3662362148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f392c15bbe0 con 0x7f392c15d7b0 2024-04-06T12:31:03.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.701+0000 7f3930e7b700 1 -- 172.21.15.189:0/3662362148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f392c067bb0 con 0x7f392c15d7b0 2024-04-06T12:31:03.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.702+0000 7f391affd700 1 -- 172.21.15.189:0/3662362148 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 71) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f3920003a20 con 0x7f392c15d7b0 2024-04-06T12:31:03.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.702+0000 7f391affd700 1 --2- 172.21.15.189:0/3662362148 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f390c06aed0 0x7f390c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:03.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.703+0000 7f392a59c700 1 -- 172.21.15.189:0/3662362148 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f390c06aed0 msgr2=0x7f390c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:03.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.703+0000 7f391affd700 1 -- 172.21.15.189:0/3662362148 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f3920089180 con 0x7f392c15d7b0 2024-04-06T12:31:03.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.703+0000 7f392a59c700 1 --2- 172.21.15.189:0/3662362148 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f390c06aed0 0x7f390c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:03.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.706+0000 7f391affd700 1 -- 172.21.15.189:0/3662362148 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3920053e10 con 0x7f392c15d7b0 2024-04-06T12:31:03.729 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:03.732+0000 7f6920c15200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:31:03.751 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:03.754+0000 7f3500729200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:31:03.814 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:03.817+0000 7f6920c15200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:31:03.825 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:03.828+0000 7f3500729200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:31:03.893 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:03.895+0000 7f6920c15200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:31:03.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.904+0000 7f392a59c700 1 -- 172.21.15.189:0/3662362148 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f390c06aed0 msgr2=0x7f390c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:03.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.904+0000 7f392a59c700 1 --2- 172.21.15.189:0/3662362148 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f390c06aed0 0x7f390c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:03.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.910+0000 7f3930e7b700 1 -- 172.21.15.189:0/3662362148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f392c074990 con 0x7f392c15d7b0 2024-04-06T12:31:03.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.910+0000 7f391affd700 1 -- 172.21.15.189:0/3662362148 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v129) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3920017020 con 0x7f392c15d7b0 2024-04-06T12:31:03.908 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:31:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.912+0000 7f3930e7b700 1 -- 172.21.15.189:0/3662362148 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f390c06aed0 msgr2=0x7f390c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.912+0000 7f3930e7b700 1 --2- 172.21.15.189:0/3662362148 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f390c06aed0 0x7f390c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.913+0000 7f3930e7b700 1 -- 172.21.15.189:0/3662362148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c15d7b0 msgr2=0x7f392c146a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.913+0000 7f3930e7b700 1 --2- 172.21.15.189:0/3662362148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c15d7b0 0x7f392c146a10 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f392000c4e0 tx=0x7f392000c7f0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.913+0000 7f3930e7b700 1 -- 172.21.15.189:0/3662362148 shutdown_connections 2024-04-06T12:31:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.913+0000 7f3930e7b700 1 --2- 172.21.15.189:0/3662362148 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f390c06aed0 0x7f390c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.913+0000 7f3930e7b700 1 --2- 172.21.15.189:0/3662362148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f392c15e500 0x7f392c158b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.913+0000 7f3930e7b700 1 --2- 172.21.15.189:0/3662362148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f392c06a730 0x7f392c15c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.913+0000 7f3930e7b700 1 --2- 172.21.15.189:0/3662362148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c15d7b0 0x7f392c146a10 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.913+0000 7f3930e7b700 1 -- 172.21.15.189:0/3662362148 >> 172.21.15.189:0/3662362148 conn(0x7f392c0d00b0 msgr2=0x7f392c06eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.913+0000 7f3930e7b700 1 -- 172.21.15.189:0/3662362148 shutdown_connections 2024-04-06T12:31:03.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:03.913+0000 7f3930e7b700 1 -- 172.21.15.189:0/3662362148 wait complete. 2024-04-06T12:31:03.922 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:31:04.029 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:04.032+0000 7f3500729200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:31:04.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.057+0000 7f870b2f7700 1 Processor -- start 2024-04-06T12:31:04.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.057+0000 7f870b2f7700 1 -- start start 2024-04-06T12:31:04.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.058+0000 7f870b2f7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8704153c80 0x7f870415d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:04.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.058+0000 7f870b2f7700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f870415a530 0x7f87040b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:04.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.058+0000 7f870b2f7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f87040b3000 0x7f870406c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:04.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.058+0000 7f870b2f7700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87040b3470 con 0x7f87040b3000 2024-04-06T12:31:04.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.058+0000 7f870b2f7700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8704071130 con 0x7f870415a530 2024-04-06T12:31:04.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.058+0000 7f870b2f7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870415d680 con 0x7f8704153c80 2024-04-06T12:31:04.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.058+0000 7f8708892700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f870415a530 0x7f87040b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:04.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.058+0000 7f8709093700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8704153c80 0x7f870415d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:04.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.059+0000 7f8709093700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8704153c80 0x7f870415d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:43236/0 (socket says 172.21.15.189:43236) 2024-04-06T12:31:04.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.059+0000 7f8709093700 1 -- 172.21.15.189:0/1892385576 learned_addr learned my addr 172.21.15.189:0/1892385576 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:04.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.059+0000 7f8709894700 1 --2- 172.21.15.189:0/1892385576 >> v2:172.21.15.189:3300/0 conn(0x7f87040b3000 0x7f870406c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:04.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.059+0000 7f8709093700 1 -- 172.21.15.189:0/1892385576 >> v2:172.21.15.192:3300/0 conn(0x7f870415a530 msgr2=0x7f87040b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:04.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.059+0000 7f8709093700 1 --2- 172.21.15.189:0/1892385576 >> v2:172.21.15.192:3300/0 conn(0x7f870415a530 0x7f87040b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.059+0000 7f8709093700 1 -- 172.21.15.189:0/1892385576 >> v2:172.21.15.189:3300/0 conn(0x7f87040b3000 msgr2=0x7f870406c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:04.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.059+0000 7f8709093700 1 --2- 172.21.15.189:0/1892385576 >> v2:172.21.15.189:3300/0 conn(0x7f87040b3000 0x7f870406c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.059+0000 7f8709093700 1 -- 172.21.15.189:0/1892385576 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870415d960 con 0x7f8704153c80 2024-04-06T12:31:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.059+0000 7f8709093700 1 --2- 172.21.15.189:0/1892385576 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8704153c80 0x7f870415d0a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f870406d570 tx=0x7f86fc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59d2d37a592a33a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.059+0000 7f86f7fff700 1 -- 172.21.15.189:0/1892385576 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc00e070 con 0x7f8704153c80 2024-04-06T12:31:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.059+0000 7f86f7fff700 1 -- 172.21.15.189:0/1892385576 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86fc00b910 con 0x7f8704153c80 2024-04-06T12:31:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.059+0000 7f86f7fff700 1 -- 172.21.15.189:0/1892385576 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc00bcc0 con 0x7f8704153c80 2024-04-06T12:31:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.060+0000 7f870b2f7700 1 -- 172.21.15.189:0/1892385576 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8704153c80 msgr2=0x7f870415d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.060+0000 7f870b2f7700 1 --2- 172.21.15.189:0/1892385576 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8704153c80 0x7f870415d0a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f870406d570 tx=0x7f86fc00b330 comp rx=0 tx=0).stop 2024-04-06T12:31:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.060+0000 7f870b2f7700 1 -- 172.21.15.189:0/1892385576 shutdown_connections 2024-04-06T12:31:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.060+0000 7f870b2f7700 1 --2- 172.21.15.189:0/1892385576 >> v2:172.21.15.189:3300/0 conn(0x7f87040b3000 0x7f870406c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.060+0000 7f870b2f7700 1 --2- 172.21.15.189:0/1892385576 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8704153c80 0x7f870415d0a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.060+0000 7f870b2f7700 1 --2- 172.21.15.189:0/1892385576 >> v2:172.21.15.192:3300/0 conn(0x7f870415a530 0x7f87040b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.060+0000 7f870b2f7700 1 -- 172.21.15.189:0/1892385576 >> 172.21.15.189:0/1892385576 conn(0x7f870406a730 msgr2=0x7f870406ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.060+0000 7f870b2f7700 1 -- 172.21.15.189:0/1892385576 shutdown_connections 2024-04-06T12:31:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.060+0000 7f870b2f7700 1 -- 172.21.15.189:0/1892385576 wait complete. 2024-04-06T12:31:04.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.062+0000 7f870b2f7700 1 Processor -- start 2024-04-06T12:31:04.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.062+0000 7f870b2f7700 1 -- start start 2024-04-06T12:31:04.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.063+0000 7f870b2f7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87040b3000 0x7f870414bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:04.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.063+0000 7f870b2f7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8704153c80 0x7f87040cd070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:04.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.063+0000 7f870b2f7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f870415a530 0x7f870414f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:04.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.063+0000 7f870b2f7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87040ce4c0 con 0x7f870415a530 2024-04-06T12:31:04.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.063+0000 7f870b2f7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870414c040 con 0x7f87040b3000 2024-04-06T12:31:04.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.063+0000 7f870b2f7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87040cd5b0 con 0x7f8704153c80 2024-04-06T12:31:04.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.064+0000 7f8709894700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f870415a530 0x7f870414f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:04.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.064+0000 7f8709894700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f870415a530 0x7f870414f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36730/0 (socket says 172.21.15.189:36730) 2024-04-06T12:31:04.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.064+0000 7f8709894700 1 -- 172.21.15.189:0/2397631620 learned_addr learned my addr 172.21.15.189:0/2397631620 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:04.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.064+0000 7f8708892700 1 --2- 172.21.15.189:0/2397631620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8704153c80 0x7f87040cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:04.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.064+0000 7f8709093700 1 --2- 172.21.15.189:0/2397631620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87040b3000 0x7f870414bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:04.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.064+0000 7f8709894700 1 -- 172.21.15.189:0/2397631620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8704153c80 msgr2=0x7f87040cd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:04.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.064+0000 7f8709894700 1 --2- 172.21.15.189:0/2397631620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8704153c80 0x7f87040cd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.064+0000 7f8709894700 1 -- 172.21.15.189:0/2397631620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87040b3000 msgr2=0x7f870414bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:04.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.064+0000 7f8709894700 1 --2- 172.21.15.189:0/2397631620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87040b3000 0x7f870414bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.064+0000 7f8709894700 1 -- 172.21.15.189:0/2397631620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86fc009b30 con 0x7f870415a530 2024-04-06T12:31:04.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.064+0000 7f8709894700 1 --2- 172.21.15.189:0/2397631620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f870415a530 0x7f870414f7c0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f870000e950 tx=0x7f870000ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:04.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.065+0000 7f86f5ffb700 1 -- 172.21.15.189:0/2397631620 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8700009730 con 0x7f870415a530 2024-04-06T12:31:04.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.065+0000 7f86f5ffb700 1 -- 172.21.15.189:0/2397631620 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8700013680 con 0x7f870415a530 2024-04-06T12:31:04.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.065+0000 7f86f5ffb700 1 -- 172.21.15.189:0/2397631620 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8700013a10 con 0x7f870415a530 2024-04-06T12:31:04.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.066+0000 7f870b2f7700 1 -- 172.21.15.189:0/2397631620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87040cdf00 con 0x7f870415a530 2024-04-06T12:31:04.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.066+0000 7f870b2f7700 1 -- 172.21.15.189:0/2397631620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87040cc870 con 0x7f870415a530 2024-04-06T12:31:04.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.067+0000 7f870b2f7700 1 -- 172.21.15.189:0/2397631620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87040cc6d0 con 0x7f870415a530 2024-04-06T12:31:04.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.069+0000 7f86f5ffb700 1 -- 172.21.15.189:0/2397631620 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 71) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f8700013b70 con 0x7f870415a530 2024-04-06T12:31:04.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.070+0000 7f86f5ffb700 1 --2- 172.21.15.189:0/2397631620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86e806aed0 0x7f86e806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:04.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.070+0000 7f8709093700 1 -- 172.21.15.189:0/2397631620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86e806aed0 msgr2=0x7f86e806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:04.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.070+0000 7f86f5ffb700 1 -- 172.21.15.189:0/2397631620 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f87000898b0 con 0x7f870415a530 2024-04-06T12:31:04.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.070+0000 7f8709093700 1 --2- 172.21.15.189:0/2397631620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86e806aed0 0x7f86e806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:04.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.077+0000 7f86f5ffb700 1 -- 172.21.15.189:0/2397631620 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f87040cc6d0 con 0x7f870415a530 2024-04-06T12:31:04.106 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:04.109+0000 7f3500729200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:31:04.108 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:04.111+0000 7f6920c15200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:31:04.186 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:04.189+0000 7f6920c15200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:31:04.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.271+0000 7f8709093700 1 -- 172.21.15.189:0/2397631620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86e806aed0 msgr2=0x7f86e806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:04.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.271+0000 7f8709093700 1 --2- 172.21.15.189:0/2397631620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86e806aed0 0x7f86e806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:04.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.300+0000 7f870b2f7700 1 -- 172.21.15.189:0/2397631620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f87040cc6d0 con 0x7f870415a530 2024-04-06T12:31:04.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.300+0000 7f86f5ffb700 1 -- 172.21.15.189:0/2397631620 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v129) v1 ==== 100+0+11 (secure 0 0 0) 0x7f87040cc6d0 con 0x7f870415a530 2024-04-06T12:31:04.297 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:31:04.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.302+0000 7f870b2f7700 1 -- 172.21.15.189:0/2397631620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86e806aed0 msgr2=0x7f86e806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:04.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.302+0000 7f870b2f7700 1 --2- 172.21.15.189:0/2397631620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86e806aed0 0x7f86e806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.302+0000 7f870b2f7700 1 -- 172.21.15.189:0/2397631620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f870415a530 msgr2=0x7f870414f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:04.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.302+0000 7f870b2f7700 1 --2- 172.21.15.189:0/2397631620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f870415a530 0x7f870414f7c0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f870000e950 tx=0x7f870000ec60 comp rx=0 tx=0).stop 2024-04-06T12:31:04.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.302+0000 7f870b2f7700 1 -- 172.21.15.189:0/2397631620 shutdown_connections 2024-04-06T12:31:04.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.302+0000 7f870b2f7700 1 --2- 172.21.15.189:0/2397631620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f870415a530 0x7f870414f7c0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.302+0000 7f870b2f7700 1 --2- 172.21.15.189:0/2397631620 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86e806aed0 0x7f86e806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.302+0000 7f870b2f7700 1 --2- 172.21.15.189:0/2397631620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87040b3000 0x7f870414bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.302+0000 7f870b2f7700 1 --2- 172.21.15.189:0/2397631620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8704153c80 0x7f87040cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.302+0000 7f870b2f7700 1 -- 172.21.15.189:0/2397631620 >> 172.21.15.189:0/2397631620 conn(0x7f870406a730 msgr2=0x7f870414ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:04.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.303+0000 7f870b2f7700 1 -- 172.21.15.189:0/2397631620 shutdown_connections 2024-04-06T12:31:04.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.303+0000 7f870b2f7700 1 -- 172.21.15.189:0/2397631620 wait complete. 2024-04-06T12:31:04.313 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T12:31:04.423 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:04.426+0000 7f3500729200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:31:04.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.444+0000 7fd4acf06700 1 Processor -- start 2024-04-06T12:31:04.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.444+0000 7fd4acf06700 1 -- start start 2024-04-06T12:31:04.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.445+0000 7fd4acf06700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd4a80b5380 0x7fd4a80d2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:04.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.445+0000 7fd4acf06700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4a80b5cc0 con 0x7fd4a815d590 2024-04-06T12:31:04.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.445+0000 7fd4acf06700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4a80b57d0 con 0x7fd4a80b4ed0 2024-04-06T12:31:04.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.445+0000 7fd4acf06700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4a80d1500 con 0x7fd4a80b5380 2024-04-06T12:31:04.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.445+0000 7fd4a659c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd4a815d590 0x7fd4a80b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45314/0 (socket says 172.21.15.189:45314) 2024-04-06T12:31:04.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.445+0000 7fd4a659c700 1 -- 172.21.15.189:0/479824925 learned_addr learned my addr 172.21.15.189:0/479824925 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:04.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.445+0000 7fd4a6d9d700 1 --2- 172.21.15.189:0/479824925 >> v2:172.21.15.192:3300/0 conn(0x7fd4a80b5380 0x7fd4a80d2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:04.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.446+0000 7fd4a559a700 1 -- 172.21.15.189:0/479824925 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461808356 0 0) 0x7fd4a80b5cc0 con 0x7fd4a815d590 2024-04-06T12:31:04.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.446+0000 7fd4a559a700 1 -- 172.21.15.189:0/479824925 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4900035b0 con 0x7fd4a815d590 2024-04-06T12:31:04.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.446+0000 7fd4a559a700 1 -- 172.21.15.189:0/479824925 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684828290 0 0) 0x7fd4a80b57d0 con 0x7fd4a80b4ed0 2024-04-06T12:31:04.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.446+0000 7fd4a559a700 1 -- 172.21.15.189:0/479824925 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4a80b5cc0 con 0x7fd4a80b4ed0 2024-04-06T12:31:04.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.446+0000 7fd4a559a700 1 -- 172.21.15.189:0/479824925 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 851288079 0 0) 0x7fd4900035b0 con 0x7fd4a815d590 2024-04-06T12:31:04.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.446+0000 7fd4a6d9d700 1 -- 172.21.15.189:0/479824925 >> v1:172.21.15.192:6789/0 conn(0x7fd4a80b4ed0 legacy=0x7fd4a80d2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:04.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.446+0000 7fd4a6d9d700 1 -- 172.21.15.189:0/479824925 >> v1:172.21.15.189:6789/0 conn(0x7fd4a815d590 legacy=0x7fd4a80b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:04.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.446+0000 7fd4a6d9d700 1 -- 172.21.15.189:0/479824925 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a80d17e0 con 0x7fd4a80b5380 2024-04-06T12:31:04.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.446+0000 7fd4a6d9d700 1 --2- 172.21.15.189:0/479824925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4a80b5380 0x7fd4a80d2f90 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0009d90 tx=0x7fd4a000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c4906196b3c24b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:04.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.446+0000 7fd4a559a700 1 -- 172.21.15.189:0/479824925 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a000e070 con 0x7fd4a80b5380 2024-04-06T12:31:04.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.446+0000 7fd4a559a700 1 -- 172.21.15.189:0/479824925 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a0003a20 con 0x7fd4a80b5380 2024-04-06T12:31:04.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.446+0000 7fd4a559a700 1 -- 172.21.15.189:0/479824925 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd498001670 con 0x7fd4a80b5380 2024-04-06T12:31:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.447+0000 7fd4acf06700 1 -- 172.21.15.189:0/479824925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4a80b5380 msgr2=0x7fd4a80d2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.447+0000 7fd4acf06700 1 --2- 172.21.15.189:0/479824925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4a80b5380 0x7fd4a80d2f90 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0009d90 tx=0x7fd4a000b330 comp rx=0 tx=0).stop 2024-04-06T12:31:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.447+0000 7fd4acf06700 1 -- 172.21.15.189:0/479824925 shutdown_connections 2024-04-06T12:31:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.447+0000 7fd4acf06700 1 --2- 172.21.15.189:0/479824925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4a80b5380 0x7fd4a80d2f90 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.447+0000 7fd4acf06700 1 -- 172.21.15.189:0/479824925 >> 172.21.15.189:0/479824925 conn(0x7fd4a806bbc0 msgr2=0x7fd4a806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.447+0000 7fd4acf06700 1 -- 172.21.15.189:0/479824925 shutdown_connections 2024-04-06T12:31:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.447+0000 7fd4acf06700 1 -- 172.21.15.189:0/479824925 wait complete. 2024-04-06T12:31:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.448+0000 7fd4acf06700 1 Processor -- start 2024-04-06T12:31:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.448+0000 7fd4acf06700 1 -- start start 2024-04-06T12:31:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.449+0000 7fd4acf06700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a80b4ed0 0x7fd4a80c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.449+0000 7fd4acf06700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4a80b5380 0x7fd4a81f63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.449+0000 7fd4acf06700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4a815d590 0x7fd4a81f68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.449+0000 7fd4acf06700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a815ff80 con 0x7fd4a80b4ed0 2024-04-06T12:31:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.449+0000 7fd4acf06700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a8160260 con 0x7fd4a815d590 2024-04-06T12:31:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.449+0000 7fd4acf06700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a8160540 con 0x7fd4a80b5380 2024-04-06T12:31:04.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.449+0000 7fd4a659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a80b4ed0 0x7fd4a80c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:04.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.449+0000 7fd4a659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a80b4ed0 0x7fd4a80c96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36736/0 (socket says 172.21.15.189:36736) 2024-04-06T12:31:04.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.449+0000 7fd4a659c700 1 -- 172.21.15.189:0/2639178946 learned_addr learned my addr 172.21.15.189:0/2639178946 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:04.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.449+0000 7fd4a6d9d700 1 --2- 172.21.15.189:0/2639178946 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4a815d590 0x7fd4a81f68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:04.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.449+0000 7fd4a5d9b700 1 --2- 172.21.15.189:0/2639178946 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4a80b5380 0x7fd4a81f63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:04.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.449+0000 7fd4a659c700 1 -- 172.21.15.189:0/2639178946 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4a80b5380 msgr2=0x7fd4a81f63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:04.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.449+0000 7fd4a659c700 1 --2- 172.21.15.189:0/2639178946 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4a80b5380 0x7fd4a81f63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.449+0000 7fd4a659c700 1 -- 172.21.15.189:0/2639178946 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4a815d590 msgr2=0x7fd4a81f68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:04.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.449+0000 7fd4a659c700 1 --2- 172.21.15.189:0/2639178946 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4a815d590 0x7fd4a81f68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.449+0000 7fd4a659c700 1 -- 172.21.15.189:0/2639178946 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a0009a70 con 0x7fd4a80b4ed0 2024-04-06T12:31:04.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.449+0000 7fd4a5d9b700 1 --2- 172.21.15.189:0/2639178946 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4a80b5380 0x7fd4a81f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:04.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.450+0000 7fd4a659c700 1 --2- 172.21.15.189:0/2639178946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a80b4ed0 0x7fd4a80c96f0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fd49800c6b0 tx=0x7fd49800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:04.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.450+0000 7fd496ffd700 1 -- 172.21.15.189:0/2639178946 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd498007620 con 0x7fd4a80b4ed0 2024-04-06T12:31:04.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.450+0000 7fd496ffd700 1 -- 172.21.15.189:0/2639178946 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd498004870 con 0x7fd4a80b4ed0 2024-04-06T12:31:04.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.450+0000 7fd4acf06700 1 -- 172.21.15.189:0/2639178946 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a815de70 con 0x7fd4a80b4ed0 2024-04-06T12:31:04.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.450+0000 7fd496ffd700 1 -- 172.21.15.189:0/2639178946 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd498004c00 con 0x7fd4a80b4ed0 2024-04-06T12:31:04.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.450+0000 7fd4acf06700 1 -- 172.21.15.189:0/2639178946 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a815b300 con 0x7fd4a80b4ed0 2024-04-06T12:31:04.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.451+0000 7fd4acf06700 1 -- 172.21.15.189:0/2639178946 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a8074990 con 0x7fd4a80b4ed0 2024-04-06T12:31:04.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.451+0000 7fd496ffd700 1 -- 172.21.15.189:0/2639178946 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 71) v1 ==== 89361+0+0 (secure 0 0 0) 0x7fd4980133c0 con 0x7fd4a80b4ed0 2024-04-06T12:31:04.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.452+0000 7fd496ffd700 1 --2- 172.21.15.189:0/2639178946 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd48806aed0 0x7fd48806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:04.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.452+0000 7fd4a5d9b700 1 -- 172.21.15.189:0/2639178946 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd48806aed0 msgr2=0x7fd48806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:04.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.452+0000 7fd496ffd700 1 -- 172.21.15.189:0/2639178946 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fd4980896c0 con 0x7fd4a80b4ed0 2024-04-06T12:31:04.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.452+0000 7fd4a5d9b700 1 --2- 172.21.15.189:0/2639178946 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd48806aed0 0x7fd48806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:04.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.456+0000 7fd496ffd700 1 -- 172.21.15.189:0/2639178946 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd498054380 con 0x7fd4a80b4ed0 2024-04-06T12:31:04.526 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:04.529+0000 7f6920c15200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:31:04.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.653+0000 7fd4a5d9b700 1 -- 172.21.15.189:0/2639178946 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd48806aed0 msgr2=0x7fd48806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:04.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.653+0000 7fd4a5d9b700 1 --2- 172.21.15.189:0/2639178946 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd48806aed0 0x7fd48806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:04.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.674+0000 7fd4acf06700 1 -- 172.21.15.189:0/2639178946 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd4a81600c0 con 0x7fd4a80b4ed0 2024-04-06T12:31:04.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.675+0000 7fd496ffd700 1 -- 172.21.15.189:0/2639178946 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v129) v1 ==== 80+0+465 (secure 0 0 0) 0x7fd4a81600c0 con 0x7fd4a80b4ed0 2024-04-06T12:31:04.672 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/4170508796 2024-04-07T12:30:31.084606+0000 2024-04-06T12:31:04.672 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/169651662 2024-04-07T12:30:31.084606+0000 2024-04-06T12:31:04.672 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/29183104 2024-04-07T12:30:31.084606+0000 2024-04-06T12:31:04.672 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/3994101871 2024-04-07T12:30:31.084606+0000 2024-04-06T12:31:04.672 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1300227922 2024-04-07T12:30:31.084606+0000 2024-04-06T12:31:04.672 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6833/47128 2024-04-07T12:30:31.084606+0000 2024-04-06T12:31:04.672 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6832/47128 2024-04-07T12:30:31.084606+0000 2024-04-06T12:31:04.672 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1086259216 2024-04-07T12:30:31.084606+0000 2024-04-06T12:31:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.677+0000 7fd4acf06700 1 -- 172.21.15.189:0/2639178946 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd48806aed0 msgr2=0x7fd48806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.677+0000 7fd4acf06700 1 --2- 172.21.15.189:0/2639178946 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd48806aed0 0x7fd48806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.677+0000 7fd4acf06700 1 -- 172.21.15.189:0/2639178946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a80b4ed0 msgr2=0x7fd4a80c96f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.677+0000 7fd4acf06700 1 --2- 172.21.15.189:0/2639178946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a80b4ed0 0x7fd4a80c96f0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fd49800c6b0 tx=0x7fd49800c9c0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.677+0000 7fd4acf06700 1 -- 172.21.15.189:0/2639178946 shutdown_connections 2024-04-06T12:31:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.677+0000 7fd4acf06700 1 --2- 172.21.15.189:0/2639178946 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd48806aed0 0x7fd48806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.677+0000 7fd4acf06700 1 --2- 172.21.15.189:0/2639178946 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4a815d590 0x7fd4a81f68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.677+0000 7fd4acf06700 1 --2- 172.21.15.189:0/2639178946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a80b4ed0 0x7fd4a80c96f0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.677+0000 7fd4acf06700 1 --2- 172.21.15.189:0/2639178946 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4a80b5380 0x7fd4a81f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.677+0000 7fd4acf06700 1 -- 172.21.15.189:0/2639178946 >> 172.21.15.189:0/2639178946 conn(0x7fd4a806bbc0 msgr2=0x7fd4a8156fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.677+0000 7fd4acf06700 1 -- 172.21.15.189:0/2639178946 shutdown_connections 2024-04-06T12:31:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.677+0000 7fd4acf06700 1 -- 172.21.15.189:0/2639178946 wait complete. 2024-04-06T12:31:04.676 INFO:teuthology.orchestra.run.smithi189.stderr:listed 8 entries 2024-04-06T12:31:04.687 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-06T12:31:04.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:04.687 DEBUG:teuthology.orchestra.run.smithi189:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-06T12:31:04.727 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T12:31:04.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:04.728 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-06T12:31:04.745 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:04.748+0000 7f3500729200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:31:04.786 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:31:04.786 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:31:04.803 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:31:04.804 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:31:04.804 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:31:04.843 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:31:04.843 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:31:04.843 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:31:04.861 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:31:04.862 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:31:04.868 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:04.871+0000 7f6920c15200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:31:04.908 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:04.911+0000 7f3500729200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:31:04.981 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:04.983+0000 7f3500729200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:31:04.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.995+0000 7f32925e3700 1 Processor -- start 2024-04-06T12:31:04.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.995+0000 7f32925e3700 1 -- start start 2024-04-06T12:31:05.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.996+0000 7f32925e3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f328c143470 0x7f328c0b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:05.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.996+0000 7f32925e3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328c15eb40 con 0x7f328c160b60 2024-04-06T12:31:05.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.996+0000 7f32925e3700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328c0d0bf0 con 0x7f328c15db10 2024-04-06T12:31:05.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.996+0000 7f32925e3700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328c160f40 con 0x7f328c143470 2024-04-06T12:31:05.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.996+0000 7f3290b80700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f328c143470 0x7f328c0b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:05.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.996+0000 7f328bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f328c160b60 0x7f328c142d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45320/0 (socket says 172.21.15.189:45320) 2024-04-06T12:31:05.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.996+0000 7f328bfff700 1 -- 172.21.15.189:0/3741052442 learned_addr learned my addr 172.21.15.189:0/3741052442 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:05.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.996+0000 7f3290b80700 1 -- 172.21.15.189:0/3741052442 >> v1:172.21.15.192:6789/0 conn(0x7f328c15db10 legacy=0x7f328c0cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:05.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.996+0000 7f3290b80700 1 -- 172.21.15.189:0/3741052442 >> v1:172.21.15.189:6789/0 conn(0x7f328c160b60 legacy=0x7f328c142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:05.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.996+0000 7f328affd700 1 -- 172.21.15.189:0/3741052442 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93002558 0 0) 0x7f3280001110 con 0x7f328c160b60 2024-04-06T12:31:05.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.996+0000 7f3290b80700 1 -- 172.21.15.189:0/3741052442 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c15dfc0 con 0x7f328c143470 2024-04-06T12:31:05.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.996+0000 7f3290b80700 1 --2- 172.21.15.189:0/3741052442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c143470 0x7f328c0b37f0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f3284009d90 tx=0x7f328400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902fd67da8989838 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:05.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.997+0000 7f328affd700 1 -- 172.21.15.189:0/3741052442 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328400e070 con 0x7f328c143470 2024-04-06T12:31:05.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.997+0000 7f328affd700 1 -- 172.21.15.189:0/3741052442 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328400bbd0 con 0x7f328c143470 2024-04-06T12:31:05.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.997+0000 7f328affd700 1 -- 172.21.15.189:0/3741052442 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3284012430 con 0x7f328c143470 2024-04-06T12:31:05.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.997+0000 7f32925e3700 1 -- 172.21.15.189:0/3741052442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c143470 msgr2=0x7f328c0b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:05.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.997+0000 7f32925e3700 1 --2- 172.21.15.189:0/3741052442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c143470 0x7f328c0b37f0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f3284009d90 tx=0x7f328400b330 comp rx=0 tx=0).stop 2024-04-06T12:31:05.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.997+0000 7f32925e3700 1 -- 172.21.15.189:0/3741052442 shutdown_connections 2024-04-06T12:31:05.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.997+0000 7f32925e3700 1 --2- 172.21.15.189:0/3741052442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c143470 0x7f328c0b37f0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.997+0000 7f32925e3700 1 -- 172.21.15.189:0/3741052442 >> 172.21.15.189:0/3741052442 conn(0x7f328c06a730 msgr2=0x7f328c06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:05.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.997+0000 7f32925e3700 1 -- 172.21.15.189:0/3741052442 shutdown_connections 2024-04-06T12:31:05.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.997+0000 7f32925e3700 1 -- 172.21.15.189:0/3741052442 wait complete. 2024-04-06T12:31:05.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.999+0000 7f32925e3700 1 Processor -- start 2024-04-06T12:31:05.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.999+0000 7f32925e3700 1 -- start start 2024-04-06T12:31:05.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.999+0000 7f32925e3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f328c143470 0x7f328c15ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:05.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.999+0000 7f32925e3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c15db10 0x7f328c0c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:05.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.999+0000 7f32925e3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328c160b60 0x7f328c15f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:05.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.999+0000 7f32925e3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c15d210 con 0x7f328c15db10 2024-04-06T12:31:05.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.999+0000 7f32925e3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c0c8ea0 con 0x7f328c143470 2024-04-06T12:31:05.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:04.999+0000 7f32925e3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328c0c6370 con 0x7f328c160b60 2024-04-06T12:31:05.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.000+0000 7f328b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c15db10 0x7f328c0c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:05.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.000+0000 7f328b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c15db10 0x7f328c0c7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36754/0 (socket says 172.21.15.189:36754) 2024-04-06T12:31:05.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.000+0000 7f328b7fe700 1 -- 172.21.15.189:0/3165964944 learned_addr learned my addr 172.21.15.189:0/3165964944 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:05.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.000+0000 7f328bfff700 1 --2- 172.21.15.189:0/3165964944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f328c143470 0x7f328c15ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:05.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.000+0000 7f3290b80700 1 --2- 172.21.15.189:0/3165964944 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328c160b60 0x7f328c15f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:05.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.000+0000 7f328b7fe700 1 -- 172.21.15.189:0/3165964944 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328c160b60 msgr2=0x7f328c15f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:05.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.000+0000 7f328b7fe700 1 --2- 172.21.15.189:0/3165964944 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328c160b60 0x7f328c15f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.000+0000 7f328b7fe700 1 -- 172.21.15.189:0/3165964944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f328c143470 msgr2=0x7f328c15ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:05.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.000+0000 7f328b7fe700 1 --2- 172.21.15.189:0/3165964944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f328c143470 0x7f328c15ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.000+0000 7f328b7fe700 1 -- 172.21.15.189:0/3165964944 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3284009a70 con 0x7f328c15db10 2024-04-06T12:31:05.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.000+0000 7f328bfff700 1 --2- 172.21.15.189:0/3165964944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f328c143470 0x7f328c15ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:05.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.000+0000 7f328b7fe700 1 --2- 172.21.15.189:0/3165964944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c15db10 0x7f328c0c7410 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f327c00c4d0 tx=0x7f327c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:05.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.000+0000 7f3288ff9700 1 -- 172.21.15.189:0/3165964944 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c0052d0 con 0x7f328c15db10 2024-04-06T12:31:05.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.000+0000 7f3288ff9700 1 -- 172.21.15.189:0/3165964944 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327c005430 con 0x7f328c15db10 2024-04-06T12:31:05.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.000+0000 7f32925e3700 1 -- 172.21.15.189:0/3165964944 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328c0c6650 con 0x7f328c15db10 2024-04-06T12:31:05.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.000+0000 7f3288ff9700 1 -- 172.21.15.189:0/3165964944 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c0038f0 con 0x7f328c15db10 2024-04-06T12:31:05.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.000+0000 7f32925e3700 1 -- 172.21.15.189:0/3165964944 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328c0c6c10 con 0x7f328c15db10 2024-04-06T12:31:05.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.001+0000 7f32925e3700 1 -- 172.21.15.189:0/3165964944 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328c074990 con 0x7f328c15db10 2024-04-06T12:31:05.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.002+0000 7f3288ff9700 1 -- 172.21.15.189:0/3165964944 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 71) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f327c003a50 con 0x7f328c15db10 2024-04-06T12:31:05.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.002+0000 7f3288ff9700 1 --2- 172.21.15.189:0/3165964944 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f327806aed0 0x7f327806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:05.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.002+0000 7f328bfff700 1 -- 172.21.15.189:0/3165964944 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f327806aed0 msgr2=0x7f327806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:05.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.002+0000 7f3288ff9700 1 -- 172.21.15.189:0/3165964944 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f327c088d20 con 0x7f328c15db10 2024-04-06T12:31:05.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.002+0000 7f328bfff700 1 --2- 172.21.15.189:0/3165964944 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f327806aed0 0x7f327806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:05.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.007+0000 7f3288ff9700 1 -- 172.21.15.189:0/3165964944 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f327c0539d0 con 0x7f328c15db10 2024-04-06T12:31:05.040 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:05.043+0000 7f6920c15200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:31:05.116 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:05.119+0000 7f6920c15200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:31:05.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.203+0000 7f328bfff700 1 -- 172.21.15.189:0/3165964944 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f327806aed0 msgr2=0x7f327806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:05.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.203+0000 7f328bfff700 1 --2- 172.21.15.189:0/3165964944 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f327806aed0 0x7f327806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:05.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.259+0000 7f32925e3700 1 -- 172.21.15.189:0/3165964944 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f328c0b35d0 con 0x7f328c15db10 2024-04-06T12:31:05.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.260+0000 7f3288ff9700 1 -- 172.21.15.189:0/3165964944 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f327c017090 con 0x7f328c15db10 2024-04-06T12:31:05.257 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:05.257 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/749266547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":749266547},{"type":"v1","addr":"172.21.15.192:6835","nonce":749266547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":6764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/947183051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":947183051},{"type":"v1","addr":"172.21.15.192:6834","nonce":947183051}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7492,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3990051444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3990051444},{"type":"v1","addr":"172.21.15.189:6837","nonce":3990051444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:30:17.127295+0000","modified":"2024-04-06T12:30:20.144458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8088},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8088":{"gid":8088,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/393232044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":393232044},{"type":"v1","addr":"172.21.15.189:6836","nonce":393232044}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-06T12:31:05.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.262+0000 7f32925e3700 1 -- 172.21.15.189:0/3165964944 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f327806aed0 msgr2=0x7f327806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:05.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.262+0000 7f32925e3700 1 --2- 172.21.15.189:0/3165964944 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f327806aed0 0x7f327806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.262+0000 7f32925e3700 1 -- 172.21.15.189:0/3165964944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c15db10 msgr2=0x7f328c0c7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:05.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.262+0000 7f32925e3700 1 --2- 172.21.15.189:0/3165964944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c15db10 0x7f328c0c7410 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f327c00c4d0 tx=0x7f327c00c7e0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.262+0000 7f32925e3700 1 -- 172.21.15.189:0/3165964944 shutdown_connections 2024-04-06T12:31:05.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.262+0000 7f32925e3700 1 --2- 172.21.15.189:0/3165964944 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f328c160b60 0x7f328c15f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.262+0000 7f32925e3700 1 --2- 172.21.15.189:0/3165964944 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f327806aed0 0x7f327806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.262+0000 7f32925e3700 1 --2- 172.21.15.189:0/3165964944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f328c143470 0x7f328c15ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.262+0000 7f32925e3700 1 --2- 172.21.15.189:0/3165964944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f328c15db10 0x7f328c0c7410 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.262+0000 7f32925e3700 1 -- 172.21.15.189:0/3165964944 >> 172.21.15.189:0/3165964944 conn(0x7f328c06a730 msgr2=0x7f328c0b2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:05.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.262+0000 7f32925e3700 1 -- 172.21.15.189:0/3165964944 shutdown_connections 2024-04-06T12:31:05.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.262+0000 7f32925e3700 1 -- 172.21.15.189:0/3165964944 wait complete. 2024-04-06T12:31:05.260 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 54 2024-04-06T12:31:05.272 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:31:05.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.403+0000 7f9132c76700 1 Processor -- start 2024-04-06T12:31:05.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.403+0000 7f9132c76700 1 -- start start 2024-04-06T12:31:05.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.404+0000 7f9132c76700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f912c0d11b0 0x7f912c0d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:05.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.404+0000 7f9132c76700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f912c0cfa00 0x7f912c0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:05.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.404+0000 7f9132c76700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912c0d1ad0 con 0x7f912c0cb870 2024-04-06T12:31:05.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.404+0000 7f9132c76700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f912c0ccdc0 con 0x7f912c0cfa00 2024-04-06T12:31:05.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.404+0000 7f9132c76700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f912c0d0b50 con 0x7f912c0d11b0 2024-04-06T12:31:05.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.405+0000 7f9130a12700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f912c0d11b0 0x7f912c0d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:05.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.405+0000 7f9131213700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f912c0cfa00 0x7f912c0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:05.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.405+0000 7f912bfff700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f912c0cb870 0x7f912c15c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:46968/0 (socket says 172.21.15.189:46968) 2024-04-06T12:31:05.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.405+0000 7f912bfff700 1 -- 172.21.15.189:0/1856698077 learned_addr learned my addr 172.21.15.189:0/1856698077 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:05.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.405+0000 7f9130a12700 1 -- 172.21.15.189:0/1856698077 >> v2:172.21.15.192:3300/0 conn(0x7f912c0cfa00 msgr2=0x7f912c0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:05.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.405+0000 7f9130a12700 1 --2- 172.21.15.189:0/1856698077 >> v2:172.21.15.192:3300/0 conn(0x7f912c0cfa00 0x7f912c0cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.405+0000 7f9130a12700 1 -- 172.21.15.189:0/1856698077 >> v1:172.21.15.192:6789/0 conn(0x7f912c0cb870 legacy=0x7f912c15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:05.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.405+0000 7f9130a12700 1 -- 172.21.15.189:0/1856698077 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912c0cbd80 con 0x7f912c0d11b0 2024-04-06T12:31:05.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.405+0000 7f912b7fe700 1 -- 172.21.15.189:0/1856698077 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441035378 0 0) 0x7f912c0d1ad0 con 0x7f912c0cb870 2024-04-06T12:31:05.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.405+0000 7f9131213700 1 --2- 172.21.15.189:0/1856698077 >> v2:172.21.15.192:3300/0 conn(0x7f912c0cfa00 0x7f912c0cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:05.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.405+0000 7f9130a12700 1 --2- 172.21.15.189:0/1856698077 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f912c0d11b0 0x7f912c0d1590 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9120009840 tx=0x7f9120009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=801f2d7dfd4cadbd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:05.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.405+0000 7f912b7fe700 1 -- 172.21.15.189:0/1856698077 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120011830 con 0x7f912c0d11b0 2024-04-06T12:31:05.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.405+0000 7f912b7fe700 1 -- 172.21.15.189:0/1856698077 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9120011990 con 0x7f912c0d11b0 2024-04-06T12:31:05.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.405+0000 7f912b7fe700 1 -- 172.21.15.189:0/1856698077 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120011d20 con 0x7f912c0d11b0 2024-04-06T12:31:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.406+0000 7f9132c76700 1 -- 172.21.15.189:0/1856698077 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f912c0d11b0 msgr2=0x7f912c0d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.406+0000 7f9132c76700 1 --2- 172.21.15.189:0/1856698077 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f912c0d11b0 0x7f912c0d1590 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9120009840 tx=0x7f9120009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.406+0000 7f9132c76700 1 -- 172.21.15.189:0/1856698077 shutdown_connections 2024-04-06T12:31:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.406+0000 7f9132c76700 1 --2- 172.21.15.189:0/1856698077 >> v2:172.21.15.192:3300/0 conn(0x7f912c0cfa00 0x7f912c0cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.406+0000 7f9132c76700 1 --2- 172.21.15.189:0/1856698077 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f912c0d11b0 0x7f912c0d1590 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.406+0000 7f9132c76700 1 -- 172.21.15.189:0/1856698077 >> 172.21.15.189:0/1856698077 conn(0x7f912c0b2db0 msgr2=0x7f912c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.406+0000 7f9132c76700 1 -- 172.21.15.189:0/1856698077 shutdown_connections 2024-04-06T12:31:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.406+0000 7f9132c76700 1 -- 172.21.15.189:0/1856698077 wait complete. 2024-04-06T12:31:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.408+0000 7f9132c76700 1 Processor -- start 2024-04-06T12:31:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.408+0000 7f9132c76700 1 -- start start 2024-04-06T12:31:05.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.408+0000 7f9132c76700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f912c0cb870 0x7f912c1579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:05.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.408+0000 7f9132c76700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f912c0cfa00 0x7f912c1f3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:05.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.408+0000 7f9132c76700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f912c0d11b0 0x7f912c15e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:05.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.408+0000 7f9132c76700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912c1f30b0 con 0x7f912c0d11b0 2024-04-06T12:31:05.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.408+0000 7f9132c76700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912c1581f0 con 0x7f912c0cb870 2024-04-06T12:31:05.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.408+0000 7f9132c76700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f912c1584d0 con 0x7f912c0cfa00 2024-04-06T12:31:05.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.409+0000 7f9131213700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f912c0d11b0 0x7f912c15e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:05.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.409+0000 7f9130a12700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f912c0cb870 0x7f912c1579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:05.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.409+0000 7f912bfff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f912c0cfa00 0x7f912c1f3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:05.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.409+0000 7f9131213700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f912c0d11b0 0x7f912c15e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36756/0 (socket says 172.21.15.189:36756) 2024-04-06T12:31:05.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.409+0000 7f9130a12700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f912c0cb870 0x7f912c1579c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:51644/0 (socket says 172.21.15.189:51644) 2024-04-06T12:31:05.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.409+0000 7f9131213700 1 -- 172.21.15.189:0/861670003 learned_addr learned my addr 172.21.15.189:0/861670003 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:05.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.409+0000 7f9131213700 1 -- 172.21.15.189:0/861670003 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f912c0cfa00 msgr2=0x7f912c1f3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:05.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.409+0000 7f9131213700 1 --2- 172.21.15.189:0/861670003 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f912c0cfa00 0x7f912c1f3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.409+0000 7f9131213700 1 -- 172.21.15.189:0/861670003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f912c0cb870 msgr2=0x7f912c1579c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:05.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.409+0000 7f9131213700 1 --2- 172.21.15.189:0/861670003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f912c0cb870 0x7f912c1579c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.409+0000 7f9131213700 1 -- 172.21.15.189:0/861670003 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9124009a70 con 0x7f912c0d11b0 2024-04-06T12:31:05.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.409+0000 7f9130a12700 1 --2- 172.21.15.189:0/861670003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f912c0cb870 0x7f912c1579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:05.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.409+0000 7f9131213700 1 --2- 172.21.15.189:0/861670003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f912c0d11b0 0x7f912c15e6c0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f912400efc0 tx=0x7f912400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:05.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.409+0000 7f91297fa700 1 -- 172.21.15.189:0/861670003 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912400ca90 con 0x7f912c0d11b0 2024-04-06T12:31:05.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.409+0000 7f91297fa700 1 -- 172.21.15.189:0/861670003 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912400f040 con 0x7f912c0d11b0 2024-04-06T12:31:05.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.409+0000 7f9132c76700 1 -- 172.21.15.189:0/861670003 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91200094a0 con 0x7f912c0d11b0 2024-04-06T12:31:05.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.409+0000 7f91297fa700 1 -- 172.21.15.189:0/861670003 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912400ce10 con 0x7f912c0d11b0 2024-04-06T12:31:05.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.409+0000 7f9132c76700 1 -- 172.21.15.189:0/861670003 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f912c157f00 con 0x7f912c0d11b0 2024-04-06T12:31:05.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.410+0000 7f9132c76700 1 -- 172.21.15.189:0/861670003 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f912c074990 con 0x7f912c0d11b0 2024-04-06T12:31:05.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.411+0000 7f91297fa700 1 -- 172.21.15.189:0/861670003 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 71) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f9124014480 con 0x7f912c0d11b0 2024-04-06T12:31:05.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.411+0000 7f91297fa700 1 --2- 172.21.15.189:0/861670003 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f911006aed0 0x7f911006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:05.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.411+0000 7f9130a12700 1 -- 172.21.15.189:0/861670003 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f911006aed0 msgr2=0x7f911006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:05.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.411+0000 7f91297fa700 1 -- 172.21.15.189:0/861670003 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f912408a0b0 con 0x7f912c0d11b0 2024-04-06T12:31:05.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.411+0000 7f9130a12700 1 --2- 172.21.15.189:0/861670003 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f911006aed0 0x7f911006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:05.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.416+0000 7f91297fa700 1 -- 172.21.15.189:0/861670003 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9124054c10 con 0x7f912c0d11b0 2024-04-06T12:31:05.602 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:05.605+0000 7f3500729200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:31:05.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.612+0000 7f9130a12700 1 -- 172.21.15.189:0/861670003 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f911006aed0 msgr2=0x7f911006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:05.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.612+0000 7f9130a12700 1 --2- 172.21.15.189:0/861670003 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f911006aed0 0x7f911006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:05.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.620+0000 7f9132c76700 1 -- 172.21.15.189:0/861670003 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f912c0b0a00 con 0x7f912c0d11b0 2024-04-06T12:31:05.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.621+0000 7f91297fa700 1 -- 172.21.15.189:0/861670003 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16792 (secure 0 0 0) 0x7f9124081e60 con 0x7f912c0d11b0 2024-04-06T12:31:05.619 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:05.619 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":129,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:30:31.084655+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:30:14.958335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-06T12:30:16.104585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/4170508796":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/169651662":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/29183104":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/3994101871":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/1300227922":"2024-04-07T12:30:31.084606+0000","172.21.15.189:6833/47128":"2024-04-07T12:30:31.084606+0000","172.21.15.189:6832/47128":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/1086259216":"2024-04-07T12:30:31.084606+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:31:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.624+0000 7f9132c76700 1 -- 172.21.15.189:0/861670003 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f911006aed0 msgr2=0x7f911006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.624+0000 7f9132c76700 1 --2- 172.21.15.189:0/861670003 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f911006aed0 0x7f911006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.624+0000 7f9132c76700 1 -- 172.21.15.189:0/861670003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f912c0d11b0 msgr2=0x7f912c15e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.624+0000 7f9132c76700 1 --2- 172.21.15.189:0/861670003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f912c0d11b0 0x7f912c15e6c0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f912400efc0 tx=0x7f912400c3a0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.624+0000 7f9132c76700 1 -- 172.21.15.189:0/861670003 shutdown_connections 2024-04-06T12:31:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.624+0000 7f9132c76700 1 --2- 172.21.15.189:0/861670003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f912c0d11b0 0x7f912c15e6c0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.624+0000 7f9132c76700 1 --2- 172.21.15.189:0/861670003 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f911006aed0 0x7f911006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.624+0000 7f9132c76700 1 --2- 172.21.15.189:0/861670003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f912c0cb870 0x7f912c1579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.624+0000 7f9132c76700 1 --2- 172.21.15.189:0/861670003 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f912c0cfa00 0x7f912c1f3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.624+0000 7f9132c76700 1 -- 172.21.15.189:0/861670003 >> 172.21.15.189:0/861670003 conn(0x7f912c0b2db0 msgr2=0x7f912c0cddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.624+0000 7f9132c76700 1 -- 172.21.15.189:0/861670003 shutdown_connections 2024-04-06T12:31:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.624+0000 7f9132c76700 1 -- 172.21.15.189:0/861670003 wait complete. 2024-04-06T12:31:05.635 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:30:14.958335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:30:16.104585+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:31:05.635 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:31:05.681 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:05.683+0000 7f3500729200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:31:05.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.768+0000 7f035bba4700 1 Processor -- start 2024-04-06T12:31:05.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.768+0000 7f035bba4700 1 -- start start 2024-04-06T12:31:05.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.769+0000 7f035bba4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f03540b4af0 0x7f03540b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:05.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.769+0000 7f035bba4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03540cdbe0 con 0x7f035415a550 2024-04-06T12:31:05.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.769+0000 7f035bba4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f035415a940 con 0x7f03540b4660 2024-04-06T12:31:05.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.769+0000 7f035bba4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f035415ac00 con 0x7f03540b4af0 2024-04-06T12:31:05.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.769+0000 7f0359940700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f035415a550 0x7f03540cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45332/0 (socket says 172.21.15.189:45332) 2024-04-06T12:31:05.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.769+0000 7f0359940700 1 -- 172.21.15.189:0/3098402855 learned_addr learned my addr 172.21.15.189:0/3098402855 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:05.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.770+0000 7f035a141700 1 --2- 172.21.15.189:0/3098402855 >> v2:172.21.15.192:3300/0 conn(0x7f03540b4af0 0x7f03540b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:05.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.770+0000 7f035893e700 1 -- 172.21.15.189:0/3098402855 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170576379 0 0) 0x7f03540cdbe0 con 0x7f035415a550 2024-04-06T12:31:05.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.770+0000 7f035893e700 1 -- 172.21.15.189:0/3098402855 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03400035b0 con 0x7f035415a550 2024-04-06T12:31:05.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.770+0000 7f035893e700 1 -- 172.21.15.189:0/3098402855 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236581708 0 0) 0x7f035415a940 con 0x7f03540b4660 2024-04-06T12:31:05.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.770+0000 7f035893e700 1 -- 172.21.15.189:0/3098402855 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03540cdbe0 con 0x7f03540b4660 2024-04-06T12:31:05.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.770+0000 7f035a141700 1 -- 172.21.15.189:0/3098402855 >> v1:172.21.15.192:6789/0 conn(0x7f03540b4660 legacy=0x7f03540b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:05.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.770+0000 7f035a141700 1 -- 172.21.15.189:0/3098402855 >> v1:172.21.15.189:6789/0 conn(0x7f035415a550 legacy=0x7f03540cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:05.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.770+0000 7f035a141700 1 -- 172.21.15.189:0/3098402855 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f035415aee0 con 0x7f03540b4af0 2024-04-06T12:31:05.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.770+0000 7f035a141700 1 --2- 172.21.15.189:0/3098402855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03540b4af0 0x7f03540b36d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f034c009d90 tx=0x7f034c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3de7b63ff73e314c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:05.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.770+0000 7f035893e700 1 -- 172.21.15.189:0/3098402855 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034c00e070 con 0x7f03540b4af0 2024-04-06T12:31:05.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.770+0000 7f035893e700 1 -- 172.21.15.189:0/3098402855 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034c00bb20 con 0x7f03540b4af0 2024-04-06T12:31:05.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.770+0000 7f035893e700 1 -- 172.21.15.189:0/3098402855 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034c012430 con 0x7f03540b4af0 2024-04-06T12:31:05.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.771+0000 7f035bba4700 1 -- 172.21.15.189:0/3098402855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03540b4af0 msgr2=0x7f03540b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:05.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.771+0000 7f035bba4700 1 --2- 172.21.15.189:0/3098402855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03540b4af0 0x7f03540b36d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f034c009d90 tx=0x7f034c00b330 comp rx=0 tx=0).stop 2024-04-06T12:31:05.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.771+0000 7f035bba4700 1 -- 172.21.15.189:0/3098402855 shutdown_connections 2024-04-06T12:31:05.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.771+0000 7f035bba4700 1 --2- 172.21.15.189:0/3098402855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03540b4af0 0x7f03540b36d0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.771+0000 7f035bba4700 1 -- 172.21.15.189:0/3098402855 >> 172.21.15.189:0/3098402855 conn(0x7f03540d00b0 msgr2=0x7f03540d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:05.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.771+0000 7f035bba4700 1 -- 172.21.15.189:0/3098402855 shutdown_connections 2024-04-06T12:31:05.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.771+0000 7f035bba4700 1 -- 172.21.15.189:0/3098402855 wait complete. 2024-04-06T12:31:05.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.773+0000 7f035bba4700 1 Processor -- start 2024-04-06T12:31:05.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.773+0000 7f035bba4700 1 -- start start 2024-04-06T12:31:05.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.774+0000 7f035bba4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03540b4660 0x7f03540ad220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:05.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.774+0000 7f035bba4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03540b4af0 0x7f03541f3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:05.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.774+0000 7f035bba4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0354149a40 0x7f03540cde10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:05.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.774+0000 7f035bba4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03541f3d90 con 0x7f03540b4660 2024-04-06T12:31:05.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.774+0000 7f035bba4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035415b8c0 con 0x7f0354149a40 2024-04-06T12:31:05.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.774+0000 7f035bba4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03540ad8c0 con 0x7f03540b4af0 2024-04-06T12:31:05.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.774+0000 7f0359940700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03540b4660 0x7f03540ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:05.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.774+0000 7f0359940700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03540b4660 0x7f03540ad220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36768/0 (socket says 172.21.15.189:36768) 2024-04-06T12:31:05.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.774+0000 7f0359940700 1 -- 172.21.15.189:0/543417657 learned_addr learned my addr 172.21.15.189:0/543417657 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:05.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.774+0000 7f035a141700 1 --2- 172.21.15.189:0/543417657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0354149a40 0x7f03540cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:05.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.774+0000 7f035913f700 1 --2- 172.21.15.189:0/543417657 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03540b4af0 0x7f03541f3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:05.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.775+0000 7f0359940700 1 -- 172.21.15.189:0/543417657 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03540b4af0 msgr2=0x7f03541f3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:05.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.775+0000 7f0359940700 1 --2- 172.21.15.189:0/543417657 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03540b4af0 0x7f03541f3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.775+0000 7f0359940700 1 -- 172.21.15.189:0/543417657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0354149a40 msgr2=0x7f03540cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:05.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.775+0000 7f0359940700 1 --2- 172.21.15.189:0/543417657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0354149a40 0x7f03540cde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:05.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.775+0000 7f0359940700 1 -- 172.21.15.189:0/543417657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034c009a70 con 0x7f03540b4660 2024-04-06T12:31:05.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.775+0000 7f0359940700 1 --2- 172.21.15.189:0/543417657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03540b4660 0x7f03540ad220 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f0350001eb0 tx=0x7f035000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:05.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.775+0000 7f034a7fc700 1 -- 172.21.15.189:0/543417657 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0350005d70 con 0x7f03540b4660 2024-04-06T12:31:05.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.775+0000 7f034a7fc700 1 -- 172.21.15.189:0/543417657 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03500070d0 con 0x7f03540b4660 2024-04-06T12:31:05.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.775+0000 7f035bba4700 1 -- 172.21.15.189:0/543417657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f035415bb10 con 0x7f03540b4660 2024-04-06T12:31:05.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.775+0000 7f034a7fc700 1 -- 172.21.15.189:0/543417657 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0350007460 con 0x7f03540b4660 2024-04-06T12:31:05.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.775+0000 7f035bba4700 1 -- 172.21.15.189:0/543417657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f035415c0d0 con 0x7f03540b4660 2024-04-06T12:31:05.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.776+0000 7f035bba4700 1 -- 172.21.15.189:0/543417657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0354067bb0 con 0x7f03540b4660 2024-04-06T12:31:05.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.776+0000 7f034a7fc700 1 -- 172.21.15.189:0/543417657 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 71) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f03500075c0 con 0x7f03540b4660 2024-04-06T12:31:05.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.777+0000 7f034a7fc700 1 --2- 172.21.15.189:0/543417657 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f033806aed0 0x7f033806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:05.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.777+0000 7f035913f700 1 -- 172.21.15.189:0/543417657 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f033806aed0 msgr2=0x7f033806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:05.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.777+0000 7f034a7fc700 1 -- 172.21.15.189:0/543417657 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f0350088de0 con 0x7f03540b4660 2024-04-06T12:31:05.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.777+0000 7f035913f700 1 --2- 172.21.15.189:0/543417657 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f033806aed0 0x7f033806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:05.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.781+0000 7f034a7fc700 1 -- 172.21.15.189:0/543417657 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f03500539c0 con 0x7f03540b4660 2024-04-06T12:31:05.781 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:05.784+0000 7f6920c15200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:31:05.807 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:05.810+0000 7f3500729200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:31:05.894 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:05.897+0000 7f6920c15200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:31:05.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.978+0000 7f035913f700 1 -- 172.21.15.189:0/543417657 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f033806aed0 msgr2=0x7f033806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:05.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:05.978+0000 7f035913f700 1 --2- 172.21.15.189:0/543417657 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f033806aed0 0x7f033806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:06.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.025+0000 7f035bba4700 1 -- 172.21.15.189:0/543417657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0354074990 con 0x7f03540b4660 2024-04-06T12:31:06.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.026+0000 7f034a7fc700 1 -- 172.21.15.189:0/543417657 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) v1 ==== 93+0+10 (secure 0 0 0) 0x7f03500170a0 con 0x7f03540b4660 2024-04-06T12:31:06.024 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:31:06.024 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:06.027+0000 7f6920c15200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:31:06.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.028+0000 7f035bba4700 1 -- 172.21.15.189:0/543417657 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f033806aed0 msgr2=0x7f033806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:06.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.028+0000 7f035bba4700 1 --2- 172.21.15.189:0/543417657 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f033806aed0 0x7f033806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.028+0000 7f035bba4700 1 -- 172.21.15.189:0/543417657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03540b4660 msgr2=0x7f03540ad220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:06.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.028+0000 7f035bba4700 1 --2- 172.21.15.189:0/543417657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03540b4660 0x7f03540ad220 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f0350001eb0 tx=0x7f035000c7b0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.028+0000 7f035bba4700 1 -- 172.21.15.189:0/543417657 shutdown_connections 2024-04-06T12:31:06.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.028+0000 7f035bba4700 1 --2- 172.21.15.189:0/543417657 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f033806aed0 0x7f033806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.028+0000 7f035bba4700 1 --2- 172.21.15.189:0/543417657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0354149a40 0x7f03540cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.028+0000 7f035bba4700 1 --2- 172.21.15.189:0/543417657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03540b4660 0x7f03540ad220 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.028+0000 7f035bba4700 1 --2- 172.21.15.189:0/543417657 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03540b4af0 0x7f03541f3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.028+0000 7f035bba4700 1 -- 172.21.15.189:0/543417657 >> 172.21.15.189:0/543417657 conn(0x7f03540d00b0 msgr2=0x7f0354154dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:06.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.028+0000 7f035bba4700 1 -- 172.21.15.189:0/543417657 shutdown_connections 2024-04-06T12:31:06.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.028+0000 7f035bba4700 1 -- 172.21.15.189:0/543417657 wait complete. 2024-04-06T12:31:06.040 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:31:06.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.174+0000 7f5f5c09e700 1 Processor -- start 2024-04-06T12:31:06.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.175+0000 7f5f5c09e700 1 -- start start 2024-04-06T12:31:06.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.175+0000 7f5f5c09e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f540d11b0 0x7f5f540d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:06.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.176+0000 7f5f5c09e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f540d1ad0 con 0x7f5f540cb870 2024-04-06T12:31:06.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.176+0000 7f5f5c09e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f540cbd80 con 0x7f5f540cfa00 2024-04-06T12:31:06.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.176+0000 7f5f5c09e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f540ccdc0 con 0x7f5f540d11b0 2024-04-06T12:31:06.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.176+0000 7f5f59639700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5f540cb870 0x7f5f5415c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45334/0 (socket says 172.21.15.189:45334) 2024-04-06T12:31:06.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.176+0000 7f5f59639700 1 -- 172.21.15.189:0/2153985101 learned_addr learned my addr 172.21.15.189:0/2153985101 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:06.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.176+0000 7f5f59e3a700 1 --2- 172.21.15.189:0/2153985101 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f540d11b0 0x7f5f540d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:06.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.176+0000 7f5f58e38700 1 -- 172.21.15.189:0/2153985101 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280841475 0 0) 0x7f5f540d1ad0 con 0x7f5f540cb870 2024-04-06T12:31:06.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.176+0000 7f5f58e38700 1 -- 172.21.15.189:0/2153985101 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f400035b0 con 0x7f5f540cb870 2024-04-06T12:31:06.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.176+0000 7f5f58e38700 1 -- 172.21.15.189:0/2153985101 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502434447 0 0) 0x7f5f540cbd80 con 0x7f5f540cfa00 2024-04-06T12:31:06.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.176+0000 7f5f58e38700 1 -- 172.21.15.189:0/2153985101 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f540d1ad0 con 0x7f5f540cfa00 2024-04-06T12:31:06.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.176+0000 7f5f59e3a700 1 -- 172.21.15.189:0/2153985101 >> v1:172.21.15.192:6789/0 conn(0x7f5f540cfa00 legacy=0x7f5f540cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:06.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.176+0000 7f5f59e3a700 1 -- 172.21.15.189:0/2153985101 >> v1:172.21.15.189:6789/0 conn(0x7f5f540cb870 legacy=0x7f5f5415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:06.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.176+0000 7f5f59e3a700 1 -- 172.21.15.189:0/2153985101 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f540cef20 con 0x7f5f540d11b0 2024-04-06T12:31:06.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.176+0000 7f5f59e3a700 1 --2- 172.21.15.189:0/2153985101 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f540d11b0 0x7f5f540d1590 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5f50009870 tx=0x7f5f50009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5e53feae3fc82b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:06.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.177+0000 7f5f58e38700 1 -- 172.21.15.189:0/2153985101 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f50011830 con 0x7f5f540d11b0 2024-04-06T12:31:06.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.177+0000 7f5f58e38700 1 -- 172.21.15.189:0/2153985101 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f50011990 con 0x7f5f540d11b0 2024-04-06T12:31:06.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.177+0000 7f5f58e38700 1 -- 172.21.15.189:0/2153985101 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f50011d20 con 0x7f5f540d11b0 2024-04-06T12:31:06.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.177+0000 7f5f5c09e700 1 -- 172.21.15.189:0/2153985101 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f540d11b0 msgr2=0x7f5f540d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:06.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.177+0000 7f5f5c09e700 1 --2- 172.21.15.189:0/2153985101 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f540d11b0 0x7f5f540d1590 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5f50009870 tx=0x7f5f50009b80 comp rx=0 tx=0).stop 2024-04-06T12:31:06.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.177+0000 7f5f5c09e700 1 -- 172.21.15.189:0/2153985101 shutdown_connections 2024-04-06T12:31:06.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.177+0000 7f5f5c09e700 1 --2- 172.21.15.189:0/2153985101 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f540d11b0 0x7f5f540d1590 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.177+0000 7f5f5c09e700 1 -- 172.21.15.189:0/2153985101 >> 172.21.15.189:0/2153985101 conn(0x7f5f540b2db0 msgr2=0x7f5f540b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:06.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.177+0000 7f5f5c09e700 1 -- 172.21.15.189:0/2153985101 shutdown_connections 2024-04-06T12:31:06.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.177+0000 7f5f5c09e700 1 -- 172.21.15.189:0/2153985101 wait complete. 2024-04-06T12:31:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.179+0000 7f5f5c09e700 1 Processor -- start 2024-04-06T12:31:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.179+0000 7f5f5c09e700 1 -- start start 2024-04-06T12:31:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.179+0000 7f5f5c09e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f540cb870 0x7f5f5415e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.179+0000 7f5f5c09e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f540cfa00 0x7f5f540c66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.179+0000 7f5f5c09e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f540d11b0 0x7f5f5406b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.179+0000 7f5f5c09e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f540c7c70 con 0x7f5f540cfa00 2024-04-06T12:31:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.179+0000 7f5f5c09e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f540c7060 con 0x7f5f540cb870 2024-04-06T12:31:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.179+0000 7f5f5c09e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f540c7340 con 0x7f5f540d11b0 2024-04-06T12:31:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.179+0000 7f5f59639700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f540cfa00 0x7f5f540c66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:06.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.180+0000 7f5f59e3a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f540cb870 0x7f5f5415e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.180+0000 7f5f59639700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f540cfa00 0x7f5f540c66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36782/0 (socket says 172.21.15.189:36782) 2024-04-06T12:31:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.180+0000 7f5f59639700 1 -- 172.21.15.189:0/2051209676 learned_addr learned my addr 172.21.15.189:0/2051209676 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.180+0000 7f5f5a63b700 1 --2- 172.21.15.189:0/2051209676 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f540d11b0 0x7f5f5406b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.180+0000 7f5f59639700 1 -- 172.21.15.189:0/2051209676 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f540d11b0 msgr2=0x7f5f5406b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.180+0000 7f5f59639700 1 --2- 172.21.15.189:0/2051209676 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f540d11b0 0x7f5f5406b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.180+0000 7f5f59639700 1 -- 172.21.15.189:0/2051209676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f540cb870 msgr2=0x7f5f5415e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.180+0000 7f5f59639700 1 --2- 172.21.15.189:0/2051209676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f540cb870 0x7f5f5415e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.180+0000 7f5f59639700 1 -- 172.21.15.189:0/2051209676 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f500094a0 con 0x7f5f540cfa00 2024-04-06T12:31:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.180+0000 7f5f5a63b700 1 --2- 172.21.15.189:0/2051209676 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f540d11b0 0x7f5f5406b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.180+0000 7f5f59e3a700 1 --2- 172.21.15.189:0/2051209676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f540cb870 0x7f5f5415e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:06.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.180+0000 7f5f59639700 1 --2- 172.21.15.189:0/2051209676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f540cfa00 0x7f5f540c66c0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f5f44005910 tx=0x7f5f4400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:06.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.180+0000 7f5f4a7fc700 1 -- 172.21.15.189:0/2051209676 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f44005d70 con 0x7f5f540cfa00 2024-04-06T12:31:06.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.180+0000 7f5f4a7fc700 1 -- 172.21.15.189:0/2051209676 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f440070d0 con 0x7f5f540cfa00 2024-04-06T12:31:06.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.180+0000 7f5f5c09e700 1 -- 172.21.15.189:0/2051209676 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f540c7620 con 0x7f5f540cfa00 2024-04-06T12:31:06.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.180+0000 7f5f4a7fc700 1 -- 172.21.15.189:0/2051209676 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f44007440 con 0x7f5f540cfa00 2024-04-06T12:31:06.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.180+0000 7f5f5c09e700 1 -- 172.21.15.189:0/2051209676 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5415ec00 con 0x7f5f540cfa00 2024-04-06T12:31:06.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.181+0000 7f5f5c09e700 1 -- 172.21.15.189:0/2051209676 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f5415ed90 con 0x7f5f540cfa00 2024-04-06T12:31:06.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.182+0000 7f5f4a7fc700 1 -- 172.21.15.189:0/2051209676 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 71) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f5f440075a0 con 0x7f5f540cfa00 2024-04-06T12:31:06.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.182+0000 7f5f4a7fc700 1 --2- 172.21.15.189:0/2051209676 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5f3806aed0 0x7f5f3806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:06.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.182+0000 7f5f59e3a700 1 -- 172.21.15.189:0/2051209676 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5f3806aed0 msgr2=0x7f5f3806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:06.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.183+0000 7f5f4a7fc700 1 -- 172.21.15.189:0/2051209676 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f5f44088e30 con 0x7f5f540cfa00 2024-04-06T12:31:06.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.183+0000 7f5f59e3a700 1 --2- 172.21.15.189:0/2051209676 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5f3806aed0 0x7f5f3806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:06.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.187+0000 7f5f4a7fc700 1 -- 172.21.15.189:0/2051209676 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5f440539a0 con 0x7f5f540cfa00 2024-04-06T12:31:06.315 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:06.317+0000 7f3500729200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:31:06.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.384+0000 7f5f59e3a700 1 -- 172.21.15.189:0/2051209676 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5f3806aed0 msgr2=0x7f5f3806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:06.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.384+0000 7f5f59e3a700 1 --2- 172.21.15.189:0/2051209676 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5f3806aed0 0x7f5f3806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:06.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.390+0000 7f5f5c09e700 1 -- 172.21.15.189:0/2051209676 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5f54074990 con 0x7f5f540cfa00 2024-04-06T12:31:06.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.390+0000 7f5f4a7fc700 1 -- 172.21.15.189:0/2051209676 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v129) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5f44056fc0 con 0x7f5f540cfa00 2024-04-06T12:31:06.387 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:31:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.392+0000 7f5f5c09e700 1 -- 172.21.15.189:0/2051209676 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5f3806aed0 msgr2=0x7f5f3806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.392+0000 7f5f5c09e700 1 --2- 172.21.15.189:0/2051209676 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5f3806aed0 0x7f5f3806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.392+0000 7f5f5c09e700 1 -- 172.21.15.189:0/2051209676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f540cfa00 msgr2=0x7f5f540c66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.392+0000 7f5f5c09e700 1 --2- 172.21.15.189:0/2051209676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f540cfa00 0x7f5f540c66c0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f5f44005910 tx=0x7f5f4400c9e0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.392+0000 7f5f5c09e700 1 -- 172.21.15.189:0/2051209676 shutdown_connections 2024-04-06T12:31:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.392+0000 7f5f5c09e700 1 --2- 172.21.15.189:0/2051209676 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f540d11b0 0x7f5f5406b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.392+0000 7f5f5c09e700 1 --2- 172.21.15.189:0/2051209676 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5f3806aed0 0x7f5f3806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.392+0000 7f5f5c09e700 1 --2- 172.21.15.189:0/2051209676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f540cb870 0x7f5f5415e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.392+0000 7f5f5c09e700 1 --2- 172.21.15.189:0/2051209676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f540cfa00 0x7f5f540c66c0 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.392+0000 7f5f5c09e700 1 -- 172.21.15.189:0/2051209676 >> 172.21.15.189:0/2051209676 conn(0x7f5f540b2db0 msgr2=0x7f5f5415a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:06.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.392+0000 7f5f5c09e700 1 -- 172.21.15.189:0/2051209676 shutdown_connections 2024-04-06T12:31:06.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.392+0000 7f5f5c09e700 1 -- 172.21.15.189:0/2051209676 wait complete. 2024-04-06T12:31:06.402 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:31:06.468 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:06.470+0000 7f3500729200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:31:06.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.534+0000 7ff781998700 1 Processor -- start 2024-04-06T12:31:06.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.535+0000 7ff781998700 1 -- start start 2024-04-06T12:31:06.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.535+0000 7ff781998700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff77c15d7c0 0x7ff77c06b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:06.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.535+0000 7ff781998700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff77c06b6f0 0x7ff77c06dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:06.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.535+0000 7ff781998700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff77c06ceb0 con 0x7ff77c0cd3e0 2024-04-06T12:31:06.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.535+0000 7ff781998700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff77c0cdfa0 con 0x7ff77c06b6f0 2024-04-06T12:31:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.535+0000 7ff781998700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c06d750 con 0x7ff77c15d7c0 2024-04-06T12:31:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.536+0000 7ff77b7fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff77c06b6f0 0x7ff77c06dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.536+0000 7ff77affd700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7ff77c0cd3e0 0x7ff77c0cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:46992/0 (socket says 172.21.15.189:46992) 2024-04-06T12:31:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.536+0000 7ff77affd700 1 -- 172.21.15.189:0/11098420 learned_addr learned my addr 172.21.15.189:0/11098420 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.536+0000 7ff77a7fc700 1 --2- 172.21.15.189:0/11098420 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff77c15d7c0 0x7ff77c06b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.536+0000 7ff77b7fe700 1 -- 172.21.15.189:0/11098420 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff77c15d7c0 msgr2=0x7ff77c06b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.536+0000 7ff77b7fe700 1 --2- 172.21.15.189:0/11098420 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff77c15d7c0 0x7ff77c06b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.536+0000 7ff77b7fe700 1 -- 172.21.15.189:0/11098420 >> v1:172.21.15.192:6789/0 conn(0x7ff77c0cd3e0 legacy=0x7ff77c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.536+0000 7ff779ffb700 1 -- 172.21.15.189:0/11098420 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777835379 0 0) 0x7ff77c06ceb0 con 0x7ff77c0cd3e0 2024-04-06T12:31:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.536+0000 7ff77b7fe700 1 -- 172.21.15.189:0/11098420 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c06bf00 con 0x7ff77c06b6f0 2024-04-06T12:31:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.536+0000 7ff77a7fc700 1 --2- 172.21.15.189:0/11098420 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff77c15d7c0 0x7ff77c06b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.536+0000 7ff77b7fe700 1 --2- 172.21.15.189:0/11098420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff77c06b6f0 0x7ff77c06dbe0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff774009d90 tx=0x7ff77400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6164b5d2b05bb8f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.536+0000 7ff779ffb700 1 -- 172.21.15.189:0/11098420 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77400e070 con 0x7ff77c06b6f0 2024-04-06T12:31:06.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.536+0000 7ff779ffb700 1 -- 172.21.15.189:0/11098420 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff77400bb20 con 0x7ff77c06b6f0 2024-04-06T12:31:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.536+0000 7ff779ffb700 1 -- 172.21.15.189:0/11098420 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff774012430 con 0x7ff77c06b6f0 2024-04-06T12:31:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.537+0000 7ff781998700 1 -- 172.21.15.189:0/11098420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff77c06b6f0 msgr2=0x7ff77c06dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.537+0000 7ff781998700 1 --2- 172.21.15.189:0/11098420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff77c06b6f0 0x7ff77c06dbe0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff774009d90 tx=0x7ff77400b330 comp rx=0 tx=0).stop 2024-04-06T12:31:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.537+0000 7ff781998700 1 -- 172.21.15.189:0/11098420 shutdown_connections 2024-04-06T12:31:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.537+0000 7ff781998700 1 --2- 172.21.15.189:0/11098420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff77c06b6f0 0x7ff77c06dbe0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.537+0000 7ff781998700 1 --2- 172.21.15.189:0/11098420 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff77c15d7c0 0x7ff77c06b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.537+0000 7ff781998700 1 -- 172.21.15.189:0/11098420 >> 172.21.15.189:0/11098420 conn(0x7ff77c0cc290 msgr2=0x7ff77c0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.537+0000 7ff781998700 1 -- 172.21.15.189:0/11098420 shutdown_connections 2024-04-06T12:31:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.537+0000 7ff781998700 1 -- 172.21.15.189:0/11098420 wait complete. 2024-04-06T12:31:06.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.538+0000 7ff781998700 1 Processor -- start 2024-04-06T12:31:06.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.539+0000 7ff781998700 1 -- start start 2024-04-06T12:31:06.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff781998700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff77c06b6f0 0x7ff77c0b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:06.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff781998700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff77c0cd3e0 0x7ff77c15c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:06.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff781998700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff77c15d7c0 0x7ff77c1600d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:06.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff781998700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c0c61f0 con 0x7ff77c15d7c0 2024-04-06T12:31:06.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff781998700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c0b4c60 con 0x7ff77c06b6f0 2024-04-06T12:31:06.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff781998700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c0b43c0 con 0x7ff77c0cd3e0 2024-04-06T12:31:06.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff77affd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff77c06b6f0 0x7ff77c0b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:06.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff77a7fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff77c0cd3e0 0x7ff77c15c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:06.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff77affd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff77c06b6f0 0x7ff77c0b3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:51694/0 (socket says 172.21.15.189:51694) 2024-04-06T12:31:06.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff77affd700 1 -- 172.21.15.189:0/1649556990 learned_addr learned my addr 172.21.15.189:0/1649556990 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:06.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff77b7fe700 1 --2- 172.21.15.189:0/1649556990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff77c15d7c0 0x7ff77c1600d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:06.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff77affd700 1 -- 172.21.15.189:0/1649556990 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff77c0cd3e0 msgr2=0x7ff77c15c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:06.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff77affd700 1 --2- 172.21.15.189:0/1649556990 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff77c0cd3e0 0x7ff77c15c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff77affd700 1 -- 172.21.15.189:0/1649556990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff77c15d7c0 msgr2=0x7ff77c1600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:06.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff77affd700 1 --2- 172.21.15.189:0/1649556990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff77c15d7c0 0x7ff77c1600d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff77affd700 1 -- 172.21.15.189:0/1649556990 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7700094a0 con 0x7ff77c06b6f0 2024-04-06T12:31:06.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff77a7fc700 1 --2- 172.21.15.189:0/1649556990 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff77c0cd3e0 0x7ff77c15c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:06.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff77affd700 1 --2- 172.21.15.189:0/1649556990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff77c06b6f0 0x7ff77c0b3d00 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff76c0011e0 tx=0x7ff76c00ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:06.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff763fff700 1 -- 172.21.15.189:0/1649556990 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76c004c80 con 0x7ff77c06b6f0 2024-04-06T12:31:06.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.540+0000 7ff763fff700 1 -- 172.21.15.189:0/1649556990 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76c004e00 con 0x7ff77c06b6f0 2024-04-06T12:31:06.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.541+0000 7ff781998700 1 -- 172.21.15.189:0/1649556990 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff774009a70 con 0x7ff77c06b6f0 2024-04-06T12:31:06.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.541+0000 7ff763fff700 1 -- 172.21.15.189:0/1649556990 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76c007310 con 0x7ff77c06b6f0 2024-04-06T12:31:06.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.541+0000 7ff781998700 1 -- 172.21.15.189:0/1649556990 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77c0b4980 con 0x7ff77c06b6f0 2024-04-06T12:31:06.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.541+0000 7ff781998700 1 -- 172.21.15.189:0/1649556990 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77c074990 con 0x7ff77c06b6f0 2024-04-06T12:31:06.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.542+0000 7ff763fff700 1 -- 172.21.15.189:0/1649556990 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 71) v1 ==== 89361+0+0 (secure 0 0 0) 0x7ff76c007470 con 0x7ff77c06b6f0 2024-04-06T12:31:06.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.542+0000 7ff763fff700 1 --2- 172.21.15.189:0/1649556990 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff76806af00 0x7ff76806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:06.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.542+0000 7ff77a7fc700 1 -- 172.21.15.189:0/1649556990 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff76806af00 msgr2=0x7ff76806d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:06.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.542+0000 7ff763fff700 1 -- 172.21.15.189:0/1649556990 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6373+0+0 (secure 0 0 0) 0x7ff76c0896b0 con 0x7ff77c06b6f0 2024-04-06T12:31:06.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.542+0000 7ff77a7fc700 1 --2- 172.21.15.189:0/1649556990 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff76806af00 0x7ff76806d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:06.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.547+0000 7ff763fff700 1 -- 172.21.15.189:0/1649556990 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff76c0541f0 con 0x7ff77c06b6f0 2024-04-06T12:31:06.573 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:06.576+0000 7f6920c15200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:31:06.625 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:06.628+0000 7f3500729200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:31:06.734 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:06.737+0000 7f6920c15200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:31:06.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.743+0000 7ff77a7fc700 1 -- 172.21.15.189:0/1649556990 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff76806af00 msgr2=0x7ff76806d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:06.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.743+0000 7ff77a7fc700 1 --2- 172.21.15.189:0/1649556990 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff76806af00 0x7ff76806d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:06.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.750+0000 7ff781998700 1 -- 172.21.15.189:0/1649556990 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff77c0aeb30 con 0x7ff77c06b6f0 2024-04-06T12:31:06.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.750+0000 7ff763fff700 1 -- 172.21.15.189:0/1649556990 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v129) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff76c057810 con 0x7ff77c06b6f0 2024-04-06T12:31:06.747 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:31:06.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.753+0000 7ff781998700 1 -- 172.21.15.189:0/1649556990 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff76806af00 msgr2=0x7ff76806d3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:06.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.753+0000 7ff781998700 1 --2- 172.21.15.189:0/1649556990 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff76806af00 0x7ff76806d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.753+0000 7ff781998700 1 -- 172.21.15.189:0/1649556990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff77c06b6f0 msgr2=0x7ff77c0b3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:06.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.753+0000 7ff781998700 1 --2- 172.21.15.189:0/1649556990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff77c06b6f0 0x7ff77c0b3d00 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff76c0011e0 tx=0x7ff76c00ca30 comp rx=0 tx=0).stop 2024-04-06T12:31:06.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.753+0000 7ff781998700 1 -- 172.21.15.189:0/1649556990 shutdown_connections 2024-04-06T12:31:06.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.753+0000 7ff781998700 1 --2- 172.21.15.189:0/1649556990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff77c15d7c0 0x7ff77c1600d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.753+0000 7ff781998700 1 --2- 172.21.15.189:0/1649556990 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff76806af00 0x7ff76806d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.753+0000 7ff781998700 1 --2- 172.21.15.189:0/1649556990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff77c06b6f0 0x7ff77c0b3d00 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.753+0000 7ff781998700 1 --2- 172.21.15.189:0/1649556990 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff77c0cd3e0 0x7ff77c15c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.753+0000 7ff781998700 1 -- 172.21.15.189:0/1649556990 >> 172.21.15.189:0/1649556990 conn(0x7ff77c0cc290 msgr2=0x7ff77c0adc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:06.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.753+0000 7ff781998700 1 -- 172.21.15.189:0/1649556990 shutdown_connections 2024-04-06T12:31:06.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.753+0000 7ff781998700 1 -- 172.21.15.189:0/1649556990 wait complete. 2024-04-06T12:31:06.763 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:31:06.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.891+0000 7f1ca00f8700 1 Processor -- start 2024-04-06T12:31:06.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.891+0000 7f1ca00f8700 1 -- start start 2024-04-06T12:31:06.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.892+0000 7f1ca00f8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1c9815d7b0 0x7f1c980b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:06.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.892+0000 7f1ca00f8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1c9815e500 0x7f1c9806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:06.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.892+0000 7f1ca00f8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c980b3990 con 0x7f1c9806a730 2024-04-06T12:31:06.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.892+0000 7f1ca00f8700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c9806b0f0 con 0x7f1c9815e500 2024-04-06T12:31:06.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.892+0000 7f1ca00f8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c980b3060 con 0x7f1c9815d7b0 2024-04-06T12:31:06.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.892+0000 7f1c9e695700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1c9815e500 0x7f1c9806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:06.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.892+0000 7f1c9e695700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1c9815e500 0x7f1c9806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36802/0 (socket says 172.21.15.189:36802) 2024-04-06T12:31:06.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.892+0000 7f1c9e695700 1 -- 172.21.15.189:0/4122456443 learned_addr learned my addr 172.21.15.189:0/4122456443 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:06.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.893+0000 7f1c9de94700 1 --2- 172.21.15.189:0/4122456443 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1c9815d7b0 0x7f1c980b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:06.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.893+0000 7f1c9e695700 1 -- 172.21.15.189:0/4122456443 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1c9815d7b0 msgr2=0x7f1c980b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:06.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.893+0000 7f1c9e695700 1 --2- 172.21.15.189:0/4122456443 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1c9815d7b0 0x7f1c980b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.893+0000 7f1c9e695700 1 -- 172.21.15.189:0/4122456443 >> v1:172.21.15.192:6789/0 conn(0x7f1c9806a730 legacy=0x7f1c9815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:06.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.893+0000 7f1c9e695700 1 -- 172.21.15.189:0/4122456443 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c980b3340 con 0x7f1c9815e500 2024-04-06T12:31:06.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.893+0000 7f1c9e695700 1 --2- 172.21.15.189:0/4122456443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c9815e500 0x7f1c9806abb0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f1c90009e50 tx=0x7f1c9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=242445d9823b25be server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:06.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.893+0000 7f1c9ce92700 1 -- 172.21.15.189:0/4122456443 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9000e070 con 0x7f1c9815e500 2024-04-06T12:31:06.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.893+0000 7f1c9ce92700 1 -- 172.21.15.189:0/4122456443 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9000b910 con 0x7f1c9815e500 2024-04-06T12:31:06.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.893+0000 7f1c9ce92700 1 -- 172.21.15.189:0/4122456443 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9000bcc0 con 0x7f1c9815e500 2024-04-06T12:31:06.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.894+0000 7f1ca00f8700 1 -- 172.21.15.189:0/4122456443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c9815e500 msgr2=0x7f1c9806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:06.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.894+0000 7f1ca00f8700 1 --2- 172.21.15.189:0/4122456443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c9815e500 0x7f1c9806abb0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f1c90009e50 tx=0x7f1c9000b330 comp rx=0 tx=0).stop 2024-04-06T12:31:06.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.894+0000 7f1ca00f8700 1 -- 172.21.15.189:0/4122456443 shutdown_connections 2024-04-06T12:31:06.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.894+0000 7f1ca00f8700 1 --2- 172.21.15.189:0/4122456443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c9815e500 0x7f1c9806abb0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.894+0000 7f1ca00f8700 1 --2- 172.21.15.189:0/4122456443 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1c9815d7b0 0x7f1c980b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.894+0000 7f1ca00f8700 1 -- 172.21.15.189:0/4122456443 >> 172.21.15.189:0/4122456443 conn(0x7f1c980d00b0 msgr2=0x7f1c980d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:06.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.894+0000 7f1ca00f8700 1 -- 172.21.15.189:0/4122456443 shutdown_connections 2024-04-06T12:31:06.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.894+0000 7f1ca00f8700 1 -- 172.21.15.189:0/4122456443 wait complete. 2024-04-06T12:31:06.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.895+0000 7f1ca00f8700 1 Processor -- start 2024-04-06T12:31:06.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.895+0000 7f1ca00f8700 1 -- start start 2024-04-06T12:31:06.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1ca00f8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c9806a730 0x7f1c981f3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:06.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1ca00f8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1c9815d7b0 0x7f1c980cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:06.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1ca00f8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c9815e500 0x7f1c981f6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:06.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1ca00f8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c980cd750 con 0x7f1c9806a730 2024-04-06T12:31:06.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1ca00f8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c980cdf20 con 0x7f1c9815e500 2024-04-06T12:31:06.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1ca00f8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c981f7410 con 0x7f1c9815d7b0 2024-04-06T12:31:06.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1c9de94700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c9806a730 0x7f1c981f3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:06.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1c9de94700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c9806a730 0x7f1c981f3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36818/0 (socket says 172.21.15.189:36818) 2024-04-06T12:31:06.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1c9de94700 1 -- 172.21.15.189:0/3347429610 learned_addr learned my addr 172.21.15.189:0/3347429610 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:06.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1c9e695700 1 --2- 172.21.15.189:0/3347429610 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c9815e500 0x7f1c981f6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:06.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1c9d693700 1 --2- 172.21.15.189:0/3347429610 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1c9815d7b0 0x7f1c980cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:06.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1c9de94700 1 -- 172.21.15.189:0/3347429610 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1c9815d7b0 msgr2=0x7f1c980cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:06.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1c9de94700 1 --2- 172.21.15.189:0/3347429610 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1c9815d7b0 0x7f1c980cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1c9de94700 1 -- 172.21.15.189:0/3347429610 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c9815e500 msgr2=0x7f1c981f6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:06.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1c9de94700 1 --2- 172.21.15.189:0/3347429610 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c9815e500 0x7f1c981f6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:06.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1c9de94700 1 -- 172.21.15.189:0/3347429610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c90009b30 con 0x7f1c9806a730 2024-04-06T12:31:06.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1c9de94700 1 --2- 172.21.15.189:0/3347429610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c9806a730 0x7f1c981f3a00 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f1c8800e9b0 tx=0x7f1c8800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:06.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1c8e7fc700 1 -- 172.21.15.189:0/3347429610 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c88009710 con 0x7f1c9806a730 2024-04-06T12:31:06.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1c8e7fc700 1 -- 172.21.15.189:0/3347429610 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c8800ee50 con 0x7f1c9806a730 2024-04-06T12:31:06.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1ca00f8700 1 -- 172.21.15.189:0/3347429610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c980cd960 con 0x7f1c9806a730 2024-04-06T12:31:06.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1c8e7fc700 1 -- 172.21.15.189:0/3347429610 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8800ca50 con 0x7f1c9806a730 2024-04-06T12:31:06.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.896+0000 7f1ca00f8700 1 -- 172.21.15.189:0/3347429610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c981f66d0 con 0x7f1c9806a730 2024-04-06T12:31:06.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.897+0000 7f1ca00f8700 1 -- 172.21.15.189:0/3347429610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c981f6530 con 0x7f1c9806a730 2024-04-06T12:31:06.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.898+0000 7f1c8e7fc700 1 -- 172.21.15.189:0/3347429610 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 71) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f1c8800cbb0 con 0x7f1c9806a730 2024-04-06T12:31:06.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.898+0000 7f1c8e7fc700 1 --2- 172.21.15.189:0/3347429610 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1c7c06aed0 0x7f1c7c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:06.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.898+0000 7f1c9d693700 1 -- 172.21.15.189:0/3347429610 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1c7c06aed0 msgr2=0x7f1c7c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:06.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.899+0000 7f1c8e7fc700 1 -- 172.21.15.189:0/3347429610 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f1c880899d0 con 0x7f1c9806a730 2024-04-06T12:31:06.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.899+0000 7f1c9d693700 1 --2- 172.21.15.189:0/3347429610 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1c7c06aed0 0x7f1c7c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:06.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:06.903+0000 7f1c8e7fc700 1 -- 172.21.15.189:0/3347429610 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1c981f6530 con 0x7f1c9806a730 2024-04-06T12:31:06.907 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:06.910+0000 7f6920c15200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:31:07.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.100+0000 7f1c9d693700 1 -- 172.21.15.189:0/3347429610 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1c7c06aed0 msgr2=0x7f1c7c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:07.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.100+0000 7f1c9d693700 1 --2- 172.21.15.189:0/3347429610 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1c7c06aed0 0x7f1c7c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:07.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.149+0000 7f1ca00f8700 1 -- 172.21.15.189:0/3347429610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c981f6530 con 0x7f1c9806a730 2024-04-06T12:31:07.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.150+0000 7f1c8e7fc700 1 -- 172.21.15.189:0/3347429610 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f1c981f6530 con 0x7f1c9806a730 2024-04-06T12:31:07.147 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:07.147 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/749266547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":749266547},{"type":"v1","addr":"172.21.15.192:6835","nonce":749266547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":6764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/947183051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":947183051},{"type":"v1","addr":"172.21.15.192:6834","nonce":947183051}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7492,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3990051444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3990051444},{"type":"v1","addr":"172.21.15.189:6837","nonce":3990051444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:30:17.127295+0000","modified":"2024-04-06T12:30:20.144458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8088},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8088":{"gid":8088,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/393232044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":393232044},{"type":"v1","addr":"172.21.15.189:6836","nonce":393232044}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-06T12:31:07.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.152+0000 7f1ca00f8700 1 -- 172.21.15.189:0/3347429610 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1c7c06aed0 msgr2=0x7f1c7c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:07.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.152+0000 7f1ca00f8700 1 --2- 172.21.15.189:0/3347429610 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1c7c06aed0 0x7f1c7c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.152+0000 7f1ca00f8700 1 -- 172.21.15.189:0/3347429610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c9806a730 msgr2=0x7f1c981f3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:07.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.152+0000 7f1ca00f8700 1 --2- 172.21.15.189:0/3347429610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c9806a730 0x7f1c981f3a00 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f1c8800e9b0 tx=0x7f1c8800ecc0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.152+0000 7f1ca00f8700 1 -- 172.21.15.189:0/3347429610 shutdown_connections 2024-04-06T12:31:07.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.152+0000 7f1ca00f8700 1 --2- 172.21.15.189:0/3347429610 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1c7c06aed0 0x7f1c7c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.152+0000 7f1ca00f8700 1 --2- 172.21.15.189:0/3347429610 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c9815e500 0x7f1c981f6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.152+0000 7f1ca00f8700 1 --2- 172.21.15.189:0/3347429610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c9806a730 0x7f1c981f3a00 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.152+0000 7f1ca00f8700 1 --2- 172.21.15.189:0/3347429610 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1c9815d7b0 0x7f1c980cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.152+0000 7f1ca00f8700 1 -- 172.21.15.189:0/3347429610 >> 172.21.15.189:0/3347429610 conn(0x7f1c980d00b0 msgr2=0x7f1c980724d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:07.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.152+0000 7f1ca00f8700 1 -- 172.21.15.189:0/3347429610 shutdown_connections 2024-04-06T12:31:07.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.152+0000 7f1ca00f8700 1 -- 172.21.15.189:0/3347429610 wait complete. 2024-04-06T12:31:07.151 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 54 2024-04-06T12:31:07.162 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:31:07.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.288+0000 7f52c8251700 1 Processor -- start 2024-04-06T12:31:07.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.288+0000 7f52c8251700 1 -- start start 2024-04-06T12:31:07.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.289+0000 7f52c8251700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00cb870 0x7f52c015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:07.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.289+0000 7f52c8251700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c00cbd80 con 0x7f52c00d11b0 2024-04-06T12:31:07.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.289+0000 7f52c8251700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c00cef20 con 0x7f52c00cfa00 2024-04-06T12:31:07.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.289+0000 7f52c8251700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c00ccdc0 con 0x7f52c00cb870 2024-04-06T12:31:07.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.289+0000 7f52c5fed700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f52c00d11b0 0x7f52c00d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45348/0 (socket says 172.21.15.189:45348) 2024-04-06T12:31:07.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.289+0000 7f52c5fed700 1 -- 172.21.15.189:0/2993075490 learned_addr learned my addr 172.21.15.189:0/2993075490 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:07.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.289+0000 7f52c57ec700 1 --2- 172.21.15.189:0/2993075490 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00cb870 0x7f52c015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:07.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.290+0000 7f52c4feb700 1 -- 172.21.15.189:0/2993075490 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1649385310 0 0) 0x7f52c00cbd80 con 0x7f52c00d11b0 2024-04-06T12:31:07.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.290+0000 7f52c4feb700 1 -- 172.21.15.189:0/2993075490 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52ac0035b0 con 0x7f52c00d11b0 2024-04-06T12:31:07.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.290+0000 7f52c4feb700 1 -- 172.21.15.189:0/2993075490 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815901908 0 0) 0x7f52c00cef20 con 0x7f52c00cfa00 2024-04-06T12:31:07.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.290+0000 7f52c4feb700 1 -- 172.21.15.189:0/2993075490 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52c00cbd80 con 0x7f52c00cfa00 2024-04-06T12:31:07.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.290+0000 7f52c57ec700 1 -- 172.21.15.189:0/2993075490 >> v1:172.21.15.192:6789/0 conn(0x7f52c00cfa00 legacy=0x7f52c00cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:07.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.290+0000 7f52c57ec700 1 -- 172.21.15.189:0/2993075490 >> v1:172.21.15.189:6789/0 conn(0x7f52c00d11b0 legacy=0x7f52c00d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:07.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.290+0000 7f52c57ec700 1 -- 172.21.15.189:0/2993075490 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c00cf200 con 0x7f52c00cb870 2024-04-06T12:31:07.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.290+0000 7f52c57ec700 1 --2- 172.21.15.189:0/2993075490 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00cb870 0x7f52c015c280 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f52b0009870 tx=0x7f52b0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34e16b0ba2f08d5d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:07.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.290+0000 7f52c4feb700 1 -- 172.21.15.189:0/2993075490 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0011830 con 0x7f52c00cb870 2024-04-06T12:31:07.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.290+0000 7f52c4feb700 1 -- 172.21.15.189:0/2993075490 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b0011990 con 0x7f52c00cb870 2024-04-06T12:31:07.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.290+0000 7f52c4feb700 1 -- 172.21.15.189:0/2993075490 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0011d20 con 0x7f52c00cb870 2024-04-06T12:31:07.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.291+0000 7f52c8251700 1 -- 172.21.15.189:0/2993075490 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00cb870 msgr2=0x7f52c015c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:07.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.291+0000 7f52c8251700 1 --2- 172.21.15.189:0/2993075490 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00cb870 0x7f52c015c280 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f52b0009870 tx=0x7f52b0009b80 comp rx=0 tx=0).stop 2024-04-06T12:31:07.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.291+0000 7f52c8251700 1 -- 172.21.15.189:0/2993075490 shutdown_connections 2024-04-06T12:31:07.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.291+0000 7f52c8251700 1 --2- 172.21.15.189:0/2993075490 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00cb870 0x7f52c015c280 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.291+0000 7f52c8251700 1 -- 172.21.15.189:0/2993075490 >> 172.21.15.189:0/2993075490 conn(0x7f52c00b2db0 msgr2=0x7f52c00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:07.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.291+0000 7f52c8251700 1 -- 172.21.15.189:0/2993075490 shutdown_connections 2024-04-06T12:31:07.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.291+0000 7f52c8251700 1 -- 172.21.15.189:0/2993075490 wait complete. 2024-04-06T12:31:07.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.292+0000 7f52c8251700 1 Processor -- start 2024-04-06T12:31:07.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.292+0000 7f52c8251700 1 -- start start 2024-04-06T12:31:07.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.293+0000 7f52c8251700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52c00cb870 0x7f52c015c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:07.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.293+0000 7f52c8251700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00cfa00 0x7f52c00ca920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:07.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.293+0000 7f52c8251700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52c00d11b0 0x7f52c01f6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:07.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.293+0000 7f52c8251700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c015e890 con 0x7f52c00d11b0 2024-04-06T12:31:07.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.293+0000 7f52c8251700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c015eb70 con 0x7f52c00cb870 2024-04-06T12:31:07.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.293+0000 7f52c8251700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c015ee50 con 0x7f52c00cfa00 2024-04-06T12:31:07.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.293+0000 7f52c67ee700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52c00d11b0 0x7f52c01f6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:07.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.293+0000 7f52c67ee700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52c00d11b0 0x7f52c01f6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36826/0 (socket says 172.21.15.189:36826) 2024-04-06T12:31:07.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.293+0000 7f52c67ee700 1 -- 172.21.15.189:0/3096403038 learned_addr learned my addr 172.21.15.189:0/3096403038 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:07.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.293+0000 7f52c57ec700 1 --2- 172.21.15.189:0/3096403038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00cfa00 0x7f52c00ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:07.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.293+0000 7f52c5fed700 1 --2- 172.21.15.189:0/3096403038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52c00cb870 0x7f52c015c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:07.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.293+0000 7f52c67ee700 1 -- 172.21.15.189:0/3096403038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00cfa00 msgr2=0x7f52c00ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:07.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.293+0000 7f52c67ee700 1 --2- 172.21.15.189:0/3096403038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00cfa00 0x7f52c00ca920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.293+0000 7f52c67ee700 1 -- 172.21.15.189:0/3096403038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52c00cb870 msgr2=0x7f52c015c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:07.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.293+0000 7f52c67ee700 1 --2- 172.21.15.189:0/3096403038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52c00cb870 0x7f52c015c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.293+0000 7f52c67ee700 1 -- 172.21.15.189:0/3096403038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b00094a0 con 0x7f52c00d11b0 2024-04-06T12:31:07.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.294+0000 7f52c67ee700 1 --2- 172.21.15.189:0/3096403038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52c00d11b0 0x7f52c01f6800 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f52b800cc20 tx=0x7f52b800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:07.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.294+0000 7f52b67fc700 1 -- 172.21.15.189:0/3096403038 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b8003c00 con 0x7f52c00d11b0 2024-04-06T12:31:07.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.294+0000 7f52b67fc700 1 -- 172.21.15.189:0/3096403038 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b8003d60 con 0x7f52c00d11b0 2024-04-06T12:31:07.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.294+0000 7f52c8251700 1 -- 172.21.15.189:0/3096403038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c015f130 con 0x7f52c00d11b0 2024-04-06T12:31:07.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.294+0000 7f52b67fc700 1 -- 172.21.15.189:0/3096403038 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b800aae0 con 0x7f52c00d11b0 2024-04-06T12:31:07.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.294+0000 7f52c8251700 1 -- 172.21.15.189:0/3096403038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c01f7630 con 0x7f52c00d11b0 2024-04-06T12:31:07.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.295+0000 7f52c8251700 1 -- 172.21.15.189:0/3096403038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c015f650 con 0x7f52c00d11b0 2024-04-06T12:31:07.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.295+0000 7f52b67fc700 1 -- 172.21.15.189:0/3096403038 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 71) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f52b800ac40 con 0x7f52c00d11b0 2024-04-06T12:31:07.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.296+0000 7f52b67fc700 1 --2- 172.21.15.189:0/3096403038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f52a406aed0 0x7f52a406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:07.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.296+0000 7f52c5fed700 1 -- 172.21.15.189:0/3096403038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f52a406aed0 msgr2=0x7f52a406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:07.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.296+0000 7f52b67fc700 1 -- 172.21.15.189:0/3096403038 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f52b8089a50 con 0x7f52c00d11b0 2024-04-06T12:31:07.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.296+0000 7f52c5fed700 1 --2- 172.21.15.189:0/3096403038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f52a406aed0 0x7f52a406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:07.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.300+0000 7f52b67fc700 1 -- 172.21.15.189:0/3096403038 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f52b8054630 con 0x7f52c00d11b0 2024-04-06T12:31:07.349 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:07.352+0000 7f3500729200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:31:07.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.497+0000 7f52c5fed700 1 -- 172.21.15.189:0/3096403038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f52a406aed0 msgr2=0x7f52a406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:07.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.497+0000 7f52c5fed700 1 --2- 172.21.15.189:0/3096403038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f52a406aed0 0x7f52a406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:07.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.502+0000 7f52c8251700 1 -- 172.21.15.189:0/3096403038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52c015bb60 con 0x7f52c00d11b0 2024-04-06T12:31:07.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.503+0000 7f52b67fc700 1 -- 172.21.15.189:0/3096403038 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16792 (secure 0 0 0) 0x7f52b8057c50 con 0x7f52c00d11b0 2024-04-06T12:31:07.500 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:07.501 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":129,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:30:31.084655+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:30:14.958335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-06T12:30:16.104585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/4170508796":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/169651662":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/29183104":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/3994101871":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/1300227922":"2024-04-07T12:30:31.084606+0000","172.21.15.189:6833/47128":"2024-04-07T12:30:31.084606+0000","172.21.15.189:6832/47128":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/1086259216":"2024-04-07T12:30:31.084606+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:31:07.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.505+0000 7f52c8251700 1 -- 172.21.15.189:0/3096403038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f52a406aed0 msgr2=0x7f52a406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:07.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.505+0000 7f52c8251700 1 --2- 172.21.15.189:0/3096403038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f52a406aed0 0x7f52a406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.505+0000 7f52c8251700 1 -- 172.21.15.189:0/3096403038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52c00d11b0 msgr2=0x7f52c01f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:07.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.505+0000 7f52c8251700 1 --2- 172.21.15.189:0/3096403038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52c00d11b0 0x7f52c01f6800 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f52b800cc20 tx=0x7f52b800cf30 comp rx=0 tx=0).stop 2024-04-06T12:31:07.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.505+0000 7f52c8251700 1 -- 172.21.15.189:0/3096403038 shutdown_connections 2024-04-06T12:31:07.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.505+0000 7f52c8251700 1 --2- 172.21.15.189:0/3096403038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52c00d11b0 0x7f52c01f6800 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.505+0000 7f52c8251700 1 --2- 172.21.15.189:0/3096403038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f52a406aed0 0x7f52a406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.505+0000 7f52c8251700 1 --2- 172.21.15.189:0/3096403038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52c00cb870 0x7f52c015c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.505+0000 7f52c8251700 1 --2- 172.21.15.189:0/3096403038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00cfa00 0x7f52c00ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.505+0000 7f52c8251700 1 -- 172.21.15.189:0/3096403038 >> 172.21.15.189:0/3096403038 conn(0x7f52c00b2db0 msgr2=0x7f52c0159a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:07.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.506+0000 7f52c8251700 1 -- 172.21.15.189:0/3096403038 shutdown_connections 2024-04-06T12:31:07.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.506+0000 7f52c8251700 1 -- 172.21.15.189:0/3096403038 wait complete. 2024-04-06T12:31:07.515 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-06T12:31:07.516 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:31:07.590 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:07.592+0000 7f3500729200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:31:07.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.643+0000 7f10e4de6700 1 Processor -- start 2024-04-06T12:31:07.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.643+0000 7f10e4de6700 1 -- start start 2024-04-06T12:31:07.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.645+0000 7f10e4de6700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f10e00b4ed0 0x7f10e00b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:07.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.645+0000 7f10e4de6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f10e00d2880 0x7f10e00d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:07.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.645+0000 7f10e4de6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e00b5cc0 con 0x7f10e015d590 2024-04-06T12:31:07.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.645+0000 7f10e4de6700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e00b58c0 con 0x7f10e00d2880 2024-04-06T12:31:07.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.645+0000 7f10e4de6700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e00d1500 con 0x7f10e00b4ed0 2024-04-06T12:31:07.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.645+0000 7f10ded9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f10e00d2880 0x7f10e00d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:07.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.645+0000 7f10de59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f10e015d590 0x7f10e00b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45364/0 (socket says 172.21.15.189:45364) 2024-04-06T12:31:07.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.645+0000 7f10de59c700 1 -- 172.21.15.189:0/71484460 learned_addr learned my addr 172.21.15.189:0/71484460 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:07.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.645+0000 7f10ddd9b700 1 --2- 172.21.15.189:0/71484460 >> v2:172.21.15.192:3300/0 conn(0x7f10e00b4ed0 0x7f10e00b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:07.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.645+0000 7f10ded9d700 1 -- 172.21.15.189:0/71484460 >> v2:172.21.15.192:3300/0 conn(0x7f10e00b4ed0 msgr2=0x7f10e00b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:07.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.645+0000 7f10ded9d700 1 --2- 172.21.15.189:0/71484460 >> v2:172.21.15.192:3300/0 conn(0x7f10e00b4ed0 0x7f10e00b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.645+0000 7f10ded9d700 1 -- 172.21.15.189:0/71484460 >> v1:172.21.15.189:6789/0 conn(0x7f10e015d590 legacy=0x7f10e00b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:07.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.645+0000 7f10ded9d700 1 -- 172.21.15.189:0/71484460 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e00d17e0 con 0x7f10e00d2880 2024-04-06T12:31:07.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.645+0000 7f10dd59a700 1 -- 172.21.15.189:0/71484460 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875454942 0 0) 0x7f10e00b5cc0 con 0x7f10e015d590 2024-04-06T12:31:07.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.645+0000 7f10ddd9b700 1 --2- 172.21.15.189:0/71484460 >> v2:172.21.15.192:3300/0 conn(0x7f10e00b4ed0 0x7f10e00b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:07.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.646+0000 7f10ded9d700 1 --2- 172.21.15.189:0/71484460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e00d2880 0x7f10e00d2cd0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f10e00b6330 tx=0x7f10d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa3a747a3eb0494 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:07.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.646+0000 7f10dd59a700 1 -- 172.21.15.189:0/71484460 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d800e070 con 0x7f10e00d2880 2024-04-06T12:31:07.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.646+0000 7f10dd59a700 1 -- 172.21.15.189:0/71484460 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d800bb20 con 0x7f10e00d2880 2024-04-06T12:31:07.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.646+0000 7f10dd59a700 1 -- 172.21.15.189:0/71484460 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d8012430 con 0x7f10e00d2880 2024-04-06T12:31:07.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.646+0000 7f10e4de6700 1 -- 172.21.15.189:0/71484460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e00d2880 msgr2=0x7f10e00d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:07.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.646+0000 7f10e4de6700 1 --2- 172.21.15.189:0/71484460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e00d2880 0x7f10e00d2cd0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f10e00b6330 tx=0x7f10d800b330 comp rx=0 tx=0).stop 2024-04-06T12:31:07.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.647+0000 7f10e4de6700 1 -- 172.21.15.189:0/71484460 shutdown_connections 2024-04-06T12:31:07.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.647+0000 7f10e4de6700 1 --2- 172.21.15.189:0/71484460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e00d2880 0x7f10e00d2cd0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.647+0000 7f10e4de6700 1 --2- 172.21.15.189:0/71484460 >> v2:172.21.15.192:3300/0 conn(0x7f10e00b4ed0 0x7f10e00b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.647+0000 7f10e4de6700 1 -- 172.21.15.189:0/71484460 >> 172.21.15.189:0/71484460 conn(0x7f10e006bbc0 msgr2=0x7f10e006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:07.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.647+0000 7f10e4de6700 1 -- 172.21.15.189:0/71484460 shutdown_connections 2024-04-06T12:31:07.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.647+0000 7f10e4de6700 1 -- 172.21.15.189:0/71484460 wait complete. 2024-04-06T12:31:07.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.648+0000 7f10e4de6700 1 Processor -- start 2024-04-06T12:31:07.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.648+0000 7f10e4de6700 1 -- start start 2024-04-06T12:31:07.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.649+0000 7f10e4de6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e00b4ed0 0x7f10e014ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:07.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.649+0000 7f10e4de6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e00d2880 0x7f10e014f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:07.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.649+0000 7f10e4de6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e015d590 0x7f10e00b8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:07.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.649+0000 7f10e4de6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e00cb6a0 con 0x7f10e015d590 2024-04-06T12:31:07.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.649+0000 7f10e4de6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e00caa20 con 0x7f10e00d2880 2024-04-06T12:31:07.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.649+0000 7f10e4de6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e00cad00 con 0x7f10e00b4ed0 2024-04-06T12:31:07.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.649+0000 7f10ded9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e015d590 0x7f10e00b8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:07.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.649+0000 7f10ded9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e015d590 0x7f10e00b8830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36834/0 (socket says 172.21.15.189:36834) 2024-04-06T12:31:07.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.649+0000 7f10ded9d700 1 -- 172.21.15.189:0/4024781095 learned_addr learned my addr 172.21.15.189:0/4024781095 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:07.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.649+0000 7f10ddd9b700 1 --2- 172.21.15.189:0/4024781095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e00d2880 0x7f10e014f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:07.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.649+0000 7f10de59c700 1 --2- 172.21.15.189:0/4024781095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e00b4ed0 0x7f10e014ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:07.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.649+0000 7f10ded9d700 1 -- 172.21.15.189:0/4024781095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e00b4ed0 msgr2=0x7f10e014ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:07.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.649+0000 7f10ded9d700 1 --2- 172.21.15.189:0/4024781095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e00b4ed0 0x7f10e014ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.649+0000 7f10ded9d700 1 -- 172.21.15.189:0/4024781095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e00d2880 msgr2=0x7f10e014f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:07.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.649+0000 7f10ded9d700 1 --2- 172.21.15.189:0/4024781095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e00d2880 0x7f10e014f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.649+0000 7f10ded9d700 1 -- 172.21.15.189:0/4024781095 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d40094a0 con 0x7f10e015d590 2024-04-06T12:31:07.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.649+0000 7f10ded9d700 1 --2- 172.21.15.189:0/4024781095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e015d590 0x7f10e00b8830 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f10e00d41b0 tx=0x7f10d8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:07.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.650+0000 7f10ceffd700 1 -- 172.21.15.189:0/4024781095 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d8012430 con 0x7f10e015d590 2024-04-06T12:31:07.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.650+0000 7f10ceffd700 1 -- 172.21.15.189:0/4024781095 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d8020070 con 0x7f10e015d590 2024-04-06T12:31:07.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.650+0000 7f10ceffd700 1 -- 172.21.15.189:0/4024781095 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d8004280 con 0x7f10e015d590 2024-04-06T12:31:07.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.650+0000 7f10e4de6700 1 -- 172.21.15.189:0/4024781095 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d8009a70 con 0x7f10e015d590 2024-04-06T12:31:07.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.650+0000 7f10e4de6700 1 -- 172.21.15.189:0/4024781095 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e00cb2c0 con 0x7f10e015d590 2024-04-06T12:31:07.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.651+0000 7f10e4de6700 1 -- 172.21.15.189:0/4024781095 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e00cb7e0 con 0x7f10e015d590 2024-04-06T12:31:07.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.651+0000 7f10ceffd700 1 -- 172.21.15.189:0/4024781095 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 71) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f10d80043e0 con 0x7f10e015d590 2024-04-06T12:31:07.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.652+0000 7f10ceffd700 1 --2- 172.21.15.189:0/4024781095 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10c006aed0 0x7f10c006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:07.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.652+0000 7f10de59c700 1 -- 172.21.15.189:0/4024781095 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10c006aed0 msgr2=0x7f10c006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:07.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.652+0000 7f10ceffd700 1 -- 172.21.15.189:0/4024781095 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f10d8022070 con 0x7f10e015d590 2024-04-06T12:31:07.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.652+0000 7f10de59c700 1 --2- 172.21.15.189:0/4024781095 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10c006aed0 0x7f10c006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:07.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.656+0000 7f10ceffd700 1 -- 172.21.15.189:0/4024781095 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f10e00cb7e0 con 0x7f10e015d590 2024-04-06T12:31:07.670 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:07.673+0000 7f3500729200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:31:07.676 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:07.679+0000 7f6920c15200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:31:07.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.853+0000 7f10de59c700 1 -- 172.21.15.189:0/4024781095 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10c006aed0 msgr2=0x7f10c006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:07.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.853+0000 7f10de59c700 1 --2- 172.21.15.189:0/4024781095 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10c006aed0 0x7f10c006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:07.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.916+0000 7f10e4de6700 1 -- 172.21.15.189:0/4024781095 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10e00cb7e0 con 0x7f10e015d590 2024-04-06T12:31:07.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.917+0000 7f10ceffd700 1 -- 172.21.15.189:0/4024781095 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f10e00cb7e0 con 0x7f10e015d590 2024-04-06T12:31:07.914 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:07.914 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/749266547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":749266547},{"type":"v1","addr":"172.21.15.192:6835","nonce":749266547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":6764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/947183051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":947183051},{"type":"v1","addr":"172.21.15.192:6834","nonce":947183051}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7492,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3990051444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3990051444},{"type":"v1","addr":"172.21.15.189:6837","nonce":3990051444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:30:17.127295+0000","modified":"2024-04-06T12:30:20.144458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8088},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8088":{"gid":8088,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/393232044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":393232044},{"type":"v1","addr":"172.21.15.189:6836","nonce":393232044}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-06T12:31:07.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.919+0000 7f10e4de6700 1 -- 172.21.15.189:0/4024781095 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10c006aed0 msgr2=0x7f10c006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:07.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.919+0000 7f10e4de6700 1 --2- 172.21.15.189:0/4024781095 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10c006aed0 0x7f10c006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.919+0000 7f10e4de6700 1 -- 172.21.15.189:0/4024781095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e015d590 msgr2=0x7f10e00b8830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:07.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.919+0000 7f10e4de6700 1 --2- 172.21.15.189:0/4024781095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e015d590 0x7f10e00b8830 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f10e00d41b0 tx=0x7f10d8004ad0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.919+0000 7f10e4de6700 1 -- 172.21.15.189:0/4024781095 shutdown_connections 2024-04-06T12:31:07.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.919+0000 7f10e4de6700 1 --2- 172.21.15.189:0/4024781095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e015d590 0x7f10e00b8830 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.919+0000 7f10e4de6700 1 --2- 172.21.15.189:0/4024781095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e00b4ed0 0x7f10e014ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.919+0000 7f10e4de6700 1 --2- 172.21.15.189:0/4024781095 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f10c006aed0 0x7f10c006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.919+0000 7f10e4de6700 1 --2- 172.21.15.189:0/4024781095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e00d2880 0x7f10e014f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:07.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.919+0000 7f10e4de6700 1 -- 172.21.15.189:0/4024781095 >> 172.21.15.189:0/4024781095 conn(0x7f10e006bbc0 msgr2=0x7f10e00d3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:07.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.919+0000 7f10e4de6700 1 -- 172.21.15.189:0/4024781095 shutdown_connections 2024-04-06T12:31:07.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:07.919+0000 7f10e4de6700 1 -- 172.21.15.189:0/4024781095 wait complete. 2024-04-06T12:31:07.918 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 54 2024-04-06T12:31:07.929 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:31:07.937 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:07.940+0000 7f6920c15200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:31:08.019 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:08.021+0000 7f6920c15200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:31:08.046 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:08.049+0000 7f3500729200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:31:08.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.057+0000 7fd233a4a700 1 Processor -- start 2024-04-06T12:31:08.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.057+0000 7fd233a4a700 1 -- start start 2024-04-06T12:31:08.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.058+0000 7fd233a4a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd22c0b4ed0 0x7fd22c0b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:08.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.058+0000 7fd233a4a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd22c0d2880 0x7fd22c0d2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:08.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.058+0000 7fd233a4a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22c0b5cc0 con 0x7fd22c15d590 2024-04-06T12:31:08.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.058+0000 7fd233a4a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd22c0b58c0 con 0x7fd22c0b4ed0 2024-04-06T12:31:08.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.058+0000 7fd233a4a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0d1500 con 0x7fd22c0d2880 2024-04-06T12:31:08.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.058+0000 7fd231fe7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd22c0d2880 0x7fd22c0d2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:08.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.058+0000 7fd230fe5700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd22c0b4ed0 0x7fd22c0b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:08.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.058+0000 7fd2317e6700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fd22c15d590 0x7fd22c0b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:47016/0 (socket says 172.21.15.189:47016) 2024-04-06T12:31:08.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.058+0000 7fd2317e6700 1 -- 172.21.15.189:0/3348077886 learned_addr learned my addr 172.21.15.189:0/3348077886 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:08.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.059+0000 7fd223fff700 1 -- 172.21.15.189:0/3348077886 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50698447 0 0) 0x7fd22c0b5cc0 con 0x7fd22c15d590 2024-04-06T12:31:08.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.059+0000 7fd223fff700 1 -- 172.21.15.189:0/3348077886 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2180035b0 con 0x7fd22c15d590 2024-04-06T12:31:08.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.059+0000 7fd231fe7700 1 -- 172.21.15.189:0/3348077886 >> v2:172.21.15.192:3300/0 conn(0x7fd22c0b4ed0 msgr2=0x7fd22c0b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:08.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.059+0000 7fd231fe7700 1 --2- 172.21.15.189:0/3348077886 >> v2:172.21.15.192:3300/0 conn(0x7fd22c0b4ed0 0x7fd22c0b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.059+0000 7fd231fe7700 1 -- 172.21.15.189:0/3348077886 >> v1:172.21.15.192:6789/0 conn(0x7fd22c15d590 legacy=0x7fd22c0b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:08.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.059+0000 7fd231fe7700 1 -- 172.21.15.189:0/3348077886 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22c0d17e0 con 0x7fd22c0d2880 2024-04-06T12:31:08.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.059+0000 7fd230fe5700 1 --2- 172.21.15.189:0/3348077886 >> v2:172.21.15.192:3300/0 conn(0x7fd22c0b4ed0 0x7fd22c0b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:08.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.059+0000 7fd231fe7700 1 --2- 172.21.15.189:0/3348077886 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd22c0d2880 0x7fd22c0d2cf0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd224009d90 tx=0x7fd22400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8c14942d060061 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:08.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.059+0000 7fd223fff700 1 -- 172.21.15.189:0/3348077886 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22400e070 con 0x7fd22c0d2880 2024-04-06T12:31:08.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.059+0000 7fd223fff700 1 -- 172.21.15.189:0/3348077886 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22400bb20 con 0x7fd22c0d2880 2024-04-06T12:31:08.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.059+0000 7fd223fff700 1 -- 172.21.15.189:0/3348077886 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd224012430 con 0x7fd22c0d2880 2024-04-06T12:31:08.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.060+0000 7fd233a4a700 1 -- 172.21.15.189:0/3348077886 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd22c0d2880 msgr2=0x7fd22c0d2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:08.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.060+0000 7fd233a4a700 1 --2- 172.21.15.189:0/3348077886 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd22c0d2880 0x7fd22c0d2cf0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd224009d90 tx=0x7fd22400b330 comp rx=0 tx=0).stop 2024-04-06T12:31:08.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.060+0000 7fd233a4a700 1 -- 172.21.15.189:0/3348077886 shutdown_connections 2024-04-06T12:31:08.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.060+0000 7fd233a4a700 1 --2- 172.21.15.189:0/3348077886 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd22c0d2880 0x7fd22c0d2cf0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.060+0000 7fd233a4a700 1 --2- 172.21.15.189:0/3348077886 >> v2:172.21.15.192:3300/0 conn(0x7fd22c0b4ed0 0x7fd22c0b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.060+0000 7fd233a4a700 1 -- 172.21.15.189:0/3348077886 >> 172.21.15.189:0/3348077886 conn(0x7fd22c06bbc0 msgr2=0x7fd22c06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:08.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.060+0000 7fd233a4a700 1 -- 172.21.15.189:0/3348077886 shutdown_connections 2024-04-06T12:31:08.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.060+0000 7fd233a4a700 1 -- 172.21.15.189:0/3348077886 wait complete. 2024-04-06T12:31:08.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.061+0000 7fd233a4a700 1 Processor -- start 2024-04-06T12:31:08.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.061+0000 7fd233a4a700 1 -- start start 2024-04-06T12:31:08.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.062+0000 7fd233a4a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd22c0b4ed0 0x7fd22c14ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:08.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.062+0000 7fd233a4a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd22c0d2880 0x7fd22c15c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:08.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.062+0000 7fd233a4a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd22c15d590 0x7fd22c15a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:08.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.062+0000 7fd233a4a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c1530a0 con 0x7fd22c0b4ed0 2024-04-06T12:31:08.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.062+0000 7fd233a4a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c152800 con 0x7fd22c0d2880 2024-04-06T12:31:08.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.062+0000 7fd233a4a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c152ae0 con 0x7fd22c15d590 2024-04-06T12:31:08.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.062+0000 7fd2317e6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd22c0b4ed0 0x7fd22c14ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.062+0000 7fd2317e6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd22c0b4ed0 0x7fd22c14ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36840/0 (socket says 172.21.15.189:36840) 2024-04-06T12:31:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.062+0000 7fd2317e6700 1 -- 172.21.15.189:0/1064847972 learned_addr learned my addr 172.21.15.189:0/1064847972 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.062+0000 7fd231fe7700 1 --2- 172.21.15.189:0/1064847972 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd22c15d590 0x7fd22c15a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.062+0000 7fd230fe5700 1 --2- 172.21.15.189:0/1064847972 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd22c0d2880 0x7fd22c15c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.062+0000 7fd231fe7700 1 -- 172.21.15.189:0/1064847972 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd22c0d2880 msgr2=0x7fd22c15c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.062+0000 7fd231fe7700 1 --2- 172.21.15.189:0/1064847972 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd22c0d2880 0x7fd22c15c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.062+0000 7fd231fe7700 1 -- 172.21.15.189:0/1064847972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd22c0b4ed0 msgr2=0x7fd22c14ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.062+0000 7fd231fe7700 1 --2- 172.21.15.189:0/1064847972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd22c0b4ed0 0x7fd22c14ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.062+0000 7fd231fe7700 1 -- 172.21.15.189:0/1064847972 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd21c0094a0 con 0x7fd22c15d590 2024-04-06T12:31:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.062+0000 7fd2317e6700 1 --2- 172.21.15.189:0/1064847972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd22c0b4ed0 0x7fd22c14ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.063+0000 7fd231fe7700 1 --2- 172.21.15.189:0/1064847972 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd22c15d590 0x7fd22c15a0c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd224016000 tx=0x7fd224004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.063+0000 7fd221ffb700 1 -- 172.21.15.189:0/1064847972 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd224012430 con 0x7fd22c15d590 2024-04-06T12:31:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.063+0000 7fd221ffb700 1 -- 172.21.15.189:0/1064847972 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd224020070 con 0x7fd22c15d590 2024-04-06T12:31:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.063+0000 7fd233a4a700 1 -- 172.21.15.189:0/1064847972 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd224009a70 con 0x7fd22c15d590 2024-04-06T12:31:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.063+0000 7fd221ffb700 1 -- 172.21.15.189:0/1064847972 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2240098f0 con 0x7fd22c15d590 2024-04-06T12:31:08.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.063+0000 7fd233a4a700 1 -- 172.21.15.189:0/1064847972 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd22c0ca5c0 con 0x7fd22c15d590 2024-04-06T12:31:08.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.064+0000 7fd233a4a700 1 -- 172.21.15.189:0/1064847972 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd22c074990 con 0x7fd22c15d590 2024-04-06T12:31:08.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.064+0000 7fd221ffb700 1 -- 172.21.15.189:0/1064847972 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 71) v1 ==== 89361+0+0 (secure 0 0 0) 0x7fd224004040 con 0x7fd22c15d590 2024-04-06T12:31:08.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.065+0000 7fd221ffb700 1 --2- 172.21.15.189:0/1064847972 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd21006aed0 0x7fd21006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:08.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.065+0000 7fd2317e6700 1 -- 172.21.15.189:0/1064847972 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd21006aed0 msgr2=0x7fd21006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:08.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.065+0000 7fd221ffb700 1 -- 172.21.15.189:0/1064847972 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6429+0+0 (secure 0 0 0) 0x7fd224022070 con 0x7fd22c15d590 2024-04-06T12:31:08.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.065+0000 7fd2317e6700 1 --2- 172.21.15.189:0/1064847972 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd21006aed0 0x7fd21006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:08.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.069+0000 7fd221ffb700 1 -- 172.21.15.189:0/1064847972 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd224056500 con 0x7fd22c15d590 2024-04-06T12:31:08.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.266+0000 7fd2317e6700 1 -- 172.21.15.189:0/1064847972 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd21006aed0 msgr2=0x7fd21006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:08.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.266+0000 7fd2317e6700 1 --2- 172.21.15.189:0/1064847972 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd21006aed0 0x7fd21006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:08.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.319+0000 7fd233a4a700 1 -- 172.21.15.189:0/1064847972 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd22c0d3ba0 con 0x7fd22c15d590 2024-04-06T12:31:08.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.320+0000 7fd221ffb700 1 -- 172.21.15.189:0/1064847972 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4739 (secure 0 0 0) 0x7fd224059b20 con 0x7fd22c15d590 2024-04-06T12:31:08.317 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:08.317 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/749266547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":749266547},{"type":"v1","addr":"172.21.15.192:6835","nonce":749266547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":6764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/947183051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":947183051},{"type":"v1","addr":"172.21.15.192:6834","nonce":947183051}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7492,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3990051444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3990051444},{"type":"v1","addr":"172.21.15.189:6837","nonce":3990051444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:30:17.127295+0000","modified":"2024-04-06T12:30:20.144458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8088},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8088":{"gid":8088,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/393232044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":393232044},{"type":"v1","addr":"172.21.15.189:6836","nonce":393232044}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-06T12:31:08.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.322+0000 7fd233a4a700 1 -- 172.21.15.189:0/1064847972 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd21006aed0 msgr2=0x7fd21006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:08.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.322+0000 7fd233a4a700 1 --2- 172.21.15.189:0/1064847972 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd21006aed0 0x7fd21006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.322+0000 7fd233a4a700 1 -- 172.21.15.189:0/1064847972 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd22c15d590 msgr2=0x7fd22c15a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:08.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.322+0000 7fd233a4a700 1 --2- 172.21.15.189:0/1064847972 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd22c15d590 0x7fd22c15a0c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd224016000 tx=0x7fd224004ad0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.322+0000 7fd233a4a700 1 -- 172.21.15.189:0/1064847972 shutdown_connections 2024-04-06T12:31:08.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.322+0000 7fd233a4a700 1 --2- 172.21.15.189:0/1064847972 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd22c15d590 0x7fd22c15a0c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.322+0000 7fd233a4a700 1 --2- 172.21.15.189:0/1064847972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd22c0b4ed0 0x7fd22c14ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.322+0000 7fd233a4a700 1 --2- 172.21.15.189:0/1064847972 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd21006aed0 0x7fd21006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.322+0000 7fd233a4a700 1 --2- 172.21.15.189:0/1064847972 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd22c0d2880 0x7fd22c15c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.322+0000 7fd233a4a700 1 -- 172.21.15.189:0/1064847972 >> 172.21.15.189:0/1064847972 conn(0x7fd22c06bbc0 msgr2=0x7fd22c06bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:08.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.322+0000 7fd233a4a700 1 -- 172.21.15.189:0/1064847972 shutdown_connections 2024-04-06T12:31:08.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.322+0000 7fd233a4a700 1 -- 172.21.15.189:0/1064847972 wait complete. 2024-04-06T12:31:08.321 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 54 2024-04-06T12:31:08.332 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:31:08.414 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:08.417+0000 7f6920c15200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:31:08.499 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:08.502+0000 7f689932e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:08.499 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:08.502+0000 7f689932e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:08.499 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:08.502+0000 7f689932e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:08.499 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:08.502+0000 7f689932e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:08.499 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:08.502+0000 7f689932e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:08.500 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:08.502+0000 7f689c334700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:08.500 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:08.502+0000 7f689c334700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:08.500 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:08.502+0000 7f689c334700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:08.500 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:08.502+0000 7f689c334700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:08.500 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:08.502+0000 7f689c334700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:08.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.514+0000 7f2481cc8700 1 Processor -- start 2024-04-06T12:31:08.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.514+0000 7f2481cc8700 1 -- start start 2024-04-06T12:31:08.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.515+0000 7f2481cc8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f247c0ce730 0x7f247c15dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:08.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.515+0000 7f2481cc8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247c15eac0 con 0x7f247c15d6f0 2024-04-06T12:31:08.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.515+0000 7f2481cc8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247c15e220 con 0x7f247c0b8d70 2024-04-06T12:31:08.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.515+0000 7f2481cc8700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f247c15e010 con 0x7f247c0ce730 2024-04-06T12:31:08.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.515+0000 7f247b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f247c15d6f0 0x7f247c0ce020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45374/0 (socket says 172.21.15.189:45374) 2024-04-06T12:31:08.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.515+0000 7f247b7fe700 1 -- 172.21.15.189:0/3011203078 learned_addr learned my addr 172.21.15.189:0/3011203078 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:08.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.515+0000 7f247affd700 1 --2- 172.21.15.189:0/3011203078 >> v2:172.21.15.192:3300/0 conn(0x7f247c0ce730 0x7f247c15dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:08.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.516+0000 7f247a7fc700 1 -- 172.21.15.189:0/3011203078 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067046223 0 0) 0x7f247c15eac0 con 0x7f247c15d6f0 2024-04-06T12:31:08.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.516+0000 7f247a7fc700 1 -- 172.21.15.189:0/3011203078 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24640035b0 con 0x7f247c15d6f0 2024-04-06T12:31:08.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.516+0000 7f247affd700 1 -- 172.21.15.189:0/3011203078 >> v1:172.21.15.192:6789/0 conn(0x7f247c0b8d70 legacy=0x7f247c0d1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:08.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.516+0000 7f247affd700 1 -- 172.21.15.189:0/3011203078 >> v1:172.21.15.189:6789/0 conn(0x7f247c15d6f0 legacy=0x7f247c0ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:08.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.516+0000 7f247affd700 1 -- 172.21.15.189:0/3011203078 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247c15e500 con 0x7f247c0ce730 2024-04-06T12:31:08.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.516+0000 7f247affd700 1 --2- 172.21.15.189:0/3011203078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f247c0ce730 0x7f247c15dad0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f246c0097c0 tx=0x7f246c009ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b10353cfdeff4d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:08.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.516+0000 7f247a7fc700 1 -- 172.21.15.189:0/3011203078 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c004230 con 0x7f247c0ce730 2024-04-06T12:31:08.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.516+0000 7f247a7fc700 1 -- 172.21.15.189:0/3011203078 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246c004390 con 0x7f247c0ce730 2024-04-06T12:31:08.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.516+0000 7f247a7fc700 1 -- 172.21.15.189:0/3011203078 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c0036a0 con 0x7f247c0ce730 2024-04-06T12:31:08.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.517+0000 7f2481cc8700 1 -- 172.21.15.189:0/3011203078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f247c0ce730 msgr2=0x7f247c15dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:08.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.517+0000 7f2481cc8700 1 --2- 172.21.15.189:0/3011203078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f247c0ce730 0x7f247c15dad0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f246c0097c0 tx=0x7f246c009ad0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.517+0000 7f2481cc8700 1 -- 172.21.15.189:0/3011203078 shutdown_connections 2024-04-06T12:31:08.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.517+0000 7f2481cc8700 1 --2- 172.21.15.189:0/3011203078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f247c0ce730 0x7f247c15dad0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.517+0000 7f2481cc8700 1 -- 172.21.15.189:0/3011203078 >> 172.21.15.189:0/3011203078 conn(0x7f247c072490 msgr2=0x7f247c06b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:08.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.517+0000 7f2481cc8700 1 -- 172.21.15.189:0/3011203078 shutdown_connections 2024-04-06T12:31:08.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.517+0000 7f2481cc8700 1 -- 172.21.15.189:0/3011203078 wait complete. 2024-04-06T12:31:08.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.518+0000 7f2481cc8700 1 Processor -- start 2024-04-06T12:31:08.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.518+0000 7f2481cc8700 1 -- start start 2024-04-06T12:31:08.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.519+0000 7f2481cc8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f247c0b8d70 0x7f247c0b52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:08.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.519+0000 7f2481cc8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f247c0ce730 0x7f247c0b5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:08.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.519+0000 7f2481cc8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f247c15d6f0 0x7f247c15a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:08.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.519+0000 7f2481cc8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247c158170 con 0x7f247c0b8d70 2024-04-06T12:31:08.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.519+0000 7f2481cc8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247c15ba80 con 0x7f247c0ce730 2024-04-06T12:31:08.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.519+0000 7f2481cc8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247c15ae70 con 0x7f247c15d6f0 2024-04-06T12:31:08.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.519+0000 7f247b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f247c0b8d70 0x7f247c0b52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:08.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.519+0000 7f247b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f247c0b8d70 0x7f247c0b52d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36942/0 (socket says 172.21.15.189:36942) 2024-04-06T12:31:08.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.519+0000 7f247b7fe700 1 -- 172.21.15.189:0/3511580785 learned_addr learned my addr 172.21.15.189:0/3511580785 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:08.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.519+0000 7f247affd700 1 --2- 172.21.15.189:0/3511580785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f247c0ce730 0x7f247c0b5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:08.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.519+0000 7f247bfff700 1 --2- 172.21.15.189:0/3511580785 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f247c15d6f0 0x7f247c15a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:08.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.519+0000 7f247b7fe700 1 -- 172.21.15.189:0/3511580785 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f247c15d6f0 msgr2=0x7f247c15a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:08.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.519+0000 7f247b7fe700 1 --2- 172.21.15.189:0/3511580785 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f247c15d6f0 0x7f247c15a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.519+0000 7f247b7fe700 1 -- 172.21.15.189:0/3511580785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f247c0ce730 msgr2=0x7f247c0b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:08.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.519+0000 7f247b7fe700 1 --2- 172.21.15.189:0/3511580785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f247c0ce730 0x7f247c0b5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.519+0000 7f247b7fe700 1 -- 172.21.15.189:0/3511580785 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246c0094a0 con 0x7f247c0b8d70 2024-04-06T12:31:08.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.519+0000 7f247b7fe700 1 --2- 172.21.15.189:0/3511580785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f247c0b8d70 0x7f247c0b52d0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f2470001eb0 tx=0x7f247000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:08.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.519+0000 7f2463fff700 1 -- 172.21.15.189:0/3511580785 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2470005d70 con 0x7f247c0b8d70 2024-04-06T12:31:08.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.520+0000 7f2481cc8700 1 -- 172.21.15.189:0/3511580785 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f247c15b150 con 0x7f247c0b8d70 2024-04-06T12:31:08.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.520+0000 7f2463fff700 1 -- 172.21.15.189:0/3511580785 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24700070d0 con 0x7f247c0b8d70 2024-04-06T12:31:08.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.520+0000 7f2463fff700 1 -- 172.21.15.189:0/3511580785 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2470007460 con 0x7f247c0b8d70 2024-04-06T12:31:08.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.520+0000 7f2481cc8700 1 -- 172.21.15.189:0/3511580785 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247c15b710 con 0x7f247c0b8d70 2024-04-06T12:31:08.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.520+0000 7f2481cc8700 1 -- 172.21.15.189:0/3511580785 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f245c0052f0 con 0x7f247c0b8d70 2024-04-06T12:31:08.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.521+0000 7f2463fff700 1 -- 172.21.15.189:0/3511580785 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 72) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f24700075c0 con 0x7f247c0b8d70 2024-04-06T12:31:08.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.522+0000 7f2463fff700 1 -- 172.21.15.189:0/3511580785 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f24700886e0 con 0x7f247c0b8d70 2024-04-06T12:31:08.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.526+0000 7f2463fff700 1 -- 172.21.15.189:0/3511580785 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2470053790 con 0x7f247c0b8d70 2024-04-06T12:31:08.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.724+0000 7f2481cc8700 1 -- 172.21.15.189:0/3511580785 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f245c0055a0 con 0x7f247c0b8d70 2024-04-06T12:31:08.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.725+0000 7f2463fff700 1 -- 172.21.15.189:0/3511580785 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+17362 (secure 0 0 0) 0x7f2470056db0 con 0x7f247c0b8d70 2024-04-06T12:31:08.722 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:08.722 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":131,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:31:08.430396+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:30:14.958335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-06T12:30:16.104585+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":130,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/3321271257":"2024-04-07T12:31:08.430330+0000","172.21.15.189:0/1720782798":"2024-04-07T12:31:08.430330+0000","172.21.15.189:0/1827318961":"2024-04-07T12:31:08.430330+0000","172.21.15.189:0/1722722952":"2024-04-07T12:31:08.430330+0000","172.21.15.189:0/647042888":"2024-04-07T12:31:08.430330+0000","172.21.15.189:0/3715480594":"2024-04-07T12:31:08.430330+0000","172.21.15.189:0/1176667678":"2024-04-07T12:31:08.430330+0000","172.21.15.189:0/4170508796":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/169651662":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/29183104":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/1762502349":"2024-04-07T12:31:08.430330+0000","172.21.15.189:0/3994101871":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/1300227922":"2024-04-07T12:30:31.084606+0000","172.21.15.189:6833/47128":"2024-04-07T12:30:31.084606+0000","172.21.15.189:6832/47128":"2024-04-07T12:30:31.084606+0000","172.21.15.189:0/1086259216":"2024-04-07T12:30:31.084606+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:31:08.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.727+0000 7f2481cc8700 1 -- 172.21.15.189:0/3511580785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f247c0b8d70 msgr2=0x7f247c0b52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:08.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.727+0000 7f2481cc8700 1 --2- 172.21.15.189:0/3511580785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f247c0b8d70 0x7f247c0b52d0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f2470001eb0 tx=0x7f247000c7b0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.727+0000 7f2481cc8700 1 -- 172.21.15.189:0/3511580785 shutdown_connections 2024-04-06T12:31:08.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.727+0000 7f2481cc8700 1 --2- 172.21.15.189:0/3511580785 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f247c15d6f0 0x7f247c15a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.727+0000 7f2481cc8700 1 --2- 172.21.15.189:0/3511580785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f247c0b8d70 0x7f247c0b52d0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.727+0000 7f2481cc8700 1 --2- 172.21.15.189:0/3511580785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f247c0ce730 0x7f247c0b5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.727+0000 7f2481cc8700 1 -- 172.21.15.189:0/3511580785 >> 172.21.15.189:0/3511580785 conn(0x7f247c072490 msgr2=0x7f247c06b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:08.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.727+0000 7f2481cc8700 1 -- 172.21.15.189:0/3511580785 shutdown_connections 2024-04-06T12:31:08.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.727+0000 7f2481cc8700 1 -- 172.21.15.189:0/3511580785 wait complete. 2024-04-06T12:31:08.736 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-06T12:31:08.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.902+0000 7fe60a0e4700 1 Processor -- start 2024-04-06T12:31:08.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.902+0000 7fe60a0e4700 1 -- start start 2024-04-06T12:31:08.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.902+0000 7fe60a0e4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe60406ae40 0x7fe6040b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:08.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.902+0000 7fe60a0e4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe60415ddf0 0x7fe60415e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:08.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.902+0000 7fe60a0e4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6040b3990 con 0x7fe60415d7b0 2024-04-06T12:31:08.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.902+0000 7fe60a0e4700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6040b3060 con 0x7fe60406ae40 2024-04-06T12:31:08.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.902+0000 7fe60a0e4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6040b3340 con 0x7fe60415ddf0 2024-04-06T12:31:08.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.903+0000 7fe602ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe60406ae40 0x7fe6040b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:08.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.903+0000 7fe602ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe60406ae40 0x7fe6040b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36946/0 (socket says 172.21.15.189:36946) 2024-04-06T12:31:08.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.903+0000 7fe6037fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe60415d7b0 0x7fe60406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45384/0 (socket says 172.21.15.189:45384) 2024-04-06T12:31:08.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.903+0000 7fe602ffd700 1 -- 172.21.15.189:0/1669185153 learned_addr learned my addr 172.21.15.189:0/1669185153 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:08.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.903+0000 7fe603fff700 1 --2- 172.21.15.189:0/1669185153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe60415ddf0 0x7fe60415e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:08.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.903+0000 7fe602ffd700 1 -- 172.21.15.189:0/1669185153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe60415ddf0 msgr2=0x7fe60415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:08.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.903+0000 7fe602ffd700 1 --2- 172.21.15.189:0/1669185153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe60415ddf0 0x7fe60415e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.903+0000 7fe602ffd700 1 -- 172.21.15.189:0/1669185153 >> v1:172.21.15.189:6789/0 conn(0x7fe60415d7b0 legacy=0x7fe60406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:08.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.903+0000 7fe602ffd700 1 -- 172.21.15.189:0/1669185153 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6040b3620 con 0x7fe60406ae40 2024-04-06T12:31:08.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.903+0000 7fe6027fc700 1 -- 172.21.15.189:0/1669185153 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430427746 0 0) 0x7fe6040b3990 con 0x7fe60415d7b0 2024-04-06T12:31:08.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.903+0000 7fe603fff700 1 --2- 172.21.15.189:0/1669185153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe60415ddf0 0x7fe60415e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:08.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.903+0000 7fe602ffd700 1 --2- 172.21.15.189:0/1669185153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe60406ae40 0x7fe6040b3e20 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80098f0 tx=0x7fe5f8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0cedc7656c9e33 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:08.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.903+0000 7fe6027fc700 1 -- 172.21.15.189:0/1669185153 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f800b8a0 con 0x7fe60406ae40 2024-04-06T12:31:08.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.903+0000 7fe6027fc700 1 -- 172.21.15.189:0/1669185153 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f800ba00 con 0x7fe60406ae40 2024-04-06T12:31:08.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.903+0000 7fe6027fc700 1 -- 172.21.15.189:0/1669185153 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f800bd90 con 0x7fe60406ae40 2024-04-06T12:31:08.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.904+0000 7fe60a0e4700 1 -- 172.21.15.189:0/1669185153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe60406ae40 msgr2=0x7fe6040b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:08.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.904+0000 7fe60a0e4700 1 --2- 172.21.15.189:0/1669185153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe60406ae40 0x7fe6040b3e20 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80098f0 tx=0x7fe5f8009c00 comp rx=0 tx=0).stop 2024-04-06T12:31:08.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.904+0000 7fe60a0e4700 1 -- 172.21.15.189:0/1669185153 shutdown_connections 2024-04-06T12:31:08.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.904+0000 7fe60a0e4700 1 --2- 172.21.15.189:0/1669185153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe60415ddf0 0x7fe60415e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.904+0000 7fe60a0e4700 1 --2- 172.21.15.189:0/1669185153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe60406ae40 0x7fe6040b3e20 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.904+0000 7fe60a0e4700 1 -- 172.21.15.189:0/1669185153 >> 172.21.15.189:0/1669185153 conn(0x7fe6040d00b0 msgr2=0x7fe6040d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:08.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.904+0000 7fe60a0e4700 1 -- 172.21.15.189:0/1669185153 shutdown_connections 2024-04-06T12:31:08.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.904+0000 7fe60a0e4700 1 -- 172.21.15.189:0/1669185153 wait complete. 2024-04-06T12:31:08.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.905+0000 7fe60a0e4700 1 Processor -- start 2024-04-06T12:31:08.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.905+0000 7fe60a0e4700 1 -- start start 2024-04-06T12:31:08.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.906+0000 7fe60a0e4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe60406ae40 0x7fe60414f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:08.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.906+0000 7fe60a0e4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe60415d7b0 0x7fe6040c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:08.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.906+0000 7fe60a0e4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe60415ddf0 0x7fe6040c5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:08.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.906+0000 7fe60a0e4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6040cbf30 con 0x7fe60415ddf0 2024-04-06T12:31:08.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.906+0000 7fe60a0e4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6040b1b50 con 0x7fe60406ae40 2024-04-06T12:31:08.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.906+0000 7fe60a0e4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6040cdd70 con 0x7fe60415d7b0 2024-04-06T12:31:08.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.906+0000 7fe603fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe60415ddf0 0x7fe6040c5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:08.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.906+0000 7fe603fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe60415ddf0 0x7fe6040c5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36962/0 (socket says 172.21.15.189:36962) 2024-04-06T12:31:08.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.906+0000 7fe603fff700 1 -- 172.21.15.189:0/3773679227 learned_addr learned my addr 172.21.15.189:0/3773679227 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:08.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.906+0000 7fe602ffd700 1 --2- 172.21.15.189:0/3773679227 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe60415d7b0 0x7fe6040c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:08.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.906+0000 7fe6037fe700 1 --2- 172.21.15.189:0/3773679227 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe60406ae40 0x7fe60414f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:08.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.906+0000 7fe603fff700 1 -- 172.21.15.189:0/3773679227 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe60415d7b0 msgr2=0x7fe6040c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:08.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.906+0000 7fe603fff700 1 --2- 172.21.15.189:0/3773679227 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe60415d7b0 0x7fe6040c4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.906+0000 7fe603fff700 1 -- 172.21.15.189:0/3773679227 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe60406ae40 msgr2=0x7fe60414f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:08.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.906+0000 7fe603fff700 1 --2- 172.21.15.189:0/3773679227 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe60406ae40 0x7fe60414f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:08.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.906+0000 7fe603fff700 1 -- 172.21.15.189:0/3773679227 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f8009550 con 0x7fe60415ddf0 2024-04-06T12:31:08.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.906+0000 7fe6037fe700 1 --2- 172.21.15.189:0/3773679227 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe60406ae40 0x7fe60414f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:08.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.907+0000 7fe603fff700 1 --2- 172.21.15.189:0/3773679227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe60415ddf0 0x7fe6040c5470 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc00cd50 tx=0x7fe5fc00a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:08.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.907+0000 7fe5ebfff700 1 -- 172.21.15.189:0/3773679227 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc00cea0 con 0x7fe60415ddf0 2024-04-06T12:31:08.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.907+0000 7fe5ebfff700 1 -- 172.21.15.189:0/3773679227 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5fc00f480 con 0x7fe60415ddf0 2024-04-06T12:31:08.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.907+0000 7fe5ebfff700 1 -- 172.21.15.189:0/3773679227 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc00f830 con 0x7fe60415ddf0 2024-04-06T12:31:08.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.907+0000 7fe60a0e4700 1 -- 172.21.15.189:0/3773679227 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60414fa90 con 0x7fe60415ddf0 2024-04-06T12:31:08.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.907+0000 7fe60a0e4700 1 -- 172.21.15.189:0/3773679227 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6040cd720 con 0x7fe60415ddf0 2024-04-06T12:31:08.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.908+0000 7fe60a0e4700 1 -- 172.21.15.189:0/3773679227 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe604067bb0 con 0x7fe60415ddf0 2024-04-06T12:31:08.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.908+0000 7fe5ebfff700 1 -- 172.21.15.189:0/3773679227 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 72) v1 ==== 88856+0+0 (secure 0 0 0) 0x7fe5fc00b040 con 0x7fe60415ddf0 2024-04-06T12:31:08.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.909+0000 7fe5ebfff700 1 -- 172.21.15.189:0/3773679227 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fe5fc0898b0 con 0x7fe60415ddf0 2024-04-06T12:31:08.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:08.913+0000 7fe5ebfff700 1 -- 172.21.15.189:0/3773679227 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe5fc054ad0 con 0x7fe60415ddf0 2024-04-06T12:31:09.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.151+0000 7fe60a0e4700 1 -- 172.21.15.189:0/3773679227 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe6040cdee0 con 0x7fe60415ddf0 2024-04-06T12:31:09.149 INFO:tasks.ceph.mon.a.smithi189.stderr:2024-04-06T12:31:09.152+0000 7fa9fe1ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T12:31:09.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.164+0000 7fe5ebfff700 1 -- 172.21.15.189:0/3773679227 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v55) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe5fc01d090 con 0x7fe60415ddf0 2024-04-06T12:31:09.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.166+0000 7fe60a0e4700 1 -- 172.21.15.189:0/3773679227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe60415ddf0 msgr2=0x7fe6040c5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:09.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.166+0000 7fe60a0e4700 1 --2- 172.21.15.189:0/3773679227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe60415ddf0 0x7fe6040c5470 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc00cd50 tx=0x7fe5fc00a560 comp rx=0 tx=0).stop 2024-04-06T12:31:09.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.166+0000 7fe60a0e4700 1 -- 172.21.15.189:0/3773679227 shutdown_connections 2024-04-06T12:31:09.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.166+0000 7fe60a0e4700 1 --2- 172.21.15.189:0/3773679227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe60415ddf0 0x7fe6040c5470 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:09.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.166+0000 7fe60a0e4700 1 --2- 172.21.15.189:0/3773679227 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe60406ae40 0x7fe60414f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:09.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.166+0000 7fe60a0e4700 1 --2- 172.21.15.189:0/3773679227 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe60415d7b0 0x7fe6040c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:09.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.166+0000 7fe60a0e4700 1 -- 172.21.15.189:0/3773679227 >> 172.21.15.189:0/3773679227 conn(0x7fe6040d00b0 msgr2=0x7fe6040ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:09.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.166+0000 7fe60a0e4700 1 -- 172.21.15.189:0/3773679227 shutdown_connections 2024-04-06T12:31:09.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.166+0000 7fe60a0e4700 1 -- 172.21.15.189:0/3773679227 wait complete. 2024-04-06T12:31:09.164 INFO:teuthology.orchestra.run.smithi189.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-06T12:31:09.165 INFO:tasks.ceph.mds.c.smithi189.stderr: -9178> 2024-04-06T12:30:22.708+0000 7f6a24e1c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T12:31:09.176 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-06T12:31:09.199 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T12:31:09.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.307+0000 7f4b0d887700 1 Processor -- start 2024-04-06T12:31:09.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.307+0000 7f4b0d887700 1 -- start start 2024-04-06T12:31:09.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.307+0000 7f4b0d887700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4b080b4660 0x7f4b080b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:09.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.307+0000 7f4b0d887700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b080b2fc0 0x7f4b080b3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:09.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.307+0000 7f4b0d887700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b080b5030 con 0x7f4b0815a550 2024-04-06T12:31:09.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.307+0000 7f4b0d887700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b080cdbe0 con 0x7f4b080b4660 2024-04-06T12:31:09.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.307+0000 7f4b0d887700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0815a940 con 0x7f4b080b2fc0 2024-04-06T12:31:09.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.308+0000 7f4b067fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4b080b4660 0x7f4b080b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:09.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.308+0000 7f4b067fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4b080b4660 0x7f4b080b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36262/0 (socket says 172.21.15.189:36262) 2024-04-06T12:31:09.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.308+0000 7f4b067fc700 1 -- 172.21.15.189:0/1445361081 learned_addr learned my addr 172.21.15.189:0/1445361081 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:09.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.308+0000 7f4b077fe700 1 --2- 172.21.15.189:0/1445361081 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b080b2fc0 0x7f4b080b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:09.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.308+0000 7f4b067fc700 1 -- 172.21.15.189:0/1445361081 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b080b2fc0 msgr2=0x7f4b080b3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:09.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.308+0000 7f4b067fc700 1 --2- 172.21.15.189:0/1445361081 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b080b2fc0 0x7f4b080b3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:09.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.308+0000 7f4b067fc700 1 -- 172.21.15.189:0/1445361081 >> v1:172.21.15.192:6789/0 conn(0x7f4b0815a550 legacy=0x7f4b080cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:09.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.308+0000 7f4b067fc700 1 -- 172.21.15.189:0/1445361081 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0815ac00 con 0x7f4b080b4660 2024-04-06T12:31:09.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.308+0000 7f4b067fc700 1 --2- 172.21.15.189:0/1445361081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b080b4660 0x7f4b080b4af0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f4af000bb90 tx=0x7f4af000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73618498410c1c2e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:09.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.308+0000 7f4b05ffb700 1 -- 172.21.15.189:0/1445361081 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af000dd30 con 0x7f4b080b4660 2024-04-06T12:31:09.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.308+0000 7f4b05ffb700 1 -- 172.21.15.189:0/1445361081 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af000de90 con 0x7f4b080b4660 2024-04-06T12:31:09.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.308+0000 7f4b05ffb700 1 -- 172.21.15.189:0/1445361081 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af0010690 con 0x7f4b080b4660 2024-04-06T12:31:09.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.309+0000 7f4b0d887700 1 -- 172.21.15.189:0/1445361081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b080b4660 msgr2=0x7f4b080b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:09.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.309+0000 7f4b0d887700 1 --2- 172.21.15.189:0/1445361081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b080b4660 0x7f4b080b4af0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f4af000bb90 tx=0x7f4af000bea0 comp rx=0 tx=0).stop 2024-04-06T12:31:09.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.309+0000 7f4b0d887700 1 -- 172.21.15.189:0/1445361081 shutdown_connections 2024-04-06T12:31:09.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.309+0000 7f4b0d887700 1 --2- 172.21.15.189:0/1445361081 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b080b2fc0 0x7f4b080b3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:09.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.309+0000 7f4b0d887700 1 --2- 172.21.15.189:0/1445361081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b080b4660 0x7f4b080b4af0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:09.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.309+0000 7f4b0d887700 1 -- 172.21.15.189:0/1445361081 >> 172.21.15.189:0/1445361081 conn(0x7f4b080d00b0 msgr2=0x7f4b080d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:09.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.309+0000 7f4b0d887700 1 -- 172.21.15.189:0/1445361081 shutdown_connections 2024-04-06T12:31:09.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.309+0000 7f4b0d887700 1 -- 172.21.15.189:0/1445361081 wait complete. 2024-04-06T12:31:09.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.310+0000 7f4b0d887700 1 Processor -- start 2024-04-06T12:31:09.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.310+0000 7f4b0d887700 1 -- start start 2024-04-06T12:31:09.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.311+0000 7f4b0d887700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b080b2fc0 0x7f4b081537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:09.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.311+0000 7f4b0d887700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b080b4660 0x7f4b08156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:09.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.311+0000 7f4b0d887700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b080a9bb0 0x7f4b081f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:09.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.311+0000 7f4b0d887700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b080b00d0 con 0x7f4b080a9bb0 2024-04-06T12:31:09.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.311+0000 7f4b0d887700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b08157b10 con 0x7f4b080b2fc0 2024-04-06T12:31:09.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.311+0000 7f4b0d887700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b08157550 con 0x7f4b080b4660 2024-04-06T12:31:09.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.311+0000 7f4b077fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b080a9bb0 0x7f4b081f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:09.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.311+0000 7f4b077fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b080a9bb0 0x7f4b081f1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36278/0 (socket says 172.21.15.189:36278) 2024-04-06T12:31:09.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.311+0000 7f4b077fe700 1 -- 172.21.15.189:0/1687827401 learned_addr learned my addr 172.21.15.189:0/1687827401 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:09.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.311+0000 7f4b067fc700 1 --2- 172.21.15.189:0/1687827401 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b080b4660 0x7f4b08156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:09.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.311+0000 7f4b077fe700 1 -- 172.21.15.189:0/1687827401 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b080b4660 msgr2=0x7f4b08156c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:09.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.311+0000 7f4b077fe700 1 --2- 172.21.15.189:0/1687827401 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b080b4660 0x7f4b08156c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:09.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.311+0000 7f4b06ffd700 1 --2- 172.21.15.189:0/1687827401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b080b2fc0 0x7f4b081537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:09.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.311+0000 7f4b077fe700 1 -- 172.21.15.189:0/1687827401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b080b2fc0 msgr2=0x7f4b081537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:09.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.311+0000 7f4b077fe700 1 --2- 172.21.15.189:0/1687827401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b080b2fc0 0x7f4b081537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:09.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.311+0000 7f4b077fe700 1 -- 172.21.15.189:0/1687827401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af000b7f0 con 0x7f4b080a9bb0 2024-04-06T12:31:09.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.311+0000 7f4b06ffd700 1 --2- 172.21.15.189:0/1687827401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b080b2fc0 0x7f4b081537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:09.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.312+0000 7f4b077fe700 1 --2- 172.21.15.189:0/1687827401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b080a9bb0 0x7f4b081f1da0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f4af800e7d0 tx=0x7f4af800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:09.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.312+0000 7f4afffff700 1 -- 172.21.15.189:0/1687827401 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af80095b0 con 0x7f4b080a9bb0 2024-04-06T12:31:09.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.312+0000 7f4afffff700 1 -- 172.21.15.189:0/1687827401 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af8013680 con 0x7f4b080a9bb0 2024-04-06T12:31:09.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.312+0000 7f4b0d887700 1 -- 172.21.15.189:0/1687827401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b08157830 con 0x7f4b080a9bb0 2024-04-06T12:31:09.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.312+0000 7f4afffff700 1 -- 172.21.15.189:0/1687827401 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af8013a10 con 0x7f4b080a9bb0 2024-04-06T12:31:09.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.312+0000 7f4b0d887700 1 -- 172.21.15.189:0/1687827401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b08153d00 con 0x7f4b080a9bb0 2024-04-06T12:31:09.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.313+0000 7f4b0d887700 1 -- 172.21.15.189:0/1687827401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b08067bb0 con 0x7f4b080a9bb0 2024-04-06T12:31:09.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.313+0000 7f4afffff700 1 -- 172.21.15.189:0/1687827401 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 72) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f4af8013b70 con 0x7f4b080a9bb0 2024-04-06T12:31:09.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.313+0000 7f4afffff700 1 -- 172.21.15.189:0/1687827401 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f4af8089200 con 0x7f4b080a9bb0 2024-04-06T12:31:09.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.318+0000 7f4afffff700 1 -- 172.21.15.189:0/1687827401 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4af80543d0 con 0x7f4b080a9bb0 2024-04-06T12:31:09.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.439+0000 7f4afffff700 1 -- 172.21.15.189:0/1687827401 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 73) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f4af8053e80 con 0x7f4b080a9bb0 2024-04-06T12:31:09.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.440+0000 7f4afffff700 1 --2- 172.21.15.189:0/1687827401 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4ae8036150 0x7f4ae8072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:09.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.440+0000 7f4b06ffd700 1 --2- 172.21.15.189:0/1687827401 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4ae8036150 0x7f4ae8072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:09.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.441+0000 7f4b06ffd700 1 --2- 172.21.15.189:0/1687827401 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4ae8036150 0x7f4ae8072a90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4b000012e0 tx=0x7f4b00009040 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:09.565+0000 7f4b0d887700 1 -- 172.21.15.189:0/1687827401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4b08074990 con 0x7f4b080a9bb0 2024-04-06T12:31:10.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.166+0000 7f4afffff700 1 -- 172.21.15.189:0/1687827401 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v56) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4af8087370 con 0x7f4b080a9bb0 2024-04-06T12:31:10.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.169+0000 7f4b0d887700 1 -- 172.21.15.189:0/1687827401 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4ae8036150 msgr2=0x7f4ae8072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:10.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.169+0000 7f4b0d887700 1 --2- 172.21.15.189:0/1687827401 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4ae8036150 0x7f4ae8072a90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4b000012e0 tx=0x7f4b00009040 comp rx=0 tx=0).stop 2024-04-06T12:31:10.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.169+0000 7f4b0d887700 1 -- 172.21.15.189:0/1687827401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b080a9bb0 msgr2=0x7f4b081f1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:10.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.169+0000 7f4b0d887700 1 --2- 172.21.15.189:0/1687827401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b080a9bb0 0x7f4b081f1da0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f4af800e7d0 tx=0x7f4af800eb90 comp rx=0 tx=0).stop 2024-04-06T12:31:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.169+0000 7f4b0d887700 1 -- 172.21.15.189:0/1687827401 shutdown_connections 2024-04-06T12:31:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.169+0000 7f4b0d887700 1 --2- 172.21.15.189:0/1687827401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b080a9bb0 0x7f4b081f1da0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.169+0000 7f4b0d887700 1 --2- 172.21.15.189:0/1687827401 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4ae8036150 0x7f4ae8072a90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.169+0000 7f4b0d887700 1 --2- 172.21.15.189:0/1687827401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b080b2fc0 0x7f4b081537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.169+0000 7f4b0d887700 1 --2- 172.21.15.189:0/1687827401 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b080b4660 0x7f4b08156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.169+0000 7f4b0d887700 1 -- 172.21.15.189:0/1687827401 >> 172.21.15.189:0/1687827401 conn(0x7f4b080d00b0 msgr2=0x7f4b08157cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.169+0000 7f4b0d887700 1 -- 172.21.15.189:0/1687827401 shutdown_connections 2024-04-06T12:31:10.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.169+0000 7f4b0d887700 1 -- 172.21.15.189:0/1687827401 wait complete. 2024-04-06T12:31:10.181 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-06T12:31:10.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.356+0000 7f7192d65700 1 Processor -- start 2024-04-06T12:31:10.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.356+0000 7f7192d65700 1 -- start start 2024-04-06T12:31:10.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.357+0000 7f7192d65700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f718c0d11b0 0x7f718c0d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:10.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.357+0000 7f7192d65700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f718c0cfa00 0x7f718c0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:10.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.357+0000 7f7192d65700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f718c0d1ad0 con 0x7f718c0cb870 2024-04-06T12:31:10.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.357+0000 7f7192d65700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f718c0ccdc0 con 0x7f718c0d11b0 2024-04-06T12:31:10.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.357+0000 7f7192d65700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f718c0d0b50 con 0x7f718c0cfa00 2024-04-06T12:31:10.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.357+0000 7f718bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f718c0cb870 0x7f718c15c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55902/0 (socket says 172.21.15.189:55902) 2024-04-06T12:31:10.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.357+0000 7f718bfff700 1 -- 172.21.15.189:0/2883031544 learned_addr learned my addr 172.21.15.189:0/2883031544 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:10.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.357+0000 7f7190b01700 1 --2- 172.21.15.189:0/2883031544 >> v2:172.21.15.192:3300/0 conn(0x7f718c0d11b0 0x7f718c0d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:10.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.357+0000 7f7191302700 1 --2- 172.21.15.189:0/2883031544 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f718c0cfa00 0x7f718c0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:10.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.357+0000 7f718b7fe700 1 -- 172.21.15.189:0/2883031544 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882922548 0 0) 0x7f718c0d1ad0 con 0x7f718c0cb870 2024-04-06T12:31:10.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.357+0000 7f718b7fe700 1 -- 172.21.15.189:0/2883031544 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f717c0035b0 con 0x7f718c0cb870 2024-04-06T12:31:10.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.358+0000 7f7190b01700 1 -- 172.21.15.189:0/2883031544 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f718c0cfa00 msgr2=0x7f718c0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:10.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.358+0000 7f7190b01700 1 --2- 172.21.15.189:0/2883031544 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f718c0cfa00 0x7f718c0cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:10.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.358+0000 7f718b7fe700 1 -- 172.21.15.189:0/2883031544 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 702982855 0 0) 0x7f717c0035b0 con 0x7f718c0cb870 2024-04-06T12:31:10.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.358+0000 7f7190b01700 1 -- 172.21.15.189:0/2883031544 >> v1:172.21.15.189:6789/0 conn(0x7f718c0cb870 legacy=0x7f718c15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:10.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.358+0000 7f7190b01700 1 -- 172.21.15.189:0/2883031544 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f718c0cbd80 con 0x7f718c0d11b0 2024-04-06T12:31:10.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.358+0000 7f7190b01700 1 --2- 172.21.15.189:0/2883031544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f718c0d11b0 0x7f718c0d1590 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f7180009840 tx=0x7f7180009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34ee14a5216426f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:10.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.358+0000 7f718b7fe700 1 -- 172.21.15.189:0/2883031544 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7178001670 con 0x7f718c0d11b0 2024-04-06T12:31:10.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.358+0000 7f718b7fe700 1 -- 172.21.15.189:0/2883031544 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7180004030 con 0x7f718c0d11b0 2024-04-06T12:31:10.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.358+0000 7f718b7fe700 1 -- 172.21.15.189:0/2883031544 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71800043c0 con 0x7f718c0d11b0 2024-04-06T12:31:10.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.359+0000 7f7192d65700 1 -- 172.21.15.189:0/2883031544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f718c0d11b0 msgr2=0x7f718c0d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:10.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.359+0000 7f7192d65700 1 --2- 172.21.15.189:0/2883031544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f718c0d11b0 0x7f718c0d1590 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f7180009840 tx=0x7f7180009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:10.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.359+0000 7f7192d65700 1 -- 172.21.15.189:0/2883031544 shutdown_connections 2024-04-06T12:31:10.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.359+0000 7f7192d65700 1 --2- 172.21.15.189:0/2883031544 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f718c0cfa00 0x7f718c0cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:10.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.359+0000 7f7192d65700 1 --2- 172.21.15.189:0/2883031544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f718c0d11b0 0x7f718c0d1590 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:10.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.359+0000 7f7192d65700 1 -- 172.21.15.189:0/2883031544 >> 172.21.15.189:0/2883031544 conn(0x7f718c0b2db0 msgr2=0x7f718c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:10.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.359+0000 7f7192d65700 1 -- 172.21.15.189:0/2883031544 shutdown_connections 2024-04-06T12:31:10.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.359+0000 7f7192d65700 1 -- 172.21.15.189:0/2883031544 wait complete. 2024-04-06T12:31:10.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.360+0000 7f7192d65700 1 Processor -- start 2024-04-06T12:31:10.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.360+0000 7f7192d65700 1 -- start start 2024-04-06T12:31:10.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f7192d65700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f718c0cb870 0x7f718c15e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:10.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f7192d65700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f718c0cfa00 0x7f718c15e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:10.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f7192d65700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f718c0d11b0 0x7f718c0b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:10.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f7192d65700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718c1f1ed0 con 0x7f718c0cfa00 2024-04-06T12:31:10.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f7192d65700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718c15ee10 con 0x7f718c0cb870 2024-04-06T12:31:10.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f7192d65700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f718c15f0f0 con 0x7f718c0d11b0 2024-04-06T12:31:10.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f718bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f718c0cfa00 0x7f718c15e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:10.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f718bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f718c0cfa00 0x7f718c15e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36286/0 (socket says 172.21.15.189:36286) 2024-04-06T12:31:10.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f718bfff700 1 -- 172.21.15.189:0/4203059249 learned_addr learned my addr 172.21.15.189:0/4203059249 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:10.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f7190b01700 1 --2- 172.21.15.189:0/4203059249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f718c0cb870 0x7f718c15e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:10.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f7191302700 1 --2- 172.21.15.189:0/4203059249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f718c0d11b0 0x7f718c0b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:10.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f718bfff700 1 -- 172.21.15.189:0/4203059249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f718c0d11b0 msgr2=0x7f718c0b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:10.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f718bfff700 1 --2- 172.21.15.189:0/4203059249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f718c0d11b0 0x7f718c0b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:10.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f718bfff700 1 -- 172.21.15.189:0/4203059249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f718c0cb870 msgr2=0x7f718c15e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:10.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f718bfff700 1 --2- 172.21.15.189:0/4203059249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f718c0cb870 0x7f718c15e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:10.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f718bfff700 1 -- 172.21.15.189:0/4203059249 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71800094a0 con 0x7f718c0cfa00 2024-04-06T12:31:10.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f718bfff700 1 --2- 172.21.15.189:0/4203059249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f718c0cfa00 0x7f718c15e790 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f717800c750 tx=0x7f717800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:10.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f71897fa700 1 -- 172.21.15.189:0/4203059249 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7178004c80 con 0x7f718c0cfa00 2024-04-06T12:31:10.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f71897fa700 1 -- 172.21.15.189:0/4203059249 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7178004de0 con 0x7f718c0cfa00 2024-04-06T12:31:10.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f7192d65700 1 -- 172.21.15.189:0/4203059249 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f718c15f3d0 con 0x7f718c0cfa00 2024-04-06T12:31:10.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.361+0000 7f71897fa700 1 -- 172.21.15.189:0/4203059249 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71780072d0 con 0x7f718c0cfa00 2024-04-06T12:31:10.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.362+0000 7f7192d65700 1 -- 172.21.15.189:0/4203059249 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f718c14b2a0 con 0x7f718c0cfa00 2024-04-06T12:31:10.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.362+0000 7f7192d65700 1 -- 172.21.15.189:0/4203059249 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f718c1f22f0 con 0x7f718c0cfa00 2024-04-06T12:31:10.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.363+0000 7f71897fa700 1 -- 172.21.15.189:0/4203059249 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 73) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f7178007430 con 0x7f718c0cfa00 2024-04-06T12:31:10.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.363+0000 7f71897fa700 1 --2- 172.21.15.189:0/4203059249 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f717006aae0 0x7f717006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:10.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.363+0000 7f71897fa700 1 -- 172.21.15.189:0/4203059249 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f7178089180 con 0x7f718c0cfa00 2024-04-06T12:31:10.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.363+0000 7f7190b01700 1 --2- 172.21.15.189:0/4203059249 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f717006aae0 0x7f717006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:10.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.364+0000 7f7190b01700 1 --2- 172.21.15.189:0/4203059249 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f717006aae0 0x7f717006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f71800090b0 tx=0x7f7180008030 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:10.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.368+0000 7f71897fa700 1 -- 172.21.15.189:0/4203059249 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f718c1f22f0 con 0x7f718c0cfa00 2024-04-06T12:31:10.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:10.572+0000 7f7192d65700 1 -- 172.21.15.189:0/4203059249 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f718c1f22f0 con 0x7f718c0cfa00 2024-04-06T12:31:11.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.448+0000 7f71897fa700 1 -- 172.21.15.189:0/4203059249 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 74) v1 ==== 89191+0+0 (secure 0 0 0) 0x7f7178054540 con 0x7f718c0cfa00 2024-04-06T12:31:11.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.451+0000 7f71897fa700 1 -- 172.21.15.189:0/4203059249 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v133) v1 ==== 181+0+0 (secure 0 0 0) 0x7f7178072890 con 0x7f718c0cfa00 2024-04-06T12:31:11.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.454+0000 7f7192d65700 1 -- 172.21.15.189:0/4203059249 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f717006aae0 msgr2=0x7f717006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:11.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.454+0000 7f7192d65700 1 --2- 172.21.15.189:0/4203059249 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f717006aae0 0x7f717006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f71800090b0 tx=0x7f7180008030 comp rx=0 tx=0).stop 2024-04-06T12:31:11.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.454+0000 7f7192d65700 1 -- 172.21.15.189:0/4203059249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f718c0cfa00 msgr2=0x7f718c15e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:11.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.454+0000 7f7192d65700 1 --2- 172.21.15.189:0/4203059249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f718c0cfa00 0x7f718c15e790 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f717800c750 tx=0x7f717800ca60 comp rx=0 tx=0).stop 2024-04-06T12:31:11.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.454+0000 7f7192d65700 1 -- 172.21.15.189:0/4203059249 shutdown_connections 2024-04-06T12:31:11.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.454+0000 7f7192d65700 1 --2- 172.21.15.189:0/4203059249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f718c0d11b0 0x7f718c0b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:11.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.454+0000 7f7192d65700 1 --2- 172.21.15.189:0/4203059249 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f717006aae0 0x7f717006cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:11.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.454+0000 7f7192d65700 1 --2- 172.21.15.189:0/4203059249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f718c0cb870 0x7f718c15e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:11.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.454+0000 7f7192d65700 1 --2- 172.21.15.189:0/4203059249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f718c0cfa00 0x7f718c15e790 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:11.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.454+0000 7f7192d65700 1 -- 172.21.15.189:0/4203059249 >> 172.21.15.189:0/4203059249 conn(0x7f718c0b2db0 msgr2=0x7f718c15a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:11.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.454+0000 7f7192d65700 1 -- 172.21.15.189:0/4203059249 shutdown_connections 2024-04-06T12:31:11.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.454+0000 7f7192d65700 1 -- 172.21.15.189:0/4203059249 wait complete. 2024-04-06T12:31:11.453 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' removed 2024-04-06T12:31:11.465 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-06T12:31:11.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.599+0000 7fc4a0568700 1 Processor -- start 2024-04-06T12:31:11.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.599+0000 7fc4a0568700 1 -- start start 2024-04-06T12:31:11.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.600+0000 7fc4a0568700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4980cb870 0x7fc49815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:11.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.600+0000 7fc4a0568700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc4980cfa00 0x7fc4980cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:11.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.600+0000 7fc4a0568700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4980cbd80 con 0x7fc4980d11b0 2024-04-06T12:31:11.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.600+0000 7fc4a0568700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4980ccdc0 con 0x7fc4980cfa00 2024-04-06T12:31:11.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.600+0000 7fc4a0568700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4980d0b50 con 0x7fc4980cb870 2024-04-06T12:31:11.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.600+0000 7fc49db03700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4980cb870 0x7fc49815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:11.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.600+0000 7fc49eb05700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc4980cfa00 0x7fc4980cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:11.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.600+0000 7fc49db03700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4980cb870 0x7fc49815c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:40630/0 (socket says 172.21.15.189:40630) 2024-04-06T12:31:11.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.600+0000 7fc49db03700 1 -- 172.21.15.189:0/219018405 learned_addr learned my addr 172.21.15.189:0/219018405 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:11.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.600+0000 7fc49e304700 1 --1- 172.21.15.189:0/219018405 >> v1:172.21.15.192:6789/0 conn(0x7fc4980d11b0 0x7fc4980d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:59302/0 (socket says 172.21.15.189:59302) 2024-04-06T12:31:11.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.600+0000 7fc49db03700 1 -- 172.21.15.189:0/219018405 >> v2:172.21.15.192:3300/0 conn(0x7fc4980cfa00 msgr2=0x7fc4980cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:11.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.600+0000 7fc49db03700 1 --2- 172.21.15.189:0/219018405 >> v2:172.21.15.192:3300/0 conn(0x7fc4980cfa00 0x7fc4980cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:11.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.600+0000 7fc49db03700 1 -- 172.21.15.189:0/219018405 >> v1:172.21.15.192:6789/0 conn(0x7fc4980d11b0 legacy=0x7fc4980d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:11.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.600+0000 7fc49db03700 1 -- 172.21.15.189:0/219018405 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4980cef20 con 0x7fc4980cb870 2024-04-06T12:31:11.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.600+0000 7fc49eb05700 1 --2- 172.21.15.189:0/219018405 >> v2:172.21.15.192:3300/0 conn(0x7fc4980cfa00 0x7fc4980cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:11.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.600+0000 7fc49db03700 1 --2- 172.21.15.189:0/219018405 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4980cb870 0x7fc49815c280 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc4880098f0 tx=0x7fc488009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=946337c9aa3ca07f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:11.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.600+0000 7fc49d302700 1 -- 172.21.15.189:0/219018405 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc488011830 con 0x7fc4980cb870 2024-04-06T12:31:11.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.601+0000 7fc49d302700 1 -- 172.21.15.189:0/219018405 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc488011990 con 0x7fc4980cb870 2024-04-06T12:31:11.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.601+0000 7fc49d302700 1 -- 172.21.15.189:0/219018405 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc488011d20 con 0x7fc4980cb870 2024-04-06T12:31:11.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.601+0000 7fc4a0568700 1 -- 172.21.15.189:0/219018405 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4980cb870 msgr2=0x7fc49815c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:11.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.601+0000 7fc4a0568700 1 --2- 172.21.15.189:0/219018405 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4980cb870 0x7fc49815c280 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc4880098f0 tx=0x7fc488009c00 comp rx=0 tx=0).stop 2024-04-06T12:31:11.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.601+0000 7fc4a0568700 1 -- 172.21.15.189:0/219018405 shutdown_connections 2024-04-06T12:31:11.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.601+0000 7fc4a0568700 1 --2- 172.21.15.189:0/219018405 >> v2:172.21.15.192:3300/0 conn(0x7fc4980cfa00 0x7fc4980cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:11.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.601+0000 7fc4a0568700 1 --2- 172.21.15.189:0/219018405 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4980cb870 0x7fc49815c280 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:11.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.601+0000 7fc4a0568700 1 -- 172.21.15.189:0/219018405 >> 172.21.15.189:0/219018405 conn(0x7fc4980b2db0 msgr2=0x7fc4980b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:11.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.601+0000 7fc4a0568700 1 -- 172.21.15.189:0/219018405 shutdown_connections 2024-04-06T12:31:11.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.601+0000 7fc4a0568700 1 -- 172.21.15.189:0/219018405 wait complete. 2024-04-06T12:31:11.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.602+0000 7fc4a0568700 1 Processor -- start 2024-04-06T12:31:11.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.602+0000 7fc4a0568700 1 -- start start 2024-04-06T12:31:11.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.603+0000 7fc4a0568700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4980cb870 0x7fc498150840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:11.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.603+0000 7fc4a0568700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4980cfa00 0x7fc4981579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:11.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.603+0000 7fc4a0568700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4980d11b0 0x7fc49815f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:11.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.603+0000 7fc4a0568700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49815fb60 con 0x7fc4980cb870 2024-04-06T12:31:11.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.603+0000 7fc4a0568700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4980b4d30 con 0x7fc4980d11b0 2024-04-06T12:31:11.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.603+0000 7fc4a0568700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc498158980 con 0x7fc4980cfa00 2024-04-06T12:31:11.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.603+0000 7fc49e304700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4980cb870 0x7fc498150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:11.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.603+0000 7fc49e304700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4980cb870 0x7fc498150840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36318/0 (socket says 172.21.15.189:36318) 2024-04-06T12:31:11.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.603+0000 7fc49e304700 1 -- 172.21.15.189:0/811032011 learned_addr learned my addr 172.21.15.189:0/811032011 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:11.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.603+0000 7fc49db03700 1 --2- 172.21.15.189:0/811032011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4980cfa00 0x7fc4981579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:11.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.603+0000 7fc49eb05700 1 --2- 172.21.15.189:0/811032011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4980d11b0 0x7fc49815f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:11.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.603+0000 7fc49e304700 1 -- 172.21.15.189:0/811032011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4980cfa00 msgr2=0x7fc4981579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:11.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.603+0000 7fc49e304700 1 --2- 172.21.15.189:0/811032011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4980cfa00 0x7fc4981579e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:11.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.603+0000 7fc49e304700 1 -- 172.21.15.189:0/811032011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4980d11b0 msgr2=0x7fc49815f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:11.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.603+0000 7fc49e304700 1 --2- 172.21.15.189:0/811032011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4980d11b0 0x7fc49815f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:11.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.604+0000 7fc49e304700 1 -- 172.21.15.189:0/811032011 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc490009a70 con 0x7fc4980cb870 2024-04-06T12:31:11.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.604+0000 7fc49e304700 1 --2- 172.21.15.189:0/811032011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4980cb870 0x7fc498150840 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fc49400c500 tx=0x7fc49400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:11.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.604+0000 7fc48effd700 1 -- 172.21.15.189:0/811032011 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49400cea0 con 0x7fc4980cb870 2024-04-06T12:31:11.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.604+0000 7fc48effd700 1 -- 172.21.15.189:0/811032011 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc494005cb0 con 0x7fc4980cb870 2024-04-06T12:31:11.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.604+0000 7fc4a0568700 1 -- 172.21.15.189:0/811032011 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc488009550 con 0x7fc4980cb870 2024-04-06T12:31:11.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.604+0000 7fc48effd700 1 -- 172.21.15.189:0/811032011 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4940072d0 con 0x7fc4980cb870 2024-04-06T12:31:11.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.604+0000 7fc4a0568700 1 -- 172.21.15.189:0/811032011 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4981583c0 con 0x7fc4980cb870 2024-04-06T12:31:11.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.605+0000 7fc4a0568700 1 -- 172.21.15.189:0/811032011 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc498074990 con 0x7fc4980cb870 2024-04-06T12:31:11.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.605+0000 7fc48effd700 1 -- 172.21.15.189:0/811032011 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 74) v1 ==== 89191+0+0 (secure 0 0 0) 0x7fc494007430 con 0x7fc4980cb870 2024-04-06T12:31:11.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.606+0000 7fc48effd700 1 --2- 172.21.15.189:0/811032011 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc47c06ad20 0x7fc47c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:11.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.606+0000 7fc48effd700 1 -- 172.21.15.189:0/811032011 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6393+0+0 (secure 0 0 0) 0x7fc4940886d0 con 0x7fc4980cb870 2024-04-06T12:31:11.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.606+0000 7fc49db03700 1 --2- 172.21.15.189:0/811032011 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc47c06ad20 0x7fc47c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:11.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.606+0000 7fc49db03700 1 --2- 172.21.15.189:0/811032011 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc47c06ad20 0x7fc47c06d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc4980cbcb0 tx=0x7fc488008030 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:11.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.610+0000 7fc48effd700 1 -- 172.21.15.189:0/811032011 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc494053750 con 0x7fc4980cb870 2024-04-06T12:31:11.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:11.807+0000 7fc4a0568700 1 -- 172.21.15.189:0/811032011 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc4980d1e20 con 0x7fc4980cb870 2024-04-06T12:31:12.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.657+0000 7fc48effd700 1 -- 172.21.15.189:0/811032011 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fc494053b20 con 0x7fc4980cb870 2024-04-06T12:31:12.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.658+0000 7fc48effd700 1 -- 172.21.15.189:0/811032011 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v135) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc494050190 con 0x7fc4980cb870 2024-04-06T12:31:12.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.661+0000 7fc4a0568700 1 -- 172.21.15.189:0/811032011 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc47c06ad20 msgr2=0x7fc47c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:12.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.661+0000 7fc4a0568700 1 --2- 172.21.15.189:0/811032011 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc47c06ad20 0x7fc47c06d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc4980cbcb0 tx=0x7fc488008030 comp rx=0 tx=0).stop 2024-04-06T12:31:12.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.661+0000 7fc4a0568700 1 -- 172.21.15.189:0/811032011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4980cb870 msgr2=0x7fc498150840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:12.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.661+0000 7fc4a0568700 1 --2- 172.21.15.189:0/811032011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4980cb870 0x7fc498150840 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fc49400c500 tx=0x7fc49400c810 comp rx=0 tx=0).stop 2024-04-06T12:31:12.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.661+0000 7fc4a0568700 1 -- 172.21.15.189:0/811032011 shutdown_connections 2024-04-06T12:31:12.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.661+0000 7fc4a0568700 1 --2- 172.21.15.189:0/811032011 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc47c06ad20 0x7fc47c06d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:12.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.661+0000 7fc4a0568700 1 --2- 172.21.15.189:0/811032011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4980d11b0 0x7fc49815f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:12.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.661+0000 7fc4a0568700 1 --2- 172.21.15.189:0/811032011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4980cb870 0x7fc498150840 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:12.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.661+0000 7fc4a0568700 1 --2- 172.21.15.189:0/811032011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4980cfa00 0x7fc4981579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:12.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.661+0000 7fc4a0568700 1 -- 172.21.15.189:0/811032011 >> 172.21.15.189:0/811032011 conn(0x7fc4980b2db0 msgr2=0x7fc49815b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:12.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.661+0000 7fc4a0568700 1 -- 172.21.15.189:0/811032011 shutdown_connections 2024-04-06T12:31:12.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.661+0000 7fc4a0568700 1 -- 172.21.15.189:0/811032011 wait complete. 2024-04-06T12:31:12.660 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' removed 2024-04-06T12:31:12.672 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-06T12:31:12.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.804+0000 7f0893821700 1 Processor -- start 2024-04-06T12:31:12.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.805+0000 7f0893821700 1 -- start start 2024-04-06T12:31:12.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.805+0000 7f0893821700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c15a550 0x7f088c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:12.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.805+0000 7f0893821700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f088c0cf900 0x7f088c0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:12.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.805+0000 7f0893821700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f088c0cdbe0 con 0x7f088c0b4ba0 2024-04-06T12:31:12.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.805+0000 7f0893821700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f088c0b5010 con 0x7f088c0cf900 2024-04-06T12:31:12.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.805+0000 7f0893821700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088c15a940 con 0x7f088c15a550 2024-04-06T12:31:12.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.806+0000 7f08915bd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c15a550 0x7f088c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:12.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.806+0000 7f0890dbc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f088c0cf900 0x7f088c0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:12.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.806+0000 7f08915bd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c15a550 0x7f088c0cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:40648/0 (socket says 172.21.15.189:40648) 2024-04-06T12:31:12.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.806+0000 7f08915bd700 1 -- 172.21.15.189:0/1073059792 learned_addr learned my addr 172.21.15.189:0/1073059792 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:12.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.806+0000 7f0891dbe700 1 --1- 172.21.15.189:0/1073059792 >> v1:172.21.15.192:6789/0 conn(0x7f088c0b4ba0 0x7f088c06eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:59312/0 (socket says 172.21.15.189:59312) 2024-04-06T12:31:12.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.806+0000 7f08915bd700 1 -- 172.21.15.189:0/1073059792 >> v2:172.21.15.192:3300/0 conn(0x7f088c0cf900 msgr2=0x7f088c0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:12.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.806+0000 7f08915bd700 1 --2- 172.21.15.189:0/1073059792 >> v2:172.21.15.192:3300/0 conn(0x7f088c0cf900 0x7f088c0b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:12.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.806+0000 7f08915bd700 1 -- 172.21.15.189:0/1073059792 >> v1:172.21.15.192:6789/0 conn(0x7f088c0b4ba0 legacy=0x7f088c06eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:12.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.806+0000 7f08915bd700 1 -- 172.21.15.189:0/1073059792 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c15ac00 con 0x7f088c15a550 2024-04-06T12:31:12.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.806+0000 7f0890dbc700 1 --2- 172.21.15.189:0/1073059792 >> v2:172.21.15.192:3300/0 conn(0x7f088c0cf900 0x7f088c0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:12.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.806+0000 7f08915bd700 1 --2- 172.21.15.189:0/1073059792 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c15a550 0x7f088c0cf3c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f087c0098f0 tx=0x7f087c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9e2f422de8c4d93 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:12.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.807+0000 7f0883fff700 1 -- 172.21.15.189:0/1073059792 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c011830 con 0x7f088c15a550 2024-04-06T12:31:12.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.807+0000 7f0883fff700 1 -- 172.21.15.189:0/1073059792 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087c011990 con 0x7f088c15a550 2024-04-06T12:31:12.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.807+0000 7f0883fff700 1 -- 172.21.15.189:0/1073059792 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c011d20 con 0x7f088c15a550 2024-04-06T12:31:12.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.807+0000 7f0893821700 1 -- 172.21.15.189:0/1073059792 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c15a550 msgr2=0x7f088c0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:12.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.807+0000 7f0893821700 1 --2- 172.21.15.189:0/1073059792 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c15a550 0x7f088c0cf3c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f087c0098f0 tx=0x7f087c009c00 comp rx=0 tx=0).stop 2024-04-06T12:31:12.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.807+0000 7f0893821700 1 -- 172.21.15.189:0/1073059792 shutdown_connections 2024-04-06T12:31:12.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.807+0000 7f0893821700 1 --2- 172.21.15.189:0/1073059792 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c15a550 0x7f088c0cf3c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:12.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.807+0000 7f0893821700 1 --2- 172.21.15.189:0/1073059792 >> v2:172.21.15.192:3300/0 conn(0x7f088c0cf900 0x7f088c0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:12.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.807+0000 7f0893821700 1 -- 172.21.15.189:0/1073059792 >> 172.21.15.189:0/1073059792 conn(0x7f088c0d00b0 msgr2=0x7f088c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:12.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.807+0000 7f0893821700 1 -- 172.21.15.189:0/1073059792 shutdown_connections 2024-04-06T12:31:12.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.807+0000 7f0893821700 1 -- 172.21.15.189:0/1073059792 wait complete. 2024-04-06T12:31:12.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.808+0000 7f0893821700 1 Processor -- start 2024-04-06T12:31:12.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.808+0000 7f0893821700 1 -- start start 2024-04-06T12:31:12.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.809+0000 7f0893821700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f088c0b4ba0 0x7f088c1f41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:12.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.809+0000 7f0893821700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c0cf900 0x7f088c141350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:12.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.809+0000 7f0893821700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c13e000 0x7f088c1593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:12.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.809+0000 7f0893821700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c13e970 con 0x7f088c0cf900 2024-04-06T12:31:12.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.809+0000 7f0893821700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c141890 con 0x7f088c0b4ba0 2024-04-06T12:31:12.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.809+0000 7f0893821700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088c1f47d0 con 0x7f088c13e000 2024-04-06T12:31:12.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.809+0000 7f0890dbc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c0cf900 0x7f088c141350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:12.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.809+0000 7f0890dbc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c0cf900 0x7f088c141350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36326/0 (socket says 172.21.15.189:36326) 2024-04-06T12:31:12.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.809+0000 7f0890dbc700 1 -- 172.21.15.189:0/719543188 learned_addr learned my addr 172.21.15.189:0/719543188 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:12.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.809+0000 7f0891dbe700 1 --2- 172.21.15.189:0/719543188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c13e000 0x7f088c1593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:12.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.809+0000 7f08915bd700 1 --2- 172.21.15.189:0/719543188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f088c0b4ba0 0x7f088c1f41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:12.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.810+0000 7f0890dbc700 1 -- 172.21.15.189:0/719543188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c13e000 msgr2=0x7f088c1593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:12.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.810+0000 7f0890dbc700 1 --2- 172.21.15.189:0/719543188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c13e000 0x7f088c1593a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:12.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.810+0000 7f0890dbc700 1 -- 172.21.15.189:0/719543188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f088c0b4ba0 msgr2=0x7f088c1f41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:12.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.810+0000 7f0890dbc700 1 --2- 172.21.15.189:0/719543188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f088c0b4ba0 0x7f088c1f41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:12.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.810+0000 7f0890dbc700 1 -- 172.21.15.189:0/719543188 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08880094a0 con 0x7f088c0cf900 2024-04-06T12:31:12.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.810+0000 7f0890dbc700 1 --2- 172.21.15.189:0/719543188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c0cf900 0x7f088c141350 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f088800ec10 tx=0x7f0888009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:12.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.810+0000 7f0881ffb700 1 -- 172.21.15.189:0/719543188 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088800c8e0 con 0x7f088c0cf900 2024-04-06T12:31:12.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.810+0000 7f0881ffb700 1 -- 172.21.15.189:0/719543188 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088800ca40 con 0x7f088c0cf900 2024-04-06T12:31:12.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.810+0000 7f0893821700 1 -- 172.21.15.189:0/719543188 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087c009550 con 0x7f088c0cf900 2024-04-06T12:31:12.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.810+0000 7f0881ffb700 1 -- 172.21.15.189:0/719543188 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088800cdd0 con 0x7f088c0cf900 2024-04-06T12:31:12.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.810+0000 7f0893821700 1 -- 172.21.15.189:0/719543188 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f088c1f4d90 con 0x7f088c0cf900 2024-04-06T12:31:12.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.811+0000 7f0893821700 1 -- 172.21.15.189:0/719543188 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f088c13eb20 con 0x7f088c0cf900 2024-04-06T12:31:12.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.811+0000 7f0881ffb700 1 -- 172.21.15.189:0/719543188 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0888013470 con 0x7f088c0cf900 2024-04-06T12:31:12.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.812+0000 7f0881ffb700 1 --2- 172.21.15.189:0/719543188 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f087006adb0 0x7f087006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:12.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.812+0000 7f0881ffb700 1 -- 172.21.15.189:0/719543188 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f08880896b0 con 0x7f088c0cf900 2024-04-06T12:31:12.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.812+0000 7f08915bd700 1 --2- 172.21.15.189:0/719543188 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f087006adb0 0x7f087006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:12.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.812+0000 7f08915bd700 1 --2- 172.21.15.189:0/719543188 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f087006adb0 0x7f087006d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f087c009c40 tx=0x7f087c008030 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:12.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:12.816+0000 7f0881ffb700 1 -- 172.21.15.189:0/719543188 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f088c13eb20 con 0x7f088c0cf900 2024-04-06T12:31:13.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.016+0000 7f0893821700 1 -- 172.21.15.189:0/719543188 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f088c13eb20 con 0x7f088c0cf900 2024-04-06T12:31:13.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.020+0000 7f0881ffb700 1 -- 172.21.15.189:0/719543188 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7f088c13eb20 con 0x7f088c0cf900 2024-04-06T12:31:13.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.022+0000 7f0893821700 1 -- 172.21.15.189:0/719543188 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f087006adb0 msgr2=0x7f087006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:13.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.022+0000 7f0893821700 1 --2- 172.21.15.189:0/719543188 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f087006adb0 0x7f087006d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f087c009c40 tx=0x7f087c008030 comp rx=0 tx=0).stop 2024-04-06T12:31:13.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.022+0000 7f0893821700 1 -- 172.21.15.189:0/719543188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c0cf900 msgr2=0x7f088c141350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:13.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.022+0000 7f0893821700 1 --2- 172.21.15.189:0/719543188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c0cf900 0x7f088c141350 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f088800ec10 tx=0x7f0888009e20 comp rx=0 tx=0).stop 2024-04-06T12:31:13.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.022+0000 7f0893821700 1 -- 172.21.15.189:0/719543188 shutdown_connections 2024-04-06T12:31:13.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.022+0000 7f0893821700 1 --2- 172.21.15.189:0/719543188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c13e000 0x7f088c1593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.022+0000 7f0893821700 1 --2- 172.21.15.189:0/719543188 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f087006adb0 0x7f087006d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.022+0000 7f0893821700 1 --2- 172.21.15.189:0/719543188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f088c0b4ba0 0x7f088c1f41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.022+0000 7f0893821700 1 --2- 172.21.15.189:0/719543188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c0cf900 0x7f088c141350 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.022+0000 7f0893821700 1 -- 172.21.15.189:0/719543188 >> 172.21.15.189:0/719543188 conn(0x7f088c0d00b0 msgr2=0x7f088c15bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:13.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.022+0000 7f0893821700 1 -- 172.21.15.189:0/719543188 shutdown_connections 2024-04-06T12:31:13.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.022+0000 7f0893821700 1 -- 172.21.15.189:0/719543188 wait complete. 2024-04-06T12:31:13.032 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-06T12:31:13.116 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:31:13.119 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:31:13.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.173+0000 7fd42dad9700 1 Processor -- start 2024-04-06T12:31:13.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.173+0000 7fd42dad9700 1 -- start start 2024-04-06T12:31:13.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.174+0000 7fd42dad9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4280cd3e0 0x7fd4280cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.174+0000 7fd42dad9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd42815d7c0 0x7fd42806b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.174+0000 7fd42dad9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd42806b6f0 0x7fd4281610f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.174+0000 7fd42dad9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4280cdfa0 con 0x7fd42806b6f0 2024-04-06T12:31:13.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.174+0000 7fd42dad9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd42806ceb0 con 0x7fd42815d7c0 2024-04-06T12:31:13.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.174+0000 7fd42dad9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd42806d750 con 0x7fd4280cd3e0 2024-04-06T12:31:13.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.174+0000 7fd427fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd42806b6f0 0x7fd4281610f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.174+0000 7fd427fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd42806b6f0 0x7fd4281610f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36332/0 (socket says 172.21.15.189:36332) 2024-04-06T12:31:13.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.174+0000 7fd427fff700 1 -- 172.21.15.189:0/3306552380 learned_addr learned my addr 172.21.15.189:0/3306552380 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:13.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.175+0000 7fd426ffd700 1 --2- 172.21.15.189:0/3306552380 >> v2:172.21.15.192:3300/0 conn(0x7fd42815d7c0 0x7fd42806b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.175+0000 7fd4277fe700 1 --2- 172.21.15.189:0/3306552380 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4280cd3e0 0x7fd4280cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.175+0000 7fd427fff700 1 -- 172.21.15.189:0/3306552380 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4280cd3e0 msgr2=0x7fd4280cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:13.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.175+0000 7fd427fff700 1 --2- 172.21.15.189:0/3306552380 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4280cd3e0 0x7fd4280cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.175+0000 7fd427fff700 1 -- 172.21.15.189:0/3306552380 >> v2:172.21.15.192:3300/0 conn(0x7fd42815d7c0 msgr2=0x7fd42806b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:13.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.175+0000 7fd427fff700 1 --2- 172.21.15.189:0/3306552380 >> v2:172.21.15.192:3300/0 conn(0x7fd42815d7c0 0x7fd42806b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.175+0000 7fd427fff700 1 -- 172.21.15.189:0/3306552380 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4280cdd00 con 0x7fd42806b6f0 2024-04-06T12:31:13.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.175+0000 7fd426ffd700 1 --2- 172.21.15.189:0/3306552380 >> v2:172.21.15.192:3300/0 conn(0x7fd42815d7c0 0x7fd42806b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:13.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.175+0000 7fd427fff700 1 --2- 172.21.15.189:0/3306552380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd42806b6f0 0x7fd4281610f0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fd428073360 tx=0x7fd42000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7123db2a0dae5a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:13.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.175+0000 7fd4267fc700 1 -- 172.21.15.189:0/3306552380 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42000e070 con 0x7fd42806b6f0 2024-04-06T12:31:13.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.175+0000 7fd4267fc700 1 -- 172.21.15.189:0/3306552380 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd42000b9c0 con 0x7fd42806b6f0 2024-04-06T12:31:13.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.175+0000 7fd4267fc700 1 -- 172.21.15.189:0/3306552380 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42000bd70 con 0x7fd42806b6f0 2024-04-06T12:31:13.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.176+0000 7fd42dad9700 1 -- 172.21.15.189:0/3306552380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd42806b6f0 msgr2=0x7fd4281610f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:13.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.176+0000 7fd42dad9700 1 --2- 172.21.15.189:0/3306552380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd42806b6f0 0x7fd4281610f0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fd428073360 tx=0x7fd42000b330 comp rx=0 tx=0).stop 2024-04-06T12:31:13.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.176+0000 7fd42dad9700 1 -- 172.21.15.189:0/3306552380 shutdown_connections 2024-04-06T12:31:13.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.176+0000 7fd42dad9700 1 --2- 172.21.15.189:0/3306552380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd42806b6f0 0x7fd4281610f0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.176+0000 7fd42dad9700 1 --2- 172.21.15.189:0/3306552380 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4280cd3e0 0x7fd4280cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.176+0000 7fd42dad9700 1 --2- 172.21.15.189:0/3306552380 >> v2:172.21.15.192:3300/0 conn(0x7fd42815d7c0 0x7fd42806b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.176+0000 7fd42dad9700 1 -- 172.21.15.189:0/3306552380 >> 172.21.15.189:0/3306552380 conn(0x7fd4280cc290 msgr2=0x7fd4280cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:13.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.176+0000 7fd42dad9700 1 -- 172.21.15.189:0/3306552380 shutdown_connections 2024-04-06T12:31:13.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.176+0000 7fd42dad9700 1 -- 172.21.15.189:0/3306552380 wait complete. 2024-04-06T12:31:13.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.177+0000 7fd42dad9700 1 Processor -- start 2024-04-06T12:31:13.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.177+0000 7fd42dad9700 1 -- start start 2024-04-06T12:31:13.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.178+0000 7fd42dad9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd42806b6f0 0x7fd42815f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.178+0000 7fd42dad9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4280cd3e0 0x7fd4281f3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.178+0000 7fd42dad9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd42815d7c0 0x7fd42806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.178+0000 7fd42dad9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4281f3a50 con 0x7fd4280cd3e0 2024-04-06T12:31:13.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.178+0000 7fd42dad9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4281f6060 con 0x7fd42806b6f0 2024-04-06T12:31:13.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.178+0000 7fd42dad9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd42815fbb0 con 0x7fd42815d7c0 2024-04-06T12:31:13.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.178+0000 7fd426ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4280cd3e0 0x7fd4281f3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.178+0000 7fd426ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4280cd3e0 0x7fd4281f3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36336/0 (socket says 172.21.15.189:36336) 2024-04-06T12:31:13.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.178+0000 7fd426ffd700 1 -- 172.21.15.189:0/333754794 learned_addr learned my addr 172.21.15.189:0/333754794 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:13.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.179+0000 7fd427fff700 1 --2- 172.21.15.189:0/333754794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd42815d7c0 0x7fd42806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.179+0000 7fd4277fe700 1 --2- 172.21.15.189:0/333754794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd42806b6f0 0x7fd42815f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.179+0000 7fd426ffd700 1 -- 172.21.15.189:0/333754794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd42815d7c0 msgr2=0x7fd42806e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:13.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.179+0000 7fd426ffd700 1 --2- 172.21.15.189:0/333754794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd42815d7c0 0x7fd42806e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.179+0000 7fd426ffd700 1 -- 172.21.15.189:0/333754794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd42806b6f0 msgr2=0x7fd42815f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:13.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.179+0000 7fd426ffd700 1 --2- 172.21.15.189:0/333754794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd42806b6f0 0x7fd42815f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.179+0000 7fd426ffd700 1 -- 172.21.15.189:0/333754794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd420009b30 con 0x7fd4280cd3e0 2024-04-06T12:31:13.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.179+0000 7fd426ffd700 1 --2- 172.21.15.189:0/333754794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4280cd3e0 0x7fd4281f3510 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fd41c00c7f0 tx=0x7fd41c00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:13.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.179+0000 7fd40ffff700 1 -- 172.21.15.189:0/333754794 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41c011800 con 0x7fd4280cd3e0 2024-04-06T12:31:13.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.179+0000 7fd40ffff700 1 -- 172.21.15.189:0/333754794 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd41c011960 con 0x7fd4280cd3e0 2024-04-06T12:31:13.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.179+0000 7fd40ffff700 1 -- 172.21.15.189:0/333754794 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41c011cf0 con 0x7fd4280cd3e0 2024-04-06T12:31:13.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.179+0000 7fd42dad9700 1 -- 172.21.15.189:0/333754794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4281f62b0 con 0x7fd4280cd3e0 2024-04-06T12:31:13.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.179+0000 7fd42dad9700 1 -- 172.21.15.189:0/333754794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4281f6870 con 0x7fd4280cd3e0 2024-04-06T12:31:13.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.180+0000 7fd42dad9700 1 -- 172.21.15.189:0/333754794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4281f6a60 con 0x7fd4280cd3e0 2024-04-06T12:31:13.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.180+0000 7fd40ffff700 1 -- 172.21.15.189:0/333754794 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fd41c018360 con 0x7fd4280cd3e0 2024-04-06T12:31:13.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.181+0000 7fd40ffff700 1 --2- 172.21.15.189:0/333754794 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd41406adb0 0x7fd41406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.181+0000 7fd40ffff700 1 -- 172.21.15.189:0/333754794 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5951+0+0 (secure 0 0 0) 0x7fd41c0894e0 con 0x7fd4280cd3e0 2024-04-06T12:31:13.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.181+0000 7fd4277fe700 1 --2- 172.21.15.189:0/333754794 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd41406adb0 0x7fd41406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.181+0000 7fd4277fe700 1 --2- 172.21.15.189:0/333754794 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd41406adb0 0x7fd41406d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd418006fd0 tx=0x7fd418008040 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:13.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.185+0000 7fd40ffff700 1 -- 172.21.15.189:0/333754794 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd41c0543b0 con 0x7fd4280cd3e0 2024-04-06T12:31:13.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.380+0000 7fd42dad9700 1 -- 172.21.15.189:0/333754794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fd428074990 con 0x7fd4280cd3e0 2024-04-06T12:31:13.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.384+0000 7fd40ffff700 1 -- 172.21.15.189:0/333754794 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v44) v1 ==== 115+0+0 (secure 0 0 0) 0x7fd41c0579d0 con 0x7fd4280cd3e0 2024-04-06T12:31:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.387+0000 7fd42dad9700 1 -- 172.21.15.189:0/333754794 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd41406adb0 msgr2=0x7fd41406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.387+0000 7fd42dad9700 1 --2- 172.21.15.189:0/333754794 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd41406adb0 0x7fd41406d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd418006fd0 tx=0x7fd418008040 comp rx=0 tx=0).stop 2024-04-06T12:31:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.387+0000 7fd42dad9700 1 -- 172.21.15.189:0/333754794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4280cd3e0 msgr2=0x7fd4281f3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.387+0000 7fd42dad9700 1 --2- 172.21.15.189:0/333754794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4280cd3e0 0x7fd4281f3510 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fd41c00c7f0 tx=0x7fd41c00cb00 comp rx=0 tx=0).stop 2024-04-06T12:31:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.387+0000 7fd42dad9700 1 -- 172.21.15.189:0/333754794 shutdown_connections 2024-04-06T12:31:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.387+0000 7fd42dad9700 1 --2- 172.21.15.189:0/333754794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd42815d7c0 0x7fd42806e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.387+0000 7fd42dad9700 1 --2- 172.21.15.189:0/333754794 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd41406adb0 0x7fd41406d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.387+0000 7fd42dad9700 1 --2- 172.21.15.189:0/333754794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd42806b6f0 0x7fd42815f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.387+0000 7fd42dad9700 1 --2- 172.21.15.189:0/333754794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4280cd3e0 0x7fd4281f3510 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.387+0000 7fd42dad9700 1 -- 172.21.15.189:0/333754794 >> 172.21.15.189:0/333754794 conn(0x7fd4280cc290 msgr2=0x7fd4280b0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:13.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.387+0000 7fd42dad9700 1 -- 172.21.15.189:0/333754794 shutdown_connections 2024-04-06T12:31:13.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.387+0000 7fd42dad9700 1 -- 172.21.15.189:0/333754794 wait complete. 2024-04-06T12:31:13.397 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-06T12:31:13.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.564+0000 7f5419411700 1 Processor -- start 2024-04-06T12:31:13.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.564+0000 7f5419411700 1 -- start start 2024-04-06T12:31:13.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.565+0000 7f5419411700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5414160cc0 0x7f54140d13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.565+0000 7f5419411700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f54140d18f0 0x7f54140b4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.565+0000 7f5419411700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f54140b52f0 0x7f541406b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.565+0000 7f5419411700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54140b5740 con 0x7f54140b52f0 2024-04-06T12:31:13.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.565+0000 7f5419411700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54140d1e70 con 0x7f54140d18f0 2024-04-06T12:31:13.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.565+0000 7f5419411700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54140d2150 con 0x7f5414160cc0 2024-04-06T12:31:13.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.565+0000 7f54137fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f54140b52f0 0x7f541406b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.565+0000 7f54137fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f54140b52f0 0x7f541406b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36338/0 (socket says 172.21.15.189:36338) 2024-04-06T12:31:13.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.565+0000 7f54137fe700 1 -- 172.21.15.189:0/2575936450 learned_addr learned my addr 172.21.15.189:0/2575936450 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:13.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.565+0000 7f5412ffd700 1 --2- 172.21.15.189:0/2575936450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5414160cc0 0x7f54140d13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.565+0000 7f54127fc700 1 --2- 172.21.15.189:0/2575936450 >> v2:172.21.15.192:3300/0 conn(0x7f54140d18f0 0x7f54140b4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.565+0000 7f54137fe700 1 -- 172.21.15.189:0/2575936450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5414160cc0 msgr2=0x7f54140d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:13.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.565+0000 7f54137fe700 1 --2- 172.21.15.189:0/2575936450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5414160cc0 0x7f54140d13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.565+0000 7f54137fe700 1 -- 172.21.15.189:0/2575936450 >> v2:172.21.15.192:3300/0 conn(0x7f54140d18f0 msgr2=0x7f54140b4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:13.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.565+0000 7f54137fe700 1 --2- 172.21.15.189:0/2575936450 >> v2:172.21.15.192:3300/0 conn(0x7f54140d18f0 0x7f54140b4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.565+0000 7f54137fe700 1 -- 172.21.15.189:0/2575936450 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54140d2430 con 0x7f54140b52f0 2024-04-06T12:31:13.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.565+0000 7f54137fe700 1 --2- 172.21.15.189:0/2575936450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54140b52f0 0x7f541406b6a0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f541408b940 tx=0x7f540c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0828379af04e60c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:13.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.566+0000 7f5411ffb700 1 -- 172.21.15.189:0/2575936450 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540c00e070 con 0x7f54140b52f0 2024-04-06T12:31:13.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.566+0000 7f5411ffb700 1 -- 172.21.15.189:0/2575936450 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540c00b9c0 con 0x7f54140b52f0 2024-04-06T12:31:13.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.566+0000 7f5411ffb700 1 -- 172.21.15.189:0/2575936450 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540c00bd70 con 0x7f54140b52f0 2024-04-06T12:31:13.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.566+0000 7f5419411700 1 -- 172.21.15.189:0/2575936450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54140b52f0 msgr2=0x7f541406b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:13.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.566+0000 7f5419411700 1 --2- 172.21.15.189:0/2575936450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54140b52f0 0x7f541406b6a0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f541408b940 tx=0x7f540c00b330 comp rx=0 tx=0).stop 2024-04-06T12:31:13.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.566+0000 7f5419411700 1 -- 172.21.15.189:0/2575936450 shutdown_connections 2024-04-06T12:31:13.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.566+0000 7f5419411700 1 --2- 172.21.15.189:0/2575936450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54140b52f0 0x7f541406b6a0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.566+0000 7f5419411700 1 --2- 172.21.15.189:0/2575936450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5414160cc0 0x7f54140d13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.566+0000 7f5419411700 1 --2- 172.21.15.189:0/2575936450 >> v2:172.21.15.192:3300/0 conn(0x7f54140d18f0 0x7f54140b4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.566+0000 7f5419411700 1 -- 172.21.15.189:0/2575936450 >> 172.21.15.189:0/2575936450 conn(0x7f54140cfdb0 msgr2=0x7f54140d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:13.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.566+0000 7f5419411700 1 -- 172.21.15.189:0/2575936450 shutdown_connections 2024-04-06T12:31:13.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.566+0000 7f5419411700 1 -- 172.21.15.189:0/2575936450 wait complete. 2024-04-06T12:31:13.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.567+0000 7f5419411700 1 Processor -- start 2024-04-06T12:31:13.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.567+0000 7f5419411700 1 -- start start 2024-04-06T12:31:13.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.568+0000 7f5419411700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54140b52f0 0x7f54140cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.568+0000 7f5419411700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54140d18f0 0x7f5414159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.568+0000 7f5419411700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5414160cc0 0x7f54141f5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.568+0000 7f5419411700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54141f60f0 con 0x7f5414160cc0 2024-04-06T12:31:13.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.568+0000 7f5419411700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541415ad40 con 0x7f54140b52f0 2024-04-06T12:31:13.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.568+0000 7f5419411700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54140cc920 con 0x7f54140d18f0 2024-04-06T12:31:13.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.568+0000 7f54137fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5414160cc0 0x7f54141f5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.568+0000 7f54137fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5414160cc0 0x7f54141f5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36348/0 (socket says 172.21.15.189:36348) 2024-04-06T12:31:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.568+0000 7f54137fe700 1 -- 172.21.15.189:0/2028344608 learned_addr learned my addr 172.21.15.189:0/2028344608 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.569+0000 7f54127fc700 1 --2- 172.21.15.189:0/2028344608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54140d18f0 0x7f5414159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.569+0000 7f5412ffd700 1 --2- 172.21.15.189:0/2028344608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54140b52f0 0x7f54140cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.569+0000 7f54137fe700 1 -- 172.21.15.189:0/2028344608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54140d18f0 msgr2=0x7f5414159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.569+0000 7f54137fe700 1 --2- 172.21.15.189:0/2028344608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54140d18f0 0x7f5414159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.569+0000 7f54137fe700 1 -- 172.21.15.189:0/2028344608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54140b52f0 msgr2=0x7f54140cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.569+0000 7f54137fe700 1 --2- 172.21.15.189:0/2028344608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54140b52f0 0x7f54140cb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.569+0000 7f54137fe700 1 -- 172.21.15.189:0/2028344608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540c009b30 con 0x7f5414160cc0 2024-04-06T12:31:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.569+0000 7f54127fc700 1 --2- 172.21.15.189:0/2028344608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54140d18f0 0x7f5414159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.569+0000 7f5412ffd700 1 --2- 172.21.15.189:0/2028344608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54140b52f0 0x7f54140cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.569+0000 7f54137fe700 1 --2- 172.21.15.189:0/2028344608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5414160cc0 0x7f54141f5bb0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f5414068080 tx=0x7f540c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.569+0000 7f53fb7fe700 1 -- 172.21.15.189:0/2028344608 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540c004250 con 0x7f5414160cc0 2024-04-06T12:31:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.569+0000 7f53fb7fe700 1 -- 172.21.15.189:0/2028344608 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540c0043d0 con 0x7f5414160cc0 2024-04-06T12:31:13.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.569+0000 7f53fb7fe700 1 -- 172.21.15.189:0/2028344608 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540c004780 con 0x7f5414160cc0 2024-04-06T12:31:13.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.569+0000 7f5419411700 1 -- 172.21.15.189:0/2028344608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54140d3090 con 0x7f5414160cc0 2024-04-06T12:31:13.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.569+0000 7f5419411700 1 -- 172.21.15.189:0/2028344608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54140cbff0 con 0x7f5414160cc0 2024-04-06T12:31:13.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.570+0000 7f5419411700 1 -- 172.21.15.189:0/2028344608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5414074990 con 0x7f5414160cc0 2024-04-06T12:31:13.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.570+0000 7f53fb7fe700 1 -- 172.21.15.189:0/2028344608 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f540c01c440 con 0x7f5414160cc0 2024-04-06T12:31:13.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.571+0000 7f53fb7fe700 1 --2- 172.21.15.189:0/2028344608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f540006adb0 0x7f540006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.571+0000 7f53fb7fe700 1 -- 172.21.15.189:0/2028344608 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f540c08ade0 con 0x7f5414160cc0 2024-04-06T12:31:13.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.571+0000 7f5412ffd700 1 --2- 172.21.15.189:0/2028344608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f540006adb0 0x7f540006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.571+0000 7f5412ffd700 1 --2- 172.21.15.189:0/2028344608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f540006adb0 0x7f540006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f54080097a0 tx=0x7f5408008040 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:13.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.575+0000 7f53fb7fe700 1 -- 172.21.15.189:0/2028344608 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f540c055e60 con 0x7f5414160cc0 2024-04-06T12:31:13.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.770+0000 7f5419411700 1 -- 172.21.15.189:0/2028344608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f54140cca60 con 0x7f5414160cc0 2024-04-06T12:31:13.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.775+0000 7f53fb7fe700 1 -- 172.21.15.189:0/2028344608 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v45) v1 ==== 120+0+0 (secure 0 0 0) 0x7f540c059480 con 0x7f5414160cc0 2024-04-06T12:31:13.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.778+0000 7f5419411700 1 -- 172.21.15.189:0/2028344608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f540006adb0 msgr2=0x7f540006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:13.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.778+0000 7f5419411700 1 --2- 172.21.15.189:0/2028344608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f540006adb0 0x7f540006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f54080097a0 tx=0x7f5408008040 comp rx=0 tx=0).stop 2024-04-06T12:31:13.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.778+0000 7f5419411700 1 -- 172.21.15.189:0/2028344608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5414160cc0 msgr2=0x7f54141f5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:13.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.778+0000 7f5419411700 1 --2- 172.21.15.189:0/2028344608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5414160cc0 0x7f54141f5bb0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f5414068080 tx=0x7f540c004ad0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.778+0000 7f5419411700 1 -- 172.21.15.189:0/2028344608 shutdown_connections 2024-04-06T12:31:13.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.778+0000 7f5419411700 1 --2- 172.21.15.189:0/2028344608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5414160cc0 0x7f54141f5bb0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.778+0000 7f5419411700 1 --2- 172.21.15.189:0/2028344608 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f540006adb0 0x7f540006d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.778+0000 7f5419411700 1 --2- 172.21.15.189:0/2028344608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54140b52f0 0x7f54140cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.778+0000 7f5419411700 1 --2- 172.21.15.189:0/2028344608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54140d18f0 0x7f5414159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.778+0000 7f5419411700 1 -- 172.21.15.189:0/2028344608 >> 172.21.15.189:0/2028344608 conn(0x7f54140cfdb0 msgr2=0x7f54140d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:13.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.778+0000 7f5419411700 1 -- 172.21.15.189:0/2028344608 shutdown_connections 2024-04-06T12:31:13.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.778+0000 7f5419411700 1 -- 172.21.15.189:0/2028344608 wait complete. 2024-04-06T12:31:13.788 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2024-04-06T12:31:13.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.913+0000 7feca2f78700 1 Processor -- start 2024-04-06T12:31:13.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.913+0000 7feca2f78700 1 -- start start 2024-04-06T12:31:13.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.914+0000 7feca2f78700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fec9c0b3e20 0x7fec9c15e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.914+0000 7feca2f78700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec9c0b3990 con 0x7fec9c15d7b0 2024-04-06T12:31:13.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.914+0000 7feca2f78700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec9c0b3060 con 0x7fec9c06ae40 2024-04-06T12:31:13.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.914+0000 7feca2f78700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec9c0b4230 con 0x7fec9c0b3e20 2024-04-06T12:31:13.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.914+0000 7feca0d14700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fec9c15d7b0 0x7fec9c06a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55904/0 (socket says 172.21.15.189:55904) 2024-04-06T12:31:13.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.914+0000 7feca0d14700 1 -- 172.21.15.189:0/4113694006 learned_addr learned my addr 172.21.15.189:0/4113694006 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:13.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.914+0000 7fec9b7fe700 1 -- 172.21.15.189:0/4113694006 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459710056 0 0) 0x7fec9c0b3990 con 0x7fec9c15d7b0 2024-04-06T12:31:13.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.914+0000 7fec9b7fe700 1 -- 172.21.15.189:0/4113694006 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec8c0035b0 con 0x7fec9c15d7b0 2024-04-06T12:31:13.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.914+0000 7feca1515700 1 --2- 172.21.15.189:0/4113694006 >> v2:172.21.15.192:3300/0 conn(0x7fec9c0b3e20 0x7fec9c15e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7fec9b7fe700 1 -- 172.21.15.189:0/4113694006 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1012619989 0 0) 0x7fec8c0035b0 con 0x7fec9c15d7b0 2024-04-06T12:31:13.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7fec9b7fe700 1 -- 172.21.15.189:0/4113694006 >> v2:172.21.15.192:3300/0 conn(0x7fec9c0b3e20 msgr2=0x7fec9c15e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:13.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7fec9b7fe700 1 --2- 172.21.15.189:0/4113694006 >> v2:172.21.15.192:3300/0 conn(0x7fec9c0b3e20 0x7fec9c15e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7fec9b7fe700 1 -- 172.21.15.189:0/4113694006 >> v1:172.21.15.192:6789/0 conn(0x7fec9c06ae40 legacy=0x7fec9c15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:13.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7fec9b7fe700 1 -- 172.21.15.189:0/4113694006 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9c0b3340 con 0x7fec9c15d7b0 2024-04-06T12:31:13.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7fec9b7fe700 1 -- 172.21.15.189:0/4113694006 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fec90001670 con 0x7fec9c15d7b0 2024-04-06T12:31:13.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7fec9b7fe700 1 -- 172.21.15.189:0/4113694006 >> v1:172.21.15.189:6789/0 conn(0x7fec9c15d7b0 legacy=0x7fec9c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:13.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7fec9b7fe700 1 --2- 172.21.15.189:0/4113694006 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec8c003a00 0x7fec8c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7fec9b7fe700 1 --2- 172.21.15.189:0/4113694006 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec8c004850 0x7fec8c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7fec9b7fe700 1 --2- 172.21.15.189:0/4113694006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec8c005210 0x7fec8c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7fec9b7fe700 1 -- 172.21.15.189:0/4113694006 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c0b4230 con 0x7fec8c005210 2024-04-06T12:31:13.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7fec9b7fe700 1 -- 172.21.15.189:0/4113694006 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c00c1b0 con 0x7fec8c004850 2024-04-06T12:31:13.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7fec9b7fe700 1 -- 172.21.15.189:0/4113694006 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c00c370 con 0x7fec8c003a00 2024-04-06T12:31:13.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7feca1515700 1 --2- 172.21.15.189:0/4113694006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec8c005210 0x7fec8c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7fec9bfff700 1 --2- 172.21.15.189:0/4113694006 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec8c004850 0x7fec8c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7feca0d14700 1 --2- 172.21.15.189:0/4113694006 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec8c003a00 0x7fec8c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7feca1515700 1 -- 172.21.15.189:0/4113694006 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec8c003a00 msgr2=0x7fec8c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:13.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7feca1515700 1 --2- 172.21.15.189:0/4113694006 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec8c003a00 0x7fec8c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7feca1515700 1 -- 172.21.15.189:0/4113694006 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec8c004850 msgr2=0x7fec8c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:13.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7feca1515700 1 --2- 172.21.15.189:0/4113694006 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec8c004850 0x7fec8c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7feca1515700 1 -- 172.21.15.189:0/4113694006 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8c0035b0 con 0x7fec8c005210 2024-04-06T12:31:13.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7feca1515700 1 -- 172.21.15.189:0/4113694006 reap_dead start 2024-04-06T12:31:13.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7feca1515700 1 --2- 172.21.15.189:0/4113694006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec8c005210 0x7fec8c00ba10 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fec94009fc0 tx=0x7fec8c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3548814b8152c46 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:13.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.915+0000 7fec9b7fe700 1 -- 172.21.15.189:0/4113694006 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec94003a20 con 0x7fec8c005210 2024-04-06T12:31:13.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.916+0000 7fec9b7fe700 1 -- 172.21.15.189:0/4113694006 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9400f040 con 0x7fec8c005210 2024-04-06T12:31:13.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.916+0000 7fec9b7fe700 1 -- 172.21.15.189:0/4113694006 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec94003de0 con 0x7fec8c005210 2024-04-06T12:31:13.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.916+0000 7feca2f78700 1 -- 172.21.15.189:0/4113694006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec8c005210 msgr2=0x7fec8c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:13.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.916+0000 7feca2f78700 1 --2- 172.21.15.189:0/4113694006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec8c005210 0x7fec8c00ba10 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fec94009fc0 tx=0x7fec8c005880 comp rx=0 tx=0).stop 2024-04-06T12:31:13.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.916+0000 7feca2f78700 1 -- 172.21.15.189:0/4113694006 shutdown_connections 2024-04-06T12:31:13.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.916+0000 7feca2f78700 1 --2- 172.21.15.189:0/4113694006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec8c005210 0x7fec8c00ba10 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.916+0000 7feca2f78700 1 -- 172.21.15.189:0/4113694006 >> 172.21.15.189:0/4113694006 conn(0x7fec9c0d00b0 msgr2=0x7fec9c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:13.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.916+0000 7feca2f78700 1 -- 172.21.15.189:0/4113694006 shutdown_connections 2024-04-06T12:31:13.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.916+0000 7feca2f78700 1 -- 172.21.15.189:0/4113694006 wait complete. 2024-04-06T12:31:13.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.918+0000 7feca2f78700 1 Processor -- start 2024-04-06T12:31:13.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.918+0000 7feca2f78700 1 -- start start 2024-04-06T12:31:13.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7feca2f78700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec8c005210 0x7fec9c15a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7feca2f78700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec9c0b0a90 0x7fec9c1f6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7feca2f78700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec9c1f6ec0 0x7fec9c153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7feca2f78700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c0b1ac0 con 0x7fec9c1f6ec0 2024-04-06T12:31:13.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7feca2f78700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c15a850 con 0x7fec8c005210 2024-04-06T12:31:13.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7feca2f78700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c1f64f0 con 0x7fec9c0b0a90 2024-04-06T12:31:13.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7feca1515700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec9c1f6ec0 0x7fec9c153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7feca1515700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec9c1f6ec0 0x7fec9c153ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36372/0 (socket says 172.21.15.189:36372) 2024-04-06T12:31:13.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7feca1515700 1 -- 172.21.15.189:0/1654774391 learned_addr learned my addr 172.21.15.189:0/1654774391 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:13.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7feca0d14700 1 --2- 172.21.15.189:0/1654774391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec8c005210 0x7fec9c15a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7fec9bfff700 1 --2- 172.21.15.189:0/1654774391 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec9c0b0a90 0x7fec9c1f6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7feca1515700 1 -- 172.21.15.189:0/1654774391 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec9c0b0a90 msgr2=0x7fec9c1f6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:13.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7feca1515700 1 --2- 172.21.15.189:0/1654774391 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec9c0b0a90 0x7fec9c1f6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7feca1515700 1 -- 172.21.15.189:0/1654774391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec8c005210 msgr2=0x7fec9c15a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:13.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7feca1515700 1 --2- 172.21.15.189:0/1654774391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec8c005210 0x7fec9c15a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:13.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7feca1515700 1 -- 172.21.15.189:0/1654774391 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9c0b1220 con 0x7fec9c1f6ec0 2024-04-06T12:31:13.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7fec9bfff700 1 --2- 172.21.15.189:0/1654774391 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec9c0b0a90 0x7fec9c1f6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:13.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7feca1515700 1 --2- 172.21.15.189:0/1654774391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec9c1f6ec0 0x7fec9c153ee0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fec9400e700 tx=0x7fec94009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:13.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7fec997fa700 1 -- 172.21.15.189:0/1654774391 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec94003e10 con 0x7fec9c1f6ec0 2024-04-06T12:31:13.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7fec997fa700 1 -- 172.21.15.189:0/1654774391 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec94004230 con 0x7fec9c1f6ec0 2024-04-06T12:31:13.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7feca2f78700 1 -- 172.21.15.189:0/1654774391 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec9c0b1500 con 0x7fec9c1f6ec0 2024-04-06T12:31:13.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.919+0000 7fec997fa700 1 -- 172.21.15.189:0/1654774391 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec940045e0 con 0x7fec9c1f6ec0 2024-04-06T12:31:13.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.920+0000 7feca2f78700 1 -- 172.21.15.189:0/1654774391 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec9c0b0f10 con 0x7fec9c1f6ec0 2024-04-06T12:31:13.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.920+0000 7feca2f78700 1 -- 172.21.15.189:0/1654774391 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec9c1f62c0 con 0x7fec9c1f6ec0 2024-04-06T12:31:13.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.921+0000 7fec997fa700 1 -- 172.21.15.189:0/1654774391 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fec94004740 con 0x7fec9c1f6ec0 2024-04-06T12:31:13.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.921+0000 7fec997fa700 1 --2- 172.21.15.189:0/1654774391 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec8006adb0 0x7fec8006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:13.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.921+0000 7fec997fa700 1 -- 172.21.15.189:0/1654774391 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5951+0+0 (secure 0 0 0) 0x7fec9408a9c0 con 0x7fec9c1f6ec0 2024-04-06T12:31:13.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.921+0000 7feca0d14700 1 --2- 172.21.15.189:0/1654774391 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec8006adb0 0x7fec8006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:13.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.922+0000 7feca0d14700 1 --2- 172.21.15.189:0/1654774391 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec8006adb0 0x7fec8006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fec90000c00 tx=0x7fec90000ed0 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:13.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:13.926+0000 7fec997fa700 1 -- 172.21.15.189:0/1654774391 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fec9c1f62c0 con 0x7fec9c1f6ec0 2024-04-06T12:31:14.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.201+0000 7feca2f78700 1 -- 172.21.15.189:0/1654774391 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7fec9c1f6680 con 0x7fec9c1f6ec0 2024-04-06T12:31:14.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.655+0000 7fec997fa700 1 -- 172.21.15.189:0/1654774391 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v729) v1 ==== 148+0+0 (secure 0 0 0) 0x7fec94018070 con 0x7fec9c1f6ec0 2024-04-06T12:31:14.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.658+0000 7feca2f78700 1 -- 172.21.15.189:0/1654774391 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec8006adb0 msgr2=0x7fec8006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:14.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.658+0000 7feca2f78700 1 --2- 172.21.15.189:0/1654774391 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec8006adb0 0x7fec8006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fec90000c00 tx=0x7fec90000ed0 comp rx=0 tx=0).stop 2024-04-06T12:31:14.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.658+0000 7feca2f78700 1 -- 172.21.15.189:0/1654774391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec9c1f6ec0 msgr2=0x7fec9c153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:14.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.658+0000 7feca2f78700 1 --2- 172.21.15.189:0/1654774391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec9c1f6ec0 0x7fec9c153ee0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fec9400e700 tx=0x7fec94009750 comp rx=0 tx=0).stop 2024-04-06T12:31:14.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.658+0000 7feca2f78700 1 -- 172.21.15.189:0/1654774391 shutdown_connections 2024-04-06T12:31:14.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.658+0000 7feca2f78700 1 --2- 172.21.15.189:0/1654774391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec9c1f6ec0 0x7fec9c153ee0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:14.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.658+0000 7feca2f78700 1 --2- 172.21.15.189:0/1654774391 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec8006adb0 0x7fec8006d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:14.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.658+0000 7feca2f78700 1 --2- 172.21.15.189:0/1654774391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec8c005210 0x7fec9c15a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:14.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.658+0000 7feca2f78700 1 --2- 172.21.15.189:0/1654774391 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec9c0b0a90 0x7fec9c1f6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:14.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.658+0000 7feca2f78700 1 -- 172.21.15.189:0/1654774391 >> 172.21.15.189:0/1654774391 conn(0x7fec9c0d00b0 msgr2=0x7fec9c156df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:14.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.658+0000 7feca2f78700 1 -- 172.21.15.189:0/1654774391 shutdown_connections 2024-04-06T12:31:14.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.658+0000 7feca2f78700 1 -- 172.21.15.189:0/1654774391 wait complete. 2024-04-06T12:31:14.669 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2024-04-06T12:31:14.669 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-06T12:31:14.670 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2024-04-06T12:31:14.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.800+0000 7f1fa7b50700 1 Processor -- start 2024-04-06T12:31:14.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.800+0000 7f1fa7b50700 1 -- start start 2024-04-06T12:31:14.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.801+0000 7f1fa7b50700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1fa006ae40 0x7f1fa00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:14.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.801+0000 7f1fa7b50700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1fa015ddf0 0x7f1fa015e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:14.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.801+0000 7f1fa7b50700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa00b3990 con 0x7f1fa015d7b0 2024-04-06T12:31:14.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.801+0000 7f1fa7b50700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fa00b3060 con 0x7f1fa006ae40 2024-04-06T12:31:14.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.801+0000 7f1fa7b50700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fa00b3340 con 0x7f1fa015ddf0 2024-04-06T12:31:14.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.801+0000 7f1fa50eb700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1fa006ae40 0x7f1fa00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:14.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.801+0000 7f1fa50eb700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1fa006ae40 0x7f1fa00b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36388/0 (socket says 172.21.15.189:36388) 2024-04-06T12:31:14.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.801+0000 7f1fa50eb700 1 -- 172.21.15.189:0/1070178263 learned_addr learned my addr 172.21.15.189:0/1070178263 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:14.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.801+0000 7f1fa50eb700 1 -- 172.21.15.189:0/1070178263 >> v2:172.21.15.192:3300/0 conn(0x7f1fa015ddf0 msgr2=0x7f1fa015e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:14.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.801+0000 7f1fa50eb700 1 --2- 172.21.15.189:0/1070178263 >> v2:172.21.15.192:3300/0 conn(0x7f1fa015ddf0 0x7f1fa015e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:14.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.801+0000 7f1fa50eb700 1 -- 172.21.15.189:0/1070178263 >> v1:172.21.15.192:6789/0 conn(0x7f1fa015d7b0 legacy=0x7f1fa006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:14.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.801+0000 7f1fa50eb700 1 -- 172.21.15.189:0/1070178263 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa00b3620 con 0x7f1fa006ae40 2024-04-06T12:31:14.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.801+0000 7f1fa50eb700 1 --2- 172.21.15.189:0/1070178263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fa006ae40 0x7f1fa00b3e20 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f1fa006d2c0 tx=0x7f1f90009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25bff9c2264d2219 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:14.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.801+0000 7f1fa48ea700 1 -- 172.21.15.189:0/1070178263 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f90004230 con 0x7f1fa006ae40 2024-04-06T12:31:14.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.801+0000 7f1fa48ea700 1 -- 172.21.15.189:0/1070178263 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f90004390 con 0x7f1fa006ae40 2024-04-06T12:31:14.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.801+0000 7f1fa48ea700 1 -- 172.21.15.189:0/1070178263 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f900036a0 con 0x7f1fa006ae40 2024-04-06T12:31:14.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.802+0000 7f1fa7b50700 1 -- 172.21.15.189:0/1070178263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fa006ae40 msgr2=0x7f1fa00b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:14.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.802+0000 7f1fa7b50700 1 --2- 172.21.15.189:0/1070178263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fa006ae40 0x7f1fa00b3e20 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f1fa006d2c0 tx=0x7f1f90009bd0 comp rx=0 tx=0).stop 2024-04-06T12:31:14.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.802+0000 7f1fa7b50700 1 -- 172.21.15.189:0/1070178263 shutdown_connections 2024-04-06T12:31:14.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.802+0000 7f1fa7b50700 1 --2- 172.21.15.189:0/1070178263 >> v2:172.21.15.192:3300/0 conn(0x7f1fa015ddf0 0x7f1fa015e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:14.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.802+0000 7f1fa7b50700 1 --2- 172.21.15.189:0/1070178263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fa006ae40 0x7f1fa00b3e20 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:14.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.802+0000 7f1fa7b50700 1 -- 172.21.15.189:0/1070178263 >> 172.21.15.189:0/1070178263 conn(0x7f1fa00d00b0 msgr2=0x7f1fa00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:14.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.802+0000 7f1fa7b50700 1 -- 172.21.15.189:0/1070178263 shutdown_connections 2024-04-06T12:31:14.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.802+0000 7f1fa7b50700 1 -- 172.21.15.189:0/1070178263 wait complete. 2024-04-06T12:31:14.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.803+0000 7f1fa7b50700 1 Processor -- start 2024-04-06T12:31:14.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.803+0000 7f1fa7b50700 1 -- start start 2024-04-06T12:31:14.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.804+0000 7f1fa7b50700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fa006ae40 0x7f1fa00cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:14.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.804+0000 7f1fa7b50700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fa015d7b0 0x7f1fa00aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:14.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.804+0000 7f1fa7b50700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fa015ddf0 0x7f1fa0160220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:14.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.804+0000 7f1fa7b50700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa01536e0 con 0x7f1fa015ddf0 2024-04-06T12:31:14.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.804+0000 7f1fa7b50700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa01f5f20 con 0x7f1fa006ae40 2024-04-06T12:31:14.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.804+0000 7f1fa7b50700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa01f6200 con 0x7f1fa015d7b0 2024-04-06T12:31:14.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.804+0000 7f1fa60ed700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fa015ddf0 0x7f1fa0160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:14.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.804+0000 7f1fa60ed700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fa015ddf0 0x7f1fa0160220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36404/0 (socket says 172.21.15.189:36404) 2024-04-06T12:31:14.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.804+0000 7f1fa60ed700 1 -- 172.21.15.189:0/3621954678 learned_addr learned my addr 172.21.15.189:0/3621954678 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:14.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.804+0000 7f1fa58ec700 1 --2- 172.21.15.189:0/3621954678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fa006ae40 0x7f1fa00cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:14.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.804+0000 7f1fa50eb700 1 --2- 172.21.15.189:0/3621954678 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fa015d7b0 0x7f1fa00aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:14.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.804+0000 7f1fa60ed700 1 -- 172.21.15.189:0/3621954678 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fa015d7b0 msgr2=0x7f1fa00aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:14.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.804+0000 7f1fa60ed700 1 --2- 172.21.15.189:0/3621954678 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fa015d7b0 0x7f1fa00aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:14.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.804+0000 7f1fa60ed700 1 -- 172.21.15.189:0/3621954678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fa006ae40 msgr2=0x7f1fa00cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:14.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.804+0000 7f1fa60ed700 1 --2- 172.21.15.189:0/3621954678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fa006ae40 0x7f1fa00cefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:14.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.804+0000 7f1fa60ed700 1 -- 172.21.15.189:0/3621954678 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f90009550 con 0x7f1fa015ddf0 2024-04-06T12:31:14.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.804+0000 7f1fa58ec700 1 --2- 172.21.15.189:0/3621954678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fa006ae40 0x7f1fa00cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:14.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.805+0000 7f1fa60ed700 1 --2- 172.21.15.189:0/3621954678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fa015ddf0 0x7f1fa0160220 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f1f9800b970 tx=0x7f1f9800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:14.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.805+0000 7f1f967fc700 1 -- 172.21.15.189:0/3621954678 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9800c9b0 con 0x7f1fa015ddf0 2024-04-06T12:31:14.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.805+0000 7f1f967fc700 1 -- 172.21.15.189:0/3621954678 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9800cb30 con 0x7f1fa015ddf0 2024-04-06T12:31:14.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.805+0000 7f1fa7b50700 1 -- 172.21.15.189:0/3621954678 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fa01f64e0 con 0x7f1fa015ddf0 2024-04-06T12:31:14.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.805+0000 7f1f967fc700 1 -- 172.21.15.189:0/3621954678 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f98014440 con 0x7f1fa015ddf0 2024-04-06T12:31:14.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.805+0000 7f1fa7b50700 1 -- 172.21.15.189:0/3621954678 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fa00af660 con 0x7f1fa015ddf0 2024-04-06T12:31:14.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.806+0000 7f1fa7b50700 1 -- 172.21.15.189:0/3621954678 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa00af880 con 0x7f1fa015ddf0 2024-04-06T12:31:14.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.806+0000 7f1f967fc700 1 -- 172.21.15.189:0/3621954678 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f1f9801d3b0 con 0x7f1fa015ddf0 2024-04-06T12:31:14.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.806+0000 7f1f967fc700 1 --2- 172.21.15.189:0/3621954678 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1f8406adb0 0x7f1f8406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:14.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.806+0000 7f1f967fc700 1 -- 172.21.15.189:0/3621954678 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f1f980892e0 con 0x7f1fa015ddf0 2024-04-06T12:31:14.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.806+0000 7f1fa58ec700 1 --2- 172.21.15.189:0/3621954678 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1f8406adb0 0x7f1f8406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:14.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.807+0000 7f1fa58ec700 1 --2- 172.21.15.189:0/3621954678 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1f8406adb0 0x7f1f8406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c0017d0 tx=0x7f1f9c009040 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:14.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:14.811+0000 7f1f967fc700 1 -- 172.21.15.189:0/3621954678 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1fa00af880 con 0x7f1fa015ddf0 2024-04-06T12:31:15.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.087+0000 7f1fa7b50700 1 -- 172.21.15.189:0/3621954678 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7f1fa01f6ce0 con 0x7f1fa015ddf0 2024-04-06T12:31:15.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.656+0000 7f1f967fc700 1 -- 172.21.15.189:0/3621954678 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v730) v1 ==== 163+0+0 (secure 0 0 0) 0x7f1fa01f6ce0 con 0x7f1fa015ddf0 2024-04-06T12:31:15.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.659+0000 7f1fa7b50700 1 -- 172.21.15.189:0/3621954678 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1f8406adb0 msgr2=0x7f1f8406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:15.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.659+0000 7f1fa7b50700 1 --2- 172.21.15.189:0/3621954678 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1f8406adb0 0x7f1f8406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c0017d0 tx=0x7f1f9c009040 comp rx=0 tx=0).stop 2024-04-06T12:31:15.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.659+0000 7f1fa7b50700 1 -- 172.21.15.189:0/3621954678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fa015ddf0 msgr2=0x7f1fa0160220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:15.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.659+0000 7f1fa7b50700 1 --2- 172.21.15.189:0/3621954678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fa015ddf0 0x7f1fa0160220 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f1f9800b970 tx=0x7f1f9800bd30 comp rx=0 tx=0).stop 2024-04-06T12:31:15.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.659+0000 7f1fa7b50700 1 -- 172.21.15.189:0/3621954678 shutdown_connections 2024-04-06T12:31:15.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.659+0000 7f1fa7b50700 1 --2- 172.21.15.189:0/3621954678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fa015ddf0 0x7f1fa0160220 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:15.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.659+0000 7f1fa7b50700 1 --2- 172.21.15.189:0/3621954678 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1f8406adb0 0x7f1f8406d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:15.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.659+0000 7f1fa7b50700 1 --2- 172.21.15.189:0/3621954678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fa006ae40 0x7f1fa00cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:15.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.659+0000 7f1fa7b50700 1 --2- 172.21.15.189:0/3621954678 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fa015d7b0 0x7f1fa00aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:15.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.659+0000 7f1fa7b50700 1 -- 172.21.15.189:0/3621954678 >> 172.21.15.189:0/3621954678 conn(0x7f1fa00d00b0 msgr2=0x7f1fa015ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:15.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.659+0000 7f1fa7b50700 1 -- 172.21.15.189:0/3621954678 shutdown_connections 2024-04-06T12:31:15.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.659+0000 7f1fa7b50700 1 -- 172.21.15.189:0/3621954678 wait complete. 2024-04-06T12:31:15.670 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-06T12:31:15.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.801+0000 7f0d77c2a700 1 Processor -- start 2024-04-06T12:31:15.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.801+0000 7f0d77c2a700 1 -- start start 2024-04-06T12:31:15.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.802+0000 7f0d77c2a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0d7006ae40 0x7f0d700b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:15.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.802+0000 7f0d77c2a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d700b3990 con 0x7f0d7015d7b0 2024-04-06T12:31:15.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.802+0000 7f0d77c2a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d700b3060 con 0x7f0d7015ddf0 2024-04-06T12:31:15.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.802+0000 7f0d77c2a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d7015e950 con 0x7f0d7006ae40 2024-04-06T12:31:15.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.802+0000 7f0d751c5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0d7006ae40 0x7f0d700b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:15.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.802+0000 7f0d751c5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0d7006ae40 0x7f0d700b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36416/0 (socket says 172.21.15.189:36416) 2024-04-06T12:31:15.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.802+0000 7f0d751c5700 1 -- 172.21.15.189:0/3673251445 learned_addr learned my addr 172.21.15.189:0/3673251445 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:15.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.802+0000 7f0d759c6700 1 --1- 172.21.15.189:0/3673251445 >> v1:172.21.15.189:6789/0 conn(0x7f0d7015d7b0 0x7f0d7006a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55908/0 (socket says 172.21.15.189:55908) 2024-04-06T12:31:15.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.802+0000 7f0d751c5700 1 -- 172.21.15.189:0/3673251445 >> v1:172.21.15.192:6789/0 conn(0x7f0d7015ddf0 legacy=0x7f0d7015e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:15.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.802+0000 7f0d751c5700 1 -- 172.21.15.189:0/3673251445 >> v1:172.21.15.189:6789/0 conn(0x7f0d7015d7b0 legacy=0x7f0d7006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:15.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.802+0000 7f0d751c5700 1 -- 172.21.15.189:0/3673251445 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d700b3340 con 0x7f0d7006ae40 2024-04-06T12:31:15.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.803+0000 7f0d751c5700 1 --2- 172.21.15.189:0/3673251445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d7006ae40 0x7f0d700b3e20 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c0098f0 tx=0x7f0d6c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ad0e418a53f3fe server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:15.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.803+0000 7f0d749c4700 1 -- 172.21.15.189:0/3673251445 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c004230 con 0x7f0d7006ae40 2024-04-06T12:31:15.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.803+0000 7f0d749c4700 1 -- 172.21.15.189:0/3673251445 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6c004390 con 0x7f0d7006ae40 2024-04-06T12:31:15.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.803+0000 7f0d749c4700 1 -- 172.21.15.189:0/3673251445 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c0036a0 con 0x7f0d7006ae40 2024-04-06T12:31:15.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.803+0000 7f0d77c2a700 1 -- 172.21.15.189:0/3673251445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d7006ae40 msgr2=0x7f0d700b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:15.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.803+0000 7f0d77c2a700 1 --2- 172.21.15.189:0/3673251445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d7006ae40 0x7f0d700b3e20 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c0098f0 tx=0x7f0d6c009c00 comp rx=0 tx=0).stop 2024-04-06T12:31:15.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.803+0000 7f0d77c2a700 1 -- 172.21.15.189:0/3673251445 shutdown_connections 2024-04-06T12:31:15.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.803+0000 7f0d77c2a700 1 --2- 172.21.15.189:0/3673251445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d7006ae40 0x7f0d700b3e20 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:15.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.803+0000 7f0d77c2a700 1 -- 172.21.15.189:0/3673251445 >> 172.21.15.189:0/3673251445 conn(0x7f0d700d00b0 msgr2=0x7f0d700d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:15.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.803+0000 7f0d77c2a700 1 -- 172.21.15.189:0/3673251445 shutdown_connections 2024-04-06T12:31:15.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.803+0000 7f0d77c2a700 1 -- 172.21.15.189:0/3673251445 wait complete. 2024-04-06T12:31:15.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.804+0000 7f0d77c2a700 1 Processor -- start 2024-04-06T12:31:15.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.805+0000 7f0d77c2a700 1 -- start start 2024-04-06T12:31:15.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.805+0000 7f0d77c2a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d7006ae40 0x7f0d700cef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:15.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.805+0000 7f0d77c2a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d7015d7b0 0x7f0d70142480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:15.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.805+0000 7f0d77c2a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d7015ddf0 0x7f0d7015b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:15.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.805+0000 7f0d77c2a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7015b520 con 0x7f0d7015ddf0 2024-04-06T12:31:15.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.805+0000 7f0d77c2a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d701429c0 con 0x7f0d7015d7b0 2024-04-06T12:31:15.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.805+0000 7f0d77c2a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7015af30 con 0x7f0d7006ae40 2024-04-06T12:31:15.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.805+0000 7f0d761c7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d7015ddf0 0x7f0d7015b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:15.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.805+0000 7f0d761c7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d7015ddf0 0x7f0d7015b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36418/0 (socket says 172.21.15.189:36418) 2024-04-06T12:31:15.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.805+0000 7f0d761c7700 1 -- 172.21.15.189:0/1245916518 learned_addr learned my addr 172.21.15.189:0/1245916518 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:15.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.806+0000 7f0d759c6700 1 --2- 172.21.15.189:0/1245916518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d7006ae40 0x7f0d700cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:15.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.806+0000 7f0d751c5700 1 --2- 172.21.15.189:0/1245916518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d7015d7b0 0x7f0d70142480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:15.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.806+0000 7f0d761c7700 1 -- 172.21.15.189:0/1245916518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d7006ae40 msgr2=0x7f0d700cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:15.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.806+0000 7f0d761c7700 1 --2- 172.21.15.189:0/1245916518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d7006ae40 0x7f0d700cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:15.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.806+0000 7f0d761c7700 1 -- 172.21.15.189:0/1245916518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d7015d7b0 msgr2=0x7f0d70142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:15.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.806+0000 7f0d761c7700 1 --2- 172.21.15.189:0/1245916518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d7015d7b0 0x7f0d70142480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:15.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.806+0000 7f0d761c7700 1 -- 172.21.15.189:0/1245916518 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c009550 con 0x7f0d7015ddf0 2024-04-06T12:31:15.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.806+0000 7f0d761c7700 1 --2- 172.21.15.189:0/1245916518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d7015ddf0 0x7f0d7015b940 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f0d6800b970 tx=0x7f0d6800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:15.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.806+0000 7f0d667fc700 1 -- 172.21.15.189:0/1245916518 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6800c9b0 con 0x7f0d7015ddf0 2024-04-06T12:31:15.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.806+0000 7f0d667fc700 1 -- 172.21.15.189:0/1245916518 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6800cb10 con 0x7f0d7015ddf0 2024-04-06T12:31:15.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.806+0000 7f0d667fc700 1 -- 172.21.15.189:0/1245916518 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d68014440 con 0x7f0d7015ddf0 2024-04-06T12:31:15.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.806+0000 7f0d77c2a700 1 -- 172.21.15.189:0/1245916518 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d700af1e0 con 0x7f0d7015ddf0 2024-04-06T12:31:15.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.806+0000 7f0d77c2a700 1 -- 172.21.15.189:0/1245916518 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d70152590 con 0x7f0d7015ddf0 2024-04-06T12:31:15.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.807+0000 7f0d77c2a700 1 -- 172.21.15.189:0/1245916518 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d7015b660 con 0x7f0d7015ddf0 2024-04-06T12:31:15.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.807+0000 7f0d667fc700 1 -- 172.21.15.189:0/1245916518 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0d6801d3b0 con 0x7f0d7015ddf0 2024-04-06T12:31:15.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.808+0000 7f0d667fc700 1 --2- 172.21.15.189:0/1245916518 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0d5406adb0 0x7f0d5406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:15.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.808+0000 7f0d667fc700 1 -- 172.21.15.189:0/1245916518 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f0d68088fa0 con 0x7f0d7015ddf0 2024-04-06T12:31:15.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.808+0000 7f0d759c6700 1 --2- 172.21.15.189:0/1245916518 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0d5406adb0 0x7f0d5406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:15.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.808+0000 7f0d759c6700 1 --2- 172.21.15.189:0/1245916518 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0d5406adb0 0x7f0d5406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0d701f6740 tx=0x7f0d60007d70 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:15.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:15.812+0000 7f0d667fc700 1 -- 172.21.15.189:0/1245916518 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0d680bc650 con 0x7f0d7015ddf0 2024-04-06T12:31:16.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.007+0000 7f0d77c2a700 1 -- 172.21.15.189:0/1245916518 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0d70067bb0 con 0x7f0d7015ddf0 2024-04-06T12:31:16.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.011+0000 7f0d667fc700 1 -- 172.21.15.189:0/1245916518 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0d7015b660 con 0x7f0d7015ddf0 2024-04-06T12:31:16.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.013+0000 7f0d77c2a700 1 -- 172.21.15.189:0/1245916518 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0d5406adb0 msgr2=0x7f0d5406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:16.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.013+0000 7f0d77c2a700 1 --2- 172.21.15.189:0/1245916518 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0d5406adb0 0x7f0d5406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0d701f6740 tx=0x7f0d60007d70 comp rx=0 tx=0).stop 2024-04-06T12:31:16.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.013+0000 7f0d77c2a700 1 -- 172.21.15.189:0/1245916518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d7015ddf0 msgr2=0x7f0d7015b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:16.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.013+0000 7f0d77c2a700 1 --2- 172.21.15.189:0/1245916518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d7015ddf0 0x7f0d7015b940 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f0d6800b970 tx=0x7f0d6800bd30 comp rx=0 tx=0).stop 2024-04-06T12:31:16.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.013+0000 7f0d77c2a700 1 -- 172.21.15.189:0/1245916518 shutdown_connections 2024-04-06T12:31:16.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.013+0000 7f0d77c2a700 1 --2- 172.21.15.189:0/1245916518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d7015ddf0 0x7f0d7015b940 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:16.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.013+0000 7f0d77c2a700 1 --2- 172.21.15.189:0/1245916518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d7006ae40 0x7f0d700cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:16.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.013+0000 7f0d77c2a700 1 --2- 172.21.15.189:0/1245916518 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0d5406adb0 0x7f0d5406d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:16.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.013+0000 7f0d77c2a700 1 --2- 172.21.15.189:0/1245916518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d7015d7b0 0x7f0d70142480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:16.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.013+0000 7f0d77c2a700 1 -- 172.21.15.189:0/1245916518 >> 172.21.15.189:0/1245916518 conn(0x7f0d700d00b0 msgr2=0x7f0d7006eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:16.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.013+0000 7f0d77c2a700 1 -- 172.21.15.189:0/1245916518 shutdown_connections 2024-04-06T12:31:16.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.013+0000 7f0d77c2a700 1 -- 172.21.15.189:0/1245916518 wait complete. 2024-04-06T12:31:16.023 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T12:31:16.023 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:31:16.061 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:31:16.114 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T12:31:16.115 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:31:16.115 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:31:16.132 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:31:16.133 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:31:16.133 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:31:16.153 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:31:16.153 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:31:16.153 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:31:16.190 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:31:16.190 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:31:16.190 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:31:16.212 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:31:16.212 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-06T12:31:16.212 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:31:16.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.354+0000 7fdd8a90b700 1 Processor -- start 2024-04-06T12:31:16.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.354+0000 7fdd8a90b700 1 -- start start 2024-04-06T12:31:16.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.355+0000 7fdd8a90b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdd8415d7b0 0x7fdd840b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:16.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.355+0000 7fdd8a90b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd840b3990 con 0x7fdd8415e500 2024-04-06T12:31:16.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.355+0000 7fdd8a90b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd840b3060 con 0x7fdd8406a730 2024-04-06T12:31:16.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.355+0000 7fdd8a90b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd8415e950 con 0x7fdd8415d7b0 2024-04-06T12:31:16.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.355+0000 7fdd83fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdd8415d7b0 0x7fdd840b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:16.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.355+0000 7fdd83fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdd8415d7b0 0x7fdd840b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36432/0 (socket says 172.21.15.189:36432) 2024-04-06T12:31:16.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.355+0000 7fdd83fff700 1 -- 172.21.15.189:0/1810793820 learned_addr learned my addr 172.21.15.189:0/1810793820 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:16.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.355+0000 7fdd88ea8700 1 --1- 172.21.15.189:0/1810793820 >> v1:172.21.15.189:6789/0 conn(0x7fdd8415e500 0x7fdd840717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55916/0 (socket says 172.21.15.189:55916) 2024-04-06T12:31:16.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.355+0000 7fdd83fff700 1 -- 172.21.15.189:0/1810793820 >> v1:172.21.15.192:6789/0 conn(0x7fdd8406a730 legacy=0x7fdd8415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:16.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.355+0000 7fdd83fff700 1 -- 172.21.15.189:0/1810793820 >> v1:172.21.15.189:6789/0 conn(0x7fdd8415e500 legacy=0x7fdd840717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:16.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.355+0000 7fdd82ffd700 1 -- 172.21.15.189:0/1810793820 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131214133 0 0) 0x7fdd840b3990 con 0x7fdd8415e500 2024-04-06T12:31:16.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.356+0000 7fdd83fff700 1 -- 172.21.15.189:0/1810793820 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd840b3340 con 0x7fdd8415d7b0 2024-04-06T12:31:16.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.356+0000 7fdd83fff700 1 --2- 172.21.15.189:0/1810793820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd8415d7b0 0x7fdd840b3e20 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fdd780098f0 tx=0x7fdd78009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe4ab4b5bc1dac0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:16.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.356+0000 7fdd82ffd700 1 -- 172.21.15.189:0/1810793820 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7800b8a0 con 0x7fdd8415d7b0 2024-04-06T12:31:16.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.356+0000 7fdd82ffd700 1 -- 172.21.15.189:0/1810793820 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7800ba00 con 0x7fdd8415d7b0 2024-04-06T12:31:16.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.356+0000 7fdd82ffd700 1 -- 172.21.15.189:0/1810793820 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7800bd90 con 0x7fdd8415d7b0 2024-04-06T12:31:16.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.356+0000 7fdd8a90b700 1 -- 172.21.15.189:0/1810793820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd8415d7b0 msgr2=0x7fdd840b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:16.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.356+0000 7fdd8a90b700 1 --2- 172.21.15.189:0/1810793820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd8415d7b0 0x7fdd840b3e20 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fdd780098f0 tx=0x7fdd78009c00 comp rx=0 tx=0).stop 2024-04-06T12:31:16.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.356+0000 7fdd8a90b700 1 -- 172.21.15.189:0/1810793820 shutdown_connections 2024-04-06T12:31:16.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.356+0000 7fdd8a90b700 1 --2- 172.21.15.189:0/1810793820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd8415d7b0 0x7fdd840b3e20 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:16.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.356+0000 7fdd8a90b700 1 -- 172.21.15.189:0/1810793820 >> 172.21.15.189:0/1810793820 conn(0x7fdd840d00b0 msgr2=0x7fdd840d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:16.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.356+0000 7fdd8a90b700 1 -- 172.21.15.189:0/1810793820 shutdown_connections 2024-04-06T12:31:16.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.356+0000 7fdd8a90b700 1 -- 172.21.15.189:0/1810793820 wait complete. 2024-04-06T12:31:16.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.358+0000 7fdd8a90b700 1 Processor -- start 2024-04-06T12:31:16.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.358+0000 7fdd8a90b700 1 -- start start 2024-04-06T12:31:16.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.358+0000 7fdd8a90b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd8406a730 0x7fdd84153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:16.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.358+0000 7fdd8a90b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd8415d7b0 0x7fdd840ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:16.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.358+0000 7fdd8a90b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd8415e500 0x7fdd8415c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:16.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.358+0000 7fdd8a90b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd840af270 con 0x7fdd8415d7b0 2024-04-06T12:31:16.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.359+0000 7fdd8a90b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd840cee50 con 0x7fdd8415e500 2024-04-06T12:31:16.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.359+0000 7fdd8a90b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd840cf130 con 0x7fdd8406a730 2024-04-06T12:31:16.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.359+0000 7fdd837fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd8415d7b0 0x7fdd840ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:16.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.359+0000 7fdd837fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd8415d7b0 0x7fdd840ce3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36440/0 (socket says 172.21.15.189:36440) 2024-04-06T12:31:16.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.359+0000 7fdd837fe700 1 -- 172.21.15.189:0/453979105 learned_addr learned my addr 172.21.15.189:0/453979105 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:16.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.359+0000 7fdd83fff700 1 --2- 172.21.15.189:0/453979105 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd8406a730 0x7fdd84153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:16.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.359+0000 7fdd88ea8700 1 --2- 172.21.15.189:0/453979105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd8415e500 0x7fdd8415c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:16.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.359+0000 7fdd837fe700 1 -- 172.21.15.189:0/453979105 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd8406a730 msgr2=0x7fdd84153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:16.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.359+0000 7fdd837fe700 1 --2- 172.21.15.189:0/453979105 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd8406a730 0x7fdd84153ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:16.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.359+0000 7fdd837fe700 1 -- 172.21.15.189:0/453979105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd8415e500 msgr2=0x7fdd8415c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:16.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.359+0000 7fdd837fe700 1 --2- 172.21.15.189:0/453979105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd8415e500 0x7fdd8415c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:16.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.359+0000 7fdd837fe700 1 -- 172.21.15.189:0/453979105 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd78009550 con 0x7fdd8415d7b0 2024-04-06T12:31:16.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.359+0000 7fdd837fe700 1 --2- 172.21.15.189:0/453979105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd8415d7b0 0x7fdd840ce3d0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fdd7400c700 tx=0x7fdd7400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:16.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.359+0000 7fdd80ff9700 1 -- 172.21.15.189:0/453979105 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd740052d0 con 0x7fdd8415d7b0 2024-04-06T12:31:16.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.359+0000 7fdd80ff9700 1 -- 172.21.15.189:0/453979105 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd74005430 con 0x7fdd8415d7b0 2024-04-06T12:31:16.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.359+0000 7fdd8a90b700 1 -- 172.21.15.189:0/453979105 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd840aec20 con 0x7fdd8415d7b0 2024-04-06T12:31:16.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.359+0000 7fdd80ff9700 1 -- 172.21.15.189:0/453979105 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd74003880 con 0x7fdd8415d7b0 2024-04-06T12:31:16.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.359+0000 7fdd8a90b700 1 -- 172.21.15.189:0/453979105 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd840cd6d0 con 0x7fdd8415d7b0 2024-04-06T12:31:16.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.360+0000 7fdd8a90b700 1 -- 172.21.15.189:0/453979105 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd840af420 con 0x7fdd8415d7b0 2024-04-06T12:31:16.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.361+0000 7fdd80ff9700 1 -- 172.21.15.189:0/453979105 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fdd740039e0 con 0x7fdd8415d7b0 2024-04-06T12:31:16.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.361+0000 7fdd80ff9700 1 --2- 172.21.15.189:0/453979105 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdd7006adb0 0x7fdd7006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:16.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.361+0000 7fdd83fff700 1 --2- 172.21.15.189:0/453979105 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdd7006adb0 0x7fdd7006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:16.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.361+0000 7fdd80ff9700 1 -- 172.21.15.189:0/453979105 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5951+0+0 (secure 0 0 0) 0x7fdd74088800 con 0x7fdd8415d7b0 2024-04-06T12:31:16.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.362+0000 7fdd83fff700 1 --2- 172.21.15.189:0/453979105 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdd7006adb0 0x7fdd7006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdd78000c00 tx=0x7fdd78008030 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:16.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.366+0000 7fdd80ff9700 1 -- 172.21.15.189:0/453979105 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdd840af420 con 0x7fdd8415d7b0 2024-04-06T12:31:16.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.607+0000 7fdd8a90b700 1 -- 172.21.15.189:0/453979105 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd84067bb0 con 0x7fdd8415d7b0 2024-04-06T12:31:16.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.607+0000 7fdd80ff9700 1 -- 172.21.15.189:0/453979105 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+3535 (secure 0 0 0) 0x7fdd74017090 con 0x7fdd8415d7b0 2024-04-06T12:31:16.605 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:16.605 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":56,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/749266547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":749266547},{"type":"v1","addr":"172.21.15.192:6835","nonce":749266547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":6764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/947183051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":947183051},{"type":"v1","addr":"172.21.15.192:6834","nonce":947183051}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":7492,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3990051444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3990051444},{"type":"v1","addr":"172.21.15.189:6837","nonce":3990051444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":8508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2469285938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2469285938},{"type":"v1","addr":"172.21.15.189:6836","nonce":2469285938}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":56}],"filesystems":[]} 2024-04-06T12:31:16.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.609+0000 7fdd8a90b700 1 -- 172.21.15.189:0/453979105 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdd7006adb0 msgr2=0x7fdd7006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:16.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.609+0000 7fdd8a90b700 1 --2- 172.21.15.189:0/453979105 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdd7006adb0 0x7fdd7006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdd78000c00 tx=0x7fdd78008030 comp rx=0 tx=0).stop 2024-04-06T12:31:16.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.609+0000 7fdd8a90b700 1 -- 172.21.15.189:0/453979105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd8415d7b0 msgr2=0x7fdd840ce3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:16.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.609+0000 7fdd8a90b700 1 --2- 172.21.15.189:0/453979105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd8415d7b0 0x7fdd840ce3d0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fdd7400c700 tx=0x7fdd7400ca10 comp rx=0 tx=0).stop 2024-04-06T12:31:16.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.609+0000 7fdd8a90b700 1 -- 172.21.15.189:0/453979105 shutdown_connections 2024-04-06T12:31:16.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.609+0000 7fdd8a90b700 1 --2- 172.21.15.189:0/453979105 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdd7006adb0 0x7fdd7006d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:16.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.609+0000 7fdd8a90b700 1 --2- 172.21.15.189:0/453979105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd8415e500 0x7fdd8415c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:16.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.609+0000 7fdd8a90b700 1 --2- 172.21.15.189:0/453979105 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd8406a730 0x7fdd84153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:16.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.609+0000 7fdd8a90b700 1 --2- 172.21.15.189:0/453979105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd8415d7b0 0x7fdd840ce3d0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:16.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.609+0000 7fdd8a90b700 1 -- 172.21.15.189:0/453979105 >> 172.21.15.189:0/453979105 conn(0x7fdd840d00b0 msgr2=0x7fdd8415f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:16.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.610+0000 7fdd8a90b700 1 -- 172.21.15.189:0/453979105 shutdown_connections 2024-04-06T12:31:16.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.610+0000 7fdd8a90b700 1 -- 172.21.15.189:0/453979105 wait complete. 2024-04-06T12:31:16.608 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 56 2024-04-06T12:31:16.619 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-06T12:31:16.619 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-06T12:31:16.619 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-06T12:31:16.620 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:31:16.620 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-06T12:31:16.620 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-06T12:31:16.620 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-06T12:31:16.620 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:31:16.620 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-06T12:31:16.620 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-06T12:31:16.621 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-06T12:31:16.621 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:31:16.621 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-06T12:31:16.621 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-06T12:31:16.621 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-06T12:31:16.621 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:31:16.628 INFO:tasks.ceph.mds.a:Stopped 2024-04-06T12:31:16.629 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-06T12:31:16.630 INFO:tasks.ceph.mds.c:Stopped 2024-04-06T12:31:16.630 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-06T12:31:16.632 INFO:tasks.ceph.mds.b:Stopped 2024-04-06T12:31:16.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-06T12:31:16.633 INFO:tasks.ceph.mds.d:Stopped 2024-04-06T12:31:16.633 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-06T12:31:16.671 INFO:tasks.ceph.mds.a:Started 2024-04-06T12:31:16.672 DEBUG:teuthology.parallel:result is None 2024-04-06T12:31:16.672 INFO:tasks.ceph.mds.c:Started 2024-04-06T12:31:16.673 DEBUG:teuthology.parallel:result is None 2024-04-06T12:31:16.674 INFO:tasks.ceph.mds.b:Started 2024-04-06T12:31:16.675 DEBUG:teuthology.parallel:result is None 2024-04-06T12:31:16.675 INFO:tasks.ceph.mds.d:Started 2024-04-06T12:31:16.676 DEBUG:teuthology.parallel:result is None 2024-04-06T12:31:16.676 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-06T12:31:16.752 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2024-04-06T12:31:16.752 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T12:31:16.753 INFO:tasks.ceph.mds.a.smithi189.stderr:2024-04-06T12:31:16.756+0000 7f5a6317f700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-04-06T12:31:16.762 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T12:31:16.763 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2024-04-06T12:31:16.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.809+0000 7f9ab6217700 1 Processor -- start 2024-04-06T12:31:16.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.809+0000 7f9ab6217700 1 -- start start 2024-04-06T12:31:16.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.810+0000 7f9ab6217700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9ab015d7e0 0x7f9ab00d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:16.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.810+0000 7f9ab6217700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9ab00d0890 0x7f9ab00d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:16.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.810+0000 7f9ab6217700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ab00d1af0 0x7f9ab00b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:16.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.810+0000 7f9ab6217700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ab00d1f80 con 0x7f9ab00d0890 2024-04-06T12:31:16.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.810+0000 7f9ab6217700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ab00cdf80 con 0x7f9ab015d7e0 2024-04-06T12:31:16.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.810+0000 7f9ab6217700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab00d0cf0 con 0x7f9ab00d1af0 2024-04-06T12:31:16.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.810+0000 7f9aaeffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9ab00d0890 0x7f9ab00d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:16.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.810+0000 7f9aaeffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9ab00d0890 0x7f9ab00d15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36498/0 (socket says 172.21.15.189:36498) 2024-04-06T12:31:16.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.810+0000 7f9aaeffd700 1 -- 172.21.15.189:0/4082271537 learned_addr learned my addr 172.21.15.189:0/4082271537 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:16.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.810+0000 7f9aaffff700 1 --2- 172.21.15.189:0/4082271537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ab00d1af0 0x7f9ab00b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:16.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.810+0000 7f9aaf7fe700 1 --2- 172.21.15.189:0/4082271537 >> v2:172.21.15.192:3300/0 conn(0x7f9ab015d7e0 0x7f9ab00d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:16.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.810+0000 7f9aaeffd700 1 -- 172.21.15.189:0/4082271537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ab00d1af0 msgr2=0x7f9ab00b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:16.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.810+0000 7f9aaeffd700 1 --2- 172.21.15.189:0/4082271537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ab00d1af0 0x7f9ab00b4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:16.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.810+0000 7f9aaeffd700 1 -- 172.21.15.189:0/4082271537 >> v2:172.21.15.192:3300/0 conn(0x7f9ab015d7e0 msgr2=0x7f9ab00d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:16.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.810+0000 7f9aaeffd700 1 --2- 172.21.15.189:0/4082271537 >> v2:172.21.15.192:3300/0 conn(0x7f9ab015d7e0 0x7f9ab00d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:16.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.810+0000 7f9aaeffd700 1 -- 172.21.15.189:0/4082271537 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab00d3230 con 0x7f9ab00d0890 2024-04-06T12:31:16.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.810+0000 7f9aaf7fe700 1 --2- 172.21.15.189:0/4082271537 >> v2:172.21.15.192:3300/0 conn(0x7f9ab015d7e0 0x7f9ab00d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:16.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.810+0000 7f9aaffff700 1 --2- 172.21.15.189:0/4082271537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ab00d1af0 0x7f9ab00b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:16.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.810+0000 7f9aaeffd700 1 --2- 172.21.15.189:0/4082271537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ab00d0890 0x7f9ab00d15b0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f9ab008c190 tx=0x7f9aa0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be0aa07f5f36737b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:16.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.811+0000 7f9aae7fc700 1 -- 172.21.15.189:0/4082271537 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa000b870 con 0x7f9ab00d0890 2024-04-06T12:31:16.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.811+0000 7f9aae7fc700 1 -- 172.21.15.189:0/4082271537 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa000b9d0 con 0x7f9ab00d0890 2024-04-06T12:31:16.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.811+0000 7f9aae7fc700 1 -- 172.21.15.189:0/4082271537 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa000bd60 con 0x7f9ab00d0890 2024-04-06T12:31:16.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.811+0000 7f9ab6217700 1 -- 172.21.15.189:0/4082271537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ab00d0890 msgr2=0x7f9ab00d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:16.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.811+0000 7f9ab6217700 1 --2- 172.21.15.189:0/4082271537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ab00d0890 0x7f9ab00d15b0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f9ab008c190 tx=0x7f9aa0009bd0 comp rx=0 tx=0).stop 2024-04-06T12:31:16.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.811+0000 7f9ab6217700 1 -- 172.21.15.189:0/4082271537 shutdown_connections 2024-04-06T12:31:16.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.811+0000 7f9ab6217700 1 --2- 172.21.15.189:0/4082271537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ab00d1af0 0x7f9ab00b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:16.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.811+0000 7f9ab6217700 1 --2- 172.21.15.189:0/4082271537 >> v2:172.21.15.192:3300/0 conn(0x7f9ab015d7e0 0x7f9ab00d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:16.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.811+0000 7f9ab6217700 1 --2- 172.21.15.189:0/4082271537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ab00d0890 0x7f9ab00d15b0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:16.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.811+0000 7f9ab6217700 1 -- 172.21.15.189:0/4082271537 >> 172.21.15.189:0/4082271537 conn(0x7f9ab0072250 msgr2=0x7f9ab0072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:16.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.811+0000 7f9ab6217700 1 -- 172.21.15.189:0/4082271537 shutdown_connections 2024-04-06T12:31:16.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.811+0000 7f9ab6217700 1 -- 172.21.15.189:0/4082271537 wait complete. 2024-04-06T12:31:16.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.812+0000 7f9ab6217700 1 Processor -- start 2024-04-06T12:31:16.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.813+0000 7f9ab6217700 1 -- start start 2024-04-06T12:31:16.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.813+0000 7f9ab6217700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ab00d0890 0x7f9ab00c7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:16.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.813+0000 7f9ab6217700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ab00d1af0 0x7f9ab01f2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:16.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.813+0000 7f9ab6217700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ab015d7e0 0x7f9ab00b3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:16.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.813+0000 7f9ab6217700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab00cbe40 con 0x7f9ab00d1af0 2024-04-06T12:31:16.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.813+0000 7f9ab6217700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab0150120 con 0x7f9ab015d7e0 2024-04-06T12:31:16.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.813+0000 7f9ab6217700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab00cb510 con 0x7f9ab00d0890 2024-04-06T12:31:16.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.813+0000 7f9aaeffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ab00d1af0 0x7f9ab01f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:16.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.813+0000 7f9aaeffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ab00d1af0 0x7f9ab01f2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36514/0 (socket says 172.21.15.189:36514) 2024-04-06T12:31:16.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.813+0000 7f9aaeffd700 1 -- 172.21.15.189:0/4107735407 learned_addr learned my addr 172.21.15.189:0/4107735407 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:16.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.814+0000 7f9aaffff700 1 --2- 172.21.15.189:0/4107735407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ab015d7e0 0x7f9ab00b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:16.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.814+0000 7f9aaeffd700 1 -- 172.21.15.189:0/4107735407 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ab00d0890 msgr2=0x7f9ab00c7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:16.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.814+0000 7f9aaeffd700 1 --2- 172.21.15.189:0/4107735407 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ab00d0890 0x7f9ab00c7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:16.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.814+0000 7f9aaeffd700 1 -- 172.21.15.189:0/4107735407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ab015d7e0 msgr2=0x7f9ab00b3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:16.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.814+0000 7f9aaeffd700 1 --2- 172.21.15.189:0/4107735407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ab015d7e0 0x7f9ab00b3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:16.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.814+0000 7f9aaeffd700 1 -- 172.21.15.189:0/4107735407 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa0009550 con 0x7f9ab00d1af0 2024-04-06T12:31:16.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.814+0000 7f9aaeffd700 1 --2- 172.21.15.189:0/4107735407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ab00d1af0 0x7f9ab01f2250 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0000c00 tx=0x7f9aa0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:16.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.814+0000 7f9a97fff700 1 -- 172.21.15.189:0/4107735407 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0004520 con 0x7f9ab00d1af0 2024-04-06T12:31:16.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.814+0000 7f9a97fff700 1 -- 172.21.15.189:0/4107735407 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa001e070 con 0x7f9ab00d1af0 2024-04-06T12:31:16.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.814+0000 7f9ab6217700 1 -- 172.21.15.189:0/4107735407 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab00cb7f0 con 0x7f9ab00d1af0 2024-04-06T12:31:16.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.814+0000 7f9a97fff700 1 -- 172.21.15.189:0/4107735407 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa00038c0 con 0x7f9ab00d1af0 2024-04-06T12:31:16.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.814+0000 7f9ab6217700 1 -- 172.21.15.189:0/4107735407 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab00c7db0 con 0x7f9ab00d1af0 2024-04-06T12:31:16.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.815+0000 7f9ab6217700 1 -- 172.21.15.189:0/4107735407 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab0074990 con 0x7f9ab00d1af0 2024-04-06T12:31:16.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.815+0000 7f9a97fff700 1 -- 172.21.15.189:0/4107735407 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f9aa0003a20 con 0x7f9ab00d1af0 2024-04-06T12:31:16.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.816+0000 7f9a97fff700 1 --2- 172.21.15.189:0/4107735407 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9a9c06adb0 0x7f9a9c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:16.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.816+0000 7f9aaf7fe700 1 --2- 172.21.15.189:0/4107735407 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9a9c06adb0 0x7f9a9c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:16.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.816+0000 7f9a97fff700 1 -- 172.21.15.189:0/4107735407 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f9aa0020070 con 0x7f9ab00d1af0 2024-04-06T12:31:16.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.816+0000 7f9aaf7fe700 1 --2- 172.21.15.189:0/4107735407 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9a9c06adb0 0x7f9a9c06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4006fd0 tx=0x7f9aa4008040 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:16.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:16.820+0000 7f9a97fff700 1 -- 172.21.15.189:0/4107735407 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9aa0055920 con 0x7f9ab00d1af0 2024-04-06T12:31:17.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:17.031+0000 7f9ab6217700 1 -- 172.21.15.189:0/4107735407 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f9ab00d3230 con 0x7f9ab00d1af0 2024-04-06T12:31:18.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.014+0000 7f9a97fff700 1 -- 172.21.15.189:0/4107735407 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v140) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9ab00d3230 con 0x7f9ab00d1af0 2024-04-06T12:31:18.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.017+0000 7f9ab6217700 1 -- 172.21.15.189:0/4107735407 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9a9c06adb0 msgr2=0x7f9a9c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:18.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.017+0000 7f9ab6217700 1 --2- 172.21.15.189:0/4107735407 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9a9c06adb0 0x7f9a9c06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4006fd0 tx=0x7f9aa4008040 comp rx=0 tx=0).stop 2024-04-06T12:31:18.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.017+0000 7f9ab6217700 1 -- 172.21.15.189:0/4107735407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ab00d1af0 msgr2=0x7f9ab01f2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:18.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.017+0000 7f9ab6217700 1 --2- 172.21.15.189:0/4107735407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ab00d1af0 0x7f9ab01f2250 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0000c00 tx=0x7f9aa0004750 comp rx=0 tx=0).stop 2024-04-06T12:31:18.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.017+0000 7f9ab6217700 1 -- 172.21.15.189:0/4107735407 shutdown_connections 2024-04-06T12:31:18.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.017+0000 7f9ab6217700 1 --2- 172.21.15.189:0/4107735407 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9a9c06adb0 0x7f9a9c06d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.017+0000 7f9ab6217700 1 --2- 172.21.15.189:0/4107735407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ab015d7e0 0x7f9ab00b3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.017+0000 7f9ab6217700 1 --2- 172.21.15.189:0/4107735407 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ab00d0890 0x7f9ab00c7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.017+0000 7f9ab6217700 1 --2- 172.21.15.189:0/4107735407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ab00d1af0 0x7f9ab01f2250 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.017+0000 7f9ab6217700 1 -- 172.21.15.189:0/4107735407 >> 172.21.15.189:0/4107735407 conn(0x7f9ab0072250 msgr2=0x7f9ab015bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:18.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.017+0000 7f9ab6217700 1 -- 172.21.15.189:0/4107735407 shutdown_connections 2024-04-06T12:31:18.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.017+0000 7f9ab6217700 1 -- 172.21.15.189:0/4107735407 wait complete. 2024-04-06T12:31:18.016 INFO:teuthology.orchestra.run.smithi189.stderr: removed all blocklist entries 2024-04-06T12:31:18.030 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-06T12:31:18.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.159+0000 7f8c80f12700 1 Processor -- start 2024-04-06T12:31:18.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.160+0000 7f8c80f12700 1 -- start start 2024-04-06T12:31:18.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.160+0000 7f8c80f12700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8c7c0d17a0 0x7f8c7c06b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.160+0000 7f8c80f12700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7c06d850 con 0x7f8c7c0cedf0 2024-04-06T12:31:18.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.160+0000 7f8c80f12700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7c06c0a0 con 0x7f8c7c146dd0 2024-04-06T12:31:18.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.160+0000 7f8c80f12700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c7c06c380 con 0x7f8c7c0d17a0 2024-04-06T12:31:18.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7ad9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8c7c0cedf0 0x7f8c7c0cf1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55932/0 (socket says 172.21.15.189:55932) 2024-04-06T12:31:18.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7ad9d700 1 -- 172.21.15.189:0/1710902952 learned_addr learned my addr 172.21.15.189:0/1710902952 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:18.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7a59c700 1 --2- 172.21.15.189:0/1710902952 >> v2:172.21.15.192:3300/0 conn(0x7f8c7c0d17a0 0x7f8c7c06b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7959a700 1 -- 172.21.15.189:0/1710902952 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272154081 0 0) 0x7f8c7c06d850 con 0x7f8c7c0cedf0 2024-04-06T12:31:18.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7959a700 1 -- 172.21.15.189:0/1710902952 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c680035b0 con 0x7f8c7c0cedf0 2024-04-06T12:31:18.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7959a700 1 -- 172.21.15.189:0/1710902952 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3594790542 0 0) 0x7f8c680035b0 con 0x7f8c7c0cedf0 2024-04-06T12:31:18.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7959a700 1 -- 172.21.15.189:0/1710902952 >> v2:172.21.15.192:3300/0 conn(0x7f8c7c0d17a0 msgr2=0x7f8c7c06b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:18.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7959a700 1 --2- 172.21.15.189:0/1710902952 >> v2:172.21.15.192:3300/0 conn(0x7f8c7c0d17a0 0x7f8c7c06b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7959a700 1 -- 172.21.15.189:0/1710902952 >> v1:172.21.15.192:6789/0 conn(0x7f8c7c146dd0 legacy=0x7f8c7c06d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:18.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7959a700 1 -- 172.21.15.189:0/1710902952 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c06c660 con 0x7f8c7c0cedf0 2024-04-06T12:31:18.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7959a700 1 -- 172.21.15.189:0/1710902952 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f8c740019f0 con 0x7f8c7c0cedf0 2024-04-06T12:31:18.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7959a700 1 -- 172.21.15.189:0/1710902952 >> v1:172.21.15.189:6789/0 conn(0x7f8c7c0cedf0 legacy=0x7f8c7c0cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:18.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7959a700 1 --2- 172.21.15.189:0/1710902952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c68003a00 0x7f8c68003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7959a700 1 --2- 172.21.15.189:0/1710902952 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c68004850 0x7f8c68004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7959a700 1 --2- 172.21.15.189:0/1710902952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c68005210 0x7f8c6800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7959a700 1 -- 172.21.15.189:0/1710902952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c06c380 con 0x7f8c68003a00 2024-04-06T12:31:18.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7959a700 1 -- 172.21.15.189:0/1710902952 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6800c0b0 con 0x7f8c68005210 2024-04-06T12:31:18.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7959a700 1 -- 172.21.15.189:0/1710902952 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6800c270 con 0x7f8c68004850 2024-04-06T12:31:18.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7a59c700 1 --2- 172.21.15.189:0/1710902952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c68003a00 0x7f8c68003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c7ad9d700 1 --2- 172.21.15.189:0/1710902952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c68005210 0x7f8c6800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.161+0000 7f8c79d9b700 1 --2- 172.21.15.189:0/1710902952 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c68004850 0x7f8c68004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.162+0000 7f8c7a59c700 1 -- 172.21.15.189:0/1710902952 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c68004850 msgr2=0x7f8c68004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:18.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.162+0000 7f8c7a59c700 1 --2- 172.21.15.189:0/1710902952 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c68004850 0x7f8c68004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.162+0000 7f8c7a59c700 1 -- 172.21.15.189:0/1710902952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c68005210 msgr2=0x7f8c6800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:18.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.162+0000 7f8c7a59c700 1 --2- 172.21.15.189:0/1710902952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c68005210 0x7f8c6800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.162+0000 7f8c7a59c700 1 -- 172.21.15.189:0/1710902952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c680035b0 con 0x7f8c68003a00 2024-04-06T12:31:18.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.162+0000 7f8c7ad9d700 1 -- 172.21.15.189:0/1710902952 reap_dead start 2024-04-06T12:31:18.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.162+0000 7f8c79d9b700 1 --2- 172.21.15.189:0/1710902952 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c68004850 0x7f8c68004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:18.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.162+0000 7f8c7a59c700 1 --2- 172.21.15.189:0/1710902952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c68003a00 0x7f8c68003e20 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c009b60 tx=0x7f8c68005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6cc51d064f02077 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:18.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.162+0000 7f8c7959a700 1 -- 172.21.15.189:0/1710902952 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c00e6c0 con 0x7f8c68003a00 2024-04-06T12:31:18.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.162+0000 7f8c7959a700 1 -- 172.21.15.189:0/1710902952 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6c00e820 con 0x7f8c68003a00 2024-04-06T12:31:18.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.162+0000 7f8c7959a700 1 -- 172.21.15.189:0/1710902952 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c00eb90 con 0x7f8c68003a00 2024-04-06T12:31:18.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.162+0000 7f8c80f12700 1 -- 172.21.15.189:0/1710902952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c68003a00 msgr2=0x7f8c68003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:18.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.162+0000 7f8c80f12700 1 --2- 172.21.15.189:0/1710902952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c68003a00 0x7f8c68003e20 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c009b60 tx=0x7f8c68005880 comp rx=0 tx=0).stop 2024-04-06T12:31:18.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.163+0000 7f8c80f12700 1 -- 172.21.15.189:0/1710902952 shutdown_connections 2024-04-06T12:31:18.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.163+0000 7f8c80f12700 1 --2- 172.21.15.189:0/1710902952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c68003a00 0x7f8c68003e20 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.163+0000 7f8c80f12700 1 -- 172.21.15.189:0/1710902952 >> 172.21.15.189:0/1710902952 conn(0x7f8c7c0b2db0 msgr2=0x7f8c7c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:18.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.163+0000 7f8c80f12700 1 -- 172.21.15.189:0/1710902952 shutdown_connections 2024-04-06T12:31:18.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.163+0000 7f8c80f12700 1 -- 172.21.15.189:0/1710902952 wait complete. 2024-04-06T12:31:18.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.164+0000 7f8c80f12700 1 Processor -- start 2024-04-06T12:31:18.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.164+0000 7f8c80f12700 1 -- start start 2024-04-06T12:31:18.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c80f12700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c68003a00 0x7f8c7c0b1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c80f12700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c7c0cfa00 0x7f8c7c0b55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c80f12700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c7c0b3ea0 0x7f8c7c0b4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c80f12700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0c7420 con 0x7f8c7c0cfa00 2024-04-06T12:31:18.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c80f12700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0c6850 con 0x7f8c68003a00 2024-04-06T12:31:18.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c80f12700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0c6af0 con 0x7f8c7c0b3ea0 2024-04-06T12:31:18.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c79d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c7c0cfa00 0x7f8c7c0b55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c79d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c7c0cfa00 0x7f8c7c0b55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36534/0 (socket says 172.21.15.189:36534) 2024-04-06T12:31:18.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c79d9b700 1 -- 172.21.15.189:0/1673282378 learned_addr learned my addr 172.21.15.189:0/1673282378 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:18.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c7a59c700 1 --2- 172.21.15.189:0/1673282378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c68003a00 0x7f8c7c0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c7ad9d700 1 --2- 172.21.15.189:0/1673282378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c7c0b3ea0 0x7f8c7c0b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c79d9b700 1 -- 172.21.15.189:0/1673282378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c7c0b3ea0 msgr2=0x7f8c7c0b4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:18.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c79d9b700 1 --2- 172.21.15.189:0/1673282378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c7c0b3ea0 0x7f8c7c0b4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c79d9b700 1 -- 172.21.15.189:0/1673282378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c68003a00 msgr2=0x7f8c7c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:18.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c79d9b700 1 --2- 172.21.15.189:0/1673282378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c68003a00 0x7f8c7c0b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c79d9b700 1 -- 172.21.15.189:0/1673282378 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c0c6dd0 con 0x7f8c7c0cfa00 2024-04-06T12:31:18.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c7ad9d700 1 --2- 172.21.15.189:0/1673282378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c7c0b3ea0 0x7f8c7c0b4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:18.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c79d9b700 1 --2- 172.21.15.189:0/1673282378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c7c0cfa00 0x7f8c7c0b55d0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400e460 tx=0x7f8c6400e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:18.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c72ffd700 1 -- 172.21.15.189:0/1673282378 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c640052e0 con 0x7f8c7c0cfa00 2024-04-06T12:31:18.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c72ffd700 1 -- 172.21.15.189:0/1673282378 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c64005440 con 0x7f8c7c0cfa00 2024-04-06T12:31:18.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c80f12700 1 -- 172.21.15.189:0/1673282378 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7c0c70b0 con 0x7f8c7c0cfa00 2024-04-06T12:31:18.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c72ffd700 1 -- 172.21.15.189:0/1673282378 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c640038a0 con 0x7f8c7c0cfa00 2024-04-06T12:31:18.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.165+0000 7f8c80f12700 1 -- 172.21.15.189:0/1673282378 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7c160600 con 0x7f8c7c0cfa00 2024-04-06T12:31:18.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.166+0000 7f8c80f12700 1 -- 172.21.15.189:0/1673282378 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7c0c75d0 con 0x7f8c7c0cfa00 2024-04-06T12:31:18.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.167+0000 7f8c72ffd700 1 -- 172.21.15.189:0/1673282378 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f8c64003a00 con 0x7f8c7c0cfa00 2024-04-06T12:31:18.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.167+0000 7f8c72ffd700 1 --2- 172.21.15.189:0/1673282378 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8c5c06ade0 0x7f8c5c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.167+0000 7f8c72ffd700 1 -- 172.21.15.189:0/1673282378 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f8c640888c0 con 0x7f8c7c0cfa00 2024-04-06T12:31:18.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.167+0000 7f8c7a59c700 1 --2- 172.21.15.189:0/1673282378 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8c5c06ade0 0x7f8c5c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.168+0000 7f8c7a59c700 1 --2- 172.21.15.189:0/1673282378 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8c5c06ade0 0x7f8c5c06d2a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c0090b0 tx=0x7f8c6c01e040 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:18.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.172+0000 7f8c72ffd700 1 -- 172.21.15.189:0/1673282378 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8c7c0c75d0 con 0x7f8c7c0cfa00 2024-04-06T12:31:18.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.445+0000 7f8c80f12700 1 -- 172.21.15.189:0/1673282378 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8c7c06f890 con 0x7f8c7c0cfa00 2024-04-06T12:31:18.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.446+0000 7f8c72ffd700 1 -- 172.21.15.189:0/1673282378 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f8c64019020 con 0x7f8c7c0cfa00 2024-04-06T12:31:18.445 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:18.445 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T12:31:18.445 INFO:teuthology.orchestra.run.smithi189.stdout: "auth_dump": [ 2024-04-06T12:31:18.445 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:18.445 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.a", 2024-04-06T12:31:18.446 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmTjYMNBAA5voWGV4Pa7pE27VogrdipQ==", 2024-04-06T12:31:18.446 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:18.446 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:31:18.446 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:31:18.446 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:31:18.446 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:18.447 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.447 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:18.447 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:18.447 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.b", 2024-04-06T12:31:18.447 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm9A1OBRAAnHzubCvRUX8UdW8hmDzKbQ==", 2024-04-06T12:31:18.447 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:18.447 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:31:18.448 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:31:18.448 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:31:18.448 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:18.448 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.448 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:18.448 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:18.449 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.c", 2024-04-06T12:31:18.449 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmz0hhOxAAD3/GINNmGoFk+oPyiHHJXg==", 2024-04-06T12:31:18.449 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:18.449 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:31:18.449 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:31:18.449 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:31:18.449 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:18.450 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.450 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:18.450 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:18.450 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.d", 2024-04-06T12:31:18.450 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmhoAeDRAAoQC0vicWumBRTwpi3PtYaw==", 2024-04-06T12:31:18.450 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:18.451 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:31:18.451 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:31:18.451 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:31:18.451 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:18.451 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.451 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:18.452 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:18.452 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.0", 2024-04-06T12:31:18.452 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDkOxFmkxLWJhAAFXewCNOFX7dhAZVgUmewpQ==", 2024-04-06T12:31:18.452 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:18.452 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:31:18.452 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:31:18.452 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:18.453 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.453 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:18.453 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:18.453 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.1", 2024-04-06T12:31:18.453 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDnOxFmHYqWJRAABlsVCiv3e7YJMbfodJTWag==", 2024-04-06T12:31:18.453 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:18.454 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:31:18.454 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:31:18.454 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:18.454 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.454 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:18.454 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:18.455 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.2", 2024-04-06T12:31:18.455 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDqOxFmkhcvJBAA8NMEpHVGbGALjV9RISKP8g==", 2024-04-06T12:31:18.455 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:18.455 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:31:18.455 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:31:18.455 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:18.456 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.456 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:18.456 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:18.456 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.3", 2024-04-06T12:31:18.456 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDtOxFmDGzhIxAAamBoAbE8+lAel4os67meyQ==", 2024-04-06T12:31:18.456 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:18.457 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:31:18.457 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:31:18.457 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:18.457 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.457 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:18.457 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:18.458 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.4", 2024-04-06T12:31:18.458 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDzOxFmobZDJxAA9l8wdS81Eyw6+17I5ac6bw==", 2024-04-06T12:31:18.458 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:18.458 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:31:18.458 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:31:18.458 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:18.459 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.459 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:18.459 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:18.459 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.5", 2024-04-06T12:31:18.459 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD2OxFmi+ruJRAASlQ0o9jX/wA7U5AosnL+Wg==", 2024-04-06T12:31:18.459 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:18.459 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:31:18.460 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:31:18.460 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:18.460 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.460 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:18.460 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:18.460 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.6", 2024-04-06T12:31:18.461 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD5OxFmbwuuJBAA0wqJRmnJyskQPZJ7/LnpvA==", 2024-04-06T12:31:18.461 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:18.461 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:31:18.461 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:31:18.461 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:18.461 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.462 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:18.462 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:18.462 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.7", 2024-04-06T12:31:18.462 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD8OxFmDmRgIxAAzyW9AGAbmb67ZvB5qunwLQ==", 2024-04-06T12:31:18.462 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:18.462 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:31:18.463 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:31:18.463 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:18.463 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.463 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:18.463 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:18.463 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.0", 2024-04-06T12:31:18.464 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm0ByJERAAoEV8Yfc09+Go6u95XafElQ==", 2024-04-06T12:31:18.464 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:18.464 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:31:18.464 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow r", 2024-04-06T12:31:18.464 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-06T12:31:18.464 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.464 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:18.465 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:18.465 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.1", 2024-04-06T12:31:18.465 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmSV5lGBAAavDEo2LGy0IHpWSAuitHXg==", 2024-04-06T12:31:18.465 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:18.465 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:31:18.465 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:31:18.466 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:31:18.466 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:31:18.466 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.466 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:18.466 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:18.466 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.2", 2024-04-06T12:31:18.467 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm/Gz5FBAA3xX/rsAyWZ1JoUVlQlBIvQ==", 2024-04-06T12:31:18.467 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:18.467 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:31:18.467 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:31:18.467 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:31:18.467 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:31:18.467 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.467 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:18.468 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:18.468 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.3", 2024-04-06T12:31:18.468 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmzra7GxAADbnzIvWnRTo65LvPwtxWrg==", 2024-04-06T12:31:18.468 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:18.468 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:31:18.468 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:31:18.468 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:31:18.468 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:31:18.469 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.469 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:18.469 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:18.469 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.admin", 2024-04-06T12:31:18.469 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmxG+wCBAANg5kt6TOUQdj3/X0Mqgx1g==", 2024-04-06T12:31:18.469 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:18.469 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:31:18.470 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:31:18.470 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:31:18.470 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:18.470 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.470 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:18.470 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:18.470 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.x", 2024-04-06T12:31:18.470 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmPLyBMBAAYfhkI9Z1CPlf1Tijm+L7xA==", 2024-04-06T12:31:18.470 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:18.471 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:31:18.471 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T12:31:18.471 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:18.471 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.471 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:18.471 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:18.471 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.y", 2024-04-06T12:31:18.471 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFm1nHJHhAAQWRe8cU8duWuUJiYaDeTaQ==", 2024-04-06T12:31:18.471 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:18.471 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:31:18.471 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T12:31:18.471 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:18.472 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.472 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:18.472 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-04-06T12:31:18.472 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T12:31:18.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.448+0000 7f8c80f12700 1 -- 172.21.15.189:0/1673282378 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8c5c06ade0 msgr2=0x7f8c5c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:18.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.448+0000 7f8c80f12700 1 --2- 172.21.15.189:0/1673282378 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8c5c06ade0 0x7f8c5c06d2a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c0090b0 tx=0x7f8c6c01e040 comp rx=0 tx=0).stop 2024-04-06T12:31:18.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.448+0000 7f8c80f12700 1 -- 172.21.15.189:0/1673282378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c7c0cfa00 msgr2=0x7f8c7c0b55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:18.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.448+0000 7f8c80f12700 1 --2- 172.21.15.189:0/1673282378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c7c0cfa00 0x7f8c7c0b55d0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400e460 tx=0x7f8c6400e820 comp rx=0 tx=0).stop 2024-04-06T12:31:18.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.448+0000 7f8c80f12700 1 -- 172.21.15.189:0/1673282378 shutdown_connections 2024-04-06T12:31:18.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.448+0000 7f8c80f12700 1 --2- 172.21.15.189:0/1673282378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c7c0b3ea0 0x7f8c7c0b4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.448+0000 7f8c80f12700 1 --2- 172.21.15.189:0/1673282378 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8c5c06ade0 0x7f8c5c06d2a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.448+0000 7f8c80f12700 1 --2- 172.21.15.189:0/1673282378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c68003a00 0x7f8c7c0b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.448+0000 7f8c80f12700 1 --2- 172.21.15.189:0/1673282378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c7c0cfa00 0x7f8c7c0b55d0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.448+0000 7f8c80f12700 1 -- 172.21.15.189:0/1673282378 >> 172.21.15.189:0/1673282378 conn(0x7f8c7c0b2db0 msgr2=0x7f8c7c15c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:18.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.448+0000 7f8c80f12700 1 -- 172.21.15.189:0/1673282378 shutdown_connections 2024-04-06T12:31:18.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.448+0000 7f8c80f12700 1 -- 172.21.15.189:0/1673282378 wait complete. 2024-04-06T12:31:18.478 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:31:18.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.602+0000 7eff59c3c700 1 Processor -- start 2024-04-06T12:31:18.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.602+0000 7eff59c3c700 1 -- start start 2024-04-06T12:31:18.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.603+0000 7eff59c3c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7eff5415d7b0 0x7eff540b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.603+0000 7eff59c3c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff5415e500 0x7eff5406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.603+0000 7eff59c3c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff540b3990 con 0x7eff5406a730 2024-04-06T12:31:18.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.603+0000 7eff59c3c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff5406b0f0 con 0x7eff5415d7b0 2024-04-06T12:31:18.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.603+0000 7eff59c3c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff540b3060 con 0x7eff5415e500 2024-04-06T12:31:18.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.603+0000 7eff537fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7eff5415d7b0 0x7eff540b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.603+0000 7eff52ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7eff5406a730 0x7eff5415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55944/0 (socket says 172.21.15.189:55944) 2024-04-06T12:31:18.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.603+0000 7eff52ffd700 1 -- 172.21.15.189:0/2282327238 learned_addr learned my addr 172.21.15.189:0/2282327238 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:18.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.603+0000 7eff53fff700 1 --2- 172.21.15.189:0/2282327238 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff5415e500 0x7eff5406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.603+0000 7eff537fe700 1 -- 172.21.15.189:0/2282327238 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff5415e500 msgr2=0x7eff5406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:18.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.603+0000 7eff537fe700 1 --2- 172.21.15.189:0/2282327238 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff5415e500 0x7eff5406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.603+0000 7eff537fe700 1 -- 172.21.15.189:0/2282327238 >> v1:172.21.15.189:6789/0 conn(0x7eff5406a730 legacy=0x7eff5415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:18.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.603+0000 7eff537fe700 1 -- 172.21.15.189:0/2282327238 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff540b3340 con 0x7eff5415d7b0 2024-04-06T12:31:18.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.603+0000 7eff53fff700 1 --2- 172.21.15.189:0/2282327238 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff5415e500 0x7eff5406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:18.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.603+0000 7eff537fe700 1 --2- 172.21.15.189:0/2282327238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5415d7b0 0x7eff540b3e20 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7eff44009840 tx=0x7eff44009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4694995a716ef2a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:18.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.603+0000 7eff527fc700 1 -- 172.21.15.189:0/2282327238 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff44011830 con 0x7eff5415d7b0 2024-04-06T12:31:18.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.603+0000 7eff527fc700 1 -- 172.21.15.189:0/2282327238 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff44011990 con 0x7eff5415d7b0 2024-04-06T12:31:18.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.603+0000 7eff527fc700 1 -- 172.21.15.189:0/2282327238 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff44011d20 con 0x7eff5415d7b0 2024-04-06T12:31:18.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.604+0000 7eff59c3c700 1 -- 172.21.15.189:0/2282327238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5415d7b0 msgr2=0x7eff540b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:18.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.604+0000 7eff59c3c700 1 --2- 172.21.15.189:0/2282327238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5415d7b0 0x7eff540b3e20 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7eff44009840 tx=0x7eff44009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:18.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.604+0000 7eff59c3c700 1 -- 172.21.15.189:0/2282327238 shutdown_connections 2024-04-06T12:31:18.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.604+0000 7eff59c3c700 1 --2- 172.21.15.189:0/2282327238 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff5415e500 0x7eff5406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.604+0000 7eff59c3c700 1 --2- 172.21.15.189:0/2282327238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5415d7b0 0x7eff540b3e20 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.604+0000 7eff59c3c700 1 -- 172.21.15.189:0/2282327238 >> 172.21.15.189:0/2282327238 conn(0x7eff540d00b0 msgr2=0x7eff540d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:18.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.604+0000 7eff59c3c700 1 -- 172.21.15.189:0/2282327238 shutdown_connections 2024-04-06T12:31:18.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.604+0000 7eff59c3c700 1 -- 172.21.15.189:0/2282327238 wait complete. 2024-04-06T12:31:18.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.605+0000 7eff59c3c700 1 Processor -- start 2024-04-06T12:31:18.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.605+0000 7eff59c3c700 1 -- start start 2024-04-06T12:31:18.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.606+0000 7eff59c3c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff5406a730 0x7eff5414fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.606+0000 7eff59c3c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5415d7b0 0x7eff541600e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.606+0000 7eff59c3c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff5415e500 0x7eff541f3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.606+0000 7eff59c3c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5415fc50 con 0x7eff5415d7b0 2024-04-06T12:31:18.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.606+0000 7eff59c3c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff541502a0 con 0x7eff5415e500 2024-04-06T12:31:18.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.606+0000 7eff59c3c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff54160620 con 0x7eff5406a730 2024-04-06T12:31:18.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.606+0000 7eff52ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5415d7b0 0x7eff541600e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.606+0000 7eff52ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5415d7b0 0x7eff541600e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36552/0 (socket says 172.21.15.189:36552) 2024-04-06T12:31:18.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.606+0000 7eff52ffd700 1 -- 172.21.15.189:0/3153145709 learned_addr learned my addr 172.21.15.189:0/3153145709 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:18.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.606+0000 7eff537fe700 1 --2- 172.21.15.189:0/3153145709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff5406a730 0x7eff5414fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.606+0000 7eff53fff700 1 --2- 172.21.15.189:0/3153145709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff5415e500 0x7eff541f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.607+0000 7eff52ffd700 1 -- 172.21.15.189:0/3153145709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff5406a730 msgr2=0x7eff5414fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:18.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.607+0000 7eff52ffd700 1 --2- 172.21.15.189:0/3153145709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff5406a730 0x7eff5414fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.607+0000 7eff52ffd700 1 -- 172.21.15.189:0/3153145709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff5415e500 msgr2=0x7eff541f3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:18.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.607+0000 7eff52ffd700 1 --2- 172.21.15.189:0/3153145709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff5415e500 0x7eff541f3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.607+0000 7eff52ffd700 1 -- 172.21.15.189:0/3153145709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff440094a0 con 0x7eff5415d7b0 2024-04-06T12:31:18.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.607+0000 7eff52ffd700 1 --2- 172.21.15.189:0/3153145709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5415d7b0 0x7eff541600e0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7eff4800c520 tx=0x7eff4800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:18.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.607+0000 7eff3bfff700 1 -- 172.21.15.189:0/3153145709 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff48005d70 con 0x7eff5415d7b0 2024-04-06T12:31:18.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.607+0000 7eff3bfff700 1 -- 172.21.15.189:0/3153145709 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff480070d0 con 0x7eff5415d7b0 2024-04-06T12:31:18.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.607+0000 7eff3bfff700 1 -- 172.21.15.189:0/3153145709 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff48007440 con 0x7eff5415d7b0 2024-04-06T12:31:18.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.607+0000 7eff59c3c700 1 -- 172.21.15.189:0/3153145709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff541f3c90 con 0x7eff5415d7b0 2024-04-06T12:31:18.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.607+0000 7eff59c3c700 1 -- 172.21.15.189:0/3153145709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff5415f320 con 0x7eff5415d7b0 2024-04-06T12:31:18.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.608+0000 7eff59c3c700 1 -- 172.21.15.189:0/3153145709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff54067bb0 con 0x7eff5415d7b0 2024-04-06T12:31:18.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.608+0000 7eff3bfff700 1 -- 172.21.15.189:0/3153145709 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7eff480075a0 con 0x7eff5415d7b0 2024-04-06T12:31:18.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.609+0000 7eff3bfff700 1 --2- 172.21.15.189:0/3153145709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7eff4006adb0 0x7eff4006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.609+0000 7eff3bfff700 1 -- 172.21.15.189:0/3153145709 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5177+0+0 (secure 0 0 0) 0x7eff480885a0 con 0x7eff5415d7b0 2024-04-06T12:31:18.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.609+0000 7eff537fe700 1 --2- 172.21.15.189:0/3153145709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7eff4006adb0 0x7eff4006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.609+0000 7eff537fe700 1 --2- 172.21.15.189:0/3153145709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7eff4006adb0 0x7eff4006d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7eff440090b0 tx=0x7eff44008030 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:18.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.613+0000 7eff3bfff700 1 -- 172.21.15.189:0/3153145709 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7eff480539c0 con 0x7eff5415d7b0 2024-04-06T12:31:18.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.812+0000 7eff59c3c700 1 -- 172.21.15.189:0/3153145709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff54074990 con 0x7eff5415d7b0 2024-04-06T12:31:18.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.813+0000 7eff3bfff700 1 -- 172.21.15.189:0/3153145709 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+12652 (secure 0 0 0) 0x7eff48056fe0 con 0x7eff5415d7b0 2024-04-06T12:31:18.811 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:18.811 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":140,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:31:18.010859+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:31:18.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.815+0000 7eff59c3c700 1 -- 172.21.15.189:0/3153145709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7eff4006adb0 msgr2=0x7eff4006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:18.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.815+0000 7eff59c3c700 1 --2- 172.21.15.189:0/3153145709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7eff4006adb0 0x7eff4006d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7eff440090b0 tx=0x7eff44008030 comp rx=0 tx=0).stop 2024-04-06T12:31:18.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.815+0000 7eff59c3c700 1 -- 172.21.15.189:0/3153145709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5415d7b0 msgr2=0x7eff541600e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:18.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.815+0000 7eff59c3c700 1 --2- 172.21.15.189:0/3153145709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5415d7b0 0x7eff541600e0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7eff4800c520 tx=0x7eff4800c8e0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.815+0000 7eff59c3c700 1 -- 172.21.15.189:0/3153145709 shutdown_connections 2024-04-06T12:31:18.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.815+0000 7eff59c3c700 1 --2- 172.21.15.189:0/3153145709 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7eff4006adb0 0x7eff4006d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.815+0000 7eff59c3c700 1 --2- 172.21.15.189:0/3153145709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff5415e500 0x7eff541f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.815+0000 7eff59c3c700 1 --2- 172.21.15.189:0/3153145709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff5406a730 0x7eff5414fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.815+0000 7eff59c3c700 1 --2- 172.21.15.189:0/3153145709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5415d7b0 0x7eff541600e0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.815+0000 7eff59c3c700 1 -- 172.21.15.189:0/3153145709 >> 172.21.15.189:0/3153145709 conn(0x7eff540d00b0 msgr2=0x7eff5406eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:18.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.815+0000 7eff59c3c700 1 -- 172.21.15.189:0/3153145709 shutdown_connections 2024-04-06T12:31:18.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.815+0000 7eff59c3c700 1 -- 172.21.15.189:0/3153145709 wait complete. 2024-04-06T12:31:18.826 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:31:18.826 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:31:18.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.953+0000 7f1f99c7e700 1 Processor -- start 2024-04-06T12:31:18.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.953+0000 7f1f99c7e700 1 -- start start 2024-04-06T12:31:18.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.954+0000 7f1f99c7e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1f940cb870 0x7f1f9415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.954+0000 7f1f99c7e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1f940cfa00 0x7f1f940cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.954+0000 7f1f99c7e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f940cbd80 con 0x7f1f940d11b0 2024-04-06T12:31:18.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.954+0000 7f1f99c7e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f940ccdc0 con 0x7f1f940cfa00 2024-04-06T12:31:18.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.954+0000 7f1f99c7e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f940d0b50 con 0x7f1f940cb870 2024-04-06T12:31:18.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.954+0000 7f1f93fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1f940cfa00 0x7f1f940cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.954+0000 7f1f93fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1f940cfa00 0x7f1f940cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36558/0 (socket says 172.21.15.189:36558) 2024-04-06T12:31:18.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.954+0000 7f1f93fff700 1 -- 172.21.15.189:0/474082794 learned_addr learned my addr 172.21.15.189:0/474082794 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:18.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.954+0000 7f1f92ffd700 1 --2- 172.21.15.189:0/474082794 >> v2:172.21.15.192:3300/0 conn(0x7f1f940cb870 0x7f1f9415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.954+0000 7f1f93fff700 1 -- 172.21.15.189:0/474082794 >> v2:172.21.15.192:3300/0 conn(0x7f1f940cb870 msgr2=0x7f1f9415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:18.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.954+0000 7f1f93fff700 1 --2- 172.21.15.189:0/474082794 >> v2:172.21.15.192:3300/0 conn(0x7f1f940cb870 0x7f1f9415c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.954+0000 7f1f93fff700 1 -- 172.21.15.189:0/474082794 >> v1:172.21.15.192:6789/0 conn(0x7f1f940d11b0 legacy=0x7f1f940d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:18.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.954+0000 7f1f93fff700 1 -- 172.21.15.189:0/474082794 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f940cef20 con 0x7f1f940cfa00 2024-04-06T12:31:18.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.954+0000 7f1f93fff700 1 --2- 172.21.15.189:0/474082794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f940cfa00 0x7f1f940cd560 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f1f9404fb40 tx=0x7f1f8c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7309c6dd419892c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:18.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.955+0000 7f1f927fc700 1 -- 172.21.15.189:0/474082794 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c00e070 con 0x7f1f940cfa00 2024-04-06T12:31:18.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.955+0000 7f1f927fc700 1 -- 172.21.15.189:0/474082794 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8c003a20 con 0x7f1f940cfa00 2024-04-06T12:31:18.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.955+0000 7f1f927fc700 1 -- 172.21.15.189:0/474082794 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c003dd0 con 0x7f1f940cfa00 2024-04-06T12:31:18.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.955+0000 7f1f99c7e700 1 -- 172.21.15.189:0/474082794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f940cfa00 msgr2=0x7f1f940cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:18.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.955+0000 7f1f99c7e700 1 --2- 172.21.15.189:0/474082794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f940cfa00 0x7f1f940cd560 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f1f9404fb40 tx=0x7f1f8c00b330 comp rx=0 tx=0).stop 2024-04-06T12:31:18.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.955+0000 7f1f99c7e700 1 -- 172.21.15.189:0/474082794 shutdown_connections 2024-04-06T12:31:18.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.955+0000 7f1f99c7e700 1 --2- 172.21.15.189:0/474082794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f940cfa00 0x7f1f940cd560 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.955+0000 7f1f99c7e700 1 --2- 172.21.15.189:0/474082794 >> v2:172.21.15.192:3300/0 conn(0x7f1f940cb870 0x7f1f9415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.955+0000 7f1f99c7e700 1 -- 172.21.15.189:0/474082794 >> 172.21.15.189:0/474082794 conn(0x7f1f940b2db0 msgr2=0x7f1f940b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:18.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.955+0000 7f1f99c7e700 1 -- 172.21.15.189:0/474082794 shutdown_connections 2024-04-06T12:31:18.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.955+0000 7f1f99c7e700 1 -- 172.21.15.189:0/474082794 wait complete. 2024-04-06T12:31:18.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.957+0000 7f1f99c7e700 1 Processor -- start 2024-04-06T12:31:18.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.957+0000 7f1f99c7e700 1 -- start start 2024-04-06T12:31:18.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.958+0000 7f1f99c7e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f940cb870 0x7f1f94154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.958+0000 7f1f99c7e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f940cfa00 0x7f1f940ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.958+0000 7f1f99c7e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1f940d11b0 0x7f1f94157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.958+0000 7f1f99c7e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f940b3ea0 con 0x7f1f940cfa00 2024-04-06T12:31:18.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.958+0000 7f1f99c7e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f940c4d30 con 0x7f1f940cb870 2024-04-06T12:31:18.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.958+0000 7f1f99c7e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f940cadc0 con 0x7f1f940d11b0 2024-04-06T12:31:18.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.958+0000 7f1f92ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f940cfa00 0x7f1f940ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.958+0000 7f1f92ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f940cfa00 0x7f1f940ca450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36572/0 (socket says 172.21.15.189:36572) 2024-04-06T12:31:18.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.958+0000 7f1f92ffd700 1 -- 172.21.15.189:0/39200845 learned_addr learned my addr 172.21.15.189:0/39200845 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:18.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.958+0000 7f1f937fe700 1 --2- 172.21.15.189:0/39200845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f940cb870 0x7f1f94154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.958+0000 7f1f93fff700 1 --2- 172.21.15.189:0/39200845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1f940d11b0 0x7f1f94157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.958+0000 7f1f92ffd700 1 -- 172.21.15.189:0/39200845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1f940d11b0 msgr2=0x7f1f94157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:18.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.958+0000 7f1f92ffd700 1 --2- 172.21.15.189:0/39200845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1f940d11b0 0x7f1f94157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.958+0000 7f1f92ffd700 1 -- 172.21.15.189:0/39200845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f940cb870 msgr2=0x7f1f94154d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:18.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.958+0000 7f1f92ffd700 1 --2- 172.21.15.189:0/39200845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f940cb870 0x7f1f94154d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:18.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.958+0000 7f1f92ffd700 1 -- 172.21.15.189:0/39200845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8c009b30 con 0x7f1f940cfa00 2024-04-06T12:31:18.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.958+0000 7f1f93fff700 1 --2- 172.21.15.189:0/39200845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1f940d11b0 0x7f1f94157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:18.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.958+0000 7f1f92ffd700 1 --2- 172.21.15.189:0/39200845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f940cfa00 0x7f1f940ca450 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f1f8800d800 tx=0x7f1f8800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:18.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.958+0000 7f1f7bfff700 1 -- 172.21.15.189:0/39200845 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f880125f0 con 0x7f1f940cfa00 2024-04-06T12:31:18.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.959+0000 7f1f7bfff700 1 -- 172.21.15.189:0/39200845 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f88012750 con 0x7f1f940cfa00 2024-04-06T12:31:18.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.959+0000 7f1f7bfff700 1 -- 172.21.15.189:0/39200845 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f88012ae0 con 0x7f1f940cfa00 2024-04-06T12:31:18.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.959+0000 7f1f99c7e700 1 -- 172.21.15.189:0/39200845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f940cb0a0 con 0x7f1f940cfa00 2024-04-06T12:31:18.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.959+0000 7f1f99c7e700 1 -- 172.21.15.189:0/39200845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f940b43d0 con 0x7f1f940cfa00 2024-04-06T12:31:18.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.959+0000 7f1f99c7e700 1 -- 172.21.15.189:0/39200845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f94074990 con 0x7f1f940cfa00 2024-04-06T12:31:18.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.960+0000 7f1f7bfff700 1 -- 172.21.15.189:0/39200845 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f1f88012c40 con 0x7f1f940cfa00 2024-04-06T12:31:18.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.960+0000 7f1f7bfff700 1 --2- 172.21.15.189:0/39200845 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1f8006adb0 0x7f1f8006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:18.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.960+0000 7f1f7bfff700 1 -- 172.21.15.189:0/39200845 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f1f880892d0 con 0x7f1f940cfa00 2024-04-06T12:31:18.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.960+0000 7f1f937fe700 1 --2- 172.21.15.189:0/39200845 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1f8006adb0 0x7f1f8006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:18.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.961+0000 7f1f937fe700 1 --2- 172.21.15.189:0/39200845 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1f8006adb0 0x7f1f8006d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1f940b70b0 tx=0x7f1f84007ec0 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:18.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:18.965+0000 7f1f7bfff700 1 -- 172.21.15.189:0/39200845 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1f88054640 con 0x7f1f940cfa00 2024-04-06T12:31:19.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.158+0000 7f1f99c7e700 1 -- 172.21.15.189:0/39200845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1f9406f7e0 con 0x7f1f940cfa00 2024-04-06T12:31:19.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.158+0000 7f1f7bfff700 1 -- 172.21.15.189:0/39200845 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v140) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1f88057c60 con 0x7f1f940cfa00 2024-04-06T12:31:19.157 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:31:19.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.160+0000 7f1f99c7e700 1 -- 172.21.15.189:0/39200845 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1f8006adb0 msgr2=0x7f1f8006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:19.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.160+0000 7f1f99c7e700 1 --2- 172.21.15.189:0/39200845 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1f8006adb0 0x7f1f8006d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1f940b70b0 tx=0x7f1f84007ec0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.160+0000 7f1f99c7e700 1 -- 172.21.15.189:0/39200845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f940cfa00 msgr2=0x7f1f940ca450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:19.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.160+0000 7f1f99c7e700 1 --2- 172.21.15.189:0/39200845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f940cfa00 0x7f1f940ca450 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f1f8800d800 tx=0x7f1f8800db10 comp rx=0 tx=0).stop 2024-04-06T12:31:19.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.160+0000 7f1f99c7e700 1 -- 172.21.15.189:0/39200845 shutdown_connections 2024-04-06T12:31:19.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.160+0000 7f1f99c7e700 1 --2- 172.21.15.189:0/39200845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1f940d11b0 0x7f1f94157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.160+0000 7f1f99c7e700 1 --2- 172.21.15.189:0/39200845 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1f8006adb0 0x7f1f8006d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.160+0000 7f1f99c7e700 1 --2- 172.21.15.189:0/39200845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f940cb870 0x7f1f94154d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.160+0000 7f1f99c7e700 1 --2- 172.21.15.189:0/39200845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f940cfa00 0x7f1f940ca450 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.160+0000 7f1f99c7e700 1 -- 172.21.15.189:0/39200845 >> 172.21.15.189:0/39200845 conn(0x7f1f940b2db0 msgr2=0x7f1f940b25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:19.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.161+0000 7f1f99c7e700 1 -- 172.21.15.189:0/39200845 shutdown_connections 2024-04-06T12:31:19.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.161+0000 7f1f99c7e700 1 -- 172.21.15.189:0/39200845 wait complete. 2024-04-06T12:31:19.171 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-06T12:31:19.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.294+0000 7f440506c700 1 Processor -- start 2024-04-06T12:31:19.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.294+0000 7f440506c700 1 -- start start 2024-04-06T12:31:19.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.295+0000 7f440506c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f440015a550 0x7f44000cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:19.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.295+0000 7f440506c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f44000cf900 0x7f44000b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:19.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.295+0000 7f440506c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44000b4ba0 0x7f440006eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:19.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.295+0000 7f440506c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44000cdbe0 con 0x7f44000cf900 2024-04-06T12:31:19.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.295+0000 7f440506c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44000b4ff0 con 0x7f440015a550 2024-04-06T12:31:19.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.295+0000 7f440506c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f440015a940 con 0x7f44000b4ba0 2024-04-06T12:31:19.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.295+0000 7f43fe59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f44000cf900 0x7f44000b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:19.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.295+0000 7f43fe59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f44000cf900 0x7f44000b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50762/0 (socket says 172.21.15.189:50762) 2024-04-06T12:31:19.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.295+0000 7f43fe59c700 1 -- 172.21.15.189:0/143853321 learned_addr learned my addr 172.21.15.189:0/143853321 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:19.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f43ff59e700 1 --2- 172.21.15.189:0/143853321 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44000b4ba0 0x7f440006eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:19.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f43fe59c700 1 -- 172.21.15.189:0/143853321 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44000b4ba0 msgr2=0x7f440006eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:19.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f43fe59c700 1 --2- 172.21.15.189:0/143853321 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44000b4ba0 0x7f440006eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f43fed9d700 1 --2- 172.21.15.189:0/143853321 >> v2:172.21.15.192:3300/0 conn(0x7f440015a550 0x7f44000cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:19.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f43fe59c700 1 -- 172.21.15.189:0/143853321 >> v2:172.21.15.192:3300/0 conn(0x7f440015a550 msgr2=0x7f44000cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:19.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f43fe59c700 1 --2- 172.21.15.189:0/143853321 >> v2:172.21.15.192:3300/0 conn(0x7f440015a550 0x7f44000cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f43fe59c700 1 -- 172.21.15.189:0/143853321 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440015ac00 con 0x7f44000cf900 2024-04-06T12:31:19.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f43fe59c700 1 --2- 172.21.15.189:0/143853321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44000cf900 0x7f44000b4660 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f4400073470 tx=0x7f43f000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6944858efe287c8f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:19.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f43fdd9b700 1 -- 172.21.15.189:0/143853321 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f000dd30 con 0x7f44000cf900 2024-04-06T12:31:19.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f43fdd9b700 1 -- 172.21.15.189:0/143853321 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f000de90 con 0x7f44000cf900 2024-04-06T12:31:19.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f43fdd9b700 1 -- 172.21.15.189:0/143853321 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0010690 con 0x7f44000cf900 2024-04-06T12:31:19.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f440506c700 1 -- 172.21.15.189:0/143853321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44000cf900 msgr2=0x7f44000b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:19.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f440506c700 1 --2- 172.21.15.189:0/143853321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44000cf900 0x7f44000b4660 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f4400073470 tx=0x7f43f000be70 comp rx=0 tx=0).stop 2024-04-06T12:31:19.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f440506c700 1 -- 172.21.15.189:0/143853321 shutdown_connections 2024-04-06T12:31:19.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f440506c700 1 --2- 172.21.15.189:0/143853321 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44000b4ba0 0x7f440006eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f440506c700 1 --2- 172.21.15.189:0/143853321 >> v2:172.21.15.192:3300/0 conn(0x7f440015a550 0x7f44000cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f440506c700 1 --2- 172.21.15.189:0/143853321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44000cf900 0x7f44000b4660 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f440506c700 1 -- 172.21.15.189:0/143853321 >> 172.21.15.189:0/143853321 conn(0x7f44000d00b0 msgr2=0x7f44000d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:19.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f440506c700 1 -- 172.21.15.189:0/143853321 shutdown_connections 2024-04-06T12:31:19.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.296+0000 7f440506c700 1 -- 172.21.15.189:0/143853321 wait complete. 2024-04-06T12:31:19.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.298+0000 7f440506c700 1 Processor -- start 2024-04-06T12:31:19.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.298+0000 7f440506c700 1 -- start start 2024-04-06T12:31:19.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f440506c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44000b4ba0 0x7f440014a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:19.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f440506c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44000cf900 0x7f4400158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:19.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f440506c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4400149a20 0x7f4400149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:19.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f440506c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4400157340 con 0x7f44000b4ba0 2024-04-06T12:31:19.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f440506c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440014a410 con 0x7f44000cf900 2024-04-06T12:31:19.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f440506c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44001566c0 con 0x7f4400149a20 2024-04-06T12:31:19.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f43fed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44000b4ba0 0x7f440014a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:19.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f43fed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44000b4ba0 0x7f440014a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50778/0 (socket says 172.21.15.189:50778) 2024-04-06T12:31:19.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f43fed9d700 1 -- 172.21.15.189:0/922126491 learned_addr learned my addr 172.21.15.189:0/922126491 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:19.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f43fe59c700 1 --2- 172.21.15.189:0/922126491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44000cf900 0x7f4400158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:19.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f43ff59e700 1 --2- 172.21.15.189:0/922126491 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4400149a20 0x7f4400149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:19.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f43fed9d700 1 -- 172.21.15.189:0/922126491 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4400149a20 msgr2=0x7f4400149ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:19.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f43fed9d700 1 --2- 172.21.15.189:0/922126491 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4400149a20 0x7f4400149ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f43fed9d700 1 -- 172.21.15.189:0/922126491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44000cf900 msgr2=0x7f4400158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:19.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f43fed9d700 1 --2- 172.21.15.189:0/922126491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44000cf900 0x7f4400158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f43fed9d700 1 -- 172.21.15.189:0/922126491 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f000b7f0 con 0x7f44000b4ba0 2024-04-06T12:31:19.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f43fed9d700 1 --2- 172.21.15.189:0/922126491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44000b4ba0 0x7f440014a910 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f43f400c820 tx=0x7f43f400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:19.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f43e77fe700 1 -- 172.21.15.189:0/922126491 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f4004220 con 0x7f44000b4ba0 2024-04-06T12:31:19.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f43e77fe700 1 -- 172.21.15.189:0/922126491 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f4004380 con 0x7f44000b4ba0 2024-04-06T12:31:19.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f43e77fe700 1 -- 172.21.15.189:0/922126491 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f40036a0 con 0x7f44000b4ba0 2024-04-06T12:31:19.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.299+0000 7f440506c700 1 -- 172.21.15.189:0/922126491 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44001569a0 con 0x7f44000b4ba0 2024-04-06T12:31:19.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.300+0000 7f440506c700 1 -- 172.21.15.189:0/922126491 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4400156f60 con 0x7f44000b4ba0 2024-04-06T12:31:19.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.301+0000 7f440506c700 1 -- 172.21.15.189:0/922126491 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4400067bb0 con 0x7f44000b4ba0 2024-04-06T12:31:19.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.301+0000 7f43e77fe700 1 -- 172.21.15.189:0/922126491 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f43f4003800 con 0x7f44000b4ba0 2024-04-06T12:31:19.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.301+0000 7f43e77fe700 1 --2- 172.21.15.189:0/922126491 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f43ec06adb0 0x7f43ec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:19.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.301+0000 7f43e77fe700 1 -- 172.21.15.189:0/922126491 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f43f4088f30 con 0x7f44000b4ba0 2024-04-06T12:31:19.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.302+0000 7f43fe59c700 1 --2- 172.21.15.189:0/922126491 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f43ec06adb0 0x7f43ec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:19.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.302+0000 7f43fe59c700 1 --2- 172.21.15.189:0/922126491 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f43ec06adb0 0x7f43ec06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f43f000beb0 tx=0x7f43f001c040 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:19.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.306+0000 7f43e77fe700 1 -- 172.21.15.189:0/922126491 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f43f4054070 con 0x7f44000b4ba0 2024-04-06T12:31:19.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.500+0000 7f440506c700 1 -- 172.21.15.189:0/922126491 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4400074990 con 0x7f44000b4ba0 2024-04-06T12:31:19.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.500+0000 7f43e77fe700 1 -- 172.21.15.189:0/922126491 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v140) v1 ==== 84+0+65 (secure 0 0 0) 0x7f43f401e070 con 0x7f44000b4ba0 2024-04-06T12:31:19.502 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:19.502 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T12:31:19.502 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:19.502 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 1, 2024-04-06T12:31:19.503 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": ".mgr" 2024-04-06T12:31:19.503 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:19.503 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T12:31:19.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.502+0000 7f440506c700 1 -- 172.21.15.189:0/922126491 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f43ec06adb0 msgr2=0x7f43ec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:19.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.502+0000 7f440506c700 1 --2- 172.21.15.189:0/922126491 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f43ec06adb0 0x7f43ec06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f43f000beb0 tx=0x7f43f001c040 comp rx=0 tx=0).stop 2024-04-06T12:31:19.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.502+0000 7f440506c700 1 -- 172.21.15.189:0/922126491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44000b4ba0 msgr2=0x7f440014a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:19.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.502+0000 7f440506c700 1 --2- 172.21.15.189:0/922126491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44000b4ba0 0x7f440014a910 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f43f400c820 tx=0x7f43f400cb30 comp rx=0 tx=0).stop 2024-04-06T12:31:19.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.502+0000 7f440506c700 1 -- 172.21.15.189:0/922126491 shutdown_connections 2024-04-06T12:31:19.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.502+0000 7f440506c700 1 --2- 172.21.15.189:0/922126491 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4400149a20 0x7f4400149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.502+0000 7f440506c700 1 --2- 172.21.15.189:0/922126491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44000b4ba0 0x7f440014a910 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.502+0000 7f440506c700 1 --2- 172.21.15.189:0/922126491 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f43ec06adb0 0x7f43ec06d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.502+0000 7f440506c700 1 --2- 172.21.15.189:0/922126491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44000cf900 0x7f4400158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.502+0000 7f440506c700 1 -- 172.21.15.189:0/922126491 >> 172.21.15.189:0/922126491 conn(0x7f44000d00b0 msgr2=0x7f440006eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:19.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.503+0000 7f440506c700 1 -- 172.21.15.189:0/922126491 shutdown_connections 2024-04-06T12:31:19.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.503+0000 7f440506c700 1 -- 172.21.15.189:0/922126491 wait complete. 2024-04-06T12:31:19.513 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-06T12:31:19.513 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-06T12:31:19.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.640+0000 7fe7b2313700 1 Processor -- start 2024-04-06T12:31:19.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.640+0000 7fe7b2313700 1 -- start start 2024-04-06T12:31:19.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.641+0000 7fe7b2313700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe7ac15d7b0 0x7fe7ac0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:19.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.641+0000 7fe7b2313700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe7ac06a730 0x7fe7ac06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:19.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.641+0000 7fe7b2313700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac15ddf0 0x7fe7ac15e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:19.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.641+0000 7fe7b2313700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7ac06b0f0 con 0x7fe7ac15d7b0 2024-04-06T12:31:19.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.641+0000 7fe7b2313700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7ac0b3990 con 0x7fe7ac06a730 2024-04-06T12:31:19.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.641+0000 7fe7b2313700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac0b3060 con 0x7fe7ac15ddf0 2024-04-06T12:31:19.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.641+0000 7fe7abfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe7ac15d7b0 0x7fe7ac0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:19.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.641+0000 7fe7abfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe7ac15d7b0 0x7fe7ac0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50780/0 (socket says 172.21.15.189:50780) 2024-04-06T12:31:19.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.641+0000 7fe7abfff700 1 -- 172.21.15.189:0/1859377162 learned_addr learned my addr 172.21.15.189:0/1859377162 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:19.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.641+0000 7fe7b08b0700 1 --2- 172.21.15.189:0/1859377162 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac15ddf0 0x7fe7ac15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:19.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.641+0000 7fe7ab7fe700 1 --2- 172.21.15.189:0/1859377162 >> v2:172.21.15.192:3300/0 conn(0x7fe7ac06a730 0x7fe7ac06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:19.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.641+0000 7fe7abfff700 1 -- 172.21.15.189:0/1859377162 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac15ddf0 msgr2=0x7fe7ac15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:19.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.641+0000 7fe7abfff700 1 --2- 172.21.15.189:0/1859377162 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac15ddf0 0x7fe7ac15e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.641+0000 7fe7abfff700 1 -- 172.21.15.189:0/1859377162 >> v2:172.21.15.192:3300/0 conn(0x7fe7ac06a730 msgr2=0x7fe7ac06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:19.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.641+0000 7fe7abfff700 1 --2- 172.21.15.189:0/1859377162 >> v2:172.21.15.192:3300/0 conn(0x7fe7ac06a730 0x7fe7ac06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.641+0000 7fe7abfff700 1 -- 172.21.15.189:0/1859377162 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ac0b3340 con 0x7fe7ac15d7b0 2024-04-06T12:31:19.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.641+0000 7fe7abfff700 1 --2- 172.21.15.189:0/1859377162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7ac15d7b0 0x7fe7ac0b3e20 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac0afc00 tx=0x7fe7a0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2750ff92503241d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:19.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.642+0000 7fe7aaffd700 1 -- 172.21.15.189:0/1859377162 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a000b870 con 0x7fe7ac15d7b0 2024-04-06T12:31:19.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.642+0000 7fe7aaffd700 1 -- 172.21.15.189:0/1859377162 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a000b9d0 con 0x7fe7ac15d7b0 2024-04-06T12:31:19.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.642+0000 7fe7aaffd700 1 -- 172.21.15.189:0/1859377162 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a000bd60 con 0x7fe7ac15d7b0 2024-04-06T12:31:19.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.642+0000 7fe7b2313700 1 -- 172.21.15.189:0/1859377162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7ac15d7b0 msgr2=0x7fe7ac0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:19.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.642+0000 7fe7b2313700 1 --2- 172.21.15.189:0/1859377162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7ac15d7b0 0x7fe7ac0b3e20 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac0afc00 tx=0x7fe7a0009bd0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.642+0000 7fe7b2313700 1 -- 172.21.15.189:0/1859377162 shutdown_connections 2024-04-06T12:31:19.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.642+0000 7fe7b2313700 1 --2- 172.21.15.189:0/1859377162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7ac15d7b0 0x7fe7ac0b3e20 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.642+0000 7fe7b2313700 1 --2- 172.21.15.189:0/1859377162 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac15ddf0 0x7fe7ac15e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.642+0000 7fe7b2313700 1 --2- 172.21.15.189:0/1859377162 >> v2:172.21.15.192:3300/0 conn(0x7fe7ac06a730 0x7fe7ac06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.642+0000 7fe7b2313700 1 -- 172.21.15.189:0/1859377162 >> 172.21.15.189:0/1859377162 conn(0x7fe7ac0d00b0 msgr2=0x7fe7ac0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:19.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.642+0000 7fe7b2313700 1 -- 172.21.15.189:0/1859377162 shutdown_connections 2024-04-06T12:31:19.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.642+0000 7fe7b2313700 1 -- 172.21.15.189:0/1859377162 wait complete. 2024-04-06T12:31:19.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.643+0000 7fe7b2313700 1 Processor -- start 2024-04-06T12:31:19.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.644+0000 7fe7b2313700 1 -- start start 2024-04-06T12:31:19.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.644+0000 7fe7b2313700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7ac06a730 0x7fe7ac14d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:19.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.644+0000 7fe7b2313700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac15d7b0 0x7fe7ac1f6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:19.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.644+0000 7fe7b2313700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7ac15ddf0 0x7fe7ac0acee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:19.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.644+0000 7fe7b2313700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac15ab70 con 0x7fe7ac06a730 2024-04-06T12:31:19.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.644+0000 7fe7b2313700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac1f2cd0 con 0x7fe7ac15ddf0 2024-04-06T12:31:19.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.644+0000 7fe7b2313700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac1f7260 con 0x7fe7ac15d7b0 2024-04-06T12:31:19.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.645+0000 7fe7abfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7ac06a730 0x7fe7ac14d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:19.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.645+0000 7fe7abfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7ac06a730 0x7fe7ac14d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50790/0 (socket says 172.21.15.189:50790) 2024-04-06T12:31:19.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.645+0000 7fe7abfff700 1 -- 172.21.15.189:0/960522420 learned_addr learned my addr 172.21.15.189:0/960522420 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:19.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.645+0000 7fe7b08b0700 1 --2- 172.21.15.189:0/960522420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7ac15ddf0 0x7fe7ac0acee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:19.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.645+0000 7fe7ab7fe700 1 --2- 172.21.15.189:0/960522420 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac15d7b0 0x7fe7ac1f6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:19.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.645+0000 7fe7abfff700 1 -- 172.21.15.189:0/960522420 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac15d7b0 msgr2=0x7fe7ac1f6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:19.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.645+0000 7fe7abfff700 1 --2- 172.21.15.189:0/960522420 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac15d7b0 0x7fe7ac1f6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.645+0000 7fe7abfff700 1 -- 172.21.15.189:0/960522420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7ac15ddf0 msgr2=0x7fe7ac0acee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:19.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.645+0000 7fe7abfff700 1 --2- 172.21.15.189:0/960522420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7ac15ddf0 0x7fe7ac0acee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:19.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.645+0000 7fe7abfff700 1 -- 172.21.15.189:0/960522420 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a0009550 con 0x7fe7ac06a730 2024-04-06T12:31:19.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.645+0000 7fe7abfff700 1 --2- 172.21.15.189:0/960522420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7ac06a730 0x7fe7ac14d530 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac06c820 tx=0x7fe7a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:19.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.645+0000 7fe7a8ff9700 1 -- 172.21.15.189:0/960522420 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a0004480 con 0x7fe7ac06a730 2024-04-06T12:31:19.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.645+0000 7fe7a8ff9700 1 -- 172.21.15.189:0/960522420 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a00045e0 con 0x7fe7ac06a730 2024-04-06T12:31:19.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.645+0000 7fe7b2313700 1 -- 172.21.15.189:0/960522420 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7ac1f6ca0 con 0x7fe7ac06a730 2024-04-06T12:31:19.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.645+0000 7fe7a8ff9700 1 -- 172.21.15.189:0/960522420 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a00038e0 con 0x7fe7ac06a730 2024-04-06T12:31:19.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.645+0000 7fe7b2313700 1 -- 172.21.15.189:0/960522420 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7ac15a790 con 0x7fe7ac06a730 2024-04-06T12:31:19.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.646+0000 7fe7b2313700 1 -- 172.21.15.189:0/960522420 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7ac067bb0 con 0x7fe7ac06a730 2024-04-06T12:31:19.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.647+0000 7fe7a8ff9700 1 -- 172.21.15.189:0/960522420 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fe7a0021070 con 0x7fe7ac06a730 2024-04-06T12:31:19.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.647+0000 7fe7a8ff9700 1 --2- 172.21.15.189:0/960522420 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe79806adb0 0x7fe79806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:19.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.647+0000 7fe7a8ff9700 1 -- 172.21.15.189:0/960522420 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fe7a008a920 con 0x7fe7ac06a730 2024-04-06T12:31:19.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.647+0000 7fe7ab7fe700 1 --2- 172.21.15.189:0/960522420 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe79806adb0 0x7fe79806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:19.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.648+0000 7fe7ab7fe700 1 --2- 172.21.15.189:0/960522420 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe79806adb0 0x7fe79806d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac15c410 tx=0x7fe79c008040 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:19.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.652+0000 7fe7a8ff9700 1 -- 172.21.15.189:0/960522420 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe7a0055bf0 con 0x7fe7ac06a730 2024-04-06T12:31:19.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:19.846+0000 7fe7b2313700 1 -- 172.21.15.189:0/960522420 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe7ac074990 con 0x7fe7ac06a730 2024-04-06T12:31:20.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.671+0000 7fe7a8ff9700 1 -- 172.21.15.189:0/960522420 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v141) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe7a00290b0 con 0x7fe7ac06a730 2024-04-06T12:31:20.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.675+0000 7fe7b2313700 1 -- 172.21.15.189:0/960522420 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe79806adb0 msgr2=0x7fe79806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:20.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.675+0000 7fe7b2313700 1 --2- 172.21.15.189:0/960522420 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe79806adb0 0x7fe79806d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac15c410 tx=0x7fe79c008040 comp rx=0 tx=0).stop 2024-04-06T12:31:20.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.675+0000 7fe7b2313700 1 -- 172.21.15.189:0/960522420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7ac06a730 msgr2=0x7fe7ac14d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:20.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.675+0000 7fe7b2313700 1 --2- 172.21.15.189:0/960522420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7ac06a730 0x7fe7ac14d530 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac06c820 tx=0x7fe7a0004750 comp rx=0 tx=0).stop 2024-04-06T12:31:20.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.676+0000 7fe7b2313700 1 -- 172.21.15.189:0/960522420 shutdown_connections 2024-04-06T12:31:20.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.676+0000 7fe7b2313700 1 --2- 172.21.15.189:0/960522420 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe79806adb0 0x7fe79806d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:20.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.676+0000 7fe7b2313700 1 --2- 172.21.15.189:0/960522420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7ac15ddf0 0x7fe7ac0acee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:20.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.676+0000 7fe7b2313700 1 --2- 172.21.15.189:0/960522420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7ac06a730 0x7fe7ac14d530 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:20.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.676+0000 7fe7b2313700 1 --2- 172.21.15.189:0/960522420 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac15d7b0 0x7fe7ac1f6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:20.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.676+0000 7fe7b2313700 1 -- 172.21.15.189:0/960522420 >> 172.21.15.189:0/960522420 conn(0x7fe7ac0d00b0 msgr2=0x7fe7ac150610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:20.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.677+0000 7fe7b2313700 1 -- 172.21.15.189:0/960522420 shutdown_connections 2024-04-06T12:31:20.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.677+0000 7fe7b2313700 1 -- 172.21.15.189:0/960522420 wait complete. 2024-04-06T12:31:20.676 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' created 2024-04-06T12:31:20.690 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-06T12:31:20.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.824+0000 7f7d3ea2b700 1 Processor -- start 2024-04-06T12:31:20.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.824+0000 7f7d3ea2b700 1 -- start start 2024-04-06T12:31:20.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.825+0000 7f7d3ea2b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d38144f40 0x7f7d3806a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:20.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.825+0000 7f7d3ea2b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d3815c150 0x7f7d3806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:20.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.825+0000 7f7d3ea2b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d3815c740 con 0x7f7d3815fcd0 2024-04-06T12:31:20.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.825+0000 7f7d3ea2b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d380cfcb0 con 0x7f7d38144f40 2024-04-06T12:31:20.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.825+0000 7f7d3ea2b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3815ca20 con 0x7f7d3815c150 2024-04-06T12:31:20.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.825+0000 7f7d37fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d38144f40 0x7f7d3806a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:20.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.825+0000 7f7d377fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7d3815fcd0 0x7f7d3815ba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48166/0 (socket says 172.21.15.189:48166) 2024-04-06T12:31:20.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.825+0000 7f7d377fe700 1 -- 172.21.15.189:0/2932543264 learned_addr learned my addr 172.21.15.189:0/2932543264 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:20.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.825+0000 7f7d3cfc8700 1 --2- 172.21.15.189:0/2932543264 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d3815c150 0x7f7d3806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:20.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.825+0000 7f7d37fff700 1 -- 172.21.15.189:0/2932543264 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d3815c150 msgr2=0x7f7d3806ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:20.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.825+0000 7f7d37fff700 1 --2- 172.21.15.189:0/2932543264 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d3815c150 0x7f7d3806ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:20.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.825+0000 7f7d37fff700 1 -- 172.21.15.189:0/2932543264 >> v1:172.21.15.189:6789/0 conn(0x7f7d3815fcd0 legacy=0x7f7d3815ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:20.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.825+0000 7f7d37fff700 1 -- 172.21.15.189:0/2932543264 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3815f1f0 con 0x7f7d38144f40 2024-04-06T12:31:20.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.825+0000 7f7d36ffd700 1 -- 172.21.15.189:0/2932543264 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502300811 0 0) 0x7f7d3815c740 con 0x7f7d3815fcd0 2024-04-06T12:31:20.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.825+0000 7f7d37fff700 1 --2- 172.21.15.189:0/2932543264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d38144f40 0x7f7d3806a730 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009840 tx=0x7f7d28009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e0fdb35cdefc3e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:20.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.825+0000 7f7d36ffd700 1 -- 172.21.15.189:0/2932543264 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d28011830 con 0x7f7d38144f40 2024-04-06T12:31:20.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.825+0000 7f7d36ffd700 1 -- 172.21.15.189:0/2932543264 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d28011990 con 0x7f7d38144f40 2024-04-06T12:31:20.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.825+0000 7f7d36ffd700 1 -- 172.21.15.189:0/2932543264 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d28011d20 con 0x7f7d38144f40 2024-04-06T12:31:20.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.826+0000 7f7d3ea2b700 1 -- 172.21.15.189:0/2932543264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d38144f40 msgr2=0x7f7d3806a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:20.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.826+0000 7f7d3ea2b700 1 --2- 172.21.15.189:0/2932543264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d38144f40 0x7f7d3806a730 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009840 tx=0x7f7d28009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:20.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.826+0000 7f7d3ea2b700 1 -- 172.21.15.189:0/2932543264 shutdown_connections 2024-04-06T12:31:20.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.826+0000 7f7d3ea2b700 1 --2- 172.21.15.189:0/2932543264 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d3815c150 0x7f7d3806ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:20.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.826+0000 7f7d3ea2b700 1 --2- 172.21.15.189:0/2932543264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d38144f40 0x7f7d3806a730 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:20.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.826+0000 7f7d3ea2b700 1 -- 172.21.15.189:0/2932543264 >> 172.21.15.189:0/2932543264 conn(0x7f7d380d00b0 msgr2=0x7f7d380d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:20.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.826+0000 7f7d3ea2b700 1 -- 172.21.15.189:0/2932543264 shutdown_connections 2024-04-06T12:31:20.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.826+0000 7f7d3ea2b700 1 -- 172.21.15.189:0/2932543264 wait complete. 2024-04-06T12:31:20.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.827+0000 7f7d3ea2b700 1 Processor -- start 2024-04-06T12:31:20.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.827+0000 7f7d3ea2b700 1 -- start start 2024-04-06T12:31:20.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.828+0000 7f7d3ea2b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d38144f40 0x7f7d380c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:20.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.828+0000 7f7d3ea2b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3815c150 0x7f7d381f1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:20.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.828+0000 7f7d3ea2b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d3815fcd0 0x7f7d380cb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:20.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.828+0000 7f7d3ea2b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d381f1980 con 0x7f7d3815c150 2024-04-06T12:31:20.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.828+0000 7f7d3ea2b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d381f4e80 con 0x7f7d3815fcd0 2024-04-06T12:31:20.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.828+0000 7f7d3ea2b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d380cbac0 con 0x7f7d38144f40 2024-04-06T12:31:20.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.828+0000 7f7d377fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3815c150 0x7f7d381f1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:20.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.828+0000 7f7d377fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3815c150 0x7f7d381f1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50804/0 (socket says 172.21.15.189:50804) 2024-04-06T12:31:20.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.828+0000 7f7d377fe700 1 -- 172.21.15.189:0/2721119222 learned_addr learned my addr 172.21.15.189:0/2721119222 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:20.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.828+0000 7f7d3cfc8700 1 --2- 172.21.15.189:0/2721119222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d3815fcd0 0x7f7d380cb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:20.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.828+0000 7f7d377fe700 1 -- 172.21.15.189:0/2721119222 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d38144f40 msgr2=0x7f7d380c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:20.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.828+0000 7f7d37fff700 1 --2- 172.21.15.189:0/2721119222 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d38144f40 0x7f7d380c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:20.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.828+0000 7f7d377fe700 1 --2- 172.21.15.189:0/2721119222 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d38144f40 0x7f7d380c66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:20.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.828+0000 7f7d377fe700 1 -- 172.21.15.189:0/2721119222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d3815fcd0 msgr2=0x7f7d380cb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:20.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.828+0000 7f7d377fe700 1 --2- 172.21.15.189:0/2721119222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d3815fcd0 0x7f7d380cb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:20.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.829+0000 7f7d377fe700 1 -- 172.21.15.189:0/2721119222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d280094a0 con 0x7f7d3815c150 2024-04-06T12:31:20.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.829+0000 7f7d377fe700 1 --2- 172.21.15.189:0/2721119222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3815c150 0x7f7d381f1440 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c00c430 tx=0x7f7d2c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:20.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.829+0000 7f7d34ff9700 1 -- 172.21.15.189:0/2721119222 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c004c80 con 0x7f7d3815c150 2024-04-06T12:31:20.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.829+0000 7f7d34ff9700 1 -- 172.21.15.189:0/2721119222 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2c004e00 con 0x7f7d3815c150 2024-04-06T12:31:20.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.829+0000 7f7d3ea2b700 1 -- 172.21.15.189:0/2721119222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d381f4300 con 0x7f7d3815c150 2024-04-06T12:31:20.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.829+0000 7f7d34ff9700 1 -- 172.21.15.189:0/2721119222 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c007310 con 0x7f7d3815c150 2024-04-06T12:31:20.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.829+0000 7f7d3ea2b700 1 -- 172.21.15.189:0/2721119222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d381f48c0 con 0x7f7d3815c150 2024-04-06T12:31:20.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.830+0000 7f7d3ea2b700 1 -- 172.21.15.189:0/2721119222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d38141a90 con 0x7f7d3815c150 2024-04-06T12:31:20.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.830+0000 7f7d34ff9700 1 -- 172.21.15.189:0/2721119222 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7d2c007470 con 0x7f7d3815c150 2024-04-06T12:31:20.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.831+0000 7f7d34ff9700 1 --2- 172.21.15.189:0/2721119222 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d2406adb0 0x7f7d2406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:20.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.831+0000 7f7d34ff9700 1 -- 172.21.15.189:0/2721119222 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f7d2c089050 con 0x7f7d3815c150 2024-04-06T12:31:20.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.831+0000 7f7d37fff700 1 --2- 172.21.15.189:0/2721119222 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d2406adb0 0x7f7d2406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:20.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.831+0000 7f7d37fff700 1 --2- 172.21.15.189:0/2721119222 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d2406adb0 0x7f7d2406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7d280090b0 tx=0x7f7d28008030 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:20.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:20.835+0000 7f7d34ff9700 1 -- 172.21.15.189:0/2721119222 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7d38141a90 con 0x7f7d3815c150 2024-04-06T12:31:21.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.028+0000 7f7d3ea2b700 1 -- 172.21.15.189:0/2721119222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7d38141a90 con 0x7f7d3815c150 2024-04-06T12:31:21.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.674+0000 7f7d34ff9700 1 -- 172.21.15.189:0/2721119222 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v142) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7d38141a90 con 0x7f7d3815c150 2024-04-06T12:31:21.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.683+0000 7f7d3ea2b700 1 -- 172.21.15.189:0/2721119222 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d2406adb0 msgr2=0x7f7d2406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:21.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.683+0000 7f7d3ea2b700 1 --2- 172.21.15.189:0/2721119222 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d2406adb0 0x7f7d2406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7d280090b0 tx=0x7f7d28008030 comp rx=0 tx=0).stop 2024-04-06T12:31:21.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.683+0000 7f7d3ea2b700 1 -- 172.21.15.189:0/2721119222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3815c150 msgr2=0x7f7d381f1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:21.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.683+0000 7f7d3ea2b700 1 --2- 172.21.15.189:0/2721119222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3815c150 0x7f7d381f1440 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c00c430 tx=0x7f7d2c00c7f0 comp rx=0 tx=0).stop 2024-04-06T12:31:21.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.683+0000 7f7d3ea2b700 1 -- 172.21.15.189:0/2721119222 shutdown_connections 2024-04-06T12:31:21.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.683+0000 7f7d3ea2b700 1 --2- 172.21.15.189:0/2721119222 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7d2406adb0 0x7f7d2406d270 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7d280090b0 tx=0x7f7d28008030 comp rx=0 tx=0).stop 2024-04-06T12:31:21.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.683+0000 7f7d3ea2b700 1 --2- 172.21.15.189:0/2721119222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d3815fcd0 0x7f7d380cb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:21.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.683+0000 7f7d3ea2b700 1 --2- 172.21.15.189:0/2721119222 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d38144f40 0x7f7d380c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:21.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.683+0000 7f7d3ea2b700 1 --2- 172.21.15.189:0/2721119222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3815c150 0x7f7d381f1440 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:21.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.683+0000 7f7d3ea2b700 1 -- 172.21.15.189:0/2721119222 >> 172.21.15.189:0/2721119222 conn(0x7f7d380d00b0 msgr2=0x7f7d3806f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:21.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.684+0000 7f7d3ea2b700 1 -- 172.21.15.189:0/2721119222 shutdown_connections 2024-04-06T12:31:21.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.686+0000 7f7d3ea2b700 1 -- 172.21.15.189:0/2721119222 wait complete. 2024-04-06T12:31:21.689 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' created 2024-04-06T12:31:21.705 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-06T12:31:21.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.835+0000 7f15805b8700 1 Processor -- start 2024-04-06T12:31:21.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.835+0000 7f15805b8700 1 -- start start 2024-04-06T12:31:21.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.836+0000 7f15805b8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15780d11b0 0x7f15780d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:21.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.836+0000 7f15805b8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f15780cb870 0x7f157815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:21.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.836+0000 7f15805b8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15780d1ad0 con 0x7f15780cfa00 2024-04-06T12:31:21.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.836+0000 7f15805b8700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15780ccdc0 con 0x7f15780cb870 2024-04-06T12:31:21.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.836+0000 7f15805b8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15780d0b50 con 0x7f15780d11b0 2024-04-06T12:31:21.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.836+0000 7f157db53700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f15780cb870 0x7f157815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.836+0000 7f157db53700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f15780cb870 0x7f157815c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50806/0 (socket says 172.21.15.189:50806) 2024-04-06T12:31:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.836+0000 7f157db53700 1 -- 172.21.15.189:0/2882755693 learned_addr learned my addr 172.21.15.189:0/2882755693 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.837+0000 7f157e354700 1 --2- 172.21.15.189:0/2882755693 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15780d11b0 0x7f15780d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.837+0000 7f157db53700 1 -- 172.21.15.189:0/2882755693 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15780d11b0 msgr2=0x7f15780d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.837+0000 7f157db53700 1 --2- 172.21.15.189:0/2882755693 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15780d11b0 0x7f15780d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.837+0000 7f157db53700 1 -- 172.21.15.189:0/2882755693 >> v1:172.21.15.192:6789/0 conn(0x7f15780cfa00 legacy=0x7f15780cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.837+0000 7f157db53700 1 -- 172.21.15.189:0/2882755693 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15780cbd80 con 0x7f15780cb870 2024-04-06T12:31:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.837+0000 7f157db53700 1 --2- 172.21.15.189:0/2882755693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15780cb870 0x7f157815c280 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f15740098f0 tx=0x7f1574009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2566c06864297fb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.837+0000 7f157d352700 1 -- 172.21.15.189:0/2882755693 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157400b8a0 con 0x7f15780cb870 2024-04-06T12:31:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.837+0000 7f157d352700 1 -- 172.21.15.189:0/2882755693 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157400ba00 con 0x7f15780cb870 2024-04-06T12:31:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.837+0000 7f157d352700 1 -- 172.21.15.189:0/2882755693 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157400bd90 con 0x7f15780cb870 2024-04-06T12:31:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.838+0000 7f15805b8700 1 -- 172.21.15.189:0/2882755693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15780cb870 msgr2=0x7f157815c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.838+0000 7f15805b8700 1 --2- 172.21.15.189:0/2882755693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15780cb870 0x7f157815c280 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f15740098f0 tx=0x7f1574009c00 comp rx=0 tx=0).stop 2024-04-06T12:31:21.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.838+0000 7f15805b8700 1 -- 172.21.15.189:0/2882755693 shutdown_connections 2024-04-06T12:31:21.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.838+0000 7f15805b8700 1 --2- 172.21.15.189:0/2882755693 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15780d11b0 0x7f15780d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:21.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.838+0000 7f15805b8700 1 --2- 172.21.15.189:0/2882755693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15780cb870 0x7f157815c280 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:21.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.838+0000 7f15805b8700 1 -- 172.21.15.189:0/2882755693 >> 172.21.15.189:0/2882755693 conn(0x7f15780b2db0 msgr2=0x7f15780b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:21.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.838+0000 7f15805b8700 1 -- 172.21.15.189:0/2882755693 shutdown_connections 2024-04-06T12:31:21.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.838+0000 7f15805b8700 1 -- 172.21.15.189:0/2882755693 wait complete. 2024-04-06T12:31:21.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.839+0000 7f15805b8700 1 Processor -- start 2024-04-06T12:31:21.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.839+0000 7f15805b8700 1 -- start start 2024-04-06T12:31:21.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.839+0000 7f15805b8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15780cb870 0x7f15780c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:21.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f15805b8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15780cfa00 0x7f157806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:21.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f15805b8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15780d11b0 0x7f1578158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:21.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f15805b8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15780ca210 con 0x7f15780cb870 2024-04-06T12:31:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f15805b8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157806d8a0 con 0x7f15780d11b0 2024-04-06T12:31:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f15805b8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15780c9650 con 0x7f15780cfa00 2024-04-06T12:31:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f157e354700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15780cb870 0x7f15780c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f157e354700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15780cb870 0x7f15780c8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50822/0 (socket says 172.21.15.189:50822) 2024-04-06T12:31:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f157e354700 1 -- 172.21.15.189:0/4191920993 learned_addr learned my addr 172.21.15.189:0/4191920993 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f157db53700 1 --2- 172.21.15.189:0/4191920993 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15780cfa00 0x7f157806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f157eb55700 1 --2- 172.21.15.189:0/4191920993 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15780d11b0 0x7f1578158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f157e354700 1 -- 172.21.15.189:0/4191920993 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15780cfa00 msgr2=0x7f157806d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f157e354700 1 --2- 172.21.15.189:0/4191920993 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15780cfa00 0x7f157806d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f157e354700 1 -- 172.21.15.189:0/4191920993 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15780d11b0 msgr2=0x7f1578158080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f157e354700 1 --2- 172.21.15.189:0/4191920993 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15780d11b0 0x7f1578158080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f157e354700 1 -- 172.21.15.189:0/4191920993 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1574009550 con 0x7f15780cb870 2024-04-06T12:31:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f157db53700 1 --2- 172.21.15.189:0/4191920993 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15780cfa00 0x7f157806d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f157eb55700 1 --2- 172.21.15.189:0/4191920993 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15780d11b0 0x7f1578158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f157e354700 1 --2- 172.21.15.189:0/4191920993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15780cb870 0x7f15780c8d00 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f156800d800 tx=0x7f156800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f156effd700 1 -- 172.21.15.189:0/4191920993 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15680125f0 con 0x7f15780cb870 2024-04-06T12:31:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f156effd700 1 -- 172.21.15.189:0/4191920993 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1568012750 con 0x7f15780cb870 2024-04-06T12:31:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f15805b8700 1 -- 172.21.15.189:0/4191920993 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15780c9370 con 0x7f15780cb870 2024-04-06T12:31:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f156effd700 1 -- 172.21.15.189:0/4191920993 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1568012b00 con 0x7f15780cb870 2024-04-06T12:31:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.840+0000 7f15805b8700 1 -- 172.21.15.189:0/4191920993 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15780c9e30 con 0x7f15780cb870 2024-04-06T12:31:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.841+0000 7f15805b8700 1 -- 172.21.15.189:0/4191920993 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15780ca480 con 0x7f15780cb870 2024-04-06T12:31:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.842+0000 7f156effd700 1 -- 172.21.15.189:0/4191920993 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f1568012c60 con 0x7f15780cb870 2024-04-06T12:31:21.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.842+0000 7f156effd700 1 --2- 172.21.15.189:0/4191920993 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f155c06adb0 0x7f155c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:21.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.842+0000 7f156effd700 1 -- 172.21.15.189:0/4191920993 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f1568089870 con 0x7f15780cb870 2024-04-06T12:31:21.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.842+0000 7f157db53700 1 --2- 172.21.15.189:0/4191920993 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f155c06adb0 0x7f155c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:21.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.843+0000 7f157db53700 1 --2- 172.21.15.189:0/4191920993 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f155c06adb0 0x7f155c06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f15780cbcb0 tx=0x7f1574008030 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:21.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:21.847+0000 7f156effd700 1 -- 172.21.15.189:0/4191920993 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f15680548f0 con 0x7f15780cb870 2024-04-06T12:31:22.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.081+0000 7f15805b8700 1 -- 172.21.15.189:0/4191920993 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1578074990 con 0x7f15780cb870 2024-04-06T12:31:22.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.087+0000 7f156effd700 1 -- 172.21.15.189:0/4191920993 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v58) v1 ==== 174+0+0 (secure 0 0 0) 0x7f1568057f10 con 0x7f15780cb870 2024-04-06T12:31:22.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.091+0000 7f15805b8700 1 -- 172.21.15.189:0/4191920993 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f155c06adb0 msgr2=0x7f155c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:22.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.091+0000 7f15805b8700 1 --2- 172.21.15.189:0/4191920993 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f155c06adb0 0x7f155c06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f15780cbcb0 tx=0x7f1574008030 comp rx=0 tx=0).stop 2024-04-06T12:31:22.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.091+0000 7f15805b8700 1 -- 172.21.15.189:0/4191920993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15780cb870 msgr2=0x7f15780c8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:22.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.091+0000 7f15805b8700 1 --2- 172.21.15.189:0/4191920993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15780cb870 0x7f15780c8d00 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f156800d800 tx=0x7f156800dbc0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.091+0000 7f15805b8700 1 -- 172.21.15.189:0/4191920993 shutdown_connections 2024-04-06T12:31:22.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.091+0000 7f15805b8700 1 --2- 172.21.15.189:0/4191920993 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f155c06adb0 0x7f155c06d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.091+0000 7f15805b8700 1 --2- 172.21.15.189:0/4191920993 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15780d11b0 0x7f1578158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.091+0000 7f15805b8700 1 --2- 172.21.15.189:0/4191920993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15780cb870 0x7f15780c8d00 secure :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f156800d800 tx=0x7f156800dbc0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.091+0000 7f15805b8700 1 --2- 172.21.15.189:0/4191920993 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15780cfa00 0x7f157806d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.091+0000 7f15805b8700 1 -- 172.21.15.189:0/4191920993 >> 172.21.15.189:0/4191920993 conn(0x7f15780b2db0 msgr2=0x7f157815a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:22.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.092+0000 7f15805b8700 1 -- 172.21.15.189:0/4191920993 shutdown_connections 2024-04-06T12:31:22.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.092+0000 7f15805b8700 1 -- 172.21.15.189:0/4191920993 wait complete. 2024-04-06T12:31:22.106 INFO:teuthology.orchestra.run.smithi189.stderr:new fs with metadata pool 17 and data pool 18 2024-04-06T12:31:22.116 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:31:22.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.239+0000 7f90b56f2700 1 Processor -- start 2024-04-06T12:31:22.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.239+0000 7f90b56f2700 1 -- start start 2024-04-06T12:31:22.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.240+0000 7f90b56f2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f90b015d590 0x7f90b00b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:22.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.240+0000 7f90b56f2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f90b00b4ed0 0x7f90b00b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:22.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.240+0000 7f90b56f2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f90b00b6580 0x7f90b00b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:22.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.240+0000 7f90b56f2700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b00b5890 con 0x7f90b00b6580 2024-04-06T12:31:22.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.240+0000 7f90b56f2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b00b5cc0 con 0x7f90b015d590 2024-04-06T12:31:22.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.240+0000 7f90b56f2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b00d1500 con 0x7f90b00b4ed0 2024-04-06T12:31:22.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.241+0000 7f90af7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f90b00b6580 0x7f90b00b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:22.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.241+0000 7f90af7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f90b00b6580 0x7f90b00b6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50830/0 (socket says 172.21.15.189:50830) 2024-04-06T12:31:22.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.241+0000 7f90af7fe700 1 -- 172.21.15.189:0/3895599243 learned_addr learned my addr 172.21.15.189:0/3895599243 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:22.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.241+0000 7f90aeffd700 1 --2- 172.21.15.189:0/3895599243 >> v2:172.21.15.192:3300/0 conn(0x7f90b015d590 0x7f90b00b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:22.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.241+0000 7f90ae7fc700 1 --2- 172.21.15.189:0/3895599243 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f90b00b4ed0 0x7f90b00b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:22.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.241+0000 7f90af7fe700 1 -- 172.21.15.189:0/3895599243 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f90b00b4ed0 msgr2=0x7f90b00b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:22.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.241+0000 7f90af7fe700 1 --2- 172.21.15.189:0/3895599243 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f90b00b4ed0 0x7f90b00b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.241+0000 7f90af7fe700 1 -- 172.21.15.189:0/3895599243 >> v2:172.21.15.192:3300/0 conn(0x7f90b015d590 msgr2=0x7f90b00b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:22.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.241+0000 7f90af7fe700 1 --2- 172.21.15.189:0/3895599243 >> v2:172.21.15.192:3300/0 conn(0x7f90b015d590 0x7f90b00b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.241+0000 7f90af7fe700 1 -- 172.21.15.189:0/3895599243 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b00d17e0 con 0x7f90b00b6580 2024-04-06T12:31:22.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.241+0000 7f90af7fe700 1 --2- 172.21.15.189:0/3895599243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90b00b6580 0x7f90b00b6a10 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f90a8009e50 tx=0x7f90a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb81b20e31d1d05 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:22.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.241+0000 7f90adffb700 1 -- 172.21.15.189:0/3895599243 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a800e070 con 0x7f90b00b6580 2024-04-06T12:31:22.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.241+0000 7f90adffb700 1 -- 172.21.15.189:0/3895599243 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a800b980 con 0x7f90b00b6580 2024-04-06T12:31:22.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.241+0000 7f90adffb700 1 -- 172.21.15.189:0/3895599243 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a800bd30 con 0x7f90b00b6580 2024-04-06T12:31:22.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.242+0000 7f90b56f2700 1 -- 172.21.15.189:0/3895599243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90b00b6580 msgr2=0x7f90b00b6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:22.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.242+0000 7f90b56f2700 1 --2- 172.21.15.189:0/3895599243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90b00b6580 0x7f90b00b6a10 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f90a8009e50 tx=0x7f90a800b330 comp rx=0 tx=0).stop 2024-04-06T12:31:22.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.242+0000 7f90b56f2700 1 -- 172.21.15.189:0/3895599243 shutdown_connections 2024-04-06T12:31:22.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.242+0000 7f90b56f2700 1 --2- 172.21.15.189:0/3895599243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90b00b6580 0x7f90b00b6a10 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.242+0000 7f90b56f2700 1 --2- 172.21.15.189:0/3895599243 >> v2:172.21.15.192:3300/0 conn(0x7f90b015d590 0x7f90b00b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.242+0000 7f90b56f2700 1 --2- 172.21.15.189:0/3895599243 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f90b00b4ed0 0x7f90b00b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.242+0000 7f90b56f2700 1 -- 172.21.15.189:0/3895599243 >> 172.21.15.189:0/3895599243 conn(0x7f90b006bbc0 msgr2=0x7f90b006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:22.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.242+0000 7f90b56f2700 1 -- 172.21.15.189:0/3895599243 shutdown_connections 2024-04-06T12:31:22.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.242+0000 7f90b56f2700 1 -- 172.21.15.189:0/3895599243 wait complete. 2024-04-06T12:31:22.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.243+0000 7f90b56f2700 1 Processor -- start 2024-04-06T12:31:22.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.244+0000 7f90b56f2700 1 -- start start 2024-04-06T12:31:22.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.244+0000 7f90b56f2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90b00b4ed0 0x7f90b00caf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.244+0000 7f90b56f2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90b00b6580 0x7f90b01f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.244+0000 7f90b56f2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f90b015d590 0x7f90b0158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.244+0000 7f90b56f2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b00cd2a0 con 0x7f90b00b4ed0 2024-04-06T12:31:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.244+0000 7f90b56f2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b01f30a0 con 0x7f90b00b6580 2024-04-06T12:31:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.244+0000 7f90b56f2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b01f3380 con 0x7f90b015d590 2024-04-06T12:31:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.245+0000 7f90aeffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90b00b4ed0 0x7f90b00caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.245+0000 7f90aeffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90b00b4ed0 0x7f90b00caf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50834/0 (socket says 172.21.15.189:50834) 2024-04-06T12:31:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.245+0000 7f90aeffd700 1 -- 172.21.15.189:0/273420310 learned_addr learned my addr 172.21.15.189:0/273420310 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.245+0000 7f90ae7fc700 1 --2- 172.21.15.189:0/273420310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90b00b6580 0x7f90b01f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.245+0000 7f90af7fe700 1 --2- 172.21.15.189:0/273420310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f90b015d590 0x7f90b0158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.245+0000 7f90aeffd700 1 -- 172.21.15.189:0/273420310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f90b015d590 msgr2=0x7f90b0158bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:22.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.245+0000 7f90aeffd700 1 --2- 172.21.15.189:0/273420310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f90b015d590 0x7f90b0158bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.245+0000 7f90aeffd700 1 -- 172.21.15.189:0/273420310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90b00b6580 msgr2=0x7f90b01f3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.245+0000 7f90aeffd700 1 --2- 172.21.15.189:0/273420310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90b00b6580 0x7f90b01f3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.245+0000 7f90aeffd700 1 -- 172.21.15.189:0/273420310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a8009b30 con 0x7f90b00b4ed0 2024-04-06T12:31:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.245+0000 7f90aeffd700 1 --2- 172.21.15.189:0/273420310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90b00b4ed0 0x7f90b00caf30 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f90a400d800 tx=0x7f90a400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.245+0000 7f90977fe700 1 -- 172.21.15.189:0/273420310 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a40125f0 con 0x7f90b00b4ed0 2024-04-06T12:31:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.245+0000 7f90977fe700 1 -- 172.21.15.189:0/273420310 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a4012750 con 0x7f90b00b4ed0 2024-04-06T12:31:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.245+0000 7f90b56f2700 1 -- 172.21.15.189:0/273420310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b01f3660 con 0x7f90b00b4ed0 2024-04-06T12:31:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.245+0000 7f90977fe700 1 -- 172.21.15.189:0/273420310 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a4012b00 con 0x7f90b00b4ed0 2024-04-06T12:31:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.245+0000 7f90b56f2700 1 -- 172.21.15.189:0/273420310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b01f3940 con 0x7f90b00b4ed0 2024-04-06T12:31:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.246+0000 7f90b56f2700 1 -- 172.21.15.189:0/273420310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b0074990 con 0x7f90b00b4ed0 2024-04-06T12:31:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.247+0000 7f90977fe700 1 -- 172.21.15.189:0/273420310 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f90a4012c60 con 0x7f90b00b4ed0 2024-04-06T12:31:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.247+0000 7f90977fe700 1 --2- 172.21.15.189:0/273420310 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f909c06adb0 0x7f909c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.247+0000 7f90977fe700 1 -- 172.21.15.189:0/273420310 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f90a4089570 con 0x7f90b00b4ed0 2024-04-06T12:31:22.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.247+0000 7f90ae7fc700 1 --2- 172.21.15.189:0/273420310 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f909c06adb0 0x7f909c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:22.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.248+0000 7f90ae7fc700 1 --2- 172.21.15.189:0/273420310 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f909c06adb0 0x7f909c06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f90a0009950 tx=0x7f90a0008040 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:22.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.252+0000 7f90977fe700 1 -- 172.21.15.189:0/273420310 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f90a4054640 con 0x7f90b00b4ed0 2024-04-06T12:31:22.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.448+0000 7f90b56f2700 1 -- 172.21.15.189:0/273420310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90b006a730 con 0x7f90b00b4ed0 2024-04-06T12:31:22.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.449+0000 7f90977fe700 1 -- 172.21.15.189:0/273420310 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16377 (secure 0 0 0) 0x7f90a4057c60 con 0x7f90b00b4ed0 2024-04-06T12:31:22.448 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:22.448 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":143,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:31:22.083528+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:31:19.866331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-06T12:31:21.048344+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:31:22.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.452+0000 7f90b56f2700 1 -- 172.21.15.189:0/273420310 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f909c06adb0 msgr2=0x7f909c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:22.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.452+0000 7f90b56f2700 1 --2- 172.21.15.189:0/273420310 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f909c06adb0 0x7f909c06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f90a0009950 tx=0x7f90a0008040 comp rx=0 tx=0).stop 2024-04-06T12:31:22.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.452+0000 7f90b56f2700 1 -- 172.21.15.189:0/273420310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90b00b4ed0 msgr2=0x7f90b00caf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:22.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.452+0000 7f90b56f2700 1 --2- 172.21.15.189:0/273420310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90b00b4ed0 0x7f90b00caf30 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f90a400d800 tx=0x7f90a400db10 comp rx=0 tx=0).stop 2024-04-06T12:31:22.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.452+0000 7f90b56f2700 1 -- 172.21.15.189:0/273420310 shutdown_connections 2024-04-06T12:31:22.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.452+0000 7f90b56f2700 1 --2- 172.21.15.189:0/273420310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f90b015d590 0x7f90b0158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.452+0000 7f90b56f2700 1 --2- 172.21.15.189:0/273420310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90b00b4ed0 0x7f90b00caf30 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.452+0000 7f90b56f2700 1 --2- 172.21.15.189:0/273420310 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f909c06adb0 0x7f909c06d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.452+0000 7f90b56f2700 1 --2- 172.21.15.189:0/273420310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90b00b6580 0x7f90b01f3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.452+0000 7f90b56f2700 1 -- 172.21.15.189:0/273420310 >> 172.21.15.189:0/273420310 conn(0x7f90b006bbc0 msgr2=0x7f90b00d26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:22.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.452+0000 7f90b56f2700 1 -- 172.21.15.189:0/273420310 shutdown_connections 2024-04-06T12:31:22.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.452+0000 7f90b56f2700 1 -- 172.21.15.189:0/273420310 wait complete. 2024-04-06T12:31:22.463 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:31:22.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.587+0000 7fb61402a700 1 Processor -- start 2024-04-06T12:31:22.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.588+0000 7fb61402a700 1 -- start start 2024-04-06T12:31:22.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.588+0000 7fb61402a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb60c15d7b0 0x7fb60c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:22.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.588+0000 7fb61402a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb60c06a730 0x7fb60c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:22.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.588+0000 7fb61402a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb60c0b3990 con 0x7fb60c15ddf0 2024-04-06T12:31:22.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.588+0000 7fb61402a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb60c15e950 con 0x7fb60c15d7b0 2024-04-06T12:31:22.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.588+0000 7fb61402a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb60c06b0f0 con 0x7fb60c06a730 2024-04-06T12:31:22.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.589+0000 7fb611dc6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb60c15d7b0 0x7fb60c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:22.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.589+0000 7fb611dc6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb60c15d7b0 0x7fb60c0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50850/0 (socket says 172.21.15.189:50850) 2024-04-06T12:31:22.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.589+0000 7fb611dc6700 1 -- 172.21.15.189:0/3338853680 learned_addr learned my addr 172.21.15.189:0/3338853680 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:22.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.589+0000 7fb6115c5700 1 --2- 172.21.15.189:0/3338853680 >> v2:172.21.15.192:3300/0 conn(0x7fb60c06a730 0x7fb60c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:22.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.589+0000 7fb611dc6700 1 -- 172.21.15.189:0/3338853680 >> v2:172.21.15.192:3300/0 conn(0x7fb60c06a730 msgr2=0x7fb60c06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:22.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.589+0000 7fb611dc6700 1 --2- 172.21.15.189:0/3338853680 >> v2:172.21.15.192:3300/0 conn(0x7fb60c06a730 0x7fb60c06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.589+0000 7fb611dc6700 1 -- 172.21.15.189:0/3338853680 >> v1:172.21.15.192:6789/0 conn(0x7fb60c15ddf0 legacy=0x7fb60c15e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:22.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.589+0000 7fb611dc6700 1 -- 172.21.15.189:0/3338853680 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60c0b3060 con 0x7fb60c15d7b0 2024-04-06T12:31:22.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.589+0000 7fb611dc6700 1 --2- 172.21.15.189:0/3338853680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c15d7b0 0x7fb60c0b3e20 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fb60c0afc00 tx=0x7fb608009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dabd63e7a4e209ed server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:22.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.589+0000 7fb610dc4700 1 -- 172.21.15.189:0/3338853680 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb608004230 con 0x7fb60c15d7b0 2024-04-06T12:31:22.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.589+0000 7fb610dc4700 1 -- 172.21.15.189:0/3338853680 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb608004390 con 0x7fb60c15d7b0 2024-04-06T12:31:22.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.589+0000 7fb610dc4700 1 -- 172.21.15.189:0/3338853680 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6080036a0 con 0x7fb60c15d7b0 2024-04-06T12:31:22.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.590+0000 7fb61402a700 1 -- 172.21.15.189:0/3338853680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c15d7b0 msgr2=0x7fb60c0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:22.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.590+0000 7fb61402a700 1 --2- 172.21.15.189:0/3338853680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c15d7b0 0x7fb60c0b3e20 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fb60c0afc00 tx=0x7fb608009bd0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.590+0000 7fb61402a700 1 -- 172.21.15.189:0/3338853680 shutdown_connections 2024-04-06T12:31:22.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.590+0000 7fb61402a700 1 --2- 172.21.15.189:0/3338853680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c15d7b0 0x7fb60c0b3e20 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.590+0000 7fb61402a700 1 --2- 172.21.15.189:0/3338853680 >> v2:172.21.15.192:3300/0 conn(0x7fb60c06a730 0x7fb60c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.590+0000 7fb61402a700 1 -- 172.21.15.189:0/3338853680 >> 172.21.15.189:0/3338853680 conn(0x7fb60c0d00b0 msgr2=0x7fb60c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:22.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.590+0000 7fb61402a700 1 -- 172.21.15.189:0/3338853680 shutdown_connections 2024-04-06T12:31:22.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.590+0000 7fb61402a700 1 -- 172.21.15.189:0/3338853680 wait complete. 2024-04-06T12:31:22.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.591+0000 7fb61402a700 1 Processor -- start 2024-04-06T12:31:22.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.591+0000 7fb61402a700 1 -- start start 2024-04-06T12:31:22.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.592+0000 7fb61402a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb60c06a730 0x7fb60c1f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:22.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.592+0000 7fb61402a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb60c15d7b0 0x7fb60c1f6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:22.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.592+0000 7fb61402a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c15ddf0 0x7fb60c0cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:22.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.592+0000 7fb61402a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c14f400 con 0x7fb60c15ddf0 2024-04-06T12:31:22.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.592+0000 7fb61402a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c15a670 con 0x7fb60c06a730 2024-04-06T12:31:22.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.592+0000 7fb61402a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c15ac30 con 0x7fb60c15d7b0 2024-04-06T12:31:22.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.592+0000 7fb6125c7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c15ddf0 0x7fb60c0cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:22.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.592+0000 7fb6125c7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c15ddf0 0x7fb60c0cdc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50858/0 (socket says 172.21.15.189:50858) 2024-04-06T12:31:22.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.592+0000 7fb6125c7700 1 -- 172.21.15.189:0/701827357 learned_addr learned my addr 172.21.15.189:0/701827357 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:22.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.592+0000 7fb611dc6700 1 --2- 172.21.15.189:0/701827357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb60c06a730 0x7fb60c1f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:22.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.592+0000 7fb6115c5700 1 --2- 172.21.15.189:0/701827357 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb60c15d7b0 0x7fb60c1f6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:22.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.592+0000 7fb6125c7700 1 -- 172.21.15.189:0/701827357 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb60c15d7b0 msgr2=0x7fb60c1f6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:22.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.592+0000 7fb6125c7700 1 --2- 172.21.15.189:0/701827357 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb60c15d7b0 0x7fb60c1f6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.592+0000 7fb6125c7700 1 -- 172.21.15.189:0/701827357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb60c06a730 msgr2=0x7fb60c1f25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:22.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.592+0000 7fb6125c7700 1 --2- 172.21.15.189:0/701827357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb60c06a730 0x7fb60c1f25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.592+0000 7fb6125c7700 1 -- 172.21.15.189:0/701827357 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb608009550 con 0x7fb60c15ddf0 2024-04-06T12:31:22.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.592+0000 7fb6115c5700 1 --2- 172.21.15.189:0/701827357 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb60c15d7b0 0x7fb60c1f6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:22.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.592+0000 7fb6125c7700 1 --2- 172.21.15.189:0/701827357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c15ddf0 0x7fb60c0cdc90 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fb60400b9a0 tx=0x7fb60400bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:22.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.592+0000 7fb5fe7fc700 1 -- 172.21.15.189:0/701827357 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60400c930 con 0x7fb60c15ddf0 2024-04-06T12:31:22.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.593+0000 7fb5fe7fc700 1 -- 172.21.15.189:0/701827357 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb60400ca90 con 0x7fb60c15ddf0 2024-04-06T12:31:22.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.593+0000 7fb61402a700 1 -- 172.21.15.189:0/701827357 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60c1f6de0 con 0x7fb60c15ddf0 2024-04-06T12:31:22.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.593+0000 7fb5fe7fc700 1 -- 172.21.15.189:0/701827357 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60400ce40 con 0x7fb60c15ddf0 2024-04-06T12:31:22.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.593+0000 7fb61402a700 1 -- 172.21.15.189:0/701827357 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb60c15a8c0 con 0x7fb60c15ddf0 2024-04-06T12:31:22.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.593+0000 7fb61402a700 1 -- 172.21.15.189:0/701827357 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb60c067bb0 con 0x7fb60c15ddf0 2024-04-06T12:31:22.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.594+0000 7fb5fe7fc700 1 -- 172.21.15.189:0/701827357 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fb60401d3b0 con 0x7fb60c15ddf0 2024-04-06T12:31:22.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.594+0000 7fb5fe7fc700 1 --2- 172.21.15.189:0/701827357 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb5f006adb0 0x7fb5f006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:22.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.594+0000 7fb5fe7fc700 1 -- 172.21.15.189:0/701827357 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb6040890c0 con 0x7fb60c15ddf0 2024-04-06T12:31:22.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.594+0000 7fb611dc6700 1 --2- 172.21.15.189:0/701827357 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb5f006adb0 0x7fb5f006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:22.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.595+0000 7fb611dc6700 1 --2- 172.21.15.189:0/701827357 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb5f006adb0 0x7fb5f006d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb608009130 tx=0x7fb608008030 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:22.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.599+0000 7fb5fe7fc700 1 -- 172.21.15.189:0/701827357 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb6040540e0 con 0x7fb60c15ddf0 2024-04-06T12:31:22.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.799+0000 7fb61402a700 1 -- 172.21.15.189:0/701827357 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb60c074990 con 0x7fb60c15ddf0 2024-04-06T12:31:22.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.800+0000 7fb5fe7fc700 1 -- 172.21.15.189:0/701827357 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16377 (secure 0 0 0) 0x7fb604057700 con 0x7fb60c15ddf0 2024-04-06T12:31:22.798 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:22.799 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":143,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:31:22.083528+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:31:19.866331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-06T12:31:21.048344+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:31:22.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.803+0000 7fb61402a700 1 -- 172.21.15.189:0/701827357 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb5f006adb0 msgr2=0x7fb5f006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:22.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.803+0000 7fb61402a700 1 --2- 172.21.15.189:0/701827357 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb5f006adb0 0x7fb5f006d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb608009130 tx=0x7fb608008030 comp rx=0 tx=0).stop 2024-04-06T12:31:22.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.803+0000 7fb61402a700 1 -- 172.21.15.189:0/701827357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c15ddf0 msgr2=0x7fb60c0cdc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:22.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.803+0000 7fb61402a700 1 --2- 172.21.15.189:0/701827357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c15ddf0 0x7fb60c0cdc90 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fb60400b9a0 tx=0x7fb60400bcb0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.803+0000 7fb61402a700 1 -- 172.21.15.189:0/701827357 shutdown_connections 2024-04-06T12:31:22.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.803+0000 7fb61402a700 1 --2- 172.21.15.189:0/701827357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb60c15ddf0 0x7fb60c0cdc90 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.803+0000 7fb61402a700 1 --2- 172.21.15.189:0/701827357 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb5f006adb0 0x7fb5f006d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.803+0000 7fb61402a700 1 --2- 172.21.15.189:0/701827357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb60c06a730 0x7fb60c1f25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.803+0000 7fb61402a700 1 --2- 172.21.15.189:0/701827357 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb60c15d7b0 0x7fb60c1f6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.803+0000 7fb61402a700 1 -- 172.21.15.189:0/701827357 >> 172.21.15.189:0/701827357 conn(0x7fb60c0d00b0 msgr2=0x7fb60c0af410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:22.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.803+0000 7fb61402a700 1 -- 172.21.15.189:0/701827357 shutdown_connections 2024-04-06T12:31:22.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.803+0000 7fb61402a700 1 -- 172.21.15.189:0/701827357 wait complete. 2024-04-06T12:31:22.813 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-06T12:31:22.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.940+0000 7f8c11964700 1 Processor -- start 2024-04-06T12:31:22.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.940+0000 7f8c11964700 1 -- start start 2024-04-06T12:31:22.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.940+0000 7f8c11964700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8c0c15d5e0 0x7f8c0c06d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:22.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.940+0000 7f8c11964700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0c06ae20 con 0x7f8c0c0b3290 2024-04-06T12:31:22.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.940+0000 7f8c11964700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0c15d9c0 con 0x7f8c0c0aede0 2024-04-06T12:31:22.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.940+0000 7f8c11964700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c0c06e250 con 0x7f8c0c15d5e0 2024-04-06T12:31:22.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.941+0000 7f8c0affd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8c0c15d5e0 0x7f8c0c06d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:22.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.941+0000 7f8c0b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8c0c0b3290 0x7f8c0c06b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48170/0 (socket says 172.21.15.189:48170) 2024-04-06T12:31:22.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.941+0000 7f8c0b7fe700 1 -- 172.21.15.189:0/3457741391 learned_addr learned my addr 172.21.15.189:0/3457741391 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:22.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.941+0000 7f8c0affd700 1 -- 172.21.15.189:0/3457741391 >> v1:172.21.15.192:6789/0 conn(0x7f8c0c0aede0 legacy=0x7f8c0c0b2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:22.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.941+0000 7f8c0affd700 1 -- 172.21.15.189:0/3457741391 >> v1:172.21.15.189:6789/0 conn(0x7f8c0c0b3290 legacy=0x7f8c0c06b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:22.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.941+0000 7f8c09ffb700 1 -- 172.21.15.189:0/3457741391 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858161525 0 0) 0x7f8c0c06ae20 con 0x7f8c0c0b3290 2024-04-06T12:31:22.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.941+0000 7f8c0affd700 1 -- 172.21.15.189:0/3457741391 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0c06df70 con 0x7f8c0c15d5e0 2024-04-06T12:31:22.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.941+0000 7f8c0affd700 1 --2- 172.21.15.189:0/3457741391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c0c15d5e0 0x7f8c0c06d720 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc009840 tx=0x7f8bfc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa8d4ad59a50f4f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:22.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.941+0000 7f8c09ffb700 1 -- 172.21.15.189:0/3457741391 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc011830 con 0x7f8c0c15d5e0 2024-04-06T12:31:22.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.941+0000 7f8c09ffb700 1 -- 172.21.15.189:0/3457741391 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bfc011990 con 0x7f8c0c15d5e0 2024-04-06T12:31:22.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.941+0000 7f8c09ffb700 1 -- 172.21.15.189:0/3457741391 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc011d20 con 0x7f8c0c15d5e0 2024-04-06T12:31:22.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.942+0000 7f8c11964700 1 -- 172.21.15.189:0/3457741391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c0c15d5e0 msgr2=0x7f8c0c06d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:22.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.942+0000 7f8c11964700 1 --2- 172.21.15.189:0/3457741391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c0c15d5e0 0x7f8c0c06d720 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc009840 tx=0x7f8bfc009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:22.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.942+0000 7f8c11964700 1 -- 172.21.15.189:0/3457741391 shutdown_connections 2024-04-06T12:31:22.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.942+0000 7f8c11964700 1 --2- 172.21.15.189:0/3457741391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c0c15d5e0 0x7f8c0c06d720 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.942+0000 7f8c11964700 1 -- 172.21.15.189:0/3457741391 >> 172.21.15.189:0/3457741391 conn(0x7f8c0c0b1d90 msgr2=0x7f8c0c0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:22.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.942+0000 7f8c11964700 1 -- 172.21.15.189:0/3457741391 shutdown_connections 2024-04-06T12:31:22.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.942+0000 7f8c11964700 1 -- 172.21.15.189:0/3457741391 wait complete. 2024-04-06T12:31:22.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.943+0000 7f8c11964700 1 Processor -- start 2024-04-06T12:31:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.943+0000 7f8c11964700 1 -- start start 2024-04-06T12:31:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.944+0000 7f8c11964700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c0c0aede0 0x7f8c0c14e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.944+0000 7f8c11964700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c0c0b3290 0x7f8c0c0b0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.944+0000 7f8c11964700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c0c15d5e0 0x7f8c0c0b2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.944+0000 7f8c11964700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c1f6080 con 0x7f8c0c15d5e0 2024-04-06T12:31:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.944+0000 7f8c11964700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c1f6ec0 con 0x7f8c0c0b3290 2024-04-06T12:31:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.944+0000 7f8c11964700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c1f62b0 con 0x7f8c0c0aede0 2024-04-06T12:31:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.944+0000 7f8c0b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c0c15d5e0 0x7f8c0c0b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.944+0000 7f8c0b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c0c15d5e0 0x7f8c0c0b2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50862/0 (socket says 172.21.15.189:50862) 2024-04-06T12:31:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.944+0000 7f8c0b7fe700 1 -- 172.21.15.189:0/3273667261 learned_addr learned my addr 172.21.15.189:0/3273667261 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.944+0000 7f8c0affd700 1 --2- 172.21.15.189:0/3273667261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c0c0aede0 0x7f8c0c14e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.944+0000 7f8c0b7fe700 1 -- 172.21.15.189:0/3273667261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c0c0aede0 msgr2=0x7f8c0c14e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.944+0000 7f8c0a7fc700 1 --2- 172.21.15.189:0/3273667261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c0c0b3290 0x7f8c0c0b0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:22.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.944+0000 7f8c0b7fe700 1 --2- 172.21.15.189:0/3273667261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c0c0aede0 0x7f8c0c14e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.944+0000 7f8c0b7fe700 1 -- 172.21.15.189:0/3273667261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c0c0b3290 msgr2=0x7f8c0c0b0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:22.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.944+0000 7f8c0b7fe700 1 --2- 172.21.15.189:0/3273667261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c0c0b3290 0x7f8c0c0b0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:22.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.944+0000 7f8c0b7fe700 1 -- 172.21.15.189:0/3273667261 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc0094a0 con 0x7f8c0c15d5e0 2024-04-06T12:31:22.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.944+0000 7f8c0affd700 1 --2- 172.21.15.189:0/3273667261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c0c0aede0 0x7f8c0c14e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:22.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.944+0000 7f8c0a7fc700 1 --2- 172.21.15.189:0/3273667261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c0c0b3290 0x7f8c0c0b0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:22.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.945+0000 7f8c0b7fe700 1 --2- 172.21.15.189:0/3273667261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c0c15d5e0 0x7f8c0c0b2610 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400cbe0 tx=0x7f8c0400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:22.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.945+0000 7f8bf3fff700 1 -- 172.21.15.189:0/3273667261 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0400abc0 con 0x7f8c0c15d5e0 2024-04-06T12:31:22.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.945+0000 7f8bf3fff700 1 -- 172.21.15.189:0/3273667261 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0400ad40 con 0x7f8c0c15d5e0 2024-04-06T12:31:22.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.945+0000 7f8c11964700 1 -- 172.21.15.189:0/3273667261 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0c1f6590 con 0x7f8c0c15d5e0 2024-04-06T12:31:22.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.945+0000 7f8bf3fff700 1 -- 172.21.15.189:0/3273667261 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04007310 con 0x7f8c0c15d5e0 2024-04-06T12:31:22.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.945+0000 7f8c11964700 1 -- 172.21.15.189:0/3273667261 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c0c1f6b50 con 0x7f8c0c15d5e0 2024-04-06T12:31:22.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.946+0000 7f8c11964700 1 -- 172.21.15.189:0/3273667261 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c0c074990 con 0x7f8c0c15d5e0 2024-04-06T12:31:22.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.946+0000 7f8bf3fff700 1 -- 172.21.15.189:0/3273667261 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f8c04007470 con 0x7f8c0c15d5e0 2024-04-06T12:31:22.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.947+0000 7f8bf3fff700 1 --2- 172.21.15.189:0/3273667261 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8bf806adb0 0x7f8bf806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:22.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.947+0000 7f8bf3fff700 1 -- 172.21.15.189:0/3273667261 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8c040894f0 con 0x7f8c0c15d5e0 2024-04-06T12:31:22.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.947+0000 7f8c0affd700 1 --2- 172.21.15.189:0/3273667261 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8bf806adb0 0x7f8bf806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:22.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.947+0000 7f8c0affd700 1 --2- 172.21.15.189:0/3273667261 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8bf806adb0 0x7f8bf806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc000c00 tx=0x7f8bfc01c040 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:22.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:22.951+0000 7f8bf3fff700 1 -- 172.21.15.189:0/3273667261 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8c040543e0 con 0x7f8c0c15d5e0 2024-04-06T12:31:23.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:23.191+0000 7f8c11964700 1 -- 172.21.15.189:0/3273667261 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8c0c0ad350 con 0x7f8c0c15d5e0 2024-04-06T12:31:24.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.105+0000 7f8bf3fff700 1 -- 172.21.15.189:0/3273667261 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v61) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8c04057a00 con 0x7f8c0c15d5e0 2024-04-06T12:31:24.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.108+0000 7f8c11964700 1 -- 172.21.15.189:0/3273667261 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8bf806adb0 msgr2=0x7f8bf806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:24.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.108+0000 7f8c11964700 1 --2- 172.21.15.189:0/3273667261 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8bf806adb0 0x7f8bf806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc000c00 tx=0x7f8bfc01c040 comp rx=0 tx=0).stop 2024-04-06T12:31:24.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.108+0000 7f8c11964700 1 -- 172.21.15.189:0/3273667261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c0c15d5e0 msgr2=0x7f8c0c0b2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:24.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.108+0000 7f8c11964700 1 --2- 172.21.15.189:0/3273667261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c0c15d5e0 0x7f8c0c0b2610 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400cbe0 tx=0x7f8c0400a060 comp rx=0 tx=0).stop 2024-04-06T12:31:24.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.108+0000 7f8c11964700 1 -- 172.21.15.189:0/3273667261 shutdown_connections 2024-04-06T12:31:24.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.108+0000 7f8c11964700 1 --2- 172.21.15.189:0/3273667261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c0c15d5e0 0x7f8c0c0b2610 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:24.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.108+0000 7f8c11964700 1 --2- 172.21.15.189:0/3273667261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c0c0aede0 0x7f8c0c14e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:24.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.108+0000 7f8c11964700 1 --2- 172.21.15.189:0/3273667261 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8bf806adb0 0x7f8bf806d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:24.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.108+0000 7f8c11964700 1 --2- 172.21.15.189:0/3273667261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c0c0b3290 0x7f8c0c0b0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:24.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.108+0000 7f8c11964700 1 -- 172.21.15.189:0/3273667261 >> 172.21.15.189:0/3273667261 conn(0x7f8c0c0b1d90 msgr2=0x7f8c0c1517f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:24.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.108+0000 7f8c11964700 1 -- 172.21.15.189:0/3273667261 shutdown_connections 2024-04-06T12:31:24.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.108+0000 7f8c11964700 1 -- 172.21.15.189:0/3273667261 wait complete. 2024-04-06T12:31:24.121 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-06T12:31:24.121 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-06T12:31:24.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.254+0000 7f06b13bf700 1 Processor -- start 2024-04-06T12:31:24.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.254+0000 7f06b13bf700 1 -- start start 2024-04-06T12:31:24.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.254+0000 7f06b13bf700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f06ac160b60 0x7f06ac0b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:24.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.254+0000 7f06b13bf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06ac15db10 0x7f06ac142d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:24.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.254+0000 7f06b13bf700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06ac1432a0 0x7f06ac0cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:24.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.254+0000 7f06b13bf700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06ac160f40 con 0x7f06ac1432a0 2024-04-06T12:31:24.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.254+0000 7f06b13bf700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06ac15eb40 con 0x7f06ac160b60 2024-04-06T12:31:24.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.254+0000 7f06b13bf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac1436e0 con 0x7f06ac15db10 2024-04-06T12:31:24.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.255+0000 7f06ab7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06ac1432a0 0x7f06ac0cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:24.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.255+0000 7f06ab7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06ac1432a0 0x7f06ac0cd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50874/0 (socket says 172.21.15.189:50874) 2024-04-06T12:31:24.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.255+0000 7f06ab7fe700 1 -- 172.21.15.189:0/1033064399 learned_addr learned my addr 172.21.15.189:0/1033064399 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:24.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.255+0000 7f06aa7fc700 1 --2- 172.21.15.189:0/1033064399 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06ac15db10 0x7f06ac142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:24.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.255+0000 7f06aaffd700 1 --2- 172.21.15.189:0/1033064399 >> v2:172.21.15.192:3300/0 conn(0x7f06ac160b60 0x7f06ac0b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:24.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.255+0000 7f06ab7fe700 1 -- 172.21.15.189:0/1033064399 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06ac15db10 msgr2=0x7f06ac142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:24.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.255+0000 7f06ab7fe700 1 --2- 172.21.15.189:0/1033064399 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06ac15db10 0x7f06ac142d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:24.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.255+0000 7f06ab7fe700 1 -- 172.21.15.189:0/1033064399 >> v2:172.21.15.192:3300/0 conn(0x7f06ac160b60 msgr2=0x7f06ac0b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:24.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.255+0000 7f06ab7fe700 1 --2- 172.21.15.189:0/1033064399 >> v2:172.21.15.192:3300/0 conn(0x7f06ac160b60 0x7f06ac0b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:24.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.255+0000 7f06ab7fe700 1 -- 172.21.15.189:0/1033064399 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ac0d0bf0 con 0x7f06ac1432a0 2024-04-06T12:31:24.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.255+0000 7f06aa7fc700 1 --2- 172.21.15.189:0/1033064399 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06ac15db10 0x7f06ac142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:24.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.255+0000 7f06aaffd700 1 --2- 172.21.15.189:0/1033064399 >> v2:172.21.15.192:3300/0 conn(0x7f06ac160b60 0x7f06ac0b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:24.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.255+0000 7f06ab7fe700 1 --2- 172.21.15.189:0/1033064399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06ac1432a0 0x7f06ac0cd890 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f06ac06b340 tx=0x7f06a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3f7ac1a22abc2e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:24.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.255+0000 7f06a9ffb700 1 -- 172.21.15.189:0/1033064399 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a400e070 con 0x7f06ac1432a0 2024-04-06T12:31:24.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.255+0000 7f06a9ffb700 1 -- 172.21.15.189:0/1033064399 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a400b9c0 con 0x7f06ac1432a0 2024-04-06T12:31:24.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.255+0000 7f06a9ffb700 1 -- 172.21.15.189:0/1033064399 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a400bd70 con 0x7f06ac1432a0 2024-04-06T12:31:24.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.256+0000 7f06b13bf700 1 -- 172.21.15.189:0/1033064399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06ac1432a0 msgr2=0x7f06ac0cd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:24.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.256+0000 7f06b13bf700 1 --2- 172.21.15.189:0/1033064399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06ac1432a0 0x7f06ac0cd890 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f06ac06b340 tx=0x7f06a400b330 comp rx=0 tx=0).stop 2024-04-06T12:31:24.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.256+0000 7f06b13bf700 1 -- 172.21.15.189:0/1033064399 shutdown_connections 2024-04-06T12:31:24.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.256+0000 7f06b13bf700 1 --2- 172.21.15.189:0/1033064399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06ac1432a0 0x7f06ac0cd890 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:24.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.256+0000 7f06b13bf700 1 --2- 172.21.15.189:0/1033064399 >> v2:172.21.15.192:3300/0 conn(0x7f06ac160b60 0x7f06ac0b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:24.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.256+0000 7f06b13bf700 1 --2- 172.21.15.189:0/1033064399 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06ac15db10 0x7f06ac142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:24.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.256+0000 7f06b13bf700 1 -- 172.21.15.189:0/1033064399 >> 172.21.15.189:0/1033064399 conn(0x7f06ac06a730 msgr2=0x7f06ac06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:24.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.256+0000 7f06b13bf700 1 -- 172.21.15.189:0/1033064399 shutdown_connections 2024-04-06T12:31:24.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.256+0000 7f06b13bf700 1 -- 172.21.15.189:0/1033064399 wait complete. 2024-04-06T12:31:24.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.257+0000 7f06b13bf700 1 Processor -- start 2024-04-06T12:31:24.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.257+0000 7f06b13bf700 1 -- start start 2024-04-06T12:31:24.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.258+0000 7f06b13bf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06ac1432a0 0x7f06ac0c4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:24.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.258+0000 7f06b13bf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06ac15db10 0x7f06ac14be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:24.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.258+0000 7f06b13bf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06ac160b60 0x7f06ac14c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:24.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.258+0000 7f06b13bf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac0b3f90 con 0x7f06ac15db10 2024-04-06T12:31:24.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.258+0000 7f06b13bf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac0b3cb0 con 0x7f06ac1432a0 2024-04-06T12:31:24.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.258+0000 7f06b13bf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac0cd3e0 con 0x7f06ac160b60 2024-04-06T12:31:24.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.258+0000 7f06aa7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06ac15db10 0x7f06ac14be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:24.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.258+0000 7f06aa7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06ac15db10 0x7f06ac14be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50880/0 (socket says 172.21.15.189:50880) 2024-04-06T12:31:24.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.258+0000 7f06aa7fc700 1 -- 172.21.15.189:0/3428287005 learned_addr learned my addr 172.21.15.189:0/3428287005 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:24.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.258+0000 7f06aa7fc700 1 -- 172.21.15.189:0/3428287005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06ac160b60 msgr2=0x7f06ac14c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:24.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.258+0000 7f06aa7fc700 1 --2- 172.21.15.189:0/3428287005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06ac160b60 0x7f06ac14c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:24.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.258+0000 7f06aa7fc700 1 -- 172.21.15.189:0/3428287005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06ac1432a0 msgr2=0x7f06ac0c4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:24.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.258+0000 7f06aaffd700 1 --2- 172.21.15.189:0/3428287005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06ac1432a0 0x7f06ac0c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:24.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.258+0000 7f06aa7fc700 1 --2- 172.21.15.189:0/3428287005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06ac1432a0 0x7f06ac0c4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:24.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.258+0000 7f06aa7fc700 1 -- 172.21.15.189:0/3428287005 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a4009b30 con 0x7f06ac15db10 2024-04-06T12:31:24.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.258+0000 7f06aaffd700 1 --2- 172.21.15.189:0/3428287005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06ac1432a0 0x7f06ac0c4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:24.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.258+0000 7f06aa7fc700 1 --2- 172.21.15.189:0/3428287005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06ac15db10 0x7f06ac14be10 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f06a000e790 tx=0x7f06a000eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:24.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.259+0000 7f06937fe700 1 -- 172.21.15.189:0/3428287005 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a0005c20 con 0x7f06ac15db10 2024-04-06T12:31:24.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.259+0000 7f06937fe700 1 -- 172.21.15.189:0/3428287005 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a0005d80 con 0x7f06ac15db10 2024-04-06T12:31:24.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.259+0000 7f06b13bf700 1 -- 172.21.15.189:0/3428287005 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ac0cd6c0 con 0x7f06ac15db10 2024-04-06T12:31:24.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.259+0000 7f06937fe700 1 -- 172.21.15.189:0/3428287005 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a0006270 con 0x7f06ac15db10 2024-04-06T12:31:24.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.259+0000 7f06b13bf700 1 -- 172.21.15.189:0/3428287005 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ac152140 con 0x7f06ac15db10 2024-04-06T12:31:24.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.260+0000 7f06b13bf700 1 -- 172.21.15.189:0/3428287005 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ac074990 con 0x7f06ac15db10 2024-04-06T12:31:24.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.260+0000 7f06937fe700 1 -- 172.21.15.189:0/3428287005 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f06a00063d0 con 0x7f06ac15db10 2024-04-06T12:31:24.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.260+0000 7f06937fe700 1 --2- 172.21.15.189:0/3428287005 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f069806adb0 0x7f069806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:24.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.260+0000 7f06937fe700 1 -- 172.21.15.189:0/3428287005 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f06a0089230 con 0x7f06ac15db10 2024-04-06T12:31:24.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.260+0000 7f06aaffd700 1 --2- 172.21.15.189:0/3428287005 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f069806adb0 0x7f069806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:24.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.261+0000 7f06aaffd700 1 --2- 172.21.15.189:0/3428287005 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f069806adb0 0x7f069806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f069c0097a0 tx=0x7f069c008040 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:24.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.265+0000 7f06937fe700 1 -- 172.21.15.189:0/3428287005 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f06a00540a0 con 0x7f06ac15db10 2024-04-06T12:31:24.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:24.505+0000 7f06b13bf700 1 -- 172.21.15.189:0/3428287005 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f06ac0b71a0 con 0x7f06ac15db10 2024-04-06T12:31:25.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.109+0000 7f06937fe700 1 -- 172.21.15.189:0/3428287005 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v62) v1 ==== 122+0+0 (secure 0 0 0) 0x7f06a00576c0 con 0x7f06ac15db10 2024-04-06T12:31:25.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.112+0000 7f06b13bf700 1 -- 172.21.15.189:0/3428287005 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f069806adb0 msgr2=0x7f069806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:25.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.112+0000 7f06b13bf700 1 --2- 172.21.15.189:0/3428287005 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f069806adb0 0x7f069806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f069c0097a0 tx=0x7f069c008040 comp rx=0 tx=0).stop 2024-04-06T12:31:25.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.112+0000 7f06b13bf700 1 -- 172.21.15.189:0/3428287005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06ac15db10 msgr2=0x7f06ac14be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:25.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.112+0000 7f06b13bf700 1 --2- 172.21.15.189:0/3428287005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06ac15db10 0x7f06ac14be10 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f06a000e790 tx=0x7f06a000eb50 comp rx=0 tx=0).stop 2024-04-06T12:31:25.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.112+0000 7f06b13bf700 1 -- 172.21.15.189:0/3428287005 shutdown_connections 2024-04-06T12:31:25.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.112+0000 7f06b13bf700 1 --2- 172.21.15.189:0/3428287005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06ac160b60 0x7f06ac14c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.112+0000 7f06b13bf700 1 --2- 172.21.15.189:0/3428287005 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f069806adb0 0x7f069806d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.112+0000 7f06b13bf700 1 --2- 172.21.15.189:0/3428287005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06ac1432a0 0x7f06ac0c4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.112+0000 7f06b13bf700 1 --2- 172.21.15.189:0/3428287005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06ac15db10 0x7f06ac14be10 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.112+0000 7f06b13bf700 1 -- 172.21.15.189:0/3428287005 >> 172.21.15.189:0/3428287005 conn(0x7f06ac06a730 msgr2=0x7f06ac0b0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:25.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.112+0000 7f06b13bf700 1 -- 172.21.15.189:0/3428287005 shutdown_connections 2024-04-06T12:31:25.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.112+0000 7f06b13bf700 1 -- 172.21.15.189:0/3428287005 wait complete. 2024-04-06T12:31:25.124 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-06T12:31:25.124 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:31:25.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.252+0000 7fb37f6d5700 1 Processor -- start 2024-04-06T12:31:25.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.252+0000 7fb37f6d5700 1 -- start start 2024-04-06T12:31:25.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.253+0000 7fb37f6d5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb37815d7b0 0x7fb3780b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.253+0000 7fb37f6d5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb37806a730 0x7fb37806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.253+0000 7fb37f6d5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3780b3990 con 0x7fb37815ddf0 2024-04-06T12:31:25.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.253+0000 7fb37f6d5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb37806b0f0 con 0x7fb37806a730 2024-04-06T12:31:25.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.253+0000 7fb37f6d5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3780b3060 con 0x7fb37815d7b0 2024-04-06T12:31:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.253+0000 7fb37cc70700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb37806a730 0x7fb37806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.253+0000 7fb37cc70700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb37806a730 0x7fb37806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50884/0 (socket says 172.21.15.189:50884) 2024-04-06T12:31:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.253+0000 7fb37cc70700 1 -- 172.21.15.189:0/639363965 learned_addr learned my addr 172.21.15.189:0/639363965 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.253+0000 7fb37cc70700 1 -- 172.21.15.189:0/639363965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb37815d7b0 msgr2=0x7fb3780b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.253+0000 7fb37cc70700 1 --2- 172.21.15.189:0/639363965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb37815d7b0 0x7fb3780b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.253+0000 7fb37cc70700 1 -- 172.21.15.189:0/639363965 >> v1:172.21.15.192:6789/0 conn(0x7fb37815ddf0 legacy=0x7fb37815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.253+0000 7fb37cc70700 1 -- 172.21.15.189:0/639363965 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3780b3340 con 0x7fb37806a730 2024-04-06T12:31:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.253+0000 7fb37cc70700 1 --2- 172.21.15.189:0/639363965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb37806a730 0x7fb37806abb0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fb3740098f0 tx=0x7fb374009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1b4705713a7b135 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.253+0000 7fb36ffff700 1 -- 172.21.15.189:0/639363965 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb374004230 con 0x7fb37806a730 2024-04-06T12:31:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.253+0000 7fb36ffff700 1 -- 172.21.15.189:0/639363965 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb374004390 con 0x7fb37806a730 2024-04-06T12:31:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.253+0000 7fb36ffff700 1 -- 172.21.15.189:0/639363965 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3740036a0 con 0x7fb37806a730 2024-04-06T12:31:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.254+0000 7fb37f6d5700 1 -- 172.21.15.189:0/639363965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb37806a730 msgr2=0x7fb37806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.254+0000 7fb37f6d5700 1 --2- 172.21.15.189:0/639363965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb37806a730 0x7fb37806abb0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fb3740098f0 tx=0x7fb374009c00 comp rx=0 tx=0).stop 2024-04-06T12:31:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.254+0000 7fb37f6d5700 1 -- 172.21.15.189:0/639363965 shutdown_connections 2024-04-06T12:31:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.254+0000 7fb37f6d5700 1 --2- 172.21.15.189:0/639363965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb37815d7b0 0x7fb3780b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.254+0000 7fb37f6d5700 1 --2- 172.21.15.189:0/639363965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb37806a730 0x7fb37806abb0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.254+0000 7fb37f6d5700 1 -- 172.21.15.189:0/639363965 >> 172.21.15.189:0/639363965 conn(0x7fb3780d00b0 msgr2=0x7fb3780d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:25.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.254+0000 7fb37f6d5700 1 -- 172.21.15.189:0/639363965 shutdown_connections 2024-04-06T12:31:25.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.254+0000 7fb37f6d5700 1 -- 172.21.15.189:0/639363965 wait complete. 2024-04-06T12:31:25.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.255+0000 7fb37f6d5700 1 Processor -- start 2024-04-06T12:31:25.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.255+0000 7fb37f6d5700 1 -- start start 2024-04-06T12:31:25.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.256+0000 7fb37f6d5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb37806a730 0x7fb3781510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.256+0000 7fb37f6d5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb37815d7b0 0x7fb3780ce590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.256+0000 7fb37f6d5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb37815ddf0 0x7fb3781556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.256+0000 7fb37f6d5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3781515e0 con 0x7fb37806a730 2024-04-06T12:31:25.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.256+0000 7fb37f6d5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3780cead0 con 0x7fb37815ddf0 2024-04-06T12:31:25.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.256+0000 7fb37f6d5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3781f6810 con 0x7fb37815d7b0 2024-04-06T12:31:25.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.256+0000 7fb37d471700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb37806a730 0x7fb3781510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:25.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.256+0000 7fb37d471700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb37806a730 0x7fb3781510a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50900/0 (socket says 172.21.15.189:50900) 2024-04-06T12:31:25.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.256+0000 7fb37d471700 1 -- 172.21.15.189:0/419284236 learned_addr learned my addr 172.21.15.189:0/419284236 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:25.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.256+0000 7fb37cc70700 1 --2- 172.21.15.189:0/419284236 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb37815d7b0 0x7fb3780ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:25.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.256+0000 7fb37dc72700 1 --2- 172.21.15.189:0/419284236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb37815ddf0 0x7fb3781556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:25.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.256+0000 7fb37d471700 1 -- 172.21.15.189:0/419284236 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb37815d7b0 msgr2=0x7fb3780ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:25.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.256+0000 7fb37d471700 1 --2- 172.21.15.189:0/419284236 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb37815d7b0 0x7fb3780ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.256+0000 7fb37d471700 1 -- 172.21.15.189:0/419284236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb37815ddf0 msgr2=0x7fb3781556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:25.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.256+0000 7fb37d471700 1 --2- 172.21.15.189:0/419284236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb37815ddf0 0x7fb3781556e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.256+0000 7fb37d471700 1 -- 172.21.15.189:0/419284236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb374009550 con 0x7fb37806a730 2024-04-06T12:31:25.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.257+0000 7fb37d471700 1 --2- 172.21.15.189:0/419284236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb37806a730 0x7fb3781510a0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fb36800b560 tx=0x7fb36800b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:25.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.257+0000 7fb36dffb700 1 -- 172.21.15.189:0/419284236 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb368011830 con 0x7fb37806a730 2024-04-06T12:31:25.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.257+0000 7fb36dffb700 1 -- 172.21.15.189:0/419284236 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb368011990 con 0x7fb37806a730 2024-04-06T12:31:25.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.257+0000 7fb37f6d5700 1 -- 172.21.15.189:0/419284236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3781f6af0 con 0x7fb37806a730 2024-04-06T12:31:25.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.257+0000 7fb36dffb700 1 -- 172.21.15.189:0/419284236 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb368011d40 con 0x7fb37806a730 2024-04-06T12:31:25.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.257+0000 7fb37f6d5700 1 -- 172.21.15.189:0/419284236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3781f70b0 con 0x7fb37806a730 2024-04-06T12:31:25.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.258+0000 7fb37f6d5700 1 -- 172.21.15.189:0/419284236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb378067bb0 con 0x7fb37806a730 2024-04-06T12:31:25.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.258+0000 7fb36dffb700 1 -- 172.21.15.189:0/419284236 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fb36801a400 con 0x7fb37806a730 2024-04-06T12:31:25.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.258+0000 7fb36dffb700 1 --2- 172.21.15.189:0/419284236 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb35c06adb0 0x7fb35c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.259+0000 7fb36dffb700 1 -- 172.21.15.189:0/419284236 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb368089590 con 0x7fb37806a730 2024-04-06T12:31:25.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.259+0000 7fb37cc70700 1 --2- 172.21.15.189:0/419284236 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb35c06adb0 0x7fb35c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:25.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.259+0000 7fb37cc70700 1 --2- 172.21.15.189:0/419284236 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb35c06adb0 0x7fb35c06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb374009130 tx=0x7fb374008030 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:25.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.263+0000 7fb36dffb700 1 -- 172.21.15.189:0/419284236 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb3680545b0 con 0x7fb37806a730 2024-04-06T12:31:25.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.502+0000 7fb37f6d5700 1 -- 172.21.15.189:0/419284236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb378074990 con 0x7fb37806a730 2024-04-06T12:31:25.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.503+0000 7fb36dffb700 1 -- 172.21.15.189:0/419284236 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4745 (secure 0 0 0) 0x7fb368057bd0 con 0x7fb37806a730 2024-04-06T12:31:25.501 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:25.501 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7115,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/538134862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":538134862},{"type":"v1","addr":"172.21.15.192:6834","nonce":538134862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7915,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1788966297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1788966297},{"type":"v1","addr":"172.21.15.189:6837","nonce":1788966297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7930,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2932927074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2932927074},{"type":"v1","addr":"172.21.15.192:6835","nonce":2932927074}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:31:22.083923+0000","modified":"2024-04-06T12:31:25.104510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/4249639368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":4249639368},{"type":"v1","addr":"172.21.15.189:6836","nonce":4249639368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-06T12:31:25.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.505+0000 7fb37f6d5700 1 -- 172.21.15.189:0/419284236 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb35c06adb0 msgr2=0x7fb35c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:25.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.505+0000 7fb37f6d5700 1 --2- 172.21.15.189:0/419284236 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb35c06adb0 0x7fb35c06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb374009130 tx=0x7fb374008030 comp rx=0 tx=0).stop 2024-04-06T12:31:25.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.505+0000 7fb37f6d5700 1 -- 172.21.15.189:0/419284236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb37806a730 msgr2=0x7fb3781510a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:25.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.505+0000 7fb37f6d5700 1 --2- 172.21.15.189:0/419284236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb37806a730 0x7fb3781510a0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fb36800b560 tx=0x7fb36800b920 comp rx=0 tx=0).stop 2024-04-06T12:31:25.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.505+0000 7fb37f6d5700 1 -- 172.21.15.189:0/419284236 shutdown_connections 2024-04-06T12:31:25.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.505+0000 7fb37f6d5700 1 --2- 172.21.15.189:0/419284236 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb35c06adb0 0x7fb35c06d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.505+0000 7fb37f6d5700 1 --2- 172.21.15.189:0/419284236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb37815ddf0 0x7fb3781556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.505+0000 7fb37f6d5700 1 --2- 172.21.15.189:0/419284236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb37806a730 0x7fb3781510a0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.505+0000 7fb37f6d5700 1 --2- 172.21.15.189:0/419284236 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb37815d7b0 0x7fb3780ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.505+0000 7fb37f6d5700 1 -- 172.21.15.189:0/419284236 >> 172.21.15.189:0/419284236 conn(0x7fb3780d00b0 msgr2=0x7fb3780b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:25.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.505+0000 7fb37f6d5700 1 -- 172.21.15.189:0/419284236 shutdown_connections 2024-04-06T12:31:25.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.505+0000 7fb37f6d5700 1 -- 172.21.15.189:0/419284236 wait complete. 2024-04-06T12:31:25.504 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 62 2024-04-06T12:31:25.515 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:31:25.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.639+0000 7fd54400f700 1 Processor -- start 2024-04-06T12:31:25.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.640+0000 7fd54400f700 1 -- start start 2024-04-06T12:31:25.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.641+0000 7fd54400f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd53c06ae40 0x7fd53c15bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.641+0000 7fd54400f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd53c15c100 0x7fd53c0ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.641+0000 7fd54400f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd53c15f9e0 con 0x7fd53c158fe0 2024-04-06T12:31:25.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.641+0000 7fd54400f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd53c15f0d0 con 0x7fd53c06ae40 2024-04-06T12:31:25.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.641+0000 7fd54400f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c15c8f0 con 0x7fd53c15c100 2024-04-06T12:31:25.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.641+0000 7fd5415aa700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd53c06ae40 0x7fd53c15bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:25.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.641+0000 7fd541dab700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd53c158fe0 0x7fd53c06a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48184/0 (socket says 172.21.15.189:48184) 2024-04-06T12:31:25.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.641+0000 7fd541dab700 1 -- 172.21.15.189:0/3641695185 learned_addr learned my addr 172.21.15.189:0/3641695185 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:25.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.641+0000 7fd5425ac700 1 --2- 172.21.15.189:0/3641695185 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd53c15c100 0x7fd53c0ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:25.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.641+0000 7fd5415aa700 1 -- 172.21.15.189:0/3641695185 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd53c15c100 msgr2=0x7fd53c0ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:25.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.641+0000 7fd5415aa700 1 --2- 172.21.15.189:0/3641695185 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd53c15c100 0x7fd53c0ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.641+0000 7fd5415aa700 1 -- 172.21.15.189:0/3641695185 >> v1:172.21.15.189:6789/0 conn(0x7fd53c158fe0 legacy=0x7fd53c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:25.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.641+0000 7fd540da9700 1 -- 172.21.15.189:0/3641695185 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033012447 0 0) 0x7fd53c15f9e0 con 0x7fd53c158fe0 2024-04-06T12:31:25.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.641+0000 7fd5415aa700 1 -- 172.21.15.189:0/3641695185 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53c15f390 con 0x7fd53c06ae40 2024-04-06T12:31:25.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.641+0000 7fd5425ac700 1 --2- 172.21.15.189:0/3641695185 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd53c15c100 0x7fd53c0ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:25.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.641+0000 7fd5415aa700 1 --2- 172.21.15.189:0/3641695185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd53c06ae40 0x7fd53c15bbc0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fd530009840 tx=0x7fd530009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3142da0d731fc8b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:25.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.642+0000 7fd540da9700 1 -- 172.21.15.189:0/3641695185 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd530011830 con 0x7fd53c06ae40 2024-04-06T12:31:25.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.642+0000 7fd540da9700 1 -- 172.21.15.189:0/3641695185 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd530011990 con 0x7fd53c06ae40 2024-04-06T12:31:25.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.642+0000 7fd540da9700 1 -- 172.21.15.189:0/3641695185 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd530011d20 con 0x7fd53c06ae40 2024-04-06T12:31:25.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.642+0000 7fd54400f700 1 -- 172.21.15.189:0/3641695185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd53c06ae40 msgr2=0x7fd53c15bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:25.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.642+0000 7fd54400f700 1 --2- 172.21.15.189:0/3641695185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd53c06ae40 0x7fd53c15bbc0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fd530009840 tx=0x7fd530009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:25.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.642+0000 7fd54400f700 1 -- 172.21.15.189:0/3641695185 shutdown_connections 2024-04-06T12:31:25.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.642+0000 7fd54400f700 1 --2- 172.21.15.189:0/3641695185 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd53c15c100 0x7fd53c0ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.642+0000 7fd54400f700 1 --2- 172.21.15.189:0/3641695185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd53c06ae40 0x7fd53c15bbc0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.642+0000 7fd54400f700 1 -- 172.21.15.189:0/3641695185 >> 172.21.15.189:0/3641695185 conn(0x7fd53c0d00b0 msgr2=0x7fd53c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:25.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.642+0000 7fd54400f700 1 -- 172.21.15.189:0/3641695185 shutdown_connections 2024-04-06T12:31:25.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.642+0000 7fd54400f700 1 -- 172.21.15.189:0/3641695185 wait complete. 2024-04-06T12:31:25.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.644+0000 7fd54400f700 1 Processor -- start 2024-04-06T12:31:25.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.644+0000 7fd54400f700 1 -- start start 2024-04-06T12:31:25.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd54400f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd53c06ae40 0x7fd53c13ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd54400f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd53c158fe0 0x7fd53c157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd54400f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd53c15c100 0x7fd53c1f4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd54400f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c158330 con 0x7fd53c06ae40 2024-04-06T12:31:25.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd54400f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c1f4e20 con 0x7fd53c15c100 2024-04-06T12:31:25.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd54400f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c1f5100 con 0x7fd53c158fe0 2024-04-06T12:31:25.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd541dab700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd53c06ae40 0x7fd53c13ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:25.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd541dab700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd53c06ae40 0x7fd53c13ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50918/0 (socket says 172.21.15.189:50918) 2024-04-06T12:31:25.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd541dab700 1 -- 172.21.15.189:0/28014119 learned_addr learned my addr 172.21.15.189:0/28014119 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:25.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd5415aa700 1 --2- 172.21.15.189:0/28014119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd53c158fe0 0x7fd53c157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:25.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd5425ac700 1 --2- 172.21.15.189:0/28014119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd53c15c100 0x7fd53c1f4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:25.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd541dab700 1 -- 172.21.15.189:0/28014119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd53c158fe0 msgr2=0x7fd53c157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:25.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd541dab700 1 --2- 172.21.15.189:0/28014119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd53c158fe0 0x7fd53c157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd541dab700 1 -- 172.21.15.189:0/28014119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd53c15c100 msgr2=0x7fd53c1f4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:25.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd541dab700 1 --2- 172.21.15.189:0/28014119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd53c15c100 0x7fd53c1f4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd541dab700 1 -- 172.21.15.189:0/28014119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5300094a0 con 0x7fd53c06ae40 2024-04-06T12:31:25.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd5415aa700 1 --2- 172.21.15.189:0/28014119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd53c158fe0 0x7fd53c157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:25.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd541dab700 1 --2- 172.21.15.189:0/28014119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd53c06ae40 0x7fd53c13ea80 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fd5380011e0 tx=0x7fd53800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:25.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd52e7fc700 1 -- 172.21.15.189:0/28014119 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd538004c80 con 0x7fd53c06ae40 2024-04-06T12:31:25.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd52e7fc700 1 -- 172.21.15.189:0/28014119 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd538004e00 con 0x7fd53c06ae40 2024-04-06T12:31:25.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd54400f700 1 -- 172.21.15.189:0/28014119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd53c1f53e0 con 0x7fd53c06ae40 2024-04-06T12:31:25.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd52e7fc700 1 -- 172.21.15.189:0/28014119 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd538007310 con 0x7fd53c06ae40 2024-04-06T12:31:25.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.645+0000 7fd54400f700 1 -- 172.21.15.189:0/28014119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd53c141570 con 0x7fd53c06ae40 2024-04-06T12:31:25.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.646+0000 7fd54400f700 1 -- 172.21.15.189:0/28014119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd53c067bb0 con 0x7fd53c06ae40 2024-04-06T12:31:25.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.647+0000 7fd52e7fc700 1 -- 172.21.15.189:0/28014119 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fd538007470 con 0x7fd53c06ae40 2024-04-06T12:31:25.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.647+0000 7fd52e7fc700 1 --2- 172.21.15.189:0/28014119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd52006ade0 0x7fd52006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.647+0000 7fd52e7fc700 1 -- 172.21.15.189:0/28014119 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd5380893a0 con 0x7fd53c06ae40 2024-04-06T12:31:25.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.647+0000 7fd5415aa700 1 --2- 172.21.15.189:0/28014119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd52006ade0 0x7fd52006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:25.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.648+0000 7fd5415aa700 1 --2- 172.21.15.189:0/28014119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd52006ade0 0x7fd52006d2a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd5300090b0 tx=0x7fd530008030 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:25.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.652+0000 7fd52e7fc700 1 -- 172.21.15.189:0/28014119 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd538054330 con 0x7fd53c06ae40 2024-04-06T12:31:25.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.848+0000 7fd54400f700 1 -- 172.21.15.189:0/28014119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd53c141760 con 0x7fd53c06ae40 2024-04-06T12:31:25.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.849+0000 7fd52e7fc700 1 -- 172.21.15.189:0/28014119 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+16364 (secure 0 0 0) 0x7fd538057950 con 0x7fd53c06ae40 2024-04-06T12:31:25.847 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:25.847 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":144,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:31:23.086230+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:31:19.866331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-06T12:31:21.048344+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:31:25.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.851+0000 7fd54400f700 1 -- 172.21.15.189:0/28014119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd52006ade0 msgr2=0x7fd52006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:25.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.851+0000 7fd54400f700 1 --2- 172.21.15.189:0/28014119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd52006ade0 0x7fd52006d2a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd5300090b0 tx=0x7fd530008030 comp rx=0 tx=0).stop 2024-04-06T12:31:25.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.851+0000 7fd54400f700 1 -- 172.21.15.189:0/28014119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd53c06ae40 msgr2=0x7fd53c13ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:25.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.851+0000 7fd54400f700 1 --2- 172.21.15.189:0/28014119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd53c06ae40 0x7fd53c13ea80 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fd5380011e0 tx=0x7fd53800cae0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.851+0000 7fd54400f700 1 -- 172.21.15.189:0/28014119 shutdown_connections 2024-04-06T12:31:25.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.851+0000 7fd54400f700 1 --2- 172.21.15.189:0/28014119 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd52006ade0 0x7fd52006d2a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.851+0000 7fd54400f700 1 --2- 172.21.15.189:0/28014119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd53c15c100 0x7fd53c1f4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.851+0000 7fd54400f700 1 --2- 172.21.15.189:0/28014119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd53c06ae40 0x7fd53c13ea80 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.851+0000 7fd54400f700 1 --2- 172.21.15.189:0/28014119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd53c158fe0 0x7fd53c157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.851+0000 7fd54400f700 1 -- 172.21.15.189:0/28014119 >> 172.21.15.189:0/28014119 conn(0x7fd53c0d00b0 msgr2=0x7fd53c1438e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:25.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.851+0000 7fd54400f700 1 -- 172.21.15.189:0/28014119 shutdown_connections 2024-04-06T12:31:25.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.851+0000 7fd54400f700 1 -- 172.21.15.189:0/28014119 wait complete. 2024-04-06T12:31:25.861 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-06T12:31:25.862 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:31:25.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.984+0000 7f363b26e700 1 Processor -- start 2024-04-06T12:31:25.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.984+0000 7f363b26e700 1 -- start start 2024-04-06T12:31:25.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.985+0000 7f363b26e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f36340b4660 0x7f36340b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.985+0000 7f363b26e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36340b5030 con 0x7f363415a550 2024-04-06T12:31:25.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.985+0000 7f363b26e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36340cdbe0 con 0x7f36340b2fc0 2024-04-06T12:31:25.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.985+0000 7f363b26e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f363415a940 con 0x7f36340b4660 2024-04-06T12:31:25.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.985+0000 7f363900a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f363415a550 0x7f36340cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48190/0 (socket says 172.21.15.189:48190) 2024-04-06T12:31:25.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.985+0000 7f363900a700 1 -- 172.21.15.189:0/4084175580 learned_addr learned my addr 172.21.15.189:0/4084175580 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:25.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.985+0000 7f3638809700 1 --2- 172.21.15.189:0/4084175580 >> v2:172.21.15.192:3300/0 conn(0x7f36340b4660 0x7f36340b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:25.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.985+0000 7f362bfff700 1 -- 172.21.15.189:0/4084175580 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254487092 0 0) 0x7f36340b5030 con 0x7f363415a550 2024-04-06T12:31:25.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f362bfff700 1 -- 172.21.15.189:0/4084175580 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36200035b0 con 0x7f363415a550 2024-04-06T12:31:25.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f362bfff700 1 -- 172.21.15.189:0/4084175580 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4134616777 0 0) 0x7f36200035b0 con 0x7f363415a550 2024-04-06T12:31:25.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f362bfff700 1 -- 172.21.15.189:0/4084175580 >> v2:172.21.15.192:3300/0 conn(0x7f36340b4660 msgr2=0x7f36340b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:25.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f362bfff700 1 --2- 172.21.15.189:0/4084175580 >> v2:172.21.15.192:3300/0 conn(0x7f36340b4660 0x7f36340b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f362bfff700 1 -- 172.21.15.189:0/4084175580 >> v1:172.21.15.192:6789/0 conn(0x7f36340b2fc0 legacy=0x7f36340b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:25.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f362bfff700 1 -- 172.21.15.189:0/4084175580 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363415ac00 con 0x7f363415a550 2024-04-06T12:31:25.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f362bfff700 1 -- 172.21.15.189:0/4084175580 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f3624001670 con 0x7f363415a550 2024-04-06T12:31:25.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f362bfff700 1 -- 172.21.15.189:0/4084175580 >> v1:172.21.15.189:6789/0 conn(0x7f363415a550 legacy=0x7f36340cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:25.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f362bfff700 1 --2- 172.21.15.189:0/4084175580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3620003a00 0x7f3620003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f362bfff700 1 --2- 172.21.15.189:0/4084175580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3620004850 0x7f3620004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f362bfff700 1 --2- 172.21.15.189:0/4084175580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3620005210 0x7f362000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f362bfff700 1 -- 172.21.15.189:0/4084175580 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363415a940 con 0x7f3620004850 2024-04-06T12:31:25.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f362bfff700 1 -- 172.21.15.189:0/4084175580 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362000c0b0 con 0x7f3620003a00 2024-04-06T12:31:25.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f362bfff700 1 -- 172.21.15.189:0/4084175580 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362000c270 con 0x7f3620005210 2024-04-06T12:31:25.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f3638809700 1 --2- 172.21.15.189:0/4084175580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3620004850 0x7f3620004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:25.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f363900a700 1 --2- 172.21.15.189:0/4084175580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3620003a00 0x7f3620003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:25.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f363980b700 1 --2- 172.21.15.189:0/4084175580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3620005210 0x7f362000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:25.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f3638809700 1 -- 172.21.15.189:0/4084175580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3620005210 msgr2=0x7f362000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:25.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f3638809700 1 --2- 172.21.15.189:0/4084175580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3620005210 0x7f362000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f3638809700 1 -- 172.21.15.189:0/4084175580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3620003a00 msgr2=0x7f3620003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:25.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f3638809700 1 --2- 172.21.15.189:0/4084175580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3620003a00 0x7f3620003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f3638809700 1 -- 172.21.15.189:0/4084175580 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36200035b0 con 0x7f3620004850 2024-04-06T12:31:25.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f363980b700 1 -- 172.21.15.189:0/4084175580 reap_dead start 2024-04-06T12:31:25.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.986+0000 7f3638809700 1 --2- 172.21.15.189:0/4084175580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3620004850 0x7f3620004cd0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f3630009b60 tx=0x7f3620005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d4f99f734f2ff1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:25.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.987+0000 7f362bfff700 1 -- 172.21.15.189:0/4084175580 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363000e6c0 con 0x7f3620004850 2024-04-06T12:31:25.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.987+0000 7f362bfff700 1 -- 172.21.15.189:0/4084175580 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f363000e820 con 0x7f3620004850 2024-04-06T12:31:25.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.987+0000 7f362bfff700 1 -- 172.21.15.189:0/4084175580 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363000eb90 con 0x7f3620004850 2024-04-06T12:31:25.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.987+0000 7f363b26e700 1 -- 172.21.15.189:0/4084175580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3620004850 msgr2=0x7f3620004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:25.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.987+0000 7f363b26e700 1 --2- 172.21.15.189:0/4084175580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3620004850 0x7f3620004cd0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f3630009b60 tx=0x7f3620005880 comp rx=0 tx=0).stop 2024-04-06T12:31:25.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.987+0000 7f363b26e700 1 -- 172.21.15.189:0/4084175580 shutdown_connections 2024-04-06T12:31:25.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.987+0000 7f363b26e700 1 --2- 172.21.15.189:0/4084175580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3620004850 0x7f3620004cd0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.987+0000 7f363b26e700 1 -- 172.21.15.189:0/4084175580 >> 172.21.15.189:0/4084175580 conn(0x7f36340d00b0 msgr2=0x7f36340d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:25.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.987+0000 7f363b26e700 1 -- 172.21.15.189:0/4084175580 shutdown_connections 2024-04-06T12:31:25.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.987+0000 7f363b26e700 1 -- 172.21.15.189:0/4084175580 wait complete. 2024-04-06T12:31:25.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.988+0000 7f363b26e700 1 Processor -- start 2024-04-06T12:31:25.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.988+0000 7f363b26e700 1 -- start start 2024-04-06T12:31:25.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.989+0000 7f363b26e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3620004850 0x7f3634157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.989+0000 7f363b26e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f363414ae60 0x7f363406a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.989+0000 7f363b26e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3634144660 0x7f3634144af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.989+0000 7f363b26e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36341487c0 con 0x7f363414ae60 2024-04-06T12:31:25.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.989+0000 7f363b26e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3634154bc0 con 0x7f3634144660 2024-04-06T12:31:25.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.989+0000 7f363b26e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3634145030 con 0x7f3620004850 2024-04-06T12:31:25.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.989+0000 7f3638809700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f363414ae60 0x7f363406a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:25.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.989+0000 7f3638809700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f363414ae60 0x7f363406a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50948/0 (socket says 172.21.15.189:50948) 2024-04-06T12:31:25.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.989+0000 7f3638809700 1 -- 172.21.15.189:0/4005679067 learned_addr learned my addr 172.21.15.189:0/4005679067 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:25.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.989+0000 7f363980b700 1 --2- 172.21.15.189:0/4005679067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3634144660 0x7f3634144af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:25.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.989+0000 7f363900a700 1 --2- 172.21.15.189:0/4005679067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3620004850 0x7f3634157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:25.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.990+0000 7f3638809700 1 -- 172.21.15.189:0/4005679067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3620004850 msgr2=0x7f3634157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:25.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.990+0000 7f3638809700 1 --2- 172.21.15.189:0/4005679067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3620004850 0x7f3634157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.990+0000 7f3638809700 1 -- 172.21.15.189:0/4005679067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3634144660 msgr2=0x7f3634144af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:25.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.990+0000 7f3638809700 1 --2- 172.21.15.189:0/4005679067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3634144660 0x7f3634144af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:25.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.990+0000 7f3638809700 1 -- 172.21.15.189:0/4005679067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36341583e0 con 0x7f363414ae60 2024-04-06T12:31:25.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.990+0000 7f3638809700 1 --2- 172.21.15.189:0/4005679067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f363414ae60 0x7f363406a9a0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f362000c080 tx=0x7f36300047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:25.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.990+0000 7f3629ffb700 1 -- 172.21.15.189:0/4005679067 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363000e6c0 con 0x7f363414ae60 2024-04-06T12:31:25.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.990+0000 7f3629ffb700 1 -- 172.21.15.189:0/4005679067 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3630003ad0 con 0x7f363414ae60 2024-04-06T12:31:25.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.990+0000 7f363b26e700 1 -- 172.21.15.189:0/4005679067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3634154e60 con 0x7f363414ae60 2024-04-06T12:31:25.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.990+0000 7f3629ffb700 1 -- 172.21.15.189:0/4005679067 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3630003e80 con 0x7f363414ae60 2024-04-06T12:31:25.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.990+0000 7f363b26e700 1 -- 172.21.15.189:0/4005679067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36341586c0 con 0x7f363414ae60 2024-04-06T12:31:25.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.991+0000 7f363b26e700 1 -- 172.21.15.189:0/4005679067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3634155380 con 0x7f363414ae60 2024-04-06T12:31:25.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.991+0000 7f3629ffb700 1 -- 172.21.15.189:0/4005679067 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f3630013470 con 0x7f363414ae60 2024-04-06T12:31:25.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.992+0000 7f3629ffb700 1 --2- 172.21.15.189:0/4005679067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f361806adb0 0x7f361806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:25.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.992+0000 7f3629ffb700 1 -- 172.21.15.189:0/4005679067 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f363008ad10 con 0x7f363414ae60 2024-04-06T12:31:25.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.992+0000 7f363900a700 1 --2- 172.21.15.189:0/4005679067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f361806adb0 0x7f361806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:25.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.992+0000 7f363900a700 1 --2- 172.21.15.189:0/4005679067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f361806adb0 0x7f361806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3624001790 tx=0x7f3624009040 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:25.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:25.996+0000 7f3629ffb700 1 -- 172.21.15.189:0/4005679067 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3634155380 con 0x7f363414ae60 2024-04-06T12:31:26.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.188+0000 7f363b26e700 1 -- 172.21.15.189:0/4005679067 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f36341487c0 con 0x7f361806adb0 2024-04-06T12:31:26.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.196+0000 7f3629ffb700 1 -- 172.21.15.189:0/4005679067 <== mgr.8376 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289707 (secure 0 0 0) 0x7f36341487c0 con 0x7f361806adb0 2024-04-06T12:31:26.198 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:26.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.199+0000 7f363b26e700 1 -- 172.21.15.189:0/4005679067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f361806adb0 msgr2=0x7f361806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:26.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.199+0000 7f363b26e700 1 --2- 172.21.15.189:0/4005679067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f361806adb0 0x7f361806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3624001790 tx=0x7f3624009040 comp rx=0 tx=0).stop 2024-04-06T12:31:26.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.199+0000 7f363b26e700 1 -- 172.21.15.189:0/4005679067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f363414ae60 msgr2=0x7f363406a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:26.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.199+0000 7f363b26e700 1 --2- 172.21.15.189:0/4005679067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f363414ae60 0x7f363406a9a0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f362000c080 tx=0x7f36300047c0 comp rx=0 tx=0).stop 2024-04-06T12:31:26.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.199+0000 7f363b26e700 1 -- 172.21.15.189:0/4005679067 shutdown_connections 2024-04-06T12:31:26.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.199+0000 7f363b26e700 1 --2- 172.21.15.189:0/4005679067 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f361806adb0 0x7f361806d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:26.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.199+0000 7f363b26e700 1 --2- 172.21.15.189:0/4005679067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3634144660 0x7f3634144af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:26.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.199+0000 7f363b26e700 1 --2- 172.21.15.189:0/4005679067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3620004850 0x7f3634157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:26.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.199+0000 7f363b26e700 1 --2- 172.21.15.189:0/4005679067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f363414ae60 0x7f363406a9a0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:26.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.199+0000 7f363b26e700 1 -- 172.21.15.189:0/4005679067 >> 172.21.15.189:0/4005679067 conn(0x7f36340d00b0 msgr2=0x7f36340cf5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:26.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.200+0000 7f363b26e700 1 -- 172.21.15.189:0/4005679067 shutdown_connections 2024-04-06T12:31:26.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.200+0000 7f363b26e700 1 -- 172.21.15.189:0/4005679067 wait complete. 2024-04-06T12:31:26.201 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:31:26.210 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-06T12:31:24.447504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":403,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":265,"ondisk_log_size":265,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":129,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330552,"kb_used_data":15140,"kb_used_omap":12,"kb_used_meta":315251,"kb_avail":754644168,"statfs":{"total":773094113280,"available":772755628032,"internally_reserved":0,"allocated":15503360,"data_stored":8342445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":322817592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000646"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.114426+0000","last_change":"2024-04-06T12:31:22.113995+0000","last_active":"2024-04-06T12:31:22.114426+0000","last_peered":"2024-04-06T12:31:22.114426+0000","last_clean":"2024-04-06T12:31:22.114426+0000","last_became_active":"2024-04-06T12:31:22.113724+0000","last_became_peered":"2024-04-06T12:31:22.113724+0000","last_unstale":"2024-04-06T12:31:22.114426+0000","last_undegraded":"2024-04-06T12:31:22.114426+0000","last_fullsized":"2024-04-06T12:31:22.114426+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:42.850998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119229+0000","last_change":"2024-04-06T12:31:22.112749+0000","last_active":"2024-04-06T12:31:22.119229+0000","last_peered":"2024-04-06T12:31:22.119229+0000","last_clean":"2024-04-06T12:31:22.119229+0000","last_became_active":"2024-04-06T12:31:22.112498+0000","last_became_peered":"2024-04-06T12:31:22.112498+0000","last_unstale":"2024-04-06T12:31:22.119229+0000","last_undegraded":"2024-04-06T12:31:22.119229+0000","last_fullsized":"2024-04-06T12:31:22.119229+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:34:53.631590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.125107+0000","last_change":"2024-04-06T12:31:22.120490+0000","last_active":"2024-04-06T12:31:22.125107+0000","last_peered":"2024-04-06T12:31:22.125107+0000","last_clean":"2024-04-06T12:31:22.125107+0000","last_became_active":"2024-04-06T12:31:22.120251+0000","last_became_peered":"2024-04-06T12:31:22.120251+0000","last_unstale":"2024-04-06T12:31:22.125107+0000","last_undegraded":"2024-04-06T12:31:22.125107+0000","last_fullsized":"2024-04-06T12:31:22.125107+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:15:07.210571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099417+0000","last_change":"2024-04-06T12:31:22.109722+0000","last_active":"2024-04-06T12:31:23.099417+0000","last_peered":"2024-04-06T12:31:23.099417+0000","last_clean":"2024-04-06T12:31:23.099417+0000","last_became_active":"2024-04-06T12:31:22.108764+0000","last_became_peered":"2024-04-06T12:31:22.108764+0000","last_unstale":"2024-04-06T12:31:23.099417+0000","last_undegraded":"2024-04-06T12:31:23.099417+0000","last_fullsized":"2024-04-06T12:31:23.099417+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:11:35.249474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.110940+0000","last_change":"2024-04-06T12:31:22.108982+0000","last_active":"2024-04-06T12:31:22.110940+0000","last_peered":"2024-04-06T12:31:22.110940+0000","last_clean":"2024-04-06T12:31:22.110940+0000","last_became_active":"2024-04-06T12:31:22.108693+0000","last_became_peered":"2024-04-06T12:31:22.108693+0000","last_unstale":"2024-04-06T12:31:22.110940+0000","last_undegraded":"2024-04-06T12:31:22.110940+0000","last_fullsized":"2024-04-06T12:31:22.110940+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:40:35.155072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.113755+0000","last_change":"2024-04-06T12:31:22.112906+0000","last_active":"2024-04-06T12:31:22.113755+0000","last_peered":"2024-04-06T12:31:22.113755+0000","last_clean":"2024-04-06T12:31:22.113755+0000","last_became_active":"2024-04-06T12:31:22.112637+0000","last_became_peered":"2024-04-06T12:31:22.112637+0000","last_unstale":"2024-04-06T12:31:22.113755+0000","last_undegraded":"2024-04-06T12:31:22.113755+0000","last_fullsized":"2024-04-06T12:31:22.113755+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:50:56.202624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.112252+0000","last_change":"2024-04-06T12:31:22.110024+0000","last_active":"2024-04-06T12:31:22.112252+0000","last_peered":"2024-04-06T12:31:22.112252+0000","last_clean":"2024-04-06T12:31:22.112252+0000","last_became_active":"2024-04-06T12:31:22.109769+0000","last_became_peered":"2024-04-06T12:31:22.109769+0000","last_unstale":"2024-04-06T12:31:22.112252+0000","last_undegraded":"2024-04-06T12:31:22.112252+0000","last_fullsized":"2024-04-06T12:31:22.112252+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:08:57.991115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.122220+0000","last_change":"2024-04-06T12:31:22.119216+0000","last_active":"2024-04-06T12:31:22.122220+0000","last_peered":"2024-04-06T12:31:22.122220+0000","last_clean":"2024-04-06T12:31:22.122220+0000","last_became_active":"2024-04-06T12:31:22.118924+0000","last_became_peered":"2024-04-06T12:31:22.118924+0000","last_unstale":"2024-04-06T12:31:22.122220+0000","last_undegraded":"2024-04-06T12:31:22.122220+0000","last_fullsized":"2024-04-06T12:31:22.122220+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:00:53.973128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099679+0000","last_change":"2024-04-06T12:31:22.113133+0000","last_active":"2024-04-06T12:31:23.099679+0000","last_peered":"2024-04-06T12:31:23.099679+0000","last_clean":"2024-04-06T12:31:23.099679+0000","last_became_active":"2024-04-06T12:31:22.112779+0000","last_became_peered":"2024-04-06T12:31:22.112779+0000","last_unstale":"2024-04-06T12:31:23.099679+0000","last_undegraded":"2024-04-06T12:31:23.099679+0000","last_fullsized":"2024-04-06T12:31:23.099679+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:57:14.126836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.700178+0000","last_change":"2024-04-06T12:31:22.107148+0000","last_active":"2024-04-06T12:31:23.700178+0000","last_peered":"2024-04-06T12:31:23.700178+0000","last_clean":"2024-04-06T12:31:23.700178+0000","last_became_active":"2024-04-06T12:31:22.106890+0000","last_became_peered":"2024-04-06T12:31:22.106890+0000","last_unstale":"2024-04-06T12:31:23.700178+0000","last_undegraded":"2024-04-06T12:31:23.700178+0000","last_fullsized":"2024-04-06T12:31:23.700178+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:57:14.126836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.097497+0000","last_change":"2024-04-06T12:31:22.096782+0000","last_active":"2024-04-06T12:31:22.097497+0000","last_peered":"2024-04-06T12:31:22.097497+0000","last_clean":"2024-04-06T12:31:22.097497+0000","last_became_active":"2024-04-06T12:31:22.096496+0000","last_became_peered":"2024-04-06T12:31:22.096496+0000","last_unstale":"2024-04-06T12:31:22.097497+0000","last_undegraded":"2024-04-06T12:31:22.097497+0000","last_fullsized":"2024-04-06T12:31:22.097497+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:03:54.584558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.111230+0000","last_change":"2024-04-06T12:31:22.104964+0000","last_active":"2024-04-06T12:31:22.111230+0000","last_peered":"2024-04-06T12:31:22.111230+0000","last_clean":"2024-04-06T12:31:22.111230+0000","last_became_active":"2024-04-06T12:31:22.104553+0000","last_became_peered":"2024-04-06T12:31:22.104553+0000","last_unstale":"2024-04-06T12:31:22.111230+0000","last_undegraded":"2024-04-06T12:31:22.111230+0000","last_fullsized":"2024-04-06T12:31:22.111230+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:37:56.980318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.699934+0000","last_change":"2024-04-06T12:31:22.106527+0000","last_active":"2024-04-06T12:31:23.699934+0000","last_peered":"2024-04-06T12:31:23.699934+0000","last_clean":"2024-04-06T12:31:23.699934+0000","last_became_active":"2024-04-06T12:31:22.106265+0000","last_became_peered":"2024-04-06T12:31:22.106265+0000","last_unstale":"2024-04-06T12:31:23.699934+0000","last_undegraded":"2024-04-06T12:31:23.699934+0000","last_fullsized":"2024-04-06T12:31:23.699934+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:58:51.171247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119659+0000","last_change":"2024-04-06T12:31:22.117244+0000","last_active":"2024-04-06T12:31:22.119659+0000","last_peered":"2024-04-06T12:31:22.119659+0000","last_clean":"2024-04-06T12:31:22.119659+0000","last_became_active":"2024-04-06T12:31:22.117002+0000","last_became_peered":"2024-04-06T12:31:22.117002+0000","last_unstale":"2024-04-06T12:31:22.119659+0000","last_undegraded":"2024-04-06T12:31:22.119659+0000","last_fullsized":"2024-04-06T12:31:22.119659+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:22:31.614589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.098119+0000","last_change":"2024-04-06T12:31:22.097043+0000","last_active":"2024-04-06T12:31:22.098119+0000","last_peered":"2024-04-06T12:31:22.098119+0000","last_clean":"2024-04-06T12:31:22.098119+0000","last_became_active":"2024-04-06T12:31:22.096738+0000","last_became_peered":"2024-04-06T12:31:22.096738+0000","last_unstale":"2024-04-06T12:31:22.098119+0000","last_undegraded":"2024-04-06T12:31:22.098119+0000","last_fullsized":"2024-04-06T12:31:22.098119+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:11:39.129105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.106455+0000","last_change":"2024-04-06T12:31:22.102695+0000","last_active":"2024-04-06T12:31:22.106455+0000","last_peered":"2024-04-06T12:31:22.106455+0000","last_clean":"2024-04-06T12:31:22.106455+0000","last_became_active":"2024-04-06T12:31:22.102340+0000","last_became_peered":"2024-04-06T12:31:22.102340+0000","last_unstale":"2024-04-06T12:31:22.106455+0000","last_undegraded":"2024-04-06T12:31:22.106455+0000","last_fullsized":"2024-04-06T12:31:22.106455+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:17:56.649966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.700216+0000","last_change":"2024-04-06T12:31:22.105864+0000","last_active":"2024-04-06T12:31:23.700216+0000","last_peered":"2024-04-06T12:31:23.700216+0000","last_clean":"2024-04-06T12:31:23.700216+0000","last_became_active":"2024-04-06T12:31:22.105709+0000","last_became_peered":"2024-04-06T12:31:22.105709+0000","last_unstale":"2024-04-06T12:31:23.700216+0000","last_undegraded":"2024-04-06T12:31:23.700216+0000","last_fullsized":"2024-04-06T12:31:23.700216+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:40:35.155072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119212+0000","last_change":"2024-04-06T12:31:22.112614+0000","last_active":"2024-04-06T12:31:22.119212+0000","last_peered":"2024-04-06T12:31:22.119212+0000","last_clean":"2024-04-06T12:31:22.119212+0000","last_became_active":"2024-04-06T12:31:22.112196+0000","last_became_peered":"2024-04-06T12:31:22.112196+0000","last_unstale":"2024-04-06T12:31:22.119212+0000","last_undegraded":"2024-04-06T12:31:22.119212+0000","last_fullsized":"2024-04-06T12:31:22.119212+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:15:12.602022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097669+0000","last_change":"2024-04-06T12:31:22.112676+0000","last_active":"2024-04-06T12:31:23.097669+0000","last_peered":"2024-04-06T12:31:23.097669+0000","last_clean":"2024-04-06T12:31:23.097669+0000","last_became_active":"2024-04-06T12:31:22.112421+0000","last_became_peered":"2024-04-06T12:31:22.112421+0000","last_unstale":"2024-04-06T12:31:23.097669+0000","last_undegraded":"2024-04-06T12:31:23.097669+0000","last_fullsized":"2024-04-06T12:31:23.097669+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:47.974510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.107214+0000","last_change":"2024-04-06T12:31:22.102663+0000","last_active":"2024-04-06T12:31:22.107214+0000","last_peered":"2024-04-06T12:31:22.107214+0000","last_clean":"2024-04-06T12:31:22.107214+0000","last_became_active":"2024-04-06T12:31:22.102413+0000","last_became_peered":"2024-04-06T12:31:22.102413+0000","last_unstale":"2024-04-06T12:31:22.107214+0000","last_undegraded":"2024-04-06T12:31:22.107214+0000","last_fullsized":"2024-04-06T12:31:22.107214+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:16.549838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099319+0000","last_change":"2024-04-06T12:31:22.109091+0000","last_active":"2024-04-06T12:31:23.099319+0000","last_peered":"2024-04-06T12:31:23.099319+0000","last_clean":"2024-04-06T12:31:23.099319+0000","last_became_active":"2024-04-06T12:31:22.108814+0000","last_became_peered":"2024-04-06T12:31:22.108814+0000","last_unstale":"2024-04-06T12:31:23.099319+0000","last_undegraded":"2024-04-06T12:31:23.099319+0000","last_fullsized":"2024-04-06T12:31:23.099319+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:42.850998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.116897+0000","last_change":"2024-04-06T12:31:22.115284+0000","last_active":"2024-04-06T12:31:22.116897+0000","last_peered":"2024-04-06T12:31:22.116897+0000","last_clean":"2024-04-06T12:31:22.116897+0000","last_became_active":"2024-04-06T12:31:22.114791+0000","last_became_peered":"2024-04-06T12:31:22.114791+0000","last_unstale":"2024-04-06T12:31:22.116897+0000","last_undegraded":"2024-04-06T12:31:22.116897+0000","last_fullsized":"2024-04-06T12:31:22.116897+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:57:14.126836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099082+0000","last_change":"2024-04-06T12:31:22.116553+0000","last_active":"2024-04-06T12:31:23.099082+0000","last_peered":"2024-04-06T12:31:23.099082+0000","last_clean":"2024-04-06T12:31:23.099082+0000","last_became_active":"2024-04-06T12:31:22.116301+0000","last_became_peered":"2024-04-06T12:31:22.116301+0000","last_unstale":"2024-04-06T12:31:23.099082+0000","last_undegraded":"2024-04-06T12:31:23.099082+0000","last_fullsized":"2024-04-06T12:31:23.099082+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:40:35.155072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097153+0000","last_change":"2024-04-06T12:31:22.109723+0000","last_active":"2024-04-06T12:31:23.097153+0000","last_peered":"2024-04-06T12:31:23.097153+0000","last_clean":"2024-04-06T12:31:23.097153+0000","last_became_active":"2024-04-06T12:31:22.109420+0000","last_became_peered":"2024-04-06T12:31:22.109420+0000","last_unstale":"2024-04-06T12:31:23.097153+0000","last_undegraded":"2024-04-06T12:31:23.097153+0000","last_fullsized":"2024-04-06T12:31:23.097153+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:05:59.961267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.098108+0000","last_change":"2024-04-06T12:31:22.097346+0000","last_active":"2024-04-06T12:31:23.098108+0000","last_peered":"2024-04-06T12:31:23.098108+0000","last_clean":"2024-04-06T12:31:23.098108+0000","last_became_active":"2024-04-06T12:31:22.097002+0000","last_became_peered":"2024-04-06T12:31:22.097002+0000","last_unstale":"2024-04-06T12:31:23.098108+0000","last_undegraded":"2024-04-06T12:31:23.098108+0000","last_fullsized":"2024-04-06T12:31:23.098108+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:35:36.730191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.111670+0000","last_change":"2024-04-06T12:31:22.108009+0000","last_active":"2024-04-06T12:31:22.111670+0000","last_peered":"2024-04-06T12:31:22.111670+0000","last_clean":"2024-04-06T12:31:22.111670+0000","last_became_active":"2024-04-06T12:31:22.107749+0000","last_became_peered":"2024-04-06T12:31:22.107749+0000","last_unstale":"2024-04-06T12:31:22.111670+0000","last_undegraded":"2024-04-06T12:31:22.111670+0000","last_fullsized":"2024-04-06T12:31:22.111670+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:52.611199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097799+0000","last_change":"2024-04-06T12:31:22.112068+0000","last_active":"2024-04-06T12:31:23.097799+0000","last_peered":"2024-04-06T12:31:23.097799+0000","last_clean":"2024-04-06T12:31:23.097799+0000","last_became_active":"2024-04-06T12:31:22.111808+0000","last_became_peered":"2024-04-06T12:31:22.111808+0000","last_unstale":"2024-04-06T12:31:23.097799+0000","last_undegraded":"2024-04-06T12:31:23.097799+0000","last_fullsized":"2024-04-06T12:31:23.097799+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:52.934051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.102920+0000","last_change":"2024-04-06T12:31:22.100138+0000","last_active":"2024-04-06T12:31:22.102920+0000","last_peered":"2024-04-06T12:31:22.102920+0000","last_clean":"2024-04-06T12:31:22.102920+0000","last_became_active":"2024-04-06T12:31:22.099876+0000","last_became_peered":"2024-04-06T12:31:22.099876+0000","last_unstale":"2024-04-06T12:31:22.102920+0000","last_undegraded":"2024-04-06T12:31:22.102920+0000","last_fullsized":"2024-04-06T12:31:22.102920+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:23:01.349287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.115931+0000","last_change":"2024-04-06T12:31:22.115047+0000","last_active":"2024-04-06T12:31:22.115931+0000","last_peered":"2024-04-06T12:31:22.115931+0000","last_clean":"2024-04-06T12:31:22.115931+0000","last_became_active":"2024-04-06T12:31:22.114798+0000","last_became_peered":"2024-04-06T12:31:22.114798+0000","last_unstale":"2024-04-06T12:31:22.115931+0000","last_undegraded":"2024-04-06T12:31:22.115931+0000","last_fullsized":"2024-04-06T12:31:22.115931+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:40.865129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119314+0000","last_change":"2024-04-06T12:31:22.114823+0000","last_active":"2024-04-06T12:31:22.119314+0000","last_peered":"2024-04-06T12:31:22.119314+0000","last_clean":"2024-04-06T12:31:22.119314+0000","last_became_active":"2024-04-06T12:31:22.114199+0000","last_became_peered":"2024-04-06T12:31:22.114199+0000","last_unstale":"2024-04-06T12:31:22.119314+0000","last_undegraded":"2024-04-06T12:31:22.119314+0000","last_fullsized":"2024-04-06T12:31:22.119314+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:29:27.106970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097365+0000","last_change":"2024-04-06T12:31:22.099930+0000","last_active":"2024-04-06T12:31:23.097365+0000","last_peered":"2024-04-06T12:31:23.097365+0000","last_clean":"2024-04-06T12:31:23.097365+0000","last_became_active":"2024-04-06T12:31:22.099670+0000","last_became_peered":"2024-04-06T12:31:22.099670+0000","last_unstale":"2024-04-06T12:31:23.097365+0000","last_undegraded":"2024-04-06T12:31:23.097365+0000","last_fullsized":"2024-04-06T12:31:23.097365+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:47.974510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097871+0000","last_change":"2024-04-06T12:31:22.109490+0000","last_active":"2024-04-06T12:31:23.097871+0000","last_peered":"2024-04-06T12:31:23.097871+0000","last_clean":"2024-04-06T12:31:23.097871+0000","last_became_active":"2024-04-06T12:31:22.109213+0000","last_became_peered":"2024-04-06T12:31:22.109213+0000","last_unstale":"2024-04-06T12:31:23.097871+0000","last_undegraded":"2024-04-06T12:31:23.097871+0000","last_fullsized":"2024-04-06T12:31:23.097871+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:41.101353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.700144+0000","last_change":"2024-04-06T12:31:22.106880+0000","last_active":"2024-04-06T12:31:23.700144+0000","last_peered":"2024-04-06T12:31:23.700144+0000","last_clean":"2024-04-06T12:31:23.700144+0000","last_became_active":"2024-04-06T12:31:22.106647+0000","last_became_peered":"2024-04-06T12:31:22.106647+0000","last_unstale":"2024-04-06T12:31:23.700144+0000","last_undegraded":"2024-04-06T12:31:23.700144+0000","last_fullsized":"2024-04-06T12:31:23.700144+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:11:39.129105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"143'4","reported_seq":15,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097696+0000","last_change":"2024-04-06T12:31:21.686002+0000","last_active":"2024-04-06T12:31:23.097696+0000","last_peered":"2024-04-06T12:31:23.097696+0000","last_clean":"2024-04-06T12:31:23.097696+0000","last_became_active":"2024-04-06T12:31:21.685729+0000","last_became_peered":"2024-04-06T12:31:21.685729+0000","last_unstale":"2024-04-06T12:31:23.097696+0000","last_undegraded":"2024-04-06T12:31:23.097696+0000","last_fullsized":"2024-04-06T12:31:23.097696+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:43:58.040962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.113846+0000","last_change":"2024-04-06T12:31:22.113109+0000","last_active":"2024-04-06T12:31:22.113846+0000","last_peered":"2024-04-06T12:31:22.113846+0000","last_clean":"2024-04-06T12:31:22.113846+0000","last_became_active":"2024-04-06T12:31:22.112853+0000","last_became_peered":"2024-04-06T12:31:22.112853+0000","last_unstale":"2024-04-06T12:31:22.113846+0000","last_undegraded":"2024-04-06T12:31:22.113846+0000","last_fullsized":"2024-04-06T12:31:22.113846+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:30:20.270321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"143'1","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.700171+0000","last_change":"2024-04-06T12:31:21.684151+0000","last_active":"2024-04-06T12:31:23.700171+0000","last_peered":"2024-04-06T12:31:23.700171+0000","last_clean":"2024-04-06T12:31:23.700171+0000","last_became_active":"2024-04-06T12:31:21.683896+0000","last_became_peered":"2024-04-06T12:31:21.683896+0000","last_unstale":"2024-04-06T12:31:23.700171+0000","last_undegraded":"2024-04-06T12:31:23.700171+0000","last_fullsized":"2024-04-06T12:31:23.700171+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:05:58.955969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097806+0000","last_change":"2024-04-06T12:31:22.103398+0000","last_active":"2024-04-06T12:31:23.097806+0000","last_peered":"2024-04-06T12:31:23.097806+0000","last_clean":"2024-04-06T12:31:23.097806+0000","last_became_active":"2024-04-06T12:31:22.103141+0000","last_became_peered":"2024-04-06T12:31:22.103141+0000","last_unstale":"2024-04-06T12:31:23.097806+0000","last_undegraded":"2024-04-06T12:31:23.097806+0000","last_fullsized":"2024-04-06T12:31:23.097806+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:06:04.683437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"143'2","reported_seq":13,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099564+0000","last_change":"2024-04-06T12:31:21.689017+0000","last_active":"2024-04-06T12:31:23.099564+0000","last_peered":"2024-04-06T12:31:23.099564+0000","last_clean":"2024-04-06T12:31:23.099564+0000","last_became_active":"2024-04-06T12:31:21.688768+0000","last_became_peered":"2024-04-06T12:31:21.688768+0000","last_unstale":"2024-04-06T12:31:23.099564+0000","last_undegraded":"2024-04-06T12:31:23.099564+0000","last_fullsized":"2024-04-06T12:31:23.099564+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:46.969211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099707+0000","last_change":"2024-04-06T12:31:22.110126+0000","last_active":"2024-04-06T12:31:23.099707+0000","last_peered":"2024-04-06T12:31:23.099707+0000","last_clean":"2024-04-06T12:31:23.099707+0000","last_became_active":"2024-04-06T12:31:22.109147+0000","last_became_peered":"2024-04-06T12:31:22.109147+0000","last_unstale":"2024-04-06T12:31:23.099707+0000","last_undegraded":"2024-04-06T12:31:23.099707+0000","last_fullsized":"2024-04-06T12:31:23.099707+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:17.208811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"143'2","reported_seq":13,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.700133+0000","last_change":"2024-04-06T12:31:21.689407+0000","last_active":"2024-04-06T12:31:23.700133+0000","last_peered":"2024-04-06T12:31:23.700133+0000","last_clean":"2024-04-06T12:31:23.700133+0000","last_became_active":"2024-04-06T12:31:21.689126+0000","last_became_peered":"2024-04-06T12:31:21.689126+0000","last_unstale":"2024-04-06T12:31:23.700133+0000","last_undegraded":"2024-04-06T12:31:23.700133+0000","last_fullsized":"2024-04-06T12:31:23.700133+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:51.928752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.699934+0000","last_change":"2024-04-06T12:31:22.110058+0000","last_active":"2024-04-06T12:31:23.699934+0000","last_peered":"2024-04-06T12:31:23.699934+0000","last_clean":"2024-04-06T12:31:23.699934+0000","last_became_active":"2024-04-06T12:31:22.109137+0000","last_became_peered":"2024-04-06T12:31:22.109137+0000","last_unstale":"2024-04-06T12:31:23.699934+0000","last_undegraded":"2024-04-06T12:31:23.699934+0000","last_fullsized":"2024-04-06T12:31:23.699934+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:42.850998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"143'1","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.106972+0000","last_change":"2024-04-06T12:31:21.685817+0000","last_active":"2024-04-06T12:31:22.106972+0000","last_peered":"2024-04-06T12:31:22.106972+0000","last_clean":"2024-04-06T12:31:22.106972+0000","last_became_active":"2024-04-06T12:31:21.685360+0000","last_became_peered":"2024-04-06T12:31:21.685360+0000","last_unstale":"2024-04-06T12:31:22.106972+0000","last_undegraded":"2024-04-06T12:31:22.106972+0000","last_fullsized":"2024-04-06T12:31:22.106972+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:06:03.678138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.107601+0000","last_change":"2024-04-06T12:31:22.106184+0000","last_active":"2024-04-06T12:31:22.107601+0000","last_peered":"2024-04-06T12:31:22.107601+0000","last_clean":"2024-04-06T12:31:22.107601+0000","last_became_active":"2024-04-06T12:31:22.105905+0000","last_became_peered":"2024-04-06T12:31:22.105905+0000","last_unstale":"2024-04-06T12:31:22.107601+0000","last_undegraded":"2024-04-06T12:31:22.107601+0000","last_fullsized":"2024-04-06T12:31:22.107601+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:11:35.249474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"143'1","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.111756+0000","last_change":"2024-04-06T12:31:21.694224+0000","last_active":"2024-04-06T12:31:22.111756+0000","last_peered":"2024-04-06T12:31:22.111756+0000","last_clean":"2024-04-06T12:31:22.111756+0000","last_became_active":"2024-04-06T12:31:21.693927+0000","last_became_peered":"2024-04-06T12:31:21.693927+0000","last_unstale":"2024-04-06T12:31:22.111756+0000","last_undegraded":"2024-04-06T12:31:22.111756+0000","last_fullsized":"2024-04-06T12:31:22.111756+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:46:29.474190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119572+0000","last_change":"2024-04-06T12:31:22.116921+0000","last_active":"2024-04-06T12:31:22.119572+0000","last_peered":"2024-04-06T12:31:22.119572+0000","last_clean":"2024-04-06T12:31:22.119572+0000","last_became_active":"2024-04-06T12:31:22.114327+0000","last_became_peered":"2024-04-06T12:31:22.114327+0000","last_unstale":"2024-04-06T12:31:22.119572+0000","last_undegraded":"2024-04-06T12:31:22.119572+0000","last_fullsized":"2024-04-06T12:31:22.119572+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:36.711304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097459+0000","last_change":"2024-04-06T12:31:21.691092+0000","last_active":"2024-04-06T12:31:23.097459+0000","last_peered":"2024-04-06T12:31:23.097459+0000","last_clean":"2024-04-06T12:31:23.097459+0000","last_became_active":"2024-04-06T12:31:21.690783+0000","last_became_peered":"2024-04-06T12:31:21.690783+0000","last_unstale":"2024-04-06T12:31:23.097459+0000","last_undegraded":"2024-04-06T12:31:23.097459+0000","last_fullsized":"2024-04-06T12:31:23.097459+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:12:51.214120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.699934+0000","last_change":"2024-04-06T12:31:22.115026+0000","last_active":"2024-04-06T12:31:23.699934+0000","last_peered":"2024-04-06T12:31:23.699934+0000","last_clean":"2024-04-06T12:31:23.699934+0000","last_became_active":"2024-04-06T12:31:22.114784+0000","last_became_peered":"2024-04-06T12:31:22.114784+0000","last_unstale":"2024-04-06T12:31:23.699934+0000","last_undegraded":"2024-04-06T12:31:23.699934+0000","last_fullsized":"2024-04-06T12:31:23.699934+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:59:14.197440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.095728+0000","last_change":"2024-04-06T12:31:21.697322+0000","last_active":"2024-04-06T12:31:22.095728+0000","last_peered":"2024-04-06T12:31:22.095728+0000","last_clean":"2024-04-06T12:31:22.095728+0000","last_became_active":"2024-04-06T12:31:21.696959+0000","last_became_peered":"2024-04-06T12:31:21.696959+0000","last_unstale":"2024-04-06T12:31:22.095728+0000","last_undegraded":"2024-04-06T12:31:22.095728+0000","last_fullsized":"2024-04-06T12:31:22.095728+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:08:37.545709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097268+0000","last_change":"2024-04-06T12:31:22.113648+0000","last_active":"2024-04-06T12:31:23.097268+0000","last_peered":"2024-04-06T12:31:23.097268+0000","last_clean":"2024-04-06T12:31:23.097268+0000","last_became_active":"2024-04-06T12:31:22.113243+0000","last_became_peered":"2024-04-06T12:31:22.113243+0000","last_unstale":"2024-04-06T12:31:23.097268+0000","last_undegraded":"2024-04-06T12:31:23.097268+0000","last_fullsized":"2024-04-06T12:31:23.097268+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:05:59.961267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097839+0000","last_change":"2024-04-06T12:31:21.696014+0000","last_active":"2024-04-06T12:31:23.097839+0000","last_peered":"2024-04-06T12:31:23.097839+0000","last_clean":"2024-04-06T12:31:23.097839+0000","last_became_active":"2024-04-06T12:31:21.695744+0000","last_became_peered":"2024-04-06T12:31:21.695744+0000","last_unstale":"2024-04-06T12:31:23.097839+0000","last_undegraded":"2024-04-06T12:31:23.097839+0000","last_fullsized":"2024-04-06T12:31:23.097839+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:51.605900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097617+0000","last_change":"2024-04-06T12:31:22.115749+0000","last_active":"2024-04-06T12:31:23.097617+0000","last_peered":"2024-04-06T12:31:23.097617+0000","last_clean":"2024-04-06T12:31:23.097617+0000","last_became_active":"2024-04-06T12:31:22.115472+0000","last_became_peered":"2024-04-06T12:31:22.115472+0000","last_unstale":"2024-04-06T12:31:23.097617+0000","last_undegraded":"2024-04-06T12:31:23.097617+0000","last_fullsized":"2024-04-06T12:31:23.097617+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:09:03.752277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097893+0000","last_change":"2024-04-06T12:31:21.691574+0000","last_active":"2024-04-06T12:31:23.097893+0000","last_peered":"2024-04-06T12:31:23.097893+0000","last_clean":"2024-04-06T12:31:23.097893+0000","last_became_active":"2024-04-06T12:31:21.691280+0000","last_became_peered":"2024-04-06T12:31:21.691280+0000","last_unstale":"2024-04-06T12:31:23.097893+0000","last_undegraded":"2024-04-06T12:31:23.097893+0000","last_fullsized":"2024-04-06T12:31:23.097893+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:18:36.586968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119193+0000","last_change":"2024-04-06T12:31:22.112466+0000","last_active":"2024-04-06T12:31:22.119193+0000","last_peered":"2024-04-06T12:31:22.119193+0000","last_clean":"2024-04-06T12:31:22.119193+0000","last_became_active":"2024-04-06T12:31:22.112168+0000","last_became_peered":"2024-04-06T12:31:22.112168+0000","last_unstale":"2024-04-06T12:31:22.119193+0000","last_undegraded":"2024-04-06T12:31:22.119193+0000","last_fullsized":"2024-04-06T12:31:22.119193+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:27.379210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"143'3","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097719+0000","last_change":"2024-04-06T12:31:21.690621+0000","last_active":"2024-04-06T12:31:23.097719+0000","last_peered":"2024-04-06T12:31:23.097719+0000","last_clean":"2024-04-06T12:31:23.097719+0000","last_became_active":"2024-04-06T12:31:21.690327+0000","last_became_peered":"2024-04-06T12:31:21.690327+0000","last_unstale":"2024-04-06T12:31:23.097719+0000","last_undegraded":"2024-04-06T12:31:23.097719+0000","last_fullsized":"2024-04-06T12:31:23.097719+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:53:06.620534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.098117+0000","last_change":"2024-04-06T12:31:22.110418+0000","last_active":"2024-04-06T12:31:23.098117+0000","last_peered":"2024-04-06T12:31:23.098117+0000","last_clean":"2024-04-06T12:31:23.098117+0000","last_became_active":"2024-04-06T12:31:22.110133+0000","last_became_peered":"2024-04-06T12:31:22.110133+0000","last_unstale":"2024-04-06T12:31:23.098117+0000","last_undegraded":"2024-04-06T12:31:23.098117+0000","last_fullsized":"2024-04-06T12:31:23.098117+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:02:35.211216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"143'6","reported_seq":16,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.127772+0000","last_change":"2024-04-06T12:31:21.690830+0000","last_active":"2024-04-06T12:31:22.127772+0000","last_peered":"2024-04-06T12:31:22.127772+0000","last_clean":"2024-04-06T12:31:22.127772+0000","last_became_active":"2024-04-06T12:31:21.690252+0000","last_became_peered":"2024-04-06T12:31:21.690252+0000","last_unstale":"2024-04-06T12:31:22.127772+0000","last_undegraded":"2024-04-06T12:31:22.127772+0000","last_fullsized":"2024-04-06T12:31:22.127772+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:45:34.281066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097382+0000","last_change":"2024-04-06T12:31:22.119711+0000","last_active":"2024-04-06T12:31:23.097382+0000","last_peered":"2024-04-06T12:31:23.097382+0000","last_clean":"2024-04-06T12:31:23.097382+0000","last_became_active":"2024-04-06T12:31:22.119435+0000","last_became_peered":"2024-04-06T12:31:22.119435+0000","last_unstale":"2024-04-06T12:31:23.097382+0000","last_undegraded":"2024-04-06T12:31:23.097382+0000","last_fullsized":"2024-04-06T12:31:23.097382+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:02:35.211216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"143'2","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.118565+0000","last_change":"2024-04-06T12:31:21.689856+0000","last_active":"2024-04-06T12:31:22.118565+0000","last_peered":"2024-04-06T12:31:22.118565+0000","last_clean":"2024-04-06T12:31:22.118565+0000","last_became_active":"2024-04-06T12:31:21.689628+0000","last_became_peered":"2024-04-06T12:31:21.689628+0000","last_unstale":"2024-04-06T12:31:22.118565+0000","last_undegraded":"2024-04-06T12:31:22.118565+0000","last_fullsized":"2024-04-06T12:31:22.118565+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:16.203513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.700028+0000","last_change":"2024-04-06T12:31:22.109860+0000","last_active":"2024-04-06T12:31:23.700028+0000","last_peered":"2024-04-06T12:31:23.700028+0000","last_clean":"2024-04-06T12:31:23.700028+0000","last_became_active":"2024-04-06T12:31:22.109641+0000","last_became_peered":"2024-04-06T12:31:22.109641+0000","last_unstale":"2024-04-06T12:31:23.700028+0000","last_undegraded":"2024-04-06T12:31:23.700028+0000","last_fullsized":"2024-04-06T12:31:23.700028+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:09:03.294158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.103362+0000","last_change":"2024-04-06T12:31:21.691247+0000","last_active":"2024-04-06T12:31:22.103362+0000","last_peered":"2024-04-06T12:31:22.103362+0000","last_clean":"2024-04-06T12:31:22.103362+0000","last_became_active":"2024-04-06T12:31:21.690386+0000","last_became_peered":"2024-04-06T12:31:21.690386+0000","last_unstale":"2024-04-06T12:31:22.103362+0000","last_undegraded":"2024-04-06T12:31:22.103362+0000","last_fullsized":"2024-04-06T12:31:22.103362+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:03:53.579260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.100240+0000","last_change":"2024-04-06T12:31:22.097282+0000","last_active":"2024-04-06T12:31:22.100240+0000","last_peered":"2024-04-06T12:31:22.100240+0000","last_clean":"2024-04-06T12:31:22.100240+0000","last_became_active":"2024-04-06T12:31:22.097047+0000","last_became_peered":"2024-04-06T12:31:22.097047+0000","last_unstale":"2024-04-06T12:31:22.100240+0000","last_undegraded":"2024-04-06T12:31:22.100240+0000","last_fullsized":"2024-04-06T12:31:22.100240+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:19:42.830042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.097898+0000","last_change":"2024-04-06T12:31:21.690399+0000","last_active":"2024-04-06T12:31:22.097898+0000","last_peered":"2024-04-06T12:31:22.097898+0000","last_clean":"2024-04-06T12:31:22.097898+0000","last_became_active":"2024-04-06T12:31:21.689993+0000","last_became_peered":"2024-04-06T12:31:21.689993+0000","last_unstale":"2024-04-06T12:31:22.097898+0000","last_undegraded":"2024-04-06T12:31:22.097898+0000","last_fullsized":"2024-04-06T12:31:22.097898+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:34:09.064491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097668+0000","last_change":"2024-04-06T12:31:22.104431+0000","last_active":"2024-04-06T12:31:23.097668+0000","last_peered":"2024-04-06T12:31:23.097668+0000","last_clean":"2024-04-06T12:31:23.097668+0000","last_became_active":"2024-04-06T12:31:22.104082+0000","last_became_peered":"2024-04-06T12:31:22.104082+0000","last_unstale":"2024-04-06T12:31:23.097668+0000","last_undegraded":"2024-04-06T12:31:23.097668+0000","last_fullsized":"2024-04-06T12:31:23.097668+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:17:13.685324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"143'1","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.700267+0000","last_change":"2024-04-06T12:31:21.691715+0000","last_active":"2024-04-06T12:31:23.700267+0000","last_peered":"2024-04-06T12:31:23.700267+0000","last_clean":"2024-04-06T12:31:23.700267+0000","last_became_active":"2024-04-06T12:31:21.691439+0000","last_became_peered":"2024-04-06T12:31:21.691439+0000","last_unstale":"2024-04-06T12:31:23.700267+0000","last_undegraded":"2024-04-06T12:31:23.700267+0000","last_fullsized":"2024-04-06T12:31:23.700267+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:46.969211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.116882+0000","last_change":"2024-04-06T12:31:22.115140+0000","last_active":"2024-04-06T12:31:22.116882+0000","last_peered":"2024-04-06T12:31:22.116882+0000","last_clean":"2024-04-06T12:31:22.116882+0000","last_became_active":"2024-04-06T12:31:22.114819+0000","last_became_peered":"2024-04-06T12:31:22.114819+0000","last_unstale":"2024-04-06T12:31:22.116882+0000","last_undegraded":"2024-04-06T12:31:22.116882+0000","last_fullsized":"2024-04-06T12:31:22.116882+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:17.208811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"143'2","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.102282+0000","last_change":"2024-04-06T12:31:21.694984+0000","last_active":"2024-04-06T12:31:22.102282+0000","last_peered":"2024-04-06T12:31:22.102282+0000","last_clean":"2024-04-06T12:31:22.102282+0000","last_became_active":"2024-04-06T12:31:21.694716+0000","last_became_peered":"2024-04-06T12:31:21.694716+0000","last_unstale":"2024-04-06T12:31:22.102282+0000","last_undegraded":"2024-04-06T12:31:22.102282+0000","last_fullsized":"2024-04-06T12:31:22.102282+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:18:36.586968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.111641+0000","last_change":"2024-04-06T12:31:22.108873+0000","last_active":"2024-04-06T12:31:22.111641+0000","last_peered":"2024-04-06T12:31:22.111641+0000","last_clean":"2024-04-06T12:31:22.111641+0000","last_became_active":"2024-04-06T12:31:22.108637+0000","last_became_peered":"2024-04-06T12:31:22.108637+0000","last_unstale":"2024-04-06T12:31:22.111641+0000","last_undegraded":"2024-04-06T12:31:22.111641+0000","last_fullsized":"2024-04-06T12:31:22.111641+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:51:23.611053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"143'1","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099644+0000","last_change":"2024-04-06T12:31:21.692516+0000","last_active":"2024-04-06T12:31:23.099644+0000","last_peered":"2024-04-06T12:31:23.099644+0000","last_clean":"2024-04-06T12:31:23.099644+0000","last_became_active":"2024-04-06T12:31:21.692077+0000","last_became_peered":"2024-04-06T12:31:21.692077+0000","last_unstale":"2024-04-06T12:31:23.099644+0000","last_undegraded":"2024-04-06T12:31:23.099644+0000","last_fullsized":"2024-04-06T12:31:23.099644+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:02:34.205917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.098187+0000","last_change":"2024-04-06T12:31:22.103426+0000","last_active":"2024-04-06T12:31:23.098187+0000","last_peered":"2024-04-06T12:31:23.098187+0000","last_clean":"2024-04-06T12:31:23.098187+0000","last_became_active":"2024-04-06T12:31:22.103145+0000","last_became_peered":"2024-04-06T12:31:22.103145+0000","last_unstale":"2024-04-06T12:31:23.098187+0000","last_undegraded":"2024-04-06T12:31:23.098187+0000","last_fullsized":"2024-04-06T12:31:23.098187+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:21:27.019989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099516+0000","last_change":"2024-04-06T12:31:21.697577+0000","last_active":"2024-04-06T12:31:23.099516+0000","last_peered":"2024-04-06T12:31:23.099516+0000","last_clean":"2024-04-06T12:31:23.099516+0000","last_became_active":"2024-04-06T12:31:21.697245+0000","last_became_peered":"2024-04-06T12:31:21.697245+0000","last_unstale":"2024-04-06T12:31:23.099516+0000","last_undegraded":"2024-04-06T12:31:23.099516+0000","last_fullsized":"2024-04-06T12:31:23.099516+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:05:58.955969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.111672+0000","last_change":"2024-04-06T12:31:22.108729+0000","last_active":"2024-04-06T12:31:22.111672+0000","last_peered":"2024-04-06T12:31:22.111672+0000","last_clean":"2024-04-06T12:31:22.111672+0000","last_became_active":"2024-04-06T12:31:22.108453+0000","last_became_peered":"2024-04-06T12:31:22.108453+0000","last_unstale":"2024-04-06T12:31:22.111672+0000","last_undegraded":"2024-04-06T12:31:22.111672+0000","last_fullsized":"2024-04-06T12:31:22.111672+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:37:19.783545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"143'2","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.113099+0000","last_change":"2024-04-06T12:31:21.694840+0000","last_active":"2024-04-06T12:31:22.113099+0000","last_peered":"2024-04-06T12:31:22.113099+0000","last_clean":"2024-04-06T12:31:22.113099+0000","last_became_active":"2024-04-06T12:31:21.694553+0000","last_became_peered":"2024-04-06T12:31:21.694553+0000","last_unstale":"2024-04-06T12:31:22.113099+0000","last_undegraded":"2024-04-06T12:31:22.113099+0000","last_fullsized":"2024-04-06T12:31:22.113099+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:14:26.385594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119150+0000","last_change":"2024-04-06T12:31:22.112779+0000","last_active":"2024-04-06T12:31:22.119150+0000","last_peered":"2024-04-06T12:31:22.119150+0000","last_clean":"2024-04-06T12:31:22.119150+0000","last_became_active":"2024-04-06T12:31:22.112256+0000","last_became_peered":"2024-04-06T12:31:22.112256+0000","last_unstale":"2024-04-06T12:31:22.119150+0000","last_undegraded":"2024-04-06T12:31:22.119150+0000","last_fullsized":"2024-04-06T12:31:22.119150+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:51:52.795093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099282+0000","last_change":"2024-04-06T12:31:21.685503+0000","last_active":"2024-04-06T12:31:23.099282+0000","last_peered":"2024-04-06T12:31:23.099282+0000","last_clean":"2024-04-06T12:31:23.099282+0000","last_became_active":"2024-04-06T12:31:21.684600+0000","last_became_peered":"2024-04-06T12:31:21.684600+0000","last_unstale":"2024-04-06T12:31:23.099282+0000","last_undegraded":"2024-04-06T12:31:23.099282+0000","last_fullsized":"2024-04-06T12:31:23.099282+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:09:02.746979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119627+0000","last_change":"2024-04-06T12:31:22.117136+0000","last_active":"2024-04-06T12:31:22.119627+0000","last_peered":"2024-04-06T12:31:22.119627+0000","last_clean":"2024-04-06T12:31:22.119627+0000","last_became_active":"2024-04-06T12:31:22.115170+0000","last_became_peered":"2024-04-06T12:31:22.115170+0000","last_unstale":"2024-04-06T12:31:22.119627+0000","last_undegraded":"2024-04-06T12:31:22.119627+0000","last_fullsized":"2024-04-06T12:31:22.119627+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:14:56.389425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.097919+0000","last_change":"2024-04-06T12:31:21.692888+0000","last_active":"2024-04-06T12:31:22.097919+0000","last_peered":"2024-04-06T12:31:22.097919+0000","last_clean":"2024-04-06T12:31:22.097919+0000","last_became_active":"2024-04-06T12:31:21.692608+0000","last_became_peered":"2024-04-06T12:31:21.692608+0000","last_unstale":"2024-04-06T12:31:22.097919+0000","last_undegraded":"2024-04-06T12:31:22.097919+0000","last_fullsized":"2024-04-06T12:31:22.097919+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:11:38.123806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097702+0000","last_change":"2024-04-06T12:31:22.096039+0000","last_active":"2024-04-06T12:31:23.097702+0000","last_peered":"2024-04-06T12:31:23.097702+0000","last_clean":"2024-04-06T12:31:23.097702+0000","last_became_active":"2024-04-06T12:31:22.095741+0000","last_became_peered":"2024-04-06T12:31:22.095741+0000","last_unstale":"2024-04-06T12:31:23.097702+0000","last_undegraded":"2024-04-06T12:31:23.097702+0000","last_fullsized":"2024-04-06T12:31:23.097702+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:52.934051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.098885+0000","last_change":"2024-04-06T12:31:21.694530+0000","last_active":"2024-04-06T12:31:23.098885+0000","last_peered":"2024-04-06T12:31:23.098885+0000","last_clean":"2024-04-06T12:31:23.098885+0000","last_became_active":"2024-04-06T12:31:21.694267+0000","last_became_peered":"2024-04-06T12:31:21.694267+0000","last_unstale":"2024-04-06T12:31:23.098885+0000","last_undegraded":"2024-04-06T12:31:23.098885+0000","last_fullsized":"2024-04-06T12:31:23.098885+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:51.928752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.111706+0000","last_change":"2024-04-06T12:31:22.108820+0000","last_active":"2024-04-06T12:31:22.111706+0000","last_peered":"2024-04-06T12:31:22.111706+0000","last_clean":"2024-04-06T12:31:22.111706+0000","last_became_active":"2024-04-06T12:31:22.108557+0000","last_became_peered":"2024-04-06T12:31:22.108557+0000","last_unstale":"2024-04-06T12:31:22.111706+0000","last_undegraded":"2024-04-06T12:31:22.111706+0000","last_fullsized":"2024-04-06T12:31:22.111706+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:08:21.428907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"143'2","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.103787+0000","last_change":"2024-04-06T12:31:21.694543+0000","last_active":"2024-04-06T12:31:22.103787+0000","last_peered":"2024-04-06T12:31:22.103787+0000","last_clean":"2024-04-06T12:31:22.103787+0000","last_became_active":"2024-04-06T12:31:21.692046+0000","last_became_peered":"2024-04-06T12:31:21.692046+0000","last_unstale":"2024-04-06T12:31:22.103787+0000","last_undegraded":"2024-04-06T12:31:22.103787+0000","last_fullsized":"2024-04-06T12:31:22.103787+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:41:18.763861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.700074+0000","last_change":"2024-04-06T12:31:22.108770+0000","last_active":"2024-04-06T12:31:23.700074+0000","last_peered":"2024-04-06T12:31:23.700074+0000","last_clean":"2024-04-06T12:31:23.700074+0000","last_became_active":"2024-04-06T12:31:22.108521+0000","last_became_peered":"2024-04-06T12:31:22.108521+0000","last_unstale":"2024-04-06T12:31:23.700074+0000","last_undegraded":"2024-04-06T12:31:23.700074+0000","last_fullsized":"2024-04-06T12:31:23.700074+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:17.208811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097731+0000","last_change":"2024-04-06T12:31:21.697407+0000","last_active":"2024-04-06T12:31:23.097731+0000","last_peered":"2024-04-06T12:31:23.097731+0000","last_clean":"2024-04-06T12:31:23.097731+0000","last_became_active":"2024-04-06T12:31:21.697125+0000","last_became_peered":"2024-04-06T12:31:21.697125+0000","last_unstale":"2024-04-06T12:31:23.097731+0000","last_undegraded":"2024-04-06T12:31:23.097731+0000","last_fullsized":"2024-04-06T12:31:23.097731+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:19:41.824744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099667+0000","last_change":"2024-04-06T12:31:22.108787+0000","last_active":"2024-04-06T12:31:23.099667+0000","last_peered":"2024-04-06T12:31:23.099667+0000","last_clean":"2024-04-06T12:31:23.099667+0000","last_became_active":"2024-04-06T12:31:22.108532+0000","last_became_peered":"2024-04-06T12:31:22.108532+0000","last_unstale":"2024-04-06T12:31:23.099667+0000","last_undegraded":"2024-04-06T12:31:23.099667+0000","last_fullsized":"2024-04-06T12:31:23.099667+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:38:56.895638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097911+0000","last_change":"2024-04-06T12:31:21.694921+0000","last_active":"2024-04-06T12:31:23.097911+0000","last_peered":"2024-04-06T12:31:23.097911+0000","last_clean":"2024-04-06T12:31:23.097911+0000","last_became_active":"2024-04-06T12:31:21.694615+0000","last_became_peered":"2024-04-06T12:31:21.694615+0000","last_unstale":"2024-04-06T12:31:23.097911+0000","last_undegraded":"2024-04-06T12:31:23.097911+0000","last_fullsized":"2024-04-06T12:31:23.097911+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:02:24.639229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.099925+0000","last_change":"2024-04-06T12:31:22.096553+0000","last_active":"2024-04-06T12:31:22.099925+0000","last_peered":"2024-04-06T12:31:22.099925+0000","last_clean":"2024-04-06T12:31:22.099925+0000","last_became_active":"2024-04-06T12:31:22.095236+0000","last_became_peered":"2024-04-06T12:31:22.095236+0000","last_unstale":"2024-04-06T12:31:22.099925+0000","last_undegraded":"2024-04-06T12:31:22.099925+0000","last_fullsized":"2024-04-06T12:31:22.099925+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:14:19.606564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.096910+0000","last_change":"2024-04-06T12:31:21.697287+0000","last_active":"2024-04-06T12:31:22.096910+0000","last_peered":"2024-04-06T12:31:22.096910+0000","last_clean":"2024-04-06T12:31:22.096910+0000","last_became_active":"2024-04-06T12:31:21.696979+0000","last_became_peered":"2024-04-06T12:31:21.696979+0000","last_unstale":"2024-04-06T12:31:22.096910+0000","last_undegraded":"2024-04-06T12:31:22.096910+0000","last_fullsized":"2024-04-06T12:31:22.096910+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:21:09.615951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.102779+0000","last_change":"2024-04-06T12:31:22.100612+0000","last_active":"2024-04-06T12:31:22.102779+0000","last_peered":"2024-04-06T12:31:22.102779+0000","last_clean":"2024-04-06T12:31:22.102779+0000","last_became_active":"2024-04-06T12:31:22.100355+0000","last_became_peered":"2024-04-06T12:31:22.100355+0000","last_unstale":"2024-04-06T12:31:22.102779+0000","last_undegraded":"2024-04-06T12:31:22.102779+0000","last_fullsized":"2024-04-06T12:31:22.102779+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:55:07.104225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.088805+0000","last_change":"2024-04-06T12:31:21.681396+0000","last_active":"2024-04-06T12:31:22.088805+0000","last_peered":"2024-04-06T12:31:22.088805+0000","last_clean":"2024-04-06T12:31:22.088805+0000","last_became_active":"2024-04-06T12:31:21.681133+0000","last_became_peered":"2024-04-06T12:31:21.681133+0000","last_unstale":"2024-04-06T12:31:22.088805+0000","last_undegraded":"2024-04-06T12:31:22.088805+0000","last_fullsized":"2024-04-06T12:31:22.088805+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:17:12.680025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119170+0000","last_change":"2024-04-06T12:31:22.114235+0000","last_active":"2024-04-06T12:31:22.119170+0000","last_peered":"2024-04-06T12:31:22.119170+0000","last_clean":"2024-04-06T12:31:22.119170+0000","last_became_active":"2024-04-06T12:31:22.112517+0000","last_became_peered":"2024-04-06T12:31:22.112517+0000","last_unstale":"2024-04-06T12:31:22.119170+0000","last_undegraded":"2024-04-06T12:31:22.119170+0000","last_fullsized":"2024-04-06T12:31:22.119170+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:01:34.580024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.088573+0000","last_change":"2024-04-06T12:31:21.689691+0000","last_active":"2024-04-06T12:31:22.088573+0000","last_peered":"2024-04-06T12:31:22.088573+0000","last_clean":"2024-04-06T12:31:22.088573+0000","last_became_active":"2024-04-06T12:31:21.689428+0000","last_became_peered":"2024-04-06T12:31:21.689428+0000","last_unstale":"2024-04-06T12:31:22.088573+0000","last_undegraded":"2024-04-06T12:31:22.088573+0000","last_fullsized":"2024-04-06T12:31:22.088573+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:09:02.746979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099162+0000","last_change":"2024-04-06T12:31:22.108269+0000","last_active":"2024-04-06T12:31:23.099162+0000","last_peered":"2024-04-06T12:31:23.099162+0000","last_clean":"2024-04-06T12:31:23.099162+0000","last_became_active":"2024-04-06T12:31:22.108017+0000","last_became_peered":"2024-04-06T12:31:22.108017+0000","last_unstale":"2024-04-06T12:31:23.099162+0000","last_undegraded":"2024-04-06T12:31:23.099162+0000","last_fullsized":"2024-04-06T12:31:23.099162+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:11:39.129105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.095731+0000","last_change":"2024-04-06T12:31:21.691758+0000","last_active":"2024-04-06T12:31:22.095731+0000","last_peered":"2024-04-06T12:31:22.095731+0000","last_clean":"2024-04-06T12:31:22.095731+0000","last_became_active":"2024-04-06T12:31:21.691462+0000","last_became_peered":"2024-04-06T12:31:21.691462+0000","last_unstale":"2024-04-06T12:31:22.095731+0000","last_undegraded":"2024-04-06T12:31:22.095731+0000","last_fullsized":"2024-04-06T12:31:22.095731+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:50:17.677522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.700036+0000","last_change":"2024-04-06T12:31:22.110654+0000","last_active":"2024-04-06T12:31:23.700036+0000","last_peered":"2024-04-06T12:31:23.700036+0000","last_clean":"2024-04-06T12:31:23.700036+0000","last_became_active":"2024-04-06T12:31:22.110189+0000","last_became_peered":"2024-04-06T12:31:22.110189+0000","last_unstale":"2024-04-06T12:31:23.700036+0000","last_undegraded":"2024-04-06T12:31:23.700036+0000","last_fullsized":"2024-04-06T12:31:23.700036+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:38:56.895638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.098919+0000","last_change":"2024-04-06T12:31:21.693170+0000","last_active":"2024-04-06T12:31:23.098919+0000","last_peered":"2024-04-06T12:31:23.098919+0000","last_clean":"2024-04-06T12:31:23.098919+0000","last_became_active":"2024-04-06T12:31:21.692910+0000","last_became_peered":"2024-04-06T12:31:21.692910+0000","last_unstale":"2024-04-06T12:31:23.098919+0000","last_undegraded":"2024-04-06T12:31:23.098919+0000","last_fullsized":"2024-04-06T12:31:23.098919+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:40.096055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099370+0000","last_change":"2024-04-06T12:31:22.112180+0000","last_active":"2024-04-06T12:31:23.099370+0000","last_peered":"2024-04-06T12:31:23.099370+0000","last_clean":"2024-04-06T12:31:23.099370+0000","last_became_active":"2024-04-06T12:31:22.111672+0000","last_became_peered":"2024-04-06T12:31:22.111672+0000","last_unstale":"2024-04-06T12:31:23.099370+0000","last_undegraded":"2024-04-06T12:31:23.099370+0000","last_fullsized":"2024-04-06T12:31:23.099370+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:58:51.171247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"143'8","reported_seq":18,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119958+0000","last_change":"2024-04-06T12:31:21.689849+0000","last_active":"2024-04-06T12:31:22.119958+0000","last_peered":"2024-04-06T12:31:22.119958+0000","last_clean":"2024-04-06T12:31:22.119958+0000","last_became_active":"2024-04-06T12:31:21.689589+0000","last_became_peered":"2024-04-06T12:31:21.689589+0000","last_unstale":"2024-04-06T12:31:22.119958+0000","last_undegraded":"2024-04-06T12:31:22.119958+0000","last_fullsized":"2024-04-06T12:31:22.119958+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:57:13.121538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"132'227","reported_seq":687,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.094977+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:31:22.094977+0000","last_peered":"2024-04-06T12:31:22.094977+0000","last_clean":"2024-04-06T12:31:22.094977+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:31:22.094977+0000","last_undegraded":"2024-04-06T12:31:22.094977+0000","last_fullsized":"2024-04-06T12:31:22.094977+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575078,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59700,"kb_used_data":2344,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312140,"statfs":{"total":96636764160,"available":96575631360,"internally_reserved":0,"allocated":2400256,"data_stored":1489714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53100000000000003,"15min":0.55300000000000005},"min":{"1min":0.33800000000000002,"5min":0.311,"15min":0.219},"max":{"1min":0.73099999999999998,"5min":0.80100000000000005,"15min":1.143},"last":0.249},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54000000000000004,"15min":0.55100000000000005},"min":{"1min":0.36399999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.72099999999999997,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.34100000000000003}]},{"osd":1,"last update":"Sat Apr 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54200000000000004,"15min":0.55500000000000005},"min":{"1min":0.39800000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71299999999999997,"5min":0.75,"15min":0.79100000000000004},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.55100000000000005,"15min":0.56799999999999995},"min":{"1min":0.433,"5min":0.36599999999999999,"15min":0.27900000000000003},"max":{"1min":0.754,"5min":0.77400000000000002,"15min":0.79600000000000004},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52700000000000002,"15min":0.55200000000000005},"min":{"1min":0.377,"5min":0.316,"15min":0.249},"max":{"1min":0.71199999999999997,"5min":0.77100000000000002,"15min":0.83299999999999996},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56699999999999995,"15min":0.57799999999999996},"min":{"1min":0.38800000000000001,"5min":0.36699999999999999,"15min":0.29599999999999999},"max":{"1min":0.78200000000000003,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.315}]},{"osd":3,"last update":"Sat Apr 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55300000000000005,"15min":0.57199999999999995},"min":{"1min":0.36299999999999999,"5min":0.35099999999999998,"15min":0.34399999999999997},"max":{"1min":0.68200000000000005,"5min":0.75800000000000001,"15min":1.357},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57499999999999996,"15min":0.58799999999999997},"min":{"1min":0.377,"5min":0.35599999999999998,"15min":0.33100000000000002},"max":{"1min":0.75800000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.39100000000000001}]},{"osd":4,"last update":"Sat Apr 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.438,"15min":0.443},"min":{"1min":0.29299999999999998,"5min":0.27000000000000002,"15min":0.25},"max":{"1min":0.59099999999999997,"5min":0.71399999999999997,"15min":0.72999999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.53000000000000003,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.35399999999999998,"15min":0.32800000000000001},"max":{"1min":0.62,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.378}]},{"osd":5,"last update":"Sat Apr 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.502,"15min":0.495},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.23799999999999999},"max":{"1min":0.73499999999999999,"5min":0.81899999999999995,"15min":0.86099999999999999},"last":0.432},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.33600000000000002,"5min":0.32800000000000001,"15min":0.314},"max":{"1min":0.73599999999999999,"5min":0.79900000000000004,"15min":0.81599999999999995},"last":0.42199999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.28799999999999998},"max":{"1min":0.73199999999999998,"5min":0.84499999999999997,"15min":0.88600000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.38500000000000001,"5min":0.312,"15min":0.30399999999999999},"max":{"1min":0.70999999999999996,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.44600000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575077,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32868,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338972,"statfs":{"total":96636764160,"available":96603107328,"internally_reserved":0,"allocated":1789952,"data_stored":898133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.52900000000000003,"15min":0.53900000000000003},"min":{"1min":0.37,"5min":0.34000000000000002,"15min":0.26200000000000001},"max":{"1min":0.628,"5min":0.77500000000000002,"15min":0.82099999999999995},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53200000000000003,"15min":0.53600000000000003},"min":{"1min":0.38700000000000001,"5min":0.35299999999999998,"15min":0.245},"max":{"1min":0.63100000000000001,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.70799999999999996}]},{"osd":1,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57099999999999995,"15min":0.58599999999999997},"min":{"1min":0.433,"5min":0.32700000000000001,"15min":0.24399999999999999},"max":{"1min":0.67900000000000005,"5min":0.78200000000000003,"15min":0.84599999999999997},"last":0.629},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59399999999999997,"15min":0.59999999999999998},"min":{"1min":0.41899999999999998,"5min":0.38400000000000001,"15min":0.221},"max":{"1min":0.68899999999999995,"5min":0.81999999999999995,"15min":0.83099999999999996},"last":0.61899999999999999}]},{"osd":2,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.55300000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.29899999999999999},"max":{"1min":0.66300000000000003,"5min":0.749,"15min":0.77700000000000002},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56200000000000006,"15min":0.56999999999999995},"min":{"1min":0.35699999999999998,"5min":0.32300000000000001,"15min":0.254},"max":{"1min":0.68500000000000005,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.58799999999999997}]},{"osd":3,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56999999999999995,"15min":0.57899999999999996},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.23300000000000001},"max":{"1min":0.66900000000000004,"5min":0.745,"15min":0.81200000000000006},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.61199999999999999,"15min":0.61899999999999999},"min":{"1min":0.442,"5min":0.39700000000000002,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.79200000000000004,"15min":0.872},"last":0.60099999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.51500000000000001,"15min":0.501},"min":{"1min":0.32600000000000001,"5min":0.28599999999999998,"15min":0.27500000000000002},"max":{"1min":0.73699999999999999,"5min":0.754,"15min":0.754},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53700000000000003,"15min":0.53200000000000003},"min":{"1min":0.32700000000000001,"5min":0.29599999999999999,"15min":0.28799999999999998},"max":{"1min":0.72799999999999998,"5min":0.73199999999999998,"15min":0.82199999999999995},"last":0.48699999999999999}]},{"osd":5,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.503,"15min":0.49399999999999999},"min":{"1min":0.37,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.70599999999999996,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51800000000000002,"15min":0.51200000000000001},"min":{"1min":0.316,"5min":0.31,"15min":0.27000000000000002},"max":{"1min":0.79600000000000004,"5min":0.80200000000000005,"15min":0.83699999999999997},"last":0.46999999999999997}]},{"osd":7,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54700000000000004,"15min":0.53600000000000003},"min":{"1min":0.33500000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.749,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57499999999999996,"15min":0.56000000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.32100000000000001},"max":{"1min":0.77300000000000002,"5min":0.84799999999999998,"15min":0.88900000000000001},"last":0.52300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575076,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32872,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338968,"statfs":{"total":96636764160,"available":96603103232,"internally_reserved":0,"allocated":1789952,"data_stored":897573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36499999999999999,"15min":0.35999999999999999},"min":{"1min":0.25,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.58999999999999997},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.375,"15min":0.39100000000000001},"min":{"1min":0.26500000000000001,"5min":0.245,"15min":0.215},"max":{"1min":0.59099999999999997,"5min":0.628,"15min":0.628},"last":0.38900000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38800000000000001,"15min":0.39300000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.19400000000000001},"max":{"1min":0.57599999999999996,"5min":0.61599999999999999,"15min":0.67600000000000005},"last":0.435},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.28799999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.42399999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.44700000000000001,"15min":0.44600000000000001},"min":{"1min":0.29999999999999999,"5min":0.29199999999999998,"15min":0.28399999999999997},"max":{"1min":0.66400000000000003,"5min":0.67500000000000004,"15min":0.67800000000000005},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41199999999999998,"15min":0.41799999999999998},"min":{"1min":0.32100000000000001,"5min":0.29099999999999998,"15min":0.249},"max":{"1min":0.60899999999999999,"5min":0.65900000000000003,"15min":0.86299999999999999},"last":0.46600000000000003}]},{"osd":4,"last update":"Sat Apr 6 12:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.61399999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.34799999999999998},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.623,"15min":0.621},"min":{"1min":0.45700000000000002,"5min":0.36299999999999999,"15min":0.30399999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.94899999999999995},"last":0.65300000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60899999999999999,"15min":0.61299999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.38700000000000001},"max":{"1min":0.80000000000000004,"5min":0.85899999999999999,"15min":0.91300000000000003},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.627,"5min":0.629,"15min":0.627},"min":{"1min":0.437,"5min":0.41499999999999998,"15min":0.38300000000000001},"max":{"1min":0.871,"5min":0.999,"15min":0.999},"last":0.77500000000000002}]},{"osd":6,"last update":"Sat Apr 6 12:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61499999999999999,"15min":0.61399999999999999},"min":{"1min":0.40699999999999997,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82599999999999996,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.753},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63300000000000001,"15min":0.64400000000000002},"min":{"1min":0.45900000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.84799999999999998,"5min":0.88700000000000001,"15min":0.92600000000000005},"last":0.73899999999999999}]},{"osd":7,"last update":"Sat Apr 6 12:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65000000000000002,"15min":0.63900000000000001},"min":{"1min":0.48999999999999999,"5min":0.44600000000000001,"15min":0.38800000000000001},"max":{"1min":0.85699999999999998,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.624,"15min":0.63300000000000001},"min":{"1min":0.42199999999999999,"5min":0.40999999999999998,"15min":0.40200000000000002},"max":{"1min":0.75700000000000001,"5min":0.91400000000000003,"15min":0.95599999999999996},"last":0.64200000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575076,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33444,"kb_used_data":2316,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338396,"statfs":{"total":96636764160,"available":96602517504,"internally_reserved":0,"allocated":2371584,"data_stored":1481321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.37,"15min":0.38200000000000001},"min":{"1min":0.22,"5min":0.216,"15min":0.19},"max":{"1min":0.52500000000000002,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.38600000000000001,"15min":0.40500000000000003},"min":{"1min":0.26700000000000002,"5min":0.23499999999999999,"15min":0.217},"max":{"1min":0.61299999999999999,"5min":0.625,"15min":1.4359999999999999},"last":0.26000000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.39400000000000002,"15min":0.40699999999999997},"min":{"1min":0.26800000000000002,"5min":0.255,"15min":0.19700000000000001},"max":{"1min":0.58399999999999996,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41799999999999998,"15min":0.432},"min":{"1min":0.29299999999999998,"5min":0.26800000000000002,"15min":0.24299999999999999},"max":{"1min":0.59399999999999997,"5min":0.89900000000000002,"15min":1.4790000000000001},"last":0.376}]},{"osd":3,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.41199999999999998,"15min":0.41899999999999998},"min":{"1min":0.29899999999999999,"5min":0.28799999999999998,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":1.25},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.443,"15min":0.46000000000000002},"min":{"1min":0.29599999999999999,"5min":0.255,"15min":0.255},"max":{"1min":0.70999999999999996,"5min":0.74099999999999999,"15min":0.748},"last":0.40400000000000003}]},{"osd":4,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60199999999999998,"15min":0.624},"min":{"1min":0.41299999999999998,"5min":0.38400000000000001,"15min":0.35699999999999998},"max":{"1min":0.84799999999999998,"5min":1.1100000000000001,"15min":1.514},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60799999999999998,"15min":0.629},"min":{"1min":0.41299999999999998,"5min":0.36699999999999999,"15min":0.35199999999999998},"max":{"1min":0.84599999999999997,"5min":0.93999999999999995,"15min":1.202},"last":0.64300000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.59799999999999998,"15min":0.624},"min":{"1min":0.44400000000000001,"5min":0.34399999999999997,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.88400000000000001,"15min":1.0920000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.63,"15min":0.64600000000000002},"min":{"1min":0.42499999999999999,"5min":0.40899999999999997,"15min":0.315},"max":{"1min":0.83599999999999997,"5min":1.006,"15min":1.006},"last":0.55800000000000005}]},{"osd":6,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.627,"15min":0.64800000000000002},"min":{"1min":0.42999999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.63200000000000001,"15min":0.66200000000000003},"min":{"1min":0.47799999999999998,"5min":0.40300000000000002,"15min":0.38100000000000001},"max":{"1min":0.92000000000000004,"5min":1.052,"15min":1.613},"last":0.63300000000000001}]},{"osd":7,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63400000000000001,"15min":0.66300000000000003},"min":{"1min":0.47399999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88200000000000001,"5min":1.0820000000000001,"15min":1.54},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.63600000000000001,"15min":0.65900000000000003},"min":{"1min":0.49099999999999999,"5min":0.39300000000000002,"15min":0.375},"max":{"1min":0.90800000000000003,"5min":1.014,"15min":1.498},"last":0.53600000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575078,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55000,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316840,"statfs":{"total":96636764160,"available":96580444160,"internally_reserved":0,"allocated":1773568,"data_stored":890931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.372,"15min":0.38200000000000001},"min":{"1min":0.254,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.53400000000000003,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40699999999999997,"15min":0.41099999999999998},"min":{"1min":0.27300000000000002,"5min":0.246,"15min":0.20000000000000001},"max":{"1min":0.56999999999999995,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.27900000000000003}]},{"osd":1,"last update":"Sat Apr 6 12:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.38500000000000001,"15min":0.39900000000000002},"min":{"1min":0.25600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.45000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41699999999999998,"15min":0.43099999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.23599999999999999},"max":{"1min":0.66800000000000004,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.35599999999999998}]},{"osd":3,"last update":"Sat Apr 6 12:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.44,"15min":0.45000000000000001},"min":{"1min":0.30099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.86899999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.434,"15min":0.438},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.29099999999999998},"max":{"1min":0.64900000000000002,"5min":0.97099999999999997,"15min":1.0549999999999999},"last":0.26500000000000001}]},{"osd":4,"last update":"Sat Apr 6 12:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60299999999999998,"15min":0.61399999999999999},"min":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.35199999999999998},"max":{"1min":0.79500000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.623},"min":{"1min":0.45600000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.79700000000000004,"5min":0.91200000000000003,"15min":0.92600000000000005},"last":0.51700000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61899999999999999,"15min":0.63500000000000001},"min":{"1min":0.45600000000000002,"5min":0.374,"15min":0.36499999999999999},"max":{"1min":0.84999999999999998,"5min":1.0369999999999999,"15min":1.6870000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61699999999999999,"15min":0.627},"min":{"1min":0.46600000000000003,"5min":0.35399999999999998,"15min":0.31},"max":{"1min":0.80500000000000005,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.49099999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.63200000000000001},"min":{"1min":0.52600000000000002,"5min":0.41799999999999998,"15min":0.36399999999999999},"max":{"1min":0.91000000000000003,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.623,"15min":0.65000000000000002},"min":{"1min":0.44600000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81899999999999995,"5min":1.0760000000000001,"15min":1.728},"last":0.55600000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.621,"15min":0.64400000000000002},"min":{"1min":0.443,"5min":0.40699999999999997,"15min":0.35599999999999998},"max":{"1min":0.85299999999999998,"5min":0.95699999999999996,"15min":0.95799999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.64000000000000001,"15min":0.66000000000000003},"min":{"1min":0.42299999999999999,"5min":0.41499999999999998,"15min":0.39000000000000001},"max":{"1min":0.88300000000000001,"5min":1.0049999999999999,"15min":1.758},"last":0.53000000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575077,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32876,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338964,"statfs":{"total":96636764160,"available":96603099136,"internally_reserved":0,"allocated":1794048,"data_stored":893548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.247,"5min":0.23599999999999999,"15min":0.191},"max":{"1min":0.52900000000000003,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.36399999999999999,"15min":0.38200000000000001},"min":{"1min":0.245,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.73499999999999999},"last":0.26800000000000002}]},{"osd":1,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.376,"15min":0.38},"min":{"1min":0.222,"5min":0.222,"15min":0.217},"max":{"1min":0.49399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.318},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.40300000000000002,"15min":0.41199999999999998},"min":{"1min":0.26800000000000002,"5min":0.26100000000000001,"15min":0.25700000000000001},"max":{"1min":0.54700000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.28999999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.44700000000000001,"15min":0.45800000000000002},"min":{"1min":0.28999999999999998,"5min":0.28799999999999998,"15min":0.27200000000000002},"max":{"1min":0.56799999999999995,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.46500000000000002,"15min":0.47399999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.59299999999999997,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.371}]},{"osd":4,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57699999999999996,"15min":0.60199999999999998},"min":{"1min":0.41199999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.95999999999999996},"last":0.505},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.62},"min":{"1min":0.42299999999999999,"5min":0.39600000000000002,"15min":0.23699999999999999},"max":{"1min":0.80800000000000005,"5min":0.90100000000000002,"15min":0.93500000000000005},"last":0.60899999999999999}]},{"osd":5,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.623,"15min":0.65200000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.309},"max":{"1min":0.70999999999999996,"5min":0.92200000000000004,"15min":0.96699999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63600000000000001,"15min":0.65900000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.38200000000000001},"max":{"1min":0.93899999999999995,"5min":0.94499999999999995,"15min":0.95599999999999996},"last":0.56999999999999995}]},{"osd":6,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59899999999999998,"15min":0.61799999999999999},"min":{"1min":0.41499999999999998,"5min":0.38,"15min":0.36899999999999999},"max":{"1min":0.83799999999999997,"5min":1.026,"15min":1.026},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.64400000000000002,"15min":0.65500000000000003},"min":{"1min":0.45400000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.81399999999999995,"5min":1.716,"15min":1.716},"last":0.495}]},{"osd":7,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61899999999999999,"15min":0.63900000000000001},"min":{"1min":0.39800000000000002,"5min":0.377,"15min":0.33300000000000002},"max":{"1min":0.79500000000000004,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.64000000000000001,"15min":0.66000000000000003},"min":{"1min":0.442,"5min":0.40100000000000002,"15min":0.373},"max":{"1min":0.92500000000000004,"5min":1.002,"15min":1.002},"last":0.58799999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575077,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32872,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338968,"statfs":{"total":96636764160,"available":96603103232,"internally_reserved":0,"allocated":1789952,"data_stored":893514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55900000000000005,"15min":0.56200000000000006},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.23899999999999999},"max":{"1min":0.83399999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.375,"5min":0.33800000000000002,"15min":0.25900000000000001},"max":{"1min":0.79400000000000004,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.56399999999999995}]},{"osd":1,"last update":"Sat Apr 6 12:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59599999999999997,"15min":0.60799999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.29799999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58999999999999997,"15min":0.59299999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.57499999999999996}]},{"osd":2,"last update":"Sat Apr 6 12:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.57099999999999995,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.78100000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.495},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57499999999999996,"15min":0.58299999999999996},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.80900000000000005},"last":0.53400000000000003}]},{"osd":3,"last update":"Sat Apr 6 12:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56899999999999995,"15min":0.56299999999999994},"min":{"1min":0.39700000000000002,"5min":0.373,"15min":0.36399999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.54600000000000004}]},{"osd":5,"last update":"Sat Apr 6 12:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.47599999999999998,"15min":0.47899999999999998},"min":{"1min":0.33100000000000002,"5min":0.308,"15min":0.27200000000000002},"max":{"1min":0.67600000000000005,"5min":0.73599999999999999,"15min":0.83599999999999997},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48599999999999999,"15min":0.496},"min":{"1min":0.33500000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.68899999999999995,"5min":0.76600000000000001,"15min":0.77000000000000002},"last":0.37}]},{"osd":6,"last update":"Sat Apr 6 12:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.503,"15min":0.502},"min":{"1min":0.32400000000000001,"5min":0.307,"15min":0.30599999999999999},"max":{"1min":0.753,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38800000000000001,"5min":0.34100000000000003,"15min":0.32100000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.41099999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53500000000000003,"15min":0.53400000000000003},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72199999999999998,"5min":0.77600000000000002,"15min":0.82299999999999995},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.38700000000000001,"5min":0.36699999999999999,"15min":0.35599999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.39100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575077,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50920,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320920,"statfs":{"total":96636764160,"available":96584622080,"internally_reserved":0,"allocated":1794048,"data_stored":897711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54600000000000004,"15min":0.57099999999999995},"min":{"1min":0.371,"5min":0.32700000000000001,"15min":0.29099999999999998},"max":{"1min":0.68799999999999994,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.52400000000000002,"15min":0.54600000000000004},"min":{"1min":0.36699999999999999,"5min":0.29499999999999998,"15min":0.22500000000000001},"max":{"1min":0.63500000000000001,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.443}]},{"osd":1,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53100000000000003,"15min":0.55500000000000005},"min":{"1min":0.371,"5min":0.30399999999999999,"15min":0.26000000000000001},"max":{"1min":0.71199999999999997,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.53200000000000003,"15min":0.55300000000000005},"min":{"1min":0.39800000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.63600000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.42799999999999999}]},{"osd":2,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54300000000000004,"15min":0.56399999999999995},"min":{"1min":0.376,"5min":0.33200000000000002,"15min":0.21099999999999999},"max":{"1min":0.69199999999999995,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.56599999999999995},"min":{"1min":0.40600000000000003,"5min":0.35999999999999999,"15min":0.27400000000000002},"max":{"1min":0.68600000000000005,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.48599999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55600000000000005,"15min":0.56999999999999995},"min":{"1min":0.376,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75600000000000001,"5min":0.874,"15min":0.874},"last":0.503},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56799999999999995,"15min":0.59099999999999997},"min":{"1min":0.38700000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.65600000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55000000000000004}]},{"osd":4,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46899999999999997,"15min":0.46800000000000003},"min":{"1min":0.34100000000000003,"5min":0.28899999999999998,"15min":0.28299999999999997},"max":{"1min":0.66800000000000004,"5min":0.755,"15min":0.81200000000000006},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54500000000000004,"15min":0.53200000000000003},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.57999999999999996}]},{"osd":6,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48399999999999999,"15min":0.47699999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.28000000000000003},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.48999999999999999,"15min":0.48799999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.29699999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.56799999999999995}]},{"osd":7,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.501,"15min":0.502},"min":{"1min":0.33600000000000002,"5min":0.29799999999999999,"15min":0.26600000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.42399999999999999,"5min":0.40200000000000002,"15min":0.35199999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.61199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:31:26.215 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:31:26.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.374+0000 7fb61985c700 1 Processor -- start 2024-04-06T12:31:26.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.374+0000 7fb61985c700 1 -- start start 2024-04-06T12:31:26.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.375+0000 7fb61985c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6140b76c0 0x7fb6141682f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:26.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.375+0000 7fb61985c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6140cdf80 con 0x7fb6140d0a60 2024-04-06T12:31:26.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.375+0000 7fb61985c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6140d3230 con 0x7fb61415d7e0 2024-04-06T12:31:26.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.375+0000 7fb61985c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6140d2da0 con 0x7fb6140b76c0 2024-04-06T12:31:26.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.375+0000 7fb6127fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb6140d0a60 0x7fb6140d15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48196/0 (socket says 172.21.15.189:48196) 2024-04-06T12:31:26.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.375+0000 7fb6127fc700 1 -- 172.21.15.189:0/166021325 learned_addr learned my addr 172.21.15.189:0/166021325 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:26.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.375+0000 7fb611ffb700 1 -- 172.21.15.189:0/166021325 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2683677662 0 0) 0x7fb6140cdf80 con 0x7fb6140d0a60 2024-04-06T12:31:26.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.375+0000 7fb611ffb700 1 -- 172.21.15.189:0/166021325 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6000035b0 con 0x7fb6140d0a60 2024-04-06T12:31:26.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.375+0000 7fb6137fe700 1 --2- 172.21.15.189:0/166021325 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6140b76c0 0x7fb6141682f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:26.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.375+0000 7fb611ffb700 1 -- 172.21.15.189:0/166021325 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2953064838 0 0) 0x7fb6000035b0 con 0x7fb6140d0a60 2024-04-06T12:31:26.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.375+0000 7fb611ffb700 1 -- 172.21.15.189:0/166021325 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6140b76c0 msgr2=0x7fb6141682f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:26.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.375+0000 7fb611ffb700 1 --2- 172.21.15.189:0/166021325 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6140b76c0 0x7fb6141682f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:26.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.375+0000 7fb611ffb700 1 -- 172.21.15.189:0/166021325 >> v1:172.21.15.192:6789/0 conn(0x7fb61415d7e0 legacy=0x7fb6140d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:26.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.375+0000 7fb611ffb700 1 -- 172.21.15.189:0/166021325 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb61415dbc0 con 0x7fb6140d0a60 2024-04-06T12:31:26.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.375+0000 7fb611ffb700 1 -- 172.21.15.189:0/166021325 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fb604001670 con 0x7fb6140d0a60 2024-04-06T12:31:26.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb611ffb700 1 -- 172.21.15.189:0/166021325 >> v1:172.21.15.189:6789/0 conn(0x7fb6140d0a60 legacy=0x7fb6140d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:26.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb611ffb700 1 --2- 172.21.15.189:0/166021325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb600003a30 0x7fb600003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:26.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb611ffb700 1 --2- 172.21.15.189:0/166021325 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6140b76c0 0x7fb600004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:26.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb611ffb700 1 --2- 172.21.15.189:0/166021325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb600004d90 0x7fb600005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:26.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb611ffb700 1 -- 172.21.15.189:0/166021325 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6140d2da0 con 0x7fb600003a30 2024-04-06T12:31:26.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb611ffb700 1 -- 172.21.15.189:0/166021325 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60000baf0 con 0x7fb600004d90 2024-04-06T12:31:26.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb611ffb700 1 -- 172.21.15.189:0/166021325 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60000bcb0 con 0x7fb6140b76c0 2024-04-06T12:31:26.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb612ffd700 1 --2- 172.21.15.189:0/166021325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb600003a30 0x7fb600003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:26.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb6127fc700 1 --2- 172.21.15.189:0/166021325 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6140b76c0 0x7fb600004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:26.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb6137fe700 1 --2- 172.21.15.189:0/166021325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb600004d90 0x7fb600005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:26.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb612ffd700 1 -- 172.21.15.189:0/166021325 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6140b76c0 msgr2=0x7fb600004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:26.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb612ffd700 1 --2- 172.21.15.189:0/166021325 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6140b76c0 0x7fb600004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:26.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb612ffd700 1 -- 172.21.15.189:0/166021325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb600004d90 msgr2=0x7fb600005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:26.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb612ffd700 1 --2- 172.21.15.189:0/166021325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb600004d90 0x7fb600005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:26.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb612ffd700 1 -- 172.21.15.189:0/166021325 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6000035b0 con 0x7fb600003a30 2024-04-06T12:31:26.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb6127fc700 1 --2- 172.21.15.189:0/166021325 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6140b76c0 0x7fb600004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:26.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb6137fe700 1 --2- 172.21.15.189:0/166021325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb600004d90 0x7fb600005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:26.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb612ffd700 1 --2- 172.21.15.189:0/166021325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb600003a30 0x7fb600003e50 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc007fd0 tx=0x7fb600005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b67508ea216719fc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:26.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb611ffb700 1 -- 172.21.15.189:0/166021325 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc0052d0 con 0x7fb600003a30 2024-04-06T12:31:26.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb611ffb700 1 -- 172.21.15.189:0/166021325 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5fc005470 con 0x7fb600003a30 2024-04-06T12:31:26.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.376+0000 7fb611ffb700 1 -- 172.21.15.189:0/166021325 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc0038c0 con 0x7fb600003a30 2024-04-06T12:31:26.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.377+0000 7fb61985c700 1 -- 172.21.15.189:0/166021325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb600003a30 msgr2=0x7fb600003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:26.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.377+0000 7fb61985c700 1 --2- 172.21.15.189:0/166021325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb600003a30 0x7fb600003e50 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc007fd0 tx=0x7fb600005880 comp rx=0 tx=0).stop 2024-04-06T12:31:26.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.377+0000 7fb6137fe700 1 -- 172.21.15.189:0/166021325 reap_dead start 2024-04-06T12:31:26.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.377+0000 7fb61985c700 1 -- 172.21.15.189:0/166021325 shutdown_connections 2024-04-06T12:31:26.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.377+0000 7fb61985c700 1 -- 172.21.15.189:0/166021325 >> 172.21.15.189:0/166021325 conn(0x7fb614072250 msgr2=0x7fb614072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:26.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.377+0000 7fb61985c700 1 -- 172.21.15.189:0/166021325 shutdown_connections 2024-04-06T12:31:26.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.377+0000 7fb61985c700 1 -- 172.21.15.189:0/166021325 wait complete. 2024-04-06T12:31:26.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.378+0000 7fb61985c700 1 Processor -- start 2024-04-06T12:31:26.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.379+0000 7fb61985c700 1 -- start start 2024-04-06T12:31:26.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.379+0000 7fb61985c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb600003a30 0x7fb614159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:26.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.379+0000 7fb61985c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6140a9e80 0x7fb6141f3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:26.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.379+0000 7fb61985c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6141f63c0 0x7fb6140aa770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:26.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.379+0000 7fb61985c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6141f72f0 con 0x7fb600003a30 2024-04-06T12:31:26.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.380+0000 7fb61985c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb614159e80 con 0x7fb6140a9e80 2024-04-06T12:31:26.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.380+0000 7fb61985c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6141f6d30 con 0x7fb6141f63c0 2024-04-06T12:31:26.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.380+0000 7fb612ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb600003a30 0x7fb614159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:26.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.380+0000 7fb612ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb600003a30 0x7fb614159940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50966/0 (socket says 172.21.15.189:50966) 2024-04-06T12:31:26.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.380+0000 7fb612ffd700 1 -- 172.21.15.189:0/454875922 learned_addr learned my addr 172.21.15.189:0/454875922 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:26.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.380+0000 7fb6137fe700 1 --2- 172.21.15.189:0/454875922 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6141f63c0 0x7fb6140aa770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:26.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.380+0000 7fb6127fc700 1 --2- 172.21.15.189:0/454875922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6140a9e80 0x7fb6141f3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:26.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.380+0000 7fb612ffd700 1 -- 172.21.15.189:0/454875922 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6141f63c0 msgr2=0x7fb6140aa770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:26.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.380+0000 7fb612ffd700 1 --2- 172.21.15.189:0/454875922 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6141f63c0 0x7fb6140aa770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:26.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.380+0000 7fb612ffd700 1 -- 172.21.15.189:0/454875922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6140a9e80 msgr2=0x7fb6141f3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:26.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.380+0000 7fb612ffd700 1 --2- 172.21.15.189:0/454875922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6140a9e80 0x7fb6141f3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:26.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.380+0000 7fb612ffd700 1 -- 172.21.15.189:0/454875922 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6141f7010 con 0x7fb600003a30 2024-04-06T12:31:26.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.380+0000 7fb6137fe700 1 --2- 172.21.15.189:0/454875922 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6141f63c0 0x7fb6140aa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:26.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.380+0000 7fb612ffd700 1 --2- 172.21.15.189:0/454875922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb600003a30 0x7fb614159940 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc004000 tx=0x7fb5fc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:26.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.380+0000 7fb60bfff700 1 -- 172.21.15.189:0/454875922 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc006a60 con 0x7fb600003a30 2024-04-06T12:31:26.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.380+0000 7fb60bfff700 1 -- 172.21.15.189:0/454875922 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5fc006bc0 con 0x7fb600003a30 2024-04-06T12:31:26.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.380+0000 7fb61985c700 1 -- 172.21.15.189:0/454875922 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6141f2f70 con 0x7fb600003a30 2024-04-06T12:31:26.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.380+0000 7fb60bfff700 1 -- 172.21.15.189:0/454875922 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc005280 con 0x7fb600003a30 2024-04-06T12:31:26.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.380+0000 7fb61985c700 1 -- 172.21.15.189:0/454875922 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6141f3cb0 con 0x7fb600003a30 2024-04-06T12:31:26.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.381+0000 7fb61985c700 1 -- 172.21.15.189:0/454875922 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6141f2dd0 con 0x7fb600003a30 2024-04-06T12:31:26.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.382+0000 7fb60bfff700 1 -- 172.21.15.189:0/454875922 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fb5fc006fb0 con 0x7fb600003a30 2024-04-06T12:31:26.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.382+0000 7fb60bfff700 1 --2- 172.21.15.189:0/454875922 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb5f406adb0 0x7fb5f406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:26.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.382+0000 7fb60bfff700 1 -- 172.21.15.189:0/454875922 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb5fc08acc0 con 0x7fb600003a30 2024-04-06T12:31:26.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.382+0000 7fb6127fc700 1 --2- 172.21.15.189:0/454875922 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb5f406adb0 0x7fb5f406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:26.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.382+0000 7fb6127fc700 1 --2- 172.21.15.189:0/454875922 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb5f406adb0 0x7fb5f406d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb6140ccef0 tx=0x7fb604009040 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:26.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.386+0000 7fb60bfff700 1 -- 172.21.15.189:0/454875922 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb6141f2dd0 con 0x7fb600003a30 2024-04-06T12:31:26.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.580+0000 7fb61985c700 1 -- 172.21.15.189:0/454875922 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb6140d2750 con 0x7fb5f406adb0 2024-04-06T12:31:26.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.587+0000 7fb60bfff700 1 -- 172.21.15.189:0/454875922 <== mgr.8376 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289707 (secure 0 0 0) 0x7fb604004870 con 0x7fb5f406adb0 2024-04-06T12:31:26.589 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:26.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.590+0000 7fb61985c700 1 -- 172.21.15.189:0/454875922 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb5f406adb0 msgr2=0x7fb5f406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:26.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.590+0000 7fb61985c700 1 --2- 172.21.15.189:0/454875922 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb5f406adb0 0x7fb5f406d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb6140ccef0 tx=0x7fb604009040 comp rx=0 tx=0).stop 2024-04-06T12:31:26.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.590+0000 7fb61985c700 1 -- 172.21.15.189:0/454875922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb600003a30 msgr2=0x7fb614159940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:26.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.591+0000 7fb61985c700 1 --2- 172.21.15.189:0/454875922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb600003a30 0x7fb614159940 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc004000 tx=0x7fb5fc001120 comp rx=0 tx=0).stop 2024-04-06T12:31:26.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.591+0000 7fb61985c700 1 -- 172.21.15.189:0/454875922 shutdown_connections 2024-04-06T12:31:26.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.591+0000 7fb61985c700 1 --2- 172.21.15.189:0/454875922 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6141f63c0 0x7fb6140aa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:26.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.591+0000 7fb61985c700 1 --2- 172.21.15.189:0/454875922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb600003a30 0x7fb614159940 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:26.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.591+0000 7fb61985c700 1 --2- 172.21.15.189:0/454875922 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb5f406adb0 0x7fb5f406d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:26.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.591+0000 7fb61985c700 1 --2- 172.21.15.189:0/454875922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6140a9e80 0x7fb6141f3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:26.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.591+0000 7fb61985c700 1 -- 172.21.15.189:0/454875922 >> 172.21.15.189:0/454875922 conn(0x7fb614072250 msgr2=0x7fb61415f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:26.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.591+0000 7fb61985c700 1 -- 172.21.15.189:0/454875922 shutdown_connections 2024-04-06T12:31:26.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.591+0000 7fb61985c700 1 -- 172.21.15.189:0/454875922 wait complete. 2024-04-06T12:31:26.592 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:31:26.601 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-04-06T12:31:26.447927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":403,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":265,"ondisk_log_size":265,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":129,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330552,"kb_used_data":15140,"kb_used_omap":12,"kb_used_meta":315251,"kb_avail":754644168,"statfs":{"total":773094113280,"available":772755628032,"internally_reserved":0,"allocated":15503360,"data_stored":8342445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":322817592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.777049"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.114426+0000","last_change":"2024-04-06T12:31:22.113995+0000","last_active":"2024-04-06T12:31:22.114426+0000","last_peered":"2024-04-06T12:31:22.114426+0000","last_clean":"2024-04-06T12:31:22.114426+0000","last_became_active":"2024-04-06T12:31:22.113724+0000","last_became_peered":"2024-04-06T12:31:22.113724+0000","last_unstale":"2024-04-06T12:31:22.114426+0000","last_undegraded":"2024-04-06T12:31:22.114426+0000","last_fullsized":"2024-04-06T12:31:22.114426+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:42.850998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119229+0000","last_change":"2024-04-06T12:31:22.112749+0000","last_active":"2024-04-06T12:31:22.119229+0000","last_peered":"2024-04-06T12:31:22.119229+0000","last_clean":"2024-04-06T12:31:22.119229+0000","last_became_active":"2024-04-06T12:31:22.112498+0000","last_became_peered":"2024-04-06T12:31:22.112498+0000","last_unstale":"2024-04-06T12:31:22.119229+0000","last_undegraded":"2024-04-06T12:31:22.119229+0000","last_fullsized":"2024-04-06T12:31:22.119229+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:34:53.631590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.125107+0000","last_change":"2024-04-06T12:31:22.120490+0000","last_active":"2024-04-06T12:31:22.125107+0000","last_peered":"2024-04-06T12:31:22.125107+0000","last_clean":"2024-04-06T12:31:22.125107+0000","last_became_active":"2024-04-06T12:31:22.120251+0000","last_became_peered":"2024-04-06T12:31:22.120251+0000","last_unstale":"2024-04-06T12:31:22.125107+0000","last_undegraded":"2024-04-06T12:31:22.125107+0000","last_fullsized":"2024-04-06T12:31:22.125107+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:15:07.210571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099417+0000","last_change":"2024-04-06T12:31:22.109722+0000","last_active":"2024-04-06T12:31:23.099417+0000","last_peered":"2024-04-06T12:31:23.099417+0000","last_clean":"2024-04-06T12:31:23.099417+0000","last_became_active":"2024-04-06T12:31:22.108764+0000","last_became_peered":"2024-04-06T12:31:22.108764+0000","last_unstale":"2024-04-06T12:31:23.099417+0000","last_undegraded":"2024-04-06T12:31:23.099417+0000","last_fullsized":"2024-04-06T12:31:23.099417+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:11:35.249474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.110940+0000","last_change":"2024-04-06T12:31:22.108982+0000","last_active":"2024-04-06T12:31:22.110940+0000","last_peered":"2024-04-06T12:31:22.110940+0000","last_clean":"2024-04-06T12:31:22.110940+0000","last_became_active":"2024-04-06T12:31:22.108693+0000","last_became_peered":"2024-04-06T12:31:22.108693+0000","last_unstale":"2024-04-06T12:31:22.110940+0000","last_undegraded":"2024-04-06T12:31:22.110940+0000","last_fullsized":"2024-04-06T12:31:22.110940+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:40:35.155072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.113755+0000","last_change":"2024-04-06T12:31:22.112906+0000","last_active":"2024-04-06T12:31:22.113755+0000","last_peered":"2024-04-06T12:31:22.113755+0000","last_clean":"2024-04-06T12:31:22.113755+0000","last_became_active":"2024-04-06T12:31:22.112637+0000","last_became_peered":"2024-04-06T12:31:22.112637+0000","last_unstale":"2024-04-06T12:31:22.113755+0000","last_undegraded":"2024-04-06T12:31:22.113755+0000","last_fullsized":"2024-04-06T12:31:22.113755+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:50:56.202624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.112252+0000","last_change":"2024-04-06T12:31:22.110024+0000","last_active":"2024-04-06T12:31:22.112252+0000","last_peered":"2024-04-06T12:31:22.112252+0000","last_clean":"2024-04-06T12:31:22.112252+0000","last_became_active":"2024-04-06T12:31:22.109769+0000","last_became_peered":"2024-04-06T12:31:22.109769+0000","last_unstale":"2024-04-06T12:31:22.112252+0000","last_undegraded":"2024-04-06T12:31:22.112252+0000","last_fullsized":"2024-04-06T12:31:22.112252+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:08:57.991115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.122220+0000","last_change":"2024-04-06T12:31:22.119216+0000","last_active":"2024-04-06T12:31:22.122220+0000","last_peered":"2024-04-06T12:31:22.122220+0000","last_clean":"2024-04-06T12:31:22.122220+0000","last_became_active":"2024-04-06T12:31:22.118924+0000","last_became_peered":"2024-04-06T12:31:22.118924+0000","last_unstale":"2024-04-06T12:31:22.122220+0000","last_undegraded":"2024-04-06T12:31:22.122220+0000","last_fullsized":"2024-04-06T12:31:22.122220+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:00:53.973128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099679+0000","last_change":"2024-04-06T12:31:22.113133+0000","last_active":"2024-04-06T12:31:23.099679+0000","last_peered":"2024-04-06T12:31:23.099679+0000","last_clean":"2024-04-06T12:31:23.099679+0000","last_became_active":"2024-04-06T12:31:22.112779+0000","last_became_peered":"2024-04-06T12:31:22.112779+0000","last_unstale":"2024-04-06T12:31:23.099679+0000","last_undegraded":"2024-04-06T12:31:23.099679+0000","last_fullsized":"2024-04-06T12:31:23.099679+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:57:14.126836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.700178+0000","last_change":"2024-04-06T12:31:22.107148+0000","last_active":"2024-04-06T12:31:23.700178+0000","last_peered":"2024-04-06T12:31:23.700178+0000","last_clean":"2024-04-06T12:31:23.700178+0000","last_became_active":"2024-04-06T12:31:22.106890+0000","last_became_peered":"2024-04-06T12:31:22.106890+0000","last_unstale":"2024-04-06T12:31:23.700178+0000","last_undegraded":"2024-04-06T12:31:23.700178+0000","last_fullsized":"2024-04-06T12:31:23.700178+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:57:14.126836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.097497+0000","last_change":"2024-04-06T12:31:22.096782+0000","last_active":"2024-04-06T12:31:22.097497+0000","last_peered":"2024-04-06T12:31:22.097497+0000","last_clean":"2024-04-06T12:31:22.097497+0000","last_became_active":"2024-04-06T12:31:22.096496+0000","last_became_peered":"2024-04-06T12:31:22.096496+0000","last_unstale":"2024-04-06T12:31:22.097497+0000","last_undegraded":"2024-04-06T12:31:22.097497+0000","last_fullsized":"2024-04-06T12:31:22.097497+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:03:54.584558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.111230+0000","last_change":"2024-04-06T12:31:22.104964+0000","last_active":"2024-04-06T12:31:22.111230+0000","last_peered":"2024-04-06T12:31:22.111230+0000","last_clean":"2024-04-06T12:31:22.111230+0000","last_became_active":"2024-04-06T12:31:22.104553+0000","last_became_peered":"2024-04-06T12:31:22.104553+0000","last_unstale":"2024-04-06T12:31:22.111230+0000","last_undegraded":"2024-04-06T12:31:22.111230+0000","last_fullsized":"2024-04-06T12:31:22.111230+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:37:56.980318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.699934+0000","last_change":"2024-04-06T12:31:22.106527+0000","last_active":"2024-04-06T12:31:23.699934+0000","last_peered":"2024-04-06T12:31:23.699934+0000","last_clean":"2024-04-06T12:31:23.699934+0000","last_became_active":"2024-04-06T12:31:22.106265+0000","last_became_peered":"2024-04-06T12:31:22.106265+0000","last_unstale":"2024-04-06T12:31:23.699934+0000","last_undegraded":"2024-04-06T12:31:23.699934+0000","last_fullsized":"2024-04-06T12:31:23.699934+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:58:51.171247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119659+0000","last_change":"2024-04-06T12:31:22.117244+0000","last_active":"2024-04-06T12:31:22.119659+0000","last_peered":"2024-04-06T12:31:22.119659+0000","last_clean":"2024-04-06T12:31:22.119659+0000","last_became_active":"2024-04-06T12:31:22.117002+0000","last_became_peered":"2024-04-06T12:31:22.117002+0000","last_unstale":"2024-04-06T12:31:22.119659+0000","last_undegraded":"2024-04-06T12:31:22.119659+0000","last_fullsized":"2024-04-06T12:31:22.119659+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:22:31.614589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.098119+0000","last_change":"2024-04-06T12:31:22.097043+0000","last_active":"2024-04-06T12:31:22.098119+0000","last_peered":"2024-04-06T12:31:22.098119+0000","last_clean":"2024-04-06T12:31:22.098119+0000","last_became_active":"2024-04-06T12:31:22.096738+0000","last_became_peered":"2024-04-06T12:31:22.096738+0000","last_unstale":"2024-04-06T12:31:22.098119+0000","last_undegraded":"2024-04-06T12:31:22.098119+0000","last_fullsized":"2024-04-06T12:31:22.098119+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:11:39.129105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.106455+0000","last_change":"2024-04-06T12:31:22.102695+0000","last_active":"2024-04-06T12:31:22.106455+0000","last_peered":"2024-04-06T12:31:22.106455+0000","last_clean":"2024-04-06T12:31:22.106455+0000","last_became_active":"2024-04-06T12:31:22.102340+0000","last_became_peered":"2024-04-06T12:31:22.102340+0000","last_unstale":"2024-04-06T12:31:22.106455+0000","last_undegraded":"2024-04-06T12:31:22.106455+0000","last_fullsized":"2024-04-06T12:31:22.106455+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:17:56.649966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.700216+0000","last_change":"2024-04-06T12:31:22.105864+0000","last_active":"2024-04-06T12:31:23.700216+0000","last_peered":"2024-04-06T12:31:23.700216+0000","last_clean":"2024-04-06T12:31:23.700216+0000","last_became_active":"2024-04-06T12:31:22.105709+0000","last_became_peered":"2024-04-06T12:31:22.105709+0000","last_unstale":"2024-04-06T12:31:23.700216+0000","last_undegraded":"2024-04-06T12:31:23.700216+0000","last_fullsized":"2024-04-06T12:31:23.700216+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:40:35.155072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119212+0000","last_change":"2024-04-06T12:31:22.112614+0000","last_active":"2024-04-06T12:31:22.119212+0000","last_peered":"2024-04-06T12:31:22.119212+0000","last_clean":"2024-04-06T12:31:22.119212+0000","last_became_active":"2024-04-06T12:31:22.112196+0000","last_became_peered":"2024-04-06T12:31:22.112196+0000","last_unstale":"2024-04-06T12:31:22.119212+0000","last_undegraded":"2024-04-06T12:31:22.119212+0000","last_fullsized":"2024-04-06T12:31:22.119212+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:15:12.602022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097669+0000","last_change":"2024-04-06T12:31:22.112676+0000","last_active":"2024-04-06T12:31:23.097669+0000","last_peered":"2024-04-06T12:31:23.097669+0000","last_clean":"2024-04-06T12:31:23.097669+0000","last_became_active":"2024-04-06T12:31:22.112421+0000","last_became_peered":"2024-04-06T12:31:22.112421+0000","last_unstale":"2024-04-06T12:31:23.097669+0000","last_undegraded":"2024-04-06T12:31:23.097669+0000","last_fullsized":"2024-04-06T12:31:23.097669+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:47.974510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.107214+0000","last_change":"2024-04-06T12:31:22.102663+0000","last_active":"2024-04-06T12:31:22.107214+0000","last_peered":"2024-04-06T12:31:22.107214+0000","last_clean":"2024-04-06T12:31:22.107214+0000","last_became_active":"2024-04-06T12:31:22.102413+0000","last_became_peered":"2024-04-06T12:31:22.102413+0000","last_unstale":"2024-04-06T12:31:22.107214+0000","last_undegraded":"2024-04-06T12:31:22.107214+0000","last_fullsized":"2024-04-06T12:31:22.107214+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:16.549838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099319+0000","last_change":"2024-04-06T12:31:22.109091+0000","last_active":"2024-04-06T12:31:23.099319+0000","last_peered":"2024-04-06T12:31:23.099319+0000","last_clean":"2024-04-06T12:31:23.099319+0000","last_became_active":"2024-04-06T12:31:22.108814+0000","last_became_peered":"2024-04-06T12:31:22.108814+0000","last_unstale":"2024-04-06T12:31:23.099319+0000","last_undegraded":"2024-04-06T12:31:23.099319+0000","last_fullsized":"2024-04-06T12:31:23.099319+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:42.850998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.116897+0000","last_change":"2024-04-06T12:31:22.115284+0000","last_active":"2024-04-06T12:31:22.116897+0000","last_peered":"2024-04-06T12:31:22.116897+0000","last_clean":"2024-04-06T12:31:22.116897+0000","last_became_active":"2024-04-06T12:31:22.114791+0000","last_became_peered":"2024-04-06T12:31:22.114791+0000","last_unstale":"2024-04-06T12:31:22.116897+0000","last_undegraded":"2024-04-06T12:31:22.116897+0000","last_fullsized":"2024-04-06T12:31:22.116897+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:57:14.126836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099082+0000","last_change":"2024-04-06T12:31:22.116553+0000","last_active":"2024-04-06T12:31:23.099082+0000","last_peered":"2024-04-06T12:31:23.099082+0000","last_clean":"2024-04-06T12:31:23.099082+0000","last_became_active":"2024-04-06T12:31:22.116301+0000","last_became_peered":"2024-04-06T12:31:22.116301+0000","last_unstale":"2024-04-06T12:31:23.099082+0000","last_undegraded":"2024-04-06T12:31:23.099082+0000","last_fullsized":"2024-04-06T12:31:23.099082+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:40:35.155072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097153+0000","last_change":"2024-04-06T12:31:22.109723+0000","last_active":"2024-04-06T12:31:23.097153+0000","last_peered":"2024-04-06T12:31:23.097153+0000","last_clean":"2024-04-06T12:31:23.097153+0000","last_became_active":"2024-04-06T12:31:22.109420+0000","last_became_peered":"2024-04-06T12:31:22.109420+0000","last_unstale":"2024-04-06T12:31:23.097153+0000","last_undegraded":"2024-04-06T12:31:23.097153+0000","last_fullsized":"2024-04-06T12:31:23.097153+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:05:59.961267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.098108+0000","last_change":"2024-04-06T12:31:22.097346+0000","last_active":"2024-04-06T12:31:23.098108+0000","last_peered":"2024-04-06T12:31:23.098108+0000","last_clean":"2024-04-06T12:31:23.098108+0000","last_became_active":"2024-04-06T12:31:22.097002+0000","last_became_peered":"2024-04-06T12:31:22.097002+0000","last_unstale":"2024-04-06T12:31:23.098108+0000","last_undegraded":"2024-04-06T12:31:23.098108+0000","last_fullsized":"2024-04-06T12:31:23.098108+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:35:36.730191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.111670+0000","last_change":"2024-04-06T12:31:22.108009+0000","last_active":"2024-04-06T12:31:22.111670+0000","last_peered":"2024-04-06T12:31:22.111670+0000","last_clean":"2024-04-06T12:31:22.111670+0000","last_became_active":"2024-04-06T12:31:22.107749+0000","last_became_peered":"2024-04-06T12:31:22.107749+0000","last_unstale":"2024-04-06T12:31:22.111670+0000","last_undegraded":"2024-04-06T12:31:22.111670+0000","last_fullsized":"2024-04-06T12:31:22.111670+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:52.611199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097799+0000","last_change":"2024-04-06T12:31:22.112068+0000","last_active":"2024-04-06T12:31:23.097799+0000","last_peered":"2024-04-06T12:31:23.097799+0000","last_clean":"2024-04-06T12:31:23.097799+0000","last_became_active":"2024-04-06T12:31:22.111808+0000","last_became_peered":"2024-04-06T12:31:22.111808+0000","last_unstale":"2024-04-06T12:31:23.097799+0000","last_undegraded":"2024-04-06T12:31:23.097799+0000","last_fullsized":"2024-04-06T12:31:23.097799+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:52.934051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.102920+0000","last_change":"2024-04-06T12:31:22.100138+0000","last_active":"2024-04-06T12:31:22.102920+0000","last_peered":"2024-04-06T12:31:22.102920+0000","last_clean":"2024-04-06T12:31:22.102920+0000","last_became_active":"2024-04-06T12:31:22.099876+0000","last_became_peered":"2024-04-06T12:31:22.099876+0000","last_unstale":"2024-04-06T12:31:22.102920+0000","last_undegraded":"2024-04-06T12:31:22.102920+0000","last_fullsized":"2024-04-06T12:31:22.102920+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:23:01.349287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.115931+0000","last_change":"2024-04-06T12:31:22.115047+0000","last_active":"2024-04-06T12:31:22.115931+0000","last_peered":"2024-04-06T12:31:22.115931+0000","last_clean":"2024-04-06T12:31:22.115931+0000","last_became_active":"2024-04-06T12:31:22.114798+0000","last_became_peered":"2024-04-06T12:31:22.114798+0000","last_unstale":"2024-04-06T12:31:22.115931+0000","last_undegraded":"2024-04-06T12:31:22.115931+0000","last_fullsized":"2024-04-06T12:31:22.115931+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:40.865129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119314+0000","last_change":"2024-04-06T12:31:22.114823+0000","last_active":"2024-04-06T12:31:22.119314+0000","last_peered":"2024-04-06T12:31:22.119314+0000","last_clean":"2024-04-06T12:31:22.119314+0000","last_became_active":"2024-04-06T12:31:22.114199+0000","last_became_peered":"2024-04-06T12:31:22.114199+0000","last_unstale":"2024-04-06T12:31:22.119314+0000","last_undegraded":"2024-04-06T12:31:22.119314+0000","last_fullsized":"2024-04-06T12:31:22.119314+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:29:27.106970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097365+0000","last_change":"2024-04-06T12:31:22.099930+0000","last_active":"2024-04-06T12:31:23.097365+0000","last_peered":"2024-04-06T12:31:23.097365+0000","last_clean":"2024-04-06T12:31:23.097365+0000","last_became_active":"2024-04-06T12:31:22.099670+0000","last_became_peered":"2024-04-06T12:31:22.099670+0000","last_unstale":"2024-04-06T12:31:23.097365+0000","last_undegraded":"2024-04-06T12:31:23.097365+0000","last_fullsized":"2024-04-06T12:31:23.097365+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:47.974510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097871+0000","last_change":"2024-04-06T12:31:22.109490+0000","last_active":"2024-04-06T12:31:23.097871+0000","last_peered":"2024-04-06T12:31:23.097871+0000","last_clean":"2024-04-06T12:31:23.097871+0000","last_became_active":"2024-04-06T12:31:22.109213+0000","last_became_peered":"2024-04-06T12:31:22.109213+0000","last_unstale":"2024-04-06T12:31:23.097871+0000","last_undegraded":"2024-04-06T12:31:23.097871+0000","last_fullsized":"2024-04-06T12:31:23.097871+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:41.101353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.700144+0000","last_change":"2024-04-06T12:31:22.106880+0000","last_active":"2024-04-06T12:31:23.700144+0000","last_peered":"2024-04-06T12:31:23.700144+0000","last_clean":"2024-04-06T12:31:23.700144+0000","last_became_active":"2024-04-06T12:31:22.106647+0000","last_became_peered":"2024-04-06T12:31:22.106647+0000","last_unstale":"2024-04-06T12:31:23.700144+0000","last_undegraded":"2024-04-06T12:31:23.700144+0000","last_fullsized":"2024-04-06T12:31:23.700144+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:11:39.129105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"143'4","reported_seq":15,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097696+0000","last_change":"2024-04-06T12:31:21.686002+0000","last_active":"2024-04-06T12:31:23.097696+0000","last_peered":"2024-04-06T12:31:23.097696+0000","last_clean":"2024-04-06T12:31:23.097696+0000","last_became_active":"2024-04-06T12:31:21.685729+0000","last_became_peered":"2024-04-06T12:31:21.685729+0000","last_unstale":"2024-04-06T12:31:23.097696+0000","last_undegraded":"2024-04-06T12:31:23.097696+0000","last_fullsized":"2024-04-06T12:31:23.097696+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:43:58.040962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.113846+0000","last_change":"2024-04-06T12:31:22.113109+0000","last_active":"2024-04-06T12:31:22.113846+0000","last_peered":"2024-04-06T12:31:22.113846+0000","last_clean":"2024-04-06T12:31:22.113846+0000","last_became_active":"2024-04-06T12:31:22.112853+0000","last_became_peered":"2024-04-06T12:31:22.112853+0000","last_unstale":"2024-04-06T12:31:22.113846+0000","last_undegraded":"2024-04-06T12:31:22.113846+0000","last_fullsized":"2024-04-06T12:31:22.113846+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:30:20.270321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"143'1","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.700171+0000","last_change":"2024-04-06T12:31:21.684151+0000","last_active":"2024-04-06T12:31:23.700171+0000","last_peered":"2024-04-06T12:31:23.700171+0000","last_clean":"2024-04-06T12:31:23.700171+0000","last_became_active":"2024-04-06T12:31:21.683896+0000","last_became_peered":"2024-04-06T12:31:21.683896+0000","last_unstale":"2024-04-06T12:31:23.700171+0000","last_undegraded":"2024-04-06T12:31:23.700171+0000","last_fullsized":"2024-04-06T12:31:23.700171+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:05:58.955969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097806+0000","last_change":"2024-04-06T12:31:22.103398+0000","last_active":"2024-04-06T12:31:23.097806+0000","last_peered":"2024-04-06T12:31:23.097806+0000","last_clean":"2024-04-06T12:31:23.097806+0000","last_became_active":"2024-04-06T12:31:22.103141+0000","last_became_peered":"2024-04-06T12:31:22.103141+0000","last_unstale":"2024-04-06T12:31:23.097806+0000","last_undegraded":"2024-04-06T12:31:23.097806+0000","last_fullsized":"2024-04-06T12:31:23.097806+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:06:04.683437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"143'2","reported_seq":13,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099564+0000","last_change":"2024-04-06T12:31:21.689017+0000","last_active":"2024-04-06T12:31:23.099564+0000","last_peered":"2024-04-06T12:31:23.099564+0000","last_clean":"2024-04-06T12:31:23.099564+0000","last_became_active":"2024-04-06T12:31:21.688768+0000","last_became_peered":"2024-04-06T12:31:21.688768+0000","last_unstale":"2024-04-06T12:31:23.099564+0000","last_undegraded":"2024-04-06T12:31:23.099564+0000","last_fullsized":"2024-04-06T12:31:23.099564+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:46.969211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099707+0000","last_change":"2024-04-06T12:31:22.110126+0000","last_active":"2024-04-06T12:31:23.099707+0000","last_peered":"2024-04-06T12:31:23.099707+0000","last_clean":"2024-04-06T12:31:23.099707+0000","last_became_active":"2024-04-06T12:31:22.109147+0000","last_became_peered":"2024-04-06T12:31:22.109147+0000","last_unstale":"2024-04-06T12:31:23.099707+0000","last_undegraded":"2024-04-06T12:31:23.099707+0000","last_fullsized":"2024-04-06T12:31:23.099707+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:17.208811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"143'2","reported_seq":13,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.700133+0000","last_change":"2024-04-06T12:31:21.689407+0000","last_active":"2024-04-06T12:31:23.700133+0000","last_peered":"2024-04-06T12:31:23.700133+0000","last_clean":"2024-04-06T12:31:23.700133+0000","last_became_active":"2024-04-06T12:31:21.689126+0000","last_became_peered":"2024-04-06T12:31:21.689126+0000","last_unstale":"2024-04-06T12:31:23.700133+0000","last_undegraded":"2024-04-06T12:31:23.700133+0000","last_fullsized":"2024-04-06T12:31:23.700133+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:51.928752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.699934+0000","last_change":"2024-04-06T12:31:22.110058+0000","last_active":"2024-04-06T12:31:23.699934+0000","last_peered":"2024-04-06T12:31:23.699934+0000","last_clean":"2024-04-06T12:31:23.699934+0000","last_became_active":"2024-04-06T12:31:22.109137+0000","last_became_peered":"2024-04-06T12:31:22.109137+0000","last_unstale":"2024-04-06T12:31:23.699934+0000","last_undegraded":"2024-04-06T12:31:23.699934+0000","last_fullsized":"2024-04-06T12:31:23.699934+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:42.850998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"143'1","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.106972+0000","last_change":"2024-04-06T12:31:21.685817+0000","last_active":"2024-04-06T12:31:22.106972+0000","last_peered":"2024-04-06T12:31:22.106972+0000","last_clean":"2024-04-06T12:31:22.106972+0000","last_became_active":"2024-04-06T12:31:21.685360+0000","last_became_peered":"2024-04-06T12:31:21.685360+0000","last_unstale":"2024-04-06T12:31:22.106972+0000","last_undegraded":"2024-04-06T12:31:22.106972+0000","last_fullsized":"2024-04-06T12:31:22.106972+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:06:03.678138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.107601+0000","last_change":"2024-04-06T12:31:22.106184+0000","last_active":"2024-04-06T12:31:22.107601+0000","last_peered":"2024-04-06T12:31:22.107601+0000","last_clean":"2024-04-06T12:31:22.107601+0000","last_became_active":"2024-04-06T12:31:22.105905+0000","last_became_peered":"2024-04-06T12:31:22.105905+0000","last_unstale":"2024-04-06T12:31:22.107601+0000","last_undegraded":"2024-04-06T12:31:22.107601+0000","last_fullsized":"2024-04-06T12:31:22.107601+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:11:35.249474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"143'1","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.111756+0000","last_change":"2024-04-06T12:31:21.694224+0000","last_active":"2024-04-06T12:31:22.111756+0000","last_peered":"2024-04-06T12:31:22.111756+0000","last_clean":"2024-04-06T12:31:22.111756+0000","last_became_active":"2024-04-06T12:31:21.693927+0000","last_became_peered":"2024-04-06T12:31:21.693927+0000","last_unstale":"2024-04-06T12:31:22.111756+0000","last_undegraded":"2024-04-06T12:31:22.111756+0000","last_fullsized":"2024-04-06T12:31:22.111756+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:46:29.474190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119572+0000","last_change":"2024-04-06T12:31:22.116921+0000","last_active":"2024-04-06T12:31:22.119572+0000","last_peered":"2024-04-06T12:31:22.119572+0000","last_clean":"2024-04-06T12:31:22.119572+0000","last_became_active":"2024-04-06T12:31:22.114327+0000","last_became_peered":"2024-04-06T12:31:22.114327+0000","last_unstale":"2024-04-06T12:31:22.119572+0000","last_undegraded":"2024-04-06T12:31:22.119572+0000","last_fullsized":"2024-04-06T12:31:22.119572+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:36.711304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097459+0000","last_change":"2024-04-06T12:31:21.691092+0000","last_active":"2024-04-06T12:31:23.097459+0000","last_peered":"2024-04-06T12:31:23.097459+0000","last_clean":"2024-04-06T12:31:23.097459+0000","last_became_active":"2024-04-06T12:31:21.690783+0000","last_became_peered":"2024-04-06T12:31:21.690783+0000","last_unstale":"2024-04-06T12:31:23.097459+0000","last_undegraded":"2024-04-06T12:31:23.097459+0000","last_fullsized":"2024-04-06T12:31:23.097459+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:12:51.214120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.699934+0000","last_change":"2024-04-06T12:31:22.115026+0000","last_active":"2024-04-06T12:31:23.699934+0000","last_peered":"2024-04-06T12:31:23.699934+0000","last_clean":"2024-04-06T12:31:23.699934+0000","last_became_active":"2024-04-06T12:31:22.114784+0000","last_became_peered":"2024-04-06T12:31:22.114784+0000","last_unstale":"2024-04-06T12:31:23.699934+0000","last_undegraded":"2024-04-06T12:31:23.699934+0000","last_fullsized":"2024-04-06T12:31:23.699934+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:59:14.197440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.095728+0000","last_change":"2024-04-06T12:31:21.697322+0000","last_active":"2024-04-06T12:31:22.095728+0000","last_peered":"2024-04-06T12:31:22.095728+0000","last_clean":"2024-04-06T12:31:22.095728+0000","last_became_active":"2024-04-06T12:31:21.696959+0000","last_became_peered":"2024-04-06T12:31:21.696959+0000","last_unstale":"2024-04-06T12:31:22.095728+0000","last_undegraded":"2024-04-06T12:31:22.095728+0000","last_fullsized":"2024-04-06T12:31:22.095728+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:08:37.545709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097268+0000","last_change":"2024-04-06T12:31:22.113648+0000","last_active":"2024-04-06T12:31:23.097268+0000","last_peered":"2024-04-06T12:31:23.097268+0000","last_clean":"2024-04-06T12:31:23.097268+0000","last_became_active":"2024-04-06T12:31:22.113243+0000","last_became_peered":"2024-04-06T12:31:22.113243+0000","last_unstale":"2024-04-06T12:31:23.097268+0000","last_undegraded":"2024-04-06T12:31:23.097268+0000","last_fullsized":"2024-04-06T12:31:23.097268+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:05:59.961267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097839+0000","last_change":"2024-04-06T12:31:21.696014+0000","last_active":"2024-04-06T12:31:23.097839+0000","last_peered":"2024-04-06T12:31:23.097839+0000","last_clean":"2024-04-06T12:31:23.097839+0000","last_became_active":"2024-04-06T12:31:21.695744+0000","last_became_peered":"2024-04-06T12:31:21.695744+0000","last_unstale":"2024-04-06T12:31:23.097839+0000","last_undegraded":"2024-04-06T12:31:23.097839+0000","last_fullsized":"2024-04-06T12:31:23.097839+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:51.605900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097617+0000","last_change":"2024-04-06T12:31:22.115749+0000","last_active":"2024-04-06T12:31:23.097617+0000","last_peered":"2024-04-06T12:31:23.097617+0000","last_clean":"2024-04-06T12:31:23.097617+0000","last_became_active":"2024-04-06T12:31:22.115472+0000","last_became_peered":"2024-04-06T12:31:22.115472+0000","last_unstale":"2024-04-06T12:31:23.097617+0000","last_undegraded":"2024-04-06T12:31:23.097617+0000","last_fullsized":"2024-04-06T12:31:23.097617+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:09:03.752277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097893+0000","last_change":"2024-04-06T12:31:21.691574+0000","last_active":"2024-04-06T12:31:23.097893+0000","last_peered":"2024-04-06T12:31:23.097893+0000","last_clean":"2024-04-06T12:31:23.097893+0000","last_became_active":"2024-04-06T12:31:21.691280+0000","last_became_peered":"2024-04-06T12:31:21.691280+0000","last_unstale":"2024-04-06T12:31:23.097893+0000","last_undegraded":"2024-04-06T12:31:23.097893+0000","last_fullsized":"2024-04-06T12:31:23.097893+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:18:36.586968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119193+0000","last_change":"2024-04-06T12:31:22.112466+0000","last_active":"2024-04-06T12:31:22.119193+0000","last_peered":"2024-04-06T12:31:22.119193+0000","last_clean":"2024-04-06T12:31:22.119193+0000","last_became_active":"2024-04-06T12:31:22.112168+0000","last_became_peered":"2024-04-06T12:31:22.112168+0000","last_unstale":"2024-04-06T12:31:22.119193+0000","last_undegraded":"2024-04-06T12:31:22.119193+0000","last_fullsized":"2024-04-06T12:31:22.119193+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:27.379210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"143'3","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097719+0000","last_change":"2024-04-06T12:31:21.690621+0000","last_active":"2024-04-06T12:31:23.097719+0000","last_peered":"2024-04-06T12:31:23.097719+0000","last_clean":"2024-04-06T12:31:23.097719+0000","last_became_active":"2024-04-06T12:31:21.690327+0000","last_became_peered":"2024-04-06T12:31:21.690327+0000","last_unstale":"2024-04-06T12:31:23.097719+0000","last_undegraded":"2024-04-06T12:31:23.097719+0000","last_fullsized":"2024-04-06T12:31:23.097719+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:53:06.620534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.098117+0000","last_change":"2024-04-06T12:31:22.110418+0000","last_active":"2024-04-06T12:31:23.098117+0000","last_peered":"2024-04-06T12:31:23.098117+0000","last_clean":"2024-04-06T12:31:23.098117+0000","last_became_active":"2024-04-06T12:31:22.110133+0000","last_became_peered":"2024-04-06T12:31:22.110133+0000","last_unstale":"2024-04-06T12:31:23.098117+0000","last_undegraded":"2024-04-06T12:31:23.098117+0000","last_fullsized":"2024-04-06T12:31:23.098117+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:02:35.211216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"143'6","reported_seq":16,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.127772+0000","last_change":"2024-04-06T12:31:21.690830+0000","last_active":"2024-04-06T12:31:22.127772+0000","last_peered":"2024-04-06T12:31:22.127772+0000","last_clean":"2024-04-06T12:31:22.127772+0000","last_became_active":"2024-04-06T12:31:21.690252+0000","last_became_peered":"2024-04-06T12:31:21.690252+0000","last_unstale":"2024-04-06T12:31:22.127772+0000","last_undegraded":"2024-04-06T12:31:22.127772+0000","last_fullsized":"2024-04-06T12:31:22.127772+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:45:34.281066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097382+0000","last_change":"2024-04-06T12:31:22.119711+0000","last_active":"2024-04-06T12:31:23.097382+0000","last_peered":"2024-04-06T12:31:23.097382+0000","last_clean":"2024-04-06T12:31:23.097382+0000","last_became_active":"2024-04-06T12:31:22.119435+0000","last_became_peered":"2024-04-06T12:31:22.119435+0000","last_unstale":"2024-04-06T12:31:23.097382+0000","last_undegraded":"2024-04-06T12:31:23.097382+0000","last_fullsized":"2024-04-06T12:31:23.097382+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:02:35.211216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"143'2","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.118565+0000","last_change":"2024-04-06T12:31:21.689856+0000","last_active":"2024-04-06T12:31:22.118565+0000","last_peered":"2024-04-06T12:31:22.118565+0000","last_clean":"2024-04-06T12:31:22.118565+0000","last_became_active":"2024-04-06T12:31:21.689628+0000","last_became_peered":"2024-04-06T12:31:21.689628+0000","last_unstale":"2024-04-06T12:31:22.118565+0000","last_undegraded":"2024-04-06T12:31:22.118565+0000","last_fullsized":"2024-04-06T12:31:22.118565+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:16.203513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.700028+0000","last_change":"2024-04-06T12:31:22.109860+0000","last_active":"2024-04-06T12:31:23.700028+0000","last_peered":"2024-04-06T12:31:23.700028+0000","last_clean":"2024-04-06T12:31:23.700028+0000","last_became_active":"2024-04-06T12:31:22.109641+0000","last_became_peered":"2024-04-06T12:31:22.109641+0000","last_unstale":"2024-04-06T12:31:23.700028+0000","last_undegraded":"2024-04-06T12:31:23.700028+0000","last_fullsized":"2024-04-06T12:31:23.700028+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:09:03.294158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.103362+0000","last_change":"2024-04-06T12:31:21.691247+0000","last_active":"2024-04-06T12:31:22.103362+0000","last_peered":"2024-04-06T12:31:22.103362+0000","last_clean":"2024-04-06T12:31:22.103362+0000","last_became_active":"2024-04-06T12:31:21.690386+0000","last_became_peered":"2024-04-06T12:31:21.690386+0000","last_unstale":"2024-04-06T12:31:22.103362+0000","last_undegraded":"2024-04-06T12:31:22.103362+0000","last_fullsized":"2024-04-06T12:31:22.103362+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:03:53.579260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.100240+0000","last_change":"2024-04-06T12:31:22.097282+0000","last_active":"2024-04-06T12:31:22.100240+0000","last_peered":"2024-04-06T12:31:22.100240+0000","last_clean":"2024-04-06T12:31:22.100240+0000","last_became_active":"2024-04-06T12:31:22.097047+0000","last_became_peered":"2024-04-06T12:31:22.097047+0000","last_unstale":"2024-04-06T12:31:22.100240+0000","last_undegraded":"2024-04-06T12:31:22.100240+0000","last_fullsized":"2024-04-06T12:31:22.100240+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:19:42.830042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.097898+0000","last_change":"2024-04-06T12:31:21.690399+0000","last_active":"2024-04-06T12:31:22.097898+0000","last_peered":"2024-04-06T12:31:22.097898+0000","last_clean":"2024-04-06T12:31:22.097898+0000","last_became_active":"2024-04-06T12:31:21.689993+0000","last_became_peered":"2024-04-06T12:31:21.689993+0000","last_unstale":"2024-04-06T12:31:22.097898+0000","last_undegraded":"2024-04-06T12:31:22.097898+0000","last_fullsized":"2024-04-06T12:31:22.097898+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:34:09.064491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097668+0000","last_change":"2024-04-06T12:31:22.104431+0000","last_active":"2024-04-06T12:31:23.097668+0000","last_peered":"2024-04-06T12:31:23.097668+0000","last_clean":"2024-04-06T12:31:23.097668+0000","last_became_active":"2024-04-06T12:31:22.104082+0000","last_became_peered":"2024-04-06T12:31:22.104082+0000","last_unstale":"2024-04-06T12:31:23.097668+0000","last_undegraded":"2024-04-06T12:31:23.097668+0000","last_fullsized":"2024-04-06T12:31:23.097668+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:17:13.685324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"143'1","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.700267+0000","last_change":"2024-04-06T12:31:21.691715+0000","last_active":"2024-04-06T12:31:23.700267+0000","last_peered":"2024-04-06T12:31:23.700267+0000","last_clean":"2024-04-06T12:31:23.700267+0000","last_became_active":"2024-04-06T12:31:21.691439+0000","last_became_peered":"2024-04-06T12:31:21.691439+0000","last_unstale":"2024-04-06T12:31:23.700267+0000","last_undegraded":"2024-04-06T12:31:23.700267+0000","last_fullsized":"2024-04-06T12:31:23.700267+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:46.969211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.116882+0000","last_change":"2024-04-06T12:31:22.115140+0000","last_active":"2024-04-06T12:31:22.116882+0000","last_peered":"2024-04-06T12:31:22.116882+0000","last_clean":"2024-04-06T12:31:22.116882+0000","last_became_active":"2024-04-06T12:31:22.114819+0000","last_became_peered":"2024-04-06T12:31:22.114819+0000","last_unstale":"2024-04-06T12:31:22.116882+0000","last_undegraded":"2024-04-06T12:31:22.116882+0000","last_fullsized":"2024-04-06T12:31:22.116882+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:17.208811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"143'2","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.102282+0000","last_change":"2024-04-06T12:31:21.694984+0000","last_active":"2024-04-06T12:31:22.102282+0000","last_peered":"2024-04-06T12:31:22.102282+0000","last_clean":"2024-04-06T12:31:22.102282+0000","last_became_active":"2024-04-06T12:31:21.694716+0000","last_became_peered":"2024-04-06T12:31:21.694716+0000","last_unstale":"2024-04-06T12:31:22.102282+0000","last_undegraded":"2024-04-06T12:31:22.102282+0000","last_fullsized":"2024-04-06T12:31:22.102282+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:18:36.586968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.111641+0000","last_change":"2024-04-06T12:31:22.108873+0000","last_active":"2024-04-06T12:31:22.111641+0000","last_peered":"2024-04-06T12:31:22.111641+0000","last_clean":"2024-04-06T12:31:22.111641+0000","last_became_active":"2024-04-06T12:31:22.108637+0000","last_became_peered":"2024-04-06T12:31:22.108637+0000","last_unstale":"2024-04-06T12:31:22.111641+0000","last_undegraded":"2024-04-06T12:31:22.111641+0000","last_fullsized":"2024-04-06T12:31:22.111641+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:51:23.611053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"143'1","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099644+0000","last_change":"2024-04-06T12:31:21.692516+0000","last_active":"2024-04-06T12:31:23.099644+0000","last_peered":"2024-04-06T12:31:23.099644+0000","last_clean":"2024-04-06T12:31:23.099644+0000","last_became_active":"2024-04-06T12:31:21.692077+0000","last_became_peered":"2024-04-06T12:31:21.692077+0000","last_unstale":"2024-04-06T12:31:23.099644+0000","last_undegraded":"2024-04-06T12:31:23.099644+0000","last_fullsized":"2024-04-06T12:31:23.099644+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:02:34.205917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.098187+0000","last_change":"2024-04-06T12:31:22.103426+0000","last_active":"2024-04-06T12:31:23.098187+0000","last_peered":"2024-04-06T12:31:23.098187+0000","last_clean":"2024-04-06T12:31:23.098187+0000","last_became_active":"2024-04-06T12:31:22.103145+0000","last_became_peered":"2024-04-06T12:31:22.103145+0000","last_unstale":"2024-04-06T12:31:23.098187+0000","last_undegraded":"2024-04-06T12:31:23.098187+0000","last_fullsized":"2024-04-06T12:31:23.098187+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:21:27.019989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099516+0000","last_change":"2024-04-06T12:31:21.697577+0000","last_active":"2024-04-06T12:31:23.099516+0000","last_peered":"2024-04-06T12:31:23.099516+0000","last_clean":"2024-04-06T12:31:23.099516+0000","last_became_active":"2024-04-06T12:31:21.697245+0000","last_became_peered":"2024-04-06T12:31:21.697245+0000","last_unstale":"2024-04-06T12:31:23.099516+0000","last_undegraded":"2024-04-06T12:31:23.099516+0000","last_fullsized":"2024-04-06T12:31:23.099516+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:05:58.955969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.111672+0000","last_change":"2024-04-06T12:31:22.108729+0000","last_active":"2024-04-06T12:31:22.111672+0000","last_peered":"2024-04-06T12:31:22.111672+0000","last_clean":"2024-04-06T12:31:22.111672+0000","last_became_active":"2024-04-06T12:31:22.108453+0000","last_became_peered":"2024-04-06T12:31:22.108453+0000","last_unstale":"2024-04-06T12:31:22.111672+0000","last_undegraded":"2024-04-06T12:31:22.111672+0000","last_fullsized":"2024-04-06T12:31:22.111672+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:37:19.783545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"143'2","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.113099+0000","last_change":"2024-04-06T12:31:21.694840+0000","last_active":"2024-04-06T12:31:22.113099+0000","last_peered":"2024-04-06T12:31:22.113099+0000","last_clean":"2024-04-06T12:31:22.113099+0000","last_became_active":"2024-04-06T12:31:21.694553+0000","last_became_peered":"2024-04-06T12:31:21.694553+0000","last_unstale":"2024-04-06T12:31:22.113099+0000","last_undegraded":"2024-04-06T12:31:22.113099+0000","last_fullsized":"2024-04-06T12:31:22.113099+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:14:26.385594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119150+0000","last_change":"2024-04-06T12:31:22.112779+0000","last_active":"2024-04-06T12:31:22.119150+0000","last_peered":"2024-04-06T12:31:22.119150+0000","last_clean":"2024-04-06T12:31:22.119150+0000","last_became_active":"2024-04-06T12:31:22.112256+0000","last_became_peered":"2024-04-06T12:31:22.112256+0000","last_unstale":"2024-04-06T12:31:22.119150+0000","last_undegraded":"2024-04-06T12:31:22.119150+0000","last_fullsized":"2024-04-06T12:31:22.119150+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:51:52.795093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099282+0000","last_change":"2024-04-06T12:31:21.685503+0000","last_active":"2024-04-06T12:31:23.099282+0000","last_peered":"2024-04-06T12:31:23.099282+0000","last_clean":"2024-04-06T12:31:23.099282+0000","last_became_active":"2024-04-06T12:31:21.684600+0000","last_became_peered":"2024-04-06T12:31:21.684600+0000","last_unstale":"2024-04-06T12:31:23.099282+0000","last_undegraded":"2024-04-06T12:31:23.099282+0000","last_fullsized":"2024-04-06T12:31:23.099282+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:09:02.746979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119627+0000","last_change":"2024-04-06T12:31:22.117136+0000","last_active":"2024-04-06T12:31:22.119627+0000","last_peered":"2024-04-06T12:31:22.119627+0000","last_clean":"2024-04-06T12:31:22.119627+0000","last_became_active":"2024-04-06T12:31:22.115170+0000","last_became_peered":"2024-04-06T12:31:22.115170+0000","last_unstale":"2024-04-06T12:31:22.119627+0000","last_undegraded":"2024-04-06T12:31:22.119627+0000","last_fullsized":"2024-04-06T12:31:22.119627+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:14:56.389425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.097919+0000","last_change":"2024-04-06T12:31:21.692888+0000","last_active":"2024-04-06T12:31:22.097919+0000","last_peered":"2024-04-06T12:31:22.097919+0000","last_clean":"2024-04-06T12:31:22.097919+0000","last_became_active":"2024-04-06T12:31:21.692608+0000","last_became_peered":"2024-04-06T12:31:21.692608+0000","last_unstale":"2024-04-06T12:31:22.097919+0000","last_undegraded":"2024-04-06T12:31:22.097919+0000","last_fullsized":"2024-04-06T12:31:22.097919+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:11:38.123806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097702+0000","last_change":"2024-04-06T12:31:22.096039+0000","last_active":"2024-04-06T12:31:23.097702+0000","last_peered":"2024-04-06T12:31:23.097702+0000","last_clean":"2024-04-06T12:31:23.097702+0000","last_became_active":"2024-04-06T12:31:22.095741+0000","last_became_peered":"2024-04-06T12:31:22.095741+0000","last_unstale":"2024-04-06T12:31:23.097702+0000","last_undegraded":"2024-04-06T12:31:23.097702+0000","last_fullsized":"2024-04-06T12:31:23.097702+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:52.934051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.098885+0000","last_change":"2024-04-06T12:31:21.694530+0000","last_active":"2024-04-06T12:31:23.098885+0000","last_peered":"2024-04-06T12:31:23.098885+0000","last_clean":"2024-04-06T12:31:23.098885+0000","last_became_active":"2024-04-06T12:31:21.694267+0000","last_became_peered":"2024-04-06T12:31:21.694267+0000","last_unstale":"2024-04-06T12:31:23.098885+0000","last_undegraded":"2024-04-06T12:31:23.098885+0000","last_fullsized":"2024-04-06T12:31:23.098885+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:51.928752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.111706+0000","last_change":"2024-04-06T12:31:22.108820+0000","last_active":"2024-04-06T12:31:22.111706+0000","last_peered":"2024-04-06T12:31:22.111706+0000","last_clean":"2024-04-06T12:31:22.111706+0000","last_became_active":"2024-04-06T12:31:22.108557+0000","last_became_peered":"2024-04-06T12:31:22.108557+0000","last_unstale":"2024-04-06T12:31:22.111706+0000","last_undegraded":"2024-04-06T12:31:22.111706+0000","last_fullsized":"2024-04-06T12:31:22.111706+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:08:21.428907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"143'2","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.103787+0000","last_change":"2024-04-06T12:31:21.694543+0000","last_active":"2024-04-06T12:31:22.103787+0000","last_peered":"2024-04-06T12:31:22.103787+0000","last_clean":"2024-04-06T12:31:22.103787+0000","last_became_active":"2024-04-06T12:31:21.692046+0000","last_became_peered":"2024-04-06T12:31:21.692046+0000","last_unstale":"2024-04-06T12:31:22.103787+0000","last_undegraded":"2024-04-06T12:31:22.103787+0000","last_fullsized":"2024-04-06T12:31:22.103787+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:41:18.763861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.700074+0000","last_change":"2024-04-06T12:31:22.108770+0000","last_active":"2024-04-06T12:31:23.700074+0000","last_peered":"2024-04-06T12:31:23.700074+0000","last_clean":"2024-04-06T12:31:23.700074+0000","last_became_active":"2024-04-06T12:31:22.108521+0000","last_became_peered":"2024-04-06T12:31:22.108521+0000","last_unstale":"2024-04-06T12:31:23.700074+0000","last_undegraded":"2024-04-06T12:31:23.700074+0000","last_fullsized":"2024-04-06T12:31:23.700074+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:17.208811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097731+0000","last_change":"2024-04-06T12:31:21.697407+0000","last_active":"2024-04-06T12:31:23.097731+0000","last_peered":"2024-04-06T12:31:23.097731+0000","last_clean":"2024-04-06T12:31:23.097731+0000","last_became_active":"2024-04-06T12:31:21.697125+0000","last_became_peered":"2024-04-06T12:31:21.697125+0000","last_unstale":"2024-04-06T12:31:23.097731+0000","last_undegraded":"2024-04-06T12:31:23.097731+0000","last_fullsized":"2024-04-06T12:31:23.097731+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:19:41.824744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099667+0000","last_change":"2024-04-06T12:31:22.108787+0000","last_active":"2024-04-06T12:31:23.099667+0000","last_peered":"2024-04-06T12:31:23.099667+0000","last_clean":"2024-04-06T12:31:23.099667+0000","last_became_active":"2024-04-06T12:31:22.108532+0000","last_became_peered":"2024-04-06T12:31:22.108532+0000","last_unstale":"2024-04-06T12:31:23.099667+0000","last_undegraded":"2024-04-06T12:31:23.099667+0000","last_fullsized":"2024-04-06T12:31:23.099667+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:38:56.895638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.097911+0000","last_change":"2024-04-06T12:31:21.694921+0000","last_active":"2024-04-06T12:31:23.097911+0000","last_peered":"2024-04-06T12:31:23.097911+0000","last_clean":"2024-04-06T12:31:23.097911+0000","last_became_active":"2024-04-06T12:31:21.694615+0000","last_became_peered":"2024-04-06T12:31:21.694615+0000","last_unstale":"2024-04-06T12:31:23.097911+0000","last_undegraded":"2024-04-06T12:31:23.097911+0000","last_fullsized":"2024-04-06T12:31:23.097911+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:02:24.639229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.099925+0000","last_change":"2024-04-06T12:31:22.096553+0000","last_active":"2024-04-06T12:31:22.099925+0000","last_peered":"2024-04-06T12:31:22.099925+0000","last_clean":"2024-04-06T12:31:22.099925+0000","last_became_active":"2024-04-06T12:31:22.095236+0000","last_became_peered":"2024-04-06T12:31:22.095236+0000","last_unstale":"2024-04-06T12:31:22.099925+0000","last_undegraded":"2024-04-06T12:31:22.099925+0000","last_fullsized":"2024-04-06T12:31:22.099925+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:14:19.606564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.096910+0000","last_change":"2024-04-06T12:31:21.697287+0000","last_active":"2024-04-06T12:31:22.096910+0000","last_peered":"2024-04-06T12:31:22.096910+0000","last_clean":"2024-04-06T12:31:22.096910+0000","last_became_active":"2024-04-06T12:31:21.696979+0000","last_became_peered":"2024-04-06T12:31:21.696979+0000","last_unstale":"2024-04-06T12:31:22.096910+0000","last_undegraded":"2024-04-06T12:31:22.096910+0000","last_fullsized":"2024-04-06T12:31:22.096910+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:21:09.615951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.102779+0000","last_change":"2024-04-06T12:31:22.100612+0000","last_active":"2024-04-06T12:31:22.102779+0000","last_peered":"2024-04-06T12:31:22.102779+0000","last_clean":"2024-04-06T12:31:22.102779+0000","last_became_active":"2024-04-06T12:31:22.100355+0000","last_became_peered":"2024-04-06T12:31:22.100355+0000","last_unstale":"2024-04-06T12:31:22.102779+0000","last_undegraded":"2024-04-06T12:31:22.102779+0000","last_fullsized":"2024-04-06T12:31:22.102779+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:55:07.104225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.088805+0000","last_change":"2024-04-06T12:31:21.681396+0000","last_active":"2024-04-06T12:31:22.088805+0000","last_peered":"2024-04-06T12:31:22.088805+0000","last_clean":"2024-04-06T12:31:22.088805+0000","last_became_active":"2024-04-06T12:31:21.681133+0000","last_became_peered":"2024-04-06T12:31:21.681133+0000","last_unstale":"2024-04-06T12:31:22.088805+0000","last_undegraded":"2024-04-06T12:31:22.088805+0000","last_fullsized":"2024-04-06T12:31:22.088805+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:17:12.680025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119170+0000","last_change":"2024-04-06T12:31:22.114235+0000","last_active":"2024-04-06T12:31:22.119170+0000","last_peered":"2024-04-06T12:31:22.119170+0000","last_clean":"2024-04-06T12:31:22.119170+0000","last_became_active":"2024-04-06T12:31:22.112517+0000","last_became_peered":"2024-04-06T12:31:22.112517+0000","last_unstale":"2024-04-06T12:31:22.119170+0000","last_undegraded":"2024-04-06T12:31:22.119170+0000","last_fullsized":"2024-04-06T12:31:22.119170+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:01:34.580024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.088573+0000","last_change":"2024-04-06T12:31:21.689691+0000","last_active":"2024-04-06T12:31:22.088573+0000","last_peered":"2024-04-06T12:31:22.088573+0000","last_clean":"2024-04-06T12:31:22.088573+0000","last_became_active":"2024-04-06T12:31:21.689428+0000","last_became_peered":"2024-04-06T12:31:21.689428+0000","last_unstale":"2024-04-06T12:31:22.088573+0000","last_undegraded":"2024-04-06T12:31:22.088573+0000","last_fullsized":"2024-04-06T12:31:22.088573+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:09:02.746979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099162+0000","last_change":"2024-04-06T12:31:22.108269+0000","last_active":"2024-04-06T12:31:23.099162+0000","last_peered":"2024-04-06T12:31:23.099162+0000","last_clean":"2024-04-06T12:31:23.099162+0000","last_became_active":"2024-04-06T12:31:22.108017+0000","last_became_peered":"2024-04-06T12:31:22.108017+0000","last_unstale":"2024-04-06T12:31:23.099162+0000","last_undegraded":"2024-04-06T12:31:23.099162+0000","last_fullsized":"2024-04-06T12:31:23.099162+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:11:39.129105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.095731+0000","last_change":"2024-04-06T12:31:21.691758+0000","last_active":"2024-04-06T12:31:22.095731+0000","last_peered":"2024-04-06T12:31:22.095731+0000","last_clean":"2024-04-06T12:31:22.095731+0000","last_became_active":"2024-04-06T12:31:21.691462+0000","last_became_peered":"2024-04-06T12:31:21.691462+0000","last_unstale":"2024-04-06T12:31:22.095731+0000","last_undegraded":"2024-04-06T12:31:22.095731+0000","last_fullsized":"2024-04-06T12:31:22.095731+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:50:17.677522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.700036+0000","last_change":"2024-04-06T12:31:22.110654+0000","last_active":"2024-04-06T12:31:23.700036+0000","last_peered":"2024-04-06T12:31:23.700036+0000","last_clean":"2024-04-06T12:31:23.700036+0000","last_became_active":"2024-04-06T12:31:22.110189+0000","last_became_peered":"2024-04-06T12:31:22.110189+0000","last_unstale":"2024-04-06T12:31:23.700036+0000","last_undegraded":"2024-04-06T12:31:23.700036+0000","last_fullsized":"2024-04-06T12:31:23.700036+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:38:56.895638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.098919+0000","last_change":"2024-04-06T12:31:21.693170+0000","last_active":"2024-04-06T12:31:23.098919+0000","last_peered":"2024-04-06T12:31:23.098919+0000","last_clean":"2024-04-06T12:31:23.098919+0000","last_became_active":"2024-04-06T12:31:21.692910+0000","last_became_peered":"2024-04-06T12:31:21.692910+0000","last_unstale":"2024-04-06T12:31:23.098919+0000","last_undegraded":"2024-04-06T12:31:23.098919+0000","last_fullsized":"2024-04-06T12:31:23.098919+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:40.096055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-06T12:31:23.099370+0000","last_change":"2024-04-06T12:31:22.112180+0000","last_active":"2024-04-06T12:31:23.099370+0000","last_peered":"2024-04-06T12:31:23.099370+0000","last_clean":"2024-04-06T12:31:23.099370+0000","last_became_active":"2024-04-06T12:31:22.111672+0000","last_became_peered":"2024-04-06T12:31:22.111672+0000","last_unstale":"2024-04-06T12:31:23.099370+0000","last_undegraded":"2024-04-06T12:31:23.099370+0000","last_fullsized":"2024-04-06T12:31:23.099370+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:21.670494+0000","last_clean_scrub_stamp":"2024-04-06T12:31:21.670494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:58:51.171247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"143'8","reported_seq":18,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.119958+0000","last_change":"2024-04-06T12:31:21.689849+0000","last_active":"2024-04-06T12:31:22.119958+0000","last_peered":"2024-04-06T12:31:22.119958+0000","last_clean":"2024-04-06T12:31:22.119958+0000","last_became_active":"2024-04-06T12:31:21.689589+0000","last_became_peered":"2024-04-06T12:31:21.689589+0000","last_unstale":"2024-04-06T12:31:22.119958+0000","last_undegraded":"2024-04-06T12:31:22.119958+0000","last_fullsized":"2024-04-06T12:31:22.119958+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:31:20.665195+0000","last_clean_scrub_stamp":"2024-04-06T12:31:20.665195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:57:13.121538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"132'227","reported_seq":687,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-06T12:31:22.094977+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:31:22.094977+0000","last_peered":"2024-04-06T12:31:22.094977+0000","last_clean":"2024-04-06T12:31:22.094977+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:31:22.094977+0000","last_undegraded":"2024-04-06T12:31:22.094977+0000","last_fullsized":"2024-04-06T12:31:22.094977+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575078,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59700,"kb_used_data":2344,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312140,"statfs":{"total":96636764160,"available":96575631360,"internally_reserved":0,"allocated":2400256,"data_stored":1489714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53100000000000003,"15min":0.55300000000000005},"min":{"1min":0.33800000000000002,"5min":0.311,"15min":0.219},"max":{"1min":0.73099999999999998,"5min":0.80100000000000005,"15min":1.143},"last":0.249},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54000000000000004,"15min":0.55100000000000005},"min":{"1min":0.36399999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.72099999999999997,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.34100000000000003}]},{"osd":1,"last update":"Sat Apr 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54200000000000004,"15min":0.55500000000000005},"min":{"1min":0.39800000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71299999999999997,"5min":0.75,"15min":0.79100000000000004},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.55100000000000005,"15min":0.56799999999999995},"min":{"1min":0.433,"5min":0.36599999999999999,"15min":0.27900000000000003},"max":{"1min":0.754,"5min":0.77400000000000002,"15min":0.79600000000000004},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52700000000000002,"15min":0.55200000000000005},"min":{"1min":0.377,"5min":0.316,"15min":0.249},"max":{"1min":0.71199999999999997,"5min":0.77100000000000002,"15min":0.83299999999999996},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56699999999999995,"15min":0.57799999999999996},"min":{"1min":0.38800000000000001,"5min":0.36699999999999999,"15min":0.29599999999999999},"max":{"1min":0.78200000000000003,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.315}]},{"osd":3,"last update":"Sat Apr 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55300000000000005,"15min":0.57199999999999995},"min":{"1min":0.36299999999999999,"5min":0.35099999999999998,"15min":0.34399999999999997},"max":{"1min":0.68200000000000005,"5min":0.75800000000000001,"15min":1.357},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57499999999999996,"15min":0.58799999999999997},"min":{"1min":0.377,"5min":0.35599999999999998,"15min":0.33100000000000002},"max":{"1min":0.75800000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.39100000000000001}]},{"osd":4,"last update":"Sat Apr 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.438,"15min":0.443},"min":{"1min":0.29299999999999998,"5min":0.27000000000000002,"15min":0.25},"max":{"1min":0.59099999999999997,"5min":0.71399999999999997,"15min":0.72999999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.53000000000000003,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.35399999999999998,"15min":0.32800000000000001},"max":{"1min":0.62,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.378}]},{"osd":5,"last update":"Sat Apr 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.502,"15min":0.495},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.23799999999999999},"max":{"1min":0.73499999999999999,"5min":0.81899999999999995,"15min":0.86099999999999999},"last":0.432},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.33600000000000002,"5min":0.32800000000000001,"15min":0.314},"max":{"1min":0.73599999999999999,"5min":0.79900000000000004,"15min":0.81599999999999995},"last":0.42199999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.28799999999999998},"max":{"1min":0.73199999999999998,"5min":0.84499999999999997,"15min":0.88600000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.38500000000000001,"5min":0.312,"15min":0.30399999999999999},"max":{"1min":0.70999999999999996,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.44600000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575077,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32868,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338972,"statfs":{"total":96636764160,"available":96603107328,"internally_reserved":0,"allocated":1789952,"data_stored":898133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.52900000000000003,"15min":0.53900000000000003},"min":{"1min":0.37,"5min":0.34000000000000002,"15min":0.26200000000000001},"max":{"1min":0.628,"5min":0.77500000000000002,"15min":0.82099999999999995},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53200000000000003,"15min":0.53600000000000003},"min":{"1min":0.38700000000000001,"5min":0.35299999999999998,"15min":0.245},"max":{"1min":0.63100000000000001,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.70799999999999996}]},{"osd":1,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57099999999999995,"15min":0.58599999999999997},"min":{"1min":0.433,"5min":0.32700000000000001,"15min":0.24399999999999999},"max":{"1min":0.67900000000000005,"5min":0.78200000000000003,"15min":0.84599999999999997},"last":0.629},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59399999999999997,"15min":0.59999999999999998},"min":{"1min":0.41899999999999998,"5min":0.38400000000000001,"15min":0.221},"max":{"1min":0.68899999999999995,"5min":0.81999999999999995,"15min":0.83099999999999996},"last":0.61899999999999999}]},{"osd":2,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.55300000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.29899999999999999},"max":{"1min":0.66300000000000003,"5min":0.749,"15min":0.77700000000000002},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56200000000000006,"15min":0.56999999999999995},"min":{"1min":0.35699999999999998,"5min":0.32300000000000001,"15min":0.254},"max":{"1min":0.68500000000000005,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.58799999999999997}]},{"osd":3,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56999999999999995,"15min":0.57899999999999996},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.23300000000000001},"max":{"1min":0.66900000000000004,"5min":0.745,"15min":0.81200000000000006},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.61199999999999999,"15min":0.61899999999999999},"min":{"1min":0.442,"5min":0.39700000000000002,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.79200000000000004,"15min":0.872},"last":0.60099999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.51500000000000001,"15min":0.501},"min":{"1min":0.32600000000000001,"5min":0.28599999999999998,"15min":0.27500000000000002},"max":{"1min":0.73699999999999999,"5min":0.754,"15min":0.754},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53700000000000003,"15min":0.53200000000000003},"min":{"1min":0.32700000000000001,"5min":0.29599999999999999,"15min":0.28799999999999998},"max":{"1min":0.72799999999999998,"5min":0.73199999999999998,"15min":0.82199999999999995},"last":0.48699999999999999}]},{"osd":5,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.503,"15min":0.49399999999999999},"min":{"1min":0.37,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.70599999999999996,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51800000000000002,"15min":0.51200000000000001},"min":{"1min":0.316,"5min":0.31,"15min":0.27000000000000002},"max":{"1min":0.79600000000000004,"5min":0.80200000000000005,"15min":0.83699999999999997},"last":0.46999999999999997}]},{"osd":7,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54700000000000004,"15min":0.53600000000000003},"min":{"1min":0.33500000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.749,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57499999999999996,"15min":0.56000000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.32100000000000001},"max":{"1min":0.77300000000000002,"5min":0.84799999999999998,"15min":0.88900000000000001},"last":0.52300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575076,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32872,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338968,"statfs":{"total":96636764160,"available":96603103232,"internally_reserved":0,"allocated":1789952,"data_stored":897573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36499999999999999,"15min":0.35999999999999999},"min":{"1min":0.25,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.58999999999999997},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.375,"15min":0.39100000000000001},"min":{"1min":0.26500000000000001,"5min":0.245,"15min":0.215},"max":{"1min":0.59099999999999997,"5min":0.628,"15min":0.628},"last":0.38900000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38800000000000001,"15min":0.39300000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.19400000000000001},"max":{"1min":0.57599999999999996,"5min":0.61599999999999999,"15min":0.67600000000000005},"last":0.435},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.28799999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.42399999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.44700000000000001,"15min":0.44600000000000001},"min":{"1min":0.29999999999999999,"5min":0.29199999999999998,"15min":0.28399999999999997},"max":{"1min":0.66400000000000003,"5min":0.67500000000000004,"15min":0.67800000000000005},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41199999999999998,"15min":0.41799999999999998},"min":{"1min":0.32100000000000001,"5min":0.29099999999999998,"15min":0.249},"max":{"1min":0.60899999999999999,"5min":0.65900000000000003,"15min":0.86299999999999999},"last":0.46600000000000003}]},{"osd":4,"last update":"Sat Apr 6 12:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.61399999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.34799999999999998},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.623,"15min":0.621},"min":{"1min":0.45700000000000002,"5min":0.36299999999999999,"15min":0.30399999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.94899999999999995},"last":0.65300000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60899999999999999,"15min":0.61299999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.38700000000000001},"max":{"1min":0.80000000000000004,"5min":0.85899999999999999,"15min":0.91300000000000003},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.627,"5min":0.629,"15min":0.627},"min":{"1min":0.437,"5min":0.41499999999999998,"15min":0.38300000000000001},"max":{"1min":0.871,"5min":0.999,"15min":0.999},"last":0.77500000000000002}]},{"osd":6,"last update":"Sat Apr 6 12:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61499999999999999,"15min":0.61399999999999999},"min":{"1min":0.40699999999999997,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82599999999999996,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.753},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63300000000000001,"15min":0.64400000000000002},"min":{"1min":0.45900000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.84799999999999998,"5min":0.88700000000000001,"15min":0.92600000000000005},"last":0.73899999999999999}]},{"osd":7,"last update":"Sat Apr 6 12:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65000000000000002,"15min":0.63900000000000001},"min":{"1min":0.48999999999999999,"5min":0.44600000000000001,"15min":0.38800000000000001},"max":{"1min":0.85699999999999998,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.624,"15min":0.63300000000000001},"min":{"1min":0.42199999999999999,"5min":0.40999999999999998,"15min":0.40200000000000002},"max":{"1min":0.75700000000000001,"5min":0.91400000000000003,"15min":0.95599999999999996},"last":0.64200000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575076,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33444,"kb_used_data":2316,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338396,"statfs":{"total":96636764160,"available":96602517504,"internally_reserved":0,"allocated":2371584,"data_stored":1481321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.37,"15min":0.38200000000000001},"min":{"1min":0.22,"5min":0.216,"15min":0.19},"max":{"1min":0.52500000000000002,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.38600000000000001,"15min":0.40500000000000003},"min":{"1min":0.26700000000000002,"5min":0.23499999999999999,"15min":0.217},"max":{"1min":0.61299999999999999,"5min":0.625,"15min":1.4359999999999999},"last":0.26000000000000001}]},{"osd":2,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.39400000000000002,"15min":0.40699999999999997},"min":{"1min":0.26800000000000002,"5min":0.255,"15min":0.19700000000000001},"max":{"1min":0.58399999999999996,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41799999999999998,"15min":0.432},"min":{"1min":0.29299999999999998,"5min":0.26800000000000002,"15min":0.24299999999999999},"max":{"1min":0.59399999999999997,"5min":0.89900000000000002,"15min":1.4790000000000001},"last":0.376}]},{"osd":3,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.41199999999999998,"15min":0.41899999999999998},"min":{"1min":0.29899999999999999,"5min":0.28799999999999998,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":1.25},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.443,"15min":0.46000000000000002},"min":{"1min":0.29599999999999999,"5min":0.255,"15min":0.255},"max":{"1min":0.70999999999999996,"5min":0.74099999999999999,"15min":0.748},"last":0.40400000000000003}]},{"osd":4,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60199999999999998,"15min":0.624},"min":{"1min":0.41299999999999998,"5min":0.38400000000000001,"15min":0.35699999999999998},"max":{"1min":0.84799999999999998,"5min":1.1100000000000001,"15min":1.514},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60799999999999998,"15min":0.629},"min":{"1min":0.41299999999999998,"5min":0.36699999999999999,"15min":0.35199999999999998},"max":{"1min":0.84599999999999997,"5min":0.93999999999999995,"15min":1.202},"last":0.64300000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.59799999999999998,"15min":0.624},"min":{"1min":0.44400000000000001,"5min":0.34399999999999997,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.88400000000000001,"15min":1.0920000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.63,"15min":0.64600000000000002},"min":{"1min":0.42499999999999999,"5min":0.40899999999999997,"15min":0.315},"max":{"1min":0.83599999999999997,"5min":1.006,"15min":1.006},"last":0.55800000000000005}]},{"osd":6,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.627,"15min":0.64800000000000002},"min":{"1min":0.42999999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.63200000000000001,"15min":0.66200000000000003},"min":{"1min":0.47799999999999998,"5min":0.40300000000000002,"15min":0.38100000000000001},"max":{"1min":0.92000000000000004,"5min":1.052,"15min":1.613},"last":0.63300000000000001}]},{"osd":7,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63400000000000001,"15min":0.66300000000000003},"min":{"1min":0.47399999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88200000000000001,"5min":1.0820000000000001,"15min":1.54},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.63600000000000001,"15min":0.65900000000000003},"min":{"1min":0.49099999999999999,"5min":0.39300000000000002,"15min":0.375},"max":{"1min":0.90800000000000003,"5min":1.014,"15min":1.498},"last":0.53600000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575078,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55000,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316840,"statfs":{"total":96636764160,"available":96580444160,"internally_reserved":0,"allocated":1773568,"data_stored":890931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.372,"15min":0.38200000000000001},"min":{"1min":0.254,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.53400000000000003,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40699999999999997,"15min":0.41099999999999998},"min":{"1min":0.27300000000000002,"5min":0.246,"15min":0.20000000000000001},"max":{"1min":0.56999999999999995,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.27900000000000003}]},{"osd":1,"last update":"Sat Apr 6 12:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.38500000000000001,"15min":0.39900000000000002},"min":{"1min":0.25600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.45000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41699999999999998,"15min":0.43099999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.23599999999999999},"max":{"1min":0.66800000000000004,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.35599999999999998}]},{"osd":3,"last update":"Sat Apr 6 12:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.44,"15min":0.45000000000000001},"min":{"1min":0.30099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.86899999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.434,"15min":0.438},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.29099999999999998},"max":{"1min":0.64900000000000002,"5min":0.97099999999999997,"15min":1.0549999999999999},"last":0.26500000000000001}]},{"osd":4,"last update":"Sat Apr 6 12:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60299999999999998,"15min":0.61399999999999999},"min":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.35199999999999998},"max":{"1min":0.79500000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.623},"min":{"1min":0.45600000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.79700000000000004,"5min":0.91200000000000003,"15min":0.92600000000000005},"last":0.51700000000000002}]},{"osd":5,"last update":"Sat Apr 6 12:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61899999999999999,"15min":0.63500000000000001},"min":{"1min":0.45600000000000002,"5min":0.374,"15min":0.36499999999999999},"max":{"1min":0.84999999999999998,"5min":1.0369999999999999,"15min":1.6870000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61699999999999999,"15min":0.627},"min":{"1min":0.46600000000000003,"5min":0.35399999999999998,"15min":0.31},"max":{"1min":0.80500000000000005,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.49099999999999999}]},{"osd":6,"last update":"Sat Apr 6 12:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.63200000000000001},"min":{"1min":0.52600000000000002,"5min":0.41799999999999998,"15min":0.36399999999999999},"max":{"1min":0.91000000000000003,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.623,"15min":0.65000000000000002},"min":{"1min":0.44600000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81899999999999995,"5min":1.0760000000000001,"15min":1.728},"last":0.55600000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.621,"15min":0.64400000000000002},"min":{"1min":0.443,"5min":0.40699999999999997,"15min":0.35599999999999998},"max":{"1min":0.85299999999999998,"5min":0.95699999999999996,"15min":0.95799999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.64000000000000001,"15min":0.66000000000000003},"min":{"1min":0.42299999999999999,"5min":0.41499999999999998,"15min":0.39000000000000001},"max":{"1min":0.88300000000000001,"5min":1.0049999999999999,"15min":1.758},"last":0.53000000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575077,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32876,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338964,"statfs":{"total":96636764160,"available":96603099136,"internally_reserved":0,"allocated":1794048,"data_stored":893548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.247,"5min":0.23599999999999999,"15min":0.191},"max":{"1min":0.52900000000000003,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.36399999999999999,"15min":0.38200000000000001},"min":{"1min":0.245,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.73499999999999999},"last":0.26800000000000002}]},{"osd":1,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.376,"15min":0.38},"min":{"1min":0.222,"5min":0.222,"15min":0.217},"max":{"1min":0.49399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.318},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.40300000000000002,"15min":0.41199999999999998},"min":{"1min":0.26800000000000002,"5min":0.26100000000000001,"15min":0.25700000000000001},"max":{"1min":0.54700000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.28999999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.44700000000000001,"15min":0.45800000000000002},"min":{"1min":0.28999999999999998,"5min":0.28799999999999998,"15min":0.27200000000000002},"max":{"1min":0.56799999999999995,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.46500000000000002,"15min":0.47399999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.59299999999999997,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.371}]},{"osd":4,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57699999999999996,"15min":0.60199999999999998},"min":{"1min":0.41199999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.95999999999999996},"last":0.505},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.62},"min":{"1min":0.42299999999999999,"5min":0.39600000000000002,"15min":0.23699999999999999},"max":{"1min":0.80800000000000005,"5min":0.90100000000000002,"15min":0.93500000000000005},"last":0.60899999999999999}]},{"osd":5,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.623,"15min":0.65200000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.309},"max":{"1min":0.70999999999999996,"5min":0.92200000000000004,"15min":0.96699999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63600000000000001,"15min":0.65900000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.38200000000000001},"max":{"1min":0.93899999999999995,"5min":0.94499999999999995,"15min":0.95599999999999996},"last":0.56999999999999995}]},{"osd":6,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59899999999999998,"15min":0.61799999999999999},"min":{"1min":0.41499999999999998,"5min":0.38,"15min":0.36899999999999999},"max":{"1min":0.83799999999999997,"5min":1.026,"15min":1.026},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.64400000000000002,"15min":0.65500000000000003},"min":{"1min":0.45400000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.81399999999999995,"5min":1.716,"15min":1.716},"last":0.495}]},{"osd":7,"last update":"Sat Apr 6 12:30:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61899999999999999,"15min":0.63900000000000001},"min":{"1min":0.39800000000000002,"5min":0.377,"15min":0.33300000000000002},"max":{"1min":0.79500000000000004,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.64000000000000001,"15min":0.66000000000000003},"min":{"1min":0.442,"5min":0.40100000000000002,"15min":0.373},"max":{"1min":0.92500000000000004,"5min":1.002,"15min":1.002},"last":0.58799999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575077,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32872,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338968,"statfs":{"total":96636764160,"available":96603103232,"internally_reserved":0,"allocated":1789952,"data_stored":893514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55900000000000005,"15min":0.56200000000000006},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.23899999999999999},"max":{"1min":0.83399999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.375,"5min":0.33800000000000002,"15min":0.25900000000000001},"max":{"1min":0.79400000000000004,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.56399999999999995}]},{"osd":1,"last update":"Sat Apr 6 12:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59599999999999997,"15min":0.60799999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.29799999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58999999999999997,"15min":0.59299999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.57499999999999996}]},{"osd":2,"last update":"Sat Apr 6 12:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.57099999999999995,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.78100000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.495},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57499999999999996,"15min":0.58299999999999996},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.80900000000000005},"last":0.53400000000000003}]},{"osd":3,"last update":"Sat Apr 6 12:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56899999999999995,"15min":0.56299999999999994},"min":{"1min":0.39700000000000002,"5min":0.373,"15min":0.36399999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.54600000000000004}]},{"osd":5,"last update":"Sat Apr 6 12:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.47599999999999998,"15min":0.47899999999999998},"min":{"1min":0.33100000000000002,"5min":0.308,"15min":0.27200000000000002},"max":{"1min":0.67600000000000005,"5min":0.73599999999999999,"15min":0.83599999999999997},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48599999999999999,"15min":0.496},"min":{"1min":0.33500000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.68899999999999995,"5min":0.76600000000000001,"15min":0.77000000000000002},"last":0.37}]},{"osd":6,"last update":"Sat Apr 6 12:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.503,"15min":0.502},"min":{"1min":0.32400000000000001,"5min":0.307,"15min":0.30599999999999999},"max":{"1min":0.753,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38800000000000001,"5min":0.34100000000000003,"15min":0.32100000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.41099999999999998}]},{"osd":7,"last update":"Sat Apr 6 12:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53500000000000003,"15min":0.53400000000000003},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72199999999999998,"5min":0.77600000000000002,"15min":0.82299999999999995},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.38700000000000001,"5min":0.36699999999999999,"15min":0.35599999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.39100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575077,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50920,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320920,"statfs":{"total":96636764160,"available":96584622080,"internally_reserved":0,"allocated":1794048,"data_stored":897711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54600000000000004,"15min":0.57099999999999995},"min":{"1min":0.371,"5min":0.32700000000000001,"15min":0.29099999999999998},"max":{"1min":0.68799999999999994,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.52400000000000002,"15min":0.54600000000000004},"min":{"1min":0.36699999999999999,"5min":0.29499999999999998,"15min":0.22500000000000001},"max":{"1min":0.63500000000000001,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.443}]},{"osd":1,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53100000000000003,"15min":0.55500000000000005},"min":{"1min":0.371,"5min":0.30399999999999999,"15min":0.26000000000000001},"max":{"1min":0.71199999999999997,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.53200000000000003,"15min":0.55300000000000005},"min":{"1min":0.39800000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.63600000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.42799999999999999}]},{"osd":2,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54300000000000004,"15min":0.56399999999999995},"min":{"1min":0.376,"5min":0.33200000000000002,"15min":0.21099999999999999},"max":{"1min":0.69199999999999995,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.56599999999999995},"min":{"1min":0.40600000000000003,"5min":0.35999999999999999,"15min":0.27400000000000002},"max":{"1min":0.68600000000000005,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.48599999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55600000000000005,"15min":0.56999999999999995},"min":{"1min":0.376,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75600000000000001,"5min":0.874,"15min":0.874},"last":0.503},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56799999999999995,"15min":0.59099999999999997},"min":{"1min":0.38700000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.65600000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55000000000000004}]},{"osd":4,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46899999999999997,"15min":0.46800000000000003},"min":{"1min":0.34100000000000003,"5min":0.28899999999999998,"15min":0.28299999999999997},"max":{"1min":0.66800000000000004,"5min":0.755,"15min":0.81200000000000006},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54500000000000004,"15min":0.53200000000000003},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.57999999999999996}]},{"osd":6,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48399999999999999,"15min":0.47699999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.28000000000000003},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.48999999999999999,"15min":0.48799999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.29699999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.56799999999999995}]},{"osd":7,"last update":"Sat Apr 6 12:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.501,"15min":0.502},"min":{"1min":0.33600000000000002,"5min":0.29799999999999999,"15min":0.26600000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.42399999999999999,"5min":0.40200000000000002,"15min":0.35199999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.61199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:31:26.606 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-06T12:31:26.606 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-06T12:31:26.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.772+0000 7fd098e9b700 1 Processor -- start 2024-04-06T12:31:26.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.772+0000 7fd098e9b700 1 -- start start 2024-04-06T12:31:26.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.773+0000 7fd098e9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd09415a550 0x7fd0940cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:26.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.773+0000 7fd098e9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd0940cf900 0x7fd0940b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:26.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.773+0000 7fd098e9b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0940cdbe0 con 0x7fd0940b4ba0 2024-04-06T12:31:26.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.773+0000 7fd098e9b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0940b5010 con 0x7fd0940cf900 2024-04-06T12:31:26.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.773+0000 7fd098e9b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09415a940 con 0x7fd09415a550 2024-04-06T12:31:26.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.773+0000 7fd091d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd0940cf900 0x7fd0940b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:26.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.773+0000 7fd091d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd0940cf900 0x7fd0940b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50974/0 (socket says 172.21.15.189:50974) 2024-04-06T12:31:26.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.773+0000 7fd091d9b700 1 -- 172.21.15.189:0/3289293976 learned_addr learned my addr 172.21.15.189:0/3289293976 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:26.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.773+0000 7fd09259c700 1 --2- 172.21.15.189:0/3289293976 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd09415a550 0x7fd0940cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:26.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.773+0000 7fd091d9b700 1 -- 172.21.15.189:0/3289293976 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd09415a550 msgr2=0x7fd0940cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:26.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.773+0000 7fd091d9b700 1 --2- 172.21.15.189:0/3289293976 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd09415a550 0x7fd0940cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:26.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.773+0000 7fd091d9b700 1 -- 172.21.15.189:0/3289293976 >> v1:172.21.15.192:6789/0 conn(0x7fd0940b4ba0 legacy=0x7fd09406eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:26.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.773+0000 7fd091d9b700 1 -- 172.21.15.189:0/3289293976 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09415ac00 con 0x7fd0940cf900 2024-04-06T12:31:26.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.773+0000 7fd09259c700 1 --2- 172.21.15.189:0/3289293976 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd09415a550 0x7fd0940cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:26.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.773+0000 7fd091d9b700 1 --2- 172.21.15.189:0/3289293976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0940cf900 0x7fd0940b4660 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fd0880098f0 tx=0x7fd088009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d0bd6e0f4619f7c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:26.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.773+0000 7fd09159a700 1 -- 172.21.15.189:0/3289293976 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd088004230 con 0x7fd0940cf900 2024-04-06T12:31:26.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.774+0000 7fd09159a700 1 -- 172.21.15.189:0/3289293976 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd088004390 con 0x7fd0940cf900 2024-04-06T12:31:26.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.774+0000 7fd09159a700 1 -- 172.21.15.189:0/3289293976 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0880036a0 con 0x7fd0940cf900 2024-04-06T12:31:26.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.774+0000 7fd098e9b700 1 -- 172.21.15.189:0/3289293976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0940cf900 msgr2=0x7fd0940b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:26.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.774+0000 7fd098e9b700 1 --2- 172.21.15.189:0/3289293976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0940cf900 0x7fd0940b4660 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fd0880098f0 tx=0x7fd088009c00 comp rx=0 tx=0).stop 2024-04-06T12:31:26.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.774+0000 7fd098e9b700 1 -- 172.21.15.189:0/3289293976 shutdown_connections 2024-04-06T12:31:26.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.774+0000 7fd098e9b700 1 --2- 172.21.15.189:0/3289293976 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd09415a550 0x7fd0940cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:26.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.774+0000 7fd098e9b700 1 --2- 172.21.15.189:0/3289293976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0940cf900 0x7fd0940b4660 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:26.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.774+0000 7fd098e9b700 1 -- 172.21.15.189:0/3289293976 >> 172.21.15.189:0/3289293976 conn(0x7fd0940d00b0 msgr2=0x7fd0940d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:26.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.774+0000 7fd098e9b700 1 -- 172.21.15.189:0/3289293976 shutdown_connections 2024-04-06T12:31:26.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.774+0000 7fd098e9b700 1 -- 172.21.15.189:0/3289293976 wait complete. 2024-04-06T12:31:26.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.775+0000 7fd098e9b700 1 Processor -- start 2024-04-06T12:31:26.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.775+0000 7fd098e9b700 1 -- start start 2024-04-06T12:31:26.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.776+0000 7fd098e9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0940b4ba0 0x7fd0940b1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:26.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.776+0000 7fd098e9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd0940cf900 0x7fd09414f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:26.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.776+0000 7fd098e9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd09414fe80 0x7fd0940b3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:26.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.776+0000 7fd098e9b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0940af2d0 con 0x7fd0940b4ba0 2024-04-06T12:31:26.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.776+0000 7fd098e9b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0941555d0 con 0x7fd09414fe80 2024-04-06T12:31:26.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.776+0000 7fd098e9b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0941558b0 con 0x7fd0940cf900 2024-04-06T12:31:26.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.776+0000 7fd09259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0940b4ba0 0x7fd0940b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:26.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.776+0000 7fd09259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0940b4ba0 0x7fd0940b1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50984/0 (socket says 172.21.15.189:50984) 2024-04-06T12:31:26.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.776+0000 7fd09259c700 1 -- 172.21.15.189:0/2042659347 learned_addr learned my addr 172.21.15.189:0/2042659347 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:26.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.776+0000 7fd091d9b700 1 --2- 172.21.15.189:0/2042659347 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd0940cf900 0x7fd09414f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:26.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.777+0000 7fd092d9d700 1 --2- 172.21.15.189:0/2042659347 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd09414fe80 0x7fd0940b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:26.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.777+0000 7fd09259c700 1 -- 172.21.15.189:0/2042659347 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd0940cf900 msgr2=0x7fd09414f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:26.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.777+0000 7fd09259c700 1 --2- 172.21.15.189:0/2042659347 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd0940cf900 0x7fd09414f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:26.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.777+0000 7fd09259c700 1 -- 172.21.15.189:0/2042659347 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd09414fe80 msgr2=0x7fd0940b3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:26.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.777+0000 7fd09259c700 1 --2- 172.21.15.189:0/2042659347 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd09414fe80 0x7fd0940b3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:26.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.777+0000 7fd09259c700 1 -- 172.21.15.189:0/2042659347 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd088009550 con 0x7fd0940b4ba0 2024-04-06T12:31:26.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.777+0000 7fd092d9d700 1 --2- 172.21.15.189:0/2042659347 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd09414fe80 0x7fd0940b3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:26.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.777+0000 7fd09259c700 1 --2- 172.21.15.189:0/2042659347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0940b4ba0 0x7fd0940b1630 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fd08400c820 tx=0x7fd08400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:26.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.777+0000 7fd082ffd700 1 -- 172.21.15.189:0/2042659347 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd084011800 con 0x7fd0940b4ba0 2024-04-06T12:31:26.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.777+0000 7fd082ffd700 1 -- 172.21.15.189:0/2042659347 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd084011960 con 0x7fd0940b4ba0 2024-04-06T12:31:26.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.777+0000 7fd098e9b700 1 -- 172.21.15.189:0/2042659347 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0940b0420 con 0x7fd0940b4ba0 2024-04-06T12:31:26.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.777+0000 7fd082ffd700 1 -- 172.21.15.189:0/2042659347 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd084011d10 con 0x7fd0940b4ba0 2024-04-06T12:31:26.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.777+0000 7fd098e9b700 1 -- 172.21.15.189:0/2042659347 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0940b09e0 con 0x7fd0940b4ba0 2024-04-06T12:31:26.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.778+0000 7fd098e9b700 1 -- 172.21.15.189:0/2042659347 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd094067bb0 con 0x7fd0940b4ba0 2024-04-06T12:31:26.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.778+0000 7fd082ffd700 1 -- 172.21.15.189:0/2042659347 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fd084018360 con 0x7fd0940b4ba0 2024-04-06T12:31:26.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.779+0000 7fd082ffd700 1 --2- 172.21.15.189:0/2042659347 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd07406adb0 0x7fd07406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:26.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.779+0000 7fd082ffd700 1 -- 172.21.15.189:0/2042659347 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd084089690 con 0x7fd0940b4ba0 2024-04-06T12:31:26.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.779+0000 7fd091d9b700 1 --2- 172.21.15.189:0/2042659347 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd07406adb0 0x7fd07406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:26.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.779+0000 7fd091d9b700 1 --2- 172.21.15.189:0/2042659347 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd07406adb0 0x7fd07406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd088009130 tx=0x7fd088008030 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:26.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:26.783+0000 7fd082ffd700 1 -- 172.21.15.189:0/2042659347 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd084054510 con 0x7fd0940b4ba0 2024-04-06T12:31:27.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.054+0000 7fd098e9b700 1 -- 172.21.15.189:0/2042659347 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd094074990 con 0x7fd0940b4ba0 2024-04-06T12:31:27.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.058+0000 7fd082ffd700 1 -- 172.21.15.189:0/2042659347 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+0 (secure 0 0 0) 0x7fd084057b30 con 0x7fd0940b4ba0 2024-04-06T12:31:27.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.061+0000 7fd098e9b700 1 -- 172.21.15.189:0/2042659347 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd07406adb0 msgr2=0x7fd07406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:27.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.061+0000 7fd098e9b700 1 --2- 172.21.15.189:0/2042659347 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd07406adb0 0x7fd07406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd088009130 tx=0x7fd088008030 comp rx=0 tx=0).stop 2024-04-06T12:31:27.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.061+0000 7fd098e9b700 1 -- 172.21.15.189:0/2042659347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0940b4ba0 msgr2=0x7fd0940b1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:27.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.061+0000 7fd098e9b700 1 --2- 172.21.15.189:0/2042659347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0940b4ba0 0x7fd0940b1630 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fd08400c820 tx=0x7fd08400cb30 comp rx=0 tx=0).stop 2024-04-06T12:31:27.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.061+0000 7fd098e9b700 1 -- 172.21.15.189:0/2042659347 shutdown_connections 2024-04-06T12:31:27.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.061+0000 7fd098e9b700 1 --2- 172.21.15.189:0/2042659347 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd07406adb0 0x7fd07406d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:27.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.061+0000 7fd098e9b700 1 --2- 172.21.15.189:0/2042659347 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd09414fe80 0x7fd0940b3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:27.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.061+0000 7fd098e9b700 1 --2- 172.21.15.189:0/2042659347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0940b4ba0 0x7fd0940b1630 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:27.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.061+0000 7fd098e9b700 1 --2- 172.21.15.189:0/2042659347 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd0940cf900 0x7fd09414f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:27.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.061+0000 7fd098e9b700 1 -- 172.21.15.189:0/2042659347 >> 172.21.15.189:0/2042659347 conn(0x7fd0940d00b0 msgr2=0x7fd09406d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:27.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.061+0000 7fd098e9b700 1 -- 172.21.15.189:0/2042659347 shutdown_connections 2024-04-06T12:31:27.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.061+0000 7fd098e9b700 1 -- 172.21.15.189:0/2042659347 wait complete. 2024-04-06T12:31:27.060 INFO:teuthology.orchestra.run.smithi189.stderr:updated caps for client.0 2024-04-06T12:31:27.071 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:31:27.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.194+0000 7fd0235d2700 1 Processor -- start 2024-04-06T12:31:27.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.195+0000 7fd0235d2700 1 -- start start 2024-04-06T12:31:27.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.195+0000 7fd0235d2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd01c0d17a0 0x7fd01c06b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:27.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.195+0000 7fd0235d2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd01c0cedf0 0x7fd01c0cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:27.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.195+0000 7fd0235d2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01c0cf760 con 0x7fd01c146dd0 2024-04-06T12:31:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.195+0000 7fd0235d2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd01c06d850 con 0x7fd01c0cedf0 2024-04-06T12:31:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.195+0000 7fd0235d2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c06c0a0 con 0x7fd01c0d17a0 2024-04-06T12:31:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd020b6d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd01c146dd0 0x7fd01c06d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48204/0 (socket says 172.21.15.189:48204) 2024-04-06T12:31:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd020b6d700 1 -- 172.21.15.189:0/292691646 learned_addr learned my addr 172.21.15.189:0/292691646 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd02136e700 1 --2- 172.21.15.189:0/292691646 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd01c0d17a0 0x7fd01c06b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd021b6f700 1 --2- 172.21.15.189:0/292691646 >> v2:172.21.15.192:3300/0 conn(0x7fd01c0cedf0 0x7fd01c0cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd013fff700 1 -- 172.21.15.189:0/292691646 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039277587 0 0) 0x7fd01c0cf760 con 0x7fd01c146dd0 2024-04-06T12:31:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd013fff700 1 -- 172.21.15.189:0/292691646 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0080035b0 con 0x7fd01c146dd0 2024-04-06T12:31:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd013fff700 1 -- 172.21.15.189:0/292691646 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 975894766 0 0) 0x7fd0080035b0 con 0x7fd01c146dd0 2024-04-06T12:31:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd013fff700 1 -- 172.21.15.189:0/292691646 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd01c0d17a0 msgr2=0x7fd01c06b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd013fff700 1 --2- 172.21.15.189:0/292691646 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd01c0d17a0 0x7fd01c06b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd013fff700 1 -- 172.21.15.189:0/292691646 >> v2:172.21.15.192:3300/0 conn(0x7fd01c0cedf0 msgr2=0x7fd01c0cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd013fff700 1 --2- 172.21.15.189:0/292691646 >> v2:172.21.15.192:3300/0 conn(0x7fd01c0cedf0 0x7fd01c0cf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd013fff700 1 -- 172.21.15.189:0/292691646 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01c06c380 con 0x7fd01c146dd0 2024-04-06T12:31:27.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd013fff700 1 -- 172.21.15.189:0/292691646 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fd00c001670 con 0x7fd01c146dd0 2024-04-06T12:31:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd013fff700 1 -- 172.21.15.189:0/292691646 >> v1:172.21.15.189:6789/0 conn(0x7fd01c146dd0 legacy=0x7fd01c06d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd013fff700 1 --2- 172.21.15.189:0/292691646 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd01c0d17a0 0x7fd008003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd013fff700 1 --2- 172.21.15.189:0/292691646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd008003f80 0x7fd008004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd013fff700 1 --2- 172.21.15.189:0/292691646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd008004d90 0x7fd008005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd013fff700 1 -- 172.21.15.189:0/292691646 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c06d850 con 0x7fd008003f80 2024-04-06T12:31:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd013fff700 1 -- 172.21.15.189:0/292691646 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c06c0a0 con 0x7fd008004d90 2024-04-06T12:31:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd013fff700 1 -- 172.21.15.189:0/292691646 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd00800ba10 con 0x7fd01c0d17a0 2024-04-06T12:31:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd020b6d700 1 --2- 172.21.15.189:0/292691646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd008003f80 0x7fd008004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd02136e700 1 --2- 172.21.15.189:0/292691646 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd01c0d17a0 0x7fd008003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.196+0000 7fd021b6f700 1 --2- 172.21.15.189:0/292691646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd008004d90 0x7fd008005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.197+0000 7fd020b6d700 1 -- 172.21.15.189:0/292691646 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd01c0d17a0 msgr2=0x7fd008003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.197+0000 7fd020b6d700 1 --2- 172.21.15.189:0/292691646 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd01c0d17a0 0x7fd008003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.197+0000 7fd020b6d700 1 -- 172.21.15.189:0/292691646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd008004d90 msgr2=0x7fd008005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.197+0000 7fd020b6d700 1 --2- 172.21.15.189:0/292691646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd008004d90 0x7fd008005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:27.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.197+0000 7fd020b6d700 1 -- 172.21.15.189:0/292691646 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0080035b0 con 0x7fd008003f80 2024-04-06T12:31:27.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.197+0000 7fd020b6d700 1 --2- 172.21.15.189:0/292691646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd008003f80 0x7fd008004850 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fd00c00c320 tx=0x7fd008005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca9d6c231fdfac7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:27.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.197+0000 7fd013fff700 1 -- 172.21.15.189:0/292691646 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c00ce00 con 0x7fd008003f80 2024-04-06T12:31:27.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.197+0000 7fd013fff700 1 -- 172.21.15.189:0/292691646 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00c005960 con 0x7fd008003f80 2024-04-06T12:31:27.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.197+0000 7fd013fff700 1 -- 172.21.15.189:0/292691646 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c005d10 con 0x7fd008003f80 2024-04-06T12:31:27.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.198+0000 7fd0235d2700 1 -- 172.21.15.189:0/292691646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd008003f80 msgr2=0x7fd008004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:27.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.198+0000 7fd0235d2700 1 --2- 172.21.15.189:0/292691646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd008003f80 0x7fd008004850 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fd00c00c320 tx=0x7fd008005880 comp rx=0 tx=0).stop 2024-04-06T12:31:27.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.198+0000 7fd021b6f700 1 -- 172.21.15.189:0/292691646 reap_dead start 2024-04-06T12:31:27.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.198+0000 7fd0235d2700 1 -- 172.21.15.189:0/292691646 shutdown_connections 2024-04-06T12:31:27.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.198+0000 7fd0235d2700 1 -- 172.21.15.189:0/292691646 >> 172.21.15.189:0/292691646 conn(0x7fd01c0b2db0 msgr2=0x7fd01c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:27.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.198+0000 7fd0235d2700 1 -- 172.21.15.189:0/292691646 shutdown_connections 2024-04-06T12:31:27.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.198+0000 7fd0235d2700 1 -- 172.21.15.189:0/292691646 wait complete. 2024-04-06T12:31:27.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.199+0000 7fd0235d2700 1 Processor -- start 2024-04-06T12:31:27.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.199+0000 7fd0235d2700 1 -- start start 2024-04-06T12:31:27.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd0235d2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd008003f80 0x7fd01c0c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:27.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd0235d2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd01c0c4d50 0x7fd01c0c38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:27.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd0235d2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd01c159de0 0x7fd01c15a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:27.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd0235d2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c15a7a0 con 0x7fd01c0c4d50 2024-04-06T12:31:27.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd0235d2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c0ce670 con 0x7fd008003f80 2024-04-06T12:31:27.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd0235d2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c0c6b00 con 0x7fd01c159de0 2024-04-06T12:31:27.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd020b6d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd01c0c4d50 0x7fd01c0c38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:27.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd020b6d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd01c0c4d50 0x7fd01c0c38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51006/0 (socket says 172.21.15.189:51006) 2024-04-06T12:31:27.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd020b6d700 1 -- 172.21.15.189:0/3654153881 learned_addr learned my addr 172.21.15.189:0/3654153881 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:27.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd02136e700 1 --2- 172.21.15.189:0/3654153881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd008003f80 0x7fd01c0c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:27.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd021b6f700 1 --2- 172.21.15.189:0/3654153881 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd01c159de0 0x7fd01c15a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:27.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd020b6d700 1 -- 172.21.15.189:0/3654153881 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd01c159de0 msgr2=0x7fd01c15a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:27.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd020b6d700 1 --2- 172.21.15.189:0/3654153881 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd01c159de0 0x7fd01c15a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:27.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd020b6d700 1 -- 172.21.15.189:0/3654153881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd008003f80 msgr2=0x7fd01c0c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:27.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd020b6d700 1 --2- 172.21.15.189:0/3654153881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd008003f80 0x7fd01c0c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:27.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd020b6d700 1 -- 172.21.15.189:0/3654153881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01c0c6de0 con 0x7fd01c0c4d50 2024-04-06T12:31:27.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd020b6d700 1 --2- 172.21.15.189:0/3654153881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd01c0c4d50 0x7fd01c0c38f0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fd00c000c00 tx=0x7fd00c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:27.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd011ffb700 1 -- 172.21.15.189:0/3654153881 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c00ce50 con 0x7fd01c0c4d50 2024-04-06T12:31:27.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd011ffb700 1 -- 172.21.15.189:0/3654153881 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00c004a80 con 0x7fd01c0c4d50 2024-04-06T12:31:27.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd0235d2700 1 -- 172.21.15.189:0/3654153881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd01c0c70c0 con 0x7fd01c0c4d50 2024-04-06T12:31:27.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.200+0000 7fd011ffb700 1 -- 172.21.15.189:0/3654153881 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c004e30 con 0x7fd01c0c4d50 2024-04-06T12:31:27.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.201+0000 7fd0235d2700 1 -- 172.21.15.189:0/3654153881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd01c0c7680 con 0x7fd01c0c4d50 2024-04-06T12:31:27.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.201+0000 7fd0235d2700 1 -- 172.21.15.189:0/3654153881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd01c074990 con 0x7fd01c0c4d50 2024-04-06T12:31:27.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.202+0000 7fd011ffb700 1 -- 172.21.15.189:0/3654153881 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fd00c0070d0 con 0x7fd01c0c4d50 2024-04-06T12:31:27.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.202+0000 7fd011ffb700 1 --2- 172.21.15.189:0/3654153881 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd00006adb0 0x7fd00006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:27.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.202+0000 7fd011ffb700 1 -- 172.21.15.189:0/3654153881 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd00c08a960 con 0x7fd01c0c4d50 2024-04-06T12:31:27.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.202+0000 7fd02136e700 1 --2- 172.21.15.189:0/3654153881 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd00006adb0 0x7fd00006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:27.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.203+0000 7fd02136e700 1 --2- 172.21.15.189:0/3654153881 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd00006adb0 0x7fd00006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd018014430 tx=0x7fd01800b040 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:27.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.207+0000 7fd011ffb700 1 -- 172.21.15.189:0/3654153881 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd00c055b60 con 0x7fd01c0c4d50 2024-04-06T12:31:27.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.444+0000 7fd0235d2700 1 -- 172.21.15.189:0/3654153881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd01c160d90 con 0x7fd01c0c4d50 2024-04-06T12:31:27.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.445+0000 7fd011ffb700 1 -- 172.21.15.189:0/3654153881 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4745 (secure 0 0 0) 0x7fd00c08a090 con 0x7fd01c0c4d50 2024-04-06T12:31:27.443 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:27.443 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7115,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/538134862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":538134862},{"type":"v1","addr":"172.21.15.192:6834","nonce":538134862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7915,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1788966297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1788966297},{"type":"v1","addr":"172.21.15.189:6837","nonce":1788966297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7930,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2932927074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2932927074},{"type":"v1","addr":"172.21.15.192:6835","nonce":2932927074}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:31:22.083923+0000","modified":"2024-04-06T12:31:25.104510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/4249639368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":4249639368},{"type":"v1","addr":"172.21.15.189:6836","nonce":4249639368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-06T12:31:27.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.447+0000 7fd0235d2700 1 -- 172.21.15.189:0/3654153881 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd00006adb0 msgr2=0x7fd00006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:27.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.447+0000 7fd0235d2700 1 --2- 172.21.15.189:0/3654153881 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd00006adb0 0x7fd00006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd018014430 tx=0x7fd01800b040 comp rx=0 tx=0).stop 2024-04-06T12:31:27.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.447+0000 7fd0235d2700 1 -- 172.21.15.189:0/3654153881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd01c0c4d50 msgr2=0x7fd01c0c38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:27.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.447+0000 7fd0235d2700 1 --2- 172.21.15.189:0/3654153881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd01c0c4d50 0x7fd01c0c38f0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fd00c000c00 tx=0x7fd00c001a00 comp rx=0 tx=0).stop 2024-04-06T12:31:27.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.447+0000 7fd0235d2700 1 -- 172.21.15.189:0/3654153881 shutdown_connections 2024-04-06T12:31:27.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.447+0000 7fd0235d2700 1 --2- 172.21.15.189:0/3654153881 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd01c159de0 0x7fd01c15a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:27.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.447+0000 7fd0235d2700 1 --2- 172.21.15.189:0/3654153881 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd00006adb0 0x7fd00006d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:27.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.447+0000 7fd0235d2700 1 --2- 172.21.15.189:0/3654153881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd008003f80 0x7fd01c0c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:27.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.447+0000 7fd0235d2700 1 --2- 172.21.15.189:0/3654153881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd01c0c4d50 0x7fd01c0c38f0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:27.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.447+0000 7fd0235d2700 1 -- 172.21.15.189:0/3654153881 >> 172.21.15.189:0/3654153881 conn(0x7fd01c0b2db0 msgr2=0x7fd01c15e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:27.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.447+0000 7fd0235d2700 1 -- 172.21.15.189:0/3654153881 shutdown_connections 2024-04-06T12:31:27.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.447+0000 7fd0235d2700 1 -- 172.21.15.189:0/3654153881 wait complete. 2024-04-06T12:31:27.447 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 62 2024-04-06T12:31:27.665 INFO:tasks.ceph.mds.c.smithi189.stderr:2024-04-06T12:31:27.667+0000 7f9d16487700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T12:31:27.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-06T12:31:22.083923+0000', 'modified': '2024-04-06T12:31:25.104510+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8586}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8586': {'gid': 8586, 'name': 'c', 'rank': 0, 'incarnation': 58, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.189:6836/4249639368', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 4249639368}, {'type': 'v1', 'addr': '172.21.15.189:6836', 'nonce': 4249639368}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-06T12:31:27.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-06T12:31:27.861 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-06T12:31:27.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.988+0000 7f78b9b7f700 1 Processor -- start 2024-04-06T12:31:27.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.988+0000 7f78b9b7f700 1 -- start start 2024-04-06T12:31:27.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.989+0000 7f78b9b7f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f78b406ae40 0x7f78b40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:27.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.989+0000 7f78b9b7f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78b40b3990 con 0x7f78b415d7b0 2024-04-06T12:31:27.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.989+0000 7f78b9b7f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78b40b3060 con 0x7f78b415ddf0 2024-04-06T12:31:27.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.989+0000 7f78b9b7f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78b415e950 con 0x7f78b406ae40 2024-04-06T12:31:27.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.989+0000 7f78b2ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f78b406ae40 0x7f78b40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:27.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.989+0000 7f78b2ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f78b406ae40 0x7f78b40b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51014/0 (socket says 172.21.15.189:51014) 2024-04-06T12:31:27.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.989+0000 7f78b2ffd700 1 -- 172.21.15.189:0/760340430 learned_addr learned my addr 172.21.15.189:0/760340430 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:27.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.989+0000 7f78b37fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f78b415d7b0 0x7f78b406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48212/0 (socket says 172.21.15.189:48212) 2024-04-06T12:31:27.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.989+0000 7f78b2ffd700 1 -- 172.21.15.189:0/760340430 >> v1:172.21.15.192:6789/0 conn(0x7f78b415ddf0 legacy=0x7f78b415e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:27.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.989+0000 7f78b2ffd700 1 -- 172.21.15.189:0/760340430 >> v1:172.21.15.189:6789/0 conn(0x7f78b415d7b0 legacy=0x7f78b406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:27.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.989+0000 7f78b2ffd700 1 -- 172.21.15.189:0/760340430 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b40b3340 con 0x7f78b406ae40 2024-04-06T12:31:27.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.989+0000 7f78b27fc700 1 -- 172.21.15.189:0/760340430 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124111257 0 0) 0x7f78b40b3990 con 0x7f78b415d7b0 2024-04-06T12:31:27.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.989+0000 7f78b2ffd700 1 --2- 172.21.15.189:0/760340430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78b406ae40 0x7f78b40b3e20 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f78a40098f0 tx=0x7f78a4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671a88abe19cf613 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:27.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.990+0000 7f78b27fc700 1 -- 172.21.15.189:0/760340430 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a4004230 con 0x7f78b406ae40 2024-04-06T12:31:27.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.990+0000 7f78b27fc700 1 -- 172.21.15.189:0/760340430 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a4004390 con 0x7f78b406ae40 2024-04-06T12:31:27.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.990+0000 7f78b27fc700 1 -- 172.21.15.189:0/760340430 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a40036a0 con 0x7f78b406ae40 2024-04-06T12:31:27.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.990+0000 7f78b9b7f700 1 -- 172.21.15.189:0/760340430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78b406ae40 msgr2=0x7f78b40b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:27.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.990+0000 7f78b9b7f700 1 --2- 172.21.15.189:0/760340430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78b406ae40 0x7f78b40b3e20 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f78a40098f0 tx=0x7f78a4009c00 comp rx=0 tx=0).stop 2024-04-06T12:31:27.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.990+0000 7f78b9b7f700 1 -- 172.21.15.189:0/760340430 shutdown_connections 2024-04-06T12:31:27.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.990+0000 7f78b9b7f700 1 --2- 172.21.15.189:0/760340430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78b406ae40 0x7f78b40b3e20 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:27.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.990+0000 7f78b9b7f700 1 -- 172.21.15.189:0/760340430 >> 172.21.15.189:0/760340430 conn(0x7f78b40d00b0 msgr2=0x7f78b40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:27.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.990+0000 7f78b9b7f700 1 -- 172.21.15.189:0/760340430 shutdown_connections 2024-04-06T12:31:27.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.990+0000 7f78b9b7f700 1 -- 172.21.15.189:0/760340430 wait complete. 2024-04-06T12:31:27.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.991+0000 7f78b9b7f700 1 Processor -- start 2024-04-06T12:31:27.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.992+0000 7f78b9b7f700 1 -- start start 2024-04-06T12:31:27.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.992+0000 7f78b9b7f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78b406ae40 0x7f78b40cc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:27.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.992+0000 7f78b9b7f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78b415d7b0 0x7f78b40ceb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:27.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.992+0000 7f78b9b7f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78b415ddf0 0x7f78b4149ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:27.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.992+0000 7f78b9b7f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b40cf220 con 0x7f78b415ddf0 2024-04-06T12:31:27.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.992+0000 7f78b9b7f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b414a640 con 0x7f78b415d7b0 2024-04-06T12:31:27.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.992+0000 7f78b9b7f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78b414a920 con 0x7f78b406ae40 2024-04-06T12:31:27.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.992+0000 7f78b3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78b415ddf0 0x7f78b4149ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:27.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.992+0000 7f78b3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78b415ddf0 0x7f78b4149ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51020/0 (socket says 172.21.15.189:51020) 2024-04-06T12:31:27.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.992+0000 7f78b3fff700 1 -- 172.21.15.189:0/3838867995 learned_addr learned my addr 172.21.15.189:0/3838867995 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:27.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.993+0000 7f78b37fe700 1 --2- 172.21.15.189:0/3838867995 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78b406ae40 0x7f78b40cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:27.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.993+0000 7f78b2ffd700 1 --2- 172.21.15.189:0/3838867995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78b415d7b0 0x7f78b40ceb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:27.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.993+0000 7f78b3fff700 1 -- 172.21.15.189:0/3838867995 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78b406ae40 msgr2=0x7f78b40cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:27.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.993+0000 7f78b3fff700 1 --2- 172.21.15.189:0/3838867995 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78b406ae40 0x7f78b40cc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:27.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.993+0000 7f78b3fff700 1 -- 172.21.15.189:0/3838867995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78b415d7b0 msgr2=0x7f78b40ceb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:27.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.993+0000 7f78b3fff700 1 --2- 172.21.15.189:0/3838867995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78b415d7b0 0x7f78b40ceb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:27.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.993+0000 7f78b3fff700 1 -- 172.21.15.189:0/3838867995 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a4009550 con 0x7f78b415ddf0 2024-04-06T12:31:27.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.993+0000 7f78b37fe700 1 --2- 172.21.15.189:0/3838867995 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78b406ae40 0x7f78b40cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:27.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.993+0000 7f78b2ffd700 1 --2- 172.21.15.189:0/3838867995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78b415d7b0 0x7f78b40ceb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:27.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.993+0000 7f78b3fff700 1 --2- 172.21.15.189:0/3838867995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78b415ddf0 0x7f78b4149ac0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f78ac00b970 tx=0x7f78ac00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:27.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.993+0000 7f789bfff700 1 -- 172.21.15.189:0/3838867995 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ac00c9b0 con 0x7f78b415ddf0 2024-04-06T12:31:27.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.993+0000 7f789bfff700 1 -- 172.21.15.189:0/3838867995 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ac00cb10 con 0x7f78b415ddf0 2024-04-06T12:31:27.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.993+0000 7f78b9b7f700 1 -- 172.21.15.189:0/3838867995 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b414ac00 con 0x7f78b415ddf0 2024-04-06T12:31:27.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.993+0000 7f789bfff700 1 -- 172.21.15.189:0/3838867995 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ac014440 con 0x7f78b415ddf0 2024-04-06T12:31:27.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.993+0000 7f78b9b7f700 1 -- 172.21.15.189:0/3838867995 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b414d2e0 con 0x7f78b415ddf0 2024-04-06T12:31:27.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.994+0000 7f789bfff700 1 -- 172.21.15.189:0/3838867995 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f78ac01d3b0 con 0x7f78b415ddf0 2024-04-06T12:31:27.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.995+0000 7f789bfff700 1 --2- 172.21.15.189:0/3838867995 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f78a006adb0 0x7f78a006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:27.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.995+0000 7f789bfff700 1 -- 172.21.15.189:0/3838867995 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f78ac0892a0 con 0x7f78b415ddf0 2024-04-06T12:31:27.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.995+0000 7f78b37fe700 1 --2- 172.21.15.189:0/3838867995 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f78a006adb0 0x7f78a006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:27.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.995+0000 7f78b37fe700 1 --2- 172.21.15.189:0/3838867995 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f78a006adb0 0x7f78a006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f78a8000f60 tx=0x7f78a8009040 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:27.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.997+0000 7f78c5d33b80 1 Processor -- start 2024-04-06T12:31:27.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.997+0000 7f78c5d33b80 1 -- start start 2024-04-06T12:31:27.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78c5d33b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d710c20 0x555d8d714a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:27.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78c5d33b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d715fc0 0x555d8d6ec5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:27.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78c5d33b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x555d8d6eb980 0x555d8d6ecb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:27.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78c5d33b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8d645940 con 0x555d8d715fc0 2024-04-06T12:31:27.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78c5d33b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8d643300 con 0x555d8d710c20 2024-04-06T12:31:27.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78c5d33b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x555d8d63ebf0 con 0x555d8d6eb980 2024-04-06T12:31:27.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78b2ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d715fc0 0x555d8d6ec5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:27.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78b2ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d715fc0 0x555d8d6ec5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51028/0 (socket says 172.21.15.189:51028) 2024-04-06T12:31:27.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78b2ffd700 1 -- 172.21.15.189:0/2575712182 learned_addr learned my addr 172.21.15.189:0/2575712182 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:27.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78b37fe700 1 --2- 172.21.15.189:0/2575712182 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d710c20 0x555d8d714a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:27.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78b3fff700 1 --2- 172.21.15.189:0/2575712182 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x555d8d6eb980 0x555d8d6ecb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:27.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78b2ffd700 1 -- 172.21.15.189:0/2575712182 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x555d8d6eb980 msgr2=0x555d8d6ecb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:27.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78b2ffd700 1 --2- 172.21.15.189:0/2575712182 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x555d8d6eb980 0x555d8d6ecb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:27.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78b2ffd700 1 -- 172.21.15.189:0/2575712182 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d710c20 msgr2=0x555d8d714a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:27.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78b2ffd700 1 --2- 172.21.15.189:0/2575712182 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d710c20 0x555d8d714a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:27.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78b2ffd700 1 -- 172.21.15.189:0/2575712182 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d8d46cb10 con 0x555d8d715fc0 2024-04-06T12:31:27.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78b2ffd700 1 --2- 172.21.15.189:0/2575712182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d715fc0 0x555d8d6ec5f0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f78a4000c00 tx=0x7f78a4010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c6474795ce0175 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:27.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78997fa700 1 -- 172.21.15.189:0/2575712182 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a4004230 con 0x555d8d715fc0 2024-04-06T12:31:27.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78997fa700 1 -- 172.21.15.189:0/2575712182 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a4003890 con 0x555d8d715fc0 2024-04-06T12:31:27.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.998+0000 7f78997fa700 1 -- 172.21.15.189:0/2575712182 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a4003c20 con 0x555d8d715fc0 2024-04-06T12:31:27.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.999+0000 7f78c5d33b80 1 -- 172.21.15.189:0/2575712182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d715fc0 msgr2=0x555d8d6ec5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:27.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.999+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/2575712182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d715fc0 0x555d8d6ec5f0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f78a4000c00 tx=0x7f78a4010000 comp rx=0 tx=0).stop 2024-04-06T12:31:28.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.999+0000 7f78c5d33b80 1 -- 172.21.15.189:0/2575712182 shutdown_connections 2024-04-06T12:31:28.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.999+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/2575712182 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x555d8d6eb980 0x555d8d6ecb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.999+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/2575712182 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d710c20 0x555d8d714a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.999+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/2575712182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d715fc0 0x555d8d6ec5f0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.999+0000 7f78c5d33b80 1 -- 172.21.15.189:0/2575712182 >> 172.21.15.189:0/2575712182 conn(0x555d8d64c650 msgr2=0x555d8d7333e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:28.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.999+0000 7f78c5d33b80 1 -- 172.21.15.189:0/2575712182 shutdown_connections 2024-04-06T12:31:28.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:27.999+0000 7f78c5d33b80 1 -- 172.21.15.189:0/2575712182 wait complete. 2024-04-06T12:31:28.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.000+0000 7f78c5d33b80 1 -- start start 2024-04-06T12:31:28.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.000+0000 7f78c5d33b80 1 Processor -- start 2024-04-06T12:31:28.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78c5d33b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d6eb980 0x555d8d72ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:28.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78c5d33b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x555d8d710c20 0x555d8d7429f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:28.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78c5d33b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d715fc0 0x555d8d742f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:28.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78c5d33b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8d5dad30 con 0x555d8d6eb980 2024-04-06T12:31:28.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78c5d33b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8d5da9d0 con 0x555d8d715fc0 2024-04-06T12:31:28.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78c5d33b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x555d8d4c2650 con 0x555d8d710c20 2024-04-06T12:31:28.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78c5d33b80 10 client.0 _ll_register_callbacks cb 0x555d8d5e4420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:31:28.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78b37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d6eb980 0x555d8d72ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:28.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78b37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d6eb980 0x555d8d72ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51042/0 (socket says 172.21.15.189:51042) 2024-04-06T12:31:28.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78b37fe700 1 -- 172.21.15.189:0/251125642 learned_addr learned my addr 172.21.15.189:0/251125642 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:28.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78b2ffd700 1 --2- 172.21.15.189:0/251125642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x555d8d710c20 0x555d8d7429f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:28.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78b3fff700 1 --2- 172.21.15.189:0/251125642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d715fc0 0x555d8d742f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:28.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78b37fe700 1 -- 172.21.15.189:0/251125642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x555d8d710c20 msgr2=0x555d8d7429f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:28.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78b37fe700 1 --2- 172.21.15.189:0/251125642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x555d8d710c20 0x555d8d7429f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78b37fe700 1 -- 172.21.15.189:0/251125642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d715fc0 msgr2=0x555d8d742f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:28.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78b37fe700 1 --2- 172.21.15.189:0/251125642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d715fc0 0x555d8d742f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78b37fe700 1 -- 172.21.15.189:0/251125642 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b40b3340 con 0x555d8d6eb980 2024-04-06T12:31:28.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78b37fe700 1 --2- 172.21.15.189:0/251125642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d6eb980 0x555d8d72ae10 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f78a80074b0 tx=0x7f78a8001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f908545742062d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:28.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78927fc700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:31:28.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78927fc700 1 -- 172.21.15.189:0/251125642 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a800dcc0 con 0x555d8d6eb980 2024-04-06T12:31:28.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78927fc700 1 -- 172.21.15.189:0/251125642 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(144..144 src has 1..144) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f78a800de20 con 0x555d8d6eb980 2024-04-06T12:31:28.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78c5d33b80 1 -- 172.21.15.189:0/251125642 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555d8d5ff330 con 0x555d8d6eb980 2024-04-06T12:31:28.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78927fc700 1 -- 172.21.15.189:0/251125642 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78a8004870 con 0x555d8d6eb980 2024-04-06T12:31:28.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78c5d33b80 10 client.8733 fetch_fsmap learned FSMap version 62 2024-04-06T12:31:28.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.001+0000 7f78c5d33b80 1 -- 172.21.15.189:0/251125642 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) v3 -- 0x555d8d604e50 con 0x555d8d6eb980 2024-04-06T12:31:28.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.002+0000 7f78927fc700 1 -- 172.21.15.189:0/251125642 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555d8d5ff330 con 0x555d8d6eb980 2024-04-06T12:31:28.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.002+0000 7f78927fc700 1 -- 172.21.15.189:0/251125642 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 62) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f78a801f070 con 0x555d8d6eb980 2024-04-06T12:31:28.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.002+0000 7f78c5d33b80 10 client.8733 fetch_fsmap finished waiting for FSMap version 62 2024-04-06T12:31:28.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.002+0000 7f78c5d33b80 10 client.8733 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-06T12:31:28.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.002+0000 7f78c5d33b80 20 client.8733 populate_metadata read hostname 'smithi189' 2024-04-06T12:31:28.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.002+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/251125642 >> [v2:172.21.15.189:6834/4249639368,v1:172.21.15.189:6836/4249639368] conn(0x555d8d72d340 0x555d8d743d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:28.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.002+0000 7f78c5d33b80 4 client.8733 mds_command: new command op to 8586 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-06T12:31:28.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.002+0000 7f78c5d33b80 1 -- 172.21.15.189:0/251125642 --> [v2:172.21.15.189:6834/4249639368,v1:172.21.15.189:6836/4249639368] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555d8d6376b0 con 0x555d8d72d340 2024-04-06T12:31:28.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.002+0000 7f78b2ffd700 1 --2- 172.21.15.189:0/251125642 >> [v2:172.21.15.189:6834/4249639368,v1:172.21.15.189:6836/4249639368] conn(0x555d8d72d340 0x555d8d743d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:28.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.002+0000 7f78927fc700 10 client.8733 ms_handle_connect on v2:172.21.15.189:6834/4249639368 2024-04-06T12:31:28.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.002+0000 7f78b2ffd700 1 --2- 172.21.15.189:0/251125642 >> [v2:172.21.15.189:6834/4249639368,v1:172.21.15.189:6836/4249639368] conn(0x555d8d72d340 0x555d8d743d40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=618a72310a75c176 server_cookie=cc33569f4c2dc8e8 in_seq=0 out_seq=0 2024-04-06T12:31:28.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.003+0000 7f78927fc700 1 -- 172.21.15.189:0/251125642 <== mds.0 v2:172.21.15.189:6834/4249639368 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555d8d6376b0 con 0x555d8d72d340 2024-04-06T12:31:28.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.003+0000 7f78927fc700 10 client.8733 handle_command_reply: tid=0 2024-04-06T12:31:28.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.003+0000 7f78c5d33b80 1 client.8733 shutdown 2024-04-06T12:31:28.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.003+0000 7f78c5d33b80 1 -- 172.21.15.189:0/251125642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d6eb980 msgr2=0x555d8d72ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:28.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.003+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/251125642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d6eb980 0x555d8d72ae10 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f78a80074b0 tx=0x7f78a8001080 comp rx=0 tx=0).stop 2024-04-06T12:31:28.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.003+0000 7f78c5d33b80 1 -- 172.21.15.189:0/251125642 shutdown_connections 2024-04-06T12:31:28.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.003+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/251125642 >> [v2:172.21.15.189:6834/4249639368,v1:172.21.15.189:6836/4249639368] conn(0x555d8d72d340 0x555d8d743d40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.003+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/251125642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d715fc0 0x555d8d742f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.003+0000 7f78927fc700 0 client.8733 ms_handle_reset on v2:172.21.15.189:6834/4249639368 2024-04-06T12:31:28.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.003+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/251125642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d6eb980 0x555d8d72ae10 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.003+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/251125642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x555d8d710c20 0x555d8d7429f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.003+0000 7f78c5d33b80 1 -- 172.21.15.189:0/251125642 >> 172.21.15.189:0/251125642 conn(0x555d8d64c650 msgr2=0x555d8d733370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:28.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.003+0000 7f78c5d33b80 1 -- 172.21.15.189:0/251125642 shutdown_connections 2024-04-06T12:31:28.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.003+0000 7f78c5d33b80 1 -- 172.21.15.189:0/251125642 wait complete. 2024-04-06T12:31:28.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.003+0000 7f78c5d33b80 20 client.8733 trim_cache size 0 max 16384 2024-04-06T12:31:28.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.020+0000 7f78c5d33b80 1 Processor -- start 2024-04-06T12:31:28.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.020+0000 7f78c5d33b80 1 -- start start 2024-04-06T12:31:28.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.021+0000 7f78c5d33b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x555d8d6eb980 0x555d8d741ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:28.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.021+0000 7f78c5d33b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d710c20 0x555d8d6ea700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:28.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.021+0000 7f78c5d33b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d715fc0 0x555d8d6eac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:28.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.021+0000 7f78c5d33b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8d6e7a60 con 0x555d8d710c20 2024-04-06T12:31:28.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.021+0000 7f78c5d33b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8d742340 con 0x555d8d715fc0 2024-04-06T12:31:28.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.021+0000 7f78c5d33b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x555d8d6e82d0 con 0x555d8d6eb980 2024-04-06T12:31:28.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.021+0000 7f78b2ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d710c20 0x555d8d6ea700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:28.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.021+0000 7f78b2ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d710c20 0x555d8d6ea700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51054/0 (socket says 172.21.15.189:51054) 2024-04-06T12:31:28.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.021+0000 7f78b2ffd700 1 -- 172.21.15.189:0/3729733028 learned_addr learned my addr 172.21.15.189:0/3729733028 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:28.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.022+0000 7f78b2ffd700 1 -- 172.21.15.189:0/3729733028 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x555d8d6eb980 msgr2=0x555d8d741ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:28.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.022+0000 7f78b2ffd700 1 --2- 172.21.15.189:0/3729733028 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x555d8d6eb980 0x555d8d741ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.022+0000 7f78b2ffd700 1 -- 172.21.15.189:0/3729733028 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d715fc0 msgr2=0x555d8d6eac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:28.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.022+0000 7f78b3fff700 1 --2- 172.21.15.189:0/3729733028 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d715fc0 0x555d8d6eac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:28.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.022+0000 7f78b2ffd700 1 --2- 172.21.15.189:0/3729733028 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d715fc0 0x555d8d6eac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.022+0000 7f78b2ffd700 1 -- 172.21.15.189:0/3729733028 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d8d4c4c80 con 0x555d8d710c20 2024-04-06T12:31:28.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.022+0000 7f78b2ffd700 1 --2- 172.21.15.189:0/3729733028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d710c20 0x555d8d6ea700 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f78a4004000 tx=0x7f78a4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1caa2ca0de07a898 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:28.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.022+0000 7f7899ffb700 1 -- 172.21.15.189:0/3729733028 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a402a040 con 0x555d8d710c20 2024-04-06T12:31:28.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.022+0000 7f7899ffb700 1 -- 172.21.15.189:0/3729733028 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a4004580 con 0x555d8d710c20 2024-04-06T12:31:28.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.022+0000 7f7899ffb700 1 -- 172.21.15.189:0/3729733028 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a4003c20 con 0x555d8d710c20 2024-04-06T12:31:28.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.023+0000 7f78c5d33b80 1 -- 172.21.15.189:0/3729733028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d710c20 msgr2=0x555d8d6ea700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:28.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.023+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/3729733028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d710c20 0x555d8d6ea700 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f78a4004000 tx=0x7f78a4011000 comp rx=0 tx=0).stop 2024-04-06T12:31:28.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.023+0000 7f78c5d33b80 1 -- 172.21.15.189:0/3729733028 shutdown_connections 2024-04-06T12:31:28.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.023+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/3729733028 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d715fc0 0x555d8d6eac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.023+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/3729733028 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x555d8d6eb980 0x555d8d741ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.023+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/3729733028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d710c20 0x555d8d6ea700 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.023+0000 7f78c5d33b80 1 -- 172.21.15.189:0/3729733028 >> 172.21.15.189:0/3729733028 conn(0x555d8d64c650 msgr2=0x555d8d755480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:28.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.023+0000 7f78c5d33b80 1 -- 172.21.15.189:0/3729733028 shutdown_connections 2024-04-06T12:31:28.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.023+0000 7f78c5d33b80 1 -- 172.21.15.189:0/3729733028 wait complete. 2024-04-06T12:31:28.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.023+0000 7f78c5d33b80 1 -- start start 2024-04-06T12:31:28.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.023+0000 7f78c5d33b80 1 Processor -- start 2024-04-06T12:31:28.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.024+0000 7f78c5d33b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x555d8d6eb980 0x555d8d741d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:28.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.024+0000 7f78c5d33b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d710c20 0x555d8d7946b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:28.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.024+0000 7f78c5d33b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d715fc0 0x555d8d743010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:28.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.024+0000 7f78c5d33b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8d6ee280 con 0x555d8d715fc0 2024-04-06T12:31:28.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.024+0000 7f78c5d33b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8d79a5e0 con 0x555d8d710c20 2024-04-06T12:31:28.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.024+0000 7f78c5d33b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x555d8d72a150 con 0x555d8d6eb980 2024-04-06T12:31:28.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.024+0000 7f78c5d33b80 10 client.0 _ll_register_callbacks cb 0x7f78a8005c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:31:28.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.024+0000 7f78b3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d715fc0 0x555d8d743010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:28.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.024+0000 7f78b3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d715fc0 0x555d8d743010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51062/0 (socket says 172.21.15.189:51062) 2024-04-06T12:31:28.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.024+0000 7f78b3fff700 1 -- 172.21.15.189:0/330161583 learned_addr learned my addr 172.21.15.189:0/330161583 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:28.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.024+0000 7f78b2ffd700 1 --2- 172.21.15.189:0/330161583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d710c20 0x555d8d7946b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:28.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.024+0000 7f78b37fe700 1 --2- 172.21.15.189:0/330161583 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x555d8d6eb980 0x555d8d741d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:28.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.024+0000 7f78b3fff700 1 -- 172.21.15.189:0/330161583 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x555d8d6eb980 msgr2=0x555d8d741d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:28.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.024+0000 7f78b3fff700 1 --2- 172.21.15.189:0/330161583 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x555d8d6eb980 0x555d8d741d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.024+0000 7f78b3fff700 1 -- 172.21.15.189:0/330161583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d710c20 msgr2=0x555d8d7946b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:28.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.024+0000 7f78b3fff700 1 --2- 172.21.15.189:0/330161583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d710c20 0x555d8d7946b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.025+0000 7f78b3fff700 1 -- 172.21.15.189:0/330161583 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d8d46cb10 con 0x555d8d715fc0 2024-04-06T12:31:28.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.025+0000 7f78b3fff700 1 --2- 172.21.15.189:0/330161583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d715fc0 0x555d8d743010 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f78ac00b940 tx=0x7f78ac057ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f8fedf7a6768dd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:28.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.025+0000 7f78927fc700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:31:28.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.025+0000 7f78927fc700 1 -- 172.21.15.189:0/330161583 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ac0602e0 con 0x555d8d715fc0 2024-04-06T12:31:28.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.025+0000 7f78927fc700 1 -- 172.21.15.189:0/330161583 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(144..144 src has 1..144) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f78ac0890b0 con 0x555d8d715fc0 2024-04-06T12:31:28.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.025+0000 7f78c5d33b80 1 -- 172.21.15.189:0/330161583 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555d8d71c770 con 0x555d8d715fc0 2024-04-06T12:31:28.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.025+0000 7f78927fc700 1 -- 172.21.15.189:0/330161583 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78ac087260 con 0x555d8d715fc0 2024-04-06T12:31:28.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.025+0000 7f78c5d33b80 10 client.8739 fetch_fsmap learned FSMap version 62 2024-04-06T12:31:28.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.025+0000 7f78c5d33b80 1 -- 172.21.15.189:0/330161583 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) v3 -- 0x555d8d46afe0 con 0x555d8d715fc0 2024-04-06T12:31:28.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.025+0000 7f78927fc700 1 -- 172.21.15.189:0/330161583 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ac073070 con 0x555d8d715fc0 2024-04-06T12:31:28.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.025+0000 7f78927fc700 1 -- 172.21.15.189:0/330161583 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 62) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f78ac0573e0 con 0x555d8d715fc0 2024-04-06T12:31:28.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.025+0000 7f78c5d33b80 10 client.8739 fetch_fsmap finished waiting for FSMap version 62 2024-04-06T12:31:28.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.025+0000 7f78c5d33b80 10 client.8739 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-06T12:31:28.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.025+0000 7f78c5d33b80 20 client.8739 populate_metadata read hostname 'smithi189' 2024-04-06T12:31:28.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.025+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/330161583 >> [v2:172.21.15.189:6834/4249639368,v1:172.21.15.189:6836/4249639368] conn(0x555d8d743920 0x555d8d743d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:28.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.025+0000 7f78c5d33b80 4 client.8739 mds_command: new command op to 8586 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-06T12:31:28.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.025+0000 7f78c5d33b80 1 -- 172.21.15.189:0/330161583 --> [v2:172.21.15.189:6834/4249639368,v1:172.21.15.189:6836/4249639368] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f78a4003890 con 0x555d8d743920 2024-04-06T12:31:28.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.025+0000 7f78b37fe700 1 --2- 172.21.15.189:0/330161583 >> [v2:172.21.15.189:6834/4249639368,v1:172.21.15.189:6836/4249639368] conn(0x555d8d743920 0x555d8d743d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:28.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.026+0000 7f78b37fe700 1 --2- 172.21.15.189:0/330161583 >> [v2:172.21.15.189:6834/4249639368,v1:172.21.15.189:6836/4249639368] conn(0x555d8d743920 0x555d8d743d00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c577b20454b4dc32 server_cookie=9f42272063275fcc in_seq=0 out_seq=0 2024-04-06T12:31:28.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.026+0000 7f78927fc700 10 client.8739 ms_handle_connect on v2:172.21.15.189:6834/4249639368 2024-04-06T12:31:28.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.026+0000 7f78927fc700 1 -- 172.21.15.189:0/330161583 <== mds.0 v2:172.21.15.189:6834/4249639368 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+320 (crc 0 0 0) 0x7f78a4003890 con 0x555d8d743920 2024-04-06T12:31:28.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.026+0000 7f78927fc700 10 client.8739 handle_command_reply: tid=0 2024-04-06T12:31:28.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.026+0000 7f78c5d33b80 1 client.8739 shutdown 2024-04-06T12:31:28.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.026+0000 7f78c5d33b80 1 -- 172.21.15.189:0/330161583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d715fc0 msgr2=0x555d8d743010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:28.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.026+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/330161583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d715fc0 0x555d8d743010 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f78ac00b940 tx=0x7f78ac057ef0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.026+0000 7f78c5d33b80 1 -- 172.21.15.189:0/330161583 shutdown_connections 2024-04-06T12:31:28.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.026+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/330161583 >> [v2:172.21.15.189:6834/4249639368,v1:172.21.15.189:6836/4249639368] conn(0x555d8d743920 0x555d8d743d00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.026+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/330161583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d8d715fc0 0x555d8d743010 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.026+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/330161583 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x555d8d6eb980 0x555d8d741d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.026+0000 7f78927fc700 0 client.8739 ms_handle_reset on v2:172.21.15.189:6834/4249639368 2024-04-06T12:31:28.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.026+0000 7f78c5d33b80 1 --2- 172.21.15.189:0/330161583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555d8d710c20 0x555d8d7946b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.026+0000 7f78c5d33b80 1 -- 172.21.15.189:0/330161583 >> 172.21.15.189:0/330161583 conn(0x555d8d64c650 msgr2=0x555d8d6e45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:28.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.026+0000 7f78c5d33b80 1 -- 172.21.15.189:0/330161583 shutdown_connections 2024-04-06T12:31:28.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.026+0000 7f78c5d33b80 1 -- 172.21.15.189:0/330161583 wait complete. 2024-04-06T12:31:28.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.026+0000 7f78c5d33b80 20 client.8739 trim_cache size 0 max 16384 2024-04-06T12:31:28.031 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T12:31:28.032 INFO:teuthology.orchestra.run.smithi189.stdout: "cluster_fsid": "e30041de-e9a3-4351-8ca7-29d7f3e3728c", 2024-04-06T12:31:28.032 INFO:teuthology.orchestra.run.smithi189.stdout: "whoami": 0, 2024-04-06T12:31:28.032 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 8586, 2024-04-06T12:31:28.032 INFO:teuthology.orchestra.run.smithi189.stdout: "want_state": "up:active", 2024-04-06T12:31:28.032 INFO:teuthology.orchestra.run.smithi189.stdout: "state": "up:active", 2024-04-06T12:31:28.032 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs", 2024-04-06T12:31:28.032 INFO:teuthology.orchestra.run.smithi189.stdout: "rank_uptime": 5.941394496, 2024-04-06T12:31:28.032 INFO:teuthology.orchestra.run.smithi189.stdout: "mdsmap_epoch": 62, 2024-04-06T12:31:28.032 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch": 144, 2024-04-06T12:31:28.032 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch_barrier": 144, 2024-04-06T12:31:28.032 INFO:teuthology.orchestra.run.smithi189.stdout: "uptime": 11.270242651 2024-04-06T12:31:28.032 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T12:31:28.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.027+0000 7f78b9b7f700 1 -- 172.21.15.189:0/3838867995 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f78a006adb0 msgr2=0x7f78a006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:28.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.027+0000 7f78b9b7f700 1 --2- 172.21.15.189:0/3838867995 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f78a006adb0 0x7f78a006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f78a8000f60 tx=0x7f78a8009040 comp rx=0 tx=0).stop 2024-04-06T12:31:28.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.027+0000 7f78b9b7f700 1 -- 172.21.15.189:0/3838867995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78b415ddf0 msgr2=0x7f78b4149ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:28.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.027+0000 7f78b9b7f700 1 --2- 172.21.15.189:0/3838867995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78b415ddf0 0x7f78b4149ac0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f78ac00b970 tx=0x7f78ac00bd30 comp rx=0 tx=0).stop 2024-04-06T12:31:28.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.027+0000 7f78b9b7f700 1 -- 172.21.15.189:0/3838867995 shutdown_connections 2024-04-06T12:31:28.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.027+0000 7f78b9b7f700 1 --2- 172.21.15.189:0/3838867995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78b415ddf0 0x7f78b4149ac0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.027+0000 7f78b9b7f700 1 --2- 172.21.15.189:0/3838867995 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78b406ae40 0x7f78b40cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.027+0000 7f78b9b7f700 1 --2- 172.21.15.189:0/3838867995 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f78a006adb0 0x7f78a006d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.027+0000 7f78b9b7f700 1 --2- 172.21.15.189:0/3838867995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78b415d7b0 0x7f78b40ceb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.027+0000 7f78b9b7f700 1 -- 172.21.15.189:0/3838867995 >> 172.21.15.189:0/3838867995 conn(0x7f78b40d00b0 msgr2=0x7f78b406eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:28.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.027+0000 7f78b9b7f700 1 -- 172.21.15.189:0/3838867995 shutdown_connections 2024-04-06T12:31:28.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.027+0000 7f78b9b7f700 1 -- 172.21.15.189:0/3838867995 wait complete. 2024-04-06T12:31:28.037 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T12:31:28.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.160+0000 7f7a4bfc3700 1 Processor -- start 2024-04-06T12:31:28.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.160+0000 7f7a4bfc3700 1 -- start start 2024-04-06T12:31:28.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.161+0000 7f7a4bfc3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7a440b4660 0x7f7a440b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:28.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.161+0000 7f7a4bfc3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7a440b2fc0 0x7f7a440b3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:28.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.161+0000 7f7a4bfc3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a440b5030 con 0x7f7a4415a550 2024-04-06T12:31:28.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.161+0000 7f7a4bfc3700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a440cdbe0 con 0x7f7a440b4660 2024-04-06T12:31:28.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.161+0000 7f7a4bfc3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4415a940 con 0x7f7a440b2fc0 2024-04-06T12:31:28.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.161+0000 7f7a4955e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7a440b4660 0x7f7a440b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:28.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.161+0000 7f7a49d5f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7a4415a550 0x7f7a440cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48228/0 (socket says 172.21.15.189:48228) 2024-04-06T12:31:28.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.161+0000 7f7a49d5f700 1 -- 172.21.15.189:0/411306253 learned_addr learned my addr 172.21.15.189:0/411306253 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:28.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.161+0000 7f7a4a560700 1 --2- 172.21.15.189:0/411306253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7a440b2fc0 0x7f7a440b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:28.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.161+0000 7f7a4955e700 1 -- 172.21.15.189:0/411306253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7a440b2fc0 msgr2=0x7f7a440b3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:28.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.161+0000 7f7a4955e700 1 --2- 172.21.15.189:0/411306253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7a440b2fc0 0x7f7a440b3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.161+0000 7f7a4955e700 1 -- 172.21.15.189:0/411306253 >> v1:172.21.15.189:6789/0 conn(0x7f7a4415a550 legacy=0x7f7a440cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:28.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.162+0000 7f7a48d5d700 1 -- 172.21.15.189:0/411306253 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629464240 0 0) 0x7f7a440b5030 con 0x7f7a4415a550 2024-04-06T12:31:28.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.162+0000 7f7a4955e700 1 -- 172.21.15.189:0/411306253 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4415ac00 con 0x7f7a440b4660 2024-04-06T12:31:28.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.162+0000 7f7a4955e700 1 --2- 172.21.15.189:0/411306253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a440b4660 0x7f7a440b4af0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f7a40009840 tx=0x7f7a40009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a33dffd5e9e36c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:28.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.162+0000 7f7a48d5d700 1 -- 172.21.15.189:0/411306253 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a40011830 con 0x7f7a440b4660 2024-04-06T12:31:28.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.162+0000 7f7a48d5d700 1 -- 172.21.15.189:0/411306253 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a40011990 con 0x7f7a440b4660 2024-04-06T12:31:28.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.162+0000 7f7a48d5d700 1 -- 172.21.15.189:0/411306253 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a40011d20 con 0x7f7a440b4660 2024-04-06T12:31:28.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.162+0000 7f7a4bfc3700 1 -- 172.21.15.189:0/411306253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a440b4660 msgr2=0x7f7a440b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:28.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.162+0000 7f7a4bfc3700 1 --2- 172.21.15.189:0/411306253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a440b4660 0x7f7a440b4af0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f7a40009840 tx=0x7f7a40009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:28.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.162+0000 7f7a4bfc3700 1 -- 172.21.15.189:0/411306253 shutdown_connections 2024-04-06T12:31:28.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.162+0000 7f7a4bfc3700 1 --2- 172.21.15.189:0/411306253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7a440b2fc0 0x7f7a440b3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.162+0000 7f7a4bfc3700 1 --2- 172.21.15.189:0/411306253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a440b4660 0x7f7a440b4af0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.162+0000 7f7a4bfc3700 1 -- 172.21.15.189:0/411306253 >> 172.21.15.189:0/411306253 conn(0x7f7a440d00b0 msgr2=0x7f7a440d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:28.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.162+0000 7f7a4bfc3700 1 -- 172.21.15.189:0/411306253 shutdown_connections 2024-04-06T12:31:28.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.163+0000 7f7a4bfc3700 1 -- 172.21.15.189:0/411306253 wait complete. 2024-04-06T12:31:28.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.164+0000 7f7a4bfc3700 1 Processor -- start 2024-04-06T12:31:28.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.164+0000 7f7a4bfc3700 1 -- start start 2024-04-06T12:31:28.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.165+0000 7f7a4bfc3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a440b2fc0 0x7f7a4415d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:28.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.165+0000 7f7a4bfc3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a440b4660 0x7f7a4414f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:28.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.165+0000 7f7a4bfc3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7a4406a730 0x7f7a4406abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:28.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.165+0000 7f7a4bfc3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4406b100 con 0x7f7a440b2fc0 2024-04-06T12:31:28.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.165+0000 7f7a4bfc3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4415d150 con 0x7f7a440b4660 2024-04-06T12:31:28.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.165+0000 7f7a4bfc3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a441f7180 con 0x7f7a4406a730 2024-04-06T12:31:28.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.165+0000 7f7a49d5f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a440b2fc0 0x7f7a4415d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:28.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.165+0000 7f7a49d5f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a440b2fc0 0x7f7a4415d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51076/0 (socket says 172.21.15.189:51076) 2024-04-06T12:31:28.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.165+0000 7f7a49d5f700 1 -- 172.21.15.189:0/516303059 learned_addr learned my addr 172.21.15.189:0/516303059 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:28.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.165+0000 7f7a4955e700 1 --2- 172.21.15.189:0/516303059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a440b4660 0x7f7a4414f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:28.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.165+0000 7f7a4a560700 1 --2- 172.21.15.189:0/516303059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7a4406a730 0x7f7a4406abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:28.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.165+0000 7f7a49d5f700 1 -- 172.21.15.189:0/516303059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7a4406a730 msgr2=0x7f7a4406abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:28.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.165+0000 7f7a49d5f700 1 --2- 172.21.15.189:0/516303059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7a4406a730 0x7f7a4406abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.165+0000 7f7a49d5f700 1 -- 172.21.15.189:0/516303059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a440b4660 msgr2=0x7f7a4414f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:28.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.165+0000 7f7a49d5f700 1 --2- 172.21.15.189:0/516303059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a440b4660 0x7f7a4414f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.165+0000 7f7a49d5f700 1 -- 172.21.15.189:0/516303059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a400094a0 con 0x7f7a440b2fc0 2024-04-06T12:31:28.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.165+0000 7f7a4955e700 1 --2- 172.21.15.189:0/516303059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a440b4660 0x7f7a4414f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:28.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.165+0000 7f7a4a560700 1 --2- 172.21.15.189:0/516303059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7a4406a730 0x7f7a4406abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:28.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.166+0000 7f7a49d5f700 1 --2- 172.21.15.189:0/516303059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a440b2fc0 0x7f7a4415d5e0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f7a340011e0 tx=0x7f7a3400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:28.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.166+0000 7f7a3a7fc700 1 -- 172.21.15.189:0/516303059 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a34004c80 con 0x7f7a440b2fc0 2024-04-06T12:31:28.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.166+0000 7f7a3a7fc700 1 -- 172.21.15.189:0/516303059 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a34004e00 con 0x7f7a440b2fc0 2024-04-06T12:31:28.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.166+0000 7f7a4bfc3700 1 -- 172.21.15.189:0/516303059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a441f6ea0 con 0x7f7a440b2fc0 2024-04-06T12:31:28.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.166+0000 7f7a3a7fc700 1 -- 172.21.15.189:0/516303059 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a34007310 con 0x7f7a440b2fc0 2024-04-06T12:31:28.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.166+0000 7f7a4bfc3700 1 -- 172.21.15.189:0/516303059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4415cde0 con 0x7f7a440b2fc0 2024-04-06T12:31:28.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.167+0000 7f7a4bfc3700 1 -- 172.21.15.189:0/516303059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a44067bb0 con 0x7f7a440b2fc0 2024-04-06T12:31:28.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.167+0000 7f7a3a7fc700 1 -- 172.21.15.189:0/516303059 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7a34007470 con 0x7f7a440b2fc0 2024-04-06T12:31:28.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.168+0000 7f7a3a7fc700 1 --2- 172.21.15.189:0/516303059 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7a2806adb0 0x7f7a2806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:28.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.168+0000 7f7a4955e700 1 --2- 172.21.15.189:0/516303059 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7a2806adb0 0x7f7a2806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:28.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.168+0000 7f7a3a7fc700 1 -- 172.21.15.189:0/516303059 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7a340892d0 con 0x7f7a440b2fc0 2024-04-06T12:31:28.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.168+0000 7f7a4955e700 1 --2- 172.21.15.189:0/516303059 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7a2806adb0 0x7f7a2806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7a4406b470 tx=0x7f7a40008030 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:28.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.172+0000 7f7a3a7fc700 1 -- 172.21.15.189:0/516303059 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7a340542f0 con 0x7f7a440b2fc0 2024-04-06T12:31:28.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.407+0000 7f7a4bfc3700 1 -- 172.21.15.189:0/516303059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7a44074990 con 0x7f7a440b2fc0 2024-04-06T12:31:28.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.407+0000 7f7a3a7fc700 1 -- 172.21.15.189:0/516303059 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v62) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7a34057910 con 0x7f7a440b2fc0 2024-04-06T12:31:28.405 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-06T12:31:28.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.409+0000 7f7a4bfc3700 1 -- 172.21.15.189:0/516303059 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7a2806adb0 msgr2=0x7f7a2806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:28.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.409+0000 7f7a4bfc3700 1 --2- 172.21.15.189:0/516303059 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7a2806adb0 0x7f7a2806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7a4406b470 tx=0x7f7a40008030 comp rx=0 tx=0).stop 2024-04-06T12:31:28.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.409+0000 7f7a4bfc3700 1 -- 172.21.15.189:0/516303059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a440b2fc0 msgr2=0x7f7a4415d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:28.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.409+0000 7f7a4bfc3700 1 --2- 172.21.15.189:0/516303059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a440b2fc0 0x7f7a4415d5e0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f7a340011e0 tx=0x7f7a3400cae0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.409+0000 7f7a4bfc3700 1 -- 172.21.15.189:0/516303059 shutdown_connections 2024-04-06T12:31:28.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.409+0000 7f7a4bfc3700 1 --2- 172.21.15.189:0/516303059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7a4406a730 0x7f7a4406abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.409+0000 7f7a4bfc3700 1 --2- 172.21.15.189:0/516303059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a440b2fc0 0x7f7a4415d5e0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.409+0000 7f7a4bfc3700 1 --2- 172.21.15.189:0/516303059 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7a2806adb0 0x7f7a2806d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.409+0000 7f7a4bfc3700 1 --2- 172.21.15.189:0/516303059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a440b4660 0x7f7a4414f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:28.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.409+0000 7f7a4bfc3700 1 -- 172.21.15.189:0/516303059 >> 172.21.15.189:0/516303059 conn(0x7f7a440d00b0 msgr2=0x7f7a44154cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:28.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.410+0000 7f7a4bfc3700 1 -- 172.21.15.189:0/516303059 shutdown_connections 2024-04-06T12:31:28.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:28.410+0000 7f7a4bfc3700 1 -- 172.21.15.189:0/516303059 wait complete. 2024-04-06T12:31:28.420 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-06T12:31:28.420 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T12:31:28.420 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-06T12:31:28.420 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-06T12:31:28.420 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T12:31:28.420 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-06T12:31:28.420 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-06T12:31:28.420 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T12:31:28.420 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:31:28.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:28.421 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-06T12:31:28.438 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T12:31:28.438 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T12:31:28.438 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T12:31:28.438 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:31:28.438 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T12:31:28.439 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:31:28.439 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:31:28.439 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:31:28.439 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-06T12:31:28.439 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:31:28.439 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:31:28.439 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-06T12:31:28.439 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:31:28.439 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:31:28.439 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T12:31:28.439 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:31:28.439 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T12:31:28.439 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 41201sec preferred_lft 41201sec 2024-04-06T12:31:28.439 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::8592:df2c:92e2:dace/64 scope link noprefixroute 2024-04-06T12:31:28.440 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:31:28.440 INFO:teuthology.orchestra.run.smithi189.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T12:31:28.440 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff 2024-04-06T12:31:28.440 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-06T12:31:28.440 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:31:28.440 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::88e:b6ff:fe6f:1cb0/64 scope link 2024-04-06T12:31:28.440 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:31:28.440 INFO:teuthology.orchestra.run.smithi189.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T12:31:28.440 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 62:a5:8d:c5:1a:9c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T12:31:28.440 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::60a5:8dff:fec5:1a9c/64 scope link 2024-04-06T12:31:28.440 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:31:28.440 INFO:teuthology.orchestra.run.smithi189.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T12:31:28.440 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-06T12:31:28.440 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::1c7e:c6ff:febb:1dab/64 scope link 2024-04-06T12:31:28.440 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:31:28.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:28.441 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T12:31:28.504 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-06T12:31:28.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:28.506 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-06T12:31:28.566 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-06T12:31:28.566 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T12:31:28.566 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T12:31:28.567 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T12:31:28.567 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T12:31:28.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:28.567 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:31:28.567 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:31:28.567 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T12:31:28.567 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T12:31:28.567 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T12:31:28.567 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:31:28.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:28.673 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T12:31:28.690 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-06T12:31:28.690 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-06T12:31:28.690 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:31:28.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:28.690 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-04-06T12:31:28.757 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:31:28.757 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-06T12:31:28.814 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:31:28.814 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:31:28.814 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-06T12:31:28.870 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T12:31:28.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:28.871 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:31:28.957 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:31:28.957 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:31:28.957 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:31:28.966 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:31:28.966 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:31:28.966 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:31:28.966 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:31:28.966 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:31:28.985 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:31:28.985 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:31:28.985 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:31:28.985 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:31:28.985 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:31:28.985 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790" 2024-04-06T12:31:28.985 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2024-04-06T12:31:28.986 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e30041de-e9a3-4351-8ca7-29d7f3e3728c" will pass to kernel 2024-04-06T12:31:28.988 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T12:31:28.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:28.988 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-06T12:31:29.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:29.016 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:31:29.102 INFO:teuthology.orchestra.run.smithi189.stdout:client8751 2024-04-06T12:31:29.103 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:31:29.103 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:31:29.120 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-06T12:31:29.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.285+0000 7f3c9cbd9700 1 Processor -- start 2024-04-06T12:31:29.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.285+0000 7f3c9cbd9700 1 -- start start 2024-04-06T12:31:29.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.286+0000 7f3c9cbd9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3c9815a550 0x7f3c980cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:29.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.286+0000 7f3c9cbd9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3c980b4d70 0x7f3c980b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:29.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.286+0000 7f3c9cbd9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c980cdbe0 con 0x7f3c980cf900 2024-04-06T12:31:29.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.286+0000 7f3c9cbd9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c9815a940 con 0x7f3c980b4d70 2024-04-06T12:31:29.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.286+0000 7f3c9cbd9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c9815ac00 con 0x7f3c9815a550 2024-04-06T12:31:29.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.286+0000 7f3c96d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3c980b4d70 0x7f3c980b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:29.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.286+0000 7f3c96d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3c980b4d70 0x7f3c980b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56922/0 (socket says 172.21.15.189:56922) 2024-04-06T12:31:29.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.286+0000 7f3c96d9d700 1 -- 172.21.15.189:0/2226625770 learned_addr learned my addr 172.21.15.189:0/2226625770 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:29.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.286+0000 7f3c9659c700 1 --2- 172.21.15.189:0/2226625770 >> v2:172.21.15.192:3300/0 conn(0x7f3c9815a550 0x7f3c980cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:29.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.286+0000 7f3c96d9d700 1 -- 172.21.15.189:0/2226625770 >> v2:172.21.15.192:3300/0 conn(0x7f3c9815a550 msgr2=0x7f3c980cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:29.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.286+0000 7f3c96d9d700 1 --2- 172.21.15.189:0/2226625770 >> v2:172.21.15.192:3300/0 conn(0x7f3c9815a550 0x7f3c980cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:29.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.286+0000 7f3c96d9d700 1 -- 172.21.15.189:0/2226625770 >> v1:172.21.15.192:6789/0 conn(0x7f3c980cf900 legacy=0x7f3c980b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:29.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.286+0000 7f3c96d9d700 1 -- 172.21.15.189:0/2226625770 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9815aee0 con 0x7f3c980b4d70 2024-04-06T12:31:29.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.287+0000 7f3c96d9d700 1 --2- 172.21.15.189:0/2226625770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c980b4d70 0x7f3c980b2fc0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f3c98073470 tx=0x7f3c9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=748614444b22a912 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:29.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.287+0000 7f3c9559a700 1 -- 172.21.15.189:0/2226625770 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9000e070 con 0x7f3c980b4d70 2024-04-06T12:31:29.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.287+0000 7f3c9559a700 1 -- 172.21.15.189:0/2226625770 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c90003a20 con 0x7f3c980b4d70 2024-04-06T12:31:29.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.287+0000 7f3c9559a700 1 -- 172.21.15.189:0/2226625770 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c90003dd0 con 0x7f3c980b4d70 2024-04-06T12:31:29.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.287+0000 7f3c9cbd9700 1 -- 172.21.15.189:0/2226625770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c980b4d70 msgr2=0x7f3c980b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:29.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.287+0000 7f3c9cbd9700 1 --2- 172.21.15.189:0/2226625770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c980b4d70 0x7f3c980b2fc0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f3c98073470 tx=0x7f3c9000b330 comp rx=0 tx=0).stop 2024-04-06T12:31:29.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.287+0000 7f3c9cbd9700 1 -- 172.21.15.189:0/2226625770 shutdown_connections 2024-04-06T12:31:29.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.287+0000 7f3c9cbd9700 1 --2- 172.21.15.189:0/2226625770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c980b4d70 0x7f3c980b2fc0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:29.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.287+0000 7f3c9cbd9700 1 --2- 172.21.15.189:0/2226625770 >> v2:172.21.15.192:3300/0 conn(0x7f3c9815a550 0x7f3c980cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:29.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.287+0000 7f3c9cbd9700 1 -- 172.21.15.189:0/2226625770 >> 172.21.15.189:0/2226625770 conn(0x7f3c980d00b0 msgr2=0x7f3c980d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:29.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.287+0000 7f3c9cbd9700 1 -- 172.21.15.189:0/2226625770 shutdown_connections 2024-04-06T12:31:29.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.287+0000 7f3c9cbd9700 1 -- 172.21.15.189:0/2226625770 wait complete. 2024-04-06T12:31:29.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.289+0000 7f3c9cbd9700 1 Processor -- start 2024-04-06T12:31:29.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.289+0000 7f3c9cbd9700 1 -- start start 2024-04-06T12:31:29.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.289+0000 7f3c9cbd9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c980b4d70 0x7f3c9815c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:29.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.289+0000 7f3c9cbd9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c980cf900 0x7f3c9815ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:29.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.289+0000 7f3c9cbd9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c981523a0 0x7f3c98148f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:29.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.289+0000 7f3c9cbd9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c981534e0 con 0x7f3c981523a0 2024-04-06T12:31:29.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.289+0000 7f3c9cbd9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c98153900 con 0x7f3c980b4d70 2024-04-06T12:31:29.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.289+0000 7f3c9cbd9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c98152860 con 0x7f3c980cf900 2024-04-06T12:31:29.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.290+0000 7f3c96d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c981523a0 0x7f3c98148f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:29.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.290+0000 7f3c96d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c981523a0 0x7f3c98148f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56932/0 (socket says 172.21.15.189:56932) 2024-04-06T12:31:29.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.290+0000 7f3c96d9d700 1 -- 172.21.15.189:0/1082918967 learned_addr learned my addr 172.21.15.189:0/1082918967 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:29.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.290+0000 7f3c9659c700 1 --2- 172.21.15.189:0/1082918967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c980b4d70 0x7f3c9815c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:29.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.290+0000 7f3c95d9b700 1 --2- 172.21.15.189:0/1082918967 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c980cf900 0x7f3c9815ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:29.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.290+0000 7f3c96d9d700 1 -- 172.21.15.189:0/1082918967 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c980cf900 msgr2=0x7f3c9815ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:29.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.290+0000 7f3c96d9d700 1 --2- 172.21.15.189:0/1082918967 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c980cf900 0x7f3c9815ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:29.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.290+0000 7f3c96d9d700 1 -- 172.21.15.189:0/1082918967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c980b4d70 msgr2=0x7f3c9815c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:29.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.290+0000 7f3c96d9d700 1 --2- 172.21.15.189:0/1082918967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c980b4d70 0x7f3c9815c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:29.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.290+0000 7f3c96d9d700 1 -- 172.21.15.189:0/1082918967 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c90009b30 con 0x7f3c981523a0 2024-04-06T12:31:29.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.290+0000 7f3c9659c700 1 --2- 172.21.15.189:0/1082918967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c980b4d70 0x7f3c9815c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:29.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.290+0000 7f3c96d9d700 1 --2- 172.21.15.189:0/1082918967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c981523a0 0x7f3c98148f10 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000b370 tx=0x7f3c90004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:29.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.290+0000 7f3c86ffd700 1 -- 172.21.15.189:0/1082918967 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c90004440 con 0x7f3c981523a0 2024-04-06T12:31:29.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.290+0000 7f3c86ffd700 1 -- 172.21.15.189:0/1082918967 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c900045a0 con 0x7f3c981523a0 2024-04-06T12:31:29.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.290+0000 7f3c9cbd9700 1 -- 172.21.15.189:0/1082918967 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c98152b40 con 0x7f3c981523a0 2024-04-06T12:31:29.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.290+0000 7f3c86ffd700 1 -- 172.21.15.189:0/1082918967 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c90004950 con 0x7f3c981523a0 2024-04-06T12:31:29.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.290+0000 7f3c9cbd9700 1 -- 172.21.15.189:0/1082918967 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c98153100 con 0x7f3c981523a0 2024-04-06T12:31:29.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.291+0000 7f3c9cbd9700 1 -- 172.21.15.189:0/1082918967 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c98067bb0 con 0x7f3c981523a0 2024-04-06T12:31:29.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.292+0000 7f3c86ffd700 1 -- 172.21.15.189:0/1082918967 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f3c90012430 con 0x7f3c981523a0 2024-04-06T12:31:29.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.292+0000 7f3c86ffd700 1 --2- 172.21.15.189:0/1082918967 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3c7806adb0 0x7f3c7806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:29.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.292+0000 7f3c86ffd700 1 -- 172.21.15.189:0/1082918967 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3c9008ab30 con 0x7f3c981523a0 2024-04-06T12:31:29.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.292+0000 7f3c9659c700 1 --2- 172.21.15.189:0/1082918967 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3c7806adb0 0x7f3c7806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:29.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.292+0000 7f3c9659c700 1 --2- 172.21.15.189:0/1082918967 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3c7806adb0 0x7f3c7806d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3c981504e0 tx=0x7f3c8c008040 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:29.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.296+0000 7f3c86ffd700 1 -- 172.21.15.189:0/1082918967 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3c90055a40 con 0x7f3c981523a0 2024-04-06T12:31:29.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.522+0000 7f3c9cbd9700 1 -- 172.21.15.189:0/1082918967 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c9806fdd0 con 0x7f3c7806adb0 2024-04-06T12:31:29.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.523+0000 7f3c86ffd700 1 -- 172.21.15.189:0/1082918967 <== mgr.8376 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f3c8c015440 con 0x7f3c7806adb0 2024-04-06T12:31:29.521 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T12:31:29.522 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:29.522 INFO:teuthology.orchestra.run.smithi189.stdout: "name": "cephfs" 2024-04-06T12:31:29.522 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:29.522 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T12:31:29.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.525+0000 7f3c9cbd9700 1 -- 172.21.15.189:0/1082918967 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3c7806adb0 msgr2=0x7f3c7806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:29.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.525+0000 7f3c9cbd9700 1 --2- 172.21.15.189:0/1082918967 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3c7806adb0 0x7f3c7806d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3c981504e0 tx=0x7f3c8c008040 comp rx=0 tx=0).stop 2024-04-06T12:31:29.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.525+0000 7f3c9cbd9700 1 -- 172.21.15.189:0/1082918967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c981523a0 msgr2=0x7f3c98148f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:29.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.525+0000 7f3c9cbd9700 1 --2- 172.21.15.189:0/1082918967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c981523a0 0x7f3c98148f10 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000b370 tx=0x7f3c90004ad0 comp rx=0 tx=0).stop 2024-04-06T12:31:29.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.525+0000 7f3c9cbd9700 1 -- 172.21.15.189:0/1082918967 shutdown_connections 2024-04-06T12:31:29.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.525+0000 7f3c9cbd9700 1 --2- 172.21.15.189:0/1082918967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c981523a0 0x7f3c98148f10 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:29.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.525+0000 7f3c9cbd9700 1 --2- 172.21.15.189:0/1082918967 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3c7806adb0 0x7f3c7806d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:29.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.525+0000 7f3c9cbd9700 1 --2- 172.21.15.189:0/1082918967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c980b4d70 0x7f3c9815c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:29.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.525+0000 7f3c9cbd9700 1 --2- 172.21.15.189:0/1082918967 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c980cf900 0x7f3c9815ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:29.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.525+0000 7f3c9cbd9700 1 -- 172.21.15.189:0/1082918967 >> 172.21.15.189:0/1082918967 conn(0x7f3c980d00b0 msgr2=0x7f3c98154e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:29.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.526+0000 7f3c9cbd9700 1 -- 172.21.15.189:0/1082918967 shutdown_connections 2024-04-06T12:31:29.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.526+0000 7f3c9cbd9700 1 -- 172.21.15.189:0/1082918967 wait complete. 2024-04-06T12:31:29.536 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-06T12:31:29.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.661+0000 7f7735cf3700 1 Processor -- start 2024-04-06T12:31:29.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.661+0000 7f7735cf3700 1 -- start start 2024-04-06T12:31:29.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.662+0000 7f7735cf3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f77300cb870 0x7f773015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:29.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.662+0000 7f7735cf3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f77300cfa00 0x7f77300cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:29.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.662+0000 7f7735cf3700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77300cbd80 con 0x7f77300d11b0 2024-04-06T12:31:29.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.662+0000 7f7735cf3700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77300ccdc0 con 0x7f77300cfa00 2024-04-06T12:31:29.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.662+0000 7f7735cf3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77300d0b50 con 0x7f77300cb870 2024-04-06T12:31:29.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.662+0000 7f772ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f77300cfa00 0x7f77300cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:29.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.662+0000 7f772ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f77300cfa00 0x7f77300cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56946/0 (socket says 172.21.15.189:56946) 2024-04-06T12:31:29.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.662+0000 7f772ffff700 1 -- 172.21.15.189:0/2813701124 learned_addr learned my addr 172.21.15.189:0/2813701124 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:29.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.663+0000 7f772effd700 1 --2- 172.21.15.189:0/2813701124 >> v2:172.21.15.192:3300/0 conn(0x7f77300cb870 0x7f773015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:29.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.663+0000 7f772ffff700 1 -- 172.21.15.189:0/2813701124 >> v2:172.21.15.192:3300/0 conn(0x7f77300cb870 msgr2=0x7f773015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:29.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.663+0000 7f772ffff700 1 --2- 172.21.15.189:0/2813701124 >> v2:172.21.15.192:3300/0 conn(0x7f77300cb870 0x7f773015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:29.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.663+0000 7f772ffff700 1 -- 172.21.15.189:0/2813701124 >> v1:172.21.15.192:6789/0 conn(0x7f77300d11b0 legacy=0x7f77300d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:29.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.663+0000 7f772ffff700 1 -- 172.21.15.189:0/2813701124 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77300cef20 con 0x7f77300cfa00 2024-04-06T12:31:29.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.663+0000 7f772ffff700 1 --2- 172.21.15.189:0/2813701124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77300cfa00 0x7f77300cd560 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f773004fb40 tx=0x7f772800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcbc51e4ec5aa9fb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:29.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.663+0000 7f772e7fc700 1 -- 172.21.15.189:0/2813701124 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772800e070 con 0x7f77300cfa00 2024-04-06T12:31:29.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.663+0000 7f772e7fc700 1 -- 172.21.15.189:0/2813701124 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7728003a20 con 0x7f77300cfa00 2024-04-06T12:31:29.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.663+0000 7f772e7fc700 1 -- 172.21.15.189:0/2813701124 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7728003dd0 con 0x7f77300cfa00 2024-04-06T12:31:29.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.663+0000 7f7735cf3700 1 -- 172.21.15.189:0/2813701124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77300cfa00 msgr2=0x7f77300cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:29.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.663+0000 7f7735cf3700 1 --2- 172.21.15.189:0/2813701124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77300cfa00 0x7f77300cd560 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f773004fb40 tx=0x7f772800b330 comp rx=0 tx=0).stop 2024-04-06T12:31:29.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.664+0000 7f7735cf3700 1 -- 172.21.15.189:0/2813701124 shutdown_connections 2024-04-06T12:31:29.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.664+0000 7f7735cf3700 1 --2- 172.21.15.189:0/2813701124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77300cfa00 0x7f77300cd560 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:29.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.664+0000 7f7735cf3700 1 --2- 172.21.15.189:0/2813701124 >> v2:172.21.15.192:3300/0 conn(0x7f77300cb870 0x7f773015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:29.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.664+0000 7f7735cf3700 1 -- 172.21.15.189:0/2813701124 >> 172.21.15.189:0/2813701124 conn(0x7f77300b2db0 msgr2=0x7f77300b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:29.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.664+0000 7f7735cf3700 1 -- 172.21.15.189:0/2813701124 shutdown_connections 2024-04-06T12:31:29.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.664+0000 7f7735cf3700 1 -- 172.21.15.189:0/2813701124 wait complete. 2024-04-06T12:31:29.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.665+0000 7f7735cf3700 1 Processor -- start 2024-04-06T12:31:29.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.665+0000 7f7735cf3700 1 -- start start 2024-04-06T12:31:29.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.666+0000 7f7735cf3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77300cb870 0x7f7730154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:29.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.666+0000 7f7735cf3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77300cfa00 0x7f77300ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:29.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.666+0000 7f7735cf3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77300d11b0 0x7f7730157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:29.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.666+0000 7f7735cf3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77300b3ea0 con 0x7f77300cb870 2024-04-06T12:31:29.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.666+0000 7f7735cf3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77300c4d30 con 0x7f77300cfa00 2024-04-06T12:31:29.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.666+0000 7f7735cf3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77300cadc0 con 0x7f77300d11b0 2024-04-06T12:31:29.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.666+0000 7f772f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77300cb870 0x7f7730154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:29.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.666+0000 7f772f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77300cb870 0x7f7730154d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56952/0 (socket says 172.21.15.189:56952) 2024-04-06T12:31:29.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.666+0000 7f772f7fe700 1 -- 172.21.15.189:0/3826971830 learned_addr learned my addr 172.21.15.189:0/3826971830 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:29.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.667+0000 7f772effd700 1 --2- 172.21.15.189:0/3826971830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77300cfa00 0x7f77300ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:29.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.667+0000 7f772f7fe700 1 -- 172.21.15.189:0/3826971830 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77300d11b0 msgr2=0x7f7730157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:29.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.667+0000 7f772f7fe700 1 --2- 172.21.15.189:0/3826971830 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77300d11b0 0x7f7730157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:29.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.667+0000 7f772f7fe700 1 -- 172.21.15.189:0/3826971830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77300cfa00 msgr2=0x7f77300ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:29.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.667+0000 7f772f7fe700 1 --2- 172.21.15.189:0/3826971830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77300cfa00 0x7f77300ca450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:29.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.667+0000 7f772f7fe700 1 -- 172.21.15.189:0/3826971830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7728009b30 con 0x7f77300cb870 2024-04-06T12:31:29.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.667+0000 7f772f7fe700 1 --2- 172.21.15.189:0/3826971830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77300cb870 0x7f7730154d40 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f771800c4d0 tx=0x7f771800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:29.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.667+0000 7f7727fff700 1 -- 172.21.15.189:0/3826971830 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7718011790 con 0x7f77300cb870 2024-04-06T12:31:29.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.667+0000 7f7727fff700 1 -- 172.21.15.189:0/3826971830 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77180118f0 con 0x7f77300cb870 2024-04-06T12:31:29.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.667+0000 7f7735cf3700 1 -- 172.21.15.189:0/3826971830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77300cb0a0 con 0x7f77300cb870 2024-04-06T12:31:29.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.667+0000 7f7727fff700 1 -- 172.21.15.189:0/3826971830 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7718011ca0 con 0x7f77300cb870 2024-04-06T12:31:29.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.667+0000 7f7735cf3700 1 -- 172.21.15.189:0/3826971830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77300b43d0 con 0x7f77300cb870 2024-04-06T12:31:29.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.668+0000 7f7735cf3700 1 -- 172.21.15.189:0/3826971830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7730074990 con 0x7f77300cb870 2024-04-06T12:31:29.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.668+0000 7f7727fff700 1 -- 172.21.15.189:0/3826971830 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 75) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f771801c460 con 0x7f77300cb870 2024-04-06T12:31:29.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.669+0000 7f7727fff700 1 --2- 172.21.15.189:0/3826971830 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f771006adb0 0x7f771006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:29.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.669+0000 7f7727fff700 1 -- 172.21.15.189:0/3826971830 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7718089400 con 0x7f77300cb870 2024-04-06T12:31:29.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.669+0000 7f772effd700 1 --2- 172.21.15.189:0/3826971830 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f771006adb0 0x7f771006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:29.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.669+0000 7f772effd700 1 --2- 172.21.15.189:0/3826971830 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f771006adb0 0x7f771006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7730072fa0 tx=0x7f7720008040 comp rx=0 tx=0).ready entity=mgr.8376 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:29.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.673+0000 7f7727fff700 1 -- 172.21.15.189:0/3826971830 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f77180543a0 con 0x7f77300cb870 2024-04-06T12:31:29.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:29.928+0000 7f7735cf3700 1 -- 172.21.15.189:0/3826971830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f773006f7e0 con 0x7f77300cb870 2024-04-06T12:31:30.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.595+0000 7f7727fff700 1 -- 172.21.15.189:0/3826971830 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 76) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f7718054770 con 0x7f77300cb870 2024-04-06T12:31:30.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.596+0000 7f7727fff700 1 -- 172.21.15.189:0/3826971830 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v76) v1 ==== 92+0+0 (secure 0 0 0) 0x7f7718081670 con 0x7f77300cb870 2024-04-06T12:31:30.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.599+0000 7f7735cf3700 1 -- 172.21.15.189:0/3826971830 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f771006adb0 msgr2=0x7f771006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:30.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.599+0000 7f7735cf3700 1 --2- 172.21.15.189:0/3826971830 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f771006adb0 0x7f771006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7730072fa0 tx=0x7f7720008040 comp rx=0 tx=0).stop 2024-04-06T12:31:30.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.599+0000 7f7735cf3700 1 -- 172.21.15.189:0/3826971830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77300cb870 msgr2=0x7f7730154d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:30.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.599+0000 7f7735cf3700 1 --2- 172.21.15.189:0/3826971830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77300cb870 0x7f7730154d40 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f771800c4d0 tx=0x7f771800c890 comp rx=0 tx=0).stop 2024-04-06T12:31:30.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.599+0000 7f7735cf3700 1 -- 172.21.15.189:0/3826971830 shutdown_connections 2024-04-06T12:31:30.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.599+0000 7f7735cf3700 1 --2- 172.21.15.189:0/3826971830 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77300d11b0 0x7f7730157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:30.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.599+0000 7f7735cf3700 1 --2- 172.21.15.189:0/3826971830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77300cb870 0x7f7730154d40 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:30.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.599+0000 7f7735cf3700 1 --2- 172.21.15.189:0/3826971830 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f771006adb0 0x7f771006d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:30.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.599+0000 7f7735cf3700 1 --2- 172.21.15.189:0/3826971830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77300cfa00 0x7f77300ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:30.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.599+0000 7f7735cf3700 1 -- 172.21.15.189:0/3826971830 >> 172.21.15.189:0/3826971830 conn(0x7f77300b2db0 msgr2=0x7f77300b25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:30.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.599+0000 7f7735cf3700 1 -- 172.21.15.189:0/3826971830 shutdown_connections 2024-04-06T12:31:30.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.599+0000 7f7735cf3700 1 -- 172.21.15.189:0/3826971830 wait complete. 2024-04-06T12:31:30.611 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-06T12:31:30.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.743+0000 7f7543be5700 1 Processor -- start 2024-04-06T12:31:30.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.744+0000 7f7543be5700 1 -- start start 2024-04-06T12:31:30.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.744+0000 7f7543be5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f753c0b4ed0 0x7f753c0b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:30.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.744+0000 7f7543be5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f753c0d2880 0x7f753c0d2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:30.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.744+0000 7f7543be5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f753c0b5cc0 con 0x7f753c15d590 2024-04-06T12:31:30.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.744+0000 7f7543be5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f753c0b58c0 con 0x7f753c0b4ed0 2024-04-06T12:31:30.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.744+0000 7f7543be5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f753c0d1500 con 0x7f753c0d2880 2024-04-06T12:31:30.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.745+0000 7f7541180700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f753c0b4ed0 0x7f753c0b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:30.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.745+0000 7f7541180700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f753c0b4ed0 0x7f753c0b5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56988/0 (socket says 172.21.15.189:56988) 2024-04-06T12:31:30.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.745+0000 7f7541180700 1 -- 172.21.15.189:0/212386119 learned_addr learned my addr 172.21.15.189:0/212386119 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:30.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.745+0000 7f7542182700 1 --2- 172.21.15.189:0/212386119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f753c0d2880 0x7f753c0d2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:30.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.745+0000 7f7541180700 1 -- 172.21.15.189:0/212386119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f753c0d2880 msgr2=0x7f753c0d2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:30.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.745+0000 7f7541180700 1 --2- 172.21.15.189:0/212386119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f753c0d2880 0x7f753c0d2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:30.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.745+0000 7f7541180700 1 -- 172.21.15.189:0/212386119 >> v1:172.21.15.192:6789/0 conn(0x7f753c15d590 legacy=0x7f753c0b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:30.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.745+0000 7f7541180700 1 -- 172.21.15.189:0/212386119 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f753c0d17e0 con 0x7f753c0b4ed0 2024-04-06T12:31:30.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.745+0000 7f754097f700 1 -- 172.21.15.189:0/212386119 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260305834 0 0) 0x7f753c0b5cc0 con 0x7f753c15d590 2024-04-06T12:31:30.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.745+0000 7f7541180700 1 --2- 172.21.15.189:0/212386119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f753c0b4ed0 0x7f753c0b5380 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f752c0098f0 tx=0x7f752c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2323b13b6c2d21c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:30.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.745+0000 7f754097f700 1 -- 172.21.15.189:0/212386119 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f752c011830 con 0x7f753c0b4ed0 2024-04-06T12:31:30.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.745+0000 7f754097f700 1 -- 172.21.15.189:0/212386119 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f752c011990 con 0x7f753c0b4ed0 2024-04-06T12:31:30.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.745+0000 7f754097f700 1 -- 172.21.15.189:0/212386119 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f752c011d20 con 0x7f753c0b4ed0 2024-04-06T12:31:30.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.746+0000 7f7543be5700 1 -- 172.21.15.189:0/212386119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f753c0b4ed0 msgr2=0x7f753c0b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:30.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.746+0000 7f7543be5700 1 --2- 172.21.15.189:0/212386119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f753c0b4ed0 0x7f753c0b5380 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f752c0098f0 tx=0x7f752c009c00 comp rx=0 tx=0).stop 2024-04-06T12:31:30.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.746+0000 7f7543be5700 1 -- 172.21.15.189:0/212386119 shutdown_connections 2024-04-06T12:31:30.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.746+0000 7f7543be5700 1 --2- 172.21.15.189:0/212386119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f753c0d2880 0x7f753c0d2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:30.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.746+0000 7f7543be5700 1 --2- 172.21.15.189:0/212386119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f753c0b4ed0 0x7f753c0b5380 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:30.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.746+0000 7f7543be5700 1 -- 172.21.15.189:0/212386119 >> 172.21.15.189:0/212386119 conn(0x7f753c06bbc0 msgr2=0x7f753c06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:30.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.746+0000 7f7543be5700 1 -- 172.21.15.189:0/212386119 shutdown_connections 2024-04-06T12:31:30.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.746+0000 7f7543be5700 1 -- 172.21.15.189:0/212386119 wait complete. 2024-04-06T12:31:30.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.747+0000 7f7543be5700 1 Processor -- start 2024-04-06T12:31:30.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.748+0000 7f7543be5700 1 -- start start 2024-04-06T12:31:30.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.748+0000 7f7543be5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f753c0b4ed0 0x7f753c160260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:30.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.748+0000 7f7543be5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f753c0d2880 0x7f753c150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:30.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.748+0000 7f7543be5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f753c15d590 0x7f753c0cda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:30.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.748+0000 7f7543be5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f753c151720 con 0x7f753c0b4ed0 2024-04-06T12:31:30.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.748+0000 7f7543be5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f753c15e3d0 con 0x7f753c0d2880 2024-04-06T12:31:30.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.748+0000 7f7543be5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f753c15daa0 con 0x7f753c15d590 2024-04-06T12:31:30.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.749+0000 7f7541981700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f753c0b4ed0 0x7f753c160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:30.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.749+0000 7f7541981700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f753c0b4ed0 0x7f753c160260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57004/0 (socket says 172.21.15.189:57004) 2024-04-06T12:31:30.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.749+0000 7f7541981700 1 -- 172.21.15.189:0/2263186816 learned_addr learned my addr 172.21.15.189:0/2263186816 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:30.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.749+0000 7f7542182700 1 --2- 172.21.15.189:0/2263186816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f753c15d590 0x7f753c0cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:30.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.749+0000 7f7541180700 1 --2- 172.21.15.189:0/2263186816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f753c0d2880 0x7f753c150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:30.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.749+0000 7f7541981700 1 -- 172.21.15.189:0/2263186816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f753c15d590 msgr2=0x7f753c0cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:30.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.749+0000 7f7541981700 1 --2- 172.21.15.189:0/2263186816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f753c15d590 0x7f753c0cda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:30.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.749+0000 7f7541981700 1 -- 172.21.15.189:0/2263186816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f753c0d2880 msgr2=0x7f753c150c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:30.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.749+0000 7f7541981700 1 --2- 172.21.15.189:0/2263186816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f753c0d2880 0x7f753c150c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:30.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.749+0000 7f7541981700 1 -- 172.21.15.189:0/2263186816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752c009550 con 0x7f753c0b4ed0 2024-04-06T12:31:30.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.749+0000 7f7541981700 1 --2- 172.21.15.189:0/2263186816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f753c0b4ed0 0x7f753c160260 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f753800c550 tx=0x7f753800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:30.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.749+0000 7f75327fc700 1 -- 172.21.15.189:0/2263186816 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75380054e0 con 0x7f753c0b4ed0 2024-04-06T12:31:30.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.749+0000 7f75327fc700 1 -- 172.21.15.189:0/2263186816 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7538003680 con 0x7f753c0b4ed0 2024-04-06T12:31:30.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.749+0000 7f75327fc700 1 -- 172.21.15.189:0/2263186816 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7538003a30 con 0x7f753c0b4ed0 2024-04-06T12:31:30.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.749+0000 7f7543be5700 1 -- 172.21.15.189:0/2263186816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f753c15dd80 con 0x7f753c0b4ed0 2024-04-06T12:31:30.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.749+0000 7f7543be5700 1 -- 172.21.15.189:0/2263186816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f753c14ff70 con 0x7f753c0b4ed0 2024-04-06T12:31:30.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.750+0000 7f7543be5700 1 -- 172.21.15.189:0/2263186816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f753c074990 con 0x7f753c0b4ed0 2024-04-06T12:31:30.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.751+0000 7f75327fc700 1 -- 172.21.15.189:0/2263186816 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 76) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f7538003b90 con 0x7f753c0b4ed0 2024-04-06T12:31:30.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.751+0000 7f75327fc700 1 --2- 172.21.15.189:0/2263186816 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f752006ae00 0x7f752006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:30.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.751+0000 7f7541180700 1 -- 172.21.15.189:0/2263186816 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f752006ae00 msgr2=0x7f752006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:30.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.751+0000 7f75327fc700 1 -- 172.21.15.189:0/2263186816 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f75380888c0 con 0x7f753c0b4ed0 2024-04-06T12:31:30.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.751+0000 7f7541180700 1 --2- 172.21.15.189:0/2263186816 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f752006ae00 0x7f752006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:30.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.756+0000 7f75327fc700 1 -- 172.21.15.189:0/2263186816 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f75380538e0 con 0x7f753c0b4ed0 2024-04-06T12:31:30.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.952+0000 7f7541180700 1 -- 172.21.15.189:0/2263186816 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f752006ae00 msgr2=0x7f752006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:30.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.952+0000 7f7541180700 1 --2- 172.21.15.189:0/2263186816 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f752006ae00 0x7f752006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:30.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.961+0000 7f7543be5700 1 -- 172.21.15.189:0/2263186816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f753c0b73c0 con 0x7f753c0b4ed0 2024-04-06T12:31:30.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.965+0000 7f75327fc700 1 -- 172.21.15.189:0/2263186816 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v47)=0 v47) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7538017070 con 0x7f753c0b4ed0 2024-04-06T12:31:30.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.967+0000 7f7543be5700 1 -- 172.21.15.189:0/2263186816 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f752006ae00 msgr2=0x7f752006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:30.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.967+0000 7f7543be5700 1 --2- 172.21.15.189:0/2263186816 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f752006ae00 0x7f752006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:30.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.967+0000 7f7543be5700 1 -- 172.21.15.189:0/2263186816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f753c0b4ed0 msgr2=0x7f753c160260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:30.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.967+0000 7f7543be5700 1 --2- 172.21.15.189:0/2263186816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f753c0b4ed0 0x7f753c160260 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f753800c550 tx=0x7f753800c860 comp rx=0 tx=0).stop 2024-04-06T12:31:30.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.967+0000 7f7543be5700 1 -- 172.21.15.189:0/2263186816 shutdown_connections 2024-04-06T12:31:30.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.967+0000 7f7543be5700 1 --2- 172.21.15.189:0/2263186816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f753c15d590 0x7f753c0cda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:30.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.967+0000 7f7543be5700 1 --2- 172.21.15.189:0/2263186816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f753c0b4ed0 0x7f753c160260 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:30.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.967+0000 7f7543be5700 1 --2- 172.21.15.189:0/2263186816 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f752006ae00 0x7f752006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:30.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.967+0000 7f7543be5700 1 --2- 172.21.15.189:0/2263186816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f753c0d2880 0x7f753c150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:30.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.967+0000 7f7543be5700 1 -- 172.21.15.189:0/2263186816 >> 172.21.15.189:0/2263186816 conn(0x7f753c06bbc0 msgr2=0x7f753c0d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:30.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.967+0000 7f7543be5700 1 -- 172.21.15.189:0/2263186816 shutdown_connections 2024-04-06T12:31:30.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:30.967+0000 7f7543be5700 1 -- 172.21.15.189:0/2263186816 wait complete. 2024-04-06T12:31:30.978 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-06T12:31:30.981 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:30.980+0000 7fc95140e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:31:30.992 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:30.994+0000 7fb755625200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:31:31.057 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:31.059+0000 7fc95140e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:31:31.074 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:31.076+0000 7fb755625200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:31:31.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.110+0000 7f0855052700 1 Processor -- start 2024-04-06T12:31:31.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.110+0000 7f0855052700 1 -- start start 2024-04-06T12:31:31.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.111+0000 7f0855052700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f08500b5380 0x7f08500d2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:31.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.111+0000 7f0855052700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08500b5cc0 con 0x7f085015d590 2024-04-06T12:31:31.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.111+0000 7f0855052700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08500b57d0 con 0x7f08500b4ed0 2024-04-06T12:31:31.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.111+0000 7f0855052700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08500d1500 con 0x7f08500b5380 2024-04-06T12:31:31.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.111+0000 7f084f59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f08500b5380 0x7f08500d2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:31.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.111+0000 7f084ed9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f085015d590 0x7f08500b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52480/0 (socket says 172.21.15.189:52480) 2024-04-06T12:31:31.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.111+0000 7f084f59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f08500b5380 0x7f08500d2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57008/0 (socket says 172.21.15.189:57008) 2024-04-06T12:31:31.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.111+0000 7f084ed9d700 1 -- 172.21.15.189:0/3115901796 learned_addr learned my addr 172.21.15.189:0/3115901796 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:31.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.111+0000 7f084f59e700 1 -- 172.21.15.189:0/3115901796 >> v1:172.21.15.192:6789/0 conn(0x7f08500b4ed0 legacy=0x7f08500d2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:31.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.111+0000 7f084f59e700 1 -- 172.21.15.189:0/3115901796 >> v1:172.21.15.189:6789/0 conn(0x7f085015d590 legacy=0x7f08500b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:31.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.111+0000 7f084dd9b700 1 -- 172.21.15.189:0/3115901796 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999745701 0 0) 0x7f08500b57d0 con 0x7f08500b4ed0 2024-04-06T12:31:31.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.111+0000 7f084f59e700 1 -- 172.21.15.189:0/3115901796 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08500d17e0 con 0x7f08500b5380 2024-04-06T12:31:31.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.111+0000 7f084f59e700 1 --2- 172.21.15.189:0/3115901796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08500b5380 0x7f08500d2f90 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f0848009d90 tx=0x7f084800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fea7f3538e1885b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:31.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.111+0000 7f084dd9b700 1 -- 172.21.15.189:0/3115901796 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084800e070 con 0x7f08500b5380 2024-04-06T12:31:31.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.111+0000 7f084dd9b700 1 -- 172.21.15.189:0/3115901796 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084800bad0 con 0x7f08500b5380 2024-04-06T12:31:31.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.111+0000 7f084dd9b700 1 -- 172.21.15.189:0/3115901796 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084800be80 con 0x7f08500b5380 2024-04-06T12:31:31.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.112+0000 7f0855052700 1 -- 172.21.15.189:0/3115901796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08500b5380 msgr2=0x7f08500d2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:31.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.112+0000 7f0855052700 1 --2- 172.21.15.189:0/3115901796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08500b5380 0x7f08500d2f90 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f0848009d90 tx=0x7f084800b330 comp rx=0 tx=0).stop 2024-04-06T12:31:31.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.112+0000 7f0855052700 1 -- 172.21.15.189:0/3115901796 shutdown_connections 2024-04-06T12:31:31.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.112+0000 7f0855052700 1 --2- 172.21.15.189:0/3115901796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08500b5380 0x7f08500d2f90 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:31.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.112+0000 7f0855052700 1 -- 172.21.15.189:0/3115901796 >> 172.21.15.189:0/3115901796 conn(0x7f085006bbc0 msgr2=0x7f085006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:31.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.112+0000 7f0855052700 1 -- 172.21.15.189:0/3115901796 shutdown_connections 2024-04-06T12:31:31.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.112+0000 7f0855052700 1 -- 172.21.15.189:0/3115901796 wait complete. 2024-04-06T12:31:31.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.114+0000 7f0855052700 1 Processor -- start 2024-04-06T12:31:31.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.114+0000 7f0855052700 1 -- start start 2024-04-06T12:31:31.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f0855052700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08500b4ed0 0x7f085015ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:31.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f0855052700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08500b5380 0x7f085015fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:31.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f0855052700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f085015d590 0x7f08501f2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:31.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f0855052700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08500cd390 con 0x7f085015d590 2024-04-06T12:31:31.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f0855052700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08500ca5c0 con 0x7f08500b5380 2024-04-06T12:31:31.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f0855052700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08500ca8a0 con 0x7f08500b4ed0 2024-04-06T12:31:31.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f084e59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08500b5380 0x7f085015fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:31.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f084ed9d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08500b4ed0 0x7f085015ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:31.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f084f59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f085015d590 0x7f08501f2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:31.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f084e59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08500b5380 0x7f085015fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:59014/0 (socket says 172.21.15.189:59014) 2024-04-06T12:31:31.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f084ed9d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08500b4ed0 0x7f085015ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:43830/0 (socket says 172.21.15.189:43830) 2024-04-06T12:31:31.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f084f59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f085015d590 0x7f08501f2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57022/0 (socket says 172.21.15.189:57022) 2024-04-06T12:31:31.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f084e59c700 1 -- 172.21.15.189:0/2800822822 learned_addr learned my addr 172.21.15.189:0/2800822822 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:31.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f084ed9d700 1 -- 172.21.15.189:0/2800822822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08500b5380 msgr2=0x7f085015fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:31.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f084ed9d700 1 --2- 172.21.15.189:0/2800822822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08500b5380 0x7f085015fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:31.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f084ed9d700 1 -- 172.21.15.189:0/2800822822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f085015d590 msgr2=0x7f08501f2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:31.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f084ed9d700 1 --2- 172.21.15.189:0/2800822822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f085015d590 0x7f08501f2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:31.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f084ed9d700 1 -- 172.21.15.189:0/2800822822 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0848009a70 con 0x7f08500b4ed0 2024-04-06T12:31:31.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f084e59c700 1 --2- 172.21.15.189:0/2800822822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08500b5380 0x7f085015fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:31.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f084f59e700 1 --2- 172.21.15.189:0/2800822822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f085015d590 0x7f08501f2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:31.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f084ed9d700 1 --2- 172.21.15.189:0/2800822822 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08500b4ed0 0x7f085015ee20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f084000c4a0 tx=0x7f084000c7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:31.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f08377fe700 1 -- 172.21.15.189:0/2800822822 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0840004c80 con 0x7f08500b4ed0 2024-04-06T12:31:31.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f08377fe700 1 -- 172.21.15.189:0/2800822822 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0840004de0 con 0x7f08500b4ed0 2024-04-06T12:31:31.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.116+0000 7f0855052700 1 -- 172.21.15.189:0/2800822822 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08500cab80 con 0x7f08500b4ed0 2024-04-06T12:31:31.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.117+0000 7f08377fe700 1 -- 172.21.15.189:0/2800822822 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0840007310 con 0x7f08500b4ed0 2024-04-06T12:31:31.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.117+0000 7f0855052700 1 -- 172.21.15.189:0/2800822822 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08500cb140 con 0x7f08500b4ed0 2024-04-06T12:31:31.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.118+0000 7f0855052700 1 -- 172.21.15.189:0/2800822822 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08500cb360 con 0x7f08500b4ed0 2024-04-06T12:31:31.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.118+0000 7f08377fe700 1 -- 172.21.15.189:0/2800822822 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 76) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f0840007470 con 0x7f08500b4ed0 2024-04-06T12:31:31.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.118+0000 7f08377fe700 1 --2- 172.21.15.189:0/2800822822 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f083c06ae00 0x7f083c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:31.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.118+0000 7f084e59c700 1 -- 172.21.15.189:0/2800822822 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f083c06ae00 msgr2=0x7f083c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:31.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.118+0000 7f08377fe700 1 -- 172.21.15.189:0/2800822822 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f08400891e0 con 0x7f08500b4ed0 2024-04-06T12:31:31.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.118+0000 7f084e59c700 1 --2- 172.21.15.189:0/2800822822 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f083c06ae00 0x7f083c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:31.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.123+0000 7f08377fe700 1 -- 172.21.15.189:0/2800822822 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f08400540f0 con 0x7f08500b4ed0 2024-04-06T12:31:31.130 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:31.132+0000 7fc95140e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:31:31.153 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:31.155+0000 7fb755625200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:31:31.198 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:31.200+0000 7fc95140e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:31:31.224 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:31.226+0000 7fb755625200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:31:31.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.319+0000 7f084e59c700 1 -- 172.21.15.189:0/2800822822 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f083c06ae00 msgr2=0x7f083c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:31.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.319+0000 7f084e59c700 1 --2- 172.21.15.189:0/2800822822 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f083c06ae00 0x7f083c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:31.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.327+0000 7f0855052700 1 -- 172.21.15.189:0/2800822822 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f0850074990 con 0x7f08500b4ed0 2024-04-06T12:31:31.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.331+0000 7f08377fe700 1 -- 172.21.15.189:0/2800822822 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v48)=0 v48) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0840057710 con 0x7f08500b4ed0 2024-04-06T12:31:31.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.334+0000 7f0855052700 1 -- 172.21.15.189:0/2800822822 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f083c06ae00 msgr2=0x7f083c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:31.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.334+0000 7f0855052700 1 --2- 172.21.15.189:0/2800822822 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f083c06ae00 0x7f083c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:31.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.334+0000 7f0855052700 1 -- 172.21.15.189:0/2800822822 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08500b4ed0 msgr2=0x7f085015ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:31.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.334+0000 7f0855052700 1 --2- 172.21.15.189:0/2800822822 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08500b4ed0 0x7f085015ee20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f084000c4a0 tx=0x7f084000c7b0 comp rx=0 tx=0).stop 2024-04-06T12:31:31.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.334+0000 7f0855052700 1 -- 172.21.15.189:0/2800822822 shutdown_connections 2024-04-06T12:31:31.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.334+0000 7f0855052700 1 --2- 172.21.15.189:0/2800822822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f085015d590 0x7f08501f2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:31.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.334+0000 7f0855052700 1 --2- 172.21.15.189:0/2800822822 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08500b4ed0 0x7f085015ee20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:31.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.334+0000 7f0855052700 1 --2- 172.21.15.189:0/2800822822 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f083c06ae00 0x7f083c06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:31.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.334+0000 7f0855052700 1 --2- 172.21.15.189:0/2800822822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08500b5380 0x7f085015fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:31.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.334+0000 7f0855052700 1 -- 172.21.15.189:0/2800822822 >> 172.21.15.189:0/2800822822 conn(0x7f085006bbc0 msgr2=0x7f085015c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:31.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.334+0000 7f0855052700 1 -- 172.21.15.189:0/2800822822 shutdown_connections 2024-04-06T12:31:31.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.334+0000 7f0855052700 1 -- 172.21.15.189:0/2800822822 wait complete. 2024-04-06T12:31:31.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:31.345 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/minutes) 2024-04-06T12:31:31.374 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:31.376+0000 7fc95140e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:31:31.406 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:31.408+0000 7fb755625200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:31:31.408 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/minutes 1m 2024-04-06T12:31:31.531 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:31.532+0000 7fc95140e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:31:31.570 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:31.572+0000 7fb755625200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:31:31.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.577+0000 7fb273788700 1 Processor -- start 2024-04-06T12:31:31.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.578+0000 7fb273788700 1 -- start start 2024-04-06T12:31:31.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.579+0000 7fb273788700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb26c06ae40 0x7fb26c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:31.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.579+0000 7fb273788700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb26c15ddf0 0x7fb26c15e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:31.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.579+0000 7fb273788700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26c0b3990 con 0x7fb26c15d7b0 2024-04-06T12:31:31.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.579+0000 7fb273788700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb26c0b3060 con 0x7fb26c15ddf0 2024-04-06T12:31:31.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.579+0000 7fb273788700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb26c0b3340 con 0x7fb26c06ae40 2024-04-06T12:31:31.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.579+0000 7fb271d25700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb26c15ddf0 0x7fb26c15e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:31.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.579+0000 7fb271d25700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb26c15ddf0 0x7fb26c15e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57038/0 (socket says 172.21.15.189:57038) 2024-04-06T12:31:31.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.579+0000 7fb271d25700 1 -- 172.21.15.189:0/3711892253 learned_addr learned my addr 172.21.15.189:0/3711892253 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:31.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.579+0000 7fb270d23700 1 --2- 172.21.15.189:0/3711892253 >> v2:172.21.15.192:3300/0 conn(0x7fb26c06ae40 0x7fb26c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:31.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.580+0000 7fb271d25700 1 -- 172.21.15.189:0/3711892253 >> v2:172.21.15.192:3300/0 conn(0x7fb26c06ae40 msgr2=0x7fb26c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:31.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.580+0000 7fb271d25700 1 --2- 172.21.15.189:0/3711892253 >> v2:172.21.15.192:3300/0 conn(0x7fb26c06ae40 0x7fb26c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:31.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.580+0000 7fb271d25700 1 -- 172.21.15.189:0/3711892253 >> v1:172.21.15.192:6789/0 conn(0x7fb26c15d7b0 legacy=0x7fb26c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:31.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.580+0000 7fb271d25700 1 -- 172.21.15.189:0/3711892253 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c0b3620 con 0x7fb26c15ddf0 2024-04-06T12:31:31.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.580+0000 7fb271d25700 1 --2- 172.21.15.189:0/3711892253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb26c15ddf0 0x7fb26c15e240 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fb26c0afc00 tx=0x7fb26400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33fdc2e8e2184afb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:31.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.580+0000 7fb263fff700 1 -- 172.21.15.189:0/3711892253 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26400e070 con 0x7fb26c15ddf0 2024-04-06T12:31:31.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.580+0000 7fb263fff700 1 -- 172.21.15.189:0/3711892253 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26400b910 con 0x7fb26c15ddf0 2024-04-06T12:31:31.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.580+0000 7fb263fff700 1 -- 172.21.15.189:0/3711892253 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26400bcc0 con 0x7fb26c15ddf0 2024-04-06T12:31:31.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.581+0000 7fb273788700 1 -- 172.21.15.189:0/3711892253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb26c15ddf0 msgr2=0x7fb26c15e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:31.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.581+0000 7fb273788700 1 --2- 172.21.15.189:0/3711892253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb26c15ddf0 0x7fb26c15e240 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fb26c0afc00 tx=0x7fb26400b330 comp rx=0 tx=0).stop 2024-04-06T12:31:31.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.581+0000 7fb273788700 1 -- 172.21.15.189:0/3711892253 shutdown_connections 2024-04-06T12:31:31.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.581+0000 7fb273788700 1 --2- 172.21.15.189:0/3711892253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb26c15ddf0 0x7fb26c15e240 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:31.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.581+0000 7fb273788700 1 --2- 172.21.15.189:0/3711892253 >> v2:172.21.15.192:3300/0 conn(0x7fb26c06ae40 0x7fb26c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:31.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.581+0000 7fb273788700 1 -- 172.21.15.189:0/3711892253 >> 172.21.15.189:0/3711892253 conn(0x7fb26c0d00b0 msgr2=0x7fb26c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:31.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.581+0000 7fb273788700 1 -- 172.21.15.189:0/3711892253 shutdown_connections 2024-04-06T12:31:31.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.581+0000 7fb273788700 1 -- 172.21.15.189:0/3711892253 wait complete. 2024-04-06T12:31:31.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.582+0000 7fb273788700 1 Processor -- start 2024-04-06T12:31:31.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.583+0000 7fb273788700 1 -- start start 2024-04-06T12:31:31.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.584+0000 7fb273788700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb26c06ae40 0x7fb26c151350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:31.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.584+0000 7fb273788700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb26c15d7b0 0x7fb26c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:31.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.584+0000 7fb273788700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb26c15ddf0 0x7fb26c15fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:31.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.584+0000 7fb273788700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c15b340 con 0x7fb26c06ae40 2024-04-06T12:31:31.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.584+0000 7fb273788700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c15a730 con 0x7fb26c15ddf0 2024-04-06T12:31:31.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.584+0000 7fb273788700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c15aa10 con 0x7fb26c15d7b0 2024-04-06T12:31:31.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.584+0000 7fb271524700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb26c06ae40 0x7fb26c151350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:31.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.584+0000 7fb271524700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb26c06ae40 0x7fb26c151350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57042/0 (socket says 172.21.15.189:57042) 2024-04-06T12:31:31.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.584+0000 7fb271524700 1 -- 172.21.15.189:0/1533408404 learned_addr learned my addr 172.21.15.189:0/1533408404 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:31.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.584+0000 7fb270d23700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb26c15d7b0 0x7fb26c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:31.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.584+0000 7fb271d25700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb26c15ddf0 0x7fb26c15fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:31.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.584+0000 7fb271524700 1 -- 172.21.15.189:0/1533408404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb26c15d7b0 msgr2=0x7fb26c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:31.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.584+0000 7fb271524700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb26c15d7b0 0x7fb26c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:31.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.584+0000 7fb271524700 1 -- 172.21.15.189:0/1533408404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb26c15ddf0 msgr2=0x7fb26c15fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:31.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.584+0000 7fb271524700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb26c15ddf0 0x7fb26c15fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:31.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.584+0000 7fb271524700 1 -- 172.21.15.189:0/1533408404 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb264009b30 con 0x7fb26c06ae40 2024-04-06T12:31:31.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.584+0000 7fb271524700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb26c06ae40 0x7fb26c151350 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fb26800c4e0 tx=0x7fb26800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:31.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.585+0000 7fb261ffb700 1 -- 172.21.15.189:0/1533408404 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2680052d0 con 0x7fb26c06ae40 2024-04-06T12:31:31.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.585+0000 7fb261ffb700 1 -- 172.21.15.189:0/1533408404 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb268005430 con 0x7fb26c06ae40 2024-04-06T12:31:31.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.585+0000 7fb261ffb700 1 -- 172.21.15.189:0/1533408404 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2680038c0 con 0x7fb26c06ae40 2024-04-06T12:31:31.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.585+0000 7fb273788700 1 -- 172.21.15.189:0/1533408404 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb26c15acf0 con 0x7fb26c06ae40 2024-04-06T12:31:31.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.585+0000 7fb273788700 1 -- 172.21.15.189:0/1533408404 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb26c151af0 con 0x7fb26c06ae40 2024-04-06T12:31:31.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.586+0000 7fb273788700 1 -- 172.21.15.189:0/1533408404 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb26c067bb0 con 0x7fb26c06ae40 2024-04-06T12:31:31.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.586+0000 7fb261ffb700 1 -- 172.21.15.189:0/1533408404 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 76) v1 ==== 89264+0+0 (secure 0 0 0) 0x7fb268003a20 con 0x7fb26c06ae40 2024-04-06T12:31:31.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.586+0000 7fb261ffb700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25006ae00 0x7fb25006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:31.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.586+0000 7fb270d23700 1 -- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25006ae00 msgr2=0x7fb25006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:31.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.586+0000 7fb261ffb700 1 -- 172.21.15.189:0/1533408404 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb2680888f0 con 0x7fb26c06ae40 2024-04-06T12:31:31.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.587+0000 7fb270d23700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25006ae00 0x7fb25006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:31.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.591+0000 7fb261ffb700 1 -- 172.21.15.189:0/1533408404 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb268053910 con 0x7fb26c06ae40 2024-04-06T12:31:31.643 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:31.645+0000 7fc95140e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:31:31.688 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:31.690+0000 7fb755625200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:31:31.724 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:31.726+0000 7fc95140e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:31:31.773 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:31.775+0000 7fb755625200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:31:31.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.788+0000 7fb270d23700 1 -- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25006ae00 msgr2=0x7fb25006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:31.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.788+0000 7fb270d23700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25006ae00 0x7fb25006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:31.801 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:31.804+0000 7fc95140e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:31:31.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:31.822+0000 7fb273788700 1 -- 172.21.15.189:0/1533408404 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb26c0717b0 con 0x7fb25006ae00 2024-04-06T12:31:31.850 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:31.852+0000 7fb755625200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:31:32.004 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:32.006+0000 7fc95140e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:31:32.052 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:32.054+0000 7fb755625200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:31:32.081 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:32.083+0000 7fc95140e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:31:32.128 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:32.130+0000 7fb755625200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:31:32.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:32.189+0000 7fb270d23700 1 -- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25006ae00 msgr2=0x7fb25006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:32.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:32.189+0000 7fb270d23700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25006ae00 0x7fb25006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-06T12:31:32.399 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:32.401+0000 7fc95140e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:31:32.447 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:32.449+0000 7fb755625200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:31:32.716 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:32.718+0000 7fc95140e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:31:32.766 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:32.768+0000 7fb755625200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:31:32.877 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:32.879+0000 7fc95140e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:31:32.928 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:32.930+0000 7fb755625200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:31:32.949 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:32.951+0000 7fc95140e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:31:32.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:32.990+0000 7fb270d23700 1 -- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25006ae00 msgr2=0x7fb25006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:32.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:32.990+0000 7fb270d23700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25006ae00 0x7fb25006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-06T12:31:33.000 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:33.002+0000 7fb755625200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:31:33.563 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:33.565+0000 7fc95140e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:31:33.615 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:33.617+0000 7fb755625200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:31:33.640 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:33.642+0000 7fc95140e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:31:33.694 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:33.696+0000 7fb755625200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:31:33.765 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:33.767+0000 7fc95140e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:31:33.818 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:33.820+0000 7fb755625200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:31:34.273 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:34.276+0000 7fc95140e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:31:34.326 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:34.328+0000 7fb755625200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:31:34.426 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:34.429+0000 7fc95140e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:31:34.481 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:34.483+0000 7fb755625200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:31:34.580 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:34.583+0000 7fc95140e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:31:34.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:34.591+0000 7fb270d23700 1 -- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25006ae00 msgr2=0x7fb25006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:34.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:34.591+0000 7fb270d23700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25006ae00 0x7fb25006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-06T12:31:34.638 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:34.640+0000 7fb755625200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:31:35.299 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:35.302+0000 7fc95140e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:31:35.358 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:35.360+0000 7fb755625200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:31:35.538 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:35.541+0000 7fc95140e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:31:35.600 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:35.602+0000 7fb755625200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:31:35.613 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:35.616+0000 7fc95140e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:31:35.677 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:35.679+0000 7fb755625200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:31:35.983 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:35.986+0000 7fc95140e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:31:36.052 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:36.053+0000 7fb755625200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:31:36.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:36.055+0000 7fb261ffb700 1 -- 172.21.15.189:0/1533408404 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 77) v1 ==== 89264+0+0 (secure 0 0 0) 0x7fb268053ce0 con 0x7fb26c06ae40 2024-04-06T12:31:36.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:36.073+0000 7fb261ffb700 1 -- 172.21.15.189:0/1533408404 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mgrmap(e 78) v1 ==== 88873+0+0 (secure 0 0 0) 0x7fb268086ca0 con 0x7fb26c06ae40 2024-04-06T12:31:36.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:36.073+0000 7fb261ffb700 1 -- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25006ae00 msgr2=0x7fb25006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:36.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:36.073+0000 7fb261ffb700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25006ae00 0x7fb25006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:36.211 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:36.212+0000 7fb6c8533700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:36.211 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:36.212+0000 7fb6c8533700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:36.212 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:36.212+0000 7fb6c8533700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:36.212 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:36.212+0000 7fb6cb539700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:36.212 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:36.212+0000 7fb6c8533700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:36.212 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:36.212+0000 7fb6cb539700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:36.212 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:36.212+0000 7fb6cb539700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:36.212 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:36.212+0000 7fb6cb539700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:36.212 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:36.212+0000 7fb6cb539700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:36.213 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:36.212+0000 7fb6c8533700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:37.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.080+0000 7fb261ffb700 1 -- 172.21.15.189:0/1533408404 <== mon.0 v2:172.21.15.189:3300/0 9 ==== mgrmap(e 79) v1 ==== 89000+0+0 (secure 0 0 0) 0x7fb2680545b0 con 0x7fb26c06ae40 2024-04-06T12:31:37.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.080+0000 7fb261ffb700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25006ae00 0x7fb25006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:37.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.080+0000 7fb261ffb700 1 -- 172.21.15.189:0/1533408404 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb25001cd40 con 0x7fb25006ae00 2024-04-06T12:31:37.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.082+0000 7fb270d23700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25006ae00 0x7fb25006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:37.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.084+0000 7fb270d23700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25006ae00 0x7fb25006d2c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb26c15c3c0 tx=0x7fb25c009210 comp rx=0 tx=0).ready entity=mgr.8775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:37.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.150+0000 7fb261ffb700 1 -- 172.21.15.189:0/1533408404 <== mgr.8775 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7fb25001cd40 con 0x7fb25006ae00 2024-04-06T12:31:37.148 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path snap_test_dir1/minutes 2024-04-06T12:31:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.152+0000 7fb273788700 1 -- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25006ae00 msgr2=0x7fb25006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.152+0000 7fb273788700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25006ae00 0x7fb25006d2c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb26c15c3c0 tx=0x7fb25c009210 comp rx=0 tx=0).stop 2024-04-06T12:31:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.152+0000 7fb273788700 1 -- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb26c06ae40 msgr2=0x7fb26c151350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.152+0000 7fb273788700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb26c06ae40 0x7fb26c151350 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fb26800c4e0 tx=0x7fb26800c7f0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.152+0000 7fb273788700 1 -- 172.21.15.189:0/1533408404 shutdown_connections 2024-04-06T12:31:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.152+0000 7fb273788700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb25006ae00 0x7fb25006d2c0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.153+0000 7fb273788700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb26c15ddf0 0x7fb26c15fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.153+0000 7fb273788700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb26c06ae40 0x7fb26c151350 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.153+0000 7fb273788700 1 --2- 172.21.15.189:0/1533408404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb26c15d7b0 0x7fb26c15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.153+0000 7fb273788700 1 -- 172.21.15.189:0/1533408404 >> 172.21.15.189:0/1533408404 conn(0x7fb26c0d00b0 msgr2=0x7fb26c0ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.153+0000 7fb273788700 1 -- 172.21.15.189:0/1533408404 shutdown_connections 2024-04-06T12:31:37.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.153+0000 7fb273788700 1 -- 172.21.15.189:0/1533408404 wait complete. 2024-04-06T12:31:37.164 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/minutes 2024-04-06T12:31:37.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:37.164 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/hourly) 2024-04-06T12:31:37.183 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/hourly 1h 2024-04-06T12:31:37.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.350+0000 7f3a21e51700 1 Processor -- start 2024-04-06T12:31:37.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.350+0000 7f3a21e51700 1 -- start start 2024-04-06T12:31:37.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.351+0000 7f3a21e51700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3a1c15d7b0 0x7f3a1c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:37.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.351+0000 7f3a21e51700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3a1c06a730 0x7f3a1c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:37.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.351+0000 7f3a21e51700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a1c15ddf0 0x7f3a1c15e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:37.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.351+0000 7f3a21e51700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a1c06b0f0 con 0x7f3a1c06a730 2024-04-06T12:31:37.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.351+0000 7f3a21e51700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a1c0b3990 con 0x7f3a1c15d7b0 2024-04-06T12:31:37.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.351+0000 7f3a21e51700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c0b3060 con 0x7f3a1c15ddf0 2024-04-06T12:31:37.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.351+0000 7f3a1affd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3a1c06a730 0x7f3a1c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:37.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.351+0000 7f3a1affd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3a1c06a730 0x7f3a1c06abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57144/0 (socket says 172.21.15.189:57144) 2024-04-06T12:31:37.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.351+0000 7f3a1affd700 1 -- 172.21.15.189:0/463726414 learned_addr learned my addr 172.21.15.189:0/463726414 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:37.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.351+0000 7f3a1bfff700 1 --2- 172.21.15.189:0/463726414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a1c15ddf0 0x7f3a1c15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:37.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.351+0000 7f3a1b7fe700 1 --2- 172.21.15.189:0/463726414 >> v2:172.21.15.192:3300/0 conn(0x7f3a1c15d7b0 0x7f3a1c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:37.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.351+0000 7f3a1affd700 1 -- 172.21.15.189:0/463726414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a1c15ddf0 msgr2=0x7f3a1c15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:37.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.351+0000 7f3a1affd700 1 --2- 172.21.15.189:0/463726414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a1c15ddf0 0x7f3a1c15e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.351+0000 7f3a1affd700 1 -- 172.21.15.189:0/463726414 >> v2:172.21.15.192:3300/0 conn(0x7f3a1c15d7b0 msgr2=0x7f3a1c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:37.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.351+0000 7f3a1affd700 1 --2- 172.21.15.189:0/463726414 >> v2:172.21.15.192:3300/0 conn(0x7f3a1c15d7b0 0x7f3a1c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.351+0000 7f3a1affd700 1 -- 172.21.15.189:0/463726414 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1c0b3340 con 0x7f3a1c06a730 2024-04-06T12:31:37.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.352+0000 7f3a1affd700 1 --2- 172.21.15.189:0/463726414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a1c06a730 0x7f3a1c06abb0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0b6200 tx=0x7f3a0c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d26ee4fd93432ae server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:37.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.352+0000 7f3a1a7fc700 1 -- 172.21.15.189:0/463726414 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c00b870 con 0x7f3a1c06a730 2024-04-06T12:31:37.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.352+0000 7f3a1a7fc700 1 -- 172.21.15.189:0/463726414 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0c00b9d0 con 0x7f3a1c06a730 2024-04-06T12:31:37.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.352+0000 7f3a1a7fc700 1 -- 172.21.15.189:0/463726414 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c00bd60 con 0x7f3a1c06a730 2024-04-06T12:31:37.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.352+0000 7f3a21e51700 1 -- 172.21.15.189:0/463726414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a1c06a730 msgr2=0x7f3a1c06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:37.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.352+0000 7f3a21e51700 1 --2- 172.21.15.189:0/463726414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a1c06a730 0x7f3a1c06abb0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0b6200 tx=0x7f3a0c009bd0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.352+0000 7f3a21e51700 1 -- 172.21.15.189:0/463726414 shutdown_connections 2024-04-06T12:31:37.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.352+0000 7f3a21e51700 1 --2- 172.21.15.189:0/463726414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a1c15ddf0 0x7f3a1c15e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.352+0000 7f3a21e51700 1 --2- 172.21.15.189:0/463726414 >> v2:172.21.15.192:3300/0 conn(0x7f3a1c15d7b0 0x7f3a1c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.352+0000 7f3a21e51700 1 --2- 172.21.15.189:0/463726414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a1c06a730 0x7f3a1c06abb0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.352+0000 7f3a21e51700 1 -- 172.21.15.189:0/463726414 >> 172.21.15.189:0/463726414 conn(0x7f3a1c0d00b0 msgr2=0x7f3a1c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:37.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.352+0000 7f3a21e51700 1 -- 172.21.15.189:0/463726414 shutdown_connections 2024-04-06T12:31:37.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.352+0000 7f3a21e51700 1 -- 172.21.15.189:0/463726414 wait complete. 2024-04-06T12:31:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.353+0000 7f3a21e51700 1 Processor -- start 2024-04-06T12:31:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.354+0000 7f3a21e51700 1 -- start start 2024-04-06T12:31:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.354+0000 7f3a21e51700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a1c06a730 0x7f3a1c153da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.354+0000 7f3a21e51700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a1c15d7b0 0x7f3a1c14fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.354+0000 7f3a21e51700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a1c15ddf0 0x7f3a1c0ad090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.354+0000 7f3a21e51700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c0cc8b0 con 0x7f3a1c15d7b0 2024-04-06T12:31:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.354+0000 7f3a21e51700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c15a580 con 0x7f3a1c06a730 2024-04-06T12:31:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.354+0000 7f3a21e51700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c15a860 con 0x7f3a1c15ddf0 2024-04-06T12:31:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.354+0000 7f3a1affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a1c15d7b0 0x7f3a1c14fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.354+0000 7f3a1affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a1c15d7b0 0x7f3a1c14fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57146/0 (socket says 172.21.15.189:57146) 2024-04-06T12:31:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.354+0000 7f3a1affd700 1 -- 172.21.15.189:0/4186748566 learned_addr learned my addr 172.21.15.189:0/4186748566 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.355+0000 7f3a1b7fe700 1 --2- 172.21.15.189:0/4186748566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a1c06a730 0x7f3a1c153da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.355+0000 7f3a1bfff700 1 --2- 172.21.15.189:0/4186748566 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a1c15ddf0 0x7f3a1c0ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.355+0000 7f3a1affd700 1 -- 172.21.15.189:0/4186748566 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a1c15ddf0 msgr2=0x7f3a1c0ad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.355+0000 7f3a1affd700 1 --2- 172.21.15.189:0/4186748566 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a1c15ddf0 0x7f3a1c0ad090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.355+0000 7f3a1affd700 1 -- 172.21.15.189:0/4186748566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a1c06a730 msgr2=0x7f3a1c153da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.355+0000 7f3a1affd700 1 --2- 172.21.15.189:0/4186748566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a1c06a730 0x7f3a1c153da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.355+0000 7f3a1affd700 1 -- 172.21.15.189:0/4186748566 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0c009550 con 0x7f3a1c15d7b0 2024-04-06T12:31:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.355+0000 7f3a1affd700 1 --2- 172.21.15.189:0/4186748566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a1c15d7b0 0x7f3a1c14fc40 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c000c00 tx=0x7f3a0c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.355+0000 7f3a13fff700 1 -- 172.21.15.189:0/4186748566 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c004480 con 0x7f3a1c15d7b0 2024-04-06T12:31:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.355+0000 7f3a13fff700 1 -- 172.21.15.189:0/4186748566 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0c0045e0 con 0x7f3a1c15d7b0 2024-04-06T12:31:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.355+0000 7f3a21e51700 1 -- 172.21.15.189:0/4186748566 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a1c15ab40 con 0x7f3a1c15d7b0 2024-04-06T12:31:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.355+0000 7f3a13fff700 1 -- 172.21.15.189:0/4186748566 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c0038e0 con 0x7f3a1c15d7b0 2024-04-06T12:31:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.355+0000 7f3a21e51700 1 -- 172.21.15.189:0/4186748566 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a1c150180 con 0x7f3a1c15d7b0 2024-04-06T12:31:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.356+0000 7f3a21e51700 1 -- 172.21.15.189:0/4186748566 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a1c15b060 con 0x7f3a1c15d7b0 2024-04-06T12:31:37.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.356+0000 7f3a13fff700 1 -- 172.21.15.189:0/4186748566 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 79) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f3a0c021070 con 0x7f3a1c15d7b0 2024-04-06T12:31:37.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.357+0000 7f3a13fff700 1 --2- 172.21.15.189:0/4186748566 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f39fc06ab30 0x7f39fc06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:37.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.357+0000 7f3a13fff700 1 -- 172.21.15.189:0/4186748566 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f3a0c08ab90 con 0x7f3a1c15d7b0 2024-04-06T12:31:37.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.357+0000 7f3a1b7fe700 1 --2- 172.21.15.189:0/4186748566 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f39fc06ab30 0x7f39fc06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:37.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.357+0000 7f3a1b7fe700 1 --2- 172.21.15.189:0/4186748566 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f39fc06ab30 0x7f39fc06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c1609c0 tx=0x7f3a04008040 comp rx=0 tx=0).ready entity=mgr.8775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:37.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.361+0000 7f3a13fff700 1 -- 172.21.15.189:0/4186748566 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3a1c15b060 con 0x7f3a1c15d7b0 2024-04-06T12:31:37.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.584+0000 7f3a21e51700 1 -- 172.21.15.189:0/4186748566 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/hourly", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a1c15e7c0 con 0x7f39fc06ab30 2024-04-06T12:31:37.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.642+0000 7f3a13fff700 1 -- 172.21.15.189:0/4186748566 <== mgr.8775 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f3a04009e40 con 0x7f39fc06ab30 2024-04-06T12:31:37.641 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path snap_test_dir1/hourly 2024-04-06T12:31:37.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.645+0000 7f3a21e51700 1 -- 172.21.15.189:0/4186748566 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f39fc06ab30 msgr2=0x7f39fc06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:37.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.645+0000 7f3a21e51700 1 --2- 172.21.15.189:0/4186748566 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f39fc06ab30 0x7f39fc06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c1609c0 tx=0x7f3a04008040 comp rx=0 tx=0).stop 2024-04-06T12:31:37.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.645+0000 7f3a21e51700 1 -- 172.21.15.189:0/4186748566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a1c15d7b0 msgr2=0x7f3a1c14fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:37.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.645+0000 7f3a21e51700 1 --2- 172.21.15.189:0/4186748566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a1c15d7b0 0x7f3a1c14fc40 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c000c00 tx=0x7f3a0c004750 comp rx=0 tx=0).stop 2024-04-06T12:31:37.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.645+0000 7f3a21e51700 1 -- 172.21.15.189:0/4186748566 shutdown_connections 2024-04-06T12:31:37.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.645+0000 7f3a21e51700 1 --2- 172.21.15.189:0/4186748566 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a1c15ddf0 0x7f3a1c0ad090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.645+0000 7f3a21e51700 1 --2- 172.21.15.189:0/4186748566 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f39fc06ab30 0x7f39fc06cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.645+0000 7f3a21e51700 1 --2- 172.21.15.189:0/4186748566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a1c06a730 0x7f3a1c153da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.645+0000 7f3a21e51700 1 --2- 172.21.15.189:0/4186748566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a1c15d7b0 0x7f3a1c14fc40 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.645+0000 7f3a21e51700 1 -- 172.21.15.189:0/4186748566 >> 172.21.15.189:0/4186748566 conn(0x7f3a1c0d00b0 msgr2=0x7f3a1c0723d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:37.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.645+0000 7f3a21e51700 1 -- 172.21.15.189:0/4186748566 shutdown_connections 2024-04-06T12:31:37.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.645+0000 7f3a21e51700 1 -- 172.21.15.189:0/4186748566 wait complete. 2024-04-06T12:31:37.656 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/hourly 2024-04-06T12:31:37.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:37.656 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/daily) 2024-04-06T12:31:37.675 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/daily 1d 2024-04-06T12:31:37.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.842+0000 7fc4bb4f5700 1 Processor -- start 2024-04-06T12:31:37.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.842+0000 7fc4bb4f5700 1 -- start start 2024-04-06T12:31:37.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.843+0000 7fc4bb4f5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc4b415d7b0 0x7fc4b40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:37.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.843+0000 7fc4bb4f5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4b415e500 0x7fc4b406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:37.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.843+0000 7fc4bb4f5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b40b3990 con 0x7fc4b406a730 2024-04-06T12:31:37.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.843+0000 7fc4bb4f5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4b406b0f0 con 0x7fc4b415d7b0 2024-04-06T12:31:37.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.843+0000 7fc4bb4f5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b40b3060 con 0x7fc4b415e500 2024-04-06T12:31:37.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.843+0000 7fc4b9291700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc4b415d7b0 0x7fc4b40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:37.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.843+0000 7fc4b9291700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc4b415d7b0 0x7fc4b40b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57156/0 (socket says 172.21.15.189:57156) 2024-04-06T12:31:37.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.843+0000 7fc4b9291700 1 -- 172.21.15.189:0/2709868122 learned_addr learned my addr 172.21.15.189:0/2709868122 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:37.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.843+0000 7fc4b9a92700 1 --2- 172.21.15.189:0/2709868122 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4b415e500 0x7fc4b406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:37.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.844+0000 7fc4b9291700 1 -- 172.21.15.189:0/2709868122 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4b415e500 msgr2=0x7fc4b406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:37.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.844+0000 7fc4b9291700 1 --2- 172.21.15.189:0/2709868122 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4b415e500 0x7fc4b406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.844+0000 7fc4b9291700 1 -- 172.21.15.189:0/2709868122 >> v1:172.21.15.192:6789/0 conn(0x7fc4b406a730 legacy=0x7fc4b415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:37.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.844+0000 7fc4b9291700 1 -- 172.21.15.189:0/2709868122 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b40b3340 con 0x7fc4b415d7b0 2024-04-06T12:31:37.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.844+0000 7fc4b9291700 1 --2- 172.21.15.189:0/2709868122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4b415d7b0 0x7fc4b40b3e20 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fc4a40098f0 tx=0x7fc4a4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9216bdc3e1a222 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:37.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.844+0000 7fc4abfff700 1 -- 172.21.15.189:0/2709868122 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a400b8a0 con 0x7fc4b415d7b0 2024-04-06T12:31:37.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.844+0000 7fc4abfff700 1 -- 172.21.15.189:0/2709868122 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a400ba00 con 0x7fc4b415d7b0 2024-04-06T12:31:37.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.844+0000 7fc4abfff700 1 -- 172.21.15.189:0/2709868122 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a400bd90 con 0x7fc4b415d7b0 2024-04-06T12:31:37.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.844+0000 7fc4bb4f5700 1 -- 172.21.15.189:0/2709868122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4b415d7b0 msgr2=0x7fc4b40b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:37.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.844+0000 7fc4bb4f5700 1 --2- 172.21.15.189:0/2709868122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4b415d7b0 0x7fc4b40b3e20 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fc4a40098f0 tx=0x7fc4a4009c00 comp rx=0 tx=0).stop 2024-04-06T12:31:37.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.844+0000 7fc4bb4f5700 1 -- 172.21.15.189:0/2709868122 shutdown_connections 2024-04-06T12:31:37.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.844+0000 7fc4bb4f5700 1 --2- 172.21.15.189:0/2709868122 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4b415e500 0x7fc4b406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.845+0000 7fc4bb4f5700 1 --2- 172.21.15.189:0/2709868122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4b415d7b0 0x7fc4b40b3e20 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.845+0000 7fc4bb4f5700 1 -- 172.21.15.189:0/2709868122 >> 172.21.15.189:0/2709868122 conn(0x7fc4b40d00b0 msgr2=0x7fc4b40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:37.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.845+0000 7fc4bb4f5700 1 -- 172.21.15.189:0/2709868122 shutdown_connections 2024-04-06T12:31:37.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.845+0000 7fc4bb4f5700 1 -- 172.21.15.189:0/2709868122 wait complete. 2024-04-06T12:31:37.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.846+0000 7fc4bb4f5700 1 Processor -- start 2024-04-06T12:31:37.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.846+0000 7fc4bb4f5700 1 -- start start 2024-04-06T12:31:37.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.847+0000 7fc4bb4f5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4b406a730 0x7fc4b415c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:37.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.847+0000 7fc4bb4f5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4b415d7b0 0x7fc4b4146a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:37.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.847+0000 7fc4bb4f5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4b415e500 0x7fc4b4158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:37.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.847+0000 7fc4bb4f5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b415bf50 con 0x7fc4b415d7b0 2024-04-06T12:31:37.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.847+0000 7fc4bb4f5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b415c920 con 0x7fc4b415e500 2024-04-06T12:31:37.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.847+0000 7fc4bb4f5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b415b340 con 0x7fc4b406a730 2024-04-06T12:31:37.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.847+0000 7fc4b8a90700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4b415d7b0 0x7fc4b4146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:37.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.847+0000 7fc4b8a90700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4b415d7b0 0x7fc4b4146a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57158/0 (socket says 172.21.15.189:57158) 2024-04-06T12:31:37.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.847+0000 7fc4b8a90700 1 -- 172.21.15.189:0/253834856 learned_addr learned my addr 172.21.15.189:0/253834856 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:37.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.847+0000 7fc4b9291700 1 --2- 172.21.15.189:0/253834856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4b406a730 0x7fc4b415c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:37.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.847+0000 7fc4b9a92700 1 --2- 172.21.15.189:0/253834856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4b415e500 0x7fc4b4158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.847+0000 7fc4b8a90700 1 -- 172.21.15.189:0/253834856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4b406a730 msgr2=0x7fc4b415c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.847+0000 7fc4b8a90700 1 --2- 172.21.15.189:0/253834856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4b406a730 0x7fc4b415c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.847+0000 7fc4b8a90700 1 -- 172.21.15.189:0/253834856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4b415e500 msgr2=0x7fc4b4158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.847+0000 7fc4b8a90700 1 --2- 172.21.15.189:0/253834856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4b415e500 0x7fc4b4158b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.847+0000 7fc4b8a90700 1 -- 172.21.15.189:0/253834856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a4009550 con 0x7fc4b415d7b0 2024-04-06T12:31:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.848+0000 7fc4b8a90700 1 --2- 172.21.15.189:0/253834856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4b415d7b0 0x7fc4b4146a10 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000c4e0 tx=0x7fc4b000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.848+0000 7fc4a9ffb700 1 -- 172.21.15.189:0/253834856 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b00052d0 con 0x7fc4b415d7b0 2024-04-06T12:31:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.848+0000 7fc4a9ffb700 1 -- 172.21.15.189:0/253834856 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b0005430 con 0x7fc4b415d7b0 2024-04-06T12:31:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.848+0000 7fc4bb4f5700 1 -- 172.21.15.189:0/253834856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b415b620 con 0x7fc4b415d7b0 2024-04-06T12:31:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.848+0000 7fc4a9ffb700 1 -- 172.21.15.189:0/253834856 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b00038c0 con 0x7fc4b415d7b0 2024-04-06T12:31:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.848+0000 7fc4bb4f5700 1 -- 172.21.15.189:0/253834856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b415bbe0 con 0x7fc4b415d7b0 2024-04-06T12:31:37.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.849+0000 7fc4bb4f5700 1 -- 172.21.15.189:0/253834856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b4067bb0 con 0x7fc4b415d7b0 2024-04-06T12:31:37.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.849+0000 7fc4a9ffb700 1 -- 172.21.15.189:0/253834856 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 79) v1 ==== 89000+0+0 (secure 0 0 0) 0x7fc4b0003a20 con 0x7fc4b415d7b0 2024-04-06T12:31:37.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.850+0000 7fc4a9ffb700 1 --2- 172.21.15.189:0/253834856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc49806ab30 0x7fc49806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:37.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.850+0000 7fc4a9ffb700 1 -- 172.21.15.189:0/253834856 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fc4b0088710 con 0x7fc4b415d7b0 2024-04-06T12:31:37.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.850+0000 7fc4b9291700 1 --2- 172.21.15.189:0/253834856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc49806ab30 0x7fc49806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:37.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.850+0000 7fc4b9291700 1 --2- 172.21.15.189:0/253834856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc49806ab30 0x7fc49806cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4009c40 tx=0x7fc4a4008030 comp rx=0 tx=0).ready entity=mgr.8775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:37.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:37.854+0000 7fc4a9ffb700 1 -- 172.21.15.189:0/253834856 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc4b0053990 con 0x7fc4b415d7b0 2024-04-06T12:31:38.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.073+0000 7fc4bb4f5700 1 -- 172.21.15.189:0/253834856 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/daily", "snap_schedule": "1d", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4b415eb40 con 0x7fc49806ab30 2024-04-06T12:31:38.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.137+0000 7fc4a9ffb700 1 -- 172.21.15.189:0/253834856 <== mgr.8775 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fc4a400bb70 con 0x7fc49806ab30 2024-04-06T12:31:38.135 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path snap_test_dir1/daily 2024-04-06T12:31:38.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.140+0000 7fc4bb4f5700 1 -- 172.21.15.189:0/253834856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc49806ab30 msgr2=0x7fc49806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:38.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.140+0000 7fc4bb4f5700 1 --2- 172.21.15.189:0/253834856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc49806ab30 0x7fc49806cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4009c40 tx=0x7fc4a4008030 comp rx=0 tx=0).stop 2024-04-06T12:31:38.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.140+0000 7fc4bb4f5700 1 -- 172.21.15.189:0/253834856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4b415d7b0 msgr2=0x7fc4b4146a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:38.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.140+0000 7fc4bb4f5700 1 --2- 172.21.15.189:0/253834856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4b415d7b0 0x7fc4b4146a10 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000c4e0 tx=0x7fc4b000c8a0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.140+0000 7fc4bb4f5700 1 -- 172.21.15.189:0/253834856 shutdown_connections 2024-04-06T12:31:38.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.140+0000 7fc4bb4f5700 1 --2- 172.21.15.189:0/253834856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc49806ab30 0x7fc49806cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.140+0000 7fc4bb4f5700 1 --2- 172.21.15.189:0/253834856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4b415e500 0x7fc4b4158b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.140+0000 7fc4bb4f5700 1 --2- 172.21.15.189:0/253834856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4b406a730 0x7fc4b415c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.140+0000 7fc4bb4f5700 1 --2- 172.21.15.189:0/253834856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4b415d7b0 0x7fc4b4146a10 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.140+0000 7fc4bb4f5700 1 -- 172.21.15.189:0/253834856 >> 172.21.15.189:0/253834856 conn(0x7fc4b40d00b0 msgr2=0x7fc4b406eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:38.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.141+0000 7fc4bb4f5700 1 -- 172.21.15.189:0/253834856 shutdown_connections 2024-04-06T12:31:38.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.141+0000 7fc4bb4f5700 1 -- 172.21.15.189:0/253834856 wait complete. 2024-04-06T12:31:38.154 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/daily 2024-04-06T12:31:38.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:38.154 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/weekly) 2024-04-06T12:31:38.172 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/weekly 1w 2024-04-06T12:31:38.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.340+0000 7fd852087700 1 Processor -- start 2024-04-06T12:31:38.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.340+0000 7fd852087700 1 -- start start 2024-04-06T12:31:38.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.341+0000 7fd852087700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd84c15d7b0 0x7fd84c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:38.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.341+0000 7fd852087700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd84c06a730 0x7fd84c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:38.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.341+0000 7fd852087700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd84c0b3990 con 0x7fd84c15ddf0 2024-04-06T12:31:38.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.341+0000 7fd852087700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84c15e950 con 0x7fd84c06a730 2024-04-06T12:31:38.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.341+0000 7fd852087700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84c06b0f0 con 0x7fd84c15d7b0 2024-04-06T12:31:38.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.341+0000 7fd84affd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd84c06a730 0x7fd84c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:38.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.341+0000 7fd84affd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd84c06a730 0x7fd84c06abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57176/0 (socket says 172.21.15.189:57176) 2024-04-06T12:31:38.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.341+0000 7fd84affd700 1 -- 172.21.15.189:0/3404376730 learned_addr learned my addr 172.21.15.189:0/3404376730 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:38.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.341+0000 7fd84b7fe700 1 --2- 172.21.15.189:0/3404376730 >> v2:172.21.15.192:3300/0 conn(0x7fd84c15d7b0 0x7fd84c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:38.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.341+0000 7fd84affd700 1 -- 172.21.15.189:0/3404376730 >> v2:172.21.15.192:3300/0 conn(0x7fd84c15d7b0 msgr2=0x7fd84c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:38.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.341+0000 7fd84affd700 1 --2- 172.21.15.189:0/3404376730 >> v2:172.21.15.192:3300/0 conn(0x7fd84c15d7b0 0x7fd84c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.341+0000 7fd84affd700 1 -- 172.21.15.189:0/3404376730 >> v1:172.21.15.189:6789/0 conn(0x7fd84c15ddf0 legacy=0x7fd84c15e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:38.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.341+0000 7fd84affd700 1 -- 172.21.15.189:0/3404376730 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84c0b3060 con 0x7fd84c06a730 2024-04-06T12:31:38.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.341+0000 7fd84b7fe700 1 --2- 172.21.15.189:0/3404376730 >> v2:172.21.15.192:3300/0 conn(0x7fd84c15d7b0 0x7fd84c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:38.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.341+0000 7fd84affd700 1 --2- 172.21.15.189:0/3404376730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84c06a730 0x7fd84c06abb0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fd84c0b6200 tx=0x7fd83c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b26e479c04cff4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:38.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.341+0000 7fd84a7fc700 1 -- 172.21.15.189:0/3404376730 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c00b870 con 0x7fd84c06a730 2024-04-06T12:31:38.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.342+0000 7fd84a7fc700 1 -- 172.21.15.189:0/3404376730 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83c00b9d0 con 0x7fd84c06a730 2024-04-06T12:31:38.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.342+0000 7fd84a7fc700 1 -- 172.21.15.189:0/3404376730 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c00bd60 con 0x7fd84c06a730 2024-04-06T12:31:38.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.342+0000 7fd852087700 1 -- 172.21.15.189:0/3404376730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84c06a730 msgr2=0x7fd84c06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:38.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.342+0000 7fd852087700 1 --2- 172.21.15.189:0/3404376730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84c06a730 0x7fd84c06abb0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fd84c0b6200 tx=0x7fd83c009bd0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.342+0000 7fd852087700 1 -- 172.21.15.189:0/3404376730 shutdown_connections 2024-04-06T12:31:38.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.342+0000 7fd852087700 1 --2- 172.21.15.189:0/3404376730 >> v2:172.21.15.192:3300/0 conn(0x7fd84c15d7b0 0x7fd84c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.342+0000 7fd852087700 1 --2- 172.21.15.189:0/3404376730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84c06a730 0x7fd84c06abb0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.342+0000 7fd852087700 1 -- 172.21.15.189:0/3404376730 >> 172.21.15.189:0/3404376730 conn(0x7fd84c0d00b0 msgr2=0x7fd84c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:38.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.342+0000 7fd852087700 1 -- 172.21.15.189:0/3404376730 shutdown_connections 2024-04-06T12:31:38.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.342+0000 7fd852087700 1 -- 172.21.15.189:0/3404376730 wait complete. 2024-04-06T12:31:38.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.343+0000 7fd852087700 1 Processor -- start 2024-04-06T12:31:38.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.343+0000 7fd852087700 1 -- start start 2024-04-06T12:31:38.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.344+0000 7fd852087700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd84c06a730 0x7fd84c0ced80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:38.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.344+0000 7fd852087700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd84c15d7b0 0x7fd84c1f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:38.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.344+0000 7fd852087700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84c15ddf0 0x7fd84c1492a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:38.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.344+0000 7fd852087700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c0ce000 con 0x7fd84c15ddf0 2024-04-06T12:31:38.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.344+0000 7fd852087700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c0ce2a0 con 0x7fd84c06a730 2024-04-06T12:31:38.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.344+0000 7fd852087700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c0ce580 con 0x7fd84c15d7b0 2024-04-06T12:31:38.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.344+0000 7fd84bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84c15ddf0 0x7fd84c1492a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:38.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.345+0000 7fd84bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84c15ddf0 0x7fd84c1492a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57192/0 (socket says 172.21.15.189:57192) 2024-04-06T12:31:38.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.345+0000 7fd84bfff700 1 -- 172.21.15.189:0/1311777089 learned_addr learned my addr 172.21.15.189:0/1311777089 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:38.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.345+0000 7fd84b7fe700 1 --2- 172.21.15.189:0/1311777089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd84c06a730 0x7fd84c0ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:38.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.345+0000 7fd84affd700 1 --2- 172.21.15.189:0/1311777089 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd84c15d7b0 0x7fd84c1f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:38.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.345+0000 7fd84bfff700 1 -- 172.21.15.189:0/1311777089 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd84c15d7b0 msgr2=0x7fd84c1f2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.345+0000 7fd84bfff700 1 --2- 172.21.15.189:0/1311777089 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd84c15d7b0 0x7fd84c1f2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.345+0000 7fd84bfff700 1 -- 172.21.15.189:0/1311777089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd84c06a730 msgr2=0x7fd84c0ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.345+0000 7fd84bfff700 1 --2- 172.21.15.189:0/1311777089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd84c06a730 0x7fd84c0ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.345+0000 7fd84bfff700 1 -- 172.21.15.189:0/1311777089 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83c009550 con 0x7fd84c15ddf0 2024-04-06T12:31:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.345+0000 7fd84bfff700 1 --2- 172.21.15.189:0/1311777089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84c15ddf0 0x7fd84c1492a0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fd84400cb50 tx=0x7fd84400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.345+0000 7fd833fff700 1 -- 172.21.15.189:0/1311777089 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd844003c00 con 0x7fd84c15ddf0 2024-04-06T12:31:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.345+0000 7fd833fff700 1 -- 172.21.15.189:0/1311777089 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd844003d60 con 0x7fd84c15ddf0 2024-04-06T12:31:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.345+0000 7fd852087700 1 -- 172.21.15.189:0/1311777089 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd84c0ce860 con 0x7fd84c15ddf0 2024-04-06T12:31:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.345+0000 7fd833fff700 1 -- 172.21.15.189:0/1311777089 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84400aa20 con 0x7fd84c15ddf0 2024-04-06T12:31:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.345+0000 7fd852087700 1 -- 172.21.15.189:0/1311777089 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd84c0c53f0 con 0x7fd84c15ddf0 2024-04-06T12:31:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.346+0000 7fd852087700 1 -- 172.21.15.189:0/1311777089 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd84c067bb0 con 0x7fd84c15ddf0 2024-04-06T12:31:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.347+0000 7fd833fff700 1 -- 172.21.15.189:0/1311777089 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 79) v1 ==== 89000+0+0 (secure 0 0 0) 0x7fd84400ab80 con 0x7fd84c15ddf0 2024-04-06T12:31:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.347+0000 7fd833fff700 1 --2- 172.21.15.189:0/1311777089 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd83806ab30 0x7fd83806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.347+0000 7fd833fff700 1 -- 172.21.15.189:0/1311777089 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fd844089260 con 0x7fd84c15ddf0 2024-04-06T12:31:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.347+0000 7fd84b7fe700 1 --2- 172.21.15.189:0/1311777089 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd83806ab30 0x7fd83806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:38.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.347+0000 7fd84b7fe700 1 --2- 172.21.15.189:0/1311777089 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd83806ab30 0x7fd83806cff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd840006f10 tx=0x7fd840008040 comp rx=0 tx=0).ready entity=mgr.8775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:38.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.351+0000 7fd833fff700 1 -- 172.21.15.189:0/1311777089 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd8440544e0 con 0x7fd84c15ddf0 2024-04-06T12:31:38.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.571+0000 7fd852087700 1 -- 172.21.15.189:0/1311777089 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/weekly", "snap_schedule": "1w", "target": ["mon-mgr", ""]}) v1 -- 0x7fd84c0ce410 con 0x7fd83806ab30 2024-04-06T12:31:38.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.627+0000 7fd833fff700 1 -- 172.21.15.189:0/1311777089 <== mgr.8775 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7fd84c0ce410 con 0x7fd83806ab30 2024-04-06T12:31:38.626 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path snap_test_dir1/weekly 2024-04-06T12:31:38.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.630+0000 7fd852087700 1 -- 172.21.15.189:0/1311777089 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd83806ab30 msgr2=0x7fd83806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:38.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.630+0000 7fd852087700 1 --2- 172.21.15.189:0/1311777089 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd83806ab30 0x7fd83806cff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd840006f10 tx=0x7fd840008040 comp rx=0 tx=0).stop 2024-04-06T12:31:38.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.630+0000 7fd852087700 1 -- 172.21.15.189:0/1311777089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84c15ddf0 msgr2=0x7fd84c1492a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:38.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.630+0000 7fd852087700 1 --2- 172.21.15.189:0/1311777089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84c15ddf0 0x7fd84c1492a0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fd84400cb50 tx=0x7fd84400ce60 comp rx=0 tx=0).stop 2024-04-06T12:31:38.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.630+0000 7fd852087700 1 -- 172.21.15.189:0/1311777089 shutdown_connections 2024-04-06T12:31:38.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.630+0000 7fd852087700 1 --2- 172.21.15.189:0/1311777089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84c15ddf0 0x7fd84c1492a0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.630+0000 7fd852087700 1 --2- 172.21.15.189:0/1311777089 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd83806ab30 0x7fd83806cff0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.630+0000 7fd852087700 1 --2- 172.21.15.189:0/1311777089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd84c06a730 0x7fd84c0ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.630+0000 7fd852087700 1 --2- 172.21.15.189:0/1311777089 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd84c15d7b0 0x7fd84c1f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.630+0000 7fd852087700 1 -- 172.21.15.189:0/1311777089 >> 172.21.15.189:0/1311777089 conn(0x7fd84c0d00b0 msgr2=0x7fd84c0ae0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:38.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.630+0000 7fd852087700 1 -- 172.21.15.189:0/1311777089 shutdown_connections 2024-04-06T12:31:38.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.630+0000 7fd852087700 1 -- 172.21.15.189:0/1311777089 wait complete. 2024-04-06T12:31:38.640 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/weekly 2024-04-06T12:31:38.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:38.640 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/monthly) 2024-04-06T12:31:38.658 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/monthly 1M 2024-04-06T12:31:38.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.824+0000 7fd4a4775700 1 Processor -- start 2024-04-06T12:31:38.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.824+0000 7fd4a4775700 1 -- start start 2024-04-06T12:31:38.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.825+0000 7fd4a4775700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd49c06ae40 0x7fd49c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:38.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.825+0000 7fd4a4775700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd49c15ddf0 0x7fd49c15e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:38.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.825+0000 7fd4a4775700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49c0b3990 con 0x7fd49c15d7b0 2024-04-06T12:31:38.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.825+0000 7fd4a4775700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd49c0b3060 con 0x7fd49c06ae40 2024-04-06T12:31:38.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.825+0000 7fd4a4775700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd49c0b3340 con 0x7fd49c15ddf0 2024-04-06T12:31:38.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.825+0000 7fd4a1d10700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd49c06ae40 0x7fd49c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:38.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.825+0000 7fd4a1d10700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd49c06ae40 0x7fd49c0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57200/0 (socket says 172.21.15.189:57200) 2024-04-06T12:31:38.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.825+0000 7fd4a1d10700 1 -- 172.21.15.189:0/3863376723 learned_addr learned my addr 172.21.15.189:0/3863376723 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:38.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.826+0000 7fd4a2d12700 1 --2- 172.21.15.189:0/3863376723 >> v2:172.21.15.192:3300/0 conn(0x7fd49c15ddf0 0x7fd49c15e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:38.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.826+0000 7fd4a1d10700 1 -- 172.21.15.189:0/3863376723 >> v2:172.21.15.192:3300/0 conn(0x7fd49c15ddf0 msgr2=0x7fd49c15e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:38.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.826+0000 7fd4a1d10700 1 --2- 172.21.15.189:0/3863376723 >> v2:172.21.15.192:3300/0 conn(0x7fd49c15ddf0 0x7fd49c15e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.826+0000 7fd4a1d10700 1 -- 172.21.15.189:0/3863376723 >> v1:172.21.15.192:6789/0 conn(0x7fd49c15d7b0 legacy=0x7fd49c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:38.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.826+0000 7fd4a1d10700 1 -- 172.21.15.189:0/3863376723 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49c0b3620 con 0x7fd49c06ae40 2024-04-06T12:31:38.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.826+0000 7fd4a1d10700 1 --2- 172.21.15.189:0/3863376723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49c06ae40 0x7fd49c0b3e20 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fd49c06d2c0 tx=0x7fd498009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52381ccefaf79bc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:38.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.826+0000 7fd4a150f700 1 -- 172.21.15.189:0/3863376723 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd498004230 con 0x7fd49c06ae40 2024-04-06T12:31:38.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.826+0000 7fd4a150f700 1 -- 172.21.15.189:0/3863376723 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd498004390 con 0x7fd49c06ae40 2024-04-06T12:31:38.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.826+0000 7fd4a150f700 1 -- 172.21.15.189:0/3863376723 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4980036a0 con 0x7fd49c06ae40 2024-04-06T12:31:38.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.827+0000 7fd4a4775700 1 -- 172.21.15.189:0/3863376723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49c06ae40 msgr2=0x7fd49c0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:38.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.827+0000 7fd4a4775700 1 --2- 172.21.15.189:0/3863376723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49c06ae40 0x7fd49c0b3e20 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fd49c06d2c0 tx=0x7fd498009bd0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.827+0000 7fd4a4775700 1 -- 172.21.15.189:0/3863376723 shutdown_connections 2024-04-06T12:31:38.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.827+0000 7fd4a4775700 1 --2- 172.21.15.189:0/3863376723 >> v2:172.21.15.192:3300/0 conn(0x7fd49c15ddf0 0x7fd49c15e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.827+0000 7fd4a4775700 1 --2- 172.21.15.189:0/3863376723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49c06ae40 0x7fd49c0b3e20 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.827+0000 7fd4a4775700 1 -- 172.21.15.189:0/3863376723 >> 172.21.15.189:0/3863376723 conn(0x7fd49c0d00b0 msgr2=0x7fd49c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:38.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.827+0000 7fd4a4775700 1 -- 172.21.15.189:0/3863376723 shutdown_connections 2024-04-06T12:31:38.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.827+0000 7fd4a4775700 1 -- 172.21.15.189:0/3863376723 wait complete. 2024-04-06T12:31:38.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.828+0000 7fd4a4775700 1 Processor -- start 2024-04-06T12:31:38.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.828+0000 7fd4a4775700 1 -- start start 2024-04-06T12:31:38.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a4775700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49c06ae40 0x7fd49c15b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:38.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a4775700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd49c15d7b0 0x7fd49c142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:38.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a4775700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd49c15ddf0 0x7fd49c142c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:38.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a4775700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c15b540 con 0x7fd49c06ae40 2024-04-06T12:31:38.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a4775700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c0cf310 con 0x7fd49c15ddf0 2024-04-06T12:31:38.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a4775700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c0aec20 con 0x7fd49c15d7b0 2024-04-06T12:31:38.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a2511700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49c06ae40 0x7fd49c15b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:38.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a2511700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49c06ae40 0x7fd49c15b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57208/0 (socket says 172.21.15.189:57208) 2024-04-06T12:31:38.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a2511700 1 -- 172.21.15.189:0/2288828162 learned_addr learned my addr 172.21.15.189:0/2288828162 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:38.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a1d10700 1 --2- 172.21.15.189:0/2288828162 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd49c15d7b0 0x7fd49c142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:38.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a2d12700 1 --2- 172.21.15.189:0/2288828162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd49c15ddf0 0x7fd49c142c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:38.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a2511700 1 -- 172.21.15.189:0/2288828162 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd49c15d7b0 msgr2=0x7fd49c142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:38.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a2511700 1 --2- 172.21.15.189:0/2288828162 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd49c15d7b0 0x7fd49c142210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a2511700 1 -- 172.21.15.189:0/2288828162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd49c15ddf0 msgr2=0x7fd49c142c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:38.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a2511700 1 --2- 172.21.15.189:0/2288828162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd49c15ddf0 0x7fd49c142c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:38.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a2511700 1 -- 172.21.15.189:0/2288828162 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd498009550 con 0x7fd49c06ae40 2024-04-06T12:31:38.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a2d12700 1 --2- 172.21.15.189:0/2288828162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd49c15ddf0 0x7fd49c142c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:38.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a2511700 1 --2- 172.21.15.189:0/2288828162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49c06ae40 0x7fd49c15b9d0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fd48c00c4d0 tx=0x7fd48c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:38.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd492ffd700 1 -- 172.21.15.189:0/2288828162 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c011790 con 0x7fd49c06ae40 2024-04-06T12:31:38.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd492ffd700 1 -- 172.21.15.189:0/2288828162 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48c0118f0 con 0x7fd49c06ae40 2024-04-06T12:31:38.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a4775700 1 -- 172.21.15.189:0/2288828162 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd49c0aef00 con 0x7fd49c06ae40 2024-04-06T12:31:38.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd492ffd700 1 -- 172.21.15.189:0/2288828162 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c011ca0 con 0x7fd49c06ae40 2024-04-06T12:31:38.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.829+0000 7fd4a4775700 1 -- 172.21.15.189:0/2288828162 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd49c0af1e0 con 0x7fd49c06ae40 2024-04-06T12:31:38.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.830+0000 7fd4a4775700 1 -- 172.21.15.189:0/2288828162 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd49c15b820 con 0x7fd49c06ae40 2024-04-06T12:31:38.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.831+0000 7fd492ffd700 1 -- 172.21.15.189:0/2288828162 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 79) v1 ==== 89000+0+0 (secure 0 0 0) 0x7fd48c01c460 con 0x7fd49c06ae40 2024-04-06T12:31:38.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.831+0000 7fd492ffd700 1 --2- 172.21.15.189:0/2288828162 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd48006ab30 0x7fd48006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:38.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.831+0000 7fd492ffd700 1 -- 172.21.15.189:0/2288828162 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fd48c089140 con 0x7fd49c06ae40 2024-04-06T12:31:38.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.831+0000 7fd4a1d10700 1 --2- 172.21.15.189:0/2288828162 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd48006ab30 0x7fd48006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:38.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.832+0000 7fd4a1d10700 1 --2- 172.21.15.189:0/2288828162 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd48006ab30 0x7fd48006cff0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd49c06d2c0 tx=0x7fd498008030 comp rx=0 tx=0).ready entity=mgr.8775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:38.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:38.836+0000 7fd492ffd700 1 -- 172.21.15.189:0/2288828162 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd48c054330 con 0x7fd49c06ae40 2024-04-06T12:31:39.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.055+0000 7fd4a4775700 1 -- 172.21.15.189:0/2288828162 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/monthly", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd49c0717b0 con 0x7fd48006ab30 2024-04-06T12:31:39.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.087+0000 7fd492ffd700 1 -- 172.21.15.189:0/2288828162 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 80) v1 ==== 89265+0+0 (secure 0 0 0) 0x7fd48c054700 con 0x7fd49c06ae40 2024-04-06T12:31:39.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.116+0000 7fd492ffd700 1 -- 172.21.15.189:0/2288828162 <== mgr.8775 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7fd4980040b0 con 0x7fd48006ab30 2024-04-06T12:31:39.115 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path snap_test_dir1/monthly 2024-04-06T12:31:39.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.119+0000 7fd4a4775700 1 -- 172.21.15.189:0/2288828162 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd48006ab30 msgr2=0x7fd48006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:39.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.119+0000 7fd4a4775700 1 --2- 172.21.15.189:0/2288828162 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd48006ab30 0x7fd48006cff0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd49c06d2c0 tx=0x7fd498008030 comp rx=0 tx=0).stop 2024-04-06T12:31:39.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.119+0000 7fd4a4775700 1 -- 172.21.15.189:0/2288828162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49c06ae40 msgr2=0x7fd49c15b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:39.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.119+0000 7fd4a4775700 1 --2- 172.21.15.189:0/2288828162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49c06ae40 0x7fd49c15b9d0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fd48c00c4d0 tx=0x7fd48c00c890 comp rx=0 tx=0).stop 2024-04-06T12:31:39.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.119+0000 7fd4a4775700 1 -- 172.21.15.189:0/2288828162 shutdown_connections 2024-04-06T12:31:39.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.119+0000 7fd4a4775700 1 --2- 172.21.15.189:0/2288828162 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd48006ab30 0x7fd48006cff0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.119+0000 7fd4a4775700 1 --2- 172.21.15.189:0/2288828162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd49c15ddf0 0x7fd49c142c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.119+0000 7fd4a4775700 1 --2- 172.21.15.189:0/2288828162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49c06ae40 0x7fd49c15b9d0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.119+0000 7fd4a4775700 1 --2- 172.21.15.189:0/2288828162 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd49c15d7b0 0x7fd49c142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.119+0000 7fd4a4775700 1 -- 172.21.15.189:0/2288828162 >> 172.21.15.189:0/2288828162 conn(0x7fd49c0d00b0 msgr2=0x7fd49c0add50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:39.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.119+0000 7fd4a4775700 1 -- 172.21.15.189:0/2288828162 shutdown_connections 2024-04-06T12:31:39.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.119+0000 7fd4a4775700 1 -- 172.21.15.189:0/2288828162 wait complete. 2024-04-06T12:31:39.129 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/monthly 2024-04-06T12:31:39.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:39.129 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/yearly) 2024-04-06T12:31:39.148 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/yearly 1Y 2024-04-06T12:31:39.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.315+0000 7f0892314700 1 Processor -- start 2024-04-06T12:31:39.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.315+0000 7f0892314700 1 -- start start 2024-04-06T12:31:39.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.316+0000 7f0892314700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f088c0d17a0 0x7f088c06b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:39.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.316+0000 7f0892314700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c146dd0 0x7f088c06d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:39.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.316+0000 7f0892314700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f088c0cedf0 0x7f088c0cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:39.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.316+0000 7f0892314700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f088c0cf760 con 0x7f088c0d17a0 2024-04-06T12:31:39.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.316+0000 7f0892314700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f088c06c0a0 con 0x7f088c0cedf0 2024-04-06T12:31:39.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.316+0000 7f0892314700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088c06c380 con 0x7f088c146dd0 2024-04-06T12:31:39.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.316+0000 7f088bfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f088c0d17a0 0x7f088c06b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:39.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.316+0000 7f088bfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f088c0d17a0 0x7f088c06b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40466/0 (socket says 172.21.15.189:40466) 2024-04-06T12:31:39.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.316+0000 7f088bfff700 1 -- 172.21.15.189:0/1453596466 learned_addr learned my addr 172.21.15.189:0/1453596466 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:39.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.316+0000 7f08908b1700 1 --2- 172.21.15.189:0/1453596466 >> v2:172.21.15.192:3300/0 conn(0x7f088c0cedf0 0x7f088c0cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:39.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.316+0000 7f088b7fe700 1 --2- 172.21.15.189:0/1453596466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c146dd0 0x7f088c06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:39.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.317+0000 7f088bfff700 1 -- 172.21.15.189:0/1453596466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c146dd0 msgr2=0x7f088c06d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:39.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.317+0000 7f088bfff700 1 --2- 172.21.15.189:0/1453596466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c146dd0 0x7f088c06d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.317+0000 7f088bfff700 1 -- 172.21.15.189:0/1453596466 >> v2:172.21.15.192:3300/0 conn(0x7f088c0cedf0 msgr2=0x7f088c0cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:39.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.317+0000 7f088bfff700 1 --2- 172.21.15.189:0/1453596466 >> v2:172.21.15.192:3300/0 conn(0x7f088c0cedf0 0x7f088c0cf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.317+0000 7f088bfff700 1 -- 172.21.15.189:0/1453596466 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c06c660 con 0x7f088c0d17a0 2024-04-06T12:31:39.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.317+0000 7f08908b1700 1 --2- 172.21.15.189:0/1453596466 >> v2:172.21.15.192:3300/0 conn(0x7f088c0cedf0 0x7f088c0cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:39.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.317+0000 7f088bfff700 1 --2- 172.21.15.189:0/1453596466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c0d17a0 0x7f088c06b7a0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f088c162120 tx=0x7f0880009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36cb6552eba57e7f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:39.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.317+0000 7f088affd700 1 -- 172.21.15.189:0/1453596466 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088000b870 con 0x7f088c0d17a0 2024-04-06T12:31:39.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.317+0000 7f088affd700 1 -- 172.21.15.189:0/1453596466 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088000b9d0 con 0x7f088c0d17a0 2024-04-06T12:31:39.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.317+0000 7f088affd700 1 -- 172.21.15.189:0/1453596466 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088000bd60 con 0x7f088c0d17a0 2024-04-06T12:31:39.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.317+0000 7f0892314700 1 -- 172.21.15.189:0/1453596466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c0d17a0 msgr2=0x7f088c06b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:39.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.317+0000 7f0892314700 1 --2- 172.21.15.189:0/1453596466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c0d17a0 0x7f088c06b7a0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f088c162120 tx=0x7f0880009bd0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.317+0000 7f0892314700 1 -- 172.21.15.189:0/1453596466 shutdown_connections 2024-04-06T12:31:39.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.317+0000 7f0892314700 1 --2- 172.21.15.189:0/1453596466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c0d17a0 0x7f088c06b7a0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.317+0000 7f0892314700 1 --2- 172.21.15.189:0/1453596466 >> v2:172.21.15.192:3300/0 conn(0x7f088c0cedf0 0x7f088c0cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.317+0000 7f0892314700 1 --2- 172.21.15.189:0/1453596466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c146dd0 0x7f088c06d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.317+0000 7f0892314700 1 -- 172.21.15.189:0/1453596466 >> 172.21.15.189:0/1453596466 conn(0x7f088c0b2db0 msgr2=0x7f088c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:39.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.317+0000 7f0892314700 1 -- 172.21.15.189:0/1453596466 shutdown_connections 2024-04-06T12:31:39.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.318+0000 7f0892314700 1 -- 172.21.15.189:0/1453596466 wait complete. 2024-04-06T12:31:39.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.319+0000 7f0892314700 1 Processor -- start 2024-04-06T12:31:39.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.319+0000 7f0892314700 1 -- start start 2024-04-06T12:31:39.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f0892314700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c0cedf0 0x7f088c14b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:39.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f0892314700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f088c0d17a0 0x7f088c14bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:39.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f0892314700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c146dd0 0x7f088c14ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:39.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f0892314700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c0c7780 con 0x7f088c0cedf0 2024-04-06T12:31:39.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f0892314700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c149f70 con 0x7f088c0d17a0 2024-04-06T12:31:39.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f0892314700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088c0c74a0 con 0x7f088c146dd0 2024-04-06T12:31:39.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f088bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c0cedf0 0x7f088c14b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:39.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f088bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c0cedf0 0x7f088c14b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40482/0 (socket says 172.21.15.189:40482) 2024-04-06T12:31:39.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f088bfff700 1 -- 172.21.15.189:0/977141897 learned_addr learned my addr 172.21.15.189:0/977141897 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:39.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f08908b1700 1 --2- 172.21.15.189:0/977141897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c146dd0 0x7f088c14ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:39.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f088b7fe700 1 --2- 172.21.15.189:0/977141897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f088c0d17a0 0x7f088c14bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:39.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f088bfff700 1 -- 172.21.15.189:0/977141897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c146dd0 msgr2=0x7f088c14ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:39.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f088bfff700 1 --2- 172.21.15.189:0/977141897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c146dd0 0x7f088c14ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f088bfff700 1 -- 172.21.15.189:0/977141897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f088c0d17a0 msgr2=0x7f088c14bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:39.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f088bfff700 1 --2- 172.21.15.189:0/977141897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f088c0d17a0 0x7f088c14bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f088bfff700 1 -- 172.21.15.189:0/977141897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0880009550 con 0x7f088c0cedf0 2024-04-06T12:31:39.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f088bfff700 1 --2- 172.21.15.189:0/977141897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c0cedf0 0x7f088c14b7c0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f0880000c00 tx=0x7f0880004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:39.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f0888ff9700 1 -- 172.21.15.189:0/977141897 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880004310 con 0x7f088c0cedf0 2024-04-06T12:31:39.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f0888ff9700 1 -- 172.21.15.189:0/977141897 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088001e070 con 0x7f088c0cedf0 2024-04-06T12:31:39.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f0892314700 1 -- 172.21.15.189:0/977141897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f088c1498b0 con 0x7f088c0cedf0 2024-04-06T12:31:39.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f0888ff9700 1 -- 172.21.15.189:0/977141897 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08800036a0 con 0x7f088c0cedf0 2024-04-06T12:31:39.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.320+0000 7f0892314700 1 -- 172.21.15.189:0/977141897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f088c14f180 con 0x7f088c0cedf0 2024-04-06T12:31:39.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.321+0000 7f0892314700 1 -- 172.21.15.189:0/977141897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f088c074990 con 0x7f088c0cedf0 2024-04-06T12:31:39.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.322+0000 7f0888ff9700 1 -- 172.21.15.189:0/977141897 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 80) v1 ==== 89265+0+0 (secure 0 0 0) 0x7f0880020030 con 0x7f088c0cedf0 2024-04-06T12:31:39.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.322+0000 7f0888ff9700 1 --2- 172.21.15.189:0/977141897 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f087806ae00 0x7f087806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:39.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.322+0000 7f0888ff9700 1 -- 172.21.15.189:0/977141897 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f088008a600 con 0x7f088c0cedf0 2024-04-06T12:31:39.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.322+0000 7f088b7fe700 1 --2- 172.21.15.189:0/977141897 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f087806ae00 0x7f087806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:39.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.323+0000 7f088b7fe700 1 --2- 172.21.15.189:0/977141897 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f087806ae00 0x7f087806d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f088c0acc50 tx=0x7f087c008040 comp rx=0 tx=0).ready entity=mgr.8775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:39.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.326+0000 7f0888ff9700 1 -- 172.21.15.189:0/977141897 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f08800554c0 con 0x7f088c0cedf0 2024-04-06T12:31:39.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.547+0000 7f0892314700 1 -- 172.21.15.189:0/977141897 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/yearly", "snap_schedule": "1Y", "target": ["mon-mgr", ""]}) v1 -- 0x7f088c149730 con 0x7f087806ae00 2024-04-06T12:31:39.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.606+0000 7f0888ff9700 1 -- 172.21.15.189:0/977141897 <== mgr.8775 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f087c007e80 con 0x7f087806ae00 2024-04-06T12:31:39.604 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path snap_test_dir1/yearly 2024-04-06T12:31:39.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.608+0000 7f0892314700 1 -- 172.21.15.189:0/977141897 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f087806ae00 msgr2=0x7f087806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:39.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.608+0000 7f0892314700 1 --2- 172.21.15.189:0/977141897 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f087806ae00 0x7f087806d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f088c0acc50 tx=0x7f087c008040 comp rx=0 tx=0).stop 2024-04-06T12:31:39.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.608+0000 7f0892314700 1 -- 172.21.15.189:0/977141897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c0cedf0 msgr2=0x7f088c14b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:39.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.608+0000 7f0892314700 1 --2- 172.21.15.189:0/977141897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c0cedf0 0x7f088c14b7c0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f0880000c00 tx=0x7f0880004750 comp rx=0 tx=0).stop 2024-04-06T12:31:39.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.608+0000 7f0892314700 1 -- 172.21.15.189:0/977141897 shutdown_connections 2024-04-06T12:31:39.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.608+0000 7f0892314700 1 --2- 172.21.15.189:0/977141897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f088c146dd0 0x7f088c14ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.608+0000 7f0892314700 1 --2- 172.21.15.189:0/977141897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c0cedf0 0x7f088c14b7c0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.608+0000 7f0892314700 1 --2- 172.21.15.189:0/977141897 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f087806ae00 0x7f087806d2c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.608+0000 7f0892314700 1 --2- 172.21.15.189:0/977141897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f088c0d17a0 0x7f088c14bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.608+0000 7f0892314700 1 -- 172.21.15.189:0/977141897 >> 172.21.15.189:0/977141897 conn(0x7f088c0b2db0 msgr2=0x7f088c0b3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:39.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.608+0000 7f0892314700 1 -- 172.21.15.189:0/977141897 shutdown_connections 2024-04-06T12:31:39.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.608+0000 7f0892314700 1 -- 172.21.15.189:0/977141897 wait complete. 2024-04-06T12:31:39.618 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/yearly 2024-04-06T12:31:39.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:39.619 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/bad_period_spec) 2024-04-06T12:31:39.637 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1X 2024-04-06T12:31:39.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.804+0000 7f1ac7aef700 1 Processor -- start 2024-04-06T12:31:39.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.804+0000 7f1ac7aef700 1 -- start start 2024-04-06T12:31:39.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac7aef700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1ac00d17a0 0x7f1ac006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:39.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac7aef700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac006d850 con 0x7f1ac00cedf0 2024-04-06T12:31:39.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac7aef700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac006c0a0 con 0x7f1ac0146dd0 2024-04-06T12:31:39.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac7aef700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac006c380 con 0x7f1ac00d17a0 2024-04-06T12:31:39.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac608c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1ac00cedf0 0x7f1ac00cf1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49064/0 (socket says 172.21.15.189:49064) 2024-04-06T12:31:39.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac608c700 1 -- 172.21.15.189:0/3012388819 learned_addr learned my addr 172.21.15.189:0/3012388819 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:39.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac588b700 1 --2- 172.21.15.189:0/3012388819 >> v2:172.21.15.192:3300/0 conn(0x7f1ac00d17a0 0x7f1ac006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:39.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac4889700 1 -- 172.21.15.189:0/3012388819 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731739117 0 0) 0x7f1ac006d850 con 0x7f1ac00cedf0 2024-04-06T12:31:39.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac4889700 1 -- 172.21.15.189:0/3012388819 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aac0035b0 con 0x7f1ac00cedf0 2024-04-06T12:31:39.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac4889700 1 -- 172.21.15.189:0/3012388819 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3817973764 0 0) 0x7f1aac0035b0 con 0x7f1ac00cedf0 2024-04-06T12:31:39.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac4889700 1 -- 172.21.15.189:0/3012388819 >> v2:172.21.15.192:3300/0 conn(0x7f1ac00d17a0 msgr2=0x7f1ac006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:39.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac4889700 1 --2- 172.21.15.189:0/3012388819 >> v2:172.21.15.192:3300/0 conn(0x7f1ac00d17a0 0x7f1ac006b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac4889700 1 -- 172.21.15.189:0/3012388819 >> v1:172.21.15.192:6789/0 conn(0x7f1ac0146dd0 legacy=0x7f1ac006d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:39.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac4889700 1 -- 172.21.15.189:0/3012388819 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac006c660 con 0x7f1ac00cedf0 2024-04-06T12:31:39.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac4889700 1 -- 172.21.15.189:0/3012388819 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f1ab80019f0 con 0x7f1ac00cedf0 2024-04-06T12:31:39.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac4889700 1 -- 172.21.15.189:0/3012388819 >> v1:172.21.15.189:6789/0 conn(0x7f1ac00cedf0 legacy=0x7f1ac00cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:39.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac4889700 1 --2- 172.21.15.189:0/3012388819 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1aac003a00 0x7f1aac003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:39.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac4889700 1 --2- 172.21.15.189:0/3012388819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1aac004850 0x7f1aac004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:39.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac4889700 1 --2- 172.21.15.189:0/3012388819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aac005210 0x7f1aac00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:39.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac4889700 1 -- 172.21.15.189:0/3012388819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac006c380 con 0x7f1aac005210 2024-04-06T12:31:39.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac4889700 1 -- 172.21.15.189:0/3012388819 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac00c0b0 con 0x7f1aac004850 2024-04-06T12:31:39.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac4889700 1 -- 172.21.15.189:0/3012388819 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac00c270 con 0x7f1aac003a00 2024-04-06T12:31:39.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.805+0000 7f1ac608c700 1 --2- 172.21.15.189:0/3012388819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aac005210 0x7f1aac00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:39.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.806+0000 7f1ac508a700 1 --2- 172.21.15.189:0/3012388819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1aac004850 0x7f1aac004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:39.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.806+0000 7f1ac588b700 1 --2- 172.21.15.189:0/3012388819 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1aac003a00 0x7f1aac003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:39.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.806+0000 7f1ac608c700 1 -- 172.21.15.189:0/3012388819 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1aac003a00 msgr2=0x7f1aac003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:39.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.806+0000 7f1ac608c700 1 --2- 172.21.15.189:0/3012388819 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1aac003a00 0x7f1aac003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.806+0000 7f1ac608c700 1 -- 172.21.15.189:0/3012388819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1aac004850 msgr2=0x7f1aac004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:39.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.806+0000 7f1ac608c700 1 --2- 172.21.15.189:0/3012388819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1aac004850 0x7f1aac004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.806+0000 7f1ac608c700 1 -- 172.21.15.189:0/3012388819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aac0035b0 con 0x7f1aac005210 2024-04-06T12:31:39.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.806+0000 7f1ac588b700 1 --2- 172.21.15.189:0/3012388819 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1aac003a00 0x7f1aac003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:39.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.806+0000 7f1ac608c700 1 -- 172.21.15.189:0/3012388819 reap_dead start 2024-04-06T12:31:39.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.806+0000 7f1ac608c700 1 --2- 172.21.15.189:0/3012388819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aac005210 0x7f1aac00ba10 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f1ab800a830 tx=0x7f1aac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f45df124b5a5b2c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:39.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.806+0000 7f1ac4889700 1 -- 172.21.15.189:0/3012388819 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab800a2f0 con 0x7f1aac005210 2024-04-06T12:31:39.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.806+0000 7f1ac4889700 1 -- 172.21.15.189:0/3012388819 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab8012600 con 0x7f1aac005210 2024-04-06T12:31:39.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.806+0000 7f1ac4889700 1 -- 172.21.15.189:0/3012388819 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab80129b0 con 0x7f1aac005210 2024-04-06T12:31:39.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.806+0000 7f1ac7aef700 1 -- 172.21.15.189:0/3012388819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aac005210 msgr2=0x7f1aac00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:39.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.806+0000 7f1ac7aef700 1 --2- 172.21.15.189:0/3012388819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aac005210 0x7f1aac00ba10 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f1ab800a830 tx=0x7f1aac005880 comp rx=0 tx=0).stop 2024-04-06T12:31:39.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.807+0000 7f1ac7aef700 1 -- 172.21.15.189:0/3012388819 shutdown_connections 2024-04-06T12:31:39.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.807+0000 7f1ac7aef700 1 --2- 172.21.15.189:0/3012388819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aac005210 0x7f1aac00ba10 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.807+0000 7f1ac7aef700 1 -- 172.21.15.189:0/3012388819 >> 172.21.15.189:0/3012388819 conn(0x7f1ac00b2db0 msgr2=0x7f1ac00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:39.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.807+0000 7f1ac7aef700 1 -- 172.21.15.189:0/3012388819 shutdown_connections 2024-04-06T12:31:39.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.807+0000 7f1ac7aef700 1 -- 172.21.15.189:0/3012388819 wait complete. 2024-04-06T12:31:39.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.808+0000 7f1ac7aef700 1 Processor -- start 2024-04-06T12:31:39.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.808+0000 7f1ac7aef700 1 -- start start 2024-04-06T12:31:39.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.808+0000 7f1ac7aef700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1aac005210 0x7f1ac014fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:39.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.808+0000 7f1ac7aef700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ac01f6e90 0x7f1ac01f5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:39.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.808+0000 7f1ac7aef700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac01f5ea0 0x7f1ac00b1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:39.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.809+0000 7f1ac7aef700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac0150020 con 0x7f1ac01f5ea0 2024-04-06T12:31:39.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.809+0000 7f1ac7aef700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac0150e20 con 0x7f1ac01f6e90 2024-04-06T12:31:39.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.809+0000 7f1ac7aef700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac01f7310 con 0x7f1aac005210 2024-04-06T12:31:39.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.809+0000 7f1ac608c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac01f5ea0 0x7f1ac00b1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:39.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.809+0000 7f1ac608c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac01f5ea0 0x7f1ac00b1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40502/0 (socket says 172.21.15.189:40502) 2024-04-06T12:31:39.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.809+0000 7f1ac608c700 1 -- 172.21.15.189:0/1815809293 learned_addr learned my addr 172.21.15.189:0/1815809293 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:39.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.809+0000 7f1ac588b700 1 --2- 172.21.15.189:0/1815809293 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1aac005210 0x7f1ac014fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:39.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.809+0000 7f1ac508a700 1 --2- 172.21.15.189:0/1815809293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ac01f6e90 0x7f1ac01f5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:39.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.809+0000 7f1ac608c700 1 -- 172.21.15.189:0/1815809293 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1aac005210 msgr2=0x7f1ac014fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:39.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.809+0000 7f1ac608c700 1 --2- 172.21.15.189:0/1815809293 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1aac005210 0x7f1ac014fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.809+0000 7f1ac608c700 1 -- 172.21.15.189:0/1815809293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ac01f6e90 msgr2=0x7f1ac01f5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:39.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.809+0000 7f1ac608c700 1 --2- 172.21.15.189:0/1815809293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ac01f6e90 0x7f1ac01f5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:39.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.809+0000 7f1ac608c700 1 -- 172.21.15.189:0/1815809293 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac0150210 con 0x7f1ac01f5ea0 2024-04-06T12:31:39.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.809+0000 7f1ac608c700 1 --2- 172.21.15.189:0/1815809293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac01f5ea0 0x7f1ac00b1b30 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f1ac015dad0 tx=0x7f1ab8001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:39.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.809+0000 7f1ab67fc700 1 -- 172.21.15.189:0/1815809293 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8001250 con 0x7f1ac01f5ea0 2024-04-06T12:31:39.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.809+0000 7f1ab67fc700 1 -- 172.21.15.189:0/1815809293 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab80017f0 con 0x7f1ac01f5ea0 2024-04-06T12:31:39.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.809+0000 7f1ac7aef700 1 -- 172.21.15.189:0/1815809293 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac01504f0 con 0x7f1ac01f5ea0 2024-04-06T12:31:39.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.809+0000 7f1ab67fc700 1 -- 172.21.15.189:0/1815809293 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab801e670 con 0x7f1ac01f5ea0 2024-04-06T12:31:39.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.809+0000 7f1ac7aef700 1 -- 172.21.15.189:0/1815809293 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac0150ab0 con 0x7f1ac01f5ea0 2024-04-06T12:31:39.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.810+0000 7f1ac7aef700 1 -- 172.21.15.189:0/1815809293 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac0150c40 con 0x7f1ac01f5ea0 2024-04-06T12:31:39.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.811+0000 7f1ab67fc700 1 -- 172.21.15.189:0/1815809293 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 80) v1 ==== 89265+0+0 (secure 0 0 0) 0x7f1ab801e7d0 con 0x7f1ac01f5ea0 2024-04-06T12:31:39.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.811+0000 7f1ab67fc700 1 --2- 172.21.15.189:0/1815809293 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1aa406ae00 0x7f1aa406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:39.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.811+0000 7f1ab67fc700 1 -- 172.21.15.189:0/1815809293 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f1ab808b760 con 0x7f1ac01f5ea0 2024-04-06T12:31:39.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.811+0000 7f1ac588b700 1 --2- 172.21.15.189:0/1815809293 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1aa406ae00 0x7f1aa406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:39.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.811+0000 7f1ac588b700 1 --2- 172.21.15.189:0/1815809293 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1aa406ae00 0x7f1aa406d2c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0009b40 tx=0x7f1ab0008040 comp rx=0 tx=0).ready entity=mgr.8775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:39.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:39.816+0000 7f1ab67fc700 1 -- 172.21.15.189:0/1815809293 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1ab80565a0 con 0x7f1ac01f5ea0 2024-04-06T12:31:40.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.037+0000 7f1ac7aef700 1 -- 172.21.15.189:0/1815809293 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1X", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ac00d1b80 con 0x7f1aa406ae00 2024-04-06T12:31:40.036 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:40.038+0000 7fb6d2547700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - multiplier should be one of h,d,w,M,Y 2024-04-06T12:31:40.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.038+0000 7f1ab67fc700 1 -- 172.21.15.189:0/1815809293 <== mgr.8775 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - multiplier should be one of h,d,w,M,Y) v1 ==== 74+0+0 (secure 0 0 0) 0x7f1ab0013890 con 0x7f1aa406ae00 2024-04-06T12:31:40.038 INFO:teuthology.orchestra.run.smithi189.stderr:Error ENOENT: invalid schedule specified - multiplier should be one of h,d,w,M,Y 2024-04-06T12:31:40.041 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-06T12:31:40.042 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1MM 2024-04-06T12:31:40.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.173+0000 7f97ea5a2700 1 Processor -- start 2024-04-06T12:31:40.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.173+0000 7f97ea5a2700 1 -- start start 2024-04-06T12:31:40.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.174+0000 7f97ea5a2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f97e40d11b0 0x7f97e40d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:40.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.174+0000 7f97ea5a2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e40d1ad0 con 0x7f97e40cb870 2024-04-06T12:31:40.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.174+0000 7f97ea5a2700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e40cbd80 con 0x7f97e40cfa00 2024-04-06T12:31:40.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.174+0000 7f97ea5a2700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e40ccdc0 con 0x7f97e40d11b0 2024-04-06T12:31:40.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.174+0000 7f97e3fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f97e40d11b0 0x7f97e40d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:40.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.174+0000 7f97e3fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f97e40d11b0 0x7f97e40d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40504/0 (socket says 172.21.15.189:40504) 2024-04-06T12:31:40.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.174+0000 7f97e3fff700 1 -- 172.21.15.189:0/4214728961 learned_addr learned my addr 172.21.15.189:0/4214728961 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:40.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.174+0000 7f97e3fff700 1 -- 172.21.15.189:0/4214728961 >> v1:172.21.15.192:6789/0 conn(0x7f97e40cfa00 legacy=0x7f97e40cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:40.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.174+0000 7f97e3fff700 1 -- 172.21.15.189:0/4214728961 >> v1:172.21.15.189:6789/0 conn(0x7f97e40cb870 legacy=0x7f97e415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:40.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.174+0000 7f97e3fff700 1 -- 172.21.15.189:0/4214728961 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e40cef20 con 0x7f97e40d11b0 2024-04-06T12:31:40.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.174+0000 7f97e3fff700 1 --2- 172.21.15.189:0/4214728961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97e40d11b0 0x7f97e40d1590 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f97d80098f0 tx=0x7f97d8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f50a6b13a9d1bb9f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:40.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.175+0000 7f97e2ffd700 1 -- 172.21.15.189:0/4214728961 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d8004230 con 0x7f97e40d11b0 2024-04-06T12:31:40.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.175+0000 7f97e2ffd700 1 -- 172.21.15.189:0/4214728961 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d8004390 con 0x7f97e40d11b0 2024-04-06T12:31:40.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.175+0000 7f97e2ffd700 1 -- 172.21.15.189:0/4214728961 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d80036a0 con 0x7f97e40d11b0 2024-04-06T12:31:40.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.175+0000 7f97ea5a2700 1 -- 172.21.15.189:0/4214728961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97e40d11b0 msgr2=0x7f97e40d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:40.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.175+0000 7f97ea5a2700 1 --2- 172.21.15.189:0/4214728961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97e40d11b0 0x7f97e40d1590 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f97d80098f0 tx=0x7f97d8009c00 comp rx=0 tx=0).stop 2024-04-06T12:31:40.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.175+0000 7f97ea5a2700 1 -- 172.21.15.189:0/4214728961 shutdown_connections 2024-04-06T12:31:40.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.175+0000 7f97ea5a2700 1 --2- 172.21.15.189:0/4214728961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97e40d11b0 0x7f97e40d1590 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:40.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.175+0000 7f97ea5a2700 1 -- 172.21.15.189:0/4214728961 >> 172.21.15.189:0/4214728961 conn(0x7f97e40b2db0 msgr2=0x7f97e40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:40.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.175+0000 7f97ea5a2700 1 -- 172.21.15.189:0/4214728961 shutdown_connections 2024-04-06T12:31:40.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.175+0000 7f97ea5a2700 1 -- 172.21.15.189:0/4214728961 wait complete. 2024-04-06T12:31:40.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.176+0000 7f97ea5a2700 1 Processor -- start 2024-04-06T12:31:40.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.177+0000 7f97ea5a2700 1 -- start start 2024-04-06T12:31:40.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.177+0000 7f97ea5a2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97e40cb870 0x7f97e4148730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:40.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.177+0000 7f97ea5a2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97e40cfa00 0x7f97e40b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:40.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.177+0000 7f97ea5a2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97e40d11b0 0x7f97e40b5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:40.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.177+0000 7f97ea5a2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e4155930 con 0x7f97e40d11b0 2024-04-06T12:31:40.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.177+0000 7f97ea5a2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e40c4ed0 con 0x7f97e40cfa00 2024-04-06T12:31:40.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.177+0000 7f97ea5a2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e40c5170 con 0x7f97e40cb870 2024-04-06T12:31:40.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.177+0000 7f97e8b3f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97e40d11b0 0x7f97e40b5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:40.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.177+0000 7f97e8b3f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97e40d11b0 0x7f97e40b5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40518/0 (socket says 172.21.15.189:40518) 2024-04-06T12:31:40.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.177+0000 7f97e8b3f700 1 -- 172.21.15.189:0/444663991 learned_addr learned my addr 172.21.15.189:0/444663991 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:40.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.178+0000 7f97e3fff700 1 --2- 172.21.15.189:0/444663991 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97e40cb870 0x7f97e4148730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:40.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.178+0000 7f97e37fe700 1 --2- 172.21.15.189:0/444663991 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97e40cfa00 0x7f97e40b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:40.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.178+0000 7f97e8b3f700 1 -- 172.21.15.189:0/444663991 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97e40cb870 msgr2=0x7f97e4148730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:40.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.178+0000 7f97e8b3f700 1 --2- 172.21.15.189:0/444663991 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97e40cb870 0x7f97e4148730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:40.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.178+0000 7f97e8b3f700 1 -- 172.21.15.189:0/444663991 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97e40cfa00 msgr2=0x7f97e40b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:40.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.178+0000 7f97e8b3f700 1 --2- 172.21.15.189:0/444663991 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97e40cfa00 0x7f97e40b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:40.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.178+0000 7f97e8b3f700 1 -- 172.21.15.189:0/444663991 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d8009550 con 0x7f97e40d11b0 2024-04-06T12:31:40.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.178+0000 7f97e3fff700 1 --2- 172.21.15.189:0/444663991 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97e40cb870 0x7f97e4148730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:40.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.178+0000 7f97e8b3f700 1 --2- 172.21.15.189:0/444663991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97e40d11b0 0x7f97e40b5ff0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f97dc00c7a0 tx=0x7f97dc00cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:40.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.178+0000 7f97e0ff9700 1 -- 172.21.15.189:0/444663991 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc003e10 con 0x7f97e40d11b0 2024-04-06T12:31:40.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.178+0000 7f97e0ff9700 1 -- 172.21.15.189:0/444663991 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97dc005070 con 0x7f97e40d11b0 2024-04-06T12:31:40.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.178+0000 7f97e0ff9700 1 -- 172.21.15.189:0/444663991 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc005420 con 0x7f97e40d11b0 2024-04-06T12:31:40.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.178+0000 7f97ea5a2700 1 -- 172.21.15.189:0/444663991 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e40c5450 con 0x7f97e40d11b0 2024-04-06T12:31:40.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.178+0000 7f97ea5a2700 1 -- 172.21.15.189:0/444663991 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e40c5a10 con 0x7f97e40d11b0 2024-04-06T12:31:40.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.179+0000 7f97ea5a2700 1 -- 172.21.15.189:0/444663991 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e40c5c30 con 0x7f97e40d11b0 2024-04-06T12:31:40.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.180+0000 7f97e0ff9700 1 -- 172.21.15.189:0/444663991 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 81) v1 ==== 89321+0+0 (secure 0 0 0) 0x7f97dc005580 con 0x7f97e40d11b0 2024-04-06T12:31:40.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.180+0000 7f97e0ff9700 1 --2- 172.21.15.189:0/444663991 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f97d006ae90 0x7f97d006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:40.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.180+0000 7f97e0ff9700 1 -- 172.21.15.189:0/444663991 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f97dc088b40 con 0x7f97e40d11b0 2024-04-06T12:31:40.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.180+0000 7f97e3fff700 1 --2- 172.21.15.189:0/444663991 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f97d006ae90 0x7f97d006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:40.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.180+0000 7f97e3fff700 1 --2- 172.21.15.189:0/444663991 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f97d006ae90 0x7f97d006d350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f97d8009130 tx=0x7f97d8008030 comp rx=0 tx=0).ready entity=mgr.8775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:40.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.184+0000 7f97e0ff9700 1 -- 172.21.15.189:0/444663991 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f97dc053800 con 0x7f97e40d11b0 2024-04-06T12:31:40.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.407+0000 7f97ea5a2700 1 -- 172.21.15.189:0/444663991 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1MM", "target": ["mon-mgr", ""]}) v1 -- 0x7f97e40cc060 con 0x7f97d006ae90 2024-04-06T12:31:40.406 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:40.408+0000 7fb6d2547700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-04-06T12:31:40.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.408+0000 7f97e0ff9700 1 -- 172.21.15.189:0/444663991 <== mgr.8775 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f97d8020070 con 0x7f97d006ae90 2024-04-06T12:31:40.410 INFO:teuthology.orchestra.run.smithi189.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-04-06T12:31:40.411 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-06T12:31:40.412 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1 2024-04-06T12:31:40.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.539+0000 7f64744d0700 1 Processor -- start 2024-04-06T12:31:40.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.539+0000 7f64744d0700 1 -- start start 2024-04-06T12:31:40.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.540+0000 7f64744d0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f646c0b4660 0x7f646c0b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:40.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.540+0000 7f64744d0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646c0cdbe0 con 0x7f646c0b2fc0 2024-04-06T12:31:40.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.540+0000 7f64744d0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646c0b5000 con 0x7f646c15a550 2024-04-06T12:31:40.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.540+0000 7f64744d0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646c15a940 con 0x7f646c0b4660 2024-04-06T12:31:40.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.540+0000 7f6472a6d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f646c0b2fc0 0x7f646c0b3430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49088/0 (socket says 172.21.15.189:49088) 2024-04-06T12:31:40.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.540+0000 7f6472a6d700 1 -- 172.21.15.189:0/1538091600 learned_addr learned my addr 172.21.15.189:0/1538091600 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:40.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.540+0000 7f6471a6b700 1 --2- 172.21.15.189:0/1538091600 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f646c0b4660 0x7f646c0b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:40.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.540+0000 7f647126a700 1 -- 172.21.15.189:0/1538091600 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956348644 0 0) 0x7f646c0cdbe0 con 0x7f646c0b2fc0 2024-04-06T12:31:40.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.540+0000 7f647126a700 1 -- 172.21.15.189:0/1538091600 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64580035b0 con 0x7f646c0b2fc0 2024-04-06T12:31:40.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.540+0000 7f647126a700 1 -- 172.21.15.189:0/1538091600 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747549804 0 0) 0x7f646c0b5000 con 0x7f646c15a550 2024-04-06T12:31:40.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.540+0000 7f647126a700 1 -- 172.21.15.189:0/1538091600 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f646c0cdbe0 con 0x7f646c15a550 2024-04-06T12:31:40.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.541+0000 7f6471a6b700 1 -- 172.21.15.189:0/1538091600 >> v1:172.21.15.192:6789/0 conn(0x7f646c15a550 legacy=0x7f646c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:40.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.541+0000 7f6471a6b700 1 -- 172.21.15.189:0/1538091600 >> v1:172.21.15.189:6789/0 conn(0x7f646c0b2fc0 legacy=0x7f646c0b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:40.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.541+0000 7f6471a6b700 1 -- 172.21.15.189:0/1538091600 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c15ac00 con 0x7f646c0b4660 2024-04-06T12:31:40.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.541+0000 7f6471a6b700 1 --2- 172.21.15.189:0/1538091600 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f646c0b4660 0x7f646c0b4ac0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6468009870 tx=0x7f6468009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97ccaf0b1ebfa16e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:40.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.541+0000 7f647126a700 1 -- 172.21.15.189:0/1538091600 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6468011830 con 0x7f646c0b4660 2024-04-06T12:31:40.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.541+0000 7f647126a700 1 -- 172.21.15.189:0/1538091600 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6468011990 con 0x7f646c0b4660 2024-04-06T12:31:40.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.541+0000 7f647126a700 1 -- 172.21.15.189:0/1538091600 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6468011d00 con 0x7f646c0b4660 2024-04-06T12:31:40.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.541+0000 7f64744d0700 1 -- 172.21.15.189:0/1538091600 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f646c0b4660 msgr2=0x7f646c0b4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:40.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.541+0000 7f64744d0700 1 --2- 172.21.15.189:0/1538091600 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f646c0b4660 0x7f646c0b4ac0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6468009870 tx=0x7f6468009b80 comp rx=0 tx=0).stop 2024-04-06T12:31:40.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.542+0000 7f64744d0700 1 -- 172.21.15.189:0/1538091600 shutdown_connections 2024-04-06T12:31:40.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.542+0000 7f64744d0700 1 --2- 172.21.15.189:0/1538091600 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f646c0b4660 0x7f646c0b4ac0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:40.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.542+0000 7f64744d0700 1 -- 172.21.15.189:0/1538091600 >> 172.21.15.189:0/1538091600 conn(0x7f646c0d00b0 msgr2=0x7f646c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:40.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.542+0000 7f64744d0700 1 -- 172.21.15.189:0/1538091600 shutdown_connections 2024-04-06T12:31:40.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.542+0000 7f64744d0700 1 -- 172.21.15.189:0/1538091600 wait complete. 2024-04-06T12:31:40.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.543+0000 7f64744d0700 1 Processor -- start 2024-04-06T12:31:40.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.543+0000 7f64744d0700 1 -- start start 2024-04-06T12:31:40.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.543+0000 7f64744d0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f646c0b2fc0 0x7f646c157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:40.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.543+0000 7f64744d0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f646c0b4660 0x7f646c0aebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:40.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.543+0000 7f64744d0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f646c0af100 0x7f646c15cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:40.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.543+0000 7f64744d0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646c0ae730 con 0x7f646c0b4660 2024-04-06T12:31:40.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.543+0000 7f64744d0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646c0adb20 con 0x7f646c0af100 2024-04-06T12:31:40.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.543+0000 7f64744d0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646c0ade00 con 0x7f646c0b2fc0 2024-04-06T12:31:40.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.544+0000 7f6471a6b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f646c0b4660 0x7f646c0aebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:40.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.544+0000 7f6471a6b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f646c0b4660 0x7f646c0aebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40526/0 (socket says 172.21.15.189:40526) 2024-04-06T12:31:40.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.544+0000 7f6471a6b700 1 -- 172.21.15.189:0/869730152 learned_addr learned my addr 172.21.15.189:0/869730152 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:40.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.544+0000 7f6472a6d700 1 --2- 172.21.15.189:0/869730152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f646c0af100 0x7f646c15cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:40.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.544+0000 7f647226c700 1 --2- 172.21.15.189:0/869730152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f646c0b2fc0 0x7f646c157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:40.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.544+0000 7f6471a6b700 1 -- 172.21.15.189:0/869730152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f646c0b2fc0 msgr2=0x7f646c157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:40.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.544+0000 7f6471a6b700 1 --2- 172.21.15.189:0/869730152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f646c0b2fc0 0x7f646c157830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:40.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.544+0000 7f6471a6b700 1 -- 172.21.15.189:0/869730152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f646c0af100 msgr2=0x7f646c15cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:40.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.544+0000 7f6471a6b700 1 --2- 172.21.15.189:0/869730152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f646c0af100 0x7f646c15cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:40.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.544+0000 7f6471a6b700 1 -- 172.21.15.189:0/869730152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64680094a0 con 0x7f646c0b4660 2024-04-06T12:31:40.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.544+0000 7f6471a6b700 1 --2- 172.21.15.189:0/869730152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f646c0b4660 0x7f646c0aebc0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f64680097c0 tx=0x7f6468004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:40.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.544+0000 7f6462ffd700 1 -- 172.21.15.189:0/869730152 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64680044d0 con 0x7f646c0b4660 2024-04-06T12:31:40.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.544+0000 7f6462ffd700 1 -- 172.21.15.189:0/869730152 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646801f070 con 0x7f646c0b4660 2024-04-06T12:31:40.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.544+0000 7f64744d0700 1 -- 172.21.15.189:0/869730152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f646c0ae0e0 con 0x7f646c0b4660 2024-04-06T12:31:40.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.544+0000 7f6462ffd700 1 -- 172.21.15.189:0/869730152 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64680038e0 con 0x7f646c0b4660 2024-04-06T12:31:40.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.544+0000 7f64744d0700 1 -- 172.21.15.189:0/869730152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f646c157d70 con 0x7f646c0b4660 2024-04-06T12:31:40.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.545+0000 7f64744d0700 1 -- 172.21.15.189:0/869730152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f646c067bb0 con 0x7f646c0b4660 2024-04-06T12:31:40.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.546+0000 7f6462ffd700 1 -- 172.21.15.189:0/869730152 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 81) v1 ==== 89321+0+0 (secure 0 0 0) 0x7f6468003a40 con 0x7f646c0b4660 2024-04-06T12:31:40.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.546+0000 7f6462ffd700 1 --2- 172.21.15.189:0/869730152 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f645006ae90 0x7f645006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:40.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.546+0000 7f647226c700 1 --2- 172.21.15.189:0/869730152 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f645006ae90 0x7f645006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:40.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.546+0000 7f6462ffd700 1 -- 172.21.15.189:0/869730152 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f6468021070 con 0x7f646c0b4660 2024-04-06T12:31:40.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.546+0000 7f647226c700 1 --2- 172.21.15.189:0/869730152 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f645006ae90 0x7f645006d350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f645c001830 tx=0x7f645c009040 comp rx=0 tx=0).ready entity=mgr.8775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:40.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.551+0000 7f6462ffd700 1 -- 172.21.15.189:0/869730152 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6468055da0 con 0x7f646c0b4660 2024-04-06T12:31:40.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.772+0000 7f64744d0700 1 -- 172.21.15.189:0/869730152 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f646c15b4a0 con 0x7f645006ae90 2024-04-06T12:31:40.771 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:40.774+0000 7fb6d2547700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-04-06T12:31:40.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.774+0000 7f6462ffd700 1 -- 172.21.15.189:0/869730152 <== mgr.8775 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f645c001d00 con 0x7f645006ae90 2024-04-06T12:31:40.773 INFO:teuthology.orchestra.run.smithi189.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-04-06T12:31:40.777 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-06T12:31:40.777 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec M 2024-04-06T12:31:40.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.903+0000 7f8815872700 1 Processor -- start 2024-04-06T12:31:40.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.903+0000 7f8815872700 1 -- start start 2024-04-06T12:31:40.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.904+0000 7f8815872700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f881015d6f0 0x7f881015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:40.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.904+0000 7f8815872700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f881015eac0 con 0x7f88100ce730 2024-04-06T12:31:40.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.904+0000 7f8815872700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f881015e220 con 0x7f88100b8d70 2024-04-06T12:31:40.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.904+0000 7f8815872700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f881015e010 con 0x7f881015d6f0 2024-04-06T12:31:40.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.904+0000 7f880effd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f881015d6f0 0x7f881015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:40.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.904+0000 7f880effd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f881015d6f0 0x7f881015dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40530/0 (socket says 172.21.15.189:40530) 2024-04-06T12:31:40.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.904+0000 7f880effd700 1 -- 172.21.15.189:0/963695031 learned_addr learned my addr 172.21.15.189:0/963695031 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:40.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.904+0000 7f880effd700 1 -- 172.21.15.189:0/963695031 >> v1:172.21.15.192:6789/0 conn(0x7f88100b8d70 legacy=0x7f88100ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:40.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.904+0000 7f880effd700 1 -- 172.21.15.189:0/963695031 >> v1:172.21.15.189:6789/0 conn(0x7f88100ce730 legacy=0x7f88100d1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:40.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.904+0000 7f880dffb700 1 -- 172.21.15.189:0/963695031 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681738928 0 0) 0x7f881015e220 con 0x7f88100b8d70 2024-04-06T12:31:40.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.904+0000 7f880effd700 1 -- 172.21.15.189:0/963695031 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881015e500 con 0x7f881015d6f0 2024-04-06T12:31:40.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.904+0000 7f880effd700 1 --2- 172.21.15.189:0/963695031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f881015d6f0 0x7f881015dad0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f88000098f0 tx=0x7f8800009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b707056581980e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:40.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.904+0000 7f880dffb700 1 -- 172.21.15.189:0/963695031 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8800004230 con 0x7f881015d6f0 2024-04-06T12:31:40.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.905+0000 7f880dffb700 1 -- 172.21.15.189:0/963695031 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8800004390 con 0x7f881015d6f0 2024-04-06T12:31:40.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.905+0000 7f880dffb700 1 -- 172.21.15.189:0/963695031 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88000036a0 con 0x7f881015d6f0 2024-04-06T12:31:40.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.905+0000 7f8815872700 1 -- 172.21.15.189:0/963695031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f881015d6f0 msgr2=0x7f881015dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:40.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.905+0000 7f8815872700 1 --2- 172.21.15.189:0/963695031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f881015d6f0 0x7f881015dad0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f88000098f0 tx=0x7f8800009c00 comp rx=0 tx=0).stop 2024-04-06T12:31:40.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.905+0000 7f8815872700 1 -- 172.21.15.189:0/963695031 shutdown_connections 2024-04-06T12:31:40.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.905+0000 7f8815872700 1 --2- 172.21.15.189:0/963695031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f881015d6f0 0x7f881015dad0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:40.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.905+0000 7f8815872700 1 -- 172.21.15.189:0/963695031 >> 172.21.15.189:0/963695031 conn(0x7f8810072490 msgr2=0x7f881006b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:40.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.905+0000 7f8815872700 1 -- 172.21.15.189:0/963695031 shutdown_connections 2024-04-06T12:31:40.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.905+0000 7f8815872700 1 -- 172.21.15.189:0/963695031 wait complete. 2024-04-06T12:31:40.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.906+0000 7f8815872700 1 Processor -- start 2024-04-06T12:31:40.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.906+0000 7f8815872700 1 -- start start 2024-04-06T12:31:40.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.907+0000 7f8815872700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88100b8d70 0x7f88101f3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:40.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.907+0000 7f8815872700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88100ce730 0x7f88100cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:40.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.907+0000 7f8815872700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f881015d6f0 0x7f88100b6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:40.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.907+0000 7f8815872700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88101f4140 con 0x7f88100ce730 2024-04-06T12:31:40.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.907+0000 7f8815872700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88100cc6f0 con 0x7f88100b8d70 2024-04-06T12:31:40.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.907+0000 7f8815872700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88101f3770 con 0x7f881015d6f0 2024-04-06T12:31:40.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.907+0000 7f880e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88100ce730 0x7f88100cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:40.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.907+0000 7f880e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88100ce730 0x7f88100cbac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40534/0 (socket says 172.21.15.189:40534) 2024-04-06T12:31:40.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.907+0000 7f880e7fc700 1 -- 172.21.15.189:0/2281546121 learned_addr learned my addr 172.21.15.189:0/2281546121 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:40.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.908+0000 7f880effd700 1 --2- 172.21.15.189:0/2281546121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88100b8d70 0x7f88101f3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:40.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.908+0000 7f880f7fe700 1 --2- 172.21.15.189:0/2281546121 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f881015d6f0 0x7f88100b6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:40.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.908+0000 7f880e7fc700 1 -- 172.21.15.189:0/2281546121 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f881015d6f0 msgr2=0x7f88100b6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:40.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.908+0000 7f880e7fc700 1 --2- 172.21.15.189:0/2281546121 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f881015d6f0 0x7f88100b6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:40.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.908+0000 7f880e7fc700 1 -- 172.21.15.189:0/2281546121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88100b8d70 msgr2=0x7f88101f3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:40.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.908+0000 7f880e7fc700 1 --2- 172.21.15.189:0/2281546121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88100b8d70 0x7f88101f3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:40.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.908+0000 7f880e7fc700 1 -- 172.21.15.189:0/2281546121 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8800009550 con 0x7f88100ce730 2024-04-06T12:31:40.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.908+0000 7f880f7fe700 1 --2- 172.21.15.189:0/2281546121 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f881015d6f0 0x7f88100b6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:40.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.908+0000 7f880e7fc700 1 --2- 172.21.15.189:0/2281546121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88100ce730 0x7f88100cbac0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f880400c550 tx=0x7f880400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:40.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.908+0000 7f87f7fff700 1 -- 172.21.15.189:0/2281546121 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88040054e0 con 0x7f88100ce730 2024-04-06T12:31:40.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.908+0000 7f87f7fff700 1 -- 172.21.15.189:0/2281546121 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8804003680 con 0x7f88100ce730 2024-04-06T12:31:40.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.908+0000 7f8815872700 1 -- 172.21.15.189:0/2281546121 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88101f2ec0 con 0x7f88100ce730 2024-04-06T12:31:40.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.908+0000 7f87f7fff700 1 -- 172.21.15.189:0/2281546121 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8804003a30 con 0x7f88100ce730 2024-04-06T12:31:40.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.908+0000 7f8815872700 1 -- 172.21.15.189:0/2281546121 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88101f3120 con 0x7f88100ce730 2024-04-06T12:31:40.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.909+0000 7f8815872700 1 -- 172.21.15.189:0/2281546121 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88100b1ee0 con 0x7f88100ce730 2024-04-06T12:31:40.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.909+0000 7f87f7fff700 1 -- 172.21.15.189:0/2281546121 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 81) v1 ==== 89321+0+0 (secure 0 0 0) 0x7f8804003b90 con 0x7f88100ce730 2024-04-06T12:31:40.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.910+0000 7f87f7fff700 1 --2- 172.21.15.189:0/2281546121 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f87fc06ae90 0x7f87fc06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:40.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.910+0000 7f880effd700 1 --2- 172.21.15.189:0/2281546121 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f87fc06ae90 0x7f87fc06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:40.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.910+0000 7f87f7fff700 1 -- 172.21.15.189:0/2281546121 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f8804088e20 con 0x7f88100ce730 2024-04-06T12:31:40.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.910+0000 7f880effd700 1 --2- 172.21.15.189:0/2281546121 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f87fc06ae90 0x7f87fc06d350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8800009130 tx=0x7f8800008030 comp rx=0 tx=0).ready entity=mgr.8775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:40.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:40.915+0000 7f87f7fff700 1 -- 172.21.15.189:0/2281546121 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8804053ae0 con 0x7f88100ce730 2024-04-06T12:31:41.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.133+0000 7f8815872700 1 -- 172.21.15.189:0/2281546121 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "M", "target": ["mon-mgr", ""]}) v1 -- 0x7f88100d0750 con 0x7f87fc06ae90 2024-04-06T12:31:41.132 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:41.134+0000 7fb6d2547700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-04-06T12:31:41.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.134+0000 7f87f7fff700 1 -- 172.21.15.189:0/2281546121 <== mgr.8775 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8800003850 con 0x7f87fc06ae90 2024-04-06T12:31:41.134 INFO:teuthology.orchestra.run.smithi189.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-04-06T12:31:41.137 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-06T12:31:41.138 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec -1m 2024-04-06T12:31:41.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.265+0000 7fa1fc2f9700 1 Processor -- start 2024-04-06T12:31:41.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.265+0000 7fa1fc2f9700 1 -- start start 2024-04-06T12:31:41.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.266+0000 7fa1fc2f9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa1f415a550 0x7fa1f40cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:41.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.266+0000 7fa1fc2f9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa1f40cf900 0x7fa1f40b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:41.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.266+0000 7fa1fc2f9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1f40b4ba0 0x7fa1f406eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:41.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.266+0000 7fa1fc2f9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1f40cdbe0 con 0x7fa1f40cf900 2024-04-06T12:31:41.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.266+0000 7fa1fc2f9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1f40b4ff0 con 0x7fa1f415a550 2024-04-06T12:31:41.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.266+0000 7fa1fc2f9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f415a940 con 0x7fa1f40b4ba0 2024-04-06T12:31:41.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.266+0000 7fa1f9894700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa1f40cf900 0x7fa1f40b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:41.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.266+0000 7fa1f9894700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa1f40cf900 0x7fa1f40b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40538/0 (socket says 172.21.15.189:40538) 2024-04-06T12:31:41.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.266+0000 7fa1f9894700 1 -- 172.21.15.189:0/55981251 learned_addr learned my addr 172.21.15.189:0/55981251 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:41.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.266+0000 7fa1fa095700 1 --2- 172.21.15.189:0/55981251 >> v2:172.21.15.192:3300/0 conn(0x7fa1f415a550 0x7fa1f40cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:41.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.266+0000 7fa1fa896700 1 --2- 172.21.15.189:0/55981251 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1f40b4ba0 0x7fa1f406eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:41.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.266+0000 7fa1f9894700 1 -- 172.21.15.189:0/55981251 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1f40b4ba0 msgr2=0x7fa1f406eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:41.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.266+0000 7fa1f9894700 1 --2- 172.21.15.189:0/55981251 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1f40b4ba0 0x7fa1f406eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:41.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.266+0000 7fa1f9894700 1 -- 172.21.15.189:0/55981251 >> v2:172.21.15.192:3300/0 conn(0x7fa1f415a550 msgr2=0x7fa1f40cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:41.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.266+0000 7fa1f9894700 1 --2- 172.21.15.189:0/55981251 >> v2:172.21.15.192:3300/0 conn(0x7fa1f415a550 0x7fa1f40cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:41.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.266+0000 7fa1f9894700 1 -- 172.21.15.189:0/55981251 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f415ac00 con 0x7fa1f40cf900 2024-04-06T12:31:41.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.267+0000 7fa1f9894700 1 --2- 172.21.15.189:0/55981251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1f40cf900 0x7fa1f40b4660 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4073470 tx=0x7fa1f0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef826071730dbd1f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:41.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.267+0000 7fa1f9093700 1 -- 172.21.15.189:0/55981251 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f000b870 con 0x7fa1f40cf900 2024-04-06T12:31:41.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.267+0000 7fa1f9093700 1 -- 172.21.15.189:0/55981251 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f000b9d0 con 0x7fa1f40cf900 2024-04-06T12:31:41.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.267+0000 7fa1f9093700 1 -- 172.21.15.189:0/55981251 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f000bd60 con 0x7fa1f40cf900 2024-04-06T12:31:41.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.267+0000 7fa1fc2f9700 1 -- 172.21.15.189:0/55981251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1f40cf900 msgr2=0x7fa1f40b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:41.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.267+0000 7fa1fc2f9700 1 --2- 172.21.15.189:0/55981251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1f40cf900 0x7fa1f40b4660 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4073470 tx=0x7fa1f0009bd0 comp rx=0 tx=0).stop 2024-04-06T12:31:41.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.267+0000 7fa1fc2f9700 1 -- 172.21.15.189:0/55981251 shutdown_connections 2024-04-06T12:31:41.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.267+0000 7fa1fc2f9700 1 --2- 172.21.15.189:0/55981251 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1f40b4ba0 0x7fa1f406eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:41.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.267+0000 7fa1fc2f9700 1 --2- 172.21.15.189:0/55981251 >> v2:172.21.15.192:3300/0 conn(0x7fa1f415a550 0x7fa1f40cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:41.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.267+0000 7fa1fc2f9700 1 --2- 172.21.15.189:0/55981251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1f40cf900 0x7fa1f40b4660 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:41.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.267+0000 7fa1fc2f9700 1 -- 172.21.15.189:0/55981251 >> 172.21.15.189:0/55981251 conn(0x7fa1f40d00b0 msgr2=0x7fa1f40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:41.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.267+0000 7fa1fc2f9700 1 -- 172.21.15.189:0/55981251 shutdown_connections 2024-04-06T12:31:41.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.267+0000 7fa1fc2f9700 1 -- 172.21.15.189:0/55981251 wait complete. 2024-04-06T12:31:41.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.268+0000 7fa1fc2f9700 1 Processor -- start 2024-04-06T12:31:41.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.269+0000 7fa1fc2f9700 1 -- start start 2024-04-06T12:31:41.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.269+0000 7fa1fc2f9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1f40b4ba0 0x7fa1f40aee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:41.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.269+0000 7fa1fc2f9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1f40cf900 0x7fa1f4152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:41.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.269+0000 7fa1fc2f9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1f40ac8c0 0x7fa1f406d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:41.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.269+0000 7fa1fc2f9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f4156710 con 0x7fa1f40cf900 2024-04-06T12:31:41.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.269+0000 7fa1fc2f9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f40ace60 con 0x7fa1f40b4ba0 2024-04-06T12:31:41.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.269+0000 7fa1fc2f9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f40ad140 con 0x7fa1f40ac8c0 2024-04-06T12:31:41.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.269+0000 7fa1f9894700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1f40cf900 0x7fa1f4152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:41.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.269+0000 7fa1f9894700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1f40cf900 0x7fa1f4152010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40544/0 (socket says 172.21.15.189:40544) 2024-04-06T12:31:41.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.269+0000 7fa1f9894700 1 -- 172.21.15.189:0/1332001472 learned_addr learned my addr 172.21.15.189:0/1332001472 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:41.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.269+0000 7fa1fa896700 1 --2- 172.21.15.189:0/1332001472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1f40ac8c0 0x7fa1f406d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:41.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.270+0000 7fa1fa095700 1 --2- 172.21.15.189:0/1332001472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1f40b4ba0 0x7fa1f40aee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:41.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.270+0000 7fa1f9894700 1 -- 172.21.15.189:0/1332001472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1f40ac8c0 msgr2=0x7fa1f406d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:41.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.270+0000 7fa1f9894700 1 --2- 172.21.15.189:0/1332001472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1f40ac8c0 0x7fa1f406d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:41.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.270+0000 7fa1f9894700 1 -- 172.21.15.189:0/1332001472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1f40b4ba0 msgr2=0x7fa1f40aee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:41.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.270+0000 7fa1f9894700 1 --2- 172.21.15.189:0/1332001472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1f40b4ba0 0x7fa1f40aee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:41.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.270+0000 7fa1f9894700 1 -- 172.21.15.189:0/1332001472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f0009550 con 0x7fa1f40cf900 2024-04-06T12:31:41.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.270+0000 7fa1f9894700 1 --2- 172.21.15.189:0/1332001472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1f40cf900 0x7fa1f4152010 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fa1f406c820 tx=0x7fa1f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:41.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.270+0000 7fa1eaffd700 1 -- 172.21.15.189:0/1332001472 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0004310 con 0x7fa1f40cf900 2024-04-06T12:31:41.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.270+0000 7fa1eaffd700 1 -- 172.21.15.189:0/1332001472 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f001f070 con 0x7fa1f40cf900 2024-04-06T12:31:41.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.270+0000 7fa1fc2f9700 1 -- 172.21.15.189:0/1332001472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f40ad420 con 0x7fa1f40cf900 2024-04-06T12:31:41.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.270+0000 7fa1eaffd700 1 -- 172.21.15.189:0/1332001472 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f00036a0 con 0x7fa1f40cf900 2024-04-06T12:31:41.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.270+0000 7fa1fc2f9700 1 -- 172.21.15.189:0/1332001472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f4152550 con 0x7fa1f40cf900 2024-04-06T12:31:41.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.271+0000 7fa1fc2f9700 1 -- 172.21.15.189:0/1332001472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f40ad940 con 0x7fa1f40cf900 2024-04-06T12:31:41.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.271+0000 7fa1eaffd700 1 -- 172.21.15.189:0/1332001472 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 81) v1 ==== 89321+0+0 (secure 0 0 0) 0x7fa1f0021030 con 0x7fa1f40cf900 2024-04-06T12:31:41.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.272+0000 7fa1eaffd700 1 --2- 172.21.15.189:0/1332001472 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa1d806ae90 0x7fa1d806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:41.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.272+0000 7fa1eaffd700 1 -- 172.21.15.189:0/1332001472 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fa1f008ad80 con 0x7fa1f40cf900 2024-04-06T12:31:41.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.272+0000 7fa1fa095700 1 --2- 172.21.15.189:0/1332001472 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa1d806ae90 0x7fa1d806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:41.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.272+0000 7fa1fa095700 1 --2- 172.21.15.189:0/1332001472 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa1d806ae90 0x7fa1d806d350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa1e400a990 tx=0x7fa1e4008040 comp rx=0 tx=0).ready entity=mgr.8775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:41.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.276+0000 7fa1eaffd700 1 -- 172.21.15.189:0/1332001472 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa1f0055bf0 con 0x7fa1f40cf900 2024-04-06T12:31:41.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.494+0000 7fa1fc2f9700 1 -- 172.21.15.189:0/1332001472 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "-1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1f415b4a0 con 0x7fa1d806ae90 2024-04-06T12:31:41.493 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:41.495+0000 7fb6d2547700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2024-04-06T12:31:41.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.495+0000 7fa1eaffd700 1 -- 172.21.15.189:0/1332001472 <== mgr.8775 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc.) v1 ==== 96+0+0 (secure 0 0 0) 0x7fa1e400a660 con 0x7fa1d806ae90 2024-04-06T12:31:41.495 INFO:teuthology.orchestra.run.smithi189.stderr:Error ENOENT: invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2024-04-06T12:31:41.498 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-06T12:31:41.498 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec '' 2024-04-06T12:31:41.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.633+0000 7f6cb341b700 1 Processor -- start 2024-04-06T12:31:41.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.633+0000 7f6cb341b700 1 -- start start 2024-04-06T12:31:41.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.634+0000 7f6cb341b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6cac15d7b0 0x7f6cac0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:41.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.634+0000 7f6cb341b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6cac15e500 0x7f6cac06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:41.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.634+0000 7f6cb341b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cac0b3990 con 0x7f6cac06a730 2024-04-06T12:31:41.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.634+0000 7f6cb341b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cac15e950 con 0x7f6cac15e500 2024-04-06T12:31:41.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.634+0000 7f6cb341b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cac06b0f0 con 0x7f6cac15d7b0 2024-04-06T12:31:41.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.634+0000 7f6cb19b8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6cac15e500 0x7f6cac06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:41.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.634+0000 7f6cb19b8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6cac15e500 0x7f6cac06abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40550/0 (socket says 172.21.15.189:40550) 2024-04-06T12:31:41.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.634+0000 7f6cb19b8700 1 -- 172.21.15.189:0/2148302990 learned_addr learned my addr 172.21.15.189:0/2148302990 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:41.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.634+0000 7f6cb19b8700 1 -- 172.21.15.189:0/2148302990 >> v2:172.21.15.192:3300/0 conn(0x7f6cac15d7b0 msgr2=0x7f6cac0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:41.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.634+0000 7f6cb11b7700 1 --2- 172.21.15.189:0/2148302990 >> v2:172.21.15.192:3300/0 conn(0x7f6cac15d7b0 0x7f6cac0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:41.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.634+0000 7f6cb19b8700 1 --2- 172.21.15.189:0/2148302990 >> v2:172.21.15.192:3300/0 conn(0x7f6cac15d7b0 0x7f6cac0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:41.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.634+0000 7f6cb19b8700 1 -- 172.21.15.189:0/2148302990 >> v1:172.21.15.192:6789/0 conn(0x7f6cac06a730 legacy=0x7f6cac15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:41.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.634+0000 7f6cb19b8700 1 -- 172.21.15.189:0/2148302990 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cac0b3060 con 0x7f6cac15e500 2024-04-06T12:31:41.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.634+0000 7f6cb11b7700 1 --2- 172.21.15.189:0/2148302990 >> v2:172.21.15.192:3300/0 conn(0x7f6cac15d7b0 0x7f6cac0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:41.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.634+0000 7f6cb19b8700 1 --2- 172.21.15.189:0/2148302990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cac15e500 0x7f6cac06abb0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f6cac0b6200 tx=0x7f6ca400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e52c95800f898f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:41.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.635+0000 7f6ca3fff700 1 -- 172.21.15.189:0/2148302990 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca400e070 con 0x7f6cac15e500 2024-04-06T12:31:41.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.635+0000 7f6ca3fff700 1 -- 172.21.15.189:0/2148302990 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca4003a20 con 0x7f6cac15e500 2024-04-06T12:31:41.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.635+0000 7f6ca3fff700 1 -- 172.21.15.189:0/2148302990 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4003dd0 con 0x7f6cac15e500 2024-04-06T12:31:41.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.635+0000 7f6cb341b700 1 -- 172.21.15.189:0/2148302990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cac15e500 msgr2=0x7f6cac06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:41.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.635+0000 7f6cb341b700 1 --2- 172.21.15.189:0/2148302990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cac15e500 0x7f6cac06abb0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f6cac0b6200 tx=0x7f6ca400b330 comp rx=0 tx=0).stop 2024-04-06T12:31:41.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.635+0000 7f6cb341b700 1 -- 172.21.15.189:0/2148302990 shutdown_connections 2024-04-06T12:31:41.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.635+0000 7f6cb341b700 1 --2- 172.21.15.189:0/2148302990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cac15e500 0x7f6cac06abb0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:41.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.635+0000 7f6cb341b700 1 --2- 172.21.15.189:0/2148302990 >> v2:172.21.15.192:3300/0 conn(0x7f6cac15d7b0 0x7f6cac0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:41.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.635+0000 7f6cb341b700 1 -- 172.21.15.189:0/2148302990 >> 172.21.15.189:0/2148302990 conn(0x7f6cac0d00b0 msgr2=0x7f6cac0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:41.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.635+0000 7f6cb341b700 1 -- 172.21.15.189:0/2148302990 shutdown_connections 2024-04-06T12:31:41.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.635+0000 7f6cb341b700 1 -- 172.21.15.189:0/2148302990 wait complete. 2024-04-06T12:31:41.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.637+0000 7f6cb341b700 1 Processor -- start 2024-04-06T12:31:41.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.637+0000 7f6cb341b700 1 -- start start 2024-04-06T12:31:41.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6cb341b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cac06a730 0x7f6cac150090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:41.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6cb341b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6cac15d7b0 0x7f6cac15ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:41.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6cb341b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6cac15e500 0x7f6cac0ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:41.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6cb341b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac15c4f0 con 0x7f6cac06a730 2024-04-06T12:31:41.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6cb341b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac15b8e0 con 0x7f6cac15d7b0 2024-04-06T12:31:41.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6cb341b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac15bbc0 con 0x7f6cac15e500 2024-04-06T12:31:41.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6cb11b7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cac06a730 0x7f6cac150090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:41.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6cb11b7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cac06a730 0x7f6cac150090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40562/0 (socket says 172.21.15.189:40562) 2024-04-06T12:31:41.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6cb11b7700 1 -- 172.21.15.189:0/1331106370 learned_addr learned my addr 172.21.15.189:0/1331106370 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:41.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6cb19b8700 1 --2- 172.21.15.189:0/1331106370 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6cac15e500 0x7f6cac0ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:41.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6cb09b6700 1 --2- 172.21.15.189:0/1331106370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6cac15d7b0 0x7f6cac15ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:41.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6cb11b7700 1 -- 172.21.15.189:0/1331106370 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6cac15e500 msgr2=0x7f6cac0ccf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:41.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6cb11b7700 1 --2- 172.21.15.189:0/1331106370 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6cac15e500 0x7f6cac0ccf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:41.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6cb11b7700 1 -- 172.21.15.189:0/1331106370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6cac15d7b0 msgr2=0x7f6cac15ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:41.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6cb11b7700 1 --2- 172.21.15.189:0/1331106370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6cac15d7b0 0x7f6cac15ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:41.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6cb11b7700 1 -- 172.21.15.189:0/1331106370 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca4009b30 con 0x7f6cac06a730 2024-04-06T12:31:41.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6cb11b7700 1 --2- 172.21.15.189:0/1331106370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cac06a730 0x7f6cac150090 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f6ca800c750 tx=0x7f6ca800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:41.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6ca1ffb700 1 -- 172.21.15.189:0/1331106370 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca8011800 con 0x7f6cac06a730 2024-04-06T12:31:41.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6ca1ffb700 1 -- 172.21.15.189:0/1331106370 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca8011960 con 0x7f6cac06a730 2024-04-06T12:31:41.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6ca1ffb700 1 -- 172.21.15.189:0/1331106370 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca8011cd0 con 0x7f6cac06a730 2024-04-06T12:31:41.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.638+0000 7f6cb341b700 1 -- 172.21.15.189:0/1331106370 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cac15bea0 con 0x7f6cac06a730 2024-04-06T12:31:41.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.639+0000 7f6cb341b700 1 -- 172.21.15.189:0/1331106370 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cac1505d0 con 0x7f6cac06a730 2024-04-06T12:31:41.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.640+0000 7f6cb341b700 1 -- 172.21.15.189:0/1331106370 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cac067bb0 con 0x7f6cac06a730 2024-04-06T12:31:41.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.640+0000 7f6ca1ffb700 1 -- 172.21.15.189:0/1331106370 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 81) v1 ==== 89321+0+0 (secure 0 0 0) 0x7f6ca8021350 con 0x7f6cac06a730 2024-04-06T12:31:41.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.640+0000 7f6ca1ffb700 1 --2- 172.21.15.189:0/1331106370 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c9006ae90 0x7f6c9006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:41.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.641+0000 7f6ca1ffb700 1 -- 172.21.15.189:0/1331106370 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f6ca8089940 con 0x7f6cac06a730 2024-04-06T12:31:41.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.641+0000 7f6cb09b6700 1 --2- 172.21.15.189:0/1331106370 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c9006ae90 0x7f6c9006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:41.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.641+0000 7f6cb09b6700 1 --2- 172.21.15.189:0/1331106370 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6c9006ae90 0x7f6c9006d350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6cac15adb0 tx=0x7f6c9c007ec0 comp rx=0 tx=0).ready entity=mgr.8775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:41.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.645+0000 7f6ca1ffb700 1 -- 172.21.15.189:0/1331106370 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6ca8054450 con 0x7f6cac06a730 2024-04-06T12:31:41.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.867+0000 7f6cb341b700 1 -- 172.21.15.189:0/1331106370 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f6cac0b3620 con 0x7f6c9006ae90 2024-04-06T12:31:41.866 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:41.868+0000 7fb6d2547700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-04-06T12:31:41.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:41.868+0000 7f6ca1ffb700 1 -- 172.21.15.189:0/1331106370 <== mgr.8775 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6c9c003a90 con 0x7f6c9006ae90 2024-04-06T12:31:41.868 INFO:teuthology.orchestra.run.smithi189.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-04-06T12:31:41.871 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-06T12:31:41.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:41.872 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/minutes) 2024-04-06T12:31:41.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:41.890 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/hourly) 2024-04-06T12:31:41.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:41.948 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/daily) 2024-04-06T12:31:42.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:42.006 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/weekly) 2024-04-06T12:31:42.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:42.064 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/monthly) 2024-04-06T12:31:42.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:42.122 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/yearly) 2024-04-06T12:31:42.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:42.181 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/bad_period_spec) 2024-04-06T12:31:42.240 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-06T12:31:42.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.405+0000 7f01da9d0700 1 Processor -- start 2024-04-06T12:31:42.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.406+0000 7f01da9d0700 1 -- start start 2024-04-06T12:31:42.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.406+0000 7f01da9d0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f01d415d7b0 0x7f01d40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:42.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.406+0000 7f01da9d0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01d406a730 0x7f01d406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:42.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.406+0000 7f01da9d0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d40b3990 con 0x7f01d415ddf0 2024-04-06T12:31:42.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.406+0000 7f01da9d0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d406b0f0 con 0x7f01d415d7b0 2024-04-06T12:31:42.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.406+0000 7f01da9d0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d40b3060 con 0x7f01d406a730 2024-04-06T12:31:42.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.406+0000 7f01d3fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f01d415d7b0 0x7f01d40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:42.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.407+0000 7f01d3fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f01d415d7b0 0x7f01d40b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40566/0 (socket says 172.21.15.189:40566) 2024-04-06T12:31:42.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.407+0000 7f01d3fff700 1 -- 172.21.15.189:0/2490715355 learned_addr learned my addr 172.21.15.189:0/2490715355 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:42.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.407+0000 7f01d37fe700 1 --2- 172.21.15.189:0/2490715355 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01d406a730 0x7f01d406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:42.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.407+0000 7f01d3fff700 1 -- 172.21.15.189:0/2490715355 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01d406a730 msgr2=0x7f01d406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:42.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.407+0000 7f01d3fff700 1 --2- 172.21.15.189:0/2490715355 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01d406a730 0x7f01d406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:42.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.407+0000 7f01d3fff700 1 -- 172.21.15.189:0/2490715355 >> v1:172.21.15.189:6789/0 conn(0x7f01d415ddf0 legacy=0x7f01d415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:42.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.407+0000 7f01d3fff700 1 -- 172.21.15.189:0/2490715355 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d40b3340 con 0x7f01d415d7b0 2024-04-06T12:31:42.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.407+0000 7f01d3fff700 1 --2- 172.21.15.189:0/2490715355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d415d7b0 0x7f01d40b3e20 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f01c80098f0 tx=0x7f01c8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63db628e20dad5b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:42.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.407+0000 7f01d2ffd700 1 -- 172.21.15.189:0/2490715355 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c800b8a0 con 0x7f01d415d7b0 2024-04-06T12:31:42.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.407+0000 7f01d2ffd700 1 -- 172.21.15.189:0/2490715355 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c800ba00 con 0x7f01d415d7b0 2024-04-06T12:31:42.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.407+0000 7f01d2ffd700 1 -- 172.21.15.189:0/2490715355 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c800bd90 con 0x7f01d415d7b0 2024-04-06T12:31:42.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.408+0000 7f01da9d0700 1 -- 172.21.15.189:0/2490715355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d415d7b0 msgr2=0x7f01d40b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:42.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.408+0000 7f01da9d0700 1 --2- 172.21.15.189:0/2490715355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d415d7b0 0x7f01d40b3e20 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f01c80098f0 tx=0x7f01c8009c00 comp rx=0 tx=0).stop 2024-04-06T12:31:42.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.408+0000 7f01da9d0700 1 -- 172.21.15.189:0/2490715355 shutdown_connections 2024-04-06T12:31:42.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.408+0000 7f01da9d0700 1 --2- 172.21.15.189:0/2490715355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d415d7b0 0x7f01d40b3e20 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:42.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.408+0000 7f01da9d0700 1 --2- 172.21.15.189:0/2490715355 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01d406a730 0x7f01d406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:42.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.408+0000 7f01da9d0700 1 -- 172.21.15.189:0/2490715355 >> 172.21.15.189:0/2490715355 conn(0x7f01d40d00b0 msgr2=0x7f01d40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:42.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.408+0000 7f01da9d0700 1 -- 172.21.15.189:0/2490715355 shutdown_connections 2024-04-06T12:31:42.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.408+0000 7f01da9d0700 1 -- 172.21.15.189:0/2490715355 wait complete. 2024-04-06T12:31:42.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.409+0000 7f01da9d0700 1 Processor -- start 2024-04-06T12:31:42.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.409+0000 7f01da9d0700 1 -- start start 2024-04-06T12:31:42.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.410+0000 7f01da9d0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01d406a730 0x7f01d41f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:42.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.410+0000 7f01da9d0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01d415d7b0 0x7f01d41f6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:42.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.410+0000 7f01da9d0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d415ddf0 0x7f01d41f6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:42.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.410+0000 7f01da9d0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d4148c90 con 0x7f01d415ddf0 2024-04-06T12:31:42.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.410+0000 7f01da9d0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d40b0d70 con 0x7f01d406a730 2024-04-06T12:31:42.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.410+0000 7f01da9d0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d40b1050 con 0x7f01d415d7b0 2024-04-06T12:31:42.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.410+0000 7f01d8f6d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d415ddf0 0x7f01d41f6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:42.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.410+0000 7f01d8f6d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d415ddf0 0x7f01d41f6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40580/0 (socket says 172.21.15.189:40580) 2024-04-06T12:31:42.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.410+0000 7f01d8f6d700 1 -- 172.21.15.189:0/3579462075 learned_addr learned my addr 172.21.15.189:0/3579462075 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:42.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.410+0000 7f01d3fff700 1 --2- 172.21.15.189:0/3579462075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01d406a730 0x7f01d41f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:42.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.410+0000 7f01d37fe700 1 --2- 172.21.15.189:0/3579462075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01d415d7b0 0x7f01d41f6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:42.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.410+0000 7f01d8f6d700 1 -- 172.21.15.189:0/3579462075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01d415d7b0 msgr2=0x7f01d41f6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:42.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.410+0000 7f01d8f6d700 1 --2- 172.21.15.189:0/3579462075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01d415d7b0 0x7f01d41f6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:42.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.410+0000 7f01d8f6d700 1 -- 172.21.15.189:0/3579462075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01d406a730 msgr2=0x7f01d41f28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:42.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.410+0000 7f01d8f6d700 1 --2- 172.21.15.189:0/3579462075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01d406a730 0x7f01d41f28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:42.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.410+0000 7f01d8f6d700 1 -- 172.21.15.189:0/3579462075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c8009550 con 0x7f01d415ddf0 2024-04-06T12:31:42.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.410+0000 7f01d8f6d700 1 --2- 172.21.15.189:0/3579462075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d415ddf0 0x7f01d41f6990 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00cb50 tx=0x7f01cc00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:42.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.411+0000 7f01d0ff9700 1 -- 172.21.15.189:0/3579462075 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc003c00 con 0x7f01d415ddf0 2024-04-06T12:31:42.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.411+0000 7f01d0ff9700 1 -- 172.21.15.189:0/3579462075 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01cc003d60 con 0x7f01d415ddf0 2024-04-06T12:31:42.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.411+0000 7f01da9d0700 1 -- 172.21.15.189:0/3579462075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d40b1330 con 0x7f01d415ddf0 2024-04-06T12:31:42.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.411+0000 7f01d0ff9700 1 -- 172.21.15.189:0/3579462075 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc00aa20 con 0x7f01d415ddf0 2024-04-06T12:31:42.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.411+0000 7f01da9d0700 1 -- 172.21.15.189:0/3579462075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d4148460 con 0x7f01d415ddf0 2024-04-06T12:31:42.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.412+0000 7f01da9d0700 1 -- 172.21.15.189:0/3579462075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d40b1850 con 0x7f01d415ddf0 2024-04-06T12:31:42.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.412+0000 7f01d0ff9700 1 -- 172.21.15.189:0/3579462075 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 81) v1 ==== 89321+0+0 (secure 0 0 0) 0x7f01cc00ab80 con 0x7f01d415ddf0 2024-04-06T12:31:42.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.413+0000 7f01d0ff9700 1 --2- 172.21.15.189:0/3579462075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f01c006ae90 0x7f01c006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:42.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.413+0000 7f01d0ff9700 1 -- 172.21.15.189:0/3579462075 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f01cc0898d0 con 0x7f01d415ddf0 2024-04-06T12:31:42.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.413+0000 7f01d3fff700 1 --2- 172.21.15.189:0/3579462075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f01c006ae90 0x7f01c006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:42.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.413+0000 7f01d3fff700 1 --2- 172.21.15.189:0/3579462075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f01c006ae90 0x7f01c006d350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f01c8009c40 tx=0x7f01c8008030 comp rx=0 tx=0).ready entity=mgr.8775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:42.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.417+0000 7f01d0ff9700 1 -- 172.21.15.189:0/3579462075 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f01d40b1850 con 0x7f01d415ddf0 2024-04-06T12:31:42.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:42.668+0000 7f01da9d0700 1 -- 172.21.15.189:0/3579462075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f01d40b1850 con 0x7f01d415ddf0 2024-04-06T12:31:43.216 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:31:43.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:31:43.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.601+0000 7f01d0ff9700 1 -- 172.21.15.189:0/3579462075 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 82) v1 ==== 89304+0+0 (secure 0 0 0) 0x7f01cc0549f0 con 0x7f01d415ddf0 2024-04-06T12:31:43.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.603+0000 7f01d0ff9700 1 -- 172.21.15.189:0/3579462075 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v82) v1 ==== 93+0+0 (secure 0 0 0) 0x7f01cc072240 con 0x7f01d415ddf0 2024-04-06T12:31:43.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.606+0000 7f01da9d0700 1 -- 172.21.15.189:0/3579462075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f01c006ae90 msgr2=0x7f01c006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:43.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.606+0000 7f01da9d0700 1 --2- 172.21.15.189:0/3579462075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f01c006ae90 0x7f01c006d350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f01c8009c40 tx=0x7f01c8008030 comp rx=0 tx=0).stop 2024-04-06T12:31:43.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.606+0000 7f01da9d0700 1 -- 172.21.15.189:0/3579462075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d415ddf0 msgr2=0x7f01d41f6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:43.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.606+0000 7f01da9d0700 1 --2- 172.21.15.189:0/3579462075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d415ddf0 0x7f01d41f6990 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00cb50 tx=0x7f01cc00ce60 comp rx=0 tx=0).stop 2024-04-06T12:31:43.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.607+0000 7f01da9d0700 1 -- 172.21.15.189:0/3579462075 shutdown_connections 2024-04-06T12:31:43.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.607+0000 7f01da9d0700 1 --2- 172.21.15.189:0/3579462075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d415ddf0 0x7f01d41f6990 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:43.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.607+0000 7f01da9d0700 1 --2- 172.21.15.189:0/3579462075 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f01c006ae90 0x7f01c006d350 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:43.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.607+0000 7f01da9d0700 1 --2- 172.21.15.189:0/3579462075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01d406a730 0x7f01d41f28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:43.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.607+0000 7f01da9d0700 1 --2- 172.21.15.189:0/3579462075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01d415d7b0 0x7f01d41f6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:43.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.607+0000 7f01da9d0700 1 -- 172.21.15.189:0/3579462075 >> 172.21.15.189:0/3579462075 conn(0x7f01d40d00b0 msgr2=0x7f01d40adbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:43.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.608+0000 7f01da9d0700 1 -- 172.21.15.189:0/3579462075 shutdown_connections 2024-04-06T12:31:43.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.608+0000 7f01da9d0700 1 -- 172.21.15.189:0/3579462075 wait complete. 2024-04-06T12:31:43.621 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T12:31:43.621 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:31:43.669 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:31:43.714 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T12:31:43.714 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:31:43.714 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:31:43.732 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:31:43.732 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:31:43.790 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:31:43.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.956+0000 7f5c6c7aa700 1 Processor -- start 2024-04-06T12:31:43.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.956+0000 7f5c6c7aa700 1 -- start start 2024-04-06T12:31:43.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.957+0000 7f5c6c7aa700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5c640d11b0 0x7f5c640d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:43.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.957+0000 7f5c6c7aa700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5c640cb870 0x7f5c6415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:43.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.957+0000 7f5c6c7aa700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c640d1ad0 con 0x7f5c640cfa00 2024-04-06T12:31:43.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.957+0000 7f5c6c7aa700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c640ccdc0 con 0x7f5c640cb870 2024-04-06T12:31:43.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.957+0000 7f5c6c7aa700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c640d0b50 con 0x7f5c640d11b0 2024-04-06T12:31:43.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.958+0000 7f5c69d45700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5c640cb870 0x7f5c6415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:43.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.958+0000 7f5c69d45700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5c640cb870 0x7f5c6415c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40602/0 (socket says 172.21.15.189:40602) 2024-04-06T12:31:43.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.958+0000 7f5c69d45700 1 -- 172.21.15.189:0/3649562249 learned_addr learned my addr 172.21.15.189:0/3649562249 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:43.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.958+0000 7f5c6a546700 1 --2- 172.21.15.189:0/3649562249 >> v2:172.21.15.192:3300/0 conn(0x7f5c640d11b0 0x7f5c640d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:43.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.958+0000 7f5c69d45700 1 -- 172.21.15.189:0/3649562249 >> v2:172.21.15.192:3300/0 conn(0x7f5c640d11b0 msgr2=0x7f5c640d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:43.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.958+0000 7f5c69d45700 1 --2- 172.21.15.189:0/3649562249 >> v2:172.21.15.192:3300/0 conn(0x7f5c640d11b0 0x7f5c640d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:43.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.958+0000 7f5c69d45700 1 -- 172.21.15.189:0/3649562249 >> v1:172.21.15.192:6789/0 conn(0x7f5c640cfa00 legacy=0x7f5c640cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:43.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.958+0000 7f5c69d45700 1 -- 172.21.15.189:0/3649562249 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c640cbd80 con 0x7f5c640cb870 2024-04-06T12:31:43.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.958+0000 7f5c69544700 1 -- 172.21.15.189:0/3649562249 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186017457 0 0) 0x7f5c640d1ad0 con 0x7f5c640cfa00 2024-04-06T12:31:43.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.958+0000 7f5c69d45700 1 --2- 172.21.15.189:0/3649562249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c640cb870 0x7f5c6415c280 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f5c640ced00 tx=0x7f5c54009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b04237dfd5be88bd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:43.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.958+0000 7f5c69544700 1 -- 172.21.15.189:0/3649562249 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c54011830 con 0x7f5c640cb870 2024-04-06T12:31:43.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.958+0000 7f5c69544700 1 -- 172.21.15.189:0/3649562249 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c54011990 con 0x7f5c640cb870 2024-04-06T12:31:43.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.958+0000 7f5c69544700 1 -- 172.21.15.189:0/3649562249 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c54011d20 con 0x7f5c640cb870 2024-04-06T12:31:43.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.959+0000 7f5c6c7aa700 1 -- 172.21.15.189:0/3649562249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c640cb870 msgr2=0x7f5c6415c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:43.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.959+0000 7f5c6c7aa700 1 --2- 172.21.15.189:0/3649562249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c640cb870 0x7f5c6415c280 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f5c640ced00 tx=0x7f5c54009bd0 comp rx=0 tx=0).stop 2024-04-06T12:31:43.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.959+0000 7f5c6c7aa700 1 -- 172.21.15.189:0/3649562249 shutdown_connections 2024-04-06T12:31:43.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.959+0000 7f5c6c7aa700 1 --2- 172.21.15.189:0/3649562249 >> v2:172.21.15.192:3300/0 conn(0x7f5c640d11b0 0x7f5c640d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:43.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.959+0000 7f5c6c7aa700 1 --2- 172.21.15.189:0/3649562249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c640cb870 0x7f5c6415c280 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:43.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.959+0000 7f5c6c7aa700 1 -- 172.21.15.189:0/3649562249 >> 172.21.15.189:0/3649562249 conn(0x7f5c640b2db0 msgr2=0x7f5c640b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:43.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.959+0000 7f5c6c7aa700 1 -- 172.21.15.189:0/3649562249 shutdown_connections 2024-04-06T12:31:43.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.959+0000 7f5c6c7aa700 1 -- 172.21.15.189:0/3649562249 wait complete. 2024-04-06T12:31:43.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.961+0000 7f5c6c7aa700 1 Processor -- start 2024-04-06T12:31:43.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.961+0000 7f5c6c7aa700 1 -- start start 2024-04-06T12:31:43.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c6c7aa700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c640cb870 0x7f5c640c4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:43.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c6c7aa700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c640cfa00 0x7f5c6406c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:43.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c6c7aa700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c640d11b0 0x7f5c6415c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:43.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c6c7aa700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c640c4b70 con 0x7f5c640d11b0 2024-04-06T12:31:43.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c6c7aa700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c640c8800 con 0x7f5c640cfa00 2024-04-06T12:31:43.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c6c7aa700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c64156360 con 0x7f5c640cb870 2024-04-06T12:31:43.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c6ad47700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c640d11b0 0x7f5c6415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:43.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c69d45700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c640cfa00 0x7f5c6406c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:43.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c6a546700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c640cb870 0x7f5c640c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:43.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c6ad47700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c640d11b0 0x7f5c6415c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40618/0 (socket says 172.21.15.189:40618) 2024-04-06T12:31:43.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c6ad47700 1 -- 172.21.15.189:0/2156189115 learned_addr learned my addr 172.21.15.189:0/2156189115 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:43.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c69d45700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c640cfa00 0x7f5c6406c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:47256/0 (socket says 172.21.15.189:47256) 2024-04-06T12:31:43.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c6ad47700 1 -- 172.21.15.189:0/2156189115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c640cb870 msgr2=0x7f5c640c4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:43.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c6ad47700 1 --2- 172.21.15.189:0/2156189115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c640cb870 0x7f5c640c4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:43.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c6ad47700 1 -- 172.21.15.189:0/2156189115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c640cfa00 msgr2=0x7f5c6406c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:43.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c6ad47700 1 --2- 172.21.15.189:0/2156189115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c640cfa00 0x7f5c6406c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:43.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c6ad47700 1 -- 172.21.15.189:0/2156189115 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c54009550 con 0x7f5c640d11b0 2024-04-06T12:31:43.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c6ad47700 1 --2- 172.21.15.189:0/2156189115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c640d11b0 0x7f5c6415c280 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c00c410 tx=0x7f5c5c00c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:43.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c5affd700 1 -- 172.21.15.189:0/2156189115 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c0052d0 con 0x7f5c640d11b0 2024-04-06T12:31:43.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c5affd700 1 -- 172.21.15.189:0/2156189115 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c5c005450 con 0x7f5c640d11b0 2024-04-06T12:31:43.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c5affd700 1 -- 172.21.15.189:0/2156189115 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c0038c0 con 0x7f5c640d11b0 2024-04-06T12:31:43.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.962+0000 7f5c6c7aa700 1 -- 172.21.15.189:0/2156189115 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c64156640 con 0x7f5c640d11b0 2024-04-06T12:31:43.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.963+0000 7f5c6c7aa700 1 -- 172.21.15.189:0/2156189115 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c64156c00 con 0x7f5c640d11b0 2024-04-06T12:31:43.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.963+0000 7f5c6c7aa700 1 -- 172.21.15.189:0/2156189115 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c64074990 con 0x7f5c640d11b0 2024-04-06T12:31:43.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.964+0000 7f5c5affd700 1 -- 172.21.15.189:0/2156189115 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 82) v1 ==== 89304+0+0 (secure 0 0 0) 0x7f5c5c003c30 con 0x7f5c640d11b0 2024-04-06T12:31:43.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.964+0000 7f5c5affd700 1 --2- 172.21.15.189:0/2156189115 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c4806ae40 0x7f5c4806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:43.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.964+0000 7f5c6a546700 1 -- 172.21.15.189:0/2156189115 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c4806ae40 msgr2=0x7f5c4806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:43.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.964+0000 7f5c5affd700 1 -- 172.21.15.189:0/2156189115 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f5c5c088e90 con 0x7f5c640d11b0 2024-04-06T12:31:43.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.964+0000 7f5c6a546700 1 --2- 172.21.15.189:0/2156189115 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c4806ae40 0x7f5c4806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:43.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:43.969+0000 7f5c5affd700 1 -- 172.21.15.189:0/2156189115 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5c5c053ac0 con 0x7f5c640d11b0 2024-04-06T12:31:43.987 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:43.990+0000 7f75c0c26200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:31:44.010 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:44.012+0000 7f4b58d91200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:31:44.064 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:44.067+0000 7f75c0c26200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:31:44.091 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:44.093+0000 7f4b58d91200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:31:44.137 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:44.140+0000 7f75c0c26200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:31:44.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.165+0000 7f5c6a546700 1 -- 172.21.15.189:0/2156189115 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c4806ae40 msgr2=0x7f5c4806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:44.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.165+0000 7f5c6a546700 1 --2- 172.21.15.189:0/2156189115 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c4806ae40 0x7f5c4806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:44.166 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:44.169+0000 7f4b58d91200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:31:44.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.172+0000 7f5c6c7aa700 1 -- 172.21.15.189:0/2156189115 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c640cb320 con 0x7f5c640d11b0 2024-04-06T12:31:44.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.174+0000 7f5c5affd700 1 -- 172.21.15.189:0/2156189115 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+16862 (secure 0 0 0) 0x7f5c5c017070 con 0x7f5c640d11b0 2024-04-06T12:31:44.172 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:44.172 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":145,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:31:36.068439+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:31:19.866331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-06T12:31:21.048344+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/2323784940":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/2597819342":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/3011108552":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/814932379":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/2994336395":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/2720887145":"2024-04-07T12:31:36.068397+0000","172.21.15.189:6833/47128":"2024-04-07T12:31:36.068397+0000","172.21.15.189:6832/47128":"2024-04-07T12:31:36.068397+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:31:44.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.176+0000 7f5c6c7aa700 1 -- 172.21.15.189:0/2156189115 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c4806ae40 msgr2=0x7f5c4806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:44.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.176+0000 7f5c6c7aa700 1 --2- 172.21.15.189:0/2156189115 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c4806ae40 0x7f5c4806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.176+0000 7f5c6c7aa700 1 -- 172.21.15.189:0/2156189115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c640d11b0 msgr2=0x7f5c6415c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:44.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.176+0000 7f5c6c7aa700 1 --2- 172.21.15.189:0/2156189115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c640d11b0 0x7f5c6415c280 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c00c410 tx=0x7f5c5c00c720 comp rx=0 tx=0).stop 2024-04-06T12:31:44.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.176+0000 7f5c6c7aa700 1 -- 172.21.15.189:0/2156189115 shutdown_connections 2024-04-06T12:31:44.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.176+0000 7f5c6c7aa700 1 --2- 172.21.15.189:0/2156189115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c640d11b0 0x7f5c6415c280 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.176+0000 7f5c6c7aa700 1 --2- 172.21.15.189:0/2156189115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c640cb870 0x7f5c640c4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.176+0000 7f5c6c7aa700 1 --2- 172.21.15.189:0/2156189115 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c4806ae40 0x7f5c4806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.176+0000 7f5c6c7aa700 1 --2- 172.21.15.189:0/2156189115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c640cfa00 0x7f5c6406c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.176+0000 7f5c6c7aa700 1 -- 172.21.15.189:0/2156189115 >> 172.21.15.189:0/2156189115 conn(0x7f5c640b2db0 msgr2=0x7f5c64153540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:44.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.176+0000 7f5c6c7aa700 1 -- 172.21.15.189:0/2156189115 shutdown_connections 2024-04-06T12:31:44.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.176+0000 7f5c6c7aa700 1 -- 172.21.15.189:0/2156189115 wait complete. 2024-04-06T12:31:44.187 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:31:19.866331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:31:21.048344+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:31:44.187 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:31:44.205 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:44.208+0000 7f75c0c26200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:31:44.243 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:44.245+0000 7f4b58d91200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:31:44.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.318+0000 7ff81385f700 1 Processor -- start 2024-04-06T12:31:44.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.318+0000 7ff81385f700 1 -- start start 2024-04-06T12:31:44.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.319+0000 7ff81385f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff80c0d0a60 0x7ff80c0d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:44.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.319+0000 7ff81385f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80c0cdf80 con 0x7ff80c0d1af0 2024-04-06T12:31:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.319+0000 7ff81385f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80c0d3230 con 0x7ff80c15d7e0 2024-04-06T12:31:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.319+0000 7ff81385f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c0d1f80 con 0x7ff80c0d0a60 2024-04-06T12:31:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.319+0000 7ff811dfc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff80c0d1af0 0x7ff80c0b66b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49122/0 (socket says 172.21.15.189:49122) 2024-04-06T12:31:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.319+0000 7ff811dfc700 1 -- 172.21.15.189:0/1877075075 learned_addr learned my addr 172.21.15.189:0/1877075075 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.319+0000 7ff810dfa700 1 --2- 172.21.15.189:0/1877075075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff80c0d0a60 0x7ff80c0d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.320+0000 7ff803fff700 1 -- 172.21.15.189:0/1877075075 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216472251 0 0) 0x7ff80c0cdf80 con 0x7ff80c0d1af0 2024-04-06T12:31:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.320+0000 7ff803fff700 1 -- 172.21.15.189:0/1877075075 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7f80035b0 con 0x7ff80c0d1af0 2024-04-06T12:31:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.320+0000 7ff803fff700 1 -- 172.21.15.189:0/1877075075 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950102108 0 0) 0x7ff808001110 con 0x7ff80c15d7e0 2024-04-06T12:31:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.320+0000 7ff803fff700 1 -- 172.21.15.189:0/1877075075 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff80c0cdf80 con 0x7ff80c15d7e0 2024-04-06T12:31:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.320+0000 7ff810dfa700 1 -- 172.21.15.189:0/1877075075 >> v1:172.21.15.192:6789/0 conn(0x7ff80c15d7e0 legacy=0x7ff80c0d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.320+0000 7ff810dfa700 1 -- 172.21.15.189:0/1877075075 >> v1:172.21.15.189:6789/0 conn(0x7ff80c0d1af0 legacy=0x7ff80c0b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:44.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.320+0000 7ff810dfa700 1 -- 172.21.15.189:0/1877075075 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80c0d2da0 con 0x7ff80c0d0a60 2024-04-06T12:31:44.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.320+0000 7ff810dfa700 1 --2- 172.21.15.189:0/1877075075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff80c0d0a60 0x7ff80c0d15b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc009870 tx=0x7ff7fc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eea3752420350f53 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:44.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.320+0000 7ff803fff700 1 -- 172.21.15.189:0/1877075075 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc011830 con 0x7ff80c0d0a60 2024-04-06T12:31:44.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.320+0000 7ff803fff700 1 -- 172.21.15.189:0/1877075075 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7fc011990 con 0x7ff80c0d0a60 2024-04-06T12:31:44.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.320+0000 7ff803fff700 1 -- 172.21.15.189:0/1877075075 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc011d20 con 0x7ff80c0d0a60 2024-04-06T12:31:44.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.321+0000 7ff81385f700 1 -- 172.21.15.189:0/1877075075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff80c0d0a60 msgr2=0x7ff80c0d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:44.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.321+0000 7ff81385f700 1 --2- 172.21.15.189:0/1877075075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff80c0d0a60 0x7ff80c0d15b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc009870 tx=0x7ff7fc009b80 comp rx=0 tx=0).stop 2024-04-06T12:31:44.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.321+0000 7ff81385f700 1 -- 172.21.15.189:0/1877075075 shutdown_connections 2024-04-06T12:31:44.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.321+0000 7ff81385f700 1 --2- 172.21.15.189:0/1877075075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff80c0d0a60 0x7ff80c0d15b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.321+0000 7ff81385f700 1 -- 172.21.15.189:0/1877075075 >> 172.21.15.189:0/1877075075 conn(0x7ff80c072250 msgr2=0x7ff80c072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:44.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.321+0000 7ff81385f700 1 -- 172.21.15.189:0/1877075075 shutdown_connections 2024-04-06T12:31:44.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.321+0000 7ff81385f700 1 -- 172.21.15.189:0/1877075075 wait complete. 2024-04-06T12:31:44.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.322+0000 7ff81385f700 1 Processor -- start 2024-04-06T12:31:44.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.322+0000 7ff81385f700 1 -- start start 2024-04-06T12:31:44.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.323+0000 7ff81385f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff80c0d0a60 0x7ff80c0b7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:44.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.323+0000 7ff81385f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff80c0d1af0 0x7ff80c0b5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:44.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.323+0000 7ff81385f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff80c15d7e0 0x7ff80c0b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:44.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.323+0000 7ff81385f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c15c660 con 0x7ff80c15d7e0 2024-04-06T12:31:44.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.323+0000 7ff81385f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c0b3fb0 con 0x7ff80c0d1af0 2024-04-06T12:31:44.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.323+0000 7ff81385f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c0c9270 con 0x7ff80c0d0a60 2024-04-06T12:31:44.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.323+0000 7ff811dfc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff80c15d7e0 0x7ff80c0b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:44.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.323+0000 7ff811dfc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff80c15d7e0 0x7ff80c0b4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40622/0 (socket says 172.21.15.189:40622) 2024-04-06T12:31:44.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.323+0000 7ff811dfc700 1 -- 172.21.15.189:0/3211693498 learned_addr learned my addr 172.21.15.189:0/3211693498 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:44.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.323+0000 7ff810dfa700 1 --2- 172.21.15.189:0/3211693498 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff80c0d1af0 0x7ff80c0b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:44.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.323+0000 7ff8115fb700 1 --2- 172.21.15.189:0/3211693498 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff80c0d0a60 0x7ff80c0b7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:44.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.323+0000 7ff811dfc700 1 -- 172.21.15.189:0/3211693498 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff80c0d0a60 msgr2=0x7ff80c0b7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:44.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.323+0000 7ff811dfc700 1 --2- 172.21.15.189:0/3211693498 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff80c0d0a60 0x7ff80c0b7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.323+0000 7ff811dfc700 1 -- 172.21.15.189:0/3211693498 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff80c0d1af0 msgr2=0x7ff80c0b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:44.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.323+0000 7ff811dfc700 1 --2- 172.21.15.189:0/3211693498 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff80c0d1af0 0x7ff80c0b5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.323+0000 7ff811dfc700 1 -- 172.21.15.189:0/3211693498 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7fc0094a0 con 0x7ff80c15d7e0 2024-04-06T12:31:44.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.324+0000 7ff811dfc700 1 --2- 172.21.15.189:0/3211693498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff80c15d7e0 0x7ff80c0b4a00 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7ff80400cba0 tx=0x7ff80400ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:44.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.324+0000 7ff801ffb700 1 -- 172.21.15.189:0/3211693498 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80400aa40 con 0x7ff80c15d7e0 2024-04-06T12:31:44.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.324+0000 7ff801ffb700 1 -- 172.21.15.189:0/3211693498 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff80400aba0 con 0x7ff80c15d7e0 2024-04-06T12:31:44.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.324+0000 7ff801ffb700 1 -- 172.21.15.189:0/3211693498 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff804012400 con 0x7ff80c15d7e0 2024-04-06T12:31:44.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.324+0000 7ff81385f700 1 -- 172.21.15.189:0/3211693498 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80c0c9550 con 0x7ff80c15d7e0 2024-04-06T12:31:44.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.324+0000 7ff81385f700 1 -- 172.21.15.189:0/3211693498 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff80c0b4200 con 0x7ff80c15d7e0 2024-04-06T12:31:44.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.325+0000 7ff81385f700 1 -- 172.21.15.189:0/3211693498 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80c074990 con 0x7ff80c15d7e0 2024-04-06T12:31:44.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.325+0000 7ff801ffb700 1 -- 172.21.15.189:0/3211693498 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 82) v1 ==== 89304+0+0 (secure 0 0 0) 0x7ff80401b3b0 con 0x7ff80c15d7e0 2024-04-06T12:31:44.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.326+0000 7ff801ffb700 1 --2- 172.21.15.189:0/3211693498 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff7f006ae40 0x7ff7f006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:44.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.326+0000 7ff8115fb700 1 -- 172.21.15.189:0/3211693498 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff7f006ae40 msgr2=0x7ff7f006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:44.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.326+0000 7ff801ffb700 1 -- 172.21.15.189:0/3211693498 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7ff804089590 con 0x7ff80c15d7e0 2024-04-06T12:31:44.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.326+0000 7ff8115fb700 1 --2- 172.21.15.189:0/3211693498 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff7f006ae40 0x7ff7f006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:44.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.330+0000 7ff801ffb700 1 -- 172.21.15.189:0/3211693498 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff804054250 con 0x7ff80c15d7e0 2024-04-06T12:31:44.382 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:44.385+0000 7f75c0c26200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:31:44.429 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:44.431+0000 7f4b58d91200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:31:44.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.527+0000 7ff8115fb700 1 -- 172.21.15.189:0/3211693498 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff7f006ae40 msgr2=0x7ff7f006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:44.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.527+0000 7ff8115fb700 1 --2- 172.21.15.189:0/3211693498 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff7f006ae40 0x7ff7f006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:44.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.534+0000 7ff81385f700 1 -- 172.21.15.189:0/3211693498 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff80c15eaa0 con 0x7ff80c15d7e0 2024-04-06T12:31:44.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.535+0000 7ff801ffb700 1 -- 172.21.15.189:0/3211693498 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v145) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff804057870 con 0x7ff80c15d7e0 2024-04-06T12:31:44.532 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:31:44.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.537+0000 7ff81385f700 1 -- 172.21.15.189:0/3211693498 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff7f006ae40 msgr2=0x7ff7f006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:44.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.537+0000 7ff81385f700 1 --2- 172.21.15.189:0/3211693498 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff7f006ae40 0x7ff7f006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.537+0000 7ff81385f700 1 -- 172.21.15.189:0/3211693498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff80c15d7e0 msgr2=0x7ff80c0b4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:44.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.537+0000 7ff81385f700 1 --2- 172.21.15.189:0/3211693498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff80c15d7e0 0x7ff80c0b4a00 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7ff80400cba0 tx=0x7ff80400ceb0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.537+0000 7ff81385f700 1 -- 172.21.15.189:0/3211693498 shutdown_connections 2024-04-06T12:31:44.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.537+0000 7ff81385f700 1 --2- 172.21.15.189:0/3211693498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff80c15d7e0 0x7ff80c0b4a00 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.537+0000 7ff81385f700 1 --2- 172.21.15.189:0/3211693498 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff80c0d0a60 0x7ff80c0b7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.537+0000 7ff81385f700 1 --2- 172.21.15.189:0/3211693498 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff7f006ae40 0x7ff7f006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.537+0000 7ff81385f700 1 --2- 172.21.15.189:0/3211693498 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff80c0d1af0 0x7ff80c0b5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.537+0000 7ff81385f700 1 -- 172.21.15.189:0/3211693498 >> 172.21.15.189:0/3211693498 conn(0x7ff80c072250 msgr2=0x7ff80c0b83f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:44.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.537+0000 7ff81385f700 1 -- 172.21.15.189:0/3211693498 shutdown_connections 2024-04-06T12:31:44.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.537+0000 7ff81385f700 1 -- 172.21.15.189:0/3211693498 wait complete. 2024-04-06T12:31:44.537 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:44.540+0000 7f75c0c26200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:31:44.547 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:31:44.596 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:44.599+0000 7f4b58d91200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:31:44.647 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:44.650+0000 7f75c0c26200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:31:44.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.679+0000 7fec2859c700 1 Processor -- start 2024-04-06T12:31:44.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.679+0000 7fec2859c700 1 -- start start 2024-04-06T12:31:44.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.680+0000 7fec2859c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fec200b4660 0x7fec200b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:44.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.680+0000 7fec2859c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec200b2fc0 0x7fec200b3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:44.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.680+0000 7fec2859c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec200b5030 con 0x7fec2015a550 2024-04-06T12:31:44.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.680+0000 7fec2859c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec200cdbe0 con 0x7fec200b4660 2024-04-06T12:31:44.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.680+0000 7fec2859c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2015a940 con 0x7fec200b2fc0 2024-04-06T12:31:44.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.681+0000 7fec25b37700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fec200b4660 0x7fec200b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:44.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.681+0000 7fec25b37700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fec200b4660 0x7fec200b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40632/0 (socket says 172.21.15.189:40632) 2024-04-06T12:31:44.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.681+0000 7fec25b37700 1 -- 172.21.15.189:0/83549280 learned_addr learned my addr 172.21.15.189:0/83549280 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:44.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.681+0000 7fec26b39700 1 --2- 172.21.15.189:0/83549280 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec200b2fc0 0x7fec200b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:44.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.681+0000 7fec25b37700 1 -- 172.21.15.189:0/83549280 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec200b2fc0 msgr2=0x7fec200b3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:44.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.681+0000 7fec25b37700 1 --2- 172.21.15.189:0/83549280 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec200b2fc0 0x7fec200b3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.681+0000 7fec25b37700 1 -- 172.21.15.189:0/83549280 >> v1:172.21.15.189:6789/0 conn(0x7fec2015a550 legacy=0x7fec200cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:44.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.681+0000 7fec25b37700 1 -- 172.21.15.189:0/83549280 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2015ac00 con 0x7fec200b4660 2024-04-06T12:31:44.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.681+0000 7fec25b37700 1 --2- 172.21.15.189:0/83549280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec200b4660 0x7fec200b4af0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fec1c00bb90 tx=0x7fec1c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27280980ce702c1b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:44.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.681+0000 7fec25336700 1 -- 172.21.15.189:0/83549280 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c00dd30 con 0x7fec200b4660 2024-04-06T12:31:44.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.681+0000 7fec25336700 1 -- 172.21.15.189:0/83549280 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1c00de90 con 0x7fec200b4660 2024-04-06T12:31:44.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.681+0000 7fec25336700 1 -- 172.21.15.189:0/83549280 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c010690 con 0x7fec200b4660 2024-04-06T12:31:44.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.682+0000 7fec2859c700 1 -- 172.21.15.189:0/83549280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec200b4660 msgr2=0x7fec200b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:44.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.682+0000 7fec2859c700 1 --2- 172.21.15.189:0/83549280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec200b4660 0x7fec200b4af0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fec1c00bb90 tx=0x7fec1c00bea0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.682+0000 7fec2859c700 1 -- 172.21.15.189:0/83549280 shutdown_connections 2024-04-06T12:31:44.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.682+0000 7fec2859c700 1 --2- 172.21.15.189:0/83549280 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec200b2fc0 0x7fec200b3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.682+0000 7fec2859c700 1 --2- 172.21.15.189:0/83549280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec200b4660 0x7fec200b4af0 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.682+0000 7fec2859c700 1 -- 172.21.15.189:0/83549280 >> 172.21.15.189:0/83549280 conn(0x7fec200d00b0 msgr2=0x7fec200d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:44.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.682+0000 7fec2859c700 1 -- 172.21.15.189:0/83549280 shutdown_connections 2024-04-06T12:31:44.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.682+0000 7fec2859c700 1 -- 172.21.15.189:0/83549280 wait complete. 2024-04-06T12:31:44.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.683+0000 7fec2859c700 1 Processor -- start 2024-04-06T12:31:44.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.683+0000 7fec2859c700 1 -- start start 2024-04-06T12:31:44.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.684+0000 7fec2859c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec200b2fc0 0x7fec200b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:44.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.684+0000 7fec2859c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec200b4660 0x7fec20146a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:44.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.684+0000 7fec2859c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec20157230 0x7fec201576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:44.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.684+0000 7fec2859c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec200b0b20 con 0x7fec20157230 2024-04-06T12:31:44.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.684+0000 7fec2859c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec200afc30 con 0x7fec200b2fc0 2024-04-06T12:31:44.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.684+0000 7fec2859c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec200aff10 con 0x7fec200b4660 2024-04-06T12:31:44.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.684+0000 7fec26b39700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec20157230 0x7fec201576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:44.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.684+0000 7fec26b39700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec20157230 0x7fec201576e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40640/0 (socket says 172.21.15.189:40640) 2024-04-06T12:31:44.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.684+0000 7fec26b39700 1 -- 172.21.15.189:0/214600435 learned_addr learned my addr 172.21.15.189:0/214600435 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:44.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.684+0000 7fec25b37700 1 --2- 172.21.15.189:0/214600435 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec200b4660 0x7fec20146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:44.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.684+0000 7fec26338700 1 --2- 172.21.15.189:0/214600435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec200b2fc0 0x7fec200b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:44.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.684+0000 7fec26b39700 1 -- 172.21.15.189:0/214600435 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec200b4660 msgr2=0x7fec20146a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:44.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.684+0000 7fec26b39700 1 --2- 172.21.15.189:0/214600435 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec200b4660 0x7fec20146a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.684+0000 7fec26b39700 1 -- 172.21.15.189:0/214600435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec200b2fc0 msgr2=0x7fec200b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:44.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.684+0000 7fec26b39700 1 --2- 172.21.15.189:0/214600435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec200b2fc0 0x7fec200b0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.684+0000 7fec26b39700 1 -- 172.21.15.189:0/214600435 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1c00b7f0 con 0x7fec20157230 2024-04-06T12:31:44.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.685+0000 7fec26b39700 1 --2- 172.21.15.189:0/214600435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec20157230 0x7fec201576e0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fec1800efc0 tx=0x7fec1800c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:44.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.685+0000 7fec12ffd700 1 -- 172.21.15.189:0/214600435 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1800cad0 con 0x7fec20157230 2024-04-06T12:31:44.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.685+0000 7fec12ffd700 1 -- 172.21.15.189:0/214600435 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1800f040 con 0x7fec20157230 2024-04-06T12:31:44.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.685+0000 7fec12ffd700 1 -- 172.21.15.189:0/214600435 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1800ce70 con 0x7fec20157230 2024-04-06T12:31:44.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.685+0000 7fec2859c700 1 -- 172.21.15.189:0/214600435 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec200b01f0 con 0x7fec20157230 2024-04-06T12:31:44.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.685+0000 7fec2859c700 1 -- 172.21.15.189:0/214600435 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec200b07b0 con 0x7fec20157230 2024-04-06T12:31:44.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.686+0000 7fec2859c700 1 -- 172.21.15.189:0/214600435 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec20067bb0 con 0x7fec20157230 2024-04-06T12:31:44.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.686+0000 7fec12ffd700 1 -- 172.21.15.189:0/214600435 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 82) v1 ==== 89304+0+0 (secure 0 0 0) 0x7fec18014480 con 0x7fec20157230 2024-04-06T12:31:44.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.687+0000 7fec12ffd700 1 --2- 172.21.15.189:0/214600435 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec0406ae40 0x7fec0406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:44.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.687+0000 7fec26338700 1 -- 172.21.15.189:0/214600435 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec0406ae40 msgr2=0x7fec0406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:44.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.687+0000 7fec12ffd700 1 -- 172.21.15.189:0/214600435 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fec1808a140 con 0x7fec20157230 2024-04-06T12:31:44.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.687+0000 7fec26338700 1 --2- 172.21.15.189:0/214600435 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec0406ae40 0x7fec0406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:44.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.691+0000 7fec12ffd700 1 -- 172.21.15.189:0/214600435 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fec18054cf0 con 0x7fec20157230 2024-04-06T12:31:44.716 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:44.719+0000 7f4b58d91200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:31:44.729 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:44.732+0000 7f75c0c26200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:31:44.800 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:44.803+0000 7f4b58d91200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:31:44.805 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:44.808+0000 7f75c0c26200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:31:44.877 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:44.880+0000 7f4b58d91200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:31:44.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.888+0000 7fec26338700 1 -- 172.21.15.189:0/214600435 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec0406ae40 msgr2=0x7fec0406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:44.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.888+0000 7fec26338700 1 --2- 172.21.15.189:0/214600435 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec0406ae40 0x7fec0406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:44.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.894+0000 7fec2859c700 1 -- 172.21.15.189:0/214600435 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fec20074990 con 0x7fec20157230 2024-04-06T12:31:44.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.895+0000 7fec12ffd700 1 -- 172.21.15.189:0/214600435 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v145) v1 ==== 104+0+11 (secure 0 0 0) 0x7fec18058310 con 0x7fec20157230 2024-04-06T12:31:44.892 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:31:44.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.897+0000 7fec2859c700 1 -- 172.21.15.189:0/214600435 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec0406ae40 msgr2=0x7fec0406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:44.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.897+0000 7fec2859c700 1 --2- 172.21.15.189:0/214600435 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec0406ae40 0x7fec0406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.897+0000 7fec2859c700 1 -- 172.21.15.189:0/214600435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec20157230 msgr2=0x7fec201576e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:44.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.897+0000 7fec2859c700 1 --2- 172.21.15.189:0/214600435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec20157230 0x7fec201576e0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fec1800efc0 tx=0x7fec1800c0c0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.897+0000 7fec2859c700 1 -- 172.21.15.189:0/214600435 shutdown_connections 2024-04-06T12:31:44.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.897+0000 7fec2859c700 1 --2- 172.21.15.189:0/214600435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec20157230 0x7fec201576e0 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.897+0000 7fec2859c700 1 --2- 172.21.15.189:0/214600435 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fec0406ae40 0x7fec0406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.897+0000 7fec2859c700 1 --2- 172.21.15.189:0/214600435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec200b2fc0 0x7fec200b0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.897+0000 7fec2859c700 1 --2- 172.21.15.189:0/214600435 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec200b4660 0x7fec20146a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:44.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.897+0000 7fec2859c700 1 -- 172.21.15.189:0/214600435 >> 172.21.15.189:0/214600435 conn(0x7fec200d00b0 msgr2=0x7fec20157cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:44.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.898+0000 7fec2859c700 1 -- 172.21.15.189:0/214600435 shutdown_connections 2024-04-06T12:31:44.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:44.898+0000 7fec2859c700 1 -- 172.21.15.189:0/214600435 wait complete. 2024-04-06T12:31:44.907 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:31:45.008 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:45.011+0000 7f75c0c26200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:31:45.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.045+0000 7fc387f59700 1 Processor -- start 2024-04-06T12:31:45.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.045+0000 7fc387f59700 1 -- start start 2024-04-06T12:31:45.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.046+0000 7fc387f59700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc38015a550 0x7fc3800cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:45.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.046+0000 7fc387f59700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3800cdbe0 con 0x7fc3800b4d70 2024-04-06T12:31:45.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.046+0000 7fc387f59700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38015a940 con 0x7fc3800cf900 2024-04-06T12:31:45.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.046+0000 7fc387f59700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38015ac00 con 0x7fc38015a550 2024-04-06T12:31:45.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.047+0000 7fc3864f6700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc3800b4d70 0x7fc3800b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49138/0 (socket says 172.21.15.189:49138) 2024-04-06T12:31:45.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.047+0000 7fc3864f6700 1 -- 172.21.15.189:0/1129288673 learned_addr learned my addr 172.21.15.189:0/1129288673 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:45.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.048+0000 7fc385cf5700 1 --2- 172.21.15.189:0/1129288673 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc38015a550 0x7fc3800cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:45.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.048+0000 7fc384cf3700 1 -- 172.21.15.189:0/1129288673 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931613614 0 0) 0x7fc3800cdbe0 con 0x7fc3800b4d70 2024-04-06T12:31:45.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.048+0000 7fc384cf3700 1 -- 172.21.15.189:0/1129288673 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc36c0035b0 con 0x7fc3800b4d70 2024-04-06T12:31:45.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.048+0000 7fc384cf3700 1 -- 172.21.15.189:0/1129288673 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637702278 0 0) 0x7fc38015a940 con 0x7fc3800cf900 2024-04-06T12:31:45.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.048+0000 7fc384cf3700 1 -- 172.21.15.189:0/1129288673 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3800cdbe0 con 0x7fc3800cf900 2024-04-06T12:31:45.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.048+0000 7fc385cf5700 1 -- 172.21.15.189:0/1129288673 >> v1:172.21.15.192:6789/0 conn(0x7fc3800cf900 legacy=0x7fc3800b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:45.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.048+0000 7fc385cf5700 1 -- 172.21.15.189:0/1129288673 >> v1:172.21.15.189:6789/0 conn(0x7fc3800b4d70 legacy=0x7fc3800b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:45.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.048+0000 7fc385cf5700 1 -- 172.21.15.189:0/1129288673 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38015aee0 con 0x7fc38015a550 2024-04-06T12:31:45.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.048+0000 7fc385cf5700 1 --2- 172.21.15.189:0/1129288673 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc38015a550 0x7fc3800cf3c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009870 tx=0x7fc37c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a73439680e89e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:45.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.048+0000 7fc384cf3700 1 -- 172.21.15.189:0/1129288673 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37c011830 con 0x7fc38015a550 2024-04-06T12:31:45.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.048+0000 7fc384cf3700 1 -- 172.21.15.189:0/1129288673 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37c011990 con 0x7fc38015a550 2024-04-06T12:31:45.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.048+0000 7fc384cf3700 1 -- 172.21.15.189:0/1129288673 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37c011d00 con 0x7fc38015a550 2024-04-06T12:31:45.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.049+0000 7fc387f59700 1 -- 172.21.15.189:0/1129288673 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc38015a550 msgr2=0x7fc3800cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:45.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.049+0000 7fc387f59700 1 --2- 172.21.15.189:0/1129288673 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc38015a550 0x7fc3800cf3c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009870 tx=0x7fc37c009b80 comp rx=0 tx=0).stop 2024-04-06T12:31:45.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.049+0000 7fc387f59700 1 -- 172.21.15.189:0/1129288673 shutdown_connections 2024-04-06T12:31:45.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.049+0000 7fc387f59700 1 --2- 172.21.15.189:0/1129288673 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc38015a550 0x7fc3800cf3c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.049+0000 7fc387f59700 1 -- 172.21.15.189:0/1129288673 >> 172.21.15.189:0/1129288673 conn(0x7fc3800d00b0 msgr2=0x7fc3800d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:45.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.050+0000 7fc387f59700 1 -- 172.21.15.189:0/1129288673 shutdown_connections 2024-04-06T12:31:45.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.050+0000 7fc387f59700 1 -- 172.21.15.189:0/1129288673 wait complete. 2024-04-06T12:31:45.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.051+0000 7fc387f59700 1 Processor -- start 2024-04-06T12:31:45.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.051+0000 7fc387f59700 1 -- start start 2024-04-06T12:31:45.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.052+0000 7fc387f59700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3800b4d70 0x7fc380148bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:45.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.052+0000 7fc387f59700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3800cf900 0x7fc3800b0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:45.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.052+0000 7fc387f59700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc3800b0cc0 0x7fc3800a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:45.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.052+0000 7fc387f59700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3800b1150 con 0x7fc3800cf900 2024-04-06T12:31:45.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.052+0000 7fc387f59700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc380149130 con 0x7fc3800b4d70 2024-04-06T12:31:45.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.052+0000 7fc387f59700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3800aa280 con 0x7fc3800b0cc0 2024-04-06T12:31:45.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.053+0000 7fc3854f4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3800cf900 0x7fc3800b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:45.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.053+0000 7fc3854f4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3800cf900 0x7fc3800b0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40656/0 (socket says 172.21.15.189:40656) 2024-04-06T12:31:45.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.053+0000 7fc3854f4700 1 -- 172.21.15.189:0/1793878801 learned_addr learned my addr 172.21.15.189:0/1793878801 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:45.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.053+0000 7fc3864f6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc3800b0cc0 0x7fc3800a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:45.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.053+0000 7fc385cf5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3800b4d70 0x7fc380148bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:45.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.053+0000 7fc3864f6700 1 -- 172.21.15.189:0/1793878801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3800b4d70 msgr2=0x7fc380148bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:45.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.053+0000 7fc3864f6700 1 --2- 172.21.15.189:0/1793878801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3800b4d70 0x7fc380148bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.053+0000 7fc3864f6700 1 -- 172.21.15.189:0/1793878801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3800cf900 msgr2=0x7fc3800b0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:45.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.053+0000 7fc3864f6700 1 --2- 172.21.15.189:0/1793878801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3800cf900 0x7fc3800b0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.053+0000 7fc3864f6700 1 -- 172.21.15.189:0/1793878801 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37c0094a0 con 0x7fc3800b0cc0 2024-04-06T12:31:45.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.053+0000 7fc3864f6700 1 --2- 172.21.15.189:0/1793878801 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc3800b0cc0 0x7fc3800a84b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc37800ca80 tx=0x7fc37800cd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:45.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.053+0000 7fc3767fc700 1 -- 172.21.15.189:0/1793878801 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37800a9a0 con 0x7fc3800b0cc0 2024-04-06T12:31:45.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.054+0000 7fc3767fc700 1 -- 172.21.15.189:0/1793878801 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37800ab00 con 0x7fc3800b0cc0 2024-04-06T12:31:45.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.054+0000 7fc3767fc700 1 -- 172.21.15.189:0/1793878801 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37800ae70 con 0x7fc3800b0cc0 2024-04-06T12:31:45.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.054+0000 7fc387f59700 1 -- 172.21.15.189:0/1793878801 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3800a91e0 con 0x7fc3800b0cc0 2024-04-06T12:31:45.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.054+0000 7fc387f59700 1 -- 172.21.15.189:0/1793878801 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3800a97a0 con 0x7fc3800b0cc0 2024-04-06T12:31:45.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.055+0000 7fc387f59700 1 -- 172.21.15.189:0/1793878801 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc380067bb0 con 0x7fc3800b0cc0 2024-04-06T12:31:45.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.056+0000 7fc3767fc700 1 -- 172.21.15.189:0/1793878801 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 82) v1 ==== 89304+0+0 (secure 0 0 0) 0x7fc378012400 con 0x7fc3800b0cc0 2024-04-06T12:31:45.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.056+0000 7fc3767fc700 1 --2- 172.21.15.189:0/1793878801 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc36406ae40 0x7fc36406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:45.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.056+0000 7fc385cf5700 1 -- 172.21.15.189:0/1793878801 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc36406ae40 msgr2=0x7fc36406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:45.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.056+0000 7fc385cf5700 1 --2- 172.21.15.189:0/1793878801 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc36406ae40 0x7fc36406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:45.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.056+0000 7fc3767fc700 1 -- 172.21.15.189:0/1793878801 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fc378089350 con 0x7fc3800b0cc0 2024-04-06T12:31:45.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.061+0000 7fc3767fc700 1 -- 172.21.15.189:0/1793878801 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc3780540c0 con 0x7fc3800b0cc0 2024-04-06T12:31:45.085 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:45.088+0000 7f75c0c26200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:31:45.095 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:45.098+0000 7f4b58d91200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:31:45.173 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:45.176+0000 7f4b58d91200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:31:45.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.257+0000 7fc385cf5700 1 -- 172.21.15.189:0/1793878801 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc36406ae40 msgr2=0x7fc36406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:45.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.257+0000 7fc385cf5700 1 --2- 172.21.15.189:0/1793878801 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc36406ae40 0x7fc36406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:45.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.287+0000 7fc387f59700 1 -- 172.21.15.189:0/1793878801 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc380074990 con 0x7fc3800b0cc0 2024-04-06T12:31:45.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.288+0000 7fc3767fc700 1 -- 172.21.15.189:0/1793878801 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v145) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc3780576e0 con 0x7fc3800b0cc0 2024-04-06T12:31:45.285 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:31:45.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.290+0000 7fc387f59700 1 -- 172.21.15.189:0/1793878801 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc36406ae40 msgr2=0x7fc36406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:45.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.290+0000 7fc387f59700 1 --2- 172.21.15.189:0/1793878801 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc36406ae40 0x7fc36406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.290+0000 7fc387f59700 1 -- 172.21.15.189:0/1793878801 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc3800b0cc0 msgr2=0x7fc3800a84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:45.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.290+0000 7fc387f59700 1 --2- 172.21.15.189:0/1793878801 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc3800b0cc0 0x7fc3800a84b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc37800ca80 tx=0x7fc37800cd90 comp rx=0 tx=0).stop 2024-04-06T12:31:45.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.290+0000 7fc387f59700 1 -- 172.21.15.189:0/1793878801 shutdown_connections 2024-04-06T12:31:45.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.290+0000 7fc387f59700 1 --2- 172.21.15.189:0/1793878801 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc3800b0cc0 0x7fc3800a84b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.290+0000 7fc387f59700 1 --2- 172.21.15.189:0/1793878801 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc36406ae40 0x7fc36406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.290+0000 7fc387f59700 1 --2- 172.21.15.189:0/1793878801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3800b4d70 0x7fc380148bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.290+0000 7fc387f59700 1 --2- 172.21.15.189:0/1793878801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3800cf900 0x7fc3800b0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.290+0000 7fc387f59700 1 -- 172.21.15.189:0/1793878801 >> 172.21.15.189:0/1793878801 conn(0x7fc3800d00b0 msgr2=0x7fc3800b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:45.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.290+0000 7fc387f59700 1 -- 172.21.15.189:0/1793878801 shutdown_connections 2024-04-06T12:31:45.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.290+0000 7fc387f59700 1 -- 172.21.15.189:0/1793878801 wait complete. 2024-04-06T12:31:45.300 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T12:31:45.404 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:45.407+0000 7f75c0c26200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:31:45.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.432+0000 7f986dcd1700 1 Processor -- start 2024-04-06T12:31:45.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.432+0000 7f986dcd1700 1 -- start start 2024-04-06T12:31:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.433+0000 7f986dcd1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f986815d7e0 0x7f98680d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.433+0000 7f986dcd1700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f98680d0890 0x7f98680d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.433+0000 7f986dcd1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98680d1af0 0x7f98680b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.433+0000 7f986dcd1700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98680d1f80 con 0x7f986815d7e0 2024-04-06T12:31:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.433+0000 7f986dcd1700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98680cdf80 con 0x7f98680d0890 2024-04-06T12:31:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.433+0000 7f986dcd1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98680d0cf0 con 0x7f98680d1af0 2024-04-06T12:31:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.433+0000 7f98677fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f986815d7e0 0x7f98680d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.433+0000 7f98677fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f986815d7e0 0x7f98680d0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40664/0 (socket says 172.21.15.189:40664) 2024-04-06T12:31:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.433+0000 7f98677fe700 1 -- 172.21.15.189:0/1609826508 learned_addr learned my addr 172.21.15.189:0/1609826508 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:45.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.433+0000 7f9866ffd700 1 --2- 172.21.15.189:0/1609826508 >> v2:172.21.15.192:3300/0 conn(0x7f98680d0890 0x7f98680d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:45.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.433+0000 7f9867fff700 1 --2- 172.21.15.189:0/1609826508 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98680d1af0 0x7f98680b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:45.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.433+0000 7f98677fe700 1 -- 172.21.15.189:0/1609826508 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98680d1af0 msgr2=0x7f98680b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:45.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.433+0000 7f98677fe700 1 --2- 172.21.15.189:0/1609826508 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98680d1af0 0x7f98680b4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.433+0000 7f98677fe700 1 -- 172.21.15.189:0/1609826508 >> v2:172.21.15.192:3300/0 conn(0x7f98680d0890 msgr2=0x7f98680d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:45.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.433+0000 7f98677fe700 1 --2- 172.21.15.189:0/1609826508 >> v2:172.21.15.192:3300/0 conn(0x7f98680d0890 0x7f98680d15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.433+0000 7f98677fe700 1 -- 172.21.15.189:0/1609826508 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98680d3230 con 0x7f986815d7e0 2024-04-06T12:31:45.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.434+0000 7f98677fe700 1 --2- 172.21.15.189:0/1609826508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f986815d7e0 0x7f98680d0350 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f98680cfda0 tx=0x7f985800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dacdb3d68280306e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:45.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.434+0000 7f98667fc700 1 -- 172.21.15.189:0/1609826508 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985800e070 con 0x7f986815d7e0 2024-04-06T12:31:45.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.434+0000 7f98667fc700 1 -- 172.21.15.189:0/1609826508 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985800b910 con 0x7f986815d7e0 2024-04-06T12:31:45.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.434+0000 7f98667fc700 1 -- 172.21.15.189:0/1609826508 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985800bcc0 con 0x7f986815d7e0 2024-04-06T12:31:45.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.434+0000 7f986dcd1700 1 -- 172.21.15.189:0/1609826508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f986815d7e0 msgr2=0x7f98680d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:45.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.434+0000 7f986dcd1700 1 --2- 172.21.15.189:0/1609826508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f986815d7e0 0x7f98680d0350 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f98680cfda0 tx=0x7f985800b330 comp rx=0 tx=0).stop 2024-04-06T12:31:45.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.434+0000 7f986dcd1700 1 -- 172.21.15.189:0/1609826508 shutdown_connections 2024-04-06T12:31:45.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.434+0000 7f986dcd1700 1 --2- 172.21.15.189:0/1609826508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f986815d7e0 0x7f98680d0350 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.434+0000 7f986dcd1700 1 --2- 172.21.15.189:0/1609826508 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98680d1af0 0x7f98680b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.434+0000 7f986dcd1700 1 --2- 172.21.15.189:0/1609826508 >> v2:172.21.15.192:3300/0 conn(0x7f98680d0890 0x7f98680d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.434+0000 7f986dcd1700 1 -- 172.21.15.189:0/1609826508 >> 172.21.15.189:0/1609826508 conn(0x7f9868072250 msgr2=0x7f9868072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:45.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.434+0000 7f986dcd1700 1 -- 172.21.15.189:0/1609826508 shutdown_connections 2024-04-06T12:31:45.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.434+0000 7f986dcd1700 1 -- 172.21.15.189:0/1609826508 wait complete. 2024-04-06T12:31:45.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.436+0000 7f986dcd1700 1 Processor -- start 2024-04-06T12:31:45.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.436+0000 7f986dcd1700 1 -- start start 2024-04-06T12:31:45.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.436+0000 7f986dcd1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98680d0890 0x7f98681f23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:45.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f986dcd1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98680d1af0 0x7f98680ac090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:45.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f986dcd1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f986815d7e0 0x7f98680ac5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:45.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f986dcd1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98680cb390 con 0x7f98680d1af0 2024-04-06T12:31:45.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f986dcd1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98680cb670 con 0x7f986815d7e0 2024-04-06T12:31:45.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f986dcd1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98680cb950 con 0x7f98680d0890 2024-04-06T12:31:45.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f9866ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98680d1af0 0x7f98680ac090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:45.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f98677fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98680d0890 0x7f98681f23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:45.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f9866ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98680d1af0 0x7f98680ac090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40676/0 (socket says 172.21.15.189:40676) 2024-04-06T12:31:45.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f9867fff700 1 --2- 172.21.15.189:0/3745715268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f986815d7e0 0x7f98680ac5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:45.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f9866ffd700 1 -- 172.21.15.189:0/3745715268 learned_addr learned my addr 172.21.15.189:0/3745715268 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:45.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f98677fe700 1 -- 172.21.15.189:0/3745715268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f986815d7e0 msgr2=0x7f98680ac5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:45.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f98677fe700 1 --2- 172.21.15.189:0/3745715268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f986815d7e0 0x7f98680ac5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f98677fe700 1 -- 172.21.15.189:0/3745715268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98680d1af0 msgr2=0x7f98680ac090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:45.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f98677fe700 1 --2- 172.21.15.189:0/3745715268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98680d1af0 0x7f98680ac090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f98677fe700 1 -- 172.21.15.189:0/3745715268 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9858009b30 con 0x7f98680d0890 2024-04-06T12:31:45.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f9867fff700 1 --2- 172.21.15.189:0/3745715268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f986815d7e0 0x7f98680ac5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:45.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f98677fe700 1 --2- 172.21.15.189:0/3745715268 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98680d0890 0x7f98681f23e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9858015000 tx=0x7f9858004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:45.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f985ffff700 1 -- 172.21.15.189:0/3745715268 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98580099b0 con 0x7f98680d0890 2024-04-06T12:31:45.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f985ffff700 1 -- 172.21.15.189:0/3745715268 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985800cea0 con 0x7f98680d0890 2024-04-06T12:31:45.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f986dcd1700 1 -- 172.21.15.189:0/3745715268 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98680cbc30 con 0x7f98680d0890 2024-04-06T12:31:45.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.437+0000 7f985ffff700 1 -- 172.21.15.189:0/3745715268 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9858004470 con 0x7f98680d0890 2024-04-06T12:31:45.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.438+0000 7f986dcd1700 1 -- 172.21.15.189:0/3745715268 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98680ca660 con 0x7f98680d0890 2024-04-06T12:31:45.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.438+0000 7f986dcd1700 1 -- 172.21.15.189:0/3745715268 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98680cb7b0 con 0x7f98680d0890 2024-04-06T12:31:45.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.439+0000 7f985ffff700 1 -- 172.21.15.189:0/3745715268 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 82) v1 ==== 89304+0+0 (secure 0 0 0) 0x7f98580045d0 con 0x7f98680d0890 2024-04-06T12:31:45.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.439+0000 7f985ffff700 1 --2- 172.21.15.189:0/3745715268 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f984806ae40 0x7f984806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:45.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.439+0000 7f9866ffd700 1 -- 172.21.15.189:0/3745715268 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f984806ae40 msgr2=0x7f984806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:45.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.439+0000 7f985ffff700 1 -- 172.21.15.189:0/3745715268 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f985808ab50 con 0x7f98680d0890 2024-04-06T12:31:45.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.439+0000 7f9866ffd700 1 --2- 172.21.15.189:0/3745715268 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f984806ae40 0x7f984806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:45.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.444+0000 7f985ffff700 1 -- 172.21.15.189:0/3745715268 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f98680cb7b0 con 0x7f98680d0890 2024-04-06T12:31:45.511 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:45.514+0000 7f4b58d91200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:31:45.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.640+0000 7f9866ffd700 1 -- 172.21.15.189:0/3745715268 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f984806ae40 msgr2=0x7f984806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:45.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.640+0000 7f9866ffd700 1 --2- 172.21.15.189:0/3745715268 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f984806ae40 0x7f984806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:45.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.663+0000 7f986dcd1700 1 -- 172.21.15.189:0/3745715268 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f98680d3230 con 0x7f98680d0890 2024-04-06T12:31:45.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.664+0000 7f985ffff700 1 -- 172.21.15.189:0/3745715268 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v145) v1 ==== 80+0+467 (secure 0 0 0) 0x7f98680d3230 con 0x7f98680d0890 2024-04-06T12:31:45.661 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2323784940 2024-04-07T12:31:36.068397+0000 2024-04-06T12:31:45.661 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2597819342 2024-04-07T12:31:36.068397+0000 2024-04-06T12:31:45.661 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/3011108552 2024-04-07T12:31:36.068397+0000 2024-04-06T12:31:45.661 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/814932379 2024-04-07T12:31:36.068397+0000 2024-04-06T12:31:45.661 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2994336395 2024-04-07T12:31:36.068397+0000 2024-04-06T12:31:45.661 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2720887145 2024-04-07T12:31:36.068397+0000 2024-04-06T12:31:45.661 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6833/47128 2024-04-07T12:31:36.068397+0000 2024-04-06T12:31:45.661 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6832/47128 2024-04-07T12:31:36.068397+0000 2024-04-06T12:31:45.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.666+0000 7f986dcd1700 1 -- 172.21.15.189:0/3745715268 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f984806ae40 msgr2=0x7f984806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:45.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.666+0000 7f986dcd1700 1 --2- 172.21.15.189:0/3745715268 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f984806ae40 0x7f984806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.666+0000 7f986dcd1700 1 -- 172.21.15.189:0/3745715268 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98680d0890 msgr2=0x7f98681f23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:45.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.666+0000 7f986dcd1700 1 --2- 172.21.15.189:0/3745715268 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98680d0890 0x7f98681f23e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9858015000 tx=0x7f9858004ad0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.666+0000 7f986dcd1700 1 -- 172.21.15.189:0/3745715268 shutdown_connections 2024-04-06T12:31:45.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.666+0000 7f986dcd1700 1 --2- 172.21.15.189:0/3745715268 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f984806ae40 0x7f984806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.666+0000 7f986dcd1700 1 --2- 172.21.15.189:0/3745715268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f986815d7e0 0x7f98680ac5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.666+0000 7f986dcd1700 1 --2- 172.21.15.189:0/3745715268 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98680d0890 0x7f98681f23e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.666+0000 7f986dcd1700 1 --2- 172.21.15.189:0/3745715268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98680d1af0 0x7f98680ac090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.666+0000 7f986dcd1700 1 -- 172.21.15.189:0/3745715268 >> 172.21.15.189:0/3745715268 conn(0x7f9868072250 msgr2=0x7f986815f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:45.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.666+0000 7f986dcd1700 1 -- 172.21.15.189:0/3745715268 shutdown_connections 2024-04-06T12:31:45.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.666+0000 7f986dcd1700 1 -- 172.21.15.189:0/3745715268 wait complete. 2024-04-06T12:31:45.665 INFO:teuthology.orchestra.run.smithi189.stderr:listed 8 entries 2024-04-06T12:31:45.676 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-06T12:31:45.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:45.676 DEBUG:teuthology.orchestra.run.smithi189:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-06T12:31:45.729 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:45.732+0000 7f75c0c26200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:31:45.730 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T12:31:45.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:31:45.730 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-06T12:31:45.747 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:31:45.747 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:31:45.765 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:31:45.765 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:31:45.765 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:31:45.805 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:31:45.805 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:31:45.805 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:31:45.824 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:31:45.825 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:31:45.851 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:45.854+0000 7f4b58d91200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:31:45.896 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:45.899+0000 7f75c0c26200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:31:45.968 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:45.971+0000 7f75c0c26200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:31:45.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.973+0000 7f6190a89700 1 Processor -- start 2024-04-06T12:31:45.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.973+0000 7f6190a89700 1 -- start start 2024-04-06T12:31:45.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.974+0000 7f6190a89700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f618c06ae40 0x7f618c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:45.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.974+0000 7f6190a89700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f618c0b3990 con 0x7f618c15ddf0 2024-04-06T12:31:45.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.974+0000 7f6190a89700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f618c0b3060 con 0x7f618c15d7b0 2024-04-06T12:31:45.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.974+0000 7f6190a89700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f618c0b3340 con 0x7f618c06ae40 2024-04-06T12:31:45.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.974+0000 7f618ad9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f618c15ddf0 0x7f618c15e2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49150/0 (socket says 172.21.15.189:49150) 2024-04-06T12:31:45.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.974+0000 7f618ad9d700 1 -- 172.21.15.189:0/3857335427 learned_addr learned my addr 172.21.15.189:0/3857335427 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:45.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.974+0000 7f6189d9b700 1 --2- 172.21.15.189:0/3857335427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f618c06ae40 0x7f618c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:45.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.974+0000 7f618959a700 1 -- 172.21.15.189:0/3857335427 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424378882 0 0) 0x7f618c0b3990 con 0x7f618c15ddf0 2024-04-06T12:31:45.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.974+0000 7f618959a700 1 -- 172.21.15.189:0/3857335427 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61740035b0 con 0x7f618c15ddf0 2024-04-06T12:31:45.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.974+0000 7f618959a700 1 -- 172.21.15.189:0/3857335427 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167052200 0 0) 0x7f618c0b3060 con 0x7f618c15d7b0 2024-04-06T12:31:45.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.974+0000 7f618959a700 1 -- 172.21.15.189:0/3857335427 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f618c0b3990 con 0x7f618c15d7b0 2024-04-06T12:31:45.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.974+0000 7f6189d9b700 1 -- 172.21.15.189:0/3857335427 >> v1:172.21.15.192:6789/0 conn(0x7f618c15d7b0 legacy=0x7f618c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:45.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.974+0000 7f6189d9b700 1 -- 172.21.15.189:0/3857335427 >> v1:172.21.15.189:6789/0 conn(0x7f618c15ddf0 legacy=0x7f618c15e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:45.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.974+0000 7f6189d9b700 1 -- 172.21.15.189:0/3857335427 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f618c0b3620 con 0x7f618c06ae40 2024-04-06T12:31:45.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.974+0000 7f6189d9b700 1 --2- 172.21.15.189:0/3857335427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f618c06ae40 0x7f618c0b3e20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6180009870 tx=0x7f6180009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b0e3687f804aee6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:45.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.975+0000 7f618959a700 1 -- 172.21.15.189:0/3857335427 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6180011830 con 0x7f618c06ae40 2024-04-06T12:31:45.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.975+0000 7f618959a700 1 -- 172.21.15.189:0/3857335427 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6180011990 con 0x7f618c06ae40 2024-04-06T12:31:45.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.975+0000 7f618959a700 1 -- 172.21.15.189:0/3857335427 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6180011d20 con 0x7f618c06ae40 2024-04-06T12:31:45.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.975+0000 7f6190a89700 1 -- 172.21.15.189:0/3857335427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f618c06ae40 msgr2=0x7f618c0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:45.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.975+0000 7f6190a89700 1 --2- 172.21.15.189:0/3857335427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f618c06ae40 0x7f618c0b3e20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6180009870 tx=0x7f6180009b80 comp rx=0 tx=0).stop 2024-04-06T12:31:45.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.975+0000 7f6190a89700 1 -- 172.21.15.189:0/3857335427 shutdown_connections 2024-04-06T12:31:45.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.975+0000 7f6190a89700 1 --2- 172.21.15.189:0/3857335427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f618c06ae40 0x7f618c0b3e20 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.975+0000 7f6190a89700 1 -- 172.21.15.189:0/3857335427 >> 172.21.15.189:0/3857335427 conn(0x7f618c0d00b0 msgr2=0x7f618c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:45.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.975+0000 7f6190a89700 1 -- 172.21.15.189:0/3857335427 shutdown_connections 2024-04-06T12:31:45.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.975+0000 7f6190a89700 1 -- 172.21.15.189:0/3857335427 wait complete. 2024-04-06T12:31:45.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.977+0000 7f6190a89700 1 Processor -- start 2024-04-06T12:31:45.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.977+0000 7f6190a89700 1 -- start start 2024-04-06T12:31:45.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.978+0000 7f6190a89700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f618c06ae40 0x7f618c1439e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:45.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.978+0000 7f6190a89700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f618c15d7b0 0x7f618c155a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:45.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.978+0000 7f6190a89700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f618c15ddf0 0x7f618c155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:45.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.978+0000 7f6190a89700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618c1565c0 con 0x7f618c06ae40 2024-04-06T12:31:45.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.978+0000 7f6190a89700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618c1568a0 con 0x7f618c15ddf0 2024-04-06T12:31:45.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.978+0000 7f6190a89700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f618c156b80 con 0x7f618c15d7b0 2024-04-06T12:31:45.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.978+0000 7f618a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f618c06ae40 0x7f618c1439e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:45.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.978+0000 7f618a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f618c06ae40 0x7f618c1439e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40684/0 (socket says 172.21.15.189:40684) 2024-04-06T12:31:45.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.978+0000 7f618a59c700 1 -- 172.21.15.189:0/1190085114 learned_addr learned my addr 172.21.15.189:0/1190085114 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:45.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.978+0000 7f6189d9b700 1 --2- 172.21.15.189:0/1190085114 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f618c15d7b0 0x7f618c155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:45.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.978+0000 7f618ad9d700 1 --2- 172.21.15.189:0/1190085114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f618c15ddf0 0x7f618c155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:45.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.978+0000 7f6189d9b700 1 -- 172.21.15.189:0/1190085114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f618c15ddf0 msgr2=0x7f618c155f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:45.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.978+0000 7f6189d9b700 1 --2- 172.21.15.189:0/1190085114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f618c15ddf0 0x7f618c155f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.978+0000 7f6189d9b700 1 -- 172.21.15.189:0/1190085114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f618c06ae40 msgr2=0x7f618c1439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:45.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.978+0000 7f6189d9b700 1 --2- 172.21.15.189:0/1190085114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f618c06ae40 0x7f618c1439e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:45.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.978+0000 7f6189d9b700 1 -- 172.21.15.189:0/1190085114 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61800094a0 con 0x7f618c15d7b0 2024-04-06T12:31:45.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.978+0000 7f618a59c700 1 --2- 172.21.15.189:0/1190085114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f618c06ae40 0x7f618c1439e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:45.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.979+0000 7f6189d9b700 1 --2- 172.21.15.189:0/1190085114 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f618c15d7b0 0x7f618c155a50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6180016000 tx=0x7f6180004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:45.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.979+0000 7f617affd700 1 -- 172.21.15.189:0/1190085114 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618001a6e0 con 0x7f618c15d7b0 2024-04-06T12:31:45.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.979+0000 7f617affd700 1 -- 172.21.15.189:0/1190085114 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f618001a860 con 0x7f618c15d7b0 2024-04-06T12:31:45.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.979+0000 7f6190a89700 1 -- 172.21.15.189:0/1190085114 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f618c156e60 con 0x7f618c15d7b0 2024-04-06T12:31:45.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.979+0000 7f617affd700 1 -- 172.21.15.189:0/1190085114 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618001ac10 con 0x7f618c15d7b0 2024-04-06T12:31:45.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.979+0000 7f6190a89700 1 -- 172.21.15.189:0/1190085114 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f618c154dd0 con 0x7f618c15d7b0 2024-04-06T12:31:45.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.980+0000 7f6190a89700 1 -- 172.21.15.189:0/1190085114 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f618c157380 con 0x7f618c15d7b0 2024-04-06T12:31:45.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.980+0000 7f617affd700 1 -- 172.21.15.189:0/1190085114 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 82) v1 ==== 89304+0+0 (secure 0 0 0) 0x7f6180022070 con 0x7f618c15d7b0 2024-04-06T12:31:45.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.981+0000 7f617affd700 1 --2- 172.21.15.189:0/1190085114 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f616c06ae40 0x7f616c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:45.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.981+0000 7f618a59c700 1 -- 172.21.15.189:0/1190085114 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f616c06ae40 msgr2=0x7f616c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:45.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.981+0000 7f617affd700 1 -- 172.21.15.189:0/1190085114 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f618008a7f0 con 0x7f618c15d7b0 2024-04-06T12:31:45.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.981+0000 7f618a59c700 1 --2- 172.21.15.189:0/1190085114 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f616c06ae40 0x7f616c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:45.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:45.985+0000 7f617affd700 1 -- 172.21.15.189:0/1190085114 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f618c157380 con 0x7f618c15d7b0 2024-04-06T12:31:46.021 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:46.024+0000 7f4b58d91200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:31:46.099 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:46.102+0000 7f4b58d91200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:31:46.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.182+0000 7f618a59c700 1 -- 172.21.15.189:0/1190085114 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f616c06ae40 msgr2=0x7f616c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:46.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.182+0000 7f618a59c700 1 --2- 172.21.15.189:0/1190085114 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f616c06ae40 0x7f616c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:46.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.241+0000 7f6190a89700 1 -- 172.21.15.189:0/1190085114 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f618c157380 con 0x7f618c15d7b0 2024-04-06T12:31:46.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.242+0000 7f617affd700 1 -- 172.21.15.189:0/1190085114 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f618005c2a0 con 0x7f618c15d7b0 2024-04-06T12:31:46.239 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:46.239 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7115,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/538134862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":538134862},{"type":"v1","addr":"172.21.15.192:6834","nonce":538134862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7915,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1788966297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1788966297},{"type":"v1","addr":"172.21.15.189:6837","nonce":1788966297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7930,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2932927074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2932927074},{"type":"v1","addr":"172.21.15.192:6835","nonce":2932927074}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:31:22.083923+0000","modified":"2024-04-06T12:31:25.104510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/4249639368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":4249639368},{"type":"v1","addr":"172.21.15.189:6836","nonce":4249639368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-06T12:31:46.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.245+0000 7f6190a89700 1 -- 172.21.15.189:0/1190085114 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f616c06ae40 msgr2=0x7f616c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:46.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.245+0000 7f6190a89700 1 --2- 172.21.15.189:0/1190085114 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f616c06ae40 0x7f616c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:46.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.245+0000 7f6190a89700 1 -- 172.21.15.189:0/1190085114 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f618c15d7b0 msgr2=0x7f618c155a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:46.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.245+0000 7f6190a89700 1 --2- 172.21.15.189:0/1190085114 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f618c15d7b0 0x7f618c155a50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6180016000 tx=0x7f6180004750 comp rx=0 tx=0).stop 2024-04-06T12:31:46.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.245+0000 7f6190a89700 1 -- 172.21.15.189:0/1190085114 shutdown_connections 2024-04-06T12:31:46.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.245+0000 7f6190a89700 1 --2- 172.21.15.189:0/1190085114 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f616c06ae40 0x7f616c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:46.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.245+0000 7f6190a89700 1 --2- 172.21.15.189:0/1190085114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f618c15ddf0 0x7f618c155f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:46.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.245+0000 7f6190a89700 1 --2- 172.21.15.189:0/1190085114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f618c06ae40 0x7f618c1439e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:46.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.245+0000 7f6190a89700 1 --2- 172.21.15.189:0/1190085114 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f618c15d7b0 0x7f618c155a50 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:46.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.245+0000 7f6190a89700 1 -- 172.21.15.189:0/1190085114 >> 172.21.15.189:0/1190085114 conn(0x7f618c0d00b0 msgr2=0x7f618c0adf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:46.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.245+0000 7f6190a89700 1 -- 172.21.15.189:0/1190085114 shutdown_connections 2024-04-06T12:31:46.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.245+0000 7f6190a89700 1 -- 172.21.15.189:0/1190085114 wait complete. 2024-04-06T12:31:46.243 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 62 2024-04-06T12:31:46.259 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:31:46.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.431+0000 7f82a772e700 1 Processor -- start 2024-04-06T12:31:46.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.431+0000 7f82a772e700 1 -- start start 2024-04-06T12:31:46.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.432+0000 7f82a772e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f82a015d7e0 0x7f82a00d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:46.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.432+0000 7f82a772e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a00cdf80 con 0x7f82a00d1cc0 2024-04-06T12:31:46.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.432+0000 7f82a772e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a00d0cf0 con 0x7f82a00d0890 2024-04-06T12:31:46.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.432+0000 7f82a772e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82a00d3230 con 0x7f82a015d7e0 2024-04-06T12:31:46.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.432+0000 7f82a5ccb700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f82a00d1cc0 0x7f82a00b66b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49158/0 (socket says 172.21.15.189:49158) 2024-04-06T12:31:46.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.432+0000 7f82a5ccb700 1 -- 172.21.15.189:0/616185725 learned_addr learned my addr 172.21.15.189:0/616185725 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:46.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.432+0000 7f82a54ca700 1 --2- 172.21.15.189:0/616185725 >> v2:172.21.15.192:3300/0 conn(0x7f82a015d7e0 0x7f82a00d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:46.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.433+0000 7f8297fff700 1 -- 172.21.15.189:0/616185725 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71880037 0 0) 0x7f82a00cdf80 con 0x7f82a00d1cc0 2024-04-06T12:31:46.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.433+0000 7f8297fff700 1 -- 172.21.15.189:0/616185725 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f828c0035b0 con 0x7f82a00d1cc0 2024-04-06T12:31:46.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.433+0000 7f8297fff700 1 -- 172.21.15.189:0/616185725 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096487765 0 0) 0x7f82a00d0cf0 con 0x7f82a00d0890 2024-04-06T12:31:46.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.433+0000 7f8297fff700 1 -- 172.21.15.189:0/616185725 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a00cdf80 con 0x7f82a00d0890 2024-04-06T12:31:46.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.433+0000 7f82a54ca700 1 -- 172.21.15.189:0/616185725 >> v1:172.21.15.192:6789/0 conn(0x7f82a00d0890 legacy=0x7f82a00d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:46.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.433+0000 7f82a54ca700 1 -- 172.21.15.189:0/616185725 >> v1:172.21.15.189:6789/0 conn(0x7f82a00d1cc0 legacy=0x7f82a00b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:46.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.433+0000 7f82a54ca700 1 -- 172.21.15.189:0/616185725 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a00d2da0 con 0x7f82a015d7e0 2024-04-06T12:31:46.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.433+0000 7f82a54ca700 1 --2- 172.21.15.189:0/616185725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82a015d7e0 0x7f82a00d0350 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f829c009840 tx=0x7f829c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b03ac9910627c2d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:46.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.433+0000 7f8297fff700 1 -- 172.21.15.189:0/616185725 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829c011830 con 0x7f82a015d7e0 2024-04-06T12:31:46.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.433+0000 7f8297fff700 1 -- 172.21.15.189:0/616185725 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829c011990 con 0x7f82a015d7e0 2024-04-06T12:31:46.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.433+0000 7f8297fff700 1 -- 172.21.15.189:0/616185725 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829c011d20 con 0x7f82a015d7e0 2024-04-06T12:31:46.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.434+0000 7f82a772e700 1 -- 172.21.15.189:0/616185725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82a015d7e0 msgr2=0x7f82a00d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:46.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.434+0000 7f82a772e700 1 --2- 172.21.15.189:0/616185725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82a015d7e0 0x7f82a00d0350 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f829c009840 tx=0x7f829c009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:46.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.434+0000 7f82a772e700 1 -- 172.21.15.189:0/616185725 shutdown_connections 2024-04-06T12:31:46.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.434+0000 7f82a772e700 1 --2- 172.21.15.189:0/616185725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82a015d7e0 0x7f82a00d0350 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:46.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.434+0000 7f82a772e700 1 -- 172.21.15.189:0/616185725 >> 172.21.15.189:0/616185725 conn(0x7f82a0072250 msgr2=0x7f82a0072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:46.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.434+0000 7f82a772e700 1 -- 172.21.15.189:0/616185725 shutdown_connections 2024-04-06T12:31:46.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.434+0000 7f82a772e700 1 -- 172.21.15.189:0/616185725 wait complete. 2024-04-06T12:31:46.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.435+0000 7f82a772e700 1 Processor -- start 2024-04-06T12:31:46.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.435+0000 7f82a772e700 1 -- start start 2024-04-06T12:31:46.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.436+0000 7f82a772e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82a00d0890 0x7f82a0151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:46.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.436+0000 7f82a772e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82a00d1cc0 0x7f82a00cc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:46.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.436+0000 7f82a772e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82a015d7e0 0x7f82a00ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:46.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.436+0000 7f82a772e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a00ca4a0 con 0x7f82a00d1cc0 2024-04-06T12:31:46.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.436+0000 7f82a772e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a014f400 con 0x7f82a015d7e0 2024-04-06T12:31:46.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.436+0000 7f82a772e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82a00c8500 con 0x7f82a00d0890 2024-04-06T12:31:46.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.436+0000 7f82a4cc9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82a00d1cc0 0x7f82a00cc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:46.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.436+0000 7f82a4cc9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82a00d1cc0 0x7f82a00cc5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40690/0 (socket says 172.21.15.189:40690) 2024-04-06T12:31:46.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.436+0000 7f82a4cc9700 1 -- 172.21.15.189:0/4158656446 learned_addr learned my addr 172.21.15.189:0/4158656446 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:46.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.436+0000 7f82a54ca700 1 --2- 172.21.15.189:0/4158656446 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82a00d0890 0x7f82a0151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:46.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.436+0000 7f82a5ccb700 1 --2- 172.21.15.189:0/4158656446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82a015d7e0 0x7f82a00ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:46.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.437+0000 7f82a54ca700 1 -- 172.21.15.189:0/4158656446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82a015d7e0 msgr2=0x7f82a00ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:46.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.437+0000 7f82a54ca700 1 --2- 172.21.15.189:0/4158656446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82a015d7e0 0x7f82a00ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:46.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.437+0000 7f82a54ca700 1 -- 172.21.15.189:0/4158656446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82a00d1cc0 msgr2=0x7f82a00cc5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:46.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.437+0000 7f82a54ca700 1 --2- 172.21.15.189:0/4158656446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82a00d1cc0 0x7f82a00cc5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:46.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.437+0000 7f82a54ca700 1 -- 172.21.15.189:0/4158656446 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c0094a0 con 0x7f82a00d0890 2024-04-06T12:31:46.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.437+0000 7f82a4cc9700 1 --2- 172.21.15.189:0/4158656446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82a00d1cc0 0x7f82a00cc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:46.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.437+0000 7f82a54ca700 1 --2- 172.21.15.189:0/4158656446 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82a00d0890 0x7f82a0151ea0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f829c016000 tx=0x7f829c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:46.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.437+0000 7f8295ffb700 1 -- 172.21.15.189:0/4158656446 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829c01a6e0 con 0x7f82a00d0890 2024-04-06T12:31:46.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.437+0000 7f8295ffb700 1 -- 172.21.15.189:0/4158656446 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829c01a860 con 0x7f82a00d0890 2024-04-06T12:31:46.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.437+0000 7f8295ffb700 1 -- 172.21.15.189:0/4158656446 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829c01ac10 con 0x7f82a00d0890 2024-04-06T12:31:46.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.437+0000 7f82a772e700 1 -- 172.21.15.189:0/4158656446 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82a00ca6f0 con 0x7f82a00d0890 2024-04-06T12:31:46.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.437+0000 7f82a772e700 1 -- 172.21.15.189:0/4158656446 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a00cacb0 con 0x7f82a00d0890 2024-04-06T12:31:46.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.438+0000 7f82a772e700 1 -- 172.21.15.189:0/4158656446 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a0074990 con 0x7f82a00d0890 2024-04-06T12:31:46.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.439+0000 7f8295ffb700 1 -- 172.21.15.189:0/4158656446 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 82) v1 ==== 89304+0+0 (secure 0 0 0) 0x7f829c022070 con 0x7f82a00d0890 2024-04-06T12:31:46.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.439+0000 7f8295ffb700 1 --2- 172.21.15.189:0/4158656446 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f828406ae40 0x7f828406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:46.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.439+0000 7f82a4cc9700 1 -- 172.21.15.189:0/4158656446 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f828406ae40 msgr2=0x7f828406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:46.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.439+0000 7f8295ffb700 1 -- 172.21.15.189:0/4158656446 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f829c08a390 con 0x7f82a00d0890 2024-04-06T12:31:46.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.439+0000 7f82a4cc9700 1 --2- 172.21.15.189:0/4158656446 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f828406ae40 0x7f828406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:46.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.444+0000 7f8295ffb700 1 -- 172.21.15.189:0/4158656446 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f829c0550e0 con 0x7f82a00d0890 2024-04-06T12:31:46.594 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:46.597+0000 7f75c0c26200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:31:46.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.640+0000 7f82a4cc9700 1 -- 172.21.15.189:0/4158656446 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f828406ae40 msgr2=0x7f828406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:46.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.640+0000 7f82a4cc9700 1 --2- 172.21.15.189:0/4158656446 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f828406ae40 0x7f828406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:46.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.653+0000 7f82a772e700 1 -- 172.21.15.189:0/4158656446 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82a00d04a0 con 0x7f82a00d0890 2024-04-06T12:31:46.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.654+0000 7f8295ffb700 1 -- 172.21.15.189:0/4158656446 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+16862 (secure 0 0 0) 0x7f829c058700 con 0x7f82a00d0890 2024-04-06T12:31:46.651 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:46.651 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":145,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:31:36.068439+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:31:19.866331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-06T12:31:21.048344+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/2323784940":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/2597819342":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/3011108552":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/814932379":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/2994336395":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/2720887145":"2024-04-07T12:31:36.068397+0000","172.21.15.189:6833/47128":"2024-04-07T12:31:36.068397+0000","172.21.15.189:6832/47128":"2024-04-07T12:31:36.068397+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:31:46.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.656+0000 7f82a772e700 1 -- 172.21.15.189:0/4158656446 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f828406ae40 msgr2=0x7f828406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:46.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.656+0000 7f82a772e700 1 --2- 172.21.15.189:0/4158656446 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f828406ae40 0x7f828406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:46.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.656+0000 7f82a772e700 1 -- 172.21.15.189:0/4158656446 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82a00d0890 msgr2=0x7f82a0151ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:46.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.656+0000 7f82a772e700 1 --2- 172.21.15.189:0/4158656446 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82a00d0890 0x7f82a0151ea0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f829c016000 tx=0x7f829c004750 comp rx=0 tx=0).stop 2024-04-06T12:31:46.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.656+0000 7f82a772e700 1 -- 172.21.15.189:0/4158656446 shutdown_connections 2024-04-06T12:31:46.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.656+0000 7f82a772e700 1 --2- 172.21.15.189:0/4158656446 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f828406ae40 0x7f828406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:46.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.656+0000 7f82a772e700 1 --2- 172.21.15.189:0/4158656446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82a015d7e0 0x7f82a00ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:46.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.656+0000 7f82a772e700 1 --2- 172.21.15.189:0/4158656446 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82a00d0890 0x7f82a0151ea0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:46.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.656+0000 7f82a772e700 1 --2- 172.21.15.189:0/4158656446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82a00d1cc0 0x7f82a00cc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:46.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.656+0000 7f82a772e700 1 -- 172.21.15.189:0/4158656446 >> 172.21.15.189:0/4158656446 conn(0x7f82a0072250 msgr2=0x7f82a00ce110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:46.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.656+0000 7f82a772e700 1 -- 172.21.15.189:0/4158656446 shutdown_connections 2024-04-06T12:31:46.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.656+0000 7f82a772e700 1 -- 172.21.15.189:0/4158656446 wait complete. 2024-04-06T12:31:46.667 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:31:19.866331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:31:21.048344+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:31:46.667 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:31:46.674 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:46.677+0000 7f75c0c26200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:31:46.764 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:46.767+0000 7f4b58d91200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:31:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.798+0000 7f893f66f700 1 Processor -- start 2024-04-06T12:31:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.799+0000 7f893f66f700 1 -- start start 2024-04-06T12:31:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.799+0000 7f893f66f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f89380d11b0 0x7f89380d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.799+0000 7f893f66f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89380d1ad0 con 0x7f89380cb870 2024-04-06T12:31:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.799+0000 7f893f66f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89380cbd80 con 0x7f89380cfa00 2024-04-06T12:31:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.799+0000 7f893f66f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89380ccdc0 con 0x7f89380d11b0 2024-04-06T12:31:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.799+0000 7f893d40b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f89380d11b0 0x7f89380d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.800+0000 7f893cc0a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f89380cb870 0x7f893815c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49174/0 (socket says 172.21.15.189:49174) 2024-04-06T12:31:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.800+0000 7f893cc0a700 1 -- 172.21.15.189:0/742253969 learned_addr learned my addr 172.21.15.189:0/742253969 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.800+0000 7f893d40b700 1 --2- 172.21.15.189:0/742253969 >> v2:172.21.15.189:3300/0 conn(0x7f89380d11b0 0x7f89380d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40692/0 (socket says 172.21.15.189:40692) 2024-04-06T12:31:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.800+0000 7f893d40b700 1 -- 172.21.15.189:0/742253969 >> v1:172.21.15.192:6789/0 conn(0x7f89380cfa00 legacy=0x7f89380cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.800+0000 7f893d40b700 1 -- 172.21.15.189:0/742253969 >> v1:172.21.15.189:6789/0 conn(0x7f89380cb870 legacy=0x7f893815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.800+0000 7f893d40b700 1 -- 172.21.15.189:0/742253969 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89380cef20 con 0x7f89380d11b0 2024-04-06T12:31:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.800+0000 7f892ffff700 1 -- 172.21.15.189:0/742253969 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888336331 0 0) 0x7f89380d1ad0 con 0x7f89380cb870 2024-04-06T12:31:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.800+0000 7f893d40b700 1 --2- 172.21.15.189:0/742253969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89380d11b0 0x7f89380d1590 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f8934009840 tx=0x7f8934009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27e4eef02e728d98 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.800+0000 7f892ffff700 1 -- 172.21.15.189:0/742253969 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934011830 con 0x7f89380d11b0 2024-04-06T12:31:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.800+0000 7f892ffff700 1 -- 172.21.15.189:0/742253969 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8934011990 con 0x7f89380d11b0 2024-04-06T12:31:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.800+0000 7f892ffff700 1 -- 172.21.15.189:0/742253969 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934011d20 con 0x7f89380d11b0 2024-04-06T12:31:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.801+0000 7f893f66f700 1 -- 172.21.15.189:0/742253969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89380d11b0 msgr2=0x7f89380d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.801+0000 7f893f66f700 1 --2- 172.21.15.189:0/742253969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89380d11b0 0x7f89380d1590 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f8934009840 tx=0x7f8934009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.801+0000 7f893f66f700 1 -- 172.21.15.189:0/742253969 shutdown_connections 2024-04-06T12:31:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.801+0000 7f893f66f700 1 --2- 172.21.15.189:0/742253969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89380d11b0 0x7f89380d1590 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:46.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.801+0000 7f893f66f700 1 -- 172.21.15.189:0/742253969 >> 172.21.15.189:0/742253969 conn(0x7f89380b2db0 msgr2=0x7f89380b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:46.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.801+0000 7f893f66f700 1 -- 172.21.15.189:0/742253969 shutdown_connections 2024-04-06T12:31:46.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.801+0000 7f893f66f700 1 -- 172.21.15.189:0/742253969 wait complete. 2024-04-06T12:31:46.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.802+0000 7f893f66f700 1 Processor -- start 2024-04-06T12:31:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.802+0000 7f893f66f700 1 -- start start 2024-04-06T12:31:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.803+0000 7f893f66f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89380cb870 0x7f89381f3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.803+0000 7f893f66f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89380cfa00 0x7f8938157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.803+0000 7f893f66f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89380d11b0 0x7f89381550b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.803+0000 7f893f66f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89380c82f0 con 0x7f89380d11b0 2024-04-06T12:31:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.803+0000 7f893f66f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89381f3f70 con 0x7f89380cfa00 2024-04-06T12:31:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.803+0000 7f893f66f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89381f4170 con 0x7f89380cb870 2024-04-06T12:31:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.803+0000 7f893dc0c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89380d11b0 0x7f89381550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.804+0000 7f893dc0c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89380d11b0 0x7f89381550b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40698/0 (socket says 172.21.15.189:40698) 2024-04-06T12:31:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.804+0000 7f893dc0c700 1 -- 172.21.15.189:0/3904796790 learned_addr learned my addr 172.21.15.189:0/3904796790 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.804+0000 7f893d40b700 1 --2- 172.21.15.189:0/3904796790 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89380cb870 0x7f89381f3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.804+0000 7f893cc0a700 1 --2- 172.21.15.189:0/3904796790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89380cfa00 0x7f8938157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.804+0000 7f893dc0c700 1 -- 172.21.15.189:0/3904796790 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89380cb870 msgr2=0x7f89381f3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:46.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.804+0000 7f893dc0c700 1 --2- 172.21.15.189:0/3904796790 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89380cb870 0x7f89381f3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:46.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.804+0000 7f893dc0c700 1 -- 172.21.15.189:0/3904796790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89380cfa00 msgr2=0x7f8938157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:46.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.804+0000 7f893dc0c700 1 --2- 172.21.15.189:0/3904796790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89380cfa00 0x7f8938157940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:46.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.804+0000 7f893dc0c700 1 -- 172.21.15.189:0/3904796790 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89340094a0 con 0x7f89380d11b0 2024-04-06T12:31:46.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.804+0000 7f893dc0c700 1 --2- 172.21.15.189:0/3904796790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89380d11b0 0x7f89381550b0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f893000cc50 tx=0x7f893000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:46.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.804+0000 7f892dffb700 1 -- 172.21.15.189:0/3904796790 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8930003c00 con 0x7f89380d11b0 2024-04-06T12:31:46.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.804+0000 7f892dffb700 1 -- 172.21.15.189:0/3904796790 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8930003d60 con 0x7f89380d11b0 2024-04-06T12:31:46.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.804+0000 7f893f66f700 1 -- 172.21.15.189:0/3904796790 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89380c8540 con 0x7f89380d11b0 2024-04-06T12:31:46.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.804+0000 7f892dffb700 1 -- 172.21.15.189:0/3904796790 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893000aab0 con 0x7f89380d11b0 2024-04-06T12:31:46.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.804+0000 7f893f66f700 1 -- 172.21.15.189:0/3904796790 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89380c8b00 con 0x7f89380d11b0 2024-04-06T12:31:46.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.805+0000 7f893f66f700 1 -- 172.21.15.189:0/3904796790 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8938074990 con 0x7f89380d11b0 2024-04-06T12:31:46.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.806+0000 7f892dffb700 1 -- 172.21.15.189:0/3904796790 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 82) v1 ==== 89304+0+0 (secure 0 0 0) 0x7f893000ac10 con 0x7f89380d11b0 2024-04-06T12:31:46.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.807+0000 7f892dffb700 1 --2- 172.21.15.189:0/3904796790 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f891c06ae40 0x7f891c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:46.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.807+0000 7f893d40b700 1 -- 172.21.15.189:0/3904796790 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f891c06ae40 msgr2=0x7f891c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:46.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.807+0000 7f892dffb700 1 -- 172.21.15.189:0/3904796790 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f8930089960 con 0x7f89380d11b0 2024-04-06T12:31:46.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.807+0000 7f893d40b700 1 --2- 172.21.15.189:0/3904796790 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f891c06ae40 0x7f891c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:46.807 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:46.804+0000 7f75c0c26200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:31:46.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:46.811+0000 7f892dffb700 1 -- 172.21.15.189:0/3904796790 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8930054580 con 0x7f89380d11b0 2024-04-06T12:31:46.848 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:46.851+0000 7f4b58d91200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:31:46.978 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:46.981+0000 7f4b58d91200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:31:47.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.008+0000 7f893d40b700 1 -- 172.21.15.189:0/3904796790 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f891c06ae40 msgr2=0x7f891c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:47.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.008+0000 7f893d40b700 1 --2- 172.21.15.189:0/3904796790 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f891c06ae40 0x7f891c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:47.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.012+0000 7f893f66f700 1 -- 172.21.15.189:0/3904796790 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f893815bb20 con 0x7f89380d11b0 2024-04-06T12:31:47.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.012+0000 7f892dffb700 1 -- 172.21.15.189:0/3904796790 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v145) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8930057ba0 con 0x7f89380d11b0 2024-04-06T12:31:47.009 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:31:47.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.014+0000 7f893f66f700 1 -- 172.21.15.189:0/3904796790 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f891c06ae40 msgr2=0x7f891c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:47.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.014+0000 7f893f66f700 1 --2- 172.21.15.189:0/3904796790 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f891c06ae40 0x7f891c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.014+0000 7f893f66f700 1 -- 172.21.15.189:0/3904796790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89380d11b0 msgr2=0x7f89381550b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:47.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.014+0000 7f893f66f700 1 --2- 172.21.15.189:0/3904796790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89380d11b0 0x7f89381550b0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f893000cc50 tx=0x7f893000cf60 comp rx=0 tx=0).stop 2024-04-06T12:31:47.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.014+0000 7f893f66f700 1 -- 172.21.15.189:0/3904796790 shutdown_connections 2024-04-06T12:31:47.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.014+0000 7f893f66f700 1 --2- 172.21.15.189:0/3904796790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89380d11b0 0x7f89381550b0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.014+0000 7f893f66f700 1 --2- 172.21.15.189:0/3904796790 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89380cb870 0x7f89381f3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.014+0000 7f893f66f700 1 --2- 172.21.15.189:0/3904796790 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f891c06ae40 0x7f891c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.014+0000 7f893f66f700 1 --2- 172.21.15.189:0/3904796790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89380cfa00 0x7f8938157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.014+0000 7f893f66f700 1 -- 172.21.15.189:0/3904796790 >> 172.21.15.189:0/3904796790 conn(0x7f89380b2db0 msgr2=0x7f893806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:47.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.014+0000 7f893f66f700 1 -- 172.21.15.189:0/3904796790 shutdown_connections 2024-04-06T12:31:47.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.014+0000 7f893f66f700 1 -- 172.21.15.189:0/3904796790 wait complete. 2024-04-06T12:31:47.024 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T12:31:47.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.158+0000 7f7b7badf700 1 Processor -- start 2024-04-06T12:31:47.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.159+0000 7f7b7badf700 1 -- start start 2024-04-06T12:31:47.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.160+0000 7f7b7badf700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7b740cb870 0x7f7b7415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:47.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.160+0000 7f7b7badf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b740cfa00 0x7f7b740cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:47.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.160+0000 7f7b7badf700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b740cbd80 con 0x7f7b740d11b0 2024-04-06T12:31:47.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.160+0000 7f7b7badf700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b740ccdc0 con 0x7f7b740cb870 2024-04-06T12:31:47.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.160+0000 7f7b7badf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b740d0b50 con 0x7f7b740cfa00 2024-04-06T12:31:47.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.160+0000 7f7b7907a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7b740cb870 0x7f7b7415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:47.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.160+0000 7f7b7a07c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b740cfa00 0x7f7b740cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:47.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.160+0000 7f7b7987b700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f7b740d11b0 0x7f7b740d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:37802/0 (socket says 172.21.15.189:37802) 2024-04-06T12:31:47.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.160+0000 7f7b7987b700 1 -- 172.21.15.189:0/2511254050 learned_addr learned my addr 172.21.15.189:0/2511254050 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:47.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.160+0000 7f7b7907a700 1 -- 172.21.15.189:0/2511254050 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b740cfa00 msgr2=0x7f7b740cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:47.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.160+0000 7f7b7907a700 1 --2- 172.21.15.189:0/2511254050 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b740cfa00 0x7f7b740cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.160+0000 7f7b7907a700 1 -- 172.21.15.189:0/2511254050 >> v1:172.21.15.192:6789/0 conn(0x7f7b740d11b0 legacy=0x7f7b740d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:47.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.160+0000 7f7b7907a700 1 -- 172.21.15.189:0/2511254050 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b740cef20 con 0x7f7b740cb870 2024-04-06T12:31:47.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.160+0000 7f7b78879700 1 -- 172.21.15.189:0/2511254050 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473573635 0 0) 0x7f7b740cbd80 con 0x7f7b740d11b0 2024-04-06T12:31:47.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.160+0000 7f7b7a07c700 1 --2- 172.21.15.189:0/2511254050 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b740cfa00 0x7f7b740cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:47.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.160+0000 7f7b7907a700 1 --2- 172.21.15.189:0/2511254050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b740cb870 0x7f7b7415c280 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7b64009840 tx=0x7f7b64009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5564360ef6cd3d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:47.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.161+0000 7f7b78879700 1 -- 172.21.15.189:0/2511254050 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b64011830 con 0x7f7b740cb870 2024-04-06T12:31:47.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.161+0000 7f7b78879700 1 -- 172.21.15.189:0/2511254050 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b64011990 con 0x7f7b740cb870 2024-04-06T12:31:47.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.161+0000 7f7b78879700 1 -- 172.21.15.189:0/2511254050 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b64011d20 con 0x7f7b740cb870 2024-04-06T12:31:47.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.161+0000 7f7b7badf700 1 -- 172.21.15.189:0/2511254050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b740cb870 msgr2=0x7f7b7415c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:47.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.161+0000 7f7b7badf700 1 --2- 172.21.15.189:0/2511254050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b740cb870 0x7f7b7415c280 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7b64009840 tx=0x7f7b64009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:47.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.161+0000 7f7b7badf700 1 -- 172.21.15.189:0/2511254050 shutdown_connections 2024-04-06T12:31:47.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.161+0000 7f7b7badf700 1 --2- 172.21.15.189:0/2511254050 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b740cfa00 0x7f7b740cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.161+0000 7f7b7badf700 1 --2- 172.21.15.189:0/2511254050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b740cb870 0x7f7b7415c280 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.161+0000 7f7b7badf700 1 -- 172.21.15.189:0/2511254050 >> 172.21.15.189:0/2511254050 conn(0x7f7b740b2db0 msgr2=0x7f7b740b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:47.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.161+0000 7f7b7badf700 1 -- 172.21.15.189:0/2511254050 shutdown_connections 2024-04-06T12:31:47.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.161+0000 7f7b7badf700 1 -- 172.21.15.189:0/2511254050 wait complete. 2024-04-06T12:31:47.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.162+0000 7f7b7badf700 1 Processor -- start 2024-04-06T12:31:47.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.163+0000 7f7b7badf700 1 -- start start 2024-04-06T12:31:47.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.163+0000 7f7b7badf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b740cb870 0x7f7b74159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:47.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.163+0000 7f7b7badf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b740cfa00 0x7f7b740b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:47.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.163+0000 7f7b7badf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b740d11b0 0x7f7b740b5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:47.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.163+0000 7f7b7badf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b74155f80 con 0x7f7b740d11b0 2024-04-06T12:31:47.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.163+0000 7f7b7badf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b741561d0 con 0x7f7b740cfa00 2024-04-06T12:31:47.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.163+0000 7f7b7badf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b741564b0 con 0x7f7b740cb870 2024-04-06T12:31:47.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.163+0000 7f7b7a07c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b740d11b0 0x7f7b740b5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:47.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.163+0000 7f7b7a07c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b740d11b0 0x7f7b740b5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40700/0 (socket says 172.21.15.189:40700) 2024-04-06T12:31:47.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.163+0000 7f7b7a07c700 1 -- 172.21.15.189:0/2689022202 learned_addr learned my addr 172.21.15.189:0/2689022202 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:47.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.163+0000 7f7b7907a700 1 --2- 172.21.15.189:0/2689022202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b740cfa00 0x7f7b740b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:47.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.163+0000 7f7b7987b700 1 --2- 172.21.15.189:0/2689022202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b740cb870 0x7f7b74159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:47.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.164+0000 7f7b7a07c700 1 -- 172.21.15.189:0/2689022202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b740cb870 msgr2=0x7f7b74159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:47.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.164+0000 7f7b7a07c700 1 --2- 172.21.15.189:0/2689022202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b740cb870 0x7f7b74159770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.164+0000 7f7b7a07c700 1 -- 172.21.15.189:0/2689022202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b740cfa00 msgr2=0x7f7b740b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:47.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.164+0000 7f7b7a07c700 1 --2- 172.21.15.189:0/2689022202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b740cfa00 0x7f7b740b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.164+0000 7f7b7a07c700 1 -- 172.21.15.189:0/2689022202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c009a70 con 0x7f7b740d11b0 2024-04-06T12:31:47.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.164+0000 7f7b7907a700 1 --2- 172.21.15.189:0/2689022202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b740cfa00 0x7f7b740b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:47.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.164+0000 7f7b7a07c700 1 --2- 172.21.15.189:0/2689022202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b740d11b0 0x7f7b740b5ff0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00efc0 tx=0x7f7b6c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:47.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.164+0000 7f7b6a7fc700 1 -- 172.21.15.189:0/2689022202 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c00ca90 con 0x7f7b740d11b0 2024-04-06T12:31:47.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.164+0000 7f7b6a7fc700 1 -- 172.21.15.189:0/2689022202 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c00f040 con 0x7f7b740d11b0 2024-04-06T12:31:47.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.164+0000 7f7b7badf700 1 -- 172.21.15.189:0/2689022202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b640094a0 con 0x7f7b740d11b0 2024-04-06T12:31:47.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.164+0000 7f7b6a7fc700 1 -- 172.21.15.189:0/2689022202 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c00ce10 con 0x7f7b740d11b0 2024-04-06T12:31:47.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.164+0000 7f7b7badf700 1 -- 172.21.15.189:0/2689022202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b74156a70 con 0x7f7b740d11b0 2024-04-06T12:31:47.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.165+0000 7f7b7badf700 1 -- 172.21.15.189:0/2689022202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b74074990 con 0x7f7b740d11b0 2024-04-06T12:31:47.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.166+0000 7f7b6a7fc700 1 -- 172.21.15.189:0/2689022202 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 82) v1 ==== 89304+0+0 (secure 0 0 0) 0x7f7b6c003a20 con 0x7f7b740d11b0 2024-04-06T12:31:47.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.166+0000 7f7b6a7fc700 1 --2- 172.21.15.189:0/2689022202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7b5806ae40 0x7f7b5806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:47.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.167+0000 7f7b7987b700 1 -- 172.21.15.189:0/2689022202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7b5806ae40 msgr2=0x7f7b5806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:47.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.167+0000 7f7b6a7fc700 1 -- 172.21.15.189:0/2689022202 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f7b6c08a9f0 con 0x7f7b740d11b0 2024-04-06T12:31:47.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.167+0000 7f7b7987b700 1 --2- 172.21.15.189:0/2689022202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7b5806ae40 0x7f7b5806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:47.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.171+0000 7f7b6a7fc700 1 -- 172.21.15.189:0/2689022202 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7b6c055690 con 0x7f7b740d11b0 2024-04-06T12:31:47.312 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:47.315+0000 7f75c0c26200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:31:47.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.368+0000 7f7b7987b700 1 -- 172.21.15.189:0/2689022202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7b5806ae40 msgr2=0x7f7b5806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:47.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.368+0000 7f7b7987b700 1 --2- 172.21.15.189:0/2689022202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7b5806ae40 0x7f7b5806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:47.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.371+0000 7f7b7badf700 1 -- 172.21.15.189:0/2689022202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7b740cb320 con 0x7f7b740d11b0 2024-04-06T12:31:47.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.371+0000 7f7b6a7fc700 1 -- 172.21.15.189:0/2689022202 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v145) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7b6c009d70 con 0x7f7b740d11b0 2024-04-06T12:31:47.369 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T12:31:47.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.373+0000 7f7b7badf700 1 -- 172.21.15.189:0/2689022202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7b5806ae40 msgr2=0x7f7b5806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:47.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.373+0000 7f7b7badf700 1 --2- 172.21.15.189:0/2689022202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7b5806ae40 0x7f7b5806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.373+0000 7f7b7badf700 1 -- 172.21.15.189:0/2689022202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b740d11b0 msgr2=0x7f7b740b5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:47.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.373+0000 7f7b7badf700 1 --2- 172.21.15.189:0/2689022202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b740d11b0 0x7f7b740b5ff0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00efc0 tx=0x7f7b6c00c3a0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.374+0000 7f7b7badf700 1 -- 172.21.15.189:0/2689022202 shutdown_connections 2024-04-06T12:31:47.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.374+0000 7f7b7badf700 1 --2- 172.21.15.189:0/2689022202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b740d11b0 0x7f7b740b5ff0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.374+0000 7f7b7badf700 1 --2- 172.21.15.189:0/2689022202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b740cb870 0x7f7b74159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.374+0000 7f7b7badf700 1 --2- 172.21.15.189:0/2689022202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f7b5806ae40 0x7f7b5806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.374+0000 7f7b7badf700 1 --2- 172.21.15.189:0/2689022202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b740cfa00 0x7f7b740b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.374+0000 7f7b7badf700 1 -- 172.21.15.189:0/2689022202 >> 172.21.15.189:0/2689022202 conn(0x7f7b740b2db0 msgr2=0x7f7b7415b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:47.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.374+0000 7f7b7badf700 1 -- 172.21.15.189:0/2689022202 shutdown_connections 2024-04-06T12:31:47.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.374+0000 7f7b7badf700 1 -- 172.21.15.189:0/2689022202 wait complete. 2024-04-06T12:31:47.383 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T12:31:47.466 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:47.469+0000 7f75c0c26200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:31:47.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.514+0000 7fd56cad8700 1 Processor -- start 2024-04-06T12:31:47.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.514+0000 7fd56cad8700 1 -- start start 2024-04-06T12:31:47.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.515+0000 7fd56cad8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd56815d590 0x7fd5680b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:47.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.515+0000 7fd56cad8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd5680b4ed0 0x7fd5680b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:47.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.515+0000 7fd56cad8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5680b6580 0x7fd5680b69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:47.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.515+0000 7fd56cad8700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5680b58c0 con 0x7fd56815d590 2024-04-06T12:31:47.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.515+0000 7fd56cad8700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5680b5cc0 con 0x7fd5680b4ed0 2024-04-06T12:31:47.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.515+0000 7fd56cad8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5680d1500 con 0x7fd5680b6580 2024-04-06T12:31:47.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.515+0000 7fd56659c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd56815d590 0x7fd5680b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:47.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.515+0000 7fd56659c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd56815d590 0x7fd5680b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40716/0 (socket says 172.21.15.189:40716) 2024-04-06T12:31:47.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.515+0000 7fd56659c700 1 -- 172.21.15.189:0/2755345535 learned_addr learned my addr 172.21.15.189:0/2755345535 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:47.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.515+0000 7fd566d9d700 1 --2- 172.21.15.189:0/2755345535 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5680b6580 0x7fd5680b69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:47.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.515+0000 7fd565d9b700 1 --2- 172.21.15.189:0/2755345535 >> v2:172.21.15.192:3300/0 conn(0x7fd5680b4ed0 0x7fd5680b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:47.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.515+0000 7fd56659c700 1 -- 172.21.15.189:0/2755345535 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5680b6580 msgr2=0x7fd5680b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:47.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.515+0000 7fd56659c700 1 --2- 172.21.15.189:0/2755345535 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5680b6580 0x7fd5680b69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.515+0000 7fd56659c700 1 -- 172.21.15.189:0/2755345535 >> v2:172.21.15.192:3300/0 conn(0x7fd5680b4ed0 msgr2=0x7fd5680b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:47.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.515+0000 7fd56659c700 1 --2- 172.21.15.189:0/2755345535 >> v2:172.21.15.192:3300/0 conn(0x7fd5680b4ed0 0x7fd5680b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.515+0000 7fd56659c700 1 -- 172.21.15.189:0/2755345535 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5680d17e0 con 0x7fd56815d590 2024-04-06T12:31:47.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.516+0000 7fd56659c700 1 --2- 172.21.15.189:0/2755345535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd56815d590 0x7fd5680b8d70 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fd5680b6330 tx=0x7fd558009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a45c0b5e9c78545 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:47.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.516+0000 7fd56559a700 1 -- 172.21.15.189:0/2755345535 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd558011830 con 0x7fd56815d590 2024-04-06T12:31:47.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.516+0000 7fd56559a700 1 -- 172.21.15.189:0/2755345535 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd558011990 con 0x7fd56815d590 2024-04-06T12:31:47.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.516+0000 7fd56559a700 1 -- 172.21.15.189:0/2755345535 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd558011d20 con 0x7fd56815d590 2024-04-06T12:31:47.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.516+0000 7fd56cad8700 1 -- 172.21.15.189:0/2755345535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd56815d590 msgr2=0x7fd5680b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:47.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.516+0000 7fd56cad8700 1 --2- 172.21.15.189:0/2755345535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd56815d590 0x7fd5680b8d70 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fd5680b6330 tx=0x7fd558009bd0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.516+0000 7fd56cad8700 1 -- 172.21.15.189:0/2755345535 shutdown_connections 2024-04-06T12:31:47.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.516+0000 7fd56cad8700 1 --2- 172.21.15.189:0/2755345535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd56815d590 0x7fd5680b8d70 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.516+0000 7fd56cad8700 1 --2- 172.21.15.189:0/2755345535 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5680b6580 0x7fd5680b69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.516+0000 7fd56cad8700 1 --2- 172.21.15.189:0/2755345535 >> v2:172.21.15.192:3300/0 conn(0x7fd5680b4ed0 0x7fd5680b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.516+0000 7fd56cad8700 1 -- 172.21.15.189:0/2755345535 >> 172.21.15.189:0/2755345535 conn(0x7fd56806bbc0 msgr2=0x7fd56806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:47.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.517+0000 7fd56cad8700 1 -- 172.21.15.189:0/2755345535 shutdown_connections 2024-04-06T12:31:47.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.517+0000 7fd56cad8700 1 -- 172.21.15.189:0/2755345535 wait complete. 2024-04-06T12:31:47.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.518+0000 7fd56cad8700 1 Processor -- start 2024-04-06T12:31:47.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.518+0000 7fd56cad8700 1 -- start start 2024-04-06T12:31:47.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd56cad8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5680b4ed0 0x7fd5680c9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:47.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd56cad8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5680b6580 0x7fd5680cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:47.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd56cad8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd56815d590 0x7fd5680cc930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:47.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd56cad8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd568151220 con 0x7fd5680b6580 2024-04-06T12:31:47.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd56cad8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5680ca990 con 0x7fd5680b4ed0 2024-04-06T12:31:47.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd56cad8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5680cac70 con 0x7fd56815d590 2024-04-06T12:31:47.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd565d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5680b6580 0x7fd5680cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:47.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd565d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5680b6580 0x7fd5680cba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40720/0 (socket says 172.21.15.189:40720) 2024-04-06T12:31:47.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd565d9b700 1 -- 172.21.15.189:0/2574165605 learned_addr learned my addr 172.21.15.189:0/2574165605 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:47.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd56659c700 1 --2- 172.21.15.189:0/2574165605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5680b4ed0 0x7fd5680c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:47.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd566d9d700 1 --2- 172.21.15.189:0/2574165605 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd56815d590 0x7fd5680cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:47.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd565d9b700 1 -- 172.21.15.189:0/2574165605 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd56815d590 msgr2=0x7fd5680cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:47.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd565d9b700 1 --2- 172.21.15.189:0/2574165605 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd56815d590 0x7fd5680cc930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd565d9b700 1 -- 172.21.15.189:0/2574165605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5680b4ed0 msgr2=0x7fd5680c9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:47.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd565d9b700 1 --2- 172.21.15.189:0/2574165605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5680b4ed0 0x7fd5680c9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd565d9b700 1 -- 172.21.15.189:0/2574165605 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd558009550 con 0x7fd5680b6580 2024-04-06T12:31:47.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd566d9d700 1 --2- 172.21.15.189:0/2574165605 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd56815d590 0x7fd5680cc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:47.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd565d9b700 1 --2- 172.21.15.189:0/2574165605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5680b6580 0x7fd5680cba30 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fd55c00e930 tx=0x7fd55c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:47.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd556ffd700 1 -- 172.21.15.189:0/2574165605 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c00ee80 con 0x7fd5680b6580 2024-04-06T12:31:47.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd556ffd700 1 -- 172.21.15.189:0/2574165605 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55c00c850 con 0x7fd5680b6580 2024-04-06T12:31:47.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd56cad8700 1 -- 172.21.15.189:0/2574165605 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5680caf50 con 0x7fd5680b6580 2024-04-06T12:31:47.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd556ffd700 1 -- 172.21.15.189:0/2574165605 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c00cbe0 con 0x7fd5680b6580 2024-04-06T12:31:47.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.519+0000 7fd56cad8700 1 -- 172.21.15.189:0/2574165605 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5680cb510 con 0x7fd5680b6580 2024-04-06T12:31:47.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.520+0000 7fd56cad8700 1 -- 172.21.15.189:0/2574165605 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5680cb730 con 0x7fd5680b6580 2024-04-06T12:31:47.520 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:47.521+0000 7f4b58d91200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:31:47.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.521+0000 7fd556ffd700 1 -- 172.21.15.189:0/2574165605 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 82) v1 ==== 89304+0+0 (secure 0 0 0) 0x7fd55c00cd40 con 0x7fd5680b6580 2024-04-06T12:31:47.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.521+0000 7fd556ffd700 1 --2- 172.21.15.189:0/2574165605 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd54806ae40 0x7fd54806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:47.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.521+0000 7fd56659c700 1 -- 172.21.15.189:0/2574165605 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd54806ae40 msgr2=0x7fd54806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:47.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.522+0000 7fd556ffd700 1 -- 172.21.15.189:0/2574165605 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fd55c089a60 con 0x7fd5680b6580 2024-04-06T12:31:47.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.522+0000 7fd56659c700 1 --2- 172.21.15.189:0/2574165605 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd54806ae40 0x7fd54806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:47.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.526+0000 7fd556ffd700 1 -- 172.21.15.189:0/2574165605 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd55c0545f0 con 0x7fd5680b6580 2024-04-06T12:31:47.621 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:47.623+0000 7f75c0c26200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:31:47.683 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:47.685+0000 7f4b58d91200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:31:47.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.723+0000 7fd56659c700 1 -- 172.21.15.189:0/2574165605 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd54806ae40 msgr2=0x7fd54806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:47.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.723+0000 7fd56659c700 1 --2- 172.21.15.189:0/2574165605 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd54806ae40 0x7fd54806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:47.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.730+0000 7fd56cad8700 1 -- 172.21.15.189:0/2574165605 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd568074990 con 0x7fd5680b6580 2024-04-06T12:31:47.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.731+0000 7fd556ffd700 1 -- 172.21.15.189:0/2574165605 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v145) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd55c021020 con 0x7fd5680b6580 2024-04-06T12:31:47.729 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T12:31:47.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.733+0000 7fd56cad8700 1 -- 172.21.15.189:0/2574165605 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd54806ae40 msgr2=0x7fd54806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:47.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.733+0000 7fd56cad8700 1 --2- 172.21.15.189:0/2574165605 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd54806ae40 0x7fd54806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.733+0000 7fd56cad8700 1 -- 172.21.15.189:0/2574165605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5680b6580 msgr2=0x7fd5680cba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:47.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.733+0000 7fd56cad8700 1 --2- 172.21.15.189:0/2574165605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5680b6580 0x7fd5680cba30 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fd55c00e930 tx=0x7fd55c00ec40 comp rx=0 tx=0).stop 2024-04-06T12:31:47.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.733+0000 7fd56cad8700 1 -- 172.21.15.189:0/2574165605 shutdown_connections 2024-04-06T12:31:47.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.733+0000 7fd56cad8700 1 --2- 172.21.15.189:0/2574165605 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd56815d590 0x7fd5680cc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.733+0000 7fd56cad8700 1 --2- 172.21.15.189:0/2574165605 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd54806ae40 0x7fd54806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.733+0000 7fd56cad8700 1 --2- 172.21.15.189:0/2574165605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5680b4ed0 0x7fd5680c9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.733+0000 7fd56cad8700 1 --2- 172.21.15.189:0/2574165605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5680b6580 0x7fd5680cba30 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.733+0000 7fd56cad8700 1 -- 172.21.15.189:0/2574165605 >> 172.21.15.189:0/2574165605 conn(0x7fd56806bbc0 msgr2=0x7fd56806bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:47.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.733+0000 7fd56cad8700 1 -- 172.21.15.189:0/2574165605 shutdown_connections 2024-04-06T12:31:47.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.733+0000 7fd56cad8700 1 -- 172.21.15.189:0/2574165605 wait complete. 2024-04-06T12:31:47.744 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:31:47.853 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:47.856+0000 7f4b58d91200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:31:47.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.873+0000 7f9f7540c700 1 Processor -- start 2024-04-06T12:31:47.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.873+0000 7f9f7540c700 1 -- start start 2024-04-06T12:31:47.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.874+0000 7f9f7540c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9f7015d590 0x7f9f700b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:47.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.874+0000 7f9f7540c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f700b5cc0 con 0x7f9f700d3890 2024-04-06T12:31:47.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.874+0000 7f9f7540c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f700d1500 con 0x7f9f700b4ed0 2024-04-06T12:31:47.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.874+0000 7f9f7540c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f700d17e0 con 0x7f9f7015d590 2024-04-06T12:31:47.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.874+0000 7f9f6effd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9f7015d590 0x7f9f700b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:47.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.874+0000 7f9f6f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9f700d3890 0x7f9f700b6c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49182/0 (socket says 172.21.15.189:49182) 2024-04-06T12:31:47.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.874+0000 7f9f6f7fe700 1 -- 172.21.15.189:0/2174774810 learned_addr learned my addr 172.21.15.189:0/2174774810 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:47.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.874+0000 7f9f6effd700 1 -- 172.21.15.189:0/2174774810 >> v1:172.21.15.192:6789/0 conn(0x7f9f700b4ed0 legacy=0x7f9f700b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:47.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.874+0000 7f9f6effd700 1 -- 172.21.15.189:0/2174774810 >> v1:172.21.15.189:6789/0 conn(0x7f9f700d3890 legacy=0x7f9f700b6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:47.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.874+0000 7f9f6effd700 1 -- 172.21.15.189:0/2174774810 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f700d1ac0 con 0x7f9f7015d590 2024-04-06T12:31:47.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.874+0000 7f9f6dffb700 1 -- 172.21.15.189:0/2174774810 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905917246 0 0) 0x7f9f700b5cc0 con 0x7f9f700d3890 2024-04-06T12:31:47.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.875+0000 7f9f6effd700 1 --2- 172.21.15.189:0/2174774810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f7015d590 0x7f9f700b8d70 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009840 tx=0x7f9f60009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=742c91c06379628e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:47.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.875+0000 7f9f6dffb700 1 -- 172.21.15.189:0/2174774810 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f60011830 con 0x7f9f7015d590 2024-04-06T12:31:47.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.875+0000 7f9f6dffb700 1 -- 172.21.15.189:0/2174774810 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f60011990 con 0x7f9f7015d590 2024-04-06T12:31:47.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.875+0000 7f9f6dffb700 1 -- 172.21.15.189:0/2174774810 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f60011d20 con 0x7f9f7015d590 2024-04-06T12:31:47.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.876+0000 7f9f7540c700 1 -- 172.21.15.189:0/2174774810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f7015d590 msgr2=0x7f9f700b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:47.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.876+0000 7f9f7540c700 1 --2- 172.21.15.189:0/2174774810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f7015d590 0x7f9f700b8d70 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009840 tx=0x7f9f60009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:47.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.876+0000 7f9f7540c700 1 -- 172.21.15.189:0/2174774810 shutdown_connections 2024-04-06T12:31:47.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.876+0000 7f9f7540c700 1 --2- 172.21.15.189:0/2174774810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f7015d590 0x7f9f700b8d70 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.876+0000 7f9f7540c700 1 -- 172.21.15.189:0/2174774810 >> 172.21.15.189:0/2174774810 conn(0x7f9f7006bbc0 msgr2=0x7f9f7006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:47.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.876+0000 7f9f7540c700 1 -- 172.21.15.189:0/2174774810 shutdown_connections 2024-04-06T12:31:47.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.876+0000 7f9f7540c700 1 -- 172.21.15.189:0/2174774810 wait complete. 2024-04-06T12:31:47.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.877+0000 7f9f7540c700 1 Processor -- start 2024-04-06T12:31:47.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.877+0000 7f9f7540c700 1 -- start start 2024-04-06T12:31:47.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.878+0000 7f9f7540c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f700b4ed0 0x7f9f7015b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:47.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.878+0000 7f9f7540c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f700d3890 0x7f9f7015e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:47.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.878+0000 7f9f7540c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f7015d590 0x7f9f700c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:47.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.878+0000 7f9f7540c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f701f6c00 con 0x7f9f700b4ed0 2024-04-06T12:31:47.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.878+0000 7f9f7540c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7015edf0 con 0x7f9f7015d590 2024-04-06T12:31:47.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.878+0000 7f9f7540c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f701f6540 con 0x7f9f700d3890 2024-04-06T12:31:47.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.878+0000 7f9f6effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f700b4ed0 0x7f9f7015b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:47.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.878+0000 7f9f6effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f700b4ed0 0x7f9f7015b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40740/0 (socket says 172.21.15.189:40740) 2024-04-06T12:31:47.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.878+0000 7f9f6effd700 1 -- 172.21.15.189:0/1455161426 learned_addr learned my addr 172.21.15.189:0/1455161426 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.878+0000 7f9f6f7fe700 1 --2- 172.21.15.189:0/1455161426 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f7015d590 0x7f9f700c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.878+0000 7f9f6e7fc700 1 --2- 172.21.15.189:0/1455161426 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f700d3890 0x7f9f7015e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.878+0000 7f9f6effd700 1 -- 172.21.15.189:0/1455161426 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f700d3890 msgr2=0x7f9f7015e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.878+0000 7f9f6effd700 1 --2- 172.21.15.189:0/1455161426 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f700d3890 0x7f9f7015e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.878+0000 7f9f6effd700 1 -- 172.21.15.189:0/1455161426 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f7015d590 msgr2=0x7f9f700c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.878+0000 7f9f6effd700 1 --2- 172.21.15.189:0/1455161426 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f7015d590 0x7f9f700c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.878+0000 7f9f6effd700 1 -- 172.21.15.189:0/1455161426 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f600094a0 con 0x7f9f700b4ed0 2024-04-06T12:31:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.878+0000 7f9f6effd700 1 --2- 172.21.15.189:0/1455161426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f700b4ed0 0x7f9f7015b880 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f9f60014040 tx=0x7f9f60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.878+0000 7f9f577fe700 1 -- 172.21.15.189:0/1455161426 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6001a7e0 con 0x7f9f700b4ed0 2024-04-06T12:31:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.879+0000 7f9f577fe700 1 -- 172.21.15.189:0/1455161426 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6001a940 con 0x7f9f700b4ed0 2024-04-06T12:31:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.879+0000 7f9f7540c700 1 -- 172.21.15.189:0/1455161426 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f701f6820 con 0x7f9f700b4ed0 2024-04-06T12:31:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.879+0000 7f9f577fe700 1 -- 172.21.15.189:0/1455161426 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6001acb0 con 0x7f9f700b4ed0 2024-04-06T12:31:47.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.879+0000 7f9f7540c700 1 -- 172.21.15.189:0/1455161426 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f7015bdc0 con 0x7f9f700b4ed0 2024-04-06T12:31:47.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.880+0000 7f9f7540c700 1 -- 172.21.15.189:0/1455161426 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f7015bfb0 con 0x7f9f700b4ed0 2024-04-06T12:31:47.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.880+0000 7f9f577fe700 1 -- 172.21.15.189:0/1455161426 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 82) v1 ==== 89304+0+0 (secure 0 0 0) 0x7f9f60022070 con 0x7f9f700b4ed0 2024-04-06T12:31:47.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.880+0000 7f9f577fe700 1 --2- 172.21.15.189:0/1455161426 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9f5c06ae40 0x7f9f5c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:47.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.881+0000 7f9f6e7fc700 1 -- 172.21.15.189:0/1455161426 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9f5c06ae40 msgr2=0x7f9f5c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:47.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.881+0000 7f9f577fe700 1 -- 172.21.15.189:0/1455161426 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f9f6008aa20 con 0x7f9f700b4ed0 2024-04-06T12:31:47.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.881+0000 7f9f6e7fc700 1 --2- 172.21.15.189:0/1455161426 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9f5c06ae40 0x7f9f5c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:47.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:47.885+0000 7f9f577fe700 1 -- 172.21.15.189:0/1455161426 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9f60055770 con 0x7f9f700b4ed0 2024-04-06T12:31:48.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.082+0000 7f9f6e7fc700 1 -- 172.21.15.189:0/1455161426 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9f5c06ae40 msgr2=0x7f9f5c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:48.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.082+0000 7f9f6e7fc700 1 --2- 172.21.15.189:0/1455161426 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9f5c06ae40 0x7f9f5c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:48.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.133+0000 7f9f7540c700 1 -- 172.21.15.189:0/1455161426 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f701f6d40 con 0x7f9f700b4ed0 2024-04-06T12:31:48.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.133+0000 7f9f577fe700 1 -- 172.21.15.189:0/1455161426 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f9f701f6d40 con 0x7f9f700b4ed0 2024-04-06T12:31:48.130 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:48.130 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7115,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/538134862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":538134862},{"type":"v1","addr":"172.21.15.192:6834","nonce":538134862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7915,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1788966297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1788966297},{"type":"v1","addr":"172.21.15.189:6837","nonce":1788966297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7930,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2932927074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2932927074},{"type":"v1","addr":"172.21.15.192:6835","nonce":2932927074}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:31:22.083923+0000","modified":"2024-04-06T12:31:25.104510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/4249639368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":4249639368},{"type":"v1","addr":"172.21.15.189:6836","nonce":4249639368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-06T12:31:48.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.135+0000 7f9f7540c700 1 -- 172.21.15.189:0/1455161426 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9f5c06ae40 msgr2=0x7f9f5c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:48.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.135+0000 7f9f7540c700 1 --2- 172.21.15.189:0/1455161426 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9f5c06ae40 0x7f9f5c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.135+0000 7f9f7540c700 1 -- 172.21.15.189:0/1455161426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f700b4ed0 msgr2=0x7f9f7015b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:48.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.135+0000 7f9f7540c700 1 --2- 172.21.15.189:0/1455161426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f700b4ed0 0x7f9f7015b880 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f9f60014040 tx=0x7f9f60004750 comp rx=0 tx=0).stop 2024-04-06T12:31:48.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.135+0000 7f9f7540c700 1 -- 172.21.15.189:0/1455161426 shutdown_connections 2024-04-06T12:31:48.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.135+0000 7f9f7540c700 1 --2- 172.21.15.189:0/1455161426 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9f5c06ae40 0x7f9f5c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.135+0000 7f9f7540c700 1 --2- 172.21.15.189:0/1455161426 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f7015d590 0x7f9f700c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.135+0000 7f9f7540c700 1 --2- 172.21.15.189:0/1455161426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f700b4ed0 0x7f9f7015b880 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.135+0000 7f9f7540c700 1 --2- 172.21.15.189:0/1455161426 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f700d3890 0x7f9f7015e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.135+0000 7f9f7540c700 1 -- 172.21.15.189:0/1455161426 >> 172.21.15.189:0/1455161426 conn(0x7f9f7006bbc0 msgr2=0x7f9f700ce3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:48.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.135+0000 7f9f7540c700 1 -- 172.21.15.189:0/1455161426 shutdown_connections 2024-04-06T12:31:48.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.135+0000 7f9f7540c700 1 -- 172.21.15.189:0/1455161426 wait complete. 2024-04-06T12:31:48.134 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 62 2024-04-06T12:31:48.145 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:31:48.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.271+0000 7f6f68dc6700 1 Processor -- start 2024-04-06T12:31:48.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.271+0000 7f6f68dc6700 1 -- start start 2024-04-06T12:31:48.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.272+0000 7f6f68dc6700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6f6415d7b0 0x7f6f640b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:48.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.272+0000 7f6f68dc6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f6415e500 0x7f6f6406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:48.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.272+0000 7f6f68dc6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f640b3990 con 0x7f6f6406a730 2024-04-06T12:31:48.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.272+0000 7f6f68dc6700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f6406b0f0 con 0x7f6f6415d7b0 2024-04-06T12:31:48.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.272+0000 7f6f68dc6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f640b3060 con 0x7f6f6415e500 2024-04-06T12:31:48.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.272+0000 7f6f61d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6f6406a730 0x7f6f6415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49196/0 (socket says 172.21.15.189:49196) 2024-04-06T12:31:48.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.272+0000 7f6f61d9b700 1 -- 172.21.15.189:0/4009410380 learned_addr learned my addr 172.21.15.189:0/4009410380 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:48.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.272+0000 7f6f6259c700 1 --2- 172.21.15.189:0/4009410380 >> v2:172.21.15.192:3300/0 conn(0x7f6f6415d7b0 0x7f6f640b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:48.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.273+0000 7f6f62d9d700 1 --2- 172.21.15.189:0/4009410380 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f6415e500 0x7f6f6406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:48.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.273+0000 7f6f6159a700 1 -- 172.21.15.189:0/4009410380 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434445606 0 0) 0x7f6f640b3990 con 0x7f6f6406a730 2024-04-06T12:31:48.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.273+0000 7f6f6159a700 1 -- 172.21.15.189:0/4009410380 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f4c0035b0 con 0x7f6f6406a730 2024-04-06T12:31:48.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.273+0000 7f6f6259c700 1 -- 172.21.15.189:0/4009410380 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f6415e500 msgr2=0x7f6f6406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:48.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.273+0000 7f6f6259c700 1 --2- 172.21.15.189:0/4009410380 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f6415e500 0x7f6f6406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.273+0000 7f6f6259c700 1 -- 172.21.15.189:0/4009410380 >> v1:172.21.15.189:6789/0 conn(0x7f6f6406a730 legacy=0x7f6f6415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:48.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.273+0000 7f6f6259c700 1 -- 172.21.15.189:0/4009410380 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f640b3340 con 0x7f6f6415d7b0 2024-04-06T12:31:48.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.273+0000 7f6f62d9d700 1 --2- 172.21.15.189:0/4009410380 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f6415e500 0x7f6f6406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:48.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.273+0000 7f6f6259c700 1 --2- 172.21.15.189:0/4009410380 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f6415d7b0 0x7f6f640b3e20 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009840 tx=0x7f6f54009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=147d0061e35498ff server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:48.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.273+0000 7f6f6159a700 1 -- 172.21.15.189:0/4009410380 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f54011830 con 0x7f6f6415d7b0 2024-04-06T12:31:48.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.273+0000 7f6f6159a700 1 -- 172.21.15.189:0/4009410380 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f54011990 con 0x7f6f6415d7b0 2024-04-06T12:31:48.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.273+0000 7f6f6159a700 1 -- 172.21.15.189:0/4009410380 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f54011d20 con 0x7f6f6415d7b0 2024-04-06T12:31:48.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.274+0000 7f6f68dc6700 1 -- 172.21.15.189:0/4009410380 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f6415d7b0 msgr2=0x7f6f640b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:48.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.274+0000 7f6f68dc6700 1 --2- 172.21.15.189:0/4009410380 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f6415d7b0 0x7f6f640b3e20 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009840 tx=0x7f6f54009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:48.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.274+0000 7f6f68dc6700 1 -- 172.21.15.189:0/4009410380 shutdown_connections 2024-04-06T12:31:48.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.274+0000 7f6f68dc6700 1 --2- 172.21.15.189:0/4009410380 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f6415e500 0x7f6f6406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.274+0000 7f6f68dc6700 1 --2- 172.21.15.189:0/4009410380 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f6415d7b0 0x7f6f640b3e20 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.274+0000 7f6f68dc6700 1 -- 172.21.15.189:0/4009410380 >> 172.21.15.189:0/4009410380 conn(0x7f6f640d00b0 msgr2=0x7f6f640d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:48.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.274+0000 7f6f68dc6700 1 -- 172.21.15.189:0/4009410380 shutdown_connections 2024-04-06T12:31:48.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.274+0000 7f6f68dc6700 1 -- 172.21.15.189:0/4009410380 wait complete. 2024-04-06T12:31:48.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.275+0000 7f6f68dc6700 1 Processor -- start 2024-04-06T12:31:48.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.275+0000 7f6f68dc6700 1 -- start start 2024-04-06T12:31:48.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.276+0000 7f6f68dc6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f6406a730 0x7f6f6414b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:48.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.276+0000 7f6f68dc6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f6415d7b0 0x7f6f641f6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:48.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.276+0000 7f6f68dc6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f6415e500 0x7f6f641f26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:48.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.276+0000 7f6f68dc6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6414c740 con 0x7f6f6415d7b0 2024-04-06T12:31:48.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.276+0000 7f6f68dc6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f64149940 con 0x7f6f6415e500 2024-04-06T12:31:48.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.276+0000 7f6f68dc6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f641f7350 con 0x7f6f6406a730 2024-04-06T12:31:48.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.276+0000 7f6f61d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f6415d7b0 0x7f6f641f6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:48.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.276+0000 7f6f61d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f6415d7b0 0x7f6f641f6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40744/0 (socket says 172.21.15.189:40744) 2024-04-06T12:31:48.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.276+0000 7f6f61d9b700 1 -- 172.21.15.189:0/3560379237 learned_addr learned my addr 172.21.15.189:0/3560379237 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:48.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.276+0000 7f6f6259c700 1 --2- 172.21.15.189:0/3560379237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f6406a730 0x7f6f6414b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:48.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.276+0000 7f6f62d9d700 1 --2- 172.21.15.189:0/3560379237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f6415e500 0x7f6f641f26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:48.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.277+0000 7f6f61d9b700 1 -- 172.21.15.189:0/3560379237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f6406a730 msgr2=0x7f6f6414b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:48.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.277+0000 7f6f61d9b700 1 --2- 172.21.15.189:0/3560379237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f6406a730 0x7f6f6414b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.277+0000 7f6f61d9b700 1 -- 172.21.15.189:0/3560379237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f6415e500 msgr2=0x7f6f641f26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:48.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.277+0000 7f6f61d9b700 1 --2- 172.21.15.189:0/3560379237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f6415e500 0x7f6f641f26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.277+0000 7f6f61d9b700 1 -- 172.21.15.189:0/3560379237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5c009a70 con 0x7f6f6415d7b0 2024-04-06T12:31:48.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.277+0000 7f6f61d9b700 1 --2- 172.21.15.189:0/3560379237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f6415d7b0 0x7f6f641f6e10 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f6f5800c730 tx=0x7f6f5800ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:48.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.277+0000 7f6f52ffd700 1 -- 172.21.15.189:0/3560379237 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f58005d70 con 0x7f6f6415d7b0 2024-04-06T12:31:48.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.277+0000 7f6f52ffd700 1 -- 172.21.15.189:0/3560379237 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f580070d0 con 0x7f6f6415d7b0 2024-04-06T12:31:48.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.277+0000 7f6f52ffd700 1 -- 172.21.15.189:0/3560379237 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f58007460 con 0x7f6f6415d7b0 2024-04-06T12:31:48.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.277+0000 7f6f68dc6700 1 -- 172.21.15.189:0/3560379237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f540094a0 con 0x7f6f6415d7b0 2024-04-06T12:31:48.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.277+0000 7f6f68dc6700 1 -- 172.21.15.189:0/3560379237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f6414c180 con 0x7f6f6415d7b0 2024-04-06T12:31:48.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.278+0000 7f6f68dc6700 1 -- 172.21.15.189:0/3560379237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f64067bb0 con 0x7f6f6415d7b0 2024-04-06T12:31:48.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.279+0000 7f6f52ffd700 1 -- 172.21.15.189:0/3560379237 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 82) v1 ==== 89304+0+0 (secure 0 0 0) 0x7f6f580075c0 con 0x7f6f6415d7b0 2024-04-06T12:31:48.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.279+0000 7f6f52ffd700 1 --2- 172.21.15.189:0/3560379237 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f4406ae40 0x7f6f4406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:48.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.279+0000 7f6f6259c700 1 -- 172.21.15.189:0/3560379237 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f4406ae40 msgr2=0x7f6f4406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:48.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.279+0000 7f6f52ffd700 1 -- 172.21.15.189:0/3560379237 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f6f58088d80 con 0x7f6f6415d7b0 2024-04-06T12:31:48.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.279+0000 7f6f6259c700 1 --2- 172.21.15.189:0/3560379237 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f4406ae40 0x7f6f4406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:48.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.283+0000 7f6f52ffd700 1 -- 172.21.15.189:0/3560379237 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6f58053a50 con 0x7f6f6415d7b0 2024-04-06T12:31:48.344 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:48.347+0000 7f75c0c26200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:31:48.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.480+0000 7f6f6259c700 1 -- 172.21.15.189:0/3560379237 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f4406ae40 msgr2=0x7f6f4406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:48.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.480+0000 7f6f6259c700 1 --2- 172.21.15.189:0/3560379237 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f4406ae40 0x7f6f4406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:48.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.485+0000 7f6f68dc6700 1 -- 172.21.15.189:0/3560379237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f64074990 con 0x7f6f6415d7b0 2024-04-06T12:31:48.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.486+0000 7f6f52ffd700 1 -- 172.21.15.189:0/3560379237 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+16862 (secure 0 0 0) 0x7f6f58057070 con 0x7f6f6415d7b0 2024-04-06T12:31:48.484 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:48.484 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":145,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:31:36.068439+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:31:19.866331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-06T12:31:21.048344+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/2323784940":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/2597819342":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/3011108552":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/814932379":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/2994336395":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/2720887145":"2024-04-07T12:31:36.068397+0000","172.21.15.189:6833/47128":"2024-04-07T12:31:36.068397+0000","172.21.15.189:6832/47128":"2024-04-07T12:31:36.068397+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:31:48.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.489+0000 7f6f68dc6700 1 -- 172.21.15.189:0/3560379237 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f4406ae40 msgr2=0x7f6f4406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:48.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.489+0000 7f6f68dc6700 1 --2- 172.21.15.189:0/3560379237 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f4406ae40 0x7f6f4406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.489+0000 7f6f68dc6700 1 -- 172.21.15.189:0/3560379237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f6415d7b0 msgr2=0x7f6f641f6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:48.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.489+0000 7f6f68dc6700 1 --2- 172.21.15.189:0/3560379237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f6415d7b0 0x7f6f641f6e10 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f6f5800c730 tx=0x7f6f5800ca40 comp rx=0 tx=0).stop 2024-04-06T12:31:48.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.489+0000 7f6f68dc6700 1 -- 172.21.15.189:0/3560379237 shutdown_connections 2024-04-06T12:31:48.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.489+0000 7f6f68dc6700 1 --2- 172.21.15.189:0/3560379237 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f4406ae40 0x7f6f4406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.489+0000 7f6f68dc6700 1 --2- 172.21.15.189:0/3560379237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f6415e500 0x7f6f641f26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.489+0000 7f6f68dc6700 1 --2- 172.21.15.189:0/3560379237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f6406a730 0x7f6f6414b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.489+0000 7f6f68dc6700 1 --2- 172.21.15.189:0/3560379237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f6415d7b0 0x7f6f641f6e10 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.489+0000 7f6f68dc6700 1 -- 172.21.15.189:0/3560379237 >> 172.21.15.189:0/3560379237 conn(0x7f6f640d00b0 msgr2=0x7f6f640ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:48.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.489+0000 7f6f68dc6700 1 -- 172.21.15.189:0/3560379237 shutdown_connections 2024-04-06T12:31:48.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.489+0000 7f6f68dc6700 1 -- 172.21.15.189:0/3560379237 wait complete. 2024-04-06T12:31:48.499 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-06T12:31:48.499 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:31:48.585 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:48.587+0000 7f75c0c26200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:31:48.619 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:48.622+0000 7f4b58d91200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:31:48.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.626+0000 7f828be4b700 1 Processor -- start 2024-04-06T12:31:48.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.626+0000 7f828be4b700 1 -- start start 2024-04-06T12:31:48.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.627+0000 7f828be4b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82840cb870 0x7f828415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:48.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.627+0000 7f828be4b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f82840cfa00 0x7f82840cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:48.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.627+0000 7f828be4b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82840cbd80 con 0x7f82840d11b0 2024-04-06T12:31:48.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.627+0000 7f828be4b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82840ccdc0 con 0x7f82840cfa00 2024-04-06T12:31:48.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.627+0000 7f828be4b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82840d0b50 con 0x7f82840cb870 2024-04-06T12:31:48.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.628+0000 7f8289be7700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f82840d11b0 0x7f82840d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49210/0 (socket says 172.21.15.189:49210) 2024-04-06T12:31:48.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.628+0000 7f8289be7700 1 -- 172.21.15.189:0/3658160053 learned_addr learned my addr 172.21.15.189:0/3658160053 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:48.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.628+0000 7f82893e6700 1 --2- 172.21.15.189:0/3658160053 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82840cb870 0x7f828415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:48.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.628+0000 7f828a3e8700 1 --2- 172.21.15.189:0/3658160053 >> v2:172.21.15.192:3300/0 conn(0x7f82840cfa00 0x7f82840cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:48.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.628+0000 7f8288be5700 1 -- 172.21.15.189:0/3658160053 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573924272 0 0) 0x7f82840cbd80 con 0x7f82840d11b0 2024-04-06T12:31:48.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.628+0000 7f8288be5700 1 -- 172.21.15.189:0/3658160053 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82700035b0 con 0x7f82840d11b0 2024-04-06T12:31:48.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.628+0000 7f82893e6700 1 -- 172.21.15.189:0/3658160053 >> v2:172.21.15.192:3300/0 conn(0x7f82840cfa00 msgr2=0x7f82840cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:48.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.628+0000 7f82893e6700 1 --2- 172.21.15.189:0/3658160053 >> v2:172.21.15.192:3300/0 conn(0x7f82840cfa00 0x7f82840cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.628+0000 7f82893e6700 1 -- 172.21.15.189:0/3658160053 >> v1:172.21.15.189:6789/0 conn(0x7f82840d11b0 legacy=0x7f82840d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:48.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.628+0000 7f82893e6700 1 -- 172.21.15.189:0/3658160053 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82840cef20 con 0x7f82840cb870 2024-04-06T12:31:48.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.628+0000 7f82893e6700 1 --2- 172.21.15.189:0/3658160053 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82840cb870 0x7f828415c280 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8274009840 tx=0x7f8274009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f106e6c9341b5a78 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:48.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.628+0000 7f8288be5700 1 -- 172.21.15.189:0/3658160053 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8274011830 con 0x7f82840cb870 2024-04-06T12:31:48.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.628+0000 7f8288be5700 1 -- 172.21.15.189:0/3658160053 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8274011990 con 0x7f82840cb870 2024-04-06T12:31:48.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.628+0000 7f8288be5700 1 -- 172.21.15.189:0/3658160053 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8274011d20 con 0x7f82840cb870 2024-04-06T12:31:48.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.629+0000 7f828be4b700 1 -- 172.21.15.189:0/3658160053 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82840cb870 msgr2=0x7f828415c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:48.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.629+0000 7f828be4b700 1 --2- 172.21.15.189:0/3658160053 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82840cb870 0x7f828415c280 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8274009840 tx=0x7f8274009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:48.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.629+0000 7f828be4b700 1 -- 172.21.15.189:0/3658160053 shutdown_connections 2024-04-06T12:31:48.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.629+0000 7f828be4b700 1 --2- 172.21.15.189:0/3658160053 >> v2:172.21.15.192:3300/0 conn(0x7f82840cfa00 0x7f82840cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.629+0000 7f828be4b700 1 --2- 172.21.15.189:0/3658160053 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82840cb870 0x7f828415c280 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.629+0000 7f828be4b700 1 -- 172.21.15.189:0/3658160053 >> 172.21.15.189:0/3658160053 conn(0x7f82840b2db0 msgr2=0x7f82840b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:48.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.629+0000 7f828be4b700 1 -- 172.21.15.189:0/3658160053 shutdown_connections 2024-04-06T12:31:48.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.629+0000 7f828be4b700 1 -- 172.21.15.189:0/3658160053 wait complete. 2024-04-06T12:31:48.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.630+0000 7f828be4b700 1 Processor -- start 2024-04-06T12:31:48.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.630+0000 7f828be4b700 1 -- start start 2024-04-06T12:31:48.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.631+0000 7f828be4b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82840cb870 0x7f8284155590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:48.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.631+0000 7f828be4b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82840cfa00 0x7f82840b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:48.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.631+0000 7f828be4b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82840d11b0 0x7f8284158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:48.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.631+0000 7f828be4b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828406b8e0 con 0x7f82840cfa00 2024-04-06T12:31:48.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.631+0000 7f828be4b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82840c8850 con 0x7f82840cb870 2024-04-06T12:31:48.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.631+0000 7f828be4b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82840b5a20 con 0x7f82840d11b0 2024-04-06T12:31:48.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.631+0000 7f82893e6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82840cfa00 0x7f82840b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:48.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.631+0000 7f82893e6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82840cfa00 0x7f82840b5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40746/0 (socket says 172.21.15.189:40746) 2024-04-06T12:31:48.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.631+0000 7f82893e6700 1 -- 172.21.15.189:0/378829612 learned_addr learned my addr 172.21.15.189:0/378829612 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:48.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.631+0000 7f828a3e8700 1 --2- 172.21.15.189:0/378829612 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82840d11b0 0x7f8284158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:48.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.631+0000 7f8289be7700 1 --2- 172.21.15.189:0/378829612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82840cb870 0x7f8284155590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:48.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.632+0000 7f82893e6700 1 -- 172.21.15.189:0/378829612 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82840d11b0 msgr2=0x7f8284158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:48.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.632+0000 7f82893e6700 1 --2- 172.21.15.189:0/378829612 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82840d11b0 0x7f8284158150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.632+0000 7f82893e6700 1 -- 172.21.15.189:0/378829612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82840cb870 msgr2=0x7f8284155590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:48.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.632+0000 7f82893e6700 1 --2- 172.21.15.189:0/378829612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82840cb870 0x7f8284155590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.632+0000 7f82893e6700 1 -- 172.21.15.189:0/378829612 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82740094a0 con 0x7f82840cfa00 2024-04-06T12:31:48.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.632+0000 7f828a3e8700 1 --2- 172.21.15.189:0/378829612 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82840d11b0 0x7f8284158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:48.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.632+0000 7f82893e6700 1 --2- 172.21.15.189:0/378829612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82840cfa00 0x7f82840b5040 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f8274000c00 tx=0x7f8274004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:48.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.632+0000 7f827a7fc700 1 -- 172.21.15.189:0/378829612 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8274004380 con 0x7f82840cfa00 2024-04-06T12:31:48.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.632+0000 7f827a7fc700 1 -- 172.21.15.189:0/378829612 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827401f070 con 0x7f82840cfa00 2024-04-06T12:31:48.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.632+0000 7f828be4b700 1 -- 172.21.15.189:0/378829612 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82840b5d00 con 0x7f82840cfa00 2024-04-06T12:31:48.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.632+0000 7f827a7fc700 1 -- 172.21.15.189:0/378829612 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82740036a0 con 0x7f82840cfa00 2024-04-06T12:31:48.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.632+0000 7f828be4b700 1 -- 172.21.15.189:0/378829612 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8284155ad0 con 0x7f82840cfa00 2024-04-06T12:31:48.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.633+0000 7f828be4b700 1 -- 172.21.15.189:0/378829612 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f828406bd90 con 0x7f82840cfa00 2024-04-06T12:31:48.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.633+0000 7f827a7fc700 1 -- 172.21.15.189:0/378829612 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 82) v1 ==== 89304+0+0 (secure 0 0 0) 0x7f8274021030 con 0x7f82840cfa00 2024-04-06T12:31:48.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.634+0000 7f827a7fc700 1 --2- 172.21.15.189:0/378829612 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f826806ae70 0x7f826806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:48.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.634+0000 7f8289be7700 1 -- 172.21.15.189:0/378829612 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f826806ae70 msgr2=0x7f826806d330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:48.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.634+0000 7f827a7fc700 1 -- 172.21.15.189:0/378829612 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f827408a950 con 0x7f82840cfa00 2024-04-06T12:31:48.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.634+0000 7f8289be7700 1 --2- 172.21.15.189:0/378829612 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f826806ae70 0x7f826806d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:48.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.642+0000 7f827a7fc700 1 -- 172.21.15.189:0/378829612 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f828406bd90 con 0x7f82840cfa00 2024-04-06T12:31:48.662 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:48.665+0000 7f75c0c26200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:31:48.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.835+0000 7f8289be7700 1 -- 172.21.15.189:0/378829612 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f826806ae70 msgr2=0x7f826806d330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:48.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.835+0000 7f8289be7700 1 --2- 172.21.15.189:0/378829612 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f826806ae70 0x7f826806d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:48.886 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:48.889+0000 7f4b58d91200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:31:48.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.899+0000 7f828be4b700 1 -- 172.21.15.189:0/378829612 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f828406bd90 con 0x7f82840cfa00 2024-04-06T12:31:48.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.900+0000 7f827a7fc700 1 -- 172.21.15.189:0/378829612 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f828406bd90 con 0x7f82840cfa00 2024-04-06T12:31:48.897 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:48.898 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7115,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/538134862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":538134862},{"type":"v1","addr":"172.21.15.192:6834","nonce":538134862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7915,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1788966297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1788966297},{"type":"v1","addr":"172.21.15.189:6837","nonce":1788966297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7930,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2932927074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2932927074},{"type":"v1","addr":"172.21.15.192:6835","nonce":2932927074}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:31:22.083923+0000","modified":"2024-04-06T12:31:25.104510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/4249639368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":4249639368},{"type":"v1","addr":"172.21.15.189:6836","nonce":4249639368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-06T12:31:48.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.903+0000 7f828be4b700 1 -- 172.21.15.189:0/378829612 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f826806ae70 msgr2=0x7f826806d330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:48.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.903+0000 7f828be4b700 1 --2- 172.21.15.189:0/378829612 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f826806ae70 0x7f826806d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.903+0000 7f828be4b700 1 -- 172.21.15.189:0/378829612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82840cfa00 msgr2=0x7f82840b5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:48.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.903+0000 7f828be4b700 1 --2- 172.21.15.189:0/378829612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82840cfa00 0x7f82840b5040 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f8274000c00 tx=0x7f8274004750 comp rx=0 tx=0).stop 2024-04-06T12:31:48.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.903+0000 7f828be4b700 1 -- 172.21.15.189:0/378829612 shutdown_connections 2024-04-06T12:31:48.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.903+0000 7f828be4b700 1 --2- 172.21.15.189:0/378829612 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82840d11b0 0x7f8284158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.903+0000 7f828be4b700 1 --2- 172.21.15.189:0/378829612 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f826806ae70 0x7f826806d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.903+0000 7f828be4b700 1 --2- 172.21.15.189:0/378829612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82840cb870 0x7f8284155590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.903+0000 7f828be4b700 1 --2- 172.21.15.189:0/378829612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82840cfa00 0x7f82840b5040 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:48.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.903+0000 7f828be4b700 1 -- 172.21.15.189:0/378829612 >> 172.21.15.189:0/378829612 conn(0x7f82840b2db0 msgr2=0x7f82840b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:48.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.903+0000 7f828be4b700 1 -- 172.21.15.189:0/378829612 shutdown_connections 2024-04-06T12:31:48.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:48.903+0000 7f828be4b700 1 -- 172.21.15.189:0/378829612 wait complete. 2024-04-06T12:31:48.901 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 62 2024-04-06T12:31:48.912 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:31:48.970 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:48.973+0000 7f4b58d91200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:31:49.035 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:31:49.038+0000 7f75c0c26200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:31:49.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.041+0000 7f129976e700 1 Processor -- start 2024-04-06T12:31:49.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.041+0000 7f129976e700 1 -- start start 2024-04-06T12:31:49.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.042+0000 7f129976e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f129415d7b0 0x7f12940b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:49.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.042+0000 7f129976e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f129415e500 0x7f129406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:49.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.042+0000 7f129976e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12940b3990 con 0x7f129406a730 2024-04-06T12:31:49.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.042+0000 7f129976e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f129406b0f0 con 0x7f129415d7b0 2024-04-06T12:31:49.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.042+0000 7f129976e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12940b3060 con 0x7f129415e500 2024-04-06T12:31:49.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.042+0000 7f1292ffd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f129415d7b0 0x7f12940b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:49.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.042+0000 7f12927fc700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f129406a730 0x7f129415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:37828/0 (socket says 172.21.15.189:37828) 2024-04-06T12:31:49.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.042+0000 7f12927fc700 1 -- 172.21.15.189:0/470203558 learned_addr learned my addr 172.21.15.189:0/470203558 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:49.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.042+0000 7f12937fe700 1 --2- 172.21.15.189:0/470203558 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f129415e500 0x7f129406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:49.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.042+0000 7f1292ffd700 1 -- 172.21.15.189:0/470203558 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f129415e500 msgr2=0x7f129406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:49.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.042+0000 7f1292ffd700 1 --2- 172.21.15.189:0/470203558 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f129415e500 0x7f129406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.042+0000 7f1292ffd700 1 -- 172.21.15.189:0/470203558 >> v1:172.21.15.192:6789/0 conn(0x7f129406a730 legacy=0x7f129415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:49.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.042+0000 7f1291ffb700 1 -- 172.21.15.189:0/470203558 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3057464526 0 0) 0x7f12940b3990 con 0x7f129406a730 2024-04-06T12:31:49.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.042+0000 7f1292ffd700 1 -- 172.21.15.189:0/470203558 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12940b3340 con 0x7f129415d7b0 2024-04-06T12:31:49.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.042+0000 7f1292ffd700 1 --2- 172.21.15.189:0/470203558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f129415d7b0 0x7f12940b3e20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f1284009840 tx=0x7f1284009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fea306848779c8c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:49.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.043+0000 7f1291ffb700 1 -- 172.21.15.189:0/470203558 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1284011830 con 0x7f129415d7b0 2024-04-06T12:31:49.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.043+0000 7f1291ffb700 1 -- 172.21.15.189:0/470203558 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1284011990 con 0x7f129415d7b0 2024-04-06T12:31:49.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.043+0000 7f1291ffb700 1 -- 172.21.15.189:0/470203558 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1284011d20 con 0x7f129415d7b0 2024-04-06T12:31:49.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.043+0000 7f129976e700 1 -- 172.21.15.189:0/470203558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f129415d7b0 msgr2=0x7f12940b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:49.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.043+0000 7f129976e700 1 --2- 172.21.15.189:0/470203558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f129415d7b0 0x7f12940b3e20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f1284009840 tx=0x7f1284009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:49.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.043+0000 7f129976e700 1 -- 172.21.15.189:0/470203558 shutdown_connections 2024-04-06T12:31:49.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.043+0000 7f129976e700 1 --2- 172.21.15.189:0/470203558 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f129415e500 0x7f129406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.043+0000 7f129976e700 1 --2- 172.21.15.189:0/470203558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f129415d7b0 0x7f12940b3e20 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.043+0000 7f129976e700 1 -- 172.21.15.189:0/470203558 >> 172.21.15.189:0/470203558 conn(0x7f12940d00b0 msgr2=0x7f12940d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:49.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.043+0000 7f129976e700 1 -- 172.21.15.189:0/470203558 shutdown_connections 2024-04-06T12:31:49.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.043+0000 7f129976e700 1 -- 172.21.15.189:0/470203558 wait complete. 2024-04-06T12:31:49.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.045+0000 7f129976e700 1 Processor -- start 2024-04-06T12:31:49.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.045+0000 7f129976e700 1 -- start start 2024-04-06T12:31:49.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.045+0000 7f129976e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f129406a730 0x7f1294148bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:49.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.045+0000 7f129976e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f129415d7b0 0x7f129414c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:49.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.045+0000 7f129976e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f129415e500 0x7f12941f6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:49.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.045+0000 7f129976e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129414a2c0 con 0x7f129415e500 2024-04-06T12:31:49.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.045+0000 7f129976e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12941493f0 con 0x7f129415d7b0 2024-04-06T12:31:49.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.045+0000 7f129976e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1294149110 con 0x7f129406a730 2024-04-06T12:31:49.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.045+0000 7f12937fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f129415e500 0x7f12941f6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:49.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.045+0000 7f12937fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f129415e500 0x7f12941f6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40750/0 (socket says 172.21.15.189:40750) 2024-04-06T12:31:49.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.045+0000 7f12937fe700 1 -- 172.21.15.189:0/1349011543 learned_addr learned my addr 172.21.15.189:0/1349011543 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:49.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.045+0000 7f12927fc700 1 --2- 172.21.15.189:0/1349011543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f129415d7b0 0x7f129414c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:49.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.045+0000 7f1292ffd700 1 --2- 172.21.15.189:0/1349011543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f129406a730 0x7f1294148bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:49.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.046+0000 7f12927fc700 1 -- 172.21.15.189:0/1349011543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f129406a730 msgr2=0x7f1294148bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:49.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.046+0000 7f12927fc700 1 --2- 172.21.15.189:0/1349011543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f129406a730 0x7f1294148bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.046+0000 7f12927fc700 1 -- 172.21.15.189:0/1349011543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f129415e500 msgr2=0x7f12941f6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:49.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.046+0000 7f12927fc700 1 --2- 172.21.15.189:0/1349011543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f129415e500 0x7f12941f6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.046+0000 7f12927fc700 1 -- 172.21.15.189:0/1349011543 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12840094a0 con 0x7f129415d7b0 2024-04-06T12:31:49.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.046+0000 7f1292ffd700 1 --2- 172.21.15.189:0/1349011543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f129406a730 0x7f1294148bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:49.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.046+0000 7f12937fe700 1 --2- 172.21.15.189:0/1349011543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f129415e500 0x7f12941f6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:49.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.046+0000 7f12927fc700 1 --2- 172.21.15.189:0/1349011543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f129415d7b0 0x7f129414c8e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f128800c4b0 tx=0x7f128800c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:49.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.046+0000 7f127b7fe700 1 -- 172.21.15.189:0/1349011543 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1288004c80 con 0x7f129415d7b0 2024-04-06T12:31:49.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.046+0000 7f127b7fe700 1 -- 172.21.15.189:0/1349011543 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1288004e00 con 0x7f129415d7b0 2024-04-06T12:31:49.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.046+0000 7f129976e700 1 -- 172.21.15.189:0/1349011543 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1294149640 con 0x7f129415d7b0 2024-04-06T12:31:49.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.046+0000 7f127b7fe700 1 -- 172.21.15.189:0/1349011543 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1288007310 con 0x7f129415d7b0 2024-04-06T12:31:49.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.046+0000 7f129976e700 1 -- 172.21.15.189:0/1349011543 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1294149c00 con 0x7f129415d7b0 2024-04-06T12:31:49.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.047+0000 7f129976e700 1 -- 172.21.15.189:0/1349011543 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f129414a400 con 0x7f129415d7b0 2024-04-06T12:31:49.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.048+0000 7f127b7fe700 1 -- 172.21.15.189:0/1349011543 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 82) v1 ==== 89304+0+0 (secure 0 0 0) 0x7f1288007470 con 0x7f129415d7b0 2024-04-06T12:31:49.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.048+0000 7f127b7fe700 1 --2- 172.21.15.189:0/1349011543 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f128006ae40 0x7f128006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:49.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.048+0000 7f1292ffd700 1 -- 172.21.15.189:0/1349011543 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f128006ae40 msgr2=0x7f128006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:49.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.048+0000 7f127b7fe700 1 -- 172.21.15.189:0/1349011543 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f1288089430 con 0x7f129415d7b0 2024-04-06T12:31:49.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.048+0000 7f1292ffd700 1 --2- 172.21.15.189:0/1349011543 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f128006ae40 0x7f128006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:31:49.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.053+0000 7f127b7fe700 1 -- 172.21.15.189:0/1349011543 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f129414a400 con 0x7f129415d7b0 2024-04-06T12:31:49.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.249+0000 7f1292ffd700 1 -- 172.21.15.189:0/1349011543 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f128006ae40 msgr2=0x7f128006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:31:49.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.249+0000 7f1292ffd700 1 --2- 172.21.15.189:0/1349011543 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f128006ae40 0x7f128006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:31:49.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.301+0000 7f129976e700 1 -- 172.21.15.189:0/1349011543 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f129414a400 con 0x7f129415d7b0 2024-04-06T12:31:49.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.301+0000 7f127b7fe700 1 -- 172.21.15.189:0/1349011543 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f129414a400 con 0x7f129415d7b0 2024-04-06T12:31:49.299 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:49.299 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7115,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/538134862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":538134862},{"type":"v1","addr":"172.21.15.192:6834","nonce":538134862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7915,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1788966297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1788966297},{"type":"v1","addr":"172.21.15.189:6837","nonce":1788966297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7930,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2932927074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2932927074},{"type":"v1","addr":"172.21.15.192:6835","nonce":2932927074}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:31:22.083923+0000","modified":"2024-04-06T12:31:25.104510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/4249639368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":4249639368},{"type":"v1","addr":"172.21.15.189:6836","nonce":4249639368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-06T12:31:49.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.304+0000 7f129976e700 1 -- 172.21.15.189:0/1349011543 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f128006ae40 msgr2=0x7f128006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:31:49.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.304+0000 7f129976e700 1 --2- 172.21.15.189:0/1349011543 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f128006ae40 0x7f128006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.304+0000 7f129976e700 1 -- 172.21.15.189:0/1349011543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f129415d7b0 msgr2=0x7f129414c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:49.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.304+0000 7f129976e700 1 --2- 172.21.15.189:0/1349011543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f129415d7b0 0x7f129414c8e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f128800c4b0 tx=0x7f128800c7c0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.304+0000 7f129976e700 1 -- 172.21.15.189:0/1349011543 shutdown_connections 2024-04-06T12:31:49.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.304+0000 7f129976e700 1 --2- 172.21.15.189:0/1349011543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f129415e500 0x7f12941f6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.304+0000 7f129976e700 1 --2- 172.21.15.189:0/1349011543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f129406a730 0x7f1294148bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.304+0000 7f129976e700 1 --2- 172.21.15.189:0/1349011543 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f128006ae40 0x7f128006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.304+0000 7f129976e700 1 --2- 172.21.15.189:0/1349011543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f129415d7b0 0x7f129414c8e0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.304+0000 7f129976e700 1 -- 172.21.15.189:0/1349011543 >> 172.21.15.189:0/1349011543 conn(0x7f12940d00b0 msgr2=0x7f12940ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:49.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.304+0000 7f129976e700 1 -- 172.21.15.189:0/1349011543 shutdown_connections 2024-04-06T12:31:49.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.304+0000 7f129976e700 1 -- 172.21.15.189:0/1349011543 wait complete. 2024-04-06T12:31:49.303 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 62 2024-04-06T12:31:49.315 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:31:49.367 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:49.370+0000 7f4b58d91200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:31:49.449 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:49.452+0000 7f4ad44b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:49.449 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:49.452+0000 7f4ad44b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:49.450 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:49.452+0000 7f4ad44b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:49.450 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:49.452+0000 7f4ad44b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:49.450 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:49.452+0000 7f4ad44b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:49.450 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:49.452+0000 7f4ad14aa700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:49.450 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:49.452+0000 7f4ad14aa700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:49.450 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:49.452+0000 7f4ad14aa700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:49.450 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:49.452+0000 7f4ad14aa700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:49.450 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:31:49.452+0000 7f4ad14aa700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:31:49.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.454+0000 7f0a95ccf700 1 Processor -- start 2024-04-06T12:31:49.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.454+0000 7f0a95ccf700 1 -- start start 2024-04-06T12:31:49.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.455+0000 7f0a95ccf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a90159b20 0x7f0a9006b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:49.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.455+0000 7f0a95ccf700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0a9006eb80 0x7f0a900ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:49.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.455+0000 7f0a95ccf700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9006a8d0 con 0x7f0a900cdc60 2024-04-06T12:31:49.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.455+0000 7f0a95ccf700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a900ce620 con 0x7f0a9006eb80 2024-04-06T12:31:49.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.455+0000 7f0a95ccf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a90159f00 con 0x7f0a90159b20 2024-04-06T12:31:49.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.455+0000 7f0a8f7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a90159b20 0x7f0a9006b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:49.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.455+0000 7f0a8f7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a90159b20 0x7f0a9006b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:60390/0 (socket says 172.21.15.189:60390) 2024-04-06T12:31:49.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.455+0000 7f0a8f7fe700 1 -- 172.21.15.189:0/1956713259 learned_addr learned my addr 172.21.15.189:0/1956713259 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:49.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.455+0000 7f0a8ffff700 1 --2- 172.21.15.189:0/1956713259 >> v2:172.21.15.192:3300/0 conn(0x7f0a9006eb80 0x7f0a900ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:49.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.456+0000 7f0a8ffff700 1 -- 172.21.15.189:0/1956713259 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a90159b20 msgr2=0x7f0a9006b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:49.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.456+0000 7f0a8ffff700 1 --2- 172.21.15.189:0/1956713259 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a90159b20 0x7f0a9006b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.456+0000 7f0a8ffff700 1 -- 172.21.15.189:0/1956713259 >> v1:172.21.15.192:6789/0 conn(0x7f0a900cdc60 legacy=0x7f0a9006e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:49.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.456+0000 7f0a8ffff700 1 -- 172.21.15.189:0/1956713259 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9006abb0 con 0x7f0a9006eb80 2024-04-06T12:31:49.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.456+0000 7f0a8ffff700 1 --2- 172.21.15.189:0/1956713259 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a9006eb80 0x7f0a900ce0e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009c80 tx=0x7f0a88009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d8790d0c84be3e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:49.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.456+0000 7f0a8e7fc700 1 -- 172.21.15.189:0/1956713259 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8800ca00 con 0x7f0a9006eb80 2024-04-06T12:31:49.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.456+0000 7f0a8e7fc700 1 -- 172.21.15.189:0/1956713259 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8800cb60 con 0x7f0a9006eb80 2024-04-06T12:31:49.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.456+0000 7f0a8e7fc700 1 -- 172.21.15.189:0/1956713259 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a88011400 con 0x7f0a9006eb80 2024-04-06T12:31:49.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.457+0000 7f0a95ccf700 1 -- 172.21.15.189:0/1956713259 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a9006eb80 msgr2=0x7f0a900ce0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:49.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.457+0000 7f0a95ccf700 1 --2- 172.21.15.189:0/1956713259 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a9006eb80 0x7f0a900ce0e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009c80 tx=0x7f0a88009f90 comp rx=0 tx=0).stop 2024-04-06T12:31:49.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.457+0000 7f0a95ccf700 1 -- 172.21.15.189:0/1956713259 shutdown_connections 2024-04-06T12:31:49.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.457+0000 7f0a95ccf700 1 --2- 172.21.15.189:0/1956713259 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a9006eb80 0x7f0a900ce0e0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.457+0000 7f0a95ccf700 1 --2- 172.21.15.189:0/1956713259 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a90159b20 0x7f0a9006b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.457+0000 7f0a95ccf700 1 -- 172.21.15.189:0/1956713259 >> 172.21.15.189:0/1956713259 conn(0x7f0a900cc290 msgr2=0x7f0a900cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:49.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.457+0000 7f0a95ccf700 1 -- 172.21.15.189:0/1956713259 shutdown_connections 2024-04-06T12:31:49.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.457+0000 7f0a95ccf700 1 -- 172.21.15.189:0/1956713259 wait complete. 2024-04-06T12:31:49.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.459+0000 7f0a95ccf700 1 Processor -- start 2024-04-06T12:31:49.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.459+0000 7f0a95ccf700 1 -- start start 2024-04-06T12:31:49.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.460+0000 7f0a95ccf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a9006eb80 0x7f0a9014ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:49.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.460+0000 7f0a95ccf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a900cdc60 0x7f0a9014d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:49.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.460+0000 7f0a95ccf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a90159b20 0x7f0a900c3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:49.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.460+0000 7f0a95ccf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9014df50 con 0x7f0a9006eb80 2024-04-06T12:31:49.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.460+0000 7f0a95ccf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9014e230 con 0x7f0a900cdc60 2024-04-06T12:31:49.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.460+0000 7f0a95ccf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9014e510 con 0x7f0a90159b20 2024-04-06T12:31:49.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.460+0000 7f0a8ffff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a90159b20 0x7f0a900c3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:49.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.460+0000 7f0a8ffff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a90159b20 0x7f0a900c3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:60394/0 (socket says 172.21.15.189:60394) 2024-04-06T12:31:49.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.460+0000 7f0a8ffff700 1 -- 172.21.15.189:0/1475577499 learned_addr learned my addr 172.21.15.189:0/1475577499 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:49.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.460+0000 7f0a8effd700 1 --2- 172.21.15.189:0/1475577499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a900cdc60 0x7f0a9014d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:49.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.460+0000 7f0a8f7fe700 1 --2- 172.21.15.189:0/1475577499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a9006eb80 0x7f0a9014ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:49.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.460+0000 7f0a8ffff700 1 -- 172.21.15.189:0/1475577499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a900cdc60 msgr2=0x7f0a9014d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:49.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.460+0000 7f0a8ffff700 1 --2- 172.21.15.189:0/1475577499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a900cdc60 0x7f0a9014d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.460+0000 7f0a8ffff700 1 -- 172.21.15.189:0/1475577499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a9006eb80 msgr2=0x7f0a9014ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:49.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.460+0000 7f0a8ffff700 1 --2- 172.21.15.189:0/1475577499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a9006eb80 0x7f0a9014ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.460+0000 7f0a8ffff700 1 -- 172.21.15.189:0/1475577499 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8800d040 con 0x7f0a90159b20 2024-04-06T12:31:49.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.460+0000 7f0a8f7fe700 1 --2- 172.21.15.189:0/1475577499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a9006eb80 0x7f0a9014ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:49.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.460+0000 7f0a8ffff700 1 --2- 172.21.15.189:0/1475577499 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a90159b20 0x7f0a900c3e80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0a88014000 tx=0x7f0a88004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:49.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.461+0000 7f0a77fff700 1 -- 172.21.15.189:0/1475577499 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8801f070 con 0x7f0a90159b20 2024-04-06T12:31:49.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.461+0000 7f0a77fff700 1 -- 172.21.15.189:0/1475577499 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8801a450 con 0x7f0a90159b20 2024-04-06T12:31:49.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.461+0000 7f0a77fff700 1 -- 172.21.15.189:0/1475577499 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a88011400 con 0x7f0a90159b20 2024-04-06T12:31:49.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.461+0000 7f0a95ccf700 1 -- 172.21.15.189:0/1475577499 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a9014e7f0 con 0x7f0a90159b20 2024-04-06T12:31:49.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.461+0000 7f0a95ccf700 1 -- 172.21.15.189:0/1475577499 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a9014d3c0 con 0x7f0a90159b20 2024-04-06T12:31:49.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.462+0000 7f0a95ccf700 1 -- 172.21.15.189:0/1475577499 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a9014ed10 con 0x7f0a90159b20 2024-04-06T12:31:49.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.463+0000 7f0a77fff700 1 -- 172.21.15.189:0/1475577499 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 83) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f0a8801a600 con 0x7f0a90159b20 2024-04-06T12:31:49.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.463+0000 7f0a77fff700 1 -- 172.21.15.189:0/1475577499 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6674+0+0 (secure 0 0 0) 0x7f0a8808af00 con 0x7f0a90159b20 2024-04-06T12:31:49.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.467+0000 7f0a77fff700 1 -- 172.21.15.189:0/1475577499 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0a88056210 con 0x7f0a90159b20 2024-04-06T12:31:49.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.668+0000 7f0a95ccf700 1 -- 172.21.15.189:0/1475577499 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a9014ed10 con 0x7f0a90159b20 2024-04-06T12:31:49.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.669+0000 7f0a77fff700 1 -- 172.21.15.189:0/1475577499 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+17301 (secure 0 0 0) 0x7f0a9014ed10 con 0x7f0a90159b20 2024-04-06T12:31:49.667 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:49.667 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":146,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:31:49.384171+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:31:19.866331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-06T12:31:21.048344+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/651843838":"2024-04-07T12:31:49.384120+0000","172.21.15.189:0/3582536648":"2024-04-07T12:31:49.384120+0000","172.21.15.189:0/424019374":"2024-04-07T12:31:49.384120+0000","172.21.15.189:0/1212184215":"2024-04-07T12:31:49.384120+0000","172.21.15.189:0/3411787925":"2024-04-07T12:31:49.384120+0000","172.21.15.189:0/3195414802":"2024-04-07T12:31:49.384120+0000","172.21.15.189:0/2323784940":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/2597819342":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/3011108552":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/814932379":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/2994336395":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/2720887145":"2024-04-07T12:31:36.068397+0000","172.21.15.189:0/3847769194":"2024-04-07T12:31:49.384120+0000","172.21.15.189:6833/47128":"2024-04-07T12:31:36.068397+0000","172.21.15.189:6832/47128":"2024-04-07T12:31:36.068397+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:31:49.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.671+0000 7f0a95ccf700 1 -- 172.21.15.189:0/1475577499 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a90159b20 msgr2=0x7f0a900c3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:49.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.671+0000 7f0a95ccf700 1 --2- 172.21.15.189:0/1475577499 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a90159b20 0x7f0a900c3e80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0a88014000 tx=0x7f0a88004ad0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.671+0000 7f0a95ccf700 1 -- 172.21.15.189:0/1475577499 shutdown_connections 2024-04-06T12:31:49.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.671+0000 7f0a95ccf700 1 --2- 172.21.15.189:0/1475577499 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a90159b20 0x7f0a900c3e80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.671+0000 7f0a95ccf700 1 --2- 172.21.15.189:0/1475577499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a9006eb80 0x7f0a9014ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.671+0000 7f0a95ccf700 1 --2- 172.21.15.189:0/1475577499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a900cdc60 0x7f0a9014d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.672+0000 7f0a95ccf700 1 -- 172.21.15.189:0/1475577499 >> 172.21.15.189:0/1475577499 conn(0x7f0a900cc290 msgr2=0x7f0a900cb830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:49.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.672+0000 7f0a95ccf700 1 -- 172.21.15.189:0/1475577499 shutdown_connections 2024-04-06T12:31:49.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.672+0000 7f0a95ccf700 1 -- 172.21.15.189:0/1475577499 wait complete. 2024-04-06T12:31:49.681 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-06T12:31:49.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.808+0000 7fd2707b9700 1 Processor -- start 2024-04-06T12:31:49.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.808+0000 7fd2707b9700 1 -- start start 2024-04-06T12:31:49.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.809+0000 7fd2707b9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd2680d17a0 0x7fd26806b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:49.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.809+0000 7fd2707b9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd268146dd0 0x7fd26806d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:49.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.809+0000 7fd2707b9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26806c0a0 con 0x7fd2680cedf0 2024-04-06T12:31:49.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.809+0000 7fd2707b9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd26806c380 con 0x7fd2680d17a0 2024-04-06T12:31:49.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.809+0000 7fd2707b9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd26806c660 con 0x7fd268146dd0 2024-04-06T12:31:49.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.810+0000 7fd26e555700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd2680d17a0 0x7fd26806b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:49.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.810+0000 7fd26dd54700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd268146dd0 0x7fd26806d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:49.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.810+0000 7fd26e555700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd2680d17a0 0x7fd26806b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:40450/0 (socket says 172.21.15.189:40450) 2024-04-06T12:31:49.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.810+0000 7fd26e555700 1 -- 172.21.15.189:0/1633626789 learned_addr learned my addr 172.21.15.189:0/1633626789 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:49.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.810+0000 7fd26dd54700 1 -- 172.21.15.189:0/1633626789 >> v2:172.21.15.192:3300/0 conn(0x7fd2680d17a0 msgr2=0x7fd26806b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:49.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.810+0000 7fd26dd54700 1 --2- 172.21.15.189:0/1633626789 >> v2:172.21.15.192:3300/0 conn(0x7fd2680d17a0 0x7fd26806b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.810+0000 7fd26dd54700 1 -- 172.21.15.189:0/1633626789 >> v1:172.21.15.192:6789/0 conn(0x7fd2680cedf0 legacy=0x7fd2681441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:49.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.810+0000 7fd26dd54700 1 -- 172.21.15.189:0/1633626789 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26806c940 con 0x7fd268146dd0 2024-04-06T12:31:49.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.810+0000 7fd26e555700 1 --2- 172.21.15.189:0/1633626789 >> v2:172.21.15.192:3300/0 conn(0x7fd2680d17a0 0x7fd26806b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:49.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.810+0000 7fd26dd54700 1 --2- 172.21.15.189:0/1633626789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd268146dd0 0x7fd26806d140 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd264009840 tx=0x7fd264009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4981f17d3e241721 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:49.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.810+0000 7fd26d553700 1 -- 172.21.15.189:0/1633626789 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd264011830 con 0x7fd268146dd0 2024-04-06T12:31:49.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.810+0000 7fd26d553700 1 -- 172.21.15.189:0/1633626789 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd264011990 con 0x7fd268146dd0 2024-04-06T12:31:49.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.810+0000 7fd26d553700 1 -- 172.21.15.189:0/1633626789 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd264011d20 con 0x7fd268146dd0 2024-04-06T12:31:49.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.811+0000 7fd2707b9700 1 -- 172.21.15.189:0/1633626789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd268146dd0 msgr2=0x7fd26806d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:49.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.811+0000 7fd2707b9700 1 --2- 172.21.15.189:0/1633626789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd268146dd0 0x7fd26806d140 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd264009840 tx=0x7fd264009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:49.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.811+0000 7fd2707b9700 1 -- 172.21.15.189:0/1633626789 shutdown_connections 2024-04-06T12:31:49.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.811+0000 7fd2707b9700 1 --2- 172.21.15.189:0/1633626789 >> v2:172.21.15.192:3300/0 conn(0x7fd2680d17a0 0x7fd26806b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.811+0000 7fd2707b9700 1 --2- 172.21.15.189:0/1633626789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd268146dd0 0x7fd26806d140 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.811+0000 7fd2707b9700 1 -- 172.21.15.189:0/1633626789 >> 172.21.15.189:0/1633626789 conn(0x7fd2680b2db0 msgr2=0x7fd2680b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:49.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.811+0000 7fd2707b9700 1 -- 172.21.15.189:0/1633626789 shutdown_connections 2024-04-06T12:31:49.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.811+0000 7fd2707b9700 1 -- 172.21.15.189:0/1633626789 wait complete. 2024-04-06T12:31:49.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.812+0000 7fd2707b9700 1 Processor -- start 2024-04-06T12:31:49.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.812+0000 7fd2707b9700 1 -- start start 2024-04-06T12:31:49.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.813+0000 7fd2707b9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2680cedf0 0x7fd2680b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:49.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.813+0000 7fd2707b9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2680d17a0 0x7fd2680b6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:49.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.813+0000 7fd2707b9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd268146dd0 0x7fd2680b5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:49.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.813+0000 7fd2707b9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2680b71d0 con 0x7fd2680d17a0 2024-04-06T12:31:49.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.813+0000 7fd2707b9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2680c7df0 con 0x7fd2680cedf0 2024-04-06T12:31:49.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.813+0000 7fd2707b9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2680b5b60 con 0x7fd268146dd0 2024-04-06T12:31:49.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.813+0000 7fd26dd54700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2680d17a0 0x7fd2680b6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:49.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.813+0000 7fd26dd54700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2680d17a0 0x7fd2680b6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47256/0 (socket says 172.21.15.189:47256) 2024-04-06T12:31:49.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.813+0000 7fd26dd54700 1 -- 172.21.15.189:0/2012458519 learned_addr learned my addr 172.21.15.189:0/2012458519 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:49.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.813+0000 7fd26ed56700 1 --2- 172.21.15.189:0/2012458519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd268146dd0 0x7fd2680b5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:49.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.813+0000 7fd26e555700 1 --2- 172.21.15.189:0/2012458519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2680cedf0 0x7fd2680b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:49.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.813+0000 7fd26dd54700 1 -- 172.21.15.189:0/2012458519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd268146dd0 msgr2=0x7fd2680b5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:49.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.813+0000 7fd26dd54700 1 --2- 172.21.15.189:0/2012458519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd268146dd0 0x7fd2680b5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.813+0000 7fd26dd54700 1 -- 172.21.15.189:0/2012458519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2680cedf0 msgr2=0x7fd2680b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:49.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.813+0000 7fd26dd54700 1 --2- 172.21.15.189:0/2012458519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2680cedf0 0x7fd2680b3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:49.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.813+0000 7fd26dd54700 1 -- 172.21.15.189:0/2012458519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd258009550 con 0x7fd2680d17a0 2024-04-06T12:31:49.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.813+0000 7fd26e555700 1 --2- 172.21.15.189:0/2012458519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2680cedf0 0x7fd2680b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:49.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.814+0000 7fd26dd54700 1 --2- 172.21.15.189:0/2012458519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2680d17a0 0x7fd2680b6c90 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fd264009fd0 tx=0x7fd264004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:49.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.814+0000 7fd25effd700 1 -- 172.21.15.189:0/2012458519 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2640036a0 con 0x7fd2680d17a0 2024-04-06T12:31:49.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.814+0000 7fd25effd700 1 -- 172.21.15.189:0/2012458519 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26401f070 con 0x7fd2680d17a0 2024-04-06T12:31:49.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.814+0000 7fd2707b9700 1 -- 172.21.15.189:0/2012458519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2640094a0 con 0x7fd2680d17a0 2024-04-06T12:31:49.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.814+0000 7fd25effd700 1 -- 172.21.15.189:0/2012458519 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd264003a60 con 0x7fd2680d17a0 2024-04-06T12:31:49.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.814+0000 7fd2707b9700 1 -- 172.21.15.189:0/2012458519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2680b4860 con 0x7fd2680d17a0 2024-04-06T12:31:49.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.815+0000 7fd2707b9700 1 -- 172.21.15.189:0/2012458519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd268074990 con 0x7fd2680d17a0 2024-04-06T12:31:49.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.815+0000 7fd25effd700 1 -- 172.21.15.189:0/2012458519 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 83) v1 ==== 88856+0+0 (secure 0 0 0) 0x7fd264003bc0 con 0x7fd2680d17a0 2024-04-06T12:31:49.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.816+0000 7fd25effd700 1 -- 172.21.15.189:0/2012458519 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6674+0+0 (secure 0 0 0) 0x7fd264021070 con 0x7fd2680d17a0 2024-04-06T12:31:49.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:49.820+0000 7fd25effd700 1 -- 172.21.15.189:0/2012458519 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd264055cf0 con 0x7fd2680d17a0 2024-04-06T12:31:50.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.055+0000 7fd2707b9700 1 -- 172.21.15.189:0/2012458519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd2680d1e20 con 0x7fd2680d17a0 2024-04-06T12:31:50.118 INFO:tasks.ceph.mon.a.smithi189.stderr:2024-04-06T12:31:50.056+0000 7fa9fe1ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T12:31:50.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.063+0000 7fd25effd700 1 -- 172.21.15.189:0/2012458519 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v63) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd26402b090 con 0x7fd2680d17a0 2024-04-06T12:31:50.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.066+0000 7fd2707b9700 1 -- 172.21.15.189:0/2012458519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2680d17a0 msgr2=0x7fd2680b6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:50.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.066+0000 7fd2707b9700 1 --2- 172.21.15.189:0/2012458519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2680d17a0 0x7fd2680b6c90 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fd264009fd0 tx=0x7fd264004750 comp rx=0 tx=0).stop 2024-04-06T12:31:50.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.066+0000 7fd2707b9700 1 -- 172.21.15.189:0/2012458519 shutdown_connections 2024-04-06T12:31:50.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.066+0000 7fd2707b9700 1 --2- 172.21.15.189:0/2012458519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd268146dd0 0x7fd2680b5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:50.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.066+0000 7fd2707b9700 1 --2- 172.21.15.189:0/2012458519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2680cedf0 0x7fd2680b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:50.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.066+0000 7fd2707b9700 1 --2- 172.21.15.189:0/2012458519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2680d17a0 0x7fd2680b6c90 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:50.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.066+0000 7fd2707b9700 1 -- 172.21.15.189:0/2012458519 >> 172.21.15.189:0/2012458519 conn(0x7fd2680b2db0 msgr2=0x7fd268161010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:50.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.066+0000 7fd2707b9700 1 -- 172.21.15.189:0/2012458519 shutdown_connections 2024-04-06T12:31:50.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.066+0000 7fd2707b9700 1 -- 172.21.15.189:0/2012458519 wait complete. 2024-04-06T12:31:50.120 INFO:tasks.ceph.mds.c.smithi189.stderr: -9687> 2024-04-06T12:31:27.667+0000 7f9d16487700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T12:31:50.120 INFO:teuthology.orchestra.run.smithi189.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-06T12:31:50.121 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-06T12:31:50.121 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T12:31:50.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.248+0000 7fbcba801700 1 Processor -- start 2024-04-06T12:31:50.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.248+0000 7fbcba801700 1 -- start start 2024-04-06T12:31:50.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.248+0000 7fbcba801700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbcb415d7b0 0x7fbcb40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:50.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.248+0000 7fbcba801700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb40b3990 con 0x7fbcb406a730 2024-04-06T12:31:50.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.248+0000 7fbcba801700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb40b3060 con 0x7fbcb415e500 2024-04-06T12:31:50.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.248+0000 7fbcba801700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcb415e950 con 0x7fbcb415d7b0 2024-04-06T12:31:50.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.249+0000 7fbcb37fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbcb406a730 0x7fbcb415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40824/0 (socket says 172.21.15.189:40824) 2024-04-06T12:31:50.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.249+0000 7fbcb37fe700 1 -- 172.21.15.189:0/2306470751 learned_addr learned my addr 172.21.15.189:0/2306470751 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:50.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.249+0000 7fbcb3fff700 1 --2- 172.21.15.189:0/2306470751 >> v2:172.21.15.192:3300/0 conn(0x7fbcb415d7b0 0x7fbcb40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:50.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.249+0000 7fbcb2ffd700 1 -- 172.21.15.189:0/2306470751 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935252654 0 0) 0x7fbcb40b3990 con 0x7fbcb406a730 2024-04-06T12:31:50.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.249+0000 7fbcb2ffd700 1 -- 172.21.15.189:0/2306470751 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc9c0035b0 con 0x7fbcb406a730 2024-04-06T12:31:50.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.249+0000 7fbcb3fff700 1 -- 172.21.15.189:0/2306470751 >> v1:172.21.15.192:6789/0 conn(0x7fbcb415e500 legacy=0x7fbcb40717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:50.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.249+0000 7fbcb3fff700 1 -- 172.21.15.189:0/2306470751 >> v1:172.21.15.189:6789/0 conn(0x7fbcb406a730 legacy=0x7fbcb415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:50.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.249+0000 7fbcb3fff700 1 -- 172.21.15.189:0/2306470751 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb40b3340 con 0x7fbcb415d7b0 2024-04-06T12:31:50.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.249+0000 7fbcb3fff700 1 --2- 172.21.15.189:0/2306470751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb415d7b0 0x7fbcb40b3e20 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fbca4009840 tx=0x7fbca4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bf05403ae76c3e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:50.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.250+0000 7fbcb2ffd700 1 -- 172.21.15.189:0/2306470751 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca4011830 con 0x7fbcb415d7b0 2024-04-06T12:31:50.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.250+0000 7fbcb2ffd700 1 -- 172.21.15.189:0/2306470751 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca4011990 con 0x7fbcb415d7b0 2024-04-06T12:31:50.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.250+0000 7fbcb2ffd700 1 -- 172.21.15.189:0/2306470751 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca4011d00 con 0x7fbcb415d7b0 2024-04-06T12:31:50.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.250+0000 7fbcba801700 1 -- 172.21.15.189:0/2306470751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb415d7b0 msgr2=0x7fbcb40b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:50.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.250+0000 7fbcba801700 1 --2- 172.21.15.189:0/2306470751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb415d7b0 0x7fbcb40b3e20 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fbca4009840 tx=0x7fbca4009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:50.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.250+0000 7fbcba801700 1 -- 172.21.15.189:0/2306470751 shutdown_connections 2024-04-06T12:31:50.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.250+0000 7fbcba801700 1 --2- 172.21.15.189:0/2306470751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb415d7b0 0x7fbcb40b3e20 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:50.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.250+0000 7fbcba801700 1 -- 172.21.15.189:0/2306470751 >> 172.21.15.189:0/2306470751 conn(0x7fbcb40d00b0 msgr2=0x7fbcb40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:50.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.250+0000 7fbcba801700 1 -- 172.21.15.189:0/2306470751 shutdown_connections 2024-04-06T12:31:50.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.250+0000 7fbcba801700 1 -- 172.21.15.189:0/2306470751 wait complete. 2024-04-06T12:31:50.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.251+0000 7fbcba801700 1 Processor -- start 2024-04-06T12:31:50.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.252+0000 7fbcba801700 1 -- start start 2024-04-06T12:31:50.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.252+0000 7fbcba801700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcb406a730 0x7fbcb40c58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:50.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.252+0000 7fbcba801700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbcb415d7b0 0x7fbcb414c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:50.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.252+0000 7fbcba801700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb415e500 0x7fbcb40cece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:50.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.252+0000 7fbcba801700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb40cdc40 con 0x7fbcb406a730 2024-04-06T12:31:50.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.252+0000 7fbcba801700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb40cdf20 con 0x7fbcb415e500 2024-04-06T12:31:50.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.252+0000 7fbcba801700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb40ce200 con 0x7fbcb415d7b0 2024-04-06T12:31:50.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.252+0000 7fbcb3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcb406a730 0x7fbcb40c58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:50.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.253+0000 7fbcb3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcb406a730 0x7fbcb40c58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47274/0 (socket says 172.21.15.189:47274) 2024-04-06T12:31:50.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.253+0000 7fbcb3fff700 1 -- 172.21.15.189:0/3026112252 learned_addr learned my addr 172.21.15.189:0/3026112252 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:50.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.253+0000 7fbcb8d9e700 1 --2- 172.21.15.189:0/3026112252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb415e500 0x7fbcb40cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:50.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.253+0000 7fbcb37fe700 1 --2- 172.21.15.189:0/3026112252 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbcb415d7b0 0x7fbcb414c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:50.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.253+0000 7fbcb3fff700 1 -- 172.21.15.189:0/3026112252 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbcb415d7b0 msgr2=0x7fbcb414c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:50.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.253+0000 7fbcb3fff700 1 --2- 172.21.15.189:0/3026112252 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbcb415d7b0 0x7fbcb414c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:50.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.253+0000 7fbcb3fff700 1 -- 172.21.15.189:0/3026112252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb415e500 msgr2=0x7fbcb40cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:50.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.253+0000 7fbcb3fff700 1 --2- 172.21.15.189:0/3026112252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb415e500 0x7fbcb40cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:50.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.253+0000 7fbcb3fff700 1 -- 172.21.15.189:0/3026112252 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca40094a0 con 0x7fbcb406a730 2024-04-06T12:31:50.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.253+0000 7fbcb37fe700 1 --2- 172.21.15.189:0/3026112252 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbcb415d7b0 0x7fbcb414c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:50.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.253+0000 7fbcb3fff700 1 --2- 172.21.15.189:0/3026112252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcb406a730 0x7fbcb40c58f0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fbca4009fd0 tx=0x7fbca4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:50.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.253+0000 7fbcb0ff9700 1 -- 172.21.15.189:0/3026112252 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca40045d0 con 0x7fbcb406a730 2024-04-06T12:31:50.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.253+0000 7fbcb0ff9700 1 -- 172.21.15.189:0/3026112252 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca401f070 con 0x7fbcb406a730 2024-04-06T12:31:50.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.253+0000 7fbcb0ff9700 1 -- 172.21.15.189:0/3026112252 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca40038a0 con 0x7fbcb406a730 2024-04-06T12:31:50.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.253+0000 7fbcba801700 1 -- 172.21.15.189:0/3026112252 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb40ce4e0 con 0x7fbcb406a730 2024-04-06T12:31:50.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.253+0000 7fbcba801700 1 -- 172.21.15.189:0/3026112252 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb40c5e30 con 0x7fbcb406a730 2024-04-06T12:31:50.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.254+0000 7fbcba801700 1 -- 172.21.15.189:0/3026112252 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb4067bb0 con 0x7fbcb406a730 2024-04-06T12:31:50.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.254+0000 7fbcb0ff9700 1 -- 172.21.15.189:0/3026112252 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 83) v1 ==== 88856+0+0 (secure 0 0 0) 0x7fbca4003a00 con 0x7fbcb406a730 2024-04-06T12:31:50.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.255+0000 7fbcb0ff9700 1 -- 172.21.15.189:0/3026112252 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6760+0+0 (secure 0 0 0) 0x7fbca4021070 con 0x7fbcb406a730 2024-04-06T12:31:50.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.259+0000 7fbcb0ff9700 1 -- 172.21.15.189:0/3026112252 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbca4055ca0 con 0x7fbcb406a730 2024-04-06T12:31:50.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.394+0000 7fbcb0ff9700 1 -- 172.21.15.189:0/3026112252 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 84) v1 ==== 88983+0+0 (secure 0 0 0) 0x7fbca4055750 con 0x7fbcb406a730 2024-04-06T12:31:50.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.394+0000 7fbcb0ff9700 1 --2- 172.21.15.189:0/3026112252 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbca0036150 0x7fbca00729e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:50.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.395+0000 7fbcb37fe700 1 --2- 172.21.15.189:0/3026112252 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbca0036150 0x7fbca00729e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:50.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.397+0000 7fbcb37fe700 1 --2- 172.21.15.189:0/3026112252 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbca0036150 0x7fbca00729e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbca80011e0 tx=0x7fbca8009040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:50.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:50.506+0000 7fbcba801700 1 -- 172.21.15.189:0/3026112252 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fbcb4074990 con 0x7fbcb406a730 2024-04-06T12:31:51.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.065+0000 7fbcb0ff9700 1 -- 172.21.15.189:0/3026112252 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v64) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbca406d970 con 0x7fbcb406a730 2024-04-06T12:31:51.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.067+0000 7fbcba801700 1 -- 172.21.15.189:0/3026112252 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbca0036150 msgr2=0x7fbca00729e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:51.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.067+0000 7fbcba801700 1 --2- 172.21.15.189:0/3026112252 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbca0036150 0x7fbca00729e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbca80011e0 tx=0x7fbca8009040 comp rx=0 tx=0).stop 2024-04-06T12:31:51.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.067+0000 7fbcba801700 1 -- 172.21.15.189:0/3026112252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcb406a730 msgr2=0x7fbcb40c58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:51.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.067+0000 7fbcba801700 1 --2- 172.21.15.189:0/3026112252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcb406a730 0x7fbcb40c58f0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fbca4009fd0 tx=0x7fbca4004750 comp rx=0 tx=0).stop 2024-04-06T12:31:51.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.068+0000 7fbcba801700 1 -- 172.21.15.189:0/3026112252 shutdown_connections 2024-04-06T12:31:51.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.068+0000 7fbcba801700 1 --2- 172.21.15.189:0/3026112252 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fbca0036150 0x7fbca00729e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:51.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.068+0000 7fbcba801700 1 --2- 172.21.15.189:0/3026112252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb415e500 0x7fbcb40cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:51.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.068+0000 7fbcba801700 1 --2- 172.21.15.189:0/3026112252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcb406a730 0x7fbcb40c58f0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:51.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.068+0000 7fbcba801700 1 --2- 172.21.15.189:0/3026112252 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbcb415d7b0 0x7fbcb414c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:51.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.068+0000 7fbcba801700 1 -- 172.21.15.189:0/3026112252 >> 172.21.15.189:0/3026112252 conn(0x7fbcb40d00b0 msgr2=0x7fbcb40ae1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:51.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.068+0000 7fbcba801700 1 -- 172.21.15.189:0/3026112252 shutdown_connections 2024-04-06T12:31:51.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.068+0000 7fbcba801700 1 -- 172.21.15.189:0/3026112252 wait complete. 2024-04-06T12:31:51.079 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-06T12:31:51.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.211+0000 7fc4f99cc700 1 Processor -- start 2024-04-06T12:31:51.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.211+0000 7fc4f99cc700 1 -- start start 2024-04-06T12:31:51.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.211+0000 7fc4f99cc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc4f40cf380 0x7fc4f40ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:51.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.212+0000 7fc4f99cc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc4f40aed50 0x7fc4f40cc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:51.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.212+0000 7fc4f99cc700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f40cd060 con 0x7fc4f40af200 2024-04-06T12:31:51.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.212+0000 7fc4f99cc700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f40cd620 con 0x7fc4f40aed50 2024-04-06T12:31:51.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.212+0000 7fc4f99cc700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f40cd340 con 0x7fc4f40cf380 2024-04-06T12:31:51.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.212+0000 7fc4f27fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc4f40aed50 0x7fc4f40cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:51.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.212+0000 7fc4f27fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc4f40aed50 0x7fc4f40cc400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47278/0 (socket says 172.21.15.189:47278) 2024-04-06T12:31:51.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.212+0000 7fc4f27fc700 1 -- 172.21.15.189:0/2613032527 learned_addr learned my addr 172.21.15.189:0/2613032527 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:51.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.212+0000 7fc4f2ffd700 1 --2- 172.21.15.189:0/2613032527 >> v2:172.21.15.192:3300/0 conn(0x7fc4f40cf380 0x7fc4f40ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:51.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.212+0000 7fc4f27fc700 1 -- 172.21.15.189:0/2613032527 >> v2:172.21.15.192:3300/0 conn(0x7fc4f40cf380 msgr2=0x7fc4f40ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:51.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.212+0000 7fc4f27fc700 1 --2- 172.21.15.189:0/2613032527 >> v2:172.21.15.192:3300/0 conn(0x7fc4f40cf380 0x7fc4f40ceab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:51.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.212+0000 7fc4f27fc700 1 -- 172.21.15.189:0/2613032527 >> v1:172.21.15.192:6789/0 conn(0x7fc4f40af200 legacy=0x7fc4f40cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:51.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.212+0000 7fc4f27fc700 1 -- 172.21.15.189:0/2613032527 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f40cbee0 con 0x7fc4f40aed50 2024-04-06T12:31:51.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.212+0000 7fc4f27fc700 1 --2- 172.21.15.189:0/2613032527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4f40aed50 0x7fc4f40cc400 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fc4f40cf350 tx=0x7fc4e8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af8467477d3b125 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:51.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.212+0000 7fc4f1ffb700 1 -- 172.21.15.189:0/2613032527 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8011830 con 0x7fc4f40aed50 2024-04-06T12:31:51.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.212+0000 7fc4f1ffb700 1 -- 172.21.15.189:0/2613032527 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e8011990 con 0x7fc4f40aed50 2024-04-06T12:31:51.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.212+0000 7fc4f1ffb700 1 -- 172.21.15.189:0/2613032527 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8011d20 con 0x7fc4f40aed50 2024-04-06T12:31:51.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.213+0000 7fc4f99cc700 1 -- 172.21.15.189:0/2613032527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4f40aed50 msgr2=0x7fc4f40cc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:51.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.213+0000 7fc4f99cc700 1 --2- 172.21.15.189:0/2613032527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4f40aed50 0x7fc4f40cc400 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fc4f40cf350 tx=0x7fc4e8009bd0 comp rx=0 tx=0).stop 2024-04-06T12:31:51.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.213+0000 7fc4f99cc700 1 -- 172.21.15.189:0/2613032527 shutdown_connections 2024-04-06T12:31:51.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.213+0000 7fc4f99cc700 1 --2- 172.21.15.189:0/2613032527 >> v2:172.21.15.192:3300/0 conn(0x7fc4f40cf380 0x7fc4f40ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:51.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.213+0000 7fc4f99cc700 1 --2- 172.21.15.189:0/2613032527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4f40aed50 0x7fc4f40cc400 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:51.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.213+0000 7fc4f99cc700 1 -- 172.21.15.189:0/2613032527 >> 172.21.15.189:0/2613032527 conn(0x7fc4f4071f30 msgr2=0x7fc4f4072340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:51.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.213+0000 7fc4f99cc700 1 -- 172.21.15.189:0/2613032527 shutdown_connections 2024-04-06T12:31:51.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.213+0000 7fc4f99cc700 1 -- 172.21.15.189:0/2613032527 wait complete. 2024-04-06T12:31:51.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.214+0000 7fc4f99cc700 1 Processor -- start 2024-04-06T12:31:51.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.214+0000 7fc4f99cc700 1 -- start start 2024-04-06T12:31:51.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.215+0000 7fc4f99cc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4f40aed50 0x7fc4f415c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:51.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.215+0000 7fc4f99cc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4f40af200 0x7fc4f40c2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:51.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.215+0000 7fc4f99cc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4f40cf380 0x7fc4f40af680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:51.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.215+0000 7fc4f99cc700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f415ca40 con 0x7fc4f40aed50 2024-04-06T12:31:51.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.215+0000 7fc4f99cc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f40c39e0 con 0x7fc4f40af200 2024-04-06T12:31:51.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.215+0000 7fc4f99cc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f40c3cc0 con 0x7fc4f40cf380 2024-04-06T12:31:51.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.215+0000 7fc4f2ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4f40aed50 0x7fc4f415c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:51.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.215+0000 7fc4f2ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4f40aed50 0x7fc4f415c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47280/0 (socket says 172.21.15.189:47280) 2024-04-06T12:31:51.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.215+0000 7fc4f2ffd700 1 -- 172.21.15.189:0/2583682459 learned_addr learned my addr 172.21.15.189:0/2583682459 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:51.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.215+0000 7fc4f27fc700 1 --2- 172.21.15.189:0/2583682459 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4f40af200 0x7fc4f40c2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:51.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.215+0000 7fc4f37fe700 1 --2- 172.21.15.189:0/2583682459 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4f40cf380 0x7fc4f40af680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:51.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.215+0000 7fc4f2ffd700 1 -- 172.21.15.189:0/2583682459 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4f40cf380 msgr2=0x7fc4f40af680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:51.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.215+0000 7fc4f2ffd700 1 --2- 172.21.15.189:0/2583682459 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4f40cf380 0x7fc4f40af680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:51.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.215+0000 7fc4f2ffd700 1 -- 172.21.15.189:0/2583682459 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4f40af200 msgr2=0x7fc4f40c2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:51.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.215+0000 7fc4f2ffd700 1 --2- 172.21.15.189:0/2583682459 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4f40af200 0x7fc4f40c2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:51.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.215+0000 7fc4f2ffd700 1 -- 172.21.15.189:0/2583682459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e8009550 con 0x7fc4f40aed50 2024-04-06T12:31:51.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.216+0000 7fc4f27fc700 1 --2- 172.21.15.189:0/2583682459 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4f40af200 0x7fc4f40c2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:51.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.216+0000 7fc4f2ffd700 1 --2- 172.21.15.189:0/2583682459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4f40aed50 0x7fc4f415c500 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fc4e400e870 tx=0x7fc4e400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:51.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.216+0000 7fc4dbfff700 1 -- 172.21.15.189:0/2583682459 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4009590 con 0x7fc4f40aed50 2024-04-06T12:31:51.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.216+0000 7fc4dbfff700 1 -- 172.21.15.189:0/2583682459 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e400ee70 con 0x7fc4f40aed50 2024-04-06T12:31:51.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.216+0000 7fc4f99cc700 1 -- 172.21.15.189:0/2583682459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f40c3fa0 con 0x7fc4f40aed50 2024-04-06T12:31:51.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.216+0000 7fc4dbfff700 1 -- 172.21.15.189:0/2583682459 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e400ca50 con 0x7fc4f40aed50 2024-04-06T12:31:51.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.216+0000 7fc4f99cc700 1 -- 172.21.15.189:0/2583682459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f40b5160 con 0x7fc4f40aed50 2024-04-06T12:31:51.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.217+0000 7fc4f99cc700 1 -- 172.21.15.189:0/2583682459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f4074990 con 0x7fc4f40aed50 2024-04-06T12:31:51.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.217+0000 7fc4dbfff700 1 -- 172.21.15.189:0/2583682459 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 84) v1 ==== 88983+0+0 (secure 0 0 0) 0x7fc4e400cbb0 con 0x7fc4f40aed50 2024-04-06T12:31:51.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.217+0000 7fc4dbfff700 1 --2- 172.21.15.189:0/2583682459 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc4e006aae0 0x7fc4e006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:51.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.218+0000 7fc4dbfff700 1 -- 172.21.15.189:0/2583682459 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6760+0+0 (secure 0 0 0) 0x7fc4e4089320 con 0x7fc4f40aed50 2024-04-06T12:31:51.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.218+0000 7fc4f27fc700 1 --2- 172.21.15.189:0/2583682459 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc4e006aae0 0x7fc4e006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:51.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.218+0000 7fc4f27fc700 1 --2- 172.21.15.189:0/2583682459 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc4e006aae0 0x7fc4e006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc4f40c4870 tx=0x7fc4e8008030 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:51.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.222+0000 7fc4dbfff700 1 -- 172.21.15.189:0/2583682459 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc4e4054420 con 0x7fc4f40aed50 2024-04-06T12:31:51.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:51.417+0000 7fc4f99cc700 1 -- 172.21.15.189:0/2583682459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc4f406d060 con 0x7fc4f40aed50 2024-04-06T12:31:52.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.404+0000 7fc4dbfff700 1 -- 172.21.15.189:0/2583682459 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 85) v1 ==== 89191+0+0 (secure 0 0 0) 0x7fc4e405e0d0 con 0x7fc4f40aed50 2024-04-06T12:31:52.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.406+0000 7fc4dbfff700 1 -- 172.21.15.189:0/2583682459 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v148) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc4e40890d0 con 0x7fc4f40aed50 2024-04-06T12:31:52.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.410+0000 7fc4f99cc700 1 -- 172.21.15.189:0/2583682459 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc4e006aae0 msgr2=0x7fc4e006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:52.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.410+0000 7fc4f99cc700 1 --2- 172.21.15.189:0/2583682459 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc4e006aae0 0x7fc4e006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc4f40c4870 tx=0x7fc4e8008030 comp rx=0 tx=0).stop 2024-04-06T12:31:52.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.410+0000 7fc4f99cc700 1 -- 172.21.15.189:0/2583682459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4f40aed50 msgr2=0x7fc4f415c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:52.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.410+0000 7fc4f99cc700 1 --2- 172.21.15.189:0/2583682459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4f40aed50 0x7fc4f415c500 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fc4e400e870 tx=0x7fc4e400ec30 comp rx=0 tx=0).stop 2024-04-06T12:31:52.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.410+0000 7fc4f99cc700 1 -- 172.21.15.189:0/2583682459 shutdown_connections 2024-04-06T12:31:52.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.410+0000 7fc4f99cc700 1 --2- 172.21.15.189:0/2583682459 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4f40cf380 0x7fc4f40af680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:52.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.410+0000 7fc4f99cc700 1 --2- 172.21.15.189:0/2583682459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4f40aed50 0x7fc4f415c500 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:52.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.410+0000 7fc4f99cc700 1 --2- 172.21.15.189:0/2583682459 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc4e006aae0 0x7fc4e006cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:52.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.410+0000 7fc4f99cc700 1 --2- 172.21.15.189:0/2583682459 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4f40af200 0x7fc4f40c2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:52.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.410+0000 7fc4f99cc700 1 -- 172.21.15.189:0/2583682459 >> 172.21.15.189:0/2583682459 conn(0x7fc4f4071f30 msgr2=0x7fc4f40cc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:52.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.411+0000 7fc4f99cc700 1 -- 172.21.15.189:0/2583682459 shutdown_connections 2024-04-06T12:31:52.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.411+0000 7fc4f99cc700 1 -- 172.21.15.189:0/2583682459 wait complete. 2024-04-06T12:31:52.409 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' removed 2024-04-06T12:31:52.423 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-06T12:31:52.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.556+0000 7fa76fda8700 1 Processor -- start 2024-04-06T12:31:52.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.557+0000 7fa76fda8700 1 -- start start 2024-04-06T12:31:52.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.557+0000 7fa76fda8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7680cb870 0x7fa76815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:52.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.557+0000 7fa76fda8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa7680cfa00 0x7fa7680cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:52.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.557+0000 7fa76fda8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7680cbd80 con 0x7fa7680d11b0 2024-04-06T12:31:52.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.557+0000 7fa76fda8700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7680ccdc0 con 0x7fa7680cfa00 2024-04-06T12:31:52.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.557+0000 7fa76fda8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7680d0b50 con 0x7fa7680cb870 2024-04-06T12:31:52.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.558+0000 7fa76d343700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7680cb870 0x7fa76815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:52.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.558+0000 7fa76e345700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa7680cfa00 0x7fa7680cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:52.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.558+0000 7fa76d343700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7680cb870 0x7fa76815c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:60468/0 (socket says 172.21.15.189:60468) 2024-04-06T12:31:52.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.558+0000 7fa76d343700 1 -- 172.21.15.189:0/2532913957 learned_addr learned my addr 172.21.15.189:0/2532913957 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:52.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.558+0000 7fa76e345700 1 -- 172.21.15.189:0/2532913957 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7680cb870 msgr2=0x7fa76815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:52.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.558+0000 7fa76e345700 1 --2- 172.21.15.189:0/2532913957 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7680cb870 0x7fa76815c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:52.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.558+0000 7fa76e345700 1 -- 172.21.15.189:0/2532913957 >> v1:172.21.15.192:6789/0 conn(0x7fa7680d11b0 legacy=0x7fa7680d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:52.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.558+0000 7fa76e345700 1 -- 172.21.15.189:0/2532913957 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7680cef20 con 0x7fa7680cfa00 2024-04-06T12:31:52.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.558+0000 7fa76d343700 1 --2- 172.21.15.189:0/2532913957 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7680cb870 0x7fa76815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:52.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.558+0000 7fa76cb42700 1 -- 172.21.15.189:0/2532913957 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778367451 0 0) 0x7fa7680cbd80 con 0x7fa7680d11b0 2024-04-06T12:31:52.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.559+0000 7fa76e345700 1 --2- 172.21.15.189:0/2532913957 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7680cfa00 0x7fa7680cd560 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa760009d90 tx=0x7fa76000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db18f36e19cb5aac server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:52.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.559+0000 7fa76cb42700 1 -- 172.21.15.189:0/2532913957 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76000e070 con 0x7fa7680cfa00 2024-04-06T12:31:52.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.559+0000 7fa76cb42700 1 -- 172.21.15.189:0/2532913957 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76000bb20 con 0x7fa7680cfa00 2024-04-06T12:31:52.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.559+0000 7fa76cb42700 1 -- 172.21.15.189:0/2532913957 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760012430 con 0x7fa7680cfa00 2024-04-06T12:31:52.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.559+0000 7fa76fda8700 1 -- 172.21.15.189:0/2532913957 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7680cfa00 msgr2=0x7fa7680cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:52.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.559+0000 7fa76fda8700 1 --2- 172.21.15.189:0/2532913957 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7680cfa00 0x7fa7680cd560 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa760009d90 tx=0x7fa76000b330 comp rx=0 tx=0).stop 2024-04-06T12:31:52.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.559+0000 7fa76fda8700 1 -- 172.21.15.189:0/2532913957 shutdown_connections 2024-04-06T12:31:52.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.559+0000 7fa76fda8700 1 --2- 172.21.15.189:0/2532913957 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7680cfa00 0x7fa7680cd560 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:52.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.559+0000 7fa76fda8700 1 --2- 172.21.15.189:0/2532913957 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7680cb870 0x7fa76815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:52.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.559+0000 7fa76fda8700 1 -- 172.21.15.189:0/2532913957 >> 172.21.15.189:0/2532913957 conn(0x7fa7680b2db0 msgr2=0x7fa7680b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:52.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.559+0000 7fa76fda8700 1 -- 172.21.15.189:0/2532913957 shutdown_connections 2024-04-06T12:31:52.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.559+0000 7fa76fda8700 1 -- 172.21.15.189:0/2532913957 wait complete. 2024-04-06T12:31:52.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.561+0000 7fa76fda8700 1 Processor -- start 2024-04-06T12:31:52.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.561+0000 7fa76fda8700 1 -- start start 2024-04-06T12:31:52.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.561+0000 7fa76fda8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7680cb870 0x7fa7681f6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:52.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.561+0000 7fa76fda8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7680cfa00 0x7fa7681f31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:52.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.561+0000 7fa76fda8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7680d11b0 0x7fa7680ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:52.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.561+0000 7fa76fda8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7681f41b0 con 0x7fa7680d11b0 2024-04-06T12:31:52.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.561+0000 7fa76fda8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7681f6550 con 0x7fa7680cb870 2024-04-06T12:31:52.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.561+0000 7fa76fda8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa76815e0f0 con 0x7fa7680cfa00 2024-04-06T12:31:52.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.561+0000 7fa76e345700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7680d11b0 0x7fa7680ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:52.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.561+0000 7fa76e345700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7680d11b0 0x7fa7680ca6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47300/0 (socket says 172.21.15.189:47300) 2024-04-06T12:31:52.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.561+0000 7fa76e345700 1 -- 172.21.15.189:0/2705019781 learned_addr learned my addr 172.21.15.189:0/2705019781 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:52.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.562+0000 7fa76d343700 1 --2- 172.21.15.189:0/2705019781 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7680cfa00 0x7fa7681f31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:52.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.562+0000 7fa76db44700 1 --2- 172.21.15.189:0/2705019781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7680cb870 0x7fa7681f6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:52.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.562+0000 7fa76e345700 1 -- 172.21.15.189:0/2705019781 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7680cfa00 msgr2=0x7fa7681f31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:52.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.562+0000 7fa76e345700 1 --2- 172.21.15.189:0/2705019781 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7680cfa00 0x7fa7681f31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:52.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.562+0000 7fa76e345700 1 -- 172.21.15.189:0/2705019781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7680cb870 msgr2=0x7fa7681f6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:52.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.562+0000 7fa76e345700 1 --2- 172.21.15.189:0/2705019781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7680cb870 0x7fa7681f6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:52.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.562+0000 7fa76e345700 1 -- 172.21.15.189:0/2705019781 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa764009550 con 0x7fa7680d11b0 2024-04-06T12:31:52.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.562+0000 7fa76db44700 1 --2- 172.21.15.189:0/2705019781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7680cb870 0x7fa7681f6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:52.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.562+0000 7fa76e345700 1 --2- 172.21.15.189:0/2705019781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7680d11b0 0x7fa7680ca6d0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fa760014040 tx=0x7fa760004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:52.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.562+0000 7fa75e7fc700 1 -- 172.21.15.189:0/2705019781 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760012430 con 0x7fa7680d11b0 2024-04-06T12:31:52.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.562+0000 7fa75e7fc700 1 -- 172.21.15.189:0/2705019781 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa760020070 con 0x7fa7680d11b0 2024-04-06T12:31:52.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.562+0000 7fa76fda8700 1 -- 172.21.15.189:0/2705019781 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa760009a70 con 0x7fa7680d11b0 2024-04-06T12:31:52.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.562+0000 7fa75e7fc700 1 -- 172.21.15.189:0/2705019781 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760004280 con 0x7fa7680d11b0 2024-04-06T12:31:52.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.562+0000 7fa76fda8700 1 -- 172.21.15.189:0/2705019781 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7681f3ed0 con 0x7fa7680d11b0 2024-04-06T12:31:52.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.563+0000 7fa76fda8700 1 -- 172.21.15.189:0/2705019781 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa768074990 con 0x7fa7680d11b0 2024-04-06T12:31:52.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.564+0000 7fa75e7fc700 1 -- 172.21.15.189:0/2705019781 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 85) v1 ==== 89191+0+0 (secure 0 0 0) 0x7fa7600043e0 con 0x7fa7680d11b0 2024-04-06T12:31:52.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.564+0000 7fa75e7fc700 1 --2- 172.21.15.189:0/2705019781 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa74c06ad20 0x7fa74c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:52.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.564+0000 7fa75e7fc700 1 -- 172.21.15.189:0/2705019781 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6322+0+0 (secure 0 0 0) 0x7fa760022070 con 0x7fa7680d11b0 2024-04-06T12:31:52.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.564+0000 7fa76db44700 1 --2- 172.21.15.189:0/2705019781 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa74c06ad20 0x7fa74c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:52.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.565+0000 7fa76db44700 1 --2- 172.21.15.189:0/2705019781 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa74c06ad20 0x7fa74c06d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa758000f60 tx=0x7fa758009040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:52.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.569+0000 7fa75e7fc700 1 -- 172.21.15.189:0/2705019781 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa7600562b0 con 0x7fa7680d11b0 2024-04-06T12:31:52.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:52.770+0000 7fa76fda8700 1 -- 172.21.15.189:0/2705019781 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa7680b0a00 con 0x7fa7680d11b0 2024-04-06T12:31:53.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.636+0000 7fa75e7fc700 1 -- 172.21.15.189:0/2705019781 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fa760056680 con 0x7fa7680d11b0 2024-04-06T12:31:53.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.637+0000 7fa75e7fc700 1 -- 172.21.15.189:0/2705019781 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v149) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa76006e4e0 con 0x7fa7680d11b0 2024-04-06T12:31:53.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.641+0000 7fa76fda8700 1 -- 172.21.15.189:0/2705019781 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa74c06ad20 msgr2=0x7fa74c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:53.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.641+0000 7fa76fda8700 1 --2- 172.21.15.189:0/2705019781 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa74c06ad20 0x7fa74c06d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa758000f60 tx=0x7fa758009040 comp rx=0 tx=0).stop 2024-04-06T12:31:53.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.641+0000 7fa76fda8700 1 -- 172.21.15.189:0/2705019781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7680d11b0 msgr2=0x7fa7680ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:53.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.641+0000 7fa76fda8700 1 --2- 172.21.15.189:0/2705019781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7680d11b0 0x7fa7680ca6d0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fa760014040 tx=0x7fa760004ad0 comp rx=0 tx=0).stop 2024-04-06T12:31:53.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.641+0000 7fa76fda8700 1 -- 172.21.15.189:0/2705019781 shutdown_connections 2024-04-06T12:31:53.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.641+0000 7fa76fda8700 1 --2- 172.21.15.189:0/2705019781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7680d11b0 0x7fa7680ca6d0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:53.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.641+0000 7fa76fda8700 1 --2- 172.21.15.189:0/2705019781 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa74c06ad20 0x7fa74c06d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:53.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.641+0000 7fa76fda8700 1 --2- 172.21.15.189:0/2705019781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7680cb870 0x7fa7681f6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:53.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.641+0000 7fa76fda8700 1 --2- 172.21.15.189:0/2705019781 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7680cfa00 0x7fa7681f31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:53.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.641+0000 7fa76fda8700 1 -- 172.21.15.189:0/2705019781 >> 172.21.15.189:0/2705019781 conn(0x7fa7680b2db0 msgr2=0x7fa7680b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:53.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.641+0000 7fa76fda8700 1 -- 172.21.15.189:0/2705019781 shutdown_connections 2024-04-06T12:31:53.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.641+0000 7fa76fda8700 1 -- 172.21.15.189:0/2705019781 wait complete. 2024-04-06T12:31:53.639 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' removed 2024-04-06T12:31:53.653 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-06T12:31:53.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.785+0000 7fc1a6c31700 1 Processor -- start 2024-04-06T12:31:53.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.785+0000 7fc1a6c31700 1 -- start start 2024-04-06T12:31:53.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.786+0000 7fc1a6c31700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc1a015d7b0 0x7fc1a00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:53.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.786+0000 7fc1a6c31700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a00b3990 con 0x7fc1a015e500 2024-04-06T12:31:53.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.786+0000 7fc1a6c31700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a00b3060 con 0x7fc1a006a730 2024-04-06T12:31:53.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.786+0000 7fc1a6c31700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1a015e950 con 0x7fc1a015d7b0 2024-04-06T12:31:53.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.786+0000 7fc1a51ce700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc1a015e500 0x7fc1a00717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40838/0 (socket says 172.21.15.189:40838) 2024-04-06T12:31:53.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.786+0000 7fc1a51ce700 1 -- 172.21.15.189:0/2029002553 learned_addr learned my addr 172.21.15.189:0/2029002553 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:53.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.786+0000 7fc1a49cd700 1 --2- 172.21.15.189:0/2029002553 >> v2:172.21.15.192:3300/0 conn(0x7fc1a015d7b0 0x7fc1a00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:53.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.786+0000 7fc19f7fe700 1 -- 172.21.15.189:0/2029002553 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318617500 0 0) 0x7fc1a00b3990 con 0x7fc1a015e500 2024-04-06T12:31:53.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.786+0000 7fc19f7fe700 1 -- 172.21.15.189:0/2029002553 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1900035b0 con 0x7fc1a015e500 2024-04-06T12:31:53.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.786+0000 7fc1a49cd700 1 -- 172.21.15.189:0/2029002553 >> v1:172.21.15.192:6789/0 conn(0x7fc1a006a730 legacy=0x7fc1a015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:53.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.786+0000 7fc19f7fe700 1 -- 172.21.15.189:0/2029002553 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710448633 0 0) 0x7fc1a00b3060 con 0x7fc1a006a730 2024-04-06T12:31:53.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.786+0000 7fc1a49cd700 1 -- 172.21.15.189:0/2029002553 >> v1:172.21.15.189:6789/0 conn(0x7fc1a015e500 legacy=0x7fc1a00717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:53.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.786+0000 7fc1a49cd700 1 -- 172.21.15.189:0/2029002553 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a00b3340 con 0x7fc1a015d7b0 2024-04-06T12:31:53.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.787+0000 7fc1a49cd700 1 --2- 172.21.15.189:0/2029002553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1a015d7b0 0x7fc1a00b3e20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc194009840 tx=0x7fc194009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8fcf84ccc32acbc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:53.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.787+0000 7fc19f7fe700 1 -- 172.21.15.189:0/2029002553 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1980019f0 con 0x7fc1a015d7b0 2024-04-06T12:31:53.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.787+0000 7fc19f7fe700 1 -- 172.21.15.189:0/2029002553 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc194004030 con 0x7fc1a015d7b0 2024-04-06T12:31:53.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.787+0000 7fc19f7fe700 1 -- 172.21.15.189:0/2029002553 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1940043c0 con 0x7fc1a015d7b0 2024-04-06T12:31:53.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.787+0000 7fc1a6c31700 1 -- 172.21.15.189:0/2029002553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1a015d7b0 msgr2=0x7fc1a00b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:53.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.787+0000 7fc1a6c31700 1 --2- 172.21.15.189:0/2029002553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1a015d7b0 0x7fc1a00b3e20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc194009840 tx=0x7fc194009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:53.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.787+0000 7fc1a6c31700 1 -- 172.21.15.189:0/2029002553 shutdown_connections 2024-04-06T12:31:53.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.787+0000 7fc1a6c31700 1 --2- 172.21.15.189:0/2029002553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1a015d7b0 0x7fc1a00b3e20 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:53.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.787+0000 7fc1a6c31700 1 -- 172.21.15.189:0/2029002553 >> 172.21.15.189:0/2029002553 conn(0x7fc1a00d00b0 msgr2=0x7fc1a00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:53.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.787+0000 7fc1a6c31700 1 -- 172.21.15.189:0/2029002553 shutdown_connections 2024-04-06T12:31:53.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.787+0000 7fc1a6c31700 1 -- 172.21.15.189:0/2029002553 wait complete. 2024-04-06T12:31:53.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.788+0000 7fc1a6c31700 1 Processor -- start 2024-04-06T12:31:53.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.789+0000 7fc1a6c31700 1 -- start start 2024-04-06T12:31:53.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.789+0000 7fc1a6c31700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1a006a730 0x7fc1a00c5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:53.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.789+0000 7fc1a6c31700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a015d7b0 0x7fc1a01f5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:53.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.789+0000 7fc1a6c31700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1a015e500 0x7fc1a00b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:53.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.789+0000 7fc1a6c31700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a00c6680 con 0x7fc1a015d7b0 2024-04-06T12:31:53.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.789+0000 7fc1a6c31700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a00c5a70 con 0x7fc1a015e500 2024-04-06T12:31:53.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.789+0000 7fc1a6c31700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a00c5d50 con 0x7fc1a006a730 2024-04-06T12:31:53.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.790+0000 7fc19ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a015d7b0 0x7fc1a01f5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:53.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.790+0000 7fc19ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a015d7b0 0x7fc1a01f5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47314/0 (socket says 172.21.15.189:47314) 2024-04-06T12:31:53.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.790+0000 7fc19ffff700 1 -- 172.21.15.189:0/2706855999 learned_addr learned my addr 172.21.15.189:0/2706855999 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:53.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.790+0000 7fc1a51ce700 1 --2- 172.21.15.189:0/2706855999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1a015e500 0x7fc1a00b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:53.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.790+0000 7fc1a49cd700 1 --2- 172.21.15.189:0/2706855999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1a006a730 0x7fc1a00c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:53.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.790+0000 7fc19ffff700 1 -- 172.21.15.189:0/2706855999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1a006a730 msgr2=0x7fc1a00c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:53.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.790+0000 7fc19ffff700 1 --2- 172.21.15.189:0/2706855999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1a006a730 0x7fc1a00c5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:53.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.790+0000 7fc19ffff700 1 -- 172.21.15.189:0/2706855999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1a015e500 msgr2=0x7fc1a00b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:53.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.790+0000 7fc19ffff700 1 --2- 172.21.15.189:0/2706855999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1a015e500 0x7fc1a00b0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:53.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.790+0000 7fc19ffff700 1 -- 172.21.15.189:0/2706855999 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1940094a0 con 0x7fc1a015d7b0 2024-04-06T12:31:53.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.790+0000 7fc19ffff700 1 --2- 172.21.15.189:0/2706855999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a015d7b0 0x7fc1a01f5660 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fc18c00c500 tx=0x7fc18c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:53.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.790+0000 7fc19d7fa700 1 -- 172.21.15.189:0/2706855999 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c0054e0 con 0x7fc1a015d7b0 2024-04-06T12:31:53.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.790+0000 7fc19d7fa700 1 -- 172.21.15.189:0/2706855999 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18c003680 con 0x7fc1a015d7b0 2024-04-06T12:31:53.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.790+0000 7fc1a6c31700 1 -- 172.21.15.189:0/2706855999 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a00c6030 con 0x7fc1a015d7b0 2024-04-06T12:31:53.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.790+0000 7fc19d7fa700 1 -- 172.21.15.189:0/2706855999 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c003a30 con 0x7fc1a015d7b0 2024-04-06T12:31:53.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.790+0000 7fc1a6c31700 1 -- 172.21.15.189:0/2706855999 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a00c6b10 con 0x7fc1a015d7b0 2024-04-06T12:31:53.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.791+0000 7fc1a6c31700 1 -- 172.21.15.189:0/2706855999 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a00c6830 con 0x7fc1a015d7b0 2024-04-06T12:31:53.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.792+0000 7fc19d7fa700 1 -- 172.21.15.189:0/2706855999 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fc18c003b90 con 0x7fc1a015d7b0 2024-04-06T12:31:53.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.792+0000 7fc19d7fa700 1 --2- 172.21.15.189:0/2706855999 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc18406adb0 0x7fc18406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:53.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.792+0000 7fc19d7fa700 1 -- 172.21.15.189:0/2706855999 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5908+0+0 (secure 0 0 0) 0x7fc18c088ba0 con 0x7fc1a015d7b0 2024-04-06T12:31:53.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.792+0000 7fc1a49cd700 1 --2- 172.21.15.189:0/2706855999 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc18406adb0 0x7fc18406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:53.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.793+0000 7fc1a49cd700 1 --2- 172.21.15.189:0/2706855999 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc18406adb0 0x7fc18406d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc194009b90 tx=0x7fc194008000 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:53.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:53.796+0000 7fc19d7fa700 1 -- 172.21.15.189:0/2706855999 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc18c053c90 con 0x7fc1a015d7b0 2024-04-06T12:31:53.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.000+0000 7fc1a6c31700 1 -- 172.21.15.189:0/2706855999 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc1a0067bb0 con 0x7fc1a015d7b0 2024-04-06T12:31:54.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.004+0000 7fc19d7fa700 1 -- 172.21.15.189:0/2706855999 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v49) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc18c017020 con 0x7fc1a015d7b0 2024-04-06T12:31:54.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.006+0000 7fc1a6c31700 1 -- 172.21.15.189:0/2706855999 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc18406adb0 msgr2=0x7fc18406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:54.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.006+0000 7fc1a6c31700 1 --2- 172.21.15.189:0/2706855999 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc18406adb0 0x7fc18406d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc194009b90 tx=0x7fc194008000 comp rx=0 tx=0).stop 2024-04-06T12:31:54.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.006+0000 7fc1a6c31700 1 -- 172.21.15.189:0/2706855999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a015d7b0 msgr2=0x7fc1a01f5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:54.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.006+0000 7fc1a6c31700 1 --2- 172.21.15.189:0/2706855999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a015d7b0 0x7fc1a01f5660 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fc18c00c500 tx=0x7fc18c00c810 comp rx=0 tx=0).stop 2024-04-06T12:31:54.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.006+0000 7fc1a6c31700 1 -- 172.21.15.189:0/2706855999 shutdown_connections 2024-04-06T12:31:54.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.006+0000 7fc1a6c31700 1 --2- 172.21.15.189:0/2706855999 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc18406adb0 0x7fc18406d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.006+0000 7fc1a6c31700 1 --2- 172.21.15.189:0/2706855999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1a015e500 0x7fc1a00b0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.006+0000 7fc1a6c31700 1 --2- 172.21.15.189:0/2706855999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1a006a730 0x7fc1a00c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.006+0000 7fc1a6c31700 1 --2- 172.21.15.189:0/2706855999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a015d7b0 0x7fc1a01f5660 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.006+0000 7fc1a6c31700 1 -- 172.21.15.189:0/2706855999 >> 172.21.15.189:0/2706855999 conn(0x7fc1a00d00b0 msgr2=0x7fc1a00adc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:54.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.006+0000 7fc1a6c31700 1 -- 172.21.15.189:0/2706855999 shutdown_connections 2024-04-06T12:31:54.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.006+0000 7fc1a6c31700 1 -- 172.21.15.189:0/2706855999 wait complete. 2024-04-06T12:31:54.015 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-06T12:31:54.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.143+0000 7f1ddedf3700 1 Processor -- start 2024-04-06T12:31:54.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.143+0000 7f1ddedf3700 1 -- start start 2024-04-06T12:31:54.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.144+0000 7f1ddedf3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1dd815a550 0x7f1dd80cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:54.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.144+0000 7f1ddedf3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1dd80cf900 0x7f1dd80b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:54.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.144+0000 7f1ddedf3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1dd80b4ba0 0x7f1dd806eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:54.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.144+0000 7f1ddedf3700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd80cdbe0 con 0x7f1dd815a550 2024-04-06T12:31:54.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.144+0000 7f1ddedf3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd80b5010 con 0x7f1dd80b4ba0 2024-04-06T12:31:54.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.144+0000 7f1ddedf3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd815a940 con 0x7f1dd80cf900 2024-04-06T12:31:54.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.144+0000 7f1ddcb8f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1dd815a550 0x7f1dd80cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:54.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.144+0000 7f1ddcb8f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1dd815a550 0x7f1dd80cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47328/0 (socket says 172.21.15.189:47328) 2024-04-06T12:31:54.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.144+0000 7f1ddcb8f700 1 -- 172.21.15.189:0/440454369 learned_addr learned my addr 172.21.15.189:0/440454369 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:54.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.144+0000 7f1dd7fff700 1 --2- 172.21.15.189:0/440454369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1dd80cf900 0x7f1dd80b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:54.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.144+0000 7f1ddd390700 1 --2- 172.21.15.189:0/440454369 >> v2:172.21.15.192:3300/0 conn(0x7f1dd80b4ba0 0x7f1dd806eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:54.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.144+0000 7f1ddcb8f700 1 -- 172.21.15.189:0/440454369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1dd80cf900 msgr2=0x7f1dd80b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:54.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.144+0000 7f1ddcb8f700 1 --2- 172.21.15.189:0/440454369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1dd80cf900 0x7f1dd80b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.144+0000 7f1ddcb8f700 1 -- 172.21.15.189:0/440454369 >> v2:172.21.15.192:3300/0 conn(0x7f1dd80b4ba0 msgr2=0x7f1dd806eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:54.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.144+0000 7f1ddcb8f700 1 --2- 172.21.15.189:0/440454369 >> v2:172.21.15.192:3300/0 conn(0x7f1dd80b4ba0 0x7f1dd806eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.144+0000 7f1ddcb8f700 1 -- 172.21.15.189:0/440454369 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd815ac00 con 0x7f1dd815a550 2024-04-06T12:31:54.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.144+0000 7f1ddd390700 1 --2- 172.21.15.189:0/440454369 >> v2:172.21.15.192:3300/0 conn(0x7f1dd80b4ba0 0x7f1dd806eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:54.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.145+0000 7f1ddcb8f700 1 --2- 172.21.15.189:0/440454369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dd815a550 0x7f1dd80cf3c0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80d1110 tx=0x7f1dcc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63ad36d4735f4dc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:54.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.145+0000 7f1dd77fe700 1 -- 172.21.15.189:0/440454369 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc00b870 con 0x7f1dd815a550 2024-04-06T12:31:54.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.145+0000 7f1dd77fe700 1 -- 172.21.15.189:0/440454369 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dcc00b9d0 con 0x7f1dd815a550 2024-04-06T12:31:54.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.145+0000 7f1dd77fe700 1 -- 172.21.15.189:0/440454369 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc00bd60 con 0x7f1dd815a550 2024-04-06T12:31:54.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.145+0000 7f1ddedf3700 1 -- 172.21.15.189:0/440454369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dd815a550 msgr2=0x7f1dd80cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:54.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.145+0000 7f1ddedf3700 1 --2- 172.21.15.189:0/440454369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dd815a550 0x7f1dd80cf3c0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80d1110 tx=0x7f1dcc009bd0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.145+0000 7f1ddedf3700 1 -- 172.21.15.189:0/440454369 shutdown_connections 2024-04-06T12:31:54.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.145+0000 7f1ddedf3700 1 --2- 172.21.15.189:0/440454369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dd815a550 0x7f1dd80cf3c0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.145+0000 7f1ddedf3700 1 --2- 172.21.15.189:0/440454369 >> v2:172.21.15.192:3300/0 conn(0x7f1dd80b4ba0 0x7f1dd806eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.145+0000 7f1ddedf3700 1 --2- 172.21.15.189:0/440454369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1dd80cf900 0x7f1dd80b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.145+0000 7f1ddedf3700 1 -- 172.21.15.189:0/440454369 >> 172.21.15.189:0/440454369 conn(0x7f1dd80d00b0 msgr2=0x7f1dd80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:54.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.145+0000 7f1ddedf3700 1 -- 172.21.15.189:0/440454369 shutdown_connections 2024-04-06T12:31:54.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.145+0000 7f1ddedf3700 1 -- 172.21.15.189:0/440454369 wait complete. 2024-04-06T12:31:54.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.147+0000 7f1ddedf3700 1 Processor -- start 2024-04-06T12:31:54.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.147+0000 7f1ddedf3700 1 -- start start 2024-04-06T12:31:54.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.147+0000 7f1ddedf3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1dd80b4ba0 0x7f1dd814ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:54.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.147+0000 7f1ddedf3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dd80cf900 0x7f1dd80ae790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:54.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.147+0000 7f1ddedf3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1dd81503f0 0x7f1dd8158120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:54.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.147+0000 7f1ddedf3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd80af3e0 con 0x7f1dd80cf900 2024-04-06T12:31:54.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.147+0000 7f1ddedf3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd8153040 con 0x7f1dd81503f0 2024-04-06T12:31:54.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.147+0000 7f1ddedf3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd80aee20 con 0x7f1dd80b4ba0 2024-04-06T12:31:54.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.148+0000 7f1dd7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dd80cf900 0x7f1dd80ae790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:54.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.148+0000 7f1dd7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dd80cf900 0x7f1dd80ae790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47330/0 (socket says 172.21.15.189:47330) 2024-04-06T12:31:54.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.148+0000 7f1dd7fff700 1 -- 172.21.15.189:0/190689061 learned_addr learned my addr 172.21.15.189:0/190689061 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:54.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.148+0000 7f1ddcb8f700 1 --2- 172.21.15.189:0/190689061 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1dd80b4ba0 0x7f1dd814ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:54.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.148+0000 7f1ddd390700 1 --2- 172.21.15.189:0/190689061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1dd81503f0 0x7f1dd8158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:54.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.148+0000 7f1dd7fff700 1 -- 172.21.15.189:0/190689061 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1dd80b4ba0 msgr2=0x7f1dd814ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:54.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.148+0000 7f1dd7fff700 1 --2- 172.21.15.189:0/190689061 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1dd80b4ba0 0x7f1dd814ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.148+0000 7f1dd7fff700 1 -- 172.21.15.189:0/190689061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1dd81503f0 msgr2=0x7f1dd8158120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:54.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.148+0000 7f1dd7fff700 1 --2- 172.21.15.189:0/190689061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1dd81503f0 0x7f1dd8158120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.148+0000 7f1dd7fff700 1 -- 172.21.15.189:0/190689061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc009550 con 0x7f1dd80cf900 2024-04-06T12:31:54.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.148+0000 7f1dd7fff700 1 --2- 172.21.15.189:0/190689061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dd80cf900 0x7f1dd80ae790 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f1dc400c7f0 tx=0x7f1dc400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:54.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.148+0000 7f1dd57fa700 1 -- 172.21.15.189:0/190689061 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc4004220 con 0x7f1dd80cf900 2024-04-06T12:31:54.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.148+0000 7f1dd57fa700 1 -- 172.21.15.189:0/190689061 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc4004380 con 0x7f1dd80cf900 2024-04-06T12:31:54.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.148+0000 7f1dd57fa700 1 -- 172.21.15.189:0/190689061 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc40036a0 con 0x7f1dd80cf900 2024-04-06T12:31:54.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.148+0000 7f1ddedf3700 1 -- 172.21.15.189:0/190689061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd80af100 con 0x7f1dd80cf900 2024-04-06T12:31:54.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.148+0000 7f1ddedf3700 1 -- 172.21.15.189:0/190689061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd81535c0 con 0x7f1dd80cf900 2024-04-06T12:31:54.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.149+0000 7f1ddedf3700 1 -- 172.21.15.189:0/190689061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd8067bb0 con 0x7f1dd80cf900 2024-04-06T12:31:54.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.150+0000 7f1dd57fa700 1 -- 172.21.15.189:0/190689061 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f1dc4003800 con 0x7f1dd80cf900 2024-04-06T12:31:54.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.150+0000 7f1dd57fa700 1 --2- 172.21.15.189:0/190689061 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dbc06ade0 0x7f1dbc06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:54.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.150+0000 7f1dd57fa700 1 -- 172.21.15.189:0/190689061 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f1dc40891e0 con 0x7f1dd80cf900 2024-04-06T12:31:54.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.150+0000 7f1ddcb8f700 1 --2- 172.21.15.189:0/190689061 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dbc06ade0 0x7f1dbc06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:54.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.151+0000 7f1ddcb8f700 1 --2- 172.21.15.189:0/190689061 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dbc06ade0 0x7f1dbc06d2a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc000c00 tx=0x7f1dcc008030 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:54.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.154+0000 7f1dd57fa700 1 -- 172.21.15.189:0/190689061 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1dc4054040 con 0x7f1dd80cf900 2024-04-06T12:31:54.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.349+0000 7f1ddedf3700 1 -- 172.21.15.189:0/190689061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f1dd81537b0 con 0x7f1dd80cf900 2024-04-06T12:31:54.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.352+0000 7f1dd57fa700 1 -- 172.21.15.189:0/190689061 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v50) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1dc401e0b0 con 0x7f1dd80cf900 2024-04-06T12:31:54.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.355+0000 7f1ddedf3700 1 -- 172.21.15.189:0/190689061 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dbc06ade0 msgr2=0x7f1dbc06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:54.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.355+0000 7f1ddedf3700 1 --2- 172.21.15.189:0/190689061 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dbc06ade0 0x7f1dbc06d2a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc000c00 tx=0x7f1dcc008030 comp rx=0 tx=0).stop 2024-04-06T12:31:54.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.355+0000 7f1ddedf3700 1 -- 172.21.15.189:0/190689061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dd80cf900 msgr2=0x7f1dd80ae790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:54.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.355+0000 7f1ddedf3700 1 --2- 172.21.15.189:0/190689061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dd80cf900 0x7f1dd80ae790 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f1dc400c7f0 tx=0x7f1dc400cb00 comp rx=0 tx=0).stop 2024-04-06T12:31:54.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.355+0000 7f1ddedf3700 1 -- 172.21.15.189:0/190689061 shutdown_connections 2024-04-06T12:31:54.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.355+0000 7f1ddedf3700 1 --2- 172.21.15.189:0/190689061 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1dbc06ade0 0x7f1dbc06d2a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.355+0000 7f1ddedf3700 1 --2- 172.21.15.189:0/190689061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1dd81503f0 0x7f1dd8158120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.355+0000 7f1ddedf3700 1 --2- 172.21.15.189:0/190689061 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1dd80b4ba0 0x7f1dd814ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.355+0000 7f1ddedf3700 1 --2- 172.21.15.189:0/190689061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dd80cf900 0x7f1dd80ae790 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.355+0000 7f1ddedf3700 1 -- 172.21.15.189:0/190689061 >> 172.21.15.189:0/190689061 conn(0x7f1dd80d00b0 msgr2=0x7f1dd80b0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:54.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.355+0000 7f1ddedf3700 1 -- 172.21.15.189:0/190689061 shutdown_connections 2024-04-06T12:31:54.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.355+0000 7f1ddedf3700 1 -- 172.21.15.189:0/190689061 wait complete. 2024-04-06T12:31:54.365 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-06T12:31:54.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.493+0000 7fc7cc0d5700 1 Processor -- start 2024-04-06T12:31:54.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.493+0000 7fc7cc0d5700 1 -- start start 2024-04-06T12:31:54.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.494+0000 7fc7cc0d5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7c406ae40 0x7fc7c415bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:54.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.494+0000 7fc7cc0d5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc7c415c100 0x7fc7c40ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:54.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.494+0000 7fc7cc0d5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c415f9e0 con 0x7fc7c4158fe0 2024-04-06T12:31:54.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.494+0000 7fc7cc0d5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7c415f0d0 con 0x7fc7c415c100 2024-04-06T12:31:54.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.494+0000 7fc7cc0d5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c415c8f0 con 0x7fc7c406ae40 2024-04-06T12:31:54.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.494+0000 7fc7ca672700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc7c415c100 0x7fc7c40ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:54.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.494+0000 7fc7ca672700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc7c415c100 0x7fc7c40ceca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47340/0 (socket says 172.21.15.189:47340) 2024-04-06T12:31:54.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.494+0000 7fc7ca672700 1 -- 172.21.15.189:0/2059181463 learned_addr learned my addr 172.21.15.189:0/2059181463 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:54.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.494+0000 7fc7c9670700 1 --2- 172.21.15.189:0/2059181463 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7c406ae40 0x7fc7c415bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:54.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.494+0000 7fc7ca672700 1 -- 172.21.15.189:0/2059181463 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7c406ae40 msgr2=0x7fc7c415bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:54.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.494+0000 7fc7ca672700 1 --2- 172.21.15.189:0/2059181463 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7c406ae40 0x7fc7c415bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.494+0000 7fc7ca672700 1 -- 172.21.15.189:0/2059181463 >> v1:172.21.15.189:6789/0 conn(0x7fc7c4158fe0 legacy=0x7fc7c406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:54.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.494+0000 7fc7ca672700 1 -- 172.21.15.189:0/2059181463 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c415f390 con 0x7fc7c415c100 2024-04-06T12:31:54.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.494+0000 7fc7ca672700 1 --2- 172.21.15.189:0/2059181463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7c415c100 0x7fc7c40ceca0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc009e50 tx=0x7fc7bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93663517e8bb529 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:54.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.494+0000 7fc7c8e6f700 1 -- 172.21.15.189:0/2059181463 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc00e070 con 0x7fc7c415c100 2024-04-06T12:31:54.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.495+0000 7fc7c8e6f700 1 -- 172.21.15.189:0/2059181463 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7bc00b910 con 0x7fc7c415c100 2024-04-06T12:31:54.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.495+0000 7fc7c8e6f700 1 -- 172.21.15.189:0/2059181463 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc00bcc0 con 0x7fc7c415c100 2024-04-06T12:31:54.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.495+0000 7fc7cc0d5700 1 -- 172.21.15.189:0/2059181463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7c415c100 msgr2=0x7fc7c40ceca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:54.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.495+0000 7fc7cc0d5700 1 --2- 172.21.15.189:0/2059181463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7c415c100 0x7fc7c40ceca0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc009e50 tx=0x7fc7bc00b330 comp rx=0 tx=0).stop 2024-04-06T12:31:54.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.495+0000 7fc7cc0d5700 1 -- 172.21.15.189:0/2059181463 shutdown_connections 2024-04-06T12:31:54.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.495+0000 7fc7cc0d5700 1 --2- 172.21.15.189:0/2059181463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7c415c100 0x7fc7c40ceca0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.495+0000 7fc7cc0d5700 1 --2- 172.21.15.189:0/2059181463 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7c406ae40 0x7fc7c415bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.495+0000 7fc7cc0d5700 1 -- 172.21.15.189:0/2059181463 >> 172.21.15.189:0/2059181463 conn(0x7fc7c40d00b0 msgr2=0x7fc7c40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:54.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.495+0000 7fc7cc0d5700 1 -- 172.21.15.189:0/2059181463 shutdown_connections 2024-04-06T12:31:54.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.495+0000 7fc7cc0d5700 1 -- 172.21.15.189:0/2059181463 wait complete. 2024-04-06T12:31:54.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.496+0000 7fc7cc0d5700 1 Processor -- start 2024-04-06T12:31:54.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.496+0000 7fc7cc0d5700 1 -- start start 2024-04-06T12:31:54.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.497+0000 7fc7cc0d5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7c406ae40 0x7fc7c41411b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:54.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.497+0000 7fc7cc0d5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7c4158fe0 0x7fc7c41416f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:54.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.497+0000 7fc7cc0d5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7c415c100 0x7fc7c4141c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:54.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.497+0000 7fc7cc0d5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c4158030 con 0x7fc7c406ae40 2024-04-06T12:31:54.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.497+0000 7fc7cc0d5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c4140d20 con 0x7fc7c415c100 2024-04-06T12:31:54.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.497+0000 7fc7cc0d5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c415a870 con 0x7fc7c4158fe0 2024-04-06T12:31:54.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.497+0000 7fc7c9e71700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7c406ae40 0x7fc7c41411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:54.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.497+0000 7fc7c9e71700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7c406ae40 0x7fc7c41411b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47348/0 (socket says 172.21.15.189:47348) 2024-04-06T12:31:54.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.497+0000 7fc7c9e71700 1 -- 172.21.15.189:0/1066217791 learned_addr learned my addr 172.21.15.189:0/1066217791 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:54.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.497+0000 7fc7ca672700 1 --2- 172.21.15.189:0/1066217791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7c415c100 0x7fc7c4141c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:54.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.497+0000 7fc7c9670700 1 --2- 172.21.15.189:0/1066217791 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7c4158fe0 0x7fc7c41416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:54.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.497+0000 7fc7c9e71700 1 -- 172.21.15.189:0/1066217791 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7c4158fe0 msgr2=0x7fc7c41416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:54.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.497+0000 7fc7c9e71700 1 --2- 172.21.15.189:0/1066217791 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7c4158fe0 0x7fc7c41416f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.497+0000 7fc7c9e71700 1 -- 172.21.15.189:0/1066217791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7c415c100 msgr2=0x7fc7c4141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:54.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.498+0000 7fc7c9e71700 1 --2- 172.21.15.189:0/1066217791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7c415c100 0x7fc7c4141c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.498+0000 7fc7c9e71700 1 -- 172.21.15.189:0/1066217791 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7bc009b30 con 0x7fc7c406ae40 2024-04-06T12:31:54.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.498+0000 7fc7c9e71700 1 --2- 172.21.15.189:0/1066217791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7c406ae40 0x7fc7c41411b0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000c4e0 tx=0x7fc7c000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:54.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.498+0000 7fc7ba7fc700 1 -- 172.21.15.189:0/1066217791 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c00052d0 con 0x7fc7c406ae40 2024-04-06T12:31:54.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.498+0000 7fc7ba7fc700 1 -- 172.21.15.189:0/1066217791 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c0005430 con 0x7fc7c406ae40 2024-04-06T12:31:54.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.498+0000 7fc7cc0d5700 1 -- 172.21.15.189:0/1066217791 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c415ab50 con 0x7fc7c406ae40 2024-04-06T12:31:54.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.498+0000 7fc7ba7fc700 1 -- 172.21.15.189:0/1066217791 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c00038c0 con 0x7fc7c406ae40 2024-04-06T12:31:54.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.498+0000 7fc7cc0d5700 1 -- 172.21.15.189:0/1066217791 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c415b110 con 0x7fc7c406ae40 2024-04-06T12:31:54.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.499+0000 7fc7cc0d5700 1 -- 172.21.15.189:0/1066217791 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c4067bb0 con 0x7fc7c406ae40 2024-04-06T12:31:54.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.499+0000 7fc7ba7fc700 1 -- 172.21.15.189:0/1066217791 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fc7c0003a20 con 0x7fc7c406ae40 2024-04-06T12:31:54.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.500+0000 7fc7ba7fc700 1 --2- 172.21.15.189:0/1066217791 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc7a806adb0 0x7fc7a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:54.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.500+0000 7fc7c9670700 1 --2- 172.21.15.189:0/1066217791 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc7a806adb0 0x7fc7a806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:54.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.500+0000 7fc7ba7fc700 1 -- 172.21.15.189:0/1066217791 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5908+0+0 (secure 0 0 0) 0x7fc7c0088a30 con 0x7fc7c406ae40 2024-04-06T12:31:54.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.500+0000 7fc7c9670700 1 --2- 172.21.15.189:0/1066217791 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc7a806adb0 0x7fc7a806d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4143810 tx=0x7fc7b4008040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:54.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.504+0000 7fc7ba7fc700 1 -- 172.21.15.189:0/1066217791 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc7c0053a30 con 0x7fc7c406ae40 2024-04-06T12:31:54.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.703+0000 7fc7cc0d5700 1 -- 172.21.15.189:0/1066217791 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fc7c4074990 con 0x7fc7c406ae40 2024-04-06T12:31:54.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.707+0000 7fc7ba7fc700 1 -- 172.21.15.189:0/1066217791 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v51) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc7c0057050 con 0x7fc7c406ae40 2024-04-06T12:31:54.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.709+0000 7fc7cc0d5700 1 -- 172.21.15.189:0/1066217791 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc7a806adb0 msgr2=0x7fc7a806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:54.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.709+0000 7fc7cc0d5700 1 --2- 172.21.15.189:0/1066217791 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc7a806adb0 0x7fc7a806d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4143810 tx=0x7fc7b4008040 comp rx=0 tx=0).stop 2024-04-06T12:31:54.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.709+0000 7fc7cc0d5700 1 -- 172.21.15.189:0/1066217791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7c406ae40 msgr2=0x7fc7c41411b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:54.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.709+0000 7fc7cc0d5700 1 --2- 172.21.15.189:0/1066217791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7c406ae40 0x7fc7c41411b0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000c4e0 tx=0x7fc7c000c8a0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.709+0000 7fc7cc0d5700 1 -- 172.21.15.189:0/1066217791 shutdown_connections 2024-04-06T12:31:54.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.709+0000 7fc7cc0d5700 1 --2- 172.21.15.189:0/1066217791 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc7a806adb0 0x7fc7a806d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.709+0000 7fc7cc0d5700 1 --2- 172.21.15.189:0/1066217791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7c415c100 0x7fc7c4141c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.709+0000 7fc7cc0d5700 1 --2- 172.21.15.189:0/1066217791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7c406ae40 0x7fc7c41411b0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.709+0000 7fc7cc0d5700 1 --2- 172.21.15.189:0/1066217791 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc7c4158fe0 0x7fc7c41416f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.709+0000 7fc7cc0d5700 1 -- 172.21.15.189:0/1066217791 >> 172.21.15.189:0/1066217791 conn(0x7fc7c40d00b0 msgr2=0x7fc7c406fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:54.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.709+0000 7fc7cc0d5700 1 -- 172.21.15.189:0/1066217791 shutdown_connections 2024-04-06T12:31:54.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.709+0000 7fc7cc0d5700 1 -- 172.21.15.189:0/1066217791 wait complete. 2024-04-06T12:31:54.719 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2024-04-06T12:31:54.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.844+0000 7f1f0e64e700 1 Processor -- start 2024-04-06T12:31:54.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.844+0000 7f1f0e64e700 1 -- start start 2024-04-06T12:31:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.845+0000 7f1f0e64e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1f0815e8a0 0x7f1f080cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.845+0000 7f1f0e64e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f080669d0 con 0x7f1f0815fa70 2024-04-06T12:31:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.845+0000 7f1f0e64e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f0806ede0 con 0x7f1f0815eee0 2024-04-06T12:31:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.845+0000 7f1f0e64e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0815ec80 con 0x7f1f0815e8a0 2024-04-06T12:31:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.845+0000 7f1f0cbeb700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1f0815fa70 0x7f1f0806aa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40860/0 (socket says 172.21.15.189:40860) 2024-04-06T12:31:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.845+0000 7f1f0cbeb700 1 -- 172.21.15.189:0/556514241 learned_addr learned my addr 172.21.15.189:0/556514241 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.846+0000 7f1f07fff700 1 --2- 172.21.15.189:0/556514241 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1f0815e8a0 0x7f1f080cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.846+0000 7f1f06ffd700 1 -- 172.21.15.189:0/556514241 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25161554 0 0) 0x7f1f080669d0 con 0x7f1f0815fa70 2024-04-06T12:31:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.846+0000 7f1f06ffd700 1 -- 172.21.15.189:0/556514241 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ef40035b0 con 0x7f1f0815fa70 2024-04-06T12:31:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.846+0000 7f1f06ffd700 1 -- 172.21.15.189:0/556514241 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 453781196 0 0) 0x7f1ef40035b0 con 0x7f1f0815fa70 2024-04-06T12:31:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.846+0000 7f1f07fff700 1 -- 172.21.15.189:0/556514241 >> v1:172.21.15.192:6789/0 conn(0x7f1f0815eee0 legacy=0x7f1f0815f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.846+0000 7f1f07fff700 1 -- 172.21.15.189:0/556514241 >> v1:172.21.15.189:6789/0 conn(0x7f1f0815fa70 legacy=0x7f1f0806aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.846+0000 7f1f07fff700 1 -- 172.21.15.189:0/556514241 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f080666f0 con 0x7f1f0815e8a0 2024-04-06T12:31:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.846+0000 7f1f07fff700 1 --2- 172.21.15.189:0/556514241 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1f0815e8a0 0x7f1f080cbc20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1ef00098a0 tx=0x7f1ef0009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47ed1e2e20fd93ac server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.846+0000 7f1f06ffd700 1 -- 172.21.15.189:0/556514241 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f000019f0 con 0x7f1f0815e8a0 2024-04-06T12:31:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.846+0000 7f1f06ffd700 1 -- 172.21.15.189:0/556514241 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef0004030 con 0x7f1f0815e8a0 2024-04-06T12:31:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.846+0000 7f1f06ffd700 1 -- 172.21.15.189:0/556514241 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef00043a0 con 0x7f1f0815e8a0 2024-04-06T12:31:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.847+0000 7f1f0e64e700 1 -- 172.21.15.189:0/556514241 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1f0815e8a0 msgr2=0x7f1f080cbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.847+0000 7f1f0e64e700 1 --2- 172.21.15.189:0/556514241 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1f0815e8a0 0x7f1f080cbc20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1ef00098a0 tx=0x7f1ef0009bb0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.847+0000 7f1f0e64e700 1 -- 172.21.15.189:0/556514241 shutdown_connections 2024-04-06T12:31:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.847+0000 7f1f0e64e700 1 --2- 172.21.15.189:0/556514241 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1f0815e8a0 0x7f1f080cbc20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.847+0000 7f1f0e64e700 1 -- 172.21.15.189:0/556514241 >> 172.21.15.189:0/556514241 conn(0x7f1f080cc290 msgr2=0x7f1f080cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.847+0000 7f1f0e64e700 1 -- 172.21.15.189:0/556514241 shutdown_connections 2024-04-06T12:31:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.847+0000 7f1f0e64e700 1 -- 172.21.15.189:0/556514241 wait complete. 2024-04-06T12:31:54.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.848+0000 7f1f0e64e700 1 Processor -- start 2024-04-06T12:31:54.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.848+0000 7f1f0e64e700 1 -- start start 2024-04-06T12:31:54.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.849+0000 7f1f0e64e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f0815e8a0 0x7f1f081596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:54.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.849+0000 7f1f0e64e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f0815eee0 0x7f1f081f08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:54.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.849+0000 7f1f0e64e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1f0815fa70 0x7f1f080c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:54.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.849+0000 7f1f0e64e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f080c6600 con 0x7f1f0815e8a0 2024-04-06T12:31:54.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.849+0000 7f1f0e64e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f080c8380 con 0x7f1f0815eee0 2024-04-06T12:31:54.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.849+0000 7f1f0e64e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f080c8660 con 0x7f1f0815fa70 2024-04-06T12:31:54.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.849+0000 7f1f07fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f0815e8a0 0x7f1f081596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:54.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.849+0000 7f1f07fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f0815e8a0 0x7f1f081596d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47358/0 (socket says 172.21.15.189:47358) 2024-04-06T12:31:54.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.849+0000 7f1f07fff700 1 -- 172.21.15.189:0/202430346 learned_addr learned my addr 172.21.15.189:0/202430346 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.849+0000 7f1f0cbeb700 1 --2- 172.21.15.189:0/202430346 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1f0815fa70 0x7f1f080c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.849+0000 7f1f077fe700 1 --2- 172.21.15.189:0/202430346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f0815eee0 0x7f1f081f08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.849+0000 7f1f07fff700 1 -- 172.21.15.189:0/202430346 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1f0815fa70 msgr2=0x7f1f080c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.849+0000 7f1f07fff700 1 --2- 172.21.15.189:0/202430346 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1f0815fa70 0x7f1f080c5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.849+0000 7f1f07fff700 1 -- 172.21.15.189:0/202430346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f0815eee0 msgr2=0x7f1f081f08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.849+0000 7f1f07fff700 1 --2- 172.21.15.189:0/202430346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f0815eee0 0x7f1f081f08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.850+0000 7f1f07fff700 1 -- 172.21.15.189:0/202430346 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef00094a0 con 0x7f1f0815e8a0 2024-04-06T12:31:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.850+0000 7f1f077fe700 1 --2- 172.21.15.189:0/202430346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f0815eee0 0x7f1f081f08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.850+0000 7f1f07fff700 1 --2- 172.21.15.189:0/202430346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f0815e8a0 0x7f1f081596d0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0000c00 tx=0x7f1ef0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.850+0000 7f1f04ff9700 1 -- 172.21.15.189:0/202430346 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0010590 con 0x7f1f0815e8a0 2024-04-06T12:31:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.850+0000 7f1f04ff9700 1 -- 172.21.15.189:0/202430346 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef00106f0 con 0x7f1f0815e8a0 2024-04-06T12:31:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.850+0000 7f1f0e64e700 1 -- 172.21.15.189:0/202430346 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f080c8940 con 0x7f1f0815e8a0 2024-04-06T12:31:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.850+0000 7f1f04ff9700 1 -- 172.21.15.189:0/202430346 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0010a60 con 0x7f1f0815e8a0 2024-04-06T12:31:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.850+0000 7f1f0e64e700 1 -- 172.21.15.189:0/202430346 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f080c7d40 con 0x7f1f0815e8a0 2024-04-06T12:31:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.851+0000 7f1f0e64e700 1 -- 172.21.15.189:0/202430346 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f080c8e60 con 0x7f1f0815e8a0 2024-04-06T12:31:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.851+0000 7f1f04ff9700 1 -- 172.21.15.189:0/202430346 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f1ef0010bc0 con 0x7f1f0815e8a0 2024-04-06T12:31:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.851+0000 7f1f04ff9700 1 --2- 172.21.15.189:0/202430346 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1ee806adb0 0x7f1ee806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.852+0000 7f1f04ff9700 1 -- 172.21.15.189:0/202430346 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f1ef008af10 con 0x7f1f0815e8a0 2024-04-06T12:31:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.852+0000 7f1f077fe700 1 --2- 172.21.15.189:0/202430346 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1ee806adb0 0x7f1ee806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:54.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.852+0000 7f1f077fe700 1 --2- 172.21.15.189:0/202430346 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1ee806adb0 0x7f1ee806d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8007fd0 tx=0x7f1ef8009040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:54.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:54.856+0000 7f1f04ff9700 1 -- 172.21.15.189:0/202430346 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1f080c8e60 con 0x7f1f0815e8a0 2024-04-06T12:31:55.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.133+0000 7f1f0e64e700 1 -- 172.21.15.189:0/202430346 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7f1f080c8e60 con 0x7f1f0815e8a0 2024-04-06T12:31:55.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.632+0000 7f1f04ff9700 1 -- 172.21.15.189:0/202430346 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v765) v1 ==== 160+0+0 (secure 0 0 0) 0x7f1f080c8e60 con 0x7f1f0815e8a0 2024-04-06T12:31:55.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.634+0000 7f1f0e64e700 1 -- 172.21.15.189:0/202430346 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1ee806adb0 msgr2=0x7f1ee806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:55.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.634+0000 7f1f0e64e700 1 --2- 172.21.15.189:0/202430346 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1ee806adb0 0x7f1ee806d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8007fd0 tx=0x7f1ef8009040 comp rx=0 tx=0).stop 2024-04-06T12:31:55.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.634+0000 7f1f0e64e700 1 -- 172.21.15.189:0/202430346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f0815e8a0 msgr2=0x7f1f081596d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:55.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.634+0000 7f1f0e64e700 1 --2- 172.21.15.189:0/202430346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f0815e8a0 0x7f1f081596d0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0000c00 tx=0x7f1ef0004750 comp rx=0 tx=0).stop 2024-04-06T12:31:55.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.634+0000 7f1f0e64e700 1 -- 172.21.15.189:0/202430346 shutdown_connections 2024-04-06T12:31:55.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.634+0000 7f1f0e64e700 1 --2- 172.21.15.189:0/202430346 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1f0815fa70 0x7f1f080c5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:55.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.634+0000 7f1f0e64e700 1 --2- 172.21.15.189:0/202430346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f0815e8a0 0x7f1f081596d0 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:55.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.634+0000 7f1f0e64e700 1 --2- 172.21.15.189:0/202430346 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1ee806adb0 0x7f1ee806d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:55.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.634+0000 7f1f0e64e700 1 --2- 172.21.15.189:0/202430346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f0815eee0 0x7f1f081f08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:55.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.634+0000 7f1f0e64e700 1 -- 172.21.15.189:0/202430346 >> 172.21.15.189:0/202430346 conn(0x7f1f080cc290 msgr2=0x7f1f0806b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:55.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.634+0000 7f1f0e64e700 1 -- 172.21.15.189:0/202430346 shutdown_connections 2024-04-06T12:31:55.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.634+0000 7f1f0e64e700 1 -- 172.21.15.189:0/202430346 wait complete. 2024-04-06T12:31:55.646 INFO:tasks.cephfs_test_runner:test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) ... ok 2024-04-06T12:31:55.647 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that number of snaps retained are as per user spec. 2024-04-06T12:31:55.647 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-06T12:31:55.647 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2024-04-06T12:31:55.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.777+0000 7fd9662f1700 1 Processor -- start 2024-04-06T12:31:55.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.777+0000 7fd9662f1700 1 -- start start 2024-04-06T12:31:55.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.778+0000 7fd9662f1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd96015a550 0x7fd9600cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:55.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.778+0000 7fd9662f1700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd9600cf900 0x7fd9600b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:55.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.778+0000 7fd9662f1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9600b4ba0 0x7fd96006eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:55.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.778+0000 7fd9662f1700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9600cdbe0 con 0x7fd96015a550 2024-04-06T12:31:55.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.778+0000 7fd9662f1700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9600b4ff0 con 0x7fd9600cf900 2024-04-06T12:31:55.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.778+0000 7fd9662f1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96015a940 con 0x7fd9600b4ba0 2024-04-06T12:31:55.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.778+0000 7fd95ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd96015a550 0x7fd9600cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:55.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.778+0000 7fd95ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd96015a550 0x7fd9600cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47364/0 (socket says 172.21.15.189:47364) 2024-04-06T12:31:55.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.778+0000 7fd95ffff700 1 -- 172.21.15.189:0/3176812437 learned_addr learned my addr 172.21.15.189:0/3176812437 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:55.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.779+0000 7fd96488e700 1 --2- 172.21.15.189:0/3176812437 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9600b4ba0 0x7fd96006eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:55.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.779+0000 7fd95f7fe700 1 --2- 172.21.15.189:0/3176812437 >> v2:172.21.15.192:3300/0 conn(0x7fd9600cf900 0x7fd9600b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:55.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.779+0000 7fd95ffff700 1 -- 172.21.15.189:0/3176812437 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9600b4ba0 msgr2=0x7fd96006eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:55.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.779+0000 7fd95ffff700 1 --2- 172.21.15.189:0/3176812437 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9600b4ba0 0x7fd96006eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:55.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.779+0000 7fd95ffff700 1 -- 172.21.15.189:0/3176812437 >> v2:172.21.15.192:3300/0 conn(0x7fd9600cf900 msgr2=0x7fd9600b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:55.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.779+0000 7fd95ffff700 1 --2- 172.21.15.189:0/3176812437 >> v2:172.21.15.192:3300/0 conn(0x7fd9600cf900 0x7fd9600b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:55.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.779+0000 7fd95ffff700 1 -- 172.21.15.189:0/3176812437 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96015ac00 con 0x7fd96015a550 2024-04-06T12:31:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.779+0000 7fd96488e700 1 --2- 172.21.15.189:0/3176812437 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9600b4ba0 0x7fd96006eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.779+0000 7fd95ffff700 1 --2- 172.21.15.189:0/3176812437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd96015a550 0x7fd9600cf3c0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fd9600b2d40 tx=0x7fd954009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196e3893c0d07285 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.779+0000 7fd95effd700 1 -- 172.21.15.189:0/3176812437 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95400b870 con 0x7fd96015a550 2024-04-06T12:31:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.779+0000 7fd95effd700 1 -- 172.21.15.189:0/3176812437 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95400b9d0 con 0x7fd96015a550 2024-04-06T12:31:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.779+0000 7fd95effd700 1 -- 172.21.15.189:0/3176812437 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95400bd60 con 0x7fd96015a550 2024-04-06T12:31:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.779+0000 7fd9662f1700 1 -- 172.21.15.189:0/3176812437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd96015a550 msgr2=0x7fd9600cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.779+0000 7fd9662f1700 1 --2- 172.21.15.189:0/3176812437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd96015a550 0x7fd9600cf3c0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fd9600b2d40 tx=0x7fd954009bd0 comp rx=0 tx=0).stop 2024-04-06T12:31:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.780+0000 7fd9662f1700 1 -- 172.21.15.189:0/3176812437 shutdown_connections 2024-04-06T12:31:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.780+0000 7fd9662f1700 1 --2- 172.21.15.189:0/3176812437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd96015a550 0x7fd9600cf3c0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.780+0000 7fd9662f1700 1 --2- 172.21.15.189:0/3176812437 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9600b4ba0 0x7fd96006eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.780+0000 7fd9662f1700 1 --2- 172.21.15.189:0/3176812437 >> v2:172.21.15.192:3300/0 conn(0x7fd9600cf900 0x7fd9600b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:55.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.780+0000 7fd9662f1700 1 -- 172.21.15.189:0/3176812437 >> 172.21.15.189:0/3176812437 conn(0x7fd9600d00b0 msgr2=0x7fd9600d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:55.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.780+0000 7fd9662f1700 1 -- 172.21.15.189:0/3176812437 shutdown_connections 2024-04-06T12:31:55.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.780+0000 7fd9662f1700 1 -- 172.21.15.189:0/3176812437 wait complete. 2024-04-06T12:31:55.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.781+0000 7fd9662f1700 1 Processor -- start 2024-04-06T12:31:55.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.781+0000 7fd9662f1700 1 -- start start 2024-04-06T12:31:55.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.782+0000 7fd9662f1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9600b4ba0 0x7fd960150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:55.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.782+0000 7fd9662f1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9600cf900 0x7fd96014dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.782+0000 7fd9662f1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd96014e240 0x7fd96015b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.782+0000 7fd9662f1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96014f640 con 0x7fd96014e240 2024-04-06T12:31:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.782+0000 7fd9662f1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96014eac0 con 0x7fd9600b4ba0 2024-04-06T12:31:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.782+0000 7fd9662f1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd960145e50 con 0x7fd9600cf900 2024-04-06T12:31:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.782+0000 7fd96488e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd96014e240 0x7fd96015b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.782+0000 7fd96488e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd96014e240 0x7fd96015b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47380/0 (socket says 172.21.15.189:47380) 2024-04-06T12:31:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.782+0000 7fd96488e700 1 -- 172.21.15.189:0/3604538138 learned_addr learned my addr 172.21.15.189:0/3604538138 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.782+0000 7fd95ffff700 1 --2- 172.21.15.189:0/3604538138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9600b4ba0 0x7fd960150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.783+0000 7fd95f7fe700 1 --2- 172.21.15.189:0/3604538138 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9600cf900 0x7fd96014dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.783+0000 7fd96488e700 1 -- 172.21.15.189:0/3604538138 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9600cf900 msgr2=0x7fd96014dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.783+0000 7fd96488e700 1 --2- 172.21.15.189:0/3604538138 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9600cf900 0x7fd96014dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.783+0000 7fd96488e700 1 -- 172.21.15.189:0/3604538138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9600b4ba0 msgr2=0x7fd960150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.783+0000 7fd96488e700 1 --2- 172.21.15.189:0/3604538138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9600b4ba0 0x7fd960150fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.783+0000 7fd96488e700 1 -- 172.21.15.189:0/3604538138 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd954009550 con 0x7fd96014e240 2024-04-06T12:31:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.783+0000 7fd95f7fe700 1 --2- 172.21.15.189:0/3604538138 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9600cf900 0x7fd96014dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:55.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.783+0000 7fd96488e700 1 --2- 172.21.15.189:0/3604538138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd96014e240 0x7fd96015b810 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fd958007db0 tx=0x7fd95800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:55.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.783+0000 7fd95cff9700 1 -- 172.21.15.189:0/3604538138 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95800ddd0 con 0x7fd96014e240 2024-04-06T12:31:55.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.783+0000 7fd95cff9700 1 -- 172.21.15.189:0/3604538138 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95800f040 con 0x7fd96014e240 2024-04-06T12:31:55.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.783+0000 7fd95cff9700 1 -- 172.21.15.189:0/3604538138 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd958003c60 con 0x7fd96014e240 2024-04-06T12:31:55.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.783+0000 7fd9662f1700 1 -- 172.21.15.189:0/3604538138 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96014ed10 con 0x7fd96014e240 2024-04-06T12:31:55.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.783+0000 7fd9662f1700 1 -- 172.21.15.189:0/3604538138 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96014f2d0 con 0x7fd96014e240 2024-04-06T12:31:55.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.784+0000 7fd9662f1700 1 -- 172.21.15.189:0/3604538138 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd96014f7f0 con 0x7fd96014e240 2024-04-06T12:31:55.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.785+0000 7fd95cff9700 1 -- 172.21.15.189:0/3604538138 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fd958004030 con 0x7fd96014e240 2024-04-06T12:31:55.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.785+0000 7fd95cff9700 1 --2- 172.21.15.189:0/3604538138 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd94c06adb0 0x7fd94c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:55.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.785+0000 7fd95cff9700 1 -- 172.21.15.189:0/3604538138 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5908+0+0 (secure 0 0 0) 0x7fd958089630 con 0x7fd96014e240 2024-04-06T12:31:55.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.785+0000 7fd95ffff700 1 --2- 172.21.15.189:0/3604538138 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd94c06adb0 0x7fd94c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:55.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.785+0000 7fd95ffff700 1 --2- 172.21.15.189:0/3604538138 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd94c06adb0 0x7fd94c06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd954000c00 tx=0x7fd954008030 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:55.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:55.789+0000 7fd95cff9700 1 -- 172.21.15.189:0/3604538138 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd96014f7f0 con 0x7fd96014e240 2024-04-06T12:31:56.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.063+0000 7fd9662f1700 1 -- 172.21.15.189:0/3604538138 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7fd96014f7f0 con 0x7fd96014e240 2024-04-06T12:31:56.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.634+0000 7fd95cff9700 1 -- 172.21.15.189:0/3604538138 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v766) v1 ==== 181+0+0 (secure 0 0 0) 0x7fd96014f7f0 con 0x7fd96014e240 2024-04-06T12:31:56.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.637+0000 7fd9662f1700 1 -- 172.21.15.189:0/3604538138 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd94c06adb0 msgr2=0x7fd94c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:56.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.637+0000 7fd9662f1700 1 --2- 172.21.15.189:0/3604538138 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd94c06adb0 0x7fd94c06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd954000c00 tx=0x7fd954008030 comp rx=0 tx=0).stop 2024-04-06T12:31:56.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.637+0000 7fd9662f1700 1 -- 172.21.15.189:0/3604538138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd96014e240 msgr2=0x7fd96015b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:56.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.637+0000 7fd9662f1700 1 --2- 172.21.15.189:0/3604538138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd96014e240 0x7fd96015b810 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fd958007db0 tx=0x7fd95800d3b0 comp rx=0 tx=0).stop 2024-04-06T12:31:56.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.637+0000 7fd9662f1700 1 -- 172.21.15.189:0/3604538138 shutdown_connections 2024-04-06T12:31:56.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.637+0000 7fd9662f1700 1 --2- 172.21.15.189:0/3604538138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd96014e240 0x7fd96015b810 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:56.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.637+0000 7fd9662f1700 1 --2- 172.21.15.189:0/3604538138 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd94c06adb0 0x7fd94c06d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:56.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.637+0000 7fd9662f1700 1 --2- 172.21.15.189:0/3604538138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9600b4ba0 0x7fd960150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:56.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.637+0000 7fd9662f1700 1 --2- 172.21.15.189:0/3604538138 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9600cf900 0x7fd96014dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:56.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.637+0000 7fd9662f1700 1 -- 172.21.15.189:0/3604538138 >> 172.21.15.189:0/3604538138 conn(0x7fd9600d00b0 msgr2=0x7fd96006adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:56.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.637+0000 7fd9662f1700 1 -- 172.21.15.189:0/3604538138 shutdown_connections 2024-04-06T12:31:56.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.637+0000 7fd9662f1700 1 -- 172.21.15.189:0/3604538138 wait complete. 2024-04-06T12:31:56.648 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-06T12:31:56.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.779+0000 7fb78ef0f700 1 Processor -- start 2024-04-06T12:31:56.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.779+0000 7fb78ef0f700 1 -- start start 2024-04-06T12:31:56.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.780+0000 7fb78ef0f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb788144f40 0x7fb78806a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:56.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.780+0000 7fb78ef0f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb78815fcd0 0x7fb78806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:56.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.780+0000 7fb78ef0f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78815c740 con 0x7fb78815ba40 2024-04-06T12:31:56.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.780+0000 7fb78ef0f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7880cfcb0 con 0x7fb78815fcd0 2024-04-06T12:31:56.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.780+0000 7fb78ef0f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb78815ca20 con 0x7fb788144f40 2024-04-06T12:31:56.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.780+0000 7fb787fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb78815fcd0 0x7fb78806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:56.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.780+0000 7fb787fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb78815fcd0 0x7fb78806ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47390/0 (socket says 172.21.15.189:47390) 2024-04-06T12:31:56.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.780+0000 7fb787fff700 1 -- 172.21.15.189:0/2554003887 learned_addr learned my addr 172.21.15.189:0/2554003887 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:56.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.780+0000 7fb787fff700 1 -- 172.21.15.189:0/2554003887 >> v2:172.21.15.192:3300/0 conn(0x7fb788144f40 msgr2=0x7fb78806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:56.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.780+0000 7fb787fff700 1 --2- 172.21.15.189:0/2554003887 >> v2:172.21.15.192:3300/0 conn(0x7fb788144f40 0x7fb78806a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:56.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.780+0000 7fb787fff700 1 -- 172.21.15.189:0/2554003887 >> v1:172.21.15.192:6789/0 conn(0x7fb78815ba40 legacy=0x7fb78815be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:56.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.780+0000 7fb787fff700 1 -- 172.21.15.189:0/2554003887 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78815f1f0 con 0x7fb78815fcd0 2024-04-06T12:31:56.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.780+0000 7fb787fff700 1 --2- 172.21.15.189:0/2554003887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb78815fcd0 0x7fb78806ac70 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fb788073470 tx=0x7fb774009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82cc1b39c0f09cd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:56.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.780+0000 7fb7877fe700 1 -- 172.21.15.189:0/2554003887 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb774004230 con 0x7fb78815fcd0 2024-04-06T12:31:56.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.781+0000 7fb7877fe700 1 -- 172.21.15.189:0/2554003887 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb774004390 con 0x7fb78815fcd0 2024-04-06T12:31:56.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.781+0000 7fb7877fe700 1 -- 172.21.15.189:0/2554003887 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7740036a0 con 0x7fb78815fcd0 2024-04-06T12:31:56.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.781+0000 7fb78ef0f700 1 -- 172.21.15.189:0/2554003887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb78815fcd0 msgr2=0x7fb78806ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:56.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.781+0000 7fb78ef0f700 1 --2- 172.21.15.189:0/2554003887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb78815fcd0 0x7fb78806ac70 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fb788073470 tx=0x7fb774009bd0 comp rx=0 tx=0).stop 2024-04-06T12:31:56.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.781+0000 7fb78ef0f700 1 -- 172.21.15.189:0/2554003887 shutdown_connections 2024-04-06T12:31:56.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.781+0000 7fb78ef0f700 1 --2- 172.21.15.189:0/2554003887 >> v2:172.21.15.192:3300/0 conn(0x7fb788144f40 0x7fb78806a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:56.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.781+0000 7fb78ef0f700 1 --2- 172.21.15.189:0/2554003887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb78815fcd0 0x7fb78806ac70 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:56.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.781+0000 7fb78ef0f700 1 -- 172.21.15.189:0/2554003887 >> 172.21.15.189:0/2554003887 conn(0x7fb7880d00b0 msgr2=0x7fb7880d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:56.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.781+0000 7fb78ef0f700 1 -- 172.21.15.189:0/2554003887 shutdown_connections 2024-04-06T12:31:56.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.781+0000 7fb78ef0f700 1 -- 172.21.15.189:0/2554003887 wait complete. 2024-04-06T12:31:56.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.782+0000 7fb78ef0f700 1 Processor -- start 2024-04-06T12:31:56.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.782+0000 7fb78ef0f700 1 -- start start 2024-04-06T12:31:56.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.783+0000 7fb78ef0f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb788144f40 0x7fb78813e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:56.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.783+0000 7fb78ef0f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb78815ba40 0x7fb78814f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:56.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.783+0000 7fb78ef0f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb78815fcd0 0x7fb7880cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:56.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.783+0000 7fb78ef0f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7881f0930 con 0x7fb78815ba40 2024-04-06T12:31:56.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.783+0000 7fb78ef0f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7881f3c90 con 0x7fb788144f40 2024-04-06T12:31:56.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.783+0000 7fb78ef0f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78813ebb0 con 0x7fb78815fcd0 2024-04-06T12:31:56.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.783+0000 7fb787fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb78815ba40 0x7fb78814f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:56.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.783+0000 7fb787fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb78815ba40 0x7fb78814f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47392/0 (socket says 172.21.15.189:47392) 2024-04-06T12:31:56.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.783+0000 7fb787fff700 1 -- 172.21.15.189:0/1047030025 learned_addr learned my addr 172.21.15.189:0/1047030025 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:56.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.783+0000 7fb78ccab700 1 --2- 172.21.15.189:0/1047030025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb788144f40 0x7fb78813e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:56.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.783+0000 7fb78d4ac700 1 --2- 172.21.15.189:0/1047030025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb78815fcd0 0x7fb7880cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:56.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.783+0000 7fb787fff700 1 -- 172.21.15.189:0/1047030025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb78815fcd0 msgr2=0x7fb7880cb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:56.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.784+0000 7fb787fff700 1 --2- 172.21.15.189:0/1047030025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb78815fcd0 0x7fb7880cb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:56.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.784+0000 7fb787fff700 1 -- 172.21.15.189:0/1047030025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb788144f40 msgr2=0x7fb78813e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:56.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.784+0000 7fb787fff700 1 --2- 172.21.15.189:0/1047030025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb788144f40 0x7fb78813e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:56.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.784+0000 7fb787fff700 1 -- 172.21.15.189:0/1047030025 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb774009550 con 0x7fb78815ba40 2024-04-06T12:31:56.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.784+0000 7fb78ccab700 1 --2- 172.21.15.189:0/1047030025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb788144f40 0x7fb78813e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:56.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.784+0000 7fb78d4ac700 1 --2- 172.21.15.189:0/1047030025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb78815fcd0 0x7fb7880cb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:56.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.784+0000 7fb787fff700 1 --2- 172.21.15.189:0/1047030025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb78815ba40 0x7fb78814f040 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fb774009130 tx=0x7fb774004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:56.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.784+0000 7fb7857fa700 1 -- 172.21.15.189:0/1047030025 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb774011630 con 0x7fb78815ba40 2024-04-06T12:31:56.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.784+0000 7fb7857fa700 1 -- 172.21.15.189:0/1047030025 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb774011790 con 0x7fb78815ba40 2024-04-06T12:31:56.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.784+0000 7fb7857fa700 1 -- 172.21.15.189:0/1047030025 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb774011b40 con 0x7fb78815ba40 2024-04-06T12:31:56.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.784+0000 7fb78ef0f700 1 -- 172.21.15.189:0/1047030025 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78814f990 con 0x7fb78815ba40 2024-04-06T12:31:56.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.784+0000 7fb78ef0f700 1 -- 172.21.15.189:0/1047030025 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb78814ff50 con 0x7fb78815ba40 2024-04-06T12:31:56.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.785+0000 7fb78ef0f700 1 -- 172.21.15.189:0/1047030025 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7881f0aa0 con 0x7fb78815ba40 2024-04-06T12:31:56.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.785+0000 7fb7857fa700 1 -- 172.21.15.189:0/1047030025 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fb774011ca0 con 0x7fb78815ba40 2024-04-06T12:31:56.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.786+0000 7fb7857fa700 1 --2- 172.21.15.189:0/1047030025 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb76c06adb0 0x7fb76c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:56.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.786+0000 7fb78ccab700 1 --2- 172.21.15.189:0/1047030025 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb76c06adb0 0x7fb76c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:56.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.786+0000 7fb7857fa700 1 -- 172.21.15.189:0/1047030025 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5908+0+0 (secure 0 0 0) 0x7fb77408abd0 con 0x7fb78815ba40 2024-04-06T12:31:56.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.786+0000 7fb78ccab700 1 --2- 172.21.15.189:0/1047030025 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb76c06adb0 0x7fb76c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb77c007900 tx=0x7fb77c008040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:56.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.790+0000 7fb7857fa700 1 -- 172.21.15.189:0/1047030025 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb7881f0aa0 con 0x7fb78815ba40 2024-04-06T12:31:56.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.983+0000 7fb78ef0f700 1 -- 172.21.15.189:0/1047030025 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb788067bb0 con 0x7fb78815ba40 2024-04-06T12:31:56.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.989+0000 7fb7857fa700 1 -- 172.21.15.189:0/1047030025 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb774055c60 con 0x7fb78815ba40 2024-04-06T12:31:56.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.991+0000 7fb78ef0f700 1 -- 172.21.15.189:0/1047030025 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb76c06adb0 msgr2=0x7fb76c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:56.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.991+0000 7fb78ef0f700 1 --2- 172.21.15.189:0/1047030025 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb76c06adb0 0x7fb76c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb77c007900 tx=0x7fb77c008040 comp rx=0 tx=0).stop 2024-04-06T12:31:56.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.991+0000 7fb78ef0f700 1 -- 172.21.15.189:0/1047030025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb78815ba40 msgr2=0x7fb78814f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:56.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.991+0000 7fb78ef0f700 1 --2- 172.21.15.189:0/1047030025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb78815ba40 0x7fb78814f040 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fb774009130 tx=0x7fb774004750 comp rx=0 tx=0).stop 2024-04-06T12:31:56.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.991+0000 7fb78ef0f700 1 -- 172.21.15.189:0/1047030025 shutdown_connections 2024-04-06T12:31:56.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.991+0000 7fb78ef0f700 1 --2- 172.21.15.189:0/1047030025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb78815fcd0 0x7fb7880cb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:56.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.991+0000 7fb78ef0f700 1 --2- 172.21.15.189:0/1047030025 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb76c06adb0 0x7fb76c06d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:56.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.991+0000 7fb78ef0f700 1 --2- 172.21.15.189:0/1047030025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb788144f40 0x7fb78813e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:56.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.991+0000 7fb78ef0f700 1 --2- 172.21.15.189:0/1047030025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb78815ba40 0x7fb78814f040 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:56.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.991+0000 7fb78ef0f700 1 -- 172.21.15.189:0/1047030025 >> 172.21.15.189:0/1047030025 conn(0x7fb7880d00b0 msgr2=0x7fb7880d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:56.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.991+0000 7fb78ef0f700 1 -- 172.21.15.189:0/1047030025 shutdown_connections 2024-04-06T12:31:56.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:56.991+0000 7fb78ef0f700 1 -- 172.21.15.189:0/1047030025 wait complete. 2024-04-06T12:31:57.000 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T12:31:57.001 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:31:57.044 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T12:31:57.093 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T12:31:57.094 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:31:57.094 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:31:57.111 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:31:57.112 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:31:57.112 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:31:57.132 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:31:57.132 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:31:57.132 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:31:57.169 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T12:31:57.169 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T12:31:57.169 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:31:57.191 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T12:31:57.191 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-06T12:31:57.191 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:31:57.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.333+0000 7fa374617700 1 Processor -- start 2024-04-06T12:31:57.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.333+0000 7fa374617700 1 -- start start 2024-04-06T12:31:57.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.334+0000 7fa374617700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa36c0d17a0 0x7fa36c06b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:57.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.334+0000 7fa374617700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa36c0cedf0 0x7fa36c0cf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:57.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.334+0000 7fa374617700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa36c0cf780 con 0x7fa36c146dd0 2024-04-06T12:31:57.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.334+0000 7fa374617700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa36c06d850 con 0x7fa36c0d17a0 2024-04-06T12:31:57.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.334+0000 7fa374617700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c06c0a0 con 0x7fa36c0cedf0 2024-04-06T12:31:57.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.334+0000 7fa3723b3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa36c0d17a0 0x7fa36c06b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:57.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.334+0000 7fa3723b3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa36c0d17a0 0x7fa36c06b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47394/0 (socket says 172.21.15.189:47394) 2024-04-06T12:31:57.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.334+0000 7fa3723b3700 1 -- 172.21.15.189:0/2775813801 learned_addr learned my addr 172.21.15.189:0/2775813801 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:57.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.334+0000 7fa372bb4700 1 --2- 172.21.15.189:0/2775813801 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa36c0cedf0 0x7fa36c0cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:57.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.334+0000 7fa3723b3700 1 -- 172.21.15.189:0/2775813801 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa36c0cedf0 msgr2=0x7fa36c0cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:57.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.334+0000 7fa3723b3700 1 --2- 172.21.15.189:0/2775813801 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa36c0cedf0 0x7fa36c0cf240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:57.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.334+0000 7fa3723b3700 1 -- 172.21.15.189:0/2775813801 >> v1:172.21.15.192:6789/0 conn(0x7fa36c146dd0 legacy=0x7fa36c06d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:57.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.334+0000 7fa3723b3700 1 -- 172.21.15.189:0/2775813801 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36c06c380 con 0x7fa36c0d17a0 2024-04-06T12:31:57.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.334+0000 7fa372bb4700 1 --2- 172.21.15.189:0/2775813801 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa36c0cedf0 0x7fa36c0cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:57.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.334+0000 7fa3723b3700 1 --2- 172.21.15.189:0/2775813801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa36c0d17a0 0x7fa36c06b7a0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fa3680098f0 tx=0x7fa368009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=279a6d2c78820049 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:57.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.335+0000 7fa3713b1700 1 -- 172.21.15.189:0/2775813801 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa368004230 con 0x7fa36c0d17a0 2024-04-06T12:31:57.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.335+0000 7fa3713b1700 1 -- 172.21.15.189:0/2775813801 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa368004390 con 0x7fa36c0d17a0 2024-04-06T12:31:57.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.335+0000 7fa3713b1700 1 -- 172.21.15.189:0/2775813801 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3680036a0 con 0x7fa36c0d17a0 2024-04-06T12:31:57.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.335+0000 7fa374617700 1 -- 172.21.15.189:0/2775813801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa36c0d17a0 msgr2=0x7fa36c06b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:57.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.335+0000 7fa374617700 1 --2- 172.21.15.189:0/2775813801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa36c0d17a0 0x7fa36c06b7a0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fa3680098f0 tx=0x7fa368009c00 comp rx=0 tx=0).stop 2024-04-06T12:31:57.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.335+0000 7fa374617700 1 -- 172.21.15.189:0/2775813801 shutdown_connections 2024-04-06T12:31:57.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.335+0000 7fa374617700 1 --2- 172.21.15.189:0/2775813801 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa36c0cedf0 0x7fa36c0cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:57.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.335+0000 7fa374617700 1 --2- 172.21.15.189:0/2775813801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa36c0d17a0 0x7fa36c06b7a0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:57.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.335+0000 7fa374617700 1 -- 172.21.15.189:0/2775813801 >> 172.21.15.189:0/2775813801 conn(0x7fa36c0b2db0 msgr2=0x7fa36c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:57.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.335+0000 7fa374617700 1 -- 172.21.15.189:0/2775813801 shutdown_connections 2024-04-06T12:31:57.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.335+0000 7fa374617700 1 -- 172.21.15.189:0/2775813801 wait complete. 2024-04-06T12:31:57.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.336+0000 7fa374617700 1 Processor -- start 2024-04-06T12:31:57.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.336+0000 7fa374617700 1 -- start start 2024-04-06T12:31:57.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.337+0000 7fa374617700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa36c0cedf0 0x7fa36c0c6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:57.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.337+0000 7fa374617700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa36c0d17a0 0x7fa36c0c4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:57.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.337+0000 7fa374617700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa36c146dd0 0x7fa36c0ae260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:57.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.337+0000 7fa374617700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c0c6dc0 con 0x7fa36c0cedf0 2024-04-06T12:31:57.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.337+0000 7fa374617700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c0c57e0 con 0x7fa36c146dd0 2024-04-06T12:31:57.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.337+0000 7fa374617700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c0c5ac0 con 0x7fa36c0d17a0 2024-04-06T12:31:57.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.337+0000 7fa3723b3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa36c0cedf0 0x7fa36c0c6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:57.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.337+0000 7fa3723b3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa36c0cedf0 0x7fa36c0c6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47402/0 (socket says 172.21.15.189:47402) 2024-04-06T12:31:57.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.337+0000 7fa3723b3700 1 -- 172.21.15.189:0/314319248 learned_addr learned my addr 172.21.15.189:0/314319248 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:57.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.338+0000 7fa3723b3700 1 -- 172.21.15.189:0/314319248 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa36c0d17a0 msgr2=0x7fa36c0c4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:57.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.338+0000 7fa3723b3700 1 --2- 172.21.15.189:0/314319248 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa36c0d17a0 0x7fa36c0c4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:57.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.338+0000 7fa3723b3700 1 -- 172.21.15.189:0/314319248 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa36c146dd0 msgr2=0x7fa36c0ae260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:57.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.338+0000 7fa3723b3700 1 --2- 172.21.15.189:0/314319248 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa36c146dd0 0x7fa36c0ae260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:57.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.338+0000 7fa3723b3700 1 -- 172.21.15.189:0/314319248 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa368009550 con 0x7fa36c0cedf0 2024-04-06T12:31:57.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.338+0000 7fa3723b3700 1 --2- 172.21.15.189:0/314319248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa36c0cedf0 0x7fa36c0c6880 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fa36c160fe0 tx=0x7fa368004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:57.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.338+0000 7fa362ffd700 1 -- 172.21.15.189:0/314319248 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36800b860 con 0x7fa36c0cedf0 2024-04-06T12:31:57.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.338+0000 7fa362ffd700 1 -- 172.21.15.189:0/314319248 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36800b9e0 con 0x7fa36c0cedf0 2024-04-06T12:31:57.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.338+0000 7fa374617700 1 -- 172.21.15.189:0/314319248 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa36c0c5da0 con 0x7fa36c0cedf0 2024-04-06T12:31:57.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.338+0000 7fa362ffd700 1 -- 172.21.15.189:0/314319248 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36800bd90 con 0x7fa36c0cedf0 2024-04-06T12:31:57.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.338+0000 7fa374617700 1 -- 172.21.15.189:0/314319248 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa36c0c6360 con 0x7fa36c0cedf0 2024-04-06T12:31:57.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.339+0000 7fa374617700 1 -- 172.21.15.189:0/314319248 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa36c074990 con 0x7fa36c0cedf0 2024-04-06T12:31:57.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.339+0000 7fa362ffd700 1 -- 172.21.15.189:0/314319248 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fa36801b440 con 0x7fa36c0cedf0 2024-04-06T12:31:57.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.340+0000 7fa362ffd700 1 --2- 172.21.15.189:0/314319248 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa35006adb0 0x7fa35006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:57.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.340+0000 7fa362ffd700 1 -- 172.21.15.189:0/314319248 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5908+0+0 (secure 0 0 0) 0x7fa36808ad90 con 0x7fa36c0cedf0 2024-04-06T12:31:57.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.340+0000 7fa371bb2700 1 --2- 172.21.15.189:0/314319248 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa35006adb0 0x7fa35006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:57.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.340+0000 7fa371bb2700 1 --2- 172.21.15.189:0/314319248 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa35006adb0 0x7fa35006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa36c146610 tx=0x7fa35c007a40 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:57.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.344+0000 7fa362ffd700 1 -- 172.21.15.189:0/314319248 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa368055ed0 con 0x7fa36c0cedf0 2024-04-06T12:31:57.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.586+0000 7fa374617700 1 -- 172.21.15.189:0/314319248 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa36c0cf780 con 0x7fa36c0cedf0 2024-04-06T12:31:57.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.587+0000 7fa362ffd700 1 -- 172.21.15.189:0/314319248 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+3535 (secure 0 0 0) 0x7fa36801fde0 con 0x7fa36c0cedf0 2024-04-06T12:31:57.584 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:57.584 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":64,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7115,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/538134862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":538134862},{"type":"v1","addr":"172.21.15.192:6834","nonce":538134862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7915,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1788966297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1788966297},{"type":"v1","addr":"172.21.15.189:6837","nonce":1788966297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7930,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2932927074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2932927074},{"type":"v1","addr":"172.21.15.192:6835","nonce":2932927074}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":9060,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/714340041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":714340041},{"type":"v1","addr":"172.21.15.189:6836","nonce":714340041}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":64}],"filesystems":[]} 2024-04-06T12:31:57.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.589+0000 7fa374617700 1 -- 172.21.15.189:0/314319248 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa35006adb0 msgr2=0x7fa35006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:57.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.589+0000 7fa374617700 1 --2- 172.21.15.189:0/314319248 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa35006adb0 0x7fa35006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa36c146610 tx=0x7fa35c007a40 comp rx=0 tx=0).stop 2024-04-06T12:31:57.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.589+0000 7fa374617700 1 -- 172.21.15.189:0/314319248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa36c0cedf0 msgr2=0x7fa36c0c6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:57.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.589+0000 7fa374617700 1 --2- 172.21.15.189:0/314319248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa36c0cedf0 0x7fa36c0c6880 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fa36c160fe0 tx=0x7fa368004750 comp rx=0 tx=0).stop 2024-04-06T12:31:57.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.589+0000 7fa374617700 1 -- 172.21.15.189:0/314319248 shutdown_connections 2024-04-06T12:31:57.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.589+0000 7fa374617700 1 --2- 172.21.15.189:0/314319248 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa35006adb0 0x7fa35006d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:57.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.589+0000 7fa374617700 1 --2- 172.21.15.189:0/314319248 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa36c146dd0 0x7fa36c0ae260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:57.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.589+0000 7fa374617700 1 --2- 172.21.15.189:0/314319248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa36c0cedf0 0x7fa36c0c6880 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:57.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.589+0000 7fa374617700 1 --2- 172.21.15.189:0/314319248 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa36c0d17a0 0x7fa36c0c4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:57.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.589+0000 7fa374617700 1 -- 172.21.15.189:0/314319248 >> 172.21.15.189:0/314319248 conn(0x7fa36c0b2db0 msgr2=0x7fa36c15e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:57.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.589+0000 7fa374617700 1 -- 172.21.15.189:0/314319248 shutdown_connections 2024-04-06T12:31:57.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.589+0000 7fa374617700 1 -- 172.21.15.189:0/314319248 wait complete. 2024-04-06T12:31:57.587 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 64 2024-04-06T12:31:57.599 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-06T12:31:57.599 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-06T12:31:57.599 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-06T12:31:57.599 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:31:57.600 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-06T12:31:57.600 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-06T12:31:57.600 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-06T12:31:57.600 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:31:57.600 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-06T12:31:57.600 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-06T12:31:57.600 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-06T12:31:57.601 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:31:57.601 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-06T12:31:57.601 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-06T12:31:57.601 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-06T12:31:57.601 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T12:31:57.608 INFO:tasks.ceph.mds.a:Stopped 2024-04-06T12:31:57.608 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-06T12:31:57.609 INFO:tasks.ceph.mds.c:Stopped 2024-04-06T12:31:57.609 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-06T12:31:57.611 INFO:tasks.ceph.mds.b:Stopped 2024-04-06T12:31:57.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-06T12:31:57.612 INFO:tasks.ceph.mds.d:Stopped 2024-04-06T12:31:57.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-06T12:31:57.651 INFO:tasks.ceph.mds.a:Started 2024-04-06T12:31:57.652 DEBUG:teuthology.parallel:result is None 2024-04-06T12:31:57.652 INFO:tasks.ceph.mds.c:Started 2024-04-06T12:31:57.652 DEBUG:teuthology.parallel:result is None 2024-04-06T12:31:57.654 INFO:tasks.ceph.mds.b:Started 2024-04-06T12:31:57.655 DEBUG:teuthology.parallel:result is None 2024-04-06T12:31:57.655 INFO:tasks.ceph.mds.d:Started 2024-04-06T12:31:57.655 DEBUG:teuthology.parallel:result is None 2024-04-06T12:31:57.656 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-06T12:31:57.732 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T12:31:57.732 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2024-04-06T12:31:57.743 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T12:31:57.744 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2024-04-06T12:31:57.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.789+0000 7f87a6760700 1 Processor -- start 2024-04-06T12:31:57.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.789+0000 7f87a6760700 1 -- start start 2024-04-06T12:31:57.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.790+0000 7f87a6760700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f87a015b130 0x7f87a00b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:57.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.790+0000 7f87a6760700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87a015b510 0x7f87a00b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:57.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.790+0000 7f87a6760700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87a00b1fb0 con 0x7f87a0071de0 2024-04-06T12:31:57.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.790+0000 7f87a6760700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87a00b2200 con 0x7f87a015b130 2024-04-06T12:31:57.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.790+0000 7f87a6760700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a00b24e0 con 0x7f87a015b510 2024-04-06T12:31:57.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.790+0000 7f879f7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87a015b510 0x7f87a00b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:57.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.790+0000 7f879ffff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f87a015b130 0x7f87a00b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:57.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.790+0000 7f87a4cfd700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f87a0071de0 0x7f87a0072250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:44804/0 (socket says 172.21.15.189:44804) 2024-04-06T12:31:57.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.790+0000 7f87a4cfd700 1 -- 172.21.15.189:0/2836870393 learned_addr learned my addr 172.21.15.189:0/2836870393 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:57.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.791+0000 7f879effd700 1 -- 172.21.15.189:0/2836870393 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591967342 0 0) 0x7f87a00b1fb0 con 0x7f87a0071de0 2024-04-06T12:31:57.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.791+0000 7f879effd700 1 -- 172.21.15.189:0/2836870393 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87880035b0 con 0x7f87a0071de0 2024-04-06T12:31:57.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.791+0000 7f879ffff700 1 -- 172.21.15.189:0/2836870393 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87a015b510 msgr2=0x7f87a00b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:57.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.791+0000 7f879ffff700 1 --2- 172.21.15.189:0/2836870393 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87a015b510 0x7f87a00b39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:57.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.791+0000 7f879ffff700 1 -- 172.21.15.189:0/2836870393 >> v1:172.21.15.192:6789/0 conn(0x7f87a0071de0 legacy=0x7f87a0072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:57.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.791+0000 7f879ffff700 1 -- 172.21.15.189:0/2836870393 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a00b27c0 con 0x7f87a015b130 2024-04-06T12:31:57.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.791+0000 7f879f7fe700 1 --2- 172.21.15.189:0/2836870393 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87a015b510 0x7f87a00b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:57.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.791+0000 7f879ffff700 1 --2- 172.21.15.189:0/2836870393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a015b130 0x7f87a00b3470 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f8790009840 tx=0x7f8790009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebec15fc018212fa server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:57.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.791+0000 7f879effd700 1 -- 172.21.15.189:0/2836870393 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8790011830 con 0x7f87a015b130 2024-04-06T12:31:57.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.791+0000 7f879effd700 1 -- 172.21.15.189:0/2836870393 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8790011990 con 0x7f87a015b130 2024-04-06T12:31:57.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.791+0000 7f879effd700 1 -- 172.21.15.189:0/2836870393 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8790011d20 con 0x7f87a015b130 2024-04-06T12:31:57.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.792+0000 7f87a6760700 1 -- 172.21.15.189:0/2836870393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a015b130 msgr2=0x7f87a00b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:57.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.792+0000 7f87a6760700 1 --2- 172.21.15.189:0/2836870393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a015b130 0x7f87a00b3470 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f8790009840 tx=0x7f8790009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:57.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.792+0000 7f87a6760700 1 -- 172.21.15.189:0/2836870393 shutdown_connections 2024-04-06T12:31:57.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.792+0000 7f87a6760700 1 --2- 172.21.15.189:0/2836870393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a015b130 0x7f87a00b3470 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:57.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.792+0000 7f87a6760700 1 --2- 172.21.15.189:0/2836870393 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87a015b510 0x7f87a00b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:57.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.792+0000 7f87a6760700 1 -- 172.21.15.189:0/2836870393 >> 172.21.15.189:0/2836870393 conn(0x7f87a006a730 msgr2=0x7f87a006ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:57.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.792+0000 7f87a6760700 1 -- 172.21.15.189:0/2836870393 shutdown_connections 2024-04-06T12:31:57.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.792+0000 7f87a6760700 1 -- 172.21.15.189:0/2836870393 wait complete. 2024-04-06T12:31:57.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.793+0000 7f87a6760700 1 Processor -- start 2024-04-06T12:31:57.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.793+0000 7f87a6760700 1 -- start start 2024-04-06T12:31:57.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.794+0000 7f87a6760700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a0071de0 0x7f87a00ad220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:57.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.794+0000 7f87a6760700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87a015b130 0x7f87a00aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:57.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.794+0000 7f87a6760700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87a015b510 0x7f87a00b0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:57.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.794+0000 7f87a6760700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a00ae2b0 con 0x7f87a015b510 2024-04-06T12:31:57.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.794+0000 7f87a6760700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a00aca40 con 0x7f87a0071de0 2024-04-06T12:31:57.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.794+0000 7f87a6760700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a0157a30 con 0x7f87a015b130 2024-04-06T12:31:57.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.794+0000 7f87a4cfd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87a015b510 0x7f87a00b0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:57.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.794+0000 7f87a4cfd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87a015b510 0x7f87a00b0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47446/0 (socket says 172.21.15.189:47446) 2024-04-06T12:31:57.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.794+0000 7f87a4cfd700 1 -- 172.21.15.189:0/2135104920 learned_addr learned my addr 172.21.15.189:0/2135104920 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:57.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.794+0000 7f879f7fe700 1 --2- 172.21.15.189:0/2135104920 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87a015b130 0x7f87a00aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:57.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.794+0000 7f879ffff700 1 --2- 172.21.15.189:0/2135104920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a0071de0 0x7f87a00ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.794+0000 7f87a4cfd700 1 -- 172.21.15.189:0/2135104920 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87a015b130 msgr2=0x7f87a00aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.794+0000 7f87a4cfd700 1 --2- 172.21.15.189:0/2135104920 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87a015b130 0x7f87a00aefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.794+0000 7f87a4cfd700 1 -- 172.21.15.189:0/2135104920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a0071de0 msgr2=0x7f87a00ad220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.794+0000 7f87a4cfd700 1 --2- 172.21.15.189:0/2135104920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a0071de0 0x7f87a00ad220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.794+0000 7f87a4cfd700 1 -- 172.21.15.189:0/2135104920 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87940094a0 con 0x7f87a015b510 2024-04-06T12:31:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.794+0000 7f879ffff700 1 --2- 172.21.15.189:0/2135104920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a0071de0 0x7f87a00ad220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.794+0000 7f87a4cfd700 1 --2- 172.21.15.189:0/2135104920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87a015b510 0x7f87a00b0c20 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f879800cb40 tx=0x7f879800ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.794+0000 7f879cff9700 1 -- 172.21.15.189:0/2135104920 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879800a9e0 con 0x7f87a015b510 2024-04-06T12:31:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.795+0000 7f879cff9700 1 -- 172.21.15.189:0/2135104920 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879800ab40 con 0x7f87a015b510 2024-04-06T12:31:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.795+0000 7f87a6760700 1 -- 172.21.15.189:0/2135104920 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87900094a0 con 0x7f87a015b510 2024-04-06T12:31:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.795+0000 7f879cff9700 1 -- 172.21.15.189:0/2135104920 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8798012400 con 0x7f87a015b510 2024-04-06T12:31:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.795+0000 7f87a6760700 1 -- 172.21.15.189:0/2135104920 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a0157ff0 con 0x7f87a015b510 2024-04-06T12:31:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.795+0000 7f87a6760700 1 -- 172.21.15.189:0/2135104920 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a0073270 con 0x7f87a015b510 2024-04-06T12:31:57.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.796+0000 7f879cff9700 1 -- 172.21.15.189:0/2135104920 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f879801b3b0 con 0x7f87a015b510 2024-04-06T12:31:57.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.796+0000 7f879cff9700 1 --2- 172.21.15.189:0/2135104920 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f878c06adb0 0x7f878c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:57.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.796+0000 7f879cff9700 1 -- 172.21.15.189:0/2135104920 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f87980890b0 con 0x7f87a015b510 2024-04-06T12:31:57.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.796+0000 7f879ffff700 1 --2- 172.21.15.189:0/2135104920 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f878c06adb0 0x7f878c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:57.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.797+0000 7f879ffff700 1 --2- 172.21.15.189:0/2135104920 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f878c06adb0 0x7f878c06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8790000c00 tx=0x7f8790008030 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:57.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:57.801+0000 7f879cff9700 1 -- 172.21.15.189:0/2135104920 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8798054160 con 0x7f87a015b510 2024-04-06T12:31:58.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:58.008+0000 7f87a6760700 1 -- 172.21.15.189:0/2135104920 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f87a013c8a0 con 0x7f87a015b510 2024-04-06T12:31:58.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:58.992+0000 7f879cff9700 1 -- 172.21.15.189:0/2135104920 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v154) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8798057780 con 0x7f87a015b510 2024-04-06T12:31:58.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:58.994+0000 7f87a6760700 1 -- 172.21.15.189:0/2135104920 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f878c06adb0 msgr2=0x7f878c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:58.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:58.994+0000 7f87a6760700 1 --2- 172.21.15.189:0/2135104920 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f878c06adb0 0x7f878c06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8790000c00 tx=0x7f8790008030 comp rx=0 tx=0).stop 2024-04-06T12:31:58.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:58.994+0000 7f87a6760700 1 -- 172.21.15.189:0/2135104920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87a015b510 msgr2=0x7f87a00b0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:58.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:58.994+0000 7f87a6760700 1 --2- 172.21.15.189:0/2135104920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87a015b510 0x7f87a00b0c20 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f879800cb40 tx=0x7f879800ce50 comp rx=0 tx=0).stop 2024-04-06T12:31:58.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:58.995+0000 7f87a6760700 1 -- 172.21.15.189:0/2135104920 shutdown_connections 2024-04-06T12:31:58.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:58.995+0000 7f87a6760700 1 --2- 172.21.15.189:0/2135104920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87a015b510 0x7f87a00b0c20 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:58.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:58.995+0000 7f87a6760700 1 --2- 172.21.15.189:0/2135104920 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f878c06adb0 0x7f878c06d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:58.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:58.995+0000 7f87a6760700 1 --2- 172.21.15.189:0/2135104920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a0071de0 0x7f87a00ad220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:58.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:58.995+0000 7f87a6760700 1 --2- 172.21.15.189:0/2135104920 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87a015b130 0x7f87a00aefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:58.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:58.995+0000 7f87a6760700 1 -- 172.21.15.189:0/2135104920 >> 172.21.15.189:0/2135104920 conn(0x7f87a006a730 msgr2=0x7f87a0158410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:58.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:58.995+0000 7f87a6760700 1 -- 172.21.15.189:0/2135104920 shutdown_connections 2024-04-06T12:31:58.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:58.995+0000 7f87a6760700 1 -- 172.21.15.189:0/2135104920 wait complete. 2024-04-06T12:31:58.993 INFO:teuthology.orchestra.run.smithi189.stderr: removed all blocklist entries 2024-04-06T12:31:59.007 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-06T12:31:59.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.137+0000 7f39d20dc700 1 Processor -- start 2024-04-06T12:31:59.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.137+0000 7f39d20dc700 1 -- start start 2024-04-06T12:31:59.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.138+0000 7f39d20dc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f39cc0b4660 0x7f39cc0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.138+0000 7f39d20dc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f39cc0b2fc0 0x7f39cc0b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.138+0000 7f39d20dc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39cc0b5030 con 0x7f39cc15a550 2024-04-06T12:31:59.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.138+0000 7f39d20dc700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39cc0cdbe0 con 0x7f39cc0b2fc0 2024-04-06T12:31:59.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.138+0000 7f39d20dc700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39cc15a940 con 0x7f39cc0b4660 2024-04-06T12:31:59.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.138+0000 7f39cbfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f39cc0b2fc0 0x7f39cc0b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.138+0000 7f39cb7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f39cc15a550 0x7f39cc0cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40864/0 (socket says 172.21.15.189:40864) 2024-04-06T12:31:59.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.138+0000 7f39cb7fe700 1 -- 172.21.15.189:0/3642243926 learned_addr learned my addr 172.21.15.189:0/3642243926 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:59.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.139+0000 7f39caffd700 1 --2- 172.21.15.189:0/3642243926 >> v2:172.21.15.192:3300/0 conn(0x7f39cc0b4660 0x7f39cc0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.139+0000 7f39cbfff700 1 -- 172.21.15.189:0/3642243926 >> v2:172.21.15.192:3300/0 conn(0x7f39cc0b4660 msgr2=0x7f39cc0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:59.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.139+0000 7f39cbfff700 1 --2- 172.21.15.189:0/3642243926 >> v2:172.21.15.192:3300/0 conn(0x7f39cc0b4660 0x7f39cc0b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.139+0000 7f39cbfff700 1 -- 172.21.15.189:0/3642243926 >> v1:172.21.15.189:6789/0 conn(0x7f39cc15a550 legacy=0x7f39cc0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:59.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.139+0000 7f39cbfff700 1 -- 172.21.15.189:0/3642243926 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39cc15ac00 con 0x7f39cc0b2fc0 2024-04-06T12:31:59.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.139+0000 7f39ca7fc700 1 -- 172.21.15.189:0/3642243926 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191120908 0 0) 0x7f39cc0b5030 con 0x7f39cc15a550 2024-04-06T12:31:59.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.139+0000 7f39caffd700 1 --2- 172.21.15.189:0/3642243926 >> v2:172.21.15.192:3300/0 conn(0x7f39cc0b4660 0x7f39cc0b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:31:59.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.139+0000 7f39cbfff700 1 --2- 172.21.15.189:0/3642243926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39cc0b2fc0 0x7f39cc0b33d0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0b2d40 tx=0x7f39c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22ff39d1b85d9241 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:59.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.139+0000 7f39ca7fc700 1 -- 172.21.15.189:0/3642243926 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c400e070 con 0x7f39cc0b2fc0 2024-04-06T12:31:59.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.139+0000 7f39ca7fc700 1 -- 172.21.15.189:0/3642243926 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c400bbd0 con 0x7f39cc0b2fc0 2024-04-06T12:31:59.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.139+0000 7f39ca7fc700 1 -- 172.21.15.189:0/3642243926 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c4012430 con 0x7f39cc0b2fc0 2024-04-06T12:31:59.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.139+0000 7f39d20dc700 1 -- 172.21.15.189:0/3642243926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39cc0b2fc0 msgr2=0x7f39cc0b33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:59.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.139+0000 7f39d20dc700 1 --2- 172.21.15.189:0/3642243926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39cc0b2fc0 0x7f39cc0b33d0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0b2d40 tx=0x7f39c400b330 comp rx=0 tx=0).stop 2024-04-06T12:31:59.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.139+0000 7f39d20dc700 1 -- 172.21.15.189:0/3642243926 shutdown_connections 2024-04-06T12:31:59.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.139+0000 7f39d20dc700 1 --2- 172.21.15.189:0/3642243926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39cc0b2fc0 0x7f39cc0b33d0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.139+0000 7f39d20dc700 1 --2- 172.21.15.189:0/3642243926 >> v2:172.21.15.192:3300/0 conn(0x7f39cc0b4660 0x7f39cc0b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.140+0000 7f39d20dc700 1 -- 172.21.15.189:0/3642243926 >> 172.21.15.189:0/3642243926 conn(0x7f39cc0d00b0 msgr2=0x7f39cc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:59.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.140+0000 7f39d20dc700 1 -- 172.21.15.189:0/3642243926 shutdown_connections 2024-04-06T12:31:59.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.140+0000 7f39d20dc700 1 -- 172.21.15.189:0/3642243926 wait complete. 2024-04-06T12:31:59.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.141+0000 7f39d20dc700 1 Processor -- start 2024-04-06T12:31:59.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.141+0000 7f39d20dc700 1 -- start start 2024-04-06T12:31:59.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39d20dc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39cc0b2fc0 0x7f39cc0ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39d20dc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39cc0b4660 0x7f39cc1f3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39d20dc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39cc14ca70 0x7f39cc154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39d20dc700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc1f38a0 con 0x7f39cc0b4660 2024-04-06T12:31:59.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39d20dc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc0ad2f0 con 0x7f39cc0b2fc0 2024-04-06T12:31:59.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39d20dc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc0ad590 con 0x7f39cc14ca70 2024-04-06T12:31:59.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39caffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39cc0b4660 0x7f39cc1f3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39caffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39cc0b4660 0x7f39cc1f3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47462/0 (socket says 172.21.15.189:47462) 2024-04-06T12:31:59.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39caffd700 1 -- 172.21.15.189:0/2519621489 learned_addr learned my addr 172.21.15.189:0/2519621489 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:59.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39cb7fe700 1 --2- 172.21.15.189:0/2519621489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39cc0b2fc0 0x7f39cc0ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39cbfff700 1 --2- 172.21.15.189:0/2519621489 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39cc14ca70 0x7f39cc154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39caffd700 1 -- 172.21.15.189:0/2519621489 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39cc14ca70 msgr2=0x7f39cc154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:59.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39caffd700 1 --2- 172.21.15.189:0/2519621489 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39cc14ca70 0x7f39cc154650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39caffd700 1 -- 172.21.15.189:0/2519621489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39cc0b2fc0 msgr2=0x7f39cc0ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:59.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39caffd700 1 --2- 172.21.15.189:0/2519621489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39cc0b2fc0 0x7f39cc0ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39caffd700 1 -- 172.21.15.189:0/2519621489 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c4009a70 con 0x7f39cc0b4660 2024-04-06T12:31:59.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39caffd700 1 --2- 172.21.15.189:0/2519621489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39cc0b4660 0x7f39cc1f3360 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f39b400c750 tx=0x7f39b400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:59.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39c3fff700 1 -- 172.21.15.189:0/2519621489 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b4011800 con 0x7f39cc0b4660 2024-04-06T12:31:59.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39c3fff700 1 -- 172.21.15.189:0/2519621489 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b4011960 con 0x7f39cc0b4660 2024-04-06T12:31:59.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39d20dc700 1 -- 172.21.15.189:0/2519621489 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39cc0ad870 con 0x7f39cc0b4660 2024-04-06T12:31:59.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39c3fff700 1 -- 172.21.15.189:0/2519621489 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b4011cd0 con 0x7f39cc0b4660 2024-04-06T12:31:59.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.142+0000 7f39d20dc700 1 -- 172.21.15.189:0/2519621489 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39cc0acf80 con 0x7f39cc0b4660 2024-04-06T12:31:59.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.143+0000 7f39d20dc700 1 -- 172.21.15.189:0/2519621489 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39cc0adab0 con 0x7f39cc0b4660 2024-04-06T12:31:59.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.144+0000 7f39c3fff700 1 -- 172.21.15.189:0/2519621489 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f39b4021350 con 0x7f39cc0b4660 2024-04-06T12:31:59.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.144+0000 7f39c3fff700 1 --2- 172.21.15.189:0/2519621489 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f39ac06adb0 0x7f39ac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.144+0000 7f39c3fff700 1 -- 172.21.15.189:0/2519621489 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f39b4089200 con 0x7f39cc0b4660 2024-04-06T12:31:59.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.144+0000 7f39cb7fe700 1 --2- 172.21.15.189:0/2519621489 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f39ac06adb0 0x7f39ac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.145+0000 7f39cb7fe700 1 --2- 172.21.15.189:0/2519621489 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f39ac06adb0 0x7f39ac06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f39bc007fd0 tx=0x7f39bc007de0 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:59.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.149+0000 7f39c3fff700 1 -- 172.21.15.189:0/2519621489 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f39cc0adab0 con 0x7f39cc0b4660 2024-04-06T12:31:59.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.417+0000 7f39d20dc700 1 -- 172.21.15.189:0/2519621489 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f39cc1f60f0 con 0x7f39cc0b4660 2024-04-06T12:31:59.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.418+0000 7f39c3fff700 1 -- 172.21.15.189:0/2519621489 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f39cc1f60f0 con 0x7f39cc0b4660 2024-04-06T12:31:59.415 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:59.415 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T12:31:59.415 INFO:teuthology.orchestra.run.smithi189.stdout: "auth_dump": [ 2024-04-06T12:31:59.415 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:59.416 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.a", 2024-04-06T12:31:59.416 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmTjYMNBAA5voWGV4Pa7pE27VogrdipQ==", 2024-04-06T12:31:59.416 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:59.416 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:31:59.416 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:31:59.416 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:31:59.416 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:59.416 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.416 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:59.416 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:59.416 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.b", 2024-04-06T12:31:59.416 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm9A1OBRAAnHzubCvRUX8UdW8hmDzKbQ==", 2024-04-06T12:31:59.416 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:59.416 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:31:59.417 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:31:59.417 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:31:59.417 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:59.417 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.417 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:59.417 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:59.417 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.c", 2024-04-06T12:31:59.417 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmz0hhOxAAD3/GINNmGoFk+oPyiHHJXg==", 2024-04-06T12:31:59.417 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:59.417 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:31:59.417 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:31:59.417 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:31:59.417 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:59.417 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.418 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:59.418 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:59.418 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.d", 2024-04-06T12:31:59.418 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmhoAeDRAAoQC0vicWumBRTwpi3PtYaw==", 2024-04-06T12:31:59.418 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:59.418 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:31:59.418 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:31:59.418 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:31:59.418 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:59.418 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.418 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:59.418 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:59.418 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.0", 2024-04-06T12:31:59.418 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDkOxFmkxLWJhAAFXewCNOFX7dhAZVgUmewpQ==", 2024-04-06T12:31:59.419 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:59.419 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:31:59.419 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:31:59.419 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:59.419 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.419 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:59.419 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:59.419 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.1", 2024-04-06T12:31:59.419 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDnOxFmHYqWJRAABlsVCiv3e7YJMbfodJTWag==", 2024-04-06T12:31:59.419 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:59.419 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:31:59.419 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:31:59.419 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:59.419 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.420 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:59.420 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:59.420 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.2", 2024-04-06T12:31:59.420 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDqOxFmkhcvJBAA8NMEpHVGbGALjV9RISKP8g==", 2024-04-06T12:31:59.420 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:59.420 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:31:59.420 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:31:59.420 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:59.420 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.420 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:59.420 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:59.420 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.3", 2024-04-06T12:31:59.420 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDtOxFmDGzhIxAAamBoAbE8+lAel4os67meyQ==", 2024-04-06T12:31:59.420 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:59.420 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:31:59.421 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:31:59.421 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:59.421 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.421 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:59.421 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:59.421 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.4", 2024-04-06T12:31:59.421 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDzOxFmobZDJxAA9l8wdS81Eyw6+17I5ac6bw==", 2024-04-06T12:31:59.421 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:59.421 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:31:59.421 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:31:59.421 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:59.421 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.421 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:59.421 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:59.421 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.5", 2024-04-06T12:31:59.422 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD2OxFmi+ruJRAASlQ0o9jX/wA7U5AosnL+Wg==", 2024-04-06T12:31:59.422 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:59.422 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:31:59.422 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:31:59.422 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:59.422 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.422 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:59.422 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:59.422 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.6", 2024-04-06T12:31:59.422 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD5OxFmbwuuJBAA0wqJRmnJyskQPZJ7/LnpvA==", 2024-04-06T12:31:59.422 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:59.422 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:31:59.422 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:31:59.422 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:59.422 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.423 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:59.423 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:59.423 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.7", 2024-04-06T12:31:59.423 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD8OxFmDmRgIxAAzyW9AGAbmb67ZvB5qunwLQ==", 2024-04-06T12:31:59.423 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:59.423 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T12:31:59.423 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T12:31:59.423 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:59.423 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.423 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:59.423 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:59.423 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.0", 2024-04-06T12:31:59.423 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm0ByJERAAoEV8Yfc09+Go6u95XafElQ==", 2024-04-06T12:31:59.423 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:59.424 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:31:59.424 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow r", 2024-04-06T12:31:59.424 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-06T12:31:59.424 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.424 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:59.424 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:59.424 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.1", 2024-04-06T12:31:59.424 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmSV5lGBAAavDEo2LGy0IHpWSAuitHXg==", 2024-04-06T12:31:59.424 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:59.424 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:31:59.424 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:31:59.424 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:31:59.424 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:31:59.424 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.425 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:59.425 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:59.425 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.2", 2024-04-06T12:31:59.425 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm/Gz5FBAA3xX/rsAyWZ1JoUVlQlBIvQ==", 2024-04-06T12:31:59.425 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:59.425 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:31:59.425 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:31:59.425 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:31:59.425 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:31:59.425 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.425 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:59.425 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:59.425 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.3", 2024-04-06T12:31:59.425 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmzra7GxAADbnzIvWnRTo65LvPwtxWrg==", 2024-04-06T12:31:59.426 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:59.426 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T12:31:59.426 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T12:31:59.426 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T12:31:59.426 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T12:31:59.426 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.426 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:59.426 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:59.426 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.admin", 2024-04-06T12:31:59.426 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmxG+wCBAANg5kt6TOUQdj3/X0Mqgx1g==", 2024-04-06T12:31:59.426 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:59.426 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:31:59.426 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T12:31:59.426 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T12:31:59.426 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:59.427 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.427 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:59.427 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:59.427 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.x", 2024-04-06T12:31:59.427 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmPLyBMBAAYfhkI9Z1CPlf1Tijm+L7xA==", 2024-04-06T12:31:59.427 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:59.427 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:31:59.427 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T12:31:59.427 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:59.427 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.427 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T12:31:59.427 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:31:59.427 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.y", 2024-04-06T12:31:59.427 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFm1nHJHhAAQWRe8cU8duWuUJiYaDeTaQ==", 2024-04-06T12:31:59.428 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T12:31:59.428 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T12:31:59.428 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T12:31:59.428 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T12:31:59.428 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.428 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:31:59.428 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-04-06T12:31:59.428 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T12:31:59.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.420+0000 7f39d20dc700 1 -- 172.21.15.189:0/2519621489 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f39ac06adb0 msgr2=0x7f39ac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:59.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.420+0000 7f39d20dc700 1 --2- 172.21.15.189:0/2519621489 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f39ac06adb0 0x7f39ac06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f39bc007fd0 tx=0x7f39bc007de0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.420+0000 7f39d20dc700 1 -- 172.21.15.189:0/2519621489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39cc0b4660 msgr2=0x7f39cc1f3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:59.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.420+0000 7f39d20dc700 1 --2- 172.21.15.189:0/2519621489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39cc0b4660 0x7f39cc1f3360 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f39b400c750 tx=0x7f39b400ca60 comp rx=0 tx=0).stop 2024-04-06T12:31:59.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.420+0000 7f39d20dc700 1 -- 172.21.15.189:0/2519621489 shutdown_connections 2024-04-06T12:31:59.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.420+0000 7f39d20dc700 1 --2- 172.21.15.189:0/2519621489 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39cc14ca70 0x7f39cc154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.420+0000 7f39d20dc700 1 --2- 172.21.15.189:0/2519621489 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f39ac06adb0 0x7f39ac06d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.420+0000 7f39d20dc700 1 --2- 172.21.15.189:0/2519621489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39cc0b2fc0 0x7f39cc0ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.420+0000 7f39d20dc700 1 --2- 172.21.15.189:0/2519621489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39cc0b4660 0x7f39cc1f3360 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.420+0000 7f39d20dc700 1 -- 172.21.15.189:0/2519621489 >> 172.21.15.189:0/2519621489 conn(0x7f39cc0d00b0 msgr2=0x7f39cc06f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:59.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.420+0000 7f39d20dc700 1 -- 172.21.15.189:0/2519621489 shutdown_connections 2024-04-06T12:31:59.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.420+0000 7f39d20dc700 1 -- 172.21.15.189:0/2519621489 wait complete. 2024-04-06T12:31:59.431 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:31:59.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.556+0000 7f83ca4f2700 1 Processor -- start 2024-04-06T12:31:59.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.556+0000 7f83ca4f2700 1 -- start start 2024-04-06T12:31:59.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.557+0000 7f83ca4f2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83c40d17a0 0x7f83c406b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.557+0000 7f83ca4f2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f83c4146dd0 0x7f83c406d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.557+0000 7f83ca4f2700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c406c0a0 con 0x7f83c40cedf0 2024-04-06T12:31:59.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.557+0000 7f83ca4f2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83c406c380 con 0x7f83c4146dd0 2024-04-06T12:31:59.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.557+0000 7f83ca4f2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c406c660 con 0x7f83c40d17a0 2024-04-06T12:31:59.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.557+0000 7f83c3fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83c40d17a0 0x7f83c406b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.557+0000 7f83c37fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f83c4146dd0 0x7f83c406d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.557+0000 7f83c3fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83c40d17a0 0x7f83c406b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:47180/0 (socket says 172.21.15.189:47180) 2024-04-06T12:31:59.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.557+0000 7f83c3fff700 1 -- 172.21.15.189:0/3538210362 learned_addr learned my addr 172.21.15.189:0/3538210362 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:59.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.557+0000 7f83c8a8f700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f83c40cedf0 0x7f83c41441f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:37116/0 (socket says 172.21.15.189:37116) 2024-04-06T12:31:59.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.557+0000 7f83c37fe700 1 -- 172.21.15.189:0/3538210362 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83c40d17a0 msgr2=0x7f83c406b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:59.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.557+0000 7f83c37fe700 1 --2- 172.21.15.189:0/3538210362 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83c40d17a0 0x7f83c406b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.557+0000 7f83c37fe700 1 -- 172.21.15.189:0/3538210362 >> v1:172.21.15.192:6789/0 conn(0x7f83c40cedf0 legacy=0x7f83c41441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:59.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.557+0000 7f83c37fe700 1 -- 172.21.15.189:0/3538210362 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c406c940 con 0x7f83c4146dd0 2024-04-06T12:31:59.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.558+0000 7f83c37fe700 1 --2- 172.21.15.189:0/3538210362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83c4146dd0 0x7f83c406d140 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f83b8009840 tx=0x7f83b8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4efeb2202ac734e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:59.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.558+0000 7f83c2ffd700 1 -- 172.21.15.189:0/3538210362 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b8011830 con 0x7f83c4146dd0 2024-04-06T12:31:59.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.558+0000 7f83c2ffd700 1 -- 172.21.15.189:0/3538210362 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b8011990 con 0x7f83c4146dd0 2024-04-06T12:31:59.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.558+0000 7f83c2ffd700 1 -- 172.21.15.189:0/3538210362 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b8011d20 con 0x7f83c4146dd0 2024-04-06T12:31:59.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.558+0000 7f83ca4f2700 1 -- 172.21.15.189:0/3538210362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83c4146dd0 msgr2=0x7f83c406d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:59.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.558+0000 7f83ca4f2700 1 --2- 172.21.15.189:0/3538210362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83c4146dd0 0x7f83c406d140 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f83b8009840 tx=0x7f83b8009b50 comp rx=0 tx=0).stop 2024-04-06T12:31:59.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.558+0000 7f83ca4f2700 1 -- 172.21.15.189:0/3538210362 shutdown_connections 2024-04-06T12:31:59.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.558+0000 7f83ca4f2700 1 --2- 172.21.15.189:0/3538210362 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83c40d17a0 0x7f83c406b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.558+0000 7f83ca4f2700 1 --2- 172.21.15.189:0/3538210362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83c4146dd0 0x7f83c406d140 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.558+0000 7f83ca4f2700 1 -- 172.21.15.189:0/3538210362 >> 172.21.15.189:0/3538210362 conn(0x7f83c40b2db0 msgr2=0x7f83c40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:59.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.558+0000 7f83ca4f2700 1 -- 172.21.15.189:0/3538210362 shutdown_connections 2024-04-06T12:31:59.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.558+0000 7f83ca4f2700 1 -- 172.21.15.189:0/3538210362 wait complete. 2024-04-06T12:31:59.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.559+0000 7f83ca4f2700 1 Processor -- start 2024-04-06T12:31:59.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.560+0000 7f83ca4f2700 1 -- start start 2024-04-06T12:31:59.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.560+0000 7f83ca4f2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83c40cedf0 0x7f83c40c7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.560+0000 7f83ca4f2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c40d17a0 0x7f83c415a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.560+0000 7f83ca4f2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83c4146dd0 0x7f83c415ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.560+0000 7f83ca4f2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c41f4290 con 0x7f83c40d17a0 2024-04-06T12:31:59.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.560+0000 7f83ca4f2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c40b5730 con 0x7f83c4146dd0 2024-04-06T12:31:59.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.560+0000 7f83ca4f2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c40c7ba0 con 0x7f83c40cedf0 2024-04-06T12:31:59.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.560+0000 7f83c37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c40d17a0 0x7f83c415a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.560+0000 7f83c37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c40d17a0 0x7f83c415a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60548/0 (socket says 172.21.15.189:60548) 2024-04-06T12:31:59.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.560+0000 7f83c37fe700 1 -- 172.21.15.189:0/2804611891 learned_addr learned my addr 172.21.15.189:0/2804611891 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:59.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.561+0000 7f83c8a8f700 1 --2- 172.21.15.189:0/2804611891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83c4146dd0 0x7f83c415ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.561+0000 7f83c3fff700 1 --2- 172.21.15.189:0/2804611891 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83c40cedf0 0x7f83c40c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.561+0000 7f83c37fe700 1 -- 172.21.15.189:0/2804611891 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83c40cedf0 msgr2=0x7f83c40c7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:59.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.561+0000 7f83c37fe700 1 --2- 172.21.15.189:0/2804611891 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83c40cedf0 0x7f83c40c7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.561+0000 7f83c37fe700 1 -- 172.21.15.189:0/2804611891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83c4146dd0 msgr2=0x7f83c415ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:59.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.561+0000 7f83c37fe700 1 --2- 172.21.15.189:0/2804611891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83c4146dd0 0x7f83c415ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.561+0000 7f83c37fe700 1 -- 172.21.15.189:0/2804611891 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b80094a0 con 0x7f83c40d17a0 2024-04-06T12:31:59.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.561+0000 7f83c37fe700 1 --2- 172.21.15.189:0/2804611891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c40d17a0 0x7f83c415a5e0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f83b8009fd0 tx=0x7f83b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:59.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.561+0000 7f83c0ff9700 1 -- 172.21.15.189:0/2804611891 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b80036a0 con 0x7f83c40d17a0 2024-04-06T12:31:59.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.561+0000 7f83c0ff9700 1 -- 172.21.15.189:0/2804611891 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b801f070 con 0x7f83c40d17a0 2024-04-06T12:31:59.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.561+0000 7f83ca4f2700 1 -- 172.21.15.189:0/2804611891 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c40b5120 con 0x7f83c40d17a0 2024-04-06T12:31:59.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.561+0000 7f83c0ff9700 1 -- 172.21.15.189:0/2804611891 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b8003a60 con 0x7f83c40d17a0 2024-04-06T12:31:59.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.561+0000 7f83ca4f2700 1 -- 172.21.15.189:0/2804611891 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c40b53c0 con 0x7f83c40d17a0 2024-04-06T12:31:59.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.562+0000 7f83ca4f2700 1 -- 172.21.15.189:0/2804611891 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c41f3d00 con 0x7f83c40d17a0 2024-04-06T12:31:59.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.563+0000 7f83c0ff9700 1 -- 172.21.15.189:0/2804611891 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f83b8003bc0 con 0x7f83c40d17a0 2024-04-06T12:31:59.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.563+0000 7f83c0ff9700 1 --2- 172.21.15.189:0/2804611891 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f83b006adb0 0x7f83b006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.563+0000 7f83c3fff700 1 --2- 172.21.15.189:0/2804611891 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f83b006adb0 0x7f83b006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.563+0000 7f83c0ff9700 1 -- 172.21.15.189:0/2804611891 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f83b8021070 con 0x7f83c40d17a0 2024-04-06T12:31:59.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.563+0000 7f83c3fff700 1 --2- 172.21.15.189:0/2804611891 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f83b006adb0 0x7f83b006d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f83c4160ed0 tx=0x7f83b40092c0 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:59.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.568+0000 7f83c0ff9700 1 -- 172.21.15.189:0/2804611891 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f83b8055db0 con 0x7f83c40d17a0 2024-04-06T12:31:59.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.764+0000 7f83ca4f2700 1 -- 172.21.15.189:0/2804611891 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83c4074990 con 0x7f83c40d17a0 2024-04-06T12:31:59.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.765+0000 7f83c0ff9700 1 -- 172.21.15.189:0/2804611891 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) v1 ==== 74+0+12652 (secure 0 0 0) 0x7f83b802b090 con 0x7f83c40d17a0 2024-04-06T12:31:59.763 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:31:59.763 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":154,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:31:58.988368+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:31:59.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.768+0000 7f83ca4f2700 1 -- 172.21.15.189:0/2804611891 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f83b006adb0 msgr2=0x7f83b006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:59.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.768+0000 7f83ca4f2700 1 --2- 172.21.15.189:0/2804611891 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f83b006adb0 0x7f83b006d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f83c4160ed0 tx=0x7f83b40092c0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.768+0000 7f83ca4f2700 1 -- 172.21.15.189:0/2804611891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c40d17a0 msgr2=0x7f83c415a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:59.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.768+0000 7f83ca4f2700 1 --2- 172.21.15.189:0/2804611891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c40d17a0 0x7f83c415a5e0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f83b8009fd0 tx=0x7f83b8004750 comp rx=0 tx=0).stop 2024-04-06T12:31:59.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.768+0000 7f83ca4f2700 1 -- 172.21.15.189:0/2804611891 shutdown_connections 2024-04-06T12:31:59.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.768+0000 7f83ca4f2700 1 --2- 172.21.15.189:0/2804611891 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f83b006adb0 0x7f83b006d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.768+0000 7f83ca4f2700 1 --2- 172.21.15.189:0/2804611891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83c4146dd0 0x7f83c415ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.768+0000 7f83ca4f2700 1 --2- 172.21.15.189:0/2804611891 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83c40cedf0 0x7f83c40c7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.768+0000 7f83ca4f2700 1 --2- 172.21.15.189:0/2804611891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c40d17a0 0x7f83c415a5e0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.768+0000 7f83ca4f2700 1 -- 172.21.15.189:0/2804611891 >> 172.21.15.189:0/2804611891 conn(0x7f83c40b2db0 msgr2=0x7f83c415e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:59.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.768+0000 7f83ca4f2700 1 -- 172.21.15.189:0/2804611891 shutdown_connections 2024-04-06T12:31:59.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.768+0000 7f83ca4f2700 1 -- 172.21.15.189:0/2804611891 wait complete. 2024-04-06T12:31:59.777 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T12:31:59.778 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T12:31:59.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.905+0000 7f4c5cc8a700 1 Processor -- start 2024-04-06T12:31:59.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.905+0000 7f4c5cc8a700 1 -- start start 2024-04-06T12:31:59.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c5cc8a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4c5815d7c0 0x7f4c5806b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c5cc8a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c5806b6f0 0x7f4c5806dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c5cc8a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c5806ceb0 con 0x7f4c580cd3e0 2024-04-06T12:31:59.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c5cc8a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c580cdfa0 con 0x7f4c5815d7c0 2024-04-06T12:31:59.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c5cc8a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5806d750 con 0x7f4c5806b6f0 2024-04-06T12:31:59.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c5659c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4c580cd3e0 0x7f4c580cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43918/0 (socket says 172.21.15.189:43918) 2024-04-06T12:31:59.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c5659c700 1 -- 172.21.15.189:0/1214383700 learned_addr learned my addr 172.21.15.189:0/1214383700 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:31:59.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c55d9b700 1 --2- 172.21.15.189:0/1214383700 >> v2:172.21.15.192:3300/0 conn(0x7f4c5815d7c0 0x7f4c5806b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c56d9d700 1 --2- 172.21.15.189:0/1214383700 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c5806b6f0 0x7f4c5806dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c5559a700 1 -- 172.21.15.189:0/1214383700 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523598986 0 0) 0x7f4c5806ceb0 con 0x7f4c580cd3e0 2024-04-06T12:31:59.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c5559a700 1 -- 172.21.15.189:0/1214383700 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c400035b0 con 0x7f4c580cd3e0 2024-04-06T12:31:59.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c5559a700 1 -- 172.21.15.189:0/1214383700 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3725893537 0 0) 0x7f4c400035b0 con 0x7f4c580cd3e0 2024-04-06T12:31:59.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c5559a700 1 -- 172.21.15.189:0/1214383700 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c5806b6f0 msgr2=0x7f4c5806dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:59.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c5559a700 1 --2- 172.21.15.189:0/1214383700 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c5806b6f0 0x7f4c5806dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c5559a700 1 -- 172.21.15.189:0/1214383700 >> v2:172.21.15.192:3300/0 conn(0x7f4c5815d7c0 msgr2=0x7f4c5806b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:59.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c5559a700 1 --2- 172.21.15.189:0/1214383700 >> v2:172.21.15.192:3300/0 conn(0x7f4c5815d7c0 0x7f4c5806b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c5559a700 1 -- 172.21.15.189:0/1214383700 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5806bf00 con 0x7f4c580cd3e0 2024-04-06T12:31:59.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c5559a700 1 -- 172.21.15.189:0/1214383700 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f4c48001670 con 0x7f4c580cd3e0 2024-04-06T12:31:59.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c55d9b700 1 --2- 172.21.15.189:0/1214383700 >> v2:172.21.15.192:3300/0 conn(0x7f4c5815d7c0 0x7f4c5806b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:31:59.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c5559a700 1 -- 172.21.15.189:0/1214383700 >> v1:172.21.15.189:6789/0 conn(0x7f4c580cd3e0 legacy=0x7f4c580cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:59.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.906+0000 7f4c5559a700 1 --2- 172.21.15.189:0/1214383700 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c5806b6f0 0x7f4c40003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.907+0000 7f4c5559a700 1 --2- 172.21.15.189:0/1214383700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c40003f80 0x7f4c40004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.907+0000 7f4c5559a700 1 --2- 172.21.15.189:0/1214383700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c40004d90 0x7f4c40005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.907+0000 7f4c5559a700 1 -- 172.21.15.189:0/1214383700 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c580cdfa0 con 0x7f4c40003f80 2024-04-06T12:31:59.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.907+0000 7f4c5559a700 1 -- 172.21.15.189:0/1214383700 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5806d750 con 0x7f4c40004d90 2024-04-06T12:31:59.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.907+0000 7f4c5559a700 1 -- 172.21.15.189:0/1214383700 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4000ba60 con 0x7f4c5806b6f0 2024-04-06T12:31:59.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.907+0000 7f4c55d9b700 1 --2- 172.21.15.189:0/1214383700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c40003f80 0x7f4c40004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.907+0000 7f4c5659c700 1 --2- 172.21.15.189:0/1214383700 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c5806b6f0 0x7f4c40003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.907+0000 7f4c56d9d700 1 --2- 172.21.15.189:0/1214383700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c40004d90 0x7f4c40005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.907+0000 7f4c55d9b700 1 -- 172.21.15.189:0/1214383700 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c5806b6f0 msgr2=0x7f4c40003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:59.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.907+0000 7f4c55d9b700 1 --2- 172.21.15.189:0/1214383700 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c5806b6f0 0x7f4c40003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.907+0000 7f4c55d9b700 1 -- 172.21.15.189:0/1214383700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c40004d90 msgr2=0x7f4c40005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:31:59.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.907+0000 7f4c55d9b700 1 --2- 172.21.15.189:0/1214383700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c40004d90 0x7f4c40005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.907+0000 7f4c55d9b700 1 -- 172.21.15.189:0/1214383700 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c400035b0 con 0x7f4c40003f80 2024-04-06T12:31:59.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.907+0000 7f4c5659c700 1 --2- 172.21.15.189:0/1214383700 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c5806b6f0 0x7f4c40003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:59.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.907+0000 7f4c56d9d700 1 --2- 172.21.15.189:0/1214383700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c40004d90 0x7f4c40005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:31:59.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.907+0000 7f4c55d9b700 1 --2- 172.21.15.189:0/1214383700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c40003f80 0x7f4c40004850 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009d60 tx=0x7f4c40005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd02e8065fdce2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:59.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.907+0000 7f4c5559a700 1 -- 172.21.15.189:0/1214383700 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c00e9a0 con 0x7f4c40003f80 2024-04-06T12:31:59.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.907+0000 7f4c5559a700 1 -- 172.21.15.189:0/1214383700 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c00eb00 con 0x7f4c40003f80 2024-04-06T12:31:59.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.907+0000 7f4c5559a700 1 -- 172.21.15.189:0/1214383700 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c00eeb0 con 0x7f4c40003f80 2024-04-06T12:31:59.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.908+0000 7f4c5cc8a700 1 -- 172.21.15.189:0/1214383700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c40003f80 msgr2=0x7f4c40004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:59.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.908+0000 7f4c5cc8a700 1 --2- 172.21.15.189:0/1214383700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c40003f80 0x7f4c40004850 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009d60 tx=0x7f4c40005880 comp rx=0 tx=0).stop 2024-04-06T12:31:59.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.908+0000 7f4c56d9d700 1 -- 172.21.15.189:0/1214383700 reap_dead start 2024-04-06T12:31:59.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.908+0000 7f4c5cc8a700 1 -- 172.21.15.189:0/1214383700 shutdown_connections 2024-04-06T12:31:59.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.908+0000 7f4c5cc8a700 1 -- 172.21.15.189:0/1214383700 >> 172.21.15.189:0/1214383700 conn(0x7f4c580cc290 msgr2=0x7f4c580cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:31:59.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.908+0000 7f4c5cc8a700 1 -- 172.21.15.189:0/1214383700 shutdown_connections 2024-04-06T12:31:59.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.908+0000 7f4c5cc8a700 1 -- 172.21.15.189:0/1214383700 wait complete. 2024-04-06T12:31:59.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.909+0000 7f4c5cc8a700 1 Processor -- start 2024-04-06T12:31:59.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.909+0000 7f4c5cc8a700 1 -- start start 2024-04-06T12:31:59.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.910+0000 7f4c5cc8a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c40003f80 0x7f4c58159020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.910+0000 7f4c5cc8a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c580cb810 0x7f4c580b4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.910+0000 7f4c5cc8a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c580ae2f0 0x7f4c580b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.910+0000 7f4c5cc8a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c58159d30 con 0x7f4c580ae2f0 2024-04-06T12:31:59.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.910+0000 7f4c5cc8a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5815c4b0 con 0x7f4c40003f80 2024-04-06T12:31:59.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.910+0000 7f4c5cc8a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c581596e0 con 0x7f4c580cb810 2024-04-06T12:31:59.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.910+0000 7f4c56d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c580ae2f0 0x7f4c580b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.910+0000 7f4c56d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c580ae2f0 0x7f4c580b49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60578/0 (socket says 172.21.15.189:60578) 2024-04-06T12:31:59.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.910+0000 7f4c56d9d700 1 -- 172.21.15.189:0/174793138 learned_addr learned my addr 172.21.15.189:0/174793138 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:31:59.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.910+0000 7f4c5659c700 1 --2- 172.21.15.189:0/174793138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c40003f80 0x7f4c58159020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.910+0000 7f4c55d9b700 1 --2- 172.21.15.189:0/174793138 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c580cb810 0x7f4c580b4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.910+0000 7f4c56d9d700 1 -- 172.21.15.189:0/174793138 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c580cb810 msgr2=0x7f4c580b4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:59.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.910+0000 7f4c56d9d700 1 --2- 172.21.15.189:0/174793138 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c580cb810 0x7f4c580b4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.910+0000 7f4c56d9d700 1 -- 172.21.15.189:0/174793138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c40003f80 msgr2=0x7f4c58159020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:31:59.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.910+0000 7f4c56d9d700 1 --2- 172.21.15.189:0/174793138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c40003f80 0x7f4c58159020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:31:59.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.910+0000 7f4c56d9d700 1 -- 172.21.15.189:0/174793138 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c581599c0 con 0x7f4c580ae2f0 2024-04-06T12:31:59.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.910+0000 7f4c56d9d700 1 --2- 172.21.15.189:0/174793138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c580ae2f0 0x7f4c580b49b0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f4c5806cc10 tx=0x7f4c500104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:59.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.911+0000 7f4c46ffd700 1 -- 172.21.15.189:0/174793138 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5001b070 con 0x7f4c580ae2f0 2024-04-06T12:31:59.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.911+0000 7f4c46ffd700 1 -- 172.21.15.189:0/174793138 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5000f040 con 0x7f4c580ae2f0 2024-04-06T12:31:59.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.911+0000 7f4c5cc8a700 1 -- 172.21.15.189:0/174793138 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5815c790 con 0x7f4c580ae2f0 2024-04-06T12:31:59.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.911+0000 7f4c46ffd700 1 -- 172.21.15.189:0/174793138 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5000e050 con 0x7f4c580ae2f0 2024-04-06T12:31:59.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.911+0000 7f4c5cc8a700 1 -- 172.21.15.189:0/174793138 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c5815c220 con 0x7f4c580ae2f0 2024-04-06T12:31:59.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.912+0000 7f4c5cc8a700 1 -- 172.21.15.189:0/174793138 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c58159b50 con 0x7f4c580ae2f0 2024-04-06T12:31:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.912+0000 7f4c46ffd700 1 -- 172.21.15.189:0/174793138 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f4c50010d00 con 0x7f4c580ae2f0 2024-04-06T12:31:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.912+0000 7f4c46ffd700 1 --2- 172.21.15.189:0/174793138 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4c3806ade0 0x7f4c3806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:31:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.913+0000 7f4c5659c700 1 --2- 172.21.15.189:0/174793138 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4c3806ade0 0x7f4c3806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:31:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.913+0000 7f4c46ffd700 1 -- 172.21.15.189:0/174793138 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f4c50089230 con 0x7f4c580ae2f0 2024-04-06T12:31:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.913+0000 7f4c5659c700 1 --2- 172.21.15.189:0/174793138 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4c3806ade0 0x7f4c3806d2a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4c48000fb0 tx=0x7f4c48007040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:31:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:31:59.917+0000 7f4c46ffd700 1 -- 172.21.15.189:0/174793138 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4c50054490 con 0x7f4c580ae2f0 2024-04-06T12:32:00.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.114+0000 7f4c5cc8a700 1 -- 172.21.15.189:0/174793138 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4c58074990 con 0x7f4c580ae2f0 2024-04-06T12:32:00.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.114+0000 7f4c46ffd700 1 -- 172.21.15.189:0/174793138 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v154) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4c50057ab0 con 0x7f4c580ae2f0 2024-04-06T12:32:00.111 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T12:32:00.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.116+0000 7f4c5cc8a700 1 -- 172.21.15.189:0/174793138 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4c3806ade0 msgr2=0x7f4c3806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:00.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.116+0000 7f4c5cc8a700 1 --2- 172.21.15.189:0/174793138 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4c3806ade0 0x7f4c3806d2a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4c48000fb0 tx=0x7f4c48007040 comp rx=0 tx=0).stop 2024-04-06T12:32:00.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.116+0000 7f4c5cc8a700 1 -- 172.21.15.189:0/174793138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c580ae2f0 msgr2=0x7f4c580b49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:00.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.116+0000 7f4c5cc8a700 1 --2- 172.21.15.189:0/174793138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c580ae2f0 0x7f4c580b49b0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f4c5806cc10 tx=0x7f4c500104a0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.116+0000 7f4c5cc8a700 1 -- 172.21.15.189:0/174793138 shutdown_connections 2024-04-06T12:32:00.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.116+0000 7f4c5cc8a700 1 --2- 172.21.15.189:0/174793138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c580ae2f0 0x7f4c580b49b0 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.116+0000 7f4c5cc8a700 1 --2- 172.21.15.189:0/174793138 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4c3806ade0 0x7f4c3806d2a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.116+0000 7f4c5cc8a700 1 --2- 172.21.15.189:0/174793138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c40003f80 0x7f4c58159020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.116+0000 7f4c5cc8a700 1 --2- 172.21.15.189:0/174793138 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c580cb810 0x7f4c580b4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.116+0000 7f4c5cc8a700 1 -- 172.21.15.189:0/174793138 >> 172.21.15.189:0/174793138 conn(0x7f4c580cc290 msgr2=0x7f4c580afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:00.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.116+0000 7f4c5cc8a700 1 -- 172.21.15.189:0/174793138 shutdown_connections 2024-04-06T12:32:00.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.116+0000 7f4c5cc8a700 1 -- 172.21.15.189:0/174793138 wait complete. 2024-04-06T12:32:00.126 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-06T12:32:00.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.250+0000 7fcc4c5b4700 1 Processor -- start 2024-04-06T12:32:00.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.251+0000 7fcc4c5b4700 1 -- start start 2024-04-06T12:32:00.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.252+0000 7fcc4c5b4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcc4406ae40 0x7fcc440b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:00.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.252+0000 7fcc4c5b4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcc4415ddf0 0x7fcc4415e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:00.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.252+0000 7fcc4c5b4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc440b3990 con 0x7fcc4415d7b0 2024-04-06T12:32:00.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.252+0000 7fcc4c5b4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc440b3060 con 0x7fcc4415ddf0 2024-04-06T12:32:00.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.252+0000 7fcc4c5b4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc440b3340 con 0x7fcc4406ae40 2024-04-06T12:32:00.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.252+0000 7fcc49b4f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcc4406ae40 0x7fcc440b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:00.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.252+0000 7fcc4ab51700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcc4415ddf0 0x7fcc4415e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:00.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.252+0000 7fcc4a350700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fcc4415d7b0 0x7fcc4406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:37130/0 (socket says 172.21.15.189:37130) 2024-04-06T12:32:00.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.252+0000 7fcc4ab51700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcc4415ddf0 0x7fcc4415e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:38840/0 (socket says 172.21.15.189:38840) 2024-04-06T12:32:00.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.252+0000 7fcc4a350700 1 -- 172.21.15.189:0/3802495660 learned_addr learned my addr 172.21.15.189:0/3802495660 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:32:00.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.252+0000 7fcc4ab51700 1 -- 172.21.15.189:0/3802495660 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcc4406ae40 msgr2=0x7fcc440b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:00.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.252+0000 7fcc4ab51700 1 --2- 172.21.15.189:0/3802495660 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcc4406ae40 0x7fcc440b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.252+0000 7fcc4ab51700 1 -- 172.21.15.189:0/3802495660 >> v1:172.21.15.192:6789/0 conn(0x7fcc4415d7b0 legacy=0x7fcc4406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:00.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.252+0000 7fcc4ab51700 1 -- 172.21.15.189:0/3802495660 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc440b3620 con 0x7fcc4415ddf0 2024-04-06T12:32:00.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.252+0000 7fcc4ab51700 1 --2- 172.21.15.189:0/3802495660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc4415ddf0 0x7fcc4415e2a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c009d90 tx=0x7fcc3c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e96f6e017f845284 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:00.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.253+0000 7fcc4934e700 1 -- 172.21.15.189:0/3802495660 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3c00e070 con 0x7fcc4415ddf0 2024-04-06T12:32:00.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.253+0000 7fcc4934e700 1 -- 172.21.15.189:0/3802495660 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3c00bb20 con 0x7fcc4415ddf0 2024-04-06T12:32:00.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.253+0000 7fcc4934e700 1 -- 172.21.15.189:0/3802495660 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3c012430 con 0x7fcc4415ddf0 2024-04-06T12:32:00.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.253+0000 7fcc4c5b4700 1 -- 172.21.15.189:0/3802495660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc4415ddf0 msgr2=0x7fcc4415e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:00.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.253+0000 7fcc4c5b4700 1 --2- 172.21.15.189:0/3802495660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc4415ddf0 0x7fcc4415e2a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c009d90 tx=0x7fcc3c00b330 comp rx=0 tx=0).stop 2024-04-06T12:32:00.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.253+0000 7fcc4c5b4700 1 -- 172.21.15.189:0/3802495660 shutdown_connections 2024-04-06T12:32:00.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.253+0000 7fcc4c5b4700 1 --2- 172.21.15.189:0/3802495660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc4415ddf0 0x7fcc4415e2a0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.253+0000 7fcc4c5b4700 1 --2- 172.21.15.189:0/3802495660 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcc4406ae40 0x7fcc440b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.253+0000 7fcc4c5b4700 1 -- 172.21.15.189:0/3802495660 >> 172.21.15.189:0/3802495660 conn(0x7fcc440d00b0 msgr2=0x7fcc440d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:00.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.253+0000 7fcc4c5b4700 1 -- 172.21.15.189:0/3802495660 shutdown_connections 2024-04-06T12:32:00.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.253+0000 7fcc4c5b4700 1 -- 172.21.15.189:0/3802495660 wait complete. 2024-04-06T12:32:00.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.254+0000 7fcc4c5b4700 1 Processor -- start 2024-04-06T12:32:00.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.254+0000 7fcc4c5b4700 1 -- start start 2024-04-06T12:32:00.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.255+0000 7fcc4c5b4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc4406ae40 0x7fcc44150b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:00.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.255+0000 7fcc4c5b4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc4415d7b0 0x7fcc44151eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:00.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.255+0000 7fcc4c5b4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcc4415ddf0 0x7fcc441552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:00.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.255+0000 7fcc4c5b4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc440af830 con 0x7fcc4406ae40 2024-04-06T12:32:00.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.255+0000 7fcc4c5b4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc440aec20 con 0x7fcc4415d7b0 2024-04-06T12:32:00.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.255+0000 7fcc4c5b4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc440aef00 con 0x7fcc4415ddf0 2024-04-06T12:32:00.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.255+0000 7fcc4a350700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc4406ae40 0x7fcc44150b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:00.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.255+0000 7fcc4a350700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc4406ae40 0x7fcc44150b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60590/0 (socket says 172.21.15.189:60590) 2024-04-06T12:32:00.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.255+0000 7fcc4a350700 1 -- 172.21.15.189:0/1973310160 learned_addr learned my addr 172.21.15.189:0/1973310160 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:00.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.255+0000 7fcc49b4f700 1 --2- 172.21.15.189:0/1973310160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc4415d7b0 0x7fcc44151eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:00.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.255+0000 7fcc4ab51700 1 --2- 172.21.15.189:0/1973310160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcc4415ddf0 0x7fcc441552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:00.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.255+0000 7fcc4a350700 1 -- 172.21.15.189:0/1973310160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcc4415ddf0 msgr2=0x7fcc441552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:00.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.255+0000 7fcc4a350700 1 --2- 172.21.15.189:0/1973310160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcc4415ddf0 0x7fcc441552f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.255+0000 7fcc4a350700 1 -- 172.21.15.189:0/1973310160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc4415d7b0 msgr2=0x7fcc44151eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:00.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.255+0000 7fcc4a350700 1 --2- 172.21.15.189:0/1973310160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc4415d7b0 0x7fcc44151eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.255+0000 7fcc4a350700 1 -- 172.21.15.189:0/1973310160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3c009a70 con 0x7fcc4406ae40 2024-04-06T12:32:00.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.255+0000 7fcc49b4f700 1 --2- 172.21.15.189:0/1973310160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc4415d7b0 0x7fcc44151eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:32:00.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.256+0000 7fcc4a350700 1 --2- 172.21.15.189:0/1973310160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc4406ae40 0x7fcc44150b50 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fcc4000c520 tx=0x7fcc4000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:00.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.256+0000 7fcc36ffd700 1 -- 172.21.15.189:0/1973310160 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc40005d70 con 0x7fcc4406ae40 2024-04-06T12:32:00.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.256+0000 7fcc36ffd700 1 -- 172.21.15.189:0/1973310160 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc400070d0 con 0x7fcc4406ae40 2024-04-06T12:32:00.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.256+0000 7fcc4c5b4700 1 -- 172.21.15.189:0/1973310160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc440af1e0 con 0x7fcc4406ae40 2024-04-06T12:32:00.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.256+0000 7fcc36ffd700 1 -- 172.21.15.189:0/1973310160 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc40007440 con 0x7fcc4406ae40 2024-04-06T12:32:00.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.256+0000 7fcc4c5b4700 1 -- 172.21.15.189:0/1973310160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc44151090 con 0x7fcc4406ae40 2024-04-06T12:32:00.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.257+0000 7fcc4c5b4700 1 -- 172.21.15.189:0/1973310160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc44067bb0 con 0x7fcc4406ae40 2024-04-06T12:32:00.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.257+0000 7fcc36ffd700 1 -- 172.21.15.189:0/1973310160 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fcc400075a0 con 0x7fcc4406ae40 2024-04-06T12:32:00.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.257+0000 7fcc36ffd700 1 --2- 172.21.15.189:0/1973310160 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcc2806adb0 0x7fcc2806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:00.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.258+0000 7fcc36ffd700 1 -- 172.21.15.189:0/1973310160 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fcc40088630 con 0x7fcc4406ae40 2024-04-06T12:32:00.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.258+0000 7fcc49b4f700 1 --2- 172.21.15.189:0/1973310160 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcc2806adb0 0x7fcc2806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:00.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.258+0000 7fcc49b4f700 1 --2- 172.21.15.189:0/1973310160 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcc2806adb0 0x7fcc2806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcc440cfa90 tx=0x7fcc38008040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:00.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.262+0000 7fcc36ffd700 1 -- 172.21.15.189:0/1973310160 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fcc400539c0 con 0x7fcc4406ae40 2024-04-06T12:32:00.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.459+0000 7fcc4c5b4700 1 -- 172.21.15.189:0/1973310160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fcc44074990 con 0x7fcc4406ae40 2024-04-06T12:32:00.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.459+0000 7fcc36ffd700 1 -- 172.21.15.189:0/1973310160 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v154) v1 ==== 84+0+65 (secure 0 0 0) 0x7fcc40017060 con 0x7fcc4406ae40 2024-04-06T12:32:00.457 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:32:00.457 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T12:32:00.457 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:32:00.457 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 1, 2024-04-06T12:32:00.457 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": ".mgr" 2024-04-06T12:32:00.457 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:32:00.457 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T12:32:00.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.461+0000 7fcc4c5b4700 1 -- 172.21.15.189:0/1973310160 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcc2806adb0 msgr2=0x7fcc2806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:00.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.461+0000 7fcc4c5b4700 1 --2- 172.21.15.189:0/1973310160 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcc2806adb0 0x7fcc2806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcc440cfa90 tx=0x7fcc38008040 comp rx=0 tx=0).stop 2024-04-06T12:32:00.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.461+0000 7fcc4c5b4700 1 -- 172.21.15.189:0/1973310160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc4406ae40 msgr2=0x7fcc44150b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:00.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.461+0000 7fcc4c5b4700 1 --2- 172.21.15.189:0/1973310160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc4406ae40 0x7fcc44150b50 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fcc4000c520 tx=0x7fcc4000c8e0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.461+0000 7fcc4c5b4700 1 -- 172.21.15.189:0/1973310160 shutdown_connections 2024-04-06T12:32:00.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.461+0000 7fcc4c5b4700 1 --2- 172.21.15.189:0/1973310160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcc4415ddf0 0x7fcc441552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.461+0000 7fcc4c5b4700 1 --2- 172.21.15.189:0/1973310160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc4406ae40 0x7fcc44150b50 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.461+0000 7fcc4c5b4700 1 --2- 172.21.15.189:0/1973310160 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fcc2806adb0 0x7fcc2806d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.461+0000 7fcc4c5b4700 1 --2- 172.21.15.189:0/1973310160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc4415d7b0 0x7fcc44151eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.461+0000 7fcc4c5b4700 1 -- 172.21.15.189:0/1973310160 >> 172.21.15.189:0/1973310160 conn(0x7fcc440d00b0 msgr2=0x7fcc4415c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:00.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.462+0000 7fcc4c5b4700 1 -- 172.21.15.189:0/1973310160 shutdown_connections 2024-04-06T12:32:00.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.462+0000 7fcc4c5b4700 1 -- 172.21.15.189:0/1973310160 wait complete. 2024-04-06T12:32:00.471 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-06T12:32:00.471 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-06T12:32:00.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.600+0000 7f1408700700 1 Processor -- start 2024-04-06T12:32:00.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.600+0000 7f1408700700 1 -- start start 2024-04-06T12:32:00.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.601+0000 7f1408700700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f14000b4660 0x7f14000b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:00.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.601+0000 7f1408700700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f14000b2fc0 0x7f14000b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:00.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.601+0000 7f1408700700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14000b5030 con 0x7f140015a550 2024-04-06T12:32:00.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.601+0000 7f1408700700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14000cdbe0 con 0x7f14000b2fc0 2024-04-06T12:32:00.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.601+0000 7f1408700700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f140015a940 con 0x7f14000b4660 2024-04-06T12:32:00.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.601+0000 7f1406c9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f14000b2fc0 0x7f14000b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:00.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.601+0000 7f140649c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f140015a550 0x7f14000cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43922/0 (socket says 172.21.15.189:43922) 2024-04-06T12:32:00.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.601+0000 7f140649c700 1 -- 172.21.15.189:0/3518543203 learned_addr learned my addr 172.21.15.189:0/3518543203 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:32:00.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.601+0000 7f1405c9b700 1 --2- 172.21.15.189:0/3518543203 >> v2:172.21.15.192:3300/0 conn(0x7f14000b4660 0x7f14000b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:00.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.601+0000 7f1406c9d700 1 -- 172.21.15.189:0/3518543203 >> v2:172.21.15.192:3300/0 conn(0x7f14000b4660 msgr2=0x7f14000b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:00.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.601+0000 7f1406c9d700 1 --2- 172.21.15.189:0/3518543203 >> v2:172.21.15.192:3300/0 conn(0x7f14000b4660 0x7f14000b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.601+0000 7f1406c9d700 1 -- 172.21.15.189:0/3518543203 >> v1:172.21.15.189:6789/0 conn(0x7f140015a550 legacy=0x7f14000cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.601+0000 7f1406c9d700 1 -- 172.21.15.189:0/3518543203 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f140015ac00 con 0x7f14000b2fc0 2024-04-06T12:32:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.601+0000 7f140549a700 1 -- 172.21.15.189:0/3518543203 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635699689 0 0) 0x7f14000b5030 con 0x7f140015a550 2024-04-06T12:32:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.602+0000 7f1406c9d700 1 --2- 172.21.15.189:0/3518543203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14000b2fc0 0x7f14000b33d0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f14000b2d40 tx=0x7f13f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf108893b0ae07f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.602+0000 7f140549a700 1 -- 172.21.15.189:0/3518543203 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f800e070 con 0x7f14000b2fc0 2024-04-06T12:32:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.602+0000 7f140549a700 1 -- 172.21.15.189:0/3518543203 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f800bbd0 con 0x7f14000b2fc0 2024-04-06T12:32:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.602+0000 7f140549a700 1 -- 172.21.15.189:0/3518543203 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f8012430 con 0x7f14000b2fc0 2024-04-06T12:32:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.602+0000 7f1408700700 1 -- 172.21.15.189:0/3518543203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14000b2fc0 msgr2=0x7f14000b33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.602+0000 7f1408700700 1 --2- 172.21.15.189:0/3518543203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14000b2fc0 0x7f14000b33d0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f14000b2d40 tx=0x7f13f800b330 comp rx=0 tx=0).stop 2024-04-06T12:32:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.602+0000 7f1408700700 1 -- 172.21.15.189:0/3518543203 shutdown_connections 2024-04-06T12:32:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.602+0000 7f1408700700 1 --2- 172.21.15.189:0/3518543203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14000b2fc0 0x7f14000b33d0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.602+0000 7f1408700700 1 --2- 172.21.15.189:0/3518543203 >> v2:172.21.15.192:3300/0 conn(0x7f14000b4660 0x7f14000b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.602+0000 7f1408700700 1 -- 172.21.15.189:0/3518543203 >> 172.21.15.189:0/3518543203 conn(0x7f14000d00b0 msgr2=0x7f14000d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.602+0000 7f1408700700 1 -- 172.21.15.189:0/3518543203 shutdown_connections 2024-04-06T12:32:00.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.602+0000 7f1408700700 1 -- 172.21.15.189:0/3518543203 wait complete. 2024-04-06T12:32:00.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.604+0000 7f1408700700 1 Processor -- start 2024-04-06T12:32:00.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.604+0000 7f1408700700 1 -- start start 2024-04-06T12:32:00.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.604+0000 7f1408700700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14000b2fc0 0x7f1400151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:00.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.604+0000 7f1408700700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f14000b4660 0x7f140015d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:00.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.604+0000 7f1408700700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14000adf80 0x7f140015d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:00.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.604+0000 7f1408700700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14001f6e30 con 0x7f14000b2fc0 2024-04-06T12:32:00.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.604+0000 7f1408700700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14001f2b60 con 0x7f14000adf80 2024-04-06T12:32:00.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.604+0000 7f1408700700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14001f2e40 con 0x7f14000b4660 2024-04-06T12:32:00.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.604+0000 7f140649c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14000b2fc0 0x7f1400151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:00.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.604+0000 7f140649c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14000b2fc0 0x7f1400151ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60602/0 (socket says 172.21.15.189:60602) 2024-04-06T12:32:00.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.604+0000 7f140649c700 1 -- 172.21.15.189:0/629520313 learned_addr learned my addr 172.21.15.189:0/629520313 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:00.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.605+0000 7f1406c9d700 1 --2- 172.21.15.189:0/629520313 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14000adf80 0x7f140015d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:00.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.605+0000 7f1405c9b700 1 --2- 172.21.15.189:0/629520313 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f14000b4660 0x7f140015d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:00.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.605+0000 7f140649c700 1 -- 172.21.15.189:0/629520313 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f14000b4660 msgr2=0x7f140015d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:00.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.605+0000 7f140649c700 1 --2- 172.21.15.189:0/629520313 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f14000b4660 0x7f140015d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.605+0000 7f140649c700 1 -- 172.21.15.189:0/629520313 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14000adf80 msgr2=0x7f140015d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.605+0000 7f140649c700 1 --2- 172.21.15.189:0/629520313 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14000adf80 0x7f140015d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.605+0000 7f140649c700 1 -- 172.21.15.189:0/629520313 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f8009a70 con 0x7f14000b2fc0 2024-04-06T12:32:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.605+0000 7f1406c9d700 1 --2- 172.21.15.189:0/629520313 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14000adf80 0x7f140015d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:32:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.605+0000 7f140649c700 1 --2- 172.21.15.189:0/629520313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14000b2fc0 0x7f1400151ef0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f13f00011e0 tx=0x7f13f000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.605+0000 7f13f6ffd700 1 -- 172.21.15.189:0/629520313 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f0004c80 con 0x7f14000b2fc0 2024-04-06T12:32:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.605+0000 7f13f6ffd700 1 -- 172.21.15.189:0/629520313 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f0004e00 con 0x7f14000b2fc0 2024-04-06T12:32:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.605+0000 7f1408700700 1 -- 172.21.15.189:0/629520313 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14001f3120 con 0x7f14000b2fc0 2024-04-06T12:32:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.605+0000 7f13f6ffd700 1 -- 172.21.15.189:0/629520313 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f0007310 con 0x7f14000b2fc0 2024-04-06T12:32:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.605+0000 7f1408700700 1 -- 172.21.15.189:0/629520313 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14001f36e0 con 0x7f14000b2fc0 2024-04-06T12:32:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.606+0000 7f1408700700 1 -- 172.21.15.189:0/629520313 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1400067bb0 con 0x7f14000b2fc0 2024-04-06T12:32:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.606+0000 7f13f6ffd700 1 -- 172.21.15.189:0/629520313 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f13f0007470 con 0x7f14000b2fc0 2024-04-06T12:32:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.607+0000 7f13f6ffd700 1 --2- 172.21.15.189:0/629520313 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13e406adb0 0x7f13e406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.607+0000 7f13f6ffd700 1 -- 172.21.15.189:0/629520313 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f13f0089010 con 0x7f14000b2fc0 2024-04-06T12:32:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.607+0000 7f1405c9b700 1 --2- 172.21.15.189:0/629520313 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13e406adb0 0x7f13e406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:00.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.607+0000 7f1405c9b700 1 --2- 172.21.15.189:0/629520313 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13e406adb0 0x7f13e406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1400158a20 tx=0x7f13fc008040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:00.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.611+0000 7f13f6ffd700 1 -- 172.21.15.189:0/629520313 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f13f00542f0 con 0x7f14000b2fc0 2024-04-06T12:32:00.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:00.807+0000 7f1408700700 1 -- 172.21.15.189:0/629520313 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1400074990 con 0x7f14000b2fc0 2024-04-06T12:32:01.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.654+0000 7f13f6ffd700 1 -- 172.21.15.189:0/629520313 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v155) v1 ==== 138+0+0 (secure 0 0 0) 0x7f13f0057910 con 0x7f14000b2fc0 2024-04-06T12:32:01.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.658+0000 7f1408700700 1 -- 172.21.15.189:0/629520313 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13e406adb0 msgr2=0x7f13e406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:01.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.658+0000 7f1408700700 1 --2- 172.21.15.189:0/629520313 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13e406adb0 0x7f13e406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1400158a20 tx=0x7f13fc008040 comp rx=0 tx=0).stop 2024-04-06T12:32:01.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.658+0000 7f1408700700 1 -- 172.21.15.189:0/629520313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14000b2fc0 msgr2=0x7f1400151ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:01.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.658+0000 7f1408700700 1 --2- 172.21.15.189:0/629520313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14000b2fc0 0x7f1400151ef0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f13f00011e0 tx=0x7f13f000cae0 comp rx=0 tx=0).stop 2024-04-06T12:32:01.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.659+0000 7f1408700700 1 -- 172.21.15.189:0/629520313 shutdown_connections 2024-04-06T12:32:01.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.659+0000 7f1408700700 1 --2- 172.21.15.189:0/629520313 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13e406adb0 0x7f13e406d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:01.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.659+0000 7f1408700700 1 --2- 172.21.15.189:0/629520313 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14000adf80 0x7f140015d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:01.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.659+0000 7f1408700700 1 --2- 172.21.15.189:0/629520313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14000b2fc0 0x7f1400151ef0 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:01.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.659+0000 7f1408700700 1 --2- 172.21.15.189:0/629520313 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f14000b4660 0x7f140015d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:01.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.659+0000 7f1408700700 1 -- 172.21.15.189:0/629520313 >> 172.21.15.189:0/629520313 conn(0x7f14000d00b0 msgr2=0x7f140006eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:01.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.659+0000 7f1408700700 1 -- 172.21.15.189:0/629520313 shutdown_connections 2024-04-06T12:32:01.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.659+0000 7f1408700700 1 -- 172.21.15.189:0/629520313 wait complete. 2024-04-06T12:32:01.660 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' created 2024-04-06T12:32:01.674 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-06T12:32:01.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.813+0000 7f6f2b816700 1 Processor -- start 2024-04-06T12:32:01.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.813+0000 7f6f2b816700 1 -- start start 2024-04-06T12:32:01.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.814+0000 7f6f2b816700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f2415a550 0x7f6f240cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:01.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.814+0000 7f6f2b816700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f240cdbe0 con 0x7f6f240b4d70 2024-04-06T12:32:01.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.814+0000 7f6f2b816700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2415a940 con 0x7f6f240cf900 2024-04-06T12:32:01.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.814+0000 7f6f2b816700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2415ac00 con 0x7f6f2415a550 2024-04-06T12:32:01.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.814+0000 7f6f29db3700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6f240b4d70 0x7f6f240b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43934/0 (socket says 172.21.15.189:43934) 2024-04-06T12:32:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.814+0000 7f6f29db3700 1 -- 172.21.15.189:0/2887996893 learned_addr learned my addr 172.21.15.189:0/2887996893 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:32:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.815+0000 7f6f295b2700 1 --2- 172.21.15.189:0/2887996893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f2415a550 0x7f6f240cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.815+0000 7f6f1bfff700 1 -- 172.21.15.189:0/2887996893 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405035058 0 0) 0x7f6f240cdbe0 con 0x7f6f240b4d70 2024-04-06T12:32:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.815+0000 7f6f1bfff700 1 -- 172.21.15.189:0/2887996893 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f100035b0 con 0x7f6f240b4d70 2024-04-06T12:32:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.815+0000 7f6f1bfff700 1 -- 172.21.15.189:0/2887996893 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073580439 0 0) 0x7f6f2415a940 con 0x7f6f240cf900 2024-04-06T12:32:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.815+0000 7f6f1bfff700 1 -- 172.21.15.189:0/2887996893 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f240cdbe0 con 0x7f6f240cf900 2024-04-06T12:32:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.815+0000 7f6f295b2700 1 -- 172.21.15.189:0/2887996893 >> v1:172.21.15.192:6789/0 conn(0x7f6f240cf900 legacy=0x7f6f240b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.815+0000 7f6f295b2700 1 -- 172.21.15.189:0/2887996893 >> v1:172.21.15.189:6789/0 conn(0x7f6f240b4d70 legacy=0x7f6f240b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.815+0000 7f6f295b2700 1 -- 172.21.15.189:0/2887996893 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2415aee0 con 0x7f6f2415a550 2024-04-06T12:32:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.815+0000 7f6f295b2700 1 --2- 172.21.15.189:0/2887996893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f2415a550 0x7f6f240cf3c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6f14009870 tx=0x7f6f14009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bc68d8321cf90a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.815+0000 7f6f1bfff700 1 -- 172.21.15.189:0/2887996893 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f14011830 con 0x7f6f2415a550 2024-04-06T12:32:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.815+0000 7f6f1bfff700 1 -- 172.21.15.189:0/2887996893 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f14011990 con 0x7f6f2415a550 2024-04-06T12:32:01.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.815+0000 7f6f1bfff700 1 -- 172.21.15.189:0/2887996893 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f14011d20 con 0x7f6f2415a550 2024-04-06T12:32:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.816+0000 7f6f2b816700 1 -- 172.21.15.189:0/2887996893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f2415a550 msgr2=0x7f6f240cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.816+0000 7f6f2b816700 1 --2- 172.21.15.189:0/2887996893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f2415a550 0x7f6f240cf3c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6f14009870 tx=0x7f6f14009b80 comp rx=0 tx=0).stop 2024-04-06T12:32:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.816+0000 7f6f2b816700 1 -- 172.21.15.189:0/2887996893 shutdown_connections 2024-04-06T12:32:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.816+0000 7f6f2b816700 1 --2- 172.21.15.189:0/2887996893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f2415a550 0x7f6f240cf3c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.816+0000 7f6f2b816700 1 -- 172.21.15.189:0/2887996893 >> 172.21.15.189:0/2887996893 conn(0x7f6f240d00b0 msgr2=0x7f6f240d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.816+0000 7f6f2b816700 1 -- 172.21.15.189:0/2887996893 shutdown_connections 2024-04-06T12:32:01.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.816+0000 7f6f2b816700 1 -- 172.21.15.189:0/2887996893 wait complete. 2024-04-06T12:32:01.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.817+0000 7f6f2b816700 1 Processor -- start 2024-04-06T12:32:01.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.817+0000 7f6f2b816700 1 -- start start 2024-04-06T12:32:01.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.818+0000 7f6f2b816700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f240b4d70 0x7f6f240aeed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:01.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.818+0000 7f6f2b816700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f240cf900 0x7f6f240a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:01.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.818+0000 7f6f2b816700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f24152b90 0x7f6f240a89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:01.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.818+0000 7f6f2b816700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f24158770 con 0x7f6f240b4d70 2024-04-06T12:32:01.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.818+0000 7f6f2b816700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f241423c0 con 0x7f6f240cf900 2024-04-06T12:32:01.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.818+0000 7f6f2b816700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f24143230 con 0x7f6f24152b90 2024-04-06T12:32:01.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.818+0000 7f6f295b2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f240b4d70 0x7f6f240aeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:01.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.818+0000 7f6f295b2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f240b4d70 0x7f6f240aeed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60612/0 (socket says 172.21.15.189:60612) 2024-04-06T12:32:01.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.818+0000 7f6f295b2700 1 -- 172.21.15.189:0/595855161 learned_addr learned my addr 172.21.15.189:0/595855161 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:01.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.818+0000 7f6f28db1700 1 --2- 172.21.15.189:0/595855161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f240cf900 0x7f6f240a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:01.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.818+0000 7f6f29db3700 1 --2- 172.21.15.189:0/595855161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f24152b90 0x7f6f240a89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:01.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.818+0000 7f6f295b2700 1 -- 172.21.15.189:0/595855161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f24152b90 msgr2=0x7f6f240a89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:01.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.818+0000 7f6f295b2700 1 --2- 172.21.15.189:0/595855161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f24152b90 0x7f6f240a89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:01.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.818+0000 7f6f295b2700 1 -- 172.21.15.189:0/595855161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f240cf900 msgr2=0x7f6f240a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:01.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.818+0000 7f6f295b2700 1 --2- 172.21.15.189:0/595855161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f240cf900 0x7f6f240a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:01.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.818+0000 7f6f295b2700 1 -- 172.21.15.189:0/595855161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f140094a0 con 0x7f6f240b4d70 2024-04-06T12:32:01.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.818+0000 7f6f295b2700 1 --2- 172.21.15.189:0/595855161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f240b4d70 0x7f6f240aeed0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f6f140090b0 tx=0x7f6f14004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:01.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.818+0000 7f6f19ffb700 1 -- 172.21.15.189:0/595855161 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1401aa70 con 0x7f6f240b4d70 2024-04-06T12:32:01.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.818+0000 7f6f19ffb700 1 -- 172.21.15.189:0/595855161 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f1401abd0 con 0x7f6f240b4d70 2024-04-06T12:32:01.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.819+0000 7f6f19ffb700 1 -- 172.21.15.189:0/595855161 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f140243f0 con 0x7f6f240b4d70 2024-04-06T12:32:01.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.819+0000 7f6f2b816700 1 -- 172.21.15.189:0/595855161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f24142660 con 0x7f6f240b4d70 2024-04-06T12:32:01.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.819+0000 7f6f2b816700 1 -- 172.21.15.189:0/595855161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f24142be0 con 0x7f6f240b4d70 2024-04-06T12:32:01.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.820+0000 7f6f2b816700 1 -- 172.21.15.189:0/595855161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f24067bb0 con 0x7f6f240b4d70 2024-04-06T12:32:01.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.820+0000 7f6f19ffb700 1 -- 172.21.15.189:0/595855161 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f6f1402d450 con 0x7f6f240b4d70 2024-04-06T12:32:01.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.821+0000 7f6f19ffb700 1 --2- 172.21.15.189:0/595855161 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f0806adb0 0x7f6f0806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:01.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.821+0000 7f6f19ffb700 1 -- 172.21.15.189:0/595855161 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f6f1408a740 con 0x7f6f240b4d70 2024-04-06T12:32:01.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.821+0000 7f6f28db1700 1 --2- 172.21.15.189:0/595855161 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f0806adb0 0x7f6f0806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:01.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.821+0000 7f6f28db1700 1 --2- 172.21.15.189:0/595855161 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f0806adb0 0x7f6f0806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6f241456d0 tx=0x7f6f20009040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:01.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:01.825+0000 7f6f19ffb700 1 -- 172.21.15.189:0/595855161 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6f14055940 con 0x7f6f240b4d70 2024-04-06T12:32:02.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.020+0000 7f6f2b816700 1 -- 172.21.15.189:0/595855161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f6f24074990 con 0x7f6f240b4d70 2024-04-06T12:32:02.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.659+0000 7f6f19ffb700 1 -- 172.21.15.189:0/595855161 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v156) v1 ==== 170+0+0 (secure 0 0 0) 0x7f6f14015020 con 0x7f6f240b4d70 2024-04-06T12:32:02.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.668+0000 7f6f2b816700 1 -- 172.21.15.189:0/595855161 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f0806adb0 msgr2=0x7f6f0806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:02.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.668+0000 7f6f2b816700 1 --2- 172.21.15.189:0/595855161 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f0806adb0 0x7f6f0806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6f241456d0 tx=0x7f6f20009040 comp rx=0 tx=0).stop 2024-04-06T12:32:02.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.668+0000 7f6f2b816700 1 -- 172.21.15.189:0/595855161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f240b4d70 msgr2=0x7f6f240aeed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:02.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.668+0000 7f6f2b816700 1 --2- 172.21.15.189:0/595855161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f240b4d70 0x7f6f240aeed0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f6f140090b0 tx=0x7f6f14004750 comp rx=0 tx=0).stop 2024-04-06T12:32:02.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.669+0000 7f6f2b816700 1 -- 172.21.15.189:0/595855161 shutdown_connections 2024-04-06T12:32:02.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.669+0000 7f6f2b816700 1 --2- 172.21.15.189:0/595855161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f24152b90 0x7f6f240a89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:02.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.669+0000 7f6f2b816700 1 --2- 172.21.15.189:0/595855161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f240b4d70 0x7f6f240aeed0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:02.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.669+0000 7f6f2b816700 1 --2- 172.21.15.189:0/595855161 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6f0806adb0 0x7f6f0806d270 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6f241456d0 tx=0x7f6f20009040 comp rx=0 tx=0).stop 2024-04-06T12:32:02.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.669+0000 7f6f2b816700 1 --2- 172.21.15.189:0/595855161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f240cf900 0x7f6f240a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:02.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.669+0000 7f6f2b816700 1 -- 172.21.15.189:0/595855161 >> 172.21.15.189:0/595855161 conn(0x7f6f240d00b0 msgr2=0x7f6f2406aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:02.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.669+0000 7f6f2b816700 1 -- 172.21.15.189:0/595855161 shutdown_connections 2024-04-06T12:32:02.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.670+0000 7f6f2b816700 1 -- 172.21.15.189:0/595855161 wait complete. 2024-04-06T12:32:02.671 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' created 2024-04-06T12:32:02.686 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-06T12:32:02.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.815+0000 7fba0417a700 1 Processor -- start 2024-04-06T12:32:02.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.816+0000 7fba0417a700 1 -- start start 2024-04-06T12:32:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.816+0000 7fba0417a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb9fc15a550 0x7fb9fc0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.816+0000 7fba0417a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9fc0b4d70 0x7fb9fc0b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.816+0000 7fba0417a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9fc0cdbe0 con 0x7fb9fc0cf900 2024-04-06T12:32:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba0417a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9fc15a940 con 0x7fb9fc15a550 2024-04-06T12:32:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba0417a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc15ac00 con 0x7fb9fc0b4d70 2024-04-06T12:32:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba01715700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb9fc0cf900 0x7fb9fc0b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43936/0 (socket says 172.21.15.189:43936) 2024-04-06T12:32:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba01715700 1 -- 172.21.15.189:0/3070447568 learned_addr learned my addr 172.21.15.189:0/3070447568 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:32:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba01f16700 1 --2- 172.21.15.189:0/3070447568 >> v2:172.21.15.192:3300/0 conn(0x7fb9fc15a550 0x7fb9fc0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba02717700 1 --2- 172.21.15.189:0/3070447568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9fc0b4d70 0x7fb9fc0b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba00f14700 1 -- 172.21.15.189:0/3070447568 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948818654 0 0) 0x7fb9fc0cdbe0 con 0x7fb9fc0cf900 2024-04-06T12:32:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba00f14700 1 -- 172.21.15.189:0/3070447568 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9e80035b0 con 0x7fb9fc0cf900 2024-04-06T12:32:02.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba00f14700 1 -- 172.21.15.189:0/3070447568 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1834616167 0 0) 0x7fb9e80035b0 con 0x7fb9fc0cf900 2024-04-06T12:32:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba00f14700 1 -- 172.21.15.189:0/3070447568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9fc0b4d70 msgr2=0x7fb9fc0b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba00f14700 1 --2- 172.21.15.189:0/3070447568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9fc0b4d70 0x7fb9fc0b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba00f14700 1 -- 172.21.15.189:0/3070447568 >> v2:172.21.15.192:3300/0 conn(0x7fb9fc15a550 msgr2=0x7fb9fc0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba00f14700 1 --2- 172.21.15.189:0/3070447568 >> v2:172.21.15.192:3300/0 conn(0x7fb9fc15a550 0x7fb9fc0cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba00f14700 1 -- 172.21.15.189:0/3070447568 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9fc15aee0 con 0x7fb9fc0cf900 2024-04-06T12:32:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba00f14700 1 -- 172.21.15.189:0/3070447568 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fb9f0001670 con 0x7fb9fc0cf900 2024-04-06T12:32:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba02717700 1 --2- 172.21.15.189:0/3070447568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9fc0b4d70 0x7fb9fc0b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:32:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba01f16700 1 --2- 172.21.15.189:0/3070447568 >> v2:172.21.15.192:3300/0 conn(0x7fb9fc15a550 0x7fb9fc0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:32:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba00f14700 1 -- 172.21.15.189:0/3070447568 >> v1:172.21.15.189:6789/0 conn(0x7fb9fc0cf900 legacy=0x7fb9fc0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba00f14700 1 --2- 172.21.15.189:0/3070447568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9fc0b4d70 0x7fb9e8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba00f14700 1 --2- 172.21.15.189:0/3070447568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9e8003f40 0x7fb9e8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba00f14700 1 --2- 172.21.15.189:0/3070447568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9e8004d90 0x7fb9e8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba00f14700 1 -- 172.21.15.189:0/3070447568 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc15a940 con 0x7fb9e8004d90 2024-04-06T12:32:02.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba00f14700 1 -- 172.21.15.189:0/3070447568 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc15ac00 con 0x7fb9e8003f40 2024-04-06T12:32:02.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba00f14700 1 -- 172.21.15.189:0/3070447568 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e800ba10 con 0x7fb9fc0b4d70 2024-04-06T12:32:02.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba02717700 1 --2- 172.21.15.189:0/3070447568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9e8004d90 0x7fb9e8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:02.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba01715700 1 --2- 172.21.15.189:0/3070447568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9e8003f40 0x7fb9e8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:02.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.817+0000 7fba01f16700 1 --2- 172.21.15.189:0/3070447568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9fc0b4d70 0x7fb9e8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:02.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.818+0000 7fba02717700 1 -- 172.21.15.189:0/3070447568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9fc0b4d70 msgr2=0x7fb9e8003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:02.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.818+0000 7fba02717700 1 --2- 172.21.15.189:0/3070447568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9fc0b4d70 0x7fb9e8003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:02.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.818+0000 7fba02717700 1 -- 172.21.15.189:0/3070447568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9e8003f40 msgr2=0x7fb9e8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:02.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.818+0000 7fba02717700 1 --2- 172.21.15.189:0/3070447568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9e8003f40 0x7fb9e8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:02.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.818+0000 7fba02717700 1 -- 172.21.15.189:0/3070447568 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f4009a70 con 0x7fb9e8004d90 2024-04-06T12:32:02.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.818+0000 7fba02717700 1 --2- 172.21.15.189:0/3070447568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9e8004d90 0x7fb9e8005210 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400e320 tx=0x7fb9e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e606f8720f501ab server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:02.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.818+0000 7fba00f14700 1 -- 172.21.15.189:0/3070447568 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f400ed60 con 0x7fb9e8004d90 2024-04-06T12:32:02.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.818+0000 7fba00f14700 1 -- 172.21.15.189:0/3070447568 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f400f040 con 0x7fb9e8004d90 2024-04-06T12:32:02.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.818+0000 7fba00f14700 1 -- 172.21.15.189:0/3070447568 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4003c40 con 0x7fb9e8004d90 2024-04-06T12:32:02.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.818+0000 7fba0417a700 1 -- 172.21.15.189:0/3070447568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9e8004d90 msgr2=0x7fb9e8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:02.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.818+0000 7fba0417a700 1 --2- 172.21.15.189:0/3070447568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9e8004d90 0x7fb9e8005210 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400e320 tx=0x7fb9e8005880 comp rx=0 tx=0).stop 2024-04-06T12:32:02.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.818+0000 7fba02717700 1 -- 172.21.15.189:0/3070447568 reap_dead start 2024-04-06T12:32:02.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.818+0000 7fba0417a700 1 -- 172.21.15.189:0/3070447568 shutdown_connections 2024-04-06T12:32:02.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.818+0000 7fba0417a700 1 -- 172.21.15.189:0/3070447568 >> 172.21.15.189:0/3070447568 conn(0x7fb9fc0d00b0 msgr2=0x7fb9fc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:02.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.819+0000 7fba0417a700 1 -- 172.21.15.189:0/3070447568 shutdown_connections 2024-04-06T12:32:02.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.819+0000 7fba0417a700 1 -- 172.21.15.189:0/3070447568 wait complete. 2024-04-06T12:32:02.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.820+0000 7fba0417a700 1 Processor -- start 2024-04-06T12:32:02.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.820+0000 7fba0417a700 1 -- start start 2024-04-06T12:32:02.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.820+0000 7fba0417a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9e8004d90 0x7fb9fc1f1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:02.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.820+0000 7fba0417a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9fc152f10 0x7fb9fc06acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:02.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.820+0000 7fba0417a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9fc06b210 0x7fb9fc154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:02.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.820+0000 7fba0417a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc1f1790 con 0x7fb9e8004d90 2024-04-06T12:32:02.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.820+0000 7fba0417a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc1f1e80 con 0x7fb9fc152f10 2024-04-06T12:32:02.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.820+0000 7fba0417a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc06b6a0 con 0x7fb9fc06b210 2024-04-06T12:32:02.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.821+0000 7fba01f16700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9e8004d90 0x7fb9fc1f1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:02.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.821+0000 7fba01f16700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9e8004d90 0x7fb9fc1f1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60634/0 (socket says 172.21.15.189:60634) 2024-04-06T12:32:02.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.821+0000 7fba01f16700 1 -- 172.21.15.189:0/693898541 learned_addr learned my addr 172.21.15.189:0/693898541 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:02.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.821+0000 7fba01715700 1 --2- 172.21.15.189:0/693898541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9fc152f10 0x7fb9fc06acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:02.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.821+0000 7fba02717700 1 --2- 172.21.15.189:0/693898541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9fc06b210 0x7fb9fc154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:02.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.821+0000 7fba01f16700 1 -- 172.21.15.189:0/693898541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9fc06b210 msgr2=0x7fb9fc154d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:02.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.821+0000 7fba01f16700 1 --2- 172.21.15.189:0/693898541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9fc06b210 0x7fb9fc154d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:02.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.821+0000 7fba01f16700 1 -- 172.21.15.189:0/693898541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9fc152f10 msgr2=0x7fb9fc06acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:02.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.821+0000 7fba01f16700 1 --2- 172.21.15.189:0/693898541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9fc152f10 0x7fb9fc06acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:02.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.821+0000 7fba01f16700 1 -- 172.21.15.189:0/693898541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9fc1f0b10 con 0x7fb9e8004d90 2024-04-06T12:32:02.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.821+0000 7fba01f16700 1 --2- 172.21.15.189:0/693898541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9e8004d90 0x7fb9fc1f1940 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8011c20 tx=0x7fb9f8011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:02.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.821+0000 7fb9ee7fc700 1 -- 172.21.15.189:0/693898541 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f800fa00 con 0x7fb9e8004d90 2024-04-06T12:32:02.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.821+0000 7fb9ee7fc700 1 -- 172.21.15.189:0/693898541 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f800fb60 con 0x7fb9e8004d90 2024-04-06T12:32:02.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.821+0000 7fba0417a700 1 -- 172.21.15.189:0/693898541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9fc1f0df0 con 0x7fb9e8004d90 2024-04-06T12:32:02.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.821+0000 7fb9ee7fc700 1 -- 172.21.15.189:0/693898541 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8004020 con 0x7fb9e8004d90 2024-04-06T12:32:02.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.821+0000 7fba0417a700 1 -- 172.21.15.189:0/693898541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9fc1f13b0 con 0x7fb9e8004d90 2024-04-06T12:32:02.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.822+0000 7fba0417a700 1 -- 172.21.15.189:0/693898541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9fc067bb0 con 0x7fb9e8004d90 2024-04-06T12:32:02.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.823+0000 7fb9ee7fc700 1 -- 172.21.15.189:0/693898541 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fb9f8004180 con 0x7fb9e8004d90 2024-04-06T12:32:02.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.823+0000 7fb9ee7fc700 1 --2- 172.21.15.189:0/693898541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb9e006adb0 0x7fb9e006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:02.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.823+0000 7fb9ee7fc700 1 -- 172.21.15.189:0/693898541 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5929+0+0 (secure 0 0 0) 0x7fb9f8089f10 con 0x7fb9e8004d90 2024-04-06T12:32:02.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.823+0000 7fba01715700 1 --2- 172.21.15.189:0/693898541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb9e006adb0 0x7fb9e006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:02.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.824+0000 7fba01715700 1 --2- 172.21.15.189:0/693898541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb9e006adb0 0x7fb9e006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00015d0 tx=0x7fb9f0007980 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:02.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:02.827+0000 7fb9ee7fc700 1 -- 172.21.15.189:0/693898541 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb9f8054f70 con 0x7fb9e8004d90 2024-04-06T12:32:03.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.066+0000 7fba0417a700 1 -- 172.21.15.189:0/693898541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb9fc074990 con 0x7fb9e8004d90 2024-04-06T12:32:03.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.073+0000 7fb9ee7fc700 1 -- 172.21.15.189:0/693898541 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 19 and data pool 20 v66) v1 ==== 174+0+0 (secure 0 0 0) 0x7fb9f8022070 con 0x7fb9e8004d90 2024-04-06T12:32:03.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.079+0000 7fba0417a700 1 -- 172.21.15.189:0/693898541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb9e006adb0 msgr2=0x7fb9e006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:03.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.079+0000 7fba0417a700 1 --2- 172.21.15.189:0/693898541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb9e006adb0 0x7fb9e006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00015d0 tx=0x7fb9f0007980 comp rx=0 tx=0).stop 2024-04-06T12:32:03.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.079+0000 7fba0417a700 1 -- 172.21.15.189:0/693898541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9e8004d90 msgr2=0x7fb9fc1f1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:03.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.079+0000 7fba0417a700 1 --2- 172.21.15.189:0/693898541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9e8004d90 0x7fb9fc1f1940 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8011c20 tx=0x7fb9f8011f30 comp rx=0 tx=0).stop 2024-04-06T12:32:03.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.080+0000 7fba0417a700 1 -- 172.21.15.189:0/693898541 shutdown_connections 2024-04-06T12:32:03.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.080+0000 7fba0417a700 1 --2- 172.21.15.189:0/693898541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9fc06b210 0x7fb9fc154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.080+0000 7fba0417a700 1 --2- 172.21.15.189:0/693898541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9e8004d90 0x7fb9fc1f1940 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.080+0000 7fba0417a700 1 --2- 172.21.15.189:0/693898541 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb9e006adb0 0x7fb9e006d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.080+0000 7fba0417a700 1 --2- 172.21.15.189:0/693898541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9fc152f10 0x7fb9fc06acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.080+0000 7fba0417a700 1 -- 172.21.15.189:0/693898541 >> 172.21.15.189:0/693898541 conn(0x7fb9fc0d00b0 msgr2=0x7fb9fc0cdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:03.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.085+0000 7fba0417a700 1 -- 172.21.15.189:0/693898541 shutdown_connections 2024-04-06T12:32:03.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.085+0000 7fba0417a700 1 -- 172.21.15.189:0/693898541 wait complete. 2024-04-06T12:32:03.084 INFO:teuthology.orchestra.run.smithi189.stderr:new fs with metadata pool 19 and data pool 20 2024-04-06T12:32:03.102 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:32:03.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.227+0000 7f81bdd71700 1 Processor -- start 2024-04-06T12:32:03.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.227+0000 7f81bdd71700 1 -- start start 2024-04-06T12:32:03.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.228+0000 7f81bdd71700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81b806ae40 0x7f81b815ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.228+0000 7f81bdd71700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f81b815fcd0 0x7f81b815bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.228+0000 7f81bdd71700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b815c4c0 con 0x7f81b8144f40 2024-04-06T12:32:03.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.228+0000 7f81bdd71700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b80cfcb0 con 0x7f81b815fcd0 2024-04-06T12:32:03.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.228+0000 7f81bdd71700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b815ca20 con 0x7f81b806ae40 2024-04-06T12:32:03.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.228+0000 7f81b7fff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f81b815fcd0 0x7f81b815bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.228+0000 7f81b6ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81b806ae40 0x7f81b815ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.228+0000 7f81b77fe700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f81b8144f40 0x7f81b806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:37136/0 (socket says 172.21.15.189:37136) 2024-04-06T12:32:03.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.228+0000 7f81b77fe700 1 -- 172.21.15.189:0/2414739432 learned_addr learned my addr 172.21.15.189:0/2414739432 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:32:03.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.228+0000 7f81b67fc700 1 -- 172.21.15.189:0/2414739432 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208444673 0 0) 0x7f81b815c4c0 con 0x7f81b8144f40 2024-04-06T12:32:03.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.229+0000 7f81b67fc700 1 -- 172.21.15.189:0/2414739432 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a40035b0 con 0x7f81b8144f40 2024-04-06T12:32:03.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.229+0000 7f81b6ffd700 1 -- 172.21.15.189:0/2414739432 >> v2:172.21.15.192:3300/0 conn(0x7f81b815fcd0 msgr2=0x7f81b815bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:03.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.229+0000 7f81b6ffd700 1 --2- 172.21.15.189:0/2414739432 >> v2:172.21.15.192:3300/0 conn(0x7f81b815fcd0 0x7f81b815bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.229+0000 7f81b6ffd700 1 -- 172.21.15.189:0/2414739432 >> v1:172.21.15.192:6789/0 conn(0x7f81b8144f40 legacy=0x7f81b806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:03.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.229+0000 7f81b6ffd700 1 -- 172.21.15.189:0/2414739432 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b815c740 con 0x7f81b806ae40 2024-04-06T12:32:03.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.229+0000 7f81b7fff700 1 --2- 172.21.15.189:0/2414739432 >> v2:172.21.15.192:3300/0 conn(0x7f81b815fcd0 0x7f81b815bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:32:03.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.229+0000 7f81b6ffd700 1 --2- 172.21.15.189:0/2414739432 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81b806ae40 0x7f81b815ba40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f81a0009840 tx=0x7f81a0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca20461071321217 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:03.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.229+0000 7f81b67fc700 1 -- 172.21.15.189:0/2414739432 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a0011830 con 0x7f81b806ae40 2024-04-06T12:32:03.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.229+0000 7f81b67fc700 1 -- 172.21.15.189:0/2414739432 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a0011990 con 0x7f81b806ae40 2024-04-06T12:32:03.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.229+0000 7f81b67fc700 1 -- 172.21.15.189:0/2414739432 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a0011d20 con 0x7f81b806ae40 2024-04-06T12:32:03.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.230+0000 7f81bdd71700 1 -- 172.21.15.189:0/2414739432 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81b806ae40 msgr2=0x7f81b815ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:03.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.230+0000 7f81bdd71700 1 --2- 172.21.15.189:0/2414739432 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81b806ae40 0x7f81b815ba40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f81a0009840 tx=0x7f81a0009b50 comp rx=0 tx=0).stop 2024-04-06T12:32:03.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.230+0000 7f81bdd71700 1 -- 172.21.15.189:0/2414739432 shutdown_connections 2024-04-06T12:32:03.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.230+0000 7f81bdd71700 1 --2- 172.21.15.189:0/2414739432 >> v2:172.21.15.192:3300/0 conn(0x7f81b815fcd0 0x7f81b815bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.230+0000 7f81bdd71700 1 --2- 172.21.15.189:0/2414739432 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81b806ae40 0x7f81b815ba40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.230+0000 7f81bdd71700 1 -- 172.21.15.189:0/2414739432 >> 172.21.15.189:0/2414739432 conn(0x7f81b80d00b0 msgr2=0x7f81b80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:03.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.230+0000 7f81bdd71700 1 -- 172.21.15.189:0/2414739432 shutdown_connections 2024-04-06T12:32:03.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.230+0000 7f81bdd71700 1 -- 172.21.15.189:0/2414739432 wait complete. 2024-04-06T12:32:03.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.231+0000 7f81bdd71700 1 Processor -- start 2024-04-06T12:32:03.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.231+0000 7f81bdd71700 1 -- start start 2024-04-06T12:32:03.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81bdd71700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81b806ae40 0x7f81b81f4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81bdd71700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81b8144f40 0x7f81b80c5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81bdd71700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81b815fcd0 0x7f81b80c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81bdd71700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b80c6550 con 0x7f81b815fcd0 2024-04-06T12:32:03.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81bdd71700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b80ca5d0 con 0x7f81b806ae40 2024-04-06T12:32:03.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81bdd71700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b80c6190 con 0x7f81b8144f40 2024-04-06T12:32:03.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81b7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81b815fcd0 0x7f81b80c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81b7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81b815fcd0 0x7f81b80c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60650/0 (socket says 172.21.15.189:60650) 2024-04-06T12:32:03.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81b7fff700 1 -- 172.21.15.189:0/2575545217 learned_addr learned my addr 172.21.15.189:0/2575545217 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:03.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81b6ffd700 1 --2- 172.21.15.189:0/2575545217 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81b8144f40 0x7f81b80c5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81b77fe700 1 --2- 172.21.15.189:0/2575545217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81b806ae40 0x7f81b81f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81b7fff700 1 -- 172.21.15.189:0/2575545217 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81b8144f40 msgr2=0x7f81b80c5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:03.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81b7fff700 1 --2- 172.21.15.189:0/2575545217 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81b8144f40 0x7f81b80c5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81b7fff700 1 -- 172.21.15.189:0/2575545217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81b806ae40 msgr2=0x7f81b81f4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:03.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81b7fff700 1 --2- 172.21.15.189:0/2575545217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81b806ae40 0x7f81b81f4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81b7fff700 1 -- 172.21.15.189:0/2575545217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a8009a70 con 0x7f81b815fcd0 2024-04-06T12:32:03.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81b7fff700 1 --2- 172.21.15.189:0/2575545217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81b815fcd0 0x7f81b80c8c10 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f81a800efb0 tx=0x7f81a800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:03.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81affff700 1 -- 172.21.15.189:0/2575545217 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a800cc30 con 0x7f81b815fcd0 2024-04-06T12:32:03.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81affff700 1 -- 172.21.15.189:0/2575545217 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a800f040 con 0x7f81b815fcd0 2024-04-06T12:32:03.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81affff700 1 -- 172.21.15.189:0/2575545217 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a8014480 con 0x7f81b815fcd0 2024-04-06T12:32:03.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.232+0000 7f81bdd71700 1 -- 172.21.15.189:0/2575545217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a00094a0 con 0x7f81b815fcd0 2024-04-06T12:32:03.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.233+0000 7f81bdd71700 1 -- 172.21.15.189:0/2575545217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b81f4fa0 con 0x7f81b815fcd0 2024-04-06T12:32:03.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.233+0000 7f81bdd71700 1 -- 172.21.15.189:0/2575545217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b8067bb0 con 0x7f81b815fcd0 2024-04-06T12:32:03.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.234+0000 7f81affff700 1 -- 172.21.15.189:0/2575545217 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f81a80145e0 con 0x7f81b815fcd0 2024-04-06T12:32:03.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.234+0000 7f81affff700 1 --2- 172.21.15.189:0/2575545217 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f819806adb0 0x7f819806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.234+0000 7f81affff700 1 -- 172.21.15.189:0/2575545217 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f81a808a240 con 0x7f81b815fcd0 2024-04-06T12:32:03.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.234+0000 7f81b77fe700 1 --2- 172.21.15.189:0/2575545217 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f819806adb0 0x7f819806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.235+0000 7f81b77fe700 1 --2- 172.21.15.189:0/2575545217 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f819806adb0 0x7f819806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f81b0007fd0 tx=0x7f81b0007dc0 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:03.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.239+0000 7f81affff700 1 -- 172.21.15.189:0/2575545217 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f81a80550c0 con 0x7f81b815fcd0 2024-04-06T12:32:03.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.438+0000 7f81bdd71700 1 -- 172.21.15.189:0/2575545217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81b81f4760 con 0x7f81b815fcd0 2024-04-06T12:32:03.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.439+0000 7f81affff700 1 -- 172.21.15.189:0/2575545217 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) v1 ==== 74+0+16325 (secure 0 0 0) 0x7f81a8009cb0 con 0x7f81b815fcd0 2024-04-06T12:32:03.436 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:32:03.436 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":157,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:32:03.067465+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:32:00.826039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-04-06T12:32:02.039801+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:32:03.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.441+0000 7f81bdd71700 1 -- 172.21.15.189:0/2575545217 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f819806adb0 msgr2=0x7f819806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:03.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.441+0000 7f81bdd71700 1 --2- 172.21.15.189:0/2575545217 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f819806adb0 0x7f819806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f81b0007fd0 tx=0x7f81b0007dc0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.441+0000 7f81bdd71700 1 -- 172.21.15.189:0/2575545217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81b815fcd0 msgr2=0x7f81b80c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:03.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.441+0000 7f81bdd71700 1 --2- 172.21.15.189:0/2575545217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81b815fcd0 0x7f81b80c8c10 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f81a800efb0 tx=0x7f81a800c3a0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.441+0000 7f81bdd71700 1 -- 172.21.15.189:0/2575545217 shutdown_connections 2024-04-06T12:32:03.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.441+0000 7f81bdd71700 1 --2- 172.21.15.189:0/2575545217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81b815fcd0 0x7f81b80c8c10 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.441+0000 7f81bdd71700 1 --2- 172.21.15.189:0/2575545217 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f819806adb0 0x7f819806d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.441+0000 7f81bdd71700 1 --2- 172.21.15.189:0/2575545217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81b806ae40 0x7f81b81f4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.441+0000 7f81bdd71700 1 --2- 172.21.15.189:0/2575545217 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81b8144f40 0x7f81b80c5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.441+0000 7f81bdd71700 1 -- 172.21.15.189:0/2575545217 >> 172.21.15.189:0/2575545217 conn(0x7f81b80d00b0 msgr2=0x7f81b806f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:03.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.441+0000 7f81bdd71700 1 -- 172.21.15.189:0/2575545217 shutdown_connections 2024-04-06T12:32:03.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.441+0000 7f81bdd71700 1 -- 172.21.15.189:0/2575545217 wait complete. 2024-04-06T12:32:03.451 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:32:03.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.575+0000 7f8dbb884700 1 Processor -- start 2024-04-06T12:32:03.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.576+0000 7f8dbb884700 1 -- start start 2024-04-06T12:32:03.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.576+0000 7f8dbb884700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8db40d17a0 0x7f8db406b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.576+0000 7f8dbb884700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8db4146dd0 0x7f8db406d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.576+0000 7f8dbb884700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db406c0a0 con 0x7f8db40cedf0 2024-04-06T12:32:03.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.576+0000 7f8dbb884700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8db406c380 con 0x7f8db4146dd0 2024-04-06T12:32:03.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.576+0000 7f8dbb884700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db406c660 con 0x7f8db40d17a0 2024-04-06T12:32:03.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.576+0000 7f8db9e21700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8db40cedf0 0x7f8db41441f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43942/0 (socket says 172.21.15.189:43942) 2024-04-06T12:32:03.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8db9e21700 1 -- 172.21.15.189:0/3706562616 learned_addr learned my addr 172.21.15.189:0/3706562616 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:32:03.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8db8e1f700 1 --2- 172.21.15.189:0/3706562616 >> v2:172.21.15.192:3300/0 conn(0x7f8db4146dd0 0x7f8db406d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8db9620700 1 --2- 172.21.15.189:0/3706562616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8db40d17a0 0x7f8db406b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8da7fff700 1 -- 172.21.15.189:0/3706562616 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4739333 0 0) 0x7f8db406c0a0 con 0x7f8db40cedf0 2024-04-06T12:32:03.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8da7fff700 1 -- 172.21.15.189:0/3706562616 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8da00035b0 con 0x7f8db40cedf0 2024-04-06T12:32:03.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8da7fff700 1 -- 172.21.15.189:0/3706562616 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2825827930 0 0) 0x7f8da00035b0 con 0x7f8db40cedf0 2024-04-06T12:32:03.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8da7fff700 1 -- 172.21.15.189:0/3706562616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8db40d17a0 msgr2=0x7f8db406b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:03.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8da7fff700 1 --2- 172.21.15.189:0/3706562616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8db40d17a0 0x7f8db406b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8da7fff700 1 -- 172.21.15.189:0/3706562616 >> v2:172.21.15.192:3300/0 conn(0x7f8db4146dd0 msgr2=0x7f8db406d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:03.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8da7fff700 1 --2- 172.21.15.189:0/3706562616 >> v2:172.21.15.192:3300/0 conn(0x7f8db4146dd0 0x7f8db406d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8da7fff700 1 -- 172.21.15.189:0/3706562616 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db406c940 con 0x7f8db40cedf0 2024-04-06T12:32:03.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8da7fff700 1 -- 172.21.15.189:0/3706562616 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f8dac0019f0 con 0x7f8db40cedf0 2024-04-06T12:32:03.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8da7fff700 1 -- 172.21.15.189:0/3706562616 >> v1:172.21.15.189:6789/0 conn(0x7f8db40cedf0 legacy=0x7f8db41441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:03.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8da7fff700 1 --2- 172.21.15.189:0/3706562616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8da0003a00 0x7f8da0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8da7fff700 1 --2- 172.21.15.189:0/3706562616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8db40d17a0 0x7f8da0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8da7fff700 1 --2- 172.21.15.189:0/3706562616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8da0004d90 0x7f8da0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8da7fff700 1 -- 172.21.15.189:0/3706562616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db406c380 con 0x7f8da0003a00 2024-04-06T12:32:03.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8da7fff700 1 -- 172.21.15.189:0/3706562616 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db406c660 con 0x7f8da0004d90 2024-04-06T12:32:03.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8da7fff700 1 -- 172.21.15.189:0/3706562616 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da000ba10 con 0x7f8db40d17a0 2024-04-06T12:32:03.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8db9620700 1 --2- 172.21.15.189:0/3706562616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8da0003a00 0x7f8da0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8db8e1f700 1 --2- 172.21.15.189:0/3706562616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8db40d17a0 0x7f8da0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.577+0000 7f8db9e21700 1 --2- 172.21.15.189:0/3706562616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8da0004d90 0x7f8da0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.578+0000 7f8db9620700 1 -- 172.21.15.189:0/3706562616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8db40d17a0 msgr2=0x7f8da0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:03.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.578+0000 7f8db9620700 1 --2- 172.21.15.189:0/3706562616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8db40d17a0 0x7f8da0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.578+0000 7f8db9620700 1 -- 172.21.15.189:0/3706562616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8da0004d90 msgr2=0x7f8da0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:03.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.578+0000 7f8db9620700 1 --2- 172.21.15.189:0/3706562616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8da0004d90 0x7f8da0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.578+0000 7f8db9620700 1 -- 172.21.15.189:0/3706562616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da00035b0 con 0x7f8da0003a00 2024-04-06T12:32:03.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.578+0000 7f8db9620700 1 --2- 172.21.15.189:0/3706562616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8da0003a00 0x7f8da0003e20 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f8da8009b60 tx=0x7f8da0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a89e5d97c84767 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:03.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.578+0000 7f8da7fff700 1 -- 172.21.15.189:0/3706562616 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da800e6c0 con 0x7f8da0003a00 2024-04-06T12:32:03.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.578+0000 7f8da7fff700 1 -- 172.21.15.189:0/3706562616 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da800e820 con 0x7f8da0003a00 2024-04-06T12:32:03.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.578+0000 7f8da7fff700 1 -- 172.21.15.189:0/3706562616 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da800eb90 con 0x7f8da0003a00 2024-04-06T12:32:03.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.578+0000 7f8dbb884700 1 -- 172.21.15.189:0/3706562616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8da0003a00 msgr2=0x7f8da0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:03.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.578+0000 7f8dbb884700 1 --2- 172.21.15.189:0/3706562616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8da0003a00 0x7f8da0003e20 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f8da8009b60 tx=0x7f8da0005880 comp rx=0 tx=0).stop 2024-04-06T12:32:03.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.578+0000 7f8db9e21700 1 -- 172.21.15.189:0/3706562616 reap_dead start 2024-04-06T12:32:03.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.578+0000 7f8dbb884700 1 -- 172.21.15.189:0/3706562616 shutdown_connections 2024-04-06T12:32:03.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.578+0000 7f8dbb884700 1 -- 172.21.15.189:0/3706562616 >> 172.21.15.189:0/3706562616 conn(0x7f8db40b2db0 msgr2=0x7f8db40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:03.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.579+0000 7f8dbb884700 1 -- 172.21.15.189:0/3706562616 shutdown_connections 2024-04-06T12:32:03.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.579+0000 7f8dbb884700 1 -- 172.21.15.189:0/3706562616 wait complete. 2024-04-06T12:32:03.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.580+0000 7f8dbb884700 1 Processor -- start 2024-04-06T12:32:03.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.580+0000 7f8dbb884700 1 -- start start 2024-04-06T12:32:03.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.581+0000 7f8dbb884700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8da0003a00 0x7f8db415af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.581+0000 7f8dbb884700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8db40afdc0 0x7f8db40b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.581+0000 7f8dbb884700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8db40c67b0 0x7f8db40c6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.581+0000 7f8dbb884700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db40c9470 con 0x7f8da0003a00 2024-04-06T12:32:03.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.581+0000 7f8dbb884700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db40c7780 con 0x7f8db40c67b0 2024-04-06T12:32:03.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.581+0000 7f8dbb884700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db40b0780 con 0x7f8db40afdc0 2024-04-06T12:32:03.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.581+0000 7f8db9620700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8da0003a00 0x7f8db415af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.581+0000 7f8db9620700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8da0003a00 0x7f8db415af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60672/0 (socket says 172.21.15.189:60672) 2024-04-06T12:32:03.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.581+0000 7f8db9620700 1 -- 172.21.15.189:0/3075044664 learned_addr learned my addr 172.21.15.189:0/3075044664 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:03.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.581+0000 7f8db9620700 1 -- 172.21.15.189:0/3075044664 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8db40afdc0 msgr2=0x7f8db40b0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:03.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.581+0000 7f8db9e21700 1 --2- 172.21.15.189:0/3075044664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8db40c67b0 0x7f8db40c6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.581+0000 7f8db9620700 1 --2- 172.21.15.189:0/3075044664 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8db40afdc0 0x7f8db40b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.581+0000 7f8db9620700 1 -- 172.21.15.189:0/3075044664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8db40c67b0 msgr2=0x7f8db40c6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:03.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.581+0000 7f8db9620700 1 --2- 172.21.15.189:0/3075044664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8db40c67b0 0x7f8db40c6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.581+0000 7f8db9620700 1 -- 172.21.15.189:0/3075044664 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db40c74a0 con 0x7f8da0003a00 2024-04-06T12:32:03.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.581+0000 7f8db9e21700 1 --2- 172.21.15.189:0/3075044664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8db40c67b0 0x7f8db40c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:32:03.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.582+0000 7f8db9620700 1 --2- 172.21.15.189:0/3075044664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8da0003a00 0x7f8db415af90 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f8da00046e0 tx=0x7f8da80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:03.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.582+0000 7f8da5ffb700 1 -- 172.21.15.189:0/3075044664 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da8021070 con 0x7f8da0003a00 2024-04-06T12:32:03.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.582+0000 7f8da5ffb700 1 -- 172.21.15.189:0/3075044664 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da80045d0 con 0x7f8da0003a00 2024-04-06T12:32:03.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.582+0000 7f8dbb884700 1 -- 172.21.15.189:0/3075044664 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db41f3ed0 con 0x7f8da0003a00 2024-04-06T12:32:03.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.582+0000 7f8da5ffb700 1 -- 172.21.15.189:0/3075044664 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da800e6c0 con 0x7f8da0003a00 2024-04-06T12:32:03.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.582+0000 7f8dbb884700 1 -- 172.21.15.189:0/3075044664 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db41f4490 con 0x7f8da0003a00 2024-04-06T12:32:03.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.583+0000 7f8dbb884700 1 -- 172.21.15.189:0/3075044664 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8db41f3d30 con 0x7f8da0003a00 2024-04-06T12:32:03.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.583+0000 7f8da5ffb700 1 -- 172.21.15.189:0/3075044664 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f8da8025070 con 0x7f8da0003a00 2024-04-06T12:32:03.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.584+0000 7f8da5ffb700 1 --2- 172.21.15.189:0/3075044664 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8d9806adb0 0x7f8d9806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.584+0000 7f8da5ffb700 1 -- 172.21.15.189:0/3075044664 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8da808a840 con 0x7f8da0003a00 2024-04-06T12:32:03.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.584+0000 7f8db8e1f700 1 --2- 172.21.15.189:0/3075044664 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8d9806adb0 0x7f8d9806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.584+0000 7f8db8e1f700 1 --2- 172.21.15.189:0/3075044664 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8d9806adb0 0x7f8d9806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8db0006f10 tx=0x7f8db0006e80 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:03.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.589+0000 7f8da5ffb700 1 -- 172.21.15.189:0/3075044664 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8db41f3d30 con 0x7f8da0003a00 2024-04-06T12:32:03.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.788+0000 7f8dbb884700 1 -- 172.21.15.189:0/3075044664 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8db41f3d30 con 0x7f8da0003a00 2024-04-06T12:32:03.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.789+0000 7f8da5ffb700 1 -- 172.21.15.189:0/3075044664 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) v1 ==== 74+0+16325 (secure 0 0 0) 0x7f8db41f3d30 con 0x7f8da0003a00 2024-04-06T12:32:03.786 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:32:03.786 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":157,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:32:03.067465+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:32:00.826039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-04-06T12:32:02.039801+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:32:03.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.791+0000 7f8dbb884700 1 -- 172.21.15.189:0/3075044664 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8d9806adb0 msgr2=0x7f8d9806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:03.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.791+0000 7f8dbb884700 1 --2- 172.21.15.189:0/3075044664 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8d9806adb0 0x7f8d9806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8db0006f10 tx=0x7f8db0006e80 comp rx=0 tx=0).stop 2024-04-06T12:32:03.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.791+0000 7f8dbb884700 1 -- 172.21.15.189:0/3075044664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8da0003a00 msgr2=0x7f8db415af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:03.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.791+0000 7f8dbb884700 1 --2- 172.21.15.189:0/3075044664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8da0003a00 0x7f8db415af90 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f8da00046e0 tx=0x7f8da80047c0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.791+0000 7f8dbb884700 1 -- 172.21.15.189:0/3075044664 shutdown_connections 2024-04-06T12:32:03.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.791+0000 7f8dbb884700 1 --2- 172.21.15.189:0/3075044664 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8d9806adb0 0x7f8d9806d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.791+0000 7f8dbb884700 1 --2- 172.21.15.189:0/3075044664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8db40c67b0 0x7f8db40c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.791+0000 7f8dbb884700 1 --2- 172.21.15.189:0/3075044664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8da0003a00 0x7f8db415af90 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.791+0000 7f8dbb884700 1 --2- 172.21.15.189:0/3075044664 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8db40afdc0 0x7f8db40b0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.791+0000 7f8dbb884700 1 -- 172.21.15.189:0/3075044664 >> 172.21.15.189:0/3075044664 conn(0x7f8db40b2db0 msgr2=0x7f8db415e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:03.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.791+0000 7f8dbb884700 1 -- 172.21.15.189:0/3075044664 shutdown_connections 2024-04-06T12:32:03.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.791+0000 7f8dbb884700 1 -- 172.21.15.189:0/3075044664 wait complete. 2024-04-06T12:32:03.801 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-06T12:32:03.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.928+0000 7f029b44a700 1 Processor -- start 2024-04-06T12:32:03.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.929+0000 7f029b44a700 1 -- start start 2024-04-06T12:32:03.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.929+0000 7f029b44a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0294158fe0 0x7f029406a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.929+0000 7f029b44a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f029406ac70 0x7f029415bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.930+0000 7f029b44a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029415f9e0 con 0x7f029415c100 2024-04-06T12:32:03.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.930+0000 7f029b44a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f029406b0f0 con 0x7f0294158fe0 2024-04-06T12:32:03.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.930+0000 7f029b44a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f029415f0d0 con 0x7f029406ac70 2024-04-06T12:32:03.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.930+0000 7f02991e6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0294158fe0 0x7f029406a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.930+0000 7f02991e6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0294158fe0 0x7f029406a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60680/0 (socket says 172.21.15.189:60680) 2024-04-06T12:32:03.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.930+0000 7f02991e6700 1 -- 172.21.15.189:0/3725229719 learned_addr learned my addr 172.21.15.189:0/3725229719 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:03.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.930+0000 7f02989e5700 1 --2- 172.21.15.189:0/3725229719 >> v2:172.21.15.192:3300/0 conn(0x7f029406ac70 0x7f029415bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.930+0000 7f02991e6700 1 -- 172.21.15.189:0/3725229719 >> v2:172.21.15.192:3300/0 conn(0x7f029406ac70 msgr2=0x7f029415bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:03.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.930+0000 7f02991e6700 1 --2- 172.21.15.189:0/3725229719 >> v2:172.21.15.192:3300/0 conn(0x7f029406ac70 0x7f029415bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.930+0000 7f02991e6700 1 -- 172.21.15.189:0/3725229719 >> v1:172.21.15.192:6789/0 conn(0x7f029415c100 legacy=0x7f02940717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:03.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.930+0000 7f02991e6700 1 -- 172.21.15.189:0/3725229719 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029415c8f0 con 0x7f0294158fe0 2024-04-06T12:32:03.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.930+0000 7f02991e6700 1 --2- 172.21.15.189:0/3725229719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0294158fe0 0x7f029406a730 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f0294067b80 tx=0x7f0290009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c5e64b880fe774 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:03.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.930+0000 7f028bfff700 1 -- 172.21.15.189:0/3725229719 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0290004230 con 0x7f0294158fe0 2024-04-06T12:32:03.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.930+0000 7f028bfff700 1 -- 172.21.15.189:0/3725229719 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0290004390 con 0x7f0294158fe0 2024-04-06T12:32:03.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.930+0000 7f028bfff700 1 -- 172.21.15.189:0/3725229719 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02900036a0 con 0x7f0294158fe0 2024-04-06T12:32:03.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.931+0000 7f029b44a700 1 -- 172.21.15.189:0/3725229719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0294158fe0 msgr2=0x7f029406a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:03.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.931+0000 7f029b44a700 1 --2- 172.21.15.189:0/3725229719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0294158fe0 0x7f029406a730 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f0294067b80 tx=0x7f0290009bd0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.931+0000 7f029b44a700 1 -- 172.21.15.189:0/3725229719 shutdown_connections 2024-04-06T12:32:03.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.931+0000 7f029b44a700 1 --2- 172.21.15.189:0/3725229719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0294158fe0 0x7f029406a730 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.931+0000 7f029b44a700 1 --2- 172.21.15.189:0/3725229719 >> v2:172.21.15.192:3300/0 conn(0x7f029406ac70 0x7f029415bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.931+0000 7f029b44a700 1 -- 172.21.15.189:0/3725229719 >> 172.21.15.189:0/3725229719 conn(0x7f02940d00b0 msgr2=0x7f02940d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:03.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.931+0000 7f029b44a700 1 -- 172.21.15.189:0/3725229719 shutdown_connections 2024-04-06T12:32:03.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.931+0000 7f029b44a700 1 -- 172.21.15.189:0/3725229719 wait complete. 2024-04-06T12:32:03.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.932+0000 7f029b44a700 1 Processor -- start 2024-04-06T12:32:03.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.932+0000 7f029b44a700 1 -- start start 2024-04-06T12:32:03.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.933+0000 7f029b44a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f029406ac70 0x7f02940ae710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.933+0000 7f029b44a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0294158fe0 0x7f02940b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.933+0000 7f029b44a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f029415c100 0x7f0294135800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.933+0000 7f029b44a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02940af990 con 0x7f029415c100 2024-04-06T12:32:03.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.933+0000 7f029b44a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02940aed80 con 0x7f0294158fe0 2024-04-06T12:32:03.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.933+0000 7f029b44a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02940af060 con 0x7f029406ac70 2024-04-06T12:32:03.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.933+0000 7f02999e7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f029415c100 0x7f0294135800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.933+0000 7f02999e7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f029415c100 0x7f0294135800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60696/0 (socket says 172.21.15.189:60696) 2024-04-06T12:32:03.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.933+0000 7f02999e7700 1 -- 172.21.15.189:0/4264522291 learned_addr learned my addr 172.21.15.189:0/4264522291 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:03.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.933+0000 7f02989e5700 1 --2- 172.21.15.189:0/4264522291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0294158fe0 0x7f02940b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.933+0000 7f02991e6700 1 --2- 172.21.15.189:0/4264522291 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f029406ac70 0x7f02940ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.933+0000 7f02999e7700 1 -- 172.21.15.189:0/4264522291 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f029406ac70 msgr2=0x7f02940ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:03.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.933+0000 7f02999e7700 1 --2- 172.21.15.189:0/4264522291 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f029406ac70 0x7f02940ae710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.933+0000 7f02999e7700 1 -- 172.21.15.189:0/4264522291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0294158fe0 msgr2=0x7f02940b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:03.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.933+0000 7f02999e7700 1 --2- 172.21.15.189:0/4264522291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0294158fe0 0x7f02940b4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:03.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.933+0000 7f02999e7700 1 -- 172.21.15.189:0/4264522291 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0290009550 con 0x7f029415c100 2024-04-06T12:32:03.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.933+0000 7f02991e6700 1 --2- 172.21.15.189:0/4264522291 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f029406ac70 0x7f02940ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:32:03.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.933+0000 7f02999e7700 1 --2- 172.21.15.189:0/4264522291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f029415c100 0x7f0294135800 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f028c00cc10 tx=0x7f028c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:03.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.934+0000 7f0289ffb700 1 -- 172.21.15.189:0/4264522291 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c00ab10 con 0x7f029415c100 2024-04-06T12:32:03.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.934+0000 7f0289ffb700 1 -- 172.21.15.189:0/4264522291 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028c00ac70 con 0x7f029415c100 2024-04-06T12:32:03.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.934+0000 7f0289ffb700 1 -- 172.21.15.189:0/4264522291 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c012420 con 0x7f029415c100 2024-04-06T12:32:03.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.934+0000 7f029b44a700 1 -- 172.21.15.189:0/4264522291 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02940af340 con 0x7f029415c100 2024-04-06T12:32:03.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.934+0000 7f029b44a700 1 -- 172.21.15.189:0/4264522291 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02940b5050 con 0x7f029415c100 2024-04-06T12:32:03.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.935+0000 7f029b44a700 1 -- 172.21.15.189:0/4264522291 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0294067bb0 con 0x7f029415c100 2024-04-06T12:32:03.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.935+0000 7f0289ffb700 1 -- 172.21.15.189:0/4264522291 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f028c012580 con 0x7f029415c100 2024-04-06T12:32:03.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.935+0000 7f0289ffb700 1 --2- 172.21.15.189:0/4264522291 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f027806adb0 0x7f027806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:03.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.935+0000 7f0289ffb700 1 -- 172.21.15.189:0/4264522291 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f028c0898b0 con 0x7f029415c100 2024-04-06T12:32:03.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.936+0000 7f02991e6700 1 --2- 172.21.15.189:0/4264522291 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f027806adb0 0x7f027806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:03.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.936+0000 7f02991e6700 1 --2- 172.21.15.189:0/4264522291 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f027806adb0 0x7f027806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0290009130 tx=0x7f0290008030 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:03.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:03.940+0000 7f0289ffb700 1 -- 172.21.15.189:0/4264522291 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f028c0548d0 con 0x7f029415c100 2024-04-06T12:32:04.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:04.185+0000 7f029b44a700 1 -- 172.21.15.189:0/4264522291 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0294074990 con 0x7f029415c100 2024-04-06T12:32:05.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.082+0000 7f0289ffb700 1 -- 172.21.15.189:0/4264522291 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v69) v1 ==== 118+0+0 (secure 0 0 0) 0x7f028c057ef0 con 0x7f029415c100 2024-04-06T12:32:05.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.085+0000 7f029b44a700 1 -- 172.21.15.189:0/4264522291 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f027806adb0 msgr2=0x7f027806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:05.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.085+0000 7f029b44a700 1 --2- 172.21.15.189:0/4264522291 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f027806adb0 0x7f027806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0290009130 tx=0x7f0290008030 comp rx=0 tx=0).stop 2024-04-06T12:32:05.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.085+0000 7f029b44a700 1 -- 172.21.15.189:0/4264522291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f029415c100 msgr2=0x7f0294135800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:05.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.085+0000 7f029b44a700 1 --2- 172.21.15.189:0/4264522291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f029415c100 0x7f0294135800 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f028c00cc10 tx=0x7f028c00cf20 comp rx=0 tx=0).stop 2024-04-06T12:32:05.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.085+0000 7f029b44a700 1 -- 172.21.15.189:0/4264522291 shutdown_connections 2024-04-06T12:32:05.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.085+0000 7f029b44a700 1 --2- 172.21.15.189:0/4264522291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f029415c100 0x7f0294135800 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:05.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.085+0000 7f029b44a700 1 --2- 172.21.15.189:0/4264522291 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f029406ac70 0x7f02940ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:05.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.085+0000 7f029b44a700 1 --2- 172.21.15.189:0/4264522291 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f027806adb0 0x7f027806d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:05.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.085+0000 7f029b44a700 1 --2- 172.21.15.189:0/4264522291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0294158fe0 0x7f02940b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:05.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.085+0000 7f029b44a700 1 -- 172.21.15.189:0/4264522291 >> 172.21.15.189:0/4264522291 conn(0x7f02940d00b0 msgr2=0x7f02940ccac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:05.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.085+0000 7f029b44a700 1 -- 172.21.15.189:0/4264522291 shutdown_connections 2024-04-06T12:32:05.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.085+0000 7f029b44a700 1 -- 172.21.15.189:0/4264522291 wait complete. 2024-04-06T12:32:05.097 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-06T12:32:05.097 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-06T12:32:05.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.231+0000 7f9711c7a700 1 Processor -- start 2024-04-06T12:32:05.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.231+0000 7f9711c7a700 1 -- start start 2024-04-06T12:32:05.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.232+0000 7f9711c7a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f970c15e8a0 0x7f970c0cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:05.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.232+0000 7f9711c7a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970c0669d0 con 0x7f970c15eee0 2024-04-06T12:32:05.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.232+0000 7f9711c7a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970c06ede0 con 0x7f970c15fa70 2024-04-06T12:32:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.232+0000 7f9711c7a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f970c15ec80 con 0x7f970c15e8a0 2024-04-06T12:32:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.232+0000 7f970b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f970c15e8a0 0x7f970c0cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.232+0000 7f970affd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f970c15eee0 0x7f970c15f360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43958/0 (socket says 172.21.15.189:43958) 2024-04-06T12:32:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.232+0000 7f970affd700 1 -- 172.21.15.189:0/1771158733 learned_addr learned my addr 172.21.15.189:0/1771158733 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:32:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.232+0000 7f970b7fe700 1 -- 172.21.15.189:0/1771158733 >> v1:172.21.15.192:6789/0 conn(0x7f970c15fa70 legacy=0x7f970c06aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.232+0000 7f970b7fe700 1 -- 172.21.15.189:0/1771158733 >> v1:172.21.15.189:6789/0 conn(0x7f970c15eee0 legacy=0x7f970c15f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.232+0000 7f970b7fe700 1 -- 172.21.15.189:0/1771158733 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970c0666f0 con 0x7f970c15e8a0 2024-04-06T12:32:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.232+0000 7f970a7fc700 1 -- 172.21.15.189:0/1771158733 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1467959189 0 0) 0x7f970c0669d0 con 0x7f970c15eee0 2024-04-06T12:32:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.232+0000 7f970b7fe700 1 --2- 172.21.15.189:0/1771158733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f970c15e8a0 0x7f970c0cbc20 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f96fc009840 tx=0x7f96fc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49a3f35eaeef06f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.232+0000 7f970a7fc700 1 -- 172.21.15.189:0/1771158733 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc004230 con 0x7f970c15e8a0 2024-04-06T12:32:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.232+0000 7f970a7fc700 1 -- 172.21.15.189:0/1771158733 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96fc004390 con 0x7f970c15e8a0 2024-04-06T12:32:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.232+0000 7f970a7fc700 1 -- 172.21.15.189:0/1771158733 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc0036a0 con 0x7f970c15e8a0 2024-04-06T12:32:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.233+0000 7f9711c7a700 1 -- 172.21.15.189:0/1771158733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f970c15e8a0 msgr2=0x7f970c0cbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:05.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.233+0000 7f9711c7a700 1 --2- 172.21.15.189:0/1771158733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f970c15e8a0 0x7f970c0cbc20 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f96fc009840 tx=0x7f96fc009b50 comp rx=0 tx=0).stop 2024-04-06T12:32:05.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.233+0000 7f9711c7a700 1 -- 172.21.15.189:0/1771158733 shutdown_connections 2024-04-06T12:32:05.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.233+0000 7f9711c7a700 1 --2- 172.21.15.189:0/1771158733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f970c15e8a0 0x7f970c0cbc20 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:05.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.233+0000 7f9711c7a700 1 -- 172.21.15.189:0/1771158733 >> 172.21.15.189:0/1771158733 conn(0x7f970c0cc290 msgr2=0x7f970c0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:05.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.233+0000 7f9711c7a700 1 -- 172.21.15.189:0/1771158733 shutdown_connections 2024-04-06T12:32:05.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.233+0000 7f9711c7a700 1 -- 172.21.15.189:0/1771158733 wait complete. 2024-04-06T12:32:05.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.234+0000 7f9711c7a700 1 Processor -- start 2024-04-06T12:32:05.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.234+0000 7f9711c7a700 1 -- start start 2024-04-06T12:32:05.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.235+0000 7f9711c7a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f970c15e8a0 0x7f970c0cee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:05.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.235+0000 7f9711c7a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f970c15eee0 0x7f970c0cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:05.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.235+0000 7f9711c7a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f970c15fa70 0x7f970c0ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:05.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.235+0000 7f9711c7a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970c0c3170 con 0x7f970c15e8a0 2024-04-06T12:32:05.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.235+0000 7f9711c7a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970c0b5300 con 0x7f970c15eee0 2024-04-06T12:32:05.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.235+0000 7f9711c7a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970c0cf3d0 con 0x7f970c15fa70 2024-04-06T12:32:05.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.235+0000 7f970b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f970c15e8a0 0x7f970c0cee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:05.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.235+0000 7f970b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f970c15e8a0 0x7f970c0cee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60706/0 (socket says 172.21.15.189:60706) 2024-04-06T12:32:05.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.235+0000 7f970b7fe700 1 -- 172.21.15.189:0/1804306882 learned_addr learned my addr 172.21.15.189:0/1804306882 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:05.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.235+0000 7f970affd700 1 --2- 172.21.15.189:0/1804306882 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f970c15eee0 0x7f970c0cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:05.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.235+0000 7f970b7fe700 1 -- 172.21.15.189:0/1804306882 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f970c15fa70 msgr2=0x7f970c0ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:05.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.235+0000 7f970b7fe700 1 --2- 172.21.15.189:0/1804306882 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f970c15fa70 0x7f970c0ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:05.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.235+0000 7f970b7fe700 1 -- 172.21.15.189:0/1804306882 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f970c15eee0 msgr2=0x7f970c0cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.235+0000 7f970b7fe700 1 --2- 172.21.15.189:0/1804306882 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f970c15eee0 0x7f970c0cdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.235+0000 7f970b7fe700 1 -- 172.21.15.189:0/1804306882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96fc0094a0 con 0x7f970c15e8a0 2024-04-06T12:32:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.235+0000 7f970affd700 1 --2- 172.21.15.189:0/1804306882 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f970c15eee0 0x7f970c0cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:32:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.236+0000 7f970b7fe700 1 --2- 172.21.15.189:0/1804306882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f970c15e8a0 0x7f970c0cee90 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f970c073430 tx=0x7f96fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.236+0000 7f96f3fff700 1 -- 172.21.15.189:0/1804306882 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc011630 con 0x7f970c15e8a0 2024-04-06T12:32:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.236+0000 7f96f3fff700 1 -- 172.21.15.189:0/1804306882 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96fc01e070 con 0x7f970c15e8a0 2024-04-06T12:32:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.236+0000 7f9711c7a700 1 -- 172.21.15.189:0/1804306882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f970c0ad7e0 con 0x7f970c15e8a0 2024-04-06T12:32:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.236+0000 7f96f3fff700 1 -- 172.21.15.189:0/1804306882 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc011990 con 0x7f970c15e8a0 2024-04-06T12:32:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.236+0000 7f9711c7a700 1 -- 172.21.15.189:0/1804306882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f970c0adda0 con 0x7f970c15e8a0 2024-04-06T12:32:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.237+0000 7f9711c7a700 1 -- 172.21.15.189:0/1804306882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f970c072140 con 0x7f970c15e8a0 2024-04-06T12:32:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.237+0000 7f96f3fff700 1 -- 172.21.15.189:0/1804306882 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f96fc011af0 con 0x7f970c15e8a0 2024-04-06T12:32:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.237+0000 7f96f3fff700 1 --2- 172.21.15.189:0/1804306882 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f96f806adb0 0x7f96f806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.238+0000 7f96f3fff700 1 -- 172.21.15.189:0/1804306882 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f96fc08a950 con 0x7f970c15e8a0 2024-04-06T12:32:05.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.238+0000 7f970affd700 1 --2- 172.21.15.189:0/1804306882 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f96f806adb0 0x7f96f806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:05.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.238+0000 7f970affd700 1 --2- 172.21.15.189:0/1804306882 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f96f806adb0 0x7f96f806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f970c0cf5c0 tx=0x7f9700009040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:05.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.242+0000 7f96f3fff700 1 -- 172.21.15.189:0/1804306882 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f96fc0558f0 con 0x7f970c15e8a0 2024-04-06T12:32:05.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:05.486+0000 7f9711c7a700 1 -- 172.21.15.189:0/1804306882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f970c074990 con 0x7f970c15e8a0 2024-04-06T12:32:06.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.086+0000 7f96f3fff700 1 -- 172.21.15.189:0/1804306882 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7f96fc058f10 con 0x7f970c15e8a0 2024-04-06T12:32:06.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.089+0000 7f9711c7a700 1 -- 172.21.15.189:0/1804306882 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f96f806adb0 msgr2=0x7f96f806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:06.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.089+0000 7f9711c7a700 1 --2- 172.21.15.189:0/1804306882 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f96f806adb0 0x7f96f806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f970c0cf5c0 tx=0x7f9700009040 comp rx=0 tx=0).stop 2024-04-06T12:32:06.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.089+0000 7f9711c7a700 1 -- 172.21.15.189:0/1804306882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f970c15e8a0 msgr2=0x7f970c0cee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:06.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.089+0000 7f9711c7a700 1 --2- 172.21.15.189:0/1804306882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f970c15e8a0 0x7f970c0cee90 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f970c073430 tx=0x7f96fc004750 comp rx=0 tx=0).stop 2024-04-06T12:32:06.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.089+0000 7f9711c7a700 1 -- 172.21.15.189:0/1804306882 shutdown_connections 2024-04-06T12:32:06.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.089+0000 7f9711c7a700 1 --2- 172.21.15.189:0/1804306882 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f970c15fa70 0x7f970c0ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.089+0000 7f9711c7a700 1 --2- 172.21.15.189:0/1804306882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f970c15e8a0 0x7f970c0cee90 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.089+0000 7f9711c7a700 1 --2- 172.21.15.189:0/1804306882 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f96f806adb0 0x7f96f806d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.089+0000 7f9711c7a700 1 --2- 172.21.15.189:0/1804306882 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f970c15eee0 0x7f970c0cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.089+0000 7f9711c7a700 1 -- 172.21.15.189:0/1804306882 >> 172.21.15.189:0/1804306882 conn(0x7f970c0cc290 msgr2=0x7f970c0ae7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:06.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.089+0000 7f9711c7a700 1 -- 172.21.15.189:0/1804306882 shutdown_connections 2024-04-06T12:32:06.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.089+0000 7f9711c7a700 1 -- 172.21.15.189:0/1804306882 wait complete. 2024-04-06T12:32:06.100 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-06T12:32:06.100 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:32:06.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.226+0000 7f07add30700 1 Processor -- start 2024-04-06T12:32:06.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.226+0000 7f07add30700 1 -- start start 2024-04-06T12:32:06.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.227+0000 7f07add30700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07a815d7b0 0x7f07a80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.227+0000 7f07add30700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f07a806a730 0x7f07a806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.227+0000 7f07add30700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f07a815ddf0 0x7f07a815e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.227+0000 7f07add30700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a806b0f0 con 0x7f07a806a730 2024-04-06T12:32:06.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.227+0000 7f07add30700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a80b3990 con 0x7f07a815ddf0 2024-04-06T12:32:06.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.227+0000 7f07add30700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a80b3060 con 0x7f07a815d7b0 2024-04-06T12:32:06.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.227+0000 7f07a6ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f07a806a730 0x7f07a806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.227+0000 7f07a6ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f07a806a730 0x7f07a806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60712/0 (socket says 172.21.15.189:60712) 2024-04-06T12:32:06.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.227+0000 7f07a6ffd700 1 -- 172.21.15.189:0/957745402 learned_addr learned my addr 172.21.15.189:0/957745402 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:06.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.227+0000 7f07a7fff700 1 --2- 172.21.15.189:0/957745402 >> v2:172.21.15.192:3300/0 conn(0x7f07a815ddf0 0x7f07a815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.227+0000 7f07a77fe700 1 --2- 172.21.15.189:0/957745402 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07a815d7b0 0x7f07a80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.227+0000 7f07a6ffd700 1 -- 172.21.15.189:0/957745402 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07a815d7b0 msgr2=0x7f07a80b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.227+0000 7f07a6ffd700 1 --2- 172.21.15.189:0/957745402 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07a815d7b0 0x7f07a80b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.227+0000 7f07a6ffd700 1 -- 172.21.15.189:0/957745402 >> v2:172.21.15.192:3300/0 conn(0x7f07a815ddf0 msgr2=0x7f07a815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.227+0000 7f07a6ffd700 1 --2- 172.21.15.189:0/957745402 >> v2:172.21.15.192:3300/0 conn(0x7f07a815ddf0 0x7f07a815e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.227+0000 7f07a6ffd700 1 -- 172.21.15.189:0/957745402 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a80b3340 con 0x7f07a806a730 2024-04-06T12:32:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.227+0000 7f07a6ffd700 1 --2- 172.21.15.189:0/957745402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a806a730 0x7f07a806abb0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f07a80b4800 tx=0x7f0798009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7972d336dd47f7b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.228+0000 7f07a67fc700 1 -- 172.21.15.189:0/957745402 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079800b870 con 0x7f07a806a730 2024-04-06T12:32:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.228+0000 7f07a67fc700 1 -- 172.21.15.189:0/957745402 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079800b9d0 con 0x7f07a806a730 2024-04-06T12:32:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.228+0000 7f07a67fc700 1 -- 172.21.15.189:0/957745402 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079800bd60 con 0x7f07a806a730 2024-04-06T12:32:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.228+0000 7f07add30700 1 -- 172.21.15.189:0/957745402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a806a730 msgr2=0x7f07a806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.228+0000 7f07add30700 1 --2- 172.21.15.189:0/957745402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a806a730 0x7f07a806abb0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f07a80b4800 tx=0x7f0798009bd0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.228+0000 7f07add30700 1 -- 172.21.15.189:0/957745402 shutdown_connections 2024-04-06T12:32:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.228+0000 7f07add30700 1 --2- 172.21.15.189:0/957745402 >> v2:172.21.15.192:3300/0 conn(0x7f07a815ddf0 0x7f07a815e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.228+0000 7f07add30700 1 --2- 172.21.15.189:0/957745402 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07a815d7b0 0x7f07a80b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.228+0000 7f07add30700 1 --2- 172.21.15.189:0/957745402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a806a730 0x7f07a806abb0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.228+0000 7f07add30700 1 -- 172.21.15.189:0/957745402 >> 172.21.15.189:0/957745402 conn(0x7f07a80d00b0 msgr2=0x7f07a80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:06.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.228+0000 7f07add30700 1 -- 172.21.15.189:0/957745402 shutdown_connections 2024-04-06T12:32:06.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.228+0000 7f07add30700 1 -- 172.21.15.189:0/957745402 wait complete. 2024-04-06T12:32:06.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.229+0000 7f07add30700 1 Processor -- start 2024-04-06T12:32:06.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.230+0000 7f07add30700 1 -- start start 2024-04-06T12:32:06.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.230+0000 7f07add30700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07a806a730 0x7f07a8149140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.230+0000 7f07add30700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07a815d7b0 0x7f07a814c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.230+0000 7f07add30700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a815ddf0 0x7f07a81498f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.230+0000 7f07add30700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a814fd50 con 0x7f07a815ddf0 2024-04-06T12:32:06.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.230+0000 7f07add30700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a8149680 con 0x7f07a815d7b0 2024-04-06T12:32:06.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.230+0000 7f07add30700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a814c8d0 con 0x7f07a806a730 2024-04-06T12:32:06.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.230+0000 7f07a7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a815ddf0 0x7f07a81498f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.230+0000 7f07a7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a815ddf0 0x7f07a81498f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60722/0 (socket says 172.21.15.189:60722) 2024-04-06T12:32:06.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.230+0000 7f07a7fff700 1 -- 172.21.15.189:0/1031967066 learned_addr learned my addr 172.21.15.189:0/1031967066 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:06.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.231+0000 7f07a6ffd700 1 --2- 172.21.15.189:0/1031967066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07a815d7b0 0x7f07a814c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.231+0000 7f07a7fff700 1 -- 172.21.15.189:0/1031967066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07a806a730 msgr2=0x7f07a8149140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:06.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.231+0000 7f07a7fff700 1 --2- 172.21.15.189:0/1031967066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07a806a730 0x7f07a8149140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.231+0000 7f07a7fff700 1 -- 172.21.15.189:0/1031967066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07a815d7b0 msgr2=0x7f07a814c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:06.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.231+0000 7f07a7fff700 1 --2- 172.21.15.189:0/1031967066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07a815d7b0 0x7f07a814c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.231+0000 7f07a7fff700 1 -- 172.21.15.189:0/1031967066 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0798009550 con 0x7f07a815ddf0 2024-04-06T12:32:06.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.231+0000 7f07a7fff700 1 --2- 172.21.15.189:0/1031967066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a815ddf0 0x7f07a81498f0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f07a0007e60 tx=0x7f07a000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:06.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.231+0000 7f078ffff700 1 -- 172.21.15.189:0/1031967066 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a000ede0 con 0x7f07a815ddf0 2024-04-06T12:32:06.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.231+0000 7f078ffff700 1 -- 172.21.15.189:0/1031967066 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a0014420 con 0x7f07a815ddf0 2024-04-06T12:32:06.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.231+0000 7f07add30700 1 -- 172.21.15.189:0/1031967066 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a815b460 con 0x7f07a815ddf0 2024-04-06T12:32:06.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.231+0000 7f078ffff700 1 -- 172.21.15.189:0/1031967066 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a00147d0 con 0x7f07a815ddf0 2024-04-06T12:32:06.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.231+0000 7f07add30700 1 -- 172.21.15.189:0/1031967066 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a815ba20 con 0x7f07a815ddf0 2024-04-06T12:32:06.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.232+0000 7f07add30700 1 -- 172.21.15.189:0/1031967066 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a8067bb0 con 0x7f07a815ddf0 2024-04-06T12:32:06.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.232+0000 7f078ffff700 1 -- 172.21.15.189:0/1031967066 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f07a0014930 con 0x7f07a815ddf0 2024-04-06T12:32:06.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.233+0000 7f078ffff700 1 --2- 172.21.15.189:0/1031967066 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f079406adb0 0x7f079406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.233+0000 7f078ffff700 1 -- 172.21.15.189:0/1031967066 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f07a0089cb0 con 0x7f07a815ddf0 2024-04-06T12:32:06.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.233+0000 7f07a77fe700 1 --2- 172.21.15.189:0/1031967066 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f079406adb0 0x7f079406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.233+0000 7f07a77fe700 1 --2- 172.21.15.189:0/1031967066 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f079406adb0 0x7f079406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f07a80cd060 tx=0x7f079c008040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:06.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.237+0000 7f078ffff700 1 -- 172.21.15.189:0/1031967066 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f07a0054c40 con 0x7f07a815ddf0 2024-04-06T12:32:06.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.475+0000 7f07add30700 1 -- 172.21.15.189:0/1031967066 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07a8074990 con 0x7f07a815ddf0 2024-04-06T12:32:06.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.475+0000 7f078ffff700 1 -- 172.21.15.189:0/1031967066 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4748 (secure 0 0 0) 0x7f07a0058260 con 0x7f07a815ddf0 2024-04-06T12:32:06.473 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:32:06.473 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":70,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3697447910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3697447910},{"type":"v1","addr":"172.21.15.189:6837","nonce":3697447910}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":7469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/3633157989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3633157989},{"type":"v1","addr":"172.21.15.192:6834","nonce":3633157989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2721135386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2721135386},{"type":"v1","addr":"172.21.15.189:6835","nonce":2721135386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:32:03.067806+0000","modified":"2024-04-06T12:32:06.080510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9144":{"gid":9144,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/3068270636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":3068270636},{"type":"v1","addr":"172.21.15.192:6835","nonce":3068270636}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-06T12:32:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.478+0000 7f07add30700 1 -- 172.21.15.189:0/1031967066 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f079406adb0 msgr2=0x7f079406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.478+0000 7f07add30700 1 --2- 172.21.15.189:0/1031967066 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f079406adb0 0x7f079406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f07a80cd060 tx=0x7f079c008040 comp rx=0 tx=0).stop 2024-04-06T12:32:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.478+0000 7f07add30700 1 -- 172.21.15.189:0/1031967066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a815ddf0 msgr2=0x7f07a81498f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.478+0000 7f07add30700 1 --2- 172.21.15.189:0/1031967066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a815ddf0 0x7f07a81498f0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f07a0007e60 tx=0x7f07a000e3c0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.478+0000 7f07add30700 1 -- 172.21.15.189:0/1031967066 shutdown_connections 2024-04-06T12:32:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.478+0000 7f07add30700 1 --2- 172.21.15.189:0/1031967066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a815ddf0 0x7f07a81498f0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.478+0000 7f07add30700 1 --2- 172.21.15.189:0/1031967066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07a806a730 0x7f07a8149140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.478+0000 7f07add30700 1 --2- 172.21.15.189:0/1031967066 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f079406adb0 0x7f079406d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.478+0000 7f07add30700 1 --2- 172.21.15.189:0/1031967066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07a815d7b0 0x7f07a814c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.478+0000 7f07add30700 1 -- 172.21.15.189:0/1031967066 >> 172.21.15.189:0/1031967066 conn(0x7f07a80d00b0 msgr2=0x7f07a80ad470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.478+0000 7f07add30700 1 -- 172.21.15.189:0/1031967066 shutdown_connections 2024-04-06T12:32:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.478+0000 7f07add30700 1 -- 172.21.15.189:0/1031967066 wait complete. 2024-04-06T12:32:06.476 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 70 2024-04-06T12:32:06.487 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T12:32:06.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.615+0000 7efca4a4e700 1 Processor -- start 2024-04-06T12:32:06.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.615+0000 7efca4a4e700 1 -- start start 2024-04-06T12:32:06.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.616+0000 7efca4a4e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efca015fcd0 0x7efca015c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.616+0000 7efca4a4e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca015c740 con 0x7efca0144f40 2024-04-06T12:32:06.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.616+0000 7efca4a4e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca015f1f0 con 0x7efca006ae40 2024-04-06T12:32:06.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.616+0000 7efca4a4e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca00cfcb0 con 0x7efca015fcd0 2024-04-06T12:32:06.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.616+0000 7efc9e59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efca0144f40 0x7efca006a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43968/0 (socket says 172.21.15.189:43968) 2024-04-06T12:32:06.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.616+0000 7efc9e59c700 1 -- 172.21.15.189:0/1431932623 learned_addr learned my addr 172.21.15.189:0/1431932623 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:32:06.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.616+0000 7efc9d59a700 1 -- 172.21.15.189:0/1431932623 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621294359 0 0) 0x7efca015c740 con 0x7efca0144f40 2024-04-06T12:32:06.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.616+0000 7efc9d59a700 1 -- 172.21.15.189:0/1431932623 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc880035b0 con 0x7efca0144f40 2024-04-06T12:32:06.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.616+0000 7efc9ed9d700 1 --2- 172.21.15.189:0/1431932623 >> v2:172.21.15.192:3300/0 conn(0x7efca015fcd0 0x7efca015c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9d59a700 1 -- 172.21.15.189:0/1431932623 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 287473668 0 0) 0x7efc880035b0 con 0x7efca0144f40 2024-04-06T12:32:06.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9d59a700 1 -- 172.21.15.189:0/1431932623 >> v2:172.21.15.192:3300/0 conn(0x7efca015fcd0 msgr2=0x7efca015c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:06.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9d59a700 1 --2- 172.21.15.189:0/1431932623 >> v2:172.21.15.192:3300/0 conn(0x7efca015fcd0 0x7efca015c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9d59a700 1 -- 172.21.15.189:0/1431932623 >> v1:172.21.15.192:6789/0 conn(0x7efca006ae40 legacy=0x7efca015ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:06.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9d59a700 1 -- 172.21.15.189:0/1431932623 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca015f4d0 con 0x7efca0144f40 2024-04-06T12:32:06.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9d59a700 1 -- 172.21.15.189:0/1431932623 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7efc94001670 con 0x7efca0144f40 2024-04-06T12:32:06.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9d59a700 1 -- 172.21.15.189:0/1431932623 >> v1:172.21.15.189:6789/0 conn(0x7efca0144f40 legacy=0x7efca006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:06.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9d59a700 1 --2- 172.21.15.189:0/1431932623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc88003a00 0x7efc88003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9d59a700 1 --2- 172.21.15.189:0/1431932623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc88004850 0x7efc88004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9d59a700 1 --2- 172.21.15.189:0/1431932623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc88005330 0x7efc8800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9d59a700 1 -- 172.21.15.189:0/1431932623 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca00cfcb0 con 0x7efc88005330 2024-04-06T12:32:06.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9d59a700 1 -- 172.21.15.189:0/1431932623 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8800c200 con 0x7efc88004850 2024-04-06T12:32:06.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9d59a700 1 -- 172.21.15.189:0/1431932623 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8800c3c0 con 0x7efc88003a00 2024-04-06T12:32:06.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9ed9d700 1 --2- 172.21.15.189:0/1431932623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc88005330 0x7efc8800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9dd9b700 1 --2- 172.21.15.189:0/1431932623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc88004850 0x7efc88004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9e59c700 1 --2- 172.21.15.189:0/1431932623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc88003a00 0x7efc88003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9ed9d700 1 -- 172.21.15.189:0/1431932623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc88003a00 msgr2=0x7efc88003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:06.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9ed9d700 1 --2- 172.21.15.189:0/1431932623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc88003a00 0x7efc88003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9ed9d700 1 -- 172.21.15.189:0/1431932623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc88004850 msgr2=0x7efc88004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:06.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9ed9d700 1 --2- 172.21.15.189:0/1431932623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc88004850 0x7efc88004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9ed9d700 1 -- 172.21.15.189:0/1431932623 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc880035b0 con 0x7efc88005330 2024-04-06T12:32:06.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9dd9b700 1 --2- 172.21.15.189:0/1431932623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc88004850 0x7efc88004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:32:06.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9e59c700 1 --2- 172.21.15.189:0/1431932623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc88003a00 0x7efc88003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:32:06.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9ed9d700 1 -- 172.21.15.189:0/1431932623 reap_dead start 2024-04-06T12:32:06.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9ed9d700 1 --2- 172.21.15.189:0/1431932623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc88005330 0x7efc8800ba10 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7efca01600e0 tx=0x7efc88005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c118941964fd81e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:06.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9d59a700 1 -- 172.21.15.189:0/1431932623 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc98007e20 con 0x7efc88005330 2024-04-06T12:32:06.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9d59a700 1 -- 172.21.15.189:0/1431932623 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc98003a20 con 0x7efc88005330 2024-04-06T12:32:06.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.617+0000 7efc9d59a700 1 -- 172.21.15.189:0/1431932623 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc98003dd0 con 0x7efc88005330 2024-04-06T12:32:06.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.618+0000 7efca4a4e700 1 -- 172.21.15.189:0/1431932623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc88005330 msgr2=0x7efc8800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:06.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.618+0000 7efca4a4e700 1 --2- 172.21.15.189:0/1431932623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc88005330 0x7efc8800ba10 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7efca01600e0 tx=0x7efc88005880 comp rx=0 tx=0).stop 2024-04-06T12:32:06.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.618+0000 7efca4a4e700 1 -- 172.21.15.189:0/1431932623 shutdown_connections 2024-04-06T12:32:06.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.618+0000 7efca4a4e700 1 --2- 172.21.15.189:0/1431932623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc88005330 0x7efc8800ba10 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.618+0000 7efca4a4e700 1 -- 172.21.15.189:0/1431932623 >> 172.21.15.189:0/1431932623 conn(0x7efca00d00b0 msgr2=0x7efca00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:06.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.618+0000 7efca4a4e700 1 -- 172.21.15.189:0/1431932623 shutdown_connections 2024-04-06T12:32:06.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.618+0000 7efca4a4e700 1 -- 172.21.15.189:0/1431932623 wait complete. 2024-04-06T12:32:06.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.619+0000 7efca4a4e700 1 Processor -- start 2024-04-06T12:32:06.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.620+0000 7efca4a4e700 1 -- start start 2024-04-06T12:32:06.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.620+0000 7efca4a4e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc88005330 0x7efca01442f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.620+0000 7efca4a4e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efca013c600 0x7efca0139e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.620+0000 7efca4a4e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efca013bc90 0x7efca0136470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.620+0000 7efca4a4e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca01417b0 con 0x7efca013c600 2024-04-06T12:32:06.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.620+0000 7efca4a4e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca0144830 con 0x7efc88005330 2024-04-06T12:32:06.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.620+0000 7efca4a4e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca015af50 con 0x7efca013bc90 2024-04-06T12:32:06.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.620+0000 7efc9dd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efca013c600 0x7efca0139e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.620+0000 7efc9dd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efca013c600 0x7efca0139e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60740/0 (socket says 172.21.15.189:60740) 2024-04-06T12:32:06.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.620+0000 7efc9dd9b700 1 -- 172.21.15.189:0/3022456171 learned_addr learned my addr 172.21.15.189:0/3022456171 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:06.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.620+0000 7efc9e59c700 1 --2- 172.21.15.189:0/3022456171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc88005330 0x7efca01442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.621+0000 7efc9ed9d700 1 --2- 172.21.15.189:0/3022456171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efca013bc90 0x7efca0136470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.621+0000 7efc9dd9b700 1 -- 172.21.15.189:0/3022456171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efca013bc90 msgr2=0x7efca0136470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:06.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.621+0000 7efc9dd9b700 1 --2- 172.21.15.189:0/3022456171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efca013bc90 0x7efca0136470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.621+0000 7efc9dd9b700 1 -- 172.21.15.189:0/3022456171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc88005330 msgr2=0x7efca01442f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:06.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.621+0000 7efc9dd9b700 1 --2- 172.21.15.189:0/3022456171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc88005330 0x7efca01442f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.621+0000 7efc9dd9b700 1 -- 172.21.15.189:0/3022456171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca013a350 con 0x7efca013c600 2024-04-06T12:32:06.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.621+0000 7efc9dd9b700 1 --2- 172.21.15.189:0/3022456171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efca013c600 0x7efca0139e10 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7efc9000e480 tx=0x7efc9000e840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:06.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.621+0000 7efc8effd700 1 -- 172.21.15.189:0/3022456171 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc900052e0 con 0x7efca013c600 2024-04-06T12:32:06.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.621+0000 7efc8effd700 1 -- 172.21.15.189:0/3022456171 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc90005440 con 0x7efca013c600 2024-04-06T12:32:06.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.621+0000 7efca4a4e700 1 -- 172.21.15.189:0/3022456171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca013cb50 con 0x7efca013c600 2024-04-06T12:32:06.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.621+0000 7efc8effd700 1 -- 172.21.15.189:0/3022456171 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc900038c0 con 0x7efca013c600 2024-04-06T12:32:06.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.621+0000 7efca4a4e700 1 -- 172.21.15.189:0/3022456171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca01413d0 con 0x7efca013c600 2024-04-06T12:32:06.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.622+0000 7efca4a4e700 1 -- 172.21.15.189:0/3022456171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca0140f50 con 0x7efca013c600 2024-04-06T12:32:06.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.622+0000 7efc8effd700 1 -- 172.21.15.189:0/3022456171 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7efc90003a20 con 0x7efca013c600 2024-04-06T12:32:06.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.623+0000 7efc8effd700 1 --2- 172.21.15.189:0/3022456171 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efc8006adb0 0x7efc8006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.623+0000 7efc8effd700 1 -- 172.21.15.189:0/3022456171 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6029+0+0 (secure 0 0 0) 0x7efc90088940 con 0x7efca013c600 2024-04-06T12:32:06.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.623+0000 7efc9e59c700 1 --2- 172.21.15.189:0/3022456171 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efc8006adb0 0x7efc8006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.623+0000 7efc9e59c700 1 --2- 172.21.15.189:0/3022456171 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efc8006adb0 0x7efc8006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efc94000fb0 tx=0x7efc94009040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:06.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.627+0000 7efc8effd700 1 -- 172.21.15.189:0/3022456171 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efca0140f50 con 0x7efca013c600 2024-04-06T12:32:06.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.827+0000 7efca4a4e700 1 -- 172.21.15.189:0/3022456171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efca0140f50 con 0x7efca013c600 2024-04-06T12:32:06.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.828+0000 7efc8effd700 1 -- 172.21.15.189:0/3022456171 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v158) v1 ==== 74+0+16312 (secure 0 0 0) 0x7efca0140f50 con 0x7efca013c600 2024-04-06T12:32:06.825 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:32:06.825 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":158,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T12:32:04.069679+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:32:00.826039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-04-06T12:32:02.039801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T12:32:06.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.830+0000 7efca4a4e700 1 -- 172.21.15.189:0/3022456171 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efc8006adb0 msgr2=0x7efc8006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:06.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.830+0000 7efca4a4e700 1 --2- 172.21.15.189:0/3022456171 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efc8006adb0 0x7efc8006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efc94000fb0 tx=0x7efc94009040 comp rx=0 tx=0).stop 2024-04-06T12:32:06.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.830+0000 7efca4a4e700 1 -- 172.21.15.189:0/3022456171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efca013c600 msgr2=0x7efca0139e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:06.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.830+0000 7efca4a4e700 1 --2- 172.21.15.189:0/3022456171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efca013c600 0x7efca0139e10 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7efc9000e480 tx=0x7efc9000e840 comp rx=0 tx=0).stop 2024-04-06T12:32:06.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.830+0000 7efca4a4e700 1 -- 172.21.15.189:0/3022456171 shutdown_connections 2024-04-06T12:32:06.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.830+0000 7efca4a4e700 1 --2- 172.21.15.189:0/3022456171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efca013bc90 0x7efca0136470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.830+0000 7efca4a4e700 1 --2- 172.21.15.189:0/3022456171 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efc8006adb0 0x7efc8006d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.830+0000 7efca4a4e700 1 --2- 172.21.15.189:0/3022456171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc88005330 0x7efca01442f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.830+0000 7efca4a4e700 1 --2- 172.21.15.189:0/3022456171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efca013c600 0x7efca0139e10 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.830+0000 7efca4a4e700 1 -- 172.21.15.189:0/3022456171 >> 172.21.15.189:0/3022456171 conn(0x7efca00d00b0 msgr2=0x7efca013fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:06.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.830+0000 7efca4a4e700 1 -- 172.21.15.189:0/3022456171 shutdown_connections 2024-04-06T12:32:06.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.830+0000 7efca4a4e700 1 -- 172.21.15.189:0/3022456171 wait complete. 2024-04-06T12:32:06.840 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-06T12:32:06.840 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:32:06.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.965+0000 7fe2ce103700 1 Processor -- start 2024-04-06T12:32:06.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.965+0000 7fe2ce103700 1 -- start start 2024-04-06T12:32:06.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.966+0000 7fe2ce103700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe2c80d17a0 0x7fe2c806b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.966+0000 7fe2ce103700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe2c8146dd0 0x7fe2c806d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.966+0000 7fe2ce103700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c806c0a0 con 0x7fe2c80cedf0 2024-04-06T12:32:06.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.966+0000 7fe2ce103700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c806c380 con 0x7fe2c80d17a0 2024-04-06T12:32:06.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.966+0000 7fe2ce103700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c806c660 con 0x7fe2c8146dd0 2024-04-06T12:32:06.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.966+0000 7fe2c77fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe2c80d17a0 0x7fe2c806b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.966+0000 7fe2c7fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe2c80cedf0 0x7fe2c80cf1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43976/0 (socket says 172.21.15.189:43976) 2024-04-06T12:32:06.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.966+0000 7fe2c7fff700 1 -- 172.21.15.189:0/1336935428 learned_addr learned my addr 172.21.15.189:0/1336935428 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:32:06.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.966+0000 7fe2c6ffd700 1 --2- 172.21.15.189:0/1336935428 >> v2:172.21.15.192:3300/0 conn(0x7fe2c8146dd0 0x7fe2c806d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.966+0000 7fe2c67fc700 1 -- 172.21.15.189:0/1336935428 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056066278 0 0) 0x7fe2c806c0a0 con 0x7fe2c80cedf0 2024-04-06T12:32:06.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.966+0000 7fe2c67fc700 1 -- 172.21.15.189:0/1336935428 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b40035b0 con 0x7fe2c80cedf0 2024-04-06T12:32:06.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.966+0000 7fe2c77fe700 1 -- 172.21.15.189:0/1336935428 >> v2:172.21.15.192:3300/0 conn(0x7fe2c8146dd0 msgr2=0x7fe2c806d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:06.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.966+0000 7fe2c77fe700 1 --2- 172.21.15.189:0/1336935428 >> v2:172.21.15.192:3300/0 conn(0x7fe2c8146dd0 0x7fe2c806d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.967+0000 7fe2c77fe700 1 -- 172.21.15.189:0/1336935428 >> v1:172.21.15.189:6789/0 conn(0x7fe2c80cedf0 legacy=0x7fe2c80cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:06.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.967+0000 7fe2c77fe700 1 -- 172.21.15.189:0/1336935428 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c806c940 con 0x7fe2c80d17a0 2024-04-06T12:32:06.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.967+0000 7fe2c6ffd700 1 --2- 172.21.15.189:0/1336935428 >> v2:172.21.15.192:3300/0 conn(0x7fe2c8146dd0 0x7fe2c806d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:32:06.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.967+0000 7fe2c77fe700 1 --2- 172.21.15.189:0/1336935428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c80d17a0 0x7fe2c806b7a0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fe2c80d2300 tx=0x7fe2b0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=931a2612670cdb48 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:06.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.967+0000 7fe2c67fc700 1 -- 172.21.15.189:0/1336935428 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0011830 con 0x7fe2c80d17a0 2024-04-06T12:32:06.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.967+0000 7fe2c67fc700 1 -- 172.21.15.189:0/1336935428 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b0011990 con 0x7fe2c80d17a0 2024-04-06T12:32:06.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.967+0000 7fe2c67fc700 1 -- 172.21.15.189:0/1336935428 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0011d20 con 0x7fe2c80d17a0 2024-04-06T12:32:06.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.967+0000 7fe2ce103700 1 -- 172.21.15.189:0/1336935428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c80d17a0 msgr2=0x7fe2c806b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:06.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.967+0000 7fe2ce103700 1 --2- 172.21.15.189:0/1336935428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c80d17a0 0x7fe2c806b7a0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fe2c80d2300 tx=0x7fe2b0009b20 comp rx=0 tx=0).stop 2024-04-06T12:32:06.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.967+0000 7fe2ce103700 1 -- 172.21.15.189:0/1336935428 shutdown_connections 2024-04-06T12:32:06.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.967+0000 7fe2ce103700 1 --2- 172.21.15.189:0/1336935428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c80d17a0 0x7fe2c806b7a0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.967+0000 7fe2ce103700 1 --2- 172.21.15.189:0/1336935428 >> v2:172.21.15.192:3300/0 conn(0x7fe2c8146dd0 0x7fe2c806d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.967+0000 7fe2ce103700 1 -- 172.21.15.189:0/1336935428 >> 172.21.15.189:0/1336935428 conn(0x7fe2c80b2db0 msgr2=0x7fe2c80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:06.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.967+0000 7fe2ce103700 1 -- 172.21.15.189:0/1336935428 shutdown_connections 2024-04-06T12:32:06.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.967+0000 7fe2ce103700 1 -- 172.21.15.189:0/1336935428 wait complete. 2024-04-06T12:32:06.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.969+0000 7fe2ce103700 1 Processor -- start 2024-04-06T12:32:06.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.969+0000 7fe2ce103700 1 -- start start 2024-04-06T12:32:06.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.969+0000 7fe2ce103700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c80cedf0 0x7fe2c80cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.969+0000 7fe2ce103700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2c80d17a0 0x7fe2c80ae580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.969+0000 7fe2ce103700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2c8146dd0 0x7fe2c8160230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.969+0000 7fe2ce103700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c80b03b0 con 0x7fe2c80cedf0 2024-04-06T12:32:06.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.969+0000 7fe2ce103700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c80aeac0 con 0x7fe2c80d17a0 2024-04-06T12:32:06.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.969+0000 7fe2ce103700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c80ad4e0 con 0x7fe2c8146dd0 2024-04-06T12:32:06.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.970+0000 7fe2c77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c80cedf0 0x7fe2c80cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.970+0000 7fe2c77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c80cedf0 0x7fe2c80cdb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60762/0 (socket says 172.21.15.189:60762) 2024-04-06T12:32:06.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.970+0000 7fe2c77fe700 1 -- 172.21.15.189:0/3304241594 learned_addr learned my addr 172.21.15.189:0/3304241594 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:06.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.970+0000 7fe2c6ffd700 1 --2- 172.21.15.189:0/3304241594 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2c80d17a0 0x7fe2c80ae580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.970+0000 7fe2c7fff700 1 --2- 172.21.15.189:0/3304241594 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2c8146dd0 0x7fe2c8160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.970+0000 7fe2c77fe700 1 -- 172.21.15.189:0/3304241594 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2c8146dd0 msgr2=0x7fe2c8160230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:06.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.970+0000 7fe2c77fe700 1 --2- 172.21.15.189:0/3304241594 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2c8146dd0 0x7fe2c8160230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.970+0000 7fe2c77fe700 1 -- 172.21.15.189:0/3304241594 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2c80d17a0 msgr2=0x7fe2c80ae580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:06.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.970+0000 7fe2c77fe700 1 --2- 172.21.15.189:0/3304241594 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2c80d17a0 0x7fe2c80ae580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:06.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.970+0000 7fe2c77fe700 1 -- 172.21.15.189:0/3304241594 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b00094a0 con 0x7fe2c80cedf0 2024-04-06T12:32:06.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.970+0000 7fe2c7fff700 1 --2- 172.21.15.189:0/3304241594 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2c8146dd0 0x7fe2c8160230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:32:06.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.970+0000 7fe2c77fe700 1 --2- 172.21.15.189:0/3304241594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c80cedf0 0x7fe2c80cdb70 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0009b60 tx=0x7fe2b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:06.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.970+0000 7fe2bffff700 1 -- 172.21.15.189:0/3304241594 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b00036a0 con 0x7fe2c80cedf0 2024-04-06T12:32:06.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.970+0000 7fe2bffff700 1 -- 172.21.15.189:0/3304241594 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b001f070 con 0x7fe2c80cedf0 2024-04-06T12:32:06.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.970+0000 7fe2ce103700 1 -- 172.21.15.189:0/3304241594 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c80ad7c0 con 0x7fe2c80cedf0 2024-04-06T12:32:06.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.970+0000 7fe2bffff700 1 -- 172.21.15.189:0/3304241594 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0003a40 con 0x7fe2c80cedf0 2024-04-06T12:32:06.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.970+0000 7fe2ce103700 1 -- 172.21.15.189:0/3304241594 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c80add80 con 0x7fe2c80cedf0 2024-04-06T12:32:06.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.971+0000 7fe2ce103700 1 -- 172.21.15.189:0/3304241594 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c8074990 con 0x7fe2c80cedf0 2024-04-06T12:32:06.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.972+0000 7fe2bffff700 1 -- 172.21.15.189:0/3304241594 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fe2b0003ba0 con 0x7fe2c80cedf0 2024-04-06T12:32:06.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.972+0000 7fe2bffff700 1 --2- 172.21.15.189:0/3304241594 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2a806adb0 0x7fe2a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:06.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.972+0000 7fe2bffff700 1 -- 172.21.15.189:0/3304241594 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe2b0021070 con 0x7fe2c80cedf0 2024-04-06T12:32:06.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.972+0000 7fe2c6ffd700 1 --2- 172.21.15.189:0/3304241594 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2a806adb0 0x7fe2a806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:06.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.972+0000 7fe2c6ffd700 1 --2- 172.21.15.189:0/3304241594 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2a806adb0 0x7fe2a806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe2c80c7f30 tx=0x7fe2b8008040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:06.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:06.976+0000 7fe2bffff700 1 -- 172.21.15.189:0/3304241594 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe2b0055e90 con 0x7fe2c80cedf0 2024-04-06T12:32:07.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.169+0000 7fe2ce103700 1 -- 172.21.15.189:0/3304241594 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe2c80d1b80 con 0x7fe2a806adb0 2024-04-06T12:32:07.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.177+0000 7fe2bffff700 1 -- 172.21.15.189:0/3304241594 <== mgr.8176 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287784 (secure 0 0 0) 0x7fe2b800c050 con 0x7fe2a806adb0 2024-04-06T12:32:07.179 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:32:07.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.180+0000 7fe2ce103700 1 -- 172.21.15.189:0/3304241594 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2a806adb0 msgr2=0x7fe2a806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:07.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.180+0000 7fe2ce103700 1 --2- 172.21.15.189:0/3304241594 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2a806adb0 0x7fe2a806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe2c80c7f30 tx=0x7fe2b8008040 comp rx=0 tx=0).stop 2024-04-06T12:32:07.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.180+0000 7fe2ce103700 1 -- 172.21.15.189:0/3304241594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c80cedf0 msgr2=0x7fe2c80cdb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:07.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.180+0000 7fe2ce103700 1 --2- 172.21.15.189:0/3304241594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c80cedf0 0x7fe2c80cdb70 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0009b60 tx=0x7fe2b0004750 comp rx=0 tx=0).stop 2024-04-06T12:32:07.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.180+0000 7fe2ce103700 1 -- 172.21.15.189:0/3304241594 shutdown_connections 2024-04-06T12:32:07.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.180+0000 7fe2ce103700 1 --2- 172.21.15.189:0/3304241594 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2c8146dd0 0x7fe2c8160230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:07.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.180+0000 7fe2ce103700 1 --2- 172.21.15.189:0/3304241594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c80cedf0 0x7fe2c80cdb70 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:07.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.180+0000 7fe2ce103700 1 --2- 172.21.15.189:0/3304241594 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe2a806adb0 0x7fe2a806d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:07.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.180+0000 7fe2ce103700 1 --2- 172.21.15.189:0/3304241594 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2c80d17a0 0x7fe2c80ae580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:07.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.180+0000 7fe2ce103700 1 -- 172.21.15.189:0/3304241594 >> 172.21.15.189:0/3304241594 conn(0x7fe2c80b2db0 msgr2=0x7fe2c815db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:07.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.180+0000 7fe2ce103700 1 -- 172.21.15.189:0/3304241594 shutdown_connections 2024-04-06T12:32:07.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.180+0000 7fe2ce103700 1 -- 172.21.15.189:0/3304241594 wait complete. 2024-04-06T12:32:07.185 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:32:07.190 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-06T12:32:05.400765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592148,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":436,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":281,"ondisk_log_size":281,"up":166,"acting":166,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":85,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":336012,"kb_used_data":16528,"kb_used_omap":12,"kb_used_meta":319347,"kb_avail":754638708,"statfs":{"total":773094113280,"available":772750036992,"internally_reserved":0,"allocated":16924672,"data_stored":9067714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":327011896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1780,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000619"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034823+0000","last_change":"2024-04-06T12:32:03.082505+0000","last_active":"2024-04-06T12:32:05.034823+0000","last_peered":"2024-04-06T12:32:05.034823+0000","last_clean":"2024-04-06T12:32:05.034823+0000","last_became_active":"2024-04-06T12:32:03.082250+0000","last_became_peered":"2024-04-06T12:32:03.082250+0000","last_unstale":"2024-04-06T12:32:05.034823+0000","last_undegraded":"2024-04-06T12:32:05.034823+0000","last_fullsized":"2024-04-06T12:32:05.034823+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:25:53.432862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034139+0000","last_change":"2024-04-06T12:32:03.080983+0000","last_active":"2024-04-06T12:32:05.034139+0000","last_peered":"2024-04-06T12:32:05.034139+0000","last_clean":"2024-04-06T12:32:05.034139+0000","last_became_active":"2024-04-06T12:32:03.080727+0000","last_became_peered":"2024-04-06T12:32:03.080727+0000","last_unstale":"2024-04-06T12:32:05.034139+0000","last_undegraded":"2024-04-06T12:32:05.034139+0000","last_fullsized":"2024-04-06T12:32:05.034139+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:51:37.184427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:32:02.652297+0000","last_change":"2024-04-06T12:32:02.652297+0000","last_active":"2024-04-06T12:32:02.652297+0000","last_peered":"2024-04-06T12:32:02.652297+0000","last_clean":"2024-04-06T12:32:02.652297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:32:02.652297+0000","last_undegraded":"2024-04-06T12:32:02.652297+0000","last_fullsized":"2024-04-06T12:32:02.652297+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034079+0000","last_change":"2024-04-06T12:32:03.083127+0000","last_active":"2024-04-06T12:32:05.034079+0000","last_peered":"2024-04-06T12:32:05.034079+0000","last_clean":"2024-04-06T12:32:05.034079+0000","last_became_active":"2024-04-06T12:32:03.082827+0000","last_became_peered":"2024-04-06T12:32:03.082827+0000","last_unstale":"2024-04-06T12:32:05.034079+0000","last_undegraded":"2024-04-06T12:32:05.034079+0000","last_fullsized":"2024-04-06T12:32:05.034079+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:18:37.631769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076339+0000","last_change":"2024-04-06T12:32:03.090821+0000","last_active":"2024-04-06T12:32:04.076339+0000","last_peered":"2024-04-06T12:32:04.076339+0000","last_clean":"2024-04-06T12:32:04.076339+0000","last_became_active":"2024-04-06T12:32:03.090524+0000","last_became_peered":"2024-04-06T12:32:03.090524+0000","last_unstale":"2024-04-06T12:32:04.076339+0000","last_undegraded":"2024-04-06T12:32:04.076339+0000","last_fullsized":"2024-04-06T12:32:04.076339+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:39.465068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034247+0000","last_change":"2024-04-06T12:32:03.095044+0000","last_active":"2024-04-06T12:32:05.034247+0000","last_peered":"2024-04-06T12:32:05.034247+0000","last_clean":"2024-04-06T12:32:05.034247+0000","last_became_active":"2024-04-06T12:32:03.094343+0000","last_became_peered":"2024-04-06T12:32:03.094343+0000","last_unstale":"2024-04-06T12:32:05.034247+0000","last_undegraded":"2024-04-06T12:32:05.034247+0000","last_fullsized":"2024-04-06T12:32:05.034247+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:17:55.389634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076557+0000","last_change":"2024-04-06T12:32:03.102162+0000","last_active":"2024-04-06T12:32:04.076557+0000","last_peered":"2024-04-06T12:32:04.076557+0000","last_clean":"2024-04-06T12:32:04.076557+0000","last_became_active":"2024-04-06T12:32:03.101868+0000","last_became_peered":"2024-04-06T12:32:03.101868+0000","last_unstale":"2024-04-06T12:32:04.076557+0000","last_undegraded":"2024-04-06T12:32:04.076557+0000","last_fullsized":"2024-04-06T12:32:04.076557+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:11:29.631951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.445270+0000","last_change":"2024-04-06T12:32:03.089271+0000","last_active":"2024-04-06T12:32:04.445270+0000","last_peered":"2024-04-06T12:32:04.445270+0000","last_clean":"2024-04-06T12:32:04.445270+0000","last_became_active":"2024-04-06T12:32:03.088992+0000","last_became_peered":"2024-04-06T12:32:03.088992+0000","last_unstale":"2024-04-06T12:32:04.445270+0000","last_undegraded":"2024-04-06T12:32:04.445270+0000","last_fullsized":"2024-04-06T12:32:04.445270+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:31:26.386814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034393+0000","last_change":"2024-04-06T12:32:03.080536+0000","last_active":"2024-04-06T12:32:05.034393+0000","last_peered":"2024-04-06T12:32:05.034393+0000","last_clean":"2024-04-06T12:32:05.034393+0000","last_became_active":"2024-04-06T12:32:03.080168+0000","last_became_peered":"2024-04-06T12:32:03.080168+0000","last_unstale":"2024-04-06T12:32:05.034393+0000","last_undegraded":"2024-04-06T12:32:05.034393+0000","last_fullsized":"2024-04-06T12:32:05.034393+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:24:51.136620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076436+0000","last_change":"2024-04-06T12:32:03.084520+0000","last_active":"2024-04-06T12:32:04.076436+0000","last_peered":"2024-04-06T12:32:04.076436+0000","last_clean":"2024-04-06T12:32:04.076436+0000","last_became_active":"2024-04-06T12:32:03.084255+0000","last_became_peered":"2024-04-06T12:32:03.084255+0000","last_unstale":"2024-04-06T12:32:04.076436+0000","last_undegraded":"2024-04-06T12:32:04.076436+0000","last_fullsized":"2024-04-06T12:32:04.076436+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:17:09.015831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034024+0000","last_change":"2024-04-06T12:32:03.097727+0000","last_active":"2024-04-06T12:32:05.034024+0000","last_peered":"2024-04-06T12:32:05.034024+0000","last_clean":"2024-04-06T12:32:05.034024+0000","last_became_active":"2024-04-06T12:32:03.097459+0000","last_became_peered":"2024-04-06T12:32:03.097459+0000","last_unstale":"2024-04-06T12:32:05.034024+0000","last_undegraded":"2024-04-06T12:32:05.034024+0000","last_fullsized":"2024-04-06T12:32:05.034024+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:31:01.252124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034764+0000","last_change":"2024-04-06T12:32:03.089641+0000","last_active":"2024-04-06T12:32:05.034764+0000","last_peered":"2024-04-06T12:32:05.034764+0000","last_clean":"2024-04-06T12:32:05.034764+0000","last_became_active":"2024-04-06T12:32:03.089401+0000","last_became_peered":"2024-04-06T12:32:03.089401+0000","last_unstale":"2024-04-06T12:32:05.034764+0000","last_undegraded":"2024-04-06T12:32:05.034764+0000","last_fullsized":"2024-04-06T12:32:05.034764+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:22:21.471765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034704+0000","last_change":"2024-04-06T12:32:03.091650+0000","last_active":"2024-04-06T12:32:05.034704+0000","last_peered":"2024-04-06T12:32:05.034704+0000","last_clean":"2024-04-06T12:32:05.034704+0000","last_became_active":"2024-04-06T12:32:03.091323+0000","last_became_peered":"2024-04-06T12:32:03.091323+0000","last_unstale":"2024-04-06T12:32:05.034704+0000","last_undegraded":"2024-04-06T12:32:05.034704+0000","last_fullsized":"2024-04-06T12:32:05.034704+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:48:06.372887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034665+0000","last_change":"2024-04-06T12:32:03.098994+0000","last_active":"2024-04-06T12:32:05.034665+0000","last_peered":"2024-04-06T12:32:05.034665+0000","last_clean":"2024-04-06T12:32:05.034665+0000","last_became_active":"2024-04-06T12:32:03.098789+0000","last_became_peered":"2024-04-06T12:32:03.098789+0000","last_unstale":"2024-04-06T12:32:05.034665+0000","last_undegraded":"2024-04-06T12:32:05.034665+0000","last_fullsized":"2024-04-06T12:32:05.034665+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:39:56.115752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034053+0000","last_change":"2024-04-06T12:32:03.089587+0000","last_active":"2024-04-06T12:32:05.034053+0000","last_peered":"2024-04-06T12:32:05.034053+0000","last_clean":"2024-04-06T12:32:05.034053+0000","last_became_active":"2024-04-06T12:32:03.089318+0000","last_became_peered":"2024-04-06T12:32:03.089318+0000","last_unstale":"2024-04-06T12:32:05.034053+0000","last_undegraded":"2024-04-06T12:32:05.034053+0000","last_fullsized":"2024-04-06T12:32:05.034053+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:23.985084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.033876+0000","last_change":"2024-04-06T12:32:03.084489+0000","last_active":"2024-04-06T12:32:05.033876+0000","last_peered":"2024-04-06T12:32:05.033876+0000","last_clean":"2024-04-06T12:32:05.033876+0000","last_became_active":"2024-04-06T12:32:03.084180+0000","last_became_peered":"2024-04-06T12:32:03.084180+0000","last_unstale":"2024-04-06T12:32:05.033876+0000","last_undegraded":"2024-04-06T12:32:05.033876+0000","last_fullsized":"2024-04-06T12:32:05.033876+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:31:48.007605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034907+0000","last_change":"2024-04-06T12:32:03.084356+0000","last_active":"2024-04-06T12:32:05.034907+0000","last_peered":"2024-04-06T12:32:05.034907+0000","last_clean":"2024-04-06T12:32:05.034907+0000","last_became_active":"2024-04-06T12:32:03.084107+0000","last_became_peered":"2024-04-06T12:32:03.084107+0000","last_unstale":"2024-04-06T12:32:05.034907+0000","last_undegraded":"2024-04-06T12:32:05.034907+0000","last_fullsized":"2024-04-06T12:32:05.034907+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:03:09.310724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076997+0000","last_change":"2024-04-06T12:32:03.091046+0000","last_active":"2024-04-06T12:32:04.076997+0000","last_peered":"2024-04-06T12:32:04.076997+0000","last_clean":"2024-04-06T12:32:04.076997+0000","last_became_active":"2024-04-06T12:32:03.090743+0000","last_became_peered":"2024-04-06T12:32:03.090743+0000","last_unstale":"2024-04-06T12:32:04.076997+0000","last_undegraded":"2024-04-06T12:32:04.076997+0000","last_fullsized":"2024-04-06T12:32:04.076997+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:41:07.421129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077179+0000","last_change":"2024-04-06T12:32:03.104504+0000","last_active":"2024-04-06T12:32:04.077179+0000","last_peered":"2024-04-06T12:32:04.077179+0000","last_clean":"2024-04-06T12:32:04.077179+0000","last_became_active":"2024-04-06T12:32:03.104269+0000","last_became_peered":"2024-04-06T12:32:03.104269+0000","last_unstale":"2024-04-06T12:32:04.077179+0000","last_undegraded":"2024-04-06T12:32:04.077179+0000","last_fullsized":"2024-04-06T12:32:04.077179+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:24:51.136620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076452+0000","last_change":"2024-04-06T12:32:03.107653+0000","last_active":"2024-04-06T12:32:04.076452+0000","last_peered":"2024-04-06T12:32:04.076452+0000","last_clean":"2024-04-06T12:32:04.076452+0000","last_became_active":"2024-04-06T12:32:03.107419+0000","last_became_peered":"2024-04-06T12:32:03.107419+0000","last_unstale":"2024-04-06T12:32:04.076452+0000","last_undegraded":"2024-04-06T12:32:04.076452+0000","last_fullsized":"2024-04-06T12:32:04.076452+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:58:40.468358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034078+0000","last_change":"2024-04-06T12:32:03.084284+0000","last_active":"2024-04-06T12:32:05.034078+0000","last_peered":"2024-04-06T12:32:05.034078+0000","last_clean":"2024-04-06T12:32:05.034078+0000","last_became_active":"2024-04-06T12:32:03.083950+0000","last_became_peered":"2024-04-06T12:32:03.083950+0000","last_unstale":"2024-04-06T12:32:05.034078+0000","last_undegraded":"2024-04-06T12:32:05.034078+0000","last_fullsized":"2024-04-06T12:32:05.034078+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:00.765348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034689+0000","last_change":"2024-04-06T12:32:03.087351+0000","last_active":"2024-04-06T12:32:05.034689+0000","last_peered":"2024-04-06T12:32:05.034689+0000","last_clean":"2024-04-06T12:32:05.034689+0000","last_became_active":"2024-04-06T12:32:03.087111+0000","last_became_peered":"2024-04-06T12:32:03.087111+0000","last_unstale":"2024-04-06T12:32:05.034689+0000","last_undegraded":"2024-04-06T12:32:05.034689+0000","last_fullsized":"2024-04-06T12:32:05.034689+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:09:38.972919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.033956+0000","last_change":"2024-04-06T12:32:03.090736+0000","last_active":"2024-04-06T12:32:05.033956+0000","last_peered":"2024-04-06T12:32:05.033956+0000","last_clean":"2024-04-06T12:32:05.033956+0000","last_became_active":"2024-04-06T12:32:03.090461+0000","last_became_peered":"2024-04-06T12:32:03.090461+0000","last_unstale":"2024-04-06T12:32:05.033956+0000","last_undegraded":"2024-04-06T12:32:05.033956+0000","last_fullsized":"2024-04-06T12:32:05.033956+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:09:38.972919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.033972+0000","last_change":"2024-04-06T12:32:03.093015+0000","last_active":"2024-04-06T12:32:05.033972+0000","last_peered":"2024-04-06T12:32:05.033972+0000","last_clean":"2024-04-06T12:32:05.033972+0000","last_became_active":"2024-04-06T12:32:03.092769+0000","last_became_peered":"2024-04-06T12:32:03.092769+0000","last_unstale":"2024-04-06T12:32:05.033972+0000","last_undegraded":"2024-04-06T12:32:05.033972+0000","last_fullsized":"2024-04-06T12:32:05.033972+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:25:42.693508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:32:02.652297+0000","last_change":"2024-04-06T12:32:02.652297+0000","last_active":"2024-04-06T12:32:02.652297+0000","last_peered":"2024-04-06T12:32:02.652297+0000","last_clean":"2024-04-06T12:32:02.652297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:32:02.652297+0000","last_undegraded":"2024-04-06T12:32:02.652297+0000","last_fullsized":"2024-04-06T12:32:02.652297+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:32:02.652297+0000","last_change":"2024-04-06T12:32:02.652297+0000","last_active":"2024-04-06T12:32:02.652297+0000","last_peered":"2024-04-06T12:32:02.652297+0000","last_clean":"2024-04-06T12:32:02.652297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:32:02.652297+0000","last_undegraded":"2024-04-06T12:32:02.652297+0000","last_fullsized":"2024-04-06T12:32:02.652297+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077030+0000","last_change":"2024-04-06T12:32:03.101971+0000","last_active":"2024-04-06T12:32:04.077030+0000","last_peered":"2024-04-06T12:32:04.077030+0000","last_clean":"2024-04-06T12:32:04.077030+0000","last_became_active":"2024-04-06T12:32:03.101656+0000","last_became_peered":"2024-04-06T12:32:03.101656+0000","last_unstale":"2024-04-06T12:32:04.077030+0000","last_undegraded":"2024-04-06T12:32:04.077030+0000","last_fullsized":"2024-04-06T12:32:04.077030+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:03:09.310724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076726+0000","last_change":"2024-04-06T12:32:03.082945+0000","last_active":"2024-04-06T12:32:04.076726+0000","last_peered":"2024-04-06T12:32:04.076726+0000","last_clean":"2024-04-06T12:32:04.076726+0000","last_became_active":"2024-04-06T12:32:03.082707+0000","last_became_peered":"2024-04-06T12:32:03.082707+0000","last_unstale":"2024-04-06T12:32:04.076726+0000","last_undegraded":"2024-04-06T12:32:04.076726+0000","last_fullsized":"2024-04-06T12:32:04.076726+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:58:03.740342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:32:02.652297+0000","last_change":"2024-04-06T12:32:02.652297+0000","last_active":"2024-04-06T12:32:02.652297+0000","last_peered":"2024-04-06T12:32:02.652297+0000","last_clean":"2024-04-06T12:32:02.652297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:32:02.652297+0000","last_undegraded":"2024-04-06T12:32:02.652297+0000","last_fullsized":"2024-04-06T12:32:02.652297+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:32:02.652297+0000","last_change":"2024-04-06T12:32:02.652297+0000","last_active":"2024-04-06T12:32:02.652297+0000","last_peered":"2024-04-06T12:32:02.652297+0000","last_clean":"2024-04-06T12:32:02.652297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:32:02.652297+0000","last_undegraded":"2024-04-06T12:32:02.652297+0000","last_fullsized":"2024-04-06T12:32:02.652297+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.445302+0000","last_change":"2024-04-06T12:32:03.089180+0000","last_active":"2024-04-06T12:32:04.445302+0000","last_peered":"2024-04-06T12:32:04.445302+0000","last_clean":"2024-04-06T12:32:04.445302+0000","last_became_active":"2024-04-06T12:32:03.088869+0000","last_became_peered":"2024-04-06T12:32:03.088869+0000","last_unstale":"2024-04-06T12:32:04.445302+0000","last_undegraded":"2024-04-06T12:32:04.445302+0000","last_fullsized":"2024-04-06T12:32:04.445302+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:05.556922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076117+0000","last_change":"2024-04-06T12:32:03.084883+0000","last_active":"2024-04-06T12:32:04.076117+0000","last_peered":"2024-04-06T12:32:04.076117+0000","last_clean":"2024-04-06T12:32:04.076117+0000","last_became_active":"2024-04-06T12:32:03.084643+0000","last_became_peered":"2024-04-06T12:32:03.084643+0000","last_unstale":"2024-04-06T12:32:04.076117+0000","last_undegraded":"2024-04-06T12:32:04.076117+0000","last_fullsized":"2024-04-06T12:32:04.076117+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:47:54.703707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.445270+0000","last_change":"2024-04-06T12:32:03.096345+0000","last_active":"2024-04-06T12:32:04.445270+0000","last_peered":"2024-04-06T12:32:04.445270+0000","last_clean":"2024-04-06T12:32:04.445270+0000","last_became_active":"2024-04-06T12:32:03.096091+0000","last_became_peered":"2024-04-06T12:32:03.096091+0000","last_unstale":"2024-04-06T12:32:04.445270+0000","last_undegraded":"2024-04-06T12:32:04.445270+0000","last_fullsized":"2024-04-06T12:32:04.445270+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:22:30.405552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:32:01.643253+0000","last_change":"2024-04-06T12:32:01.643253+0000","last_active":"2024-04-06T12:32:01.643253+0000","last_peered":"2024-04-06T12:32:01.643253+0000","last_clean":"2024-04-06T12:32:01.643253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:32:01.643253+0000","last_undegraded":"2024-04-06T12:32:01.643253+0000","last_fullsized":"2024-04-06T12:32:01.643253+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034165+0000","last_change":"2024-04-06T12:32:03.088020+0000","last_active":"2024-04-06T12:32:05.034165+0000","last_peered":"2024-04-06T12:32:05.034165+0000","last_clean":"2024-04-06T12:32:05.034165+0000","last_became_active":"2024-04-06T12:32:03.087581+0000","last_became_peered":"2024-04-06T12:32:03.087581+0000","last_unstale":"2024-04-06T12:32:05.034165+0000","last_undegraded":"2024-04-06T12:32:05.034165+0000","last_fullsized":"2024-04-06T12:32:05.034165+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:56:21.396821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"157'1","reported_seq":12,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077143+0000","last_change":"2024-04-06T12:32:02.670991+0000","last_active":"2024-04-06T12:32:04.077143+0000","last_peered":"2024-04-06T12:32:04.077143+0000","last_clean":"2024-04-06T12:32:04.077143+0000","last_became_active":"2024-04-06T12:32:02.670714+0000","last_became_peered":"2024-04-06T12:32:02.670714+0000","last_unstale":"2024-04-06T12:32:04.077143+0000","last_undegraded":"2024-04-06T12:32:04.077143+0000","last_fullsized":"2024-04-06T12:32:04.077143+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:03:08.301680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034011+0000","last_change":"2024-04-06T12:32:03.099273+0000","last_active":"2024-04-06T12:32:05.034011+0000","last_peered":"2024-04-06T12:32:05.034011+0000","last_clean":"2024-04-06T12:32:05.034011+0000","last_became_active":"2024-04-06T12:32:03.099021+0000","last_became_peered":"2024-04-06T12:32:03.099021+0000","last_unstale":"2024-04-06T12:32:05.034011+0000","last_undegraded":"2024-04-06T12:32:05.034011+0000","last_fullsized":"2024-04-06T12:32:05.034011+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:52:04.592856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"157'2","reported_seq":13,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077096+0000","last_change":"2024-04-06T12:32:02.674743+0000","last_active":"2024-04-06T12:32:04.077096+0000","last_peered":"2024-04-06T12:32:04.077096+0000","last_clean":"2024-04-06T12:32:04.077096+0000","last_became_active":"2024-04-06T12:32:02.674438+0000","last_became_peered":"2024-04-06T12:32:02.674438+0000","last_unstale":"2024-04-06T12:32:04.077096+0000","last_undegraded":"2024-04-06T12:32:04.077096+0000","last_fullsized":"2024-04-06T12:32:04.077096+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:29:22.645084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034488+0000","last_change":"2024-04-06T12:32:03.090270+0000","last_active":"2024-04-06T12:32:05.034488+0000","last_peered":"2024-04-06T12:32:05.034488+0000","last_clean":"2024-04-06T12:32:05.034488+0000","last_became_active":"2024-04-06T12:32:03.090029+0000","last_became_peered":"2024-04-06T12:32:03.090029+0000","last_unstale":"2024-04-06T12:32:05.034488+0000","last_undegraded":"2024-04-06T12:32:05.034488+0000","last_fullsized":"2024-04-06T12:32:05.034488+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:11:33.053576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"157'2","reported_seq":13,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077222+0000","last_change":"2024-04-06T12:32:02.667498+0000","last_active":"2024-04-06T12:32:04.077222+0000","last_peered":"2024-04-06T12:32:04.077222+0000","last_clean":"2024-04-06T12:32:04.077222+0000","last_became_active":"2024-04-06T12:32:02.667227+0000","last_became_peered":"2024-04-06T12:32:02.667227+0000","last_unstale":"2024-04-06T12:32:04.077222+0000","last_undegraded":"2024-04-06T12:32:04.077222+0000","last_fullsized":"2024-04-06T12:32:04.077222+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:45:12.515973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.445549+0000","last_change":"2024-04-06T12:32:03.089533+0000","last_active":"2024-04-06T12:32:04.445549+0000","last_peered":"2024-04-06T12:32:04.445549+0000","last_clean":"2024-04-06T12:32:04.445549+0000","last_became_active":"2024-04-06T12:32:03.088844+0000","last_became_peered":"2024-04-06T12:32:03.088844+0000","last_unstale":"2024-04-06T12:32:04.445549+0000","last_undegraded":"2024-04-06T12:32:04.445549+0000","last_fullsized":"2024-04-06T12:32:04.445549+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:12:29.374507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"157'1","reported_seq":12,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.445497+0000","last_change":"2024-04-06T12:32:02.676754+0000","last_active":"2024-04-06T12:32:04.445497+0000","last_peered":"2024-04-06T12:32:04.445497+0000","last_clean":"2024-04-06T12:32:04.445497+0000","last_became_active":"2024-04-06T12:32:02.676449+0000","last_became_peered":"2024-04-06T12:32:02.676449+0000","last_unstale":"2024-04-06T12:32:04.445497+0000","last_undegraded":"2024-04-06T12:32:04.445497+0000","last_fullsized":"2024-04-06T12:32:04.445497+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:37:59.756304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.445540+0000","last_change":"2024-04-06T12:32:03.090474+0000","last_active":"2024-04-06T12:32:04.445540+0000","last_peered":"2024-04-06T12:32:04.445540+0000","last_clean":"2024-04-06T12:32:04.445540+0000","last_became_active":"2024-04-06T12:32:03.090148+0000","last_became_peered":"2024-04-06T12:32:03.090148+0000","last_unstale":"2024-04-06T12:32:04.445540+0000","last_undegraded":"2024-04-06T12:32:04.445540+0000","last_fullsized":"2024-04-06T12:32:04.445540+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:12:19.356287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"157'1","reported_seq":12,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076272+0000","last_change":"2024-04-06T12:32:02.661929+0000","last_active":"2024-04-06T12:32:04.076272+0000","last_peered":"2024-04-06T12:32:04.076272+0000","last_clean":"2024-04-06T12:32:04.076272+0000","last_became_active":"2024-04-06T12:32:02.661649+0000","last_became_peered":"2024-04-06T12:32:02.661649+0000","last_unstale":"2024-04-06T12:32:04.076272+0000","last_undegraded":"2024-04-06T12:32:04.076272+0000","last_fullsized":"2024-04-06T12:32:04.076272+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:51:21.530690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077054+0000","last_change":"2024-04-06T12:32:03.102445+0000","last_active":"2024-04-06T12:32:04.077054+0000","last_peered":"2024-04-06T12:32:04.077054+0000","last_clean":"2024-04-06T12:32:04.077054+0000","last_became_active":"2024-04-06T12:32:03.102248+0000","last_became_peered":"2024-04-06T12:32:03.102248+0000","last_unstale":"2024-04-06T12:32:04.077054+0000","last_undegraded":"2024-04-06T12:32:04.077054+0000","last_fullsized":"2024-04-06T12:32:04.077054+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:13:01.541685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034132+0000","last_change":"2024-04-06T12:32:02.674195+0000","last_active":"2024-04-06T12:32:05.034132+0000","last_peered":"2024-04-06T12:32:05.034132+0000","last_clean":"2024-04-06T12:32:05.034132+0000","last_became_active":"2024-04-06T12:32:02.673397+0000","last_became_peered":"2024-04-06T12:32:02.673397+0000","last_unstale":"2024-04-06T12:32:05.034132+0000","last_undegraded":"2024-04-06T12:32:05.034132+0000","last_fullsized":"2024-04-06T12:32:05.034132+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:29:22.645084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077476+0000","last_change":"2024-04-06T12:32:03.096249+0000","last_active":"2024-04-06T12:32:04.077476+0000","last_peered":"2024-04-06T12:32:04.077476+0000","last_clean":"2024-04-06T12:32:04.077476+0000","last_became_active":"2024-04-06T12:32:03.095988+0000","last_became_peered":"2024-04-06T12:32:03.095988+0000","last_unstale":"2024-04-06T12:32:04.077476+0000","last_undegraded":"2024-04-06T12:32:04.077476+0000","last_fullsized":"2024-04-06T12:32:04.077476+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:31:48.007605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:32:01.643253+0000","last_change":"2024-04-06T12:32:01.643253+0000","last_active":"2024-04-06T12:32:01.643253+0000","last_peered":"2024-04-06T12:32:01.643253+0000","last_clean":"2024-04-06T12:32:01.643253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:32:01.643253+0000","last_undegraded":"2024-04-06T12:32:01.643253+0000","last_fullsized":"2024-04-06T12:32:01.643253+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.033854+0000","last_change":"2024-04-06T12:32:03.096606+0000","last_active":"2024-04-06T12:32:05.033854+0000","last_peered":"2024-04-06T12:32:05.033854+0000","last_clean":"2024-04-06T12:32:05.033854+0000","last_became_active":"2024-04-06T12:32:03.096302+0000","last_became_peered":"2024-04-06T12:32:03.096302+0000","last_unstale":"2024-04-06T12:32:05.033854+0000","last_undegraded":"2024-04-06T12:32:05.033854+0000","last_fullsized":"2024-04-06T12:32:05.033854+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:25:36.894614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034235+0000","last_change":"2024-04-06T12:32:02.672265+0000","last_active":"2024-04-06T12:32:05.034235+0000","last_peered":"2024-04-06T12:32:05.034235+0000","last_clean":"2024-04-06T12:32:05.034235+0000","last_became_active":"2024-04-06T12:32:02.671995+0000","last_became_peered":"2024-04-06T12:32:02.671995+0000","last_unstale":"2024-04-06T12:32:05.034235+0000","last_undegraded":"2024-04-06T12:32:05.034235+0000","last_fullsized":"2024-04-06T12:32:05.034235+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:20:14.679061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.033988+0000","last_change":"2024-04-06T12:32:03.082704+0000","last_active":"2024-04-06T12:32:05.033988+0000","last_peered":"2024-04-06T12:32:05.033988+0000","last_clean":"2024-04-06T12:32:05.033988+0000","last_became_active":"2024-04-06T12:32:03.082421+0000","last_became_peered":"2024-04-06T12:32:03.082421+0000","last_unstale":"2024-04-06T12:32:05.033988+0000","last_undegraded":"2024-04-06T12:32:05.033988+0000","last_fullsized":"2024-04-06T12:32:05.033988+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:32:44.600075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034510+0000","last_change":"2024-04-06T12:32:02.676453+0000","last_active":"2024-04-06T12:32:05.034510+0000","last_peered":"2024-04-06T12:32:05.034510+0000","last_clean":"2024-04-06T12:32:05.034510+0000","last_became_active":"2024-04-06T12:32:02.676191+0000","last_became_peered":"2024-04-06T12:32:02.676191+0000","last_unstale":"2024-04-06T12:32:05.034510+0000","last_undegraded":"2024-04-06T12:32:05.034510+0000","last_fullsized":"2024-04-06T12:32:05.034510+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:32:38.840045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:32:02.652297+0000","last_change":"2024-04-06T12:32:02.652297+0000","last_active":"2024-04-06T12:32:02.652297+0000","last_peered":"2024-04-06T12:32:02.652297+0000","last_clean":"2024-04-06T12:32:02.652297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:32:02.652297+0000","last_undegraded":"2024-04-06T12:32:02.652297+0000","last_fullsized":"2024-04-06T12:32:02.652297+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.15","version":"157'3","reported_seq":14,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077194+0000","last_change":"2024-04-06T12:32:02.672117+0000","last_active":"2024-04-06T12:32:04.077194+0000","last_peered":"2024-04-06T12:32:04.077194+0000","last_clean":"2024-04-06T12:32:04.077194+0000","last_became_active":"2024-04-06T12:32:02.671844+0000","last_became_peered":"2024-04-06T12:32:02.671844+0000","last_unstale":"2024-04-06T12:32:04.077194+0000","last_undegraded":"2024-04-06T12:32:04.077194+0000","last_fullsized":"2024-04-06T12:32:04.077194+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:55:47.076984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:32:02.652297+0000","last_change":"2024-04-06T12:32:02.652297+0000","last_active":"2024-04-06T12:32:02.652297+0000","last_peered":"2024-04-06T12:32:02.652297+0000","last_clean":"2024-04-06T12:32:02.652297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:32:02.652297+0000","last_undegraded":"2024-04-06T12:32:02.652297+0000","last_fullsized":"2024-04-06T12:32:02.652297+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.14","version":"157'6","reported_seq":17,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077323+0000","last_change":"2024-04-06T12:32:02.672984+0000","last_active":"2024-04-06T12:32:04.077323+0000","last_peered":"2024-04-06T12:32:04.077323+0000","last_clean":"2024-04-06T12:32:04.077323+0000","last_became_active":"2024-04-06T12:32:02.672507+0000","last_became_peered":"2024-04-06T12:32:02.672507+0000","last_unstale":"2024-04-06T12:32:04.077323+0000","last_undegraded":"2024-04-06T12:32:04.077323+0000","last_fullsized":"2024-04-06T12:32:04.077323+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:49:55.738902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076779+0000","last_change":"2024-04-06T12:32:03.081118+0000","last_active":"2024-04-06T12:32:04.076779+0000","last_peered":"2024-04-06T12:32:04.076779+0000","last_clean":"2024-04-06T12:32:04.076779+0000","last_became_active":"2024-04-06T12:32:03.080855+0000","last_became_peered":"2024-04-06T12:32:03.080855+0000","last_unstale":"2024-04-06T12:32:04.076779+0000","last_undegraded":"2024-04-06T12:32:04.076779+0000","last_fullsized":"2024-04-06T12:32:04.076779+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:26:50.050125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"157'2","reported_seq":13,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034691+0000","last_change":"2024-04-06T12:32:02.676947+0000","last_active":"2024-04-06T12:32:05.034691+0000","last_peered":"2024-04-06T12:32:05.034691+0000","last_clean":"2024-04-06T12:32:05.034691+0000","last_became_active":"2024-04-06T12:32:02.676514+0000","last_became_peered":"2024-04-06T12:32:02.676514+0000","last_unstale":"2024-04-06T12:32:05.034691+0000","last_undegraded":"2024-04-06T12:32:05.034691+0000","last_fullsized":"2024-04-06T12:32:05.034691+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:35:45.136425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077438+0000","last_change":"2024-04-06T12:32:03.089750+0000","last_active":"2024-04-06T12:32:04.077438+0000","last_peered":"2024-04-06T12:32:04.077438+0000","last_clean":"2024-04-06T12:32:04.077438+0000","last_became_active":"2024-04-06T12:32:03.089323+0000","last_became_peered":"2024-04-06T12:32:03.089323+0000","last_unstale":"2024-04-06T12:32:04.077438+0000","last_undegraded":"2024-04-06T12:32:04.077438+0000","last_fullsized":"2024-04-06T12:32:04.077438+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:14:34.054920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:32:01.643253+0000","last_change":"2024-04-06T12:32:01.643253+0000","last_active":"2024-04-06T12:32:01.643253+0000","last_peered":"2024-04-06T12:32:01.643253+0000","last_clean":"2024-04-06T12:32:01.643253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:32:01.643253+0000","last_undegraded":"2024-04-06T12:32:01.643253+0000","last_fullsized":"2024-04-06T12:32:01.643253+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077263+0000","last_change":"2024-04-06T12:32:03.090934+0000","last_active":"2024-04-06T12:32:04.077263+0000","last_peered":"2024-04-06T12:32:04.077263+0000","last_clean":"2024-04-06T12:32:04.077263+0000","last_became_active":"2024-04-06T12:32:03.090284+0000","last_became_peered":"2024-04-06T12:32:03.090284+0000","last_unstale":"2024-04-06T12:32:04.077263+0000","last_undegraded":"2024-04-06T12:32:04.077263+0000","last_fullsized":"2024-04-06T12:32:04.077263+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:21:11.872887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"157'8","reported_seq":19,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034740+0000","last_change":"2024-04-06T12:32:02.673648+0000","last_active":"2024-04-06T12:32:05.034740+0000","last_peered":"2024-04-06T12:32:05.034740+0000","last_clean":"2024-04-06T12:32:05.034740+0000","last_became_active":"2024-04-06T12:32:02.673372+0000","last_became_peered":"2024-04-06T12:32:02.673372+0000","last_unstale":"2024-04-06T12:32:05.034740+0000","last_undegraded":"2024-04-06T12:32:05.034740+0000","last_fullsized":"2024-04-06T12:32:05.034740+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:09:01.401665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:32:02.652297+0000","last_change":"2024-04-06T12:32:02.652297+0000","last_active":"2024-04-06T12:32:02.652297+0000","last_peered":"2024-04-06T12:32:02.652297+0000","last_clean":"2024-04-06T12:32:02.652297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:32:02.652297+0000","last_undegraded":"2024-04-06T12:32:02.652297+0000","last_fullsized":"2024-04-06T12:32:02.652297+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.f","version":"157'1","reported_seq":12,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034295+0000","last_change":"2024-04-06T12:32:02.676442+0000","last_active":"2024-04-06T12:32:05.034295+0000","last_peered":"2024-04-06T12:32:05.034295+0000","last_clean":"2024-04-06T12:32:05.034295+0000","last_became_active":"2024-04-06T12:32:02.676164+0000","last_became_peered":"2024-04-06T12:32:02.676164+0000","last_unstale":"2024-04-06T12:32:05.034295+0000","last_undegraded":"2024-04-06T12:32:05.034295+0000","last_fullsized":"2024-04-06T12:32:05.034295+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:41:59.741918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076562+0000","last_change":"2024-04-06T12:32:03.094666+0000","last_active":"2024-04-06T12:32:04.076562+0000","last_peered":"2024-04-06T12:32:04.076562+0000","last_clean":"2024-04-06T12:32:04.076562+0000","last_became_active":"2024-04-06T12:32:03.090349+0000","last_became_peered":"2024-04-06T12:32:03.090349+0000","last_unstale":"2024-04-06T12:32:04.076562+0000","last_undegraded":"2024-04-06T12:32:04.076562+0000","last_fullsized":"2024-04-06T12:32:04.076562+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:17:55.389634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:32:01.643253+0000","last_change":"2024-04-06T12:32:01.643253+0000","last_active":"2024-04-06T12:32:01.643253+0000","last_peered":"2024-04-06T12:32:01.643253+0000","last_clean":"2024-04-06T12:32:01.643253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:32:01.643253+0000","last_undegraded":"2024-04-06T12:32:01.643253+0000","last_fullsized":"2024-04-06T12:32:01.643253+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076733+0000","last_change":"2024-04-06T12:32:03.095614+0000","last_active":"2024-04-06T12:32:04.076733+0000","last_peered":"2024-04-06T12:32:04.076733+0000","last_clean":"2024-04-06T12:32:04.076733+0000","last_became_active":"2024-04-06T12:32:03.095324+0000","last_became_peered":"2024-04-06T12:32:03.095324+0000","last_unstale":"2024-04-06T12:32:04.076733+0000","last_undegraded":"2024-04-06T12:32:04.076733+0000","last_fullsized":"2024-04-06T12:32:04.076733+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:48:06.372887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"157'1","reported_seq":12,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034386+0000","last_change":"2024-04-06T12:32:02.673592+0000","last_active":"2024-04-06T12:32:05.034386+0000","last_peered":"2024-04-06T12:32:05.034386+0000","last_clean":"2024-04-06T12:32:05.034386+0000","last_became_active":"2024-04-06T12:32:02.673228+0000","last_became_peered":"2024-04-06T12:32:02.673228+0000","last_unstale":"2024-04-06T12:32:05.034386+0000","last_undegraded":"2024-04-06T12:32:05.034386+0000","last_fullsized":"2024-04-06T12:32:05.034386+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:37:59.756304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076157+0000","last_change":"2024-04-06T12:32:03.091263+0000","last_active":"2024-04-06T12:32:04.076157+0000","last_peered":"2024-04-06T12:32:04.076157+0000","last_clean":"2024-04-06T12:32:04.076157+0000","last_became_active":"2024-04-06T12:32:03.090998+0000","last_became_peered":"2024-04-06T12:32:03.090998+0000","last_unstale":"2024-04-06T12:32:04.076157+0000","last_undegraded":"2024-04-06T12:32:04.076157+0000","last_fullsized":"2024-04-06T12:32:04.076157+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:04:54.609161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076682+0000","last_change":"2024-04-06T12:32:02.666017+0000","last_active":"2024-04-06T12:32:04.076682+0000","last_peered":"2024-04-06T12:32:04.076682+0000","last_clean":"2024-04-06T12:32:04.076682+0000","last_became_active":"2024-04-06T12:32:02.665597+0000","last_became_peered":"2024-04-06T12:32:02.665597+0000","last_unstale":"2024-04-06T12:32:04.076682+0000","last_undegraded":"2024-04-06T12:32:04.076682+0000","last_fullsized":"2024-04-06T12:32:04.076682+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:34:34.986435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076608+0000","last_change":"2024-04-06T12:32:03.087353+0000","last_active":"2024-04-06T12:32:04.076608+0000","last_peered":"2024-04-06T12:32:04.076608+0000","last_clean":"2024-04-06T12:32:04.076608+0000","last_became_active":"2024-04-06T12:32:03.087041+0000","last_became_peered":"2024-04-06T12:32:03.087041+0000","last_unstale":"2024-04-06T12:32:04.076608+0000","last_undegraded":"2024-04-06T12:32:04.076608+0000","last_fullsized":"2024-04-06T12:32:04.076608+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:38:07.718139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"157'2","reported_seq":13,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.445540+0000","last_change":"2024-04-06T12:32:02.671169+0000","last_active":"2024-04-06T12:32:04.445540+0000","last_peered":"2024-04-06T12:32:04.445540+0000","last_clean":"2024-04-06T12:32:04.445540+0000","last_became_active":"2024-04-06T12:32:02.670908+0000","last_became_peered":"2024-04-06T12:32:02.670908+0000","last_unstale":"2024-04-06T12:32:04.445540+0000","last_undegraded":"2024-04-06T12:32:04.445540+0000","last_fullsized":"2024-04-06T12:32:04.445540+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:51:36.175382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034719+0000","last_change":"2024-04-06T12:32:03.099004+0000","last_active":"2024-04-06T12:32:05.034719+0000","last_peered":"2024-04-06T12:32:05.034719+0000","last_clean":"2024-04-06T12:32:05.034719+0000","last_became_active":"2024-04-06T12:32:03.098817+0000","last_became_peered":"2024-04-06T12:32:03.098817+0000","last_unstale":"2024-04-06T12:32:05.034719+0000","last_undegraded":"2024-04-06T12:32:05.034719+0000","last_fullsized":"2024-04-06T12:32:05.034719+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:20:15.688105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034780+0000","last_change":"2024-04-06T12:32:02.673048+0000","last_active":"2024-04-06T12:32:05.034780+0000","last_peered":"2024-04-06T12:32:05.034780+0000","last_clean":"2024-04-06T12:32:05.034780+0000","last_became_active":"2024-04-06T12:32:02.672698+0000","last_became_peered":"2024-04-06T12:32:02.672698+0000","last_unstale":"2024-04-06T12:32:05.034780+0000","last_undegraded":"2024-04-06T12:32:05.034780+0000","last_fullsized":"2024-04-06T12:32:05.034780+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:34:37.245104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077101+0000","last_change":"2024-04-06T12:32:03.096836+0000","last_active":"2024-04-06T12:32:04.077101+0000","last_peered":"2024-04-06T12:32:04.077101+0000","last_clean":"2024-04-06T12:32:04.077101+0000","last_became_active":"2024-04-06T12:32:03.096608+0000","last_became_peered":"2024-04-06T12:32:03.096608+0000","last_unstale":"2024-04-06T12:32:04.077101+0000","last_undegraded":"2024-04-06T12:32:04.077101+0000","last_fullsized":"2024-04-06T12:32:04.077101+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:35:14.481710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076714+0000","last_change":"2024-04-06T12:32:02.675977+0000","last_active":"2024-04-06T12:32:04.076714+0000","last_peered":"2024-04-06T12:32:04.076714+0000","last_clean":"2024-04-06T12:32:04.076714+0000","last_became_active":"2024-04-06T12:32:02.675688+0000","last_became_peered":"2024-04-06T12:32:02.675688+0000","last_unstale":"2024-04-06T12:32:04.076714+0000","last_undegraded":"2024-04-06T12:32:04.076714+0000","last_fullsized":"2024-04-06T12:32:04.076714+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:09:01.401665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034233+0000","last_change":"2024-04-06T12:32:03.092794+0000","last_active":"2024-04-06T12:32:05.034233+0000","last_peered":"2024-04-06T12:32:05.034233+0000","last_clean":"2024-04-06T12:32:05.034233+0000","last_became_active":"2024-04-06T12:32:03.092504+0000","last_became_peered":"2024-04-06T12:32:03.092504+0000","last_unstale":"2024-04-06T12:32:05.034233+0000","last_undegraded":"2024-04-06T12:32:05.034233+0000","last_fullsized":"2024-04-06T12:32:05.034233+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:20:54.013729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034565+0000","last_change":"2024-04-06T12:32:02.676790+0000","last_active":"2024-04-06T12:32:05.034565+0000","last_peered":"2024-04-06T12:32:05.034565+0000","last_clean":"2024-04-06T12:32:05.034565+0000","last_became_active":"2024-04-06T12:32:02.676256+0000","last_became_peered":"2024-04-06T12:32:02.676256+0000","last_unstale":"2024-04-06T12:32:05.034565+0000","last_undegraded":"2024-04-06T12:32:05.034565+0000","last_fullsized":"2024-04-06T12:32:05.034565+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:38:36.953076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076117+0000","last_change":"2024-04-06T12:32:03.107649+0000","last_active":"2024-04-06T12:32:04.076117+0000","last_peered":"2024-04-06T12:32:04.076117+0000","last_clean":"2024-04-06T12:32:04.076117+0000","last_became_active":"2024-04-06T12:32:03.107392+0000","last_became_peered":"2024-04-06T12:32:03.107392+0000","last_unstale":"2024-04-06T12:32:04.076117+0000","last_undegraded":"2024-04-06T12:32:04.076117+0000","last_fullsized":"2024-04-06T12:32:04.076117+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:09:37.978340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"157'2","reported_seq":13,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034852+0000","last_change":"2024-04-06T12:32:02.673512+0000","last_active":"2024-04-06T12:32:05.034852+0000","last_peered":"2024-04-06T12:32:05.034852+0000","last_clean":"2024-04-06T12:32:05.034852+0000","last_became_active":"2024-04-06T12:32:02.673277+0000","last_became_peered":"2024-04-06T12:32:02.673277+0000","last_unstale":"2024-04-06T12:32:05.034852+0000","last_undegraded":"2024-04-06T12:32:05.034852+0000","last_fullsized":"2024-04-06T12:32:05.034852+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:22:26.422194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:32:02.652297+0000","last_change":"2024-04-06T12:32:02.652297+0000","last_active":"2024-04-06T12:32:02.652297+0000","last_peered":"2024-04-06T12:32:02.652297+0000","last_clean":"2024-04-06T12:32:02.652297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:32:02.652297+0000","last_undegraded":"2024-04-06T12:32:02.652297+0000","last_fullsized":"2024-04-06T12:32:02.652297+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.445708+0000","last_change":"2024-04-06T12:32:02.671193+0000","last_active":"2024-04-06T12:32:04.445708+0000","last_peered":"2024-04-06T12:32:04.445708+0000","last_clean":"2024-04-06T12:32:04.445708+0000","last_became_active":"2024-04-06T12:32:02.670946+0000","last_became_peered":"2024-04-06T12:32:02.670946+0000","last_unstale":"2024-04-06T12:32:04.445708+0000","last_undegraded":"2024-04-06T12:32:04.445708+0000","last_fullsized":"2024-04-06T12:32:04.445708+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:18:36.622724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076843+0000","last_change":"2024-04-06T12:32:03.104549+0000","last_active":"2024-04-06T12:32:04.076843+0000","last_peered":"2024-04-06T12:32:04.076843+0000","last_clean":"2024-04-06T12:32:04.076843+0000","last_became_active":"2024-04-06T12:32:03.104254+0000","last_became_peered":"2024-04-06T12:32:03.104254+0000","last_unstale":"2024-04-06T12:32:04.076843+0000","last_undegraded":"2024-04-06T12:32:04.076843+0000","last_fullsized":"2024-04-06T12:32:04.076843+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:25:42.693508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.445634+0000","last_change":"2024-04-06T12:32:02.663784+0000","last_active":"2024-04-06T12:32:04.445634+0000","last_peered":"2024-04-06T12:32:04.445634+0000","last_clean":"2024-04-06T12:32:04.445634+0000","last_became_active":"2024-04-06T12:32:02.663498+0000","last_became_peered":"2024-04-06T12:32:02.663498+0000","last_unstale":"2024-04-06T12:32:04.445634+0000","last_undegraded":"2024-04-06T12:32:04.445634+0000","last_fullsized":"2024-04-06T12:32:04.445634+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:55:47.076984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076242+0000","last_change":"2024-04-06T12:32:03.082479+0000","last_active":"2024-04-06T12:32:04.076242+0000","last_peered":"2024-04-06T12:32:04.076242+0000","last_clean":"2024-04-06T12:32:04.076242+0000","last_became_active":"2024-04-06T12:32:03.082240+0000","last_became_peered":"2024-04-06T12:32:03.082240+0000","last_unstale":"2024-04-06T12:32:04.076242+0000","last_undegraded":"2024-04-06T12:32:04.076242+0000","last_fullsized":"2024-04-06T12:32:04.076242+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:35.486850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034140+0000","last_change":"2024-04-06T12:32:02.673043+0000","last_active":"2024-04-06T12:32:05.034140+0000","last_peered":"2024-04-06T12:32:05.034140+0000","last_clean":"2024-04-06T12:32:05.034140+0000","last_became_active":"2024-04-06T12:32:02.672502+0000","last_became_peered":"2024-04-06T12:32:02.672502+0000","last_unstale":"2024-04-06T12:32:05.034140+0000","last_undegraded":"2024-04-06T12:32:05.034140+0000","last_fullsized":"2024-04-06T12:32:05.034140+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:18:36.622724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034072+0000","last_change":"2024-04-06T12:32:03.086348+0000","last_active":"2024-04-06T12:32:05.034072+0000","last_peered":"2024-04-06T12:32:05.034072+0000","last_clean":"2024-04-06T12:32:05.034072+0000","last_became_active":"2024-04-06T12:32:03.086093+0000","last_became_peered":"2024-04-06T12:32:03.086093+0000","last_unstale":"2024-04-06T12:32:05.034072+0000","last_undegraded":"2024-04-06T12:32:05.034072+0000","last_fullsized":"2024-04-06T12:32:05.034072+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:14:34.054920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.033990+0000","last_change":"2024-04-06T12:32:02.675255+0000","last_active":"2024-04-06T12:32:05.033990+0000","last_peered":"2024-04-06T12:32:05.033990+0000","last_clean":"2024-04-06T12:32:05.033990+0000","last_became_active":"2024-04-06T12:32:02.675016+0000","last_became_peered":"2024-04-06T12:32:02.675016+0000","last_unstale":"2024-04-06T12:32:05.033990+0000","last_undegraded":"2024-04-06T12:32:05.033990+0000","last_fullsized":"2024-04-06T12:32:05.033990+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:04:32.029292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034254+0000","last_change":"2024-04-06T12:32:03.081731+0000","last_active":"2024-04-06T12:32:05.034254+0000","last_peered":"2024-04-06T12:32:05.034254+0000","last_clean":"2024-04-06T12:32:05.034254+0000","last_became_active":"2024-04-06T12:32:03.081462+0000","last_became_peered":"2024-04-06T12:32:03.081462+0000","last_unstale":"2024-04-06T12:32:05.034254+0000","last_undegraded":"2024-04-06T12:32:05.034254+0000","last_fullsized":"2024-04-06T12:32:05.034254+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:54:15.592322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077083+0000","last_change":"2024-04-06T12:32:02.667429+0000","last_active":"2024-04-06T12:32:04.077083+0000","last_peered":"2024-04-06T12:32:04.077083+0000","last_clean":"2024-04-06T12:32:04.077083+0000","last_became_active":"2024-04-06T12:32:02.667155+0000","last_became_peered":"2024-04-06T12:32:02.667155+0000","last_unstale":"2024-04-06T12:32:04.077083+0000","last_undegraded":"2024-04-06T12:32:04.077083+0000","last_fullsized":"2024-04-06T12:32:04.077083+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:58:22.678856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034580+0000","last_change":"2024-04-06T12:32:03.095696+0000","last_active":"2024-04-06T12:32:05.034580+0000","last_peered":"2024-04-06T12:32:05.034580+0000","last_clean":"2024-04-06T12:32:05.034580+0000","last_became_active":"2024-04-06T12:32:03.095431+0000","last_became_peered":"2024-04-06T12:32:03.095431+0000","last_unstale":"2024-04-06T12:32:05.034580+0000","last_undegraded":"2024-04-06T12:32:05.034580+0000","last_fullsized":"2024-04-06T12:32:05.034580+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:11:29.631951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034429+0000","last_change":"2024-04-06T12:32:02.674058+0000","last_active":"2024-04-06T12:32:05.034429+0000","last_peered":"2024-04-06T12:32:05.034429+0000","last_clean":"2024-04-06T12:32:05.034429+0000","last_became_active":"2024-04-06T12:32:02.673810+0000","last_became_peered":"2024-04-06T12:32:02.673810+0000","last_unstale":"2024-04-06T12:32:05.034429+0000","last_undegraded":"2024-04-06T12:32:05.034429+0000","last_fullsized":"2024-04-06T12:32:05.034429+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:58:22.678856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077490+0000","last_change":"2024-04-06T12:32:03.101458+0000","last_active":"2024-04-06T12:32:04.077490+0000","last_peered":"2024-04-06T12:32:04.077490+0000","last_clean":"2024-04-06T12:32:04.077490+0000","last_became_active":"2024-04-06T12:32:03.099130+0000","last_became_peered":"2024-04-06T12:32:03.099130+0000","last_unstale":"2024-04-06T12:32:04.077490+0000","last_undegraded":"2024-04-06T12:32:04.077490+0000","last_fullsized":"2024-04-06T12:32:04.077490+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:32:44.600075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076266+0000","last_change":"2024-04-06T12:32:02.673686+0000","last_active":"2024-04-06T12:32:04.076266+0000","last_peered":"2024-04-06T12:32:04.076266+0000","last_clean":"2024-04-06T12:32:04.076266+0000","last_became_active":"2024-04-06T12:32:02.673427+0000","last_became_peered":"2024-04-06T12:32:02.673427+0000","last_unstale":"2024-04-06T12:32:04.076266+0000","last_undegraded":"2024-04-06T12:32:04.076266+0000","last_fullsized":"2024-04-06T12:32:04.076266+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:02:06.901506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T12:32:02.652297+0000","last_change":"2024-04-06T12:32:02.652297+0000","last_active":"2024-04-06T12:32:02.652297+0000","last_peered":"2024-04-06T12:32:02.652297+0000","last_clean":"2024-04-06T12:32:02.652297+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T12:32:02.652297+0000","last_undegraded":"2024-04-06T12:32:02.652297+0000","last_fullsized":"2024-04-06T12:32:02.652297+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076938+0000","last_change":"2024-04-06T12:32:02.675915+0000","last_active":"2024-04-06T12:32:04.076938+0000","last_peered":"2024-04-06T12:32:04.076938+0000","last_clean":"2024-04-06T12:32:04.076938+0000","last_became_active":"2024-04-06T12:32:02.675690+0000","last_became_peered":"2024-04-06T12:32:02.675690+0000","last_unstale":"2024-04-06T12:32:04.076938+0000","last_undegraded":"2024-04-06T12:32:04.076938+0000","last_fullsized":"2024-04-06T12:32:04.076938+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:38:36.953076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"147'249","reported_seq":779,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-06T12:32:03.081691+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:32:03.081691+0000","last_peered":"2024-04-06T12:32:03.081691+0000","last_clean":"2024-04-06T12:32:03.081691+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:32:03.081691+0000","last_undegraded":"2024-04-06T12:32:03.081691+0000","last_fullsized":"2024-04-06T12:32:03.081691+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":108,"acting":108,"num_store_stats":7},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":1780,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":56,"acting":56,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575085,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59876,"kb_used_data":2520,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311964,"statfs":{"total":96636764160,"available":96575451136,"internally_reserved":0,"allocated":2580480,"data_stored":1582157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52300000000000002,"15min":0.55000000000000004},"min":{"1min":0.249,"5min":0.249,"15min":0.219},"max":{"1min":0.71999999999999997,"5min":0.78200000000000003,"15min":1.143},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.53100000000000003,"15min":0.54900000000000004},"min":{"1min":0.34100000000000003,"5min":0.246,"15min":0.246},"max":{"1min":0.69299999999999995,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.54400000000000004}]},{"osd":1,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53600000000000003,"15min":0.55300000000000005},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.753,"5min":0.753,"15min":0.79100000000000004},"last":0.374},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54200000000000004,"15min":0.56499999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.27900000000000003},"max":{"1min":0.67700000000000005,"5min":0.754,"15min":0.79600000000000004},"last":0.47699999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.55000000000000004},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.249},"max":{"1min":0.73199999999999998,"5min":0.76500000000000001,"15min":0.83299999999999996},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56599999999999995,"15min":0.57699999999999996},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.73899999999999999,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.57199999999999995}]},{"osd":3,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54500000000000004,"15min":0.57099999999999995},"min":{"1min":0.41199999999999998,"5min":0.35099999999999998,"15min":0.34399999999999997},"max":{"1min":0.68799999999999994,"5min":0.73999999999999999,"15min":1.357},"last":0.502},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57199999999999995,"15min":0.58799999999999997},"min":{"1min":0.36899999999999999,"5min":0.35599999999999998,"15min":0.33100000000000002},"max":{"1min":0.71999999999999997,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.70499999999999996}]},{"osd":4,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.44600000000000001,"15min":0.44500000000000001},"min":{"1min":0.28100000000000003,"5min":0.27000000000000002,"15min":0.25},"max":{"1min":0.63600000000000001,"5min":0.71399999999999997,"15min":0.72999999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.52900000000000003,"15min":0.52800000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.32800000000000001},"max":{"1min":0.72999999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.63500000000000001}]},{"osd":5,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.50600000000000001,"15min":0.498},"min":{"1min":0.39800000000000002,"5min":0.29399999999999998,"15min":0.26600000000000001},"max":{"1min":0.78500000000000003,"5min":0.81899999999999995,"15min":0.86099999999999999},"last":0.755},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.40999999999999998,"5min":0.32800000000000001,"15min":0.314},"max":{"1min":0.74099999999999999,"5min":0.79900000000000004,"15min":0.81599999999999995},"last":0.67500000000000004}]},{"osd":6,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.84499999999999997,"15min":0.88600000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56699999999999995,"15min":0.56299999999999994},"min":{"1min":0.40699999999999997,"5min":0.312,"15min":0.30399999999999999},"max":{"1min":0.73299999999999998,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.72499999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32992,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1916928,"data_stored":962677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.52100000000000002,"15min":0.53700000000000003},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.26200000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.82099999999999995},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52200000000000002,"15min":0.53700000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.245},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.502}]},{"osd":1,"last update":"Sat Apr 6 12:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.55100000000000005,"15min":0.58199999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.59999999999999998},"min":{"1min":0.40899999999999997,"5min":0.38400000000000001,"15min":0.221},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.57099999999999995}]},{"osd":2,"last update":"Sat Apr 6 12:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54200000000000004,"15min":0.55300000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.30499999999999999},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54900000000000004,"15min":0.56799999999999995},"min":{"1min":0.378,"5min":0.32300000000000001,"15min":0.254},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Apr 6 12:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55800000000000005,"15min":0.57799999999999996},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.23300000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.61799999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.58099999999999996}]},{"osd":4,"last update":"Sat Apr 6 12:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52000000000000002,"15min":0.501},"min":{"1min":0.34599999999999997,"5min":0.28599999999999998,"15min":0.27500000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.621},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54700000000000004,"15min":0.53300000000000003},"min":{"1min":0.34899999999999998,"5min":0.29799999999999999,"15min":0.28799999999999998},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.82199999999999995},"last":0.41899999999999998}]},{"osd":5,"last update":"Sat Apr 6 12:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50900000000000001,"15min":0.49399999999999999},"min":{"1min":0.30399999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.70899999999999996,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52600000000000002,"15min":0.51300000000000001},"min":{"1min":0.313,"5min":0.31,"15min":0.27000000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.53900000000000003}]},{"osd":7,"last update":"Sat Apr 6 12:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55200000000000005,"15min":0.53900000000000003},"min":{"1min":0.36099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.75900000000000001,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57899999999999996,"15min":0.56299999999999994},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.32100000000000001},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.64900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1970176,"data_stored":989186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.35999999999999999,"15min":0.36299999999999999},"min":{"1min":0.27800000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.53700000000000003,"5min":0.55400000000000005,"15min":0.58999999999999997},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.377,"15min":0.39200000000000002},"min":{"1min":0.26800000000000002,"5min":0.245,"15min":0.215},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.27800000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.38,"15min":0.39300000000000002},"min":{"1min":0.27500000000000002,"5min":0.25,"15min":0.19400000000000001},"max":{"1min":0.49199999999999999,"5min":0.58099999999999996,"15min":0.67600000000000005},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.42899999999999999},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.57099999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.39200000000000002}]},{"osd":3,"last update":"Sat Apr 6 12:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.44700000000000001},"min":{"1min":0.314,"5min":0.29199999999999998,"15min":0.28399999999999997},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.41699999999999998},"min":{"1min":0.29799999999999999,"5min":0.29099999999999998,"15min":0.249},"max":{"1min":0.55200000000000005,"5min":0.65900000000000003,"15min":0.86299999999999999},"last":0.35799999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58899999999999997,"15min":0.61499999999999999},"min":{"1min":0.45500000000000002,"5min":0.38900000000000001,"15min":0.34799999999999998},"max":{"1min":0.73699999999999999,"5min":0.872,"15min":0.872},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.437,"5min":0.36299999999999999,"15min":0.30399999999999999},"max":{"1min":0.81399999999999995,"5min":0.93799999999999994,"15min":0.94899999999999995},"last":0.496}]},{"osd":5,"last update":"Sat Apr 6 12:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60099999999999998,"15min":0.61499999999999999},"min":{"1min":0.505,"5min":0.41999999999999998,"15min":0.38700000000000001},"max":{"1min":0.79000000000000004,"5min":0.83999999999999997,"15min":0.91300000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.625,"5min":0.61399999999999999,"15min":0.63},"min":{"1min":0.45400000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.87,"5min":0.871,"15min":0.999},"last":0.47999999999999998}]},{"osd":6,"last update":"Sat Apr 6 12:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.60399999999999998,"15min":0.61899999999999999},"min":{"1min":0.44600000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81499999999999995,"5min":0.82599999999999996,"15min":0.98599999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.627,"15min":0.64800000000000002},"min":{"1min":0.48399999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81699999999999995,"5min":0.84799999999999998,"15min":0.92600000000000005},"last":0.58799999999999997}]},{"osd":7,"last update":"Sat Apr 6 12:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.64100000000000001,"15min":0.64400000000000002},"min":{"1min":0.437,"5min":0.437,"15min":0.38800000000000001},"max":{"1min":0.82999999999999996,"5min":0.85699999999999998,"15min":1.0329999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.61299999999999999,"15min":0.63600000000000001},"min":{"1min":0.436,"5min":0.40999999999999998,"15min":0.40200000000000002},"max":{"1min":0.85599999999999998,"5min":0.91400000000000003,"15min":0.95599999999999996},"last":0.50800000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575083,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37732,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334108,"statfs":{"total":96636764160,"available":96598126592,"internally_reserved":0,"allocated":2568192,"data_stored":1579540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 12:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35799999999999998,"15min":0.38100000000000001},"min":{"1min":0.26000000000000001,"5min":0.216,"15min":0.19},"max":{"1min":0.495,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.434},{"interface":"front","average":{"1min":0.371,"5min":0.376,"15min":0.40500000000000003},"min":{"1min":0.255,"5min":0.23499999999999999,"15min":0.217},"max":{"1min":0.55600000000000005,"5min":0.625,"15min":1.4359999999999999},"last":0.378}]},{"osd":2,"last update":"Sat Apr 6 12:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.39200000000000002,"15min":0.40899999999999997},"min":{"1min":0.315,"5min":0.255,"15min":0.19700000000000001},"max":{"1min":0.58599999999999997,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.40899999999999997,"15min":0.42999999999999999},"min":{"1min":0.28499999999999998,"5min":0.26800000000000002,"15min":0.24299999999999999},"max":{"1min":0.47099999999999997,"5min":0.89900000000000002,"15min":1.4790000000000001},"last":0.45400000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.39600000000000002,"15min":0.41699999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.252},"max":{"1min":0.51300000000000001,"5min":0.64300000000000002,"15min":1.25},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.432,"15min":0.46000000000000002},"min":{"1min":0.35299999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":0.74099999999999999,"15min":0.748},"last":0.51300000000000001}]},{"osd":4,"last update":"Sat Apr 6 12:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.58499999999999996,"15min":0.622},"min":{"1min":0.37,"5min":0.37,"15min":0.35699999999999998},"max":{"1min":0.65400000000000003,"5min":1.1100000000000001,"15min":1.514},"last":0.499},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58699999999999997,"15min":0.629},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72299999999999998,"5min":0.93999999999999995,"15min":1.202},"last":0.62}]},{"osd":5,"last update":"Sat Apr 6 12:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.57899999999999996,"15min":0.623},"min":{"1min":0.40999999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76500000000000001,"5min":0.88400000000000001,"15min":1.0920000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60899999999999999,"15min":0.64400000000000002},"min":{"1min":0.42799999999999999,"5min":0.40899999999999997,"15min":0.315},"max":{"1min":0.72999999999999998,"5min":1.006,"15min":1.006},"last":0.63600000000000001}]},{"osd":6,"last update":"Sat Apr 6 12:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.60599999999999998,"15min":0.64900000000000002},"min":{"1min":0.442,"5min":0.376,"15min":0.376},"max":{"1min":0.69199999999999995,"5min":1.038,"15min":1.038},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61399999999999999,"15min":0.66000000000000003},"min":{"1min":0.47099999999999997,"5min":0.40300000000000002,"15min":0.38100000000000001},"max":{"1min":0.74399999999999999,"5min":1.052,"15min":1.613},"last":0.55800000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.61199999999999999,"15min":0.66200000000000003},"min":{"1min":0.441,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.72099999999999997,"5min":1.0820000000000001,"15min":1.54},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60699999999999998,"15min":0.65700000000000003},"min":{"1min":0.41699999999999998,"5min":0.39300000000000002,"15min":0.38400000000000001},"max":{"1min":0.78300000000000003,"5min":1.014,"15min":1.498},"last":0.52200000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575085,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55176,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316664,"statfs":{"total":96636764160,"available":96580263936,"internally_reserved":0,"allocated":1961984,"data_stored":983112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36299999999999999,"15min":0.38200000000000001},"min":{"1min":0.23799999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.55100000000000005,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.38600000000000001,"15min":0.40600000000000003},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.55600000000000005,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.26900000000000002}]},{"osd":1,"last update":"Sat Apr 6 12:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.373,"15min":0.39700000000000002},"min":{"1min":0.222,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54000000000000004,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.40500000000000003,"15min":0.42699999999999999},"min":{"1min":0.29399999999999998,"5min":0.28000000000000003,"15min":0.27100000000000002},"max":{"1min":0.60399999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.39500000000000002}]},{"osd":3,"last update":"Sat Apr 6 12:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42099999999999999,"15min":0.44600000000000001},"min":{"1min":0.28799999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.52400000000000002,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41999999999999998,"15min":0.436},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":0.97099999999999997,"15min":1.0549999999999999},"last":0.40999999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58499999999999996,"15min":0.60999999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.35199999999999998},"max":{"1min":0.82099999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57999999999999996,"15min":0.62},"min":{"1min":0.40100000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77300000000000002,"5min":0.91200000000000003,"15min":0.92600000000000005},"last":0.65900000000000003}]},{"osd":5,"last update":"Sat Apr 6 12:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.58699999999999997,"15min":0.63},"min":{"1min":0.377,"5min":0.374,"15min":0.374},"max":{"1min":0.68200000000000005,"5min":1.0369999999999999,"15min":1.6870000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.59799999999999998,"15min":0.623},"min":{"1min":0.42199999999999999,"5min":0.35399999999999998,"15min":0.31},"max":{"1min":0.81000000000000005,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.68100000000000005}]},{"osd":6,"last update":"Sat Apr 6 12:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.59899999999999998,"15min":0.63},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.36399999999999999},"max":{"1min":0.79900000000000004,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.59899999999999998,"15min":0.64500000000000002},"min":{"1min":0.41199999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71399999999999997,"5min":1.0760000000000001,"15min":1.728},"last":0.80600000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59899999999999998,"15min":0.64100000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79400000000000004,"5min":0.95699999999999996,"15min":0.95799999999999996},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.61699999999999999,"15min":0.65700000000000003},"min":{"1min":0.47999999999999998,"5min":0.41499999999999998,"15min":0.39000000000000001},"max":{"1min":0.76000000000000001,"5min":1.0049999999999999,"15min":1.758},"last":0.78500000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575084,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1986560,"data_stored":991769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35799999999999998,"15min":0.36599999999999999},"min":{"1min":0.246,"5min":0.23599999999999999,"15min":0.21099999999999999},"max":{"1min":0.52600000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.434},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.35799999999999998,"15min":0.379},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.502,"5min":0.59099999999999997,"15min":0.73499999999999999},"last":0.40500000000000003}]},{"osd":1,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.37,"15min":0.378},"min":{"1min":0.23400000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.629,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39700000000000002,"15min":0.41099999999999998},"min":{"1min":0.28999999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.62,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.49399999999999999}]},{"osd":2,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.439,"15min":0.45500000000000002},"min":{"1min":0.28899999999999998,"5min":0.28799999999999998,"15min":0.27200000000000002},"max":{"1min":0.59099999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.45400000000000001,"15min":0.47199999999999998},"min":{"1min":0.34499999999999997,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.58599999999999997,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.53200000000000003}]},{"osd":4,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56799999999999995,"15min":0.59799999999999998},"min":{"1min":0.435,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.95999999999999996},"last":0.62},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.61699999999999999},"min":{"1min":0.437,"5min":0.39600000000000002,"15min":0.23699999999999999},"max":{"1min":0.85099999999999998,"5min":0.90100000000000002,"15min":0.93500000000000005},"last":0.71199999999999997}]},{"osd":5,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61599999999999999,"15min":0.64800000000000002},"min":{"1min":0.45200000000000001,"5min":0.374,"15min":0.309},"max":{"1min":0.80200000000000005,"5min":0.92200000000000004,"15min":0.96699999999999997},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.627,"15min":0.65500000000000003},"min":{"1min":0.44700000000000001,"5min":0.41499999999999998,"15min":0.38200000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.69099999999999995}]},{"osd":6,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59499999999999997,"15min":0.61399999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.36899999999999999},"max":{"1min":0.84999999999999998,"5min":1.026,"15min":1.026},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63700000000000001,"15min":0.65100000000000002},"min":{"1min":0.39900000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.94299999999999995,"5min":1.716,"15min":1.716},"last":0.66300000000000003}]},{"osd":7,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61199999999999999,"15min":0.63600000000000001},"min":{"1min":0.42499999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.92400000000000004,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.63100000000000001,"15min":0.65700000000000003},"min":{"1min":0.47499999999999998,"5min":0.40100000000000002,"15min":0.39500000000000002},"max":{"1min":0.875,"5min":1.002,"15min":1.002},"last":0.74399999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575084,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":1970176,"data_stored":990135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:31:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53700000000000003,"15min":0.55300000000000005},"min":{"1min":0.30399999999999999,"5min":0.29299999999999998,"15min":0.23899999999999999},"max":{"1min":0.66500000000000004,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.503,"5min":0.53600000000000003,"15min":0.55500000000000005},"min":{"1min":0.33900000000000002,"5min":0.33800000000000002,"15min":0.25900000000000001},"max":{"1min":0.73699999999999999,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.72699999999999998}]},{"osd":1,"last update":"Sat Apr 6 12:31:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.57999999999999996,"15min":0.59899999999999998},"min":{"1min":0.377,"5min":0.315,"15min":0.29799999999999999},"max":{"1min":0.75700000000000001,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.57599999999999996,"15min":0.58799999999999997},"min":{"1min":0.45200000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.74099999999999999,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.62}]},{"osd":2,"last update":"Sat Apr 6 12:31:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.56599999999999995},"min":{"1min":0.34200000000000003,"5min":0.312,"15min":0.312},"max":{"1min":0.78200000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.55600000000000005,"15min":0.57599999999999996},"min":{"1min":0.40500000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.76800000000000002,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.49099999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:31:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55300000000000005,"15min":0.55800000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78400000000000003,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.57099999999999995,"15min":0.59099999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.751,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.79500000000000004}]},{"osd":5,"last update":"Sat Apr 6 12:31:47 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27200000000000002},"max":{"1min":0.64000000000000001,"5min":0.73599999999999999,"15min":0.83599999999999997},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.48199999999999998,"15min":0.49299999999999999},"min":{"1min":0.34200000000000003,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72499999999999998,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.65700000000000003}]},{"osd":6,"last update":"Sat Apr 6 12:31:47 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.501,"15min":0.498},"min":{"1min":0.379,"5min":0.307,"15min":0.30599999999999999},"max":{"1min":0.76400000000000001,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53600000000000003,"15min":0.53400000000000003},"min":{"1min":0.36399999999999999,"5min":0.34100000000000003,"15min":0.32100000000000001},"max":{"1min":0.68999999999999995,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.55700000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:31:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53200000000000003},"min":{"1min":0.39600000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.754,"5min":0.77600000000000002,"15min":0.79300000000000004},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57699999999999996,"15min":0.56699999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80000000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.84899999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575084,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51088,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320752,"statfs":{"total":96636764160,"available":96584450048,"internally_reserved":0,"allocated":1970176,"data_stored":989138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54000000000000004,"15min":0.56499999999999995},"min":{"1min":0.35799999999999998,"5min":0.35199999999999998,"15min":0.29099999999999998},"max":{"1min":0.72399999999999998,"5min":0.81999999999999995,"15min":0.82599999999999996},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.51400000000000001,"15min":0.54600000000000004},"min":{"1min":0.40699999999999997,"5min":0.29499999999999998,"15min":0.22500000000000001},"max":{"1min":0.70599999999999996,"5min":0.748,"15min":0.80300000000000005},"last":0.61499999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52100000000000002,"15min":0.55300000000000005},"min":{"1min":0.40600000000000003,"5min":0.30399999999999999,"15min":0.26000000000000001},"max":{"1min":0.70399999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52500000000000002,"15min":0.55000000000000004},"min":{"1min":0.372,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.72199999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.60299999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53100000000000003,"15min":0.56100000000000005},"min":{"1min":0.38,"5min":0.33200000000000002,"15min":0.21099999999999999},"max":{"1min":0.67900000000000005,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.53800000000000003,"15min":0.56299999999999994},"min":{"1min":0.41699999999999998,"5min":0.35999999999999999,"15min":0.27400000000000002},"max":{"1min":0.67800000000000005,"5min":0.77100000000000002,"15min":0.83599999999999997},"last":0.54100000000000004}]},{"osd":3,"last update":"Sat Apr 6 12:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54700000000000004,"15min":0.56899999999999995},"min":{"1min":0.37,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70699999999999996,"5min":0.83799999999999997,"15min":0.874},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.55800000000000005,"15min":0.58899999999999997},"min":{"1min":0.41999999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.89600000000000002},"last":0.52200000000000002}]},{"osd":4,"last update":"Sat Apr 6 12:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.34000000000000002,"5min":0.28899999999999998,"15min":0.28299999999999997},"max":{"1min":0.73199999999999998,"5min":0.755,"15min":0.81200000000000006},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54300000000000004,"15min":0.53500000000000003},"min":{"1min":0.39600000000000002,"5min":0.29499999999999998,"15min":0.29299999999999998},"max":{"1min":0.72399999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.70199999999999996}]},{"osd":6,"last update":"Sat Apr 6 12:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47799999999999998,"15min":0.47899999999999998},"min":{"1min":0.307,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.70499999999999996,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.627},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.48199999999999998,"15min":0.49099999999999999},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.29699999999999999},"max":{"1min":0.65400000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.56100000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.48999999999999999,"15min":0.503},"min":{"1min":0.32700000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.70599999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.753},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.60299999999999998},"min":{"1min":0.47899999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.751,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.66700000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:32:07.195 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T12:32:07.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.354+0000 7fa44cf06700 1 Processor -- start 2024-04-06T12:32:07.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.355+0000 7fa44cf06700 1 -- start start 2024-04-06T12:32:07.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.355+0000 7fa44cf06700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa44815d590 0x7fa4480b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:07.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.355+0000 7fa44cf06700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4480b5cc0 con 0x7fa4480b4ed0 2024-04-06T12:32:07.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.355+0000 7fa44cf06700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4480d1500 con 0x7fa4480d3890 2024-04-06T12:32:07.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.355+0000 7fa44cf06700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4480d17e0 con 0x7fa44815d590 2024-04-06T12:32:07.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa445d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa4480b4ed0 0x7fa4480b6580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43982/0 (socket says 172.21.15.189:43982) 2024-04-06T12:32:07.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa445d9b700 1 -- 172.21.15.189:0/2493121214 learned_addr learned my addr 172.21.15.189:0/2493121214 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:32:07.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa44659c700 1 --2- 172.21.15.189:0/2493121214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa44815d590 0x7fa4480b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:07.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa44559a700 1 -- 172.21.15.189:0/2493121214 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880011797 0 0) 0x7fa4480b5cc0 con 0x7fa4480b4ed0 2024-04-06T12:32:07.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa44559a700 1 -- 172.21.15.189:0/2493121214 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4340035b0 con 0x7fa4480b4ed0 2024-04-06T12:32:07.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa44559a700 1 -- 172.21.15.189:0/2493121214 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128492787 0 0) 0x7fa4480d1500 con 0x7fa4480d3890 2024-04-06T12:32:07.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa44559a700 1 -- 172.21.15.189:0/2493121214 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4480b5cc0 con 0x7fa4480d3890 2024-04-06T12:32:07.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa44559a700 1 -- 172.21.15.189:0/2493121214 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1592120491 0 0) 0x7fa4340035b0 con 0x7fa4480b4ed0 2024-04-06T12:32:07.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa44559a700 1 -- 172.21.15.189:0/2493121214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa44815d590 msgr2=0x7fa4480b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:07.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa44559a700 1 --2- 172.21.15.189:0/2493121214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa44815d590 0x7fa4480b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:07.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa44559a700 1 -- 172.21.15.189:0/2493121214 >> v1:172.21.15.192:6789/0 conn(0x7fa4480d3890 legacy=0x7fa4480b6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:07.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa44559a700 1 -- 172.21.15.189:0/2493121214 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4480d1ac0 con 0x7fa4480b4ed0 2024-04-06T12:32:07.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa44559a700 1 -- 172.21.15.189:0/2493121214 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fa438001670 con 0x7fa4480b4ed0 2024-04-06T12:32:07.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa44559a700 1 -- 172.21.15.189:0/2493121214 >> v1:172.21.15.189:6789/0 conn(0x7fa4480b4ed0 legacy=0x7fa4480b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:07.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa44559a700 1 --2- 172.21.15.189:0/2493121214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa434003bc0 0x7fa434003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:07.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa44559a700 1 --2- 172.21.15.189:0/2493121214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa44815d590 0x7fa434004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:07.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa44559a700 1 --2- 172.21.15.189:0/2493121214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa434004d90 0x7fa434005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:07.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa44559a700 1 -- 172.21.15.189:0/2493121214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4480d17e0 con 0x7fa434003bc0 2024-04-06T12:32:07.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa44559a700 1 -- 172.21.15.189:0/2493121214 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43400bac0 con 0x7fa434004d90 2024-04-06T12:32:07.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa44559a700 1 -- 172.21.15.189:0/2493121214 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa43400bc30 con 0x7fa44815d590 2024-04-06T12:32:07.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa44659c700 1 --2- 172.21.15.189:0/2493121214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa434003bc0 0x7fa434003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:07.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa445d9b700 1 --2- 172.21.15.189:0/2493121214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa44815d590 0x7fa434004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:07.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.356+0000 7fa446d9d700 1 --2- 172.21.15.189:0/2493121214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa434004d90 0x7fa434005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:07.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.357+0000 7fa44659c700 1 -- 172.21.15.189:0/2493121214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa44815d590 msgr2=0x7fa434004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:07.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.357+0000 7fa44659c700 1 --2- 172.21.15.189:0/2493121214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa44815d590 0x7fa434004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:07.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.357+0000 7fa44659c700 1 -- 172.21.15.189:0/2493121214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa434004d90 msgr2=0x7fa434005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:07.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.357+0000 7fa44659c700 1 --2- 172.21.15.189:0/2493121214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa434004d90 0x7fa434005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:07.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.357+0000 7fa44659c700 1 -- 172.21.15.189:0/2493121214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4340035b0 con 0x7fa434003bc0 2024-04-06T12:32:07.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.357+0000 7fa44659c700 1 --2- 172.21.15.189:0/2493121214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa434003bc0 0x7fa434003fc0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fa430009b60 tx=0x7fa434005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c35f8f846028684 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:07.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.357+0000 7fa44559a700 1 -- 172.21.15.189:0/2493121214 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43000e6c0 con 0x7fa434003bc0 2024-04-06T12:32:07.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.357+0000 7fa44559a700 1 -- 172.21.15.189:0/2493121214 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43000e820 con 0x7fa434003bc0 2024-04-06T12:32:07.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.357+0000 7fa44559a700 1 -- 172.21.15.189:0/2493121214 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43000eb90 con 0x7fa434003bc0 2024-04-06T12:32:07.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.357+0000 7fa44cf06700 1 -- 172.21.15.189:0/2493121214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa434003bc0 msgr2=0x7fa434003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:07.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.357+0000 7fa44cf06700 1 --2- 172.21.15.189:0/2493121214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa434003bc0 0x7fa434003fc0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fa430009b60 tx=0x7fa434005880 comp rx=0 tx=0).stop 2024-04-06T12:32:07.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.357+0000 7fa446d9d700 1 -- 172.21.15.189:0/2493121214 reap_dead start 2024-04-06T12:32:07.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.358+0000 7fa44cf06700 1 -- 172.21.15.189:0/2493121214 shutdown_connections 2024-04-06T12:32:07.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.358+0000 7fa44cf06700 1 -- 172.21.15.189:0/2493121214 >> 172.21.15.189:0/2493121214 conn(0x7fa44806bbc0 msgr2=0x7fa44806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:07.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.358+0000 7fa44cf06700 1 -- 172.21.15.189:0/2493121214 shutdown_connections 2024-04-06T12:32:07.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.358+0000 7fa44cf06700 1 -- 172.21.15.189:0/2493121214 wait complete. 2024-04-06T12:32:07.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.359+0000 7fa44cf06700 1 Processor -- start 2024-04-06T12:32:07.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.359+0000 7fa44cf06700 1 -- start start 2024-04-06T12:32:07.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.360+0000 7fa44cf06700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa434003bc0 0x7fa4480cb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:07.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.360+0000 7fa44cf06700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa448146690 0x7fa4480cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:07.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.360+0000 7fa44cf06700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4480cc0c0 0x7fa448071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:07.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.360+0000 7fa44cf06700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4480cc6f0 con 0x7fa448146690 2024-04-06T12:32:07.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.360+0000 7fa44cf06700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4480cb950 con 0x7fa4480cc0c0 2024-04-06T12:32:07.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.360+0000 7fa44cf06700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4480ca650 con 0x7fa434003bc0 2024-04-06T12:32:07.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.360+0000 7fa445d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa448146690 0x7fa4480cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:07.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.360+0000 7fa445d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa448146690 0x7fa4480cbb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60778/0 (socket says 172.21.15.189:60778) 2024-04-06T12:32:07.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.360+0000 7fa445d9b700 1 -- 172.21.15.189:0/3263853291 learned_addr learned my addr 172.21.15.189:0/3263853291 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:07.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.361+0000 7fa445d9b700 1 -- 172.21.15.189:0/3263853291 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa434003bc0 msgr2=0x7fa4480cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:07.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.361+0000 7fa445d9b700 1 --2- 172.21.15.189:0/3263853291 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa434003bc0 0x7fa4480cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:07.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.361+0000 7fa445d9b700 1 -- 172.21.15.189:0/3263853291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4480cc0c0 msgr2=0x7fa448071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:07.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.361+0000 7fa445d9b700 1 --2- 172.21.15.189:0/3263853291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4480cc0c0 0x7fa448071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:07.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.361+0000 7fa445d9b700 1 -- 172.21.15.189:0/3263853291 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4480ca930 con 0x7fa448146690 2024-04-06T12:32:07.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.361+0000 7fa445d9b700 1 --2- 172.21.15.189:0/3263853291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa448146690 0x7fa4480cbb80 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fa438007310 tx=0x7fa438007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:07.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.361+0000 7fa43effd700 1 -- 172.21.15.189:0/3263853291 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa438007800 con 0x7fa448146690 2024-04-06T12:32:07.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.361+0000 7fa43effd700 1 -- 172.21.15.189:0/3263853291 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43800f7f0 con 0x7fa448146690 2024-04-06T12:32:07.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.361+0000 7fa44cf06700 1 -- 172.21.15.189:0/3263853291 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4480cac10 con 0x7fa448146690 2024-04-06T12:32:07.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.361+0000 7fa43effd700 1 -- 172.21.15.189:0/3263853291 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43800fba0 con 0x7fa448146690 2024-04-06T12:32:07.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.361+0000 7fa44cf06700 1 -- 172.21.15.189:0/3263853291 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa44814fb10 con 0x7fa448146690 2024-04-06T12:32:07.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.362+0000 7fa44cf06700 1 -- 172.21.15.189:0/3263853291 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4480cb130 con 0x7fa448146690 2024-04-06T12:32:07.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.362+0000 7fa43effd700 1 -- 172.21.15.189:0/3263853291 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fa43800fd00 con 0x7fa448146690 2024-04-06T12:32:07.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.363+0000 7fa43effd700 1 --2- 172.21.15.189:0/3263853291 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa42806ade0 0x7fa42806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:07.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.363+0000 7fa43effd700 1 -- 172.21.15.189:0/3263853291 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa438089550 con 0x7fa448146690 2024-04-06T12:32:07.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.363+0000 7fa44659c700 1 --2- 172.21.15.189:0/3263853291 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa42806ade0 0x7fa42806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:07.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.363+0000 7fa44659c700 1 --2- 172.21.15.189:0/3263853291 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa42806ade0 0x7fa42806d2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa430000c00 tx=0x7fa430013000 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:07.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.367+0000 7fa43effd700 1 -- 172.21.15.189:0/3263853291 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa4480cb130 con 0x7fa448146690 2024-04-06T12:32:07.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.560+0000 7fa44cf06700 1 -- 172.21.15.189:0/3263853291 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa4480d2360 con 0x7fa42806ade0 2024-04-06T12:32:07.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.567+0000 7fa43effd700 1 -- 172.21.15.189:0/3263853291 <== mgr.8176 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289917 (secure 0 0 0) 0x7fa43000fa20 con 0x7fa42806ade0 2024-04-06T12:32:07.571 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:32:07.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.571+0000 7fa44cf06700 1 -- 172.21.15.189:0/3263853291 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa42806ade0 msgr2=0x7fa42806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:07.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.571+0000 7fa44cf06700 1 --2- 172.21.15.189:0/3263853291 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa42806ade0 0x7fa42806d2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa430000c00 tx=0x7fa430013000 comp rx=0 tx=0).stop 2024-04-06T12:32:07.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.571+0000 7fa44cf06700 1 -- 172.21.15.189:0/3263853291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa448146690 msgr2=0x7fa4480cbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:07.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.571+0000 7fa44cf06700 1 --2- 172.21.15.189:0/3263853291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa448146690 0x7fa4480cbb80 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fa438007310 tx=0x7fa438007620 comp rx=0 tx=0).stop 2024-04-06T12:32:07.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.571+0000 7fa44cf06700 1 -- 172.21.15.189:0/3263853291 shutdown_connections 2024-04-06T12:32:07.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.571+0000 7fa44cf06700 1 --2- 172.21.15.189:0/3263853291 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa42806ade0 0x7fa42806d2a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:07.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.571+0000 7fa44cf06700 1 --2- 172.21.15.189:0/3263853291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4480cc0c0 0x7fa448071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:07.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.571+0000 7fa44cf06700 1 --2- 172.21.15.189:0/3263853291 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa434003bc0 0x7fa4480cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:07.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.571+0000 7fa44cf06700 1 --2- 172.21.15.189:0/3263853291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa448146690 0x7fa4480cbb80 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:07.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.571+0000 7fa44cf06700 1 -- 172.21.15.189:0/3263853291 >> 172.21.15.189:0/3263853291 conn(0x7fa44806bbc0 msgr2=0x7fa44806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:07.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.571+0000 7fa44cf06700 1 -- 172.21.15.189:0/3263853291 shutdown_connections 2024-04-06T12:32:07.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.571+0000 7fa44cf06700 1 -- 172.21.15.189:0/3263853291 wait complete. 2024-04-06T12:32:07.575 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T12:32:07.580 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-06T12:32:07.401109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":443,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":114,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":336064,"kb_used_data":16584,"kb_used_omap":12,"kb_used_meta":319347,"kb_avail":754638656,"statfs":{"total":773094113280,"available":772749983744,"internally_reserved":0,"allocated":16982016,"data_stored":9094825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":327011896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.744955"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034823+0000","last_change":"2024-04-06T12:32:03.082505+0000","last_active":"2024-04-06T12:32:05.034823+0000","last_peered":"2024-04-06T12:32:05.034823+0000","last_clean":"2024-04-06T12:32:05.034823+0000","last_became_active":"2024-04-06T12:32:03.082250+0000","last_became_peered":"2024-04-06T12:32:03.082250+0000","last_unstale":"2024-04-06T12:32:05.034823+0000","last_undegraded":"2024-04-06T12:32:05.034823+0000","last_fullsized":"2024-04-06T12:32:05.034823+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:25:53.432862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034139+0000","last_change":"2024-04-06T12:32:03.080983+0000","last_active":"2024-04-06T12:32:05.034139+0000","last_peered":"2024-04-06T12:32:05.034139+0000","last_clean":"2024-04-06T12:32:05.034139+0000","last_became_active":"2024-04-06T12:32:03.080727+0000","last_became_peered":"2024-04-06T12:32:03.080727+0000","last_unstale":"2024-04-06T12:32:05.034139+0000","last_undegraded":"2024-04-06T12:32:05.034139+0000","last_fullsized":"2024-04-06T12:32:05.034139+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:51:37.184427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.946187+0000","last_change":"2024-04-06T12:32:03.094137+0000","last_active":"2024-04-06T12:32:04.946187+0000","last_peered":"2024-04-06T12:32:04.946187+0000","last_clean":"2024-04-06T12:32:04.946187+0000","last_became_active":"2024-04-06T12:32:03.093781+0000","last_became_peered":"2024-04-06T12:32:03.093781+0000","last_unstale":"2024-04-06T12:32:04.946187+0000","last_undegraded":"2024-04-06T12:32:04.946187+0000","last_fullsized":"2024-04-06T12:32:04.946187+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:18:37.631769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034079+0000","last_change":"2024-04-06T12:32:03.083127+0000","last_active":"2024-04-06T12:32:05.034079+0000","last_peered":"2024-04-06T12:32:05.034079+0000","last_clean":"2024-04-06T12:32:05.034079+0000","last_became_active":"2024-04-06T12:32:03.082827+0000","last_became_peered":"2024-04-06T12:32:03.082827+0000","last_unstale":"2024-04-06T12:32:05.034079+0000","last_undegraded":"2024-04-06T12:32:05.034079+0000","last_fullsized":"2024-04-06T12:32:05.034079+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:18:37.631769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076339+0000","last_change":"2024-04-06T12:32:03.090821+0000","last_active":"2024-04-06T12:32:04.076339+0000","last_peered":"2024-04-06T12:32:04.076339+0000","last_clean":"2024-04-06T12:32:04.076339+0000","last_became_active":"2024-04-06T12:32:03.090524+0000","last_became_peered":"2024-04-06T12:32:03.090524+0000","last_unstale":"2024-04-06T12:32:04.076339+0000","last_undegraded":"2024-04-06T12:32:04.076339+0000","last_fullsized":"2024-04-06T12:32:04.076339+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:39.465068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034247+0000","last_change":"2024-04-06T12:32:03.095044+0000","last_active":"2024-04-06T12:32:05.034247+0000","last_peered":"2024-04-06T12:32:05.034247+0000","last_clean":"2024-04-06T12:32:05.034247+0000","last_became_active":"2024-04-06T12:32:03.094343+0000","last_became_peered":"2024-04-06T12:32:03.094343+0000","last_unstale":"2024-04-06T12:32:05.034247+0000","last_undegraded":"2024-04-06T12:32:05.034247+0000","last_fullsized":"2024-04-06T12:32:05.034247+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:17:55.389634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076557+0000","last_change":"2024-04-06T12:32:03.102162+0000","last_active":"2024-04-06T12:32:04.076557+0000","last_peered":"2024-04-06T12:32:04.076557+0000","last_clean":"2024-04-06T12:32:04.076557+0000","last_became_active":"2024-04-06T12:32:03.101868+0000","last_became_peered":"2024-04-06T12:32:03.101868+0000","last_unstale":"2024-04-06T12:32:04.076557+0000","last_undegraded":"2024-04-06T12:32:04.076557+0000","last_fullsized":"2024-04-06T12:32:04.076557+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:11:29.631951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.445270+0000","last_change":"2024-04-06T12:32:03.089271+0000","last_active":"2024-04-06T12:32:04.445270+0000","last_peered":"2024-04-06T12:32:04.445270+0000","last_clean":"2024-04-06T12:32:04.445270+0000","last_became_active":"2024-04-06T12:32:03.088992+0000","last_became_peered":"2024-04-06T12:32:03.088992+0000","last_unstale":"2024-04-06T12:32:04.445270+0000","last_undegraded":"2024-04-06T12:32:04.445270+0000","last_fullsized":"2024-04-06T12:32:04.445270+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:31:26.386814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034393+0000","last_change":"2024-04-06T12:32:03.080536+0000","last_active":"2024-04-06T12:32:05.034393+0000","last_peered":"2024-04-06T12:32:05.034393+0000","last_clean":"2024-04-06T12:32:05.034393+0000","last_became_active":"2024-04-06T12:32:03.080168+0000","last_became_peered":"2024-04-06T12:32:03.080168+0000","last_unstale":"2024-04-06T12:32:05.034393+0000","last_undegraded":"2024-04-06T12:32:05.034393+0000","last_fullsized":"2024-04-06T12:32:05.034393+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:24:51.136620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076436+0000","last_change":"2024-04-06T12:32:03.084520+0000","last_active":"2024-04-06T12:32:04.076436+0000","last_peered":"2024-04-06T12:32:04.076436+0000","last_clean":"2024-04-06T12:32:04.076436+0000","last_became_active":"2024-04-06T12:32:03.084255+0000","last_became_peered":"2024-04-06T12:32:03.084255+0000","last_unstale":"2024-04-06T12:32:04.076436+0000","last_undegraded":"2024-04-06T12:32:04.076436+0000","last_fullsized":"2024-04-06T12:32:04.076436+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:17:09.015831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034024+0000","last_change":"2024-04-06T12:32:03.097727+0000","last_active":"2024-04-06T12:32:05.034024+0000","last_peered":"2024-04-06T12:32:05.034024+0000","last_clean":"2024-04-06T12:32:05.034024+0000","last_became_active":"2024-04-06T12:32:03.097459+0000","last_became_peered":"2024-04-06T12:32:03.097459+0000","last_unstale":"2024-04-06T12:32:05.034024+0000","last_undegraded":"2024-04-06T12:32:05.034024+0000","last_fullsized":"2024-04-06T12:32:05.034024+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:31:01.252124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034764+0000","last_change":"2024-04-06T12:32:03.089641+0000","last_active":"2024-04-06T12:32:05.034764+0000","last_peered":"2024-04-06T12:32:05.034764+0000","last_clean":"2024-04-06T12:32:05.034764+0000","last_became_active":"2024-04-06T12:32:03.089401+0000","last_became_peered":"2024-04-06T12:32:03.089401+0000","last_unstale":"2024-04-06T12:32:05.034764+0000","last_undegraded":"2024-04-06T12:32:05.034764+0000","last_fullsized":"2024-04-06T12:32:05.034764+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:22:21.471765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034704+0000","last_change":"2024-04-06T12:32:03.091650+0000","last_active":"2024-04-06T12:32:05.034704+0000","last_peered":"2024-04-06T12:32:05.034704+0000","last_clean":"2024-04-06T12:32:05.034704+0000","last_became_active":"2024-04-06T12:32:03.091323+0000","last_became_peered":"2024-04-06T12:32:03.091323+0000","last_unstale":"2024-04-06T12:32:05.034704+0000","last_undegraded":"2024-04-06T12:32:05.034704+0000","last_fullsized":"2024-04-06T12:32:05.034704+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:48:06.372887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034665+0000","last_change":"2024-04-06T12:32:03.098994+0000","last_active":"2024-04-06T12:32:05.034665+0000","last_peered":"2024-04-06T12:32:05.034665+0000","last_clean":"2024-04-06T12:32:05.034665+0000","last_became_active":"2024-04-06T12:32:03.098789+0000","last_became_peered":"2024-04-06T12:32:03.098789+0000","last_unstale":"2024-04-06T12:32:05.034665+0000","last_undegraded":"2024-04-06T12:32:05.034665+0000","last_fullsized":"2024-04-06T12:32:05.034665+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:39:56.115752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034053+0000","last_change":"2024-04-06T12:32:03.089587+0000","last_active":"2024-04-06T12:32:05.034053+0000","last_peered":"2024-04-06T12:32:05.034053+0000","last_clean":"2024-04-06T12:32:05.034053+0000","last_became_active":"2024-04-06T12:32:03.089318+0000","last_became_peered":"2024-04-06T12:32:03.089318+0000","last_unstale":"2024-04-06T12:32:05.034053+0000","last_undegraded":"2024-04-06T12:32:05.034053+0000","last_fullsized":"2024-04-06T12:32:05.034053+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:23.985084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.033876+0000","last_change":"2024-04-06T12:32:03.084489+0000","last_active":"2024-04-06T12:32:05.033876+0000","last_peered":"2024-04-06T12:32:05.033876+0000","last_clean":"2024-04-06T12:32:05.033876+0000","last_became_active":"2024-04-06T12:32:03.084180+0000","last_became_peered":"2024-04-06T12:32:03.084180+0000","last_unstale":"2024-04-06T12:32:05.033876+0000","last_undegraded":"2024-04-06T12:32:05.033876+0000","last_fullsized":"2024-04-06T12:32:05.033876+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:31:48.007605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034907+0000","last_change":"2024-04-06T12:32:03.084356+0000","last_active":"2024-04-06T12:32:05.034907+0000","last_peered":"2024-04-06T12:32:05.034907+0000","last_clean":"2024-04-06T12:32:05.034907+0000","last_became_active":"2024-04-06T12:32:03.084107+0000","last_became_peered":"2024-04-06T12:32:03.084107+0000","last_unstale":"2024-04-06T12:32:05.034907+0000","last_undegraded":"2024-04-06T12:32:05.034907+0000","last_fullsized":"2024-04-06T12:32:05.034907+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:03:09.310724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076997+0000","last_change":"2024-04-06T12:32:03.091046+0000","last_active":"2024-04-06T12:32:04.076997+0000","last_peered":"2024-04-06T12:32:04.076997+0000","last_clean":"2024-04-06T12:32:04.076997+0000","last_became_active":"2024-04-06T12:32:03.090743+0000","last_became_peered":"2024-04-06T12:32:03.090743+0000","last_unstale":"2024-04-06T12:32:04.076997+0000","last_undegraded":"2024-04-06T12:32:04.076997+0000","last_fullsized":"2024-04-06T12:32:04.076997+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:41:07.421129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077179+0000","last_change":"2024-04-06T12:32:03.104504+0000","last_active":"2024-04-06T12:32:04.077179+0000","last_peered":"2024-04-06T12:32:04.077179+0000","last_clean":"2024-04-06T12:32:04.077179+0000","last_became_active":"2024-04-06T12:32:03.104269+0000","last_became_peered":"2024-04-06T12:32:03.104269+0000","last_unstale":"2024-04-06T12:32:04.077179+0000","last_undegraded":"2024-04-06T12:32:04.077179+0000","last_fullsized":"2024-04-06T12:32:04.077179+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:24:51.136620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076452+0000","last_change":"2024-04-06T12:32:03.107653+0000","last_active":"2024-04-06T12:32:04.076452+0000","last_peered":"2024-04-06T12:32:04.076452+0000","last_clean":"2024-04-06T12:32:04.076452+0000","last_became_active":"2024-04-06T12:32:03.107419+0000","last_became_peered":"2024-04-06T12:32:03.107419+0000","last_unstale":"2024-04-06T12:32:04.076452+0000","last_undegraded":"2024-04-06T12:32:04.076452+0000","last_fullsized":"2024-04-06T12:32:04.076452+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:58:40.468358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034078+0000","last_change":"2024-04-06T12:32:03.084284+0000","last_active":"2024-04-06T12:32:05.034078+0000","last_peered":"2024-04-06T12:32:05.034078+0000","last_clean":"2024-04-06T12:32:05.034078+0000","last_became_active":"2024-04-06T12:32:03.083950+0000","last_became_peered":"2024-04-06T12:32:03.083950+0000","last_unstale":"2024-04-06T12:32:05.034078+0000","last_undegraded":"2024-04-06T12:32:05.034078+0000","last_fullsized":"2024-04-06T12:32:05.034078+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:00.765348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034689+0000","last_change":"2024-04-06T12:32:03.087351+0000","last_active":"2024-04-06T12:32:05.034689+0000","last_peered":"2024-04-06T12:32:05.034689+0000","last_clean":"2024-04-06T12:32:05.034689+0000","last_became_active":"2024-04-06T12:32:03.087111+0000","last_became_peered":"2024-04-06T12:32:03.087111+0000","last_unstale":"2024-04-06T12:32:05.034689+0000","last_undegraded":"2024-04-06T12:32:05.034689+0000","last_fullsized":"2024-04-06T12:32:05.034689+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:09:38.972919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.033956+0000","last_change":"2024-04-06T12:32:03.090736+0000","last_active":"2024-04-06T12:32:05.033956+0000","last_peered":"2024-04-06T12:32:05.033956+0000","last_clean":"2024-04-06T12:32:05.033956+0000","last_became_active":"2024-04-06T12:32:03.090461+0000","last_became_peered":"2024-04-06T12:32:03.090461+0000","last_unstale":"2024-04-06T12:32:05.033956+0000","last_undegraded":"2024-04-06T12:32:05.033956+0000","last_fullsized":"2024-04-06T12:32:05.033956+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:09:38.972919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.033972+0000","last_change":"2024-04-06T12:32:03.093015+0000","last_active":"2024-04-06T12:32:05.033972+0000","last_peered":"2024-04-06T12:32:05.033972+0000","last_clean":"2024-04-06T12:32:05.033972+0000","last_became_active":"2024-04-06T12:32:03.092769+0000","last_became_peered":"2024-04-06T12:32:03.092769+0000","last_unstale":"2024-04-06T12:32:05.033972+0000","last_undegraded":"2024-04-06T12:32:05.033972+0000","last_fullsized":"2024-04-06T12:32:05.033972+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:25:42.693508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.946246+0000","last_change":"2024-04-06T12:32:03.099232+0000","last_active":"2024-04-06T12:32:04.946246+0000","last_peered":"2024-04-06T12:32:04.946246+0000","last_clean":"2024-04-06T12:32:04.946246+0000","last_became_active":"2024-04-06T12:32:03.098975+0000","last_became_peered":"2024-04-06T12:32:03.098975+0000","last_unstale":"2024-04-06T12:32:04.946246+0000","last_undegraded":"2024-04-06T12:32:04.946246+0000","last_fullsized":"2024-04-06T12:32:04.946246+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:09:02.410710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.946215+0000","last_change":"2024-04-06T12:32:03.108137+0000","last_active":"2024-04-06T12:32:04.946215+0000","last_peered":"2024-04-06T12:32:04.946215+0000","last_clean":"2024-04-06T12:32:04.946215+0000","last_became_active":"2024-04-06T12:32:03.107855+0000","last_became_peered":"2024-04-06T12:32:03.107855+0000","last_unstale":"2024-04-06T12:32:04.946215+0000","last_undegraded":"2024-04-06T12:32:04.946215+0000","last_fullsized":"2024-04-06T12:32:04.946215+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:51:37.184427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077030+0000","last_change":"2024-04-06T12:32:03.101971+0000","last_active":"2024-04-06T12:32:04.077030+0000","last_peered":"2024-04-06T12:32:04.077030+0000","last_clean":"2024-04-06T12:32:04.077030+0000","last_became_active":"2024-04-06T12:32:03.101656+0000","last_became_peered":"2024-04-06T12:32:03.101656+0000","last_unstale":"2024-04-06T12:32:04.077030+0000","last_undegraded":"2024-04-06T12:32:04.077030+0000","last_fullsized":"2024-04-06T12:32:04.077030+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:03:09.310724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076726+0000","last_change":"2024-04-06T12:32:03.082945+0000","last_active":"2024-04-06T12:32:04.076726+0000","last_peered":"2024-04-06T12:32:04.076726+0000","last_clean":"2024-04-06T12:32:04.076726+0000","last_became_active":"2024-04-06T12:32:03.082707+0000","last_became_peered":"2024-04-06T12:32:03.082707+0000","last_unstale":"2024-04-06T12:32:04.076726+0000","last_undegraded":"2024-04-06T12:32:04.076726+0000","last_fullsized":"2024-04-06T12:32:04.076726+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:58:03.740342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.945912+0000","last_change":"2024-04-06T12:32:03.094085+0000","last_active":"2024-04-06T12:32:04.945912+0000","last_peered":"2024-04-06T12:32:04.945912+0000","last_clean":"2024-04-06T12:32:04.945912+0000","last_became_active":"2024-04-06T12:32:03.093281+0000","last_became_peered":"2024-04-06T12:32:03.093281+0000","last_unstale":"2024-04-06T12:32:04.945912+0000","last_undegraded":"2024-04-06T12:32:04.945912+0000","last_fullsized":"2024-04-06T12:32:04.945912+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:00.765348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.945903+0000","last_change":"2024-04-06T12:32:03.100141+0000","last_active":"2024-04-06T12:32:04.945903+0000","last_peered":"2024-04-06T12:32:04.945903+0000","last_clean":"2024-04-06T12:32:04.945903+0000","last_became_active":"2024-04-06T12:32:03.099911+0000","last_became_peered":"2024-04-06T12:32:03.099911+0000","last_unstale":"2024-04-06T12:32:04.945903+0000","last_undegraded":"2024-04-06T12:32:04.945903+0000","last_fullsized":"2024-04-06T12:32:04.945903+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:31:01.252124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.445302+0000","last_change":"2024-04-06T12:32:03.089180+0000","last_active":"2024-04-06T12:32:04.445302+0000","last_peered":"2024-04-06T12:32:04.445302+0000","last_clean":"2024-04-06T12:32:04.445302+0000","last_became_active":"2024-04-06T12:32:03.088869+0000","last_became_peered":"2024-04-06T12:32:03.088869+0000","last_unstale":"2024-04-06T12:32:04.445302+0000","last_undegraded":"2024-04-06T12:32:04.445302+0000","last_fullsized":"2024-04-06T12:32:04.445302+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:05.556922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076117+0000","last_change":"2024-04-06T12:32:03.084883+0000","last_active":"2024-04-06T12:32:04.076117+0000","last_peered":"2024-04-06T12:32:04.076117+0000","last_clean":"2024-04-06T12:32:04.076117+0000","last_became_active":"2024-04-06T12:32:03.084643+0000","last_became_peered":"2024-04-06T12:32:03.084643+0000","last_unstale":"2024-04-06T12:32:04.076117+0000","last_undegraded":"2024-04-06T12:32:04.076117+0000","last_fullsized":"2024-04-06T12:32:04.076117+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:47:54.703707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.445270+0000","last_change":"2024-04-06T12:32:03.096345+0000","last_active":"2024-04-06T12:32:04.445270+0000","last_peered":"2024-04-06T12:32:04.445270+0000","last_clean":"2024-04-06T12:32:04.445270+0000","last_became_active":"2024-04-06T12:32:03.096091+0000","last_became_peered":"2024-04-06T12:32:03.096091+0000","last_unstale":"2024-04-06T12:32:04.445270+0000","last_undegraded":"2024-04-06T12:32:04.445270+0000","last_fullsized":"2024-04-06T12:32:04.445270+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:22:30.405552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"157'4","reported_seq":15,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.946307+0000","last_change":"2024-04-06T12:32:02.673852+0000","last_active":"2024-04-06T12:32:04.946307+0000","last_peered":"2024-04-06T12:32:04.946307+0000","last_clean":"2024-04-06T12:32:04.946307+0000","last_became_active":"2024-04-06T12:32:02.673317+0000","last_became_peered":"2024-04-06T12:32:02.673317+0000","last_unstale":"2024-04-06T12:32:04.946307+0000","last_undegraded":"2024-04-06T12:32:04.946307+0000","last_fullsized":"2024-04-06T12:32:04.946307+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:41:59.741918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034165+0000","last_change":"2024-04-06T12:32:03.088020+0000","last_active":"2024-04-06T12:32:05.034165+0000","last_peered":"2024-04-06T12:32:05.034165+0000","last_clean":"2024-04-06T12:32:05.034165+0000","last_became_active":"2024-04-06T12:32:03.087581+0000","last_became_peered":"2024-04-06T12:32:03.087581+0000","last_unstale":"2024-04-06T12:32:05.034165+0000","last_undegraded":"2024-04-06T12:32:05.034165+0000","last_fullsized":"2024-04-06T12:32:05.034165+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:56:21.396821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"157'1","reported_seq":12,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077143+0000","last_change":"2024-04-06T12:32:02.670991+0000","last_active":"2024-04-06T12:32:04.077143+0000","last_peered":"2024-04-06T12:32:04.077143+0000","last_clean":"2024-04-06T12:32:04.077143+0000","last_became_active":"2024-04-06T12:32:02.670714+0000","last_became_peered":"2024-04-06T12:32:02.670714+0000","last_unstale":"2024-04-06T12:32:04.077143+0000","last_undegraded":"2024-04-06T12:32:04.077143+0000","last_fullsized":"2024-04-06T12:32:04.077143+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:03:08.301680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034011+0000","last_change":"2024-04-06T12:32:03.099273+0000","last_active":"2024-04-06T12:32:05.034011+0000","last_peered":"2024-04-06T12:32:05.034011+0000","last_clean":"2024-04-06T12:32:05.034011+0000","last_became_active":"2024-04-06T12:32:03.099021+0000","last_became_peered":"2024-04-06T12:32:03.099021+0000","last_unstale":"2024-04-06T12:32:05.034011+0000","last_undegraded":"2024-04-06T12:32:05.034011+0000","last_fullsized":"2024-04-06T12:32:05.034011+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:52:04.592856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"157'2","reported_seq":13,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077096+0000","last_change":"2024-04-06T12:32:02.674743+0000","last_active":"2024-04-06T12:32:04.077096+0000","last_peered":"2024-04-06T12:32:04.077096+0000","last_clean":"2024-04-06T12:32:04.077096+0000","last_became_active":"2024-04-06T12:32:02.674438+0000","last_became_peered":"2024-04-06T12:32:02.674438+0000","last_unstale":"2024-04-06T12:32:04.077096+0000","last_undegraded":"2024-04-06T12:32:04.077096+0000","last_fullsized":"2024-04-06T12:32:04.077096+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:29:22.645084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034488+0000","last_change":"2024-04-06T12:32:03.090270+0000","last_active":"2024-04-06T12:32:05.034488+0000","last_peered":"2024-04-06T12:32:05.034488+0000","last_clean":"2024-04-06T12:32:05.034488+0000","last_became_active":"2024-04-06T12:32:03.090029+0000","last_became_peered":"2024-04-06T12:32:03.090029+0000","last_unstale":"2024-04-06T12:32:05.034488+0000","last_undegraded":"2024-04-06T12:32:05.034488+0000","last_fullsized":"2024-04-06T12:32:05.034488+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:11:33.053576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"157'2","reported_seq":13,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077222+0000","last_change":"2024-04-06T12:32:02.667498+0000","last_active":"2024-04-06T12:32:04.077222+0000","last_peered":"2024-04-06T12:32:04.077222+0000","last_clean":"2024-04-06T12:32:04.077222+0000","last_became_active":"2024-04-06T12:32:02.667227+0000","last_became_peered":"2024-04-06T12:32:02.667227+0000","last_unstale":"2024-04-06T12:32:04.077222+0000","last_undegraded":"2024-04-06T12:32:04.077222+0000","last_fullsized":"2024-04-06T12:32:04.077222+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:45:12.515973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.445549+0000","last_change":"2024-04-06T12:32:03.089533+0000","last_active":"2024-04-06T12:32:04.445549+0000","last_peered":"2024-04-06T12:32:04.445549+0000","last_clean":"2024-04-06T12:32:04.445549+0000","last_became_active":"2024-04-06T12:32:03.088844+0000","last_became_peered":"2024-04-06T12:32:03.088844+0000","last_unstale":"2024-04-06T12:32:04.445549+0000","last_undegraded":"2024-04-06T12:32:04.445549+0000","last_fullsized":"2024-04-06T12:32:04.445549+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:12:29.374507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"157'1","reported_seq":12,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.445497+0000","last_change":"2024-04-06T12:32:02.676754+0000","last_active":"2024-04-06T12:32:04.445497+0000","last_peered":"2024-04-06T12:32:04.445497+0000","last_clean":"2024-04-06T12:32:04.445497+0000","last_became_active":"2024-04-06T12:32:02.676449+0000","last_became_peered":"2024-04-06T12:32:02.676449+0000","last_unstale":"2024-04-06T12:32:04.445497+0000","last_undegraded":"2024-04-06T12:32:04.445497+0000","last_fullsized":"2024-04-06T12:32:04.445497+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:37:59.756304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.445540+0000","last_change":"2024-04-06T12:32:03.090474+0000","last_active":"2024-04-06T12:32:04.445540+0000","last_peered":"2024-04-06T12:32:04.445540+0000","last_clean":"2024-04-06T12:32:04.445540+0000","last_became_active":"2024-04-06T12:32:03.090148+0000","last_became_peered":"2024-04-06T12:32:03.090148+0000","last_unstale":"2024-04-06T12:32:04.445540+0000","last_undegraded":"2024-04-06T12:32:04.445540+0000","last_fullsized":"2024-04-06T12:32:04.445540+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:12:19.356287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"157'1","reported_seq":12,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076272+0000","last_change":"2024-04-06T12:32:02.661929+0000","last_active":"2024-04-06T12:32:04.076272+0000","last_peered":"2024-04-06T12:32:04.076272+0000","last_clean":"2024-04-06T12:32:04.076272+0000","last_became_active":"2024-04-06T12:32:02.661649+0000","last_became_peered":"2024-04-06T12:32:02.661649+0000","last_unstale":"2024-04-06T12:32:04.076272+0000","last_undegraded":"2024-04-06T12:32:04.076272+0000","last_fullsized":"2024-04-06T12:32:04.076272+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:51:21.530690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077054+0000","last_change":"2024-04-06T12:32:03.102445+0000","last_active":"2024-04-06T12:32:04.077054+0000","last_peered":"2024-04-06T12:32:04.077054+0000","last_clean":"2024-04-06T12:32:04.077054+0000","last_became_active":"2024-04-06T12:32:03.102248+0000","last_became_peered":"2024-04-06T12:32:03.102248+0000","last_unstale":"2024-04-06T12:32:04.077054+0000","last_undegraded":"2024-04-06T12:32:04.077054+0000","last_fullsized":"2024-04-06T12:32:04.077054+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:13:01.541685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034132+0000","last_change":"2024-04-06T12:32:02.674195+0000","last_active":"2024-04-06T12:32:05.034132+0000","last_peered":"2024-04-06T12:32:05.034132+0000","last_clean":"2024-04-06T12:32:05.034132+0000","last_became_active":"2024-04-06T12:32:02.673397+0000","last_became_peered":"2024-04-06T12:32:02.673397+0000","last_unstale":"2024-04-06T12:32:05.034132+0000","last_undegraded":"2024-04-06T12:32:05.034132+0000","last_fullsized":"2024-04-06T12:32:05.034132+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:29:22.645084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077476+0000","last_change":"2024-04-06T12:32:03.096249+0000","last_active":"2024-04-06T12:32:04.077476+0000","last_peered":"2024-04-06T12:32:04.077476+0000","last_clean":"2024-04-06T12:32:04.077476+0000","last_became_active":"2024-04-06T12:32:03.095988+0000","last_became_peered":"2024-04-06T12:32:03.095988+0000","last_unstale":"2024-04-06T12:32:04.077476+0000","last_undegraded":"2024-04-06T12:32:04.077476+0000","last_fullsized":"2024-04-06T12:32:04.077476+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:31:48.007605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.945885+0000","last_change":"2024-04-06T12:32:02.673724+0000","last_active":"2024-04-06T12:32:04.945885+0000","last_peered":"2024-04-06T12:32:04.945885+0000","last_clean":"2024-04-06T12:32:04.945885+0000","last_became_active":"2024-04-06T12:32:02.673350+0000","last_became_peered":"2024-04-06T12:32:02.673350+0000","last_unstale":"2024-04-06T12:32:04.945885+0000","last_undegraded":"2024-04-06T12:32:04.945885+0000","last_fullsized":"2024-04-06T12:32:04.945885+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:04:32.029292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.033854+0000","last_change":"2024-04-06T12:32:03.096606+0000","last_active":"2024-04-06T12:32:05.033854+0000","last_peered":"2024-04-06T12:32:05.033854+0000","last_clean":"2024-04-06T12:32:05.033854+0000","last_became_active":"2024-04-06T12:32:03.096302+0000","last_became_peered":"2024-04-06T12:32:03.096302+0000","last_unstale":"2024-04-06T12:32:05.033854+0000","last_undegraded":"2024-04-06T12:32:05.033854+0000","last_fullsized":"2024-04-06T12:32:05.033854+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:25:36.894614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034235+0000","last_change":"2024-04-06T12:32:02.672265+0000","last_active":"2024-04-06T12:32:05.034235+0000","last_peered":"2024-04-06T12:32:05.034235+0000","last_clean":"2024-04-06T12:32:05.034235+0000","last_became_active":"2024-04-06T12:32:02.671995+0000","last_became_peered":"2024-04-06T12:32:02.671995+0000","last_unstale":"2024-04-06T12:32:05.034235+0000","last_undegraded":"2024-04-06T12:32:05.034235+0000","last_fullsized":"2024-04-06T12:32:05.034235+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:20:14.679061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.033988+0000","last_change":"2024-04-06T12:32:03.082704+0000","last_active":"2024-04-06T12:32:05.033988+0000","last_peered":"2024-04-06T12:32:05.033988+0000","last_clean":"2024-04-06T12:32:05.033988+0000","last_became_active":"2024-04-06T12:32:03.082421+0000","last_became_peered":"2024-04-06T12:32:03.082421+0000","last_unstale":"2024-04-06T12:32:05.033988+0000","last_undegraded":"2024-04-06T12:32:05.033988+0000","last_fullsized":"2024-04-06T12:32:05.033988+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:32:44.600075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034510+0000","last_change":"2024-04-06T12:32:02.676453+0000","last_active":"2024-04-06T12:32:05.034510+0000","last_peered":"2024-04-06T12:32:05.034510+0000","last_clean":"2024-04-06T12:32:05.034510+0000","last_became_active":"2024-04-06T12:32:02.676191+0000","last_became_peered":"2024-04-06T12:32:02.676191+0000","last_unstale":"2024-04-06T12:32:05.034510+0000","last_undegraded":"2024-04-06T12:32:05.034510+0000","last_fullsized":"2024-04-06T12:32:05.034510+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:32:38.840045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.945975+0000","last_change":"2024-04-06T12:32:03.094000+0000","last_active":"2024-04-06T12:32:04.945975+0000","last_peered":"2024-04-06T12:32:04.945975+0000","last_clean":"2024-04-06T12:32:04.945975+0000","last_became_active":"2024-04-06T12:32:03.092683+0000","last_became_peered":"2024-04-06T12:32:03.092683+0000","last_unstale":"2024-04-06T12:32:04.945975+0000","last_undegraded":"2024-04-06T12:32:04.945975+0000","last_fullsized":"2024-04-06T12:32:04.945975+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:20:54.013729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"157'3","reported_seq":14,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077194+0000","last_change":"2024-04-06T12:32:02.672117+0000","last_active":"2024-04-06T12:32:04.077194+0000","last_peered":"2024-04-06T12:32:04.077194+0000","last_clean":"2024-04-06T12:32:04.077194+0000","last_became_active":"2024-04-06T12:32:02.671844+0000","last_became_peered":"2024-04-06T12:32:02.671844+0000","last_unstale":"2024-04-06T12:32:04.077194+0000","last_undegraded":"2024-04-06T12:32:04.077194+0000","last_fullsized":"2024-04-06T12:32:04.077194+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:55:47.076984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.946044+0000","last_change":"2024-04-06T12:32:03.095285+0000","last_active":"2024-04-06T12:32:04.946044+0000","last_peered":"2024-04-06T12:32:04.946044+0000","last_clean":"2024-04-06T12:32:04.946044+0000","last_became_active":"2024-04-06T12:32:03.094997+0000","last_became_peered":"2024-04-06T12:32:03.094997+0000","last_unstale":"2024-04-06T12:32:04.946044+0000","last_undegraded":"2024-04-06T12:32:04.946044+0000","last_fullsized":"2024-04-06T12:32:04.946044+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:58:23.687900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"157'6","reported_seq":17,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077323+0000","last_change":"2024-04-06T12:32:02.672984+0000","last_active":"2024-04-06T12:32:04.077323+0000","last_peered":"2024-04-06T12:32:04.077323+0000","last_clean":"2024-04-06T12:32:04.077323+0000","last_became_active":"2024-04-06T12:32:02.672507+0000","last_became_peered":"2024-04-06T12:32:02.672507+0000","last_unstale":"2024-04-06T12:32:04.077323+0000","last_undegraded":"2024-04-06T12:32:04.077323+0000","last_fullsized":"2024-04-06T12:32:04.077323+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:49:55.738902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076779+0000","last_change":"2024-04-06T12:32:03.081118+0000","last_active":"2024-04-06T12:32:04.076779+0000","last_peered":"2024-04-06T12:32:04.076779+0000","last_clean":"2024-04-06T12:32:04.076779+0000","last_became_active":"2024-04-06T12:32:03.080855+0000","last_became_peered":"2024-04-06T12:32:03.080855+0000","last_unstale":"2024-04-06T12:32:04.076779+0000","last_undegraded":"2024-04-06T12:32:04.076779+0000","last_fullsized":"2024-04-06T12:32:04.076779+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:26:50.050125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"157'2","reported_seq":13,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034691+0000","last_change":"2024-04-06T12:32:02.676947+0000","last_active":"2024-04-06T12:32:05.034691+0000","last_peered":"2024-04-06T12:32:05.034691+0000","last_clean":"2024-04-06T12:32:05.034691+0000","last_became_active":"2024-04-06T12:32:02.676514+0000","last_became_peered":"2024-04-06T12:32:02.676514+0000","last_unstale":"2024-04-06T12:32:05.034691+0000","last_undegraded":"2024-04-06T12:32:05.034691+0000","last_fullsized":"2024-04-06T12:32:05.034691+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:35:45.136425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077438+0000","last_change":"2024-04-06T12:32:03.089750+0000","last_active":"2024-04-06T12:32:04.077438+0000","last_peered":"2024-04-06T12:32:04.077438+0000","last_clean":"2024-04-06T12:32:04.077438+0000","last_became_active":"2024-04-06T12:32:03.089323+0000","last_became_peered":"2024-04-06T12:32:03.089323+0000","last_unstale":"2024-04-06T12:32:04.077438+0000","last_undegraded":"2024-04-06T12:32:04.077438+0000","last_fullsized":"2024-04-06T12:32:04.077438+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:14:34.054920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.945962+0000","last_change":"2024-04-06T12:32:02.671371+0000","last_active":"2024-04-06T12:32:04.945962+0000","last_peered":"2024-04-06T12:32:04.945962+0000","last_clean":"2024-04-06T12:32:04.945962+0000","last_became_active":"2024-04-06T12:32:02.671103+0000","last_became_peered":"2024-04-06T12:32:02.671103+0000","last_unstale":"2024-04-06T12:32:04.945962+0000","last_undegraded":"2024-04-06T12:32:04.945962+0000","last_fullsized":"2024-04-06T12:32:04.945962+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:50:58.655580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077263+0000","last_change":"2024-04-06T12:32:03.090934+0000","last_active":"2024-04-06T12:32:04.077263+0000","last_peered":"2024-04-06T12:32:04.077263+0000","last_clean":"2024-04-06T12:32:04.077263+0000","last_became_active":"2024-04-06T12:32:03.090284+0000","last_became_peered":"2024-04-06T12:32:03.090284+0000","last_unstale":"2024-04-06T12:32:04.077263+0000","last_undegraded":"2024-04-06T12:32:04.077263+0000","last_fullsized":"2024-04-06T12:32:04.077263+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:21:11.872887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"157'8","reported_seq":19,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034740+0000","last_change":"2024-04-06T12:32:02.673648+0000","last_active":"2024-04-06T12:32:05.034740+0000","last_peered":"2024-04-06T12:32:05.034740+0000","last_clean":"2024-04-06T12:32:05.034740+0000","last_became_active":"2024-04-06T12:32:02.673372+0000","last_became_peered":"2024-04-06T12:32:02.673372+0000","last_unstale":"2024-04-06T12:32:05.034740+0000","last_undegraded":"2024-04-06T12:32:05.034740+0000","last_fullsized":"2024-04-06T12:32:05.034740+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:09:01.401665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.945834+0000","last_change":"2024-04-06T12:32:03.105429+0000","last_active":"2024-04-06T12:32:04.945834+0000","last_peered":"2024-04-06T12:32:04.945834+0000","last_clean":"2024-04-06T12:32:04.945834+0000","last_became_active":"2024-04-06T12:32:03.105187+0000","last_became_peered":"2024-04-06T12:32:03.105187+0000","last_unstale":"2024-04-06T12:32:04.945834+0000","last_undegraded":"2024-04-06T12:32:04.945834+0000","last_fullsized":"2024-04-06T12:32:04.945834+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:52:04.592856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"157'1","reported_seq":12,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034295+0000","last_change":"2024-04-06T12:32:02.676442+0000","last_active":"2024-04-06T12:32:05.034295+0000","last_peered":"2024-04-06T12:32:05.034295+0000","last_clean":"2024-04-06T12:32:05.034295+0000","last_became_active":"2024-04-06T12:32:02.676164+0000","last_became_peered":"2024-04-06T12:32:02.676164+0000","last_unstale":"2024-04-06T12:32:05.034295+0000","last_undegraded":"2024-04-06T12:32:05.034295+0000","last_fullsized":"2024-04-06T12:32:05.034295+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:41:59.741918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076562+0000","last_change":"2024-04-06T12:32:03.094666+0000","last_active":"2024-04-06T12:32:04.076562+0000","last_peered":"2024-04-06T12:32:04.076562+0000","last_clean":"2024-04-06T12:32:04.076562+0000","last_became_active":"2024-04-06T12:32:03.090349+0000","last_became_peered":"2024-04-06T12:32:03.090349+0000","last_unstale":"2024-04-06T12:32:04.076562+0000","last_undegraded":"2024-04-06T12:32:04.076562+0000","last_fullsized":"2024-04-06T12:32:04.076562+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:17:55.389634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"157'2","reported_seq":13,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.946282+0000","last_change":"2024-04-06T12:32:02.674368+0000","last_active":"2024-04-06T12:32:04.946282+0000","last_peered":"2024-04-06T12:32:04.946282+0000","last_clean":"2024-04-06T12:32:04.946282+0000","last_became_active":"2024-04-06T12:32:02.674113+0000","last_became_peered":"2024-04-06T12:32:02.674113+0000","last_unstale":"2024-04-06T12:32:04.946282+0000","last_undegraded":"2024-04-06T12:32:04.946282+0000","last_fullsized":"2024-04-06T12:32:04.946282+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:21:50.594008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076733+0000","last_change":"2024-04-06T12:32:03.095614+0000","last_active":"2024-04-06T12:32:04.076733+0000","last_peered":"2024-04-06T12:32:04.076733+0000","last_clean":"2024-04-06T12:32:04.076733+0000","last_became_active":"2024-04-06T12:32:03.095324+0000","last_became_peered":"2024-04-06T12:32:03.095324+0000","last_unstale":"2024-04-06T12:32:04.076733+0000","last_undegraded":"2024-04-06T12:32:04.076733+0000","last_fullsized":"2024-04-06T12:32:04.076733+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:48:06.372887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"157'1","reported_seq":12,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034386+0000","last_change":"2024-04-06T12:32:02.673592+0000","last_active":"2024-04-06T12:32:05.034386+0000","last_peered":"2024-04-06T12:32:05.034386+0000","last_clean":"2024-04-06T12:32:05.034386+0000","last_became_active":"2024-04-06T12:32:02.673228+0000","last_became_peered":"2024-04-06T12:32:02.673228+0000","last_unstale":"2024-04-06T12:32:05.034386+0000","last_undegraded":"2024-04-06T12:32:05.034386+0000","last_fullsized":"2024-04-06T12:32:05.034386+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:37:59.756304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076157+0000","last_change":"2024-04-06T12:32:03.091263+0000","last_active":"2024-04-06T12:32:04.076157+0000","last_peered":"2024-04-06T12:32:04.076157+0000","last_clean":"2024-04-06T12:32:04.076157+0000","last_became_active":"2024-04-06T12:32:03.090998+0000","last_became_peered":"2024-04-06T12:32:03.090998+0000","last_unstale":"2024-04-06T12:32:04.076157+0000","last_undegraded":"2024-04-06T12:32:04.076157+0000","last_fullsized":"2024-04-06T12:32:04.076157+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:04:54.609161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076682+0000","last_change":"2024-04-06T12:32:02.666017+0000","last_active":"2024-04-06T12:32:04.076682+0000","last_peered":"2024-04-06T12:32:04.076682+0000","last_clean":"2024-04-06T12:32:04.076682+0000","last_became_active":"2024-04-06T12:32:02.665597+0000","last_became_peered":"2024-04-06T12:32:02.665597+0000","last_unstale":"2024-04-06T12:32:04.076682+0000","last_undegraded":"2024-04-06T12:32:04.076682+0000","last_fullsized":"2024-04-06T12:32:04.076682+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:34:34.986435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076608+0000","last_change":"2024-04-06T12:32:03.087353+0000","last_active":"2024-04-06T12:32:04.076608+0000","last_peered":"2024-04-06T12:32:04.076608+0000","last_clean":"2024-04-06T12:32:04.076608+0000","last_became_active":"2024-04-06T12:32:03.087041+0000","last_became_peered":"2024-04-06T12:32:03.087041+0000","last_unstale":"2024-04-06T12:32:04.076608+0000","last_undegraded":"2024-04-06T12:32:04.076608+0000","last_fullsized":"2024-04-06T12:32:04.076608+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:38:07.718139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"157'2","reported_seq":13,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.445540+0000","last_change":"2024-04-06T12:32:02.671169+0000","last_active":"2024-04-06T12:32:04.445540+0000","last_peered":"2024-04-06T12:32:04.445540+0000","last_clean":"2024-04-06T12:32:04.445540+0000","last_became_active":"2024-04-06T12:32:02.670908+0000","last_became_peered":"2024-04-06T12:32:02.670908+0000","last_unstale":"2024-04-06T12:32:04.445540+0000","last_undegraded":"2024-04-06T12:32:04.445540+0000","last_fullsized":"2024-04-06T12:32:04.445540+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:51:36.175382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034719+0000","last_change":"2024-04-06T12:32:03.099004+0000","last_active":"2024-04-06T12:32:05.034719+0000","last_peered":"2024-04-06T12:32:05.034719+0000","last_clean":"2024-04-06T12:32:05.034719+0000","last_became_active":"2024-04-06T12:32:03.098817+0000","last_became_peered":"2024-04-06T12:32:03.098817+0000","last_unstale":"2024-04-06T12:32:05.034719+0000","last_undegraded":"2024-04-06T12:32:05.034719+0000","last_fullsized":"2024-04-06T12:32:05.034719+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:20:15.688105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034780+0000","last_change":"2024-04-06T12:32:02.673048+0000","last_active":"2024-04-06T12:32:05.034780+0000","last_peered":"2024-04-06T12:32:05.034780+0000","last_clean":"2024-04-06T12:32:05.034780+0000","last_became_active":"2024-04-06T12:32:02.672698+0000","last_became_peered":"2024-04-06T12:32:02.672698+0000","last_unstale":"2024-04-06T12:32:05.034780+0000","last_undegraded":"2024-04-06T12:32:05.034780+0000","last_fullsized":"2024-04-06T12:32:05.034780+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:34:37.245104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077101+0000","last_change":"2024-04-06T12:32:03.096836+0000","last_active":"2024-04-06T12:32:04.077101+0000","last_peered":"2024-04-06T12:32:04.077101+0000","last_clean":"2024-04-06T12:32:04.077101+0000","last_became_active":"2024-04-06T12:32:03.096608+0000","last_became_peered":"2024-04-06T12:32:03.096608+0000","last_unstale":"2024-04-06T12:32:04.077101+0000","last_undegraded":"2024-04-06T12:32:04.077101+0000","last_fullsized":"2024-04-06T12:32:04.077101+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:35:14.481710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076714+0000","last_change":"2024-04-06T12:32:02.675977+0000","last_active":"2024-04-06T12:32:04.076714+0000","last_peered":"2024-04-06T12:32:04.076714+0000","last_clean":"2024-04-06T12:32:04.076714+0000","last_became_active":"2024-04-06T12:32:02.675688+0000","last_became_peered":"2024-04-06T12:32:02.675688+0000","last_unstale":"2024-04-06T12:32:04.076714+0000","last_undegraded":"2024-04-06T12:32:04.076714+0000","last_fullsized":"2024-04-06T12:32:04.076714+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:09:01.401665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034233+0000","last_change":"2024-04-06T12:32:03.092794+0000","last_active":"2024-04-06T12:32:05.034233+0000","last_peered":"2024-04-06T12:32:05.034233+0000","last_clean":"2024-04-06T12:32:05.034233+0000","last_became_active":"2024-04-06T12:32:03.092504+0000","last_became_peered":"2024-04-06T12:32:03.092504+0000","last_unstale":"2024-04-06T12:32:05.034233+0000","last_undegraded":"2024-04-06T12:32:05.034233+0000","last_fullsized":"2024-04-06T12:32:05.034233+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:20:54.013729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034565+0000","last_change":"2024-04-06T12:32:02.676790+0000","last_active":"2024-04-06T12:32:05.034565+0000","last_peered":"2024-04-06T12:32:05.034565+0000","last_clean":"2024-04-06T12:32:05.034565+0000","last_became_active":"2024-04-06T12:32:02.676256+0000","last_became_peered":"2024-04-06T12:32:02.676256+0000","last_unstale":"2024-04-06T12:32:05.034565+0000","last_undegraded":"2024-04-06T12:32:05.034565+0000","last_fullsized":"2024-04-06T12:32:05.034565+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:38:36.953076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076117+0000","last_change":"2024-04-06T12:32:03.107649+0000","last_active":"2024-04-06T12:32:04.076117+0000","last_peered":"2024-04-06T12:32:04.076117+0000","last_clean":"2024-04-06T12:32:04.076117+0000","last_became_active":"2024-04-06T12:32:03.107392+0000","last_became_peered":"2024-04-06T12:32:03.107392+0000","last_unstale":"2024-04-06T12:32:04.076117+0000","last_undegraded":"2024-04-06T12:32:04.076117+0000","last_fullsized":"2024-04-06T12:32:04.076117+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:09:37.978340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"157'2","reported_seq":13,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034852+0000","last_change":"2024-04-06T12:32:02.673512+0000","last_active":"2024-04-06T12:32:05.034852+0000","last_peered":"2024-04-06T12:32:05.034852+0000","last_clean":"2024-04-06T12:32:05.034852+0000","last_became_active":"2024-04-06T12:32:02.673277+0000","last_became_peered":"2024-04-06T12:32:02.673277+0000","last_unstale":"2024-04-06T12:32:05.034852+0000","last_undegraded":"2024-04-06T12:32:05.034852+0000","last_fullsized":"2024-04-06T12:32:05.034852+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:22:26.422194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.945764+0000","last_change":"2024-04-06T12:32:03.098167+0000","last_active":"2024-04-06T12:32:04.945764+0000","last_peered":"2024-04-06T12:32:04.945764+0000","last_clean":"2024-04-06T12:32:04.945764+0000","last_became_active":"2024-04-06T12:32:03.097853+0000","last_became_peered":"2024-04-06T12:32:03.097853+0000","last_unstale":"2024-04-06T12:32:04.945764+0000","last_undegraded":"2024-04-06T12:32:04.945764+0000","last_fullsized":"2024-04-06T12:32:04.945764+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:38:37.962121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.445708+0000","last_change":"2024-04-06T12:32:02.671193+0000","last_active":"2024-04-06T12:32:04.445708+0000","last_peered":"2024-04-06T12:32:04.445708+0000","last_clean":"2024-04-06T12:32:04.445708+0000","last_became_active":"2024-04-06T12:32:02.670946+0000","last_became_peered":"2024-04-06T12:32:02.670946+0000","last_unstale":"2024-04-06T12:32:04.445708+0000","last_undegraded":"2024-04-06T12:32:04.445708+0000","last_fullsized":"2024-04-06T12:32:04.445708+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:18:36.622724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076843+0000","last_change":"2024-04-06T12:32:03.104549+0000","last_active":"2024-04-06T12:32:04.076843+0000","last_peered":"2024-04-06T12:32:04.076843+0000","last_clean":"2024-04-06T12:32:04.076843+0000","last_became_active":"2024-04-06T12:32:03.104254+0000","last_became_peered":"2024-04-06T12:32:03.104254+0000","last_unstale":"2024-04-06T12:32:04.076843+0000","last_undegraded":"2024-04-06T12:32:04.076843+0000","last_fullsized":"2024-04-06T12:32:04.076843+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:25:42.693508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.445634+0000","last_change":"2024-04-06T12:32:02.663784+0000","last_active":"2024-04-06T12:32:04.445634+0000","last_peered":"2024-04-06T12:32:04.445634+0000","last_clean":"2024-04-06T12:32:04.445634+0000","last_became_active":"2024-04-06T12:32:02.663498+0000","last_became_peered":"2024-04-06T12:32:02.663498+0000","last_unstale":"2024-04-06T12:32:04.445634+0000","last_undegraded":"2024-04-06T12:32:04.445634+0000","last_fullsized":"2024-04-06T12:32:04.445634+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:55:47.076984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076242+0000","last_change":"2024-04-06T12:32:03.082479+0000","last_active":"2024-04-06T12:32:04.076242+0000","last_peered":"2024-04-06T12:32:04.076242+0000","last_clean":"2024-04-06T12:32:04.076242+0000","last_became_active":"2024-04-06T12:32:03.082240+0000","last_became_peered":"2024-04-06T12:32:03.082240+0000","last_unstale":"2024-04-06T12:32:04.076242+0000","last_undegraded":"2024-04-06T12:32:04.076242+0000","last_fullsized":"2024-04-06T12:32:04.076242+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:35.486850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034140+0000","last_change":"2024-04-06T12:32:02.673043+0000","last_active":"2024-04-06T12:32:05.034140+0000","last_peered":"2024-04-06T12:32:05.034140+0000","last_clean":"2024-04-06T12:32:05.034140+0000","last_became_active":"2024-04-06T12:32:02.672502+0000","last_became_peered":"2024-04-06T12:32:02.672502+0000","last_unstale":"2024-04-06T12:32:05.034140+0000","last_undegraded":"2024-04-06T12:32:05.034140+0000","last_fullsized":"2024-04-06T12:32:05.034140+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:18:36.622724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034072+0000","last_change":"2024-04-06T12:32:03.086348+0000","last_active":"2024-04-06T12:32:05.034072+0000","last_peered":"2024-04-06T12:32:05.034072+0000","last_clean":"2024-04-06T12:32:05.034072+0000","last_became_active":"2024-04-06T12:32:03.086093+0000","last_became_peered":"2024-04-06T12:32:03.086093+0000","last_unstale":"2024-04-06T12:32:05.034072+0000","last_undegraded":"2024-04-06T12:32:05.034072+0000","last_fullsized":"2024-04-06T12:32:05.034072+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:14:34.054920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.033990+0000","last_change":"2024-04-06T12:32:02.675255+0000","last_active":"2024-04-06T12:32:05.033990+0000","last_peered":"2024-04-06T12:32:05.033990+0000","last_clean":"2024-04-06T12:32:05.033990+0000","last_became_active":"2024-04-06T12:32:02.675016+0000","last_became_peered":"2024-04-06T12:32:02.675016+0000","last_unstale":"2024-04-06T12:32:05.033990+0000","last_undegraded":"2024-04-06T12:32:05.033990+0000","last_fullsized":"2024-04-06T12:32:05.033990+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:04:32.029292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034254+0000","last_change":"2024-04-06T12:32:03.081731+0000","last_active":"2024-04-06T12:32:05.034254+0000","last_peered":"2024-04-06T12:32:05.034254+0000","last_clean":"2024-04-06T12:32:05.034254+0000","last_became_active":"2024-04-06T12:32:03.081462+0000","last_became_peered":"2024-04-06T12:32:03.081462+0000","last_unstale":"2024-04-06T12:32:05.034254+0000","last_undegraded":"2024-04-06T12:32:05.034254+0000","last_fullsized":"2024-04-06T12:32:05.034254+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:54:15.592322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077083+0000","last_change":"2024-04-06T12:32:02.667429+0000","last_active":"2024-04-06T12:32:04.077083+0000","last_peered":"2024-04-06T12:32:04.077083+0000","last_clean":"2024-04-06T12:32:04.077083+0000","last_became_active":"2024-04-06T12:32:02.667155+0000","last_became_peered":"2024-04-06T12:32:02.667155+0000","last_unstale":"2024-04-06T12:32:04.077083+0000","last_undegraded":"2024-04-06T12:32:04.077083+0000","last_fullsized":"2024-04-06T12:32:04.077083+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:58:22.678856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034580+0000","last_change":"2024-04-06T12:32:03.095696+0000","last_active":"2024-04-06T12:32:05.034580+0000","last_peered":"2024-04-06T12:32:05.034580+0000","last_clean":"2024-04-06T12:32:05.034580+0000","last_became_active":"2024-04-06T12:32:03.095431+0000","last_became_peered":"2024-04-06T12:32:03.095431+0000","last_unstale":"2024-04-06T12:32:05.034580+0000","last_undegraded":"2024-04-06T12:32:05.034580+0000","last_fullsized":"2024-04-06T12:32:05.034580+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:11:29.631951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:05.034429+0000","last_change":"2024-04-06T12:32:02.674058+0000","last_active":"2024-04-06T12:32:05.034429+0000","last_peered":"2024-04-06T12:32:05.034429+0000","last_clean":"2024-04-06T12:32:05.034429+0000","last_became_active":"2024-04-06T12:32:02.673810+0000","last_became_peered":"2024-04-06T12:32:02.673810+0000","last_unstale":"2024-04-06T12:32:05.034429+0000","last_undegraded":"2024-04-06T12:32:05.034429+0000","last_fullsized":"2024-04-06T12:32:05.034429+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:58:22.678856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.077490+0000","last_change":"2024-04-06T12:32:03.101458+0000","last_active":"2024-04-06T12:32:04.077490+0000","last_peered":"2024-04-06T12:32:04.077490+0000","last_clean":"2024-04-06T12:32:04.077490+0000","last_became_active":"2024-04-06T12:32:03.099130+0000","last_became_peered":"2024-04-06T12:32:03.099130+0000","last_unstale":"2024-04-06T12:32:04.077490+0000","last_undegraded":"2024-04-06T12:32:04.077490+0000","last_fullsized":"2024-04-06T12:32:04.077490+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:32:44.600075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076266+0000","last_change":"2024-04-06T12:32:02.673686+0000","last_active":"2024-04-06T12:32:04.076266+0000","last_peered":"2024-04-06T12:32:04.076266+0000","last_clean":"2024-04-06T12:32:04.076266+0000","last_became_active":"2024-04-06T12:32:02.673427+0000","last_became_peered":"2024-04-06T12:32:02.673427+0000","last_unstale":"2024-04-06T12:32:04.076266+0000","last_undegraded":"2024-04-06T12:32:04.076266+0000","last_fullsized":"2024-04-06T12:32:04.076266+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:02:06.901506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":10,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.946057+0000","last_change":"2024-04-06T12:32:03.105937+0000","last_active":"2024-04-06T12:32:04.946057+0000","last_peered":"2024-04-06T12:32:04.946057+0000","last_clean":"2024-04-06T12:32:04.946057+0000","last_became_active":"2024-04-06T12:32:03.105682+0000","last_became_peered":"2024-04-06T12:32:03.105682+0000","last_unstale":"2024-04-06T12:32:04.946057+0000","last_undegraded":"2024-04-06T12:32:04.946057+0000","last_fullsized":"2024-04-06T12:32:04.946057+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:02.652297+0000","last_clean_scrub_stamp":"2024-04-06T12:32:02.652297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:55:48.086028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":11,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-06T12:32:04.076938+0000","last_change":"2024-04-06T12:32:02.675915+0000","last_active":"2024-04-06T12:32:04.076938+0000","last_peered":"2024-04-06T12:32:04.076938+0000","last_clean":"2024-04-06T12:32:04.076938+0000","last_became_active":"2024-04-06T12:32:02.675690+0000","last_became_peered":"2024-04-06T12:32:02.675690+0000","last_unstale":"2024-04-06T12:32:04.076938+0000","last_undegraded":"2024-04-06T12:32:04.076938+0000","last_fullsized":"2024-04-06T12:32:04.076938+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:32:01.643253+0000","last_clean_scrub_stamp":"2024-04-06T12:32:01.643253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:38:36.953076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"147'249","reported_seq":779,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-06T12:32:03.081691+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T12:32:03.081691+0000","last_peered":"2024-04-06T12:32:03.081691+0000","last_clean":"2024-04-06T12:32:03.081691+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T12:32:03.081691+0000","last_undegraded":"2024-04-06T12:32:03.081691+0000","last_fullsized":"2024-04-06T12:32:03.081691+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575085,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59876,"kb_used_data":2520,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311964,"statfs":{"total":96636764160,"available":96575451136,"internally_reserved":0,"allocated":2580480,"data_stored":1582157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52300000000000002,"15min":0.55000000000000004},"min":{"1min":0.249,"5min":0.249,"15min":0.219},"max":{"1min":0.71999999999999997,"5min":0.78200000000000003,"15min":1.143},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.53100000000000003,"15min":0.54900000000000004},"min":{"1min":0.34100000000000003,"5min":0.246,"15min":0.246},"max":{"1min":0.69299999999999995,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.54400000000000004}]},{"osd":1,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53600000000000003,"15min":0.55300000000000005},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.753,"5min":0.753,"15min":0.79100000000000004},"last":0.374},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54200000000000004,"15min":0.56499999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.27900000000000003},"max":{"1min":0.67700000000000005,"5min":0.754,"15min":0.79600000000000004},"last":0.47699999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.55000000000000004},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.249},"max":{"1min":0.73199999999999998,"5min":0.76500000000000001,"15min":0.83299999999999996},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56599999999999995,"15min":0.57699999999999996},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.73899999999999999,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.57199999999999995}]},{"osd":3,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54500000000000004,"15min":0.57099999999999995},"min":{"1min":0.41199999999999998,"5min":0.35099999999999998,"15min":0.34399999999999997},"max":{"1min":0.68799999999999994,"5min":0.73999999999999999,"15min":1.357},"last":0.502},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57199999999999995,"15min":0.58799999999999997},"min":{"1min":0.36899999999999999,"5min":0.35599999999999998,"15min":0.33100000000000002},"max":{"1min":0.71999999999999997,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.70499999999999996}]},{"osd":4,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.44600000000000001,"15min":0.44500000000000001},"min":{"1min":0.28100000000000003,"5min":0.27000000000000002,"15min":0.25},"max":{"1min":0.63600000000000001,"5min":0.71399999999999997,"15min":0.72999999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.52900000000000003,"15min":0.52800000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.32800000000000001},"max":{"1min":0.72999999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.63500000000000001}]},{"osd":5,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.50600000000000001,"15min":0.498},"min":{"1min":0.39800000000000002,"5min":0.29399999999999998,"15min":0.26600000000000001},"max":{"1min":0.78500000000000003,"5min":0.81899999999999995,"15min":0.86099999999999999},"last":0.755},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.40999999999999998,"5min":0.32800000000000001,"15min":0.314},"max":{"1min":0.74099999999999999,"5min":0.79900000000000004,"15min":0.81599999999999995},"last":0.67500000000000004}]},{"osd":6,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.84499999999999997,"15min":0.88600000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56699999999999995,"15min":0.56299999999999994},"min":{"1min":0.40699999999999997,"5min":0.312,"15min":0.30399999999999999},"max":{"1min":0.73299999999999998,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.72499999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575084,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1974272,"data_stored":989788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.52100000000000002,"15min":0.53700000000000003},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.26200000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.82099999999999995},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52200000000000002,"15min":0.53700000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.245},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.48099999999999998}]},{"osd":1,"last update":"Sat Apr 6 12:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.55100000000000005,"15min":0.58199999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.59999999999999998},"min":{"1min":0.40899999999999997,"5min":0.38400000000000001,"15min":0.221},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.67100000000000004}]},{"osd":2,"last update":"Sat Apr 6 12:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54200000000000004,"15min":0.55300000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.30499999999999999},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54900000000000004,"15min":0.56799999999999995},"min":{"1min":0.378,"5min":0.32300000000000001,"15min":0.254},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.56699999999999995}]},{"osd":3,"last update":"Sat Apr 6 12:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55800000000000005,"15min":0.57799999999999996},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.23300000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.61799999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.746}]},{"osd":4,"last update":"Sat Apr 6 12:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52000000000000002,"15min":0.501},"min":{"1min":0.34599999999999997,"5min":0.28599999999999998,"15min":0.27500000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.629},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54700000000000004,"15min":0.53300000000000003},"min":{"1min":0.34899999999999998,"5min":0.29799999999999999,"15min":0.28799999999999998},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.82199999999999995},"last":0.76000000000000001}]},{"osd":5,"last update":"Sat Apr 6 12:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50900000000000001,"15min":0.49399999999999999},"min":{"1min":0.30399999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.70899999999999996,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52600000000000002,"15min":0.51300000000000001},"min":{"1min":0.313,"5min":0.31,"15min":0.27000000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.73399999999999999}]},{"osd":7,"last update":"Sat Apr 6 12:31:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55200000000000005,"15min":0.53900000000000003},"min":{"1min":0.36099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.75900000000000001,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57899999999999996,"15min":0.56299999999999994},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.32100000000000001},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.77900000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1970176,"data_stored":989186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.35999999999999999,"15min":0.36299999999999999},"min":{"1min":0.27800000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.53700000000000003,"5min":0.55400000000000005,"15min":0.58999999999999997},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.377,"15min":0.39200000000000002},"min":{"1min":0.26800000000000002,"5min":0.245,"15min":0.215},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.27800000000000002}]},{"osd":2,"last update":"Sat Apr 6 12:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.38,"15min":0.39300000000000002},"min":{"1min":0.27500000000000002,"5min":0.25,"15min":0.19400000000000001},"max":{"1min":0.49199999999999999,"5min":0.58099999999999996,"15min":0.67600000000000005},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.42899999999999999},"min":{"1min":0.32000000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.57099999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.39200000000000002}]},{"osd":3,"last update":"Sat Apr 6 12:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.44700000000000001},"min":{"1min":0.314,"5min":0.29199999999999998,"15min":0.28399999999999997},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.41699999999999998},"min":{"1min":0.29799999999999999,"5min":0.29099999999999998,"15min":0.249},"max":{"1min":0.55200000000000005,"5min":0.65900000000000003,"15min":0.86299999999999999},"last":0.35799999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58899999999999997,"15min":0.61499999999999999},"min":{"1min":0.45500000000000002,"5min":0.38900000000000001,"15min":0.34799999999999998},"max":{"1min":0.73699999999999999,"5min":0.872,"15min":0.872},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.437,"5min":0.36299999999999999,"15min":0.30399999999999999},"max":{"1min":0.81399999999999995,"5min":0.93799999999999994,"15min":0.94899999999999995},"last":0.496}]},{"osd":5,"last update":"Sat Apr 6 12:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60099999999999998,"15min":0.61499999999999999},"min":{"1min":0.505,"5min":0.41999999999999998,"15min":0.38700000000000001},"max":{"1min":0.79000000000000004,"5min":0.83999999999999997,"15min":0.91300000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.625,"5min":0.61399999999999999,"15min":0.63},"min":{"1min":0.45400000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.87,"5min":0.871,"15min":0.999},"last":0.47999999999999998}]},{"osd":6,"last update":"Sat Apr 6 12:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.60399999999999998,"15min":0.61899999999999999},"min":{"1min":0.44600000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81499999999999995,"5min":0.82599999999999996,"15min":0.98599999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.627,"15min":0.64800000000000002},"min":{"1min":0.48399999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81699999999999995,"5min":0.84799999999999998,"15min":0.92600000000000005},"last":0.58799999999999997}]},{"osd":7,"last update":"Sat Apr 6 12:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.64100000000000001,"15min":0.64400000000000002},"min":{"1min":0.437,"5min":0.437,"15min":0.38800000000000001},"max":{"1min":0.82999999999999996,"5min":0.85699999999999998,"15min":1.0329999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.61299999999999999,"15min":0.63600000000000001},"min":{"1min":0.436,"5min":0.40999999999999998,"15min":0.40200000000000002},"max":{"1min":0.85599999999999998,"5min":0.91400000000000003,"15min":0.95599999999999996},"last":0.50800000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575083,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37732,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334108,"statfs":{"total":96636764160,"available":96598126592,"internally_reserved":0,"allocated":2568192,"data_stored":1579540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 12:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35799999999999998,"15min":0.38100000000000001},"min":{"1min":0.26000000000000001,"5min":0.216,"15min":0.19},"max":{"1min":0.495,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.434},{"interface":"front","average":{"1min":0.371,"5min":0.376,"15min":0.40500000000000003},"min":{"1min":0.255,"5min":0.23499999999999999,"15min":0.217},"max":{"1min":0.55600000000000005,"5min":0.625,"15min":1.4359999999999999},"last":0.378}]},{"osd":2,"last update":"Sat Apr 6 12:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.39200000000000002,"15min":0.40899999999999997},"min":{"1min":0.315,"5min":0.255,"15min":0.19700000000000001},"max":{"1min":0.58599999999999997,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.40899999999999997,"15min":0.42999999999999999},"min":{"1min":0.28499999999999998,"5min":0.26800000000000002,"15min":0.24299999999999999},"max":{"1min":0.47099999999999997,"5min":0.89900000000000002,"15min":1.4790000000000001},"last":0.45400000000000001}]},{"osd":3,"last update":"Sat Apr 6 12:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.39600000000000002,"15min":0.41699999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.252},"max":{"1min":0.51300000000000001,"5min":0.64300000000000002,"15min":1.25},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.432,"15min":0.46000000000000002},"min":{"1min":0.35299999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":0.74099999999999999,"15min":0.748},"last":0.51300000000000001}]},{"osd":4,"last update":"Sat Apr 6 12:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.58499999999999996,"15min":0.622},"min":{"1min":0.37,"5min":0.37,"15min":0.35699999999999998},"max":{"1min":0.65400000000000003,"5min":1.1100000000000001,"15min":1.514},"last":0.499},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58699999999999997,"15min":0.629},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72299999999999998,"5min":0.93999999999999995,"15min":1.202},"last":0.62}]},{"osd":5,"last update":"Sat Apr 6 12:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.57899999999999996,"15min":0.623},"min":{"1min":0.40999999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76500000000000001,"5min":0.88400000000000001,"15min":1.0920000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60899999999999999,"15min":0.64400000000000002},"min":{"1min":0.42799999999999999,"5min":0.40899999999999997,"15min":0.315},"max":{"1min":0.72999999999999998,"5min":1.006,"15min":1.006},"last":0.63600000000000001}]},{"osd":6,"last update":"Sat Apr 6 12:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.60599999999999998,"15min":0.64900000000000002},"min":{"1min":0.442,"5min":0.376,"15min":0.376},"max":{"1min":0.69199999999999995,"5min":1.038,"15min":1.038},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61399999999999999,"15min":0.66000000000000003},"min":{"1min":0.47099999999999997,"5min":0.40300000000000002,"15min":0.38100000000000001},"max":{"1min":0.74399999999999999,"5min":1.052,"15min":1.613},"last":0.55800000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:31:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.61199999999999999,"15min":0.66200000000000003},"min":{"1min":0.441,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.72099999999999997,"5min":1.0820000000000001,"15min":1.54},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60699999999999998,"15min":0.65700000000000003},"min":{"1min":0.41699999999999998,"5min":0.39300000000000002,"15min":0.38400000000000001},"max":{"1min":0.78300000000000003,"5min":1.014,"15min":1.498},"last":0.52200000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575085,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55176,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316664,"statfs":{"total":96636764160,"available":96580263936,"internally_reserved":0,"allocated":1961984,"data_stored":983112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36299999999999999,"15min":0.38200000000000001},"min":{"1min":0.23799999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.55100000000000005,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.38600000000000001,"15min":0.40600000000000003},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.55600000000000005,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.26900000000000002}]},{"osd":1,"last update":"Sat Apr 6 12:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.373,"15min":0.39700000000000002},"min":{"1min":0.222,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54000000000000004,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.40500000000000003,"15min":0.42699999999999999},"min":{"1min":0.29399999999999998,"5min":0.28000000000000003,"15min":0.27100000000000002},"max":{"1min":0.60399999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.39500000000000002}]},{"osd":3,"last update":"Sat Apr 6 12:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42099999999999999,"15min":0.44600000000000001},"min":{"1min":0.28799999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.52400000000000002,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41999999999999998,"15min":0.436},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":0.97099999999999997,"15min":1.0549999999999999},"last":0.40999999999999998}]},{"osd":4,"last update":"Sat Apr 6 12:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58499999999999996,"15min":0.60999999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.35199999999999998},"max":{"1min":0.82099999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57999999999999996,"15min":0.62},"min":{"1min":0.40100000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77300000000000002,"5min":0.91200000000000003,"15min":0.92600000000000005},"last":0.65900000000000003}]},{"osd":5,"last update":"Sat Apr 6 12:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.58699999999999997,"15min":0.63},"min":{"1min":0.377,"5min":0.374,"15min":0.374},"max":{"1min":0.68200000000000005,"5min":1.0369999999999999,"15min":1.6870000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.59799999999999998,"15min":0.623},"min":{"1min":0.42199999999999999,"5min":0.35399999999999998,"15min":0.31},"max":{"1min":0.81000000000000005,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.68100000000000005}]},{"osd":6,"last update":"Sat Apr 6 12:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.59899999999999998,"15min":0.63},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.36399999999999999},"max":{"1min":0.79900000000000004,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.59899999999999998,"15min":0.64500000000000002},"min":{"1min":0.41199999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71399999999999997,"5min":1.0760000000000001,"15min":1.728},"last":0.80600000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59899999999999998,"15min":0.64100000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79400000000000004,"5min":0.95699999999999996,"15min":0.95799999999999996},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.61699999999999999,"15min":0.65700000000000003},"min":{"1min":0.47999999999999998,"5min":0.41499999999999998,"15min":0.39000000000000001},"max":{"1min":0.76000000000000001,"5min":1.0049999999999999,"15min":1.758},"last":0.78500000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575084,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1986560,"data_stored":991769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35799999999999998,"15min":0.36599999999999999},"min":{"1min":0.246,"5min":0.23599999999999999,"15min":0.21099999999999999},"max":{"1min":0.52600000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.434},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.35799999999999998,"15min":0.379},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.502,"5min":0.59099999999999997,"15min":0.73499999999999999},"last":0.40500000000000003}]},{"osd":1,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.37,"15min":0.378},"min":{"1min":0.23400000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.629,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39700000000000002,"15min":0.41099999999999998},"min":{"1min":0.28999999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.62,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.49399999999999999}]},{"osd":2,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.439,"15min":0.45500000000000002},"min":{"1min":0.28899999999999998,"5min":0.28799999999999998,"15min":0.27200000000000002},"max":{"1min":0.59099999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.45400000000000001,"15min":0.47199999999999998},"min":{"1min":0.34499999999999997,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.58599999999999997,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.53200000000000003}]},{"osd":4,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56799999999999995,"15min":0.59799999999999998},"min":{"1min":0.435,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.95999999999999996},"last":0.62},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.61699999999999999},"min":{"1min":0.437,"5min":0.39600000000000002,"15min":0.23699999999999999},"max":{"1min":0.85099999999999998,"5min":0.90100000000000002,"15min":0.93500000000000005},"last":0.71199999999999997}]},{"osd":5,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61599999999999999,"15min":0.64800000000000002},"min":{"1min":0.45200000000000001,"5min":0.374,"15min":0.309},"max":{"1min":0.80200000000000005,"5min":0.92200000000000004,"15min":0.96699999999999997},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.627,"15min":0.65500000000000003},"min":{"1min":0.44700000000000001,"5min":0.41499999999999998,"15min":0.38200000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.69099999999999995}]},{"osd":6,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59499999999999997,"15min":0.61399999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.36899999999999999},"max":{"1min":0.84999999999999998,"5min":1.026,"15min":1.026},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63700000000000001,"15min":0.65100000000000002},"min":{"1min":0.39900000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.94299999999999995,"5min":1.716,"15min":1.716},"last":0.66300000000000003}]},{"osd":7,"last update":"Sat Apr 6 12:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61199999999999999,"15min":0.63600000000000001},"min":{"1min":0.42499999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.92400000000000004,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.63100000000000001,"15min":0.65700000000000003},"min":{"1min":0.47499999999999998,"5min":0.40100000000000002,"15min":0.39500000000000002},"max":{"1min":0.875,"5min":1.002,"15min":1.002},"last":0.74399999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575084,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":1970176,"data_stored":990135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:31:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53700000000000003,"15min":0.55300000000000005},"min":{"1min":0.30399999999999999,"5min":0.29299999999999998,"15min":0.23899999999999999},"max":{"1min":0.66500000000000004,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.503,"5min":0.53600000000000003,"15min":0.55500000000000005},"min":{"1min":0.33900000000000002,"5min":0.33800000000000002,"15min":0.25900000000000001},"max":{"1min":0.73699999999999999,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.72699999999999998}]},{"osd":1,"last update":"Sat Apr 6 12:31:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.57999999999999996,"15min":0.59899999999999998},"min":{"1min":0.377,"5min":0.315,"15min":0.29799999999999999},"max":{"1min":0.75700000000000001,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.57599999999999996,"15min":0.58799999999999997},"min":{"1min":0.45200000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.74099999999999999,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.62}]},{"osd":2,"last update":"Sat Apr 6 12:31:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.56599999999999995},"min":{"1min":0.34200000000000003,"5min":0.312,"15min":0.312},"max":{"1min":0.78200000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.55600000000000005,"15min":0.57599999999999996},"min":{"1min":0.40500000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.76800000000000002,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.49099999999999999}]},{"osd":3,"last update":"Sat Apr 6 12:31:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55300000000000005,"15min":0.55800000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78400000000000003,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.57099999999999995,"15min":0.59099999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.751,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.79500000000000004}]},{"osd":5,"last update":"Sat Apr 6 12:31:47 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27200000000000002},"max":{"1min":0.64000000000000001,"5min":0.73599999999999999,"15min":0.83599999999999997},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.48199999999999998,"15min":0.49299999999999999},"min":{"1min":0.34200000000000003,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72499999999999998,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.65700000000000003}]},{"osd":6,"last update":"Sat Apr 6 12:31:47 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.501,"15min":0.498},"min":{"1min":0.379,"5min":0.307,"15min":0.30599999999999999},"max":{"1min":0.76400000000000001,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53600000000000003,"15min":0.53400000000000003},"min":{"1min":0.36399999999999999,"5min":0.34100000000000003,"15min":0.32100000000000001},"max":{"1min":0.68999999999999995,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.55700000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:31:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53200000000000003},"min":{"1min":0.39600000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.754,"5min":0.77600000000000002,"15min":0.79300000000000004},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57699999999999996,"15min":0.56699999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80000000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.84899999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575084,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51088,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320752,"statfs":{"total":96636764160,"available":96584450048,"internally_reserved":0,"allocated":1970176,"data_stored":989138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 12:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54000000000000004,"15min":0.56499999999999995},"min":{"1min":0.35799999999999998,"5min":0.35199999999999998,"15min":0.29099999999999998},"max":{"1min":0.72399999999999998,"5min":0.81999999999999995,"15min":0.82599999999999996},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.51400000000000001,"15min":0.54600000000000004},"min":{"1min":0.40699999999999997,"5min":0.29499999999999998,"15min":0.22500000000000001},"max":{"1min":0.70599999999999996,"5min":0.748,"15min":0.80300000000000005},"last":0.61499999999999999}]},{"osd":1,"last update":"Sat Apr 6 12:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52100000000000002,"15min":0.55300000000000005},"min":{"1min":0.40600000000000003,"5min":0.30399999999999999,"15min":0.26000000000000001},"max":{"1min":0.70399999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52500000000000002,"15min":0.55000000000000004},"min":{"1min":0.372,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.72199999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.60299999999999998}]},{"osd":2,"last update":"Sat Apr 6 12:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53100000000000003,"15min":0.56100000000000005},"min":{"1min":0.38,"5min":0.33200000000000002,"15min":0.21099999999999999},"max":{"1min":0.67900000000000005,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.53800000000000003,"15min":0.56299999999999994},"min":{"1min":0.41699999999999998,"5min":0.35999999999999999,"15min":0.27400000000000002},"max":{"1min":0.67800000000000005,"5min":0.77100000000000002,"15min":0.83599999999999997},"last":0.54100000000000004}]},{"osd":3,"last update":"Sat Apr 6 12:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54700000000000004,"15min":0.56899999999999995},"min":{"1min":0.37,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70699999999999996,"5min":0.83799999999999997,"15min":0.874},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.55800000000000005,"15min":0.58899999999999997},"min":{"1min":0.41999999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.89600000000000002},"last":0.52200000000000002}]},{"osd":4,"last update":"Sat Apr 6 12:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.34000000000000002,"5min":0.28899999999999998,"15min":0.28299999999999997},"max":{"1min":0.73199999999999998,"5min":0.755,"15min":0.81200000000000006},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54300000000000004,"15min":0.53500000000000003},"min":{"1min":0.39600000000000002,"5min":0.29499999999999998,"15min":0.29299999999999998},"max":{"1min":0.72399999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.70199999999999996}]},{"osd":6,"last update":"Sat Apr 6 12:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47799999999999998,"15min":0.47899999999999998},"min":{"1min":0.307,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.70499999999999996,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.627},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.48199999999999998,"15min":0.49099999999999999},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.29699999999999999},"max":{"1min":0.65400000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.56100000000000005}]},{"osd":7,"last update":"Sat Apr 6 12:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.48999999999999999,"15min":0.503},"min":{"1min":0.32700000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.70599999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.753},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.60299999999999998},"min":{"1min":0.47899999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.751,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.66700000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T12:32:07.585 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-06T12:32:07.585 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-06T12:32:07.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.753+0000 7f1666e49700 1 Processor -- start 2024-04-06T12:32:07.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.753+0000 7f1666e49700 1 -- start start 2024-04-06T12:32:07.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f1666e49700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f166006ae40 0x7f16600b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:07.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f1666e49700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16600b3990 con 0x7f166015d7b0 2024-04-06T12:32:07.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f1666e49700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16600b3060 con 0x7f166015ddf0 2024-04-06T12:32:07.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f1666e49700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166015e950 con 0x7f166006ae40 2024-04-06T12:32:07.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f1664be5700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f166015d7b0 0x7f166006a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43998/0 (socket says 172.21.15.189:43998) 2024-04-06T12:32:07.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f1664be5700 1 -- 172.21.15.189:0/3896373482 learned_addr learned my addr 172.21.15.189:0/3896373482 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:32:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f165ffff700 1 --2- 172.21.15.189:0/3896373482 >> v2:172.21.15.192:3300/0 conn(0x7f166006ae40 0x7f16600b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f165f7fe700 1 -- 172.21.15.189:0/3896373482 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262521075 0 0) 0x7f16600b3990 con 0x7f166015d7b0 2024-04-06T12:32:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f165f7fe700 1 -- 172.21.15.189:0/3896373482 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16500035b0 con 0x7f166015d7b0 2024-04-06T12:32:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f165f7fe700 1 -- 172.21.15.189:0/3896373482 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365556634 0 0) 0x7f16600b3060 con 0x7f166015ddf0 2024-04-06T12:32:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f165f7fe700 1 -- 172.21.15.189:0/3896373482 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16600b3990 con 0x7f166015ddf0 2024-04-06T12:32:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f165f7fe700 1 -- 172.21.15.189:0/3896373482 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3611269269 0 0) 0x7f16500035b0 con 0x7f166015d7b0 2024-04-06T12:32:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f165f7fe700 1 -- 172.21.15.189:0/3896373482 >> v2:172.21.15.192:3300/0 conn(0x7f166006ae40 msgr2=0x7f16600b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f165f7fe700 1 --2- 172.21.15.189:0/3896373482 >> v2:172.21.15.192:3300/0 conn(0x7f166006ae40 0x7f16600b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f165f7fe700 1 -- 172.21.15.189:0/3896373482 >> v1:172.21.15.192:6789/0 conn(0x7f166015ddf0 legacy=0x7f166015e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f165f7fe700 1 -- 172.21.15.189:0/3896373482 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16600b3340 con 0x7f166015d7b0 2024-04-06T12:32:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f165f7fe700 1 -- 172.21.15.189:0/3896373482 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f1654001670 con 0x7f166015d7b0 2024-04-06T12:32:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f165f7fe700 1 -- 172.21.15.189:0/3896373482 >> v1:172.21.15.189:6789/0 conn(0x7f166015d7b0 legacy=0x7f166006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:07.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f165f7fe700 1 --2- 172.21.15.189:0/3896373482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1650003be0 0x7f1650003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:07.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f165f7fe700 1 --2- 172.21.15.189:0/3896373482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1650004850 0x7f1650004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:07.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f165f7fe700 1 --2- 172.21.15.189:0/3896373482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1650005260 0x7f165000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:07.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f165f7fe700 1 -- 172.21.15.189:0/3896373482 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165000c160 con 0x7f1650005260 2024-04-06T12:32:07.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f165f7fe700 1 -- 172.21.15.189:0/3896373482 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165000c2d0 con 0x7f1650004850 2024-04-06T12:32:07.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.754+0000 7f165f7fe700 1 -- 172.21.15.189:0/3896373482 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f165000c490 con 0x7f1650003be0 2024-04-06T12:32:07.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.755+0000 7f16653e6700 1 --2- 172.21.15.189:0/3896373482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1650005260 0x7f165000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:07.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.755+0000 7f1664be5700 1 --2- 172.21.15.189:0/3896373482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1650003be0 0x7f1650003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:07.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.755+0000 7f165ffff700 1 --2- 172.21.15.189:0/3896373482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1650004850 0x7f1650004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:07.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.755+0000 7f16653e6700 1 -- 172.21.15.189:0/3896373482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1650003be0 msgr2=0x7f1650003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:07.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.755+0000 7f16653e6700 1 --2- 172.21.15.189:0/3896373482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1650003be0 0x7f1650003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:07.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.755+0000 7f16653e6700 1 -- 172.21.15.189:0/3896373482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1650004850 msgr2=0x7f1650004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:07.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.755+0000 7f16653e6700 1 --2- 172.21.15.189:0/3896373482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1650004850 0x7f1650004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:07.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.755+0000 7f16653e6700 1 -- 172.21.15.189:0/3896373482 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16500035b0 con 0x7f1650005260 2024-04-06T12:32:07.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.755+0000 7f16653e6700 1 -- 172.21.15.189:0/3896373482 reap_dead start 2024-04-06T12:32:07.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.755+0000 7f16653e6700 1 --2- 172.21.15.189:0/3896373482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1650005260 0x7f165000ba10 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f165800c2f0 tx=0x7f1650005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e0494240523ac09 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:07.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.755+0000 7f165f7fe700 1 -- 172.21.15.189:0/3896373482 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658001f30 con 0x7f1650005260 2024-04-06T12:32:07.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.755+0000 7f165f7fe700 1 -- 172.21.15.189:0/3896373482 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1658003a00 con 0x7f1650005260 2024-04-06T12:32:07.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.755+0000 7f165f7fe700 1 -- 172.21.15.189:0/3896373482 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658003db0 con 0x7f1650005260 2024-04-06T12:32:07.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.756+0000 7f1666e49700 1 -- 172.21.15.189:0/3896373482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1650005260 msgr2=0x7f165000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:07.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.756+0000 7f1666e49700 1 --2- 172.21.15.189:0/3896373482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1650005260 0x7f165000ba10 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f165800c2f0 tx=0x7f1650005880 comp rx=0 tx=0).stop 2024-04-06T12:32:07.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.756+0000 7f1666e49700 1 -- 172.21.15.189:0/3896373482 shutdown_connections 2024-04-06T12:32:07.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.756+0000 7f1666e49700 1 --2- 172.21.15.189:0/3896373482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1650005260 0x7f165000ba10 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:07.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.756+0000 7f1666e49700 1 -- 172.21.15.189:0/3896373482 >> 172.21.15.189:0/3896373482 conn(0x7f16600d00b0 msgr2=0x7f16600d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:07.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.756+0000 7f1666e49700 1 -- 172.21.15.189:0/3896373482 shutdown_connections 2024-04-06T12:32:07.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.756+0000 7f1666e49700 1 -- 172.21.15.189:0/3896373482 wait complete. 2024-04-06T12:32:07.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.757+0000 7f1666e49700 1 Processor -- start 2024-04-06T12:32:07.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.757+0000 7f1666e49700 1 -- start start 2024-04-06T12:32:07.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f1666e49700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1650005260 0x7f166014a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:07.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f1666e49700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16601589c0 0x7f166014ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:07.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f1666e49700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f166015bf20 0x7f16601e65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:07.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f1666e49700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1660158e50 con 0x7f1650005260 2024-04-06T12:32:07.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f1666e49700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1660159130 con 0x7f16601589c0 2024-04-06T12:32:07.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f1666e49700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1660159410 con 0x7f166015bf20 2024-04-06T12:32:07.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f1664be5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1650005260 0x7f166014a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:07.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f1664be5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1650005260 0x7f166014a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60788/0 (socket says 172.21.15.189:60788) 2024-04-06T12:32:07.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f1664be5700 1 -- 172.21.15.189:0/2033956840 learned_addr learned my addr 172.21.15.189:0/2033956840 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:07.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f165ffff700 1 --2- 172.21.15.189:0/2033956840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16601589c0 0x7f166014ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:07.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f16653e6700 1 --2- 172.21.15.189:0/2033956840 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f166015bf20 0x7f16601e65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:07.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f1664be5700 1 -- 172.21.15.189:0/2033956840 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f166015bf20 msgr2=0x7f16601e65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:07.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f1664be5700 1 --2- 172.21.15.189:0/2033956840 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f166015bf20 0x7f16601e65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:07.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f1664be5700 1 -- 172.21.15.189:0/2033956840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16601589c0 msgr2=0x7f166014ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:07.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f1664be5700 1 --2- 172.21.15.189:0/2033956840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16601589c0 0x7f166014ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:07.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f1664be5700 1 -- 172.21.15.189:0/2033956840 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166015ac30 con 0x7f1650005260 2024-04-06T12:32:07.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f1664be5700 1 --2- 172.21.15.189:0/2033956840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1650005260 0x7f166014a500 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f1654004f80 tx=0x7f16540073f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:07.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f165d7fa700 1 -- 172.21.15.189:0/2033956840 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654011580 con 0x7f1650005260 2024-04-06T12:32:07.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f165d7fa700 1 -- 172.21.15.189:0/2033956840 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1654011700 con 0x7f1650005260 2024-04-06T12:32:07.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f165d7fa700 1 -- 172.21.15.189:0/2033956840 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654011ab0 con 0x7f1650005260 2024-04-06T12:32:07.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.758+0000 7f1666e49700 1 -- 172.21.15.189:0/2033956840 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166014aa40 con 0x7f1650005260 2024-04-06T12:32:07.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.759+0000 7f1666e49700 1 -- 172.21.15.189:0/2033956840 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166014b270 con 0x7f1650005260 2024-04-06T12:32:07.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.759+0000 7f1666e49700 1 -- 172.21.15.189:0/2033956840 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1660067bb0 con 0x7f1650005260 2024-04-06T12:32:07.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.760+0000 7f165d7fa700 1 -- 172.21.15.189:0/2033956840 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f1654011c10 con 0x7f1650005260 2024-04-06T12:32:07.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.760+0000 7f165d7fa700 1 --2- 172.21.15.189:0/2033956840 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f164406adb0 0x7f164406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:07.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.760+0000 7f165d7fa700 1 -- 172.21.15.189:0/2033956840 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1654089620 con 0x7f1650005260 2024-04-06T12:32:07.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.760+0000 7f165ffff700 1 --2- 172.21.15.189:0/2033956840 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f164406adb0 0x7f164406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:07.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.761+0000 7f165ffff700 1 --2- 172.21.15.189:0/2033956840 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f164406adb0 0x7f164406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f166015c660 tx=0x7f164c008040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:07.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:07.765+0000 7f165d7fa700 1 -- 172.21.15.189:0/2033956840 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f16540545b0 con 0x7f1650005260 2024-04-06T12:32:08.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.038+0000 7f1666e49700 1 -- 172.21.15.189:0/2033956840 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f1660074990 con 0x7f1650005260 2024-04-06T12:32:08.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.042+0000 7f165d7fa700 1 -- 172.21.15.189:0/2033956840 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 190+0+0 (secure 0 0 0) 0x7f1654057bd0 con 0x7f1650005260 2024-04-06T12:32:08.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.044+0000 7f1666e49700 1 -- 172.21.15.189:0/2033956840 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f164406adb0 msgr2=0x7f164406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:08.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.045+0000 7f1666e49700 1 --2- 172.21.15.189:0/2033956840 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f164406adb0 0x7f164406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f166015c660 tx=0x7f164c008040 comp rx=0 tx=0).stop 2024-04-06T12:32:08.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.045+0000 7f1666e49700 1 -- 172.21.15.189:0/2033956840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1650005260 msgr2=0x7f166014a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:08.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.045+0000 7f1666e49700 1 --2- 172.21.15.189:0/2033956840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1650005260 0x7f166014a500 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f1654004f80 tx=0x7f16540073f0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.045+0000 7f1666e49700 1 -- 172.21.15.189:0/2033956840 shutdown_connections 2024-04-06T12:32:08.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.045+0000 7f1666e49700 1 --2- 172.21.15.189:0/2033956840 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f166015bf20 0x7f16601e65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.045+0000 7f1666e49700 1 --2- 172.21.15.189:0/2033956840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1650005260 0x7f166014a500 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.045+0000 7f1666e49700 1 --2- 172.21.15.189:0/2033956840 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f164406adb0 0x7f164406d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.045+0000 7f1666e49700 1 --2- 172.21.15.189:0/2033956840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16601589c0 0x7f166014ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.045+0000 7f1666e49700 1 -- 172.21.15.189:0/2033956840 >> 172.21.15.189:0/2033956840 conn(0x7f16600d00b0 msgr2=0x7f1660156c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:08.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.045+0000 7f1666e49700 1 -- 172.21.15.189:0/2033956840 shutdown_connections 2024-04-06T12:32:08.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.045+0000 7f1666e49700 1 -- 172.21.15.189:0/2033956840 wait complete. 2024-04-06T12:32:08.044 INFO:teuthology.orchestra.run.smithi189.stderr:updated caps for client.0 2024-04-06T12:32:08.055 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T12:32:08.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.179+0000 7fa8c949a700 1 Processor -- start 2024-04-06T12:32:08.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.179+0000 7fa8c949a700 1 -- start start 2024-04-06T12:32:08.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.180+0000 7fa8c949a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa8c4158fe0 0x7fa8c406a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:08.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.180+0000 7fa8c949a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c415f9e0 con 0x7fa8c406ac70 2024-04-06T12:32:08.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.180+0000 7fa8c949a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c415f0d0 con 0x7fa8c415c2d0 2024-04-06T12:32:08.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.180+0000 7fa8c949a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8c406b0f0 con 0x7fa8c4158fe0 2024-04-06T12:32:08.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.180+0000 7fa8c27fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa8c406ac70 0x7fa8c415bbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44006/0 (socket says 172.21.15.189:44006) 2024-04-06T12:32:08.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.180+0000 7fa8c27fc700 1 -- 172.21.15.189:0/2171264320 learned_addr learned my addr 172.21.15.189:0/2171264320 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:32:08.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.180+0000 7fa8c2ffd700 1 --2- 172.21.15.189:0/2171264320 >> v2:172.21.15.192:3300/0 conn(0x7fa8c4158fe0 0x7fa8c406a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:08.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.180+0000 7fa8c1ffb700 1 -- 172.21.15.189:0/2171264320 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648559565 0 0) 0x7fa8c415f9e0 con 0x7fa8c406ac70 2024-04-06T12:32:08.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.180+0000 7fa8c1ffb700 1 -- 172.21.15.189:0/2171264320 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b00035b0 con 0x7fa8c406ac70 2024-04-06T12:32:08.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.180+0000 7fa8c1ffb700 1 -- 172.21.15.189:0/2171264320 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1187129016 0 0) 0x7fa8b00035b0 con 0x7fa8c406ac70 2024-04-06T12:32:08.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.180+0000 7fa8c1ffb700 1 -- 172.21.15.189:0/2171264320 >> v2:172.21.15.192:3300/0 conn(0x7fa8c4158fe0 msgr2=0x7fa8c406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:08.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.180+0000 7fa8c1ffb700 1 --2- 172.21.15.189:0/2171264320 >> v2:172.21.15.192:3300/0 conn(0x7fa8c4158fe0 0x7fa8c406a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.180+0000 7fa8c1ffb700 1 -- 172.21.15.189:0/2171264320 >> v1:172.21.15.192:6789/0 conn(0x7fa8c415c2d0 legacy=0x7fa8c40ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:08.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.180+0000 7fa8c1ffb700 1 -- 172.21.15.189:0/2171264320 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c415c8f0 con 0x7fa8c406ac70 2024-04-06T12:32:08.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.180+0000 7fa8c1ffb700 1 -- 172.21.15.189:0/2171264320 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fa8ac001670 con 0x7fa8c406ac70 2024-04-06T12:32:08.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c1ffb700 1 -- 172.21.15.189:0/2171264320 >> v1:172.21.15.189:6789/0 conn(0x7fa8c406ac70 legacy=0x7fa8c415bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:08.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c1ffb700 1 --2- 172.21.15.189:0/2171264320 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8b0003a00 0x7fa8b0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:08.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c1ffb700 1 --2- 172.21.15.189:0/2171264320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8b0004850 0x7fa8b0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:08.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c1ffb700 1 --2- 172.21.15.189:0/2171264320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8b0005210 0x7fa8b000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:08.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c1ffb700 1 -- 172.21.15.189:0/2171264320 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c406b0f0 con 0x7fa8b0005210 2024-04-06T12:32:08.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c1ffb700 1 -- 172.21.15.189:0/2171264320 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b000c1b0 con 0x7fa8b0004850 2024-04-06T12:32:08.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c1ffb700 1 -- 172.21.15.189:0/2171264320 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b000c370 con 0x7fa8b0003a00 2024-04-06T12:32:08.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c37fe700 1 --2- 172.21.15.189:0/2171264320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8b0005210 0x7fa8b000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:08.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c27fc700 1 --2- 172.21.15.189:0/2171264320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8b0004850 0x7fa8b0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:08.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c2ffd700 1 --2- 172.21.15.189:0/2171264320 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8b0003a00 0x7fa8b0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:08.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c37fe700 1 -- 172.21.15.189:0/2171264320 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8b0003a00 msgr2=0x7fa8b0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:08.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c37fe700 1 --2- 172.21.15.189:0/2171264320 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8b0003a00 0x7fa8b0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c37fe700 1 -- 172.21.15.189:0/2171264320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8b0004850 msgr2=0x7fa8b0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:08.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c37fe700 1 --2- 172.21.15.189:0/2171264320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8b0004850 0x7fa8b0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c37fe700 1 -- 172.21.15.189:0/2171264320 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b00035b0 con 0x7fa8b0005210 2024-04-06T12:32:08.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c2ffd700 1 --2- 172.21.15.189:0/2171264320 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8b0003a00 0x7fa8b0003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:32:08.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c37fe700 1 -- 172.21.15.189:0/2171264320 reap_dead start 2024-04-06T12:32:08.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c37fe700 1 --2- 172.21.15.189:0/2171264320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8b0005210 0x7fa8b000ba10 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400c240 tx=0x7fa8b0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=422b3f1a41cbd5fc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:08.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c1ffb700 1 -- 172.21.15.189:0/2171264320 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4001df0 con 0x7fa8b0005210 2024-04-06T12:32:08.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c1ffb700 1 -- 172.21.15.189:0/2171264320 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b400cda0 con 0x7fa8b0005210 2024-04-06T12:32:08.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.181+0000 7fa8c1ffb700 1 -- 172.21.15.189:0/2171264320 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4003c40 con 0x7fa8b0005210 2024-04-06T12:32:08.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.182+0000 7fa8c949a700 1 -- 172.21.15.189:0/2171264320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8b0005210 msgr2=0x7fa8b000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:08.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.182+0000 7fa8c949a700 1 --2- 172.21.15.189:0/2171264320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8b0005210 0x7fa8b000ba10 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400c240 tx=0x7fa8b0005880 comp rx=0 tx=0).stop 2024-04-06T12:32:08.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.182+0000 7fa8c949a700 1 -- 172.21.15.189:0/2171264320 shutdown_connections 2024-04-06T12:32:08.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.182+0000 7fa8c949a700 1 --2- 172.21.15.189:0/2171264320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8b0005210 0x7fa8b000ba10 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.182+0000 7fa8c949a700 1 -- 172.21.15.189:0/2171264320 >> 172.21.15.189:0/2171264320 conn(0x7fa8c40d00b0 msgr2=0x7fa8c40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:08.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.182+0000 7fa8c949a700 1 -- 172.21.15.189:0/2171264320 shutdown_connections 2024-04-06T12:32:08.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.182+0000 7fa8c949a700 1 -- 172.21.15.189:0/2171264320 wait complete. 2024-04-06T12:32:08.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.183+0000 7fa8c949a700 1 Processor -- start 2024-04-06T12:32:08.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.184+0000 7fa8c949a700 1 -- start start 2024-04-06T12:32:08.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.184+0000 7fa8c949a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8b0005210 0x7fa8c40cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:08.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.184+0000 7fa8c949a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8c4131530 0x7fa8c41f4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:08.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.184+0000 7fa8c949a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8c412f4a0 0x7fa8c41432d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:08.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.184+0000 7fa8c949a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c40cc450 con 0x7fa8c4131530 2024-04-06T12:32:08.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.184+0000 7fa8c949a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c4135870 con 0x7fa8b0005210 2024-04-06T12:32:08.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.184+0000 7fa8c949a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c41f5400 con 0x7fa8c412f4a0 2024-04-06T12:32:08.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.185+0000 7fa8c27fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8c4131530 0x7fa8c41f4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:08.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.185+0000 7fa8c27fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8c4131530 0x7fa8c41f4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60802/0 (socket says 172.21.15.189:60802) 2024-04-06T12:32:08.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.185+0000 7fa8c27fc700 1 -- 172.21.15.189:0/3329776038 learned_addr learned my addr 172.21.15.189:0/3329776038 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:08.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.185+0000 7fa8c2ffd700 1 --2- 172.21.15.189:0/3329776038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8b0005210 0x7fa8c40cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:08.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.185+0000 7fa8c37fe700 1 --2- 172.21.15.189:0/3329776038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8c412f4a0 0x7fa8c41432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:08.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.185+0000 7fa8c27fc700 1 -- 172.21.15.189:0/3329776038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8c412f4a0 msgr2=0x7fa8c41432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:08.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.185+0000 7fa8c27fc700 1 --2- 172.21.15.189:0/3329776038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8c412f4a0 0x7fa8c41432d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.185+0000 7fa8c27fc700 1 -- 172.21.15.189:0/3329776038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8b0005210 msgr2=0x7fa8c40cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:08.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.185+0000 7fa8c27fc700 1 --2- 172.21.15.189:0/3329776038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8b0005210 0x7fa8c40cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.185+0000 7fa8c27fc700 1 -- 172.21.15.189:0/3329776038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c41319f0 con 0x7fa8c4131530 2024-04-06T12:32:08.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.185+0000 7fa8c2ffd700 1 --2- 172.21.15.189:0/3329776038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8b0005210 0x7fa8c40cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:32:08.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.185+0000 7fa8c37fe700 1 --2- 172.21.15.189:0/3329776038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8c412f4a0 0x7fa8c41432d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:32:08.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.185+0000 7fa8c27fc700 1 --2- 172.21.15.189:0/3329776038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8c4131530 0x7fa8c41f4ec0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac007290 tx=0x7fa8ac007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:08.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.185+0000 7fa8bb7fe700 1 -- 172.21.15.189:0/3329776038 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac0014c0 con 0x7fa8c4131530 2024-04-06T12:32:08.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.185+0000 7fa8bb7fe700 1 -- 172.21.15.189:0/3329776038 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8ac00f850 con 0x7fa8c4131530 2024-04-06T12:32:08.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.185+0000 7fa8c949a700 1 -- 172.21.15.189:0/3329776038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c4131cd0 con 0x7fa8c4131530 2024-04-06T12:32:08.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.185+0000 7fa8bb7fe700 1 -- 172.21.15.189:0/3329776038 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac00fc00 con 0x7fa8c4131530 2024-04-06T12:32:08.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.185+0000 7fa8c949a700 1 -- 172.21.15.189:0/3329776038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c4132290 con 0x7fa8c4131530 2024-04-06T12:32:08.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.186+0000 7fa8c949a700 1 -- 172.21.15.189:0/3329776038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c4067bb0 con 0x7fa8c4131530 2024-04-06T12:32:08.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.187+0000 7fa8bb7fe700 1 -- 172.21.15.189:0/3329776038 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fa8ac00fd60 con 0x7fa8c4131530 2024-04-06T12:32:08.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.187+0000 7fa8bb7fe700 1 --2- 172.21.15.189:0/3329776038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8a406adb0 0x7fa8a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:08.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.187+0000 7fa8bb7fe700 1 -- 172.21.15.189:0/3329776038 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa8ac089610 con 0x7fa8c4131530 2024-04-06T12:32:08.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.187+0000 7fa8c2ffd700 1 --2- 172.21.15.189:0/3329776038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8a406adb0 0x7fa8a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:08.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.187+0000 7fa8c2ffd700 1 --2- 172.21.15.189:0/3329776038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8a406adb0 0x7fa8a406d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa8c412f370 tx=0x7fa8bc008040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:08.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.192+0000 7fa8bb7fe700 1 -- 172.21.15.189:0/3329776038 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa8ac0545a0 con 0x7fa8c4131530 2024-04-06T12:32:08.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.430+0000 7fa8c949a700 1 -- 172.21.15.189:0/3329776038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8c4132450 con 0x7fa8c4131530 2024-04-06T12:32:08.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.430+0000 7fa8bb7fe700 1 -- 172.21.15.189:0/3329776038 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4748 (secure 0 0 0) 0x7fa8ac057bc0 con 0x7fa8c4131530 2024-04-06T12:32:08.428 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:32:08.429 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":70,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3697447910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3697447910},{"type":"v1","addr":"172.21.15.189:6837","nonce":3697447910}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":7469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/3633157989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3633157989},{"type":"v1","addr":"172.21.15.192:6834","nonce":3633157989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2721135386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2721135386},{"type":"v1","addr":"172.21.15.189:6835","nonce":2721135386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:32:03.067806+0000","modified":"2024-04-06T12:32:06.080510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9144":{"gid":9144,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/3068270636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":3068270636},{"type":"v1","addr":"172.21.15.192:6835","nonce":3068270636}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-06T12:32:08.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.432+0000 7fa8c949a700 1 -- 172.21.15.189:0/3329776038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8a406adb0 msgr2=0x7fa8a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:08.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.432+0000 7fa8c949a700 1 --2- 172.21.15.189:0/3329776038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8a406adb0 0x7fa8a406d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa8c412f370 tx=0x7fa8bc008040 comp rx=0 tx=0).stop 2024-04-06T12:32:08.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.432+0000 7fa8c949a700 1 -- 172.21.15.189:0/3329776038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8c4131530 msgr2=0x7fa8c41f4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:08.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.432+0000 7fa8c949a700 1 --2- 172.21.15.189:0/3329776038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8c4131530 0x7fa8c41f4ec0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac007290 tx=0x7fa8ac007650 comp rx=0 tx=0).stop 2024-04-06T12:32:08.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.432+0000 7fa8c949a700 1 -- 172.21.15.189:0/3329776038 shutdown_connections 2024-04-06T12:32:08.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.432+0000 7fa8c949a700 1 --2- 172.21.15.189:0/3329776038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8c412f4a0 0x7fa8c41432d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.432+0000 7fa8c949a700 1 --2- 172.21.15.189:0/3329776038 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa8a406adb0 0x7fa8a406d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.432+0000 7fa8c949a700 1 --2- 172.21.15.189:0/3329776038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8b0005210 0x7fa8c40cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.432+0000 7fa8c949a700 1 --2- 172.21.15.189:0/3329776038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8c4131530 0x7fa8c41f4ec0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.432+0000 7fa8c949a700 1 -- 172.21.15.189:0/3329776038 >> 172.21.15.189:0/3329776038 conn(0x7fa8c40d00b0 msgr2=0x7fa8c4140310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:08.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.432+0000 7fa8c949a700 1 -- 172.21.15.189:0/3329776038 shutdown_connections 2024-04-06T12:32:08.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.432+0000 7fa8c949a700 1 -- 172.21.15.189:0/3329776038 wait complete. 2024-04-06T12:32:08.432 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 70 2024-04-06T12:32:08.640 INFO:tasks.ceph.mds.b.smithi192.stderr:2024-04-06T12:32:08.643+0000 7f9041838700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T12:32:08.845 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-06T12:32:03.067806+0000', 'modified': '2024-04-06T12:32:06.080510+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9144}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9144': {'gid': 9144, 'name': 'b', 'rank': 0, 'incarnation': 66, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.192:6835/3068270636', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6833', 'nonce': 3068270636}, {'type': 'v1', 'addr': '172.21.15.192:6835', 'nonce': 3068270636}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-06T12:32:08.845 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-06T12:32:08.846 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-06T12:32:08.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.974+0000 7fc1d0699700 1 Processor -- start 2024-04-06T12:32:08.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.974+0000 7fc1d0699700 1 -- start start 2024-04-06T12:32:08.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.975+0000 7fc1d0699700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc1c815d590 0x7fc1c80b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:08.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.975+0000 7fc1d0699700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1c80d3890 0x7fc1c80b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:08.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.975+0000 7fc1d0699700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c80b5cc0 con 0x7fc1c80b4ed0 2024-04-06T12:32:08.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.975+0000 7fc1d0699700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c80b58c0 con 0x7fc1c815d590 2024-04-06T12:32:08.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.975+0000 7fc1d0699700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c80d1500 con 0x7fc1c80d3890 2024-04-06T12:32:08.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.975+0000 7fc1ce435700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc1c815d590 0x7fc1c80b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:08.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.975+0000 7fc1ce435700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc1c815d590 0x7fc1c80b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:39124/0 (socket says 172.21.15.189:39124) 2024-04-06T12:32:08.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.975+0000 7fc1ce435700 1 -- 172.21.15.189:0/3730309653 learned_addr learned my addr 172.21.15.189:0/3730309653 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:08.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.975+0000 7fc1cec36700 1 --2- 172.21.15.189:0/3730309653 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1c80d3890 0x7fc1c80b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:08.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.975+0000 7fc1cdc34700 1 --1- 172.21.15.189:0/3730309653 >> v1:172.21.15.192:6789/0 conn(0x7fc1c80b4ed0 0x7fc1c80b6580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:37226/0 (socket says 172.21.15.189:37226) 2024-04-06T12:32:08.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.976+0000 7fc1ce435700 1 -- 172.21.15.189:0/3730309653 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1c80d3890 msgr2=0x7fc1c80b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:08.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.976+0000 7fc1ce435700 1 --2- 172.21.15.189:0/3730309653 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1c80d3890 0x7fc1c80b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.976+0000 7fc1ce435700 1 -- 172.21.15.189:0/3730309653 >> v1:172.21.15.192:6789/0 conn(0x7fc1c80b4ed0 legacy=0x7fc1c80b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:08.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.976+0000 7fc1ce435700 1 -- 172.21.15.189:0/3730309653 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c80d17e0 con 0x7fc1c815d590 2024-04-06T12:32:08.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.976+0000 7fc1cd433700 1 -- 172.21.15.189:0/3730309653 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968161045 0 0) 0x7fc1c80b5cc0 con 0x7fc1c80b4ed0 2024-04-06T12:32:08.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.976+0000 7fc1cec36700 1 --2- 172.21.15.189:0/3730309653 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1c80d3890 0x7fc1c80b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:32:08.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.976+0000 7fc1ce435700 1 --2- 172.21.15.189:0/3730309653 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1c815d590 0x7fc1c80b8d70 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc1b80098f0 tx=0x7fc1b8009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb46c2a858145217 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:08.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.976+0000 7fc1cd433700 1 -- 172.21.15.189:0/3730309653 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8011830 con 0x7fc1c815d590 2024-04-06T12:32:08.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.976+0000 7fc1cd433700 1 -- 172.21.15.189:0/3730309653 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b8011990 con 0x7fc1c815d590 2024-04-06T12:32:08.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.976+0000 7fc1cd433700 1 -- 172.21.15.189:0/3730309653 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8011d20 con 0x7fc1c815d590 2024-04-06T12:32:08.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.977+0000 7fc1d0699700 1 -- 172.21.15.189:0/3730309653 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1c815d590 msgr2=0x7fc1c80b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:08.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.977+0000 7fc1d0699700 1 --2- 172.21.15.189:0/3730309653 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1c815d590 0x7fc1c80b8d70 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc1b80098f0 tx=0x7fc1b8009c00 comp rx=0 tx=0).stop 2024-04-06T12:32:08.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.977+0000 7fc1d0699700 1 -- 172.21.15.189:0/3730309653 shutdown_connections 2024-04-06T12:32:08.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.977+0000 7fc1d0699700 1 --2- 172.21.15.189:0/3730309653 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1c80d3890 0x7fc1c80b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.977+0000 7fc1d0699700 1 --2- 172.21.15.189:0/3730309653 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1c815d590 0x7fc1c80b8d70 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.977+0000 7fc1d0699700 1 -- 172.21.15.189:0/3730309653 >> 172.21.15.189:0/3730309653 conn(0x7fc1c806bbc0 msgr2=0x7fc1c806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:08.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.977+0000 7fc1d0699700 1 -- 172.21.15.189:0/3730309653 shutdown_connections 2024-04-06T12:32:08.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.977+0000 7fc1d0699700 1 -- 172.21.15.189:0/3730309653 wait complete. 2024-04-06T12:32:08.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.978+0000 7fc1d0699700 1 Processor -- start 2024-04-06T12:32:08.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.978+0000 7fc1d0699700 1 -- start start 2024-04-06T12:32:08.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1d0699700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1c80b4ed0 0x7fc1c814ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:08.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1d0699700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1c80d3890 0x7fc1c80ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:08.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1d0699700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1c815d590 0x7fc1c815b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:08.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1d0699700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c80cc7d0 con 0x7fc1c80b4ed0 2024-04-06T12:32:08.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1d0699700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c814f450 con 0x7fc1c80d3890 2024-04-06T12:32:08.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1d0699700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c80cbb50 con 0x7fc1c815d590 2024-04-06T12:32:08.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1ce435700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1c80b4ed0 0x7fc1c814ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:08.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1ce435700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1c80b4ed0 0x7fc1c814ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60818/0 (socket says 172.21.15.189:60818) 2024-04-06T12:32:08.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1ce435700 1 -- 172.21.15.189:0/4097887926 learned_addr learned my addr 172.21.15.189:0/4097887926 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:08.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1cdc34700 1 --2- 172.21.15.189:0/4097887926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1c80d3890 0x7fc1c80ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:08.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1cec36700 1 --2- 172.21.15.189:0/4097887926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1c815d590 0x7fc1c815b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:08.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1ce435700 1 -- 172.21.15.189:0/4097887926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1c815d590 msgr2=0x7fc1c815b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:08.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1ce435700 1 --2- 172.21.15.189:0/4097887926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1c815d590 0x7fc1c815b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1ce435700 1 -- 172.21.15.189:0/4097887926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1c80d3890 msgr2=0x7fc1c80ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:08.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1ce435700 1 --2- 172.21.15.189:0/4097887926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1c80d3890 0x7fc1c80ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:08.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1ce435700 1 -- 172.21.15.189:0/4097887926 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c0009a70 con 0x7fc1c80b4ed0 2024-04-06T12:32:08.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1ce435700 1 --2- 172.21.15.189:0/4097887926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1c80b4ed0 0x7fc1c814ef10 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8009c40 tx=0x7fc1b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:08.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1beffd700 1 -- 172.21.15.189:0/4097887926 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b80044d0 con 0x7fc1c80b4ed0 2024-04-06T12:32:08.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1beffd700 1 -- 172.21.15.189:0/4097887926 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b801e070 con 0x7fc1c80b4ed0 2024-04-06T12:32:08.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1d0699700 1 -- 172.21.15.189:0/4097887926 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b8009550 con 0x7fc1c80b4ed0 2024-04-06T12:32:08.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.979+0000 7fc1beffd700 1 -- 172.21.15.189:0/4097887926 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b80038e0 con 0x7fc1c80b4ed0 2024-04-06T12:32:08.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.980+0000 7fc1d0699700 1 -- 172.21.15.189:0/4097887926 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c80cc110 con 0x7fc1c80b4ed0 2024-04-06T12:32:08.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.981+0000 7fc1beffd700 1 -- 172.21.15.189:0/4097887926 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fc1b8003a40 con 0x7fc1c80b4ed0 2024-04-06T12:32:08.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.981+0000 7fc1beffd700 1 --2- 172.21.15.189:0/4097887926 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc1ac06ade0 0x7fc1ac06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:08.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.981+0000 7fc1beffd700 1 -- 172.21.15.189:0/4097887926 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc1b808a3f0 con 0x7fc1c80b4ed0 2024-04-06T12:32:08.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.981+0000 7fc1cdc34700 1 --2- 172.21.15.189:0/4097887926 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc1ac06ade0 0x7fc1ac06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:08.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.982+0000 7fc1cdc34700 1 --2- 172.21.15.189:0/4097887926 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc1ac06ade0 0x7fc1ac06d2a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc1c815cbc0 tx=0x7fc1c4007de0 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:08.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.983+0000 7fc1dc84db80 1 Processor -- start 2024-04-06T12:32:08.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.983+0000 7fc1dc84db80 1 -- start start 2024-04-06T12:32:08.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.984+0000 7fc1dc84db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 0x55a3d49c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:08.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.984+0000 7fc1dc84db80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c9fc0 0x55a3d49a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:08.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.984+0000 7fc1dc84db80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d499f980 0x55a3d49a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:08.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.984+0000 7fc1dc84db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d48f9940 con 0x55a3d49c4c20 2024-04-06T12:32:08.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.984+0000 7fc1dc84db80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d48f7300 con 0x55a3d499f980 2024-04-06T12:32:08.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.984+0000 7fc1dc84db80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a3d48f2bf0 con 0x55a3d49c9fc0 2024-04-06T12:32:08.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.984+0000 7fc1ce435700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 0x55a3d49c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:08.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.984+0000 7fc1ce435700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 0x55a3d49c8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60832/0 (socket says 172.21.15.189:60832) 2024-04-06T12:32:08.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.984+0000 7fc1ce435700 1 -- 172.21.15.189:0/2191105593 learned_addr learned my addr 172.21.15.189:0/2191105593 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:08.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.984+0000 7fc1cec36700 1 --2- 172.21.15.189:0/2191105593 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d499f980 0x55a3d49a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:08.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.984+0000 7fc1cdc34700 1 --2- 172.21.15.189:0/2191105593 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c9fc0 0x55a3d49a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:08.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.985+0000 7fc1ce435700 1 -- 172.21.15.189:0/2191105593 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c9fc0 msgr2=0x55a3d49a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:09.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.985+0000 7fc1ce435700 1 --2- 172.21.15.189:0/2191105593 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c9fc0 0x55a3d49a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.985+0000 7fc1ce435700 1 -- 172.21.15.189:0/2191105593 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d499f980 msgr2=0x55a3d49a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:09.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.985+0000 7fc1ce435700 1 --2- 172.21.15.189:0/2191105593 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d499f980 0x55a3d49a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.985+0000 7fc1ce435700 1 -- 172.21.15.189:0/2191105593 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a3d4720b10 con 0x55a3d49c4c20 2024-04-06T12:32:09.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.985+0000 7fc1cdc34700 1 --2- 172.21.15.189:0/2191105593 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c9fc0 0x55a3d49a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:32:09.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.985+0000 7fc1ce435700 1 --2- 172.21.15.189:0/2191105593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 0x55a3d49c8a60 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8060ef0 tx=0x7fc1b8006d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b1dd5723adabf12 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:09.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.985+0000 7fc1abfff700 1 -- 172.21.15.189:0/2191105593 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8059030 con 0x55a3d49c4c20 2024-04-06T12:32:09.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.985+0000 7fc1abfff700 1 -- 172.21.15.189:0/2191105593 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b805ce60 con 0x55a3d49c4c20 2024-04-06T12:32:09.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.985+0000 7fc1abfff700 1 -- 172.21.15.189:0/2191105593 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8061420 con 0x55a3d49c4c20 2024-04-06T12:32:09.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.985+0000 7fc1dc84db80 1 -- 172.21.15.189:0/2191105593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 msgr2=0x55a3d49c8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:09.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.985+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/2191105593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 0x55a3d49c8a60 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8060ef0 tx=0x7fc1b8006d00 comp rx=0 tx=0).stop 2024-04-06T12:32:09.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.985+0000 7fc1dc84db80 1 -- 172.21.15.189:0/2191105593 shutdown_connections 2024-04-06T12:32:09.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.985+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/2191105593 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d499f980 0x55a3d49a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.985+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/2191105593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 0x55a3d49c8a60 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.985+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/2191105593 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c9fc0 0x55a3d49a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.985+0000 7fc1dc84db80 1 -- 172.21.15.189:0/2191105593 >> 172.21.15.189:0/2191105593 conn(0x55a3d4900650 msgr2=0x55a3d49e73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:09.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.986+0000 7fc1dc84db80 1 -- 172.21.15.189:0/2191105593 shutdown_connections 2024-04-06T12:32:09.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.986+0000 7fc1dc84db80 1 -- 172.21.15.189:0/2191105593 wait complete. 2024-04-06T12:32:09.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.986+0000 7fc1dc84db80 1 -- start start 2024-04-06T12:32:09.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.986+0000 7fc1dc84db80 1 Processor -- start 2024-04-06T12:32:09.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.987+0000 7fc1dc84db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d499f980 0x55a3d49cce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:09.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.987+0000 7fc1dc84db80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c4c20 0x55a3d49cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:09.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.987+0000 7fc1dc84db80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d49c9fc0 0x55a3d49c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:09.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.987+0000 7fc1dc84db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d488ed30 con 0x55a3d499f980 2024-04-06T12:32:09.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.987+0000 7fc1dc84db80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d488e9d0 con 0x55a3d49c9fc0 2024-04-06T12:32:09.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.987+0000 7fc1dc84db80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a3d4776650 con 0x55a3d49c4c20 2024-04-06T12:32:09.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.987+0000 7fc1dc84db80 10 client.0 _ll_register_callbacks cb 0x55a3d4898420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:32:09.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.987+0000 7fc1ce435700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d499f980 0x55a3d49cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:09.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.987+0000 7fc1ce435700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d499f980 0x55a3d49cce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60848/0 (socket says 172.21.15.189:60848) 2024-04-06T12:32:09.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.987+0000 7fc1ce435700 1 -- 172.21.15.189:0/2256777030 learned_addr learned my addr 172.21.15.189:0/2256777030 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:09.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.987+0000 7fc1cec36700 1 --2- 172.21.15.189:0/2256777030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d49c9fc0 0x55a3d49c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:09.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.987+0000 7fc1cdc34700 1 --2- 172.21.15.189:0/2256777030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c4c20 0x55a3d49cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:09.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.987+0000 7fc1ce435700 1 -- 172.21.15.189:0/2256777030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c4c20 msgr2=0x55a3d49cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:09.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.987+0000 7fc1ce435700 1 --2- 172.21.15.189:0/2256777030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c4c20 0x55a3d49cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1ce435700 1 -- 172.21.15.189:0/2256777030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d49c9fc0 msgr2=0x55a3d49c2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:09.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1ce435700 1 --2- 172.21.15.189:0/2256777030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d49c9fc0 0x55a3d49c2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1ce435700 1 -- 172.21.15.189:0/2256777030 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c80cc110 con 0x55a3d499f980 2024-04-06T12:32:09.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1ce435700 1 --2- 172.21.15.189:0/2256777030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d499f980 0x55a3d49cce90 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8003fd0 tx=0x7fc1b808a070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7109193ad48c383f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:09.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1a97fa700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:32:09.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1a97fa700 1 -- 172.21.15.189:0/2256777030 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8059030 con 0x55a3d499f980 2024-04-06T12:32:09.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1a97fa700 1 -- 172.21.15.189:0/2256777030 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(158..158 src has 1..158) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc1b808a0e0 con 0x55a3d499f980 2024-04-06T12:32:09.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1dc84db80 1 -- 172.21.15.189:0/2256777030 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a3d48b3330 con 0x55a3d499f980 2024-04-06T12:32:09.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1a97fa700 1 -- 172.21.15.189:0/2256777030 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1b8055740 con 0x55a3d499f980 2024-04-06T12:32:09.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1dc84db80 10 client.9261 fetch_fsmap learned FSMap version 70 2024-04-06T12:32:09.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1dc84db80 1 -- 172.21.15.189:0/2256777030 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x55a3d48b8e50 con 0x55a3d499f980 2024-04-06T12:32:09.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1a97fa700 1 -- 172.21.15.189:0/2256777030 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a3d48b3330 con 0x55a3d499f980 2024-04-06T12:32:09.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1a97fa700 1 -- 172.21.15.189:0/2256777030 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 70) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc1b8088730 con 0x55a3d499f980 2024-04-06T12:32:09.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1dc84db80 10 client.9261 fetch_fsmap finished waiting for FSMap version 70 2024-04-06T12:32:09.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1dc84db80 10 client.9261 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-06T12:32:09.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1dc84db80 20 client.9261 populate_metadata read hostname 'smithi189' 2024-04-06T12:32:09.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/2256777030 >> [v2:172.21.15.192:6833/3068270636,v1:172.21.15.192:6835/3068270636] conn(0x55a3d49c70e0 0x55a3d49cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:09.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1dc84db80 4 client.9261 mds_command: new command op to 9144 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-06T12:32:09.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1dc84db80 1 -- 172.21.15.189:0/2256777030 --> [v2:172.21.15.192:6833/3068270636,v1:172.21.15.192:6835/3068270636] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a3d48eb6b0 con 0x55a3d49c70e0 2024-04-06T12:32:09.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.988+0000 7fc1cdc34700 1 --2- 172.21.15.189:0/2256777030 >> [v2:172.21.15.192:6833/3068270636,v1:172.21.15.192:6835/3068270636] conn(0x55a3d49c70e0 0x55a3d49cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:09.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.989+0000 7fc1a97fa700 10 client.9261 ms_handle_connect on v2:172.21.15.192:6833/3068270636 2024-04-06T12:32:09.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.989+0000 7fc1cdc34700 1 --2- 172.21.15.189:0/2256777030 >> [v2:172.21.15.192:6833/3068270636,v1:172.21.15.192:6835/3068270636] conn(0x55a3d49c70e0 0x55a3d49cf750 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f53d61534c9a6f9 server_cookie=4955a61a86c2e4fb in_seq=0 out_seq=0 2024-04-06T12:32:09.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.989+0000 7fc1a97fa700 1 -- 172.21.15.189:0/2256777030 <== mds.0 v2:172.21.15.192:6833/3068270636 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a3d48eb6b0 con 0x55a3d49c70e0 2024-04-06T12:32:09.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.989+0000 7fc1a97fa700 10 client.9261 handle_command_reply: tid=0 2024-04-06T12:32:09.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.989+0000 7fc1dc84db80 1 client.9261 shutdown 2024-04-06T12:32:09.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.990+0000 7fc1dc84db80 1 -- 172.21.15.189:0/2256777030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d499f980 msgr2=0x55a3d49cce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:09.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.990+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/2256777030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d499f980 0x55a3d49cce90 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8003fd0 tx=0x7fc1b808a070 comp rx=0 tx=0).stop 2024-04-06T12:32:09.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.990+0000 7fc1dc84db80 1 -- 172.21.15.189:0/2256777030 shutdown_connections 2024-04-06T12:32:09.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.990+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/2256777030 >> [v2:172.21.15.192:6833/3068270636,v1:172.21.15.192:6835/3068270636] conn(0x55a3d49c70e0 0x55a3d49cf750 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.990+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/2256777030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d49c9fc0 0x55a3d49c2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.990+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/2256777030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d499f980 0x55a3d49cce90 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.990+0000 7fc1a97fa700 0 client.9261 ms_handle_reset on v2:172.21.15.192:6833/3068270636 2024-04-06T12:32:09.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.990+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/2256777030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c4c20 0x55a3d49cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.990+0000 7fc1dc84db80 1 -- 172.21.15.189:0/2256777030 >> 172.21.15.189:0/2256777030 conn(0x55a3d4900650 msgr2=0x55a3d49e7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:09.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.990+0000 7fc1dc84db80 1 -- 172.21.15.189:0/2256777030 shutdown_connections 2024-04-06T12:32:09.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.990+0000 7fc1dc84db80 1 -- 172.21.15.189:0/2256777030 wait complete. 2024-04-06T12:32:09.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:08.990+0000 7fc1dc84db80 20 client.9261 trim_cache size 0 max 16384 2024-04-06T12:32:09.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.007+0000 7fc1dc84db80 1 Processor -- start 2024-04-06T12:32:09.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.008+0000 7fc1dc84db80 1 -- start start 2024-04-06T12:32:09.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.008+0000 7fc1dc84db80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d499f980 0x55a3d49c5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:09.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.008+0000 7fc1dc84db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 0x55a3d49c5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:09.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.008+0000 7fc1dc84db80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c9fc0 0x55a3d49e7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:09.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.008+0000 7fc1dc84db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d499c870 con 0x55a3d49c4c20 2024-04-06T12:32:09.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.008+0000 7fc1dc84db80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d49ccc20 con 0x55a3d499f980 2024-04-06T12:32:09.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.008+0000 7fc1dc84db80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a3d49c6760 con 0x55a3d49c9fc0 2024-04-06T12:32:09.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.008+0000 7fc1cdc34700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 0x55a3d49c5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:09.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.008+0000 7fc1cdc34700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 0x55a3d49c5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60856/0 (socket says 172.21.15.189:60856) 2024-04-06T12:32:09.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.008+0000 7fc1cdc34700 1 -- 172.21.15.189:0/782305808 learned_addr learned my addr 172.21.15.189:0/782305808 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:09.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.009+0000 7fc1cdc34700 1 -- 172.21.15.189:0/782305808 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c9fc0 msgr2=0x55a3d49e7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:09.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.009+0000 7fc1cdc34700 1 --2- 172.21.15.189:0/782305808 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c9fc0 0x55a3d49e7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.009+0000 7fc1cdc34700 1 -- 172.21.15.189:0/782305808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d499f980 msgr2=0x55a3d49c5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:09.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.009+0000 7fc1cdc34700 1 --2- 172.21.15.189:0/782305808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d499f980 0x55a3d49c5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.009+0000 7fc1cdc34700 1 -- 172.21.15.189:0/782305808 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a3d4778c80 con 0x55a3d49c4c20 2024-04-06T12:32:09.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.009+0000 7fc1cdc34700 1 --2- 172.21.15.189:0/782305808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 0x55a3d49c5c80 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4007bb0 tx=0x7fc1c4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e633258db70401aa server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:09.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.009+0000 7fc1bcff9700 1 -- 172.21.15.189:0/782305808 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4025e50 con 0x55a3d49c4c20 2024-04-06T12:32:09.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.009+0000 7fc1bcff9700 1 -- 172.21.15.189:0/782305808 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c4004b10 con 0x55a3d49c4c20 2024-04-06T12:32:09.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.009+0000 7fc1bcff9700 1 -- 172.21.15.189:0/782305808 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c40255a0 con 0x55a3d49c4c20 2024-04-06T12:32:09.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.009+0000 7fc1dc84db80 1 -- 172.21.15.189:0/782305808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 msgr2=0x55a3d49c5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:09.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.009+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/782305808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 0x55a3d49c5c80 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4007bb0 tx=0x7fc1c4013000 comp rx=0 tx=0).stop 2024-04-06T12:32:09.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.009+0000 7fc1dc84db80 1 -- 172.21.15.189:0/782305808 shutdown_connections 2024-04-06T12:32:09.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.009+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/782305808 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c9fc0 0x55a3d49e7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.009+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/782305808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d499f980 0x55a3d49c5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.009+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/782305808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 0x55a3d49c5c80 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.009+0000 7fc1dc84db80 1 -- 172.21.15.189:0/782305808 >> 172.21.15.189:0/782305808 conn(0x55a3d4900650 msgr2=0x55a3d49a30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:09.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.009+0000 7fc1dc84db80 1 -- 172.21.15.189:0/782305808 shutdown_connections 2024-04-06T12:32:09.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.009+0000 7fc1dc84db80 1 -- 172.21.15.189:0/782305808 wait complete. 2024-04-06T12:32:09.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.010+0000 7fc1dc84db80 1 -- start start 2024-04-06T12:32:09.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.010+0000 7fc1dc84db80 1 Processor -- start 2024-04-06T12:32:09.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1dc84db80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d499f980 0x55a3d49e7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:09.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1dc84db80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 0x55a3d499b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:09.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1dc84db80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c9fc0 0x55a3d49c5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:09.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1dc84db80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d49e8200 con 0x55a3d49c4c20 2024-04-06T12:32:09.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1dc84db80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d49a1400 con 0x55a3d499f980 2024-04-06T12:32:09.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1dc84db80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a3d49e8370 con 0x55a3d49c9fc0 2024-04-06T12:32:09.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1dc84db80 10 client.0 _ll_register_callbacks cb 0x55a3d4898420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T12:32:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1cdc34700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 0x55a3d499b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1cdc34700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 0x55a3d499b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60860/0 (socket says 172.21.15.189:60860) 2024-04-06T12:32:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1cdc34700 1 -- 172.21.15.189:0/3811366800 learned_addr learned my addr 172.21.15.189:0/3811366800 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1ce435700 1 --2- 172.21.15.189:0/3811366800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d499f980 0x55a3d49e7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1cec36700 1 --2- 172.21.15.189:0/3811366800 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c9fc0 0x55a3d49c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1cdc34700 1 -- 172.21.15.189:0/3811366800 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c9fc0 msgr2=0x55a3d49c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1cdc34700 1 --2- 172.21.15.189:0/3811366800 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c9fc0 0x55a3d49c5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1cdc34700 1 -- 172.21.15.189:0/3811366800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d499f980 msgr2=0x55a3d49e7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1cdc34700 1 --2- 172.21.15.189:0/3811366800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d499f980 0x55a3d49e7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1cdc34700 1 -- 172.21.15.189:0/3811366800 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c4025ad0 con 0x55a3d49c4c20 2024-04-06T12:32:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1ce435700 1 --2- 172.21.15.189:0/3811366800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d499f980 0x55a3d49e7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:32:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1cdc34700 1 --2- 172.21.15.189:0/3811366800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 0x55a3d499b8c0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4007fd0 tx=0x7fc1c4000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdce0ce66e2d11b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.011+0000 7fc1a97fa700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T12:32:09.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.012+0000 7fc1a97fa700 1 -- 172.21.15.189:0/3811366800 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4001850 con 0x55a3d49c4c20 2024-04-06T12:32:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.012+0000 7fc1a97fa700 1 -- 172.21.15.189:0/3811366800 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(158..158 src has 1..158) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc1c401b040 con 0x55a3d49c4c20 2024-04-06T12:32:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.012+0000 7fc1dc84db80 1 -- 172.21.15.189:0/3811366800 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc1b805ce60 con 0x55a3d49c4c20 2024-04-06T12:32:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.012+0000 7fc1a97fa700 1 -- 172.21.15.189:0/3811366800 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1c40019b0 con 0x55a3d49c4c20 2024-04-06T12:32:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.012+0000 7fc1dc84db80 10 client.9267 fetch_fsmap learned FSMap version 70 2024-04-06T12:32:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.012+0000 7fc1dc84db80 1 -- 172.21.15.189:0/3811366800 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x55a3d471efe0 con 0x55a3d49c4c20 2024-04-06T12:32:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.012+0000 7fc1a97fa700 1 -- 172.21.15.189:0/3811366800 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b805ce60 con 0x55a3d49c4c20 2024-04-06T12:32:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.012+0000 7fc1a97fa700 1 -- 172.21.15.189:0/3811366800 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 70) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc1c4001b10 con 0x55a3d49c4c20 2024-04-06T12:32:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.012+0000 7fc1dc84db80 10 client.9267 fetch_fsmap finished waiting for FSMap version 70 2024-04-06T12:32:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.012+0000 7fc1dc84db80 10 client.9267 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-06T12:32:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.012+0000 7fc1dc84db80 20 client.9267 populate_metadata read hostname 'smithi189' 2024-04-06T12:32:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.012+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/3811366800 >> [v2:172.21.15.192:6833/3068270636,v1:172.21.15.192:6835/3068270636] conn(0x55a3d49c5bc0 0x55a3d49f5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.012+0000 7fc1dc84db80 4 client.9267 mds_command: new command op to 9144 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-06T12:32:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.012+0000 7fc1dc84db80 1 -- 172.21.15.189:0/3811366800 --> [v2:172.21.15.192:6833/3068270636,v1:172.21.15.192:6835/3068270636] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a3d4741ca0 con 0x55a3d49c5bc0 2024-04-06T12:32:09.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.012+0000 7fc1ce435700 1 --2- 172.21.15.189:0/3811366800 >> [v2:172.21.15.192:6833/3068270636,v1:172.21.15.192:6835/3068270636] conn(0x55a3d49c5bc0 0x55a3d49f5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.013+0000 7fc1a97fa700 10 client.9267 ms_handle_connect on v2:172.21.15.192:6833/3068270636 2024-04-06T12:32:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.013+0000 7fc1ce435700 1 --2- 172.21.15.189:0/3811366800 >> [v2:172.21.15.192:6833/3068270636,v1:172.21.15.192:6835/3068270636] conn(0x55a3d49c5bc0 0x55a3d49f5b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c2d438f6ea8a698 server_cookie=9afdec343d478f34 in_seq=0 out_seq=0 2024-04-06T12:32:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.013+0000 7fc1a97fa700 1 -- 172.21.15.189:0/3811366800 <== mds.0 v2:172.21.15.192:6833/3068270636 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55a3d4741ca0 con 0x55a3d49c5bc0 2024-04-06T12:32:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.013+0000 7fc1a97fa700 10 client.9267 handle_command_reply: tid=0 2024-04-06T12:32:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.013+0000 7fc1dc84db80 1 client.9267 shutdown 2024-04-06T12:32:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.013+0000 7fc1dc84db80 1 -- 172.21.15.189:0/3811366800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 msgr2=0x55a3d499b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.013+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/3811366800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 0x55a3d499b8c0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4007fd0 tx=0x7fc1c4000ed0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.013+0000 7fc1dc84db80 1 -- 172.21.15.189:0/3811366800 shutdown_connections 2024-04-06T12:32:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.013+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/3811366800 >> [v2:172.21.15.192:6833/3068270636,v1:172.21.15.192:6835/3068270636] conn(0x55a3d49c5bc0 0x55a3d49f5b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.013+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/3811366800 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a3d49c9fc0 0x55a3d49c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.013+0000 7fc1a97fa700 0 client.9267 ms_handle_reset on v2:172.21.15.192:6833/3068270636 2024-04-06T12:32:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.013+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/3811366800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a3d499f980 0x55a3d49e7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.013+0000 7fc1dc84db80 1 --2- 172.21.15.189:0/3811366800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3d49c4c20 0x55a3d499b8c0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.013+0000 7fc1dc84db80 1 -- 172.21.15.189:0/3811366800 >> 172.21.15.189:0/3811366800 conn(0x55a3d4900650 msgr2=0x55a3d49c7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:09.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.013+0000 7fc1dc84db80 1 -- 172.21.15.189:0/3811366800 shutdown_connections 2024-04-06T12:32:09.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.013+0000 7fc1dc84db80 1 -- 172.21.15.189:0/3811366800 wait complete. 2024-04-06T12:32:09.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.013+0000 7fc1dc84db80 20 client.9267 trim_cache size 0 max 16384 2024-04-06T12:32:09.018 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T12:32:09.018 INFO:teuthology.orchestra.run.smithi189.stdout: "cluster_fsid": "e30041de-e9a3-4351-8ca7-29d7f3e3728c", 2024-04-06T12:32:09.018 INFO:teuthology.orchestra.run.smithi189.stdout: "whoami": 0, 2024-04-06T12:32:09.018 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 9144, 2024-04-06T12:32:09.018 INFO:teuthology.orchestra.run.smithi189.stdout: "want_state": "up:active", 2024-04-06T12:32:09.018 INFO:teuthology.orchestra.run.smithi189.stdout: "state": "up:active", 2024-04-06T12:32:09.018 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs", 2024-04-06T12:32:09.018 INFO:teuthology.orchestra.run.smithi189.stdout: "rank_uptime": 5.9448474999999998, 2024-04-06T12:32:09.018 INFO:teuthology.orchestra.run.smithi189.stdout: "mdsmap_epoch": 70, 2024-04-06T12:32:09.019 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch": 158, 2024-04-06T12:32:09.019 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch_barrier": 158, 2024-04-06T12:32:09.019 INFO:teuthology.orchestra.run.smithi189.stdout: "uptime": 11.265079467 2024-04-06T12:32:09.019 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T12:32:09.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.014+0000 7fc1d0699700 1 -- 172.21.15.189:0/4097887926 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc1ac06ade0 msgr2=0x7fc1ac06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:09.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.014+0000 7fc1d0699700 1 --2- 172.21.15.189:0/4097887926 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc1ac06ade0 0x7fc1ac06d2a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc1c815cbc0 tx=0x7fc1c4007de0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.014+0000 7fc1d0699700 1 -- 172.21.15.189:0/4097887926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1c80b4ed0 msgr2=0x7fc1c814ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:09.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.014+0000 7fc1d0699700 1 --2- 172.21.15.189:0/4097887926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1c80b4ed0 0x7fc1c814ef10 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8009c40 tx=0x7fc1b8004750 comp rx=0 tx=0).stop 2024-04-06T12:32:09.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.014+0000 7fc1d0699700 1 -- 172.21.15.189:0/4097887926 shutdown_connections 2024-04-06T12:32:09.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.014+0000 7fc1d0699700 1 --2- 172.21.15.189:0/4097887926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1c815d590 0x7fc1c815b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.014+0000 7fc1d0699700 1 --2- 172.21.15.189:0/4097887926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1c80b4ed0 0x7fc1c814ef10 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.014+0000 7fc1d0699700 1 --2- 172.21.15.189:0/4097887926 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc1ac06ade0 0x7fc1ac06d2a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.014+0000 7fc1d0699700 1 --2- 172.21.15.189:0/4097887926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1c80d3890 0x7fc1c80ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.014+0000 7fc1d0699700 1 -- 172.21.15.189:0/4097887926 >> 172.21.15.189:0/4097887926 conn(0x7fc1c806bbc0 msgr2=0x7fc1c80b2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:09.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.014+0000 7fc1d0699700 1 -- 172.21.15.189:0/4097887926 shutdown_connections 2024-04-06T12:32:09.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.014+0000 7fc1d0699700 1 -- 172.21.15.189:0/4097887926 wait complete. 2024-04-06T12:32:09.024 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T12:32:09.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.149+0000 7f975e4a9700 1 Processor -- start 2024-04-06T12:32:09.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.149+0000 7f975e4a9700 1 -- start start 2024-04-06T12:32:09.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.150+0000 7f975e4a9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f975815a550 0x7f97580cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:09.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.150+0000 7f975e4a9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97580cf900 0x7f97580b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:09.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.150+0000 7f975e4a9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97580cdbe0 con 0x7f97580b4ba0 2024-04-06T12:32:09.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.150+0000 7f975e4a9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97580b5010 con 0x7f975815a550 2024-04-06T12:32:09.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.150+0000 7f975e4a9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975815a940 con 0x7f97580cf900 2024-04-06T12:32:09.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.150+0000 7f975ca46700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f97580b4ba0 0x7f975806eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44018/0 (socket says 172.21.15.189:44018) 2024-04-06T12:32:09.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.150+0000 7f975ca46700 1 -- 172.21.15.189:0/2032841758 learned_addr learned my addr 172.21.15.189:0/2032841758 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:32:09.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.150+0000 7f9757fff700 1 --2- 172.21.15.189:0/2032841758 >> v2:172.21.15.192:3300/0 conn(0x7f975815a550 0x7f97580cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:09.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.150+0000 7f97577fe700 1 --2- 172.21.15.189:0/2032841758 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97580cf900 0x7f97580b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:09.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.150+0000 7f9756ffd700 1 -- 172.21.15.189:0/2032841758 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466010393 0 0) 0x7f97580cdbe0 con 0x7f97580b4ba0 2024-04-06T12:32:09.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.150+0000 7f9756ffd700 1 -- 172.21.15.189:0/2032841758 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97440035b0 con 0x7f97580b4ba0 2024-04-06T12:32:09.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.150+0000 7f97577fe700 1 -- 172.21.15.189:0/2032841758 >> v2:172.21.15.192:3300/0 conn(0x7f975815a550 msgr2=0x7f97580cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:09.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.150+0000 7f97577fe700 1 --2- 172.21.15.189:0/2032841758 >> v2:172.21.15.192:3300/0 conn(0x7f975815a550 0x7f97580cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.150+0000 7f97577fe700 1 -- 172.21.15.189:0/2032841758 >> v1:172.21.15.189:6789/0 conn(0x7f97580b4ba0 legacy=0x7f975806eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:09.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.150+0000 7f97577fe700 1 -- 172.21.15.189:0/2032841758 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975815ac00 con 0x7f97580cf900 2024-04-06T12:32:09.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.150+0000 7f9757fff700 1 --2- 172.21.15.189:0/2032841758 >> v2:172.21.15.192:3300/0 conn(0x7f975815a550 0x7f97580cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:32:09.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.151+0000 7f97577fe700 1 --2- 172.21.15.189:0/2032841758 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97580cf900 0x7f97580b4660 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9748009840 tx=0x7f9748009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d929c159242cd89 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:09.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.151+0000 7f9756ffd700 1 -- 172.21.15.189:0/2032841758 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9748011830 con 0x7f97580cf900 2024-04-06T12:32:09.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.151+0000 7f9756ffd700 1 -- 172.21.15.189:0/2032841758 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9748011990 con 0x7f97580cf900 2024-04-06T12:32:09.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.151+0000 7f9756ffd700 1 -- 172.21.15.189:0/2032841758 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9748011d20 con 0x7f97580cf900 2024-04-06T12:32:09.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.151+0000 7f975e4a9700 1 -- 172.21.15.189:0/2032841758 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97580cf900 msgr2=0x7f97580b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:09.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.151+0000 7f975e4a9700 1 --2- 172.21.15.189:0/2032841758 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97580cf900 0x7f97580b4660 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9748009840 tx=0x7f9748009b50 comp rx=0 tx=0).stop 2024-04-06T12:32:09.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.151+0000 7f975e4a9700 1 -- 172.21.15.189:0/2032841758 shutdown_connections 2024-04-06T12:32:09.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.151+0000 7f975e4a9700 1 --2- 172.21.15.189:0/2032841758 >> v2:172.21.15.192:3300/0 conn(0x7f975815a550 0x7f97580cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.151+0000 7f975e4a9700 1 --2- 172.21.15.189:0/2032841758 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97580cf900 0x7f97580b4660 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.151+0000 7f975e4a9700 1 -- 172.21.15.189:0/2032841758 >> 172.21.15.189:0/2032841758 conn(0x7f97580d00b0 msgr2=0x7f97580d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:09.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.151+0000 7f975e4a9700 1 -- 172.21.15.189:0/2032841758 shutdown_connections 2024-04-06T12:32:09.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.151+0000 7f975e4a9700 1 -- 172.21.15.189:0/2032841758 wait complete. 2024-04-06T12:32:09.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.152+0000 7f975e4a9700 1 Processor -- start 2024-04-06T12:32:09.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.153+0000 7f975e4a9700 1 -- start start 2024-04-06T12:32:09.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.153+0000 7f975e4a9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97580b4ba0 0x7f975814fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:09.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.153+0000 7f975e4a9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97580cf900 0x7f97581f30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.153+0000 7f975e4a9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9758157d10 0x7f97580b2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.153+0000 7f975e4a9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97581525e0 con 0x7f9758157d10 2024-04-06T12:32:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.153+0000 7f975e4a9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97581f35e0 con 0x7f97580b4ba0 2024-04-06T12:32:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.153+0000 7f975e4a9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9758152060 con 0x7f97580cf900 2024-04-06T12:32:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.154+0000 7f975ca46700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9758157d10 0x7f97580b2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.154+0000 7f975ca46700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9758157d10 0x7f97580b2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60872/0 (socket says 172.21.15.189:60872) 2024-04-06T12:32:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.154+0000 7f975ca46700 1 -- 172.21.15.189:0/29843844 learned_addr learned my addr 172.21.15.189:0/29843844 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.154+0000 7f9757fff700 1 --2- 172.21.15.189:0/29843844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97580b4ba0 0x7f975814fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.154+0000 7f97577fe700 1 --2- 172.21.15.189:0/29843844 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97580cf900 0x7f97581f30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.154+0000 7f975ca46700 1 -- 172.21.15.189:0/29843844 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97580cf900 msgr2=0x7f97581f30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.154+0000 7f975ca46700 1 --2- 172.21.15.189:0/29843844 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97580cf900 0x7f97581f30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.154+0000 7f975ca46700 1 -- 172.21.15.189:0/29843844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97580b4ba0 msgr2=0x7f975814fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.154+0000 7f975ca46700 1 --2- 172.21.15.189:0/29843844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97580b4ba0 0x7f975814fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.154+0000 7f975ca46700 1 -- 172.21.15.189:0/29843844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97400094a0 con 0x7f9758157d10 2024-04-06T12:32:09.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.154+0000 7f975ca46700 1 --2- 172.21.15.189:0/29843844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9758157d10 0x7f97580b2cb0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f975000caa0 tx=0x7f975000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:09.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.154+0000 7f9754ff9700 1 -- 172.21.15.189:0/29843844 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975000aa80 con 0x7f9758157d10 2024-04-06T12:32:09.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.154+0000 7f9754ff9700 1 -- 172.21.15.189:0/29843844 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975000abe0 con 0x7f9758157d10 2024-04-06T12:32:09.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.154+0000 7f975e4a9700 1 -- 172.21.15.189:0/29843844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97480094a0 con 0x7f9758157d10 2024-04-06T12:32:09.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.154+0000 7f9754ff9700 1 -- 172.21.15.189:0/29843844 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9750012400 con 0x7f9758157d10 2024-04-06T12:32:09.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.154+0000 7f975e4a9700 1 -- 172.21.15.189:0/29843844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9758152300 con 0x7f9758157d10 2024-04-06T12:32:09.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.155+0000 7f975e4a9700 1 -- 172.21.15.189:0/29843844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97581f2740 con 0x7f9758157d10 2024-04-06T12:32:09.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.156+0000 7f9754ff9700 1 -- 172.21.15.189:0/29843844 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f975001b3b0 con 0x7f9758157d10 2024-04-06T12:32:09.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.156+0000 7f9754ff9700 1 --2- 172.21.15.189:0/29843844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f973806adb0 0x7f973806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:09.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.156+0000 7f9757fff700 1 --2- 172.21.15.189:0/29843844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f973806adb0 0x7f973806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:09.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.156+0000 7f9754ff9700 1 -- 172.21.15.189:0/29843844 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9750089330 con 0x7f9758157d10 2024-04-06T12:32:09.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.157+0000 7f9757fff700 1 --2- 172.21.15.189:0/29843844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f973806adb0 0x7f973806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9740009d00 tx=0x7f9740008040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:09.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.161+0000 7f9754ff9700 1 -- 172.21.15.189:0/29843844 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f97581f2740 con 0x7f9758157d10 2024-04-06T12:32:09.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.400+0000 7f975e4a9700 1 -- 172.21.15.189:0/29843844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f97581f2740 con 0x7f9758157d10 2024-04-06T12:32:09.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.401+0000 7f9754ff9700 1 -- 172.21.15.189:0/29843844 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v70) v1 ==== 53+0+73 (secure 0 0 0) 0x7f97581f2740 con 0x7f9758157d10 2024-04-06T12:32:09.399 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-06T12:32:09.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.403+0000 7f975e4a9700 1 -- 172.21.15.189:0/29843844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f973806adb0 msgr2=0x7f973806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:09.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.403+0000 7f975e4a9700 1 --2- 172.21.15.189:0/29843844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f973806adb0 0x7f973806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9740009d00 tx=0x7f9740008040 comp rx=0 tx=0).stop 2024-04-06T12:32:09.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.403+0000 7f975e4a9700 1 -- 172.21.15.189:0/29843844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9758157d10 msgr2=0x7f97580b2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:09.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.403+0000 7f975e4a9700 1 --2- 172.21.15.189:0/29843844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9758157d10 0x7f97580b2cb0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f975000caa0 tx=0x7f975000ce60 comp rx=0 tx=0).stop 2024-04-06T12:32:09.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.403+0000 7f975e4a9700 1 -- 172.21.15.189:0/29843844 shutdown_connections 2024-04-06T12:32:09.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.403+0000 7f975e4a9700 1 --2- 172.21.15.189:0/29843844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9758157d10 0x7f97580b2cb0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.403+0000 7f975e4a9700 1 --2- 172.21.15.189:0/29843844 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f973806adb0 0x7f973806d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.403+0000 7f975e4a9700 1 --2- 172.21.15.189:0/29843844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97580b4ba0 0x7f975814fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.403+0000 7f975e4a9700 1 --2- 172.21.15.189:0/29843844 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97580cf900 0x7f97581f30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:09.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.403+0000 7f975e4a9700 1 -- 172.21.15.189:0/29843844 >> 172.21.15.189:0/29843844 conn(0x7f97580d00b0 msgr2=0x7f975806f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:09.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.403+0000 7f975e4a9700 1 -- 172.21.15.189:0/29843844 shutdown_connections 2024-04-06T12:32:09.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:09.403+0000 7f975e4a9700 1 -- 172.21.15.189:0/29843844 wait complete. 2024-04-06T12:32:09.413 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-06T12:32:09.413 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T12:32:09.413 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-06T12:32:09.413 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-06T12:32:09.413 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T12:32:09.413 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-06T12:32:09.413 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-06T12:32:09.413 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T12:32:09.413 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:32:09.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:32:09.414 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-06T12:32:09.431 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T12:32:09.432 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T12:32:09.432 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T12:32:09.432 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:32:09.432 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T12:32:09.432 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:32:09.432 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:32:09.432 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:32:09.432 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-06T12:32:09.432 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:32:09.433 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:32:09.433 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-06T12:32:09.433 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T12:32:09.433 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:32:09.433 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T12:32:09.433 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-06T12:32:09.433 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T12:32:09.433 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 41160sec preferred_lft 41160sec 2024-04-06T12:32:09.433 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::8592:df2c:92e2:dace/64 scope link noprefixroute 2024-04-06T12:32:09.433 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:32:09.433 INFO:teuthology.orchestra.run.smithi189.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T12:32:09.433 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff 2024-04-06T12:32:09.433 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-06T12:32:09.433 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:32:09.434 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::88e:b6ff:fe6f:1cb0/64 scope link 2024-04-06T12:32:09.434 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:32:09.434 INFO:teuthology.orchestra.run.smithi189.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T12:32:09.434 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 62:a5:8d:c5:1a:9c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T12:32:09.434 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::60a5:8dff:fec5:1a9c/64 scope link 2024-04-06T12:32:09.434 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:32:09.434 INFO:teuthology.orchestra.run.smithi189.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T12:32:09.434 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-06T12:32:09.434 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::1c7e:c6ff:febb:1dab/64 scope link 2024-04-06T12:32:09.434 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T12:32:09.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:32:09.435 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T12:32:09.497 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-06T12:32:09.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:32:09.499 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-06T12:32:09.557 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-06T12:32:09.557 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T12:32:09.557 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T12:32:09.558 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T12:32:09.558 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T12:32:09.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:32:09.558 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T12:32:09.558 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T12:32:09.558 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T12:32:09.558 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T12:32:09.558 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T12:32:09.558 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T12:32:09.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:32:09.665 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T12:32:09.682 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-06T12:32:09.682 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-06T12:32:09.682 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:32:09.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:32:09.682 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-04-06T12:32:09.749 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:32:09.750 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-06T12:32:09.807 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-06T12:32:09.807 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T12:32:09.807 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-06T12:32:09.863 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T12:32:09.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:32:09.864 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:32:09.951 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:32:09.951 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:32:09.951 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:32:09.960 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:32:09.960 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:32:09.960 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:32:09.960 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:32:09.960 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:32:09.979 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T12:32:09.979 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T12:32:09.979 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T12:32:09.979 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T12:32:09.980 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T12:32:09.980 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790" 2024-04-06T12:32:09.980 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2024-04-06T12:32:09.980 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e30041de-e9a3-4351-8ca7-29d7f3e3728c" will pass to kernel 2024-04-06T12:32:09.982 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T12:32:09.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:32:09.982 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-06T12:32:10.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:32:10.013 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T12:32:10.110 INFO:teuthology.orchestra.run.smithi189.stdout:client8515 2024-04-06T12:32:10.114 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T12:32:10.114 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T12:32:10.132 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-06T12:32:10.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.297+0000 7f6ef206d700 1 Processor -- start 2024-04-06T12:32:10.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.297+0000 7f6ef206d700 1 -- start start 2024-04-06T12:32:10.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.298+0000 7f6ef206d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6eec146dd0 0x7f6eec06d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:10.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.298+0000 7f6ef206d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6eec0cedf0 0x7f6eec0cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:10.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.298+0000 7f6ef206d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eec0cf760 con 0x7f6eec0d17a0 2024-04-06T12:32:10.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.298+0000 7f6ef206d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eec06beb0 con 0x7f6eec0cedf0 2024-04-06T12:32:10.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.298+0000 7f6ef206d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec06c0a0 con 0x7f6eec146dd0 2024-04-06T12:32:10.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.298+0000 7f6eebfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6eec0cedf0 0x7f6eec0cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:10.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.298+0000 7f6eeb7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6eec0d17a0 0x7f6eec06b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50392/0 (socket says 172.21.15.189:50392) 2024-04-06T12:32:10.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.298+0000 7f6eeb7fe700 1 -- 172.21.15.189:0/1091658436 learned_addr learned my addr 172.21.15.189:0/1091658436 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:32:10.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.298+0000 7f6eebfff700 1 -- 172.21.15.189:0/1091658436 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6eec146dd0 msgr2=0x7f6eec06d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:10.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.298+0000 7f6eebfff700 1 --2- 172.21.15.189:0/1091658436 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6eec146dd0 0x7f6eec06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:10.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.298+0000 7f6eebfff700 1 -- 172.21.15.189:0/1091658436 >> v1:172.21.15.189:6789/0 conn(0x7f6eec0d17a0 legacy=0x7f6eec06b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:10.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.298+0000 7f6eea7fc700 1 -- 172.21.15.189:0/1091658436 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503952127 0 0) 0x7f6eec0cf760 con 0x7f6eec0d17a0 2024-04-06T12:32:10.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.298+0000 7f6eebfff700 1 -- 172.21.15.189:0/1091658436 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eec06c380 con 0x7f6eec0cedf0 2024-04-06T12:32:10.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.299+0000 7f6eebfff700 1 --2- 172.21.15.189:0/1091658436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6eec0cedf0 0x7f6eec0cf220 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4009d90 tx=0x7f6ee400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591218da01f41efa server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:10.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.299+0000 7f6eea7fc700 1 -- 172.21.15.189:0/1091658436 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee400e070 con 0x7f6eec0cedf0 2024-04-06T12:32:10.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.299+0000 7f6eea7fc700 1 -- 172.21.15.189:0/1091658436 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee400bbd0 con 0x7f6eec0cedf0 2024-04-06T12:32:10.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.299+0000 7f6eea7fc700 1 -- 172.21.15.189:0/1091658436 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee4012430 con 0x7f6eec0cedf0 2024-04-06T12:32:10.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.299+0000 7f6ef206d700 1 -- 172.21.15.189:0/1091658436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6eec0cedf0 msgr2=0x7f6eec0cf220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:10.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.299+0000 7f6ef206d700 1 --2- 172.21.15.189:0/1091658436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6eec0cedf0 0x7f6eec0cf220 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4009d90 tx=0x7f6ee400b330 comp rx=0 tx=0).stop 2024-04-06T12:32:10.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.299+0000 7f6ef206d700 1 -- 172.21.15.189:0/1091658436 shutdown_connections 2024-04-06T12:32:10.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.299+0000 7f6ef206d700 1 --2- 172.21.15.189:0/1091658436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6eec0cedf0 0x7f6eec0cf220 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:10.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.299+0000 7f6ef206d700 1 --2- 172.21.15.189:0/1091658436 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6eec146dd0 0x7f6eec06d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:10.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.299+0000 7f6ef206d700 1 -- 172.21.15.189:0/1091658436 >> 172.21.15.189:0/1091658436 conn(0x7f6eec0b2db0 msgr2=0x7f6eec0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:10.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.299+0000 7f6ef206d700 1 -- 172.21.15.189:0/1091658436 shutdown_connections 2024-04-06T12:32:10.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.299+0000 7f6ef206d700 1 -- 172.21.15.189:0/1091658436 wait complete. 2024-04-06T12:32:10.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.300+0000 7f6ef206d700 1 Processor -- start 2024-04-06T12:32:10.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.301+0000 7f6ef206d700 1 -- start start 2024-04-06T12:32:10.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.301+0000 7f6ef206d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6eec0cedf0 0x7f6eec0ae4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:10.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.301+0000 7f6ef206d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6eec0d17a0 0x7f6eec159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:10.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.301+0000 7f6ef206d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6eec146dd0 0x7f6eec0c7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:10.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.301+0000 7f6ef206d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec15a330 con 0x7f6eec146dd0 2024-04-06T12:32:10.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.301+0000 7f6ef206d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec0c7ca0 con 0x7f6eec0d17a0 2024-04-06T12:32:10.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.301+0000 7f6ef206d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec0aeae0 con 0x7f6eec0cedf0 2024-04-06T12:32:10.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.302+0000 7f6eebfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6eec146dd0 0x7f6eec0c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:10.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.302+0000 7f6eebfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6eec146dd0 0x7f6eec0c7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53050/0 (socket says 172.21.15.189:53050) 2024-04-06T12:32:10.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.302+0000 7f6eebfff700 1 -- 172.21.15.189:0/1609529161 learned_addr learned my addr 172.21.15.189:0/1609529161 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:10.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.302+0000 7f6eeb7fe700 1 --2- 172.21.15.189:0/1609529161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6eec0cedf0 0x7f6eec0ae4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:10.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.302+0000 7f6eebfff700 1 -- 172.21.15.189:0/1609529161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6eec0cedf0 msgr2=0x7f6eec0ae4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:10.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.302+0000 7f6eebfff700 1 --2- 172.21.15.189:0/1609529161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6eec0cedf0 0x7f6eec0ae4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:10.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.302+0000 7f6eebfff700 1 -- 172.21.15.189:0/1609529161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6eec0d17a0 msgr2=0x7f6eec159df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:10.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.302+0000 7f6eebfff700 1 --2- 172.21.15.189:0/1609529161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6eec0d17a0 0x7f6eec159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:10.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.302+0000 7f6eebfff700 1 -- 172.21.15.189:0/1609529161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee4009a70 con 0x7f6eec146dd0 2024-04-06T12:32:10.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.302+0000 7f6eeb7fe700 1 --2- 172.21.15.189:0/1609529161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6eec0cedf0 0x7f6eec0ae4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:32:10.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.302+0000 7f6eebfff700 1 --2- 172.21.15.189:0/1609529161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6eec146dd0 0x7f6eec0c7760 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f6eec0677e0 tx=0x7f6ee4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:10.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.302+0000 7f6ee3fff700 1 -- 172.21.15.189:0/1609529161 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee4012430 con 0x7f6eec146dd0 2024-04-06T12:32:10.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.302+0000 7f6ee3fff700 1 -- 172.21.15.189:0/1609529161 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee4020070 con 0x7f6eec146dd0 2024-04-06T12:32:10.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.302+0000 7f6ef206d700 1 -- 172.21.15.189:0/1609529161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eec0aedc0 con 0x7f6eec146dd0 2024-04-06T12:32:10.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.302+0000 7f6ee3fff700 1 -- 172.21.15.189:0/1609529161 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee4004280 con 0x7f6eec146dd0 2024-04-06T12:32:10.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.302+0000 7f6ef206d700 1 -- 172.21.15.189:0/1609529161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eec0af380 con 0x7f6eec146dd0 2024-04-06T12:32:10.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.303+0000 7f6ef206d700 1 -- 172.21.15.189:0/1609529161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eec074990 con 0x7f6eec146dd0 2024-04-06T12:32:10.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.304+0000 7f6ee3fff700 1 -- 172.21.15.189:0/1609529161 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f6ee40043e0 con 0x7f6eec146dd0 2024-04-06T12:32:10.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.304+0000 7f6ee3fff700 1 --2- 172.21.15.189:0/1609529161 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ecc06adb0 0x7f6ecc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:10.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.304+0000 7f6ee3fff700 1 -- 172.21.15.189:0/1609529161 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6ee4022070 con 0x7f6eec146dd0 2024-04-06T12:32:10.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.304+0000 7f6eeb7fe700 1 --2- 172.21.15.189:0/1609529161 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ecc06adb0 0x7f6ecc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:10.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.304+0000 7f6eeb7fe700 1 --2- 172.21.15.189:0/1609529161 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ecc06adb0 0x7f6ecc06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4007fd0 tx=0x7f6ed4009040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:10.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.308+0000 7f6ee3fff700 1 -- 172.21.15.189:0/1609529161 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6ee40564d0 con 0x7f6eec146dd0 2024-04-06T12:32:10.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.531+0000 7f6ef206d700 1 -- 172.21.15.189:0/1609529161 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f6eec06cc20 con 0x7f6ecc06adb0 2024-04-06T12:32:10.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.532+0000 7f6ee3fff700 1 -- 172.21.15.189:0/1609529161 <== mgr.8176 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6ed4001410 con 0x7f6ecc06adb0 2024-04-06T12:32:10.530 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T12:32:10.530 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T12:32:10.531 INFO:teuthology.orchestra.run.smithi189.stdout: "name": "cephfs" 2024-04-06T12:32:10.531 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T12:32:10.531 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T12:32:10.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.534+0000 7f6ef206d700 1 -- 172.21.15.189:0/1609529161 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ecc06adb0 msgr2=0x7f6ecc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:10.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.534+0000 7f6ef206d700 1 --2- 172.21.15.189:0/1609529161 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ecc06adb0 0x7f6ecc06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4007fd0 tx=0x7f6ed4009040 comp rx=0 tx=0).stop 2024-04-06T12:32:10.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.534+0000 7f6ef206d700 1 -- 172.21.15.189:0/1609529161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6eec146dd0 msgr2=0x7f6eec0c7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:10.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.534+0000 7f6ef206d700 1 --2- 172.21.15.189:0/1609529161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6eec146dd0 0x7f6eec0c7760 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f6eec0677e0 tx=0x7f6ee4004ad0 comp rx=0 tx=0).stop 2024-04-06T12:32:10.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.534+0000 7f6ef206d700 1 -- 172.21.15.189:0/1609529161 shutdown_connections 2024-04-06T12:32:10.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.534+0000 7f6ef206d700 1 --2- 172.21.15.189:0/1609529161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6eec146dd0 0x7f6eec0c7760 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:10.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.534+0000 7f6ef206d700 1 --2- 172.21.15.189:0/1609529161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6eec0cedf0 0x7f6eec0ae4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:10.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.534+0000 7f6ef206d700 1 --2- 172.21.15.189:0/1609529161 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ecc06adb0 0x7f6ecc06d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:10.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.534+0000 7f6ef206d700 1 --2- 172.21.15.189:0/1609529161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6eec0d17a0 0x7f6eec159df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:10.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.534+0000 7f6ef206d700 1 -- 172.21.15.189:0/1609529161 >> 172.21.15.189:0/1609529161 conn(0x7f6eec0b2db0 msgr2=0x7f6eec06b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:10.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.535+0000 7f6ef206d700 1 -- 172.21.15.189:0/1609529161 shutdown_connections 2024-04-06T12:32:10.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.535+0000 7f6ef206d700 1 -- 172.21.15.189:0/1609529161 wait complete. 2024-04-06T12:32:10.545 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-06T12:32:10.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.670+0000 7feb2a867700 1 Processor -- start 2024-04-06T12:32:10.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.670+0000 7feb2a867700 1 -- start start 2024-04-06T12:32:10.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.671+0000 7feb2a867700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb24146dd0 0x7feb2406d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:10.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.671+0000 7feb2a867700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2406c0a0 con 0x7feb240cedf0 2024-04-06T12:32:10.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.671+0000 7feb2a867700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2406c380 con 0x7feb240d17a0 2024-04-06T12:32:10.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.671+0000 7feb2a867700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2406beb0 con 0x7feb24146dd0 2024-04-06T12:32:10.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.671+0000 7feb237fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb24146dd0 0x7feb2406d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:10.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.671+0000 7feb28e04700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7feb240cedf0 0x7feb241441f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50408/0 (socket says 172.21.15.189:50408) 2024-04-06T12:32:10.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.671+0000 7feb28e04700 1 -- 172.21.15.189:0/3830587571 learned_addr learned my addr 172.21.15.189:0/3830587571 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:32:10.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.672+0000 7feb22ffd700 1 -- 172.21.15.189:0/3830587571 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544391592 0 0) 0x7feb2406c0a0 con 0x7feb240cedf0 2024-04-06T12:32:10.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.672+0000 7feb22ffd700 1 -- 172.21.15.189:0/3830587571 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb0c0035b0 con 0x7feb240cedf0 2024-04-06T12:32:10.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.672+0000 7feb22ffd700 1 -- 172.21.15.189:0/3830587571 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299188671 0 0) 0x7feb2406c380 con 0x7feb240d17a0 2024-04-06T12:32:10.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.672+0000 7feb22ffd700 1 -- 172.21.15.189:0/3830587571 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb2406c0a0 con 0x7feb240d17a0 2024-04-06T12:32:10.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.672+0000 7feb237fe700 1 -- 172.21.15.189:0/3830587571 >> v1:172.21.15.192:6789/0 conn(0x7feb240d17a0 legacy=0x7feb2406b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:10.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.672+0000 7feb237fe700 1 -- 172.21.15.189:0/3830587571 >> v1:172.21.15.189:6789/0 conn(0x7feb240cedf0 legacy=0x7feb241441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:10.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.672+0000 7feb237fe700 1 -- 172.21.15.189:0/3830587571 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2406c660 con 0x7feb24146dd0 2024-04-06T12:32:10.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.672+0000 7feb237fe700 1 --2- 172.21.15.189:0/3830587571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb24146dd0 0x7feb2406d140 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7feb18009870 tx=0x7feb18009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c83c9eabe6bb3f9a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:10.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.672+0000 7feb22ffd700 1 -- 172.21.15.189:0/3830587571 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb18011830 con 0x7feb24146dd0 2024-04-06T12:32:10.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.673+0000 7feb22ffd700 1 -- 172.21.15.189:0/3830587571 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb18011990 con 0x7feb24146dd0 2024-04-06T12:32:10.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.673+0000 7feb22ffd700 1 -- 172.21.15.189:0/3830587571 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb18011d00 con 0x7feb24146dd0 2024-04-06T12:32:10.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.673+0000 7feb2a867700 1 -- 172.21.15.189:0/3830587571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb24146dd0 msgr2=0x7feb2406d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:10.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.673+0000 7feb2a867700 1 --2- 172.21.15.189:0/3830587571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb24146dd0 0x7feb2406d140 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7feb18009870 tx=0x7feb18009b80 comp rx=0 tx=0).stop 2024-04-06T12:32:10.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.673+0000 7feb2a867700 1 -- 172.21.15.189:0/3830587571 shutdown_connections 2024-04-06T12:32:10.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.673+0000 7feb2a867700 1 --2- 172.21.15.189:0/3830587571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb24146dd0 0x7feb2406d140 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:10.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.673+0000 7feb2a867700 1 -- 172.21.15.189:0/3830587571 >> 172.21.15.189:0/3830587571 conn(0x7feb240b2db0 msgr2=0x7feb240b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:10.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.673+0000 7feb2a867700 1 -- 172.21.15.189:0/3830587571 shutdown_connections 2024-04-06T12:32:10.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.673+0000 7feb2a867700 1 -- 172.21.15.189:0/3830587571 wait complete. 2024-04-06T12:32:10.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.675+0000 7feb2a867700 1 Processor -- start 2024-04-06T12:32:10.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.675+0000 7feb2a867700 1 -- start start 2024-04-06T12:32:10.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.676+0000 7feb2a867700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb240cedf0 0x7feb240c3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:10.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.676+0000 7feb2a867700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb240d17a0 0x7feb2414a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:10.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.676+0000 7feb2a867700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb24146dd0 0x7feb240b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:10.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.676+0000 7feb2a867700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb240c3f50 con 0x7feb240d17a0 2024-04-06T12:32:10.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.676+0000 7feb2a867700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2414b370 con 0x7feb24146dd0 2024-04-06T12:32:10.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.676+0000 7feb2a867700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2414b650 con 0x7feb240cedf0 2024-04-06T12:32:10.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.676+0000 7feb237fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb240d17a0 0x7feb2414a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:10.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.676+0000 7feb237fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb240d17a0 0x7feb2414a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53056/0 (socket says 172.21.15.189:53056) 2024-04-06T12:32:10.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.676+0000 7feb237fe700 1 -- 172.21.15.189:0/2161704305 learned_addr learned my addr 172.21.15.189:0/2161704305 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:10.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.676+0000 7feb28e04700 1 --2- 172.21.15.189:0/2161704305 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb24146dd0 0x7feb240b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:10.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.676+0000 7feb23fff700 1 --2- 172.21.15.189:0/2161704305 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb240cedf0 0x7feb240c3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:10.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.676+0000 7feb237fe700 1 -- 172.21.15.189:0/2161704305 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb240cedf0 msgr2=0x7feb240c3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:10.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.676+0000 7feb237fe700 1 --2- 172.21.15.189:0/2161704305 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb240cedf0 0x7feb240c3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:10.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.676+0000 7feb237fe700 1 -- 172.21.15.189:0/2161704305 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb24146dd0 msgr2=0x7feb240b1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:10.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.676+0000 7feb237fe700 1 --2- 172.21.15.189:0/2161704305 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb24146dd0 0x7feb240b1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:10.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.676+0000 7feb237fe700 1 -- 172.21.15.189:0/2161704305 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb180094a0 con 0x7feb240d17a0 2024-04-06T12:32:10.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.676+0000 7feb237fe700 1 --2- 172.21.15.189:0/2161704305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb240d17a0 0x7feb2414a8f0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7feb180090b0 tx=0x7feb18004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:10.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.676+0000 7feb20ff9700 1 -- 172.21.15.189:0/2161704305 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb18004510 con 0x7feb240d17a0 2024-04-06T12:32:10.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.676+0000 7feb20ff9700 1 -- 172.21.15.189:0/2161704305 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb180036a0 con 0x7feb240d17a0 2024-04-06T12:32:10.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.677+0000 7feb2a867700 1 -- 172.21.15.189:0/2161704305 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2414b930 con 0x7feb240d17a0 2024-04-06T12:32:10.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.677+0000 7feb20ff9700 1 -- 172.21.15.189:0/2161704305 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb18003a50 con 0x7feb240d17a0 2024-04-06T12:32:10.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.677+0000 7feb2a867700 1 -- 172.21.15.189:0/2161704305 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2414bef0 con 0x7feb240d17a0 2024-04-06T12:32:10.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.677+0000 7feb2a867700 1 -- 172.21.15.189:0/2161704305 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb24074990 con 0x7feb240d17a0 2024-04-06T12:32:10.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.678+0000 7feb20ff9700 1 -- 172.21.15.189:0/2161704305 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 86) v1 ==== 89247+0+0 (secure 0 0 0) 0x7feb18003bb0 con 0x7feb240d17a0 2024-04-06T12:32:10.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.678+0000 7feb20ff9700 1 --2- 172.21.15.189:0/2161704305 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb1006adb0 0x7feb1006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:10.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.678+0000 7feb20ff9700 1 -- 172.21.15.189:0/2161704305 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6029+0+0 (secure 0 0 0) 0x7feb1808a7d0 con 0x7feb240d17a0 2024-04-06T12:32:10.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.679+0000 7feb23fff700 1 --2- 172.21.15.189:0/2161704305 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb1006adb0 0x7feb1006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:10.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.679+0000 7feb23fff700 1 --2- 172.21.15.189:0/2161704305 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb1006adb0 0x7feb1006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feb2415e620 tx=0x7feb14009040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:10.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.683+0000 7feb20ff9700 1 -- 172.21.15.189:0/2161704305 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7feb180557f0 con 0x7feb240d17a0 2024-04-06T12:32:10.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:10.940+0000 7feb2a867700 1 -- 172.21.15.189:0/2161704305 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7feb240d1e20 con 0x7feb240d17a0 2024-04-06T12:32:11.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.106+0000 7feb20ff9700 1 -- 172.21.15.189:0/2161704305 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 87) v1 ==== 89264+0+0 (secure 0 0 0) 0x7feb18055bc0 con 0x7feb240d17a0 2024-04-06T12:32:11.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.108+0000 7feb20ff9700 1 -- 172.21.15.189:0/2161704305 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v87) v1 ==== 92+0+0 (secure 0 0 0) 0x7feb1805aa10 con 0x7feb240d17a0 2024-04-06T12:32:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.110+0000 7feb2a867700 1 -- 172.21.15.189:0/2161704305 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb1006adb0 msgr2=0x7feb1006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.110+0000 7feb2a867700 1 --2- 172.21.15.189:0/2161704305 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb1006adb0 0x7feb1006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feb2415e620 tx=0x7feb14009040 comp rx=0 tx=0).stop 2024-04-06T12:32:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.110+0000 7feb2a867700 1 -- 172.21.15.189:0/2161704305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb240d17a0 msgr2=0x7feb2414a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.110+0000 7feb2a867700 1 --2- 172.21.15.189:0/2161704305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb240d17a0 0x7feb2414a8f0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7feb180090b0 tx=0x7feb18004750 comp rx=0 tx=0).stop 2024-04-06T12:32:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.110+0000 7feb2a867700 1 -- 172.21.15.189:0/2161704305 shutdown_connections 2024-04-06T12:32:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.110+0000 7feb2a867700 1 --2- 172.21.15.189:0/2161704305 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb1006adb0 0x7feb1006d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.110+0000 7feb2a867700 1 --2- 172.21.15.189:0/2161704305 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb24146dd0 0x7feb240b1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.110+0000 7feb2a867700 1 --2- 172.21.15.189:0/2161704305 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb240cedf0 0x7feb240c3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.110+0000 7feb2a867700 1 --2- 172.21.15.189:0/2161704305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb240d17a0 0x7feb2414a8f0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.110+0000 7feb2a867700 1 -- 172.21.15.189:0/2161704305 >> 172.21.15.189:0/2161704305 conn(0x7feb240b2db0 msgr2=0x7feb2415cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.110+0000 7feb2a867700 1 -- 172.21.15.189:0/2161704305 shutdown_connections 2024-04-06T12:32:11.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.110+0000 7feb2a867700 1 -- 172.21.15.189:0/2161704305 wait complete. 2024-04-06T12:32:11.122 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-06T12:32:11.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.261+0000 7f1ffe197700 1 Processor -- start 2024-04-06T12:32:11.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.261+0000 7f1ffe197700 1 -- start start 2024-04-06T12:32:11.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.262+0000 7f1ffe197700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff80b4660 0x7f1ff80b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:11.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.262+0000 7f1ffe197700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1ff80b2fc0 0x7f1ff80b3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:11.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.262+0000 7f1ffe197700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff80cdbe0 con 0x7f1ff815a550 2024-04-06T12:32:11.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.262+0000 7f1ffe197700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff80b5000 con 0x7f1ff80b2fc0 2024-04-06T12:32:11.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.262+0000 7f1ffe197700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff815a940 con 0x7f1ff80b4660 2024-04-06T12:32:11.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.262+0000 7f1ff7fff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1ff80b2fc0 0x7f1ff80b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:11.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.262+0000 7f1ff6ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff80b4660 0x7f1ff80b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:11.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.262+0000 7f1ff77fe700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f1ff815a550 0x7f1ff80cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:55068/0 (socket says 172.21.15.189:55068) 2024-04-06T12:32:11.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.262+0000 7f1ff77fe700 1 -- 172.21.15.189:0/445908715 learned_addr learned my addr 172.21.15.189:0/445908715 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:32:11.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.262+0000 7f1ff7fff700 1 -- 172.21.15.189:0/445908715 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff80b4660 msgr2=0x7f1ff80b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:11.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.262+0000 7f1ff7fff700 1 --2- 172.21.15.189:0/445908715 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff80b4660 0x7f1ff80b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.263+0000 7f1ff7fff700 1 -- 172.21.15.189:0/445908715 >> v1:172.21.15.192:6789/0 conn(0x7f1ff815a550 legacy=0x7f1ff80cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:11.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.263+0000 7f1ff7fff700 1 -- 172.21.15.189:0/445908715 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff815ac00 con 0x7f1ff80b2fc0 2024-04-06T12:32:11.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.263+0000 7f1ff67fc700 1 -- 172.21.15.189:0/445908715 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635306210 0 0) 0x7f1ff80cdbe0 con 0x7f1ff815a550 2024-04-06T12:32:11.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.263+0000 7f1ff6ffd700 1 --2- 172.21.15.189:0/445908715 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff80b4660 0x7f1ff80b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:32:11.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.263+0000 7f1ff7fff700 1 --2- 172.21.15.189:0/445908715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff80b2fc0 0x7f1ff80b3430 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0009d90 tx=0x7f1ff000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32d114422dce5880 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:11.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.263+0000 7f1ff67fc700 1 -- 172.21.15.189:0/445908715 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff000e070 con 0x7f1ff80b2fc0 2024-04-06T12:32:11.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.263+0000 7f1ff67fc700 1 -- 172.21.15.189:0/445908715 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff000bb20 con 0x7f1ff80b2fc0 2024-04-06T12:32:11.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.263+0000 7f1ff67fc700 1 -- 172.21.15.189:0/445908715 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0012430 con 0x7f1ff80b2fc0 2024-04-06T12:32:11.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.263+0000 7f1ffe197700 1 -- 172.21.15.189:0/445908715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff80b2fc0 msgr2=0x7f1ff80b3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:11.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.263+0000 7f1ffe197700 1 --2- 172.21.15.189:0/445908715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff80b2fc0 0x7f1ff80b3430 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0009d90 tx=0x7f1ff000b330 comp rx=0 tx=0).stop 2024-04-06T12:32:11.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.264+0000 7f1ffe197700 1 -- 172.21.15.189:0/445908715 shutdown_connections 2024-04-06T12:32:11.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.264+0000 7f1ffe197700 1 --2- 172.21.15.189:0/445908715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff80b2fc0 0x7f1ff80b3430 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.264+0000 7f1ffe197700 1 --2- 172.21.15.189:0/445908715 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff80b4660 0x7f1ff80b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.264+0000 7f1ffe197700 1 -- 172.21.15.189:0/445908715 >> 172.21.15.189:0/445908715 conn(0x7f1ff80d00b0 msgr2=0x7f1ff80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:11.264 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.264+0000 7f1ffe197700 1 -- 172.21.15.189:0/445908715 shutdown_connections 2024-04-06T12:32:11.264 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.264+0000 7f1ffe197700 1 -- 172.21.15.189:0/445908715 wait complete. 2024-04-06T12:32:11.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.265+0000 7f1ffe197700 1 Processor -- start 2024-04-06T12:32:11.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.265+0000 7f1ffe197700 1 -- start start 2024-04-06T12:32:11.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.266+0000 7f1ffe197700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff80b2fc0 0x7f1ff814a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:11.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.266+0000 7f1ffe197700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff80b4660 0x7f1ff8158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:11.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.266+0000 7f1ffe197700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ff814c500 0x7f1ff81415a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:11.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.266+0000 7f1ffe197700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff8153a40 con 0x7f1ff814c500 2024-04-06T12:32:11.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.266+0000 7f1ffe197700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff814b980 con 0x7f1ff80b4660 2024-04-06T12:32:11.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.266+0000 7f1ffe197700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff814ae00 con 0x7f1ff80b2fc0 2024-04-06T12:32:11.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.266+0000 7f1ff77fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff80b2fc0 0x7f1ff814a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:11.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.266+0000 7f1ff6ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff80b4660 0x7f1ff8158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:11.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.266+0000 7f1ff7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ff814c500 0x7f1ff81415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:11.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.266+0000 7f1ff77fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff80b2fc0 0x7f1ff814a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:58120/0 (socket says 172.21.15.189:58120) 2024-04-06T12:32:11.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.266+0000 7f1ff77fe700 1 -- 172.21.15.189:0/4247531510 learned_addr learned my addr 172.21.15.189:0/4247531510 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:11.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.266+0000 7f1ff6ffd700 1 --2- 172.21.15.189:0/4247531510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff80b4660 0x7f1ff8158dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:33204/0 (socket says 172.21.15.189:33204) 2024-04-06T12:32:11.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.266+0000 7f1ff6ffd700 1 -- 172.21.15.189:0/4247531510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff80b2fc0 msgr2=0x7f1ff814a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:11.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.266+0000 7f1ff6ffd700 1 --2- 172.21.15.189:0/4247531510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff80b2fc0 0x7f1ff814a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.266+0000 7f1ff6ffd700 1 -- 172.21.15.189:0/4247531510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ff814c500 msgr2=0x7f1ff81415a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:11.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.266+0000 7f1ff6ffd700 1 --2- 172.21.15.189:0/4247531510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ff814c500 0x7f1ff81415a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.266+0000 7f1ff6ffd700 1 -- 172.21.15.189:0/4247531510 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec0094a0 con 0x7f1ff80b4660 2024-04-06T12:32:11.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.266+0000 7f1ff77fe700 1 --2- 172.21.15.189:0/4247531510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff80b2fc0 0x7f1ff814a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:32:11.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.266+0000 7f1ff6ffd700 1 --2- 172.21.15.189:0/4247531510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff80b4660 0x7f1ff8158dc0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00e9b0 tx=0x7f1fec00ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:11.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.267+0000 7f1fdffff700 1 -- 172.21.15.189:0/4247531510 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec00ee50 con 0x7f1ff80b4660 2024-04-06T12:32:11.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.267+0000 7f1fdffff700 1 -- 172.21.15.189:0/4247531510 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fec00c850 con 0x7f1ff80b4660 2024-04-06T12:32:11.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.267+0000 7f1fdffff700 1 -- 172.21.15.189:0/4247531510 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec00cc00 con 0x7f1ff80b4660 2024-04-06T12:32:11.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.267+0000 7f1ffe197700 1 -- 172.21.15.189:0/4247531510 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff0009a70 con 0x7f1ff80b4660 2024-04-06T12:32:11.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.267+0000 7f1ffe197700 1 -- 172.21.15.189:0/4247531510 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff814b3c0 con 0x7f1ff80b4660 2024-04-06T12:32:11.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.268+0000 7f1ffe197700 1 -- 172.21.15.189:0/4247531510 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff8153d00 con 0x7f1ff80b4660 2024-04-06T12:32:11.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.268+0000 7f1fdffff700 1 -- 172.21.15.189:0/4247531510 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 87) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f1fec013470 con 0x7f1ff80b4660 2024-04-06T12:32:11.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.268+0000 7f1fdffff700 1 --2- 172.21.15.189:0/4247531510 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1fe406ae00 0x7f1fe406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:11.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.268+0000 7f1ff77fe700 1 -- 172.21.15.189:0/4247531510 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1fe406ae00 msgr2=0x7f1fe406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:32:11.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.268+0000 7f1fdffff700 1 -- 172.21.15.189:0/4247531510 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1fec0896e0 con 0x7f1ff80b4660 2024-04-06T12:32:11.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.268+0000 7f1ff77fe700 1 --2- 172.21.15.189:0/4247531510 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1fe406ae00 0x7f1fe406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:32:11.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.273+0000 7f1fdffff700 1 -- 172.21.15.189:0/4247531510 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1fec0545d0 con 0x7f1ff80b4660 2024-04-06T12:32:11.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.469+0000 7f1ff77fe700 1 -- 172.21.15.189:0/4247531510 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1fe406ae00 msgr2=0x7f1fe406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:32:11.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.469+0000 7f1ff77fe700 1 --2- 172.21.15.189:0/4247531510 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1fe406ae00 0x7f1fe406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:32:11.482 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:11.484+0000 7f50d294f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:32:11.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.486+0000 7f1ffe197700 1 -- 172.21.15.189:0/4247531510 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f1ff8067bb0 con 0x7f1ff80b4660 2024-04-06T12:32:11.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.490+0000 7f1fdffff700 1 -- 172.21.15.189:0/4247531510 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v53)=0 v53) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1fec057bf0 con 0x7f1ff80b4660 2024-04-06T12:32:11.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.493+0000 7f1ffe197700 1 -- 172.21.15.189:0/4247531510 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1fe406ae00 msgr2=0x7f1fe406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:32:11.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.493+0000 7f1ffe197700 1 --2- 172.21.15.189:0/4247531510 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1fe406ae00 0x7f1fe406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.493+0000 7f1ffe197700 1 -- 172.21.15.189:0/4247531510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff80b4660 msgr2=0x7f1ff8158dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:11.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.493+0000 7f1ffe197700 1 --2- 172.21.15.189:0/4247531510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff80b4660 0x7f1ff8158dc0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00e9b0 tx=0x7f1fec00ecc0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.493+0000 7f1ffe197700 1 -- 172.21.15.189:0/4247531510 shutdown_connections 2024-04-06T12:32:11.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.493+0000 7f1ffe197700 1 --2- 172.21.15.189:0/4247531510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ff814c500 0x7f1ff81415a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.493+0000 7f1ffe197700 1 --2- 172.21.15.189:0/4247531510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff80b2fc0 0x7f1ff814a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.493+0000 7f1ffe197700 1 --2- 172.21.15.189:0/4247531510 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1fe406ae00 0x7f1fe406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.493+0000 7f1ffe197700 1 --2- 172.21.15.189:0/4247531510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff80b4660 0x7f1ff8158dc0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.493+0000 7f1ffe197700 1 -- 172.21.15.189:0/4247531510 >> 172.21.15.189:0/4247531510 conn(0x7f1ff80d00b0 msgr2=0x7f1ff806ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:11.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.493+0000 7f1ffe197700 1 -- 172.21.15.189:0/4247531510 shutdown_connections 2024-04-06T12:32:11.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.493+0000 7f1ffe197700 1 -- 172.21.15.189:0/4247531510 wait complete. 2024-04-06T12:32:11.504 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-06T12:32:11.516 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:11.519+0000 7f24b1537200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T12:32:11.561 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:11.563+0000 7f50d294f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:32:11.601 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:11.603+0000 7f24b1537200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T12:32:11.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.634+0000 7ff0035ce700 1 Processor -- start 2024-04-06T12:32:11.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.635+0000 7ff0035ce700 1 -- start start 2024-04-06T12:32:11.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.636+0000 7ff0035ce700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7feffc0b4660 0x7feffc0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:11.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.636+0000 7ff0035ce700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feffc0b5030 con 0x7feffc15a550 2024-04-06T12:32:11.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.636+0000 7ff0035ce700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feffc0cdbe0 con 0x7feffc0b2fc0 2024-04-06T12:32:11.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.636+0000 7ff0035ce700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feffc15a940 con 0x7feffc0b4660 2024-04-06T12:32:11.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.636+0000 7ff000b69700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7feffc0b4660 0x7feffc0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:11.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.636+0000 7ff000b69700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7feffc0b4660 0x7feffc0b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53102/0 (socket says 172.21.15.189:53102) 2024-04-06T12:32:11.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.636+0000 7ff000b69700 1 -- 172.21.15.189:0/344461835 learned_addr learned my addr 172.21.15.189:0/344461835 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:11.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.636+0000 7ff000b69700 1 -- 172.21.15.189:0/344461835 >> v1:172.21.15.192:6789/0 conn(0x7feffc0b2fc0 legacy=0x7feffc0b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:11.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.636+0000 7ff000b69700 1 -- 172.21.15.189:0/344461835 >> v1:172.21.15.189:6789/0 conn(0x7feffc15a550 legacy=0x7feffc0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:11.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.636+0000 7ff000b69700 1 -- 172.21.15.189:0/344461835 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc15ac00 con 0x7feffc0b4660 2024-04-06T12:32:11.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.636+0000 7ff000b69700 1 --2- 172.21.15.189:0/344461835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feffc0b4660 0x7feffc0b4af0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7feff00098f0 tx=0x7feff0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2232179b84c834e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:11.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.636+0000 7fefeffff700 1 -- 172.21.15.189:0/344461835 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff000b8a0 con 0x7feffc0b4660 2024-04-06T12:32:11.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.636+0000 7fefeffff700 1 -- 172.21.15.189:0/344461835 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff000ba00 con 0x7feffc0b4660 2024-04-06T12:32:11.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.636+0000 7fefeffff700 1 -- 172.21.15.189:0/344461835 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff000bd90 con 0x7feffc0b4660 2024-04-06T12:32:11.636 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:11.638+0000 7f50d294f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:32:11.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.637+0000 7ff0035ce700 1 -- 172.21.15.189:0/344461835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feffc0b4660 msgr2=0x7feffc0b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:11.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.637+0000 7ff0035ce700 1 --2- 172.21.15.189:0/344461835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feffc0b4660 0x7feffc0b4af0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7feff00098f0 tx=0x7feff0009c00 comp rx=0 tx=0).stop 2024-04-06T12:32:11.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.637+0000 7ff0035ce700 1 -- 172.21.15.189:0/344461835 shutdown_connections 2024-04-06T12:32:11.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.637+0000 7ff0035ce700 1 --2- 172.21.15.189:0/344461835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feffc0b4660 0x7feffc0b4af0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.637+0000 7ff0035ce700 1 -- 172.21.15.189:0/344461835 >> 172.21.15.189:0/344461835 conn(0x7feffc0d00b0 msgr2=0x7feffc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:11.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.637+0000 7ff0035ce700 1 -- 172.21.15.189:0/344461835 shutdown_connections 2024-04-06T12:32:11.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.637+0000 7ff0035ce700 1 -- 172.21.15.189:0/344461835 wait complete. 2024-04-06T12:32:11.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.638+0000 7ff0035ce700 1 Processor -- start 2024-04-06T12:32:11.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.638+0000 7ff0035ce700 1 -- start start 2024-04-06T12:32:11.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.639+0000 7ff0035ce700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feffc0b2fc0 0x7feffc153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:11.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.639+0000 7ff0035ce700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feffc0b4660 0x7feffc1ef890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:11.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.639+0000 7ff0035ce700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feffc155fa0 0x7feffc06d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:11.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.639+0000 7ff0035ce700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc0a8550 con 0x7feffc155fa0 2024-04-06T12:32:11.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.639+0000 7ff0035ce700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc0a8830 con 0x7feffc0b4660 2024-04-06T12:32:11.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.639+0000 7ff0035ce700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feffc0a8b10 con 0x7feffc0b2fc0 2024-04-06T12:32:11.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.639+0000 7ff001b6b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feffc155fa0 0x7feffc06d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:11.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.639+0000 7ff001b6b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feffc155fa0 0x7feffc06d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53108/0 (socket says 172.21.15.189:53108) 2024-04-06T12:32:11.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.639+0000 7ff001b6b700 1 -- 172.21.15.189:0/2712302242 learned_addr learned my addr 172.21.15.189:0/2712302242 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:11.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.639+0000 7ff000b69700 1 --2- 172.21.15.189:0/2712302242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feffc0b4660 0x7feffc1ef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:11.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.639+0000 7ff00136a700 1 --2- 172.21.15.189:0/2712302242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feffc0b2fc0 0x7feffc153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:11.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.640+0000 7ff001b6b700 1 -- 172.21.15.189:0/2712302242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feffc0b2fc0 msgr2=0x7feffc153fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:11.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.640+0000 7ff001b6b700 1 --2- 172.21.15.189:0/2712302242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feffc0b2fc0 0x7feffc153fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.640+0000 7ff001b6b700 1 -- 172.21.15.189:0/2712302242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feffc0b4660 msgr2=0x7feffc1ef890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:11.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.640+0000 7ff001b6b700 1 --2- 172.21.15.189:0/2712302242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feffc0b4660 0x7feffc1ef890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.640+0000 7ff001b6b700 1 -- 172.21.15.189:0/2712302242 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff0009550 con 0x7feffc155fa0 2024-04-06T12:32:11.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.640+0000 7ff001b6b700 1 --2- 172.21.15.189:0/2712302242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feffc155fa0 0x7feffc06d610 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7feff400cc50 tx=0x7feff400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:11.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.640+0000 7fefedffb700 1 -- 172.21.15.189:0/2712302242 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff4003c00 con 0x7feffc155fa0 2024-04-06T12:32:11.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.640+0000 7fefedffb700 1 -- 172.21.15.189:0/2712302242 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff4003d60 con 0x7feffc155fa0 2024-04-06T12:32:11.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.640+0000 7ff0035ce700 1 -- 172.21.15.189:0/2712302242 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feffc0a8df0 con 0x7feffc155fa0 2024-04-06T12:32:11.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.640+0000 7fefedffb700 1 -- 172.21.15.189:0/2712302242 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff400aab0 con 0x7feffc155fa0 2024-04-06T12:32:11.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.640+0000 7ff0035ce700 1 -- 172.21.15.189:0/2712302242 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feffc1544e0 con 0x7feffc155fa0 2024-04-06T12:32:11.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.641+0000 7ff0035ce700 1 -- 172.21.15.189:0/2712302242 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feffc0a9310 con 0x7feffc155fa0 2024-04-06T12:32:11.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.641+0000 7fefedffb700 1 -- 172.21.15.189:0/2712302242 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 87) v1 ==== 89264+0+0 (secure 0 0 0) 0x7feff400ac10 con 0x7feffc155fa0 2024-04-06T12:32:11.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.642+0000 7fefedffb700 1 --2- 172.21.15.189:0/2712302242 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fefe006ae00 0x7fefe006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:11.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.642+0000 7ff00136a700 1 -- 172.21.15.189:0/2712302242 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fefe006ae00 msgr2=0x7fefe006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:32:11.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.642+0000 7fefedffb700 1 -- 172.21.15.189:0/2712302242 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6029+0+0 (secure 0 0 0) 0x7feff40895c0 con 0x7feffc155fa0 2024-04-06T12:32:11.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.642+0000 7ff00136a700 1 --2- 172.21.15.189:0/2712302242 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fefe006ae00 0x7fefe006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:32:11.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.646+0000 7fefedffb700 1 -- 172.21.15.189:0/2712302242 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7feffc0a9310 con 0x7feffc155fa0 2024-04-06T12:32:11.681 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:11.683+0000 7f24b1537200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T12:32:11.703 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:11.705+0000 7f50d294f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:32:11.752 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:11.754+0000 7f24b1537200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T12:32:11.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.843+0000 7ff00136a700 1 -- 172.21.15.189:0/2712302242 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fefe006ae00 msgr2=0x7fefe006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:32:11.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.843+0000 7ff00136a700 1 --2- 172.21.15.189:0/2712302242 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fefe006ae00 0x7fefe006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:32:11.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.851+0000 7ff0035ce700 1 -- 172.21.15.189:0/2712302242 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7feffc0a9310 con 0x7feffc155fa0 2024-04-06T12:32:11.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.856+0000 7fefedffb700 1 -- 172.21.15.189:0/2712302242 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v54)=0 v54) v1 ==== 133+0+0 (secure 0 0 0) 0x7feffc0a9310 con 0x7feffc155fa0 2024-04-06T12:32:11.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.859+0000 7ff0035ce700 1 -- 172.21.15.189:0/2712302242 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fefe006ae00 msgr2=0x7fefe006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:32:11.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.859+0000 7ff0035ce700 1 --2- 172.21.15.189:0/2712302242 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fefe006ae00 0x7fefe006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.859+0000 7ff0035ce700 1 -- 172.21.15.189:0/2712302242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feffc155fa0 msgr2=0x7feffc06d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:11.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.859+0000 7ff0035ce700 1 --2- 172.21.15.189:0/2712302242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feffc155fa0 0x7feffc06d610 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7feff400cc50 tx=0x7feff400cf60 comp rx=0 tx=0).stop 2024-04-06T12:32:11.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.859+0000 7ff0035ce700 1 -- 172.21.15.189:0/2712302242 shutdown_connections 2024-04-06T12:32:11.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.859+0000 7ff0035ce700 1 --2- 172.21.15.189:0/2712302242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feffc155fa0 0x7feffc06d610 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.859+0000 7ff0035ce700 1 --2- 172.21.15.189:0/2712302242 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feffc0b2fc0 0x7feffc153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.859+0000 7ff0035ce700 1 --2- 172.21.15.189:0/2712302242 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fefe006ae00 0x7fefe006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.859+0000 7ff0035ce700 1 --2- 172.21.15.189:0/2712302242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feffc0b4660 0x7feffc1ef890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:11.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.859+0000 7ff0035ce700 1 -- 172.21.15.189:0/2712302242 >> 172.21.15.189:0/2712302242 conn(0x7feffc0d00b0 msgr2=0x7feffc154dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:11.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.859+0000 7ff0035ce700 1 -- 172.21.15.189:0/2712302242 shutdown_connections 2024-04-06T12:32:11.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:11.859+0000 7ff0035ce700 1 -- 172.21.15.189:0/2712302242 wait complete. 2024-04-06T12:32:11.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T12:32:11.870 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-06T12:32:11.878 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:11.880+0000 7f50d294f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:32:11.932 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1 1m 2024-04-06T12:32:11.934 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:11.937+0000 7f24b1537200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:32:12.036 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:12.038+0000 7f50d294f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:32:12.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.064+0000 7f3829c29700 1 Processor -- start 2024-04-06T12:32:12.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.064+0000 7f3829c29700 1 -- start start 2024-04-06T12:32:12.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.064+0000 7f3829c29700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f382415d7b0 0x7f38240b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:12.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.064+0000 7f3829c29700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f382406a730 0x7f382406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:12.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.065+0000 7f3829c29700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f382415ddf0 0x7f382415e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:12.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.065+0000 7f3829c29700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f382406b0f0 con 0x7f382406a730 2024-04-06T12:32:12.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.065+0000 7f3829c29700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38240b3990 con 0x7f382415d7b0 2024-04-06T12:32:12.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.065+0000 7f3829c29700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38240b3060 con 0x7f382415ddf0 2024-04-06T12:32:12.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.065+0000 7f3822ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f382406a730 0x7f382406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:12.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.065+0000 7f3822ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f382406a730 0x7f382406abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53114/0 (socket says 172.21.15.189:53114) 2024-04-06T12:32:12.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.065+0000 7f3822ffd700 1 -- 172.21.15.189:0/2304311611 learned_addr learned my addr 172.21.15.189:0/2304311611 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:12.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.065+0000 7f38237fe700 1 --2- 172.21.15.189:0/2304311611 >> v2:172.21.15.192:3300/0 conn(0x7f382415d7b0 0x7f38240b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:12.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.065+0000 7f3823fff700 1 --2- 172.21.15.189:0/2304311611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f382415ddf0 0x7f382415e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:12.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.065+0000 7f3822ffd700 1 -- 172.21.15.189:0/2304311611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f382415ddf0 msgr2=0x7f382415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:12.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.065+0000 7f3822ffd700 1 --2- 172.21.15.189:0/2304311611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f382415ddf0 0x7f382415e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:12.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.065+0000 7f3822ffd700 1 -- 172.21.15.189:0/2304311611 >> v2:172.21.15.192:3300/0 conn(0x7f382415d7b0 msgr2=0x7f38240b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:12.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.065+0000 7f3822ffd700 1 --2- 172.21.15.189:0/2304311611 >> v2:172.21.15.192:3300/0 conn(0x7f382415d7b0 0x7f38240b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:12.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.065+0000 7f3822ffd700 1 -- 172.21.15.189:0/2304311611 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38240b3340 con 0x7f382406a730 2024-04-06T12:32:12.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.065+0000 7f38237fe700 1 --2- 172.21.15.189:0/2304311611 >> v2:172.21.15.192:3300/0 conn(0x7f382415d7b0 0x7f38240b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:32:12.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.065+0000 7f3822ffd700 1 --2- 172.21.15.189:0/2304311611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f382406a730 0x7f382406abb0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f38240b6200 tx=0x7f3818009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b59236814e2acd5f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:12.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.065+0000 7f38227fc700 1 -- 172.21.15.189:0/2304311611 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381800b870 con 0x7f382406a730 2024-04-06T12:32:12.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.065+0000 7f38227fc700 1 -- 172.21.15.189:0/2304311611 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f381800b9d0 con 0x7f382406a730 2024-04-06T12:32:12.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.065+0000 7f38227fc700 1 -- 172.21.15.189:0/2304311611 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381800bd60 con 0x7f382406a730 2024-04-06T12:32:12.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.066+0000 7f3829c29700 1 -- 172.21.15.189:0/2304311611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f382406a730 msgr2=0x7f382406abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:12.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.066+0000 7f3829c29700 1 --2- 172.21.15.189:0/2304311611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f382406a730 0x7f382406abb0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f38240b6200 tx=0x7f3818009bd0 comp rx=0 tx=0).stop 2024-04-06T12:32:12.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.066+0000 7f3829c29700 1 -- 172.21.15.189:0/2304311611 shutdown_connections 2024-04-06T12:32:12.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.066+0000 7f3829c29700 1 --2- 172.21.15.189:0/2304311611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f382415ddf0 0x7f382415e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:12.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.066+0000 7f3829c29700 1 --2- 172.21.15.189:0/2304311611 >> v2:172.21.15.192:3300/0 conn(0x7f382415d7b0 0x7f38240b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:12.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.066+0000 7f3829c29700 1 --2- 172.21.15.189:0/2304311611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f382406a730 0x7f382406abb0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:12.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.066+0000 7f3829c29700 1 -- 172.21.15.189:0/2304311611 >> 172.21.15.189:0/2304311611 conn(0x7f38240d00b0 msgr2=0x7f38240d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:12.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.066+0000 7f3829c29700 1 -- 172.21.15.189:0/2304311611 shutdown_connections 2024-04-06T12:32:12.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.066+0000 7f3829c29700 1 -- 172.21.15.189:0/2304311611 wait complete. 2024-04-06T12:32:12.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.067+0000 7f3829c29700 1 Processor -- start 2024-04-06T12:32:12.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.067+0000 7f3829c29700 1 -- start start 2024-04-06T12:32:12.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.068+0000 7f3829c29700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f382406a730 0x7f382415c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:12.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.068+0000 7f3829c29700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f382415d7b0 0x7f38241f2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:12.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.068+0000 7f3829c29700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f382415ddf0 0x7f382415fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:12.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.068+0000 7f3829c29700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382415b560 con 0x7f382415ddf0 2024-04-06T12:32:12.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.068+0000 7f3829c29700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3824151160 con 0x7f382406a730 2024-04-06T12:32:12.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.068+0000 7f3829c29700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f382415caa0 con 0x7f382415d7b0 2024-04-06T12:32:12.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.068+0000 7f3823fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f382415ddf0 0x7f382415fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:12.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.068+0000 7f3823fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f382415ddf0 0x7f382415fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53124/0 (socket says 172.21.15.189:53124) 2024-04-06T12:32:12.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.068+0000 7f3823fff700 1 -- 172.21.15.189:0/4199946574 learned_addr learned my addr 172.21.15.189:0/4199946574 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:12.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.068+0000 7f3822ffd700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f382415d7b0 0x7f38241f2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:12.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.068+0000 7f38237fe700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f382406a730 0x7f382415c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:12.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.068+0000 7f3823fff700 1 -- 172.21.15.189:0/4199946574 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f382415d7b0 msgr2=0x7f38241f2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:12.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.068+0000 7f3823fff700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f382415d7b0 0x7f38241f2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:12.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.069+0000 7f3823fff700 1 -- 172.21.15.189:0/4199946574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f382406a730 msgr2=0x7f382415c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:12.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.069+0000 7f3823fff700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f382406a730 0x7f382415c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:12.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.069+0000 7f3823fff700 1 -- 172.21.15.189:0/4199946574 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3818009550 con 0x7f382415ddf0 2024-04-06T12:32:12.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.069+0000 7f3823fff700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f382415ddf0 0x7f382415fd10 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f381c007e60 tx=0x7f381c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:12.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.069+0000 7f380bfff700 1 -- 172.21.15.189:0/4199946574 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381c00ecd0 con 0x7f382415ddf0 2024-04-06T12:32:12.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.069+0000 7f380bfff700 1 -- 172.21.15.189:0/4199946574 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f381c00ee50 con 0x7f382415ddf0 2024-04-06T12:32:12.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.069+0000 7f3829c29700 1 -- 172.21.15.189:0/4199946574 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f382415a9e0 con 0x7f382415ddf0 2024-04-06T12:32:12.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.069+0000 7f380bfff700 1 -- 172.21.15.189:0/4199946574 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381c014660 con 0x7f382415ddf0 2024-04-06T12:32:12.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.069+0000 7f3829c29700 1 -- 172.21.15.189:0/4199946574 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f382415afa0 con 0x7f382415ddf0 2024-04-06T12:32:12.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.070+0000 7f3829c29700 1 -- 172.21.15.189:0/4199946574 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3824067bb0 con 0x7f382415ddf0 2024-04-06T12:32:12.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.070+0000 7f380bfff700 1 -- 172.21.15.189:0/4199946574 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 87) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f381c0147c0 con 0x7f382415ddf0 2024-04-06T12:32:12.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.071+0000 7f380bfff700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f381006ae00 0x7f381006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:12.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.071+0000 7f38237fe700 1 -- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f381006ae00 msgr2=0x7f381006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:32:12.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.071+0000 7f380bfff700 1 -- 172.21.15.189:0/4199946574 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f381c089b50 con 0x7f382415ddf0 2024-04-06T12:32:12.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.071+0000 7f38237fe700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f381006ae00 0x7f381006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T12:32:12.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.079+0000 7f380bfff700 1 -- 172.21.15.189:0/4199946574 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f381c054b90 con 0x7f382415ddf0 2024-04-06T12:32:12.105 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:12.107+0000 7f24b1537200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T12:32:12.147 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:12.149+0000 7f50d294f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:32:12.219 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:12.222+0000 7f24b1537200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T12:32:12.228 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:12.231+0000 7f50d294f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:32:12.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.272+0000 7f38237fe700 1 -- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f381006ae00 msgr2=0x7f381006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:32:12.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.272+0000 7f38237fe700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f381006ae00 0x7f381006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T12:32:12.303 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:12.305+0000 7f50d294f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:32:12.303 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:12.305+0000 7f24b1537200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T12:32:12.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.314+0000 7f3829c29700 1 -- 172.21.15.189:0/4199946574 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f382415e7c0 con 0x7f381006ae00 2024-04-06T12:32:12.378 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:12.380+0000 7f24b1537200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T12:32:12.504 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:12.506+0000 7f50d294f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:32:12.582 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:12.582+0000 7f24b1537200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T12:32:12.582 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:12.583+0000 7f50d294f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:32:12.655 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:12.657+0000 7f24b1537200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T12:32:12.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.673+0000 7f38237fe700 1 -- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f381006ae00 msgr2=0x7f381006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:32:12.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:12.673+0000 7f38237fe700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f381006ae00 0x7f381006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-06T12:32:12.900 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:12.902+0000 7f50d294f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:32:12.974 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:12.977+0000 7f24b1537200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T12:32:13.218 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:13.220+0000 7f50d294f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:32:13.285 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:32:13.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:32:13.297 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:13.299+0000 7f24b1537200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T12:32:13.381 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:13.383+0000 7f50d294f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:32:13.454 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:13.456+0000 7f50d294f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:32:13.460 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:13.462+0000 7f24b1537200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T12:32:13.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:13.474+0000 7f38237fe700 1 -- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f381006ae00 msgr2=0x7f381006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:32:13.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:13.474+0000 7f38237fe700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f381006ae00 0x7f381006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-06T12:32:13.535 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:13.537+0000 7f24b1537200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T12:32:14.067 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:14.069+0000 7f50d294f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:32:14.145 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:14.147+0000 7f50d294f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:32:14.157 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:14.159+0000 7f24b1537200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T12:32:14.236 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:14.238+0000 7f24b1537200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T12:32:14.269 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:14.271+0000 7f50d294f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:32:14.362 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:14.363+0000 7f24b1537200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T12:32:14.776 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:14.778+0000 7f50d294f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:32:14.873 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:14.875+0000 7f24b1537200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T12:32:14.930 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:14.932+0000 7f50d294f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:32:15.028 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:15.030+0000 7f24b1537200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T12:32:15.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:15.075+0000 7f38237fe700 1 -- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f381006ae00 msgr2=0x7f381006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T12:32:15.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:15.075+0000 7f38237fe700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f381006ae00 0x7f381006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-06T12:32:15.085 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:15.087+0000 7f50d294f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:32:15.186 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:15.188+0000 7f24b1537200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T12:32:15.804 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:15.806+0000 7f50d294f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:32:15.912 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:15.914+0000 7f24b1537200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T12:32:16.046 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:16.048+0000 7f50d294f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:32:16.122 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:16.124+0000 7f50d294f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:32:16.157 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:16.159+0000 7f24b1537200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T12:32:16.233 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:16.236+0000 7f24b1537200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T12:32:16.494 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T12:32:16.495+0000 7f50d294f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:32:16.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:16.567+0000 7f380bfff700 1 -- 172.21.15.189:0/4199946574 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 88) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f381c087f30 con 0x7f382415ddf0 2024-04-06T12:32:16.613 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:16.615+0000 7f24b1537200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T12:32:16.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:16.638+0000 7f380bfff700 1 -- 172.21.15.189:0/4199946574 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mgrmap(e 89) v1 ==== 88873+0+0 (secure 0 0 0) 0x7f381c0703b0 con 0x7f382415ddf0 2024-04-06T12:32:16.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:16.639+0000 7f380bfff700 1 -- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f381006ae00 msgr2=0x7f381006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T12:32:16.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:16.639+0000 7f380bfff700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f381006ae00 0x7f381006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:16.788 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:16.790+0000 7f2424405700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:32:16.789 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:16.790+0000 7f2424405700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:32:16.789 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:16.790+0000 7f2424405700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:32:16.789 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:16.790+0000 7f2424405700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:32:16.789 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:16.790+0000 7f2424405700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:32:16.789 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:16.790+0000 7f242740b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:32:16.789 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:16.790+0000 7f242740b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:32:16.789 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:16.790+0000 7f242740b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:32:16.789 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:16.790+0000 7f242740b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:32:16.789 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:32:16.790+0000 7f242740b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:32:17.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.640+0000 7f380bfff700 1 -- 172.21.15.189:0/4199946574 <== mon.0 v2:172.21.15.189:3300/0 9 ==== mgrmap(e 90) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f381c03ced0 con 0x7f382415ddf0 2024-04-06T12:32:17.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.640+0000 7f380bfff700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f381006ae00 0x7f381006d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:17.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.640+0000 7f380bfff700 1 -- 172.21.15.189:0/4199946574 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3810035470 con 0x7f381006ae00 2024-04-06T12:32:17.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.642+0000 7f38237fe700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f381006ae00 0x7f381006d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:17.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.645+0000 7f38237fe700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f381006ae00 0x7f381006d240 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f38140095f0 tx=0x7f3814008040 comp rx=0 tx=0).ready entity=mgr.9303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:17.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.704+0000 7f380bfff700 1 -- 172.21.15.189:0/4199946574 <== mgr.9303 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f3810035470 con 0x7f381006ae00 2024-04-06T12:32:17.703 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path /snap_test_dir1 2024-04-06T12:32:17.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.706+0000 7f3829c29700 1 -- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f381006ae00 msgr2=0x7f381006d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:17.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.706+0000 7f3829c29700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f381006ae00 0x7f381006d240 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f38140095f0 tx=0x7f3814008040 comp rx=0 tx=0).stop 2024-04-06T12:32:17.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.706+0000 7f3829c29700 1 -- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f382415ddf0 msgr2=0x7f382415fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:17.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.706+0000 7f3829c29700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f382415ddf0 0x7f382415fd10 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f381c007e60 tx=0x7f381c00e3c0 comp rx=0 tx=0).stop 2024-04-06T12:32:17.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.707+0000 7f3829c29700 1 -- 172.21.15.189:0/4199946574 shutdown_connections 2024-04-06T12:32:17.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.707+0000 7f3829c29700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f382415ddf0 0x7f382415fd10 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:17.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.707+0000 7f3829c29700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f381006ae00 0x7f381006d240 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:17.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.707+0000 7f3829c29700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f382406a730 0x7f382415c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:17.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.707+0000 7f3829c29700 1 --2- 172.21.15.189:0/4199946574 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f382415d7b0 0x7f38241f2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:17.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.707+0000 7f3829c29700 1 -- 172.21.15.189:0/4199946574 >> 172.21.15.189:0/4199946574 conn(0x7f38240d00b0 msgr2=0x7f38240ae210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:17.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.707+0000 7f3829c29700 1 -- 172.21.15.189:0/4199946574 shutdown_connections 2024-04-06T12:32:17.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.707+0000 7f3829c29700 1 -- 172.21.15.189:0/4199946574 wait complete. 2024-04-06T12:32:17.718 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1 2024-04-06T12:32:17.719 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add /snap_test_dir1 55n 2024-04-06T12:32:17.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.848+0000 7fd1e95cf700 1 Processor -- start 2024-04-06T12:32:17.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.848+0000 7fd1e95cf700 1 -- start start 2024-04-06T12:32:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.849+0000 7fd1e95cf700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd1e40d17a0 0x7fd1e406b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.849+0000 7fd1e95cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1e40cedf0 0x7fd1e40cf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.849+0000 7fd1e95cf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e40cf780 con 0x7fd1e4146dd0 2024-04-06T12:32:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.849+0000 7fd1e95cf700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1e406d850 con 0x7fd1e40d17a0 2024-04-06T12:32:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.849+0000 7fd1e95cf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e406c0a0 con 0x7fd1e40cedf0 2024-04-06T12:32:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.849+0000 7fd1e27fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd1e4146dd0 0x7fd1e406d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50438/0 (socket says 172.21.15.189:50438) 2024-04-06T12:32:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.849+0000 7fd1e27fc700 1 -- 172.21.15.189:0/3082298567 learned_addr learned my addr 172.21.15.189:0/3082298567 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:32:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.849+0000 7fd1e37fe700 1 --2- 172.21.15.189:0/3082298567 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1e40cedf0 0x7fd1e40cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.849+0000 7fd1e2ffd700 1 --2- 172.21.15.189:0/3082298567 >> v2:172.21.15.192:3300/0 conn(0x7fd1e40d17a0 0x7fd1e406b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.849+0000 7fd1e1ffb700 1 -- 172.21.15.189:0/3082298567 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225132958 0 0) 0x7fd1d4001110 con 0x7fd1e4146dd0 2024-04-06T12:32:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.849+0000 7fd1e1ffb700 1 -- 172.21.15.189:0/3082298567 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1cc0035b0 con 0x7fd1e4146dd0 2024-04-06T12:32:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.849+0000 7fd1e1ffb700 1 -- 172.21.15.189:0/3082298567 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2609795222 0 0) 0x7fd1cc0035b0 con 0x7fd1e4146dd0 2024-04-06T12:32:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.849+0000 7fd1e37fe700 1 -- 172.21.15.189:0/3082298567 >> v2:172.21.15.192:3300/0 conn(0x7fd1e40d17a0 msgr2=0x7fd1e406b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.849+0000 7fd1e37fe700 1 --2- 172.21.15.189:0/3082298567 >> v2:172.21.15.192:3300/0 conn(0x7fd1e40d17a0 0x7fd1e406b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.849+0000 7fd1e37fe700 1 -- 172.21.15.189:0/3082298567 >> v1:172.21.15.189:6789/0 conn(0x7fd1e4146dd0 legacy=0x7fd1e406d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.849+0000 7fd1e37fe700 1 -- 172.21.15.189:0/3082298567 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e406c380 con 0x7fd1e40cedf0 2024-04-06T12:32:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.849+0000 7fd1e2ffd700 1 --2- 172.21.15.189:0/3082298567 >> v2:172.21.15.192:3300/0 conn(0x7fd1e40d17a0 0x7fd1e406b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T12:32:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.850+0000 7fd1e37fe700 1 --2- 172.21.15.189:0/3082298567 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1e40cedf0 0x7fd1e40cf240 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc009d90 tx=0x7fd1dc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8c8bb0f41c21dc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.850+0000 7fd1e1ffb700 1 -- 172.21.15.189:0/3082298567 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc00e070 con 0x7fd1e40cedf0 2024-04-06T12:32:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.850+0000 7fd1e1ffb700 1 -- 172.21.15.189:0/3082298567 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1dc003a20 con 0x7fd1e40cedf0 2024-04-06T12:32:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.850+0000 7fd1e1ffb700 1 -- 172.21.15.189:0/3082298567 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4001800 con 0x7fd1e40cedf0 2024-04-06T12:32:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.850+0000 7fd1e95cf700 1 -- 172.21.15.189:0/3082298567 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1e40cedf0 msgr2=0x7fd1e40cf240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.850+0000 7fd1e95cf700 1 --2- 172.21.15.189:0/3082298567 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1e40cedf0 0x7fd1e40cf240 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc009d90 tx=0x7fd1dc00b330 comp rx=0 tx=0).stop 2024-04-06T12:32:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.850+0000 7fd1e95cf700 1 -- 172.21.15.189:0/3082298567 shutdown_connections 2024-04-06T12:32:17.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.850+0000 7fd1e95cf700 1 --2- 172.21.15.189:0/3082298567 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1e40cedf0 0x7fd1e40cf240 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:17.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.850+0000 7fd1e95cf700 1 --2- 172.21.15.189:0/3082298567 >> v2:172.21.15.192:3300/0 conn(0x7fd1e40d17a0 0x7fd1e406b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:17.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.850+0000 7fd1e95cf700 1 -- 172.21.15.189:0/3082298567 >> 172.21.15.189:0/3082298567 conn(0x7fd1e40b2db0 msgr2=0x7fd1e40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:17.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.850+0000 7fd1e95cf700 1 -- 172.21.15.189:0/3082298567 shutdown_connections 2024-04-06T12:32:17.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.850+0000 7fd1e95cf700 1 -- 172.21.15.189:0/3082298567 wait complete. 2024-04-06T12:32:17.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.852+0000 7fd1e95cf700 1 Processor -- start 2024-04-06T12:32:17.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.852+0000 7fd1e95cf700 1 -- start start 2024-04-06T12:32:17.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.852+0000 7fd1e95cf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1e40cedf0 0x7fd1e415c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:17.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.852+0000 7fd1e95cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1e40d17a0 0x7fd1e40ae320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.852+0000 7fd1e95cf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1e4146dd0 0x7fd1e40ae860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1e95cf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e41f63f0 con 0x7fd1e4146dd0 2024-04-06T12:32:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1e95cf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e40af680 con 0x7fd1e40cedf0 2024-04-06T12:32:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1e95cf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e40b4cf0 con 0x7fd1e40d17a0 2024-04-06T12:32:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1e37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1e4146dd0 0x7fd1e40ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1e37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1e4146dd0 0x7fd1e40ae860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53250/0 (socket says 172.21.15.189:53250) 2024-04-06T12:32:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1e37fe700 1 -- 172.21.15.189:0/1138353069 learned_addr learned my addr 172.21.15.189:0/1138353069 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1e2ffd700 1 --2- 172.21.15.189:0/1138353069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1e40cedf0 0x7fd1e415c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1e27fc700 1 --2- 172.21.15.189:0/1138353069 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1e40d17a0 0x7fd1e40ae320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1e37fe700 1 -- 172.21.15.189:0/1138353069 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1e40d17a0 msgr2=0x7fd1e40ae320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1e37fe700 1 --2- 172.21.15.189:0/1138353069 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1e40d17a0 0x7fd1e40ae320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1e37fe700 1 -- 172.21.15.189:0/1138353069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1e40cedf0 msgr2=0x7fd1e415c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1e37fe700 1 --2- 172.21.15.189:0/1138353069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1e40cedf0 0x7fd1e415c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1e37fe700 1 -- 172.21.15.189:0/1138353069 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d80094a0 con 0x7fd1e4146dd0 2024-04-06T12:32:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1e2ffd700 1 --2- 172.21.15.189:0/1138353069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1e40cedf0 0x7fd1e415c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:32:17.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1e27fc700 1 --2- 172.21.15.189:0/1138353069 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1e40d17a0 0x7fd1e40ae320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:32:17.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1e37fe700 1 --2- 172.21.15.189:0/1138353069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1e4146dd0 0x7fd1e40ae860 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40677e0 tx=0x7fd1dc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:17.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1cb7fe700 1 -- 172.21.15.189:0/1138353069 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc0045f0 con 0x7fd1e4146dd0 2024-04-06T12:32:17.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1cb7fe700 1 -- 172.21.15.189:0/1138353069 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1dc004750 con 0x7fd1e4146dd0 2024-04-06T12:32:17.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1cb7fe700 1 -- 172.21.15.189:0/1138353069 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc009450 con 0x7fd1e4146dd0 2024-04-06T12:32:17.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1e95cf700 1 -- 172.21.15.189:0/1138353069 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1dc009a70 con 0x7fd1e4146dd0 2024-04-06T12:32:17.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.853+0000 7fd1e95cf700 1 -- 172.21.15.189:0/1138353069 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e40b52b0 con 0x7fd1e4146dd0 2024-04-06T12:32:17.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.854+0000 7fd1e95cf700 1 -- 172.21.15.189:0/1138353069 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e4074990 con 0x7fd1e4146dd0 2024-04-06T12:32:17.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.855+0000 7fd1cb7fe700 1 -- 172.21.15.189:0/1138353069 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 90) v1 ==== 89000+0+0 (secure 0 0 0) 0x7fd1dc012450 con 0x7fd1e4146dd0 2024-04-06T12:32:17.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.855+0000 7fd1cb7fe700 1 --2- 172.21.15.189:0/1138353069 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1d006ab30 0x7fd1d006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:17.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.855+0000 7fd1e2ffd700 1 --2- 172.21.15.189:0/1138353069 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1d006ab30 0x7fd1d006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:17.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.855+0000 7fd1cb7fe700 1 -- 172.21.15.189:0/1138353069 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fd1dc08abc0 con 0x7fd1e4146dd0 2024-04-06T12:32:17.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.856+0000 7fd1e2ffd700 1 --2- 172.21.15.189:0/1138353069 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1d006ab30 0x7fd1d006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8009d20 tx=0x7fd1d8008040 comp rx=0 tx=0).ready entity=mgr.9303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:17.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:17.860+0000 7fd1cb7fe700 1 -- 172.21.15.189:0/1138353069 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd1dc0598d0 con 0x7fd1e4146dd0 2024-04-06T12:32:18.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.079+0000 7fd1e95cf700 1 -- 172.21.15.189:0/1138353069 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "/snap_test_dir1", "retention_spec_or_period": "55n", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1e406cc20 con 0x7fd1d006ab30 2024-04-06T12:32:18.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.133+0000 7fd1cb7fe700 1 -- 172.21.15.189:0/1138353069 <== mgr.9303 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+40 (secure 0 0 0) 0x7fd1d8009970 con 0x7fd1d006ab30 2024-04-06T12:32:18.131 INFO:teuthology.orchestra.run.smithi189.stdout:Retention added to path /snap_test_dir1 2024-04-06T12:32:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.135+0000 7fd1e95cf700 1 -- 172.21.15.189:0/1138353069 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1d006ab30 msgr2=0x7fd1d006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.135+0000 7fd1e95cf700 1 --2- 172.21.15.189:0/1138353069 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1d006ab30 0x7fd1d006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8009d20 tx=0x7fd1d8008040 comp rx=0 tx=0).stop 2024-04-06T12:32:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.135+0000 7fd1e95cf700 1 -- 172.21.15.189:0/1138353069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1e4146dd0 msgr2=0x7fd1e40ae860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.135+0000 7fd1e95cf700 1 --2- 172.21.15.189:0/1138353069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1e4146dd0 0x7fd1e40ae860 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40677e0 tx=0x7fd1dc004ad0 comp rx=0 tx=0).stop 2024-04-06T12:32:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.135+0000 7fd1e95cf700 1 -- 172.21.15.189:0/1138353069 shutdown_connections 2024-04-06T12:32:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.135+0000 7fd1e95cf700 1 --2- 172.21.15.189:0/1138353069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1e4146dd0 0x7fd1e40ae860 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.135+0000 7fd1e95cf700 1 --2- 172.21.15.189:0/1138353069 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd1d006ab30 0x7fd1d006cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.135+0000 7fd1e95cf700 1 --2- 172.21.15.189:0/1138353069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1e40cedf0 0x7fd1e415c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.135+0000 7fd1e95cf700 1 --2- 172.21.15.189:0/1138353069 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1e40d17a0 0x7fd1e40ae320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.135+0000 7fd1e95cf700 1 -- 172.21.15.189:0/1138353069 >> 172.21.15.189:0/1138353069 conn(0x7fd1e40b2db0 msgr2=0x7fd1e415e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.135+0000 7fd1e95cf700 1 -- 172.21.15.189:0/1138353069 shutdown_connections 2024-04-06T12:32:18.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.135+0000 7fd1e95cf700 1 -- 172.21.15.189:0/1138353069 wait complete. 2024-04-06T12:32:18.146 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path /snap_test_dir1 2024-04-06T12:32:18.146 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2024-04-06T12:32:18.146 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json /snap_test_dir1 2024-04-06T12:32:18.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.272+0000 7f4e686d4700 1 Processor -- start 2024-04-06T12:32:18.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.272+0000 7f4e686d4700 1 -- start start 2024-04-06T12:32:18.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.273+0000 7f4e686d4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e600b2bf0 0x7f4e600b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:18.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.273+0000 7f4e686d4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4e6015da10 0x7f4e600ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:18.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.273+0000 7f4e686d4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e60161000 con 0x7f4e600d0a40 2024-04-06T12:32:18.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.273+0000 7f4e686d4700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e600b3510 con 0x7f4e6015da10 2024-04-06T12:32:18.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.273+0000 7f4e686d4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6015dfb0 con 0x7f4e600b2bf0 2024-04-06T12:32:18.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.273+0000 7f4e65c6f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4e6015da10 0x7f4e600ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:18.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.273+0000 7f4e66c71700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4e600d0a40 0x7f4e60161490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50450/0 (socket says 172.21.15.189:50450) 2024-04-06T12:32:18.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.273+0000 7f4e66c71700 1 -- 172.21.15.189:0/2607812439 learned_addr learned my addr 172.21.15.189:0/2607812439 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T12:32:18.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.274+0000 7f4e66470700 1 --2- 172.21.15.189:0/2607812439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e600b2bf0 0x7f4e600b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:18.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.274+0000 7f4e65c6f700 1 -- 172.21.15.189:0/2607812439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e600b2bf0 msgr2=0x7f4e600b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T12:32:18.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.274+0000 7f4e65c6f700 1 --2- 172.21.15.189:0/2607812439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e600b2bf0 0x7f4e600b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:18.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.274+0000 7f4e65c6f700 1 -- 172.21.15.189:0/2607812439 >> v1:172.21.15.189:6789/0 conn(0x7f4e600d0a40 legacy=0x7f4e60161490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:18.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.274+0000 7f4e65c6f700 1 -- 172.21.15.189:0/2607812439 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6015e290 con 0x7f4e6015da10 2024-04-06T12:32:18.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.274+0000 7f4e6546e700 1 -- 172.21.15.189:0/2607812439 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132786604 0 0) 0x7f4e60161000 con 0x7f4e600d0a40 2024-04-06T12:32:18.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.274+0000 7f4e66470700 1 --2- 172.21.15.189:0/2607812439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e600b2bf0 0x7f4e600b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T12:32:18.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.274+0000 7f4e65c6f700 1 --2- 172.21.15.189:0/2607812439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e6015da10 0x7f4e600ceee0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c009840 tx=0x7f4e5c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6ef0bd214289920 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:18.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.274+0000 7f4e6546e700 1 -- 172.21.15.189:0/2607812439 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c011830 con 0x7f4e6015da10 2024-04-06T12:32:18.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.274+0000 7f4e6546e700 1 -- 172.21.15.189:0/2607812439 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5c011990 con 0x7f4e6015da10 2024-04-06T12:32:18.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.274+0000 7f4e6546e700 1 -- 172.21.15.189:0/2607812439 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c011d20 con 0x7f4e6015da10 2024-04-06T12:32:18.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.275+0000 7f4e686d4700 1 -- 172.21.15.189:0/2607812439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e6015da10 msgr2=0x7f4e600ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:18.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.275+0000 7f4e686d4700 1 --2- 172.21.15.189:0/2607812439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e6015da10 0x7f4e600ceee0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c009840 tx=0x7f4e5c009b50 comp rx=0 tx=0).stop 2024-04-06T12:32:18.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.275+0000 7f4e686d4700 1 -- 172.21.15.189:0/2607812439 shutdown_connections 2024-04-06T12:32:18.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.275+0000 7f4e686d4700 1 --2- 172.21.15.189:0/2607812439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e600b2bf0 0x7f4e600b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:18.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.275+0000 7f4e686d4700 1 --2- 172.21.15.189:0/2607812439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e6015da10 0x7f4e600ceee0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:18.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.275+0000 7f4e686d4700 1 -- 172.21.15.189:0/2607812439 >> 172.21.15.189:0/2607812439 conn(0x7f4e60066b80 msgr2=0x7f4e6006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:18.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.275+0000 7f4e686d4700 1 -- 172.21.15.189:0/2607812439 shutdown_connections 2024-04-06T12:32:18.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.275+0000 7f4e686d4700 1 -- 172.21.15.189:0/2607812439 wait complete. 2024-04-06T12:32:18.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.276+0000 7f4e686d4700 1 Processor -- start 2024-04-06T12:32:18.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.276+0000 7f4e686d4700 1 -- start start 2024-04-06T12:32:18.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e686d4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e600b2bf0 0x7f4e601f35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:18.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e686d4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e600d0a40 0x7f4e600b5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:18.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e686d4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e6015da10 0x7f4e6015a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:18.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e686d4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e600b4ae0 con 0x7f4e600b2bf0 2024-04-06T12:32:18.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e686d4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e600b5900 con 0x7f4e6015da10 2024-04-06T12:32:18.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e686d4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e60149310 con 0x7f4e600d0a40 2024-04-06T12:32:18.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e66470700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e600b2bf0 0x7f4e601f35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:18.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e66470700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e600b2bf0 0x7f4e601f35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53276/0 (socket says 172.21.15.189:53276) 2024-04-06T12:32:18.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e66470700 1 -- 172.21.15.189:0/3434694272 learned_addr learned my addr 172.21.15.189:0/3434694272 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T12:32:18.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e65c6f700 1 --2- 172.21.15.189:0/3434694272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e600d0a40 0x7f4e600b5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:18.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e66c71700 1 --2- 172.21.15.189:0/3434694272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e6015da10 0x7f4e6015a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:18.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e66470700 1 -- 172.21.15.189:0/3434694272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e600d0a40 msgr2=0x7f4e600b5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:18.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e66470700 1 --2- 172.21.15.189:0/3434694272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e600d0a40 0x7f4e600b5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:18.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e66470700 1 -- 172.21.15.189:0/3434694272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e6015da10 msgr2=0x7f4e6015a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:18.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e66470700 1 --2- 172.21.15.189:0/3434694272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e6015da10 0x7f4e6015a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:18.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e66470700 1 -- 172.21.15.189:0/3434694272 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c0094a0 con 0x7f4e600b2bf0 2024-04-06T12:32:18.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e66c71700 1 --2- 172.21.15.189:0/3434694272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e6015da10 0x7f4e6015a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T12:32:18.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e66470700 1 --2- 172.21.15.189:0/3434694272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e600b2bf0 0x7f4e601f35b0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f4e5000c780 tx=0x7f4e5000ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:18.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e56ffd700 1 -- 172.21.15.189:0/3434694272 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e50011800 con 0x7f4e600b2bf0 2024-04-06T12:32:18.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e56ffd700 1 -- 172.21.15.189:0/3434694272 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e50011960 con 0x7f4e600b2bf0 2024-04-06T12:32:18.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e686d4700 1 -- 172.21.15.189:0/3434694272 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e600b4cf0 con 0x7f4e600b2bf0 2024-04-06T12:32:18.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e56ffd700 1 -- 172.21.15.189:0/3434694272 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e50011cd0 con 0x7f4e600b2bf0 2024-04-06T12:32:18.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.277+0000 7f4e686d4700 1 -- 172.21.15.189:0/3434694272 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e600b5080 con 0x7f4e600b2bf0 2024-04-06T12:32:18.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.278+0000 7f4e686d4700 1 -- 172.21.15.189:0/3434694272 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e600b5be0 con 0x7f4e600b2bf0 2024-04-06T12:32:18.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.279+0000 7f4e56ffd700 1 -- 172.21.15.189:0/3434694272 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 90) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f4e50023480 con 0x7f4e600b2bf0 2024-04-06T12:32:18.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.279+0000 7f4e56ffd700 1 --2- 172.21.15.189:0/3434694272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e4406ab30 0x7f4e4406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T12:32:18.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.280+0000 7f4e56ffd700 1 -- 172.21.15.189:0/3434694272 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f4e50089530 con 0x7f4e600b2bf0 2024-04-06T12:32:18.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.280+0000 7f4e65c6f700 1 --2- 172.21.15.189:0/3434694272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e4406ab30 0x7f4e4406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T12:32:18.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.280+0000 7f4e65c6f700 1 --2- 172.21.15.189:0/3434694272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e4406ab30 0x7f4e4406cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c0090b0 tx=0x7f4e5c01c040 comp rx=0 tx=0).ready entity=mgr.9303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T12:32:18.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.283+0000 7f4e56ffd700 1 -- 172.21.15.189:0/3434694272 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4e50054870 con 0x7f4e600b2bf0 2024-04-06T12:32:18.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.503+0000 7f4e686d4700 1 -- 172.21.15.189:0/3434694272 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4e6015e850 con 0x7f4e4406ab30 2024-04-06T12:32:18.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.512+0000 7f4e56ffd700 1 -- 172.21.15.189:0/3434694272 <== mgr.9303 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+74 (secure 0 0 0) 0x7f4e5c011c80 con 0x7f4e4406ab30 2024-04-06T12:32:18.510 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T12:32:18.510 INFO:teuthology.orchestra.run.smithi189.stdout:{"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2024-04-06T12:32:18.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.514+0000 7f4e686d4700 1 -- 172.21.15.189:0/3434694272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e4406ab30 msgr2=0x7f4e4406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:18.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.514+0000 7f4e686d4700 1 --2- 172.21.15.189:0/3434694272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e4406ab30 0x7f4e4406cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c0090b0 tx=0x7f4e5c01c040 comp rx=0 tx=0).stop 2024-04-06T12:32:18.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.514+0000 7f4e686d4700 1 -- 172.21.15.189:0/3434694272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e600b2bf0 msgr2=0x7f4e601f35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T12:32:18.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.514+0000 7f4e686d4700 1 --2- 172.21.15.189:0/3434694272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e600b2bf0 0x7f4e601f35b0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f4e5000c780 tx=0x7f4e5000ca90 comp rx=0 tx=0).stop 2024-04-06T12:32:18.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.514+0000 7f4e686d4700 1 -- 172.21.15.189:0/3434694272 shutdown_connections 2024-04-06T12:32:18.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.514+0000 7f4e686d4700 1 --2- 172.21.15.189:0/3434694272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4e4406ab30 0x7f4e4406cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:18.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.514+0000 7f4e686d4700 1 --2- 172.21.15.189:0/3434694272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e6015da10 0x7f4e6015a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:18.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.514+0000 7f4e686d4700 1 --2- 172.21.15.189:0/3434694272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e600b2bf0 0x7f4e601f35b0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:18.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.514+0000 7f4e686d4700 1 --2- 172.21.15.189:0/3434694272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e600d0a40 0x7f4e600b5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T12:32:18.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.514+0000 7f4e686d4700 1 -- 172.21.15.189:0/3434694272 >> 172.21.15.189:0/3434694272 conn(0x7f4e60066b80 msgr2=0x7f4e6006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T12:32:18.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.515+0000 7f4e686d4700 1 -- 172.21.15.189:0/3434694272 shutdown_connections 2024-04-06T12:32:18.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T12:32:18.515+0000 7f4e686d4700 1 -- 172.21.15.189:0/3434694272 wait complete. 2024-04-06T12:32:18.525 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2024-04-06T12:32:18.525 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': '/snap_test_dir1', 'schedule': ['1m'], 'retention': [{'n': 55}]} 2024-04-06T12:32:43.352 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:32:43.355 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:33:00.141 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:33:00.143+0000 7f24163e9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:33:00.141 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:33:00.143+0000 7f24163e9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:33:00.141 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:33:00.143+0000 7f24163e9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:33:00.141 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:33:00.143+0000 7f24163e9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:33:00.142 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T12:33:00.143+0000 7f24163e9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T12:33:13.424 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:33:13.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:33:43.505 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:33:43.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:34:13.587 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:34:13.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:34:43.659 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:34:43.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:35:13.731 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:35:13.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:35:43.811 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:35:43.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:36:13.881 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:36:13.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:36:43.955 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:36:43.959 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:37:14.092 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:37:14.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:37:44.157 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:37:44.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:38:14.236 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:38:14.240 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:38:44.325 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:38:44.329 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:39:14.389 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:39:14.393 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:39:44.461 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:39:44.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:40:14.542 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:40:14.546 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:40:44.656 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:40:44.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:41:14.785 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:41:14.788 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:41:44.942 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:41:44.946 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:42:15.014 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:42:15.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:42:45.109 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:42:45.112 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:43:15.240 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:43:15.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:43:45.377 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:43:45.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:44:15.516 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:44:15.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:44:45.620 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:44:45.623 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:45:15.752 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:45:15.755 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:45:45.874 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:45:45.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:46:16.030 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:46:16.034 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:46:46.164 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:46:46.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:47:16.281 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:47:16.284 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:47:46.437 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:47:46.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:48:16.584 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:48:16.588 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:48:46.727 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:48:46.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:49:16.844 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:49:16.847 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:49:47.003 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:49:47.007 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:50:17.150 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:50:17.154 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:50:47.310 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:50:47.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:51:17.474 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:51:17.478 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:51:47.604 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:51:47.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:52:17.904 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:52:17.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:52:48.016 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:52:48.020 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:53:18.138 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:53:18.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:53:48.260 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:53:48.264 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:54:18.419 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:54:18.422 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:54:48.520 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:54:48.524 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:55:18.675 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:55:18.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:55:48.798 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:55:48.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:56:18.903 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:56:18.906 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:56:49.033 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:56:49.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:57:19.147 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:57:19.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:57:49.318 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:57:49.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:58:19.430 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:58:19.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:58:49.570 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:58:49.573 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:59:19.682 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:59:19.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:59:49.829 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T12:59:49.833 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:00:19.995 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:00:19.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:00:50.165 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:00:50.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:01:20.303 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:01:20.306 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:01:50.420 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:01:50.424 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:02:20.566 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:02:20.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:02:50.701 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:02:50.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:03:20.844 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:03:20.847 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:03:50.992 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:03:50.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:04:21.092 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:04:21.095 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:04:51.239 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:04:51.242 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:05:21.407 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:05:21.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:05:51.563 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:05:51.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:06:21.809 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:06:21.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:06:51.949 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:06:51.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:07:22.103 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:07:22.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:07:52.250 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:07:52.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:08:22.422 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:08:22.426 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:08:52.588 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:08:52.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:09:22.653 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:09:22.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:09:52.774 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:09:52.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:10:22.913 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:10:22.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:10:53.052 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:10:53.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:11:23.214 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:11:23.217 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:11:53.345 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:11:53.351 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:12:23.483 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:12:23.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:12:53.648 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:12:53.651 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:13:23.776 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:13:23.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:13:53.924 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:13:53.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:14:24.063 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:14:24.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:14:54.209 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:14:54.213 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:15:24.339 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:15:24.342 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:15:54.488 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:15:54.491 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:16:24.627 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:16:24.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:16:54.757 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:16:54.760 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:17:24.916 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:17:24.919 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:17:55.062 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:17:55.066 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:18:25.231 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:18:25.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:18:55.361 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:18:55.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:19:25.517 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:19:25.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:19:55.609 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:19:55.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:20:25.727 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:20:25.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:20:55.843 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:20:55.847 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:21:25.990 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:21:25.993 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:21:56.096 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:21:56.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:22:26.258 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:22:26.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:22:56.337 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:22:56.341 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:23:26.486 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:23:26.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:23:56.616 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:23:56.620 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:24:26.802 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:24:26.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:24:56.910 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:24:56.915 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:25:27.048 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:25:27.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:25:57.193 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:25:57.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:26:27.334 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:26:27.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:26:57.473 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:26:57.476 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:27:27.616 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:27:27.620 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:27:57.741 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:27:57.744 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:28:15.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:15.527 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:28:15.554 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_34_00_UTC 2024-04-06T13:28:15.554 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_35_00_UTC 2024-04-06T13:28:15.554 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_36_00_UTC 2024-04-06T13:28:15.554 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_37_00_UTC 2024-04-06T13:28:15.554 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_38_00_UTC 2024-04-06T13:28:15.554 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_39_00_UTC 2024-04-06T13:28:15.554 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_40_00_UTC 2024-04-06T13:28:15.554 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_41_00_UTC 2024-04-06T13:28:15.554 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_42_00_UTC 2024-04-06T13:28:15.554 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_43_00_UTC 2024-04-06T13:28:15.554 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_44_00_UTC 2024-04-06T13:28:15.555 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_45_00_UTC 2024-04-06T13:28:15.555 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_46_00_UTC 2024-04-06T13:28:15.555 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_47_00_UTC 2024-04-06T13:28:15.555 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_48_00_UTC 2024-04-06T13:28:15.555 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_49_00_UTC 2024-04-06T13:28:15.555 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_50_00_UTC 2024-04-06T13:28:15.555 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_51_00_UTC 2024-04-06T13:28:15.555 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_52_00_UTC 2024-04-06T13:28:15.555 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_53_00_UTC 2024-04-06T13:28:15.555 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_54_00_UTC 2024-04-06T13:28:15.555 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_55_00_UTC 2024-04-06T13:28:15.555 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_56_00_UTC 2024-04-06T13:28:15.555 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_57_00_UTC 2024-04-06T13:28:15.555 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_58_00_UTC 2024-04-06T13:28:15.556 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_59_00_UTC 2024-04-06T13:28:15.556 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_00_00_UTC 2024-04-06T13:28:15.556 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_01_00_UTC 2024-04-06T13:28:15.556 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_02_00_UTC 2024-04-06T13:28:15.556 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_03_00_UTC 2024-04-06T13:28:15.556 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_04_00_UTC 2024-04-06T13:28:15.556 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_05_00_UTC 2024-04-06T13:28:15.556 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_06_00_UTC 2024-04-06T13:28:15.556 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_07_00_UTC 2024-04-06T13:28:15.556 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_08_00_UTC 2024-04-06T13:28:15.556 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_09_00_UTC 2024-04-06T13:28:15.556 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_10_00_UTC 2024-04-06T13:28:15.556 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_11_00_UTC 2024-04-06T13:28:15.556 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_12_00_UTC 2024-04-06T13:28:15.557 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_13_00_UTC 2024-04-06T13:28:15.557 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_14_00_UTC 2024-04-06T13:28:15.557 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_15_00_UTC 2024-04-06T13:28:15.557 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_16_00_UTC 2024-04-06T13:28:15.557 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_17_00_UTC 2024-04-06T13:28:15.557 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_18_00_UTC 2024-04-06T13:28:15.557 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_19_00_UTC 2024-04-06T13:28:15.557 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_20_00_UTC 2024-04-06T13:28:15.557 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_21_00_UTC 2024-04-06T13:28:15.557 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_22_00_UTC 2024-04-06T13:28:15.557 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_23_00_UTC 2024-04-06T13:28:15.557 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_24_00_UTC 2024-04-06T13:28:15.557 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_25_00_UTC 2024-04-06T13:28:15.557 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_26_00_UTC 2024-04-06T13:28:15.557 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_27_00_UTC 2024-04-06T13:28:15.558 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_28_00_UTC 2024-04-06T13:28:15.558 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-06-12_34_00_UTC', 'scheduled-2024-04-06-12_35_00_UTC', 'scheduled-2024-04-06-12_36_00_UTC', 'scheduled-2024-04-06-12_37_00_UTC', 'scheduled-2024-04-06-12_38_00_UTC', 'scheduled-2024-04-06-12_39_00_UTC', 'scheduled-2024-04-06-12_40_00_UTC', 'scheduled-2024-04-06-12_41_00_UTC', 'scheduled-2024-04-06-12_42_00_UTC', 'scheduled-2024-04-06-12_43_00_UTC', 'scheduled-2024-04-06-12_44_00_UTC', 'scheduled-2024-04-06-12_45_00_UTC', 'scheduled-2024-04-06-12_46_00_UTC', 'scheduled-2024-04-06-12_47_00_UTC', 'scheduled-2024-04-06-12_48_00_UTC', 'scheduled-2024-04-06-12_49_00_UTC', 'scheduled-2024-04-06-12_50_00_UTC', 'scheduled-2024-04-06-12_51_00_UTC', 'scheduled-2024-04-06-12_52_00_UTC', 'scheduled-2024-04-06-12_53_00_UTC', 'scheduled-2024-04-06-12_54_00_UTC', 'scheduled-2024-04-06-12_55_00_UTC', 'scheduled-2024-04-06-12_56_00_UTC', 'scheduled-2024-04-06-12_57_00_UTC', 'scheduled-2024-04-06-12_58_00_UTC', 'scheduled-2024-04-06-12_59_00_UTC', 'scheduled-2024-04-06-13_00_00_UTC', 'scheduled-2024-04-06-13_01_00_UTC', 'scheduled-2024-04-06-13_02_00_UTC', 'scheduled-2024-04-06-13_03_00_UTC', 'scheduled-2024-04-06-13_04_00_UTC', 'scheduled-2024-04-06-13_05_00_UTC', 'scheduled-2024-04-06-13_06_00_UTC', 'scheduled-2024-04-06-13_07_00_UTC', 'scheduled-2024-04-06-13_08_00_UTC', 'scheduled-2024-04-06-13_09_00_UTC', 'scheduled-2024-04-06-13_10_00_UTC', 'scheduled-2024-04-06-13_11_00_UTC', 'scheduled-2024-04-06-13_12_00_UTC', 'scheduled-2024-04-06-13_13_00_UTC', 'scheduled-2024-04-06-13_14_00_UTC', 'scheduled-2024-04-06-13_15_00_UTC', 'scheduled-2024-04-06-13_16_00_UTC', 'scheduled-2024-04-06-13_17_00_UTC', 'scheduled-2024-04-06-13_18_00_UTC', 'scheduled-2024-04-06-13_19_00_UTC', 'scheduled-2024-04-06-13_20_00_UTC', 'scheduled-2024-04-06-13_21_00_UTC', 'scheduled-2024-04-06-13_22_00_UTC', 'scheduled-2024-04-06-13_23_00_UTC', 'scheduled-2024-04-06-13_24_00_UTC', 'scheduled-2024-04-06-13_25_00_UTC', 'scheduled-2024-04-06-13_26_00_UTC', 'scheduled-2024-04-06-13_27_00_UTC', 'scheduled-2024-04-06-13_28_00_UTC'] 2024-04-06T13:28:15.558 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1 2024-04-06T13:28:15.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.726+0000 7fe1b9252700 1 Processor -- start 2024-04-06T13:28:15.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.726+0000 7fe1b9252700 1 -- start start 2024-04-06T13:28:15.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.726+0000 7fe1b9252700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe1b415e8a0 0x7fe1b40cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:15.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.726+0000 7fe1b9252700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe1b415eee0 0x7fe1b415f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:15.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.726+0000 7fe1b9252700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b415fcd0 con 0x7fe1b415f8a0 2024-04-06T13:28:15.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.726+0000 7fe1b9252700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1b40669d0 con 0x7fe1b415eee0 2024-04-06T13:28:15.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.726+0000 7fe1b9252700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1b406ede0 con 0x7fe1b415e8a0 2024-04-06T13:28:15.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.727+0000 7fe1b259c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe1b415eee0 0x7fe1b415f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:15.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.727+0000 7fe1b259c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe1b415eee0 0x7fe1b415f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42456/0 (socket says 172.21.15.189:42456) 2024-04-06T13:28:15.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.727+0000 7fe1b259c700 1 -- 172.21.15.189:0/3896531724 learned_addr learned my addr 172.21.15.189:0/3896531724 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:15.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.727+0000 7fe1b359e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe1b415f8a0 0x7fe1b406ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46418/0 (socket says 172.21.15.189:46418) 2024-04-06T13:28:15.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.727+0000 7fe1b2d9d700 1 --2- 172.21.15.189:0/3896531724 >> v2:172.21.15.192:3300/0 conn(0x7fe1b415e8a0 0x7fe1b40cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:15.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.727+0000 7fe1b259c700 1 -- 172.21.15.189:0/3896531724 >> v2:172.21.15.192:3300/0 conn(0x7fe1b415e8a0 msgr2=0x7fe1b40cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:15.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.727+0000 7fe1b259c700 1 --2- 172.21.15.189:0/3896531724 >> v2:172.21.15.192:3300/0 conn(0x7fe1b415e8a0 0x7fe1b40cbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:15.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.727+0000 7fe1b259c700 1 -- 172.21.15.189:0/3896531724 >> v1:172.21.15.189:6789/0 conn(0x7fe1b415f8a0 legacy=0x7fe1b406ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:15.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.727+0000 7fe1b259c700 1 -- 172.21.15.189:0/3896531724 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b415ec80 con 0x7fe1b415eee0 2024-04-06T13:28:15.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.727+0000 7fe1b259c700 1 --2- 172.21.15.189:0/3896531724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1b415eee0 0x7fe1b415f360 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fe1b40cf650 tx=0x7fe1a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbbd020a7cb1cae4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:15.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.727+0000 7fe1b1d9b700 1 -- 172.21.15.189:0/3896531724 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a400b870 con 0x7fe1b415eee0 2024-04-06T13:28:15.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.727+0000 7fe1b1d9b700 1 -- 172.21.15.189:0/3896531724 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a400b9d0 con 0x7fe1b415eee0 2024-04-06T13:28:15.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.727+0000 7fe1b1d9b700 1 -- 172.21.15.189:0/3896531724 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a400bd60 con 0x7fe1b415eee0 2024-04-06T13:28:15.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.728+0000 7fe1b9252700 1 -- 172.21.15.189:0/3896531724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1b415eee0 msgr2=0x7fe1b415f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:15.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.728+0000 7fe1b9252700 1 --2- 172.21.15.189:0/3896531724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1b415eee0 0x7fe1b415f360 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fe1b40cf650 tx=0x7fe1a4009bd0 comp rx=0 tx=0).stop 2024-04-06T13:28:15.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.728+0000 7fe1b9252700 1 -- 172.21.15.189:0/3896531724 shutdown_connections 2024-04-06T13:28:15.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.728+0000 7fe1b9252700 1 --2- 172.21.15.189:0/3896531724 >> v2:172.21.15.192:3300/0 conn(0x7fe1b415e8a0 0x7fe1b40cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:15.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.728+0000 7fe1b9252700 1 --2- 172.21.15.189:0/3896531724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1b415eee0 0x7fe1b415f360 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:15.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.728+0000 7fe1b9252700 1 -- 172.21.15.189:0/3896531724 >> 172.21.15.189:0/3896531724 conn(0x7fe1b40cc290 msgr2=0x7fe1b40cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:15.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.728+0000 7fe1b9252700 1 -- 172.21.15.189:0/3896531724 shutdown_connections 2024-04-06T13:28:15.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.728+0000 7fe1b9252700 1 -- 172.21.15.189:0/3896531724 wait complete. 2024-04-06T13:28:15.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.729+0000 7fe1b9252700 1 Processor -- start 2024-04-06T13:28:15.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.729+0000 7fe1b9252700 1 -- start start 2024-04-06T13:28:15.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.730+0000 7fe1b9252700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1b415e8a0 0x7fe1b41f2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:15.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.730+0000 7fe1b9252700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1b415eee0 0x7fe1b40c32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:15.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.730+0000 7fe1b9252700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1b415f8a0 0x7fe1b40b4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:15.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.730+0000 7fe1b9252700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b4150790 con 0x7fe1b415eee0 2024-04-06T13:28:15.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.730+0000 7fe1b9252700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b40c3820 con 0x7fe1b415e8a0 2024-04-06T13:28:15.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.730+0000 7fe1b9252700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b40ce570 con 0x7fe1b415f8a0 2024-04-06T13:28:15.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.730+0000 7fe1b259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1b415eee0 0x7fe1b40c32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:15.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.730+0000 7fe1b259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1b415eee0 0x7fe1b40c32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42464/0 (socket says 172.21.15.189:42464) 2024-04-06T13:28:15.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.730+0000 7fe1b259c700 1 -- 172.21.15.189:0/477936136 learned_addr learned my addr 172.21.15.189:0/477936136 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:15.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.731+0000 7fe1b2d9d700 1 --2- 172.21.15.189:0/477936136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1b415e8a0 0x7fe1b41f2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:15.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.731+0000 7fe1b259c700 1 -- 172.21.15.189:0/477936136 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1b415f8a0 msgr2=0x7fe1b40b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:15.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.731+0000 7fe1b359e700 1 --2- 172.21.15.189:0/477936136 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1b415f8a0 0x7fe1b40b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:15.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.731+0000 7fe1b259c700 1 --2- 172.21.15.189:0/477936136 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1b415f8a0 0x7fe1b40b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:15.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.731+0000 7fe1b259c700 1 -- 172.21.15.189:0/477936136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1b415e8a0 msgr2=0x7fe1b41f2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:15.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.731+0000 7fe1b259c700 1 --2- 172.21.15.189:0/477936136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1b415e8a0 0x7fe1b41f2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:15.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.731+0000 7fe1b259c700 1 -- 172.21.15.189:0/477936136 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a4009550 con 0x7fe1b415eee0 2024-04-06T13:28:15.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.731+0000 7fe1b2d9d700 1 --2- 172.21.15.189:0/477936136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1b415e8a0 0x7fe1b41f2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:28:15.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.731+0000 7fe1b259c700 1 --2- 172.21.15.189:0/477936136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1b415eee0 0x7fe1b40c32e0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4073430 tx=0x7fe1a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:15.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.731+0000 7fe19b7fe700 1 -- 172.21.15.189:0/477936136 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a4004540 con 0x7fe1b415eee0 2024-04-06T13:28:15.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.731+0000 7fe19b7fe700 1 -- 172.21.15.189:0/477936136 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a40036a0 con 0x7fe1b415eee0 2024-04-06T13:28:15.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.731+0000 7fe1b9252700 1 -- 172.21.15.189:0/477936136 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b40cf3e0 con 0x7fe1b415eee0 2024-04-06T13:28:15.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.731+0000 7fe19b7fe700 1 -- 172.21.15.189:0/477936136 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a4003a30 con 0x7fe1b415eee0 2024-04-06T13:28:15.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.731+0000 7fe1b9252700 1 -- 172.21.15.189:0/477936136 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b40cf100 con 0x7fe1b415eee0 2024-04-06T13:28:15.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.732+0000 7fe1b9252700 1 -- 172.21.15.189:0/477936136 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b4072140 con 0x7fe1b415eee0 2024-04-06T13:28:15.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.732+0000 7fe19b7fe700 1 -- 172.21.15.189:0/477936136 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 94) v1 ==== 89276+0+0 (secure 0 0 0) 0x7fe1a4021070 con 0x7fe1b415eee0 2024-04-06T13:28:15.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.733+0000 7fe19b7fe700 1 --2- 172.21.15.189:0/477936136 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe1a006ae00 0x7fe1a006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:15.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.733+0000 7fe19b7fe700 1 -- 172.21.15.189:0/477936136 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fe1a408b090 con 0x7fe1b415eee0 2024-04-06T13:28:15.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.733+0000 7fe1b2d9d700 1 --2- 172.21.15.189:0/477936136 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe1a006ae00 0x7fe1a006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:15.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.733+0000 7fe1b2d9d700 1 --2- 172.21.15.189:0/477936136 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe1a006ae00 0x7fe1a006d2c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8009a00 tx=0x7fe1a8008040 comp rx=0 tx=0).ready entity=mgr.9303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:15.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.738+0000 7fe19b7fe700 1 -- 172.21.15.189:0/477936136 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe1a4055fa0 con 0x7fe1b415eee0 2024-04-06T13:28:15.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.957+0000 7fe1b9252700 1 -- 172.21.15.189:0/477936136 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe1b40c4c20 con 0x7fe1a006ae00 2024-04-06T13:28:15.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.965+0000 7fe19b7fe700 1 -- 172.21.15.189:0/477936136 <== mgr.9303 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+373 (secure 0 0 0) 0x7fe1b40c4c20 con 0x7fe1a006ae00 2024-04-06T13:28:15.965 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:15.966 INFO:teuthology.orchestra.run.smithi189.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:32:17", "first": "2024-04-06T12:33:00", "last": "2024-04-06T13:28:00", "last_pruned": "2024-04-06T13:28:00", "created_count": 56, "pruned_count": 1, "active": true}] 2024-04-06T13:28:15.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.968+0000 7fe1b9252700 1 -- 172.21.15.189:0/477936136 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe1a006ae00 msgr2=0x7fe1a006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:15.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.968+0000 7fe1b9252700 1 --2- 172.21.15.189:0/477936136 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe1a006ae00 0x7fe1a006d2c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8009a00 tx=0x7fe1a8008040 comp rx=0 tx=0).stop 2024-04-06T13:28:15.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.968+0000 7fe1b9252700 1 -- 172.21.15.189:0/477936136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1b415eee0 msgr2=0x7fe1b40c32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:15.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.968+0000 7fe1b9252700 1 --2- 172.21.15.189:0/477936136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1b415eee0 0x7fe1b40c32e0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4073430 tx=0x7fe1a4004750 comp rx=0 tx=0).stop 2024-04-06T13:28:15.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.968+0000 7fe1b9252700 1 -- 172.21.15.189:0/477936136 shutdown_connections 2024-04-06T13:28:15.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.968+0000 7fe1b9252700 1 --2- 172.21.15.189:0/477936136 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1b415f8a0 0x7fe1b40b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:15.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.968+0000 7fe1b9252700 1 --2- 172.21.15.189:0/477936136 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe1a006ae00 0x7fe1a006d2c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:15.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.968+0000 7fe1b9252700 1 --2- 172.21.15.189:0/477936136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1b415e8a0 0x7fe1b41f2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:15.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.968+0000 7fe1b9252700 1 --2- 172.21.15.189:0/477936136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1b415eee0 0x7fe1b40c32e0 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:15.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.968+0000 7fe1b9252700 1 -- 172.21.15.189:0/477936136 >> 172.21.15.189:0/477936136 conn(0x7fe1b40cc290 msgr2=0x7fe1b40aeab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:15.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.968+0000 7fe1b9252700 1 -- 172.21.15.189:0/477936136 shutdown_connections 2024-04-06T13:28:15.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:15.968+0000 7fe1b9252700 1 -- 172.21.15.189:0/477936136 wait complete. 2024-04-06T13:28:15.980 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T12:32:17", "first": "2024-04-06T12:33:00", "last": "2024-04-06T13:28:00", "last_pruned": "2024-04-06T13:28:00", "created_count": 56, "pruned_count": 1, "active": true}] 2024-04-06T13:28:15.980 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1', 'rel_path': '/snap_test_dir1', 'schedule': '1m', 'retention': {'n': 55}, 'start': '2024-04-06T00:00:00', 'created': '2024-04-06T12:32:17', 'first': '2024-04-06T12:33:00', 'last': '2024-04-06T13:28:00', 'last_pruned': '2024-04-06T13:28:00', 'created_count': 56, 'pruned_count': 1, 'active': True} 2024-04-06T13:28:15.980 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 55 2024-04-06T13:28:15.980 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 56 2024-04-06T13:28:15.980 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1 2024-04-06T13:28:16.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.104+0000 7f649c326700 1 Processor -- start 2024-04-06T13:28:16.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.104+0000 7f649c326700 1 -- start start 2024-04-06T13:28:16.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649c326700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f64940d11b0 0x7f64940d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:16.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649c326700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f64940cfa00 0x7f64940cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:16.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649c326700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64940d1ad0 con 0x7f64940cb870 2024-04-06T13:28:16.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649c326700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64940ccdc0 con 0x7f64940d11b0 2024-04-06T13:28:16.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649c326700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64940d0b50 con 0x7f64940cfa00 2024-04-06T13:28:16.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649a0c2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f64940d11b0 0x7f64940d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:16.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649a0c2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f64940d11b0 0x7f64940d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42470/0 (socket says 172.21.15.189:42470) 2024-04-06T13:28:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649a0c2700 1 -- 172.21.15.189:0/2599658458 learned_addr learned my addr 172.21.15.189:0/2599658458 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649a8c3700 1 --2- 172.21.15.189:0/2599658458 >> v2:172.21.15.192:3300/0 conn(0x7f64940cfa00 0x7f64940cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649a0c2700 1 -- 172.21.15.189:0/2599658458 >> v2:172.21.15.192:3300/0 conn(0x7f64940cfa00 msgr2=0x7f64940cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649a0c2700 1 --2- 172.21.15.189:0/2599658458 >> v2:172.21.15.192:3300/0 conn(0x7f64940cfa00 0x7f64940cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649a0c2700 1 -- 172.21.15.189:0/2599658458 >> v1:172.21.15.192:6789/0 conn(0x7f64940cb870 legacy=0x7f649415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649a0c2700 1 -- 172.21.15.189:0/2599658458 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64940cbd80 con 0x7f64940d11b0 2024-04-06T13:28:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649a0c2700 1 --2- 172.21.15.189:0/2599658458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64940d11b0 0x7f64940d1590 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f64940b0cd0 tx=0x7f6484009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e76fa241064fe952 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f64990c0700 1 -- 172.21.15.189:0/2599658458 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648400b870 con 0x7f64940d11b0 2024-04-06T13:28:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f64990c0700 1 -- 172.21.15.189:0/2599658458 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648400b9d0 con 0x7f64940d11b0 2024-04-06T13:28:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f64990c0700 1 -- 172.21.15.189:0/2599658458 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648400bd60 con 0x7f64940d11b0 2024-04-06T13:28:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649c326700 1 -- 172.21.15.189:0/2599658458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64940d11b0 msgr2=0x7f64940d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649c326700 1 --2- 172.21.15.189:0/2599658458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64940d11b0 0x7f64940d1590 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f64940b0cd0 tx=0x7f6484009bd0 comp rx=0 tx=0).stop 2024-04-06T13:28:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649c326700 1 -- 172.21.15.189:0/2599658458 shutdown_connections 2024-04-06T13:28:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649c326700 1 --2- 172.21.15.189:0/2599658458 >> v2:172.21.15.192:3300/0 conn(0x7f64940cfa00 0x7f64940cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649c326700 1 --2- 172.21.15.189:0/2599658458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64940d11b0 0x7f64940d1590 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649c326700 1 -- 172.21.15.189:0/2599658458 >> 172.21.15.189:0/2599658458 conn(0x7f64940b2db0 msgr2=0x7f64940b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649c326700 1 -- 172.21.15.189:0/2599658458 shutdown_connections 2024-04-06T13:28:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.105+0000 7f649c326700 1 -- 172.21.15.189:0/2599658458 wait complete. 2024-04-06T13:28:16.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.107+0000 7f649c326700 1 Processor -- start 2024-04-06T13:28:16.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.107+0000 7f649c326700 1 -- start start 2024-04-06T13:28:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f649c326700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64940cb870 0x7f6494155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f649c326700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64940cfa00 0x7f64941589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f649c326700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64940d11b0 0x7f64941f1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f649c326700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6494158f30 con 0x7f64940cb870 2024-04-06T13:28:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f649c326700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649415e0d0 con 0x7f64940d11b0 2024-04-06T13:28:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f649c326700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64940b0650 con 0x7f64940cfa00 2024-04-06T13:28:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f649a0c2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64940cb870 0x7f6494155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f649a0c2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64940cb870 0x7f6494155750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:42478/0 (socket says 172.21.15.189:42478) 2024-04-06T13:28:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f649a0c2700 1 -- 172.21.15.189:0/3049541509 learned_addr learned my addr 172.21.15.189:0/3049541509 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f649a8c3700 1 --2- 172.21.15.189:0/3049541509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64940d11b0 0x7f64941f1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f64998c1700 1 --2- 172.21.15.189:0/3049541509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64940cfa00 0x7f64941589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f649a0c2700 1 -- 172.21.15.189:0/3049541509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64940cfa00 msgr2=0x7f64941589f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f649a0c2700 1 --2- 172.21.15.189:0/3049541509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64940cfa00 0x7f64941589f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f649a0c2700 1 -- 172.21.15.189:0/3049541509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64940d11b0 msgr2=0x7f64941f1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f649a0c2700 1 --2- 172.21.15.189:0/3049541509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64940d11b0 0x7f64941f1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f649a0c2700 1 -- 172.21.15.189:0/3049541509 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6484009550 con 0x7f64940cb870 2024-04-06T13:28:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f649a0c2700 1 --2- 172.21.15.189:0/3049541509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64940cb870 0x7f6494155750 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f64940677e0 tx=0x7f6484004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f648affd700 1 -- 172.21.15.189:0/3049541509 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6484004310 con 0x7f64940cb870 2024-04-06T13:28:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f648affd700 1 -- 172.21.15.189:0/3049541509 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648401f070 con 0x7f64940cb870 2024-04-06T13:28:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f649c326700 1 -- 172.21.15.189:0/3049541509 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64941f1cb0 con 0x7f64940cb870 2024-04-06T13:28:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.108+0000 7f648affd700 1 -- 172.21.15.189:0/3049541509 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64840036a0 con 0x7f64940cb870 2024-04-06T13:28:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.109+0000 7f649c326700 1 -- 172.21.15.189:0/3049541509 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6494157c30 con 0x7f64940cb870 2024-04-06T13:28:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.109+0000 7f649c326700 1 -- 172.21.15.189:0/3049541509 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6494074990 con 0x7f64940cb870 2024-04-06T13:28:16.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.110+0000 7f648affd700 1 -- 172.21.15.189:0/3049541509 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 94) v1 ==== 89276+0+0 (secure 0 0 0) 0x7f6484021030 con 0x7f64940cb870 2024-04-06T13:28:16.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.110+0000 7f648affd700 1 --2- 172.21.15.189:0/3049541509 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f647806ae00 0x7f647806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:16.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.111+0000 7f648affd700 1 -- 172.21.15.189:0/3049541509 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f648408b190 con 0x7f64940cb870 2024-04-06T13:28:16.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.111+0000 7f64998c1700 1 --2- 172.21.15.189:0/3049541509 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f647806ae00 0x7f647806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:16.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.111+0000 7f64998c1700 1 --2- 172.21.15.189:0/3049541509 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f647806ae00 0x7f647806d2c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f64940b6eb0 tx=0x7f6490009040 comp rx=0 tx=0).ready entity=mgr.9303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:16.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.115+0000 7f648affd700 1 -- 172.21.15.189:0/3049541509 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6484056200 con 0x7f64940cb870 2024-04-06T13:28:16.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.336+0000 7f649c326700 1 -- 172.21.15.189:0/3049541509 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f64940cc060 con 0x7f647806ae00 2024-04-06T13:28:16.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.411+0000 7f648affd700 1 -- 172.21.15.189:0/3049541509 <== mgr.9303 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+42 (secure 0 0 0) 0x7f64900050d0 con 0x7f647806ae00 2024-04-06T13:28:16.412 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule removed for path /snap_test_dir1 2024-04-06T13:28:16.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.413+0000 7f649c326700 1 -- 172.21.15.189:0/3049541509 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f647806ae00 msgr2=0x7f647806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:16.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.413+0000 7f649c326700 1 --2- 172.21.15.189:0/3049541509 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f647806ae00 0x7f647806d2c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f64940b6eb0 tx=0x7f6490009040 comp rx=0 tx=0).stop 2024-04-06T13:28:16.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.415+0000 7f649c326700 1 -- 172.21.15.189:0/3049541509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64940cb870 msgr2=0x7f6494155750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:16.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.415+0000 7f649c326700 1 --2- 172.21.15.189:0/3049541509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64940cb870 0x7f6494155750 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f64940677e0 tx=0x7f6484004750 comp rx=0 tx=0).stop 2024-04-06T13:28:16.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.415+0000 7f649c326700 1 -- 172.21.15.189:0/3049541509 shutdown_connections 2024-04-06T13:28:16.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.415+0000 7f649c326700 1 --2- 172.21.15.189:0/3049541509 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f647806ae00 0x7f647806d2c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:16.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.415+0000 7f649c326700 1 --2- 172.21.15.189:0/3049541509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64940d11b0 0x7f64941f1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:16.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.415+0000 7f649c326700 1 --2- 172.21.15.189:0/3049541509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64940cb870 0x7f6494155750 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:16.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.415+0000 7f649c326700 1 --2- 172.21.15.189:0/3049541509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64940cfa00 0x7f64941589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:16.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.415+0000 7f649c326700 1 -- 172.21.15.189:0/3049541509 >> 172.21.15.189:0/3049541509 conn(0x7f64940b2db0 msgr2=0x7f64940b0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:16.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.415+0000 7f649c326700 1 -- 172.21.15.189:0/3049541509 shutdown_connections 2024-04-06T13:28:16.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:16.415+0000 7f649c326700 1 -- 172.21.15.189:0/3049541509 wait complete. 2024-04-06T13:28:16.426 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1 2024-04-06T13:28:16.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:16.427 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:28:16.446 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_34_00_UTC 2024-04-06T13:28:16.447 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_35_00_UTC 2024-04-06T13:28:16.447 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_36_00_UTC 2024-04-06T13:28:16.447 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_37_00_UTC 2024-04-06T13:28:16.447 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_38_00_UTC 2024-04-06T13:28:16.447 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_39_00_UTC 2024-04-06T13:28:16.447 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_40_00_UTC 2024-04-06T13:28:16.447 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_41_00_UTC 2024-04-06T13:28:16.447 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_42_00_UTC 2024-04-06T13:28:16.447 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_43_00_UTC 2024-04-06T13:28:16.447 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_44_00_UTC 2024-04-06T13:28:16.447 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_45_00_UTC 2024-04-06T13:28:16.447 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_46_00_UTC 2024-04-06T13:28:16.447 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_47_00_UTC 2024-04-06T13:28:16.447 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_48_00_UTC 2024-04-06T13:28:16.447 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_49_00_UTC 2024-04-06T13:28:16.448 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_50_00_UTC 2024-04-06T13:28:16.448 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_51_00_UTC 2024-04-06T13:28:16.448 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_52_00_UTC 2024-04-06T13:28:16.448 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_53_00_UTC 2024-04-06T13:28:16.448 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_54_00_UTC 2024-04-06T13:28:16.448 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_55_00_UTC 2024-04-06T13:28:16.448 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_56_00_UTC 2024-04-06T13:28:16.448 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_57_00_UTC 2024-04-06T13:28:16.448 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_58_00_UTC 2024-04-06T13:28:16.448 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-12_59_00_UTC 2024-04-06T13:28:16.448 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_00_00_UTC 2024-04-06T13:28:16.448 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_01_00_UTC 2024-04-06T13:28:16.448 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_02_00_UTC 2024-04-06T13:28:16.448 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_03_00_UTC 2024-04-06T13:28:16.448 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_04_00_UTC 2024-04-06T13:28:16.449 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_05_00_UTC 2024-04-06T13:28:16.449 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_06_00_UTC 2024-04-06T13:28:16.449 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_07_00_UTC 2024-04-06T13:28:16.449 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_08_00_UTC 2024-04-06T13:28:16.449 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_09_00_UTC 2024-04-06T13:28:16.449 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_10_00_UTC 2024-04-06T13:28:16.449 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_11_00_UTC 2024-04-06T13:28:16.449 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_12_00_UTC 2024-04-06T13:28:16.449 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_13_00_UTC 2024-04-06T13:28:16.449 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_14_00_UTC 2024-04-06T13:28:16.449 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_15_00_UTC 2024-04-06T13:28:16.449 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_16_00_UTC 2024-04-06T13:28:16.449 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_17_00_UTC 2024-04-06T13:28:16.449 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_18_00_UTC 2024-04-06T13:28:16.449 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_19_00_UTC 2024-04-06T13:28:16.449 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_20_00_UTC 2024-04-06T13:28:16.450 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_21_00_UTC 2024-04-06T13:28:16.450 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_22_00_UTC 2024-04-06T13:28:16.450 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_23_00_UTC 2024-04-06T13:28:16.450 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_24_00_UTC 2024-04-06T13:28:16.450 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_25_00_UTC 2024-04-06T13:28:16.450 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_26_00_UTC 2024-04-06T13:28:16.450 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_27_00_UTC 2024-04-06T13:28:16.450 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_28_00_UTC 2024-04-06T13:28:16.450 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_34_00_UTC 2024-04-06T13:28:16.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:16.450 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_34_00_UTC) 2024-04-06T13:28:16.511 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_35_00_UTC 2024-04-06T13:28:16.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:16.511 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_35_00_UTC) 2024-04-06T13:28:16.575 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_36_00_UTC 2024-04-06T13:28:16.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:16.576 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_36_00_UTC) 2024-04-06T13:28:16.638 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_37_00_UTC 2024-04-06T13:28:16.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:16.639 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_37_00_UTC) 2024-04-06T13:28:16.702 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_38_00_UTC 2024-04-06T13:28:16.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:16.703 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_38_00_UTC) 2024-04-06T13:28:16.765 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_39_00_UTC 2024-04-06T13:28:16.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:16.765 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_39_00_UTC) 2024-04-06T13:28:16.827 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_40_00_UTC 2024-04-06T13:28:16.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:16.827 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_40_00_UTC) 2024-04-06T13:28:16.890 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_41_00_UTC 2024-04-06T13:28:16.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:16.890 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_41_00_UTC) 2024-04-06T13:28:16.953 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_42_00_UTC 2024-04-06T13:28:16.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:16.953 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_42_00_UTC) 2024-04-06T13:28:17.016 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_43_00_UTC 2024-04-06T13:28:17.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:17.017 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_43_00_UTC) 2024-04-06T13:28:17.079 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_44_00_UTC 2024-04-06T13:28:17.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:17.079 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_44_00_UTC) 2024-04-06T13:28:17.143 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_45_00_UTC 2024-04-06T13:28:17.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:17.143 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_45_00_UTC) 2024-04-06T13:28:17.206 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_46_00_UTC 2024-04-06T13:28:17.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:17.206 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_46_00_UTC) 2024-04-06T13:28:17.269 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_47_00_UTC 2024-04-06T13:28:17.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:17.269 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_47_00_UTC) 2024-04-06T13:28:17.338 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_48_00_UTC 2024-04-06T13:28:17.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:17.338 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_48_00_UTC) 2024-04-06T13:28:17.402 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_49_00_UTC 2024-04-06T13:28:17.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:17.402 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_49_00_UTC) 2024-04-06T13:28:17.463 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_50_00_UTC 2024-04-06T13:28:17.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:17.464 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_50_00_UTC) 2024-04-06T13:28:17.526 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_51_00_UTC 2024-04-06T13:28:17.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:17.527 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_51_00_UTC) 2024-04-06T13:28:17.589 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_52_00_UTC 2024-04-06T13:28:17.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:17.590 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_52_00_UTC) 2024-04-06T13:28:17.652 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_53_00_UTC 2024-04-06T13:28:17.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:17.652 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_53_00_UTC) 2024-04-06T13:28:17.714 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_54_00_UTC 2024-04-06T13:28:17.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:17.715 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_54_00_UTC) 2024-04-06T13:28:17.777 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_55_00_UTC 2024-04-06T13:28:17.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:17.777 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_55_00_UTC) 2024-04-06T13:28:17.840 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_56_00_UTC 2024-04-06T13:28:17.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:17.840 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_56_00_UTC) 2024-04-06T13:28:17.903 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_57_00_UTC 2024-04-06T13:28:17.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:17.904 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_57_00_UTC) 2024-04-06T13:28:17.966 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_58_00_UTC 2024-04-06T13:28:17.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:17.967 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_58_00_UTC) 2024-04-06T13:28:18.029 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-12_59_00_UTC 2024-04-06T13:28:18.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:18.029 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-12_59_00_UTC) 2024-04-06T13:28:18.092 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_00_00_UTC 2024-04-06T13:28:18.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:18.092 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_00_00_UTC) 2024-04-06T13:28:18.155 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_01_00_UTC 2024-04-06T13:28:18.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:18.155 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_01_00_UTC) 2024-04-06T13:28:18.217 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_02_00_UTC 2024-04-06T13:28:18.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:18.217 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_02_00_UTC) 2024-04-06T13:28:18.281 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_03_00_UTC 2024-04-06T13:28:18.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:18.281 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_03_00_UTC) 2024-04-06T13:28:18.369 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_04_00_UTC 2024-04-06T13:28:18.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:18.369 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_04_00_UTC) 2024-04-06T13:28:18.396 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_05_00_UTC 2024-04-06T13:28:18.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:18.396 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_05_00_UTC) 2024-04-06T13:28:18.461 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_06_00_UTC 2024-04-06T13:28:18.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:18.461 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_06_00_UTC) 2024-04-06T13:28:18.524 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_07_00_UTC 2024-04-06T13:28:18.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:18.524 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_07_00_UTC) 2024-04-06T13:28:18.586 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_08_00_UTC 2024-04-06T13:28:18.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:18.586 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_08_00_UTC) 2024-04-06T13:28:18.648 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_09_00_UTC 2024-04-06T13:28:18.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:18.649 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_09_00_UTC) 2024-04-06T13:28:18.711 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_10_00_UTC 2024-04-06T13:28:18.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:18.711 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_10_00_UTC) 2024-04-06T13:28:18.774 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_11_00_UTC 2024-04-06T13:28:18.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:18.775 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_11_00_UTC) 2024-04-06T13:28:18.849 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_12_00_UTC 2024-04-06T13:28:18.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:18.850 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_12_00_UTC) 2024-04-06T13:28:18.912 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_13_00_UTC 2024-04-06T13:28:18.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:18.913 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_13_00_UTC) 2024-04-06T13:28:18.975 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_14_00_UTC 2024-04-06T13:28:18.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:18.975 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_14_00_UTC) 2024-04-06T13:28:19.037 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_15_00_UTC 2024-04-06T13:28:19.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:19.037 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_15_00_UTC) 2024-04-06T13:28:19.099 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_16_00_UTC 2024-04-06T13:28:19.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:19.100 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_16_00_UTC) 2024-04-06T13:28:19.162 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_17_00_UTC 2024-04-06T13:28:19.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:19.162 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_17_00_UTC) 2024-04-06T13:28:19.226 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_18_00_UTC 2024-04-06T13:28:19.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:19.226 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_18_00_UTC) 2024-04-06T13:28:19.289 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_19_00_UTC 2024-04-06T13:28:19.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:19.289 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_19_00_UTC) 2024-04-06T13:28:19.361 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_20_00_UTC 2024-04-06T13:28:19.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:19.361 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_20_00_UTC) 2024-04-06T13:28:19.424 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_21_00_UTC 2024-04-06T13:28:19.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:19.425 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_21_00_UTC) 2024-04-06T13:28:19.487 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_22_00_UTC 2024-04-06T13:28:19.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:19.487 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_22_00_UTC) 2024-04-06T13:28:19.549 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_23_00_UTC 2024-04-06T13:28:19.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:19.549 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_23_00_UTC) 2024-04-06T13:28:19.612 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_24_00_UTC 2024-04-06T13:28:19.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:19.612 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_24_00_UTC) 2024-04-06T13:28:19.684 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_25_00_UTC 2024-04-06T13:28:19.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:19.684 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_25_00_UTC) 2024-04-06T13:28:19.746 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_26_00_UTC 2024-04-06T13:28:19.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:19.747 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_26_00_UTC) 2024-04-06T13:28:19.809 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_27_00_UTC 2024-04-06T13:28:19.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:19.810 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_27_00_UTC) 2024-04-06T13:28:19.871 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_28_00_UTC 2024-04-06T13:28:19.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:19.872 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_28_00_UTC) 2024-04-06T13:28:19.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:19.933 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-06T13:28:19.993 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-06T13:28:20.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.156+0000 7fd6f0a27700 1 Processor -- start 2024-04-06T13:28:20.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.156+0000 7fd6f0a27700 1 -- start start 2024-04-06T13:28:20.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.157+0000 7fd6f0a27700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd6ec0d17a0 0x7fd6ec06b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:20.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.157+0000 7fd6f0a27700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ec06d850 con 0x7fd6ec146dd0 2024-04-06T13:28:20.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.157+0000 7fd6f0a27700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ec06c0a0 con 0x7fd6ec0cedf0 2024-04-06T13:28:20.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.157+0000 7fd6f0a27700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6ec06c380 con 0x7fd6ec0d17a0 2024-04-06T13:28:20.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.157+0000 7fd6e9d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd6ec146dd0 0x7fd6ec06d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51404/0 (socket says 172.21.15.189:51404) 2024-04-06T13:28:20.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.157+0000 7fd6e9d9b700 1 -- 172.21.15.189:0/1039630823 learned_addr learned my addr 172.21.15.189:0/1039630823 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:20.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.157+0000 7fd6ea59c700 1 --2- 172.21.15.189:0/1039630823 >> v2:172.21.15.192:3300/0 conn(0x7fd6ec0d17a0 0x7fd6ec06b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:20.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.157+0000 7fd6e959a700 1 -- 172.21.15.189:0/1039630823 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299578410 0 0) 0x7fd6ec06d850 con 0x7fd6ec146dd0 2024-04-06T13:28:20.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.157+0000 7fd6e959a700 1 -- 172.21.15.189:0/1039630823 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6d40035b0 con 0x7fd6ec146dd0 2024-04-06T13:28:20.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.157+0000 7fd6e959a700 1 -- 172.21.15.189:0/1039630823 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081167252 0 0) 0x7fd6ec06c0a0 con 0x7fd6ec0cedf0 2024-04-06T13:28:20.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.158+0000 7fd6e959a700 1 -- 172.21.15.189:0/1039630823 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6ec06d850 con 0x7fd6ec0cedf0 2024-04-06T13:28:20.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.158+0000 7fd6ea59c700 1 -- 172.21.15.189:0/1039630823 >> v1:172.21.15.192:6789/0 conn(0x7fd6ec0cedf0 legacy=0x7fd6ec0cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:20.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.158+0000 7fd6ea59c700 1 -- 172.21.15.189:0/1039630823 >> v1:172.21.15.189:6789/0 conn(0x7fd6ec146dd0 legacy=0x7fd6ec06d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:20.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.158+0000 7fd6e959a700 1 -- 172.21.15.189:0/1039630823 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1049892745 0 0) 0x7fd6d40035b0 con 0x7fd6ec146dd0 2024-04-06T13:28:20.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.158+0000 7fd6ea59c700 1 -- 172.21.15.189:0/1039630823 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ec06c660 con 0x7fd6ec0d17a0 2024-04-06T13:28:20.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.158+0000 7fd6ea59c700 1 --2- 172.21.15.189:0/1039630823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6ec0d17a0 0x7fd6ec06b7a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009840 tx=0x7fd6e0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d18c84277a54067 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:20.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.158+0000 7fd6e959a700 1 -- 172.21.15.189:0/1039630823 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc001670 con 0x7fd6ec0d17a0 2024-04-06T13:28:20.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.158+0000 7fd6e959a700 1 -- 172.21.15.189:0/1039630823 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e0004030 con 0x7fd6ec0d17a0 2024-04-06T13:28:20.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.158+0000 7fd6e959a700 1 -- 172.21.15.189:0/1039630823 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e00043c0 con 0x7fd6ec0d17a0 2024-04-06T13:28:20.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.158+0000 7fd6f0a27700 1 -- 172.21.15.189:0/1039630823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6ec0d17a0 msgr2=0x7fd6ec06b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:20.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.158+0000 7fd6f0a27700 1 --2- 172.21.15.189:0/1039630823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6ec0d17a0 0x7fd6ec06b7a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009840 tx=0x7fd6e0009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:20.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.158+0000 7fd6f0a27700 1 -- 172.21.15.189:0/1039630823 shutdown_connections 2024-04-06T13:28:20.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.158+0000 7fd6f0a27700 1 --2- 172.21.15.189:0/1039630823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6ec0d17a0 0x7fd6ec06b7a0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:20.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.158+0000 7fd6f0a27700 1 -- 172.21.15.189:0/1039630823 >> 172.21.15.189:0/1039630823 conn(0x7fd6ec0b2db0 msgr2=0x7fd6ec0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:20.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.159+0000 7fd6f0a27700 1 -- 172.21.15.189:0/1039630823 shutdown_connections 2024-04-06T13:28:20.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.159+0000 7fd6f0a27700 1 -- 172.21.15.189:0/1039630823 wait complete. 2024-04-06T13:28:20.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.160+0000 7fd6f0a27700 1 Processor -- start 2024-04-06T13:28:20.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.160+0000 7fd6f0a27700 1 -- start start 2024-04-06T13:28:20.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.161+0000 7fd6f0a27700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6ec0cedf0 0x7fd6ec15eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:20.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.161+0000 7fd6f0a27700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6ec0d17a0 0x7fd6ec15a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:20.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.161+0000 7fd6f0a27700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6ec146dd0 0x7fd6ec0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:20.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.161+0000 7fd6f0a27700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ec15a550 con 0x7fd6ec0cedf0 2024-04-06T13:28:20.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.161+0000 7fd6f0a27700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ec0adbc0 con 0x7fd6ec146dd0 2024-04-06T13:28:20.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.161+0000 7fd6f0a27700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ec0ad320 con 0x7fd6ec0d17a0 2024-04-06T13:28:20.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.161+0000 7fd6ea59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6ec0cedf0 0x7fd6ec15eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:20.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.161+0000 7fd6ea59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6ec0cedf0 0x7fd6ec15eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55532/0 (socket says 172.21.15.189:55532) 2024-04-06T13:28:20.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.161+0000 7fd6ea59c700 1 -- 172.21.15.189:0/3798639192 learned_addr learned my addr 172.21.15.189:0/3798639192 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:20.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.161+0000 7fd6ead9d700 1 --2- 172.21.15.189:0/3798639192 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6ec146dd0 0x7fd6ec0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:20.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.161+0000 7fd6e9d9b700 1 --2- 172.21.15.189:0/3798639192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6ec0d17a0 0x7fd6ec15a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:20.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.161+0000 7fd6ea59c700 1 -- 172.21.15.189:0/3798639192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6ec0d17a0 msgr2=0x7fd6ec15a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:20.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.161+0000 7fd6ea59c700 1 --2- 172.21.15.189:0/3798639192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6ec0d17a0 0x7fd6ec15a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:20.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.161+0000 7fd6ea59c700 1 -- 172.21.15.189:0/3798639192 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6ec146dd0 msgr2=0x7fd6ec0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:20.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.161+0000 7fd6ea59c700 1 --2- 172.21.15.189:0/3798639192 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6ec146dd0 0x7fd6ec0ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:20.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.162+0000 7fd6ea59c700 1 -- 172.21.15.189:0/3798639192 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e00094a0 con 0x7fd6ec0cedf0 2024-04-06T13:28:20.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.162+0000 7fd6ead9d700 1 --2- 172.21.15.189:0/3798639192 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6ec146dd0 0x7fd6ec0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:28:20.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.162+0000 7fd6e9d9b700 1 --2- 172.21.15.189:0/3798639192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6ec0d17a0 0x7fd6ec15a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:28:20.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.162+0000 7fd6ea59c700 1 --2- 172.21.15.189:0/3798639192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6ec0cedf0 0x7fd6ec15eff0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fd6e00090b0 tx=0x7fd6e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:20.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.162+0000 7fd6daffd700 1 -- 172.21.15.189:0/3798639192 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0011830 con 0x7fd6ec0cedf0 2024-04-06T13:28:20.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.162+0000 7fd6daffd700 1 -- 172.21.15.189:0/3798639192 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e001f070 con 0x7fd6ec0cedf0 2024-04-06T13:28:20.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.162+0000 7fd6f0a27700 1 -- 172.21.15.189:0/3798639192 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6ec0ad600 con 0x7fd6ec0cedf0 2024-04-06T13:28:20.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.162+0000 7fd6daffd700 1 -- 172.21.15.189:0/3798639192 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0011bf0 con 0x7fd6ec0cedf0 2024-04-06T13:28:20.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.162+0000 7fd6f0a27700 1 -- 172.21.15.189:0/3798639192 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ec0adec0 con 0x7fd6ec0cedf0 2024-04-06T13:28:20.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.163+0000 7fd6f0a27700 1 -- 172.21.15.189:0/3798639192 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6ec074990 con 0x7fd6ec0cedf0 2024-04-06T13:28:20.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.163+0000 7fd6daffd700 1 -- 172.21.15.189:0/3798639192 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 94) v1 ==== 89276+0+0 (secure 0 0 0) 0x7fd6e0011d50 con 0x7fd6ec0cedf0 2024-04-06T13:28:20.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.164+0000 7fd6daffd700 1 --2- 172.21.15.189:0/3798639192 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd6cc06ae00 0x7fd6cc06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:20.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.164+0000 7fd6e9d9b700 1 --2- 172.21.15.189:0/3798639192 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd6cc06ae00 0x7fd6cc06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:20.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.164+0000 7fd6daffd700 1 -- 172.21.15.189:0/3798639192 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fd6e0021070 con 0x7fd6ec0cedf0 2024-04-06T13:28:20.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.164+0000 7fd6e9d9b700 1 --2- 172.21.15.189:0/3798639192 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd6cc06ae00 0x7fd6cc06d2c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc001eb0 tx=0x7fd6dc009040 comp rx=0 tx=0).ready entity=mgr.9303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:20.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.168+0000 7fd6daffd700 1 -- 172.21.15.189:0/3798639192 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd6e00be650 con 0x7fd6ec0cedf0 2024-04-06T13:28:20.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:20.456+0000 7fd6f0a27700 1 -- 172.21.15.189:0/3798639192 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fd6ec0d1e20 con 0x7fd6ec0cedf0 2024-04-06T13:28:21.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.365+0000 7fd6daffd700 1 -- 172.21.15.189:0/3798639192 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 95) v1 ==== 89259+0+0 (secure 0 0 0) 0x7fd6e00563c0 con 0x7fd6ec0cedf0 2024-04-06T13:28:21.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.366+0000 7fd6daffd700 1 -- 172.21.15.189:0/3798639192 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v95) v1 ==== 93+0+0 (secure 0 0 0) 0x7fd6e0083240 con 0x7fd6ec0cedf0 2024-04-06T13:28:21.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.369+0000 7fd6f0a27700 1 -- 172.21.15.189:0/3798639192 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd6cc06ae00 msgr2=0x7fd6cc06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:21.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.369+0000 7fd6f0a27700 1 --2- 172.21.15.189:0/3798639192 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd6cc06ae00 0x7fd6cc06d2c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc001eb0 tx=0x7fd6dc009040 comp rx=0 tx=0).stop 2024-04-06T13:28:21.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.369+0000 7fd6f0a27700 1 -- 172.21.15.189:0/3798639192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6ec0cedf0 msgr2=0x7fd6ec15eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:21.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.369+0000 7fd6f0a27700 1 --2- 172.21.15.189:0/3798639192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6ec0cedf0 0x7fd6ec15eff0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fd6e00090b0 tx=0x7fd6e0004750 comp rx=0 tx=0).stop 2024-04-06T13:28:21.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.369+0000 7fd6f0a27700 1 -- 172.21.15.189:0/3798639192 shutdown_connections 2024-04-06T13:28:21.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.369+0000 7fd6f0a27700 1 --2- 172.21.15.189:0/3798639192 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd6cc06ae00 0x7fd6cc06d2c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:21.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.369+0000 7fd6f0a27700 1 --2- 172.21.15.189:0/3798639192 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6ec146dd0 0x7fd6ec0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:21.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.369+0000 7fd6f0a27700 1 --2- 172.21.15.189:0/3798639192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6ec0cedf0 0x7fd6ec15eff0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:21.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.369+0000 7fd6f0a27700 1 --2- 172.21.15.189:0/3798639192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6ec0d17a0 0x7fd6ec15a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:21.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.369+0000 7fd6f0a27700 1 -- 172.21.15.189:0/3798639192 >> 172.21.15.189:0/3798639192 conn(0x7fd6ec0b2db0 msgr2=0x7fd6ec0c7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:21.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.369+0000 7fd6f0a27700 1 -- 172.21.15.189:0/3798639192 shutdown_connections 2024-04-06T13:28:21.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.369+0000 7fd6f0a27700 1 -- 172.21.15.189:0/3798639192 wait complete. 2024-04-06T13:28:21.383 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T13:28:21.383 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:28:21.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:28:21.475 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T13:28:21.475 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:28:21.476 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:28:21.498 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:28:21.498 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:28:21.557 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:28:21.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.722+0000 7f4fcc06d700 1 Processor -- start 2024-04-06T13:28:21.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.722+0000 7f4fcc06d700 1 -- start start 2024-04-06T13:28:21.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.723+0000 7f4fcc06d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4fc415bf30 0x7f4fc40c2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:21.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.723+0000 7f4fcc06d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4fc40cc500 0x7f4fc40c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:21.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.723+0000 7f4fcc06d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc40cc910 con 0x7f4fc415c310 2024-04-06T13:28:21.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.723+0000 7f4fcc06d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fc4163090 con 0x7f4fc415bf30 2024-04-06T13:28:21.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.723+0000 7f4fcc06d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fc40af480 con 0x7f4fc40cc500 2024-04-06T13:28:21.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.724+0000 7f4fc9e09700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4fc415bf30 0x7f4fc40c2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:21.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.724+0000 7f4fc9e09700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4fc415bf30 0x7f4fc40c2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55570/0 (socket says 172.21.15.189:55570) 2024-04-06T13:28:21.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.724+0000 7f4fc9e09700 1 -- 172.21.15.189:0/2164473454 learned_addr learned my addr 172.21.15.189:0/2164473454 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:21.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.724+0000 7f4fca60a700 1 --2- 172.21.15.189:0/2164473454 >> v2:172.21.15.192:3300/0 conn(0x7f4fc40cc500 0x7f4fc40c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:21.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.724+0000 7f4fc9e09700 1 -- 172.21.15.189:0/2164473454 >> v2:172.21.15.192:3300/0 conn(0x7f4fc40cc500 msgr2=0x7f4fc40c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:21.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.724+0000 7f4fc9e09700 1 --2- 172.21.15.189:0/2164473454 >> v2:172.21.15.192:3300/0 conn(0x7f4fc40cc500 0x7f4fc40c29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:21.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.724+0000 7f4fc9e09700 1 -- 172.21.15.189:0/2164473454 >> v1:172.21.15.189:6789/0 conn(0x7f4fc415c310 legacy=0x7f4fc40cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:21.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.724+0000 7f4fc9e09700 1 -- 172.21.15.189:0/2164473454 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc40ccb40 con 0x7f4fc415bf30 2024-04-06T13:28:21.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.724+0000 7f4fc9e09700 1 --2- 172.21.15.189:0/2164473454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fc415bf30 0x7f4fc40c2460 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40b1f40 tx=0x7f4fbc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3b19f2ed5d0dcb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:21.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.725+0000 7f4fc8e07700 1 -- 172.21.15.189:0/2164473454 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc00e070 con 0x7f4fc415bf30 2024-04-06T13:28:21.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.725+0000 7f4fc8e07700 1 -- 172.21.15.189:0/2164473454 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fbc00b9c0 con 0x7f4fc415bf30 2024-04-06T13:28:21.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.725+0000 7f4fc8e07700 1 -- 172.21.15.189:0/2164473454 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc00bd70 con 0x7f4fc415bf30 2024-04-06T13:28:21.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.726+0000 7f4fcc06d700 1 -- 172.21.15.189:0/2164473454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fc415bf30 msgr2=0x7f4fc40c2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:21.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.726+0000 7f4fcc06d700 1 --2- 172.21.15.189:0/2164473454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fc415bf30 0x7f4fc40c2460 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40b1f40 tx=0x7f4fbc00b330 comp rx=0 tx=0).stop 2024-04-06T13:28:21.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.726+0000 7f4fcc06d700 1 -- 172.21.15.189:0/2164473454 shutdown_connections 2024-04-06T13:28:21.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.726+0000 7f4fcc06d700 1 --2- 172.21.15.189:0/2164473454 >> v2:172.21.15.192:3300/0 conn(0x7f4fc40cc500 0x7f4fc40c29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:21.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.726+0000 7f4fcc06d700 1 --2- 172.21.15.189:0/2164473454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fc415bf30 0x7f4fc40c2460 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:21.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.726+0000 7f4fcc06d700 1 -- 172.21.15.189:0/2164473454 >> 172.21.15.189:0/2164473454 conn(0x7f4fc40b0fe0 msgr2=0x7f4fc40b13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:21.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.726+0000 7f4fcc06d700 1 -- 172.21.15.189:0/2164473454 shutdown_connections 2024-04-06T13:28:21.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.726+0000 7f4fcc06d700 1 -- 172.21.15.189:0/2164473454 wait complete. 2024-04-06T13:28:21.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.728+0000 7f4fcc06d700 1 Processor -- start 2024-04-06T13:28:21.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.728+0000 7f4fcc06d700 1 -- start start 2024-04-06T13:28:21.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.729+0000 7f4fcc06d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fc40cc500 0x7f4fc414aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:21.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.729+0000 7f4fcc06d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fc415bf30 0x7f4fc4150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:21.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.729+0000 7f4fcc06d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fc415c310 0x7f4fc40c5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:21.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.729+0000 7f4fcc06d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc41595c0 con 0x7f4fc415bf30 2024-04-06T13:28:21.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.729+0000 7f4fcc06d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc40c5d30 con 0x7f4fc415c310 2024-04-06T13:28:21.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.729+0000 7f4fcc06d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc4158c20 con 0x7f4fc40cc500 2024-04-06T13:28:21.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.729+0000 7f4fc9e09700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fc40cc500 0x7f4fc414aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:21.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.729+0000 7f4fc9e09700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fc40cc500 0x7f4fc414aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:44168/0 (socket says 172.21.15.189:44168) 2024-04-06T13:28:21.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.729+0000 7f4fc9e09700 1 -- 172.21.15.189:0/2701935160 learned_addr learned my addr 172.21.15.189:0/2701935160 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:21.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.729+0000 7f4fc9608700 1 --2- 172.21.15.189:0/2701935160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fc415bf30 0x7f4fc4150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:21.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.729+0000 7f4fca60a700 1 --2- 172.21.15.189:0/2701935160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fc415c310 0x7f4fc40c5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:21.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.729+0000 7f4fc9e09700 1 -- 172.21.15.189:0/2701935160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fc415c310 msgr2=0x7f4fc40c5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:21.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.729+0000 7f4fc9e09700 1 --2- 172.21.15.189:0/2701935160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fc415c310 0x7f4fc40c5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:21.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.729+0000 7f4fc9e09700 1 -- 172.21.15.189:0/2701935160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fc415bf30 msgr2=0x7f4fc4150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:21.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.729+0000 7f4fc9e09700 1 --2- 172.21.15.189:0/2701935160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fc415bf30 0x7f4fc4150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:21.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.729+0000 7f4fc9e09700 1 -- 172.21.15.189:0/2701935160 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fbc009b30 con 0x7f4fc40cc500 2024-04-06T13:28:21.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.730+0000 7f4fc9e09700 1 --2- 172.21.15.189:0/2701935160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fc40cc500 0x7f4fc414aef0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40cd010 tx=0x7f4fbc004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:21.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.730+0000 7f4fba7fc700 1 -- 172.21.15.189:0/2701935160 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc004250 con 0x7f4fc40cc500 2024-04-06T13:28:21.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.730+0000 7f4fba7fc700 1 -- 172.21.15.189:0/2701935160 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fbc0043d0 con 0x7f4fc40cc500 2024-04-06T13:28:21.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.730+0000 7f4fcc06d700 1 -- 172.21.15.189:0/2701935160 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc4158f00 con 0x7f4fc40cc500 2024-04-06T13:28:21.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.730+0000 7f4fba7fc700 1 -- 172.21.15.189:0/2701935160 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc004780 con 0x7f4fc40cc500 2024-04-06T13:28:21.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.730+0000 7f4fcc06d700 1 -- 172.21.15.189:0/2701935160 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc41511f0 con 0x7f4fc40cc500 2024-04-06T13:28:21.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.731+0000 7f4fcc06d700 1 -- 172.21.15.189:0/2701935160 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc4151380 con 0x7f4fc40cc500 2024-04-06T13:28:21.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.732+0000 7f4fba7fc700 1 -- 172.21.15.189:0/2701935160 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 95) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f4fbc01c440 con 0x7f4fc40cc500 2024-04-06T13:28:21.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.732+0000 7f4fba7fc700 1 --2- 172.21.15.189:0/2701935160 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4fa806adb0 0x7f4fa806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:21.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.732+0000 7f4fc9608700 1 -- 172.21.15.189:0/2701935160 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4fa806adb0 msgr2=0x7f4fa806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:21.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.732+0000 7f4fba7fc700 1 -- 172.21.15.189:0/2701935160 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f4fbc08aca0 con 0x7f4fc40cc500 2024-04-06T13:28:21.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.732+0000 7f4fc9608700 1 --2- 172.21.15.189:0/2701935160 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4fa806adb0 0x7f4fa806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:21.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.738+0000 7f4fba7fc700 1 -- 172.21.15.189:0/2701935160 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4fbc055ab0 con 0x7f4fc40cc500 2024-04-06T13:28:21.781 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:21.780+0000 7fca68879200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:28:21.829 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:21.828+0000 7f02c5aba200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:28:21.863 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:21.862+0000 7fca68879200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:28:21.910 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:21.910+0000 7f02c5aba200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:28:21.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.933+0000 7f4fc9608700 1 -- 172.21.15.189:0/2701935160 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4fa806adb0 msgr2=0x7f4fa806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:21.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.933+0000 7f4fc9608700 1 --2- 172.21.15.189:0/2701935160 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4fa806adb0 0x7f4fa806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:21.935 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:21.934+0000 7fca68879200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:28:21.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.990+0000 7f4fcc06d700 1 -- 172.21.15.189:0/2701935160 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fc4074990 con 0x7f4fc40cc500 2024-04-06T13:28:21.990 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:21.990+0000 7f02c5aba200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:28:21.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.991+0000 7f4fba7fc700 1 -- 172.21.15.189:0/2701935160 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v165) v1 ==== 74+0+16976 (secure 0 0 0) 0x7f4fbc020020 con 0x7f4fc40cc500 2024-04-06T13:28:21.991 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:21.991 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":165,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:28:21.351197+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:32:00.826039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-04-06T12:32:02.039801+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"165","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":112,"snap_epoch":165,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/3791371660":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/3889346008":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/628196195":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/4023078006":"2024-04-07T12:32:16.629669+0000","172.21.15.189:6833/47128":"2024-04-07T12:32:16.629669+0000","172.21.15.189:6832/47128":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/3100554908":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/383422220":"2024-04-07T12:32:16.629669+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":54},{"begin":59,"length":54}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":47,"length":10},{"begin":103,"length":10}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:28:21.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.995+0000 7f4fcc06d700 1 -- 172.21.15.189:0/2701935160 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4fa806adb0 msgr2=0x7f4fa806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:21.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.995+0000 7f4fcc06d700 1 --2- 172.21.15.189:0/2701935160 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4fa806adb0 0x7f4fa806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:21.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.995+0000 7f4fcc06d700 1 -- 172.21.15.189:0/2701935160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fc40cc500 msgr2=0x7f4fc414aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:21.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.995+0000 7f4fcc06d700 1 --2- 172.21.15.189:0/2701935160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fc40cc500 0x7f4fc414aef0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40cd010 tx=0x7f4fbc004ad0 comp rx=0 tx=0).stop 2024-04-06T13:28:21.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.995+0000 7f4fcc06d700 1 -- 172.21.15.189:0/2701935160 shutdown_connections 2024-04-06T13:28:21.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.995+0000 7f4fcc06d700 1 --2- 172.21.15.189:0/2701935160 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4fa806adb0 0x7f4fa806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:21.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.995+0000 7f4fcc06d700 1 --2- 172.21.15.189:0/2701935160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fc415c310 0x7f4fc40c5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:21.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.995+0000 7f4fcc06d700 1 --2- 172.21.15.189:0/2701935160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fc40cc500 0x7f4fc414aef0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:21.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.995+0000 7f4fcc06d700 1 --2- 172.21.15.189:0/2701935160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fc415bf30 0x7f4fc4150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:21.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.995+0000 7f4fcc06d700 1 -- 172.21.15.189:0/2701935160 >> 172.21.15.189:0/2701935160 conn(0x7f4fc40b0fe0 msgr2=0x7f4fc40af910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:21.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.995+0000 7f4fcc06d700 1 -- 172.21.15.189:0/2701935160 shutdown_connections 2024-04-06T13:28:21.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:21.995+0000 7f4fcc06d700 1 -- 172.21.15.189:0/2701935160 wait complete. 2024-04-06T13:28:22.009 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:22.009+0000 7fca68879200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:28:22.012 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:32:00.826039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '157', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:32:02.039801+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '165', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 112, 'snap_epoch': 165, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:28:22.013 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:28:22.070 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:22.070+0000 7f02c5aba200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:28:22.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.142+0000 7f70db4f6700 1 Processor -- start 2024-04-06T13:28:22.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.142+0000 7f70db4f6700 1 -- start start 2024-04-06T13:28:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.143+0000 7f70db4f6700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f70d415d7e0 0x7f70d40d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.143+0000 7f70db4f6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f70d40d0890 0x7f70d40d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.143+0000 7f70db4f6700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70d40cdf80 con 0x7f70d40d1af0 2024-04-06T13:28:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.143+0000 7f70db4f6700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70d40d1f80 con 0x7f70d40d0890 2024-04-06T13:28:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.143+0000 7f70db4f6700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70d40d3230 con 0x7f70d415d7e0 2024-04-06T13:28:22.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.143+0000 7f70d8a91700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f70d40d0890 0x7f70d40d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.143+0000 7f70d8a91700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f70d40d0890 0x7f70d40d15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55580/0 (socket says 172.21.15.189:55580) 2024-04-06T13:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.143+0000 7f70d8a91700 1 -- 172.21.15.189:0/4120271452 learned_addr learned my addr 172.21.15.189:0/4120271452 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.143+0000 7f70d9292700 1 --2- 172.21.15.189:0/4120271452 >> v2:172.21.15.192:3300/0 conn(0x7f70d415d7e0 0x7f70d40d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.143+0000 7f70d8a91700 1 -- 172.21.15.189:0/4120271452 >> v2:172.21.15.192:3300/0 conn(0x7f70d415d7e0 msgr2=0x7f70d40d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.143+0000 7f70d8a91700 1 --2- 172.21.15.189:0/4120271452 >> v2:172.21.15.192:3300/0 conn(0x7f70d415d7e0 0x7f70d40d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.143+0000 7f70d8a91700 1 -- 172.21.15.189:0/4120271452 >> v1:172.21.15.192:6789/0 conn(0x7f70d40d1af0 legacy=0x7f70d40b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.143+0000 7f70d8a91700 1 -- 172.21.15.189:0/4120271452 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d40d0cf0 con 0x7f70d40d0890 2024-04-06T13:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.143+0000 7f70d8a91700 1 --2- 172.21.15.189:0/4120271452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70d40d0890 0x7f70d40d15b0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f70d408c190 tx=0x7f70c4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8f4babf5273f71 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.144+0000 7f70cbfff700 1 -- 172.21.15.189:0/4120271452 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c4011830 con 0x7f70d40d0890 2024-04-06T13:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.144+0000 7f70cbfff700 1 -- 172.21.15.189:0/4120271452 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c4011990 con 0x7f70d40d0890 2024-04-06T13:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.144+0000 7f70cbfff700 1 -- 172.21.15.189:0/4120271452 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c4011d20 con 0x7f70d40d0890 2024-04-06T13:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.144+0000 7f70db4f6700 1 -- 172.21.15.189:0/4120271452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70d40d0890 msgr2=0x7f70d40d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.144+0000 7f70db4f6700 1 --2- 172.21.15.189:0/4120271452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70d40d0890 0x7f70d40d15b0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f70d408c190 tx=0x7f70c4009bd0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.144+0000 7f70db4f6700 1 -- 172.21.15.189:0/4120271452 shutdown_connections 2024-04-06T13:28:22.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.144+0000 7f70db4f6700 1 --2- 172.21.15.189:0/4120271452 >> v2:172.21.15.192:3300/0 conn(0x7f70d415d7e0 0x7f70d40d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.144+0000 7f70db4f6700 1 --2- 172.21.15.189:0/4120271452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70d40d0890 0x7f70d40d15b0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.144+0000 7f70db4f6700 1 -- 172.21.15.189:0/4120271452 >> 172.21.15.189:0/4120271452 conn(0x7f70d4072250 msgr2=0x7f70d4072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:22.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.144+0000 7f70db4f6700 1 -- 172.21.15.189:0/4120271452 shutdown_connections 2024-04-06T13:28:22.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.144+0000 7f70db4f6700 1 -- 172.21.15.189:0/4120271452 wait complete. 2024-04-06T13:28:22.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.145+0000 7f70db4f6700 1 Processor -- start 2024-04-06T13:28:22.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.146+0000 7f70db4f6700 1 -- start start 2024-04-06T13:28:22.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.146+0000 7f70db4f6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70d40d0890 0x7f70d415b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:22.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.146+0000 7f70db4f6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70d40d1af0 0x7f70d40ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:22.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.146+0000 7f70db4f6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70d415d7e0 0x7f70d40b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:22.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.146+0000 7f70db4f6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d40cba70 con 0x7f70d415d7e0 2024-04-06T13:28:22.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.146+0000 7f70db4f6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d41f2cc0 con 0x7f70d40d1af0 2024-04-06T13:28:22.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.146+0000 7f70db4f6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70d415bdf0 con 0x7f70d40d0890 2024-04-06T13:28:22.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.146+0000 7f70d9a93700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70d415d7e0 0x7f70d40b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:22.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.146+0000 7f70d9a93700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70d415d7e0 0x7f70d40b51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55586/0 (socket says 172.21.15.189:55586) 2024-04-06T13:28:22.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.146+0000 7f70d9a93700 1 -- 172.21.15.189:0/3619320737 learned_addr learned my addr 172.21.15.189:0/3619320737 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:22.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.146+0000 7f70d9292700 1 --2- 172.21.15.189:0/3619320737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70d40d0890 0x7f70d415b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:22.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.146+0000 7f70d8a91700 1 --2- 172.21.15.189:0/3619320737 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70d40d1af0 0x7f70d40ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:22.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.147+0000 7f70d9a93700 1 -- 172.21.15.189:0/3619320737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70d40d0890 msgr2=0x7f70d415b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:22.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.147+0000 7f70d9a93700 1 --2- 172.21.15.189:0/3619320737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70d40d0890 0x7f70d415b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.147+0000 7f70d9a93700 1 -- 172.21.15.189:0/3619320737 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70d40d1af0 msgr2=0x7f70d40ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:22.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.147+0000 7f70d9a93700 1 --2- 172.21.15.189:0/3619320737 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70d40d1af0 0x7f70d40ca4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.147+0000 7f70d9a93700 1 -- 172.21.15.189:0/3619320737 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c4009550 con 0x7f70d415d7e0 2024-04-06T13:28:22.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.147+0000 7f70d9a93700 1 --2- 172.21.15.189:0/3619320737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70d415d7e0 0x7f70d40b51f0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f70cc00cb50 tx=0x7f70cc00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:22.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.147+0000 7f70c9ffb700 1 -- 172.21.15.189:0/3619320737 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc003c00 con 0x7f70d415d7e0 2024-04-06T13:28:22.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.147+0000 7f70c9ffb700 1 -- 172.21.15.189:0/3619320737 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70cc003d60 con 0x7f70d415d7e0 2024-04-06T13:28:22.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.147+0000 7f70db4f6700 1 -- 172.21.15.189:0/3619320737 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d40caef0 con 0x7f70d415d7e0 2024-04-06T13:28:22.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.147+0000 7f70c9ffb700 1 -- 172.21.15.189:0/3619320737 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc00aa20 con 0x7f70d415d7e0 2024-04-06T13:28:22.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.147+0000 7f70db4f6700 1 -- 172.21.15.189:0/3619320737 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d40cb4b0 con 0x7f70d415d7e0 2024-04-06T13:28:22.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.148+0000 7f70db4f6700 1 -- 172.21.15.189:0/3619320737 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70d4074990 con 0x7f70d415d7e0 2024-04-06T13:28:22.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.149+0000 7f70c9ffb700 1 -- 172.21.15.189:0/3619320737 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 95) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f70cc00ab80 con 0x7f70d415d7e0 2024-04-06T13:28:22.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.150+0000 7f70c9ffb700 1 --2- 172.21.15.189:0/3619320737 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f70b806adb0 0x7f70b806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:22.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.150+0000 7f70d9292700 1 -- 172.21.15.189:0/3619320737 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f70b806adb0 msgr2=0x7f70b806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:22.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.150+0000 7f70c9ffb700 1 -- 172.21.15.189:0/3619320737 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f70cc089770 con 0x7f70d415d7e0 2024-04-06T13:28:22.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.150+0000 7f70d9292700 1 --2- 172.21.15.189:0/3619320737 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f70b806adb0 0x7f70b806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:22.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.157+0000 7f70c9ffb700 1 -- 172.21.15.189:0/3619320737 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f70cc054560 con 0x7f70d415d7e0 2024-04-06T13:28:22.183 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:22.182+0000 7fca68879200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:28:22.258 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:22.258+0000 7f02c5aba200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:28:22.340 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:22.339+0000 7fca68879200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:28:22.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.351+0000 7f70d9292700 1 -- 172.21.15.189:0/3619320737 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f70b806adb0 msgr2=0x7f70b806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:22.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.351+0000 7f70d9292700 1 --2- 172.21.15.189:0/3619320737 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f70b806adb0 0x7f70b806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:22.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.359+0000 7f70db4f6700 1 -- 172.21.15.189:0/3619320737 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f70d40b7260 con 0x7f70d415d7e0 2024-04-06T13:28:22.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.361+0000 7f70c9ffb700 1 -- 172.21.15.189:0/3619320737 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v165) v1 ==== 93+0+10 (secure 0 0 0) 0x7f70cc057b80 con 0x7f70d415d7e0 2024-04-06T13:28:22.360 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:28:22.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.364+0000 7f70db4f6700 1 -- 172.21.15.189:0/3619320737 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f70b806adb0 msgr2=0x7f70b806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:22.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.364+0000 7f70db4f6700 1 --2- 172.21.15.189:0/3619320737 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f70b806adb0 0x7f70b806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.364+0000 7f70db4f6700 1 -- 172.21.15.189:0/3619320737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70d415d7e0 msgr2=0x7f70d40b51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:22.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.364+0000 7f70db4f6700 1 --2- 172.21.15.189:0/3619320737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70d415d7e0 0x7f70d40b51f0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f70cc00cb50 tx=0x7f70cc00ce60 comp rx=0 tx=0).stop 2024-04-06T13:28:22.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.364+0000 7f70db4f6700 1 -- 172.21.15.189:0/3619320737 shutdown_connections 2024-04-06T13:28:22.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.364+0000 7f70db4f6700 1 --2- 172.21.15.189:0/3619320737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70d415d7e0 0x7f70d40b51f0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.364+0000 7f70db4f6700 1 --2- 172.21.15.189:0/3619320737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70d40d0890 0x7f70d415b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.364+0000 7f70db4f6700 1 --2- 172.21.15.189:0/3619320737 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f70b806adb0 0x7f70b806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.364+0000 7f70db4f6700 1 --2- 172.21.15.189:0/3619320737 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70d40d1af0 0x7f70d40ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.364+0000 7f70db4f6700 1 -- 172.21.15.189:0/3619320737 >> 172.21.15.189:0/3619320737 conn(0x7f70d4072250 msgr2=0x7f70d40b66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:22.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.364+0000 7f70db4f6700 1 -- 172.21.15.189:0/3619320737 shutdown_connections 2024-04-06T13:28:22.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.364+0000 7f70db4f6700 1 -- 172.21.15.189:0/3619320737 wait complete. 2024-04-06T13:28:22.380 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T13:28:22.430 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:22.430+0000 7f02c5aba200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:28:22.449 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:22.448+0000 7fca68879200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:28:22.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.509+0000 7f04425de700 1 Processor -- start 2024-04-06T13:28:22.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.509+0000 7f04425de700 1 -- start start 2024-04-06T13:28:22.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.510+0000 7f04425de700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f043c0cb870 0x7f043c15c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:22.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.510+0000 7f04425de700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f043c0cfa00 0x7f043c0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:22.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.510+0000 7f04425de700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f043c0cbd80 con 0x7f043c0d11b0 2024-04-06T13:28:22.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.510+0000 7f04425de700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f043c0ccdc0 con 0x7f043c0cfa00 2024-04-06T13:28:22.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.510+0000 7f04425de700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f043c0d0b50 con 0x7f043c0cb870 2024-04-06T13:28:22.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.510+0000 7f043bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f043c0d11b0 0x7f043c0d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51416/0 (socket says 172.21.15.189:51416) 2024-04-06T13:28:22.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.510+0000 7f043bfff700 1 -- 172.21.15.189:0/1145432819 learned_addr learned my addr 172.21.15.189:0/1145432819 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:22.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.511+0000 7f043b7fe700 1 --2- 172.21.15.189:0/1145432819 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f043c0cb870 0x7f043c15c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:22.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.511+0000 7f0440b7b700 1 --2- 172.21.15.189:0/1145432819 >> v2:172.21.15.192:3300/0 conn(0x7f043c0cfa00 0x7f043c0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:22.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.511+0000 7f043affd700 1 -- 172.21.15.189:0/1145432819 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293722444 0 0) 0x7f043c0cbd80 con 0x7f043c0d11b0 2024-04-06T13:28:22.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.511+0000 7f043affd700 1 -- 172.21.15.189:0/1145432819 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04240035b0 con 0x7f043c0d11b0 2024-04-06T13:28:22.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.511+0000 7f043affd700 1 -- 172.21.15.189:0/1145432819 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 851280103 0 0) 0x7f04240035b0 con 0x7f043c0d11b0 2024-04-06T13:28:22.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.511+0000 7f043b7fe700 1 -- 172.21.15.189:0/1145432819 >> v2:172.21.15.192:3300/0 conn(0x7f043c0cfa00 msgr2=0x7f043c0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:22.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.511+0000 7f043b7fe700 1 --2- 172.21.15.189:0/1145432819 >> v2:172.21.15.192:3300/0 conn(0x7f043c0cfa00 0x7f043c0cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.511+0000 7f043b7fe700 1 -- 172.21.15.189:0/1145432819 >> v1:172.21.15.189:6789/0 conn(0x7f043c0d11b0 legacy=0x7f043c0d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:22.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.511+0000 7f043b7fe700 1 -- 172.21.15.189:0/1145432819 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043c0cef20 con 0x7f043c0cb870 2024-04-06T13:28:22.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.511+0000 7f0440b7b700 1 --2- 172.21.15.189:0/1145432819 >> v2:172.21.15.192:3300/0 conn(0x7f043c0cfa00 0x7f043c0cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:22.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.511+0000 7f043b7fe700 1 --2- 172.21.15.189:0/1145432819 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f043c0cb870 0x7f043c15c280 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0430009840 tx=0x7f0430009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe5a6b99cb37014 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:22.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.511+0000 7f043affd700 1 -- 172.21.15.189:0/1145432819 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c001670 con 0x7f043c0cb870 2024-04-06T13:28:22.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.511+0000 7f043affd700 1 -- 172.21.15.189:0/1145432819 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0430004030 con 0x7f043c0cb870 2024-04-06T13:28:22.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.511+0000 7f043affd700 1 -- 172.21.15.189:0/1145432819 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04300043c0 con 0x7f043c0cb870 2024-04-06T13:28:22.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.512+0000 7f04425de700 1 -- 172.21.15.189:0/1145432819 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f043c0cb870 msgr2=0x7f043c15c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:22.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.512+0000 7f04425de700 1 --2- 172.21.15.189:0/1145432819 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f043c0cb870 0x7f043c15c280 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0430009840 tx=0x7f0430009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:22.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.512+0000 7f04425de700 1 -- 172.21.15.189:0/1145432819 shutdown_connections 2024-04-06T13:28:22.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.512+0000 7f04425de700 1 --2- 172.21.15.189:0/1145432819 >> v2:172.21.15.192:3300/0 conn(0x7f043c0cfa00 0x7f043c0cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.512+0000 7f04425de700 1 --2- 172.21.15.189:0/1145432819 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f043c0cb870 0x7f043c15c280 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.512+0000 7f04425de700 1 -- 172.21.15.189:0/1145432819 >> 172.21.15.189:0/1145432819 conn(0x7f043c0b2db0 msgr2=0x7f043c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:22.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.512+0000 7f04425de700 1 -- 172.21.15.189:0/1145432819 shutdown_connections 2024-04-06T13:28:22.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.512+0000 7f04425de700 1 -- 172.21.15.189:0/1145432819 wait complete. 2024-04-06T13:28:22.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.513+0000 7f04425de700 1 Processor -- start 2024-04-06T13:28:22.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.513+0000 7f04425de700 1 -- start start 2024-04-06T13:28:22.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.514+0000 7f04425de700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f043c0cb870 0x7f043c0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:22.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.514+0000 7f04425de700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f043c0cfa00 0x7f043c0cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:22.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.514+0000 7f04425de700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f043c0d11b0 0x7f043c1598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:22.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.514+0000 7f04425de700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c0c8cc0 con 0x7f043c0cfa00 2024-04-06T13:28:22.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.514+0000 7f04425de700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c0ca0f0 con 0x7f043c0cb870 2024-04-06T13:28:22.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.514+0000 7f04425de700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f043c0c8390 con 0x7f043c0d11b0 2024-04-06T13:28:22.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.514+0000 7f043b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f043c0cfa00 0x7f043c0cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:22.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.514+0000 7f043b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f043c0cfa00 0x7f043c0cab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55592/0 (socket says 172.21.15.189:55592) 2024-04-06T13:28:22.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.514+0000 7f043b7fe700 1 -- 172.21.15.189:0/3596574838 learned_addr learned my addr 172.21.15.189:0/3596574838 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:22.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.514+0000 7f043bfff700 1 --2- 172.21.15.189:0/3596574838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f043c0cb870 0x7f043c0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:22.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.514+0000 7f0440b7b700 1 --2- 172.21.15.189:0/3596574838 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f043c0d11b0 0x7f043c1598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:22.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.514+0000 7f043b7fe700 1 -- 172.21.15.189:0/3596574838 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f043c0d11b0 msgr2=0x7f043c1598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:22.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.514+0000 7f043b7fe700 1 --2- 172.21.15.189:0/3596574838 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f043c0d11b0 0x7f043c1598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.514+0000 7f043b7fe700 1 -- 172.21.15.189:0/3596574838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f043c0cb870 msgr2=0x7f043c0ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:22.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.514+0000 7f043b7fe700 1 --2- 172.21.15.189:0/3596574838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f043c0cb870 0x7f043c0ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.515+0000 7f043b7fe700 1 -- 172.21.15.189:0/3596574838 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0434009a70 con 0x7f043c0cfa00 2024-04-06T13:28:22.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.515+0000 7f043b7fe700 1 --2- 172.21.15.189:0/3596574838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f043c0cfa00 0x7f043c0cab30 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f0430009790 tx=0x7f0430004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:22.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.515+0000 7f0438ff9700 1 -- 172.21.15.189:0/3596574838 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430003cb0 con 0x7f043c0cfa00 2024-04-06T13:28:22.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.515+0000 7f0438ff9700 1 -- 172.21.15.189:0/3596574838 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043001f070 con 0x7f043c0cfa00 2024-04-06T13:28:22.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.515+0000 7f04425de700 1 -- 172.21.15.189:0/3596574838 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04300094a0 con 0x7f043c0cfa00 2024-04-06T13:28:22.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.515+0000 7f0438ff9700 1 -- 172.21.15.189:0/3596574838 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430011870 con 0x7f043c0cfa00 2024-04-06T13:28:22.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.515+0000 7f04425de700 1 -- 172.21.15.189:0/3596574838 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f043c0c88e0 con 0x7f043c0cfa00 2024-04-06T13:28:22.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.516+0000 7f04425de700 1 -- 172.21.15.189:0/3596574838 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f043c0c8e00 con 0x7f043c0cfa00 2024-04-06T13:28:22.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.516+0000 7f0438ff9700 1 -- 172.21.15.189:0/3596574838 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 95) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f04300119d0 con 0x7f043c0cfa00 2024-04-06T13:28:22.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.517+0000 7f0438ff9700 1 --2- 172.21.15.189:0/3596574838 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f042806adb0 0x7f042806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:22.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.517+0000 7f043bfff700 1 -- 172.21.15.189:0/3596574838 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f042806adb0 msgr2=0x7f042806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:22.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.517+0000 7f0438ff9700 1 -- 172.21.15.189:0/3596574838 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f0430021070 con 0x7f043c0cfa00 2024-04-06T13:28:22.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.517+0000 7f043bfff700 1 --2- 172.21.15.189:0/3596574838 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f042806adb0 0x7f042806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:22.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.521+0000 7f0438ff9700 1 -- 172.21.15.189:0/3596574838 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f043c0c8e00 con 0x7f043c0cfa00 2024-04-06T13:28:22.530 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:22.529+0000 7fca68879200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:28:22.547 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:22.547+0000 7f02c5aba200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:28:22.605 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:22.604+0000 7fca68879200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:28:22.632 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:22.632+0000 7f02c5aba200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:28:22.709 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:22.709+0000 7f02c5aba200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:28:22.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.718+0000 7f043bfff700 1 -- 172.21.15.189:0/3596574838 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f042806adb0 msgr2=0x7f042806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:22.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.718+0000 7f043bfff700 1 --2- 172.21.15.189:0/3596574838 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f042806adb0 0x7f042806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:22.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.722+0000 7f04425de700 1 -- 172.21.15.189:0/3596574838 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f043c0c8e00 con 0x7f043c0cfa00 2024-04-06T13:28:22.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.722+0000 7f0438ff9700 1 -- 172.21.15.189:0/3596574838 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v165) v1 ==== 104+0+11 (secure 0 0 0) 0x7f043c0c8e00 con 0x7f043c0cfa00 2024-04-06T13:28:22.722 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:28:22.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.724+0000 7f04425de700 1 -- 172.21.15.189:0/3596574838 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f042806adb0 msgr2=0x7f042806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:22.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.724+0000 7f04425de700 1 --2- 172.21.15.189:0/3596574838 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f042806adb0 0x7f042806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.724+0000 7f04425de700 1 -- 172.21.15.189:0/3596574838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f043c0cfa00 msgr2=0x7f043c0cab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:22.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.724+0000 7f04425de700 1 --2- 172.21.15.189:0/3596574838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f043c0cfa00 0x7f043c0cab30 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f0430009790 tx=0x7f0430004750 comp rx=0 tx=0).stop 2024-04-06T13:28:22.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.724+0000 7f04425de700 1 -- 172.21.15.189:0/3596574838 shutdown_connections 2024-04-06T13:28:22.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.724+0000 7f04425de700 1 --2- 172.21.15.189:0/3596574838 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f043c0d11b0 0x7f043c1598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.724+0000 7f04425de700 1 --2- 172.21.15.189:0/3596574838 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f042806adb0 0x7f042806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.724+0000 7f04425de700 1 --2- 172.21.15.189:0/3596574838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f043c0cb870 0x7f043c0ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.724+0000 7f04425de700 1 --2- 172.21.15.189:0/3596574838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f043c0cfa00 0x7f043c0cab30 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.724+0000 7f04425de700 1 -- 172.21.15.189:0/3596574838 >> 172.21.15.189:0/3596574838 conn(0x7f043c0b2db0 msgr2=0x7f043c0d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:22.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.724+0000 7f04425de700 1 -- 172.21.15.189:0/3596574838 shutdown_connections 2024-04-06T13:28:22.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.724+0000 7f04425de700 1 -- 172.21.15.189:0/3596574838 wait complete. 2024-04-06T13:28:22.737 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T13:28:22.807 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:22.806+0000 7fca68879200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:28:22.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.867+0000 7f3accf24700 1 Processor -- start 2024-04-06T13:28:22.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.868+0000 7f3accf24700 1 -- start start 2024-04-06T13:28:22.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.869+0000 7f3accf24700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3ac80d0300 0x7f3ac80d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:22.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.869+0000 7f3accf24700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac8148000 con 0x7f3ac8147490 2024-04-06T13:28:22.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.869+0000 7f3accf24700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac80d0c20 con 0x7f3ac8165690 2024-04-06T13:28:22.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.869+0000 7f3accf24700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ac8162360 con 0x7f3ac80d0300 2024-04-06T13:28:22.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.869+0000 7f3ac5d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3ac8147490 0x7f3ac81478f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51426/0 (socket says 172.21.15.189:51426) 2024-04-06T13:28:22.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.869+0000 7f3ac5d9b700 1 -- 172.21.15.189:0/182194477 learned_addr learned my addr 172.21.15.189:0/182194477 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:22.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.869+0000 7f3ac659c700 1 --2- 172.21.15.189:0/182194477 >> v2:172.21.15.192:3300/0 conn(0x7f3ac80d0300 0x7f3ac80d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:22.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.870+0000 7f3ac659c700 1 -- 172.21.15.189:0/182194477 >> v1:172.21.15.192:6789/0 conn(0x7f3ac8165690 legacy=0x7f3ac8165ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:22.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.870+0000 7f3ac659c700 1 -- 172.21.15.189:0/182194477 >> v1:172.21.15.189:6789/0 conn(0x7f3ac8147490 legacy=0x7f3ac81478f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:22.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.870+0000 7f3ac559a700 1 -- 172.21.15.189:0/182194477 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846565764 0 0) 0x7f3ac8148000 con 0x7f3ac8147490 2024-04-06T13:28:22.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.870+0000 7f3ac659c700 1 -- 172.21.15.189:0/182194477 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac8162640 con 0x7f3ac80d0300 2024-04-06T13:28:22.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.870+0000 7f3ac659c700 1 --2- 172.21.15.189:0/182194477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ac80d0300 0x7f3ac80d06e0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009d90 tx=0x7f3ac000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2de4cf055a4c838 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:22.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.870+0000 7f3ac559a700 1 -- 172.21.15.189:0/182194477 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac000e070 con 0x7f3ac80d0300 2024-04-06T13:28:22.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.870+0000 7f3ac559a700 1 -- 172.21.15.189:0/182194477 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac000bb90 con 0x7f3ac80d0300 2024-04-06T13:28:22.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.870+0000 7f3ac559a700 1 -- 172.21.15.189:0/182194477 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0003a20 con 0x7f3ac80d0300 2024-04-06T13:28:22.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.871+0000 7f3accf24700 1 -- 172.21.15.189:0/182194477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ac80d0300 msgr2=0x7f3ac80d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:22.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.871+0000 7f3accf24700 1 --2- 172.21.15.189:0/182194477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ac80d0300 0x7f3ac80d06e0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009d90 tx=0x7f3ac000b330 comp rx=0 tx=0).stop 2024-04-06T13:28:22.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.871+0000 7f3accf24700 1 -- 172.21.15.189:0/182194477 shutdown_connections 2024-04-06T13:28:22.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.871+0000 7f3accf24700 1 --2- 172.21.15.189:0/182194477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ac80d0300 0x7f3ac80d06e0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.871+0000 7f3accf24700 1 -- 172.21.15.189:0/182194477 >> 172.21.15.189:0/182194477 conn(0x7f3ac8072250 msgr2=0x7f3ac8072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:22.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.871+0000 7f3accf24700 1 -- 172.21.15.189:0/182194477 shutdown_connections 2024-04-06T13:28:22.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.871+0000 7f3accf24700 1 -- 172.21.15.189:0/182194477 wait complete. 2024-04-06T13:28:22.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.873+0000 7f3accf24700 1 Processor -- start 2024-04-06T13:28:22.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.873+0000 7f3accf24700 1 -- start start 2024-04-06T13:28:22.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.874+0000 7f3accf24700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ac80d0300 0x7f3ac80b56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:22.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.874+0000 7f3accf24700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ac8147490 0x7f3ac80c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:22.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.874+0000 7f3accf24700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ac8165690 0x7f3ac8164290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:22.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.874+0000 7f3accf24700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac80c9540 con 0x7f3ac80d0300 2024-04-06T13:28:22.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.874+0000 7f3accf24700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac80c9960 con 0x7f3ac8147490 2024-04-06T13:28:22.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.874+0000 7f3accf24700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac80b5c00 con 0x7f3ac8165690 2024-04-06T13:28:22.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.875+0000 7f3ac5d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ac8147490 0x7f3ac80c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:22.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.875+0000 7f3ac5d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ac8147490 0x7f3ac80c7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:39744/0 (socket says 172.21.15.189:39744) 2024-04-06T13:28:22.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.875+0000 7f3ac5d9b700 1 -- 172.21.15.189:0/2558805029 learned_addr learned my addr 172.21.15.189:0/2558805029 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:22.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.875+0000 7f3ac659c700 1 --2- 172.21.15.189:0/2558805029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ac80d0300 0x7f3ac80b56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:22.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.875+0000 7f3ac5d9b700 1 -- 172.21.15.189:0/2558805029 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ac8165690 msgr2=0x7f3ac8164290 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-06T13:28:22.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.875+0000 7f3ac5d9b700 1 --2- 172.21.15.189:0/2558805029 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ac8165690 0x7f3ac8164290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.875+0000 7f3ac5d9b700 1 -- 172.21.15.189:0/2558805029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ac80d0300 msgr2=0x7f3ac80b56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:22.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.875+0000 7f3ac5d9b700 1 --2- 172.21.15.189:0/2558805029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ac80d0300 0x7f3ac80b56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:22.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.875+0000 7f3ac5d9b700 1 -- 172.21.15.189:0/2558805029 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac0009a70 con 0x7f3ac8147490 2024-04-06T13:28:22.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.875+0000 7f3ac5d9b700 1 --2- 172.21.15.189:0/2558805029 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ac8147490 0x7f3ac80c7f50 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3ab800c520 tx=0x7f3ab800c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:22.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.875+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/2558805029 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab8004c80 con 0x7f3ac8147490 2024-04-06T13:28:22.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.875+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/2558805029 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab8004de0 con 0x7f3ac8147490 2024-04-06T13:28:22.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.876+0000 7f3accf24700 1 -- 172.21.15.189:0/2558805029 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac80c88c0 con 0x7f3ac8147490 2024-04-06T13:28:22.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.876+0000 7f3accf24700 1 -- 172.21.15.189:0/2558805029 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac80c8e80 con 0x7f3ac8147490 2024-04-06T13:28:22.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.876+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/2558805029 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab80072f0 con 0x7f3ac8147490 2024-04-06T13:28:22.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.877+0000 7f3accf24700 1 -- 172.21.15.189:0/2558805029 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aac0052f0 con 0x7f3ac8147490 2024-04-06T13:28:22.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.878+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/2558805029 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 95) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f3ab8007450 con 0x7f3ac8147490 2024-04-06T13:28:22.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.878+0000 7f3ab6ffd700 1 --2- 172.21.15.189:0/2558805029 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3aa806adb0 0x7f3aa806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:22.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.878+0000 7f3ac659c700 1 -- 172.21.15.189:0/2558805029 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3aa806adb0 msgr2=0x7f3aa806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:22.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.878+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/2558805029 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f3ab8089340 con 0x7f3ac8147490 2024-04-06T13:28:22.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.878+0000 7f3ac659c700 1 --2- 172.21.15.189:0/2558805029 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3aa806adb0 0x7f3aa806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:22.885 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:22.883+0000 7fca68879200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:28:22.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:22.883+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/2558805029 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3ab80541b0 con 0x7f3ac8147490 2024-04-06T13:28:22.927 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:22.928+0000 7f02c5aba200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:28:23.005 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:23.005+0000 7f02c5aba200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:28:23.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.079+0000 7f3ac659c700 1 -- 172.21.15.189:0/2558805029 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3aa806adb0 msgr2=0x7f3aa806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:23.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.079+0000 7f3ac659c700 1 --2- 172.21.15.189:0/2558805029 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3aa806adb0 0x7f3aa806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:23.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.112+0000 7f3accf24700 1 -- 172.21.15.189:0/2558805029 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3aac005b20 con 0x7f3ac8147490 2024-04-06T13:28:23.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.113+0000 7f3ab6ffd700 1 -- 172.21.15.189:0/2558805029 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v165) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3ab8007700 con 0x7f3ac8147490 2024-04-06T13:28:23.113 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:28:23.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.115+0000 7f3accf24700 1 -- 172.21.15.189:0/2558805029 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3aa806adb0 msgr2=0x7f3aa806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:23.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.115+0000 7f3accf24700 1 --2- 172.21.15.189:0/2558805029 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3aa806adb0 0x7f3aa806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.115+0000 7f3accf24700 1 -- 172.21.15.189:0/2558805029 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ac8147490 msgr2=0x7f3ac80c7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:23.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.115+0000 7f3accf24700 1 --2- 172.21.15.189:0/2558805029 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ac8147490 0x7f3ac80c7f50 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3ab800c520 tx=0x7f3ab800c830 comp rx=0 tx=0).stop 2024-04-06T13:28:23.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.115+0000 7f3accf24700 1 -- 172.21.15.189:0/2558805029 shutdown_connections 2024-04-06T13:28:23.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.115+0000 7f3accf24700 1 --2- 172.21.15.189:0/2558805029 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ac8165690 0x7f3ac8164290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.115+0000 7f3accf24700 1 --2- 172.21.15.189:0/2558805029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ac80d0300 0x7f3ac80b56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.115+0000 7f3accf24700 1 --2- 172.21.15.189:0/2558805029 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3aa806adb0 0x7f3aa806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.115+0000 7f3accf24700 1 --2- 172.21.15.189:0/2558805029 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ac8147490 0x7f3ac80c7f50 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.115+0000 7f3accf24700 1 -- 172.21.15.189:0/2558805029 >> 172.21.15.189:0/2558805029 conn(0x7f3ac8072250 msgr2=0x7f3ac81468f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:23.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.115+0000 7f3accf24700 1 -- 172.21.15.189:0/2558805029 shutdown_connections 2024-04-06T13:28:23.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.115+0000 7f3accf24700 1 -- 172.21.15.189:0/2558805029 wait complete. 2024-04-06T13:28:23.128 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T13:28:23.201 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:23.201+0000 7fca68879200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:28:23.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.256+0000 7f3b24bee700 1 Processor -- start 2024-04-06T13:28:23.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.256+0000 7f3b24bee700 1 -- start start 2024-04-06T13:28:23.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.257+0000 7f3b24bee700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b200d17a0 0x7f3b2006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.257+0000 7f3b24bee700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3b200cedf0 0x7f3b200cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.257+0000 7f3b24bee700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b200cf760 con 0x7f3b20146dd0 2024-04-06T13:28:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.257+0000 7f3b24bee700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2006d850 con 0x7f3b200cedf0 2024-04-06T13:28:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.257+0000 7f3b24bee700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2006c0a0 con 0x7f3b200d17a0 2024-04-06T13:28:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.257+0000 7f3b1dd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3b20146dd0 0x7f3b2006d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51432/0 (socket says 172.21.15.189:51432) 2024-04-06T13:28:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.257+0000 7f3b1dd9b700 1 -- 172.21.15.189:0/707706870 learned_addr learned my addr 172.21.15.189:0/707706870 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.257+0000 7f3b1ed9d700 1 --2- 172.21.15.189:0/707706870 >> v2:172.21.15.192:3300/0 conn(0x7f3b200cedf0 0x7f3b200cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.257+0000 7f3b1e59c700 1 --2- 172.21.15.189:0/707706870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b200d17a0 0x7f3b2006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.257+0000 7f3b1d59a700 1 -- 172.21.15.189:0/707706870 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482931975 0 0) 0x7f3b200cf760 con 0x7f3b20146dd0 2024-04-06T13:28:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.257+0000 7f3b1d59a700 1 -- 172.21.15.189:0/707706870 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b080035b0 con 0x7f3b20146dd0 2024-04-06T13:28:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.257+0000 7f3b1ed9d700 1 -- 172.21.15.189:0/707706870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b200d17a0 msgr2=0x7f3b2006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.257+0000 7f3b1ed9d700 1 --2- 172.21.15.189:0/707706870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b200d17a0 0x7f3b2006b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.257+0000 7f3b1ed9d700 1 -- 172.21.15.189:0/707706870 >> v1:172.21.15.189:6789/0 conn(0x7f3b20146dd0 legacy=0x7f3b2006d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:23.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.257+0000 7f3b1ed9d700 1 -- 172.21.15.189:0/707706870 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2006c380 con 0x7f3b200cedf0 2024-04-06T13:28:23.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.257+0000 7f3b1ed9d700 1 --2- 172.21.15.189:0/707706870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b200cedf0 0x7f3b200cf220 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009d90 tx=0x7f3b1800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0c099132604c194 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:23.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.258+0000 7f3b1d59a700 1 -- 172.21.15.189:0/707706870 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1800e070 con 0x7f3b200cedf0 2024-04-06T13:28:23.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.258+0000 7f3b1d59a700 1 -- 172.21.15.189:0/707706870 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1800bb20 con 0x7f3b200cedf0 2024-04-06T13:28:23.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.258+0000 7f3b1d59a700 1 -- 172.21.15.189:0/707706870 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b18012430 con 0x7f3b200cedf0 2024-04-06T13:28:23.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.258+0000 7f3b24bee700 1 -- 172.21.15.189:0/707706870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b200cedf0 msgr2=0x7f3b200cf220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:23.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.258+0000 7f3b24bee700 1 --2- 172.21.15.189:0/707706870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b200cedf0 0x7f3b200cf220 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009d90 tx=0x7f3b1800b330 comp rx=0 tx=0).stop 2024-04-06T13:28:23.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.258+0000 7f3b24bee700 1 -- 172.21.15.189:0/707706870 shutdown_connections 2024-04-06T13:28:23.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.258+0000 7f3b24bee700 1 --2- 172.21.15.189:0/707706870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b200cedf0 0x7f3b200cf220 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.258+0000 7f3b24bee700 1 --2- 172.21.15.189:0/707706870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b200d17a0 0x7f3b2006b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.258+0000 7f3b24bee700 1 -- 172.21.15.189:0/707706870 >> 172.21.15.189:0/707706870 conn(0x7f3b200b2db0 msgr2=0x7f3b200b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:23.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.258+0000 7f3b24bee700 1 -- 172.21.15.189:0/707706870 shutdown_connections 2024-04-06T13:28:23.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.258+0000 7f3b24bee700 1 -- 172.21.15.189:0/707706870 wait complete. 2024-04-06T13:28:23.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.260+0000 7f3b24bee700 1 Processor -- start 2024-04-06T13:28:23.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.260+0000 7f3b24bee700 1 -- start start 2024-04-06T13:28:23.265 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.260+0000 7f3b24bee700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b200cedf0 0x7f3b2015c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:23.265 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.260+0000 7f3b24bee700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b200d17a0 0x7f3b200c8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:23.265 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.260+0000 7f3b24bee700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b20146dd0 0x7f3b200c89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:23.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.260+0000 7f3b24bee700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b200af890 con 0x7f3b200cedf0 2024-04-06T13:28:23.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.260+0000 7f3b24bee700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b200b06f0 con 0x7f3b200d17a0 2024-04-06T13:28:23.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.260+0000 7f3b24bee700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b200afae0 con 0x7f3b20146dd0 2024-04-06T13:28:23.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.261+0000 7f3b1e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b200cedf0 0x7f3b2015c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:23.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.261+0000 7f3b1e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b200cedf0 0x7f3b2015c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55612/0 (socket says 172.21.15.189:55612) 2024-04-06T13:28:23.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.261+0000 7f3b1e59c700 1 -- 172.21.15.189:0/1654385272 learned_addr learned my addr 172.21.15.189:0/1654385272 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:23.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.261+0000 7f3b1dd9b700 1 --2- 172.21.15.189:0/1654385272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b200d17a0 0x7f3b200c8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:23.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.261+0000 7f3b1ed9d700 1 --2- 172.21.15.189:0/1654385272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b20146dd0 0x7f3b200c89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:23.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.261+0000 7f3b1dd9b700 1 -- 172.21.15.189:0/1654385272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b20146dd0 msgr2=0x7f3b200c89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:23.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.261+0000 7f3b1dd9b700 1 --2- 172.21.15.189:0/1654385272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b20146dd0 0x7f3b200c89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.261+0000 7f3b1dd9b700 1 -- 172.21.15.189:0/1654385272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b200cedf0 msgr2=0x7f3b2015c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:23.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.261+0000 7f3b1dd9b700 1 --2- 172.21.15.189:0/1654385272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b200cedf0 0x7f3b2015c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.261+0000 7f3b1dd9b700 1 -- 172.21.15.189:0/1654385272 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b18009a70 con 0x7f3b200d17a0 2024-04-06T13:28:23.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.261+0000 7f3b1e59c700 1 --2- 172.21.15.189:0/1654385272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b200cedf0 0x7f3b2015c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:23.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.261+0000 7f3b1dd9b700 1 --2- 172.21.15.189:0/1654385272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b200d17a0 0x7f3b200c8480 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400c490 tx=0x7f3b1400c7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:23.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.261+0000 7f3b0effd700 1 -- 172.21.15.189:0/1654385272 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14005d70 con 0x7f3b200d17a0 2024-04-06T13:28:23.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.261+0000 7f3b0effd700 1 -- 172.21.15.189:0/1654385272 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b140070d0 con 0x7f3b200d17a0 2024-04-06T13:28:23.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.261+0000 7f3b24bee700 1 -- 172.21.15.189:0/1654385272 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b200afdc0 con 0x7f3b200d17a0 2024-04-06T13:28:23.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.261+0000 7f3b0effd700 1 -- 172.21.15.189:0/1654385272 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14007440 con 0x7f3b200d17a0 2024-04-06T13:28:23.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.261+0000 7f3b24bee700 1 -- 172.21.15.189:0/1654385272 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b200b0380 con 0x7f3b200d17a0 2024-04-06T13:28:23.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.262+0000 7f3b24bee700 1 -- 172.21.15.189:0/1654385272 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b20074990 con 0x7f3b200d17a0 2024-04-06T13:28:23.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.263+0000 7f3b0effd700 1 -- 172.21.15.189:0/1654385272 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 95) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f3b140075a0 con 0x7f3b200d17a0 2024-04-06T13:28:23.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.263+0000 7f3b0effd700 1 --2- 172.21.15.189:0/1654385272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3b0006adb0 0x7f3b0006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:23.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.263+0000 7f3b1e59c700 1 -- 172.21.15.189:0/1654385272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3b0006adb0 msgr2=0x7f3b0006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:23.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.263+0000 7f3b0effd700 1 -- 172.21.15.189:0/1654385272 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f3b14088a00 con 0x7f3b200d17a0 2024-04-06T13:28:23.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.263+0000 7f3b1e59c700 1 --2- 172.21.15.189:0/1654385272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3b0006adb0 0x7f3b0006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:23.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.268+0000 7f3b0effd700 1 -- 172.21.15.189:0/1654385272 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3b14053890 con 0x7f3b200d17a0 2024-04-06T13:28:23.344 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:23.344+0000 7f02c5aba200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:28:23.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.464+0000 7f3b1e59c700 1 -- 172.21.15.189:0/1654385272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3b0006adb0 msgr2=0x7f3b0006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:23.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.464+0000 7f3b1e59c700 1 --2- 172.21.15.189:0/1654385272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3b0006adb0 0x7f3b0006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:23.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.487+0000 7f3b24bee700 1 -- 172.21.15.189:0/1654385272 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3b200d1e20 con 0x7f3b200d17a0 2024-04-06T13:28:23.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.488+0000 7f3b0effd700 1 -- 172.21.15.189:0/1654385272 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v165) v1 ==== 80+0+466 (secure 0 0 0) 0x7f3b14017080 con 0x7f3b200d17a0 2024-04-06T13:28:23.488 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/3791371660 2024-04-07T12:32:16.629669+0000 2024-04-06T13:28:23.488 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/3889346008 2024-04-07T12:32:16.629669+0000 2024-04-06T13:28:23.488 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/628196195 2024-04-07T12:32:16.629669+0000 2024-04-06T13:28:23.488 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/4023078006 2024-04-07T12:32:16.629669+0000 2024-04-06T13:28:23.488 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6833/47128 2024-04-07T12:32:16.629669+0000 2024-04-06T13:28:23.488 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6832/47128 2024-04-07T12:32:16.629669+0000 2024-04-06T13:28:23.488 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/3100554908 2024-04-07T12:32:16.629669+0000 2024-04-06T13:28:23.488 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/383422220 2024-04-07T12:32:16.629669+0000 2024-04-06T13:28:23.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.490+0000 7f3b24bee700 1 -- 172.21.15.189:0/1654385272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3b0006adb0 msgr2=0x7f3b0006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:23.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.490+0000 7f3b24bee700 1 --2- 172.21.15.189:0/1654385272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3b0006adb0 0x7f3b0006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.490+0000 7f3b24bee700 1 -- 172.21.15.189:0/1654385272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b200d17a0 msgr2=0x7f3b200c8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:23.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.490+0000 7f3b24bee700 1 --2- 172.21.15.189:0/1654385272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b200d17a0 0x7f3b200c8480 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400c490 tx=0x7f3b1400c7a0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.490+0000 7f3b24bee700 1 -- 172.21.15.189:0/1654385272 shutdown_connections 2024-04-06T13:28:23.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.490+0000 7f3b24bee700 1 --2- 172.21.15.189:0/1654385272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b20146dd0 0x7f3b200c89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.490+0000 7f3b24bee700 1 --2- 172.21.15.189:0/1654385272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b200cedf0 0x7f3b2015c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.490+0000 7f3b24bee700 1 --2- 172.21.15.189:0/1654385272 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3b0006adb0 0x7f3b0006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.490+0000 7f3b24bee700 1 --2- 172.21.15.189:0/1654385272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b200d17a0 0x7f3b200c8480 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.490+0000 7f3b24bee700 1 -- 172.21.15.189:0/1654385272 >> 172.21.15.189:0/1654385272 conn(0x7f3b200b2db0 msgr2=0x7f3b2015e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:23.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.490+0000 7f3b24bee700 1 -- 172.21.15.189:0/1654385272 shutdown_connections 2024-04-06T13:28:23.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.490+0000 7f3b24bee700 1 -- 172.21.15.189:0/1654385272 wait complete. 2024-04-06T13:28:23.491 INFO:teuthology.orchestra.run.smithi189.stderr:listed 8 entries 2024-04-06T13:28:23.502 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-06T13:28:23.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:23.503 DEBUG:teuthology.orchestra.run.smithi189:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-06T13:28:23.522 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:23.521+0000 7fca68879200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:28:23.546 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T13:28:23.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:23.546 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-06T13:28:23.604 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:28:23.604 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:28:23.622 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:28:23.622 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:28:23.622 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:28:23.661 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:28:23.661 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:28:23.661 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:28:23.679 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:28:23.679 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:28:23.680 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:23.680+0000 7f02c5aba200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:28:23.689 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:23.689+0000 7fca68879200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:28:23.763 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:23.762+0000 7fca68879200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:28:23.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.805+0000 7f1a62d28700 1 Processor -- start 2024-04-06T13:28:23.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.805+0000 7f1a62d28700 1 -- start start 2024-04-06T13:28:23.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.806+0000 7f1a62d28700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1a5c15d7b0 0x7f1a5c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:23.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.806+0000 7f1a62d28700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1a5c06a730 0x7f1a5c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:23.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.806+0000 7f1a62d28700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c0b3990 con 0x7f1a5c15ddf0 2024-04-06T13:28:23.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.806+0000 7f1a62d28700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5c15e950 con 0x7f1a5c15d7b0 2024-04-06T13:28:23.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.806+0000 7f1a62d28700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5c06b0f0 con 0x7f1a5c06a730 2024-04-06T13:28:23.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.807+0000 7f1a60ac4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1a5c15d7b0 0x7f1a5c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:23.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.807+0000 7f1a60ac4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1a5c15d7b0 0x7f1a5c0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55624/0 (socket says 172.21.15.189:55624) 2024-04-06T13:28:23.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.807+0000 7f1a60ac4700 1 -- 172.21.15.189:0/3579840468 learned_addr learned my addr 172.21.15.189:0/3579840468 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:23.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.807+0000 7f1a5bfff700 1 --2- 172.21.15.189:0/3579840468 >> v2:172.21.15.192:3300/0 conn(0x7f1a5c06a730 0x7f1a5c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:23.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.807+0000 7f1a60ac4700 1 -- 172.21.15.189:0/3579840468 >> v2:172.21.15.192:3300/0 conn(0x7f1a5c06a730 msgr2=0x7f1a5c06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:23.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.807+0000 7f1a60ac4700 1 --2- 172.21.15.189:0/3579840468 >> v2:172.21.15.192:3300/0 conn(0x7f1a5c06a730 0x7f1a5c06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.807+0000 7f1a60ac4700 1 -- 172.21.15.189:0/3579840468 >> v1:172.21.15.192:6789/0 conn(0x7f1a5c15ddf0 legacy=0x7f1a5c15e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:23.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.807+0000 7f1a60ac4700 1 -- 172.21.15.189:0/3579840468 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c0b3060 con 0x7f1a5c15d7b0 2024-04-06T13:28:23.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.807+0000 7f1a60ac4700 1 --2- 172.21.15.189:0/3579840468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5c15d7b0 0x7f1a5c0b3e20 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c0afc00 tx=0x7f1a50009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ec513077db51a5a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:23.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.807+0000 7f1a5b7fe700 1 -- 172.21.15.189:0/3579840468 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5000b870 con 0x7f1a5c15d7b0 2024-04-06T13:28:23.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.807+0000 7f1a5b7fe700 1 -- 172.21.15.189:0/3579840468 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5000b9d0 con 0x7f1a5c15d7b0 2024-04-06T13:28:23.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.807+0000 7f1a5b7fe700 1 -- 172.21.15.189:0/3579840468 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5000bd60 con 0x7f1a5c15d7b0 2024-04-06T13:28:23.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.808+0000 7f1a62d28700 1 -- 172.21.15.189:0/3579840468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5c15d7b0 msgr2=0x7f1a5c0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:23.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.808+0000 7f1a62d28700 1 --2- 172.21.15.189:0/3579840468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5c15d7b0 0x7f1a5c0b3e20 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c0afc00 tx=0x7f1a50009bd0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.808+0000 7f1a62d28700 1 -- 172.21.15.189:0/3579840468 shutdown_connections 2024-04-06T13:28:23.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.808+0000 7f1a62d28700 1 --2- 172.21.15.189:0/3579840468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5c15d7b0 0x7f1a5c0b3e20 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.808+0000 7f1a62d28700 1 --2- 172.21.15.189:0/3579840468 >> v2:172.21.15.192:3300/0 conn(0x7f1a5c06a730 0x7f1a5c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.808+0000 7f1a62d28700 1 -- 172.21.15.189:0/3579840468 >> 172.21.15.189:0/3579840468 conn(0x7f1a5c0d00b0 msgr2=0x7f1a5c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:23.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.808+0000 7f1a62d28700 1 -- 172.21.15.189:0/3579840468 shutdown_connections 2024-04-06T13:28:23.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.808+0000 7f1a62d28700 1 -- 172.21.15.189:0/3579840468 wait complete. 2024-04-06T13:28:23.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.809+0000 7f1a62d28700 1 Processor -- start 2024-04-06T13:28:23.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.809+0000 7f1a62d28700 1 -- start start 2024-04-06T13:28:23.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.810+0000 7f1a62d28700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a5c06a730 0x7f1a5c0c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:23.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.810+0000 7f1a62d28700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a5c15d7b0 0x7f1a5c1f2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:23.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.810+0000 7f1a62d28700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5c15ddf0 0x7f1a5c15b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:23.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.810+0000 7f1a62d28700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0c55c0 con 0x7f1a5c15ddf0 2024-04-06T13:28:23.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.810+0000 7f1a62d28700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c149550 con 0x7f1a5c06a730 2024-04-06T13:28:23.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.810+0000 7f1a62d28700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c149830 con 0x7f1a5c15d7b0 2024-04-06T13:28:23.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.810+0000 7f1a612c5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5c15ddf0 0x7f1a5c15b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:23.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.810+0000 7f1a612c5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5c15ddf0 0x7f1a5c15b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55634/0 (socket says 172.21.15.189:55634) 2024-04-06T13:28:23.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.810+0000 7f1a612c5700 1 -- 172.21.15.189:0/473471093 learned_addr learned my addr 172.21.15.189:0/473471093 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:23.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.810+0000 7f1a60ac4700 1 --2- 172.21.15.189:0/473471093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a5c06a730 0x7f1a5c0c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:23.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.810+0000 7f1a5bfff700 1 --2- 172.21.15.189:0/473471093 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a5c15d7b0 0x7f1a5c1f2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:23.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.810+0000 7f1a612c5700 1 -- 172.21.15.189:0/473471093 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a5c15d7b0 msgr2=0x7f1a5c1f2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:23.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.810+0000 7f1a612c5700 1 --2- 172.21.15.189:0/473471093 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a5c15d7b0 0x7f1a5c1f2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.810+0000 7f1a612c5700 1 -- 172.21.15.189:0/473471093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a5c06a730 msgr2=0x7f1a5c0c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:23.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.810+0000 7f1a612c5700 1 --2- 172.21.15.189:0/473471093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a5c06a730 0x7f1a5c0c4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:23.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.810+0000 7f1a612c5700 1 -- 172.21.15.189:0/473471093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a50009550 con 0x7f1a5c15ddf0 2024-04-06T13:28:23.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.810+0000 7f1a612c5700 1 --2- 172.21.15.189:0/473471093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5c15ddf0 0x7f1a5c15b530 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400cb50 tx=0x7f1a5400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:23.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.811+0000 7f1a597fa700 1 -- 172.21.15.189:0/473471093 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54003c00 con 0x7f1a5c15ddf0 2024-04-06T13:28:23.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.811+0000 7f1a597fa700 1 -- 172.21.15.189:0/473471093 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a54003d60 con 0x7f1a5c15ddf0 2024-04-06T13:28:23.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.811+0000 7f1a62d28700 1 -- 172.21.15.189:0/473471093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5c149b10 con 0x7f1a5c15ddf0 2024-04-06T13:28:23.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.811+0000 7f1a597fa700 1 -- 172.21.15.189:0/473471093 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5400aa20 con 0x7f1a5c15ddf0 2024-04-06T13:28:23.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.811+0000 7f1a62d28700 1 -- 172.21.15.189:0/473471093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5c1f28c0 con 0x7f1a5c15ddf0 2024-04-06T13:28:23.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.812+0000 7f1a62d28700 1 -- 172.21.15.189:0/473471093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5c14a030 con 0x7f1a5c15ddf0 2024-04-06T13:28:23.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.813+0000 7f1a597fa700 1 -- 172.21.15.189:0/473471093 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 95) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f1a5400ab80 con 0x7f1a5c15ddf0 2024-04-06T13:28:23.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.813+0000 7f1a597fa700 1 --2- 172.21.15.189:0/473471093 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a4006adb0 0x7f1a4006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:23.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.813+0000 7f1a60ac4700 1 -- 172.21.15.189:0/473471093 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a4006adb0 msgr2=0x7f1a4006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:23.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.813+0000 7f1a597fa700 1 -- 172.21.15.189:0/473471093 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f1a5408a0c0 con 0x7f1a5c15ddf0 2024-04-06T13:28:23.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.813+0000 7f1a60ac4700 1 --2- 172.21.15.189:0/473471093 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a4006adb0 0x7f1a4006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:23.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:23.817+0000 7f1a597fa700 1 -- 172.21.15.189:0/473471093 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1a54058980 con 0x7f1a5c15ddf0 2024-04-06T13:28:23.852 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:23.852+0000 7f02c5aba200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:28:23.928 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:23.929+0000 7f02c5aba200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:28:24.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.014+0000 7f1a60ac4700 1 -- 172.21.15.189:0/473471093 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a4006adb0 msgr2=0x7f1a4006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:24.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.014+0000 7f1a60ac4700 1 --2- 172.21.15.189:0/473471093 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a4006adb0 0x7f1a4006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:24.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.065+0000 7f1a62d28700 1 -- 172.21.15.189:0/473471093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a5c14a030 con 0x7f1a5c15ddf0 2024-04-06T13:28:24.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.066+0000 7f1a597fa700 1 -- 172.21.15.189:0/473471093 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4748 (secure 0 0 0) 0x7f1a5c14a030 con 0x7f1a5c15ddf0 2024-04-06T13:28:24.066 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:24.066 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":70,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3697447910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3697447910},{"type":"v1","addr":"172.21.15.189:6837","nonce":3697447910}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":7469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/3633157989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3633157989},{"type":"v1","addr":"172.21.15.192:6834","nonce":3633157989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2721135386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2721135386},{"type":"v1","addr":"172.21.15.189:6835","nonce":2721135386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:32:03.067806+0000","modified":"2024-04-06T12:32:06.080510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9144":{"gid":9144,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/3068270636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":3068270636},{"type":"v1","addr":"172.21.15.192:6835","nonce":3068270636}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-06T13:28:24.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.069+0000 7f1a62d28700 1 -- 172.21.15.189:0/473471093 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a4006adb0 msgr2=0x7f1a4006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:24.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.069+0000 7f1a62d28700 1 --2- 172.21.15.189:0/473471093 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a4006adb0 0x7f1a4006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.069+0000 7f1a62d28700 1 -- 172.21.15.189:0/473471093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5c15ddf0 msgr2=0x7f1a5c15b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:24.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.069+0000 7f1a62d28700 1 --2- 172.21.15.189:0/473471093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5c15ddf0 0x7f1a5c15b530 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400cb50 tx=0x7f1a5400ce60 comp rx=0 tx=0).stop 2024-04-06T13:28:24.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.069+0000 7f1a62d28700 1 -- 172.21.15.189:0/473471093 shutdown_connections 2024-04-06T13:28:24.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.069+0000 7f1a62d28700 1 --2- 172.21.15.189:0/473471093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5c15ddf0 0x7f1a5c15b530 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.069+0000 7f1a62d28700 1 --2- 172.21.15.189:0/473471093 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1a4006adb0 0x7f1a4006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.069+0000 7f1a62d28700 1 --2- 172.21.15.189:0/473471093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a5c06a730 0x7f1a5c0c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.069+0000 7f1a62d28700 1 --2- 172.21.15.189:0/473471093 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a5c15d7b0 0x7f1a5c1f2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.069+0000 7f1a62d28700 1 -- 172.21.15.189:0/473471093 >> 172.21.15.189:0/473471093 conn(0x7f1a5c0d00b0 msgr2=0x7f1a5c072370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:24.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.069+0000 7f1a62d28700 1 -- 172.21.15.189:0/473471093 shutdown_connections 2024-04-06T13:28:24.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.069+0000 7f1a62d28700 1 -- 172.21.15.189:0/473471093 wait complete. 2024-04-06T13:28:24.070 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 70 2024-04-06T13:28:24.082 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:28:24.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.210+0000 7f740e19d700 1 Processor -- start 2024-04-06T13:28:24.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.210+0000 7f740e19d700 1 -- start start 2024-04-06T13:28:24.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.211+0000 7f740e19d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f740806ae40 0x7f74080b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:24.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.211+0000 7f740e19d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74080b3990 con 0x7f740815d7b0 2024-04-06T13:28:24.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.211+0000 7f740e19d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74080b3060 con 0x7f740815ddf0 2024-04-06T13:28:24.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.211+0000 7f740e19d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f740815e950 con 0x7f740806ae40 2024-04-06T13:28:24.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.211+0000 7f73fffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f740806ae40 0x7f74080b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:24.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.211+0000 7f74077fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f740815d7b0 0x7f740806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51446/0 (socket says 172.21.15.189:51446) 2024-04-06T13:28:24.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.211+0000 7f74077fe700 1 -- 172.21.15.189:0/1113814340 learned_addr learned my addr 172.21.15.189:0/1113814340 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:24.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.212+0000 7f73fffff700 1 -- 172.21.15.189:0/1113814340 >> v1:172.21.15.192:6789/0 conn(0x7f740815ddf0 legacy=0x7f740815e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:24.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.212+0000 7f73fffff700 1 -- 172.21.15.189:0/1113814340 >> v1:172.21.15.189:6789/0 conn(0x7f740815d7b0 legacy=0x7f740806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:24.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.212+0000 7f73fffff700 1 -- 172.21.15.189:0/1113814340 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74080b3340 con 0x7f740806ae40 2024-04-06T13:28:24.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.212+0000 7f7406ffd700 1 -- 172.21.15.189:0/1113814340 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034978346 0 0) 0x7f74080b3990 con 0x7f740815d7b0 2024-04-06T13:28:24.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.212+0000 7f73fffff700 1 --2- 172.21.15.189:0/1113814340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f740806ae40 0x7f74080b3e20 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f73f0009840 tx=0x7f73f0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf2785907afb66a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:24.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.212+0000 7f7406ffd700 1 -- 172.21.15.189:0/1113814340 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f0011830 con 0x7f740806ae40 2024-04-06T13:28:24.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.212+0000 7f7406ffd700 1 -- 172.21.15.189:0/1113814340 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f0011990 con 0x7f740806ae40 2024-04-06T13:28:24.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.212+0000 7f7406ffd700 1 -- 172.21.15.189:0/1113814340 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f0011d20 con 0x7f740806ae40 2024-04-06T13:28:24.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.213+0000 7f740e19d700 1 -- 172.21.15.189:0/1113814340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f740806ae40 msgr2=0x7f74080b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:24.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.213+0000 7f740e19d700 1 --2- 172.21.15.189:0/1113814340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f740806ae40 0x7f74080b3e20 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f73f0009840 tx=0x7f73f0009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:24.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.213+0000 7f740e19d700 1 -- 172.21.15.189:0/1113814340 shutdown_connections 2024-04-06T13:28:24.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.213+0000 7f740e19d700 1 --2- 172.21.15.189:0/1113814340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f740806ae40 0x7f74080b3e20 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.213+0000 7f740e19d700 1 -- 172.21.15.189:0/1113814340 >> 172.21.15.189:0/1113814340 conn(0x7f74080d00b0 msgr2=0x7f74080d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:24.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.213+0000 7f740e19d700 1 -- 172.21.15.189:0/1113814340 shutdown_connections 2024-04-06T13:28:24.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.213+0000 7f740e19d700 1 -- 172.21.15.189:0/1113814340 wait complete. 2024-04-06T13:28:24.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.214+0000 7f740e19d700 1 Processor -- start 2024-04-06T13:28:24.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.214+0000 7f740e19d700 1 -- start start 2024-04-06T13:28:24.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.215+0000 7f740e19d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f740806ae40 0x7f740815c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:24.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.215+0000 7f740e19d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f740815d7b0 0x7f7408156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:24.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.215+0000 7f740e19d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f740815ddf0 0x7f74080aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:24.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.215+0000 7f740e19d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74080b1470 con 0x7f740815d7b0 2024-04-06T13:28:24.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.215+0000 7f740e19d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7408157530 con 0x7f740806ae40 2024-04-06T13:28:24.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.215+0000 7f740e19d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74080b0ad0 con 0x7f740815ddf0 2024-04-06T13:28:24.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.215+0000 7f73fffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f740815d7b0 0x7f7408156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:24.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.215+0000 7f73fffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f740815d7b0 0x7f7408156ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55644/0 (socket says 172.21.15.189:55644) 2024-04-06T13:28:24.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.215+0000 7f73fffff700 1 -- 172.21.15.189:0/946396876 learned_addr learned my addr 172.21.15.189:0/946396876 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:24.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.215+0000 7f74077fe700 1 --2- 172.21.15.189:0/946396876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f740806ae40 0x7f740815c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:24.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.215+0000 7f7407fff700 1 --2- 172.21.15.189:0/946396876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f740815ddf0 0x7f74080aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:24.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.215+0000 7f73fffff700 1 -- 172.21.15.189:0/946396876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f740815ddf0 msgr2=0x7f74080aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:24.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.215+0000 7f73fffff700 1 --2- 172.21.15.189:0/946396876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f740815ddf0 0x7f74080aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.215+0000 7f73fffff700 1 -- 172.21.15.189:0/946396876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f740806ae40 msgr2=0x7f740815c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:24.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.215+0000 7f73fffff700 1 --2- 172.21.15.189:0/946396876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f740806ae40 0x7f740815c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.215+0000 7f73fffff700 1 -- 172.21.15.189:0/946396876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f00094a0 con 0x7f740815d7b0 2024-04-06T13:28:24.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.215+0000 7f73fffff700 1 --2- 172.21.15.189:0/946396876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f740815d7b0 0x7f7408156ff0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f73f00090b0 tx=0x7f73f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:24.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.215+0000 7f7404ff9700 1 -- 172.21.15.189:0/946396876 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f001a7e0 con 0x7f740815d7b0 2024-04-06T13:28:24.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.216+0000 7f7404ff9700 1 -- 172.21.15.189:0/946396876 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f001a940 con 0x7f740815d7b0 2024-04-06T13:28:24.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.216+0000 7f7404ff9700 1 -- 172.21.15.189:0/946396876 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f001acb0 con 0x7f740815d7b0 2024-04-06T13:28:24.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.216+0000 7f740e19d700 1 -- 172.21.15.189:0/946396876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74080b0db0 con 0x7f740815d7b0 2024-04-06T13:28:24.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.216+0000 7f740e19d700 1 -- 172.21.15.189:0/946396876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74080af0c0 con 0x7f740815d7b0 2024-04-06T13:28:24.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.217+0000 7f740e19d700 1 -- 172.21.15.189:0/946396876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74080b15b0 con 0x7f740815d7b0 2024-04-06T13:28:24.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.217+0000 7f7404ff9700 1 -- 172.21.15.189:0/946396876 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 95) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f73f0022070 con 0x7f740815d7b0 2024-04-06T13:28:24.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.218+0000 7f7404ff9700 1 --2- 172.21.15.189:0/946396876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f73e806ade0 0x7f73e806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:24.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.218+0000 7f74077fe700 1 -- 172.21.15.189:0/946396876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f73e806ade0 msgr2=0x7f73e806d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:24.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.218+0000 7f7404ff9700 1 -- 172.21.15.189:0/946396876 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f73f008a490 con 0x7f740815d7b0 2024-04-06T13:28:24.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.218+0000 7f74077fe700 1 --2- 172.21.15.189:0/946396876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f73e806ade0 0x7f73e806d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:24.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.222+0000 7f7404ff9700 1 -- 172.21.15.189:0/946396876 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f74080b15b0 con 0x7f740815d7b0 2024-04-06T13:28:24.375 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:24.374+0000 7fca68879200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:28:24.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.419+0000 7f74077fe700 1 -- 172.21.15.189:0/946396876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f73e806ade0 msgr2=0x7f73e806d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:24.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.419+0000 7f74077fe700 1 --2- 172.21.15.189:0/946396876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f73e806ade0 0x7f73e806d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:24.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.428+0000 7f740e19d700 1 -- 172.21.15.189:0/946396876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74080b15b0 con 0x7f740815d7b0 2024-04-06T13:28:24.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.429+0000 7f7404ff9700 1 -- 172.21.15.189:0/946396876 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v165) v1 ==== 74+0+16976 (secure 0 0 0) 0x7f74080b15b0 con 0x7f740815d7b0 2024-04-06T13:28:24.429 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:24.429 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":165,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:28:21.351197+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:32:00.826039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-04-06T12:32:02.039801+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"165","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":112,"snap_epoch":165,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/3791371660":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/3889346008":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/628196195":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/4023078006":"2024-04-07T12:32:16.629669+0000","172.21.15.189:6833/47128":"2024-04-07T12:32:16.629669+0000","172.21.15.189:6832/47128":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/3100554908":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/383422220":"2024-04-07T12:32:16.629669+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":54},{"begin":59,"length":54}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":47,"length":10},{"begin":103,"length":10}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:28:24.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.431+0000 7f740e19d700 1 -- 172.21.15.189:0/946396876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f73e806ade0 msgr2=0x7f73e806d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:24.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.431+0000 7f740e19d700 1 --2- 172.21.15.189:0/946396876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f73e806ade0 0x7f73e806d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.431+0000 7f740e19d700 1 -- 172.21.15.189:0/946396876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f740815d7b0 msgr2=0x7f7408156ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:24.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.431+0000 7f740e19d700 1 --2- 172.21.15.189:0/946396876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f740815d7b0 0x7f7408156ff0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f73f00090b0 tx=0x7f73f0004750 comp rx=0 tx=0).stop 2024-04-06T13:28:24.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.431+0000 7f740e19d700 1 -- 172.21.15.189:0/946396876 shutdown_connections 2024-04-06T13:28:24.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.431+0000 7f740e19d700 1 --2- 172.21.15.189:0/946396876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f740815ddf0 0x7f74080aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.431+0000 7f740e19d700 1 --2- 172.21.15.189:0/946396876 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f73e806ade0 0x7f73e806d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.431+0000 7f740e19d700 1 --2- 172.21.15.189:0/946396876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f740806ae40 0x7f740815c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.431+0000 7f740e19d700 1 --2- 172.21.15.189:0/946396876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f740815d7b0 0x7f7408156ff0 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.431+0000 7f740e19d700 1 -- 172.21.15.189:0/946396876 >> 172.21.15.189:0/946396876 conn(0x7f74080d00b0 msgr2=0x7f740815e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:24.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.431+0000 7f740e19d700 1 -- 172.21.15.189:0/946396876 shutdown_connections 2024-04-06T13:28:24.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.431+0000 7f740e19d700 1 -- 172.21.15.189:0/946396876 wait complete. 2024-04-06T13:28:24.444 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T12:32:00.826039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '157', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T12:32:02.039801+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '165', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 112, 'snap_epoch': 165, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:28:24.445 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:28:24.454 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:24.453+0000 7fca68879200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:28:24.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.574+0000 7f87c1e3c700 1 Processor -- start 2024-04-06T13:28:24.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.574+0000 7f87c1e3c700 1 -- start start 2024-04-06T13:28:24.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.575+0000 7f87c1e3c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f87bc0d11b0 0x7f87bc0d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:24.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.575+0000 7f87c1e3c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87bc0cb870 0x7f87bc15c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:24.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.575+0000 7f87c1e3c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87bc0d1ad0 con 0x7f87bc0cfa00 2024-04-06T13:28:24.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.575+0000 7f87c1e3c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87bc0ccdc0 con 0x7f87bc0d11b0 2024-04-06T13:28:24.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.575+0000 7f87c1e3c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc0d0b50 con 0x7f87bc0cb870 2024-04-06T13:28:24.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.575+0000 7f87bbfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f87bc0cfa00 0x7f87bc0cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51448/0 (socket says 172.21.15.189:51448) 2024-04-06T13:28:24.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.575+0000 7f87bbfff700 1 -- 172.21.15.189:0/1757258880 learned_addr learned my addr 172.21.15.189:0/1757258880 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:24.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.575+0000 7f87bb7fe700 1 --2- 172.21.15.189:0/1757258880 >> v2:172.21.15.192:3300/0 conn(0x7f87bc0d11b0 0x7f87bc0d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:24.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.575+0000 7f87baffd700 1 --2- 172.21.15.189:0/1757258880 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87bc0cb870 0x7f87bc15c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:24.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.575+0000 7f87ba7fc700 1 -- 172.21.15.189:0/1757258880 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555784771 0 0) 0x7f87bc0d1ad0 con 0x7f87bc0cfa00 2024-04-06T13:28:24.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.575+0000 7f87ba7fc700 1 -- 172.21.15.189:0/1757258880 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87a40035b0 con 0x7f87bc0cfa00 2024-04-06T13:28:24.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.575+0000 7f87baffd700 1 -- 172.21.15.189:0/1757258880 >> v2:172.21.15.192:3300/0 conn(0x7f87bc0d11b0 msgr2=0x7f87bc0d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:24.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.575+0000 7f87baffd700 1 --2- 172.21.15.189:0/1757258880 >> v2:172.21.15.192:3300/0 conn(0x7f87bc0d11b0 0x7f87bc0d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.575+0000 7f87baffd700 1 -- 172.21.15.189:0/1757258880 >> v1:172.21.15.189:6789/0 conn(0x7f87bc0cfa00 legacy=0x7f87bc0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:24.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.575+0000 7f87baffd700 1 -- 172.21.15.189:0/1757258880 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87bc0cbd80 con 0x7f87bc0cb870 2024-04-06T13:28:24.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.576+0000 7f87bb7fe700 1 --2- 172.21.15.189:0/1757258880 >> v2:172.21.15.192:3300/0 conn(0x7f87bc0d11b0 0x7f87bc0d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:24.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.576+0000 7f87baffd700 1 --2- 172.21.15.189:0/1757258880 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87bc0cb870 0x7f87bc15c280 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f87b0009840 tx=0x7f87b0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79cbb686f34b9d33 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:24.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.576+0000 7f87ba7fc700 1 -- 172.21.15.189:0/1757258880 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b0011830 con 0x7f87bc0cb870 2024-04-06T13:28:24.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.576+0000 7f87ba7fc700 1 -- 172.21.15.189:0/1757258880 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b0011990 con 0x7f87bc0cb870 2024-04-06T13:28:24.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.576+0000 7f87ba7fc700 1 -- 172.21.15.189:0/1757258880 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b0011d20 con 0x7f87bc0cb870 2024-04-06T13:28:24.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.576+0000 7f87c1e3c700 1 -- 172.21.15.189:0/1757258880 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87bc0cb870 msgr2=0x7f87bc15c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:24.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.576+0000 7f87c1e3c700 1 --2- 172.21.15.189:0/1757258880 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87bc0cb870 0x7f87bc15c280 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f87b0009840 tx=0x7f87b0009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:24.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.576+0000 7f87c1e3c700 1 -- 172.21.15.189:0/1757258880 shutdown_connections 2024-04-06T13:28:24.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.577+0000 7f87c1e3c700 1 --2- 172.21.15.189:0/1757258880 >> v2:172.21.15.192:3300/0 conn(0x7f87bc0d11b0 0x7f87bc0d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.577+0000 7f87c1e3c700 1 --2- 172.21.15.189:0/1757258880 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87bc0cb870 0x7f87bc15c280 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.577+0000 7f87c1e3c700 1 -- 172.21.15.189:0/1757258880 >> 172.21.15.189:0/1757258880 conn(0x7f87bc0b2db0 msgr2=0x7f87bc0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:24.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.577+0000 7f87c1e3c700 1 -- 172.21.15.189:0/1757258880 shutdown_connections 2024-04-06T13:28:24.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.577+0000 7f87c1e3c700 1 -- 172.21.15.189:0/1757258880 wait complete. 2024-04-06T13:28:24.579 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:24.578+0000 7fca68879200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:28:24.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.578+0000 7f87c1e3c700 1 Processor -- start 2024-04-06T13:28:24.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.578+0000 7f87c1e3c700 1 -- start start 2024-04-06T13:28:24.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87c1e3c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87bc0cb870 0x7f87bc15e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:24.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87c1e3c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87bc0cfa00 0x7f87bc154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:24.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87c1e3c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87bc0d11b0 0x7f87bc0b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:24.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87c1e3c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc15ead0 con 0x7f87bc0cfa00 2024-04-06T13:28:24.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87c1e3c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc1557d0 con 0x7f87bc0cb870 2024-04-06T13:28:24.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87c1e3c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc154f30 con 0x7f87bc0d11b0 2024-04-06T13:28:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87baffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87bc0cfa00 0x7f87bc154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87bb7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87bc0cb870 0x7f87bc15e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87bbfff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87bc0d11b0 0x7f87bc0b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87baffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87bc0cfa00 0x7f87bc154470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55656/0 (socket says 172.21.15.189:55656) 2024-04-06T13:28:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87baffd700 1 -- 172.21.15.189:0/4012365745 learned_addr learned my addr 172.21.15.189:0/4012365745 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87baffd700 1 -- 172.21.15.189:0/4012365745 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87bc0d11b0 msgr2=0x7f87bc0b62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87baffd700 1 --2- 172.21.15.189:0/4012365745 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87bc0d11b0 0x7f87bc0b62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87baffd700 1 -- 172.21.15.189:0/4012365745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87bc0cb870 msgr2=0x7f87bc15e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87baffd700 1 --2- 172.21.15.189:0/4012365745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87bc0cb870 0x7f87bc15e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87baffd700 1 -- 172.21.15.189:0/4012365745 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ac0094a0 con 0x7f87bc0cfa00 2024-04-06T13:28:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87bb7fe700 1 --2- 172.21.15.189:0/4012365745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87bc0cb870 0x7f87bc15e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87bbfff700 1 --2- 172.21.15.189:0/4012365745 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87bc0d11b0 0x7f87bc0b62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87baffd700 1 --2- 172.21.15.189:0/4012365745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87bc0cfa00 0x7f87bc154470 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f87b00090b0 tx=0x7f87b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87a3fff700 1 -- 172.21.15.189:0/4012365745 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b0004590 con 0x7f87bc0cfa00 2024-04-06T13:28:24.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87a3fff700 1 -- 172.21.15.189:0/4012365745 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b001e070 con 0x7f87bc0cfa00 2024-04-06T13:28:24.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87c1e3c700 1 -- 172.21.15.189:0/4012365745 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b00094a0 con 0x7f87bc0cfa00 2024-04-06T13:28:24.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.579+0000 7f87a3fff700 1 -- 172.21.15.189:0/4012365745 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b00038c0 con 0x7f87bc0cfa00 2024-04-06T13:28:24.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.580+0000 7f87c1e3c700 1 -- 172.21.15.189:0/4012365745 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87bc1554f0 con 0x7f87bc0cfa00 2024-04-06T13:28:24.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.581+0000 7f87c1e3c700 1 -- 172.21.15.189:0/4012365745 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87bc074990 con 0x7f87bc0cfa00 2024-04-06T13:28:24.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.581+0000 7f87a3fff700 1 -- 172.21.15.189:0/4012365745 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 95) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f87b0003a20 con 0x7f87bc0cfa00 2024-04-06T13:28:24.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.581+0000 7f87a3fff700 1 --2- 172.21.15.189:0/4012365745 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f87a806adb0 0x7f87a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:24.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.581+0000 7f87bb7fe700 1 -- 172.21.15.189:0/4012365745 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f87a806adb0 msgr2=0x7f87a806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:24.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.581+0000 7f87a3fff700 1 -- 172.21.15.189:0/4012365745 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f87b008a530 con 0x7f87bc0cfa00 2024-04-06T13:28:24.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.582+0000 7f87bb7fe700 1 --2- 172.21.15.189:0/4012365745 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f87a806adb0 0x7f87a806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:24.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.586+0000 7f87a3fff700 1 -- 172.21.15.189:0/4012365745 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f87b0055320 con 0x7f87bc0cfa00 2024-04-06T13:28:24.589 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:24.589+0000 7f02c5aba200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:28:24.671 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:24.672+0000 7f02c5aba200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:28:24.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.783+0000 7f87bb7fe700 1 -- 172.21.15.189:0/4012365745 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f87a806adb0 msgr2=0x7f87a806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:24.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.783+0000 7f87bb7fe700 1 --2- 172.21.15.189:0/4012365745 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f87a806adb0 0x7f87a806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:24.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.786+0000 7f87c1e3c700 1 -- 172.21.15.189:0/4012365745 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f87bc0b0a00 con 0x7f87bc0cfa00 2024-04-06T13:28:24.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.787+0000 7f87a3fff700 1 -- 172.21.15.189:0/4012365745 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v165) v1 ==== 93+0+10 (secure 0 0 0) 0x7f87b0058940 con 0x7f87bc0cfa00 2024-04-06T13:28:24.786 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:28:24.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.790+0000 7f87c1e3c700 1 -- 172.21.15.189:0/4012365745 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f87a806adb0 msgr2=0x7f87a806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:24.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.790+0000 7f87c1e3c700 1 --2- 172.21.15.189:0/4012365745 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f87a806adb0 0x7f87a806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.790+0000 7f87c1e3c700 1 -- 172.21.15.189:0/4012365745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87bc0cfa00 msgr2=0x7f87bc154470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:24.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.790+0000 7f87c1e3c700 1 --2- 172.21.15.189:0/4012365745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87bc0cfa00 0x7f87bc154470 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f87b00090b0 tx=0x7f87b0004750 comp rx=0 tx=0).stop 2024-04-06T13:28:24.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.790+0000 7f87c1e3c700 1 -- 172.21.15.189:0/4012365745 shutdown_connections 2024-04-06T13:28:24.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.790+0000 7f87c1e3c700 1 --2- 172.21.15.189:0/4012365745 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87bc0d11b0 0x7f87bc0b62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.790+0000 7f87c1e3c700 1 --2- 172.21.15.189:0/4012365745 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f87a806adb0 0x7f87a806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.790+0000 7f87c1e3c700 1 --2- 172.21.15.189:0/4012365745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87bc0cb870 0x7f87bc15e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.790+0000 7f87c1e3c700 1 --2- 172.21.15.189:0/4012365745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87bc0cfa00 0x7f87bc154470 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.790+0000 7f87c1e3c700 1 -- 172.21.15.189:0/4012365745 >> 172.21.15.189:0/4012365745 conn(0x7f87bc0b2db0 msgr2=0x7f87bc0ca300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:24.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.790+0000 7f87c1e3c700 1 -- 172.21.15.189:0/4012365745 shutdown_connections 2024-04-06T13:28:24.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.790+0000 7f87c1e3c700 1 -- 172.21.15.189:0/4012365745 wait complete. 2024-04-06T13:28:24.803 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T13:28:24.804 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:24.805+0000 7f02c5aba200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:28:24.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.932+0000 7fe694f4e700 1 Processor -- start 2024-04-06T13:28:24.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.932+0000 7fe694f4e700 1 -- start start 2024-04-06T13:28:24.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.934+0000 7fe694f4e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6900d11b0 0x7fe6900d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:24.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.934+0000 7fe694f4e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe6900cb870 0x7fe69015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:24.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.934+0000 7fe694f4e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6900d1ad0 con 0x7fe6900cfa00 2024-04-06T13:28:24.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.934+0000 7fe694f4e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6900ccdc0 con 0x7fe6900cb870 2024-04-06T13:28:24.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.934+0000 7fe694f4e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6900d0b50 con 0x7fe6900d11b0 2024-04-06T13:28:24.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.934+0000 7fe68ed9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe6900cfa00 0x7fe6900cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51462/0 (socket says 172.21.15.189:51462) 2024-04-06T13:28:24.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.934+0000 7fe68ed9d700 1 -- 172.21.15.189:0/4155131254 learned_addr learned my addr 172.21.15.189:0/4155131254 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:24.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.934+0000 7fe68dd9b700 1 --2- 172.21.15.189:0/4155131254 >> v2:172.21.15.192:3300/0 conn(0x7fe6900cb870 0x7fe69015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:24.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.934+0000 7fe68e59c700 1 --2- 172.21.15.189:0/4155131254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6900d11b0 0x7fe6900d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:24.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.934+0000 7fe68d59a700 1 -- 172.21.15.189:0/4155131254 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440278822 0 0) 0x7fe6900d1ad0 con 0x7fe6900cfa00 2024-04-06T13:28:24.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.934+0000 7fe68d59a700 1 -- 172.21.15.189:0/4155131254 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6780035b0 con 0x7fe6900cfa00 2024-04-06T13:28:24.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.934+0000 7fe68dd9b700 1 -- 172.21.15.189:0/4155131254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6900d11b0 msgr2=0x7fe6900d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:24.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.934+0000 7fe68dd9b700 1 --2- 172.21.15.189:0/4155131254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6900d11b0 0x7fe6900d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.935+0000 7fe68dd9b700 1 -- 172.21.15.189:0/4155131254 >> v1:172.21.15.189:6789/0 conn(0x7fe6900cfa00 legacy=0x7fe6900cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:24.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.935+0000 7fe68dd9b700 1 -- 172.21.15.189:0/4155131254 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6900cbd80 con 0x7fe6900cb870 2024-04-06T13:28:24.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.935+0000 7fe68e59c700 1 --2- 172.21.15.189:0/4155131254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6900d11b0 0x7fe6900d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:24.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.935+0000 7fe68dd9b700 1 --2- 172.21.15.189:0/4155131254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6900cb870 0x7fe69015c280 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe684009840 tx=0x7fe684009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=712fb0e09f4dbebd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:24.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.935+0000 7fe68d59a700 1 -- 172.21.15.189:0/4155131254 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe684011830 con 0x7fe6900cb870 2024-04-06T13:28:24.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.935+0000 7fe68d59a700 1 -- 172.21.15.189:0/4155131254 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe684011990 con 0x7fe6900cb870 2024-04-06T13:28:24.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.935+0000 7fe68d59a700 1 -- 172.21.15.189:0/4155131254 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe684011d20 con 0x7fe6900cb870 2024-04-06T13:28:24.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.936+0000 7fe694f4e700 1 -- 172.21.15.189:0/4155131254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6900cb870 msgr2=0x7fe69015c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:24.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.936+0000 7fe694f4e700 1 --2- 172.21.15.189:0/4155131254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6900cb870 0x7fe69015c280 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe684009840 tx=0x7fe684009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:24.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.936+0000 7fe694f4e700 1 -- 172.21.15.189:0/4155131254 shutdown_connections 2024-04-06T13:28:24.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.936+0000 7fe694f4e700 1 --2- 172.21.15.189:0/4155131254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6900d11b0 0x7fe6900d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.936+0000 7fe694f4e700 1 --2- 172.21.15.189:0/4155131254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6900cb870 0x7fe69015c280 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.936+0000 7fe694f4e700 1 -- 172.21.15.189:0/4155131254 >> 172.21.15.189:0/4155131254 conn(0x7fe6900b2db0 msgr2=0x7fe6900b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:24.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.936+0000 7fe694f4e700 1 -- 172.21.15.189:0/4155131254 shutdown_connections 2024-04-06T13:28:24.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.936+0000 7fe694f4e700 1 -- 172.21.15.189:0/4155131254 wait complete. 2024-04-06T13:28:24.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.937+0000 7fe694f4e700 1 Processor -- start 2024-04-06T13:28:24.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.938+0000 7fe694f4e700 1 -- start start 2024-04-06T13:28:24.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.938+0000 7fe694f4e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6900cb870 0x7fe6900c7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:24.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.938+0000 7fe694f4e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6900cfa00 0x7fe690158360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:24.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.938+0000 7fe694f4e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6900d11b0 0x7fe6901588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:24.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.938+0000 7fe694f4e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69015c280 con 0x7fe6900cb870 2024-04-06T13:28:24.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.938+0000 7fe694f4e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69015d0e0 con 0x7fe6900d11b0 2024-04-06T13:28:24.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.938+0000 7fe694f4e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe690158de0 con 0x7fe6900cfa00 2024-04-06T13:28:24.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.939+0000 7fe68e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6900cb870 0x7fe6900c7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:24.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.939+0000 7fe68e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6900cb870 0x7fe6900c7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55658/0 (socket says 172.21.15.189:55658) 2024-04-06T13:28:24.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.939+0000 7fe68e59c700 1 -- 172.21.15.189:0/3767258865 learned_addr learned my addr 172.21.15.189:0/3767258865 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:24.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.939+0000 7fe68dd9b700 1 --2- 172.21.15.189:0/3767258865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6900cfa00 0x7fe690158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:24.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.939+0000 7fe68ed9d700 1 --2- 172.21.15.189:0/3767258865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6900d11b0 0x7fe6901588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:24.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.939+0000 7fe68e59c700 1 -- 172.21.15.189:0/3767258865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6900cfa00 msgr2=0x7fe690158360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:24.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.939+0000 7fe68e59c700 1 --2- 172.21.15.189:0/3767258865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6900cfa00 0x7fe690158360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.939+0000 7fe68e59c700 1 -- 172.21.15.189:0/3767258865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6900d11b0 msgr2=0x7fe6901588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:24.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.939+0000 7fe68e59c700 1 --2- 172.21.15.189:0/3767258865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6900d11b0 0x7fe6901588a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:24.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.939+0000 7fe68e59c700 1 -- 172.21.15.189:0/3767258865 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6800094a0 con 0x7fe6900cb870 2024-04-06T13:28:24.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.939+0000 7fe68e59c700 1 --2- 172.21.15.189:0/3767258865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6900cb870 0x7fe6900c7770 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fe68000ec10 tx=0x7fe68000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:24.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.939+0000 7fe67effd700 1 -- 172.21.15.189:0/3767258865 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe680009760 con 0x7fe6900cb870 2024-04-06T13:28:24.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.939+0000 7fe67effd700 1 -- 172.21.15.189:0/3767258865 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe68000c850 con 0x7fe6900cb870 2024-04-06T13:28:24.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.939+0000 7fe694f4e700 1 -- 172.21.15.189:0/3767258865 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6840094a0 con 0x7fe6900cb870 2024-04-06T13:28:24.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.939+0000 7fe67effd700 1 -- 172.21.15.189:0/3767258865 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68000cbe0 con 0x7fe6900cb870 2024-04-06T13:28:24.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.939+0000 7fe694f4e700 1 -- 172.21.15.189:0/3767258865 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6900c7cb0 con 0x7fe6900cb870 2024-04-06T13:28:24.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.940+0000 7fe694f4e700 1 -- 172.21.15.189:0/3767258865 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe690074990 con 0x7fe6900cb870 2024-04-06T13:28:24.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.941+0000 7fe67effd700 1 -- 172.21.15.189:0/3767258865 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 95) v1 ==== 89259+0+0 (secure 0 0 0) 0x7fe68000cd40 con 0x7fe6900cb870 2024-04-06T13:28:24.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.941+0000 7fe67effd700 1 --2- 172.21.15.189:0/3767258865 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe67006adb0 0x7fe67006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:24.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.941+0000 7fe68dd9b700 1 -- 172.21.15.189:0/3767258865 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe67006adb0 msgr2=0x7fe67006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:24.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.941+0000 7fe67effd700 1 -- 172.21.15.189:0/3767258865 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fe6800896c0 con 0x7fe6900cb870 2024-04-06T13:28:24.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.941+0000 7fe68dd9b700 1 --2- 172.21.15.189:0/3767258865 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe67006adb0 0x7fe67006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:24.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:24.946+0000 7fe67effd700 1 -- 172.21.15.189:0/3767258865 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe6800544b0 con 0x7fe6900cb870 2024-04-06T13:28:25.086 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:25.086+0000 7fca68879200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:28:25.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.142+0000 7fe68dd9b700 1 -- 172.21.15.189:0/3767258865 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe67006adb0 msgr2=0x7fe67006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:25.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.142+0000 7fe68dd9b700 1 --2- 172.21.15.189:0/3767258865 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe67006adb0 0x7fe67006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:25.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.151+0000 7fe694f4e700 1 -- 172.21.15.189:0/3767258865 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe6900d1e20 con 0x7fe6900cb870 2024-04-06T13:28:25.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.152+0000 7fe67effd700 1 -- 172.21.15.189:0/3767258865 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v165) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe680057ad0 con 0x7fe6900cb870 2024-04-06T13:28:25.152 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:28:25.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.154+0000 7fe694f4e700 1 -- 172.21.15.189:0/3767258865 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe67006adb0 msgr2=0x7fe67006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:25.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.154+0000 7fe694f4e700 1 --2- 172.21.15.189:0/3767258865 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe67006adb0 0x7fe67006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.154+0000 7fe694f4e700 1 -- 172.21.15.189:0/3767258865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6900cb870 msgr2=0x7fe6900c7770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:25.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.154+0000 7fe694f4e700 1 --2- 172.21.15.189:0/3767258865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6900cb870 0x7fe6900c7770 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fe68000ec10 tx=0x7fe68000ef20 comp rx=0 tx=0).stop 2024-04-06T13:28:25.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.154+0000 7fe694f4e700 1 -- 172.21.15.189:0/3767258865 shutdown_connections 2024-04-06T13:28:25.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.154+0000 7fe694f4e700 1 --2- 172.21.15.189:0/3767258865 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe67006adb0 0x7fe67006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.154+0000 7fe694f4e700 1 --2- 172.21.15.189:0/3767258865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6900d11b0 0x7fe6901588a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.154+0000 7fe694f4e700 1 --2- 172.21.15.189:0/3767258865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6900cb870 0x7fe6900c7770 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.154+0000 7fe694f4e700 1 --2- 172.21.15.189:0/3767258865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6900cfa00 0x7fe690158360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.154+0000 7fe694f4e700 1 -- 172.21.15.189:0/3767258865 >> 172.21.15.189:0/3767258865 conn(0x7fe6900b2db0 msgr2=0x7fe69015a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:25.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.154+0000 7fe694f4e700 1 -- 172.21.15.189:0/3767258865 shutdown_connections 2024-04-06T13:28:25.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.154+0000 7fe694f4e700 1 -- 172.21.15.189:0/3767258865 wait complete. 2024-04-06T13:28:25.166 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T13:28:25.237 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:25.237+0000 7fca68879200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:28:25.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.294+0000 7f5786c1c700 1 Processor -- start 2024-04-06T13:28:25.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.294+0000 7f5786c1c700 1 -- start start 2024-04-06T13:28:25.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.296+0000 7f5786c1c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f57800b4660 0x7f57800b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:25.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.296+0000 7f5786c1c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57800b2fc0 0x7f57800b3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:25.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.296+0000 7f5786c1c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57800b5030 con 0x7f578015a550 2024-04-06T13:28:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.296+0000 7f5786c1c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57800cdbe0 con 0x7f57800b4660 2024-04-06T13:28:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.296+0000 7f5786c1c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f578015a940 con 0x7f57800b2fc0 2024-04-06T13:28:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.296+0000 7f57849b8700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f578015a550 0x7f57800cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51470/0 (socket says 172.21.15.189:51470) 2024-04-06T13:28:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.296+0000 7f57849b8700 1 -- 172.21.15.189:0/745298309 learned_addr learned my addr 172.21.15.189:0/745298309 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.296+0000 7f577ffff700 1 --2- 172.21.15.189:0/745298309 >> v2:172.21.15.192:3300/0 conn(0x7f57800b4660 0x7f57800b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.296+0000 7f57851b9700 1 --2- 172.21.15.189:0/745298309 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57800b2fc0 0x7f57800b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.296+0000 7f577f7fe700 1 -- 172.21.15.189:0/745298309 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826304296 0 0) 0x7f57800b5030 con 0x7f578015a550 2024-04-06T13:28:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.296+0000 7f577f7fe700 1 -- 172.21.15.189:0/745298309 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57700035b0 con 0x7f578015a550 2024-04-06T13:28:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.296+0000 7f57851b9700 1 -- 172.21.15.189:0/745298309 >> v2:172.21.15.192:3300/0 conn(0x7f57800b4660 msgr2=0x7f57800b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.296+0000 7f57851b9700 1 --2- 172.21.15.189:0/745298309 >> v2:172.21.15.192:3300/0 conn(0x7f57800b4660 0x7f57800b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.296+0000 7f57851b9700 1 -- 172.21.15.189:0/745298309 >> v1:172.21.15.189:6789/0 conn(0x7f578015a550 legacy=0x7f57800cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.296+0000 7f57851b9700 1 -- 172.21.15.189:0/745298309 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578015ac00 con 0x7f57800b2fc0 2024-04-06T13:28:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.296+0000 7f577ffff700 1 --2- 172.21.15.189:0/745298309 >> v2:172.21.15.192:3300/0 conn(0x7f57800b4660 0x7f57800b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.297+0000 7f57851b9700 1 --2- 172.21.15.189:0/745298309 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57800b2fc0 0x7f57800b3410 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5778009d90 tx=0x7f577800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44ae0fb96c5d8385 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:25.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.297+0000 7f577f7fe700 1 -- 172.21.15.189:0/745298309 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577800e070 con 0x7f57800b2fc0 2024-04-06T13:28:25.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.297+0000 7f577f7fe700 1 -- 172.21.15.189:0/745298309 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f577800bb20 con 0x7f57800b2fc0 2024-04-06T13:28:25.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.297+0000 7f577f7fe700 1 -- 172.21.15.189:0/745298309 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5778012430 con 0x7f57800b2fc0 2024-04-06T13:28:25.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.297+0000 7f5786c1c700 1 -- 172.21.15.189:0/745298309 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57800b2fc0 msgr2=0x7f57800b3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:25.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.297+0000 7f5786c1c700 1 --2- 172.21.15.189:0/745298309 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57800b2fc0 0x7f57800b3410 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5778009d90 tx=0x7f577800b330 comp rx=0 tx=0).stop 2024-04-06T13:28:25.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.297+0000 7f5786c1c700 1 -- 172.21.15.189:0/745298309 shutdown_connections 2024-04-06T13:28:25.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.297+0000 7f5786c1c700 1 --2- 172.21.15.189:0/745298309 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57800b2fc0 0x7f57800b3410 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.297+0000 7f5786c1c700 1 --2- 172.21.15.189:0/745298309 >> v2:172.21.15.192:3300/0 conn(0x7f57800b4660 0x7f57800b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.297+0000 7f5786c1c700 1 -- 172.21.15.189:0/745298309 >> 172.21.15.189:0/745298309 conn(0x7f57800d00b0 msgr2=0x7f57800d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:25.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.297+0000 7f5786c1c700 1 -- 172.21.15.189:0/745298309 shutdown_connections 2024-04-06T13:28:25.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.297+0000 7f5786c1c700 1 -- 172.21.15.189:0/745298309 wait complete. 2024-04-06T13:28:25.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.299+0000 7f5786c1c700 1 Processor -- start 2024-04-06T13:28:25.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.299+0000 7f5786c1c700 1 -- start start 2024-04-06T13:28:25.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.300+0000 7f5786c1c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57800b2fc0 0x7f57800a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:25.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.300+0000 7f5786c1c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57800b4660 0x7f5780154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.300+0000 7f5786c1c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f578015d4e0 0x7f57800cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.300+0000 7f5786c1c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57800aa3d0 con 0x7f57800b4660 2024-04-06T13:28:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.300+0000 7f5786c1c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578015c5e0 con 0x7f578015d4e0 2024-04-06T13:28:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.300+0000 7f5786c1c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f578015b780 con 0x7f57800b2fc0 2024-04-06T13:28:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.300+0000 7f577ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57800b4660 0x7f5780154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.300+0000 7f577ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57800b4660 0x7f5780154af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55662/0 (socket says 172.21.15.189:55662) 2024-04-06T13:28:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.300+0000 7f577ffff700 1 -- 172.21.15.189:0/2464781042 learned_addr learned my addr 172.21.15.189:0/2464781042 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.300+0000 7f57849b8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57800b2fc0 0x7f57800a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.300+0000 7f57851b9700 1 --2- 172.21.15.189:0/2464781042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f578015d4e0 0x7f57800cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.300+0000 7f577ffff700 1 -- 172.21.15.189:0/2464781042 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57800b2fc0 msgr2=0x7f57800a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.300+0000 7f577ffff700 1 --2- 172.21.15.189:0/2464781042 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57800b2fc0 0x7f57800a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.300+0000 7f577ffff700 1 -- 172.21.15.189:0/2464781042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f578015d4e0 msgr2=0x7f57800cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.300+0000 7f577ffff700 1 --2- 172.21.15.189:0/2464781042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f578015d4e0 0x7f57800cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.300+0000 7f577ffff700 1 -- 172.21.15.189:0/2464781042 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c0094a0 con 0x7f57800b4660 2024-04-06T13:28:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.300+0000 7f57849b8700 1 --2- 172.21.15.189:0/2464781042 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57800b2fc0 0x7f57800a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.301+0000 7f577ffff700 1 --2- 172.21.15.189:0/2464781042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57800b4660 0x7f5780154af0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f576c00ec10 tx=0x7f576c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.301+0000 7f577d7fa700 1 -- 172.21.15.189:0/2464781042 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c00c850 con 0x7f57800b4660 2024-04-06T13:28:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.301+0000 7f577d7fa700 1 -- 172.21.15.189:0/2464781042 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576c00c9b0 con 0x7f57800b4660 2024-04-06T13:28:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.301+0000 7f5786c1c700 1 -- 172.21.15.189:0/2464781042 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5778009a70 con 0x7f57800b4660 2024-04-06T13:28:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.301+0000 7f577d7fa700 1 -- 172.21.15.189:0/2464781042 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c00cd20 con 0x7f57800b4660 2024-04-06T13:28:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.301+0000 7f5786c1c700 1 -- 172.21.15.189:0/2464781042 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5780155030 con 0x7f57800b4660 2024-04-06T13:28:25.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.302+0000 7f5786c1c700 1 -- 172.21.15.189:0/2464781042 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5780067bb0 con 0x7f57800b4660 2024-04-06T13:28:25.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.303+0000 7f577d7fa700 1 -- 172.21.15.189:0/2464781042 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 95) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f576c013470 con 0x7f57800b4660 2024-04-06T13:28:25.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.303+0000 7f577d7fa700 1 --2- 172.21.15.189:0/2464781042 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f576406adb0 0x7f576406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:25.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.303+0000 7f57849b8700 1 -- 172.21.15.189:0/2464781042 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f576406adb0 msgr2=0x7f576406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:25.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.303+0000 7f577d7fa700 1 -- 172.21.15.189:0/2464781042 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f576c089880 con 0x7f57800b4660 2024-04-06T13:28:25.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.303+0000 7f57849b8700 1 --2- 172.21.15.189:0/2464781042 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f576406adb0 0x7f576406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:25.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.308+0000 7f577d7fa700 1 -- 172.21.15.189:0/2464781042 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f576c0545d0 con 0x7f57800b4660 2024-04-06T13:28:25.374 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:25.375+0000 7f02c5aba200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:28:25.394 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:25.393+0000 7fca68879200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:28:25.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.504+0000 7f57849b8700 1 -- 172.21.15.189:0/2464781042 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f576406adb0 msgr2=0x7f576406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:25.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.504+0000 7f57849b8700 1 --2- 172.21.15.189:0/2464781042 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f576406adb0 0x7f576406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:25.536 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:25.536+0000 7f02c5aba200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:28:25.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.541+0000 7f5786c1c700 1 -- 172.21.15.189:0/2464781042 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5780074990 con 0x7f57800b4660 2024-04-06T13:28:25.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.541+0000 7f577d7fa700 1 -- 172.21.15.189:0/2464781042 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v165) v1 ==== 100+0+11 (secure 0 0 0) 0x7f576c057bf0 con 0x7f57800b4660 2024-04-06T13:28:25.542 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:28:25.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.543+0000 7f5786c1c700 1 -- 172.21.15.189:0/2464781042 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f576406adb0 msgr2=0x7f576406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:25.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.543+0000 7f5786c1c700 1 --2- 172.21.15.189:0/2464781042 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f576406adb0 0x7f576406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.543+0000 7f5786c1c700 1 -- 172.21.15.189:0/2464781042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57800b4660 msgr2=0x7f5780154af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:25.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.543+0000 7f5786c1c700 1 --2- 172.21.15.189:0/2464781042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57800b4660 0x7f5780154af0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f576c00ec10 tx=0x7f576c00ef20 comp rx=0 tx=0).stop 2024-04-06T13:28:25.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.544+0000 7f5786c1c700 1 -- 172.21.15.189:0/2464781042 shutdown_connections 2024-04-06T13:28:25.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.544+0000 7f5786c1c700 1 --2- 172.21.15.189:0/2464781042 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f576406adb0 0x7f576406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.544+0000 7f5786c1c700 1 --2- 172.21.15.189:0/2464781042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f578015d4e0 0x7f57800cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.544+0000 7f5786c1c700 1 --2- 172.21.15.189:0/2464781042 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57800b2fc0 0x7f57800a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.544+0000 7f5786c1c700 1 --2- 172.21.15.189:0/2464781042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57800b4660 0x7f5780154af0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.544+0000 7f5786c1c700 1 -- 172.21.15.189:0/2464781042 >> 172.21.15.189:0/2464781042 conn(0x7f57800d00b0 msgr2=0x7f578006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:25.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.544+0000 7f5786c1c700 1 -- 172.21.15.189:0/2464781042 shutdown_connections 2024-04-06T13:28:25.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.544+0000 7f5786c1c700 1 -- 172.21.15.189:0/2464781042 wait complete. 2024-04-06T13:28:25.556 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:28:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.681+0000 7fd5203ca700 1 Processor -- start 2024-04-06T13:28:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.681+0000 7fd5203ca700 1 -- start start 2024-04-06T13:28:25.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.682+0000 7fd5203ca700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd5180b4660 0x7fd5180b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:25.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.682+0000 7fd5203ca700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5180b2fc0 0x7fd5180b3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:25.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.682+0000 7fd5203ca700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5180b5030 con 0x7fd51815a550 2024-04-06T13:28:25.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.682+0000 7fd5203ca700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5180cdbe0 con 0x7fd5180b4660 2024-04-06T13:28:25.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.682+0000 7fd5203ca700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd51815a940 con 0x7fd5180b2fc0 2024-04-06T13:28:25.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.682+0000 7fd51d965700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd5180b4660 0x7fd5180b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:25.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.682+0000 7fd51e166700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd51815a550 0x7fd5180cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51482/0 (socket says 172.21.15.189:51482) 2024-04-06T13:28:25.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.682+0000 7fd51e166700 1 -- 172.21.15.189:0/1494551171 learned_addr learned my addr 172.21.15.189:0/1494551171 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:25.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.682+0000 7fd51e967700 1 --2- 172.21.15.189:0/1494551171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5180b2fc0 0x7fd5180b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:25.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.683+0000 7fd51d965700 1 -- 172.21.15.189:0/1494551171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5180b2fc0 msgr2=0x7fd5180b3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:25.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.683+0000 7fd51d965700 1 --2- 172.21.15.189:0/1494551171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5180b2fc0 0x7fd5180b3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.683+0000 7fd51d965700 1 -- 172.21.15.189:0/1494551171 >> v1:172.21.15.189:6789/0 conn(0x7fd51815a550 legacy=0x7fd5180cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.683+0000 7fd51d965700 1 -- 172.21.15.189:0/1494551171 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51815ac00 con 0x7fd5180b4660 2024-04-06T13:28:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.683+0000 7fd51d164700 1 -- 172.21.15.189:0/1494551171 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254300705 0 0) 0x7fd5180b5030 con 0x7fd51815a550 2024-04-06T13:28:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.683+0000 7fd51d965700 1 --2- 172.21.15.189:0/1494551171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5180b4660 0x7fd5180b4af0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7fd508009840 tx=0x7fd508009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45873c954d190ffc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.683+0000 7fd51d164700 1 -- 172.21.15.189:0/1494551171 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd508011830 con 0x7fd5180b4660 2024-04-06T13:28:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.683+0000 7fd51d164700 1 -- 172.21.15.189:0/1494551171 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd508011990 con 0x7fd5180b4660 2024-04-06T13:28:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.683+0000 7fd51d164700 1 -- 172.21.15.189:0/1494551171 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd508011d20 con 0x7fd5180b4660 2024-04-06T13:28:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.684+0000 7fd5203ca700 1 -- 172.21.15.189:0/1494551171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5180b4660 msgr2=0x7fd5180b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.684+0000 7fd5203ca700 1 --2- 172.21.15.189:0/1494551171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5180b4660 0x7fd5180b4af0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7fd508009840 tx=0x7fd508009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.684+0000 7fd5203ca700 1 -- 172.21.15.189:0/1494551171 shutdown_connections 2024-04-06T13:28:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.684+0000 7fd5203ca700 1 --2- 172.21.15.189:0/1494551171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5180b2fc0 0x7fd5180b3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.684+0000 7fd5203ca700 1 --2- 172.21.15.189:0/1494551171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5180b4660 0x7fd5180b4af0 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.684+0000 7fd5203ca700 1 -- 172.21.15.189:0/1494551171 >> 172.21.15.189:0/1494551171 conn(0x7fd5180d00b0 msgr2=0x7fd5180d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.684+0000 7fd5203ca700 1 -- 172.21.15.189:0/1494551171 shutdown_connections 2024-04-06T13:28:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.684+0000 7fd5203ca700 1 -- 172.21.15.189:0/1494551171 wait complete. 2024-04-06T13:28:25.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.685+0000 7fd5203ca700 1 Processor -- start 2024-04-06T13:28:25.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.685+0000 7fd5203ca700 1 -- start start 2024-04-06T13:28:25.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.686+0000 7fd5203ca700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5180b2fc0 0x7fd51814ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:25.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.686+0000 7fd5203ca700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5180b4660 0x7fd51814aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:25.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.686+0000 7fd5203ca700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd518154f10 0x7fd51814fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:25.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.686+0000 7fd5203ca700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5181553a0 con 0x7fd518154f10 2024-04-06T13:28:25.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.686+0000 7fd5203ca700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51814c080 con 0x7fd5180b4660 2024-04-06T13:28:25.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.686+0000 7fd5203ca700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd51814c360 con 0x7fd5180b2fc0 2024-04-06T13:28:25.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.686+0000 7fd51e967700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd518154f10 0x7fd51814fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:25.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.686+0000 7fd51e967700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd518154f10 0x7fd51814fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55684/0 (socket says 172.21.15.189:55684) 2024-04-06T13:28:25.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.686+0000 7fd51e967700 1 -- 172.21.15.189:0/3412799387 learned_addr learned my addr 172.21.15.189:0/3412799387 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:25.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.686+0000 7fd51e166700 1 --2- 172.21.15.189:0/3412799387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5180b2fc0 0x7fd51814ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:25.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.686+0000 7fd51d965700 1 --2- 172.21.15.189:0/3412799387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5180b4660 0x7fd51814aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:25.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.687+0000 7fd51e967700 1 -- 172.21.15.189:0/3412799387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5180b2fc0 msgr2=0x7fd51814ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:25.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.687+0000 7fd51e967700 1 --2- 172.21.15.189:0/3412799387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5180b2fc0 0x7fd51814ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.687+0000 7fd51e967700 1 -- 172.21.15.189:0/3412799387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5180b4660 msgr2=0x7fd51814aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:25.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.687+0000 7fd51e967700 1 --2- 172.21.15.189:0/3412799387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5180b4660 0x7fd51814aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.687+0000 7fd51e967700 1 -- 172.21.15.189:0/3412799387 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5080094a0 con 0x7fd518154f10 2024-04-06T13:28:25.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.687+0000 7fd51e166700 1 --2- 172.21.15.189:0/3412799387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5180b2fc0 0x7fd51814ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:25.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.687+0000 7fd51e967700 1 --2- 172.21.15.189:0/3412799387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd518154f10 0x7fd51814fc10 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fd51000efc0 tx=0x7fd51000c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:25.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.687+0000 7fd50effd700 1 -- 172.21.15.189:0/3412799387 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51000cad0 con 0x7fd518154f10 2024-04-06T13:28:25.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.687+0000 7fd50effd700 1 -- 172.21.15.189:0/3412799387 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51000f040 con 0x7fd518154f10 2024-04-06T13:28:25.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.687+0000 7fd50effd700 1 -- 172.21.15.189:0/3412799387 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51000ce70 con 0x7fd518154f10 2024-04-06T13:28:25.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.687+0000 7fd5203ca700 1 -- 172.21.15.189:0/3412799387 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd51814c640 con 0x7fd518154f10 2024-04-06T13:28:25.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.687+0000 7fd5203ca700 1 -- 172.21.15.189:0/3412799387 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd51814b030 con 0x7fd518154f10 2024-04-06T13:28:25.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.688+0000 7fd5203ca700 1 -- 172.21.15.189:0/3412799387 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd518067bb0 con 0x7fd518154f10 2024-04-06T13:28:25.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.689+0000 7fd50effd700 1 -- 172.21.15.189:0/3412799387 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 95) v1 ==== 89259+0+0 (secure 0 0 0) 0x7fd510014480 con 0x7fd518154f10 2024-04-06T13:28:25.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.689+0000 7fd50effd700 1 --2- 172.21.15.189:0/3412799387 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd4fc06adb0 0x7fd4fc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:25.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.689+0000 7fd51e166700 1 -- 172.21.15.189:0/3412799387 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd4fc06adb0 msgr2=0x7fd4fc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:25.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.689+0000 7fd50effd700 1 -- 172.21.15.189:0/3412799387 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fd51008a000 con 0x7fd518154f10 2024-04-06T13:28:25.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.689+0000 7fd51e166700 1 --2- 172.21.15.189:0/3412799387 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd4fc06adb0 0x7fd4fc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:25.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.693+0000 7fd50effd700 1 -- 172.21.15.189:0/3412799387 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd510054ce0 con 0x7fd518154f10 2024-04-06T13:28:25.705 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:25.706+0000 7f02c5aba200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:28:25.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.890+0000 7fd51e166700 1 -- 172.21.15.189:0/3412799387 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd4fc06adb0 msgr2=0x7fd4fc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:25.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.890+0000 7fd51e166700 1 --2- 172.21.15.189:0/3412799387 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd4fc06adb0 0x7fd4fc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:25.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.939+0000 7fd5203ca700 1 -- 172.21.15.189:0/3412799387 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd518074990 con 0x7fd518154f10 2024-04-06T13:28:25.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.940+0000 7fd50effd700 1 -- 172.21.15.189:0/3412799387 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4748 (secure 0 0 0) 0x7fd510058300 con 0x7fd518154f10 2024-04-06T13:28:25.940 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:25.941 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":70,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3697447910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3697447910},{"type":"v1","addr":"172.21.15.189:6837","nonce":3697447910}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":7469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/3633157989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3633157989},{"type":"v1","addr":"172.21.15.192:6834","nonce":3633157989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2721135386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2721135386},{"type":"v1","addr":"172.21.15.189:6835","nonce":2721135386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:32:03.067806+0000","modified":"2024-04-06T12:32:06.080510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9144":{"gid":9144,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/3068270636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":3068270636},{"type":"v1","addr":"172.21.15.192:6835","nonce":3068270636}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-06T13:28:25.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.942+0000 7fd5203ca700 1 -- 172.21.15.189:0/3412799387 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd4fc06adb0 msgr2=0x7fd4fc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:25.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.942+0000 7fd5203ca700 1 --2- 172.21.15.189:0/3412799387 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd4fc06adb0 0x7fd4fc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.942+0000 7fd5203ca700 1 -- 172.21.15.189:0/3412799387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd518154f10 msgr2=0x7fd51814fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:25.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.942+0000 7fd5203ca700 1 --2- 172.21.15.189:0/3412799387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd518154f10 0x7fd51814fc10 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fd51000efc0 tx=0x7fd51000c0c0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.942+0000 7fd5203ca700 1 -- 172.21.15.189:0/3412799387 shutdown_connections 2024-04-06T13:28:25.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.942+0000 7fd5203ca700 1 --2- 172.21.15.189:0/3412799387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd518154f10 0x7fd51814fc10 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.942+0000 7fd5203ca700 1 --2- 172.21.15.189:0/3412799387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5180b2fc0 0x7fd51814ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.942+0000 7fd5203ca700 1 --2- 172.21.15.189:0/3412799387 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd4fc06adb0 0x7fd4fc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.942+0000 7fd5203ca700 1 --2- 172.21.15.189:0/3412799387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5180b4660 0x7fd51814aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:25.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.942+0000 7fd5203ca700 1 -- 172.21.15.189:0/3412799387 >> 172.21.15.189:0/3412799387 conn(0x7fd5180d00b0 msgr2=0x7fd51806b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:25.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.942+0000 7fd5203ca700 1 -- 172.21.15.189:0/3412799387 shutdown_connections 2024-04-06T13:28:25.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:25.942+0000 7fd5203ca700 1 -- 172.21.15.189:0/3412799387 wait complete. 2024-04-06T13:28:25.945 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 70 2024-04-06T13:28:25.955 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:28:26.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.079+0000 7f4b6188b700 1 Processor -- start 2024-04-06T13:28:26.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.079+0000 7f4b6188b700 1 -- start start 2024-04-06T13:28:26.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.080+0000 7f4b6188b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b5c144f40 0x7f4b5c06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:26.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.080+0000 7f4b6188b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4b5c15fcd0 0x7f4b5c06ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:26.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.080+0000 7f4b6188b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b5c15c740 con 0x7f4b5c15ba40 2024-04-06T13:28:26.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.080+0000 7f4b6188b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b5c15c560 con 0x7f4b5c15fcd0 2024-04-06T13:28:26.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.080+0000 7f4b6188b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c0cfcb0 con 0x7f4b5c144f40 2024-04-06T13:28:26.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.080+0000 7f4b5affd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b5c144f40 0x7f4b5c06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:26.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.080+0000 7f4b5a7fc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4b5c15fcd0 0x7f4b5c06ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:26.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.080+0000 7f4b5b7fe700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f4b5c15ba40 0x7f4b5c15be50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:42938/0 (socket says 172.21.15.189:42938) 2024-04-06T13:28:26.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.080+0000 7f4b5b7fe700 1 -- 172.21.15.189:0/1865389686 learned_addr learned my addr 172.21.15.189:0/1865389686 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:26.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.081+0000 7f4b5affd700 1 -- 172.21.15.189:0/1865389686 >> v2:172.21.15.192:3300/0 conn(0x7f4b5c15fcd0 msgr2=0x7f4b5c06ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:26.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.081+0000 7f4b5affd700 1 --2- 172.21.15.189:0/1865389686 >> v2:172.21.15.192:3300/0 conn(0x7f4b5c15fcd0 0x7f4b5c06ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.081+0000 7f4b5affd700 1 -- 172.21.15.189:0/1865389686 >> v1:172.21.15.192:6789/0 conn(0x7f4b5c15ba40 legacy=0x7f4b5c15be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:26.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.081+0000 7f4b5affd700 1 -- 172.21.15.189:0/1865389686 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c15f1f0 con 0x7f4b5c144f40 2024-04-06T13:28:26.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.081+0000 7f4b5a7fc700 1 --2- 172.21.15.189:0/1865389686 >> v2:172.21.15.192:3300/0 conn(0x7f4b5c15fcd0 0x7f4b5c06ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:26.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.081+0000 7f4b5affd700 1 --2- 172.21.15.189:0/1865389686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b5c144f40 0x7f4b5c06a730 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c009840 tx=0x7f4b4c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64bcf811d194666 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:26.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.081+0000 7f4b59ffb700 1 -- 172.21.15.189:0/1865389686 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c011830 con 0x7f4b5c144f40 2024-04-06T13:28:26.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.081+0000 7f4b59ffb700 1 -- 172.21.15.189:0/1865389686 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b4c011990 con 0x7f4b5c144f40 2024-04-06T13:28:26.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.081+0000 7f4b59ffb700 1 -- 172.21.15.189:0/1865389686 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c011d20 con 0x7f4b5c144f40 2024-04-06T13:28:26.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.081+0000 7f4b6188b700 1 -- 172.21.15.189:0/1865389686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b5c144f40 msgr2=0x7f4b5c06a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:26.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.081+0000 7f4b6188b700 1 --2- 172.21.15.189:0/1865389686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b5c144f40 0x7f4b5c06a730 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c009840 tx=0x7f4b4c009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:26.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.082+0000 7f4b6188b700 1 -- 172.21.15.189:0/1865389686 shutdown_connections 2024-04-06T13:28:26.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.082+0000 7f4b6188b700 1 --2- 172.21.15.189:0/1865389686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b5c144f40 0x7f4b5c06a730 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.082+0000 7f4b6188b700 1 --2- 172.21.15.189:0/1865389686 >> v2:172.21.15.192:3300/0 conn(0x7f4b5c15fcd0 0x7f4b5c06ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.082+0000 7f4b6188b700 1 -- 172.21.15.189:0/1865389686 >> 172.21.15.189:0/1865389686 conn(0x7f4b5c0d00b0 msgr2=0x7f4b5c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:26.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.082+0000 7f4b6188b700 1 -- 172.21.15.189:0/1865389686 shutdown_connections 2024-04-06T13:28:26.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.082+0000 7f4b6188b700 1 -- 172.21.15.189:0/1865389686 wait complete. 2024-04-06T13:28:26.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.083+0000 7f4b6188b700 1 Processor -- start 2024-04-06T13:28:26.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.083+0000 7f4b6188b700 1 -- start start 2024-04-06T13:28:26.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b6188b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b5c144f40 0x7f4b5c0c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:26.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b6188b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b5c15ba40 0x7f4b5c0b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b6188b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b5c15fcd0 0x7f4b5c06f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b6188b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c0b4220 con 0x7f4b5c15fcd0 2024-04-06T13:28:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b6188b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c0c9eb0 con 0x7f4b5c144f40 2024-04-06T13:28:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b6188b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c0b3980 con 0x7f4b5c15ba40 2024-04-06T13:28:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b5affd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b5c144f40 0x7f4b5c0c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b5a7fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b5c15ba40 0x7f4b5c0b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b5b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b5c15fcd0 0x7f4b5c06f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b5affd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b5c144f40 0x7f4b5c0c9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:39876/0 (socket says 172.21.15.189:39876) 2024-04-06T13:28:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b5a7fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b5c15ba40 0x7f4b5c0b30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:44324/0 (socket says 172.21.15.189:44324) 2024-04-06T13:28:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b5affd700 1 -- 172.21.15.189:0/1378339458 learned_addr learned my addr 172.21.15.189:0/1378339458 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b5b7fe700 1 -- 172.21.15.189:0/1378339458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b5c15ba40 msgr2=0x7f4b5c0b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b5b7fe700 1 --2- 172.21.15.189:0/1378339458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b5c15ba40 0x7f4b5c0b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b5b7fe700 1 -- 172.21.15.189:0/1378339458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b5c144f40 msgr2=0x7f4b5c0c9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:26.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b5b7fe700 1 --2- 172.21.15.189:0/1378339458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b5c144f40 0x7f4b5c0c9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b5b7fe700 1 -- 172.21.15.189:0/1378339458 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b500094a0 con 0x7f4b5c15fcd0 2024-04-06T13:28:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b5affd700 1 --2- 172.21.15.189:0/1378339458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b5c144f40 0x7f4b5c0c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b5a7fc700 1 --2- 172.21.15.189:0/1378339458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b5c15ba40 0x7f4b5c0b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b5b7fe700 1 --2- 172.21.15.189:0/1378339458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b5c15fcd0 0x7f4b5c06f690 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f4b5400caa0 tx=0x7f4b5400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b43fff700 1 -- 172.21.15.189:0/1378339458 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5400a9a0 con 0x7f4b5c15fcd0 2024-04-06T13:28:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b43fff700 1 -- 172.21.15.189:0/1378339458 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5400ab00 con 0x7f4b5c15fcd0 2024-04-06T13:28:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b43fff700 1 -- 172.21.15.189:0/1378339458 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5400ae70 con 0x7f4b5c15fcd0 2024-04-06T13:28:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.084+0000 7f4b6188b700 1 -- 172.21.15.189:0/1378339458 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b4c0094a0 con 0x7f4b5c15fcd0 2024-04-06T13:28:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.085+0000 7f4b6188b700 1 -- 172.21.15.189:0/1378339458 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b5c0b3f40 con 0x7f4b5c15fcd0 2024-04-06T13:28:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.086+0000 7f4b6188b700 1 -- 172.21.15.189:0/1378339458 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b5c067bb0 con 0x7f4b5c15fcd0 2024-04-06T13:28:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.086+0000 7f4b43fff700 1 -- 172.21.15.189:0/1378339458 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 95) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f4b5401b3b0 con 0x7f4b5c15fcd0 2024-04-06T13:28:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.086+0000 7f4b43fff700 1 --2- 172.21.15.189:0/1378339458 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b4806adb0 0x7f4b4806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.086+0000 7f4b5affd700 1 -- 172.21.15.189:0/1378339458 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b4806adb0 msgr2=0x7f4b4806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.086+0000 7f4b43fff700 1 -- 172.21.15.189:0/1378339458 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f4b540891e0 con 0x7f4b5c15fcd0 2024-04-06T13:28:26.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.086+0000 7f4b5affd700 1 --2- 172.21.15.189:0/1378339458 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b4806adb0 0x7f4b4806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:26.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.095+0000 7f4b43fff700 1 -- 172.21.15.189:0/1378339458 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4b540540a0 con 0x7f4b5c15fcd0 2024-04-06T13:28:26.114 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:26.114+0000 7fca68879200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:28:26.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.287+0000 7f4b5affd700 1 -- 172.21.15.189:0/1378339458 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b4806adb0 msgr2=0x7f4b4806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:26.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.287+0000 7f4b5affd700 1 --2- 172.21.15.189:0/1378339458 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b4806adb0 0x7f4b4806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:26.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.295+0000 7f4b6188b700 1 -- 172.21.15.189:0/1378339458 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b5c074990 con 0x7f4b5c15fcd0 2024-04-06T13:28:26.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.297+0000 7f4b43fff700 1 -- 172.21.15.189:0/1378339458 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v165) v1 ==== 74+0+16976 (secure 0 0 0) 0x7f4b540576c0 con 0x7f4b5c15fcd0 2024-04-06T13:28:26.297 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:26.297 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":165,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:28:21.351197+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:32:00.826039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-04-06T12:32:02.039801+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"165","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":112,"snap_epoch":165,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/3791371660":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/3889346008":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/628196195":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/4023078006":"2024-04-07T12:32:16.629669+0000","172.21.15.189:6833/47128":"2024-04-07T12:32:16.629669+0000","172.21.15.189:6832/47128":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/3100554908":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/383422220":"2024-04-07T12:32:16.629669+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":54},{"begin":59,"length":54}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":47,"length":10},{"begin":103,"length":10}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:28:26.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.299+0000 7f4b6188b700 1 -- 172.21.15.189:0/1378339458 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b4806adb0 msgr2=0x7f4b4806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:26.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.299+0000 7f4b6188b700 1 --2- 172.21.15.189:0/1378339458 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b4806adb0 0x7f4b4806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.299+0000 7f4b6188b700 1 -- 172.21.15.189:0/1378339458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b5c15fcd0 msgr2=0x7f4b5c06f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:26.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.299+0000 7f4b6188b700 1 --2- 172.21.15.189:0/1378339458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b5c15fcd0 0x7f4b5c06f690 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f4b5400caa0 tx=0x7f4b5400cdb0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.299+0000 7f4b6188b700 1 -- 172.21.15.189:0/1378339458 shutdown_connections 2024-04-06T13:28:26.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.299+0000 7f4b6188b700 1 --2- 172.21.15.189:0/1378339458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b5c15fcd0 0x7f4b5c06f690 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.299+0000 7f4b6188b700 1 --2- 172.21.15.189:0/1378339458 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4b4806adb0 0x7f4b4806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.299+0000 7f4b6188b700 1 --2- 172.21.15.189:0/1378339458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b5c144f40 0x7f4b5c0c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.299+0000 7f4b6188b700 1 --2- 172.21.15.189:0/1378339458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b5c15ba40 0x7f4b5c0b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.299+0000 7f4b6188b700 1 -- 172.21.15.189:0/1378339458 >> 172.21.15.189:0/1378339458 conn(0x7f4b5c0d00b0 msgr2=0x7f4b5c140160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:26.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.299+0000 7f4b6188b700 1 -- 172.21.15.189:0/1378339458 shutdown_connections 2024-04-06T13:28:26.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.299+0000 7f4b6188b700 1 -- 172.21.15.189:0/1378339458 wait complete. 2024-04-06T13:28:26.312 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-06T13:28:26.312 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:28:26.354 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:26.353+0000 7fca68879200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:28:26.429 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:26.428+0000 7fca68879200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:28:26.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.479+0000 7f76cc943700 1 Processor -- start 2024-04-06T13:28:26.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.479+0000 7f76cc943700 1 -- start start 2024-04-06T13:28:26.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.480+0000 7f76cc943700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f76c8144f40 0x7f76c806a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:26.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.480+0000 7f76cc943700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76c815fcd0 0x7f76c806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:26.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.480+0000 7f76cc943700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76c815c740 con 0x7f76c815ba40 2024-04-06T13:28:26.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.480+0000 7f76cc943700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76c80cfcb0 con 0x7f76c8144f40 2024-04-06T13:28:26.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.480+0000 7f76cc943700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76c815ca20 con 0x7f76c815fcd0 2024-04-06T13:28:26.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.480+0000 7f76c659c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f76c8144f40 0x7f76c806a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:26.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.480+0000 7f76c6d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f76c815ba40 0x7f76c815be90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51486/0 (socket says 172.21.15.189:51486) 2024-04-06T13:28:26.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.480+0000 7f76c6d9d700 1 -- 172.21.15.189:0/110975732 learned_addr learned my addr 172.21.15.189:0/110975732 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:26.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.481+0000 7f76c5d9b700 1 --2- 172.21.15.189:0/110975732 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76c815fcd0 0x7f76c806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:26.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.481+0000 7f76c659c700 1 -- 172.21.15.189:0/110975732 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76c815fcd0 msgr2=0x7f76c806ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:26.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.481+0000 7f76c659c700 1 --2- 172.21.15.189:0/110975732 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76c815fcd0 0x7f76c806ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.481+0000 7f76c659c700 1 -- 172.21.15.189:0/110975732 >> v1:172.21.15.189:6789/0 conn(0x7f76c815ba40 legacy=0x7f76c815be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:26.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.481+0000 7f76c659c700 1 -- 172.21.15.189:0/110975732 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c815f1f0 con 0x7f76c8144f40 2024-04-06T13:28:26.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.481+0000 7f76c559a700 1 -- 172.21.15.189:0/110975732 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390502242 0 0) 0x7f76c815c740 con 0x7f76c815ba40 2024-04-06T13:28:26.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.481+0000 7f76c659c700 1 --2- 172.21.15.189:0/110975732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76c8144f40 0x7f76c806a730 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f76b8009840 tx=0x7f76b8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c1bae402a2c732a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:26.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.481+0000 7f76c559a700 1 -- 172.21.15.189:0/110975732 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b8011830 con 0x7f76c8144f40 2024-04-06T13:28:26.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.481+0000 7f76c559a700 1 -- 172.21.15.189:0/110975732 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b8011990 con 0x7f76c8144f40 2024-04-06T13:28:26.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.481+0000 7f76c559a700 1 -- 172.21.15.189:0/110975732 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b8011d20 con 0x7f76c8144f40 2024-04-06T13:28:26.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.482+0000 7f76cc943700 1 -- 172.21.15.189:0/110975732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76c8144f40 msgr2=0x7f76c806a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:26.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.482+0000 7f76cc943700 1 --2- 172.21.15.189:0/110975732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76c8144f40 0x7f76c806a730 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f76b8009840 tx=0x7f76b8009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:26.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.482+0000 7f76cc943700 1 -- 172.21.15.189:0/110975732 shutdown_connections 2024-04-06T13:28:26.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.482+0000 7f76cc943700 1 --2- 172.21.15.189:0/110975732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76c8144f40 0x7f76c806a730 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.482+0000 7f76cc943700 1 --2- 172.21.15.189:0/110975732 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76c815fcd0 0x7f76c806ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.482+0000 7f76cc943700 1 -- 172.21.15.189:0/110975732 >> 172.21.15.189:0/110975732 conn(0x7f76c80d00b0 msgr2=0x7f76c80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:26.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.482+0000 7f76cc943700 1 -- 172.21.15.189:0/110975732 shutdown_connections 2024-04-06T13:28:26.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.482+0000 7f76cc943700 1 -- 172.21.15.189:0/110975732 wait complete. 2024-04-06T13:28:26.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.483+0000 7f76cc943700 1 Processor -- start 2024-04-06T13:28:26.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.483+0000 7f76cc943700 1 -- start start 2024-04-06T13:28:26.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.484+0000 7f76cc943700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76c8144f40 0x7f76c8131f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:26.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.484+0000 7f76cc943700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76c815ba40 0x7f76c815add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:26.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.484+0000 7f76cc943700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76c815fcd0 0x7f76c81304b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:26.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.484+0000 7f76cc943700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c8131af0 con 0x7f76c815fcd0 2024-04-06T13:28:26.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.484+0000 7f76cc943700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c8130e70 con 0x7f76c815ba40 2024-04-06T13:28:26.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.484+0000 7f76cc943700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76c8131150 con 0x7f76c8144f40 2024-04-06T13:28:26.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.484+0000 7f76c6d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76c815fcd0 0x7f76c81304b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:26.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.484+0000 7f76c6d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76c815fcd0 0x7f76c81304b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55720/0 (socket says 172.21.15.189:55720) 2024-04-06T13:28:26.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.484+0000 7f76c6d9d700 1 -- 172.21.15.189:0/381471358 learned_addr learned my addr 172.21.15.189:0/381471358 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:26.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.484+0000 7f76c659c700 1 --2- 172.21.15.189:0/381471358 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76c8144f40 0x7f76c8131f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:26.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.484+0000 7f76c5d9b700 1 --2- 172.21.15.189:0/381471358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76c815ba40 0x7f76c815add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:26.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.484+0000 7f76c6d9d700 1 -- 172.21.15.189:0/381471358 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76c8144f40 msgr2=0x7f76c8131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:26.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.484+0000 7f76c6d9d700 1 --2- 172.21.15.189:0/381471358 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76c8144f40 0x7f76c8131f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.484+0000 7f76c6d9d700 1 -- 172.21.15.189:0/381471358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76c815ba40 msgr2=0x7f76c815add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:26.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.484+0000 7f76c6d9d700 1 --2- 172.21.15.189:0/381471358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76c815ba40 0x7f76c815add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.484+0000 7f76c6d9d700 1 -- 172.21.15.189:0/381471358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b80094a0 con 0x7f76c815fcd0 2024-04-06T13:28:26.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.485+0000 7f76c6d9d700 1 --2- 172.21.15.189:0/381471358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76c815fcd0 0x7f76c81304b0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f76c000cca0 tx=0x7f76c000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:26.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.485+0000 7f76b6ffd700 1 -- 172.21.15.189:0/381471358 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c000ab70 con 0x7f76c815fcd0 2024-04-06T13:28:26.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.485+0000 7f76b6ffd700 1 -- 172.21.15.189:0/381471358 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c000acd0 con 0x7f76c815fcd0 2024-04-06T13:28:26.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.485+0000 7f76b6ffd700 1 -- 172.21.15.189:0/381471358 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c0007310 con 0x7f76c815fcd0 2024-04-06T13:28:26.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.485+0000 7f76cc943700 1 -- 172.21.15.189:0/381471358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76c8131430 con 0x7f76c815fcd0 2024-04-06T13:28:26.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.485+0000 7f76cc943700 1 -- 172.21.15.189:0/381471358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76c815b310 con 0x7f76c815fcd0 2024-04-06T13:28:26.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.486+0000 7f76cc943700 1 -- 172.21.15.189:0/381471358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76c8131c30 con 0x7f76c815fcd0 2024-04-06T13:28:26.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.486+0000 7f76b6ffd700 1 -- 172.21.15.189:0/381471358 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 95) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f76c0007470 con 0x7f76c815fcd0 2024-04-06T13:28:26.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.487+0000 7f76b6ffd700 1 --2- 172.21.15.189:0/381471358 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f76a806adb0 0x7f76a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:26.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.487+0000 7f76c659c700 1 -- 172.21.15.189:0/381471358 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f76a806adb0 msgr2=0x7f76a806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:26.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.487+0000 7f76b6ffd700 1 -- 172.21.15.189:0/381471358 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f76c00893f0 con 0x7f76c815fcd0 2024-04-06T13:28:26.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.487+0000 7f76c659c700 1 --2- 172.21.15.189:0/381471358 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f76a806adb0 0x7f76a806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:26.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.491+0000 7f76b6ffd700 1 -- 172.21.15.189:0/381471358 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f76c8131c30 con 0x7f76c815fcd0 2024-04-06T13:28:26.519 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:26.520+0000 7f02c5aba200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:28:26.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.688+0000 7f76c659c700 1 -- 172.21.15.189:0/381471358 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f76a806adb0 msgr2=0x7f76a806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:26.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.688+0000 7f76c659c700 1 --2- 172.21.15.189:0/381471358 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f76a806adb0 0x7f76a806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:26.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.738+0000 7f76cc943700 1 -- 172.21.15.189:0/381471358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76c8131c30 con 0x7f76c815fcd0 2024-04-06T13:28:26.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.738+0000 7f76b6ffd700 1 -- 172.21.15.189:0/381471358 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4748 (secure 0 0 0) 0x7f76c8131c30 con 0x7f76c815fcd0 2024-04-06T13:28:26.738 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:26.739 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":70,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3697447910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3697447910},{"type":"v1","addr":"172.21.15.189:6837","nonce":3697447910}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":7469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/3633157989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3633157989},{"type":"v1","addr":"172.21.15.192:6834","nonce":3633157989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2721135386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2721135386},{"type":"v1","addr":"172.21.15.189:6835","nonce":2721135386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:32:03.067806+0000","modified":"2024-04-06T12:32:06.080510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9144":{"gid":9144,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/3068270636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":3068270636},{"type":"v1","addr":"172.21.15.192:6835","nonce":3068270636}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-06T13:28:26.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.740+0000 7f76cc943700 1 -- 172.21.15.189:0/381471358 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f76a806adb0 msgr2=0x7f76a806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:26.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.740+0000 7f76cc943700 1 --2- 172.21.15.189:0/381471358 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f76a806adb0 0x7f76a806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.741+0000 7f76cc943700 1 -- 172.21.15.189:0/381471358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76c815fcd0 msgr2=0x7f76c81304b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:26.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.741+0000 7f76cc943700 1 --2- 172.21.15.189:0/381471358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76c815fcd0 0x7f76c81304b0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f76c000cca0 tx=0x7f76c000a060 comp rx=0 tx=0).stop 2024-04-06T13:28:26.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.741+0000 7f76cc943700 1 -- 172.21.15.189:0/381471358 shutdown_connections 2024-04-06T13:28:26.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.741+0000 7f76cc943700 1 --2- 172.21.15.189:0/381471358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76c815fcd0 0x7f76c81304b0 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.741+0000 7f76cc943700 1 --2- 172.21.15.189:0/381471358 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76c8144f40 0x7f76c8131f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.741+0000 7f76cc943700 1 --2- 172.21.15.189:0/381471358 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f76a806adb0 0x7f76a806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.741+0000 7f76cc943700 1 --2- 172.21.15.189:0/381471358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76c815ba40 0x7f76c815add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.741+0000 7f76cc943700 1 -- 172.21.15.189:0/381471358 >> 172.21.15.189:0/381471358 conn(0x7f76c80d00b0 msgr2=0x7f76c80cf560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:26.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.741+0000 7f76cc943700 1 -- 172.21.15.189:0/381471358 shutdown_connections 2024-04-06T13:28:26.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.741+0000 7f76cc943700 1 -- 172.21.15.189:0/381471358 wait complete. 2024-04-06T13:28:26.742 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 70 2024-04-06T13:28:26.754 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:28:26.777 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:26.777+0000 7f02c5aba200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:28:26.801 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:26.800+0000 7fca68879200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:28:26.862 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:26.862+0000 7f02c5aba200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:28:26.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.880+0000 7fc1bae05700 1 Processor -- start 2024-04-06T13:28:26.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.880+0000 7fc1bae05700 1 -- start start 2024-04-06T13:28:26.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.880+0000 7fc1bae05700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc1b40d11b0 0x7fc1b40d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:26.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.880+0000 7fc1bae05700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40cb870 0x7fc1b415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:26.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.880+0000 7fc1bae05700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b40d1ad0 con 0x7fc1b40cfa00 2024-04-06T13:28:26.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.880+0000 7fc1bae05700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b40ccdc0 con 0x7fc1b40d11b0 2024-04-06T13:28:26.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.880+0000 7fc1bae05700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40d0b50 con 0x7fc1b40cb870 2024-04-06T13:28:26.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.881+0000 7fc1b8ba1700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc1b40d11b0 0x7fc1b40d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:26.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.881+0000 7fc1b3fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40cb870 0x7fc1b415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:26.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.881+0000 7fc1b93a2700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fc1b40cfa00 0x7fc1b40cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:42954/0 (socket says 172.21.15.189:42954) 2024-04-06T13:28:26.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.881+0000 7fc1b8ba1700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc1b40d11b0 0x7fc1b40d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:39902/0 (socket says 172.21.15.189:39902) 2024-04-06T13:28:26.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.881+0000 7fc1b93a2700 1 -- 172.21.15.189:0/1117620164 learned_addr learned my addr 172.21.15.189:0/1117620164 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:26.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.881+0000 7fc1b8ba1700 1 -- 172.21.15.189:0/1117620164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40cb870 msgr2=0x7fc1b415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:26.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.881+0000 7fc1b8ba1700 1 --2- 172.21.15.189:0/1117620164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40cb870 0x7fc1b415c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.881+0000 7fc1b8ba1700 1 -- 172.21.15.189:0/1117620164 >> v1:172.21.15.192:6789/0 conn(0x7fc1b40cfa00 legacy=0x7fc1b40cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:26.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.881+0000 7fc1b8ba1700 1 -- 172.21.15.189:0/1117620164 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b40cbd80 con 0x7fc1b40d11b0 2024-04-06T13:28:26.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.881+0000 7fc1b37fe700 1 -- 172.21.15.189:0/1117620164 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1574297172 0 0) 0x7fc1b40d1ad0 con 0x7fc1b40cfa00 2024-04-06T13:28:26.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.881+0000 7fc1b3fff700 1 --2- 172.21.15.189:0/1117620164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40cb870 0x7fc1b415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:26.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.881+0000 7fc1b8ba1700 1 --2- 172.21.15.189:0/1117620164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b40d11b0 0x7fc1b40d1590 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8009840 tx=0x7fc1a8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c48b1fdc01a8737 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:26.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.881+0000 7fc1b37fe700 1 -- 172.21.15.189:0/1117620164 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8011830 con 0x7fc1b40d11b0 2024-04-06T13:28:26.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.881+0000 7fc1b37fe700 1 -- 172.21.15.189:0/1117620164 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a8011990 con 0x7fc1b40d11b0 2024-04-06T13:28:26.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.882+0000 7fc1b37fe700 1 -- 172.21.15.189:0/1117620164 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8011d20 con 0x7fc1b40d11b0 2024-04-06T13:28:26.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.882+0000 7fc1bae05700 1 -- 172.21.15.189:0/1117620164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b40d11b0 msgr2=0x7fc1b40d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:26.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.882+0000 7fc1bae05700 1 --2- 172.21.15.189:0/1117620164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b40d11b0 0x7fc1b40d1590 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8009840 tx=0x7fc1a8009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:26.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.882+0000 7fc1bae05700 1 -- 172.21.15.189:0/1117620164 shutdown_connections 2024-04-06T13:28:26.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.882+0000 7fc1bae05700 1 --2- 172.21.15.189:0/1117620164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b40d11b0 0x7fc1b40d1590 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.882+0000 7fc1bae05700 1 --2- 172.21.15.189:0/1117620164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40cb870 0x7fc1b415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.882+0000 7fc1bae05700 1 -- 172.21.15.189:0/1117620164 >> 172.21.15.189:0/1117620164 conn(0x7fc1b40b2db0 msgr2=0x7fc1b40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:26.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.882+0000 7fc1bae05700 1 -- 172.21.15.189:0/1117620164 shutdown_connections 2024-04-06T13:28:26.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.882+0000 7fc1bae05700 1 -- 172.21.15.189:0/1117620164 wait complete. 2024-04-06T13:28:26.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.883+0000 7fc1bae05700 1 Processor -- start 2024-04-06T13:28:26.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.883+0000 7fc1bae05700 1 -- start start 2024-04-06T13:28:26.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.884+0000 7fc1bae05700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b40cb870 0x7fc1b41584c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:26.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.884+0000 7fc1bae05700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40cfa00 0x7fc1b40ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:26.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.884+0000 7fc1bae05700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1b40d11b0 0x7fc1b4155260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:26.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.884+0000 7fc1bae05700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40cb2b0 con 0x7fc1b40d11b0 2024-04-06T13:28:26.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.884+0000 7fc1bae05700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40cacf0 con 0x7fc1b40cb870 2024-04-06T13:28:26.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.884+0000 7fc1bae05700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40cafd0 con 0x7fc1b40cfa00 2024-04-06T13:28:26.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.884+0000 7fc1b93a2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1b40d11b0 0x7fc1b4155260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:26.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.884+0000 7fc1b93a2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1b40d11b0 0x7fc1b4155260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55734/0 (socket says 172.21.15.189:55734) 2024-04-06T13:28:26.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.884+0000 7fc1b93a2700 1 -- 172.21.15.189:0/3815661936 learned_addr learned my addr 172.21.15.189:0/3815661936 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:26.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.884+0000 7fc1b3fff700 1 --2- 172.21.15.189:0/3815661936 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40cfa00 0x7fc1b40ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:26.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.884+0000 7fc1b8ba1700 1 --2- 172.21.15.189:0/3815661936 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b40cb870 0x7fc1b41584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:26.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.885+0000 7fc1b93a2700 1 -- 172.21.15.189:0/3815661936 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40cfa00 msgr2=0x7fc1b40ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:26.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.885+0000 7fc1b93a2700 1 --2- 172.21.15.189:0/3815661936 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40cfa00 0x7fc1b40ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.885+0000 7fc1b93a2700 1 -- 172.21.15.189:0/3815661936 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b40cb870 msgr2=0x7fc1b41584c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:26.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.885+0000 7fc1b93a2700 1 --2- 172.21.15.189:0/3815661936 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b40cb870 0x7fc1b41584c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:26.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.885+0000 7fc1b93a2700 1 -- 172.21.15.189:0/3815661936 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a00094a0 con 0x7fc1b40d11b0 2024-04-06T13:28:26.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.885+0000 7fc1b8ba1700 1 --2- 172.21.15.189:0/3815661936 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b40cb870 0x7fc1b41584c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:28:26.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.885+0000 7fc1b93a2700 1 --2- 172.21.15.189:0/3815661936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1b40d11b0 0x7fc1b4155260 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac002230 tx=0x7fc1ac00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:26.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.885+0000 7fc1b17fa700 1 -- 172.21.15.189:0/3815661936 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac00ac20 con 0x7fc1b40d11b0 2024-04-06T13:28:26.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.885+0000 7fc1b17fa700 1 -- 172.21.15.189:0/3815661936 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ac00ada0 con 0x7fc1b40d11b0 2024-04-06T13:28:26.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.885+0000 7fc1b17fa700 1 -- 172.21.15.189:0/3815661936 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac007310 con 0x7fc1b40d11b0 2024-04-06T13:28:26.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.885+0000 7fc1bae05700 1 -- 172.21.15.189:0/3815661936 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a80094a0 con 0x7fc1b40d11b0 2024-04-06T13:28:26.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.885+0000 7fc1bae05700 1 -- 172.21.15.189:0/3815661936 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b41f25d0 con 0x7fc1b40d11b0 2024-04-06T13:28:26.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.886+0000 7fc1bae05700 1 -- 172.21.15.189:0/3815661936 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b406e720 con 0x7fc1b40d11b0 2024-04-06T13:28:26.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.886+0000 7fc1b17fa700 1 -- 172.21.15.189:0/3815661936 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 95) v1 ==== 89259+0+0 (secure 0 0 0) 0x7fc1ac007470 con 0x7fc1b40d11b0 2024-04-06T13:28:26.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.887+0000 7fc1b17fa700 1 --2- 172.21.15.189:0/3815661936 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc19806adb0 0x7fc19806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:26.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.887+0000 7fc1b8ba1700 1 -- 172.21.15.189:0/3815661936 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc19806adb0 msgr2=0x7fc19806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:26.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.887+0000 7fc1b17fa700 1 -- 172.21.15.189:0/3815661936 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fc1ac089390 con 0x7fc1b40d11b0 2024-04-06T13:28:26.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.887+0000 7fc1b8ba1700 1 --2- 172.21.15.189:0/3815661936 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc19806adb0 0x7fc19806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:26.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:26.891+0000 7fc1b17fa700 1 -- 172.21.15.189:0/3815661936 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc1ac0540f0 con 0x7fc1b40d11b0 2024-04-06T13:28:27.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.088+0000 7fc1b8ba1700 1 -- 172.21.15.189:0/3815661936 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc19806adb0 msgr2=0x7fc19806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:27.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.088+0000 7fc1b8ba1700 1 --2- 172.21.15.189:0/3815661936 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc19806adb0 0x7fc19806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:27.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.138+0000 7fc1bae05700 1 -- 172.21.15.189:0/3815661936 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1b4074990 con 0x7fc1b40d11b0 2024-04-06T13:28:27.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.139+0000 7fc1b17fa700 1 -- 172.21.15.189:0/3815661936 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4748 (secure 0 0 0) 0x7fc1ac057710 con 0x7fc1b40d11b0 2024-04-06T13:28:27.139 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:27.139 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":70,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3697447910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3697447910},{"type":"v1","addr":"172.21.15.189:6837","nonce":3697447910}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":7469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/3633157989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3633157989},{"type":"v1","addr":"172.21.15.192:6834","nonce":3633157989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2721135386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2721135386},{"type":"v1","addr":"172.21.15.189:6835","nonce":2721135386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T12:32:03.067806+0000","modified":"2024-04-06T12:32:06.080510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9144":{"gid":9144,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/3068270636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":3068270636},{"type":"v1","addr":"172.21.15.192:6835","nonce":3068270636}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-06T13:28:27.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.141+0000 7fc1bae05700 1 -- 172.21.15.189:0/3815661936 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc19806adb0 msgr2=0x7fc19806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:27.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.141+0000 7fc1bae05700 1 --2- 172.21.15.189:0/3815661936 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc19806adb0 0x7fc19806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.141+0000 7fc1bae05700 1 -- 172.21.15.189:0/3815661936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1b40d11b0 msgr2=0x7fc1b4155260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:27.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.141+0000 7fc1bae05700 1 --2- 172.21.15.189:0/3815661936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1b40d11b0 0x7fc1b4155260 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac002230 tx=0x7fc1ac00a060 comp rx=0 tx=0).stop 2024-04-06T13:28:27.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.141+0000 7fc1bae05700 1 -- 172.21.15.189:0/3815661936 shutdown_connections 2024-04-06T13:28:27.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.141+0000 7fc1bae05700 1 --2- 172.21.15.189:0/3815661936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1b40d11b0 0x7fc1b4155260 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.141+0000 7fc1bae05700 1 --2- 172.21.15.189:0/3815661936 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc19806adb0 0x7fc19806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.141+0000 7fc1bae05700 1 --2- 172.21.15.189:0/3815661936 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b40cb870 0x7fc1b41584c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.141+0000 7fc1bae05700 1 --2- 172.21.15.189:0/3815661936 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40cfa00 0x7fc1b40ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.141+0000 7fc1bae05700 1 -- 172.21.15.189:0/3815661936 >> 172.21.15.189:0/3815661936 conn(0x7fc1b40b2db0 msgr2=0x7fc1b40b6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:27.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.141+0000 7fc1bae05700 1 -- 172.21.15.189:0/3815661936 shutdown_connections 2024-04-06T13:28:27.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.141+0000 7fc1bae05700 1 -- 172.21.15.189:0/3815661936 wait complete. 2024-04-06T13:28:27.143 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 70 2024-04-06T13:28:27.155 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:28:27.258 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:27.259+0000 7f02c5aba200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:28:27.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.331+0000 7fde68611700 1 Processor -- start 2024-04-06T13:28:27.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.331+0000 7fde68611700 1 -- start start 2024-04-06T13:28:27.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.332+0000 7fde68611700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde600cd3e0 0x7fde600cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:27.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.332+0000 7fde68611700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6006ceb0 con 0x7fde6015d7c0 2024-04-06T13:28:27.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.332+0000 7fde68611700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6006d750 con 0x7fde6006b8c0 2024-04-06T13:28:27.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.332+0000 7fde68611700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde600cdfa0 con 0x7fde600cd3e0 2024-04-06T13:28:27.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.332+0000 7fde66bae700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fde6006b8c0 0x7fde6006dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:42964/0 (socket says 172.21.15.189:42964) 2024-04-06T13:28:27.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.332+0000 7fde66bae700 1 -- 172.21.15.189:0/3003296624 learned_addr learned my addr 172.21.15.189:0/3003296624 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:27.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.332+0000 7fde663ad700 1 --2- 172.21.15.189:0/3003296624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde600cd3e0 0x7fde600cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:27.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.332+0000 7fde663ad700 1 -- 172.21.15.189:0/3003296624 >> v1:172.21.15.192:6789/0 conn(0x7fde6006b8c0 legacy=0x7fde6006dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:27.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.332+0000 7fde663ad700 1 -- 172.21.15.189:0/3003296624 >> v1:172.21.15.189:6789/0 conn(0x7fde6015d7c0 legacy=0x7fde6006b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:27.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.332+0000 7fde663ad700 1 -- 172.21.15.189:0/3003296624 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde600cdd00 con 0x7fde600cd3e0 2024-04-06T13:28:27.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.332+0000 7fde663ad700 1 --2- 172.21.15.189:0/3003296624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde600cd3e0 0x7fde600cd7c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fde5c009870 tx=0x7fde5c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9386e864ae13f80 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:27.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.332+0000 7fde653ab700 1 -- 172.21.15.189:0/3003296624 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c004230 con 0x7fde600cd3e0 2024-04-06T13:28:27.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.333+0000 7fde653ab700 1 -- 172.21.15.189:0/3003296624 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5c004390 con 0x7fde600cd3e0 2024-04-06T13:28:27.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.333+0000 7fde653ab700 1 -- 172.21.15.189:0/3003296624 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c0036a0 con 0x7fde600cd3e0 2024-04-06T13:28:27.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.333+0000 7fde68611700 1 -- 172.21.15.189:0/3003296624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde600cd3e0 msgr2=0x7fde600cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:27.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.333+0000 7fde68611700 1 --2- 172.21.15.189:0/3003296624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde600cd3e0 0x7fde600cd7c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fde5c009870 tx=0x7fde5c009b80 comp rx=0 tx=0).stop 2024-04-06T13:28:27.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.333+0000 7fde68611700 1 -- 172.21.15.189:0/3003296624 shutdown_connections 2024-04-06T13:28:27.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.333+0000 7fde68611700 1 --2- 172.21.15.189:0/3003296624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde600cd3e0 0x7fde600cd7c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.333+0000 7fde68611700 1 -- 172.21.15.189:0/3003296624 >> 172.21.15.189:0/3003296624 conn(0x7fde600cc290 msgr2=0x7fde600cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:27.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.333+0000 7fde68611700 1 -- 172.21.15.189:0/3003296624 shutdown_connections 2024-04-06T13:28:27.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.333+0000 7fde68611700 1 -- 172.21.15.189:0/3003296624 wait complete. 2024-04-06T13:28:27.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.335+0000 7fde68611700 1 Processor -- start 2024-04-06T13:28:27.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.335+0000 7fde68611700 1 -- start start 2024-04-06T13:28:27.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.336+0000 7fde68611700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde6006b8c0 0x7fde600c5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:27.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.336+0000 7fde68611700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde600cd3e0 0x7fde600c6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:27.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.336+0000 7fde68611700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde6015d7c0 0x7fde601f3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:27.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.336+0000 7fde68611700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde600c5840 con 0x7fde6006b8c0 2024-04-06T13:28:27.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.336+0000 7fde68611700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6015af50 con 0x7fde600cd3e0 2024-04-06T13:28:27.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.336+0000 7fde68611700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde6015b230 con 0x7fde6015d7c0 2024-04-06T13:28:27.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.336+0000 7fde663ad700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde6006b8c0 0x7fde600c5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:27.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.336+0000 7fde663ad700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde6006b8c0 0x7fde600c5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55788/0 (socket says 172.21.15.189:55788) 2024-04-06T13:28:27.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.336+0000 7fde663ad700 1 -- 172.21.15.189:0/4060238165 learned_addr learned my addr 172.21.15.189:0/4060238165 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:27.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.336+0000 7fde65bac700 1 --2- 172.21.15.189:0/4060238165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde600cd3e0 0x7fde600c6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:27.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.336+0000 7fde66bae700 1 --2- 172.21.15.189:0/4060238165 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde6015d7c0 0x7fde601f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:27.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.336+0000 7fde663ad700 1 -- 172.21.15.189:0/4060238165 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde6015d7c0 msgr2=0x7fde601f3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:27.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.336+0000 7fde663ad700 1 --2- 172.21.15.189:0/4060238165 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde6015d7c0 0x7fde601f3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.336+0000 7fde663ad700 1 -- 172.21.15.189:0/4060238165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde600cd3e0 msgr2=0x7fde600c6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:27.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.336+0000 7fde663ad700 1 --2- 172.21.15.189:0/4060238165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde600cd3e0 0x7fde600c6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.336+0000 7fde663ad700 1 -- 172.21.15.189:0/4060238165 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5c0094a0 con 0x7fde6006b8c0 2024-04-06T13:28:27.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.336+0000 7fde663ad700 1 --2- 172.21.15.189:0/4060238165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde6006b8c0 0x7fde600c5300 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fde5c0090b0 tx=0x7fde5c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:27.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.337+0000 7fde56ffd700 1 -- 172.21.15.189:0/4060238165 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c011630 con 0x7fde6006b8c0 2024-04-06T13:28:27.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.337+0000 7fde56ffd700 1 -- 172.21.15.189:0/4060238165 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5c01e070 con 0x7fde6006b8c0 2024-04-06T13:28:27.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.337+0000 7fde56ffd700 1 -- 172.21.15.189:0/4060238165 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c0119d0 con 0x7fde6006b8c0 2024-04-06T13:28:27.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.337+0000 7fde68611700 1 -- 172.21.15.189:0/4060238165 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6015b510 con 0x7fde6006b8c0 2024-04-06T13:28:27.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.337+0000 7fde68611700 1 -- 172.21.15.189:0/4060238165 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde6015bad0 con 0x7fde6006b8c0 2024-04-06T13:28:27.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.338+0000 7fde68611700 1 -- 172.21.15.189:0/4060238165 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde60074990 con 0x7fde6006b8c0 2024-04-06T13:28:27.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.339+0000 7fde56ffd700 1 -- 172.21.15.189:0/4060238165 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 96) v1 ==== 88856+0+0 (secure 0 0 0) 0x7fde5c011b30 con 0x7fde6006b8c0 2024-04-06T13:28:27.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.340+0000 7fde56ffd700 1 -- 172.21.15.189:0/4060238165 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fde5c08a4b0 con 0x7fde6006b8c0 2024-04-06T13:28:27.344 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:27.345+0000 7f023d1d1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:27.344 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:27.345+0000 7f023d1d1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:27.345 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:27.345+0000 7f023d1d1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:27.345 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:27.345+0000 7f023d1d1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:27.345 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:27.345+0000 7f023d1d1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:27.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.345+0000 7fde56ffd700 1 -- 172.21.15.189:0/4060238165 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fde5c055620 con 0x7fde6006b8c0 2024-04-06T13:28:27.346 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:27.346+0000 7f02411d9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:27.346 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:27.346+0000 7f02411d9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:27.346 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:27.346+0000 7f02411d9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:27.346 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:27.346+0000 7f02411d9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:27.347 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:27.346+0000 7f02411d9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:27.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.542+0000 7fde68611700 1 -- 172.21.15.189:0/4060238165 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde600b5790 con 0x7fde6006b8c0 2024-04-06T13:28:27.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.543+0000 7fde56ffd700 1 -- 172.21.15.189:0/4060238165 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v166) v1 ==== 74+0+17281 (secure 0 0 0) 0x7fde5c020090 con 0x7fde6006b8c0 2024-04-06T13:28:27.543 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:27.544 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":166,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:28:27.273217+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-04-06T12:32:00.826039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-04-06T12:32:02.039801+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"165","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":112,"snap_epoch":165,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/3836616107":"2024-04-07T13:28:27.273156+0000","172.21.15.189:0/3921446814":"2024-04-07T13:28:27.273156+0000","172.21.15.189:0/382015867":"2024-04-07T13:28:27.273156+0000","172.21.15.189:0/1260972304":"2024-04-07T13:28:27.273156+0000","172.21.15.189:0/3822669837":"2024-04-07T13:28:27.273156+0000","172.21.15.189:0/3791371660":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/1467579759":"2024-04-07T13:28:27.273156+0000","172.21.15.189:0/3889346008":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/628196195":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/4023078006":"2024-04-07T12:32:16.629669+0000","172.21.15.189:6833/47128":"2024-04-07T12:32:16.629669+0000","172.21.15.189:6832/47128":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/3100554908":"2024-04-07T12:32:16.629669+0000","172.21.15.189:0/383422220":"2024-04-07T12:32:16.629669+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":54},{"begin":59,"length":54}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:28:27.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.545+0000 7fde68611700 1 -- 172.21.15.189:0/4060238165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde6006b8c0 msgr2=0x7fde600c5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:27.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.545+0000 7fde68611700 1 --2- 172.21.15.189:0/4060238165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde6006b8c0 0x7fde600c5300 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fde5c0090b0 tx=0x7fde5c004750 comp rx=0 tx=0).stop 2024-04-06T13:28:27.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.545+0000 7fde68611700 1 -- 172.21.15.189:0/4060238165 shutdown_connections 2024-04-06T13:28:27.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.545+0000 7fde68611700 1 --2- 172.21.15.189:0/4060238165 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde6015d7c0 0x7fde601f3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.545+0000 7fde68611700 1 --2- 172.21.15.189:0/4060238165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde6006b8c0 0x7fde600c5300 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.545+0000 7fde68611700 1 --2- 172.21.15.189:0/4060238165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde600cd3e0 0x7fde600c6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.545+0000 7fde68611700 1 -- 172.21.15.189:0/4060238165 >> 172.21.15.189:0/4060238165 conn(0x7fde600cc290 msgr2=0x7fde600b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:27.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.546+0000 7fde68611700 1 -- 172.21.15.189:0/4060238165 shutdown_connections 2024-04-06T13:28:27.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.546+0000 7fde68611700 1 -- 172.21.15.189:0/4060238165 wait complete. 2024-04-06T13:28:27.558 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-06T13:28:27.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.682+0000 7f246a709700 1 Processor -- start 2024-04-06T13:28:27.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.683+0000 7f246a709700 1 -- start start 2024-04-06T13:28:27.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.683+0000 7f246a709700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f246415a550 0x7f24640cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:27.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.683+0000 7f246a709700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f24640b4d70 0x7f24640b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:27.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.683+0000 7f246a709700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24640cdbe0 con 0x7f24640cf900 2024-04-06T13:28:27.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.683+0000 7f246a709700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246415a940 con 0x7f24640b4d70 2024-04-06T13:28:27.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.683+0000 7f246a709700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246415ac00 con 0x7f246415a550 2024-04-06T13:28:27.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.683+0000 7f2468ca6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f24640b4d70 0x7f24640b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:27.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.684+0000 7f2468ca6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f24640b4d70 0x7f24640b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55858/0 (socket says 172.21.15.189:55858) 2024-04-06T13:28:27.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.684+0000 7f2468ca6700 1 -- 172.21.15.189:0/3461569509 learned_addr learned my addr 172.21.15.189:0/3461569509 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:27.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.684+0000 7f2468ca6700 1 -- 172.21.15.189:0/3461569509 >> v2:172.21.15.192:3300/0 conn(0x7f246415a550 msgr2=0x7f24640cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:27.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.684+0000 7f2463fff700 1 --2- 172.21.15.189:0/3461569509 >> v2:172.21.15.192:3300/0 conn(0x7f246415a550 0x7f24640cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:27.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.684+0000 7f2468ca6700 1 --2- 172.21.15.189:0/3461569509 >> v2:172.21.15.192:3300/0 conn(0x7f246415a550 0x7f24640cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.684+0000 7f2468ca6700 1 -- 172.21.15.189:0/3461569509 >> v1:172.21.15.189:6789/0 conn(0x7f24640cf900 legacy=0x7f24640b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:27.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.684+0000 7f2468ca6700 1 -- 172.21.15.189:0/3461569509 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246415aee0 con 0x7f24640b4d70 2024-04-06T13:28:27.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.684+0000 7f2468ca6700 1 --2- 172.21.15.189:0/3461569509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24640b4d70 0x7f24640b2fc0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f2464073470 tx=0x7f245c00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f26c2f5e6f86d79e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:27.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.684+0000 7f2462ffd700 1 -- 172.21.15.189:0/3461569509 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c010070 con 0x7f24640b4d70 2024-04-06T13:28:27.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.684+0000 7f2462ffd700 1 -- 172.21.15.189:0/3461569509 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245c00dd90 con 0x7f24640b4d70 2024-04-06T13:28:27.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.684+0000 7f2462ffd700 1 -- 172.21.15.189:0/3461569509 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c00b310 con 0x7f24640b4d70 2024-04-06T13:28:27.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.685+0000 7f246a709700 1 -- 172.21.15.189:0/3461569509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24640b4d70 msgr2=0x7f24640b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:27.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.685+0000 7f246a709700 1 --2- 172.21.15.189:0/3461569509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24640b4d70 0x7f24640b2fc0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f2464073470 tx=0x7f245c00d430 comp rx=0 tx=0).stop 2024-04-06T13:28:27.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.685+0000 7f246a709700 1 -- 172.21.15.189:0/3461569509 shutdown_connections 2024-04-06T13:28:27.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.685+0000 7f246a709700 1 --2- 172.21.15.189:0/3461569509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24640b4d70 0x7f24640b2fc0 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.685+0000 7f246a709700 1 --2- 172.21.15.189:0/3461569509 >> v2:172.21.15.192:3300/0 conn(0x7f246415a550 0x7f24640cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.685+0000 7f246a709700 1 -- 172.21.15.189:0/3461569509 >> 172.21.15.189:0/3461569509 conn(0x7f24640d00b0 msgr2=0x7f24640d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:27.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.685+0000 7f246a709700 1 -- 172.21.15.189:0/3461569509 shutdown_connections 2024-04-06T13:28:27.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.685+0000 7f246a709700 1 -- 172.21.15.189:0/3461569509 wait complete. 2024-04-06T13:28:27.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.686+0000 7f246a709700 1 Processor -- start 2024-04-06T13:28:27.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.686+0000 7f246a709700 1 -- start start 2024-04-06T13:28:27.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f246a709700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24640b4d70 0x7f24641ee240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:27.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f246a709700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24640cf900 0x7f2464156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:27.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f246a709700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2464156b60 0x7f24640a8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:27.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f246a709700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2464157010 con 0x7f2464156b60 2024-04-06T13:28:27.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f246a709700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24641ee780 con 0x7f24640cf900 2024-04-06T13:28:27.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f246a709700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24640a9d80 con 0x7f24640b4d70 2024-04-06T13:28:27.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f2468ca6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2464156b60 0x7f24640a8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:27.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f2468ca6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2464156b60 0x7f24640a8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55862/0 (socket says 172.21.15.189:55862) 2024-04-06T13:28:27.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f2468ca6700 1 -- 172.21.15.189:0/1224008687 learned_addr learned my addr 172.21.15.189:0/1224008687 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:27.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f2463fff700 1 --2- 172.21.15.189:0/1224008687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24640b4d70 0x7f24641ee240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:27.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f24637fe700 1 --2- 172.21.15.189:0/1224008687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24640cf900 0x7f2464156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:27.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f2468ca6700 1 -- 172.21.15.189:0/1224008687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24640b4d70 msgr2=0x7f24641ee240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:27.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f2468ca6700 1 --2- 172.21.15.189:0/1224008687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24640b4d70 0x7f24641ee240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f2468ca6700 1 -- 172.21.15.189:0/1224008687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24640cf900 msgr2=0x7f2464156620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:27.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f2468ca6700 1 --2- 172.21.15.189:0/1224008687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24640cf900 0x7f2464156620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f2468ca6700 1 -- 172.21.15.189:0/1224008687 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245c00bd40 con 0x7f2464156b60 2024-04-06T13:28:27.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f2468ca6700 1 --2- 172.21.15.189:0/1224008687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2464156b60 0x7f24640a8fe0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f246406c820 tx=0x7f245c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:27.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f2460ff9700 1 -- 172.21.15.189:0/1224008687 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c00bbc0 con 0x7f2464156b60 2024-04-06T13:28:27.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f2460ff9700 1 -- 172.21.15.189:0/1224008687 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245c00b4c0 con 0x7f2464156b60 2024-04-06T13:28:27.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f2460ff9700 1 -- 172.21.15.189:0/1224008687 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c00ae90 con 0x7f2464156b60 2024-04-06T13:28:27.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.687+0000 7f246a709700 1 -- 172.21.15.189:0/1224008687 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24640aa060 con 0x7f2464156b60 2024-04-06T13:28:27.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.688+0000 7f246a709700 1 -- 172.21.15.189:0/1224008687 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24641eea20 con 0x7f2464156b60 2024-04-06T13:28:27.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.689+0000 7f246a709700 1 -- 172.21.15.189:0/1224008687 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2464067bb0 con 0x7f2464156b60 2024-04-06T13:28:27.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.689+0000 7f2460ff9700 1 -- 172.21.15.189:0/1224008687 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 96) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f245c004040 con 0x7f2464156b60 2024-04-06T13:28:27.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.689+0000 7f2460ff9700 1 -- 172.21.15.189:0/1224008687 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f245c08cd30 con 0x7f2464156b60 2024-04-06T13:28:27.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.694+0000 7f2460ff9700 1 -- 172.21.15.189:0/1224008687 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f245c057f90 con 0x7f2464156b60 2024-04-06T13:28:27.886 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:28:27.888 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:28:27.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.933+0000 7f246a709700 1 -- 172.21.15.189:0/1224008687 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f24640b34a0 con 0x7f2464156b60 2024-04-06T13:28:27.934 INFO:tasks.ceph.mon.a.smithi189.stderr:2024-04-06T13:28:27.934+0000 7fa9fe1ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T13:28:27.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.945+0000 7f2460ff9700 1 -- 172.21.15.189:0/1224008687 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v71) v1 ==== 161+0+0 (secure 0 0 0) 0x7f245c05b5b0 con 0x7f2464156b60 2024-04-06T13:28:27.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.948+0000 7f246a709700 1 -- 172.21.15.189:0/1224008687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2464156b60 msgr2=0x7f24640a8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:27.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.948+0000 7f246a709700 1 --2- 172.21.15.189:0/1224008687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2464156b60 0x7f24640a8fe0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f246406c820 tx=0x7f245c004ad0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.948+0000 7f246a709700 1 -- 172.21.15.189:0/1224008687 shutdown_connections 2024-04-06T13:28:27.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.948+0000 7f246a709700 1 --2- 172.21.15.189:0/1224008687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2464156b60 0x7f24640a8fe0 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.948+0000 7f246a709700 1 --2- 172.21.15.189:0/1224008687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24640b4d70 0x7f24641ee240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.948+0000 7f246a709700 1 --2- 172.21.15.189:0/1224008687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24640cf900 0x7f2464156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:27.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.948+0000 7f246a709700 1 -- 172.21.15.189:0/1224008687 >> 172.21.15.189:0/1224008687 conn(0x7f24640d00b0 msgr2=0x7f24640b3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:27.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.948+0000 7f246a709700 1 -- 172.21.15.189:0/1224008687 shutdown_connections 2024-04-06T13:28:27.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:27.948+0000 7f246a709700 1 -- 172.21.15.189:0/1224008687 wait complete. 2024-04-06T13:28:27.949 INFO:teuthology.orchestra.run.smithi189.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-06T13:28:27.960 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-06T13:28:27.984 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T13:28:28.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.083+0000 7f8fdfe0b700 1 Processor -- start 2024-04-06T13:28:28.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.083+0000 7f8fdfe0b700 1 -- start start 2024-04-06T13:28:28.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.084+0000 7f8fdfe0b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8fd815a550 0x7f8fd80cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:28.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.084+0000 7f8fdfe0b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd80cdbe0 con 0x7f8fd80b4d70 2024-04-06T13:28:28.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.084+0000 7f8fdfe0b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd815a940 con 0x7f8fd80cf900 2024-04-06T13:28:28.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.084+0000 7f8fdfe0b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd815ac00 con 0x7f8fd815a550 2024-04-06T13:28:28.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.084+0000 7f8fde3a8700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8fd80b4d70 0x7f8fd80b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51514/0 (socket says 172.21.15.189:51514) 2024-04-06T13:28:28.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.084+0000 7f8fde3a8700 1 -- 172.21.15.189:0/466956883 learned_addr learned my addr 172.21.15.189:0/466956883 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:28.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.084+0000 7f8fddba7700 1 --2- 172.21.15.189:0/466956883 >> v2:172.21.15.192:3300/0 conn(0x7f8fd815a550 0x7f8fd80cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:28.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.084+0000 7f8fdcba5700 1 -- 172.21.15.189:0/466956883 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348242322 0 0) 0x7f8fd80cdbe0 con 0x7f8fd80b4d70 2024-04-06T13:28:28.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.084+0000 7f8fdcba5700 1 -- 172.21.15.189:0/466956883 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fc40035b0 con 0x7f8fd80b4d70 2024-04-06T13:28:28.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.084+0000 7f8fddba7700 1 -- 172.21.15.189:0/466956883 >> v1:172.21.15.192:6789/0 conn(0x7f8fd80cf900 legacy=0x7f8fd80b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:28.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.084+0000 7f8fdcba5700 1 -- 172.21.15.189:0/466956883 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358635908 0 0) 0x7f8fd815a940 con 0x7f8fd80cf900 2024-04-06T13:28:28.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.084+0000 7f8fddba7700 1 -- 172.21.15.189:0/466956883 >> v1:172.21.15.189:6789/0 conn(0x7f8fd80b4d70 legacy=0x7f8fd80b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:28.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.084+0000 7f8fddba7700 1 -- 172.21.15.189:0/466956883 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd815aee0 con 0x7f8fd815a550 2024-04-06T13:28:28.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.084+0000 7f8fddba7700 1 --2- 172.21.15.189:0/466956883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fd815a550 0x7f8fd80cf3c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009840 tx=0x7f8fd4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=972f3f25e0fb199b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:28.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.085+0000 7f8fdcba5700 1 -- 172.21.15.189:0/466956883 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4011830 con 0x7f8fd815a550 2024-04-06T13:28:28.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.085+0000 7f8fdcba5700 1 -- 172.21.15.189:0/466956883 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd4011990 con 0x7f8fd815a550 2024-04-06T13:28:28.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.085+0000 7f8fdcba5700 1 -- 172.21.15.189:0/466956883 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4011d20 con 0x7f8fd815a550 2024-04-06T13:28:28.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.085+0000 7f8fdfe0b700 1 -- 172.21.15.189:0/466956883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fd815a550 msgr2=0x7f8fd80cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:28.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.085+0000 7f8fdfe0b700 1 --2- 172.21.15.189:0/466956883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fd815a550 0x7f8fd80cf3c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009840 tx=0x7f8fd4009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:28.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.085+0000 7f8fdfe0b700 1 -- 172.21.15.189:0/466956883 shutdown_connections 2024-04-06T13:28:28.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.085+0000 7f8fdfe0b700 1 --2- 172.21.15.189:0/466956883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fd815a550 0x7f8fd80cf3c0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:28.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.085+0000 7f8fdfe0b700 1 -- 172.21.15.189:0/466956883 >> 172.21.15.189:0/466956883 conn(0x7f8fd80d00b0 msgr2=0x7f8fd80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:28.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.085+0000 7f8fdfe0b700 1 -- 172.21.15.189:0/466956883 shutdown_connections 2024-04-06T13:28:28.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.085+0000 7f8fdfe0b700 1 -- 172.21.15.189:0/466956883 wait complete. 2024-04-06T13:28:28.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.086+0000 7f8fdfe0b700 1 Processor -- start 2024-04-06T13:28:28.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.086+0000 7f8fdfe0b700 1 -- start start 2024-04-06T13:28:28.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.087+0000 7f8fdfe0b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd80b4d70 0x7f8fd8158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:28.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.087+0000 7f8fdfe0b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fd80cf900 0x7f8fd80b1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:28.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.087+0000 7f8fdfe0b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8fd80a8640 0x7f8fd80717b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:28.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.087+0000 7f8fdfe0b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd8158640 con 0x7f8fd80b4d70 2024-04-06T13:28:28.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.087+0000 7f8fdfe0b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd8158920 con 0x7f8fd80cf900 2024-04-06T13:28:28.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.087+0000 7f8fdfe0b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd8158c00 con 0x7f8fd80a8640 2024-04-06T13:28:28.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.087+0000 7f8fddba7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd80b4d70 0x7f8fd8158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:28.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.087+0000 7f8fddba7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd80b4d70 0x7f8fd8158060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55874/0 (socket says 172.21.15.189:55874) 2024-04-06T13:28:28.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.087+0000 7f8fddba7700 1 -- 172.21.15.189:0/1018659324 learned_addr learned my addr 172.21.15.189:0/1018659324 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:28.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.087+0000 7f8fde3a8700 1 --2- 172.21.15.189:0/1018659324 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8fd80a8640 0x7f8fd80717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:28.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.087+0000 7f8fdd3a6700 1 --2- 172.21.15.189:0/1018659324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fd80cf900 0x7f8fd80b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:28.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.088+0000 7f8fddba7700 1 -- 172.21.15.189:0/1018659324 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8fd80a8640 msgr2=0x7f8fd80717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:28.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.088+0000 7f8fddba7700 1 --2- 172.21.15.189:0/1018659324 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8fd80a8640 0x7f8fd80717b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:28.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.088+0000 7f8fddba7700 1 -- 172.21.15.189:0/1018659324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fd80cf900 msgr2=0x7f8fd80b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:28.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.088+0000 7f8fddba7700 1 --2- 172.21.15.189:0/1018659324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fd80cf900 0x7f8fd80b1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:28.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.088+0000 7f8fddba7700 1 -- 172.21.15.189:0/1018659324 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd40094a0 con 0x7f8fd80b4d70 2024-04-06T13:28:28.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.088+0000 7f8fddba7700 1 --2- 172.21.15.189:0/1018659324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd80b4d70 0x7f8fd8158060 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4014040 tx=0x7f8fd4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:28.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.088+0000 7f8fce7fc700 1 -- 172.21.15.189:0/1018659324 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd401a7e0 con 0x7f8fd80b4d70 2024-04-06T13:28:28.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.088+0000 7f8fce7fc700 1 -- 172.21.15.189:0/1018659324 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd401a940 con 0x7f8fd80b4d70 2024-04-06T13:28:28.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.088+0000 7f8fce7fc700 1 -- 172.21.15.189:0/1018659324 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd401acd0 con 0x7f8fd80b4d70 2024-04-06T13:28:28.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.088+0000 7f8fdfe0b700 1 -- 172.21.15.189:0/1018659324 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd8158ee0 con 0x7f8fd80b4d70 2024-04-06T13:28:28.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.088+0000 7f8fdfe0b700 1 -- 172.21.15.189:0/1018659324 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd80a8ad0 con 0x7f8fd80b4d70 2024-04-06T13:28:28.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.089+0000 7f8fdfe0b700 1 -- 172.21.15.189:0/1018659324 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd8159400 con 0x7f8fd80b4d70 2024-04-06T13:28:28.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.089+0000 7f8fce7fc700 1 -- 172.21.15.189:0/1018659324 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 96) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f8fd4022070 con 0x7f8fd80b4d70 2024-04-06T13:28:28.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.090+0000 7f8fce7fc700 1 -- 172.21.15.189:0/1018659324 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f8fd4089e80 con 0x7f8fd80b4d70 2024-04-06T13:28:28.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.094+0000 7f8fce7fc700 1 -- 172.21.15.189:0/1018659324 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8fd8159400 con 0x7f8fd80b4d70 2024-04-06T13:28:28.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.284+0000 7f8fce7fc700 1 -- 172.21.15.189:0/1018659324 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 97) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f8fd4054b10 con 0x7f8fd80b4d70 2024-04-06T13:28:28.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.284+0000 7f8fce7fc700 1 --2- 172.21.15.189:0/1018659324 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8fbc036150 0x7f8fbc072a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:28.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.285+0000 7f8fdd3a6700 1 --2- 172.21.15.189:0/1018659324 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8fbc036150 0x7f8fbc072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:28.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.287+0000 7f8fdd3a6700 1 --2- 172.21.15.189:0/1018659324 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8fbc036150 0x7f8fbc072a10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8fd814d900 tx=0x7f8fc8009040 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:28.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.348+0000 7f8fdfe0b700 1 -- 172.21.15.189:0/1018659324 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8fd8067bb0 con 0x7f8fd80b4d70 2024-04-06T13:28:28.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.947+0000 7f8fce7fc700 1 -- 172.21.15.189:0/1018659324 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v72) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8fd4059f00 con 0x7f8fd80b4d70 2024-04-06T13:28:28.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.950+0000 7f8fdfe0b700 1 -- 172.21.15.189:0/1018659324 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8fbc036150 msgr2=0x7f8fbc072a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:28.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.950+0000 7f8fdfe0b700 1 --2- 172.21.15.189:0/1018659324 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8fbc036150 0x7f8fbc072a10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8fd814d900 tx=0x7f8fc8009040 comp rx=0 tx=0).stop 2024-04-06T13:28:28.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.950+0000 7f8fdfe0b700 1 -- 172.21.15.189:0/1018659324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd80b4d70 msgr2=0x7f8fd8158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:28.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.950+0000 7f8fdfe0b700 1 --2- 172.21.15.189:0/1018659324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd80b4d70 0x7f8fd8158060 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4014040 tx=0x7f8fd4004750 comp rx=0 tx=0).stop 2024-04-06T13:28:28.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.951+0000 7f8fdfe0b700 1 -- 172.21.15.189:0/1018659324 shutdown_connections 2024-04-06T13:28:28.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.951+0000 7f8fdfe0b700 1 --2- 172.21.15.189:0/1018659324 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8fd80a8640 0x7f8fd80717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:28.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.951+0000 7f8fdfe0b700 1 --2- 172.21.15.189:0/1018659324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd80b4d70 0x7f8fd8158060 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:28.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.951+0000 7f8fdfe0b700 1 --2- 172.21.15.189:0/1018659324 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8fbc036150 0x7f8fbc072a10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:28.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.951+0000 7f8fdfe0b700 1 --2- 172.21.15.189:0/1018659324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fd80cf900 0x7f8fd80b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:28.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.951+0000 7f8fdfe0b700 1 -- 172.21.15.189:0/1018659324 >> 172.21.15.189:0/1018659324 conn(0x7f8fd80d00b0 msgr2=0x7f8fd80b4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:28.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.951+0000 7f8fdfe0b700 1 -- 172.21.15.189:0/1018659324 shutdown_connections 2024-04-06T13:28:28.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:28.951+0000 7f8fdfe0b700 1 -- 172.21.15.189:0/1018659324 wait complete. 2024-04-06T13:28:28.966 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-06T13:28:29.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.100+0000 7f9a2696c700 1 Processor -- start 2024-04-06T13:28:29.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.100+0000 7f9a2696c700 1 -- start start 2024-04-06T13:28:29.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.101+0000 7f9a2696c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a2006ae40 0x7f9a200b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:29.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.101+0000 7f9a2696c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9a2015ddf0 0x7f9a2015e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:29.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.101+0000 7f9a2696c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a200b3990 con 0x7f9a2015d7b0 2024-04-06T13:28:29.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.101+0000 7f9a2696c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a200b3060 con 0x7f9a2015ddf0 2024-04-06T13:28:29.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.101+0000 7f9a2696c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a200b3340 con 0x7f9a2006ae40 2024-04-06T13:28:29.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.101+0000 7f9a1f7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a2006ae40 0x7f9a200b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:29.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.101+0000 7f9a24f09700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9a2015ddf0 0x7f9a2015e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:29.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.101+0000 7f9a1f7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a2006ae40 0x7f9a200b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:44478/0 (socket says 172.21.15.189:44478) 2024-04-06T13:28:29.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.101+0000 7f9a24f09700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9a2015ddf0 0x7f9a2015e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:40074/0 (socket says 172.21.15.189:40074) 2024-04-06T13:28:29.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.101+0000 7f9a1ffff700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f9a2015d7b0 0x7f9a2006a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:42982/0 (socket says 172.21.15.189:42982) 2024-04-06T13:28:29.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.101+0000 7f9a1f7fe700 1 -- 172.21.15.189:0/46327306 learned_addr learned my addr 172.21.15.189:0/46327306 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:29.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.102+0000 7f9a1f7fe700 1 -- 172.21.15.189:0/46327306 >> v2:172.21.15.192:3300/0 conn(0x7f9a2015ddf0 msgr2=0x7f9a2015e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:29.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.102+0000 7f9a1f7fe700 1 --2- 172.21.15.189:0/46327306 >> v2:172.21.15.192:3300/0 conn(0x7f9a2015ddf0 0x7f9a2015e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:29.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.102+0000 7f9a1f7fe700 1 -- 172.21.15.189:0/46327306 >> v1:172.21.15.192:6789/0 conn(0x7f9a2015d7b0 legacy=0x7f9a2006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:29.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.102+0000 7f9a1f7fe700 1 -- 172.21.15.189:0/46327306 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a200b3620 con 0x7f9a2006ae40 2024-04-06T13:28:29.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.102+0000 7f9a24f09700 1 --2- 172.21.15.189:0/46327306 >> v2:172.21.15.192:3300/0 conn(0x7f9a2015ddf0 0x7f9a2015e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:29.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.102+0000 7f9a1f7fe700 1 --2- 172.21.15.189:0/46327306 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a2006ae40 0x7f9a200b3e20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9a100098f0 tx=0x7f9a10009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d689f4041bc201b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:29.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.102+0000 7f9a1effd700 1 -- 172.21.15.189:0/46327306 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a10011830 con 0x7f9a2006ae40 2024-04-06T13:28:29.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.102+0000 7f9a1effd700 1 -- 172.21.15.189:0/46327306 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a10011990 con 0x7f9a2006ae40 2024-04-06T13:28:29.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.102+0000 7f9a1effd700 1 -- 172.21.15.189:0/46327306 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a10011d20 con 0x7f9a2006ae40 2024-04-06T13:28:29.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.103+0000 7f9a2696c700 1 -- 172.21.15.189:0/46327306 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a2006ae40 msgr2=0x7f9a200b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:29.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.103+0000 7f9a2696c700 1 --2- 172.21.15.189:0/46327306 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a2006ae40 0x7f9a200b3e20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9a100098f0 tx=0x7f9a10009c00 comp rx=0 tx=0).stop 2024-04-06T13:28:29.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.103+0000 7f9a2696c700 1 -- 172.21.15.189:0/46327306 shutdown_connections 2024-04-06T13:28:29.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.103+0000 7f9a2696c700 1 --2- 172.21.15.189:0/46327306 >> v2:172.21.15.192:3300/0 conn(0x7f9a2015ddf0 0x7f9a2015e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:29.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.103+0000 7f9a2696c700 1 --2- 172.21.15.189:0/46327306 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a2006ae40 0x7f9a200b3e20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:29.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.103+0000 7f9a2696c700 1 -- 172.21.15.189:0/46327306 >> 172.21.15.189:0/46327306 conn(0x7f9a200d00b0 msgr2=0x7f9a200d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:29.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.103+0000 7f9a2696c700 1 -- 172.21.15.189:0/46327306 shutdown_connections 2024-04-06T13:28:29.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.103+0000 7f9a2696c700 1 -- 172.21.15.189:0/46327306 wait complete. 2024-04-06T13:28:29.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.104+0000 7f9a2696c700 1 Processor -- start 2024-04-06T13:28:29.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.105+0000 7f9a2696c700 1 -- start start 2024-04-06T13:28:29.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a2696c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a2006ae40 0x7f9a20143b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:29.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a2696c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a2015d7b0 0x7f9a20154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:29.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a2696c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a2015ddf0 0x7f9a201f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:29.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a2696c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a201561a0 con 0x7f9a2015d7b0 2024-04-06T13:28:29.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a2696c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a20155590 con 0x7f9a2006ae40 2024-04-06T13:28:29.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a2696c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a20155870 con 0x7f9a2015ddf0 2024-04-06T13:28:29.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a1f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a2015d7b0 0x7f9a20154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:29.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a1f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a2015d7b0 0x7f9a20154ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55886/0 (socket says 172.21.15.189:55886) 2024-04-06T13:28:29.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a1f7fe700 1 -- 172.21.15.189:0/854877570 learned_addr learned my addr 172.21.15.189:0/854877570 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:29.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a1ffff700 1 --2- 172.21.15.189:0/854877570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a2006ae40 0x7f9a20143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:29.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a24f09700 1 --2- 172.21.15.189:0/854877570 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a2015ddf0 0x7f9a201f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:29.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a1f7fe700 1 -- 172.21.15.189:0/854877570 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a2015ddf0 msgr2=0x7f9a201f5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:29.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a1f7fe700 1 --2- 172.21.15.189:0/854877570 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a2015ddf0 0x7f9a201f5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:29.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a1f7fe700 1 -- 172.21.15.189:0/854877570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a2006ae40 msgr2=0x7f9a20143b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:29.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a1f7fe700 1 --2- 172.21.15.189:0/854877570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a2006ae40 0x7f9a20143b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:29.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a1f7fe700 1 -- 172.21.15.189:0/854877570 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a18009a70 con 0x7f9a2015d7b0 2024-04-06T13:28:29.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a1f7fe700 1 --2- 172.21.15.189:0/854877570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a2015d7b0 0x7f9a20154ab0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f9a10000c00 tx=0x7f9a10004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:29.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a1cff9700 1 -- 172.21.15.189:0/854877570 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a100043c0 con 0x7f9a2015d7b0 2024-04-06T13:28:29.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a1cff9700 1 -- 172.21.15.189:0/854877570 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a1001f070 con 0x7f9a2015d7b0 2024-04-06T13:28:29.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a2696c700 1 -- 172.21.15.189:0/854877570 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a10009550 con 0x7f9a2015d7b0 2024-04-06T13:28:29.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a1cff9700 1 -- 172.21.15.189:0/854877570 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a100036a0 con 0x7f9a2015d7b0 2024-04-06T13:28:29.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.106+0000 7f9a2696c700 1 -- 172.21.15.189:0/854877570 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a20155e30 con 0x7f9a2015d7b0 2024-04-06T13:28:29.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.108+0000 7f9a2696c700 1 -- 172.21.15.189:0/854877570 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a20067bb0 con 0x7f9a2015d7b0 2024-04-06T13:28:29.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.108+0000 7f9a1cff9700 1 -- 172.21.15.189:0/854877570 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 97) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f9a10021030 con 0x7f9a2015d7b0 2024-04-06T13:28:29.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.108+0000 7f9a1cff9700 1 --2- 172.21.15.189:0/854877570 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9a0c06aae0 0x7f9a0c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:29.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.108+0000 7f9a1cff9700 1 -- 172.21.15.189:0/854877570 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f9a1008a770 con 0x7f9a2015d7b0 2024-04-06T13:28:29.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.108+0000 7f9a1ffff700 1 --2- 172.21.15.189:0/854877570 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9a0c06aae0 0x7f9a0c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:29.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.109+0000 7f9a1ffff700 1 --2- 172.21.15.189:0/854877570 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9a0c06aae0 0x7f9a0c06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9a14000f60 tx=0x7f9a14007ed0 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:29.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.113+0000 7f9a1cff9700 1 -- 172.21.15.189:0/854877570 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9a10055840 con 0x7f9a2015d7b0 2024-04-06T13:28:29.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:29.309+0000 7f9a2696c700 1 -- 172.21.15.189:0/854877570 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9a20074990 con 0x7f9a2015d7b0 2024-04-06T13:28:30.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.291+0000 7f9a1cff9700 1 -- 172.21.15.189:0/854877570 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 98) v1 ==== 89191+0+0 (secure 0 0 0) 0x7f9a1008ccc0 con 0x7f9a2015d7b0 2024-04-06T13:28:30.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.293+0000 7f9a1cff9700 1 -- 172.21.15.189:0/854877570 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v168) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9a100592d0 con 0x7f9a2015d7b0 2024-04-06T13:28:30.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.297+0000 7f9a2696c700 1 -- 172.21.15.189:0/854877570 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9a0c06aae0 msgr2=0x7f9a0c06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:30.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.297+0000 7f9a2696c700 1 --2- 172.21.15.189:0/854877570 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9a0c06aae0 0x7f9a0c06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9a14000f60 tx=0x7f9a14007ed0 comp rx=0 tx=0).stop 2024-04-06T13:28:30.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.297+0000 7f9a2696c700 1 -- 172.21.15.189:0/854877570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a2015d7b0 msgr2=0x7f9a20154ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:30.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.297+0000 7f9a2696c700 1 --2- 172.21.15.189:0/854877570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a2015d7b0 0x7f9a20154ab0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f9a10000c00 tx=0x7f9a10004750 comp rx=0 tx=0).stop 2024-04-06T13:28:30.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.297+0000 7f9a2696c700 1 -- 172.21.15.189:0/854877570 shutdown_connections 2024-04-06T13:28:30.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.297+0000 7f9a2696c700 1 --2- 172.21.15.189:0/854877570 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a2015ddf0 0x7f9a201f5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:30.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.297+0000 7f9a2696c700 1 --2- 172.21.15.189:0/854877570 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9a0c06aae0 0x7f9a0c06cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:30.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.297+0000 7f9a2696c700 1 --2- 172.21.15.189:0/854877570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a2006ae40 0x7f9a20143b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:30.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.297+0000 7f9a2696c700 1 --2- 172.21.15.189:0/854877570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a2015d7b0 0x7f9a20154ab0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:30.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.297+0000 7f9a2696c700 1 -- 172.21.15.189:0/854877570 >> 172.21.15.189:0/854877570 conn(0x7f9a200d00b0 msgr2=0x7f9a2006de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:30.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.297+0000 7f9a2696c700 1 -- 172.21.15.189:0/854877570 shutdown_connections 2024-04-06T13:28:30.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.297+0000 7f9a2696c700 1 -- 172.21.15.189:0/854877570 wait complete. 2024-04-06T13:28:30.298 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' removed 2024-04-06T13:28:30.312 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-06T13:28:30.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.444+0000 7f11ed800700 1 Processor -- start 2024-04-06T13:28:30.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.444+0000 7f11ed800700 1 -- start start 2024-04-06T13:28:30.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.444+0000 7f11ed800700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11e80ce730 0x7f11e815dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:30.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.444+0000 7f11ed800700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f11e80b8d70 0x7f11e80d1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:30.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11ed800700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e815eac0 con 0x7f11e815d6f0 2024-04-06T13:28:30.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11ed800700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11e815e010 con 0x7f11e80b8d70 2024-04-06T13:28:30.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11ed800700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e815e220 con 0x7f11e80ce730 2024-04-06T13:28:30.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e6ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f11e815d6f0 0x7f11e80ce020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40910/0 (socket says 172.21.15.189:40910) 2024-04-06T13:28:30.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e6ffd700 1 -- 172.21.15.189:0/1736426183 learned_addr learned my addr 172.21.15.189:0/1736426183 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:30.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e67fc700 1 --2- 172.21.15.189:0/1736426183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11e80ce730 0x7f11e815dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:30.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e77fe700 1 --2- 172.21.15.189:0/1736426183 >> v2:172.21.15.192:3300/0 conn(0x7f11e80b8d70 0x7f11e80d1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:30.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e5ffb700 1 -- 172.21.15.189:0/1736426183 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916535879 0 0) 0x7f11e815eac0 con 0x7f11e815d6f0 2024-04-06T13:28:30.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e5ffb700 1 -- 172.21.15.189:0/1736426183 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d00035b0 con 0x7f11e815d6f0 2024-04-06T13:28:30.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e5ffb700 1 -- 172.21.15.189:0/1736426183 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2628905421 0 0) 0x7f11d00035b0 con 0x7f11e815d6f0 2024-04-06T13:28:30.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e5ffb700 1 -- 172.21.15.189:0/1736426183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11e80ce730 msgr2=0x7f11e815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:30.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e5ffb700 1 --2- 172.21.15.189:0/1736426183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11e80ce730 0x7f11e815dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:30.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e5ffb700 1 -- 172.21.15.189:0/1736426183 >> v2:172.21.15.192:3300/0 conn(0x7f11e80b8d70 msgr2=0x7f11e80d1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:30.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e5ffb700 1 --2- 172.21.15.189:0/1736426183 >> v2:172.21.15.192:3300/0 conn(0x7f11e80b8d70 0x7f11e80d1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:30.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e5ffb700 1 -- 172.21.15.189:0/1736426183 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e815e500 con 0x7f11e815d6f0 2024-04-06T13:28:30.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e5ffb700 1 -- 172.21.15.189:0/1736426183 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f11d8001670 con 0x7f11e815d6f0 2024-04-06T13:28:30.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e67fc700 1 --2- 172.21.15.189:0/1736426183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11e80ce730 0x7f11e815dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:30.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e5ffb700 1 -- 172.21.15.189:0/1736426183 >> v1:172.21.15.189:6789/0 conn(0x7f11e815d6f0 legacy=0x7f11e80ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:30.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e5ffb700 1 --2- 172.21.15.189:0/1736426183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11d0003a00 0x7f11d0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:30.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e5ffb700 1 --2- 172.21.15.189:0/1736426183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11d0004850 0x7f11d0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:30.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e5ffb700 1 --2- 172.21.15.189:0/1736426183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11e80ce730 0x7f11d0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:30.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e5ffb700 1 -- 172.21.15.189:0/1736426183 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e815e010 con 0x7f11d0004850 2024-04-06T13:28:30.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e5ffb700 1 -- 172.21.15.189:0/1736426183 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e815e220 con 0x7f11d0003a00 2024-04-06T13:28:30.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e5ffb700 1 -- 172.21.15.189:0/1736426183 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d000ba40 con 0x7f11e80ce730 2024-04-06T13:28:30.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e67fc700 1 --2- 172.21.15.189:0/1736426183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11d0004850 0x7f11d0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:30.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e6ffd700 1 --2- 172.21.15.189:0/1736426183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11d0003a00 0x7f11d0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:30.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.445+0000 7f11e77fe700 1 --2- 172.21.15.189:0/1736426183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11e80ce730 0x7f11d0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:30.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.446+0000 7f11e67fc700 1 -- 172.21.15.189:0/1736426183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11e80ce730 msgr2=0x7f11d0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:30.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.446+0000 7f11e67fc700 1 --2- 172.21.15.189:0/1736426183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11e80ce730 0x7f11d0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:30.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.446+0000 7f11e67fc700 1 -- 172.21.15.189:0/1736426183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11d0003a00 msgr2=0x7f11d0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:30.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.446+0000 7f11e67fc700 1 --2- 172.21.15.189:0/1736426183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11d0003a00 0x7f11d0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:30.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.446+0000 7f11e67fc700 1 -- 172.21.15.189:0/1736426183 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc0094a0 con 0x7f11d0004850 2024-04-06T13:28:30.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.446+0000 7f11e77fe700 1 --2- 172.21.15.189:0/1736426183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11e80ce730 0x7f11d0005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:28:30.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.446+0000 7f11e67fc700 1 --2- 172.21.15.189:0/1736426183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11d0004850 0x7f11d0004cd0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f11dc009d30 tx=0x7f11d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef776395d07c5321 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:30.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.446+0000 7f11e5ffb700 1 -- 172.21.15.189:0/1736426183 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11dc009e80 con 0x7f11d0004850 2024-04-06T13:28:30.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.446+0000 7f11e5ffb700 1 -- 172.21.15.189:0/1736426183 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11dc00e940 con 0x7f11d0004850 2024-04-06T13:28:30.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.446+0000 7f11e5ffb700 1 -- 172.21.15.189:0/1736426183 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11dc00ecb0 con 0x7f11d0004850 2024-04-06T13:28:30.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.446+0000 7f11ed800700 1 -- 172.21.15.189:0/1736426183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11d0004850 msgr2=0x7f11d0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:30.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.446+0000 7f11ed800700 1 --2- 172.21.15.189:0/1736426183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11d0004850 0x7f11d0004cd0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f11dc009d30 tx=0x7f11d0005880 comp rx=0 tx=0).stop 2024-04-06T13:28:30.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.446+0000 7f11e77fe700 1 -- 172.21.15.189:0/1736426183 reap_dead start 2024-04-06T13:28:30.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.446+0000 7f11ed800700 1 -- 172.21.15.189:0/1736426183 shutdown_connections 2024-04-06T13:28:30.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.446+0000 7f11ed800700 1 -- 172.21.15.189:0/1736426183 >> 172.21.15.189:0/1736426183 conn(0x7f11e8072490 msgr2=0x7f11e806b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:30.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.447+0000 7f11ed800700 1 -- 172.21.15.189:0/1736426183 shutdown_connections 2024-04-06T13:28:30.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.447+0000 7f11ed800700 1 -- 172.21.15.189:0/1736426183 wait complete. 2024-04-06T13:28:30.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.448+0000 7f11ed800700 1 Processor -- start 2024-04-06T13:28:30.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.448+0000 7f11ed800700 1 -- start start 2024-04-06T13:28:30.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11ed800700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11d0004850 0x7f11e80b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:30.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11ed800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11e814f280 0x7f11e80ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:30.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11ed800700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11e814f700 0x7f11e81f5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:30.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11ed800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e80cae00 con 0x7f11e814f280 2024-04-06T13:28:30.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11ed800700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e80cb0e0 con 0x7f11d0004850 2024-04-06T13:28:30.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11ed800700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e80cb3c0 con 0x7f11e814f700 2024-04-06T13:28:30.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11e67fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11e814f280 0x7f11e80ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:30.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11e67fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11e814f280 0x7f11e80ca4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35294/0 (socket says 172.21.15.189:35294) 2024-04-06T13:28:30.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11e67fc700 1 -- 172.21.15.189:0/2066729854 learned_addr learned my addr 172.21.15.189:0/2066729854 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:30.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11e6ffd700 1 --2- 172.21.15.189:0/2066729854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11d0004850 0x7f11e80b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:30.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11e77fe700 1 --2- 172.21.15.189:0/2066729854 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11e814f700 0x7f11e81f5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:30.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11e67fc700 1 -- 172.21.15.189:0/2066729854 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11e814f700 msgr2=0x7f11e81f5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:30.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11e67fc700 1 --2- 172.21.15.189:0/2066729854 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11e814f700 0x7f11e81f5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:30.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11e67fc700 1 -- 172.21.15.189:0/2066729854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11d0004850 msgr2=0x7f11e80b5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:30.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11e67fc700 1 --2- 172.21.15.189:0/2066729854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11d0004850 0x7f11e80b5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:30.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11e67fc700 1 -- 172.21.15.189:0/2066729854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e80cb6a0 con 0x7f11e814f280 2024-04-06T13:28:30.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11e67fc700 1 --2- 172.21.15.189:0/2066729854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11e814f280 0x7f11e80ca4b0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f11d00057d0 tx=0x7f11dc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:30.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11cf7fe700 1 -- 172.21.15.189:0/2066729854 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11dc003ab0 con 0x7f11e814f280 2024-04-06T13:28:30.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11cf7fe700 1 -- 172.21.15.189:0/2066729854 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11dc003c30 con 0x7f11e814f280 2024-04-06T13:28:30.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11cf7fe700 1 -- 172.21.15.189:0/2066729854 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11dc00feb0 con 0x7f11e814f280 2024-04-06T13:28:30.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.449+0000 7f11ed800700 1 -- 172.21.15.189:0/2066729854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e80cb980 con 0x7f11e814f280 2024-04-06T13:28:30.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.450+0000 7f11ed800700 1 -- 172.21.15.189:0/2066729854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e80ca9f0 con 0x7f11e814f280 2024-04-06T13:28:30.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.450+0000 7f11ed800700 1 -- 172.21.15.189:0/2066729854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e8074990 con 0x7f11e814f280 2024-04-06T13:28:30.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.451+0000 7f11cf7fe700 1 -- 172.21.15.189:0/2066729854 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 98) v1 ==== 89191+0+0 (secure 0 0 0) 0x7f11dc013470 con 0x7f11e814f280 2024-04-06T13:28:30.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.451+0000 7f11cf7fe700 1 --2- 172.21.15.189:0/2066729854 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f11d406ad20 0x7f11d406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:30.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.451+0000 7f11e6ffd700 1 --2- 172.21.15.189:0/2066729854 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f11d406ad20 0x7f11d406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:30.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.451+0000 7f11cf7fe700 1 -- 172.21.15.189:0/2066729854 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6323+0+0 (secure 0 0 0) 0x7f11dc08ad50 con 0x7f11e814f280 2024-04-06T13:28:30.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.452+0000 7f11e6ffd700 1 --2- 172.21.15.189:0/2066729854 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f11d406ad20 0x7f11d406d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f11d8001550 tx=0x7f11d8007980 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:30.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.456+0000 7f11cf7fe700 1 -- 172.21.15.189:0/2066729854 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f11dc055d60 con 0x7f11e814f280 2024-04-06T13:28:30.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:30.649+0000 7f11ed800700 1 -- 172.21.15.189:0/2066729854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f11e80d1350 con 0x7f11e814f280 2024-04-06T13:28:31.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.382+0000 7f11cf7fe700 1 -- 172.21.15.189:0/2066729854 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f11dc056130 con 0x7f11e814f280 2024-04-06T13:28:31.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.383+0000 7f11cf7fe700 1 -- 172.21.15.189:0/2066729854 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v169) v1 ==== 169+0+0 (secure 0 0 0) 0x7f11dc0318c0 con 0x7f11e814f280 2024-04-06T13:28:31.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.386+0000 7f11ed800700 1 -- 172.21.15.189:0/2066729854 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f11d406ad20 msgr2=0x7f11d406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:31.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.386+0000 7f11ed800700 1 --2- 172.21.15.189:0/2066729854 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f11d406ad20 0x7f11d406d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f11d8001550 tx=0x7f11d8007980 comp rx=0 tx=0).stop 2024-04-06T13:28:31.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.386+0000 7f11ed800700 1 -- 172.21.15.189:0/2066729854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11e814f280 msgr2=0x7f11e80ca4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:31.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.386+0000 7f11ed800700 1 --2- 172.21.15.189:0/2066729854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11e814f280 0x7f11e80ca4b0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f11d00057d0 tx=0x7f11dc0047c0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.386+0000 7f11ed800700 1 -- 172.21.15.189:0/2066729854 shutdown_connections 2024-04-06T13:28:31.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.386+0000 7f11ed800700 1 --2- 172.21.15.189:0/2066729854 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11e814f700 0x7f11e81f5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.386+0000 7f11ed800700 1 --2- 172.21.15.189:0/2066729854 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f11d406ad20 0x7f11d406d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.386+0000 7f11ed800700 1 --2- 172.21.15.189:0/2066729854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11d0004850 0x7f11e80b5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.386+0000 7f11ed800700 1 --2- 172.21.15.189:0/2066729854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11e814f280 0x7f11e80ca4b0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.386+0000 7f11ed800700 1 -- 172.21.15.189:0/2066729854 >> 172.21.15.189:0/2066729854 conn(0x7f11e8072490 msgr2=0x7f11e80d27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:31.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.386+0000 7f11ed800700 1 -- 172.21.15.189:0/2066729854 shutdown_connections 2024-04-06T13:28:31.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.386+0000 7f11ed800700 1 -- 172.21.15.189:0/2066729854 wait complete. 2024-04-06T13:28:31.387 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' removed 2024-04-06T13:28:31.399 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-06T13:28:31.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.525+0000 7f0a4afb5700 1 Processor -- start 2024-04-06T13:28:31.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.525+0000 7f0a4afb5700 1 -- start start 2024-04-06T13:28:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.526+0000 7f0a4afb5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a4406ae40 0x7f0a440b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.526+0000 7f0a4afb5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0a4415ddf0 0x7f0a4415e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.526+0000 7f0a4afb5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a440b3990 con 0x7f0a4415d7b0 2024-04-06T13:28:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.526+0000 7f0a4afb5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a440b3060 con 0x7f0a4415ddf0 2024-04-06T13:28:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.526+0000 7f0a4afb5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a440b3340 con 0x7f0a4406ae40 2024-04-06T13:28:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.526+0000 7f0a49552700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0a4415ddf0 0x7f0a4415e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.526+0000 7f0a48d51700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0a4415d7b0 0x7f0a4406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40914/0 (socket says 172.21.15.189:40914) 2024-04-06T13:28:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.526+0000 7f0a49552700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0a4415ddf0 0x7f0a4415e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35310/0 (socket says 172.21.15.189:35310) 2024-04-06T13:28:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.526+0000 7f0a48d51700 1 -- 172.21.15.189:0/812450398 learned_addr learned my addr 172.21.15.189:0/812450398 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.526+0000 7f0a43fff700 1 --2- 172.21.15.189:0/812450398 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a4406ae40 0x7f0a440b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.526+0000 7f0a49552700 1 -- 172.21.15.189:0/812450398 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a4406ae40 msgr2=0x7f0a440b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:31.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.526+0000 7f0a437fe700 1 -- 172.21.15.189:0/812450398 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456840687 0 0) 0x7f0a440b3990 con 0x7f0a4415d7b0 2024-04-06T13:28:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.527+0000 7f0a49552700 1 --2- 172.21.15.189:0/812450398 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a4406ae40 0x7f0a440b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.527+0000 7f0a49552700 1 -- 172.21.15.189:0/812450398 >> v1:172.21.15.189:6789/0 conn(0x7f0a4415d7b0 legacy=0x7f0a4406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.527+0000 7f0a49552700 1 -- 172.21.15.189:0/812450398 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a440b3620 con 0x7f0a4415ddf0 2024-04-06T13:28:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.527+0000 7f0a43fff700 1 --2- 172.21.15.189:0/812450398 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a4406ae40 0x7f0a440b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:28:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.527+0000 7f0a49552700 1 --2- 172.21.15.189:0/812450398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a4415ddf0 0x7f0a4415e2a0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c009d90 tx=0x7f0a3c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f85da75fb7de6df server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.527+0000 7f0a437fe700 1 -- 172.21.15.189:0/812450398 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c00e070 con 0x7f0a4415ddf0 2024-04-06T13:28:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.527+0000 7f0a437fe700 1 -- 172.21.15.189:0/812450398 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3c00bbd0 con 0x7f0a4415ddf0 2024-04-06T13:28:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.527+0000 7f0a437fe700 1 -- 172.21.15.189:0/812450398 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c012430 con 0x7f0a4415ddf0 2024-04-06T13:28:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.527+0000 7f0a4afb5700 1 -- 172.21.15.189:0/812450398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a4415ddf0 msgr2=0x7f0a4415e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.527+0000 7f0a4afb5700 1 --2- 172.21.15.189:0/812450398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a4415ddf0 0x7f0a4415e2a0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c009d90 tx=0x7f0a3c00b330 comp rx=0 tx=0).stop 2024-04-06T13:28:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.527+0000 7f0a4afb5700 1 -- 172.21.15.189:0/812450398 shutdown_connections 2024-04-06T13:28:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.527+0000 7f0a4afb5700 1 --2- 172.21.15.189:0/812450398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a4415ddf0 0x7f0a4415e2a0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.527+0000 7f0a4afb5700 1 --2- 172.21.15.189:0/812450398 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a4406ae40 0x7f0a440b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.527+0000 7f0a4afb5700 1 -- 172.21.15.189:0/812450398 >> 172.21.15.189:0/812450398 conn(0x7f0a440d00b0 msgr2=0x7f0a440d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:31.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.527+0000 7f0a4afb5700 1 -- 172.21.15.189:0/812450398 shutdown_connections 2024-04-06T13:28:31.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.527+0000 7f0a4afb5700 1 -- 172.21.15.189:0/812450398 wait complete. 2024-04-06T13:28:31.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.529+0000 7f0a4afb5700 1 Processor -- start 2024-04-06T13:28:31.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.529+0000 7f0a4afb5700 1 -- start start 2024-04-06T13:28:31.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.529+0000 7f0a4afb5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a4406ae40 0x7f0a440cc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:31.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.529+0000 7f0a4afb5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a4415d7b0 0x7f0a440c50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:31.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.529+0000 7f0a4afb5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a4415ddf0 0x7f0a440afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:31.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.529+0000 7f0a4afb5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a440c5630 con 0x7f0a4415ddf0 2024-04-06T13:28:31.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.529+0000 7f0a4afb5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a440ccf20 con 0x7f0a4406ae40 2024-04-06T13:28:31.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.529+0000 7f0a4afb5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a440cd200 con 0x7f0a4415d7b0 2024-04-06T13:28:31.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.530+0000 7f0a49552700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a4415ddf0 0x7f0a440afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:31.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.530+0000 7f0a49552700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a4415ddf0 0x7f0a440afbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35312/0 (socket says 172.21.15.189:35312) 2024-04-06T13:28:31.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.530+0000 7f0a49552700 1 -- 172.21.15.189:0/2083588220 learned_addr learned my addr 172.21.15.189:0/2083588220 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:31.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.530+0000 7f0a43fff700 1 --2- 172.21.15.189:0/2083588220 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a4415d7b0 0x7f0a440c50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:31.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.530+0000 7f0a49552700 1 -- 172.21.15.189:0/2083588220 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a4415d7b0 msgr2=0x7f0a440c50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:31.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.530+0000 7f0a49552700 1 --2- 172.21.15.189:0/2083588220 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a4415d7b0 0x7f0a440c50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.530+0000 7f0a49552700 1 -- 172.21.15.189:0/2083588220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a4406ae40 msgr2=0x7f0a440cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:31.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.530+0000 7f0a48d51700 1 --2- 172.21.15.189:0/2083588220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a4406ae40 0x7f0a440cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:31.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.530+0000 7f0a49552700 1 --2- 172.21.15.189:0/2083588220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a4406ae40 0x7f0a440cc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.530+0000 7f0a49552700 1 -- 172.21.15.189:0/2083588220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3c009a70 con 0x7f0a4415ddf0 2024-04-06T13:28:31.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.530+0000 7f0a48d51700 1 --2- 172.21.15.189:0/2083588220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a4406ae40 0x7f0a440cc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:28:31.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.530+0000 7f0a49552700 1 --2- 172.21.15.189:0/2083588220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a4415ddf0 0x7f0a440afbd0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f0a4406c820 tx=0x7f0a3c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:31.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.530+0000 7f0a417fa700 1 -- 172.21.15.189:0/2083588220 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c0098f0 con 0x7f0a4415ddf0 2024-04-06T13:28:31.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.530+0000 7f0a417fa700 1 -- 172.21.15.189:0/2083588220 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3c020070 con 0x7f0a4415ddf0 2024-04-06T13:28:31.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.530+0000 7f0a4afb5700 1 -- 172.21.15.189:0/2083588220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a440cd4e0 con 0x7f0a4415ddf0 2024-04-06T13:28:31.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.530+0000 7f0a417fa700 1 -- 172.21.15.189:0/2083588220 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c004260 con 0x7f0a4415ddf0 2024-04-06T13:28:31.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.530+0000 7f0a4afb5700 1 -- 172.21.15.189:0/2083588220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a440cdaa0 con 0x7f0a4415ddf0 2024-04-06T13:28:31.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.531+0000 7f0a4afb5700 1 -- 172.21.15.189:0/2083588220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a44067bb0 con 0x7f0a4415ddf0 2024-04-06T13:28:31.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.532+0000 7f0a417fa700 1 -- 172.21.15.189:0/2083588220 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0a3c0043c0 con 0x7f0a4415ddf0 2024-04-06T13:28:31.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.532+0000 7f0a417fa700 1 --2- 172.21.15.189:0/2083588220 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0a2806adb0 0x7f0a2806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:31.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.532+0000 7f0a417fa700 1 -- 172.21.15.189:0/2083588220 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5909+0+0 (secure 0 0 0) 0x7f0a3c022070 con 0x7f0a4415ddf0 2024-04-06T13:28:31.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.532+0000 7f0a48d51700 1 --2- 172.21.15.189:0/2083588220 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0a2806adb0 0x7f0a2806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:31.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.532+0000 7f0a48d51700 1 --2- 172.21.15.189:0/2083588220 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0a2806adb0 0x7f0a2806d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0a38000f60 tx=0x7f0a38009040 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:31.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.536+0000 7f0a417fa700 1 -- 172.21.15.189:0/2083588220 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0a3c056950 con 0x7f0a4415ddf0 2024-04-06T13:28:31.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.740+0000 7f0a4afb5700 1 -- 172.21.15.189:0/2083588220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0a440ae760 con 0x7f0a4415ddf0 2024-04-06T13:28:31.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.744+0000 7f0a417fa700 1 -- 172.21.15.189:0/2083588220 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v55) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0a3c059f70 con 0x7f0a4415ddf0 2024-04-06T13:28:31.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.746+0000 7f0a4afb5700 1 -- 172.21.15.189:0/2083588220 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0a2806adb0 msgr2=0x7f0a2806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:31.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.746+0000 7f0a4afb5700 1 --2- 172.21.15.189:0/2083588220 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0a2806adb0 0x7f0a2806d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0a38000f60 tx=0x7f0a38009040 comp rx=0 tx=0).stop 2024-04-06T13:28:31.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.746+0000 7f0a4afb5700 1 -- 172.21.15.189:0/2083588220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a4415ddf0 msgr2=0x7f0a440afbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:31.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.746+0000 7f0a4afb5700 1 --2- 172.21.15.189:0/2083588220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a4415ddf0 0x7f0a440afbd0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f0a4406c820 tx=0x7f0a3c004ad0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.746+0000 7f0a4afb5700 1 -- 172.21.15.189:0/2083588220 shutdown_connections 2024-04-06T13:28:31.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.746+0000 7f0a4afb5700 1 --2- 172.21.15.189:0/2083588220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a4415ddf0 0x7f0a440afbd0 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.746+0000 7f0a4afb5700 1 --2- 172.21.15.189:0/2083588220 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0a2806adb0 0x7f0a2806d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.746+0000 7f0a4afb5700 1 --2- 172.21.15.189:0/2083588220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a4406ae40 0x7f0a440cc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.746+0000 7f0a4afb5700 1 --2- 172.21.15.189:0/2083588220 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0a4415d7b0 0x7f0a440c50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.746+0000 7f0a4afb5700 1 -- 172.21.15.189:0/2083588220 >> 172.21.15.189:0/2083588220 conn(0x7f0a440d00b0 msgr2=0x7f0a44159580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:31.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.746+0000 7f0a4afb5700 1 -- 172.21.15.189:0/2083588220 shutdown_connections 2024-04-06T13:28:31.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.746+0000 7f0a4afb5700 1 -- 172.21.15.189:0/2083588220 wait complete. 2024-04-06T13:28:31.758 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-06T13:28:31.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.882+0000 7f03e6a10700 1 Processor -- start 2024-04-06T13:28:31.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.882+0000 7f03e6a10700 1 -- start start 2024-04-06T13:28:31.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.883+0000 7f03e6a10700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f03e015a550 0x7f03e00cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:31.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.883+0000 7f03e6a10700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03e00b4d70 0x7f03e00b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:31.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.883+0000 7f03e6a10700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03e00cdbe0 con 0x7f03e00cf900 2024-04-06T13:28:31.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.883+0000 7f03e6a10700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03e015a940 con 0x7f03e015a550 2024-04-06T13:28:31.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.883+0000 7f03e6a10700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e015ac00 con 0x7f03e00b4d70 2024-04-06T13:28:31.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.883+0000 7f03dffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f03e015a550 0x7f03e00cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:31.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.883+0000 7f03dffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f03e015a550 0x7f03e00cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35314/0 (socket says 172.21.15.189:35314) 2024-04-06T13:28:31.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.883+0000 7f03dffff700 1 -- 172.21.15.189:0/2795599095 learned_addr learned my addr 172.21.15.189:0/2795599095 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:31.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.883+0000 7f03e4fad700 1 --2- 172.21.15.189:0/2795599095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03e00b4d70 0x7f03e00b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:31.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.883+0000 7f03dffff700 1 -- 172.21.15.189:0/2795599095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03e00b4d70 msgr2=0x7f03e00b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:31.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.883+0000 7f03dffff700 1 --2- 172.21.15.189:0/2795599095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03e00b4d70 0x7f03e00b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.883+0000 7f03dffff700 1 -- 172.21.15.189:0/2795599095 >> v1:172.21.15.192:6789/0 conn(0x7f03e00cf900 legacy=0x7f03e00b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:31.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.883+0000 7f03dffff700 1 -- 172.21.15.189:0/2795599095 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e015aee0 con 0x7f03e015a550 2024-04-06T13:28:31.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.883+0000 7f03dffff700 1 --2- 172.21.15.189:0/2795599095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03e015a550 0x7f03e00cf3c0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f03d400bb90 tx=0x7f03d400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=753f6eff5a2494de server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:31.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.883+0000 7f03deffd700 1 -- 172.21.15.189:0/2795599095 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d400dd50 con 0x7f03e015a550 2024-04-06T13:28:31.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.883+0000 7f03deffd700 1 -- 172.21.15.189:0/2795599095 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d4010470 con 0x7f03e015a550 2024-04-06T13:28:31.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.883+0000 7f03deffd700 1 -- 172.21.15.189:0/2795599095 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d4010800 con 0x7f03e015a550 2024-04-06T13:28:31.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.884+0000 7f03e6a10700 1 -- 172.21.15.189:0/2795599095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03e015a550 msgr2=0x7f03e00cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:31.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.884+0000 7f03e6a10700 1 --2- 172.21.15.189:0/2795599095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03e015a550 0x7f03e00cf3c0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f03d400bb90 tx=0x7f03d400bea0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.884+0000 7f03e6a10700 1 -- 172.21.15.189:0/2795599095 shutdown_connections 2024-04-06T13:28:31.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.884+0000 7f03e6a10700 1 --2- 172.21.15.189:0/2795599095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03e00b4d70 0x7f03e00b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.884+0000 7f03e6a10700 1 --2- 172.21.15.189:0/2795599095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03e015a550 0x7f03e00cf3c0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.884+0000 7f03e6a10700 1 -- 172.21.15.189:0/2795599095 >> 172.21.15.189:0/2795599095 conn(0x7f03e00d00b0 msgr2=0x7f03e00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:31.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.884+0000 7f03e6a10700 1 -- 172.21.15.189:0/2795599095 shutdown_connections 2024-04-06T13:28:31.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.884+0000 7f03e6a10700 1 -- 172.21.15.189:0/2795599095 wait complete. 2024-04-06T13:28:31.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.885+0000 7f03e6a10700 1 Processor -- start 2024-04-06T13:28:31.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.885+0000 7f03e6a10700 1 -- start start 2024-04-06T13:28:31.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.886+0000 7f03e6a10700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03e00b4d70 0x7f03e00ad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:31.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.886+0000 7f03e6a10700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03e00cf900 0x7f03e01f4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:31.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.886+0000 7f03e6a10700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03e0152c00 0x7f03e0157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:31.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.886+0000 7f03e6a10700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e01f45b0 con 0x7f03e00cf900 2024-04-06T13:28:31.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.886+0000 7f03e6a10700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e01f4810 con 0x7f03e0152c00 2024-04-06T13:28:31.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.886+0000 7f03e6a10700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e01f4af0 con 0x7f03e00b4d70 2024-04-06T13:28:31.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.886+0000 7f03df7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03e00cf900 0x7f03e01f4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:31.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.886+0000 7f03df7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03e00cf900 0x7f03e01f4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35330/0 (socket says 172.21.15.189:35330) 2024-04-06T13:28:31.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.886+0000 7f03df7fe700 1 -- 172.21.15.189:0/55275704 learned_addr learned my addr 172.21.15.189:0/55275704 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:31.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.886+0000 7f03dffff700 1 --2- 172.21.15.189:0/55275704 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03e00b4d70 0x7f03e00ad3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:31.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.886+0000 7f03e4fad700 1 --2- 172.21.15.189:0/55275704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03e0152c00 0x7f03e0157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:31.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.886+0000 7f03df7fe700 1 -- 172.21.15.189:0/55275704 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03e00b4d70 msgr2=0x7f03e00ad3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:31.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.886+0000 7f03df7fe700 1 --2- 172.21.15.189:0/55275704 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03e00b4d70 0x7f03e00ad3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.886+0000 7f03df7fe700 1 -- 172.21.15.189:0/55275704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03e0152c00 msgr2=0x7f03e0157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:31.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.886+0000 7f03df7fe700 1 --2- 172.21.15.189:0/55275704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03e0152c00 0x7f03e0157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:31.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.886+0000 7f03df7fe700 1 -- 172.21.15.189:0/55275704 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d400b7f0 con 0x7f03e00cf900 2024-04-06T13:28:31.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.887+0000 7f03df7fe700 1 --2- 172.21.15.189:0/55275704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03e00cf900 0x7f03e01f4070 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f03d000c4e0 tx=0x7f03d000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:31.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.887+0000 7f03dcff9700 1 -- 172.21.15.189:0/55275704 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d00052d0 con 0x7f03e00cf900 2024-04-06T13:28:31.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.887+0000 7f03dcff9700 1 -- 172.21.15.189:0/55275704 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d0005430 con 0x7f03e00cf900 2024-04-06T13:28:31.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.887+0000 7f03dcff9700 1 -- 172.21.15.189:0/55275704 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d00038c0 con 0x7f03e00cf900 2024-04-06T13:28:31.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.887+0000 7f03e6a10700 1 -- 172.21.15.189:0/55275704 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e01f4dd0 con 0x7f03e00cf900 2024-04-06T13:28:31.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.887+0000 7f03e6a10700 1 -- 172.21.15.189:0/55275704 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e01f5390 con 0x7f03e00cf900 2024-04-06T13:28:31.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.888+0000 7f03e6a10700 1 -- 172.21.15.189:0/55275704 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e0067bb0 con 0x7f03e00cf900 2024-04-06T13:28:31.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.888+0000 7f03dcff9700 1 -- 172.21.15.189:0/55275704 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f03d0003a20 con 0x7f03e00cf900 2024-04-06T13:28:31.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.889+0000 7f03dcff9700 1 --2- 172.21.15.189:0/55275704 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f03cc06adb0 0x7f03cc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:31.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.889+0000 7f03dcff9700 1 -- 172.21.15.189:0/55275704 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5909+0+0 (secure 0 0 0) 0x7f03d00889b0 con 0x7f03e00cf900 2024-04-06T13:28:31.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.889+0000 7f03dffff700 1 --2- 172.21.15.189:0/55275704 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f03cc06adb0 0x7f03cc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:31.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.889+0000 7f03dffff700 1 --2- 172.21.15.189:0/55275704 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f03cc06adb0 0x7f03cc06d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f03d400bee0 tx=0x7f03d4013000 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:31.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:31.893+0000 7f03dcff9700 1 -- 172.21.15.189:0/55275704 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f03d00539b0 con 0x7f03e00cf900 2024-04-06T13:28:32.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.087+0000 7f03e6a10700 1 -- 172.21.15.189:0/55275704 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f03e0074990 con 0x7f03e00cf900 2024-04-06T13:28:32.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.091+0000 7f03dcff9700 1 -- 172.21.15.189:0/55275704 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v56) v1 ==== 115+0+0 (secure 0 0 0) 0x7f03d00170b0 con 0x7f03e00cf900 2024-04-06T13:28:32.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.094+0000 7f03e6a10700 1 -- 172.21.15.189:0/55275704 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f03cc06adb0 msgr2=0x7f03cc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:32.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.094+0000 7f03e6a10700 1 --2- 172.21.15.189:0/55275704 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f03cc06adb0 0x7f03cc06d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f03d400bee0 tx=0x7f03d4013000 comp rx=0 tx=0).stop 2024-04-06T13:28:32.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.094+0000 7f03e6a10700 1 -- 172.21.15.189:0/55275704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03e00cf900 msgr2=0x7f03e01f4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:32.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.094+0000 7f03e6a10700 1 --2- 172.21.15.189:0/55275704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03e00cf900 0x7f03e01f4070 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f03d000c4e0 tx=0x7f03d000c7f0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.094+0000 7f03e6a10700 1 -- 172.21.15.189:0/55275704 shutdown_connections 2024-04-06T13:28:32.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.094+0000 7f03e6a10700 1 --2- 172.21.15.189:0/55275704 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f03cc06adb0 0x7f03cc06d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.094+0000 7f03e6a10700 1 --2- 172.21.15.189:0/55275704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03e0152c00 0x7f03e0157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.094+0000 7f03e6a10700 1 --2- 172.21.15.189:0/55275704 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03e00b4d70 0x7f03e00ad3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.094+0000 7f03e6a10700 1 --2- 172.21.15.189:0/55275704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03e00cf900 0x7f03e01f4070 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.094+0000 7f03e6a10700 1 -- 172.21.15.189:0/55275704 >> 172.21.15.189:0/55275704 conn(0x7f03e00d00b0 msgr2=0x7f03e0155fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:32.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.094+0000 7f03e6a10700 1 -- 172.21.15.189:0/55275704 shutdown_connections 2024-04-06T13:28:32.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.094+0000 7f03e6a10700 1 -- 172.21.15.189:0/55275704 wait complete. 2024-04-06T13:28:32.106 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-06T13:28:32.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.230+0000 7fdfaf2e4700 1 Processor -- start 2024-04-06T13:28:32.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.230+0000 7fdfaf2e4700 1 -- start start 2024-04-06T13:28:32.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.231+0000 7fdfaf2e4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdfa806ae40 0x7fdfa80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:32.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.231+0000 7fdfaf2e4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa815ddf0 0x7fdfa815e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:32.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.231+0000 7fdfaf2e4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa80b3990 con 0x7fdfa815d7b0 2024-04-06T13:28:32.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.231+0000 7fdfaf2e4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa80b3060 con 0x7fdfa806ae40 2024-04-06T13:28:32.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.231+0000 7fdfaf2e4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa80b3340 con 0x7fdfa815ddf0 2024-04-06T13:28:32.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.232+0000 7fdfac87f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdfa806ae40 0x7fdfa80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:32.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.232+0000 7fdfad881700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa815ddf0 0x7fdfa815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:32.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.232+0000 7fdfad881700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa815ddf0 0x7fdfa815e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:36116/0 (socket says 172.21.15.189:36116) 2024-04-06T13:28:32.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.232+0000 7fdfad080700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fdfa815d7b0 0x7fdfa806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:53274/0 (socket says 172.21.15.189:53274) 2024-04-06T13:28:32.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.232+0000 7fdfac87f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdfa806ae40 0x7fdfa80b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:41548/0 (socket says 172.21.15.189:41548) 2024-04-06T13:28:32.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.232+0000 7fdfad881700 1 -- 172.21.15.189:0/503867721 learned_addr learned my addr 172.21.15.189:0/503867721 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:32.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.232+0000 7fdf9ffff700 1 -- 172.21.15.189:0/503867721 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094621960 0 0) 0x7fdfa80b3990 con 0x7fdfa815d7b0 2024-04-06T13:28:32.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.232+0000 7fdf9ffff700 1 -- 172.21.15.189:0/503867721 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf940035b0 con 0x7fdfa815d7b0 2024-04-06T13:28:32.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.232+0000 7fdfad881700 1 -- 172.21.15.189:0/503867721 >> v2:172.21.15.192:3300/0 conn(0x7fdfa806ae40 msgr2=0x7fdfa80b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:32.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.232+0000 7fdfad881700 1 --2- 172.21.15.189:0/503867721 >> v2:172.21.15.192:3300/0 conn(0x7fdfa806ae40 0x7fdfa80b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.232+0000 7fdfad881700 1 -- 172.21.15.189:0/503867721 >> v1:172.21.15.192:6789/0 conn(0x7fdfa815d7b0 legacy=0x7fdfa806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:32.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.232+0000 7fdfad881700 1 -- 172.21.15.189:0/503867721 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa80b3620 con 0x7fdfa815ddf0 2024-04-06T13:28:32.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.232+0000 7fdfac87f700 1 --2- 172.21.15.189:0/503867721 >> v2:172.21.15.192:3300/0 conn(0x7fdfa806ae40 0x7fdfa80b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:32.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.233+0000 7fdfad881700 1 --2- 172.21.15.189:0/503867721 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa815ddf0 0x7fdfa815e280 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0009e50 tx=0x7fdfa000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e386f10d525f1047 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:32.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.233+0000 7fdf9ffff700 1 -- 172.21.15.189:0/503867721 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa000e070 con 0x7fdfa815ddf0 2024-04-06T13:28:32.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.233+0000 7fdf9ffff700 1 -- 172.21.15.189:0/503867721 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa000b910 con 0x7fdfa815ddf0 2024-04-06T13:28:32.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.233+0000 7fdf9ffff700 1 -- 172.21.15.189:0/503867721 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa000bc80 con 0x7fdfa815ddf0 2024-04-06T13:28:32.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.233+0000 7fdfaf2e4700 1 -- 172.21.15.189:0/503867721 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa815ddf0 msgr2=0x7fdfa815e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:32.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.233+0000 7fdfaf2e4700 1 --2- 172.21.15.189:0/503867721 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa815ddf0 0x7fdfa815e280 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0009e50 tx=0x7fdfa000b330 comp rx=0 tx=0).stop 2024-04-06T13:28:32.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.233+0000 7fdfaf2e4700 1 -- 172.21.15.189:0/503867721 shutdown_connections 2024-04-06T13:28:32.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.233+0000 7fdfaf2e4700 1 --2- 172.21.15.189:0/503867721 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa815ddf0 0x7fdfa815e280 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.233+0000 7fdfaf2e4700 1 --2- 172.21.15.189:0/503867721 >> v2:172.21.15.192:3300/0 conn(0x7fdfa806ae40 0x7fdfa80b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.233+0000 7fdfaf2e4700 1 -- 172.21.15.189:0/503867721 >> 172.21.15.189:0/503867721 conn(0x7fdfa80d00b0 msgr2=0x7fdfa80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:32.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.233+0000 7fdfaf2e4700 1 -- 172.21.15.189:0/503867721 shutdown_connections 2024-04-06T13:28:32.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.233+0000 7fdfaf2e4700 1 -- 172.21.15.189:0/503867721 wait complete. 2024-04-06T13:28:32.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.235+0000 7fdfaf2e4700 1 Processor -- start 2024-04-06T13:28:32.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.235+0000 7fdfaf2e4700 1 -- start start 2024-04-06T13:28:32.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.235+0000 7fdfaf2e4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa806ae40 0x7fdfa8150b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:32.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.235+0000 7fdfaf2e4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa815d7b0 0x7fdfa80cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:32.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.235+0000 7fdfaf2e4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfa815ddf0 0x7fdfa80ce1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:32.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.235+0000 7fdfaf2e4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa8151090 con 0x7fdfa815d7b0 2024-04-06T13:28:32.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.235+0000 7fdfaf2e4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa80b04c0 con 0x7fdfa815ddf0 2024-04-06T13:28:32.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.235+0000 7fdfaf2e4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa80af8b0 con 0x7fdfa806ae40 2024-04-06T13:28:32.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.235+0000 7fdfac87f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa815d7b0 0x7fdfa80cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:32.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.235+0000 7fdfac87f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa815d7b0 0x7fdfa80cdcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35342/0 (socket says 172.21.15.189:35342) 2024-04-06T13:28:32.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.235+0000 7fdfac87f700 1 -- 172.21.15.189:0/865577965 learned_addr learned my addr 172.21.15.189:0/865577965 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:32.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.236+0000 7fdfad881700 1 --2- 172.21.15.189:0/865577965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfa815ddf0 0x7fdfa80ce1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:32.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.236+0000 7fdfad080700 1 --2- 172.21.15.189:0/865577965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa806ae40 0x7fdfa8150b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:32.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.236+0000 7fdfac87f700 1 -- 172.21.15.189:0/865577965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa806ae40 msgr2=0x7fdfa8150b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:32.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.236+0000 7fdfac87f700 1 --2- 172.21.15.189:0/865577965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa806ae40 0x7fdfa8150b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.236+0000 7fdfac87f700 1 -- 172.21.15.189:0/865577965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfa815ddf0 msgr2=0x7fdfa80ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:32.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.236+0000 7fdfac87f700 1 --2- 172.21.15.189:0/865577965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfa815ddf0 0x7fdfa80ce1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.236+0000 7fdfac87f700 1 -- 172.21.15.189:0/865577965 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa40094a0 con 0x7fdfa815d7b0 2024-04-06T13:28:32.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.236+0000 7fdfad080700 1 --2- 172.21.15.189:0/865577965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa806ae40 0x7fdfa8150b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:28:32.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.236+0000 7fdfad881700 1 --2- 172.21.15.189:0/865577965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfa815ddf0 0x7fdfa80ce1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:28:32.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.236+0000 7fdfac87f700 1 --2- 172.21.15.189:0/865577965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa815d7b0 0x7fdfa80cdcb0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fdfa400ebb0 tx=0x7fdfa400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:32.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.236+0000 7fdf9dffb700 1 -- 172.21.15.189:0/865577965 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa400c910 con 0x7fdfa815d7b0 2024-04-06T13:28:32.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.236+0000 7fdf9dffb700 1 -- 172.21.15.189:0/865577965 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa400ca70 con 0x7fdfa815d7b0 2024-04-06T13:28:32.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.236+0000 7fdf9dffb700 1 -- 172.21.15.189:0/865577965 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa400ce00 con 0x7fdfa815d7b0 2024-04-06T13:28:32.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.236+0000 7fdfaf2e4700 1 -- 172.21.15.189:0/865577965 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa0009b30 con 0x7fdfa815d7b0 2024-04-06T13:28:32.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.236+0000 7fdfaf2e4700 1 -- 172.21.15.189:0/865577965 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa80afe70 con 0x7fdfa815d7b0 2024-04-06T13:28:32.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.237+0000 7fdfaf2e4700 1 -- 172.21.15.189:0/865577965 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa8067bb0 con 0x7fdfa815d7b0 2024-04-06T13:28:32.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.238+0000 7fdf9dffb700 1 -- 172.21.15.189:0/865577965 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fdfa4013470 con 0x7fdfa815d7b0 2024-04-06T13:28:32.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.238+0000 7fdf9dffb700 1 --2- 172.21.15.189:0/865577965 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdf8c06adb0 0x7fdf8c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:32.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.238+0000 7fdf9dffb700 1 -- 172.21.15.189:0/865577965 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5909+0+0 (secure 0 0 0) 0x7fdfa4089780 con 0x7fdfa815d7b0 2024-04-06T13:28:32.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.238+0000 7fdfad080700 1 --2- 172.21.15.189:0/865577965 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdf8c06adb0 0x7fdf8c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:32.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.238+0000 7fdfad080700 1 --2- 172.21.15.189:0/865577965 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdf8c06adb0 0x7fdf8c06d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdf98007fd0 tx=0x7fdf98007c20 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:32.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.242+0000 7fdf9dffb700 1 -- 172.21.15.189:0/865577965 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdfa4054780 con 0x7fdfa815d7b0 2024-04-06T13:28:32.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.438+0000 7fdfaf2e4700 1 -- 172.21.15.189:0/865577965 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fdfa80b0630 con 0x7fdfa815d7b0 2024-04-06T13:28:32.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.441+0000 7fdf9dffb700 1 -- 172.21.15.189:0/865577965 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v57) v1 ==== 120+0+0 (secure 0 0 0) 0x7fdfa4057da0 con 0x7fdfa815d7b0 2024-04-06T13:28:32.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.444+0000 7fdfaf2e4700 1 -- 172.21.15.189:0/865577965 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdf8c06adb0 msgr2=0x7fdf8c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:32.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.444+0000 7fdfaf2e4700 1 --2- 172.21.15.189:0/865577965 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdf8c06adb0 0x7fdf8c06d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdf98007fd0 tx=0x7fdf98007c20 comp rx=0 tx=0).stop 2024-04-06T13:28:32.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.444+0000 7fdfaf2e4700 1 -- 172.21.15.189:0/865577965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa815d7b0 msgr2=0x7fdfa80cdcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:32.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.444+0000 7fdfaf2e4700 1 --2- 172.21.15.189:0/865577965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa815d7b0 0x7fdfa80cdcb0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fdfa400ebb0 tx=0x7fdfa400ef70 comp rx=0 tx=0).stop 2024-04-06T13:28:32.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.444+0000 7fdfaf2e4700 1 -- 172.21.15.189:0/865577965 shutdown_connections 2024-04-06T13:28:32.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.444+0000 7fdfaf2e4700 1 --2- 172.21.15.189:0/865577965 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdf8c06adb0 0x7fdf8c06d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.444+0000 7fdfaf2e4700 1 --2- 172.21.15.189:0/865577965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfa815ddf0 0x7fdfa80ce1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.444+0000 7fdfaf2e4700 1 --2- 172.21.15.189:0/865577965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa806ae40 0x7fdfa8150b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.444+0000 7fdfaf2e4700 1 --2- 172.21.15.189:0/865577965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa815d7b0 0x7fdfa80cdcb0 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.444+0000 7fdfaf2e4700 1 -- 172.21.15.189:0/865577965 >> 172.21.15.189:0/865577965 conn(0x7fdfa80d00b0 msgr2=0x7fdfa80ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:32.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.444+0000 7fdfaf2e4700 1 -- 172.21.15.189:0/865577965 shutdown_connections 2024-04-06T13:28:32.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.444+0000 7fdfaf2e4700 1 -- 172.21.15.189:0/865577965 wait complete. 2024-04-06T13:28:32.457 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2024-04-06T13:28:32.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.579+0000 7f05bedf6700 1 Processor -- start 2024-04-06T13:28:32.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.579+0000 7f05bedf6700 1 -- start start 2024-04-06T13:28:32.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05bedf6700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f05b815d7b0 0x7f05b80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:32.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05bedf6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05b815e500 0x7f05b806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05bedf6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b80b3990 con 0x7f05b806a730 2024-04-06T13:28:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05bedf6700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b806b0f0 con 0x7f05b815d7b0 2024-04-06T13:28:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05bedf6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b80b3060 con 0x7f05b815e500 2024-04-06T13:28:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05b7fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f05b806a730 0x7f05b815ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40926/0 (socket says 172.21.15.189:40926) 2024-04-06T13:28:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05b7fff700 1 -- 172.21.15.189:0/242575588 learned_addr learned my addr 172.21.15.189:0/242575588 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05b77fe700 1 -- 172.21.15.189:0/242575588 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705876893 0 0) 0x7f05b80b3990 con 0x7f05b806a730 2024-04-06T13:28:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05b77fe700 1 -- 172.21.15.189:0/242575588 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05a80035b0 con 0x7f05b806a730 2024-04-06T13:28:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05bd393700 1 --2- 172.21.15.189:0/242575588 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05b815e500 0x7f05b806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05bcb92700 1 --2- 172.21.15.189:0/242575588 >> v2:172.21.15.192:3300/0 conn(0x7f05b815d7b0 0x7f05b80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05b77fe700 1 -- 172.21.15.189:0/242575588 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 309086496 0 0) 0x7f05a80035b0 con 0x7f05b806a730 2024-04-06T13:28:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05b77fe700 1 -- 172.21.15.189:0/242575588 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05b815e500 msgr2=0x7f05b806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05b77fe700 1 --2- 172.21.15.189:0/242575588 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05b815e500 0x7f05b806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05b77fe700 1 -- 172.21.15.189:0/242575588 >> v2:172.21.15.192:3300/0 conn(0x7f05b815d7b0 msgr2=0x7f05b80b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:32.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05b77fe700 1 --2- 172.21.15.189:0/242575588 >> v2:172.21.15.192:3300/0 conn(0x7f05b815d7b0 0x7f05b80b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05b77fe700 1 -- 172.21.15.189:0/242575588 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b80b3340 con 0x7f05b806a730 2024-04-06T13:28:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05b77fe700 1 -- 172.21.15.189:0/242575588 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f05a4001670 con 0x7f05b806a730 2024-04-06T13:28:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05bcb92700 1 --2- 172.21.15.189:0/242575588 >> v2:172.21.15.192:3300/0 conn(0x7f05b815d7b0 0x7f05b80b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:28:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05b77fe700 1 -- 172.21.15.189:0/242575588 >> v1:172.21.15.189:6789/0 conn(0x7f05b806a730 legacy=0x7f05b815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.580+0000 7f05b77fe700 1 --2- 172.21.15.189:0/242575588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05a8003a00 0x7f05a8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.581+0000 7f05b77fe700 1 --2- 172.21.15.189:0/242575588 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05a8004850 0x7f05a8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.581+0000 7f05b77fe700 1 --2- 172.21.15.189:0/242575588 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05b815e500 0x7f05a8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.581+0000 7f05b77fe700 1 -- 172.21.15.189:0/242575588 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b806b0f0 con 0x7f05a8003a00 2024-04-06T13:28:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.581+0000 7f05b77fe700 1 -- 172.21.15.189:0/242575588 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b80b3060 con 0x7f05a8004850 2024-04-06T13:28:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.581+0000 7f05b77fe700 1 -- 172.21.15.189:0/242575588 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a800bac0 con 0x7f05b815e500 2024-04-06T13:28:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.581+0000 7f05bcb92700 1 --2- 172.21.15.189:0/242575588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05a8003a00 0x7f05a8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.581+0000 7f05bd393700 1 --2- 172.21.15.189:0/242575588 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05b815e500 0x7f05a8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.581+0000 7f05b7fff700 1 --2- 172.21.15.189:0/242575588 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05a8004850 0x7f05a8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.581+0000 7f05bcb92700 1 -- 172.21.15.189:0/242575588 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05b815e500 msgr2=0x7f05a8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:32.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.581+0000 7f05bcb92700 1 --2- 172.21.15.189:0/242575588 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05b815e500 0x7f05a8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.581+0000 7f05bcb92700 1 -- 172.21.15.189:0/242575588 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05a8004850 msgr2=0x7f05a8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.581+0000 7f05bcb92700 1 --2- 172.21.15.189:0/242575588 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05a8004850 0x7f05a8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.581+0000 7f05bcb92700 1 -- 172.21.15.189:0/242575588 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a80035b0 con 0x7f05a8003a00 2024-04-06T13:28:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.581+0000 7f05b7fff700 1 --2- 172.21.15.189:0/242575588 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05a8004850 0x7f05a8004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:28:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.581+0000 7f05bcb92700 1 --2- 172.21.15.189:0/242575588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05a8003a00 0x7f05a8003e20 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f05ac009920 tx=0x7f05a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=391769d77bf2757 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.581+0000 7f05b77fe700 1 -- 172.21.15.189:0/242575588 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ac00c070 con 0x7f05a8003a00 2024-04-06T13:28:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.581+0000 7f05b77fe700 1 -- 172.21.15.189:0/242575588 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ac013610 con 0x7f05a8003a00 2024-04-06T13:28:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.581+0000 7f05b77fe700 1 -- 172.21.15.189:0/242575588 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ac013980 con 0x7f05a8003a00 2024-04-06T13:28:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.582+0000 7f05bedf6700 1 -- 172.21.15.189:0/242575588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05a8003a00 msgr2=0x7f05a8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.582+0000 7f05bedf6700 1 --2- 172.21.15.189:0/242575588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05a8003a00 0x7f05a8003e20 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f05ac009920 tx=0x7f05a8005880 comp rx=0 tx=0).stop 2024-04-06T13:28:32.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.582+0000 7f05bd393700 1 -- 172.21.15.189:0/242575588 reap_dead start 2024-04-06T13:28:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.582+0000 7f05bedf6700 1 -- 172.21.15.189:0/242575588 shutdown_connections 2024-04-06T13:28:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.582+0000 7f05bedf6700 1 -- 172.21.15.189:0/242575588 >> 172.21.15.189:0/242575588 conn(0x7f05b80d00b0 msgr2=0x7f05b80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.582+0000 7f05bedf6700 1 -- 172.21.15.189:0/242575588 shutdown_connections 2024-04-06T13:28:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.582+0000 7f05bedf6700 1 -- 172.21.15.189:0/242575588 wait complete. 2024-04-06T13:28:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.583+0000 7f05bedf6700 1 Processor -- start 2024-04-06T13:28:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.583+0000 7f05bedf6700 1 -- start start 2024-04-06T13:28:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.584+0000 7f05bedf6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05a8003a00 0x7f05b814aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.584+0000 7f05bedf6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05b80b0670 0x7f05b80aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.584+0000 7f05bedf6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b80af0c0 0x7f05b815c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.584+0000 7f05bedf6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b80b0250 con 0x7f05b80af0c0 2024-04-06T13:28:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.584+0000 7f05bedf6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b814af70 con 0x7f05a8003a00 2024-04-06T13:28:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.584+0000 7f05bedf6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b80af5d0 con 0x7f05b80b0670 2024-04-06T13:28:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.584+0000 7f05bd393700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b80af0c0 0x7f05b815c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.584+0000 7f05bd393700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b80af0c0 0x7f05b815c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35354/0 (socket says 172.21.15.189:35354) 2024-04-06T13:28:32.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.584+0000 7f05bd393700 1 -- 172.21.15.189:0/3900214699 learned_addr learned my addr 172.21.15.189:0/3900214699 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.584+0000 7f05b7fff700 1 --2- 172.21.15.189:0/3900214699 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05b80b0670 0x7f05b80aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.584+0000 7f05bcb92700 1 --2- 172.21.15.189:0/3900214699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05a8003a00 0x7f05b814aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.584+0000 7f05bd393700 1 -- 172.21.15.189:0/3900214699 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05b80b0670 msgr2=0x7f05b80aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.584+0000 7f05bd393700 1 --2- 172.21.15.189:0/3900214699 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05b80b0670 0x7f05b80aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.584+0000 7f05bd393700 1 -- 172.21.15.189:0/3900214699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05a8003a00 msgr2=0x7f05b814aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.584+0000 7f05bd393700 1 --2- 172.21.15.189:0/3900214699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05a8003a00 0x7f05b814aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.584+0000 7f05bd393700 1 -- 172.21.15.189:0/3900214699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b80af8b0 con 0x7f05b80af0c0 2024-04-06T13:28:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.584+0000 7f05bd393700 1 --2- 172.21.15.189:0/3900214699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b80af0c0 0x7f05b815c210 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f05b806c7f0 tx=0x7f05b00114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.585+0000 7f05b57fa700 1 -- 172.21.15.189:0/3900214699 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b000e070 con 0x7f05b80af0c0 2024-04-06T13:28:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.585+0000 7f05b57fa700 1 -- 172.21.15.189:0/3900214699 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b0003cf0 con 0x7f05b80af0c0 2024-04-06T13:28:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.585+0000 7f05bedf6700 1 -- 172.21.15.189:0/3900214699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b80afb90 con 0x7f05b80af0c0 2024-04-06T13:28:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.585+0000 7f05b57fa700 1 -- 172.21.15.189:0/3900214699 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b0003ea0 con 0x7f05b80af0c0 2024-04-06T13:28:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.585+0000 7f05bedf6700 1 -- 172.21.15.189:0/3900214699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b8154290 con 0x7f05b80af0c0 2024-04-06T13:28:32.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.586+0000 7f05bedf6700 1 -- 172.21.15.189:0/3900214699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05b8154450 con 0x7f05b80af0c0 2024-04-06T13:28:32.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.586+0000 7f05b57fa700 1 -- 172.21.15.189:0/3900214699 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f05b0011960 con 0x7f05b80af0c0 2024-04-06T13:28:32.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.586+0000 7f05b57fa700 1 --2- 172.21.15.189:0/3900214699 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f059c06adb0 0x7f059c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:32.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.586+0000 7f05b57fa700 1 -- 172.21.15.189:0/3900214699 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5909+0+0 (secure 0 0 0) 0x7f05b0089a80 con 0x7f05b80af0c0 2024-04-06T13:28:32.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.587+0000 7f05bcb92700 1 --2- 172.21.15.189:0/3900214699 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f059c06adb0 0x7f059c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:32.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.587+0000 7f05bcb92700 1 --2- 172.21.15.189:0/3900214699 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f059c06adb0 0x7f059c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f05ac000c00 tx=0x7f05ac01f040 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:32.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.591+0000 7f05b57fa700 1 -- 172.21.15.189:0/3900214699 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f05b00548e0 con 0x7f05b80af0c0 2024-04-06T13:28:32.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:32.874+0000 7f05bedf6700 1 -- 172.21.15.189:0/3900214699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f05b8067bb0 con 0x7f05b80af0c0 2024-04-06T13:28:33.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.379+0000 7f05b57fa700 1 -- 172.21.15.189:0/3900214699 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v2486) v1 ==== 178+0+0 (secure 0 0 0) 0x7f05b0057f00 con 0x7f05b80af0c0 2024-04-06T13:28:33.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.381+0000 7f05bedf6700 1 -- 172.21.15.189:0/3900214699 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f059c06adb0 msgr2=0x7f059c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:33.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.381+0000 7f05bedf6700 1 --2- 172.21.15.189:0/3900214699 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f059c06adb0 0x7f059c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f05ac000c00 tx=0x7f05ac01f040 comp rx=0 tx=0).stop 2024-04-06T13:28:33.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.381+0000 7f05bedf6700 1 -- 172.21.15.189:0/3900214699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b80af0c0 msgr2=0x7f05b815c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:33.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.381+0000 7f05bedf6700 1 --2- 172.21.15.189:0/3900214699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b80af0c0 0x7f05b815c210 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f05b806c7f0 tx=0x7f05b00114b0 comp rx=0 tx=0).stop 2024-04-06T13:28:33.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.381+0000 7f05bedf6700 1 -- 172.21.15.189:0/3900214699 shutdown_connections 2024-04-06T13:28:33.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.381+0000 7f05bedf6700 1 --2- 172.21.15.189:0/3900214699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b80af0c0 0x7f05b815c210 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:33.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.381+0000 7f05bedf6700 1 --2- 172.21.15.189:0/3900214699 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f059c06adb0 0x7f059c06d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:33.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.381+0000 7f05bedf6700 1 --2- 172.21.15.189:0/3900214699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05a8003a00 0x7f05b814aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:33.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.381+0000 7f05bedf6700 1 --2- 172.21.15.189:0/3900214699 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05b80b0670 0x7f05b80aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:33.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.381+0000 7f05bedf6700 1 -- 172.21.15.189:0/3900214699 >> 172.21.15.189:0/3900214699 conn(0x7f05b80d00b0 msgr2=0x7f05b806f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:33.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.381+0000 7f05bedf6700 1 -- 172.21.15.189:0/3900214699 shutdown_connections 2024-04-06T13:28:33.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.381+0000 7f05bedf6700 1 -- 172.21.15.189:0/3900214699 wait complete. 2024-04-06T13:28:33.394 INFO:tasks.cephfs_test_runner:Test that number of snaps retained are as per user spec. ... ok 2024-04-06T13:28:33.395 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2024-04-06T13:28:33.395 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-06T13:28:33.395 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2024-04-06T13:28:33.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.519+0000 7f0c89a08700 1 Processor -- start 2024-04-06T13:28:33.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.520+0000 7f0c89a08700 1 -- start start 2024-04-06T13:28:33.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.520+0000 7f0c89a08700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c840cea80 0x7f0c840cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:33.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.520+0000 7f0c89a08700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c8415a830 con 0x7f0c840aec70 2024-04-06T13:28:33.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.520+0000 7f0c89a08700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c840cc050 con 0x7f0c840cdae0 2024-04-06T13:28:33.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.520+0000 7f0c89a08700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c840cd920 con 0x7f0c840cea80 2024-04-06T13:28:33.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.520+0000 7f0c82ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0c840aec70 0x7f0c840ce370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40932/0 (socket says 172.21.15.189:40932) 2024-04-06T13:28:33.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.520+0000 7f0c82ffd700 1 -- 172.21.15.189:0/3976366555 learned_addr learned my addr 172.21.15.189:0/3976366555 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:33.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c827fc700 1 --2- 172.21.15.189:0/3976366555 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c840cea80 0x7f0c840cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:33.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c81ffb700 1 -- 172.21.15.189:0/3976366555 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099736344 0 0) 0x7f0c8415a830 con 0x7f0c840aec70 2024-04-06T13:28:33.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c81ffb700 1 -- 172.21.15.189:0/3976366555 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c6c0035b0 con 0x7f0c840aec70 2024-04-06T13:28:33.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c81ffb700 1 -- 172.21.15.189:0/3976366555 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176802101 0 0) 0x7f0c840cc050 con 0x7f0c840cdae0 2024-04-06T13:28:33.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c81ffb700 1 -- 172.21.15.189:0/3976366555 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c8415a830 con 0x7f0c840cdae0 2024-04-06T13:28:33.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c81ffb700 1 -- 172.21.15.189:0/3976366555 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3169887795 0 0) 0x7f0c6c0035b0 con 0x7f0c840aec70 2024-04-06T13:28:33.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c81ffb700 1 -- 172.21.15.189:0/3976366555 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c840cea80 msgr2=0x7f0c840cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:33.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c81ffb700 1 --2- 172.21.15.189:0/3976366555 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c840cea80 0x7f0c840cd3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:33.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c81ffb700 1 -- 172.21.15.189:0/3976366555 >> v1:172.21.15.192:6789/0 conn(0x7f0c840cdae0 legacy=0x7f0c8406ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:33.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c81ffb700 1 -- 172.21.15.189:0/3976366555 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c840666f0 con 0x7f0c840aec70 2024-04-06T13:28:33.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c81ffb700 1 -- 172.21.15.189:0/3976366555 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f0c78001670 con 0x7f0c840aec70 2024-04-06T13:28:33.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c81ffb700 1 -- 172.21.15.189:0/3976366555 >> v1:172.21.15.189:6789/0 conn(0x7f0c840aec70 legacy=0x7f0c840ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:33.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c81ffb700 1 --2- 172.21.15.189:0/3976366555 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c840cea80 0x7f0c6c003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:33.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c81ffb700 1 --2- 172.21.15.189:0/3976366555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c6c004120 0x7f0c6c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:33.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c81ffb700 1 --2- 172.21.15.189:0/3976366555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c6c004d90 0x7f0c6c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:33.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c81ffb700 1 -- 172.21.15.189:0/3976366555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c840cd920 con 0x7f0c6c004120 2024-04-06T13:28:33.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c81ffb700 1 -- 172.21.15.189:0/3976366555 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c00bba0 con 0x7f0c6c004d90 2024-04-06T13:28:33.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c81ffb700 1 -- 172.21.15.189:0/3976366555 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c00bd30 con 0x7f0c840cea80 2024-04-06T13:28:33.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c827fc700 1 --2- 172.21.15.189:0/3976366555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c6c004120 0x7f0c6c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:33.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c82ffd700 1 --2- 172.21.15.189:0/3976366555 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c840cea80 0x7f0c6c003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:33.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c837fe700 1 --2- 172.21.15.189:0/3976366555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c6c004d90 0x7f0c6c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:33.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c827fc700 1 -- 172.21.15.189:0/3976366555 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c840cea80 msgr2=0x7f0c6c003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:33.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.521+0000 7f0c827fc700 1 --2- 172.21.15.189:0/3976366555 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c840cea80 0x7f0c6c003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:33.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.522+0000 7f0c827fc700 1 -- 172.21.15.189:0/3976366555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c6c004d90 msgr2=0x7f0c6c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:33.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.522+0000 7f0c827fc700 1 --2- 172.21.15.189:0/3976366555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c6c004d90 0x7f0c6c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:33.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.522+0000 7f0c827fc700 1 -- 172.21.15.189:0/3976366555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c0035b0 con 0x7f0c6c004120 2024-04-06T13:28:33.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.522+0000 7f0c827fc700 1 --2- 172.21.15.189:0/3976366555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c6c004120 0x7f0c6c004850 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f0c74009b60 tx=0x7f0c6c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a55f4d9892442e67 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:33.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.522+0000 7f0c81ffb700 1 -- 172.21.15.189:0/3976366555 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7400e6c0 con 0x7f0c6c004120 2024-04-06T13:28:33.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.522+0000 7f0c81ffb700 1 -- 172.21.15.189:0/3976366555 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c7400e820 con 0x7f0c6c004120 2024-04-06T13:28:33.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.522+0000 7f0c81ffb700 1 -- 172.21.15.189:0/3976366555 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7400eb90 con 0x7f0c6c004120 2024-04-06T13:28:33.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.522+0000 7f0c89a08700 1 -- 172.21.15.189:0/3976366555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c6c004120 msgr2=0x7f0c6c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:33.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.522+0000 7f0c89a08700 1 --2- 172.21.15.189:0/3976366555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c6c004120 0x7f0c6c004850 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f0c74009b60 tx=0x7f0c6c005880 comp rx=0 tx=0).stop 2024-04-06T13:28:33.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.522+0000 7f0c837fe700 1 -- 172.21.15.189:0/3976366555 reap_dead start 2024-04-06T13:28:33.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.522+0000 7f0c89a08700 1 -- 172.21.15.189:0/3976366555 shutdown_connections 2024-04-06T13:28:33.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.522+0000 7f0c89a08700 1 -- 172.21.15.189:0/3976366555 >> 172.21.15.189:0/3976366555 conn(0x7f0c840cc290 msgr2=0x7f0c840cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:33.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.522+0000 7f0c89a08700 1 -- 172.21.15.189:0/3976366555 shutdown_connections 2024-04-06T13:28:33.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.522+0000 7f0c89a08700 1 -- 172.21.15.189:0/3976366555 wait complete. 2024-04-06T13:28:33.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.524+0000 7f0c89a08700 1 Processor -- start 2024-04-06T13:28:33.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.524+0000 7f0c89a08700 1 -- start start 2024-04-06T13:28:33.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.524+0000 7f0c89a08700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c8414fd20 0x7f0c840b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:33.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.524+0000 7f0c89a08700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c840adb30 0x7f0c841586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:33.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.524+0000 7f0c89a08700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c84158c20 0x7f0c8406a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.524+0000 7f0c89a08700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c841590a0 con 0x7f0c84158c20 2024-04-06T13:28:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.525+0000 7f0c89a08700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c840b38f0 con 0x7f0c8414fd20 2024-04-06T13:28:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.525+0000 7f0c89a08700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c840b2ce0 con 0x7f0c840adb30 2024-04-06T13:28:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.525+0000 7f0c837fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c84158c20 0x7f0c8406a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.525+0000 7f0c837fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c84158c20 0x7f0c8406a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35378/0 (socket says 172.21.15.189:35378) 2024-04-06T13:28:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.525+0000 7f0c837fe700 1 -- 172.21.15.189:0/112286758 learned_addr learned my addr 172.21.15.189:0/112286758 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.525+0000 7f0c827fc700 1 --2- 172.21.15.189:0/112286758 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c840adb30 0x7f0c841586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.525+0000 7f0c82ffd700 1 --2- 172.21.15.189:0/112286758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c8414fd20 0x7f0c840b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.525+0000 7f0c837fe700 1 -- 172.21.15.189:0/112286758 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c840adb30 msgr2=0x7f0c841586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.525+0000 7f0c837fe700 1 --2- 172.21.15.189:0/112286758 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c840adb30 0x7f0c841586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.525+0000 7f0c837fe700 1 -- 172.21.15.189:0/112286758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c8414fd20 msgr2=0x7f0c840b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.525+0000 7f0c837fe700 1 --2- 172.21.15.189:0/112286758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c8414fd20 0x7f0c840b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.525+0000 7f0c837fe700 1 -- 172.21.15.189:0/112286758 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c840b2fc0 con 0x7f0c84158c20 2024-04-06T13:28:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.525+0000 7f0c837fe700 1 --2- 172.21.15.189:0/112286758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c84158c20 0x7f0c8406a730 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f0c8406cc10 tx=0x7f0c7c00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.525+0000 7f0c6bfff700 1 -- 172.21.15.189:0/112286758 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c003c10 con 0x7f0c84158c20 2024-04-06T13:28:33.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.525+0000 7f0c6bfff700 1 -- 172.21.15.189:0/112286758 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c7c003d70 con 0x7f0c84158c20 2024-04-06T13:28:33.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.525+0000 7f0c89a08700 1 -- 172.21.15.189:0/112286758 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c840b32a0 con 0x7f0c84158c20 2024-04-06T13:28:33.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.525+0000 7f0c6bfff700 1 -- 172.21.15.189:0/112286758 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c00db20 con 0x7f0c84158c20 2024-04-06T13:28:33.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.525+0000 7f0c89a08700 1 -- 172.21.15.189:0/112286758 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c84150100 con 0x7f0c84158c20 2024-04-06T13:28:33.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.526+0000 7f0c89a08700 1 -- 172.21.15.189:0/112286758 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c84073740 con 0x7f0c84158c20 2024-04-06T13:28:33.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.527+0000 7f0c6bfff700 1 -- 172.21.15.189:0/112286758 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0c7c00dc80 con 0x7f0c84158c20 2024-04-06T13:28:33.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.527+0000 7f0c6bfff700 1 --2- 172.21.15.189:0/112286758 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c7006adb0 0x7f0c7006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:33.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.527+0000 7f0c6bfff700 1 -- 172.21.15.189:0/112286758 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5909+0+0 (secure 0 0 0) 0x7f0c7c089800 con 0x7f0c84158c20 2024-04-06T13:28:33.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.527+0000 7f0c82ffd700 1 --2- 172.21.15.189:0/112286758 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c7006adb0 0x7f0c7006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:33.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.527+0000 7f0c82ffd700 1 --2- 172.21.15.189:0/112286758 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c7006adb0 0x7f0c7006d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0c780048d0 tx=0x7f0c78001e10 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:33.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.531+0000 7f0c6bfff700 1 -- 172.21.15.189:0/112286758 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0c7c0548b0 con 0x7f0c84158c20 2024-04-06T13:28:33.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:33.807+0000 7f0c89a08700 1 -- 172.21.15.189:0/112286758 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f0c84074990 con 0x7f0c84158c20 2024-04-06T13:28:34.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.380+0000 7f0c6bfff700 1 -- 172.21.15.189:0/112286758 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2487) v1 ==== 177+0+0 (secure 0 0 0) 0x7f0c7c057ed0 con 0x7f0c84158c20 2024-04-06T13:28:34.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.383+0000 7f0c89a08700 1 -- 172.21.15.189:0/112286758 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c7006adb0 msgr2=0x7f0c7006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:34.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.383+0000 7f0c89a08700 1 --2- 172.21.15.189:0/112286758 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c7006adb0 0x7f0c7006d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0c780048d0 tx=0x7f0c78001e10 comp rx=0 tx=0).stop 2024-04-06T13:28:34.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.383+0000 7f0c89a08700 1 -- 172.21.15.189:0/112286758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c84158c20 msgr2=0x7f0c8406a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:34.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.383+0000 7f0c89a08700 1 --2- 172.21.15.189:0/112286758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c84158c20 0x7f0c8406a730 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f0c8406cc10 tx=0x7f0c7c00d430 comp rx=0 tx=0).stop 2024-04-06T13:28:34.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.383+0000 7f0c89a08700 1 -- 172.21.15.189:0/112286758 shutdown_connections 2024-04-06T13:28:34.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.383+0000 7f0c89a08700 1 --2- 172.21.15.189:0/112286758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c84158c20 0x7f0c8406a730 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:34.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.383+0000 7f0c89a08700 1 --2- 172.21.15.189:0/112286758 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0c7006adb0 0x7f0c7006d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:34.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.383+0000 7f0c89a08700 1 --2- 172.21.15.189:0/112286758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c8414fd20 0x7f0c840b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:34.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.383+0000 7f0c89a08700 1 --2- 172.21.15.189:0/112286758 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c840adb30 0x7f0c841586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:34.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.383+0000 7f0c89a08700 1 -- 172.21.15.189:0/112286758 >> 172.21.15.189:0/112286758 conn(0x7f0c840cc290 msgr2=0x7f0c840af1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:34.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.383+0000 7f0c89a08700 1 -- 172.21.15.189:0/112286758 shutdown_connections 2024-04-06T13:28:34.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.383+0000 7f0c89a08700 1 -- 172.21.15.189:0/112286758 wait complete. 2024-04-06T13:28:34.397 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-06T13:28:34.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.525+0000 7f5c73df8700 1 Processor -- start 2024-04-06T13:28:34.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.526+0000 7f5c73df8700 1 -- start start 2024-04-06T13:28:34.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c73df8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c6c158fe0 0x7f5c6c06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:34.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c73df8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5c6c06ac70 0x7f5c6c15bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:34.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c73df8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c6c15f9e0 con 0x7f5c6c15c100 2024-04-06T13:28:34.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c73df8700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c6c06b0f0 con 0x7f5c6c06ac70 2024-04-06T13:28:34.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c73df8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c15f0d0 con 0x7f5c6c158fe0 2024-04-06T13:28:34.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c72395700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5c6c15c100 0x7f5c6c0717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40934/0 (socket says 172.21.15.189:40934) 2024-04-06T13:28:34.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c72395700 1 -- 172.21.15.189:0/2060089914 learned_addr learned my addr 172.21.15.189:0/2060089914 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:34.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c71393700 1 --2- 172.21.15.189:0/2060089914 >> v2:172.21.15.192:3300/0 conn(0x7f5c6c06ac70 0x7f5c6c15bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:34.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c71b94700 1 --2- 172.21.15.189:0/2060089914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c6c158fe0 0x7f5c6c06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:34.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c70b92700 1 -- 172.21.15.189:0/2060089914 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2705744031 0 0) 0x7f5c6c15f9e0 con 0x7f5c6c15c100 2024-04-06T13:28:34.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c70b92700 1 -- 172.21.15.189:0/2060089914 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c580035b0 con 0x7f5c6c15c100 2024-04-06T13:28:34.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c70b92700 1 -- 172.21.15.189:0/2060089914 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2390893141 0 0) 0x7f5c580035b0 con 0x7f5c6c15c100 2024-04-06T13:28:34.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c70b92700 1 -- 172.21.15.189:0/2060089914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c6c158fe0 msgr2=0x7f5c6c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:34.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c70b92700 1 --2- 172.21.15.189:0/2060089914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c6c158fe0 0x7f5c6c06a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:34.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c70b92700 1 -- 172.21.15.189:0/2060089914 >> v2:172.21.15.192:3300/0 conn(0x7f5c6c06ac70 msgr2=0x7f5c6c15bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:34.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c70b92700 1 --2- 172.21.15.189:0/2060089914 >> v2:172.21.15.192:3300/0 conn(0x7f5c6c06ac70 0x7f5c6c15bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:34.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c70b92700 1 -- 172.21.15.189:0/2060089914 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6c15c8f0 con 0x7f5c6c15c100 2024-04-06T13:28:34.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c70b92700 1 -- 172.21.15.189:0/2060089914 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f5c640019f0 con 0x7f5c6c15c100 2024-04-06T13:28:34.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c70b92700 1 -- 172.21.15.189:0/2060089914 >> v1:172.21.15.189:6789/0 conn(0x7f5c6c15c100 legacy=0x7f5c6c0717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:34.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.527+0000 7f5c70b92700 1 --2- 172.21.15.189:0/2060089914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c6c158fe0 0x7f5c58003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:34.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.528+0000 7f5c70b92700 1 --2- 172.21.15.189:0/2060089914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c58003f80 0x7f5c58004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:34.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.528+0000 7f5c70b92700 1 --2- 172.21.15.189:0/2060089914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c58004d90 0x7f5c58005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:34.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.528+0000 7f5c70b92700 1 -- 172.21.15.189:0/2060089914 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c06b0f0 con 0x7f5c58004d90 2024-04-06T13:28:34.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.528+0000 7f5c70b92700 1 -- 172.21.15.189:0/2060089914 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c15f0d0 con 0x7f5c58003f80 2024-04-06T13:28:34.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.528+0000 7f5c70b92700 1 -- 172.21.15.189:0/2060089914 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5800ba60 con 0x7f5c6c158fe0 2024-04-06T13:28:34.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.528+0000 7f5c72395700 1 --2- 172.21.15.189:0/2060089914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c58004d90 0x7f5c58005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:34.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.528+0000 7f5c71b94700 1 --2- 172.21.15.189:0/2060089914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c6c158fe0 0x7f5c58003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:34.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.528+0000 7f5c71393700 1 --2- 172.21.15.189:0/2060089914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c58003f80 0x7f5c58004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:34.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.528+0000 7f5c72395700 1 -- 172.21.15.189:0/2060089914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c6c158fe0 msgr2=0x7f5c58003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:34.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.528+0000 7f5c72395700 1 --2- 172.21.15.189:0/2060089914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c6c158fe0 0x7f5c58003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:34.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.528+0000 7f5c72395700 1 -- 172.21.15.189:0/2060089914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c58003f80 msgr2=0x7f5c58004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:34.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.528+0000 7f5c72395700 1 --2- 172.21.15.189:0/2060089914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c58003f80 0x7f5c58004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:34.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.528+0000 7f5c72395700 1 -- 172.21.15.189:0/2060089914 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c580035b0 con 0x7f5c58004d90 2024-04-06T13:28:34.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.528+0000 7f5c71393700 1 --2- 172.21.15.189:0/2060089914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c58003f80 0x7f5c58004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:28:34.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.528+0000 7f5c71b94700 1 --2- 172.21.15.189:0/2060089914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c6c158fe0 0x7f5c58003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:28:34.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.528+0000 7f5c72395700 1 --2- 172.21.15.189:0/2060089914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c58004d90 0x7f5c58005210 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400a860 tx=0x7f5c58005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6057a4a4b297701e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:34.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.528+0000 7f5c70b92700 1 -- 172.21.15.189:0/2060089914 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6400a320 con 0x7f5c58004d90 2024-04-06T13:28:34.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.528+0000 7f5c70b92700 1 -- 172.21.15.189:0/2060089914 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c64012600 con 0x7f5c58004d90 2024-04-06T13:28:34.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.528+0000 7f5c70b92700 1 -- 172.21.15.189:0/2060089914 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c640129b0 con 0x7f5c58004d90 2024-04-06T13:28:34.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.529+0000 7f5c73df8700 1 -- 172.21.15.189:0/2060089914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c58004d90 msgr2=0x7f5c58005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:34.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.529+0000 7f5c73df8700 1 --2- 172.21.15.189:0/2060089914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c58004d90 0x7f5c58005210 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400a860 tx=0x7f5c58005880 comp rx=0 tx=0).stop 2024-04-06T13:28:34.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.529+0000 7f5c72395700 1 -- 172.21.15.189:0/2060089914 reap_dead start 2024-04-06T13:28:34.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.529+0000 7f5c73df8700 1 -- 172.21.15.189:0/2060089914 shutdown_connections 2024-04-06T13:28:34.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.529+0000 7f5c73df8700 1 -- 172.21.15.189:0/2060089914 >> 172.21.15.189:0/2060089914 conn(0x7f5c6c0d00b0 msgr2=0x7f5c6c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:34.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.529+0000 7f5c73df8700 1 -- 172.21.15.189:0/2060089914 shutdown_connections 2024-04-06T13:28:34.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.529+0000 7f5c73df8700 1 -- 172.21.15.189:0/2060089914 wait complete. 2024-04-06T13:28:34.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.530+0000 7f5c73df8700 1 Processor -- start 2024-04-06T13:28:34.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.530+0000 7f5c73df8700 1 -- start start 2024-04-06T13:28:34.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.531+0000 7f5c73df8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c6c139c90 0x7f5c6c140db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:34.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.531+0000 7f5c73df8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c6c1412f0 0x7f5c6c13a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:34.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.531+0000 7f5c73df8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c6c13a7f0 0x7f5c6c13c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:34.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.531+0000 7f5c73df8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c13ac70 con 0x7f5c6c1412f0 2024-04-06T13:28:34.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.531+0000 7f5c73df8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c13a070 con 0x7f5c6c139c90 2024-04-06T13:28:34.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.531+0000 7f5c73df8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c141850 con 0x7f5c6c13a7f0 2024-04-06T13:28:34.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.531+0000 7f5c71393700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c6c1412f0 0x7f5c6c13a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:34.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.531+0000 7f5c71393700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c6c1412f0 0x7f5c6c13a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35398/0 (socket says 172.21.15.189:35398) 2024-04-06T13:28:34.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.531+0000 7f5c71393700 1 -- 172.21.15.189:0/4095358159 learned_addr learned my addr 172.21.15.189:0/4095358159 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:34.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.531+0000 7f5c72395700 1 --2- 172.21.15.189:0/4095358159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c6c13a7f0 0x7f5c6c13c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:34.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.531+0000 7f5c71b94700 1 --2- 172.21.15.189:0/4095358159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c6c139c90 0x7f5c6c140db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:34.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.531+0000 7f5c71393700 1 -- 172.21.15.189:0/4095358159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c6c13a7f0 msgr2=0x7f5c6c13c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:34.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.531+0000 7f5c71393700 1 --2- 172.21.15.189:0/4095358159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c6c13a7f0 0x7f5c6c13c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:34.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.531+0000 7f5c71393700 1 -- 172.21.15.189:0/4095358159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c6c139c90 msgr2=0x7f5c6c140db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:34.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.531+0000 7f5c71393700 1 --2- 172.21.15.189:0/4095358159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c6c139c90 0x7f5c6c140db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:34.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.531+0000 7f5c71393700 1 -- 172.21.15.189:0/4095358159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6c141b30 con 0x7f5c6c1412f0 2024-04-06T13:28:34.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.531+0000 7f5c71393700 1 --2- 172.21.15.189:0/4095358159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c6c1412f0 0x7f5c6c13a2b0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f5c68010850 tx=0x7f5c68010b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:34.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.531+0000 7f5c5e7fc700 1 -- 172.21.15.189:0/4095358159 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c68004020 con 0x7f5c6c1412f0 2024-04-06T13:28:34.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.532+0000 7f5c5e7fc700 1 -- 172.21.15.189:0/4095358159 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c68004180 con 0x7f5c6c1412f0 2024-04-06T13:28:34.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.532+0000 7f5c73df8700 1 -- 172.21.15.189:0/4095358159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c6c141e10 con 0x7f5c6c1412f0 2024-04-06T13:28:34.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.532+0000 7f5c5e7fc700 1 -- 172.21.15.189:0/4095358159 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c68004510 con 0x7f5c6c1412f0 2024-04-06T13:28:34.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.532+0000 7f5c73df8700 1 -- 172.21.15.189:0/4095358159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c6c1423d0 con 0x7f5c6c1412f0 2024-04-06T13:28:34.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.532+0000 7f5c73df8700 1 -- 172.21.15.189:0/4095358159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c6c142560 con 0x7f5c6c1412f0 2024-04-06T13:28:34.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.533+0000 7f5c5e7fc700 1 -- 172.21.15.189:0/4095358159 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f5c680036a0 con 0x7f5c6c1412f0 2024-04-06T13:28:34.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.533+0000 7f5c5e7fc700 1 --2- 172.21.15.189:0/4095358159 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c5006adb0 0x7f5c5006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:34.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.534+0000 7f5c71b94700 1 --2- 172.21.15.189:0/4095358159 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c5006adb0 0x7f5c5006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:34.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.534+0000 7f5c5e7fc700 1 -- 172.21.15.189:0/4095358159 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5909+0+0 (secure 0 0 0) 0x7f5c68089710 con 0x7f5c6c1412f0 2024-04-06T13:28:34.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.534+0000 7f5c71b94700 1 --2- 172.21.15.189:0/4095358159 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c5006adb0 0x7f5c5006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5c60009be0 tx=0x7f5c60008040 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:34.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.538+0000 7f5c5e7fc700 1 -- 172.21.15.189:0/4095358159 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5c68054710 con 0x7f5c6c1412f0 2024-04-06T13:28:34.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.734+0000 7f5c73df8700 1 -- 172.21.15.189:0/4095358159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5c6c074990 con 0x7f5c6c1412f0 2024-04-06T13:28:34.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.738+0000 7f5c5e7fc700 1 -- 172.21.15.189:0/4095358159 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v58)=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5c68057d30 con 0x7f5c6c1412f0 2024-04-06T13:28:34.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.741+0000 7f5c73df8700 1 -- 172.21.15.189:0/4095358159 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c5006adb0 msgr2=0x7f5c5006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:34.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.741+0000 7f5c73df8700 1 --2- 172.21.15.189:0/4095358159 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c5006adb0 0x7f5c5006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5c60009be0 tx=0x7f5c60008040 comp rx=0 tx=0).stop 2024-04-06T13:28:34.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.741+0000 7f5c73df8700 1 -- 172.21.15.189:0/4095358159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c6c1412f0 msgr2=0x7f5c6c13a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:34.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.741+0000 7f5c73df8700 1 --2- 172.21.15.189:0/4095358159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c6c1412f0 0x7f5c6c13a2b0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f5c68010850 tx=0x7f5c68010b60 comp rx=0 tx=0).stop 2024-04-06T13:28:34.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.741+0000 7f5c73df8700 1 -- 172.21.15.189:0/4095358159 shutdown_connections 2024-04-06T13:28:34.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.741+0000 7f5c73df8700 1 --2- 172.21.15.189:0/4095358159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c6c13a7f0 0x7f5c6c13c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:34.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.741+0000 7f5c73df8700 1 --2- 172.21.15.189:0/4095358159 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c5006adb0 0x7f5c5006d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:34.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.741+0000 7f5c73df8700 1 --2- 172.21.15.189:0/4095358159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c6c139c90 0x7f5c6c140db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:34.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.741+0000 7f5c73df8700 1 --2- 172.21.15.189:0/4095358159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c6c1412f0 0x7f5c6c13a2b0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:34.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.741+0000 7f5c73df8700 1 -- 172.21.15.189:0/4095358159 >> 172.21.15.189:0/4095358159 conn(0x7f5c6c0d00b0 msgr2=0x7f5c6c1585c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:34.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.741+0000 7f5c73df8700 1 -- 172.21.15.189:0/4095358159 shutdown_connections 2024-04-06T13:28:34.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:34.741+0000 7f5c73df8700 1 -- 172.21.15.189:0/4095358159 wait complete. 2024-04-06T13:28:34.753 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T13:28:34.753 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:28:34.799 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:28:34.849 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T13:28:34.850 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:28:34.850 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:28:34.867 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:28:34.867 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:28:34.867 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:28:34.888 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:28:34.888 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:28:34.888 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:28:34.925 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:28:34.925 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:28:34.925 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:28:34.947 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:28:34.947 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-06T13:28:34.948 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:28:35.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.087+0000 7fc2d693d700 1 Processor -- start 2024-04-06T13:28:35.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.087+0000 7fc2d693d700 1 -- start start 2024-04-06T13:28:35.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.088+0000 7fc2d693d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc2d0158fe0 0x7fc2d006a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:35.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.088+0000 7fc2d693d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc2d006ac70 0x7fc2d015bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:35.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.088+0000 7fc2d693d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc2d015c100 0x7fc2d00717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:35.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.088+0000 7fc2d693d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2d006b0f0 con 0x7fc2d0158fe0 2024-04-06T13:28:35.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.088+0000 7fc2d693d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2d015f9e0 con 0x7fc2d015c100 2024-04-06T13:28:35.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.088+0000 7fc2d693d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d015f0d0 con 0x7fc2d006ac70 2024-04-06T13:28:35.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.088+0000 7fc2cffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc2d0158fe0 0x7fc2d006a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:35.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.088+0000 7fc2cffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc2d0158fe0 0x7fc2d006a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35410/0 (socket says 172.21.15.189:35410) 2024-04-06T13:28:35.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.088+0000 7fc2cffff700 1 -- 172.21.15.189:0/1985513510 learned_addr learned my addr 172.21.15.189:0/1985513510 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:35.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.088+0000 7fc2d4eda700 1 --2- 172.21.15.189:0/1985513510 >> v2:172.21.15.192:3300/0 conn(0x7fc2d015c100 0x7fc2d00717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:35.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.088+0000 7fc2cf7fe700 1 --2- 172.21.15.189:0/1985513510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc2d006ac70 0x7fc2d015bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:35.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.088+0000 7fc2cffff700 1 -- 172.21.15.189:0/1985513510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc2d006ac70 msgr2=0x7fc2d015bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:35.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.088+0000 7fc2cffff700 1 --2- 172.21.15.189:0/1985513510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc2d006ac70 0x7fc2d015bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:35.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.088+0000 7fc2cffff700 1 -- 172.21.15.189:0/1985513510 >> v2:172.21.15.192:3300/0 conn(0x7fc2d015c100 msgr2=0x7fc2d00717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:35.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.088+0000 7fc2cffff700 1 --2- 172.21.15.189:0/1985513510 >> v2:172.21.15.192:3300/0 conn(0x7fc2d015c100 0x7fc2d00717b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:35.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.088+0000 7fc2cffff700 1 -- 172.21.15.189:0/1985513510 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d015c8f0 con 0x7fc2d0158fe0 2024-04-06T13:28:35.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.088+0000 7fc2cffff700 1 --2- 172.21.15.189:0/1985513510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2d0158fe0 0x7fc2d006a730 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fc2d00b2d40 tx=0x7fc2c4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=423d4192fe447809 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:35.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.089+0000 7fc2ceffd700 1 -- 172.21.15.189:0/1985513510 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c400b870 con 0x7fc2d0158fe0 2024-04-06T13:28:35.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.089+0000 7fc2ceffd700 1 -- 172.21.15.189:0/1985513510 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c400b9d0 con 0x7fc2d0158fe0 2024-04-06T13:28:35.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.089+0000 7fc2ceffd700 1 -- 172.21.15.189:0/1985513510 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c400bd60 con 0x7fc2d0158fe0 2024-04-06T13:28:35.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.089+0000 7fc2d693d700 1 -- 172.21.15.189:0/1985513510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2d0158fe0 msgr2=0x7fc2d006a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:35.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.089+0000 7fc2d693d700 1 --2- 172.21.15.189:0/1985513510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2d0158fe0 0x7fc2d006a730 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fc2d00b2d40 tx=0x7fc2c4009bd0 comp rx=0 tx=0).stop 2024-04-06T13:28:35.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.089+0000 7fc2d693d700 1 -- 172.21.15.189:0/1985513510 shutdown_connections 2024-04-06T13:28:35.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.089+0000 7fc2d693d700 1 --2- 172.21.15.189:0/1985513510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2d0158fe0 0x7fc2d006a730 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:35.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.089+0000 7fc2d693d700 1 --2- 172.21.15.189:0/1985513510 >> v2:172.21.15.192:3300/0 conn(0x7fc2d015c100 0x7fc2d00717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:35.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.089+0000 7fc2d693d700 1 --2- 172.21.15.189:0/1985513510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc2d006ac70 0x7fc2d015bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:35.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.089+0000 7fc2d693d700 1 -- 172.21.15.189:0/1985513510 >> 172.21.15.189:0/1985513510 conn(0x7fc2d00d00b0 msgr2=0x7fc2d00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:35.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.089+0000 7fc2d693d700 1 -- 172.21.15.189:0/1985513510 shutdown_connections 2024-04-06T13:28:35.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.089+0000 7fc2d693d700 1 -- 172.21.15.189:0/1985513510 wait complete. 2024-04-06T13:28:35.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.091+0000 7fc2d693d700 1 Processor -- start 2024-04-06T13:28:35.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.091+0000 7fc2d693d700 1 -- start start 2024-04-06T13:28:35.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.092+0000 7fc2d693d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc2d006ac70 0x7fc2d013cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:35.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.092+0000 7fc2d693d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2d0158fe0 0x7fc2d00af230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:35.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.092+0000 7fc2d693d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2d015c100 0x7fc2d00b0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:35.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.092+0000 7fc2d693d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d015ea40 con 0x7fc2d0158fe0 2024-04-06T13:28:35.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.092+0000 7fc2d693d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d015de30 con 0x7fc2d015c100 2024-04-06T13:28:35.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.092+0000 7fc2d693d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d013d050 con 0x7fc2d006ac70 2024-04-06T13:28:35.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.092+0000 7fc2cf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2d0158fe0 0x7fc2d00af230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:35.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.092+0000 7fc2cf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2d0158fe0 0x7fc2d00af230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35424/0 (socket says 172.21.15.189:35424) 2024-04-06T13:28:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.092+0000 7fc2cf7fe700 1 -- 172.21.15.189:0/2380481558 learned_addr learned my addr 172.21.15.189:0/2380481558 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.092+0000 7fc2cffff700 1 --2- 172.21.15.189:0/2380481558 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc2d006ac70 0x7fc2d013cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.092+0000 7fc2d4eda700 1 --2- 172.21.15.189:0/2380481558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2d015c100 0x7fc2d00b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.092+0000 7fc2cf7fe700 1 -- 172.21.15.189:0/2380481558 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc2d006ac70 msgr2=0x7fc2d013cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.092+0000 7fc2cf7fe700 1 --2- 172.21.15.189:0/2380481558 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc2d006ac70 0x7fc2d013cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.092+0000 7fc2cf7fe700 1 -- 172.21.15.189:0/2380481558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2d015c100 msgr2=0x7fc2d00b0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.092+0000 7fc2cf7fe700 1 --2- 172.21.15.189:0/2380481558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2d015c100 0x7fc2d00b0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.092+0000 7fc2cf7fe700 1 -- 172.21.15.189:0/2380481558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c4009550 con 0x7fc2d0158fe0 2024-04-06T13:28:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.092+0000 7fc2cf7fe700 1 --2- 172.21.15.189:0/2380481558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2d0158fe0 0x7fc2d00af230 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fc2c000e840 tx=0x7fc2c000eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.092+0000 7fc2ccff9700 1 -- 172.21.15.189:0/2380481558 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c000c070 con 0x7fc2d0158fe0 2024-04-06T13:28:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.092+0000 7fc2ccff9700 1 -- 172.21.15.189:0/2380481558 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c0013610 con 0x7fc2d0158fe0 2024-04-06T13:28:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.093+0000 7fc2d693d700 1 -- 172.21.15.189:0/2380481558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d015e0a0 con 0x7fc2d0158fe0 2024-04-06T13:28:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.093+0000 7fc2ccff9700 1 -- 172.21.15.189:0/2380481558 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c00139a0 con 0x7fc2d0158fe0 2024-04-06T13:28:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.093+0000 7fc2d693d700 1 -- 172.21.15.189:0/2380481558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d015e660 con 0x7fc2d0158fe0 2024-04-06T13:28:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.093+0000 7fc2d693d700 1 -- 172.21.15.189:0/2380481558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2d0067bb0 con 0x7fc2d0158fe0 2024-04-06T13:28:35.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.094+0000 7fc2ccff9700 1 -- 172.21.15.189:0/2380481558 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fc2c0013b00 con 0x7fc2d0158fe0 2024-04-06T13:28:35.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.094+0000 7fc2ccff9700 1 --2- 172.21.15.189:0/2380481558 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc2bc06adb0 0x7fc2bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:35.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.094+0000 7fc2ccff9700 1 -- 172.21.15.189:0/2380481558 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5909+0+0 (secure 0 0 0) 0x7fc2c0089230 con 0x7fc2d0158fe0 2024-04-06T13:28:35.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.094+0000 7fc2cffff700 1 --2- 172.21.15.189:0/2380481558 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc2bc06adb0 0x7fc2bc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:35.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.095+0000 7fc2cffff700 1 --2- 172.21.15.189:0/2380481558 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc2bc06adb0 0x7fc2bc06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4000c00 tx=0x7fc2c4008030 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:35.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.099+0000 7fc2ccff9700 1 -- 172.21.15.189:0/2380481558 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc2c00541b0 con 0x7fc2d0158fe0 2024-04-06T13:28:35.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.335+0000 7fc2d693d700 1 -- 172.21.15.189:0/2380481558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2d0074990 con 0x7fc2d0158fe0 2024-04-06T13:28:35.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.336+0000 7fc2ccff9700 1 -- 172.21.15.189:0/2380481558 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+3541 (secure 0 0 0) 0x7fc2c00577d0 con 0x7fc2d0158fe0 2024-04-06T13:28:35.336 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:35.336 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":72,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3697447910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3697447910},{"type":"v1","addr":"172.21.15.189:6837","nonce":3697447910}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":7469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/3633157989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3633157989},{"type":"v1","addr":"172.21.15.192:6834","nonce":3633157989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2721135386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2721135386},{"type":"v1","addr":"172.21.15.189:6835","nonce":2721135386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8752,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1121148764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":1121148764},{"type":"v1","addr":"172.21.15.192:6835","nonce":1121148764}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[]} 2024-04-06T13:28:35.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.338+0000 7fc2d693d700 1 -- 172.21.15.189:0/2380481558 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc2bc06adb0 msgr2=0x7fc2bc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:35.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.338+0000 7fc2d693d700 1 --2- 172.21.15.189:0/2380481558 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc2bc06adb0 0x7fc2bc06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4000c00 tx=0x7fc2c4008030 comp rx=0 tx=0).stop 2024-04-06T13:28:35.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.338+0000 7fc2d693d700 1 -- 172.21.15.189:0/2380481558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2d0158fe0 msgr2=0x7fc2d00af230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:35.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.338+0000 7fc2d693d700 1 --2- 172.21.15.189:0/2380481558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2d0158fe0 0x7fc2d00af230 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fc2c000e840 tx=0x7fc2c000eb50 comp rx=0 tx=0).stop 2024-04-06T13:28:35.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.338+0000 7fc2d693d700 1 -- 172.21.15.189:0/2380481558 shutdown_connections 2024-04-06T13:28:35.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.338+0000 7fc2d693d700 1 --2- 172.21.15.189:0/2380481558 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc2bc06adb0 0x7fc2bc06d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:35.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.338+0000 7fc2d693d700 1 --2- 172.21.15.189:0/2380481558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2d015c100 0x7fc2d00b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:35.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.338+0000 7fc2d693d700 1 --2- 172.21.15.189:0/2380481558 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc2d006ac70 0x7fc2d013cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:35.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.338+0000 7fc2d693d700 1 --2- 172.21.15.189:0/2380481558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2d0158fe0 0x7fc2d00af230 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:35.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.338+0000 7fc2d693d700 1 -- 172.21.15.189:0/2380481558 >> 172.21.15.189:0/2380481558 conn(0x7fc2d00d00b0 msgr2=0x7fc2d00b0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:35.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.338+0000 7fc2d693d700 1 -- 172.21.15.189:0/2380481558 shutdown_connections 2024-04-06T13:28:35.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.338+0000 7fc2d693d700 1 -- 172.21.15.189:0/2380481558 wait complete. 2024-04-06T13:28:35.340 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 72 2024-04-06T13:28:35.351 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-06T13:28:35.351 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-06T13:28:35.351 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-06T13:28:35.351 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:28:35.352 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-06T13:28:35.352 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-06T13:28:35.352 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-06T13:28:35.352 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:28:35.352 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-06T13:28:35.352 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-06T13:28:35.352 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-06T13:28:35.352 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:28:35.353 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-06T13:28:35.353 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-06T13:28:35.353 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-06T13:28:35.353 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:28:35.361 INFO:tasks.ceph.mds.a:Stopped 2024-04-06T13:28:35.362 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-06T13:28:35.362 INFO:tasks.ceph.mds.c:Stopped 2024-04-06T13:28:35.363 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-06T13:28:35.364 INFO:tasks.ceph.mds.b:Stopped 2024-04-06T13:28:35.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-06T13:28:35.365 INFO:tasks.ceph.mds.d:Stopped 2024-04-06T13:28:35.365 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-06T13:28:35.404 INFO:tasks.ceph.mds.a:Started 2024-04-06T13:28:35.405 DEBUG:teuthology.parallel:result is None 2024-04-06T13:28:35.405 INFO:tasks.ceph.mds.c:Started 2024-04-06T13:28:35.405 DEBUG:teuthology.parallel:result is None 2024-04-06T13:28:35.408 INFO:tasks.ceph.mds.b:Started 2024-04-06T13:28:35.408 DEBUG:teuthology.parallel:result is None 2024-04-06T13:28:35.408 INFO:tasks.ceph.mds.d:Started 2024-04-06T13:28:35.409 DEBUG:teuthology.parallel:result is None 2024-04-06T13:28:35.409 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-06T13:28:35.484 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2024-04-06T13:28:35.484 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T13:28:35.497 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T13:28:35.498 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2024-04-06T13:28:35.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.542+0000 7f5cb45d0700 1 Processor -- start 2024-04-06T13:28:35.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.542+0000 7f5cb45d0700 1 -- start start 2024-04-06T13:28:35.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.543+0000 7f5cb45d0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5cac06be50 0x7f5cac164f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:35.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.543+0000 7f5cb45d0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cac15ddc0 con 0x7f5cac0cf2d0 2024-04-06T13:28:35.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.543+0000 7f5cb45d0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cac15e940 con 0x7f5cac15a260 2024-04-06T13:28:35.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.543+0000 7f5cb45d0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cac0af150 con 0x7f5cac06be50 2024-04-06T13:28:35.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.543+0000 7f5cb2b6d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5cac06be50 0x7f5cac164f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:35.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.543+0000 7f5cb1b6b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5cac0cf2d0 0x7f5cac06b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40942/0 (socket says 172.21.15.189:40942) 2024-04-06T13:28:35.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.543+0000 7f5cb1b6b700 1 -- 172.21.15.189:0/4028999183 learned_addr learned my addr 172.21.15.189:0/4028999183 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:35.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.543+0000 7f5cb2b6d700 1 -- 172.21.15.189:0/4028999183 >> v1:172.21.15.192:6789/0 conn(0x7f5cac15a260 legacy=0x7f5cac0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:35.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.543+0000 7f5cb2b6d700 1 -- 172.21.15.189:0/4028999183 >> v1:172.21.15.189:6789/0 conn(0x7f5cac0cf2d0 legacy=0x7f5cac06b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:35.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.543+0000 7f5cb2b6d700 1 -- 172.21.15.189:0/4028999183 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cac15e010 con 0x7f5cac06be50 2024-04-06T13:28:35.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.543+0000 7f5cb2b6d700 1 --2- 172.21.15.189:0/4028999183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cac06be50 0x7f5cac164f40 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4009d90 tx=0x7f5ca400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b33e26ac9dc3f6d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:35.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.543+0000 7f5cb136a700 1 -- 172.21.15.189:0/4028999183 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca400e070 con 0x7f5cac06be50 2024-04-06T13:28:35.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.544+0000 7f5cb136a700 1 -- 172.21.15.189:0/4028999183 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca400bbd0 con 0x7f5cac06be50 2024-04-06T13:28:35.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.544+0000 7f5cb136a700 1 -- 172.21.15.189:0/4028999183 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca4012430 con 0x7f5cac06be50 2024-04-06T13:28:35.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.544+0000 7f5cb45d0700 1 -- 172.21.15.189:0/4028999183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cac06be50 msgr2=0x7f5cac164f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:35.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.544+0000 7f5cb45d0700 1 --2- 172.21.15.189:0/4028999183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cac06be50 0x7f5cac164f40 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4009d90 tx=0x7f5ca400b330 comp rx=0 tx=0).stop 2024-04-06T13:28:35.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.544+0000 7f5cb45d0700 1 -- 172.21.15.189:0/4028999183 shutdown_connections 2024-04-06T13:28:35.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.544+0000 7f5cb45d0700 1 --2- 172.21.15.189:0/4028999183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cac06be50 0x7f5cac164f40 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:35.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.544+0000 7f5cb45d0700 1 -- 172.21.15.189:0/4028999183 >> 172.21.15.189:0/4028999183 conn(0x7f5cac070e90 msgr2=0x7f5cac0712a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:35.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.544+0000 7f5cb45d0700 1 -- 172.21.15.189:0/4028999183 shutdown_connections 2024-04-06T13:28:35.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.544+0000 7f5cb45d0700 1 -- 172.21.15.189:0/4028999183 wait complete. 2024-04-06T13:28:35.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.546+0000 7f5cb45d0700 1 Processor -- start 2024-04-06T13:28:35.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.546+0000 7f5cb45d0700 1 -- start start 2024-04-06T13:28:35.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.547+0000 7f5cb45d0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5cac06be50 0x7f5cac14e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:35.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.547+0000 7f5cb45d0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cac0cf2d0 0x7f5cac149bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:35.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.547+0000 7f5cb45d0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cac15a260 0x7f5cac1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:35.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.547+0000 7f5cb45d0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cac14b3e0 con 0x7f5cac15a260 2024-04-06T13:28:35.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.547+0000 7f5cb45d0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cac14a110 con 0x7f5cac0cf2d0 2024-04-06T13:28:35.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.547+0000 7f5cb45d0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cac14d920 con 0x7f5cac06be50 2024-04-06T13:28:35.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.547+0000 7f5cb2b6d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cac15a260 0x7f5cac1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:35.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.547+0000 7f5cb236c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5cac06be50 0x7f5cac14e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:35.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.547+0000 7f5cb2b6d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cac15a260 0x7f5cac1529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35494/0 (socket says 172.21.15.189:35494) 2024-04-06T13:28:35.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.547+0000 7f5cb1b6b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cac0cf2d0 0x7f5cac149bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:35.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.547+0000 7f5cb2b6d700 1 -- 172.21.15.189:0/224486546 learned_addr learned my addr 172.21.15.189:0/224486546 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:35.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.547+0000 7f5cb236c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5cac06be50 0x7f5cac14e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:36232/0 (socket says 172.21.15.189:36232) 2024-04-06T13:28:35.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.547+0000 7f5cb2b6d700 1 -- 172.21.15.189:0/224486546 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5cac06be50 msgr2=0x7f5cac14e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:35.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.547+0000 7f5cb2b6d700 1 --2- 172.21.15.189:0/224486546 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5cac06be50 0x7f5cac14e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:35.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.547+0000 7f5cb2b6d700 1 -- 172.21.15.189:0/224486546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cac0cf2d0 msgr2=0x7f5cac149bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:35.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.547+0000 7f5cb2b6d700 1 --2- 172.21.15.189:0/224486546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cac0cf2d0 0x7f5cac149bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:35.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.547+0000 7f5cb2b6d700 1 -- 172.21.15.189:0/224486546 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca4009a70 con 0x7f5cac15a260 2024-04-06T13:28:35.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.547+0000 7f5cb2b6d700 1 --2- 172.21.15.189:0/224486546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cac15a260 0x7f5cac1529c0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400b370 tx=0x7f5ca4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:35.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.548+0000 7f5ca2ffd700 1 -- 172.21.15.189:0/224486546 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca4012da0 con 0x7f5cac15a260 2024-04-06T13:28:35.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.548+0000 7f5ca2ffd700 1 -- 172.21.15.189:0/224486546 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca4020070 con 0x7f5cac15a260 2024-04-06T13:28:35.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.548+0000 7f5cb45d0700 1 -- 172.21.15.189:0/224486546 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cac14dbc0 con 0x7f5cac15a260 2024-04-06T13:28:35.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.548+0000 7f5cb45d0700 1 -- 172.21.15.189:0/224486546 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cac15ba20 con 0x7f5cac15a260 2024-04-06T13:28:35.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.548+0000 7f5ca2ffd700 1 -- 172.21.15.189:0/224486546 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca4004280 con 0x7f5cac15a260 2024-04-06T13:28:35.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.549+0000 7f5cb45d0700 1 -- 172.21.15.189:0/224486546 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c940052f0 con 0x7f5cac15a260 2024-04-06T13:28:35.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.549+0000 7f5ca2ffd700 1 -- 172.21.15.189:0/224486546 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f5ca40043e0 con 0x7f5cac15a260 2024-04-06T13:28:35.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.550+0000 7f5ca2ffd700 1 --2- 172.21.15.189:0/224486546 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c9006adb0 0x7f5c9006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:35.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.550+0000 7f5ca2ffd700 1 -- 172.21.15.189:0/224486546 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5909+0+0 (secure 0 0 0) 0x7f5ca4022070 con 0x7f5cac15a260 2024-04-06T13:28:35.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.550+0000 7f5cb236c700 1 --2- 172.21.15.189:0/224486546 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c9006adb0 0x7f5c9006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:35.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.550+0000 7f5cb236c700 1 --2- 172.21.15.189:0/224486546 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c9006adb0 0x7f5c9006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5cac071b40 tx=0x7f5ca8009040 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:35.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.554+0000 7f5ca2ffd700 1 -- 172.21.15.189:0/224486546 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5ca40566d0 con 0x7f5cac15a260 2024-04-06T13:28:35.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:35.766+0000 7f5cb45d0700 1 -- 172.21.15.189:0/224486546 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5c940055a0 con 0x7f5cac15a260 2024-04-06T13:28:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.742+0000 7f5ca2ffd700 1 -- 172.21.15.189:0/224486546 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v174) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5ca401ae60 con 0x7f5cac15a260 2024-04-06T13:28:36.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.745+0000 7f5cb45d0700 1 -- 172.21.15.189:0/224486546 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c9006adb0 msgr2=0x7f5c9006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:36.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.745+0000 7f5cb45d0700 1 --2- 172.21.15.189:0/224486546 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c9006adb0 0x7f5c9006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5cac071b40 tx=0x7f5ca8009040 comp rx=0 tx=0).stop 2024-04-06T13:28:36.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.745+0000 7f5cb45d0700 1 -- 172.21.15.189:0/224486546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cac15a260 msgr2=0x7f5cac1529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:36.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.745+0000 7f5cb45d0700 1 --2- 172.21.15.189:0/224486546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cac15a260 0x7f5cac1529c0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400b370 tx=0x7f5ca4004ad0 comp rx=0 tx=0).stop 2024-04-06T13:28:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.745+0000 7f5cb45d0700 1 -- 172.21.15.189:0/224486546 shutdown_connections 2024-04-06T13:28:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.745+0000 7f5cb45d0700 1 --2- 172.21.15.189:0/224486546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cac15a260 0x7f5cac1529c0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.745+0000 7f5cb45d0700 1 --2- 172.21.15.189:0/224486546 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5cac06be50 0x7f5cac14e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.745+0000 7f5cb45d0700 1 --2- 172.21.15.189:0/224486546 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f5c9006adb0 0x7f5c9006d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.745+0000 7f5cb45d0700 1 --2- 172.21.15.189:0/224486546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cac0cf2d0 0x7f5cac149bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.745+0000 7f5cb45d0700 1 -- 172.21.15.189:0/224486546 >> 172.21.15.189:0/224486546 conn(0x7f5cac070e90 msgr2=0x7f5cac159580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.745+0000 7f5cb45d0700 1 -- 172.21.15.189:0/224486546 shutdown_connections 2024-04-06T13:28:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.745+0000 7f5cb45d0700 1 -- 172.21.15.189:0/224486546 wait complete. 2024-04-06T13:28:36.746 INFO:teuthology.orchestra.run.smithi189.stderr: removed all blocklist entries 2024-04-06T13:28:36.760 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-06T13:28:36.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.889+0000 7fe1eb8f8700 1 Processor -- start 2024-04-06T13:28:36.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.889+0000 7fe1eb8f8700 1 -- start start 2024-04-06T13:28:36.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1eb8f8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1e406ae40 0x7fe1e40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:36.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1eb8f8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e40b3990 con 0x7fe1e415ddf0 2024-04-06T13:28:36.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1eb8f8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e40b3060 con 0x7fe1e415d7b0 2024-04-06T13:28:36.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1eb8f8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e40b3340 con 0x7fe1e406ae40 2024-04-06T13:28:36.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1e9e95700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe1e415ddf0 0x7fe1e415e2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40948/0 (socket says 172.21.15.189:40948) 2024-04-06T13:28:36.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1e9e95700 1 -- 172.21.15.189:0/2766129981 learned_addr learned my addr 172.21.15.189:0/2766129981 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:36.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1e8e93700 1 --2- 172.21.15.189:0/2766129981 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1e406ae40 0x7fe1e40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:36.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1dbfff700 1 -- 172.21.15.189:0/2766129981 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639143386 0 0) 0x7fe1e40b3990 con 0x7fe1e415ddf0 2024-04-06T13:28:36.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1dbfff700 1 -- 172.21.15.189:0/2766129981 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d00035b0 con 0x7fe1e415ddf0 2024-04-06T13:28:36.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1dbfff700 1 -- 172.21.15.189:0/2766129981 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1013321536 0 0) 0x7fe1d00035b0 con 0x7fe1e415ddf0 2024-04-06T13:28:36.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1dbfff700 1 -- 172.21.15.189:0/2766129981 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1e406ae40 msgr2=0x7fe1e40b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:36.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1dbfff700 1 --2- 172.21.15.189:0/2766129981 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1e406ae40 0x7fe1e40b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:36.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1dbfff700 1 -- 172.21.15.189:0/2766129981 >> v1:172.21.15.192:6789/0 conn(0x7fe1e415d7b0 legacy=0x7fe1e406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:36.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1dbfff700 1 -- 172.21.15.189:0/2766129981 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e40b3620 con 0x7fe1e415ddf0 2024-04-06T13:28:36.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1dbfff700 1 -- 172.21.15.189:0/2766129981 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fe1dc0019f0 con 0x7fe1e415ddf0 2024-04-06T13:28:36.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1dbfff700 1 -- 172.21.15.189:0/2766129981 >> v1:172.21.15.189:6789/0 conn(0x7fe1e415ddf0 legacy=0x7fe1e415e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:36.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1dbfff700 1 --2- 172.21.15.189:0/2766129981 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1e406ae40 0x7fe1d0003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:36.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1dbfff700 1 --2- 172.21.15.189:0/2766129981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1d0003f70 0x7fe1d0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:36.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1dbfff700 1 --2- 172.21.15.189:0/2766129981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1d0004d90 0x7fe1d0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:36.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1dbfff700 1 -- 172.21.15.189:0/2766129981 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e40b3340 con 0x7fe1d0004d90 2024-04-06T13:28:36.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1dbfff700 1 -- 172.21.15.189:0/2766129981 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d000bb20 con 0x7fe1d0003f70 2024-04-06T13:28:36.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.890+0000 7fe1dbfff700 1 -- 172.21.15.189:0/2766129981 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d000bce0 con 0x7fe1e406ae40 2024-04-06T13:28:36.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.891+0000 7fe1e9e95700 1 --2- 172.21.15.189:0/2766129981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1d0004d90 0x7fe1d0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:36.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.891+0000 7fe1e9694700 1 --2- 172.21.15.189:0/2766129981 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1e406ae40 0x7fe1d0003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:36.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.891+0000 7fe1e8e93700 1 --2- 172.21.15.189:0/2766129981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1d0003f70 0x7fe1d0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:36.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.891+0000 7fe1e9e95700 1 -- 172.21.15.189:0/2766129981 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1e406ae40 msgr2=0x7fe1d0003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:36.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.891+0000 7fe1e9e95700 1 --2- 172.21.15.189:0/2766129981 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1e406ae40 0x7fe1d0003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:36.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.891+0000 7fe1e9e95700 1 -- 172.21.15.189:0/2766129981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1d0003f70 msgr2=0x7fe1d0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:36.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.891+0000 7fe1e9e95700 1 --2- 172.21.15.189:0/2766129981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1d0003f70 0x7fe1d0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:36.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.891+0000 7fe1e9e95700 1 -- 172.21.15.189:0/2766129981 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d00035b0 con 0x7fe1d0004d90 2024-04-06T13:28:36.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.891+0000 7fe1e8e93700 1 --2- 172.21.15.189:0/2766129981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1d0003f70 0x7fe1d0004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:28:36.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.891+0000 7fe1e9e95700 1 --2- 172.21.15.189:0/2766129981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1d0004d90 0x7fe1d0005210 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc00a830 tx=0x7fe1d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b793936395174ec1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:36.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.891+0000 7fe1dbfff700 1 -- 172.21.15.189:0/2766129981 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc012600 con 0x7fe1d0004d90 2024-04-06T13:28:36.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.891+0000 7fe1dbfff700 1 -- 172.21.15.189:0/2766129981 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1dc012780 con 0x7fe1d0004d90 2024-04-06T13:28:36.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.891+0000 7fe1dbfff700 1 -- 172.21.15.189:0/2766129981 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc012b30 con 0x7fe1d0004d90 2024-04-06T13:28:36.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.892+0000 7fe1eb8f8700 1 -- 172.21.15.189:0/2766129981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1d0004d90 msgr2=0x7fe1d0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:36.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.892+0000 7fe1eb8f8700 1 --2- 172.21.15.189:0/2766129981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1d0004d90 0x7fe1d0005210 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc00a830 tx=0x7fe1d0005880 comp rx=0 tx=0).stop 2024-04-06T13:28:36.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.892+0000 7fe1e9e95700 1 -- 172.21.15.189:0/2766129981 reap_dead start 2024-04-06T13:28:36.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.892+0000 7fe1eb8f8700 1 -- 172.21.15.189:0/2766129981 shutdown_connections 2024-04-06T13:28:36.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.892+0000 7fe1eb8f8700 1 -- 172.21.15.189:0/2766129981 >> 172.21.15.189:0/2766129981 conn(0x7fe1e40d00b0 msgr2=0x7fe1e40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:36.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.892+0000 7fe1eb8f8700 1 -- 172.21.15.189:0/2766129981 shutdown_connections 2024-04-06T13:28:36.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.892+0000 7fe1eb8f8700 1 -- 172.21.15.189:0/2766129981 wait complete. 2024-04-06T13:28:36.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.893+0000 7fe1eb8f8700 1 Processor -- start 2024-04-06T13:28:36.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.893+0000 7fe1eb8f8700 1 -- start start 2024-04-06T13:28:36.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1eb8f8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1d0004d90 0x7fe1e41591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:36.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1eb8f8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1e4155030 0x7fe1e4145ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:36.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1eb8f8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e415beb0 0x7fe1e415c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:36.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1eb8f8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e41460e0 con 0x7fe1e415beb0 2024-04-06T13:28:36.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1eb8f8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e40afdd0 con 0x7fe1e4155030 2024-04-06T13:28:36.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1eb8f8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e415aa10 con 0x7fe1d0004d90 2024-04-06T13:28:36.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1e9e95700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e415beb0 0x7fe1e415c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:36.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1e9e95700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e415beb0 0x7fe1e415c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35518/0 (socket says 172.21.15.189:35518) 2024-04-06T13:28:36.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1e9e95700 1 -- 172.21.15.189:0/510261194 learned_addr learned my addr 172.21.15.189:0/510261194 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:36.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1e9694700 1 --2- 172.21.15.189:0/510261194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1d0004d90 0x7fe1e41591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:36.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1e8e93700 1 --2- 172.21.15.189:0/510261194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1e4155030 0x7fe1e4145ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:36.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1e9e95700 1 -- 172.21.15.189:0/510261194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1d0004d90 msgr2=0x7fe1e41591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:36.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1e9e95700 1 --2- 172.21.15.189:0/510261194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1d0004d90 0x7fe1e41591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:36.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1e9e95700 1 -- 172.21.15.189:0/510261194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1e4155030 msgr2=0x7fe1e4145ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:36.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1e9e95700 1 --2- 172.21.15.189:0/510261194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1e4155030 0x7fe1e4145ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:36.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1e9e95700 1 -- 172.21.15.189:0/510261194 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e415cee0 con 0x7fe1e415beb0 2024-04-06T13:28:36.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1e9e95700 1 --2- 172.21.15.189:0/510261194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e415beb0 0x7fe1e415c340 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc001cb0 tx=0x7fe1dc001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:36.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1d9ffb700 1 -- 172.21.15.189:0/510261194 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc012600 con 0x7fe1e415beb0 2024-04-06T13:28:36.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1d9ffb700 1 -- 172.21.15.189:0/510261194 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1dc022070 con 0x7fe1e415beb0 2024-04-06T13:28:36.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1eb8f8700 1 -- 172.21.15.189:0/510261194 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e415a450 con 0x7fe1e415beb0 2024-04-06T13:28:36.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.894+0000 7fe1d9ffb700 1 -- 172.21.15.189:0/510261194 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc01e630 con 0x7fe1e415beb0 2024-04-06T13:28:36.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.895+0000 7fe1eb8f8700 1 -- 172.21.15.189:0/510261194 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e415c880 con 0x7fe1e415beb0 2024-04-06T13:28:36.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.895+0000 7fe1eb8f8700 1 -- 172.21.15.189:0/510261194 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e40afc30 con 0x7fe1e415beb0 2024-04-06T13:28:36.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.896+0000 7fe1d9ffb700 1 -- 172.21.15.189:0/510261194 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fe1dc01e790 con 0x7fe1e415beb0 2024-04-06T13:28:36.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.897+0000 7fe1d9ffb700 1 --2- 172.21.15.189:0/510261194 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe1c806adb0 0x7fe1c806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:36.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.897+0000 7fe1e9694700 1 --2- 172.21.15.189:0/510261194 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe1c806adb0 0x7fe1c806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:36.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.897+0000 7fe1d9ffb700 1 -- 172.21.15.189:0/510261194 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5221+0+0 (secure 0 0 0) 0x7fe1dc08bad0 con 0x7fe1e415beb0 2024-04-06T13:28:36.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.897+0000 7fe1e9694700 1 --2- 172.21.15.189:0/510261194 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe1c806adb0 0x7fe1c806d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe1e40cc1b0 tx=0x7fe1e0009040 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:36.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:36.901+0000 7fe1d9ffb700 1 -- 172.21.15.189:0/510261194 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe1e40afc30 con 0x7fe1e415beb0 2024-04-06T13:28:37.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.172+0000 7fe1eb8f8700 1 -- 172.21.15.189:0/510261194 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe1e40afc30 con 0x7fe1e415beb0 2024-04-06T13:28:37.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.172+0000 7fe1d9ffb700 1 -- 172.21.15.189:0/510261194 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v41) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fe1e40afc30 con 0x7fe1e415beb0 2024-04-06T13:28:37.172 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:37.173 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:28:37.173 INFO:teuthology.orchestra.run.smithi189.stdout: "auth_dump": [ 2024-04-06T13:28:37.173 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:37.173 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.a", 2024-04-06T13:28:37.173 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmTjYMNBAA5voWGV4Pa7pE27VogrdipQ==", 2024-04-06T13:28:37.173 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:28:37.173 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:28:37.173 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:28:37.173 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:28:37.173 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:28:37.173 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.173 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:28:37.173 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:37.174 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.b", 2024-04-06T13:28:37.174 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm9A1OBRAAnHzubCvRUX8UdW8hmDzKbQ==", 2024-04-06T13:28:37.174 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:28:37.174 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:28:37.174 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:28:37.174 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:28:37.174 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:28:37.174 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.174 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:28:37.174 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:37.174 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.c", 2024-04-06T13:28:37.174 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmz0hhOxAAD3/GINNmGoFk+oPyiHHJXg==", 2024-04-06T13:28:37.174 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:28:37.175 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:28:37.175 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:28:37.175 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:28:37.175 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:28:37.175 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.175 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:28:37.175 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:37.175 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.d", 2024-04-06T13:28:37.175 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmhoAeDRAAoQC0vicWumBRTwpi3PtYaw==", 2024-04-06T13:28:37.175 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:28:37.175 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:28:37.175 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:28:37.175 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:28:37.175 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:28:37.176 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.176 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:28:37.176 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:37.176 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.0", 2024-04-06T13:28:37.176 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDkOxFmkxLWJhAAFXewCNOFX7dhAZVgUmewpQ==", 2024-04-06T13:28:37.176 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:28:37.176 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:28:37.176 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:28:37.176 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:28:37.176 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.176 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:28:37.176 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:37.176 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.1", 2024-04-06T13:28:37.177 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDnOxFmHYqWJRAABlsVCiv3e7YJMbfodJTWag==", 2024-04-06T13:28:37.177 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:28:37.177 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:28:37.177 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:28:37.177 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:28:37.177 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.177 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:28:37.177 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:37.177 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.2", 2024-04-06T13:28:37.177 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDqOxFmkhcvJBAA8NMEpHVGbGALjV9RISKP8g==", 2024-04-06T13:28:37.177 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:28:37.177 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:28:37.177 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:28:37.178 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:28:37.178 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.178 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:28:37.178 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:37.178 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.3", 2024-04-06T13:28:37.178 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDtOxFmDGzhIxAAamBoAbE8+lAel4os67meyQ==", 2024-04-06T13:28:37.178 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:28:37.178 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:28:37.178 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:28:37.178 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:28:37.178 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.178 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:28:37.179 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:37.179 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.4", 2024-04-06T13:28:37.179 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDzOxFmobZDJxAA9l8wdS81Eyw6+17I5ac6bw==", 2024-04-06T13:28:37.179 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:28:37.179 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:28:37.181 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:28:37.181 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:28:37.181 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.181 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:28:37.181 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:37.181 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.5", 2024-04-06T13:28:37.181 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD2OxFmi+ruJRAASlQ0o9jX/wA7U5AosnL+Wg==", 2024-04-06T13:28:37.181 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:28:37.181 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:28:37.181 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:28:37.181 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:28:37.181 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.182 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:28:37.182 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:37.182 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.6", 2024-04-06T13:28:37.182 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD5OxFmbwuuJBAA0wqJRmnJyskQPZJ7/LnpvA==", 2024-04-06T13:28:37.182 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:28:37.182 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:28:37.182 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:28:37.182 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:28:37.182 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.182 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:28:37.182 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:37.182 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.7", 2024-04-06T13:28:37.182 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD8OxFmDmRgIxAAzyW9AGAbmb67ZvB5qunwLQ==", 2024-04-06T13:28:37.182 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:28:37.183 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:28:37.183 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:28:37.183 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:28:37.183 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.183 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:28:37.183 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:37.183 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.0", 2024-04-06T13:28:37.183 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm0ByJERAAoEV8Yfc09+Go6u95XafElQ==", 2024-04-06T13:28:37.183 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:28:37.183 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:28:37.183 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow r", 2024-04-06T13:28:37.183 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-06T13:28:37.183 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.183 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:28:37.183 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:37.184 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.1", 2024-04-06T13:28:37.184 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmSV5lGBAAavDEo2LGy0IHpWSAuitHXg==", 2024-04-06T13:28:37.184 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:28:37.184 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:28:37.184 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:28:37.184 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:28:37.184 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:28:37.184 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.184 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:28:37.184 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:37.184 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.2", 2024-04-06T13:28:37.184 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm/Gz5FBAA3xX/rsAyWZ1JoUVlQlBIvQ==", 2024-04-06T13:28:37.184 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:28:37.184 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:28:37.184 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:28:37.185 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:28:37.185 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:28:37.185 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.185 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:28:37.185 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:37.185 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.3", 2024-04-06T13:28:37.185 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmzra7GxAADbnzIvWnRTo65LvPwtxWrg==", 2024-04-06T13:28:37.185 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:28:37.185 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:28:37.185 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:28:37.185 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:28:37.185 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:28:37.185 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.185 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:28:37.186 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:37.186 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.admin", 2024-04-06T13:28:37.186 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmxG+wCBAANg5kt6TOUQdj3/X0Mqgx1g==", 2024-04-06T13:28:37.186 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:28:37.186 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:28:37.186 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:28:37.186 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:28:37.186 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:28:37.186 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.186 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:28:37.186 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:37.186 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.x", 2024-04-06T13:28:37.186 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmPLyBMBAAYfhkI9Z1CPlf1Tijm+L7xA==", 2024-04-06T13:28:37.186 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:28:37.186 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:28:37.187 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T13:28:37.187 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:28:37.187 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.187 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:28:37.187 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:37.187 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.y", 2024-04-06T13:28:37.187 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFm1nHJHhAAQWRe8cU8duWuUJiYaDeTaQ==", 2024-04-06T13:28:37.187 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:28:37.187 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:28:37.187 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T13:28:37.187 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:28:37.187 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.187 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:37.187 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-04-06T13:28:37.188 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:28:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.175+0000 7fe1eb8f8700 1 -- 172.21.15.189:0/510261194 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe1c806adb0 msgr2=0x7fe1c806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.175+0000 7fe1eb8f8700 1 --2- 172.21.15.189:0/510261194 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe1c806adb0 0x7fe1c806d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe1e40cc1b0 tx=0x7fe1e0009040 comp rx=0 tx=0).stop 2024-04-06T13:28:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.175+0000 7fe1eb8f8700 1 -- 172.21.15.189:0/510261194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e415beb0 msgr2=0x7fe1e415c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.175+0000 7fe1eb8f8700 1 --2- 172.21.15.189:0/510261194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e415beb0 0x7fe1e415c340 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc001cb0 tx=0x7fe1dc001d80 comp rx=0 tx=0).stop 2024-04-06T13:28:37.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.175+0000 7fe1eb8f8700 1 -- 172.21.15.189:0/510261194 shutdown_connections 2024-04-06T13:28:37.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.175+0000 7fe1eb8f8700 1 --2- 172.21.15.189:0/510261194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e415beb0 0x7fe1e415c340 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.175+0000 7fe1eb8f8700 1 --2- 172.21.15.189:0/510261194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1d0004d90 0x7fe1e41591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.175+0000 7fe1eb8f8700 1 --2- 172.21.15.189:0/510261194 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe1c806adb0 0x7fe1c806d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.175+0000 7fe1eb8f8700 1 --2- 172.21.15.189:0/510261194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1e4155030 0x7fe1e4145ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.175+0000 7fe1eb8f8700 1 -- 172.21.15.189:0/510261194 >> 172.21.15.189:0/510261194 conn(0x7fe1e40d00b0 msgr2=0x7fe1e40717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:37.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.175+0000 7fe1eb8f8700 1 -- 172.21.15.189:0/510261194 shutdown_connections 2024-04-06T13:28:37.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.175+0000 7fe1eb8f8700 1 -- 172.21.15.189:0/510261194 wait complete. 2024-04-06T13:28:37.191 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:28:37.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.313+0000 7f23ece3a700 1 Processor -- start 2024-04-06T13:28:37.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.313+0000 7f23ece3a700 1 -- start start 2024-04-06T13:28:37.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.314+0000 7f23ece3a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f23e815a550 0x7f23e80cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:37.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.314+0000 7f23ece3a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e80cdbe0 con 0x7f23e80b4d70 2024-04-06T13:28:37.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.314+0000 7f23ece3a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e815a940 con 0x7f23e80cf900 2024-04-06T13:28:37.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.314+0000 7f23ece3a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23e815ac00 con 0x7f23e815a550 2024-04-06T13:28:37.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.314+0000 7f23e659c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f23e815a550 0x7f23e80cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:37.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.314+0000 7f23e6d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f23e80b4d70 0x7f23e80b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40960/0 (socket says 172.21.15.189:40960) 2024-04-06T13:28:37.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.314+0000 7f23e6d9d700 1 -- 172.21.15.189:0/3597010902 learned_addr learned my addr 172.21.15.189:0/3597010902 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:37.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.315+0000 7f23e659c700 1 -- 172.21.15.189:0/3597010902 >> v1:172.21.15.192:6789/0 conn(0x7f23e80cf900 legacy=0x7f23e80b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:37.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.315+0000 7f23e659c700 1 -- 172.21.15.189:0/3597010902 >> v1:172.21.15.189:6789/0 conn(0x7f23e80b4d70 legacy=0x7f23e80b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:37.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.315+0000 7f23e659c700 1 -- 172.21.15.189:0/3597010902 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e815aee0 con 0x7f23e815a550 2024-04-06T13:28:37.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.315+0000 7f23e559a700 1 -- 172.21.15.189:0/3597010902 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399786418 0 0) 0x7f23e80cdbe0 con 0x7f23e80b4d70 2024-04-06T13:28:37.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.315+0000 7f23e659c700 1 --2- 172.21.15.189:0/3597010902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e815a550 0x7f23e80cf3c0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f23d800bae0 tx=0x7f23d800bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=255cb5e43de0e987 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:37.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.315+0000 7f23e559a700 1 -- 172.21.15.189:0/3597010902 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d800dcf0 con 0x7f23e815a550 2024-04-06T13:28:37.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.315+0000 7f23e559a700 1 -- 172.21.15.189:0/3597010902 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d800de50 con 0x7f23e815a550 2024-04-06T13:28:37.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.315+0000 7f23e559a700 1 -- 172.21.15.189:0/3597010902 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d8004250 con 0x7f23e815a550 2024-04-06T13:28:37.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.315+0000 7f23ece3a700 1 -- 172.21.15.189:0/3597010902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e815a550 msgr2=0x7f23e80cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:37.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.315+0000 7f23ece3a700 1 --2- 172.21.15.189:0/3597010902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e815a550 0x7f23e80cf3c0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f23d800bae0 tx=0x7f23d800bdf0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.316+0000 7f23ece3a700 1 -- 172.21.15.189:0/3597010902 shutdown_connections 2024-04-06T13:28:37.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.316+0000 7f23ece3a700 1 --2- 172.21.15.189:0/3597010902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e815a550 0x7f23e80cf3c0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.316+0000 7f23ece3a700 1 -- 172.21.15.189:0/3597010902 >> 172.21.15.189:0/3597010902 conn(0x7f23e80d00b0 msgr2=0x7f23e80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:37.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.316+0000 7f23ece3a700 1 -- 172.21.15.189:0/3597010902 shutdown_connections 2024-04-06T13:28:37.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.316+0000 7f23ece3a700 1 -- 172.21.15.189:0/3597010902 wait complete. 2024-04-06T13:28:37.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.317+0000 7f23ece3a700 1 Processor -- start 2024-04-06T13:28:37.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.317+0000 7f23ece3a700 1 -- start start 2024-04-06T13:28:37.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.318+0000 7f23ece3a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23e80b4d70 0x7f23e80aa3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:37.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.318+0000 7f23ece3a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23e80cf900 0x7f23e81efae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:37.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.318+0000 7f23ece3a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e81589c0 0x7f23e8158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:37.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.318+0000 7f23ece3a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e80aac20 con 0x7f23e81589c0 2024-04-06T13:28:37.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.318+0000 7f23ece3a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e80aaf00 con 0x7f23e80cf900 2024-04-06T13:28:37.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.318+0000 7f23ece3a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e80ab1e0 con 0x7f23e80b4d70 2024-04-06T13:28:37.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.318+0000 7f23e6d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e81589c0 0x7f23e8158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:37.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.318+0000 7f23e6d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e81589c0 0x7f23e8158e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35540/0 (socket says 172.21.15.189:35540) 2024-04-06T13:28:37.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.318+0000 7f23e6d9d700 1 -- 172.21.15.189:0/3221041891 learned_addr learned my addr 172.21.15.189:0/3221041891 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:37.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.318+0000 7f23e659c700 1 --2- 172.21.15.189:0/3221041891 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23e80b4d70 0x7f23e80aa3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:37.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.318+0000 7f23e5d9b700 1 --2- 172.21.15.189:0/3221041891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23e80cf900 0x7f23e81efae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:37.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.319+0000 7f23e6d9d700 1 -- 172.21.15.189:0/3221041891 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23e80b4d70 msgr2=0x7f23e80aa3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:37.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.319+0000 7f23e6d9d700 1 --2- 172.21.15.189:0/3221041891 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23e80b4d70 0x7f23e80aa3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.319+0000 7f23e6d9d700 1 -- 172.21.15.189:0/3221041891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23e80cf900 msgr2=0x7f23e81efae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:37.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.319+0000 7f23e6d9d700 1 --2- 172.21.15.189:0/3221041891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23e80cf900 0x7f23e81efae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.319+0000 7f23e6d9d700 1 -- 172.21.15.189:0/3221041891 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d800b740 con 0x7f23e81589c0 2024-04-06T13:28:37.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.319+0000 7f23e6d9d700 1 --2- 172.21.15.189:0/3221041891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e81589c0 0x7f23e8158e50 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f23e000cca0 tx=0x7f23e000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:37.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.319+0000 7f23d6ffd700 1 -- 172.21.15.189:0/3221041891 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e000cea0 con 0x7f23e81589c0 2024-04-06T13:28:37.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.319+0000 7f23d6ffd700 1 -- 172.21.15.189:0/3221041891 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e000ade0 con 0x7f23e81589c0 2024-04-06T13:28:37.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.319+0000 7f23ece3a700 1 -- 172.21.15.189:0/3221041891 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e80a8ab0 con 0x7f23e81589c0 2024-04-06T13:28:37.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.319+0000 7f23d6ffd700 1 -- 172.21.15.189:0/3221041891 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e0007310 con 0x7f23e81589c0 2024-04-06T13:28:37.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.319+0000 7f23ece3a700 1 -- 172.21.15.189:0/3221041891 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e80aa910 con 0x7f23e81589c0 2024-04-06T13:28:37.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.320+0000 7f23ece3a700 1 -- 172.21.15.189:0/3221041891 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e80a8cf0 con 0x7f23e81589c0 2024-04-06T13:28:37.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.320+0000 7f23d6ffd700 1 -- 172.21.15.189:0/3221041891 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f23e0007470 con 0x7f23e81589c0 2024-04-06T13:28:37.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.321+0000 7f23d6ffd700 1 --2- 172.21.15.189:0/3221041891 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f23c806adb0 0x7f23c806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:37.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.321+0000 7f23d6ffd700 1 -- 172.21.15.189:0/3221041891 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f23e0089210 con 0x7f23e81589c0 2024-04-06T13:28:37.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.321+0000 7f23e659c700 1 --2- 172.21.15.189:0/3221041891 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f23c806adb0 0x7f23c806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:37.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.321+0000 7f23e659c700 1 --2- 172.21.15.189:0/3221041891 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f23c806adb0 0x7f23c806d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f23d800be30 tx=0x7f23d801b040 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:37.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.325+0000 7f23d6ffd700 1 -- 172.21.15.189:0/3221041891 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f23e80a8cf0 con 0x7f23e81589c0 2024-04-06T13:28:37.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.522+0000 7f23ece3a700 1 -- 172.21.15.189:0/3221041891 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23e8067bb0 con 0x7f23e81589c0 2024-04-06T13:28:37.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.522+0000 7f23d6ffd700 1 -- 172.21.15.189:0/3221041891 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+12722 (secure 0 0 0) 0x7f23e0054550 con 0x7f23e81589c0 2024-04-06T13:28:37.522 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:37.523 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":174,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:28:36.737769+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:28:37.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.525+0000 7f23ece3a700 1 -- 172.21.15.189:0/3221041891 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f23c806adb0 msgr2=0x7f23c806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:37.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.525+0000 7f23ece3a700 1 --2- 172.21.15.189:0/3221041891 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f23c806adb0 0x7f23c806d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f23d800be30 tx=0x7f23d801b040 comp rx=0 tx=0).stop 2024-04-06T13:28:37.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.525+0000 7f23ece3a700 1 -- 172.21.15.189:0/3221041891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e81589c0 msgr2=0x7f23e8158e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:37.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.525+0000 7f23ece3a700 1 --2- 172.21.15.189:0/3221041891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e81589c0 0x7f23e8158e50 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f23e000cca0 tx=0x7f23e000a340 comp rx=0 tx=0).stop 2024-04-06T13:28:37.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.525+0000 7f23ece3a700 1 -- 172.21.15.189:0/3221041891 shutdown_connections 2024-04-06T13:28:37.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.525+0000 7f23ece3a700 1 --2- 172.21.15.189:0/3221041891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e81589c0 0x7f23e8158e50 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.525+0000 7f23ece3a700 1 --2- 172.21.15.189:0/3221041891 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23e80b4d70 0x7f23e80aa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.525+0000 7f23ece3a700 1 --2- 172.21.15.189:0/3221041891 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f23c806adb0 0x7f23c806d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.525+0000 7f23ece3a700 1 --2- 172.21.15.189:0/3221041891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23e80cf900 0x7f23e81efae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.525+0000 7f23ece3a700 1 -- 172.21.15.189:0/3221041891 >> 172.21.15.189:0/3221041891 conn(0x7f23e80d00b0 msgr2=0x7f23e80b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:37.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.525+0000 7f23ece3a700 1 -- 172.21.15.189:0/3221041891 shutdown_connections 2024-04-06T13:28:37.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.525+0000 7f23ece3a700 1 -- 172.21.15.189:0/3221041891 wait complete. 2024-04-06T13:28:37.537 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:28:37.538 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:28:37.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.662+0000 7fe5c6787700 1 Processor -- start 2024-04-06T13:28:37.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.662+0000 7fe5c6787700 1 -- start start 2024-04-06T13:28:37.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.663+0000 7fe5c6787700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5c015d7b0 0x7fe5c00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:37.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.663+0000 7fe5c6787700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe5c006a730 0x7fe5c006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:37.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.663+0000 7fe5c6787700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe5c015ddf0 0x7fe5c015e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:37.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.663+0000 7fe5c6787700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c006b0f0 con 0x7fe5c006a730 2024-04-06T13:28:37.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.663+0000 7fe5c6787700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c00b3990 con 0x7fe5c015ddf0 2024-04-06T13:28:37.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.663+0000 7fe5c6787700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c00b3060 con 0x7fe5c015d7b0 2024-04-06T13:28:37.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.663+0000 7fe5bf7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe5c006a730 0x7fe5c006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:37.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.663+0000 7fe5bf7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe5c006a730 0x7fe5c006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35552/0 (socket says 172.21.15.189:35552) 2024-04-06T13:28:37.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.663+0000 7fe5bf7fe700 1 -- 172.21.15.189:0/814891146 learned_addr learned my addr 172.21.15.189:0/814891146 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:37.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.663+0000 7fe5bffff700 1 --2- 172.21.15.189:0/814891146 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5c015d7b0 0x7fe5c00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:37.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.663+0000 7fe5bf7fe700 1 -- 172.21.15.189:0/814891146 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5c015d7b0 msgr2=0x7fe5c00b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:37.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.663+0000 7fe5bf7fe700 1 --2- 172.21.15.189:0/814891146 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5c015d7b0 0x7fe5c00b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.663+0000 7fe5c4d24700 1 --2- 172.21.15.189:0/814891146 >> v2:172.21.15.192:3300/0 conn(0x7fe5c015ddf0 0x7fe5c015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:37.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.663+0000 7fe5bf7fe700 1 -- 172.21.15.189:0/814891146 >> v2:172.21.15.192:3300/0 conn(0x7fe5c015ddf0 msgr2=0x7fe5c015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:37.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.663+0000 7fe5bf7fe700 1 --2- 172.21.15.189:0/814891146 >> v2:172.21.15.192:3300/0 conn(0x7fe5c015ddf0 0x7fe5c015e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.663+0000 7fe5bf7fe700 1 -- 172.21.15.189:0/814891146 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c00b3340 con 0x7fe5c006a730 2024-04-06T13:28:37.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.663+0000 7fe5bffff700 1 --2- 172.21.15.189:0/814891146 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5c015d7b0 0x7fe5c00b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:28:37.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.663+0000 7fe5c4d24700 1 --2- 172.21.15.189:0/814891146 >> v2:172.21.15.192:3300/0 conn(0x7fe5c015ddf0 0x7fe5c015e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:28:37.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.663+0000 7fe5bf7fe700 1 --2- 172.21.15.189:0/814891146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5c006a730 0x7fe5c006abb0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fe5c00b4800 tx=0x7fe5b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7277399898f16d3d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:37.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.664+0000 7fe5beffd700 1 -- 172.21.15.189:0/814891146 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b000b870 con 0x7fe5c006a730 2024-04-06T13:28:37.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.664+0000 7fe5beffd700 1 -- 172.21.15.189:0/814891146 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b000b9d0 con 0x7fe5c006a730 2024-04-06T13:28:37.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.664+0000 7fe5beffd700 1 -- 172.21.15.189:0/814891146 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b000bd60 con 0x7fe5c006a730 2024-04-06T13:28:37.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.664+0000 7fe5c6787700 1 -- 172.21.15.189:0/814891146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5c006a730 msgr2=0x7fe5c006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:37.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.664+0000 7fe5c6787700 1 --2- 172.21.15.189:0/814891146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5c006a730 0x7fe5c006abb0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fe5c00b4800 tx=0x7fe5b0009bd0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.664+0000 7fe5c6787700 1 -- 172.21.15.189:0/814891146 shutdown_connections 2024-04-06T13:28:37.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.664+0000 7fe5c6787700 1 --2- 172.21.15.189:0/814891146 >> v2:172.21.15.192:3300/0 conn(0x7fe5c015ddf0 0x7fe5c015e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.664+0000 7fe5c6787700 1 --2- 172.21.15.189:0/814891146 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5c015d7b0 0x7fe5c00b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.664+0000 7fe5c6787700 1 --2- 172.21.15.189:0/814891146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5c006a730 0x7fe5c006abb0 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.664+0000 7fe5c6787700 1 -- 172.21.15.189:0/814891146 >> 172.21.15.189:0/814891146 conn(0x7fe5c00d00b0 msgr2=0x7fe5c00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:37.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.664+0000 7fe5c6787700 1 -- 172.21.15.189:0/814891146 shutdown_connections 2024-04-06T13:28:37.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.664+0000 7fe5c6787700 1 -- 172.21.15.189:0/814891146 wait complete. 2024-04-06T13:28:37.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.665+0000 7fe5c6787700 1 Processor -- start 2024-04-06T13:28:37.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.666+0000 7fe5c6787700 1 -- start start 2024-04-06T13:28:37.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.666+0000 7fe5c6787700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5c006a730 0x7fe5c014f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:37.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.666+0000 7fe5c6787700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5c015d7b0 0x7fe5c015c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:37.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.666+0000 7fe5c6787700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5c015ddf0 0x7fe5c015b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:37.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.666+0000 7fe5c6787700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c00cd210 con 0x7fe5c015ddf0 2024-04-06T13:28:37.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.666+0000 7fe5c6787700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c00cd4f0 con 0x7fe5c006a730 2024-04-06T13:28:37.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.666+0000 7fe5c6787700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c00cd7d0 con 0x7fe5c015d7b0 2024-04-06T13:28:37.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.666+0000 7fe5c4d24700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5c015ddf0 0x7fe5c015b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:37.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.666+0000 7fe5c4d24700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5c015ddf0 0x7fe5c015b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35564/0 (socket says 172.21.15.189:35564) 2024-04-06T13:28:37.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.666+0000 7fe5c4d24700 1 -- 172.21.15.189:0/2992685821 learned_addr learned my addr 172.21.15.189:0/2992685821 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:37.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.667+0000 7fe5bffff700 1 --2- 172.21.15.189:0/2992685821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5c006a730 0x7fe5c014f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:37.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.667+0000 7fe5bf7fe700 1 --2- 172.21.15.189:0/2992685821 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5c015d7b0 0x7fe5c015c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:37.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.667+0000 7fe5c4d24700 1 -- 172.21.15.189:0/2992685821 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5c015d7b0 msgr2=0x7fe5c015c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:37.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.667+0000 7fe5c4d24700 1 --2- 172.21.15.189:0/2992685821 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5c015d7b0 0x7fe5c015c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.667+0000 7fe5c4d24700 1 -- 172.21.15.189:0/2992685821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5c006a730 msgr2=0x7fe5c014f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:37.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.667+0000 7fe5c4d24700 1 --2- 172.21.15.189:0/2992685821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5c006a730 0x7fe5c014f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.667+0000 7fe5c4d24700 1 -- 172.21.15.189:0/2992685821 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b0009550 con 0x7fe5c015ddf0 2024-04-06T13:28:37.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.667+0000 7fe5bf7fe700 1 --2- 172.21.15.189:0/2992685821 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5c015d7b0 0x7fe5c015c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:28:37.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.667+0000 7fe5bffff700 1 --2- 172.21.15.189:0/2992685821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5c006a730 0x7fe5c014f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:28:37.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.667+0000 7fe5c4d24700 1 --2- 172.21.15.189:0/2992685821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5c015ddf0 0x7fe5c015b3c0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7fe5b800cd50 tx=0x7fe5b800a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:37.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.667+0000 7fe5bcff9700 1 -- 172.21.15.189:0/2992685821 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b800f480 con 0x7fe5c015ddf0 2024-04-06T13:28:37.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.667+0000 7fe5bcff9700 1 -- 172.21.15.189:0/2992685821 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b800f600 con 0x7fe5c015ddf0 2024-04-06T13:28:37.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.667+0000 7fe5bcff9700 1 -- 172.21.15.189:0/2992685821 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b800f9b0 con 0x7fe5c015ddf0 2024-04-06T13:28:37.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.667+0000 7fe5c6787700 1 -- 172.21.15.189:0/2992685821 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c00cdab0 con 0x7fe5c015ddf0 2024-04-06T13:28:37.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.667+0000 7fe5c6787700 1 -- 172.21.15.189:0/2992685821 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c014f8e0 con 0x7fe5c015ddf0 2024-04-06T13:28:37.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.668+0000 7fe5c6787700 1 -- 172.21.15.189:0/2992685821 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c00cdfd0 con 0x7fe5c015ddf0 2024-04-06T13:28:37.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.668+0000 7fe5bcff9700 1 -- 172.21.15.189:0/2992685821 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fe5b800b040 con 0x7fe5c015ddf0 2024-04-06T13:28:37.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.669+0000 7fe5bcff9700 1 --2- 172.21.15.189:0/2992685821 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe5ac06adb0 0x7fe5ac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:37.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.669+0000 7fe5bcff9700 1 -- 172.21.15.189:0/2992685821 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5221+0+0 (secure 0 0 0) 0x7fe5b8089bb0 con 0x7fe5c015ddf0 2024-04-06T13:28:37.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.669+0000 7fe5bffff700 1 --2- 172.21.15.189:0/2992685821 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe5ac06adb0 0x7fe5ac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:37.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.669+0000 7fe5bffff700 1 --2- 172.21.15.189:0/2992685821 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe5ac06adb0 0x7fe5ac06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4006fd0 tx=0x7fe5b4008040 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:37.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.673+0000 7fe5bcff9700 1 -- 172.21.15.189:0/2992685821 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe5c00cdfd0 con 0x7fe5c015ddf0 2024-04-06T13:28:37.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.867+0000 7fe5c6787700 1 -- 172.21.15.189:0/2992685821 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe5c00cdfd0 con 0x7fe5c015ddf0 2024-04-06T13:28:37.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.867+0000 7fe5bcff9700 1 -- 172.21.15.189:0/2992685821 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v174) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe5c00cdfd0 con 0x7fe5c015ddf0 2024-04-06T13:28:37.867 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:28:37.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.869+0000 7fe5c6787700 1 -- 172.21.15.189:0/2992685821 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe5ac06adb0 msgr2=0x7fe5ac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:37.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.870+0000 7fe5c6787700 1 --2- 172.21.15.189:0/2992685821 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe5ac06adb0 0x7fe5ac06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4006fd0 tx=0x7fe5b4008040 comp rx=0 tx=0).stop 2024-04-06T13:28:37.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.870+0000 7fe5c6787700 1 -- 172.21.15.189:0/2992685821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5c015ddf0 msgr2=0x7fe5c015b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:37.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.870+0000 7fe5c6787700 1 --2- 172.21.15.189:0/2992685821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5c015ddf0 0x7fe5c015b3c0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7fe5b800cd50 tx=0x7fe5b800a560 comp rx=0 tx=0).stop 2024-04-06T13:28:37.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.870+0000 7fe5c6787700 1 -- 172.21.15.189:0/2992685821 shutdown_connections 2024-04-06T13:28:37.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.870+0000 7fe5c6787700 1 --2- 172.21.15.189:0/2992685821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5c015ddf0 0x7fe5c015b3c0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.870+0000 7fe5c6787700 1 --2- 172.21.15.189:0/2992685821 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fe5ac06adb0 0x7fe5ac06d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.870+0000 7fe5c6787700 1 --2- 172.21.15.189:0/2992685821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5c006a730 0x7fe5c014f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.870+0000 7fe5c6787700 1 --2- 172.21.15.189:0/2992685821 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5c015d7b0 0x7fe5c015c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:37.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.870+0000 7fe5c6787700 1 -- 172.21.15.189:0/2992685821 >> 172.21.15.189:0/2992685821 conn(0x7fe5c00d00b0 msgr2=0x7fe5c00acd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:37.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.870+0000 7fe5c6787700 1 -- 172.21.15.189:0/2992685821 shutdown_connections 2024-04-06T13:28:37.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:37.870+0000 7fe5c6787700 1 -- 172.21.15.189:0/2992685821 wait complete. 2024-04-06T13:28:37.882 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-06T13:28:38.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.003+0000 7f8e4f578700 1 Processor -- start 2024-04-06T13:28:38.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.003+0000 7f8e4f578700 1 -- start start 2024-04-06T13:28:38.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.004+0000 7f8e4f578700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8e4806ae40 0x7f8e480b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:38.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.004+0000 7f8e4f578700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8e4815ddf0 0x7f8e4815e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:38.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.004+0000 7f8e4f578700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e480b3990 con 0x7f8e4815d7b0 2024-04-06T13:28:38.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.004+0000 7f8e4f578700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e480b3060 con 0x7f8e4806ae40 2024-04-06T13:28:38.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.004+0000 7f8e4f578700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e480b3340 con 0x7f8e4815ddf0 2024-04-06T13:28:38.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.004+0000 7f8e4cb13700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8e4806ae40 0x7f8e480b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:38.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.004+0000 7f8e4d314700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8e4815d7b0 0x7f8e4806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40976/0 (socket says 172.21.15.189:40976) 2024-04-06T13:28:38.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.004+0000 7f8e4cb13700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8e4806ae40 0x7f8e480b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35578/0 (socket says 172.21.15.189:35578) 2024-04-06T13:28:38.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.004+0000 7f8e4d314700 1 -- 172.21.15.189:0/274706924 learned_addr learned my addr 172.21.15.189:0/274706924 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:38.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.005+0000 7f8e4db15700 1 --2- 172.21.15.189:0/274706924 >> v2:172.21.15.192:3300/0 conn(0x7f8e4815ddf0 0x7f8e4815e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:38.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.005+0000 7f8e4cb13700 1 -- 172.21.15.189:0/274706924 >> v2:172.21.15.192:3300/0 conn(0x7f8e4815ddf0 msgr2=0x7f8e4815e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:38.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.005+0000 7f8e4cb13700 1 --2- 172.21.15.189:0/274706924 >> v2:172.21.15.192:3300/0 conn(0x7f8e4815ddf0 0x7f8e4815e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:38.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.005+0000 7f8e4cb13700 1 -- 172.21.15.189:0/274706924 >> v1:172.21.15.189:6789/0 conn(0x7f8e4815d7b0 legacy=0x7f8e4806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:38.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.005+0000 7f8e3ffff700 1 -- 172.21.15.189:0/274706924 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968234810 0 0) 0x7f8e480b3990 con 0x7f8e4815d7b0 2024-04-06T13:28:38.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.005+0000 7f8e4cb13700 1 -- 172.21.15.189:0/274706924 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e480b3620 con 0x7f8e4806ae40 2024-04-06T13:28:38.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.005+0000 7f8e4db15700 1 --2- 172.21.15.189:0/274706924 >> v2:172.21.15.192:3300/0 conn(0x7f8e4815ddf0 0x7f8e4815e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:28:38.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.005+0000 7f8e4cb13700 1 --2- 172.21.15.189:0/274706924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4806ae40 0x7f8e480b3e20 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f8e4806d2c0 tx=0x7f8e44009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf6429ba145150ec server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:38.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.005+0000 7f8e3ffff700 1 -- 172.21.15.189:0/274706924 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e44011830 con 0x7f8e4806ae40 2024-04-06T13:28:38.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.005+0000 7f8e3ffff700 1 -- 172.21.15.189:0/274706924 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e44011990 con 0x7f8e4806ae40 2024-04-06T13:28:38.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.005+0000 7f8e3ffff700 1 -- 172.21.15.189:0/274706924 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e44011d20 con 0x7f8e4806ae40 2024-04-06T13:28:38.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.005+0000 7f8e4f578700 1 -- 172.21.15.189:0/274706924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4806ae40 msgr2=0x7f8e480b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:38.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.005+0000 7f8e4f578700 1 --2- 172.21.15.189:0/274706924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4806ae40 0x7f8e480b3e20 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f8e4806d2c0 tx=0x7f8e44009b20 comp rx=0 tx=0).stop 2024-04-06T13:28:38.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.006+0000 7f8e4f578700 1 -- 172.21.15.189:0/274706924 shutdown_connections 2024-04-06T13:28:38.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.006+0000 7f8e4f578700 1 --2- 172.21.15.189:0/274706924 >> v2:172.21.15.192:3300/0 conn(0x7f8e4815ddf0 0x7f8e4815e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:38.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.006+0000 7f8e4f578700 1 --2- 172.21.15.189:0/274706924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4806ae40 0x7f8e480b3e20 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:38.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.006+0000 7f8e4f578700 1 -- 172.21.15.189:0/274706924 >> 172.21.15.189:0/274706924 conn(0x7f8e480d00b0 msgr2=0x7f8e480d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:38.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.006+0000 7f8e4f578700 1 -- 172.21.15.189:0/274706924 shutdown_connections 2024-04-06T13:28:38.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.006+0000 7f8e4f578700 1 -- 172.21.15.189:0/274706924 wait complete. 2024-04-06T13:28:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.007+0000 7f8e4f578700 1 Processor -- start 2024-04-06T13:28:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.007+0000 7f8e4f578700 1 -- start start 2024-04-06T13:28:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.007+0000 7f8e4f578700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e4806ae40 0x7f8e4814a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.007+0000 7f8e4f578700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4815d7b0 0x7f8e4806eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e4f578700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e4815ddf0 0x7f8e4806f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e4f578700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e480afb90 con 0x7f8e4815d7b0 2024-04-06T13:28:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e4f578700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4814a600 con 0x7f8e4815ddf0 2024-04-06T13:28:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e4f578700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e480af8b0 con 0x7f8e4806ae40 2024-04-06T13:28:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e4cb13700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4815d7b0 0x7f8e4806eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e4cb13700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4815d7b0 0x7f8e4806eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35582/0 (socket says 172.21.15.189:35582) 2024-04-06T13:28:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e4cb13700 1 -- 172.21.15.189:0/2662347139 learned_addr learned my addr 172.21.15.189:0/2662347139 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e4db15700 1 --2- 172.21.15.189:0/2662347139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e4815ddf0 0x7f8e4806f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e4d314700 1 --2- 172.21.15.189:0/2662347139 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e4806ae40 0x7f8e4814a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e4cb13700 1 -- 172.21.15.189:0/2662347139 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e4806ae40 msgr2=0x7f8e4814a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e4cb13700 1 --2- 172.21.15.189:0/2662347139 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e4806ae40 0x7f8e4814a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e4cb13700 1 -- 172.21.15.189:0/2662347139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e4815ddf0 msgr2=0x7f8e4806f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e4cb13700 1 --2- 172.21.15.189:0/2662347139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e4815ddf0 0x7f8e4806f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e4cb13700 1 -- 172.21.15.189:0/2662347139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e440094a0 con 0x7f8e4815d7b0 2024-04-06T13:28:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e4db15700 1 --2- 172.21.15.189:0/2662347139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e4815ddf0 0x7f8e4806f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:28:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e4cb13700 1 --2- 172.21.15.189:0/2662347139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4815d7b0 0x7f8e4806eb50 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f8e44000c00 tx=0x7f8e44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e3dffb700 1 -- 172.21.15.189:0/2662347139 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e440038a0 con 0x7f8e4815d7b0 2024-04-06T13:28:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e3dffb700 1 -- 172.21.15.189:0/2662347139 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4401f070 con 0x7f8e4815d7b0 2024-04-06T13:28:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e4f578700 1 -- 172.21.15.189:0/2662347139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e481556d0 con 0x7f8e4815d7b0 2024-04-06T13:28:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e3dffb700 1 -- 172.21.15.189:0/2662347139 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e44003c40 con 0x7f8e4815d7b0 2024-04-06T13:28:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.008+0000 7f8e4f578700 1 -- 172.21.15.189:0/2662347139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e48155c90 con 0x7f8e4815d7b0 2024-04-06T13:28:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.009+0000 7f8e4f578700 1 -- 172.21.15.189:0/2662347139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e48155530 con 0x7f8e4815d7b0 2024-04-06T13:28:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.010+0000 7f8e3dffb700 1 -- 172.21.15.189:0/2662347139 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f8e44026480 con 0x7f8e4815d7b0 2024-04-06T13:28:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.010+0000 7f8e3dffb700 1 --2- 172.21.15.189:0/2662347139 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8e2c06adb0 0x7f8e2c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:38.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.010+0000 7f8e3dffb700 1 -- 172.21.15.189:0/2662347139 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f8e44021070 con 0x7f8e4815d7b0 2024-04-06T13:28:38.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.010+0000 7f8e4d314700 1 --2- 172.21.15.189:0/2662347139 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8e2c06adb0 0x7f8e2c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:38.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.011+0000 7f8e4d314700 1 --2- 172.21.15.189:0/2662347139 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8e2c06adb0 0x7f8e2c06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8e38000f60 tx=0x7f8e38009040 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:38.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.015+0000 7f8e3dffb700 1 -- 172.21.15.189:0/2662347139 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8e48155530 con 0x7f8e4815d7b0 2024-04-06T13:28:38.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.209+0000 7f8e4f578700 1 -- 172.21.15.189:0/2662347139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8e48155530 con 0x7f8e4815d7b0 2024-04-06T13:28:38.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.209+0000 7f8e3dffb700 1 -- 172.21.15.189:0/2662347139 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v174) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8e48155530 con 0x7f8e4815d7b0 2024-04-06T13:28:38.210 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:38.210 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T13:28:38.210 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:38.210 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 1, 2024-04-06T13:28:38.210 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": ".mgr" 2024-04-06T13:28:38.210 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:38.210 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T13:28:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.211+0000 7f8e4f578700 1 -- 172.21.15.189:0/2662347139 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8e2c06adb0 msgr2=0x7f8e2c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.211+0000 7f8e4f578700 1 --2- 172.21.15.189:0/2662347139 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8e2c06adb0 0x7f8e2c06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8e38000f60 tx=0x7f8e38009040 comp rx=0 tx=0).stop 2024-04-06T13:28:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.212+0000 7f8e4f578700 1 -- 172.21.15.189:0/2662347139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4815d7b0 msgr2=0x7f8e4806eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.212+0000 7f8e4f578700 1 --2- 172.21.15.189:0/2662347139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4815d7b0 0x7f8e4806eb50 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f8e44000c00 tx=0x7f8e44004750 comp rx=0 tx=0).stop 2024-04-06T13:28:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.212+0000 7f8e4f578700 1 -- 172.21.15.189:0/2662347139 shutdown_connections 2024-04-06T13:28:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.212+0000 7f8e4f578700 1 --2- 172.21.15.189:0/2662347139 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8e2c06adb0 0x7f8e2c06d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.212+0000 7f8e4f578700 1 --2- 172.21.15.189:0/2662347139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e4815ddf0 0x7f8e4806f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.212+0000 7f8e4f578700 1 --2- 172.21.15.189:0/2662347139 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e4806ae40 0x7f8e4814a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.212+0000 7f8e4f578700 1 --2- 172.21.15.189:0/2662347139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4815d7b0 0x7f8e4806eb50 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:38.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.212+0000 7f8e4f578700 1 -- 172.21.15.189:0/2662347139 >> 172.21.15.189:0/2662347139 conn(0x7f8e480d00b0 msgr2=0x7f8e4815f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:38.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.212+0000 7f8e4f578700 1 -- 172.21.15.189:0/2662347139 shutdown_connections 2024-04-06T13:28:38.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.212+0000 7f8e4f578700 1 -- 172.21.15.189:0/2662347139 wait complete. 2024-04-06T13:28:38.224 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-06T13:28:38.224 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-06T13:28:38.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.351+0000 7f2a7d8ce700 1 Processor -- start 2024-04-06T13:28:38.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.351+0000 7f2a7d8ce700 1 -- start start 2024-04-06T13:28:38.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.352+0000 7f2a7d8ce700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a780ce810 0x7f2a78168230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:38.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.352+0000 7f2a7d8ce700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a780b90a0 con 0x7f2a780b6560 2024-04-06T13:28:38.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.352+0000 7f2a7d8ce700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a780b6a40 con 0x7f2a7815d470 2024-04-06T13:28:38.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.352+0000 7f2a7d8ce700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a780b6d20 con 0x7f2a780ce810 2024-04-06T13:28:38.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.352+0000 7f2a767fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2a780b6560 0x7f2a780ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40984/0 (socket says 172.21.15.189:40984) 2024-04-06T13:28:38.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.352+0000 7f2a767fc700 1 -- 172.21.15.189:0/3740340253 learned_addr learned my addr 172.21.15.189:0/3740340253 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:38.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.352+0000 7f2a777fe700 1 --2- 172.21.15.189:0/3740340253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a780ce810 0x7f2a78168230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:38.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.352+0000 7f2a75ffb700 1 -- 172.21.15.189:0/3740340253 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124407976 0 0) 0x7f2a780b90a0 con 0x7f2a780b6560 2024-04-06T13:28:38.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.352+0000 7f2a75ffb700 1 -- 172.21.15.189:0/3740340253 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a600035b0 con 0x7f2a780b6560 2024-04-06T13:28:38.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.353+0000 7f2a75ffb700 1 -- 172.21.15.189:0/3740340253 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970242310 0 0) 0x7f2a780b6a40 con 0x7f2a7815d470 2024-04-06T13:28:38.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.353+0000 7f2a75ffb700 1 -- 172.21.15.189:0/3740340253 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a780b90a0 con 0x7f2a7815d470 2024-04-06T13:28:38.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.353+0000 7f2a777fe700 1 -- 172.21.15.189:0/3740340253 >> v1:172.21.15.192:6789/0 conn(0x7f2a7815d470 legacy=0x7f2a780d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:38.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.353+0000 7f2a75ffb700 1 -- 172.21.15.189:0/3740340253 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 123010714 0 0) 0x7f2a600035b0 con 0x7f2a780b6560 2024-04-06T13:28:38.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.353+0000 7f2a777fe700 1 -- 172.21.15.189:0/3740340253 >> v1:172.21.15.189:6789/0 conn(0x7f2a780b6560 legacy=0x7f2a780ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:38.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.353+0000 7f2a777fe700 1 -- 172.21.15.189:0/3740340253 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a780b7000 con 0x7f2a780ce810 2024-04-06T13:28:38.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.353+0000 7f2a777fe700 1 --2- 172.21.15.189:0/3740340253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a780ce810 0x7f2a78168230 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2a70009dc0 tx=0x7f2a7000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=773017000fad6880 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:38.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.353+0000 7f2a75ffb700 1 -- 172.21.15.189:0/3740340253 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7000e070 con 0x7f2a780ce810 2024-04-06T13:28:38.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.353+0000 7f2a75ffb700 1 -- 172.21.15.189:0/3740340253 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a70003a20 con 0x7f2a780ce810 2024-04-06T13:28:38.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.353+0000 7f2a75ffb700 1 -- 172.21.15.189:0/3740340253 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a68001670 con 0x7f2a780ce810 2024-04-06T13:28:38.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.353+0000 7f2a7d8ce700 1 -- 172.21.15.189:0/3740340253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a780ce810 msgr2=0x7f2a78168230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:38.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.353+0000 7f2a7d8ce700 1 --2- 172.21.15.189:0/3740340253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a780ce810 0x7f2a78168230 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2a70009dc0 tx=0x7f2a7000b330 comp rx=0 tx=0).stop 2024-04-06T13:28:38.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.354+0000 7f2a7d8ce700 1 -- 172.21.15.189:0/3740340253 shutdown_connections 2024-04-06T13:28:38.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.354+0000 7f2a7d8ce700 1 --2- 172.21.15.189:0/3740340253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a780ce810 0x7f2a78168230 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:38.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.354+0000 7f2a7d8ce700 1 -- 172.21.15.189:0/3740340253 >> 172.21.15.189:0/3740340253 conn(0x7f2a7806bbc0 msgr2=0x7f2a7806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:38.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.354+0000 7f2a7d8ce700 1 -- 172.21.15.189:0/3740340253 shutdown_connections 2024-04-06T13:28:38.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.354+0000 7f2a7d8ce700 1 -- 172.21.15.189:0/3740340253 wait complete. 2024-04-06T13:28:38.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.355+0000 7f2a7d8ce700 1 Processor -- start 2024-04-06T13:28:38.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.355+0000 7f2a7d8ce700 1 -- start start 2024-04-06T13:28:38.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a7d8ce700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a780b6560 0x7f2a781529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:38.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a7d8ce700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a780ce810 0x7f2a780b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:38.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a7d8ce700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a7815d470 0x7f2a780b84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:38.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a7d8ce700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a78152ee0 con 0x7f2a780b6560 2024-04-06T13:28:38.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a7d8ce700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a780d2020 con 0x7f2a7815d470 2024-04-06T13:28:38.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a7d8ce700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a780d0f80 con 0x7f2a780ce810 2024-04-06T13:28:38.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a76ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a780b6560 0x7f2a781529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:38.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a76ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a780b6560 0x7f2a781529a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35594/0 (socket says 172.21.15.189:35594) 2024-04-06T13:28:38.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a76ffd700 1 -- 172.21.15.189:0/3169484978 learned_addr learned my addr 172.21.15.189:0/3169484978 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:38.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a767fc700 1 --2- 172.21.15.189:0/3169484978 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a780ce810 0x7f2a780b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:38.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a777fe700 1 --2- 172.21.15.189:0/3169484978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a7815d470 0x7f2a780b84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:38.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a76ffd700 1 -- 172.21.15.189:0/3169484978 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a780ce810 msgr2=0x7f2a780b7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:38.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a76ffd700 1 --2- 172.21.15.189:0/3169484978 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a780ce810 0x7f2a780b7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:38.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a76ffd700 1 -- 172.21.15.189:0/3169484978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a7815d470 msgr2=0x7f2a780b84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:38.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a76ffd700 1 --2- 172.21.15.189:0/3169484978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a7815d470 0x7f2a780b84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:38.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a76ffd700 1 -- 172.21.15.189:0/3169484978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a70009a70 con 0x7f2a780b6560 2024-04-06T13:28:38.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a76ffd700 1 --2- 172.21.15.189:0/3169484978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a780b6560 0x7f2a781529a0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00c500 tx=0x7f2a6c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:38.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a5ffff700 1 -- 172.21.15.189:0/3169484978 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c0052d0 con 0x7f2a780b6560 2024-04-06T13:28:38.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a5ffff700 1 -- 172.21.15.189:0/3169484978 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6c005450 con 0x7f2a780b6560 2024-04-06T13:28:38.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a7d8ce700 1 -- 172.21.15.189:0/3169484978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a780d1260 con 0x7f2a780b6560 2024-04-06T13:28:38.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a5ffff700 1 -- 172.21.15.189:0/3169484978 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c0038c0 con 0x7f2a780b6560 2024-04-06T13:28:38.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.356+0000 7f2a7d8ce700 1 -- 172.21.15.189:0/3169484978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a780d1820 con 0x7f2a780b6560 2024-04-06T13:28:38.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.357+0000 7f2a7d8ce700 1 -- 172.21.15.189:0/3169484978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a78067bf0 con 0x7f2a780b6560 2024-04-06T13:28:38.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.358+0000 7f2a5ffff700 1 -- 172.21.15.189:0/3169484978 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f2a6c003a20 con 0x7f2a780b6560 2024-04-06T13:28:38.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.358+0000 7f2a5ffff700 1 --2- 172.21.15.189:0/3169484978 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2a6406adb0 0x7f2a6406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:38.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.358+0000 7f2a5ffff700 1 -- 172.21.15.189:0/3169484978 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f2a6c0885d0 con 0x7f2a780b6560 2024-04-06T13:28:38.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.358+0000 7f2a767fc700 1 --2- 172.21.15.189:0/3169484978 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2a6406adb0 0x7f2a6406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:38.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.359+0000 7f2a767fc700 1 --2- 172.21.15.189:0/3169484978 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2a6406adb0 0x7f2a6406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2a68000fd0 tx=0x7f2a68001e00 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:38.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.363+0000 7f2a5ffff700 1 -- 172.21.15.189:0/3169484978 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2a6c053910 con 0x7f2a780b6560 2024-04-06T13:28:38.558 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:38.558+0000 7f2a7d8ce700 1 -- 172.21.15.189:0/3169484978 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2a780670c0 con 0x7f2a780b6560 2024-04-06T13:28:39.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.398+0000 7f2a5ffff700 1 -- 172.21.15.189:0/3169484978 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v175) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2a6c056f30 con 0x7f2a780b6560 2024-04-06T13:28:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.403+0000 7f2a7d8ce700 1 -- 172.21.15.189:0/3169484978 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2a6406adb0 msgr2=0x7f2a6406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.403+0000 7f2a7d8ce700 1 --2- 172.21.15.189:0/3169484978 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2a6406adb0 0x7f2a6406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2a68000fd0 tx=0x7f2a68001e00 comp rx=0 tx=0).stop 2024-04-06T13:28:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.403+0000 7f2a7d8ce700 1 -- 172.21.15.189:0/3169484978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a780b6560 msgr2=0x7f2a781529a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.403+0000 7f2a7d8ce700 1 --2- 172.21.15.189:0/3169484978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a780b6560 0x7f2a781529a0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00c500 tx=0x7f2a6c00c810 comp rx=0 tx=0).stop 2024-04-06T13:28:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.403+0000 7f2a7d8ce700 1 -- 172.21.15.189:0/3169484978 shutdown_connections 2024-04-06T13:28:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.403+0000 7f2a7d8ce700 1 --2- 172.21.15.189:0/3169484978 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2a6406adb0 0x7f2a6406d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.403+0000 7f2a7d8ce700 1 --2- 172.21.15.189:0/3169484978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a7815d470 0x7f2a780b84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.403+0000 7f2a7d8ce700 1 --2- 172.21.15.189:0/3169484978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a780b6560 0x7f2a781529a0 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.403+0000 7f2a7d8ce700 1 --2- 172.21.15.189:0/3169484978 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a780ce810 0x7f2a780b7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.403+0000 7f2a7d8ce700 1 -- 172.21.15.189:0/3169484978 >> 172.21.15.189:0/3169484978 conn(0x7f2a7806bbc0 msgr2=0x7f2a780b3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.403+0000 7f2a7d8ce700 1 -- 172.21.15.189:0/3169484978 shutdown_connections 2024-04-06T13:28:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.403+0000 7f2a7d8ce700 1 -- 172.21.15.189:0/3169484978 wait complete. 2024-04-06T13:28:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' created 2024-04-06T13:28:39.419 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-06T13:28:39.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.554+0000 7f13135ee700 1 Processor -- start 2024-04-06T13:28:39.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.554+0000 7f13135ee700 1 -- start start 2024-04-06T13:28:39.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.555+0000 7f13135ee700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f130c06ae40 0x7f130c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:39.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.555+0000 7f13135ee700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f130c15ddf0 0x7f130c15e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:39.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.555+0000 7f13135ee700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f130c0b3990 con 0x7f130c15d7b0 2024-04-06T13:28:39.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.555+0000 7f13135ee700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f130c0b3060 con 0x7f130c15ddf0 2024-04-06T13:28:39.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.555+0000 7f13135ee700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130c0b3340 con 0x7f130c06ae40 2024-04-06T13:28:39.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.555+0000 7f1311b8b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f130c15ddf0 0x7f130c15e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:39.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.555+0000 7f1311b8b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f130c15ddf0 0x7f130c15e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58934/0 (socket says 172.21.15.189:58934) 2024-04-06T13:28:39.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.555+0000 7f1311b8b700 1 -- 172.21.15.189:0/335879195 learned_addr learned my addr 172.21.15.189:0/335879195 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:39.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.555+0000 7f1310b89700 1 --2- 172.21.15.189:0/335879195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f130c06ae40 0x7f130c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:39.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.555+0000 7f1311b8b700 1 -- 172.21.15.189:0/335879195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f130c06ae40 msgr2=0x7f130c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:39.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.555+0000 7f1311b8b700 1 --2- 172.21.15.189:0/335879195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f130c06ae40 0x7f130c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:39.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.555+0000 7f1311b8b700 1 -- 172.21.15.189:0/335879195 >> v1:172.21.15.192:6789/0 conn(0x7f130c15d7b0 legacy=0x7f130c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:39.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.555+0000 7f1311b8b700 1 -- 172.21.15.189:0/335879195 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130c0b3620 con 0x7f130c15ddf0 2024-04-06T13:28:39.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.555+0000 7f1310b89700 1 --2- 172.21.15.189:0/335879195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f130c06ae40 0x7f130c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:28:39.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.556+0000 7f1311b8b700 1 --2- 172.21.15.189:0/335879195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f130c15ddf0 0x7f130c15e2a0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f1304009e50 tx=0x7f130400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c210b79da6ce0394 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:39.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.556+0000 7f1303fff700 1 -- 172.21.15.189:0/335879195 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130400e070 con 0x7f130c15ddf0 2024-04-06T13:28:39.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.556+0000 7f1303fff700 1 -- 172.21.15.189:0/335879195 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1304003a20 con 0x7f130c15ddf0 2024-04-06T13:28:39.558 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.556+0000 7f1303fff700 1 -- 172.21.15.189:0/335879195 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1304003dd0 con 0x7f130c15ddf0 2024-04-06T13:28:39.558 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.556+0000 7f13135ee700 1 -- 172.21.15.189:0/335879195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f130c15ddf0 msgr2=0x7f130c15e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:39.558 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.556+0000 7f13135ee700 1 --2- 172.21.15.189:0/335879195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f130c15ddf0 0x7f130c15e2a0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f1304009e50 tx=0x7f130400b330 comp rx=0 tx=0).stop 2024-04-06T13:28:39.558 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.556+0000 7f13135ee700 1 -- 172.21.15.189:0/335879195 shutdown_connections 2024-04-06T13:28:39.558 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.556+0000 7f13135ee700 1 --2- 172.21.15.189:0/335879195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f130c15ddf0 0x7f130c15e2a0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:39.558 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.556+0000 7f13135ee700 1 --2- 172.21.15.189:0/335879195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f130c06ae40 0x7f130c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:39.558 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.556+0000 7f13135ee700 1 -- 172.21.15.189:0/335879195 >> 172.21.15.189:0/335879195 conn(0x7f130c0d00b0 msgr2=0x7f130c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:39.558 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.556+0000 7f13135ee700 1 -- 172.21.15.189:0/335879195 shutdown_connections 2024-04-06T13:28:39.558 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.556+0000 7f13135ee700 1 -- 172.21.15.189:0/335879195 wait complete. 2024-04-06T13:28:39.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.558+0000 7f13135ee700 1 Processor -- start 2024-04-06T13:28:39.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.558+0000 7f13135ee700 1 -- start start 2024-04-06T13:28:39.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.558+0000 7f13135ee700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f130c06ae40 0x7f130c15c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:39.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.558+0000 7f13135ee700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f130c15d7b0 0x7f130c0b1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:39.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.558+0000 7f13135ee700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f130c15ddf0 0x7f130c1f2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:39.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.558+0000 7f13135ee700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c15cde0 con 0x7f130c15d7b0 2024-04-06T13:28:39.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.558+0000 7f13135ee700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c1577e0 con 0x7f130c06ae40 2024-04-06T13:28:39.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.558+0000 7f13135ee700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130c0af010 con 0x7f130c15ddf0 2024-04-06T13:28:39.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.558+0000 7f1310b89700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f130c15d7b0 0x7f130c0b1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:39.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.558+0000 7f1310b89700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f130c15d7b0 0x7f130c0b1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58944/0 (socket says 172.21.15.189:58944) 2024-04-06T13:28:39.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.558+0000 7f1310b89700 1 -- 172.21.15.189:0/2500918484 learned_addr learned my addr 172.21.15.189:0/2500918484 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:39.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.559+0000 7f131138a700 1 --2- 172.21.15.189:0/2500918484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f130c06ae40 0x7f130c15c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:39.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.559+0000 7f1311b8b700 1 --2- 172.21.15.189:0/2500918484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f130c15ddf0 0x7f130c1f2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:39.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.559+0000 7f1310b89700 1 -- 172.21.15.189:0/2500918484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f130c15ddf0 msgr2=0x7f130c1f2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:39.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.559+0000 7f1310b89700 1 --2- 172.21.15.189:0/2500918484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f130c15ddf0 0x7f130c1f2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:39.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.559+0000 7f1310b89700 1 -- 172.21.15.189:0/2500918484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f130c06ae40 msgr2=0x7f130c15c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:39.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.559+0000 7f1310b89700 1 --2- 172.21.15.189:0/2500918484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f130c06ae40 0x7f130c15c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:39.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.559+0000 7f1310b89700 1 -- 172.21.15.189:0/2500918484 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1304009b30 con 0x7f130c15d7b0 2024-04-06T13:28:39.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.559+0000 7f131138a700 1 --2- 172.21.15.189:0/2500918484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f130c06ae40 0x7f130c15c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:28:39.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.559+0000 7f1310b89700 1 --2- 172.21.15.189:0/2500918484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f130c15d7b0 0x7f130c0b1490 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f130800c740 tx=0x7f130800cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:39.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.559+0000 7f1301ffb700 1 -- 172.21.15.189:0/2500918484 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1308004220 con 0x7f130c15d7b0 2024-04-06T13:28:39.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.559+0000 7f1301ffb700 1 -- 172.21.15.189:0/2500918484 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1308004380 con 0x7f130c15d7b0 2024-04-06T13:28:39.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.559+0000 7f1301ffb700 1 -- 172.21.15.189:0/2500918484 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13080036a0 con 0x7f130c15d7b0 2024-04-06T13:28:39.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.559+0000 7f13135ee700 1 -- 172.21.15.189:0/2500918484 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130c0af2f0 con 0x7f130c15d7b0 2024-04-06T13:28:39.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.559+0000 7f13135ee700 1 -- 172.21.15.189:0/2500918484 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130c1f2830 con 0x7f130c15d7b0 2024-04-06T13:28:39.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.560+0000 7f13135ee700 1 -- 172.21.15.189:0/2500918484 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f130c1f2a50 con 0x7f130c15d7b0 2024-04-06T13:28:39.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.561+0000 7f1301ffb700 1 -- 172.21.15.189:0/2500918484 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f1308003800 con 0x7f130c15d7b0 2024-04-06T13:28:39.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.561+0000 7f1301ffb700 1 --2- 172.21.15.189:0/2500918484 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f12f006ade0 0x7f12f006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:39.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.561+0000 7f1301ffb700 1 -- 172.21.15.189:0/2500918484 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5591+0+0 (secure 0 0 0) 0x7f1308089830 con 0x7f130c15d7b0 2024-04-06T13:28:39.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.561+0000 7f131138a700 1 --2- 172.21.15.189:0/2500918484 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f12f006ade0 0x7f12f006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:39.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.561+0000 7f131138a700 1 --2- 172.21.15.189:0/2500918484 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f12f006ade0 0x7f12f006d2a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f130c0ae530 tx=0x7f12fc009040 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:39.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.566+0000 7f1301ffb700 1 -- 172.21.15.189:0/2500918484 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1308054870 con 0x7f130c15d7b0 2024-04-06T13:28:39.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:39.759+0000 7f13135ee700 1 -- 172.21.15.189:0/2500918484 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f130c067bb0 con 0x7f130c15d7b0 2024-04-06T13:28:40.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.400+0000 7f1301ffb700 1 -- 172.21.15.189:0/2500918484 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v176) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1308057e90 con 0x7f130c15d7b0 2024-04-06T13:28:40.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.405+0000 7f13135ee700 1 -- 172.21.15.189:0/2500918484 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f12f006ade0 msgr2=0x7f12f006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:40.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.405+0000 7f13135ee700 1 --2- 172.21.15.189:0/2500918484 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f12f006ade0 0x7f12f006d2a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f130c0ae530 tx=0x7f12fc009040 comp rx=0 tx=0).stop 2024-04-06T13:28:40.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.405+0000 7f13135ee700 1 -- 172.21.15.189:0/2500918484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f130c15d7b0 msgr2=0x7f130c0b1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:40.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.405+0000 7f13135ee700 1 --2- 172.21.15.189:0/2500918484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f130c15d7b0 0x7f130c0b1490 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f130800c740 tx=0x7f130800cb00 comp rx=0 tx=0).stop 2024-04-06T13:28:40.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.407+0000 7f13135ee700 1 -- 172.21.15.189:0/2500918484 shutdown_connections 2024-04-06T13:28:40.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.407+0000 7f13135ee700 1 --2- 172.21.15.189:0/2500918484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f130c15ddf0 0x7f130c1f2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:40.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.407+0000 7f13135ee700 1 --2- 172.21.15.189:0/2500918484 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f12f006ade0 0x7f12f006d2a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:40.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.407+0000 7f13135ee700 1 --2- 172.21.15.189:0/2500918484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f130c06ae40 0x7f130c15c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:40.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.407+0000 7f13135ee700 1 --2- 172.21.15.189:0/2500918484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f130c15d7b0 0x7f130c0b1490 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:40.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.407+0000 7f13135ee700 1 -- 172.21.15.189:0/2500918484 >> 172.21.15.189:0/2500918484 conn(0x7f130c0d00b0 msgr2=0x7f130c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:40.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.407+0000 7f13135ee700 1 -- 172.21.15.189:0/2500918484 shutdown_connections 2024-04-06T13:28:40.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.408+0000 7f13135ee700 1 -- 172.21.15.189:0/2500918484 wait complete. 2024-04-06T13:28:40.414 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' created 2024-04-06T13:28:40.432 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-06T13:28:40.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.560+0000 7f670db4f700 1 Processor -- start 2024-04-06T13:28:40.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.560+0000 7f670db4f700 1 -- start start 2024-04-06T13:28:40.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.561+0000 7f670db4f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f670806ae40 0x7f67080b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:40.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.561+0000 7f670db4f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f670815ddf0 0x7f670815e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:40.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.561+0000 7f670db4f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67080b3990 con 0x7f670815d7b0 2024-04-06T13:28:40.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.561+0000 7f670db4f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67080b3060 con 0x7f670806ae40 2024-04-06T13:28:40.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.561+0000 7f670db4f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67080b3340 con 0x7f670815ddf0 2024-04-06T13:28:40.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.561+0000 7f67077fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f670815d7b0 0x7f670806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55842/0 (socket says 172.21.15.189:55842) 2024-04-06T13:28:40.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.561+0000 7f67077fe700 1 -- 172.21.15.189:0/4103880078 learned_addr learned my addr 172.21.15.189:0/4103880078 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:40.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f6707fff700 1 --2- 172.21.15.189:0/4103880078 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f670815ddf0 0x7f670815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:40.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f6706ffd700 1 --2- 172.21.15.189:0/4103880078 >> v2:172.21.15.192:3300/0 conn(0x7f670806ae40 0x7f67080b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:40.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f67067fc700 1 -- 172.21.15.189:0/4103880078 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254801747 0 0) 0x7f67080b3990 con 0x7f670815d7b0 2024-04-06T13:28:40.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f67067fc700 1 -- 172.21.15.189:0/4103880078 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66f40035b0 con 0x7f670815d7b0 2024-04-06T13:28:40.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f67067fc700 1 -- 172.21.15.189:0/4103880078 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 694922011 0 0) 0x7f66f40035b0 con 0x7f670815d7b0 2024-04-06T13:28:40.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f67067fc700 1 -- 172.21.15.189:0/4103880078 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f670815ddf0 msgr2=0x7f670815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:40.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f67067fc700 1 --2- 172.21.15.189:0/4103880078 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f670815ddf0 0x7f670815e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:40.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f67067fc700 1 -- 172.21.15.189:0/4103880078 >> v2:172.21.15.192:3300/0 conn(0x7f670806ae40 msgr2=0x7f67080b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:40.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f67067fc700 1 --2- 172.21.15.189:0/4103880078 >> v2:172.21.15.192:3300/0 conn(0x7f670806ae40 0x7f67080b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:40.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f67067fc700 1 -- 172.21.15.189:0/4103880078 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67080b3620 con 0x7f670815d7b0 2024-04-06T13:28:40.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f67067fc700 1 -- 172.21.15.189:0/4103880078 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f66f0001670 con 0x7f670815d7b0 2024-04-06T13:28:40.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f67067fc700 1 -- 172.21.15.189:0/4103880078 >> v1:172.21.15.189:6789/0 conn(0x7f670815d7b0 legacy=0x7f670806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:40.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f67067fc700 1 --2- 172.21.15.189:0/4103880078 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f670815ddf0 0x7f66f4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:40.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f67067fc700 1 --2- 172.21.15.189:0/4103880078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66f4003f80 0x7f66f4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:40.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f67067fc700 1 --2- 172.21.15.189:0/4103880078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66f4004d90 0x7f66f4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:40.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f67067fc700 1 -- 172.21.15.189:0/4103880078 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67080b3060 con 0x7f66f4004d90 2024-04-06T13:28:40.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f67067fc700 1 -- 172.21.15.189:0/4103880078 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67080b3340 con 0x7f66f4003f80 2024-04-06T13:28:40.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f67067fc700 1 -- 172.21.15.189:0/4103880078 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f400ba60 con 0x7f670815ddf0 2024-04-06T13:28:40.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f6707fff700 1 --2- 172.21.15.189:0/4103880078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66f4004d90 0x7f66f4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:40.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f67077fe700 1 --2- 172.21.15.189:0/4103880078 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f670815ddf0 0x7f66f4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:40.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f6706ffd700 1 --2- 172.21.15.189:0/4103880078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66f4003f80 0x7f66f4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:40.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f6707fff700 1 -- 172.21.15.189:0/4103880078 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f670815ddf0 msgr2=0x7f66f4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:40.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f6707fff700 1 --2- 172.21.15.189:0/4103880078 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f670815ddf0 0x7f66f4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:40.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f6707fff700 1 -- 172.21.15.189:0/4103880078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66f4003f80 msgr2=0x7f66f4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:40.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f6707fff700 1 --2- 172.21.15.189:0/4103880078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66f4003f80 0x7f66f4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:40.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.562+0000 7f6707fff700 1 -- 172.21.15.189:0/4103880078 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f40035b0 con 0x7f66f4004d90 2024-04-06T13:28:40.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.563+0000 7f6707fff700 1 --2- 172.21.15.189:0/4103880078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66f4004d90 0x7f66f4005210 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f670000e170 tx=0x7f66f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a5e28168dbbd92c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:40.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.563+0000 7f67067fc700 1 -- 172.21.15.189:0/4103880078 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670000ec90 con 0x7f66f4004d90 2024-04-06T13:28:40.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.563+0000 7f67067fc700 1 -- 172.21.15.189:0/4103880078 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670000f040 con 0x7f66f4004d90 2024-04-06T13:28:40.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.563+0000 7f67067fc700 1 -- 172.21.15.189:0/4103880078 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6700014480 con 0x7f66f4004d90 2024-04-06T13:28:40.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.563+0000 7f670db4f700 1 -- 172.21.15.189:0/4103880078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66f4004d90 msgr2=0x7f66f4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:40.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.563+0000 7f670db4f700 1 --2- 172.21.15.189:0/4103880078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66f4004d90 0x7f66f4005210 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f670000e170 tx=0x7f66f4005880 comp rx=0 tx=0).stop 2024-04-06T13:28:40.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.563+0000 7f6707fff700 1 -- 172.21.15.189:0/4103880078 reap_dead start 2024-04-06T13:28:40.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.563+0000 7f670db4f700 1 -- 172.21.15.189:0/4103880078 shutdown_connections 2024-04-06T13:28:40.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.563+0000 7f670db4f700 1 -- 172.21.15.189:0/4103880078 >> 172.21.15.189:0/4103880078 conn(0x7f67080d00b0 msgr2=0x7f67080d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:40.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.563+0000 7f670db4f700 1 -- 172.21.15.189:0/4103880078 shutdown_connections 2024-04-06T13:28:40.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.563+0000 7f670db4f700 1 -- 172.21.15.189:0/4103880078 wait complete. 2024-04-06T13:28:40.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.564+0000 7f670db4f700 1 Processor -- start 2024-04-06T13:28:40.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.565+0000 7f670db4f700 1 -- start start 2024-04-06T13:28:40.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.565+0000 7f670db4f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66f4004d90 0x7f6708147b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:40.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.565+0000 7f670db4f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67080cd300 0x7f67080b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:40.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.565+0000 7f670db4f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67080b11c0 0x7f670815c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:40.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.565+0000 7f670db4f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67080b1640 con 0x7f66f4004d90 2024-04-06T13:28:40.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.565+0000 7f670db4f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6708148090 con 0x7f67080cd300 2024-04-06T13:28:40.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.565+0000 7f670db4f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67080cc260 con 0x7f67080b11c0 2024-04-06T13:28:40.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.565+0000 7f67077fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66f4004d90 0x7f6708147b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:40.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.565+0000 7f67077fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66f4004d90 0x7f6708147b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58956/0 (socket says 172.21.15.189:58956) 2024-04-06T13:28:40.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.565+0000 7f67077fe700 1 -- 172.21.15.189:0/3311689326 learned_addr learned my addr 172.21.15.189:0/3311689326 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:40.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.566+0000 7f6706ffd700 1 --2- 172.21.15.189:0/3311689326 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67080cd300 0x7f67080b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:40.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.566+0000 7f6707fff700 1 --2- 172.21.15.189:0/3311689326 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67080b11c0 0x7f670815c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:40.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.566+0000 7f67077fe700 1 -- 172.21.15.189:0/3311689326 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67080b11c0 msgr2=0x7f670815c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:40.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.566+0000 7f67077fe700 1 --2- 172.21.15.189:0/3311689326 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67080b11c0 0x7f670815c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:40.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.566+0000 7f67077fe700 1 -- 172.21.15.189:0/3311689326 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67080cd300 msgr2=0x7f67080b0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:40.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.566+0000 7f67077fe700 1 --2- 172.21.15.189:0/3311689326 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67080cd300 0x7f67080b0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:40.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.566+0000 7f67077fe700 1 -- 172.21.15.189:0/3311689326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67080cc540 con 0x7f66f4004d90 2024-04-06T13:28:40.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.566+0000 7f6707fff700 1 --2- 172.21.15.189:0/3311689326 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67080b11c0 0x7f670815c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:28:40.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.566+0000 7f6706ffd700 1 --2- 172.21.15.189:0/3311689326 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67080cd300 0x7f67080b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:28:40.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.566+0000 7f67077fe700 1 --2- 172.21.15.189:0/3311689326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66f4004d90 0x7f6708147b50 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f66f0007310 tx=0x7f66f0007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:40.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.566+0000 7f66fffff700 1 -- 172.21.15.189:0/3311689326 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f000f8d0 con 0x7f66f4004d90 2024-04-06T13:28:40.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.566+0000 7f66fffff700 1 -- 172.21.15.189:0/3311689326 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f000fa30 con 0x7f66f4004d90 2024-04-06T13:28:40.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.566+0000 7f670db4f700 1 -- 172.21.15.189:0/3311689326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67080cc820 con 0x7f66f4004d90 2024-04-06T13:28:40.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.566+0000 7f66fffff700 1 -- 172.21.15.189:0/3311689326 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f000fde0 con 0x7f66f4004d90 2024-04-06T13:28:40.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.566+0000 7f670db4f700 1 -- 172.21.15.189:0/3311689326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67080ccde0 con 0x7f66f4004d90 2024-04-06T13:28:40.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.567+0000 7f670db4f700 1 -- 172.21.15.189:0/3311689326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6708067bb0 con 0x7f66f4004d90 2024-04-06T13:28:40.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.568+0000 7f66fffff700 1 -- 172.21.15.189:0/3311689326 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f66f001d460 con 0x7f66f4004d90 2024-04-06T13:28:40.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.568+0000 7f66fffff700 1 --2- 172.21.15.189:0/3311689326 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66e806adb0 0x7f66e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:40.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.568+0000 7f66fffff700 1 -- 172.21.15.189:0/3311689326 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5973+0+0 (secure 0 0 0) 0x7f66f0089660 con 0x7f66f4004d90 2024-04-06T13:28:40.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.568+0000 7f6706ffd700 1 --2- 172.21.15.189:0/3311689326 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66e806adb0 0x7f66e806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:40.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.568+0000 7f6706ffd700 1 --2- 172.21.15.189:0/3311689326 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66e806adb0 0x7f66e806d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f67080cd990 tx=0x7f66f8008040 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:40.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.573+0000 7f66fffff700 1 -- 172.21.15.189:0/3311689326 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f66f00546b0 con 0x7f66f4004d90 2024-04-06T13:28:40.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.810+0000 7f670db4f700 1 -- 172.21.15.189:0/3311689326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6708074990 con 0x7f66f4004d90 2024-04-06T13:28:40.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.831+0000 7f66fffff700 1 -- 172.21.15.189:0/3311689326 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 21 and data pool 22 v74) v1 ==== 174+0+0 (secure 0 0 0) 0x7f66f0057cd0 con 0x7f66f4004d90 2024-04-06T13:28:40.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.837+0000 7f670db4f700 1 -- 172.21.15.189:0/3311689326 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66e806adb0 msgr2=0x7f66e806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:40.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.837+0000 7f670db4f700 1 --2- 172.21.15.189:0/3311689326 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66e806adb0 0x7f66e806d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f67080cd990 tx=0x7f66f8008040 comp rx=0 tx=0).stop 2024-04-06T13:28:40.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.837+0000 7f670db4f700 1 -- 172.21.15.189:0/3311689326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66f4004d90 msgr2=0x7f6708147b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:40.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.837+0000 7f670db4f700 1 --2- 172.21.15.189:0/3311689326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66f4004d90 0x7f6708147b50 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f66f0007310 tx=0x7f66f0007620 comp rx=0 tx=0).stop 2024-04-06T13:28:40.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.838+0000 7f670db4f700 1 -- 172.21.15.189:0/3311689326 shutdown_connections 2024-04-06T13:28:40.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.838+0000 7f670db4f700 1 --2- 172.21.15.189:0/3311689326 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67080b11c0 0x7f670815c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:40.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.838+0000 7f670db4f700 1 --2- 172.21.15.189:0/3311689326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66f4004d90 0x7f6708147b50 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:40.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.838+0000 7f670db4f700 1 --2- 172.21.15.189:0/3311689326 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f66e806adb0 0x7f66e806d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:40.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.838+0000 7f670db4f700 1 --2- 172.21.15.189:0/3311689326 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67080cd300 0x7f67080b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:40.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.838+0000 7f670db4f700 1 -- 172.21.15.189:0/3311689326 >> 172.21.15.189:0/3311689326 conn(0x7f67080d00b0 msgr2=0x7f670806eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:40.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.838+0000 7f670db4f700 1 -- 172.21.15.189:0/3311689326 shutdown_connections 2024-04-06T13:28:40.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.838+0000 7f670db4f700 1 -- 172.21.15.189:0/3311689326 wait complete. 2024-04-06T13:28:40.840 INFO:teuthology.orchestra.run.smithi189.stderr:new fs with metadata pool 21 and data pool 22 2024-04-06T13:28:40.854 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:28:40.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.975+0000 7f64d04b6700 1 Processor -- start 2024-04-06T13:28:40.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.975+0000 7f64d04b6700 1 -- start start 2024-04-06T13:28:40.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.976+0000 7f64d04b6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64c80cb870 0x7f64c815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:40.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.976+0000 7f64d04b6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c80cbd80 con 0x7f64c80d11b0 2024-04-06T13:28:40.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.976+0000 7f64d04b6700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c80cef20 con 0x7f64c80cfa00 2024-04-06T13:28:40.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.976+0000 7f64d04b6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c80ccdc0 con 0x7f64c80cb870 2024-04-06T13:28:40.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.976+0000 7f64ce252700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f64c80d11b0 0x7f64c80d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55854/0 (socket says 172.21.15.189:55854) 2024-04-06T13:28:40.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.976+0000 7f64ce252700 1 -- 172.21.15.189:0/839760939 learned_addr learned my addr 172.21.15.189:0/839760939 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:40.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cda51700 1 --2- 172.21.15.189:0/839760939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64c80cb870 0x7f64c815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:40.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cd250700 1 -- 172.21.15.189:0/839760939 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971523062 0 0) 0x7f64c80cbd80 con 0x7f64c80d11b0 2024-04-06T13:28:40.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cd250700 1 -- 172.21.15.189:0/839760939 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b40035b0 con 0x7f64c80d11b0 2024-04-06T13:28:40.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cd250700 1 -- 172.21.15.189:0/839760939 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418845437 0 0) 0x7f64c80cef20 con 0x7f64c80cfa00 2024-04-06T13:28:40.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cd250700 1 -- 172.21.15.189:0/839760939 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c80cbd80 con 0x7f64c80cfa00 2024-04-06T13:28:40.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cd250700 1 -- 172.21.15.189:0/839760939 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1801306312 0 0) 0x7f64b40035b0 con 0x7f64c80d11b0 2024-04-06T13:28:40.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cd250700 1 -- 172.21.15.189:0/839760939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64c80cb870 msgr2=0x7f64c815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:40.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cd250700 1 --2- 172.21.15.189:0/839760939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64c80cb870 0x7f64c815c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:40.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cd250700 1 -- 172.21.15.189:0/839760939 >> v1:172.21.15.192:6789/0 conn(0x7f64c80cfa00 legacy=0x7f64c80cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:40.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cd250700 1 -- 172.21.15.189:0/839760939 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c80cf200 con 0x7f64c80d11b0 2024-04-06T13:28:40.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cd250700 1 -- 172.21.15.189:0/839760939 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f64bc001670 con 0x7f64c80d11b0 2024-04-06T13:28:40.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cda51700 1 --2- 172.21.15.189:0/839760939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64c80cb870 0x7f64c815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:40.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cd250700 1 -- 172.21.15.189:0/839760939 >> v1:172.21.15.189:6789/0 conn(0x7f64c80d11b0 legacy=0x7f64c80d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:40.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cd250700 1 --2- 172.21.15.189:0/839760939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64c80cb870 0x7f64b4003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:40.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cd250700 1 --2- 172.21.15.189:0/839760939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64b4004100 0x7f64b4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:40.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cd250700 1 --2- 172.21.15.189:0/839760939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64b4004d90 0x7f64b4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:40.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cd250700 1 -- 172.21.15.189:0/839760939 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b400bb20 con 0x7f64b4004100 2024-04-06T13:28:40.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cd250700 1 -- 172.21.15.189:0/839760939 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b400bc90 con 0x7f64b4004d90 2024-04-06T13:28:40.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cd250700 1 -- 172.21.15.189:0/839760939 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b400be50 con 0x7f64c80cb870 2024-04-06T13:28:40.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cda51700 1 --2- 172.21.15.189:0/839760939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64b4004100 0x7f64b4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:40.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64ce252700 1 --2- 172.21.15.189:0/839760939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64c80cb870 0x7f64b4003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:40.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cea53700 1 --2- 172.21.15.189:0/839760939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64b4004d90 0x7f64b4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:40.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cda51700 1 -- 172.21.15.189:0/839760939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64c80cb870 msgr2=0x7f64b4003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:40.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cda51700 1 --2- 172.21.15.189:0/839760939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64c80cb870 0x7f64b4003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:40.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cda51700 1 -- 172.21.15.189:0/839760939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64b4004d90 msgr2=0x7f64b4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:40.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cda51700 1 --2- 172.21.15.189:0/839760939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64b4004d90 0x7f64b4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:40.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.977+0000 7f64cda51700 1 -- 172.21.15.189:0/839760939 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c40094a0 con 0x7f64b4004100 2024-04-06T13:28:40.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.978+0000 7f64cda51700 1 --2- 172.21.15.189:0/839760939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64b4004100 0x7f64b4004850 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009ce0 tx=0x7f64b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe38829f67313782 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:40.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.978+0000 7f64cd250700 1 -- 172.21.15.189:0/839760939 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4004030 con 0x7f64b4004100 2024-04-06T13:28:40.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.978+0000 7f64cd250700 1 -- 172.21.15.189:0/839760939 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c4004190 con 0x7f64b4004100 2024-04-06T13:28:40.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.978+0000 7f64cd250700 1 -- 172.21.15.189:0/839760939 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4004540 con 0x7f64b4004100 2024-04-06T13:28:40.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.978+0000 7f64d04b6700 1 -- 172.21.15.189:0/839760939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64b4004100 msgr2=0x7f64b4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:40.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.978+0000 7f64d04b6700 1 --2- 172.21.15.189:0/839760939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64b4004100 0x7f64b4004850 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009ce0 tx=0x7f64b4005880 comp rx=0 tx=0).stop 2024-04-06T13:28:40.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.978+0000 7f64cea53700 1 -- 172.21.15.189:0/839760939 reap_dead start 2024-04-06T13:28:40.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.978+0000 7f64d04b6700 1 -- 172.21.15.189:0/839760939 shutdown_connections 2024-04-06T13:28:40.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.978+0000 7f64d04b6700 1 -- 172.21.15.189:0/839760939 >> 172.21.15.189:0/839760939 conn(0x7f64c80b2db0 msgr2=0x7f64c80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:40.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.978+0000 7f64d04b6700 1 -- 172.21.15.189:0/839760939 shutdown_connections 2024-04-06T13:28:40.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.978+0000 7f64d04b6700 1 -- 172.21.15.189:0/839760939 wait complete. 2024-04-06T13:28:40.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.980+0000 7f64d04b6700 1 Processor -- start 2024-04-06T13:28:40.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.980+0000 7f64d04b6700 1 -- start start 2024-04-06T13:28:40.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.980+0000 7f64d04b6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64b4004100 0x7f64c8156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:40.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.980+0000 7f64d04b6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64c8155f80 0x7f64c80b5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:40.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.980+0000 7f64d04b6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64c81550b0 0x7f64c8155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:40.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.980+0000 7f64d04b6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c8155a80 con 0x7f64b4004100 2024-04-06T13:28:40.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.980+0000 7f64d04b6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c8156480 con 0x7f64c8155f80 2024-04-06T13:28:40.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.980+0000 7f64d04b6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c815d4a0 con 0x7f64c81550b0 2024-04-06T13:28:40.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.980+0000 7f64ce252700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64b4004100 0x7f64c8156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:40.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.980+0000 7f64ce252700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64b4004100 0x7f64c8156910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58972/0 (socket says 172.21.15.189:58972) 2024-04-06T13:28:40.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.980+0000 7f64ce252700 1 -- 172.21.15.189:0/2269953481 learned_addr learned my addr 172.21.15.189:0/2269953481 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:40.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.981+0000 7f64cea53700 1 --2- 172.21.15.189:0/2269953481 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64c81550b0 0x7f64c8155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:40.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.981+0000 7f64cda51700 1 --2- 172.21.15.189:0/2269953481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64c8155f80 0x7f64c80b5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:40.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.981+0000 7f64ce252700 1 -- 172.21.15.189:0/2269953481 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64c81550b0 msgr2=0x7f64c8155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:40.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.981+0000 7f64ce252700 1 --2- 172.21.15.189:0/2269953481 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64c81550b0 0x7f64c8155540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:40.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.981+0000 7f64ce252700 1 -- 172.21.15.189:0/2269953481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64c8155f80 msgr2=0x7f64c80b5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:40.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.981+0000 7f64ce252700 1 --2- 172.21.15.189:0/2269953481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64c8155f80 0x7f64c80b5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:40.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.981+0000 7f64ce252700 1 -- 172.21.15.189:0/2269953481 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c815d780 con 0x7f64b4004100 2024-04-06T13:28:40.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.981+0000 7f64ce252700 1 --2- 172.21.15.189:0/2269953481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64b4004100 0x7f64c8156910 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f64bc004fd0 tx=0x7f64bc007340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:40.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.981+0000 7f64baffd700 1 -- 172.21.15.189:0/2269953481 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64bc012560 con 0x7f64b4004100 2024-04-06T13:28:40.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.981+0000 7f64baffd700 1 -- 172.21.15.189:0/2269953481 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64bc0126e0 con 0x7f64b4004100 2024-04-06T13:28:40.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.981+0000 7f64d04b6700 1 -- 172.21.15.189:0/2269953481 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c815da60 con 0x7f64b4004100 2024-04-06T13:28:40.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.981+0000 7f64baffd700 1 -- 172.21.15.189:0/2269953481 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64bc012a90 con 0x7f64b4004100 2024-04-06T13:28:40.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.981+0000 7f64d04b6700 1 -- 172.21.15.189:0/2269953481 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c80b57a0 con 0x7f64b4004100 2024-04-06T13:28:40.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.982+0000 7f64d04b6700 1 -- 172.21.15.189:0/2269953481 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c815dca0 con 0x7f64b4004100 2024-04-06T13:28:40.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.982+0000 7f64baffd700 1 -- 172.21.15.189:0/2269953481 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f64bc012bf0 con 0x7f64b4004100 2024-04-06T13:28:40.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.983+0000 7f64baffd700 1 --2- 172.21.15.189:0/2269953481 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f64ac06ade0 0x7f64ac06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:40.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.983+0000 7f64cda51700 1 --2- 172.21.15.189:0/2269953481 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f64ac06ade0 0x7f64ac06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:40.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.983+0000 7f64baffd700 1 -- 172.21.15.189:0/2269953481 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f64bc089540 con 0x7f64b4004100 2024-04-06T13:28:40.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.983+0000 7f64cda51700 1 --2- 172.21.15.189:0/2269953481 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f64ac06ade0 0x7f64ac06d2a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f64c40090b0 tx=0x7f64c4008030 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:40.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:40.987+0000 7f64baffd700 1 -- 172.21.15.189:0/2269953481 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f64c815dca0 con 0x7f64b4004100 2024-04-06T13:28:41.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.183+0000 7f64d04b6700 1 -- 172.21.15.189:0/2269953481 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64c815dca0 con 0x7f64b4004100 2024-04-06T13:28:41.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.184+0000 7f64baffd700 1 -- 172.21.15.189:0/2269953481 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v177) v1 ==== 74+0+16387 (secure 0 0 0) 0x7f64c815dca0 con 0x7f64b4004100 2024-04-06T13:28:41.185 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:41.185 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":177,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:28:40.812686+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:28:38.579643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-04-06T13:28:39.779769+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:28:41.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.186+0000 7f64d04b6700 1 -- 172.21.15.189:0/2269953481 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f64ac06ade0 msgr2=0x7f64ac06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:41.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.186+0000 7f64d04b6700 1 --2- 172.21.15.189:0/2269953481 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f64ac06ade0 0x7f64ac06d2a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f64c40090b0 tx=0x7f64c4008030 comp rx=0 tx=0).stop 2024-04-06T13:28:41.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.186+0000 7f64d04b6700 1 -- 172.21.15.189:0/2269953481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64b4004100 msgr2=0x7f64c8156910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:41.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.186+0000 7f64d04b6700 1 --2- 172.21.15.189:0/2269953481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64b4004100 0x7f64c8156910 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f64bc004fd0 tx=0x7f64bc007340 comp rx=0 tx=0).stop 2024-04-06T13:28:41.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.186+0000 7f64d04b6700 1 -- 172.21.15.189:0/2269953481 shutdown_connections 2024-04-06T13:28:41.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.186+0000 7f64d04b6700 1 --2- 172.21.15.189:0/2269953481 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64c81550b0 0x7f64c8155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:41.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.186+0000 7f64d04b6700 1 --2- 172.21.15.189:0/2269953481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64b4004100 0x7f64c8156910 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:41.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.186+0000 7f64d04b6700 1 --2- 172.21.15.189:0/2269953481 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f64ac06ade0 0x7f64ac06d2a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:41.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.186+0000 7f64d04b6700 1 --2- 172.21.15.189:0/2269953481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64c8155f80 0x7f64c80b5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:41.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.186+0000 7f64d04b6700 1 -- 172.21.15.189:0/2269953481 >> 172.21.15.189:0/2269953481 conn(0x7f64c80b2db0 msgr2=0x7f64c815adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:41.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.186+0000 7f64d04b6700 1 -- 172.21.15.189:0/2269953481 shutdown_connections 2024-04-06T13:28:41.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.186+0000 7f64d04b6700 1 -- 172.21.15.189:0/2269953481 wait complete. 2024-04-06T13:28:41.201 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:28:41.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.321+0000 7f6df5979700 1 Processor -- start 2024-04-06T13:28:41.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.321+0000 7f6df5979700 1 -- start start 2024-04-06T13:28:41.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.322+0000 7f6df5979700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6df015d590 0x7f6df00b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:41.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.322+0000 7f6df5979700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6df00b4ed0 0x7f6df00b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:41.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.322+0000 7f6df5979700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df00b5cc0 con 0x7f6df00b6580 2024-04-06T13:28:41.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.322+0000 7f6df5979700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6df00b5890 con 0x7f6df00b4ed0 2024-04-06T13:28:41.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.322+0000 7f6df5979700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df00d1500 con 0x7f6df015d590 2024-04-06T13:28:41.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.322+0000 7f6dee7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6df00b4ed0 0x7f6df00b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:41.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.322+0000 7f6dee7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6df00b4ed0 0x7f6df00b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58984/0 (socket says 172.21.15.189:58984) 2024-04-06T13:28:41.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.322+0000 7f6dee7fc700 1 -- 172.21.15.189:0/2171655005 learned_addr learned my addr 172.21.15.189:0/2171655005 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:41.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.322+0000 7f6deeffd700 1 --2- 172.21.15.189:0/2171655005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6df015d590 0x7f6df00b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:41.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.322+0000 7f6dee7fc700 1 -- 172.21.15.189:0/2171655005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6df015d590 msgr2=0x7f6df00b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:41.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.322+0000 7f6dee7fc700 1 --2- 172.21.15.189:0/2171655005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6df015d590 0x7f6df00b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:41.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.322+0000 7f6dee7fc700 1 -- 172.21.15.189:0/2171655005 >> v1:172.21.15.192:6789/0 conn(0x7f6df00b6580 legacy=0x7f6df00b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:41.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.323+0000 7f6dee7fc700 1 -- 172.21.15.189:0/2171655005 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df00d17e0 con 0x7f6df00b4ed0 2024-04-06T13:28:41.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.323+0000 7f6dee7fc700 1 --2- 172.21.15.189:0/2171655005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6df00b4ed0 0x7f6df00b5350 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009920 tx=0x7f6de0009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf11f9b08dfae369 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:41.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.323+0000 7f6dedffb700 1 -- 172.21.15.189:0/2171655005 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de000ba20 con 0x7f6df00b4ed0 2024-04-06T13:28:41.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.323+0000 7f6dedffb700 1 -- 172.21.15.189:0/2171655005 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de000bb80 con 0x7f6df00b4ed0 2024-04-06T13:28:41.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.323+0000 7f6dedffb700 1 -- 172.21.15.189:0/2171655005 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de0009e70 con 0x7f6df00b4ed0 2024-04-06T13:28:41.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.323+0000 7f6df5979700 1 -- 172.21.15.189:0/2171655005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6df00b4ed0 msgr2=0x7f6df00b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:41.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.323+0000 7f6df5979700 1 --2- 172.21.15.189:0/2171655005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6df00b4ed0 0x7f6df00b5350 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009920 tx=0x7f6de0009c30 comp rx=0 tx=0).stop 2024-04-06T13:28:41.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.323+0000 7f6df5979700 1 -- 172.21.15.189:0/2171655005 shutdown_connections 2024-04-06T13:28:41.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.323+0000 7f6df5979700 1 --2- 172.21.15.189:0/2171655005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6df015d590 0x7f6df00b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:41.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.323+0000 7f6df5979700 1 --2- 172.21.15.189:0/2171655005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6df00b4ed0 0x7f6df00b5350 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:41.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.323+0000 7f6df5979700 1 -- 172.21.15.189:0/2171655005 >> 172.21.15.189:0/2171655005 conn(0x7f6df006bbc0 msgr2=0x7f6df006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:41.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.323+0000 7f6df5979700 1 -- 172.21.15.189:0/2171655005 shutdown_connections 2024-04-06T13:28:41.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.323+0000 7f6df5979700 1 -- 172.21.15.189:0/2171655005 wait complete. 2024-04-06T13:28:41.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.325+0000 7f6df5979700 1 Processor -- start 2024-04-06T13:28:41.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.325+0000 7f6df5979700 1 -- start start 2024-04-06T13:28:41.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6df5979700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6df00b4ed0 0x7f6df0156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:41.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6df5979700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6df00b6580 0x7f6df015d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:41.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6df5979700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6df015d590 0x7f6df015a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:41.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6df5979700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df01f3f50 con 0x7f6df00b6580 2024-04-06T13:28:41.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6df5979700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df0157320 con 0x7f6df00b4ed0 2024-04-06T13:28:41.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6df5979700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df0158790 con 0x7f6df015d590 2024-04-06T13:28:41.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6dee7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6df00b6580 0x7f6df015d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:41.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6dee7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6df00b6580 0x7f6df015d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58994/0 (socket says 172.21.15.189:58994) 2024-04-06T13:28:41.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6dee7fc700 1 -- 172.21.15.189:0/988366977 learned_addr learned my addr 172.21.15.189:0/988366977 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:41.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6def7fe700 1 --2- 172.21.15.189:0/988366977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6df015d590 0x7f6df015a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:41.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6deeffd700 1 --2- 172.21.15.189:0/988366977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6df00b4ed0 0x7f6df0156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:41.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6dee7fc700 1 -- 172.21.15.189:0/988366977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6df015d590 msgr2=0x7f6df015a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:41.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6dee7fc700 1 --2- 172.21.15.189:0/988366977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6df015d590 0x7f6df015a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:41.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6dee7fc700 1 -- 172.21.15.189:0/988366977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6df00b4ed0 msgr2=0x7f6df0156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:41.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6dee7fc700 1 --2- 172.21.15.189:0/988366977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6df00b4ed0 0x7f6df0156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:41.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6dee7fc700 1 -- 172.21.15.189:0/988366977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de0009550 con 0x7f6df00b6580 2024-04-06T13:28:41.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6deeffd700 1 --2- 172.21.15.189:0/988366977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6df00b4ed0 0x7f6df0156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:28:41.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6dee7fc700 1 --2- 172.21.15.189:0/988366977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6df00b6580 0x7f6df015d970 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f6de000c010 tx=0x7f6de0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:41.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6dd7fff700 1 -- 172.21.15.189:0/988366977 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de0004300 con 0x7f6df00b6580 2024-04-06T13:28:41.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6dd7fff700 1 -- 172.21.15.189:0/988366977 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de001f070 con 0x7f6df00b6580 2024-04-06T13:28:41.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6dd7fff700 1 -- 172.21.15.189:0/988366977 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de00036a0 con 0x7f6df00b6580 2024-04-06T13:28:41.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6df5979700 1 -- 172.21.15.189:0/988366977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df0158a70 con 0x7f6df00b6580 2024-04-06T13:28:41.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.326+0000 7f6df5979700 1 -- 172.21.15.189:0/988366977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df0159030 con 0x7f6df00b6580 2024-04-06T13:28:41.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.327+0000 7f6df5979700 1 -- 172.21.15.189:0/988366977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df0074990 con 0x7f6df00b6580 2024-04-06T13:28:41.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.328+0000 7f6dd7fff700 1 -- 172.21.15.189:0/988366977 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f6de0021030 con 0x7f6df00b6580 2024-04-06T13:28:41.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.328+0000 7f6dd7fff700 1 --2- 172.21.15.189:0/988366977 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ddc06adb0 0x7f6ddc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:41.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.328+0000 7f6dd7fff700 1 -- 172.21.15.189:0/988366977 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6de008aac0 con 0x7f6df00b6580 2024-04-06T13:28:41.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.328+0000 7f6deeffd700 1 --2- 172.21.15.189:0/988366977 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ddc06adb0 0x7f6ddc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:41.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.329+0000 7f6deeffd700 1 --2- 172.21.15.189:0/988366977 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ddc06adb0 0x7f6ddc06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6df015f5a0 tx=0x7f6de4008040 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:41.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.333+0000 7f6dd7fff700 1 -- 172.21.15.189:0/988366977 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6de0055b50 con 0x7f6df00b6580 2024-04-06T13:28:41.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.529+0000 7f6df5979700 1 -- 172.21.15.189:0/988366977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6df00c9900 con 0x7f6df00b6580 2024-04-06T13:28:41.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.530+0000 7f6dd7fff700 1 -- 172.21.15.189:0/988366977 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v177) v1 ==== 74+0+16387 (secure 0 0 0) 0x7f6de0015020 con 0x7f6df00b6580 2024-04-06T13:28:41.531 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:41.531 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":177,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:28:40.812686+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:28:38.579643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-04-06T13:28:39.779769+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:28:41.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.533+0000 7f6df5979700 1 -- 172.21.15.189:0/988366977 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ddc06adb0 msgr2=0x7f6ddc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:41.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.533+0000 7f6df5979700 1 --2- 172.21.15.189:0/988366977 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ddc06adb0 0x7f6ddc06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6df015f5a0 tx=0x7f6de4008040 comp rx=0 tx=0).stop 2024-04-06T13:28:41.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.533+0000 7f6df5979700 1 -- 172.21.15.189:0/988366977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6df00b6580 msgr2=0x7f6df015d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:41.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.533+0000 7f6df5979700 1 --2- 172.21.15.189:0/988366977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6df00b6580 0x7f6df015d970 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f6de000c010 tx=0x7f6de0004750 comp rx=0 tx=0).stop 2024-04-06T13:28:41.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.533+0000 7f6df5979700 1 -- 172.21.15.189:0/988366977 shutdown_connections 2024-04-06T13:28:41.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.533+0000 7f6df5979700 1 --2- 172.21.15.189:0/988366977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6df015d590 0x7f6df015a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:41.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.533+0000 7f6df5979700 1 --2- 172.21.15.189:0/988366977 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6ddc06adb0 0x7f6ddc06d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:41.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.533+0000 7f6df5979700 1 --2- 172.21.15.189:0/988366977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6df00b4ed0 0x7f6df0156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:41.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.533+0000 7f6df5979700 1 --2- 172.21.15.189:0/988366977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6df00b6580 0x7f6df015d970 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:41.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.533+0000 7f6df5979700 1 -- 172.21.15.189:0/988366977 >> 172.21.15.189:0/988366977 conn(0x7f6df006bbc0 msgr2=0x7f6df00b3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:41.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.533+0000 7f6df5979700 1 -- 172.21.15.189:0/988366977 shutdown_connections 2024-04-06T13:28:41.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.533+0000 7f6df5979700 1 -- 172.21.15.189:0/988366977 wait complete. 2024-04-06T13:28:41.546 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-06T13:28:41.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.670+0000 7fa5dfc7e700 1 Processor -- start 2024-04-06T13:28:41.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.670+0000 7fa5dfc7e700 1 -- start start 2024-04-06T13:28:41.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.671+0000 7fa5dfc7e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5d806ae40 0x7fa5d80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:41.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.671+0000 7fa5dfc7e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d80b3990 con 0x7fa5d815ddf0 2024-04-06T13:28:41.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.671+0000 7fa5dfc7e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d80b3060 con 0x7fa5d815d7b0 2024-04-06T13:28:41.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.671+0000 7fa5dfc7e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80b3340 con 0x7fa5d806ae40 2024-04-06T13:28:41.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.671+0000 7fa5de21b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa5d815ddf0 0x7fa5d815e2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55864/0 (socket says 172.21.15.189:55864) 2024-04-06T13:28:41.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.671+0000 7fa5de21b700 1 -- 172.21.15.189:0/842025141 learned_addr learned my addr 172.21.15.189:0/842025141 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:41.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.671+0000 7fa5dd219700 1 --2- 172.21.15.189:0/842025141 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5d806ae40 0x7fa5d80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:41.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.671+0000 7fa5dca18700 1 -- 172.21.15.189:0/842025141 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250842204 0 0) 0x7fa5d80b3990 con 0x7fa5d815ddf0 2024-04-06T13:28:41.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.671+0000 7fa5dca18700 1 -- 172.21.15.189:0/842025141 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5c40035b0 con 0x7fa5d815ddf0 2024-04-06T13:28:41.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.671+0000 7fa5dca18700 1 -- 172.21.15.189:0/842025141 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205137898 0 0) 0x7fa5d80b3060 con 0x7fa5d815d7b0 2024-04-06T13:28:41.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.671+0000 7fa5dd219700 1 -- 172.21.15.189:0/842025141 >> v1:172.21.15.192:6789/0 conn(0x7fa5d815d7b0 legacy=0x7fa5d806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:41.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.671+0000 7fa5dd219700 1 -- 172.21.15.189:0/842025141 >> v1:172.21.15.189:6789/0 conn(0x7fa5d815ddf0 legacy=0x7fa5d815e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:41.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.671+0000 7fa5dd219700 1 -- 172.21.15.189:0/842025141 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d80b3620 con 0x7fa5d806ae40 2024-04-06T13:28:41.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.672+0000 7fa5dd219700 1 --2- 172.21.15.189:0/842025141 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5d806ae40 0x7fa5d80b3e20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8009870 tx=0x7fa5c8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3a7424ba5492581 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:41.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.672+0000 7fa5dca18700 1 -- 172.21.15.189:0/842025141 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8011830 con 0x7fa5d806ae40 2024-04-06T13:28:41.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.672+0000 7fa5dca18700 1 -- 172.21.15.189:0/842025141 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c8011990 con 0x7fa5d806ae40 2024-04-06T13:28:41.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.672+0000 7fa5dca18700 1 -- 172.21.15.189:0/842025141 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8011d20 con 0x7fa5d806ae40 2024-04-06T13:28:41.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.672+0000 7fa5dfc7e700 1 -- 172.21.15.189:0/842025141 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5d806ae40 msgr2=0x7fa5d80b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:41.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.672+0000 7fa5dfc7e700 1 --2- 172.21.15.189:0/842025141 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5d806ae40 0x7fa5d80b3e20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8009870 tx=0x7fa5c8009b80 comp rx=0 tx=0).stop 2024-04-06T13:28:41.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.672+0000 7fa5dfc7e700 1 -- 172.21.15.189:0/842025141 shutdown_connections 2024-04-06T13:28:41.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.672+0000 7fa5dfc7e700 1 --2- 172.21.15.189:0/842025141 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5d806ae40 0x7fa5d80b3e20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:41.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.672+0000 7fa5dfc7e700 1 -- 172.21.15.189:0/842025141 >> 172.21.15.189:0/842025141 conn(0x7fa5d80d00b0 msgr2=0x7fa5d80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:41.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.673+0000 7fa5dfc7e700 1 -- 172.21.15.189:0/842025141 shutdown_connections 2024-04-06T13:28:41.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.673+0000 7fa5dfc7e700 1 -- 172.21.15.189:0/842025141 wait complete. 2024-04-06T13:28:41.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.674+0000 7fa5dfc7e700 1 Processor -- start 2024-04-06T13:28:41.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.674+0000 7fa5dfc7e700 1 -- start start 2024-04-06T13:28:41.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.675+0000 7fa5dfc7e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d806ae40 0x7fa5d8154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:41.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.675+0000 7fa5dfc7e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5d815d7b0 0x7fa5d806f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:41.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.675+0000 7fa5dfc7e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5d815ddf0 0x7fa5d806fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:41.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.675+0000 7fa5dfc7e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d81f3030 con 0x7fa5d806ae40 2024-04-06T13:28:41.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.675+0000 7fa5dfc7e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80af870 con 0x7fa5d815d7b0 2024-04-06T13:28:41.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.675+0000 7fa5dfc7e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80afb50 con 0x7fa5d815ddf0 2024-04-06T13:28:41.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.675+0000 7fa5dda1a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d806ae40 0x7fa5d8154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:41.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.675+0000 7fa5dda1a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d806ae40 0x7fa5d8154d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59006/0 (socket says 172.21.15.189:59006) 2024-04-06T13:28:41.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.675+0000 7fa5dda1a700 1 -- 172.21.15.189:0/2493605152 learned_addr learned my addr 172.21.15.189:0/2493605152 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:41.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.675+0000 7fa5de21b700 1 --2- 172.21.15.189:0/2493605152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5d815ddf0 0x7fa5d806fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:41.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.675+0000 7fa5dd219700 1 --2- 172.21.15.189:0/2493605152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5d815d7b0 0x7fa5d806f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:41.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.675+0000 7fa5dda1a700 1 -- 172.21.15.189:0/2493605152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5d815ddf0 msgr2=0x7fa5d806fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:41.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.675+0000 7fa5dda1a700 1 --2- 172.21.15.189:0/2493605152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5d815ddf0 0x7fa5d806fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:41.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.675+0000 7fa5dda1a700 1 -- 172.21.15.189:0/2493605152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5d815d7b0 msgr2=0x7fa5d806f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:41.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.675+0000 7fa5dda1a700 1 --2- 172.21.15.189:0/2493605152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5d815d7b0 0x7fa5d806f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:41.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.675+0000 7fa5dda1a700 1 -- 172.21.15.189:0/2493605152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c80094a0 con 0x7fa5d806ae40 2024-04-06T13:28:41.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.676+0000 7fa5dda1a700 1 --2- 172.21.15.189:0/2493605152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d806ae40 0x7fa5d8154d40 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fa5d40011e0 tx=0x7fa5d400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:41.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.676+0000 7fa5ce7fc700 1 -- 172.21.15.189:0/2493605152 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d40054e0 con 0x7fa5d806ae40 2024-04-06T13:28:41.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.676+0000 7fa5ce7fc700 1 -- 172.21.15.189:0/2493605152 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d4003680 con 0x7fa5d806ae40 2024-04-06T13:28:41.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.676+0000 7fa5ce7fc700 1 -- 172.21.15.189:0/2493605152 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d4003a30 con 0x7fa5d806ae40 2024-04-06T13:28:41.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.676+0000 7fa5dfc7e700 1 -- 172.21.15.189:0/2493605152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d80afe30 con 0x7fa5d806ae40 2024-04-06T13:28:41.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.676+0000 7fa5dfc7e700 1 -- 172.21.15.189:0/2493605152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d8155280 con 0x7fa5d806ae40 2024-04-06T13:28:41.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.677+0000 7fa5dfc7e700 1 -- 172.21.15.189:0/2493605152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d8155410 con 0x7fa5d806ae40 2024-04-06T13:28:41.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.677+0000 7fa5ce7fc700 1 -- 172.21.15.189:0/2493605152 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fa5d4003b90 con 0x7fa5d806ae40 2024-04-06T13:28:41.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.678+0000 7fa5ce7fc700 1 --2- 172.21.15.189:0/2493605152 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5bc06adb0 0x7fa5bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:41.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.678+0000 7fa5dd219700 1 --2- 172.21.15.189:0/2493605152 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5bc06adb0 0x7fa5bc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:41.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.678+0000 7fa5ce7fc700 1 -- 172.21.15.189:0/2493605152 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa5d4088a50 con 0x7fa5d806ae40 2024-04-06T13:28:41.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.678+0000 7fa5dd219700 1 --2- 172.21.15.189:0/2493605152 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5bc06adb0 0x7fa5bc06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8009870 tx=0x7fa5c801c040 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:41.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.682+0000 7fa5ce7fc700 1 -- 172.21.15.189:0/2493605152 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa5d40539a0 con 0x7fa5d806ae40 2024-04-06T13:28:41.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:41.930+0000 7fa5dfc7e700 1 -- 172.21.15.189:0/2493605152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa5d8067bb0 con 0x7fa5d806ae40 2024-04-06T13:28:42.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.828+0000 7fa5ce7fc700 1 -- 172.21.15.189:0/2493605152 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v77) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa5d4017020 con 0x7fa5d806ae40 2024-04-06T13:28:42.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.830+0000 7fa5dfc7e700 1 -- 172.21.15.189:0/2493605152 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5bc06adb0 msgr2=0x7fa5bc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:42.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.830+0000 7fa5dfc7e700 1 --2- 172.21.15.189:0/2493605152 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5bc06adb0 0x7fa5bc06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8009870 tx=0x7fa5c801c040 comp rx=0 tx=0).stop 2024-04-06T13:28:42.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.830+0000 7fa5dfc7e700 1 -- 172.21.15.189:0/2493605152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d806ae40 msgr2=0x7fa5d8154d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:42.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.830+0000 7fa5dfc7e700 1 --2- 172.21.15.189:0/2493605152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d806ae40 0x7fa5d8154d40 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fa5d40011e0 tx=0x7fa5d400ca30 comp rx=0 tx=0).stop 2024-04-06T13:28:42.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.831+0000 7fa5dfc7e700 1 -- 172.21.15.189:0/2493605152 shutdown_connections 2024-04-06T13:28:42.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.831+0000 7fa5dfc7e700 1 --2- 172.21.15.189:0/2493605152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5d815ddf0 0x7fa5d806fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:42.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.831+0000 7fa5dfc7e700 1 --2- 172.21.15.189:0/2493605152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d806ae40 0x7fa5d8154d40 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:42.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.831+0000 7fa5dfc7e700 1 --2- 172.21.15.189:0/2493605152 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa5bc06adb0 0x7fa5bc06d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:42.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.831+0000 7fa5dfc7e700 1 --2- 172.21.15.189:0/2493605152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5d815d7b0 0x7fa5d806f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:42.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.831+0000 7fa5dfc7e700 1 -- 172.21.15.189:0/2493605152 >> 172.21.15.189:0/2493605152 conn(0x7fa5d80d00b0 msgr2=0x7fa5d81570a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:42.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.831+0000 7fa5dfc7e700 1 -- 172.21.15.189:0/2493605152 shutdown_connections 2024-04-06T13:28:42.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.831+0000 7fa5dfc7e700 1 -- 172.21.15.189:0/2493605152 wait complete. 2024-04-06T13:28:42.847 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-06T13:28:42.847 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-06T13:28:42.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.977+0000 7f8acd628700 1 Processor -- start 2024-04-06T13:28:42.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.977+0000 7f8acd628700 1 -- start start 2024-04-06T13:28:42.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.977+0000 7f8acd628700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ac80cd3e0 0x7f8ac80cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:42.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.977+0000 7f8acd628700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8ac815d7c0 0x7f8ac806b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:42.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.977+0000 7f8acd628700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac806ceb0 con 0x7f8ac806b6f0 2024-04-06T13:28:42.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.978+0000 7f8acd628700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac80cdfa0 con 0x7f8ac815d7c0 2024-04-06T13:28:42.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.978+0000 7f8acd628700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac806d750 con 0x7f8ac80cd3e0 2024-04-06T13:28:42.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.978+0000 7f8ac67fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8ac815d7c0 0x7f8ac806b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:42.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.978+0000 7f8ac77fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8ac806b6f0 0x7f8ac81610f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55874/0 (socket says 172.21.15.189:55874) 2024-04-06T13:28:42.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.978+0000 7f8ac77fe700 1 -- 172.21.15.189:0/2617099287 learned_addr learned my addr 172.21.15.189:0/2617099287 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:42.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.978+0000 7f8ac6ffd700 1 --2- 172.21.15.189:0/2617099287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ac80cd3e0 0x7f8ac80cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:42.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.978+0000 7f8ac67fc700 1 -- 172.21.15.189:0/2617099287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ac80cd3e0 msgr2=0x7f8ac80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:42.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.978+0000 7f8ac67fc700 1 --2- 172.21.15.189:0/2617099287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ac80cd3e0 0x7f8ac80cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:42.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.978+0000 7f8ac67fc700 1 -- 172.21.15.189:0/2617099287 >> v1:172.21.15.189:6789/0 conn(0x7f8ac806b6f0 legacy=0x7f8ac81610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:42.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.978+0000 7f8ac67fc700 1 -- 172.21.15.189:0/2617099287 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac80cdd00 con 0x7f8ac815d7c0 2024-04-06T13:28:42.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.978+0000 7f8ac5ffb700 1 -- 172.21.15.189:0/2617099287 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619651499 0 0) 0x7f8ac806ceb0 con 0x7f8ac806b6f0 2024-04-06T13:28:42.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.978+0000 7f8ac67fc700 1 --2- 172.21.15.189:0/2617099287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ac815d7c0 0x7f8ac806b1b0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009840 tx=0x7f8abc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a6ee3516262d403 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:42.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.978+0000 7f8ac5ffb700 1 -- 172.21.15.189:0/2617099287 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc011830 con 0x7f8ac815d7c0 2024-04-06T13:28:42.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.978+0000 7f8ac5ffb700 1 -- 172.21.15.189:0/2617099287 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8abc011990 con 0x7f8ac815d7c0 2024-04-06T13:28:42.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.978+0000 7f8ac5ffb700 1 -- 172.21.15.189:0/2617099287 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc011d20 con 0x7f8ac815d7c0 2024-04-06T13:28:42.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.979+0000 7f8acd628700 1 -- 172.21.15.189:0/2617099287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ac815d7c0 msgr2=0x7f8ac806b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:42.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.979+0000 7f8acd628700 1 --2- 172.21.15.189:0/2617099287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ac815d7c0 0x7f8ac806b1b0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009840 tx=0x7f8abc009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:42.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.979+0000 7f8acd628700 1 -- 172.21.15.189:0/2617099287 shutdown_connections 2024-04-06T13:28:42.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.979+0000 7f8acd628700 1 --2- 172.21.15.189:0/2617099287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ac80cd3e0 0x7f8ac80cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:42.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.979+0000 7f8acd628700 1 --2- 172.21.15.189:0/2617099287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ac815d7c0 0x7f8ac806b1b0 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:42.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.979+0000 7f8acd628700 1 -- 172.21.15.189:0/2617099287 >> 172.21.15.189:0/2617099287 conn(0x7f8ac80cc290 msgr2=0x7f8ac80cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:42.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.979+0000 7f8acd628700 1 -- 172.21.15.189:0/2617099287 shutdown_connections 2024-04-06T13:28:42.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.979+0000 7f8acd628700 1 -- 172.21.15.189:0/2617099287 wait complete. 2024-04-06T13:28:42.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.980+0000 7f8acd628700 1 Processor -- start 2024-04-06T13:28:42.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.980+0000 7f8acd628700 1 -- start start 2024-04-06T13:28:42.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.981+0000 7f8acd628700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ac806b6f0 0x7f8ac815b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:42.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.981+0000 7f8acd628700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac80cd3e0 0x7f8ac80b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:42.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.981+0000 7f8acd628700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ac815d7c0 0x7f8ac8157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:42.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.981+0000 7f8acd628700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac80b3c50 con 0x7f8ac815d7c0 2024-04-06T13:28:42.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.981+0000 7f8acd628700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac80cb670 con 0x7f8ac80cd3e0 2024-04-06T13:28:42.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.981+0000 7f8acd628700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac80b2cf0 con 0x7f8ac806b6f0 2024-04-06T13:28:42.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.981+0000 7f8ac77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ac815d7c0 0x7f8ac8157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:42.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.981+0000 7f8ac77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ac815d7c0 0x7f8ac8157970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59016/0 (socket says 172.21.15.189:59016) 2024-04-06T13:28:42.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.981+0000 7f8ac77fe700 1 -- 172.21.15.189:0/3132783711 learned_addr learned my addr 172.21.15.189:0/3132783711 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:42.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.981+0000 7f8ac6ffd700 1 --2- 172.21.15.189:0/3132783711 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ac806b6f0 0x7f8ac815b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:42.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.981+0000 7f8ac77fe700 1 -- 172.21.15.189:0/3132783711 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ac806b6f0 msgr2=0x7f8ac815b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:42.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.981+0000 7f8ac77fe700 1 --2- 172.21.15.189:0/3132783711 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ac806b6f0 0x7f8ac815b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:42.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.981+0000 7f8ac67fc700 1 --2- 172.21.15.189:0/3132783711 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac80cd3e0 0x7f8ac80b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:42.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.981+0000 7f8ac77fe700 1 -- 172.21.15.189:0/3132783711 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac80cd3e0 msgr2=0x7f8ac80b4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:42.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.981+0000 7f8ac77fe700 1 --2- 172.21.15.189:0/3132783711 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac80cd3e0 0x7f8ac80b4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:42.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.981+0000 7f8ac77fe700 1 -- 172.21.15.189:0/3132783711 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8abc0094a0 con 0x7f8ac815d7c0 2024-04-06T13:28:42.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.981+0000 7f8ac67fc700 1 --2- 172.21.15.189:0/3132783711 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac80cd3e0 0x7f8ac80b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:28:42.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.981+0000 7f8ac77fe700 1 --2- 172.21.15.189:0/3132783711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ac815d7c0 0x7f8ac8157970 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f8ac000cba0 tx=0x7f8ac000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:42.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.982+0000 7f8aaf7fe700 1 -- 172.21.15.189:0/3132783711 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac000ab80 con 0x7f8ac815d7c0 2024-04-06T13:28:42.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.982+0000 7f8aaf7fe700 1 -- 172.21.15.189:0/3132783711 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac000ad00 con 0x7f8ac815d7c0 2024-04-06T13:28:42.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.982+0000 7f8acd628700 1 -- 172.21.15.189:0/3132783711 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac80b2fd0 con 0x7f8ac815d7c0 2024-04-06T13:28:42.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.982+0000 7f8aaf7fe700 1 -- 172.21.15.189:0/3132783711 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0007310 con 0x7f8ac815d7c0 2024-04-06T13:28:42.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.982+0000 7f8acd628700 1 -- 172.21.15.189:0/3132783711 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac80b3590 con 0x7f8ac815d7c0 2024-04-06T13:28:42.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.982+0000 7f8acd628700 1 -- 172.21.15.189:0/3132783711 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac8159590 con 0x7f8ac815d7c0 2024-04-06T13:28:42.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.983+0000 7f8aaf7fe700 1 -- 172.21.15.189:0/3132783711 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f8ac0007470 con 0x7f8ac815d7c0 2024-04-06T13:28:42.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.984+0000 7f8aaf7fe700 1 --2- 172.21.15.189:0/3132783711 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8ab406adb0 0x7f8ab406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:42.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.984+0000 7f8aaf7fe700 1 -- 172.21.15.189:0/3132783711 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8ac0089400 con 0x7f8ac815d7c0 2024-04-06T13:28:42.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.984+0000 7f8ac6ffd700 1 --2- 172.21.15.189:0/3132783711 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8ab406adb0 0x7f8ab406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:42.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.984+0000 7f8ac6ffd700 1 --2- 172.21.15.189:0/3132783711 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8ab406adb0 0x7f8ab406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8ab800b9b0 tx=0x7f8ab8008040 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:42.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:42.988+0000 7f8aaf7fe700 1 -- 172.21.15.189:0/3132783711 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8ac8159590 con 0x7f8ac815d7c0 2024-04-06T13:28:43.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.223+0000 7f8acd628700 1 -- 172.21.15.189:0/3132783711 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8ac8159590 con 0x7f8ac815d7c0 2024-04-06T13:28:43.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.832+0000 7f8aaf7fe700 1 -- 172.21.15.189:0/3132783711 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v78) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8ac8159590 con 0x7f8ac815d7c0 2024-04-06T13:28:43.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.835+0000 7f8acd628700 1 -- 172.21.15.189:0/3132783711 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8ab406adb0 msgr2=0x7f8ab406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:43.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.835+0000 7f8acd628700 1 --2- 172.21.15.189:0/3132783711 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8ab406adb0 0x7f8ab406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8ab800b9b0 tx=0x7f8ab8008040 comp rx=0 tx=0).stop 2024-04-06T13:28:43.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.835+0000 7f8acd628700 1 -- 172.21.15.189:0/3132783711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ac815d7c0 msgr2=0x7f8ac8157970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:43.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.835+0000 7f8acd628700 1 --2- 172.21.15.189:0/3132783711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ac815d7c0 0x7f8ac8157970 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f8ac000cba0 tx=0x7f8ac000cf60 comp rx=0 tx=0).stop 2024-04-06T13:28:43.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.835+0000 7f8acd628700 1 -- 172.21.15.189:0/3132783711 shutdown_connections 2024-04-06T13:28:43.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.835+0000 7f8acd628700 1 --2- 172.21.15.189:0/3132783711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ac815d7c0 0x7f8ac8157970 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:43.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.835+0000 7f8acd628700 1 --2- 172.21.15.189:0/3132783711 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ac806b6f0 0x7f8ac815b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:43.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.835+0000 7f8acd628700 1 --2- 172.21.15.189:0/3132783711 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f8ab406adb0 0x7f8ab406d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:43.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.835+0000 7f8acd628700 1 --2- 172.21.15.189:0/3132783711 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac80cd3e0 0x7f8ac80b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:43.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.835+0000 7f8acd628700 1 -- 172.21.15.189:0/3132783711 >> 172.21.15.189:0/3132783711 conn(0x7f8ac80cc290 msgr2=0x7f8ac806dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:43.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.835+0000 7f8acd628700 1 -- 172.21.15.189:0/3132783711 shutdown_connections 2024-04-06T13:28:43.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.835+0000 7f8acd628700 1 -- 172.21.15.189:0/3132783711 wait complete. 2024-04-06T13:28:43.851 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-06T13:28:43.851 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:28:43.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.977+0000 7ff421055700 1 Processor -- start 2024-04-06T13:28:43.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.977+0000 7ff421055700 1 -- start start 2024-04-06T13:28:43.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.977+0000 7ff421055700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff41c15d7b0 0x7ff41c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:43.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.977+0000 7ff421055700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff41c06a730 0x7ff41c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:43.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.977+0000 7ff421055700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41c0b3990 con 0x7ff41c15ddf0 2024-04-06T13:28:43.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.977+0000 7ff421055700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff41c06b0f0 con 0x7ff41c15d7b0 2024-04-06T13:28:43.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.977+0000 7ff421055700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c0b3060 con 0x7ff41c06a730 2024-04-06T13:28:43.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff41b59e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff41c15ddf0 0x7ff41c15e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55882/0 (socket says 172.21.15.189:55882) 2024-04-06T13:28:43.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff41b59e700 1 -- 172.21.15.189:0/902470987 learned_addr learned my addr 172.21.15.189:0/902470987 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:43.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff41ad9d700 1 --2- 172.21.15.189:0/902470987 >> v2:172.21.15.192:3300/0 conn(0x7ff41c15d7b0 0x7ff41c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:43.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff41a59c700 1 --2- 172.21.15.189:0/902470987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff41c06a730 0x7ff41c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:43.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff419d9b700 1 -- 172.21.15.189:0/902470987 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212500776 0 0) 0x7ff41c0b3990 con 0x7ff41c15ddf0 2024-04-06T13:28:43.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff419d9b700 1 -- 172.21.15.189:0/902470987 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4040035b0 con 0x7ff41c15ddf0 2024-04-06T13:28:43.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff419d9b700 1 -- 172.21.15.189:0/902470987 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 758940399 0 0) 0x7ff4040035b0 con 0x7ff41c15ddf0 2024-04-06T13:28:43.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff419d9b700 1 -- 172.21.15.189:0/902470987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff41c06a730 msgr2=0x7ff41c06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:43.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff419d9b700 1 --2- 172.21.15.189:0/902470987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff41c06a730 0x7ff41c06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:43.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff419d9b700 1 -- 172.21.15.189:0/902470987 >> v2:172.21.15.192:3300/0 conn(0x7ff41c15d7b0 msgr2=0x7ff41c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:43.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff419d9b700 1 --2- 172.21.15.189:0/902470987 >> v2:172.21.15.192:3300/0 conn(0x7ff41c15d7b0 0x7ff41c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:43.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff419d9b700 1 -- 172.21.15.189:0/902470987 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41c0b3340 con 0x7ff41c15ddf0 2024-04-06T13:28:43.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff419d9b700 1 -- 172.21.15.189:0/902470987 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7ff4140019f0 con 0x7ff41c15ddf0 2024-04-06T13:28:43.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff41ad9d700 1 --2- 172.21.15.189:0/902470987 >> v2:172.21.15.192:3300/0 conn(0x7ff41c15d7b0 0x7ff41c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:43.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff419d9b700 1 -- 172.21.15.189:0/902470987 >> v1:172.21.15.189:6789/0 conn(0x7ff41c15ddf0 legacy=0x7ff41c15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:43.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff419d9b700 1 --2- 172.21.15.189:0/902470987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff404003a00 0x7ff404003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:43.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff419d9b700 1 --2- 172.21.15.189:0/902470987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff41c06a730 0x7ff404004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:43.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff419d9b700 1 --2- 172.21.15.189:0/902470987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff404004d90 0x7ff404005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:43.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff419d9b700 1 -- 172.21.15.189:0/902470987 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c06b0f0 con 0x7ff404003a00 2024-04-06T13:28:43.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff419d9b700 1 -- 172.21.15.189:0/902470987 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c0b3060 con 0x7ff404004d90 2024-04-06T13:28:43.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff419d9b700 1 -- 172.21.15.189:0/902470987 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40400ba10 con 0x7ff41c06a730 2024-04-06T13:28:43.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff41ad9d700 1 --2- 172.21.15.189:0/902470987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff404003a00 0x7ff404003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:43.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff41a59c700 1 --2- 172.21.15.189:0/902470987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff41c06a730 0x7ff404004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.978+0000 7ff41b59e700 1 --2- 172.21.15.189:0/902470987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff404004d90 0x7ff404005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.979+0000 7ff41ad9d700 1 -- 172.21.15.189:0/902470987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff41c06a730 msgr2=0x7ff404004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.979+0000 7ff41ad9d700 1 --2- 172.21.15.189:0/902470987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff41c06a730 0x7ff404004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.979+0000 7ff41ad9d700 1 -- 172.21.15.189:0/902470987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff404004d90 msgr2=0x7ff404005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.979+0000 7ff41ad9d700 1 --2- 172.21.15.189:0/902470987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff404004d90 0x7ff404005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.979+0000 7ff41ad9d700 1 -- 172.21.15.189:0/902470987 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4040035b0 con 0x7ff404003a00 2024-04-06T13:28:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.979+0000 7ff41a59c700 1 --2- 172.21.15.189:0/902470987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff41c06a730 0x7ff404004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:28:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.979+0000 7ff41b59e700 1 --2- 172.21.15.189:0/902470987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff404004d90 0x7ff404005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:28:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.979+0000 7ff41ad9d700 1 --2- 172.21.15.189:0/902470987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff404003a00 0x7ff404003e20 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7ff40c009d30 tx=0x7ff404005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b2dd7dcf0b83311 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.979+0000 7ff419d9b700 1 -- 172.21.15.189:0/902470987 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40c009e80 con 0x7ff404003a00 2024-04-06T13:28:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.979+0000 7ff419d9b700 1 -- 172.21.15.189:0/902470987 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40c00e9a0 con 0x7ff404003a00 2024-04-06T13:28:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.979+0000 7ff419d9b700 1 -- 172.21.15.189:0/902470987 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40c00ed30 con 0x7ff404003a00 2024-04-06T13:28:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.979+0000 7ff421055700 1 -- 172.21.15.189:0/902470987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff404003a00 msgr2=0x7ff404003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.979+0000 7ff421055700 1 --2- 172.21.15.189:0/902470987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff404003a00 0x7ff404003e20 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7ff40c009d30 tx=0x7ff404005880 comp rx=0 tx=0).stop 2024-04-06T13:28:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.979+0000 7ff41b59e700 1 -- 172.21.15.189:0/902470987 reap_dead start 2024-04-06T13:28:43.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.979+0000 7ff421055700 1 -- 172.21.15.189:0/902470987 shutdown_connections 2024-04-06T13:28:43.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.979+0000 7ff421055700 1 -- 172.21.15.189:0/902470987 >> 172.21.15.189:0/902470987 conn(0x7ff41c0d00b0 msgr2=0x7ff41c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:43.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.979+0000 7ff421055700 1 -- 172.21.15.189:0/902470987 shutdown_connections 2024-04-06T13:28:43.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.979+0000 7ff421055700 1 -- 172.21.15.189:0/902470987 wait complete. 2024-04-06T13:28:43.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.981+0000 7ff421055700 1 Processor -- start 2024-04-06T13:28:43.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.981+0000 7ff421055700 1 -- start start 2024-04-06T13:28:43.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.982+0000 7ff421055700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff404003a00 0x7ff41c0cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:43.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.982+0000 7ff421055700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff41c15aef0 0x7ff41c0ce940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:43.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.982+0000 7ff421055700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff41c0cee80 0x7ff41c1f2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:43.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.982+0000 7ff421055700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c15a200 con 0x7ff404003a00 2024-04-06T13:28:43.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.982+0000 7ff421055700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c0cf300 con 0x7ff41c15aef0 2024-04-06T13:28:43.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.982+0000 7ff421055700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c15aa60 con 0x7ff41c0cee80 2024-04-06T13:28:43.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.982+0000 7ff41ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff404003a00 0x7ff41c0cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:43.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.982+0000 7ff41ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff404003a00 0x7ff41c0cd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59034/0 (socket says 172.21.15.189:59034) 2024-04-06T13:28:43.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.982+0000 7ff41ad9d700 1 -- 172.21.15.189:0/2942423490 learned_addr learned my addr 172.21.15.189:0/2942423490 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:43.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.982+0000 7ff41a59c700 1 --2- 172.21.15.189:0/2942423490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff41c15aef0 0x7ff41c0ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:43.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.982+0000 7ff41b59e700 1 --2- 172.21.15.189:0/2942423490 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff41c0cee80 0x7ff41c1f2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:43.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.983+0000 7ff41ad9d700 1 -- 172.21.15.189:0/2942423490 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff41c0cee80 msgr2=0x7ff41c1f2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:43.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.983+0000 7ff41ad9d700 1 --2- 172.21.15.189:0/2942423490 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff41c0cee80 0x7ff41c1f2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:43.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.983+0000 7ff41ad9d700 1 -- 172.21.15.189:0/2942423490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff41c15aef0 msgr2=0x7ff41c0ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:43.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.983+0000 7ff41ad9d700 1 --2- 172.21.15.189:0/2942423490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff41c15aef0 0x7ff41c0ce940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:43.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.983+0000 7ff41ad9d700 1 -- 172.21.15.189:0/2942423490 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41c1f38c0 con 0x7ff404003a00 2024-04-06T13:28:43.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.983+0000 7ff41ad9d700 1 --2- 172.21.15.189:0/2942423490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff404003a00 0x7ff41c0cd810 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7ff404004730 tx=0x7ff40c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:43.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.983+0000 7ff4037fe700 1 -- 172.21.15.189:0/2942423490 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40c021070 con 0x7ff404003a00 2024-04-06T13:28:43.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.983+0000 7ff4037fe700 1 -- 172.21.15.189:0/2942423490 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40c013b50 con 0x7ff404003a00 2024-04-06T13:28:43.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.983+0000 7ff421055700 1 -- 172.21.15.189:0/2942423490 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff41c15a410 con 0x7ff404003a00 2024-04-06T13:28:43.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.983+0000 7ff4037fe700 1 -- 172.21.15.189:0/2942423490 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40c009e80 con 0x7ff404003a00 2024-04-06T13:28:43.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.983+0000 7ff421055700 1 -- 172.21.15.189:0/2942423490 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff41c0cdd50 con 0x7ff404003a00 2024-04-06T13:28:43.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.984+0000 7ff421055700 1 -- 172.21.15.189:0/2942423490 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff41c15ad40 con 0x7ff404003a00 2024-04-06T13:28:43.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.984+0000 7ff4037fe700 1 -- 172.21.15.189:0/2942423490 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7ff40c025070 con 0x7ff404003a00 2024-04-06T13:28:43.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.985+0000 7ff4037fe700 1 --2- 172.21.15.189:0/2942423490 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff40806adb0 0x7ff40806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:43.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.985+0000 7ff4037fe700 1 -- 172.21.15.189:0/2942423490 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff40c08aad0 con 0x7ff404003a00 2024-04-06T13:28:43.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.985+0000 7ff41a59c700 1 --2- 172.21.15.189:0/2942423490 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff40806adb0 0x7ff40806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:43.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.985+0000 7ff41a59c700 1 --2- 172.21.15.189:0/2942423490 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff40806adb0 0x7ff40806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff41c160590 tx=0x7ff410008040 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:43.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:43.989+0000 7ff4037fe700 1 -- 172.21.15.189:0/2942423490 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff40c055b60 con 0x7ff404003a00 2024-04-06T13:28:44.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.224+0000 7ff421055700 1 -- 172.21.15.189:0/2942423490 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff41c0740b0 con 0x7ff404003a00 2024-04-06T13:28:44.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.224+0000 7ff4037fe700 1 -- 172.21.15.189:0/2942423490 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4750 (secure 0 0 0) 0x7ff40c02c090 con 0x7ff404003a00 2024-04-06T13:28:44.225 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:44.225 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":78,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/3388956983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3388956983},{"type":"v1","addr":"172.21.15.192:6834","nonce":3388956983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":8845,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1844918354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":1844918354},{"type":"v1","addr":"172.21.15.192:6835","nonce":1844918354}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":9618,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3742852104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3742852104},{"type":"v1","addr":"172.21.15.189:6836","nonce":3742852104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:28:40.813074+0000","modified":"2024-04-06T13:28:43.827822+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9621":{"gid":9621,"name":"c","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/2089191015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2089191015},{"type":"v1","addr":"172.21.15.189:6837","nonce":2089191015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-06T13:28:44.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.226+0000 7ff421055700 1 -- 172.21.15.189:0/2942423490 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff40806adb0 msgr2=0x7ff40806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:44.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.226+0000 7ff421055700 1 --2- 172.21.15.189:0/2942423490 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff40806adb0 0x7ff40806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff41c160590 tx=0x7ff410008040 comp rx=0 tx=0).stop 2024-04-06T13:28:44.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.226+0000 7ff421055700 1 -- 172.21.15.189:0/2942423490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff404003a00 msgr2=0x7ff41c0cd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:44.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.226+0000 7ff421055700 1 --2- 172.21.15.189:0/2942423490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff404003a00 0x7ff41c0cd810 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7ff404004730 tx=0x7ff40c0047c0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.226+0000 7ff421055700 1 -- 172.21.15.189:0/2942423490 shutdown_connections 2024-04-06T13:28:44.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.226+0000 7ff421055700 1 --2- 172.21.15.189:0/2942423490 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff41c0cee80 0x7ff41c1f2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.226+0000 7ff421055700 1 --2- 172.21.15.189:0/2942423490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff404003a00 0x7ff41c0cd810 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.226+0000 7ff421055700 1 --2- 172.21.15.189:0/2942423490 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff40806adb0 0x7ff40806d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.226+0000 7ff421055700 1 --2- 172.21.15.189:0/2942423490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff41c15aef0 0x7ff41c0ce940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.226+0000 7ff421055700 1 -- 172.21.15.189:0/2942423490 >> 172.21.15.189:0/2942423490 conn(0x7ff41c0d00b0 msgr2=0x7ff41c0ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:44.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.227+0000 7ff421055700 1 -- 172.21.15.189:0/2942423490 shutdown_connections 2024-04-06T13:28:44.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.227+0000 7ff421055700 1 -- 172.21.15.189:0/2942423490 wait complete. 2024-04-06T13:28:44.228 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 78 2024-04-06T13:28:44.240 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:28:44.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.365+0000 7f61b4931700 1 Processor -- start 2024-04-06T13:28:44.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.365+0000 7f61b4931700 1 -- start start 2024-04-06T13:28:44.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.366+0000 7f61b4931700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f61b0158fe0 0x7f61b006a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:44.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.366+0000 7f61b4931700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f61b006ac70 0x7f61b015bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:44.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.366+0000 7f61b4931700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b015c100 0x7f61b00717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:44.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.366+0000 7f61b4931700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b006b0f0 con 0x7f61b0158fe0 2024-04-06T13:28:44.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.366+0000 7f61b4931700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b015f9e0 con 0x7f61b006ac70 2024-04-06T13:28:44.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.366+0000 7f61b4931700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b015f0d0 con 0x7f61b015c100 2024-04-06T13:28:44.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.366+0000 7f61ae59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f61b0158fe0 0x7f61b006a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:44.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.366+0000 7f61ae59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f61b0158fe0 0x7f61b006a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59040/0 (socket says 172.21.15.189:59040) 2024-04-06T13:28:44.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.366+0000 7f61ae59c700 1 -- 172.21.15.189:0/492464737 learned_addr learned my addr 172.21.15.189:0/492464737 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:44.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.366+0000 7f61aed9d700 1 --2- 172.21.15.189:0/492464737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b015c100 0x7f61b00717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:44.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.366+0000 7f61add9b700 1 --2- 172.21.15.189:0/492464737 >> v2:172.21.15.192:3300/0 conn(0x7f61b006ac70 0x7f61b015bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:44.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.366+0000 7f61ae59c700 1 -- 172.21.15.189:0/492464737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b015c100 msgr2=0x7f61b00717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:44.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.366+0000 7f61ae59c700 1 --2- 172.21.15.189:0/492464737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b015c100 0x7f61b00717b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.366+0000 7f61ae59c700 1 -- 172.21.15.189:0/492464737 >> v2:172.21.15.192:3300/0 conn(0x7f61b006ac70 msgr2=0x7f61b015bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:44.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.367+0000 7f61ae59c700 1 --2- 172.21.15.189:0/492464737 >> v2:172.21.15.192:3300/0 conn(0x7f61b006ac70 0x7f61b015bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.367+0000 7f61ae59c700 1 -- 172.21.15.189:0/492464737 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b015c8f0 con 0x7f61b0158fe0 2024-04-06T13:28:44.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.367+0000 7f61ae59c700 1 --2- 172.21.15.189:0/492464737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b0158fe0 0x7f61b006a730 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f61b0067b80 tx=0x7f61a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b043fca93df445d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:44.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.367+0000 7f61ad59a700 1 -- 172.21.15.189:0/492464737 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a4011830 con 0x7f61b0158fe0 2024-04-06T13:28:44.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.367+0000 7f61ad59a700 1 -- 172.21.15.189:0/492464737 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a4011990 con 0x7f61b0158fe0 2024-04-06T13:28:44.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.367+0000 7f61ad59a700 1 -- 172.21.15.189:0/492464737 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a4011d20 con 0x7f61b0158fe0 2024-04-06T13:28:44.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.367+0000 7f61b4931700 1 -- 172.21.15.189:0/492464737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b0158fe0 msgr2=0x7f61b006a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:44.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.367+0000 7f61b4931700 1 --2- 172.21.15.189:0/492464737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b0158fe0 0x7f61b006a730 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f61b0067b80 tx=0x7f61a4009bd0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.367+0000 7f61b4931700 1 -- 172.21.15.189:0/492464737 shutdown_connections 2024-04-06T13:28:44.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.367+0000 7f61b4931700 1 --2- 172.21.15.189:0/492464737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b0158fe0 0x7f61b006a730 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.367+0000 7f61b4931700 1 --2- 172.21.15.189:0/492464737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b015c100 0x7f61b00717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.367+0000 7f61b4931700 1 --2- 172.21.15.189:0/492464737 >> v2:172.21.15.192:3300/0 conn(0x7f61b006ac70 0x7f61b015bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.367+0000 7f61b4931700 1 -- 172.21.15.189:0/492464737 >> 172.21.15.189:0/492464737 conn(0x7f61b00d00b0 msgr2=0x7f61b00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:44.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.368+0000 7f61b4931700 1 -- 172.21.15.189:0/492464737 shutdown_connections 2024-04-06T13:28:44.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.368+0000 7f61b4931700 1 -- 172.21.15.189:0/492464737 wait complete. 2024-04-06T13:28:44.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.369+0000 7f61b4931700 1 Processor -- start 2024-04-06T13:28:44.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.369+0000 7f61b4931700 1 -- start start 2024-04-06T13:28:44.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.370+0000 7f61b4931700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b006ac70 0x7f61b0157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:44.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.370+0000 7f61b4931700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b0158fe0 0x7f61b015a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:44.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.370+0000 7f61b4931700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61b015c100 0x7f61b015a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:44.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.370+0000 7f61b4931700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b01579d0 con 0x7f61b006ac70 2024-04-06T13:28:44.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.370+0000 7f61b4931700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b013a3c0 con 0x7f61b015c100 2024-04-06T13:28:44.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.370+0000 7f61b4931700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b013ca50 con 0x7f61b0158fe0 2024-04-06T13:28:44.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.370+0000 7f61ae59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b006ac70 0x7f61b0157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:44.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.370+0000 7f61ae59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b006ac70 0x7f61b0157490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59048/0 (socket says 172.21.15.189:59048) 2024-04-06T13:28:44.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.370+0000 7f61ae59c700 1 -- 172.21.15.189:0/1582578141 learned_addr learned my addr 172.21.15.189:0/1582578141 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:44.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.370+0000 7f61ae59c700 1 -- 172.21.15.189:0/1582578141 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b0158fe0 msgr2=0x7f61b015a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:44.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.370+0000 7f61ae59c700 1 --2- 172.21.15.189:0/1582578141 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b0158fe0 0x7f61b015a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.370+0000 7f61ae59c700 1 -- 172.21.15.189:0/1582578141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61b015c100 msgr2=0x7f61b015a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:44.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.370+0000 7f61ae59c700 1 --2- 172.21.15.189:0/1582578141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61b015c100 0x7f61b015a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.370+0000 7f61ae59c700 1 -- 172.21.15.189:0/1582578141 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a4009550 con 0x7f61b006ac70 2024-04-06T13:28:44.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.371+0000 7f61ae59c700 1 --2- 172.21.15.189:0/1582578141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b006ac70 0x7f61b0157490 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f61a4000c00 tx=0x7f61a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:44.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.371+0000 7f619effd700 1 -- 172.21.15.189:0/1582578141 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a4004300 con 0x7f61b006ac70 2024-04-06T13:28:44.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.371+0000 7f619effd700 1 -- 172.21.15.189:0/1582578141 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a401f070 con 0x7f61b006ac70 2024-04-06T13:28:44.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.371+0000 7f619effd700 1 -- 172.21.15.189:0/1582578141 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a40036a0 con 0x7f61b006ac70 2024-04-06T13:28:44.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.371+0000 7f61b4931700 1 -- 172.21.15.189:0/1582578141 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b013cd30 con 0x7f61b006ac70 2024-04-06T13:28:44.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.371+0000 7f61b4931700 1 -- 172.21.15.189:0/1582578141 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b013d2f0 con 0x7f61b006ac70 2024-04-06T13:28:44.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.372+0000 7f61b4931700 1 -- 172.21.15.189:0/1582578141 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b0067bb0 con 0x7f61b006ac70 2024-04-06T13:28:44.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.373+0000 7f619effd700 1 -- 172.21.15.189:0/1582578141 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f61a4021030 con 0x7f61b006ac70 2024-04-06T13:28:44.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.373+0000 7f619effd700 1 --2- 172.21.15.189:0/1582578141 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f619006adb0 0x7f619006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:44.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.373+0000 7f619effd700 1 -- 172.21.15.189:0/1582578141 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f61a408abe0 con 0x7f61b006ac70 2024-04-06T13:28:44.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.373+0000 7f61add9b700 1 --2- 172.21.15.189:0/1582578141 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f619006adb0 0x7f619006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:44.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.374+0000 7f61add9b700 1 --2- 172.21.15.189:0/1582578141 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f619006adb0 0x7f619006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f61b006b810 tx=0x7f61a0009500 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:44.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.377+0000 7f619effd700 1 -- 172.21.15.189:0/1582578141 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f61a4055cd0 con 0x7f61b006ac70 2024-04-06T13:28:44.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.609+0000 7f61b4931700 1 -- 172.21.15.189:0/1582578141 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61b0074990 con 0x7f61b006ac70 2024-04-06T13:28:44.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.610+0000 7f619effd700 1 -- 172.21.15.189:0/1582578141 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) v1 ==== 74+0+16374 (secure 0 0 0) 0x7f61a4015090 con 0x7f61b006ac70 2024-04-06T13:28:44.611 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:44.611 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":178,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:28:41.814911+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:28:38.579643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-04-06T13:28:39.779769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:28:44.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.612+0000 7f61b4931700 1 -- 172.21.15.189:0/1582578141 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f619006adb0 msgr2=0x7f619006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:44.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.612+0000 7f61b4931700 1 --2- 172.21.15.189:0/1582578141 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f619006adb0 0x7f619006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f61b006b810 tx=0x7f61a0009500 comp rx=0 tx=0).stop 2024-04-06T13:28:44.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.612+0000 7f61b4931700 1 -- 172.21.15.189:0/1582578141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b006ac70 msgr2=0x7f61b0157490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:44.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.612+0000 7f61b4931700 1 --2- 172.21.15.189:0/1582578141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b006ac70 0x7f61b0157490 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f61a4000c00 tx=0x7f61a4004750 comp rx=0 tx=0).stop 2024-04-06T13:28:44.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.612+0000 7f61b4931700 1 -- 172.21.15.189:0/1582578141 shutdown_connections 2024-04-06T13:28:44.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.612+0000 7f61b4931700 1 --2- 172.21.15.189:0/1582578141 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f619006adb0 0x7f619006d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.612+0000 7f61b4931700 1 --2- 172.21.15.189:0/1582578141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61b015c100 0x7f61b015a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.612+0000 7f61b4931700 1 --2- 172.21.15.189:0/1582578141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b006ac70 0x7f61b0157490 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.612+0000 7f61b4931700 1 --2- 172.21.15.189:0/1582578141 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b0158fe0 0x7f61b015a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.612+0000 7f61b4931700 1 -- 172.21.15.189:0/1582578141 >> 172.21.15.189:0/1582578141 conn(0x7f61b00d00b0 msgr2=0x7f61b00ccac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:44.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.612+0000 7f61b4931700 1 -- 172.21.15.189:0/1582578141 shutdown_connections 2024-04-06T13:28:44.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.612+0000 7f61b4931700 1 -- 172.21.15.189:0/1582578141 wait complete. 2024-04-06T13:28:44.625 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-06T13:28:44.625 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:28:44.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.747+0000 7f567aa3b700 1 Processor -- start 2024-04-06T13:28:44.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.747+0000 7f567aa3b700 1 -- start start 2024-04-06T13:28:44.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.748+0000 7f567aa3b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f56740d11b0 0x7f56740d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:44.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.748+0000 7f567aa3b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f56740cb870 0x7f567415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:44.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.748+0000 7f567aa3b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56740cfa00 0x7f56740cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:44.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.748+0000 7f567aa3b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56740ccdc0 con 0x7f56740cb870 2024-04-06T13:28:44.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.748+0000 7f567aa3b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56740d0b50 con 0x7f56740d11b0 2024-04-06T13:28:44.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.748+0000 7f567aa3b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56740d1ad0 con 0x7f56740cfa00 2024-04-06T13:28:44.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.748+0000 7f56737fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f56740cb870 0x7f567415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:44.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.748+0000 7f56737fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f56740cb870 0x7f567415c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59054/0 (socket says 172.21.15.189:59054) 2024-04-06T13:28:44.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.748+0000 7f56737fe700 1 -- 172.21.15.189:0/4022488206 learned_addr learned my addr 172.21.15.189:0/4022488206 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:44.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.748+0000 7f5673fff700 1 --2- 172.21.15.189:0/4022488206 >> v2:172.21.15.192:3300/0 conn(0x7f56740d11b0 0x7f56740d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:44.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.748+0000 7f5678fd8700 1 --2- 172.21.15.189:0/4022488206 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56740cfa00 0x7f56740cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:44.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.748+0000 7f56737fe700 1 -- 172.21.15.189:0/4022488206 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56740cfa00 msgr2=0x7f56740cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:44.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.748+0000 7f56737fe700 1 --2- 172.21.15.189:0/4022488206 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56740cfa00 0x7f56740cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.748+0000 7f56737fe700 1 -- 172.21.15.189:0/4022488206 >> v2:172.21.15.192:3300/0 conn(0x7f56740d11b0 msgr2=0x7f56740d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:44.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.748+0000 7f56737fe700 1 --2- 172.21.15.189:0/4022488206 >> v2:172.21.15.192:3300/0 conn(0x7f56740d11b0 0x7f56740d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.748+0000 7f56737fe700 1 -- 172.21.15.189:0/4022488206 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56740cbd80 con 0x7f56740cb870 2024-04-06T13:28:44.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.748+0000 7f5678fd8700 1 --2- 172.21.15.189:0/4022488206 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56740cfa00 0x7f56740cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:28:44.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.748+0000 7f5673fff700 1 --2- 172.21.15.189:0/4022488206 >> v2:172.21.15.192:3300/0 conn(0x7f56740d11b0 0x7f56740d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:28:44.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.748+0000 7f56737fe700 1 --2- 172.21.15.189:0/4022488206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56740cb870 0x7f567415c280 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f56740ced00 tx=0x7f5664009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d5098a2b77ba6d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:44.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.749+0000 7f5672ffd700 1 -- 172.21.15.189:0/4022488206 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566400b870 con 0x7f56740cb870 2024-04-06T13:28:44.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.749+0000 7f5672ffd700 1 -- 172.21.15.189:0/4022488206 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566400b9d0 con 0x7f56740cb870 2024-04-06T13:28:44.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.749+0000 7f5672ffd700 1 -- 172.21.15.189:0/4022488206 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566400bd60 con 0x7f56740cb870 2024-04-06T13:28:44.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.749+0000 7f567aa3b700 1 -- 172.21.15.189:0/4022488206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56740cb870 msgr2=0x7f567415c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:44.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.749+0000 7f567aa3b700 1 --2- 172.21.15.189:0/4022488206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56740cb870 0x7f567415c280 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f56740ced00 tx=0x7f5664009bd0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.749+0000 7f567aa3b700 1 -- 172.21.15.189:0/4022488206 shutdown_connections 2024-04-06T13:28:44.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.749+0000 7f567aa3b700 1 --2- 172.21.15.189:0/4022488206 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56740cfa00 0x7f56740cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.749+0000 7f567aa3b700 1 --2- 172.21.15.189:0/4022488206 >> v2:172.21.15.192:3300/0 conn(0x7f56740d11b0 0x7f56740d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.749+0000 7f567aa3b700 1 --2- 172.21.15.189:0/4022488206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56740cb870 0x7f567415c280 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.749+0000 7f567aa3b700 1 -- 172.21.15.189:0/4022488206 >> 172.21.15.189:0/4022488206 conn(0x7f56740b2db0 msgr2=0x7f56740b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:44.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.749+0000 7f567aa3b700 1 -- 172.21.15.189:0/4022488206 shutdown_connections 2024-04-06T13:28:44.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.749+0000 7f567aa3b700 1 -- 172.21.15.189:0/4022488206 wait complete. 2024-04-06T13:28:44.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.751+0000 7f567aa3b700 1 Processor -- start 2024-04-06T13:28:44.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.751+0000 7f567aa3b700 1 -- start start 2024-04-06T13:28:44.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.751+0000 7f567aa3b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56740cb870 0x7f567415dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:44.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.751+0000 7f567aa3b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56740cfa00 0x7f567414b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:44.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.751+0000 7f567aa3b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56740d11b0 0x7f567415c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:44.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.751+0000 7f567aa3b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567415d930 con 0x7f56740cb870 2024-04-06T13:28:44.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.751+0000 7f567aa3b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567415ccb0 con 0x7f56740cfa00 2024-04-06T13:28:44.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.751+0000 7f567aa3b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567415cf90 con 0x7f56740d11b0 2024-04-06T13:28:44.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.751+0000 7f5673fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56740cb870 0x7f567415dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:44.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.752+0000 7f5673fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56740cb870 0x7f567415dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59056/0 (socket says 172.21.15.189:59056) 2024-04-06T13:28:44.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.752+0000 7f5673fff700 1 -- 172.21.15.189:0/1355306856 learned_addr learned my addr 172.21.15.189:0/1355306856 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:44.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.752+0000 7f5673fff700 1 -- 172.21.15.189:0/1355306856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56740d11b0 msgr2=0x7f567415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:44.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.752+0000 7f5673fff700 1 --2- 172.21.15.189:0/1355306856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56740d11b0 0x7f567415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.752+0000 7f5673fff700 1 -- 172.21.15.189:0/1355306856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56740cfa00 msgr2=0x7f567414b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:44.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.752+0000 7f5673fff700 1 --2- 172.21.15.189:0/1355306856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56740cfa00 0x7f567414b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.752+0000 7f5673fff700 1 -- 172.21.15.189:0/1355306856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5664009550 con 0x7f56740cb870 2024-04-06T13:28:44.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.752+0000 7f5673fff700 1 --2- 172.21.15.189:0/1355306856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56740cb870 0x7f567415dd50 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f566800c750 tx=0x7f566800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:44.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.752+0000 7f5670ff9700 1 -- 172.21.15.189:0/1355306856 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668011800 con 0x7f56740cb870 2024-04-06T13:28:44.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.752+0000 7f5670ff9700 1 -- 172.21.15.189:0/1355306856 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5668011960 con 0x7f56740cb870 2024-04-06T13:28:44.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.752+0000 7f567aa3b700 1 -- 172.21.15.189:0/1355306856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f567415d270 con 0x7f56740cb870 2024-04-06T13:28:44.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.752+0000 7f5670ff9700 1 -- 172.21.15.189:0/1355306856 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668011cd0 con 0x7f56740cb870 2024-04-06T13:28:44.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.752+0000 7f567aa3b700 1 -- 172.21.15.189:0/1355306856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f567414b970 con 0x7f56740cb870 2024-04-06T13:28:44.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.753+0000 7f567aa3b700 1 -- 172.21.15.189:0/1355306856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567415da70 con 0x7f56740cb870 2024-04-06T13:28:44.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.754+0000 7f5670ff9700 1 -- 172.21.15.189:0/1355306856 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f5668021350 con 0x7f56740cb870 2024-04-06T13:28:44.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.754+0000 7f5670ff9700 1 --2- 172.21.15.189:0/1355306856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f566006adb0 0x7f566006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:44.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.754+0000 7f5670ff9700 1 -- 172.21.15.189:0/1355306856 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5668089740 con 0x7f56740cb870 2024-04-06T13:28:44.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.754+0000 7f56737fe700 1 --2- 172.21.15.189:0/1355306856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f566006adb0 0x7f566006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:44.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.754+0000 7f56737fe700 1 --2- 172.21.15.189:0/1355306856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f566006adb0 0x7f566006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f56740cbcb0 tx=0x7f5664010000 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:44.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.758+0000 7f5670ff9700 1 -- 172.21.15.189:0/1355306856 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5668054590 con 0x7f56740cb870 2024-04-06T13:28:44.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.949+0000 7f567aa3b700 1 -- 172.21.15.189:0/1355306856 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f56740cf4e0 con 0x7f566006adb0 2024-04-06T13:28:44.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.957+0000 7f5670ff9700 1 -- 172.21.15.189:0/1355306856 <== mgr.8614 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278920 (secure 0 0 0) 0x7f566401e070 con 0x7f566006adb0 2024-04-06T13:28:44.962 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:44.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.961+0000 7f567aa3b700 1 -- 172.21.15.189:0/1355306856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f566006adb0 msgr2=0x7f566006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:44.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.961+0000 7f567aa3b700 1 --2- 172.21.15.189:0/1355306856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f566006adb0 0x7f566006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f56740cbcb0 tx=0x7f5664010000 comp rx=0 tx=0).stop 2024-04-06T13:28:44.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.961+0000 7f567aa3b700 1 -- 172.21.15.189:0/1355306856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56740cb870 msgr2=0x7f567415dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:44.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.961+0000 7f567aa3b700 1 --2- 172.21.15.189:0/1355306856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56740cb870 0x7f567415dd50 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f566800c750 tx=0x7f566800cb10 comp rx=0 tx=0).stop 2024-04-06T13:28:44.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.961+0000 7f567aa3b700 1 -- 172.21.15.189:0/1355306856 shutdown_connections 2024-04-06T13:28:44.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.961+0000 7f567aa3b700 1 --2- 172.21.15.189:0/1355306856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56740d11b0 0x7f567415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.961+0000 7f567aa3b700 1 --2- 172.21.15.189:0/1355306856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56740cb870 0x7f567415dd50 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.961+0000 7f567aa3b700 1 --2- 172.21.15.189:0/1355306856 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f566006adb0 0x7f566006d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.961+0000 7f567aa3b700 1 --2- 172.21.15.189:0/1355306856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56740cfa00 0x7f567414b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:44.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.961+0000 7f567aa3b700 1 -- 172.21.15.189:0/1355306856 >> 172.21.15.189:0/1355306856 conn(0x7f56740b2db0 msgr2=0x7f567415ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:44.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.961+0000 7f567aa3b700 1 -- 172.21.15.189:0/1355306856 shutdown_connections 2024-04-06T13:28:44.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:44.961+0000 7f567aa3b700 1 -- 172.21.15.189:0/1355306856 wait complete. 2024-04-06T13:28:44.965 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:28:44.974 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-06T13:28:43.289238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590550,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":448,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":380396,"kb_used_data":18240,"kb_used_omap":12,"kb_used_meta":361971,"kb_avail":754594324,"statfs":{"total":773094113280,"available":772704587776,"internally_reserved":0,"allocated":18677760,"data_stored":10011909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":370658872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":182,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000613"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823295+0000","last_change":"2024-04-06T13:28:40.844338+0000","last_active":"2024-04-06T13:28:41.823295+0000","last_peered":"2024-04-06T13:28:41.823295+0000","last_clean":"2024-04-06T13:28:41.823295+0000","last_became_active":"2024-04-06T13:28:40.844116+0000","last_became_peered":"2024-04-06T13:28:40.844116+0000","last_unstale":"2024-04-06T13:28:41.823295+0000","last_undegraded":"2024-04-06T13:28:41.823295+0000","last_fullsized":"2024-04-06T13:28:41.823295+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:40:55.218858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823252+0000","last_change":"2024-04-06T13:28:40.830807+0000","last_active":"2024-04-06T13:28:41.823252+0000","last_peered":"2024-04-06T13:28:41.823252+0000","last_clean":"2024-04-06T13:28:41.823252+0000","last_became_active":"2024-04-06T13:28:40.830567+0000","last_became_peered":"2024-04-06T13:28:40.830567+0000","last_unstale":"2024-04-06T13:28:41.823252+0000","last_undegraded":"2024-04-06T13:28:41.823252+0000","last_fullsized":"2024-04-06T13:28:41.823252+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:49:30.966991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228605+0000","last_change":"2024-04-06T13:28:40.831798+0000","last_active":"2024-04-06T13:28:42.228605+0000","last_peered":"2024-04-06T13:28:42.228605+0000","last_clean":"2024-04-06T13:28:42.228605+0000","last_became_active":"2024-04-06T13:28:40.831163+0000","last_became_peered":"2024-04-06T13:28:40.831163+0000","last_unstale":"2024-04-06T13:28:42.228605+0000","last_undegraded":"2024-04-06T13:28:42.228605+0000","last_fullsized":"2024-04-06T13:28:42.228605+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:44:23.562469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823039+0000","last_change":"2024-04-06T13:28:40.842233+0000","last_active":"2024-04-06T13:28:41.823039+0000","last_peered":"2024-04-06T13:28:41.823039+0000","last_clean":"2024-04-06T13:28:41.823039+0000","last_became_active":"2024-04-06T13:28:40.841786+0000","last_became_peered":"2024-04-06T13:28:40.841786+0000","last_unstale":"2024-04-06T13:28:41.823039+0000","last_undegraded":"2024-04-06T13:28:41.823039+0000","last_fullsized":"2024-04-06T13:28:41.823039+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:00:38.061764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228926+0000","last_change":"2024-04-06T13:28:40.832719+0000","last_active":"2024-04-06T13:28:42.228926+0000","last_peered":"2024-04-06T13:28:42.228926+0000","last_clean":"2024-04-06T13:28:42.228926+0000","last_became_active":"2024-04-06T13:28:40.832481+0000","last_became_peered":"2024-04-06T13:28:40.832481+0000","last_unstale":"2024-04-06T13:28:42.228926+0000","last_undegraded":"2024-04-06T13:28:42.228926+0000","last_fullsized":"2024-04-06T13:28:42.228926+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:17:00.268284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228122+0000","last_change":"2024-04-06T13:28:40.827058+0000","last_active":"2024-04-06T13:28:42.228122+0000","last_peered":"2024-04-06T13:28:42.228122+0000","last_clean":"2024-04-06T13:28:42.228122+0000","last_became_active":"2024-04-06T13:28:40.826834+0000","last_became_peered":"2024-04-06T13:28:40.826834+0000","last_unstale":"2024-04-06T13:28:42.228122+0000","last_undegraded":"2024-04-06T13:28:42.228122+0000","last_fullsized":"2024-04-06T13:28:42.228122+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:25:10.763749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228509+0000","last_change":"2024-04-06T13:28:40.835375+0000","last_active":"2024-04-06T13:28:42.228509+0000","last_peered":"2024-04-06T13:28:42.228509+0000","last_clean":"2024-04-06T13:28:42.228509+0000","last_became_active":"2024-04-06T13:28:40.835085+0000","last_became_peered":"2024-04-06T13:28:40.835085+0000","last_unstale":"2024-04-06T13:28:42.228509+0000","last_undegraded":"2024-04-06T13:28:42.228509+0000","last_fullsized":"2024-04-06T13:28:42.228509+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:52:39.138833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228202+0000","last_change":"2024-04-06T13:28:40.844690+0000","last_active":"2024-04-06T13:28:42.228202+0000","last_peered":"2024-04-06T13:28:42.228202+0000","last_clean":"2024-04-06T13:28:42.228202+0000","last_became_active":"2024-04-06T13:28:40.844454+0000","last_became_peered":"2024-04-06T13:28:40.844454+0000","last_unstale":"2024-04-06T13:28:42.228202+0000","last_undegraded":"2024-04-06T13:28:42.228202+0000","last_fullsized":"2024-04-06T13:28:42.228202+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:56:20.132200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823314+0000","last_change":"2024-04-06T13:28:40.830212+0000","last_active":"2024-04-06T13:28:41.823314+0000","last_peered":"2024-04-06T13:28:41.823314+0000","last_clean":"2024-04-06T13:28:41.823314+0000","last_became_active":"2024-04-06T13:28:40.829971+0000","last_became_peered":"2024-04-06T13:28:40.829971+0000","last_unstale":"2024-04-06T13:28:41.823314+0000","last_undegraded":"2024-04-06T13:28:41.823314+0000","last_fullsized":"2024-04-06T13:28:41.823314+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:11:48.235841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823305+0000","last_change":"2024-04-06T13:28:40.841928+0000","last_active":"2024-04-06T13:28:41.823305+0000","last_peered":"2024-04-06T13:28:41.823305+0000","last_clean":"2024-04-06T13:28:41.823305+0000","last_became_active":"2024-04-06T13:28:40.841704+0000","last_became_peered":"2024-04-06T13:28:40.841704+0000","last_unstale":"2024-04-06T13:28:41.823305+0000","last_undegraded":"2024-04-06T13:28:41.823305+0000","last_fullsized":"2024-04-06T13:28:41.823305+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:57:08.430945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1c","version":"177'2","reported_seq":13,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823374+0000","last_change":"2024-04-06T13:28:40.416928+0000","last_active":"2024-04-06T13:28:41.823374+0000","last_peered":"2024-04-06T13:28:41.823374+0000","last_clean":"2024-04-06T13:28:41.823374+0000","last_became_active":"2024-04-06T13:28:40.416637+0000","last_became_peered":"2024-04-06T13:28:40.416637+0000","last_unstale":"2024-04-06T13:28:41.823374+0000","last_undegraded":"2024-04-06T13:28:41.823374+0000","last_fullsized":"2024-04-06T13:28:41.823374+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:32:44.959236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823070+0000","last_change":"2024-04-06T13:28:40.832830+0000","last_active":"2024-04-06T13:28:41.823070+0000","last_peered":"2024-04-06T13:28:41.823070+0000","last_clean":"2024-04-06T13:28:41.823070+0000","last_became_active":"2024-04-06T13:28:40.832566+0000","last_became_peered":"2024-04-06T13:28:40.832566+0000","last_unstale":"2024-04-06T13:28:41.823070+0000","last_undegraded":"2024-04-06T13:28:41.823070+0000","last_fullsized":"2024-04-06T13:28:41.823070+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:13:15.481226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228534+0000","last_change":"2024-04-06T13:28:40.831953+0000","last_active":"2024-04-06T13:28:42.228534+0000","last_peered":"2024-04-06T13:28:42.228534+0000","last_clean":"2024-04-06T13:28:42.228534+0000","last_became_active":"2024-04-06T13:28:40.831529+0000","last_became_peered":"2024-04-06T13:28:40.831529+0000","last_unstale":"2024-04-06T13:28:42.228534+0000","last_undegraded":"2024-04-06T13:28:42.228534+0000","last_fullsized":"2024-04-06T13:28:42.228534+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:23:52.673261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228927+0000","last_change":"2024-04-06T13:28:40.832562+0000","last_active":"2024-04-06T13:28:42.228927+0000","last_peered":"2024-04-06T13:28:42.228927+0000","last_clean":"2024-04-06T13:28:42.228927+0000","last_became_active":"2024-04-06T13:28:40.832361+0000","last_became_peered":"2024-04-06T13:28:40.832361+0000","last_unstale":"2024-04-06T13:28:42.228927+0000","last_undegraded":"2024-04-06T13:28:42.228927+0000","last_fullsized":"2024-04-06T13:28:42.228927+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:18:26.747951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"177'3","reported_seq":14,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228627+0000","last_change":"2024-04-06T13:28:40.411188+0000","last_active":"2024-04-06T13:28:42.228627+0000","last_peered":"2024-04-06T13:28:42.228627+0000","last_clean":"2024-04-06T13:28:42.228627+0000","last_became_active":"2024-04-06T13:28:40.410450+0000","last_became_peered":"2024-04-06T13:28:40.410450+0000","last_unstale":"2024-04-06T13:28:42.228627+0000","last_undegraded":"2024-04-06T13:28:42.228627+0000","last_fullsized":"2024-04-06T13:28:42.228627+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:07:22.365454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823157+0000","last_change":"2024-04-06T13:28:40.842464+0000","last_active":"2024-04-06T13:28:41.823157+0000","last_peered":"2024-04-06T13:28:41.823157+0000","last_clean":"2024-04-06T13:28:41.823157+0000","last_became_active":"2024-04-06T13:28:40.842238+0000","last_became_peered":"2024-04-06T13:28:40.842238+0000","last_unstale":"2024-04-06T13:28:41.823157+0000","last_undegraded":"2024-04-06T13:28:41.823157+0000","last_fullsized":"2024-04-06T13:28:41.823157+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:59:44.613608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823413+0000","last_change":"2024-04-06T13:28:40.829903+0000","last_active":"2024-04-06T13:28:41.823413+0000","last_peered":"2024-04-06T13:28:41.823413+0000","last_clean":"2024-04-06T13:28:41.823413+0000","last_became_active":"2024-04-06T13:28:40.829654+0000","last_became_peered":"2024-04-06T13:28:40.829654+0000","last_unstale":"2024-04-06T13:28:41.823413+0000","last_undegraded":"2024-04-06T13:28:41.823413+0000","last_fullsized":"2024-04-06T13:28:41.823413+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:29:15.092554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.f","version":"177'1","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228576+0000","last_change":"2024-04-06T13:28:40.409911+0000","last_active":"2024-04-06T13:28:42.228576+0000","last_peered":"2024-04-06T13:28:42.228576+0000","last_clean":"2024-04-06T13:28:42.228576+0000","last_became_active":"2024-04-06T13:28:40.409605+0000","last_became_peered":"2024-04-06T13:28:40.409605+0000","last_unstale":"2024-04-06T13:28:42.228576+0000","last_undegraded":"2024-04-06T13:28:42.228576+0000","last_fullsized":"2024-04-06T13:28:42.228576+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:37:00.320010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.d","version":"177'1","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823356+0000","last_change":"2024-04-06T13:28:40.412433+0000","last_active":"2024-04-06T13:28:41.823356+0000","last_peered":"2024-04-06T13:28:41.823356+0000","last_clean":"2024-04-06T13:28:41.823356+0000","last_became_active":"2024-04-06T13:28:40.411519+0000","last_became_peered":"2024-04-06T13:28:40.411519+0000","last_unstale":"2024-04-06T13:28:41.823356+0000","last_undegraded":"2024-04-06T13:28:41.823356+0000","last_fullsized":"2024-04-06T13:28:41.823356+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:11.032370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823188+0000","last_change":"2024-04-06T13:28:40.831641+0000","last_active":"2024-04-06T13:28:41.823188+0000","last_peered":"2024-04-06T13:28:41.823188+0000","last_clean":"2024-04-06T13:28:41.823188+0000","last_became_active":"2024-04-06T13:28:40.831383+0000","last_became_peered":"2024-04-06T13:28:40.831383+0000","last_unstale":"2024-04-06T13:28:41.823188+0000","last_undegraded":"2024-04-06T13:28:41.823188+0000","last_fullsized":"2024-04-06T13:28:41.823188+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:09:51.321262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823219+0000","last_change":"2024-04-06T13:28:40.843971+0000","last_active":"2024-04-06T13:28:41.823219+0000","last_peered":"2024-04-06T13:28:41.823219+0000","last_clean":"2024-04-06T13:28:41.823219+0000","last_became_active":"2024-04-06T13:28:40.843734+0000","last_became_peered":"2024-04-06T13:28:40.843734+0000","last_unstale":"2024-04-06T13:28:41.823219+0000","last_undegraded":"2024-04-06T13:28:41.823219+0000","last_fullsized":"2024-04-06T13:28:41.823219+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:25.783071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823399+0000","last_change":"2024-04-06T13:28:40.416331+0000","last_active":"2024-04-06T13:28:41.823399+0000","last_peered":"2024-04-06T13:28:41.823399+0000","last_clean":"2024-04-06T13:28:41.823399+0000","last_became_active":"2024-04-06T13:28:40.416050+0000","last_became_peered":"2024-04-06T13:28:40.416050+0000","last_unstale":"2024-04-06T13:28:41.823399+0000","last_undegraded":"2024-04-06T13:28:41.823399+0000","last_fullsized":"2024-04-06T13:28:41.823399+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:33:13.846379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228459+0000","last_change":"2024-04-06T13:28:40.422136+0000","last_active":"2024-04-06T13:28:42.228459+0000","last_peered":"2024-04-06T13:28:42.228459+0000","last_clean":"2024-04-06T13:28:42.228459+0000","last_became_active":"2024-04-06T13:28:40.421816+0000","last_became_peered":"2024-04-06T13:28:40.421816+0000","last_unstale":"2024-04-06T13:28:42.228459+0000","last_undegraded":"2024-04-06T13:28:42.228459+0000","last_fullsized":"2024-04-06T13:28:42.228459+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:05:41.512140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823188+0000","last_change":"2024-04-06T13:28:40.838259+0000","last_active":"2024-04-06T13:28:41.823188+0000","last_peered":"2024-04-06T13:28:41.823188+0000","last_clean":"2024-04-06T13:28:41.823188+0000","last_became_active":"2024-04-06T13:28:40.838005+0000","last_became_peered":"2024-04-06T13:28:40.838005+0000","last_unstale":"2024-04-06T13:28:41.823188+0000","last_undegraded":"2024-04-06T13:28:41.823188+0000","last_fullsized":"2024-04-06T13:28:41.823188+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:23:38.800047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:40.394568+0000","last_change":"2024-04-06T13:28:40.394568+0000","last_active":"2024-04-06T13:28:40.394568+0000","last_peered":"2024-04-06T13:28:40.394568+0000","last_clean":"2024-04-06T13:28:40.394568+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:40.394568+0000","last_undegraded":"2024-04-06T13:28:40.394568+0000","last_fullsized":"2024-04-06T13:28:40.394568+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:28:39.389487+0000","last_change":"2024-04-06T13:28:39.389487+0000","last_active":"2024-04-06T13:28:39.389487+0000","last_peered":"2024-04-06T13:28:39.389487+0000","last_clean":"2024-04-06T13:28:39.389487+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:28:39.389487+0000","last_undegraded":"2024-04-06T13:28:39.389487+0000","last_fullsized":"2024-04-06T13:28:39.389487+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"167'271","reported_seq":902,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-04-06T13:28:35.548013+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:28:35.548013+0000","last_peered":"2024-04-06T13:28:35.548013+0000","last_clean":"2024-04-06T13:28:35.548013+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:28:35.548013+0000","last_undegraded":"2024-04-06T13:28:35.548013+0000","last_fullsized":"2024-04-06T13:28:35.548013+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":182,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":12,"acting":12,"num_store_stats":2},{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":36,"acting":36,"num_store_stats":2},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51884,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319956,"statfs":{"total":96636764160,"available":96583634944,"internally_reserved":0,"allocated":2768896,"data_stored":1688223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.58799999999999997,"15min":0.59299999999999997},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.78200000000000003,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59599999999999997,"15min":0.60099999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.35899999999999999},"max":{"1min":0.78500000000000003,"5min":0.86099999999999999,"15min":1.4790000000000001},"last":0.376}]},{"osd":1,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.60799999999999998,"15min":0.60999999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.79500000000000004,"5min":0.82499999999999996,"15min":0.83299999999999996},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.40300000000000002,"5min":0.39300000000000002,"15min":0.39000000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":1.5249999999999999},"last":0.70799999999999996}]},{"osd":2,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59099999999999997,"15min":0.58199999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59699999999999998,"15min":0.61099999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":1.198},"last":0.46999999999999997}]},{"osd":3,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.60699999999999998,"15min":0.60799999999999998},"min":{"1min":0.437,"5min":0.36499999999999999,"15min":0.35499999999999998},"max":{"1min":0.77500000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61599999999999999,"15min":0.61199999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.78600000000000003,"5min":0.79600000000000004,"15min":1.1200000000000001},"last":0.55400000000000005}]},{"osd":4,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45400000000000001,"15min":0.45200000000000001},"min":{"1min":0.29599999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.97399999999999998},"last":0.747},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55800000000000005,"15min":0.54000000000000004},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.33600000000000002},"max":{"1min":0.72399999999999998,"5min":0.77600000000000002,"15min":0.78000000000000003},"last":0.58399999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.52700000000000002,"15min":0.51500000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69599999999999995,"5min":0.80000000000000004,"15min":1.024},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.58499999999999996,"15min":0.57999999999999996},"min":{"1min":0.36899999999999999,"5min":0.35599999999999998,"15min":0.33000000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":1.046},"last":0.877}]},{"osd":6,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56699999999999995,"15min":0.56100000000000005},"min":{"1min":0.41399999999999998,"5min":0.38800000000000001,"15min":0.35299999999999998},"max":{"1min":0.81100000000000005,"5min":0.82199999999999995,"15min":1.4159999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.59599999999999997,"15min":0.57899999999999996},"min":{"1min":0.35799999999999998,"5min":0.34599999999999997,"15min":0.33700000000000002},"max":{"1min":0.75600000000000001,"5min":0.80700000000000005,"15min":0.873},"last":0.63800000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575761,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37344,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334496,"statfs":{"total":96636764160,"available":96598523904,"internally_reserved":0,"allocated":2170880,"data_stored":1096836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57799999999999996,"15min":0.56799999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.34399999999999997},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.83499999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58699999999999997,"15min":0.57699999999999996},"min":{"1min":0.40400000000000003,"5min":0.40300000000000002,"15min":0.35899999999999999},"max":{"1min":0.73399999999999999,"5min":0.76200000000000001,"15min":0.78800000000000003},"last":0.55700000000000005}]},{"osd":1,"last update":"Sat Apr 6 13:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61899999999999999,"15min":0.624},"min":{"1min":0.42799999999999999,"5min":0.37,"15min":0.35199999999999998},"max":{"1min":0.77100000000000002,"5min":0.80300000000000005,"15min":0.81799999999999995},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.64800000000000002,"15min":0.64200000000000002},"min":{"1min":0.52600000000000002,"5min":0.48599999999999999,"15min":0.311},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.67500000000000004}]},{"osd":2,"last update":"Sat Apr 6 13:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.47699999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.84499999999999997},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60899999999999999,"15min":0.60199999999999998},"min":{"1min":0.45000000000000001,"5min":0.42699999999999999,"15min":0.373},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.80900000000000005},"last":0.623}]},{"osd":3,"last update":"Sat Apr 6 13:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.626,"15min":0.61899999999999999},"min":{"1min":0.50600000000000001,"5min":0.42399999999999999,"15min":0.40799999999999997},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.66500000000000004,"15min":0.66200000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.877},"last":0.60999999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.31},"max":{"1min":0.64100000000000001,"5min":0.746,"15min":0.81000000000000005},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54400000000000004,"15min":0.54700000000000004},"min":{"1min":0.38900000000000001,"5min":0.29499999999999998,"15min":0.29299999999999998},"max":{"1min":0.74199999999999999,"5min":0.80200000000000005,"15min":0.873},"last":0.378}]},{"osd":5,"last update":"Sat Apr 6 13:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51500000000000001,"15min":0.51600000000000001},"min":{"1min":0.36499999999999999,"5min":0.32600000000000001,"15min":0.307},"max":{"1min":0.72799999999999998,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53300000000000003,"15min":0.53600000000000003},"min":{"1min":0.36399999999999999,"5min":0.36199999999999999,"15min":0.32800000000000001},"max":{"1min":0.70999999999999996,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.442}]},{"osd":7,"last update":"Sat Apr 6 13:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56000000000000005,"15min":0.56100000000000005},"min":{"1min":0.41599999999999998,"5min":0.35299999999999998,"15min":0.34899999999999998},"max":{"1min":0.81100000000000005,"5min":0.85399999999999998,"15min":0.85599999999999998},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.38700000000000001,"5min":0.375,"15min":0.36499999999999999},"max":{"1min":0.82599999999999996,"5min":0.83299999999999996,"15min":0.85999999999999999},"last":0.56999999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575761,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37384,"kb_used_data":2172,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334456,"statfs":{"total":96636764160,"available":96598482944,"internally_reserved":0,"allocated":2224128,"data_stored":1123529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.39900000000000002,"15min":0.38800000000000001},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.50900000000000001,"5min":0.54800000000000004,"15min":0.64300000000000002},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.434,"15min":0.42899999999999999},"min":{"1min":0.28799999999999998,"5min":0.27000000000000002,"15min":0.26700000000000002},"max":{"1min":0.56100000000000005,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.32300000000000001}]},{"osd":2,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.42999999999999999,"15min":0.42499999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.52400000000000002,"5min":0.64000000000000001,"15min":0.67400000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.47699999999999998,"15min":0.46500000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59399999999999997,"5min":0.70499999999999996,"15min":0.73299999999999998},"last":0.47399999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.48499999999999999,"15min":0.47299999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.30599999999999999},"max":{"1min":0.64200000000000002,"5min":0.70899999999999996,"15min":0.77100000000000002},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.45600000000000002,"15min":0.44700000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.63200000000000001,"5min":0.65500000000000003,"15min":0.68000000000000005},"last":0.42299999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65900000000000003,"15min":0.64700000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.85799999999999998,"5min":0.92500000000000004,"15min":1.008},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.66900000000000004,"15min":0.65900000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.57499999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.67300000000000004,"15min":0.66200000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.371},"max":{"1min":0.93799999999999994,"5min":0.95499999999999996,"15min":0.98299999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.69099999999999995,"15min":0.67400000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91800000000000004,"5min":0.94199999999999995,"15min":1.0189999999999999},"last":0.64400000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.67600000000000005,"15min":0.66700000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.97199999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.69499999999999995,"15min":0.68500000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.877,"5min":1,"15min":1},"last":0.71499999999999997}]},{"osd":7,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.71199999999999997,"15min":0.69499999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.89000000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68000000000000005,"15min":0.66900000000000004},"min":{"1min":0.48699999999999999,"5min":0.46899999999999997,"15min":0.44},"max":{"1min":0.91200000000000003,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.627}]}]},{"osd":0,"up_from":13,"seq":55834575760,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37920,"kb_used_data":2700,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94333920,"statfs":{"total":96636764160,"available":96597934080,"internally_reserved":0,"allocated":2764800,"data_stored":1687204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39700000000000002,"15min":0.39800000000000002},"min":{"1min":0.27600000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.46700000000000003,"5min":0.58799999999999997,"15min":0.65000000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42599999999999999,"15min":0.42499999999999999},"min":{"1min":0.28499999999999998,"5min":0.28000000000000003,"15min":0.26400000000000001},"max":{"1min":0.57199999999999995,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.35999999999999999}]},{"osd":2,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.434,"15min":0.434},"min":{"1min":0.313,"5min":0.29699999999999999,"15min":0.27000000000000002},"max":{"1min":0.58699999999999997,"5min":0.67200000000000004,"15min":0.71199999999999997},"last":0.371},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.45000000000000001,"15min":0.45300000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.51400000000000001,"5min":0.63,"15min":0.77000000000000002},"last":0.33200000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.436,"15min":0.435},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.27900000000000003},"max":{"1min":0.51100000000000001,"5min":0.61899999999999999,"15min":0.63200000000000001},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.39700000000000002,"5min":0.34399999999999997,"15min":0.307},"max":{"1min":0.64500000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.39100000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.67200000000000004,"15min":0.65400000000000003},"min":{"1min":0.47899999999999998,"5min":0.46899999999999997,"15min":0.45600000000000002},"max":{"1min":0.86299999999999999,"5min":0.93000000000000005,"15min":0.999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.67900000000000005,"15min":0.66800000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.41499999999999998},"max":{"1min":0.82499999999999996,"5min":0.96699999999999997,"15min":1.103},"last":0.63200000000000001}]},{"osd":5,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.66900000000000004,"15min":0.65700000000000003},"min":{"1min":0.46700000000000003,"5min":0.46100000000000002,"15min":0.46000000000000002},"max":{"1min":0.77100000000000002,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70499999999999996,"15min":0.68400000000000005},"min":{"1min":0.497,"5min":0.48999999999999999,"15min":0.41299999999999998},"max":{"1min":0.89100000000000001,"5min":1.0169999999999999,"15min":1.0469999999999999},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.70399999999999996,"15min":0.68899999999999995},"min":{"1min":0.54000000000000004,"5min":0.49399999999999999,"15min":0.46000000000000002},"max":{"1min":0.91100000000000003,"5min":1.0629999999999999,"15min":1.0920000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.70399999999999996,"15min":0.68799999999999994},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.46000000000000002},"max":{"1min":0.871,"5min":0.92900000000000005,"15min":0.98599999999999999},"last":0.55900000000000005}]},{"osd":7,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.71999999999999997,"15min":0.70299999999999996},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.46400000000000002},"max":{"1min":0.81399999999999995,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.71499999999999997,"15min":0.69299999999999995},"min":{"1min":0.502,"5min":0.48899999999999999,"15min":0.438},"max":{"1min":0.92400000000000004,"5min":1.042,"15min":1.071},"last":0.55000000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575763,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55444,"kb_used_data":2176,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316396,"statfs":{"total":96636764160,"available":96579989504,"internally_reserved":0,"allocated":2228224,"data_stored":1123571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.39400000000000002,"15min":0.39200000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51300000000000001,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":2.157},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.437,"15min":0.434},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.59199999999999997,"5min":0.64300000000000002,"15min":0.68700000000000006},"last":0.50800000000000001}]},{"osd":1,"last update":"Sat Apr 6 13:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.42199999999999999,"15min":0.41499999999999998},"min":{"1min":0.28999999999999998,"5min":0.27500000000000002,"15min":0.254},"max":{"1min":0.62,"5min":0.62,"15min":0.63300000000000001},"last":2.6190000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.45900000000000002,"15min":0.45300000000000001},"min":{"1min":0.32800000000000001,"5min":0.32400000000000001,"15min":0.308},"max":{"1min":0.49299999999999999,"5min":0.628,"15min":0.68899999999999995},"last":2.0870000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.46899999999999997,"15min":0.47099999999999997},"min":{"1min":0.35899999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.57099999999999995,"5min":0.65800000000000003,"15min":0.68100000000000005},"last":3.9980000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45500000000000002,"15min":0.45400000000000001},"min":{"1min":0.33600000000000002,"5min":0.33300000000000002,"15min":0.29699999999999999},"max":{"1min":0.53500000000000003,"5min":0.627,"15min":0.65300000000000002},"last":4.0350000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65700000000000003,"15min":0.65100000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85299999999999998,"5min":0.85799999999999998,"15min":0.93700000000000006},"last":6.335},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67500000000000004,"15min":0.66800000000000004},"min":{"1min":0.52700000000000002,"5min":0.47799999999999998,"15min":0.434},"max":{"1min":0.85199999999999998,"5min":0.91500000000000004,"15min":0.91800000000000004},"last":5.2610000000000001}]},{"osd":5,"last update":"Sat Apr 6 13:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67500000000000004},"min":{"1min":0.52100000000000002,"5min":0.46800000000000003,"15min":0.42699999999999999},"max":{"1min":0.82799999999999996,"5min":0.95699999999999996,"15min":0.97399999999999998},"last":2.7599999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.66600000000000004,"15min":0.66900000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.40699999999999997},"max":{"1min":0.88400000000000001,"5min":0.89000000000000001,"15min":0.95399999999999996},"last":2.206}]},{"osd":6,"last update":"Sat Apr 6 13:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.67700000000000005,"15min":0.67400000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.40500000000000003},"max":{"1min":0.83399999999999996,"5min":0.93899999999999995,"15min":1.022},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69599999999999995,"15min":0.69799999999999995},"min":{"1min":0.47699999999999998,"5min":0.47299999999999998,"15min":0.41999999999999998},"max":{"1min":0.84499999999999997,"5min":0.95099999999999996,"15min":1.069},"last":1.55}]},{"osd":7,"last update":"Sat Apr 6 13:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.70199999999999996,"15min":0.69099999999999995},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.438},"max":{"1min":0.85499999999999998,"5min":0.95399999999999996,"15min":0.998},"last":2.0110000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71499999999999997,"15min":0.71199999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.43099999999999999},"max":{"1min":0.88500000000000001,"5min":0.93300000000000005,"15min":1.099},"last":2.8090000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575761,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":59492,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312348,"statfs":{"total":96636764160,"available":96575844352,"internally_reserved":0,"allocated":2174976,"data_stored":1097855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.39400000000000002,"15min":0.38700000000000001},"min":{"1min":0.27400000000000002,"5min":0.25700000000000001,"15min":0.248},"max":{"1min":0.54400000000000004,"5min":0.61199999999999999,"15min":0.621},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.41199999999999998,"15min":0.40500000000000003},"min":{"1min":0.27100000000000002,"5min":0.253,"15min":0.24099999999999999},"max":{"1min":0.51400000000000001,"5min":0.66000000000000003,"15min":0.67100000000000004},"last":0.27700000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.42099999999999999,"15min":0.41299999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.58999999999999997,"5min":0.64100000000000001,"15min":0.64900000000000002},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.443,"15min":0.441},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.56699999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.28999999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.5,"15min":0.49299999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.629,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.503,"15min":0.501},"min":{"1min":0.38,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.66200000000000003,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.40400000000000003}]},{"osd":4,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65800000000000003,"15min":0.63600000000000001},"min":{"1min":0.49199999999999999,"5min":0.46400000000000002,"15min":0.34200000000000003},"max":{"1min":0.86599999999999999,"5min":0.88400000000000001,"15min":0.96799999999999997},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.66100000000000003,"15min":0.65700000000000003},"min":{"1min":0.47299999999999998,"5min":0.436,"15min":0.436},"max":{"1min":0.76200000000000001,"5min":0.97199999999999998,"15min":0.97299999999999998},"last":0.53300000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69199999999999995,"15min":0.68200000000000005},"min":{"1min":0.48299999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.89400000000000002,"5min":0.97699999999999998,"15min":1.024},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.71299999999999997,"15min":0.70099999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.41999999999999998},"max":{"1min":0.81399999999999995,"5min":0.98699999999999999,"15min":1.0069999999999999},"last":0.58699999999999997}]},{"osd":6,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.68700000000000006,"15min":0.67200000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45200000000000001},"max":{"1min":0.79500000000000004,"5min":1.0149999999999999,"15min":1.0569999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.69299999999999995,"15min":0.67900000000000005},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.38900000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.98899999999999999},"last":0.52000000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.68600000000000005,"15min":0.68000000000000005},"min":{"1min":0.50800000000000001,"5min":0.47599999999999998,"15min":0.42199999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":1.0009999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.72299999999999998,"15min":0.70299999999999996},"min":{"1min":0.495,"5min":0.47999999999999998,"15min":0.46800000000000003},"max":{"1min":0.88,"5min":1.028,"15min":1.028},"last":0.495}]}]},{"osd":4,"up_from":13,"seq":55834575761,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41440,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330400,"statfs":{"total":96636764160,"available":96594329600,"internally_reserved":0,"allocated":2174976,"data_stored":1097855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59399999999999997,"15min":0.59999999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.745,"5min":0.78900000000000003,"15min":1.2609999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.45800000000000002,"5min":0.40699999999999997,"15min":0.376},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.434}]},{"osd":1,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64700000000000002,"15min":0.64900000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.35199999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.86699999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.64000000000000001},"min":{"1min":0.45100000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.78100000000000003,"5min":0.79300000000000004,"15min":0.82499999999999996},"last":0.47299999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59599999999999997,"15min":0.60699999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.33600000000000002},"max":{"1min":0.81399999999999995,"5min":0.82399999999999995,"15min":1.5109999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.622,"15min":0.623},"min":{"1min":0.436,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.64600000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.60399999999999998,"15min":0.60299999999999998},"min":{"1min":0.39100000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73399999999999999,"5min":0.78600000000000003,"15min":1.1439999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63300000000000001,"15min":0.63400000000000001},"min":{"1min":0.38800000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":1.319},"last":0.629}]},{"osd":5,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48899999999999999,"15min":0.504},"min":{"1min":0.35199999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66900000000000004,"5min":0.748,"15min":0.878},"last":0.495},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51800000000000002,"15min":0.52200000000000002},"min":{"1min":0.38300000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":1.3839999999999999},"last":0.55700000000000005}]},{"osd":6,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.308},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":1.6850000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40500000000000003,"5min":0.39900000000000002,"15min":0.36699999999999999},"max":{"1min":0.755,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.53700000000000003}]},{"osd":7,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.374,"5min":0.36899999999999999,"15min":0.35099999999999998},"max":{"1min":0.79000000000000004,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.61399999999999999,"15min":0.60299999999999998},"min":{"1min":0.46999999999999997,"5min":0.41499999999999998,"15min":0.35799999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.7410000000000001},"last":0.67000000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575761,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":59488,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312352,"statfs":{"total":96636764160,"available":96575848448,"internally_reserved":0,"allocated":2170880,"data_stored":1096836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59899999999999998,"15min":0.59199999999999997},"min":{"1min":0.46700000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.75900000000000001,"5min":0.79100000000000004,"15min":0.81299999999999994},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60299999999999998,"15min":0.59499999999999997},"min":{"1min":0.39300000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81899999999999995,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.56599999999999995}]},{"osd":1,"last update":"Sat Apr 6 13:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.60399999999999998,"15min":0.59499999999999997},"min":{"1min":0.45000000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.76400000000000001,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58899999999999997,"15min":0.59199999999999997},"min":{"1min":0.36899999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78700000000000003},"last":0.59899999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60899999999999999,"15min":0.60599999999999998},"min":{"1min":0.433,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73599999999999999,"5min":0.80300000000000005,"15min":0.81999999999999995},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59999999999999998,"15min":0.59599999999999997},"min":{"1min":0.48199999999999998,"5min":0.38600000000000001,"15min":0.35899999999999999},"max":{"1min":0.72699999999999998,"5min":0.76200000000000001,"15min":0.76600000000000001},"last":0.68600000000000005}]},{"osd":3,"last update":"Sat Apr 6 13:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61599999999999999,"15min":0.60599999999999998},"min":{"1min":0.45000000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.63200000000000001,"15min":0.625},"min":{"1min":0.52600000000000002,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.80100000000000005,"5min":0.82599999999999996,"15min":0.82899999999999996},"last":0.74299999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.502,"15min":0.499},"min":{"1min":0.373,"5min":0.33000000000000002,"15min":0.32100000000000001},"max":{"1min":0.66300000000000003,"5min":0.72499999999999998,"15min":0.85199999999999998},"last":0.503},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.56000000000000005,"15min":0.55300000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.29099999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.627}]},{"osd":6,"last update":"Sat Apr 6 13:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.51200000000000001,"15min":0.504},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.29699999999999999},"max":{"1min":0.67300000000000004,"5min":0.80000000000000004,"15min":0.82699999999999996},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.52100000000000002,"15min":0.50600000000000001},"min":{"1min":0.36599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.80300000000000005},"last":0.70299999999999996}]},{"osd":7,"last update":"Sat Apr 6 13:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.53300000000000003,"15min":0.52200000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.314},"max":{"1min":0.68600000000000005,"5min":0.78300000000000003,"15min":0.80900000000000005},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.63100000000000001,"15min":0.61799999999999999},"min":{"1min":0.42099999999999999,"5min":0.41599999999999998,"15min":0.376},"max":{"1min":0.79700000000000004,"5min":0.84599999999999997,"15min":0.84699999999999998},"last":0.71599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:28:44.979 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:28:45.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.136+0000 7f3d54646700 1 Processor -- start 2024-04-06T13:28:45.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.136+0000 7f3d54646700 1 -- start start 2024-04-06T13:28:45.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.137+0000 7f3d54646700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3d4c146dd0 0x7f3d4c06d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:45.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.137+0000 7f3d54646700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3d4c0cedf0 0x7f3d4c0cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:45.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.137+0000 7f3d54646700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d4c0cf730 con 0x7f3d4c0d17a0 2024-04-06T13:28:45.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.137+0000 7f3d54646700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d4c06beb0 con 0x7f3d4c146dd0 2024-04-06T13:28:45.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.137+0000 7f3d54646700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d4c06c0a0 con 0x7f3d4c0cedf0 2024-04-06T13:28:45.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.137+0000 7f3d51be1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3d4c146dd0 0x7f3d4c06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:45.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.137+0000 7f3d51be1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3d4c146dd0 0x7f3d4c06d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59060/0 (socket says 172.21.15.189:59060) 2024-04-06T13:28:45.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.137+0000 7f3d51be1700 1 -- 172.21.15.189:0/547416884 learned_addr learned my addr 172.21.15.189:0/547416884 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:45.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.137+0000 7f3d52be3700 1 --2- 172.21.15.189:0/547416884 >> v2:172.21.15.192:3300/0 conn(0x7f3d4c0cedf0 0x7f3d4c0cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:45.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.137+0000 7f3d51be1700 1 -- 172.21.15.189:0/547416884 >> v2:172.21.15.192:3300/0 conn(0x7f3d4c0cedf0 msgr2=0x7f3d4c0cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:45.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.137+0000 7f3d51be1700 1 --2- 172.21.15.189:0/547416884 >> v2:172.21.15.192:3300/0 conn(0x7f3d4c0cedf0 0x7f3d4c0cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.137+0000 7f3d51be1700 1 -- 172.21.15.189:0/547416884 >> v1:172.21.15.192:6789/0 conn(0x7f3d4c0d17a0 legacy=0x7f3d4c06b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:45.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.137+0000 7f3d51be1700 1 -- 172.21.15.189:0/547416884 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4c06c380 con 0x7f3d4c146dd0 2024-04-06T13:28:45.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.137+0000 7f3d51be1700 1 --2- 172.21.15.189:0/547416884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d4c146dd0 0x7f3d4c06d140 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c160fb0 tx=0x7f3d3c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec437a74d353b51 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:45.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.137+0000 7f3d513e0700 1 -- 172.21.15.189:0/547416884 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c00b870 con 0x7f3d4c146dd0 2024-04-06T13:28:45.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.137+0000 7f3d513e0700 1 -- 172.21.15.189:0/547416884 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d3c00b9d0 con 0x7f3d4c146dd0 2024-04-06T13:28:45.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.137+0000 7f3d513e0700 1 -- 172.21.15.189:0/547416884 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c00bd60 con 0x7f3d4c146dd0 2024-04-06T13:28:45.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.138+0000 7f3d54646700 1 -- 172.21.15.189:0/547416884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d4c146dd0 msgr2=0x7f3d4c06d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:45.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.138+0000 7f3d54646700 1 --2- 172.21.15.189:0/547416884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d4c146dd0 0x7f3d4c06d140 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c160fb0 tx=0x7f3d3c009bd0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.138+0000 7f3d54646700 1 -- 172.21.15.189:0/547416884 shutdown_connections 2024-04-06T13:28:45.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.138+0000 7f3d54646700 1 --2- 172.21.15.189:0/547416884 >> v2:172.21.15.192:3300/0 conn(0x7f3d4c0cedf0 0x7f3d4c0cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.138+0000 7f3d54646700 1 --2- 172.21.15.189:0/547416884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d4c146dd0 0x7f3d4c06d140 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.138+0000 7f3d54646700 1 -- 172.21.15.189:0/547416884 >> 172.21.15.189:0/547416884 conn(0x7f3d4c0b2db0 msgr2=0x7f3d4c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:45.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.138+0000 7f3d54646700 1 -- 172.21.15.189:0/547416884 shutdown_connections 2024-04-06T13:28:45.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.138+0000 7f3d54646700 1 -- 172.21.15.189:0/547416884 wait complete. 2024-04-06T13:28:45.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.139+0000 7f3d54646700 1 Processor -- start 2024-04-06T13:28:45.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.139+0000 7f3d54646700 1 -- start start 2024-04-06T13:28:45.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.140+0000 7f3d54646700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d4c0cedf0 0x7f3d4c0b5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:45.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.140+0000 7f3d54646700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3d4c0d17a0 0x7f3d4c0b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:45.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.140+0000 7f3d54646700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d4c146dd0 0x7f3d4c1f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:45.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.140+0000 7f3d54646700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c1f65e0 con 0x7f3d4c146dd0 2024-04-06T13:28:45.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.140+0000 7f3d54646700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c15a6a0 con 0x7f3d4c0cedf0 2024-04-06T13:28:45.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.140+0000 7f3d54646700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c0b0720 con 0x7f3d4c0d17a0 2024-04-06T13:28:45.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.140+0000 7f3d52be3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d4c146dd0 0x7f3d4c1f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:45.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.140+0000 7f3d52be3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d4c146dd0 0x7f3d4c1f2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59070/0 (socket says 172.21.15.189:59070) 2024-04-06T13:28:45.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.140+0000 7f3d52be3700 1 -- 172.21.15.189:0/3121942819 learned_addr learned my addr 172.21.15.189:0/3121942819 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:45.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.140+0000 7f3d51be1700 1 --2- 172.21.15.189:0/3121942819 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3d4c0d17a0 0x7f3d4c0b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:45.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.140+0000 7f3d52be3700 1 -- 172.21.15.189:0/3121942819 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3d4c0d17a0 msgr2=0x7f3d4c0b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:45.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.140+0000 7f3d523e2700 1 --2- 172.21.15.189:0/3121942819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d4c0cedf0 0x7f3d4c0b5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:45.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.140+0000 7f3d52be3700 1 --2- 172.21.15.189:0/3121942819 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3d4c0d17a0 0x7f3d4c0b01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.140+0000 7f3d52be3700 1 -- 172.21.15.189:0/3121942819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d4c0cedf0 msgr2=0x7f3d4c0b5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:45.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.140+0000 7f3d52be3700 1 --2- 172.21.15.189:0/3121942819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d4c0cedf0 0x7f3d4c0b5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.140+0000 7f3d52be3700 1 -- 172.21.15.189:0/3121942819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3c009550 con 0x7f3d4c146dd0 2024-04-06T13:28:45.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.141+0000 7f3d52be3700 1 --2- 172.21.15.189:0/3121942819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d4c146dd0 0x7f3d4c1f2c30 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f3d44007e60 tx=0x7f3d4400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:45.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.141+0000 7f3d42ffd700 1 -- 172.21.15.189:0/3121942819 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4400ede0 con 0x7f3d4c146dd0 2024-04-06T13:28:45.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.141+0000 7f3d42ffd700 1 -- 172.21.15.189:0/3121942819 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d44014420 con 0x7f3d4c146dd0 2024-04-06T13:28:45.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.141+0000 7f3d42ffd700 1 -- 172.21.15.189:0/3121942819 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d440147d0 con 0x7f3d4c146dd0 2024-04-06T13:28:45.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.141+0000 7f3d54646700 1 -- 172.21.15.189:0/3121942819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d4c1f6240 con 0x7f3d4c146dd0 2024-04-06T13:28:45.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.141+0000 7f3d54646700 1 -- 172.21.15.189:0/3121942819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d4c1f37f0 con 0x7f3d4c146dd0 2024-04-06T13:28:45.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.142+0000 7f3d54646700 1 -- 172.21.15.189:0/3121942819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d4c1f6720 con 0x7f3d4c146dd0 2024-04-06T13:28:45.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.142+0000 7f3d42ffd700 1 -- 172.21.15.189:0/3121942819 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f3d44014930 con 0x7f3d4c146dd0 2024-04-06T13:28:45.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.142+0000 7f3d42ffd700 1 --2- 172.21.15.189:0/3121942819 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3d3006adb0 0x7f3d3006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:45.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.143+0000 7f3d42ffd700 1 -- 172.21.15.189:0/3121942819 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3d44089d90 con 0x7f3d4c146dd0 2024-04-06T13:28:45.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.143+0000 7f3d523e2700 1 --2- 172.21.15.189:0/3121942819 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3d3006adb0 0x7f3d3006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:45.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.143+0000 7f3d523e2700 1 --2- 172.21.15.189:0/3121942819 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3d3006adb0 0x7f3d3006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3d48000f60 tx=0x7f3d48009040 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:45.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.147+0000 7f3d42ffd700 1 -- 172.21.15.189:0/3121942819 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3d4c1f6720 con 0x7f3d4c146dd0 2024-04-06T13:28:45.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.338+0000 7f3d54646700 1 -- 172.21.15.189:0/3121942819 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3d4c06cc20 con 0x7f3d3006adb0 2024-04-06T13:28:45.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.345+0000 7f3d42ffd700 1 -- 172.21.15.189:0/3121942819 <== mgr.8614 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290432 (secure 0 0 0) 0x7f3d480050d0 con 0x7f3d3006adb0 2024-04-06T13:28:45.350 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:45.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.349+0000 7f3d54646700 1 -- 172.21.15.189:0/3121942819 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3d3006adb0 msgr2=0x7f3d3006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.349+0000 7f3d54646700 1 --2- 172.21.15.189:0/3121942819 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3d3006adb0 0x7f3d3006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3d48000f60 tx=0x7f3d48009040 comp rx=0 tx=0).stop 2024-04-06T13:28:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.349+0000 7f3d54646700 1 -- 172.21.15.189:0/3121942819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d4c146dd0 msgr2=0x7f3d4c1f2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.349+0000 7f3d54646700 1 --2- 172.21.15.189:0/3121942819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d4c146dd0 0x7f3d4c1f2c30 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f3d44007e60 tx=0x7f3d4400e3c0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.349+0000 7f3d54646700 1 -- 172.21.15.189:0/3121942819 shutdown_connections 2024-04-06T13:28:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.349+0000 7f3d54646700 1 --2- 172.21.15.189:0/3121942819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d4c146dd0 0x7f3d4c1f2c30 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.349+0000 7f3d54646700 1 --2- 172.21.15.189:0/3121942819 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f3d3006adb0 0x7f3d3006d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.349+0000 7f3d54646700 1 --2- 172.21.15.189:0/3121942819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d4c0cedf0 0x7f3d4c0b5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.349+0000 7f3d54646700 1 --2- 172.21.15.189:0/3121942819 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3d4c0d17a0 0x7f3d4c0b01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.349+0000 7f3d54646700 1 -- 172.21.15.189:0/3121942819 >> 172.21.15.189:0/3121942819 conn(0x7f3d4c0b2db0 msgr2=0x7f3d4c15dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.349+0000 7f3d54646700 1 -- 172.21.15.189:0/3121942819 shutdown_connections 2024-04-06T13:28:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.349+0000 7f3d54646700 1 -- 172.21.15.189:0/3121942819 wait complete. 2024-04-06T13:28:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:28:45.362 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-06T13:28:45.289565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":483,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":126,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":380684,"kb_used_data":18580,"kb_used_omap":12,"kb_used_meta":361971,"kb_avail":754594036,"statfs":{"total":773094113280,"available":772704292864,"internally_reserved":0,"allocated":19025920,"data_stored":10176189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":370658872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.894117"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229647+0000","last_change":"2024-04-06T13:28:40.841070+0000","last_active":"2024-04-06T13:28:42.229647+0000","last_peered":"2024-04-06T13:28:42.229647+0000","last_clean":"2024-04-06T13:28:42.229647+0000","last_became_active":"2024-04-06T13:28:40.840767+0000","last_became_peered":"2024-04-06T13:28:40.840767+0000","last_unstale":"2024-04-06T13:28:42.229647+0000","last_undegraded":"2024-04-06T13:28:42.229647+0000","last_fullsized":"2024-04-06T13:28:42.229647+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:32:45.964316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823295+0000","last_change":"2024-04-06T13:28:40.844338+0000","last_active":"2024-04-06T13:28:41.823295+0000","last_peered":"2024-04-06T13:28:41.823295+0000","last_clean":"2024-04-06T13:28:41.823295+0000","last_became_active":"2024-04-06T13:28:40.844116+0000","last_became_peered":"2024-04-06T13:28:40.844116+0000","last_unstale":"2024-04-06T13:28:41.823295+0000","last_undegraded":"2024-04-06T13:28:41.823295+0000","last_fullsized":"2024-04-06T13:28:41.823295+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:40:55.218858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823252+0000","last_change":"2024-04-06T13:28:40.830807+0000","last_active":"2024-04-06T13:28:41.823252+0000","last_peered":"2024-04-06T13:28:41.823252+0000","last_clean":"2024-04-06T13:28:41.823252+0000","last_became_active":"2024-04-06T13:28:40.830567+0000","last_became_peered":"2024-04-06T13:28:40.830567+0000","last_unstale":"2024-04-06T13:28:41.823252+0000","last_undegraded":"2024-04-06T13:28:41.823252+0000","last_fullsized":"2024-04-06T13:28:41.823252+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:49:30.966991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228837+0000","last_change":"2024-04-06T13:28:40.843798+0000","last_active":"2024-04-06T13:28:42.228837+0000","last_peered":"2024-04-06T13:28:42.228837+0000","last_clean":"2024-04-06T13:28:42.228837+0000","last_became_active":"2024-04-06T13:28:40.843564+0000","last_became_peered":"2024-04-06T13:28:40.843564+0000","last_unstale":"2024-04-06T13:28:42.228837+0000","last_undegraded":"2024-04-06T13:28:42.228837+0000","last_fullsized":"2024-04-06T13:28:42.228837+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:21:54.204666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.824532+0000","last_change":"2024-04-06T13:28:40.832880+0000","last_active":"2024-04-06T13:28:41.824532+0000","last_peered":"2024-04-06T13:28:41.824532+0000","last_clean":"2024-04-06T13:28:41.824532+0000","last_became_active":"2024-04-06T13:28:40.832613+0000","last_became_peered":"2024-04-06T13:28:40.832613+0000","last_unstale":"2024-04-06T13:28:41.824532+0000","last_undegraded":"2024-04-06T13:28:41.824532+0000","last_fullsized":"2024-04-06T13:28:41.824532+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:39:50.716990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823921+0000","last_change":"2024-04-06T13:28:40.830608+0000","last_active":"2024-04-06T13:28:41.823921+0000","last_peered":"2024-04-06T13:28:41.823921+0000","last_clean":"2024-04-06T13:28:41.823921+0000","last_became_active":"2024-04-06T13:28:40.830343+0000","last_became_peered":"2024-04-06T13:28:40.830343+0000","last_unstale":"2024-04-06T13:28:41.823921+0000","last_undegraded":"2024-04-06T13:28:41.823921+0000","last_fullsized":"2024-04-06T13:28:41.823921+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:57.479394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228894+0000","last_change":"2024-04-06T13:28:40.831344+0000","last_active":"2024-04-06T13:28:42.228894+0000","last_peered":"2024-04-06T13:28:42.228894+0000","last_clean":"2024-04-06T13:28:42.228894+0000","last_became_active":"2024-04-06T13:28:40.830972+0000","last_became_peered":"2024-04-06T13:28:40.830972+0000","last_unstale":"2024-04-06T13:28:42.228894+0000","last_undegraded":"2024-04-06T13:28:42.228894+0000","last_fullsized":"2024-04-06T13:28:42.228894+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:21:54.204666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229816+0000","last_change":"2024-04-06T13:28:40.830062+0000","last_active":"2024-04-06T13:28:42.229816+0000","last_peered":"2024-04-06T13:28:42.229816+0000","last_clean":"2024-04-06T13:28:42.229816+0000","last_became_active":"2024-04-06T13:28:40.829316+0000","last_became_peered":"2024-04-06T13:28:40.829316+0000","last_unstale":"2024-04-06T13:28:42.229816+0000","last_undegraded":"2024-04-06T13:28:42.229816+0000","last_fullsized":"2024-04-06T13:28:42.229816+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:11:48.235841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.824691+0000","last_change":"2024-04-06T13:28:40.830062+0000","last_active":"2024-04-06T13:28:41.824691+0000","last_peered":"2024-04-06T13:28:41.824691+0000","last_clean":"2024-04-06T13:28:41.824691+0000","last_became_active":"2024-04-06T13:28:40.829821+0000","last_became_peered":"2024-04-06T13:28:40.829821+0000","last_unstale":"2024-04-06T13:28:41.824691+0000","last_undegraded":"2024-04-06T13:28:41.824691+0000","last_fullsized":"2024-04-06T13:28:41.824691+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:46:34.113622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.824673+0000","last_change":"2024-04-06T13:28:40.828571+0000","last_active":"2024-04-06T13:28:41.824673+0000","last_peered":"2024-04-06T13:28:41.824673+0000","last_clean":"2024-04-06T13:28:41.824673+0000","last_became_active":"2024-04-06T13:28:40.828311+0000","last_became_peered":"2024-04-06T13:28:40.828311+0000","last_unstale":"2024-04-06T13:28:41.824673+0000","last_undegraded":"2024-04-06T13:28:41.824673+0000","last_fullsized":"2024-04-06T13:28:41.824673+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:18:08.870534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228605+0000","last_change":"2024-04-06T13:28:40.831798+0000","last_active":"2024-04-06T13:28:42.228605+0000","last_peered":"2024-04-06T13:28:42.228605+0000","last_clean":"2024-04-06T13:28:42.228605+0000","last_became_active":"2024-04-06T13:28:40.831163+0000","last_became_peered":"2024-04-06T13:28:40.831163+0000","last_unstale":"2024-04-06T13:28:42.228605+0000","last_undegraded":"2024-04-06T13:28:42.228605+0000","last_fullsized":"2024-04-06T13:28:42.228605+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:44:23.562469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228935+0000","last_change":"2024-04-06T13:28:40.844570+0000","last_active":"2024-04-06T13:28:42.228935+0000","last_peered":"2024-04-06T13:28:42.228935+0000","last_clean":"2024-04-06T13:28:42.228935+0000","last_became_active":"2024-04-06T13:28:40.843965+0000","last_became_peered":"2024-04-06T13:28:40.843965+0000","last_unstale":"2024-04-06T13:28:42.228935+0000","last_undegraded":"2024-04-06T13:28:42.228935+0000","last_fullsized":"2024-04-06T13:28:42.228935+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:12.037450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.824446+0000","last_change":"2024-04-06T13:28:40.832838+0000","last_active":"2024-04-06T13:28:41.824446+0000","last_peered":"2024-04-06T13:28:41.824446+0000","last_clean":"2024-04-06T13:28:41.824446+0000","last_became_active":"2024-04-06T13:28:40.832531+0000","last_became_peered":"2024-04-06T13:28:40.832531+0000","last_unstale":"2024-04-06T13:28:41.824446+0000","last_undegraded":"2024-04-06T13:28:41.824446+0000","last_fullsized":"2024-04-06T13:28:41.824446+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:02:46.457277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229414+0000","last_change":"2024-04-06T13:28:40.843876+0000","last_active":"2024-04-06T13:28:42.229414+0000","last_peered":"2024-04-06T13:28:42.229414+0000","last_clean":"2024-04-06T13:28:42.229414+0000","last_became_active":"2024-04-06T13:28:40.843424+0000","last_became_peered":"2024-04-06T13:28:40.843424+0000","last_unstale":"2024-04-06T13:28:42.229414+0000","last_undegraded":"2024-04-06T13:28:42.229414+0000","last_fullsized":"2024-04-06T13:28:42.229414+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:33:14.851459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823039+0000","last_change":"2024-04-06T13:28:40.842233+0000","last_active":"2024-04-06T13:28:41.823039+0000","last_peered":"2024-04-06T13:28:41.823039+0000","last_clean":"2024-04-06T13:28:41.823039+0000","last_became_active":"2024-04-06T13:28:40.841786+0000","last_became_peered":"2024-04-06T13:28:40.841786+0000","last_unstale":"2024-04-06T13:28:41.823039+0000","last_undegraded":"2024-04-06T13:28:41.823039+0000","last_fullsized":"2024-04-06T13:28:41.823039+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:00:38.061764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823713+0000","last_change":"2024-04-06T13:28:40.835775+0000","last_active":"2024-04-06T13:28:41.823713+0000","last_peered":"2024-04-06T13:28:41.823713+0000","last_clean":"2024-04-06T13:28:41.823713+0000","last_became_active":"2024-04-06T13:28:40.835497+0000","last_became_peered":"2024-04-06T13:28:40.835497+0000","last_unstale":"2024-04-06T13:28:41.823713+0000","last_undegraded":"2024-04-06T13:28:41.823713+0000","last_fullsized":"2024-04-06T13:28:41.823713+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:44:42.919189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229105+0000","last_change":"2024-04-06T13:28:40.844021+0000","last_active":"2024-04-06T13:28:42.229105+0000","last_peered":"2024-04-06T13:28:42.229105+0000","last_clean":"2024-04-06T13:28:42.229105+0000","last_became_active":"2024-04-06T13:28:40.841545+0000","last_became_peered":"2024-04-06T13:28:40.841545+0000","last_unstale":"2024-04-06T13:28:42.229105+0000","last_undegraded":"2024-04-06T13:28:42.229105+0000","last_fullsized":"2024-04-06T13:28:42.229105+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:50:27.060357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228926+0000","last_change":"2024-04-06T13:28:40.832719+0000","last_active":"2024-04-06T13:28:42.228926+0000","last_peered":"2024-04-06T13:28:42.228926+0000","last_clean":"2024-04-06T13:28:42.228926+0000","last_became_active":"2024-04-06T13:28:40.832481+0000","last_became_peered":"2024-04-06T13:28:40.832481+0000","last_unstale":"2024-04-06T13:28:42.228926+0000","last_undegraded":"2024-04-06T13:28:42.228926+0000","last_fullsized":"2024-04-06T13:28:42.228926+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:17:00.268284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.825006+0000","last_change":"2024-04-06T13:28:40.840499+0000","last_active":"2024-04-06T13:28:41.825006+0000","last_peered":"2024-04-06T13:28:41.825006+0000","last_clean":"2024-04-06T13:28:41.825006+0000","last_became_active":"2024-04-06T13:28:40.839816+0000","last_became_peered":"2024-04-06T13:28:40.839816+0000","last_unstale":"2024-04-06T13:28:41.825006+0000","last_undegraded":"2024-04-06T13:28:41.825006+0000","last_fullsized":"2024-04-06T13:28:41.825006+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:33:32.596787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228321+0000","last_change":"2024-04-06T13:28:40.833785+0000","last_active":"2024-04-06T13:28:42.228321+0000","last_peered":"2024-04-06T13:28:42.228321+0000","last_clean":"2024-04-06T13:28:42.228321+0000","last_became_active":"2024-04-06T13:28:40.833497+0000","last_became_peered":"2024-04-06T13:28:40.833497+0000","last_unstale":"2024-04-06T13:28:42.228321+0000","last_undegraded":"2024-04-06T13:28:42.228321+0000","last_fullsized":"2024-04-06T13:28:42.228321+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:10.713595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.824362+0000","last_change":"2024-04-06T13:28:40.831975+0000","last_active":"2024-04-06T13:28:41.824362+0000","last_peered":"2024-04-06T13:28:41.824362+0000","last_clean":"2024-04-06T13:28:41.824362+0000","last_became_active":"2024-04-06T13:28:40.829627+0000","last_became_peered":"2024-04-06T13:28:40.829627+0000","last_unstale":"2024-04-06T13:28:41.824362+0000","last_undegraded":"2024-04-06T13:28:41.824362+0000","last_fullsized":"2024-04-06T13:28:41.824362+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:49:40.834156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228822+0000","last_change":"2024-04-06T13:28:40.837461+0000","last_active":"2024-04-06T13:28:42.228822+0000","last_peered":"2024-04-06T13:28:42.228822+0000","last_clean":"2024-04-06T13:28:42.228822+0000","last_became_active":"2024-04-06T13:28:40.836362+0000","last_became_peered":"2024-04-06T13:28:40.836362+0000","last_unstale":"2024-04-06T13:28:42.228822+0000","last_undegraded":"2024-04-06T13:28:42.228822+0000","last_fullsized":"2024-04-06T13:28:42.228822+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:18:26.747951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229628+0000","last_change":"2024-04-06T13:28:40.834811+0000","last_active":"2024-04-06T13:28:42.229628+0000","last_peered":"2024-04-06T13:28:42.229628+0000","last_clean":"2024-04-06T13:28:42.229628+0000","last_became_active":"2024-04-06T13:28:40.834546+0000","last_became_peered":"2024-04-06T13:28:40.834546+0000","last_unstale":"2024-04-06T13:28:42.229628+0000","last_undegraded":"2024-04-06T13:28:42.229628+0000","last_fullsized":"2024-04-06T13:28:42.229628+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:16.736960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228122+0000","last_change":"2024-04-06T13:28:40.827058+0000","last_active":"2024-04-06T13:28:42.228122+0000","last_peered":"2024-04-06T13:28:42.228122+0000","last_clean":"2024-04-06T13:28:42.228122+0000","last_became_active":"2024-04-06T13:28:40.826834+0000","last_became_peered":"2024-04-06T13:28:40.826834+0000","last_unstale":"2024-04-06T13:28:42.228122+0000","last_undegraded":"2024-04-06T13:28:42.228122+0000","last_fullsized":"2024-04-06T13:28:42.228122+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:25:10.763749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229756+0000","last_change":"2024-04-06T13:28:40.838187+0000","last_active":"2024-04-06T13:28:42.229756+0000","last_peered":"2024-04-06T13:28:42.229756+0000","last_clean":"2024-04-06T13:28:42.229756+0000","last_became_active":"2024-04-06T13:28:40.837915+0000","last_became_peered":"2024-04-06T13:28:40.837915+0000","last_unstale":"2024-04-06T13:28:42.229756+0000","last_undegraded":"2024-04-06T13:28:42.229756+0000","last_fullsized":"2024-04-06T13:28:42.229756+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:12:29.166946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.230132+0000","last_change":"2024-04-06T13:28:40.837274+0000","last_active":"2024-04-06T13:28:42.230132+0000","last_peered":"2024-04-06T13:28:42.230132+0000","last_clean":"2024-04-06T13:28:42.230132+0000","last_became_active":"2024-04-06T13:28:40.837020+0000","last_became_peered":"2024-04-06T13:28:40.837020+0000","last_unstale":"2024-04-06T13:28:42.230132+0000","last_undegraded":"2024-04-06T13:28:42.230132+0000","last_fullsized":"2024-04-06T13:28:42.230132+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:44:35.610303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.824642+0000","last_change":"2024-04-06T13:28:40.828197+0000","last_active":"2024-04-06T13:28:41.824642+0000","last_peered":"2024-04-06T13:28:41.824642+0000","last_clean":"2024-04-06T13:28:41.824642+0000","last_became_active":"2024-04-06T13:28:40.827992+0000","last_became_peered":"2024-04-06T13:28:40.827992+0000","last_unstale":"2024-04-06T13:28:41.824642+0000","last_undegraded":"2024-04-06T13:28:41.824642+0000","last_fullsized":"2024-04-06T13:28:41.824642+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:52:36.015609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228509+0000","last_change":"2024-04-06T13:28:40.835375+0000","last_active":"2024-04-06T13:28:42.228509+0000","last_peered":"2024-04-06T13:28:42.228509+0000","last_clean":"2024-04-06T13:28:42.228509+0000","last_became_active":"2024-04-06T13:28:40.835085+0000","last_became_peered":"2024-04-06T13:28:40.835085+0000","last_unstale":"2024-04-06T13:28:42.228509+0000","last_undegraded":"2024-04-06T13:28:42.228509+0000","last_fullsized":"2024-04-06T13:28:42.228509+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:52:39.138833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.230050+0000","last_change":"2024-04-06T13:28:40.837856+0000","last_active":"2024-04-06T13:28:42.230050+0000","last_peered":"2024-04-06T13:28:42.230050+0000","last_clean":"2024-04-06T13:28:42.230050+0000","last_became_active":"2024-04-06T13:28:40.837563+0000","last_became_peered":"2024-04-06T13:28:40.837563+0000","last_unstale":"2024-04-06T13:28:42.230050+0000","last_undegraded":"2024-04-06T13:28:42.230050+0000","last_fullsized":"2024-04-06T13:28:42.230050+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:49:30.966991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228202+0000","last_change":"2024-04-06T13:28:40.844690+0000","last_active":"2024-04-06T13:28:42.228202+0000","last_peered":"2024-04-06T13:28:42.228202+0000","last_clean":"2024-04-06T13:28:42.228202+0000","last_became_active":"2024-04-06T13:28:40.844454+0000","last_became_peered":"2024-04-06T13:28:40.844454+0000","last_unstale":"2024-04-06T13:28:42.228202+0000","last_undegraded":"2024-04-06T13:28:42.228202+0000","last_fullsized":"2024-04-06T13:28:42.228202+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:56:20.132200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823819+0000","last_change":"2024-04-06T13:28:40.827755+0000","last_active":"2024-04-06T13:28:41.823819+0000","last_peered":"2024-04-06T13:28:41.823819+0000","last_clean":"2024-04-06T13:28:41.823819+0000","last_became_active":"2024-04-06T13:28:40.827090+0000","last_became_peered":"2024-04-06T13:28:40.827090+0000","last_unstale":"2024-04-06T13:28:41.823819+0000","last_undegraded":"2024-04-06T13:28:41.823819+0000","last_fullsized":"2024-04-06T13:28:41.823819+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:23:46.286047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229632+0000","last_change":"2024-04-06T13:28:40.837123+0000","last_active":"2024-04-06T13:28:42.229632+0000","last_peered":"2024-04-06T13:28:42.229632+0000","last_clean":"2024-04-06T13:28:42.229632+0000","last_became_active":"2024-04-06T13:28:40.836861+0000","last_became_peered":"2024-04-06T13:28:40.836861+0000","last_unstale":"2024-04-06T13:28:42.229632+0000","last_undegraded":"2024-04-06T13:28:42.229632+0000","last_fullsized":"2024-04-06T13:28:42.229632+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:49:42.615169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823314+0000","last_change":"2024-04-06T13:28:40.830212+0000","last_active":"2024-04-06T13:28:41.823314+0000","last_peered":"2024-04-06T13:28:41.823314+0000","last_clean":"2024-04-06T13:28:41.823314+0000","last_became_active":"2024-04-06T13:28:40.829971+0000","last_became_peered":"2024-04-06T13:28:40.829971+0000","last_unstale":"2024-04-06T13:28:41.823314+0000","last_undegraded":"2024-04-06T13:28:41.823314+0000","last_fullsized":"2024-04-06T13:28:41.823314+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:11:48.235841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"177'4","reported_seq":15,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.824744+0000","last_change":"2024-04-06T13:28:40.414045+0000","last_active":"2024-04-06T13:28:41.824744+0000","last_peered":"2024-04-06T13:28:41.824744+0000","last_clean":"2024-04-06T13:28:41.824744+0000","last_became_active":"2024-04-06T13:28:40.413718+0000","last_became_peered":"2024-04-06T13:28:40.413718+0000","last_unstale":"2024-04-06T13:28:41.824744+0000","last_undegraded":"2024-04-06T13:28:41.824744+0000","last_fullsized":"2024-04-06T13:28:41.824744+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:25.246240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823305+0000","last_change":"2024-04-06T13:28:40.841928+0000","last_active":"2024-04-06T13:28:41.823305+0000","last_peered":"2024-04-06T13:28:41.823305+0000","last_clean":"2024-04-06T13:28:41.823305+0000","last_became_active":"2024-04-06T13:28:40.841704+0000","last_became_peered":"2024-04-06T13:28:40.841704+0000","last_unstale":"2024-04-06T13:28:41.823305+0000","last_undegraded":"2024-04-06T13:28:41.823305+0000","last_fullsized":"2024-04-06T13:28:41.823305+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:57:08.430945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"177'1","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.824744+0000","last_change":"2024-04-06T13:28:40.412559+0000","last_active":"2024-04-06T13:28:41.824744+0000","last_peered":"2024-04-06T13:28:41.824744+0000","last_clean":"2024-04-06T13:28:41.824744+0000","last_became_active":"2024-04-06T13:28:40.412260+0000","last_became_peered":"2024-04-06T13:28:40.412260+0000","last_unstale":"2024-04-06T13:28:41.824744+0000","last_undegraded":"2024-04-06T13:28:41.824744+0000","last_fullsized":"2024-04-06T13:28:41.824744+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:24.777990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.230501+0000","last_change":"2024-04-06T13:28:40.843699+0000","last_active":"2024-04-06T13:28:42.230501+0000","last_peered":"2024-04-06T13:28:42.230501+0000","last_clean":"2024-04-06T13:28:42.230501+0000","last_became_active":"2024-04-06T13:28:40.843420+0000","last_became_peered":"2024-04-06T13:28:40.843420+0000","last_unstale":"2024-04-06T13:28:42.230501+0000","last_undegraded":"2024-04-06T13:28:42.230501+0000","last_fullsized":"2024-04-06T13:28:42.230501+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:13:15.481226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"177'2","reported_seq":13,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.825068+0000","last_change":"2024-04-06T13:28:40.410683+0000","last_active":"2024-04-06T13:28:41.825068+0000","last_peered":"2024-04-06T13:28:41.825068+0000","last_clean":"2024-04-06T13:28:41.825068+0000","last_became_active":"2024-04-06T13:28:40.410390+0000","last_became_peered":"2024-04-06T13:28:40.410390+0000","last_unstale":"2024-04-06T13:28:41.825068+0000","last_undegraded":"2024-04-06T13:28:41.825068+0000","last_fullsized":"2024-04-06T13:28:41.825068+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:13:14.476145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.824735+0000","last_change":"2024-04-06T13:28:40.837565+0000","last_active":"2024-04-06T13:28:41.824735+0000","last_peered":"2024-04-06T13:28:41.824735+0000","last_clean":"2024-04-06T13:28:41.824735+0000","last_became_active":"2024-04-06T13:28:40.837296+0000","last_became_peered":"2024-04-06T13:28:40.837296+0000","last_unstale":"2024-04-06T13:28:41.824735+0000","last_undegraded":"2024-04-06T13:28:41.824735+0000","last_fullsized":"2024-04-06T13:28:41.824735+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:10:27.023865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"177'2","reported_seq":13,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823374+0000","last_change":"2024-04-06T13:28:40.416928+0000","last_active":"2024-04-06T13:28:41.823374+0000","last_peered":"2024-04-06T13:28:41.823374+0000","last_clean":"2024-04-06T13:28:41.823374+0000","last_became_active":"2024-04-06T13:28:40.416637+0000","last_became_peered":"2024-04-06T13:28:40.416637+0000","last_unstale":"2024-04-06T13:28:41.823374+0000","last_undegraded":"2024-04-06T13:28:41.823374+0000","last_fullsized":"2024-04-06T13:28:41.823374+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:32:44.959236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228174+0000","last_change":"2024-04-06T13:28:40.835222+0000","last_active":"2024-04-06T13:28:42.228174+0000","last_peered":"2024-04-06T13:28:42.228174+0000","last_clean":"2024-04-06T13:28:42.228174+0000","last_became_active":"2024-04-06T13:28:40.834960+0000","last_became_peered":"2024-04-06T13:28:40.834960+0000","last_unstale":"2024-04-06T13:28:42.228174+0000","last_undegraded":"2024-04-06T13:28:42.228174+0000","last_fullsized":"2024-04-06T13:28:42.228174+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:18:08.870534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"177'1","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.230154+0000","last_change":"2024-04-06T13:28:40.415119+0000","last_active":"2024-04-06T13:28:42.230154+0000","last_peered":"2024-04-06T13:28:42.230154+0000","last_clean":"2024-04-06T13:28:42.230154+0000","last_became_active":"2024-04-06T13:28:40.414799+0000","last_became_peered":"2024-04-06T13:28:40.414799+0000","last_unstale":"2024-04-06T13:28:42.230154+0000","last_undegraded":"2024-04-06T13:28:42.230154+0000","last_fullsized":"2024-04-06T13:28:42.230154+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:11.032370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228872+0000","last_change":"2024-04-06T13:28:40.845440+0000","last_active":"2024-04-06T13:28:42.228872+0000","last_peered":"2024-04-06T13:28:42.228872+0000","last_clean":"2024-04-06T13:28:42.228872+0000","last_became_active":"2024-04-06T13:28:40.841567+0000","last_became_peered":"2024-04-06T13:28:40.841567+0000","last_unstale":"2024-04-06T13:28:42.228872+0000","last_undegraded":"2024-04-06T13:28:42.228872+0000","last_fullsized":"2024-04-06T13:28:42.228872+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:12:29.166946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"177'1","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.824093+0000","last_change":"2024-04-06T13:28:40.418488+0000","last_active":"2024-04-06T13:28:41.824093+0000","last_peered":"2024-04-06T13:28:41.824093+0000","last_clean":"2024-04-06T13:28:41.824093+0000","last_became_active":"2024-04-06T13:28:40.418197+0000","last_became_peered":"2024-04-06T13:28:40.418197+0000","last_unstale":"2024-04-06T13:28:41.824093+0000","last_undegraded":"2024-04-06T13:28:41.824093+0000","last_fullsized":"2024-04-06T13:28:41.824093+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:38:55.807954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823070+0000","last_change":"2024-04-06T13:28:40.832830+0000","last_active":"2024-04-06T13:28:41.823070+0000","last_peered":"2024-04-06T13:28:41.823070+0000","last_clean":"2024-04-06T13:28:41.823070+0000","last_became_active":"2024-04-06T13:28:40.832566+0000","last_became_peered":"2024-04-06T13:28:40.832566+0000","last_unstale":"2024-04-06T13:28:41.823070+0000","last_undegraded":"2024-04-06T13:28:41.823070+0000","last_fullsized":"2024-04-06T13:28:41.823070+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:13:15.481226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228980+0000","last_change":"2024-04-06T13:28:40.414770+0000","last_active":"2024-04-06T13:28:42.228980+0000","last_peered":"2024-04-06T13:28:42.228980+0000","last_clean":"2024-04-06T13:28:42.228980+0000","last_became_active":"2024-04-06T13:28:40.414419+0000","last_became_peered":"2024-04-06T13:28:40.414419+0000","last_unstale":"2024-04-06T13:28:42.228980+0000","last_undegraded":"2024-04-06T13:28:42.228980+0000","last_fullsized":"2024-04-06T13:28:42.228980+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:25.499120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229765+0000","last_change":"2024-04-06T13:28:40.833411+0000","last_active":"2024-04-06T13:28:42.229765+0000","last_peered":"2024-04-06T13:28:42.229765+0000","last_clean":"2024-04-06T13:28:42.229765+0000","last_became_active":"2024-04-06T13:28:40.831004+0000","last_became_peered":"2024-04-06T13:28:40.831004+0000","last_unstale":"2024-04-06T13:28:42.229765+0000","last_undegraded":"2024-04-06T13:28:42.229765+0000","last_fullsized":"2024-04-06T13:28:42.229765+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:12.037450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823743+0000","last_change":"2024-04-06T13:28:40.413021+0000","last_active":"2024-04-06T13:28:41.823743+0000","last_peered":"2024-04-06T13:28:41.823743+0000","last_clean":"2024-04-06T13:28:41.823743+0000","last_became_active":"2024-04-06T13:28:40.412757+0000","last_became_peered":"2024-04-06T13:28:40.412757+0000","last_unstale":"2024-04-06T13:28:41.823743+0000","last_undegraded":"2024-04-06T13:28:41.823743+0000","last_fullsized":"2024-04-06T13:28:41.823743+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:02:03.234569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.230174+0000","last_change":"2024-04-06T13:28:40.833524+0000","last_active":"2024-04-06T13:28:42.230174+0000","last_peered":"2024-04-06T13:28:42.230174+0000","last_clean":"2024-04-06T13:28:42.230174+0000","last_became_active":"2024-04-06T13:28:40.833271+0000","last_became_peered":"2024-04-06T13:28:40.833271+0000","last_unstale":"2024-04-06T13:28:42.230174+0000","last_undegraded":"2024-04-06T13:28:42.230174+0000","last_fullsized":"2024-04-06T13:28:42.230174+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:29:15.092554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.824979+0000","last_change":"2024-04-06T13:28:40.412527+0000","last_active":"2024-04-06T13:28:41.824979+0000","last_peered":"2024-04-06T13:28:41.824979+0000","last_clean":"2024-04-06T13:28:41.824979+0000","last_became_active":"2024-04-06T13:28:40.411946+0000","last_became_peered":"2024-04-06T13:28:40.411946+0000","last_unstale":"2024-04-06T13:28:41.824979+0000","last_undegraded":"2024-04-06T13:28:41.824979+0000","last_fullsized":"2024-04-06T13:28:41.824979+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:59:43.608527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228534+0000","last_change":"2024-04-06T13:28:40.831953+0000","last_active":"2024-04-06T13:28:42.228534+0000","last_peered":"2024-04-06T13:28:42.228534+0000","last_clean":"2024-04-06T13:28:42.228534+0000","last_became_active":"2024-04-06T13:28:40.831529+0000","last_became_peered":"2024-04-06T13:28:40.831529+0000","last_unstale":"2024-04-06T13:28:42.228534+0000","last_undegraded":"2024-04-06T13:28:42.228534+0000","last_fullsized":"2024-04-06T13:28:42.228534+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:23:52.673261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229937+0000","last_change":"2024-04-06T13:28:40.422285+0000","last_active":"2024-04-06T13:28:42.229937+0000","last_peered":"2024-04-06T13:28:42.229937+0000","last_clean":"2024-04-06T13:28:42.229937+0000","last_became_active":"2024-04-06T13:28:40.422002+0000","last_became_peered":"2024-04-06T13:28:40.422002+0000","last_unstale":"2024-04-06T13:28:42.229937+0000","last_undegraded":"2024-04-06T13:28:42.229937+0000","last_fullsized":"2024-04-06T13:28:42.229937+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:54:48.293603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228927+0000","last_change":"2024-04-06T13:28:40.832562+0000","last_active":"2024-04-06T13:28:42.228927+0000","last_peered":"2024-04-06T13:28:42.228927+0000","last_clean":"2024-04-06T13:28:42.228927+0000","last_became_active":"2024-04-06T13:28:40.832361+0000","last_became_peered":"2024-04-06T13:28:40.832361+0000","last_unstale":"2024-04-06T13:28:42.228927+0000","last_undegraded":"2024-04-06T13:28:42.228927+0000","last_fullsized":"2024-04-06T13:28:42.228927+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:18:26.747951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"177'3","reported_seq":14,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228627+0000","last_change":"2024-04-06T13:28:40.411188+0000","last_active":"2024-04-06T13:28:42.228627+0000","last_peered":"2024-04-06T13:28:42.228627+0000","last_clean":"2024-04-06T13:28:42.228627+0000","last_became_active":"2024-04-06T13:28:40.410450+0000","last_became_peered":"2024-04-06T13:28:40.410450+0000","last_unstale":"2024-04-06T13:28:42.228627+0000","last_undegraded":"2024-04-06T13:28:42.228627+0000","last_fullsized":"2024-04-06T13:28:42.228627+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:07:22.365454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229345+0000","last_change":"2024-04-06T13:28:40.836457+0000","last_active":"2024-04-06T13:28:42.229345+0000","last_peered":"2024-04-06T13:28:42.229345+0000","last_clean":"2024-04-06T13:28:42.229345+0000","last_became_active":"2024-04-06T13:28:40.835883+0000","last_became_peered":"2024-04-06T13:28:40.835883+0000","last_unstale":"2024-04-06T13:28:42.229345+0000","last_undegraded":"2024-04-06T13:28:42.229345+0000","last_fullsized":"2024-04-06T13:28:42.229345+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:40:55.218858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"177'2","reported_seq":13,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.824297+0000","last_change":"2024-04-06T13:28:40.416717+0000","last_active":"2024-04-06T13:28:41.824297+0000","last_peered":"2024-04-06T13:28:41.824297+0000","last_clean":"2024-04-06T13:28:41.824297+0000","last_became_active":"2024-04-06T13:28:40.414160+0000","last_became_peered":"2024-04-06T13:28:40.414160+0000","last_unstale":"2024-04-06T13:28:41.824297+0000","last_undegraded":"2024-04-06T13:28:41.824297+0000","last_fullsized":"2024-04-06T13:28:41.824297+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:10:26.018784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823157+0000","last_change":"2024-04-06T13:28:40.842464+0000","last_active":"2024-04-06T13:28:41.823157+0000","last_peered":"2024-04-06T13:28:41.823157+0000","last_clean":"2024-04-06T13:28:41.823157+0000","last_became_active":"2024-04-06T13:28:40.842238+0000","last_became_peered":"2024-04-06T13:28:40.842238+0000","last_unstale":"2024-04-06T13:28:41.823157+0000","last_undegraded":"2024-04-06T13:28:41.823157+0000","last_fullsized":"2024-04-06T13:28:41.823157+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:59:44.613608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229500+0000","last_change":"2024-04-06T13:28:40.419303+0000","last_active":"2024-04-06T13:28:42.229500+0000","last_peered":"2024-04-06T13:28:42.229500+0000","last_clean":"2024-04-06T13:28:42.229500+0000","last_became_active":"2024-04-06T13:28:40.418997+0000","last_became_peered":"2024-04-06T13:28:40.418997+0000","last_unstale":"2024-04-06T13:28:42.229500+0000","last_undegraded":"2024-04-06T13:28:42.229500+0000","last_fullsized":"2024-04-06T13:28:42.229500+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:52:38.133753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823998+0000","last_change":"2024-04-06T13:28:40.824612+0000","last_active":"2024-04-06T13:28:41.823998+0000","last_peered":"2024-04-06T13:28:41.823998+0000","last_clean":"2024-04-06T13:28:41.823998+0000","last_became_active":"2024-04-06T13:28:40.824358+0000","last_became_peered":"2024-04-06T13:28:40.824358+0000","last_unstale":"2024-04-06T13:28:41.823998+0000","last_undegraded":"2024-04-06T13:28:41.823998+0000","last_fullsized":"2024-04-06T13:28:41.823998+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:19:31.528546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229186+0000","last_change":"2024-04-06T13:28:40.406854+0000","last_active":"2024-04-06T13:28:42.229186+0000","last_peered":"2024-04-06T13:28:42.229186+0000","last_clean":"2024-04-06T13:28:42.229186+0000","last_became_active":"2024-04-06T13:28:40.406566+0000","last_became_peered":"2024-04-06T13:28:40.406566+0000","last_unstale":"2024-04-06T13:28:42.229186+0000","last_undegraded":"2024-04-06T13:28:42.229186+0000","last_fullsized":"2024-04-06T13:28:42.229186+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:34:25.499120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823413+0000","last_change":"2024-04-06T13:28:40.829903+0000","last_active":"2024-04-06T13:28:41.823413+0000","last_peered":"2024-04-06T13:28:41.823413+0000","last_clean":"2024-04-06T13:28:41.823413+0000","last_became_active":"2024-04-06T13:28:40.829654+0000","last_became_peered":"2024-04-06T13:28:40.829654+0000","last_unstale":"2024-04-06T13:28:41.823413+0000","last_undegraded":"2024-04-06T13:28:41.823413+0000","last_fullsized":"2024-04-06T13:28:41.823413+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:29:15.092554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"177'8","reported_seq":19,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228633+0000","last_change":"2024-04-06T13:28:40.411428+0000","last_active":"2024-04-06T13:28:42.228633+0000","last_peered":"2024-04-06T13:28:42.228633+0000","last_clean":"2024-04-06T13:28:42.228633+0000","last_became_active":"2024-04-06T13:28:40.411127+0000","last_became_peered":"2024-04-06T13:28:40.411127+0000","last_unstale":"2024-04-06T13:28:42.228633+0000","last_undegraded":"2024-04-06T13:28:42.228633+0000","last_fullsized":"2024-04-06T13:28:42.228633+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:08:04.805205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.825329+0000","last_change":"2024-04-06T13:28:40.828670+0000","last_active":"2024-04-06T13:28:41.825329+0000","last_peered":"2024-04-06T13:28:41.825329+0000","last_clean":"2024-04-06T13:28:41.825329+0000","last_became_active":"2024-04-06T13:28:40.827846+0000","last_became_peered":"2024-04-06T13:28:40.827846+0000","last_unstale":"2024-04-06T13:28:41.825329+0000","last_undegraded":"2024-04-06T13:28:41.825329+0000","last_fullsized":"2024-04-06T13:28:41.825329+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:02:04.239649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"177'1","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228576+0000","last_change":"2024-04-06T13:28:40.409911+0000","last_active":"2024-04-06T13:28:42.228576+0000","last_peered":"2024-04-06T13:28:42.228576+0000","last_clean":"2024-04-06T13:28:42.228576+0000","last_became_active":"2024-04-06T13:28:40.409605+0000","last_became_peered":"2024-04-06T13:28:40.409605+0000","last_unstale":"2024-04-06T13:28:42.228576+0000","last_undegraded":"2024-04-06T13:28:42.228576+0000","last_fullsized":"2024-04-06T13:28:42.228576+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:37:00.320010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.824934+0000","last_change":"2024-04-06T13:28:40.840101+0000","last_active":"2024-04-06T13:28:41.824934+0000","last_peered":"2024-04-06T13:28:41.824934+0000","last_clean":"2024-04-06T13:28:41.824934+0000","last_became_active":"2024-04-06T13:28:40.839462+0000","last_became_peered":"2024-04-06T13:28:40.839462+0000","last_unstale":"2024-04-06T13:28:41.824934+0000","last_undegraded":"2024-04-06T13:28:41.824934+0000","last_fullsized":"2024-04-06T13:28:41.824934+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:49:31.035605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"177'2","reported_seq":13,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.824643+0000","last_change":"2024-04-06T13:28:40.408732+0000","last_active":"2024-04-06T13:28:41.824643+0000","last_peered":"2024-04-06T13:28:41.824643+0000","last_clean":"2024-04-06T13:28:41.824643+0000","last_became_active":"2024-04-06T13:28:40.408186+0000","last_became_peered":"2024-04-06T13:28:40.408186+0000","last_unstale":"2024-04-06T13:28:41.824643+0000","last_undegraded":"2024-04-06T13:28:41.824643+0000","last_fullsized":"2024-04-06T13:28:41.824643+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:40:54.213777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.230046+0000","last_change":"2024-04-06T13:28:40.833359+0000","last_active":"2024-04-06T13:28:42.230046+0000","last_peered":"2024-04-06T13:28:42.230046+0000","last_clean":"2024-04-06T13:28:42.230046+0000","last_became_active":"2024-04-06T13:28:40.833096+0000","last_became_peered":"2024-04-06T13:28:40.833096+0000","last_unstale":"2024-04-06T13:28:42.230046+0000","last_undegraded":"2024-04-06T13:28:42.230046+0000","last_fullsized":"2024-04-06T13:28:42.230046+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:21:26.361084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"177'1","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823356+0000","last_change":"2024-04-06T13:28:40.412433+0000","last_active":"2024-04-06T13:28:41.823356+0000","last_peered":"2024-04-06T13:28:41.823356+0000","last_clean":"2024-04-06T13:28:41.823356+0000","last_became_active":"2024-04-06T13:28:40.411519+0000","last_became_peered":"2024-04-06T13:28:40.411519+0000","last_unstale":"2024-04-06T13:28:41.823356+0000","last_undegraded":"2024-04-06T13:28:41.823356+0000","last_fullsized":"2024-04-06T13:28:41.823356+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:11.032370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229857+0000","last_change":"2024-04-06T13:28:40.845446+0000","last_active":"2024-04-06T13:28:42.229857+0000","last_peered":"2024-04-06T13:28:42.229857+0000","last_clean":"2024-04-06T13:28:42.229857+0000","last_became_active":"2024-04-06T13:28:40.845172+0000","last_became_peered":"2024-04-06T13:28:40.845172+0000","last_unstale":"2024-04-06T13:28:42.229857+0000","last_undegraded":"2024-04-06T13:28:42.229857+0000","last_fullsized":"2024-04-06T13:28:42.229857+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:11:50.750310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229552+0000","last_change":"2024-04-06T13:28:40.416917+0000","last_active":"2024-04-06T13:28:42.229552+0000","last_peered":"2024-04-06T13:28:42.229552+0000","last_clean":"2024-04-06T13:28:42.229552+0000","last_became_active":"2024-04-06T13:28:40.416632+0000","last_became_peered":"2024-04-06T13:28:40.416632+0000","last_unstale":"2024-04-06T13:28:42.229552+0000","last_undegraded":"2024-04-06T13:28:42.229552+0000","last_fullsized":"2024-04-06T13:28:42.229552+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:40:37.721893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.824192+0000","last_change":"2024-04-06T13:28:40.832265+0000","last_active":"2024-04-06T13:28:41.824192+0000","last_peered":"2024-04-06T13:28:41.824192+0000","last_clean":"2024-04-06T13:28:41.824192+0000","last_became_active":"2024-04-06T13:28:40.832001+0000","last_became_peered":"2024-04-06T13:28:40.832001+0000","last_unstale":"2024-04-06T13:28:41.824192+0000","last_undegraded":"2024-04-06T13:28:41.824192+0000","last_fullsized":"2024-04-06T13:28:41.824192+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:05:26.251321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"177'2","reported_seq":13,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228615+0000","last_change":"2024-04-06T13:28:40.421037+0000","last_active":"2024-04-06T13:28:42.228615+0000","last_peered":"2024-04-06T13:28:42.228615+0000","last_clean":"2024-04-06T13:28:42.228615+0000","last_became_active":"2024-04-06T13:28:40.420742+0000","last_became_peered":"2024-04-06T13:28:40.420742+0000","last_unstale":"2024-04-06T13:28:42.228615+0000","last_undegraded":"2024-04-06T13:28:42.228615+0000","last_fullsized":"2024-04-06T13:28:42.228615+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:56.474313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823188+0000","last_change":"2024-04-06T13:28:40.831641+0000","last_active":"2024-04-06T13:28:41.823188+0000","last_peered":"2024-04-06T13:28:41.823188+0000","last_clean":"2024-04-06T13:28:41.823188+0000","last_became_active":"2024-04-06T13:28:40.831383+0000","last_became_peered":"2024-04-06T13:28:40.831383+0000","last_unstale":"2024-04-06T13:28:41.823188+0000","last_undegraded":"2024-04-06T13:28:41.823188+0000","last_fullsized":"2024-04-06T13:28:41.823188+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:09:51.321262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229530+0000","last_change":"2024-04-06T13:28:40.416010+0000","last_active":"2024-04-06T13:28:42.229530+0000","last_peered":"2024-04-06T13:28:42.229530+0000","last_clean":"2024-04-06T13:28:42.229530+0000","last_became_active":"2024-04-06T13:28:40.415615+0000","last_became_peered":"2024-04-06T13:28:40.415615+0000","last_unstale":"2024-04-06T13:28:42.229530+0000","last_undegraded":"2024-04-06T13:28:42.229530+0000","last_fullsized":"2024-04-06T13:28:42.229530+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:33:13.846379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.824906+0000","last_change":"2024-04-06T13:28:40.835137+0000","last_active":"2024-04-06T13:28:41.824906+0000","last_peered":"2024-04-06T13:28:41.824906+0000","last_clean":"2024-04-06T13:28:41.824906+0000","last_became_active":"2024-04-06T13:28:40.834860+0000","last_became_peered":"2024-04-06T13:28:40.834860+0000","last_unstale":"2024-04-06T13:28:41.824906+0000","last_undegraded":"2024-04-06T13:28:41.824906+0000","last_fullsized":"2024-04-06T13:28:41.824906+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:52.566973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228184+0000","last_change":"2024-04-06T13:28:40.421702+0000","last_active":"2024-04-06T13:28:42.228184+0000","last_peered":"2024-04-06T13:28:42.228184+0000","last_clean":"2024-04-06T13:28:42.228184+0000","last_became_active":"2024-04-06T13:28:40.421344+0000","last_became_peered":"2024-04-06T13:28:40.421344+0000","last_unstale":"2024-04-06T13:28:42.228184+0000","last_undegraded":"2024-04-06T13:28:42.228184+0000","last_fullsized":"2024-04-06T13:28:42.228184+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:17:42.161859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823219+0000","last_change":"2024-04-06T13:28:40.843971+0000","last_active":"2024-04-06T13:28:41.823219+0000","last_peered":"2024-04-06T13:28:41.823219+0000","last_clean":"2024-04-06T13:28:41.823219+0000","last_became_active":"2024-04-06T13:28:40.843734+0000","last_became_peered":"2024-04-06T13:28:40.843734+0000","last_unstale":"2024-04-06T13:28:41.823219+0000","last_undegraded":"2024-04-06T13:28:41.823219+0000","last_fullsized":"2024-04-06T13:28:41.823219+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:25.783071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229719+0000","last_change":"2024-04-06T13:28:40.420732+0000","last_active":"2024-04-06T13:28:42.229719+0000","last_peered":"2024-04-06T13:28:42.229719+0000","last_clean":"2024-04-06T13:28:42.229719+0000","last_became_active":"2024-04-06T13:28:40.420444+0000","last_became_peered":"2024-04-06T13:28:40.420444+0000","last_unstale":"2024-04-06T13:28:42.229719+0000","last_undegraded":"2024-04-06T13:28:42.229719+0000","last_fullsized":"2024-04-06T13:28:42.229719+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:32:44.959236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823989+0000","last_change":"2024-04-06T13:28:40.830081+0000","last_active":"2024-04-06T13:28:41.823989+0000","last_peered":"2024-04-06T13:28:41.823989+0000","last_clean":"2024-04-06T13:28:41.823989+0000","last_became_active":"2024-04-06T13:28:40.829401+0000","last_became_peered":"2024-04-06T13:28:40.829401+0000","last_unstale":"2024-04-06T13:28:41.823989+0000","last_undegraded":"2024-04-06T13:28:41.823989+0000","last_fullsized":"2024-04-06T13:28:41.823989+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:10.713595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"177'2","reported_seq":13,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229343+0000","last_change":"2024-04-06T13:28:40.416275+0000","last_active":"2024-04-06T13:28:42.229343+0000","last_peered":"2024-04-06T13:28:42.229343+0000","last_clean":"2024-04-06T13:28:42.229343+0000","last_became_active":"2024-04-06T13:28:40.415994+0000","last_became_peered":"2024-04-06T13:28:40.415994+0000","last_unstale":"2024-04-06T13:28:42.229343+0000","last_undegraded":"2024-04-06T13:28:42.229343+0000","last_fullsized":"2024-04-06T13:28:42.229343+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:44:22.557389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229290+0000","last_change":"2024-04-06T13:28:40.843577+0000","last_active":"2024-04-06T13:28:42.229290+0000","last_peered":"2024-04-06T13:28:42.229290+0000","last_clean":"2024-04-06T13:28:42.229290+0000","last_became_active":"2024-04-06T13:28:40.843053+0000","last_became_peered":"2024-04-06T13:28:40.843053+0000","last_unstale":"2024-04-06T13:28:42.229290+0000","last_undegraded":"2024-04-06T13:28:42.229290+0000","last_fullsized":"2024-04-06T13:28:42.229290+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:32:45.964316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229855+0000","last_change":"2024-04-06T13:28:40.418634+0000","last_active":"2024-04-06T13:28:42.229855+0000","last_peered":"2024-04-06T13:28:42.229855+0000","last_clean":"2024-04-06T13:28:42.229855+0000","last_became_active":"2024-04-06T13:28:40.417902+0000","last_became_peered":"2024-04-06T13:28:40.417902+0000","last_unstale":"2024-04-06T13:28:42.229855+0000","last_undegraded":"2024-04-06T13:28:42.229855+0000","last_fullsized":"2024-04-06T13:28:42.229855+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:31:14.067752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.230041+0000","last_change":"2024-04-06T13:28:40.840601+0000","last_active":"2024-04-06T13:28:42.230041+0000","last_peered":"2024-04-06T13:28:42.230041+0000","last_clean":"2024-04-06T13:28:42.230041+0000","last_became_active":"2024-04-06T13:28:40.840304+0000","last_became_peered":"2024-04-06T13:28:40.840304+0000","last_unstale":"2024-04-06T13:28:42.230041+0000","last_undegraded":"2024-04-06T13:28:42.230041+0000","last_fullsized":"2024-04-06T13:28:42.230041+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:33:14.851459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823399+0000","last_change":"2024-04-06T13:28:40.416331+0000","last_active":"2024-04-06T13:28:41.823399+0000","last_peered":"2024-04-06T13:28:41.823399+0000","last_clean":"2024-04-06T13:28:41.823399+0000","last_became_active":"2024-04-06T13:28:40.416050+0000","last_became_peered":"2024-04-06T13:28:40.416050+0000","last_unstale":"2024-04-06T13:28:41.823399+0000","last_undegraded":"2024-04-06T13:28:41.823399+0000","last_fullsized":"2024-04-06T13:28:41.823399+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:33:13.846379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.824746+0000","last_change":"2024-04-06T13:28:40.843753+0000","last_active":"2024-04-06T13:28:41.824746+0000","last_peered":"2024-04-06T13:28:41.824746+0000","last_clean":"2024-04-06T13:28:41.824746+0000","last_became_active":"2024-04-06T13:28:40.843579+0000","last_became_peered":"2024-04-06T13:28:40.843579+0000","last_unstale":"2024-04-06T13:28:41.824746+0000","last_undegraded":"2024-04-06T13:28:41.824746+0000","last_fullsized":"2024-04-06T13:28:41.824746+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:40:22.169564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.824136+0000","last_change":"2024-04-06T13:28:40.422805+0000","last_active":"2024-04-06T13:28:41.824136+0000","last_peered":"2024-04-06T13:28:41.824136+0000","last_clean":"2024-04-06T13:28:41.824136+0000","last_became_active":"2024-04-06T13:28:40.422554+0000","last_became_peered":"2024-04-06T13:28:40.422554+0000","last_unstale":"2024-04-06T13:28:41.824136+0000","last_undegraded":"2024-04-06T13:28:41.824136+0000","last_fullsized":"2024-04-06T13:28:41.824136+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:03:11.985300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229240+0000","last_change":"2024-04-06T13:28:40.836286+0000","last_active":"2024-04-06T13:28:42.229240+0000","last_peered":"2024-04-06T13:28:42.229240+0000","last_clean":"2024-04-06T13:28:42.229240+0000","last_became_active":"2024-04-06T13:28:40.835439+0000","last_became_peered":"2024-04-06T13:28:40.835439+0000","last_unstale":"2024-04-06T13:28:42.229240+0000","last_undegraded":"2024-04-06T13:28:42.229240+0000","last_fullsized":"2024-04-06T13:28:42.229240+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:25.783071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229896+0000","last_change":"2024-04-06T13:28:40.413990+0000","last_active":"2024-04-06T13:28:42.229896+0000","last_peered":"2024-04-06T13:28:42.229896+0000","last_clean":"2024-04-06T13:28:42.229896+0000","last_became_active":"2024-04-06T13:28:40.413372+0000","last_became_peered":"2024-04-06T13:28:40.413372+0000","last_unstale":"2024-04-06T13:28:42.229896+0000","last_undegraded":"2024-04-06T13:28:42.229896+0000","last_fullsized":"2024-04-06T13:28:42.229896+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:27:26.130961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228991+0000","last_change":"2024-04-06T13:28:40.845072+0000","last_active":"2024-04-06T13:28:42.228991+0000","last_peered":"2024-04-06T13:28:42.228991+0000","last_clean":"2024-04-06T13:28:42.228991+0000","last_became_active":"2024-04-06T13:28:40.843106+0000","last_became_peered":"2024-04-06T13:28:40.843106+0000","last_unstale":"2024-04-06T13:28:42.228991+0000","last_undegraded":"2024-04-06T13:28:42.228991+0000","last_fullsized":"2024-04-06T13:28:42.228991+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:11:50.750310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228459+0000","last_change":"2024-04-06T13:28:40.422136+0000","last_active":"2024-04-06T13:28:42.228459+0000","last_peered":"2024-04-06T13:28:42.228459+0000","last_clean":"2024-04-06T13:28:42.228459+0000","last_became_active":"2024-04-06T13:28:40.421816+0000","last_became_peered":"2024-04-06T13:28:40.421816+0000","last_unstale":"2024-04-06T13:28:42.228459+0000","last_undegraded":"2024-04-06T13:28:42.228459+0000","last_fullsized":"2024-04-06T13:28:42.228459+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:05:41.512140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.823188+0000","last_change":"2024-04-06T13:28:40.838259+0000","last_active":"2024-04-06T13:28:41.823188+0000","last_peered":"2024-04-06T13:28:41.823188+0000","last_clean":"2024-04-06T13:28:41.823188+0000","last_became_active":"2024-04-06T13:28:40.838005+0000","last_became_peered":"2024-04-06T13:28:40.838005+0000","last_unstale":"2024-04-06T13:28:41.823188+0000","last_undegraded":"2024-04-06T13:28:41.823188+0000","last_fullsized":"2024-04-06T13:28:41.823188+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:23:38.800047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.229066+0000","last_change":"2024-04-06T13:28:40.414242+0000","last_active":"2024-04-06T13:28:42.229066+0000","last_peered":"2024-04-06T13:28:42.229066+0000","last_clean":"2024-04-06T13:28:42.229066+0000","last_became_active":"2024-04-06T13:28:40.413386+0000","last_became_peered":"2024-04-06T13:28:40.413386+0000","last_unstale":"2024-04-06T13:28:42.229066+0000","last_undegraded":"2024-04-06T13:28:42.229066+0000","last_fullsized":"2024-04-06T13:28:42.229066+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:40:37.721893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.230069+0000","last_change":"2024-04-06T13:28:40.832252+0000","last_active":"2024-04-06T13:28:42.230069+0000","last_peered":"2024-04-06T13:28:42.230069+0000","last_clean":"2024-04-06T13:28:42.230069+0000","last_became_active":"2024-04-06T13:28:40.831936+0000","last_became_peered":"2024-04-06T13:28:40.831936+0000","last_unstale":"2024-04-06T13:28:42.230069+0000","last_undegraded":"2024-04-06T13:28:42.230069+0000","last_fullsized":"2024-04-06T13:28:42.230069+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:00:38.061764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:41.824359+0000","last_change":"2024-04-06T13:28:40.411481+0000","last_active":"2024-04-06T13:28:41.824359+0000","last_peered":"2024-04-06T13:28:41.824359+0000","last_clean":"2024-04-06T13:28:41.824359+0000","last_became_active":"2024-04-06T13:28:40.411223+0000","last_became_peered":"2024-04-06T13:28:40.411223+0000","last_unstale":"2024-04-06T13:28:41.824359+0000","last_undegraded":"2024-04-06T13:28:41.824359+0000","last_fullsized":"2024-04-06T13:28:41.824359+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:11:47.230761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":10,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.230296+0000","last_change":"2024-04-06T13:28:40.829900+0000","last_active":"2024-04-06T13:28:42.230296+0000","last_peered":"2024-04-06T13:28:42.230296+0000","last_clean":"2024-04-06T13:28:42.230296+0000","last_became_active":"2024-04-06T13:28:40.828404+0000","last_became_peered":"2024-04-06T13:28:40.828404+0000","last_unstale":"2024-04-06T13:28:42.230296+0000","last_undegraded":"2024-04-06T13:28:42.230296+0000","last_fullsized":"2024-04-06T13:28:42.230296+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:40.394568+0000","last_clean_scrub_stamp":"2024-04-06T13:28:40.394568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:59:44.613608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"177'6","reported_seq":17,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-06T13:28:42.228373+0000","last_change":"2024-04-06T13:28:40.421559+0000","last_active":"2024-04-06T13:28:42.228373+0000","last_peered":"2024-04-06T13:28:42.228373+0000","last_clean":"2024-04-06T13:28:42.228373+0000","last_became_active":"2024-04-06T13:28:40.420777+0000","last_became_peered":"2024-04-06T13:28:40.420777+0000","last_unstale":"2024-04-06T13:28:42.228373+0000","last_undegraded":"2024-04-06T13:28:42.228373+0000","last_fullsized":"2024-04-06T13:28:42.228373+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:28:39.389487+0000","last_clean_scrub_stamp":"2024-04-06T13:28:39.389487+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:40:20.186258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"167'271","reported_seq":906,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-04-06T13:28:40.823069+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:28:40.823069+0000","last_peered":"2024-04-06T13:28:40.823069+0000","last_clean":"2024-04-06T13:28:40.823069+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:28:40.823069+0000","last_undegraded":"2024-04-06T13:28:40.823069+0000","last_fullsized":"2024-04-06T13:28:40.823069+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575763,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51928,"kb_used_data":2756,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319912,"statfs":{"total":96636764160,"available":96583589888,"internally_reserved":0,"allocated":2822144,"data_stored":1714868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.58799999999999997,"15min":0.59299999999999997},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.78200000000000003,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59599999999999997,"15min":0.60099999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.35899999999999999},"max":{"1min":0.78500000000000003,"5min":0.86099999999999999,"15min":1.4790000000000001},"last":0.60299999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.60799999999999998,"15min":0.60999999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.79500000000000004,"5min":0.82499999999999996,"15min":0.83299999999999996},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.40300000000000002,"5min":0.39300000000000002,"15min":0.39000000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":1.5249999999999999},"last":0.57999999999999996}]},{"osd":2,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59099999999999997,"15min":0.58199999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59699999999999998,"15min":0.61099999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":1.198},"last":0.65100000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.60699999999999998,"15min":0.60799999999999998},"min":{"1min":0.437,"5min":0.36499999999999999,"15min":0.35499999999999998},"max":{"1min":0.77500000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.499},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61599999999999999,"15min":0.61199999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.78600000000000003,"5min":0.79600000000000004,"15min":1.1200000000000001},"last":0.56399999999999995}]},{"osd":4,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45400000000000001,"15min":0.45200000000000001},"min":{"1min":0.29599999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.97399999999999998},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55800000000000005,"15min":0.54000000000000004},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.33600000000000002},"max":{"1min":0.72399999999999998,"5min":0.77600000000000002,"15min":0.78000000000000003},"last":0.41599999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.52700000000000002,"15min":0.51500000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69599999999999995,"5min":0.80000000000000004,"15min":1.024},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.58499999999999996,"15min":0.57999999999999996},"min":{"1min":0.36899999999999999,"5min":0.35599999999999998,"15min":0.33000000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":1.046},"last":0.47899999999999998}]},{"osd":6,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56699999999999995,"15min":0.56100000000000005},"min":{"1min":0.41399999999999998,"5min":0.38800000000000001,"15min":0.35299999999999998},"max":{"1min":0.81100000000000005,"5min":0.82199999999999995,"15min":1.4159999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.59599999999999997,"15min":0.57899999999999996},"min":{"1min":0.35799999999999998,"5min":0.34599999999999997,"15min":0.33700000000000002},"max":{"1min":0.75600000000000001,"5min":0.80700000000000005,"15min":0.873},"last":0.45900000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575762,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37388,"kb_used_data":2172,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334452,"statfs":{"total":96636764160,"available":96598478848,"internally_reserved":0,"allocated":2224128,"data_stored":1123529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57799999999999996,"15min":0.56799999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.34399999999999997},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.83499999999999996},"last":0.621},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58699999999999997,"15min":0.57699999999999996},"min":{"1min":0.40400000000000003,"5min":0.40300000000000002,"15min":0.35899999999999999},"max":{"1min":0.73399999999999999,"5min":0.76200000000000001,"15min":0.78800000000000003},"last":0.72799999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61899999999999999,"15min":0.624},"min":{"1min":0.42799999999999999,"5min":0.37,"15min":0.35199999999999998},"max":{"1min":0.77100000000000002,"5min":0.80300000000000005,"15min":0.81799999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.64800000000000002,"15min":0.64200000000000002},"min":{"1min":0.52600000000000002,"5min":0.48599999999999999,"15min":0.311},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.75900000000000001}]},{"osd":2,"last update":"Sat Apr 6 13:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.47699999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.84499999999999997},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60899999999999999,"15min":0.60199999999999998},"min":{"1min":0.45000000000000001,"5min":0.42699999999999999,"15min":0.373},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.80900000000000005},"last":0.81399999999999995}]},{"osd":3,"last update":"Sat Apr 6 13:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.626,"15min":0.61899999999999999},"min":{"1min":0.50600000000000001,"5min":0.42399999999999999,"15min":0.40799999999999997},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.66500000000000004,"15min":0.66200000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.877},"last":0.70999999999999996}]},{"osd":4,"last update":"Sat Apr 6 13:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.31},"max":{"1min":0.64100000000000001,"5min":0.746,"15min":0.81000000000000005},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54400000000000004,"15min":0.54700000000000004},"min":{"1min":0.38900000000000001,"5min":0.29499999999999998,"15min":0.29299999999999998},"max":{"1min":0.74199999999999999,"5min":0.80200000000000005,"15min":0.873},"last":0.83499999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51500000000000001,"15min":0.51600000000000001},"min":{"1min":0.36499999999999999,"5min":0.32600000000000001,"15min":0.307},"max":{"1min":0.72799999999999998,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53300000000000003,"15min":0.53600000000000003},"min":{"1min":0.36399999999999999,"5min":0.36199999999999999,"15min":0.32800000000000001},"max":{"1min":0.70999999999999996,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.85999999999999999}]},{"osd":7,"last update":"Sat Apr 6 13:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56000000000000005,"15min":0.56100000000000005},"min":{"1min":0.41599999999999998,"5min":0.35299999999999998,"15min":0.34899999999999998},"max":{"1min":0.81100000000000005,"5min":0.85399999999999998,"15min":0.85599999999999998},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.38700000000000001,"5min":0.375,"15min":0.36499999999999999},"max":{"1min":0.82599999999999996,"5min":0.83299999999999996,"15min":0.85999999999999999},"last":0.89500000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575761,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37384,"kb_used_data":2172,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334456,"statfs":{"total":96636764160,"available":96598482944,"internally_reserved":0,"allocated":2224128,"data_stored":1123529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.39900000000000002,"15min":0.38800000000000001},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.50900000000000001,"5min":0.54800000000000004,"15min":0.64300000000000002},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.434,"15min":0.42899999999999999},"min":{"1min":0.28799999999999998,"5min":0.27000000000000002,"15min":0.26700000000000002},"max":{"1min":0.56100000000000005,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.32300000000000001}]},{"osd":2,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.42999999999999999,"15min":0.42499999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.52400000000000002,"5min":0.64000000000000001,"15min":0.67400000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.47699999999999998,"15min":0.46500000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59399999999999997,"5min":0.70499999999999996,"15min":0.73299999999999998},"last":0.47399999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.48499999999999999,"15min":0.47299999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.30599999999999999},"max":{"1min":0.64200000000000002,"5min":0.70899999999999996,"15min":0.77100000000000002},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.45600000000000002,"15min":0.44700000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.63200000000000001,"5min":0.65500000000000003,"15min":0.68000000000000005},"last":0.42299999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65900000000000003,"15min":0.64700000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.85799999999999998,"5min":0.92500000000000004,"15min":1.008},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.66900000000000004,"15min":0.65900000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.57499999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.67300000000000004,"15min":0.66200000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.371},"max":{"1min":0.93799999999999994,"5min":0.95499999999999996,"15min":0.98299999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.69099999999999995,"15min":0.67400000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91800000000000004,"5min":0.94199999999999995,"15min":1.0189999999999999},"last":0.64400000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.67600000000000005,"15min":0.66700000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.97199999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.69499999999999995,"15min":0.68500000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.877,"5min":1,"15min":1},"last":0.71499999999999997}]},{"osd":7,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.71199999999999997,"15min":0.69499999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.89000000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68000000000000005,"15min":0.66900000000000004},"min":{"1min":0.48699999999999999,"5min":0.46899999999999997,"15min":0.44},"max":{"1min":0.91200000000000003,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.627}]}]},{"osd":0,"up_from":13,"seq":55834575761,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37984,"kb_used_data":2772,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94333856,"statfs":{"total":96636764160,"available":96597868544,"internally_reserved":0,"allocated":2838528,"data_stored":1716097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39700000000000002,"15min":0.39800000000000002},"min":{"1min":0.27600000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.46700000000000003,"5min":0.58799999999999997,"15min":0.65000000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42599999999999999,"15min":0.42499999999999999},"min":{"1min":0.28499999999999998,"5min":0.28000000000000003,"15min":0.26400000000000001},"max":{"1min":0.57199999999999995,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.35999999999999999}]},{"osd":2,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.434,"15min":0.434},"min":{"1min":0.313,"5min":0.29699999999999999,"15min":0.27000000000000002},"max":{"1min":0.58699999999999997,"5min":0.67200000000000004,"15min":0.71199999999999997},"last":0.371},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.45000000000000001,"15min":0.45300000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.51400000000000001,"5min":0.63,"15min":0.77000000000000002},"last":0.33200000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.436,"15min":0.435},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.27900000000000003},"max":{"1min":0.51100000000000001,"5min":0.61899999999999999,"15min":0.63200000000000001},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.39700000000000002,"5min":0.34399999999999997,"15min":0.307},"max":{"1min":0.64500000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.39100000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.67200000000000004,"15min":0.65400000000000003},"min":{"1min":0.47899999999999998,"5min":0.46899999999999997,"15min":0.45600000000000002},"max":{"1min":0.86299999999999999,"5min":0.93000000000000005,"15min":0.999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.67900000000000005,"15min":0.66800000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.41499999999999998},"max":{"1min":0.82499999999999996,"5min":0.96699999999999997,"15min":1.103},"last":0.63200000000000001}]},{"osd":5,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.66900000000000004,"15min":0.65700000000000003},"min":{"1min":0.46700000000000003,"5min":0.46100000000000002,"15min":0.46000000000000002},"max":{"1min":0.77100000000000002,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70499999999999996,"15min":0.68400000000000005},"min":{"1min":0.497,"5min":0.48999999999999999,"15min":0.41299999999999998},"max":{"1min":0.89100000000000001,"5min":1.0169999999999999,"15min":1.0469999999999999},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.70399999999999996,"15min":0.68899999999999995},"min":{"1min":0.54000000000000004,"5min":0.49399999999999999,"15min":0.46000000000000002},"max":{"1min":0.91100000000000003,"5min":1.0629999999999999,"15min":1.0920000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.70399999999999996,"15min":0.68799999999999994},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.46000000000000002},"max":{"1min":0.871,"5min":0.92900000000000005,"15min":0.98599999999999999},"last":0.55900000000000005}]},{"osd":7,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.71999999999999997,"15min":0.70299999999999996},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.46400000000000002},"max":{"1min":0.81399999999999995,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.71499999999999997,"15min":0.69299999999999995},"min":{"1min":0.502,"5min":0.48899999999999999,"15min":0.438},"max":{"1min":0.92400000000000004,"5min":1.042,"15min":1.071},"last":0.55000000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575763,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55444,"kb_used_data":2176,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316396,"statfs":{"total":96636764160,"available":96579989504,"internally_reserved":0,"allocated":2228224,"data_stored":1123571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.39400000000000002,"15min":0.39200000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51300000000000001,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":2.157},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.437,"15min":0.434},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.59199999999999997,"5min":0.64300000000000002,"15min":0.68700000000000006},"last":0.50800000000000001}]},{"osd":1,"last update":"Sat Apr 6 13:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.42199999999999999,"15min":0.41499999999999998},"min":{"1min":0.28999999999999998,"5min":0.27500000000000002,"15min":0.254},"max":{"1min":0.62,"5min":0.62,"15min":0.63300000000000001},"last":2.6190000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.45900000000000002,"15min":0.45300000000000001},"min":{"1min":0.32800000000000001,"5min":0.32400000000000001,"15min":0.308},"max":{"1min":0.49299999999999999,"5min":0.628,"15min":0.68899999999999995},"last":2.0870000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.46899999999999997,"15min":0.47099999999999997},"min":{"1min":0.35899999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.57099999999999995,"5min":0.65800000000000003,"15min":0.68100000000000005},"last":3.9980000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45500000000000002,"15min":0.45400000000000001},"min":{"1min":0.33600000000000002,"5min":0.33300000000000002,"15min":0.29699999999999999},"max":{"1min":0.53500000000000003,"5min":0.627,"15min":0.65300000000000002},"last":4.0350000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65700000000000003,"15min":0.65100000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85299999999999998,"5min":0.85799999999999998,"15min":0.93700000000000006},"last":6.335},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67500000000000004,"15min":0.66800000000000004},"min":{"1min":0.52700000000000002,"5min":0.47799999999999998,"15min":0.434},"max":{"1min":0.85199999999999998,"5min":0.91500000000000004,"15min":0.91800000000000004},"last":5.2610000000000001}]},{"osd":5,"last update":"Sat Apr 6 13:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67500000000000004},"min":{"1min":0.52100000000000002,"5min":0.46800000000000003,"15min":0.42699999999999999},"max":{"1min":0.82799999999999996,"5min":0.95699999999999996,"15min":0.97399999999999998},"last":2.7599999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.66600000000000004,"15min":0.66900000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.40699999999999997},"max":{"1min":0.88400000000000001,"5min":0.89000000000000001,"15min":0.95399999999999996},"last":2.206}]},{"osd":6,"last update":"Sat Apr 6 13:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.67700000000000005,"15min":0.67400000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.40500000000000003},"max":{"1min":0.83399999999999996,"5min":0.93899999999999995,"15min":1.022},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69599999999999995,"15min":0.69799999999999995},"min":{"1min":0.47699999999999998,"5min":0.47299999999999998,"15min":0.41999999999999998},"max":{"1min":0.84499999999999997,"5min":0.95099999999999996,"15min":1.069},"last":1.55}]},{"osd":7,"last update":"Sat Apr 6 13:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.70199999999999996,"15min":0.69099999999999995},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.438},"max":{"1min":0.85499999999999998,"5min":0.95399999999999996,"15min":0.998},"last":2.0110000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71499999999999997,"15min":0.71199999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.43099999999999999},"max":{"1min":0.88500000000000001,"5min":0.93300000000000005,"15min":1.099},"last":2.8090000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575762,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59540,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312300,"statfs":{"total":96636764160,"available":96575795200,"internally_reserved":0,"allocated":2232320,"data_stored":1125094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.39400000000000002,"15min":0.38700000000000001},"min":{"1min":0.27400000000000002,"5min":0.25700000000000001,"15min":0.248},"max":{"1min":0.54400000000000004,"5min":0.61199999999999999,"15min":0.621},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.41199999999999998,"15min":0.40500000000000003},"min":{"1min":0.27100000000000002,"5min":0.253,"15min":0.24099999999999999},"max":{"1min":0.51400000000000001,"5min":0.66000000000000003,"15min":0.67100000000000004},"last":0.33500000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.42099999999999999,"15min":0.41299999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.58999999999999997,"5min":0.64100000000000001,"15min":0.64900000000000002},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.443,"15min":0.441},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.56699999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.36199999999999999}]},{"osd":2,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.5,"15min":0.49299999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.629,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.503,"15min":0.501},"min":{"1min":0.38,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.66200000000000003,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.46999999999999997}]},{"osd":4,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65800000000000003,"15min":0.63600000000000001},"min":{"1min":0.49199999999999999,"5min":0.46400000000000002,"15min":0.34200000000000003},"max":{"1min":0.86599999999999999,"5min":0.88400000000000001,"15min":0.96799999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.66100000000000003,"15min":0.65700000000000003},"min":{"1min":0.47299999999999998,"5min":0.436,"15min":0.436},"max":{"1min":0.76200000000000001,"5min":0.97199999999999998,"15min":0.97299999999999998},"last":0.49299999999999999}]},{"osd":5,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69199999999999995,"15min":0.68200000000000005},"min":{"1min":0.48299999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.89400000000000002,"5min":0.97699999999999998,"15min":1.024},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.71299999999999997,"15min":0.70099999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.41999999999999998},"max":{"1min":0.81399999999999995,"5min":0.98699999999999999,"15min":1.0069999999999999},"last":0.52500000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.68700000000000006,"15min":0.67200000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45200000000000001},"max":{"1min":0.79500000000000004,"5min":1.0149999999999999,"15min":1.0569999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.69299999999999995,"15min":0.67900000000000005},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.38900000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.98899999999999999},"last":0.56100000000000005}]},{"osd":7,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.68600000000000005,"15min":0.68000000000000005},"min":{"1min":0.50800000000000001,"5min":0.47599999999999998,"15min":0.42199999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":1.0009999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.72299999999999998,"15min":0.70299999999999996},"min":{"1min":0.495,"5min":0.47999999999999998,"15min":0.46800000000000003},"max":{"1min":0.88,"5min":1.028,"15min":1.028},"last":0.54200000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575762,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41492,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330348,"statfs":{"total":96636764160,"available":96594276352,"internally_reserved":0,"allocated":2236416,"data_stored":1126042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59399999999999997,"15min":0.59999999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.745,"5min":0.78900000000000003,"15min":1.2609999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.45800000000000002,"5min":0.40699999999999997,"15min":0.376},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.54800000000000004}]},{"osd":1,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64700000000000002,"15min":0.64900000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.35199999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.86699999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.64000000000000001},"min":{"1min":0.45100000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.78100000000000003,"5min":0.79300000000000004,"15min":0.82499999999999996},"last":0.67100000000000004}]},{"osd":2,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59599999999999997,"15min":0.60699999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.33600000000000002},"max":{"1min":0.81399999999999995,"5min":0.82399999999999995,"15min":1.5109999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.622,"15min":0.623},"min":{"1min":0.436,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.65500000000000003}]},{"osd":3,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.60399999999999998,"15min":0.60299999999999998},"min":{"1min":0.39100000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73399999999999999,"5min":0.78600000000000003,"15min":1.1439999999999999},"last":0.623},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63300000000000001,"15min":0.63400000000000001},"min":{"1min":0.38800000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":1.319},"last":0.52900000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48899999999999999,"15min":0.504},"min":{"1min":0.35199999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66900000000000004,"5min":0.748,"15min":0.878},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51800000000000002,"15min":0.52200000000000002},"min":{"1min":0.38300000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":1.3839999999999999},"last":0.40799999999999997}]},{"osd":6,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.308},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":1.6850000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40500000000000003,"5min":0.39900000000000002,"15min":0.36699999999999999},"max":{"1min":0.755,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.48899999999999999}]},{"osd":7,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.374,"5min":0.36899999999999999,"15min":0.35099999999999998},"max":{"1min":0.79000000000000004,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.61399999999999999,"15min":0.60299999999999998},"min":{"1min":0.46999999999999997,"5min":0.41499999999999998,"15min":0.35799999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.7410000000000001},"last":0.439}]}]},{"osd":5,"up_from":13,"seq":55834575762,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59524,"kb_used_data":2168,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312316,"statfs":{"total":96636764160,"available":96575811584,"internally_reserved":0,"allocated":2220032,"data_stored":1123459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59899999999999998,"15min":0.59199999999999997},"min":{"1min":0.46700000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.75900000000000001,"5min":0.79100000000000004,"15min":0.81299999999999994},"last":0.755},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60299999999999998,"15min":0.59499999999999997},"min":{"1min":0.39300000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81899999999999995,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.64900000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.60399999999999998,"15min":0.59499999999999997},"min":{"1min":0.45000000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.76400000000000001,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58899999999999997,"15min":0.59199999999999997},"min":{"1min":0.36899999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78700000000000003},"last":0.66500000000000004}]},{"osd":2,"last update":"Sat Apr 6 13:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60899999999999999,"15min":0.60599999999999998},"min":{"1min":0.433,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73599999999999999,"5min":0.80300000000000005,"15min":0.81999999999999995},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59999999999999998,"15min":0.59599999999999997},"min":{"1min":0.48199999999999998,"5min":0.38600000000000001,"15min":0.35899999999999999},"max":{"1min":0.72699999999999998,"5min":0.76200000000000001,"15min":0.76600000000000001},"last":0.58399999999999996}]},{"osd":3,"last update":"Sat Apr 6 13:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61599999999999999,"15min":0.60599999999999998},"min":{"1min":0.45000000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.63200000000000001,"15min":0.625},"min":{"1min":0.52600000000000002,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.80100000000000005,"5min":0.82599999999999996,"15min":0.82899999999999996},"last":0.69699999999999995}]},{"osd":4,"last update":"Sat Apr 6 13:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.502,"15min":0.499},"min":{"1min":0.373,"5min":0.33000000000000002,"15min":0.32100000000000001},"max":{"1min":0.66300000000000003,"5min":0.72499999999999998,"15min":0.85199999999999998},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.56000000000000005,"15min":0.55300000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.29099999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.56200000000000006}]},{"osd":6,"last update":"Sat Apr 6 13:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.51200000000000001,"15min":0.504},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.29699999999999999},"max":{"1min":0.67300000000000004,"5min":0.80000000000000004,"15min":0.82699999999999996},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.52100000000000002,"15min":0.50600000000000001},"min":{"1min":0.36599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.80300000000000005},"last":0.52500000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.53300000000000003,"15min":0.52200000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.314},"max":{"1min":0.68600000000000005,"5min":0.78300000000000003,"15min":0.80900000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.63100000000000001,"15min":0.61799999999999999},"min":{"1min":0.42099999999999999,"5min":0.41599999999999998,"15min":0.376},"max":{"1min":0.79700000000000004,"5min":0.84599999999999997,"15min":0.84699999999999998},"last":0.73099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:28:45.367 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-06T13:28:45.367 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-06T13:28:45.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.532+0000 7f973dc0c700 1 Processor -- start 2024-04-06T13:28:45.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.532+0000 7f973dc0c700 1 -- start start 2024-04-06T13:28:45.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.532+0000 7f973dc0c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f973815d7b0 0x7f97380b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:45.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.532+0000 7f973dc0c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97380b3990 con 0x7f973815e500 2024-04-06T13:28:45.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.533+0000 7f973dc0c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97380b3060 con 0x7f973806a730 2024-04-06T13:28:45.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.533+0000 7f973dc0c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f973815e950 con 0x7f973815d7b0 2024-04-06T13:28:45.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.533+0000 7f97377fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f973815d7b0 0x7f97380b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:45.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.533+0000 7f9737fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f973815e500 0x7f97380717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55888/0 (socket says 172.21.15.189:55888) 2024-04-06T13:28:45.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.533+0000 7f9737fff700 1 -- 172.21.15.189:0/2388024976 learned_addr learned my addr 172.21.15.189:0/2388024976 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:45.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.533+0000 7f97377fe700 1 --2- 172.21.15.189:0/2388024976 >> v2:172.21.15.189:3300/0 conn(0x7f973815d7b0 0x7f97380b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59084/0 (socket says 172.21.15.189:59084) 2024-04-06T13:28:45.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.533+0000 7f97377fe700 1 -- 172.21.15.189:0/2388024976 >> v1:172.21.15.192:6789/0 conn(0x7f973806a730 legacy=0x7f973815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:45.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.533+0000 7f97377fe700 1 -- 172.21.15.189:0/2388024976 >> v1:172.21.15.189:6789/0 conn(0x7f973815e500 legacy=0x7f97380717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:45.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.533+0000 7f97377fe700 1 -- 172.21.15.189:0/2388024976 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97380b3340 con 0x7f973815d7b0 2024-04-06T13:28:45.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.533+0000 7f97367fc700 1 -- 172.21.15.189:0/2388024976 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733008747 0 0) 0x7f97380b3990 con 0x7f973815e500 2024-04-06T13:28:45.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.533+0000 7f97377fe700 1 --2- 172.21.15.189:0/2388024976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f973815d7b0 0x7f97380b3e20 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f9728009840 tx=0x7f9728009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c199b47d9f2729 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:45.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.533+0000 7f97367fc700 1 -- 172.21.15.189:0/2388024976 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9728004230 con 0x7f973815d7b0 2024-04-06T13:28:45.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.533+0000 7f97367fc700 1 -- 172.21.15.189:0/2388024976 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9728004390 con 0x7f973815d7b0 2024-04-06T13:28:45.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.533+0000 7f97367fc700 1 -- 172.21.15.189:0/2388024976 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97280036a0 con 0x7f973815d7b0 2024-04-06T13:28:45.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.534+0000 7f973dc0c700 1 -- 172.21.15.189:0/2388024976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f973815d7b0 msgr2=0x7f97380b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:45.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.534+0000 7f973dc0c700 1 --2- 172.21.15.189:0/2388024976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f973815d7b0 0x7f97380b3e20 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f9728009840 tx=0x7f9728009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:45.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.534+0000 7f973dc0c700 1 -- 172.21.15.189:0/2388024976 shutdown_connections 2024-04-06T13:28:45.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.534+0000 7f973dc0c700 1 --2- 172.21.15.189:0/2388024976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f973815d7b0 0x7f97380b3e20 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.534+0000 7f973dc0c700 1 -- 172.21.15.189:0/2388024976 >> 172.21.15.189:0/2388024976 conn(0x7f97380d00b0 msgr2=0x7f97380d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:45.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.534+0000 7f973dc0c700 1 -- 172.21.15.189:0/2388024976 shutdown_connections 2024-04-06T13:28:45.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.534+0000 7f973dc0c700 1 -- 172.21.15.189:0/2388024976 wait complete. 2024-04-06T13:28:45.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.535+0000 7f973dc0c700 1 Processor -- start 2024-04-06T13:28:45.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.535+0000 7f973dc0c700 1 -- start start 2024-04-06T13:28:45.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.536+0000 7f973dc0c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f973806a730 0x7f97380c51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:45.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.536+0000 7f973dc0c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f973815d7b0 0x7f97380cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:45.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.536+0000 7f973dc0c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f973815e500 0x7f97380ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:45.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.536+0000 7f973dc0c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97380cef70 con 0x7f973815d7b0 2024-04-06T13:28:45.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.536+0000 7f973dc0c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97380cea20 con 0x7f973815e500 2024-04-06T13:28:45.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.536+0000 7f973dc0c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97380cec90 con 0x7f973806a730 2024-04-06T13:28:45.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.536+0000 7f9736ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f973815d7b0 0x7f97380cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:45.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.536+0000 7f9736ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f973815d7b0 0x7f97380cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59094/0 (socket says 172.21.15.189:59094) 2024-04-06T13:28:45.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.536+0000 7f9736ffd700 1 -- 172.21.15.189:0/238433472 learned_addr learned my addr 172.21.15.189:0/238433472 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:45.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.536+0000 7f97377fe700 1 --2- 172.21.15.189:0/238433472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f973806a730 0x7f97380c51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:45.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.536+0000 7f9737fff700 1 --2- 172.21.15.189:0/238433472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f973815e500 0x7f97380ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:45.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.536+0000 7f9736ffd700 1 -- 172.21.15.189:0/238433472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f973806a730 msgr2=0x7f97380c51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:45.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.536+0000 7f9736ffd700 1 --2- 172.21.15.189:0/238433472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f973806a730 0x7f97380c51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.536+0000 7f9736ffd700 1 -- 172.21.15.189:0/238433472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f973815e500 msgr2=0x7f97380ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:45.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.536+0000 7f9736ffd700 1 --2- 172.21.15.189:0/238433472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f973815e500 0x7f97380ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.536+0000 7f9736ffd700 1 -- 172.21.15.189:0/238433472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97280094a0 con 0x7f973815d7b0 2024-04-06T13:28:45.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.536+0000 7f9736ffd700 1 --2- 172.21.15.189:0/238433472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f973815d7b0 0x7f97380cdfa0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f972c00c500 tx=0x7f972c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:45.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.536+0000 7f971ffff700 1 -- 172.21.15.189:0/238433472 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c011790 con 0x7f973815d7b0 2024-04-06T13:28:45.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.537+0000 7f971ffff700 1 -- 172.21.15.189:0/238433472 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972c0118f0 con 0x7f973815d7b0 2024-04-06T13:28:45.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.537+0000 7f973dc0c700 1 -- 172.21.15.189:0/238433472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97381498b0 con 0x7f973815d7b0 2024-04-06T13:28:45.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.537+0000 7f971ffff700 1 -- 172.21.15.189:0/238433472 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c011ca0 con 0x7f973815d7b0 2024-04-06T13:28:45.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.537+0000 7f973dc0c700 1 -- 172.21.15.189:0/238433472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9738149e70 con 0x7f973815d7b0 2024-04-06T13:28:45.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.537+0000 7f973dc0c700 1 -- 172.21.15.189:0/238433472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9738067bb0 con 0x7f973815d7b0 2024-04-06T13:28:45.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.538+0000 7f971ffff700 1 -- 172.21.15.189:0/238433472 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f972c01c460 con 0x7f973815d7b0 2024-04-06T13:28:45.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.538+0000 7f971ffff700 1 --2- 172.21.15.189:0/238433472 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f972406adb0 0x7f972406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:45.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.538+0000 7f971ffff700 1 -- 172.21.15.189:0/238433472 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f972c089450 con 0x7f973815d7b0 2024-04-06T13:28:45.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.538+0000 7f97377fe700 1 --2- 172.21.15.189:0/238433472 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f972406adb0 0x7f972406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:45.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.539+0000 7f97377fe700 1 --2- 172.21.15.189:0/238433472 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f972406adb0 0x7f972406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9728009b90 tx=0x7f9728008030 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:45.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.543+0000 7f971ffff700 1 -- 172.21.15.189:0/238433472 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f972c0543b0 con 0x7f973815d7b0 2024-04-06T13:28:45.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.820+0000 7f973dc0c700 1 -- 172.21.15.189:0/238433472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f9738074990 con 0x7f973815d7b0 2024-04-06T13:28:45.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.824+0000 7f971ffff700 1 -- 172.21.15.189:0/238433472 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v42) v1 ==== 190+0+0 (secure 0 0 0) 0x7f972c0579d0 con 0x7f973815d7b0 2024-04-06T13:28:45.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.826+0000 7f973dc0c700 1 -- 172.21.15.189:0/238433472 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f972406adb0 msgr2=0x7f972406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:45.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.826+0000 7f973dc0c700 1 --2- 172.21.15.189:0/238433472 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f972406adb0 0x7f972406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9728009b90 tx=0x7f9728008030 comp rx=0 tx=0).stop 2024-04-06T13:28:45.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.826+0000 7f973dc0c700 1 -- 172.21.15.189:0/238433472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f973815d7b0 msgr2=0x7f97380cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:45.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.826+0000 7f973dc0c700 1 --2- 172.21.15.189:0/238433472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f973815d7b0 0x7f97380cdfa0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f972c00c500 tx=0x7f972c00c8c0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.826+0000 7f973dc0c700 1 -- 172.21.15.189:0/238433472 shutdown_connections 2024-04-06T13:28:45.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.826+0000 7f973dc0c700 1 --2- 172.21.15.189:0/238433472 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f972406adb0 0x7f972406d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.826+0000 7f973dc0c700 1 --2- 172.21.15.189:0/238433472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f973815e500 0x7f97380ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.826+0000 7f973dc0c700 1 --2- 172.21.15.189:0/238433472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f973806a730 0x7f97380c51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.826+0000 7f973dc0c700 1 --2- 172.21.15.189:0/238433472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f973815d7b0 0x7f97380cdfa0 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.826+0000 7f973dc0c700 1 -- 172.21.15.189:0/238433472 >> 172.21.15.189:0/238433472 conn(0x7f97380d00b0 msgr2=0x7f97380ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:45.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.826+0000 7f973dc0c700 1 -- 172.21.15.189:0/238433472 shutdown_connections 2024-04-06T13:28:45.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.826+0000 7f973dc0c700 1 -- 172.21.15.189:0/238433472 wait complete. 2024-04-06T13:28:45.829 INFO:teuthology.orchestra.run.smithi189.stderr:updated caps for client.0 2024-04-06T13:28:45.840 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:28:45.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.961+0000 7ff5ec015700 1 Processor -- start 2024-04-06T13:28:45.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.961+0000 7ff5ec015700 1 -- start start 2024-04-06T13:28:45.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.962+0000 7ff5ec015700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff5e40d11b0 0x7ff5e40d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:45.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.962+0000 7ff5ec015700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff5e40cb870 0x7ff5e415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:45.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.962+0000 7ff5ec015700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e40d1ad0 con 0x7ff5e40cfa00 2024-04-06T13:28:45.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.962+0000 7ff5ec015700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e40ccdc0 con 0x7ff5e40d11b0 2024-04-06T13:28:45.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.962+0000 7ff5ec015700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e40d0b50 con 0x7ff5e40cb870 2024-04-06T13:28:45.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.962+0000 7ff5e9db1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff5e40d11b0 0x7ff5e40d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:45.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.962+0000 7ff5e9db1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff5e40d11b0 0x7ff5e40d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59096/0 (socket says 172.21.15.189:59096) 2024-04-06T13:28:45.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.962+0000 7ff5e9db1700 1 -- 172.21.15.189:0/3190884116 learned_addr learned my addr 172.21.15.189:0/3190884116 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:45.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.962+0000 7ff5e95b0700 1 --2- 172.21.15.189:0/3190884116 >> v2:172.21.15.192:3300/0 conn(0x7ff5e40cb870 0x7ff5e415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:45.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.962+0000 7ff5e9db1700 1 -- 172.21.15.189:0/3190884116 >> v2:172.21.15.192:3300/0 conn(0x7ff5e40cb870 msgr2=0x7ff5e415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:45.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.962+0000 7ff5e9db1700 1 --2- 172.21.15.189:0/3190884116 >> v2:172.21.15.192:3300/0 conn(0x7ff5e40cb870 0x7ff5e415c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.962+0000 7ff5e9db1700 1 -- 172.21.15.189:0/3190884116 >> v1:172.21.15.192:6789/0 conn(0x7ff5e40cfa00 legacy=0x7ff5e40cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:45.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.962+0000 7ff5e9db1700 1 -- 172.21.15.189:0/3190884116 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e40cbd80 con 0x7ff5e40d11b0 2024-04-06T13:28:45.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.963+0000 7ff5e9db1700 1 --2- 172.21.15.189:0/3190884116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5e40d11b0 0x7ff5e40d1590 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7ff5e404fb40 tx=0x7ff5d4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf38d4475b4b7e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:45.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.963+0000 7ff5e8daf700 1 -- 172.21.15.189:0/3190884116 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d400b870 con 0x7ff5e40d11b0 2024-04-06T13:28:45.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.963+0000 7ff5e8daf700 1 -- 172.21.15.189:0/3190884116 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d400b9d0 con 0x7ff5e40d11b0 2024-04-06T13:28:45.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.963+0000 7ff5e8daf700 1 -- 172.21.15.189:0/3190884116 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d400bd60 con 0x7ff5e40d11b0 2024-04-06T13:28:45.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.963+0000 7ff5ec015700 1 -- 172.21.15.189:0/3190884116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5e40d11b0 msgr2=0x7ff5e40d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:45.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.963+0000 7ff5ec015700 1 --2- 172.21.15.189:0/3190884116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5e40d11b0 0x7ff5e40d1590 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7ff5e404fb40 tx=0x7ff5d4009bd0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.963+0000 7ff5ec015700 1 -- 172.21.15.189:0/3190884116 shutdown_connections 2024-04-06T13:28:45.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.963+0000 7ff5ec015700 1 --2- 172.21.15.189:0/3190884116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5e40d11b0 0x7ff5e40d1590 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.963+0000 7ff5ec015700 1 --2- 172.21.15.189:0/3190884116 >> v2:172.21.15.192:3300/0 conn(0x7ff5e40cb870 0x7ff5e415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.963+0000 7ff5ec015700 1 -- 172.21.15.189:0/3190884116 >> 172.21.15.189:0/3190884116 conn(0x7ff5e40b2db0 msgr2=0x7ff5e40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:45.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.963+0000 7ff5ec015700 1 -- 172.21.15.189:0/3190884116 shutdown_connections 2024-04-06T13:28:45.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.963+0000 7ff5ec015700 1 -- 172.21.15.189:0/3190884116 wait complete. 2024-04-06T13:28:45.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.964+0000 7ff5ec015700 1 Processor -- start 2024-04-06T13:28:45.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.965+0000 7ff5ec015700 1 -- start start 2024-04-06T13:28:45.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.965+0000 7ff5ec015700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5e40cb870 0x7ff5e4155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:45.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.965+0000 7ff5ec015700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5e40cfa00 0x7ff5e41549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:45.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.965+0000 7ff5ec015700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5e40d11b0 0x7ff5e41573c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:45.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.965+0000 7ff5ec015700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e4154ee0 con 0x7ff5e40d11b0 2024-04-06T13:28:45.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.965+0000 7ff5ec015700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e40c82f0 con 0x7ff5e40cfa00 2024-04-06T13:28:45.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.965+0000 7ff5ec015700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e40c85d0 con 0x7ff5e40cb870 2024-04-06T13:28:45.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.965+0000 7ff5ea5b2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5e40d11b0 0x7ff5e41573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:45.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.965+0000 7ff5ea5b2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5e40d11b0 0x7ff5e41573c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59110/0 (socket says 172.21.15.189:59110) 2024-04-06T13:28:45.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.965+0000 7ff5ea5b2700 1 -- 172.21.15.189:0/50598684 learned_addr learned my addr 172.21.15.189:0/50598684 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:45.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.966+0000 7ff5e9db1700 1 --2- 172.21.15.189:0/50598684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5e40cb870 0x7ff5e4155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:45.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.966+0000 7ff5e95b0700 1 --2- 172.21.15.189:0/50598684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5e40cfa00 0x7ff5e41549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:45.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.966+0000 7ff5ea5b2700 1 -- 172.21.15.189:0/50598684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5e40cb870 msgr2=0x7ff5e4155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:45.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.966+0000 7ff5ea5b2700 1 --2- 172.21.15.189:0/50598684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5e40cb870 0x7ff5e4155130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.966+0000 7ff5ea5b2700 1 -- 172.21.15.189:0/50598684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5e40cfa00 msgr2=0x7ff5e41549a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:45.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.966+0000 7ff5ea5b2700 1 --2- 172.21.15.189:0/50598684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5e40cfa00 0x7ff5e41549a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:45.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.966+0000 7ff5ea5b2700 1 -- 172.21.15.189:0/50598684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d4009550 con 0x7ff5e40d11b0 2024-04-06T13:28:45.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.966+0000 7ff5e9db1700 1 --2- 172.21.15.189:0/50598684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5e40cb870 0x7ff5e4155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:28:45.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.966+0000 7ff5ea5b2700 1 --2- 172.21.15.189:0/50598684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5e40d11b0 0x7ff5e41573c0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc00cb50 tx=0x7ff5dc00cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:45.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.966+0000 7ff5da7fc700 1 -- 172.21.15.189:0/50598684 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc003c00 con 0x7ff5e40d11b0 2024-04-06T13:28:45.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.966+0000 7ff5da7fc700 1 -- 172.21.15.189:0/50598684 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5dc003d60 con 0x7ff5e40d11b0 2024-04-06T13:28:45.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.966+0000 7ff5da7fc700 1 -- 172.21.15.189:0/50598684 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc00ab70 con 0x7ff5e40d11b0 2024-04-06T13:28:45.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.966+0000 7ff5ec015700 1 -- 172.21.15.189:0/50598684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e40c88b0 con 0x7ff5e40d11b0 2024-04-06T13:28:45.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.966+0000 7ff5ec015700 1 -- 172.21.15.189:0/50598684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e40c8e70 con 0x7ff5e40d11b0 2024-04-06T13:28:45.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.967+0000 7ff5ec015700 1 -- 172.21.15.189:0/50598684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e4074990 con 0x7ff5e40d11b0 2024-04-06T13:28:45.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.967+0000 7ff5da7fc700 1 -- 172.21.15.189:0/50598684 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7ff5dc00acd0 con 0x7ff5e40d11b0 2024-04-06T13:28:45.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.968+0000 7ff5da7fc700 1 --2- 172.21.15.189:0/50598684 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff5c806adb0 0x7ff5c806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:45.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.968+0000 7ff5da7fc700 1 -- 172.21.15.189:0/50598684 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff5dc089780 con 0x7ff5e40d11b0 2024-04-06T13:28:45.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.968+0000 7ff5e9db1700 1 --2- 172.21.15.189:0/50598684 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff5c806adb0 0x7ff5c806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:45.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.968+0000 7ff5e9db1700 1 --2- 172.21.15.189:0/50598684 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff5c806adb0 0x7ff5c806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4000c00 tx=0x7ff5d4008030 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:45.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:45.972+0000 7ff5da7fc700 1 -- 172.21.15.189:0/50598684 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff5dc0545b0 con 0x7ff5e40d11b0 2024-04-06T13:28:46.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.208+0000 7ff5ec015700 1 -- 172.21.15.189:0/50598684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5e40b0a00 con 0x7ff5e40d11b0 2024-04-06T13:28:46.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.209+0000 7ff5da7fc700 1 -- 172.21.15.189:0/50598684 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4750 (secure 0 0 0) 0x7ff5dc057bd0 con 0x7ff5e40d11b0 2024-04-06T13:28:46.210 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:46.210 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":78,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/3388956983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3388956983},{"type":"v1","addr":"172.21.15.192:6834","nonce":3388956983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":8845,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1844918354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":1844918354},{"type":"v1","addr":"172.21.15.192:6835","nonce":1844918354}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":9618,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3742852104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3742852104},{"type":"v1","addr":"172.21.15.189:6836","nonce":3742852104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:28:40.813074+0000","modified":"2024-04-06T13:28:43.827822+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9621":{"gid":9621,"name":"c","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/2089191015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2089191015},{"type":"v1","addr":"172.21.15.189:6837","nonce":2089191015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-06T13:28:46.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.211+0000 7ff5ec015700 1 -- 172.21.15.189:0/50598684 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff5c806adb0 msgr2=0x7ff5c806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:46.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.211+0000 7ff5ec015700 1 --2- 172.21.15.189:0/50598684 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff5c806adb0 0x7ff5c806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4000c00 tx=0x7ff5d4008030 comp rx=0 tx=0).stop 2024-04-06T13:28:46.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.211+0000 7ff5ec015700 1 -- 172.21.15.189:0/50598684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5e40d11b0 msgr2=0x7ff5e41573c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:46.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.211+0000 7ff5ec015700 1 --2- 172.21.15.189:0/50598684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5e40d11b0 0x7ff5e41573c0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc00cb50 tx=0x7ff5dc00cf10 comp rx=0 tx=0).stop 2024-04-06T13:28:46.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.211+0000 7ff5ec015700 1 -- 172.21.15.189:0/50598684 shutdown_connections 2024-04-06T13:28:46.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.211+0000 7ff5ec015700 1 --2- 172.21.15.189:0/50598684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5e40d11b0 0x7ff5e41573c0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.211+0000 7ff5ec015700 1 --2- 172.21.15.189:0/50598684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5e40cb870 0x7ff5e4155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.211+0000 7ff5ec015700 1 --2- 172.21.15.189:0/50598684 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff5c806adb0 0x7ff5c806d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.211+0000 7ff5ec015700 1 --2- 172.21.15.189:0/50598684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5e40cfa00 0x7ff5e41549a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.211+0000 7ff5ec015700 1 -- 172.21.15.189:0/50598684 >> 172.21.15.189:0/50598684 conn(0x7ff5e40b2db0 msgr2=0x7ff5e40b0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:46.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.211+0000 7ff5ec015700 1 -- 172.21.15.189:0/50598684 shutdown_connections 2024-04-06T13:28:46.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.211+0000 7ff5ec015700 1 -- 172.21.15.189:0/50598684 wait complete. 2024-04-06T13:28:46.213 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 78 2024-04-06T13:28:46.388 INFO:tasks.ceph.mds.c.smithi189.stderr:2024-04-06T13:28:46.388+0000 7fd24da6c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T13:28:46.628 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-06T13:28:40.813074+0000', 'modified': '2024-04-06T13:28:43.827822+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9621}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9621': {'gid': 9621, 'name': 'c', 'rank': 0, 'incarnation': 74, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.189:6837/2089191015', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6835', 'nonce': 2089191015}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 2089191015}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-06T13:28:46.628 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-06T13:28:46.628 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-06T13:28:46.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.752+0000 7f70679fd700 1 Processor -- start 2024-04-06T13:28:46.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.752+0000 7f70679fd700 1 -- start start 2024-04-06T13:28:46.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.753+0000 7f70679fd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7060146dd0 0x7f706006d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.753+0000 7f70679fd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f70600cedf0 0x7f70600cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.753+0000 7f70679fd700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70600cf730 con 0x7f70600d17a0 2024-04-06T13:28:46.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.753+0000 7f70679fd700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f706006beb0 con 0x7f70600cedf0 2024-04-06T13:28:46.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.753+0000 7f70679fd700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f706006c0a0 con 0x7f7060146dd0 2024-04-06T13:28:46.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.753+0000 7f7065f9a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f70600cedf0 0x7f70600cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.753+0000 7f7065f9a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f70600cedf0 0x7f70600cf1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59118/0 (socket says 172.21.15.189:59118) 2024-04-06T13:28:46.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.753+0000 7f7065f9a700 1 -- 172.21.15.189:0/3201097442 learned_addr learned my addr 172.21.15.189:0/3201097442 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:46.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.753+0000 7f7064f98700 1 --2- 172.21.15.189:0/3201097442 >> v2:172.21.15.192:3300/0 conn(0x7f7060146dd0 0x7f706006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.753+0000 7f7065f9a700 1 -- 172.21.15.189:0/3201097442 >> v2:172.21.15.192:3300/0 conn(0x7f7060146dd0 msgr2=0x7f706006d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:46.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.753+0000 7f7065f9a700 1 --2- 172.21.15.189:0/3201097442 >> v2:172.21.15.192:3300/0 conn(0x7f7060146dd0 0x7f706006d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.753+0000 7f7065f9a700 1 -- 172.21.15.189:0/3201097442 >> v1:172.21.15.192:6789/0 conn(0x7f70600d17a0 legacy=0x7f706006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:46.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.753+0000 7f7065f9a700 1 -- 172.21.15.189:0/3201097442 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706006c380 con 0x7f70600cedf0 2024-04-06T13:28:46.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.754+0000 7f7065f9a700 1 --2- 172.21.15.189:0/3201097442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70600cedf0 0x7f70600cf1f0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f70600d2300 tx=0x7f705800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=febd9f95ab7dbacc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:46.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.754+0000 7f7053fff700 1 -- 172.21.15.189:0/3201097442 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705800e070 con 0x7f70600cedf0 2024-04-06T13:28:46.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.754+0000 7f7053fff700 1 -- 172.21.15.189:0/3201097442 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7058003a20 con 0x7f70600cedf0 2024-04-06T13:28:46.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.754+0000 7f7053fff700 1 -- 172.21.15.189:0/3201097442 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7058003dd0 con 0x7f70600cedf0 2024-04-06T13:28:46.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.754+0000 7f70679fd700 1 -- 172.21.15.189:0/3201097442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70600cedf0 msgr2=0x7f70600cf1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:46.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.754+0000 7f70679fd700 1 --2- 172.21.15.189:0/3201097442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70600cedf0 0x7f70600cf1f0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f70600d2300 tx=0x7f705800b330 comp rx=0 tx=0).stop 2024-04-06T13:28:46.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.754+0000 7f70679fd700 1 -- 172.21.15.189:0/3201097442 shutdown_connections 2024-04-06T13:28:46.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.754+0000 7f70679fd700 1 --2- 172.21.15.189:0/3201097442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70600cedf0 0x7f70600cf1f0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.754+0000 7f70679fd700 1 --2- 172.21.15.189:0/3201097442 >> v2:172.21.15.192:3300/0 conn(0x7f7060146dd0 0x7f706006d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.754+0000 7f70679fd700 1 -- 172.21.15.189:0/3201097442 >> 172.21.15.189:0/3201097442 conn(0x7f70600b2db0 msgr2=0x7f70600b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:46.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.754+0000 7f70679fd700 1 -- 172.21.15.189:0/3201097442 shutdown_connections 2024-04-06T13:28:46.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.754+0000 7f70679fd700 1 -- 172.21.15.189:0/3201097442 wait complete. 2024-04-06T13:28:46.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.755+0000 7f70679fd700 1 Processor -- start 2024-04-06T13:28:46.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.756+0000 7f70679fd700 1 -- start start 2024-04-06T13:28:46.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.756+0000 7f70679fd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70600cedf0 0x7f70600b6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.756+0000 7f70679fd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70600d17a0 0x7f706014a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.756+0000 7f70679fd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7060146dd0 0x7f70601f1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.756+0000 7f70679fd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7060147480 con 0x7f7060146dd0 2024-04-06T13:28:46.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.756+0000 7f70679fd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706014ac90 con 0x7f70600d17a0 2024-04-06T13:28:46.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.756+0000 7f70679fd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706014af70 con 0x7f70600cedf0 2024-04-06T13:28:46.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.756+0000 7f7065f9a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7060146dd0 0x7f70601f1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.756+0000 7f7065f9a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7060146dd0 0x7f70601f1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59130/0 (socket says 172.21.15.189:59130) 2024-04-06T13:28:46.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.756+0000 7f7065f9a700 1 -- 172.21.15.189:0/3186980553 learned_addr learned my addr 172.21.15.189:0/3186980553 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:46.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.757+0000 7f7064f98700 1 --2- 172.21.15.189:0/3186980553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70600d17a0 0x7f706014a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.757+0000 7f7065799700 1 --2- 172.21.15.189:0/3186980553 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70600cedf0 0x7f70600b6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.757+0000 7f7065f9a700 1 -- 172.21.15.189:0/3186980553 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70600cedf0 msgr2=0x7f70600b6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:46.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.757+0000 7f7065f9a700 1 --2- 172.21.15.189:0/3186980553 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70600cedf0 0x7f70600b6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.757+0000 7f7065f9a700 1 -- 172.21.15.189:0/3186980553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70600d17a0 msgr2=0x7f706014a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:46.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.757+0000 7f7065f9a700 1 --2- 172.21.15.189:0/3186980553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70600d17a0 0x7f706014a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.757+0000 7f7065f9a700 1 -- 172.21.15.189:0/3186980553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7058009b30 con 0x7f7060146dd0 2024-04-06T13:28:46.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.757+0000 7f7065f9a700 1 --2- 172.21.15.189:0/3186980553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7060146dd0 0x7f70601f1f70 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f7058015040 tx=0x7f7058004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:46.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.757+0000 7f7051ffb700 1 -- 172.21.15.189:0/3186980553 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7058004440 con 0x7f7060146dd0 2024-04-06T13:28:46.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.757+0000 7f7051ffb700 1 -- 172.21.15.189:0/3186980553 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70580045a0 con 0x7f7060146dd0 2024-04-06T13:28:46.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.757+0000 7f70679fd700 1 -- 172.21.15.189:0/3186980553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f706014b250 con 0x7f7060146dd0 2024-04-06T13:28:46.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.757+0000 7f7051ffb700 1 -- 172.21.15.189:0/3186980553 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7058004950 con 0x7f7060146dd0 2024-04-06T13:28:46.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.757+0000 7f70679fd700 1 -- 172.21.15.189:0/3186980553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70600b6bb0 con 0x7f7060146dd0 2024-04-06T13:28:46.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.759+0000 7f7051ffb700 1 -- 172.21.15.189:0/3186980553 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7058012430 con 0x7f7060146dd0 2024-04-06T13:28:46.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.759+0000 7f7051ffb700 1 --2- 172.21.15.189:0/3186980553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f704406adb0 0x7f704406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.759+0000 7f7051ffb700 1 -- 172.21.15.189:0/3186980553 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f705808ae10 con 0x7f7060146dd0 2024-04-06T13:28:46.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.759+0000 7f7065799700 1 --2- 172.21.15.189:0/3186980553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f704406adb0 0x7f704406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.760+0000 7f7065799700 1 --2- 172.21.15.189:0/3186980553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f704406adb0 0x7f704406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f705400a7b0 tx=0x7f7054008040 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:46.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.761+0000 7f7073bb1b80 1 Processor -- start 2024-04-06T13:28:46.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.761+0000 7f7073bb1b80 1 -- start start 2024-04-06T13:28:46.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.762+0000 7f7073bb1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373794c20 0x558373798a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.762+0000 7f7073bb1b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x558373799fc0 0x5583737705f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.762+0000 7f7073bb1b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55837376f980 0x558373770b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.762+0000 7f7073bb1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583736c9940 con 0x558373794c20 2024-04-06T13:28:46.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.762+0000 7f7073bb1b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583736c7300 con 0x55837376f980 2024-04-06T13:28:46.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.762+0000 7f7073bb1b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x5583736c2bf0 con 0x558373799fc0 2024-04-06T13:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.762+0000 7f7065799700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373794c20 0x558373798a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.762+0000 7f7065799700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373794c20 0x558373798a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59136/0 (socket says 172.21.15.189:59136) 2024-04-06T13:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.762+0000 7f7065799700 1 -- 172.21.15.189:0/481410469 learned_addr learned my addr 172.21.15.189:0/481410469 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.762+0000 7f7064f98700 1 --2- 172.21.15.189:0/481410469 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x558373799fc0 0x5583737705f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.762+0000 7f7065f9a700 1 --2- 172.21.15.189:0/481410469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55837376f980 0x558373770b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.762+0000 7f7065799700 1 -- 172.21.15.189:0/481410469 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x558373799fc0 msgr2=0x5583737705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:46.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.762+0000 7f7065799700 1 --2- 172.21.15.189:0/481410469 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x558373799fc0 0x5583737705f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.762+0000 7f7065799700 1 -- 172.21.15.189:0/481410469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55837376f980 msgr2=0x558373770b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:46.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.762+0000 7f7065799700 1 --2- 172.21.15.189:0/481410469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55837376f980 0x558373770b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.762+0000 7f7065799700 1 -- 172.21.15.189:0/481410469 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583734f0b10 con 0x558373794c20 2024-04-06T13:28:46.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.763+0000 7f7065799700 1 --2- 172.21.15.189:0/481410469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373794c20 0x558373798a60 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f70540138c0 tx=0x7f705400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76b988c842a1e57c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:46.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.763+0000 7f704affd700 1 -- 172.21.15.189:0/481410469 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7054021070 con 0x558373794c20 2024-04-06T13:28:46.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.763+0000 7f704affd700 1 -- 172.21.15.189:0/481410469 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7054004020 con 0x558373794c20 2024-04-06T13:28:46.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.763+0000 7f704affd700 1 -- 172.21.15.189:0/481410469 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7054004390 con 0x558373794c20 2024-04-06T13:28:46.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.763+0000 7f7073bb1b80 1 -- 172.21.15.189:0/481410469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373794c20 msgr2=0x558373798a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:46.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.763+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/481410469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373794c20 0x558373798a60 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f70540138c0 tx=0x7f705400f000 comp rx=0 tx=0).stop 2024-04-06T13:28:46.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.763+0000 7f7073bb1b80 1 -- 172.21.15.189:0/481410469 shutdown_connections 2024-04-06T13:28:46.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.763+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/481410469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55837376f980 0x558373770b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.763+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/481410469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373794c20 0x558373798a60 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.763+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/481410469 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x558373799fc0 0x5583737705f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.763+0000 7f7073bb1b80 1 -- 172.21.15.189:0/481410469 >> 172.21.15.189:0/481410469 conn(0x5583736d0650 msgr2=0x5583737b73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:46.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.763+0000 7f7073bb1b80 1 -- 172.21.15.189:0/481410469 shutdown_connections 2024-04-06T13:28:46.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.763+0000 7f7073bb1b80 1 -- 172.21.15.189:0/481410469 wait complete. 2024-04-06T13:28:46.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.764+0000 7f7073bb1b80 1 -- start start 2024-04-06T13:28:46.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.764+0000 7f7073bb1b80 1 Processor -- start 2024-04-06T13:28:46.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.765+0000 7f7073bb1b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55837376f980 0x5583737ae870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.765+0000 7f7073bb1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373794c20 0x55837379df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.765+0000 7f7073bb1b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558373799fc0 0x558373772040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.765+0000 7f7073bb1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55837365ed30 con 0x558373794c20 2024-04-06T13:28:46.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.765+0000 7f7073bb1b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55837365e9d0 con 0x558373799fc0 2024-04-06T13:28:46.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.765+0000 7f7073bb1b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x558373546650 con 0x55837376f980 2024-04-06T13:28:46.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.765+0000 7f7073bb1b80 10 client.0 _ll_register_callbacks cb 0x558373668420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T13:28:46.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.765+0000 7f7064f98700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373794c20 0x55837379df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.765+0000 7f7064f98700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373794c20 0x55837379df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59140/0 (socket says 172.21.15.189:59140) 2024-04-06T13:28:46.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.765+0000 7f7064f98700 1 -- 172.21.15.189:0/2154302185 learned_addr learned my addr 172.21.15.189:0/2154302185 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:46.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.765+0000 7f7065799700 1 --2- 172.21.15.189:0/2154302185 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55837376f980 0x5583737ae870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.765+0000 7f7065f9a700 1 --2- 172.21.15.189:0/2154302185 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558373799fc0 0x558373772040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.765+0000 7f7064f98700 1 -- 172.21.15.189:0/2154302185 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55837376f980 msgr2=0x5583737ae870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.765+0000 7f7064f98700 1 --2- 172.21.15.189:0/2154302185 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55837376f980 0x5583737ae870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.765+0000 7f7064f98700 1 -- 172.21.15.189:0/2154302185 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558373799fc0 msgr2=0x558373772040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.765+0000 7f7064f98700 1 --2- 172.21.15.189:0/2154302185 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558373799fc0 0x558373772040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.765+0000 7f7064f98700 1 -- 172.21.15.189:0/2154302185 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70540135b0 con 0x558373794c20 2024-04-06T13:28:46.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.765+0000 7f7065f9a700 1 --2- 172.21.15.189:0/2154302185 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558373799fc0 0x558373772040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.766+0000 7f7064f98700 1 --2- 172.21.15.189:0/2154302185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373794c20 0x55837379df90 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f705c010530 tx=0x7f705c010560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=166c0b2fc4f52487 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.766+0000 7f703ffff700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T13:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.766+0000 7f703ffff700 1 -- 172.21.15.189:0/2154302185 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c013430 con 0x558373794c20 2024-04-06T13:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.766+0000 7f703ffff700 1 -- 172.21.15.189:0/2154302185 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(178..178 src has 1..178) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f705c013590 con 0x558373794c20 2024-04-06T13:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.766+0000 7f7073bb1b80 1 -- 172.21.15.189:0/2154302185 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558373683330 con 0x558373794c20 2024-04-06T13:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.766+0000 7f703ffff700 1 -- 172.21.15.189:0/2154302185 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f705c013750 con 0x558373794c20 2024-04-06T13:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.766+0000 7f7073bb1b80 10 client.9768 fetch_fsmap learned FSMap version 78 2024-04-06T13:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.766+0000 7f7073bb1b80 1 -- 172.21.15.189:0/2154302185 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x558373688e50 con 0x558373794c20 2024-04-06T13:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.766+0000 7f703ffff700 1 -- 172.21.15.189:0/2154302185 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558373683330 con 0x558373794c20 2024-04-06T13:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.766+0000 7f703ffff700 1 -- 172.21.15.189:0/2154302185 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 78) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f705c01a020 con 0x558373794c20 2024-04-06T13:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.766+0000 7f7073bb1b80 10 client.9768 fetch_fsmap finished waiting for FSMap version 78 2024-04-06T13:28:46.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.766+0000 7f7073bb1b80 10 client.9768 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-06T13:28:46.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.766+0000 7f7073bb1b80 20 client.9768 populate_metadata read hostname 'smithi189' 2024-04-06T13:28:46.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.766+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/2154302185 >> [v2:172.21.15.189:6835/2089191015,v1:172.21.15.189:6837/2089191015] conn(0x558373793fb0 0x55837379fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.766+0000 7f7073bb1b80 4 client.9768 mds_command: new command op to 9621 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-06T13:28:46.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.766+0000 7f7073bb1b80 1 -- 172.21.15.189:0/2154302185 --> [v2:172.21.15.189:6835/2089191015,v1:172.21.15.189:6837/2089191015] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5583736bb6b0 con 0x558373793fb0 2024-04-06T13:28:46.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.766+0000 7f7065799700 1 --2- 172.21.15.189:0/2154302185 >> [v2:172.21.15.189:6835/2089191015,v1:172.21.15.189:6837/2089191015] conn(0x558373793fb0 0x55837379fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.767+0000 7f703ffff700 10 client.9768 ms_handle_connect on v2:172.21.15.189:6835/2089191015 2024-04-06T13:28:46.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.767+0000 7f7065799700 1 --2- 172.21.15.189:0/2154302185 >> [v2:172.21.15.189:6835/2089191015,v1:172.21.15.189:6837/2089191015] conn(0x558373793fb0 0x55837379fa90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=230ae240b2f064e server_cookie=b3a46c0090ce3f1a in_seq=0 out_seq=0 2024-04-06T13:28:46.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.767+0000 7f703ffff700 1 -- 172.21.15.189:0/2154302185 <== mds.0 v2:172.21.15.189:6835/2089191015 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5583736bb6b0 con 0x558373793fb0 2024-04-06T13:28:46.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.767+0000 7f703ffff700 10 client.9768 handle_command_reply: tid=0 2024-04-06T13:28:46.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.767+0000 7f7073bb1b80 1 client.9768 shutdown 2024-04-06T13:28:46.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.767+0000 7f7073bb1b80 1 -- 172.21.15.189:0/2154302185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373794c20 msgr2=0x55837379df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:46.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.767+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/2154302185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373794c20 0x55837379df90 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f705c010530 tx=0x7f705c010560 comp rx=0 tx=0).stop 2024-04-06T13:28:46.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.767+0000 7f7073bb1b80 1 -- 172.21.15.189:0/2154302185 shutdown_connections 2024-04-06T13:28:46.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.767+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/2154302185 >> [v2:172.21.15.189:6835/2089191015,v1:172.21.15.189:6837/2089191015] conn(0x558373793fb0 0x55837379fa90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.767+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/2154302185 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558373799fc0 0x558373772040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.767+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/2154302185 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55837376f980 0x5583737ae870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.767+0000 7f703ffff700 0 client.9768 ms_handle_reset on v2:172.21.15.189:6835/2089191015 2024-04-06T13:28:46.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.767+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/2154302185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373794c20 0x55837379df90 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.767+0000 7f7073bb1b80 1 -- 172.21.15.189:0/2154302185 >> 172.21.15.189:0/2154302185 conn(0x5583736d0650 msgr2=0x5583737b7370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:46.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.767+0000 7f7073bb1b80 1 -- 172.21.15.189:0/2154302185 shutdown_connections 2024-04-06T13:28:46.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.767+0000 7f7073bb1b80 1 -- 172.21.15.189:0/2154302185 wait complete. 2024-04-06T13:28:46.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.768+0000 7f7073bb1b80 20 client.9768 trim_cache size 0 max 16384 2024-04-06T13:28:46.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.785+0000 7f7073bb1b80 1 Processor -- start 2024-04-06T13:28:46.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.785+0000 7f7073bb1b80 1 -- start start 2024-04-06T13:28:46.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.786+0000 7f7073bb1b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55837376f980 0x558373770440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.786+0000 7f7073bb1b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558373794c20 0x558373798850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.786+0000 7f7073bb1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373799fc0 0x55837379de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.786+0000 7f7073bb1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55837379fb20 con 0x558373799fc0 2024-04-06T13:28:46.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.786+0000 7f7073bb1b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583737afa20 con 0x558373794c20 2024-04-06T13:28:46.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.786+0000 7f7073bb1b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x558373794550 con 0x55837376f980 2024-04-06T13:28:46.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.786+0000 7f7065f9a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373799fc0 0x55837379de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.786+0000 7f7065f9a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373799fc0 0x55837379de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59142/0 (socket says 172.21.15.189:59142) 2024-04-06T13:28:46.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.786+0000 7f7065f9a700 1 -- 172.21.15.189:0/576189717 learned_addr learned my addr 172.21.15.189:0/576189717 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:46.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.786+0000 7f7064f98700 1 --2- 172.21.15.189:0/576189717 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558373794c20 0x558373798850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.786+0000 7f7065f9a700 1 -- 172.21.15.189:0/576189717 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55837376f980 msgr2=0x558373770440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:46.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.786+0000 7f7065f9a700 1 --2- 172.21.15.189:0/576189717 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55837376f980 0x558373770440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.786+0000 7f7065f9a700 1 -- 172.21.15.189:0/576189717 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558373794c20 msgr2=0x558373798850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:46.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.786+0000 7f7065f9a700 1 --2- 172.21.15.189:0/576189717 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558373794c20 0x558373798850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.786+0000 7f7065f9a700 1 -- 172.21.15.189:0/576189717 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558373548c80 con 0x558373799fc0 2024-04-06T13:28:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.786+0000 7f7065f9a700 1 --2- 172.21.15.189:0/576189717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373799fc0 0x55837379de90 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f705800b370 tx=0x7f7058059f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67598059ac8e98a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.786+0000 7f704b7fe700 1 -- 172.21.15.189:0/576189717 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7058061c80 con 0x558373799fc0 2024-04-06T13:28:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.786+0000 7f704b7fe700 1 -- 172.21.15.189:0/576189717 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7058071120 con 0x558373799fc0 2024-04-06T13:28:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.786+0000 7f704b7fe700 1 -- 172.21.15.189:0/576189717 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7058071290 con 0x558373799fc0 2024-04-06T13:28:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.787+0000 7f7073bb1b80 1 -- 172.21.15.189:0/576189717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373799fc0 msgr2=0x55837379de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.787+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/576189717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373799fc0 0x55837379de90 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f705800b370 tx=0x7f7058059f90 comp rx=0 tx=0).stop 2024-04-06T13:28:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.787+0000 7f7073bb1b80 1 -- 172.21.15.189:0/576189717 shutdown_connections 2024-04-06T13:28:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.787+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/576189717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373799fc0 0x55837379de90 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.787+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/576189717 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55837376f980 0x558373770440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.787+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/576189717 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558373794c20 0x558373798850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.787+0000 7f7073bb1b80 1 -- 172.21.15.189:0/576189717 >> 172.21.15.189:0/576189717 conn(0x5583736d0650 msgr2=0x5583737d8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.787+0000 7f7073bb1b80 1 -- 172.21.15.189:0/576189717 shutdown_connections 2024-04-06T13:28:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.787+0000 7f7073bb1b80 1 -- 172.21.15.189:0/576189717 wait complete. 2024-04-06T13:28:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.788+0000 7f7073bb1b80 1 -- start start 2024-04-06T13:28:46.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.788+0000 7f7073bb1b80 1 Processor -- start 2024-04-06T13:28:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7073bb1b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55837376f980 0x5583738017b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7073bb1b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x558373794c20 0x5583737cf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7073bb1b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373799fc0 0x55837376b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7073bb1b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55837381c660 con 0x558373799fc0 2024-04-06T13:28:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7073bb1b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x558373804cf0 con 0x55837376f980 2024-04-06T13:28:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7073bb1b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x5583737cfe70 con 0x558373794c20 2024-04-06T13:28:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7073bb1b80 10 client.0 _ll_register_callbacks cb 0x558373668420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T13:28:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7065f9a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373799fc0 0x55837376b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7065f9a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373799fc0 0x55837376b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59148/0 (socket says 172.21.15.189:59148) 2024-04-06T13:28:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7065f9a700 1 -- 172.21.15.189:0/532660500 learned_addr learned my addr 172.21.15.189:0/532660500 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7064f98700 1 --2- 172.21.15.189:0/532660500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x558373794c20 0x5583737cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7065799700 1 --2- 172.21.15.189:0/532660500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55837376f980 0x5583738017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7065f9a700 1 -- 172.21.15.189:0/532660500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x558373794c20 msgr2=0x5583737cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:46.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7065f9a700 1 --2- 172.21.15.189:0/532660500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x558373794c20 0x5583737cf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7065f9a700 1 -- 172.21.15.189:0/532660500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55837376f980 msgr2=0x5583738017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:46.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7065f9a700 1 --2- 172.21.15.189:0/532660500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55837376f980 0x5583738017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7065f9a700 1 -- 172.21.15.189:0/532660500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70600b6bb0 con 0x558373799fc0 2024-04-06T13:28:46.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7065f9a700 1 --2- 172.21.15.189:0/532660500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373799fc0 0x55837376b0c0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f7058055d90 tx=0x7f7058009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de03bcc516b115d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:46.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f703ffff700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T13:28:46.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f703ffff700 1 -- 172.21.15.189:0/532660500 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7058082ea0 con 0x558373799fc0 2024-04-06T13:28:46.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f703ffff700 1 -- 172.21.15.189:0/532660500 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(178..178 src has 1..178) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7058061c80 con 0x558373799fc0 2024-04-06T13:28:46.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7073bb1b80 1 -- 172.21.15.189:0/532660500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7054004020 con 0x558373799fc0 2024-04-06T13:28:46.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f703ffff700 1 -- 172.21.15.189:0/532660500 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7058071290 con 0x558373799fc0 2024-04-06T13:28:46.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7073bb1b80 10 client.9774 fetch_fsmap learned FSMap version 78 2024-04-06T13:28:46.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.789+0000 7f7073bb1b80 1 -- 172.21.15.189:0/532660500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x5583734eefe0 con 0x558373799fc0 2024-04-06T13:28:46.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.790+0000 7f703ffff700 1 -- 172.21.15.189:0/532660500 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7054004020 con 0x558373799fc0 2024-04-06T13:28:46.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.790+0000 7f703ffff700 1 -- 172.21.15.189:0/532660500 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 78) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7058058d80 con 0x558373799fc0 2024-04-06T13:28:46.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.790+0000 7f7073bb1b80 10 client.9774 fetch_fsmap finished waiting for FSMap version 78 2024-04-06T13:28:46.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.790+0000 7f7073bb1b80 10 client.9774 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-06T13:28:46.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.790+0000 7f7073bb1b80 20 client.9774 populate_metadata read hostname 'smithi189' 2024-04-06T13:28:46.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.790+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/532660500 >> [v2:172.21.15.189:6835/2089191015,v1:172.21.15.189:6837/2089191015] conn(0x558373822200 0x5583738225e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.790+0000 7f7073bb1b80 4 client.9774 mds_command: new command op to 9621 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-06T13:28:46.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.790+0000 7f7073bb1b80 1 -- 172.21.15.189:0/532660500 --> [v2:172.21.15.189:6835/2089191015,v1:172.21.15.189:6837/2089191015] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558373511ca0 con 0x558373822200 2024-04-06T13:28:46.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.790+0000 7f7065799700 1 --2- 172.21.15.189:0/532660500 >> [v2:172.21.15.189:6835/2089191015,v1:172.21.15.189:6837/2089191015] conn(0x558373822200 0x5583738225e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.790+0000 7f703ffff700 10 client.9774 ms_handle_connect on v2:172.21.15.189:6835/2089191015 2024-04-06T13:28:46.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.790+0000 7f7065799700 1 --2- 172.21.15.189:0/532660500 >> [v2:172.21.15.189:6835/2089191015,v1:172.21.15.189:6837/2089191015] conn(0x558373822200 0x5583738225e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fda8e59ac0548622 server_cookie=2de72e7b919d5495 in_seq=0 out_seq=0 2024-04-06T13:28:46.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.790+0000 7f703ffff700 1 -- 172.21.15.189:0/532660500 <== mds.0 v2:172.21.15.189:6835/2089191015 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x558373511ca0 con 0x558373822200 2024-04-06T13:28:46.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.790+0000 7f703ffff700 10 client.9774 handle_command_reply: tid=0 2024-04-06T13:28:46.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.790+0000 7f7073bb1b80 1 client.9774 shutdown 2024-04-06T13:28:46.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.790+0000 7f7073bb1b80 1 -- 172.21.15.189:0/532660500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373799fc0 msgr2=0x55837376b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:46.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.790+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/532660500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373799fc0 0x55837376b0c0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f7058055d90 tx=0x7f7058009810 comp rx=0 tx=0).stop 2024-04-06T13:28:46.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f7073bb1b80 1 -- 172.21.15.189:0/532660500 shutdown_connections 2024-04-06T13:28:46.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/532660500 >> [v2:172.21.15.189:6835/2089191015,v1:172.21.15.189:6837/2089191015] conn(0x558373822200 0x5583738225e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/532660500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558373799fc0 0x55837376b0c0 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f703ffff700 0 client.9774 ms_handle_reset on v2:172.21.15.189:6835/2089191015 2024-04-06T13:28:46.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/532660500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55837376f980 0x5583738017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f7073bb1b80 1 --2- 172.21.15.189:0/532660500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x558373794c20 0x5583737cf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f7073bb1b80 1 -- 172.21.15.189:0/532660500 >> 172.21.15.189:0/532660500 conn(0x5583736d0650 msgr2=0x55837379f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:46.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f7073bb1b80 1 -- 172.21.15.189:0/532660500 shutdown_connections 2024-04-06T13:28:46.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f7073bb1b80 1 -- 172.21.15.189:0/532660500 wait complete. 2024-04-06T13:28:46.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f7073bb1b80 20 client.9774 trim_cache size 0 max 16384 2024-04-06T13:28:46.802 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:28:46.802 INFO:teuthology.orchestra.run.smithi189.stdout: "cluster_fsid": "e30041de-e9a3-4351-8ca7-29d7f3e3728c", 2024-04-06T13:28:46.802 INFO:teuthology.orchestra.run.smithi189.stdout: "whoami": 0, 2024-04-06T13:28:46.802 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 9621, 2024-04-06T13:28:46.803 INFO:teuthology.orchestra.run.smithi189.stdout: "want_state": "up:active", 2024-04-06T13:28:46.803 INFO:teuthology.orchestra.run.smithi189.stdout: "state": "up:active", 2024-04-06T13:28:46.803 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs", 2024-04-06T13:28:46.803 INFO:teuthology.orchestra.run.smithi189.stdout: "rank_uptime": 5.9772696710000002, 2024-04-06T13:28:46.803 INFO:teuthology.orchestra.run.smithi189.stdout: "mdsmap_epoch": 78, 2024-04-06T13:28:46.803 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch": 178, 2024-04-06T13:28:46.803 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch_barrier": 178, 2024-04-06T13:28:46.803 INFO:teuthology.orchestra.run.smithi189.stdout: "uptime": 11.305928743000001 2024-04-06T13:28:46.803 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:28:46.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f70679fd700 1 -- 172.21.15.189:0/3186980553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f704406adb0 msgr2=0x7f704406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f70679fd700 1 --2- 172.21.15.189:0/3186980553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f704406adb0 0x7f704406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f705400a7b0 tx=0x7f7054008040 comp rx=0 tx=0).stop 2024-04-06T13:28:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f70679fd700 1 -- 172.21.15.189:0/3186980553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7060146dd0 msgr2=0x7f70601f1f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f70679fd700 1 --2- 172.21.15.189:0/3186980553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7060146dd0 0x7f70601f1f70 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f7058015040 tx=0x7f7058004ad0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f70679fd700 1 -- 172.21.15.189:0/3186980553 shutdown_connections 2024-04-06T13:28:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f70679fd700 1 --2- 172.21.15.189:0/3186980553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7060146dd0 0x7f70601f1f70 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f70679fd700 1 --2- 172.21.15.189:0/3186980553 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70600cedf0 0x7f70600b6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f70679fd700 1 --2- 172.21.15.189:0/3186980553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f704406adb0 0x7f704406d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f70679fd700 1 --2- 172.21.15.189:0/3186980553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70600d17a0 0x7f706014a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f70679fd700 1 -- 172.21.15.189:0/3186980553 >> 172.21.15.189:0/3186980553 conn(0x7f70600b2db0 msgr2=0x7f706006b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f70679fd700 1 -- 172.21.15.189:0/3186980553 shutdown_connections 2024-04-06T13:28:46.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.791+0000 7f70679fd700 1 -- 172.21.15.189:0/3186980553 wait complete. 2024-04-06T13:28:46.806 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T13:28:46.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.927+0000 7f07cd743700 1 Processor -- start 2024-04-06T13:28:46.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.927+0000 7f07cd743700 1 -- start start 2024-04-06T13:28:46.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.928+0000 7f07cd743700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f07c815d590 0x7f07c80b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.928+0000 7f07cd743700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f07c80b4ed0 0x7f07c80b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.928+0000 7f07cd743700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07c80b6580 0x7f07c80b69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.928+0000 7f07cd743700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07c80b58c0 con 0x7f07c815d590 2024-04-06T13:28:46.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.928+0000 7f07cd743700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07c80b5cc0 con 0x7f07c80b4ed0 2024-04-06T13:28:46.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.928+0000 7f07cd743700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c80d1500 con 0x7f07c80b6580 2024-04-06T13:28:46.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.928+0000 7f07c6ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f07c815d590 0x7f07c80b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.928+0000 7f07c6ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f07c815d590 0x7f07c80b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59162/0 (socket says 172.21.15.189:59162) 2024-04-06T13:28:46.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.928+0000 7f07c6ffd700 1 -- 172.21.15.189:0/3300177266 learned_addr learned my addr 172.21.15.189:0/3300177266 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:46.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.928+0000 7f07c67fc700 1 --2- 172.21.15.189:0/3300177266 >> v2:172.21.15.192:3300/0 conn(0x7f07c80b4ed0 0x7f07c80b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.928+0000 7f07c77fe700 1 --2- 172.21.15.189:0/3300177266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07c80b6580 0x7f07c80b69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.928+0000 7f07c6ffd700 1 -- 172.21.15.189:0/3300177266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07c80b6580 msgr2=0x7f07c80b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:46.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.928+0000 7f07c6ffd700 1 --2- 172.21.15.189:0/3300177266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07c80b6580 0x7f07c80b69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.928+0000 7f07c6ffd700 1 -- 172.21.15.189:0/3300177266 >> v2:172.21.15.192:3300/0 conn(0x7f07c80b4ed0 msgr2=0x7f07c80b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:46.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.928+0000 7f07c6ffd700 1 --2- 172.21.15.189:0/3300177266 >> v2:172.21.15.192:3300/0 conn(0x7f07c80b4ed0 0x7f07c80b5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.929+0000 7f07c6ffd700 1 -- 172.21.15.189:0/3300177266 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c80d17e0 con 0x7f07c815d590 2024-04-06T13:28:46.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.929+0000 7f07c6ffd700 1 --2- 172.21.15.189:0/3300177266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07c815d590 0x7f07c80b8d70 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f07c80b6330 tx=0x7f07bc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f52c04a26918ed server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:46.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.929+0000 7f07c5ffb700 1 -- 172.21.15.189:0/3300177266 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07bc00b870 con 0x7f07c815d590 2024-04-06T13:28:46.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.929+0000 7f07c5ffb700 1 -- 172.21.15.189:0/3300177266 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07bc00b9d0 con 0x7f07c815d590 2024-04-06T13:28:46.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.929+0000 7f07c5ffb700 1 -- 172.21.15.189:0/3300177266 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07bc00bd60 con 0x7f07c815d590 2024-04-06T13:28:46.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.929+0000 7f07cd743700 1 -- 172.21.15.189:0/3300177266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07c815d590 msgr2=0x7f07c80b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:46.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.929+0000 7f07cd743700 1 --2- 172.21.15.189:0/3300177266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07c815d590 0x7f07c80b8d70 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f07c80b6330 tx=0x7f07bc009bd0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.929+0000 7f07cd743700 1 -- 172.21.15.189:0/3300177266 shutdown_connections 2024-04-06T13:28:46.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.929+0000 7f07cd743700 1 --2- 172.21.15.189:0/3300177266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07c815d590 0x7f07c80b8d70 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.929+0000 7f07cd743700 1 --2- 172.21.15.189:0/3300177266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07c80b6580 0x7f07c80b69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.929+0000 7f07cd743700 1 --2- 172.21.15.189:0/3300177266 >> v2:172.21.15.192:3300/0 conn(0x7f07c80b4ed0 0x7f07c80b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.929+0000 7f07cd743700 1 -- 172.21.15.189:0/3300177266 >> 172.21.15.189:0/3300177266 conn(0x7f07c806bbc0 msgr2=0x7f07c806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:46.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.929+0000 7f07cd743700 1 -- 172.21.15.189:0/3300177266 shutdown_connections 2024-04-06T13:28:46.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.929+0000 7f07cd743700 1 -- 172.21.15.189:0/3300177266 wait complete. 2024-04-06T13:28:46.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.931+0000 7f07cd743700 1 Processor -- start 2024-04-06T13:28:46.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.931+0000 7f07cd743700 1 -- start start 2024-04-06T13:28:46.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.932+0000 7f07cd743700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07c80b4ed0 0x7f07c8160490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.932+0000 7f07cd743700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07c80b6580 0x7f07c80cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.932+0000 7f07cd743700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07c815d590 0x7f07c814f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.932+0000 7f07cd743700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c81609d0 con 0x7f07c80b6580 2024-04-06T13:28:46.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.932+0000 7f07cd743700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c80cbb90 con 0x7f07c80b4ed0 2024-04-06T13:28:46.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.932+0000 7f07cd743700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c80cbdc0 con 0x7f07c815d590 2024-04-06T13:28:46.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.932+0000 7f07c67fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07c80b6580 0x7f07c80cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.932+0000 7f07c67fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07c80b6580 0x7f07c80cb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59172/0 (socket says 172.21.15.189:59172) 2024-04-06T13:28:46.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.932+0000 7f07c67fc700 1 -- 172.21.15.189:0/2238910715 learned_addr learned my addr 172.21.15.189:0/2238910715 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:46.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.932+0000 7f07c6ffd700 1 --2- 172.21.15.189:0/2238910715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07c80b4ed0 0x7f07c8160490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.932+0000 7f07c77fe700 1 --2- 172.21.15.189:0/2238910715 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07c815d590 0x7f07c814f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.933+0000 7f07c67fc700 1 -- 172.21.15.189:0/2238910715 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07c815d590 msgr2=0x7f07c814f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:46.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.933+0000 7f07c67fc700 1 --2- 172.21.15.189:0/2238910715 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07c815d590 0x7f07c814f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.933+0000 7f07c67fc700 1 -- 172.21.15.189:0/2238910715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07c80b4ed0 msgr2=0x7f07c8160490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:46.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.933+0000 7f07c67fc700 1 --2- 172.21.15.189:0/2238910715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07c80b4ed0 0x7f07c8160490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:46.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.933+0000 7f07c67fc700 1 -- 172.21.15.189:0/2238910715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07bc009550 con 0x7f07c80b6580 2024-04-06T13:28:46.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.933+0000 7f07c67fc700 1 --2- 172.21.15.189:0/2238910715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07c80b6580 0x7f07c80cb650 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f07b800c820 tx=0x7f07b800cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:46.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.933+0000 7f07af7fe700 1 -- 172.21.15.189:0/2238910715 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b8004220 con 0x7f07c80b6580 2024-04-06T13:28:46.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.933+0000 7f07af7fe700 1 -- 172.21.15.189:0/2238910715 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b8004380 con 0x7f07c80b6580 2024-04-06T13:28:46.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.933+0000 7f07cd743700 1 -- 172.21.15.189:0/2238910715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c80cc0a0 con 0x7f07c80b6580 2024-04-06T13:28:46.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.933+0000 7f07af7fe700 1 -- 172.21.15.189:0/2238910715 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b80036a0 con 0x7f07c80b6580 2024-04-06T13:28:46.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.933+0000 7f07cd743700 1 -- 172.21.15.189:0/2238910715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c80cc660 con 0x7f07c80b6580 2024-04-06T13:28:46.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.934+0000 7f07cd743700 1 -- 172.21.15.189:0/2238910715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c8074990 con 0x7f07c80b6580 2024-04-06T13:28:46.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.935+0000 7f07af7fe700 1 -- 172.21.15.189:0/2238910715 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f07b8003820 con 0x7f07c80b6580 2024-04-06T13:28:46.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.935+0000 7f07af7fe700 1 --2- 172.21.15.189:0/2238910715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f07b406adb0 0x7f07b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:46.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.935+0000 7f07af7fe700 1 -- 172.21.15.189:0/2238910715 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f07b8089600 con 0x7f07c80b6580 2024-04-06T13:28:46.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.935+0000 7f07c6ffd700 1 --2- 172.21.15.189:0/2238910715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f07b406adb0 0x7f07b406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:46.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.935+0000 7f07c6ffd700 1 --2- 172.21.15.189:0/2238910715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f07b406adb0 0x7f07b406d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f07bc014040 tx=0x7f07bc008030 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:46.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:46.939+0000 7f07af7fe700 1 -- 172.21.15.189:0/2238910715 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f07b80544a0 con 0x7f07c80b6580 2024-04-06T13:28:47.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:47.177+0000 7f07cd743700 1 -- 172.21.15.189:0/2238910715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f07c80d2820 con 0x7f07c80b6580 2024-04-06T13:28:47.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:47.177+0000 7f07af7fe700 1 -- 172.21.15.189:0/2238910715 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v78) v1 ==== 53+0+73 (secure 0 0 0) 0x7f07b801e070 con 0x7f07c80b6580 2024-04-06T13:28:47.179 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-06T13:28:47.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:47.179+0000 7f07cd743700 1 -- 172.21.15.189:0/2238910715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f07b406adb0 msgr2=0x7f07b406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:47.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:47.179+0000 7f07cd743700 1 --2- 172.21.15.189:0/2238910715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f07b406adb0 0x7f07b406d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f07bc014040 tx=0x7f07bc008030 comp rx=0 tx=0).stop 2024-04-06T13:28:47.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:47.179+0000 7f07cd743700 1 -- 172.21.15.189:0/2238910715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07c80b6580 msgr2=0x7f07c80cb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:47.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:47.179+0000 7f07cd743700 1 --2- 172.21.15.189:0/2238910715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07c80b6580 0x7f07c80cb650 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f07b800c820 tx=0x7f07b800cb30 comp rx=0 tx=0).stop 2024-04-06T13:28:47.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:47.179+0000 7f07cd743700 1 -- 172.21.15.189:0/2238910715 shutdown_connections 2024-04-06T13:28:47.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:47.179+0000 7f07cd743700 1 --2- 172.21.15.189:0/2238910715 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07c815d590 0x7f07c814f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:47.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:47.179+0000 7f07cd743700 1 --2- 172.21.15.189:0/2238910715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f07b406adb0 0x7f07b406d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:47.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:47.180+0000 7f07cd743700 1 --2- 172.21.15.189:0/2238910715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07c80b4ed0 0x7f07c8160490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:47.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:47.180+0000 7f07cd743700 1 --2- 172.21.15.189:0/2238910715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07c80b6580 0x7f07c80cb650 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:47.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:47.180+0000 7f07cd743700 1 -- 172.21.15.189:0/2238910715 >> 172.21.15.189:0/2238910715 conn(0x7f07c806bbc0 msgr2=0x7f07c80b37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:47.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:47.180+0000 7f07cd743700 1 -- 172.21.15.189:0/2238910715 shutdown_connections 2024-04-06T13:28:47.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:47.180+0000 7f07cd743700 1 -- 172.21.15.189:0/2238910715 wait complete. 2024-04-06T13:28:47.193 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-06T13:28:47.193 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T13:28:47.193 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-06T13:28:47.193 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-06T13:28:47.193 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T13:28:47.193 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-06T13:28:47.193 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-06T13:28:47.193 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T13:28:47.193 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:28:47.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:47.194 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-06T13:28:47.211 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T13:28:47.211 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T13:28:47.211 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T13:28:47.211 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:28:47.211 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T13:28:47.212 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:28:47.212 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:28:47.212 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:28:47.212 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-06T13:28:47.212 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:28:47.212 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:28:47.212 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-06T13:28:47.212 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:28:47.212 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:28:47.212 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T13:28:47.212 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:28:47.212 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T13:28:47.212 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 37762sec preferred_lft 37762sec 2024-04-06T13:28:47.212 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::8592:df2c:92e2:dace/64 scope link noprefixroute 2024-04-06T13:28:47.213 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:28:47.213 INFO:teuthology.orchestra.run.smithi189.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T13:28:47.213 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff 2024-04-06T13:28:47.213 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-06T13:28:47.213 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:28:47.213 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::88e:b6ff:fe6f:1cb0/64 scope link 2024-04-06T13:28:47.213 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:28:47.213 INFO:teuthology.orchestra.run.smithi189.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T13:28:47.213 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 62:a5:8d:c5:1a:9c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T13:28:47.213 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::60a5:8dff:fec5:1a9c/64 scope link 2024-04-06T13:28:47.213 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:28:47.213 INFO:teuthology.orchestra.run.smithi189.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T13:28:47.213 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-06T13:28:47.213 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::1c7e:c6ff:febb:1dab/64 scope link 2024-04-06T13:28:47.213 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:28:47.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:47.214 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T13:28:47.277 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-06T13:28:47.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:47.279 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-06T13:28:47.338 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-06T13:28:47.338 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T13:28:47.338 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T13:28:47.338 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T13:28:47.338 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T13:28:47.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:47.339 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T13:28:47.339 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T13:28:47.339 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T13:28:47.339 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T13:28:47.339 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T13:28:47.339 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T13:28:47.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:47.445 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T13:28:47.462 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-06T13:28:47.462 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-06T13:28:47.462 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:28:47.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:47.462 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-04-06T13:28:47.528 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T13:28:47.529 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-06T13:28:47.585 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:28:47.586 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T13:28:47.586 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-06T13:28:47.642 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T13:28:47.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:47.642 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:28:47.729 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:28:47.729 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:28:47.729 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:28:47.738 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:28:47.738 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:28:47.738 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:28:47.738 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T13:28:47.738 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T13:28:47.757 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:28:47.757 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:28:47.757 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:28:47.757 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T13:28:47.757 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T13:28:47.757 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790" 2024-04-06T13:28:47.758 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2024-04-06T13:28:47.758 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e30041de-e9a3-4351-8ca7-29d7f3e3728c" will pass to kernel 2024-04-06T13:28:47.759 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T13:28:47.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:47.760 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-06T13:28:47.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:47.787 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T13:28:47.870 INFO:teuthology.orchestra.run.smithi189.stdout:client7898 2024-04-06T13:28:47.873 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:28:47.873 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:28:47.928 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-06T13:28:48.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.093+0000 7f89c0c66700 1 Processor -- start 2024-04-06T13:28:48.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.093+0000 7f89c0c66700 1 -- start start 2024-04-06T13:28:48.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.094+0000 7f89c0c66700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f89bc13e340 0x7f89bc13e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:48.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.094+0000 7f89c0c66700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f89bc15c380 0x7f89bc0cbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:48.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.094+0000 7f89c0c66700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89bc0cdeb0 con 0x7f89bc0cb9d0 2024-04-06T13:28:48.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.094+0000 7f89c0c66700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89bc0cc3a0 con 0x7f89bc13e340 2024-04-06T13:28:48.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.094+0000 7f89c0c66700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89bc15c810 con 0x7f89bc15c380 2024-04-06T13:28:48.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.094+0000 7f89ba59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f89bc13e340 0x7f89bc13e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:48.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.094+0000 7f89ba59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f89bc13e340 0x7f89bc13e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59184/0 (socket says 172.21.15.189:59184) 2024-04-06T13:28:48.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.094+0000 7f89ba59c700 1 -- 172.21.15.189:0/1719942256 learned_addr learned my addr 172.21.15.189:0/1719942256 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:48.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.094+0000 7f89bad9d700 1 --2- 172.21.15.189:0/1719942256 >> v2:172.21.15.192:3300/0 conn(0x7f89bc15c380 0x7f89bc0cbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:48.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.094+0000 7f89ba59c700 1 -- 172.21.15.189:0/1719942256 >> v2:172.21.15.192:3300/0 conn(0x7f89bc15c380 msgr2=0x7f89bc0cbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:48.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.094+0000 7f89ba59c700 1 --2- 172.21.15.189:0/1719942256 >> v2:172.21.15.192:3300/0 conn(0x7f89bc15c380 0x7f89bc0cbe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:48.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.094+0000 7f89ba59c700 1 -- 172.21.15.189:0/1719942256 >> v1:172.21.15.192:6789/0 conn(0x7f89bc0cb9d0 legacy=0x7f89bc15bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:48.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.094+0000 7f89ba59c700 1 -- 172.21.15.189:0/1719942256 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89bc0ce190 con 0x7f89bc13e340 2024-04-06T13:28:48.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.095+0000 7f89ba59c700 1 --2- 172.21.15.189:0/1719942256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc13e340 0x7f89bc13e8b0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f89bc06b430 tx=0x7f89ac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e72250135f8133 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:48.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.095+0000 7f89b959a700 1 -- 172.21.15.189:0/1719942256 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ac011830 con 0x7f89bc13e340 2024-04-06T13:28:48.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.095+0000 7f89b959a700 1 -- 172.21.15.189:0/1719942256 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89ac011990 con 0x7f89bc13e340 2024-04-06T13:28:48.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.095+0000 7f89b959a700 1 -- 172.21.15.189:0/1719942256 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ac011d20 con 0x7f89bc13e340 2024-04-06T13:28:48.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.095+0000 7f89c0c66700 1 -- 172.21.15.189:0/1719942256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc13e340 msgr2=0x7f89bc13e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:48.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.095+0000 7f89c0c66700 1 --2- 172.21.15.189:0/1719942256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc13e340 0x7f89bc13e8b0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f89bc06b430 tx=0x7f89ac009c00 comp rx=0 tx=0).stop 2024-04-06T13:28:48.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.095+0000 7f89c0c66700 1 -- 172.21.15.189:0/1719942256 shutdown_connections 2024-04-06T13:28:48.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.095+0000 7f89c0c66700 1 --2- 172.21.15.189:0/1719942256 >> v2:172.21.15.192:3300/0 conn(0x7f89bc15c380 0x7f89bc0cbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:48.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.095+0000 7f89c0c66700 1 --2- 172.21.15.189:0/1719942256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc13e340 0x7f89bc13e8b0 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:48.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.095+0000 7f89c0c66700 1 -- 172.21.15.189:0/1719942256 >> 172.21.15.189:0/1719942256 conn(0x7f89bc06a730 msgr2=0x7f89bc06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:48.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.095+0000 7f89c0c66700 1 -- 172.21.15.189:0/1719942256 shutdown_connections 2024-04-06T13:28:48.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.096+0000 7f89c0c66700 1 -- 172.21.15.189:0/1719942256 wait complete. 2024-04-06T13:28:48.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.097+0000 7f89c0c66700 1 Processor -- start 2024-04-06T13:28:48.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.097+0000 7f89c0c66700 1 -- start start 2024-04-06T13:28:48.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89c0c66700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89bc0cb9d0 0x7f89bc13d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:48.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89c0c66700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc13e340 0x7f89bc159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:48.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89c0c66700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89bc15c380 0x7f89bc15a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:48.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89c0c66700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc158850 con 0x7f89bc13e340 2024-04-06T13:28:48.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89c0c66700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc158b30 con 0x7f89bc15c380 2024-04-06T13:28:48.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89c0c66700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc158e10 con 0x7f89bc0cb9d0 2024-04-06T13:28:48.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89b9d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc13e340 0x7f89bc159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:48.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89b9d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc13e340 0x7f89bc159dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59188/0 (socket says 172.21.15.189:59188) 2024-04-06T13:28:48.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89b9d9b700 1 -- 172.21.15.189:0/1513560851 learned_addr learned my addr 172.21.15.189:0/1513560851 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:48.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89bad9d700 1 --2- 172.21.15.189:0/1513560851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89bc15c380 0x7f89bc15a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:48.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89ba59c700 1 --2- 172.21.15.189:0/1513560851 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89bc0cb9d0 0x7f89bc13d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:48.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89b9d9b700 1 -- 172.21.15.189:0/1513560851 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89bc0cb9d0 msgr2=0x7f89bc13d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:48.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89b9d9b700 1 --2- 172.21.15.189:0/1513560851 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89bc0cb9d0 0x7f89bc13d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:48.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89b9d9b700 1 -- 172.21.15.189:0/1513560851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89bc15c380 msgr2=0x7f89bc15a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:48.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89b9d9b700 1 --2- 172.21.15.189:0/1513560851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89bc15c380 0x7f89bc15a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:48.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89b9d9b700 1 -- 172.21.15.189:0/1513560851 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ac009550 con 0x7f89bc13e340 2024-04-06T13:28:48.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89b9d9b700 1 --2- 172.21.15.189:0/1513560851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc13e340 0x7f89bc159dc0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f89b000c450 tx=0x7f89b000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:48.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89aaffd700 1 -- 172.21.15.189:0/1513560851 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b00052d0 con 0x7f89bc13e340 2024-04-06T13:28:48.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89aaffd700 1 -- 172.21.15.189:0/1513560851 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b0005430 con 0x7f89bc13e340 2024-04-06T13:28:48.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89aaffd700 1 -- 172.21.15.189:0/1513560851 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b00038c0 con 0x7f89bc13e340 2024-04-06T13:28:48.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89c0c66700 1 -- 172.21.15.189:0/1513560851 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89bc1590f0 con 0x7f89bc13e340 2024-04-06T13:28:48.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.098+0000 7f89c0c66700 1 -- 172.21.15.189:0/1513560851 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89bc13ddc0 con 0x7f89bc13e340 2024-04-06T13:28:48.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.100+0000 7f89c0c66700 1 -- 172.21.15.189:0/1513560851 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89bc073270 con 0x7f89bc13e340 2024-04-06T13:28:48.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.100+0000 7f89aaffd700 1 -- 172.21.15.189:0/1513560851 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f89b0003a20 con 0x7f89bc13e340 2024-04-06T13:28:48.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.100+0000 7f89aaffd700 1 --2- 172.21.15.189:0/1513560851 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f899c06adb0 0x7f899c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:48.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.100+0000 7f89aaffd700 1 -- 172.21.15.189:0/1513560851 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f89b0088950 con 0x7f89bc13e340 2024-04-06T13:28:48.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.100+0000 7f89ba59c700 1 --2- 172.21.15.189:0/1513560851 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f899c06adb0 0x7f899c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:48.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.101+0000 7f89ba59c700 1 --2- 172.21.15.189:0/1513560851 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f899c06adb0 0x7f899c06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f89ac014040 tx=0x7f89ac008030 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:48.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.105+0000 7f89aaffd700 1 -- 172.21.15.189:0/1513560851 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f89b0053820 con 0x7f89bc13e340 2024-04-06T13:28:48.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.356+0000 7f89c0c66700 1 -- 172.21.15.189:0/1513560851 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f89bc074990 con 0x7f899c06adb0 2024-04-06T13:28:48.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.358+0000 7f89aaffd700 1 -- 172.21.15.189:0/1513560851 <== mgr.8614 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f89bc074990 con 0x7f899c06adb0 2024-04-06T13:28:48.359 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T13:28:48.359 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:28:48.359 INFO:teuthology.orchestra.run.smithi189.stdout: "name": "cephfs" 2024-04-06T13:28:48.359 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:28:48.359 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T13:28:48.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.360+0000 7f89c0c66700 1 -- 172.21.15.189:0/1513560851 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f899c06adb0 msgr2=0x7f899c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:48.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.360+0000 7f89c0c66700 1 --2- 172.21.15.189:0/1513560851 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f899c06adb0 0x7f899c06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f89ac014040 tx=0x7f89ac008030 comp rx=0 tx=0).stop 2024-04-06T13:28:48.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.360+0000 7f89c0c66700 1 -- 172.21.15.189:0/1513560851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc13e340 msgr2=0x7f89bc159dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:48.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.360+0000 7f89c0c66700 1 --2- 172.21.15.189:0/1513560851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc13e340 0x7f89bc159dc0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f89b000c450 tx=0x7f89b000c760 comp rx=0 tx=0).stop 2024-04-06T13:28:48.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.360+0000 7f89c0c66700 1 -- 172.21.15.189:0/1513560851 shutdown_connections 2024-04-06T13:28:48.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.360+0000 7f89c0c66700 1 --2- 172.21.15.189:0/1513560851 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f899c06adb0 0x7f899c06d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:48.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.360+0000 7f89c0c66700 1 --2- 172.21.15.189:0/1513560851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89bc15c380 0x7f89bc15a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:48.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.360+0000 7f89c0c66700 1 --2- 172.21.15.189:0/1513560851 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89bc0cb9d0 0x7f89bc13d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:48.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.360+0000 7f89c0c66700 1 --2- 172.21.15.189:0/1513560851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc13e340 0x7f89bc159dc0 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:48.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.360+0000 7f89c0c66700 1 -- 172.21.15.189:0/1513560851 >> 172.21.15.189:0/1513560851 conn(0x7f89bc06a730 msgr2=0x7f89bc0ced00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:48.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.360+0000 7f89c0c66700 1 -- 172.21.15.189:0/1513560851 shutdown_connections 2024-04-06T13:28:48.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.360+0000 7f89c0c66700 1 -- 172.21.15.189:0/1513560851 wait complete. 2024-04-06T13:28:48.375 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-06T13:28:48.505 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.504+0000 7fa91968a700 1 Processor -- start 2024-04-06T13:28:48.505 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.504+0000 7fa91968a700 1 -- start start 2024-04-06T13:28:48.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.505+0000 7fa91968a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa91415d040 0x7fa91415dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:48.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.505+0000 7fa91968a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa91406cbf0 0x7fa9140cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:48.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.505+0000 7fa91968a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91415d8d0 con 0x7fa91415e2a0 2024-04-06T13:28:48.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.505+0000 7fa91968a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9140cba40 con 0x7fa91415d040 2024-04-06T13:28:48.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.505+0000 7fa91968a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91415d580 con 0x7fa91406cbf0 2024-04-06T13:28:48.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.505+0000 7fa9127fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa91415e2a0 0x7fa91406c4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55900/0 (socket says 172.21.15.189:55900) 2024-04-06T13:28:48.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.505+0000 7fa9127fc700 1 -- 172.21.15.189:0/1490034598 learned_addr learned my addr 172.21.15.189:0/1490034598 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:48.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.505+0000 7fa912ffd700 1 --2- 172.21.15.189:0/1490034598 >> v2:172.21.15.192:3300/0 conn(0x7fa91415d040 0x7fa91415dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:48.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.505+0000 7fa9137fe700 1 --2- 172.21.15.189:0/1490034598 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa91406cbf0 0x7fa9140cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:48.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.505+0000 7fa911ffb700 1 -- 172.21.15.189:0/1490034598 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755751131 0 0) 0x7fa91415d8d0 con 0x7fa91415e2a0 2024-04-06T13:28:48.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.505+0000 7fa911ffb700 1 -- 172.21.15.189:0/1490034598 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8fc0035b0 con 0x7fa91415e2a0 2024-04-06T13:28:48.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.506+0000 7fa912ffd700 1 -- 172.21.15.189:0/1490034598 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa91406cbf0 msgr2=0x7fa9140cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:48.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.506+0000 7fa912ffd700 1 --2- 172.21.15.189:0/1490034598 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa91406cbf0 0x7fa9140cc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:48.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.506+0000 7fa912ffd700 1 -- 172.21.15.189:0/1490034598 >> v1:172.21.15.189:6789/0 conn(0x7fa91415e2a0 legacy=0x7fa91406c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:48.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.506+0000 7fa912ffd700 1 -- 172.21.15.189:0/1490034598 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9140b3240 con 0x7fa91415d040 2024-04-06T13:28:48.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.506+0000 7fa912ffd700 1 --2- 172.21.15.189:0/1490034598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa91415d040 0x7fa91415dd60 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa904009840 tx=0x7fa904009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1ebd34f9b508dc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:48.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.506+0000 7fa911ffb700 1 -- 172.21.15.189:0/1490034598 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa904011830 con 0x7fa91415d040 2024-04-06T13:28:48.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.506+0000 7fa911ffb700 1 -- 172.21.15.189:0/1490034598 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa904011990 con 0x7fa91415d040 2024-04-06T13:28:48.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.506+0000 7fa911ffb700 1 -- 172.21.15.189:0/1490034598 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa904011d20 con 0x7fa91415d040 2024-04-06T13:28:48.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.507+0000 7fa91968a700 1 -- 172.21.15.189:0/1490034598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa91415d040 msgr2=0x7fa91415dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:48.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.507+0000 7fa91968a700 1 --2- 172.21.15.189:0/1490034598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa91415d040 0x7fa91415dd60 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa904009840 tx=0x7fa904009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:48.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.507+0000 7fa91968a700 1 -- 172.21.15.189:0/1490034598 shutdown_connections 2024-04-06T13:28:48.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.507+0000 7fa91968a700 1 --2- 172.21.15.189:0/1490034598 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa91406cbf0 0x7fa9140cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:48.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.507+0000 7fa91968a700 1 --2- 172.21.15.189:0/1490034598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa91415d040 0x7fa91415dd60 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:48.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.507+0000 7fa91968a700 1 -- 172.21.15.189:0/1490034598 >> 172.21.15.189:0/1490034598 conn(0x7fa91406a730 msgr2=0x7fa91406ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:48.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.507+0000 7fa91968a700 1 -- 172.21.15.189:0/1490034598 shutdown_connections 2024-04-06T13:28:48.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.507+0000 7fa91968a700 1 -- 172.21.15.189:0/1490034598 wait complete. 2024-04-06T13:28:48.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.508+0000 7fa91968a700 1 Processor -- start 2024-04-06T13:28:48.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.508+0000 7fa91968a700 1 -- start start 2024-04-06T13:28:48.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.509+0000 7fa91968a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa91406cbf0 0x7fa914145bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:48.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.509+0000 7fa91968a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa91415d040 0x7fa9141552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:48.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.509+0000 7fa91968a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9140c4900 0x7fa9140c4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:48.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.509+0000 7fa91968a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa914152ff0 con 0x7fa91415d040 2024-04-06T13:28:48.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.509+0000 7fa91968a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa914153ff0 con 0x7fa91406cbf0 2024-04-06T13:28:48.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.509+0000 7fa91968a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa914154e50 con 0x7fa9140c4900 2024-04-06T13:28:48.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.509+0000 7fa9127fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa91415d040 0x7fa9141552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:48.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.509+0000 7fa9127fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa91415d040 0x7fa9141552e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59194/0 (socket says 172.21.15.189:59194) 2024-04-06T13:28:48.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.509+0000 7fa9127fc700 1 -- 172.21.15.189:0/1986556171 learned_addr learned my addr 172.21.15.189:0/1986556171 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:48.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.509+0000 7fa912ffd700 1 --2- 172.21.15.189:0/1986556171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa91406cbf0 0x7fa914145bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:48.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.509+0000 7fa9137fe700 1 --2- 172.21.15.189:0/1986556171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9140c4900 0x7fa9140c4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:48.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.509+0000 7fa9127fc700 1 -- 172.21.15.189:0/1986556171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9140c4900 msgr2=0x7fa9140c4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:48.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.509+0000 7fa9127fc700 1 --2- 172.21.15.189:0/1986556171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9140c4900 0x7fa9140c4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:48.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.509+0000 7fa9127fc700 1 -- 172.21.15.189:0/1986556171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa91406cbf0 msgr2=0x7fa914145bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:48.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.509+0000 7fa9127fc700 1 --2- 172.21.15.189:0/1986556171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa91406cbf0 0x7fa914145bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:48.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.510+0000 7fa9127fc700 1 -- 172.21.15.189:0/1986556171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9040094a0 con 0x7fa91415d040 2024-04-06T13:28:48.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.510+0000 7fa912ffd700 1 --2- 172.21.15.189:0/1986556171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa91406cbf0 0x7fa914145bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:28:48.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.510+0000 7fa9137fe700 1 --2- 172.21.15.189:0/1986556171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9140c4900 0x7fa9140c4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:28:48.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.510+0000 7fa9127fc700 1 --2- 172.21.15.189:0/1986556171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa91415d040 0x7fa9141552e0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7fa908001eb0 tx=0x7fa90800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:48.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.510+0000 7fa8fb7fe700 1 -- 172.21.15.189:0/1986556171 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa908005d70 con 0x7fa91415d040 2024-04-06T13:28:48.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.510+0000 7fa8fb7fe700 1 -- 172.21.15.189:0/1986556171 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9080070d0 con 0x7fa91415d040 2024-04-06T13:28:48.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.510+0000 7fa91968a700 1 -- 172.21.15.189:0/1986556171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa914155820 con 0x7fa91415d040 2024-04-06T13:28:48.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.510+0000 7fa8fb7fe700 1 -- 172.21.15.189:0/1986556171 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa908007460 con 0x7fa91415d040 2024-04-06T13:28:48.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.510+0000 7fa91968a700 1 -- 172.21.15.189:0/1986556171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa914154520 con 0x7fa91415d040 2024-04-06T13:28:48.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.511+0000 7fa91968a700 1 -- 172.21.15.189:0/1986556171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa914073270 con 0x7fa91415d040 2024-04-06T13:28:48.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.511+0000 7fa8fb7fe700 1 -- 172.21.15.189:0/1986556171 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 99) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fa9080075c0 con 0x7fa91415d040 2024-04-06T13:28:48.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.512+0000 7fa8fb7fe700 1 --2- 172.21.15.189:0/1986556171 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa90006adb0 0x7fa90006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:48.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.512+0000 7fa8fb7fe700 1 -- 172.21.15.189:0/1986556171 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa908088b60 con 0x7fa91415d040 2024-04-06T13:28:48.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.512+0000 7fa912ffd700 1 --2- 172.21.15.189:0/1986556171 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa90006adb0 0x7fa90006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:48.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.512+0000 7fa912ffd700 1 --2- 172.21.15.189:0/1986556171 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa90006adb0 0x7fa90006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa904000c00 tx=0x7fa904008030 comp rx=0 tx=0).ready entity=mgr.8614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:48.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.516+0000 7fa8fb7fe700 1 -- 172.21.15.189:0/1986556171 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa908053a40 con 0x7fa91415d040 2024-04-06T13:28:48.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.778+0000 7fa91968a700 1 -- 172.21.15.189:0/1986556171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fa914074990 con 0x7fa91415d040 2024-04-06T13:28:48.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.851+0000 7fa8fb7fe700 1 -- 172.21.15.189:0/1986556171 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 100) v1 ==== 89264+0+0 (secure 0 0 0) 0x7fa908074d50 con 0x7fa91415d040 2024-04-06T13:28:48.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.853+0000 7fa8fb7fe700 1 -- 172.21.15.189:0/1986556171 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v100) v1 ==== 92+0+0 (secure 0 0 0) 0x7fa908083e20 con 0x7fa91415d040 2024-04-06T13:28:48.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.856+0000 7fa91968a700 1 -- 172.21.15.189:0/1986556171 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa90006adb0 msgr2=0x7fa90006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:48.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.856+0000 7fa91968a700 1 --2- 172.21.15.189:0/1986556171 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa90006adb0 0x7fa90006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa904000c00 tx=0x7fa904008030 comp rx=0 tx=0).stop 2024-04-06T13:28:48.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.856+0000 7fa91968a700 1 -- 172.21.15.189:0/1986556171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa91415d040 msgr2=0x7fa9141552e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:48.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.856+0000 7fa91968a700 1 --2- 172.21.15.189:0/1986556171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa91415d040 0x7fa9141552e0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7fa908001eb0 tx=0x7fa90800c860 comp rx=0 tx=0).stop 2024-04-06T13:28:48.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.856+0000 7fa91968a700 1 -- 172.21.15.189:0/1986556171 shutdown_connections 2024-04-06T13:28:48.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.856+0000 7fa91968a700 1 --2- 172.21.15.189:0/1986556171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9140c4900 0x7fa9140c4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:48.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.856+0000 7fa91968a700 1 --2- 172.21.15.189:0/1986556171 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa90006adb0 0x7fa90006d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:48.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.856+0000 7fa91968a700 1 --2- 172.21.15.189:0/1986556171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa91406cbf0 0x7fa914145bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:48.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.856+0000 7fa91968a700 1 --2- 172.21.15.189:0/1986556171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa91415d040 0x7fa9141552e0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:48.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.856+0000 7fa91968a700 1 -- 172.21.15.189:0/1986556171 >> 172.21.15.189:0/1986556171 conn(0x7fa91406a730 msgr2=0x7fa9140ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:48.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.856+0000 7fa91968a700 1 -- 172.21.15.189:0/1986556171 shutdown_connections 2024-04-06T13:28:48.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:48.856+0000 7fa91968a700 1 -- 172.21.15.189:0/1986556171 wait complete. 2024-04-06T13:28:48.871 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-06T13:28:49.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.003+0000 7fdb437d4700 1 Processor -- start 2024-04-06T13:28:49.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.003+0000 7fdb437d4700 1 -- start start 2024-04-06T13:28:49.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.004+0000 7fdb437d4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdb3c15a550 0x7fdb3c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:49.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.004+0000 7fdb437d4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c0cdbe0 con 0x7fdb3c0cf900 2024-04-06T13:28:49.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.004+0000 7fdb437d4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c15a940 con 0x7fdb3c0b4d70 2024-04-06T13:28:49.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.004+0000 7fdb437d4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb3c15ac00 con 0x7fdb3c15a550 2024-04-06T13:28:49.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.004+0000 7fdb40d6f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdb3c0cf900 0x7fdb3c0b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55918/0 (socket says 172.21.15.189:55918) 2024-04-06T13:28:49.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.004+0000 7fdb40d6f700 1 -- 172.21.15.189:0/1864447176 learned_addr learned my addr 172.21.15.189:0/1864447176 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:49.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.005+0000 7fdb41570700 1 --2- 172.21.15.189:0/1864447176 >> v2:172.21.15.192:3300/0 conn(0x7fdb3c15a550 0x7fdb3c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:49.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.005+0000 7fdb33fff700 1 -- 172.21.15.189:0/1864447176 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822569438 0 0) 0x7fdb3c0cdbe0 con 0x7fdb3c0cf900 2024-04-06T13:28:49.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.005+0000 7fdb33fff700 1 -- 172.21.15.189:0/1864447176 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb280035b0 con 0x7fdb3c0cf900 2024-04-06T13:28:49.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.005+0000 7fdb41570700 1 -- 172.21.15.189:0/1864447176 >> v1:172.21.15.192:6789/0 conn(0x7fdb3c0b4d70 legacy=0x7fdb3c0b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:49.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.005+0000 7fdb41570700 1 -- 172.21.15.189:0/1864447176 >> v1:172.21.15.189:6789/0 conn(0x7fdb3c0cf900 legacy=0x7fdb3c0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:49.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.005+0000 7fdb41570700 1 -- 172.21.15.189:0/1864447176 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c15aee0 con 0x7fdb3c15a550 2024-04-06T13:28:49.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.005+0000 7fdb41570700 1 --2- 172.21.15.189:0/1864447176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb3c15a550 0x7fdb3c0cf3c0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fdb38009840 tx=0x7fdb38009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dfd27a056764ef6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:49.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.005+0000 7fdb33fff700 1 -- 172.21.15.189:0/1864447176 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb38011830 con 0x7fdb3c15a550 2024-04-06T13:28:49.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.005+0000 7fdb33fff700 1 -- 172.21.15.189:0/1864447176 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb38011990 con 0x7fdb3c15a550 2024-04-06T13:28:49.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.005+0000 7fdb33fff700 1 -- 172.21.15.189:0/1864447176 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb38011d20 con 0x7fdb3c15a550 2024-04-06T13:28:49.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.006+0000 7fdb437d4700 1 -- 172.21.15.189:0/1864447176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb3c15a550 msgr2=0x7fdb3c0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:49.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.006+0000 7fdb437d4700 1 --2- 172.21.15.189:0/1864447176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb3c15a550 0x7fdb3c0cf3c0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fdb38009840 tx=0x7fdb38009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:49.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.006+0000 7fdb437d4700 1 -- 172.21.15.189:0/1864447176 shutdown_connections 2024-04-06T13:28:49.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.006+0000 7fdb437d4700 1 --2- 172.21.15.189:0/1864447176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb3c15a550 0x7fdb3c0cf3c0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.006+0000 7fdb437d4700 1 -- 172.21.15.189:0/1864447176 >> 172.21.15.189:0/1864447176 conn(0x7fdb3c0d00b0 msgr2=0x7fdb3c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:49.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.006+0000 7fdb437d4700 1 -- 172.21.15.189:0/1864447176 shutdown_connections 2024-04-06T13:28:49.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.006+0000 7fdb437d4700 1 -- 172.21.15.189:0/1864447176 wait complete. 2024-04-06T13:28:49.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.007+0000 7fdb437d4700 1 Processor -- start 2024-04-06T13:28:49.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.007+0000 7fdb437d4700 1 -- start start 2024-04-06T13:28:49.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.008+0000 7fdb437d4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb3c0b4d70 0x7fdb3c14f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:49.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.008+0000 7fdb437d4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb3c0cf900 0x7fdb3c14f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:49.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.008+0000 7fdb437d4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb3c14ca20 0x7fdb3c14ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:49.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.008+0000 7fdb437d4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c0a8ec0 con 0x7fdb3c0cf900 2024-04-06T13:28:49.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.008+0000 7fdb437d4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c0a9ce0 con 0x7fdb3c14ca20 2024-04-06T13:28:49.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.008+0000 7fdb437d4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c0a90d0 con 0x7fdb3c0b4d70 2024-04-06T13:28:49.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.008+0000 7fdb40d6f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb3c0cf900 0x7fdb3c14f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:49.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.008+0000 7fdb40d6f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb3c0cf900 0x7fdb3c14f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59236/0 (socket says 172.21.15.189:59236) 2024-04-06T13:28:49.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.008+0000 7fdb40d6f700 1 -- 172.21.15.189:0/4016404005 learned_addr learned my addr 172.21.15.189:0/4016404005 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:49.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.008+0000 7fdb41570700 1 --2- 172.21.15.189:0/4016404005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb3c0b4d70 0x7fdb3c14f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:49.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.008+0000 7fdb41d71700 1 --2- 172.21.15.189:0/4016404005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb3c14ca20 0x7fdb3c14ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:49.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.009+0000 7fdb41570700 1 -- 172.21.15.189:0/4016404005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb3c14ca20 msgr2=0x7fdb3c14ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:49.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.009+0000 7fdb41570700 1 --2- 172.21.15.189:0/4016404005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb3c14ca20 0x7fdb3c14ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.009+0000 7fdb41570700 1 -- 172.21.15.189:0/4016404005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb3c0cf900 msgr2=0x7fdb3c14f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:49.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.009+0000 7fdb41570700 1 --2- 172.21.15.189:0/4016404005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb3c0cf900 0x7fdb3c14f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.009+0000 7fdb41570700 1 -- 172.21.15.189:0/4016404005 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb380094a0 con 0x7fdb3c0b4d70 2024-04-06T13:28:49.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.009+0000 7fdb41570700 1 --2- 172.21.15.189:0/4016404005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb3c0b4d70 0x7fdb3c14f280 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdb38016000 tx=0x7fdb38004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:49.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.009+0000 7fdb31ffb700 1 -- 172.21.15.189:0/4016404005 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb380036a0 con 0x7fdb3c0b4d70 2024-04-06T13:28:49.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.009+0000 7fdb31ffb700 1 -- 172.21.15.189:0/4016404005 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3801e070 con 0x7fdb3c0b4d70 2024-04-06T13:28:49.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.009+0000 7fdb437d4700 1 -- 172.21.15.189:0/4016404005 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb3c0a93b0 con 0x7fdb3c0b4d70 2024-04-06T13:28:49.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.009+0000 7fdb31ffb700 1 -- 172.21.15.189:0/4016404005 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb38003a40 con 0x7fdb3c0b4d70 2024-04-06T13:28:49.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.009+0000 7fdb437d4700 1 -- 172.21.15.189:0/4016404005 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3c0a9970 con 0x7fdb3c0b4d70 2024-04-06T13:28:49.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.010+0000 7fdb437d4700 1 -- 172.21.15.189:0/4016404005 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb3c067bb0 con 0x7fdb3c0b4d70 2024-04-06T13:28:49.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.011+0000 7fdb31ffb700 1 -- 172.21.15.189:0/4016404005 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 100) v1 ==== 89264+0+0 (secure 0 0 0) 0x7fdb38003ba0 con 0x7fdb3c0b4d70 2024-04-06T13:28:49.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.011+0000 7fdb31ffb700 1 --2- 172.21.15.189:0/4016404005 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdb2006ae00 0x7fdb2006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:49.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.011+0000 7fdb40d6f700 1 -- 172.21.15.189:0/4016404005 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdb2006ae00 msgr2=0x7fdb2006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:49.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.011+0000 7fdb31ffb700 1 -- 172.21.15.189:0/4016404005 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdb3808a540 con 0x7fdb3c0b4d70 2024-04-06T13:28:49.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.011+0000 7fdb40d6f700 1 --2- 172.21.15.189:0/4016404005 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdb2006ae00 0x7fdb2006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:49.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.016+0000 7fdb31ffb700 1 -- 172.21.15.189:0/4016404005 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdb38055530 con 0x7fdb3c0b4d70 2024-04-06T13:28:49.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.212+0000 7fdb40d6f700 1 -- 172.21.15.189:0/4016404005 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdb2006ae00 msgr2=0x7fdb2006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:49.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.212+0000 7fdb40d6f700 1 --2- 172.21.15.189:0/4016404005 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdb2006ae00 0x7fdb2006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:49.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.222+0000 7fdb437d4700 1 -- 172.21.15.189:0/4016404005 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fdb3c074990 con 0x7fdb3c0b4d70 2024-04-06T13:28:49.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.227+0000 7fdb31ffb700 1 -- 172.21.15.189:0/4016404005 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v59)=0 v59) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdb38058b50 con 0x7fdb3c0b4d70 2024-04-06T13:28:49.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.229+0000 7fdb437d4700 1 -- 172.21.15.189:0/4016404005 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdb2006ae00 msgr2=0x7fdb2006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:49.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.229+0000 7fdb437d4700 1 --2- 172.21.15.189:0/4016404005 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdb2006ae00 0x7fdb2006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.229+0000 7fdb437d4700 1 -- 172.21.15.189:0/4016404005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb3c0b4d70 msgr2=0x7fdb3c14f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:49.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.229+0000 7fdb437d4700 1 --2- 172.21.15.189:0/4016404005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb3c0b4d70 0x7fdb3c14f280 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdb38016000 tx=0x7fdb38004750 comp rx=0 tx=0).stop 2024-04-06T13:28:49.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.229+0000 7fdb437d4700 1 -- 172.21.15.189:0/4016404005 shutdown_connections 2024-04-06T13:28:49.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.229+0000 7fdb437d4700 1 --2- 172.21.15.189:0/4016404005 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdb2006ae00 0x7fdb2006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.229+0000 7fdb437d4700 1 --2- 172.21.15.189:0/4016404005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb3c14ca20 0x7fdb3c14ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.229+0000 7fdb437d4700 1 --2- 172.21.15.189:0/4016404005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb3c0b4d70 0x7fdb3c14f280 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.229+0000 7fdb437d4700 1 --2- 172.21.15.189:0/4016404005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb3c0cf900 0x7fdb3c14f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.229+0000 7fdb437d4700 1 -- 172.21.15.189:0/4016404005 >> 172.21.15.189:0/4016404005 conn(0x7fdb3c0d00b0 msgr2=0x7fdb3c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:49.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.229+0000 7fdb437d4700 1 -- 172.21.15.189:0/4016404005 shutdown_connections 2024-04-06T13:28:49.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.229+0000 7fdb437d4700 1 -- 172.21.15.189:0/4016404005 wait complete. 2024-04-06T13:28:49.236 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:49.236+0000 7fcbfe6f1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:28:49.244 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-06T13:28:49.262 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:49.261+0000 7f3536eb6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:28:49.314 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:49.314+0000 7fcbfe6f1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:28:49.346 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:49.345+0000 7f3536eb6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:28:49.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.375+0000 7fa5114ba700 1 Processor -- start 2024-04-06T13:28:49.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.375+0000 7fa5114ba700 1 -- start start 2024-04-06T13:28:49.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.376+0000 7fa5114ba700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa50c15a550 0x7fa50c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:49.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.376+0000 7fa5114ba700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa50c0cf900 0x7fa50c0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:49.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.376+0000 7fa5114ba700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa50c0b4ba0 0x7fa50c06eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:49.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.376+0000 7fa5114ba700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa50c0cdbe0 con 0x7fa50c0b4ba0 2024-04-06T13:28:49.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.376+0000 7fa5114ba700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa50c0b5010 con 0x7fa50c15a550 2024-04-06T13:28:49.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.376+0000 7fa5114ba700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c15a940 con 0x7fa50c0cf900 2024-04-06T13:28:49.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.376+0000 7fa50b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa50c0b4ba0 0x7fa50c06eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:49.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.376+0000 7fa50b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa50c0b4ba0 0x7fa50c06eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36198/0 (socket says 172.21.15.189:36198) 2024-04-06T13:28:49.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.376+0000 7fa50b7fe700 1 -- 172.21.15.189:0/2998394737 learned_addr learned my addr 172.21.15.189:0/2998394737 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:49.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.376+0000 7fa50affd700 1 --2- 172.21.15.189:0/2998394737 >> v2:172.21.15.192:3300/0 conn(0x7fa50c15a550 0x7fa50c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:49.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.376+0000 7fa50a7fc700 1 --2- 172.21.15.189:0/2998394737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa50c0cf900 0x7fa50c0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:49.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.377+0000 7fa50b7fe700 1 -- 172.21.15.189:0/2998394737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa50c0cf900 msgr2=0x7fa50c0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:49.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.377+0000 7fa50b7fe700 1 --2- 172.21.15.189:0/2998394737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa50c0cf900 0x7fa50c0b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.377+0000 7fa50b7fe700 1 -- 172.21.15.189:0/2998394737 >> v2:172.21.15.192:3300/0 conn(0x7fa50c15a550 msgr2=0x7fa50c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:49.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.377+0000 7fa50b7fe700 1 --2- 172.21.15.189:0/2998394737 >> v2:172.21.15.192:3300/0 conn(0x7fa50c15a550 0x7fa50c0cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.377+0000 7fa50b7fe700 1 -- 172.21.15.189:0/2998394737 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50c15ac00 con 0x7fa50c0b4ba0 2024-04-06T13:28:49.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.377+0000 7fa50b7fe700 1 --2- 172.21.15.189:0/2998394737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa50c0b4ba0 0x7fa50c06eb50 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fa50c073470 tx=0x7fa50400d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b7cd53c733448e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:49.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.377+0000 7fa509ffb700 1 -- 172.21.15.189:0/2998394737 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa504010070 con 0x7fa50c0b4ba0 2024-04-06T13:28:49.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.377+0000 7fa509ffb700 1 -- 172.21.15.189:0/2998394737 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50400dce0 con 0x7fa50c0b4ba0 2024-04-06T13:28:49.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.377+0000 7fa509ffb700 1 -- 172.21.15.189:0/2998394737 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50400b310 con 0x7fa50c0b4ba0 2024-04-06T13:28:49.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.378+0000 7fa5114ba700 1 -- 172.21.15.189:0/2998394737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa50c0b4ba0 msgr2=0x7fa50c06eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:49.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.378+0000 7fa5114ba700 1 --2- 172.21.15.189:0/2998394737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa50c0b4ba0 0x7fa50c06eb50 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fa50c073470 tx=0x7fa50400d430 comp rx=0 tx=0).stop 2024-04-06T13:28:49.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.378+0000 7fa5114ba700 1 -- 172.21.15.189:0/2998394737 shutdown_connections 2024-04-06T13:28:49.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.378+0000 7fa5114ba700 1 --2- 172.21.15.189:0/2998394737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa50c0b4ba0 0x7fa50c06eb50 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.378+0000 7fa5114ba700 1 --2- 172.21.15.189:0/2998394737 >> v2:172.21.15.192:3300/0 conn(0x7fa50c15a550 0x7fa50c0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.378+0000 7fa5114ba700 1 --2- 172.21.15.189:0/2998394737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa50c0cf900 0x7fa50c0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.378+0000 7fa5114ba700 1 -- 172.21.15.189:0/2998394737 >> 172.21.15.189:0/2998394737 conn(0x7fa50c0d00b0 msgr2=0x7fa50c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:49.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.378+0000 7fa5114ba700 1 -- 172.21.15.189:0/2998394737 shutdown_connections 2024-04-06T13:28:49.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.378+0000 7fa5114ba700 1 -- 172.21.15.189:0/2998394737 wait complete. 2024-04-06T13:28:49.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.380+0000 7fa5114ba700 1 Processor -- start 2024-04-06T13:28:49.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.380+0000 7fa5114ba700 1 -- start start 2024-04-06T13:28:49.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.380+0000 7fa5114ba700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa50c0b4ba0 0x7fa50c0ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:49.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.380+0000 7fa5114ba700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa50c0cf900 0x7fa50c1518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:49.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.380+0000 7fa5114ba700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa50c14d880 0x7fa50c0ad340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:49.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.380+0000 7fa5114ba700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c156a20 con 0x7fa50c0cf900 2024-04-06T13:28:49.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.380+0000 7fa5114ba700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c14e040 con 0x7fa50c0b4ba0 2024-04-06T13:28:49.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.380+0000 7fa5114ba700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c156080 con 0x7fa50c14d880 2024-04-06T13:28:49.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.381+0000 7fa50a7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa50c0cf900 0x7fa50c1518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:49.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.381+0000 7fa50a7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa50c0cf900 0x7fa50c1518e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36208/0 (socket says 172.21.15.189:36208) 2024-04-06T13:28:49.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.381+0000 7fa50a7fc700 1 -- 172.21.15.189:0/540120162 learned_addr learned my addr 172.21.15.189:0/540120162 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:49.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.381+0000 7fa50affd700 1 --2- 172.21.15.189:0/540120162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa50c0b4ba0 0x7fa50c0ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:49.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.381+0000 7fa50b7fe700 1 --2- 172.21.15.189:0/540120162 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa50c14d880 0x7fa50c0ad340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:49.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.381+0000 7fa50affd700 1 -- 172.21.15.189:0/540120162 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa50c14d880 msgr2=0x7fa50c0ad340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:49.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.381+0000 7fa50affd700 1 --2- 172.21.15.189:0/540120162 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa50c14d880 0x7fa50c0ad340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.381+0000 7fa50affd700 1 -- 172.21.15.189:0/540120162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa50c0cf900 msgr2=0x7fa50c1518e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:49.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.381+0000 7fa50affd700 1 --2- 172.21.15.189:0/540120162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa50c0cf900 0x7fa50c1518e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.381+0000 7fa50affd700 1 -- 172.21.15.189:0/540120162 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50400bd40 con 0x7fa50c0b4ba0 2024-04-06T13:28:49.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.381+0000 7fa50a7fc700 1 --2- 172.21.15.189:0/540120162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa50c0cf900 0x7fa50c1518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:49.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.381+0000 7fa50affd700 1 --2- 172.21.15.189:0/540120162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa50c0b4ba0 0x7fa50c0ac8c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00e900 tx=0x7fa4fc00ec10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:49.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.381+0000 7fa4f37fe700 1 -- 172.21.15.189:0/540120162 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc013680 con 0x7fa50c0b4ba0 2024-04-06T13:28:49.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.381+0000 7fa4f37fe700 1 -- 172.21.15.189:0/540120162 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4fc0137e0 con 0x7fa50c0b4ba0 2024-04-06T13:28:49.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.381+0000 7fa5114ba700 1 -- 172.21.15.189:0/540120162 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa50c156360 con 0x7fa50c0b4ba0 2024-04-06T13:28:49.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.381+0000 7fa4f37fe700 1 -- 172.21.15.189:0/540120162 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc013b50 con 0x7fa50c0b4ba0 2024-04-06T13:28:49.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.381+0000 7fa5114ba700 1 -- 172.21.15.189:0/540120162 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa50c158500 con 0x7fa50c0b4ba0 2024-04-06T13:28:49.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.382+0000 7fa5114ba700 1 -- 172.21.15.189:0/540120162 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa50c0af370 con 0x7fa50c0b4ba0 2024-04-06T13:28:49.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.383+0000 7fa4f37fe700 1 -- 172.21.15.189:0/540120162 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 100) v1 ==== 89264+0+0 (secure 0 0 0) 0x7fa4fc013cb0 con 0x7fa50c0b4ba0 2024-04-06T13:28:49.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.384+0000 7fa4f37fe700 1 --2- 172.21.15.189:0/540120162 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa4f806ae00 0x7fa4f806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:49.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.384+0000 7fa50a7fc700 1 -- 172.21.15.189:0/540120162 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa4f806ae00 msgr2=0x7fa4f806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:49.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.384+0000 7fa4f37fe700 1 -- 172.21.15.189:0/540120162 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa4fc089850 con 0x7fa50c0b4ba0 2024-04-06T13:28:49.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.384+0000 7fa50a7fc700 1 --2- 172.21.15.189:0/540120162 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa4f806ae00 0x7fa4f806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:49.386 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:49.387+0000 7fcbfe6f1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:28:49.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.388+0000 7fa4f37fe700 1 -- 172.21.15.189:0/540120162 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa4fc054840 con 0x7fa50c0b4ba0 2024-04-06T13:28:49.426 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:49.425+0000 7f3536eb6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:28:49.455 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:49.455+0000 7fcbfe6f1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:28:49.497 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:49.496+0000 7f3536eb6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:28:49.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.585+0000 7fa50a7fc700 1 -- 172.21.15.189:0/540120162 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa4f806ae00 msgr2=0x7fa4f806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:49.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.585+0000 7fa50a7fc700 1 --2- 172.21.15.189:0/540120162 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa4f806ae00 0x7fa4f806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:49.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.592+0000 7fa5114ba700 1 -- 172.21.15.189:0/540120162 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fa50c156b60 con 0x7fa50c0b4ba0 2024-04-06T13:28:49.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.597+0000 7fa4f37fe700 1 -- 172.21.15.189:0/540120162 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v60)=0 v60) v1 ==== 133+0+0 (secure 0 0 0) 0x7fa4fc017020 con 0x7fa50c0b4ba0 2024-04-06T13:28:49.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.599+0000 7fa5114ba700 1 -- 172.21.15.189:0/540120162 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa4f806ae00 msgr2=0x7fa4f806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:49.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.599+0000 7fa5114ba700 1 --2- 172.21.15.189:0/540120162 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa4f806ae00 0x7fa4f806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.599+0000 7fa5114ba700 1 -- 172.21.15.189:0/540120162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa50c0b4ba0 msgr2=0x7fa50c0ac8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:49.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.599+0000 7fa5114ba700 1 --2- 172.21.15.189:0/540120162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa50c0b4ba0 0x7fa50c0ac8c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00e900 tx=0x7fa4fc00ec10 comp rx=0 tx=0).stop 2024-04-06T13:28:49.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.599+0000 7fa5114ba700 1 -- 172.21.15.189:0/540120162 shutdown_connections 2024-04-06T13:28:49.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.599+0000 7fa5114ba700 1 --2- 172.21.15.189:0/540120162 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa50c14d880 0x7fa50c0ad340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.599+0000 7fa5114ba700 1 --2- 172.21.15.189:0/540120162 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa4f806ae00 0x7fa4f806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.599+0000 7fa5114ba700 1 --2- 172.21.15.189:0/540120162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa50c0b4ba0 0x7fa50c0ac8c0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.599+0000 7fa5114ba700 1 --2- 172.21.15.189:0/540120162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa50c0cf900 0x7fa50c1518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.599+0000 7fa5114ba700 1 -- 172.21.15.189:0/540120162 >> 172.21.15.189:0/540120162 conn(0x7fa50c0d00b0 msgr2=0x7fa50c06ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:49.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.599+0000 7fa5114ba700 1 -- 172.21.15.189:0/540120162 shutdown_connections 2024-04-06T13:28:49.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.599+0000 7fa5114ba700 1 -- 172.21.15.189:0/540120162 wait complete. 2024-04-06T13:28:49.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:49.613 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-06T13:28:49.629 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:49.629+0000 7fcbfe6f1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:28:49.676 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1h 2024-04-06T13:28:49.684 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:49.684+0000 7f3536eb6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:28:49.785 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:49.785+0000 7fcbfe6f1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:28:49.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.811+0000 7f1eb03d5700 1 Processor -- start 2024-04-06T13:28:49.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.811+0000 7f1eb03d5700 1 -- start start 2024-04-06T13:28:49.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.812+0000 7f1eb03d5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ea80b4660 0x7f1ea80b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:49.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.812+0000 7f1eb03d5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1ea80b2fc0 0x7f1ea80b3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:49.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.812+0000 7f1eb03d5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea80cdbe0 con 0x7f1ea815a550 2024-04-06T13:28:49.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.812+0000 7f1eb03d5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ea80b5000 con 0x7f1ea80b2fc0 2024-04-06T13:28:49.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.812+0000 7f1eb03d5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea815a940 con 0x7f1ea80b4660 2024-04-06T13:28:49.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.812+0000 7f1eae972700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1ea80b2fc0 0x7f1ea80b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:49.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.812+0000 7f1eae171700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1ea815a550 0x7f1ea80cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50042/0 (socket says 172.21.15.189:50042) 2024-04-06T13:28:49.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.812+0000 7f1eae171700 1 -- 172.21.15.189:0/1786036536 learned_addr learned my addr 172.21.15.189:0/1786036536 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:49.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.812+0000 7f1ead970700 1 --2- 172.21.15.189:0/1786036536 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ea80b4660 0x7f1ea80b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:49.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.812+0000 7f1eae972700 1 -- 172.21.15.189:0/1786036536 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ea80b4660 msgr2=0x7f1ea80b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:49.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.812+0000 7f1eae972700 1 --2- 172.21.15.189:0/1786036536 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ea80b4660 0x7f1ea80b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.812+0000 7f1eae972700 1 -- 172.21.15.189:0/1786036536 >> v1:172.21.15.189:6789/0 conn(0x7f1ea815a550 legacy=0x7f1ea80cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:49.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.812+0000 7f1eae972700 1 -- 172.21.15.189:0/1786036536 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea815ac00 con 0x7f1ea80b2fc0 2024-04-06T13:28:49.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.812+0000 7f1ead16f700 1 -- 172.21.15.189:0/1786036536 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098647504 0 0) 0x7f1ea80cdbe0 con 0x7f1ea815a550 2024-04-06T13:28:49.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.812+0000 7f1eae972700 1 --2- 172.21.15.189:0/1786036536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea80b2fc0 0x7f1ea80b3430 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0009d90 tx=0x7f1ea000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7890340ea4547d40 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:49.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.813+0000 7f1ead16f700 1 -- 172.21.15.189:0/1786036536 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea000e070 con 0x7f1ea80b2fc0 2024-04-06T13:28:49.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.813+0000 7f1ead16f700 1 -- 172.21.15.189:0/1786036536 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea000bb20 con 0x7f1ea80b2fc0 2024-04-06T13:28:49.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.813+0000 7f1ead16f700 1 -- 172.21.15.189:0/1786036536 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0012430 con 0x7f1ea80b2fc0 2024-04-06T13:28:49.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.813+0000 7f1eb03d5700 1 -- 172.21.15.189:0/1786036536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea80b2fc0 msgr2=0x7f1ea80b3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:49.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.813+0000 7f1eb03d5700 1 --2- 172.21.15.189:0/1786036536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea80b2fc0 0x7f1ea80b3430 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0009d90 tx=0x7f1ea000b330 comp rx=0 tx=0).stop 2024-04-06T13:28:49.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.813+0000 7f1eb03d5700 1 -- 172.21.15.189:0/1786036536 shutdown_connections 2024-04-06T13:28:49.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.813+0000 7f1eb03d5700 1 --2- 172.21.15.189:0/1786036536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea80b2fc0 0x7f1ea80b3430 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.813+0000 7f1eb03d5700 1 --2- 172.21.15.189:0/1786036536 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ea80b4660 0x7f1ea80b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.813+0000 7f1eb03d5700 1 -- 172.21.15.189:0/1786036536 >> 172.21.15.189:0/1786036536 conn(0x7f1ea80d00b0 msgr2=0x7f1ea80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:49.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.813+0000 7f1eb03d5700 1 -- 172.21.15.189:0/1786036536 shutdown_connections 2024-04-06T13:28:49.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.813+0000 7f1eb03d5700 1 -- 172.21.15.189:0/1786036536 wait complete. 2024-04-06T13:28:49.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.815+0000 7f1eb03d5700 1 Processor -- start 2024-04-06T13:28:49.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.815+0000 7f1eb03d5700 1 -- start start 2024-04-06T13:28:49.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.815+0000 7f1eb03d5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ea80b2fc0 0x7f1ea8153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:49.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.815+0000 7f1eb03d5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ea80b4660 0x7f1ea80cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:49.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.815+0000 7f1eb03d5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea80ce300 0x7f1ea81f0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:49.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.815+0000 7f1eb03d5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea80ce7b0 con 0x7f1ea80ce300 2024-04-06T13:28:49.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.815+0000 7f1eb03d5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea81f4d60 con 0x7f1ea80b4660 2024-04-06T13:28:49.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.815+0000 7f1eb03d5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea81f3fa0 con 0x7f1ea80b2fc0 2024-04-06T13:28:49.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.815+0000 7f1eae972700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea80ce300 0x7f1ea81f0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:49.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.815+0000 7f1eae972700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea80ce300 0x7f1ea81f0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36218/0 (socket says 172.21.15.189:36218) 2024-04-06T13:28:49.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.815+0000 7f1eae972700 1 -- 172.21.15.189:0/1557291619 learned_addr learned my addr 172.21.15.189:0/1557291619 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:49.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.815+0000 7f1eae171700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ea80b2fc0 0x7f1ea8153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:49.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.815+0000 7f1ead970700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ea80b4660 0x7f1ea80cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:49.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.816+0000 7f1eae972700 1 -- 172.21.15.189:0/1557291619 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ea80b2fc0 msgr2=0x7f1ea8153200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:49.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.816+0000 7f1eae972700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ea80b2fc0 0x7f1ea8153200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.816+0000 7f1eae972700 1 -- 172.21.15.189:0/1557291619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ea80b4660 msgr2=0x7f1ea80cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:49.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.816+0000 7f1eae972700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ea80b4660 0x7f1ea80cddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:49.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.816+0000 7f1eae972700 1 -- 172.21.15.189:0/1557291619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea0009a70 con 0x7f1ea80ce300 2024-04-06T13:28:49.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.816+0000 7f1ead970700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ea80b4660 0x7f1ea80cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:49.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.816+0000 7f1eae171700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ea80b2fc0 0x7f1ea8153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:49.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.816+0000 7f1eae972700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea80ce300 0x7f1ea81f0a30 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f1ea806c820 tx=0x7f1ea0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:49.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.816+0000 7f1e9effd700 1 -- 172.21.15.189:0/1557291619 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0012430 con 0x7f1ea80ce300 2024-04-06T13:28:49.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.816+0000 7f1e9effd700 1 -- 172.21.15.189:0/1557291619 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea0020070 con 0x7f1ea80ce300 2024-04-06T13:28:49.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.816+0000 7f1eb03d5700 1 -- 172.21.15.189:0/1557291619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea81f4280 con 0x7f1ea80ce300 2024-04-06T13:28:49.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.816+0000 7f1e9effd700 1 -- 172.21.15.189:0/1557291619 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0004280 con 0x7f1ea80ce300 2024-04-06T13:28:49.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.816+0000 7f1eb03d5700 1 -- 172.21.15.189:0/1557291619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea81f4840 con 0x7f1ea80ce300 2024-04-06T13:28:49.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.817+0000 7f1eb03d5700 1 -- 172.21.15.189:0/1557291619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea8067bb0 con 0x7f1ea80ce300 2024-04-06T13:28:49.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.818+0000 7f1e9effd700 1 -- 172.21.15.189:0/1557291619 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 100) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f1ea00043e0 con 0x7f1ea80ce300 2024-04-06T13:28:49.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.818+0000 7f1e9effd700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e8c06ae00 0x7f1e8c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:49.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.819+0000 7f1eae171700 1 -- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e8c06ae00 msgr2=0x7f1e8c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:49.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.819+0000 7f1e9effd700 1 -- 172.21.15.189:0/1557291619 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1ea0022070 con 0x7f1ea80ce300 2024-04-06T13:28:49.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.819+0000 7f1eae171700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e8c06ae00 0x7f1e8c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:49.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:49.823+0000 7f1e9effd700 1 -- 172.21.15.189:0/1557291619 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1ea00568c0 con 0x7f1ea80ce300 2024-04-06T13:28:49.855 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:49.854+0000 7f3536eb6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:28:49.895 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:49.895+0000 7fcbfe6f1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:28:49.969 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:49.968+0000 7f3536eb6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:28:49.975 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:49.975+0000 7fcbfe6f1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:28:50.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:50.020+0000 7f1eae171700 1 -- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e8c06ae00 msgr2=0x7f1e8c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:50.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:50.020+0000 7f1eae171700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e8c06ae00 0x7f1e8c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:50.049 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:50.050+0000 7fcbfe6f1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:28:50.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:50.052+0000 7f1eb03d5700 1 -- 172.21.15.189:0/1557291619 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ea815b4a0 con 0x7f1e8c06ae00 2024-04-06T13:28:50.053 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:50.052+0000 7f3536eb6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:28:50.131 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:50.130+0000 7f3536eb6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:28:50.250 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:50.250+0000 7fcbfe6f1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:28:50.327 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:50.327+0000 7fcbfe6f1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:28:50.338 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:50.337+0000 7f3536eb6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:28:50.414 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:50.413+0000 7f3536eb6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:28:50.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:50.421+0000 7f1eae171700 1 -- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e8c06ae00 msgr2=0x7f1e8c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:50.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:50.421+0000 7f1eae171700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e8c06ae00 0x7f1e8c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-06T13:28:50.646 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:50.646+0000 7fcbfe6f1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:28:50.734 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:50.733+0000 7f3536eb6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:28:50.964 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:50.964+0000 7fcbfe6f1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:28:51.053 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:51.053+0000 7f3536eb6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:28:51.122 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:51.122+0000 7fcbfe6f1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:28:51.194 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:51.194+0000 7fcbfe6f1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:28:51.213 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:51.212+0000 7f3536eb6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:28:51.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:51.222+0000 7f1eae171700 1 -- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e8c06ae00 msgr2=0x7f1e8c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:51.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:51.222+0000 7f1eae171700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e8c06ae00 0x7f1e8c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-06T13:28:51.287 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:51.286+0000 7f3536eb6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:28:51.808 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:51.808+0000 7fcbfe6f1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:28:51.887 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:51.888+0000 7fcbfe6f1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:28:51.904 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:51.904+0000 7f3536eb6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:28:51.982 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:51.981+0000 7f3536eb6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:28:52.012 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:52.013+0000 7fcbfe6f1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:28:52.106 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:52.106+0000 7f3536eb6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:28:52.524 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:52.524+0000 7fcbfe6f1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:28:52.615 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:52.614+0000 7f3536eb6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:28:52.678 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:52.678+0000 7fcbfe6f1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:28:52.769 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:52.768+0000 7f3536eb6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:28:52.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:52.823+0000 7f1eae171700 1 -- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e8c06ae00 msgr2=0x7f1e8c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:52.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:52.823+0000 7f1eae171700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e8c06ae00 0x7f1e8c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-06T13:28:52.833 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:52.833+0000 7fcbfe6f1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:28:52.926 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:52.926+0000 7f3536eb6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:28:53.555 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:53.555+0000 7fcbfe6f1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:28:53.650 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:53.649+0000 7f3536eb6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:28:53.797 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:53.798+0000 7fcbfe6f1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:28:53.873 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:53.873+0000 7fcbfe6f1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:28:53.893 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:53.892+0000 7f3536eb6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:28:53.969 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:53.968+0000 7f3536eb6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:28:54.244 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:54.244+0000 7fcbfe6f1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:28:54.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:54.314+0000 7f1e9effd700 1 -- 172.21.15.189:0/1557291619 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 101) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f1ea0056c90 con 0x7f1ea80ce300 2024-04-06T13:28:54.345 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:54.345+0000 7f3536eb6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:28:54.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:54.367+0000 7f1e9effd700 1 -- 172.21.15.189:0/1557291619 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mgrmap(e 102) v1 ==== 88873+0+0 (secure 0 0 0) 0x7f1ea008ca60 con 0x7f1ea80ce300 2024-04-06T13:28:54.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:54.367+0000 7f1e9effd700 1 -- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e8c06ae00 msgr2=0x7f1e8c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:54.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:54.367+0000 7f1e9effd700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e8c06ae00 0x7f1e8c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:54.511 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:54.509+0000 7f34a9583700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:54.511 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:54.509+0000 7f34a9583700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:54.511 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:54.509+0000 7f34a9583700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:54.511 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:54.510+0000 7f34a9583700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:54.511 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:54.510+0000 7f34a9583700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:54.511 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:54.510+0000 7f34acd8a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:54.511 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:54.510+0000 7f34acd8a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:54.511 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:54.510+0000 7f34acd8a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:54.512 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:54.510+0000 7f34acd8a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:54.512 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:54.510+0000 7f34acd8a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:28:55.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.372+0000 7f1e9effd700 1 -- 172.21.15.189:0/1557291619 <== mon.0 v2:172.21.15.189:3300/0 9 ==== mgrmap(e 103) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f1ea0014d20 con 0x7f1ea80ce300 2024-04-06T13:28:55.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.373+0000 7f1e9effd700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e8c06ae00 0x7f1e8c06d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:55.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.373+0000 7f1e9effd700 1 -- 172.21.15.189:0/1557291619 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e8c01d5b0 con 0x7f1e8c06ae00 2024-04-06T13:28:55.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.373+0000 7f1eae171700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e8c06ae00 0x7f1e8c06d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:55.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.376+0000 7f1eae171700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e8c06ae00 0x7f1e8c06d240 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4007fd0 tx=0x7f1ea4007de0 comp rx=0 tx=0).ready entity=mgr.9804 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:55.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.443+0000 7f1e9effd700 1 -- 172.21.15.189:0/1557291619 <== mgr.9804 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1e8c01d5b0 con 0x7f1e8c06ae00 2024-04-06T13:28:55.444 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path snap_test_dir1 2024-04-06T13:28:55.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.445+0000 7f1eb03d5700 1 -- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e8c06ae00 msgr2=0x7f1e8c06d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:55.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.445+0000 7f1eb03d5700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e8c06ae00 0x7f1e8c06d240 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4007fd0 tx=0x7f1ea4007de0 comp rx=0 tx=0).stop 2024-04-06T13:28:55.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.445+0000 7f1eb03d5700 1 -- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea80ce300 msgr2=0x7f1ea81f0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:55.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.445+0000 7f1eb03d5700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea80ce300 0x7f1ea81f0a30 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f1ea806c820 tx=0x7f1ea0004ad0 comp rx=0 tx=0).stop 2024-04-06T13:28:55.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.445+0000 7f1eb03d5700 1 -- 172.21.15.189:0/1557291619 shutdown_connections 2024-04-06T13:28:55.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.445+0000 7f1eb03d5700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea80ce300 0x7f1ea81f0a30 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:55.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.445+0000 7f1eb03d5700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ea80b2fc0 0x7f1ea8153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:55.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.445+0000 7f1eb03d5700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f1e8c06ae00 0x7f1e8c06d240 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:55.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.445+0000 7f1eb03d5700 1 --2- 172.21.15.189:0/1557291619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ea80b4660 0x7f1ea80cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:55.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.445+0000 7f1eb03d5700 1 -- 172.21.15.189:0/1557291619 >> 172.21.15.189:0/1557291619 conn(0x7f1ea80d00b0 msgr2=0x7f1ea81579b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:55.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.445+0000 7f1eb03d5700 1 -- 172.21.15.189:0/1557291619 shutdown_connections 2024-04-06T13:28:55.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.445+0000 7f1eb03d5700 1 -- 172.21.15.189:0/1557291619 wait complete. 2024-04-06T13:28:55.460 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-04-06T13:28:55.460 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-04-06T13:28:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.584+0000 7f2dd0a10700 1 Processor -- start 2024-04-06T13:28:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.584+0000 7f2dd0a10700 1 -- start start 2024-04-06T13:28:55.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.585+0000 7f2dd0a10700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2dcc06ae40 0x7f2dcc0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:55.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.585+0000 7f2dd0a10700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dcc0b3990 con 0x7f2dcc15ddf0 2024-04-06T13:28:55.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.585+0000 7f2dd0a10700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dcc0b3060 con 0x7f2dcc15d7b0 2024-04-06T13:28:55.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.585+0000 7f2dd0a10700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dcc15e950 con 0x7f2dcc06ae40 2024-04-06T13:28:55.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.586+0000 7f2dcad9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2dcc15ddf0 0x7f2dcc15e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50052/0 (socket says 172.21.15.189:50052) 2024-04-06T13:28:55.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.586+0000 7f2dcad9d700 1 -- 172.21.15.189:0/2842298213 learned_addr learned my addr 172.21.15.189:0/2842298213 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:55.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.586+0000 7f2dc9d9b700 1 --2- 172.21.15.189:0/2842298213 >> v2:172.21.15.192:3300/0 conn(0x7f2dcc06ae40 0x7f2dcc0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:55.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.586+0000 7f2dc959a700 1 -- 172.21.15.189:0/2842298213 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567439760 0 0) 0x7f2dcc0b3990 con 0x7f2dcc15ddf0 2024-04-06T13:28:55.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.586+0000 7f2dc959a700 1 -- 172.21.15.189:0/2842298213 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db40035b0 con 0x7f2dcc15ddf0 2024-04-06T13:28:55.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.586+0000 7f2dc9d9b700 1 -- 172.21.15.189:0/2842298213 >> v1:172.21.15.192:6789/0 conn(0x7f2dcc15d7b0 legacy=0x7f2dcc06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:55.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.586+0000 7f2dc9d9b700 1 -- 172.21.15.189:0/2842298213 >> v1:172.21.15.189:6789/0 conn(0x7f2dcc15ddf0 legacy=0x7f2dcc15e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:55.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.586+0000 7f2dc9d9b700 1 -- 172.21.15.189:0/2842298213 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dcc0b3340 con 0x7f2dcc06ae40 2024-04-06T13:28:55.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.586+0000 7f2dc9d9b700 1 --2- 172.21.15.189:0/2842298213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2dcc06ae40 0x7f2dcc0b3e20 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc009840 tx=0x7f2dbc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7e4718f2e285ddd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:55.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.586+0000 7f2dc959a700 1 -- 172.21.15.189:0/2842298213 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc011830 con 0x7f2dcc06ae40 2024-04-06T13:28:55.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.586+0000 7f2dc959a700 1 -- 172.21.15.189:0/2842298213 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dbc011990 con 0x7f2dcc06ae40 2024-04-06T13:28:55.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.586+0000 7f2dc959a700 1 -- 172.21.15.189:0/2842298213 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc011d20 con 0x7f2dcc06ae40 2024-04-06T13:28:55.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.587+0000 7f2dd0a10700 1 -- 172.21.15.189:0/2842298213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2dcc06ae40 msgr2=0x7f2dcc0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:55.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.587+0000 7f2dd0a10700 1 --2- 172.21.15.189:0/2842298213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2dcc06ae40 0x7f2dcc0b3e20 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc009840 tx=0x7f2dbc009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:55.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.587+0000 7f2dd0a10700 1 -- 172.21.15.189:0/2842298213 shutdown_connections 2024-04-06T13:28:55.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.587+0000 7f2dd0a10700 1 --2- 172.21.15.189:0/2842298213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2dcc06ae40 0x7f2dcc0b3e20 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:55.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.587+0000 7f2dd0a10700 1 -- 172.21.15.189:0/2842298213 >> 172.21.15.189:0/2842298213 conn(0x7f2dcc0d00b0 msgr2=0x7f2dcc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:55.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.587+0000 7f2dd0a10700 1 -- 172.21.15.189:0/2842298213 shutdown_connections 2024-04-06T13:28:55.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.587+0000 7f2dd0a10700 1 -- 172.21.15.189:0/2842298213 wait complete. 2024-04-06T13:28:55.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.588+0000 7f2dd0a10700 1 Processor -- start 2024-04-06T13:28:55.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.588+0000 7f2dd0a10700 1 -- start start 2024-04-06T13:28:55.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.589+0000 7f2dd0a10700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2dcc06ae40 0x7f2dcc0aee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:55.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.589+0000 7f2dd0a10700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2dcc15d7b0 0x7f2dcc146c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:55.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.589+0000 7f2dd0a10700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dcc15ddf0 0x7f2dcc1f6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:55.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.589+0000 7f2dd0a10700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc1f3fb0 con 0x7f2dcc15ddf0 2024-04-06T13:28:55.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.589+0000 7f2dd0a10700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc147180 con 0x7f2dcc06ae40 2024-04-06T13:28:55.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.589+0000 7f2dd0a10700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc1f3c10 con 0x7f2dcc15d7b0 2024-04-06T13:28:55.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.589+0000 7f2dcad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dcc15ddf0 0x7f2dcc1f6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:55.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.589+0000 7f2dcad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dcc15ddf0 0x7f2dcc1f6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36346/0 (socket says 172.21.15.189:36346) 2024-04-06T13:28:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.589+0000 7f2dcad9d700 1 -- 172.21.15.189:0/3166679925 learned_addr learned my addr 172.21.15.189:0/3166679925 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.589+0000 7f2dca59c700 1 --2- 172.21.15.189:0/3166679925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2dcc06ae40 0x7f2dcc0aee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.589+0000 7f2dc9d9b700 1 --2- 172.21.15.189:0/3166679925 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2dcc15d7b0 0x7f2dcc146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.589+0000 7f2dcad9d700 1 -- 172.21.15.189:0/3166679925 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2dcc15d7b0 msgr2=0x7f2dcc146c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.589+0000 7f2dcad9d700 1 --2- 172.21.15.189:0/3166679925 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2dcc15d7b0 0x7f2dcc146c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.589+0000 7f2dcad9d700 1 -- 172.21.15.189:0/3166679925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2dcc06ae40 msgr2=0x7f2dcc0aee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.589+0000 7f2dcad9d700 1 --2- 172.21.15.189:0/3166679925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2dcc06ae40 0x7f2dcc0aee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.589+0000 7f2dcad9d700 1 -- 172.21.15.189:0/3166679925 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc0094a0 con 0x7f2dcc15ddf0 2024-04-06T13:28:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.590+0000 7f2dcad9d700 1 --2- 172.21.15.189:0/3166679925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dcc15ddf0 0x7f2dcc1f6040 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f2dc400caa0 tx=0x7f2dc400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.590+0000 7f2dbaffd700 1 -- 172.21.15.189:0/3166679925 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc400aa80 con 0x7f2dcc15ddf0 2024-04-06T13:28:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.590+0000 7f2dbaffd700 1 -- 172.21.15.189:0/3166679925 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc400abe0 con 0x7f2dcc15ddf0 2024-04-06T13:28:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.590+0000 7f2dbaffd700 1 -- 172.21.15.189:0/3166679925 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc4012400 con 0x7f2dcc15ddf0 2024-04-06T13:28:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.590+0000 7f2dd0a10700 1 -- 172.21.15.189:0/3166679925 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dcc1f6890 con 0x7f2dcc15ddf0 2024-04-06T13:28:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.590+0000 7f2dd0a10700 1 -- 172.21.15.189:0/3166679925 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dcc1f6e50 con 0x7f2dcc15ddf0 2024-04-06T13:28:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.591+0000 7f2dd0a10700 1 -- 172.21.15.189:0/3166679925 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dcc1f40f0 con 0x7f2dcc15ddf0 2024-04-06T13:28:55.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.591+0000 7f2dbaffd700 1 -- 172.21.15.189:0/3166679925 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 103) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f2dc401b3b0 con 0x7f2dcc15ddf0 2024-04-06T13:28:55.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.592+0000 7f2dbaffd700 1 --2- 172.21.15.189:0/3166679925 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2dac06ab30 0x7f2dac06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:55.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.592+0000 7f2dca59c700 1 --2- 172.21.15.189:0/3166679925 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2dac06ab30 0x7f2dac06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:55.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.592+0000 7f2dbaffd700 1 -- 172.21.15.189:0/3166679925 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f2dc4089330 con 0x7f2dcc15ddf0 2024-04-06T13:28:55.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.592+0000 7f2dca59c700 1 --2- 172.21.15.189:0/3166679925 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2dac06ab30 0x7f2dac06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0000f60 tx=0x7f2dc0009040 comp rx=0 tx=0).ready entity=mgr.9804 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:55.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.596+0000 7f2dbaffd700 1 -- 172.21.15.189:0/3166679925 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2dc4054530 con 0x7f2dcc15ddf0 2024-04-06T13:28:55.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.815+0000 7f2dd0a10700 1 -- 172.21.15.189:0/3166679925 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2dcc15eb40 con 0x7f2dac06ab30 2024-04-06T13:28:55.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.878+0000 7f2dbaffd700 1 -- 172.21.15.189:0/3166679925 <== mgr.9804 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f2dc00050d0 con 0x7f2dac06ab30 2024-04-06T13:28:55.879 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule removed for path snap_test_dir1 2024-04-06T13:28:55.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.880+0000 7f2dd0a10700 1 -- 172.21.15.189:0/3166679925 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2dac06ab30 msgr2=0x7f2dac06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:55.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.880+0000 7f2dd0a10700 1 --2- 172.21.15.189:0/3166679925 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2dac06ab30 0x7f2dac06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0000f60 tx=0x7f2dc0009040 comp rx=0 tx=0).stop 2024-04-06T13:28:55.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.880+0000 7f2dd0a10700 1 -- 172.21.15.189:0/3166679925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dcc15ddf0 msgr2=0x7f2dcc1f6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:55.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.880+0000 7f2dd0a10700 1 --2- 172.21.15.189:0/3166679925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dcc15ddf0 0x7f2dcc1f6040 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f2dc400caa0 tx=0x7f2dc400ce60 comp rx=0 tx=0).stop 2024-04-06T13:28:55.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.880+0000 7f2dd0a10700 1 -- 172.21.15.189:0/3166679925 shutdown_connections 2024-04-06T13:28:55.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.880+0000 7f2dd0a10700 1 --2- 172.21.15.189:0/3166679925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dcc15ddf0 0x7f2dcc1f6040 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:55.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.880+0000 7f2dd0a10700 1 --2- 172.21.15.189:0/3166679925 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2dac06ab30 0x7f2dac06cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:55.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.880+0000 7f2dd0a10700 1 --2- 172.21.15.189:0/3166679925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2dcc06ae40 0x7f2dcc0aee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:55.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.880+0000 7f2dd0a10700 1 --2- 172.21.15.189:0/3166679925 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2dcc15d7b0 0x7f2dcc146c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:55.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.880+0000 7f2dd0a10700 1 -- 172.21.15.189:0/3166679925 >> 172.21.15.189:0/3166679925 conn(0x7f2dcc0d00b0 msgr2=0x7f2dcc0adc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:55.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.880+0000 7f2dd0a10700 1 -- 172.21.15.189:0/3166679925 shutdown_connections 2024-04-06T13:28:55.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:55.880+0000 7f2dd0a10700 1 -- 172.21.15.189:0/3166679925 wait complete. 2024-04-06T13:28:55.894 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-04-06T13:28:55.894 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-04-06T13:28:56.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.017+0000 7f16c2a84700 1 Processor -- start 2024-04-06T13:28:56.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.018+0000 7f16c2a84700 1 -- start start 2024-04-06T13:28:56.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.018+0000 7f16c2a84700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f16bc0b4660 0x7f16bc0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:56.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.018+0000 7f16c2a84700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc0b5030 con 0x7f16bc15a550 2024-04-06T13:28:56.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.018+0000 7f16c2a84700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc0cdbe0 con 0x7f16bc0b2fc0 2024-04-06T13:28:56.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.018+0000 7f16c2a84700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16bc15a940 con 0x7f16bc0b4660 2024-04-06T13:28:56.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.019+0000 7f16c0820700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f16bc15a550 0x7f16bc0cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50060/0 (socket says 172.21.15.189:50060) 2024-04-06T13:28:56.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.019+0000 7f16c0820700 1 -- 172.21.15.189:0/4063880474 learned_addr learned my addr 172.21.15.189:0/4063880474 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:56.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.019+0000 7f16bbfff700 1 --2- 172.21.15.189:0/4063880474 >> v2:172.21.15.192:3300/0 conn(0x7f16bc0b4660 0x7f16bc0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:56.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.019+0000 7f16bb7fe700 1 -- 172.21.15.189:0/4063880474 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3538255639 0 0) 0x7f16bc0b5030 con 0x7f16bc15a550 2024-04-06T13:28:56.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.019+0000 7f16bb7fe700 1 -- 172.21.15.189:0/4063880474 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16ac0035b0 con 0x7f16bc15a550 2024-04-06T13:28:56.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.019+0000 7f16bb7fe700 1 -- 172.21.15.189:0/4063880474 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110421289 0 0) 0x7f16bc0cdbe0 con 0x7f16bc0b2fc0 2024-04-06T13:28:56.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.019+0000 7f16bb7fe700 1 -- 172.21.15.189:0/4063880474 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16bc0b5030 con 0x7f16bc0b2fc0 2024-04-06T13:28:56.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.019+0000 7f16bb7fe700 1 -- 172.21.15.189:0/4063880474 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3109564580 0 0) 0x7f16ac0035b0 con 0x7f16bc15a550 2024-04-06T13:28:56.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.019+0000 7f16bbfff700 1 -- 172.21.15.189:0/4063880474 >> v1:172.21.15.192:6789/0 conn(0x7f16bc0b2fc0 legacy=0x7f16bc0b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:56.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.019+0000 7f16bbfff700 1 -- 172.21.15.189:0/4063880474 >> v1:172.21.15.189:6789/0 conn(0x7f16bc15a550 legacy=0x7f16bc0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:56.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.019+0000 7f16bbfff700 1 -- 172.21.15.189:0/4063880474 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc15ac00 con 0x7f16bc0b4660 2024-04-06T13:28:56.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.019+0000 7f16bbfff700 1 --2- 172.21.15.189:0/4063880474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16bc0b4660 0x7f16bc0b4af0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f16a8009840 tx=0x7f16a8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b537a2ae5a571904 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:56.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.019+0000 7f16bb7fe700 1 -- 172.21.15.189:0/4063880474 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b0001670 con 0x7f16bc0b4660 2024-04-06T13:28:56.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.019+0000 7f16bb7fe700 1 -- 172.21.15.189:0/4063880474 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a8004030 con 0x7f16bc0b4660 2024-04-06T13:28:56.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.019+0000 7f16bb7fe700 1 -- 172.21.15.189:0/4063880474 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a80043c0 con 0x7f16bc0b4660 2024-04-06T13:28:56.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.020+0000 7f16c2a84700 1 -- 172.21.15.189:0/4063880474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16bc0b4660 msgr2=0x7f16bc0b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:56.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.020+0000 7f16c2a84700 1 --2- 172.21.15.189:0/4063880474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16bc0b4660 0x7f16bc0b4af0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f16a8009840 tx=0x7f16a8009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:56.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.020+0000 7f16c2a84700 1 -- 172.21.15.189:0/4063880474 shutdown_connections 2024-04-06T13:28:56.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.020+0000 7f16c2a84700 1 --2- 172.21.15.189:0/4063880474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16bc0b4660 0x7f16bc0b4af0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:56.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.020+0000 7f16c2a84700 1 -- 172.21.15.189:0/4063880474 >> 172.21.15.189:0/4063880474 conn(0x7f16bc0d00b0 msgr2=0x7f16bc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:56.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.020+0000 7f16c2a84700 1 -- 172.21.15.189:0/4063880474 shutdown_connections 2024-04-06T13:28:56.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.020+0000 7f16c2a84700 1 -- 172.21.15.189:0/4063880474 wait complete. 2024-04-06T13:28:56.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.021+0000 7f16c2a84700 1 Processor -- start 2024-04-06T13:28:56.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.021+0000 7f16c2a84700 1 -- start start 2024-04-06T13:28:56.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.022+0000 7f16c2a84700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f16bc0b2fc0 0x7f16bc13f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:56.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.022+0000 7f16c2a84700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16bc0b4660 0x7f16bc06a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:56.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.022+0000 7f16c2a84700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc1561e0 0x7f16bc151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:56.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.022+0000 7f16c2a84700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc13fe80 con 0x7f16bc1561e0 2024-04-06T13:28:56.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.022+0000 7f16c2a84700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc140160 con 0x7f16bc0b4660 2024-04-06T13:28:56.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.022+0000 7f16c2a84700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc140440 con 0x7f16bc0b2fc0 2024-04-06T13:28:56.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.022+0000 7f16c1021700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc1561e0 0x7f16bc151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:56.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.022+0000 7f16c1021700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc1561e0 0x7f16bc151b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36360/0 (socket says 172.21.15.189:36360) 2024-04-06T13:28:56.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.022+0000 7f16c1021700 1 -- 172.21.15.189:0/4108200516 learned_addr learned my addr 172.21.15.189:0/4108200516 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:56.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.022+0000 7f16bbfff700 1 --2- 172.21.15.189:0/4108200516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16bc0b4660 0x7f16bc06a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:56.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.022+0000 7f16c0820700 1 --2- 172.21.15.189:0/4108200516 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f16bc0b2fc0 0x7f16bc13f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:56.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.022+0000 7f16c1021700 1 -- 172.21.15.189:0/4108200516 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f16bc0b2fc0 msgr2=0x7f16bc13f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:56.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.023+0000 7f16c1021700 1 --2- 172.21.15.189:0/4108200516 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f16bc0b2fc0 0x7f16bc13f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:56.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.023+0000 7f16c1021700 1 -- 172.21.15.189:0/4108200516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16bc0b4660 msgr2=0x7f16bc06a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:56.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.023+0000 7f16c1021700 1 --2- 172.21.15.189:0/4108200516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16bc0b4660 0x7f16bc06a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:56.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.023+0000 7f16c1021700 1 -- 172.21.15.189:0/4108200516 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a80094a0 con 0x7f16bc1561e0 2024-04-06T13:28:56.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.023+0000 7f16c1021700 1 --2- 172.21.15.189:0/4108200516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc1561e0 0x7f16bc151b80 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f16b400cc20 tx=0x7f16b400cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:56.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.023+0000 7f16b97fa700 1 -- 172.21.15.189:0/4108200516 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b4003c00 con 0x7f16bc1561e0 2024-04-06T13:28:56.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.023+0000 7f16b97fa700 1 -- 172.21.15.189:0/4108200516 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b4003d60 con 0x7f16bc1561e0 2024-04-06T13:28:56.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.023+0000 7f16c2a84700 1 -- 172.21.15.189:0/4108200516 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16bc13d590 con 0x7f16bc1561e0 2024-04-06T13:28:56.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.023+0000 7f16b97fa700 1 -- 172.21.15.189:0/4108200516 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b400aae0 con 0x7f16bc1561e0 2024-04-06T13:28:56.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.023+0000 7f16c2a84700 1 -- 172.21.15.189:0/4108200516 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16bc13fac0 con 0x7f16bc1561e0 2024-04-06T13:28:56.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.024+0000 7f16c2a84700 1 -- 172.21.15.189:0/4108200516 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16bc13fc50 con 0x7f16bc1561e0 2024-04-06T13:28:56.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.024+0000 7f16b97fa700 1 -- 172.21.15.189:0/4108200516 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 103) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f16b400ac40 con 0x7f16bc1561e0 2024-04-06T13:28:56.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.025+0000 7f16b97fa700 1 --2- 172.21.15.189:0/4108200516 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f16a006ab30 0x7f16a006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:56.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.025+0000 7f16c0820700 1 --2- 172.21.15.189:0/4108200516 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f16a006ab30 0x7f16a006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:56.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.025+0000 7f16b97fa700 1 -- 172.21.15.189:0/4108200516 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f16b4089410 con 0x7f16bc1561e0 2024-04-06T13:28:56.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.025+0000 7f16c0820700 1 --2- 172.21.15.189:0/4108200516 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f16a006ab30 0x7f16a006cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f16b00017a0 tx=0x7f16b0001670 comp rx=0 tx=0).ready entity=mgr.9804 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:56.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.029+0000 7f16b97fa700 1 -- 172.21.15.189:0/4108200516 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f16b4054510 con 0x7f16bc1561e0 2024-04-06T13:28:56.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.249+0000 7f16c2a84700 1 -- 172.21.15.189:0/4108200516 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f16bc1402d0 con 0x7f16a006ab30 2024-04-06T13:28:56.256 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:56.255+0000 7f34b3d98700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-04-06T13:28:56.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.255+0000 7f16b97fa700 1 -- 172.21.15.189:0/4108200516 <== mgr.9804 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f16bc1402d0 con 0x7f16a006ab30 2024-04-06T13:28:56.258 INFO:teuthology.orchestra.run.smithi189.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-04-06T13:28:56.261 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-06T13:28:56.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:56.262 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-06T13:28:56.284 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-06T13:28:56.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.455+0000 7fc6f2406700 1 Processor -- start 2024-04-06T13:28:56.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.456+0000 7fc6f2406700 1 -- start start 2024-04-06T13:28:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.456+0000 7fc6f2406700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec0b4ed0 0x7fc6ec0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.456+0000 7fc6f2406700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc6ec0d2880 0x7fc6ec0d2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.456+0000 7fc6f2406700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6ec0b5cc0 con 0x7fc6ec15d590 2024-04-06T13:28:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.456+0000 7fc6f2406700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6ec0b5890 con 0x7fc6ec0d2880 2024-04-06T13:28:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.456+0000 7fc6f2406700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec0d1500 con 0x7fc6ec0b4ed0 2024-04-06T13:28:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.456+0000 7fc6f09a3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc6ec0d2880 0x7fc6ec0d2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.457+0000 7fc6ebfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc6ec15d590 0x7fc6ec0b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50076/0 (socket says 172.21.15.189:50076) 2024-04-06T13:28:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.457+0000 7fc6ebfff700 1 -- 172.21.15.189:0/2475306374 learned_addr learned my addr 172.21.15.189:0/2475306374 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.457+0000 7fc6eb7fe700 1 --2- 172.21.15.189:0/2475306374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec0b4ed0 0x7fc6ec0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.457+0000 7fc6f09a3700 1 -- 172.21.15.189:0/2475306374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec0b4ed0 msgr2=0x7fc6ec0b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.457+0000 7fc6f09a3700 1 --2- 172.21.15.189:0/2475306374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec0b4ed0 0x7fc6ec0b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.457+0000 7fc6f09a3700 1 -- 172.21.15.189:0/2475306374 >> v1:172.21.15.189:6789/0 conn(0x7fc6ec15d590 legacy=0x7fc6ec0b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.457+0000 7fc6f09a3700 1 -- 172.21.15.189:0/2475306374 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ec0d17e0 con 0x7fc6ec0d2880 2024-04-06T13:28:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.457+0000 7fc6eaffd700 1 -- 172.21.15.189:0/2475306374 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310892922 0 0) 0x7fc6ec0b5cc0 con 0x7fc6ec15d590 2024-04-06T13:28:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.457+0000 7fc6eb7fe700 1 --2- 172.21.15.189:0/2475306374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec0b4ed0 0x7fc6ec0b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:28:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.457+0000 7fc6f09a3700 1 --2- 172.21.15.189:0/2475306374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0d2880 0x7fc6ec0d2d10 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4009d90 tx=0x7fc6e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9852bdc8df0b1ee4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.457+0000 7fc6eaffd700 1 -- 172.21.15.189:0/2475306374 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e400e070 con 0x7fc6ec0d2880 2024-04-06T13:28:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.457+0000 7fc6eaffd700 1 -- 172.21.15.189:0/2475306374 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e400bbd0 con 0x7fc6ec0d2880 2024-04-06T13:28:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.457+0000 7fc6eaffd700 1 -- 172.21.15.189:0/2475306374 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4012430 con 0x7fc6ec0d2880 2024-04-06T13:28:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.458+0000 7fc6f2406700 1 -- 172.21.15.189:0/2475306374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0d2880 msgr2=0x7fc6ec0d2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.458+0000 7fc6f2406700 1 --2- 172.21.15.189:0/2475306374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0d2880 0x7fc6ec0d2d10 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4009d90 tx=0x7fc6e400b330 comp rx=0 tx=0).stop 2024-04-06T13:28:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.458+0000 7fc6f2406700 1 -- 172.21.15.189:0/2475306374 shutdown_connections 2024-04-06T13:28:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.458+0000 7fc6f2406700 1 --2- 172.21.15.189:0/2475306374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0d2880 0x7fc6ec0d2d10 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.458+0000 7fc6f2406700 1 --2- 172.21.15.189:0/2475306374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec0b4ed0 0x7fc6ec0b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.458+0000 7fc6f2406700 1 -- 172.21.15.189:0/2475306374 >> 172.21.15.189:0/2475306374 conn(0x7fc6ec06bbc0 msgr2=0x7fc6ec06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:56.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.458+0000 7fc6f2406700 1 -- 172.21.15.189:0/2475306374 shutdown_connections 2024-04-06T13:28:56.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.458+0000 7fc6f2406700 1 -- 172.21.15.189:0/2475306374 wait complete. 2024-04-06T13:28:56.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.459+0000 7fc6f2406700 1 Processor -- start 2024-04-06T13:28:56.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.459+0000 7fc6f2406700 1 -- start start 2024-04-06T13:28:56.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.460+0000 7fc6f2406700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0b4ed0 0x7fc6ec0c9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:56.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.460+0000 7fc6f2406700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec0d2880 0x7fc6ec1f7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:56.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.460+0000 7fc6f2406700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6ec15d590 0x7fc6ec158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:56.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.460+0000 7fc6f2406700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec153b80 con 0x7fc6ec0b4ed0 2024-04-06T13:28:56.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.460+0000 7fc6f2406700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec15e0b0 con 0x7fc6ec15d590 2024-04-06T13:28:56.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.460+0000 7fc6f2406700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec0c95a0 con 0x7fc6ec0d2880 2024-04-06T13:28:56.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.460+0000 7fc6ebfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0b4ed0 0x7fc6ec0c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:56.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.460+0000 7fc6ebfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0b4ed0 0x7fc6ec0c9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36396/0 (socket says 172.21.15.189:36396) 2024-04-06T13:28:56.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.460+0000 7fc6ebfff700 1 -- 172.21.15.189:0/2031585846 learned_addr learned my addr 172.21.15.189:0/2031585846 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:56.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.460+0000 7fc6eb7fe700 1 --2- 172.21.15.189:0/2031585846 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec0d2880 0x7fc6ec1f7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:56.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.460+0000 7fc6f09a3700 1 --2- 172.21.15.189:0/2031585846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6ec15d590 0x7fc6ec158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:56.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.460+0000 7fc6ebfff700 1 -- 172.21.15.189:0/2031585846 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec0d2880 msgr2=0x7fc6ec1f7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:56.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.460+0000 7fc6ebfff700 1 --2- 172.21.15.189:0/2031585846 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec0d2880 0x7fc6ec1f7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:56.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.460+0000 7fc6ebfff700 1 -- 172.21.15.189:0/2031585846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6ec15d590 msgr2=0x7fc6ec158870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:56.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.460+0000 7fc6ebfff700 1 --2- 172.21.15.189:0/2031585846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6ec15d590 0x7fc6ec158870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:56.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.460+0000 7fc6ebfff700 1 -- 172.21.15.189:0/2031585846 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e4009a70 con 0x7fc6ec0b4ed0 2024-04-06T13:28:56.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.461+0000 7fc6ebfff700 1 --2- 172.21.15.189:0/2031585846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0b4ed0 0x7fc6ec0c9060 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7fc6e00011e0 tx=0x7fc6e000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:56.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.461+0000 7fc6e8ff9700 1 -- 172.21.15.189:0/2031585846 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0004c80 con 0x7fc6ec0b4ed0 2024-04-06T13:28:56.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.461+0000 7fc6e8ff9700 1 -- 172.21.15.189:0/2031585846 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e0004e00 con 0x7fc6ec0b4ed0 2024-04-06T13:28:56.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.461+0000 7fc6f2406700 1 -- 172.21.15.189:0/2031585846 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6ec1f7770 con 0x7fc6ec0b4ed0 2024-04-06T13:28:56.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.461+0000 7fc6e8ff9700 1 -- 172.21.15.189:0/2031585846 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0007310 con 0x7fc6ec0b4ed0 2024-04-06T13:28:56.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.461+0000 7fc6f2406700 1 -- 172.21.15.189:0/2031585846 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ec15daa0 con 0x7fc6ec0b4ed0 2024-04-06T13:28:56.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.462+0000 7fc6f2406700 1 -- 172.21.15.189:0/2031585846 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6ec074990 con 0x7fc6ec0b4ed0 2024-04-06T13:28:56.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.462+0000 7fc6e8ff9700 1 -- 172.21.15.189:0/2031585846 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 103) v1 ==== 89000+0+0 (secure 0 0 0) 0x7fc6e0007470 con 0x7fc6ec0b4ed0 2024-04-06T13:28:56.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.462+0000 7fc6e8ff9700 1 --2- 172.21.15.189:0/2031585846 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc6d806ab30 0x7fc6d806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:56.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.463+0000 7fc6e8ff9700 1 -- 172.21.15.189:0/2031585846 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fc6e0088eb0 con 0x7fc6ec0b4ed0 2024-04-06T13:28:56.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.463+0000 7fc6eb7fe700 1 --2- 172.21.15.189:0/2031585846 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc6d806ab30 0x7fc6d806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:56.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.463+0000 7fc6eb7fe700 1 --2- 172.21.15.189:0/2031585846 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc6d806ab30 0x7fc6d806cff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc006fd0 tx=0x7fc6dc008040 comp rx=0 tx=0).ready entity=mgr.9804 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:56.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.467+0000 7fc6e8ff9700 1 -- 172.21.15.189:0/2031585846 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc6e0054100 con 0x7fc6ec0b4ed0 2024-04-06T13:28:56.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:56.718+0000 7fc6f2406700 1 -- 172.21.15.189:0/2031585846 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fc6ec0cc470 con 0x7fc6ec0b4ed0 2024-04-06T13:28:57.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.378+0000 7fc6e8ff9700 1 -- 172.21.15.189:0/2031585846 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 104) v1 ==== 89248+0+0 (secure 0 0 0) 0x7fc6e00544d0 con 0x7fc6ec0b4ed0 2024-04-06T13:28:57.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.379+0000 7fc6e8ff9700 1 -- 172.21.15.189:0/2031585846 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v104) v1 ==== 93+0+0 (secure 0 0 0) 0x7fc6e00743d0 con 0x7fc6ec0b4ed0 2024-04-06T13:28:57.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.382+0000 7fc6f2406700 1 -- 172.21.15.189:0/2031585846 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc6d806ab30 msgr2=0x7fc6d806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:57.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.382+0000 7fc6f2406700 1 --2- 172.21.15.189:0/2031585846 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc6d806ab30 0x7fc6d806cff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc006fd0 tx=0x7fc6dc008040 comp rx=0 tx=0).stop 2024-04-06T13:28:57.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.382+0000 7fc6f2406700 1 -- 172.21.15.189:0/2031585846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0b4ed0 msgr2=0x7fc6ec0c9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:57.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.382+0000 7fc6f2406700 1 --2- 172.21.15.189:0/2031585846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0b4ed0 0x7fc6ec0c9060 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7fc6e00011e0 tx=0x7fc6e000ca30 comp rx=0 tx=0).stop 2024-04-06T13:28:57.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.382+0000 7fc6f2406700 1 -- 172.21.15.189:0/2031585846 shutdown_connections 2024-04-06T13:28:57.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.382+0000 7fc6f2406700 1 --2- 172.21.15.189:0/2031585846 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc6d806ab30 0x7fc6d806cff0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:57.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.382+0000 7fc6f2406700 1 --2- 172.21.15.189:0/2031585846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6ec15d590 0x7fc6ec158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:57.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.382+0000 7fc6f2406700 1 --2- 172.21.15.189:0/2031585846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0b4ed0 0x7fc6ec0c9060 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:57.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.382+0000 7fc6f2406700 1 --2- 172.21.15.189:0/2031585846 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6ec0d2880 0x7fc6ec1f7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:57.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.382+0000 7fc6f2406700 1 -- 172.21.15.189:0/2031585846 >> 172.21.15.189:0/2031585846 conn(0x7fc6ec06bbc0 msgr2=0x7fc6ec06bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:57.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.382+0000 7fc6f2406700 1 -- 172.21.15.189:0/2031585846 shutdown_connections 2024-04-06T13:28:57.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.382+0000 7fc6f2406700 1 -- 172.21.15.189:0/2031585846 wait complete. 2024-04-06T13:28:57.396 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T13:28:57.396 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:28:57.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:28:57.484 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T13:28:57.484 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:28:57.484 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:28:57.502 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:28:57.503 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:28:57.560 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:28:57.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.724+0000 7ffb3cf96700 1 Processor -- start 2024-04-06T13:28:57.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.724+0000 7ffb3cf96700 1 -- start start 2024-04-06T13:28:57.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.725+0000 7ffb3cf96700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffb380b4660 0x7ffb380b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:57.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.725+0000 7ffb3cf96700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ffb380b2fc0 0x7ffb380b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:57.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.725+0000 7ffb3cf96700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb380b5030 con 0x7ffb3815a550 2024-04-06T13:28:57.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.725+0000 7ffb3cf96700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb380cdbe0 con 0x7ffb380b4660 2024-04-06T13:28:57.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.725+0000 7ffb3cf96700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb3815a940 con 0x7ffb380b2fc0 2024-04-06T13:28:57.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.726+0000 7ffb35d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffb380b4660 0x7ffb380b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:57.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.726+0000 7ffb35d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffb380b4660 0x7ffb380b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36428/0 (socket says 172.21.15.189:36428) 2024-04-06T13:28:57.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.726+0000 7ffb35d9b700 1 -- 172.21.15.189:0/4288546253 learned_addr learned my addr 172.21.15.189:0/4288546253 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:57.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.726+0000 7ffb36d9d700 1 --2- 172.21.15.189:0/4288546253 >> v2:172.21.15.192:3300/0 conn(0x7ffb380b2fc0 0x7ffb380b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:57.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.726+0000 7ffb35d9b700 1 -- 172.21.15.189:0/4288546253 >> v2:172.21.15.192:3300/0 conn(0x7ffb380b2fc0 msgr2=0x7ffb380b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:57.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.726+0000 7ffb35d9b700 1 --2- 172.21.15.189:0/4288546253 >> v2:172.21.15.192:3300/0 conn(0x7ffb380b2fc0 0x7ffb380b33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:57.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.726+0000 7ffb35d9b700 1 -- 172.21.15.189:0/4288546253 >> v1:172.21.15.192:6789/0 conn(0x7ffb3815a550 legacy=0x7ffb380cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:57.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.726+0000 7ffb35d9b700 1 -- 172.21.15.189:0/4288546253 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb3815ac00 con 0x7ffb380b4660 2024-04-06T13:28:57.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.726+0000 7ffb35d9b700 1 --2- 172.21.15.189:0/4288546253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb380b4660 0x7ffb380b4af0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7ffb38072830 tx=0x7ffb2000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b84875e85953d2a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:57.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.726+0000 7ffb3559a700 1 -- 172.21.15.189:0/4288546253 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2000dc70 con 0x7ffb380b4660 2024-04-06T13:28:57.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.726+0000 7ffb3559a700 1 -- 172.21.15.189:0/4288546253 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb2000ddd0 con 0x7ffb380b4660 2024-04-06T13:28:57.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.726+0000 7ffb3559a700 1 -- 172.21.15.189:0/4288546253 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb20010690 con 0x7ffb380b4660 2024-04-06T13:28:57.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.727+0000 7ffb3cf96700 1 -- 172.21.15.189:0/4288546253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb380b4660 msgr2=0x7ffb380b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:57.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.727+0000 7ffb3cf96700 1 --2- 172.21.15.189:0/4288546253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb380b4660 0x7ffb380b4af0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7ffb38072830 tx=0x7ffb2000be70 comp rx=0 tx=0).stop 2024-04-06T13:28:57.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.727+0000 7ffb3cf96700 1 -- 172.21.15.189:0/4288546253 shutdown_connections 2024-04-06T13:28:57.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.727+0000 7ffb3cf96700 1 --2- 172.21.15.189:0/4288546253 >> v2:172.21.15.192:3300/0 conn(0x7ffb380b2fc0 0x7ffb380b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:57.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.727+0000 7ffb3cf96700 1 --2- 172.21.15.189:0/4288546253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb380b4660 0x7ffb380b4af0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:57.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.727+0000 7ffb3cf96700 1 -- 172.21.15.189:0/4288546253 >> 172.21.15.189:0/4288546253 conn(0x7ffb380d00b0 msgr2=0x7ffb380d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:57.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.727+0000 7ffb3cf96700 1 -- 172.21.15.189:0/4288546253 shutdown_connections 2024-04-06T13:28:57.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.727+0000 7ffb3cf96700 1 -- 172.21.15.189:0/4288546253 wait complete. 2024-04-06T13:28:57.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.728+0000 7ffb3cf96700 1 Processor -- start 2024-04-06T13:28:57.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.729+0000 7ffb3cf96700 1 -- start start 2024-04-06T13:28:57.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.729+0000 7ffb3cf96700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb380b2fc0 0x7ffb38149e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:57.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.729+0000 7ffb3cf96700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb380b4660 0x7ffb380ad020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:57.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.729+0000 7ffb3cf96700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb380b0c80 0x7ffb380ad560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:57.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.729+0000 7ffb3cf96700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3815c2d0 con 0x7ffb380b2fc0 2024-04-06T13:28:57.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.729+0000 7ffb3cf96700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb380b11b0 con 0x7ffb380b4660 2024-04-06T13:28:57.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.729+0000 7ffb3cf96700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb380b1490 con 0x7ffb380b0c80 2024-04-06T13:28:57.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.729+0000 7ffb3659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb380b2fc0 0x7ffb38149e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:57.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.729+0000 7ffb3659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb380b2fc0 0x7ffb38149e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36430/0 (socket says 172.21.15.189:36430) 2024-04-06T13:28:57.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.729+0000 7ffb3659c700 1 -- 172.21.15.189:0/2628404445 learned_addr learned my addr 172.21.15.189:0/2628404445 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:57.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.730+0000 7ffb35d9b700 1 --2- 172.21.15.189:0/2628404445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb380b4660 0x7ffb380ad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:57.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.730+0000 7ffb36d9d700 1 --2- 172.21.15.189:0/2628404445 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb380b0c80 0x7ffb380ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:57.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.730+0000 7ffb3659c700 1 -- 172.21.15.189:0/2628404445 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb380b0c80 msgr2=0x7ffb380ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:57.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.730+0000 7ffb3659c700 1 --2- 172.21.15.189:0/2628404445 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb380b0c80 0x7ffb380ad560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:57.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.730+0000 7ffb3659c700 1 -- 172.21.15.189:0/2628404445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb380b4660 msgr2=0x7ffb380ad020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:57.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.730+0000 7ffb3659c700 1 --2- 172.21.15.189:0/2628404445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb380b4660 0x7ffb380ad020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:57.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.730+0000 7ffb3659c700 1 -- 172.21.15.189:0/2628404445 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2000b7f0 con 0x7ffb380b2fc0 2024-04-06T13:28:57.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.730+0000 7ffb36d9d700 1 --2- 172.21.15.189:0/2628404445 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb380b0c80 0x7ffb380ad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:57.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.730+0000 7ffb35d9b700 1 --2- 172.21.15.189:0/2628404445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb380b4660 0x7ffb380ad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:57.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.730+0000 7ffb3659c700 1 --2- 172.21.15.189:0/2628404445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb380b2fc0 0x7ffb38149e20 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7ffb2800c4e0 tx=0x7ffb2800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:57.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.730+0000 7ffb2effd700 1 -- 172.21.15.189:0/2628404445 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb280052d0 con 0x7ffb380b2fc0 2024-04-06T13:28:57.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.730+0000 7ffb2effd700 1 -- 172.21.15.189:0/2628404445 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb28005430 con 0x7ffb380b2fc0 2024-04-06T13:28:57.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.730+0000 7ffb3cf96700 1 -- 172.21.15.189:0/2628404445 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb380b1770 con 0x7ffb380b2fc0 2024-04-06T13:28:57.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.730+0000 7ffb2effd700 1 -- 172.21.15.189:0/2628404445 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb280038c0 con 0x7ffb380b2fc0 2024-04-06T13:28:57.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.730+0000 7ffb3cf96700 1 -- 172.21.15.189:0/2628404445 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb381555d0 con 0x7ffb380b2fc0 2024-04-06T13:28:57.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.731+0000 7ffb3cf96700 1 -- 172.21.15.189:0/2628404445 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb380b1960 con 0x7ffb380b2fc0 2024-04-06T13:28:57.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.732+0000 7ffb2effd700 1 -- 172.21.15.189:0/2628404445 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 104) v1 ==== 89248+0+0 (secure 0 0 0) 0x7ffb28003a20 con 0x7ffb380b2fc0 2024-04-06T13:28:57.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.732+0000 7ffb2effd700 1 --2- 172.21.15.189:0/2628404445 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb1806adb0 0x7ffb1806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:57.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.732+0000 7ffb35d9b700 1 -- 172.21.15.189:0/2628404445 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb1806adb0 msgr2=0x7ffb1806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:57.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.732+0000 7ffb2effd700 1 -- 172.21.15.189:0/2628404445 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6417+0+0 (secure 0 0 0) 0x7ffb28088a70 con 0x7ffb380b2fc0 2024-04-06T13:28:57.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.732+0000 7ffb35d9b700 1 --2- 172.21.15.189:0/2628404445 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb1806adb0 0x7ffb1806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:57.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.737+0000 7ffb2effd700 1 -- 172.21.15.189:0/2628404445 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ffb380b1960 con 0x7ffb380b2fc0 2024-04-06T13:28:57.765 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:57.765+0000 7f27f75d3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:28:57.786 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:57.785+0000 7ff5abd26200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:28:57.843 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:57.843+0000 7f27f75d3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:28:57.868 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:57.867+0000 7ff5abd26200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:28:57.915 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:57.915+0000 7f27f75d3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:28:57.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.933+0000 7ffb35d9b700 1 -- 172.21.15.189:0/2628404445 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb1806adb0 msgr2=0x7ffb1806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:57.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.933+0000 7ffb35d9b700 1 --2- 172.21.15.189:0/2628404445 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb1806adb0 0x7ffb1806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:57.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.941+0000 7ffb3cf96700 1 -- 172.21.15.189:0/2628404445 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb3815c720 con 0x7ffb380b2fc0 2024-04-06T13:28:57.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.943+0000 7ffb2effd700 1 -- 172.21.15.189:0/2628404445 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+16873 (secure 0 0 0) 0x7ffb3815c720 con 0x7ffb380b2fc0 2024-04-06T13:28:57.944 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:28:57.944 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":179,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:28:54.359536+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:28:38.579643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-04-06T13:28:39.779769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/2773566419":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/2407993541":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/2622703801":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/1999978665":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/1375122784":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/1621522008":"2024-04-07T13:28:54.359501+0000","172.21.15.189:6833/47128":"2024-04-07T13:28:54.359501+0000","172.21.15.189:6832/47128":"2024-04-07T13:28:54.359501+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:28:57.944 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:57.943+0000 7ff5abd26200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:28:57.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.945+0000 7ffb3cf96700 1 -- 172.21.15.189:0/2628404445 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb1806adb0 msgr2=0x7ffb1806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:57.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.945+0000 7ffb3cf96700 1 --2- 172.21.15.189:0/2628404445 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb1806adb0 0x7ffb1806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:57.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.945+0000 7ffb3cf96700 1 -- 172.21.15.189:0/2628404445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb380b2fc0 msgr2=0x7ffb38149e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:57.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.945+0000 7ffb3cf96700 1 --2- 172.21.15.189:0/2628404445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb380b2fc0 0x7ffb38149e20 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7ffb2800c4e0 tx=0x7ffb2800c7f0 comp rx=0 tx=0).stop 2024-04-06T13:28:57.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.945+0000 7ffb3cf96700 1 -- 172.21.15.189:0/2628404445 shutdown_connections 2024-04-06T13:28:57.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.945+0000 7ffb3cf96700 1 --2- 172.21.15.189:0/2628404445 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb380b0c80 0x7ffb380ad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:57.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.945+0000 7ffb3cf96700 1 --2- 172.21.15.189:0/2628404445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb380b2fc0 0x7ffb38149e20 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:57.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.945+0000 7ffb3cf96700 1 --2- 172.21.15.189:0/2628404445 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ffb1806adb0 0x7ffb1806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:57.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.945+0000 7ffb3cf96700 1 --2- 172.21.15.189:0/2628404445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb380b4660 0x7ffb380ad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:57.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.945+0000 7ffb3cf96700 1 -- 172.21.15.189:0/2628404445 >> 172.21.15.189:0/2628404445 conn(0x7ffb380d00b0 msgr2=0x7ffb3806d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:57.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.945+0000 7ffb3cf96700 1 -- 172.21.15.189:0/2628404445 shutdown_connections 2024-04-06T13:28:57.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:57.945+0000 7ffb3cf96700 1 -- 172.21.15.189:0/2628404445 wait complete. 2024-04-06T13:28:57.959 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T13:28:38.579643+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '177', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T13:28:39.779769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '178', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:28:57.959 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:28:57.978 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:28:57.982 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:57.982+0000 7f27f75d3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:28:58.005 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:28:58.022 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:58.022+0000 7ff5abd26200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:28:58.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.090+0000 7fb897d78700 1 Processor -- start 2024-04-06T13:28:58.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.090+0000 7fb897d78700 1 -- start start 2024-04-06T13:28:58.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.091+0000 7fb897d78700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb89015d7b0 0x7fb8900b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:58.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.091+0000 7fb897d78700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb89006a730 0x7fb89006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:58.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.091+0000 7fb897d78700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8900b3990 con 0x7fb89015ddf0 2024-04-06T13:28:58.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.091+0000 7fb897d78700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb89006b0f0 con 0x7fb89006a730 2024-04-06T13:28:58.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.091+0000 7fb897d78700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8900b3060 con 0x7fb89015d7b0 2024-04-06T13:28:58.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.091+0000 7fb895313700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb89006a730 0x7fb89006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:58.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.091+0000 7fb896315700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb89015ddf0 0x7fb89015e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50086/0 (socket says 172.21.15.189:50086) 2024-04-06T13:28:58.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.091+0000 7fb896315700 1 -- 172.21.15.189:0/3251228690 learned_addr learned my addr 172.21.15.189:0/3251228690 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:58.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.091+0000 7fb895b14700 1 --2- 172.21.15.189:0/3251228690 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb89015d7b0 0x7fb8900b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:58.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.092+0000 7fb895313700 1 -- 172.21.15.189:0/3251228690 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb89015d7b0 msgr2=0x7fb8900b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:58.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.092+0000 7fb895313700 1 --2- 172.21.15.189:0/3251228690 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb89015d7b0 0x7fb8900b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.092+0000 7fb895313700 1 -- 172.21.15.189:0/3251228690 >> v1:172.21.15.189:6789/0 conn(0x7fb89015ddf0 legacy=0x7fb89015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:58.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.092+0000 7fb895313700 1 -- 172.21.15.189:0/3251228690 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8900b3340 con 0x7fb89006a730 2024-04-06T13:28:58.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.092+0000 7fb894b12700 1 -- 172.21.15.189:0/3251228690 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2161843781 0 0) 0x7fb8900b3990 con 0x7fb89015ddf0 2024-04-06T13:28:58.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.092+0000 7fb895313700 1 --2- 172.21.15.189:0/3251228690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb89006a730 0x7fb89006abb0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fb88c009840 tx=0x7fb88c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=366b5a5caa467101 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:58.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.092+0000 7fb894b12700 1 -- 172.21.15.189:0/3251228690 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c011830 con 0x7fb89006a730 2024-04-06T13:28:58.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.092+0000 7fb894b12700 1 -- 172.21.15.189:0/3251228690 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88c011990 con 0x7fb89006a730 2024-04-06T13:28:58.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.092+0000 7fb894b12700 1 -- 172.21.15.189:0/3251228690 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c011d20 con 0x7fb89006a730 2024-04-06T13:28:58.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.093+0000 7fb897d78700 1 -- 172.21.15.189:0/3251228690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb89006a730 msgr2=0x7fb89006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:58.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.093+0000 7fb897d78700 1 --2- 172.21.15.189:0/3251228690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb89006a730 0x7fb89006abb0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fb88c009840 tx=0x7fb88c009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:58.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.093+0000 7fb897d78700 1 -- 172.21.15.189:0/3251228690 shutdown_connections 2024-04-06T13:28:58.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.093+0000 7fb897d78700 1 --2- 172.21.15.189:0/3251228690 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb89015d7b0 0x7fb8900b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.093+0000 7fb897d78700 1 --2- 172.21.15.189:0/3251228690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb89006a730 0x7fb89006abb0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.093+0000 7fb897d78700 1 -- 172.21.15.189:0/3251228690 >> 172.21.15.189:0/3251228690 conn(0x7fb8900d00b0 msgr2=0x7fb8900d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:58.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.093+0000 7fb897d78700 1 -- 172.21.15.189:0/3251228690 shutdown_connections 2024-04-06T13:28:58.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.093+0000 7fb897d78700 1 -- 172.21.15.189:0/3251228690 wait complete. 2024-04-06T13:28:58.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.094+0000 7fb897d78700 1 Processor -- start 2024-04-06T13:28:58.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.094+0000 7fb897d78700 1 -- start start 2024-04-06T13:28:58.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb897d78700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb89006a730 0x7fb8900c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:58.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb897d78700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb89015d7b0 0x7fb8901f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:58.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb897d78700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb89015ddf0 0x7fb89015fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:58.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb897d78700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89014b8c0 con 0x7fb89015ddf0 2024-04-06T13:28:58.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb897d78700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89014bba0 con 0x7fb89015d7b0 2024-04-06T13:28:58.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb897d78700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89014be80 con 0x7fb89006a730 2024-04-06T13:28:58.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb896315700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb89015ddf0 0x7fb89015fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:58.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb895b14700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb89006a730 0x7fb8900c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:58.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb896315700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb89015ddf0 0x7fb89015fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36446/0 (socket says 172.21.15.189:36446) 2024-04-06T13:28:58.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb895b14700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb89006a730 0x7fb8900c4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:47798/0 (socket says 172.21.15.189:47798) 2024-04-06T13:28:58.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb896315700 1 -- 172.21.15.189:0/1224859908 learned_addr learned my addr 172.21.15.189:0/1224859908 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:58.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb895313700 1 --2- 172.21.15.189:0/1224859908 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb89015d7b0 0x7fb8901f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:58.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb896315700 1 -- 172.21.15.189:0/1224859908 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb89006a730 msgr2=0x7fb8900c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:58.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb896315700 1 --2- 172.21.15.189:0/1224859908 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb89006a730 0x7fb8900c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb896315700 1 -- 172.21.15.189:0/1224859908 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb89015d7b0 msgr2=0x7fb8901f22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:58.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb896315700 1 --2- 172.21.15.189:0/1224859908 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb89015d7b0 0x7fb8901f22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb896315700 1 -- 172.21.15.189:0/1224859908 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88c0094a0 con 0x7fb89015ddf0 2024-04-06T13:28:58.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb895b14700 1 --2- 172.21.15.189:0/1224859908 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb89006a730 0x7fb8900c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:58.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb896315700 1 --2- 172.21.15.189:0/1224859908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb89015ddf0 0x7fb89015fd10 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fb88800cba0 tx=0x7fb88800ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:58.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.095+0000 7fb8867fc700 1 -- 172.21.15.189:0/1224859908 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88800aaa0 con 0x7fb89015ddf0 2024-04-06T13:28:58.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.096+0000 7fb8867fc700 1 -- 172.21.15.189:0/1224859908 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88800ac20 con 0x7fb89015ddf0 2024-04-06T13:28:58.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.096+0000 7fb897d78700 1 -- 172.21.15.189:0/1224859908 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb89014c160 con 0x7fb89015ddf0 2024-04-06T13:28:58.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.096+0000 7fb8867fc700 1 -- 172.21.15.189:0/1224859908 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8880070d0 con 0x7fb89015ddf0 2024-04-06T13:28:58.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.096+0000 7fb897d78700 1 -- 172.21.15.189:0/1224859908 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb89014b5e0 con 0x7fb89015ddf0 2024-04-06T13:28:58.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.097+0000 7fb897d78700 1 -- 172.21.15.189:0/1224859908 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb890067bb0 con 0x7fb89015ddf0 2024-04-06T13:28:58.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.097+0000 7fb8867fc700 1 -- 172.21.15.189:0/1224859908 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 104) v1 ==== 89248+0+0 (secure 0 0 0) 0x7fb888007230 con 0x7fb89015ddf0 2024-04-06T13:28:58.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.097+0000 7fb8867fc700 1 --2- 172.21.15.189:0/1224859908 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb87406adb0 0x7fb87406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:58.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.098+0000 7fb895b14700 1 -- 172.21.15.189:0/1224859908 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb87406adb0 msgr2=0x7fb87406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:58.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.098+0000 7fb8867fc700 1 -- 172.21.15.189:0/1224859908 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fb888089480 con 0x7fb89015ddf0 2024-04-06T13:28:58.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.098+0000 7fb895b14700 1 --2- 172.21.15.189:0/1224859908 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb87406adb0 0x7fb87406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:58.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.102+0000 7fb8867fc700 1 -- 172.21.15.189:0/1224859908 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb888054280 con 0x7fb89015ddf0 2024-04-06T13:28:58.162 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:58.162+0000 7f27f75d3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:28:58.211 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:58.210+0000 7ff5abd26200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:28:58.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.299+0000 7fb895b14700 1 -- 172.21.15.189:0/1224859908 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb87406adb0 msgr2=0x7fb87406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:58.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.299+0000 7fb895b14700 1 --2- 172.21.15.189:0/1224859908 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb87406adb0 0x7fb87406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:58.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.302+0000 7fb897d78700 1 -- 172.21.15.189:0/1224859908 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb890074990 con 0x7fb89015ddf0 2024-04-06T13:28:58.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.302+0000 7fb8867fc700 1 -- 172.21.15.189:0/1224859908 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v179) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb8880578a0 con 0x7fb89015ddf0 2024-04-06T13:28:58.303 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:28:58.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.304+0000 7fb897d78700 1 -- 172.21.15.189:0/1224859908 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb87406adb0 msgr2=0x7fb87406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:58.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.304+0000 7fb897d78700 1 --2- 172.21.15.189:0/1224859908 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb87406adb0 0x7fb87406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.304+0000 7fb897d78700 1 -- 172.21.15.189:0/1224859908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb89015ddf0 msgr2=0x7fb89015fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:58.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.304+0000 7fb897d78700 1 --2- 172.21.15.189:0/1224859908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb89015ddf0 0x7fb89015fd10 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fb88800cba0 tx=0x7fb88800ceb0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.304+0000 7fb897d78700 1 -- 172.21.15.189:0/1224859908 shutdown_connections 2024-04-06T13:28:58.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.304+0000 7fb897d78700 1 --2- 172.21.15.189:0/1224859908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb89015ddf0 0x7fb89015fd10 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.304+0000 7fb897d78700 1 --2- 172.21.15.189:0/1224859908 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb89006a730 0x7fb8900c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.304+0000 7fb897d78700 1 --2- 172.21.15.189:0/1224859908 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fb87406adb0 0x7fb87406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.304+0000 7fb897d78700 1 --2- 172.21.15.189:0/1224859908 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb89015d7b0 0x7fb8901f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.304+0000 7fb897d78700 1 -- 172.21.15.189:0/1224859908 >> 172.21.15.189:0/1224859908 conn(0x7fb8900d00b0 msgr2=0x7fb8900aeb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:58.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.304+0000 7fb897d78700 1 -- 172.21.15.189:0/1224859908 shutdown_connections 2024-04-06T13:28:58.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.304+0000 7fb897d78700 1 -- 172.21.15.189:0/1224859908 wait complete. 2024-04-06T13:28:58.317 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:58.317+0000 7f27f75d3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:28:58.318 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T13:28:58.378 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:58.377+0000 7ff5abd26200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:28:58.427 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:58.427+0000 7f27f75d3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:28:58.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.488+0000 7f2dce11b700 1 Processor -- start 2024-04-06T13:28:58.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.488+0000 7f2dce11b700 1 -- start start 2024-04-06T13:28:58.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.488+0000 7f2dce11b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2dc815d7b0 0x7f2dc80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:58.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.489+0000 7f2dce11b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc80b3990 con 0x7f2dc806a730 2024-04-06T13:28:58.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.489+0000 7f2dce11b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc80b3060 con 0x7f2dc815e500 2024-04-06T13:28:58.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.489+0000 7f2dce11b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc815e950 con 0x7f2dc815d7b0 2024-04-06T13:28:58.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.489+0000 7f2dc77fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2dc815d7b0 0x7f2dc80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:58.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.489+0000 7f2dc6ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2dc806a730 0x7f2dc815ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50102/0 (socket says 172.21.15.189:50102) 2024-04-06T13:28:58.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.489+0000 7f2dc6ffd700 1 -- 172.21.15.189:0/569122216 learned_addr learned my addr 172.21.15.189:0/569122216 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:58.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.489+0000 7f2dc77fe700 1 -- 172.21.15.189:0/569122216 >> v1:172.21.15.192:6789/0 conn(0x7f2dc815e500 legacy=0x7f2dc80717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:58.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.489+0000 7f2dc77fe700 1 -- 172.21.15.189:0/569122216 >> v1:172.21.15.189:6789/0 conn(0x7f2dc806a730 legacy=0x7f2dc815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:58.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.489+0000 7f2dc77fe700 1 -- 172.21.15.189:0/569122216 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc80b3340 con 0x7f2dc815d7b0 2024-04-06T13:28:58.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.489+0000 7f2dc77fe700 1 --2- 172.21.15.189:0/569122216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dc815d7b0 0x7f2dc80b3e20 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f2db8009840 tx=0x7f2db8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b40bc0b70f061fc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:58.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.489+0000 7f2dc67fc700 1 -- 172.21.15.189:0/569122216 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db8011830 con 0x7f2dc815d7b0 2024-04-06T13:28:58.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.489+0000 7f2dc67fc700 1 -- 172.21.15.189:0/569122216 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db8011990 con 0x7f2dc815d7b0 2024-04-06T13:28:58.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.489+0000 7f2dc67fc700 1 -- 172.21.15.189:0/569122216 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db8011d20 con 0x7f2dc815d7b0 2024-04-06T13:28:58.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.490+0000 7f2dce11b700 1 -- 172.21.15.189:0/569122216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dc815d7b0 msgr2=0x7f2dc80b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:58.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.490+0000 7f2dce11b700 1 --2- 172.21.15.189:0/569122216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dc815d7b0 0x7f2dc80b3e20 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f2db8009840 tx=0x7f2db8009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:58.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.490+0000 7f2dce11b700 1 -- 172.21.15.189:0/569122216 shutdown_connections 2024-04-06T13:28:58.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.490+0000 7f2dce11b700 1 --2- 172.21.15.189:0/569122216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dc815d7b0 0x7f2dc80b3e20 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.490+0000 7f2dce11b700 1 -- 172.21.15.189:0/569122216 >> 172.21.15.189:0/569122216 conn(0x7f2dc80d00b0 msgr2=0x7f2dc80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:58.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.490+0000 7f2dce11b700 1 -- 172.21.15.189:0/569122216 shutdown_connections 2024-04-06T13:28:58.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.490+0000 7f2dce11b700 1 -- 172.21.15.189:0/569122216 wait complete. 2024-04-06T13:28:58.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.491+0000 7f2dce11b700 1 Processor -- start 2024-04-06T13:28:58.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.491+0000 7f2dce11b700 1 -- start start 2024-04-06T13:28:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.492+0000 7f2dce11b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2dc806a730 0x7f2dc8072930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.492+0000 7f2dce11b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dc815d7b0 0x7f2dc8152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.492+0000 7f2dce11b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2dc815e500 0x7f2dc815fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.492+0000 7f2dce11b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc8151a80 con 0x7f2dc815d7b0 2024-04-06T13:28:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.492+0000 7f2dce11b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc814e460 con 0x7f2dc806a730 2024-04-06T13:28:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.492+0000 7f2dce11b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc81511e0 con 0x7f2dc815e500 2024-04-06T13:28:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.492+0000 7f2dc6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dc815d7b0 0x7f2dc8152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.492+0000 7f2dc6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dc815d7b0 0x7f2dc8152580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36454/0 (socket says 172.21.15.189:36454) 2024-04-06T13:28:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.492+0000 7f2dc6ffd700 1 -- 172.21.15.189:0/3936290451 learned_addr learned my addr 172.21.15.189:0/3936290451 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:58.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.492+0000 7f2dc77fe700 1 --2- 172.21.15.189:0/3936290451 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2dc806a730 0x7f2dc8072930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:58.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.492+0000 7f2dc7fff700 1 --2- 172.21.15.189:0/3936290451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2dc815e500 0x7f2dc815fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:58.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.492+0000 7f2dc6ffd700 1 -- 172.21.15.189:0/3936290451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2dc815e500 msgr2=0x7f2dc815fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:58.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.492+0000 7f2dc6ffd700 1 --2- 172.21.15.189:0/3936290451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2dc815e500 0x7f2dc815fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.492+0000 7f2dc6ffd700 1 -- 172.21.15.189:0/3936290451 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2dc806a730 msgr2=0x7f2dc8072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:58.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.492+0000 7f2dc6ffd700 1 --2- 172.21.15.189:0/3936290451 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2dc806a730 0x7f2dc8072930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.492+0000 7f2dc6ffd700 1 -- 172.21.15.189:0/3936290451 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db80094a0 con 0x7f2dc815d7b0 2024-04-06T13:28:58.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.493+0000 7f2dc6ffd700 1 --2- 172.21.15.189:0/3936290451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dc815d7b0 0x7f2dc8152580 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc00c520 tx=0x7f2dbc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:58.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.493+0000 7f2daffff700 1 -- 172.21.15.189:0/3936290451 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc005d70 con 0x7f2dc815d7b0 2024-04-06T13:28:58.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.493+0000 7f2daffff700 1 -- 172.21.15.189:0/3936290451 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dbc0070d0 con 0x7f2dc815d7b0 2024-04-06T13:28:58.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.493+0000 7f2dce11b700 1 -- 172.21.15.189:0/3936290451 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc8150e60 con 0x7f2dc815d7b0 2024-04-06T13:28:58.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.493+0000 7f2daffff700 1 -- 172.21.15.189:0/3936290451 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc007440 con 0x7f2dc815d7b0 2024-04-06T13:28:58.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.493+0000 7f2dce11b700 1 -- 172.21.15.189:0/3936290451 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc8151430 con 0x7f2dc815d7b0 2024-04-06T13:28:58.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.494+0000 7f2dce11b700 1 -- 172.21.15.189:0/3936290451 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc81547b0 con 0x7f2dc815d7b0 2024-04-06T13:28:58.497 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:58.494+0000 7ff5abd26200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:28:58.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.494+0000 7f2daffff700 1 -- 172.21.15.189:0/3936290451 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 104) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f2dbc0075a0 con 0x7f2dc815d7b0 2024-04-06T13:28:58.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.495+0000 7f2daffff700 1 --2- 172.21.15.189:0/3936290451 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2db406adb0 0x7f2db406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:58.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.495+0000 7f2dc77fe700 1 -- 172.21.15.189:0/3936290451 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2db406adb0 msgr2=0x7f2db406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:58.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.495+0000 7f2daffff700 1 -- 172.21.15.189:0/3936290451 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f2dbc088a20 con 0x7f2dc815d7b0 2024-04-06T13:28:58.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.495+0000 7f2dc77fe700 1 --2- 172.21.15.189:0/3936290451 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2db406adb0 0x7f2db406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:58.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.499+0000 7f2daffff700 1 -- 172.21.15.189:0/3936290451 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2dbc0538d0 con 0x7f2dc815d7b0 2024-04-06T13:28:58.509 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:58.509+0000 7f27f75d3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:28:58.581 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:58.580+0000 7ff5abd26200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:28:58.584 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:58.584+0000 7f27f75d3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:28:58.659 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:58.658+0000 7ff5abd26200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:28:58.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.696+0000 7f2dc77fe700 1 -- 172.21.15.189:0/3936290451 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2db406adb0 msgr2=0x7f2db406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:58.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.696+0000 7f2dc77fe700 1 --2- 172.21.15.189:0/3936290451 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2db406adb0 0x7f2db406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:58.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.701+0000 7f2dce11b700 1 -- 172.21.15.189:0/3936290451 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2dc8067bb0 con 0x7f2dc815d7b0 2024-04-06T13:28:58.702 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.702+0000 7f2daffff700 1 -- 172.21.15.189:0/3936290451 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v179) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2dbc017020 con 0x7f2dc815d7b0 2024-04-06T13:28:58.703 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:28:58.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.704+0000 7f2dce11b700 1 -- 172.21.15.189:0/3936290451 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2db406adb0 msgr2=0x7f2db406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:58.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.704+0000 7f2dce11b700 1 --2- 172.21.15.189:0/3936290451 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2db406adb0 0x7f2db406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.704+0000 7f2dce11b700 1 -- 172.21.15.189:0/3936290451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dc815d7b0 msgr2=0x7f2dc8152580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:58.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.704+0000 7f2dce11b700 1 --2- 172.21.15.189:0/3936290451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dc815d7b0 0x7f2dc8152580 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc00c520 tx=0x7f2dbc00c830 comp rx=0 tx=0).stop 2024-04-06T13:28:58.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.704+0000 7f2dce11b700 1 -- 172.21.15.189:0/3936290451 shutdown_connections 2024-04-06T13:28:58.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.704+0000 7f2dce11b700 1 --2- 172.21.15.189:0/3936290451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2dc815e500 0x7f2dc815fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.704+0000 7f2dce11b700 1 --2- 172.21.15.189:0/3936290451 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2db406adb0 0x7f2db406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.704+0000 7f2dce11b700 1 --2- 172.21.15.189:0/3936290451 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2dc806a730 0x7f2dc8072930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.704+0000 7f2dce11b700 1 --2- 172.21.15.189:0/3936290451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dc815d7b0 0x7f2dc8152580 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.704+0000 7f2dce11b700 1 -- 172.21.15.189:0/3936290451 >> 172.21.15.189:0/3936290451 conn(0x7f2dc80d00b0 msgr2=0x7f2dc806eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:58.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.704+0000 7f2dce11b700 1 -- 172.21.15.189:0/3936290451 shutdown_connections 2024-04-06T13:28:58.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.704+0000 7f2dce11b700 1 -- 172.21.15.189:0/3936290451 wait complete. 2024-04-06T13:28:58.718 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T13:28:58.786 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:58.786+0000 7f27f75d3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:28:58.863 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:58.863+0000 7f27f75d3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:28:58.874 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:58.873+0000 7ff5abd26200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:28:58.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.908+0000 7fde82b2c700 1 Processor -- start 2024-04-06T13:28:58.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.908+0000 7fde82b2c700 1 -- start start 2024-04-06T13:28:58.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.909+0000 7fde82b2c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fde7c15d7e0 0x7fde7c0d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:58.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.909+0000 7fde82b2c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde7c0cdf80 con 0x7fde7c0d1cc0 2024-04-06T13:28:58.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.909+0000 7fde82b2c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde7c0d0cf0 con 0x7fde7c0d0890 2024-04-06T13:28:58.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.909+0000 7fde82b2c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde7c0d3230 con 0x7fde7c15d7e0 2024-04-06T13:28:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.909+0000 7fde808c8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fde7c15d7e0 0x7fde7c0d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.909+0000 7fde810c9700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fde7c0d1cc0 0x7fde7c0b66b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50106/0 (socket says 172.21.15.189:50106) 2024-04-06T13:28:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.909+0000 7fde810c9700 1 -- 172.21.15.189:0/2588638799 learned_addr learned my addr 172.21.15.189:0/2588638799 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.909+0000 7fde808c8700 1 -- 172.21.15.189:0/2588638799 >> v1:172.21.15.192:6789/0 conn(0x7fde7c0d0890 legacy=0x7fde7c0d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.909+0000 7fde808c8700 1 -- 172.21.15.189:0/2588638799 >> v1:172.21.15.189:6789/0 conn(0x7fde7c0d1cc0 legacy=0x7fde7c0b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.909+0000 7fde808c8700 1 -- 172.21.15.189:0/2588638799 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7c0d2da0 con 0x7fde7c15d7e0 2024-04-06T13:28:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.910+0000 7fde808c8700 1 --2- 172.21.15.189:0/2588638799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde7c15d7e0 0x7fde7c0d0350 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fde70009840 tx=0x7fde70009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6ad046e974b4022 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.910+0000 7fde7b7fe700 1 -- 172.21.15.189:0/2588638799 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde70011830 con 0x7fde7c15d7e0 2024-04-06T13:28:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.910+0000 7fde7b7fe700 1 -- 172.21.15.189:0/2588638799 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde70011990 con 0x7fde7c15d7e0 2024-04-06T13:28:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.910+0000 7fde7b7fe700 1 -- 172.21.15.189:0/2588638799 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde70011d20 con 0x7fde7c15d7e0 2024-04-06T13:28:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.910+0000 7fde82b2c700 1 -- 172.21.15.189:0/2588638799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde7c15d7e0 msgr2=0x7fde7c0d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.910+0000 7fde82b2c700 1 --2- 172.21.15.189:0/2588638799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde7c15d7e0 0x7fde7c0d0350 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fde70009840 tx=0x7fde70009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.910+0000 7fde82b2c700 1 -- 172.21.15.189:0/2588638799 shutdown_connections 2024-04-06T13:28:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.910+0000 7fde82b2c700 1 --2- 172.21.15.189:0/2588638799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde7c15d7e0 0x7fde7c0d0350 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.910+0000 7fde82b2c700 1 -- 172.21.15.189:0/2588638799 >> 172.21.15.189:0/2588638799 conn(0x7fde7c072250 msgr2=0x7fde7c072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:58.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.910+0000 7fde82b2c700 1 -- 172.21.15.189:0/2588638799 shutdown_connections 2024-04-06T13:28:58.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.910+0000 7fde82b2c700 1 -- 172.21.15.189:0/2588638799 wait complete. 2024-04-06T13:28:58.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.912+0000 7fde82b2c700 1 Processor -- start 2024-04-06T13:28:58.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.912+0000 7fde82b2c700 1 -- start start 2024-04-06T13:28:58.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.912+0000 7fde82b2c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde7c0d0890 0x7fde7c148370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:58.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde82b2c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde7c0d1cc0 0x7fde7c148a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:58.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde82b2c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde7c15d7e0 0x7fde7c14ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:58.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde82b2c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c149550 con 0x7fde7c15d7e0 2024-04-06T13:28:58.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde82b2c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c149830 con 0x7fde7c0d0890 2024-04-06T13:28:58.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde82b2c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c149b10 con 0x7fde7c0d1cc0 2024-04-06T13:28:58.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde810c9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde7c15d7e0 0x7fde7c14ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:58.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde810c9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde7c15d7e0 0x7fde7c14ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36466/0 (socket says 172.21.15.189:36466) 2024-04-06T13:28:58.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde810c9700 1 -- 172.21.15.189:0/2547955738 learned_addr learned my addr 172.21.15.189:0/2547955738 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:58.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde808c8700 1 --2- 172.21.15.189:0/2547955738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde7c0d0890 0x7fde7c148370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:58.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde7bfff700 1 --2- 172.21.15.189:0/2547955738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde7c0d1cc0 0x7fde7c148a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:58.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde810c9700 1 -- 172.21.15.189:0/2547955738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde7c0d1cc0 msgr2=0x7fde7c148a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:58.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde810c9700 1 --2- 172.21.15.189:0/2547955738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde7c0d1cc0 0x7fde7c148a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde810c9700 1 -- 172.21.15.189:0/2547955738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde7c0d0890 msgr2=0x7fde7c148370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:58.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde810c9700 1 --2- 172.21.15.189:0/2547955738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde7c0d0890 0x7fde7c148370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:58.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde810c9700 1 -- 172.21.15.189:0/2547955738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde700094a0 con 0x7fde7c15d7e0 2024-04-06T13:28:58.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde810c9700 1 --2- 172.21.15.189:0/2547955738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde7c15d7e0 0x7fde7c14ab20 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7fde7400cb10 tx=0x7fde7400ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:58.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde797fa700 1 -- 172.21.15.189:0/2547955738 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7400a9e0 con 0x7fde7c15d7e0 2024-04-06T13:28:58.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde797fa700 1 -- 172.21.15.189:0/2547955738 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7400ab40 con 0x7fde7c15d7e0 2024-04-06T13:28:58.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde797fa700 1 -- 172.21.15.189:0/2547955738 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74012400 con 0x7fde7c15d7e0 2024-04-06T13:28:58.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.913+0000 7fde82b2c700 1 -- 172.21.15.189:0/2547955738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7c149df0 con 0x7fde7c15d7e0 2024-04-06T13:28:58.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.914+0000 7fde82b2c700 1 -- 172.21.15.189:0/2547955738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7c15bdb0 con 0x7fde7c15d7e0 2024-04-06T13:28:58.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.914+0000 7fde82b2c700 1 -- 172.21.15.189:0/2547955738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde7c149970 con 0x7fde7c15d7e0 2024-04-06T13:28:58.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.916+0000 7fde797fa700 1 -- 172.21.15.189:0/2547955738 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 104) v1 ==== 89248+0+0 (secure 0 0 0) 0x7fde7401b3b0 con 0x7fde7c15d7e0 2024-04-06T13:28:58.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.916+0000 7fde797fa700 1 --2- 172.21.15.189:0/2547955738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fde6006adb0 0x7fde6006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:58.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.916+0000 7fde808c8700 1 -- 172.21.15.189:0/2547955738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fde6006adb0 msgr2=0x7fde6006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:58.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.916+0000 7fde797fa700 1 -- 172.21.15.189:0/2547955738 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fde74089120 con 0x7fde7c15d7e0 2024-04-06T13:28:58.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.916+0000 7fde808c8700 1 --2- 172.21.15.189:0/2547955738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fde6006adb0 0x7fde6006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:58.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:58.920+0000 7fde797fa700 1 -- 172.21.15.189:0/2547955738 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fde7c149970 con 0x7fde7c15d7e0 2024-04-06T13:28:58.955 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:58.954+0000 7ff5abd26200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:28:59.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.117+0000 7fde808c8700 1 -- 172.21.15.189:0/2547955738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fde6006adb0 msgr2=0x7fde6006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:59.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.117+0000 7fde808c8700 1 --2- 172.21.15.189:0/2547955738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fde6006adb0 0x7fde6006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:59.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.122+0000 7fde82b2c700 1 -- 172.21.15.189:0/2547955738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fde7c149970 con 0x7fde7c15d7e0 2024-04-06T13:28:59.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.123+0000 7fde797fa700 1 -- 172.21.15.189:0/2547955738 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v179) v1 ==== 100+0+11 (secure 0 0 0) 0x7fde7c149970 con 0x7fde7c15d7e0 2024-04-06T13:28:59.124 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:28:59.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.125+0000 7fde82b2c700 1 -- 172.21.15.189:0/2547955738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fde6006adb0 msgr2=0x7fde6006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:59.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.125+0000 7fde82b2c700 1 --2- 172.21.15.189:0/2547955738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fde6006adb0 0x7fde6006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:59.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.125+0000 7fde82b2c700 1 -- 172.21.15.189:0/2547955738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde7c15d7e0 msgr2=0x7fde7c14ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:59.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.125+0000 7fde82b2c700 1 --2- 172.21.15.189:0/2547955738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde7c15d7e0 0x7fde7c14ab20 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7fde7400cb10 tx=0x7fde7400ce20 comp rx=0 tx=0).stop 2024-04-06T13:28:59.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.126+0000 7fde82b2c700 1 -- 172.21.15.189:0/2547955738 shutdown_connections 2024-04-06T13:28:59.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.126+0000 7fde82b2c700 1 --2- 172.21.15.189:0/2547955738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde7c15d7e0 0x7fde7c14ab20 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:59.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.126+0000 7fde82b2c700 1 --2- 172.21.15.189:0/2547955738 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fde6006adb0 0x7fde6006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:59.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.126+0000 7fde82b2c700 1 --2- 172.21.15.189:0/2547955738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde7c0d0890 0x7fde7c148370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:59.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.126+0000 7fde82b2c700 1 --2- 172.21.15.189:0/2547955738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde7c0d1cc0 0x7fde7c148a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:59.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.126+0000 7fde82b2c700 1 -- 172.21.15.189:0/2547955738 >> 172.21.15.189:0/2547955738 conn(0x7fde7c072250 msgr2=0x7fde7c15e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:59.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.126+0000 7fde82b2c700 1 -- 172.21.15.189:0/2547955738 shutdown_connections 2024-04-06T13:28:59.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.126+0000 7fde82b2c700 1 -- 172.21.15.189:0/2547955738 wait complete. 2024-04-06T13:28:59.140 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T13:28:59.182 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:59.183+0000 7f27f75d3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:28:59.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.266+0000 7f50199a9700 1 Processor -- start 2024-04-06T13:28:59.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.266+0000 7f50199a9700 1 -- start start 2024-04-06T13:28:59.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.266+0000 7f50199a9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50140d11b0 0x7f50140d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:59.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.266+0000 7f50199a9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f50140cb870 0x7f501415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:59.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.266+0000 7f50199a9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50140d1ad0 con 0x7f50140cfa00 2024-04-06T13:28:59.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.266+0000 7f50199a9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50140ccdc0 con 0x7f50140cb870 2024-04-06T13:28:59.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.266+0000 7f50199a9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50140d0b50 con 0x7f50140d11b0 2024-04-06T13:28:59.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.267+0000 7f50127fc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f50140cb870 0x7f501415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.267+0000 7f5012ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50140d11b0 0x7f50140d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.267+0000 7f50137fe700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f50140cfa00 0x7f50140cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:58636/0 (socket says 172.21.15.189:58636) 2024-04-06T13:28:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.267+0000 7f50137fe700 1 -- 172.21.15.189:0/3495864836 learned_addr learned my addr 172.21.15.189:0/3495864836 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.267+0000 7f50127fc700 1 -- 172.21.15.189:0/3495864836 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50140d11b0 msgr2=0x7f50140d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:28:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.267+0000 7f50127fc700 1 --2- 172.21.15.189:0/3495864836 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50140d11b0 0x7f50140d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.267+0000 7f50127fc700 1 -- 172.21.15.189:0/3495864836 >> v1:172.21.15.192:6789/0 conn(0x7f50140cfa00 legacy=0x7f50140cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.267+0000 7f50127fc700 1 -- 172.21.15.189:0/3495864836 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50140cbd80 con 0x7f50140cb870 2024-04-06T13:28:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.267+0000 7f5011ffb700 1 -- 172.21.15.189:0/3495864836 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994166644 0 0) 0x7f50140d1ad0 con 0x7f50140cfa00 2024-04-06T13:28:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.267+0000 7f50127fc700 1 --2- 172.21.15.189:0/3495864836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50140cb870 0x7f501415c280 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f5008009840 tx=0x7f5008009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c345bd61222c518 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.267+0000 7f5011ffb700 1 -- 172.21.15.189:0/3495864836 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5008011830 con 0x7f50140cb870 2024-04-06T13:28:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.267+0000 7f5011ffb700 1 -- 172.21.15.189:0/3495864836 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5008011990 con 0x7f50140cb870 2024-04-06T13:28:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.267+0000 7f5011ffb700 1 -- 172.21.15.189:0/3495864836 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5008011d20 con 0x7f50140cb870 2024-04-06T13:28:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.268+0000 7f50199a9700 1 -- 172.21.15.189:0/3495864836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50140cb870 msgr2=0x7f501415c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:59.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.268+0000 7f50199a9700 1 --2- 172.21.15.189:0/3495864836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50140cb870 0x7f501415c280 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f5008009840 tx=0x7f5008009b50 comp rx=0 tx=0).stop 2024-04-06T13:28:59.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.268+0000 7f50199a9700 1 -- 172.21.15.189:0/3495864836 shutdown_connections 2024-04-06T13:28:59.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.268+0000 7f50199a9700 1 --2- 172.21.15.189:0/3495864836 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50140d11b0 0x7f50140d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:59.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.268+0000 7f50199a9700 1 --2- 172.21.15.189:0/3495864836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50140cb870 0x7f501415c280 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:59.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.268+0000 7f50199a9700 1 -- 172.21.15.189:0/3495864836 >> 172.21.15.189:0/3495864836 conn(0x7f50140b2db0 msgr2=0x7f50140b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:59.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.268+0000 7f50199a9700 1 -- 172.21.15.189:0/3495864836 shutdown_connections 2024-04-06T13:28:59.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.268+0000 7f50199a9700 1 -- 172.21.15.189:0/3495864836 wait complete. 2024-04-06T13:28:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.269+0000 7f50199a9700 1 Processor -- start 2024-04-06T13:28:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.269+0000 7f50199a9700 1 -- start start 2024-04-06T13:28:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.270+0000 7f50199a9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50140cb870 0x7f5014154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.270+0000 7f50199a9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50140cfa00 0x7f50140c7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.270+0000 7f50199a9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50140d11b0 0x7f50140ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.270+0000 7f50199a9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50140b58d0 con 0x7f50140d11b0 2024-04-06T13:28:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.270+0000 7f50199a9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5014155110 con 0x7f50140cb870 2024-04-06T13:28:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.270+0000 7f50199a9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50140b5b50 con 0x7f50140cfa00 2024-04-06T13:28:59.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.270+0000 7f50137fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50140d11b0 0x7f50140ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:59.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.270+0000 7f5012ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50140cb870 0x7f5014154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:59.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.270+0000 7f5012ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50140cb870 0x7f5014154bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:37626/0 (socket says 172.21.15.189:37626) 2024-04-06T13:28:59.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.270+0000 7f50127fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50140cfa00 0x7f50140c7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:59.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.270+0000 7f50137fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50140d11b0 0x7f50140ca060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:36468/0 (socket says 172.21.15.189:36468) 2024-04-06T13:28:59.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.270+0000 7f5012ffd700 1 -- 172.21.15.189:0/3626677101 learned_addr learned my addr 172.21.15.189:0/3626677101 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:59.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.271+0000 7f5012ffd700 1 -- 172.21.15.189:0/3626677101 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50140cfa00 msgr2=0x7f50140c7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:59.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.271+0000 7f5012ffd700 1 --2- 172.21.15.189:0/3626677101 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50140cfa00 0x7f50140c7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:59.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.271+0000 7f5012ffd700 1 -- 172.21.15.189:0/3626677101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50140d11b0 msgr2=0x7f50140ca060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:59.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.271+0000 7f5012ffd700 1 --2- 172.21.15.189:0/3626677101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50140d11b0 0x7f50140ca060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:59.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.271+0000 7f5012ffd700 1 -- 172.21.15.189:0/3626677101 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50080094a0 con 0x7f50140cb870 2024-04-06T13:28:59.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.271+0000 7f50137fe700 1 --2- 172.21.15.189:0/3626677101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50140d11b0 0x7f50140ca060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:59.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.271+0000 7f5012ffd700 1 --2- 172.21.15.189:0/3626677101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50140cb870 0x7f5014154bd0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f500400e870 tx=0x7f500400eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:59.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.271+0000 7f4ffbfff700 1 -- 172.21.15.189:0/3626677101 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5004009590 con 0x7f50140cb870 2024-04-06T13:28:59.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.271+0000 7f4ffbfff700 1 -- 172.21.15.189:0/3626677101 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5004013680 con 0x7f50140cb870 2024-04-06T13:28:59.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.271+0000 7f4ffbfff700 1 -- 172.21.15.189:0/3626677101 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50040139f0 con 0x7f50140cb870 2024-04-06T13:28:59.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.271+0000 7f50199a9700 1 -- 172.21.15.189:0/3626677101 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50140b5e30 con 0x7f50140cb870 2024-04-06T13:28:59.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.271+0000 7f50199a9700 1 -- 172.21.15.189:0/3626677101 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50140b63f0 con 0x7f50140cb870 2024-04-06T13:28:59.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.272+0000 7f50199a9700 1 -- 172.21.15.189:0/3626677101 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5014074990 con 0x7f50140cb870 2024-04-06T13:28:59.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.273+0000 7f4ffbfff700 1 -- 172.21.15.189:0/3626677101 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 104) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f5004013b50 con 0x7f50140cb870 2024-04-06T13:28:59.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.273+0000 7f4ffbfff700 1 --2- 172.21.15.189:0/3626677101 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f500006adb0 0x7f500006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:59.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.273+0000 7f50127fc700 1 -- 172.21.15.189:0/3626677101 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f500006adb0 msgr2=0x7f500006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:59.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.273+0000 7f4ffbfff700 1 -- 172.21.15.189:0/3626677101 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f50040895a0 con 0x7f50140cb870 2024-04-06T13:28:59.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.273+0000 7f50127fc700 1 --2- 172.21.15.189:0/3626677101 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f500006adb0 0x7f500006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:59.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.277+0000 7f4ffbfff700 1 -- 172.21.15.189:0/3626677101 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5004054430 con 0x7f50140cb870 2024-04-06T13:28:59.296 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:59.295+0000 7ff5abd26200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:28:59.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.474+0000 7f50127fc700 1 -- 172.21.15.189:0/3626677101 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f500006adb0 msgr2=0x7f500006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:59.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.474+0000 7f50127fc700 1 --2- 172.21.15.189:0/3626677101 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f500006adb0 0x7f500006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:28:59.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.498+0000 7f50199a9700 1 -- 172.21.15.189:0/3626677101 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f50140cb630 con 0x7f50140cb870 2024-04-06T13:28:59.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.498+0000 7f4ffbfff700 1 -- 172.21.15.189:0/3626677101 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v179) v1 ==== 80+0+468 (secure 0 0 0) 0x7f5004057a50 con 0x7f50140cb870 2024-04-06T13:28:59.499 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2773566419 2024-04-07T13:28:54.359501+0000 2024-04-06T13:28:59.499 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2407993541 2024-04-07T13:28:54.359501+0000 2024-04-06T13:28:59.499 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2622703801 2024-04-07T13:28:54.359501+0000 2024-04-06T13:28:59.499 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1999978665 2024-04-07T13:28:54.359501+0000 2024-04-06T13:28:59.499 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1375122784 2024-04-07T13:28:54.359501+0000 2024-04-06T13:28:59.499 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1621522008 2024-04-07T13:28:54.359501+0000 2024-04-06T13:28:59.499 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6833/47128 2024-04-07T13:28:54.359501+0000 2024-04-06T13:28:59.500 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6832/47128 2024-04-07T13:28:54.359501+0000 2024-04-06T13:28:59.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.500+0000 7f50199a9700 1 -- 172.21.15.189:0/3626677101 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f500006adb0 msgr2=0x7f500006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:28:59.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.500+0000 7f50199a9700 1 --2- 172.21.15.189:0/3626677101 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f500006adb0 0x7f500006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:59.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.500+0000 7f50199a9700 1 -- 172.21.15.189:0/3626677101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50140cb870 msgr2=0x7f5014154bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:59.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.500+0000 7f50199a9700 1 --2- 172.21.15.189:0/3626677101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50140cb870 0x7f5014154bd0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f500400e870 tx=0x7f500400eb80 comp rx=0 tx=0).stop 2024-04-06T13:28:59.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.501+0000 7f50199a9700 1 -- 172.21.15.189:0/3626677101 shutdown_connections 2024-04-06T13:28:59.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.501+0000 7f50199a9700 1 --2- 172.21.15.189:0/3626677101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50140d11b0 0x7f50140ca060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:59.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.501+0000 7f50199a9700 1 --2- 172.21.15.189:0/3626677101 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f500006adb0 0x7f500006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:59.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.501+0000 7f50199a9700 1 --2- 172.21.15.189:0/3626677101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50140cb870 0x7f5014154bd0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:59.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.501+0000 7f50199a9700 1 --2- 172.21.15.189:0/3626677101 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50140cfa00 0x7f50140c7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:59.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.501+0000 7f50199a9700 1 -- 172.21.15.189:0/3626677101 >> 172.21.15.189:0/3626677101 conn(0x7f50140b2db0 msgr2=0x7f50140cd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:59.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.501+0000 7f50199a9700 1 -- 172.21.15.189:0/3626677101 shutdown_connections 2024-04-06T13:28:59.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.501+0000 7f50199a9700 1 -- 172.21.15.189:0/3626677101 wait complete. 2024-04-06T13:28:59.503 INFO:teuthology.orchestra.run.smithi189.stderr:listed 8 entries 2024-04-06T13:28:59.503 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:59.503+0000 7f27f75d3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:28:59.514 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-06T13:28:59.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:59.515 DEBUG:teuthology.orchestra.run.smithi189:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-06T13:28:59.557 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T13:28:59.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:28:59.558 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-06T13:28:59.616 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:28:59.616 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:28:59.633 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:28:59.633 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:28:59.633 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:28:59.638 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:59.637+0000 7ff5abd26200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:28:59.655 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:28:59.655 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:28:59.655 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:28:59.664 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:59.665+0000 7f27f75d3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:28:59.681 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:28:59.681 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:28:59.739 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:28:59.739+0000 7f27f75d3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:28:59.808 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:59.807+0000 7ff5abd26200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:28:59.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.820+0000 7f6fea86b700 1 Processor -- start 2024-04-06T13:28:59.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.821+0000 7f6fea86b700 1 -- start start 2024-04-06T13:28:59.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.821+0000 7f6fea86b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fe406ae40 0x7f6fe40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:59.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.821+0000 7f6fea86b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe40b3990 con 0x7f6fe415ddf0 2024-04-06T13:28:59.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.821+0000 7f6fea86b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe40b3060 con 0x7f6fe415d7b0 2024-04-06T13:28:59.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.821+0000 7f6fea86b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe40b3340 con 0x7f6fe406ae40 2024-04-06T13:28:59.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.822+0000 7f6fe8e08700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6fe415ddf0 0x7f6fe415e2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34160/0 (socket says 172.21.15.189:34160) 2024-04-06T13:28:59.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.822+0000 7f6fe8e08700 1 -- 172.21.15.189:0/4156190928 learned_addr learned my addr 172.21.15.189:0/4156190928 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:28:59.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.822+0000 7f6fe37fe700 1 --2- 172.21.15.189:0/4156190928 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fe406ae40 0x7f6fe40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:59.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.822+0000 7f6fe2ffd700 1 -- 172.21.15.189:0/4156190928 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488486705 0 0) 0x7f6fe40b3990 con 0x7f6fe415ddf0 2024-04-06T13:28:59.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.822+0000 7f6fe2ffd700 1 -- 172.21.15.189:0/4156190928 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fcc0035b0 con 0x7f6fe415ddf0 2024-04-06T13:28:59.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.822+0000 7f6fe2ffd700 1 -- 172.21.15.189:0/4156190928 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528577004 0 0) 0x7f6fe40b3060 con 0x7f6fe415d7b0 2024-04-06T13:28:59.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.822+0000 7f6fe2ffd700 1 -- 172.21.15.189:0/4156190928 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe40b3990 con 0x7f6fe415d7b0 2024-04-06T13:28:59.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.822+0000 7f6fe37fe700 1 -- 172.21.15.189:0/4156190928 >> v1:172.21.15.192:6789/0 conn(0x7f6fe415d7b0 legacy=0x7f6fe406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:59.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.822+0000 7f6fe37fe700 1 -- 172.21.15.189:0/4156190928 >> v1:172.21.15.189:6789/0 conn(0x7f6fe415ddf0 legacy=0x7f6fe415e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:59.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.822+0000 7f6fe37fe700 1 -- 172.21.15.189:0/4156190928 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe40b3620 con 0x7f6fe406ae40 2024-04-06T13:28:59.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.822+0000 7f6fe37fe700 1 --2- 172.21.15.189:0/4156190928 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fe406ae40 0x7f6fe40b3e20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8009870 tx=0x7f6fd8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49c317e427a0ecd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:59.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.822+0000 7f6fe2ffd700 1 -- 172.21.15.189:0/4156190928 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8011830 con 0x7f6fe406ae40 2024-04-06T13:28:59.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.822+0000 7f6fe2ffd700 1 -- 172.21.15.189:0/4156190928 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd8011990 con 0x7f6fe406ae40 2024-04-06T13:28:59.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.822+0000 7f6fe2ffd700 1 -- 172.21.15.189:0/4156190928 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8011d20 con 0x7f6fe406ae40 2024-04-06T13:28:59.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.823+0000 7f6fea86b700 1 -- 172.21.15.189:0/4156190928 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fe406ae40 msgr2=0x7f6fe40b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:59.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.823+0000 7f6fea86b700 1 --2- 172.21.15.189:0/4156190928 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fe406ae40 0x7f6fe40b3e20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8009870 tx=0x7f6fd8009b80 comp rx=0 tx=0).stop 2024-04-06T13:28:59.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.823+0000 7f6fea86b700 1 -- 172.21.15.189:0/4156190928 shutdown_connections 2024-04-06T13:28:59.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.823+0000 7f6fea86b700 1 --2- 172.21.15.189:0/4156190928 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fe406ae40 0x7f6fe40b3e20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:59.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.823+0000 7f6fea86b700 1 -- 172.21.15.189:0/4156190928 >> 172.21.15.189:0/4156190928 conn(0x7f6fe40d00b0 msgr2=0x7f6fe40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:28:59.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.823+0000 7f6fea86b700 1 -- 172.21.15.189:0/4156190928 shutdown_connections 2024-04-06T13:28:59.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.823+0000 7f6fea86b700 1 -- 172.21.15.189:0/4156190928 wait complete. 2024-04-06T13:28:59.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.824+0000 7f6fea86b700 1 Processor -- start 2024-04-06T13:28:59.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.825+0000 7f6fea86b700 1 -- start start 2024-04-06T13:28:59.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.826+0000 7f6fea86b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fe406ae40 0x7f6fe41574d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:59.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.826+0000 7f6fea86b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fe415d7b0 0x7f6fe4150790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:59.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.826+0000 7f6fea86b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fe415ddf0 0x7f6fe41f5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:59.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.826+0000 7f6fea86b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe4157a10 con 0x7f6fe415ddf0 2024-04-06T13:28:59.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.826+0000 7f6fea86b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe4155490 con 0x7f6fe415d7b0 2024-04-06T13:28:59.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.826+0000 7f6fea86b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe4154b60 con 0x7f6fe406ae40 2024-04-06T13:28:59.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.826+0000 7f6fe3fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fe406ae40 0x7f6fe41574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:59.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.826+0000 7f6fe37fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fe415d7b0 0x7f6fe4150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:59.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.826+0000 7f6fe8e08700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fe415ddf0 0x7f6fe41f5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:28:59.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.826+0000 7f6fe3fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fe406ae40 0x7f6fe41574d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:57004/0 (socket says 172.21.15.189:57004) 2024-04-06T13:28:59.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.826+0000 7f6fe3fff700 1 -- 172.21.15.189:0/1582718998 learned_addr learned my addr 172.21.15.189:0/1582718998 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:28:59.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.826+0000 7f6fe3fff700 1 -- 172.21.15.189:0/1582718998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fe415d7b0 msgr2=0x7f6fe4150790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:59.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.826+0000 7f6fe3fff700 1 --2- 172.21.15.189:0/1582718998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fe415d7b0 0x7f6fe4150790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:59.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.826+0000 7f6fe3fff700 1 -- 172.21.15.189:0/1582718998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fe415ddf0 msgr2=0x7f6fe41f5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:28:59.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.827+0000 7f6fe3fff700 1 --2- 172.21.15.189:0/1582718998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fe415ddf0 0x7f6fe41f5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:28:59.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.827+0000 7f6fe3fff700 1 -- 172.21.15.189:0/1582718998 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd80094a0 con 0x7f6fe406ae40 2024-04-06T13:28:59.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.827+0000 7f6fe8e08700 1 --2- 172.21.15.189:0/1582718998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fe415ddf0 0x7f6fe41f5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:28:59.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.827+0000 7f6fe3fff700 1 --2- 172.21.15.189:0/1582718998 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fe406ae40 0x7f6fe41574d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400c550 tx=0x7f6fd400c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:28:59.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.827+0000 7f6fe0ff9700 1 -- 172.21.15.189:0/1582718998 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd40052d0 con 0x7f6fe406ae40 2024-04-06T13:28:59.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.827+0000 7f6fe0ff9700 1 -- 172.21.15.189:0/1582718998 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd4005430 con 0x7f6fe406ae40 2024-04-06T13:28:59.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.827+0000 7f6fea86b700 1 -- 172.21.15.189:0/1582718998 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe4154e40 con 0x7f6fe406ae40 2024-04-06T13:28:59.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.827+0000 7f6fe0ff9700 1 -- 172.21.15.189:0/1582718998 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd40038c0 con 0x7f6fe406ae40 2024-04-06T13:28:59.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.827+0000 7f6fea86b700 1 -- 172.21.15.189:0/1582718998 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe4157c00 con 0x7f6fe406ae40 2024-04-06T13:28:59.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.828+0000 7f6fea86b700 1 -- 172.21.15.189:0/1582718998 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe4067bb0 con 0x7f6fe406ae40 2024-04-06T13:28:59.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.828+0000 7f6fe0ff9700 1 -- 172.21.15.189:0/1582718998 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 104) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f6fd4003a20 con 0x7f6fe406ae40 2024-04-06T13:28:59.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.829+0000 7f6fe0ff9700 1 --2- 172.21.15.189:0/1582718998 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6fd006adb0 0x7f6fd006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:28:59.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.829+0000 7f6fe37fe700 1 -- 172.21.15.189:0/1582718998 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6fd006adb0 msgr2=0x7f6fd006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:28:59.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.829+0000 7f6fe0ff9700 1 -- 172.21.15.189:0/1582718998 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f6fd4088bf0 con 0x7f6fe406ae40 2024-04-06T13:28:59.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.829+0000 7f6fe37fe700 1 --2- 172.21.15.189:0/1582718998 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6fd006adb0 0x7f6fd006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:28:59.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:28:59.834+0000 7f6fe0ff9700 1 -- 172.21.15.189:0/1582718998 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6fd4053960 con 0x7f6fe406ae40 2024-04-06T13:28:59.889 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:28:59.888+0000 7ff5abd26200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:29:00.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.030+0000 7f6fe37fe700 1 -- 172.21.15.189:0/1582718998 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6fd006adb0 msgr2=0x7f6fd006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:00.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.030+0000 7f6fe37fe700 1 --2- 172.21.15.189:0/1582718998 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6fd006adb0 0x7f6fd006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:29:00.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.085+0000 7f6fea86b700 1 -- 172.21.15.189:0/1582718998 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fe4074990 con 0x7f6fe406ae40 2024-04-06T13:29:00.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.086+0000 7f6fe0ff9700 1 -- 172.21.15.189:0/1582718998 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4750 (secure 0 0 0) 0x7f6fd4017020 con 0x7f6fe406ae40 2024-04-06T13:29:00.087 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:29:00.087 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":78,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/3388956983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3388956983},{"type":"v1","addr":"172.21.15.192:6834","nonce":3388956983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":8845,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1844918354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":1844918354},{"type":"v1","addr":"172.21.15.192:6835","nonce":1844918354}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":9618,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3742852104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3742852104},{"type":"v1","addr":"172.21.15.189:6836","nonce":3742852104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:28:40.813074+0000","modified":"2024-04-06T13:28:43.827822+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9621":{"gid":9621,"name":"c","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/2089191015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2089191015},{"type":"v1","addr":"172.21.15.189:6837","nonce":2089191015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-06T13:29:00.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.090+0000 7f6fea86b700 1 -- 172.21.15.189:0/1582718998 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6fd006adb0 msgr2=0x7f6fd006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:29:00.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.090+0000 7f6fea86b700 1 --2- 172.21.15.189:0/1582718998 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6fd006adb0 0x7f6fd006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.090+0000 7f6fea86b700 1 -- 172.21.15.189:0/1582718998 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fe406ae40 msgr2=0x7f6fe41574d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:00.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.090+0000 7f6fea86b700 1 --2- 172.21.15.189:0/1582718998 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fe406ae40 0x7f6fe41574d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400c550 tx=0x7f6fd400c860 comp rx=0 tx=0).stop 2024-04-06T13:29:00.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.090+0000 7f6fea86b700 1 -- 172.21.15.189:0/1582718998 shutdown_connections 2024-04-06T13:29:00.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.090+0000 7f6fea86b700 1 --2- 172.21.15.189:0/1582718998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fe415ddf0 0x7f6fe41f5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.090+0000 7f6fea86b700 1 --2- 172.21.15.189:0/1582718998 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fe406ae40 0x7f6fe41574d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.090+0000 7f6fea86b700 1 --2- 172.21.15.189:0/1582718998 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6fd006adb0 0x7f6fd006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.090+0000 7f6fea86b700 1 --2- 172.21.15.189:0/1582718998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fe415d7b0 0x7f6fe4150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.090+0000 7f6fea86b700 1 -- 172.21.15.189:0/1582718998 >> 172.21.15.189:0/1582718998 conn(0x7f6fe40d00b0 msgr2=0x7f6fe40adf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:00.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.090+0000 7f6fea86b700 1 -- 172.21.15.189:0/1582718998 shutdown_connections 2024-04-06T13:29:00.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.090+0000 7f6fea86b700 1 -- 172.21.15.189:0/1582718998 wait complete. 2024-04-06T13:29:00.092 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 78 2024-04-06T13:29:00.108 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:29:00.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.237+0000 7f19f9459700 1 Processor -- start 2024-04-06T13:29:00.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.237+0000 7f19f9459700 1 -- start start 2024-04-06T13:29:00.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.237+0000 7f19f9459700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19f415d7e0 0x7f19f40d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:00.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.237+0000 7f19f9459700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f19f40d0890 0x7f19f40d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:00.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.237+0000 7f19f9459700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f19f40d1af0 0x7f19f40b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:00.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.237+0000 7f19f9459700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f40d0d10 con 0x7f19f40d1af0 2024-04-06T13:29:00.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.237+0000 7f19f9459700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f40d1f80 con 0x7f19f40d0890 2024-04-06T13:29:00.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.237+0000 7f19f9459700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f40cdf80 con 0x7f19f415d7e0 2024-04-06T13:29:00.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.238+0000 7f19f37fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f19f40d1af0 0x7f19f40b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:00.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.238+0000 7f19f37fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f19f40d1af0 0x7f19f40b4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35236/0 (socket says 172.21.15.189:35236) 2024-04-06T13:29:00.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.238+0000 7f19f37fe700 1 -- 172.21.15.189:0/2332968973 learned_addr learned my addr 172.21.15.189:0/2332968973 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:00.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.238+0000 7f19f2ffd700 1 --2- 172.21.15.189:0/2332968973 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19f415d7e0 0x7f19f40d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:00.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.238+0000 7f19f27fc700 1 --2- 172.21.15.189:0/2332968973 >> v2:172.21.15.192:3300/0 conn(0x7f19f40d0890 0x7f19f40d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:00.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.238+0000 7f19f37fe700 1 -- 172.21.15.189:0/2332968973 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19f415d7e0 msgr2=0x7f19f40d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:00.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.238+0000 7f19f37fe700 1 --2- 172.21.15.189:0/2332968973 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19f415d7e0 0x7f19f40d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.238+0000 7f19f37fe700 1 -- 172.21.15.189:0/2332968973 >> v2:172.21.15.192:3300/0 conn(0x7f19f40d0890 msgr2=0x7f19f40d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:00.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.238+0000 7f19f37fe700 1 --2- 172.21.15.189:0/2332968973 >> v2:172.21.15.192:3300/0 conn(0x7f19f40d0890 0x7f19f40d15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.238+0000 7f19f37fe700 1 -- 172.21.15.189:0/2332968973 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f40d3230 con 0x7f19f40d1af0 2024-04-06T13:29:00.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.238+0000 7f19f37fe700 1 --2- 172.21.15.189:0/2332968973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19f40d1af0 0x7f19f40b4e50 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f19f40b5f20 tx=0x7f19ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45f8873a670d206 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:00.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.238+0000 7f19f1ffb700 1 -- 172.21.15.189:0/2332968973 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec00e070 con 0x7f19f40d1af0 2024-04-06T13:29:00.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.238+0000 7f19f1ffb700 1 -- 172.21.15.189:0/2332968973 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ec00b910 con 0x7f19f40d1af0 2024-04-06T13:29:00.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.238+0000 7f19f1ffb700 1 -- 172.21.15.189:0/2332968973 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec00bcc0 con 0x7f19f40d1af0 2024-04-06T13:29:00.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.239+0000 7f19f9459700 1 -- 172.21.15.189:0/2332968973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19f40d1af0 msgr2=0x7f19f40b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:00.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.239+0000 7f19f9459700 1 --2- 172.21.15.189:0/2332968973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19f40d1af0 0x7f19f40b4e50 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f19f40b5f20 tx=0x7f19ec00b330 comp rx=0 tx=0).stop 2024-04-06T13:29:00.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.239+0000 7f19f9459700 1 -- 172.21.15.189:0/2332968973 shutdown_connections 2024-04-06T13:29:00.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.239+0000 7f19f9459700 1 --2- 172.21.15.189:0/2332968973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19f40d1af0 0x7f19f40b4e50 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.239+0000 7f19f9459700 1 --2- 172.21.15.189:0/2332968973 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19f415d7e0 0x7f19f40d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.239+0000 7f19f9459700 1 --2- 172.21.15.189:0/2332968973 >> v2:172.21.15.192:3300/0 conn(0x7f19f40d0890 0x7f19f40d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.239+0000 7f19f9459700 1 -- 172.21.15.189:0/2332968973 >> 172.21.15.189:0/2332968973 conn(0x7f19f4072250 msgr2=0x7f19f4072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:00.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.239+0000 7f19f9459700 1 -- 172.21.15.189:0/2332968973 shutdown_connections 2024-04-06T13:29:00.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.239+0000 7f19f9459700 1 -- 172.21.15.189:0/2332968973 wait complete. 2024-04-06T13:29:00.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.240+0000 7f19f9459700 1 Processor -- start 2024-04-06T13:29:00.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.241+0000 7f19f9459700 1 -- start start 2024-04-06T13:29:00.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.241+0000 7f19f9459700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19f40d0890 0x7f19f414bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:00.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.241+0000 7f19f9459700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19f40d1af0 0x7f19f414c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:00.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.241+0000 7f19f9459700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19f415d7e0 0x7f19f40ac530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:00.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.241+0000 7f19f9459700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f40ae030 con 0x7f19f415d7e0 2024-04-06T13:29:00.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.241+0000 7f19f9459700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f40cbec0 con 0x7f19f40d1af0 2024-04-06T13:29:00.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.241+0000 7f19f9459700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f41f3fc0 con 0x7f19f40d0890 2024-04-06T13:29:00.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.241+0000 7f19f37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19f415d7e0 0x7f19f40ac530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:00.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.241+0000 7f19f37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19f415d7e0 0x7f19f40ac530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35244/0 (socket says 172.21.15.189:35244) 2024-04-06T13:29:00.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.241+0000 7f19f2ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19f40d0890 0x7f19f414bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:00.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.241+0000 7f19f37fe700 1 -- 172.21.15.189:0/2010487646 learned_addr learned my addr 172.21.15.189:0/2010487646 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:00.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.241+0000 7f19f27fc700 1 --2- 172.21.15.189:0/2010487646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19f40d1af0 0x7f19f414c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:00.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.242+0000 7f19f37fe700 1 -- 172.21.15.189:0/2010487646 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19f40d0890 msgr2=0x7f19f414bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:00.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.242+0000 7f19f37fe700 1 --2- 172.21.15.189:0/2010487646 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19f40d0890 0x7f19f414bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.242+0000 7f19f37fe700 1 -- 172.21.15.189:0/2010487646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19f40d1af0 msgr2=0x7f19f414c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:00.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.242+0000 7f19f37fe700 1 --2- 172.21.15.189:0/2010487646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19f40d1af0 0x7f19f414c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.242+0000 7f19f37fe700 1 -- 172.21.15.189:0/2010487646 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec009b30 con 0x7f19f415d7e0 2024-04-06T13:29:00.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.242+0000 7f19f27fc700 1 --2- 172.21.15.189:0/2010487646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19f40d1af0 0x7f19f414c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:29:00.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.242+0000 7f19f2ffd700 1 --2- 172.21.15.189:0/2010487646 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19f40d0890 0x7f19f414bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:29:00.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.242+0000 7f19f37fe700 1 --2- 172.21.15.189:0/2010487646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19f415d7e0 0x7f19f40ac530 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f19f406b8c0 tx=0x7f19ec004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:00.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.242+0000 7f19db7fe700 1 -- 172.21.15.189:0/2010487646 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec0099b0 con 0x7f19f415d7e0 2024-04-06T13:29:00.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.242+0000 7f19db7fe700 1 -- 172.21.15.189:0/2010487646 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ec00ce80 con 0x7f19f415d7e0 2024-04-06T13:29:00.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.242+0000 7f19f9459700 1 -- 172.21.15.189:0/2010487646 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f40cb590 con 0x7f19f415d7e0 2024-04-06T13:29:00.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.242+0000 7f19db7fe700 1 -- 172.21.15.189:0/2010487646 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec004490 con 0x7f19f415d7e0 2024-04-06T13:29:00.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.242+0000 7f19f9459700 1 -- 172.21.15.189:0/2010487646 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f40cbb50 con 0x7f19f415d7e0 2024-04-06T13:29:00.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.243+0000 7f19f9459700 1 -- 172.21.15.189:0/2010487646 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f40ca160 con 0x7f19f415d7e0 2024-04-06T13:29:00.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.244+0000 7f19db7fe700 1 -- 172.21.15.189:0/2010487646 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 104) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f19ec0045f0 con 0x7f19f415d7e0 2024-04-06T13:29:00.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.244+0000 7f19db7fe700 1 --2- 172.21.15.189:0/2010487646 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f19e006adb0 0x7f19e006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:00.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.244+0000 7f19f2ffd700 1 -- 172.21.15.189:0/2010487646 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f19e006adb0 msgr2=0x7f19e006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:00.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.244+0000 7f19db7fe700 1 -- 172.21.15.189:0/2010487646 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f19ec08a860 con 0x7f19f415d7e0 2024-04-06T13:29:00.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.244+0000 7f19f2ffd700 1 --2- 172.21.15.189:0/2010487646 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f19e006adb0 0x7f19e006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:29:00.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.249+0000 7f19db7fe700 1 -- 172.21.15.189:0/2010487646 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f19ec0555e0 con 0x7f19f415d7e0 2024-04-06T13:29:00.359 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:00.359+0000 7f27f75d3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:29:00.438 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:00.438+0000 7f27f75d3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:29:00.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.445+0000 7f19f2ffd700 1 -- 172.21.15.189:0/2010487646 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f19e006adb0 msgr2=0x7f19e006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:00.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.445+0000 7f19f2ffd700 1 --2- 172.21.15.189:0/2010487646 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f19e006adb0 0x7f19e006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:29:00.453 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.452+0000 7f19f9459700 1 -- 172.21.15.189:0/2010487646 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19f40d3230 con 0x7f19f415d7e0 2024-04-06T13:29:00.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.453+0000 7f19db7fe700 1 -- 172.21.15.189:0/2010487646 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+16873 (secure 0 0 0) 0x7f19f40d3230 con 0x7f19f415d7e0 2024-04-06T13:29:00.454 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:29:00.455 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":179,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:28:54.359536+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:28:38.579643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-04-06T13:28:39.779769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/2773566419":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/2407993541":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/2622703801":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/1999978665":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/1375122784":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/1621522008":"2024-04-07T13:28:54.359501+0000","172.21.15.189:6833/47128":"2024-04-07T13:28:54.359501+0000","172.21.15.189:6832/47128":"2024-04-07T13:28:54.359501+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:29:00.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.455+0000 7f19f9459700 1 -- 172.21.15.189:0/2010487646 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f19e006adb0 msgr2=0x7f19e006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:29:00.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.455+0000 7f19f9459700 1 --2- 172.21.15.189:0/2010487646 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f19e006adb0 0x7f19e006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.455+0000 7f19f9459700 1 -- 172.21.15.189:0/2010487646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19f415d7e0 msgr2=0x7f19f40ac530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:00.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.455+0000 7f19f9459700 1 --2- 172.21.15.189:0/2010487646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19f415d7e0 0x7f19f40ac530 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f19f406b8c0 tx=0x7f19ec004ad0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.456+0000 7f19f9459700 1 -- 172.21.15.189:0/2010487646 shutdown_connections 2024-04-06T13:29:00.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.456+0000 7f19f9459700 1 --2- 172.21.15.189:0/2010487646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19f415d7e0 0x7f19f40ac530 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.456+0000 7f19f9459700 1 --2- 172.21.15.189:0/2010487646 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19f40d0890 0x7f19f414bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.456+0000 7f19f9459700 1 --2- 172.21.15.189:0/2010487646 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f19e006adb0 0x7f19e006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.456+0000 7f19f9459700 1 --2- 172.21.15.189:0/2010487646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19f40d1af0 0x7f19f414c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.456+0000 7f19f9459700 1 -- 172.21.15.189:0/2010487646 >> 172.21.15.189:0/2010487646 conn(0x7f19f4072250 msgr2=0x7f19f40c8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:00.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.456+0000 7f19f9459700 1 -- 172.21.15.189:0/2010487646 shutdown_connections 2024-04-06T13:29:00.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.456+0000 7f19f9459700 1 -- 172.21.15.189:0/2010487646 wait complete. 2024-04-06T13:29:00.469 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T13:28:38.579643+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '177', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T13:28:39.779769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '178', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:29:00.470 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:29:00.557 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:00.556+0000 7ff5abd26200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:29:00.562 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:00.562+0000 7f27f75d3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:29:00.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.597+0000 7fc60a01e700 1 Processor -- start 2024-04-06T13:29:00.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.598+0000 7fc60a01e700 1 -- start start 2024-04-06T13:29:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.598+0000 7fc60a01e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc60415a550 0x7fc6040cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.598+0000 7fc60a01e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc6040b4d70 0x7fc6040b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.598+0000 7fc60a01e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6040cdbe0 con 0x7fc6040cf900 2024-04-06T13:29:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.598+0000 7fc60a01e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc60415a940 con 0x7fc6040b4d70 2024-04-06T13:29:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.599+0000 7fc60a01e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc60415ac00 con 0x7fc60415a550 2024-04-06T13:29:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.599+0000 7fc603fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc6040b4d70 0x7fc6040b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.599+0000 7fc5fbfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc6040cf900 0x7fc6040b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34176/0 (socket says 172.21.15.189:34176) 2024-04-06T13:29:00.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.599+0000 7fc5fbfff700 1 -- 172.21.15.189:0/895757278 learned_addr learned my addr 172.21.15.189:0/895757278 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:00.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.599+0000 7fc6037fe700 1 --2- 172.21.15.189:0/895757278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc60415a550 0x7fc6040cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:00.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.599+0000 7fc602ffd700 1 -- 172.21.15.189:0/895757278 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782882459 0 0) 0x7fc6040cdbe0 con 0x7fc6040cf900 2024-04-06T13:29:00.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.599+0000 7fc602ffd700 1 -- 172.21.15.189:0/895757278 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5f00035b0 con 0x7fc6040cf900 2024-04-06T13:29:00.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.599+0000 7fc6037fe700 1 -- 172.21.15.189:0/895757278 >> v2:172.21.15.189:3300/0 conn(0x7fc6040b4d70 msgr2=0x7fc6040b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:00.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.599+0000 7fc6037fe700 1 --2- 172.21.15.189:0/895757278 >> v2:172.21.15.189:3300/0 conn(0x7fc6040b4d70 0x7fc6040b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.599+0000 7fc6037fe700 1 -- 172.21.15.189:0/895757278 >> v1:172.21.15.189:6789/0 conn(0x7fc6040cf900 legacy=0x7fc6040b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:00.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.599+0000 7fc6037fe700 1 -- 172.21.15.189:0/895757278 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60415aee0 con 0x7fc60415a550 2024-04-06T13:29:00.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.599+0000 7fc6037fe700 1 --2- 172.21.15.189:0/895757278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc60415a550 0x7fc6040cf3c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4009840 tx=0x7fc5f4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff573380ee93be57 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:00.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.599+0000 7fc602ffd700 1 -- 172.21.15.189:0/895757278 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4011830 con 0x7fc60415a550 2024-04-06T13:29:00.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.599+0000 7fc602ffd700 1 -- 172.21.15.189:0/895757278 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f4011990 con 0x7fc60415a550 2024-04-06T13:29:00.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.599+0000 7fc602ffd700 1 -- 172.21.15.189:0/895757278 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4011d20 con 0x7fc60415a550 2024-04-06T13:29:00.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.600+0000 7fc60a01e700 1 -- 172.21.15.189:0/895757278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc60415a550 msgr2=0x7fc6040cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:00.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.600+0000 7fc60a01e700 1 --2- 172.21.15.189:0/895757278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc60415a550 0x7fc6040cf3c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4009840 tx=0x7fc5f4009b50 comp rx=0 tx=0).stop 2024-04-06T13:29:00.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.600+0000 7fc60a01e700 1 -- 172.21.15.189:0/895757278 shutdown_connections 2024-04-06T13:29:00.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.600+0000 7fc60a01e700 1 --2- 172.21.15.189:0/895757278 >> v2:172.21.15.189:3300/0 conn(0x7fc6040b4d70 0x7fc6040b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.600+0000 7fc60a01e700 1 --2- 172.21.15.189:0/895757278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc60415a550 0x7fc6040cf3c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.600+0000 7fc60a01e700 1 -- 172.21.15.189:0/895757278 >> 172.21.15.189:0/895757278 conn(0x7fc6040d00b0 msgr2=0x7fc6040d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:00.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.600+0000 7fc60a01e700 1 -- 172.21.15.189:0/895757278 shutdown_connections 2024-04-06T13:29:00.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.600+0000 7fc60a01e700 1 -- 172.21.15.189:0/895757278 wait complete. 2024-04-06T13:29:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.601+0000 7fc60a01e700 1 Processor -- start 2024-04-06T13:29:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.601+0000 7fc60a01e700 1 -- start start 2024-04-06T13:29:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.602+0000 7fc60a01e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6040b4d70 0x7fc6041597b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.602+0000 7fc60a01e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6040cf900 0x7fc6040ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.602+0000 7fc60a01e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6040ace00 0x7fc6040ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.602+0000 7fc60a01e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc604159cf0 con 0x7fc6040b4d70 2024-04-06T13:29:00.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.602+0000 7fc60a01e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6040ad380 con 0x7fc6040ace00 2024-04-06T13:29:00.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.602+0000 7fc60a01e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc60413fcb0 con 0x7fc6040cf900 2024-04-06T13:29:00.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.602+0000 7fc6037fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6040b4d70 0x7fc6041597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:00.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.602+0000 7fc6037fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6040b4d70 0x7fc6041597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35262/0 (socket says 172.21.15.189:35262) 2024-04-06T13:29:00.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.602+0000 7fc6037fe700 1 -- 172.21.15.189:0/2304811397 learned_addr learned my addr 172.21.15.189:0/2304811397 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:00.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.602+0000 7fc5fbfff700 1 --2- 172.21.15.189:0/2304811397 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6040cf900 0x7fc6040ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:00.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.602+0000 7fc603fff700 1 --2- 172.21.15.189:0/2304811397 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6040ace00 0x7fc6040ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:00.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.602+0000 7fc6037fe700 1 -- 172.21.15.189:0/2304811397 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6040cf900 msgr2=0x7fc6040ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:00.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.602+0000 7fc6037fe700 1 --2- 172.21.15.189:0/2304811397 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6040cf900 0x7fc6040ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.602+0000 7fc6037fe700 1 -- 172.21.15.189:0/2304811397 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6040ace00 msgr2=0x7fc6040ae8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:00.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.602+0000 7fc6037fe700 1 --2- 172.21.15.189:0/2304811397 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6040ace00 0x7fc6040ae8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.602+0000 7fc6037fe700 1 -- 172.21.15.189:0/2304811397 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f40094a0 con 0x7fc6040b4d70 2024-04-06T13:29:00.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.603+0000 7fc6037fe700 1 --2- 172.21.15.189:0/2304811397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6040b4d70 0x7fc6041597b0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4014040 tx=0x7fc5f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:00.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.603+0000 7fc600ff9700 1 -- 172.21.15.189:0/2304811397 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4004590 con 0x7fc6040b4d70 2024-04-06T13:29:00.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.603+0000 7fc600ff9700 1 -- 172.21.15.189:0/2304811397 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f401f070 con 0x7fc6040b4d70 2024-04-06T13:29:00.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.603+0000 7fc60a01e700 1 -- 172.21.15.189:0/2304811397 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc604140140 con 0x7fc6040b4d70 2024-04-06T13:29:00.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.603+0000 7fc600ff9700 1 -- 172.21.15.189:0/2304811397 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f40038a0 con 0x7fc6040b4d70 2024-04-06T13:29:00.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.603+0000 7fc60a01e700 1 -- 172.21.15.189:0/2304811397 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc60413f660 con 0x7fc6040b4d70 2024-04-06T13:29:00.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.604+0000 7fc60a01e700 1 -- 172.21.15.189:0/2304811397 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc604067bb0 con 0x7fc6040b4d70 2024-04-06T13:29:00.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.604+0000 7fc600ff9700 1 -- 172.21.15.189:0/2304811397 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 104) v1 ==== 89248+0+0 (secure 0 0 0) 0x7fc5f4003a00 con 0x7fc6040b4d70 2024-04-06T13:29:00.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.605+0000 7fc600ff9700 1 --2- 172.21.15.189:0/2304811397 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc5e406adb0 0x7fc5e406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:00.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.605+0000 7fc5fbfff700 1 -- 172.21.15.189:0/2304811397 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc5e406adb0 msgr2=0x7fc5e406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:00.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.605+0000 7fc600ff9700 1 -- 172.21.15.189:0/2304811397 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fc5f4021070 con 0x7fc6040b4d70 2024-04-06T13:29:00.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.605+0000 7fc5fbfff700 1 --2- 172.21.15.189:0/2304811397 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc5e406adb0 0x7fc5e406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:29:00.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.609+0000 7fc600ff9700 1 -- 172.21.15.189:0/2304811397 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc5f4055c80 con 0x7fc6040b4d70 2024-04-06T13:29:00.642 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:00.641+0000 7ff5abd26200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:29:00.773 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:00.772+0000 7ff5abd26200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:29:00.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.806+0000 7fc5fbfff700 1 -- 172.21.15.189:0/2304811397 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc5e406adb0 msgr2=0x7fc5e406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:00.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.806+0000 7fc5fbfff700 1 --2- 172.21.15.189:0/2304811397 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc5e406adb0 0x7fc5e406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:29:00.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.809+0000 7fc60a01e700 1 -- 172.21.15.189:0/2304811397 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc604074990 con 0x7fc6040b4d70 2024-04-06T13:29:00.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.809+0000 7fc600ff9700 1 -- 172.21.15.189:0/2304811397 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v179) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc5f4003cb0 con 0x7fc6040b4d70 2024-04-06T13:29:00.810 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:29:00.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.811+0000 7fc60a01e700 1 -- 172.21.15.189:0/2304811397 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc5e406adb0 msgr2=0x7fc5e406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:29:00.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.811+0000 7fc60a01e700 1 --2- 172.21.15.189:0/2304811397 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc5e406adb0 0x7fc5e406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.811+0000 7fc60a01e700 1 -- 172.21.15.189:0/2304811397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6040b4d70 msgr2=0x7fc6041597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:00.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.811+0000 7fc60a01e700 1 --2- 172.21.15.189:0/2304811397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6040b4d70 0x7fc6041597b0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4014040 tx=0x7fc5f4004750 comp rx=0 tx=0).stop 2024-04-06T13:29:00.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.811+0000 7fc60a01e700 1 -- 172.21.15.189:0/2304811397 shutdown_connections 2024-04-06T13:29:00.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.811+0000 7fc60a01e700 1 --2- 172.21.15.189:0/2304811397 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc5e406adb0 0x7fc5e406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.811+0000 7fc60a01e700 1 --2- 172.21.15.189:0/2304811397 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6040ace00 0x7fc6040ae8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.811+0000 7fc60a01e700 1 --2- 172.21.15.189:0/2304811397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6040b4d70 0x7fc6041597b0 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.811+0000 7fc60a01e700 1 --2- 172.21.15.189:0/2304811397 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6040cf900 0x7fc6040ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.811+0000 7fc60a01e700 1 -- 172.21.15.189:0/2304811397 >> 172.21.15.189:0/2304811397 conn(0x7fc6040d00b0 msgr2=0x7fc60406ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:00.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.811+0000 7fc60a01e700 1 -- 172.21.15.189:0/2304811397 shutdown_connections 2024-04-06T13:29:00.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.811+0000 7fc60a01e700 1 -- 172.21.15.189:0/2304811397 wait complete. 2024-04-06T13:29:00.825 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T13:29:00.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.956+0000 7f4fb76e1700 1 Processor -- start 2024-04-06T13:29:00.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.957+0000 7f4fb76e1700 1 -- start start 2024-04-06T13:29:00.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.958+0000 7f4fb76e1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4fb00d11b0 0x7f4fb00d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:00.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.958+0000 7f4fb76e1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fb00cb870 0x7f4fb015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:00.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.958+0000 7f4fb76e1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fb00d1ad0 con 0x7f4fb00cfa00 2024-04-06T13:29:00.960 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.958+0000 7f4fb76e1700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb00ccdc0 con 0x7f4fb00d11b0 2024-04-06T13:29:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.958+0000 7f4fb76e1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb00d0b50 con 0x7f4fb00cb870 2024-04-06T13:29:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.958+0000 7f4fb547d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4fb00d11b0 0x7f4fb00d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.958+0000 7f4fb5c7e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4fb00cfa00 0x7f4fb00cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34178/0 (socket says 172.21.15.189:34178) 2024-04-06T13:29:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.958+0000 7f4fb547d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4fb00d11b0 0x7f4fb00d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35276/0 (socket says 172.21.15.189:35276) 2024-04-06T13:29:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.958+0000 7f4fb5c7e700 1 -- 172.21.15.189:0/2762673015 learned_addr learned my addr 172.21.15.189:0/2762673015 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.958+0000 7f4fb4c7c700 1 --2- 172.21.15.189:0/2762673015 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fb00cb870 0x7f4fb015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.958+0000 7f4fb547d700 1 -- 172.21.15.189:0/2762673015 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fb00cb870 msgr2=0x7f4fb015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.958+0000 7f4fb547d700 1 --2- 172.21.15.189:0/2762673015 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fb00cb870 0x7f4fb015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.958+0000 7f4fb547d700 1 -- 172.21.15.189:0/2762673015 >> v1:172.21.15.189:6789/0 conn(0x7f4fb00cfa00 legacy=0x7f4fb00cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.958+0000 7f4fb547d700 1 -- 172.21.15.189:0/2762673015 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb00cbd80 con 0x7f4fb00d11b0 2024-04-06T13:29:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.958+0000 7f4fa7fff700 1 -- 172.21.15.189:0/2762673015 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195782210 0 0) 0x7f4fb00d1ad0 con 0x7f4fb00cfa00 2024-04-06T13:29:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.958+0000 7f4fb547d700 1 --2- 172.21.15.189:0/2762673015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb00d11b0 0x7f4fb00d1590 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f4fac009840 tx=0x7f4fac009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dd196065e5c1836 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.959+0000 7f4fa7fff700 1 -- 172.21.15.189:0/2762673015 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fac011830 con 0x7f4fb00d11b0 2024-04-06T13:29:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.959+0000 7f4fa7fff700 1 -- 172.21.15.189:0/2762673015 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fac011990 con 0x7f4fb00d11b0 2024-04-06T13:29:00.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.959+0000 7f4fa7fff700 1 -- 172.21.15.189:0/2762673015 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fac011d20 con 0x7f4fb00d11b0 2024-04-06T13:29:00.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.959+0000 7f4fb76e1700 1 -- 172.21.15.189:0/2762673015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb00d11b0 msgr2=0x7f4fb00d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:00.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.959+0000 7f4fb76e1700 1 --2- 172.21.15.189:0/2762673015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb00d11b0 0x7f4fb00d1590 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f4fac009840 tx=0x7f4fac009b50 comp rx=0 tx=0).stop 2024-04-06T13:29:00.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.959+0000 7f4fb76e1700 1 -- 172.21.15.189:0/2762673015 shutdown_connections 2024-04-06T13:29:00.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.959+0000 7f4fb76e1700 1 --2- 172.21.15.189:0/2762673015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb00d11b0 0x7f4fb00d1590 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.959+0000 7f4fb76e1700 1 --2- 172.21.15.189:0/2762673015 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fb00cb870 0x7f4fb015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.959+0000 7f4fb76e1700 1 -- 172.21.15.189:0/2762673015 >> 172.21.15.189:0/2762673015 conn(0x7f4fb00b2db0 msgr2=0x7f4fb00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:00.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.959+0000 7f4fb76e1700 1 -- 172.21.15.189:0/2762673015 shutdown_connections 2024-04-06T13:29:00.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.959+0000 7f4fb76e1700 1 -- 172.21.15.189:0/2762673015 wait complete. 2024-04-06T13:29:00.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.961+0000 7f4fb76e1700 1 Processor -- start 2024-04-06T13:29:00.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.961+0000 7f4fb76e1700 1 -- start start 2024-04-06T13:29:00.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.962+0000 7f4fb76e1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fb00cb870 0x7f4fb0154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:00.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.962+0000 7f4fb76e1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fb00cfa00 0x7f4fb00c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:00.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.962+0000 7f4fb76e1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb00d11b0 0x7f4fb01e6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:00.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.962+0000 7f4fb76e1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb0156140 con 0x7f4fb00d11b0 2024-04-06T13:29:00.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.962+0000 7f4fb76e1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb00c90d0 con 0x7f4fb00cb870 2024-04-06T13:29:00.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.962+0000 7f4fb76e1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb006d840 con 0x7f4fb00cfa00 2024-04-06T13:29:00.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.962+0000 7f4fb5c7e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb00d11b0 0x7f4fb01e6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:00.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.962+0000 7f4fb5c7e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb00d11b0 0x7f4fb01e6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35288/0 (socket says 172.21.15.189:35288) 2024-04-06T13:29:00.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.962+0000 7f4fb5c7e700 1 -- 172.21.15.189:0/2233650050 learned_addr learned my addr 172.21.15.189:0/2233650050 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:00.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.962+0000 7f4fb4c7c700 1 --2- 172.21.15.189:0/2233650050 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fb00cfa00 0x7f4fb00c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:00.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.962+0000 7f4fb547d700 1 --2- 172.21.15.189:0/2233650050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fb00cb870 0x7f4fb0154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:00.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.962+0000 7f4fb5c7e700 1 -- 172.21.15.189:0/2233650050 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fb00cfa00 msgr2=0x7f4fb00c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:00.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.962+0000 7f4fb5c7e700 1 --2- 172.21.15.189:0/2233650050 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fb00cfa00 0x7f4fb00c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.962+0000 7f4fb5c7e700 1 -- 172.21.15.189:0/2233650050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fb00cb870 msgr2=0x7f4fb0154c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:00.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.962+0000 7f4fb5c7e700 1 --2- 172.21.15.189:0/2233650050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fb00cb870 0x7f4fb0154c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:00.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.962+0000 7f4fb5c7e700 1 -- 172.21.15.189:0/2233650050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fac0094a0 con 0x7f4fb00d11b0 2024-04-06T13:29:00.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.962+0000 7f4fb5c7e700 1 --2- 172.21.15.189:0/2233650050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb00d11b0 0x7f4fb01e6ee0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f4fa800cca0 tx=0x7f4fa800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:00.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.963+0000 7f4fa5ffb700 1 -- 172.21.15.189:0/2233650050 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa800ab70 con 0x7f4fb00d11b0 2024-04-06T13:29:00.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.963+0000 7f4fa5ffb700 1 -- 172.21.15.189:0/2233650050 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa800acd0 con 0x7f4fb00d11b0 2024-04-06T13:29:00.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.963+0000 7f4fa5ffb700 1 -- 172.21.15.189:0/2233650050 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8007310 con 0x7f4fb00d11b0 2024-04-06T13:29:00.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.963+0000 7f4fb76e1700 1 -- 172.21.15.189:0/2233650050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb006cfa0 con 0x7f4fb00d11b0 2024-04-06T13:29:00.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.963+0000 7f4fb76e1700 1 -- 172.21.15.189:0/2233650050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb006d560 con 0x7f4fb00d11b0 2024-04-06T13:29:00.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.964+0000 7f4fb76e1700 1 -- 172.21.15.189:0/2233650050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb01562b0 con 0x7f4fb00d11b0 2024-04-06T13:29:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.965+0000 7f4fa5ffb700 1 -- 172.21.15.189:0/2233650050 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 104) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f4fa8007470 con 0x7f4fb00d11b0 2024-04-06T13:29:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.965+0000 7f4fa5ffb700 1 --2- 172.21.15.189:0/2233650050 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f9406adb0 0x7f4f9406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.965+0000 7f4fb547d700 1 -- 172.21.15.189:0/2233650050 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f9406adb0 msgr2=0x7f4f9406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.965+0000 7f4fa5ffb700 1 -- 172.21.15.189:0/2233650050 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f4fa8089580 con 0x7f4fb00d11b0 2024-04-06T13:29:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.965+0000 7f4fb547d700 1 --2- 172.21.15.189:0/2233650050 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f9406adb0 0x7f4f9406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:29:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:00.969+0000 7f4fa5ffb700 1 -- 172.21.15.189:0/2233650050 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4fa80544f0 con 0x7f4fb00d11b0 2024-04-06T13:29:01.069 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:01.069+0000 7f27f75d3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:29:01.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.166+0000 7f4fb547d700 1 -- 172.21.15.189:0/2233650050 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f9406adb0 msgr2=0x7f4f9406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:01.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.166+0000 7f4fb547d700 1 --2- 172.21.15.189:0/2233650050 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f9406adb0 0x7f4f9406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:29:01.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.171+0000 7f4fb76e1700 1 -- 172.21.15.189:0/2233650050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4fb0074990 con 0x7f4fb00d11b0 2024-04-06T13:29:01.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.172+0000 7f4fa5ffb700 1 -- 172.21.15.189:0/2233650050 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v179) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4fa8018020 con 0x7f4fb00d11b0 2024-04-06T13:29:01.172 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:29:01.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.174+0000 7f4fb76e1700 1 -- 172.21.15.189:0/2233650050 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f9406adb0 msgr2=0x7f4f9406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:29:01.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.174+0000 7f4fb76e1700 1 --2- 172.21.15.189:0/2233650050 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f9406adb0 0x7f4f9406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.174+0000 7f4fb76e1700 1 -- 172.21.15.189:0/2233650050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb00d11b0 msgr2=0x7f4fb01e6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:01.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.174+0000 7f4fb76e1700 1 --2- 172.21.15.189:0/2233650050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb00d11b0 0x7f4fb01e6ee0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f4fa800cca0 tx=0x7f4fa800a060 comp rx=0 tx=0).stop 2024-04-06T13:29:01.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.174+0000 7f4fb76e1700 1 -- 172.21.15.189:0/2233650050 shutdown_connections 2024-04-06T13:29:01.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.174+0000 7f4fb76e1700 1 --2- 172.21.15.189:0/2233650050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb00d11b0 0x7f4fb01e6ee0 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.174+0000 7f4fb76e1700 1 --2- 172.21.15.189:0/2233650050 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4f9406adb0 0x7f4f9406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.174+0000 7f4fb76e1700 1 --2- 172.21.15.189:0/2233650050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fb00cb870 0x7f4fb0154c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.174+0000 7f4fb76e1700 1 --2- 172.21.15.189:0/2233650050 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4fb00cfa00 0x7f4fb00c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.174+0000 7f4fb76e1700 1 -- 172.21.15.189:0/2233650050 >> 172.21.15.189:0/2233650050 conn(0x7f4fb00b2db0 msgr2=0x7f4fb015a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:01.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.174+0000 7f4fb76e1700 1 -- 172.21.15.189:0/2233650050 shutdown_connections 2024-04-06T13:29:01.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.174+0000 7f4fb76e1700 1 -- 172.21.15.189:0/2233650050 wait complete. 2024-04-06T13:29:01.187 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T13:29:01.221 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:01.221+0000 7f27f75d3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:29:01.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.315+0000 7f0d0ebd6700 1 Processor -- start 2024-04-06T13:29:01.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.315+0000 7f0d0ebd6700 1 -- start start 2024-04-06T13:29:01.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.315+0000 7f0d0ebd6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0d0815d590 0x7f0d080b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:01.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.315+0000 7f0d0ebd6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d080b4ed0 0x7f0d080b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:01.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.315+0000 7f0d0ebd6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d080b5cc0 con 0x7f0d080b6580 2024-04-06T13:29:01.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.315+0000 7f0d0ebd6700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d080b5890 con 0x7f0d0815d590 2024-04-06T13:29:01.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.315+0000 7f0d0ebd6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d080d1500 con 0x7f0d080b4ed0 2024-04-06T13:29:01.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.316+0000 7f0d0c972700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0d0815d590 0x7f0d080b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:01.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.316+0000 7f0d0d173700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0d080b6580 0x7f0d080b6a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34194/0 (socket says 172.21.15.189:34194) 2024-04-06T13:29:01.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.316+0000 7f0d0d173700 1 -- 172.21.15.189:0/2645598889 learned_addr learned my addr 172.21.15.189:0/2645598889 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:01.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.316+0000 7f0d07fff700 1 --2- 172.21.15.189:0/2645598889 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d080b4ed0 0x7f0d080b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:01.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.316+0000 7f0d0c972700 1 -- 172.21.15.189:0/2645598889 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d080b4ed0 msgr2=0x7f0d080b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:01.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.316+0000 7f0d0c972700 1 --2- 172.21.15.189:0/2645598889 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d080b4ed0 0x7f0d080b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.316+0000 7f0d0c972700 1 -- 172.21.15.189:0/2645598889 >> v1:172.21.15.189:6789/0 conn(0x7f0d080b6580 legacy=0x7f0d080b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:01.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.316+0000 7f0d0c972700 1 -- 172.21.15.189:0/2645598889 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d080d17e0 con 0x7f0d0815d590 2024-04-06T13:29:01.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.316+0000 7f0d077fe700 1 -- 172.21.15.189:0/2645598889 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929679219 0 0) 0x7f0d080b5cc0 con 0x7f0d080b6580 2024-04-06T13:29:01.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.316+0000 7f0d0c972700 1 --2- 172.21.15.189:0/2645598889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d0815d590 0x7f0d080b8d70 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc009870 tx=0x7f0cfc009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=305d3b6eb05d6e59 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:01.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.316+0000 7f0d077fe700 1 -- 172.21.15.189:0/2645598889 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc011830 con 0x7f0d0815d590 2024-04-06T13:29:01.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.316+0000 7f0d077fe700 1 -- 172.21.15.189:0/2645598889 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cfc011990 con 0x7f0d0815d590 2024-04-06T13:29:01.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.316+0000 7f0d077fe700 1 -- 172.21.15.189:0/2645598889 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc011d20 con 0x7f0d0815d590 2024-04-06T13:29:01.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.317+0000 7f0d0ebd6700 1 -- 172.21.15.189:0/2645598889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d0815d590 msgr2=0x7f0d080b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:01.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.317+0000 7f0d0ebd6700 1 --2- 172.21.15.189:0/2645598889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d0815d590 0x7f0d080b8d70 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc009870 tx=0x7f0cfc009b80 comp rx=0 tx=0).stop 2024-04-06T13:29:01.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.317+0000 7f0d0ebd6700 1 -- 172.21.15.189:0/2645598889 shutdown_connections 2024-04-06T13:29:01.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.317+0000 7f0d0ebd6700 1 --2- 172.21.15.189:0/2645598889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d0815d590 0x7f0d080b8d70 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.319 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:01.317+0000 7ff5abd26200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:29:01.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.317+0000 7f0d0ebd6700 1 --2- 172.21.15.189:0/2645598889 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d080b4ed0 0x7f0d080b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.317+0000 7f0d0ebd6700 1 -- 172.21.15.189:0/2645598889 >> 172.21.15.189:0/2645598889 conn(0x7f0d0806bbc0 msgr2=0x7f0d0806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:01.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.317+0000 7f0d0ebd6700 1 -- 172.21.15.189:0/2645598889 shutdown_connections 2024-04-06T13:29:01.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.317+0000 7f0d0ebd6700 1 -- 172.21.15.189:0/2645598889 wait complete. 2024-04-06T13:29:01.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.318+0000 7f0d0ebd6700 1 Processor -- start 2024-04-06T13:29:01.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.318+0000 7f0d0ebd6700 1 -- start start 2024-04-06T13:29:01.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.319+0000 7f0d0ebd6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d080b4ed0 0x7f0d081f6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:01.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.319+0000 7f0d0ebd6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d080b6580 0x7f0d081f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:01.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.319+0000 7f0d0ebd6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d0815d590 0x7f0d08146720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:01.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.319+0000 7f0d0ebd6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d081f73f0 con 0x7f0d080b6580 2024-04-06T13:29:01.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.319+0000 7f0d0ebd6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0815dc90 con 0x7f0d0815d590 2024-04-06T13:29:01.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.319+0000 7f0d0ebd6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0815ed90 con 0x7f0d080b4ed0 2024-04-06T13:29:01.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.319+0000 7f0d07fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d080b6580 0x7f0d081f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:01.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.319+0000 7f0d07fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d080b6580 0x7f0d081f38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35300/0 (socket says 172.21.15.189:35300) 2024-04-06T13:29:01.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.319+0000 7f0d07fff700 1 -- 172.21.15.189:0/3409317374 learned_addr learned my addr 172.21.15.189:0/3409317374 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:01.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.319+0000 7f0d0c972700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d080b4ed0 0x7f0d081f6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:01.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.319+0000 7f0d0d173700 1 --2- 172.21.15.189:0/3409317374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d0815d590 0x7f0d08146720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:01.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.320+0000 7f0d07fff700 1 -- 172.21.15.189:0/3409317374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d080b4ed0 msgr2=0x7f0d081f6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:01.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.320+0000 7f0d07fff700 1 --2- 172.21.15.189:0/3409317374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d080b4ed0 0x7f0d081f6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.320+0000 7f0d07fff700 1 -- 172.21.15.189:0/3409317374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d0815d590 msgr2=0x7f0d08146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:01.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.320+0000 7f0d07fff700 1 --2- 172.21.15.189:0/3409317374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d0815d590 0x7f0d08146720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.320+0000 7f0d07fff700 1 -- 172.21.15.189:0/3409317374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cfc0094a0 con 0x7f0d080b6580 2024-04-06T13:29:01.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.320+0000 7f0d0c972700 1 --2- 172.21.15.189:0/3409317374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d080b4ed0 0x7f0d081f6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:29:01.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.320+0000 7f0d07fff700 1 --2- 172.21.15.189:0/3409317374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d080b6580 0x7f0d081f38a0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f0cf400e9b0 tx=0x7f0cf400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:01.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.320+0000 7f0d057fa700 1 -- 172.21.15.189:0/3409317374 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4009710 con 0x7f0d080b6580 2024-04-06T13:29:01.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.320+0000 7f0d057fa700 1 -- 172.21.15.189:0/3409317374 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf400ee50 con 0x7f0d080b6580 2024-04-06T13:29:01.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.320+0000 7f0d0ebd6700 1 -- 172.21.15.189:0/3409317374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0815f070 con 0x7f0d080b6580 2024-04-06T13:29:01.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.320+0000 7f0d057fa700 1 -- 172.21.15.189:0/3409317374 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf400ca50 con 0x7f0d080b6580 2024-04-06T13:29:01.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.320+0000 7f0d0ebd6700 1 -- 172.21.15.189:0/3409317374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d081f3380 con 0x7f0d080b6580 2024-04-06T13:29:01.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.321+0000 7f0d0ebd6700 1 -- 172.21.15.189:0/3409317374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d081f36f0 con 0x7f0d080b6580 2024-04-06T13:29:01.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.321+0000 7f0d057fa700 1 -- 172.21.15.189:0/3409317374 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 104) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f0cf400cbb0 con 0x7f0d080b6580 2024-04-06T13:29:01.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.322+0000 7f0d057fa700 1 --2- 172.21.15.189:0/3409317374 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0cec06adb0 0x7f0cec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:01.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.322+0000 7f0d0c972700 1 -- 172.21.15.189:0/3409317374 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0cec06adb0 msgr2=0x7f0cec06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:01.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.322+0000 7f0d057fa700 1 -- 172.21.15.189:0/3409317374 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f0cf4089670 con 0x7f0d080b6580 2024-04-06T13:29:01.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.322+0000 7f0d0c972700 1 --2- 172.21.15.189:0/3409317374 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0cec06adb0 0x7f0cec06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:29:01.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.326+0000 7f0d057fa700 1 -- 172.21.15.189:0/3409317374 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0cf40544e0 con 0x7f0d080b6580 2024-04-06T13:29:01.377 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:01.377+0000 7f27f75d3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:29:01.483 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:01.482+0000 7ff5abd26200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:29:01.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.523+0000 7f0d0c972700 1 -- 172.21.15.189:0/3409317374 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0cec06adb0 msgr2=0x7f0cec06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:01.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.523+0000 7f0d0c972700 1 --2- 172.21.15.189:0/3409317374 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0cec06adb0 0x7f0cec06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:29:01.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.529+0000 7f0d0ebd6700 1 -- 172.21.15.189:0/3409317374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0d08074990 con 0x7f0d080b6580 2024-04-06T13:29:01.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.530+0000 7f0d057fa700 1 -- 172.21.15.189:0/3409317374 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v179) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0cf400ce60 con 0x7f0d080b6580 2024-04-06T13:29:01.531 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:29:01.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.532+0000 7f0d0ebd6700 1 -- 172.21.15.189:0/3409317374 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0cec06adb0 msgr2=0x7f0cec06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:29:01.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.532+0000 7f0d0ebd6700 1 --2- 172.21.15.189:0/3409317374 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0cec06adb0 0x7f0cec06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.532+0000 7f0d0ebd6700 1 -- 172.21.15.189:0/3409317374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d080b6580 msgr2=0x7f0d081f38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:01.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.532+0000 7f0d0ebd6700 1 --2- 172.21.15.189:0/3409317374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d080b6580 0x7f0d081f38a0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f0cf400e9b0 tx=0x7f0cf400ecc0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.532+0000 7f0d0ebd6700 1 -- 172.21.15.189:0/3409317374 shutdown_connections 2024-04-06T13:29:01.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.532+0000 7f0d0ebd6700 1 --2- 172.21.15.189:0/3409317374 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0cec06adb0 0x7f0cec06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.532+0000 7f0d0ebd6700 1 --2- 172.21.15.189:0/3409317374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d0815d590 0x7f0d08146720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.532+0000 7f0d0ebd6700 1 --2- 172.21.15.189:0/3409317374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d080b4ed0 0x7f0d081f6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.532+0000 7f0d0ebd6700 1 --2- 172.21.15.189:0/3409317374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d080b6580 0x7f0d081f38a0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.532+0000 7f0d0ebd6700 1 -- 172.21.15.189:0/3409317374 >> 172.21.15.189:0/3409317374 conn(0x7f0d0806bbc0 msgr2=0x7f0d080b7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:01.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.532+0000 7f0d0ebd6700 1 -- 172.21.15.189:0/3409317374 shutdown_connections 2024-04-06T13:29:01.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.532+0000 7f0d0ebd6700 1 -- 172.21.15.189:0/3409317374 wait complete. 2024-04-06T13:29:01.546 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:29:01.652 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:01.651+0000 7ff5abd26200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:29:01.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.670+0000 7f926ddb3700 1 Processor -- start 2024-04-06T13:29:01.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.670+0000 7f926ddb3700 1 -- start start 2024-04-06T13:29:01.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.671+0000 7f926ddb3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f92680b4ed0 0x7f92680b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:01.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.671+0000 7f926ddb3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92680b5cc0 con 0x7f92680d2880 2024-04-06T13:29:01.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.671+0000 7f926ddb3700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92680d1500 con 0x7f926815d590 2024-04-06T13:29:01.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.671+0000 7f926ddb3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92680b58c0 con 0x7f92680b4ed0 2024-04-06T13:29:01.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.671+0000 7f9267fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f92680d2880 0x7f92680d2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34206/0 (socket says 172.21.15.189:34206) 2024-04-06T13:29:01.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.671+0000 7f9267fff700 1 -- 172.21.15.189:0/2194393819 learned_addr learned my addr 172.21.15.189:0/2194393819 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:01.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.671+0000 7f9266ffd700 1 --2- 172.21.15.189:0/2194393819 >> v2:172.21.15.192:3300/0 conn(0x7f92680b4ed0 0x7f92680b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:01.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.671+0000 7f92667fc700 1 -- 172.21.15.189:0/2194393819 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768612458 0 0) 0x7f92680b5cc0 con 0x7f92680d2880 2024-04-06T13:29:01.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.671+0000 7f92667fc700 1 -- 172.21.15.189:0/2194393819 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92540035b0 con 0x7f92680d2880 2024-04-06T13:29:01.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.671+0000 7f9266ffd700 1 -- 172.21.15.189:0/2194393819 >> v1:172.21.15.192:6789/0 conn(0x7f926815d590 legacy=0x7f92680b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:01.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.672+0000 7f9266ffd700 1 -- 172.21.15.189:0/2194393819 >> v1:172.21.15.189:6789/0 conn(0x7f92680d2880 legacy=0x7f92680d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:01.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.672+0000 7f9266ffd700 1 -- 172.21.15.189:0/2194393819 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92680d17e0 con 0x7f92680b4ed0 2024-04-06T13:29:01.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.672+0000 7f9266ffd700 1 --2- 172.21.15.189:0/2194393819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92680b4ed0 0x7f92680b5380 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9250009840 tx=0x7f9250009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=567041f0ee32c1ed server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:01.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.672+0000 7f92667fc700 1 -- 172.21.15.189:0/2194393819 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9250011830 con 0x7f92680b4ed0 2024-04-06T13:29:01.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.672+0000 7f92667fc700 1 -- 172.21.15.189:0/2194393819 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9250011990 con 0x7f92680b4ed0 2024-04-06T13:29:01.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.672+0000 7f92667fc700 1 -- 172.21.15.189:0/2194393819 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9250011d00 con 0x7f92680b4ed0 2024-04-06T13:29:01.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.672+0000 7f926ddb3700 1 -- 172.21.15.189:0/2194393819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92680b4ed0 msgr2=0x7f92680b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:01.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.672+0000 7f926ddb3700 1 --2- 172.21.15.189:0/2194393819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92680b4ed0 0x7f92680b5380 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9250009840 tx=0x7f9250009b50 comp rx=0 tx=0).stop 2024-04-06T13:29:01.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.672+0000 7f926ddb3700 1 -- 172.21.15.189:0/2194393819 shutdown_connections 2024-04-06T13:29:01.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.673+0000 7f926ddb3700 1 --2- 172.21.15.189:0/2194393819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92680b4ed0 0x7f92680b5380 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.673+0000 7f926ddb3700 1 -- 172.21.15.189:0/2194393819 >> 172.21.15.189:0/2194393819 conn(0x7f926806bbc0 msgr2=0x7f926806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:01.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.673+0000 7f926ddb3700 1 -- 172.21.15.189:0/2194393819 shutdown_connections 2024-04-06T13:29:01.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.673+0000 7f926ddb3700 1 -- 172.21.15.189:0/2194393819 wait complete. 2024-04-06T13:29:01.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.674+0000 7f926ddb3700 1 Processor -- start 2024-04-06T13:29:01.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.675+0000 7f926ddb3700 1 -- start start 2024-04-06T13:29:01.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.676+0000 7f926ddb3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92680b4ed0 0x7f9268144750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:01.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.676+0000 7f926ddb3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92680d2880 0x7f926814eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:01.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.676+0000 7f926ddb3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f926815d590 0x7f92680cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:01.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.676+0000 7f926ddb3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9268144c90 con 0x7f92680d2880 2024-04-06T13:29:01.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.676+0000 7f926ddb3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92681f3d60 con 0x7f926815d590 2024-04-06T13:29:01.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.676+0000 7f926ddb3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92681f3fc0 con 0x7f92680b4ed0 2024-04-06T13:29:01.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.676+0000 7f9266ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92680d2880 0x7f926814eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:01.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.676+0000 7f9266ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92680d2880 0x7f926814eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35316/0 (socket says 172.21.15.189:35316) 2024-04-06T13:29:01.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.676+0000 7f9266ffd700 1 -- 172.21.15.189:0/2827432553 learned_addr learned my addr 172.21.15.189:0/2827432553 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:01.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.676+0000 7f9267fff700 1 --2- 172.21.15.189:0/2827432553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f926815d590 0x7f92680cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:01.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.676+0000 7f92677fe700 1 --2- 172.21.15.189:0/2827432553 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92680b4ed0 0x7f9268144750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:01.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.676+0000 7f9266ffd700 1 -- 172.21.15.189:0/2827432553 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92680b4ed0 msgr2=0x7f9268144750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:01.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.676+0000 7f9266ffd700 1 --2- 172.21.15.189:0/2827432553 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92680b4ed0 0x7f9268144750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.676+0000 7f9266ffd700 1 -- 172.21.15.189:0/2827432553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f926815d590 msgr2=0x7f92680cbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:01.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.676+0000 7f9266ffd700 1 --2- 172.21.15.189:0/2827432553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f926815d590 0x7f92680cbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.676+0000 7f9266ffd700 1 -- 172.21.15.189:0/2827432553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92500094a0 con 0x7f92680d2880 2024-04-06T13:29:01.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.676+0000 7f9267fff700 1 --2- 172.21.15.189:0/2827432553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f926815d590 0x7f92680cbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:29:01.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.676+0000 7f9266ffd700 1 --2- 172.21.15.189:0/2827432553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92680d2880 0x7f926814eed0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f9250009fd0 tx=0x7f9250004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:01.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.677+0000 7f925ffff700 1 -- 172.21.15.189:0/2827432553 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9250004550 con 0x7f92680d2880 2024-04-06T13:29:01.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.677+0000 7f925ffff700 1 -- 172.21.15.189:0/2827432553 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925001f070 con 0x7f92680d2880 2024-04-06T13:29:01.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.677+0000 7f925ffff700 1 -- 172.21.15.189:0/2827432553 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92500038c0 con 0x7f92680d2880 2024-04-06T13:29:01.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.677+0000 7f926ddb3700 1 -- 172.21.15.189:0/2827432553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92681f42a0 con 0x7f92680d2880 2024-04-06T13:29:01.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.677+0000 7f926ddb3700 1 -- 172.21.15.189:0/2827432553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92681f4860 con 0x7f92680d2880 2024-04-06T13:29:01.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.678+0000 7f926ddb3700 1 -- 172.21.15.189:0/2827432553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9268074990 con 0x7f92680d2880 2024-04-06T13:29:01.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.678+0000 7f925ffff700 1 -- 172.21.15.189:0/2827432553 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 104) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f9250003a20 con 0x7f92680d2880 2024-04-06T13:29:01.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.679+0000 7f925ffff700 1 --2- 172.21.15.189:0/2827432553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f924806adb0 0x7f924806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:01.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.679+0000 7f92677fe700 1 -- 172.21.15.189:0/2827432553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f924806adb0 msgr2=0x7f924806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:01.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.679+0000 7f925ffff700 1 -- 172.21.15.189:0/2827432553 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f9250021070 con 0x7f92680d2880 2024-04-06T13:29:01.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.679+0000 7f92677fe700 1 --2- 172.21.15.189:0/2827432553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f924806adb0 0x7f924806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:29:01.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.686+0000 7f925ffff700 1 -- 172.21.15.189:0/2827432553 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9250056310 con 0x7f92680d2880 2024-04-06T13:29:01.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.880+0000 7f92677fe700 1 -- 172.21.15.189:0/2827432553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f924806adb0 msgr2=0x7f924806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:01.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.880+0000 7f92677fe700 1 --2- 172.21.15.189:0/2827432553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f924806adb0 0x7f924806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:29:01.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.937+0000 7f926ddb3700 1 -- 172.21.15.189:0/2827432553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92680d39e0 con 0x7f92680d2880 2024-04-06T13:29:01.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.937+0000 7f925ffff700 1 -- 172.21.15.189:0/2827432553 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4750 (secure 0 0 0) 0x7f9250059930 con 0x7f92680d2880 2024-04-06T13:29:01.938 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:29:01.938 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":78,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/3388956983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3388956983},{"type":"v1","addr":"172.21.15.192:6834","nonce":3388956983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":8845,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1844918354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":1844918354},{"type":"v1","addr":"172.21.15.192:6835","nonce":1844918354}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":9618,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3742852104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3742852104},{"type":"v1","addr":"172.21.15.189:6836","nonce":3742852104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:28:40.813074+0000","modified":"2024-04-06T13:28:43.827822+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9621":{"gid":9621,"name":"c","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/2089191015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2089191015},{"type":"v1","addr":"172.21.15.189:6837","nonce":2089191015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-06T13:29:01.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.940+0000 7f926ddb3700 1 -- 172.21.15.189:0/2827432553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f924806adb0 msgr2=0x7f924806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:29:01.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.940+0000 7f926ddb3700 1 --2- 172.21.15.189:0/2827432553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f924806adb0 0x7f924806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.940+0000 7f926ddb3700 1 -- 172.21.15.189:0/2827432553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92680d2880 msgr2=0x7f926814eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:01.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.940+0000 7f926ddb3700 1 --2- 172.21.15.189:0/2827432553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92680d2880 0x7f926814eed0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f9250009fd0 tx=0x7f9250004750 comp rx=0 tx=0).stop 2024-04-06T13:29:01.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.940+0000 7f926ddb3700 1 -- 172.21.15.189:0/2827432553 shutdown_connections 2024-04-06T13:29:01.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.940+0000 7f926ddb3700 1 --2- 172.21.15.189:0/2827432553 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f924806adb0 0x7f924806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.940+0000 7f926ddb3700 1 --2- 172.21.15.189:0/2827432553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f926815d590 0x7f92680cbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.940+0000 7f926ddb3700 1 --2- 172.21.15.189:0/2827432553 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92680b4ed0 0x7f9268144750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.940+0000 7f926ddb3700 1 --2- 172.21.15.189:0/2827432553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92680d2880 0x7f926814eed0 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:01.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.940+0000 7f926ddb3700 1 -- 172.21.15.189:0/2827432553 >> 172.21.15.189:0/2827432553 conn(0x7f926806bbc0 msgr2=0x7f92680ce070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:01.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.940+0000 7f926ddb3700 1 -- 172.21.15.189:0/2827432553 shutdown_connections 2024-04-06T13:29:01.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:01.940+0000 7f926ddb3700 1 -- 172.21.15.189:0/2827432553 wait complete. 2024-04-06T13:29:01.942 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 78 2024-04-06T13:29:01.953 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:29:02.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.076+0000 7fa2cbb30700 1 Processor -- start 2024-04-06T13:29:02.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.077+0000 7fa2cbb30700 1 -- start start 2024-04-06T13:29:02.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.077+0000 7fa2cbb30700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa2c40d11b0 0x7fa2c40d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:02.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.077+0000 7fa2cbb30700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa2c40cb870 0x7fa2c415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:02.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.078+0000 7fa2cbb30700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c40d1ad0 con 0x7fa2c40cfa00 2024-04-06T13:29:02.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.078+0000 7fa2cbb30700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2c40ccdc0 con 0x7fa2c40d11b0 2024-04-06T13:29:02.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.078+0000 7fa2cbb30700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2c40d0b50 con 0x7fa2c40cb870 2024-04-06T13:29:02.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.078+0000 7fa2c98cc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa2c40d11b0 0x7fa2c40d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:02.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.078+0000 7fa2c98cc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa2c40d11b0 0x7fa2c40d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35318/0 (socket says 172.21.15.189:35318) 2024-04-06T13:29:02.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.078+0000 7fa2c98cc700 1 -- 172.21.15.189:0/2389337166 learned_addr learned my addr 172.21.15.189:0/2389337166 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:02.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.078+0000 7fa2c90cb700 1 --2- 172.21.15.189:0/2389337166 >> v2:172.21.15.192:3300/0 conn(0x7fa2c40cb870 0x7fa2c415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:02.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.078+0000 7fa2c98cc700 1 -- 172.21.15.189:0/2389337166 >> v2:172.21.15.192:3300/0 conn(0x7fa2c40cb870 msgr2=0x7fa2c415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:02.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.078+0000 7fa2c98cc700 1 --2- 172.21.15.189:0/2389337166 >> v2:172.21.15.192:3300/0 conn(0x7fa2c40cb870 0x7fa2c415c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.078+0000 7fa2c98cc700 1 -- 172.21.15.189:0/2389337166 >> v1:172.21.15.192:6789/0 conn(0x7fa2c40cfa00 legacy=0x7fa2c40cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:02.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.078+0000 7fa2c98cc700 1 -- 172.21.15.189:0/2389337166 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c40cbd80 con 0x7fa2c40d11b0 2024-04-06T13:29:02.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.078+0000 7fa2c98cc700 1 --2- 172.21.15.189:0/2389337166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2c40d11b0 0x7fa2c40d1590 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fa2c404fb40 tx=0x7fa2c0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5760c64bc79b976 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:02.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.078+0000 7fa2c88ca700 1 -- 172.21.15.189:0/2389337166 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0011830 con 0x7fa2c40d11b0 2024-04-06T13:29:02.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.078+0000 7fa2c88ca700 1 -- 172.21.15.189:0/2389337166 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c0011990 con 0x7fa2c40d11b0 2024-04-06T13:29:02.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.078+0000 7fa2c88ca700 1 -- 172.21.15.189:0/2389337166 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0011d20 con 0x7fa2c40d11b0 2024-04-06T13:29:02.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.079+0000 7fa2cbb30700 1 -- 172.21.15.189:0/2389337166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2c40d11b0 msgr2=0x7fa2c40d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:02.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.079+0000 7fa2cbb30700 1 --2- 172.21.15.189:0/2389337166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2c40d11b0 0x7fa2c40d1590 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fa2c404fb40 tx=0x7fa2c0009bd0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.079+0000 7fa2cbb30700 1 -- 172.21.15.189:0/2389337166 shutdown_connections 2024-04-06T13:29:02.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.079+0000 7fa2cbb30700 1 --2- 172.21.15.189:0/2389337166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2c40d11b0 0x7fa2c40d1590 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.079+0000 7fa2cbb30700 1 --2- 172.21.15.189:0/2389337166 >> v2:172.21.15.192:3300/0 conn(0x7fa2c40cb870 0x7fa2c415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.079+0000 7fa2cbb30700 1 -- 172.21.15.189:0/2389337166 >> 172.21.15.189:0/2389337166 conn(0x7fa2c40b2db0 msgr2=0x7fa2c40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:02.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.079+0000 7fa2cbb30700 1 -- 172.21.15.189:0/2389337166 shutdown_connections 2024-04-06T13:29:02.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.079+0000 7fa2cbb30700 1 -- 172.21.15.189:0/2389337166 wait complete. 2024-04-06T13:29:02.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.080+0000 7fa2cbb30700 1 Processor -- start 2024-04-06T13:29:02.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.080+0000 7fa2cbb30700 1 -- start start 2024-04-06T13:29:02.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.081+0000 7fa2cbb30700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2c40cb870 0x7fa2c41e67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:02.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.081+0000 7fa2cbb30700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2c40cfa00 0x7fa2c4155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:02.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.081+0000 7fa2cbb30700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2c40d11b0 0x7fa2c415e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:02.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.081+0000 7fa2cbb30700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c40ca1d0 con 0x7fa2c40cfa00 2024-04-06T13:29:02.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.081+0000 7fa2cbb30700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c40c9270 con 0x7fa2c40d11b0 2024-04-06T13:29:02.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.081+0000 7fa2cbb30700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c40c9550 con 0x7fa2c40cb870 2024-04-06T13:29:02.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.081+0000 7fa2c90cb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2c40cfa00 0x7fa2c4155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:02.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.081+0000 7fa2c90cb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2c40cfa00 0x7fa2c4155670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35328/0 (socket says 172.21.15.189:35328) 2024-04-06T13:29:02.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.081+0000 7fa2c90cb700 1 -- 172.21.15.189:0/1367489556 learned_addr learned my addr 172.21.15.189:0/1367489556 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:02.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.081+0000 7fa2c98cc700 1 --2- 172.21.15.189:0/1367489556 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2c40cb870 0x7fa2c41e67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:02.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.081+0000 7fa2ca0cd700 1 --2- 172.21.15.189:0/1367489556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2c40d11b0 0x7fa2c415e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:02.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.081+0000 7fa2c90cb700 1 -- 172.21.15.189:0/1367489556 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2c40cb870 msgr2=0x7fa2c41e67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:02.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.081+0000 7fa2c90cb700 1 --2- 172.21.15.189:0/1367489556 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2c40cb870 0x7fa2c41e67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.081+0000 7fa2c90cb700 1 -- 172.21.15.189:0/1367489556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2c40d11b0 msgr2=0x7fa2c415e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:02.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.081+0000 7fa2c90cb700 1 --2- 172.21.15.189:0/1367489556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2c40d11b0 0x7fa2c415e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.081+0000 7fa2c90cb700 1 -- 172.21.15.189:0/1367489556 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c0009550 con 0x7fa2c40cfa00 2024-04-06T13:29:02.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.081+0000 7fa2ca0cd700 1 --2- 172.21.15.189:0/1367489556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2c40d11b0 0x7fa2c415e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:29:02.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.082+0000 7fa2c90cb700 1 --2- 172.21.15.189:0/1367489556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2c40cfa00 0x7fa2c4155670 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400e840 tx=0x7fa2b400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:02.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.082+0000 7fa2ba7fc700 1 -- 172.21.15.189:0/1367489556 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4009590 con 0x7fa2c40cfa00 2024-04-06T13:29:02.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.082+0000 7fa2ba7fc700 1 -- 172.21.15.189:0/1367489556 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b4013680 con 0x7fa2c40cfa00 2024-04-06T13:29:02.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.082+0000 7fa2cbb30700 1 -- 172.21.15.189:0/1367489556 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c40c9830 con 0x7fa2c40cfa00 2024-04-06T13:29:02.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.082+0000 7fa2ba7fc700 1 -- 172.21.15.189:0/1367489556 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4013a30 con 0x7fa2c40cfa00 2024-04-06T13:29:02.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.082+0000 7fa2cbb30700 1 -- 172.21.15.189:0/1367489556 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c40c9df0 con 0x7fa2c40cfa00 2024-04-06T13:29:02.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.083+0000 7fa2cbb30700 1 -- 172.21.15.189:0/1367489556 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c4156160 con 0x7fa2c40cfa00 2024-04-06T13:29:02.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.084+0000 7fa2ba7fc700 1 -- 172.21.15.189:0/1367489556 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 104) v1 ==== 89248+0+0 (secure 0 0 0) 0x7fa2b4013b90 con 0x7fa2c40cfa00 2024-04-06T13:29:02.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.084+0000 7fa2ba7fc700 1 --2- 172.21.15.189:0/1367489556 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2a806adb0 0x7fa2a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:02.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.084+0000 7fa2c98cc700 1 -- 172.21.15.189:0/1367489556 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2a806adb0 msgr2=0x7fa2a806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:02.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.084+0000 7fa2ba7fc700 1 -- 172.21.15.189:0/1367489556 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fa2b4089670 con 0x7fa2c40cfa00 2024-04-06T13:29:02.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.084+0000 7fa2c98cc700 1 --2- 172.21.15.189:0/1367489556 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2a806adb0 0x7fa2a806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:29:02.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.089+0000 7fa2ba7fc700 1 -- 172.21.15.189:0/1367489556 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa2b40543d0 con 0x7fa2c40cfa00 2024-04-06T13:29:02.098 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:02.098+0000 7f27f75d3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:29:02.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.285+0000 7fa2c98cc700 1 -- 172.21.15.189:0/1367489556 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2a806adb0 msgr2=0x7fa2a806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:02.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.285+0000 7fa2c98cc700 1 --2- 172.21.15.189:0/1367489556 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2a806adb0 0x7fa2a806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:29:02.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.290+0000 7fa2cbb30700 1 -- 172.21.15.189:0/1367489556 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2c4074990 con 0x7fa2c40cfa00 2024-04-06T13:29:02.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.292+0000 7fa2ba7fc700 1 -- 172.21.15.189:0/1367489556 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+16873 (secure 0 0 0) 0x7fa2b40579f0 con 0x7fa2c40cfa00 2024-04-06T13:29:02.293 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:29:02.293 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":179,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:28:54.359536+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:28:38.579643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-04-06T13:28:39.779769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/2773566419":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/2407993541":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/2622703801":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/1999978665":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/1375122784":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/1621522008":"2024-04-07T13:28:54.359501+0000","172.21.15.189:6833/47128":"2024-04-07T13:28:54.359501+0000","172.21.15.189:6832/47128":"2024-04-07T13:28:54.359501+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:29:02.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.294+0000 7fa2cbb30700 1 -- 172.21.15.189:0/1367489556 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2a806adb0 msgr2=0x7fa2a806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:29:02.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.294+0000 7fa2cbb30700 1 --2- 172.21.15.189:0/1367489556 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2a806adb0 0x7fa2a806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.294+0000 7fa2cbb30700 1 -- 172.21.15.189:0/1367489556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2c40cfa00 msgr2=0x7fa2c4155670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:02.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.294+0000 7fa2cbb30700 1 --2- 172.21.15.189:0/1367489556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2c40cfa00 0x7fa2c4155670 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400e840 tx=0x7fa2b400eb50 comp rx=0 tx=0).stop 2024-04-06T13:29:02.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.294+0000 7fa2cbb30700 1 -- 172.21.15.189:0/1367489556 shutdown_connections 2024-04-06T13:29:02.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.294+0000 7fa2cbb30700 1 --2- 172.21.15.189:0/1367489556 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa2a806adb0 0x7fa2a806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.294+0000 7fa2cbb30700 1 --2- 172.21.15.189:0/1367489556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2c40d11b0 0x7fa2c415e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.294+0000 7fa2cbb30700 1 --2- 172.21.15.189:0/1367489556 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2c40cb870 0x7fa2c41e67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.294+0000 7fa2cbb30700 1 --2- 172.21.15.189:0/1367489556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2c40cfa00 0x7fa2c4155670 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.294+0000 7fa2cbb30700 1 -- 172.21.15.189:0/1367489556 >> 172.21.15.189:0/1367489556 conn(0x7fa2c40b2db0 msgr2=0x7fa2c40b5fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:02.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.294+0000 7fa2cbb30700 1 -- 172.21.15.189:0/1367489556 shutdown_connections 2024-04-06T13:29:02.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.294+0000 7fa2cbb30700 1 -- 172.21.15.189:0/1367489556 wait complete. 2024-04-06T13:29:02.309 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-06T13:29:02.309 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:29:02.340 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:02.340+0000 7f27f75d3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:29:02.415 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:02.415+0000 7f27f75d3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:29:02.432 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:02.431+0000 7ff5abd26200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:29:02.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.434+0000 7f6bd3318700 1 Processor -- start 2024-04-06T13:29:02.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.434+0000 7f6bd3318700 1 -- start start 2024-04-06T13:29:02.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.435+0000 7f6bd3318700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bcc0b3e20 0x7f6bcc15e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:02.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.435+0000 7f6bd3318700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bcc0b3990 con 0x7f6bcc06ae40 2024-04-06T13:29:02.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.435+0000 7f6bd3318700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bcc0b3060 con 0x7f6bcc15d7b0 2024-04-06T13:29:02.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.435+0000 7f6bd3318700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc0b3340 con 0x7f6bcc0b3e20 2024-04-06T13:29:02.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.435+0000 7f6bd08b3700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6bcc06ae40 0x7f6bcc15ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34214/0 (socket says 172.21.15.189:34214) 2024-04-06T13:29:02.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.435+0000 7f6bd08b3700 1 -- 172.21.15.189:0/2792619150 learned_addr learned my addr 172.21.15.189:0/2792619150 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:02.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.435+0000 7f6bd18b5700 1 --2- 172.21.15.189:0/2792619150 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bcc0b3e20 0x7f6bcc15e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:02.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.435+0000 7f6bc3fff700 1 -- 172.21.15.189:0/2792619150 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399305890 0 0) 0x7f6bcc0b3990 con 0x7f6bcc06ae40 2024-04-06T13:29:02.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.435+0000 7f6bc3fff700 1 -- 172.21.15.189:0/2792619150 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bb80035b0 con 0x7f6bcc06ae40 2024-04-06T13:29:02.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.435+0000 7f6bc3fff700 1 -- 172.21.15.189:0/2792619150 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458827923 0 0) 0x7f6bcc0b3060 con 0x7f6bcc15d7b0 2024-04-06T13:29:02.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.435+0000 7f6bd18b5700 1 -- 172.21.15.189:0/2792619150 >> v1:172.21.15.192:6789/0 conn(0x7f6bcc15d7b0 legacy=0x7f6bcc06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:02.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.435+0000 7f6bd18b5700 1 -- 172.21.15.189:0/2792619150 >> v1:172.21.15.189:6789/0 conn(0x7f6bcc06ae40 legacy=0x7f6bcc15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:02.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.435+0000 7f6bd18b5700 1 -- 172.21.15.189:0/2792619150 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc0b3620 con 0x7f6bcc0b3e20 2024-04-06T13:29:02.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.436+0000 7f6bd18b5700 1 --2- 172.21.15.189:0/2792619150 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bcc0b3e20 0x7f6bcc15e500 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4009dc0 tx=0x7f6bc400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=749eebb3888ebc41 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:02.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.436+0000 7f6bc3fff700 1 -- 172.21.15.189:0/2792619150 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc400e070 con 0x7f6bcc0b3e20 2024-04-06T13:29:02.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.436+0000 7f6bc3fff700 1 -- 172.21.15.189:0/2792619150 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc400bb20 con 0x7f6bcc0b3e20 2024-04-06T13:29:02.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.436+0000 7f6bc3fff700 1 -- 172.21.15.189:0/2792619150 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc400be90 con 0x7f6bcc0b3e20 2024-04-06T13:29:02.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.436+0000 7f6bd3318700 1 -- 172.21.15.189:0/2792619150 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bcc0b3e20 msgr2=0x7f6bcc15e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:02.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.436+0000 7f6bd3318700 1 --2- 172.21.15.189:0/2792619150 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bcc0b3e20 0x7f6bcc15e500 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4009dc0 tx=0x7f6bc400b330 comp rx=0 tx=0).stop 2024-04-06T13:29:02.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.436+0000 7f6bd3318700 1 -- 172.21.15.189:0/2792619150 shutdown_connections 2024-04-06T13:29:02.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.436+0000 7f6bd3318700 1 --2- 172.21.15.189:0/2792619150 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bcc0b3e20 0x7f6bcc15e500 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.436+0000 7f6bd3318700 1 -- 172.21.15.189:0/2792619150 >> 172.21.15.189:0/2792619150 conn(0x7f6bcc0d00b0 msgr2=0x7f6bcc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:02.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.437+0000 7f6bd3318700 1 -- 172.21.15.189:0/2792619150 shutdown_connections 2024-04-06T13:29:02.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.437+0000 7f6bd3318700 1 -- 172.21.15.189:0/2792619150 wait complete. 2024-04-06T13:29:02.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.438+0000 7f6bd3318700 1 Processor -- start 2024-04-06T13:29:02.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.438+0000 7f6bd3318700 1 -- start start 2024-04-06T13:29:02.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.438+0000 7f6bd3318700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bcc06ae40 0x7f6bcc156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:02.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.438+0000 7f6bd3318700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bcc0b3e20 0x7f6bcc06d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:02.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.438+0000 7f6bd3318700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bcc15d7b0 0x7f6bcc06db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:02.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.438+0000 7f6bd3318700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc1f7010 con 0x7f6bcc15d7b0 2024-04-06T13:29:02.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.438+0000 7f6bd3318700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc157180 con 0x7f6bcc0b3e20 2024-04-06T13:29:02.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.438+0000 7f6bd3318700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc1f6400 con 0x7f6bcc06ae40 2024-04-06T13:29:02.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.439+0000 7f6bd18b5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bcc15d7b0 0x7f6bcc06db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:02.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.439+0000 7f6bd18b5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bcc15d7b0 0x7f6bcc06db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35338/0 (socket says 172.21.15.189:35338) 2024-04-06T13:29:02.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.439+0000 7f6bd18b5700 1 -- 172.21.15.189:0/2298103629 learned_addr learned my addr 172.21.15.189:0/2298103629 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:02.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.439+0000 7f6bd08b3700 1 --2- 172.21.15.189:0/2298103629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bcc0b3e20 0x7f6bcc06d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:02.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.439+0000 7f6bd10b4700 1 --2- 172.21.15.189:0/2298103629 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bcc06ae40 0x7f6bcc156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:02.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.439+0000 7f6bd18b5700 1 -- 172.21.15.189:0/2298103629 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bcc06ae40 msgr2=0x7f6bcc156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:02.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.439+0000 7f6bd18b5700 1 --2- 172.21.15.189:0/2298103629 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bcc06ae40 0x7f6bcc156c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.439+0000 7f6bd18b5700 1 -- 172.21.15.189:0/2298103629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bcc0b3e20 msgr2=0x7f6bcc06d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:02.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.439+0000 7f6bd18b5700 1 --2- 172.21.15.189:0/2298103629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bcc0b3e20 0x7f6bcc06d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.439+0000 7f6bd18b5700 1 -- 172.21.15.189:0/2298103629 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc4009a70 con 0x7f6bcc15d7b0 2024-04-06T13:29:02.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.439+0000 7f6bd18b5700 1 --2- 172.21.15.189:0/2298103629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bcc15d7b0 0x7f6bcc06db50 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc06c820 tx=0x7f6bc4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:02.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.439+0000 7f6bc1ffb700 1 -- 172.21.15.189:0/2298103629 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc40046b0 con 0x7f6bcc15d7b0 2024-04-06T13:29:02.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.439+0000 7f6bd3318700 1 -- 172.21.15.189:0/2298103629 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bcc1f66e0 con 0x7f6bcc15d7b0 2024-04-06T13:29:02.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.439+0000 7f6bc1ffb700 1 -- 172.21.15.189:0/2298103629 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc4021070 con 0x7f6bcc15d7b0 2024-04-06T13:29:02.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.439+0000 7f6bc1ffb700 1 -- 172.21.15.189:0/2298103629 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc40098f0 con 0x7f6bcc15d7b0 2024-04-06T13:29:02.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.439+0000 7f6bd3318700 1 -- 172.21.15.189:0/2298103629 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bcc1f6ca0 con 0x7f6bcc15d7b0 2024-04-06T13:29:02.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.440+0000 7f6bd3318700 1 -- 172.21.15.189:0/2298103629 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bcc1f71c0 con 0x7f6bcc15d7b0 2024-04-06T13:29:02.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.442+0000 7f6bc1ffb700 1 -- 172.21.15.189:0/2298103629 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 104) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f6bc4023030 con 0x7f6bcc15d7b0 2024-04-06T13:29:02.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.442+0000 7f6bc1ffb700 1 --2- 172.21.15.189:0/2298103629 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6bb006adb0 0x7f6bb006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:02.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.442+0000 7f6bd10b4700 1 -- 172.21.15.189:0/2298103629 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6bb006adb0 msgr2=0x7f6bb006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:02.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.443+0000 7f6bc1ffb700 1 -- 172.21.15.189:0/2298103629 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f6bc408b750 con 0x7f6bcc15d7b0 2024-04-06T13:29:02.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.443+0000 7f6bd10b4700 1 --2- 172.21.15.189:0/2298103629 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6bb006adb0 0x7f6bb006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:29:02.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.446+0000 7f6bc1ffb700 1 -- 172.21.15.189:0/2298103629 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6bc40565e0 con 0x7f6bcc15d7b0 2024-04-06T13:29:02.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.644+0000 7f6bd10b4700 1 -- 172.21.15.189:0/2298103629 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6bb006adb0 msgr2=0x7f6bb006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:02.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.644+0000 7f6bd10b4700 1 --2- 172.21.15.189:0/2298103629 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6bb006adb0 0x7f6bb006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:29:02.690 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:02.689+0000 7ff5abd26200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:29:02.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.694+0000 7f6bd3318700 1 -- 172.21.15.189:0/2298103629 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bcc067bb0 con 0x7f6bcc15d7b0 2024-04-06T13:29:02.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.695+0000 7f6bc1ffb700 1 -- 172.21.15.189:0/2298103629 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4750 (secure 0 0 0) 0x7f6bc4059c00 con 0x7f6bcc15d7b0 2024-04-06T13:29:02.696 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:29:02.696 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":78,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/3388956983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3388956983},{"type":"v1","addr":"172.21.15.192:6834","nonce":3388956983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":8845,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1844918354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":1844918354},{"type":"v1","addr":"172.21.15.192:6835","nonce":1844918354}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":9618,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3742852104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3742852104},{"type":"v1","addr":"172.21.15.189:6836","nonce":3742852104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:28:40.813074+0000","modified":"2024-04-06T13:28:43.827822+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9621":{"gid":9621,"name":"c","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/2089191015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2089191015},{"type":"v1","addr":"172.21.15.189:6837","nonce":2089191015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-06T13:29:02.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.697+0000 7f6bd3318700 1 -- 172.21.15.189:0/2298103629 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6bb006adb0 msgr2=0x7f6bb006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:29:02.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.697+0000 7f6bd3318700 1 --2- 172.21.15.189:0/2298103629 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6bb006adb0 0x7f6bb006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.697+0000 7f6bd3318700 1 -- 172.21.15.189:0/2298103629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bcc15d7b0 msgr2=0x7f6bcc06db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:02.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.697+0000 7f6bd3318700 1 --2- 172.21.15.189:0/2298103629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bcc15d7b0 0x7f6bcc06db50 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc06c820 tx=0x7f6bc4004ad0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.697+0000 7f6bd3318700 1 -- 172.21.15.189:0/2298103629 shutdown_connections 2024-04-06T13:29:02.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.697+0000 7f6bd3318700 1 --2- 172.21.15.189:0/2298103629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bcc15d7b0 0x7f6bcc06db50 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.697+0000 7f6bd3318700 1 --2- 172.21.15.189:0/2298103629 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bcc06ae40 0x7f6bcc156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.697+0000 7f6bd3318700 1 --2- 172.21.15.189:0/2298103629 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f6bb006adb0 0x7f6bb006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.697+0000 7f6bd3318700 1 --2- 172.21.15.189:0/2298103629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bcc0b3e20 0x7f6bcc06d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.697+0000 7f6bd3318700 1 -- 172.21.15.189:0/2298103629 >> 172.21.15.189:0/2298103629 conn(0x7f6bcc0d00b0 msgr2=0x7f6bcc06f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:02.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.697+0000 7f6bd3318700 1 -- 172.21.15.189:0/2298103629 shutdown_connections 2024-04-06T13:29:02.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.697+0000 7f6bd3318700 1 -- 172.21.15.189:0/2298103629 wait complete. 2024-04-06T13:29:02.700 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 78 2024-04-06T13:29:02.716 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:29:02.778 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:02.778+0000 7ff5abd26200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:29:02.792 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:02.792+0000 7f27f75d3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:29:02.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.840+0000 7f9f82916700 1 Processor -- start 2024-04-06T13:29:02.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.840+0000 7f9f82916700 1 -- start start 2024-04-06T13:29:02.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.841+0000 7f9f82916700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f7c0d17a0 0x7f9f7c06b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:02.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.841+0000 7f9f82916700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9f7c146dd0 0x7f9f7c06d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:02.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.841+0000 7f9f82916700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9f7c0cedf0 0x7f9f7c0cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:02.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.841+0000 7f9f82916700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f7c0cf760 con 0x7f9f7c0cedf0 2024-04-06T13:29:02.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.841+0000 7f9f82916700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f7c06c0a0 con 0x7f9f7c146dd0 2024-04-06T13:29:02.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.841+0000 7f9f82916700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7c06c380 con 0x7f9f7c0d17a0 2024-04-06T13:29:02.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.841+0000 7f9f80eb3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9f7c0cedf0 0x7f9f7c0cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:02.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.841+0000 7f9f80eb3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9f7c0cedf0 0x7f9f7c0cf220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35356/0 (socket says 172.21.15.189:35356) 2024-04-06T13:29:02.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.841+0000 7f9f80eb3700 1 -- 172.21.15.189:0/1698742485 learned_addr learned my addr 172.21.15.189:0/1698742485 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:02.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.841+0000 7f9f7b7fe700 1 --2- 172.21.15.189:0/1698742485 >> v2:172.21.15.192:3300/0 conn(0x7f9f7c146dd0 0x7f9f7c06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:02.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.841+0000 7f9f7bfff700 1 --2- 172.21.15.189:0/1698742485 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f7c0d17a0 0x7f9f7c06b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:02.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.841+0000 7f9f80eb3700 1 -- 172.21.15.189:0/1698742485 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f7c0d17a0 msgr2=0x7f9f7c06b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:02.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.841+0000 7f9f80eb3700 1 --2- 172.21.15.189:0/1698742485 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f7c0d17a0 0x7f9f7c06b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.841+0000 7f9f80eb3700 1 -- 172.21.15.189:0/1698742485 >> v2:172.21.15.192:3300/0 conn(0x7f9f7c146dd0 msgr2=0x7f9f7c06d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:02.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.841+0000 7f9f80eb3700 1 --2- 172.21.15.189:0/1698742485 >> v2:172.21.15.192:3300/0 conn(0x7f9f7c146dd0 0x7f9f7c06d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.841+0000 7f9f80eb3700 1 -- 172.21.15.189:0/1698742485 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f7c06c660 con 0x7f9f7c0cedf0 2024-04-06T13:29:02.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.842+0000 7f9f80eb3700 1 --2- 172.21.15.189:0/1698742485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f7c0cedf0 0x7f9f7c0cf220 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c0ced30 tx=0x7f9f7400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aaadd7a04bac674 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:02.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.842+0000 7f9f7affd700 1 -- 172.21.15.189:0/1698742485 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7400e070 con 0x7f9f7c0cedf0 2024-04-06T13:29:02.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.842+0000 7f9f7affd700 1 -- 172.21.15.189:0/1698742485 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7400b9c0 con 0x7f9f7c0cedf0 2024-04-06T13:29:02.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.842+0000 7f9f7affd700 1 -- 172.21.15.189:0/1698742485 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7400bd70 con 0x7f9f7c0cedf0 2024-04-06T13:29:02.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.842+0000 7f9f82916700 1 -- 172.21.15.189:0/1698742485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f7c0cedf0 msgr2=0x7f9f7c0cf220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:02.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.842+0000 7f9f82916700 1 --2- 172.21.15.189:0/1698742485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f7c0cedf0 0x7f9f7c0cf220 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c0ced30 tx=0x7f9f7400b330 comp rx=0 tx=0).stop 2024-04-06T13:29:02.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.842+0000 7f9f82916700 1 -- 172.21.15.189:0/1698742485 shutdown_connections 2024-04-06T13:29:02.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.842+0000 7f9f82916700 1 --2- 172.21.15.189:0/1698742485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f7c0cedf0 0x7f9f7c0cf220 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.842+0000 7f9f82916700 1 --2- 172.21.15.189:0/1698742485 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f7c0d17a0 0x7f9f7c06b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.843+0000 7f9f82916700 1 --2- 172.21.15.189:0/1698742485 >> v2:172.21.15.192:3300/0 conn(0x7f9f7c146dd0 0x7f9f7c06d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.843+0000 7f9f82916700 1 -- 172.21.15.189:0/1698742485 >> 172.21.15.189:0/1698742485 conn(0x7f9f7c0b2db0 msgr2=0x7f9f7c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:02.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.843+0000 7f9f82916700 1 -- 172.21.15.189:0/1698742485 shutdown_connections 2024-04-06T13:29:02.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.843+0000 7f9f82916700 1 -- 172.21.15.189:0/1698742485 wait complete. 2024-04-06T13:29:02.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.844+0000 7f9f82916700 1 Processor -- start 2024-04-06T13:29:02.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.845+0000 7f9f82916700 1 -- start start 2024-04-06T13:29:02.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.846+0000 7f9f82916700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f7c0cedf0 0x7f9f7c159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:02.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.846+0000 7f9f82916700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f7c0d17a0 0x7f9f7c1f6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:02.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.846+0000 7f9f82916700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f7c146dd0 0x7f9f7c1f6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:02.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.846+0000 7f9f82916700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7c0c7c60 con 0x7f9f7c0cedf0 2024-04-06T13:29:02.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.846+0000 7f9f82916700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7c1f4260 con 0x7f9f7c146dd0 2024-04-06T13:29:02.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.846+0000 7f9f82916700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7c1f4540 con 0x7f9f7c0d17a0 2024-04-06T13:29:02.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.846+0000 7f9f7bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f7c0cedf0 0x7f9f7c159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:02.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.846+0000 7f9f7bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f7c0cedf0 0x7f9f7c159f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35358/0 (socket says 172.21.15.189:35358) 2024-04-06T13:29:02.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.846+0000 7f9f7bfff700 1 -- 172.21.15.189:0/415765202 learned_addr learned my addr 172.21.15.189:0/415765202 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:02.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.846+0000 7f9f80eb3700 1 --2- 172.21.15.189:0/415765202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f7c146dd0 0x7f9f7c1f6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:02.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.846+0000 7f9f7b7fe700 1 --2- 172.21.15.189:0/415765202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f7c0d17a0 0x7f9f7c1f6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:02.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.846+0000 7f9f7bfff700 1 -- 172.21.15.189:0/415765202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f7c0d17a0 msgr2=0x7f9f7c1f6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:02.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.846+0000 7f9f7bfff700 1 --2- 172.21.15.189:0/415765202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f7c0d17a0 0x7f9f7c1f6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.846+0000 7f9f7bfff700 1 -- 172.21.15.189:0/415765202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f7c146dd0 msgr2=0x7f9f7c1f6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:02.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.846+0000 7f9f7bfff700 1 --2- 172.21.15.189:0/415765202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f7c146dd0 0x7f9f7c1f6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:02.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.846+0000 7f9f7bfff700 1 -- 172.21.15.189:0/415765202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f74009b30 con 0x7f9f7c0cedf0 2024-04-06T13:29:02.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.846+0000 7f9f7b7fe700 1 --2- 172.21.15.189:0/415765202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f7c0d17a0 0x7f9f7c1f6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:29:02.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.846+0000 7f9f7bfff700 1 --2- 172.21.15.189:0/415765202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f7c0cedf0 0x7f9f7c159f80 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00c7f0 tx=0x7f9f6c00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:02.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.846+0000 7f9f78ff9700 1 -- 172.21.15.189:0/415765202 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c011800 con 0x7f9f7c0cedf0 2024-04-06T13:29:02.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.847+0000 7f9f82916700 1 -- 172.21.15.189:0/415765202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f7c1f4820 con 0x7f9f7c0cedf0 2024-04-06T13:29:02.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.847+0000 7f9f78ff9700 1 -- 172.21.15.189:0/415765202 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6c011960 con 0x7f9f7c0cedf0 2024-04-06T13:29:02.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.847+0000 7f9f78ff9700 1 -- 172.21.15.189:0/415765202 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c011cd0 con 0x7f9f7c0cedf0 2024-04-06T13:29:02.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.847+0000 7f9f82916700 1 -- 172.21.15.189:0/415765202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f7c0c6ec0 con 0x7f9f7c0cedf0 2024-04-06T13:29:02.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.848+0000 7f9f82916700 1 -- 172.21.15.189:0/415765202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f7c074990 con 0x7f9f7c0cedf0 2024-04-06T13:29:02.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.848+0000 7f9f78ff9700 1 -- 172.21.15.189:0/415765202 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 104) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f9f6c010350 con 0x7f9f7c0cedf0 2024-04-06T13:29:02.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.849+0000 7f9f78ff9700 1 --2- 172.21.15.189:0/415765202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9f6806adb0 0x7f9f6806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:02.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.849+0000 7f9f7b7fe700 1 -- 172.21.15.189:0/415765202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9f6806adb0 msgr2=0x7f9f6806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:02.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.849+0000 7f9f78ff9700 1 -- 172.21.15.189:0/415765202 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f9f6c0894a0 con 0x7f9f7c0cedf0 2024-04-06T13:29:02.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.849+0000 7f9f7b7fe700 1 --2- 172.21.15.189:0/415765202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9f6806adb0 0x7f9f6806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:29:02.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:02.853+0000 7f9f78ff9700 1 -- 172.21.15.189:0/415765202 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9f6c054100 con 0x7f9f7c0cedf0 2024-04-06T13:29:03.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.050+0000 7f9f7b7fe700 1 -- 172.21.15.189:0/415765202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9f6806adb0 msgr2=0x7f9f6806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:03.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.050+0000 7f9f7b7fe700 1 --2- 172.21.15.189:0/415765202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9f6806adb0 0x7f9f6806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:29:03.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.102+0000 7f9f82916700 1 -- 172.21.15.189:0/415765202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f7c0b3900 con 0x7f9f7c0cedf0 2024-04-06T13:29:03.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.102+0000 7f9f78ff9700 1 -- 172.21.15.189:0/415765202 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4750 (secure 0 0 0) 0x7f9f6c057720 con 0x7f9f7c0cedf0 2024-04-06T13:29:03.103 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:29:03.104 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":78,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/3388956983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3388956983},{"type":"v1","addr":"172.21.15.192:6834","nonce":3388956983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":8845,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1844918354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":1844918354},{"type":"v1","addr":"172.21.15.192:6835","nonce":1844918354}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":9618,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3742852104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3742852104},{"type":"v1","addr":"172.21.15.189:6836","nonce":3742852104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:28:40.813074+0000","modified":"2024-04-06T13:28:43.827822+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9621":{"gid":9621,"name":"c","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/2089191015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2089191015},{"type":"v1","addr":"172.21.15.189:6837","nonce":2089191015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-06T13:29:03.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.105+0000 7f9f82916700 1 -- 172.21.15.189:0/415765202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9f6806adb0 msgr2=0x7f9f6806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:29:03.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.105+0000 7f9f82916700 1 --2- 172.21.15.189:0/415765202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9f6806adb0 0x7f9f6806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.105+0000 7f9f82916700 1 -- 172.21.15.189:0/415765202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f7c0cedf0 msgr2=0x7f9f7c159f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:03.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.105+0000 7f9f82916700 1 --2- 172.21.15.189:0/415765202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f7c0cedf0 0x7f9f7c159f80 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00c7f0 tx=0x7f9f6c00cb00 comp rx=0 tx=0).stop 2024-04-06T13:29:03.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.105+0000 7f9f82916700 1 -- 172.21.15.189:0/415765202 shutdown_connections 2024-04-06T13:29:03.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.105+0000 7f9f82916700 1 --2- 172.21.15.189:0/415765202 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f9f6806adb0 0x7f9f6806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.105+0000 7f9f82916700 1 --2- 172.21.15.189:0/415765202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f7c146dd0 0x7f9f7c1f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.105+0000 7f9f82916700 1 --2- 172.21.15.189:0/415765202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f7c0cedf0 0x7f9f7c159f80 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.105+0000 7f9f82916700 1 --2- 172.21.15.189:0/415765202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f7c0d17a0 0x7f9f7c1f6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.105+0000 7f9f82916700 1 -- 172.21.15.189:0/415765202 >> 172.21.15.189:0/415765202 conn(0x7f9f7c0b2db0 msgr2=0x7f9f7c0b3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:03.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.105+0000 7f9f82916700 1 -- 172.21.15.189:0/415765202 shutdown_connections 2024-04-06T13:29:03.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.105+0000 7f9f82916700 1 -- 172.21.15.189:0/415765202 wait complete. 2024-04-06T13:29:03.109 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 78 2024-04-06T13:29:03.119 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:29:03.183 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:03.182+0000 7ff5abd26200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:29:03.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.257+0000 7fdb406ea700 1 Processor -- start 2024-04-06T13:29:03.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.257+0000 7fdb406ea700 1 -- start start 2024-04-06T13:29:03.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.259+0000 7fdb406ea700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb380d11b0 0x7fdb380d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:03.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.259+0000 7fdb406ea700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdb380cb870 0x7fdb3815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:03.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.259+0000 7fdb406ea700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdb380cfa00 0x7fdb380cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:03.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.259+0000 7fdb406ea700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb380ccdc0 con 0x7fdb380cb870 2024-04-06T13:29:03.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.259+0000 7fdb406ea700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb380d0b50 con 0x7fdb380cfa00 2024-04-06T13:29:03.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.259+0000 7fdb406ea700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb380d1ad0 con 0x7fdb380d11b0 2024-04-06T13:29:03.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.259+0000 7fdb3dc85700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdb380cb870 0x7fdb3815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:03.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.259+0000 7fdb3dc85700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdb380cb870 0x7fdb3815c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35432/0 (socket says 172.21.15.189:35432) 2024-04-06T13:29:03.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.259+0000 7fdb3dc85700 1 -- 172.21.15.189:0/2890218298 learned_addr learned my addr 172.21.15.189:0/2890218298 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:03.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.259+0000 7fdb3e486700 1 --2- 172.21.15.189:0/2890218298 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb380d11b0 0x7fdb380d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:03.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.259+0000 7fdb3ec87700 1 --2- 172.21.15.189:0/2890218298 >> v2:172.21.15.192:3300/0 conn(0x7fdb380cfa00 0x7fdb380cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:03.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.259+0000 7fdb3e486700 1 -- 172.21.15.189:0/2890218298 >> v2:172.21.15.192:3300/0 conn(0x7fdb380cfa00 msgr2=0x7fdb380cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:03.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.259+0000 7fdb3e486700 1 --2- 172.21.15.189:0/2890218298 >> v2:172.21.15.192:3300/0 conn(0x7fdb380cfa00 0x7fdb380cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.259+0000 7fdb3e486700 1 -- 172.21.15.189:0/2890218298 >> v2:172.21.15.189:3300/0 conn(0x7fdb380cb870 msgr2=0x7fdb3815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:03.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.259+0000 7fdb3e486700 1 --2- 172.21.15.189:0/2890218298 >> v2:172.21.15.189:3300/0 conn(0x7fdb380cb870 0x7fdb3815c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.259+0000 7fdb3e486700 1 -- 172.21.15.189:0/2890218298 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb380cbd80 con 0x7fdb380d11b0 2024-04-06T13:29:03.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.259+0000 7fdb3dc85700 1 --2- 172.21.15.189:0/2890218298 >> v2:172.21.15.189:3300/0 conn(0x7fdb380cb870 0x7fdb3815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:29:03.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.259+0000 7fdb3e486700 1 --2- 172.21.15.189:0/2890218298 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb380d11b0 0x7fdb380d1590 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdb380b0990 tx=0x7fdb34009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf2fe61666996b04 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:03.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.260+0000 7fdb3d484700 1 -- 172.21.15.189:0/2890218298 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34011830 con 0x7fdb380d11b0 2024-04-06T13:29:03.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.260+0000 7fdb3d484700 1 -- 172.21.15.189:0/2890218298 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb34011990 con 0x7fdb380d11b0 2024-04-06T13:29:03.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.260+0000 7fdb3d484700 1 -- 172.21.15.189:0/2890218298 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34011d20 con 0x7fdb380d11b0 2024-04-06T13:29:03.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.260+0000 7fdb406ea700 1 -- 172.21.15.189:0/2890218298 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb380d11b0 msgr2=0x7fdb380d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:03.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.260+0000 7fdb406ea700 1 --2- 172.21.15.189:0/2890218298 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb380d11b0 0x7fdb380d1590 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdb380b0990 tx=0x7fdb34009b20 comp rx=0 tx=0).stop 2024-04-06T13:29:03.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.260+0000 7fdb406ea700 1 -- 172.21.15.189:0/2890218298 shutdown_connections 2024-04-06T13:29:03.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.260+0000 7fdb406ea700 1 --2- 172.21.15.189:0/2890218298 >> v2:172.21.15.192:3300/0 conn(0x7fdb380cfa00 0x7fdb380cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.260+0000 7fdb406ea700 1 --2- 172.21.15.189:0/2890218298 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb380d11b0 0x7fdb380d1590 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.260+0000 7fdb406ea700 1 --2- 172.21.15.189:0/2890218298 >> v2:172.21.15.189:3300/0 conn(0x7fdb380cb870 0x7fdb3815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.260+0000 7fdb406ea700 1 -- 172.21.15.189:0/2890218298 >> 172.21.15.189:0/2890218298 conn(0x7fdb380b2db0 msgr2=0x7fdb380b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:03.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.260+0000 7fdb406ea700 1 -- 172.21.15.189:0/2890218298 shutdown_connections 2024-04-06T13:29:03.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.260+0000 7fdb406ea700 1 -- 172.21.15.189:0/2890218298 wait complete. 2024-04-06T13:29:03.265 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.262+0000 7fdb406ea700 1 Processor -- start 2024-04-06T13:29:03.265 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.262+0000 7fdb406ea700 1 -- start start 2024-04-06T13:29:03.265 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.263+0000 7fdb406ea700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb380cb870 0x7fdb381586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:03.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.263+0000 7fdb406ea700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb380cfa00 0x7fdb38155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:03.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.263+0000 7fdb406ea700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb380d11b0 0x7fdb381f0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:03.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.263+0000 7fdb406ea700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb381564c0 con 0x7fdb380cb870 2024-04-06T13:29:03.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.263+0000 7fdb406ea700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb38158cb0 con 0x7fdb380cfa00 2024-04-06T13:29:03.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.263+0000 7fdb406ea700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb38158f90 con 0x7fdb380d11b0 2024-04-06T13:29:03.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.263+0000 7fdb3e486700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb380cb870 0x7fdb381586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:03.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.263+0000 7fdb3ec87700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb380d11b0 0x7fdb381f0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:03.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.263+0000 7fdb3dc85700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb380cfa00 0x7fdb38155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:03.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.263+0000 7fdb3e486700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb380cb870 0x7fdb381586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35438/0 (socket says 172.21.15.189:35438) 2024-04-06T13:29:03.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.263+0000 7fdb3e486700 1 -- 172.21.15.189:0/2019044217 learned_addr learned my addr 172.21.15.189:0/2019044217 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:03.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.264+0000 7fdb3ec87700 1 -- 172.21.15.189:0/2019044217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb380cfa00 msgr2=0x7fdb38155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:03.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.264+0000 7fdb3ec87700 1 --2- 172.21.15.189:0/2019044217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb380cfa00 0x7fdb38155f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.264+0000 7fdb3ec87700 1 -- 172.21.15.189:0/2019044217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb380cb870 msgr2=0x7fdb381586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:03.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.264+0000 7fdb3ec87700 1 --2- 172.21.15.189:0/2019044217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb380cb870 0x7fdb381586b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.264+0000 7fdb3ec87700 1 -- 172.21.15.189:0/2019044217 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb28009550 con 0x7fdb380d11b0 2024-04-06T13:29:03.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.264+0000 7fdb3ec87700 1 --2- 172.21.15.189:0/2019044217 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb380d11b0 0x7fdb381f0480 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000efc0 tx=0x7fdb3000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:03.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.264+0000 7fdb2effd700 1 -- 172.21.15.189:0/2019044217 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3000cb30 con 0x7fdb380d11b0 2024-04-06T13:29:03.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.264+0000 7fdb2effd700 1 -- 172.21.15.189:0/2019044217 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3000f040 con 0x7fdb380d11b0 2024-04-06T13:29:03.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.264+0000 7fdb2effd700 1 -- 172.21.15.189:0/2019044217 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb30014480 con 0x7fdb380d11b0 2024-04-06T13:29:03.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.264+0000 7fdb406ea700 1 -- 172.21.15.189:0/2019044217 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb340094a0 con 0x7fdb380d11b0 2024-04-06T13:29:03.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.264+0000 7fdb406ea700 1 -- 172.21.15.189:0/2019044217 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb380b3f40 con 0x7fdb380d11b0 2024-04-06T13:29:03.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.265+0000 7fdb406ea700 1 -- 172.21.15.189:0/2019044217 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb38074990 con 0x7fdb380d11b0 2024-04-06T13:29:03.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.266+0000 7fdb2effd700 1 -- 172.21.15.189:0/2019044217 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 105) v1 ==== 88856+0+0 (secure 0 0 0) 0x7fdb300145e0 con 0x7fdb380d11b0 2024-04-06T13:29:03.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.266+0000 7fdb2effd700 1 -- 172.21.15.189:0/2019044217 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fdb30089c10 con 0x7fdb380d11b0 2024-04-06T13:29:03.269 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:03.269+0000 7ff527445700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:03.270 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:03.269+0000 7ff527445700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:03.270 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:03.269+0000 7ff527445700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:03.270 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:03.269+0000 7ff527445700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:03.270 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:03.269+0000 7ff527445700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:03.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.271+0000 7fdb2effd700 1 -- 172.21.15.189:0/2019044217 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdb30054e70 con 0x7fdb380d11b0 2024-04-06T13:29:03.274 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:03.273+0000 7ff52443f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:03.274 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:03.273+0000 7ff52443f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:03.274 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:03.273+0000 7ff52443f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:03.274 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:03.273+0000 7ff52443f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:03.274 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:03.273+0000 7ff52443f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:03.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.470+0000 7fdb406ea700 1 -- 172.21.15.189:0/2019044217 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb380b0a00 con 0x7fdb380d11b0 2024-04-06T13:29:03.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.472+0000 7fdb2effd700 1 -- 172.21.15.189:0/2019044217 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+17250 (secure 0 0 0) 0x7fdb30018020 con 0x7fdb380d11b0 2024-04-06T13:29:03.473 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:29:03.473 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":180,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:29:03.197082+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:28:38.579643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-04-06T13:28:39.779769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/793459212":"2024-04-07T13:29:03.197024+0000","172.21.15.189:0/1307579035":"2024-04-07T13:29:03.197024+0000","172.21.15.189:0/3378582407":"2024-04-07T13:29:03.197024+0000","172.21.15.189:0/2392003074":"2024-04-07T13:29:03.197024+0000","172.21.15.189:0/2773566419":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/2407993541":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/2622703801":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/1999978665":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/1283289671":"2024-04-07T13:29:03.197024+0000","172.21.15.189:0/2543617292":"2024-04-07T13:29:03.197024+0000","172.21.15.189:0/1375122784":"2024-04-07T13:28:54.359501+0000","172.21.15.189:0/1621522008":"2024-04-07T13:28:54.359501+0000","172.21.15.189:6833/47128":"2024-04-07T13:28:54.359501+0000","172.21.15.189:6832/47128":"2024-04-07T13:28:54.359501+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:29:03.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.474+0000 7fdb406ea700 1 -- 172.21.15.189:0/2019044217 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb380d11b0 msgr2=0x7fdb381f0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:03.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.474+0000 7fdb406ea700 1 --2- 172.21.15.189:0/2019044217 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb380d11b0 0x7fdb381f0480 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000efc0 tx=0x7fdb3000c3a0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.474+0000 7fdb406ea700 1 -- 172.21.15.189:0/2019044217 shutdown_connections 2024-04-06T13:29:03.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.474+0000 7fdb406ea700 1 --2- 172.21.15.189:0/2019044217 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb380d11b0 0x7fdb381f0480 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.474+0000 7fdb406ea700 1 --2- 172.21.15.189:0/2019044217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb380cb870 0x7fdb381586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.474+0000 7fdb406ea700 1 --2- 172.21.15.189:0/2019044217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb380cfa00 0x7fdb38155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.474+0000 7fdb406ea700 1 -- 172.21.15.189:0/2019044217 >> 172.21.15.189:0/2019044217 conn(0x7fdb380b2db0 msgr2=0x7fdb380cd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:03.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.474+0000 7fdb406ea700 1 -- 172.21.15.189:0/2019044217 shutdown_connections 2024-04-06T13:29:03.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.474+0000 7fdb406ea700 1 -- 172.21.15.189:0/2019044217 wait complete. 2024-04-06T13:29:03.487 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-06T13:29:03.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.609+0000 7fc97922e700 1 Processor -- start 2024-04-06T13:29:03.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.609+0000 7fc97922e700 1 -- start start 2024-04-06T13:29:03.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.610+0000 7fc97922e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc97406ae40 0x7fc9740b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:03.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.610+0000 7fc97922e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc97415ddf0 0x7fc97415e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:03.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.610+0000 7fc97922e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9740b3990 con 0x7fc97415d7b0 2024-04-06T13:29:03.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.610+0000 7fc97922e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9740b3060 con 0x7fc97406ae40 2024-04-06T13:29:03.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.610+0000 7fc97922e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9740b3340 con 0x7fc97415ddf0 2024-04-06T13:29:03.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.610+0000 7fc972d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc97415d7b0 0x7fc97406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34218/0 (socket says 172.21.15.189:34218) 2024-04-06T13:29:03.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.610+0000 7fc972d9d700 1 -- 172.21.15.189:0/482321479 learned_addr learned my addr 172.21.15.189:0/482321479 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:03.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.610+0000 7fc971d9b700 1 -- 172.21.15.189:0/482321479 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723789834 0 0) 0x7fc9740b3990 con 0x7fc97415d7b0 2024-04-06T13:29:03.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.610+0000 7fc971d9b700 1 -- 172.21.15.189:0/482321479 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc95c0035b0 con 0x7fc97415d7b0 2024-04-06T13:29:03.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.610+0000 7fc97359e700 1 --2- 172.21.15.189:0/482321479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc97415ddf0 0x7fc97415e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:03.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.610+0000 7fc97259c700 1 --2- 172.21.15.189:0/482321479 >> v2:172.21.15.192:3300/0 conn(0x7fc97406ae40 0x7fc9740b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:03.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.610+0000 7fc971d9b700 1 -- 172.21.15.189:0/482321479 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3615797800 0 0) 0x7fc95c0035b0 con 0x7fc97415d7b0 2024-04-06T13:29:03.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc971d9b700 1 -- 172.21.15.189:0/482321479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc97415ddf0 msgr2=0x7fc97415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:03.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc971d9b700 1 --2- 172.21.15.189:0/482321479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc97415ddf0 0x7fc97415e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc971d9b700 1 -- 172.21.15.189:0/482321479 >> v2:172.21.15.192:3300/0 conn(0x7fc97406ae40 msgr2=0x7fc9740b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:03.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc971d9b700 1 --2- 172.21.15.189:0/482321479 >> v2:172.21.15.192:3300/0 conn(0x7fc97406ae40 0x7fc9740b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc971d9b700 1 -- 172.21.15.189:0/482321479 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9740b3620 con 0x7fc97415d7b0 2024-04-06T13:29:03.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc971d9b700 1 -- 172.21.15.189:0/482321479 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fc968001670 con 0x7fc97415d7b0 2024-04-06T13:29:03.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc971d9b700 1 -- 172.21.15.189:0/482321479 >> v1:172.21.15.189:6789/0 conn(0x7fc97415d7b0 legacy=0x7fc97406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:03.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc971d9b700 1 --2- 172.21.15.189:0/482321479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc95c003a00 0x7fc95c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:03.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc971d9b700 1 --2- 172.21.15.189:0/482321479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc95c004850 0x7fc95c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:03.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc971d9b700 1 --2- 172.21.15.189:0/482321479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc97415ddf0 0x7fc95c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:03.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc971d9b700 1 -- 172.21.15.189:0/482321479 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9740b3060 con 0x7fc95c004850 2024-04-06T13:29:03.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc971d9b700 1 -- 172.21.15.189:0/482321479 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9740b3340 con 0x7fc95c003a00 2024-04-06T13:29:03.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc971d9b700 1 -- 172.21.15.189:0/482321479 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c00ba10 con 0x7fc97415ddf0 2024-04-06T13:29:03.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc97259c700 1 --2- 172.21.15.189:0/482321479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc95c004850 0x7fc95c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:03.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc972d9d700 1 --2- 172.21.15.189:0/482321479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc95c003a00 0x7fc95c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:03.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc97359e700 1 --2- 172.21.15.189:0/482321479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc97415ddf0 0x7fc95c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:03.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc97259c700 1 -- 172.21.15.189:0/482321479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc97415ddf0 msgr2=0x7fc95c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:03.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc97259c700 1 --2- 172.21.15.189:0/482321479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc97415ddf0 0x7fc95c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc97259c700 1 -- 172.21.15.189:0/482321479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc95c003a00 msgr2=0x7fc95c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:03.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc97259c700 1 --2- 172.21.15.189:0/482321479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc95c003a00 0x7fc95c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc97259c700 1 -- 172.21.15.189:0/482321479 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95c0035b0 con 0x7fc95c004850 2024-04-06T13:29:03.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc97259c700 1 --2- 172.21.15.189:0/482321479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc95c004850 0x7fc95c004cd0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7fc964009b60 tx=0x7fc95c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db810de7fd6f2eb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:03.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc971d9b700 1 -- 172.21.15.189:0/482321479 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96400e6c0 con 0x7fc95c004850 2024-04-06T13:29:03.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc971d9b700 1 -- 172.21.15.189:0/482321479 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc96400e820 con 0x7fc95c004850 2024-04-06T13:29:03.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.611+0000 7fc971d9b700 1 -- 172.21.15.189:0/482321479 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96400eb90 con 0x7fc95c004850 2024-04-06T13:29:03.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.612+0000 7fc97922e700 1 -- 172.21.15.189:0/482321479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc95c004850 msgr2=0x7fc95c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:03.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.612+0000 7fc97922e700 1 --2- 172.21.15.189:0/482321479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc95c004850 0x7fc95c004cd0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7fc964009b60 tx=0x7fc95c005880 comp rx=0 tx=0).stop 2024-04-06T13:29:03.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.612+0000 7fc97359e700 1 -- 172.21.15.189:0/482321479 reap_dead start 2024-04-06T13:29:03.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.612+0000 7fc97922e700 1 -- 172.21.15.189:0/482321479 shutdown_connections 2024-04-06T13:29:03.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.612+0000 7fc97922e700 1 -- 172.21.15.189:0/482321479 >> 172.21.15.189:0/482321479 conn(0x7fc9740d00b0 msgr2=0x7fc9740d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:03.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.612+0000 7fc97922e700 1 -- 172.21.15.189:0/482321479 shutdown_connections 2024-04-06T13:29:03.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.612+0000 7fc97922e700 1 -- 172.21.15.189:0/482321479 wait complete. 2024-04-06T13:29:03.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.613+0000 7fc97922e700 1 Processor -- start 2024-04-06T13:29:03.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.613+0000 7fc97922e700 1 -- start start 2024-04-06T13:29:03.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.614+0000 7fc97922e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc95c004850 0x7fc9741f04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:03.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.614+0000 7fc97922e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc974145300 0x7fc974152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:03.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.614+0000 7fc97922e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc974153470 0x7fc97415b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:03.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.614+0000 7fc97922e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9741538f0 con 0x7fc974145300 2024-04-06T13:29:03.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.614+0000 7fc97922e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9741f3d80 con 0x7fc95c004850 2024-04-06T13:29:03.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.614+0000 7fc97922e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9741f3170 con 0x7fc974153470 2024-04-06T13:29:03.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.614+0000 7fc97259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc974145300 0x7fc974152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:03.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.614+0000 7fc97259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc974145300 0x7fc974152f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35492/0 (socket says 172.21.15.189:35492) 2024-04-06T13:29:03.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.614+0000 7fc97259c700 1 -- 172.21.15.189:0/64788097 learned_addr learned my addr 172.21.15.189:0/64788097 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:03.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.614+0000 7fc972d9d700 1 --2- 172.21.15.189:0/64788097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc95c004850 0x7fc9741f04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:03.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.614+0000 7fc97359e700 1 --2- 172.21.15.189:0/64788097 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc974153470 0x7fc97415b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:03.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.615+0000 7fc97259c700 1 -- 172.21.15.189:0/64788097 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc974153470 msgr2=0x7fc97415b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:03.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.615+0000 7fc97259c700 1 --2- 172.21.15.189:0/64788097 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc974153470 0x7fc97415b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.615+0000 7fc97259c700 1 -- 172.21.15.189:0/64788097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc95c004850 msgr2=0x7fc9741f04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:03.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.615+0000 7fc97259c700 1 --2- 172.21.15.189:0/64788097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc95c004850 0x7fc9741f04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.615+0000 7fc97259c700 1 -- 172.21.15.189:0/64788097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9741f3450 con 0x7fc974145300 2024-04-06T13:29:03.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.615+0000 7fc97259c700 1 --2- 172.21.15.189:0/64788097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc974145300 0x7fc974152f30 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7fc96400e4b0 tx=0x7fc9640047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:03.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.615+0000 7fc95b7fe700 1 -- 172.21.15.189:0/64788097 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96400e6c0 con 0x7fc974145300 2024-04-06T13:29:03.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.615+0000 7fc95b7fe700 1 -- 172.21.15.189:0/64788097 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc964013cd0 con 0x7fc974145300 2024-04-06T13:29:03.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.615+0000 7fc97922e700 1 -- 172.21.15.189:0/64788097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9741f3730 con 0x7fc974145300 2024-04-06T13:29:03.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.615+0000 7fc95b7fe700 1 -- 172.21.15.189:0/64788097 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc964004270 con 0x7fc974145300 2024-04-06T13:29:03.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.615+0000 7fc97922e700 1 -- 172.21.15.189:0/64788097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9741f2e60 con 0x7fc974145300 2024-04-06T13:29:03.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.616+0000 7fc97922e700 1 -- 172.21.15.189:0/64788097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc974067bb0 con 0x7fc974145300 2024-04-06T13:29:03.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.616+0000 7fc95b7fe700 1 -- 172.21.15.189:0/64788097 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 105) v1 ==== 88856+0+0 (secure 0 0 0) 0x7fc9640043d0 con 0x7fc974145300 2024-04-06T13:29:03.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.617+0000 7fc95b7fe700 1 -- 172.21.15.189:0/64788097 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fc96408a8b0 con 0x7fc974145300 2024-04-06T13:29:03.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.621+0000 7fc95b7fe700 1 -- 172.21.15.189:0/64788097 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc964055c30 con 0x7fc974145300 2024-04-06T13:29:03.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.864+0000 7fc97922e700 1 -- 172.21.15.189:0/64788097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc974074990 con 0x7fc974145300 2024-04-06T13:29:03.866 INFO:tasks.ceph.mon.a.smithi189.stderr:2024-04-06T13:29:03.865+0000 7fa9fe1ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T13:29:03.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.872+0000 7fc95b7fe700 1 -- 172.21.15.189:0/64788097 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v79) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc964059250 con 0x7fc974145300 2024-04-06T13:29:03.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.875+0000 7fc97922e700 1 -- 172.21.15.189:0/64788097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc974145300 msgr2=0x7fc974152f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:03.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.875+0000 7fc97922e700 1 --2- 172.21.15.189:0/64788097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc974145300 0x7fc974152f30 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7fc96400e4b0 tx=0x7fc9640047c0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.875+0000 7fc97922e700 1 -- 172.21.15.189:0/64788097 shutdown_connections 2024-04-06T13:29:03.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.875+0000 7fc97922e700 1 --2- 172.21.15.189:0/64788097 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc974153470 0x7fc97415b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.875+0000 7fc97922e700 1 --2- 172.21.15.189:0/64788097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc95c004850 0x7fc9741f04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.875+0000 7fc97922e700 1 --2- 172.21.15.189:0/64788097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc974145300 0x7fc974152f30 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:03.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.875+0000 7fc97922e700 1 -- 172.21.15.189:0/64788097 >> 172.21.15.189:0/64788097 conn(0x7fc9740d00b0 msgr2=0x7fc9740717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:03.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.875+0000 7fc97922e700 1 -- 172.21.15.189:0/64788097 shutdown_connections 2024-04-06T13:29:03.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:03.875+0000 7fc97922e700 1 -- 172.21.15.189:0/64788097 wait complete. 2024-04-06T13:29:03.877 INFO:teuthology.orchestra.run.smithi189.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-06T13:29:03.877 INFO:tasks.ceph.mds.c.smithi189.stderr: -4995> 2024-04-06T13:28:46.388+0000 7fd24da6c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T13:29:03.890 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-06T13:29:03.908 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T13:29:04.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.015+0000 7f331671a700 1 Processor -- start 2024-04-06T13:29:04.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.015+0000 7f331671a700 1 -- start start 2024-04-06T13:29:04.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.015+0000 7f331671a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f33100cfa00 0x7f33100cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:04.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.015+0000 7f331671a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33100cbd80 con 0x7f33100cb870 2024-04-06T13:29:04.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.015+0000 7f331671a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33100cef20 con 0x7f33100d11b0 2024-04-06T13:29:04.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.015+0000 7f331671a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33100ccdc0 con 0x7f33100cfa00 2024-04-06T13:29:04.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.015+0000 7f3314cb7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f33100cfa00 0x7f33100cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:04.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.016+0000 7f330f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f33100cb870 0x7f331015c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34238/0 (socket says 172.21.15.189:34238) 2024-04-06T13:29:04.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.016+0000 7f3314cb7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f33100cfa00 0x7f33100cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35510/0 (socket says 172.21.15.189:35510) 2024-04-06T13:29:04.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.016+0000 7f330f7fe700 1 -- 172.21.15.189:0/667146299 learned_addr learned my addr 172.21.15.189:0/667146299 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:04.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.016+0000 7f3314cb7700 1 -- 172.21.15.189:0/667146299 >> v1:172.21.15.192:6789/0 conn(0x7f33100d11b0 legacy=0x7f33100d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:04.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.016+0000 7f3314cb7700 1 -- 172.21.15.189:0/667146299 >> v1:172.21.15.189:6789/0 conn(0x7f33100cb870 legacy=0x7f331015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:04.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.016+0000 7f3314cb7700 1 -- 172.21.15.189:0/667146299 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33100cf200 con 0x7f33100cfa00 2024-04-06T13:29:04.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.016+0000 7f330effd700 1 -- 172.21.15.189:0/667146299 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500827703 0 0) 0x7f33100cbd80 con 0x7f33100cb870 2024-04-06T13:29:04.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.016+0000 7f3314cb7700 1 --2- 172.21.15.189:0/667146299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33100cfa00 0x7f33100cd560 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f3308009d90 tx=0x7f330800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ee1e91a47ca820 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:04.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.016+0000 7f330effd700 1 -- 172.21.15.189:0/667146299 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330800e070 con 0x7f33100cfa00 2024-04-06T13:29:04.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.016+0000 7f330effd700 1 -- 172.21.15.189:0/667146299 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330800bb90 con 0x7f33100cfa00 2024-04-06T13:29:04.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.016+0000 7f330effd700 1 -- 172.21.15.189:0/667146299 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3308003a20 con 0x7f33100cfa00 2024-04-06T13:29:04.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.017+0000 7f331671a700 1 -- 172.21.15.189:0/667146299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33100cfa00 msgr2=0x7f33100cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:04.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.017+0000 7f331671a700 1 --2- 172.21.15.189:0/667146299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33100cfa00 0x7f33100cd560 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f3308009d90 tx=0x7f330800b330 comp rx=0 tx=0).stop 2024-04-06T13:29:04.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.017+0000 7f331671a700 1 -- 172.21.15.189:0/667146299 shutdown_connections 2024-04-06T13:29:04.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.017+0000 7f331671a700 1 --2- 172.21.15.189:0/667146299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33100cfa00 0x7f33100cd560 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:04.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.017+0000 7f331671a700 1 -- 172.21.15.189:0/667146299 >> 172.21.15.189:0/667146299 conn(0x7f33100b2db0 msgr2=0x7f33100b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:04.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.017+0000 7f331671a700 1 -- 172.21.15.189:0/667146299 shutdown_connections 2024-04-06T13:29:04.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.017+0000 7f331671a700 1 -- 172.21.15.189:0/667146299 wait complete. 2024-04-06T13:29:04.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.018+0000 7f331671a700 1 Processor -- start 2024-04-06T13:29:04.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.018+0000 7f331671a700 1 -- start start 2024-04-06T13:29:04.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.019+0000 7f331671a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33100cb870 0x7f3310156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:04.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.019+0000 7f331671a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f33100cfa00 0x7f33100c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:04.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.019+0000 7f331671a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f33100d11b0 0x7f3310155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:04.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.019+0000 7f331671a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3310158120 con 0x7f33100cb870 2024-04-06T13:29:04.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.019+0000 7f331671a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33101572b0 con 0x7f33100cfa00 2024-04-06T13:29:04.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.019+0000 7f331671a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33100ca3e0 con 0x7f33100d11b0 2024-04-06T13:29:04.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.019+0000 7f330ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33100cb870 0x7f3310156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:04.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.019+0000 7f330ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33100cb870 0x7f3310156d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35520/0 (socket says 172.21.15.189:35520) 2024-04-06T13:29:04.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.019+0000 7f330ffff700 1 -- 172.21.15.189:0/3641935407 learned_addr learned my addr 172.21.15.189:0/3641935407 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:04.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.019+0000 7f3314cb7700 1 --2- 172.21.15.189:0/3641935407 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f33100d11b0 0x7f3310155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:04.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.019+0000 7f330f7fe700 1 --2- 172.21.15.189:0/3641935407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f33100cfa00 0x7f33100c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:04.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.019+0000 7f330ffff700 1 -- 172.21.15.189:0/3641935407 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f33100d11b0 msgr2=0x7f3310155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:04.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.019+0000 7f330ffff700 1 --2- 172.21.15.189:0/3641935407 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f33100d11b0 0x7f3310155700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:04.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.019+0000 7f330ffff700 1 -- 172.21.15.189:0/3641935407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f33100cfa00 msgr2=0x7f33100c9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:04.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.019+0000 7f330ffff700 1 --2- 172.21.15.189:0/3641935407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f33100cfa00 0x7f33100c9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:04.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.019+0000 7f330ffff700 1 -- 172.21.15.189:0/3641935407 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3308009a70 con 0x7f33100cb870 2024-04-06T13:29:04.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.019+0000 7f330ffff700 1 --2- 172.21.15.189:0/3641935407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33100cb870 0x7f3310156d70 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f330000c4a0 tx=0x7f330000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:04.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.019+0000 7f330cff9700 1 -- 172.21.15.189:0/3641935407 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33000054e0 con 0x7f33100cb870 2024-04-06T13:29:04.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.019+0000 7f330cff9700 1 -- 172.21.15.189:0/3641935407 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3300003680 con 0x7f33100cb870 2024-04-06T13:29:04.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.020+0000 7f331671a700 1 -- 172.21.15.189:0/3641935407 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33101574a0 con 0x7f33100cb870 2024-04-06T13:29:04.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.020+0000 7f330cff9700 1 -- 172.21.15.189:0/3641935407 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3300003a30 con 0x7f33100cb870 2024-04-06T13:29:04.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.020+0000 7f331671a700 1 -- 172.21.15.189:0/3641935407 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3310157a60 con 0x7f33100cb870 2024-04-06T13:29:04.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.020+0000 7f331671a700 1 -- 172.21.15.189:0/3641935407 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3310074990 con 0x7f33100cb870 2024-04-06T13:29:04.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.021+0000 7f330cff9700 1 -- 172.21.15.189:0/3641935407 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 105) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f3300003b90 con 0x7f33100cb870 2024-04-06T13:29:04.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.021+0000 7f330cff9700 1 -- 172.21.15.189:0/3641935407 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6761+0+0 (secure 0 0 0) 0x7f3300088720 con 0x7f33100cb870 2024-04-06T13:29:04.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.026+0000 7f330cff9700 1 -- 172.21.15.189:0/3641935407 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3300053930 con 0x7f33100cb870 2024-04-06T13:29:04.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.207+0000 7f330cff9700 1 -- 172.21.15.189:0/3641935407 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 106) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f33000533e0 con 0x7f33100cb870 2024-04-06T13:29:04.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.208+0000 7f330cff9700 1 --2- 172.21.15.189:0/3641935407 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f32fc036150 0x7f32fc0729b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:04.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.209+0000 7f330f7fe700 1 --2- 172.21.15.189:0/3641935407 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f32fc036150 0x7f32fc0729b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:04.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.210+0000 7f330f7fe700 1 --2- 172.21.15.189:0/3641935407 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f32fc036150 0x7f32fc0729b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3304007fd0 tx=0x7f3304009040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:04.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.268+0000 7f331671a700 1 -- 172.21.15.189:0/3641935407 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f33100b0a00 con 0x7f33100cb870 2024-04-06T13:29:04.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.874+0000 7f330cff9700 1 -- 172.21.15.189:0/3641935407 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v80) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3300072640 con 0x7f33100cb870 2024-04-06T13:29:04.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.876+0000 7f331671a700 1 -- 172.21.15.189:0/3641935407 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f32fc036150 msgr2=0x7f32fc0729b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:04.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.876+0000 7f331671a700 1 --2- 172.21.15.189:0/3641935407 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f32fc036150 0x7f32fc0729b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3304007fd0 tx=0x7f3304009040 comp rx=0 tx=0).stop 2024-04-06T13:29:04.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.876+0000 7f331671a700 1 -- 172.21.15.189:0/3641935407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33100cb870 msgr2=0x7f3310156d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:04.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.876+0000 7f331671a700 1 --2- 172.21.15.189:0/3641935407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33100cb870 0x7f3310156d70 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f330000c4a0 tx=0x7f330000c860 comp rx=0 tx=0).stop 2024-04-06T13:29:04.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.876+0000 7f331671a700 1 -- 172.21.15.189:0/3641935407 shutdown_connections 2024-04-06T13:29:04.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.876+0000 7f331671a700 1 --2- 172.21.15.189:0/3641935407 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f33100d11b0 0x7f3310155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:04.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.876+0000 7f331671a700 1 --2- 172.21.15.189:0/3641935407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33100cb870 0x7f3310156d70 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:04.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.876+0000 7f331671a700 1 --2- 172.21.15.189:0/3641935407 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f32fc036150 0x7f32fc0729b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:04.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.876+0000 7f331671a700 1 --2- 172.21.15.189:0/3641935407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f33100cfa00 0x7f33100c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:04.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.876+0000 7f331671a700 1 -- 172.21.15.189:0/3641935407 >> 172.21.15.189:0/3641935407 conn(0x7f33100b2db0 msgr2=0x7f33100d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:04.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.877+0000 7f331671a700 1 -- 172.21.15.189:0/3641935407 shutdown_connections 2024-04-06T13:29:04.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:04.877+0000 7f331671a700 1 -- 172.21.15.189:0/3641935407 wait complete. 2024-04-06T13:29:04.890 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-06T13:29:05.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.055+0000 7fde8734f700 1 Processor -- start 2024-04-06T13:29:05.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.055+0000 7fde8734f700 1 -- start start 2024-04-06T13:29:05.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.056+0000 7fde8734f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fde8015d7b0 0x7fde800b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:05.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.056+0000 7fde8734f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde800b3990 con 0x7fde8006a730 2024-04-06T13:29:05.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.056+0000 7fde8734f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde800b3060 con 0x7fde8015e500 2024-04-06T13:29:05.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.056+0000 7fde8734f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde8015e950 con 0x7fde8015d7b0 2024-04-06T13:29:05.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.056+0000 7fde848ea700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fde8006a730 0x7fde8015ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34242/0 (socket says 172.21.15.189:34242) 2024-04-06T13:29:05.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.056+0000 7fde848ea700 1 -- 172.21.15.189:0/1926196092 learned_addr learned my addr 172.21.15.189:0/1926196092 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:05.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.057+0000 7fde850eb700 1 --2- 172.21.15.189:0/1926196092 >> v2:172.21.15.192:3300/0 conn(0x7fde8015d7b0 0x7fde800b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:05.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.057+0000 7fde73fff700 1 -- 172.21.15.189:0/1926196092 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716985456 0 0) 0x7fde800b3990 con 0x7fde8006a730 2024-04-06T13:29:05.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.057+0000 7fde73fff700 1 -- 172.21.15.189:0/1926196092 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde6c0035b0 con 0x7fde8006a730 2024-04-06T13:29:05.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.057+0000 7fde850eb700 1 -- 172.21.15.189:0/1926196092 >> v1:172.21.15.192:6789/0 conn(0x7fde8015e500 legacy=0x7fde800717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:05.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.057+0000 7fde850eb700 1 -- 172.21.15.189:0/1926196092 >> v1:172.21.15.189:6789/0 conn(0x7fde8006a730 legacy=0x7fde8015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:05.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.057+0000 7fde73fff700 1 -- 172.21.15.189:0/1926196092 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4030376737 0 0) 0x7fde6c0035b0 con 0x7fde8006a730 2024-04-06T13:29:05.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.057+0000 7fde850eb700 1 -- 172.21.15.189:0/1926196092 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde800b3340 con 0x7fde8015d7b0 2024-04-06T13:29:05.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.057+0000 7fde850eb700 1 --2- 172.21.15.189:0/1926196092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde8015d7b0 0x7fde800b3e20 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fde74009840 tx=0x7fde74009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f69795f9d9941758 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:05.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.057+0000 7fde73fff700 1 -- 172.21.15.189:0/1926196092 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c001670 con 0x7fde8015d7b0 2024-04-06T13:29:05.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.057+0000 7fde73fff700 1 -- 172.21.15.189:0/1926196092 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde74004030 con 0x7fde8015d7b0 2024-04-06T13:29:05.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.057+0000 7fde73fff700 1 -- 172.21.15.189:0/1926196092 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde740043a0 con 0x7fde8015d7b0 2024-04-06T13:29:05.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.058+0000 7fde8734f700 1 -- 172.21.15.189:0/1926196092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde8015d7b0 msgr2=0x7fde800b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:05.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.058+0000 7fde8734f700 1 --2- 172.21.15.189:0/1926196092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde8015d7b0 0x7fde800b3e20 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fde74009840 tx=0x7fde74009b50 comp rx=0 tx=0).stop 2024-04-06T13:29:05.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.058+0000 7fde8734f700 1 -- 172.21.15.189:0/1926196092 shutdown_connections 2024-04-06T13:29:05.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.058+0000 7fde8734f700 1 --2- 172.21.15.189:0/1926196092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde8015d7b0 0x7fde800b3e20 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:05.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.058+0000 7fde8734f700 1 -- 172.21.15.189:0/1926196092 >> 172.21.15.189:0/1926196092 conn(0x7fde800d00b0 msgr2=0x7fde800d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:05.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.058+0000 7fde8734f700 1 -- 172.21.15.189:0/1926196092 shutdown_connections 2024-04-06T13:29:05.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.058+0000 7fde8734f700 1 -- 172.21.15.189:0/1926196092 wait complete. 2024-04-06T13:29:05.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.059+0000 7fde8734f700 1 Processor -- start 2024-04-06T13:29:05.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.059+0000 7fde8734f700 1 -- start start 2024-04-06T13:29:05.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.060+0000 7fde8734f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde8006a730 0x7fde8015cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:05.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.060+0000 7fde8734f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde8015d7b0 0x7fde800cf300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:05.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.060+0000 7fde8734f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde8015e500 0x7fde801f2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:05.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.060+0000 7fde8734f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde801f2c80 con 0x7fde8006a730 2024-04-06T13:29:05.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.060+0000 7fde8734f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8015c7d0 con 0x7fde8015d7b0 2024-04-06T13:29:05.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.060+0000 7fde8734f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde800cde80 con 0x7fde8015e500 2024-04-06T13:29:05.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.060+0000 7fde850eb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde8006a730 0x7fde8015cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:05.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.060+0000 7fde850eb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde8006a730 0x7fde8015cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35530/0 (socket says 172.21.15.189:35530) 2024-04-06T13:29:05.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.060+0000 7fde850eb700 1 -- 172.21.15.189:0/799710097 learned_addr learned my addr 172.21.15.189:0/799710097 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:05.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.060+0000 7fde858ec700 1 --2- 172.21.15.189:0/799710097 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde8015e500 0x7fde801f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:05.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.060+0000 7fde848ea700 1 --2- 172.21.15.189:0/799710097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde8015d7b0 0x7fde800cf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:05.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.060+0000 7fde850eb700 1 -- 172.21.15.189:0/799710097 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde8015e500 msgr2=0x7fde801f2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:05.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.060+0000 7fde850eb700 1 --2- 172.21.15.189:0/799710097 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde8015e500 0x7fde801f2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:05.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.060+0000 7fde850eb700 1 -- 172.21.15.189:0/799710097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde8015d7b0 msgr2=0x7fde800cf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:05.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.060+0000 7fde850eb700 1 --2- 172.21.15.189:0/799710097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde8015d7b0 0x7fde800cf300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:05.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.060+0000 7fde850eb700 1 -- 172.21.15.189:0/799710097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde740094a0 con 0x7fde8006a730 2024-04-06T13:29:05.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.060+0000 7fde848ea700 1 --2- 172.21.15.189:0/799710097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde8015d7b0 0x7fde800cf300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:29:05.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.060+0000 7fde858ec700 1 --2- 172.21.15.189:0/799710097 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde8015e500 0x7fde801f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:29:05.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.060+0000 7fde850eb700 1 --2- 172.21.15.189:0/799710097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde8006a730 0x7fde8015cc60 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fde74000c00 tx=0x7fde74004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:05.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.061+0000 7fde71ffb700 1 -- 172.21.15.189:0/799710097 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74011630 con 0x7fde8006a730 2024-04-06T13:29:05.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.061+0000 7fde71ffb700 1 -- 172.21.15.189:0/799710097 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7401f070 con 0x7fde8006a730 2024-04-06T13:29:05.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.061+0000 7fde71ffb700 1 -- 172.21.15.189:0/799710097 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde740119f0 con 0x7fde8006a730 2024-04-06T13:29:05.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.061+0000 7fde8734f700 1 -- 172.21.15.189:0/799710097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde800cdba0 con 0x7fde8006a730 2024-04-06T13:29:05.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.061+0000 7fde8734f700 1 -- 172.21.15.189:0/799710097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8015c460 con 0x7fde8006a730 2024-04-06T13:29:05.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.062+0000 7fde8734f700 1 -- 172.21.15.189:0/799710097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde80067bb0 con 0x7fde8006a730 2024-04-06T13:29:05.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.062+0000 7fde71ffb700 1 -- 172.21.15.189:0/799710097 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 106) v1 ==== 88983+0+0 (secure 0 0 0) 0x7fde74011b50 con 0x7fde8006a730 2024-04-06T13:29:05.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.063+0000 7fde71ffb700 1 --2- 172.21.15.189:0/799710097 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fde6406aae0 0x7fde6406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:05.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.063+0000 7fde71ffb700 1 -- 172.21.15.189:0/799710097 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6761+0+0 (secure 0 0 0) 0x7fde74021070 con 0x7fde8006a730 2024-04-06T13:29:05.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.063+0000 7fde848ea700 1 --2- 172.21.15.189:0/799710097 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fde6406aae0 0x7fde6406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:05.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.063+0000 7fde848ea700 1 --2- 172.21.15.189:0/799710097 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fde6406aae0 0x7fde6406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fde7c001eb0 tx=0x7fde7c001d00 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:05.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.067+0000 7fde71ffb700 1 -- 172.21.15.189:0/799710097 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fde74055ea0 con 0x7fde8006a730 2024-04-06T13:29:05.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:05.271+0000 7fde8734f700 1 -- 172.21.15.189:0/799710097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fde80074990 con 0x7fde8006a730 2024-04-06T13:29:06.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.220+0000 7fde71ffb700 1 -- 172.21.15.189:0/799710097 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 107) v1 ==== 89191+0+0 (secure 0 0 0) 0x7fde74056270 con 0x7fde8006a730 2024-04-06T13:29:06.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.222+0000 7fde71ffb700 1 -- 172.21.15.189:0/799710097 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v182) v1 ==== 181+0+0 (secure 0 0 0) 0x7fde7405cd60 con 0x7fde8006a730 2024-04-06T13:29:06.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.225+0000 7fde8734f700 1 -- 172.21.15.189:0/799710097 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fde6406aae0 msgr2=0x7fde6406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.225+0000 7fde8734f700 1 --2- 172.21.15.189:0/799710097 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fde6406aae0 0x7fde6406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fde7c001eb0 tx=0x7fde7c001d00 comp rx=0 tx=0).stop 2024-04-06T13:29:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.225+0000 7fde8734f700 1 -- 172.21.15.189:0/799710097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde8006a730 msgr2=0x7fde8015cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.225+0000 7fde8734f700 1 --2- 172.21.15.189:0/799710097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde8006a730 0x7fde8015cc60 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fde74000c00 tx=0x7fde74004750 comp rx=0 tx=0).stop 2024-04-06T13:29:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.225+0000 7fde8734f700 1 -- 172.21.15.189:0/799710097 shutdown_connections 2024-04-06T13:29:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.225+0000 7fde8734f700 1 --2- 172.21.15.189:0/799710097 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde8015e500 0x7fde801f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.225+0000 7fde8734f700 1 --2- 172.21.15.189:0/799710097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde8006a730 0x7fde8015cc60 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.225+0000 7fde8734f700 1 --2- 172.21.15.189:0/799710097 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fde6406aae0 0x7fde6406cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.225+0000 7fde8734f700 1 --2- 172.21.15.189:0/799710097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde8015d7b0 0x7fde800cf300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.225+0000 7fde8734f700 1 -- 172.21.15.189:0/799710097 >> 172.21.15.189:0/799710097 conn(0x7fde800d00b0 msgr2=0x7fde801505a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.225+0000 7fde8734f700 1 -- 172.21.15.189:0/799710097 shutdown_connections 2024-04-06T13:29:06.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.225+0000 7fde8734f700 1 -- 172.21.15.189:0/799710097 wait complete. 2024-04-06T13:29:06.228 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' removed 2024-04-06T13:29:06.241 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-06T13:29:06.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.375+0000 7efd1ce12700 1 Processor -- start 2024-04-06T13:29:06.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.375+0000 7efd1ce12700 1 -- start start 2024-04-06T13:29:06.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1ce12700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efd180b4af0 0x7efd180b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:06.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1ce12700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd180cdbe0 con 0x7efd180b4660 2024-04-06T13:29:06.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1ce12700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd1815a940 con 0x7efd1815a550 2024-04-06T13:29:06.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1ce12700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd1815ac00 con 0x7efd180b4af0 2024-04-06T13:29:06.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd15d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efd180b4660 0x7efd180b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34256/0 (socket says 172.21.15.189:34256) 2024-04-06T13:29:06.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd15d9b700 1 -- 172.21.15.189:0/2005769476 learned_addr learned my addr 172.21.15.189:0/2005769476 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:06.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd16d9d700 1 --2- 172.21.15.189:0/2005769476 >> v2:172.21.15.192:3300/0 conn(0x7efd180b4af0 0x7efd180b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:06.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1559a700 1 -- 172.21.15.189:0/2005769476 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367329468 0 0) 0x7efd180cdbe0 con 0x7efd180b4660 2024-04-06T13:29:06.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1559a700 1 -- 172.21.15.189:0/2005769476 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd000035b0 con 0x7efd180b4660 2024-04-06T13:29:06.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1559a700 1 -- 172.21.15.189:0/2005769476 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2302029555 0 0) 0x7efd000035b0 con 0x7efd180b4660 2024-04-06T13:29:06.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1559a700 1 -- 172.21.15.189:0/2005769476 >> v2:172.21.15.192:3300/0 conn(0x7efd180b4af0 msgr2=0x7efd180b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:06.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1559a700 1 --2- 172.21.15.189:0/2005769476 >> v2:172.21.15.192:3300/0 conn(0x7efd180b4af0 0x7efd180b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:06.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1559a700 1 -- 172.21.15.189:0/2005769476 >> v1:172.21.15.192:6789/0 conn(0x7efd1815a550 legacy=0x7efd180cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:06.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1559a700 1 -- 172.21.15.189:0/2005769476 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1815aee0 con 0x7efd180b4660 2024-04-06T13:29:06.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1559a700 1 -- 172.21.15.189:0/2005769476 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7efd08001670 con 0x7efd180b4660 2024-04-06T13:29:06.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1559a700 1 -- 172.21.15.189:0/2005769476 >> v1:172.21.15.189:6789/0 conn(0x7efd180b4660 legacy=0x7efd180b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:06.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1559a700 1 --2- 172.21.15.189:0/2005769476 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd00003a00 0x7efd00003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:06.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1559a700 1 --2- 172.21.15.189:0/2005769476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd00004850 0x7efd00004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:06.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1559a700 1 --2- 172.21.15.189:0/2005769476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd00005210 0x7efd0000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:06.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1559a700 1 -- 172.21.15.189:0/2005769476 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1815ac00 con 0x7efd00005210 2024-04-06T13:29:06.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1559a700 1 -- 172.21.15.189:0/2005769476 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0000c0b0 con 0x7efd00004850 2024-04-06T13:29:06.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1559a700 1 -- 172.21.15.189:0/2005769476 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd0000c270 con 0x7efd00003a00 2024-04-06T13:29:06.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd16d9d700 1 --2- 172.21.15.189:0/2005769476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd00005210 0x7efd0000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:06.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd15d9b700 1 --2- 172.21.15.189:0/2005769476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd00004850 0x7efd00004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:06.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1659c700 1 --2- 172.21.15.189:0/2005769476 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd00003a00 0x7efd00003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:06.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd16d9d700 1 -- 172.21.15.189:0/2005769476 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd00003a00 msgr2=0x7efd00003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:06.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd16d9d700 1 --2- 172.21.15.189:0/2005769476 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd00003a00 0x7efd00003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:06.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd16d9d700 1 -- 172.21.15.189:0/2005769476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd00004850 msgr2=0x7efd00004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:06.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd16d9d700 1 --2- 172.21.15.189:0/2005769476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd00004850 0x7efd00004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:06.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd16d9d700 1 -- 172.21.15.189:0/2005769476 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd000035b0 con 0x7efd00005210 2024-04-06T13:29:06.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd16d9d700 1 -- 172.21.15.189:0/2005769476 reap_dead start 2024-04-06T13:29:06.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd16d9d700 1 --2- 172.21.15.189:0/2005769476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd00005210 0x7efd0000ba10 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7efd10009fc0 tx=0x7efd00005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f913306c5157567d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:06.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1559a700 1 -- 172.21.15.189:0/2005769476 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd10003a20 con 0x7efd00005210 2024-04-06T13:29:06.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1559a700 1 -- 172.21.15.189:0/2005769476 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd1000f040 con 0x7efd00005210 2024-04-06T13:29:06.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.377+0000 7efd1559a700 1 -- 172.21.15.189:0/2005769476 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd10003de0 con 0x7efd00005210 2024-04-06T13:29:06.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.378+0000 7efd1ce12700 1 -- 172.21.15.189:0/2005769476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd00005210 msgr2=0x7efd0000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:06.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.378+0000 7efd1ce12700 1 --2- 172.21.15.189:0/2005769476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd00005210 0x7efd0000ba10 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7efd10009fc0 tx=0x7efd00005880 comp rx=0 tx=0).stop 2024-04-06T13:29:06.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.378+0000 7efd1ce12700 1 -- 172.21.15.189:0/2005769476 shutdown_connections 2024-04-06T13:29:06.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.378+0000 7efd1ce12700 1 --2- 172.21.15.189:0/2005769476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd00005210 0x7efd0000ba10 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:06.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.378+0000 7efd1ce12700 1 -- 172.21.15.189:0/2005769476 >> 172.21.15.189:0/2005769476 conn(0x7efd180d00b0 msgr2=0x7efd180d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:06.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.378+0000 7efd1ce12700 1 -- 172.21.15.189:0/2005769476 shutdown_connections 2024-04-06T13:29:06.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.378+0000 7efd1ce12700 1 -- 172.21.15.189:0/2005769476 wait complete. 2024-04-06T13:29:06.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.379+0000 7efd1ce12700 1 Processor -- start 2024-04-06T13:29:06.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.380+0000 7efd1ce12700 1 -- start start 2024-04-06T13:29:06.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.380+0000 7efd1ce12700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd00005210 0x7efd18169b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:06.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.380+0000 7efd1ce12700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd180b11c0 0x7efd18141170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:06.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.380+0000 7efd1ce12700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd1814dc80 0x7efd1814e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:06.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.380+0000 7efd1ce12700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1816a070 con 0x7efd00005210 2024-04-06T13:29:06.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.380+0000 7efd1ce12700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1814e640 con 0x7efd180b11c0 2024-04-06T13:29:06.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.380+0000 7efd1ce12700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd181416b0 con 0x7efd1814dc80 2024-04-06T13:29:06.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.380+0000 7efd1659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd00005210 0x7efd18169b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:06.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.380+0000 7efd1659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd00005210 0x7efd18169b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35566/0 (socket says 172.21.15.189:35566) 2024-04-06T13:29:06.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.380+0000 7efd1659c700 1 -- 172.21.15.189:0/1618632824 learned_addr learned my addr 172.21.15.189:0/1618632824 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:06.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.381+0000 7efd15d9b700 1 --2- 172.21.15.189:0/1618632824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd180b11c0 0x7efd18141170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:06.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.381+0000 7efd16d9d700 1 --2- 172.21.15.189:0/1618632824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd1814dc80 0x7efd1814e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:06.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.381+0000 7efd1659c700 1 -- 172.21.15.189:0/1618632824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd1814dc80 msgr2=0x7efd1814e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:06.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.381+0000 7efd1659c700 1 --2- 172.21.15.189:0/1618632824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd1814dc80 0x7efd1814e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:06.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.381+0000 7efd1659c700 1 -- 172.21.15.189:0/1618632824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd180b11c0 msgr2=0x7efd18141170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:06.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.381+0000 7efd1659c700 1 --2- 172.21.15.189:0/1618632824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd180b11c0 0x7efd18141170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:06.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.381+0000 7efd1659c700 1 -- 172.21.15.189:0/1618632824 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd18141980 con 0x7efd00005210 2024-04-06T13:29:06.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.381+0000 7efd15d9b700 1 --2- 172.21.15.189:0/1618632824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd180b11c0 0x7efd18141170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:29:06.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.381+0000 7efd16d9d700 1 --2- 172.21.15.189:0/1618632824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd1814dc80 0x7efd1814e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:29:06.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.381+0000 7efd1659c700 1 --2- 172.21.15.189:0/1618632824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd00005210 0x7efd18169b30 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7efd0c00f820 tx=0x7efd0c00fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:06.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.381+0000 7efd06ffd700 1 -- 172.21.15.189:0/1618632824 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0c0052e0 con 0x7efd00005210 2024-04-06T13:29:06.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.381+0000 7efd06ffd700 1 -- 172.21.15.189:0/1618632824 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd0c005460 con 0x7efd00005210 2024-04-06T13:29:06.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.381+0000 7efd1ce12700 1 -- 172.21.15.189:0/1618632824 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd18141c60 con 0x7efd00005210 2024-04-06T13:29:06.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.381+0000 7efd06ffd700 1 -- 172.21.15.189:0/1618632824 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0c0038c0 con 0x7efd00005210 2024-04-06T13:29:06.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.381+0000 7efd1ce12700 1 -- 172.21.15.189:0/1618632824 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd18142220 con 0x7efd00005210 2024-04-06T13:29:06.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.382+0000 7efd1ce12700 1 -- 172.21.15.189:0/1618632824 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd18067bb0 con 0x7efd00005210 2024-04-06T13:29:06.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.382+0000 7efd06ffd700 1 -- 172.21.15.189:0/1618632824 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 107) v1 ==== 89191+0+0 (secure 0 0 0) 0x7efd0c003a20 con 0x7efd00005210 2024-04-06T13:29:06.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.383+0000 7efd06ffd700 1 --2- 172.21.15.189:0/1618632824 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efcf806ad20 0x7efcf806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:06.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.383+0000 7efd06ffd700 1 -- 172.21.15.189:0/1618632824 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6323+0+0 (secure 0 0 0) 0x7efd0c0894b0 con 0x7efd00005210 2024-04-06T13:29:06.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.383+0000 7efd15d9b700 1 --2- 172.21.15.189:0/1618632824 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efcf806ad20 0x7efcf806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:06.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.383+0000 7efd15d9b700 1 --2- 172.21.15.189:0/1618632824 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efcf806ad20 0x7efcf806d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efd180b16c0 tx=0x7efd08009040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:06.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.387+0000 7efd06ffd700 1 -- 172.21.15.189:0/1618632824 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efd0c054330 con 0x7efd00005210 2024-04-06T13:29:06.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:06.585+0000 7efd1ce12700 1 -- 172.21.15.189:0/1618632824 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efd18074990 con 0x7efd00005210 2024-04-06T13:29:07.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.438+0000 7efd06ffd700 1 -- 172.21.15.189:0/1618632824 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7efd0c003cd0 con 0x7efd00005210 2024-04-06T13:29:07.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.438+0000 7efd06ffd700 1 -- 172.21.15.189:0/1618632824 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v183) v1 ==== 169+0+0 (secure 0 0 0) 0x7efd0c0509a0 con 0x7efd00005210 2024-04-06T13:29:07.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.443+0000 7efd1ce12700 1 -- 172.21.15.189:0/1618632824 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efcf806ad20 msgr2=0x7efcf806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:07.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.443+0000 7efd1ce12700 1 --2- 172.21.15.189:0/1618632824 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efcf806ad20 0x7efcf806d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efd180b16c0 tx=0x7efd08009040 comp rx=0 tx=0).stop 2024-04-06T13:29:07.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.443+0000 7efd1ce12700 1 -- 172.21.15.189:0/1618632824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd00005210 msgr2=0x7efd18169b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:07.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.443+0000 7efd1ce12700 1 --2- 172.21.15.189:0/1618632824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd00005210 0x7efd18169b30 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7efd0c00f820 tx=0x7efd0c00fbe0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.443+0000 7efd1ce12700 1 -- 172.21.15.189:0/1618632824 shutdown_connections 2024-04-06T13:29:07.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.443+0000 7efd1ce12700 1 --2- 172.21.15.189:0/1618632824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd1814dc80 0x7efd1814e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.443+0000 7efd1ce12700 1 --2- 172.21.15.189:0/1618632824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd00005210 0x7efd18169b30 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.443+0000 7efd1ce12700 1 --2- 172.21.15.189:0/1618632824 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efcf806ad20 0x7efcf806d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.443+0000 7efd1ce12700 1 --2- 172.21.15.189:0/1618632824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd180b11c0 0x7efd18141170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.443+0000 7efd1ce12700 1 -- 172.21.15.189:0/1618632824 >> 172.21.15.189:0/1618632824 conn(0x7efd180d00b0 msgr2=0x7efd180b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:07.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.443+0000 7efd1ce12700 1 -- 172.21.15.189:0/1618632824 shutdown_connections 2024-04-06T13:29:07.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.443+0000 7efd1ce12700 1 -- 172.21.15.189:0/1618632824 wait complete. 2024-04-06T13:29:07.444 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' removed 2024-04-06T13:29:07.458 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-06T13:29:07.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.586+0000 7f69762ca700 1 Processor -- start 2024-04-06T13:29:07.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.586+0000 7f69762ca700 1 -- start start 2024-04-06T13:29:07.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.587+0000 7f69762ca700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f697015d7b0 0x7f69700b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:07.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.587+0000 7f69762ca700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f697015e500 0x7f697006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:07.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.587+0000 7f69762ca700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69700b3990 con 0x7f697006a730 2024-04-06T13:29:07.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.587+0000 7f69762ca700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f697006b0f0 con 0x7f697015e500 2024-04-06T13:29:07.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.587+0000 7f69762ca700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69700b3060 con 0x7f697015d7b0 2024-04-06T13:29:07.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.587+0000 7f6974867700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f697015e500 0x7f697006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:07.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.587+0000 7f6974867700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f697015e500 0x7f697006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35580/0 (socket says 172.21.15.189:35580) 2024-04-06T13:29:07.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.587+0000 7f6974867700 1 -- 172.21.15.189:0/2605170647 learned_addr learned my addr 172.21.15.189:0/2605170647 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:07.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.588+0000 7f696ffff700 1 --2- 172.21.15.189:0/2605170647 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f697015d7b0 0x7f69700b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:07.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.588+0000 7f6974867700 1 -- 172.21.15.189:0/2605170647 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f697015d7b0 msgr2=0x7f69700b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:07.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.588+0000 7f6974867700 1 --2- 172.21.15.189:0/2605170647 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f697015d7b0 0x7f69700b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.588+0000 7f6974867700 1 -- 172.21.15.189:0/2605170647 >> v1:172.21.15.192:6789/0 conn(0x7f697006a730 legacy=0x7f697015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:07.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.588+0000 7f6974867700 1 -- 172.21.15.189:0/2605170647 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69700b3340 con 0x7f697015e500 2024-04-06T13:29:07.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.588+0000 7f6974867700 1 --2- 172.21.15.189:0/2605170647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f697015e500 0x7f697006abb0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f6968009e50 tx=0x7f696800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e7044137f577f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:07.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.588+0000 7f696effd700 1 -- 172.21.15.189:0/2605170647 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696800e070 con 0x7f697015e500 2024-04-06T13:29:07.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.588+0000 7f696effd700 1 -- 172.21.15.189:0/2605170647 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6968003a20 con 0x7f697015e500 2024-04-06T13:29:07.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.588+0000 7f696effd700 1 -- 172.21.15.189:0/2605170647 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6968003dd0 con 0x7f697015e500 2024-04-06T13:29:07.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.588+0000 7f69762ca700 1 -- 172.21.15.189:0/2605170647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f697015e500 msgr2=0x7f697006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:07.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.588+0000 7f69762ca700 1 --2- 172.21.15.189:0/2605170647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f697015e500 0x7f697006abb0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f6968009e50 tx=0x7f696800b330 comp rx=0 tx=0).stop 2024-04-06T13:29:07.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.588+0000 7f69762ca700 1 -- 172.21.15.189:0/2605170647 shutdown_connections 2024-04-06T13:29:07.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.588+0000 7f69762ca700 1 --2- 172.21.15.189:0/2605170647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f697015e500 0x7f697006abb0 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.588+0000 7f69762ca700 1 --2- 172.21.15.189:0/2605170647 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f697015d7b0 0x7f69700b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.589+0000 7f69762ca700 1 -- 172.21.15.189:0/2605170647 >> 172.21.15.189:0/2605170647 conn(0x7f69700d00b0 msgr2=0x7f69700d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:07.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.589+0000 7f69762ca700 1 -- 172.21.15.189:0/2605170647 shutdown_connections 2024-04-06T13:29:07.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.589+0000 7f69762ca700 1 -- 172.21.15.189:0/2605170647 wait complete. 2024-04-06T13:29:07.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.590+0000 7f69762ca700 1 Processor -- start 2024-04-06T13:29:07.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.590+0000 7f69762ca700 1 -- start start 2024-04-06T13:29:07.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f69762ca700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f697006a730 0x7f697014fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:07.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f69762ca700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f697015d7b0 0x7f6970153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:07.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f69762ca700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f697015e500 0x7f69700ceb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:07.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f69762ca700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69700cd330 con 0x7f697015e500 2024-04-06T13:29:07.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f69762ca700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697015b160 con 0x7f697015d7b0 2024-04-06T13:29:07.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f69762ca700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697015b440 con 0x7f697006a730 2024-04-06T13:29:07.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f6974867700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f697015e500 0x7f69700ceb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:07.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f6974867700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f697015e500 0x7f69700ceb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35588/0 (socket says 172.21.15.189:35588) 2024-04-06T13:29:07.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f6974867700 1 -- 172.21.15.189:0/1691829783 learned_addr learned my addr 172.21.15.189:0/1691829783 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:07.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f696ffff700 1 --2- 172.21.15.189:0/1691829783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f697006a730 0x7f697014fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:07.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f696f7fe700 1 --2- 172.21.15.189:0/1691829783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f697015d7b0 0x7f6970153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:07.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f6974867700 1 -- 172.21.15.189:0/1691829783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f697006a730 msgr2=0x7f697014fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:07.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f6974867700 1 --2- 172.21.15.189:0/1691829783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f697006a730 0x7f697014fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f6974867700 1 -- 172.21.15.189:0/1691829783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f697015d7b0 msgr2=0x7f6970153d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:07.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f6974867700 1 --2- 172.21.15.189:0/1691829783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f697015d7b0 0x7f6970153d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f6974867700 1 -- 172.21.15.189:0/1691829783 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6968009b30 con 0x7f697015e500 2024-04-06T13:29:07.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f6974867700 1 --2- 172.21.15.189:0/1691829783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f697015e500 0x7f69700ceb40 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f697006c820 tx=0x7f6968004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:07.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f696cff9700 1 -- 172.21.15.189:0/1691829783 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69680092e0 con 0x7f697015e500 2024-04-06T13:29:07.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f696cff9700 1 -- 172.21.15.189:0/1691829783 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69680094d0 con 0x7f697015e500 2024-04-06T13:29:07.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f69762ca700 1 -- 172.21.15.189:0/1691829783 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f697015b720 con 0x7f697015e500 2024-04-06T13:29:07.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.591+0000 7f696cff9700 1 -- 172.21.15.189:0/1691829783 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69680044a0 con 0x7f697015e500 2024-04-06T13:29:07.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.592+0000 7f69762ca700 1 -- 172.21.15.189:0/1691829783 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69700cc6d0 con 0x7f697015e500 2024-04-06T13:29:07.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.592+0000 7f69762ca700 1 -- 172.21.15.189:0/1691829783 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6970067bb0 con 0x7f697015e500 2024-04-06T13:29:07.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.593+0000 7f696cff9700 1 -- 172.21.15.189:0/1691829783 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f6968019070 con 0x7f697015e500 2024-04-06T13:29:07.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.593+0000 7f696cff9700 1 --2- 172.21.15.189:0/1691829783 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f695c06adb0 0x7f695c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:07.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.593+0000 7f696cff9700 1 -- 172.21.15.189:0/1691829783 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5909+0+0 (secure 0 0 0) 0x7f696808a420 con 0x7f697015e500 2024-04-06T13:29:07.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.593+0000 7f696ffff700 1 --2- 172.21.15.189:0/1691829783 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f695c06adb0 0x7f695c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:07.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.594+0000 7f696ffff700 1 --2- 172.21.15.189:0/1691829783 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f695c06adb0 0x7f695c06d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f69701599f0 tx=0x7f6960008040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:07.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.598+0000 7f696cff9700 1 -- 172.21.15.189:0/1691829783 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6968058e30 con 0x7f697015e500 2024-04-06T13:29:07.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.791+0000 7f69762ca700 1 -- 172.21.15.189:0/1691829783 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6970074990 con 0x7f697015e500 2024-04-06T13:29:07.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.795+0000 7f696cff9700 1 -- 172.21.15.189:0/1691829783 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v61) v1 ==== 104+0+0 (secure 0 0 0) 0x7f69680589c0 con 0x7f697015e500 2024-04-06T13:29:07.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.797+0000 7f69762ca700 1 -- 172.21.15.189:0/1691829783 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f695c06adb0 msgr2=0x7f695c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:07.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.797+0000 7f69762ca700 1 --2- 172.21.15.189:0/1691829783 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f695c06adb0 0x7f695c06d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f69701599f0 tx=0x7f6960008040 comp rx=0 tx=0).stop 2024-04-06T13:29:07.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.797+0000 7f69762ca700 1 -- 172.21.15.189:0/1691829783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f697015e500 msgr2=0x7f69700ceb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:07.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.797+0000 7f69762ca700 1 --2- 172.21.15.189:0/1691829783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f697015e500 0x7f69700ceb40 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f697006c820 tx=0x7f6968004ad0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.797+0000 7f69762ca700 1 -- 172.21.15.189:0/1691829783 shutdown_connections 2024-04-06T13:29:07.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.797+0000 7f69762ca700 1 --2- 172.21.15.189:0/1691829783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f697015e500 0x7f69700ceb40 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.797+0000 7f69762ca700 1 --2- 172.21.15.189:0/1691829783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f697006a730 0x7f697014fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.797+0000 7f69762ca700 1 --2- 172.21.15.189:0/1691829783 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f695c06adb0 0x7f695c06d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.797+0000 7f69762ca700 1 --2- 172.21.15.189:0/1691829783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f697015d7b0 0x7f6970153d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.797+0000 7f69762ca700 1 -- 172.21.15.189:0/1691829783 >> 172.21.15.189:0/1691829783 conn(0x7f69700d00b0 msgr2=0x7f697006eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:07.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.798+0000 7f69762ca700 1 -- 172.21.15.189:0/1691829783 shutdown_connections 2024-04-06T13:29:07.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.798+0000 7f69762ca700 1 -- 172.21.15.189:0/1691829783 wait complete. 2024-04-06T13:29:07.809 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-06T13:29:07.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.935+0000 7f48049de700 1 Processor -- start 2024-04-06T13:29:07.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.935+0000 7f48049de700 1 -- start start 2024-04-06T13:29:07.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.936+0000 7f48049de700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f480006ae40 0x7f48000b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:07.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.936+0000 7f48049de700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f480015ddf0 0x7f480015e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:07.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.936+0000 7f48049de700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48000b3990 con 0x7f480015d7b0 2024-04-06T13:29:07.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.936+0000 7f48049de700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48000b3060 con 0x7f480015ddf0 2024-04-06T13:29:07.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.936+0000 7f48049de700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48000b3340 con 0x7f480006ae40 2024-04-06T13:29:07.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.936+0000 7f47fed9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f480015ddf0 0x7f480015e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:07.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.936+0000 7f47fed9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f480015ddf0 0x7f480015e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35598/0 (socket says 172.21.15.189:35598) 2024-04-06T13:29:07.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.936+0000 7f47fed9d700 1 -- 172.21.15.189:0/685794448 learned_addr learned my addr 172.21.15.189:0/685794448 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:07.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.936+0000 7f47fdd9b700 1 --2- 172.21.15.189:0/685794448 >> v2:172.21.15.192:3300/0 conn(0x7f480006ae40 0x7f48000b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:07.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.936+0000 7f47fed9d700 1 -- 172.21.15.189:0/685794448 >> v2:172.21.15.192:3300/0 conn(0x7f480006ae40 msgr2=0x7f48000b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:07.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.936+0000 7f47fed9d700 1 --2- 172.21.15.189:0/685794448 >> v2:172.21.15.192:3300/0 conn(0x7f480006ae40 0x7f48000b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.936+0000 7f47fed9d700 1 -- 172.21.15.189:0/685794448 >> v1:172.21.15.192:6789/0 conn(0x7f480015d7b0 legacy=0x7f480006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:07.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.936+0000 7f47fed9d700 1 -- 172.21.15.189:0/685794448 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48000b3620 con 0x7f480015ddf0 2024-04-06T13:29:07.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.936+0000 7f47fdd9b700 1 --2- 172.21.15.189:0/685794448 >> v2:172.21.15.192:3300/0 conn(0x7f480006ae40 0x7f48000b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:29:07.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.936+0000 7f47fed9d700 1 --2- 172.21.15.189:0/685794448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f480015ddf0 0x7f480015e240 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f48000afc00 tx=0x7f47f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=103701bbd5ca06a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:07.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.937+0000 7f47fd59a700 1 -- 172.21.15.189:0/685794448 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f800e070 con 0x7f480015ddf0 2024-04-06T13:29:07.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.937+0000 7f47fd59a700 1 -- 172.21.15.189:0/685794448 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f8003a20 con 0x7f480015ddf0 2024-04-06T13:29:07.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.937+0000 7f47fd59a700 1 -- 172.21.15.189:0/685794448 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f8003dd0 con 0x7f480015ddf0 2024-04-06T13:29:07.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.937+0000 7f48049de700 1 -- 172.21.15.189:0/685794448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f480015ddf0 msgr2=0x7f480015e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:07.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.937+0000 7f48049de700 1 --2- 172.21.15.189:0/685794448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f480015ddf0 0x7f480015e240 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f48000afc00 tx=0x7f47f800b330 comp rx=0 tx=0).stop 2024-04-06T13:29:07.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.937+0000 7f48049de700 1 -- 172.21.15.189:0/685794448 shutdown_connections 2024-04-06T13:29:07.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.937+0000 7f48049de700 1 --2- 172.21.15.189:0/685794448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f480015ddf0 0x7f480015e240 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.937+0000 7f48049de700 1 --2- 172.21.15.189:0/685794448 >> v2:172.21.15.192:3300/0 conn(0x7f480006ae40 0x7f48000b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.937+0000 7f48049de700 1 -- 172.21.15.189:0/685794448 >> 172.21.15.189:0/685794448 conn(0x7f48000d00b0 msgr2=0x7f48000d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:07.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.937+0000 7f48049de700 1 -- 172.21.15.189:0/685794448 shutdown_connections 2024-04-06T13:29:07.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.937+0000 7f48049de700 1 -- 172.21.15.189:0/685794448 wait complete. 2024-04-06T13:29:07.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.938+0000 7f48049de700 1 Processor -- start 2024-04-06T13:29:07.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.939+0000 7f48049de700 1 -- start start 2024-04-06T13:29:07.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.939+0000 7f48049de700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f480006ae40 0x7f480015b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:07.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.939+0000 7f48049de700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f480015d7b0 0x7f4800146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:07.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.939+0000 7f48049de700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f480015ddf0 0x7f4800154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:07.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.939+0000 7f48049de700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4800147110 con 0x7f480015ddf0 2024-04-06T13:29:07.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.939+0000 7f48049de700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480015a530 con 0x7f480015d7b0 2024-04-06T13:29:07.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.939+0000 7f48049de700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f480015a810 con 0x7f480006ae40 2024-04-06T13:29:07.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.939+0000 7f47fed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f480015ddf0 0x7f4800154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:07.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.939+0000 7f47fed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f480015ddf0 0x7f4800154450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35610/0 (socket says 172.21.15.189:35610) 2024-04-06T13:29:07.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.939+0000 7f47fed9d700 1 -- 172.21.15.189:0/707368909 learned_addr learned my addr 172.21.15.189:0/707368909 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:07.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.940+0000 7f47fed9d700 1 -- 172.21.15.189:0/707368909 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f480006ae40 msgr2=0x7f480015b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:07.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.940+0000 7f47fe59c700 1 --2- 172.21.15.189:0/707368909 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f480006ae40 0x7f480015b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:07.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.940+0000 7f47fdd9b700 1 --2- 172.21.15.189:0/707368909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f480015d7b0 0x7f4800146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:07.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.940+0000 7f47fed9d700 1 --2- 172.21.15.189:0/707368909 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f480006ae40 0x7f480015b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.940+0000 7f47fed9d700 1 -- 172.21.15.189:0/707368909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f480015d7b0 msgr2=0x7f4800146bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:07.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.940+0000 7f47fed9d700 1 --2- 172.21.15.189:0/707368909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f480015d7b0 0x7f4800146bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:07.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.940+0000 7f47fed9d700 1 -- 172.21.15.189:0/707368909 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f8009b30 con 0x7f480015ddf0 2024-04-06T13:29:07.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.940+0000 7f47fed9d700 1 --2- 172.21.15.189:0/707368909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f480015ddf0 0x7f4800154450 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f47f800b370 tx=0x7f47f8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:07.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.940+0000 7f47eeffd700 1 -- 172.21.15.189:0/707368909 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f8004500 con 0x7f480015ddf0 2024-04-06T13:29:07.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.940+0000 7f47eeffd700 1 -- 172.21.15.189:0/707368909 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f8004680 con 0x7f480015ddf0 2024-04-06T13:29:07.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.940+0000 7f48049de700 1 -- 172.21.15.189:0/707368909 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f480015aaf0 con 0x7f480015ddf0 2024-04-06T13:29:07.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.940+0000 7f47eeffd700 1 -- 172.21.15.189:0/707368909 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f8012430 con 0x7f480015ddf0 2024-04-06T13:29:07.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.940+0000 7f48049de700 1 -- 172.21.15.189:0/707368909 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f480015b0b0 con 0x7f480015ddf0 2024-04-06T13:29:07.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.941+0000 7f48049de700 1 -- 172.21.15.189:0/707368909 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4800067bb0 con 0x7f480015ddf0 2024-04-06T13:29:07.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.941+0000 7f47eeffd700 1 -- 172.21.15.189:0/707368909 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f47f8019030 con 0x7f480015ddf0 2024-04-06T13:29:07.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.942+0000 7f47eeffd700 1 --2- 172.21.15.189:0/707368909 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f47e006adb0 0x7f47e006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:07.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.942+0000 7f47eeffd700 1 -- 172.21.15.189:0/707368909 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5909+0+0 (secure 0 0 0) 0x7f47f808ad00 con 0x7f480015ddf0 2024-04-06T13:29:07.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.942+0000 7f47fe59c700 1 --2- 172.21.15.189:0/707368909 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f47e006adb0 0x7f47e006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:07.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.942+0000 7f47fe59c700 1 --2- 172.21.15.189:0/707368909 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f47e006adb0 0x7f47e006d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f47f0003e60 tx=0x7f47f0009040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:07.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:07.946+0000 7f47eeffd700 1 -- 172.21.15.189:0/707368909 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f47f8059710 con 0x7f480015ddf0 2024-04-06T13:29:08.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.140+0000 7f48049de700 1 -- 172.21.15.189:0/707368909 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f4800074990 con 0x7f480015ddf0 2024-04-06T13:29:08.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.143+0000 7f47eeffd700 1 -- 172.21.15.189:0/707368909 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v62) v1 ==== 115+0+0 (secure 0 0 0) 0x7f47f80170b0 con 0x7f480015ddf0 2024-04-06T13:29:08.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.146+0000 7f48049de700 1 -- 172.21.15.189:0/707368909 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f47e006adb0 msgr2=0x7f47e006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:08.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.146+0000 7f48049de700 1 --2- 172.21.15.189:0/707368909 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f47e006adb0 0x7f47e006d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f47f0003e60 tx=0x7f47f0009040 comp rx=0 tx=0).stop 2024-04-06T13:29:08.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.146+0000 7f48049de700 1 -- 172.21.15.189:0/707368909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f480015ddf0 msgr2=0x7f4800154450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:08.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.146+0000 7f48049de700 1 --2- 172.21.15.189:0/707368909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f480015ddf0 0x7f4800154450 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f47f800b370 tx=0x7f47f8004ad0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.146+0000 7f48049de700 1 -- 172.21.15.189:0/707368909 shutdown_connections 2024-04-06T13:29:08.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.146+0000 7f48049de700 1 --2- 172.21.15.189:0/707368909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f480015ddf0 0x7f4800154450 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.146+0000 7f48049de700 1 --2- 172.21.15.189:0/707368909 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f480006ae40 0x7f480015b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.146+0000 7f48049de700 1 --2- 172.21.15.189:0/707368909 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f47e006adb0 0x7f47e006d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.146+0000 7f48049de700 1 --2- 172.21.15.189:0/707368909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f480015d7b0 0x7f4800146bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.146+0000 7f48049de700 1 -- 172.21.15.189:0/707368909 >> 172.21.15.189:0/707368909 conn(0x7f48000d00b0 msgr2=0x7f48000ad360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:08.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.146+0000 7f48049de700 1 -- 172.21.15.189:0/707368909 shutdown_connections 2024-04-06T13:29:08.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.146+0000 7f48049de700 1 -- 172.21.15.189:0/707368909 wait complete. 2024-04-06T13:29:08.158 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-06T13:29:08.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.284+0000 7fa56ff94700 1 Processor -- start 2024-04-06T13:29:08.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.284+0000 7fa56ff94700 1 -- start start 2024-04-06T13:29:08.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.285+0000 7fa56ff94700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa56806ae40 0x7fa5680b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:08.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.285+0000 7fa56ff94700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa56815ddf0 0x7fa56815e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:08.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.285+0000 7fa56ff94700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5680b3990 con 0x7fa56815d7b0 2024-04-06T13:29:08.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.285+0000 7fa56ff94700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5680b3060 con 0x7fa56806ae40 2024-04-06T13:29:08.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.285+0000 7fa56ff94700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5680b3340 con 0x7fa56815ddf0 2024-04-06T13:29:08.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.285+0000 7fa56d52f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa56806ae40 0x7fa5680b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:08.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.285+0000 7fa56dd30700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa56815d7b0 0x7fa56806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34258/0 (socket says 172.21.15.189:34258) 2024-04-06T13:29:08.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.285+0000 7fa56dd30700 1 -- 172.21.15.189:0/584980842 learned_addr learned my addr 172.21.15.189:0/584980842 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:08.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.285+0000 7fa56e531700 1 --2- 172.21.15.189:0/584980842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa56815ddf0 0x7fa56815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:08.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.285+0000 7fa56d52f700 1 -- 172.21.15.189:0/584980842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa56815ddf0 msgr2=0x7fa56815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:08.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.285+0000 7fa56d52f700 1 --2- 172.21.15.189:0/584980842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa56815ddf0 0x7fa56815e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.285+0000 7fa56d52f700 1 -- 172.21.15.189:0/584980842 >> v1:172.21.15.189:6789/0 conn(0x7fa56815d7b0 legacy=0x7fa56806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:08.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.285+0000 7fa56d52f700 1 -- 172.21.15.189:0/584980842 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5680b3620 con 0x7fa56806ae40 2024-04-06T13:29:08.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.285+0000 7fa56cd2e700 1 -- 172.21.15.189:0/584980842 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2616977489 0 0) 0x7fa5680b3990 con 0x7fa56815d7b0 2024-04-06T13:29:08.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.285+0000 7fa56e531700 1 --2- 172.21.15.189:0/584980842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa56815ddf0 0x7fa56815e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:29:08.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.285+0000 7fa56d52f700 1 --2- 172.21.15.189:0/584980842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56806ae40 0x7fa5680b3e20 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fa558009840 tx=0x7fa558009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ad769697d35d22e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:08.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.285+0000 7fa56cd2e700 1 -- 172.21.15.189:0/584980842 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa558011830 con 0x7fa56806ae40 2024-04-06T13:29:08.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.285+0000 7fa56cd2e700 1 -- 172.21.15.189:0/584980842 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa558011990 con 0x7fa56806ae40 2024-04-06T13:29:08.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.285+0000 7fa56cd2e700 1 -- 172.21.15.189:0/584980842 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa558011d20 con 0x7fa56806ae40 2024-04-06T13:29:08.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.286+0000 7fa56ff94700 1 -- 172.21.15.189:0/584980842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56806ae40 msgr2=0x7fa5680b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:08.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.286+0000 7fa56ff94700 1 --2- 172.21.15.189:0/584980842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56806ae40 0x7fa5680b3e20 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fa558009840 tx=0x7fa558009b50 comp rx=0 tx=0).stop 2024-04-06T13:29:08.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.286+0000 7fa56ff94700 1 -- 172.21.15.189:0/584980842 shutdown_connections 2024-04-06T13:29:08.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.286+0000 7fa56ff94700 1 --2- 172.21.15.189:0/584980842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa56815ddf0 0x7fa56815e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.286+0000 7fa56ff94700 1 --2- 172.21.15.189:0/584980842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56806ae40 0x7fa5680b3e20 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.286+0000 7fa56ff94700 1 -- 172.21.15.189:0/584980842 >> 172.21.15.189:0/584980842 conn(0x7fa5680d00b0 msgr2=0x7fa5680d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:08.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.286+0000 7fa56ff94700 1 -- 172.21.15.189:0/584980842 shutdown_connections 2024-04-06T13:29:08.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.286+0000 7fa56ff94700 1 -- 172.21.15.189:0/584980842 wait complete. 2024-04-06T13:29:08.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.287+0000 7fa56ff94700 1 Processor -- start 2024-04-06T13:29:08.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.287+0000 7fa56ff94700 1 -- start start 2024-04-06T13:29:08.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.288+0000 7fa56ff94700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa56806ae40 0x7fa568149b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:08.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.288+0000 7fa56ff94700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa56815d7b0 0x7fa568155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:08.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.288+0000 7fa56ff94700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56815ddf0 0x7fa56815b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:08.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.288+0000 7fa56ff94700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5681544c0 con 0x7fa56815ddf0 2024-04-06T13:29:08.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.288+0000 7fa56ff94700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56814a0b0 con 0x7fa56815d7b0 2024-04-06T13:29:08.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.288+0000 7fa56ff94700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5681567b0 con 0x7fa56806ae40 2024-04-06T13:29:08.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.288+0000 7fa56e531700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56815ddf0 0x7fa56815b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:08.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.288+0000 7fa56e531700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56815ddf0 0x7fa56815b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35634/0 (socket says 172.21.15.189:35634) 2024-04-06T13:29:08.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.288+0000 7fa56e531700 1 -- 172.21.15.189:0/3591974209 learned_addr learned my addr 172.21.15.189:0/3591974209 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:08.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.288+0000 7fa56dd30700 1 --2- 172.21.15.189:0/3591974209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa56806ae40 0x7fa568149b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:08.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.288+0000 7fa56d52f700 1 --2- 172.21.15.189:0/3591974209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa56815d7b0 0x7fa568155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:08.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.288+0000 7fa56e531700 1 -- 172.21.15.189:0/3591974209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa56806ae40 msgr2=0x7fa568149b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:08.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.288+0000 7fa56e531700 1 --2- 172.21.15.189:0/3591974209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa56806ae40 0x7fa568149b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.288+0000 7fa56e531700 1 -- 172.21.15.189:0/3591974209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa56815d7b0 msgr2=0x7fa568155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:08.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.288+0000 7fa56e531700 1 --2- 172.21.15.189:0/3591974209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa56815d7b0 0x7fa568155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.288+0000 7fa56e531700 1 -- 172.21.15.189:0/3591974209 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5580094a0 con 0x7fa56815ddf0 2024-04-06T13:29:08.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.288+0000 7fa56e531700 1 --2- 172.21.15.189:0/3591974209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56815ddf0 0x7fa56815b610 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fa56000cd50 tx=0x7fa56000a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:08.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.289+0000 7fa55e7fc700 1 -- 172.21.15.189:0/3591974209 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56000f480 con 0x7fa56815ddf0 2024-04-06T13:29:08.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.289+0000 7fa55e7fc700 1 -- 172.21.15.189:0/3591974209 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56000f600 con 0x7fa56815ddf0 2024-04-06T13:29:08.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.289+0000 7fa55e7fc700 1 -- 172.21.15.189:0/3591974209 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56000f9b0 con 0x7fa56815ddf0 2024-04-06T13:29:08.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.289+0000 7fa56ff94700 1 -- 172.21.15.189:0/3591974209 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa568156a90 con 0x7fa56815ddf0 2024-04-06T13:29:08.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.289+0000 7fa56ff94700 1 -- 172.21.15.189:0/3591974209 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa568157050 con 0x7fa56815ddf0 2024-04-06T13:29:08.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.290+0000 7fa56ff94700 1 -- 172.21.15.189:0/3591974209 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa568154600 con 0x7fa56815ddf0 2024-04-06T13:29:08.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.290+0000 7fa55e7fc700 1 -- 172.21.15.189:0/3591974209 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fa56000b040 con 0x7fa56815ddf0 2024-04-06T13:29:08.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.290+0000 7fa55e7fc700 1 --2- 172.21.15.189:0/3591974209 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa54c06adb0 0x7fa54c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:08.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.290+0000 7fa55e7fc700 1 -- 172.21.15.189:0/3591974209 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5909+0+0 (secure 0 0 0) 0x7fa560089e20 con 0x7fa56815ddf0 2024-04-06T13:29:08.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.290+0000 7fa56dd30700 1 --2- 172.21.15.189:0/3591974209 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa54c06adb0 0x7fa54c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:08.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.291+0000 7fa56dd30700 1 --2- 172.21.15.189:0/3591974209 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa54c06adb0 0x7fa54c06d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa564000f60 tx=0x7fa564007be0 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:08.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.295+0000 7fa55e7fc700 1 -- 172.21.15.189:0/3591974209 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa568154600 con 0x7fa56815ddf0 2024-04-06T13:29:08.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.490+0000 7fa56ff94700 1 -- 172.21.15.189:0/3591974209 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fa568154600 con 0x7fa56815ddf0 2024-04-06T13:29:08.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.494+0000 7fa55e7fc700 1 -- 172.21.15.189:0/3591974209 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v63) v1 ==== 120+0+0 (secure 0 0 0) 0x7fa568154600 con 0x7fa56815ddf0 2024-04-06T13:29:08.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.496+0000 7fa56ff94700 1 -- 172.21.15.189:0/3591974209 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa54c06adb0 msgr2=0x7fa54c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:08.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.496+0000 7fa56ff94700 1 --2- 172.21.15.189:0/3591974209 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa54c06adb0 0x7fa54c06d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa564000f60 tx=0x7fa564007be0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.496+0000 7fa56ff94700 1 -- 172.21.15.189:0/3591974209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56815ddf0 msgr2=0x7fa56815b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:08.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.496+0000 7fa56ff94700 1 --2- 172.21.15.189:0/3591974209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56815ddf0 0x7fa56815b610 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fa56000cd50 tx=0x7fa56000a560 comp rx=0 tx=0).stop 2024-04-06T13:29:08.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.496+0000 7fa56ff94700 1 -- 172.21.15.189:0/3591974209 shutdown_connections 2024-04-06T13:29:08.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.496+0000 7fa56ff94700 1 --2- 172.21.15.189:0/3591974209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56815ddf0 0x7fa56815b610 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.496+0000 7fa56ff94700 1 --2- 172.21.15.189:0/3591974209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa56806ae40 0x7fa568149b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.496+0000 7fa56ff94700 1 --2- 172.21.15.189:0/3591974209 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa54c06adb0 0x7fa54c06d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.496+0000 7fa56ff94700 1 --2- 172.21.15.189:0/3591974209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa56815d7b0 0x7fa568155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.496+0000 7fa56ff94700 1 -- 172.21.15.189:0/3591974209 >> 172.21.15.189:0/3591974209 conn(0x7fa5680d00b0 msgr2=0x7fa568071cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:08.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.496+0000 7fa56ff94700 1 -- 172.21.15.189:0/3591974209 shutdown_connections 2024-04-06T13:29:08.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.496+0000 7fa56ff94700 1 -- 172.21.15.189:0/3591974209 wait complete. 2024-04-06T13:29:08.509 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2024-04-06T13:29:08.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.630+0000 7faa201ee700 1 Processor -- start 2024-04-06T13:29:08.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.630+0000 7faa201ee700 1 -- start start 2024-04-06T13:29:08.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.631+0000 7faa201ee700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7faa1806ae40 0x7faa180b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:08.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.631+0000 7faa201ee700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faa1815ddf0 0x7faa1815e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:08.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.631+0000 7faa201ee700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa180b3990 con 0x7faa1815d7b0 2024-04-06T13:29:08.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.631+0000 7faa201ee700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa180b3060 con 0x7faa1815ddf0 2024-04-06T13:29:08.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.631+0000 7faa201ee700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa180b3340 con 0x7faa1806ae40 2024-04-06T13:29:08.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.631+0000 7faa1e78b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faa1815ddf0 0x7faa1815e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:08.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.631+0000 7faa1e78b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faa1815ddf0 0x7faa1815e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35638/0 (socket says 172.21.15.189:35638) 2024-04-06T13:29:08.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.631+0000 7faa1e78b700 1 -- 172.21.15.189:0/2303497704 learned_addr learned my addr 172.21.15.189:0/2303497704 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:08.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.631+0000 7faa1d789700 1 --2- 172.21.15.189:0/2303497704 >> v2:172.21.15.192:3300/0 conn(0x7faa1806ae40 0x7faa180b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:08.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.631+0000 7faa1e78b700 1 -- 172.21.15.189:0/2303497704 >> v2:172.21.15.192:3300/0 conn(0x7faa1806ae40 msgr2=0x7faa180b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:08.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.631+0000 7faa1e78b700 1 --2- 172.21.15.189:0/2303497704 >> v2:172.21.15.192:3300/0 conn(0x7faa1806ae40 0x7faa180b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.631+0000 7faa1e78b700 1 -- 172.21.15.189:0/2303497704 >> v1:172.21.15.192:6789/0 conn(0x7faa1815d7b0 legacy=0x7faa1806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:08.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.631+0000 7faa1e78b700 1 -- 172.21.15.189:0/2303497704 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa180b3620 con 0x7faa1815ddf0 2024-04-06T13:29:08.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.632+0000 7faa1e78b700 1 --2- 172.21.15.189:0/2303497704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa1815ddf0 0x7faa1815e240 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7faa180afc00 tx=0x7faa1000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a2cbd7af60e9e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:08.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.632+0000 7faa1cf88700 1 -- 172.21.15.189:0/2303497704 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1000e070 con 0x7faa1815ddf0 2024-04-06T13:29:08.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.632+0000 7faa1cf88700 1 -- 172.21.15.189:0/2303497704 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa10003a20 con 0x7faa1815ddf0 2024-04-06T13:29:08.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.632+0000 7faa1cf88700 1 -- 172.21.15.189:0/2303497704 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa10003dd0 con 0x7faa1815ddf0 2024-04-06T13:29:08.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.632+0000 7faa201ee700 1 -- 172.21.15.189:0/2303497704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa1815ddf0 msgr2=0x7faa1815e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:08.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.632+0000 7faa201ee700 1 --2- 172.21.15.189:0/2303497704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa1815ddf0 0x7faa1815e240 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7faa180afc00 tx=0x7faa1000b330 comp rx=0 tx=0).stop 2024-04-06T13:29:08.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.632+0000 7faa201ee700 1 -- 172.21.15.189:0/2303497704 shutdown_connections 2024-04-06T13:29:08.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.632+0000 7faa201ee700 1 --2- 172.21.15.189:0/2303497704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa1815ddf0 0x7faa1815e240 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.632+0000 7faa201ee700 1 --2- 172.21.15.189:0/2303497704 >> v2:172.21.15.192:3300/0 conn(0x7faa1806ae40 0x7faa180b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.632+0000 7faa201ee700 1 -- 172.21.15.189:0/2303497704 >> 172.21.15.189:0/2303497704 conn(0x7faa180d00b0 msgr2=0x7faa180d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:08.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.632+0000 7faa201ee700 1 -- 172.21.15.189:0/2303497704 shutdown_connections 2024-04-06T13:29:08.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.632+0000 7faa201ee700 1 -- 172.21.15.189:0/2303497704 wait complete. 2024-04-06T13:29:08.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.634+0000 7faa201ee700 1 Processor -- start 2024-04-06T13:29:08.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.634+0000 7faa201ee700 1 -- start start 2024-04-06T13:29:08.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.634+0000 7faa201ee700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa1806ae40 0x7faa18157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:08.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.634+0000 7faa201ee700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa1815d7b0 0x7faa18153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:08.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.634+0000 7faa201ee700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa1815ddf0 0x7faa180cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:08.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.634+0000 7faa201ee700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa180ccf50 con 0x7faa1815d7b0 2024-04-06T13:29:08.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.634+0000 7faa201ee700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa18160210 con 0x7faa1806ae40 2024-04-06T13:29:08.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.634+0000 7faa201ee700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa180cd1a0 con 0x7faa1815ddf0 2024-04-06T13:29:08.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.634+0000 7faa1d789700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa1815d7b0 0x7faa18153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:08.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.634+0000 7faa1d789700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa1815d7b0 0x7faa18153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35644/0 (socket says 172.21.15.189:35644) 2024-04-06T13:29:08.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.634+0000 7faa1d789700 1 -- 172.21.15.189:0/171412633 learned_addr learned my addr 172.21.15.189:0/171412633 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:08.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.635+0000 7faa1df8a700 1 --2- 172.21.15.189:0/171412633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa1806ae40 0x7faa18157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:08.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.635+0000 7faa1e78b700 1 --2- 172.21.15.189:0/171412633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa1815ddf0 0x7faa180cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:08.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.635+0000 7faa1d789700 1 -- 172.21.15.189:0/171412633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa1815ddf0 msgr2=0x7faa180cdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:08.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.635+0000 7faa1d789700 1 --2- 172.21.15.189:0/171412633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa1815ddf0 0x7faa180cdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.635+0000 7faa1d789700 1 -- 172.21.15.189:0/171412633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa1806ae40 msgr2=0x7faa18157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:08.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.635+0000 7faa1d789700 1 --2- 172.21.15.189:0/171412633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa1806ae40 0x7faa18157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:08.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.635+0000 7faa1d789700 1 -- 172.21.15.189:0/171412633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa10009b30 con 0x7faa1815d7b0 2024-04-06T13:29:08.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.635+0000 7faa1d789700 1 --2- 172.21.15.189:0/171412633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa1815d7b0 0x7faa18153950 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7faa1400c7f0 tx=0x7faa1400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:08.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.635+0000 7faa0e7fc700 1 -- 172.21.15.189:0/171412633 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa14004220 con 0x7faa1815d7b0 2024-04-06T13:29:08.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.635+0000 7faa0e7fc700 1 -- 172.21.15.189:0/171412633 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa14004380 con 0x7faa1815d7b0 2024-04-06T13:29:08.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.635+0000 7faa201ee700 1 -- 172.21.15.189:0/171412633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa180cd480 con 0x7faa1815d7b0 2024-04-06T13:29:08.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.635+0000 7faa0e7fc700 1 -- 172.21.15.189:0/171412633 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa140036a0 con 0x7faa1815d7b0 2024-04-06T13:29:08.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.635+0000 7faa201ee700 1 -- 172.21.15.189:0/171412633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa181540a0 con 0x7faa1815d7b0 2024-04-06T13:29:08.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.636+0000 7faa201ee700 1 -- 172.21.15.189:0/171412633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa18067bb0 con 0x7faa1815d7b0 2024-04-06T13:29:08.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.636+0000 7faa0e7fc700 1 -- 172.21.15.189:0/171412633 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7faa14003800 con 0x7faa1815d7b0 2024-04-06T13:29:08.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.637+0000 7faa0e7fc700 1 --2- 172.21.15.189:0/171412633 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa9fc06adb0 0x7fa9fc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:08.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.637+0000 7faa0e7fc700 1 -- 172.21.15.189:0/171412633 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5909+0+0 (secure 0 0 0) 0x7faa14089260 con 0x7faa1815d7b0 2024-04-06T13:29:08.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.637+0000 7faa1df8a700 1 --2- 172.21.15.189:0/171412633 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa9fc06adb0 0x7fa9fc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:08.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.637+0000 7faa1df8a700 1 --2- 172.21.15.189:0/171412633 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa9fc06adb0 0x7fa9fc06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7faa08001730 tx=0x7faa08009040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:08.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.641+0000 7faa0e7fc700 1 -- 172.21.15.189:0/171412633 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7faa14054160 con 0x7faa1815d7b0 2024-04-06T13:29:08.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:08.917+0000 7faa201ee700 1 -- 172.21.15.189:0/171412633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7faa18074990 con 0x7faa1815d7b0 2024-04-06T13:29:09.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.425+0000 7faa0e7fc700 1 -- 172.21.15.189:0/171412633 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2518) v1 ==== 174+0+0 (secure 0 0 0) 0x7faa14057780 con 0x7faa1815d7b0 2024-04-06T13:29:09.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.427+0000 7faa201ee700 1 -- 172.21.15.189:0/171412633 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa9fc06adb0 msgr2=0x7fa9fc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:09.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.427+0000 7faa201ee700 1 --2- 172.21.15.189:0/171412633 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa9fc06adb0 0x7fa9fc06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7faa08001730 tx=0x7faa08009040 comp rx=0 tx=0).stop 2024-04-06T13:29:09.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.427+0000 7faa201ee700 1 -- 172.21.15.189:0/171412633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa1815d7b0 msgr2=0x7faa18153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:09.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.427+0000 7faa201ee700 1 --2- 172.21.15.189:0/171412633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa1815d7b0 0x7faa18153950 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7faa1400c7f0 tx=0x7faa1400cbb0 comp rx=0 tx=0).stop 2024-04-06T13:29:09.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.427+0000 7faa201ee700 1 -- 172.21.15.189:0/171412633 shutdown_connections 2024-04-06T13:29:09.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.427+0000 7faa201ee700 1 --2- 172.21.15.189:0/171412633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa1815ddf0 0x7faa180cdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:09.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.427+0000 7faa201ee700 1 --2- 172.21.15.189:0/171412633 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa9fc06adb0 0x7fa9fc06d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:09.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.427+0000 7faa201ee700 1 --2- 172.21.15.189:0/171412633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa1806ae40 0x7faa18157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:09.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.427+0000 7faa201ee700 1 --2- 172.21.15.189:0/171412633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa1815d7b0 0x7faa18153950 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:09.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.427+0000 7faa201ee700 1 -- 172.21.15.189:0/171412633 >> 172.21.15.189:0/171412633 conn(0x7faa180d00b0 msgr2=0x7faa18072400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:09.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.428+0000 7faa201ee700 1 -- 172.21.15.189:0/171412633 shutdown_connections 2024-04-06T13:29:09.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.428+0000 7faa201ee700 1 -- 172.21.15.189:0/171412633 wait complete. 2024-04-06T13:29:09.441 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2024-04-06T13:29:09.441 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2024-04-06T13:29:09.441 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-06T13:29:09.441 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2024-04-06T13:29:09.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.566+0000 7f2ca5801700 1 Processor -- start 2024-04-06T13:29:09.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.566+0000 7f2ca5801700 1 -- start start 2024-04-06T13:29:09.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.567+0000 7f2ca5801700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2ca015b310 0x7f2ca00b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:09.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.567+0000 7f2ca5801700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2ca00d1a50 0x7f2ca015ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:09.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.567+0000 7f2ca5801700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca00cb910 con 0x7f2ca00cdfd0 2024-04-06T13:29:09.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.567+0000 7f2ca5801700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ca00cbb80 con 0x7f2ca00d1a50 2024-04-06T13:29:09.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.567+0000 7f2ca5801700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ca00cbe60 con 0x7f2ca015b310 2024-04-06T13:29:09.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.567+0000 7f2c9f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2ca00d1a50 0x7f2ca015ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:09.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.567+0000 7f2c9f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2ca00d1a50 0x7f2ca015ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49774/0 (socket says 172.21.15.189:49774) 2024-04-06T13:29:09.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.567+0000 7f2c9f7fe700 1 -- 172.21.15.189:0/1192400977 learned_addr learned my addr 172.21.15.189:0/1192400977 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:09.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.567+0000 7f2c9effd700 1 --2- 172.21.15.189:0/1192400977 >> v2:172.21.15.192:3300/0 conn(0x7f2ca015b310 0x7f2ca00b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:09.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.567+0000 7f2c9f7fe700 1 -- 172.21.15.189:0/1192400977 >> v2:172.21.15.192:3300/0 conn(0x7f2ca015b310 msgr2=0x7f2ca00b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:09.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.567+0000 7f2c9f7fe700 1 --2- 172.21.15.189:0/1192400977 >> v2:172.21.15.192:3300/0 conn(0x7f2ca015b310 0x7f2ca00b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:09.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.567+0000 7f2c9f7fe700 1 -- 172.21.15.189:0/1192400977 >> v1:172.21.15.192:6789/0 conn(0x7f2ca00cdfd0 legacy=0x7f2ca00d1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:09.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.567+0000 7f2c9f7fe700 1 -- 172.21.15.189:0/1192400977 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca00d0b40 con 0x7f2ca00d1a50 2024-04-06T13:29:09.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.567+0000 7f2c9effd700 1 --2- 172.21.15.189:0/1192400977 >> v2:172.21.15.192:3300/0 conn(0x7f2ca015b310 0x7f2ca00b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:29:09.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.568+0000 7f2c9f7fe700 1 --2- 172.21.15.189:0/1192400977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ca00d1a50 0x7f2ca015ef10 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f2ca00d2290 tx=0x7f2c9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=995d4b1492659a48 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:09.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.568+0000 7f2c9dffb700 1 -- 172.21.15.189:0/1192400977 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9800e070 con 0x7f2ca00d1a50 2024-04-06T13:29:09.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.568+0000 7f2c9dffb700 1 -- 172.21.15.189:0/1192400977 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c98003a20 con 0x7f2ca00d1a50 2024-04-06T13:29:09.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.568+0000 7f2c9dffb700 1 -- 172.21.15.189:0/1192400977 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c98003dd0 con 0x7f2ca00d1a50 2024-04-06T13:29:09.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.568+0000 7f2ca5801700 1 -- 172.21.15.189:0/1192400977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ca00d1a50 msgr2=0x7f2ca015ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:09.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.568+0000 7f2ca5801700 1 --2- 172.21.15.189:0/1192400977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ca00d1a50 0x7f2ca015ef10 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f2ca00d2290 tx=0x7f2c9800b330 comp rx=0 tx=0).stop 2024-04-06T13:29:09.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.568+0000 7f2ca5801700 1 -- 172.21.15.189:0/1192400977 shutdown_connections 2024-04-06T13:29:09.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.568+0000 7f2ca5801700 1 --2- 172.21.15.189:0/1192400977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ca00d1a50 0x7f2ca015ef10 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:09.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.568+0000 7f2ca5801700 1 --2- 172.21.15.189:0/1192400977 >> v2:172.21.15.192:3300/0 conn(0x7f2ca015b310 0x7f2ca00b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:09.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.568+0000 7f2ca5801700 1 -- 172.21.15.189:0/1192400977 >> 172.21.15.189:0/1192400977 conn(0x7f2ca006a730 msgr2=0x7f2ca006ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:09.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.568+0000 7f2ca5801700 1 -- 172.21.15.189:0/1192400977 shutdown_connections 2024-04-06T13:29:09.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.568+0000 7f2ca5801700 1 -- 172.21.15.189:0/1192400977 wait complete. 2024-04-06T13:29:09.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.570+0000 7f2ca5801700 1 Processor -- start 2024-04-06T13:29:09.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.570+0000 7f2ca5801700 1 -- start start 2024-04-06T13:29:09.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.570+0000 7f2ca5801700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ca00cdfd0 0x7f2ca01f06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:09.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.570+0000 7f2ca5801700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ca00d1a50 0x7f2ca01f4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:09.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.570+0000 7f2ca5801700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ca015b310 0x7f2ca015ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:09.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.570+0000 7f2ca5801700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca015d9a0 con 0x7f2ca00d1a50 2024-04-06T13:29:09.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.570+0000 7f2ca5801700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca01f5170 con 0x7f2ca015b310 2024-04-06T13:29:09.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.570+0000 7f2ca5801700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca015cd90 con 0x7f2ca00cdfd0 2024-04-06T13:29:09.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.571+0000 7f2c9e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ca00d1a50 0x7f2ca01f4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:09.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.571+0000 7f2c9e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ca00d1a50 0x7f2ca01f4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49790/0 (socket says 172.21.15.189:49790) 2024-04-06T13:29:09.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.571+0000 7f2c9e7fc700 1 -- 172.21.15.189:0/2314294972 learned_addr learned my addr 172.21.15.189:0/2314294972 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:09.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.571+0000 7f2c9effd700 1 --2- 172.21.15.189:0/2314294972 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ca00cdfd0 0x7f2ca01f06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:09.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.571+0000 7f2c9f7fe700 1 --2- 172.21.15.189:0/2314294972 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ca015b310 0x7f2ca015ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:09.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.571+0000 7f2c9e7fc700 1 -- 172.21.15.189:0/2314294972 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ca00cdfd0 msgr2=0x7f2ca01f06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:09.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.571+0000 7f2c9e7fc700 1 --2- 172.21.15.189:0/2314294972 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ca00cdfd0 0x7f2ca01f06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:09.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.571+0000 7f2c9e7fc700 1 -- 172.21.15.189:0/2314294972 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ca015b310 msgr2=0x7f2ca015ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:09.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.571+0000 7f2c9e7fc700 1 --2- 172.21.15.189:0/2314294972 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ca015b310 0x7f2ca015ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:09.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.571+0000 7f2c9e7fc700 1 -- 172.21.15.189:0/2314294972 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c98009b30 con 0x7f2ca00d1a50 2024-04-06T13:29:09.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.571+0000 7f2c9e7fc700 1 --2- 172.21.15.189:0/2314294972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ca00d1a50 0x7f2ca01f4c30 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f2c9000c4b0 tx=0x7f2c9000c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:09.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.571+0000 7f2c877fe700 1 -- 172.21.15.189:0/2314294972 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c90011790 con 0x7f2ca00d1a50 2024-04-06T13:29:09.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.571+0000 7f2c877fe700 1 -- 172.21.15.189:0/2314294972 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c900118f0 con 0x7f2ca00d1a50 2024-04-06T13:29:09.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.571+0000 7f2ca5801700 1 -- 172.21.15.189:0/2314294972 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca015d070 con 0x7f2ca00d1a50 2024-04-06T13:29:09.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.571+0000 7f2c877fe700 1 -- 172.21.15.189:0/2314294972 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c90011ca0 con 0x7f2ca00d1a50 2024-04-06T13:29:09.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.571+0000 7f2ca5801700 1 -- 172.21.15.189:0/2314294972 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca015d630 con 0x7f2ca00d1a50 2024-04-06T13:29:09.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.572+0000 7f2ca5801700 1 -- 172.21.15.189:0/2314294972 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca015d7c0 con 0x7f2ca00d1a50 2024-04-06T13:29:09.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.573+0000 7f2c877fe700 1 -- 172.21.15.189:0/2314294972 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f2c9001c460 con 0x7f2ca00d1a50 2024-04-06T13:29:09.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.573+0000 7f2c877fe700 1 --2- 172.21.15.189:0/2314294972 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2c8c06adb0 0x7f2c8c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:09.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.573+0000 7f2c877fe700 1 -- 172.21.15.189:0/2314294972 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5909+0+0 (secure 0 0 0) 0x7f2c90089330 con 0x7f2ca00d1a50 2024-04-06T13:29:09.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.573+0000 7f2c9effd700 1 --2- 172.21.15.189:0/2314294972 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2c8c06adb0 0x7f2c8c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:09.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.574+0000 7f2c9effd700 1 --2- 172.21.15.189:0/2314294972 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2c8c06adb0 0x7f2c8c06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2c94006fd0 tx=0x7f2c94008040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:09.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.578+0000 7f2c877fe700 1 -- 172.21.15.189:0/2314294972 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2c900543c0 con 0x7f2ca00d1a50 2024-04-06T13:29:09.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:09.858+0000 7f2ca5801700 1 -- 172.21.15.189:0/2314294972 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7f2ca0074990 con 0x7f2ca00d1a50 2024-04-06T13:29:10.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.427+0000 7f2c877fe700 1 -- 172.21.15.189:0/2314294972 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2519) v1 ==== 168+0+0 (secure 0 0 0) 0x7f2c900579e0 con 0x7f2ca00d1a50 2024-04-06T13:29:10.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.429+0000 7f2ca5801700 1 -- 172.21.15.189:0/2314294972 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2c8c06adb0 msgr2=0x7f2c8c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:10.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.429+0000 7f2ca5801700 1 --2- 172.21.15.189:0/2314294972 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2c8c06adb0 0x7f2c8c06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2c94006fd0 tx=0x7f2c94008040 comp rx=0 tx=0).stop 2024-04-06T13:29:10.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.429+0000 7f2ca5801700 1 -- 172.21.15.189:0/2314294972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ca00d1a50 msgr2=0x7f2ca01f4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:10.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.429+0000 7f2ca5801700 1 --2- 172.21.15.189:0/2314294972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ca00d1a50 0x7f2ca01f4c30 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f2c9000c4b0 tx=0x7f2c9000c870 comp rx=0 tx=0).stop 2024-04-06T13:29:10.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.429+0000 7f2ca5801700 1 -- 172.21.15.189:0/2314294972 shutdown_connections 2024-04-06T13:29:10.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.429+0000 7f2ca5801700 1 --2- 172.21.15.189:0/2314294972 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f2c8c06adb0 0x7f2c8c06d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:10.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.429+0000 7f2ca5801700 1 --2- 172.21.15.189:0/2314294972 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ca015b310 0x7f2ca015ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:10.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.429+0000 7f2ca5801700 1 --2- 172.21.15.189:0/2314294972 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ca00cdfd0 0x7f2ca01f06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:10.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.429+0000 7f2ca5801700 1 --2- 172.21.15.189:0/2314294972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ca00d1a50 0x7f2ca01f4c30 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:10.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.429+0000 7f2ca5801700 1 -- 172.21.15.189:0/2314294972 >> 172.21.15.189:0/2314294972 conn(0x7f2ca006a730 msgr2=0x7f2ca00b3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:10.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.429+0000 7f2ca5801700 1 -- 172.21.15.189:0/2314294972 shutdown_connections 2024-04-06T13:29:10.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.429+0000 7f2ca5801700 1 -- 172.21.15.189:0/2314294972 wait complete. 2024-04-06T13:29:10.444 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-06T13:29:10.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.575+0000 7ff3d2fa2700 1 Processor -- start 2024-04-06T13:29:10.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.575+0000 7ff3d2fa2700 1 -- start start 2024-04-06T13:29:10.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.575+0000 7ff3d2fa2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3cc15d7b0 0x7ff3cc0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:10.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.575+0000 7ff3d2fa2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff3cc06a730 0x7ff3cc06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:10.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.575+0000 7ff3d2fa2700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3cc0b3990 con 0x7ff3cc15ddf0 2024-04-06T13:29:10.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.575+0000 7ff3d2fa2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3cc06b0f0 con 0x7ff3cc06a730 2024-04-06T13:29:10.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.575+0000 7ff3d2fa2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0b3060 con 0x7ff3cc15d7b0 2024-04-06T13:29:10.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.576+0000 7ff3cbfff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff3cc06a730 0x7ff3cc06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:10.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.576+0000 7ff3d0d3e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3cc15d7b0 0x7ff3cc0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:10.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.576+0000 7ff3d153f700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7ff3cc15ddf0 0x7ff3cc15e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:36292/0 (socket says 172.21.15.189:36292) 2024-04-06T13:29:10.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.576+0000 7ff3d153f700 1 -- 172.21.15.189:0/1749673444 learned_addr learned my addr 172.21.15.189:0/1749673444 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:10.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.576+0000 7ff3cbfff700 1 -- 172.21.15.189:0/1749673444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3cc15d7b0 msgr2=0x7ff3cc0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:10.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.576+0000 7ff3cbfff700 1 --2- 172.21.15.189:0/1749673444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3cc15d7b0 0x7ff3cc0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:10.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.576+0000 7ff3cbfff700 1 -- 172.21.15.189:0/1749673444 >> v1:172.21.15.192:6789/0 conn(0x7ff3cc15ddf0 legacy=0x7ff3cc15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:10.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.576+0000 7ff3cbfff700 1 -- 172.21.15.189:0/1749673444 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc0b3340 con 0x7ff3cc06a730 2024-04-06T13:29:10.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.576+0000 7ff3d0d3e700 1 --2- 172.21.15.189:0/1749673444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3cc15d7b0 0x7ff3cc0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:29:10.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.576+0000 7ff3cbfff700 1 --2- 172.21.15.189:0/1749673444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3cc06a730 0x7ff3cc06abb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009840 tx=0x7ff3b8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=723cbfc31bb1f4ca server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:10.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.577+0000 7ff3cb7fe700 1 -- 172.21.15.189:0/1749673444 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8011830 con 0x7ff3cc06a730 2024-04-06T13:29:10.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.577+0000 7ff3cb7fe700 1 -- 172.21.15.189:0/1749673444 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b8011990 con 0x7ff3cc06a730 2024-04-06T13:29:10.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.577+0000 7ff3cb7fe700 1 -- 172.21.15.189:0/1749673444 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8011d20 con 0x7ff3cc06a730 2024-04-06T13:29:10.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.577+0000 7ff3d2fa2700 1 -- 172.21.15.189:0/1749673444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3cc06a730 msgr2=0x7ff3cc06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:10.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.577+0000 7ff3d2fa2700 1 --2- 172.21.15.189:0/1749673444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3cc06a730 0x7ff3cc06abb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009840 tx=0x7ff3b8009b50 comp rx=0 tx=0).stop 2024-04-06T13:29:10.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.577+0000 7ff3d2fa2700 1 -- 172.21.15.189:0/1749673444 shutdown_connections 2024-04-06T13:29:10.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.577+0000 7ff3d2fa2700 1 --2- 172.21.15.189:0/1749673444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3cc15d7b0 0x7ff3cc0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:10.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.577+0000 7ff3d2fa2700 1 --2- 172.21.15.189:0/1749673444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3cc06a730 0x7ff3cc06abb0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:10.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.577+0000 7ff3d2fa2700 1 -- 172.21.15.189:0/1749673444 >> 172.21.15.189:0/1749673444 conn(0x7ff3cc0d00b0 msgr2=0x7ff3cc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:10.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.577+0000 7ff3d2fa2700 1 -- 172.21.15.189:0/1749673444 shutdown_connections 2024-04-06T13:29:10.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.577+0000 7ff3d2fa2700 1 -- 172.21.15.189:0/1749673444 wait complete. 2024-04-06T13:29:10.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.578+0000 7ff3d2fa2700 1 Processor -- start 2024-04-06T13:29:10.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.578+0000 7ff3d2fa2700 1 -- start start 2024-04-06T13:29:10.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.579+0000 7ff3d2fa2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3cc06a730 0x7ff3cc0cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:10.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.579+0000 7ff3d2fa2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3cc15d7b0 0x7ff3cc14f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:10.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.579+0000 7ff3d2fa2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3cc15ddf0 0x7ff3cc0ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:10.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.579+0000 7ff3d2fa2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc157ea0 con 0x7ff3cc06a730 2024-04-06T13:29:10.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.579+0000 7ff3d2fa2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc150e20 con 0x7ff3cc15d7b0 2024-04-06T13:29:10.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.579+0000 7ff3d2fa2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0cee40 con 0x7ff3cc15ddf0 2024-04-06T13:29:10.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.579+0000 7ff3d0d3e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3cc06a730 0x7ff3cc0cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:10.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.579+0000 7ff3d0d3e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3cc06a730 0x7ff3cc0cc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49792/0 (socket says 172.21.15.189:49792) 2024-04-06T13:29:10.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.579+0000 7ff3d0d3e700 1 -- 172.21.15.189:0/215770945 learned_addr learned my addr 172.21.15.189:0/215770945 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:10.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.580+0000 7ff3cbfff700 1 --2- 172.21.15.189:0/215770945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3cc15d7b0 0x7ff3cc14f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:10.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.580+0000 7ff3d153f700 1 --2- 172.21.15.189:0/215770945 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3cc15ddf0 0x7ff3cc0ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:10.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.580+0000 7ff3d0d3e700 1 -- 172.21.15.189:0/215770945 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3cc15ddf0 msgr2=0x7ff3cc0ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:10.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.580+0000 7ff3d0d3e700 1 --2- 172.21.15.189:0/215770945 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3cc15ddf0 0x7ff3cc0ccb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:10.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.580+0000 7ff3d0d3e700 1 -- 172.21.15.189:0/215770945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3cc15d7b0 msgr2=0x7ff3cc14f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:10.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.580+0000 7ff3d0d3e700 1 --2- 172.21.15.189:0/215770945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3cc15d7b0 0x7ff3cc14f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:10.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.580+0000 7ff3d0d3e700 1 -- 172.21.15.189:0/215770945 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c00094a0 con 0x7ff3cc06a730 2024-04-06T13:29:10.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.580+0000 7ff3cbfff700 1 --2- 172.21.15.189:0/215770945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3cc15d7b0 0x7ff3cc14f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:29:10.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.580+0000 7ff3d0d3e700 1 --2- 172.21.15.189:0/215770945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3cc06a730 0x7ff3cc0cc370 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000ebe0 tx=0x7ff3c0009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:10.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.580+0000 7ff3c97fa700 1 -- 172.21.15.189:0/215770945 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c000c910 con 0x7ff3cc06a730 2024-04-06T13:29:10.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.580+0000 7ff3c97fa700 1 -- 172.21.15.189:0/215770945 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c000ca70 con 0x7ff3cc06a730 2024-04-06T13:29:10.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.580+0000 7ff3d2fa2700 1 -- 172.21.15.189:0/215770945 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b80094a0 con 0x7ff3cc06a730 2024-04-06T13:29:10.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.580+0000 7ff3c97fa700 1 -- 172.21.15.189:0/215770945 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c000cde0 con 0x7ff3cc06a730 2024-04-06T13:29:10.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.580+0000 7ff3d2fa2700 1 -- 172.21.15.189:0/215770945 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3cc157ac0 con 0x7ff3cc06a730 2024-04-06T13:29:10.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.581+0000 7ff3d2fa2700 1 -- 172.21.15.189:0/215770945 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3cc0cf360 con 0x7ff3cc06a730 2024-04-06T13:29:10.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.581+0000 7ff3c97fa700 1 -- 172.21.15.189:0/215770945 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7ff3c0013470 con 0x7ff3cc06a730 2024-04-06T13:29:10.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.582+0000 7ff3c97fa700 1 --2- 172.21.15.189:0/215770945 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff3b006adb0 0x7ff3b006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:10.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.582+0000 7ff3c97fa700 1 -- 172.21.15.189:0/215770945 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5909+0+0 (secure 0 0 0) 0x7ff3c00897a0 con 0x7ff3cc06a730 2024-04-06T13:29:10.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.582+0000 7ff3cbfff700 1 --2- 172.21.15.189:0/215770945 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff3b006adb0 0x7ff3b006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:10.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.582+0000 7ff3cbfff700 1 --2- 172.21.15.189:0/215770945 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff3b006adb0 0x7ff3b006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc1f61b0 tx=0x7ff3b8008030 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:10.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.586+0000 7ff3c97fa700 1 -- 172.21.15.189:0/215770945 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff3cc0cf360 con 0x7ff3cc06a730 2024-04-06T13:29:10.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.780+0000 7ff3d2fa2700 1 -- 172.21.15.189:0/215770945 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff3cc157920 con 0x7ff3cc06a730 2024-04-06T13:29:10.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.784+0000 7ff3c97fa700 1 -- 172.21.15.189:0/215770945 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff3cc157920 con 0x7ff3cc06a730 2024-04-06T13:29:10.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.786+0000 7ff3d2fa2700 1 -- 172.21.15.189:0/215770945 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff3b006adb0 msgr2=0x7ff3b006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:10.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.786+0000 7ff3d2fa2700 1 --2- 172.21.15.189:0/215770945 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff3b006adb0 0x7ff3b006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc1f61b0 tx=0x7ff3b8008030 comp rx=0 tx=0).stop 2024-04-06T13:29:10.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.786+0000 7ff3d2fa2700 1 -- 172.21.15.189:0/215770945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3cc06a730 msgr2=0x7ff3cc0cc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:10.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.786+0000 7ff3d2fa2700 1 --2- 172.21.15.189:0/215770945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3cc06a730 0x7ff3cc0cc370 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000ebe0 tx=0x7ff3c0009ec0 comp rx=0 tx=0).stop 2024-04-06T13:29:10.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.786+0000 7ff3d2fa2700 1 -- 172.21.15.189:0/215770945 shutdown_connections 2024-04-06T13:29:10.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.786+0000 7ff3d2fa2700 1 --2- 172.21.15.189:0/215770945 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3cc15ddf0 0x7ff3cc0ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:10.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.786+0000 7ff3d2fa2700 1 --2- 172.21.15.189:0/215770945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3cc06a730 0x7ff3cc0cc370 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:10.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.786+0000 7ff3d2fa2700 1 --2- 172.21.15.189:0/215770945 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff3b006adb0 0x7ff3b006d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:10.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.786+0000 7ff3d2fa2700 1 --2- 172.21.15.189:0/215770945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3cc15d7b0 0x7ff3cc14f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:10.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.786+0000 7ff3d2fa2700 1 -- 172.21.15.189:0/215770945 >> 172.21.15.189:0/215770945 conn(0x7ff3cc0d00b0 msgr2=0x7ff3cc06fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:10.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.786+0000 7ff3d2fa2700 1 -- 172.21.15.189:0/215770945 shutdown_connections 2024-04-06T13:29:10.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:10.786+0000 7ff3d2fa2700 1 -- 172.21.15.189:0/215770945 wait complete. 2024-04-06T13:29:10.798 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T13:29:10.801 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:29:10.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:29:10.888 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T13:29:10.889 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:29:10.889 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:29:10.908 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:29:10.908 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:29:10.908 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:29:10.928 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:29:10.928 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:29:10.928 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:29:10.965 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:29:10.965 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:29:10.966 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:29:10.987 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:29:10.987 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-06T13:29:10.987 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:29:11.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.127+0000 7f1878383700 1 Processor -- start 2024-04-06T13:29:11.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.127+0000 7f1878383700 1 -- start start 2024-04-06T13:29:11.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.128+0000 7f1878383700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f187015a550 0x7f18700cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:11.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.128+0000 7f1878383700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f18700b4d70 0x7f18700b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:11.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.128+0000 7f1878383700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18700cdbe0 con 0x7f18700cf900 2024-04-06T13:29:11.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.128+0000 7f1878383700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f187015a940 con 0x7f18700b4d70 2024-04-06T13:29:11.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.128+0000 7f1878383700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f187015ac00 con 0x7f187015a550 2024-04-06T13:29:11.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.128+0000 7f187591e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f18700cf900 0x7f18700b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42956/0 (socket says 172.21.15.189:42956) 2024-04-06T13:29:11.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.128+0000 7f187591e700 1 -- 172.21.15.189:0/249876057 learned_addr learned my addr 172.21.15.189:0/249876057 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:11.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.128+0000 7f187511d700 1 -- 172.21.15.189:0/249876057 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893767629 0 0) 0x7f18700cdbe0 con 0x7f18700cf900 2024-04-06T13:29:11.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.128+0000 7f1876920700 1 --2- 172.21.15.189:0/249876057 >> v2:172.21.15.192:3300/0 conn(0x7f18700b4d70 0x7f18700b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:11.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.128+0000 7f187511d700 1 -- 172.21.15.189:0/249876057 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f185c0035b0 con 0x7f18700cf900 2024-04-06T13:29:11.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.128+0000 7f187611f700 1 --2- 172.21.15.189:0/249876057 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f187015a550 0x7f18700cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:11.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f187511d700 1 -- 172.21.15.189:0/249876057 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3855348429 0 0) 0x7f185c0035b0 con 0x7f18700cf900 2024-04-06T13:29:11.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f187511d700 1 -- 172.21.15.189:0/249876057 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f187015a550 msgr2=0x7f18700cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:11.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f187511d700 1 --2- 172.21.15.189:0/249876057 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f187015a550 0x7f18700cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:11.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f187511d700 1 -- 172.21.15.189:0/249876057 >> v2:172.21.15.192:3300/0 conn(0x7f18700b4d70 msgr2=0x7f18700b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:11.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f187511d700 1 --2- 172.21.15.189:0/249876057 >> v2:172.21.15.192:3300/0 conn(0x7f18700b4d70 0x7f18700b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:11.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f187511d700 1 -- 172.21.15.189:0/249876057 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187015aee0 con 0x7f18700cf900 2024-04-06T13:29:11.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f187511d700 1 -- 172.21.15.189:0/249876057 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f186c001670 con 0x7f18700cf900 2024-04-06T13:29:11.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f187511d700 1 -- 172.21.15.189:0/249876057 >> v1:172.21.15.189:6789/0 conn(0x7f18700cf900 legacy=0x7f18700b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:11.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f187511d700 1 --2- 172.21.15.189:0/249876057 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f185c003a00 0x7f185c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:11.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f187511d700 1 --2- 172.21.15.189:0/249876057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f185c004360 0x7f185c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:11.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f187511d700 1 --2- 172.21.15.189:0/249876057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185c004f50 0x7f185c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:11.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f187511d700 1 -- 172.21.15.189:0/249876057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187015a940 con 0x7f185c004f50 2024-04-06T13:29:11.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f187511d700 1 -- 172.21.15.189:0/249876057 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187015ac00 con 0x7f185c004360 2024-04-06T13:29:11.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f187511d700 1 -- 172.21.15.189:0/249876057 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185c00bf50 con 0x7f185c003a00 2024-04-06T13:29:11.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f1876920700 1 --2- 172.21.15.189:0/249876057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185c004f50 0x7f185c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:11.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f187611f700 1 --2- 172.21.15.189:0/249876057 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f185c003a00 0x7f185c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:11.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f187591e700 1 --2- 172.21.15.189:0/249876057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f185c004360 0x7f185c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:11.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f1876920700 1 -- 172.21.15.189:0/249876057 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f185c003a00 msgr2=0x7f185c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:11.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f1876920700 1 --2- 172.21.15.189:0/249876057 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f185c003a00 0x7f185c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:11.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f1876920700 1 -- 172.21.15.189:0/249876057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f185c004360 msgr2=0x7f185c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:11.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f1876920700 1 --2- 172.21.15.189:0/249876057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f185c004360 0x7f185c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:11.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f1876920700 1 -- 172.21.15.189:0/249876057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c0035b0 con 0x7f185c004f50 2024-04-06T13:29:11.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f1876920700 1 --2- 172.21.15.189:0/249876057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185c004f50 0x7f185c00ba10 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f1868009f90 tx=0x7f185c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=598beb4a7bda857c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:11.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f187511d700 1 -- 172.21.15.189:0/249876057 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186800ec30 con 0x7f185c004f50 2024-04-06T13:29:11.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f187511d700 1 -- 172.21.15.189:0/249876057 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186800f040 con 0x7f185c004f50 2024-04-06T13:29:11.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.129+0000 7f187511d700 1 -- 172.21.15.189:0/249876057 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1868014480 con 0x7f185c004f50 2024-04-06T13:29:11.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.130+0000 7f1878383700 1 -- 172.21.15.189:0/249876057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185c004f50 msgr2=0x7f185c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:11.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.130+0000 7f1878383700 1 --2- 172.21.15.189:0/249876057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185c004f50 0x7f185c00ba10 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f1868009f90 tx=0x7f185c005880 comp rx=0 tx=0).stop 2024-04-06T13:29:11.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.130+0000 7f1876920700 1 -- 172.21.15.189:0/249876057 reap_dead start 2024-04-06T13:29:11.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.130+0000 7f1878383700 1 -- 172.21.15.189:0/249876057 shutdown_connections 2024-04-06T13:29:11.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.130+0000 7f1878383700 1 -- 172.21.15.189:0/249876057 >> 172.21.15.189:0/249876057 conn(0x7f18700d00b0 msgr2=0x7f18700d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:11.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.130+0000 7f1878383700 1 -- 172.21.15.189:0/249876057 shutdown_connections 2024-04-06T13:29:11.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.130+0000 7f1878383700 1 -- 172.21.15.189:0/249876057 wait complete. 2024-04-06T13:29:11.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.131+0000 7f1878383700 1 Processor -- start 2024-04-06T13:29:11.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.132+0000 7f1878383700 1 -- start start 2024-04-06T13:29:11.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.132+0000 7f1878383700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185c004f50 0x7f18701517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:11.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.132+0000 7f1878383700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f187006eb50 0x7f187006efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:11.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.132+0000 7f1878383700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18700a8f80 0x7f18700a9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:11.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.132+0000 7f1878383700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18700a9c70 con 0x7f185c004f50 2024-04-06T13:29:11.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.132+0000 7f1878383700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18701f3b30 con 0x7f18700a8f80 2024-04-06T13:29:11.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.132+0000 7f1878383700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f187006f510 con 0x7f187006eb50 2024-04-06T13:29:11.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.132+0000 7f187611f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185c004f50 0x7f18701517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:11.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.132+0000 7f187611f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185c004f50 0x7f18701517a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49810/0 (socket says 172.21.15.189:49810) 2024-04-06T13:29:11.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.132+0000 7f187611f700 1 -- 172.21.15.189:0/529678883 learned_addr learned my addr 172.21.15.189:0/529678883 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:11.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.133+0000 7f1876920700 1 --2- 172.21.15.189:0/529678883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18700a8f80 0x7f18700a9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:11.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.133+0000 7f187591e700 1 --2- 172.21.15.189:0/529678883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f187006eb50 0x7f187006efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:11.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.133+0000 7f187611f700 1 -- 172.21.15.189:0/529678883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f187006eb50 msgr2=0x7f187006efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:11.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.133+0000 7f187611f700 1 --2- 172.21.15.189:0/529678883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f187006eb50 0x7f187006efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:11.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.133+0000 7f187611f700 1 -- 172.21.15.189:0/529678883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18700a8f80 msgr2=0x7f18700a9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:11.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.133+0000 7f187611f700 1 --2- 172.21.15.189:0/529678883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18700a8f80 0x7f18700a9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:11.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.133+0000 7f187611f700 1 -- 172.21.15.189:0/529678883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18701f4670 con 0x7f185c004f50 2024-04-06T13:29:11.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.133+0000 7f187591e700 1 --2- 172.21.15.189:0/529678883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f187006eb50 0x7f187006efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:29:11.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.133+0000 7f187611f700 1 --2- 172.21.15.189:0/529678883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185c004f50 0x7f18701517a0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f18600116c0 tx=0x7f1860011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:11.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.133+0000 7f1866ffd700 1 -- 172.21.15.189:0/529678883 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186000c070 con 0x7f185c004f50 2024-04-06T13:29:11.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.133+0000 7f1866ffd700 1 -- 172.21.15.189:0/529678883 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1860015870 con 0x7f185c004f50 2024-04-06T13:29:11.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.133+0000 7f1866ffd700 1 -- 172.21.15.189:0/529678883 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1860015c20 con 0x7f185c004f50 2024-04-06T13:29:11.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.133+0000 7f1878383700 1 -- 172.21.15.189:0/529678883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18701f3d40 con 0x7f185c004f50 2024-04-06T13:29:11.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.133+0000 7f1878383700 1 -- 172.21.15.189:0/529678883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18701f4300 con 0x7f185c004f50 2024-04-06T13:29:11.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.134+0000 7f1878383700 1 -- 172.21.15.189:0/529678883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18701f44c0 con 0x7f185c004f50 2024-04-06T13:29:11.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.134+0000 7f1866ffd700 1 -- 172.21.15.189:0/529678883 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f186001e450 con 0x7f185c004f50 2024-04-06T13:29:11.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.135+0000 7f1866ffd700 1 --2- 172.21.15.189:0/529678883 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f185406ade0 0x7f185406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:11.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.135+0000 7f187591e700 1 --2- 172.21.15.189:0/529678883 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f185406ade0 0x7f185406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:11.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.135+0000 7f1866ffd700 1 -- 172.21.15.189:0/529678883 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5909+0+0 (secure 0 0 0) 0x7f1860089bf0 con 0x7f185c004f50 2024-04-06T13:29:11.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.135+0000 7f187591e700 1 --2- 172.21.15.189:0/529678883 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f185406ade0 0x7f185406d2a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f186c000fb0 tx=0x7f186c009040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:11.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.139+0000 7f1866ffd700 1 -- 172.21.15.189:0/529678883 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1860054ae0 con 0x7f185c004f50 2024-04-06T13:29:11.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.377+0000 7f1878383700 1 -- 172.21.15.189:0/529678883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1870067bb0 con 0x7f185c004f50 2024-04-06T13:29:11.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.378+0000 7f1866ffd700 1 -- 172.21.15.189:0/529678883 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+3542 (secure 0 0 0) 0x7f1860058100 con 0x7f185c004f50 2024-04-06T13:29:11.378 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:29:11.378 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":80,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8842,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/3388956983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3388956983},{"type":"v1","addr":"172.21.15.192:6834","nonce":3388956983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":8845,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1844918354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":1844918354},{"type":"v1","addr":"172.21.15.192:6835","nonce":1844918354}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":9618,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3742852104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3742852104},{"type":"v1","addr":"172.21.15.189:6836","nonce":3742852104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":10047,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1751580619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1751580619},{"type":"v1","addr":"172.21.15.189:6837","nonce":1751580619}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80}],"filesystems":[]} 2024-04-06T13:29:11.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.380+0000 7f1878383700 1 -- 172.21.15.189:0/529678883 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f185406ade0 msgr2=0x7f185406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:11.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.380+0000 7f1878383700 1 --2- 172.21.15.189:0/529678883 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f185406ade0 0x7f185406d2a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f186c000fb0 tx=0x7f186c009040 comp rx=0 tx=0).stop 2024-04-06T13:29:11.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.380+0000 7f1878383700 1 -- 172.21.15.189:0/529678883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185c004f50 msgr2=0x7f18701517a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:11.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.380+0000 7f1878383700 1 --2- 172.21.15.189:0/529678883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185c004f50 0x7f18701517a0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f18600116c0 tx=0x7f1860011a80 comp rx=0 tx=0).stop 2024-04-06T13:29:11.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.380+0000 7f1878383700 1 -- 172.21.15.189:0/529678883 shutdown_connections 2024-04-06T13:29:11.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.380+0000 7f1878383700 1 --2- 172.21.15.189:0/529678883 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f185406ade0 0x7f185406d2a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:11.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.380+0000 7f1878383700 1 --2- 172.21.15.189:0/529678883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18700a8f80 0x7f18700a9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:11.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.380+0000 7f1878383700 1 --2- 172.21.15.189:0/529678883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185c004f50 0x7f18701517a0 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:11.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.380+0000 7f1878383700 1 --2- 172.21.15.189:0/529678883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f187006eb50 0x7f187006efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:11.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.380+0000 7f1878383700 1 -- 172.21.15.189:0/529678883 >> 172.21.15.189:0/529678883 conn(0x7f18700d00b0 msgr2=0x7f18700b2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:11.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.380+0000 7f1878383700 1 -- 172.21.15.189:0/529678883 shutdown_connections 2024-04-06T13:29:11.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.380+0000 7f1878383700 1 -- 172.21.15.189:0/529678883 wait complete. 2024-04-06T13:29:11.381 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 80 2024-04-06T13:29:11.392 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-06T13:29:11.392 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-06T13:29:11.392 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-06T13:29:11.392 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:29:11.393 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-06T13:29:11.393 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-06T13:29:11.393 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-06T13:29:11.393 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:29:11.393 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-06T13:29:11.393 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-06T13:29:11.393 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-06T13:29:11.394 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:29:11.394 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-06T13:29:11.394 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-06T13:29:11.394 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-06T13:29:11.394 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:29:11.401 INFO:tasks.ceph.mds.a:Stopped 2024-04-06T13:29:11.401 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-06T13:29:11.402 INFO:tasks.ceph.mds.c:Stopped 2024-04-06T13:29:11.402 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-06T13:29:11.405 INFO:tasks.ceph.mds.b:Stopped 2024-04-06T13:29:11.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-06T13:29:11.406 INFO:tasks.ceph.mds.d:Stopped 2024-04-06T13:29:11.406 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-06T13:29:11.444 INFO:tasks.ceph.mds.a:Started 2024-04-06T13:29:11.445 DEBUG:teuthology.parallel:result is None 2024-04-06T13:29:11.445 INFO:tasks.ceph.mds.c:Started 2024-04-06T13:29:11.445 DEBUG:teuthology.parallel:result is None 2024-04-06T13:29:11.448 INFO:tasks.ceph.mds.b:Started 2024-04-06T13:29:11.448 DEBUG:teuthology.parallel:result is None 2024-04-06T13:29:11.449 INFO:tasks.ceph.mds.d:Started 2024-04-06T13:29:11.449 DEBUG:teuthology.parallel:result is None 2024-04-06T13:29:11.449 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-06T13:29:11.524 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T13:29:11.524 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2024-04-06T13:29:11.537 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T13:29:11.538 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2024-04-06T13:29:11.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.581+0000 7f1903960700 1 Processor -- start 2024-04-06T13:29:11.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.582+0000 7f1903960700 1 -- start start 2024-04-06T13:29:11.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.582+0000 7f1903960700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f18fc15d7e0 0x7f18fc0d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:11.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.582+0000 7f1903960700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18fc0d1cc0 0x7f18fc0b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:11.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.582+0000 7f1903960700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18fc0cdf80 con 0x7f18fc0d0890 2024-04-06T13:29:11.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.582+0000 7f1903960700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18fc0d3230 con 0x7f18fc15d7e0 2024-04-06T13:29:11.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.582+0000 7f1903960700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc0d2da0 con 0x7f18fc0d1cc0 2024-04-06T13:29:11.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f1900efb700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f18fc0d0890 0x7f18fc0d15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42974/0 (socket says 172.21.15.189:42974) 2024-04-06T13:29:11.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f1900efb700 1 -- 172.21.15.189:0/3791889195 learned_addr learned my addr 172.21.15.189:0/3791889195 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:11.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f1901efd700 1 --2- 172.21.15.189:0/3791889195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18fc0d1cc0 0x7f18fc0b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:11.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f19016fc700 1 --2- 172.21.15.189:0/3791889195 >> v2:172.21.15.192:3300/0 conn(0x7f18fc15d7e0 0x7f18fc0d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:11.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f18fbfff700 1 -- 172.21.15.189:0/3791889195 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178401570 0 0) 0x7f18fc0cdf80 con 0x7f18fc0d0890 2024-04-06T13:29:11.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f18fbfff700 1 -- 172.21.15.189:0/3791889195 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18e40035b0 con 0x7f18fc0d0890 2024-04-06T13:29:11.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f18fbfff700 1 -- 172.21.15.189:0/3791889195 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3677806768 0 0) 0x7f18e40035b0 con 0x7f18fc0d0890 2024-04-06T13:29:11.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f18fbfff700 1 -- 172.21.15.189:0/3791889195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18fc0d1cc0 msgr2=0x7f18fc0b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:11.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f18fbfff700 1 --2- 172.21.15.189:0/3791889195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18fc0d1cc0 0x7f18fc0b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:11.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f18fbfff700 1 -- 172.21.15.189:0/3791889195 >> v2:172.21.15.192:3300/0 conn(0x7f18fc15d7e0 msgr2=0x7f18fc0d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:11.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f18fbfff700 1 --2- 172.21.15.189:0/3791889195 >> v2:172.21.15.192:3300/0 conn(0x7f18fc15d7e0 0x7f18fc0d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:11.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f18fbfff700 1 -- 172.21.15.189:0/3791889195 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18fc0d0cf0 con 0x7f18fc0d0890 2024-04-06T13:29:11.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f18fbfff700 1 -- 172.21.15.189:0/3791889195 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f18f0001670 con 0x7f18fc0d0890 2024-04-06T13:29:11.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f18fbfff700 1 -- 172.21.15.189:0/3791889195 >> v1:172.21.15.189:6789/0 conn(0x7f18fc0d0890 legacy=0x7f18fc0d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:11.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f18fbfff700 1 --2- 172.21.15.189:0/3791889195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e4003a00 0x7f18e4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:11.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f18fbfff700 1 --2- 172.21.15.189:0/3791889195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18e4004850 0x7f18e4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:11.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f18fbfff700 1 --2- 172.21.15.189:0/3791889195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18fc0d1cc0 0x7f18e4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:11.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f18fbfff700 1 -- 172.21.15.189:0/3791889195 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc0d3230 con 0x7f18e4003a00 2024-04-06T13:29:11.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f18fbfff700 1 -- 172.21.15.189:0/3791889195 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc0d2da0 con 0x7f18e4004850 2024-04-06T13:29:11.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f18fbfff700 1 -- 172.21.15.189:0/3791889195 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e400ba10 con 0x7f18fc0d1cc0 2024-04-06T13:29:11.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f19016fc700 1 --2- 172.21.15.189:0/3791889195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e4003a00 0x7f18e4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:11.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f1900efb700 1 --2- 172.21.15.189:0/3791889195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18e4004850 0x7f18e4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:11.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.583+0000 7f1901efd700 1 --2- 172.21.15.189:0/3791889195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18fc0d1cc0 0x7f18e4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:11.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.584+0000 7f19016fc700 1 -- 172.21.15.189:0/3791889195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18fc0d1cc0 msgr2=0x7f18e4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:11.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.584+0000 7f19016fc700 1 --2- 172.21.15.189:0/3791889195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18fc0d1cc0 0x7f18e4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:11.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.584+0000 7f19016fc700 1 -- 172.21.15.189:0/3791889195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18e4004850 msgr2=0x7f18e4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:11.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.584+0000 7f19016fc700 1 --2- 172.21.15.189:0/3791889195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18e4004850 0x7f18e4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:11.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.584+0000 7f19016fc700 1 -- 172.21.15.189:0/3791889195 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e40035b0 con 0x7f18e4003a00 2024-04-06T13:29:11.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.584+0000 7f1901efd700 1 --2- 172.21.15.189:0/3791889195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18fc0d1cc0 0x7f18e4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:29:11.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.584+0000 7f19016fc700 1 --2- 172.21.15.189:0/3791889195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e4003a00 0x7f18e4003e20 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f18f4009fc0 tx=0x7f18e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76ee80346ccab74 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:11.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.584+0000 7f18fbfff700 1 -- 172.21.15.189:0/3791889195 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f400ec30 con 0x7f18e4003a00 2024-04-06T13:29:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.584+0000 7f18fbfff700 1 -- 172.21.15.189:0/3791889195 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f400f040 con 0x7f18e4003a00 2024-04-06T13:29:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.584+0000 7f18fbfff700 1 -- 172.21.15.189:0/3791889195 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f4014480 con 0x7f18e4003a00 2024-04-06T13:29:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.584+0000 7f1903960700 1 -- 172.21.15.189:0/3791889195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e4003a00 msgr2=0x7f18e4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.584+0000 7f1903960700 1 --2- 172.21.15.189:0/3791889195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e4003a00 0x7f18e4003e20 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f18f4009fc0 tx=0x7f18e4005880 comp rx=0 tx=0).stop 2024-04-06T13:29:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.584+0000 7f1901efd700 1 -- 172.21.15.189:0/3791889195 reap_dead start 2024-04-06T13:29:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.584+0000 7f1903960700 1 -- 172.21.15.189:0/3791889195 shutdown_connections 2024-04-06T13:29:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.584+0000 7f1903960700 1 -- 172.21.15.189:0/3791889195 >> 172.21.15.189:0/3791889195 conn(0x7f18fc072250 msgr2=0x7f18fc072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.584+0000 7f1903960700 1 -- 172.21.15.189:0/3791889195 shutdown_connections 2024-04-06T13:29:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.584+0000 7f1903960700 1 -- 172.21.15.189:0/3791889195 wait complete. 2024-04-06T13:29:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.586+0000 7f1903960700 1 Processor -- start 2024-04-06T13:29:11.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.586+0000 7f1903960700 1 -- start start 2024-04-06T13:29:11.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.586+0000 7f1903960700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e4003a00 0x7f18fc159710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:11.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.586+0000 7f1903960700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18fc0ca4a0 0x7f18fc1f2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:11.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.586+0000 7f1903960700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18fc0caeb0 0x7f18fc0cb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:11.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.586+0000 7f1903960700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc0cb880 con 0x7f18e4003a00 2024-04-06T13:29:11.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.586+0000 7f1903960700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc0b4570 con 0x7f18fc0caeb0 2024-04-06T13:29:11.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.586+0000 7f1903960700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc0b4240 con 0x7f18fc0ca4a0 2024-04-06T13:29:11.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.586+0000 7f19016fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e4003a00 0x7f18fc159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:11.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.586+0000 7f19016fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e4003a00 0x7f18fc159710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49848/0 (socket says 172.21.15.189:49848) 2024-04-06T13:29:11.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.586+0000 7f19016fc700 1 -- 172.21.15.189:0/2871970905 learned_addr learned my addr 172.21.15.189:0/2871970905 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:11.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.587+0000 7f1900efb700 1 --2- 172.21.15.189:0/2871970905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18fc0ca4a0 0x7f18fc1f2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:11.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.587+0000 7f1901efd700 1 --2- 172.21.15.189:0/2871970905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18fc0caeb0 0x7f18fc0cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:11.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.587+0000 7f19016fc700 1 -- 172.21.15.189:0/2871970905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18fc0ca4a0 msgr2=0x7f18fc1f2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:11.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.587+0000 7f19016fc700 1 --2- 172.21.15.189:0/2871970905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18fc0ca4a0 0x7f18fc1f2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:11.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.587+0000 7f19016fc700 1 -- 172.21.15.189:0/2871970905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18fc0caeb0 msgr2=0x7f18fc0cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:11.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.587+0000 7f19016fc700 1 --2- 172.21.15.189:0/2871970905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18fc0caeb0 0x7f18fc0cb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:11.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.587+0000 7f19016fc700 1 -- 172.21.15.189:0/2871970905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18fc1f38b0 con 0x7f18e4003a00 2024-04-06T13:29:11.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.587+0000 7f19016fc700 1 --2- 172.21.15.189:0/2871970905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e4003a00 0x7f18fc159710 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f18f400e700 tx=0x7f18f4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:11.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.587+0000 7f18f9ffb700 1 -- 172.21.15.189:0/2871970905 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f4004450 con 0x7f18e4003a00 2024-04-06T13:29:11.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.587+0000 7f18f9ffb700 1 -- 172.21.15.189:0/2871970905 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f40045d0 con 0x7f18e4003a00 2024-04-06T13:29:11.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.587+0000 7f1903960700 1 -- 172.21.15.189:0/2871970905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18fc1f3b90 con 0x7f18e4003a00 2024-04-06T13:29:11.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.587+0000 7f18f9ffb700 1 -- 172.21.15.189:0/2871970905 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f4003e10 con 0x7f18e4003a00 2024-04-06T13:29:11.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.587+0000 7f1903960700 1 -- 172.21.15.189:0/2871970905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18fc159c50 con 0x7f18e4003a00 2024-04-06T13:29:11.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.588+0000 7f1903960700 1 -- 172.21.15.189:0/2871970905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18fc074990 con 0x7f18e4003a00 2024-04-06T13:29:11.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.589+0000 7f18f9ffb700 1 -- 172.21.15.189:0/2871970905 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f18f4014480 con 0x7f18e4003a00 2024-04-06T13:29:11.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.589+0000 7f18f9ffb700 1 --2- 172.21.15.189:0/2871970905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f18e806adb0 0x7f18e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:11.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.589+0000 7f18f9ffb700 1 -- 172.21.15.189:0/2871970905 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5909+0+0 (secure 0 0 0) 0x7f18f408af00 con 0x7f18e4003a00 2024-04-06T13:29:11.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.589+0000 7f1900efb700 1 --2- 172.21.15.189:0/2871970905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f18e806adb0 0x7f18e806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:11.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.590+0000 7f1900efb700 1 --2- 172.21.15.189:0/2871970905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f18e806adb0 0x7f18e806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f18f0001cf0 tx=0x7f18f0007980 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:11.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.594+0000 7f18f9ffb700 1 -- 172.21.15.189:0/2871970905 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f18f4055e80 con 0x7f18e4003a00 2024-04-06T13:29:11.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:11.804+0000 7f1903960700 1 -- 172.21.15.189:0/2871970905 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f18fc15e5d0 con 0x7f18e4003a00 2024-04-06T13:29:12.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.788+0000 7f18f9ffb700 1 -- 172.21.15.189:0/2871970905 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v188) v1 ==== 97+0+0 (secure 0 0 0) 0x7f18f40594a0 con 0x7f18e4003a00 2024-04-06T13:29:12.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.791+0000 7f1903960700 1 -- 172.21.15.189:0/2871970905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f18e806adb0 msgr2=0x7f18e806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:12.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.791+0000 7f1903960700 1 --2- 172.21.15.189:0/2871970905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f18e806adb0 0x7f18e806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f18f0001cf0 tx=0x7f18f0007980 comp rx=0 tx=0).stop 2024-04-06T13:29:12.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.791+0000 7f1903960700 1 -- 172.21.15.189:0/2871970905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e4003a00 msgr2=0x7f18fc159710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:12.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.791+0000 7f1903960700 1 --2- 172.21.15.189:0/2871970905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e4003a00 0x7f18fc159710 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f18f400e700 tx=0x7f18f4009750 comp rx=0 tx=0).stop 2024-04-06T13:29:12.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.791+0000 7f1903960700 1 -- 172.21.15.189:0/2871970905 shutdown_connections 2024-04-06T13:29:12.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.791+0000 7f1903960700 1 --2- 172.21.15.189:0/2871970905 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f18e806adb0 0x7f18e806d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:12.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.791+0000 7f1903960700 1 --2- 172.21.15.189:0/2871970905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18fc0caeb0 0x7f18fc0cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:12.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.791+0000 7f1903960700 1 --2- 172.21.15.189:0/2871970905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e4003a00 0x7f18fc159710 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:12.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.791+0000 7f1903960700 1 --2- 172.21.15.189:0/2871970905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18fc0ca4a0 0x7f18fc1f2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:12.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.792+0000 7f1903960700 1 -- 172.21.15.189:0/2871970905 >> 172.21.15.189:0/2871970905 conn(0x7f18fc072250 msgr2=0x7f18fc15a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:12.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.792+0000 7f1903960700 1 -- 172.21.15.189:0/2871970905 shutdown_connections 2024-04-06T13:29:12.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.792+0000 7f1903960700 1 -- 172.21.15.189:0/2871970905 wait complete. 2024-04-06T13:29:12.793 INFO:teuthology.orchestra.run.smithi189.stderr: removed all blocklist entries 2024-04-06T13:29:12.806 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-06T13:29:12.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.934+0000 7fa2588fd700 1 Processor -- start 2024-04-06T13:29:12.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.934+0000 7fa2588fd700 1 -- start start 2024-04-06T13:29:12.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa2588fd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa2540b3e20 0x7fa25415e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa2588fd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2540b3990 con 0x7fa25406ae40 2024-04-06T13:29:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa2588fd700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2540b3060 con 0x7fa25415d7b0 2024-04-06T13:29:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa2588fd700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2540b4230 con 0x7fa2540b3e20 2024-04-06T13:29:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa251d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa25406ae40 0x7fa25415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42990/0 (socket says 172.21.15.189:42990) 2024-04-06T13:29:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa251d9b700 1 -- 172.21.15.189:0/3240409865 learned_addr learned my addr 172.21.15.189:0/3240409865 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa252d9d700 1 --2- 172.21.15.189:0/3240409865 >> v2:172.21.15.192:3300/0 conn(0x7fa2540b3e20 0x7fa25415e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa25159a700 1 -- 172.21.15.189:0/3240409865 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518389959 0 0) 0x7fa2540b3990 con 0x7fa25406ae40 2024-04-06T13:29:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa25159a700 1 -- 172.21.15.189:0/3240409865 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2400035b0 con 0x7fa25406ae40 2024-04-06T13:29:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa25159a700 1 -- 172.21.15.189:0/3240409865 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4154857445 0 0) 0x7fa2400035b0 con 0x7fa25406ae40 2024-04-06T13:29:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa25159a700 1 -- 172.21.15.189:0/3240409865 >> v2:172.21.15.192:3300/0 conn(0x7fa2540b3e20 msgr2=0x7fa25415e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa25159a700 1 --2- 172.21.15.189:0/3240409865 >> v2:172.21.15.192:3300/0 conn(0x7fa2540b3e20 0x7fa25415e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa25159a700 1 -- 172.21.15.189:0/3240409865 >> v1:172.21.15.192:6789/0 conn(0x7fa25415d7b0 legacy=0x7fa25406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:12.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa25159a700 1 -- 172.21.15.189:0/3240409865 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2540b3340 con 0x7fa25406ae40 2024-04-06T13:29:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa25159a700 1 -- 172.21.15.189:0/3240409865 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fa23c001670 con 0x7fa25406ae40 2024-04-06T13:29:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa25159a700 1 -- 172.21.15.189:0/3240409865 >> v1:172.21.15.189:6789/0 conn(0x7fa25406ae40 legacy=0x7fa25415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa25159a700 1 --2- 172.21.15.189:0/3240409865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa240003a00 0x7fa240003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa25159a700 1 --2- 172.21.15.189:0/3240409865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa240004850 0x7fa240004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa25159a700 1 --2- 172.21.15.189:0/3240409865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa240005210 0x7fa24000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa25159a700 1 -- 172.21.15.189:0/3240409865 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2540b4230 con 0x7fa240004850 2024-04-06T13:29:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa25159a700 1 -- 172.21.15.189:0/3240409865 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24000c1b0 con 0x7fa240003a00 2024-04-06T13:29:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa25159a700 1 -- 172.21.15.189:0/3240409865 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24000c370 con 0x7fa240005210 2024-04-06T13:29:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa251d9b700 1 --2- 172.21.15.189:0/3240409865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa240004850 0x7fa240004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.936+0000 7fa25259c700 1 --2- 172.21.15.189:0/3240409865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa240003a00 0x7fa240003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.937+0000 7fa252d9d700 1 --2- 172.21.15.189:0/3240409865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa240005210 0x7fa24000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.937+0000 7fa251d9b700 1 -- 172.21.15.189:0/3240409865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa240005210 msgr2=0x7fa24000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:12.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.937+0000 7fa251d9b700 1 --2- 172.21.15.189:0/3240409865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa240005210 0x7fa24000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.937+0000 7fa251d9b700 1 -- 172.21.15.189:0/3240409865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa240003a00 msgr2=0x7fa240003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.937+0000 7fa251d9b700 1 --2- 172.21.15.189:0/3240409865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa240003a00 0x7fa240003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.937+0000 7fa251d9b700 1 -- 172.21.15.189:0/3240409865 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2400035b0 con 0x7fa240004850 2024-04-06T13:29:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.937+0000 7fa25259c700 1 --2- 172.21.15.189:0/3240409865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa240003a00 0x7fa240003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:29:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.937+0000 7fa252d9d700 1 --2- 172.21.15.189:0/3240409865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa240005210 0x7fa24000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:29:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.937+0000 7fa252d9d700 1 -- 172.21.15.189:0/3240409865 reap_dead start 2024-04-06T13:29:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.937+0000 7fa251d9b700 1 --2- 172.21.15.189:0/3240409865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa240004850 0x7fa240004cd0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fa23c00c240 tx=0x7fa240005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290c4d02a8f737a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.937+0000 7fa25159a700 1 -- 172.21.15.189:0/3240409865 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c007c10 con 0x7fa240004850 2024-04-06T13:29:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.937+0000 7fa25159a700 1 -- 172.21.15.189:0/3240409865 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23c00cde0 con 0x7fa240004850 2024-04-06T13:29:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.937+0000 7fa25159a700 1 -- 172.21.15.189:0/3240409865 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c005ba0 con 0x7fa240004850 2024-04-06T13:29:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.937+0000 7fa2588fd700 1 -- 172.21.15.189:0/3240409865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa240004850 msgr2=0x7fa240004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.937+0000 7fa2588fd700 1 --2- 172.21.15.189:0/3240409865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa240004850 0x7fa240004cd0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fa23c00c240 tx=0x7fa240005880 comp rx=0 tx=0).stop 2024-04-06T13:29:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.938+0000 7fa2588fd700 1 -- 172.21.15.189:0/3240409865 shutdown_connections 2024-04-06T13:29:12.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.938+0000 7fa2588fd700 1 --2- 172.21.15.189:0/3240409865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa240004850 0x7fa240004cd0 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:12.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.938+0000 7fa2588fd700 1 -- 172.21.15.189:0/3240409865 >> 172.21.15.189:0/3240409865 conn(0x7fa2540d00b0 msgr2=0x7fa2540d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:12.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.938+0000 7fa2588fd700 1 -- 172.21.15.189:0/3240409865 shutdown_connections 2024-04-06T13:29:12.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.938+0000 7fa2588fd700 1 -- 172.21.15.189:0/3240409865 wait complete. 2024-04-06T13:29:12.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.939+0000 7fa2588fd700 1 Processor -- start 2024-04-06T13:29:12.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.939+0000 7fa2588fd700 1 -- start start 2024-04-06T13:29:12.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa2588fd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa240004850 0x7fa254150c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:12.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa2588fd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa254155540 0x7fa2540afcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:12.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa2588fd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2541559c0 0x7fa2540b0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:12.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa2588fd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa254156ba0 con 0x7fa254155540 2024-04-06T13:29:12.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa2588fd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2541511b0 con 0x7fa2541559c0 2024-04-06T13:29:12.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa2588fd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa254155f90 con 0x7fa240004850 2024-04-06T13:29:12.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa251d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa254155540 0x7fa2540afcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:12.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa251d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa254155540 0x7fa2540afcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49854/0 (socket says 172.21.15.189:49854) 2024-04-06T13:29:12.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa251d9b700 1 -- 172.21.15.189:0/2190487565 learned_addr learned my addr 172.21.15.189:0/2190487565 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:12.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa25259c700 1 --2- 172.21.15.189:0/2190487565 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa240004850 0x7fa254150c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:12.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa252d9d700 1 --2- 172.21.15.189:0/2190487565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2541559c0 0x7fa2540b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:12.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa251d9b700 1 -- 172.21.15.189:0/2190487565 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa240004850 msgr2=0x7fa254150c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:12.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa251d9b700 1 --2- 172.21.15.189:0/2190487565 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa240004850 0x7fa254150c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:12.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa251d9b700 1 -- 172.21.15.189:0/2190487565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2541559c0 msgr2=0x7fa2540b0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:12.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa251d9b700 1 --2- 172.21.15.189:0/2190487565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2541559c0 0x7fa2540b0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:12.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa251d9b700 1 -- 172.21.15.189:0/2190487565 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa254156270 con 0x7fa254155540 2024-04-06T13:29:12.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa251d9b700 1 --2- 172.21.15.189:0/2190487565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa254155540 0x7fa2540afcd0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fa24000c180 tx=0x7fa23c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:12.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa24affd700 1 -- 172.21.15.189:0/2190487565 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c001640 con 0x7fa254155540 2024-04-06T13:29:12.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa24affd700 1 -- 172.21.15.189:0/2190487565 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23c0070d0 con 0x7fa254155540 2024-04-06T13:29:12.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa2588fd700 1 -- 172.21.15.189:0/2190487565 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa254156550 con 0x7fa254155540 2024-04-06T13:29:12.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa24affd700 1 -- 172.21.15.189:0/2190487565 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c007480 con 0x7fa254155540 2024-04-06T13:29:12.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.940+0000 7fa2588fd700 1 -- 172.21.15.189:0/2190487565 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa254157b90 con 0x7fa254155540 2024-04-06T13:29:12.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.941+0000 7fa2588fd700 1 -- 172.21.15.189:0/2190487565 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa254156d50 con 0x7fa254155540 2024-04-06T13:29:12.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.942+0000 7fa24affd700 1 -- 172.21.15.189:0/2190487565 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fa23c0075e0 con 0x7fa254155540 2024-04-06T13:29:12.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.942+0000 7fa24affd700 1 --2- 172.21.15.189:0/2190487565 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa23406adb0 0x7fa23406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:12.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.942+0000 7fa24affd700 1 -- 172.21.15.189:0/2190487565 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5221+0+0 (secure 0 0 0) 0x7fa23c08a670 con 0x7fa254155540 2024-04-06T13:29:12.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.942+0000 7fa25259c700 1 --2- 172.21.15.189:0/2190487565 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa23406adb0 0x7fa23406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:12.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.943+0000 7fa25259c700 1 --2- 172.21.15.189:0/2190487565 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa23406adb0 0x7fa23406d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa244000c00 tx=0x7fa244009040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:12.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:12.947+0000 7fa24affd700 1 -- 172.21.15.189:0/2190487565 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa254156d50 con 0x7fa254155540 2024-04-06T13:29:13.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.221+0000 7fa2588fd700 1 -- 172.21.15.189:0/2190487565 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa254067bb0 con 0x7fa254155540 2024-04-06T13:29:13.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.222+0000 7fa24affd700 1 -- 172.21.15.189:0/2190487565 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v43) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fa23c0559b0 con 0x7fa254155540 2024-04-06T13:29:13.222 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:29:13.222 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:29:13.222 INFO:teuthology.orchestra.run.smithi189.stdout: "auth_dump": [ 2024-04-06T13:29:13.222 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:13.222 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.a", 2024-04-06T13:29:13.222 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmTjYMNBAA5voWGV4Pa7pE27VogrdipQ==", 2024-04-06T13:29:13.222 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:29:13.222 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:29:13.222 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:29:13.222 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:29:13.222 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:29:13.222 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.223 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:29:13.223 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:13.223 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.b", 2024-04-06T13:29:13.223 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm9A1OBRAAnHzubCvRUX8UdW8hmDzKbQ==", 2024-04-06T13:29:13.223 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:29:13.223 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:29:13.223 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:29:13.223 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:29:13.223 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:29:13.223 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.223 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:29:13.223 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:13.223 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.c", 2024-04-06T13:29:13.224 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmz0hhOxAAD3/GINNmGoFk+oPyiHHJXg==", 2024-04-06T13:29:13.224 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:29:13.224 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:29:13.224 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:29:13.224 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:29:13.224 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:29:13.224 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.224 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:29:13.224 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:13.224 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.d", 2024-04-06T13:29:13.224 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmhoAeDRAAoQC0vicWumBRTwpi3PtYaw==", 2024-04-06T13:29:13.224 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:29:13.224 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:29:13.224 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:29:13.224 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:29:13.225 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:29:13.225 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.225 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:29:13.225 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:13.225 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.0", 2024-04-06T13:29:13.225 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDkOxFmkxLWJhAAFXewCNOFX7dhAZVgUmewpQ==", 2024-04-06T13:29:13.225 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:29:13.225 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:29:13.225 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:29:13.225 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:29:13.225 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.225 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:29:13.225 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:13.225 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.1", 2024-04-06T13:29:13.226 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDnOxFmHYqWJRAABlsVCiv3e7YJMbfodJTWag==", 2024-04-06T13:29:13.226 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:29:13.226 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:29:13.226 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:29:13.226 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:29:13.226 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.226 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:29:13.226 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:13.226 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.2", 2024-04-06T13:29:13.226 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDqOxFmkhcvJBAA8NMEpHVGbGALjV9RISKP8g==", 2024-04-06T13:29:13.226 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:29:13.226 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:29:13.226 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:29:13.227 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:29:13.227 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.227 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:29:13.227 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:13.227 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.3", 2024-04-06T13:29:13.227 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDtOxFmDGzhIxAAamBoAbE8+lAel4os67meyQ==", 2024-04-06T13:29:13.227 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:29:13.227 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:29:13.227 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:29:13.227 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:29:13.227 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.227 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:29:13.227 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:13.228 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.4", 2024-04-06T13:29:13.228 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDzOxFmobZDJxAA9l8wdS81Eyw6+17I5ac6bw==", 2024-04-06T13:29:13.228 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:29:13.228 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:29:13.228 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:29:13.228 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:29:13.228 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.228 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:29:13.228 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:13.228 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.5", 2024-04-06T13:29:13.228 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD2OxFmi+ruJRAASlQ0o9jX/wA7U5AosnL+Wg==", 2024-04-06T13:29:13.228 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:29:13.228 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:29:13.228 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:29:13.228 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:29:13.228 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.229 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:29:13.229 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:13.229 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.6", 2024-04-06T13:29:13.229 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD5OxFmbwuuJBAA0wqJRmnJyskQPZJ7/LnpvA==", 2024-04-06T13:29:13.229 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:29:13.229 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:29:13.229 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:29:13.229 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:29:13.229 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.229 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:29:13.229 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:13.229 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.7", 2024-04-06T13:29:13.229 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD8OxFmDmRgIxAAzyW9AGAbmb67ZvB5qunwLQ==", 2024-04-06T13:29:13.229 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:29:13.229 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:29:13.230 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:29:13.230 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:29:13.230 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.230 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:29:13.230 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:13.230 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.0", 2024-04-06T13:29:13.230 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm0ByJERAAoEV8Yfc09+Go6u95XafElQ==", 2024-04-06T13:29:13.230 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:29:13.230 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:29:13.230 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow r", 2024-04-06T13:29:13.230 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-06T13:29:13.230 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.230 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:29:13.230 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:13.231 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.1", 2024-04-06T13:29:13.231 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmSV5lGBAAavDEo2LGy0IHpWSAuitHXg==", 2024-04-06T13:29:13.231 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:29:13.231 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:29:13.231 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:29:13.231 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:29:13.231 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:29:13.231 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.231 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:29:13.231 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:13.231 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.2", 2024-04-06T13:29:13.231 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm/Gz5FBAA3xX/rsAyWZ1JoUVlQlBIvQ==", 2024-04-06T13:29:13.231 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:29:13.231 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:29:13.232 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:29:13.232 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:29:13.232 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:29:13.232 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.232 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:29:13.232 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:13.232 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.3", 2024-04-06T13:29:13.232 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmzra7GxAADbnzIvWnRTo65LvPwtxWrg==", 2024-04-06T13:29:13.232 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:29:13.232 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:29:13.232 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:29:13.232 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:29:13.232 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:29:13.232 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.232 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:29:13.232 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:13.233 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.admin", 2024-04-06T13:29:13.233 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmxG+wCBAANg5kt6TOUQdj3/X0Mqgx1g==", 2024-04-06T13:29:13.233 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:29:13.233 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:29:13.233 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:29:13.233 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:29:13.233 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:29:13.233 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.233 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:29:13.233 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:13.233 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.x", 2024-04-06T13:29:13.233 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmPLyBMBAAYfhkI9Z1CPlf1Tijm+L7xA==", 2024-04-06T13:29:13.233 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:29:13.233 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:29:13.233 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T13:29:13.234 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:29:13.234 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.234 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:29:13.234 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:13.234 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.y", 2024-04-06T13:29:13.234 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFm1nHJHhAAQWRe8cU8duWuUJiYaDeTaQ==", 2024-04-06T13:29:13.234 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:29:13.234 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:29:13.234 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T13:29:13.234 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:29:13.234 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.234 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:13.234 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-04-06T13:29:13.234 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:29:13.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.224+0000 7fa2588fd700 1 -- 172.21.15.189:0/2190487565 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa23406adb0 msgr2=0x7fa23406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:13.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.224+0000 7fa2588fd700 1 --2- 172.21.15.189:0/2190487565 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa23406adb0 0x7fa23406d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa244000c00 tx=0x7fa244009040 comp rx=0 tx=0).stop 2024-04-06T13:29:13.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.224+0000 7fa2588fd700 1 -- 172.21.15.189:0/2190487565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa254155540 msgr2=0x7fa2540afcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:13.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.224+0000 7fa2588fd700 1 --2- 172.21.15.189:0/2190487565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa254155540 0x7fa2540afcd0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fa24000c180 tx=0x7fa23c001a00 comp rx=0 tx=0).stop 2024-04-06T13:29:13.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.224+0000 7fa2588fd700 1 -- 172.21.15.189:0/2190487565 shutdown_connections 2024-04-06T13:29:13.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.224+0000 7fa2588fd700 1 --2- 172.21.15.189:0/2190487565 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa23406adb0 0x7fa23406d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.224+0000 7fa2588fd700 1 --2- 172.21.15.189:0/2190487565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2541559c0 0x7fa2540b0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.224+0000 7fa2588fd700 1 --2- 172.21.15.189:0/2190487565 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa240004850 0x7fa254150c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.224+0000 7fa2588fd700 1 --2- 172.21.15.189:0/2190487565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa254155540 0x7fa2540afcd0 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.224+0000 7fa2588fd700 1 -- 172.21.15.189:0/2190487565 >> 172.21.15.189:0/2190487565 conn(0x7fa2540d00b0 msgr2=0x7fa25406eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:13.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.224+0000 7fa2588fd700 1 -- 172.21.15.189:0/2190487565 shutdown_connections 2024-04-06T13:29:13.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.224+0000 7fa2588fd700 1 -- 172.21.15.189:0/2190487565 wait complete. 2024-04-06T13:29:13.237 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:29:13.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.360+0000 7f527c526700 1 Processor -- start 2024-04-06T13:29:13.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.360+0000 7f527c526700 1 -- start start 2024-04-06T13:29:13.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f527c526700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52740b4ed0 0x7f52740b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:13.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f527c526700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f52740d2880 0x7f52740d2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:13.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f527c526700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52740b5cc0 con 0x7f527415d590 2024-04-06T13:29:13.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f527c526700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52740b5890 con 0x7f52740d2880 2024-04-06T13:29:13.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f527c526700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52740d1500 con 0x7f52740b4ed0 2024-04-06T13:29:13.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f527a2c2700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f527415d590 0x7f52740b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43000/0 (socket says 172.21.15.189:43000) 2024-04-06T13:29:13.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f527a2c2700 1 -- 172.21.15.189:0/775776680 learned_addr learned my addr 172.21.15.189:0/775776680 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:13.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f52792c0700 1 -- 172.21.15.189:0/775776680 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194082641 0 0) 0x7f52740b5cc0 con 0x7f527415d590 2024-04-06T13:29:13.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f5279ac1700 1 --2- 172.21.15.189:0/775776680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52740b4ed0 0x7f52740b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:13.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f52792c0700 1 -- 172.21.15.189:0/775776680 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52600035b0 con 0x7f527415d590 2024-04-06T13:29:13.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f527aac3700 1 --2- 172.21.15.189:0/775776680 >> v2:172.21.15.192:3300/0 conn(0x7f52740d2880 0x7f52740d2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:13.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f52792c0700 1 -- 172.21.15.189:0/775776680 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 431976952 0 0) 0x7f52600035b0 con 0x7f527415d590 2024-04-06T13:29:13.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f52792c0700 1 -- 172.21.15.189:0/775776680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52740b4ed0 msgr2=0x7f52740b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:13.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f52792c0700 1 --2- 172.21.15.189:0/775776680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52740b4ed0 0x7f52740b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f52792c0700 1 -- 172.21.15.189:0/775776680 >> v2:172.21.15.192:3300/0 conn(0x7f52740d2880 msgr2=0x7f52740d2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f52792c0700 1 --2- 172.21.15.189:0/775776680 >> v2:172.21.15.192:3300/0 conn(0x7f52740d2880 0x7f52740d2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f52792c0700 1 -- 172.21.15.189:0/775776680 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52740d17e0 con 0x7f527415d590 2024-04-06T13:29:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f52792c0700 1 -- 172.21.15.189:0/775776680 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f5270001670 con 0x7f527415d590 2024-04-06T13:29:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f527aac3700 1 --2- 172.21.15.189:0/775776680 >> v2:172.21.15.192:3300/0 conn(0x7f52740d2880 0x7f52740d2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:29:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f52792c0700 1 -- 172.21.15.189:0/775776680 >> v1:172.21.15.189:6789/0 conn(0x7f527415d590 legacy=0x7f52740b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f52792c0700 1 --2- 172.21.15.189:0/775776680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52740b4ed0 0x7f5260003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f52792c0700 1 --2- 172.21.15.189:0/775776680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5260003f80 0x7f5260004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f52792c0700 1 --2- 172.21.15.189:0/775776680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5260004d90 0x7f5260005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f52792c0700 1 -- 172.21.15.189:0/775776680 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52740b5890 con 0x7f5260003f80 2024-04-06T13:29:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.361+0000 7f52792c0700 1 -- 172.21.15.189:0/775776680 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52740d1500 con 0x7f5260004d90 2024-04-06T13:29:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.362+0000 7f52792c0700 1 -- 172.21.15.189:0/775776680 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f526000ba60 con 0x7f52740b4ed0 2024-04-06T13:29:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.362+0000 7f5279ac1700 1 --2- 172.21.15.189:0/775776680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5260003f80 0x7f5260004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.362+0000 7f527a2c2700 1 --2- 172.21.15.189:0/775776680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52740b4ed0 0x7f5260003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.362+0000 7f527aac3700 1 --2- 172.21.15.189:0/775776680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5260004d90 0x7f5260005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:13.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.362+0000 7f5279ac1700 1 -- 172.21.15.189:0/775776680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52740b4ed0 msgr2=0x7f5260003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:13.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.362+0000 7f5279ac1700 1 --2- 172.21.15.189:0/775776680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52740b4ed0 0x7f5260003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.362+0000 7f5279ac1700 1 -- 172.21.15.189:0/775776680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5260004d90 msgr2=0x7f5260005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:13.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.362+0000 7f5279ac1700 1 --2- 172.21.15.189:0/775776680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5260004d90 0x7f5260005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.362+0000 7f5279ac1700 1 -- 172.21.15.189:0/775776680 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52600035b0 con 0x7f5260003f80 2024-04-06T13:29:13.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.362+0000 7f5279ac1700 1 --2- 172.21.15.189:0/775776680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5260003f80 0x7f5260004850 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f5264009b60 tx=0x7f5260005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc5fb4596cf6bcc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:13.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.362+0000 7f52792c0700 1 -- 172.21.15.189:0/775776680 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526400e6c0 con 0x7f5260003f80 2024-04-06T13:29:13.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.362+0000 7f52792c0700 1 -- 172.21.15.189:0/775776680 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526400e820 con 0x7f5260003f80 2024-04-06T13:29:13.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.362+0000 7f52792c0700 1 -- 172.21.15.189:0/775776680 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526400eb90 con 0x7f5260003f80 2024-04-06T13:29:13.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.363+0000 7f527c526700 1 -- 172.21.15.189:0/775776680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5260003f80 msgr2=0x7f5260004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:13.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.363+0000 7f527c526700 1 --2- 172.21.15.189:0/775776680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5260003f80 0x7f5260004850 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f5264009b60 tx=0x7f5260005880 comp rx=0 tx=0).stop 2024-04-06T13:29:13.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.363+0000 7f527aac3700 1 -- 172.21.15.189:0/775776680 reap_dead start 2024-04-06T13:29:13.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.363+0000 7f527c526700 1 -- 172.21.15.189:0/775776680 shutdown_connections 2024-04-06T13:29:13.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.363+0000 7f527c526700 1 -- 172.21.15.189:0/775776680 >> 172.21.15.189:0/775776680 conn(0x7f527406bbc0 msgr2=0x7f527406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:13.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.363+0000 7f527c526700 1 -- 172.21.15.189:0/775776680 shutdown_connections 2024-04-06T13:29:13.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.363+0000 7f527c526700 1 -- 172.21.15.189:0/775776680 wait complete. 2024-04-06T13:29:13.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.364+0000 7f527c526700 1 Processor -- start 2024-04-06T13:29:13.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.364+0000 7f527c526700 1 -- start start 2024-04-06T13:29:13.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f527c526700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5260003f80 0x7f52740c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:13.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f527c526700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52740ccca0 0x7f5274155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f527c526700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52740cd120 0x7f52740cded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f527c526700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527415d970 con 0x7f5260003f80 2024-04-06T13:29:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f527c526700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52740c9860 con 0x7f52740cd120 2024-04-06T13:29:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f527c526700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f527415dbf0 con 0x7f52740ccca0 2024-04-06T13:29:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f527a2c2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5260003f80 0x7f52740c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f527a2c2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5260003f80 0x7f52740c9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49882/0 (socket says 172.21.15.189:49882) 2024-04-06T13:29:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f527a2c2700 1 -- 172.21.15.189:0/1426613504 learned_addr learned my addr 172.21.15.189:0/1426613504 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f5279ac1700 1 --2- 172.21.15.189:0/1426613504 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52740ccca0 0x7f5274155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f527aac3700 1 --2- 172.21.15.189:0/1426613504 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52740cd120 0x7f52740cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f527a2c2700 1 -- 172.21.15.189:0/1426613504 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52740ccca0 msgr2=0x7f5274155020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f527a2c2700 1 --2- 172.21.15.189:0/1426613504 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52740ccca0 0x7f5274155020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f527a2c2700 1 -- 172.21.15.189:0/1426613504 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52740cd120 msgr2=0x7f52740cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f527a2c2700 1 --2- 172.21.15.189:0/1426613504 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52740cd120 0x7f52740cded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f527a2c2700 1 -- 172.21.15.189:0/1426613504 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527415ded0 con 0x7f5260003f80 2024-04-06T13:29:13.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f527a2c2700 1 --2- 172.21.15.189:0/1426613504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5260003f80 0x7f52740c9320 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f5270000c00 tx=0x7f52700075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:13.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f526affd700 1 -- 172.21.15.189:0/1426613504 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5270011580 con 0x7f5260003f80 2024-04-06T13:29:13.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f526affd700 1 -- 172.21.15.189:0/1426613504 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5270011700 con 0x7f5260003f80 2024-04-06T13:29:13.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f527c526700 1 -- 172.21.15.189:0/1426613504 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f527415e1b0 con 0x7f5260003f80 2024-04-06T13:29:13.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f526affd700 1 -- 172.21.15.189:0/1426613504 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5270011ab0 con 0x7f5260003f80 2024-04-06T13:29:13.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.365+0000 7f527c526700 1 -- 172.21.15.189:0/1426613504 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f527415e770 con 0x7f5260003f80 2024-04-06T13:29:13.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.366+0000 7f527c526700 1 -- 172.21.15.189:0/1426613504 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5274067a30 con 0x7f5260003f80 2024-04-06T13:29:13.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.367+0000 7f526affd700 1 -- 172.21.15.189:0/1426613504 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f5270011c10 con 0x7f5260003f80 2024-04-06T13:29:13.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.367+0000 7f526affd700 1 --2- 172.21.15.189:0/1426613504 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f525806adb0 0x7f525806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:13.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.367+0000 7f526affd700 1 -- 172.21.15.189:0/1426613504 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f52700892f0 con 0x7f5260003f80 2024-04-06T13:29:13.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.367+0000 7f5279ac1700 1 --2- 172.21.15.189:0/1426613504 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f525806adb0 0x7f525806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:13.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.368+0000 7f5279ac1700 1 --2- 172.21.15.189:0/1426613504 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f525806adb0 0x7f525806d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f52640090b0 tx=0x7f526401f040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:13.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.372+0000 7f526affd700 1 -- 172.21.15.189:0/1426613504 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5270057f90 con 0x7f5260003f80 2024-04-06T13:29:13.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.568+0000 7f527c526700 1 -- 172.21.15.189:0/1426613504 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5274074990 con 0x7f5260003f80 2024-04-06T13:29:13.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.569+0000 7f526affd700 1 -- 172.21.15.189:0/1426613504 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+12722 (secure 0 0 0) 0x7f5270057b20 con 0x7f5260003f80 2024-04-06T13:29:13.568 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:29:13.569 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":188,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:29:12.783897+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:29:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.571+0000 7f527c526700 1 -- 172.21.15.189:0/1426613504 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f525806adb0 msgr2=0x7f525806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.571+0000 7f527c526700 1 --2- 172.21.15.189:0/1426613504 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f525806adb0 0x7f525806d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f52640090b0 tx=0x7f526401f040 comp rx=0 tx=0).stop 2024-04-06T13:29:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.571+0000 7f527c526700 1 -- 172.21.15.189:0/1426613504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5260003f80 msgr2=0x7f52740c9320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.571+0000 7f527c526700 1 --2- 172.21.15.189:0/1426613504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5260003f80 0x7f52740c9320 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f5270000c00 tx=0x7f52700075f0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.571+0000 7f527c526700 1 -- 172.21.15.189:0/1426613504 shutdown_connections 2024-04-06T13:29:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.571+0000 7f527c526700 1 --2- 172.21.15.189:0/1426613504 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f525806adb0 0x7f525806d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.571+0000 7f527c526700 1 --2- 172.21.15.189:0/1426613504 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52740cd120 0x7f52740cded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.571+0000 7f527c526700 1 --2- 172.21.15.189:0/1426613504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5260003f80 0x7f52740c9320 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.571+0000 7f527c526700 1 --2- 172.21.15.189:0/1426613504 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52740ccca0 0x7f5274155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.571+0000 7f527c526700 1 -- 172.21.15.189:0/1426613504 >> 172.21.15.189:0/1426613504 conn(0x7f527406bbc0 msgr2=0x7f527415a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:13.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.571+0000 7f527c526700 1 -- 172.21.15.189:0/1426613504 shutdown_connections 2024-04-06T13:29:13.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.571+0000 7f527c526700 1 -- 172.21.15.189:0/1426613504 wait complete. 2024-04-06T13:29:13.583 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:29:13.584 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:29:13.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.709+0000 7fdaebb86700 1 Processor -- start 2024-04-06T13:29:13.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.710+0000 7fdaebb86700 1 -- start start 2024-04-06T13:29:13.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.710+0000 7fdaebb86700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdae40cb870 0x7fdae415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:13.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.710+0000 7fdaebb86700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdae40cfa00 0x7fdae40cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:13.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.710+0000 7fdaebb86700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae40cbd80 con 0x7fdae40d11b0 2024-04-06T13:29:13.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.710+0000 7fdaebb86700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdae40ccdc0 con 0x7fdae40cfa00 2024-04-06T13:29:13.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.710+0000 7fdaebb86700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdae40d0b50 con 0x7fdae40cb870 2024-04-06T13:29:13.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.710+0000 7fdaea123700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdae40cfa00 0x7fdae40cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:13.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.711+0000 7fdae9922700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdae40d11b0 0x7fdae40d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43012/0 (socket says 172.21.15.189:43012) 2024-04-06T13:29:13.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.711+0000 7fdae9922700 1 -- 172.21.15.189:0/1681556057 learned_addr learned my addr 172.21.15.189:0/1681556057 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:13.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.711+0000 7fdae9121700 1 --2- 172.21.15.189:0/1681556057 >> v2:172.21.15.192:3300/0 conn(0x7fdae40cb870 0x7fdae415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:13.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.711+0000 7fdaea123700 1 -- 172.21.15.189:0/1681556057 >> v2:172.21.15.192:3300/0 conn(0x7fdae40cb870 msgr2=0x7fdae415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:13.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.711+0000 7fdaea123700 1 --2- 172.21.15.189:0/1681556057 >> v2:172.21.15.192:3300/0 conn(0x7fdae40cb870 0x7fdae415c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.711+0000 7fdaea123700 1 -- 172.21.15.189:0/1681556057 >> v1:172.21.15.189:6789/0 conn(0x7fdae40d11b0 legacy=0x7fdae40d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:13.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.711+0000 7fdaea123700 1 -- 172.21.15.189:0/1681556057 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae40cef20 con 0x7fdae40cfa00 2024-04-06T13:29:13.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.711+0000 7fdae8920700 1 -- 172.21.15.189:0/1681556057 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417539047 0 0) 0x7fdae40cbd80 con 0x7fdae40d11b0 2024-04-06T13:29:13.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.711+0000 7fdaea123700 1 --2- 172.21.15.189:0/1681556057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae40cfa00 0x7fdae40cd560 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fdae404fb40 tx=0x7fdadc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48250acecf3e1bca server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:13.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.711+0000 7fdae8920700 1 -- 172.21.15.189:0/1681556057 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc00e070 con 0x7fdae40cfa00 2024-04-06T13:29:13.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.711+0000 7fdae8920700 1 -- 172.21.15.189:0/1681556057 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdadc00bbd0 con 0x7fdae40cfa00 2024-04-06T13:29:13.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.711+0000 7fdae8920700 1 -- 172.21.15.189:0/1681556057 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc012430 con 0x7fdae40cfa00 2024-04-06T13:29:13.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.712+0000 7fdaebb86700 1 -- 172.21.15.189:0/1681556057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae40cfa00 msgr2=0x7fdae40cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:13.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.712+0000 7fdaebb86700 1 --2- 172.21.15.189:0/1681556057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae40cfa00 0x7fdae40cd560 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fdae404fb40 tx=0x7fdadc00b330 comp rx=0 tx=0).stop 2024-04-06T13:29:13.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.712+0000 7fdaebb86700 1 -- 172.21.15.189:0/1681556057 shutdown_connections 2024-04-06T13:29:13.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.712+0000 7fdaebb86700 1 --2- 172.21.15.189:0/1681556057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae40cfa00 0x7fdae40cd560 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.712+0000 7fdaebb86700 1 --2- 172.21.15.189:0/1681556057 >> v2:172.21.15.192:3300/0 conn(0x7fdae40cb870 0x7fdae415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.712+0000 7fdaebb86700 1 -- 172.21.15.189:0/1681556057 >> 172.21.15.189:0/1681556057 conn(0x7fdae40b2db0 msgr2=0x7fdae40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:13.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.712+0000 7fdaebb86700 1 -- 172.21.15.189:0/1681556057 shutdown_connections 2024-04-06T13:29:13.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.712+0000 7fdaebb86700 1 -- 172.21.15.189:0/1681556057 wait complete. 2024-04-06T13:29:13.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.713+0000 7fdaebb86700 1 Processor -- start 2024-04-06T13:29:13.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.713+0000 7fdaebb86700 1 -- start start 2024-04-06T13:29:13.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdaebb86700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdae40cb870 0x7fdae4152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:13.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdaebb86700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdae40cfa00 0x7fdae40c8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:13.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdaebb86700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae40d11b0 0x7fdae41f0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:13.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdaebb86700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae41f1e70 con 0x7fdae40d11b0 2024-04-06T13:29:13.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdaebb86700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae41f18b0 con 0x7fdae40cfa00 2024-04-06T13:29:13.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdaebb86700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae41f1b90 con 0x7fdae40cb870 2024-04-06T13:29:13.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdaea123700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae40d11b0 0x7fdae41f0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:13.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdaea123700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae40d11b0 0x7fdae41f0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49900/0 (socket says 172.21.15.189:49900) 2024-04-06T13:29:13.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdaea123700 1 -- 172.21.15.189:0/3613015484 learned_addr learned my addr 172.21.15.189:0/3613015484 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:13.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdae9922700 1 --2- 172.21.15.189:0/3613015484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdae40cb870 0x7fdae4152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:13.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdae9121700 1 --2- 172.21.15.189:0/3613015484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdae40cfa00 0x7fdae40c8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:13.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdaea123700 1 -- 172.21.15.189:0/3613015484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdae40cb870 msgr2=0x7fdae4152860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:13.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdaea123700 1 --2- 172.21.15.189:0/3613015484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdae40cb870 0x7fdae4152860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdaea123700 1 -- 172.21.15.189:0/3613015484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdae40cfa00 msgr2=0x7fdae40c8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:13.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdaea123700 1 --2- 172.21.15.189:0/3613015484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdae40cfa00 0x7fdae40c8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdaea123700 1 -- 172.21.15.189:0/3613015484 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdadc009a70 con 0x7fdae40d11b0 2024-04-06T13:29:13.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdae9121700 1 --2- 172.21.15.189:0/3613015484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdae40cfa00 0x7fdae40c8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:29:13.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdaea123700 1 --2- 172.21.15.189:0/3613015484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae40d11b0 0x7fdae41f0f40 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fdae40677e0 tx=0x7fdadc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:13.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdada7fc700 1 -- 172.21.15.189:0/3613015484 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc012430 con 0x7fdae40d11b0 2024-04-06T13:29:13.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdada7fc700 1 -- 172.21.15.189:0/3613015484 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdadc020070 con 0x7fdae40d11b0 2024-04-06T13:29:13.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdaebb86700 1 -- 172.21.15.189:0/3613015484 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae40c8420 con 0x7fdae40d11b0 2024-04-06T13:29:13.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdada7fc700 1 -- 172.21.15.189:0/3613015484 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc004280 con 0x7fdae40d11b0 2024-04-06T13:29:13.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.714+0000 7fdaebb86700 1 -- 172.21.15.189:0/3613015484 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae4152da0 con 0x7fdae40d11b0 2024-04-06T13:29:13.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.715+0000 7fdaebb86700 1 -- 172.21.15.189:0/3613015484 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdae4152f30 con 0x7fdae40d11b0 2024-04-06T13:29:13.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.716+0000 7fdada7fc700 1 -- 172.21.15.189:0/3613015484 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fdadc0043e0 con 0x7fdae40d11b0 2024-04-06T13:29:13.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.716+0000 7fdada7fc700 1 --2- 172.21.15.189:0/3613015484 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdac806adb0 0x7fdac806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:13.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.717+0000 7fdae9922700 1 --2- 172.21.15.189:0/3613015484 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdac806adb0 0x7fdac806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:13.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.717+0000 7fdada7fc700 1 -- 172.21.15.189:0/3613015484 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5221+0+0 (secure 0 0 0) 0x7fdadc022070 con 0x7fdae40d11b0 2024-04-06T13:29:13.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.717+0000 7fdae9922700 1 --2- 172.21.15.189:0/3613015484 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdac806adb0 0x7fdac806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdad4007fd0 tx=0x7fdad4007de0 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:13.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.721+0000 7fdada7fc700 1 -- 172.21.15.189:0/3613015484 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdae4152f30 con 0x7fdae40d11b0 2024-04-06T13:29:13.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.914+0000 7fdaebb86700 1 -- 172.21.15.189:0/3613015484 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdae40c8940 con 0x7fdae40d11b0 2024-04-06T13:29:13.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.914+0000 7fdada7fc700 1 -- 172.21.15.189:0/3613015484 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v188) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdae40c8940 con 0x7fdae40d11b0 2024-04-06T13:29:13.914 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:29:13.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.916+0000 7fdaebb86700 1 -- 172.21.15.189:0/3613015484 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdac806adb0 msgr2=0x7fdac806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:13.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.916+0000 7fdaebb86700 1 --2- 172.21.15.189:0/3613015484 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdac806adb0 0x7fdac806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdad4007fd0 tx=0x7fdad4007de0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.916+0000 7fdaebb86700 1 -- 172.21.15.189:0/3613015484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae40d11b0 msgr2=0x7fdae41f0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:13.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.916+0000 7fdaebb86700 1 --2- 172.21.15.189:0/3613015484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae40d11b0 0x7fdae41f0f40 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fdae40677e0 tx=0x7fdadc004ad0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.916+0000 7fdaebb86700 1 -- 172.21.15.189:0/3613015484 shutdown_connections 2024-04-06T13:29:13.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.916+0000 7fdaebb86700 1 --2- 172.21.15.189:0/3613015484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdae40d11b0 0x7fdae41f0f40 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.916+0000 7fdaebb86700 1 --2- 172.21.15.189:0/3613015484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdae40cb870 0x7fdae4152860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.916+0000 7fdaebb86700 1 --2- 172.21.15.189:0/3613015484 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fdac806adb0 0x7fdac806d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.916+0000 7fdaebb86700 1 --2- 172.21.15.189:0/3613015484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdae40cfa00 0x7fdae40c8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:13.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.916+0000 7fdaebb86700 1 -- 172.21.15.189:0/3613015484 >> 172.21.15.189:0/3613015484 conn(0x7fdae40b2db0 msgr2=0x7fdae415af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:13.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.916+0000 7fdaebb86700 1 -- 172.21.15.189:0/3613015484 shutdown_connections 2024-04-06T13:29:13.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:13.916+0000 7fdaebb86700 1 -- 172.21.15.189:0/3613015484 wait complete. 2024-04-06T13:29:13.929 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-06T13:29:14.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.051+0000 7ff11556d700 1 Processor -- start 2024-04-06T13:29:14.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.051+0000 7ff11556d700 1 -- start start 2024-04-06T13:29:14.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.052+0000 7ff11556d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1100b4ed0 0x7ff1100b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:14.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.052+0000 7ff11556d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff1100d2880 0x7ff1100d2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:14.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.052+0000 7ff11556d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1100b5cc0 con 0x7ff11015d590 2024-04-06T13:29:14.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.052+0000 7ff11556d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1100b5890 con 0x7ff1100d2880 2024-04-06T13:29:14.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.052+0000 7ff11556d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1100d1500 con 0x7ff1100b4ed0 2024-04-06T13:29:14.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.052+0000 7ff10f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff1100d2880 0x7ff1100d2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:14.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.052+0000 7ff10effd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff11015d590 0x7ff1100b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43022/0 (socket says 172.21.15.189:43022) 2024-04-06T13:29:14.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.052+0000 7ff10effd700 1 -- 172.21.15.189:0/3594039866 learned_addr learned my addr 172.21.15.189:0/3594039866 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:14.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.052+0000 7ff10e7fc700 1 --2- 172.21.15.189:0/3594039866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1100b4ed0 0x7ff1100b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:14.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.052+0000 7ff10f7fe700 1 -- 172.21.15.189:0/3594039866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1100b4ed0 msgr2=0x7ff1100b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:14.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.052+0000 7ff10f7fe700 1 --2- 172.21.15.189:0/3594039866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1100b4ed0 0x7ff1100b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:14.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.052+0000 7ff10f7fe700 1 -- 172.21.15.189:0/3594039866 >> v1:172.21.15.189:6789/0 conn(0x7ff11015d590 legacy=0x7ff1100b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:14.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.052+0000 7ff10dffb700 1 -- 172.21.15.189:0/3594039866 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937105048 0 0) 0x7ff1100b5cc0 con 0x7ff11015d590 2024-04-06T13:29:14.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.052+0000 7ff10f7fe700 1 -- 172.21.15.189:0/3594039866 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1100d17e0 con 0x7ff1100d2880 2024-04-06T13:29:14.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.052+0000 7ff10e7fc700 1 --2- 172.21.15.189:0/3594039866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1100b4ed0 0x7ff1100b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:29:14.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.053+0000 7ff10f7fe700 1 --2- 172.21.15.189:0/3594039866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1100d2880 0x7ff1100d2d10 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7ff108009d90 tx=0x7ff10800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26c5a66e59a5cc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:14.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.053+0000 7ff10dffb700 1 -- 172.21.15.189:0/3594039866 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10800e070 con 0x7ff1100d2880 2024-04-06T13:29:14.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.053+0000 7ff10dffb700 1 -- 172.21.15.189:0/3594039866 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10800bbd0 con 0x7ff1100d2880 2024-04-06T13:29:14.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.053+0000 7ff10dffb700 1 -- 172.21.15.189:0/3594039866 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff108012430 con 0x7ff1100d2880 2024-04-06T13:29:14.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.053+0000 7ff11556d700 1 -- 172.21.15.189:0/3594039866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1100d2880 msgr2=0x7ff1100d2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:14.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.053+0000 7ff11556d700 1 --2- 172.21.15.189:0/3594039866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1100d2880 0x7ff1100d2d10 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7ff108009d90 tx=0x7ff10800b330 comp rx=0 tx=0).stop 2024-04-06T13:29:14.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.053+0000 7ff11556d700 1 -- 172.21.15.189:0/3594039866 shutdown_connections 2024-04-06T13:29:14.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.053+0000 7ff11556d700 1 --2- 172.21.15.189:0/3594039866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1100d2880 0x7ff1100d2d10 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:14.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.053+0000 7ff11556d700 1 --2- 172.21.15.189:0/3594039866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1100b4ed0 0x7ff1100b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:14.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.053+0000 7ff11556d700 1 -- 172.21.15.189:0/3594039866 >> 172.21.15.189:0/3594039866 conn(0x7ff11006bbc0 msgr2=0x7ff11006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:14.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.053+0000 7ff11556d700 1 -- 172.21.15.189:0/3594039866 shutdown_connections 2024-04-06T13:29:14.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.053+0000 7ff11556d700 1 -- 172.21.15.189:0/3594039866 wait complete. 2024-04-06T13:29:14.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.055+0000 7ff11556d700 1 Processor -- start 2024-04-06T13:29:14.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.055+0000 7ff11556d700 1 -- start start 2024-04-06T13:29:14.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.055+0000 7ff11556d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1100b4ed0 0x7ff110159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:14.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.055+0000 7ff11556d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1100d2880 0x7ff110158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:14.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.055+0000 7ff11556d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff11015d590 0x7ff1100b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:14.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.055+0000 7ff11556d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff110158ff0 con 0x7ff11015d590 2024-04-06T13:29:14.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.055+0000 7ff11556d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff110158d90 con 0x7ff1100b4ed0 2024-04-06T13:29:14.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.055+0000 7ff11556d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1100cba70 con 0x7ff1100d2880 2024-04-06T13:29:14.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.056+0000 7ff10f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff11015d590 0x7ff1100b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:14.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.056+0000 7ff10f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff11015d590 0x7ff1100b8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49930/0 (socket says 172.21.15.189:49930) 2024-04-06T13:29:14.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.056+0000 7ff10f7fe700 1 -- 172.21.15.189:0/1425731750 learned_addr learned my addr 172.21.15.189:0/1425731750 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:14.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.056+0000 7ff10e7fc700 1 --2- 172.21.15.189:0/1425731750 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1100d2880 0x7ff110158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:14.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.056+0000 7ff10effd700 1 --2- 172.21.15.189:0/1425731750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1100b4ed0 0x7ff110159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:14.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.056+0000 7ff10f7fe700 1 -- 172.21.15.189:0/1425731750 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1100d2880 msgr2=0x7ff110158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:14.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.056+0000 7ff10f7fe700 1 --2- 172.21.15.189:0/1425731750 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1100d2880 0x7ff110158850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:14.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.056+0000 7ff10f7fe700 1 -- 172.21.15.189:0/1425731750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1100b4ed0 msgr2=0x7ff110159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:14.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.056+0000 7ff10f7fe700 1 --2- 172.21.15.189:0/1425731750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1100b4ed0 0x7ff110159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:14.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.056+0000 7ff10f7fe700 1 -- 172.21.15.189:0/1425731750 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff108009a70 con 0x7ff11015d590 2024-04-06T13:29:14.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.056+0000 7ff10effd700 1 --2- 172.21.15.189:0/1425731750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1100b4ed0 0x7ff110159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:29:14.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.056+0000 7ff10f7fe700 1 --2- 172.21.15.189:0/1425731750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff11015d590 0x7ff1100b8aa0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7ff108014040 tx=0x7ff108004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:14.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.056+0000 7ff0f77fe700 1 -- 172.21.15.189:0/1425731750 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff108012430 con 0x7ff11015d590 2024-04-06T13:29:14.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.056+0000 7ff0f77fe700 1 -- 172.21.15.189:0/1425731750 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff108020070 con 0x7ff11015d590 2024-04-06T13:29:14.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.056+0000 7ff11556d700 1 -- 172.21.15.189:0/1425731750 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1100cacb0 con 0x7ff11015d590 2024-04-06T13:29:14.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.056+0000 7ff0f77fe700 1 -- 172.21.15.189:0/1425731750 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff108004280 con 0x7ff11015d590 2024-04-06T13:29:14.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.056+0000 7ff11556d700 1 -- 172.21.15.189:0/1425731750 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1100cb270 con 0x7ff11015d590 2024-04-06T13:29:14.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.057+0000 7ff11556d700 1 -- 172.21.15.189:0/1425731750 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff110074990 con 0x7ff11015d590 2024-04-06T13:29:14.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.058+0000 7ff0f77fe700 1 -- 172.21.15.189:0/1425731750 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7ff1080043e0 con 0x7ff11015d590 2024-04-06T13:29:14.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.058+0000 7ff0f77fe700 1 --2- 172.21.15.189:0/1425731750 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff0fc06adb0 0x7ff0fc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:14.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.058+0000 7ff0f77fe700 1 -- 172.21.15.189:0/1425731750 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5221+0+0 (secure 0 0 0) 0x7ff108022070 con 0x7ff11015d590 2024-04-06T13:29:14.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.058+0000 7ff10effd700 1 --2- 172.21.15.189:0/1425731750 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff0fc06adb0 0x7ff0fc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:14.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.058+0000 7ff10effd700 1 --2- 172.21.15.189:0/1425731750 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff0fc06adb0 0x7ff0fc06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff104000f60 tx=0x7ff104007ca0 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:14.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.062+0000 7ff0f77fe700 1 -- 172.21.15.189:0/1425731750 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff1080563e0 con 0x7ff11015d590 2024-04-06T13:29:14.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.258+0000 7ff11556d700 1 -- 172.21.15.189:0/1425731750 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff1100cdd20 con 0x7ff11015d590 2024-04-06T13:29:14.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.258+0000 7ff0f77fe700 1 -- 172.21.15.189:0/1425731750 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v188) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff10802c060 con 0x7ff11015d590 2024-04-06T13:29:14.258 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:29:14.258 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T13:29:14.258 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:14.258 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 1, 2024-04-06T13:29:14.258 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": ".mgr" 2024-04-06T13:29:14.259 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:14.259 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T13:29:14.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.260+0000 7ff11556d700 1 -- 172.21.15.189:0/1425731750 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff0fc06adb0 msgr2=0x7ff0fc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:14.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.260+0000 7ff11556d700 1 --2- 172.21.15.189:0/1425731750 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff0fc06adb0 0x7ff0fc06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff104000f60 tx=0x7ff104007ca0 comp rx=0 tx=0).stop 2024-04-06T13:29:14.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.260+0000 7ff11556d700 1 -- 172.21.15.189:0/1425731750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff11015d590 msgr2=0x7ff1100b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:14.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.260+0000 7ff11556d700 1 --2- 172.21.15.189:0/1425731750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff11015d590 0x7ff1100b8aa0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7ff108014040 tx=0x7ff108004ad0 comp rx=0 tx=0).stop 2024-04-06T13:29:14.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.260+0000 7ff11556d700 1 -- 172.21.15.189:0/1425731750 shutdown_connections 2024-04-06T13:29:14.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.260+0000 7ff11556d700 1 --2- 172.21.15.189:0/1425731750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff11015d590 0x7ff1100b8aa0 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:14.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.260+0000 7ff11556d700 1 --2- 172.21.15.189:0/1425731750 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff0fc06adb0 0x7ff0fc06d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:14.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.260+0000 7ff11556d700 1 --2- 172.21.15.189:0/1425731750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1100b4ed0 0x7ff110159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:14.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.260+0000 7ff11556d700 1 --2- 172.21.15.189:0/1425731750 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1100d2880 0x7ff110158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:14.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.260+0000 7ff11556d700 1 -- 172.21.15.189:0/1425731750 >> 172.21.15.189:0/1425731750 conn(0x7ff11006bbc0 msgr2=0x7ff11015c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:14.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.260+0000 7ff11556d700 1 -- 172.21.15.189:0/1425731750 shutdown_connections 2024-04-06T13:29:14.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.260+0000 7ff11556d700 1 -- 172.21.15.189:0/1425731750 wait complete. 2024-04-06T13:29:14.272 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-06T13:29:14.273 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-06T13:29:14.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.398+0000 7f870e198700 1 Processor -- start 2024-04-06T13:29:14.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.398+0000 7f870e198700 1 -- start start 2024-04-06T13:29:14.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.399+0000 7f870e198700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8708158fe0 0x7f870806a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:14.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.399+0000 7f870e198700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f870806ac70 0x7f870815bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:14.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.399+0000 7f870e198700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f870815c100 0x7f87080717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:14.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.399+0000 7f870e198700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f870806b0f0 con 0x7f870806ac70 2024-04-06T13:29:14.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.399+0000 7f870e198700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f870815f9e0 con 0x7f8708158fe0 2024-04-06T13:29:14.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.399+0000 7f870e198700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870815f0d0 con 0x7f870815c100 2024-04-06T13:29:14.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.399+0000 7f8706ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f870806ac70 0x7f870815bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:14.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.399+0000 7f8706ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f870806ac70 0x7f870815bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49944/0 (socket says 172.21.15.189:49944) 2024-04-06T13:29:14.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.399+0000 7f8706ffd700 1 -- 172.21.15.189:0/3837494752 learned_addr learned my addr 172.21.15.189:0/3837494752 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:14.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.399+0000 7f87077fe700 1 --2- 172.21.15.189:0/3837494752 >> v2:172.21.15.192:3300/0 conn(0x7f8708158fe0 0x7f870806a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:14.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.399+0000 7f8707fff700 1 --2- 172.21.15.189:0/3837494752 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f870815c100 0x7f87080717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:14.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.399+0000 7f8706ffd700 1 -- 172.21.15.189:0/3837494752 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f870815c100 msgr2=0x7f87080717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:14.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.399+0000 7f8706ffd700 1 --2- 172.21.15.189:0/3837494752 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f870815c100 0x7f87080717b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:14.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.399+0000 7f8706ffd700 1 -- 172.21.15.189:0/3837494752 >> v2:172.21.15.192:3300/0 conn(0x7f8708158fe0 msgr2=0x7f870806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:14.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.399+0000 7f8706ffd700 1 --2- 172.21.15.189:0/3837494752 >> v2:172.21.15.192:3300/0 conn(0x7f8708158fe0 0x7f870806a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:14.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.399+0000 7f8706ffd700 1 -- 172.21.15.189:0/3837494752 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870815c8f0 con 0x7f870806ac70 2024-04-06T13:29:14.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.400+0000 7f8706ffd700 1 --2- 172.21.15.189:0/3837494752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f870806ac70 0x7f870815bbc0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f8708073470 tx=0x7f86f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d7cb7ddb830a07 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:14.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.400+0000 7f87067fc700 1 -- 172.21.15.189:0/3837494752 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f800b870 con 0x7f870806ac70 2024-04-06T13:29:14.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.400+0000 7f87067fc700 1 -- 172.21.15.189:0/3837494752 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f800b9d0 con 0x7f870806ac70 2024-04-06T13:29:14.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.400+0000 7f87067fc700 1 -- 172.21.15.189:0/3837494752 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f800bd60 con 0x7f870806ac70 2024-04-06T13:29:14.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.400+0000 7f870e198700 1 -- 172.21.15.189:0/3837494752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f870806ac70 msgr2=0x7f870815bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:14.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.400+0000 7f870e198700 1 --2- 172.21.15.189:0/3837494752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f870806ac70 0x7f870815bbc0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f8708073470 tx=0x7f86f8009bd0 comp rx=0 tx=0).stop 2024-04-06T13:29:14.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.400+0000 7f870e198700 1 -- 172.21.15.189:0/3837494752 shutdown_connections 2024-04-06T13:29:14.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.400+0000 7f870e198700 1 --2- 172.21.15.189:0/3837494752 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f870815c100 0x7f87080717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:14.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.400+0000 7f870e198700 1 --2- 172.21.15.189:0/3837494752 >> v2:172.21.15.192:3300/0 conn(0x7f8708158fe0 0x7f870806a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:14.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.400+0000 7f870e198700 1 --2- 172.21.15.189:0/3837494752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f870806ac70 0x7f870815bbc0 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:14.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.400+0000 7f870e198700 1 -- 172.21.15.189:0/3837494752 >> 172.21.15.189:0/3837494752 conn(0x7f87080d00b0 msgr2=0x7f87080d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:14.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.400+0000 7f870e198700 1 -- 172.21.15.189:0/3837494752 shutdown_connections 2024-04-06T13:29:14.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.400+0000 7f870e198700 1 -- 172.21.15.189:0/3837494752 wait complete. 2024-04-06T13:29:14.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.401+0000 7f870e198700 1 Processor -- start 2024-04-06T13:29:14.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.402+0000 7f870e198700 1 -- start start 2024-04-06T13:29:14.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.402+0000 7f870e198700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f870806ac70 0x7f870813cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:14.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.402+0000 7f870e198700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8708158fe0 0x7f870813d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:14.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.402+0000 7f870e198700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f870815c100 0x7f87081f0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.402+0000 7f870e198700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870813ac20 con 0x7f870815c100 2024-04-06T13:29:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.402+0000 7f870e198700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870813af00 con 0x7f870806ac70 2024-04-06T13:29:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.402+0000 7f870e198700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870813b1e0 con 0x7f8708158fe0 2024-04-06T13:29:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.403+0000 7f8707fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f870815c100 0x7f87081f0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.403+0000 7f8707fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f870815c100 0x7f87081f0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49954/0 (socket says 172.21.15.189:49954) 2024-04-06T13:29:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.403+0000 7f8707fff700 1 -- 172.21.15.189:0/467017304 learned_addr learned my addr 172.21.15.189:0/467017304 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.403+0000 7f87077fe700 1 --2- 172.21.15.189:0/467017304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f870806ac70 0x7f870813cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.403+0000 7f8706ffd700 1 --2- 172.21.15.189:0/467017304 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8708158fe0 0x7f870813d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.403+0000 7f8707fff700 1 -- 172.21.15.189:0/467017304 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8708158fe0 msgr2=0x7f870813d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.403+0000 7f8707fff700 1 --2- 172.21.15.189:0/467017304 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8708158fe0 0x7f870813d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.403+0000 7f8707fff700 1 -- 172.21.15.189:0/467017304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f870806ac70 msgr2=0x7f870813cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.403+0000 7f8707fff700 1 --2- 172.21.15.189:0/467017304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f870806ac70 0x7f870813cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.403+0000 7f8707fff700 1 -- 172.21.15.189:0/467017304 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f8009550 con 0x7f870815c100 2024-04-06T13:29:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.403+0000 7f87077fe700 1 --2- 172.21.15.189:0/467017304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f870806ac70 0x7f870813cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:29:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.403+0000 7f8706ffd700 1 --2- 172.21.15.189:0/467017304 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8708158fe0 0x7f870813d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:29:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.403+0000 7f8707fff700 1 --2- 172.21.15.189:0/467017304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f870815c100 0x7f87081f0fa0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f8700007e40 tx=0x7f870000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.403+0000 7f86effff700 1 -- 172.21.15.189:0/467017304 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870000eea0 con 0x7f870815c100 2024-04-06T13:29:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.403+0000 7f86effff700 1 -- 172.21.15.189:0/467017304 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8700014420 con 0x7f870815c100 2024-04-06T13:29:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.403+0000 7f870e198700 1 -- 172.21.15.189:0/467017304 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f870813b4c0 con 0x7f870815c100 2024-04-06T13:29:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.403+0000 7f86effff700 1 -- 172.21.15.189:0/467017304 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87000147b0 con 0x7f870815c100 2024-04-06T13:29:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.403+0000 7f870e198700 1 -- 172.21.15.189:0/467017304 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f870813ba80 con 0x7f870815c100 2024-04-06T13:29:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.404+0000 7f870e198700 1 -- 172.21.15.189:0/467017304 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8708067bb0 con 0x7f870815c100 2024-04-06T13:29:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.405+0000 7f86effff700 1 -- 172.21.15.189:0/467017304 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f8700014910 con 0x7f870815c100 2024-04-06T13:29:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.405+0000 7f86effff700 1 --2- 172.21.15.189:0/467017304 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86f406adb0 0x7f86f406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.405+0000 7f86effff700 1 -- 172.21.15.189:0/467017304 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f8700089b70 con 0x7f870815c100 2024-04-06T13:29:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.405+0000 7f87077fe700 1 --2- 172.21.15.189:0/467017304 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86f406adb0 0x7f86f406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.405+0000 7f87077fe700 1 --2- 172.21.15.189:0/467017304 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86f406adb0 0x7f86f406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009970 tx=0x7f86fc008040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:14.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.409+0000 7f86effff700 1 -- 172.21.15.189:0/467017304 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8700054da0 con 0x7f870815c100 2024-04-06T13:29:14.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:14.605+0000 7f870e198700 1 -- 172.21.15.189:0/467017304 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8708074990 con 0x7f870815c100 2024-04-06T13:29:15.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.444+0000 7f86effff700 1 -- 172.21.15.189:0/467017304 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v189) v1 ==== 138+0+0 (secure 0 0 0) 0x7f87000583c0 con 0x7f870815c100 2024-04-06T13:29:15.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.451+0000 7f870e198700 1 -- 172.21.15.189:0/467017304 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86f406adb0 msgr2=0x7f86f406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:15.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.451+0000 7f870e198700 1 --2- 172.21.15.189:0/467017304 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86f406adb0 0x7f86f406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009970 tx=0x7f86fc008040 comp rx=0 tx=0).stop 2024-04-06T13:29:15.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.451+0000 7f870e198700 1 -- 172.21.15.189:0/467017304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f870815c100 msgr2=0x7f87081f0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:15.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.451+0000 7f870e198700 1 --2- 172.21.15.189:0/467017304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f870815c100 0x7f87081f0fa0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f8700007e40 tx=0x7f870000e3c0 comp rx=0 tx=0).stop 2024-04-06T13:29:15.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.451+0000 7f870e198700 1 -- 172.21.15.189:0/467017304 shutdown_connections 2024-04-06T13:29:15.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.451+0000 7f870e198700 1 --2- 172.21.15.189:0/467017304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f870815c100 0x7f87081f0fa0 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:15.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.451+0000 7f870e198700 1 --2- 172.21.15.189:0/467017304 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f86f406adb0 0x7f86f406d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:15.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.451+0000 7f870e198700 1 --2- 172.21.15.189:0/467017304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f870806ac70 0x7f870813cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:15.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.451+0000 7f870e198700 1 --2- 172.21.15.189:0/467017304 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8708158fe0 0x7f870813d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:15.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.451+0000 7f870e198700 1 -- 172.21.15.189:0/467017304 >> 172.21.15.189:0/467017304 conn(0x7f87080d00b0 msgr2=0x7f87080ccac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:15.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.451+0000 7f870e198700 1 -- 172.21.15.189:0/467017304 shutdown_connections 2024-04-06T13:29:15.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.451+0000 7f870e198700 1 -- 172.21.15.189:0/467017304 wait complete. 2024-04-06T13:29:15.453 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' created 2024-04-06T13:29:15.466 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-06T13:29:15.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.602+0000 7f621c5ea700 1 Processor -- start 2024-04-06T13:29:15.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.602+0000 7f621c5ea700 1 -- start start 2024-04-06T13:29:15.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.603+0000 7f621c5ea700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f621415d7b0 0x7f62140b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:15.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.603+0000 7f621c5ea700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f621415e500 0x7f621406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:15.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.603+0000 7f621c5ea700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62140b3990 con 0x7f621406a730 2024-04-06T13:29:15.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.603+0000 7f621c5ea700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f621406b0f0 con 0x7f621415e500 2024-04-06T13:29:15.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.603+0000 7f621c5ea700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62140b3060 con 0x7f621415d7b0 2024-04-06T13:29:15.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.603+0000 7f6219b85700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f621406a730 0x7f621415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43038/0 (socket says 172.21.15.189:43038) 2024-04-06T13:29:15.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.603+0000 7f6219b85700 1 -- 172.21.15.189:0/340258892 learned_addr learned my addr 172.21.15.189:0/340258892 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:15.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.603+0000 7f621a386700 1 --2- 172.21.15.189:0/340258892 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f621415d7b0 0x7f62140b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:15.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.603+0000 7f621ab87700 1 --2- 172.21.15.189:0/340258892 >> v2:172.21.15.192:3300/0 conn(0x7f621415e500 0x7f621406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:15.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.603+0000 7f6219384700 1 -- 172.21.15.189:0/340258892 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007498554 0 0) 0x7f62140b3990 con 0x7f621406a730 2024-04-06T13:29:15.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.603+0000 7f6219384700 1 -- 172.21.15.189:0/340258892 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62000035b0 con 0x7f621406a730 2024-04-06T13:29:15.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.603+0000 7f621a386700 1 -- 172.21.15.189:0/340258892 >> v2:172.21.15.192:3300/0 conn(0x7f621415e500 msgr2=0x7f621406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:15.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.603+0000 7f621a386700 1 --2- 172.21.15.189:0/340258892 >> v2:172.21.15.192:3300/0 conn(0x7f621415e500 0x7f621406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:15.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.603+0000 7f621a386700 1 -- 172.21.15.189:0/340258892 >> v1:172.21.15.189:6789/0 conn(0x7f621406a730 legacy=0x7f621415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:15.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.603+0000 7f621a386700 1 -- 172.21.15.189:0/340258892 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62140b3340 con 0x7f621415d7b0 2024-04-06T13:29:15.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.603+0000 7f621ab87700 1 --2- 172.21.15.189:0/340258892 >> v2:172.21.15.192:3300/0 conn(0x7f621415e500 0x7f621406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:29:15.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.604+0000 7f621a386700 1 --2- 172.21.15.189:0/340258892 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f621415d7b0 0x7f62140b3e20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6210009840 tx=0x7f6210009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d8ec1e9f88021cb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:15.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.604+0000 7f6219384700 1 -- 172.21.15.189:0/340258892 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6210011830 con 0x7f621415d7b0 2024-04-06T13:29:15.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.604+0000 7f6219384700 1 -- 172.21.15.189:0/340258892 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6210011990 con 0x7f621415d7b0 2024-04-06T13:29:15.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.604+0000 7f6219384700 1 -- 172.21.15.189:0/340258892 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6210011d00 con 0x7f621415d7b0 2024-04-06T13:29:15.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.604+0000 7f621c5ea700 1 -- 172.21.15.189:0/340258892 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f621415d7b0 msgr2=0x7f62140b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:15.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.604+0000 7f621c5ea700 1 --2- 172.21.15.189:0/340258892 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f621415d7b0 0x7f62140b3e20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6210009840 tx=0x7f6210009b50 comp rx=0 tx=0).stop 2024-04-06T13:29:15.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.604+0000 7f621c5ea700 1 -- 172.21.15.189:0/340258892 shutdown_connections 2024-04-06T13:29:15.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.604+0000 7f621c5ea700 1 --2- 172.21.15.189:0/340258892 >> v2:172.21.15.192:3300/0 conn(0x7f621415e500 0x7f621406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:15.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.604+0000 7f621c5ea700 1 --2- 172.21.15.189:0/340258892 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f621415d7b0 0x7f62140b3e20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:15.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.604+0000 7f621c5ea700 1 -- 172.21.15.189:0/340258892 >> 172.21.15.189:0/340258892 conn(0x7f62140d00b0 msgr2=0x7f62140d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:15.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.604+0000 7f621c5ea700 1 -- 172.21.15.189:0/340258892 shutdown_connections 2024-04-06T13:29:15.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.604+0000 7f621c5ea700 1 -- 172.21.15.189:0/340258892 wait complete. 2024-04-06T13:29:15.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.606+0000 7f621c5ea700 1 Processor -- start 2024-04-06T13:29:15.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.606+0000 7f621c5ea700 1 -- start start 2024-04-06T13:29:15.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.606+0000 7f621c5ea700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f621406a730 0x7f621415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:15.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.606+0000 7f621c5ea700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f621415d7b0 0x7f621415c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:15.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.606+0000 7f621c5ea700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f621415e500 0x7f621415c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:15.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.606+0000 7f621c5ea700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62140cda10 con 0x7f621406a730 2024-04-06T13:29:15.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.606+0000 7f621c5ea700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621415ab30 con 0x7f621415d7b0 2024-04-06T13:29:15.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.606+0000 7f621c5ea700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f621415ae10 con 0x7f621415e500 2024-04-06T13:29:15.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.606+0000 7f621a386700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f621406a730 0x7f621415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:15.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.606+0000 7f621a386700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f621406a730 0x7f621415a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49968/0 (socket says 172.21.15.189:49968) 2024-04-06T13:29:15.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.606+0000 7f621a386700 1 -- 172.21.15.189:0/2082611519 learned_addr learned my addr 172.21.15.189:0/2082611519 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:15.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.607+0000 7f6219b85700 1 --2- 172.21.15.189:0/2082611519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f621415d7b0 0x7f621415c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:15.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.607+0000 7f621ab87700 1 --2- 172.21.15.189:0/2082611519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f621415e500 0x7f621415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:15.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.607+0000 7f621a386700 1 -- 172.21.15.189:0/2082611519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f621415e500 msgr2=0x7f621415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:15.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.607+0000 7f621a386700 1 --2- 172.21.15.189:0/2082611519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f621415e500 0x7f621415c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:15.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.607+0000 7f621a386700 1 -- 172.21.15.189:0/2082611519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f621415d7b0 msgr2=0x7f621415c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:15.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.607+0000 7f621a386700 1 --2- 172.21.15.189:0/2082611519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f621415d7b0 0x7f621415c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:15.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.607+0000 7f621a386700 1 -- 172.21.15.189:0/2082611519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620c009a70 con 0x7f621406a730 2024-04-06T13:29:15.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.607+0000 7f621a386700 1 --2- 172.21.15.189:0/2082611519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f621406a730 0x7f621415a2e0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f62100090b0 tx=0x7f6210004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:15.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.607+0000 7f620affd700 1 -- 172.21.15.189:0/2082611519 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62100036a0 con 0x7f621406a730 2024-04-06T13:29:15.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.607+0000 7f620affd700 1 -- 172.21.15.189:0/2082611519 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621001f070 con 0x7f621406a730 2024-04-06T13:29:15.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.607+0000 7f621c5ea700 1 -- 172.21.15.189:0/2082611519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62100094a0 con 0x7f621406a730 2024-04-06T13:29:15.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.607+0000 7f620affd700 1 -- 172.21.15.189:0/2082611519 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6210003a00 con 0x7f621406a730 2024-04-06T13:29:15.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.607+0000 7f621c5ea700 1 -- 172.21.15.189:0/2082611519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f621415b3d0 con 0x7f621406a730 2024-04-06T13:29:15.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.608+0000 7f621c5ea700 1 -- 172.21.15.189:0/2082611519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f621415b560 con 0x7f621406a730 2024-04-06T13:29:15.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.608+0000 7f620affd700 1 -- 172.21.15.189:0/2082611519 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f6210003b60 con 0x7f621406a730 2024-04-06T13:29:15.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.609+0000 7f620affd700 1 --2- 172.21.15.189:0/2082611519 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f61f806adb0 0x7f61f806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:15.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.609+0000 7f620affd700 1 -- 172.21.15.189:0/2082611519 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5591+0+0 (secure 0 0 0) 0x7f6210021070 con 0x7f621406a730 2024-04-06T13:29:15.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.609+0000 7f6219b85700 1 --2- 172.21.15.189:0/2082611519 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f61f806adb0 0x7f61f806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:15.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.609+0000 7f6219b85700 1 --2- 172.21.15.189:0/2082611519 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f61f806adb0 0x7f61f806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f621415ba80 tx=0x7f6204009040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:15.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.614+0000 7f620affd700 1 -- 172.21.15.189:0/2082611519 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6210055f00 con 0x7f621406a730 2024-04-06T13:29:15.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:15.807+0000 7f621c5ea700 1 -- 172.21.15.189:0/2082611519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f6214067bb0 con 0x7f621406a730 2024-04-06T13:29:16.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.448+0000 7f620affd700 1 -- 172.21.15.189:0/2082611519 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v190) v1 ==== 170+0+0 (secure 0 0 0) 0x7f621002b020 con 0x7f621406a730 2024-04-06T13:29:16.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.459+0000 7f621c5ea700 1 -- 172.21.15.189:0/2082611519 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f61f806adb0 msgr2=0x7f61f806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:16.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.459+0000 7f621c5ea700 1 --2- 172.21.15.189:0/2082611519 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f61f806adb0 0x7f61f806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f621415ba80 tx=0x7f6204009040 comp rx=0 tx=0).stop 2024-04-06T13:29:16.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.459+0000 7f621c5ea700 1 -- 172.21.15.189:0/2082611519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f621406a730 msgr2=0x7f621415a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:16.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.459+0000 7f621c5ea700 1 --2- 172.21.15.189:0/2082611519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f621406a730 0x7f621415a2e0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f62100090b0 tx=0x7f6210004750 comp rx=0 tx=0).stop 2024-04-06T13:29:16.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.460+0000 7f621c5ea700 1 -- 172.21.15.189:0/2082611519 shutdown_connections 2024-04-06T13:29:16.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.460+0000 7f621c5ea700 1 --2- 172.21.15.189:0/2082611519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f621415e500 0x7f621415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:16.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.460+0000 7f621c5ea700 1 --2- 172.21.15.189:0/2082611519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f621406a730 0x7f621415a2e0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:16.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.460+0000 7f621c5ea700 1 --2- 172.21.15.189:0/2082611519 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f61f806adb0 0x7f61f806d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:16.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.460+0000 7f621c5ea700 1 --2- 172.21.15.189:0/2082611519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f621415d7b0 0x7f621415c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:16.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.460+0000 7f621c5ea700 1 -- 172.21.15.189:0/2082611519 >> 172.21.15.189:0/2082611519 conn(0x7f62140d00b0 msgr2=0x7f621406f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:16.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.460+0000 7f621c5ea700 1 -- 172.21.15.189:0/2082611519 shutdown_connections 2024-04-06T13:29:16.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.460+0000 7f621c5ea700 1 -- 172.21.15.189:0/2082611519 wait complete. 2024-04-06T13:29:16.462 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' created 2024-04-06T13:29:16.478 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-06T13:29:16.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.605+0000 7fa044cfd700 1 Processor -- start 2024-04-06T13:29:16.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.605+0000 7fa044cfd700 1 -- start start 2024-04-06T13:29:16.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.606+0000 7fa044cfd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa0400b4af0 0x7fa0400b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:16.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.606+0000 7fa044cfd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0400cdbe0 con 0x7fa0400b4660 2024-04-06T13:29:16.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.606+0000 7fa044cfd700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa04015a940 con 0x7fa04015a550 2024-04-06T13:29:16.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.606+0000 7fa044cfd700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa04015ac00 con 0x7fa0400b4af0 2024-04-06T13:29:16.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.606+0000 7fa03dd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa0400b4660 0x7fa0400b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43042/0 (socket says 172.21.15.189:43042) 2024-04-06T13:29:16.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.606+0000 7fa03dd9b700 1 -- 172.21.15.189:0/2396115189 learned_addr learned my addr 172.21.15.189:0/2396115189 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:16.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.606+0000 7fa03d59a700 1 -- 172.21.15.189:0/2396115189 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495238324 0 0) 0x7fa0400cdbe0 con 0x7fa0400b4660 2024-04-06T13:29:16.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.606+0000 7fa03ed9d700 1 --2- 172.21.15.189:0/2396115189 >> v2:172.21.15.192:3300/0 conn(0x7fa0400b4af0 0x7fa0400b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:16.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.606+0000 7fa03d59a700 1 -- 172.21.15.189:0/2396115189 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0280035b0 con 0x7fa0400b4660 2024-04-06T13:29:16.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.606+0000 7fa03d59a700 1 -- 172.21.15.189:0/2396115189 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3413910003 0 0) 0x7fa0280035b0 con 0x7fa0400b4660 2024-04-06T13:29:16.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03d59a700 1 -- 172.21.15.189:0/2396115189 >> v2:172.21.15.192:3300/0 conn(0x7fa0400b4af0 msgr2=0x7fa0400b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:16.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03d59a700 1 --2- 172.21.15.189:0/2396115189 >> v2:172.21.15.192:3300/0 conn(0x7fa0400b4af0 0x7fa0400b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:16.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03d59a700 1 -- 172.21.15.189:0/2396115189 >> v1:172.21.15.192:6789/0 conn(0x7fa04015a550 legacy=0x7fa0400cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:16.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03d59a700 1 -- 172.21.15.189:0/2396115189 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04015aee0 con 0x7fa0400b4660 2024-04-06T13:29:16.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03d59a700 1 -- 172.21.15.189:0/2396115189 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fa030001670 con 0x7fa0400b4660 2024-04-06T13:29:16.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03d59a700 1 -- 172.21.15.189:0/2396115189 >> v1:172.21.15.189:6789/0 conn(0x7fa0400b4660 legacy=0x7fa0400b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:16.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03d59a700 1 --2- 172.21.15.189:0/2396115189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa028003a00 0x7fa028003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:16.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03d59a700 1 --2- 172.21.15.189:0/2396115189 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa028004850 0x7fa028004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:16.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03d59a700 1 --2- 172.21.15.189:0/2396115189 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa028005210 0x7fa02800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:16.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03d59a700 1 -- 172.21.15.189:0/2396115189 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04015ac00 con 0x7fa028003a00 2024-04-06T13:29:16.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03d59a700 1 -- 172.21.15.189:0/2396115189 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02800c0b0 con 0x7fa028005210 2024-04-06T13:29:16.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03d59a700 1 -- 172.21.15.189:0/2396115189 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa02800c270 con 0x7fa028004850 2024-04-06T13:29:16.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03e59c700 1 --2- 172.21.15.189:0/2396115189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa028003a00 0x7fa028003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:16.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03ed9d700 1 --2- 172.21.15.189:0/2396115189 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa028005210 0x7fa02800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:16.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03dd9b700 1 --2- 172.21.15.189:0/2396115189 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa028004850 0x7fa028004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:16.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03e59c700 1 -- 172.21.15.189:0/2396115189 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa028004850 msgr2=0x7fa028004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:16.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03e59c700 1 --2- 172.21.15.189:0/2396115189 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa028004850 0x7fa028004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:16.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03e59c700 1 -- 172.21.15.189:0/2396115189 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa028005210 msgr2=0x7fa02800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:16.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03e59c700 1 --2- 172.21.15.189:0/2396115189 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa028005210 0x7fa02800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:16.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03e59c700 1 -- 172.21.15.189:0/2396115189 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0280035b0 con 0x7fa028003a00 2024-04-06T13:29:16.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03ed9d700 1 -- 172.21.15.189:0/2396115189 reap_dead start 2024-04-06T13:29:16.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03e59c700 1 --2- 172.21.15.189:0/2396115189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa028003a00 0x7fa028003e20 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fa0340017c0 tx=0x7fa028005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89af1b4de7992752 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:16.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03d59a700 1 -- 172.21.15.189:0/2396115189 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0340052d0 con 0x7fa028003a00 2024-04-06T13:29:16.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03d59a700 1 -- 172.21.15.189:0/2396115189 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa034005450 con 0x7fa028003a00 2024-04-06T13:29:16.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.607+0000 7fa03d59a700 1 -- 172.21.15.189:0/2396115189 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0340038c0 con 0x7fa028003a00 2024-04-06T13:29:16.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.608+0000 7fa044cfd700 1 -- 172.21.15.189:0/2396115189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa028003a00 msgr2=0x7fa028003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:16.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.608+0000 7fa044cfd700 1 --2- 172.21.15.189:0/2396115189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa028003a00 0x7fa028003e20 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fa0340017c0 tx=0x7fa028005880 comp rx=0 tx=0).stop 2024-04-06T13:29:16.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.608+0000 7fa044cfd700 1 -- 172.21.15.189:0/2396115189 shutdown_connections 2024-04-06T13:29:16.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.608+0000 7fa044cfd700 1 --2- 172.21.15.189:0/2396115189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa028003a00 0x7fa028003e20 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:16.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.608+0000 7fa044cfd700 1 -- 172.21.15.189:0/2396115189 >> 172.21.15.189:0/2396115189 conn(0x7fa0400d00b0 msgr2=0x7fa0400d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:16.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.608+0000 7fa044cfd700 1 -- 172.21.15.189:0/2396115189 shutdown_connections 2024-04-06T13:29:16.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.608+0000 7fa044cfd700 1 -- 172.21.15.189:0/2396115189 wait complete. 2024-04-06T13:29:16.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.609+0000 7fa044cfd700 1 Processor -- start 2024-04-06T13:29:16.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.609+0000 7fa044cfd700 1 -- start start 2024-04-06T13:29:16.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.610+0000 7fa044cfd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa028003a00 0x7fa040154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:16.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.610+0000 7fa044cfd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0401f0670 0x7fa040152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:16.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.610+0000 7fa044cfd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0401529d0 0x7fa0400b1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:16.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.610+0000 7fa044cfd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa040153bd0 con 0x7fa028003a00 2024-04-06T13:29:16.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.610+0000 7fa044cfd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0401f0af0 con 0x7fa0401529d0 2024-04-06T13:29:16.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.610+0000 7fa044cfd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa040152fc0 con 0x7fa0401f0670 2024-04-06T13:29:16.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.610+0000 7fa03e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa028003a00 0x7fa040154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:16.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.610+0000 7fa03e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa028003a00 0x7fa040154060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49990/0 (socket says 172.21.15.189:49990) 2024-04-06T13:29:16.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.610+0000 7fa03e59c700 1 -- 172.21.15.189:0/657401450 learned_addr learned my addr 172.21.15.189:0/657401450 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:16.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.610+0000 7fa03dd9b700 1 --2- 172.21.15.189:0/657401450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0401f0670 0x7fa040152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:16.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.610+0000 7fa03ed9d700 1 --2- 172.21.15.189:0/657401450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0401529d0 0x7fa0400b1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:16.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.610+0000 7fa03e59c700 1 -- 172.21.15.189:0/657401450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0401f0670 msgr2=0x7fa040152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:16.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.610+0000 7fa03e59c700 1 --2- 172.21.15.189:0/657401450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0401f0670 0x7fa040152490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:16.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.610+0000 7fa03e59c700 1 -- 172.21.15.189:0/657401450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0401529d0 msgr2=0x7fa0400b1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:16.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.610+0000 7fa03e59c700 1 --2- 172.21.15.189:0/657401450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0401529d0 0x7fa0400b1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:16.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.610+0000 7fa03e59c700 1 -- 172.21.15.189:0/657401450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0401532a0 con 0x7fa028003a00 2024-04-06T13:29:16.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.610+0000 7fa03ed9d700 1 --2- 172.21.15.189:0/657401450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0401529d0 0x7fa0400b1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:29:16.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.611+0000 7fa03e59c700 1 --2- 172.21.15.189:0/657401450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa028003a00 0x7fa040154060 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fa034000c00 tx=0x7fa034001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:16.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.611+0000 7fa02effd700 1 -- 172.21.15.189:0/657401450 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa034001290 con 0x7fa028003a00 2024-04-06T13:29:16.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.611+0000 7fa02effd700 1 -- 172.21.15.189:0/657401450 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa034007040 con 0x7fa028003a00 2024-04-06T13:29:16.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.611+0000 7fa044cfd700 1 -- 172.21.15.189:0/657401450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa040153580 con 0x7fa028003a00 2024-04-06T13:29:16.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.611+0000 7fa02effd700 1 -- 172.21.15.189:0/657401450 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0340073d0 con 0x7fa028003a00 2024-04-06T13:29:16.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.611+0000 7fa044cfd700 1 -- 172.21.15.189:0/657401450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa040144180 con 0x7fa028003a00 2024-04-06T13:29:16.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.612+0000 7fa044cfd700 1 -- 172.21.15.189:0/657401450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa040144310 con 0x7fa028003a00 2024-04-06T13:29:16.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.612+0000 7fa02effd700 1 -- 172.21.15.189:0/657401450 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fa034007530 con 0x7fa028003a00 2024-04-06T13:29:16.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.613+0000 7fa02effd700 1 --2- 172.21.15.189:0/657401450 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa02006ade0 0x7fa02006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:16.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.613+0000 7fa02effd700 1 -- 172.21.15.189:0/657401450 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5973+0+0 (secure 0 0 0) 0x7fa034089fb0 con 0x7fa028003a00 2024-04-06T13:29:16.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.613+0000 7fa03dd9b700 1 --2- 172.21.15.189:0/657401450 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa02006ade0 0x7fa02006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:16.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.613+0000 7fa03dd9b700 1 --2- 172.21.15.189:0/657401450 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa02006ade0 0x7fa02006d2a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa030001050 tx=0x7fa030000fe0 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:16.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.617+0000 7fa02effd700 1 -- 172.21.15.189:0/657401450 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa034055080 con 0x7fa028003a00 2024-04-06T13:29:16.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.857+0000 7fa044cfd700 1 -- 172.21.15.189:0/657401450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa040153eb0 con 0x7fa028003a00 2024-04-06T13:29:16.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.866+0000 7fa02effd700 1 -- 172.21.15.189:0/657401450 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 23 and data pool 24 v82) v1 ==== 174+0+0 (secure 0 0 0) 0x7fa03402b0c0 con 0x7fa028003a00 2024-04-06T13:29:16.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.870+0000 7fa044cfd700 1 -- 172.21.15.189:0/657401450 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa02006ade0 msgr2=0x7fa02006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:16.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.870+0000 7fa044cfd700 1 --2- 172.21.15.189:0/657401450 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa02006ade0 0x7fa02006d2a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa030001050 tx=0x7fa030000fe0 comp rx=0 tx=0).stop 2024-04-06T13:29:16.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.870+0000 7fa044cfd700 1 -- 172.21.15.189:0/657401450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa028003a00 msgr2=0x7fa040154060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.870+0000 7fa044cfd700 1 --2- 172.21.15.189:0/657401450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa028003a00 0x7fa040154060 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fa034000c00 tx=0x7fa034001120 comp rx=0 tx=0).stop 2024-04-06T13:29:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.870+0000 7fa044cfd700 1 -- 172.21.15.189:0/657401450 shutdown_connections 2024-04-06T13:29:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.870+0000 7fa044cfd700 1 --2- 172.21.15.189:0/657401450 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fa02006ade0 0x7fa02006d2a0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa030001050 tx=0x7fa030000fe0 comp rx=0 tx=0).stop 2024-04-06T13:29:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.870+0000 7fa044cfd700 1 --2- 172.21.15.189:0/657401450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0401529d0 0x7fa0400b1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.870+0000 7fa044cfd700 1 --2- 172.21.15.189:0/657401450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa028003a00 0x7fa040154060 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.870+0000 7fa044cfd700 1 --2- 172.21.15.189:0/657401450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0401f0670 0x7fa040152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.870+0000 7fa044cfd700 1 -- 172.21.15.189:0/657401450 >> 172.21.15.189:0/657401450 conn(0x7fa0400d00b0 msgr2=0x7fa0401595f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.872+0000 7fa044cfd700 1 -- 172.21.15.189:0/657401450 shutdown_connections 2024-04-06T13:29:16.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:16.873+0000 7fa044cfd700 1 -- 172.21.15.189:0/657401450 wait complete. 2024-04-06T13:29:16.883 INFO:teuthology.orchestra.run.smithi189.stderr:new fs with metadata pool 23 and data pool 24 2024-04-06T13:29:16.896 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:29:17.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.018+0000 7fd19dcb6700 1 Processor -- start 2024-04-06T13:29:17.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.018+0000 7fd19dcb6700 1 -- start start 2024-04-06T13:29:17.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.019+0000 7fd19dcb6700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd19815a550 0x7fd1980cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.019+0000 7fd19dcb6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1980cf900 0x7fd1980b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.019+0000 7fd19dcb6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1980cdbe0 con 0x7fd1980b4ba0 2024-04-06T13:29:17.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.019+0000 7fd19dcb6700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1980b5010 con 0x7fd19815a550 2024-04-06T13:29:17.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.019+0000 7fd19dcb6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd19815a940 con 0x7fd1980cf900 2024-04-06T13:29:17.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.019+0000 7fd197fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd1980b4ba0 0x7fd19806eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43054/0 (socket says 172.21.15.189:43054) 2024-04-06T13:29:17.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.019+0000 7fd197fff700 1 -- 172.21.15.189:0/3479002595 learned_addr learned my addr 172.21.15.189:0/3479002595 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:17.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.019+0000 7fd1977fe700 1 --2- 172.21.15.189:0/3479002595 >> v2:172.21.15.192:3300/0 conn(0x7fd19815a550 0x7fd1980cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.019+0000 7fd1967fc700 1 -- 172.21.15.189:0/3479002595 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125524822 0 0) 0x7fd1980cdbe0 con 0x7fd1980b4ba0 2024-04-06T13:29:17.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.019+0000 7fd196ffd700 1 --2- 172.21.15.189:0/3479002595 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1980cf900 0x7fd1980b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.019+0000 7fd1967fc700 1 -- 172.21.15.189:0/3479002595 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1800057c0 con 0x7fd1980b4ba0 2024-04-06T13:29:17.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd1967fc700 1 -- 172.21.15.189:0/3479002595 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2188250887 0 0) 0x7fd1800057c0 con 0x7fd1980b4ba0 2024-04-06T13:29:17.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd1967fc700 1 -- 172.21.15.189:0/3479002595 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1980cf900 msgr2=0x7fd1980b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:17.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd1967fc700 1 --2- 172.21.15.189:0/3479002595 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1980cf900 0x7fd1980b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd1967fc700 1 -- 172.21.15.189:0/3479002595 >> v2:172.21.15.192:3300/0 conn(0x7fd19815a550 msgr2=0x7fd1980cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:17.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd1967fc700 1 --2- 172.21.15.189:0/3479002595 >> v2:172.21.15.192:3300/0 conn(0x7fd19815a550 0x7fd1980cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd1967fc700 1 -- 172.21.15.189:0/3479002595 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19815ac00 con 0x7fd1980b4ba0 2024-04-06T13:29:17.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd1967fc700 1 -- 172.21.15.189:0/3479002595 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fd1900019f0 con 0x7fd1980b4ba0 2024-04-06T13:29:17.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd1967fc700 1 -- 172.21.15.189:0/3479002595 >> v1:172.21.15.189:6789/0 conn(0x7fd1980b4ba0 legacy=0x7fd19806eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:17.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd1967fc700 1 --2- 172.21.15.189:0/3479002595 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1980cf900 0x7fd180005c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd1967fc700 1 --2- 172.21.15.189:0/3479002595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd180006190 0x7fd180006a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd1967fc700 1 --2- 172.21.15.189:0/3479002595 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd180006fa0 0x7fd180007420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd1967fc700 1 -- 172.21.15.189:0/3479002595 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1980b5010 con 0x7fd180006190 2024-04-06T13:29:17.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd1967fc700 1 -- 172.21.15.189:0/3479002595 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19815a940 con 0x7fd180006fa0 2024-04-06T13:29:17.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd1967fc700 1 -- 172.21.15.189:0/3479002595 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd18000dc20 con 0x7fd1980cf900 2024-04-06T13:29:17.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd196ffd700 1 --2- 172.21.15.189:0/3479002595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd180006190 0x7fd180006a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd1977fe700 1 --2- 172.21.15.189:0/3479002595 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1980cf900 0x7fd180005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd197fff700 1 --2- 172.21.15.189:0/3479002595 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd180006fa0 0x7fd180007420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd196ffd700 1 -- 172.21.15.189:0/3479002595 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1980cf900 msgr2=0x7fd180005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:17.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd196ffd700 1 --2- 172.21.15.189:0/3479002595 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1980cf900 0x7fd180005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd196ffd700 1 -- 172.21.15.189:0/3479002595 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd180006fa0 msgr2=0x7fd180007420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:17.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd196ffd700 1 --2- 172.21.15.189:0/3479002595 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd180006fa0 0x7fd180007420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd196ffd700 1 -- 172.21.15.189:0/3479002595 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1800057c0 con 0x7fd180006190 2024-04-06T13:29:17.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd1977fe700 1 --2- 172.21.15.189:0/3479002595 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1980cf900 0x7fd180005c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:29:17.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd196ffd700 1 --2- 172.21.15.189:0/3479002595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd180006190 0x7fd180006a60 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fd188009b60 tx=0x7fd180007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2420b1ab2c878c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:17.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.020+0000 7fd1967fc700 1 -- 172.21.15.189:0/3479002595 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18800e660 con 0x7fd180006190 2024-04-06T13:29:17.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.021+0000 7fd1967fc700 1 -- 172.21.15.189:0/3479002595 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd18800e7c0 con 0x7fd180006190 2024-04-06T13:29:17.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.021+0000 7fd1967fc700 1 -- 172.21.15.189:0/3479002595 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18800eb30 con 0x7fd180006190 2024-04-06T13:29:17.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.021+0000 7fd19dcb6700 1 -- 172.21.15.189:0/3479002595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd180006190 msgr2=0x7fd180006a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:17.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.021+0000 7fd19dcb6700 1 --2- 172.21.15.189:0/3479002595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd180006190 0x7fd180006a60 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fd188009b60 tx=0x7fd180007a90 comp rx=0 tx=0).stop 2024-04-06T13:29:17.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.021+0000 7fd197fff700 1 -- 172.21.15.189:0/3479002595 reap_dead start 2024-04-06T13:29:17.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.021+0000 7fd19dcb6700 1 -- 172.21.15.189:0/3479002595 shutdown_connections 2024-04-06T13:29:17.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.021+0000 7fd19dcb6700 1 -- 172.21.15.189:0/3479002595 >> 172.21.15.189:0/3479002595 conn(0x7fd1980d00b0 msgr2=0x7fd1980d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:17.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.021+0000 7fd19dcb6700 1 -- 172.21.15.189:0/3479002595 shutdown_connections 2024-04-06T13:29:17.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.021+0000 7fd19dcb6700 1 -- 172.21.15.189:0/3479002595 wait complete. 2024-04-06T13:29:17.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.022+0000 7fd19dcb6700 1 Processor -- start 2024-04-06T13:29:17.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.022+0000 7fd19dcb6700 1 -- start start 2024-04-06T13:29:17.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.023+0000 7fd19dcb6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd180006190 0x7fd1981575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.023+0000 7fd19dcb6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1980ac9a0 0x7fd198157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.023+0000 7fd19dcb6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd198150160 0x7fd19815c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.023+0000 7fd19dcb6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19815b8c0 con 0x7fd198150160 2024-04-06T13:29:17.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.023+0000 7fd19dcb6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19814c4d0 con 0x7fd180006190 2024-04-06T13:29:17.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.023+0000 7fd19dcb6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1981585e0 con 0x7fd1980ac9a0 2024-04-06T13:29:17.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.023+0000 7fd197fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd198150160 0x7fd19815c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.023+0000 7fd197fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd198150160 0x7fd19815c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50020/0 (socket says 172.21.15.189:50020) 2024-04-06T13:29:17.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.023+0000 7fd197fff700 1 -- 172.21.15.189:0/860439050 learned_addr learned my addr 172.21.15.189:0/860439050 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:17.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.023+0000 7fd196ffd700 1 --2- 172.21.15.189:0/860439050 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1980ac9a0 0x7fd198157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.023+0000 7fd1977fe700 1 --2- 172.21.15.189:0/860439050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd180006190 0x7fd1981575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.024+0000 7fd197fff700 1 -- 172.21.15.189:0/860439050 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1980ac9a0 msgr2=0x7fd198157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:17.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.024+0000 7fd197fff700 1 --2- 172.21.15.189:0/860439050 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1980ac9a0 0x7fd198157b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.024+0000 7fd197fff700 1 -- 172.21.15.189:0/860439050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd180006190 msgr2=0x7fd1981575e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:17.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.024+0000 7fd197fff700 1 --2- 172.21.15.189:0/860439050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd180006190 0x7fd1981575e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.024+0000 7fd197fff700 1 -- 172.21.15.189:0/860439050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19815bad0 con 0x7fd198150160 2024-04-06T13:29:17.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.024+0000 7fd197fff700 1 --2- 172.21.15.189:0/860439050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd198150160 0x7fd19815c5b0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fd198072830 tx=0x7fd19000a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:17.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.024+0000 7fd17ffff700 1 -- 172.21.15.189:0/860439050 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19000ae10 con 0x7fd198150160 2024-04-06T13:29:17.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.024+0000 7fd17ffff700 1 -- 172.21.15.189:0/860439050 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1900133f0 con 0x7fd198150160 2024-04-06T13:29:17.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.024+0000 7fd19dcb6700 1 -- 172.21.15.189:0/860439050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd19815bdb0 con 0x7fd198150160 2024-04-06T13:29:17.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.024+0000 7fd17ffff700 1 -- 172.21.15.189:0/860439050 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1900137a0 con 0x7fd198150160 2024-04-06T13:29:17.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.024+0000 7fd19dcb6700 1 -- 172.21.15.189:0/860439050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd198158060 con 0x7fd198150160 2024-04-06T13:29:17.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.025+0000 7fd19dcb6700 1 -- 172.21.15.189:0/860439050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd198067bb0 con 0x7fd198150160 2024-04-06T13:29:17.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.026+0000 7fd17ffff700 1 -- 172.21.15.189:0/860439050 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fd190013900 con 0x7fd198150160 2024-04-06T13:29:17.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.026+0000 7fd17ffff700 1 --2- 172.21.15.189:0/860439050 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd18406adb0 0x7fd18406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.026+0000 7fd17ffff700 1 -- 172.21.15.189:0/860439050 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd19008a920 con 0x7fd198150160 2024-04-06T13:29:17.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.026+0000 7fd1977fe700 1 --2- 172.21.15.189:0/860439050 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd18406adb0 0x7fd18406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.026+0000 7fd1977fe700 1 --2- 172.21.15.189:0/860439050 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd18406adb0 0x7fd18406d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd18c009bb0 tx=0x7fd18c008040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:17.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.030+0000 7fd17ffff700 1 -- 172.21.15.189:0/860439050 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd1900559f0 con 0x7fd198150160 2024-04-06T13:29:17.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.226+0000 7fd19dcb6700 1 -- 172.21.15.189:0/860439050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd198074990 con 0x7fd198150160 2024-04-06T13:29:17.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.228+0000 7fd17ffff700 1 -- 172.21.15.189:0/860439050 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v191) v1 ==== 74+0+16379 (secure 0 0 0) 0x7fd190059010 con 0x7fd198150160 2024-04-06T13:29:17.227 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:29:17.228 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":191,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:29:16.858380+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:29:14.625778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-04-06T13:29:15.827532+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:29:17.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.230+0000 7fd19dcb6700 1 -- 172.21.15.189:0/860439050 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd18406adb0 msgr2=0x7fd18406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:17.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.230+0000 7fd19dcb6700 1 --2- 172.21.15.189:0/860439050 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd18406adb0 0x7fd18406d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd18c009bb0 tx=0x7fd18c008040 comp rx=0 tx=0).stop 2024-04-06T13:29:17.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.230+0000 7fd19dcb6700 1 -- 172.21.15.189:0/860439050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd198150160 msgr2=0x7fd19815c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:17.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.230+0000 7fd19dcb6700 1 --2- 172.21.15.189:0/860439050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd198150160 0x7fd19815c5b0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fd198072830 tx=0x7fd19000a640 comp rx=0 tx=0).stop 2024-04-06T13:29:17.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.230+0000 7fd19dcb6700 1 -- 172.21.15.189:0/860439050 shutdown_connections 2024-04-06T13:29:17.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.230+0000 7fd19dcb6700 1 --2- 172.21.15.189:0/860439050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd198150160 0x7fd19815c5b0 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.230+0000 7fd19dcb6700 1 --2- 172.21.15.189:0/860439050 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd18406adb0 0x7fd18406d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.230+0000 7fd19dcb6700 1 --2- 172.21.15.189:0/860439050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd180006190 0x7fd1981575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.230+0000 7fd19dcb6700 1 --2- 172.21.15.189:0/860439050 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1980ac9a0 0x7fd198157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.230+0000 7fd19dcb6700 1 -- 172.21.15.189:0/860439050 >> 172.21.15.189:0/860439050 conn(0x7fd1980d00b0 msgr2=0x7fd19806f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:17.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.230+0000 7fd19dcb6700 1 -- 172.21.15.189:0/860439050 shutdown_connections 2024-04-06T13:29:17.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.230+0000 7fd19dcb6700 1 -- 172.21.15.189:0/860439050 wait complete. 2024-04-06T13:29:17.243 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:29:17.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.366+0000 7fc51fecb700 1 Processor -- start 2024-04-06T13:29:17.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.366+0000 7fc51fecb700 1 -- start start 2024-04-06T13:29:17.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.367+0000 7fc51fecb700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc51815d7b0 0x7fc5180b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.367+0000 7fc51fecb700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc51806a730 0x7fc51806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.367+0000 7fc51fecb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5180b3990 con 0x7fc51815ddf0 2024-04-06T13:29:17.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.367+0000 7fc51fecb700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc51815e950 con 0x7fc51815d7b0 2024-04-06T13:29:17.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.367+0000 7fc51fecb700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc51806b0f0 con 0x7fc51806a730 2024-04-06T13:29:17.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.367+0000 7fc51dc67700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc51815d7b0 0x7fc5180b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.367+0000 7fc51e468700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc51815ddf0 0x7fc51815e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43070/0 (socket says 172.21.15.189:43070) 2024-04-06T13:29:17.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.367+0000 7fc51e468700 1 -- 172.21.15.189:0/4060233195 learned_addr learned my addr 172.21.15.189:0/4060233195 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:17.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.367+0000 7fc51d466700 1 --2- 172.21.15.189:0/4060233195 >> v2:172.21.15.192:3300/0 conn(0x7fc51806a730 0x7fc51806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.367+0000 7fc51dc67700 1 -- 172.21.15.189:0/4060233195 >> v2:172.21.15.192:3300/0 conn(0x7fc51806a730 msgr2=0x7fc51806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:17.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.367+0000 7fc51dc67700 1 --2- 172.21.15.189:0/4060233195 >> v2:172.21.15.192:3300/0 conn(0x7fc51806a730 0x7fc51806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.367+0000 7fc51dc67700 1 -- 172.21.15.189:0/4060233195 >> v1:172.21.15.189:6789/0 conn(0x7fc51815ddf0 legacy=0x7fc51815e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:17.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.367+0000 7fc51dc67700 1 -- 172.21.15.189:0/4060233195 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5180b3060 con 0x7fc51815d7b0 2024-04-06T13:29:17.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.367+0000 7fc51cc65700 1 -- 172.21.15.189:0/4060233195 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664152962 0 0) 0x7fc5180b3990 con 0x7fc51815ddf0 2024-04-06T13:29:17.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.367+0000 7fc51dc67700 1 --2- 172.21.15.189:0/4060233195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc51815d7b0 0x7fc5180b3e20 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fc5180afc00 tx=0x7fc514009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea1141327d7e456 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:17.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.368+0000 7fc51cc65700 1 -- 172.21.15.189:0/4060233195 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc514011830 con 0x7fc51815d7b0 2024-04-06T13:29:17.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.368+0000 7fc51cc65700 1 -- 172.21.15.189:0/4060233195 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc514011990 con 0x7fc51815d7b0 2024-04-06T13:29:17.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.368+0000 7fc51cc65700 1 -- 172.21.15.189:0/4060233195 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc514011d20 con 0x7fc51815d7b0 2024-04-06T13:29:17.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.368+0000 7fc51fecb700 1 -- 172.21.15.189:0/4060233195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc51815d7b0 msgr2=0x7fc5180b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:17.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.368+0000 7fc51fecb700 1 --2- 172.21.15.189:0/4060233195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc51815d7b0 0x7fc5180b3e20 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fc5180afc00 tx=0x7fc514009b20 comp rx=0 tx=0).stop 2024-04-06T13:29:17.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.368+0000 7fc51fecb700 1 -- 172.21.15.189:0/4060233195 shutdown_connections 2024-04-06T13:29:17.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.368+0000 7fc51fecb700 1 --2- 172.21.15.189:0/4060233195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc51815d7b0 0x7fc5180b3e20 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.368+0000 7fc51fecb700 1 --2- 172.21.15.189:0/4060233195 >> v2:172.21.15.192:3300/0 conn(0x7fc51806a730 0x7fc51806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.368+0000 7fc51fecb700 1 -- 172.21.15.189:0/4060233195 >> 172.21.15.189:0/4060233195 conn(0x7fc5180d00b0 msgr2=0x7fc5180d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:17.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.368+0000 7fc51fecb700 1 -- 172.21.15.189:0/4060233195 shutdown_connections 2024-04-06T13:29:17.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.368+0000 7fc51fecb700 1 -- 172.21.15.189:0/4060233195 wait complete. 2024-04-06T13:29:17.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.369+0000 7fc51fecb700 1 Processor -- start 2024-04-06T13:29:17.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.370+0000 7fc51fecb700 1 -- start start 2024-04-06T13:29:17.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.370+0000 7fc51fecb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc51806a730 0x7fc5181f5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.370+0000 7fc51fecb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc51815d7b0 0x7fc5180ced80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.370+0000 7fc51fecb700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc51815ddf0 0x7fc5181f6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.370+0000 7fc51fecb700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51815b2d0 con 0x7fc51815d7b0 2024-04-06T13:29:17.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.370+0000 7fc51fecb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51815b5b0 con 0x7fc51806a730 2024-04-06T13:29:17.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.370+0000 7fc51fecb700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc51815b890 con 0x7fc51815ddf0 2024-04-06T13:29:17.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.370+0000 7fc51d466700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc51815d7b0 0x7fc5180ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.370+0000 7fc51d466700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc51815d7b0 0x7fc5180ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50034/0 (socket says 172.21.15.189:50034) 2024-04-06T13:29:17.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.370+0000 7fc51d466700 1 -- 172.21.15.189:0/966585822 learned_addr learned my addr 172.21.15.189:0/966585822 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:17.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.370+0000 7fc51dc67700 1 --2- 172.21.15.189:0/966585822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc51806a730 0x7fc5181f5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.371+0000 7fc51e468700 1 --2- 172.21.15.189:0/966585822 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc51815ddf0 0x7fc5181f6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.371+0000 7fc51d466700 1 -- 172.21.15.189:0/966585822 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc51815ddf0 msgr2=0x7fc5181f6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:17.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.371+0000 7fc51d466700 1 --2- 172.21.15.189:0/966585822 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc51815ddf0 0x7fc5181f6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.371+0000 7fc51d466700 1 -- 172.21.15.189:0/966585822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc51806a730 msgr2=0x7fc5181f5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:17.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.371+0000 7fc51d466700 1 --2- 172.21.15.189:0/966585822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc51806a730 0x7fc5181f5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.371+0000 7fc51d466700 1 -- 172.21.15.189:0/966585822 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5140094a0 con 0x7fc51815d7b0 2024-04-06T13:29:17.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.371+0000 7fc51d466700 1 --2- 172.21.15.189:0/966585822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc51815d7b0 0x7fc5180ced80 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fc50800e9b0 tx=0x7fc50800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:17.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.371+0000 7fc50e7fc700 1 -- 172.21.15.189:0/966585822 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc508009710 con 0x7fc51815d7b0 2024-04-06T13:29:17.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.371+0000 7fc50e7fc700 1 -- 172.21.15.189:0/966585822 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50800c850 con 0x7fc51815d7b0 2024-04-06T13:29:17.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.371+0000 7fc51fecb700 1 -- 172.21.15.189:0/966585822 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc51815bb70 con 0x7fc51815d7b0 2024-04-06T13:29:17.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.371+0000 7fc50e7fc700 1 -- 172.21.15.189:0/966585822 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50800cbc0 con 0x7fc51815d7b0 2024-04-06T13:29:17.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.371+0000 7fc51fecb700 1 -- 172.21.15.189:0/966585822 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc51815aea0 con 0x7fc51815d7b0 2024-04-06T13:29:17.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.372+0000 7fc51fecb700 1 -- 172.21.15.189:0/966585822 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5181f3bf0 con 0x7fc51815d7b0 2024-04-06T13:29:17.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.372+0000 7fc50e7fc700 1 -- 172.21.15.189:0/966585822 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fc50800cd20 con 0x7fc51815d7b0 2024-04-06T13:29:17.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.373+0000 7fc50e7fc700 1 --2- 172.21.15.189:0/966585822 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc4fc06adb0 0x7fc4fc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.373+0000 7fc50e7fc700 1 -- 172.21.15.189:0/966585822 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc5080895d0 con 0x7fc51815d7b0 2024-04-06T13:29:17.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.373+0000 7fc51dc67700 1 --2- 172.21.15.189:0/966585822 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc4fc06adb0 0x7fc4fc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.373+0000 7fc51dc67700 1 --2- 172.21.15.189:0/966585822 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc4fc06adb0 0x7fc4fc06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc5140090b0 tx=0x7fc514008030 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:17.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.377+0000 7fc50e7fc700 1 -- 172.21.15.189:0/966585822 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc5181f3bf0 con 0x7fc51815d7b0 2024-04-06T13:29:17.438 INFO:tasks.ceph.mds.b.smithi192.stderr:2024-04-06T13:29:17.437+0000 7fda583e4700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T13:29:17.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.574+0000 7fc51fecb700 1 -- 172.21.15.189:0/966585822 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc51815bee0 con 0x7fc51815d7b0 2024-04-06T13:29:17.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.575+0000 7fc50e7fc700 1 -- 172.21.15.189:0/966585822 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v191) v1 ==== 74+0+16379 (secure 0 0 0) 0x7fc51815bee0 con 0x7fc51815d7b0 2024-04-06T13:29:17.575 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:29:17.575 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":191,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:29:16.858380+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:29:14.625778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-04-06T13:29:15.827532+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:29:17.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.578+0000 7fc51fecb700 1 -- 172.21.15.189:0/966585822 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc4fc06adb0 msgr2=0x7fc4fc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.578+0000 7fc51fecb700 1 --2- 172.21.15.189:0/966585822 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc4fc06adb0 0x7fc4fc06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc5140090b0 tx=0x7fc514008030 comp rx=0 tx=0).stop 2024-04-06T13:29:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.578+0000 7fc51fecb700 1 -- 172.21.15.189:0/966585822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc51815d7b0 msgr2=0x7fc5180ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.578+0000 7fc51fecb700 1 --2- 172.21.15.189:0/966585822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc51815d7b0 0x7fc5180ced80 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fc50800e9b0 tx=0x7fc50800ecc0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.578+0000 7fc51fecb700 1 -- 172.21.15.189:0/966585822 shutdown_connections 2024-04-06T13:29:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.578+0000 7fc51fecb700 1 --2- 172.21.15.189:0/966585822 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc51815ddf0 0x7fc5181f6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.578+0000 7fc51fecb700 1 --2- 172.21.15.189:0/966585822 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fc4fc06adb0 0x7fc4fc06d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.578+0000 7fc51fecb700 1 --2- 172.21.15.189:0/966585822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc51806a730 0x7fc5181f5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.578+0000 7fc51fecb700 1 --2- 172.21.15.189:0/966585822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc51815d7b0 0x7fc5180ced80 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.578+0000 7fc51fecb700 1 -- 172.21.15.189:0/966585822 >> 172.21.15.189:0/966585822 conn(0x7fc5180d00b0 msgr2=0x7fc51806fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.578+0000 7fc51fecb700 1 -- 172.21.15.189:0/966585822 shutdown_connections 2024-04-06T13:29:17.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.578+0000 7fc51fecb700 1 -- 172.21.15.189:0/966585822 wait complete. 2024-04-06T13:29:17.590 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-06T13:29:17.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.716+0000 7f9243418700 1 Processor -- start 2024-04-06T13:29:17.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.716+0000 7f9243418700 1 -- start start 2024-04-06T13:29:17.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.717+0000 7f9243418700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f923c146dd0 0x7f923c06d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.717+0000 7f9243418700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f923c0cedf0 0x7f923c0cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.717+0000 7f9243418700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f923c0cf730 con 0x7f923c0d17a0 2024-04-06T13:29:17.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.717+0000 7f9243418700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f923c06beb0 con 0x7f923c146dd0 2024-04-06T13:29:17.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.717+0000 7f9243418700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f923c06c0a0 con 0x7f923c0cedf0 2024-04-06T13:29:17.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.717+0000 7f92409b3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f923c146dd0 0x7f923c06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.717+0000 7f92411b4700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f923c0d17a0 0x7f923c06b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43072/0 (socket says 172.21.15.189:43072) 2024-04-06T13:29:17.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.717+0000 7f92411b4700 1 -- 172.21.15.189:0/999715060 learned_addr learned my addr 172.21.15.189:0/999715060 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:17.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.717+0000 7f92419b5700 1 --2- 172.21.15.189:0/999715060 >> v2:172.21.15.192:3300/0 conn(0x7f923c0cedf0 0x7f923c0cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.717+0000 7f92409b3700 1 -- 172.21.15.189:0/999715060 >> v2:172.21.15.192:3300/0 conn(0x7f923c0cedf0 msgr2=0x7f923c0cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:17.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.717+0000 7f92409b3700 1 --2- 172.21.15.189:0/999715060 >> v2:172.21.15.192:3300/0 conn(0x7f923c0cedf0 0x7f923c0cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.717+0000 7f92409b3700 1 -- 172.21.15.189:0/999715060 >> v1:172.21.15.189:6789/0 conn(0x7f923c0d17a0 legacy=0x7f923c06b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:17.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.717+0000 7f9233fff700 1 -- 172.21.15.189:0/999715060 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680987568 0 0) 0x7f923c0cf730 con 0x7f923c0d17a0 2024-04-06T13:29:17.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.717+0000 7f92409b3700 1 -- 172.21.15.189:0/999715060 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f923c06c380 con 0x7f923c146dd0 2024-04-06T13:29:17.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.717+0000 7f92409b3700 1 --2- 172.21.15.189:0/999715060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f923c146dd0 0x7f923c06d140 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f923c160fb0 tx=0x7f9238009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71cacb7baac1c7d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:17.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.717+0000 7f9233fff700 1 -- 172.21.15.189:0/999715060 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9238011830 con 0x7f923c146dd0 2024-04-06T13:29:17.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.717+0000 7f9233fff700 1 -- 172.21.15.189:0/999715060 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9238011990 con 0x7f923c146dd0 2024-04-06T13:29:17.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.717+0000 7f9233fff700 1 -- 172.21.15.189:0/999715060 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9238011d20 con 0x7f923c146dd0 2024-04-06T13:29:17.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.718+0000 7f9243418700 1 -- 172.21.15.189:0/999715060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f923c146dd0 msgr2=0x7f923c06d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:17.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.718+0000 7f9243418700 1 --2- 172.21.15.189:0/999715060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f923c146dd0 0x7f923c06d140 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f923c160fb0 tx=0x7f9238009b20 comp rx=0 tx=0).stop 2024-04-06T13:29:17.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.718+0000 7f9243418700 1 -- 172.21.15.189:0/999715060 shutdown_connections 2024-04-06T13:29:17.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.718+0000 7f9243418700 1 --2- 172.21.15.189:0/999715060 >> v2:172.21.15.192:3300/0 conn(0x7f923c0cedf0 0x7f923c0cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.718+0000 7f9243418700 1 --2- 172.21.15.189:0/999715060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f923c146dd0 0x7f923c06d140 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.718+0000 7f9243418700 1 -- 172.21.15.189:0/999715060 >> 172.21.15.189:0/999715060 conn(0x7f923c0b2db0 msgr2=0x7f923c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:17.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.718+0000 7f9243418700 1 -- 172.21.15.189:0/999715060 shutdown_connections 2024-04-06T13:29:17.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.718+0000 7f9243418700 1 -- 172.21.15.189:0/999715060 wait complete. 2024-04-06T13:29:17.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.719+0000 7f9243418700 1 Processor -- start 2024-04-06T13:29:17.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.719+0000 7f9243418700 1 -- start start 2024-04-06T13:29:17.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.720+0000 7f9243418700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f923c0cedf0 0x7f923c15c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.720+0000 7f9243418700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f923c0d17a0 0x7f923c0c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.720+0000 7f9243418700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f923c146dd0 0x7f923c1f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.720+0000 7f9243418700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923c0c82f0 con 0x7f923c0cedf0 2024-04-06T13:29:17.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.720+0000 7f9243418700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923c0af6a0 con 0x7f923c146dd0 2024-04-06T13:29:17.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.720+0000 7f9243418700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f923c1f39a0 con 0x7f923c0d17a0 2024-04-06T13:29:17.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.720+0000 7f92411b4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f923c0cedf0 0x7f923c15c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.720+0000 7f92411b4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f923c0cedf0 0x7f923c15c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50062/0 (socket says 172.21.15.189:50062) 2024-04-06T13:29:17.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.720+0000 7f92411b4700 1 -- 172.21.15.189:0/3180197894 learned_addr learned my addr 172.21.15.189:0/3180197894 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:17.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.720+0000 7f92419b5700 1 --2- 172.21.15.189:0/3180197894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f923c146dd0 0x7f923c1f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.720+0000 7f92409b3700 1 --2- 172.21.15.189:0/3180197894 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f923c0d17a0 0x7f923c0c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.720+0000 7f92411b4700 1 -- 172.21.15.189:0/3180197894 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f923c0d17a0 msgr2=0x7f923c0c8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:17.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.720+0000 7f92411b4700 1 --2- 172.21.15.189:0/3180197894 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f923c0d17a0 0x7f923c0c8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.720+0000 7f92411b4700 1 -- 172.21.15.189:0/3180197894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f923c146dd0 msgr2=0x7f923c1f2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:17.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.720+0000 7f92411b4700 1 --2- 172.21.15.189:0/3180197894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f923c146dd0 0x7f923c1f2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:17.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.720+0000 7f92411b4700 1 -- 172.21.15.189:0/3180197894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92380094a0 con 0x7f923c0cedf0 2024-04-06T13:29:17.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.720+0000 7f92419b5700 1 --2- 172.21.15.189:0/3180197894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f923c146dd0 0x7f923c1f2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:29:17.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.720+0000 7f92409b3700 1 --2- 172.21.15.189:0/3180197894 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f923c0d17a0 0x7f923c0c8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:29:17.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.720+0000 7f92411b4700 1 --2- 172.21.15.189:0/3180197894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f923c0cedf0 0x7f923c15c7a0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f922c00c550 tx=0x7f922c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:17.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.721+0000 7f9231ffb700 1 -- 172.21.15.189:0/3180197894 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c004c80 con 0x7f923c0cedf0 2024-04-06T13:29:17.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.721+0000 7f9231ffb700 1 -- 172.21.15.189:0/3180197894 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922c004e00 con 0x7f923c0cedf0 2024-04-06T13:29:17.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.721+0000 7f9231ffb700 1 -- 172.21.15.189:0/3180197894 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c007310 con 0x7f923c0cedf0 2024-04-06T13:29:17.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.721+0000 7f9243418700 1 -- 172.21.15.189:0/3180197894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f923c1f3c80 con 0x7f923c0cedf0 2024-04-06T13:29:17.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.721+0000 7f9243418700 1 -- 172.21.15.189:0/3180197894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f923c0c8890 con 0x7f923c0cedf0 2024-04-06T13:29:17.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.722+0000 7f9243418700 1 -- 172.21.15.189:0/3180197894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f923c06c900 con 0x7f923c0cedf0 2024-04-06T13:29:17.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.722+0000 7f9231ffb700 1 -- 172.21.15.189:0/3180197894 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f922c007470 con 0x7f923c0cedf0 2024-04-06T13:29:17.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.723+0000 7f9231ffb700 1 --2- 172.21.15.189:0/3180197894 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f922006adb0 0x7f922006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:17.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.723+0000 7f9231ffb700 1 -- 172.21.15.189:0/3180197894 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f922c089380 con 0x7f923c0cedf0 2024-04-06T13:29:17.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.723+0000 7f92409b3700 1 --2- 172.21.15.189:0/3180197894 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f922006adb0 0x7f922006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:17.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.723+0000 7f92409b3700 1 --2- 172.21.15.189:0/3180197894 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f922006adb0 0x7f922006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f923c1471e0 tx=0x7f9238008030 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:17.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.727+0000 7f9231ffb700 1 -- 172.21.15.189:0/3180197894 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f922c054240 con 0x7f923c0cedf0 2024-04-06T13:29:17.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:17.964+0000 7f9243418700 1 -- 172.21.15.189:0/3180197894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f923c0c8a80 con 0x7f923c0cedf0 2024-04-06T13:29:18.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:18.870+0000 7f9231ffb700 1 -- 172.21.15.189:0/3180197894 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v84) v1 ==== 118+0+0 (secure 0 0 0) 0x7f922c057860 con 0x7f923c0cedf0 2024-04-06T13:29:18.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:18.873+0000 7f9243418700 1 -- 172.21.15.189:0/3180197894 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f922006adb0 msgr2=0x7f922006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:18.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:18.873+0000 7f9243418700 1 --2- 172.21.15.189:0/3180197894 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f922006adb0 0x7f922006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f923c1471e0 tx=0x7f9238008030 comp rx=0 tx=0).stop 2024-04-06T13:29:18.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:18.873+0000 7f9243418700 1 -- 172.21.15.189:0/3180197894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f923c0cedf0 msgr2=0x7f923c15c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:18.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:18.873+0000 7f9243418700 1 --2- 172.21.15.189:0/3180197894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f923c0cedf0 0x7f923c15c7a0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f922c00c550 tx=0x7f922c00c910 comp rx=0 tx=0).stop 2024-04-06T13:29:18.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:18.873+0000 7f9243418700 1 -- 172.21.15.189:0/3180197894 shutdown_connections 2024-04-06T13:29:18.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:18.873+0000 7f9243418700 1 --2- 172.21.15.189:0/3180197894 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f922006adb0 0x7f922006d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:18.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:18.873+0000 7f9243418700 1 --2- 172.21.15.189:0/3180197894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f923c146dd0 0x7f923c1f2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:18.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:18.873+0000 7f9243418700 1 --2- 172.21.15.189:0/3180197894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f923c0cedf0 0x7f923c15c7a0 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:18.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:18.873+0000 7f9243418700 1 --2- 172.21.15.189:0/3180197894 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f923c0d17a0 0x7f923c0c8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:18.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:18.873+0000 7f9243418700 1 -- 172.21.15.189:0/3180197894 >> 172.21.15.189:0/3180197894 conn(0x7f923c0b2db0 msgr2=0x7f923c0b24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:18.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:18.873+0000 7f9243418700 1 -- 172.21.15.189:0/3180197894 shutdown_connections 2024-04-06T13:29:18.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:18.873+0000 7f9243418700 1 -- 172.21.15.189:0/3180197894 wait complete. 2024-04-06T13:29:18.888 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-06T13:29:18.888 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-06T13:29:19.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.019+0000 7f4265adc700 1 Processor -- start 2024-04-06T13:29:19.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.019+0000 7f4265adc700 1 -- start start 2024-04-06T13:29:19.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.020+0000 7f4265adc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f42600aec70 0x7f42600cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:19.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.020+0000 7f4265adc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600cdae0 0x7f42600ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:19.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.020+0000 7f4265adc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f42600ce8b0 0x7f426015df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:19.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.020+0000 7f4265adc700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42600cd920 con 0x7f42600aec70 2024-04-06T13:29:19.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.020+0000 7f4265adc700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42600ced20 con 0x7f42600ce8b0 2024-04-06T13:29:19.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.020+0000 7f4265adc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f426015a830 con 0x7f42600cdae0 2024-04-06T13:29:19.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.020+0000 7f425f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f42600aec70 0x7f42600cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:19.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.020+0000 7f425f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f42600aec70 0x7f42600cd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50076/0 (socket says 172.21.15.189:50076) 2024-04-06T13:29:19.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.020+0000 7f425f7fe700 1 -- 172.21.15.189:0/3426440270 learned_addr learned my addr 172.21.15.189:0/3426440270 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:19.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.020+0000 7f425effd700 1 --2- 172.21.15.189:0/3426440270 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600cdae0 0x7f42600ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:19.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.020+0000 7f425ffff700 1 --2- 172.21.15.189:0/3426440270 >> v2:172.21.15.192:3300/0 conn(0x7f42600ce8b0 0x7f426015df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:19.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.020+0000 7f425f7fe700 1 -- 172.21.15.189:0/3426440270 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600cdae0 msgr2=0x7f42600ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:19.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.020+0000 7f425f7fe700 1 --2- 172.21.15.189:0/3426440270 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600cdae0 0x7f42600ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:19.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.020+0000 7f425f7fe700 1 -- 172.21.15.189:0/3426440270 >> v2:172.21.15.192:3300/0 conn(0x7f42600ce8b0 msgr2=0x7f426015df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:19.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.020+0000 7f425f7fe700 1 --2- 172.21.15.189:0/3426440270 >> v2:172.21.15.192:3300/0 conn(0x7f42600ce8b0 0x7f426015df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:19.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.020+0000 7f425f7fe700 1 -- 172.21.15.189:0/3426440270 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42600cc050 con 0x7f42600aec70 2024-04-06T13:29:19.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.021+0000 7f425f7fe700 1 --2- 172.21.15.189:0/3426440270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600aec70 0x7f42600cd3e0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f42600727f0 tx=0x7f4254009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5598c4586aa3697b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:19.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.021+0000 7f425e7fc700 1 -- 172.21.15.189:0/3426440270 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425400b870 con 0x7f42600aec70 2024-04-06T13:29:19.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.021+0000 7f425e7fc700 1 -- 172.21.15.189:0/3426440270 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425400b9d0 con 0x7f42600aec70 2024-04-06T13:29:19.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.021+0000 7f425e7fc700 1 -- 172.21.15.189:0/3426440270 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425400bd60 con 0x7f42600aec70 2024-04-06T13:29:19.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.021+0000 7f4265adc700 1 -- 172.21.15.189:0/3426440270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600aec70 msgr2=0x7f42600cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:19.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.021+0000 7f4265adc700 1 --2- 172.21.15.189:0/3426440270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600aec70 0x7f42600cd3e0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f42600727f0 tx=0x7f4254009bd0 comp rx=0 tx=0).stop 2024-04-06T13:29:19.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.021+0000 7f4265adc700 1 -- 172.21.15.189:0/3426440270 shutdown_connections 2024-04-06T13:29:19.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.021+0000 7f4265adc700 1 --2- 172.21.15.189:0/3426440270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600aec70 0x7f42600cd3e0 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:19.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.021+0000 7f4265adc700 1 --2- 172.21.15.189:0/3426440270 >> v2:172.21.15.192:3300/0 conn(0x7f42600ce8b0 0x7f426015df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:19.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.021+0000 7f4265adc700 1 --2- 172.21.15.189:0/3426440270 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600cdae0 0x7f42600ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:19.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.021+0000 7f4265adc700 1 -- 172.21.15.189:0/3426440270 >> 172.21.15.189:0/3426440270 conn(0x7f42600cc290 msgr2=0x7f42600cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:19.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.021+0000 7f4265adc700 1 -- 172.21.15.189:0/3426440270 shutdown_connections 2024-04-06T13:29:19.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.021+0000 7f4265adc700 1 -- 172.21.15.189:0/3426440270 wait complete. 2024-04-06T13:29:19.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.023+0000 7f4265adc700 1 Processor -- start 2024-04-06T13:29:19.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.023+0000 7f4265adc700 1 -- start start 2024-04-06T13:29:19.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.023+0000 7f4265adc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600aec70 0x7f42601582f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:19.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.023+0000 7f4265adc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600cdae0 0x7f42601f25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:19.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.023+0000 7f4265adc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42600ce8b0 0x7f4260155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:19.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.023+0000 7f4265adc700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42600b2760 con 0x7f42600aec70 2024-04-06T13:29:19.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.023+0000 7f4265adc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42600adde0 con 0x7f42600ce8b0 2024-04-06T13:29:19.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.023+0000 7f4265adc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42600ae0c0 con 0x7f42600cdae0 2024-04-06T13:29:19.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.024+0000 7f425f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600aec70 0x7f42601582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:19.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.024+0000 7f425f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600aec70 0x7f42601582f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50080/0 (socket says 172.21.15.189:50080) 2024-04-06T13:29:19.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.024+0000 7f425f7fe700 1 -- 172.21.15.189:0/1421107429 learned_addr learned my addr 172.21.15.189:0/1421107429 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:19.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.024+0000 7f425ffff700 1 --2- 172.21.15.189:0/1421107429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42600ce8b0 0x7f4260155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:19.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.024+0000 7f425f7fe700 1 -- 172.21.15.189:0/1421107429 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600cdae0 msgr2=0x7f42601f25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:19.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.024+0000 7f425effd700 1 --2- 172.21.15.189:0/1421107429 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600cdae0 0x7f42601f25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:19.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.024+0000 7f425f7fe700 1 --2- 172.21.15.189:0/1421107429 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600cdae0 0x7f42601f25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:19.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.024+0000 7f425f7fe700 1 -- 172.21.15.189:0/1421107429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42600ce8b0 msgr2=0x7f4260155dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:19.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.024+0000 7f425f7fe700 1 --2- 172.21.15.189:0/1421107429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42600ce8b0 0x7f4260155dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:19.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.024+0000 7f425f7fe700 1 -- 172.21.15.189:0/1421107429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4254009550 con 0x7f42600aec70 2024-04-06T13:29:19.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.024+0000 7f425effd700 1 --2- 172.21.15.189:0/1421107429 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600cdae0 0x7f42601f25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:29:19.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.024+0000 7f425ffff700 1 --2- 172.21.15.189:0/1421107429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42600ce8b0 0x7f4260155dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:29:19.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.024+0000 7f425f7fe700 1 --2- 172.21.15.189:0/1421107429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600aec70 0x7f42601582f0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f4254000c00 tx=0x7f4254004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:19.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.024+0000 7f4247fff700 1 -- 172.21.15.189:0/1421107429 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4254004520 con 0x7f42600aec70 2024-04-06T13:29:19.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.024+0000 7f4247fff700 1 -- 172.21.15.189:0/1421107429 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425401e070 con 0x7f42600aec70 2024-04-06T13:29:19.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.024+0000 7f4265adc700 1 -- 172.21.15.189:0/1421107429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42600b29b0 con 0x7f42600aec70 2024-04-06T13:29:19.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.024+0000 7f4247fff700 1 -- 172.21.15.189:0/1421107429 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42540038c0 con 0x7f42600aec70 2024-04-06T13:29:19.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.024+0000 7f4265adc700 1 -- 172.21.15.189:0/1421107429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4260158830 con 0x7f42600aec70 2024-04-06T13:29:19.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.025+0000 7f4265adc700 1 -- 172.21.15.189:0/1421107429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4260074990 con 0x7f42600aec70 2024-04-06T13:29:19.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.026+0000 7f4247fff700 1 -- 172.21.15.189:0/1421107429 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f4254003a20 con 0x7f42600aec70 2024-04-06T13:29:19.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.026+0000 7f4247fff700 1 --2- 172.21.15.189:0/1421107429 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f424c06adb0 0x7f424c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:19.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.026+0000 7f4247fff700 1 -- 172.21.15.189:0/1421107429 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4254020070 con 0x7f42600aec70 2024-04-06T13:29:19.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.026+0000 7f425effd700 1 --2- 172.21.15.189:0/1421107429 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f424c06adb0 0x7f424c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:19.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.026+0000 7f425effd700 1 --2- 172.21.15.189:0/1421107429 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f424c06adb0 0x7f424c06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f426006c1a0 tx=0x7f4250008040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:19.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.030+0000 7f4247fff700 1 -- 172.21.15.189:0/1421107429 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4254055af0 con 0x7f42600aec70 2024-04-06T13:29:19.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.273+0000 7f4265adc700 1 -- 172.21.15.189:0/1421107429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f42600b5790 con 0x7f42600aec70 2024-04-06T13:29:19.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.871+0000 7f4247fff700 1 -- 172.21.15.189:0/1421107429 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v85) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4254015020 con 0x7f42600aec70 2024-04-06T13:29:19.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.875+0000 7f4265adc700 1 -- 172.21.15.189:0/1421107429 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f424c06adb0 msgr2=0x7f424c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:19.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.875+0000 7f4265adc700 1 --2- 172.21.15.189:0/1421107429 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f424c06adb0 0x7f424c06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f426006c1a0 tx=0x7f4250008040 comp rx=0 tx=0).stop 2024-04-06T13:29:19.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.875+0000 7f4265adc700 1 -- 172.21.15.189:0/1421107429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600aec70 msgr2=0x7f42601582f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:19.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.875+0000 7f4265adc700 1 --2- 172.21.15.189:0/1421107429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600aec70 0x7f42601582f0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f4254000c00 tx=0x7f4254004750 comp rx=0 tx=0).stop 2024-04-06T13:29:19.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.875+0000 7f4265adc700 1 -- 172.21.15.189:0/1421107429 shutdown_connections 2024-04-06T13:29:19.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.875+0000 7f4265adc700 1 --2- 172.21.15.189:0/1421107429 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f424c06adb0 0x7f424c06d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:19.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.875+0000 7f4265adc700 1 --2- 172.21.15.189:0/1421107429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42600ce8b0 0x7f4260155dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:19.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.875+0000 7f4265adc700 1 --2- 172.21.15.189:0/1421107429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600aec70 0x7f42601582f0 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:19.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.875+0000 7f4265adc700 1 --2- 172.21.15.189:0/1421107429 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42600cdae0 0x7f42601f25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:19.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.875+0000 7f4265adc700 1 -- 172.21.15.189:0/1421107429 >> 172.21.15.189:0/1421107429 conn(0x7f42600cc290 msgr2=0x7f426006e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:19.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.875+0000 7f4265adc700 1 -- 172.21.15.189:0/1421107429 shutdown_connections 2024-04-06T13:29:19.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:19.875+0000 7f4265adc700 1 -- 172.21.15.189:0/1421107429 wait complete. 2024-04-06T13:29:19.890 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-06T13:29:19.890 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:29:20.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.018+0000 7ff2df1ec700 1 Processor -- start 2024-04-06T13:29:20.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.018+0000 7ff2df1ec700 1 -- start start 2024-04-06T13:29:20.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.019+0000 7ff2df1ec700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff2d815d7b0 0x7ff2d80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:20.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.019+0000 7ff2df1ec700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2d806a730 0x7ff2d806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:20.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.019+0000 7ff2df1ec700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff2d815ddf0 0x7ff2d815e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:20.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.019+0000 7ff2df1ec700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2d806b0f0 con 0x7ff2d815d7b0 2024-04-06T13:29:20.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.019+0000 7ff2df1ec700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2d80b3990 con 0x7ff2d815ddf0 2024-04-06T13:29:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.019+0000 7ff2df1ec700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d80b3060 con 0x7ff2d806a730 2024-04-06T13:29:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.019+0000 7ff2dcf88700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff2d815d7b0 0x7ff2d80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.019+0000 7ff2dcf88700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff2d815d7b0 0x7ff2d80b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54228/0 (socket says 172.21.15.189:54228) 2024-04-06T13:29:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.019+0000 7ff2dcf88700 1 -- 172.21.15.189:0/4049932833 learned_addr learned my addr 172.21.15.189:0/4049932833 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.019+0000 7ff2d7fff700 1 --2- 172.21.15.189:0/4049932833 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2d806a730 0x7ff2d806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.019+0000 7ff2dd789700 1 --2- 172.21.15.189:0/4049932833 >> v2:172.21.15.192:3300/0 conn(0x7ff2d815ddf0 0x7ff2d815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.019+0000 7ff2dcf88700 1 -- 172.21.15.189:0/4049932833 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2d806a730 msgr2=0x7ff2d806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.019+0000 7ff2dcf88700 1 --2- 172.21.15.189:0/4049932833 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2d806a730 0x7ff2d806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.019+0000 7ff2dcf88700 1 -- 172.21.15.189:0/4049932833 >> v2:172.21.15.192:3300/0 conn(0x7ff2d815ddf0 msgr2=0x7ff2d815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.019+0000 7ff2dcf88700 1 --2- 172.21.15.189:0/4049932833 >> v2:172.21.15.192:3300/0 conn(0x7ff2d815ddf0 0x7ff2d815e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.019+0000 7ff2dcf88700 1 -- 172.21.15.189:0/4049932833 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d80b3340 con 0x7ff2d815d7b0 2024-04-06T13:29:20.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.019+0000 7ff2dcf88700 1 --2- 172.21.15.189:0/4049932833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d815d7b0 0x7ff2d80b3e20 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7ff2d80b4360 tx=0x7ff2cc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f53f504d4a68ac9a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.019+0000 7ff2d77fe700 1 -- 172.21.15.189:0/4049932833 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc00b870 con 0x7ff2d815d7b0 2024-04-06T13:29:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.020+0000 7ff2d77fe700 1 -- 172.21.15.189:0/4049932833 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2cc00b9d0 con 0x7ff2d815d7b0 2024-04-06T13:29:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.020+0000 7ff2d77fe700 1 -- 172.21.15.189:0/4049932833 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc00bd60 con 0x7ff2d815d7b0 2024-04-06T13:29:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.020+0000 7ff2df1ec700 1 -- 172.21.15.189:0/4049932833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d815d7b0 msgr2=0x7ff2d80b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.020+0000 7ff2df1ec700 1 --2- 172.21.15.189:0/4049932833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d815d7b0 0x7ff2d80b3e20 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7ff2d80b4360 tx=0x7ff2cc009bd0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.020+0000 7ff2df1ec700 1 -- 172.21.15.189:0/4049932833 shutdown_connections 2024-04-06T13:29:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.020+0000 7ff2df1ec700 1 --2- 172.21.15.189:0/4049932833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d815d7b0 0x7ff2d80b3e20 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.020+0000 7ff2df1ec700 1 --2- 172.21.15.189:0/4049932833 >> v2:172.21.15.192:3300/0 conn(0x7ff2d815ddf0 0x7ff2d815e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.020+0000 7ff2df1ec700 1 --2- 172.21.15.189:0/4049932833 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2d806a730 0x7ff2d806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.020+0000 7ff2df1ec700 1 -- 172.21.15.189:0/4049932833 >> 172.21.15.189:0/4049932833 conn(0x7ff2d80d00b0 msgr2=0x7ff2d80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.020+0000 7ff2df1ec700 1 -- 172.21.15.189:0/4049932833 shutdown_connections 2024-04-06T13:29:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.020+0000 7ff2df1ec700 1 -- 172.21.15.189:0/4049932833 wait complete. 2024-04-06T13:29:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.022+0000 7ff2df1ec700 1 Processor -- start 2024-04-06T13:29:20.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.022+0000 7ff2df1ec700 1 -- start start 2024-04-06T13:29:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2df1ec700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d806a730 0x7ff2d81492d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2df1ec700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2d815d7b0 0x7ff2d80cf570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2df1ec700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2d815ddf0 0x7ff2d8146a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2df1ec700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d8147420 con 0x7ff2d806a730 2024-04-06T13:29:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2df1ec700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d80cfab0 con 0x7ff2d815ddf0 2024-04-06T13:29:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2df1ec700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d814a220 con 0x7ff2d815d7b0 2024-04-06T13:29:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2dcf88700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d806a730 0x7ff2d81492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2dcf88700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d806a730 0x7ff2d81492d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54240/0 (socket says 172.21.15.189:54240) 2024-04-06T13:29:20.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2dcf88700 1 -- 172.21.15.189:0/267978931 learned_addr learned my addr 172.21.15.189:0/267978931 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2d7fff700 1 --2- 172.21.15.189:0/267978931 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2d815d7b0 0x7ff2d80cf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2dd789700 1 --2- 172.21.15.189:0/267978931 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2d815ddf0 0x7ff2d8146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2dcf88700 1 -- 172.21.15.189:0/267978931 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2d815d7b0 msgr2=0x7ff2d80cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2dcf88700 1 --2- 172.21.15.189:0/267978931 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2d815d7b0 0x7ff2d80cf570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2dcf88700 1 -- 172.21.15.189:0/267978931 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2d815ddf0 msgr2=0x7ff2d8146a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2dcf88700 1 --2- 172.21.15.189:0/267978931 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2d815ddf0 0x7ff2d8146a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2dcf88700 1 -- 172.21.15.189:0/267978931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2cc009550 con 0x7ff2d806a730 2024-04-06T13:29:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2dd789700 1 --2- 172.21.15.189:0/267978931 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2d815ddf0 0x7ff2d8146a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:29:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2dcf88700 1 --2- 172.21.15.189:0/267978931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d806a730 0x7ff2d81492d0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7ff2d806c820 tx=0x7ff2cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2d57fa700 1 -- 172.21.15.189:0/267978931 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc004540 con 0x7ff2d806a730 2024-04-06T13:29:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2d57fa700 1 -- 172.21.15.189:0/267978931 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2cc0036a0 con 0x7ff2d806a730 2024-04-06T13:29:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2df1ec700 1 -- 172.21.15.189:0/267978931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2d8146f50 con 0x7ff2d806a730 2024-04-06T13:29:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2d57fa700 1 -- 172.21.15.189:0/267978931 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc003a50 con 0x7ff2d806a730 2024-04-06T13:29:20.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.023+0000 7ff2df1ec700 1 -- 172.21.15.189:0/267978931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2d8149c60 con 0x7ff2d806a730 2024-04-06T13:29:20.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.024+0000 7ff2df1ec700 1 -- 172.21.15.189:0/267978931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2d8067bb0 con 0x7ff2d806a730 2024-04-06T13:29:20.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.025+0000 7ff2d57fa700 1 -- 172.21.15.189:0/267978931 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7ff2cc021070 con 0x7ff2d806a730 2024-04-06T13:29:20.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.025+0000 7ff2d57fa700 1 --2- 172.21.15.189:0/267978931 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff2bc06adb0 0x7ff2bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:20.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.025+0000 7ff2d57fa700 1 -- 172.21.15.189:0/267978931 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff2cc08ae30 con 0x7ff2d806a730 2024-04-06T13:29:20.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.025+0000 7ff2d7fff700 1 --2- 172.21.15.189:0/267978931 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff2bc06adb0 0x7ff2bc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:20.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.026+0000 7ff2d7fff700 1 --2- 172.21.15.189:0/267978931 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff2bc06adb0 0x7ff2bc06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8160940 tx=0x7ff2c4008040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:20.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.030+0000 7ff2d57fa700 1 -- 172.21.15.189:0/267978931 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff2cc055f40 con 0x7ff2d806a730 2024-04-06T13:29:20.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.268+0000 7ff2df1ec700 1 -- 172.21.15.189:0/267978931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2d815e200 con 0x7ff2d806a730 2024-04-06T13:29:20.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.269+0000 7ff2d57fa700 1 -- 172.21.15.189:0/267978931 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+4749 (secure 0 0 0) 0x7ff2cc019e60 con 0x7ff2d806a730 2024-04-06T13:29:20.269 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:29:20.269 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":85,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8141,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2979280081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2979280081},{"type":"v1","addr":"172.21.15.192:6835","nonce":2979280081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":10125,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/286991908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":286991908},{"type":"v1","addr":"172.21.15.189:6836","nonce":286991908}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":10128,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/704323171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":704323171},{"type":"v1","addr":"172.21.15.189:6837","nonce":704323171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:29:16.858698+0000","modified":"2024-04-06T13:29:19.868525+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10137":{"gid":10137,"name":"b","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6834/2662267274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2662267274},{"type":"v1","addr":"172.21.15.192:6834","nonce":2662267274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-04-06T13:29:20.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.271+0000 7ff2df1ec700 1 -- 172.21.15.189:0/267978931 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff2bc06adb0 msgr2=0x7ff2bc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:20.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.271+0000 7ff2df1ec700 1 --2- 172.21.15.189:0/267978931 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff2bc06adb0 0x7ff2bc06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8160940 tx=0x7ff2c4008040 comp rx=0 tx=0).stop 2024-04-06T13:29:20.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.271+0000 7ff2df1ec700 1 -- 172.21.15.189:0/267978931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d806a730 msgr2=0x7ff2d81492d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:20.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.271+0000 7ff2df1ec700 1 --2- 172.21.15.189:0/267978931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d806a730 0x7ff2d81492d0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7ff2d806c820 tx=0x7ff2cc004750 comp rx=0 tx=0).stop 2024-04-06T13:29:20.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.271+0000 7ff2df1ec700 1 -- 172.21.15.189:0/267978931 shutdown_connections 2024-04-06T13:29:20.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.271+0000 7ff2df1ec700 1 --2- 172.21.15.189:0/267978931 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7ff2bc06adb0 0x7ff2bc06d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.271+0000 7ff2df1ec700 1 --2- 172.21.15.189:0/267978931 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2d815ddf0 0x7ff2d8146a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.271+0000 7ff2df1ec700 1 --2- 172.21.15.189:0/267978931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d806a730 0x7ff2d81492d0 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.271+0000 7ff2df1ec700 1 --2- 172.21.15.189:0/267978931 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2d815d7b0 0x7ff2d80cf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.271+0000 7ff2df1ec700 1 -- 172.21.15.189:0/267978931 >> 172.21.15.189:0/267978931 conn(0x7ff2d80d00b0 msgr2=0x7ff2d815f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:20.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.271+0000 7ff2df1ec700 1 -- 172.21.15.189:0/267978931 shutdown_connections 2024-04-06T13:29:20.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.271+0000 7ff2df1ec700 1 -- 172.21.15.189:0/267978931 wait complete. 2024-04-06T13:29:20.272 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 85 2024-04-06T13:29:20.283 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:29:20.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.405+0000 7f90761c2700 1 Processor -- start 2024-04-06T13:29:20.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.405+0000 7f90761c2700 1 -- start start 2024-04-06T13:29:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.406+0000 7f90761c2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f907006ae40 0x7f907015ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.406+0000 7f90761c2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f907015fcd0 0x7f907015bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.406+0000 7f90761c2700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907015c4c0 con 0x7f9070144f40 2024-04-06T13:29:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.406+0000 7f90761c2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90700cfcb0 con 0x7f907006ae40 2024-04-06T13:29:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.406+0000 7f90761c2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f907015ca20 con 0x7f907015fcd0 2024-04-06T13:29:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.406+0000 7f906effd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f907006ae40 0x7f907015ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.406+0000 7f906ffff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f907015fcd0 0x7f907015bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.406+0000 7f906effd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f907006ae40 0x7f907015ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:51912/0 (socket says 172.21.15.189:51912) 2024-04-06T13:29:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.406+0000 7f906effd700 1 -- 172.21.15.189:0/3677846251 learned_addr learned my addr 172.21.15.189:0/3677846251 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.407+0000 7f906ffff700 1 -- 172.21.15.189:0/3677846251 >> v2:172.21.15.192:3300/0 conn(0x7f907006ae40 msgr2=0x7f907015ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.407+0000 7f906ffff700 1 --2- 172.21.15.189:0/3677846251 >> v2:172.21.15.192:3300/0 conn(0x7f907006ae40 0x7f907015ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.407+0000 7f906ffff700 1 -- 172.21.15.189:0/3677846251 >> v1:172.21.15.192:6789/0 conn(0x7f9070144f40 legacy=0x7f907006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:20.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.407+0000 7f906ffff700 1 -- 172.21.15.189:0/3677846251 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907015c740 con 0x7f907015fcd0 2024-04-06T13:29:20.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.407+0000 7f906effd700 1 --2- 172.21.15.189:0/3677846251 >> v2:172.21.15.192:3300/0 conn(0x7f907006ae40 0x7f907015ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:29:20.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.407+0000 7f906ffff700 1 --2- 172.21.15.189:0/3677846251 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f907015fcd0 0x7f907015bf80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9068009d90 tx=0x7f906800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d536139c4dc3f81 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:20.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.407+0000 7f906e7fc700 1 -- 172.21.15.189:0/3677846251 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906800e070 con 0x7f907015fcd0 2024-04-06T13:29:20.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.407+0000 7f906e7fc700 1 -- 172.21.15.189:0/3677846251 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906800bb20 con 0x7f907015fcd0 2024-04-06T13:29:20.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.407+0000 7f906e7fc700 1 -- 172.21.15.189:0/3677846251 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9068012430 con 0x7f907015fcd0 2024-04-06T13:29:20.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.408+0000 7f90761c2700 1 -- 172.21.15.189:0/3677846251 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f907015fcd0 msgr2=0x7f907015bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:20.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.408+0000 7f90761c2700 1 --2- 172.21.15.189:0/3677846251 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f907015fcd0 0x7f907015bf80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9068009d90 tx=0x7f906800b330 comp rx=0 tx=0).stop 2024-04-06T13:29:20.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.408+0000 7f90761c2700 1 -- 172.21.15.189:0/3677846251 shutdown_connections 2024-04-06T13:29:20.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.408+0000 7f90761c2700 1 --2- 172.21.15.189:0/3677846251 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f907015fcd0 0x7f907015bf80 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.408+0000 7f90761c2700 1 --2- 172.21.15.189:0/3677846251 >> v2:172.21.15.192:3300/0 conn(0x7f907006ae40 0x7f907015ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.408+0000 7f90761c2700 1 -- 172.21.15.189:0/3677846251 >> 172.21.15.189:0/3677846251 conn(0x7f90700d00b0 msgr2=0x7f90700d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:20.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.408+0000 7f90761c2700 1 -- 172.21.15.189:0/3677846251 shutdown_connections 2024-04-06T13:29:20.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.408+0000 7f90761c2700 1 -- 172.21.15.189:0/3677846251 wait complete. 2024-04-06T13:29:20.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.409+0000 7f90761c2700 1 Processor -- start 2024-04-06T13:29:20.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.409+0000 7f90761c2700 1 -- start start 2024-04-06T13:29:20.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.410+0000 7f90761c2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f907006ae40 0x7f907013e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:20.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.410+0000 7f90761c2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9070144f40 0x7f90700c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:20.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.410+0000 7f90761c2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f907015fcd0 0x7f90701436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:20.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.410+0000 7f90761c2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90700c9360 con 0x7f9070144f40 2024-04-06T13:29:20.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.410+0000 7f90761c2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90700c9640 con 0x7f907015fcd0 2024-04-06T13:29:20.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.410+0000 7f90761c2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90700c9920 con 0x7f907006ae40 2024-04-06T13:29:20.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.410+0000 7f906effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9070144f40 0x7f90700c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:20.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.410+0000 7f906effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9070144f40 0x7f90700c88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54256/0 (socket says 172.21.15.189:54256) 2024-04-06T13:29:20.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.410+0000 7f906effd700 1 -- 172.21.15.189:0/1676035724 learned_addr learned my addr 172.21.15.189:0/1676035724 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:20.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.410+0000 7f906f7fe700 1 --2- 172.21.15.189:0/1676035724 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f907006ae40 0x7f907013e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:20.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.410+0000 7f906ffff700 1 --2- 172.21.15.189:0/1676035724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f907015fcd0 0x7f90701436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:20.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.410+0000 7f906effd700 1 -- 172.21.15.189:0/1676035724 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f907006ae40 msgr2=0x7f907013e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:20.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.410+0000 7f906effd700 1 --2- 172.21.15.189:0/1676035724 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f907006ae40 0x7f907013e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.410+0000 7f906effd700 1 -- 172.21.15.189:0/1676035724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f907015fcd0 msgr2=0x7f90701436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:20.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.410+0000 7f906effd700 1 --2- 172.21.15.189:0/1676035724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f907015fcd0 0x7f90701436b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.410+0000 7f906effd700 1 -- 172.21.15.189:0/1676035724 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9060009550 con 0x7f9070144f40 2024-04-06T13:29:20.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.410+0000 7f906effd700 1 --2- 172.21.15.189:0/1676035724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9070144f40 0x7f90700c88a0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f906000ebb0 tx=0x7f906000eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:20.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.411+0000 7f9057fff700 1 -- 172.21.15.189:0/1676035724 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9060004240 con 0x7f9070144f40 2024-04-06T13:29:20.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.411+0000 7f9057fff700 1 -- 172.21.15.189:0/1676035724 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90600043a0 con 0x7f9070144f40 2024-04-06T13:29:20.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.411+0000 7f9057fff700 1 -- 172.21.15.189:0/1676035724 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90600036a0 con 0x7f9070144f40 2024-04-06T13:29:20.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.411+0000 7f90761c2700 1 -- 172.21.15.189:0/1676035724 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9068009a70 con 0x7f9070144f40 2024-04-06T13:29:20.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.411+0000 7f90761c2700 1 -- 172.21.15.189:0/1676035724 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907013f190 con 0x7f9070144f40 2024-04-06T13:29:20.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.412+0000 7f90761c2700 1 -- 172.21.15.189:0/1676035724 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9070067bb0 con 0x7f9070144f40 2024-04-06T13:29:20.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.412+0000 7f9057fff700 1 -- 172.21.15.189:0/1676035724 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f9060003800 con 0x7f9070144f40 2024-04-06T13:29:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.412+0000 7f9057fff700 1 --2- 172.21.15.189:0/1676035724 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f905c06adb0 0x7f905c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.412+0000 7f9057fff700 1 -- 172.21.15.189:0/1676035724 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f90600897a0 con 0x7f9070144f40 2024-04-06T13:29:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.412+0000 7f906f7fe700 1 --2- 172.21.15.189:0/1676035724 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f905c06adb0 0x7f905c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.413+0000 7f906f7fe700 1 --2- 172.21.15.189:0/1676035724 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f905c06adb0 0x7f905c06d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9064007fd0 tx=0x7f9064009040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.417+0000 7f9057fff700 1 -- 172.21.15.189:0/1676035724 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9060054700 con 0x7f9070144f40 2024-04-06T13:29:20.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.614+0000 7f90761c2700 1 -- 172.21.15.189:0/1676035724 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9070074990 con 0x7f9070144f40 2024-04-06T13:29:20.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.615+0000 7f9057fff700 1 -- 172.21.15.189:0/1676035724 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) v1 ==== 74+0+16366 (secure 0 0 0) 0x7f9060057d20 con 0x7f9070144f40 2024-04-06T13:29:20.615 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:29:20.615 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":192,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:29:17.860424+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:29:14.625778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-04-06T13:29:15.827532+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:29:20.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.617+0000 7f90761c2700 1 -- 172.21.15.189:0/1676035724 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f905c06adb0 msgr2=0x7f905c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:20.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.617+0000 7f90761c2700 1 --2- 172.21.15.189:0/1676035724 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f905c06adb0 0x7f905c06d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9064007fd0 tx=0x7f9064009040 comp rx=0 tx=0).stop 2024-04-06T13:29:20.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.617+0000 7f90761c2700 1 -- 172.21.15.189:0/1676035724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9070144f40 msgr2=0x7f90700c88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:20.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.617+0000 7f90761c2700 1 --2- 172.21.15.189:0/1676035724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9070144f40 0x7f90700c88a0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f906000ebb0 tx=0x7f906000eec0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.617+0000 7f90761c2700 1 -- 172.21.15.189:0/1676035724 shutdown_connections 2024-04-06T13:29:20.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.617+0000 7f90761c2700 1 --2- 172.21.15.189:0/1676035724 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f905c06adb0 0x7f905c06d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.617+0000 7f90761c2700 1 --2- 172.21.15.189:0/1676035724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f907015fcd0 0x7f90701436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.617+0000 7f90761c2700 1 --2- 172.21.15.189:0/1676035724 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f907006ae40 0x7f907013e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.617+0000 7f90761c2700 1 --2- 172.21.15.189:0/1676035724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9070144f40 0x7f90700c88a0 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.617+0000 7f90761c2700 1 -- 172.21.15.189:0/1676035724 >> 172.21.15.189:0/1676035724 conn(0x7f90700d00b0 msgr2=0x7f907015b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:20.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.617+0000 7f90761c2700 1 -- 172.21.15.189:0/1676035724 shutdown_connections 2024-04-06T13:29:20.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.617+0000 7f90761c2700 1 -- 172.21.15.189:0/1676035724 wait complete. 2024-04-06T13:29:20.630 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-06T13:29:20.630 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:29:20.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.752+0000 7fd53e3b2700 1 Processor -- start 2024-04-06T13:29:20.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.752+0000 7fd53e3b2700 1 -- start start 2024-04-06T13:29:20.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.753+0000 7fd53e3b2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5380cedf0 0x7fd5380cf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:20.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.753+0000 7fd53e3b2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5380cf780 con 0x7fd5380d17a0 2024-04-06T13:29:20.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.753+0000 7fd53e3b2700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd53806d850 con 0x7fd538146dd0 2024-04-06T13:29:20.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.753+0000 7fd53e3b2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53806beb0 con 0x7fd5380cedf0 2024-04-06T13:29:20.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.753+0000 7fd537fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd5380d17a0 0x7fd53806b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51966/0 (socket says 172.21.15.189:51966) 2024-04-06T13:29:20.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.753+0000 7fd537fff700 1 -- 172.21.15.189:0/1063930935 learned_addr learned my addr 172.21.15.189:0/1063930935 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:20.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.754+0000 7fd53c94f700 1 --2- 172.21.15.189:0/1063930935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5380cedf0 0x7fd5380cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:20.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.754+0000 7fd536ffd700 1 -- 172.21.15.189:0/1063930935 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824703232 0 0) 0x7fd528001110 con 0x7fd5380d17a0 2024-04-06T13:29:20.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.754+0000 7fd536ffd700 1 -- 172.21.15.189:0/1063930935 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5240035b0 con 0x7fd5380d17a0 2024-04-06T13:29:20.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.754+0000 7fd536ffd700 1 -- 172.21.15.189:0/1063930935 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800860601 0 0) 0x7fd53806d850 con 0x7fd538146dd0 2024-04-06T13:29:20.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.754+0000 7fd536ffd700 1 -- 172.21.15.189:0/1063930935 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd528001110 con 0x7fd538146dd0 2024-04-06T13:29:20.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.754+0000 7fd536ffd700 1 -- 172.21.15.189:0/1063930935 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4094638116 0 0) 0x7fd5240035b0 con 0x7fd5380d17a0 2024-04-06T13:29:20.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.754+0000 7fd53c94f700 1 -- 172.21.15.189:0/1063930935 >> v1:172.21.15.192:6789/0 conn(0x7fd538146dd0 legacy=0x7fd53806d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:20.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.754+0000 7fd53c94f700 1 -- 172.21.15.189:0/1063930935 >> v1:172.21.15.189:6789/0 conn(0x7fd5380d17a0 legacy=0x7fd53806b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:20.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.754+0000 7fd53c94f700 1 -- 172.21.15.189:0/1063930935 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53806c0a0 con 0x7fd5380cedf0 2024-04-06T13:29:20.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.754+0000 7fd53c94f700 1 --2- 172.21.15.189:0/1063930935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5380cedf0 0x7fd5380cf240 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd530009dc0 tx=0x7fd53000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cb3716f3eefa45 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:20.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.754+0000 7fd536ffd700 1 -- 172.21.15.189:0/1063930935 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53000e070 con 0x7fd5380cedf0 2024-04-06T13:29:20.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.754+0000 7fd536ffd700 1 -- 172.21.15.189:0/1063930935 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd530003a20 con 0x7fd5380cedf0 2024-04-06T13:29:20.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.754+0000 7fd536ffd700 1 -- 172.21.15.189:0/1063930935 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd528001800 con 0x7fd5380cedf0 2024-04-06T13:29:20.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.755+0000 7fd53e3b2700 1 -- 172.21.15.189:0/1063930935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5380cedf0 msgr2=0x7fd5380cf240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:20.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.755+0000 7fd53e3b2700 1 --2- 172.21.15.189:0/1063930935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5380cedf0 0x7fd5380cf240 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd530009dc0 tx=0x7fd53000b330 comp rx=0 tx=0).stop 2024-04-06T13:29:20.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.755+0000 7fd53e3b2700 1 -- 172.21.15.189:0/1063930935 shutdown_connections 2024-04-06T13:29:20.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.755+0000 7fd53e3b2700 1 --2- 172.21.15.189:0/1063930935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5380cedf0 0x7fd5380cf240 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.755+0000 7fd53e3b2700 1 -- 172.21.15.189:0/1063930935 >> 172.21.15.189:0/1063930935 conn(0x7fd5380b2db0 msgr2=0x7fd5380b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:20.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.755+0000 7fd53e3b2700 1 -- 172.21.15.189:0/1063930935 shutdown_connections 2024-04-06T13:29:20.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.755+0000 7fd53e3b2700 1 -- 172.21.15.189:0/1063930935 wait complete. 2024-04-06T13:29:20.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.756+0000 7fd53e3b2700 1 Processor -- start 2024-04-06T13:29:20.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.756+0000 7fd53e3b2700 1 -- start start 2024-04-06T13:29:20.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.757+0000 7fd53e3b2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5380cedf0 0x7fd5380ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:20.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.757+0000 7fd53e3b2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5380d17a0 0x7fd538151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:20.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.757+0000 7fd53e3b2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd538146dd0 0x7fd5380c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:20.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.757+0000 7fd53e3b2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5380af7d0 con 0x7fd538146dd0 2024-04-06T13:29:20.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.757+0000 7fd53e3b2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5380adb30 con 0x7fd5380cedf0 2024-04-06T13:29:20.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.757+0000 7fd53e3b2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd538152300 con 0x7fd5380d17a0 2024-04-06T13:29:20.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.757+0000 7fd53c94f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd538146dd0 0x7fd5380c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:20.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.757+0000 7fd53c94f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd538146dd0 0x7fd5380c7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54262/0 (socket says 172.21.15.189:54262) 2024-04-06T13:29:20.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.757+0000 7fd53c94f700 1 -- 172.21.15.189:0/907799989 learned_addr learned my addr 172.21.15.189:0/907799989 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:20.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.757+0000 7fd537fff700 1 --2- 172.21.15.189:0/907799989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5380cedf0 0x7fd5380ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:20.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.757+0000 7fd5377fe700 1 --2- 172.21.15.189:0/907799989 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5380d17a0 0x7fd538151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:20.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.757+0000 7fd53c94f700 1 -- 172.21.15.189:0/907799989 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5380d17a0 msgr2=0x7fd538151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:20.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.757+0000 7fd53c94f700 1 --2- 172.21.15.189:0/907799989 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5380d17a0 0x7fd538151dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.757+0000 7fd53c94f700 1 -- 172.21.15.189:0/907799989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5380cedf0 msgr2=0x7fd5380ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:20.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.757+0000 7fd53c94f700 1 --2- 172.21.15.189:0/907799989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5380cedf0 0x7fd5380ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.757+0000 7fd53c94f700 1 -- 172.21.15.189:0/907799989 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd530009a70 con 0x7fd538146dd0 2024-04-06T13:29:20.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.757+0000 7fd537fff700 1 --2- 172.21.15.189:0/907799989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5380cedf0 0x7fd5380ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:29:20.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.757+0000 7fd5377fe700 1 --2- 172.21.15.189:0/907799989 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5380d17a0 0x7fd538151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:29:20.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.758+0000 7fd53c94f700 1 --2- 172.21.15.189:0/907799989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd538146dd0 0x7fd5380c7040 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fd530015040 tx=0x7fd530004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:20.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.758+0000 7fd534ff9700 1 -- 172.21.15.189:0/907799989 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd530004400 con 0x7fd538146dd0 2024-04-06T13:29:20.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.758+0000 7fd534ff9700 1 -- 172.21.15.189:0/907799989 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd530004560 con 0x7fd538146dd0 2024-04-06T13:29:20.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.758+0000 7fd53e3b2700 1 -- 172.21.15.189:0/907799989 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5380aceb0 con 0x7fd538146dd0 2024-04-06T13:29:20.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.758+0000 7fd534ff9700 1 -- 172.21.15.189:0/907799989 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd530004910 con 0x7fd538146dd0 2024-04-06T13:29:20.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.758+0000 7fd53e3b2700 1 -- 172.21.15.189:0/907799989 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5380ad470 con 0x7fd538146dd0 2024-04-06T13:29:20.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.759+0000 7fd53e3b2700 1 -- 172.21.15.189:0/907799989 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5380adc70 con 0x7fd538146dd0 2024-04-06T13:29:20.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.759+0000 7fd534ff9700 1 -- 172.21.15.189:0/907799989 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fd53001c460 con 0x7fd538146dd0 2024-04-06T13:29:20.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.759+0000 7fd534ff9700 1 --2- 172.21.15.189:0/907799989 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd51806adb0 0x7fd51806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:20.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.760+0000 7fd534ff9700 1 -- 172.21.15.189:0/907799989 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd53008ada0 con 0x7fd538146dd0 2024-04-06T13:29:20.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.760+0000 7fd537fff700 1 --2- 172.21.15.189:0/907799989 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd51806adb0 0x7fd51806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:20.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.760+0000 7fd537fff700 1 --2- 172.21.15.189:0/907799989 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd51806adb0 0x7fd51806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd528007a40 tx=0x7fd528009040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:20.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.764+0000 7fd534ff9700 1 -- 172.21.15.189:0/907799989 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd5380adc70 con 0x7fd538146dd0 2024-04-06T13:29:20.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.956+0000 7fd53e3b2700 1 -- 172.21.15.189:0/907799989 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd53806c940 con 0x7fd51806adb0 2024-04-06T13:29:20.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.964+0000 7fd534ff9700 1 -- 172.21.15.189:0/907799989 <== mgr.9034 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289989 (secure 0 0 0) 0x7fd5280070d0 con 0x7fd51806adb0 2024-04-06T13:29:20.968 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:29:20.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.968+0000 7fd53e3b2700 1 -- 172.21.15.189:0/907799989 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd51806adb0 msgr2=0x7fd51806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:20.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.968+0000 7fd53e3b2700 1 --2- 172.21.15.189:0/907799989 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd51806adb0 0x7fd51806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd528007a40 tx=0x7fd528009040 comp rx=0 tx=0).stop 2024-04-06T13:29:20.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.968+0000 7fd53e3b2700 1 -- 172.21.15.189:0/907799989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd538146dd0 msgr2=0x7fd5380c7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:20.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.968+0000 7fd53e3b2700 1 --2- 172.21.15.189:0/907799989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd538146dd0 0x7fd5380c7040 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fd530015040 tx=0x7fd530004ad0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.968+0000 7fd53e3b2700 1 -- 172.21.15.189:0/907799989 shutdown_connections 2024-04-06T13:29:20.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.968+0000 7fd53e3b2700 1 --2- 172.21.15.189:0/907799989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd538146dd0 0x7fd5380c7040 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.968+0000 7fd53e3b2700 1 --2- 172.21.15.189:0/907799989 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7fd51806adb0 0x7fd51806d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.968+0000 7fd53e3b2700 1 --2- 172.21.15.189:0/907799989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5380cedf0 0x7fd5380ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.968+0000 7fd53e3b2700 1 --2- 172.21.15.189:0/907799989 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5380d17a0 0x7fd538151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:20.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.968+0000 7fd53e3b2700 1 -- 172.21.15.189:0/907799989 >> 172.21.15.189:0/907799989 conn(0x7fd5380b2db0 msgr2=0x7fd5380b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:20.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.968+0000 7fd53e3b2700 1 -- 172.21.15.189:0/907799989 shutdown_connections 2024-04-06T13:29:20.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:20.968+0000 7fd53e3b2700 1 -- 172.21.15.189:0/907799989 wait complete. 2024-04-06T13:29:20.972 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:29:20.980 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-06T13:29:19.217108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":523,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":331,"ondisk_log_size":331,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":184,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":386168,"kb_used_data":19952,"kb_used_omap":12,"kb_used_meta":366067,"kb_avail":754588552,"statfs":{"total":773094113280,"available":772698677248,"internally_reserved":0,"allocated":20430848,"data_stored":10890269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":374853176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000666"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870006+0000","last_change":"2024-04-06T13:29:16.887629+0000","last_active":"2024-04-06T13:29:17.870006+0000","last_peered":"2024-04-06T13:29:17.870006+0000","last_clean":"2024-04-06T13:29:17.870006+0000","last_became_active":"2024-04-06T13:29:16.887340+0000","last_became_peered":"2024-04-06T13:29:16.887340+0000","last_unstale":"2024-04-06T13:29:17.870006+0000","last_undegraded":"2024-04-06T13:29:17.870006+0000","last_fullsized":"2024-04-06T13:29:17.870006+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:32.732586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870840+0000","last_change":"2024-04-06T13:29:16.882535+0000","last_active":"2024-04-06T13:29:17.870840+0000","last_peered":"2024-04-06T13:29:17.870840+0000","last_clean":"2024-04-06T13:29:17.870840+0000","last_became_active":"2024-04-06T13:29:16.882219+0000","last_became_peered":"2024-04-06T13:29:16.882219+0000","last_unstale":"2024-04-06T13:29:17.870840+0000","last_undegraded":"2024-04-06T13:29:17.870840+0000","last_fullsized":"2024-04-06T13:29:17.870840+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:51:38.673432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870321+0000","last_change":"2024-04-06T13:29:16.885378+0000","last_active":"2024-04-06T13:29:17.870321+0000","last_peered":"2024-04-06T13:29:17.870321+0000","last_clean":"2024-04-06T13:29:17.870321+0000","last_became_active":"2024-04-06T13:29:16.885055+0000","last_became_peered":"2024-04-06T13:29:16.885055+0000","last_unstale":"2024-04-06T13:29:17.870321+0000","last_undegraded":"2024-04-06T13:29:17.870321+0000","last_fullsized":"2024-04-06T13:29:17.870321+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:38:42.932002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869716+0000","last_change":"2024-04-06T13:29:16.887481+0000","last_active":"2024-04-06T13:29:17.869716+0000","last_peered":"2024-04-06T13:29:17.869716+0000","last_clean":"2024-04-06T13:29:17.869716+0000","last_became_active":"2024-04-06T13:29:16.887096+0000","last_became_peered":"2024-04-06T13:29:16.887096+0000","last_unstale":"2024-04-06T13:29:17.869716+0000","last_undegraded":"2024-04-06T13:29:17.869716+0000","last_fullsized":"2024-04-06T13:29:17.869716+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:06:40.389835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015524+0000","last_change":"2024-04-06T13:29:16.888478+0000","last_active":"2024-04-06T13:29:19.015524+0000","last_peered":"2024-04-06T13:29:19.015524+0000","last_clean":"2024-04-06T13:29:19.015524+0000","last_became_active":"2024-04-06T13:29:16.888164+0000","last_became_peered":"2024-04-06T13:29:16.888164+0000","last_unstale":"2024-04-06T13:29:19.015524+0000","last_undegraded":"2024-04-06T13:29:19.015524+0000","last_fullsized":"2024-04-06T13:29:19.015524+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:10:03.720777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870250+0000","last_change":"2024-04-06T13:29:16.879186+0000","last_active":"2024-04-06T13:29:17.870250+0000","last_peered":"2024-04-06T13:29:17.870250+0000","last_clean":"2024-04-06T13:29:17.870250+0000","last_became_active":"2024-04-06T13:29:16.878952+0000","last_became_peered":"2024-04-06T13:29:16.878952+0000","last_unstale":"2024-04-06T13:29:17.870250+0000","last_undegraded":"2024-04-06T13:29:17.870250+0000","last_fullsized":"2024-04-06T13:29:17.870250+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:31:46.993248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031667+0000","last_change":"2024-04-06T13:29:16.880932+0000","last_active":"2024-04-06T13:29:18.031667+0000","last_peered":"2024-04-06T13:29:18.031667+0000","last_clean":"2024-04-06T13:29:18.031667+0000","last_became_active":"2024-04-06T13:29:16.880184+0000","last_became_peered":"2024-04-06T13:29:16.880184+0000","last_unstale":"2024-04-06T13:29:18.031667+0000","last_undegraded":"2024-04-06T13:29:18.031667+0000","last_fullsized":"2024-04-06T13:29:18.031667+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:55:24.254991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031614+0000","last_change":"2024-04-06T13:29:16.886799+0000","last_active":"2024-04-06T13:29:18.031614+0000","last_peered":"2024-04-06T13:29:18.031614+0000","last_clean":"2024-04-06T13:29:18.031614+0000","last_became_active":"2024-04-06T13:29:16.885980+0000","last_became_peered":"2024-04-06T13:29:16.885980+0000","last_unstale":"2024-04-06T13:29:18.031614+0000","last_undegraded":"2024-04-06T13:29:18.031614+0000","last_fullsized":"2024-04-06T13:29:18.031614+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:17:51.615500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.032000+0000","last_change":"2024-04-06T13:29:16.879765+0000","last_active":"2024-04-06T13:29:18.032000+0000","last_peered":"2024-04-06T13:29:18.032000+0000","last_clean":"2024-04-06T13:29:18.032000+0000","last_became_active":"2024-04-06T13:29:16.878170+0000","last_became_peered":"2024-04-06T13:29:16.878170+0000","last_unstale":"2024-04-06T13:29:18.032000+0000","last_undegraded":"2024-04-06T13:29:18.032000+0000","last_fullsized":"2024-04-06T13:29:18.032000+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:51:38.673432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.032040+0000","last_change":"2024-04-06T13:29:16.885817+0000","last_active":"2024-04-06T13:29:18.032040+0000","last_peered":"2024-04-06T13:29:18.032040+0000","last_clean":"2024-04-06T13:29:18.032040+0000","last_became_active":"2024-04-06T13:29:16.885479+0000","last_became_peered":"2024-04-06T13:29:16.885479+0000","last_unstale":"2024-04-06T13:29:18.032040+0000","last_undegraded":"2024-04-06T13:29:18.032040+0000","last_fullsized":"2024-04-06T13:29:18.032040+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:29:16.732050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015859+0000","last_change":"2024-04-06T13:29:16.883659+0000","last_active":"2024-04-06T13:29:19.015859+0000","last_peered":"2024-04-06T13:29:19.015859+0000","last_clean":"2024-04-06T13:29:19.015859+0000","last_became_active":"2024-04-06T13:29:16.883354+0000","last_became_peered":"2024-04-06T13:29:16.883354+0000","last_unstale":"2024-04-06T13:29:19.015859+0000","last_undegraded":"2024-04-06T13:29:19.015859+0000","last_fullsized":"2024-04-06T13:29:19.015859+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:45:02.241916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869830+0000","last_change":"2024-04-06T13:29:16.887846+0000","last_active":"2024-04-06T13:29:17.869830+0000","last_peered":"2024-04-06T13:29:17.869830+0000","last_clean":"2024-04-06T13:29:17.869830+0000","last_became_active":"2024-04-06T13:29:16.887563+0000","last_became_peered":"2024-04-06T13:29:16.887563+0000","last_unstale":"2024-04-06T13:29:17.869830+0000","last_undegraded":"2024-04-06T13:29:17.869830+0000","last_fullsized":"2024-04-06T13:29:17.869830+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:44:50.624713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869522+0000","last_change":"2024-04-06T13:29:16.870100+0000","last_active":"2024-04-06T13:29:17.869522+0000","last_peered":"2024-04-06T13:29:17.869522+0000","last_clean":"2024-04-06T13:29:17.869522+0000","last_became_active":"2024-04-06T13:29:16.869874+0000","last_became_peered":"2024-04-06T13:29:16.869874+0000","last_unstale":"2024-04-06T13:29:17.869522+0000","last_undegraded":"2024-04-06T13:29:17.869522+0000","last_fullsized":"2024-04-06T13:29:17.869522+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:40:22.459776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015446+0000","last_change":"2024-04-06T13:29:16.889997+0000","last_active":"2024-04-06T13:29:19.015446+0000","last_peered":"2024-04-06T13:29:19.015446+0000","last_clean":"2024-04-06T13:29:19.015446+0000","last_became_active":"2024-04-06T13:29:16.889378+0000","last_became_peered":"2024-04-06T13:29:16.889378+0000","last_unstale":"2024-04-06T13:29:19.015446+0000","last_undegraded":"2024-04-06T13:29:19.015446+0000","last_fullsized":"2024-04-06T13:29:19.015446+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:06:59.197190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.016173+0000","last_change":"2024-04-06T13:29:16.891204+0000","last_active":"2024-04-06T13:29:19.016173+0000","last_peered":"2024-04-06T13:29:19.016173+0000","last_clean":"2024-04-06T13:29:19.016173+0000","last_became_active":"2024-04-06T13:29:16.890910+0000","last_became_peered":"2024-04-06T13:29:16.890910+0000","last_unstale":"2024-04-06T13:29:19.016173+0000","last_undegraded":"2024-04-06T13:29:19.016173+0000","last_fullsized":"2024-04-06T13:29:19.016173+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:17:57.757521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869361+0000","last_change":"2024-04-06T13:29:16.885388+0000","last_active":"2024-04-06T13:29:17.869361+0000","last_peered":"2024-04-06T13:29:17.869361+0000","last_clean":"2024-04-06T13:29:17.869361+0000","last_became_active":"2024-04-06T13:29:16.885101+0000","last_became_peered":"2024-04-06T13:29:16.885101+0000","last_unstale":"2024-04-06T13:29:17.869361+0000","last_undegraded":"2024-04-06T13:29:17.869361+0000","last_fullsized":"2024-04-06T13:29:17.869361+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:28:11.407947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.032039+0000","last_change":"2024-04-06T13:29:16.882768+0000","last_active":"2024-04-06T13:29:18.032039+0000","last_peered":"2024-04-06T13:29:18.032039+0000","last_clean":"2024-04-06T13:29:18.032039+0000","last_became_active":"2024-04-06T13:29:16.882522+0000","last_became_peered":"2024-04-06T13:29:16.882522+0000","last_unstale":"2024-04-06T13:29:18.032039+0000","last_undegraded":"2024-04-06T13:29:18.032039+0000","last_fullsized":"2024-04-06T13:29:18.032039+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:44:33.412863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015810+0000","last_change":"2024-04-06T13:29:16.892552+0000","last_active":"2024-04-06T13:29:19.015810+0000","last_peered":"2024-04-06T13:29:19.015810+0000","last_clean":"2024-04-06T13:29:19.015810+0000","last_became_active":"2024-04-06T13:29:16.892251+0000","last_became_peered":"2024-04-06T13:29:16.892251+0000","last_unstale":"2024-04-06T13:29:19.015810+0000","last_undegraded":"2024-04-06T13:29:19.015810+0000","last_fullsized":"2024-04-06T13:29:19.015810+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:47:10.159438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015958+0000","last_change":"2024-04-06T13:29:16.891253+0000","last_active":"2024-04-06T13:29:19.015958+0000","last_peered":"2024-04-06T13:29:19.015958+0000","last_clean":"2024-04-06T13:29:19.015958+0000","last_became_active":"2024-04-06T13:29:16.890946+0000","last_became_peered":"2024-04-06T13:29:16.890946+0000","last_unstale":"2024-04-06T13:29:19.015958+0000","last_undegraded":"2024-04-06T13:29:19.015958+0000","last_fullsized":"2024-04-06T13:29:19.015958+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:25:07.172886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.514888+0000","last_change":"2024-04-06T13:29:16.882939+0000","last_active":"2024-04-06T13:29:18.514888+0000","last_peered":"2024-04-06T13:29:18.514888+0000","last_clean":"2024-04-06T13:29:18.514888+0000","last_became_active":"2024-04-06T13:29:16.882654+0000","last_became_peered":"2024-04-06T13:29:16.882654+0000","last_unstale":"2024-04-06T13:29:18.514888+0000","last_undegraded":"2024-04-06T13:29:18.514888+0000","last_fullsized":"2024-04-06T13:29:18.514888+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:18:44.916349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869613+0000","last_change":"2024-04-06T13:29:16.887439+0000","last_active":"2024-04-06T13:29:17.869613+0000","last_peered":"2024-04-06T13:29:17.869613+0000","last_clean":"2024-04-06T13:29:17.869613+0000","last_became_active":"2024-04-06T13:29:16.887073+0000","last_became_peered":"2024-04-06T13:29:16.887073+0000","last_unstale":"2024-04-06T13:29:17.869613+0000","last_undegraded":"2024-04-06T13:29:17.869613+0000","last_fullsized":"2024-04-06T13:29:17.869613+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:55:35.395643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870360+0000","last_change":"2024-04-06T13:29:16.882888+0000","last_active":"2024-04-06T13:29:17.870360+0000","last_peered":"2024-04-06T13:29:17.870360+0000","last_clean":"2024-04-06T13:29:17.870360+0000","last_became_active":"2024-04-06T13:29:16.882462+0000","last_became_peered":"2024-04-06T13:29:16.882462+0000","last_unstale":"2024-04-06T13:29:17.870360+0000","last_undegraded":"2024-04-06T13:29:17.870360+0000","last_fullsized":"2024-04-06T13:29:17.870360+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:41:05.238951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015442+0000","last_change":"2024-04-06T13:29:16.880186+0000","last_active":"2024-04-06T13:29:19.015442+0000","last_peered":"2024-04-06T13:29:19.015442+0000","last_clean":"2024-04-06T13:29:19.015442+0000","last_became_active":"2024-04-06T13:29:16.879763+0000","last_became_peered":"2024-04-06T13:29:16.879763+0000","last_unstale":"2024-04-06T13:29:19.015442+0000","last_undegraded":"2024-04-06T13:29:19.015442+0000","last_fullsized":"2024-04-06T13:29:19.015442+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:40:51.382582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869453+0000","last_change":"2024-04-06T13:29:16.885554+0000","last_active":"2024-04-06T13:29:17.869453+0000","last_peered":"2024-04-06T13:29:17.869453+0000","last_clean":"2024-04-06T13:29:17.869453+0000","last_became_active":"2024-04-06T13:29:16.885109+0000","last_became_peered":"2024-04-06T13:29:16.885109+0000","last_unstale":"2024-04-06T13:29:17.869453+0000","last_undegraded":"2024-04-06T13:29:17.869453+0000","last_fullsized":"2024-04-06T13:29:17.869453+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:27:59.484229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869787+0000","last_change":"2024-04-06T13:29:16.882750+0000","last_active":"2024-04-06T13:29:17.869787+0000","last_peered":"2024-04-06T13:29:17.869787+0000","last_clean":"2024-04-06T13:29:17.869787+0000","last_became_active":"2024-04-06T13:29:16.882522+0000","last_became_peered":"2024-04-06T13:29:16.882522+0000","last_unstale":"2024-04-06T13:29:17.869787+0000","last_undegraded":"2024-04-06T13:29:17.869787+0000","last_fullsized":"2024-04-06T13:29:17.869787+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:00:02.805980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.515109+0000","last_change":"2024-04-06T13:29:16.892594+0000","last_active":"2024-04-06T13:29:18.515109+0000","last_peered":"2024-04-06T13:29:18.515109+0000","last_clean":"2024-04-06T13:29:18.515109+0000","last_became_active":"2024-04-06T13:29:16.892326+0000","last_became_peered":"2024-04-06T13:29:16.892326+0000","last_unstale":"2024-04-06T13:29:18.515109+0000","last_undegraded":"2024-04-06T13:29:18.515109+0000","last_fullsized":"2024-04-06T13:29:18.515109+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:24:22.331863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870741+0000","last_change":"2024-04-06T13:29:16.882855+0000","last_active":"2024-04-06T13:29:17.870741+0000","last_peered":"2024-04-06T13:29:17.870741+0000","last_clean":"2024-04-06T13:29:17.870741+0000","last_became_active":"2024-04-06T13:29:16.882554+0000","last_became_peered":"2024-04-06T13:29:16.882554+0000","last_unstale":"2024-04-06T13:29:17.870741+0000","last_undegraded":"2024-04-06T13:29:17.870741+0000","last_fullsized":"2024-04-06T13:29:17.870741+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:34:35.301684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869518+0000","last_change":"2024-04-06T13:29:16.888153+0000","last_active":"2024-04-06T13:29:17.869518+0000","last_peered":"2024-04-06T13:29:17.869518+0000","last_clean":"2024-04-06T13:29:17.869518+0000","last_became_active":"2024-04-06T13:29:16.887688+0000","last_became_peered":"2024-04-06T13:29:16.887688+0000","last_unstale":"2024-04-06T13:29:17.869518+0000","last_undegraded":"2024-04-06T13:29:17.869518+0000","last_fullsized":"2024-04-06T13:29:17.869518+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:03:10.474463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870723+0000","last_change":"2024-04-06T13:29:16.891407+0000","last_active":"2024-04-06T13:29:17.870723+0000","last_peered":"2024-04-06T13:29:17.870723+0000","last_clean":"2024-04-06T13:29:17.870723+0000","last_became_active":"2024-04-06T13:29:16.891087+0000","last_became_peered":"2024-04-06T13:29:16.891087+0000","last_unstale":"2024-04-06T13:29:17.870723+0000","last_undegraded":"2024-04-06T13:29:17.870723+0000","last_fullsized":"2024-04-06T13:29:17.870723+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:26:30.192329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015798+0000","last_change":"2024-04-06T13:29:16.888009+0000","last_active":"2024-04-06T13:29:19.015798+0000","last_peered":"2024-04-06T13:29:19.015798+0000","last_clean":"2024-04-06T13:29:19.015798+0000","last_became_active":"2024-04-06T13:29:16.887653+0000","last_became_peered":"2024-04-06T13:29:16.887653+0000","last_unstale":"2024-04-06T13:29:19.015798+0000","last_undegraded":"2024-04-06T13:29:19.015798+0000","last_fullsized":"2024-04-06T13:29:19.015798+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:03:22.503093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015765+0000","last_change":"2024-04-06T13:29:16.879536+0000","last_active":"2024-04-06T13:29:19.015765+0000","last_peered":"2024-04-06T13:29:19.015765+0000","last_clean":"2024-04-06T13:29:19.015765+0000","last_became_active":"2024-04-06T13:29:16.879275+0000","last_became_peered":"2024-04-06T13:29:16.879275+0000","last_unstale":"2024-04-06T13:29:19.015765+0000","last_undegraded":"2024-04-06T13:29:19.015765+0000","last_fullsized":"2024-04-06T13:29:19.015765+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:49:08.120568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.016009+0000","last_change":"2024-04-06T13:29:16.876576+0000","last_active":"2024-04-06T13:29:19.016009+0000","last_peered":"2024-04-06T13:29:19.016009+0000","last_clean":"2024-04-06T13:29:19.016009+0000","last_became_active":"2024-04-06T13:29:16.876306+0000","last_became_peered":"2024-04-06T13:29:16.876306+0000","last_unstale":"2024-04-06T13:29:19.016009+0000","last_undegraded":"2024-04-06T13:29:19.016009+0000","last_fullsized":"2024-04-06T13:29:19.016009+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:08:24.097613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869887+0000","last_change":"2024-04-06T13:29:16.868296+0000","last_active":"2024-04-06T13:29:17.869887+0000","last_peered":"2024-04-06T13:29:17.869887+0000","last_clean":"2024-04-06T13:29:17.869887+0000","last_became_active":"2024-04-06T13:29:16.867999+0000","last_became_peered":"2024-04-06T13:29:16.867999+0000","last_unstale":"2024-04-06T13:29:17.869887+0000","last_undegraded":"2024-04-06T13:29:17.869887+0000","last_fullsized":"2024-04-06T13:29:17.869887+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:47:29.405915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"191'4","reported_seq":15,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870176+0000","last_change":"2024-04-06T13:29:16.451701+0000","last_active":"2024-04-06T13:29:17.870176+0000","last_peered":"2024-04-06T13:29:17.870176+0000","last_clean":"2024-04-06T13:29:17.870176+0000","last_became_active":"2024-04-06T13:29:16.451385+0000","last_became_peered":"2024-04-06T13:29:16.451385+0000","last_unstale":"2024-04-06T13:29:17.870176+0000","last_undegraded":"2024-04-06T13:29:17.870176+0000","last_fullsized":"2024-04-06T13:29:17.870176+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:45:01.236287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869339+0000","last_change":"2024-04-06T13:29:16.876384+0000","last_active":"2024-04-06T13:29:17.869339+0000","last_peered":"2024-04-06T13:29:17.869339+0000","last_clean":"2024-04-06T13:29:17.869339+0000","last_became_active":"2024-04-06T13:29:16.876091+0000","last_became_peered":"2024-04-06T13:29:16.876091+0000","last_unstale":"2024-04-06T13:29:17.869339+0000","last_undegraded":"2024-04-06T13:29:17.869339+0000","last_fullsized":"2024-04-06T13:29:17.869339+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:29:57.354201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"191'1","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015766+0000","last_change":"2024-04-06T13:29:16.466716+0000","last_active":"2024-04-06T13:29:19.015766+0000","last_peered":"2024-04-06T13:29:19.015766+0000","last_clean":"2024-04-06T13:29:19.015766+0000","last_became_active":"2024-04-06T13:29:16.466462+0000","last_became_peered":"2024-04-06T13:29:16.466462+0000","last_unstale":"2024-04-06T13:29:19.015766+0000","last_undegraded":"2024-04-06T13:29:19.015766+0000","last_fullsized":"2024-04-06T13:29:19.015766+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:49:57.644107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015484+0000","last_change":"2024-04-06T13:29:16.889839+0000","last_active":"2024-04-06T13:29:19.015484+0000","last_peered":"2024-04-06T13:29:19.015484+0000","last_clean":"2024-04-06T13:29:19.015484+0000","last_became_active":"2024-04-06T13:29:16.889050+0000","last_became_peered":"2024-04-06T13:29:16.889050+0000","last_unstale":"2024-04-06T13:29:19.015484+0000","last_undegraded":"2024-04-06T13:29:19.015484+0000","last_fullsized":"2024-04-06T13:29:19.015484+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:51:13.835661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"191'2","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.016116+0000","last_change":"2024-04-06T13:29:16.463291+0000","last_active":"2024-04-06T13:29:19.016116+0000","last_peered":"2024-04-06T13:29:19.016116+0000","last_clean":"2024-04-06T13:29:19.016116+0000","last_became_active":"2024-04-06T13:29:16.462877+0000","last_became_peered":"2024-04-06T13:29:16.462877+0000","last_unstale":"2024-04-06T13:29:19.016116+0000","last_undegraded":"2024-04-06T13:29:19.016116+0000","last_fullsized":"2024-04-06T13:29:19.016116+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:45.753781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870718+0000","last_change":"2024-04-06T13:29:16.875052+0000","last_active":"2024-04-06T13:29:17.870718+0000","last_peered":"2024-04-06T13:29:17.870718+0000","last_clean":"2024-04-06T13:29:17.870718+0000","last_became_active":"2024-04-06T13:29:16.874779+0000","last_became_peered":"2024-04-06T13:29:16.874779+0000","last_unstale":"2024-04-06T13:29:17.870718+0000","last_undegraded":"2024-04-06T13:29:17.870718+0000","last_fullsized":"2024-04-06T13:29:17.870718+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:09:34.642057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"191'2","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.514989+0000","last_change":"2024-04-06T13:29:16.456231+0000","last_active":"2024-04-06T13:29:18.514989+0000","last_peered":"2024-04-06T13:29:18.514989+0000","last_clean":"2024-04-06T13:29:18.514989+0000","last_became_active":"2024-04-06T13:29:16.455951+0000","last_became_peered":"2024-04-06T13:29:16.455951+0000","last_unstale":"2024-04-06T13:29:18.514989+0000","last_undegraded":"2024-04-06T13:29:18.514989+0000","last_fullsized":"2024-04-06T13:29:18.514989+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:20:30.598041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031811+0000","last_change":"2024-04-06T13:29:16.885700+0000","last_active":"2024-04-06T13:29:18.031811+0000","last_peered":"2024-04-06T13:29:18.031811+0000","last_clean":"2024-04-06T13:29:18.031811+0000","last_became_active":"2024-04-06T13:29:16.885425+0000","last_became_peered":"2024-04-06T13:29:16.885425+0000","last_unstale":"2024-04-06T13:29:18.031811+0000","last_undegraded":"2024-04-06T13:29:18.031811+0000","last_fullsized":"2024-04-06T13:29:18.031811+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:44:50.624713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"191'1","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031794+0000","last_change":"2024-04-06T13:29:16.466892+0000","last_active":"2024-04-06T13:29:18.031794+0000","last_peered":"2024-04-06T13:29:18.031794+0000","last_clean":"2024-04-06T13:29:18.031794+0000","last_became_active":"2024-04-06T13:29:16.466630+0000","last_became_peered":"2024-04-06T13:29:16.466630+0000","last_unstale":"2024-04-06T13:29:18.031794+0000","last_undegraded":"2024-04-06T13:29:18.031794+0000","last_fullsized":"2024-04-06T13:29:18.031794+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:27:51.741781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.515085+0000","last_change":"2024-04-06T13:29:16.874261+0000","last_active":"2024-04-06T13:29:18.515085+0000","last_peered":"2024-04-06T13:29:18.515085+0000","last_clean":"2024-04-06T13:29:18.515085+0000","last_became_active":"2024-04-06T13:29:16.874024+0000","last_became_peered":"2024-04-06T13:29:16.874024+0000","last_unstale":"2024-04-06T13:29:18.515085+0000","last_undegraded":"2024-04-06T13:29:18.515085+0000","last_fullsized":"2024-04-06T13:29:18.515085+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:40:26.762806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"191'1","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015576+0000","last_change":"2024-04-06T13:29:16.465100+0000","last_active":"2024-04-06T13:29:19.015576+0000","last_peered":"2024-04-06T13:29:19.015576+0000","last_clean":"2024-04-06T13:29:19.015576+0000","last_became_active":"2024-04-06T13:29:16.464833+0000","last_became_peered":"2024-04-06T13:29:16.464833+0000","last_unstale":"2024-04-06T13:29:19.015576+0000","last_undegraded":"2024-04-06T13:29:19.015576+0000","last_fullsized":"2024-04-06T13:29:19.015576+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:26:29.186700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.515103+0000","last_change":"2024-04-06T13:29:16.875862+0000","last_active":"2024-04-06T13:29:18.515103+0000","last_peered":"2024-04-06T13:29:18.515103+0000","last_clean":"2024-04-06T13:29:18.515103+0000","last_became_active":"2024-04-06T13:29:16.875600+0000","last_became_peered":"2024-04-06T13:29:16.875600+0000","last_unstale":"2024-04-06T13:29:18.515103+0000","last_undegraded":"2024-04-06T13:29:18.515103+0000","last_fullsized":"2024-04-06T13:29:18.515103+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:46.759410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.514867+0000","last_change":"2024-04-06T13:29:16.464039+0000","last_active":"2024-04-06T13:29:18.514867+0000","last_peered":"2024-04-06T13:29:18.514867+0000","last_clean":"2024-04-06T13:29:18.514867+0000","last_became_active":"2024-04-06T13:29:16.463345+0000","last_became_peered":"2024-04-06T13:29:16.463345+0000","last_unstale":"2024-04-06T13:29:18.514867+0000","last_undegraded":"2024-04-06T13:29:18.514867+0000","last_fullsized":"2024-04-06T13:29:18.514867+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:33:20.459267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870110+0000","last_change":"2024-04-06T13:29:16.887269+0000","last_active":"2024-04-06T13:29:17.870110+0000","last_peered":"2024-04-06T13:29:17.870110+0000","last_clean":"2024-04-06T13:29:17.870110+0000","last_became_active":"2024-04-06T13:29:16.886536+0000","last_became_peered":"2024-04-06T13:29:16.886536+0000","last_unstale":"2024-04-06T13:29:17.870110+0000","last_undegraded":"2024-04-06T13:29:17.870110+0000","last_fullsized":"2024-04-06T13:29:17.870110+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:03:29.165222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031619+0000","last_change":"2024-04-06T13:29:16.466241+0000","last_active":"2024-04-06T13:29:18.031619+0000","last_peered":"2024-04-06T13:29:18.031619+0000","last_clean":"2024-04-06T13:29:18.031619+0000","last_became_active":"2024-04-06T13:29:16.465829+0000","last_became_peered":"2024-04-06T13:29:16.465829+0000","last_unstale":"2024-04-06T13:29:18.031619+0000","last_undegraded":"2024-04-06T13:29:18.031619+0000","last_fullsized":"2024-04-06T13:29:18.031619+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:16:21.330544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870066+0000","last_change":"2024-04-06T13:29:16.870112+0000","last_active":"2024-04-06T13:29:17.870066+0000","last_peered":"2024-04-06T13:29:17.870066+0000","last_clean":"2024-04-06T13:29:17.870066+0000","last_became_active":"2024-04-06T13:29:16.869840+0000","last_became_peered":"2024-04-06T13:29:16.869840+0000","last_unstale":"2024-04-06T13:29:17.870066+0000","last_undegraded":"2024-04-06T13:29:17.870066+0000","last_fullsized":"2024-04-06T13:29:17.870066+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:33:43.850720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869778+0000","last_change":"2024-04-06T13:29:16.468871+0000","last_active":"2024-04-06T13:29:17.869778+0000","last_peered":"2024-04-06T13:29:17.869778+0000","last_clean":"2024-04-06T13:29:17.869778+0000","last_became_active":"2024-04-06T13:29:16.468455+0000","last_became_peered":"2024-04-06T13:29:16.468455+0000","last_unstale":"2024-04-06T13:29:17.869778+0000","last_undegraded":"2024-04-06T13:29:17.869778+0000","last_fullsized":"2024-04-06T13:29:17.869778+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:53:38.647912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869479+0000","last_change":"2024-04-06T13:29:16.878934+0000","last_active":"2024-04-06T13:29:17.869479+0000","last_peered":"2024-04-06T13:29:17.869479+0000","last_clean":"2024-04-06T13:29:17.869479+0000","last_became_active":"2024-04-06T13:29:16.878688+0000","last_became_peered":"2024-04-06T13:29:16.878688+0000","last_unstale":"2024-04-06T13:29:17.869479+0000","last_undegraded":"2024-04-06T13:29:17.869479+0000","last_fullsized":"2024-04-06T13:29:17.869479+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:34:29.517268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"191'3","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869910+0000","last_change":"2024-04-06T13:29:16.457567+0000","last_active":"2024-04-06T13:29:17.869910+0000","last_peered":"2024-04-06T13:29:17.869910+0000","last_clean":"2024-04-06T13:29:17.869910+0000","last_became_active":"2024-04-06T13:29:16.457261+0000","last_became_peered":"2024-04-06T13:29:16.457261+0000","last_unstale":"2024-04-06T13:29:17.869910+0000","last_undegraded":"2024-04-06T13:29:17.869910+0000","last_fullsized":"2024-04-06T13:29:17.869910+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:54:16.663929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.515071+0000","last_change":"2024-04-06T13:29:16.877435+0000","last_active":"2024-04-06T13:29:18.515071+0000","last_peered":"2024-04-06T13:29:18.515071+0000","last_clean":"2024-04-06T13:29:18.515071+0000","last_became_active":"2024-04-06T13:29:16.877233+0000","last_became_peered":"2024-04-06T13:29:16.877233+0000","last_unstale":"2024-04-06T13:29:18.515071+0000","last_undegraded":"2024-04-06T13:29:18.515071+0000","last_fullsized":"2024-04-06T13:29:18.515071+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:03:22.503093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"191'6","reported_seq":17,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869585+0000","last_change":"2024-04-06T13:29:16.463484+0000","last_active":"2024-04-06T13:29:17.869585+0000","last_peered":"2024-04-06T13:29:17.869585+0000","last_clean":"2024-04-06T13:29:17.869585+0000","last_became_active":"2024-04-06T13:29:16.463181+0000","last_became_peered":"2024-04-06T13:29:16.463181+0000","last_unstale":"2024-04-06T13:29:17.869585+0000","last_undegraded":"2024-04-06T13:29:17.869585+0000","last_fullsized":"2024-04-06T13:29:17.869585+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:31:50.119403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015989+0000","last_change":"2024-04-06T13:29:16.875897+0000","last_active":"2024-04-06T13:29:19.015989+0000","last_peered":"2024-04-06T13:29:19.015989+0000","last_clean":"2024-04-06T13:29:19.015989+0000","last_became_active":"2024-04-06T13:29:16.875619+0000","last_became_peered":"2024-04-06T13:29:16.875619+0000","last_unstale":"2024-04-06T13:29:19.015989+0000","last_undegraded":"2024-04-06T13:29:19.015989+0000","last_fullsized":"2024-04-06T13:29:19.015989+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:46.759410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"191'2","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015626+0000","last_change":"2024-04-06T13:29:16.463527+0000","last_active":"2024-04-06T13:29:19.015626+0000","last_peered":"2024-04-06T13:29:19.015626+0000","last_clean":"2024-04-06T13:29:19.015626+0000","last_became_active":"2024-04-06T13:29:16.463231+0000","last_became_peered":"2024-04-06T13:29:16.463231+0000","last_unstale":"2024-04-06T13:29:19.015626+0000","last_undegraded":"2024-04-06T13:29:19.015626+0000","last_fullsized":"2024-04-06T13:29:19.015626+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:24:10.368926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031979+0000","last_change":"2024-04-06T13:29:16.884167+0000","last_active":"2024-04-06T13:29:18.031979+0000","last_peered":"2024-04-06T13:29:18.031979+0000","last_clean":"2024-04-06T13:29:18.031979+0000","last_became_active":"2024-04-06T13:29:16.883539+0000","last_became_peered":"2024-04-06T13:29:16.883539+0000","last_unstale":"2024-04-06T13:29:18.031979+0000","last_undegraded":"2024-04-06T13:29:18.031979+0000","last_fullsized":"2024-04-06T13:29:18.031979+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:31:46.993248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869445+0000","last_change":"2024-04-06T13:29:16.455567+0000","last_active":"2024-04-06T13:29:17.869445+0000","last_peered":"2024-04-06T13:29:17.869445+0000","last_clean":"2024-04-06T13:29:17.869445+0000","last_became_active":"2024-04-06T13:29:16.455280+0000","last_became_peered":"2024-04-06T13:29:16.455280+0000","last_unstale":"2024-04-06T13:29:17.869445+0000","last_undegraded":"2024-04-06T13:29:17.869445+0000","last_fullsized":"2024-04-06T13:29:17.869445+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:40:50.376953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015609+0000","last_change":"2024-04-06T13:29:16.877580+0000","last_active":"2024-04-06T13:29:19.015609+0000","last_peered":"2024-04-06T13:29:19.015609+0000","last_clean":"2024-04-06T13:29:19.015609+0000","last_became_active":"2024-04-06T13:29:16.877327+0000","last_became_peered":"2024-04-06T13:29:16.877327+0000","last_unstale":"2024-04-06T13:29:19.015609+0000","last_undegraded":"2024-04-06T13:29:19.015609+0000","last_fullsized":"2024-04-06T13:29:19.015609+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:17:44.072132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015631+0000","last_change":"2024-04-06T13:29:16.467262+0000","last_active":"2024-04-06T13:29:19.015631+0000","last_peered":"2024-04-06T13:29:19.015631+0000","last_clean":"2024-04-06T13:29:19.015631+0000","last_became_active":"2024-04-06T13:29:16.467036+0000","last_became_peered":"2024-04-06T13:29:16.467036+0000","last_unstale":"2024-04-06T13:29:19.015631+0000","last_undegraded":"2024-04-06T13:29:19.015631+0000","last_fullsized":"2024-04-06T13:29:19.015631+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:18:43.910720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.515126+0000","last_change":"2024-04-06T13:29:16.892789+0000","last_active":"2024-04-06T13:29:18.515126+0000","last_peered":"2024-04-06T13:29:18.515126+0000","last_clean":"2024-04-06T13:29:18.515126+0000","last_became_active":"2024-04-06T13:29:16.892524+0000","last_became_peered":"2024-04-06T13:29:16.892524+0000","last_unstale":"2024-04-06T13:29:18.515126+0000","last_undegraded":"2024-04-06T13:29:18.515126+0000","last_fullsized":"2024-04-06T13:29:18.515126+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:45:18.965004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"191'8","reported_seq":19,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031716+0000","last_change":"2024-04-06T13:29:16.466074+0000","last_active":"2024-04-06T13:29:18.031716+0000","last_peered":"2024-04-06T13:29:18.031716+0000","last_clean":"2024-04-06T13:29:18.031716+0000","last_became_active":"2024-04-06T13:29:16.465473+0000","last_became_peered":"2024-04-06T13:29:16.465473+0000","last_unstale":"2024-04-06T13:29:18.031716+0000","last_undegraded":"2024-04-06T13:29:18.031716+0000","last_fullsized":"2024-04-06T13:29:18.031716+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:00:46.031669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031733+0000","last_change":"2024-04-06T13:29:16.886226+0000","last_active":"2024-04-06T13:29:18.031733+0000","last_peered":"2024-04-06T13:29:18.031733+0000","last_clean":"2024-04-06T13:29:18.031733+0000","last_became_active":"2024-04-06T13:29:16.885464+0000","last_became_peered":"2024-04-06T13:29:16.885464+0000","last_unstale":"2024-04-06T13:29:18.031733+0000","last_undegraded":"2024-04-06T13:29:18.031733+0000","last_fullsized":"2024-04-06T13:29:18.031733+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:40:59.094743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"191'1","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869840+0000","last_change":"2024-04-06T13:29:16.457609+0000","last_active":"2024-04-06T13:29:17.869840+0000","last_peered":"2024-04-06T13:29:17.869840+0000","last_clean":"2024-04-06T13:29:17.869840+0000","last_became_active":"2024-04-06T13:29:16.457227+0000","last_became_peered":"2024-04-06T13:29:16.457227+0000","last_unstale":"2024-04-06T13:29:17.869840+0000","last_undegraded":"2024-04-06T13:29:17.869840+0000","last_fullsized":"2024-04-06T13:29:17.869840+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:39:30.175672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015942+0000","last_change":"2024-04-06T13:29:16.886090+0000","last_active":"2024-04-06T13:29:19.015942+0000","last_peered":"2024-04-06T13:29:19.015942+0000","last_clean":"2024-04-06T13:29:19.015942+0000","last_became_active":"2024-04-06T13:29:16.885825+0000","last_became_peered":"2024-04-06T13:29:16.885825+0000","last_unstale":"2024-04-06T13:29:19.015942+0000","last_undegraded":"2024-04-06T13:29:19.015942+0000","last_fullsized":"2024-04-06T13:29:19.015942+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:18:44.916349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"191'2","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031967+0000","last_change":"2024-04-06T13:29:16.465514+0000","last_active":"2024-04-06T13:29:18.031967+0000","last_peered":"2024-04-06T13:29:18.031967+0000","last_clean":"2024-04-06T13:29:18.031967+0000","last_became_active":"2024-04-06T13:29:16.462836+0000","last_became_peered":"2024-04-06T13:29:16.462836+0000","last_unstale":"2024-04-06T13:29:18.031967+0000","last_undegraded":"2024-04-06T13:29:18.031967+0000","last_fullsized":"2024-04-06T13:29:18.031967+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:25:06.167257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.016200+0000","last_change":"2024-04-06T13:29:16.887037+0000","last_active":"2024-04-06T13:29:19.016200+0000","last_peered":"2024-04-06T13:29:19.016200+0000","last_clean":"2024-04-06T13:29:19.016200+0000","last_became_active":"2024-04-06T13:29:16.886767+0000","last_became_peered":"2024-04-06T13:29:16.886767+0000","last_unstale":"2024-04-06T13:29:19.016200+0000","last_undegraded":"2024-04-06T13:29:19.016200+0000","last_fullsized":"2024-04-06T13:29:19.016200+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:40:41.556933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"191'1","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015900+0000","last_change":"2024-04-06T13:29:16.460961+0000","last_active":"2024-04-06T13:29:19.015900+0000","last_peered":"2024-04-06T13:29:19.015900+0000","last_clean":"2024-04-06T13:29:19.015900+0000","last_became_active":"2024-04-06T13:29:16.460609+0000","last_became_peered":"2024-04-06T13:29:16.460609+0000","last_unstale":"2024-04-06T13:29:19.015900+0000","last_undegraded":"2024-04-06T13:29:19.015900+0000","last_fullsized":"2024-04-06T13:29:19.015900+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:04:44.686319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015661+0000","last_change":"2024-04-06T13:29:16.880697+0000","last_active":"2024-04-06T13:29:19.015661+0000","last_peered":"2024-04-06T13:29:19.015661+0000","last_clean":"2024-04-06T13:29:19.015661+0000","last_became_active":"2024-04-06T13:29:16.880383+0000","last_became_peered":"2024-04-06T13:29:16.880383+0000","last_unstale":"2024-04-06T13:29:19.015661+0000","last_undegraded":"2024-04-06T13:29:19.015661+0000","last_fullsized":"2024-04-06T13:29:19.015661+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:54:17.669558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031900+0000","last_change":"2024-04-06T13:29:16.466282+0000","last_active":"2024-04-06T13:29:18.031900+0000","last_peered":"2024-04-06T13:29:18.031900+0000","last_clean":"2024-04-06T13:29:18.031900+0000","last_became_active":"2024-04-06T13:29:16.466042+0000","last_became_peered":"2024-04-06T13:29:16.466042+0000","last_unstale":"2024-04-06T13:29:18.031900+0000","last_undegraded":"2024-04-06T13:29:18.031900+0000","last_fullsized":"2024-04-06T13:29:18.031900+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:22:01.776384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869538+0000","last_change":"2024-04-06T13:29:16.881141+0000","last_active":"2024-04-06T13:29:17.869538+0000","last_peered":"2024-04-06T13:29:17.869538+0000","last_clean":"2024-04-06T13:29:17.869538+0000","last_became_active":"2024-04-06T13:29:16.880905+0000","last_became_peered":"2024-04-06T13:29:16.880905+0000","last_unstale":"2024-04-06T13:29:17.869538+0000","last_undegraded":"2024-04-06T13:29:17.869538+0000","last_fullsized":"2024-04-06T13:29:17.869538+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:58:28.333504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"191'2","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015651+0000","last_change":"2024-04-06T13:29:16.460100+0000","last_active":"2024-04-06T13:29:19.015651+0000","last_peered":"2024-04-06T13:29:19.015651+0000","last_clean":"2024-04-06T13:29:19.015651+0000","last_became_active":"2024-04-06T13:29:16.459855+0000","last_became_peered":"2024-04-06T13:29:16.459855+0000","last_unstale":"2024-04-06T13:29:19.015651+0000","last_undegraded":"2024-04-06T13:29:19.015651+0000","last_fullsized":"2024-04-06T13:29:19.015651+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:22:23.627784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015831+0000","last_change":"2024-04-06T13:29:16.890376+0000","last_active":"2024-04-06T13:29:19.015831+0000","last_peered":"2024-04-06T13:29:19.015831+0000","last_clean":"2024-04-06T13:29:19.015831+0000","last_became_active":"2024-04-06T13:29:16.890117+0000","last_became_peered":"2024-04-06T13:29:16.890117+0000","last_unstale":"2024-04-06T13:29:19.015831+0000","last_undegraded":"2024-04-06T13:29:19.015831+0000","last_fullsized":"2024-04-06T13:29:19.015831+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:39:49.010777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.016173+0000","last_change":"2024-04-06T13:29:16.457256+0000","last_active":"2024-04-06T13:29:19.016173+0000","last_peered":"2024-04-06T13:29:19.016173+0000","last_clean":"2024-04-06T13:29:19.016173+0000","last_became_active":"2024-04-06T13:29:16.457050+0000","last_became_peered":"2024-04-06T13:29:16.457050+0000","last_unstale":"2024-04-06T13:29:19.016173+0000","last_undegraded":"2024-04-06T13:29:19.016173+0000","last_fullsized":"2024-04-06T13:29:19.016173+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:50:15.874342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015746+0000","last_change":"2024-04-06T13:29:16.892221+0000","last_active":"2024-04-06T13:29:19.015746+0000","last_peered":"2024-04-06T13:29:19.015746+0000","last_clean":"2024-04-06T13:29:19.015746+0000","last_became_active":"2024-04-06T13:29:16.891987+0000","last_became_peered":"2024-04-06T13:29:16.891987+0000","last_unstale":"2024-04-06T13:29:19.015746+0000","last_undegraded":"2024-04-06T13:29:19.015746+0000","last_fullsized":"2024-04-06T13:29:19.015746+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:22:33.381500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.514961+0000","last_change":"2024-04-06T13:29:16.459264+0000","last_active":"2024-04-06T13:29:18.514961+0000","last_peered":"2024-04-06T13:29:18.514961+0000","last_clean":"2024-04-06T13:29:18.514961+0000","last_became_active":"2024-04-06T13:29:16.458974+0000","last_became_peered":"2024-04-06T13:29:16.458974+0000","last_unstale":"2024-04-06T13:29:18.514961+0000","last_undegraded":"2024-04-06T13:29:18.514961+0000","last_fullsized":"2024-04-06T13:29:18.514961+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:02:10.950359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870850+0000","last_change":"2024-04-06T13:29:16.871003+0000","last_active":"2024-04-06T13:29:17.870850+0000","last_peered":"2024-04-06T13:29:17.870850+0000","last_clean":"2024-04-06T13:29:17.870850+0000","last_became_active":"2024-04-06T13:29:16.870709+0000","last_became_peered":"2024-04-06T13:29:16.870709+0000","last_unstale":"2024-04-06T13:29:17.870850+0000","last_undegraded":"2024-04-06T13:29:17.870850+0000","last_fullsized":"2024-04-06T13:29:17.870850+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:29:16.732050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869973+0000","last_change":"2024-04-06T13:29:16.455130+0000","last_active":"2024-04-06T13:29:17.869973+0000","last_peered":"2024-04-06T13:29:17.869973+0000","last_clean":"2024-04-06T13:29:17.869973+0000","last_became_active":"2024-04-06T13:29:16.454892+0000","last_became_peered":"2024-04-06T13:29:16.454892+0000","last_unstale":"2024-04-06T13:29:17.869973+0000","last_undegraded":"2024-04-06T13:29:17.869973+0000","last_fullsized":"2024-04-06T13:29:17.869973+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:31.726957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015502+0000","last_change":"2024-04-06T13:29:16.874944+0000","last_active":"2024-04-06T13:29:19.015502+0000","last_peered":"2024-04-06T13:29:19.015502+0000","last_clean":"2024-04-06T13:29:19.015502+0000","last_became_active":"2024-04-06T13:29:16.874618+0000","last_became_peered":"2024-04-06T13:29:16.874618+0000","last_unstale":"2024-04-06T13:29:19.015502+0000","last_undegraded":"2024-04-06T13:29:19.015502+0000","last_fullsized":"2024-04-06T13:29:19.015502+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:28:59.228532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"191'2","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015860+0000","last_change":"2024-04-06T13:29:16.461497+0000","last_active":"2024-04-06T13:29:19.015860+0000","last_peered":"2024-04-06T13:29:19.015860+0000","last_clean":"2024-04-06T13:29:19.015860+0000","last_became_active":"2024-04-06T13:29:16.460573+0000","last_became_peered":"2024-04-06T13:29:16.460573+0000","last_unstale":"2024-04-06T13:29:19.015860+0000","last_undegraded":"2024-04-06T13:29:19.015860+0000","last_fullsized":"2024-04-06T13:29:19.015860+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:39:40.540329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031748+0000","last_change":"2024-04-06T13:29:16.883616+0000","last_active":"2024-04-06T13:29:18.031748+0000","last_peered":"2024-04-06T13:29:18.031748+0000","last_clean":"2024-04-06T13:29:18.031748+0000","last_became_active":"2024-04-06T13:29:16.883357+0000","last_became_peered":"2024-04-06T13:29:16.883357+0000","last_unstale":"2024-04-06T13:29:18.031748+0000","last_undegraded":"2024-04-06T13:29:18.031748+0000","last_fullsized":"2024-04-06T13:29:18.031748+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:39:21.730370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869753+0000","last_change":"2024-04-06T13:29:16.469434+0000","last_active":"2024-04-06T13:29:17.869753+0000","last_peered":"2024-04-06T13:29:17.869753+0000","last_clean":"2024-04-06T13:29:17.869753+0000","last_became_active":"2024-04-06T13:29:16.468793+0000","last_became_peered":"2024-04-06T13:29:16.468793+0000","last_unstale":"2024-04-06T13:29:17.869753+0000","last_undegraded":"2024-04-06T13:29:17.869753+0000","last_fullsized":"2024-04-06T13:29:17.869753+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:30:45.086967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.514875+0000","last_change":"2024-04-06T13:29:16.873996+0000","last_active":"2024-04-06T13:29:18.514875+0000","last_peered":"2024-04-06T13:29:18.514875+0000","last_clean":"2024-04-06T13:29:18.514875+0000","last_became_active":"2024-04-06T13:29:16.873773+0000","last_became_peered":"2024-04-06T13:29:16.873773+0000","last_unstale":"2024-04-06T13:29:18.514875+0000","last_undegraded":"2024-04-06T13:29:18.514875+0000","last_fullsized":"2024-04-06T13:29:18.514875+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:20:07.574362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869666+0000","last_change":"2024-04-06T13:29:16.456790+0000","last_active":"2024-04-06T13:29:17.869666+0000","last_peered":"2024-04-06T13:29:17.869666+0000","last_clean":"2024-04-06T13:29:17.869666+0000","last_became_active":"2024-04-06T13:29:16.456544+0000","last_became_peered":"2024-04-06T13:29:16.456544+0000","last_unstale":"2024-04-06T13:29:17.869666+0000","last_undegraded":"2024-04-06T13:29:17.869666+0000","last_fullsized":"2024-04-06T13:29:17.869666+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:42:08.635751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031715+0000","last_change":"2024-04-06T13:29:16.884104+0000","last_active":"2024-04-06T13:29:18.031715+0000","last_peered":"2024-04-06T13:29:18.031715+0000","last_clean":"2024-04-06T13:29:18.031715+0000","last_became_active":"2024-04-06T13:29:16.883871+0000","last_became_peered":"2024-04-06T13:29:16.883871+0000","last_unstale":"2024-04-06T13:29:18.031715+0000","last_undegraded":"2024-04-06T13:29:18.031715+0000","last_fullsized":"2024-04-06T13:29:18.031715+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:43:19.361499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.514984+0000","last_change":"2024-04-06T13:29:16.463035+0000","last_active":"2024-04-06T13:29:18.514984+0000","last_peered":"2024-04-06T13:29:18.514984+0000","last_clean":"2024-04-06T13:29:18.514984+0000","last_became_active":"2024-04-06T13:29:16.462760+0000","last_became_peered":"2024-04-06T13:29:16.462760+0000","last_unstale":"2024-04-06T13:29:18.514984+0000","last_undegraded":"2024-04-06T13:29:18.514984+0000","last_fullsized":"2024-04-06T13:29:18.514984+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:35:44.802474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869754+0000","last_change":"2024-04-06T13:29:16.888214+0000","last_active":"2024-04-06T13:29:17.869754+0000","last_peered":"2024-04-06T13:29:17.869754+0000","last_clean":"2024-04-06T13:29:17.869754+0000","last_became_active":"2024-04-06T13:29:16.887759+0000","last_became_peered":"2024-04-06T13:29:16.887759+0000","last_unstale":"2024-04-06T13:29:17.869754+0000","last_undegraded":"2024-04-06T13:29:17.869754+0000","last_fullsized":"2024-04-06T13:29:17.869754+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:17:42.492434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869875+0000","last_change":"2024-04-06T13:29:16.465013+0000","last_active":"2024-04-06T13:29:17.869875+0000","last_peered":"2024-04-06T13:29:17.869875+0000","last_clean":"2024-04-06T13:29:17.869875+0000","last_became_active":"2024-04-06T13:29:16.464805+0000","last_became_peered":"2024-04-06T13:29:16.464805+0000","last_unstale":"2024-04-06T13:29:17.869875+0000","last_undegraded":"2024-04-06T13:29:17.869875+0000","last_fullsized":"2024-04-06T13:29:17.869875+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:25:06.167257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870795+0000","last_change":"2024-04-06T13:29:16.875804+0000","last_active":"2024-04-06T13:29:17.870795+0000","last_peered":"2024-04-06T13:29:17.870795+0000","last_clean":"2024-04-06T13:29:17.870795+0000","last_became_active":"2024-04-06T13:29:16.874222+0000","last_became_peered":"2024-04-06T13:29:16.874222+0000","last_unstale":"2024-04-06T13:29:17.870795+0000","last_undegraded":"2024-04-06T13:29:17.870795+0000","last_fullsized":"2024-04-06T13:29:17.870795+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:45:02.533582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869532+0000","last_change":"2024-04-06T13:29:16.459589+0000","last_active":"2024-04-06T13:29:17.869532+0000","last_peered":"2024-04-06T13:29:17.869532+0000","last_clean":"2024-04-06T13:29:17.869532+0000","last_became_active":"2024-04-06T13:29:16.459340+0000","last_became_peered":"2024-04-06T13:29:16.459340+0000","last_unstale":"2024-04-06T13:29:17.869532+0000","last_undegraded":"2024-04-06T13:29:17.869532+0000","last_fullsized":"2024-04-06T13:29:17.869532+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:51:12.830032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869678+0000","last_change":"2024-04-06T13:29:16.878926+0000","last_active":"2024-04-06T13:29:17.869678+0000","last_peered":"2024-04-06T13:29:17.869678+0000","last_clean":"2024-04-06T13:29:17.869678+0000","last_became_active":"2024-04-06T13:29:16.878659+0000","last_became_peered":"2024-04-06T13:29:16.878659+0000","last_unstale":"2024-04-06T13:29:17.869678+0000","last_undegraded":"2024-04-06T13:29:17.869678+0000","last_fullsized":"2024-04-06T13:29:17.869678+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:57:16.557906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015858+0000","last_change":"2024-04-06T13:29:16.460240+0000","last_active":"2024-04-06T13:29:19.015858+0000","last_peered":"2024-04-06T13:29:19.015858+0000","last_clean":"2024-04-06T13:29:19.015858+0000","last_became_active":"2024-04-06T13:29:16.459981+0000","last_became_peered":"2024-04-06T13:29:16.459981+0000","last_unstale":"2024-04-06T13:29:19.015858+0000","last_undegraded":"2024-04-06T13:29:19.015858+0000","last_fullsized":"2024-04-06T13:29:19.015858+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:08:23.091984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.515169+0000","last_change":"2024-04-06T13:29:16.876422+0000","last_active":"2024-04-06T13:29:18.515169+0000","last_peered":"2024-04-06T13:29:18.515169+0000","last_clean":"2024-04-06T13:29:18.515169+0000","last_became_active":"2024-04-06T13:29:16.876144+0000","last_became_peered":"2024-04-06T13:29:16.876144+0000","last_unstale":"2024-04-06T13:29:18.515169+0000","last_undegraded":"2024-04-06T13:29:18.515169+0000","last_fullsized":"2024-04-06T13:29:18.515169+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:47:10.159438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015347+0000","last_change":"2024-04-06T13:29:16.461121+0000","last_active":"2024-04-06T13:29:19.015347+0000","last_peered":"2024-04-06T13:29:19.015347+0000","last_clean":"2024-04-06T13:29:19.015347+0000","last_became_active":"2024-04-06T13:29:16.460542+0000","last_became_peered":"2024-04-06T13:29:16.460542+0000","last_unstale":"2024-04-06T13:29:19.015347+0000","last_undegraded":"2024-04-06T13:29:19.015347+0000","last_fullsized":"2024-04-06T13:29:19.015347+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:33:57.575734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869473+0000","last_change":"2024-04-06T13:29:16.875390+0000","last_active":"2024-04-06T13:29:17.869473+0000","last_peered":"2024-04-06T13:29:17.869473+0000","last_clean":"2024-04-06T13:29:17.869473+0000","last_became_active":"2024-04-06T13:29:16.875010+0000","last_became_peered":"2024-04-06T13:29:16.875010+0000","last_unstale":"2024-04-06T13:29:17.869473+0000","last_undegraded":"2024-04-06T13:29:17.869473+0000","last_fullsized":"2024-04-06T13:29:17.869473+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:44:33.412863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869691+0000","last_change":"2024-04-06T13:29:16.469239+0000","last_active":"2024-04-06T13:29:17.869691+0000","last_peered":"2024-04-06T13:29:17.869691+0000","last_clean":"2024-04-06T13:29:17.869691+0000","last_became_active":"2024-04-06T13:29:16.468944+0000","last_became_peered":"2024-04-06T13:29:16.468944+0000","last_unstale":"2024-04-06T13:29:17.869691+0000","last_undegraded":"2024-04-06T13:29:17.869691+0000","last_fullsized":"2024-04-06T13:29:17.869691+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:59.794032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"181'293","reported_seq":998,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-04-06T13:29:16.864144+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:29:16.864144+0000","last_peered":"2024-04-06T13:29:16.864144+0000","last_clean":"2024-04-06T13:29:16.864144+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:29:16.864144+0000","last_undegraded":"2024-04-06T13:29:16.864144+0000","last_fullsized":"2024-04-06T13:29:16.864144+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575769,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52100,"kb_used_data":2928,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319740,"statfs":{"total":96636764160,"available":96583413760,"internally_reserved":0,"allocated":2998272,"data_stored":1806605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.58799999999999997,"15min":0.59299999999999997},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.78200000000000003,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59599999999999997,"15min":0.60099999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.35899999999999999},"max":{"1min":0.78500000000000003,"5min":0.86099999999999999,"15min":1.4790000000000001},"last":0.48399999999999999}]},{"osd":1,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.60799999999999998,"15min":0.60999999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.79500000000000004,"5min":0.82499999999999996,"15min":0.83299999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.40300000000000002,"5min":0.39300000000000002,"15min":0.39000000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":1.5249999999999999},"last":0.67600000000000005}]},{"osd":2,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59099999999999997,"15min":0.58199999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59699999999999998,"15min":0.61099999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":1.198},"last":0.61099999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.60699999999999998,"15min":0.60799999999999998},"min":{"1min":0.437,"5min":0.36499999999999999,"15min":0.35499999999999998},"max":{"1min":0.77500000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61599999999999999,"15min":0.61199999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.78600000000000003,"5min":0.79600000000000004,"15min":1.1200000000000001},"last":0.66000000000000003}]},{"osd":4,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45400000000000001,"15min":0.45200000000000001},"min":{"1min":0.29599999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.97399999999999998},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55800000000000005,"15min":0.54000000000000004},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.33600000000000002},"max":{"1min":0.72399999999999998,"5min":0.77600000000000002,"15min":0.78000000000000003},"last":0.64800000000000002}]},{"osd":5,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.52700000000000002,"15min":0.51500000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69599999999999995,"5min":0.80000000000000004,"15min":1.024},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.58499999999999996,"15min":0.57999999999999996},"min":{"1min":0.36899999999999999,"5min":0.35599999999999998,"15min":0.33000000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":1.046},"last":0.83499999999999996}]},{"osd":6,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56699999999999995,"15min":0.56100000000000005},"min":{"1min":0.41399999999999998,"5min":0.38800000000000001,"15min":0.35299999999999998},"max":{"1min":0.81100000000000005,"5min":0.82199999999999995,"15min":1.4159999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.59599999999999997,"15min":0.57899999999999996},"min":{"1min":0.35799999999999998,"5min":0.34599999999999997,"15min":0.33700000000000002},"max":{"1min":0.75600000000000001,"5min":0.80700000000000005,"15min":0.873},"last":0.70199999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575768,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37552,"kb_used_data":2332,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334288,"statfs":{"total":96636764160,"available":96598310912,"internally_reserved":0,"allocated":2387968,"data_stored":1208554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.55700000000000005,"15min":0.56000000000000005},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.65900000000000003,"5min":0.79700000000000004,"15min":0.83499999999999996},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.75800000000000001,"5min":0.76200000000000001,"15min":0.78800000000000003},"last":0.54100000000000004}]},{"osd":1,"last update":"Sat Apr 6 13:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.60299999999999998,"15min":0.61799999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.628,"15min":0.63700000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.311},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.505}]},{"osd":2,"last update":"Sat Apr 6 13:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.58199999999999996,"15min":0.58999999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.748,"5min":0.80700000000000005,"15min":0.84499999999999997},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.58399999999999996,"15min":0.59299999999999997},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.56100000000000005}]},{"osd":3,"last update":"Sat Apr 6 13:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.60999999999999999,"15min":0.61199999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79200000000000004,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.64000000000000001,"15min":0.65400000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.74199999999999999,"5min":0.83999999999999997,"15min":0.877},"last":0.57599999999999996}]},{"osd":4,"last update":"Sat Apr 6 13:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51800000000000002,"15min":0.52100000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.81000000000000005},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53600000000000003,"15min":0.54700000000000004},"min":{"1min":0.33100000000000002,"5min":0.29499999999999998,"15min":0.29299999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.873},"last":0.39000000000000001}]},{"osd":5,"last update":"Sat Apr 6 13:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.505,"15min":0.51600000000000001},"min":{"1min":0.34799999999999998,"5min":0.32600000000000001,"15min":0.314},"max":{"1min":0.78600000000000003,"5min":0.79200000000000004,"15min":0.82299999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52100000000000002,"15min":0.53500000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.42399999999999999}]},{"osd":7,"last update":"Sat Apr 6 13:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55200000000000005,"15min":0.56000000000000005},"min":{"1min":0.37,"5min":0.35299999999999998,"15min":0.34899999999999998},"max":{"1min":0.81399999999999995,"5min":0.83499999999999996,"15min":0.85599999999999998},"last":0.499},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58299999999999996,"15min":0.58999999999999997},"min":{"1min":0.41799999999999998,"5min":0.375,"15min":0.36499999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.60399999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575767,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37568,"kb_used_data":2356,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334272,"statfs":{"total":96636764160,"available":96598294528,"internally_reserved":0,"allocated":2412544,"data_stored":1216782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.39900000000000002,"15min":0.38800000000000001},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.50900000000000001,"5min":0.54800000000000004,"15min":0.64300000000000002},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.434,"15min":0.42899999999999999},"min":{"1min":0.28799999999999998,"5min":0.27000000000000002,"15min":0.26700000000000002},"max":{"1min":0.56100000000000005,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.505}]},{"osd":2,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.42999999999999999,"15min":0.42499999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.52400000000000002,"5min":0.64000000000000001,"15min":0.67400000000000004},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.47699999999999998,"15min":0.46500000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59399999999999997,"5min":0.70499999999999996,"15min":0.73299999999999998},"last":0.55000000000000004}]},{"osd":3,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.48499999999999999,"15min":0.47299999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.30599999999999999},"max":{"1min":0.64200000000000002,"5min":0.70899999999999996,"15min":0.77100000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.45600000000000002,"15min":0.44700000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.63200000000000001,"5min":0.65500000000000003,"15min":0.68000000000000005},"last":0.621}]},{"osd":4,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65900000000000003,"15min":0.64700000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.85799999999999998,"5min":0.92500000000000004,"15min":1.008},"last":0.877},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.66900000000000004,"15min":0.65900000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.93000000000000005}]},{"osd":5,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.67300000000000004,"15min":0.66200000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.371},"max":{"1min":0.93799999999999994,"5min":0.95499999999999996,"15min":0.98299999999999998},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.69099999999999995,"15min":0.67400000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91800000000000004,"5min":0.94199999999999995,"15min":1.0189999999999999},"last":0.90000000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.67600000000000005,"15min":0.66700000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.97199999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.69499999999999995,"15min":0.68500000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.877,"5min":1,"15min":1},"last":0.88900000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.71199999999999997,"15min":0.69499999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.89000000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68000000000000005,"15min":0.66900000000000004},"min":{"1min":0.48699999999999999,"5min":0.46899999999999997,"15min":0.44},"max":{"1min":0.91200000000000003,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.93899999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575767,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38140,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94333700,"statfs":{"total":96636764160,"available":96597708800,"internally_reserved":0,"allocated":2990080,"data_stored":1799026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39700000000000002,"15min":0.39800000000000002},"min":{"1min":0.27600000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.46700000000000003,"5min":0.58799999999999997,"15min":0.65000000000000002},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42599999999999999,"15min":0.42499999999999999},"min":{"1min":0.28499999999999998,"5min":0.28000000000000003,"15min":0.26400000000000001},"max":{"1min":0.57199999999999995,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.36599999999999999}]},{"osd":2,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.434,"15min":0.434},"min":{"1min":0.313,"5min":0.29699999999999999,"15min":0.27000000000000002},"max":{"1min":0.58699999999999997,"5min":0.67200000000000004,"15min":0.71199999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.45000000000000001,"15min":0.45300000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.51400000000000001,"5min":0.63,"15min":0.77000000000000002},"last":0.32700000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.436,"15min":0.435},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.27900000000000003},"max":{"1min":0.51100000000000001,"5min":0.61899999999999999,"15min":0.63200000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.39700000000000002,"5min":0.34399999999999997,"15min":0.307},"max":{"1min":0.64500000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.377}]},{"osd":4,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.67200000000000004,"15min":0.65400000000000003},"min":{"1min":0.47899999999999998,"5min":0.46899999999999997,"15min":0.45600000000000002},"max":{"1min":0.86299999999999999,"5min":0.93000000000000005,"15min":0.999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.67900000000000005,"15min":0.66800000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.41499999999999998},"max":{"1min":0.82499999999999996,"5min":0.96699999999999997,"15min":1.103},"last":0.53300000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.66900000000000004,"15min":0.65700000000000003},"min":{"1min":0.46700000000000003,"5min":0.46100000000000002,"15min":0.46000000000000002},"max":{"1min":0.77100000000000002,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70499999999999996,"15min":0.68400000000000005},"min":{"1min":0.497,"5min":0.48999999999999999,"15min":0.41299999999999998},"max":{"1min":0.89100000000000001,"5min":1.0169999999999999,"15min":1.0469999999999999},"last":0.52000000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.70399999999999996,"15min":0.68899999999999995},"min":{"1min":0.54000000000000004,"5min":0.49399999999999999,"15min":0.46000000000000002},"max":{"1min":0.91100000000000003,"5min":1.0629999999999999,"15min":1.0920000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.70399999999999996,"15min":0.68799999999999994},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.46000000000000002},"max":{"1min":0.871,"5min":0.92900000000000005,"15min":0.98599999999999999},"last":0.56499999999999995}]},{"osd":7,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.71999999999999997,"15min":0.70299999999999996},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.46400000000000002},"max":{"1min":0.81399999999999995,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.71499999999999997,"15min":0.69299999999999995},"min":{"1min":0.502,"5min":0.48899999999999999,"15min":0.438},"max":{"1min":0.92400000000000004,"5min":1.042,"15min":1.071},"last":0.57199999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575769,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59708,"kb_used_data":2344,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312132,"statfs":{"total":96636764160,"available":96575623168,"internally_reserved":0,"allocated":2400256,"data_stored":1215218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.39400000000000002,"15min":0.39300000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41599999999999998,"15min":0.42899999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.68700000000000006},"last":0.315}]},{"osd":1,"last update":"Sat Apr 6 13:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.42499999999999999,"15min":0.41899999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.6190000000000002,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":0.378},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":0.41399999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.49399999999999999,"15min":0.48199999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":3.9980000000000002,"5min":3.9980000000000002,"15min":3.9980000000000002},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.47399999999999998,"15min":0.46300000000000002},"min":{"1min":0.33500000000000002,"5min":0.33300000000000002,"15min":0.29699999999999999},"max":{"1min":4.0350000000000001,"5min":4.0350000000000001,"15min":4.0350000000000001},"last":0.433}]},{"osd":4,"last update":"Sat Apr 6 13:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.70099999999999996,"15min":0.66800000000000004},"min":{"1min":0.42099999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":6.335,"5min":6.335,"15min":6.335},"last":0.501},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.70699999999999996,"15min":0.67700000000000005},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.434},"max":{"1min":5.2610000000000001,"5min":5.2610000000000001,"15min":5.2610000000000001},"last":0.52100000000000002}]},{"osd":5,"last update":"Sat Apr 6 13:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.69599999999999995,"15min":0.68200000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":2.7599999999999998,"5min":2.7599999999999998,"15min":2.7599999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67000000000000004,"15min":0.66800000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.206,"5min":2.206,"15min":2.206},"last":0.53300000000000003}]},{"osd":6,"last update":"Sat Apr 6 13:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.66200000000000003,"15min":0.66700000000000004},"min":{"1min":0.42499999999999999,"5min":0.42199999999999999,"15min":0.40500000000000003},"max":{"1min":0.91200000000000003,"5min":0.93899999999999995,"15min":1.022},"last":0.621},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.69799999999999995,"15min":0.69899999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.41999999999999998},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":0.443}]},{"osd":7,"last update":"Sat Apr 6 13:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.70699999999999996,"15min":0.69399999999999995},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.438},"max":{"1min":2.0110000000000001,"5min":2.0110000000000001,"15min":2.0110000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72999999999999998,"15min":0.71799999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.43099999999999999},"max":{"1min":2.8090000000000002,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":0.51100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575768,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59724,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312116,"statfs":{"total":96636764160,"available":96575606784,"internally_reserved":0,"allocated":2424832,"data_stored":1218405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.39400000000000002,"15min":0.38700000000000001},"min":{"1min":0.27400000000000002,"5min":0.25700000000000001,"15min":0.248},"max":{"1min":0.54400000000000004,"5min":0.61199999999999999,"15min":0.621},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.41199999999999998,"15min":0.40500000000000003},"min":{"1min":0.27100000000000002,"5min":0.253,"15min":0.24099999999999999},"max":{"1min":0.51400000000000001,"5min":0.66000000000000003,"15min":0.67100000000000004},"last":0.28000000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.42099999999999999,"15min":0.41299999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.58999999999999997,"5min":0.64100000000000001,"15min":0.64900000000000002},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.443,"15min":0.441},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.56699999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.29399999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.5,"15min":0.49299999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.629,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.37},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.503,"15min":0.501},"min":{"1min":0.38,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.66200000000000003,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.38}]},{"osd":4,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65800000000000003,"15min":0.63600000000000001},"min":{"1min":0.49199999999999999,"5min":0.46400000000000002,"15min":0.34200000000000003},"max":{"1min":0.86599999999999999,"5min":0.88400000000000001,"15min":0.96799999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.66100000000000003,"15min":0.65700000000000003},"min":{"1min":0.47299999999999998,"5min":0.436,"15min":0.436},"max":{"1min":0.76200000000000001,"5min":0.97199999999999998,"15min":0.97299999999999998},"last":0.58199999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69199999999999995,"15min":0.68200000000000005},"min":{"1min":0.48299999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.89400000000000002,"5min":0.97699999999999998,"15min":1.024},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.71299999999999997,"15min":0.70099999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.41999999999999998},"max":{"1min":0.81399999999999995,"5min":0.98699999999999999,"15min":1.0069999999999999},"last":0.53300000000000003}]},{"osd":6,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.68700000000000006,"15min":0.67200000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45200000000000001},"max":{"1min":0.79500000000000004,"5min":1.0149999999999999,"15min":1.0569999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.69299999999999995,"15min":0.67900000000000005},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.38900000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.98899999999999999},"last":0.47799999999999998}]},{"osd":7,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.68600000000000005,"15min":0.68000000000000005},"min":{"1min":0.50800000000000001,"5min":0.47599999999999998,"15min":0.42199999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":1.0009999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.72299999999999998,"15min":0.70299999999999996},"min":{"1min":0.495,"5min":0.47999999999999998,"15min":0.46800000000000003},"max":{"1min":0.88,"5min":1.028,"15min":1.028},"last":0.56599999999999995}]}]},{"osd":4,"up_from":13,"seq":55834575768,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41672,"kb_used_data":2364,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330168,"statfs":{"total":96636764160,"available":96594092032,"internally_reserved":0,"allocated":2420736,"data_stored":1217001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59399999999999997,"15min":0.59999999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.745,"5min":0.78900000000000003,"15min":1.2609999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.45800000000000002,"5min":0.40699999999999997,"15min":0.376},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.52300000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64700000000000002,"15min":0.64900000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.35199999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.86699999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.64000000000000001},"min":{"1min":0.45100000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.78100000000000003,"5min":0.79300000000000004,"15min":0.82499999999999996},"last":0.40699999999999997}]},{"osd":2,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59599999999999997,"15min":0.60699999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.33600000000000002},"max":{"1min":0.81399999999999995,"5min":0.82399999999999995,"15min":1.5109999999999999},"last":0.375},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.622,"15min":0.623},"min":{"1min":0.436,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.43099999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.60399999999999998,"15min":0.60299999999999998},"min":{"1min":0.39100000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73399999999999999,"5min":0.78600000000000003,"15min":1.1439999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63300000000000001,"15min":0.63400000000000001},"min":{"1min":0.38800000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":1.319},"last":0.54200000000000004}]},{"osd":5,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48899999999999999,"15min":0.504},"min":{"1min":0.35199999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66900000000000004,"5min":0.748,"15min":0.878},"last":0.623},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51800000000000002,"15min":0.52200000000000002},"min":{"1min":0.38300000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":1.3839999999999999},"last":0.47099999999999997}]},{"osd":6,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.308},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":1.6850000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40500000000000003,"5min":0.39900000000000002,"15min":0.36699999999999999},"max":{"1min":0.755,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.59999999999999998}]},{"osd":7,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.374,"5min":0.36899999999999999,"15min":0.35099999999999998},"max":{"1min":0.79000000000000004,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.61399999999999999,"15min":0.60299999999999998},"min":{"1min":0.46999999999999997,"5min":0.41499999999999998,"15min":0.35799999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.7410000000000001},"last":0.61199999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575768,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59704,"kb_used_data":2340,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312136,"statfs":{"total":96636764160,"available":96575627264,"internally_reserved":0,"allocated":2396160,"data_stored":1208678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.755,"5min":0.79100000000000004,"15min":0.81299999999999994},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.64900000000000002,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.53100000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.59699999999999998,"15min":0.59099999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.58899999999999997,"15min":0.58999999999999997},"min":{"1min":0.38900000000000001,"5min":0.36899999999999999,"15min":0.30599999999999999},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.504}]},{"osd":2,"last update":"Sat Apr 6 13:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.60199999999999998,"15min":0.60099999999999998},"min":{"1min":0.40799999999999997,"5min":0.33700000000000002,"15min":0.33200000000000002},"max":{"1min":0.63200000000000001,"5min":0.80300000000000005,"15min":0.81999999999999995},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.59599999999999997,"15min":0.59299999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76600000000000001},"last":0.55800000000000005}]},{"osd":3,"last update":"Sat Apr 6 13:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.60699999999999998,"15min":0.60299999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35499999999999998},"max":{"1min":0.78400000000000003,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.628,"15min":0.622},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.745,"5min":0.82599999999999996,"15min":0.82899999999999996},"last":0.622}]},{"osd":4,"last update":"Sat Apr 6 13:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.505,"15min":0.501},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72299999999999998,"5min":0.72499999999999998,"15min":0.85199999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.56699999999999995,"15min":0.55400000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.29099999999999998},"max":{"1min":0.80100000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.68400000000000005}]},{"osd":6,"last update":"Sat Apr 6 13:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.51500000000000001,"15min":0.504},"min":{"1min":0.33100000000000002,"5min":0.32300000000000001,"15min":0.29699999999999999},"max":{"1min":0.68600000000000005,"5min":0.80000000000000004,"15min":0.82699999999999996},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.52200000000000002,"15min":0.50600000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.77600000000000002,"5min":0.77900000000000003,"15min":0.80300000000000005},"last":0.58799999999999997}]},{"osd":7,"last update":"Sat Apr 6 13:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54200000000000004,"15min":0.52700000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.314},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64500000000000002,"15min":0.621},"min":{"1min":0.441,"5min":0.42099999999999999,"15min":0.376},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":0.72399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:29:21.026 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:29:21.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.148+0000 7f4cbf56f700 1 Processor -- start 2024-04-06T13:29:21.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.148+0000 7f4cbf56f700 1 -- start start 2024-04-06T13:29:21.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.149+0000 7f4cbf56f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4cb815a550 0x7f4cb80cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:21.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.149+0000 7f4cbf56f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4cb80b4d70 0x7f4cb80b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:21.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.149+0000 7f4cbf56f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb80cdbe0 con 0x7f4cb80cf900 2024-04-06T13:29:21.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.149+0000 7f4cbf56f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb815a940 con 0x7f4cb815a550 2024-04-06T13:29:21.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.149+0000 7f4cbf56f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb815ac00 con 0x7f4cb80b4d70 2024-04-06T13:29:21.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.149+0000 7f4cbd30b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4cb815a550 0x7f4cb80cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:21.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.149+0000 7f4cbdb0c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4cb80b4d70 0x7f4cb80b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:21.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.149+0000 7f4cbd30b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4cb815a550 0x7f4cb80cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:51932/0 (socket says 172.21.15.189:51932) 2024-04-06T13:29:21.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.149+0000 7f4cbd30b700 1 -- 172.21.15.189:0/1226202943 learned_addr learned my addr 172.21.15.189:0/1226202943 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:21.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.150+0000 7f4cbd30b700 1 -- 172.21.15.189:0/1226202943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4cb80b4d70 msgr2=0x7f4cb80b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:21.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.150+0000 7f4cbd30b700 1 --2- 172.21.15.189:0/1226202943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4cb80b4d70 0x7f4cb80b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.150+0000 7f4cbd30b700 1 -- 172.21.15.189:0/1226202943 >> v1:172.21.15.192:6789/0 conn(0x7f4cb80cf900 legacy=0x7f4cb80b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:21.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.150+0000 7f4caffff700 1 -- 172.21.15.189:0/1226202943 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970366418 0 0) 0x7f4cb80cdbe0 con 0x7f4cb80cf900 2024-04-06T13:29:21.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.150+0000 7f4cbd30b700 1 -- 172.21.15.189:0/1226202943 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb815aee0 con 0x7f4cb815a550 2024-04-06T13:29:21.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.150+0000 7f4cbdb0c700 1 --2- 172.21.15.189:0/1226202943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4cb80b4d70 0x7f4cb80b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:29:21.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.150+0000 7f4cbd30b700 1 --2- 172.21.15.189:0/1226202943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cb815a550 0x7f4cb80cf3c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f4cb40098f0 tx=0x7f4cb4009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f955d9d5014cc876 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:21.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.150+0000 7f4caffff700 1 -- 172.21.15.189:0/1226202943 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4011830 con 0x7f4cb815a550 2024-04-06T13:29:21.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.150+0000 7f4caffff700 1 -- 172.21.15.189:0/1226202943 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb4011990 con 0x7f4cb815a550 2024-04-06T13:29:21.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.150+0000 7f4caffff700 1 -- 172.21.15.189:0/1226202943 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4011d20 con 0x7f4cb815a550 2024-04-06T13:29:21.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.150+0000 7f4cbf56f700 1 -- 172.21.15.189:0/1226202943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cb815a550 msgr2=0x7f4cb80cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:21.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.150+0000 7f4cbf56f700 1 --2- 172.21.15.189:0/1226202943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cb815a550 0x7f4cb80cf3c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f4cb40098f0 tx=0x7f4cb4009c00 comp rx=0 tx=0).stop 2024-04-06T13:29:21.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.150+0000 7f4cbf56f700 1 -- 172.21.15.189:0/1226202943 shutdown_connections 2024-04-06T13:29:21.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.150+0000 7f4cbf56f700 1 --2- 172.21.15.189:0/1226202943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4cb80b4d70 0x7f4cb80b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.150+0000 7f4cbf56f700 1 --2- 172.21.15.189:0/1226202943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cb815a550 0x7f4cb80cf3c0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.150+0000 7f4cbf56f700 1 -- 172.21.15.189:0/1226202943 >> 172.21.15.189:0/1226202943 conn(0x7f4cb80d00b0 msgr2=0x7f4cb80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:21.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.151+0000 7f4cbf56f700 1 -- 172.21.15.189:0/1226202943 shutdown_connections 2024-04-06T13:29:21.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.151+0000 7f4cbf56f700 1 -- 172.21.15.189:0/1226202943 wait complete. 2024-04-06T13:29:21.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.152+0000 7f4cbf56f700 1 Processor -- start 2024-04-06T13:29:21.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.152+0000 7f4cbf56f700 1 -- start start 2024-04-06T13:29:21.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.152+0000 7f4cbf56f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4cb80b4d70 0x7f4cb814cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:21.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.152+0000 7f4cbf56f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cb80cf900 0x7f4cb814d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:21.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.152+0000 7f4cbf56f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cb806a730 0x7f4cb806abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:21.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.152+0000 7f4cbf56f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb806b100 con 0x7f4cb80cf900 2024-04-06T13:29:21.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.152+0000 7f4cbf56f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb8147730 con 0x7f4cb806a730 2024-04-06T13:29:21.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.152+0000 7f4cbf56f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb8145a50 con 0x7f4cb80b4d70 2024-04-06T13:29:21.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.153+0000 7f4cbcb0a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cb80cf900 0x7f4cb814d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:21.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.153+0000 7f4cbcb0a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cb80cf900 0x7f4cb814d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54264/0 (socket says 172.21.15.189:54264) 2024-04-06T13:29:21.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.153+0000 7f4cbcb0a700 1 -- 172.21.15.189:0/522994853 learned_addr learned my addr 172.21.15.189:0/522994853 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:21.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.153+0000 7f4cbdb0c700 1 --2- 172.21.15.189:0/522994853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cb806a730 0x7f4cb806abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:21.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.153+0000 7f4cbd30b700 1 --2- 172.21.15.189:0/522994853 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4cb80b4d70 0x7f4cb814cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:21.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.153+0000 7f4cbcb0a700 1 -- 172.21.15.189:0/522994853 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4cb80b4d70 msgr2=0x7f4cb814cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:21.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.153+0000 7f4cbcb0a700 1 --2- 172.21.15.189:0/522994853 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4cb80b4d70 0x7f4cb814cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.153+0000 7f4cbcb0a700 1 -- 172.21.15.189:0/522994853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cb806a730 msgr2=0x7f4cb806abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:21.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.153+0000 7f4cbcb0a700 1 --2- 172.21.15.189:0/522994853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cb806a730 0x7f4cb806abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.153+0000 7f4cbcb0a700 1 -- 172.21.15.189:0/522994853 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb0009a70 con 0x7f4cb80cf900 2024-04-06T13:29:21.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.153+0000 7f4cbdb0c700 1 --2- 172.21.15.189:0/522994853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cb806a730 0x7f4cb806abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:29:21.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.153+0000 7f4cbcb0a700 1 --2- 172.21.15.189:0/522994853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cb80cf900 0x7f4cb814d820 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f4ca80011e0 tx=0x7f4ca800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:21.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.153+0000 7f4cadffb700 1 -- 172.21.15.189:0/522994853 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca80054e0 con 0x7f4cb80cf900 2024-04-06T13:29:21.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.153+0000 7f4cadffb700 1 -- 172.21.15.189:0/522994853 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca8003680 con 0x7f4cb80cf900 2024-04-06T13:29:21.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.153+0000 7f4cbf56f700 1 -- 172.21.15.189:0/522994853 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb4009550 con 0x7f4cb80cf900 2024-04-06T13:29:21.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.153+0000 7f4cadffb700 1 -- 172.21.15.189:0/522994853 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8003a30 con 0x7f4cb80cf900 2024-04-06T13:29:21.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.154+0000 7f4cbf56f700 1 -- 172.21.15.189:0/522994853 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb8146010 con 0x7f4cb80cf900 2024-04-06T13:29:21.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.154+0000 7f4cbf56f700 1 -- 172.21.15.189:0/522994853 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb8147590 con 0x7f4cb80cf900 2024-04-06T13:29:21.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.155+0000 7f4cadffb700 1 -- 172.21.15.189:0/522994853 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f4ca8003b90 con 0x7f4cb80cf900 2024-04-06T13:29:21.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.155+0000 7f4cadffb700 1 --2- 172.21.15.189:0/522994853 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4c9c06adb0 0x7f4c9c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:21.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.155+0000 7f4cadffb700 1 -- 172.21.15.189:0/522994853 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4ca8088ba0 con 0x7f4cb80cf900 2024-04-06T13:29:21.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.155+0000 7f4cbd30b700 1 --2- 172.21.15.189:0/522994853 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4c9c06adb0 0x7f4c9c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:21.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.156+0000 7f4cbd30b700 1 --2- 172.21.15.189:0/522994853 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4c9c06adb0 0x7f4c9c06d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4014040 tx=0x7f4cb4008030 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:21.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.160+0000 7f4cadffb700 1 -- 172.21.15.189:0/522994853 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4cb8147590 con 0x7f4cb80cf900 2024-04-06T13:29:21.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.349+0000 7f4cbf56f700 1 -- 172.21.15.189:0/522994853 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4cb8145bc0 con 0x7f4c9c06adb0 2024-04-06T13:29:21.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.357+0000 7f4cadffb700 1 -- 172.21.15.189:0/522994853 <== mgr.9034 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289989 (secure 0 0 0) 0x7f4cb8145bc0 con 0x7f4c9c06adb0 2024-04-06T13:29:21.361 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:29:21.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.361+0000 7f4cbf56f700 1 -- 172.21.15.189:0/522994853 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4c9c06adb0 msgr2=0x7f4c9c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:21.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.361+0000 7f4cbf56f700 1 --2- 172.21.15.189:0/522994853 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4c9c06adb0 0x7f4c9c06d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4014040 tx=0x7f4cb4008030 comp rx=0 tx=0).stop 2024-04-06T13:29:21.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.361+0000 7f4cbf56f700 1 -- 172.21.15.189:0/522994853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cb80cf900 msgr2=0x7f4cb814d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:21.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.361+0000 7f4cbf56f700 1 --2- 172.21.15.189:0/522994853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cb80cf900 0x7f4cb814d820 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f4ca80011e0 tx=0x7f4ca800cae0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.361+0000 7f4cbf56f700 1 -- 172.21.15.189:0/522994853 shutdown_connections 2024-04-06T13:29:21.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.361+0000 7f4cbf56f700 1 --2- 172.21.15.189:0/522994853 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f4c9c06adb0 0x7f4c9c06d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.361+0000 7f4cbf56f700 1 --2- 172.21.15.189:0/522994853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cb806a730 0x7f4cb806abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.361+0000 7f4cbf56f700 1 --2- 172.21.15.189:0/522994853 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4cb80b4d70 0x7f4cb814cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.361+0000 7f4cbf56f700 1 --2- 172.21.15.189:0/522994853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cb80cf900 0x7f4cb814d820 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.361+0000 7f4cbf56f700 1 -- 172.21.15.189:0/522994853 >> 172.21.15.189:0/522994853 conn(0x7f4cb80d00b0 msgr2=0x7f4cb8151c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:21.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.361+0000 7f4cbf56f700 1 -- 172.21.15.189:0/522994853 shutdown_connections 2024-04-06T13:29:21.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.361+0000 7f4cbf56f700 1 -- 172.21.15.189:0/522994853 wait complete. 2024-04-06T13:29:21.364 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:29:21.373 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-06T13:29:21.217524+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":523,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":331,"ondisk_log_size":331,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":184,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":386168,"kb_used_data":19952,"kb_used_omap":12,"kb_used_meta":366067,"kb_avail":754588552,"statfs":{"total":773094113280,"available":772698677248,"internally_reserved":0,"allocated":20430848,"data_stored":10890269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":374853176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.776703"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870006+0000","last_change":"2024-04-06T13:29:16.887629+0000","last_active":"2024-04-06T13:29:17.870006+0000","last_peered":"2024-04-06T13:29:17.870006+0000","last_clean":"2024-04-06T13:29:17.870006+0000","last_became_active":"2024-04-06T13:29:16.887340+0000","last_became_peered":"2024-04-06T13:29:16.887340+0000","last_unstale":"2024-04-06T13:29:17.870006+0000","last_undegraded":"2024-04-06T13:29:17.870006+0000","last_fullsized":"2024-04-06T13:29:17.870006+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:32.732586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870840+0000","last_change":"2024-04-06T13:29:16.882535+0000","last_active":"2024-04-06T13:29:17.870840+0000","last_peered":"2024-04-06T13:29:17.870840+0000","last_clean":"2024-04-06T13:29:17.870840+0000","last_became_active":"2024-04-06T13:29:16.882219+0000","last_became_peered":"2024-04-06T13:29:16.882219+0000","last_unstale":"2024-04-06T13:29:17.870840+0000","last_undegraded":"2024-04-06T13:29:17.870840+0000","last_fullsized":"2024-04-06T13:29:17.870840+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:51:38.673432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870321+0000","last_change":"2024-04-06T13:29:16.885378+0000","last_active":"2024-04-06T13:29:17.870321+0000","last_peered":"2024-04-06T13:29:17.870321+0000","last_clean":"2024-04-06T13:29:17.870321+0000","last_became_active":"2024-04-06T13:29:16.885055+0000","last_became_peered":"2024-04-06T13:29:16.885055+0000","last_unstale":"2024-04-06T13:29:17.870321+0000","last_undegraded":"2024-04-06T13:29:17.870321+0000","last_fullsized":"2024-04-06T13:29:17.870321+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:38:42.932002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869716+0000","last_change":"2024-04-06T13:29:16.887481+0000","last_active":"2024-04-06T13:29:17.869716+0000","last_peered":"2024-04-06T13:29:17.869716+0000","last_clean":"2024-04-06T13:29:17.869716+0000","last_became_active":"2024-04-06T13:29:16.887096+0000","last_became_peered":"2024-04-06T13:29:16.887096+0000","last_unstale":"2024-04-06T13:29:17.869716+0000","last_undegraded":"2024-04-06T13:29:17.869716+0000","last_fullsized":"2024-04-06T13:29:17.869716+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:06:40.389835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015524+0000","last_change":"2024-04-06T13:29:16.888478+0000","last_active":"2024-04-06T13:29:19.015524+0000","last_peered":"2024-04-06T13:29:19.015524+0000","last_clean":"2024-04-06T13:29:19.015524+0000","last_became_active":"2024-04-06T13:29:16.888164+0000","last_became_peered":"2024-04-06T13:29:16.888164+0000","last_unstale":"2024-04-06T13:29:19.015524+0000","last_undegraded":"2024-04-06T13:29:19.015524+0000","last_fullsized":"2024-04-06T13:29:19.015524+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:10:03.720777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870250+0000","last_change":"2024-04-06T13:29:16.879186+0000","last_active":"2024-04-06T13:29:17.870250+0000","last_peered":"2024-04-06T13:29:17.870250+0000","last_clean":"2024-04-06T13:29:17.870250+0000","last_became_active":"2024-04-06T13:29:16.878952+0000","last_became_peered":"2024-04-06T13:29:16.878952+0000","last_unstale":"2024-04-06T13:29:17.870250+0000","last_undegraded":"2024-04-06T13:29:17.870250+0000","last_fullsized":"2024-04-06T13:29:17.870250+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:31:46.993248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031667+0000","last_change":"2024-04-06T13:29:16.880932+0000","last_active":"2024-04-06T13:29:18.031667+0000","last_peered":"2024-04-06T13:29:18.031667+0000","last_clean":"2024-04-06T13:29:18.031667+0000","last_became_active":"2024-04-06T13:29:16.880184+0000","last_became_peered":"2024-04-06T13:29:16.880184+0000","last_unstale":"2024-04-06T13:29:18.031667+0000","last_undegraded":"2024-04-06T13:29:18.031667+0000","last_fullsized":"2024-04-06T13:29:18.031667+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:55:24.254991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031614+0000","last_change":"2024-04-06T13:29:16.886799+0000","last_active":"2024-04-06T13:29:18.031614+0000","last_peered":"2024-04-06T13:29:18.031614+0000","last_clean":"2024-04-06T13:29:18.031614+0000","last_became_active":"2024-04-06T13:29:16.885980+0000","last_became_peered":"2024-04-06T13:29:16.885980+0000","last_unstale":"2024-04-06T13:29:18.031614+0000","last_undegraded":"2024-04-06T13:29:18.031614+0000","last_fullsized":"2024-04-06T13:29:18.031614+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:17:51.615500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.032000+0000","last_change":"2024-04-06T13:29:16.879765+0000","last_active":"2024-04-06T13:29:18.032000+0000","last_peered":"2024-04-06T13:29:18.032000+0000","last_clean":"2024-04-06T13:29:18.032000+0000","last_became_active":"2024-04-06T13:29:16.878170+0000","last_became_peered":"2024-04-06T13:29:16.878170+0000","last_unstale":"2024-04-06T13:29:18.032000+0000","last_undegraded":"2024-04-06T13:29:18.032000+0000","last_fullsized":"2024-04-06T13:29:18.032000+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:51:38.673432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.032040+0000","last_change":"2024-04-06T13:29:16.885817+0000","last_active":"2024-04-06T13:29:18.032040+0000","last_peered":"2024-04-06T13:29:18.032040+0000","last_clean":"2024-04-06T13:29:18.032040+0000","last_became_active":"2024-04-06T13:29:16.885479+0000","last_became_peered":"2024-04-06T13:29:16.885479+0000","last_unstale":"2024-04-06T13:29:18.032040+0000","last_undegraded":"2024-04-06T13:29:18.032040+0000","last_fullsized":"2024-04-06T13:29:18.032040+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:29:16.732050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015859+0000","last_change":"2024-04-06T13:29:16.883659+0000","last_active":"2024-04-06T13:29:19.015859+0000","last_peered":"2024-04-06T13:29:19.015859+0000","last_clean":"2024-04-06T13:29:19.015859+0000","last_became_active":"2024-04-06T13:29:16.883354+0000","last_became_peered":"2024-04-06T13:29:16.883354+0000","last_unstale":"2024-04-06T13:29:19.015859+0000","last_undegraded":"2024-04-06T13:29:19.015859+0000","last_fullsized":"2024-04-06T13:29:19.015859+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:45:02.241916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869830+0000","last_change":"2024-04-06T13:29:16.887846+0000","last_active":"2024-04-06T13:29:17.869830+0000","last_peered":"2024-04-06T13:29:17.869830+0000","last_clean":"2024-04-06T13:29:17.869830+0000","last_became_active":"2024-04-06T13:29:16.887563+0000","last_became_peered":"2024-04-06T13:29:16.887563+0000","last_unstale":"2024-04-06T13:29:17.869830+0000","last_undegraded":"2024-04-06T13:29:17.869830+0000","last_fullsized":"2024-04-06T13:29:17.869830+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:44:50.624713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869522+0000","last_change":"2024-04-06T13:29:16.870100+0000","last_active":"2024-04-06T13:29:17.869522+0000","last_peered":"2024-04-06T13:29:17.869522+0000","last_clean":"2024-04-06T13:29:17.869522+0000","last_became_active":"2024-04-06T13:29:16.869874+0000","last_became_peered":"2024-04-06T13:29:16.869874+0000","last_unstale":"2024-04-06T13:29:17.869522+0000","last_undegraded":"2024-04-06T13:29:17.869522+0000","last_fullsized":"2024-04-06T13:29:17.869522+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:40:22.459776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015446+0000","last_change":"2024-04-06T13:29:16.889997+0000","last_active":"2024-04-06T13:29:19.015446+0000","last_peered":"2024-04-06T13:29:19.015446+0000","last_clean":"2024-04-06T13:29:19.015446+0000","last_became_active":"2024-04-06T13:29:16.889378+0000","last_became_peered":"2024-04-06T13:29:16.889378+0000","last_unstale":"2024-04-06T13:29:19.015446+0000","last_undegraded":"2024-04-06T13:29:19.015446+0000","last_fullsized":"2024-04-06T13:29:19.015446+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:06:59.197190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.016173+0000","last_change":"2024-04-06T13:29:16.891204+0000","last_active":"2024-04-06T13:29:19.016173+0000","last_peered":"2024-04-06T13:29:19.016173+0000","last_clean":"2024-04-06T13:29:19.016173+0000","last_became_active":"2024-04-06T13:29:16.890910+0000","last_became_peered":"2024-04-06T13:29:16.890910+0000","last_unstale":"2024-04-06T13:29:19.016173+0000","last_undegraded":"2024-04-06T13:29:19.016173+0000","last_fullsized":"2024-04-06T13:29:19.016173+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:17:57.757521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869361+0000","last_change":"2024-04-06T13:29:16.885388+0000","last_active":"2024-04-06T13:29:17.869361+0000","last_peered":"2024-04-06T13:29:17.869361+0000","last_clean":"2024-04-06T13:29:17.869361+0000","last_became_active":"2024-04-06T13:29:16.885101+0000","last_became_peered":"2024-04-06T13:29:16.885101+0000","last_unstale":"2024-04-06T13:29:17.869361+0000","last_undegraded":"2024-04-06T13:29:17.869361+0000","last_fullsized":"2024-04-06T13:29:17.869361+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:28:11.407947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.032039+0000","last_change":"2024-04-06T13:29:16.882768+0000","last_active":"2024-04-06T13:29:18.032039+0000","last_peered":"2024-04-06T13:29:18.032039+0000","last_clean":"2024-04-06T13:29:18.032039+0000","last_became_active":"2024-04-06T13:29:16.882522+0000","last_became_peered":"2024-04-06T13:29:16.882522+0000","last_unstale":"2024-04-06T13:29:18.032039+0000","last_undegraded":"2024-04-06T13:29:18.032039+0000","last_fullsized":"2024-04-06T13:29:18.032039+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:44:33.412863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015810+0000","last_change":"2024-04-06T13:29:16.892552+0000","last_active":"2024-04-06T13:29:19.015810+0000","last_peered":"2024-04-06T13:29:19.015810+0000","last_clean":"2024-04-06T13:29:19.015810+0000","last_became_active":"2024-04-06T13:29:16.892251+0000","last_became_peered":"2024-04-06T13:29:16.892251+0000","last_unstale":"2024-04-06T13:29:19.015810+0000","last_undegraded":"2024-04-06T13:29:19.015810+0000","last_fullsized":"2024-04-06T13:29:19.015810+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:47:10.159438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015958+0000","last_change":"2024-04-06T13:29:16.891253+0000","last_active":"2024-04-06T13:29:19.015958+0000","last_peered":"2024-04-06T13:29:19.015958+0000","last_clean":"2024-04-06T13:29:19.015958+0000","last_became_active":"2024-04-06T13:29:16.890946+0000","last_became_peered":"2024-04-06T13:29:16.890946+0000","last_unstale":"2024-04-06T13:29:19.015958+0000","last_undegraded":"2024-04-06T13:29:19.015958+0000","last_fullsized":"2024-04-06T13:29:19.015958+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:25:07.172886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.514888+0000","last_change":"2024-04-06T13:29:16.882939+0000","last_active":"2024-04-06T13:29:18.514888+0000","last_peered":"2024-04-06T13:29:18.514888+0000","last_clean":"2024-04-06T13:29:18.514888+0000","last_became_active":"2024-04-06T13:29:16.882654+0000","last_became_peered":"2024-04-06T13:29:16.882654+0000","last_unstale":"2024-04-06T13:29:18.514888+0000","last_undegraded":"2024-04-06T13:29:18.514888+0000","last_fullsized":"2024-04-06T13:29:18.514888+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:18:44.916349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869613+0000","last_change":"2024-04-06T13:29:16.887439+0000","last_active":"2024-04-06T13:29:17.869613+0000","last_peered":"2024-04-06T13:29:17.869613+0000","last_clean":"2024-04-06T13:29:17.869613+0000","last_became_active":"2024-04-06T13:29:16.887073+0000","last_became_peered":"2024-04-06T13:29:16.887073+0000","last_unstale":"2024-04-06T13:29:17.869613+0000","last_undegraded":"2024-04-06T13:29:17.869613+0000","last_fullsized":"2024-04-06T13:29:17.869613+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:55:35.395643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870360+0000","last_change":"2024-04-06T13:29:16.882888+0000","last_active":"2024-04-06T13:29:17.870360+0000","last_peered":"2024-04-06T13:29:17.870360+0000","last_clean":"2024-04-06T13:29:17.870360+0000","last_became_active":"2024-04-06T13:29:16.882462+0000","last_became_peered":"2024-04-06T13:29:16.882462+0000","last_unstale":"2024-04-06T13:29:17.870360+0000","last_undegraded":"2024-04-06T13:29:17.870360+0000","last_fullsized":"2024-04-06T13:29:17.870360+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:41:05.238951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015442+0000","last_change":"2024-04-06T13:29:16.880186+0000","last_active":"2024-04-06T13:29:19.015442+0000","last_peered":"2024-04-06T13:29:19.015442+0000","last_clean":"2024-04-06T13:29:19.015442+0000","last_became_active":"2024-04-06T13:29:16.879763+0000","last_became_peered":"2024-04-06T13:29:16.879763+0000","last_unstale":"2024-04-06T13:29:19.015442+0000","last_undegraded":"2024-04-06T13:29:19.015442+0000","last_fullsized":"2024-04-06T13:29:19.015442+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:40:51.382582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869453+0000","last_change":"2024-04-06T13:29:16.885554+0000","last_active":"2024-04-06T13:29:17.869453+0000","last_peered":"2024-04-06T13:29:17.869453+0000","last_clean":"2024-04-06T13:29:17.869453+0000","last_became_active":"2024-04-06T13:29:16.885109+0000","last_became_peered":"2024-04-06T13:29:16.885109+0000","last_unstale":"2024-04-06T13:29:17.869453+0000","last_undegraded":"2024-04-06T13:29:17.869453+0000","last_fullsized":"2024-04-06T13:29:17.869453+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:27:59.484229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869787+0000","last_change":"2024-04-06T13:29:16.882750+0000","last_active":"2024-04-06T13:29:17.869787+0000","last_peered":"2024-04-06T13:29:17.869787+0000","last_clean":"2024-04-06T13:29:17.869787+0000","last_became_active":"2024-04-06T13:29:16.882522+0000","last_became_peered":"2024-04-06T13:29:16.882522+0000","last_unstale":"2024-04-06T13:29:17.869787+0000","last_undegraded":"2024-04-06T13:29:17.869787+0000","last_fullsized":"2024-04-06T13:29:17.869787+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:00:02.805980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.515109+0000","last_change":"2024-04-06T13:29:16.892594+0000","last_active":"2024-04-06T13:29:18.515109+0000","last_peered":"2024-04-06T13:29:18.515109+0000","last_clean":"2024-04-06T13:29:18.515109+0000","last_became_active":"2024-04-06T13:29:16.892326+0000","last_became_peered":"2024-04-06T13:29:16.892326+0000","last_unstale":"2024-04-06T13:29:18.515109+0000","last_undegraded":"2024-04-06T13:29:18.515109+0000","last_fullsized":"2024-04-06T13:29:18.515109+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:24:22.331863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870741+0000","last_change":"2024-04-06T13:29:16.882855+0000","last_active":"2024-04-06T13:29:17.870741+0000","last_peered":"2024-04-06T13:29:17.870741+0000","last_clean":"2024-04-06T13:29:17.870741+0000","last_became_active":"2024-04-06T13:29:16.882554+0000","last_became_peered":"2024-04-06T13:29:16.882554+0000","last_unstale":"2024-04-06T13:29:17.870741+0000","last_undegraded":"2024-04-06T13:29:17.870741+0000","last_fullsized":"2024-04-06T13:29:17.870741+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:34:35.301684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869518+0000","last_change":"2024-04-06T13:29:16.888153+0000","last_active":"2024-04-06T13:29:17.869518+0000","last_peered":"2024-04-06T13:29:17.869518+0000","last_clean":"2024-04-06T13:29:17.869518+0000","last_became_active":"2024-04-06T13:29:16.887688+0000","last_became_peered":"2024-04-06T13:29:16.887688+0000","last_unstale":"2024-04-06T13:29:17.869518+0000","last_undegraded":"2024-04-06T13:29:17.869518+0000","last_fullsized":"2024-04-06T13:29:17.869518+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:03:10.474463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870723+0000","last_change":"2024-04-06T13:29:16.891407+0000","last_active":"2024-04-06T13:29:17.870723+0000","last_peered":"2024-04-06T13:29:17.870723+0000","last_clean":"2024-04-06T13:29:17.870723+0000","last_became_active":"2024-04-06T13:29:16.891087+0000","last_became_peered":"2024-04-06T13:29:16.891087+0000","last_unstale":"2024-04-06T13:29:17.870723+0000","last_undegraded":"2024-04-06T13:29:17.870723+0000","last_fullsized":"2024-04-06T13:29:17.870723+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:26:30.192329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015798+0000","last_change":"2024-04-06T13:29:16.888009+0000","last_active":"2024-04-06T13:29:19.015798+0000","last_peered":"2024-04-06T13:29:19.015798+0000","last_clean":"2024-04-06T13:29:19.015798+0000","last_became_active":"2024-04-06T13:29:16.887653+0000","last_became_peered":"2024-04-06T13:29:16.887653+0000","last_unstale":"2024-04-06T13:29:19.015798+0000","last_undegraded":"2024-04-06T13:29:19.015798+0000","last_fullsized":"2024-04-06T13:29:19.015798+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:03:22.503093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015765+0000","last_change":"2024-04-06T13:29:16.879536+0000","last_active":"2024-04-06T13:29:19.015765+0000","last_peered":"2024-04-06T13:29:19.015765+0000","last_clean":"2024-04-06T13:29:19.015765+0000","last_became_active":"2024-04-06T13:29:16.879275+0000","last_became_peered":"2024-04-06T13:29:16.879275+0000","last_unstale":"2024-04-06T13:29:19.015765+0000","last_undegraded":"2024-04-06T13:29:19.015765+0000","last_fullsized":"2024-04-06T13:29:19.015765+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:49:08.120568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.016009+0000","last_change":"2024-04-06T13:29:16.876576+0000","last_active":"2024-04-06T13:29:19.016009+0000","last_peered":"2024-04-06T13:29:19.016009+0000","last_clean":"2024-04-06T13:29:19.016009+0000","last_became_active":"2024-04-06T13:29:16.876306+0000","last_became_peered":"2024-04-06T13:29:16.876306+0000","last_unstale":"2024-04-06T13:29:19.016009+0000","last_undegraded":"2024-04-06T13:29:19.016009+0000","last_fullsized":"2024-04-06T13:29:19.016009+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:08:24.097613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869887+0000","last_change":"2024-04-06T13:29:16.868296+0000","last_active":"2024-04-06T13:29:17.869887+0000","last_peered":"2024-04-06T13:29:17.869887+0000","last_clean":"2024-04-06T13:29:17.869887+0000","last_became_active":"2024-04-06T13:29:16.867999+0000","last_became_peered":"2024-04-06T13:29:16.867999+0000","last_unstale":"2024-04-06T13:29:17.869887+0000","last_undegraded":"2024-04-06T13:29:17.869887+0000","last_fullsized":"2024-04-06T13:29:17.869887+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:47:29.405915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"191'4","reported_seq":15,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870176+0000","last_change":"2024-04-06T13:29:16.451701+0000","last_active":"2024-04-06T13:29:17.870176+0000","last_peered":"2024-04-06T13:29:17.870176+0000","last_clean":"2024-04-06T13:29:17.870176+0000","last_became_active":"2024-04-06T13:29:16.451385+0000","last_became_peered":"2024-04-06T13:29:16.451385+0000","last_unstale":"2024-04-06T13:29:17.870176+0000","last_undegraded":"2024-04-06T13:29:17.870176+0000","last_fullsized":"2024-04-06T13:29:17.870176+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:45:01.236287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869339+0000","last_change":"2024-04-06T13:29:16.876384+0000","last_active":"2024-04-06T13:29:17.869339+0000","last_peered":"2024-04-06T13:29:17.869339+0000","last_clean":"2024-04-06T13:29:17.869339+0000","last_became_active":"2024-04-06T13:29:16.876091+0000","last_became_peered":"2024-04-06T13:29:16.876091+0000","last_unstale":"2024-04-06T13:29:17.869339+0000","last_undegraded":"2024-04-06T13:29:17.869339+0000","last_fullsized":"2024-04-06T13:29:17.869339+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:29:57.354201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"191'1","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015766+0000","last_change":"2024-04-06T13:29:16.466716+0000","last_active":"2024-04-06T13:29:19.015766+0000","last_peered":"2024-04-06T13:29:19.015766+0000","last_clean":"2024-04-06T13:29:19.015766+0000","last_became_active":"2024-04-06T13:29:16.466462+0000","last_became_peered":"2024-04-06T13:29:16.466462+0000","last_unstale":"2024-04-06T13:29:19.015766+0000","last_undegraded":"2024-04-06T13:29:19.015766+0000","last_fullsized":"2024-04-06T13:29:19.015766+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:49:57.644107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015484+0000","last_change":"2024-04-06T13:29:16.889839+0000","last_active":"2024-04-06T13:29:19.015484+0000","last_peered":"2024-04-06T13:29:19.015484+0000","last_clean":"2024-04-06T13:29:19.015484+0000","last_became_active":"2024-04-06T13:29:16.889050+0000","last_became_peered":"2024-04-06T13:29:16.889050+0000","last_unstale":"2024-04-06T13:29:19.015484+0000","last_undegraded":"2024-04-06T13:29:19.015484+0000","last_fullsized":"2024-04-06T13:29:19.015484+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:51:13.835661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"191'2","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.016116+0000","last_change":"2024-04-06T13:29:16.463291+0000","last_active":"2024-04-06T13:29:19.016116+0000","last_peered":"2024-04-06T13:29:19.016116+0000","last_clean":"2024-04-06T13:29:19.016116+0000","last_became_active":"2024-04-06T13:29:16.462877+0000","last_became_peered":"2024-04-06T13:29:16.462877+0000","last_unstale":"2024-04-06T13:29:19.016116+0000","last_undegraded":"2024-04-06T13:29:19.016116+0000","last_fullsized":"2024-04-06T13:29:19.016116+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:45.753781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870718+0000","last_change":"2024-04-06T13:29:16.875052+0000","last_active":"2024-04-06T13:29:17.870718+0000","last_peered":"2024-04-06T13:29:17.870718+0000","last_clean":"2024-04-06T13:29:17.870718+0000","last_became_active":"2024-04-06T13:29:16.874779+0000","last_became_peered":"2024-04-06T13:29:16.874779+0000","last_unstale":"2024-04-06T13:29:17.870718+0000","last_undegraded":"2024-04-06T13:29:17.870718+0000","last_fullsized":"2024-04-06T13:29:17.870718+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:09:34.642057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"191'2","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.514989+0000","last_change":"2024-04-06T13:29:16.456231+0000","last_active":"2024-04-06T13:29:18.514989+0000","last_peered":"2024-04-06T13:29:18.514989+0000","last_clean":"2024-04-06T13:29:18.514989+0000","last_became_active":"2024-04-06T13:29:16.455951+0000","last_became_peered":"2024-04-06T13:29:16.455951+0000","last_unstale":"2024-04-06T13:29:18.514989+0000","last_undegraded":"2024-04-06T13:29:18.514989+0000","last_fullsized":"2024-04-06T13:29:18.514989+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:20:30.598041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031811+0000","last_change":"2024-04-06T13:29:16.885700+0000","last_active":"2024-04-06T13:29:18.031811+0000","last_peered":"2024-04-06T13:29:18.031811+0000","last_clean":"2024-04-06T13:29:18.031811+0000","last_became_active":"2024-04-06T13:29:16.885425+0000","last_became_peered":"2024-04-06T13:29:16.885425+0000","last_unstale":"2024-04-06T13:29:18.031811+0000","last_undegraded":"2024-04-06T13:29:18.031811+0000","last_fullsized":"2024-04-06T13:29:18.031811+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:44:50.624713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"191'1","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031794+0000","last_change":"2024-04-06T13:29:16.466892+0000","last_active":"2024-04-06T13:29:18.031794+0000","last_peered":"2024-04-06T13:29:18.031794+0000","last_clean":"2024-04-06T13:29:18.031794+0000","last_became_active":"2024-04-06T13:29:16.466630+0000","last_became_peered":"2024-04-06T13:29:16.466630+0000","last_unstale":"2024-04-06T13:29:18.031794+0000","last_undegraded":"2024-04-06T13:29:18.031794+0000","last_fullsized":"2024-04-06T13:29:18.031794+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:27:51.741781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.515085+0000","last_change":"2024-04-06T13:29:16.874261+0000","last_active":"2024-04-06T13:29:18.515085+0000","last_peered":"2024-04-06T13:29:18.515085+0000","last_clean":"2024-04-06T13:29:18.515085+0000","last_became_active":"2024-04-06T13:29:16.874024+0000","last_became_peered":"2024-04-06T13:29:16.874024+0000","last_unstale":"2024-04-06T13:29:18.515085+0000","last_undegraded":"2024-04-06T13:29:18.515085+0000","last_fullsized":"2024-04-06T13:29:18.515085+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:40:26.762806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"191'1","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015576+0000","last_change":"2024-04-06T13:29:16.465100+0000","last_active":"2024-04-06T13:29:19.015576+0000","last_peered":"2024-04-06T13:29:19.015576+0000","last_clean":"2024-04-06T13:29:19.015576+0000","last_became_active":"2024-04-06T13:29:16.464833+0000","last_became_peered":"2024-04-06T13:29:16.464833+0000","last_unstale":"2024-04-06T13:29:19.015576+0000","last_undegraded":"2024-04-06T13:29:19.015576+0000","last_fullsized":"2024-04-06T13:29:19.015576+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:26:29.186700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.515103+0000","last_change":"2024-04-06T13:29:16.875862+0000","last_active":"2024-04-06T13:29:18.515103+0000","last_peered":"2024-04-06T13:29:18.515103+0000","last_clean":"2024-04-06T13:29:18.515103+0000","last_became_active":"2024-04-06T13:29:16.875600+0000","last_became_peered":"2024-04-06T13:29:16.875600+0000","last_unstale":"2024-04-06T13:29:18.515103+0000","last_undegraded":"2024-04-06T13:29:18.515103+0000","last_fullsized":"2024-04-06T13:29:18.515103+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:46.759410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.514867+0000","last_change":"2024-04-06T13:29:16.464039+0000","last_active":"2024-04-06T13:29:18.514867+0000","last_peered":"2024-04-06T13:29:18.514867+0000","last_clean":"2024-04-06T13:29:18.514867+0000","last_became_active":"2024-04-06T13:29:16.463345+0000","last_became_peered":"2024-04-06T13:29:16.463345+0000","last_unstale":"2024-04-06T13:29:18.514867+0000","last_undegraded":"2024-04-06T13:29:18.514867+0000","last_fullsized":"2024-04-06T13:29:18.514867+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:33:20.459267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870110+0000","last_change":"2024-04-06T13:29:16.887269+0000","last_active":"2024-04-06T13:29:17.870110+0000","last_peered":"2024-04-06T13:29:17.870110+0000","last_clean":"2024-04-06T13:29:17.870110+0000","last_became_active":"2024-04-06T13:29:16.886536+0000","last_became_peered":"2024-04-06T13:29:16.886536+0000","last_unstale":"2024-04-06T13:29:17.870110+0000","last_undegraded":"2024-04-06T13:29:17.870110+0000","last_fullsized":"2024-04-06T13:29:17.870110+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:03:29.165222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031619+0000","last_change":"2024-04-06T13:29:16.466241+0000","last_active":"2024-04-06T13:29:18.031619+0000","last_peered":"2024-04-06T13:29:18.031619+0000","last_clean":"2024-04-06T13:29:18.031619+0000","last_became_active":"2024-04-06T13:29:16.465829+0000","last_became_peered":"2024-04-06T13:29:16.465829+0000","last_unstale":"2024-04-06T13:29:18.031619+0000","last_undegraded":"2024-04-06T13:29:18.031619+0000","last_fullsized":"2024-04-06T13:29:18.031619+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:16:21.330544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870066+0000","last_change":"2024-04-06T13:29:16.870112+0000","last_active":"2024-04-06T13:29:17.870066+0000","last_peered":"2024-04-06T13:29:17.870066+0000","last_clean":"2024-04-06T13:29:17.870066+0000","last_became_active":"2024-04-06T13:29:16.869840+0000","last_became_peered":"2024-04-06T13:29:16.869840+0000","last_unstale":"2024-04-06T13:29:17.870066+0000","last_undegraded":"2024-04-06T13:29:17.870066+0000","last_fullsized":"2024-04-06T13:29:17.870066+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:33:43.850720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869778+0000","last_change":"2024-04-06T13:29:16.468871+0000","last_active":"2024-04-06T13:29:17.869778+0000","last_peered":"2024-04-06T13:29:17.869778+0000","last_clean":"2024-04-06T13:29:17.869778+0000","last_became_active":"2024-04-06T13:29:16.468455+0000","last_became_peered":"2024-04-06T13:29:16.468455+0000","last_unstale":"2024-04-06T13:29:17.869778+0000","last_undegraded":"2024-04-06T13:29:17.869778+0000","last_fullsized":"2024-04-06T13:29:17.869778+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:53:38.647912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869479+0000","last_change":"2024-04-06T13:29:16.878934+0000","last_active":"2024-04-06T13:29:17.869479+0000","last_peered":"2024-04-06T13:29:17.869479+0000","last_clean":"2024-04-06T13:29:17.869479+0000","last_became_active":"2024-04-06T13:29:16.878688+0000","last_became_peered":"2024-04-06T13:29:16.878688+0000","last_unstale":"2024-04-06T13:29:17.869479+0000","last_undegraded":"2024-04-06T13:29:17.869479+0000","last_fullsized":"2024-04-06T13:29:17.869479+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:34:29.517268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"191'3","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869910+0000","last_change":"2024-04-06T13:29:16.457567+0000","last_active":"2024-04-06T13:29:17.869910+0000","last_peered":"2024-04-06T13:29:17.869910+0000","last_clean":"2024-04-06T13:29:17.869910+0000","last_became_active":"2024-04-06T13:29:16.457261+0000","last_became_peered":"2024-04-06T13:29:16.457261+0000","last_unstale":"2024-04-06T13:29:17.869910+0000","last_undegraded":"2024-04-06T13:29:17.869910+0000","last_fullsized":"2024-04-06T13:29:17.869910+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:54:16.663929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.515071+0000","last_change":"2024-04-06T13:29:16.877435+0000","last_active":"2024-04-06T13:29:18.515071+0000","last_peered":"2024-04-06T13:29:18.515071+0000","last_clean":"2024-04-06T13:29:18.515071+0000","last_became_active":"2024-04-06T13:29:16.877233+0000","last_became_peered":"2024-04-06T13:29:16.877233+0000","last_unstale":"2024-04-06T13:29:18.515071+0000","last_undegraded":"2024-04-06T13:29:18.515071+0000","last_fullsized":"2024-04-06T13:29:18.515071+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:03:22.503093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"191'6","reported_seq":17,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869585+0000","last_change":"2024-04-06T13:29:16.463484+0000","last_active":"2024-04-06T13:29:17.869585+0000","last_peered":"2024-04-06T13:29:17.869585+0000","last_clean":"2024-04-06T13:29:17.869585+0000","last_became_active":"2024-04-06T13:29:16.463181+0000","last_became_peered":"2024-04-06T13:29:16.463181+0000","last_unstale":"2024-04-06T13:29:17.869585+0000","last_undegraded":"2024-04-06T13:29:17.869585+0000","last_fullsized":"2024-04-06T13:29:17.869585+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:31:50.119403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015989+0000","last_change":"2024-04-06T13:29:16.875897+0000","last_active":"2024-04-06T13:29:19.015989+0000","last_peered":"2024-04-06T13:29:19.015989+0000","last_clean":"2024-04-06T13:29:19.015989+0000","last_became_active":"2024-04-06T13:29:16.875619+0000","last_became_peered":"2024-04-06T13:29:16.875619+0000","last_unstale":"2024-04-06T13:29:19.015989+0000","last_undegraded":"2024-04-06T13:29:19.015989+0000","last_fullsized":"2024-04-06T13:29:19.015989+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:46.759410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"191'2","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015626+0000","last_change":"2024-04-06T13:29:16.463527+0000","last_active":"2024-04-06T13:29:19.015626+0000","last_peered":"2024-04-06T13:29:19.015626+0000","last_clean":"2024-04-06T13:29:19.015626+0000","last_became_active":"2024-04-06T13:29:16.463231+0000","last_became_peered":"2024-04-06T13:29:16.463231+0000","last_unstale":"2024-04-06T13:29:19.015626+0000","last_undegraded":"2024-04-06T13:29:19.015626+0000","last_fullsized":"2024-04-06T13:29:19.015626+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:24:10.368926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031979+0000","last_change":"2024-04-06T13:29:16.884167+0000","last_active":"2024-04-06T13:29:18.031979+0000","last_peered":"2024-04-06T13:29:18.031979+0000","last_clean":"2024-04-06T13:29:18.031979+0000","last_became_active":"2024-04-06T13:29:16.883539+0000","last_became_peered":"2024-04-06T13:29:16.883539+0000","last_unstale":"2024-04-06T13:29:18.031979+0000","last_undegraded":"2024-04-06T13:29:18.031979+0000","last_fullsized":"2024-04-06T13:29:18.031979+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:31:46.993248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869445+0000","last_change":"2024-04-06T13:29:16.455567+0000","last_active":"2024-04-06T13:29:17.869445+0000","last_peered":"2024-04-06T13:29:17.869445+0000","last_clean":"2024-04-06T13:29:17.869445+0000","last_became_active":"2024-04-06T13:29:16.455280+0000","last_became_peered":"2024-04-06T13:29:16.455280+0000","last_unstale":"2024-04-06T13:29:17.869445+0000","last_undegraded":"2024-04-06T13:29:17.869445+0000","last_fullsized":"2024-04-06T13:29:17.869445+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:40:50.376953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015609+0000","last_change":"2024-04-06T13:29:16.877580+0000","last_active":"2024-04-06T13:29:19.015609+0000","last_peered":"2024-04-06T13:29:19.015609+0000","last_clean":"2024-04-06T13:29:19.015609+0000","last_became_active":"2024-04-06T13:29:16.877327+0000","last_became_peered":"2024-04-06T13:29:16.877327+0000","last_unstale":"2024-04-06T13:29:19.015609+0000","last_undegraded":"2024-04-06T13:29:19.015609+0000","last_fullsized":"2024-04-06T13:29:19.015609+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:17:44.072132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015631+0000","last_change":"2024-04-06T13:29:16.467262+0000","last_active":"2024-04-06T13:29:19.015631+0000","last_peered":"2024-04-06T13:29:19.015631+0000","last_clean":"2024-04-06T13:29:19.015631+0000","last_became_active":"2024-04-06T13:29:16.467036+0000","last_became_peered":"2024-04-06T13:29:16.467036+0000","last_unstale":"2024-04-06T13:29:19.015631+0000","last_undegraded":"2024-04-06T13:29:19.015631+0000","last_fullsized":"2024-04-06T13:29:19.015631+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:18:43.910720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.515126+0000","last_change":"2024-04-06T13:29:16.892789+0000","last_active":"2024-04-06T13:29:18.515126+0000","last_peered":"2024-04-06T13:29:18.515126+0000","last_clean":"2024-04-06T13:29:18.515126+0000","last_became_active":"2024-04-06T13:29:16.892524+0000","last_became_peered":"2024-04-06T13:29:16.892524+0000","last_unstale":"2024-04-06T13:29:18.515126+0000","last_undegraded":"2024-04-06T13:29:18.515126+0000","last_fullsized":"2024-04-06T13:29:18.515126+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:45:18.965004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"191'8","reported_seq":19,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031716+0000","last_change":"2024-04-06T13:29:16.466074+0000","last_active":"2024-04-06T13:29:18.031716+0000","last_peered":"2024-04-06T13:29:18.031716+0000","last_clean":"2024-04-06T13:29:18.031716+0000","last_became_active":"2024-04-06T13:29:16.465473+0000","last_became_peered":"2024-04-06T13:29:16.465473+0000","last_unstale":"2024-04-06T13:29:18.031716+0000","last_undegraded":"2024-04-06T13:29:18.031716+0000","last_fullsized":"2024-04-06T13:29:18.031716+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:00:46.031669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031733+0000","last_change":"2024-04-06T13:29:16.886226+0000","last_active":"2024-04-06T13:29:18.031733+0000","last_peered":"2024-04-06T13:29:18.031733+0000","last_clean":"2024-04-06T13:29:18.031733+0000","last_became_active":"2024-04-06T13:29:16.885464+0000","last_became_peered":"2024-04-06T13:29:16.885464+0000","last_unstale":"2024-04-06T13:29:18.031733+0000","last_undegraded":"2024-04-06T13:29:18.031733+0000","last_fullsized":"2024-04-06T13:29:18.031733+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:40:59.094743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"191'1","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869840+0000","last_change":"2024-04-06T13:29:16.457609+0000","last_active":"2024-04-06T13:29:17.869840+0000","last_peered":"2024-04-06T13:29:17.869840+0000","last_clean":"2024-04-06T13:29:17.869840+0000","last_became_active":"2024-04-06T13:29:16.457227+0000","last_became_peered":"2024-04-06T13:29:16.457227+0000","last_unstale":"2024-04-06T13:29:17.869840+0000","last_undegraded":"2024-04-06T13:29:17.869840+0000","last_fullsized":"2024-04-06T13:29:17.869840+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:39:30.175672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015942+0000","last_change":"2024-04-06T13:29:16.886090+0000","last_active":"2024-04-06T13:29:19.015942+0000","last_peered":"2024-04-06T13:29:19.015942+0000","last_clean":"2024-04-06T13:29:19.015942+0000","last_became_active":"2024-04-06T13:29:16.885825+0000","last_became_peered":"2024-04-06T13:29:16.885825+0000","last_unstale":"2024-04-06T13:29:19.015942+0000","last_undegraded":"2024-04-06T13:29:19.015942+0000","last_fullsized":"2024-04-06T13:29:19.015942+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:18:44.916349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"191'2","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031967+0000","last_change":"2024-04-06T13:29:16.465514+0000","last_active":"2024-04-06T13:29:18.031967+0000","last_peered":"2024-04-06T13:29:18.031967+0000","last_clean":"2024-04-06T13:29:18.031967+0000","last_became_active":"2024-04-06T13:29:16.462836+0000","last_became_peered":"2024-04-06T13:29:16.462836+0000","last_unstale":"2024-04-06T13:29:18.031967+0000","last_undegraded":"2024-04-06T13:29:18.031967+0000","last_fullsized":"2024-04-06T13:29:18.031967+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:25:06.167257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.016200+0000","last_change":"2024-04-06T13:29:16.887037+0000","last_active":"2024-04-06T13:29:19.016200+0000","last_peered":"2024-04-06T13:29:19.016200+0000","last_clean":"2024-04-06T13:29:19.016200+0000","last_became_active":"2024-04-06T13:29:16.886767+0000","last_became_peered":"2024-04-06T13:29:16.886767+0000","last_unstale":"2024-04-06T13:29:19.016200+0000","last_undegraded":"2024-04-06T13:29:19.016200+0000","last_fullsized":"2024-04-06T13:29:19.016200+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:40:41.556933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"191'1","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015900+0000","last_change":"2024-04-06T13:29:16.460961+0000","last_active":"2024-04-06T13:29:19.015900+0000","last_peered":"2024-04-06T13:29:19.015900+0000","last_clean":"2024-04-06T13:29:19.015900+0000","last_became_active":"2024-04-06T13:29:16.460609+0000","last_became_peered":"2024-04-06T13:29:16.460609+0000","last_unstale":"2024-04-06T13:29:19.015900+0000","last_undegraded":"2024-04-06T13:29:19.015900+0000","last_fullsized":"2024-04-06T13:29:19.015900+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:04:44.686319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015661+0000","last_change":"2024-04-06T13:29:16.880697+0000","last_active":"2024-04-06T13:29:19.015661+0000","last_peered":"2024-04-06T13:29:19.015661+0000","last_clean":"2024-04-06T13:29:19.015661+0000","last_became_active":"2024-04-06T13:29:16.880383+0000","last_became_peered":"2024-04-06T13:29:16.880383+0000","last_unstale":"2024-04-06T13:29:19.015661+0000","last_undegraded":"2024-04-06T13:29:19.015661+0000","last_fullsized":"2024-04-06T13:29:19.015661+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:54:17.669558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031900+0000","last_change":"2024-04-06T13:29:16.466282+0000","last_active":"2024-04-06T13:29:18.031900+0000","last_peered":"2024-04-06T13:29:18.031900+0000","last_clean":"2024-04-06T13:29:18.031900+0000","last_became_active":"2024-04-06T13:29:16.466042+0000","last_became_peered":"2024-04-06T13:29:16.466042+0000","last_unstale":"2024-04-06T13:29:18.031900+0000","last_undegraded":"2024-04-06T13:29:18.031900+0000","last_fullsized":"2024-04-06T13:29:18.031900+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:22:01.776384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869538+0000","last_change":"2024-04-06T13:29:16.881141+0000","last_active":"2024-04-06T13:29:17.869538+0000","last_peered":"2024-04-06T13:29:17.869538+0000","last_clean":"2024-04-06T13:29:17.869538+0000","last_became_active":"2024-04-06T13:29:16.880905+0000","last_became_peered":"2024-04-06T13:29:16.880905+0000","last_unstale":"2024-04-06T13:29:17.869538+0000","last_undegraded":"2024-04-06T13:29:17.869538+0000","last_fullsized":"2024-04-06T13:29:17.869538+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:58:28.333504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"191'2","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015651+0000","last_change":"2024-04-06T13:29:16.460100+0000","last_active":"2024-04-06T13:29:19.015651+0000","last_peered":"2024-04-06T13:29:19.015651+0000","last_clean":"2024-04-06T13:29:19.015651+0000","last_became_active":"2024-04-06T13:29:16.459855+0000","last_became_peered":"2024-04-06T13:29:16.459855+0000","last_unstale":"2024-04-06T13:29:19.015651+0000","last_undegraded":"2024-04-06T13:29:19.015651+0000","last_fullsized":"2024-04-06T13:29:19.015651+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:22:23.627784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015831+0000","last_change":"2024-04-06T13:29:16.890376+0000","last_active":"2024-04-06T13:29:19.015831+0000","last_peered":"2024-04-06T13:29:19.015831+0000","last_clean":"2024-04-06T13:29:19.015831+0000","last_became_active":"2024-04-06T13:29:16.890117+0000","last_became_peered":"2024-04-06T13:29:16.890117+0000","last_unstale":"2024-04-06T13:29:19.015831+0000","last_undegraded":"2024-04-06T13:29:19.015831+0000","last_fullsized":"2024-04-06T13:29:19.015831+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:39:49.010777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.016173+0000","last_change":"2024-04-06T13:29:16.457256+0000","last_active":"2024-04-06T13:29:19.016173+0000","last_peered":"2024-04-06T13:29:19.016173+0000","last_clean":"2024-04-06T13:29:19.016173+0000","last_became_active":"2024-04-06T13:29:16.457050+0000","last_became_peered":"2024-04-06T13:29:16.457050+0000","last_unstale":"2024-04-06T13:29:19.016173+0000","last_undegraded":"2024-04-06T13:29:19.016173+0000","last_fullsized":"2024-04-06T13:29:19.016173+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:50:15.874342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015746+0000","last_change":"2024-04-06T13:29:16.892221+0000","last_active":"2024-04-06T13:29:19.015746+0000","last_peered":"2024-04-06T13:29:19.015746+0000","last_clean":"2024-04-06T13:29:19.015746+0000","last_became_active":"2024-04-06T13:29:16.891987+0000","last_became_peered":"2024-04-06T13:29:16.891987+0000","last_unstale":"2024-04-06T13:29:19.015746+0000","last_undegraded":"2024-04-06T13:29:19.015746+0000","last_fullsized":"2024-04-06T13:29:19.015746+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:22:33.381500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.514961+0000","last_change":"2024-04-06T13:29:16.459264+0000","last_active":"2024-04-06T13:29:18.514961+0000","last_peered":"2024-04-06T13:29:18.514961+0000","last_clean":"2024-04-06T13:29:18.514961+0000","last_became_active":"2024-04-06T13:29:16.458974+0000","last_became_peered":"2024-04-06T13:29:16.458974+0000","last_unstale":"2024-04-06T13:29:18.514961+0000","last_undegraded":"2024-04-06T13:29:18.514961+0000","last_fullsized":"2024-04-06T13:29:18.514961+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:02:10.950359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870850+0000","last_change":"2024-04-06T13:29:16.871003+0000","last_active":"2024-04-06T13:29:17.870850+0000","last_peered":"2024-04-06T13:29:17.870850+0000","last_clean":"2024-04-06T13:29:17.870850+0000","last_became_active":"2024-04-06T13:29:16.870709+0000","last_became_peered":"2024-04-06T13:29:16.870709+0000","last_unstale":"2024-04-06T13:29:17.870850+0000","last_undegraded":"2024-04-06T13:29:17.870850+0000","last_fullsized":"2024-04-06T13:29:17.870850+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:29:16.732050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869973+0000","last_change":"2024-04-06T13:29:16.455130+0000","last_active":"2024-04-06T13:29:17.869973+0000","last_peered":"2024-04-06T13:29:17.869973+0000","last_clean":"2024-04-06T13:29:17.869973+0000","last_became_active":"2024-04-06T13:29:16.454892+0000","last_became_peered":"2024-04-06T13:29:16.454892+0000","last_unstale":"2024-04-06T13:29:17.869973+0000","last_undegraded":"2024-04-06T13:29:17.869973+0000","last_fullsized":"2024-04-06T13:29:17.869973+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:31.726957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015502+0000","last_change":"2024-04-06T13:29:16.874944+0000","last_active":"2024-04-06T13:29:19.015502+0000","last_peered":"2024-04-06T13:29:19.015502+0000","last_clean":"2024-04-06T13:29:19.015502+0000","last_became_active":"2024-04-06T13:29:16.874618+0000","last_became_peered":"2024-04-06T13:29:16.874618+0000","last_unstale":"2024-04-06T13:29:19.015502+0000","last_undegraded":"2024-04-06T13:29:19.015502+0000","last_fullsized":"2024-04-06T13:29:19.015502+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:28:59.228532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"191'2","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015860+0000","last_change":"2024-04-06T13:29:16.461497+0000","last_active":"2024-04-06T13:29:19.015860+0000","last_peered":"2024-04-06T13:29:19.015860+0000","last_clean":"2024-04-06T13:29:19.015860+0000","last_became_active":"2024-04-06T13:29:16.460573+0000","last_became_peered":"2024-04-06T13:29:16.460573+0000","last_unstale":"2024-04-06T13:29:19.015860+0000","last_undegraded":"2024-04-06T13:29:19.015860+0000","last_fullsized":"2024-04-06T13:29:19.015860+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:39:40.540329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031748+0000","last_change":"2024-04-06T13:29:16.883616+0000","last_active":"2024-04-06T13:29:18.031748+0000","last_peered":"2024-04-06T13:29:18.031748+0000","last_clean":"2024-04-06T13:29:18.031748+0000","last_became_active":"2024-04-06T13:29:16.883357+0000","last_became_peered":"2024-04-06T13:29:16.883357+0000","last_unstale":"2024-04-06T13:29:18.031748+0000","last_undegraded":"2024-04-06T13:29:18.031748+0000","last_fullsized":"2024-04-06T13:29:18.031748+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:39:21.730370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869753+0000","last_change":"2024-04-06T13:29:16.469434+0000","last_active":"2024-04-06T13:29:17.869753+0000","last_peered":"2024-04-06T13:29:17.869753+0000","last_clean":"2024-04-06T13:29:17.869753+0000","last_became_active":"2024-04-06T13:29:16.468793+0000","last_became_peered":"2024-04-06T13:29:16.468793+0000","last_unstale":"2024-04-06T13:29:17.869753+0000","last_undegraded":"2024-04-06T13:29:17.869753+0000","last_fullsized":"2024-04-06T13:29:17.869753+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:30:45.086967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.514875+0000","last_change":"2024-04-06T13:29:16.873996+0000","last_active":"2024-04-06T13:29:18.514875+0000","last_peered":"2024-04-06T13:29:18.514875+0000","last_clean":"2024-04-06T13:29:18.514875+0000","last_became_active":"2024-04-06T13:29:16.873773+0000","last_became_peered":"2024-04-06T13:29:16.873773+0000","last_unstale":"2024-04-06T13:29:18.514875+0000","last_undegraded":"2024-04-06T13:29:18.514875+0000","last_fullsized":"2024-04-06T13:29:18.514875+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:20:07.574362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869666+0000","last_change":"2024-04-06T13:29:16.456790+0000","last_active":"2024-04-06T13:29:17.869666+0000","last_peered":"2024-04-06T13:29:17.869666+0000","last_clean":"2024-04-06T13:29:17.869666+0000","last_became_active":"2024-04-06T13:29:16.456544+0000","last_became_peered":"2024-04-06T13:29:16.456544+0000","last_unstale":"2024-04-06T13:29:17.869666+0000","last_undegraded":"2024-04-06T13:29:17.869666+0000","last_fullsized":"2024-04-06T13:29:17.869666+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:42:08.635751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.031715+0000","last_change":"2024-04-06T13:29:16.884104+0000","last_active":"2024-04-06T13:29:18.031715+0000","last_peered":"2024-04-06T13:29:18.031715+0000","last_clean":"2024-04-06T13:29:18.031715+0000","last_became_active":"2024-04-06T13:29:16.883871+0000","last_became_peered":"2024-04-06T13:29:16.883871+0000","last_unstale":"2024-04-06T13:29:18.031715+0000","last_undegraded":"2024-04-06T13:29:18.031715+0000","last_fullsized":"2024-04-06T13:29:18.031715+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:43:19.361499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.514984+0000","last_change":"2024-04-06T13:29:16.463035+0000","last_active":"2024-04-06T13:29:18.514984+0000","last_peered":"2024-04-06T13:29:18.514984+0000","last_clean":"2024-04-06T13:29:18.514984+0000","last_became_active":"2024-04-06T13:29:16.462760+0000","last_became_peered":"2024-04-06T13:29:16.462760+0000","last_unstale":"2024-04-06T13:29:18.514984+0000","last_undegraded":"2024-04-06T13:29:18.514984+0000","last_fullsized":"2024-04-06T13:29:18.514984+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:35:44.802474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869754+0000","last_change":"2024-04-06T13:29:16.888214+0000","last_active":"2024-04-06T13:29:17.869754+0000","last_peered":"2024-04-06T13:29:17.869754+0000","last_clean":"2024-04-06T13:29:17.869754+0000","last_became_active":"2024-04-06T13:29:16.887759+0000","last_became_peered":"2024-04-06T13:29:16.887759+0000","last_unstale":"2024-04-06T13:29:17.869754+0000","last_undegraded":"2024-04-06T13:29:17.869754+0000","last_fullsized":"2024-04-06T13:29:17.869754+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:17:42.492434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869875+0000","last_change":"2024-04-06T13:29:16.465013+0000","last_active":"2024-04-06T13:29:17.869875+0000","last_peered":"2024-04-06T13:29:17.869875+0000","last_clean":"2024-04-06T13:29:17.869875+0000","last_became_active":"2024-04-06T13:29:16.464805+0000","last_became_peered":"2024-04-06T13:29:16.464805+0000","last_unstale":"2024-04-06T13:29:17.869875+0000","last_undegraded":"2024-04-06T13:29:17.869875+0000","last_fullsized":"2024-04-06T13:29:17.869875+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:25:06.167257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.870795+0000","last_change":"2024-04-06T13:29:16.875804+0000","last_active":"2024-04-06T13:29:17.870795+0000","last_peered":"2024-04-06T13:29:17.870795+0000","last_clean":"2024-04-06T13:29:17.870795+0000","last_became_active":"2024-04-06T13:29:16.874222+0000","last_became_peered":"2024-04-06T13:29:16.874222+0000","last_unstale":"2024-04-06T13:29:17.870795+0000","last_undegraded":"2024-04-06T13:29:17.870795+0000","last_fullsized":"2024-04-06T13:29:17.870795+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:45:02.533582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869532+0000","last_change":"2024-04-06T13:29:16.459589+0000","last_active":"2024-04-06T13:29:17.869532+0000","last_peered":"2024-04-06T13:29:17.869532+0000","last_clean":"2024-04-06T13:29:17.869532+0000","last_became_active":"2024-04-06T13:29:16.459340+0000","last_became_peered":"2024-04-06T13:29:16.459340+0000","last_unstale":"2024-04-06T13:29:17.869532+0000","last_undegraded":"2024-04-06T13:29:17.869532+0000","last_fullsized":"2024-04-06T13:29:17.869532+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:51:12.830032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869678+0000","last_change":"2024-04-06T13:29:16.878926+0000","last_active":"2024-04-06T13:29:17.869678+0000","last_peered":"2024-04-06T13:29:17.869678+0000","last_clean":"2024-04-06T13:29:17.869678+0000","last_became_active":"2024-04-06T13:29:16.878659+0000","last_became_peered":"2024-04-06T13:29:16.878659+0000","last_unstale":"2024-04-06T13:29:17.869678+0000","last_undegraded":"2024-04-06T13:29:17.869678+0000","last_fullsized":"2024-04-06T13:29:17.869678+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:57:16.557906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015858+0000","last_change":"2024-04-06T13:29:16.460240+0000","last_active":"2024-04-06T13:29:19.015858+0000","last_peered":"2024-04-06T13:29:19.015858+0000","last_clean":"2024-04-06T13:29:19.015858+0000","last_became_active":"2024-04-06T13:29:16.459981+0000","last_became_peered":"2024-04-06T13:29:16.459981+0000","last_unstale":"2024-04-06T13:29:19.015858+0000","last_undegraded":"2024-04-06T13:29:19.015858+0000","last_fullsized":"2024-04-06T13:29:19.015858+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:08:23.091984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:18.515169+0000","last_change":"2024-04-06T13:29:16.876422+0000","last_active":"2024-04-06T13:29:18.515169+0000","last_peered":"2024-04-06T13:29:18.515169+0000","last_clean":"2024-04-06T13:29:18.515169+0000","last_became_active":"2024-04-06T13:29:16.876144+0000","last_became_peered":"2024-04-06T13:29:16.876144+0000","last_unstale":"2024-04-06T13:29:18.515169+0000","last_undegraded":"2024-04-06T13:29:18.515169+0000","last_fullsized":"2024-04-06T13:29:18.515169+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:47:10.159438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:19.015347+0000","last_change":"2024-04-06T13:29:16.461121+0000","last_active":"2024-04-06T13:29:19.015347+0000","last_peered":"2024-04-06T13:29:19.015347+0000","last_clean":"2024-04-06T13:29:19.015347+0000","last_became_active":"2024-04-06T13:29:16.460542+0000","last_became_peered":"2024-04-06T13:29:16.460542+0000","last_unstale":"2024-04-06T13:29:19.015347+0000","last_undegraded":"2024-04-06T13:29:19.015347+0000","last_fullsized":"2024-04-06T13:29:19.015347+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:33:57.575734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869473+0000","last_change":"2024-04-06T13:29:16.875390+0000","last_active":"2024-04-06T13:29:17.869473+0000","last_peered":"2024-04-06T13:29:17.869473+0000","last_clean":"2024-04-06T13:29:17.869473+0000","last_became_active":"2024-04-06T13:29:16.875010+0000","last_became_peered":"2024-04-06T13:29:16.875010+0000","last_unstale":"2024-04-06T13:29:17.869473+0000","last_undegraded":"2024-04-06T13:29:17.869473+0000","last_fullsized":"2024-04-06T13:29:17.869473+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:16.440383+0000","last_clean_scrub_stamp":"2024-04-06T13:29:16.440383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:44:33.412863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-06T13:29:17.869691+0000","last_change":"2024-04-06T13:29:16.469239+0000","last_active":"2024-04-06T13:29:17.869691+0000","last_peered":"2024-04-06T13:29:17.869691+0000","last_clean":"2024-04-06T13:29:17.869691+0000","last_became_active":"2024-04-06T13:29:16.468944+0000","last_became_peered":"2024-04-06T13:29:16.468944+0000","last_unstale":"2024-04-06T13:29:17.869691+0000","last_undegraded":"2024-04-06T13:29:17.869691+0000","last_fullsized":"2024-04-06T13:29:17.869691+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:29:15.434754+0000","last_clean_scrub_stamp":"2024-04-06T13:29:15.434754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:59.794032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"181'293","reported_seq":998,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-04-06T13:29:16.864144+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:29:16.864144+0000","last_peered":"2024-04-06T13:29:16.864144+0000","last_clean":"2024-04-06T13:29:16.864144+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:29:16.864144+0000","last_undegraded":"2024-04-06T13:29:16.864144+0000","last_fullsized":"2024-04-06T13:29:16.864144+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575769,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52100,"kb_used_data":2928,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319740,"statfs":{"total":96636764160,"available":96583413760,"internally_reserved":0,"allocated":2998272,"data_stored":1806605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.58799999999999997,"15min":0.59299999999999997},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.78200000000000003,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59599999999999997,"15min":0.60099999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.35899999999999999},"max":{"1min":0.78500000000000003,"5min":0.86099999999999999,"15min":1.4790000000000001},"last":0.48399999999999999}]},{"osd":1,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.60799999999999998,"15min":0.60999999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.79500000000000004,"5min":0.82499999999999996,"15min":0.83299999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.40300000000000002,"5min":0.39300000000000002,"15min":0.39000000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":1.5249999999999999},"last":0.67600000000000005}]},{"osd":2,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59099999999999997,"15min":0.58199999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59699999999999998,"15min":0.61099999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":1.198},"last":0.61099999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.60699999999999998,"15min":0.60799999999999998},"min":{"1min":0.437,"5min":0.36499999999999999,"15min":0.35499999999999998},"max":{"1min":0.77500000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61599999999999999,"15min":0.61199999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.78600000000000003,"5min":0.79600000000000004,"15min":1.1200000000000001},"last":0.66000000000000003}]},{"osd":4,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45400000000000001,"15min":0.45200000000000001},"min":{"1min":0.29599999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.97399999999999998},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55800000000000005,"15min":0.54000000000000004},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.33600000000000002},"max":{"1min":0.72399999999999998,"5min":0.77600000000000002,"15min":0.78000000000000003},"last":0.64800000000000002}]},{"osd":5,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.52700000000000002,"15min":0.51500000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69599999999999995,"5min":0.80000000000000004,"15min":1.024},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.58499999999999996,"15min":0.57999999999999996},"min":{"1min":0.36899999999999999,"5min":0.35599999999999998,"15min":0.33000000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":1.046},"last":0.83499999999999996}]},{"osd":6,"last update":"Sat Apr 6 13:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56699999999999995,"15min":0.56100000000000005},"min":{"1min":0.41399999999999998,"5min":0.38800000000000001,"15min":0.35299999999999998},"max":{"1min":0.81100000000000005,"5min":0.82199999999999995,"15min":1.4159999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.59599999999999997,"15min":0.57899999999999996},"min":{"1min":0.35799999999999998,"5min":0.34599999999999997,"15min":0.33700000000000002},"max":{"1min":0.75600000000000001,"5min":0.80700000000000005,"15min":0.873},"last":0.70199999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575768,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37552,"kb_used_data":2332,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334288,"statfs":{"total":96636764160,"available":96598310912,"internally_reserved":0,"allocated":2387968,"data_stored":1208554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.55700000000000005,"15min":0.56000000000000005},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.65900000000000003,"5min":0.79700000000000004,"15min":0.83499999999999996},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.75800000000000001,"5min":0.76200000000000001,"15min":0.78800000000000003},"last":0.54100000000000004}]},{"osd":1,"last update":"Sat Apr 6 13:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.60299999999999998,"15min":0.61799999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.628,"15min":0.63700000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.311},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.505}]},{"osd":2,"last update":"Sat Apr 6 13:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.58199999999999996,"15min":0.58999999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.748,"5min":0.80700000000000005,"15min":0.84499999999999997},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.58399999999999996,"15min":0.59299999999999997},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.56100000000000005}]},{"osd":3,"last update":"Sat Apr 6 13:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.60999999999999999,"15min":0.61199999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79200000000000004,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.64000000000000001,"15min":0.65400000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.74199999999999999,"5min":0.83999999999999997,"15min":0.877},"last":0.57599999999999996}]},{"osd":4,"last update":"Sat Apr 6 13:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51800000000000002,"15min":0.52100000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.81000000000000005},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53600000000000003,"15min":0.54700000000000004},"min":{"1min":0.33100000000000002,"5min":0.29499999999999998,"15min":0.29299999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.873},"last":0.39000000000000001}]},{"osd":5,"last update":"Sat Apr 6 13:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.505,"15min":0.51600000000000001},"min":{"1min":0.34799999999999998,"5min":0.32600000000000001,"15min":0.314},"max":{"1min":0.78600000000000003,"5min":0.79200000000000004,"15min":0.82299999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52100000000000002,"15min":0.53500000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.42399999999999999}]},{"osd":7,"last update":"Sat Apr 6 13:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55200000000000005,"15min":0.56000000000000005},"min":{"1min":0.37,"5min":0.35299999999999998,"15min":0.34899999999999998},"max":{"1min":0.81399999999999995,"5min":0.83499999999999996,"15min":0.85599999999999998},"last":0.499},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58299999999999996,"15min":0.58999999999999997},"min":{"1min":0.41799999999999998,"5min":0.375,"15min":0.36499999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.60399999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575767,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37568,"kb_used_data":2356,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334272,"statfs":{"total":96636764160,"available":96598294528,"internally_reserved":0,"allocated":2412544,"data_stored":1216782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.39900000000000002,"15min":0.38800000000000001},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.50900000000000001,"5min":0.54800000000000004,"15min":0.64300000000000002},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.434,"15min":0.42899999999999999},"min":{"1min":0.28799999999999998,"5min":0.27000000000000002,"15min":0.26700000000000002},"max":{"1min":0.56100000000000005,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.505}]},{"osd":2,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.42999999999999999,"15min":0.42499999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.52400000000000002,"5min":0.64000000000000001,"15min":0.67400000000000004},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.47699999999999998,"15min":0.46500000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59399999999999997,"5min":0.70499999999999996,"15min":0.73299999999999998},"last":0.55000000000000004}]},{"osd":3,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.48499999999999999,"15min":0.47299999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.30599999999999999},"max":{"1min":0.64200000000000002,"5min":0.70899999999999996,"15min":0.77100000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.45600000000000002,"15min":0.44700000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.63200000000000001,"5min":0.65500000000000003,"15min":0.68000000000000005},"last":0.621}]},{"osd":4,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65900000000000003,"15min":0.64700000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.85799999999999998,"5min":0.92500000000000004,"15min":1.008},"last":0.877},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.66900000000000004,"15min":0.65900000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.93000000000000005}]},{"osd":5,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.67300000000000004,"15min":0.66200000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.371},"max":{"1min":0.93799999999999994,"5min":0.95499999999999996,"15min":0.98299999999999998},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.69099999999999995,"15min":0.67400000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91800000000000004,"5min":0.94199999999999995,"15min":1.0189999999999999},"last":0.90000000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.67600000000000005,"15min":0.66700000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.97199999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.69499999999999995,"15min":0.68500000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.877,"5min":1,"15min":1},"last":0.88900000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.71199999999999997,"15min":0.69499999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.89000000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68000000000000005,"15min":0.66900000000000004},"min":{"1min":0.48699999999999999,"5min":0.46899999999999997,"15min":0.44},"max":{"1min":0.91200000000000003,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.93899999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575767,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38140,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94333700,"statfs":{"total":96636764160,"available":96597708800,"internally_reserved":0,"allocated":2990080,"data_stored":1799026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39700000000000002,"15min":0.39800000000000002},"min":{"1min":0.27600000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.46700000000000003,"5min":0.58799999999999997,"15min":0.65000000000000002},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42599999999999999,"15min":0.42499999999999999},"min":{"1min":0.28499999999999998,"5min":0.28000000000000003,"15min":0.26400000000000001},"max":{"1min":0.57199999999999995,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.36599999999999999}]},{"osd":2,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.434,"15min":0.434},"min":{"1min":0.313,"5min":0.29699999999999999,"15min":0.27000000000000002},"max":{"1min":0.58699999999999997,"5min":0.67200000000000004,"15min":0.71199999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.45000000000000001,"15min":0.45300000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.51400000000000001,"5min":0.63,"15min":0.77000000000000002},"last":0.32700000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.436,"15min":0.435},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.27900000000000003},"max":{"1min":0.51100000000000001,"5min":0.61899999999999999,"15min":0.63200000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.39700000000000002,"5min":0.34399999999999997,"15min":0.307},"max":{"1min":0.64500000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.377}]},{"osd":4,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.67200000000000004,"15min":0.65400000000000003},"min":{"1min":0.47899999999999998,"5min":0.46899999999999997,"15min":0.45600000000000002},"max":{"1min":0.86299999999999999,"5min":0.93000000000000005,"15min":0.999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.67900000000000005,"15min":0.66800000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.41499999999999998},"max":{"1min":0.82499999999999996,"5min":0.96699999999999997,"15min":1.103},"last":0.53300000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.66900000000000004,"15min":0.65700000000000003},"min":{"1min":0.46700000000000003,"5min":0.46100000000000002,"15min":0.46000000000000002},"max":{"1min":0.77100000000000002,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70499999999999996,"15min":0.68400000000000005},"min":{"1min":0.497,"5min":0.48999999999999999,"15min":0.41299999999999998},"max":{"1min":0.89100000000000001,"5min":1.0169999999999999,"15min":1.0469999999999999},"last":0.52000000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.70399999999999996,"15min":0.68899999999999995},"min":{"1min":0.54000000000000004,"5min":0.49399999999999999,"15min":0.46000000000000002},"max":{"1min":0.91100000000000003,"5min":1.0629999999999999,"15min":1.0920000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.70399999999999996,"15min":0.68799999999999994},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.46000000000000002},"max":{"1min":0.871,"5min":0.92900000000000005,"15min":0.98599999999999999},"last":0.56499999999999995}]},{"osd":7,"last update":"Sat Apr 6 13:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.71999999999999997,"15min":0.70299999999999996},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.46400000000000002},"max":{"1min":0.81399999999999995,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.71499999999999997,"15min":0.69299999999999995},"min":{"1min":0.502,"5min":0.48899999999999999,"15min":0.438},"max":{"1min":0.92400000000000004,"5min":1.042,"15min":1.071},"last":0.57199999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575769,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59708,"kb_used_data":2344,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312132,"statfs":{"total":96636764160,"available":96575623168,"internally_reserved":0,"allocated":2400256,"data_stored":1215218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.39400000000000002,"15min":0.39300000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41599999999999998,"15min":0.42899999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.68700000000000006},"last":0.315}]},{"osd":1,"last update":"Sat Apr 6 13:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.42499999999999999,"15min":0.41899999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.6190000000000002,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":0.378},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":0.41399999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.49399999999999999,"15min":0.48199999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":3.9980000000000002,"5min":3.9980000000000002,"15min":3.9980000000000002},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.47399999999999998,"15min":0.46300000000000002},"min":{"1min":0.33500000000000002,"5min":0.33300000000000002,"15min":0.29699999999999999},"max":{"1min":4.0350000000000001,"5min":4.0350000000000001,"15min":4.0350000000000001},"last":0.433}]},{"osd":4,"last update":"Sat Apr 6 13:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.70099999999999996,"15min":0.66800000000000004},"min":{"1min":0.42099999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":6.335,"5min":6.335,"15min":6.335},"last":0.501},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.70699999999999996,"15min":0.67700000000000005},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.434},"max":{"1min":5.2610000000000001,"5min":5.2610000000000001,"15min":5.2610000000000001},"last":0.52100000000000002}]},{"osd":5,"last update":"Sat Apr 6 13:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.69599999999999995,"15min":0.68200000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":2.7599999999999998,"5min":2.7599999999999998,"15min":2.7599999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67000000000000004,"15min":0.66800000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.206,"5min":2.206,"15min":2.206},"last":0.53300000000000003}]},{"osd":6,"last update":"Sat Apr 6 13:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.66200000000000003,"15min":0.66700000000000004},"min":{"1min":0.42499999999999999,"5min":0.42199999999999999,"15min":0.40500000000000003},"max":{"1min":0.91200000000000003,"5min":0.93899999999999995,"15min":1.022},"last":0.621},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.69799999999999995,"15min":0.69899999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.41999999999999998},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":0.443}]},{"osd":7,"last update":"Sat Apr 6 13:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.70699999999999996,"15min":0.69399999999999995},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.438},"max":{"1min":2.0110000000000001,"5min":2.0110000000000001,"15min":2.0110000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72999999999999998,"15min":0.71799999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.43099999999999999},"max":{"1min":2.8090000000000002,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":0.51100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575768,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59724,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312116,"statfs":{"total":96636764160,"available":96575606784,"internally_reserved":0,"allocated":2424832,"data_stored":1218405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.39400000000000002,"15min":0.38700000000000001},"min":{"1min":0.27400000000000002,"5min":0.25700000000000001,"15min":0.248},"max":{"1min":0.54400000000000004,"5min":0.61199999999999999,"15min":0.621},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.41199999999999998,"15min":0.40500000000000003},"min":{"1min":0.27100000000000002,"5min":0.253,"15min":0.24099999999999999},"max":{"1min":0.51400000000000001,"5min":0.66000000000000003,"15min":0.67100000000000004},"last":0.28000000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.42099999999999999,"15min":0.41299999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.58999999999999997,"5min":0.64100000000000001,"15min":0.64900000000000002},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.443,"15min":0.441},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.56699999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.29399999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.5,"15min":0.49299999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.629,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.37},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.503,"15min":0.501},"min":{"1min":0.38,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.66200000000000003,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.38}]},{"osd":4,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65800000000000003,"15min":0.63600000000000001},"min":{"1min":0.49199999999999999,"5min":0.46400000000000002,"15min":0.34200000000000003},"max":{"1min":0.86599999999999999,"5min":0.88400000000000001,"15min":0.96799999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.66100000000000003,"15min":0.65700000000000003},"min":{"1min":0.47299999999999998,"5min":0.436,"15min":0.436},"max":{"1min":0.76200000000000001,"5min":0.97199999999999998,"15min":0.97299999999999998},"last":0.58199999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69199999999999995,"15min":0.68200000000000005},"min":{"1min":0.48299999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.89400000000000002,"5min":0.97699999999999998,"15min":1.024},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.71299999999999997,"15min":0.70099999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.41999999999999998},"max":{"1min":0.81399999999999995,"5min":0.98699999999999999,"15min":1.0069999999999999},"last":0.53300000000000003}]},{"osd":6,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.68700000000000006,"15min":0.67200000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45200000000000001},"max":{"1min":0.79500000000000004,"5min":1.0149999999999999,"15min":1.0569999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.69299999999999995,"15min":0.67900000000000005},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.38900000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.98899999999999999},"last":0.47799999999999998}]},{"osd":7,"last update":"Sat Apr 6 13:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.68600000000000005,"15min":0.68000000000000005},"min":{"1min":0.50800000000000001,"5min":0.47599999999999998,"15min":0.42199999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":1.0009999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.72299999999999998,"15min":0.70299999999999996},"min":{"1min":0.495,"5min":0.47999999999999998,"15min":0.46800000000000003},"max":{"1min":0.88,"5min":1.028,"15min":1.028},"last":0.56599999999999995}]}]},{"osd":4,"up_from":13,"seq":55834575768,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41672,"kb_used_data":2364,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330168,"statfs":{"total":96636764160,"available":96594092032,"internally_reserved":0,"allocated":2420736,"data_stored":1217001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59399999999999997,"15min":0.59999999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.745,"5min":0.78900000000000003,"15min":1.2609999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.45800000000000002,"5min":0.40699999999999997,"15min":0.376},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.52300000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64700000000000002,"15min":0.64900000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.35199999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.86699999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.64000000000000001},"min":{"1min":0.45100000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.78100000000000003,"5min":0.79300000000000004,"15min":0.82499999999999996},"last":0.40699999999999997}]},{"osd":2,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59599999999999997,"15min":0.60699999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.33600000000000002},"max":{"1min":0.81399999999999995,"5min":0.82399999999999995,"15min":1.5109999999999999},"last":0.375},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.622,"15min":0.623},"min":{"1min":0.436,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.43099999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.60399999999999998,"15min":0.60299999999999998},"min":{"1min":0.39100000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73399999999999999,"5min":0.78600000000000003,"15min":1.1439999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63300000000000001,"15min":0.63400000000000001},"min":{"1min":0.38800000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":1.319},"last":0.54200000000000004}]},{"osd":5,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48899999999999999,"15min":0.504},"min":{"1min":0.35199999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66900000000000004,"5min":0.748,"15min":0.878},"last":0.623},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51800000000000002,"15min":0.52200000000000002},"min":{"1min":0.38300000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":1.3839999999999999},"last":0.47099999999999997}]},{"osd":6,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.308},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":1.6850000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40500000000000003,"5min":0.39900000000000002,"15min":0.36699999999999999},"max":{"1min":0.755,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.59999999999999998}]},{"osd":7,"last update":"Sat Apr 6 13:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.374,"5min":0.36899999999999999,"15min":0.35099999999999998},"max":{"1min":0.79000000000000004,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.61399999999999999,"15min":0.60299999999999998},"min":{"1min":0.46999999999999997,"5min":0.41499999999999998,"15min":0.35799999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.7410000000000001},"last":0.61199999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575768,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59704,"kb_used_data":2340,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312136,"statfs":{"total":96636764160,"available":96575627264,"internally_reserved":0,"allocated":2396160,"data_stored":1208678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.755,"5min":0.79100000000000004,"15min":0.81299999999999994},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.64900000000000002,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.53100000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.59699999999999998,"15min":0.59099999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.58899999999999997,"15min":0.58999999999999997},"min":{"1min":0.38900000000000001,"5min":0.36899999999999999,"15min":0.30599999999999999},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.504}]},{"osd":2,"last update":"Sat Apr 6 13:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.60199999999999998,"15min":0.60099999999999998},"min":{"1min":0.40799999999999997,"5min":0.33700000000000002,"15min":0.33200000000000002},"max":{"1min":0.63200000000000001,"5min":0.80300000000000005,"15min":0.81999999999999995},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.59599999999999997,"15min":0.59299999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76600000000000001},"last":0.55800000000000005}]},{"osd":3,"last update":"Sat Apr 6 13:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.60699999999999998,"15min":0.60299999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35499999999999998},"max":{"1min":0.78400000000000003,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.628,"15min":0.622},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.745,"5min":0.82599999999999996,"15min":0.82899999999999996},"last":0.622}]},{"osd":4,"last update":"Sat Apr 6 13:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.505,"15min":0.501},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72299999999999998,"5min":0.72499999999999998,"15min":0.85199999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.56699999999999995,"15min":0.55400000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.29099999999999998},"max":{"1min":0.80100000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.68400000000000005}]},{"osd":6,"last update":"Sat Apr 6 13:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.51500000000000001,"15min":0.504},"min":{"1min":0.33100000000000002,"5min":0.32300000000000001,"15min":0.29699999999999999},"max":{"1min":0.68600000000000005,"5min":0.80000000000000004,"15min":0.82699999999999996},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.52200000000000002,"15min":0.50600000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.77600000000000002,"5min":0.77900000000000003,"15min":0.80300000000000005},"last":0.58799999999999997}]},{"osd":7,"last update":"Sat Apr 6 13:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54200000000000004,"15min":0.52700000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.314},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64500000000000002,"15min":0.621},"min":{"1min":0.441,"5min":0.42099999999999999,"15min":0.376},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":0.72399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:29:21.378 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-06T13:29:21.378 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-06T13:29:21.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.543+0000 7f70741ab700 1 Processor -- start 2024-04-06T13:29:21.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.544+0000 7f70741ab700 1 -- start start 2024-04-06T13:29:21.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.544+0000 7f70741ab700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f706c159b20 0x7f706c06b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:21.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.544+0000 7f70741ab700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f706c0cdc60 0x7f706c0ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:21.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.544+0000 7f70741ab700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706c06a8d0 con 0x7f706c06e470 2024-04-06T13:29:21.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.544+0000 7f70741ab700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f706c0ce650 con 0x7f706c159b20 2024-04-06T13:29:21.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.544+0000 7f70741ab700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f706c159f00 con 0x7f706c0cdc60 2024-04-06T13:29:21.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.545+0000 7f7071f47700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f706c159b20 0x7f706c06b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:21.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.545+0000 7f7072748700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f706c06e470 0x7f706c06e880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51982/0 (socket says 172.21.15.189:51982) 2024-04-06T13:29:21.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.545+0000 7f7072748700 1 -- 172.21.15.189:0/2602142043 learned_addr learned my addr 172.21.15.189:0/2602142043 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:21.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.545+0000 7f7071f47700 1 -- 172.21.15.189:0/2602142043 >> v2:172.21.15.192:3300/0 conn(0x7f706c0cdc60 msgr2=0x7f706c0ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:21.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.545+0000 7f7071746700 1 --2- 172.21.15.189:0/2602142043 >> v2:172.21.15.192:3300/0 conn(0x7f706c0cdc60 0x7f706c0ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:21.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.545+0000 7f7071f47700 1 --2- 172.21.15.189:0/2602142043 >> v2:172.21.15.192:3300/0 conn(0x7f706c0cdc60 0x7f706c0ce110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.545+0000 7f7071f47700 1 -- 172.21.15.189:0/2602142043 >> v1:172.21.15.189:6789/0 conn(0x7f706c06e470 legacy=0x7f706c06e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:21.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.545+0000 7f7071f47700 1 -- 172.21.15.189:0/2602142043 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c06abb0 con 0x7f706c159b20 2024-04-06T13:29:21.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.545+0000 7f7070f45700 1 -- 172.21.15.189:0/2602142043 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4182499863 0 0) 0x7f706c06a8d0 con 0x7f706c06e470 2024-04-06T13:29:21.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.545+0000 7f7071f47700 1 --2- 172.21.15.189:0/2602142043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f706c159b20 0x7f706c06b970 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f706c0ca010 tx=0x7f706800bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=643360f34179569 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:21.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.545+0000 7f7070f45700 1 -- 172.21.15.189:0/2602142043 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706800dbe0 con 0x7f706c159b20 2024-04-06T13:29:21.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.545+0000 7f7070f45700 1 -- 172.21.15.189:0/2602142043 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706800dd40 con 0x7f706c159b20 2024-04-06T13:29:21.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.545+0000 7f7070f45700 1 -- 172.21.15.189:0/2602142043 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7068017630 con 0x7f706c159b20 2024-04-06T13:29:21.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.546+0000 7f70741ab700 1 -- 172.21.15.189:0/2602142043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f706c159b20 msgr2=0x7f706c06b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:21.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.546+0000 7f70741ab700 1 --2- 172.21.15.189:0/2602142043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f706c159b20 0x7f706c06b970 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f706c0ca010 tx=0x7f706800bdf0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.546+0000 7f70741ab700 1 -- 172.21.15.189:0/2602142043 shutdown_connections 2024-04-06T13:29:21.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.546+0000 7f70741ab700 1 --2- 172.21.15.189:0/2602142043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f706c159b20 0x7f706c06b970 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.546+0000 7f70741ab700 1 --2- 172.21.15.189:0/2602142043 >> v2:172.21.15.192:3300/0 conn(0x7f706c0cdc60 0x7f706c0ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.546+0000 7f70741ab700 1 -- 172.21.15.189:0/2602142043 >> 172.21.15.189:0/2602142043 conn(0x7f706c0cc290 msgr2=0x7f706c0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:21.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.546+0000 7f70741ab700 1 -- 172.21.15.189:0/2602142043 shutdown_connections 2024-04-06T13:29:21.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.546+0000 7f70741ab700 1 -- 172.21.15.189:0/2602142043 wait complete. 2024-04-06T13:29:21.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.547+0000 7f70741ab700 1 Processor -- start 2024-04-06T13:29:21.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.547+0000 7f70741ab700 1 -- start start 2024-04-06T13:29:21.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.548+0000 7f70741ab700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f706c06e470 0x7f706c0c7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:21.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.548+0000 7f70741ab700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f706c0cdc60 0x7f706c158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:21.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.548+0000 7f70741ab700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f706c159b20 0x7f706c150e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:21.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.548+0000 7f70741ab700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706c158c40 con 0x7f706c159b20 2024-04-06T13:29:21.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.548+0000 7f70741ab700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706c0c7970 con 0x7f706c0cdc60 2024-04-06T13:29:21.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.548+0000 7f70741ab700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706c0c7c50 con 0x7f706c06e470 2024-04-06T13:29:21.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.548+0000 7f7072748700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f706c159b20 0x7f706c150e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:21.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.548+0000 7f7072748700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f706c159b20 0x7f706c150e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54288/0 (socket says 172.21.15.189:54288) 2024-04-06T13:29:21.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.548+0000 7f7072748700 1 -- 172.21.15.189:0/310154032 learned_addr learned my addr 172.21.15.189:0/310154032 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:21.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.548+0000 7f7071746700 1 --2- 172.21.15.189:0/310154032 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f706c0cdc60 0x7f706c158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:21.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.548+0000 7f7071f47700 1 --2- 172.21.15.189:0/310154032 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f706c06e470 0x7f706c0c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:21.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.548+0000 7f7072748700 1 -- 172.21.15.189:0/310154032 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f706c06e470 msgr2=0x7f706c0c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:21.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.548+0000 7f7072748700 1 --2- 172.21.15.189:0/310154032 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f706c06e470 0x7f706c0c7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.548+0000 7f7072748700 1 -- 172.21.15.189:0/310154032 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f706c0cdc60 msgr2=0x7f706c158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:21.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.548+0000 7f7072748700 1 --2- 172.21.15.189:0/310154032 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f706c0cdc60 0x7f706c158ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.548+0000 7f7072748700 1 -- 172.21.15.189:0/310154032 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706800b740 con 0x7f706c159b20 2024-04-06T13:29:21.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.548+0000 7f7072748700 1 --2- 172.21.15.189:0/310154032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f706c159b20 0x7f706c150e50 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f706400cca0 tx=0x7f706400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:21.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.548+0000 7f70627fc700 1 -- 172.21.15.189:0/310154032 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706400cea0 con 0x7f706c159b20 2024-04-06T13:29:21.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.548+0000 7f70627fc700 1 -- 172.21.15.189:0/310154032 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706400ade0 con 0x7f706c159b20 2024-04-06T13:29:21.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.549+0000 7f70741ab700 1 -- 172.21.15.189:0/310154032 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f706c0c7f30 con 0x7f706c159b20 2024-04-06T13:29:21.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.549+0000 7f70627fc700 1 -- 172.21.15.189:0/310154032 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7064007310 con 0x7f706c159b20 2024-04-06T13:29:21.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.549+0000 7f70741ab700 1 -- 172.21.15.189:0/310154032 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f706c0c84f0 con 0x7f706c159b20 2024-04-06T13:29:21.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.549+0000 7f70741ab700 1 -- 172.21.15.189:0/310154032 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f706c06b130 con 0x7f706c159b20 2024-04-06T13:29:21.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.550+0000 7f70627fc700 1 -- 172.21.15.189:0/310154032 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7064007490 con 0x7f706c159b20 2024-04-06T13:29:21.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.550+0000 7f70627fc700 1 --2- 172.21.15.189:0/310154032 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f705006adb0 0x7f705006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:21.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.550+0000 7f70627fc700 1 -- 172.21.15.189:0/310154032 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7064089520 con 0x7f706c159b20 2024-04-06T13:29:21.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.550+0000 7f7071f47700 1 --2- 172.21.15.189:0/310154032 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f705006adb0 0x7f705006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:21.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.551+0000 7f7071f47700 1 --2- 172.21.15.189:0/310154032 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f705006adb0 0x7f705006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f706800ba60 tx=0x7f7068014000 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:21.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.555+0000 7f70627fc700 1 -- 172.21.15.189:0/310154032 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7064054480 con 0x7f706c159b20 2024-04-06T13:29:21.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.830+0000 7f70741ab700 1 -- 172.21.15.189:0/310154032 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f706c074990 con 0x7f706c159b20 2024-04-06T13:29:21.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.834+0000 7f70627fc700 1 -- 172.21.15.189:0/310154032 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v44) v1 ==== 190+0+0 (secure 0 0 0) 0x7f7064057aa0 con 0x7f706c159b20 2024-04-06T13:29:21.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.836+0000 7f70741ab700 1 -- 172.21.15.189:0/310154032 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f705006adb0 msgr2=0x7f705006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:21.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.836+0000 7f70741ab700 1 --2- 172.21.15.189:0/310154032 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f705006adb0 0x7f705006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f706800ba60 tx=0x7f7068014000 comp rx=0 tx=0).stop 2024-04-06T13:29:21.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.836+0000 7f70741ab700 1 -- 172.21.15.189:0/310154032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f706c159b20 msgr2=0x7f706c150e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:21.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.836+0000 7f70741ab700 1 --2- 172.21.15.189:0/310154032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f706c159b20 0x7f706c150e50 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f706400cca0 tx=0x7f706400a340 comp rx=0 tx=0).stop 2024-04-06T13:29:21.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.836+0000 7f70741ab700 1 -- 172.21.15.189:0/310154032 shutdown_connections 2024-04-06T13:29:21.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.836+0000 7f70741ab700 1 --2- 172.21.15.189:0/310154032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f706c159b20 0x7f706c150e50 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.836+0000 7f70741ab700 1 --2- 172.21.15.189:0/310154032 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f706c06e470 0x7f706c0c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.836+0000 7f70741ab700 1 --2- 172.21.15.189:0/310154032 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f705006adb0 0x7f705006d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.836+0000 7f70741ab700 1 --2- 172.21.15.189:0/310154032 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f706c0cdc60 0x7f706c158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.836+0000 7f70741ab700 1 -- 172.21.15.189:0/310154032 >> 172.21.15.189:0/310154032 conn(0x7f706c0cc290 msgr2=0x7f706c066550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:21.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.836+0000 7f70741ab700 1 -- 172.21.15.189:0/310154032 shutdown_connections 2024-04-06T13:29:21.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.836+0000 7f70741ab700 1 -- 172.21.15.189:0/310154032 wait complete. 2024-04-06T13:29:21.840 INFO:teuthology.orchestra.run.smithi189.stderr:updated caps for client.0 2024-04-06T13:29:21.850 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:29:21.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.972+0000 7f0f67af8700 1 Processor -- start 2024-04-06T13:29:21.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.972+0000 7f0f67af8700 1 -- start start 2024-04-06T13:29:21.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.973+0000 7f0f67af8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0f600709d0 0x7f0f600cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:21.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.973+0000 7f0f67af8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0f60070e40 0x7f0f600cf480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:21.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.973+0000 7f0f67af8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f60071340 con 0x7f0f600b4ce0 2024-04-06T13:29:21.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.973+0000 7f0f67af8700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f600cd000 con 0x7f0f60070e40 2024-04-06T13:29:21.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.973+0000 7f0f67af8700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f6006ce40 con 0x7f0f600709d0 2024-04-06T13:29:21.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.973+0000 7f0f66095700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0f60070e40 0x7f0f600cf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:21.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.973+0000 7f0f66095700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0f60070e40 0x7f0f600cf480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54292/0 (socket says 172.21.15.189:54292) 2024-04-06T13:29:21.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.973+0000 7f0f66095700 1 -- 172.21.15.189:0/565510202 learned_addr learned my addr 172.21.15.189:0/565510202 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:21.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.973+0000 7f0f65093700 1 --2- 172.21.15.189:0/565510202 >> v2:172.21.15.192:3300/0 conn(0x7f0f600709d0 0x7f0f600cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:21.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.974+0000 7f0f66095700 1 -- 172.21.15.189:0/565510202 >> v2:172.21.15.192:3300/0 conn(0x7f0f600709d0 msgr2=0x7f0f600cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:21.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.974+0000 7f0f66095700 1 --2- 172.21.15.189:0/565510202 >> v2:172.21.15.192:3300/0 conn(0x7f0f600709d0 0x7f0f600cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.974+0000 7f0f66095700 1 -- 172.21.15.189:0/565510202 >> v1:172.21.15.192:6789/0 conn(0x7f0f600b4ce0 legacy=0x7f0f6006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:21.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.974+0000 7f0f66095700 1 -- 172.21.15.189:0/565510202 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f60071620 con 0x7f0f60070e40 2024-04-06T13:29:21.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.974+0000 7f0f66095700 1 --2- 172.21.15.189:0/565510202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f60070e40 0x7f0f600cf480 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f0f600d0020 tx=0x7f0f5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbff58e6a25356c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:21.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.974+0000 7f0f64892700 1 -- 172.21.15.189:0/565510202 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5000e070 con 0x7f0f60070e40 2024-04-06T13:29:21.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.974+0000 7f0f64892700 1 -- 172.21.15.189:0/565510202 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f50003a20 con 0x7f0f60070e40 2024-04-06T13:29:21.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.974+0000 7f0f64892700 1 -- 172.21.15.189:0/565510202 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f50003dd0 con 0x7f0f60070e40 2024-04-06T13:29:21.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.974+0000 7f0f67af8700 1 -- 172.21.15.189:0/565510202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f60070e40 msgr2=0x7f0f600cf480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:21.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.974+0000 7f0f67af8700 1 --2- 172.21.15.189:0/565510202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f60070e40 0x7f0f600cf480 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f0f600d0020 tx=0x7f0f5000b330 comp rx=0 tx=0).stop 2024-04-06T13:29:21.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.974+0000 7f0f67af8700 1 -- 172.21.15.189:0/565510202 shutdown_connections 2024-04-06T13:29:21.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.974+0000 7f0f67af8700 1 --2- 172.21.15.189:0/565510202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f60070e40 0x7f0f600cf480 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.974+0000 7f0f67af8700 1 --2- 172.21.15.189:0/565510202 >> v2:172.21.15.192:3300/0 conn(0x7f0f600709d0 0x7f0f600cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.974+0000 7f0f67af8700 1 -- 172.21.15.189:0/565510202 >> 172.21.15.189:0/565510202 conn(0x7f0f600cba70 msgr2=0x7f0f600cbe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:21.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.975+0000 7f0f67af8700 1 -- 172.21.15.189:0/565510202 shutdown_connections 2024-04-06T13:29:21.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.975+0000 7f0f67af8700 1 -- 172.21.15.189:0/565510202 wait complete. 2024-04-06T13:29:21.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.976+0000 7f0f67af8700 1 Processor -- start 2024-04-06T13:29:21.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.976+0000 7f0f67af8700 1 -- start start 2024-04-06T13:29:21.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.977+0000 7f0f67af8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f600709d0 0x7f0f6014bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:21.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.977+0000 7f0f67af8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f60070e40 0x7f0f60149af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:21.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.977+0000 7f0f67af8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f600b4ce0 0x7f0f601576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:21.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.977+0000 7f0f67af8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f60158ae0 con 0x7f0f600b4ce0 2024-04-06T13:29:21.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.977+0000 7f0f67af8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f60157f60 con 0x7f0f60070e40 2024-04-06T13:29:21.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.977+0000 7f0f67af8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f60158240 con 0x7f0f600709d0 2024-04-06T13:29:21.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.977+0000 7f0f66095700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f600b4ce0 0x7f0f601576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:21.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.977+0000 7f0f66095700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f600b4ce0 0x7f0f601576e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54294/0 (socket says 172.21.15.189:54294) 2024-04-06T13:29:21.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.977+0000 7f0f66095700 1 -- 172.21.15.189:0/3931473695 learned_addr learned my addr 172.21.15.189:0/3931473695 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:21.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.977+0000 7f0f66095700 1 -- 172.21.15.189:0/3931473695 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f600709d0 msgr2=0x7f0f6014bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:21.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.977+0000 7f0f65894700 1 --2- 172.21.15.189:0/3931473695 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f600709d0 0x7f0f6014bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:21.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.977+0000 7f0f65093700 1 --2- 172.21.15.189:0/3931473695 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f60070e40 0x7f0f60149af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:21.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.977+0000 7f0f66095700 1 --2- 172.21.15.189:0/3931473695 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f600709d0 0x7f0f6014bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.977+0000 7f0f66095700 1 -- 172.21.15.189:0/3931473695 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f60070e40 msgr2=0x7f0f60149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:21.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.977+0000 7f0f66095700 1 --2- 172.21.15.189:0/3931473695 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f60070e40 0x7f0f60149af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:21.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.977+0000 7f0f66095700 1 -- 172.21.15.189:0/3931473695 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f50009b30 con 0x7f0f600b4ce0 2024-04-06T13:29:21.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.977+0000 7f0f65093700 1 --2- 172.21.15.189:0/3931473695 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f60070e40 0x7f0f60149af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:29:21.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.977+0000 7f0f65894700 1 --2- 172.21.15.189:0/3931473695 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f600709d0 0x7f0f6014bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:29:21.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.978+0000 7f0f66095700 1 --2- 172.21.15.189:0/3931473695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f600b4ce0 0x7f0f601576e0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f0f6006c030 tx=0x7f0f50004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:21.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.978+0000 7f0f5e7fc700 1 -- 172.21.15.189:0/3931473695 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f50004440 con 0x7f0f600b4ce0 2024-04-06T13:29:21.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.978+0000 7f0f5e7fc700 1 -- 172.21.15.189:0/3931473695 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f500045e0 con 0x7f0f600b4ce0 2024-04-06T13:29:21.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.978+0000 7f0f67af8700 1 -- 172.21.15.189:0/3931473695 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f60158520 con 0x7f0f600b4ce0 2024-04-06T13:29:21.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.978+0000 7f0f5e7fc700 1 -- 172.21.15.189:0/3931473695 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f500094d0 con 0x7f0f600b4ce0 2024-04-06T13:29:21.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.978+0000 7f0f67af8700 1 -- 172.21.15.189:0/3931473695 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f60157c20 con 0x7f0f600b4ce0 2024-04-06T13:29:21.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.979+0000 7f0f67af8700 1 -- 172.21.15.189:0/3931473695 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f600748f0 con 0x7f0f600b4ce0 2024-04-06T13:29:21.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.979+0000 7f0f5e7fc700 1 -- 172.21.15.189:0/3931473695 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0f50012430 con 0x7f0f600b4ce0 2024-04-06T13:29:21.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.979+0000 7f0f5e7fc700 1 --2- 172.21.15.189:0/3931473695 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0f4c06adb0 0x7f0f4c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:21.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.980+0000 7f0f5e7fc700 1 -- 172.21.15.189:0/3931473695 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0f5008bcd0 con 0x7f0f600b4ce0 2024-04-06T13:29:21.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.980+0000 7f0f65894700 1 --2- 172.21.15.189:0/3931473695 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0f4c06adb0 0x7f0f4c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:21.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.980+0000 7f0f65894700 1 --2- 172.21.15.189:0/3931473695 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0f4c06adb0 0x7f0f4c06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0f6006d2c0 tx=0x7f0f58008040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:21.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:21.984+0000 7f0f5e7fc700 1 -- 172.21.15.189:0/3931473695 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0f50056c60 con 0x7f0f600b4ce0 2024-04-06T13:29:22.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.221+0000 7f0f67af8700 1 -- 172.21.15.189:0/3931473695 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f600b15d0 con 0x7f0f600b4ce0 2024-04-06T13:29:22.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.222+0000 7f0f5e7fc700 1 -- 172.21.15.189:0/3931473695 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+4749 (secure 0 0 0) 0x7f0f50017030 con 0x7f0f600b4ce0 2024-04-06T13:29:22.222 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:29:22.222 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":85,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8141,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2979280081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2979280081},{"type":"v1","addr":"172.21.15.192:6835","nonce":2979280081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":10125,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/286991908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":286991908},{"type":"v1","addr":"172.21.15.189:6836","nonce":286991908}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":10128,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/704323171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":704323171},{"type":"v1","addr":"172.21.15.189:6837","nonce":704323171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:29:16.858698+0000","modified":"2024-04-06T13:29:19.868525+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10137":{"gid":10137,"name":"b","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6834/2662267274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2662267274},{"type":"v1","addr":"172.21.15.192:6834","nonce":2662267274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-04-06T13:29:22.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.224+0000 7f0f67af8700 1 -- 172.21.15.189:0/3931473695 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0f4c06adb0 msgr2=0x7f0f4c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:22.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.224+0000 7f0f67af8700 1 --2- 172.21.15.189:0/3931473695 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0f4c06adb0 0x7f0f4c06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0f6006d2c0 tx=0x7f0f58008040 comp rx=0 tx=0).stop 2024-04-06T13:29:22.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.224+0000 7f0f67af8700 1 -- 172.21.15.189:0/3931473695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f600b4ce0 msgr2=0x7f0f601576e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:22.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.224+0000 7f0f67af8700 1 --2- 172.21.15.189:0/3931473695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f600b4ce0 0x7f0f601576e0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f0f6006c030 tx=0x7f0f50004ad0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.224+0000 7f0f67af8700 1 -- 172.21.15.189:0/3931473695 shutdown_connections 2024-04-06T13:29:22.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.224+0000 7f0f67af8700 1 --2- 172.21.15.189:0/3931473695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f600b4ce0 0x7f0f601576e0 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.224+0000 7f0f67af8700 1 --2- 172.21.15.189:0/3931473695 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f600709d0 0x7f0f6014bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.224+0000 7f0f67af8700 1 --2- 172.21.15.189:0/3931473695 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f0f4c06adb0 0x7f0f4c06d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.224+0000 7f0f67af8700 1 --2- 172.21.15.189:0/3931473695 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f60070e40 0x7f0f60149af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.224+0000 7f0f67af8700 1 -- 172.21.15.189:0/3931473695 >> 172.21.15.189:0/3931473695 conn(0x7f0f600cba70 msgr2=0x7f0f600b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:22.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.224+0000 7f0f67af8700 1 -- 172.21.15.189:0/3931473695 shutdown_connections 2024-04-06T13:29:22.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.224+0000 7f0f67af8700 1 -- 172.21.15.189:0/3931473695 wait complete. 2024-04-06T13:29:22.226 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 85 2024-04-06T13:29:22.639 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 85, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-06T13:29:16.858698+0000', 'modified': '2024-04-06T13:29:19.868525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10137}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10137': {'gid': 10137, 'name': 'b', 'rank': 0, 'incarnation': 82, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.192:6834/2662267274', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6832', 'nonce': 2662267274}, {'type': 'v1', 'addr': '172.21.15.192:6834', 'nonce': 2662267274}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-06T13:29:22.639 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-06T13:29:22.639 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-06T13:29:22.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.764+0000 7f574ebea700 1 Processor -- start 2024-04-06T13:29:22.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.764+0000 7f574ebea700 1 -- start start 2024-04-06T13:29:22.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.765+0000 7f574ebea700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5748158fe0 0x7f574806a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.765+0000 7f574ebea700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f574815c2d0 0x7f57480ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.765+0000 7f574ebea700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f574815f9e0 con 0x7f574806ac70 2024-04-06T13:29:22.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.765+0000 7f574ebea700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574806b0f0 con 0x7f574815c2d0 2024-04-06T13:29:22.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.765+0000 7f574ebea700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f574815f0d0 con 0x7f5748158fe0 2024-04-06T13:29:22.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.765+0000 7f574d187700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f574815c2d0 0x7f57480ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.765+0000 7f574d187700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f574815c2d0 0x7f57480ceca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54306/0 (socket says 172.21.15.189:54306) 2024-04-06T13:29:22.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.765+0000 7f574d187700 1 -- 172.21.15.189:0/1131055036 learned_addr learned my addr 172.21.15.189:0/1131055036 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:22.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.765+0000 7f574c986700 1 --2- 172.21.15.189:0/1131055036 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5748158fe0 0x7f574806a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.765+0000 7f574d187700 1 -- 172.21.15.189:0/1131055036 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5748158fe0 msgr2=0x7f574806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:22.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.765+0000 7f574d187700 1 --2- 172.21.15.189:0/1131055036 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5748158fe0 0x7f574806a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.765+0000 7f574d187700 1 -- 172.21.15.189:0/1131055036 >> v1:172.21.15.192:6789/0 conn(0x7f574806ac70 legacy=0x7f574815bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:22.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.765+0000 7f574d187700 1 -- 172.21.15.189:0/1131055036 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574815c8f0 con 0x7f574815c2d0 2024-04-06T13:29:22.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.765+0000 7f574c986700 1 --2- 172.21.15.189:0/1131055036 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5748158fe0 0x7f574806a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:29:22.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.765+0000 7f574d187700 1 --2- 172.21.15.189:0/1131055036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574815c2d0 0x7f57480ceca0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f5740009e50 tx=0x7f574000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=decde248c2523922 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:22.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.766+0000 7f57477fe700 1 -- 172.21.15.189:0/1131055036 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574000e070 con 0x7f574815c2d0 2024-04-06T13:29:22.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.766+0000 7f57477fe700 1 -- 172.21.15.189:0/1131055036 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574000b9c0 con 0x7f574815c2d0 2024-04-06T13:29:22.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.766+0000 7f57477fe700 1 -- 172.21.15.189:0/1131055036 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574000bd70 con 0x7f574815c2d0 2024-04-06T13:29:22.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.766+0000 7f574ebea700 1 -- 172.21.15.189:0/1131055036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574815c2d0 msgr2=0x7f57480ceca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:22.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.766+0000 7f574ebea700 1 --2- 172.21.15.189:0/1131055036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574815c2d0 0x7f57480ceca0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f5740009e50 tx=0x7f574000b330 comp rx=0 tx=0).stop 2024-04-06T13:29:22.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.766+0000 7f574ebea700 1 -- 172.21.15.189:0/1131055036 shutdown_connections 2024-04-06T13:29:22.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.766+0000 7f574ebea700 1 --2- 172.21.15.189:0/1131055036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574815c2d0 0x7f57480ceca0 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.766+0000 7f574ebea700 1 --2- 172.21.15.189:0/1131055036 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5748158fe0 0x7f574806a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.766+0000 7f574ebea700 1 -- 172.21.15.189:0/1131055036 >> 172.21.15.189:0/1131055036 conn(0x7f57480d00b0 msgr2=0x7f57480d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:22.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.766+0000 7f574ebea700 1 -- 172.21.15.189:0/1131055036 shutdown_connections 2024-04-06T13:29:22.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.766+0000 7f574ebea700 1 -- 172.21.15.189:0/1131055036 wait complete. 2024-04-06T13:29:22.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.767+0000 7f574ebea700 1 Processor -- start 2024-04-06T13:29:22.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.767+0000 7f574ebea700 1 -- start start 2024-04-06T13:29:22.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.768+0000 7f574ebea700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f574806ac70 0x7f57480ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.768+0000 7f574ebea700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5748158fe0 0x7f5748139390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.768+0000 7f574ebea700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f574815c2d0 0x7f57480ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.768+0000 7f574ebea700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57480ace00 con 0x7f5748158fe0 2024-04-06T13:29:22.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.768+0000 7f574ebea700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57481398d0 con 0x7f574815c2d0 2024-04-06T13:29:22.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.768+0000 7f574ebea700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57480acff0 con 0x7f574806ac70 2024-04-06T13:29:22.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.768+0000 7f5747fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5748158fe0 0x7f5748139390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.768+0000 7f5747fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5748158fe0 0x7f5748139390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54314/0 (socket says 172.21.15.189:54314) 2024-04-06T13:29:22.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.768+0000 7f5747fff700 1 -- 172.21.15.189:0/780530791 learned_addr learned my addr 172.21.15.189:0/780530791 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:22.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.768+0000 7f574c986700 1 --2- 172.21.15.189:0/780530791 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f574806ac70 0x7f57480ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.769+0000 7f574d187700 1 --2- 172.21.15.189:0/780530791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f574815c2d0 0x7f57480ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.769+0000 7f5747fff700 1 -- 172.21.15.189:0/780530791 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f574806ac70 msgr2=0x7f57480ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:22.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.769+0000 7f5747fff700 1 --2- 172.21.15.189:0/780530791 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f574806ac70 0x7f57480ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.769+0000 7f5747fff700 1 -- 172.21.15.189:0/780530791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f574815c2d0 msgr2=0x7f57480ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:22.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.769+0000 7f5747fff700 1 --2- 172.21.15.189:0/780530791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f574815c2d0 0x7f57480ccd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.769+0000 7f5747fff700 1 -- 172.21.15.189:0/780530791 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5740009b30 con 0x7f5748158fe0 2024-04-06T13:29:22.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.769+0000 7f574c986700 1 --2- 172.21.15.189:0/780530791 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f574806ac70 0x7f57480ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:29:22.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.769+0000 7f5747fff700 1 --2- 172.21.15.189:0/780530791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5748158fe0 0x7f5748139390 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f573400c4e0 tx=0x7f573400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:22.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.769+0000 7f57457fa700 1 -- 172.21.15.189:0/780530791 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57340052d0 con 0x7f5748158fe0 2024-04-06T13:29:22.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.769+0000 7f57457fa700 1 -- 172.21.15.189:0/780530791 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5734005430 con 0x7f5748158fe0 2024-04-06T13:29:22.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.769+0000 7f574ebea700 1 -- 172.21.15.189:0/780530791 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57480b4b10 con 0x7f5748158fe0 2024-04-06T13:29:22.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.769+0000 7f57457fa700 1 -- 172.21.15.189:0/780530791 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57340038c0 con 0x7f5748158fe0 2024-04-06T13:29:22.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.769+0000 7f574ebea700 1 -- 172.21.15.189:0/780530791 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57480ad3d0 con 0x7f5748158fe0 2024-04-06T13:29:22.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.770+0000 7f57457fa700 1 -- 172.21.15.189:0/780530791 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f5734003a20 con 0x7f5748158fe0 2024-04-06T13:29:22.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.771+0000 7f57457fa700 1 --2- 172.21.15.189:0/780530791 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f572c06adb0 0x7f572c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.771+0000 7f57457fa700 1 -- 172.21.15.189:0/780530791 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5734088ae0 con 0x7f5748158fe0 2024-04-06T13:29:22.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.771+0000 7f574c986700 1 --2- 172.21.15.189:0/780530791 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f572c06adb0 0x7f572c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.771+0000 7f574c986700 1 --2- 172.21.15.189:0/780530791 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f572c06adb0 0x7f572c06d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f573c0099f0 tx=0x7f573c008040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:22.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.773+0000 7f575ad9eb80 1 Processor -- start 2024-04-06T13:29:22.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.773+0000 7f575ad9eb80 1 -- start start 2024-04-06T13:29:22.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f575ad9eb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5588f14f8c20 0x5588f14fca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f575ad9eb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14fdfc0 0x5588f14d45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f575ad9eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 0x5588f14d4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f575ad9eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588f142d940 con 0x5588f14d3980 2024-04-06T13:29:22.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f575ad9eb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588f142b300 con 0x5588f14fdfc0 2024-04-06T13:29:22.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f575ad9eb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588f1426bf0 con 0x5588f14f8c20 2024-04-06T13:29:22.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f574d187700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 0x5588f14d4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f574d187700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 0x5588f14d4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54316/0 (socket says 172.21.15.189:54316) 2024-04-06T13:29:22.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f574d187700 1 -- 172.21.15.189:0/4132600719 learned_addr learned my addr 172.21.15.189:0/4132600719 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:22.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f5747fff700 1 --2- 172.21.15.189:0/4132600719 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14fdfc0 0x5588f14d45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f574c986700 1 --2- 172.21.15.189:0/4132600719 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5588f14f8c20 0x5588f14fca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f574d187700 1 -- 172.21.15.189:0/4132600719 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5588f14f8c20 msgr2=0x5588f14fca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:22.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f574d187700 1 --2- 172.21.15.189:0/4132600719 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5588f14f8c20 0x5588f14fca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f574d187700 1 -- 172.21.15.189:0/4132600719 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14fdfc0 msgr2=0x5588f14d45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:22.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f574d187700 1 --2- 172.21.15.189:0/4132600719 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14fdfc0 0x5588f14d45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f574d187700 1 -- 172.21.15.189:0/4132600719 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588f1254b10 con 0x5588f14d3980 2024-04-06T13:29:22.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f574d187700 1 --2- 172.21.15.189:0/4132600719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 0x5588f14d4b30 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f574806c820 tx=0x7f5740010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a2c30e7cdce4e91 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:22.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f57327fc700 1 -- 172.21.15.189:0/4132600719 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5740003a20 con 0x5588f14d3980 2024-04-06T13:29:22.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f57327fc700 1 -- 172.21.15.189:0/4132600719 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5740003b80 con 0x5588f14d3980 2024-04-06T13:29:22.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.774+0000 7f57327fc700 1 -- 172.21.15.189:0/4132600719 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5740004040 con 0x5588f14d3980 2024-04-06T13:29:22.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.775+0000 7f575ad9eb80 1 -- 172.21.15.189:0/4132600719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 msgr2=0x5588f14d4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:22.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.775+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/4132600719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 0x5588f14d4b30 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f574806c820 tx=0x7f5740010000 comp rx=0 tx=0).stop 2024-04-06T13:29:22.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.775+0000 7f575ad9eb80 1 -- 172.21.15.189:0/4132600719 shutdown_connections 2024-04-06T13:29:22.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.775+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/4132600719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 0x5588f14d4b30 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.775+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/4132600719 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5588f14f8c20 0x5588f14fca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.775+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/4132600719 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14fdfc0 0x5588f14d45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.775+0000 7f575ad9eb80 1 -- 172.21.15.189:0/4132600719 >> 172.21.15.189:0/4132600719 conn(0x5588f1434650 msgr2=0x5588f151b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:22.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.775+0000 7f575ad9eb80 1 -- 172.21.15.189:0/4132600719 shutdown_connections 2024-04-06T13:29:22.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.775+0000 7f575ad9eb80 1 -- 172.21.15.189:0/4132600719 wait complete. 2024-04-06T13:29:22.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.775+0000 7f575ad9eb80 1 -- start start 2024-04-06T13:29:22.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.776+0000 7f575ad9eb80 1 Processor -- start 2024-04-06T13:29:22.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.776+0000 7f575ad9eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 0x5588f14f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.776+0000 7f575ad9eb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14f8c20 0x5588f1500d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.776+0000 7f575ad9eb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5588f14fdfc0 0x5588f1514620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.776+0000 7f575ad9eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588f13c2d30 con 0x5588f14d3980 2024-04-06T13:29:22.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.776+0000 7f575ad9eb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588f13c29d0 con 0x5588f14f8c20 2024-04-06T13:29:22.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.776+0000 7f575ad9eb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588f12aa650 con 0x5588f14fdfc0 2024-04-06T13:29:22.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f575ad9eb80 10 client.0 _ll_register_callbacks cb 0x5588f13cc420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T13:29:22.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f574c986700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 0x5588f14f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f574c986700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 0x5588f14f76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54328/0 (socket says 172.21.15.189:54328) 2024-04-06T13:29:22.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f574c986700 1 -- 172.21.15.189:0/1634234382 learned_addr learned my addr 172.21.15.189:0/1634234382 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:22.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f574d187700 1 --2- 172.21.15.189:0/1634234382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5588f14fdfc0 0x5588f1514620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f5747fff700 1 --2- 172.21.15.189:0/1634234382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14f8c20 0x5588f1500d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f574c986700 1 -- 172.21.15.189:0/1634234382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5588f14fdfc0 msgr2=0x5588f1514620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:22.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f574c986700 1 --2- 172.21.15.189:0/1634234382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5588f14fdfc0 0x5588f1514620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f574c986700 1 -- 172.21.15.189:0/1634234382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14f8c20 msgr2=0x5588f1500d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:22.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f574c986700 1 --2- 172.21.15.189:0/1634234382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14f8c20 0x5588f1500d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f574c986700 1 -- 172.21.15.189:0/1634234382 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574815c8f0 con 0x5588f14d3980 2024-04-06T13:29:22.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f5747fff700 1 --2- 172.21.15.189:0/1634234382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14f8c20 0x5588f1500d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:29:22.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f574c986700 1 --2- 172.21.15.189:0/1634234382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 0x5588f14f76a0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f573c0154f0 tx=0x7f573c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d9342a947b9165 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:22.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f57277fe700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T13:29:22.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f57277fe700 1 -- 172.21.15.189:0/1634234382 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c01c050 con 0x5588f14d3980 2024-04-06T13:29:22.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f57277fe700 1 -- 172.21.15.189:0/1634234382 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(192..192 src has 1..192) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f573c022660 con 0x5588f14d3980 2024-04-06T13:29:22.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f575ad9eb80 1 -- 172.21.15.189:0/1634234382 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5588f13e7330 con 0x5588f14d3980 2024-04-06T13:29:22.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f57277fe700 1 -- 172.21.15.189:0/1634234382 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f573c022820 con 0x5588f14d3980 2024-04-06T13:29:22.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f575ad9eb80 10 client.10269 fetch_fsmap learned FSMap version 85 2024-04-06T13:29:22.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f575ad9eb80 1 -- 172.21.15.189:0/1634234382 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=85,monmap=2+}) v3 -- 0x5588f13ece50 con 0x5588f14d3980 2024-04-06T13:29:22.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f57277fe700 1 -- 172.21.15.189:0/1634234382 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5588f13e7330 con 0x5588f14d3980 2024-04-06T13:29:22.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.777+0000 7f57277fe700 1 -- 172.21.15.189:0/1634234382 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 85) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f573c022980 con 0x5588f14d3980 2024-04-06T13:29:22.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.778+0000 7f575ad9eb80 10 client.10269 fetch_fsmap finished waiting for FSMap version 85 2024-04-06T13:29:22.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.778+0000 7f575ad9eb80 10 client.10269 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-06T13:29:22.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.778+0000 7f575ad9eb80 20 client.10269 populate_metadata read hostname 'smithi189' 2024-04-06T13:29:22.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.778+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/1634234382 >> [v2:172.21.15.192:6832/2662267274,v1:172.21.15.192:6834/2662267274] conn(0x5588f1500310 0x5588f1538440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.778+0000 7f575ad9eb80 4 client.10269 mds_command: new command op to 10137 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-06T13:29:22.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.778+0000 7f575ad9eb80 1 -- 172.21.15.189:0/1634234382 --> [v2:172.21.15.192:6832/2662267274,v1:172.21.15.192:6834/2662267274] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5588f141f6b0 con 0x5588f1500310 2024-04-06T13:29:22.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.778+0000 7f5747fff700 1 --2- 172.21.15.189:0/1634234382 >> [v2:172.21.15.192:6832/2662267274,v1:172.21.15.192:6834/2662267274] conn(0x5588f1500310 0x5588f1538440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.778+0000 7f57277fe700 10 client.10269 ms_handle_connect on v2:172.21.15.192:6832/2662267274 2024-04-06T13:29:22.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.778+0000 7f5747fff700 1 --2- 172.21.15.189:0/1634234382 >> [v2:172.21.15.192:6832/2662267274,v1:172.21.15.192:6834/2662267274] conn(0x5588f1500310 0x5588f1538440 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eddfcaeec78c7a93 server_cookie=b1b1fb7a7125245f in_seq=0 out_seq=0 2024-04-06T13:29:22.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.779+0000 7f57277fe700 1 -- 172.21.15.189:0/1634234382 <== mds.0 v2:172.21.15.192:6832/2662267274 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5588f141f6b0 con 0x5588f1500310 2024-04-06T13:29:22.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.779+0000 7f57277fe700 10 client.10269 handle_command_reply: tid=0 2024-04-06T13:29:22.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.779+0000 7f575ad9eb80 1 client.10269 shutdown 2024-04-06T13:29:22.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.779+0000 7f575ad9eb80 1 -- 172.21.15.189:0/1634234382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 msgr2=0x5588f14f76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:22.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.779+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/1634234382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 0x5588f14f76a0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f573c0154f0 tx=0x7f573c0047c0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.779+0000 7f575ad9eb80 1 -- 172.21.15.189:0/1634234382 shutdown_connections 2024-04-06T13:29:22.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.779+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/1634234382 >> [v2:172.21.15.192:6832/2662267274,v1:172.21.15.192:6834/2662267274] conn(0x5588f1500310 0x5588f1538440 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.779+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/1634234382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5588f14fdfc0 0x5588f1514620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.779+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/1634234382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 0x5588f14f76a0 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.779+0000 7f57277fe700 0 client.10269 ms_handle_reset on v2:172.21.15.192:6832/2662267274 2024-04-06T13:29:22.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.779+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/1634234382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14f8c20 0x5588f1500d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.779+0000 7f575ad9eb80 1 -- 172.21.15.189:0/1634234382 >> 172.21.15.189:0/1634234382 conn(0x5588f1434650 msgr2=0x5588f14d4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:22.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.779+0000 7f575ad9eb80 1 -- 172.21.15.189:0/1634234382 shutdown_connections 2024-04-06T13:29:22.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.779+0000 7f575ad9eb80 1 -- 172.21.15.189:0/1634234382 wait complete. 2024-04-06T13:29:22.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.779+0000 7f575ad9eb80 20 client.10269 trim_cache size 0 max 16384 2024-04-06T13:29:22.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.797+0000 7f575ad9eb80 1 Processor -- start 2024-04-06T13:29:22.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.797+0000 7f575ad9eb80 1 -- start start 2024-04-06T13:29:22.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.798+0000 7f575ad9eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 0x5588f14fe590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.798+0000 7f575ad9eb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14f8c20 0x5588f1501580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.798+0000 7f575ad9eb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5588f14fdfc0 0x5588f1501ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.798+0000 7f575ad9eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588f14cd1e0 con 0x5588f14d3980 2024-04-06T13:29:22.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.798+0000 7f575ad9eb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588f14f74b0 con 0x5588f14f8c20 2024-04-06T13:29:22.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.798+0000 7f575ad9eb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588f151b370 con 0x5588f14fdfc0 2024-04-06T13:29:22.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.798+0000 7f574c986700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 0x5588f14fe590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.798+0000 7f574c986700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 0x5588f14fe590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54340/0 (socket says 172.21.15.189:54340) 2024-04-06T13:29:22.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.798+0000 7f574c986700 1 -- 172.21.15.189:0/2094407299 learned_addr learned my addr 172.21.15.189:0/2094407299 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:22.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.798+0000 7f574c986700 1 -- 172.21.15.189:0/2094407299 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5588f14fdfc0 msgr2=0x5588f1501ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:22.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.798+0000 7f574c986700 1 --2- 172.21.15.189:0/2094407299 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5588f14fdfc0 0x5588f1501ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.798+0000 7f574c986700 1 -- 172.21.15.189:0/2094407299 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14f8c20 msgr2=0x5588f1501580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:22.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.798+0000 7f574c986700 1 --2- 172.21.15.189:0/2094407299 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14f8c20 0x5588f1501580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.798+0000 7f574c986700 1 -- 172.21.15.189:0/2094407299 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588f12acc80 con 0x5588f14d3980 2024-04-06T13:29:22.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.798+0000 7f574c986700 1 --2- 172.21.15.189:0/2094407299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 0x5588f14fe590 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f573c010550 tx=0x7f573c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27955bbddf1f292d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:22.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.799+0000 7f5732ffd700 1 -- 172.21.15.189:0/2094407299 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c01c070 con 0x5588f14d3980 2024-04-06T13:29:22.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.799+0000 7f5732ffd700 1 -- 172.21.15.189:0/2094407299 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573c01d040 con 0x5588f14d3980 2024-04-06T13:29:22.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.799+0000 7f5732ffd700 1 -- 172.21.15.189:0/2094407299 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c0228a0 con 0x5588f14d3980 2024-04-06T13:29:22.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.799+0000 7f575ad9eb80 1 -- 172.21.15.189:0/2094407299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 msgr2=0x5588f14fe590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:22.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.799+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/2094407299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 0x5588f14fe590 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f573c010550 tx=0x7f573c01f000 comp rx=0 tx=0).stop 2024-04-06T13:29:22.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.799+0000 7f575ad9eb80 1 -- 172.21.15.189:0/2094407299 shutdown_connections 2024-04-06T13:29:22.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.799+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/2094407299 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5588f14fdfc0 0x5588f1501ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.799+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/2094407299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14d3980 0x5588f14fe590 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.799+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/2094407299 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14f8c20 0x5588f1501580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.799+0000 7f575ad9eb80 1 -- 172.21.15.189:0/2094407299 >> 172.21.15.189:0/2094407299 conn(0x5588f1434650 msgr2=0x5588f1515050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:22.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.799+0000 7f575ad9eb80 1 -- 172.21.15.189:0/2094407299 shutdown_connections 2024-04-06T13:29:22.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.799+0000 7f575ad9eb80 1 -- 172.21.15.189:0/2094407299 wait complete. 2024-04-06T13:29:22.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.800+0000 7f575ad9eb80 1 -- start start 2024-04-06T13:29:22.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.800+0000 7f575ad9eb80 1 Processor -- start 2024-04-06T13:29:22.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f575ad9eb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5588f14d3980 0x5588f14d5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f575ad9eb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14f8c20 0x5588f151f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f575ad9eb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14fdfc0 0x5588f14fe590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f575ad9eb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588f151e870 con 0x5588f14f8c20 2024-04-06T13:29:22.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f575ad9eb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588f14d5c00 con 0x5588f14fdfc0 2024-04-06T13:29:22.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f575ad9eb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588f151f7c0 con 0x5588f14d3980 2024-04-06T13:29:22.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f575ad9eb80 10 client.0 _ll_register_callbacks cb 0x5588f13cc420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T13:29:22.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f5747fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14f8c20 0x5588f151f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f5747fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14f8c20 0x5588f151f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54352/0 (socket says 172.21.15.189:54352) 2024-04-06T13:29:22.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f5747fff700 1 -- 172.21.15.189:0/982073590 learned_addr learned my addr 172.21.15.189:0/982073590 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:22.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f574c986700 1 --2- 172.21.15.189:0/982073590 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5588f14d3980 0x5588f14d5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f574d187700 1 --2- 172.21.15.189:0/982073590 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14fdfc0 0x5588f14fe590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f5747fff700 1 -- 172.21.15.189:0/982073590 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5588f14d3980 msgr2=0x5588f14d5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:22.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f5747fff700 1 --2- 172.21.15.189:0/982073590 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5588f14d3980 0x5588f14d5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f5747fff700 1 -- 172.21.15.189:0/982073590 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14fdfc0 msgr2=0x5588f14fe590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:22.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f5747fff700 1 --2- 172.21.15.189:0/982073590 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14fdfc0 0x5588f14fe590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f5747fff700 1 -- 172.21.15.189:0/982073590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5588f13ece50 con 0x5588f14f8c20 2024-04-06T13:29:22.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f5747fff700 1 --2- 172.21.15.189:0/982073590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14f8c20 0x5588f151f280 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f573400cca0 tx=0x7f5734057830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf78081164d6aedc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:22.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f57277fe700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T13:29:22.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f57277fe700 1 -- 172.21.15.189:0/982073590 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573405b360 con 0x5588f14f8c20 2024-04-06T13:29:22.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f57277fe700 1 -- 172.21.15.189:0/982073590 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(192..192 src has 1..192) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f573405fae0 con 0x5588f14f8c20 2024-04-06T13:29:22.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.801+0000 7f575ad9eb80 1 -- 172.21.15.189:0/982073590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5740003b80 con 0x5588f14f8c20 2024-04-06T13:29:22.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.802+0000 7f57277fe700 1 -- 172.21.15.189:0/982073590 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5734057950 con 0x5588f14f8c20 2024-04-06T13:29:22.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.802+0000 7f575ad9eb80 10 client.10275 fetch_fsmap learned FSMap version 85 2024-04-06T13:29:22.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.802+0000 7f575ad9eb80 1 -- 172.21.15.189:0/982073590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=85,monmap=2+}) v3 -- 0x5588f1252fe0 con 0x5588f14f8c20 2024-04-06T13:29:22.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.802+0000 7f57277fe700 1 -- 172.21.15.189:0/982073590 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5740003b80 con 0x5588f14f8c20 2024-04-06T13:29:22.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.802+0000 7f57277fe700 1 -- 172.21.15.189:0/982073590 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 85) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5734056be0 con 0x5588f14f8c20 2024-04-06T13:29:22.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.802+0000 7f575ad9eb80 10 client.10275 fetch_fsmap finished waiting for FSMap version 85 2024-04-06T13:29:22.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.802+0000 7f575ad9eb80 10 client.10275 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-06T13:29:22.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.802+0000 7f575ad9eb80 20 client.10275 populate_metadata read hostname 'smithi189' 2024-04-06T13:29:22.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.802+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/982073590 >> [v2:172.21.15.192:6832/2662267274,v1:172.21.15.192:6834/2662267274] conn(0x5588f1586f70 0x5588f1587350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.802+0000 7f575ad9eb80 4 client.10275 mds_command: new command op to 10137 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-06T13:29:22.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.802+0000 7f575ad9eb80 1 -- 172.21.15.189:0/982073590 --> [v2:172.21.15.192:6832/2662267274,v1:172.21.15.192:6834/2662267274] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5588f1275ca0 con 0x5588f1586f70 2024-04-06T13:29:22.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.802+0000 7f574c986700 1 --2- 172.21.15.189:0/982073590 >> [v2:172.21.15.192:6832/2662267274,v1:172.21.15.192:6834/2662267274] conn(0x5588f1586f70 0x5588f1587350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.802+0000 7f57277fe700 10 client.10275 ms_handle_connect on v2:172.21.15.192:6832/2662267274 2024-04-06T13:29:22.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.802+0000 7f574c986700 1 --2- 172.21.15.189:0/982073590 >> [v2:172.21.15.192:6832/2662267274,v1:172.21.15.192:6834/2662267274] conn(0x5588f1586f70 0x5588f1587350 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31b2677328827684 server_cookie=72892540c9b6c585 in_seq=0 out_seq=0 2024-04-06T13:29:22.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.803+0000 7f57277fe700 1 -- 172.21.15.189:0/982073590 <== mds.0 v2:172.21.15.192:6832/2662267274 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x5588f1275ca0 con 0x5588f1586f70 2024-04-06T13:29:22.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.803+0000 7f57277fe700 10 client.10275 handle_command_reply: tid=0 2024-04-06T13:29:22.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.803+0000 7f575ad9eb80 1 client.10275 shutdown 2024-04-06T13:29:22.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.803+0000 7f575ad9eb80 1 -- 172.21.15.189:0/982073590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14f8c20 msgr2=0x5588f151f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:22.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.803+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/982073590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14f8c20 0x5588f151f280 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f573400cca0 tx=0x7f5734057830 comp rx=0 tx=0).stop 2024-04-06T13:29:22.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.803+0000 7f575ad9eb80 1 -- 172.21.15.189:0/982073590 shutdown_connections 2024-04-06T13:29:22.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.803+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/982073590 >> [v2:172.21.15.192:6832/2662267274,v1:172.21.15.192:6834/2662267274] conn(0x5588f1586f70 0x5588f1587350 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.803+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/982073590 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588f14fdfc0 0x5588f14fe590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.803+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/982073590 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5588f14d3980 0x5588f14d5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.803+0000 7f57277fe700 0 client.10275 ms_handle_reset on v2:172.21.15.192:6832/2662267274 2024-04-06T13:29:22.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.803+0000 7f575ad9eb80 1 --2- 172.21.15.189:0/982073590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588f14f8c20 0x5588f151f280 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.803+0000 7f575ad9eb80 1 -- 172.21.15.189:0/982073590 >> 172.21.15.189:0/982073590 conn(0x5588f1434650 msgr2=0x5588f1515050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:22.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.803+0000 7f575ad9eb80 1 -- 172.21.15.189:0/982073590 shutdown_connections 2024-04-06T13:29:22.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.803+0000 7f575ad9eb80 1 -- 172.21.15.189:0/982073590 wait complete. 2024-04-06T13:29:22.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.803+0000 7f575ad9eb80 20 client.10275 trim_cache size 0 max 16384 2024-04-06T13:29:22.809 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:29:22.809 INFO:teuthology.orchestra.run.smithi189.stdout: "cluster_fsid": "e30041de-e9a3-4351-8ca7-29d7f3e3728c", 2024-04-06T13:29:22.809 INFO:teuthology.orchestra.run.smithi189.stdout: "whoami": 0, 2024-04-06T13:29:22.809 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 10137, 2024-04-06T13:29:22.809 INFO:teuthology.orchestra.run.smithi189.stdout: "want_state": "up:active", 2024-04-06T13:29:22.810 INFO:teuthology.orchestra.run.smithi189.stdout: "state": "up:active", 2024-04-06T13:29:22.810 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs", 2024-04-06T13:29:22.810 INFO:teuthology.orchestra.run.smithi189.stdout: "rank_uptime": 5.9427017720000004, 2024-04-06T13:29:22.810 INFO:teuthology.orchestra.run.smithi189.stdout: "mdsmap_epoch": 85, 2024-04-06T13:29:22.810 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch": 192, 2024-04-06T13:29:22.810 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch_barrier": 192, 2024-04-06T13:29:22.810 INFO:teuthology.orchestra.run.smithi189.stdout: "uptime": 11.263344942 2024-04-06T13:29:22.810 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:29:22.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.803+0000 7f574ebea700 1 -- 172.21.15.189:0/780530791 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f572c06adb0 msgr2=0x7f572c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:22.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.803+0000 7f574ebea700 1 --2- 172.21.15.189:0/780530791 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f572c06adb0 0x7f572c06d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f573c0099f0 tx=0x7f573c008040 comp rx=0 tx=0).stop 2024-04-06T13:29:22.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.803+0000 7f574ebea700 1 -- 172.21.15.189:0/780530791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5748158fe0 msgr2=0x7f5748139390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:22.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.803+0000 7f574ebea700 1 --2- 172.21.15.189:0/780530791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5748158fe0 0x7f5748139390 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f573400c4e0 tx=0x7f573400c8a0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.804+0000 7f574ebea700 1 -- 172.21.15.189:0/780530791 shutdown_connections 2024-04-06T13:29:22.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.804+0000 7f574ebea700 1 --2- 172.21.15.189:0/780530791 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f572c06adb0 0x7f572c06d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.804+0000 7f574ebea700 1 --2- 172.21.15.189:0/780530791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f574815c2d0 0x7f57480ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.804+0000 7f574ebea700 1 --2- 172.21.15.189:0/780530791 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f574806ac70 0x7f57480ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.804+0000 7f574ebea700 1 --2- 172.21.15.189:0/780530791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5748158fe0 0x7f5748139390 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.804+0000 7f574ebea700 1 -- 172.21.15.189:0/780530791 >> 172.21.15.189:0/780530791 conn(0x7f57480d00b0 msgr2=0x7f5748142b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:22.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.804+0000 7f574ebea700 1 -- 172.21.15.189:0/780530791 shutdown_connections 2024-04-06T13:29:22.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.804+0000 7f574ebea700 1 -- 172.21.15.189:0/780530791 wait complete. 2024-04-06T13:29:22.815 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T13:29:22.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.940+0000 7f16a230e700 1 Processor -- start 2024-04-06T13:29:22.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.940+0000 7f16a230e700 1 -- start start 2024-04-06T13:29:22.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.940+0000 7f16a230e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f169c158fe0 0x7f169c06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.940+0000 7f16a230e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f169c15f9e0 con 0x7f169c15c2d0 2024-04-06T13:29:22.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.940+0000 7f16a230e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f169c15f0d0 con 0x7f169c06ac70 2024-04-06T13:29:22.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.940+0000 7f16a230e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169c06b0f0 con 0x7f169c158fe0 2024-04-06T13:29:22.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.940+0000 7f16a08ab700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f169c15c2d0 0x7f169c0ceca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51988/0 (socket says 172.21.15.189:51988) 2024-04-06T13:29:22.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.940+0000 7f16a08ab700 1 -- 172.21.15.189:0/3673148880 learned_addr learned my addr 172.21.15.189:0/3673148880 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:22.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.941+0000 7f169bfff700 1 --2- 172.21.15.189:0/3673148880 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f169c158fe0 0x7f169c06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.941+0000 7f169affd700 1 -- 172.21.15.189:0/3673148880 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686730394 0 0) 0x7f169c15f9e0 con 0x7f169c15c2d0 2024-04-06T13:29:22.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.941+0000 7f169affd700 1 -- 172.21.15.189:0/3673148880 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16840035b0 con 0x7f169c15c2d0 2024-04-06T13:29:22.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.941+0000 7f169affd700 1 -- 172.21.15.189:0/3673148880 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376580104 0 0) 0x7f169c15f0d0 con 0x7f169c06ac70 2024-04-06T13:29:22.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.941+0000 7f169affd700 1 -- 172.21.15.189:0/3673148880 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f169c15f9e0 con 0x7f169c06ac70 2024-04-06T13:29:22.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.941+0000 7f169affd700 1 -- 172.21.15.189:0/3673148880 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 712025355 0 0) 0x7f16840035b0 con 0x7f169c15c2d0 2024-04-06T13:29:22.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.941+0000 7f169bfff700 1 -- 172.21.15.189:0/3673148880 >> v1:172.21.15.192:6789/0 conn(0x7f169c06ac70 legacy=0x7f169c15bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:22.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.941+0000 7f169bfff700 1 -- 172.21.15.189:0/3673148880 >> v1:172.21.15.189:6789/0 conn(0x7f169c15c2d0 legacy=0x7f169c0ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:22.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.941+0000 7f169bfff700 1 -- 172.21.15.189:0/3673148880 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c15c8f0 con 0x7f169c158fe0 2024-04-06T13:29:22.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.941+0000 7f169bfff700 1 --2- 172.21.15.189:0/3673148880 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f169c158fe0 0x7f169c06a730 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1690009870 tx=0x7f1690009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7d0c93859548b83 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:22.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.941+0000 7f169affd700 1 -- 172.21.15.189:0/3673148880 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16940019f0 con 0x7f169c158fe0 2024-04-06T13:29:22.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.941+0000 7f169affd700 1 -- 172.21.15.189:0/3673148880 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1690004030 con 0x7f169c158fe0 2024-04-06T13:29:22.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.941+0000 7f169affd700 1 -- 172.21.15.189:0/3673148880 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16900043a0 con 0x7f169c158fe0 2024-04-06T13:29:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.942+0000 7f16a230e700 1 -- 172.21.15.189:0/3673148880 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f169c158fe0 msgr2=0x7f169c06a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.942+0000 7f16a230e700 1 --2- 172.21.15.189:0/3673148880 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f169c158fe0 0x7f169c06a730 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1690009870 tx=0x7f1690009b80 comp rx=0 tx=0).stop 2024-04-06T13:29:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.942+0000 7f16a230e700 1 -- 172.21.15.189:0/3673148880 shutdown_connections 2024-04-06T13:29:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.942+0000 7f16a230e700 1 --2- 172.21.15.189:0/3673148880 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f169c158fe0 0x7f169c06a730 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.942+0000 7f16a230e700 1 -- 172.21.15.189:0/3673148880 >> 172.21.15.189:0/3673148880 conn(0x7f169c0d00b0 msgr2=0x7f169c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.942+0000 7f16a230e700 1 -- 172.21.15.189:0/3673148880 shutdown_connections 2024-04-06T13:29:22.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.942+0000 7f16a230e700 1 -- 172.21.15.189:0/3673148880 wait complete. 2024-04-06T13:29:22.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.943+0000 7f16a230e700 1 Processor -- start 2024-04-06T13:29:22.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.943+0000 7f16a230e700 1 -- start start 2024-04-06T13:29:22.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.944+0000 7f16a230e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f169c06ac70 0x7f169c0b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.944+0000 7f16a230e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f169c158fe0 0x7f169c14f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.944+0000 7f16a230e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f169c15c2d0 0x7f169c15d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.944+0000 7f16a230e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0b5150 con 0x7f169c06ac70 2024-04-06T13:29:22.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.944+0000 7f16a230e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0b3b70 con 0x7f169c15c2d0 2024-04-06T13:29:22.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.944+0000 7f16a230e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0b3e50 con 0x7f169c158fe0 2024-04-06T13:29:22.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.944+0000 7f169bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f169c06ac70 0x7f169c0b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.944+0000 7f169bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f169c06ac70 0x7f169c0b4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54360/0 (socket says 172.21.15.189:54360) 2024-04-06T13:29:22.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.944+0000 7f169bfff700 1 -- 172.21.15.189:0/3548932148 learned_addr learned my addr 172.21.15.189:0/3548932148 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:22.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.944+0000 7f169bfff700 1 -- 172.21.15.189:0/3548932148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f169c158fe0 msgr2=0x7f169c14f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:22.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.944+0000 7f169bfff700 1 --2- 172.21.15.189:0/3548932148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f169c158fe0 0x7f169c14f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.944+0000 7f169bfff700 1 -- 172.21.15.189:0/3548932148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f169c15c2d0 msgr2=0x7f169c15d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:22.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.944+0000 7f169bfff700 1 --2- 172.21.15.189:0/3548932148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f169c15c2d0 0x7f169c15d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:22.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.944+0000 7f169bfff700 1 -- 172.21.15.189:0/3548932148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16900094a0 con 0x7f169c06ac70 2024-04-06T13:29:22.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.945+0000 7f169bfff700 1 --2- 172.21.15.189:0/3548932148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f169c06ac70 0x7f169c0b4c10 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f16900097c0 tx=0x7f1690004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:22.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.945+0000 7f1698ff9700 1 -- 172.21.15.189:0/3548932148 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16900045b0 con 0x7f169c06ac70 2024-04-06T13:29:22.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.945+0000 7f1698ff9700 1 -- 172.21.15.189:0/3548932148 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169001f070 con 0x7f169c06ac70 2024-04-06T13:29:22.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.945+0000 7f1698ff9700 1 -- 172.21.15.189:0/3548932148 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690011850 con 0x7f169c06ac70 2024-04-06T13:29:22.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.945+0000 7f16a230e700 1 -- 172.21.15.189:0/3548932148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169c0b4130 con 0x7f169c06ac70 2024-04-06T13:29:22.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.945+0000 7f16a230e700 1 -- 172.21.15.189:0/3548932148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f169c0b46f0 con 0x7f169c06ac70 2024-04-06T13:29:22.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.946+0000 7f1698ff9700 1 -- 172.21.15.189:0/3548932148 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f16900119b0 con 0x7f169c06ac70 2024-04-06T13:29:22.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.946+0000 7f16a230e700 1 -- 172.21.15.189:0/3548932148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169c0b4930 con 0x7f169c06ac70 2024-04-06T13:29:22.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.947+0000 7f1698ff9700 1 --2- 172.21.15.189:0/3548932148 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f168806adb0 0x7f168806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:22.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.947+0000 7f1698ff9700 1 -- 172.21.15.189:0/3548932148 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1690021070 con 0x7f169c06ac70 2024-04-06T13:29:22.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.947+0000 7f169b7fe700 1 --2- 172.21.15.189:0/3548932148 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f168806adb0 0x7f168806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:22.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.947+0000 7f169b7fe700 1 --2- 172.21.15.189:0/3548932148 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f168806adb0 0x7f168806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f168c000f60 tx=0x7f168c0079a0 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:22.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:22.951+0000 7f1698ff9700 1 -- 172.21.15.189:0/3548932148 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1690055cb0 con 0x7f169c06ac70 2024-04-06T13:29:23.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:23.189+0000 7f16a230e700 1 -- 172.21.15.189:0/3548932148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f169c067bb0 con 0x7f169c06ac70 2024-04-06T13:29:23.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:23.189+0000 7f1698ff9700 1 -- 172.21.15.189:0/3548932148 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v85) v1 ==== 53+0+73 (secure 0 0 0) 0x7f169002b020 con 0x7f169c06ac70 2024-04-06T13:29:23.189 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-06T13:29:23.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:23.191+0000 7f16a230e700 1 -- 172.21.15.189:0/3548932148 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f168806adb0 msgr2=0x7f168806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:23.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:23.191+0000 7f16a230e700 1 --2- 172.21.15.189:0/3548932148 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f168806adb0 0x7f168806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f168c000f60 tx=0x7f168c0079a0 comp rx=0 tx=0).stop 2024-04-06T13:29:23.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:23.191+0000 7f16a230e700 1 -- 172.21.15.189:0/3548932148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f169c06ac70 msgr2=0x7f169c0b4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:23.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:23.191+0000 7f16a230e700 1 --2- 172.21.15.189:0/3548932148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f169c06ac70 0x7f169c0b4c10 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f16900097c0 tx=0x7f1690004750 comp rx=0 tx=0).stop 2024-04-06T13:29:23.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:23.191+0000 7f16a230e700 1 -- 172.21.15.189:0/3548932148 shutdown_connections 2024-04-06T13:29:23.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:23.191+0000 7f16a230e700 1 --2- 172.21.15.189:0/3548932148 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f168806adb0 0x7f168806d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:23.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:23.191+0000 7f16a230e700 1 --2- 172.21.15.189:0/3548932148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f169c15c2d0 0x7f169c15d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:23.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:23.191+0000 7f16a230e700 1 --2- 172.21.15.189:0/3548932148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f169c06ac70 0x7f169c0b4c10 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:23.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:23.191+0000 7f16a230e700 1 --2- 172.21.15.189:0/3548932148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f169c158fe0 0x7f169c14f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:23.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:23.191+0000 7f16a230e700 1 -- 172.21.15.189:0/3548932148 >> 172.21.15.189:0/3548932148 conn(0x7f169c0d00b0 msgr2=0x7f169c06eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:23.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:23.191+0000 7f16a230e700 1 -- 172.21.15.189:0/3548932148 shutdown_connections 2024-04-06T13:29:23.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:23.192+0000 7f16a230e700 1 -- 172.21.15.189:0/3548932148 wait complete. 2024-04-06T13:29:23.204 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-06T13:29:23.204 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T13:29:23.204 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-06T13:29:23.204 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-06T13:29:23.204 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T13:29:23.204 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-06T13:29:23.204 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-06T13:29:23.204 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T13:29:23.204 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:29:23.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:29:23.204 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-06T13:29:23.222 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T13:29:23.222 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T13:29:23.222 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T13:29:23.222 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:29:23.223 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T13:29:23.223 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:29:23.223 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:29:23.223 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:29:23.223 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-06T13:29:23.223 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:29:23.223 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:29:23.223 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-06T13:29:23.223 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:29:23.223 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:29:23.223 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T13:29:23.223 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:29:23.223 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T13:29:23.223 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 37726sec preferred_lft 37726sec 2024-04-06T13:29:23.223 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::8592:df2c:92e2:dace/64 scope link noprefixroute 2024-04-06T13:29:23.224 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:29:23.224 INFO:teuthology.orchestra.run.smithi189.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T13:29:23.224 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff 2024-04-06T13:29:23.224 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-06T13:29:23.224 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:29:23.224 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::88e:b6ff:fe6f:1cb0/64 scope link 2024-04-06T13:29:23.224 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:29:23.224 INFO:teuthology.orchestra.run.smithi189.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T13:29:23.224 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 62:a5:8d:c5:1a:9c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T13:29:23.224 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::60a5:8dff:fec5:1a9c/64 scope link 2024-04-06T13:29:23.224 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:29:23.224 INFO:teuthology.orchestra.run.smithi189.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T13:29:23.224 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-06T13:29:23.224 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::1c7e:c6ff:febb:1dab/64 scope link 2024-04-06T13:29:23.224 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:29:23.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:29:23.225 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T13:29:23.291 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-06T13:29:23.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:29:23.293 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-06T13:29:23.353 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-06T13:29:23.353 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T13:29:23.353 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T13:29:23.353 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T13:29:23.353 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T13:29:23.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:29:23.353 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T13:29:23.353 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T13:29:23.353 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T13:29:23.354 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T13:29:23.354 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T13:29:23.354 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T13:29:23.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:29:23.460 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T13:29:23.477 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-06T13:29:23.477 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-06T13:29:23.478 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:29:23.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:29:23.478 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-04-06T13:29:23.545 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T13:29:23.545 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-06T13:29:23.602 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:29:23.603 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T13:29:23.603 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-06T13:29:23.659 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T13:29:23.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:29:23.660 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:29:23.746 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:29:23.746 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:29:23.746 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:29:23.755 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:29:23.755 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:29:23.756 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:29:23.756 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T13:29:23.756 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T13:29:23.774 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:29:23.774 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:29:23.774 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:29:23.774 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T13:29:23.774 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T13:29:23.775 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790" 2024-04-06T13:29:23.775 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2024-04-06T13:29:23.775 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e30041de-e9a3-4351-8ca7-29d7f3e3728c" will pass to kernel 2024-04-06T13:29:23.776 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T13:29:23.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:29:23.777 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-06T13:29:23.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:29:23.805 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T13:29:23.894 INFO:teuthology.orchestra.run.smithi189.stdout:client9340 2024-04-06T13:29:23.894 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:29:23.894 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:29:23.912 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-06T13:29:24.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.076+0000 7f14a32cd700 1 Processor -- start 2024-04-06T13:29:24.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.076+0000 7f14a32cd700 1 -- start start 2024-04-06T13:29:24.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f14a32cd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f149c0b4660 0x7f149c0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:24.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f14a32cd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f149c0b2fc0 0x7f149c0b3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:24.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f14a32cd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f149c0b5030 con 0x7f149c15a550 2024-04-06T13:29:24.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f14a32cd700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f149c0cdbe0 con 0x7f149c0b4660 2024-04-06T13:29:24.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f14a32cd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f149c15a940 con 0x7f149c0b2fc0 2024-04-06T13:29:24.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f14a1069700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f149c15a550 0x7f149c0cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51994/0 (socket says 172.21.15.189:51994) 2024-04-06T13:29:24.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f14a1069700 1 -- 172.21.15.189:0/2966956552 learned_addr learned my addr 172.21.15.189:0/2966956552 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:24.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f1493fff700 1 -- 172.21.15.189:0/2966956552 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297021013 0 0) 0x7f149c0b5030 con 0x7f149c15a550 2024-04-06T13:29:24.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f1493fff700 1 -- 172.21.15.189:0/2966956552 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14880035b0 con 0x7f149c15a550 2024-04-06T13:29:24.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f14a0868700 1 --2- 172.21.15.189:0/2966956552 >> v2:172.21.15.192:3300/0 conn(0x7f149c0b4660 0x7f149c0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:24.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f14a186a700 1 --2- 172.21.15.189:0/2966956552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f149c0b2fc0 0x7f149c0b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:24.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f1493fff700 1 -- 172.21.15.189:0/2966956552 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 215109141 0 0) 0x7f14880035b0 con 0x7f149c15a550 2024-04-06T13:29:24.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f1493fff700 1 -- 172.21.15.189:0/2966956552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f149c0b2fc0 msgr2=0x7f149c0b3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:24.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f1493fff700 1 --2- 172.21.15.189:0/2966956552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f149c0b2fc0 0x7f149c0b3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f1493fff700 1 -- 172.21.15.189:0/2966956552 >> v2:172.21.15.192:3300/0 conn(0x7f149c0b4660 msgr2=0x7f149c0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:24.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f1493fff700 1 --2- 172.21.15.189:0/2966956552 >> v2:172.21.15.192:3300/0 conn(0x7f149c0b4660 0x7f149c0b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f1493fff700 1 -- 172.21.15.189:0/2966956552 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f149c15ac00 con 0x7f149c15a550 2024-04-06T13:29:24.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f1493fff700 1 -- 172.21.15.189:0/2966956552 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f1498001670 con 0x7f149c15a550 2024-04-06T13:29:24.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f1493fff700 1 -- 172.21.15.189:0/2966956552 >> v1:172.21.15.189:6789/0 conn(0x7f149c15a550 legacy=0x7f149c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:24.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f1493fff700 1 --2- 172.21.15.189:0/2966956552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1488003a00 0x7f1488003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:24.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f1493fff700 1 --2- 172.21.15.189:0/2966956552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1488004850 0x7f1488004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:24.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f1493fff700 1 --2- 172.21.15.189:0/2966956552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f149c0b2fc0 0x7f1488005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:24.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f1493fff700 1 -- 172.21.15.189:0/2966956552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149c0cdbe0 con 0x7f1488003a00 2024-04-06T13:29:24.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.077+0000 7f1493fff700 1 -- 172.21.15.189:0/2966956552 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149c15a940 con 0x7f1488004850 2024-04-06T13:29:24.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.078+0000 7f1493fff700 1 -- 172.21.15.189:0/2966956552 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148800ba10 con 0x7f149c0b2fc0 2024-04-06T13:29:24.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.078+0000 7f14a1069700 1 --2- 172.21.15.189:0/2966956552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1488003a00 0x7f1488003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:24.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.078+0000 7f14a0868700 1 --2- 172.21.15.189:0/2966956552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1488004850 0x7f1488004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:24.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.078+0000 7f14a186a700 1 --2- 172.21.15.189:0/2966956552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f149c0b2fc0 0x7f1488005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:24.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.078+0000 7f14a1069700 1 -- 172.21.15.189:0/2966956552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f149c0b2fc0 msgr2=0x7f1488005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:24.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.078+0000 7f14a1069700 1 --2- 172.21.15.189:0/2966956552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f149c0b2fc0 0x7f1488005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.078+0000 7f14a1069700 1 -- 172.21.15.189:0/2966956552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1488004850 msgr2=0x7f1488004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:24.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.078+0000 7f14a1069700 1 --2- 172.21.15.189:0/2966956552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1488004850 0x7f1488004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.078+0000 7f14a1069700 1 -- 172.21.15.189:0/2966956552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14880035b0 con 0x7f1488003a00 2024-04-06T13:29:24.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.078+0000 7f14a186a700 1 --2- 172.21.15.189:0/2966956552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f149c0b2fc0 0x7f1488005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:29:24.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.078+0000 7f14a1069700 1 --2- 172.21.15.189:0/2966956552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1488003a00 0x7f1488003e20 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f149800c370 tx=0x7f1488005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c479784d73afc8ed server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:24.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.078+0000 7f1493fff700 1 -- 172.21.15.189:0/2966956552 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498007c60 con 0x7f1488003a00 2024-04-06T13:29:24.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.078+0000 7f1493fff700 1 -- 172.21.15.189:0/2966956552 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1498005960 con 0x7f1488003a00 2024-04-06T13:29:24.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.078+0000 7f1493fff700 1 -- 172.21.15.189:0/2966956552 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498005d10 con 0x7f1488003a00 2024-04-06T13:29:24.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.079+0000 7f14a32cd700 1 -- 172.21.15.189:0/2966956552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1488003a00 msgr2=0x7f1488003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:24.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.079+0000 7f14a32cd700 1 --2- 172.21.15.189:0/2966956552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1488003a00 0x7f1488003e20 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f149800c370 tx=0x7f1488005880 comp rx=0 tx=0).stop 2024-04-06T13:29:24.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.079+0000 7f14a186a700 1 -- 172.21.15.189:0/2966956552 reap_dead start 2024-04-06T13:29:24.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.079+0000 7f14a32cd700 1 -- 172.21.15.189:0/2966956552 shutdown_connections 2024-04-06T13:29:24.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.079+0000 7f14a32cd700 1 -- 172.21.15.189:0/2966956552 >> 172.21.15.189:0/2966956552 conn(0x7f149c0d00b0 msgr2=0x7f149c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:24.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.079+0000 7f14a32cd700 1 -- 172.21.15.189:0/2966956552 shutdown_connections 2024-04-06T13:29:24.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.079+0000 7f14a32cd700 1 -- 172.21.15.189:0/2966956552 wait complete. 2024-04-06T13:29:24.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.080+0000 7f14a32cd700 1 Processor -- start 2024-04-06T13:29:24.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.080+0000 7f14a32cd700 1 -- start start 2024-04-06T13:29:24.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f14a32cd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1488003a00 0x7f149c14ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:24.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f14a32cd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f149c150380 0x7f149c06d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:24.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f14a32cd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f149c150800 0x7f149c06db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:24.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f14a32cd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149c153a10 con 0x7f1488003a00 2024-04-06T13:29:24.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f14a32cd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149c153cf0 con 0x7f149c150800 2024-04-06T13:29:24.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f14a32cd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f149c153fd0 con 0x7f149c150380 2024-04-06T13:29:24.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f14a1069700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1488003a00 0x7f149c14ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:24.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f14a1069700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1488003a00 0x7f149c14ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54370/0 (socket says 172.21.15.189:54370) 2024-04-06T13:29:24.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f14a1069700 1 -- 172.21.15.189:0/216823508 learned_addr learned my addr 172.21.15.189:0/216823508 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:24.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f14a186a700 1 --2- 172.21.15.189:0/216823508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f149c150800 0x7f149c06db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:24.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f14a0868700 1 --2- 172.21.15.189:0/216823508 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f149c150380 0x7f149c06d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:24.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f14a1069700 1 -- 172.21.15.189:0/216823508 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f149c150380 msgr2=0x7f149c06d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:24.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f14a1069700 1 --2- 172.21.15.189:0/216823508 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f149c150380 0x7f149c06d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f14a1069700 1 -- 172.21.15.189:0/216823508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f149c150800 msgr2=0x7f149c06db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:24.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f14a1069700 1 --2- 172.21.15.189:0/216823508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f149c150800 0x7f149c06db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f14a1069700 1 -- 172.21.15.189:0/216823508 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f149c1542b0 con 0x7f1488003a00 2024-04-06T13:29:24.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f14a1069700 1 --2- 172.21.15.189:0/216823508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1488003a00 0x7f149c14ce80 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f14880046a0 tx=0x7f1498000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:24.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f1491ffb700 1 -- 172.21.15.189:0/216823508 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498004870 con 0x7f1488003a00 2024-04-06T13:29:24.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f1491ffb700 1 -- 172.21.15.189:0/216823508 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14980049f0 con 0x7f1488003a00 2024-04-06T13:29:24.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f14a32cd700 1 -- 172.21.15.189:0/216823508 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f149c153a10 con 0x7f1488003a00 2024-04-06T13:29:24.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.081+0000 7f1491ffb700 1 -- 172.21.15.189:0/216823508 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498004da0 con 0x7f1488003a00 2024-04-06T13:29:24.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.082+0000 7f14a32cd700 1 -- 172.21.15.189:0/216823508 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f149c14d3c0 con 0x7f1488003a00 2024-04-06T13:29:24.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.082+0000 7f14a32cd700 1 -- 172.21.15.189:0/216823508 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f149c067bb0 con 0x7f1488003a00 2024-04-06T13:29:24.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.083+0000 7f1491ffb700 1 -- 172.21.15.189:0/216823508 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f14980070d0 con 0x7f1488003a00 2024-04-06T13:29:24.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.083+0000 7f1491ffb700 1 --2- 172.21.15.189:0/216823508 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f148006adb0 0x7f148006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:24.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.083+0000 7f1491ffb700 1 -- 172.21.15.189:0/216823508 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f149808aa80 con 0x7f1488003a00 2024-04-06T13:29:24.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.083+0000 7f14a0868700 1 --2- 172.21.15.189:0/216823508 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f148006adb0 0x7f148006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:24.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.084+0000 7f14a0868700 1 --2- 172.21.15.189:0/216823508 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f148006adb0 0x7f148006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f148c004510 tx=0x7f148c008040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:24.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.088+0000 7f1491ffb700 1 -- 172.21.15.189:0/216823508 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1498055a70 con 0x7f1488003a00 2024-04-06T13:29:24.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.308+0000 7f14a32cd700 1 -- 172.21.15.189:0/216823508 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f149c15b4a0 con 0x7f148006adb0 2024-04-06T13:29:24.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.309+0000 7f1491ffb700 1 -- 172.21.15.189:0/216823508 <== mgr.9034 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f148c006c60 con 0x7f148006adb0 2024-04-06T13:29:24.309 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T13:29:24.309 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:29:24.309 INFO:teuthology.orchestra.run.smithi189.stdout: "name": "cephfs" 2024-04-06T13:29:24.309 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:29:24.309 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T13:29:24.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.311+0000 7f14a32cd700 1 -- 172.21.15.189:0/216823508 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f148006adb0 msgr2=0x7f148006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:24.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.311+0000 7f14a32cd700 1 --2- 172.21.15.189:0/216823508 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f148006adb0 0x7f148006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f148c004510 tx=0x7f148c008040 comp rx=0 tx=0).stop 2024-04-06T13:29:24.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.311+0000 7f14a32cd700 1 -- 172.21.15.189:0/216823508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1488003a00 msgr2=0x7f149c14ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:24.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.311+0000 7f14a32cd700 1 --2- 172.21.15.189:0/216823508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1488003a00 0x7f149c14ce80 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f14880046a0 tx=0x7f1498000fe0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.311+0000 7f14a32cd700 1 -- 172.21.15.189:0/216823508 shutdown_connections 2024-04-06T13:29:24.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.311+0000 7f14a32cd700 1 --2- 172.21.15.189:0/216823508 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f148006adb0 0x7f148006d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.311+0000 7f14a32cd700 1 --2- 172.21.15.189:0/216823508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f149c150800 0x7f149c06db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.311+0000 7f14a32cd700 1 --2- 172.21.15.189:0/216823508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1488003a00 0x7f149c14ce80 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.311+0000 7f14a32cd700 1 --2- 172.21.15.189:0/216823508 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f149c150380 0x7f149c06d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.311+0000 7f14a32cd700 1 -- 172.21.15.189:0/216823508 >> 172.21.15.189:0/216823508 conn(0x7f149c0d00b0 msgr2=0x7f149c159440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:24.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.311+0000 7f14a32cd700 1 -- 172.21.15.189:0/216823508 shutdown_connections 2024-04-06T13:29:24.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.311+0000 7f14a32cd700 1 -- 172.21.15.189:0/216823508 wait complete. 2024-04-06T13:29:24.323 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-06T13:29:24.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.448+0000 7feb4db41700 1 Processor -- start 2024-04-06T13:29:24.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.448+0000 7feb4db41700 1 -- start start 2024-04-06T13:29:24.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.449+0000 7feb4db41700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7feb4806ae40 0x7feb480b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:24.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.449+0000 7feb4db41700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb4815ddf0 0x7feb4815e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:24.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.449+0000 7feb4db41700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb480b3990 con 0x7feb4815d7b0 2024-04-06T13:29:24.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.449+0000 7feb4db41700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb480b3060 con 0x7feb4806ae40 2024-04-06T13:29:24.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.449+0000 7feb4db41700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb480b3340 con 0x7feb4815ddf0 2024-04-06T13:29:24.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.449+0000 7feb477fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7feb4815d7b0 0x7feb4806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51996/0 (socket says 172.21.15.189:51996) 2024-04-06T13:29:24.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.449+0000 7feb477fe700 1 -- 172.21.15.189:0/334553338 learned_addr learned my addr 172.21.15.189:0/334553338 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:24.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.449+0000 7feb47fff700 1 --2- 172.21.15.189:0/334553338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb4815ddf0 0x7feb4815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:24.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.449+0000 7feb46ffd700 1 --2- 172.21.15.189:0/334553338 >> v2:172.21.15.192:3300/0 conn(0x7feb4806ae40 0x7feb480b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:24.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.449+0000 7feb467fc700 1 -- 172.21.15.189:0/334553338 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409442511 0 0) 0x7feb480b3990 con 0x7feb4815d7b0 2024-04-06T13:29:24.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.449+0000 7feb467fc700 1 -- 172.21.15.189:0/334553338 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb300035b0 con 0x7feb4815d7b0 2024-04-06T13:29:24.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.449+0000 7feb467fc700 1 -- 172.21.15.189:0/334553338 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2032096026 0 0) 0x7feb300035b0 con 0x7feb4815d7b0 2024-04-06T13:29:24.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.449+0000 7feb47fff700 1 -- 172.21.15.189:0/334553338 >> v2:172.21.15.192:3300/0 conn(0x7feb4806ae40 msgr2=0x7feb480b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:24.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.449+0000 7feb47fff700 1 --2- 172.21.15.189:0/334553338 >> v2:172.21.15.192:3300/0 conn(0x7feb4806ae40 0x7feb480b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.449+0000 7feb47fff700 1 -- 172.21.15.189:0/334553338 >> v1:172.21.15.189:6789/0 conn(0x7feb4815d7b0 legacy=0x7feb4806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:24.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.449+0000 7feb47fff700 1 -- 172.21.15.189:0/334553338 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb480b3620 con 0x7feb4815ddf0 2024-04-06T13:29:24.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.449+0000 7feb46ffd700 1 --2- 172.21.15.189:0/334553338 >> v2:172.21.15.192:3300/0 conn(0x7feb4806ae40 0x7feb480b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:29:24.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.450+0000 7feb47fff700 1 --2- 172.21.15.189:0/334553338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb4815ddf0 0x7feb4815e280 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7feb40009d90 tx=0x7feb4000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf96c3bd07c15efe server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:24.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.450+0000 7feb467fc700 1 -- 172.21.15.189:0/334553338 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4000e070 con 0x7feb4815ddf0 2024-04-06T13:29:24.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.450+0000 7feb467fc700 1 -- 172.21.15.189:0/334553338 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb40003a20 con 0x7feb4815ddf0 2024-04-06T13:29:24.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.450+0000 7feb467fc700 1 -- 172.21.15.189:0/334553338 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c001670 con 0x7feb4815ddf0 2024-04-06T13:29:24.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.450+0000 7feb4db41700 1 -- 172.21.15.189:0/334553338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb4815ddf0 msgr2=0x7feb4815e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:24.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.450+0000 7feb4db41700 1 --2- 172.21.15.189:0/334553338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb4815ddf0 0x7feb4815e280 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7feb40009d90 tx=0x7feb4000b330 comp rx=0 tx=0).stop 2024-04-06T13:29:24.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.450+0000 7feb4db41700 1 -- 172.21.15.189:0/334553338 shutdown_connections 2024-04-06T13:29:24.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.450+0000 7feb4db41700 1 --2- 172.21.15.189:0/334553338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb4815ddf0 0x7feb4815e280 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.450+0000 7feb4db41700 1 --2- 172.21.15.189:0/334553338 >> v2:172.21.15.192:3300/0 conn(0x7feb4806ae40 0x7feb480b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.450+0000 7feb4db41700 1 -- 172.21.15.189:0/334553338 >> 172.21.15.189:0/334553338 conn(0x7feb480d00b0 msgr2=0x7feb480d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:24.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.450+0000 7feb4db41700 1 -- 172.21.15.189:0/334553338 shutdown_connections 2024-04-06T13:29:24.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.450+0000 7feb4db41700 1 -- 172.21.15.189:0/334553338 wait complete. 2024-04-06T13:29:24.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.451+0000 7feb4db41700 1 Processor -- start 2024-04-06T13:29:24.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.452+0000 7feb4db41700 1 -- start start 2024-04-06T13:29:24.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.452+0000 7feb4db41700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb4806ae40 0x7feb480cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:24.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.452+0000 7feb4db41700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb4815d7b0 0x7feb480cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:24.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.452+0000 7feb4db41700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb4815ddf0 0x7feb4806f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:24.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.452+0000 7feb4db41700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb480cdd00 con 0x7feb4806ae40 2024-04-06T13:29:24.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.452+0000 7feb4db41700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4815ba40 con 0x7feb4815ddf0 2024-04-06T13:29:24.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.452+0000 7feb4db41700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb480cda20 con 0x7feb4815d7b0 2024-04-06T13:29:24.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.453+0000 7feb477fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb4806ae40 0x7feb480cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:24.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.453+0000 7feb477fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb4806ae40 0x7feb480cceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54372/0 (socket says 172.21.15.189:54372) 2024-04-06T13:29:24.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.453+0000 7feb477fe700 1 -- 172.21.15.189:0/4140444404 learned_addr learned my addr 172.21.15.189:0/4140444404 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:24.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.453+0000 7feb47fff700 1 --2- 172.21.15.189:0/4140444404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb4815ddf0 0x7feb4806f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:24.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.453+0000 7feb46ffd700 1 --2- 172.21.15.189:0/4140444404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb4815d7b0 0x7feb480cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:24.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.453+0000 7feb477fe700 1 -- 172.21.15.189:0/4140444404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb4815d7b0 msgr2=0x7feb480cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:24.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.453+0000 7feb477fe700 1 --2- 172.21.15.189:0/4140444404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb4815d7b0 0x7feb480cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.453+0000 7feb477fe700 1 -- 172.21.15.189:0/4140444404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb4815ddf0 msgr2=0x7feb4806f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:24.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.453+0000 7feb477fe700 1 --2- 172.21.15.189:0/4140444404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb4815ddf0 0x7feb4806f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.453+0000 7feb477fe700 1 -- 172.21.15.189:0/4140444404 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb380094a0 con 0x7feb4806ae40 2024-04-06T13:29:24.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.453+0000 7feb46ffd700 1 --2- 172.21.15.189:0/4140444404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb4815d7b0 0x7feb480cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:29:24.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.453+0000 7feb477fe700 1 --2- 172.21.15.189:0/4140444404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb4806ae40 0x7feb480cceb0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7feb3c00c7b0 tx=0x7feb3c00cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:24.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.453+0000 7feb2ffff700 1 -- 172.21.15.189:0/4140444404 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c004c80 con 0x7feb4806ae40 2024-04-06T13:29:24.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.453+0000 7feb2ffff700 1 -- 172.21.15.189:0/4140444404 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3c004de0 con 0x7feb4806ae40 2024-04-06T13:29:24.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.453+0000 7feb4db41700 1 -- 172.21.15.189:0/4140444404 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb40009a70 con 0x7feb4806ae40 2024-04-06T13:29:24.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.453+0000 7feb2ffff700 1 -- 172.21.15.189:0/4140444404 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c0072f0 con 0x7feb4806ae40 2024-04-06T13:29:24.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.453+0000 7feb4db41700 1 -- 172.21.15.189:0/4140444404 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb4815b6d0 con 0x7feb4806ae40 2024-04-06T13:29:24.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.454+0000 7feb4db41700 1 -- 172.21.15.189:0/4140444404 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb48067bb0 con 0x7feb4806ae40 2024-04-06T13:29:24.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.455+0000 7feb2ffff700 1 -- 172.21.15.189:0/4140444404 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 108) v1 ==== 89247+0+0 (secure 0 0 0) 0x7feb3c007450 con 0x7feb4806ae40 2024-04-06T13:29:24.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.455+0000 7feb2ffff700 1 --2- 172.21.15.189:0/4140444404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb3406adb0 0x7feb3406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:24.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.455+0000 7feb46ffd700 1 --2- 172.21.15.189:0/4140444404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb3406adb0 0x7feb3406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:24.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.455+0000 7feb2ffff700 1 -- 172.21.15.189:0/4140444404 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6073+0+0 (secure 0 0 0) 0x7feb3c088a00 con 0x7feb4806ae40 2024-04-06T13:29:24.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.456+0000 7feb46ffd700 1 --2- 172.21.15.189:0/4140444404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb3406adb0 0x7feb3406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feb4815ab80 tx=0x7feb38008040 comp rx=0 tx=0).ready entity=mgr.9034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:24.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.460+0000 7feb2ffff700 1 -- 172.21.15.189:0/4140444404 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7feb3c0539f0 con 0x7feb4806ae40 2024-04-06T13:29:24.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.710+0000 7feb4db41700 1 -- 172.21.15.189:0/4140444404 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7feb48074990 con 0x7feb4806ae40 2024-04-06T13:29:24.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.893+0000 7feb2ffff700 1 -- 172.21.15.189:0/4140444404 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 109) v1 ==== 89264+0+0 (secure 0 0 0) 0x7feb3c053dc0 con 0x7feb4806ae40 2024-04-06T13:29:24.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.894+0000 7feb2ffff700 1 -- 172.21.15.189:0/4140444404 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v109) v1 ==== 92+0+0 (secure 0 0 0) 0x7feb3c073080 con 0x7feb4806ae40 2024-04-06T13:29:24.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.897+0000 7feb4db41700 1 -- 172.21.15.189:0/4140444404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb3406adb0 msgr2=0x7feb3406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:24.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.897+0000 7feb4db41700 1 --2- 172.21.15.189:0/4140444404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb3406adb0 0x7feb3406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feb4815ab80 tx=0x7feb38008040 comp rx=0 tx=0).stop 2024-04-06T13:29:24.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.897+0000 7feb4db41700 1 -- 172.21.15.189:0/4140444404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb4806ae40 msgr2=0x7feb480cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:24.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.897+0000 7feb4db41700 1 --2- 172.21.15.189:0/4140444404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb4806ae40 0x7feb480cceb0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7feb3c00c7b0 tx=0x7feb3c00cac0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.897+0000 7feb4db41700 1 -- 172.21.15.189:0/4140444404 shutdown_connections 2024-04-06T13:29:24.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.897+0000 7feb4db41700 1 --2- 172.21.15.189:0/4140444404 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7feb3406adb0 0x7feb3406d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.897+0000 7feb4db41700 1 --2- 172.21.15.189:0/4140444404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb4815ddf0 0x7feb4806f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.897+0000 7feb4db41700 1 --2- 172.21.15.189:0/4140444404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb4806ae40 0x7feb480cceb0 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.897+0000 7feb4db41700 1 --2- 172.21.15.189:0/4140444404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb4815d7b0 0x7feb480cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:24.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.897+0000 7feb4db41700 1 -- 172.21.15.189:0/4140444404 >> 172.21.15.189:0/4140444404 conn(0x7feb480d00b0 msgr2=0x7feb48158ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:24.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.897+0000 7feb4db41700 1 -- 172.21.15.189:0/4140444404 shutdown_connections 2024-04-06T13:29:24.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:24.897+0000 7feb4db41700 1 -- 172.21.15.189:0/4140444404 wait complete. 2024-04-06T13:29:24.914 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-06T13:29:25.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.044+0000 7f13e1c28700 1 Processor -- start 2024-04-06T13:29:25.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.045+0000 7f13e1c28700 1 -- start start 2024-04-06T13:29:25.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.045+0000 7f13e1c28700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f13dc0cb870 0x7f13dc15c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:25.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.045+0000 7f13e1c28700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13dc0cbd80 con 0x7f13dc0cfa00 2024-04-06T13:29:25.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.045+0000 7f13e1c28700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13dc0cef20 con 0x7f13dc0d11b0 2024-04-06T13:29:25.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.045+0000 7f13e1c28700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13dc0ccdc0 con 0x7f13dc0cb870 2024-04-06T13:29:25.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.046+0000 7f13dbfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f13dc0cfa00 0x7f13dc0cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52024/0 (socket says 172.21.15.189:52024) 2024-04-06T13:29:25.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.046+0000 7f13dbfff700 1 -- 172.21.15.189:0/3514773746 learned_addr learned my addr 172.21.15.189:0/3514773746 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:25.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.046+0000 7f13d2ffd700 1 --2- 172.21.15.189:0/3514773746 >> v2:172.21.15.192:3300/0 conn(0x7f13dc0cb870 0x7f13dc15c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:25.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.046+0000 7f13daffd700 1 -- 172.21.15.189:0/3514773746 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112755686 0 0) 0x7f13dc0cbd80 con 0x7f13dc0cfa00 2024-04-06T13:29:25.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.046+0000 7f13daffd700 1 -- 172.21.15.189:0/3514773746 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c80035b0 con 0x7f13dc0cfa00 2024-04-06T13:29:25.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.046+0000 7f13d2ffd700 1 -- 172.21.15.189:0/3514773746 >> v1:172.21.15.192:6789/0 conn(0x7f13dc0d11b0 legacy=0x7f13dc0d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:25.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.046+0000 7f13d2ffd700 1 -- 172.21.15.189:0/3514773746 >> v1:172.21.15.189:6789/0 conn(0x7f13dc0cfa00 legacy=0x7f13dc0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:25.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.046+0000 7f13d2ffd700 1 -- 172.21.15.189:0/3514773746 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13dc0cf200 con 0x7f13dc0cb870 2024-04-06T13:29:25.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.046+0000 7f13d2ffd700 1 --2- 172.21.15.189:0/3514773746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13dc0cb870 0x7f13dc15c280 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f13c4009840 tx=0x7f13c4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4a58d242f17e4aa server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:25.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.046+0000 7f13daffd700 1 -- 172.21.15.189:0/3514773746 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c4011830 con 0x7f13dc0cb870 2024-04-06T13:29:25.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.046+0000 7f13daffd700 1 -- 172.21.15.189:0/3514773746 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c4011990 con 0x7f13dc0cb870 2024-04-06T13:29:25.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.046+0000 7f13daffd700 1 -- 172.21.15.189:0/3514773746 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c4011d00 con 0x7f13dc0cb870 2024-04-06T13:29:25.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.047+0000 7f13e1c28700 1 -- 172.21.15.189:0/3514773746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13dc0cb870 msgr2=0x7f13dc15c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:25.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.047+0000 7f13e1c28700 1 --2- 172.21.15.189:0/3514773746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13dc0cb870 0x7f13dc15c280 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f13c4009840 tx=0x7f13c4009b50 comp rx=0 tx=0).stop 2024-04-06T13:29:25.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.047+0000 7f13e1c28700 1 -- 172.21.15.189:0/3514773746 shutdown_connections 2024-04-06T13:29:25.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.047+0000 7f13e1c28700 1 --2- 172.21.15.189:0/3514773746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13dc0cb870 0x7f13dc15c280 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.047+0000 7f13e1c28700 1 -- 172.21.15.189:0/3514773746 >> 172.21.15.189:0/3514773746 conn(0x7f13dc0b2db0 msgr2=0x7f13dc0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:25.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.047+0000 7f13e1c28700 1 -- 172.21.15.189:0/3514773746 shutdown_connections 2024-04-06T13:29:25.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.047+0000 7f13e1c28700 1 -- 172.21.15.189:0/3514773746 wait complete. 2024-04-06T13:29:25.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.048+0000 7f13e1c28700 1 Processor -- start 2024-04-06T13:29:25.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.048+0000 7f13e1c28700 1 -- start start 2024-04-06T13:29:25.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.049+0000 7f13e1c28700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13dc0cb870 0x7f13dc0affd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:25.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.049+0000 7f13e1c28700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13dc0cfa00 0x7f13dc1515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:25.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.049+0000 7f13e1c28700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13dc0d11b0 0x7f13dc06c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:25.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.049+0000 7f13e1c28700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc0c5da0 con 0x7f13dc0d11b0 2024-04-06T13:29:25.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.049+0000 7f13e1c28700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc0c6320 con 0x7f13dc0cb870 2024-04-06T13:29:25.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.049+0000 7f13e1c28700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc06d400 con 0x7f13dc0cfa00 2024-04-06T13:29:25.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.049+0000 7f13dbfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13dc0d11b0 0x7f13dc06c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:25.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.049+0000 7f13dbfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13dc0d11b0 0x7f13dc06c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54406/0 (socket says 172.21.15.189:54406) 2024-04-06T13:29:25.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.049+0000 7f13dbfff700 1 -- 172.21.15.189:0/1025026881 learned_addr learned my addr 172.21.15.189:0/1025026881 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:25.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.049+0000 7f13db7fe700 1 --2- 172.21.15.189:0/1025026881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13dc0cb870 0x7f13dc0affd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:25.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.049+0000 7f13d2ffd700 1 --2- 172.21.15.189:0/1025026881 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13dc0cfa00 0x7f13dc1515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:25.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.050+0000 7f13dbfff700 1 -- 172.21.15.189:0/1025026881 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13dc0cfa00 msgr2=0x7f13dc1515c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:25.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.050+0000 7f13dbfff700 1 --2- 172.21.15.189:0/1025026881 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13dc0cfa00 0x7f13dc1515c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.050+0000 7f13dbfff700 1 -- 172.21.15.189:0/1025026881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13dc0cb870 msgr2=0x7f13dc0affd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:25.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.050+0000 7f13dbfff700 1 --2- 172.21.15.189:0/1025026881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13dc0cb870 0x7f13dc0affd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.050+0000 7f13dbfff700 1 -- 172.21.15.189:0/1025026881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c40094a0 con 0x7f13dc0d11b0 2024-04-06T13:29:25.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.050+0000 7f13dbfff700 1 --2- 172.21.15.189:0/1025026881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13dc0d11b0 0x7f13dc06c0d0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f13d400cb40 tx=0x7f13d400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:25.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.050+0000 7f13d8ff9700 1 -- 172.21.15.189:0/1025026881 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d400a980 con 0x7f13dc0d11b0 2024-04-06T13:29:25.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.050+0000 7f13d8ff9700 1 -- 172.21.15.189:0/1025026881 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d400aae0 con 0x7f13dc0d11b0 2024-04-06T13:29:25.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.050+0000 7f13e1c28700 1 -- 172.21.15.189:0/1025026881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13dc06d6c0 con 0x7f13dc0d11b0 2024-04-06T13:29:25.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.050+0000 7f13d8ff9700 1 -- 172.21.15.189:0/1025026881 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d400ae50 con 0x7f13dc0d11b0 2024-04-06T13:29:25.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.050+0000 7f13e1c28700 1 -- 172.21.15.189:0/1025026881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13dc06d040 con 0x7f13dc0d11b0 2024-04-06T13:29:25.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.051+0000 7f13e1c28700 1 -- 172.21.15.189:0/1025026881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13dc074990 con 0x7f13dc0d11b0 2024-04-06T13:29:25.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.052+0000 7f13d8ff9700 1 -- 172.21.15.189:0/1025026881 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 109) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f13d401b3b0 con 0x7f13dc0d11b0 2024-04-06T13:29:25.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.052+0000 7f13d8ff9700 1 --2- 172.21.15.189:0/1025026881 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13bc06ae00 0x7f13bc06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:25.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.052+0000 7f13db7fe700 1 -- 172.21.15.189:0/1025026881 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13bc06ae00 msgr2=0x7f13bc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:25.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.052+0000 7f13d8ff9700 1 -- 172.21.15.189:0/1025026881 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f13d4089670 con 0x7f13dc0d11b0 2024-04-06T13:29:25.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.052+0000 7f13db7fe700 1 --2- 172.21.15.189:0/1025026881 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13bc06ae00 0x7f13bc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:29:25.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.056+0000 7f13d8ff9700 1 -- 172.21.15.189:0/1025026881 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f13d4054710 con 0x7f13dc0d11b0 2024-04-06T13:29:25.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.253+0000 7f13db7fe700 1 -- 172.21.15.189:0/1025026881 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13bc06ae00 msgr2=0x7f13bc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.253+0000 7f13db7fe700 1 --2- 172.21.15.189:0/1025026881 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13bc06ae00 0x7f13bc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:29:25.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.258+0000 7f13e1c28700 1 -- 172.21.15.189:0/1025026881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f13dc0c6460 con 0x7f13dc0d11b0 2024-04-06T13:29:25.264 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.264+0000 7f13d8ff9700 1 -- 172.21.15.189:0/1025026881 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v65)=0 v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7f13d4057d30 con 0x7f13dc0d11b0 2024-04-06T13:29:25.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.266+0000 7f13e1c28700 1 -- 172.21.15.189:0/1025026881 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13bc06ae00 msgr2=0x7f13bc06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:29:25.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.266+0000 7f13e1c28700 1 --2- 172.21.15.189:0/1025026881 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13bc06ae00 0x7f13bc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.266+0000 7f13e1c28700 1 -- 172.21.15.189:0/1025026881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13dc0d11b0 msgr2=0x7f13dc06c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:25.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.266+0000 7f13e1c28700 1 --2- 172.21.15.189:0/1025026881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13dc0d11b0 0x7f13dc06c0d0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f13d400cb40 tx=0x7f13d400ce50 comp rx=0 tx=0).stop 2024-04-06T13:29:25.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.266+0000 7f13e1c28700 1 -- 172.21.15.189:0/1025026881 shutdown_connections 2024-04-06T13:29:25.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.266+0000 7f13e1c28700 1 --2- 172.21.15.189:0/1025026881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13dc0d11b0 0x7f13dc06c0d0 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.266+0000 7f13e1c28700 1 --2- 172.21.15.189:0/1025026881 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f13bc06ae00 0x7f13bc06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.266+0000 7f13e1c28700 1 --2- 172.21.15.189:0/1025026881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13dc0cb870 0x7f13dc0affd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.266+0000 7f13e1c28700 1 --2- 172.21.15.189:0/1025026881 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13dc0cfa00 0x7f13dc1515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.266+0000 7f13e1c28700 1 -- 172.21.15.189:0/1025026881 >> 172.21.15.189:0/1025026881 conn(0x7f13dc0b2db0 msgr2=0x7f13dc0b2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:25.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.267+0000 7f13e1c28700 1 -- 172.21.15.189:0/1025026881 shutdown_connections 2024-04-06T13:29:25.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.267+0000 7f13e1c28700 1 -- 172.21.15.189:0/1025026881 wait complete. 2024-04-06T13:29:25.278 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:25.277+0000 7fa6c21fd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:29:25.279 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-06T13:29:25.296 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:25.297+0000 7f20c3c01200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:29:25.356 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:25.356+0000 7fa6c21fd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:29:25.379 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:25.380+0000 7f20c3c01200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:29:25.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.407+0000 7f9939870700 1 Processor -- start 2024-04-06T13:29:25.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.407+0000 7f9939870700 1 -- start start 2024-04-06T13:29:25.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.408+0000 7f9939870700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f99340d11b0 0x7f99340d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:25.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.408+0000 7f9939870700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f99340cfa00 0x7f99340cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:25.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.408+0000 7f9939870700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99340d1ad0 con 0x7f99340cb870 2024-04-06T13:29:25.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.408+0000 7f9939870700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99340ccdc0 con 0x7f99340cfa00 2024-04-06T13:29:25.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.408+0000 7f9939870700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99340d0b50 con 0x7f99340d11b0 2024-04-06T13:29:25.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.408+0000 7f99337fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f99340cfa00 0x7f99340cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:25.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.408+0000 7f99337fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f99340cfa00 0x7f99340cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54420/0 (socket says 172.21.15.189:54420) 2024-04-06T13:29:25.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.408+0000 7f99337fe700 1 -- 172.21.15.189:0/236264938 learned_addr learned my addr 172.21.15.189:0/236264938 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:25.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.408+0000 7f9932ffd700 1 --2- 172.21.15.189:0/236264938 >> v2:172.21.15.192:3300/0 conn(0x7f99340d11b0 0x7f99340d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:25.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.408+0000 7f99337fe700 1 -- 172.21.15.189:0/236264938 >> v2:172.21.15.192:3300/0 conn(0x7f99340d11b0 msgr2=0x7f99340d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:25.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.408+0000 7f99337fe700 1 --2- 172.21.15.189:0/236264938 >> v2:172.21.15.192:3300/0 conn(0x7f99340d11b0 0x7f99340d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.408+0000 7f99337fe700 1 -- 172.21.15.189:0/236264938 >> v1:172.21.15.192:6789/0 conn(0x7f99340cb870 legacy=0x7f993415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:25.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.408+0000 7f99337fe700 1 -- 172.21.15.189:0/236264938 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99340cbd80 con 0x7f99340cfa00 2024-04-06T13:29:25.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.408+0000 7f99337fe700 1 --2- 172.21.15.189:0/236264938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99340cfa00 0x7f99340cd560 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f99340ced00 tx=0x7f992c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa5fadb7358f97d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:25.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.408+0000 7f9931ffb700 1 -- 172.21.15.189:0/236264938 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992c00e070 con 0x7f99340cfa00 2024-04-06T13:29:25.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.409+0000 7f9931ffb700 1 -- 172.21.15.189:0/236264938 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f992c00b910 con 0x7f99340cfa00 2024-04-06T13:29:25.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.409+0000 7f9931ffb700 1 -- 172.21.15.189:0/236264938 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992c00bcc0 con 0x7f99340cfa00 2024-04-06T13:29:25.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.409+0000 7f9939870700 1 -- 172.21.15.189:0/236264938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99340cfa00 msgr2=0x7f99340cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:25.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.409+0000 7f9939870700 1 --2- 172.21.15.189:0/236264938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99340cfa00 0x7f99340cd560 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f99340ced00 tx=0x7f992c00b330 comp rx=0 tx=0).stop 2024-04-06T13:29:25.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.409+0000 7f9939870700 1 -- 172.21.15.189:0/236264938 shutdown_connections 2024-04-06T13:29:25.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.409+0000 7f9939870700 1 --2- 172.21.15.189:0/236264938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99340cfa00 0x7f99340cd560 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.409+0000 7f9939870700 1 --2- 172.21.15.189:0/236264938 >> v2:172.21.15.192:3300/0 conn(0x7f99340d11b0 0x7f99340d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.409+0000 7f9939870700 1 -- 172.21.15.189:0/236264938 >> 172.21.15.189:0/236264938 conn(0x7f99340b2db0 msgr2=0x7f99340b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:25.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.409+0000 7f9939870700 1 -- 172.21.15.189:0/236264938 shutdown_connections 2024-04-06T13:29:25.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.409+0000 7f9939870700 1 -- 172.21.15.189:0/236264938 wait complete. 2024-04-06T13:29:25.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.410+0000 7f9939870700 1 Processor -- start 2024-04-06T13:29:25.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.411+0000 7f9939870700 1 -- start start 2024-04-06T13:29:25.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.411+0000 7f9939870700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99340cb870 0x7f99340c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:25.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.411+0000 7f9939870700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99340cfa00 0x7f99340c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:25.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.411+0000 7f9939870700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99340d11b0 0x7f993414ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:25.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.411+0000 7f9939870700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993414f950 con 0x7f99340cfa00 2024-04-06T13:29:25.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.411+0000 7f9939870700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99340c9370 con 0x7f99340d11b0 2024-04-06T13:29:25.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.411+0000 7f9939870700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99341e4db0 con 0x7f99340cb870 2024-04-06T13:29:25.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.411+0000 7f99327fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99340cfa00 0x7f99340c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:25.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.411+0000 7f99327fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99340cfa00 0x7f99340c46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54436/0 (socket says 172.21.15.189:54436) 2024-04-06T13:29:25.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.411+0000 7f99327fc700 1 -- 172.21.15.189:0/48114464 learned_addr learned my addr 172.21.15.189:0/48114464 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:25.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.411+0000 7f99337fe700 1 --2- 172.21.15.189:0/48114464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99340d11b0 0x7f993414ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:25.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.411+0000 7f9932ffd700 1 --2- 172.21.15.189:0/48114464 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99340cb870 0x7f99340c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:25.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.412+0000 7f99327fc700 1 -- 172.21.15.189:0/48114464 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99340cb870 msgr2=0x7f99340c8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:25.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.412+0000 7f99327fc700 1 --2- 172.21.15.189:0/48114464 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99340cb870 0x7f99340c8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.412+0000 7f99327fc700 1 -- 172.21.15.189:0/48114464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99340d11b0 msgr2=0x7f993414ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:25.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.412+0000 7f99327fc700 1 --2- 172.21.15.189:0/48114464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99340d11b0 0x7f993414ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.412+0000 7f99327fc700 1 -- 172.21.15.189:0/48114464 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f992c009b30 con 0x7f99340cfa00 2024-04-06T13:29:25.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.412+0000 7f99327fc700 1 --2- 172.21.15.189:0/48114464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99340cfa00 0x7f99340c46f0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f992800c3c0 tx=0x7f992800c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:25.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.412+0000 7f991bfff700 1 -- 172.21.15.189:0/48114464 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99280052d0 con 0x7f99340cfa00 2024-04-06T13:29:25.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.412+0000 7f991bfff700 1 -- 172.21.15.189:0/48114464 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9928005430 con 0x7f99340cfa00 2024-04-06T13:29:25.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.412+0000 7f9939870700 1 -- 172.21.15.189:0/48114464 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99341e5090 con 0x7f99340cfa00 2024-04-06T13:29:25.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.412+0000 7f991bfff700 1 -- 172.21.15.189:0/48114464 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9928006a80 con 0x7f99340cfa00 2024-04-06T13:29:25.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.412+0000 7f9939870700 1 -- 172.21.15.189:0/48114464 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99341e5650 con 0x7f99340cfa00 2024-04-06T13:29:25.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.413+0000 7f9939870700 1 -- 172.21.15.189:0/48114464 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9934074990 con 0x7f99340cfa00 2024-04-06T13:29:25.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.414+0000 7f991bfff700 1 -- 172.21.15.189:0/48114464 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 109) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f9928003680 con 0x7f99340cfa00 2024-04-06T13:29:25.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.414+0000 7f991bfff700 1 --2- 172.21.15.189:0/48114464 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f992006ae00 0x7f992006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:25.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.414+0000 7f9932ffd700 1 -- 172.21.15.189:0/48114464 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f992006ae00 msgr2=0x7f992006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:25.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.414+0000 7f991bfff700 1 -- 172.21.15.189:0/48114464 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9928088730 con 0x7f99340cfa00 2024-04-06T13:29:25.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.414+0000 7f9932ffd700 1 --2- 172.21.15.189:0/48114464 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f992006ae00 0x7f992006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:29:25.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.418+0000 7f991bfff700 1 -- 172.21.15.189:0/48114464 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9928053690 con 0x7f99340cfa00 2024-04-06T13:29:25.429 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:25.429+0000 7fa6c21fd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:29:25.458 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:25.458+0000 7f20c3c01200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:29:25.497 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:25.497+0000 7fa6c21fd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:29:25.529 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:25.529+0000 7f20c3c01200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:29:25.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.615+0000 7f9932ffd700 1 -- 172.21.15.189:0/48114464 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f992006ae00 msgr2=0x7f992006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:25.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.615+0000 7f9932ffd700 1 --2- 172.21.15.189:0/48114464 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f992006ae00 0x7f992006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:29:25.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.623+0000 7f9939870700 1 -- 172.21.15.189:0/48114464 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f99340d1e20 con 0x7f99340cfa00 2024-04-06T13:29:25.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.628+0000 7f991bfff700 1 -- 172.21.15.189:0/48114464 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v66)=0 v66) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9928056cb0 con 0x7f99340cfa00 2024-04-06T13:29:25.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.631+0000 7f9939870700 1 -- 172.21.15.189:0/48114464 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f992006ae00 msgr2=0x7f992006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:29:25.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.631+0000 7f9939870700 1 --2- 172.21.15.189:0/48114464 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f992006ae00 0x7f992006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.631+0000 7f9939870700 1 -- 172.21.15.189:0/48114464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99340cfa00 msgr2=0x7f99340c46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:25.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.631+0000 7f9939870700 1 --2- 172.21.15.189:0/48114464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99340cfa00 0x7f99340c46f0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f992800c3c0 tx=0x7f992800c6d0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.631+0000 7f9939870700 1 -- 172.21.15.189:0/48114464 shutdown_connections 2024-04-06T13:29:25.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.631+0000 7f9939870700 1 --2- 172.21.15.189:0/48114464 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f992006ae00 0x7f992006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.631+0000 7f9939870700 1 --2- 172.21.15.189:0/48114464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99340d11b0 0x7f993414ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.631+0000 7f9939870700 1 --2- 172.21.15.189:0/48114464 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99340cb870 0x7f99340c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.631+0000 7f9939870700 1 --2- 172.21.15.189:0/48114464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99340cfa00 0x7f99340c46f0 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.631+0000 7f9939870700 1 -- 172.21.15.189:0/48114464 >> 172.21.15.189:0/48114464 conn(0x7f99340b2db0 msgr2=0x7f99340b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:25.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.631+0000 7f9939870700 1 -- 172.21.15.189:0/48114464 shutdown_connections 2024-04-06T13:29:25.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.631+0000 7f9939870700 1 -- 172.21.15.189:0/48114464 wait complete. 2024-04-06T13:29:25.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:29:25.644 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-06T13:29:25.675 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:25.675+0000 7fa6c21fd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:29:25.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:29:25.705 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2024-04-06T13:29:25.712 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:25.712+0000 7f20c3c01200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:29:25.739 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/test_restart 1m 2024-04-06T13:29:25.831 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:25.831+0000 7fa6c21fd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:29:25.877 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:25.877+0000 7f20c3c01200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:29:25.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.909+0000 7efeee16a700 1 Processor -- start 2024-04-06T13:29:25.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.909+0000 7efeee16a700 1 -- start start 2024-04-06T13:29:25.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.910+0000 7efeee16a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efee80d11b0 0x7efee80d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:25.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.910+0000 7efeee16a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee80d1ad0 con 0x7efee80cb870 2024-04-06T13:29:25.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.910+0000 7efeee16a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee80cbd80 con 0x7efee80cfa00 2024-04-06T13:29:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.910+0000 7efeee16a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efee80ccdc0 con 0x7efee80d11b0 2024-04-06T13:29:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.910+0000 7efee6ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efee80cb870 0x7efee815c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52026/0 (socket says 172.21.15.189:52026) 2024-04-06T13:29:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.910+0000 7efee6ffd700 1 -- 172.21.15.189:0/3622157329 learned_addr learned my addr 172.21.15.189:0/3622157329 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:29:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.910+0000 7efee77fe700 1 --2- 172.21.15.189:0/3622157329 >> v2:172.21.15.192:3300/0 conn(0x7efee80d11b0 0x7efee80d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.910+0000 7efee67fc700 1 -- 172.21.15.189:0/3622157329 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877524830 0 0) 0x7efee80d1ad0 con 0x7efee80cb870 2024-04-06T13:29:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.910+0000 7efee67fc700 1 -- 172.21.15.189:0/3622157329 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed00035b0 con 0x7efee80cb870 2024-04-06T13:29:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.910+0000 7efee67fc700 1 -- 172.21.15.189:0/3622157329 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220589269 0 0) 0x7efee80cbd80 con 0x7efee80cfa00 2024-04-06T13:29:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.910+0000 7efee67fc700 1 -- 172.21.15.189:0/3622157329 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efee80d1ad0 con 0x7efee80cfa00 2024-04-06T13:29:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.910+0000 7efee77fe700 1 -- 172.21.15.189:0/3622157329 >> v1:172.21.15.192:6789/0 conn(0x7efee80cfa00 legacy=0x7efee80cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.910+0000 7efee77fe700 1 -- 172.21.15.189:0/3622157329 >> v1:172.21.15.189:6789/0 conn(0x7efee80cb870 legacy=0x7efee815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.910+0000 7efee67fc700 1 -- 172.21.15.189:0/3622157329 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 818140374 0 0) 0x7efed00035b0 con 0x7efee80cb870 2024-04-06T13:29:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.910+0000 7efee77fe700 1 -- 172.21.15.189:0/3622157329 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee80cef20 con 0x7efee80d11b0 2024-04-06T13:29:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.911+0000 7efee77fe700 1 --2- 172.21.15.189:0/3622157329 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efee80d11b0 0x7efee80d1590 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7efed8009840 tx=0x7efed8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da1e73e922216e39 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.911+0000 7efee67fc700 1 -- 172.21.15.189:0/3622157329 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc001670 con 0x7efee80d11b0 2024-04-06T13:29:25.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.911+0000 7efee67fc700 1 -- 172.21.15.189:0/3622157329 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed8004030 con 0x7efee80d11b0 2024-04-06T13:29:25.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.911+0000 7efee67fc700 1 -- 172.21.15.189:0/3622157329 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed80043c0 con 0x7efee80d11b0 2024-04-06T13:29:25.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.911+0000 7efeee16a700 1 -- 172.21.15.189:0/3622157329 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efee80d11b0 msgr2=0x7efee80d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:25.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.911+0000 7efeee16a700 1 --2- 172.21.15.189:0/3622157329 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efee80d11b0 0x7efee80d1590 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7efed8009840 tx=0x7efed8009b50 comp rx=0 tx=0).stop 2024-04-06T13:29:25.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.911+0000 7efeee16a700 1 -- 172.21.15.189:0/3622157329 shutdown_connections 2024-04-06T13:29:25.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.911+0000 7efeee16a700 1 --2- 172.21.15.189:0/3622157329 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efee80d11b0 0x7efee80d1590 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.911+0000 7efeee16a700 1 -- 172.21.15.189:0/3622157329 >> 172.21.15.189:0/3622157329 conn(0x7efee80b2db0 msgr2=0x7efee80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:25.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.911+0000 7efeee16a700 1 -- 172.21.15.189:0/3622157329 shutdown_connections 2024-04-06T13:29:25.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.911+0000 7efeee16a700 1 -- 172.21.15.189:0/3622157329 wait complete. 2024-04-06T13:29:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.913+0000 7efeee16a700 1 Processor -- start 2024-04-06T13:29:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.913+0000 7efeee16a700 1 -- start start 2024-04-06T13:29:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efeee16a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efee80cb870 0x7efee8154ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efeee16a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efee80cfa00 0x7efee8158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efeee16a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efee80d11b0 0x7efee80c6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efeee16a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee8158470 con 0x7efee80cfa00 2024-04-06T13:29:25.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efeee16a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee8158e40 con 0x7efee80cb870 2024-04-06T13:29:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efeee16a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee8157860 con 0x7efee80d11b0 2024-04-06T13:29:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efee6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efee80cfa00 0x7efee8158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efee6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efee80cfa00 0x7efee8158900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54448/0 (socket says 172.21.15.189:54448) 2024-04-06T13:29:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efee6ffd700 1 -- 172.21.15.189:0/662471715 learned_addr learned my addr 172.21.15.189:0/662471715 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efee77fe700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efee80cb870 0x7efee8154ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efee7fff700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efee80d11b0 0x7efee80c6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efee6ffd700 1 -- 172.21.15.189:0/662471715 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efee80d11b0 msgr2=0x7efee80c6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efee6ffd700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efee80d11b0 0x7efee80c6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efee6ffd700 1 -- 172.21.15.189:0/662471715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efee80cb870 msgr2=0x7efee8154ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efee6ffd700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efee80cb870 0x7efee8154ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efee6ffd700 1 -- 172.21.15.189:0/662471715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed80094a0 con 0x7efee80cfa00 2024-04-06T13:29:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efee6ffd700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efee80cfa00 0x7efee8158900 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7efedc00c750 tx=0x7efedc00ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efecffff700 1 -- 172.21.15.189:0/662471715 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc004c80 con 0x7efee80cfa00 2024-04-06T13:29:25.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efecffff700 1 -- 172.21.15.189:0/662471715 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efedc004de0 con 0x7efee80cfa00 2024-04-06T13:29:25.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efeee16a700 1 -- 172.21.15.189:0/662471715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee8157b40 con 0x7efee80cfa00 2024-04-06T13:29:25.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.914+0000 7efecffff700 1 -- 172.21.15.189:0/662471715 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc0072d0 con 0x7efee80cfa00 2024-04-06T13:29:25.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.915+0000 7efeee16a700 1 -- 172.21.15.189:0/662471715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee8158100 con 0x7efee80cfa00 2024-04-06T13:29:25.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.916+0000 7efeee16a700 1 -- 172.21.15.189:0/662471715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee8074990 con 0x7efee80cfa00 2024-04-06T13:29:25.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.916+0000 7efecffff700 1 -- 172.21.15.189:0/662471715 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 109) v1 ==== 89264+0+0 (secure 0 0 0) 0x7efedc007430 con 0x7efee80cfa00 2024-04-06T13:29:25.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.916+0000 7efecffff700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed406ae00 0x7efed406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:25.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.917+0000 7efee77fe700 1 -- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed406ae00 msgr2=0x7efed406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:25.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.917+0000 7efecffff700 1 -- 172.21.15.189:0/662471715 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efedc089450 con 0x7efee80cfa00 2024-04-06T13:29:25.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.917+0000 7efee77fe700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed406ae00 0x7efed406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:29:25.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:25.921+0000 7efecffff700 1 -- 172.21.15.189:0/662471715 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efedc0543b0 con 0x7efee80cfa00 2024-04-06T13:29:25.943 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:25.943+0000 7fa6c21fd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:29:25.994 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:25.994+0000 7f20c3c01200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:29:26.024 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:26.025+0000 7fa6c21fd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:29:26.078 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:26.078+0000 7f20c3c01200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:29:26.099 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:26.099+0000 7fa6c21fd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:29:26.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:26.118+0000 7efee77fe700 1 -- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed406ae00 msgr2=0x7efed406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:26.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:26.118+0000 7efee77fe700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed406ae00 0x7efed406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:29:26.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:26.151+0000 7efeee16a700 1 -- 172.21.15.189:0/662471715 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7efee80cc060 con 0x7efed406ae00 2024-04-06T13:29:26.155 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:26.156+0000 7f20c3c01200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:29:26.300 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:26.301+0000 7fa6c21fd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:29:26.359 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:26.360+0000 7f20c3c01200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:29:26.377 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:26.377+0000 7fa6c21fd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:29:26.435 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:26.435+0000 7f20c3c01200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:29:26.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:26.519+0000 7efee77fe700 1 -- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed406ae00 msgr2=0x7efed406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:26.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:26.519+0000 7efee77fe700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed406ae00 0x7efed406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-06T13:29:26.694 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:26.694+0000 7fa6c21fd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:29:26.757 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:26.757+0000 7f20c3c01200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:29:27.015 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:27.015+0000 7fa6c21fd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:29:27.079 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:27.079+0000 7f20c3c01200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:29:27.173 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:27.173+0000 7fa6c21fd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:29:27.238 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:27.238+0000 7f20c3c01200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:29:27.246 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:27.246+0000 7fa6c21fd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:29:27.310 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:27.311+0000 7f20c3c01200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:29:27.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:27.320+0000 7efee77fe700 1 -- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed406ae00 msgr2=0x7efed406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:27.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:27.320+0000 7efee77fe700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed406ae00 0x7efed406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-06T13:29:27.863 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:27.864+0000 7fa6c21fd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:29:27.929 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:27.930+0000 7f20c3c01200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:29:27.941 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:27.941+0000 7fa6c21fd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:29:28.009 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:28.009+0000 7f20c3c01200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:29:28.067 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:28.065+0000 7fa6c21fd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:29:28.122 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:29:28.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:29:28.136 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:28.136+0000 7f20c3c01200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:29:28.576 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:28.575+0000 7fa6c21fd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:29:28.652 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:28.653+0000 7f20c3c01200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:29:28.729 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:28.729+0000 7fa6c21fd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:29:28.807 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:28.807+0000 7f20c3c01200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:29:28.884 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:28.884+0000 7fa6c21fd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:29:28.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:28.921+0000 7efee77fe700 1 -- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed406ae00 msgr2=0x7efed406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/47128 2024-04-06T13:29:28.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:28.921+0000 7efee77fe700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed406ae00 0x7efed406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-06T13:29:28.964 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:28.964+0000 7f20c3c01200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:29:29.621 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:29.621+0000 7fa6c21fd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:29:29.690 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:29.691+0000 7f20c3c01200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:29:29.861 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:29.861+0000 7fa6c21fd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:29:29.935 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:29.934+0000 7f20c3c01200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:29:29.936 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:29.937+0000 7fa6c21fd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:29:30.013 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:30.012+0000 7f20c3c01200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:29:30.309 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:29:30.310+0000 7fa6c21fd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:29:30.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:30.379+0000 7efecffff700 1 -- 172.21.15.189:0/662471715 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 110) v1 ==== 89264+0+0 (secure 0 0 0) 0x7efedc087800 con 0x7efee80cfa00 2024-04-06T13:29:30.393 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:30.392+0000 7f20c3c01200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:29:30.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:30.412+0000 7efecffff700 1 -- 172.21.15.189:0/662471715 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mgrmap(e 111) v1 ==== 88873+0+0 (secure 0 0 0) 0x7efedc054780 con 0x7efee80cfa00 2024-04-06T13:29:30.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:30.412+0000 7efecffff700 1 -- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed406ae00 msgr2=0x7efed406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:29:30.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:30.412+0000 7efecffff700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed406ae00 0x7efed406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:30.564 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:30.563+0000 7f203630e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:30.564 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:30.563+0000 7f203630e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:30.564 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:30.563+0000 7f203630e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:30.564 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:30.563+0000 7f203630e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:30.564 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:30.563+0000 7f203630e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:30.564 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:30.563+0000 7f2039b15700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:30.564 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:30.563+0000 7f2039b15700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:30.564 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:30.563+0000 7f2039b15700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:30.564 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:30.563+0000 7f2039b15700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:30.565 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:29:30.563+0000 7f2039b15700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:29:31.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.418+0000 7efecffff700 1 -- 172.21.15.189:0/662471715 <== mon.0 v2:172.21.15.189:3300/0 9 ==== mgrmap(e 112) v1 ==== 89000+0+0 (secure 0 0 0) 0x7efedc055050 con 0x7efee80cfa00 2024-04-06T13:29:31.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.418+0000 7efecffff700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed406ae00 0x7efed406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:31.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.418+0000 7efecffff700 1 -- 172.21.15.189:0/662471715 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7efed401cd40 con 0x7efed406ae00 2024-04-06T13:29:31.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.420+0000 7efee77fe700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed406ae00 0x7efed406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:31.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.422+0000 7efee77fe700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed406ae00 0x7efed406d2c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efed800a010 tx=0x7efed8008030 comp rx=0 tx=0).ready entity=mgr.10308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:31.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.473+0000 7efecffff700 1 -- 172.21.15.189:0/662471715 <== mgr.10308 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7efed401cd40 con 0x7efed406ae00 2024-04-06T13:29:31.475 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path /snap_test_dir1/test_restart 2024-04-06T13:29:31.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.476+0000 7efeee16a700 1 -- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed406ae00 msgr2=0x7efed406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:31.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.476+0000 7efeee16a700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed406ae00 0x7efed406d2c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efed800a010 tx=0x7efed8008030 comp rx=0 tx=0).stop 2024-04-06T13:29:31.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.476+0000 7efeee16a700 1 -- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efee80cfa00 msgr2=0x7efee8158900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:31.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.476+0000 7efeee16a700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efee80cfa00 0x7efee8158900 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7efedc00c750 tx=0x7efedc00ca60 comp rx=0 tx=0).stop 2024-04-06T13:29:31.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.476+0000 7efeee16a700 1 -- 172.21.15.189:0/662471715 shutdown_connections 2024-04-06T13:29:31.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.476+0000 7efeee16a700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efee80d11b0 0x7efee80c6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:31.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.476+0000 7efeee16a700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7efed406ae00 0x7efed406d2c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:31.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.476+0000 7efeee16a700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efee80cb870 0x7efee8154ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:31.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.476+0000 7efeee16a700 1 --2- 172.21.15.189:0/662471715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efee80cfa00 0x7efee8158900 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:31.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.476+0000 7efeee16a700 1 -- 172.21.15.189:0/662471715 >> 172.21.15.189:0/662471715 conn(0x7efee80b2db0 msgr2=0x7efee80b5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:31.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.476+0000 7efeee16a700 1 -- 172.21.15.189:0/662471715 shutdown_connections 2024-04-06T13:29:31.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.476+0000 7efeee16a700 1 -- 172.21.15.189:0/662471715 wait complete. 2024-04-06T13:29:31.490 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2024-04-06T13:29:31.490 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/test_restart 1m 2024-04-06T13:29:31.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.617+0000 7f019962c700 1 Processor -- start 2024-04-06T13:29:31.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.617+0000 7f019962c700 1 -- start start 2024-04-06T13:29:31.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.618+0000 7f019962c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f019415d590 0x7f01940b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.618+0000 7f019962c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f01940b4ed0 0x7f01940b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.618+0000 7f019962c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01940b6580 0x7f01940b69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.618+0000 7f019962c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01940b58c0 con 0x7f019415d590 2024-04-06T13:29:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.618+0000 7f019962c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01940b5cc0 con 0x7f01940b4ed0 2024-04-06T13:29:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.618+0000 7f019962c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01940d1500 con 0x7f01940b6580 2024-04-06T13:29:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.618+0000 7f0192ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f019415d590 0x7f01940b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.618+0000 7f0192ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f019415d590 0x7f01940b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48112/0 (socket says 172.21.15.189:48112) 2024-04-06T13:29:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.618+0000 7f0192ffd700 1 -- 172.21.15.189:0/3256086849 learned_addr learned my addr 172.21.15.189:0/3256086849 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.618+0000 7f01937fe700 1 --2- 172.21.15.189:0/3256086849 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01940b6580 0x7f01940b69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.618+0000 7f0192ffd700 1 -- 172.21.15.189:0/3256086849 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01940b6580 msgr2=0x7f01940b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.619+0000 7f0192ffd700 1 --2- 172.21.15.189:0/3256086849 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01940b6580 0x7f01940b69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.619+0000 7f0192ffd700 1 -- 172.21.15.189:0/3256086849 >> v2:172.21.15.192:3300/0 conn(0x7f01940b4ed0 msgr2=0x7f01940b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:29:31.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.619+0000 7f01927fc700 1 --2- 172.21.15.189:0/3256086849 >> v2:172.21.15.192:3300/0 conn(0x7f01940b4ed0 0x7f01940b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:31.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.619+0000 7f0192ffd700 1 --2- 172.21.15.189:0/3256086849 >> v2:172.21.15.192:3300/0 conn(0x7f01940b4ed0 0x7f01940b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:31.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.619+0000 7f0192ffd700 1 -- 172.21.15.189:0/3256086849 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01940d17e0 con 0x7f019415d590 2024-04-06T13:29:31.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.619+0000 7f01927fc700 1 --2- 172.21.15.189:0/3256086849 >> v2:172.21.15.192:3300/0 conn(0x7f01940b4ed0 0x7f01940b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:29:31.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.619+0000 7f0192ffd700 1 --2- 172.21.15.189:0/3256086849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f019415d590 0x7f01940b8d70 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f01940b6330 tx=0x7f0188009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8477f091aafc1f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:31.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.619+0000 7f0191ffb700 1 -- 172.21.15.189:0/3256086849 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018800b870 con 0x7f019415d590 2024-04-06T13:29:31.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.619+0000 7f0191ffb700 1 -- 172.21.15.189:0/3256086849 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f018800b9d0 con 0x7f019415d590 2024-04-06T13:29:31.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.619+0000 7f0191ffb700 1 -- 172.21.15.189:0/3256086849 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018800bd60 con 0x7f019415d590 2024-04-06T13:29:31.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.619+0000 7f019962c700 1 -- 172.21.15.189:0/3256086849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f019415d590 msgr2=0x7f01940b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:31.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.619+0000 7f019962c700 1 --2- 172.21.15.189:0/3256086849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f019415d590 0x7f01940b8d70 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f01940b6330 tx=0x7f0188009bd0 comp rx=0 tx=0).stop 2024-04-06T13:29:31.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.619+0000 7f019962c700 1 -- 172.21.15.189:0/3256086849 shutdown_connections 2024-04-06T13:29:31.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.619+0000 7f019962c700 1 --2- 172.21.15.189:0/3256086849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f019415d590 0x7f01940b8d70 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:31.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.619+0000 7f019962c700 1 --2- 172.21.15.189:0/3256086849 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01940b6580 0x7f01940b69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:31.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.619+0000 7f019962c700 1 --2- 172.21.15.189:0/3256086849 >> v2:172.21.15.192:3300/0 conn(0x7f01940b4ed0 0x7f01940b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:31.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.619+0000 7f019962c700 1 -- 172.21.15.189:0/3256086849 >> 172.21.15.189:0/3256086849 conn(0x7f019406bbc0 msgr2=0x7f019406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:31.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.620+0000 7f019962c700 1 -- 172.21.15.189:0/3256086849 shutdown_connections 2024-04-06T13:29:31.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.620+0000 7f019962c700 1 -- 172.21.15.189:0/3256086849 wait complete. 2024-04-06T13:29:31.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.621+0000 7f019962c700 1 Processor -- start 2024-04-06T13:29:31.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.621+0000 7f019962c700 1 -- start start 2024-04-06T13:29:31.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f019962c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01940b4ed0 0x7f01940cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:31.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f019962c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01940b6580 0x7f019415d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:31.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f019962c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f019415d590 0x7f019414cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:31.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f019962c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019415e380 con 0x7f01940b4ed0 2024-04-06T13:29:31.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f019962c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019414d4e0 con 0x7f01940b6580 2024-04-06T13:29:31.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f019962c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01941f7070 con 0x7f019415d590 2024-04-06T13:29:31.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f0192ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01940b4ed0 0x7f01940cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:31.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f0192ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01940b4ed0 0x7f01940cb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48126/0 (socket says 172.21.15.189:48126) 2024-04-06T13:29:31.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f0192ffd700 1 -- 172.21.15.189:0/787781921 learned_addr learned my addr 172.21.15.189:0/787781921 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:29:31.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f01937fe700 1 --2- 172.21.15.189:0/787781921 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f019415d590 0x7f019414cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:31.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f01927fc700 1 --2- 172.21.15.189:0/787781921 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01940b6580 0x7f019415d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:31.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f0192ffd700 1 -- 172.21.15.189:0/787781921 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f019415d590 msgr2=0x7f019414cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:31.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f0192ffd700 1 --2- 172.21.15.189:0/787781921 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f019415d590 0x7f019414cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:31.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f0192ffd700 1 -- 172.21.15.189:0/787781921 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01940b6580 msgr2=0x7f019415d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:31.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f0192ffd700 1 --2- 172.21.15.189:0/787781921 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01940b6580 0x7f019415d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:31.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f0192ffd700 1 -- 172.21.15.189:0/787781921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0188009550 con 0x7f01940b4ed0 2024-04-06T13:29:31.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f0192ffd700 1 --2- 172.21.15.189:0/787781921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01940b4ed0 0x7f01940cb2f0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f0188014040 tx=0x7f0188004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:31.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f017b7fe700 1 -- 172.21.15.189:0/787781921 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0188004520 con 0x7f01940b4ed0 2024-04-06T13:29:31.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f017b7fe700 1 -- 172.21.15.189:0/787781921 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f018801e070 con 0x7f01940b4ed0 2024-04-06T13:29:31.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f019962c700 1 -- 172.21.15.189:0/787781921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f019415e590 con 0x7f01940b4ed0 2024-04-06T13:29:31.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f017b7fe700 1 -- 172.21.15.189:0/787781921 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01880038e0 con 0x7f01940b4ed0 2024-04-06T13:29:31.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.622+0000 7f019962c700 1 -- 172.21.15.189:0/787781921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f019415fea0 con 0x7f01940b4ed0 2024-04-06T13:29:31.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.623+0000 7f019962c700 1 -- 172.21.15.189:0/787781921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0194074990 con 0x7f01940b4ed0 2024-04-06T13:29:31.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.624+0000 7f017b7fe700 1 -- 172.21.15.189:0/787781921 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 112) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f0188003a40 con 0x7f01940b4ed0 2024-04-06T13:29:31.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.624+0000 7f017b7fe700 1 --2- 172.21.15.189:0/787781921 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f018006ab60 0x7f018006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:29:31.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.624+0000 7f017b7fe700 1 -- 172.21.15.189:0/787781921 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f0188020070 con 0x7f01940b4ed0 2024-04-06T13:29:31.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.624+0000 7f01927fc700 1 --2- 172.21.15.189:0/787781921 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f018006ab60 0x7f018006d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:29:31.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.625+0000 7f01927fc700 1 --2- 172.21.15.189:0/787781921 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f018006ab60 0x7f018006d020 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f01940cd520 tx=0x7f0184008040 comp rx=0 tx=0).ready entity=mgr.10308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:29:31.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.629+0000 7f017b7fe700 1 -- 172.21.15.189:0/787781921 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0188055b40 con 0x7f01940b4ed0 2024-04-06T13:29:31.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.847+0000 7f019962c700 1 -- 172.21.15.189:0/787781921 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f01940d2080 con 0x7f018006ab60 2024-04-06T13:29:31.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.888+0000 7f017b7fe700 1 -- 172.21.15.189:0/787781921 <== mgr.10308 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7f018400a6a0 con 0x7f018006ab60 2024-04-06T13:29:31.889 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule activated for path /snap_test_dir1/test_restart 2024-04-06T13:29:31.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.890+0000 7f019962c700 1 -- 172.21.15.189:0/787781921 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f018006ab60 msgr2=0x7f018006d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:31.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.890+0000 7f019962c700 1 --2- 172.21.15.189:0/787781921 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f018006ab60 0x7f018006d020 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f01940cd520 tx=0x7f0184008040 comp rx=0 tx=0).stop 2024-04-06T13:29:31.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.890+0000 7f019962c700 1 -- 172.21.15.189:0/787781921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01940b4ed0 msgr2=0x7f01940cb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:29:31.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.890+0000 7f019962c700 1 --2- 172.21.15.189:0/787781921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01940b4ed0 0x7f01940cb2f0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f0188014040 tx=0x7f0188004750 comp rx=0 tx=0).stop 2024-04-06T13:29:31.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.890+0000 7f019962c700 1 -- 172.21.15.189:0/787781921 shutdown_connections 2024-04-06T13:29:31.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.890+0000 7f019962c700 1 --2- 172.21.15.189:0/787781921 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f019415d590 0x7f019414cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:31.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.890+0000 7f019962c700 1 --2- 172.21.15.189:0/787781921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01940b4ed0 0x7f01940cb2f0 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:31.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.890+0000 7f019962c700 1 --2- 172.21.15.189:0/787781921 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f018006ab60 0x7f018006d020 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:31.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.890+0000 7f019962c700 1 --2- 172.21.15.189:0/787781921 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01940b6580 0x7f019415d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:29:31.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.890+0000 7f019962c700 1 -- 172.21.15.189:0/787781921 >> 172.21.15.189:0/787781921 conn(0x7f019406bbc0 msgr2=0x7f01940b2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:29:31.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.890+0000 7f019962c700 1 -- 172.21.15.189:0/787781921 shutdown_connections 2024-04-06T13:29:31.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:29:31.891+0000 7f019962c700 1 -- 172.21.15.189:0/787781921 wait complete. 2024-04-06T13:29:31.904 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2024-04-06T13:29:58.291 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:29:58.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:30:00.898 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:30:00.898+0000 7f2028ab3700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:30:00.898 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:30:00.898+0000 7f2028ab3700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:30:00.899 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:30:00.898+0000 7f2028ab3700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:30:00.899 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:30:00.898+0000 7f2028ab3700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:30:00.899 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:30:00.898+0000 7f2028ab3700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:30:28.654 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:30:28.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:30:58.722 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:30:58.726 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:31:28.881 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:31:28.885 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:31:59.008 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:31:59.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:32:29.155 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:32:29.158 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:32:59.280 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:32:59.284 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:33:29.394 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:33:29.397 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:33:59.532 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:33:59.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:34:29.625 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:34:29.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:34:59.742 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:34:59.746 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:35:29.889 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:35:29.892 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:36:00.019 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:36:00.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:36:30.164 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:36:30.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:37:00.320 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:37:00.325 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:37:30.433 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:37:30.437 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:38:00.564 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:38:00.568 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:38:30.693 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:38:30.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:39:00.816 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:39:00.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:39:30.972 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:39:30.976 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:40:01.117 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:40:01.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:40:15.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:40:15.905 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-04-06T13:40:15.929 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_30_00_UTC 2024-04-06T13:40:15.929 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_31_00_UTC 2024-04-06T13:40:15.929 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_32_00_UTC 2024-04-06T13:40:15.929 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_33_00_UTC 2024-04-06T13:40:15.929 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_34_00_UTC 2024-04-06T13:40:15.929 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_35_00_UTC 2024-04-06T13:40:15.929 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_36_00_UTC 2024-04-06T13:40:15.929 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_37_00_UTC 2024-04-06T13:40:15.929 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_38_00_UTC 2024-04-06T13:40:15.929 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_39_00_UTC 2024-04-06T13:40:15.929 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_40_00_UTC 2024-04-06T13:40:15.930 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-06-13_30_00_UTC', 'scheduled-2024-04-06-13_31_00_UTC', 'scheduled-2024-04-06-13_32_00_UTC', 'scheduled-2024-04-06-13_33_00_UTC', 'scheduled-2024-04-06-13_34_00_UTC', 'scheduled-2024-04-06-13_35_00_UTC', 'scheduled-2024-04-06-13_36_00_UTC', 'scheduled-2024-04-06-13_37_00_UTC', 'scheduled-2024-04-06-13_38_00_UTC', 'scheduled-2024-04-06-13_39_00_UTC', 'scheduled-2024-04-06-13_40_00_UTC'] 2024-04-06T13:40:15.930 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2024-04-06T13:40:16.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.105+0000 7f761e56b700 1 Processor -- start 2024-04-06T13:40:16.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.105+0000 7f761e56b700 1 -- start start 2024-04-06T13:40:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.106+0000 7f761e56b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f761815d7b0 0x7f76180b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:40:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.106+0000 7f761e56b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76180b3990 con 0x7f761806a730 2024-04-06T13:40:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.106+0000 7f761e56b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76180b3060 con 0x7f761815e500 2024-04-06T13:40:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.106+0000 7f761e56b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76180b3340 con 0x7f761815d7b0 2024-04-06T13:40:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.106+0000 7f76177fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f761806a730 0x7f761815ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46658/0 (socket says 172.21.15.189:46658) 2024-04-06T13:40:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.106+0000 7f76177fe700 1 -- 172.21.15.189:0/2647081550 learned_addr learned my addr 172.21.15.189:0/2647081550 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:40:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.106+0000 7f7617fff700 1 --2- 172.21.15.189:0/2647081550 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f761815d7b0 0x7f76180b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:40:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.106+0000 7f7616ffd700 1 -- 172.21.15.189:0/2647081550 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 867880526 0 0) 0x7f76180b3990 con 0x7f761806a730 2024-04-06T13:40:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.106+0000 7f7616ffd700 1 -- 172.21.15.189:0/2647081550 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76040035b0 con 0x7f761806a730 2024-04-06T13:40:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.106+0000 7f7616ffd700 1 -- 172.21.15.189:0/2647081550 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775487225 0 0) 0x7f76180b3060 con 0x7f761815e500 2024-04-06T13:40:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.107+0000 7f7616ffd700 1 -- 172.21.15.189:0/2647081550 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76180b3990 con 0x7f761815e500 2024-04-06T13:40:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.107+0000 7f7616ffd700 1 -- 172.21.15.189:0/2647081550 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1961730294 0 0) 0x7f76040035b0 con 0x7f761806a730 2024-04-06T13:40:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.107+0000 7f7617fff700 1 -- 172.21.15.189:0/2647081550 >> v1:172.21.15.192:6789/0 conn(0x7f761815e500 legacy=0x7f76180717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:40:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.107+0000 7f7617fff700 1 -- 172.21.15.189:0/2647081550 >> v1:172.21.15.189:6789/0 conn(0x7f761806a730 legacy=0x7f761815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:40:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.107+0000 7f7617fff700 1 -- 172.21.15.189:0/2647081550 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76180b3620 con 0x7f761815d7b0 2024-04-06T13:40:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.107+0000 7f7617fff700 1 --2- 172.21.15.189:0/2647081550 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f761815d7b0 0x7f76180b3e20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7600009870 tx=0x7f7600009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a06ab986fbcc261d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:40:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.107+0000 7f7616ffd700 1 -- 172.21.15.189:0/2647081550 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7608001670 con 0x7f761815d7b0 2024-04-06T13:40:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.107+0000 7f7616ffd700 1 -- 172.21.15.189:0/2647081550 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7600004030 con 0x7f761815d7b0 2024-04-06T13:40:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.107+0000 7f7616ffd700 1 -- 172.21.15.189:0/2647081550 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76000043a0 con 0x7f761815d7b0 2024-04-06T13:40:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.108+0000 7f761e56b700 1 -- 172.21.15.189:0/2647081550 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f761815d7b0 msgr2=0x7f76180b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:40:16.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.108+0000 7f761e56b700 1 --2- 172.21.15.189:0/2647081550 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f761815d7b0 0x7f76180b3e20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7600009870 tx=0x7f7600009b80 comp rx=0 tx=0).stop 2024-04-06T13:40:16.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.108+0000 7f761e56b700 1 -- 172.21.15.189:0/2647081550 shutdown_connections 2024-04-06T13:40:16.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.108+0000 7f761e56b700 1 --2- 172.21.15.189:0/2647081550 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f761815d7b0 0x7f76180b3e20 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:40:16.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.108+0000 7f761e56b700 1 -- 172.21.15.189:0/2647081550 >> 172.21.15.189:0/2647081550 conn(0x7f76180d00b0 msgr2=0x7f76180d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:40:16.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.108+0000 7f761e56b700 1 -- 172.21.15.189:0/2647081550 shutdown_connections 2024-04-06T13:40:16.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.108+0000 7f761e56b700 1 -- 172.21.15.189:0/2647081550 wait complete. 2024-04-06T13:40:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.109+0000 7f761e56b700 1 Processor -- start 2024-04-06T13:40:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.110+0000 7f761e56b700 1 -- start start 2024-04-06T13:40:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.110+0000 7f761e56b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f761806a730 0x7f7618153ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:40:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.110+0000 7f761e56b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f761815d7b0 0x7f76181f6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:40:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.110+0000 7f761e56b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f761815e500 0x7f76181f2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:40:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.110+0000 7f761e56b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7618154430 con 0x7f761815e500 2024-04-06T13:40:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.110+0000 7f761e56b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76181f7000 con 0x7f761806a730 2024-04-06T13:40:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.110+0000 7f761e56b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76181f2fe0 con 0x7f761815d7b0 2024-04-06T13:40:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.110+0000 7f761cb08700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f761815e500 0x7f76181f2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:40:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.111+0000 7f761cb08700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f761815e500 0x7f76181f2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40170/0 (socket says 172.21.15.189:40170) 2024-04-06T13:40:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.111+0000 7f761cb08700 1 -- 172.21.15.189:0/3009954883 learned_addr learned my addr 172.21.15.189:0/3009954883 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:40:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.111+0000 7f76177fe700 1 --2- 172.21.15.189:0/3009954883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f761815d7b0 0x7f76181f6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:40:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.111+0000 7f7617fff700 1 --2- 172.21.15.189:0/3009954883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f761806a730 0x7f7618153ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:40:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.111+0000 7f761cb08700 1 -- 172.21.15.189:0/3009954883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f761815d7b0 msgr2=0x7f76181f6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:40:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.111+0000 7f761cb08700 1 --2- 172.21.15.189:0/3009954883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f761815d7b0 0x7f76181f6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:40:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.111+0000 7f761cb08700 1 -- 172.21.15.189:0/3009954883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f761806a730 msgr2=0x7f7618153ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:40:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.111+0000 7f761cb08700 1 --2- 172.21.15.189:0/3009954883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f761806a730 0x7f7618153ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:40:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.111+0000 7f761cb08700 1 -- 172.21.15.189:0/3009954883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76000094a0 con 0x7f761815e500 2024-04-06T13:40:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.111+0000 7f76177fe700 1 --2- 172.21.15.189:0/3009954883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f761815d7b0 0x7f76181f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:40:16.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.111+0000 7f7617fff700 1 --2- 172.21.15.189:0/3009954883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f761806a730 0x7f7618153ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:40:16.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.111+0000 7f761cb08700 1 --2- 172.21.15.189:0/3009954883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f761815e500 0x7f76181f2aa0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f761000cc50 tx=0x7f761000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:40:16.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.111+0000 7f7614ff9700 1 -- 172.21.15.189:0/3009954883 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761000ada0 con 0x7f761815e500 2024-04-06T13:40:16.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.111+0000 7f7614ff9700 1 -- 172.21.15.189:0/3009954883 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761000ce50 con 0x7f761815e500 2024-04-06T13:40:16.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.111+0000 7f761e56b700 1 -- 172.21.15.189:0/3009954883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76181f3260 con 0x7f761815e500 2024-04-06T13:40:16.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.111+0000 7f7614ff9700 1 -- 172.21.15.189:0/3009954883 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7610003c40 con 0x7f761815e500 2024-04-06T13:40:16.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.111+0000 7f761e56b700 1 -- 172.21.15.189:0/3009954883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76181f3820 con 0x7f761815e500 2024-04-06T13:40:16.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.112+0000 7f761e56b700 1 -- 172.21.15.189:0/3009954883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7618067bb0 con 0x7f761815e500 2024-04-06T13:40:16.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.113+0000 7f7614ff9700 1 -- 172.21.15.189:0/3009954883 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 116) v1 ==== 89276+0+0 (secure 0 0 0) 0x7f76100050d0 con 0x7f761815e500 2024-04-06T13:40:16.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.113+0000 7f7614ff9700 1 --2- 172.21.15.189:0/3009954883 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f75f806ae00 0x7f75f806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:40:16.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.113+0000 7f7614ff9700 1 -- 172.21.15.189:0/3009954883 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f76100899e0 con 0x7f761815e500 2024-04-06T13:40:16.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.113+0000 7f7617fff700 1 --2- 172.21.15.189:0/3009954883 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f75f806ae00 0x7f75f806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:40:16.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.113+0000 7f7617fff700 1 --2- 172.21.15.189:0/3009954883 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f75f806ae00 0x7f75f806d2c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f76181578f0 tx=0x7f7600008030 comp rx=0 tx=0).ready entity=mgr.10308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:40:16.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.118+0000 7f7614ff9700 1 -- 172.21.15.189:0/3009954883 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7610054750 con 0x7f761815e500 2024-04-06T13:40:16.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.339+0000 7f761e56b700 1 -- 172.21.15.189:0/3009954883 --> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7618074990 con 0x7f75f806ae00 2024-04-06T13:40:16.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.344+0000 7f7614ff9700 1 -- 172.21.15.189:0/3009954883 <== mgr.10308 v2:172.21.15.189:6832/47128 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+375 (secure 0 0 0) 0x7f7618074990 con 0x7f75f806ae00 2024-04-06T13:40:16.344 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:40:16.344 INFO:teuthology.orchestra.run.smithi189.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T13:29:31", "first": "2024-04-06T13:30:00", "last": "2024-04-06T13:40:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2024-04-06T13:40:16.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.347+0000 7f761e56b700 1 -- 172.21.15.189:0/3009954883 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f75f806ae00 msgr2=0x7f75f806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:40:16.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.347+0000 7f761e56b700 1 --2- 172.21.15.189:0/3009954883 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f75f806ae00 0x7f75f806d2c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f76181578f0 tx=0x7f7600008030 comp rx=0 tx=0).stop 2024-04-06T13:40:16.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.347+0000 7f761e56b700 1 -- 172.21.15.189:0/3009954883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f761815e500 msgr2=0x7f76181f2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:40:16.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.347+0000 7f761e56b700 1 --2- 172.21.15.189:0/3009954883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f761815e500 0x7f76181f2aa0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f761000cc50 tx=0x7f761000a340 comp rx=0 tx=0).stop 2024-04-06T13:40:16.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.347+0000 7f761e56b700 1 -- 172.21.15.189:0/3009954883 shutdown_connections 2024-04-06T13:40:16.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.347+0000 7f761e56b700 1 --2- 172.21.15.189:0/3009954883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f761815e500 0x7f76181f2aa0 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:40:16.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.347+0000 7f761e56b700 1 --2- 172.21.15.189:0/3009954883 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f75f806ae00 0x7f75f806d2c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:40:16.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.347+0000 7f761e56b700 1 --2- 172.21.15.189:0/3009954883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f761806a730 0x7f7618153ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:40:16.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.347+0000 7f761e56b700 1 --2- 172.21.15.189:0/3009954883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f761815d7b0 0x7f76181f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:40:16.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.347+0000 7f761e56b700 1 -- 172.21.15.189:0/3009954883 >> 172.21.15.189:0/3009954883 conn(0x7f76180d00b0 msgr2=0x7f76180ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:40:16.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.347+0000 7f761e56b700 1 -- 172.21.15.189:0/3009954883 shutdown_connections 2024-04-06T13:40:16.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.347+0000 7f761e56b700 1 -- 172.21.15.189:0/3009954883 wait complete. 2024-04-06T13:40:16.358 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T13:29:31", "first": "2024-04-06T13:30:00", "last": "2024-04-06T13:40:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2024-04-06T13:40:16.358 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2024-04-06T00:00:00', 'created': '2024-04-06T13:29:31', 'first': '2024-04-06T13:30:00', 'last': '2024-04-06T13:40:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2024-04-06T13:40:16.359 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-06T13:40:16.359 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-06T13:40:16.359 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-06T13:40:16.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.480+0000 7f371377a700 1 Processor -- start 2024-04-06T13:40:16.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.480+0000 7f371377a700 1 -- start start 2024-04-06T13:40:16.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.481+0000 7f371377a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f370c06ae40 0x7f370c0b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:40:16.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.481+0000 7f371377a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f370c06ec50 0x7f370c06f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:40:16.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.481+0000 7f371377a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f370c15d9f0 con 0x7f370c15d610 2024-04-06T13:40:16.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.481+0000 7f371377a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f370c0d1040 con 0x7f370c06ae40 2024-04-06T13:40:16.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.481+0000 7f371377a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370c072760 con 0x7f370c06ec50 2024-04-06T13:40:16.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.481+0000 7f3710d15700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f370c06ae40 0x7f370c0b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:40:16.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.481+0000 7f3710d15700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f370c06ae40 0x7f370c0b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40184/0 (socket says 172.21.15.189:40184) 2024-04-06T13:40:16.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.481+0000 7f3710d15700 1 -- 172.21.15.189:0/3874393188 learned_addr learned my addr 172.21.15.189:0/3874393188 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:40:16.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.481+0000 7f3711d17700 1 --2- 172.21.15.189:0/3874393188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f370c06ec50 0x7f370c06f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:40:16.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.481+0000 7f3710d15700 1 -- 172.21.15.189:0/3874393188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f370c06ec50 msgr2=0x7f370c06f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:40:16.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.481+0000 7f3710d15700 1 --2- 172.21.15.189:0/3874393188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f370c06ec50 0x7f370c06f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:40:16.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.481+0000 7f3710d15700 1 -- 172.21.15.189:0/3874393188 >> v1:172.21.15.192:6789/0 conn(0x7f370c15d610 legacy=0x7f370c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:40:16.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.481+0000 7f3710d15700 1 -- 172.21.15.189:0/3874393188 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f370c0719a0 con 0x7f370c06ae40 2024-04-06T13:40:16.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.481+0000 7f3710d15700 1 --2- 172.21.15.189:0/3874393188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f370c06ae40 0x7f370c0b3de0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f36fc0098f0 tx=0x7f36fc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e0fb67d5aedab53 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:40:16.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.481+0000 7f3703fff700 1 -- 172.21.15.189:0/3874393188 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36fc004230 con 0x7f370c06ae40 2024-04-06T13:40:16.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.482+0000 7f3703fff700 1 -- 172.21.15.189:0/3874393188 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36fc004390 con 0x7f370c06ae40 2024-04-06T13:40:16.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.482+0000 7f3703fff700 1 -- 172.21.15.189:0/3874393188 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36fc0036a0 con 0x7f370c06ae40 2024-04-06T13:40:16.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.482+0000 7f371377a700 1 -- 172.21.15.189:0/3874393188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f370c06ae40 msgr2=0x7f370c0b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:40:16.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.482+0000 7f371377a700 1 --2- 172.21.15.189:0/3874393188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f370c06ae40 0x7f370c0b3de0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f36fc0098f0 tx=0x7f36fc009c00 comp rx=0 tx=0).stop 2024-04-06T13:40:16.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.482+0000 7f371377a700 1 -- 172.21.15.189:0/3874393188 shutdown_connections 2024-04-06T13:40:16.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.482+0000 7f371377a700 1 --2- 172.21.15.189:0/3874393188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f370c06ec50 0x7f370c06f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:40:16.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.482+0000 7f371377a700 1 --2- 172.21.15.189:0/3874393188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f370c06ae40 0x7f370c0b3de0 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:40:16.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.482+0000 7f371377a700 1 -- 172.21.15.189:0/3874393188 >> 172.21.15.189:0/3874393188 conn(0x7f370c0d00b0 msgr2=0x7f370c0d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:40:16.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.482+0000 7f371377a700 1 -- 172.21.15.189:0/3874393188 shutdown_connections 2024-04-06T13:40:16.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.482+0000 7f371377a700 1 -- 172.21.15.189:0/3874393188 wait complete. 2024-04-06T13:40:16.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.483+0000 7f371377a700 1 Processor -- start 2024-04-06T13:40:16.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.483+0000 7f371377a700 1 -- start start 2024-04-06T13:40:16.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.484+0000 7f371377a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f370c06ae40 0x7f370c0ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:40:16.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.484+0000 7f371377a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f370c06ec50 0x7f370c1f28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:40:16.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.484+0000 7f371377a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f370c15d610 0x7f370c1f66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:40:16.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.484+0000 7f371377a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370c1f2e00 con 0x7f370c15d610 2024-04-06T13:40:16.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.484+0000 7f371377a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370c1f30d0 con 0x7f370c06ec50 2024-04-06T13:40:16.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.484+0000 7f371377a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370c1f33b0 con 0x7f370c06ae40 2024-04-06T13:40:16.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.484+0000 7f3711d17700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f370c15d610 0x7f370c1f66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:40:16.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.484+0000 7f3711d17700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f370c15d610 0x7f370c1f66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40188/0 (socket says 172.21.15.189:40188) 2024-04-06T13:40:16.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.484+0000 7f3711d17700 1 -- 172.21.15.189:0/1230355070 learned_addr learned my addr 172.21.15.189:0/1230355070 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:40:16.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.484+0000 7f3711516700 1 --2- 172.21.15.189:0/1230355070 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f370c06ae40 0x7f370c0ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:40:16.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.484+0000 7f3710d15700 1 --2- 172.21.15.189:0/1230355070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f370c06ec50 0x7f370c1f28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:40:16.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.484+0000 7f3711d17700 1 -- 172.21.15.189:0/1230355070 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f370c06ae40 msgr2=0x7f370c0ae5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:40:16.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.484+0000 7f3711d17700 1 --2- 172.21.15.189:0/1230355070 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f370c06ae40 0x7f370c0ae5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:40:16.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.484+0000 7f3711d17700 1 -- 172.21.15.189:0/1230355070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f370c06ec50 msgr2=0x7f370c1f28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:40:16.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.484+0000 7f3711d17700 1 --2- 172.21.15.189:0/1230355070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f370c06ec50 0x7f370c1f28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:40:16.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.484+0000 7f3711d17700 1 -- 172.21.15.189:0/1230355070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36fc009550 con 0x7f370c15d610 2024-04-06T13:40:16.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.484+0000 7f3710d15700 1 --2- 172.21.15.189:0/1230355070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f370c06ec50 0x7f370c1f28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:40:16.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.484+0000 7f3711516700 1 --2- 172.21.15.189:0/1230355070 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f370c06ae40 0x7f370c0ae5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:40:16.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.485+0000 7f3711d17700 1 --2- 172.21.15.189:0/1230355070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f370c15d610 0x7f370c1f66b0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f3704007e40 tx=0x7f370400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:40:16.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.485+0000 7f3701ffb700 1 -- 172.21.15.189:0/1230355070 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370400eea0 con 0x7f370c15d610 2024-04-06T13:40:16.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.485+0000 7f3701ffb700 1 -- 172.21.15.189:0/1230355070 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3704014420 con 0x7f370c15d610 2024-04-06T13:40:16.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.485+0000 7f371377a700 1 -- 172.21.15.189:0/1230355070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f370c1f3690 con 0x7f370c15d610 2024-04-06T13:40:16.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.485+0000 7f3701ffb700 1 -- 172.21.15.189:0/1230355070 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37040147b0 con 0x7f370c15d610 2024-04-06T13:40:16.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.485+0000 7f371377a700 1 -- 172.21.15.189:0/1230355070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f370c1f6190 con 0x7f370c15d610 2024-04-06T13:40:16.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.486+0000 7f371377a700 1 -- 172.21.15.189:0/1230355070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f370c074990 con 0x7f370c15d610 2024-04-06T13:40:16.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.486+0000 7f3701ffb700 1 -- 172.21.15.189:0/1230355070 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 116) v1 ==== 89276+0+0 (secure 0 0 0) 0x7f3704014910 con 0x7f370c15d610 2024-04-06T13:40:16.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.487+0000 7f3701ffb700 1 --2- 172.21.15.189:0/1230355070 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f36f006ae00 0x7f36f006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:40:16.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.487+0000 7f3701ffb700 1 -- 172.21.15.189:0/1230355070 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f370408a000 con 0x7f370c15d610 2024-04-06T13:40:16.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.487+0000 7f3711516700 1 --2- 172.21.15.189:0/1230355070 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f36f006ae00 0x7f36f006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:40:16.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.487+0000 7f3711516700 1 --2- 172.21.15.189:0/1230355070 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f36f006ae00 0x7f36f006d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3708009930 tx=0x7f3708008040 comp rx=0 tx=0).ready entity=mgr.10308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:40:16.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.491+0000 7f3701ffb700 1 -- 172.21.15.189:0/1230355070 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3704054e00 con 0x7f370c15d610 2024-04-06T13:40:16.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.743+0000 7f371377a700 1 -- 172.21.15.189:0/1230355070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f370c0b37e0 con 0x7f370c15d610 2024-04-06T13:40:16.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.746+0000 7f3701ffb700 1 -- 172.21.15.189:0/1230355070 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v116) v1 ==== 74+0+171155 (secure 0 0 0) 0x7f3704058420 con 0x7f370c15d610 2024-04-06T13:40:16.749 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:40:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.749+0000 7f371377a700 1 -- 172.21.15.189:0/1230355070 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f36f006ae00 msgr2=0x7f36f006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:40:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.749+0000 7f371377a700 1 --2- 172.21.15.189:0/1230355070 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f36f006ae00 0x7f36f006d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3708009930 tx=0x7f3708008040 comp rx=0 tx=0).stop 2024-04-06T13:40:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.749+0000 7f371377a700 1 -- 172.21.15.189:0/1230355070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f370c15d610 msgr2=0x7f370c1f66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:40:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.749+0000 7f371377a700 1 --2- 172.21.15.189:0/1230355070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f370c15d610 0x7f370c1f66b0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f3704007e40 tx=0x7f370400e3c0 comp rx=0 tx=0).stop 2024-04-06T13:40:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.749+0000 7f371377a700 1 -- 172.21.15.189:0/1230355070 shutdown_connections 2024-04-06T13:40:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.749+0000 7f371377a700 1 --2- 172.21.15.189:0/1230355070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f370c15d610 0x7f370c1f66b0 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:40:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.749+0000 7f371377a700 1 --2- 172.21.15.189:0/1230355070 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f370c06ae40 0x7f370c0ae5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:40:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.749+0000 7f371377a700 1 --2- 172.21.15.189:0/1230355070 >> [v2:172.21.15.189:6832/47128,v1:172.21.15.189:6833/47128] conn(0x7f36f006ae00 0x7f36f006d2c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:40:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.749+0000 7f371377a700 1 --2- 172.21.15.189:0/1230355070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f370c06ec50 0x7f370c1f28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:40:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.749+0000 7f371377a700 1 -- 172.21.15.189:0/1230355070 >> 172.21.15.189:0/1230355070 conn(0x7f370c0d00b0 msgr2=0x7f370c0b0480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:40:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.749+0000 7f371377a700 1 -- 172.21.15.189:0/1230355070 shutdown_connections 2024-04-06T13:40:16.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:40:16.749+0000 7f371377a700 1 -- 172.21.15.189:0/1230355070 wait complete. 2024-04-06T13:40:16.762 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":116,"flags":0,"active_gid":10308,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":47128},{"type":"v1","addr":"172.21.15.189:6833","nonce":47128}]},"active_addr":"172.21.15.189:6833/47128","active_change":"2024-04-06T13:29:30.406519+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":8237,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":193,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":3656453012}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":1974012472}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":2710277231}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":1953253738}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":2983879481}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":1090714269}]}]} 2024-04-06T13:40:16.765 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: y 2024-04-06T13:40:16.766 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-06T13:40:16.766 INFO:tasks.ceph.mgr.y:Stopping old one... 2024-04-06T13:40:16.766 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-06T13:40:16.766 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:40:16.795 INFO:tasks.ceph.mgr.y:Stopped 2024-04-06T13:40:16.795 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-06T13:40:16.838 INFO:tasks.ceph.mgr.y:Started 2024-04-06T13:40:17.252 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:17.251+0000 7f19a4a47200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:40:17.330 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:17.330+0000 7f19a4a47200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:40:17.403 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:17.402+0000 7f19a4a47200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:40:17.471 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:17.470+0000 7f19a4a47200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:40:17.646 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:17.645+0000 7f19a4a47200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:40:17.802 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:17.801+0000 7f19a4a47200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:40:17.911 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:17.910+0000 7f19a4a47200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:40:17.992 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:17.992+0000 7f19a4a47200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:40:18.067 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:18.066+0000 7f19a4a47200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:40:18.272 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:18.271+0000 7f19a4a47200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:40:18.348 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:18.347+0000 7f19a4a47200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:40:18.667 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:18.666+0000 7f19a4a47200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:40:18.988 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:18.988+0000 7f19a4a47200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:40:19.148 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:19.147+0000 7f19a4a47200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:40:19.220 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:19.219+0000 7f19a4a47200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:40:19.838 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:19.838+0000 7f19a4a47200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:40:19.915 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:19.915+0000 7f19a4a47200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:40:20.041 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:20.040+0000 7f19a4a47200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:40:20.553 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:20.552+0000 7f19a4a47200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:40:20.708 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:20.707+0000 7f19a4a47200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:40:20.865 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:20.864+0000 7f19a4a47200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:40:21.589 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:21.589+0000 7f19a4a47200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:40:21.831 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:21.830+0000 7f19a4a47200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:40:21.909 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:21.908+0000 7f19a4a47200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:40:22.286 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:22.286+0000 7f19a4a47200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:40:22.444 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:22.443+0000 7f191a0da700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:40:22.444 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:22.443+0000 7f191a0da700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:40:22.444 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:22.443+0000 7f191a0da700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:40:22.444 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:22.443+0000 7f191a0da700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:40:22.444 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:22.443+0000 7f191a0da700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:40:22.445 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:22.444+0000 7f19168d3700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:40:22.445 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:22.444+0000 7f19168d3700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:40:22.445 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:22.444+0000 7f19168d3700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:40:22.445 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:22.444+0000 7f19168d3700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:40:22.445 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:40:22.444+0000 7f19168d3700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:40:31.197 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:40:31.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:41:00.442 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:41:00.441+0000 7f19220ea700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:41:00.442 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:41:00.442+0000 7f19220ea700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:41:00.442 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:41:00.442+0000 7f19220ea700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:41:00.442 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:41:00.442+0000 7f19220ea700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:41:00.442 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:41:00.442+0000 7f19220ea700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:41:01.336 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:41:01.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:41:31.460 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:41:31.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:42:01.665 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:42:01.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:42:31.739 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:42:31.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:43:01.934 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:43:01.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:43:32.072 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:43:32.076 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:44:02.162 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:44:02.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:44:32.305 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:44:32.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:45:02.444 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:45:02.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:45:16.839 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/test_restart 1m 2024-04-06T13:45:16.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.980+0000 7fef4cb78700 1 Processor -- start 2024-04-06T13:45:16.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.980+0000 7fef4cb78700 1 -- start start 2024-04-06T13:45:16.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.981+0000 7fef4cb78700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fef4815d7b0 0x7fef480b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:16.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.981+0000 7fef4cb78700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef480b3990 con 0x7fef4806a730 2024-04-06T13:45:16.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.981+0000 7fef4cb78700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef480b3060 con 0x7fef4815e500 2024-04-06T13:45:16.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.981+0000 7fef4cb78700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef4815e950 con 0x7fef4815d7b0 2024-04-06T13:45:16.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.981+0000 7fef45d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fef4806a730 0x7fef4815ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60822/0 (socket says 172.21.15.189:60822) 2024-04-06T13:45:16.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.981+0000 7fef45d9b700 1 -- 172.21.15.189:0/3461169729 learned_addr learned my addr 172.21.15.189:0/3461169729 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:16.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.981+0000 7fef4659c700 1 --2- 172.21.15.189:0/3461169729 >> v2:172.21.15.192:3300/0 conn(0x7fef4815d7b0 0x7fef480b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:16.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.981+0000 7fef4559a700 1 -- 172.21.15.189:0/3461169729 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314509948 0 0) 0x7fef480b3990 con 0x7fef4806a730 2024-04-06T13:45:16.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.981+0000 7fef4559a700 1 -- 172.21.15.189:0/3461169729 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef300035b0 con 0x7fef4806a730 2024-04-06T13:45:16.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4559a700 1 -- 172.21.15.189:0/3461169729 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3452659757 0 0) 0x7fef300035b0 con 0x7fef4806a730 2024-04-06T13:45:16.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4559a700 1 -- 172.21.15.189:0/3461169729 >> v2:172.21.15.192:3300/0 conn(0x7fef4815d7b0 msgr2=0x7fef480b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:16.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4559a700 1 --2- 172.21.15.189:0/3461169729 >> v2:172.21.15.192:3300/0 conn(0x7fef4815d7b0 0x7fef480b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:16.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4559a700 1 -- 172.21.15.189:0/3461169729 >> v1:172.21.15.192:6789/0 conn(0x7fef4815e500 legacy=0x7fef480717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:16.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4559a700 1 -- 172.21.15.189:0/3461169729 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef480b3340 con 0x7fef4806a730 2024-04-06T13:45:16.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4559a700 1 -- 172.21.15.189:0/3461169729 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fef38001670 con 0x7fef4806a730 2024-04-06T13:45:16.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4559a700 1 -- 172.21.15.189:0/3461169729 >> v1:172.21.15.189:6789/0 conn(0x7fef4806a730 legacy=0x7fef4815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:16.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4559a700 1 --2- 172.21.15.189:0/3461169729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30003a00 0x7fef30003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:16.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4559a700 1 --2- 172.21.15.189:0/3461169729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef30004850 0x7fef30004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:16.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4559a700 1 --2- 172.21.15.189:0/3461169729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef30005210 0x7fef3000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:16.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4559a700 1 -- 172.21.15.189:0/3461169729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3000c180 con 0x7fef30003a00 2024-04-06T13:45:16.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4559a700 1 -- 172.21.15.189:0/3461169729 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3000c2f0 con 0x7fef30004850 2024-04-06T13:45:16.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4559a700 1 -- 172.21.15.189:0/3461169729 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3000c4b0 con 0x7fef30005210 2024-04-06T13:45:16.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4659c700 1 --2- 172.21.15.189:0/3461169729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30003a00 0x7fef30003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:16.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef45d9b700 1 --2- 172.21.15.189:0/3461169729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef30004850 0x7fef30004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:16.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef46d9d700 1 --2- 172.21.15.189:0/3461169729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef30005210 0x7fef3000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:16.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4659c700 1 -- 172.21.15.189:0/3461169729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef30005210 msgr2=0x7fef3000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:16.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4659c700 1 --2- 172.21.15.189:0/3461169729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef30005210 0x7fef3000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:16.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4659c700 1 -- 172.21.15.189:0/3461169729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef30004850 msgr2=0x7fef30004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:16.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4659c700 1 --2- 172.21.15.189:0/3461169729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef30004850 0x7fef30004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:16.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4659c700 1 -- 172.21.15.189:0/3461169729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef300035b0 con 0x7fef30003a00 2024-04-06T13:45:16.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef46d9d700 1 -- 172.21.15.189:0/3461169729 reap_dead start 2024-04-06T13:45:16.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef45d9b700 1 --2- 172.21.15.189:0/3461169729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef30004850 0x7fef30004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:16.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4659c700 1 --2- 172.21.15.189:0/3461169729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30003a00 0x7fef30003e20 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7fef3c009b60 tx=0x7fef30005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9928b9c01d0f0d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:16.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4559a700 1 -- 172.21.15.189:0/3461169729 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c00e6c0 con 0x7fef30003a00 2024-04-06T13:45:16.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4559a700 1 -- 172.21.15.189:0/3461169729 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3c00e820 con 0x7fef30003a00 2024-04-06T13:45:16.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.982+0000 7fef4559a700 1 -- 172.21.15.189:0/3461169729 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c00eb90 con 0x7fef30003a00 2024-04-06T13:45:16.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.983+0000 7fef4cb78700 1 -- 172.21.15.189:0/3461169729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30003a00 msgr2=0x7fef30003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:16.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.983+0000 7fef4cb78700 1 --2- 172.21.15.189:0/3461169729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30003a00 0x7fef30003e20 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7fef3c009b60 tx=0x7fef30005880 comp rx=0 tx=0).stop 2024-04-06T13:45:16.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.983+0000 7fef4cb78700 1 -- 172.21.15.189:0/3461169729 shutdown_connections 2024-04-06T13:45:16.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.983+0000 7fef4cb78700 1 --2- 172.21.15.189:0/3461169729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30003a00 0x7fef30003e20 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:16.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.983+0000 7fef4cb78700 1 -- 172.21.15.189:0/3461169729 >> 172.21.15.189:0/3461169729 conn(0x7fef480d00b0 msgr2=0x7fef480d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:16.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.983+0000 7fef4cb78700 1 -- 172.21.15.189:0/3461169729 shutdown_connections 2024-04-06T13:45:16.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.983+0000 7fef4cb78700 1 -- 172.21.15.189:0/3461169729 wait complete. 2024-04-06T13:45:16.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.984+0000 7fef4cb78700 1 Processor -- start 2024-04-06T13:45:16.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.985+0000 7fef4cb78700 1 -- start start 2024-04-06T13:45:16.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.985+0000 7fef4cb78700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30003a00 0x7fef48151d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:16.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.985+0000 7fef4cb78700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef481f61a0 0x7fef480b0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:16.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.985+0000 7fef4cb78700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef480cd8a0 0x7fef480ad120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:16.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.985+0000 7fef4cb78700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef48160c20 con 0x7fef30003a00 2024-04-06T13:45:16.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.985+0000 7fef4cb78700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef480b0dd0 con 0x7fef481f61a0 2024-04-06T13:45:16.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.985+0000 7fef4cb78700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef481f6710 con 0x7fef480cd8a0 2024-04-06T13:45:16.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.985+0000 7fef4659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30003a00 0x7fef48151d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:16.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.985+0000 7fef4659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30003a00 0x7fef48151d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45192/0 (socket says 172.21.15.189:45192) 2024-04-06T13:45:16.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.985+0000 7fef4659c700 1 -- 172.21.15.189:0/4023346686 learned_addr learned my addr 172.21.15.189:0/4023346686 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:16.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.986+0000 7fef45d9b700 1 --2- 172.21.15.189:0/4023346686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef481f61a0 0x7fef480b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:16.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.986+0000 7fef46d9d700 1 --2- 172.21.15.189:0/4023346686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef480cd8a0 0x7fef480ad120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:16.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.986+0000 7fef4659c700 1 -- 172.21.15.189:0/4023346686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef480cd8a0 msgr2=0x7fef480ad120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:16.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.986+0000 7fef4659c700 1 --2- 172.21.15.189:0/4023346686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef480cd8a0 0x7fef480ad120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:16.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.986+0000 7fef4659c700 1 -- 172.21.15.189:0/4023346686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef481f61a0 msgr2=0x7fef480b0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:16.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.986+0000 7fef4659c700 1 --2- 172.21.15.189:0/4023346686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef481f61a0 0x7fef480b0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:16.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.986+0000 7fef4659c700 1 -- 172.21.15.189:0/4023346686 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef481f69f0 con 0x7fef30003a00 2024-04-06T13:45:16.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.986+0000 7fef4659c700 1 --2- 172.21.15.189:0/4023346686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30003a00 0x7fef48151d80 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7fef3000c2c0 tx=0x7fef3c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:16.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.986+0000 7fef36ffd700 1 -- 172.21.15.189:0/4023346686 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c00e6c0 con 0x7fef30003a00 2024-04-06T13:45:16.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.986+0000 7fef36ffd700 1 -- 172.21.15.189:0/4023346686 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3c003ad0 con 0x7fef30003a00 2024-04-06T13:45:16.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.986+0000 7fef36ffd700 1 -- 172.21.15.189:0/4023346686 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c003e80 con 0x7fef30003a00 2024-04-06T13:45:16.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.986+0000 7fef4cb78700 1 -- 172.21.15.189:0/4023346686 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef481f6cd0 con 0x7fef30003a00 2024-04-06T13:45:16.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.986+0000 7fef4cb78700 1 -- 172.21.15.189:0/4023346686 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef48158990 con 0x7fef30003a00 2024-04-06T13:45:16.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.987+0000 7fef4cb78700 1 -- 172.21.15.189:0/4023346686 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef48158d00 con 0x7fef30003a00 2024-04-06T13:45:16.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.987+0000 7fef36ffd700 1 -- 172.21.15.189:0/4023346686 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 122) v1 ==== 89276+0+0 (secure 0 0 0) 0x7fef3c013470 con 0x7fef30003a00 2024-04-06T13:45:16.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.988+0000 7fef36ffd700 1 --2- 172.21.15.189:0/4023346686 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fef2806ae00 0x7fef2806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:16.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.988+0000 7fef45d9b700 1 --2- 172.21.15.189:0/4023346686 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fef2806ae00 0x7fef2806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:16.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.988+0000 7fef36ffd700 1 -- 172.21.15.189:0/4023346686 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fef3c08af20 con 0x7fef30003a00 2024-04-06T13:45:16.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.988+0000 7fef45d9b700 1 --2- 172.21.15.189:0/4023346686 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fef2806ae00 0x7fef2806d2c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fef48160860 tx=0x7fef38009040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:16.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:16.993+0000 7fef36ffd700 1 -- 172.21.15.189:0/4023346686 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fef3c055dd0 con 0x7fef30003a00 2024-04-06T13:45:17.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.213+0000 7fef4cb78700 1 -- 172.21.15.189:0/4023346686 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fef4815eb40 con 0x7fef2806ae00 2024-04-06T13:45:17.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.257+0000 7fef36ffd700 1 -- 172.21.15.189:0/4023346686 <== mgr.10398 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fef380015b0 con 0x7fef2806ae00 2024-04-06T13:45:17.257 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2024-04-06T13:45:17.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.259+0000 7fef4cb78700 1 -- 172.21.15.189:0/4023346686 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fef2806ae00 msgr2=0x7fef2806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:17.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.259+0000 7fef4cb78700 1 --2- 172.21.15.189:0/4023346686 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fef2806ae00 0x7fef2806d2c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fef48160860 tx=0x7fef38009040 comp rx=0 tx=0).stop 2024-04-06T13:45:17.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.259+0000 7fef4cb78700 1 -- 172.21.15.189:0/4023346686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30003a00 msgr2=0x7fef48151d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:17.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.259+0000 7fef4cb78700 1 --2- 172.21.15.189:0/4023346686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30003a00 0x7fef48151d80 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7fef3000c2c0 tx=0x7fef3c0047c0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.259+0000 7fef4cb78700 1 -- 172.21.15.189:0/4023346686 shutdown_connections 2024-04-06T13:45:17.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.259+0000 7fef4cb78700 1 --2- 172.21.15.189:0/4023346686 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fef2806ae00 0x7fef2806d2c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.259+0000 7fef4cb78700 1 --2- 172.21.15.189:0/4023346686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef480cd8a0 0x7fef480ad120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.259+0000 7fef4cb78700 1 --2- 172.21.15.189:0/4023346686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef30003a00 0x7fef48151d80 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.259+0000 7fef4cb78700 1 --2- 172.21.15.189:0/4023346686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef481f61a0 0x7fef480b0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.259+0000 7fef4cb78700 1 -- 172.21.15.189:0/4023346686 >> 172.21.15.189:0/4023346686 conn(0x7fef480d00b0 msgr2=0x7fef480717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:17.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.260+0000 7fef4cb78700 1 -- 172.21.15.189:0/4023346686 shutdown_connections 2024-04-06T13:45:17.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.260+0000 7fef4cb78700 1 -- 172.21.15.189:0/4023346686 wait complete. 2024-04-06T13:45:17.272 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2024-04-06T13:45:17.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:17.272 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-04-06T13:45:17.290 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_30_00_UTC 2024-04-06T13:45:17.290 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_31_00_UTC 2024-04-06T13:45:17.290 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_32_00_UTC 2024-04-06T13:45:17.290 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_33_00_UTC 2024-04-06T13:45:17.290 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_34_00_UTC 2024-04-06T13:45:17.291 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_35_00_UTC 2024-04-06T13:45:17.291 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_36_00_UTC 2024-04-06T13:45:17.291 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_37_00_UTC 2024-04-06T13:45:17.291 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_38_00_UTC 2024-04-06T13:45:17.291 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_39_00_UTC 2024-04-06T13:45:17.291 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_40_00_UTC 2024-04-06T13:45:17.291 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_41_00_UTC 2024-04-06T13:45:17.291 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_42_00_UTC 2024-04-06T13:45:17.291 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_43_00_UTC 2024-04-06T13:45:17.291 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_44_00_UTC 2024-04-06T13:45:17.291 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_45_00_UTC 2024-04-06T13:45:17.291 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-06-13_30_00_UTC', 'scheduled-2024-04-06-13_31_00_UTC', 'scheduled-2024-04-06-13_32_00_UTC', 'scheduled-2024-04-06-13_33_00_UTC', 'scheduled-2024-04-06-13_34_00_UTC', 'scheduled-2024-04-06-13_35_00_UTC', 'scheduled-2024-04-06-13_36_00_UTC', 'scheduled-2024-04-06-13_37_00_UTC', 'scheduled-2024-04-06-13_38_00_UTC', 'scheduled-2024-04-06-13_39_00_UTC', 'scheduled-2024-04-06-13_40_00_UTC', 'scheduled-2024-04-06-13_41_00_UTC', 'scheduled-2024-04-06-13_42_00_UTC', 'scheduled-2024-04-06-13_43_00_UTC', 'scheduled-2024-04-06-13_44_00_UTC', 'scheduled-2024-04-06-13_45_00_UTC'] 2024-04-06T13:45:17.292 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2024-04-06T13:45:17.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.454+0000 7fe3e9f1d700 1 Processor -- start 2024-04-06T13:45:17.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.454+0000 7fe3e9f1d700 1 -- start start 2024-04-06T13:45:17.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.455+0000 7fe3e9f1d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe3e415a550 0x7fe3e40cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:17.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.455+0000 7fe3e9f1d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e40cf900 0x7fe3e40b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:17.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.455+0000 7fe3e9f1d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3e40cdbe0 con 0x7fe3e40b4ba0 2024-04-06T13:45:17.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.455+0000 7fe3e9f1d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3e40b5010 con 0x7fe3e415a550 2024-04-06T13:45:17.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.455+0000 7fe3e9f1d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e415a940 con 0x7fe3e40cf900 2024-04-06T13:45:17.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.455+0000 7fe3e37fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe3e415a550 0x7fe3e40cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:17.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.455+0000 7fe3e3fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe3e40b4ba0 0x7fe3e406eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60836/0 (socket says 172.21.15.189:60836) 2024-04-06T13:45:17.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.455+0000 7fe3e37fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe3e415a550 0x7fe3e40cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45198/0 (socket says 172.21.15.189:45198) 2024-04-06T13:45:17.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.455+0000 7fe3e3fff700 1 -- 172.21.15.189:0/1617579367 learned_addr learned my addr 172.21.15.189:0/1617579367 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:17.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.455+0000 7fe3e2ffd700 1 --2- 172.21.15.189:0/1617579367 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e40cf900 0x7fe3e40b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:17.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.455+0000 7fe3e37fe700 1 -- 172.21.15.189:0/1617579367 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e40cf900 msgr2=0x7fe3e40b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:17.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.455+0000 7fe3e37fe700 1 --2- 172.21.15.189:0/1617579367 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e40cf900 0x7fe3e40b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.455+0000 7fe3e37fe700 1 -- 172.21.15.189:0/1617579367 >> v1:172.21.15.189:6789/0 conn(0x7fe3e40b4ba0 legacy=0x7fe3e406eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:17.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.455+0000 7fe3e27fc700 1 -- 172.21.15.189:0/1617579367 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850691301 0 0) 0x7fe3e40cdbe0 con 0x7fe3e40b4ba0 2024-04-06T13:45:17.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.455+0000 7fe3e37fe700 1 -- 172.21.15.189:0/1617579367 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e415ac00 con 0x7fe3e415a550 2024-04-06T13:45:17.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.455+0000 7fe3e2ffd700 1 --2- 172.21.15.189:0/1617579367 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e40cf900 0x7fe3e40b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:45:17.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.455+0000 7fe3e37fe700 1 --2- 172.21.15.189:0/1617579367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e415a550 0x7fe3e40cf3c0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8009840 tx=0x7fe3d8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93eb6d8f3b207d41 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:17.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.456+0000 7fe3e27fc700 1 -- 172.21.15.189:0/1617579367 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8011830 con 0x7fe3e415a550 2024-04-06T13:45:17.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.456+0000 7fe3e27fc700 1 -- 172.21.15.189:0/1617579367 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d8011990 con 0x7fe3e415a550 2024-04-06T13:45:17.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.456+0000 7fe3e27fc700 1 -- 172.21.15.189:0/1617579367 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8011d20 con 0x7fe3e415a550 2024-04-06T13:45:17.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.456+0000 7fe3e9f1d700 1 -- 172.21.15.189:0/1617579367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e415a550 msgr2=0x7fe3e40cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:17.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.456+0000 7fe3e9f1d700 1 --2- 172.21.15.189:0/1617579367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e415a550 0x7fe3e40cf3c0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8009840 tx=0x7fe3d8009b50 comp rx=0 tx=0).stop 2024-04-06T13:45:17.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.456+0000 7fe3e9f1d700 1 -- 172.21.15.189:0/1617579367 shutdown_connections 2024-04-06T13:45:17.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.456+0000 7fe3e9f1d700 1 --2- 172.21.15.189:0/1617579367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e415a550 0x7fe3e40cf3c0 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.456+0000 7fe3e9f1d700 1 --2- 172.21.15.189:0/1617579367 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e40cf900 0x7fe3e40b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.456+0000 7fe3e9f1d700 1 -- 172.21.15.189:0/1617579367 >> 172.21.15.189:0/1617579367 conn(0x7fe3e40d00b0 msgr2=0x7fe3e40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:17.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.456+0000 7fe3e9f1d700 1 -- 172.21.15.189:0/1617579367 shutdown_connections 2024-04-06T13:45:17.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.456+0000 7fe3e9f1d700 1 -- 172.21.15.189:0/1617579367 wait complete. 2024-04-06T13:45:17.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.457+0000 7fe3e9f1d700 1 Processor -- start 2024-04-06T13:45:17.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.458+0000 7fe3e9f1d700 1 -- start start 2024-04-06T13:45:17.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.458+0000 7fe3e9f1d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e40b4ba0 0x7fe3e41582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:17.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.458+0000 7fe3e9f1d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e40cf900 0x7fe3e4159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:17.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.458+0000 7fe3e9f1d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3e41f24d0 0x7fe3e40ada00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:17.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.458+0000 7fe3e9f1d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e4159520 con 0x7fe3e40b4ba0 2024-04-06T13:45:17.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.458+0000 7fe3e9f1d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e41588a0 con 0x7fe3e41f24d0 2024-04-06T13:45:17.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.458+0000 7fe3e9f1d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e4158b80 con 0x7fe3e40cf900 2024-04-06T13:45:17.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.458+0000 7fe3e37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e40b4ba0 0x7fe3e41582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:17.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.458+0000 7fe3e37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e40b4ba0 0x7fe3e41582c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45212/0 (socket says 172.21.15.189:45212) 2024-04-06T13:45:17.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.459+0000 7fe3e37fe700 1 -- 172.21.15.189:0/2624267277 learned_addr learned my addr 172.21.15.189:0/2624267277 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:17.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.459+0000 7fe3e3fff700 1 --2- 172.21.15.189:0/2624267277 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3e41f24d0 0x7fe3e40ada00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:17.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.459+0000 7fe3e2ffd700 1 --2- 172.21.15.189:0/2624267277 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e40cf900 0x7fe3e4159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:17.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.459+0000 7fe3e37fe700 1 -- 172.21.15.189:0/2624267277 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e40cf900 msgr2=0x7fe3e4159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:17.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.459+0000 7fe3e37fe700 1 --2- 172.21.15.189:0/2624267277 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e40cf900 0x7fe3e4159940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.459+0000 7fe3e37fe700 1 -- 172.21.15.189:0/2624267277 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3e41f24d0 msgr2=0x7fe3e40ada00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:17.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.459+0000 7fe3e37fe700 1 --2- 172.21.15.189:0/2624267277 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3e41f24d0 0x7fe3e40ada00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.459+0000 7fe3e37fe700 1 -- 172.21.15.189:0/2624267277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d80094a0 con 0x7fe3e40b4ba0 2024-04-06T13:45:17.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.459+0000 7fe3e37fe700 1 --2- 172.21.15.189:0/2624267277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e40b4ba0 0x7fe3e41582c0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8000c00 tx=0x7fe3d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:17.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.459+0000 7fe3cbfff700 1 -- 172.21.15.189:0/2624267277 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d80045f0 con 0x7fe3e40b4ba0 2024-04-06T13:45:17.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.459+0000 7fe3cbfff700 1 -- 172.21.15.189:0/2624267277 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d801f070 con 0x7fe3e40b4ba0 2024-04-06T13:45:17.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.459+0000 7fe3e9f1d700 1 -- 172.21.15.189:0/2624267277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e4158e60 con 0x7fe3e40b4ba0 2024-04-06T13:45:17.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.459+0000 7fe3cbfff700 1 -- 172.21.15.189:0/2624267277 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8003ae0 con 0x7fe3e40b4ba0 2024-04-06T13:45:17.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.459+0000 7fe3e9f1d700 1 -- 172.21.15.189:0/2624267277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3e40adf40 con 0x7fe3e40b4ba0 2024-04-06T13:45:17.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.460+0000 7fe3e9f1d700 1 -- 172.21.15.189:0/2624267277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e40ae0d0 con 0x7fe3e40b4ba0 2024-04-06T13:45:17.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.460+0000 7fe3cbfff700 1 -- 172.21.15.189:0/2624267277 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 122) v1 ==== 89276+0+0 (secure 0 0 0) 0x7fe3d8003c40 con 0x7fe3e40b4ba0 2024-04-06T13:45:17.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.461+0000 7fe3cbfff700 1 --2- 172.21.15.189:0/2624267277 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe3d006ae00 0x7fe3d006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:17.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.461+0000 7fe3cbfff700 1 -- 172.21.15.189:0/2624267277 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fe3d8021070 con 0x7fe3e40b4ba0 2024-04-06T13:45:17.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.461+0000 7fe3e2ffd700 1 --2- 172.21.15.189:0/2624267277 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe3d006ae00 0x7fe3d006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:17.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.461+0000 7fe3e2ffd700 1 --2- 172.21.15.189:0/2624267277 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe3d006ae00 0x7fe3d006d2c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4156f80 tx=0x7fe3d4008040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:17.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.465+0000 7fe3cbfff700 1 -- 172.21.15.189:0/2624267277 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe3e40ae0d0 con 0x7fe3e40b4ba0 2024-04-06T13:45:17.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.684+0000 7fe3e9f1d700 1 -- 172.21.15.189:0/2624267277 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe3e415b4a0 con 0x7fe3d006ae00 2024-04-06T13:45:17.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.690+0000 7fe3cbfff700 1 -- 172.21.15.189:0/2624267277 <== mgr.10398 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+376 (secure 0 0 0) 0x7fe3d400c050 con 0x7fe3d006ae00 2024-04-06T13:45:17.690 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:17.690 INFO:teuthology.orchestra.run.smithi189.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T13:29:31", "first": "2024-04-06T13:30:00", "last": "2024-04-06T13:45:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2024-04-06T13:45:17.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.692+0000 7fe3e9f1d700 1 -- 172.21.15.189:0/2624267277 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe3d006ae00 msgr2=0x7fe3d006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:17.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.692+0000 7fe3e9f1d700 1 --2- 172.21.15.189:0/2624267277 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe3d006ae00 0x7fe3d006d2c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4156f80 tx=0x7fe3d4008040 comp rx=0 tx=0).stop 2024-04-06T13:45:17.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.692+0000 7fe3e9f1d700 1 -- 172.21.15.189:0/2624267277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e40b4ba0 msgr2=0x7fe3e41582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:17.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.692+0000 7fe3e9f1d700 1 --2- 172.21.15.189:0/2624267277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e40b4ba0 0x7fe3e41582c0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8000c00 tx=0x7fe3d8004750 comp rx=0 tx=0).stop 2024-04-06T13:45:17.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.692+0000 7fe3e9f1d700 1 -- 172.21.15.189:0/2624267277 shutdown_connections 2024-04-06T13:45:17.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.692+0000 7fe3e9f1d700 1 --2- 172.21.15.189:0/2624267277 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe3d006ae00 0x7fe3d006d2c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.692+0000 7fe3e9f1d700 1 --2- 172.21.15.189:0/2624267277 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3e41f24d0 0x7fe3e40ada00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.692+0000 7fe3e9f1d700 1 --2- 172.21.15.189:0/2624267277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e40b4ba0 0x7fe3e41582c0 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.692+0000 7fe3e9f1d700 1 --2- 172.21.15.189:0/2624267277 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3e40cf900 0x7fe3e4159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.692+0000 7fe3e9f1d700 1 -- 172.21.15.189:0/2624267277 >> 172.21.15.189:0/2624267277 conn(0x7fe3e40d00b0 msgr2=0x7fe3e40afcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:17.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.692+0000 7fe3e9f1d700 1 -- 172.21.15.189:0/2624267277 shutdown_connections 2024-04-06T13:45:17.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.692+0000 7fe3e9f1d700 1 -- 172.21.15.189:0/2624267277 wait complete. 2024-04-06T13:45:17.704 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T13:29:31", "first": "2024-04-06T13:30:00", "last": "2024-04-06T13:45:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2024-04-06T13:45:17.704 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2024-04-06T00:00:00', 'created': '2024-04-06T13:29:31', 'first': '2024-04-06T13:30:00', 'last': '2024-04-06T13:45:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2024-04-06T13:45:17.704 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 16 2024-04-06T13:45:17.704 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 16 2024-04-06T13:45:17.704 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/test_restart 1m 2024-04-06T13:45:17.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.827+0000 7f94ae57f700 1 Processor -- start 2024-04-06T13:45:17.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.827+0000 7f94ae57f700 1 -- start start 2024-04-06T13:45:17.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.828+0000 7f94ae57f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f94a815d7b0 0x7f94a80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:17.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.828+0000 7f94ae57f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f94a815e500 0x7f94a806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:17.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.828+0000 7f94ae57f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a80b3990 con 0x7f94a806a730 2024-04-06T13:45:17.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.828+0000 7f94ae57f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94a815e950 con 0x7f94a815d7b0 2024-04-06T13:45:17.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.828+0000 7f94ae57f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94a806b0f0 con 0x7f94a815e500 2024-04-06T13:45:17.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.828+0000 7f94a7fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f94a815d7b0 0x7f94a80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:17.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.829+0000 7f94a7fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f94a815d7b0 0x7f94a80b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45226/0 (socket says 172.21.15.189:45226) 2024-04-06T13:45:17.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.829+0000 7f94a7fff700 1 -- 172.21.15.189:0/93010157 learned_addr learned my addr 172.21.15.189:0/93010157 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:17.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.829+0000 7f94acb1c700 1 --2- 172.21.15.189:0/93010157 >> v2:172.21.15.192:3300/0 conn(0x7f94a815e500 0x7f94a806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:17.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.829+0000 7f94a7fff700 1 -- 172.21.15.189:0/93010157 >> v2:172.21.15.192:3300/0 conn(0x7f94a815e500 msgr2=0x7f94a806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:17.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.829+0000 7f94a7fff700 1 --2- 172.21.15.189:0/93010157 >> v2:172.21.15.192:3300/0 conn(0x7f94a815e500 0x7f94a806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.829+0000 7f94a7fff700 1 -- 172.21.15.189:0/93010157 >> v1:172.21.15.192:6789/0 conn(0x7f94a806a730 legacy=0x7f94a815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:17.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.829+0000 7f94a7fff700 1 -- 172.21.15.189:0/93010157 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a80b3060 con 0x7f94a815d7b0 2024-04-06T13:45:17.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.829+0000 7f94a7fff700 1 --2- 172.21.15.189:0/93010157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a815d7b0 0x7f94a80b3e20 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f94a80673e0 tx=0x7f9498009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b60ec3b42e77be8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:17.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.829+0000 7f94a6ffd700 1 -- 172.21.15.189:0/93010157 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949800b870 con 0x7f94a815d7b0 2024-04-06T13:45:17.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.829+0000 7f94a6ffd700 1 -- 172.21.15.189:0/93010157 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f949800b9d0 con 0x7f94a815d7b0 2024-04-06T13:45:17.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.829+0000 7f94a6ffd700 1 -- 172.21.15.189:0/93010157 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949800bd60 con 0x7f94a815d7b0 2024-04-06T13:45:17.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.830+0000 7f94ae57f700 1 -- 172.21.15.189:0/93010157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a815d7b0 msgr2=0x7f94a80b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:17.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.830+0000 7f94ae57f700 1 --2- 172.21.15.189:0/93010157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a815d7b0 0x7f94a80b3e20 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f94a80673e0 tx=0x7f9498009bd0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.830+0000 7f94ae57f700 1 -- 172.21.15.189:0/93010157 shutdown_connections 2024-04-06T13:45:17.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.830+0000 7f94ae57f700 1 --2- 172.21.15.189:0/93010157 >> v2:172.21.15.192:3300/0 conn(0x7f94a815e500 0x7f94a806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.830+0000 7f94ae57f700 1 --2- 172.21.15.189:0/93010157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a815d7b0 0x7f94a80b3e20 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.830+0000 7f94ae57f700 1 -- 172.21.15.189:0/93010157 >> 172.21.15.189:0/93010157 conn(0x7f94a80d00b0 msgr2=0x7f94a80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:17.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.830+0000 7f94ae57f700 1 -- 172.21.15.189:0/93010157 shutdown_connections 2024-04-06T13:45:17.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.830+0000 7f94ae57f700 1 -- 172.21.15.189:0/93010157 wait complete. 2024-04-06T13:45:17.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.831+0000 7f94ae57f700 1 Processor -- start 2024-04-06T13:45:17.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.831+0000 7f94ae57f700 1 -- start start 2024-04-06T13:45:17.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.832+0000 7f94ae57f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94a806a730 0x7f94a81f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:17.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.832+0000 7f94ae57f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a815d7b0 0x7f94a815c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:17.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.832+0000 7f94ae57f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94a815e500 0x7f94a81f6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:17.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.832+0000 7f94ae57f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a81f6a40 con 0x7f94a815d7b0 2024-04-06T13:45:17.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.832+0000 7f94ae57f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a81f6710 con 0x7f94a815e500 2024-04-06T13:45:17.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.832+0000 7f94ae57f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a81f39e0 con 0x7f94a806a730 2024-04-06T13:45:17.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.832+0000 7f94a77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a815d7b0 0x7f94a815c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:17.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.832+0000 7f94a77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a815d7b0 0x7f94a815c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45242/0 (socket says 172.21.15.189:45242) 2024-04-06T13:45:17.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.832+0000 7f94a77fe700 1 -- 172.21.15.189:0/2246321301 learned_addr learned my addr 172.21.15.189:0/2246321301 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:17.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.832+0000 7f94a7fff700 1 --2- 172.21.15.189:0/2246321301 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94a806a730 0x7f94a81f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:17.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.832+0000 7f94acb1c700 1 --2- 172.21.15.189:0/2246321301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94a815e500 0x7f94a81f6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:17.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.832+0000 7f94a77fe700 1 -- 172.21.15.189:0/2246321301 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94a806a730 msgr2=0x7f94a81f3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:17.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.832+0000 7f94a77fe700 1 --2- 172.21.15.189:0/2246321301 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94a806a730 0x7f94a81f3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.832+0000 7f94a77fe700 1 -- 172.21.15.189:0/2246321301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94a815e500 msgr2=0x7f94a81f6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:17.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.832+0000 7f94a77fe700 1 --2- 172.21.15.189:0/2246321301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94a815e500 0x7f94a81f6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:17.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.832+0000 7f94a77fe700 1 -- 172.21.15.189:0/2246321301 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9498009550 con 0x7f94a815d7b0 2024-04-06T13:45:17.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.832+0000 7f94a77fe700 1 --2- 172.21.15.189:0/2246321301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a815d7b0 0x7f94a815c320 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f949000c4e0 tx=0x7f949000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:17.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.833+0000 7f94a4ff9700 1 -- 172.21.15.189:0/2246321301 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94900052d0 con 0x7f94a815d7b0 2024-04-06T13:45:17.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.833+0000 7f94a4ff9700 1 -- 172.21.15.189:0/2246321301 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9490005430 con 0x7f94a815d7b0 2024-04-06T13:45:17.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.833+0000 7f94ae57f700 1 -- 172.21.15.189:0/2246321301 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a81f3cc0 con 0x7f94a815d7b0 2024-04-06T13:45:17.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.833+0000 7f94a4ff9700 1 -- 172.21.15.189:0/2246321301 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94900038c0 con 0x7f94a815d7b0 2024-04-06T13:45:17.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.833+0000 7f94ae57f700 1 -- 172.21.15.189:0/2246321301 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a81f3680 con 0x7f94a815d7b0 2024-04-06T13:45:17.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.834+0000 7f94ae57f700 1 -- 172.21.15.189:0/2246321301 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a8067bb0 con 0x7f94a815d7b0 2024-04-06T13:45:17.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.834+0000 7f94a4ff9700 1 -- 172.21.15.189:0/2246321301 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 122) v1 ==== 89276+0+0 (secure 0 0 0) 0x7f9490003a20 con 0x7f94a815d7b0 2024-04-06T13:45:17.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.834+0000 7f94a4ff9700 1 --2- 172.21.15.189:0/2246321301 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f948806ae00 0x7f948806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:17.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.834+0000 7f94a7fff700 1 --2- 172.21.15.189:0/2246321301 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f948806ae00 0x7f948806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:17.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.834+0000 7f94a4ff9700 1 -- 172.21.15.189:0/2246321301 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f9490088be0 con 0x7f94a815d7b0 2024-04-06T13:45:17.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.835+0000 7f94a7fff700 1 --2- 172.21.15.189:0/2246321301 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f948806ae00 0x7f948806d2c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9498000c00 tx=0x7f9498008030 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:17.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:17.839+0000 7f94a4ff9700 1 -- 172.21.15.189:0/2246321301 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9490053970 con 0x7f94a815d7b0 2024-04-06T13:45:18.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:18.057+0000 7f94ae57f700 1 -- 172.21.15.189:0/2246321301 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f94a815eb40 con 0x7f948806ae00 2024-04-06T13:45:18.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:18.107+0000 7f94a4ff9700 1 -- 172.21.15.189:0/2246321301 <== mgr.10398 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f949800bb40 con 0x7f948806ae00 2024-04-06T13:45:18.109 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule removed for path /snap_test_dir1/test_restart 2024-04-06T13:45:18.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:18.110+0000 7f94ae57f700 1 -- 172.21.15.189:0/2246321301 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f948806ae00 msgr2=0x7f948806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:18.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:18.110+0000 7f94ae57f700 1 --2- 172.21.15.189:0/2246321301 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f948806ae00 0x7f948806d2c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9498000c00 tx=0x7f9498008030 comp rx=0 tx=0).stop 2024-04-06T13:45:18.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:18.110+0000 7f94ae57f700 1 -- 172.21.15.189:0/2246321301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a815d7b0 msgr2=0x7f94a815c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:18.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:18.110+0000 7f94ae57f700 1 --2- 172.21.15.189:0/2246321301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a815d7b0 0x7f94a815c320 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f949000c4e0 tx=0x7f949000c7f0 comp rx=0 tx=0).stop 2024-04-06T13:45:18.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:18.110+0000 7f94ae57f700 1 -- 172.21.15.189:0/2246321301 shutdown_connections 2024-04-06T13:45:18.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:18.110+0000 7f94ae57f700 1 --2- 172.21.15.189:0/2246321301 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f948806ae00 0x7f948806d2c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:18.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:18.110+0000 7f94ae57f700 1 --2- 172.21.15.189:0/2246321301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94a815e500 0x7f94a81f6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:18.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:18.110+0000 7f94ae57f700 1 --2- 172.21.15.189:0/2246321301 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94a806a730 0x7f94a81f3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:18.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:18.110+0000 7f94ae57f700 1 --2- 172.21.15.189:0/2246321301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a815d7b0 0x7f94a815c320 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:18.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:18.110+0000 7f94ae57f700 1 -- 172.21.15.189:0/2246321301 >> 172.21.15.189:0/2246321301 conn(0x7f94a80d00b0 msgr2=0x7f94a80af410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:18.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:18.110+0000 7f94ae57f700 1 -- 172.21.15.189:0/2246321301 shutdown_connections 2024-04-06T13:45:18.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:18.110+0000 7f94ae57f700 1 -- 172.21.15.189:0/2246321301 wait complete. 2024-04-06T13:45:18.124 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2024-04-06T13:45:18.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:18.124 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-04-06T13:45:18.143 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_30_00_UTC 2024-04-06T13:45:18.143 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_31_00_UTC 2024-04-06T13:45:18.144 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_32_00_UTC 2024-04-06T13:45:18.144 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_33_00_UTC 2024-04-06T13:45:18.144 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_34_00_UTC 2024-04-06T13:45:18.144 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_35_00_UTC 2024-04-06T13:45:18.144 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_36_00_UTC 2024-04-06T13:45:18.144 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_37_00_UTC 2024-04-06T13:45:18.144 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_38_00_UTC 2024-04-06T13:45:18.144 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_39_00_UTC 2024-04-06T13:45:18.145 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_40_00_UTC 2024-04-06T13:45:18.145 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_41_00_UTC 2024-04-06T13:45:18.145 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_42_00_UTC 2024-04-06T13:45:18.145 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_43_00_UTC 2024-04-06T13:45:18.145 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_44_00_UTC 2024-04-06T13:45:18.145 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_45_00_UTC 2024-04-06T13:45:18.145 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_30_00_UTC 2024-04-06T13:45:18.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:18.146 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_30_00_UTC) 2024-04-06T13:45:18.206 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_31_00_UTC 2024-04-06T13:45:18.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:18.207 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_31_00_UTC) 2024-04-06T13:45:18.269 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_32_00_UTC 2024-04-06T13:45:18.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:18.269 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_32_00_UTC) 2024-04-06T13:45:18.330 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_33_00_UTC 2024-04-06T13:45:18.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:18.331 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_33_00_UTC) 2024-04-06T13:45:18.393 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_34_00_UTC 2024-04-06T13:45:18.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:18.393 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_34_00_UTC) 2024-04-06T13:45:18.455 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_35_00_UTC 2024-04-06T13:45:18.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:18.456 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_35_00_UTC) 2024-04-06T13:45:18.517 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_36_00_UTC 2024-04-06T13:45:18.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:18.518 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_36_00_UTC) 2024-04-06T13:45:18.580 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_37_00_UTC 2024-04-06T13:45:18.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:18.580 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_37_00_UTC) 2024-04-06T13:45:18.643 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_38_00_UTC 2024-04-06T13:45:18.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:18.644 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_38_00_UTC) 2024-04-06T13:45:18.704 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_39_00_UTC 2024-04-06T13:45:18.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:18.705 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_39_00_UTC) 2024-04-06T13:45:18.767 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_40_00_UTC 2024-04-06T13:45:18.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:18.767 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_40_00_UTC) 2024-04-06T13:45:18.829 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_41_00_UTC 2024-04-06T13:45:18.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:18.830 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_41_00_UTC) 2024-04-06T13:45:18.892 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_42_00_UTC 2024-04-06T13:45:18.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:18.892 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_42_00_UTC) 2024-04-06T13:45:18.955 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_43_00_UTC 2024-04-06T13:45:18.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:18.955 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_43_00_UTC) 2024-04-06T13:45:19.018 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_44_00_UTC 2024-04-06T13:45:19.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:19.018 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_44_00_UTC) 2024-04-06T13:45:19.080 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_45_00_UTC 2024-04-06T13:45:19.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:19.080 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-06-13_45_00_UTC) 2024-04-06T13:45:19.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:19.143 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2024-04-06T13:45:19.203 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-06T13:45:19.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.366+0000 7fa0e0853700 1 Processor -- start 2024-04-06T13:45:19.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.366+0000 7fa0e0853700 1 -- start start 2024-04-06T13:45:19.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.367+0000 7fa0e0853700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa0dc0b4660 0x7fa0dc0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:19.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.367+0000 7fa0e0853700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa0dc0b2fc0 0x7fa0dc0b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:19.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.367+0000 7fa0e0853700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0dc0b5030 con 0x7fa0dc15a550 2024-04-06T13:45:19.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.367+0000 7fa0e0853700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0dc0cdbe0 con 0x7fa0dc0b2fc0 2024-04-06T13:45:19.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.367+0000 7fa0e0853700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0dc15a940 con 0x7fa0dc0b4660 2024-04-06T13:45:19.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.367+0000 7fa0dad9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa0dc0b2fc0 0x7fa0dc0b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:19.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.367+0000 7fa0da59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa0dc15a550 0x7fa0dc0cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38398/0 (socket says 172.21.15.189:38398) 2024-04-06T13:45:19.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.367+0000 7fa0da59c700 1 -- 172.21.15.189:0/188220793 learned_addr learned my addr 172.21.15.189:0/188220793 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:19.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.368+0000 7fa0dad9d700 1 -- 172.21.15.189:0/188220793 >> v2:172.21.15.192:3300/0 conn(0x7fa0dc0b4660 msgr2=0x7fa0dc0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:19.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.368+0000 7fa0d9d9b700 1 --2- 172.21.15.189:0/188220793 >> v2:172.21.15.192:3300/0 conn(0x7fa0dc0b4660 0x7fa0dc0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:19.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.368+0000 7fa0dad9d700 1 --2- 172.21.15.189:0/188220793 >> v2:172.21.15.192:3300/0 conn(0x7fa0dc0b4660 0x7fa0dc0b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:19.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.368+0000 7fa0d959a700 1 -- 172.21.15.189:0/188220793 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049159422 0 0) 0x7fa0dc0b5030 con 0x7fa0dc15a550 2024-04-06T13:45:19.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.368+0000 7fa0dad9d700 1 -- 172.21.15.189:0/188220793 >> v1:172.21.15.189:6789/0 conn(0x7fa0dc15a550 legacy=0x7fa0dc0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:19.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.368+0000 7fa0dad9d700 1 -- 172.21.15.189:0/188220793 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0dc15ac00 con 0x7fa0dc0b2fc0 2024-04-06T13:45:19.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.368+0000 7fa0dad9d700 1 --2- 172.21.15.189:0/188220793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0dc0b2fc0 0x7fa0dc0b33d0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc0b2d40 tx=0x7fa0d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f5b1eb0911688d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:19.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.368+0000 7fa0d959a700 1 -- 172.21.15.189:0/188220793 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d400e070 con 0x7fa0dc0b2fc0 2024-04-06T13:45:19.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.368+0000 7fa0d959a700 1 -- 172.21.15.189:0/188220793 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d400bbd0 con 0x7fa0dc0b2fc0 2024-04-06T13:45:19.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.368+0000 7fa0d959a700 1 -- 172.21.15.189:0/188220793 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4012430 con 0x7fa0dc0b2fc0 2024-04-06T13:45:19.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.369+0000 7fa0e0853700 1 -- 172.21.15.189:0/188220793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0dc0b2fc0 msgr2=0x7fa0dc0b33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:19.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.369+0000 7fa0e0853700 1 --2- 172.21.15.189:0/188220793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0dc0b2fc0 0x7fa0dc0b33d0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc0b2d40 tx=0x7fa0d400b330 comp rx=0 tx=0).stop 2024-04-06T13:45:19.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.369+0000 7fa0e0853700 1 -- 172.21.15.189:0/188220793 shutdown_connections 2024-04-06T13:45:19.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.369+0000 7fa0e0853700 1 --2- 172.21.15.189:0/188220793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0dc0b2fc0 0x7fa0dc0b33d0 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:19.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.369+0000 7fa0e0853700 1 --2- 172.21.15.189:0/188220793 >> v2:172.21.15.192:3300/0 conn(0x7fa0dc0b4660 0x7fa0dc0b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:19.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.369+0000 7fa0e0853700 1 -- 172.21.15.189:0/188220793 >> 172.21.15.189:0/188220793 conn(0x7fa0dc0d00b0 msgr2=0x7fa0dc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:19.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.369+0000 7fa0e0853700 1 -- 172.21.15.189:0/188220793 shutdown_connections 2024-04-06T13:45:19.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.369+0000 7fa0e0853700 1 -- 172.21.15.189:0/188220793 wait complete. 2024-04-06T13:45:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.370+0000 7fa0e0853700 1 Processor -- start 2024-04-06T13:45:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.370+0000 7fa0e0853700 1 -- start start 2024-04-06T13:45:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0e0853700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0dc0b2fc0 0x7fa0dc1f0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0e0853700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0dc0b4660 0x7fa0dc151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0e0853700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0dc0a88e0 0x7fa0dc1f3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0e0853700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc1f4190 con 0x7fa0dc0a88e0 2024-04-06T13:45:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0e0853700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc1f01f0 con 0x7fa0dc0b4660 2024-04-06T13:45:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0e0853700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc1efc20 con 0x7fa0dc0b2fc0 2024-04-06T13:45:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0dad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0dc0a88e0 0x7fa0dc1f3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0dad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0dc0a88e0 0x7fa0dc1f3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50510/0 (socket says 172.21.15.189:50510) 2024-04-06T13:45:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0dad9d700 1 -- 172.21.15.189:0/770283553 learned_addr learned my addr 172.21.15.189:0/770283553 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0d9d9b700 1 --2- 172.21.15.189:0/770283553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0dc0b4660 0x7fa0dc151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0da59c700 1 --2- 172.21.15.189:0/770283553 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0dc0b2fc0 0x7fa0dc1f0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0dad9d700 1 -- 172.21.15.189:0/770283553 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0dc0b2fc0 msgr2=0x7fa0dc1f0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0dad9d700 1 --2- 172.21.15.189:0/770283553 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0dc0b2fc0 0x7fa0dc1f0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0dad9d700 1 -- 172.21.15.189:0/770283553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0dc0b4660 msgr2=0x7fa0dc151cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0dad9d700 1 --2- 172.21.15.189:0/770283553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0dc0b4660 0x7fa0dc151cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0dad9d700 1 -- 172.21.15.189:0/770283553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d4009a70 con 0x7fa0dc0a88e0 2024-04-06T13:45:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0dad9d700 1 --2- 172.21.15.189:0/770283553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0dc0a88e0 0x7fa0dc1f3260 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc06c820 tx=0x7fa0d4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0caffd700 1 -- 172.21.15.189:0/770283553 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4012430 con 0x7fa0dc0a88e0 2024-04-06T13:45:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0caffd700 1 -- 172.21.15.189:0/770283553 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d4020070 con 0x7fa0dc0a88e0 2024-04-06T13:45:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.371+0000 7fa0caffd700 1 -- 172.21.15.189:0/770283553 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4004280 con 0x7fa0dc0a88e0 2024-04-06T13:45:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.372+0000 7fa0e0853700 1 -- 172.21.15.189:0/770283553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0dc1f0bc0 con 0x7fa0dc0a88e0 2024-04-06T13:45:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.372+0000 7fa0e0853700 1 -- 172.21.15.189:0/770283553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0dc1f3bd0 con 0x7fa0dc0a88e0 2024-04-06T13:45:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.372+0000 7fa0e0853700 1 -- 172.21.15.189:0/770283553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0dc067bb0 con 0x7fa0dc0a88e0 2024-04-06T13:45:19.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.373+0000 7fa0caffd700 1 -- 172.21.15.189:0/770283553 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 122) v1 ==== 89276+0+0 (secure 0 0 0) 0x7fa0d40043e0 con 0x7fa0dc0a88e0 2024-04-06T13:45:19.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.374+0000 7fa0caffd700 1 --2- 172.21.15.189:0/770283553 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa0bc06ae00 0x7fa0bc06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:19.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.374+0000 7fa0caffd700 1 -- 172.21.15.189:0/770283553 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fa0d4022070 con 0x7fa0dc0a88e0 2024-04-06T13:45:19.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.374+0000 7fa0da59c700 1 --2- 172.21.15.189:0/770283553 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa0bc06ae00 0x7fa0bc06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:19.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.374+0000 7fa0da59c700 1 --2- 172.21.15.189:0/770283553 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa0bc06ae00 0x7fa0bc06d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0007fd0 tx=0x7fa0d0007de0 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:19.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.378+0000 7fa0caffd700 1 -- 172.21.15.189:0/770283553 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa0d4056a10 con 0x7fa0dc0a88e0 2024-04-06T13:45:19.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:19.632+0000 7fa0e0853700 1 -- 172.21.15.189:0/770283553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fa0dc074990 con 0x7fa0dc0a88e0 2024-04-06T13:45:20.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.580+0000 7fa0caffd700 1 -- 172.21.15.189:0/770283553 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 123) v1 ==== 89259+0+0 (secure 0 0 0) 0x7fa0d4056de0 con 0x7fa0dc0a88e0 2024-04-06T13:45:20.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.581+0000 7fa0caffd700 1 -- 172.21.15.189:0/770283553 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v123) v1 ==== 93+0+0 (secure 0 0 0) 0x7fa0d405a4a0 con 0x7fa0dc0a88e0 2024-04-06T13:45:20.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.584+0000 7fa0e0853700 1 -- 172.21.15.189:0/770283553 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa0bc06ae00 msgr2=0x7fa0bc06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:20.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.584+0000 7fa0e0853700 1 --2- 172.21.15.189:0/770283553 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa0bc06ae00 0x7fa0bc06d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0007fd0 tx=0x7fa0d0007de0 comp rx=0 tx=0).stop 2024-04-06T13:45:20.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.584+0000 7fa0e0853700 1 -- 172.21.15.189:0/770283553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0dc0a88e0 msgr2=0x7fa0dc1f3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:20.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.584+0000 7fa0e0853700 1 --2- 172.21.15.189:0/770283553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0dc0a88e0 0x7fa0dc1f3260 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc06c820 tx=0x7fa0d4004ad0 comp rx=0 tx=0).stop 2024-04-06T13:45:20.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.584+0000 7fa0e0853700 1 -- 172.21.15.189:0/770283553 shutdown_connections 2024-04-06T13:45:20.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.584+0000 7fa0e0853700 1 --2- 172.21.15.189:0/770283553 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa0bc06ae00 0x7fa0bc06d2c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:20.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.584+0000 7fa0e0853700 1 --2- 172.21.15.189:0/770283553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0dc0a88e0 0x7fa0dc1f3260 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:20.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.584+0000 7fa0e0853700 1 --2- 172.21.15.189:0/770283553 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0dc0b2fc0 0x7fa0dc1f0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:20.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.584+0000 7fa0e0853700 1 --2- 172.21.15.189:0/770283553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0dc0b4660 0x7fa0dc151cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:20.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.584+0000 7fa0e0853700 1 -- 172.21.15.189:0/770283553 >> 172.21.15.189:0/770283553 conn(0x7fa0dc0d00b0 msgr2=0x7fa0dc157cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:20.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.585+0000 7fa0e0853700 1 -- 172.21.15.189:0/770283553 shutdown_connections 2024-04-06T13:45:20.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.585+0000 7fa0e0853700 1 -- 172.21.15.189:0/770283553 wait complete. 2024-04-06T13:45:20.600 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T13:45:20.600 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:45:20.643 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:45:20.688 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T13:45:20.689 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:45:20.689 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:45:20.706 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:45:20.707 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:45:20.765 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:45:20.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.950+0000 7f6bfd17d700 1 Processor -- start 2024-04-06T13:45:20.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.950+0000 7f6bfd17d700 1 -- start start 2024-04-06T13:45:20.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.951+0000 7f6bfd17d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6bf815a550 0x7f6bf80cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:20.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.951+0000 7f6bfd17d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf80cf900 0x7f6bf80b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:20.962 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.951+0000 7f6bfd17d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf80cdbe0 con 0x7f6bf80b4ba0 2024-04-06T13:45:20.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.951+0000 7f6bfd17d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bf80b5010 con 0x7f6bf815a550 2024-04-06T13:45:20.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.951+0000 7f6bfd17d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf815a940 con 0x7f6bf80cf900 2024-04-06T13:45:20.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.951+0000 7f6bf6d9d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6bf815a550 0x7f6bf80cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:20.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.951+0000 7f6bf659c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf80cf900 0x7f6bf80b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:20.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.951+0000 7f6bf759e700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f6bf80b4ba0 0x7f6bf806eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:56702/0 (socket says 172.21.15.189:56702) 2024-04-06T13:45:20.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.951+0000 7f6bf759e700 1 -- 172.21.15.189:0/2662569339 learned_addr learned my addr 172.21.15.189:0/2662569339 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:20.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.952+0000 7f6bf6d9d700 1 -- 172.21.15.189:0/2662569339 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf80cf900 msgr2=0x7f6bf80b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:20.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.952+0000 7f6bf6d9d700 1 --2- 172.21.15.189:0/2662569339 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf80cf900 0x7f6bf80b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:20.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.952+0000 7f6bf6d9d700 1 -- 172.21.15.189:0/2662569339 >> v1:172.21.15.192:6789/0 conn(0x7f6bf80b4ba0 legacy=0x7f6bf806eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:20.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.952+0000 7f6bf6d9d700 1 -- 172.21.15.189:0/2662569339 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf815ac00 con 0x7f6bf815a550 2024-04-06T13:45:20.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.952+0000 7f6bf5d9b700 1 -- 172.21.15.189:0/2662569339 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2234455982 0 0) 0x7f6bf80cdbe0 con 0x7f6bf80b4ba0 2024-04-06T13:45:20.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.952+0000 7f6bf659c700 1 --2- 172.21.15.189:0/2662569339 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf80cf900 0x7f6bf80b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:45:20.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.952+0000 7f6bf6d9d700 1 --2- 172.21.15.189:0/2662569339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf815a550 0x7f6bf80cf3c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009840 tx=0x7f6be8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af8efbfa6a48c72e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:20.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.952+0000 7f6bf5d9b700 1 -- 172.21.15.189:0/2662569339 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8011830 con 0x7f6bf815a550 2024-04-06T13:45:20.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.952+0000 7f6bf5d9b700 1 -- 172.21.15.189:0/2662569339 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be8011990 con 0x7f6bf815a550 2024-04-06T13:45:20.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.952+0000 7f6bf5d9b700 1 -- 172.21.15.189:0/2662569339 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8011d20 con 0x7f6bf815a550 2024-04-06T13:45:20.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.953+0000 7f6bfd17d700 1 -- 172.21.15.189:0/2662569339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf815a550 msgr2=0x7f6bf80cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:20.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.953+0000 7f6bfd17d700 1 --2- 172.21.15.189:0/2662569339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf815a550 0x7f6bf80cf3c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009840 tx=0x7f6be8009b50 comp rx=0 tx=0).stop 2024-04-06T13:45:20.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.953+0000 7f6bfd17d700 1 -- 172.21.15.189:0/2662569339 shutdown_connections 2024-04-06T13:45:20.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.953+0000 7f6bfd17d700 1 --2- 172.21.15.189:0/2662569339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf815a550 0x7f6bf80cf3c0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:20.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.953+0000 7f6bfd17d700 1 --2- 172.21.15.189:0/2662569339 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf80cf900 0x7f6bf80b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:20.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.953+0000 7f6bfd17d700 1 -- 172.21.15.189:0/2662569339 >> 172.21.15.189:0/2662569339 conn(0x7f6bf80d00b0 msgr2=0x7f6bf80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:20.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.953+0000 7f6bfd17d700 1 -- 172.21.15.189:0/2662569339 shutdown_connections 2024-04-06T13:45:20.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.953+0000 7f6bfd17d700 1 -- 172.21.15.189:0/2662569339 wait complete. 2024-04-06T13:45:20.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.954+0000 7f6bfd17d700 1 Processor -- start 2024-04-06T13:45:20.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.955+0000 7f6bfd17d700 1 -- start start 2024-04-06T13:45:20.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.955+0000 7f6bfd17d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bf80b4ba0 0x7f6bf8141130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:20.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.955+0000 7f6bfd17d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf80cf900 0x7f6bf814dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:20.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.955+0000 7f6bfd17d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf81477d0 0x7f6bf8147c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:20.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.955+0000 7f6bfd17d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf814d7e0 con 0x7f6bf80b4ba0 2024-04-06T13:45:20.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.955+0000 7f6bfd17d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf8148290 con 0x7f6bf80cf900 2024-04-06T13:45:20.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.955+0000 7f6bfd17d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf814ceb0 con 0x7f6bf81477d0 2024-04-06T13:45:20.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.955+0000 7f6bf6d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bf80b4ba0 0x7f6bf8141130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:20.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.956+0000 7f6bf6d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bf80b4ba0 0x7f6bf8141130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50528/0 (socket says 172.21.15.189:50528) 2024-04-06T13:45:20.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.956+0000 7f6bf6d9d700 1 -- 172.21.15.189:0/4284005858 learned_addr learned my addr 172.21.15.189:0/4284005858 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:20.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.956+0000 7f6bf659c700 1 --2- 172.21.15.189:0/4284005858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf80cf900 0x7f6bf814dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:20.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.956+0000 7f6bf759e700 1 --2- 172.21.15.189:0/4284005858 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf81477d0 0x7f6bf8147c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:20.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.956+0000 7f6bf6d9d700 1 -- 172.21.15.189:0/4284005858 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf81477d0 msgr2=0x7f6bf8147c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:20.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.956+0000 7f6bf6d9d700 1 --2- 172.21.15.189:0/4284005858 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf81477d0 0x7f6bf8147c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:20.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.956+0000 7f6bf6d9d700 1 -- 172.21.15.189:0/4284005858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf80cf900 msgr2=0x7f6bf814dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:20.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.956+0000 7f6bf6d9d700 1 --2- 172.21.15.189:0/4284005858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf80cf900 0x7f6bf814dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:20.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.956+0000 7f6bf6d9d700 1 -- 172.21.15.189:0/4284005858 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bec0094a0 con 0x7f6bf80b4ba0 2024-04-06T13:45:20.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.956+0000 7f6bf6d9d700 1 --2- 172.21.15.189:0/4284005858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bf80b4ba0 0x7f6bf8141130 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009b90 tx=0x7f6be8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:20.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.956+0000 7f6bdf7fe700 1 -- 172.21.15.189:0/4284005858 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be80036a0 con 0x7f6bf80b4ba0 2024-04-06T13:45:20.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.956+0000 7f6bdf7fe700 1 -- 172.21.15.189:0/4284005858 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be801e070 con 0x7f6bf80b4ba0 2024-04-06T13:45:20.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.956+0000 7f6bfd17d700 1 -- 172.21.15.189:0/4284005858 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be80094a0 con 0x7f6bf80b4ba0 2024-04-06T13:45:20.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.956+0000 7f6bdf7fe700 1 -- 172.21.15.189:0/4284005858 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8003a20 con 0x7f6bf80b4ba0 2024-04-06T13:45:20.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.956+0000 7f6bfd17d700 1 -- 172.21.15.189:0/4284005858 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf814d470 con 0x7f6bf80b4ba0 2024-04-06T13:45:20.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.957+0000 7f6bfd17d700 1 -- 172.21.15.189:0/4284005858 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf814dac0 con 0x7f6bf80b4ba0 2024-04-06T13:45:20.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.958+0000 7f6bdf7fe700 1 -- 172.21.15.189:0/4284005858 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 123) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f6be8003b80 con 0x7f6bf80b4ba0 2024-04-06T13:45:20.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.958+0000 7f6bdf7fe700 1 --2- 172.21.15.189:0/4284005858 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6be406adb0 0x7f6be406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:20.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.958+0000 7f6bf659c700 1 -- 172.21.15.189:0/4284005858 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6be406adb0 msgr2=0x7f6be406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:20.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.958+0000 7f6bdf7fe700 1 -- 172.21.15.189:0/4284005858 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f6be808a6f0 con 0x7f6bf80b4ba0 2024-04-06T13:45:20.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.958+0000 7f6bf659c700 1 --2- 172.21.15.189:0/4284005858 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6be406adb0 0x7f6be406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:45:20.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:20.962+0000 7f6bdf7fe700 1 -- 172.21.15.189:0/4284005858 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6be8055480 con 0x7f6bf80b4ba0 2024-04-06T13:45:20.973 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:20.968+0000 7fbe06afc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:45:21.008 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:21.008+0000 7f3f48896200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:45:21.046 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:21.046+0000 7fbe06afc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:45:21.090 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:21.089+0000 7f3f48896200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:45:21.119 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:21.119+0000 7fbe06afc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:45:21.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.159+0000 7f6bf659c700 1 -- 172.21.15.189:0/4284005858 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6be406adb0 msgr2=0x7f6be406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:21.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.159+0000 7f6bf659c700 1 --2- 172.21.15.189:0/4284005858 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6be406adb0 0x7f6be406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:45:21.165 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:21.164+0000 7f3f48896200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:45:21.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.169+0000 7f6bfd17d700 1 -- 172.21.15.189:0/4284005858 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bf8067bb0 con 0x7f6bf80b4ba0 2024-04-06T13:45:21.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.170+0000 7f6bdf7fe700 1 -- 172.21.15.189:0/4284005858 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v194) v1 ==== 74+0+17242 (secure 0 0 0) 0x7f6be8058aa0 con 0x7f6bf80b4ba0 2024-04-06T13:45:21.172 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:21.172 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":194,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:40:22.300437+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:29:14.625778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-04-06T13:29:15.827532+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/1090714269":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/3656453012":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/2710277231":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/1953253738":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/1974012472":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/820436148":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/2599831431":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/1401962292":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/2983879481":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/3870180304":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/1073864749":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/3525693740":"2024-04-07T13:29:30.406338+0000","172.21.15.189:6833/47128":"2024-04-07T13:29:30.406338+0000","172.21.15.189:6832/47128":"2024-04-07T13:29:30.406338+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:45:21.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.173+0000 7f6bfd17d700 1 -- 172.21.15.189:0/4284005858 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6be406adb0 msgr2=0x7f6be406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:45:21.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.173+0000 7f6bfd17d700 1 --2- 172.21.15.189:0/4284005858 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6be406adb0 0x7f6be406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.173+0000 7f6bfd17d700 1 -- 172.21.15.189:0/4284005858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bf80b4ba0 msgr2=0x7f6bf8141130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:21.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.173+0000 7f6bfd17d700 1 --2- 172.21.15.189:0/4284005858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bf80b4ba0 0x7f6bf8141130 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009b90 tx=0x7f6be8004750 comp rx=0 tx=0).stop 2024-04-06T13:45:21.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.173+0000 7f6bfd17d700 1 -- 172.21.15.189:0/4284005858 shutdown_connections 2024-04-06T13:45:21.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.173+0000 7f6bfd17d700 1 --2- 172.21.15.189:0/4284005858 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6be406adb0 0x7f6be406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.173+0000 7f6bfd17d700 1 --2- 172.21.15.189:0/4284005858 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf81477d0 0x7f6bf8147c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.173+0000 7f6bfd17d700 1 --2- 172.21.15.189:0/4284005858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bf80b4ba0 0x7f6bf8141130 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.173+0000 7f6bfd17d700 1 --2- 172.21.15.189:0/4284005858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf80cf900 0x7f6bf814dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.173+0000 7f6bfd17d700 1 -- 172.21.15.189:0/4284005858 >> 172.21.15.189:0/4284005858 conn(0x7f6bf80d00b0 msgr2=0x7f6bf806f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:21.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.173+0000 7f6bfd17d700 1 -- 172.21.15.189:0/4284005858 shutdown_connections 2024-04-06T13:45:21.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.173+0000 7f6bfd17d700 1 -- 172.21.15.189:0/4284005858 wait complete. 2024-04-06T13:45:21.185 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:21.186+0000 7fbe06afc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:45:21.187 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T13:29:14.625778+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '191', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T13:29:15.827532+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '192', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:45:21.188 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:45:21.238 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:21.238+0000 7f3f48896200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:45:21.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.315+0000 7f601213b700 1 Processor -- start 2024-04-06T13:45:21.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.315+0000 7f601213b700 1 -- start start 2024-04-06T13:45:21.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.316+0000 7f601213b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f600c0d11b0 0x7f600c0d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:21.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.316+0000 7f601213b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c0d1ad0 con 0x7f600c0cfa00 2024-04-06T13:45:21.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.316+0000 7f601213b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c0cbd80 con 0x7f600c0cb870 2024-04-06T13:45:21.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.316+0000 7f601213b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f600c0ccdc0 con 0x7f600c0d11b0 2024-04-06T13:45:21.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.316+0000 7f600b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f600c0d11b0 0x7f600c0d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:21.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.316+0000 7f600b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f600c0d11b0 0x7f600c0d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50530/0 (socket says 172.21.15.189:50530) 2024-04-06T13:45:21.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.316+0000 7f600bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f600c0cfa00 0x7f600c0cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38420/0 (socket says 172.21.15.189:38420) 2024-04-06T13:45:21.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.316+0000 7f600b7fe700 1 -- 172.21.15.189:0/1306809430 learned_addr learned my addr 172.21.15.189:0/1306809430 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:21.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.317+0000 7f600b7fe700 1 -- 172.21.15.189:0/1306809430 >> v1:172.21.15.192:6789/0 conn(0x7f600c0cb870 legacy=0x7f600c15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:21.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.317+0000 7f600b7fe700 1 -- 172.21.15.189:0/1306809430 >> v1:172.21.15.189:6789/0 conn(0x7f600c0cfa00 legacy=0x7f600c0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:21.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.317+0000 7f600b7fe700 1 -- 172.21.15.189:0/1306809430 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c0cef20 con 0x7f600c0d11b0 2024-04-06T13:45:21.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.317+0000 7f600b7fe700 1 --2- 172.21.15.189:0/1306809430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f600c0d11b0 0x7f600c0d1590 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009e50 tx=0x7f5ffc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c100dfeebcdb1ce7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:21.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.317+0000 7f600a7fc700 1 -- 172.21.15.189:0/1306809430 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc00e070 con 0x7f600c0d11b0 2024-04-06T13:45:21.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.317+0000 7f600a7fc700 1 -- 172.21.15.189:0/1306809430 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ffc00b9c0 con 0x7f600c0d11b0 2024-04-06T13:45:21.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.317+0000 7f600a7fc700 1 -- 172.21.15.189:0/1306809430 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc00bd70 con 0x7f600c0d11b0 2024-04-06T13:45:21.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.318+0000 7f601213b700 1 -- 172.21.15.189:0/1306809430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f600c0d11b0 msgr2=0x7f600c0d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:21.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.318+0000 7f601213b700 1 --2- 172.21.15.189:0/1306809430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f600c0d11b0 0x7f600c0d1590 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009e50 tx=0x7f5ffc00b330 comp rx=0 tx=0).stop 2024-04-06T13:45:21.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.318+0000 7f601213b700 1 -- 172.21.15.189:0/1306809430 shutdown_connections 2024-04-06T13:45:21.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.318+0000 7f601213b700 1 --2- 172.21.15.189:0/1306809430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f600c0d11b0 0x7f600c0d1590 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.318+0000 7f601213b700 1 -- 172.21.15.189:0/1306809430 >> 172.21.15.189:0/1306809430 conn(0x7f600c0b2db0 msgr2=0x7f600c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:21.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.318+0000 7f601213b700 1 -- 172.21.15.189:0/1306809430 shutdown_connections 2024-04-06T13:45:21.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.318+0000 7f601213b700 1 -- 172.21.15.189:0/1306809430 wait complete. 2024-04-06T13:45:21.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.319+0000 7f601213b700 1 Processor -- start 2024-04-06T13:45:21.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.319+0000 7f601213b700 1 -- start start 2024-04-06T13:45:21.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.320+0000 7f601213b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f600c0cb870 0x7f600c155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:21.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.320+0000 7f601213b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f600c0cfa00 0x7f600c0ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:21.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.320+0000 7f601213b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f600c0d11b0 0x7f600c0c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:21.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.320+0000 7f601213b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c1f0e40 con 0x7f600c0cb870 2024-04-06T13:45:21.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.320+0000 7f601213b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c1f10c0 con 0x7f600c0d11b0 2024-04-06T13:45:21.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.320+0000 7f601213b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600c0ca830 con 0x7f600c0cfa00 2024-04-06T13:45:21.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.320+0000 7f600b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f600c0cb870 0x7f600c155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:21.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.320+0000 7f600b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f600c0cb870 0x7f600c155890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50546/0 (socket says 172.21.15.189:50546) 2024-04-06T13:45:21.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.320+0000 7f600b7fe700 1 -- 172.21.15.189:0/2439201596 learned_addr learned my addr 172.21.15.189:0/2439201596 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:21.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.320+0000 7f600bfff700 1 --2- 172.21.15.189:0/2439201596 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f600c0d11b0 0x7f600c0c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:21.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.320+0000 7f600affd700 1 --2- 172.21.15.189:0/2439201596 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f600c0cfa00 0x7f600c0ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:21.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.320+0000 7f600b7fe700 1 -- 172.21.15.189:0/2439201596 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f600c0cfa00 msgr2=0x7f600c0ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:21.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.320+0000 7f600b7fe700 1 --2- 172.21.15.189:0/2439201596 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f600c0cfa00 0x7f600c0ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.320+0000 7f600b7fe700 1 -- 172.21.15.189:0/2439201596 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f600c0d11b0 msgr2=0x7f600c0c76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:21.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.320+0000 7f600b7fe700 1 --2- 172.21.15.189:0/2439201596 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f600c0d11b0 0x7f600c0c76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.320+0000 7f600b7fe700 1 -- 172.21.15.189:0/2439201596 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ffc009b30 con 0x7f600c0cb870 2024-04-06T13:45:21.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.321+0000 7f600b7fe700 1 --2- 172.21.15.189:0/2439201596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f600c0cb870 0x7f600c155890 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f600c0677e0 tx=0x7f5ffc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:21.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.321+0000 7f6003fff700 1 -- 172.21.15.189:0/2439201596 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc0048c0 con 0x7f600c0cb870 2024-04-06T13:45:21.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.321+0000 7f6003fff700 1 -- 172.21.15.189:0/2439201596 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ffc0099b0 con 0x7f600c0cb870 2024-04-06T13:45:21.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.321+0000 7f601213b700 1 -- 172.21.15.189:0/2439201596 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600c0caae0 con 0x7f600c0cb870 2024-04-06T13:45:21.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.321+0000 7f6003fff700 1 -- 172.21.15.189:0/2439201596 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc01c680 con 0x7f600c0cb870 2024-04-06T13:45:21.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.321+0000 7f601213b700 1 -- 172.21.15.189:0/2439201596 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f600c0cb0a0 con 0x7f600c0cb870 2024-04-06T13:45:21.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.322+0000 7f601213b700 1 -- 172.21.15.189:0/2439201596 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f600c074990 con 0x7f600c0cb870 2024-04-06T13:45:21.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.322+0000 7f6003fff700 1 -- 172.21.15.189:0/2439201596 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 123) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f5ffc01c7e0 con 0x7f600c0cb870 2024-04-06T13:45:21.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.323+0000 7f6003fff700 1 --2- 172.21.15.189:0/2439201596 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5fec06adb0 0x7f5fec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:21.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.323+0000 7f600affd700 1 -- 172.21.15.189:0/2439201596 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5fec06adb0 msgr2=0x7f5fec06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:21.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.323+0000 7f6003fff700 1 -- 172.21.15.189:0/2439201596 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f5ffc08b330 con 0x7f600c0cb870 2024-04-06T13:45:21.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.323+0000 7f600affd700 1 --2- 172.21.15.189:0/2439201596 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5fec06adb0 0x7f5fec06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:45:21.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.327+0000 7f6003fff700 1 -- 172.21.15.189:0/2439201596 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5ffc055fb0 con 0x7f600c0cb870 2024-04-06T13:45:21.361 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:21.362+0000 7fbe06afc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:45:21.423 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:21.422+0000 7f3f48896200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:45:21.518 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:21.518+0000 7fbe06afc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:45:21.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.524+0000 7f600affd700 1 -- 172.21.15.189:0/2439201596 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5fec06adb0 msgr2=0x7f5fec06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:21.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.524+0000 7f600affd700 1 --2- 172.21.15.189:0/2439201596 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5fec06adb0 0x7f5fec06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:45:21.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.529+0000 7f601213b700 1 -- 172.21.15.189:0/2439201596 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f600c0b2530 con 0x7f600c0cb870 2024-04-06T13:45:21.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.529+0000 7f6003fff700 1 -- 172.21.15.189:0/2439201596 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v194) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5ffc0595d0 con 0x7f600c0cb870 2024-04-06T13:45:21.530 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:45:21.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.531+0000 7f601213b700 1 -- 172.21.15.189:0/2439201596 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5fec06adb0 msgr2=0x7f5fec06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:45:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.531+0000 7f601213b700 1 --2- 172.21.15.189:0/2439201596 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5fec06adb0 0x7f5fec06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.531+0000 7f601213b700 1 -- 172.21.15.189:0/2439201596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f600c0cb870 msgr2=0x7f600c155890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.531+0000 7f601213b700 1 --2- 172.21.15.189:0/2439201596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f600c0cb870 0x7f600c155890 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f600c0677e0 tx=0x7f5ffc004ad0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.531+0000 7f601213b700 1 -- 172.21.15.189:0/2439201596 shutdown_connections 2024-04-06T13:45:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.531+0000 7f601213b700 1 --2- 172.21.15.189:0/2439201596 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5fec06adb0 0x7f5fec06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.531+0000 7f601213b700 1 --2- 172.21.15.189:0/2439201596 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f600c0d11b0 0x7f600c0c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.531+0000 7f601213b700 1 --2- 172.21.15.189:0/2439201596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f600c0cb870 0x7f600c155890 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.531+0000 7f601213b700 1 --2- 172.21.15.189:0/2439201596 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f600c0cfa00 0x7f600c0ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.531+0000 7f601213b700 1 -- 172.21.15.189:0/2439201596 >> 172.21.15.189:0/2439201596 conn(0x7f600c0b2db0 msgr2=0x7f600c0b3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.531+0000 7f601213b700 1 -- 172.21.15.189:0/2439201596 shutdown_connections 2024-04-06T13:45:21.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.532+0000 7f601213b700 1 -- 172.21.15.189:0/2439201596 wait complete. 2024-04-06T13:45:21.546 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T13:45:21.593 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:21.592+0000 7f3f48896200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:45:21.675 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:21.673+0000 7fbe06afc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:45:21.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.723+0000 7f4b9728c700 1 Processor -- start 2024-04-06T13:45:21.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.724+0000 7f4b9728c700 1 -- start start 2024-04-06T13:45:21.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.725+0000 7f4b9728c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b9015b1b0 0x7f4b9015f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:21.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.725+0000 7f4b9728c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4b900b3fa0 0x7f4b900b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:21.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.725+0000 7f4b9728c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b900d0590 con 0x7f4b900cb7a0 2024-04-06T13:45:21.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.725+0000 7f4b9728c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b900cd700 con 0x7f4b900b3fa0 2024-04-06T13:45:21.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.725+0000 7f4b9728c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b900cd9e0 con 0x7f4b9015b1b0 2024-04-06T13:45:21.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.726+0000 7f4b95028700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b9015b1b0 0x7f4b9015f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:21.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.726+0000 7f4b95829700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f4b900cb7a0 0x7f4b900ced20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:56726/0 (socket says 172.21.15.189:56726) 2024-04-06T13:45:21.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.726+0000 7f4b95829700 1 -- 172.21.15.189:0/1165314624 learned_addr learned my addr 172.21.15.189:0/1165314624 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:21.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.726+0000 7f4b94827700 1 --2- 172.21.15.189:0/1165314624 >> v2:172.21.15.189:3300/0 conn(0x7f4b900b3fa0 0x7f4b900b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:21.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.726+0000 7f4b94827700 1 -- 172.21.15.189:0/1165314624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b9015b1b0 msgr2=0x7f4b9015f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:21.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.727+0000 7f4b94827700 1 --2- 172.21.15.189:0/1165314624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b9015b1b0 0x7f4b9015f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.727+0000 7f4b94827700 1 -- 172.21.15.189:0/1165314624 >> v1:172.21.15.192:6789/0 conn(0x7f4b900cb7a0 legacy=0x7f4b900ced20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:21.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.727+0000 7f4b94827700 1 -- 172.21.15.189:0/1165314624 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b900d0940 con 0x7f4b900b3fa0 2024-04-06T13:45:21.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.727+0000 7f4b94827700 1 --2- 172.21.15.189:0/1165314624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b900b3fa0 0x7f4b900b2bf0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f4b80009840 tx=0x7f4b80009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5f6b77c594d0b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:21.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.728+0000 7f4b87fff700 1 -- 172.21.15.189:0/1165314624 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b80004030 con 0x7f4b900b3fa0 2024-04-06T13:45:21.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.728+0000 7f4b87fff700 1 -- 172.21.15.189:0/1165314624 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b80004190 con 0x7f4b900b3fa0 2024-04-06T13:45:21.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.728+0000 7f4b87fff700 1 -- 172.21.15.189:0/1165314624 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b80004520 con 0x7f4b900b3fa0 2024-04-06T13:45:21.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.729+0000 7f4b9728c700 1 -- 172.21.15.189:0/1165314624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b900b3fa0 msgr2=0x7f4b900b2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:21.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.729+0000 7f4b9728c700 1 --2- 172.21.15.189:0/1165314624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b900b3fa0 0x7f4b900b2bf0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f4b80009840 tx=0x7f4b80009b50 comp rx=0 tx=0).stop 2024-04-06T13:45:21.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.729+0000 7f4b9728c700 1 -- 172.21.15.189:0/1165314624 shutdown_connections 2024-04-06T13:45:21.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.729+0000 7f4b9728c700 1 --2- 172.21.15.189:0/1165314624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b9015b1b0 0x7f4b9015f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.729+0000 7f4b9728c700 1 --2- 172.21.15.189:0/1165314624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b900b3fa0 0x7f4b900b2bf0 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.729+0000 7f4b9728c700 1 -- 172.21.15.189:0/1165314624 >> 172.21.15.189:0/1165314624 conn(0x7f4b90070300 msgr2=0x7f4b900706e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:21.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.729+0000 7f4b9728c700 1 -- 172.21.15.189:0/1165314624 shutdown_connections 2024-04-06T13:45:21.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.729+0000 7f4b9728c700 1 -- 172.21.15.189:0/1165314624 wait complete. 2024-04-06T13:45:21.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.731+0000 7f4b9728c700 1 Processor -- start 2024-04-06T13:45:21.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.731+0000 7f4b9728c700 1 -- start start 2024-04-06T13:45:21.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.732+0000 7f4b9728c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b900b3fa0 0x7f4b9014b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:21.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.732+0000 7f4b9728c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b900cb7a0 0x7f4b900cab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:21.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.732+0000 7f4b9728c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b9015b1b0 0x7f4b900c7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:21.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.732+0000 7f4b9728c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b90157e90 con 0x7f4b900cb7a0 2024-04-06T13:45:21.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.733+0000 7f4b9728c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b900c6dc0 con 0x7f4b900b3fa0 2024-04-06T13:45:21.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.733+0000 7f4b9728c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b900c61b0 con 0x7f4b9015b1b0 2024-04-06T13:45:21.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.733+0000 7f4b94827700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b900cb7a0 0x7f4b900cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:21.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.733+0000 7f4b94827700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b900cb7a0 0x7f4b900cab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50564/0 (socket says 172.21.15.189:50564) 2024-04-06T13:45:21.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.733+0000 7f4b94827700 1 -- 172.21.15.189:0/3038460345 learned_addr learned my addr 172.21.15.189:0/3038460345 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:21.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.733+0000 7f4b95028700 1 --2- 172.21.15.189:0/3038460345 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b900b3fa0 0x7f4b9014b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:21.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.733+0000 7f4b95829700 1 --2- 172.21.15.189:0/3038460345 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b9015b1b0 0x7f4b900c7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:21.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.733+0000 7f4b95028700 1 -- 172.21.15.189:0/3038460345 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b9015b1b0 msgr2=0x7f4b900c7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:21.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.733+0000 7f4b95028700 1 --2- 172.21.15.189:0/3038460345 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b9015b1b0 0x7f4b900c7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.733+0000 7f4b95028700 1 -- 172.21.15.189:0/3038460345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b900cb7a0 msgr2=0x7f4b900cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:21.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.733+0000 7f4b95028700 1 --2- 172.21.15.189:0/3038460345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b900cb7a0 0x7f4b900cab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.733+0000 7f4b95028700 1 -- 172.21.15.189:0/3038460345 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b800094a0 con 0x7f4b900b3fa0 2024-04-06T13:45:21.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.734+0000 7f4b95028700 1 --2- 172.21.15.189:0/3038460345 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b900b3fa0 0x7f4b9014b500 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c014cf0 tx=0x7f4b8c00fe60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:21.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.734+0000 7f4b85ffb700 1 -- 172.21.15.189:0/3038460345 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c00b070 con 0x7f4b900b3fa0 2024-04-06T13:45:21.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.734+0000 7f4b85ffb700 1 -- 172.21.15.189:0/3038460345 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8c012960 con 0x7f4b900b3fa0 2024-04-06T13:45:21.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.734+0000 7f4b85ffb700 1 -- 172.21.15.189:0/3038460345 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c012cf0 con 0x7f4b900b3fa0 2024-04-06T13:45:21.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.734+0000 7f4b9728c700 1 -- 172.21.15.189:0/3038460345 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b900c6490 con 0x7f4b900b3fa0 2024-04-06T13:45:21.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.735+0000 7f4b9728c700 1 -- 172.21.15.189:0/3038460345 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b900c6a50 con 0x7f4b900b3fa0 2024-04-06T13:45:21.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.736+0000 7f4b9728c700 1 -- 172.21.15.189:0/3038460345 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b900c70a0 con 0x7f4b900b3fa0 2024-04-06T13:45:21.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.738+0000 7f4b85ffb700 1 -- 172.21.15.189:0/3038460345 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 123) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f4b8c015440 con 0x7f4b900b3fa0 2024-04-06T13:45:21.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.739+0000 7f4b85ffb700 1 --2- 172.21.15.189:0/3038460345 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4b7406ae80 0x7f4b7406d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:21.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.739+0000 7f4b85ffb700 1 -- 172.21.15.189:0/3038460345 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6763+0+0 (secure 0 0 0) 0x7f4b8c08a440 con 0x7f4b900b3fa0 2024-04-06T13:45:21.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.739+0000 7f4b94827700 1 -- 172.21.15.189:0/3038460345 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4b7406ae80 msgr2=0x7f4b7406d340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:21.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.739+0000 7f4b94827700 1 --2- 172.21.15.189:0/3038460345 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4b7406ae80 0x7f4b7406d340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:45:21.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.744+0000 7f4b85ffb700 1 -- 172.21.15.189:0/3038460345 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4b8c054fc0 con 0x7f4b900b3fa0 2024-04-06T13:45:21.779 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:21.778+0000 7f3f48896200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:45:21.780 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:21.780+0000 7fbe06afc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:45:21.854 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:21.854+0000 7fbe06afc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:45:21.863 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:21.862+0000 7f3f48896200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:45:21.940 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:21.939+0000 7f3f48896200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:45:21.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.940+0000 7f4b94827700 1 -- 172.21.15.189:0/3038460345 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4b7406ae80 msgr2=0x7f4b7406d340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:21.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.940+0000 7f4b94827700 1 --2- 172.21.15.189:0/3038460345 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4b7406ae80 0x7f4b7406d340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:45:21.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.950+0000 7f4b9728c700 1 -- 172.21.15.189:0/3038460345 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4b900748f0 con 0x7f4b900b3fa0 2024-04-06T13:45:21.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.951+0000 7f4b85ffb700 1 -- 172.21.15.189:0/3038460345 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v195) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4b8c0585e0 con 0x7f4b900b3fa0 2024-04-06T13:45:21.951 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:45:21.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.953+0000 7f4b9728c700 1 -- 172.21.15.189:0/3038460345 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4b7406ae80 msgr2=0x7f4b7406d340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:45:21.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.953+0000 7f4b9728c700 1 --2- 172.21.15.189:0/3038460345 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4b7406ae80 0x7f4b7406d340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.953+0000 7f4b9728c700 1 -- 172.21.15.189:0/3038460345 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b900b3fa0 msgr2=0x7f4b9014b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:21.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.953+0000 7f4b9728c700 1 --2- 172.21.15.189:0/3038460345 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b900b3fa0 0x7f4b9014b500 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c014cf0 tx=0x7f4b8c00fe60 comp rx=0 tx=0).stop 2024-04-06T13:45:21.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.953+0000 7f4b9728c700 1 -- 172.21.15.189:0/3038460345 shutdown_connections 2024-04-06T13:45:21.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.953+0000 7f4b9728c700 1 --2- 172.21.15.189:0/3038460345 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4b7406ae80 0x7f4b7406d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.953+0000 7f4b9728c700 1 --2- 172.21.15.189:0/3038460345 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b9015b1b0 0x7f4b900c7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.953+0000 7f4b9728c700 1 --2- 172.21.15.189:0/3038460345 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b900b3fa0 0x7f4b9014b500 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.953+0000 7f4b9728c700 1 --2- 172.21.15.189:0/3038460345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b900cb7a0 0x7f4b900cab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:21.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.953+0000 7f4b9728c700 1 -- 172.21.15.189:0/3038460345 >> 172.21.15.189:0/3038460345 conn(0x7f4b90070300 msgr2=0x7f4b900b4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:21.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.953+0000 7f4b9728c700 1 -- 172.21.15.189:0/3038460345 shutdown_connections 2024-04-06T13:45:21.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:21.953+0000 7f4b9728c700 1 -- 172.21.15.189:0/3038460345 wait complete. 2024-04-06T13:45:21.966 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T13:45:22.058 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:22.058+0000 7fbe06afc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:45:22.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.094+0000 7f5332532700 1 Processor -- start 2024-04-06T13:45:22.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.095+0000 7f5332532700 1 -- start start 2024-04-06T13:45:22.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.096+0000 7f5332532700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f532c0cb870 0x7f532c15c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:22.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.096+0000 7f5332532700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f532c0cbd80 con 0x7f532c0cfa00 2024-04-06T13:45:22.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.096+0000 7f5332532700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f532c0cef20 con 0x7f532c0d11b0 2024-04-06T13:45:22.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.096+0000 7f5332532700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f532c0ccdc0 con 0x7f532c0cb870 2024-04-06T13:45:22.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.096+0000 7f532bfff700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f532c0d11b0 0x7f532c0d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:56742/0 (socket says 172.21.15.189:56742) 2024-04-06T13:45:22.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.096+0000 7f532bfff700 1 -- 172.21.15.189:0/1045280608 learned_addr learned my addr 172.21.15.189:0/1045280608 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:22.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.096+0000 7f532b7fe700 1 --2- 172.21.15.189:0/1045280608 >> v2:172.21.15.189:3300/0 conn(0x7f532c0cb870 0x7f532c15c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:22.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.096+0000 7f532affd700 1 -- 172.21.15.189:0/1045280608 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443494908 0 0) 0x7f532c0cef20 con 0x7f532c0d11b0 2024-04-06T13:45:22.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.096+0000 7f532affd700 1 -- 172.21.15.189:0/1045280608 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53140035b0 con 0x7f532c0d11b0 2024-04-06T13:45:22.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.096+0000 7f532affd700 1 -- 172.21.15.189:0/1045280608 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712386538 0 0) 0x7f532c0cbd80 con 0x7f532c0cfa00 2024-04-06T13:45:22.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.096+0000 7f532affd700 1 -- 172.21.15.189:0/1045280608 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f532c0cef20 con 0x7f532c0cfa00 2024-04-06T13:45:22.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.097+0000 7f532b7fe700 1 -- 172.21.15.189:0/1045280608 >> v1:172.21.15.192:6789/0 conn(0x7f532c0d11b0 legacy=0x7f532c0d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:22.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.097+0000 7f532b7fe700 1 -- 172.21.15.189:0/1045280608 >> v1:172.21.15.189:6789/0 conn(0x7f532c0cfa00 legacy=0x7f532c0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:22.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.097+0000 7f532b7fe700 1 -- 172.21.15.189:0/1045280608 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532c0cf200 con 0x7f532c0cb870 2024-04-06T13:45:22.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.097+0000 7f532b7fe700 1 --2- 172.21.15.189:0/1045280608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f532c0cb870 0x7f532c15c280 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f5320009840 tx=0x7f5320009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce0c04cbb621e41 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:22.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.097+0000 7f532affd700 1 -- 172.21.15.189:0/1045280608 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5320011830 con 0x7f532c0cb870 2024-04-06T13:45:22.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.097+0000 7f532affd700 1 -- 172.21.15.189:0/1045280608 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5320011990 con 0x7f532c0cb870 2024-04-06T13:45:22.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.097+0000 7f532affd700 1 -- 172.21.15.189:0/1045280608 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5320011d20 con 0x7f532c0cb870 2024-04-06T13:45:22.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.098+0000 7f5332532700 1 -- 172.21.15.189:0/1045280608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f532c0cb870 msgr2=0x7f532c15c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:22.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.098+0000 7f5332532700 1 --2- 172.21.15.189:0/1045280608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f532c0cb870 0x7f532c15c280 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f5320009840 tx=0x7f5320009b50 comp rx=0 tx=0).stop 2024-04-06T13:45:22.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.098+0000 7f5332532700 1 -- 172.21.15.189:0/1045280608 shutdown_connections 2024-04-06T13:45:22.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.098+0000 7f5332532700 1 --2- 172.21.15.189:0/1045280608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f532c0cb870 0x7f532c15c280 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:22.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.098+0000 7f5332532700 1 -- 172.21.15.189:0/1045280608 >> 172.21.15.189:0/1045280608 conn(0x7f532c0b2db0 msgr2=0x7f532c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:22.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.098+0000 7f5332532700 1 -- 172.21.15.189:0/1045280608 shutdown_connections 2024-04-06T13:45:22.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.098+0000 7f5332532700 1 -- 172.21.15.189:0/1045280608 wait complete. 2024-04-06T13:45:22.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.100+0000 7f5332532700 1 Processor -- start 2024-04-06T13:45:22.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.100+0000 7f5332532700 1 -- start start 2024-04-06T13:45:22.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.101+0000 7f5332532700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f532c0cb870 0x7f532c158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:22.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.101+0000 7f5332532700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f532c0cfa00 0x7f532c1508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:22.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.101+0000 7f5332532700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f532c0d11b0 0x7f532c154400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:22.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.101+0000 7f5332532700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c1f7050 con 0x7f532c0cfa00 2024-04-06T13:45:22.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.101+0000 7f5332532700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c0c52c0 con 0x7f532c0cb870 2024-04-06T13:45:22.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.101+0000 7f5332532700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532c0c55a0 con 0x7f532c0d11b0 2024-04-06T13:45:22.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.101+0000 7f532b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f532c0cfa00 0x7f532c1508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:22.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.101+0000 7f532b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f532c0cfa00 0x7f532c1508b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50592/0 (socket says 172.21.15.189:50592) 2024-04-06T13:45:22.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.101+0000 7f532bfff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f532c0cb870 0x7f532c158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:22.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.101+0000 7f532b7fe700 1 -- 172.21.15.189:0/125473810 learned_addr learned my addr 172.21.15.189:0/125473810 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:22.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.102+0000 7f5330acf700 1 --2- 172.21.15.189:0/125473810 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f532c0d11b0 0x7f532c154400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:22.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.102+0000 7f532bfff700 1 -- 172.21.15.189:0/125473810 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f532c0d11b0 msgr2=0x7f532c154400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:22.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.102+0000 7f532bfff700 1 --2- 172.21.15.189:0/125473810 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f532c0d11b0 0x7f532c154400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:22.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.102+0000 7f532bfff700 1 -- 172.21.15.189:0/125473810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f532c0cfa00 msgr2=0x7f532c1508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:22.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.102+0000 7f532bfff700 1 --2- 172.21.15.189:0/125473810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f532c0cfa00 0x7f532c1508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:22.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.102+0000 7f532bfff700 1 -- 172.21.15.189:0/125473810 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53200094a0 con 0x7f532c0cb870 2024-04-06T13:45:22.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.102+0000 7f532b7fe700 1 --2- 172.21.15.189:0/125473810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f532c0cfa00 0x7f532c1508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:45:22.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.102+0000 7f532bfff700 1 --2- 172.21.15.189:0/125473810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f532c0cb870 0x7f532c158a80 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f531c00c390 tx=0x7f531c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:22.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.102+0000 7f5328ff9700 1 -- 172.21.15.189:0/125473810 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531c005b40 con 0x7f532c0cb870 2024-04-06T13:45:22.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.102+0000 7f5328ff9700 1 -- 172.21.15.189:0/125473810 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f531c005cc0 con 0x7f532c0cb870 2024-04-06T13:45:22.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.102+0000 7f5328ff9700 1 -- 172.21.15.189:0/125473810 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531c007310 con 0x7f532c0cb870 2024-04-06T13:45:22.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.102+0000 7f5332532700 1 -- 172.21.15.189:0/125473810 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f532c0c5880 con 0x7f532c0cb870 2024-04-06T13:45:22.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.102+0000 7f5332532700 1 -- 172.21.15.189:0/125473810 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f532c0c5e40 con 0x7f532c0cb870 2024-04-06T13:45:22.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.104+0000 7f5332532700 1 -- 172.21.15.189:0/125473810 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f532c0c6030 con 0x7f532c0cb870 2024-04-06T13:45:22.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.105+0000 7f5328ff9700 1 -- 172.21.15.189:0/125473810 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 123) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f531c007470 con 0x7f532c0cb870 2024-04-06T13:45:22.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.105+0000 7f5328ff9700 1 --2- 172.21.15.189:0/125473810 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f531806adb0 0x7f531806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:22.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.105+0000 7f5328ff9700 1 -- 172.21.15.189:0/125473810 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6763+0+0 (secure 0 0 0) 0x7f531c088bc0 con 0x7f532c0cb870 2024-04-06T13:45:22.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.105+0000 7f532b7fe700 1 -- 172.21.15.189:0/125473810 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f531806adb0 msgr2=0x7f531806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:22.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.105+0000 7f532b7fe700 1 --2- 172.21.15.189:0/125473810 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f531806adb0 0x7f531806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:45:22.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.110+0000 7f5328ff9700 1 -- 172.21.15.189:0/125473810 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f531c053870 con 0x7f532c0cb870 2024-04-06T13:45:22.133 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:22.133+0000 7fbe06afc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:45:22.157 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:22.155+0000 7f3f48896200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:45:22.232 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:22.231+0000 7f3f48896200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:45:22.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.306+0000 7f532b7fe700 1 -- 172.21.15.189:0/125473810 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f531806adb0 msgr2=0x7f531806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:22.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.306+0000 7f532b7fe700 1 --2- 172.21.15.189:0/125473810 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f531806adb0 0x7f531806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:45:22.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.338+0000 7f5332532700 1 -- 172.21.15.189:0/125473810 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f532c074990 con 0x7f532c0cb870 2024-04-06T13:45:22.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.339+0000 7f5328ff9700 1 -- 172.21.15.189:0/125473810 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v195) v1 ==== 100+0+11 (secure 0 0 0) 0x7f531c0170a0 con 0x7f532c0cb870 2024-04-06T13:45:22.340 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:45:22.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.341+0000 7f5332532700 1 -- 172.21.15.189:0/125473810 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f531806adb0 msgr2=0x7f531806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:45:22.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.341+0000 7f5332532700 1 --2- 172.21.15.189:0/125473810 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f531806adb0 0x7f531806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:22.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.341+0000 7f5332532700 1 -- 172.21.15.189:0/125473810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f532c0cb870 msgr2=0x7f532c158a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:22.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.341+0000 7f5332532700 1 --2- 172.21.15.189:0/125473810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f532c0cb870 0x7f532c158a80 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f531c00c390 tx=0x7f531c00c6a0 comp rx=0 tx=0).stop 2024-04-06T13:45:22.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.341+0000 7f5332532700 1 -- 172.21.15.189:0/125473810 shutdown_connections 2024-04-06T13:45:22.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.341+0000 7f5332532700 1 --2- 172.21.15.189:0/125473810 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f531806adb0 0x7f531806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:22.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.341+0000 7f5332532700 1 --2- 172.21.15.189:0/125473810 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f532c0d11b0 0x7f532c154400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:22.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.341+0000 7f5332532700 1 --2- 172.21.15.189:0/125473810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f532c0cb870 0x7f532c158a80 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:22.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.341+0000 7f5332532700 1 --2- 172.21.15.189:0/125473810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f532c0cfa00 0x7f532c1508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:22.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.341+0000 7f5332532700 1 -- 172.21.15.189:0/125473810 >> 172.21.15.189:0/125473810 conn(0x7f532c0b2db0 msgr2=0x7f532c152530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:22.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.341+0000 7f5332532700 1 -- 172.21.15.189:0/125473810 shutdown_connections 2024-04-06T13:45:22.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.341+0000 7f5332532700 1 -- 172.21.15.189:0/125473810 wait complete. 2024-04-06T13:45:22.355 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T13:45:22.451 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:22.451+0000 7fbe06afc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:45:22.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.482+0000 7f6caf3a8700 1 Processor -- start 2024-04-06T13:45:22.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.482+0000 7f6caf3a8700 1 -- start start 2024-04-06T13:45:22.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.483+0000 7f6caf3a8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6ca806ae40 0x7f6ca815ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:22.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.483+0000 7f6caf3a8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca815fcd0 0x7f6ca815bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:22.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.483+0000 7f6caf3a8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca815c4c0 con 0x7f6ca8144f40 2024-04-06T13:45:22.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.483+0000 7f6caf3a8700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca80cfcb0 con 0x7f6ca806ae40 2024-04-06T13:45:22.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.483+0000 7f6caf3a8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca815ca20 con 0x7f6ca815fcd0 2024-04-06T13:45:22.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.483+0000 7f6cad144700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6ca8144f40 0x7f6ca806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38436/0 (socket says 172.21.15.189:38436) 2024-04-06T13:45:22.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.483+0000 7f6cad144700 1 -- 172.21.15.189:0/701531176 learned_addr learned my addr 172.21.15.189:0/701531176 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:22.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.483+0000 7f6cad945700 1 --2- 172.21.15.189:0/701531176 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca815fcd0 0x7f6ca815bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:22.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.483+0000 7f6cac943700 1 --2- 172.21.15.189:0/701531176 >> v2:172.21.15.192:3300/0 conn(0x7f6ca806ae40 0x7f6ca815ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:22.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.483+0000 7f6c9ffff700 1 -- 172.21.15.189:0/701531176 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729571833 0 0) 0x7f6ca815c4c0 con 0x7f6ca8144f40 2024-04-06T13:45:22.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.484+0000 7f6c9ffff700 1 -- 172.21.15.189:0/701531176 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c940035b0 con 0x7f6ca8144f40 2024-04-06T13:45:22.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.484+0000 7f6cac943700 1 -- 172.21.15.189:0/701531176 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca815fcd0 msgr2=0x7f6ca815bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:22.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.484+0000 7f6cac943700 1 --2- 172.21.15.189:0/701531176 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca815fcd0 0x7f6ca815bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:22.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.484+0000 7f6cac943700 1 -- 172.21.15.189:0/701531176 >> v1:172.21.15.189:6789/0 conn(0x7f6ca8144f40 legacy=0x7f6ca806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:22.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.484+0000 7f6cac943700 1 -- 172.21.15.189:0/701531176 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca815c740 con 0x7f6ca806ae40 2024-04-06T13:45:22.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.484+0000 7f6cad945700 1 --2- 172.21.15.189:0/701531176 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca815fcd0 0x7f6ca815bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:45:22.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.484+0000 7f6cac943700 1 --2- 172.21.15.189:0/701531176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca806ae40 0x7f6ca815ba40 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009840 tx=0x7f6c98009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4687f04df37b0d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:22.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.484+0000 7f6c9ffff700 1 -- 172.21.15.189:0/701531176 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98011830 con 0x7f6ca806ae40 2024-04-06T13:45:22.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.484+0000 7f6c9ffff700 1 -- 172.21.15.189:0/701531176 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c98011990 con 0x7f6ca806ae40 2024-04-06T13:45:22.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.484+0000 7f6c9ffff700 1 -- 172.21.15.189:0/701531176 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98011d00 con 0x7f6ca806ae40 2024-04-06T13:45:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.485+0000 7f6caf3a8700 1 -- 172.21.15.189:0/701531176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca806ae40 msgr2=0x7f6ca815ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.485+0000 7f6caf3a8700 1 --2- 172.21.15.189:0/701531176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca806ae40 0x7f6ca815ba40 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009840 tx=0x7f6c98009b50 comp rx=0 tx=0).stop 2024-04-06T13:45:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.485+0000 7f6caf3a8700 1 -- 172.21.15.189:0/701531176 shutdown_connections 2024-04-06T13:45:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.485+0000 7f6caf3a8700 1 --2- 172.21.15.189:0/701531176 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca815fcd0 0x7f6ca815bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.485+0000 7f6caf3a8700 1 --2- 172.21.15.189:0/701531176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca806ae40 0x7f6ca815ba40 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.485+0000 7f6caf3a8700 1 -- 172.21.15.189:0/701531176 >> 172.21.15.189:0/701531176 conn(0x7f6ca80d00b0 msgr2=0x7f6ca80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:22.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.485+0000 7f6caf3a8700 1 -- 172.21.15.189:0/701531176 shutdown_connections 2024-04-06T13:45:22.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.485+0000 7f6caf3a8700 1 -- 172.21.15.189:0/701531176 wait complete. 2024-04-06T13:45:22.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.487+0000 7f6caf3a8700 1 Processor -- start 2024-04-06T13:45:22.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.487+0000 7f6caf3a8700 1 -- start start 2024-04-06T13:45:22.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.488+0000 7f6caf3a8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca806ae40 0x7f6ca8154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:22.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.488+0000 7f6caf3a8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca8144f40 0x7f6ca813f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:22.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.488+0000 7f6caf3a8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca815fcd0 0x7f6ca813f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:22.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.488+0000 7f6caf3a8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca81414a0 con 0x7f6ca815fcd0 2024-04-06T13:45:22.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.488+0000 7f6caf3a8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca81408f0 con 0x7f6ca8144f40 2024-04-06T13:45:22.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.488+0000 7f6caf3a8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca8140b00 con 0x7f6ca806ae40 2024-04-06T13:45:22.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.488+0000 7f6cad945700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca815fcd0 0x7f6ca813f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:22.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.488+0000 7f6cad945700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca815fcd0 0x7f6ca813f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50602/0 (socket says 172.21.15.189:50602) 2024-04-06T13:45:22.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.488+0000 7f6cad945700 1 -- 172.21.15.189:0/1632366580 learned_addr learned my addr 172.21.15.189:0/1632366580 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:22.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.488+0000 7f6cac943700 1 --2- 172.21.15.189:0/1632366580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca8144f40 0x7f6ca813f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:22.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.488+0000 7f6cad144700 1 --2- 172.21.15.189:0/1632366580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca806ae40 0x7f6ca8154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:22.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.489+0000 7f6cad144700 1 -- 172.21.15.189:0/1632366580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca8144f40 msgr2=0x7f6ca813f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:22.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.489+0000 7f6cad144700 1 --2- 172.21.15.189:0/1632366580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca8144f40 0x7f6ca813f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:22.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.489+0000 7f6cad144700 1 -- 172.21.15.189:0/1632366580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca815fcd0 msgr2=0x7f6ca813f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:22.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.489+0000 7f6cad144700 1 --2- 172.21.15.189:0/1632366580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca815fcd0 0x7f6ca813f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:22.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.489+0000 7f6cad144700 1 -- 172.21.15.189:0/1632366580 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca0009a70 con 0x7f6ca806ae40 2024-04-06T13:45:22.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.489+0000 7f6cac943700 1 --2- 172.21.15.189:0/1632366580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca8144f40 0x7f6ca813f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:45:22.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.489+0000 7f6cad945700 1 --2- 172.21.15.189:0/1632366580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca815fcd0 0x7f6ca813f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:45:22.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.489+0000 7f6cad144700 1 --2- 172.21.15.189:0/1632366580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca806ae40 0x7f6ca8154e70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6ca400c420 tx=0x7f6ca400c730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:22.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.489+0000 7f6c9dffb700 1 -- 172.21.15.189:0/1632366580 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4005d70 con 0x7f6ca806ae40 2024-04-06T13:45:22.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.489+0000 7f6c9dffb700 1 -- 172.21.15.189:0/1632366580 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca400ce30 con 0x7f6ca806ae40 2024-04-06T13:45:22.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.489+0000 7f6c9dffb700 1 -- 172.21.15.189:0/1632366580 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca40072f0 con 0x7f6ca806ae40 2024-04-06T13:45:22.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.489+0000 7f6caf3a8700 1 -- 172.21.15.189:0/1632366580 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c980094a0 con 0x7f6ca806ae40 2024-04-06T13:45:22.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.489+0000 7f6caf3a8700 1 -- 172.21.15.189:0/1632366580 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca81410c0 con 0x7f6ca806ae40 2024-04-06T13:45:22.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.490+0000 7f6caf3a8700 1 -- 172.21.15.189:0/1632366580 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca81415e0 con 0x7f6ca806ae40 2024-04-06T13:45:22.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.491+0000 7f6c9dffb700 1 -- 172.21.15.189:0/1632366580 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 123) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f6ca4007450 con 0x7f6ca806ae40 2024-04-06T13:45:22.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.491+0000 7f6c9dffb700 1 --2- 172.21.15.189:0/1632366580 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6c8c06adb0 0x7f6c8c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:22.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.491+0000 7f6cac943700 1 -- 172.21.15.189:0/1632366580 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6c8c06adb0 msgr2=0x7f6c8c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:22.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.491+0000 7f6c9dffb700 1 -- 172.21.15.189:0/1632366580 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6763+0+0 (secure 0 0 0) 0x7f6ca4088be0 con 0x7f6ca806ae40 2024-04-06T13:45:22.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.491+0000 7f6cac943700 1 --2- 172.21.15.189:0/1632366580 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6c8c06adb0 0x7f6c8c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:45:22.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.496+0000 7f6c9dffb700 1 -- 172.21.15.189:0/1632366580 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6ca81415e0 con 0x7f6ca806ae40 2024-04-06T13:45:22.573 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:22.572+0000 7f3f48896200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:45:22.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.692+0000 7f6cac943700 1 -- 172.21.15.189:0/1632366580 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6c8c06adb0 msgr2=0x7f6c8c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:22.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.692+0000 7f6cac943700 1 --2- 172.21.15.189:0/1632366580 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6c8c06adb0 0x7f6c8c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:45:22.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.722+0000 7f6caf3a8700 1 -- 172.21.15.189:0/1632366580 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6ca81415e0 con 0x7f6ca806ae40 2024-04-06T13:45:22.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.723+0000 7f6c9dffb700 1 -- 172.21.15.189:0/1632366580 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 14 entries v196) v1 ==== 81+0+821 (secure 0 0 0) 0x7f6ca81415e0 con 0x7f6ca806ae40 2024-04-06T13:45:22.724 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1090714269 2024-04-07T13:40:22.300382+0000 2024-04-06T13:45:22.724 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/3656453012 2024-04-07T13:40:22.300382+0000 2024-04-06T13:45:22.724 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2710277231 2024-04-07T13:40:22.300382+0000 2024-04-06T13:45:22.724 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1953253738 2024-04-07T13:40:22.300382+0000 2024-04-06T13:45:22.724 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1974012472 2024-04-07T13:40:22.300382+0000 2024-04-06T13:45:22.724 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/820436148 2024-04-07T13:29:30.406338+0000 2024-04-06T13:45:22.724 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2599831431 2024-04-07T13:29:30.406338+0000 2024-04-06T13:45:22.724 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1401962292 2024-04-07T13:29:30.406338+0000 2024-04-06T13:45:22.724 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2983879481 2024-04-07T13:40:22.300382+0000 2024-04-06T13:45:22.724 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/3870180304 2024-04-07T13:29:30.406338+0000 2024-04-06T13:45:22.725 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1073864749 2024-04-07T13:29:30.406338+0000 2024-04-06T13:45:22.725 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/3525693740 2024-04-07T13:29:30.406338+0000 2024-04-06T13:45:22.725 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6833/47128 2024-04-07T13:29:30.406338+0000 2024-04-06T13:45:22.725 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6832/47128 2024-04-07T13:29:30.406338+0000 2024-04-06T13:45:22.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.725+0000 7f6caf3a8700 1 -- 172.21.15.189:0/1632366580 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6c8c06adb0 msgr2=0x7f6c8c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:45:22.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.725+0000 7f6caf3a8700 1 --2- 172.21.15.189:0/1632366580 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6c8c06adb0 0x7f6c8c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:22.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.725+0000 7f6caf3a8700 1 -- 172.21.15.189:0/1632366580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca806ae40 msgr2=0x7f6ca8154e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:22.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.725+0000 7f6caf3a8700 1 --2- 172.21.15.189:0/1632366580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca806ae40 0x7f6ca8154e70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6ca400c420 tx=0x7f6ca400c730 comp rx=0 tx=0).stop 2024-04-06T13:45:22.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.725+0000 7f6caf3a8700 1 -- 172.21.15.189:0/1632366580 shutdown_connections 2024-04-06T13:45:22.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.725+0000 7f6caf3a8700 1 --2- 172.21.15.189:0/1632366580 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6c8c06adb0 0x7f6c8c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:22.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.725+0000 7f6caf3a8700 1 --2- 172.21.15.189:0/1632366580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ca815fcd0 0x7f6ca813f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:22.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.725+0000 7f6caf3a8700 1 --2- 172.21.15.189:0/1632366580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca806ae40 0x7f6ca8154e70 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:22.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.725+0000 7f6caf3a8700 1 --2- 172.21.15.189:0/1632366580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca8144f40 0x7f6ca813f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:22.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.725+0000 7f6caf3a8700 1 -- 172.21.15.189:0/1632366580 >> 172.21.15.189:0/1632366580 conn(0x7f6ca80d00b0 msgr2=0x7f6ca80d0520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:22.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.725+0000 7f6caf3a8700 1 -- 172.21.15.189:0/1632366580 shutdown_connections 2024-04-06T13:45:22.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:22.725+0000 7f6caf3a8700 1 -- 172.21.15.189:0/1632366580 wait complete. 2024-04-06T13:45:22.727 INFO:teuthology.orchestra.run.smithi189.stderr:listed 14 entries 2024-04-06T13:45:22.739 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-06T13:45:22.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:22.739 DEBUG:teuthology.orchestra.run.smithi189:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-06T13:45:22.782 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:22.782+0000 7fbe06afc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:45:22.783 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T13:45:22.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:22.784 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-06T13:45:22.841 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:45:22.842 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:45:22.859 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:45:22.859 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:45:22.859 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:45:22.899 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:45:22.899 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:45:22.899 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:45:22.915 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:22.915+0000 7f3f48896200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:45:22.917 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:45:22.918 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:45:22.950 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:22.950+0000 7fbe06afc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:45:23.025 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:23.025+0000 7fbe06afc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:45:23.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.045+0000 7f7e20cd9700 1 Processor -- start 2024-04-06T13:45:23.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.045+0000 7f7e20cd9700 1 -- start start 2024-04-06T13:45:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.046+0000 7f7e20cd9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7e1c0d11b0 0x7f7e1c0d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.046+0000 7f7e20cd9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e1c0cb870 0x7f7e1c15c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.046+0000 7f7e20cd9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7e1c0cfa00 0x7f7e1c0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.046+0000 7f7e20cd9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e1c0ccdc0 con 0x7f7e1c0cfa00 2024-04-06T13:45:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.046+0000 7f7e20cd9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e1c0d0b50 con 0x7f7e1c0d11b0 2024-04-06T13:45:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.046+0000 7f7e20cd9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c0d1ad0 con 0x7f7e1c0cb870 2024-04-06T13:45:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.046+0000 7f7e1ad9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7e1c0cfa00 0x7f7e1c0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.046+0000 7f7e1ad9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7e1c0cfa00 0x7f7e1c0cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50604/0 (socket says 172.21.15.189:50604) 2024-04-06T13:45:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.046+0000 7f7e1a59c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7e1c0d11b0 0x7f7e1c0d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:23.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.046+0000 7f7e19d9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e1c0cb870 0x7f7e1c15c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.046+0000 7f7e1ad9d700 1 -- 172.21.15.189:0/1358246371 learned_addr learned my addr 172.21.15.189:0/1358246371 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.046+0000 7f7e1ad9d700 1 -- 172.21.15.189:0/1358246371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e1c0cb870 msgr2=0x7f7e1c15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.046+0000 7f7e1ad9d700 1 --2- 172.21.15.189:0/1358246371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e1c0cb870 0x7f7e1c15c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.046+0000 7f7e1ad9d700 1 -- 172.21.15.189:0/1358246371 >> v2:172.21.15.192:3300/0 conn(0x7f7e1c0d11b0 msgr2=0x7f7e1c0d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.046+0000 7f7e1ad9d700 1 --2- 172.21.15.189:0/1358246371 >> v2:172.21.15.192:3300/0 conn(0x7f7e1c0d11b0 0x7f7e1c0d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.046+0000 7f7e1ad9d700 1 -- 172.21.15.189:0/1358246371 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c0cbd80 con 0x7f7e1c0cfa00 2024-04-06T13:45:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.047+0000 7f7e1ad9d700 1 --2- 172.21.15.189:0/1358246371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e1c0cfa00 0x7f7e1c0cd560 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c0ced00 tx=0x7f7e1400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcf1463e54b3bcb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.047+0000 7f7e1959a700 1 -- 172.21.15.189:0/1358246371 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1400e070 con 0x7f7e1c0cfa00 2024-04-06T13:45:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.047+0000 7f7e1959a700 1 -- 172.21.15.189:0/1358246371 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1400b910 con 0x7f7e1c0cfa00 2024-04-06T13:45:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.047+0000 7f7e1959a700 1 -- 172.21.15.189:0/1358246371 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1400bcc0 con 0x7f7e1c0cfa00 2024-04-06T13:45:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.048+0000 7f7e20cd9700 1 -- 172.21.15.189:0/1358246371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e1c0cfa00 msgr2=0x7f7e1c0cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.048+0000 7f7e20cd9700 1 --2- 172.21.15.189:0/1358246371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e1c0cfa00 0x7f7e1c0cd560 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c0ced00 tx=0x7f7e1400b330 comp rx=0 tx=0).stop 2024-04-06T13:45:23.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.048+0000 7f7e20cd9700 1 -- 172.21.15.189:0/1358246371 shutdown_connections 2024-04-06T13:45:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.048+0000 7f7e20cd9700 1 --2- 172.21.15.189:0/1358246371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e1c0cfa00 0x7f7e1c0cd560 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.048+0000 7f7e20cd9700 1 --2- 172.21.15.189:0/1358246371 >> v2:172.21.15.192:3300/0 conn(0x7f7e1c0d11b0 0x7f7e1c0d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.048+0000 7f7e20cd9700 1 --2- 172.21.15.189:0/1358246371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e1c0cb870 0x7f7e1c15c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.048+0000 7f7e20cd9700 1 -- 172.21.15.189:0/1358246371 >> 172.21.15.189:0/1358246371 conn(0x7f7e1c0b2db0 msgr2=0x7f7e1c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.048+0000 7f7e20cd9700 1 -- 172.21.15.189:0/1358246371 shutdown_connections 2024-04-06T13:45:23.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.048+0000 7f7e20cd9700 1 -- 172.21.15.189:0/1358246371 wait complete. 2024-04-06T13:45:23.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.050+0000 7f7e20cd9700 1 Processor -- start 2024-04-06T13:45:23.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.050+0000 7f7e20cd9700 1 -- start start 2024-04-06T13:45:23.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.051+0000 7f7e20cd9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e1c0cb870 0x7f7e1c0c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:23.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.051+0000 7f7e20cd9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e1c0cfa00 0x7f7e1c0c3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:23.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.051+0000 7f7e20cd9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e1c0d11b0 0x7f7e1c0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:23.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.051+0000 7f7e20cd9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c0c4250 con 0x7f7e1c0d11b0 2024-04-06T13:45:23.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.051+0000 7f7e20cd9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c0c5b90 con 0x7f7e1c0cfa00 2024-04-06T13:45:23.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.051+0000 7f7e20cd9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c0c55d0 con 0x7f7e1c0cb870 2024-04-06T13:45:23.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.051+0000 7f7e1ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e1c0d11b0 0x7f7e1c0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:23.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.051+0000 7f7e1ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e1c0d11b0 0x7f7e1c0c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50616/0 (socket says 172.21.15.189:50616) 2024-04-06T13:45:23.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.051+0000 7f7e1ad9d700 1 -- 172.21.15.189:0/51720682 learned_addr learned my addr 172.21.15.189:0/51720682 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:23.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.051+0000 7f7e19d9b700 1 --2- 172.21.15.189:0/51720682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e1c0cfa00 0x7f7e1c0c3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.051+0000 7f7e1a59c700 1 --2- 172.21.15.189:0/51720682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e1c0cb870 0x7f7e1c0c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.051+0000 7f7e1ad9d700 1 -- 172.21.15.189:0/51720682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e1c0cb870 msgr2=0x7f7e1c0c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.051+0000 7f7e1ad9d700 1 --2- 172.21.15.189:0/51720682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e1c0cb870 0x7f7e1c0c82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.051+0000 7f7e1ad9d700 1 -- 172.21.15.189:0/51720682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e1c0cfa00 msgr2=0x7f7e1c0c3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.051+0000 7f7e1ad9d700 1 --2- 172.21.15.189:0/51720682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e1c0cfa00 0x7f7e1c0c3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.051+0000 7f7e1ad9d700 1 -- 172.21.15.189:0/51720682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e14009b30 con 0x7f7e1c0d11b0 2024-04-06T13:45:23.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.052+0000 7f7e1ad9d700 1 --2- 172.21.15.189:0/51720682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e1c0d11b0 0x7f7e1c0c4b80 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c0677e0 tx=0x7f7e14004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:23.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.052+0000 7f7e0affd700 1 -- 172.21.15.189:0/51720682 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e140099b0 con 0x7f7e1c0d11b0 2024-04-06T13:45:23.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.052+0000 7f7e0affd700 1 -- 172.21.15.189:0/51720682 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1400ce80 con 0x7f7e1c0d11b0 2024-04-06T13:45:23.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.052+0000 7f7e20cd9700 1 -- 172.21.15.189:0/51720682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1c0c58b0 con 0x7f7e1c0d11b0 2024-04-06T13:45:23.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.052+0000 7f7e0affd700 1 -- 172.21.15.189:0/51720682 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e14004490 con 0x7f7e1c0d11b0 2024-04-06T13:45:23.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.052+0000 7f7e20cd9700 1 -- 172.21.15.189:0/51720682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1c0c6830 con 0x7f7e1c0d11b0 2024-04-06T13:45:23.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.053+0000 7f7e20cd9700 1 -- 172.21.15.189:0/51720682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1c074990 con 0x7f7e1c0d11b0 2024-04-06T13:45:23.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.053+0000 7f7e0affd700 1 -- 172.21.15.189:0/51720682 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 123) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f7e140045f0 con 0x7f7e1c0d11b0 2024-04-06T13:45:23.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.054+0000 7f7e0affd700 1 --2- 172.21.15.189:0/51720682 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7dfc06adb0 0x7f7dfc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:23.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.054+0000 7f7e1a59c700 1 -- 172.21.15.189:0/51720682 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7dfc06adb0 msgr2=0x7f7dfc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:23.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.054+0000 7f7e0affd700 1 -- 172.21.15.189:0/51720682 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6747+0+0 (secure 0 0 0) 0x7f7e1408a8a0 con 0x7f7e1c0d11b0 2024-04-06T13:45:23.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.054+0000 7f7e1a59c700 1 --2- 172.21.15.189:0/51720682 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7dfc06adb0 0x7f7dfc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:45:23.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.058+0000 7f7e0affd700 1 -- 172.21.15.189:0/51720682 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7e140554e0 con 0x7f7e1c0d11b0 2024-04-06T13:45:23.093 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:23.092+0000 7f3f48896200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:45:23.169 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:23.169+0000 7f3f48896200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:45:23.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.255+0000 7f7e1a59c700 1 -- 172.21.15.189:0/51720682 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7dfc06adb0 msgr2=0x7f7dfc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:23.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.255+0000 7f7e1a59c700 1 --2- 172.21.15.189:0/51720682 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7dfc06adb0 0x7f7dfc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:45:23.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.308+0000 7f7e20cd9700 1 -- 172.21.15.189:0/51720682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e1c155c70 con 0x7f7e1c0d11b0 2024-04-06T13:45:23.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.308+0000 7f7e0affd700 1 -- 172.21.15.189:0/51720682 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+4749 (secure 0 0 0) 0x7f7e14058b00 con 0x7f7e1c0d11b0 2024-04-06T13:45:23.309 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:23.309 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":85,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8141,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2979280081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2979280081},{"type":"v1","addr":"172.21.15.192:6835","nonce":2979280081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":10125,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/286991908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":286991908},{"type":"v1","addr":"172.21.15.189:6836","nonce":286991908}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":10128,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/704323171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":704323171},{"type":"v1","addr":"172.21.15.189:6837","nonce":704323171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:29:16.858698+0000","modified":"2024-04-06T13:29:19.868525+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10137":{"gid":10137,"name":"b","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6834/2662267274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2662267274},{"type":"v1","addr":"172.21.15.192:6834","nonce":2662267274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-04-06T13:45:23.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.311+0000 7f7e20cd9700 1 -- 172.21.15.189:0/51720682 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7dfc06adb0 msgr2=0x7f7dfc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:45:23.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.311+0000 7f7e20cd9700 1 --2- 172.21.15.189:0/51720682 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7dfc06adb0 0x7f7dfc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.311+0000 7f7e20cd9700 1 -- 172.21.15.189:0/51720682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e1c0d11b0 msgr2=0x7f7e1c0c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:23.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.311+0000 7f7e20cd9700 1 --2- 172.21.15.189:0/51720682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e1c0d11b0 0x7f7e1c0c4b80 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c0677e0 tx=0x7f7e14004ad0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.311+0000 7f7e20cd9700 1 -- 172.21.15.189:0/51720682 shutdown_connections 2024-04-06T13:45:23.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.311+0000 7f7e20cd9700 1 --2- 172.21.15.189:0/51720682 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7dfc06adb0 0x7f7dfc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.311+0000 7f7e20cd9700 1 --2- 172.21.15.189:0/51720682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e1c0d11b0 0x7f7e1c0c4b80 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.311+0000 7f7e20cd9700 1 --2- 172.21.15.189:0/51720682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e1c0cb870 0x7f7e1c0c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.311+0000 7f7e20cd9700 1 --2- 172.21.15.189:0/51720682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e1c0cfa00 0x7f7e1c0c3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.311+0000 7f7e20cd9700 1 -- 172.21.15.189:0/51720682 >> 172.21.15.189:0/51720682 conn(0x7f7e1c0b2db0 msgr2=0x7f7e1c0b25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:23.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.311+0000 7f7e20cd9700 1 -- 172.21.15.189:0/51720682 shutdown_connections 2024-04-06T13:45:23.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.311+0000 7f7e20cd9700 1 -- 172.21.15.189:0/51720682 wait complete. 2024-04-06T13:45:23.313 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 85 2024-04-06T13:45:23.325 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:45:23.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.451+0000 7fb7ce042700 1 Processor -- start 2024-04-06T13:45:23.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.451+0000 7fb7ce042700 1 -- start start 2024-04-06T13:45:23.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.452+0000 7fb7ce042700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7c8159b20 0x7fb7c806b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:23.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.452+0000 7fb7ce042700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c806a8d0 con 0x7fb7c80cdc60 2024-04-06T13:45:23.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.452+0000 7fb7ce042700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c806abb0 con 0x7fb7c806eb80 2024-04-06T13:45:23.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.452+0000 7fb7ce042700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c8159f00 con 0x7fb7c8159b20 2024-04-06T13:45:23.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.452+0000 7fb7c6ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb7c80cdc60 0x7fb7c806e470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38438/0 (socket says 172.21.15.189:38438) 2024-04-06T13:45:23.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.452+0000 7fb7c6ffd700 1 -- 172.21.15.189:0/1025914325 learned_addr learned my addr 172.21.15.189:0/1025914325 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:23.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.452+0000 7fb7c77fe700 1 --2- 172.21.15.189:0/1025914325 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7c8159b20 0x7fb7c806b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:23.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.452+0000 7fb7c67fc700 1 -- 172.21.15.189:0/1025914325 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874860184 0 0) 0x7fb7c806a8d0 con 0x7fb7c80cdc60 2024-04-06T13:45:23.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.452+0000 7fb7c67fc700 1 -- 172.21.15.189:0/1025914325 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7b00046c0 con 0x7fb7c80cdc60 2024-04-06T13:45:23.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.452+0000 7fb7c67fc700 1 -- 172.21.15.189:0/1025914325 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529848350 0 0) 0x7fb7c806abb0 con 0x7fb7c806eb80 2024-04-06T13:45:23.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.453+0000 7fb7c67fc700 1 -- 172.21.15.189:0/1025914325 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c806a8d0 con 0x7fb7c806eb80 2024-04-06T13:45:23.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.453+0000 7fb7c77fe700 1 -- 172.21.15.189:0/1025914325 >> v1:172.21.15.192:6789/0 conn(0x7fb7c806eb80 legacy=0x7fb7c80ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:23.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.453+0000 7fb7c77fe700 1 -- 172.21.15.189:0/1025914325 >> v1:172.21.15.189:6789/0 conn(0x7fb7c80cdc60 legacy=0x7fb7c806e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:23.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.453+0000 7fb7c77fe700 1 -- 172.21.15.189:0/1025914325 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c806ae90 con 0x7fb7c8159b20 2024-04-06T13:45:23.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.453+0000 7fb7c77fe700 1 --2- 172.21.15.189:0/1025914325 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7c8159b20 0x7fb7c806b970 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009dc0 tx=0x7fb7b800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aadbdcffe5a9abb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:23.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.453+0000 7fb7c67fc700 1 -- 172.21.15.189:0/1025914325 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b800bd20 con 0x7fb7c8159b20 2024-04-06T13:45:23.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.453+0000 7fb7c67fc700 1 -- 172.21.15.189:0/1025914325 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b800be80 con 0x7fb7c8159b20 2024-04-06T13:45:23.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.453+0000 7fb7c67fc700 1 -- 172.21.15.189:0/1025914325 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b80116a0 con 0x7fb7c8159b20 2024-04-06T13:45:23.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.454+0000 7fb7ce042700 1 -- 172.21.15.189:0/1025914325 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7c8159b20 msgr2=0x7fb7c806b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:23.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.454+0000 7fb7ce042700 1 --2- 172.21.15.189:0/1025914325 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7c8159b20 0x7fb7c806b970 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009dc0 tx=0x7fb7b800b330 comp rx=0 tx=0).stop 2024-04-06T13:45:23.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.454+0000 7fb7ce042700 1 -- 172.21.15.189:0/1025914325 shutdown_connections 2024-04-06T13:45:23.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.454+0000 7fb7ce042700 1 --2- 172.21.15.189:0/1025914325 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7c8159b20 0x7fb7c806b970 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.454+0000 7fb7ce042700 1 -- 172.21.15.189:0/1025914325 >> 172.21.15.189:0/1025914325 conn(0x7fb7c80cc290 msgr2=0x7fb7c80cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:23.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.454+0000 7fb7ce042700 1 -- 172.21.15.189:0/1025914325 shutdown_connections 2024-04-06T13:45:23.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.454+0000 7fb7ce042700 1 -- 172.21.15.189:0/1025914325 wait complete. 2024-04-06T13:45:23.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.455+0000 7fb7ce042700 1 Processor -- start 2024-04-06T13:45:23.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.455+0000 7fb7ce042700 1 -- start start 2024-04-06T13:45:23.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.456+0000 7fb7ce042700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7c806eb80 0x7fb7c814aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:23.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.456+0000 7fb7ce042700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7c80cdc60 0x7fb7c815a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:23.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.456+0000 7fb7ce042700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7c8159b20 0x7fb7c814d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:23.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.456+0000 7fb7ce042700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c814af80 con 0x7fb7c806eb80 2024-04-06T13:45:23.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.456+0000 7fb7ce042700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c80ac970 con 0x7fb7c80cdc60 2024-04-06T13:45:23.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.456+0000 7fb7ce042700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c815cca0 con 0x7fb7c8159b20 2024-04-06T13:45:23.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.456+0000 7fb7c77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7c806eb80 0x7fb7c814aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.456+0000 7fb7c77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7c806eb80 0x7fb7c814aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50624/0 (socket says 172.21.15.189:50624) 2024-04-06T13:45:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.456+0000 7fb7c77fe700 1 -- 172.21.15.189:0/613486982 learned_addr learned my addr 172.21.15.189:0/613486982 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.456+0000 7fb7c7fff700 1 --2- 172.21.15.189:0/613486982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7c8159b20 0x7fb7c814d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.456+0000 7fb7c6ffd700 1 --2- 172.21.15.189:0/613486982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7c80cdc60 0x7fb7c815a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.456+0000 7fb7c7fff700 1 -- 172.21.15.189:0/613486982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7c80cdc60 msgr2=0x7fb7c815a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.456+0000 7fb7c7fff700 1 --2- 172.21.15.189:0/613486982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7c80cdc60 0x7fb7c815a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.456+0000 7fb7c7fff700 1 -- 172.21.15.189:0/613486982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7c806eb80 msgr2=0x7fb7c814aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.456+0000 7fb7c7fff700 1 --2- 172.21.15.189:0/613486982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7c806eb80 0x7fb7c814aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.456+0000 7fb7c7fff700 1 -- 172.21.15.189:0/613486982 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b8009a70 con 0x7fb7c8159b20 2024-04-06T13:45:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.457+0000 7fb7c77fe700 1 --2- 172.21.15.189:0/613486982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7c806eb80 0x7fb7c814aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:45:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.457+0000 7fb7c7fff700 1 --2- 172.21.15.189:0/613486982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7c8159b20 0x7fb7c814d240 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000c390 tx=0x7fb7c000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.457+0000 7fb7affff700 1 -- 172.21.15.189:0/613486982 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c00052d0 con 0x7fb7c8159b20 2024-04-06T13:45:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.457+0000 7fb7affff700 1 -- 172.21.15.189:0/613486982 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c0005430 con 0x7fb7c8159b20 2024-04-06T13:45:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.457+0000 7fb7ce042700 1 -- 172.21.15.189:0/613486982 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c80acbe0 con 0x7fb7c8159b20 2024-04-06T13:45:23.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.457+0000 7fb7affff700 1 -- 172.21.15.189:0/613486982 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c00038c0 con 0x7fb7c8159b20 2024-04-06T13:45:23.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.457+0000 7fb7ce042700 1 -- 172.21.15.189:0/613486982 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c815cf80 con 0x7fb7c8159b20 2024-04-06T13:45:23.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.458+0000 7fb7ce042700 1 -- 172.21.15.189:0/613486982 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c8074990 con 0x7fb7c8159b20 2024-04-06T13:45:23.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.458+0000 7fb7affff700 1 -- 172.21.15.189:0/613486982 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 123) v1 ==== 89259+0+0 (secure 0 0 0) 0x7fb7c0003a20 con 0x7fb7c8159b20 2024-04-06T13:45:23.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.459+0000 7fb7affff700 1 --2- 172.21.15.189:0/613486982 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb7b406adb0 0x7fb7b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:23.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.459+0000 7fb7c77fe700 1 -- 172.21.15.189:0/613486982 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb7b406adb0 msgr2=0x7fb7b406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:23.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.459+0000 7fb7affff700 1 -- 172.21.15.189:0/613486982 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6747+0+0 (secure 0 0 0) 0x7fb7c0088aa0 con 0x7fb7c8159b20 2024-04-06T13:45:23.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.459+0000 7fb7c77fe700 1 --2- 172.21.15.189:0/613486982 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb7b406adb0 0x7fb7b406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:45:23.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.463+0000 7fb7affff700 1 -- 172.21.15.189:0/613486982 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb7c0053760 con 0x7fb7c8159b20 2024-04-06T13:45:23.638 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:23.638+0000 7fbe06afc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:45:23.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.660+0000 7fb7c77fe700 1 -- 172.21.15.189:0/613486982 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb7b406adb0 msgr2=0x7fb7b406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:23.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.660+0000 7fb7c77fe700 1 --2- 172.21.15.189:0/613486982 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb7b406adb0 0x7fb7b406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:45:23.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.677+0000 7fb7ce042700 1 -- 172.21.15.189:0/613486982 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7c80aeea0 con 0x7fb7c8159b20 2024-04-06T13:45:23.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.679+0000 7fb7affff700 1 -- 172.21.15.189:0/613486982 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+17381 (secure 0 0 0) 0x7fb7c005d070 con 0x7fb7c8159b20 2024-04-06T13:45:23.680 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:23.680 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":196,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:45:22.587972+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:29:14.625778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-04-06T13:29:15.827532+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":196,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/1090714269":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/3656453012":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/2710277231":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/1953253738":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/1974012472":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/820436148":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/2599831431":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/1401962292":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/2983879481":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/3870180304":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/1073864749":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/3525693740":"2024-04-07T13:29:30.406338+0000","172.21.15.189:6833/47128":"2024-04-07T13:29:30.406338+0000","172.21.15.189:6832/47128":"2024-04-07T13:29:30.406338+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:45:23.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.681+0000 7fb7ce042700 1 -- 172.21.15.189:0/613486982 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb7b406adb0 msgr2=0x7fb7b406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:45:23.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.681+0000 7fb7ce042700 1 --2- 172.21.15.189:0/613486982 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb7b406adb0 0x7fb7b406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.681+0000 7fb7ce042700 1 -- 172.21.15.189:0/613486982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7c8159b20 msgr2=0x7fb7c814d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:23.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.681+0000 7fb7ce042700 1 --2- 172.21.15.189:0/613486982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7c8159b20 0x7fb7c814d240 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000c390 tx=0x7fb7c000c6a0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.681+0000 7fb7ce042700 1 -- 172.21.15.189:0/613486982 shutdown_connections 2024-04-06T13:45:23.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.681+0000 7fb7ce042700 1 --2- 172.21.15.189:0/613486982 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb7b406adb0 0x7fb7b406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.681+0000 7fb7ce042700 1 --2- 172.21.15.189:0/613486982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7c8159b20 0x7fb7c814d240 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.681+0000 7fb7ce042700 1 --2- 172.21.15.189:0/613486982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7c806eb80 0x7fb7c814aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.681+0000 7fb7ce042700 1 --2- 172.21.15.189:0/613486982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7c80cdc60 0x7fb7c815a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.681+0000 7fb7ce042700 1 -- 172.21.15.189:0/613486982 >> 172.21.15.189:0/613486982 conn(0x7fb7c80cc290 msgr2=0x7fb7c806e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:23.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.681+0000 7fb7ce042700 1 -- 172.21.15.189:0/613486982 shutdown_connections 2024-04-06T13:45:23.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.681+0000 7fb7ce042700 1 -- 172.21.15.189:0/613486982 wait complete. 2024-04-06T13:45:23.696 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T13:29:14.625778+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '191', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T13:29:15.827532+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '196', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 196, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:45:23.696 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:45:23.716 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:23.716+0000 7fbe06afc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:45:23.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.824+0000 7f5173b50700 1 Processor -- start 2024-04-06T13:45:23.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.824+0000 7f5173b50700 1 -- start start 2024-04-06T13:45:23.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.825+0000 7f5173b50700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f516c144f40 0x7f516c06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:23.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.825+0000 7f5173b50700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f516c15fcd0 0x7f516c06ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:23.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.825+0000 7f5173b50700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f516c15c740 con 0x7f516c15ba40 2024-04-06T13:45:23.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.825+0000 7f5173b50700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f516c0cfcb0 con 0x7f516c15fcd0 2024-04-06T13:45:23.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.825+0000 7f5173b50700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f516c15ca20 con 0x7f516c144f40 2024-04-06T13:45:23.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.825+0000 7f51710eb700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f516c15fcd0 0x7f516c06ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:23.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.825+0000 7f51720ed700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f516c15ba40 0x7f516c15be60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38442/0 (socket says 172.21.15.189:38442) 2024-04-06T13:45:23.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.825+0000 7f51720ed700 1 -- 172.21.15.189:0/1103170366 learned_addr learned my addr 172.21.15.189:0/1103170366 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:23.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.825+0000 7f51718ec700 1 --2- 172.21.15.189:0/1103170366 >> v2:172.21.15.192:3300/0 conn(0x7f516c144f40 0x7f516c06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:23.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.826+0000 7f51718ec700 1 -- 172.21.15.189:0/1103170366 >> v2:172.21.15.189:3300/0 conn(0x7f516c15fcd0 msgr2=0x7f516c06ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:23.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.826+0000 7f51718ec700 1 --2- 172.21.15.189:0/1103170366 >> v2:172.21.15.189:3300/0 conn(0x7f516c15fcd0 0x7f516c06ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.826+0000 7f51718ec700 1 -- 172.21.15.189:0/1103170366 >> v1:172.21.15.189:6789/0 conn(0x7f516c15ba40 legacy=0x7f516c15be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:23.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.826+0000 7f51718ec700 1 -- 172.21.15.189:0/1103170366 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516c15f1f0 con 0x7f516c144f40 2024-04-06T13:45:23.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.826+0000 7f51708ea700 1 -- 172.21.15.189:0/1103170366 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86006296 0 0) 0x7f516c15c740 con 0x7f516c15ba40 2024-04-06T13:45:23.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.826+0000 7f51710eb700 1 --2- 172.21.15.189:0/1103170366 >> v2:172.21.15.189:3300/0 conn(0x7f516c15fcd0 0x7f516c06ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:45:23.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.826+0000 7f51718ec700 1 --2- 172.21.15.189:0/1103170366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f516c144f40 0x7f516c06a730 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f516c073470 tx=0x7f5168009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=700559431e3c19d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:23.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.826+0000 7f51708ea700 1 -- 172.21.15.189:0/1103170366 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5168011830 con 0x7f516c144f40 2024-04-06T13:45:23.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.826+0000 7f51708ea700 1 -- 172.21.15.189:0/1103170366 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5168011990 con 0x7f516c144f40 2024-04-06T13:45:23.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.826+0000 7f51708ea700 1 -- 172.21.15.189:0/1103170366 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5168011d20 con 0x7f516c144f40 2024-04-06T13:45:23.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.827+0000 7f5173b50700 1 -- 172.21.15.189:0/1103170366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f516c144f40 msgr2=0x7f516c06a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:23.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.827+0000 7f5173b50700 1 --2- 172.21.15.189:0/1103170366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f516c144f40 0x7f516c06a730 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f516c073470 tx=0x7f5168009b20 comp rx=0 tx=0).stop 2024-04-06T13:45:23.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.827+0000 7f5173b50700 1 -- 172.21.15.189:0/1103170366 shutdown_connections 2024-04-06T13:45:23.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.827+0000 7f5173b50700 1 --2- 172.21.15.189:0/1103170366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f516c144f40 0x7f516c06a730 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.827+0000 7f5173b50700 1 --2- 172.21.15.189:0/1103170366 >> v2:172.21.15.189:3300/0 conn(0x7f516c15fcd0 0x7f516c06ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.827+0000 7f5173b50700 1 -- 172.21.15.189:0/1103170366 >> 172.21.15.189:0/1103170366 conn(0x7f516c0d00b0 msgr2=0x7f516c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:23.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.827+0000 7f5173b50700 1 -- 172.21.15.189:0/1103170366 shutdown_connections 2024-04-06T13:45:23.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.827+0000 7f5173b50700 1 -- 172.21.15.189:0/1103170366 wait complete. 2024-04-06T13:45:23.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.828+0000 7f5173b50700 1 Processor -- start 2024-04-06T13:45:23.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.828+0000 7f5173b50700 1 -- start start 2024-04-06T13:45:23.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f5173b50700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f516c144f40 0x7f516c1f0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:23.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f5173b50700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f516c15ba40 0x7f516c15cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:23.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f5173b50700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f516c15fcd0 0x7f516c15d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:23.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f5173b50700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516c0cc220 con 0x7f516c15fcd0 2024-04-06T13:45:23.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f5173b50700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516c0cb610 con 0x7f516c15ba40 2024-04-06T13:45:23.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f5173b50700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516c0cb8f0 con 0x7f516c144f40 2024-04-06T13:45:23.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f51720ed700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f516c15fcd0 0x7f516c15d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:23.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f51720ed700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f516c15fcd0 0x7f516c15d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50650/0 (socket says 172.21.15.189:50650) 2024-04-06T13:45:23.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f51720ed700 1 -- 172.21.15.189:0/2063987849 learned_addr learned my addr 172.21.15.189:0/2063987849 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:23.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f51710eb700 1 --2- 172.21.15.189:0/2063987849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f516c15ba40 0x7f516c15cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:23.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f51718ec700 1 --2- 172.21.15.189:0/2063987849 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f516c144f40 0x7f516c1f0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:23.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f51720ed700 1 -- 172.21.15.189:0/2063987849 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f516c144f40 msgr2=0x7f516c1f0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:23.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f51720ed700 1 --2- 172.21.15.189:0/2063987849 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f516c144f40 0x7f516c1f0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f51720ed700 1 -- 172.21.15.189:0/2063987849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f516c15ba40 msgr2=0x7f516c15cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:23.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f51720ed700 1 --2- 172.21.15.189:0/2063987849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f516c15ba40 0x7f516c15cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:23.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f51720ed700 1 -- 172.21.15.189:0/2063987849 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515c0094a0 con 0x7f516c15fcd0 2024-04-06T13:45:23.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f51710eb700 1 --2- 172.21.15.189:0/2063987849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f516c15ba40 0x7f516c15cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:45:23.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f51720ed700 1 --2- 172.21.15.189:0/2063987849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f516c15fcd0 0x7f516c15d310 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f516400cca0 tx=0x7f516400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:23.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f51627fc700 1 -- 172.21.15.189:0/2063987849 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516400ab70 con 0x7f516c15fcd0 2024-04-06T13:45:23.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f51627fc700 1 -- 172.21.15.189:0/2063987849 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516400acd0 con 0x7f516c15fcd0 2024-04-06T13:45:23.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f5173b50700 1 -- 172.21.15.189:0/2063987849 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51680094a0 con 0x7f516c15fcd0 2024-04-06T13:45:23.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.829+0000 7f51627fc700 1 -- 172.21.15.189:0/2063987849 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5164007310 con 0x7f516c15fcd0 2024-04-06T13:45:23.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.830+0000 7f5173b50700 1 -- 172.21.15.189:0/2063987849 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f516c0cbeb0 con 0x7f516c15fcd0 2024-04-06T13:45:23.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.830+0000 7f5173b50700 1 -- 172.21.15.189:0/2063987849 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f516c067bb0 con 0x7f516c15fcd0 2024-04-06T13:45:23.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.831+0000 7f51627fc700 1 -- 172.21.15.189:0/2063987849 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 123) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f5164007470 con 0x7f516c15fcd0 2024-04-06T13:45:23.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.831+0000 7f51627fc700 1 --2- 172.21.15.189:0/2063987849 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f515006adb0 0x7f515006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:23.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.831+0000 7f51718ec700 1 -- 172.21.15.189:0/2063987849 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f515006adb0 msgr2=0x7f515006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:23.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.831+0000 7f51627fc700 1 -- 172.21.15.189:0/2063987849 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6747+0+0 (secure 0 0 0) 0x7f5164089370 con 0x7f516c15fcd0 2024-04-06T13:45:23.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.831+0000 7f51718ec700 1 --2- 172.21.15.189:0/2063987849 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f515006adb0 0x7f515006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:45:23.835 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:23.834+0000 7f3f48896200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:45:23.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:23.836+0000 7f51627fc700 1 -- 172.21.15.189:0/2063987849 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f51640540c0 con 0x7f516c15fcd0 2024-04-06T13:45:23.841 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:23.842+0000 7fbe06afc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:45:23.917 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:23.916+0000 7f3f48896200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:45:24.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.032+0000 7f51718ec700 1 -- 172.21.15.189:0/2063987849 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f515006adb0 msgr2=0x7f515006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:24.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.032+0000 7f51718ec700 1 --2- 172.21.15.189:0/2063987849 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f515006adb0 0x7f515006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:45:24.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.032+0000 7f5173b50700 1 -- 172.21.15.189:0/2063987849 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f516c074990 con 0x7f516c15fcd0 2024-04-06T13:45:24.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.032+0000 7f51627fc700 1 -- 172.21.15.189:0/2063987849 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v196) v1 ==== 93+0+10 (secure 0 0 0) 0x7f51640180a0 con 0x7f516c15fcd0 2024-04-06T13:45:24.034 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:45:24.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.035+0000 7f5173b50700 1 -- 172.21.15.189:0/2063987849 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f515006adb0 msgr2=0x7f515006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:45:24.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.035+0000 7f5173b50700 1 --2- 172.21.15.189:0/2063987849 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f515006adb0 0x7f515006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.035+0000 7f5173b50700 1 -- 172.21.15.189:0/2063987849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f516c15fcd0 msgr2=0x7f516c15d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:24.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.035+0000 7f5173b50700 1 --2- 172.21.15.189:0/2063987849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f516c15fcd0 0x7f516c15d310 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f516400cca0 tx=0x7f516400a060 comp rx=0 tx=0).stop 2024-04-06T13:45:24.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.035+0000 7f5173b50700 1 -- 172.21.15.189:0/2063987849 shutdown_connections 2024-04-06T13:45:24.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.035+0000 7f5173b50700 1 --2- 172.21.15.189:0/2063987849 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f515006adb0 0x7f515006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.035+0000 7f5173b50700 1 --2- 172.21.15.189:0/2063987849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f516c15fcd0 0x7f516c15d310 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.035+0000 7f5173b50700 1 --2- 172.21.15.189:0/2063987849 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f516c144f40 0x7f516c1f0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.035+0000 7f5173b50700 1 --2- 172.21.15.189:0/2063987849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f516c15ba40 0x7f516c15cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.035+0000 7f5173b50700 1 -- 172.21.15.189:0/2063987849 >> 172.21.15.189:0/2063987849 conn(0x7f516c0d00b0 msgr2=0x7f516c06aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:24.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.035+0000 7f5173b50700 1 -- 172.21.15.189:0/2063987849 shutdown_connections 2024-04-06T13:45:24.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.035+0000 7f5173b50700 1 -- 172.21.15.189:0/2063987849 wait complete. 2024-04-06T13:45:24.047 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:24.046+0000 7f3f48896200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:45:24.048 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T13:45:24.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.178+0000 7f8a40891700 1 Processor -- start 2024-04-06T13:45:24.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.178+0000 7f8a40891700 1 -- start start 2024-04-06T13:45:24.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.179+0000 7f8a40891700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8a3c0d11b0 0x7f8a3c0d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:24.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.179+0000 7f8a40891700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a3c0cfa00 0x7f8a3c0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:24.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.179+0000 7f8a40891700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a3c0d1ad0 con 0x7f8a3c0cb870 2024-04-06T13:45:24.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.179+0000 7f8a40891700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a3c0ccdc0 con 0x7f8a3c0d11b0 2024-04-06T13:45:24.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.179+0000 7f8a40891700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c0d0b50 con 0x7f8a3c0cfa00 2024-04-06T13:45:24.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.179+0000 7f8a3a59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8a3c0d11b0 0x7f8a3c0d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:24.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.179+0000 7f8a3a59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8a3c0d11b0 0x7f8a3c0d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50662/0 (socket says 172.21.15.189:50662) 2024-04-06T13:45:24.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.179+0000 7f8a3a59c700 1 -- 172.21.15.189:0/326894496 learned_addr learned my addr 172.21.15.189:0/326894496 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:24.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.179+0000 7f8a3ad9d700 1 --2- 172.21.15.189:0/326894496 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a3c0cfa00 0x7f8a3c0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:24.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.180+0000 7f8a3a59c700 1 -- 172.21.15.189:0/326894496 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a3c0cfa00 msgr2=0x7f8a3c0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:24.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.180+0000 7f8a3a59c700 1 --2- 172.21.15.189:0/326894496 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a3c0cfa00 0x7f8a3c0cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.180+0000 7f8a3a59c700 1 -- 172.21.15.189:0/326894496 >> v1:172.21.15.192:6789/0 conn(0x7f8a3c0cb870 legacy=0x7f8a3c15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:24.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.180+0000 7f8a3a59c700 1 -- 172.21.15.189:0/326894496 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c0cbd80 con 0x7f8a3c0d11b0 2024-04-06T13:45:24.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.180+0000 7f8a3a59c700 1 --2- 172.21.15.189:0/326894496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a3c0d11b0 0x7f8a3c0d1590 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c0098f0 tx=0x7f8a2c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=619cf98d0b14f621 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:24.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.180+0000 7f8a3959a700 1 -- 172.21.15.189:0/326894496 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c011830 con 0x7f8a3c0d11b0 2024-04-06T13:45:24.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.180+0000 7f8a3959a700 1 -- 172.21.15.189:0/326894496 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2c011990 con 0x7f8a3c0d11b0 2024-04-06T13:45:24.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.180+0000 7f8a3959a700 1 -- 172.21.15.189:0/326894496 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c011d20 con 0x7f8a3c0d11b0 2024-04-06T13:45:24.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.181+0000 7f8a40891700 1 -- 172.21.15.189:0/326894496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a3c0d11b0 msgr2=0x7f8a3c0d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:24.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.181+0000 7f8a40891700 1 --2- 172.21.15.189:0/326894496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a3c0d11b0 0x7f8a3c0d1590 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c0098f0 tx=0x7f8a2c009c00 comp rx=0 tx=0).stop 2024-04-06T13:45:24.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.181+0000 7f8a40891700 1 -- 172.21.15.189:0/326894496 shutdown_connections 2024-04-06T13:45:24.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.181+0000 7f8a40891700 1 --2- 172.21.15.189:0/326894496 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a3c0cfa00 0x7f8a3c0cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.181+0000 7f8a40891700 1 --2- 172.21.15.189:0/326894496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a3c0d11b0 0x7f8a3c0d1590 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.181+0000 7f8a40891700 1 -- 172.21.15.189:0/326894496 >> 172.21.15.189:0/326894496 conn(0x7f8a3c0b2db0 msgr2=0x7f8a3c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:24.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.181+0000 7f8a40891700 1 -- 172.21.15.189:0/326894496 shutdown_connections 2024-04-06T13:45:24.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.181+0000 7f8a40891700 1 -- 172.21.15.189:0/326894496 wait complete. 2024-04-06T13:45:24.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.182+0000 7f8a40891700 1 Processor -- start 2024-04-06T13:45:24.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.182+0000 7f8a40891700 1 -- start start 2024-04-06T13:45:24.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.183+0000 7f8a40891700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a3c0cb870 0x7f8a3c1f03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:24.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.183+0000 7f8a40891700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a3c0cfa00 0x7f8a3c158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:24.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.183+0000 7f8a40891700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a3c0d11b0 0x7f8a3c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:24.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.183+0000 7f8a40891700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c1447c0 con 0x7f8a3c0cb870 2024-04-06T13:45:24.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.183+0000 7f8a40891700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c158eb0 con 0x7f8a3c0d11b0 2024-04-06T13:45:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.183+0000 7f8a40891700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c159190 con 0x7f8a3c0cfa00 2024-04-06T13:45:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.183+0000 7f8a3a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a3c0cb870 0x7f8a3c1f03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.183+0000 7f8a3a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a3c0cb870 0x7f8a3c1f03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50664/0 (socket says 172.21.15.189:50664) 2024-04-06T13:45:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.183+0000 7f8a3a59c700 1 -- 172.21.15.189:0/3207543702 learned_addr learned my addr 172.21.15.189:0/3207543702 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.183+0000 7f8a39d9b700 1 --2- 172.21.15.189:0/3207543702 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a3c0cfa00 0x7f8a3c158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.183+0000 7f8a3ad9d700 1 --2- 172.21.15.189:0/3207543702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a3c0d11b0 0x7f8a3c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.183+0000 7f8a3a59c700 1 -- 172.21.15.189:0/3207543702 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a3c0cfa00 msgr2=0x7f8a3c158460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.183+0000 7f8a3a59c700 1 --2- 172.21.15.189:0/3207543702 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a3c0cfa00 0x7f8a3c158460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.183+0000 7f8a3a59c700 1 -- 172.21.15.189:0/3207543702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a3c0d11b0 msgr2=0x7f8a3c15e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.184+0000 7f8a3a59c700 1 --2- 172.21.15.189:0/3207543702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a3c0d11b0 0x7f8a3c15e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.184+0000 7f8a3a59c700 1 -- 172.21.15.189:0/3207543702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a2c009550 con 0x7f8a3c0cb870 2024-04-06T13:45:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.184+0000 7f8a3a59c700 1 --2- 172.21.15.189:0/3207543702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a3c0cb870 0x7f8a3c1f03f0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c009c40 tx=0x7f8a2c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.184+0000 7f8a2affd700 1 -- 172.21.15.189:0/3207543702 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c004340 con 0x7f8a3c0cb870 2024-04-06T13:45:24.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.184+0000 7f8a2affd700 1 -- 172.21.15.189:0/3207543702 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2c01f070 con 0x7f8a3c0cb870 2024-04-06T13:45:24.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.184+0000 7f8a40891700 1 -- 172.21.15.189:0/3207543702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a3c0c93e0 con 0x7f8a3c0cb870 2024-04-06T13:45:24.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.184+0000 7f8a2affd700 1 -- 172.21.15.189:0/3207543702 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c0036a0 con 0x7f8a3c0cb870 2024-04-06T13:45:24.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.184+0000 7f8a40891700 1 -- 172.21.15.189:0/3207543702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a3c0c99a0 con 0x7f8a3c0cb870 2024-04-06T13:45:24.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.185+0000 7f8a40891700 1 -- 172.21.15.189:0/3207543702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a3c074990 con 0x7f8a3c0cb870 2024-04-06T13:45:24.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.186+0000 7f8a2affd700 1 -- 172.21.15.189:0/3207543702 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 123) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f8a2c021030 con 0x7f8a3c0cb870 2024-04-06T13:45:24.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.187+0000 7f8a2affd700 1 --2- 172.21.15.189:0/3207543702 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8a1c06adb0 0x7f8a1c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:24.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.187+0000 7f8a39d9b700 1 -- 172.21.15.189:0/3207543702 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8a1c06adb0 msgr2=0x7f8a1c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:24.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.187+0000 7f8a2affd700 1 -- 172.21.15.189:0/3207543702 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6747+0+0 (secure 0 0 0) 0x7f8a2c08b160 con 0x7f8a3c0cb870 2024-04-06T13:45:24.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.187+0000 7f8a39d9b700 1 --2- 172.21.15.189:0/3207543702 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8a1c06adb0 0x7f8a1c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:45:24.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.190+0000 7f8a2affd700 1 -- 172.21.15.189:0/3207543702 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8a2c055d90 con 0x7f8a3c0cb870 2024-04-06T13:45:24.351 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:24.351+0000 7fbe06afc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:45:24.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.388+0000 7f8a39d9b700 1 -- 172.21.15.189:0/3207543702 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8a1c06adb0 msgr2=0x7f8a1c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:24.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.388+0000 7f8a39d9b700 1 --2- 172.21.15.189:0/3207543702 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8a1c06adb0 0x7f8a1c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:45:24.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.390+0000 7f8a40891700 1 -- 172.21.15.189:0/3207543702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8a3c0cb630 con 0x7f8a3c0cb870 2024-04-06T13:45:24.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.391+0000 7f8a2affd700 1 -- 172.21.15.189:0/3207543702 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v196) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8a2c0593b0 con 0x7f8a3c0cb870 2024-04-06T13:45:24.392 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:45:24.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.393+0000 7f8a40891700 1 -- 172.21.15.189:0/3207543702 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8a1c06adb0 msgr2=0x7f8a1c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:45:24.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.393+0000 7f8a40891700 1 --2- 172.21.15.189:0/3207543702 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8a1c06adb0 0x7f8a1c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.393+0000 7f8a40891700 1 -- 172.21.15.189:0/3207543702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a3c0cb870 msgr2=0x7f8a3c1f03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:24.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.393+0000 7f8a40891700 1 --2- 172.21.15.189:0/3207543702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a3c0cb870 0x7f8a3c1f03f0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c009c40 tx=0x7f8a2c004750 comp rx=0 tx=0).stop 2024-04-06T13:45:24.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.393+0000 7f8a40891700 1 -- 172.21.15.189:0/3207543702 shutdown_connections 2024-04-06T13:45:24.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.393+0000 7f8a40891700 1 --2- 172.21.15.189:0/3207543702 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8a1c06adb0 0x7f8a1c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.393+0000 7f8a40891700 1 --2- 172.21.15.189:0/3207543702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a3c0d11b0 0x7f8a3c15e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.393+0000 7f8a40891700 1 --2- 172.21.15.189:0/3207543702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a3c0cb870 0x7f8a3c1f03f0 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.393+0000 7f8a40891700 1 --2- 172.21.15.189:0/3207543702 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a3c0cfa00 0x7f8a3c158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.393+0000 7f8a40891700 1 -- 172.21.15.189:0/3207543702 >> 172.21.15.189:0/3207543702 conn(0x7f8a3c0b2db0 msgr2=0x7f8a3c0b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:24.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.393+0000 7f8a40891700 1 -- 172.21.15.189:0/3207543702 shutdown_connections 2024-04-06T13:45:24.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.393+0000 7f8a40891700 1 -- 172.21.15.189:0/3207543702 wait complete. 2024-04-06T13:45:24.407 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T13:45:24.505 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:24.504+0000 7fbe06afc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:45:24.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.537+0000 7f1efab50700 1 Processor -- start 2024-04-06T13:45:24.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.537+0000 7f1efab50700 1 -- start start 2024-04-06T13:45:24.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.538+0000 7f1efab50700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ef40d17a0 0x7f1ef406b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:24.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.538+0000 7f1efab50700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1ef40cedf0 0x7f1ef40cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:24.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.538+0000 7f1efab50700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef40cf760 con 0x7f1ef4146dd0 2024-04-06T13:45:24.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.538+0000 7f1efab50700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ef406d850 con 0x7f1ef40cedf0 2024-04-06T13:45:24.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.538+0000 7f1efab50700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef406c0a0 con 0x7f1ef40d17a0 2024-04-06T13:45:24.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.538+0000 7f1ef88ec700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ef40d17a0 0x7f1ef406b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:24.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.538+0000 7f1ef90ed700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1ef40cedf0 0x7f1ef40cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:24.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.538+0000 7f1ef3fff700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f1ef4146dd0 0x7f1ef406d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:56770/0 (socket says 172.21.15.189:56770) 2024-04-06T13:45:24.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.538+0000 7f1ef3fff700 1 -- 172.21.15.189:0/2817479276 learned_addr learned my addr 172.21.15.189:0/2817479276 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:24.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.538+0000 7f1ef37fe700 1 -- 172.21.15.189:0/2817479276 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808946592 0 0) 0x7f1ef40cf760 con 0x7f1ef4146dd0 2024-04-06T13:45:24.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.538+0000 7f1ef37fe700 1 -- 172.21.15.189:0/2817479276 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ee40035b0 con 0x7f1ef4146dd0 2024-04-06T13:45:24.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.538+0000 7f1ef90ed700 1 -- 172.21.15.189:0/2817479276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ef40d17a0 msgr2=0x7f1ef406b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:24.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.538+0000 7f1ef90ed700 1 --2- 172.21.15.189:0/2817479276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ef40d17a0 0x7f1ef406b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.538+0000 7f1ef90ed700 1 -- 172.21.15.189:0/2817479276 >> v1:172.21.15.192:6789/0 conn(0x7f1ef4146dd0 legacy=0x7f1ef406d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:24.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.538+0000 7f1ef90ed700 1 -- 172.21.15.189:0/2817479276 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef406c380 con 0x7f1ef40cedf0 2024-04-06T13:45:24.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.538+0000 7f1ef88ec700 1 --2- 172.21.15.189:0/2817479276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ef40d17a0 0x7f1ef406b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:45:24.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.539+0000 7f1ef90ed700 1 --2- 172.21.15.189:0/2817479276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ef40cedf0 0x7f1ef40cf220 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f1eec009d90 tx=0x7f1eec00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d12c82ea3df6111 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:24.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.539+0000 7f1ef37fe700 1 -- 172.21.15.189:0/2817479276 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eec00e070 con 0x7f1ef40cedf0 2024-04-06T13:45:24.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.539+0000 7f1ef37fe700 1 -- 172.21.15.189:0/2817479276 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eec00bb20 con 0x7f1ef40cedf0 2024-04-06T13:45:24.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.539+0000 7f1ef37fe700 1 -- 172.21.15.189:0/2817479276 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eec00be90 con 0x7f1ef40cedf0 2024-04-06T13:45:24.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.539+0000 7f1efab50700 1 -- 172.21.15.189:0/2817479276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ef40cedf0 msgr2=0x7f1ef40cf220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:24.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.539+0000 7f1efab50700 1 --2- 172.21.15.189:0/2817479276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ef40cedf0 0x7f1ef40cf220 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f1eec009d90 tx=0x7f1eec00b330 comp rx=0 tx=0).stop 2024-04-06T13:45:24.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.539+0000 7f1efab50700 1 -- 172.21.15.189:0/2817479276 shutdown_connections 2024-04-06T13:45:24.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.539+0000 7f1efab50700 1 --2- 172.21.15.189:0/2817479276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ef40cedf0 0x7f1ef40cf220 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.539+0000 7f1efab50700 1 --2- 172.21.15.189:0/2817479276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ef40d17a0 0x7f1ef406b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.539+0000 7f1efab50700 1 -- 172.21.15.189:0/2817479276 >> 172.21.15.189:0/2817479276 conn(0x7f1ef40b2db0 msgr2=0x7f1ef40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:24.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.539+0000 7f1efab50700 1 -- 172.21.15.189:0/2817479276 shutdown_connections 2024-04-06T13:45:24.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.539+0000 7f1efab50700 1 -- 172.21.15.189:0/2817479276 wait complete. 2024-04-06T13:45:24.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.541+0000 7f1efab50700 1 Processor -- start 2024-04-06T13:45:24.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.541+0000 7f1efab50700 1 -- start start 2024-04-06T13:45:24.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.542+0000 7f1efab50700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ef40cedf0 0x7f1ef40ad350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:24.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.542+0000 7f1efab50700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ef40d17a0 0x7f1ef40ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:24.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.542+0000 7f1efab50700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ef4146dd0 0x7f1ef40c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:24.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.542+0000 7f1efab50700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef415c820 con 0x7f1ef40cedf0 2024-04-06T13:45:24.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.542+0000 7f1efab50700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef40af160 con 0x7f1ef40d17a0 2024-04-06T13:45:24.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.542+0000 7f1efab50700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef40addd0 con 0x7f1ef4146dd0 2024-04-06T13:45:24.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.542+0000 7f1ef88ec700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ef40cedf0 0x7f1ef40ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:24.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.542+0000 7f1ef88ec700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ef40cedf0 0x7f1ef40ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50672/0 (socket says 172.21.15.189:50672) 2024-04-06T13:45:24.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.542+0000 7f1ef88ec700 1 -- 172.21.15.189:0/209138408 learned_addr learned my addr 172.21.15.189:0/209138408 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:24.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.542+0000 7f1ef3fff700 1 --2- 172.21.15.189:0/209138408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ef40d17a0 0x7f1ef40ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:24.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.542+0000 7f1ef90ed700 1 --2- 172.21.15.189:0/209138408 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ef4146dd0 0x7f1ef40c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:24.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.543+0000 7f1ef88ec700 1 -- 172.21.15.189:0/209138408 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ef4146dd0 msgr2=0x7f1ef40c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:24.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.543+0000 7f1ef88ec700 1 --2- 172.21.15.189:0/209138408 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ef4146dd0 0x7f1ef40c82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.543+0000 7f1ef88ec700 1 -- 172.21.15.189:0/209138408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ef40d17a0 msgr2=0x7f1ef40ad890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:24.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.543+0000 7f1ef88ec700 1 --2- 172.21.15.189:0/209138408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ef40d17a0 0x7f1ef40ad890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.543+0000 7f1ef88ec700 1 -- 172.21.15.189:0/209138408 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee80094a0 con 0x7f1ef40cedf0 2024-04-06T13:45:24.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.543+0000 7f1ef88ec700 1 --2- 172.21.15.189:0/209138408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ef40cedf0 0x7f1ef40ad350 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f1ee800ebe0 tx=0x7f1ee800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:24.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.543+0000 7f1ef17fa700 1 -- 172.21.15.189:0/209138408 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee8009760 con 0x7f1ef40cedf0 2024-04-06T13:45:24.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.543+0000 7f1ef17fa700 1 -- 172.21.15.189:0/209138408 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee800c850 con 0x7f1ef40cedf0 2024-04-06T13:45:24.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.543+0000 7f1efab50700 1 -- 172.21.15.189:0/209138408 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eec009a70 con 0x7f1ef40cedf0 2024-04-06T13:45:24.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.543+0000 7f1ef17fa700 1 -- 172.21.15.189:0/209138408 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee800cbe0 con 0x7f1ef40cedf0 2024-04-06T13:45:24.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.543+0000 7f1efab50700 1 -- 172.21.15.189:0/209138408 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef40ae360 con 0x7f1ef40cedf0 2024-04-06T13:45:24.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.544+0000 7f1efab50700 1 -- 172.21.15.189:0/209138408 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef40af440 con 0x7f1ef40cedf0 2024-04-06T13:45:24.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.544+0000 7f1ef17fa700 1 -- 172.21.15.189:0/209138408 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 123) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f1ee800cd40 con 0x7f1ef40cedf0 2024-04-06T13:45:24.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.545+0000 7f1ef17fa700 1 --2- 172.21.15.189:0/209138408 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ed806adb0 0x7f1ed806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:24.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.545+0000 7f1ef3fff700 1 -- 172.21.15.189:0/209138408 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ed806adb0 msgr2=0x7f1ed806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:24.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.545+0000 7f1ef17fa700 1 -- 172.21.15.189:0/209138408 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6747+0+0 (secure 0 0 0) 0x7f1ee8089950 con 0x7f1ef40cedf0 2024-04-06T13:45:24.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.545+0000 7f1ef3fff700 1 --2- 172.21.15.189:0/209138408 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ed806adb0 0x7f1ed806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:45:24.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.549+0000 7f1ef17fa700 1 -- 172.21.15.189:0/209138408 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1ee8054570 con 0x7f1ef40cedf0 2024-04-06T13:45:24.598 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:24.597+0000 7f3f48896200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:45:24.660 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:24.660+0000 7fbe06afc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:45:24.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.746+0000 7f1ef3fff700 1 -- 172.21.15.189:0/209138408 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ed806adb0 msgr2=0x7f1ed806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:24.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.746+0000 7f1ef3fff700 1 --2- 172.21.15.189:0/209138408 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ed806adb0 0x7f1ed806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:45:24.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.752+0000 7f1efab50700 1 -- 172.21.15.189:0/209138408 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1ef4074990 con 0x7f1ef40cedf0 2024-04-06T13:45:24.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.752+0000 7f1ef17fa700 1 -- 172.21.15.189:0/209138408 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v196) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1ee8057b90 con 0x7f1ef40cedf0 2024-04-06T13:45:24.753 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:45:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.754+0000 7f1efab50700 1 -- 172.21.15.189:0/209138408 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ed806adb0 msgr2=0x7f1ed806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:45:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.754+0000 7f1efab50700 1 --2- 172.21.15.189:0/209138408 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ed806adb0 0x7f1ed806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.754+0000 7f1efab50700 1 -- 172.21.15.189:0/209138408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ef40cedf0 msgr2=0x7f1ef40ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.754+0000 7f1efab50700 1 --2- 172.21.15.189:0/209138408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ef40cedf0 0x7f1ef40ad350 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f1ee800ebe0 tx=0x7f1ee800eef0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.754+0000 7f1efab50700 1 -- 172.21.15.189:0/209138408 shutdown_connections 2024-04-06T13:45:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.754+0000 7f1efab50700 1 --2- 172.21.15.189:0/209138408 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ed806adb0 0x7f1ed806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.754+0000 7f1efab50700 1 --2- 172.21.15.189:0/209138408 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ef4146dd0 0x7f1ef40c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.754+0000 7f1efab50700 1 --2- 172.21.15.189:0/209138408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ef40cedf0 0x7f1ef40ad350 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.754+0000 7f1efab50700 1 --2- 172.21.15.189:0/209138408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ef40d17a0 0x7f1ef40ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.754+0000 7f1efab50700 1 -- 172.21.15.189:0/209138408 >> 172.21.15.189:0/209138408 conn(0x7f1ef40b2db0 msgr2=0x7f1ef41607c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.755+0000 7f1efab50700 1 -- 172.21.15.189:0/209138408 shutdown_connections 2024-04-06T13:45:24.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.755+0000 7f1efab50700 1 -- 172.21.15.189:0/209138408 wait complete. 2024-04-06T13:45:24.762 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:24.761+0000 7f3f48896200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:45:24.768 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:45:24.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.892+0000 7f537ef51700 1 Processor -- start 2024-04-06T13:45:24.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.892+0000 7f537ef51700 1 -- start start 2024-04-06T13:45:24.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.893+0000 7f537ef51700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f537815d7e0 0x7f53780d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:24.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.893+0000 7f537ef51700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f53780d0890 0x7f53780d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:24.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.893+0000 7f537ef51700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53780cdf80 con 0x7f53780d1af0 2024-04-06T13:45:24.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.893+0000 7f537ef51700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53780d1f80 con 0x7f53780d0890 2024-04-06T13:45:24.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.893+0000 7f537ef51700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53780d3230 con 0x7f537815d7e0 2024-04-06T13:45:24.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.893+0000 7f5377fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f53780d0890 0x7f53780d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:24.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.893+0000 7f537d4ee700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f53780d1af0 0x7f53780b4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38452/0 (socket says 172.21.15.189:38452) 2024-04-06T13:45:24.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.893+0000 7f537d4ee700 1 -- 172.21.15.189:0/3693894053 learned_addr learned my addr 172.21.15.189:0/3693894053 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:24.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.893+0000 7f5377fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f53780d0890 0x7f53780d15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50684/0 (socket says 172.21.15.189:50684) 2024-04-06T13:45:24.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.893+0000 7f537cced700 1 --2- 172.21.15.189:0/3693894053 >> v2:172.21.15.192:3300/0 conn(0x7f537815d7e0 0x7f53780d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:24.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.893+0000 7f5377fff700 1 -- 172.21.15.189:0/3693894053 >> v2:172.21.15.192:3300/0 conn(0x7f537815d7e0 msgr2=0x7f53780d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:24.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.893+0000 7f5377fff700 1 --2- 172.21.15.189:0/3693894053 >> v2:172.21.15.192:3300/0 conn(0x7f537815d7e0 0x7f53780d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.893+0000 7f5377fff700 1 -- 172.21.15.189:0/3693894053 >> v1:172.21.15.189:6789/0 conn(0x7f53780d1af0 legacy=0x7f53780b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:24.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.893+0000 7f5377fff700 1 -- 172.21.15.189:0/3693894053 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53780d0cf0 con 0x7f53780d0890 2024-04-06T13:45:24.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.893+0000 7f53777fe700 1 -- 172.21.15.189:0/3693894053 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10370953 0 0) 0x7f53780cdf80 con 0x7f53780d1af0 2024-04-06T13:45:24.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.894+0000 7f5377fff700 1 --2- 172.21.15.189:0/3693894053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53780d0890 0x7f53780d15b0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f537808c190 tx=0x7f5364009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ee243e4765e38c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:24.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.894+0000 7f53777fe700 1 -- 172.21.15.189:0/3693894053 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5364011830 con 0x7f53780d0890 2024-04-06T13:45:24.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.894+0000 7f53777fe700 1 -- 172.21.15.189:0/3693894053 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5364011990 con 0x7f53780d0890 2024-04-06T13:45:24.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.894+0000 7f53777fe700 1 -- 172.21.15.189:0/3693894053 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5364011d20 con 0x7f53780d0890 2024-04-06T13:45:24.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.894+0000 7f537ef51700 1 -- 172.21.15.189:0/3693894053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53780d0890 msgr2=0x7f53780d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:24.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.894+0000 7f537ef51700 1 --2- 172.21.15.189:0/3693894053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53780d0890 0x7f53780d15b0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f537808c190 tx=0x7f5364009b20 comp rx=0 tx=0).stop 2024-04-06T13:45:24.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.894+0000 7f537ef51700 1 -- 172.21.15.189:0/3693894053 shutdown_connections 2024-04-06T13:45:24.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.894+0000 7f537ef51700 1 --2- 172.21.15.189:0/3693894053 >> v2:172.21.15.192:3300/0 conn(0x7f537815d7e0 0x7f53780d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.894+0000 7f537ef51700 1 --2- 172.21.15.189:0/3693894053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53780d0890 0x7f53780d15b0 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.894+0000 7f537ef51700 1 -- 172.21.15.189:0/3693894053 >> 172.21.15.189:0/3693894053 conn(0x7f5378072250 msgr2=0x7f5378072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:24.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.894+0000 7f537ef51700 1 -- 172.21.15.189:0/3693894053 shutdown_connections 2024-04-06T13:45:24.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.894+0000 7f537ef51700 1 -- 172.21.15.189:0/3693894053 wait complete. 2024-04-06T13:45:24.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.896+0000 7f537ef51700 1 Processor -- start 2024-04-06T13:45:24.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.896+0000 7f537ef51700 1 -- start start 2024-04-06T13:45:24.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.897+0000 7f537ef51700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53780d0890 0x7f53780b4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:24.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.897+0000 7f537ef51700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53780d1af0 0x7f537814daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:24.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.897+0000 7f537ef51700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f537815d7e0 0x7f53780b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:24.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.897+0000 7f537ef51700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5378157e90 con 0x7f53780d0890 2024-04-06T13:45:24.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.897+0000 7f537ef51700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5378158730 con 0x7f53780d1af0 2024-04-06T13:45:24.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.897+0000 7f537ef51700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53780b55f0 con 0x7f537815d7e0 2024-04-06T13:45:24.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.897+0000 7f537cced700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53780d0890 0x7f53780b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:24.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.897+0000 7f537cced700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53780d0890 0x7f53780b4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50698/0 (socket says 172.21.15.189:50698) 2024-04-06T13:45:24.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.897+0000 7f537cced700 1 -- 172.21.15.189:0/3875276359 learned_addr learned my addr 172.21.15.189:0/3875276359 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:24.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.897+0000 7f5377fff700 1 --2- 172.21.15.189:0/3875276359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53780d1af0 0x7f537814daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:24.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.897+0000 7f537d4ee700 1 --2- 172.21.15.189:0/3875276359 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f537815d7e0 0x7f53780b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:24.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.897+0000 7f537cced700 1 -- 172.21.15.189:0/3875276359 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f537815d7e0 msgr2=0x7f53780b50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:24.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.897+0000 7f537cced700 1 --2- 172.21.15.189:0/3875276359 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f537815d7e0 0x7f53780b50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.897+0000 7f537cced700 1 -- 172.21.15.189:0/3875276359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53780d1af0 msgr2=0x7f537814daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.897+0000 7f537cced700 1 --2- 172.21.15.189:0/3875276359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53780d1af0 0x7f537814daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.897+0000 7f537cced700 1 -- 172.21.15.189:0/3875276359 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53640094a0 con 0x7f53780d0890 2024-04-06T13:45:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.898+0000 7f537cced700 1 --2- 172.21.15.189:0/3875276359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53780d0890 0x7f53780b4740 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f536c00e9b0 tx=0x7f536c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.898+0000 7f53757fa700 1 -- 172.21.15.189:0/3875276359 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c009710 con 0x7f53780d0890 2024-04-06T13:45:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.898+0000 7f53757fa700 1 -- 172.21.15.189:0/3875276359 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536c00ee50 con 0x7f53780d0890 2024-04-06T13:45:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.898+0000 7f53757fa700 1 -- 172.21.15.189:0/3875276359 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c00ca50 con 0x7f53780d0890 2024-04-06T13:45:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.898+0000 7f537ef51700 1 -- 172.21.15.189:0/3875276359 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53780b4d20 con 0x7f53780d0890 2024-04-06T13:45:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.898+0000 7f537ef51700 1 -- 172.21.15.189:0/3875276359 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53781583c0 con 0x7f53780d0890 2024-04-06T13:45:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.899+0000 7f537ef51700 1 -- 172.21.15.189:0/3875276359 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5378158a10 con 0x7f53780d0890 2024-04-06T13:45:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.899+0000 7f53757fa700 1 -- 172.21.15.189:0/3875276359 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 123) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f536c00cbb0 con 0x7f53780d0890 2024-04-06T13:45:24.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.900+0000 7f53757fa700 1 --2- 172.21.15.189:0/3875276359 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f535c06adb0 0x7f535c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:24.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.900+0000 7f5377fff700 1 -- 172.21.15.189:0/3875276359 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f535c06adb0 msgr2=0x7f535c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:24.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.900+0000 7f53757fa700 1 -- 172.21.15.189:0/3875276359 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6747+0+0 (secure 0 0 0) 0x7f536c089900 con 0x7f53780d0890 2024-04-06T13:45:24.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.900+0000 7f5377fff700 1 --2- 172.21.15.189:0/3875276359 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f535c06adb0 0x7f535c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:45:24.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:24.907+0000 7f53757fa700 1 -- 172.21.15.189:0/3875276359 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f536c054530 con 0x7f53780d0890 2024-04-06T13:45:24.936 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:24.936+0000 7f3f48896200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:45:25.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.101+0000 7f5377fff700 1 -- 172.21.15.189:0/3875276359 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f535c06adb0 msgr2=0x7f535c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:25.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.101+0000 7f5377fff700 1 --2- 172.21.15.189:0/3875276359 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f535c06adb0 0x7f535c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:45:25.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.158+0000 7f537ef51700 1 -- 172.21.15.189:0/3875276359 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5378074990 con 0x7f53780d0890 2024-04-06T13:45:25.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.158+0000 7f53757fa700 1 -- 172.21.15.189:0/3875276359 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+4749 (secure 0 0 0) 0x7f536c00ce60 con 0x7f53780d0890 2024-04-06T13:45:25.160 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:25.160 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":85,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8141,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2979280081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2979280081},{"type":"v1","addr":"172.21.15.192:6835","nonce":2979280081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":10125,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/286991908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":286991908},{"type":"v1","addr":"172.21.15.189:6836","nonce":286991908}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":10128,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/704323171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":704323171},{"type":"v1","addr":"172.21.15.189:6837","nonce":704323171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:29:16.858698+0000","modified":"2024-04-06T13:29:19.868525+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10137":{"gid":10137,"name":"b","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6834/2662267274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2662267274},{"type":"v1","addr":"172.21.15.192:6834","nonce":2662267274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-04-06T13:45:25.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.161+0000 7f537ef51700 1 -- 172.21.15.189:0/3875276359 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f535c06adb0 msgr2=0x7f535c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:45:25.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.161+0000 7f537ef51700 1 --2- 172.21.15.189:0/3875276359 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f535c06adb0 0x7f535c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.161+0000 7f537ef51700 1 -- 172.21.15.189:0/3875276359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53780d0890 msgr2=0x7f53780b4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:25.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.161+0000 7f537ef51700 1 --2- 172.21.15.189:0/3875276359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53780d0890 0x7f53780b4740 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f536c00e9b0 tx=0x7f536c00ecc0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.161+0000 7f537ef51700 1 -- 172.21.15.189:0/3875276359 shutdown_connections 2024-04-06T13:45:25.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.161+0000 7f537ef51700 1 --2- 172.21.15.189:0/3875276359 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f535c06adb0 0x7f535c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.161+0000 7f537ef51700 1 --2- 172.21.15.189:0/3875276359 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f537815d7e0 0x7f53780b50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.161+0000 7f537ef51700 1 --2- 172.21.15.189:0/3875276359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53780d0890 0x7f53780b4740 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.161+0000 7f537ef51700 1 --2- 172.21.15.189:0/3875276359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53780d1af0 0x7f537814daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.161+0000 7f537ef51700 1 -- 172.21.15.189:0/3875276359 >> 172.21.15.189:0/3875276359 conn(0x7f5378072250 msgr2=0x7f53780b66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.161+0000 7f537ef51700 1 -- 172.21.15.189:0/3875276359 shutdown_connections 2024-04-06T13:45:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.161+0000 7f537ef51700 1 -- 172.21.15.189:0/3875276359 wait complete. 2024-04-06T13:45:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 85 2024-04-06T13:45:25.175 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:45:25.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.299+0000 7f65f96f5700 1 Processor -- start 2024-04-06T13:45:25.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.299+0000 7f65f96f5700 1 -- start start 2024-04-06T13:45:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.299+0000 7f65f96f5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f65f415d590 0x7f65f40b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.299+0000 7f65f96f5700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f65f40d3890 0x7f65f40b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.299+0000 7f65f96f5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f40b5cc0 con 0x7f65f40b4ed0 2024-04-06T13:45:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.299+0000 7f65f96f5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65f40b58c0 con 0x7f65f415d590 2024-04-06T13:45:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.299+0000 7f65f96f5700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65f40d1500 con 0x7f65f40d3890 2024-04-06T13:45:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.300+0000 7f65f2ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f65f415d590 0x7f65f40b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.300+0000 7f65f27fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f65f40b4ed0 0x7f65f40b6580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38464/0 (socket says 172.21.15.189:38464) 2024-04-06T13:45:25.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.300+0000 7f65f27fc700 1 -- 172.21.15.189:0/1456364832 learned_addr learned my addr 172.21.15.189:0/1456364832 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.300+0000 7f65f37fe700 1 --2- 172.21.15.189:0/1456364832 >> v2:172.21.15.192:3300/0 conn(0x7f65f40d3890 0x7f65f40b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.300+0000 7f65f2ffd700 1 -- 172.21.15.189:0/1456364832 >> v2:172.21.15.192:3300/0 conn(0x7f65f40d3890 msgr2=0x7f65f40b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.300+0000 7f65f2ffd700 1 --2- 172.21.15.189:0/1456364832 >> v2:172.21.15.192:3300/0 conn(0x7f65f40d3890 0x7f65f40b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.300+0000 7f65f2ffd700 1 -- 172.21.15.189:0/1456364832 >> v1:172.21.15.189:6789/0 conn(0x7f65f40b4ed0 legacy=0x7f65f40b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.300+0000 7f65f2ffd700 1 -- 172.21.15.189:0/1456364832 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f40d17e0 con 0x7f65f415d590 2024-04-06T13:45:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.300+0000 7f65f1ffb700 1 -- 172.21.15.189:0/1456364832 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593213559 0 0) 0x7f65f40b5cc0 con 0x7f65f40b4ed0 2024-04-06T13:45:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.300+0000 7f65f2ffd700 1 --2- 172.21.15.189:0/1456364832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65f415d590 0x7f65f40b8d70 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f65f40cf880 tx=0x7f65e4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb418764293c5dd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.300+0000 7f65f1ffb700 1 -- 172.21.15.189:0/1456364832 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e4011830 con 0x7f65f415d590 2024-04-06T13:45:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.300+0000 7f65f1ffb700 1 -- 172.21.15.189:0/1456364832 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e4011990 con 0x7f65f415d590 2024-04-06T13:45:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.300+0000 7f65f1ffb700 1 -- 172.21.15.189:0/1456364832 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e4011d20 con 0x7f65f415d590 2024-04-06T13:45:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.301+0000 7f65f96f5700 1 -- 172.21.15.189:0/1456364832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65f415d590 msgr2=0x7f65f40b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.301+0000 7f65f96f5700 1 --2- 172.21.15.189:0/1456364832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65f415d590 0x7f65f40b8d70 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f65f40cf880 tx=0x7f65e4009b20 comp rx=0 tx=0).stop 2024-04-06T13:45:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.301+0000 7f65f96f5700 1 -- 172.21.15.189:0/1456364832 shutdown_connections 2024-04-06T13:45:25.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.301+0000 7f65f96f5700 1 --2- 172.21.15.189:0/1456364832 >> v2:172.21.15.192:3300/0 conn(0x7f65f40d3890 0x7f65f40b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.301+0000 7f65f96f5700 1 --2- 172.21.15.189:0/1456364832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65f415d590 0x7f65f40b8d70 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.301+0000 7f65f96f5700 1 -- 172.21.15.189:0/1456364832 >> 172.21.15.189:0/1456364832 conn(0x7f65f406bbc0 msgr2=0x7f65f406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:25.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.301+0000 7f65f96f5700 1 -- 172.21.15.189:0/1456364832 shutdown_connections 2024-04-06T13:45:25.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.301+0000 7f65f96f5700 1 -- 172.21.15.189:0/1456364832 wait complete. 2024-04-06T13:45:25.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.302+0000 7f65f96f5700 1 Processor -- start 2024-04-06T13:45:25.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.302+0000 7f65f96f5700 1 -- start start 2024-04-06T13:45:25.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.303+0000 7f65f96f5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65f40b4ed0 0x7f65f415ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:25.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.303+0000 7f65f96f5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65f40d3890 0x7f65f41f5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:25.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.303+0000 7f65f96f5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65f415d590 0x7f65f4160250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:25.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.303+0000 7f65f96f5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f41f5f60 con 0x7f65f415d590 2024-04-06T13:45:25.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.303+0000 7f65f96f5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f415ca30 con 0x7f65f40b4ed0 2024-04-06T13:45:25.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.303+0000 7f65f96f5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f40cb290 con 0x7f65f40d3890 2024-04-06T13:45:25.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.303+0000 7f65f37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65f415d590 0x7f65f4160250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:25.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.303+0000 7f65f37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65f415d590 0x7f65f4160250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50710/0 (socket says 172.21.15.189:50710) 2024-04-06T13:45:25.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.303+0000 7f65f37fe700 1 -- 172.21.15.189:0/3399989546 learned_addr learned my addr 172.21.15.189:0/3399989546 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:25.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.303+0000 7f65f2ffd700 1 --2- 172.21.15.189:0/3399989546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65f40b4ed0 0x7f65f415ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:25.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.303+0000 7f65f27fc700 1 --2- 172.21.15.189:0/3399989546 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65f40d3890 0x7f65f41f5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:25.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.303+0000 7f65f37fe700 1 -- 172.21.15.189:0/3399989546 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65f40d3890 msgr2=0x7f65f41f5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:25.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.303+0000 7f65f37fe700 1 --2- 172.21.15.189:0/3399989546 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65f40d3890 0x7f65f41f5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.303+0000 7f65f37fe700 1 -- 172.21.15.189:0/3399989546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65f40b4ed0 msgr2=0x7f65f415ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:25.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.303+0000 7f65f37fe700 1 --2- 172.21.15.189:0/3399989546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65f40b4ed0 0x7f65f415ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.303+0000 7f65f37fe700 1 -- 172.21.15.189:0/3399989546 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e40094a0 con 0x7f65f415d590 2024-04-06T13:45:25.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.303+0000 7f65f2ffd700 1 --2- 172.21.15.189:0/3399989546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65f40b4ed0 0x7f65f415ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:45:25.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.304+0000 7f65f37fe700 1 --2- 172.21.15.189:0/3399989546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65f415d590 0x7f65f4160250 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f65ec009dd0 tx=0x7f65ec00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:25.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.304+0000 7f65db7fe700 1 -- 172.21.15.189:0/3399989546 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec009c40 con 0x7f65f415d590 2024-04-06T13:45:25.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.304+0000 7f65db7fe700 1 -- 172.21.15.189:0/3399989546 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ec00f040 con 0x7f65f415d590 2024-04-06T13:45:25.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.304+0000 7f65db7fe700 1 -- 172.21.15.189:0/3399989546 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec00ceb0 con 0x7f65f415d590 2024-04-06T13:45:25.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.304+0000 7f65f96f5700 1 -- 172.21.15.189:0/3399989546 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f4160790 con 0x7f65f415d590 2024-04-06T13:45:25.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.304+0000 7f65f96f5700 1 -- 172.21.15.189:0/3399989546 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f415c470 con 0x7f65f415d590 2024-04-06T13:45:25.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.305+0000 7f65f96f5700 1 -- 172.21.15.189:0/3399989546 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f4074990 con 0x7f65f415d590 2024-04-06T13:45:25.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.306+0000 7f65db7fe700 1 -- 172.21.15.189:0/3399989546 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 123) v1 ==== 89259+0+0 (secure 0 0 0) 0x7f65ec014480 con 0x7f65f415d590 2024-04-06T13:45:25.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.306+0000 7f65db7fe700 1 --2- 172.21.15.189:0/3399989546 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f65e006adb0 0x7f65e006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:25.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.306+0000 7f65f2ffd700 1 -- 172.21.15.189:0/3399989546 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f65e006adb0 msgr2=0x7f65e006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:25.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.306+0000 7f65db7fe700 1 -- 172.21.15.189:0/3399989546 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6747+0+0 (secure 0 0 0) 0x7f65ec08a4b0 con 0x7f65f415d590 2024-04-06T13:45:25.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.306+0000 7f65f2ffd700 1 --2- 172.21.15.189:0/3399989546 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f65e006adb0 0x7f65e006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:45:25.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.311+0000 7f65db7fe700 1 -- 172.21.15.189:0/3399989546 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f65ec0550f0 con 0x7f65f415d590 2024-04-06T13:45:25.383 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:25.383+0000 7fbe06afc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:45:25.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.507+0000 7f65f2ffd700 1 -- 172.21.15.189:0/3399989546 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f65e006adb0 msgr2=0x7f65e006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:25.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.507+0000 7f65f2ffd700 1 --2- 172.21.15.189:0/3399989546 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f65e006adb0 0x7f65e006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:45:25.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.513+0000 7f65f96f5700 1 -- 172.21.15.189:0/3399989546 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65f40cb3d0 con 0x7f65f415d590 2024-04-06T13:45:25.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.514+0000 7f65db7fe700 1 -- 172.21.15.189:0/3399989546 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+17381 (secure 0 0 0) 0x7f65ec058710 con 0x7f65f415d590 2024-04-06T13:45:25.516 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:25.516 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":196,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:45:22.587972+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:29:14.625778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-04-06T13:29:15.827532+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":196,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/1090714269":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/3656453012":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/2710277231":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/1953253738":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/1974012472":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/820436148":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/2599831431":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/1401962292":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/2983879481":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/3870180304":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/1073864749":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/3525693740":"2024-04-07T13:29:30.406338+0000","172.21.15.189:6833/47128":"2024-04-07T13:29:30.406338+0000","172.21.15.189:6832/47128":"2024-04-07T13:29:30.406338+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:45:25.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.517+0000 7f65f96f5700 1 -- 172.21.15.189:0/3399989546 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f65e006adb0 msgr2=0x7f65e006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:45:25.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.517+0000 7f65f96f5700 1 --2- 172.21.15.189:0/3399989546 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f65e006adb0 0x7f65e006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.517+0000 7f65f96f5700 1 -- 172.21.15.189:0/3399989546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65f415d590 msgr2=0x7f65f4160250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:25.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.517+0000 7f65f96f5700 1 --2- 172.21.15.189:0/3399989546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65f415d590 0x7f65f4160250 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f65ec009dd0 tx=0x7f65ec00c3a0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.517+0000 7f65f96f5700 1 -- 172.21.15.189:0/3399989546 shutdown_connections 2024-04-06T13:45:25.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.517+0000 7f65f96f5700 1 --2- 172.21.15.189:0/3399989546 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f65e006adb0 0x7f65e006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.517+0000 7f65f96f5700 1 --2- 172.21.15.189:0/3399989546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65f415d590 0x7f65f4160250 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.517+0000 7f65f96f5700 1 --2- 172.21.15.189:0/3399989546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65f40b4ed0 0x7f65f415ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.517+0000 7f65f96f5700 1 --2- 172.21.15.189:0/3399989546 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65f40d3890 0x7f65f41f5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.517+0000 7f65f96f5700 1 -- 172.21.15.189:0/3399989546 >> 172.21.15.189:0/3399989546 conn(0x7f65f406bbc0 msgr2=0x7f65f40b2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:25.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.517+0000 7f65f96f5700 1 -- 172.21.15.189:0/3399989546 shutdown_connections 2024-04-06T13:45:25.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.517+0000 7f65f96f5700 1 -- 172.21.15.189:0/3399989546 wait complete. 2024-04-06T13:45:25.531 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-06T13:45:25.531 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:45:25.623 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:25.623+0000 7fbe06afc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:45:25.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.657+0000 7faf9a1ec700 1 Processor -- start 2024-04-06T13:45:25.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.657+0000 7faf9a1ec700 1 -- start start 2024-04-06T13:45:25.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.657+0000 7faf9a1ec700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faf9406ae40 0x7faf940b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:25.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.657+0000 7faf9a1ec700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf940b3990 con 0x7faf9415ddf0 2024-04-06T13:45:25.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.657+0000 7faf9a1ec700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf940b3060 con 0x7faf9415d7b0 2024-04-06T13:45:25.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.657+0000 7faf9a1ec700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf9415e950 con 0x7faf9406ae40 2024-04-06T13:45:25.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.658+0000 7faf92ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faf9406ae40 0x7faf940b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:25.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.658+0000 7faf93fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7faf9415ddf0 0x7faf9415e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38468/0 (socket says 172.21.15.189:38468) 2024-04-06T13:45:25.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.658+0000 7faf93fff700 1 -- 172.21.15.189:0/1877327594 learned_addr learned my addr 172.21.15.189:0/1877327594 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:25.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.658+0000 7faf92ffd700 1 -- 172.21.15.189:0/1877327594 >> v1:172.21.15.192:6789/0 conn(0x7faf9415d7b0 legacy=0x7faf9406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:25.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.658+0000 7faf92ffd700 1 -- 172.21.15.189:0/1877327594 >> v1:172.21.15.189:6789/0 conn(0x7faf9415ddf0 legacy=0x7faf9415e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:25.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.658+0000 7faf927fc700 1 -- 172.21.15.189:0/1877327594 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048221355 0 0) 0x7faf940b3990 con 0x7faf9415ddf0 2024-04-06T13:45:25.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.658+0000 7faf92ffd700 1 -- 172.21.15.189:0/1877327594 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf940b3340 con 0x7faf9406ae40 2024-04-06T13:45:25.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.658+0000 7faf92ffd700 1 --2- 172.21.15.189:0/1877327594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf9406ae40 0x7faf940b3e20 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7faf88009840 tx=0x7faf88009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59d0a86a3c809103 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:25.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.658+0000 7faf927fc700 1 -- 172.21.15.189:0/1877327594 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88011830 con 0x7faf9406ae40 2024-04-06T13:45:25.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.658+0000 7faf927fc700 1 -- 172.21.15.189:0/1877327594 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf88011990 con 0x7faf9406ae40 2024-04-06T13:45:25.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.658+0000 7faf927fc700 1 -- 172.21.15.189:0/1877327594 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88011d20 con 0x7faf9406ae40 2024-04-06T13:45:25.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.659+0000 7faf9a1ec700 1 -- 172.21.15.189:0/1877327594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf9406ae40 msgr2=0x7faf940b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:25.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.659+0000 7faf9a1ec700 1 --2- 172.21.15.189:0/1877327594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf9406ae40 0x7faf940b3e20 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7faf88009840 tx=0x7faf88009b50 comp rx=0 tx=0).stop 2024-04-06T13:45:25.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.659+0000 7faf9a1ec700 1 -- 172.21.15.189:0/1877327594 shutdown_connections 2024-04-06T13:45:25.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.659+0000 7faf9a1ec700 1 --2- 172.21.15.189:0/1877327594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf9406ae40 0x7faf940b3e20 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.659+0000 7faf9a1ec700 1 -- 172.21.15.189:0/1877327594 >> 172.21.15.189:0/1877327594 conn(0x7faf940d00b0 msgr2=0x7faf940d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:25.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.659+0000 7faf9a1ec700 1 -- 172.21.15.189:0/1877327594 shutdown_connections 2024-04-06T13:45:25.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.659+0000 7faf9a1ec700 1 -- 172.21.15.189:0/1877327594 wait complete. 2024-04-06T13:45:25.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.660+0000 7faf9a1ec700 1 Processor -- start 2024-04-06T13:45:25.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.660+0000 7faf9a1ec700 1 -- start start 2024-04-06T13:45:25.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.661+0000 7faf9a1ec700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf9406ae40 0x7faf9415c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:25.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.661+0000 7faf9a1ec700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf9415d7b0 0x7faf94156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:25.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.661+0000 7faf9a1ec700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faf9415ddf0 0x7faf940aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:25.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.661+0000 7faf9a1ec700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf940b1470 con 0x7faf9406ae40 2024-04-06T13:45:25.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.661+0000 7faf9a1ec700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf94157530 con 0x7faf9415d7b0 2024-04-06T13:45:25.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.661+0000 7faf9a1ec700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf940b0ad0 con 0x7faf9415ddf0 2024-04-06T13:45:25.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.661+0000 7faf937fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf9406ae40 0x7faf9415c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:25.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.661+0000 7faf937fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf9406ae40 0x7faf9415c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50726/0 (socket says 172.21.15.189:50726) 2024-04-06T13:45:25.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.661+0000 7faf937fe700 1 -- 172.21.15.189:0/2568211555 learned_addr learned my addr 172.21.15.189:0/2568211555 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:25.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.661+0000 7faf92ffd700 1 --2- 172.21.15.189:0/2568211555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf9415d7b0 0x7faf94156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:25.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.661+0000 7faf93fff700 1 --2- 172.21.15.189:0/2568211555 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faf9415ddf0 0x7faf940aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:25.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.662+0000 7faf937fe700 1 -- 172.21.15.189:0/2568211555 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faf9415ddf0 msgr2=0x7faf940aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:25.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.662+0000 7faf937fe700 1 --2- 172.21.15.189:0/2568211555 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faf9415ddf0 0x7faf940aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.662+0000 7faf937fe700 1 -- 172.21.15.189:0/2568211555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf9415d7b0 msgr2=0x7faf94156ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:25.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.662+0000 7faf937fe700 1 --2- 172.21.15.189:0/2568211555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf9415d7b0 0x7faf94156ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.662+0000 7faf937fe700 1 -- 172.21.15.189:0/2568211555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf880094a0 con 0x7faf9406ae40 2024-04-06T13:45:25.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.662+0000 7faf937fe700 1 --2- 172.21.15.189:0/2568211555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf9406ae40 0x7faf9415c2f0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7faf8400c4e0 tx=0x7faf8400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:25.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.662+0000 7faf7bfff700 1 -- 172.21.15.189:0/2568211555 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf840052d0 con 0x7faf9406ae40 2024-04-06T13:45:25.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.662+0000 7faf7bfff700 1 -- 172.21.15.189:0/2568211555 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf84005430 con 0x7faf9406ae40 2024-04-06T13:45:25.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.662+0000 7faf9a1ec700 1 -- 172.21.15.189:0/2568211555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf940b0db0 con 0x7faf9406ae40 2024-04-06T13:45:25.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.662+0000 7faf7bfff700 1 -- 172.21.15.189:0/2568211555 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf840038c0 con 0x7faf9406ae40 2024-04-06T13:45:25.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.662+0000 7faf9a1ec700 1 -- 172.21.15.189:0/2568211555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf940af0c0 con 0x7faf9406ae40 2024-04-06T13:45:25.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.663+0000 7faf9a1ec700 1 -- 172.21.15.189:0/2568211555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf940b15b0 con 0x7faf9406ae40 2024-04-06T13:45:25.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.664+0000 7faf7bfff700 1 -- 172.21.15.189:0/2568211555 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 123) v1 ==== 89259+0+0 (secure 0 0 0) 0x7faf84003a20 con 0x7faf9406ae40 2024-04-06T13:45:25.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.664+0000 7faf7bfff700 1 --2- 172.21.15.189:0/2568211555 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faf8006ade0 0x7faf8006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:25.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.664+0000 7faf92ffd700 1 -- 172.21.15.189:0/2568211555 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faf8006ade0 msgr2=0x7faf8006d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:25.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.664+0000 7faf7bfff700 1 -- 172.21.15.189:0/2568211555 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6747+0+0 (secure 0 0 0) 0x7faf84088cf0 con 0x7faf9406ae40 2024-04-06T13:45:25.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.664+0000 7faf92ffd700 1 --2- 172.21.15.189:0/2568211555 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faf8006ade0 0x7faf8006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:45:25.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.668+0000 7faf7bfff700 1 -- 172.21.15.189:0/2568211555 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7faf84053910 con 0x7faf9406ae40 2024-04-06T13:45:25.699 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:25.700+0000 7fbe06afc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:45:25.706 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:25.706+0000 7f3f48896200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:45:25.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.865+0000 7faf92ffd700 1 -- 172.21.15.189:0/2568211555 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faf8006ade0 msgr2=0x7faf8006d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:25.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.865+0000 7faf92ffd700 1 --2- 172.21.15.189:0/2568211555 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faf8006ade0 0x7faf8006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:45:25.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.917+0000 7faf9a1ec700 1 -- 172.21.15.189:0/2568211555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf94067bb0 con 0x7faf9406ae40 2024-04-06T13:45:25.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.918+0000 7faf7bfff700 1 -- 172.21.15.189:0/2568211555 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+4749 (secure 0 0 0) 0x7faf84056f30 con 0x7faf9406ae40 2024-04-06T13:45:25.919 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:25.919 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":85,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8141,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2979280081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2979280081},{"type":"v1","addr":"172.21.15.192:6835","nonce":2979280081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":10125,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/286991908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":286991908},{"type":"v1","addr":"172.21.15.189:6836","nonce":286991908}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":10128,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/704323171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":704323171},{"type":"v1","addr":"172.21.15.189:6837","nonce":704323171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:29:16.858698+0000","modified":"2024-04-06T13:29:19.868525+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10137":{"gid":10137,"name":"b","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6834/2662267274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2662267274},{"type":"v1","addr":"172.21.15.192:6834","nonce":2662267274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-04-06T13:45:25.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.920+0000 7faf9a1ec700 1 -- 172.21.15.189:0/2568211555 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faf8006ade0 msgr2=0x7faf8006d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:45:25.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.920+0000 7faf9a1ec700 1 --2- 172.21.15.189:0/2568211555 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faf8006ade0 0x7faf8006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.920+0000 7faf9a1ec700 1 -- 172.21.15.189:0/2568211555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf9406ae40 msgr2=0x7faf9415c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:25.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.920+0000 7faf9a1ec700 1 --2- 172.21.15.189:0/2568211555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf9406ae40 0x7faf9415c2f0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7faf8400c4e0 tx=0x7faf8400c7f0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.920+0000 7faf9a1ec700 1 -- 172.21.15.189:0/2568211555 shutdown_connections 2024-04-06T13:45:25.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.920+0000 7faf9a1ec700 1 --2- 172.21.15.189:0/2568211555 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faf8006ade0 0x7faf8006d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.920+0000 7faf9a1ec700 1 --2- 172.21.15.189:0/2568211555 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faf9415ddf0 0x7faf940aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.920+0000 7faf9a1ec700 1 --2- 172.21.15.189:0/2568211555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf9406ae40 0x7faf9415c2f0 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.920+0000 7faf9a1ec700 1 --2- 172.21.15.189:0/2568211555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf9415d7b0 0x7faf94156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:25.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.920+0000 7faf9a1ec700 1 -- 172.21.15.189:0/2568211555 >> 172.21.15.189:0/2568211555 conn(0x7faf940d00b0 msgr2=0x7faf9415e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:25.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.920+0000 7faf9a1ec700 1 -- 172.21.15.189:0/2568211555 shutdown_connections 2024-04-06T13:45:25.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:25.920+0000 7faf9a1ec700 1 -- 172.21.15.189:0/2568211555 wait complete. 2024-04-06T13:45:25.923 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 85 2024-04-06T13:45:25.934 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:45:25.963 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:25.962+0000 7f3f48896200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:45:26.046 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:26.045+0000 7f3f48896200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:45:26.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.058+0000 7fa48f8bf700 1 Processor -- start 2024-04-06T13:45:26.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.058+0000 7fa48f8bf700 1 -- start start 2024-04-06T13:45:26.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.059+0000 7fa48f8bf700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa48806ae40 0x7fa4880b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:26.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.059+0000 7fa48f8bf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4880b3990 con 0x7fa48815d7b0 2024-04-06T13:45:26.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.059+0000 7fa48f8bf700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4880b3060 con 0x7fa48815ddf0 2024-04-06T13:45:26.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.059+0000 7fa48f8bf700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa48815e950 con 0x7fa48806ae40 2024-04-06T13:45:26.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.059+0000 7fa48d65b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa48815d7b0 0x7fa48806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38470/0 (socket says 172.21.15.189:38470) 2024-04-06T13:45:26.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.059+0000 7fa48d65b700 1 -- 172.21.15.189:0/3689274068 learned_addr learned my addr 172.21.15.189:0/3689274068 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:26.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.059+0000 7fa48ce5a700 1 --2- 172.21.15.189:0/3689274068 >> v2:172.21.15.192:3300/0 conn(0x7fa48806ae40 0x7fa4880b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:26.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.060+0000 7fa47ffff700 1 -- 172.21.15.189:0/3689274068 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225623180 0 0) 0x7fa4880b3990 con 0x7fa48815d7b0 2024-04-06T13:45:26.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.060+0000 7fa47ffff700 1 -- 172.21.15.189:0/3689274068 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4740035b0 con 0x7fa48815d7b0 2024-04-06T13:45:26.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.060+0000 7fa47ffff700 1 -- 172.21.15.189:0/3689274068 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028761589 0 0) 0x7fa4880b3060 con 0x7fa48815ddf0 2024-04-06T13:45:26.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.060+0000 7fa47ffff700 1 -- 172.21.15.189:0/3689274068 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4880b3990 con 0x7fa48815ddf0 2024-04-06T13:45:26.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.060+0000 7fa48ce5a700 1 -- 172.21.15.189:0/3689274068 >> v1:172.21.15.192:6789/0 conn(0x7fa48815ddf0 legacy=0x7fa48815e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:26.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.060+0000 7fa48ce5a700 1 -- 172.21.15.189:0/3689274068 >> v1:172.21.15.189:6789/0 conn(0x7fa48815d7b0 legacy=0x7fa48806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:26.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.060+0000 7fa48ce5a700 1 -- 172.21.15.189:0/3689274068 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4880b3340 con 0x7fa48806ae40 2024-04-06T13:45:26.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.060+0000 7fa48ce5a700 1 --2- 172.21.15.189:0/3689274068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa48806ae40 0x7fa4880b3e20 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fa484009840 tx=0x7fa484009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1616da44944c7d9c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:26.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.060+0000 7fa47ffff700 1 -- 172.21.15.189:0/3689274068 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa484011830 con 0x7fa48806ae40 2024-04-06T13:45:26.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.060+0000 7fa47ffff700 1 -- 172.21.15.189:0/3689274068 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa484011990 con 0x7fa48806ae40 2024-04-06T13:45:26.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.060+0000 7fa47ffff700 1 -- 172.21.15.189:0/3689274068 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa484011d20 con 0x7fa48806ae40 2024-04-06T13:45:26.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.061+0000 7fa48f8bf700 1 -- 172.21.15.189:0/3689274068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa48806ae40 msgr2=0x7fa4880b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:26.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.061+0000 7fa48f8bf700 1 --2- 172.21.15.189:0/3689274068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa48806ae40 0x7fa4880b3e20 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fa484009840 tx=0x7fa484009b50 comp rx=0 tx=0).stop 2024-04-06T13:45:26.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.061+0000 7fa48f8bf700 1 -- 172.21.15.189:0/3689274068 shutdown_connections 2024-04-06T13:45:26.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.061+0000 7fa48f8bf700 1 --2- 172.21.15.189:0/3689274068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa48806ae40 0x7fa4880b3e20 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.061+0000 7fa48f8bf700 1 -- 172.21.15.189:0/3689274068 >> 172.21.15.189:0/3689274068 conn(0x7fa4880d00b0 msgr2=0x7fa4880d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:26.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.061+0000 7fa48f8bf700 1 -- 172.21.15.189:0/3689274068 shutdown_connections 2024-04-06T13:45:26.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.061+0000 7fa48f8bf700 1 -- 172.21.15.189:0/3689274068 wait complete. 2024-04-06T13:45:26.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.062+0000 7fa48f8bf700 1 Processor -- start 2024-04-06T13:45:26.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.062+0000 7fa48f8bf700 1 -- start start 2024-04-06T13:45:26.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.063+0000 7fa48f8bf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa48806ae40 0x7fa4880b0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:26.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.063+0000 7fa48f8bf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa48815d7b0 0x7fa488150790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:26.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.063+0000 7fa48f8bf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa48815ddf0 0x7fa4880b1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:26.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.063+0000 7fa48f8bf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa488150cd0 con 0x7fa48815ddf0 2024-04-06T13:45:26.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.063+0000 7fa48f8bf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa488156430 con 0x7fa48806ae40 2024-04-06T13:45:26.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.063+0000 7fa48f8bf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4881566d0 con 0x7fa48815d7b0 2024-04-06T13:45:26.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.063+0000 7fa48de5c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa48815ddf0 0x7fa4880b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:26.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.063+0000 7fa48d65b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa48806ae40 0x7fa4880b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:26.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.063+0000 7fa48de5c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa48815ddf0 0x7fa4880b1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50740/0 (socket says 172.21.15.189:50740) 2024-04-06T13:45:26.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.063+0000 7fa48de5c700 1 -- 172.21.15.189:0/512059777 learned_addr learned my addr 172.21.15.189:0/512059777 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:26.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.063+0000 7fa48ce5a700 1 --2- 172.21.15.189:0/512059777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa48815d7b0 0x7fa488150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:26.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.063+0000 7fa48de5c700 1 -- 172.21.15.189:0/512059777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa48815d7b0 msgr2=0x7fa488150790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:26.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.063+0000 7fa48de5c700 1 --2- 172.21.15.189:0/512059777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa48815d7b0 0x7fa488150790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.063+0000 7fa48de5c700 1 -- 172.21.15.189:0/512059777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa48806ae40 msgr2=0x7fa4880b0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:26.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.063+0000 7fa48de5c700 1 --2- 172.21.15.189:0/512059777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa48806ae40 0x7fa4880b0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.063+0000 7fa48de5c700 1 -- 172.21.15.189:0/512059777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4840094a0 con 0x7fa48815ddf0 2024-04-06T13:45:26.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.063+0000 7fa48d65b700 1 --2- 172.21.15.189:0/512059777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa48806ae40 0x7fa4880b0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:45:26.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.063+0000 7fa48de5c700 1 --2- 172.21.15.189:0/512059777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa48815ddf0 0x7fa4880b1330 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fa48000cbc0 tx=0x7fa48000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:26.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.064+0000 7fa47dffb700 1 -- 172.21.15.189:0/512059777 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480003c00 con 0x7fa48815ddf0 2024-04-06T13:45:26.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.064+0000 7fa47dffb700 1 -- 172.21.15.189:0/512059777 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa480003d60 con 0x7fa48815ddf0 2024-04-06T13:45:26.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.064+0000 7fa48f8bf700 1 -- 172.21.15.189:0/512059777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4881569b0 con 0x7fa48815ddf0 2024-04-06T13:45:26.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.064+0000 7fa47dffb700 1 -- 172.21.15.189:0/512059777 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48000aaa0 con 0x7fa48815ddf0 2024-04-06T13:45:26.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.064+0000 7fa48f8bf700 1 -- 172.21.15.189:0/512059777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa488156f70 con 0x7fa48815ddf0 2024-04-06T13:45:26.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.065+0000 7fa48f8bf700 1 -- 172.21.15.189:0/512059777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa488067bb0 con 0x7fa48815ddf0 2024-04-06T13:45:26.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.065+0000 7fa47dffb700 1 -- 172.21.15.189:0/512059777 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 123) v1 ==== 89259+0+0 (secure 0 0 0) 0x7fa48000ac00 con 0x7fa48815ddf0 2024-04-06T13:45:26.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.065+0000 7fa47dffb700 1 --2- 172.21.15.189:0/512059777 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa46c06adb0 0x7fa46c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:26.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.066+0000 7fa48d65b700 1 -- 172.21.15.189:0/512059777 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa46c06adb0 msgr2=0x7fa46c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:26.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.066+0000 7fa47dffb700 1 -- 172.21.15.189:0/512059777 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6747+0+0 (secure 0 0 0) 0x7fa480089850 con 0x7fa48815ddf0 2024-04-06T13:45:26.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.066+0000 7fa48d65b700 1 --2- 172.21.15.189:0/512059777 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa46c06adb0 0x7fa46c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:45:26.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.070+0000 7fa47dffb700 1 -- 172.21.15.189:0/512059777 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa480054510 con 0x7fa48815ddf0 2024-04-06T13:45:26.071 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:26.071+0000 7fbe06afc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:45:26.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.267+0000 7fa48d65b700 1 -- 172.21.15.189:0/512059777 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa46c06adb0 msgr2=0x7fa46c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:26.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.267+0000 7fa48d65b700 1 --2- 172.21.15.189:0/512059777 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa46c06adb0 0x7fa46c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:45:26.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.318+0000 7fa48f8bf700 1 -- 172.21.15.189:0/512059777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa488074990 con 0x7fa48815ddf0 2024-04-06T13:45:26.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.319+0000 7fa47dffb700 1 -- 172.21.15.189:0/512059777 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+4749 (secure 0 0 0) 0x7fa480057b30 con 0x7fa48815ddf0 2024-04-06T13:45:26.319 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:26.320 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":85,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8141,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2979280081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2979280081},{"type":"v1","addr":"172.21.15.192:6835","nonce":2979280081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":10125,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/286991908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":286991908},{"type":"v1","addr":"172.21.15.189:6836","nonce":286991908}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":10128,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/704323171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":704323171},{"type":"v1","addr":"172.21.15.189:6837","nonce":704323171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:29:16.858698+0000","modified":"2024-04-06T13:29:19.868525+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10137":{"gid":10137,"name":"b","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6834/2662267274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2662267274},{"type":"v1","addr":"172.21.15.192:6834","nonce":2662267274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-04-06T13:45:26.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.321+0000 7fa48f8bf700 1 -- 172.21.15.189:0/512059777 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa46c06adb0 msgr2=0x7fa46c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:45:26.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.321+0000 7fa48f8bf700 1 --2- 172.21.15.189:0/512059777 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa46c06adb0 0x7fa46c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.321+0000 7fa48f8bf700 1 -- 172.21.15.189:0/512059777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa48815ddf0 msgr2=0x7fa4880b1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:26.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.321+0000 7fa48f8bf700 1 --2- 172.21.15.189:0/512059777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa48815ddf0 0x7fa4880b1330 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fa48000cbc0 tx=0x7fa48000ced0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.321+0000 7fa48f8bf700 1 -- 172.21.15.189:0/512059777 shutdown_connections 2024-04-06T13:45:26.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.321+0000 7fa48f8bf700 1 --2- 172.21.15.189:0/512059777 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa46c06adb0 0x7fa46c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.321+0000 7fa48f8bf700 1 --2- 172.21.15.189:0/512059777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa48815ddf0 0x7fa4880b1330 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.321+0000 7fa48f8bf700 1 --2- 172.21.15.189:0/512059777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa48806ae40 0x7fa4880b0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.321+0000 7fa48f8bf700 1 --2- 172.21.15.189:0/512059777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa48815d7b0 0x7fa488150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.321+0000 7fa48f8bf700 1 -- 172.21.15.189:0/512059777 >> 172.21.15.189:0/512059777 conn(0x7fa4880d00b0 msgr2=0x7fa4880ae070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:26.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.321+0000 7fa48f8bf700 1 -- 172.21.15.189:0/512059777 shutdown_connections 2024-04-06T13:45:26.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.321+0000 7fa48f8bf700 1 -- 172.21.15.189:0/512059777 wait complete. 2024-04-06T13:45:26.323 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 85 2024-04-06T13:45:26.335 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:45:26.446 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:26.445+0000 7f3f48896200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:45:26.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.506+0000 7fad20a23700 1 Processor -- start 2024-04-06T13:45:26.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.506+0000 7fad20a23700 1 -- start start 2024-04-06T13:45:26.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.507+0000 7fad20a23700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fad1c15a550 0x7fad1c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:26.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.508+0000 7fad20a23700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fad1c0cf900 0x7fad1c0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:26.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.508+0000 7fad20a23700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1c0cdbe0 con 0x7fad1c0b4ba0 2024-04-06T13:45:26.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.508+0000 7fad20a23700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad1c15a940 con 0x7fad1c0cf900 2024-04-06T13:45:26.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.508+0000 7fad20a23700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad1c15ac00 con 0x7fad1c15a550 2024-04-06T13:45:26.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.508+0000 7fad1ad9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fad1c0b4ba0 0x7fad1c06eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38474/0 (socket says 172.21.15.189:38474) 2024-04-06T13:45:26.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.508+0000 7fad1ad9d700 1 -- 172.21.15.189:0/89791236 learned_addr learned my addr 172.21.15.189:0/89791236 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:26.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.508+0000 7fad1a59c700 1 --2- 172.21.15.189:0/89791236 >> v2:172.21.15.192:3300/0 conn(0x7fad1c15a550 0x7fad1c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:26.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.508+0000 7fad19d9b700 1 --2- 172.21.15.189:0/89791236 >> v2:172.21.15.189:3300/0 conn(0x7fad1c0cf900 0x7fad1c0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:26.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.508+0000 7fad1a59c700 1 -- 172.21.15.189:0/89791236 >> v2:172.21.15.189:3300/0 conn(0x7fad1c0cf900 msgr2=0x7fad1c0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:26.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.508+0000 7fad1a59c700 1 --2- 172.21.15.189:0/89791236 >> v2:172.21.15.189:3300/0 conn(0x7fad1c0cf900 0x7fad1c0b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.508+0000 7fad1a59c700 1 -- 172.21.15.189:0/89791236 >> v1:172.21.15.189:6789/0 conn(0x7fad1c0b4ba0 legacy=0x7fad1c06eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:26.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.508+0000 7fad1a59c700 1 -- 172.21.15.189:0/89791236 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1c15aee0 con 0x7fad1c15a550 2024-04-06T13:45:26.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.508+0000 7fad1a59c700 1 --2- 172.21.15.189:0/89791236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad1c15a550 0x7fad1c0cf3c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fad1c073470 tx=0x7fad0c009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22fd075597e58667 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:26.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.508+0000 7fad1959a700 1 -- 172.21.15.189:0/89791236 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c011830 con 0x7fad1c15a550 2024-04-06T13:45:26.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.508+0000 7fad1959a700 1 -- 172.21.15.189:0/89791236 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad0c011990 con 0x7fad1c15a550 2024-04-06T13:45:26.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.508+0000 7fad1959a700 1 -- 172.21.15.189:0/89791236 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c011d20 con 0x7fad1c15a550 2024-04-06T13:45:26.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.509+0000 7fad20a23700 1 -- 172.21.15.189:0/89791236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad1c15a550 msgr2=0x7fad1c0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:26.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.509+0000 7fad20a23700 1 --2- 172.21.15.189:0/89791236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad1c15a550 0x7fad1c0cf3c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fad1c073470 tx=0x7fad0c009b20 comp rx=0 tx=0).stop 2024-04-06T13:45:26.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.509+0000 7fad20a23700 1 -- 172.21.15.189:0/89791236 shutdown_connections 2024-04-06T13:45:26.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.509+0000 7fad20a23700 1 --2- 172.21.15.189:0/89791236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad1c15a550 0x7fad1c0cf3c0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.509+0000 7fad20a23700 1 --2- 172.21.15.189:0/89791236 >> v2:172.21.15.189:3300/0 conn(0x7fad1c0cf900 0x7fad1c0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.509+0000 7fad20a23700 1 -- 172.21.15.189:0/89791236 >> 172.21.15.189:0/89791236 conn(0x7fad1c0d00b0 msgr2=0x7fad1c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:26.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.509+0000 7fad20a23700 1 -- 172.21.15.189:0/89791236 shutdown_connections 2024-04-06T13:45:26.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.509+0000 7fad20a23700 1 -- 172.21.15.189:0/89791236 wait complete. 2024-04-06T13:45:26.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.511+0000 7fad20a23700 1 Processor -- start 2024-04-06T13:45:26.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.511+0000 7fad20a23700 1 -- start start 2024-04-06T13:45:26.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.512+0000 7fad20a23700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad1c0b4ba0 0x7fad1c14d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:26.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.512+0000 7fad20a23700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad1c0cf900 0x7fad1c143a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:26.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.512+0000 7fad20a23700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad1c15c9b0 0x7fad1c144a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:26.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.512+0000 7fad20a23700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c15c6f0 con 0x7fad1c15c9b0 2024-04-06T13:45:26.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.512+0000 7fad20a23700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c15cee0 con 0x7fad1c0b4ba0 2024-04-06T13:45:26.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.512+0000 7fad20a23700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c14b5d0 con 0x7fad1c0cf900 2024-04-06T13:45:26.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.512+0000 7fad19d9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad1c0cf900 0x7fad1c143a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:26.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.512+0000 7fad1a59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad1c0b4ba0 0x7fad1c14d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:26.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.512+0000 7fad19d9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad1c0cf900 0x7fad1c143a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:45382/0 (socket says 172.21.15.189:45382) 2024-04-06T13:45:26.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.512+0000 7fad19d9b700 1 -- 172.21.15.189:0/493053146 learned_addr learned my addr 172.21.15.189:0/493053146 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:26.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.513+0000 7fad1ad9d700 1 --2- 172.21.15.189:0/493053146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad1c15c9b0 0x7fad1c144a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:26.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.513+0000 7fad19d9b700 1 -- 172.21.15.189:0/493053146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad1c0b4ba0 msgr2=0x7fad1c14d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:26.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.513+0000 7fad19d9b700 1 --2- 172.21.15.189:0/493053146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad1c0b4ba0 0x7fad1c14d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.513+0000 7fad19d9b700 1 -- 172.21.15.189:0/493053146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad1c15c9b0 msgr2=0x7fad1c144a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:26.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.513+0000 7fad19d9b700 1 --2- 172.21.15.189:0/493053146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad1c15c9b0 0x7fad1c144a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.513+0000 7fad19d9b700 1 -- 172.21.15.189:0/493053146 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0c0094a0 con 0x7fad1c0cf900 2024-04-06T13:45:26.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.513+0000 7fad1a59c700 1 --2- 172.21.15.189:0/493053146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad1c0b4ba0 0x7fad1c14d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:45:26.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.513+0000 7fad1ad9d700 1 --2- 172.21.15.189:0/493053146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad1c15c9b0 0x7fad1c144a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:45:26.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.513+0000 7fad19d9b700 1 --2- 172.21.15.189:0/493053146 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad1c0cf900 0x7fad1c143a80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fad1000e9e0 tx=0x7fad1000ecf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:26.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.513+0000 7fad0affd700 1 -- 172.21.15.189:0/493053146 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad10009710 con 0x7fad1c0cf900 2024-04-06T13:45:26.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.513+0000 7fad0affd700 1 -- 172.21.15.189:0/493053146 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1000ee80 con 0x7fad1c0cf900 2024-04-06T13:45:26.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.513+0000 7fad20a23700 1 -- 172.21.15.189:0/493053146 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad1c14b8b0 con 0x7fad1c0cf900 2024-04-06T13:45:26.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.513+0000 7fad20a23700 1 -- 172.21.15.189:0/493053146 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad1c14bb90 con 0x7fad1c0cf900 2024-04-06T13:45:26.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.513+0000 7fad0affd700 1 -- 172.21.15.189:0/493053146 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1000ca50 con 0x7fad1c0cf900 2024-04-06T13:45:26.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.514+0000 7fad20a23700 1 -- 172.21.15.189:0/493053146 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad000052f0 con 0x7fad1c0cf900 2024-04-06T13:45:26.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.515+0000 7fad0affd700 1 -- 172.21.15.189:0/493053146 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 124) v1 ==== 88856+0+0 (secure 0 0 0) 0x7fad1000cbb0 con 0x7fad1c0cf900 2024-04-06T13:45:26.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.516+0000 7fad0affd700 1 -- 172.21.15.189:0/493053146 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 7047+0+0 (secure 0 0 0) 0x7fad10089220 con 0x7fad1c0cf900 2024-04-06T13:45:26.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.520+0000 7fad0affd700 1 -- 172.21.15.189:0/493053146 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fad10054310 con 0x7fad1c0cf900 2024-04-06T13:45:26.530 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:26.529+0000 7f3ec3f75700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:26.530 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:26.529+0000 7f3ec3f75700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:26.530 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:26.529+0000 7f3ec3f75700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:26.530 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:26.529+0000 7f3ec3f75700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:26.530 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:26.529+0000 7f3ec3f75700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:26.530 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:26.529+0000 7f3ec0f6f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:26.530 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:26.529+0000 7f3ec0f6f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:26.530 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:26.529+0000 7f3ec0f6f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:26.531 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:26.529+0000 7f3ec0f6f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:26.531 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:26.529+0000 7f3ec0f6f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:26.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.726+0000 7fad20a23700 1 -- 172.21.15.189:0/493053146 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad000055a0 con 0x7fad1c0cf900 2024-04-06T13:45:26.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.728+0000 7fad0affd700 1 -- 172.21.15.189:0/493053146 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+17814 (secure 0 0 0) 0x7fad10057930 con 0x7fad1c0cf900 2024-04-06T13:45:26.729 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:26.729 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":197,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:45:26.459405+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:29:14.625778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-04-06T13:29:15.827532+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":196,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/1910258936":"2024-04-07T13:45:26.459370+0000","172.21.15.189:0/1856725622":"2024-04-07T13:45:26.459370+0000","172.21.15.189:0/1012667766":"2024-04-07T13:45:26.459370+0000","172.21.15.189:0/1369160062":"2024-04-07T13:45:26.459370+0000","172.21.15.189:6832/111704":"2024-04-07T13:45:26.459370+0000","172.21.15.189:0/2221593394":"2024-04-07T13:45:26.459370+0000","172.21.15.189:0/1090714269":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/3656453012":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/2710277231":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/1953253738":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/1974012472":"2024-04-07T13:40:22.300382+0000","172.21.15.189:6833/111704":"2024-04-07T13:45:26.459370+0000","172.21.15.189:0/820436148":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/2599831431":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/1401962292":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/2983879481":"2024-04-07T13:40:22.300382+0000","172.21.15.189:0/3870180304":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/1073864749":"2024-04-07T13:29:30.406338+0000","172.21.15.189:0/2616071243":"2024-04-07T13:45:26.459370+0000","172.21.15.189:0/3525693740":"2024-04-07T13:29:30.406338+0000","172.21.15.189:6833/47128":"2024-04-07T13:29:30.406338+0000","172.21.15.189:6832/47128":"2024-04-07T13:29:30.406338+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:45:26.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.730+0000 7fad20a23700 1 -- 172.21.15.189:0/493053146 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad1c0cf900 msgr2=0x7fad1c143a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:26.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.730+0000 7fad20a23700 1 --2- 172.21.15.189:0/493053146 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad1c0cf900 0x7fad1c143a80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fad1000e9e0 tx=0x7fad1000ecf0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.730+0000 7fad20a23700 1 -- 172.21.15.189:0/493053146 shutdown_connections 2024-04-06T13:45:26.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.730+0000 7fad20a23700 1 --2- 172.21.15.189:0/493053146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad1c15c9b0 0x7fad1c144a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.730+0000 7fad20a23700 1 --2- 172.21.15.189:0/493053146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad1c0b4ba0 0x7fad1c14d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.730+0000 7fad20a23700 1 --2- 172.21.15.189:0/493053146 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad1c0cf900 0x7fad1c143a80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.730+0000 7fad20a23700 1 -- 172.21.15.189:0/493053146 >> 172.21.15.189:0/493053146 conn(0x7fad1c0d00b0 msgr2=0x7fad1c06b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:26.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.730+0000 7fad20a23700 1 -- 172.21.15.189:0/493053146 shutdown_connections 2024-04-06T13:45:26.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.730+0000 7fad20a23700 1 -- 172.21.15.189:0/493053146 wait complete. 2024-04-06T13:45:26.743 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-06T13:45:26.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.865+0000 7f65eac23700 1 Processor -- start 2024-04-06T13:45:26.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.865+0000 7f65eac23700 1 -- start start 2024-04-06T13:45:26.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.866+0000 7f65eac23700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f65e415a550 0x7f65e40cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:26.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.866+0000 7f65eac23700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f65e40cf900 0x7f65e40b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:26.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.866+0000 7f65eac23700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65e40b4ba0 0x7f65e406eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:26.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.866+0000 7f65eac23700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e40cdbe0 con 0x7f65e40cf900 2024-04-06T13:45:26.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.866+0000 7f65eac23700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e40b4ff0 con 0x7f65e415a550 2024-04-06T13:45:26.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.866+0000 7f65eac23700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e415a940 con 0x7f65e40b4ba0 2024-04-06T13:45:26.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.866+0000 7f65e3fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f65e40cf900 0x7f65e40b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:26.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.866+0000 7f65e3fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f65e40cf900 0x7f65e40b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50866/0 (socket says 172.21.15.189:50866) 2024-04-06T13:45:26.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.867+0000 7f65e3fff700 1 -- 172.21.15.189:0/3483320028 learned_addr learned my addr 172.21.15.189:0/3483320028 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:26.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.867+0000 7f65e89bf700 1 --2- 172.21.15.189:0/3483320028 >> v2:172.21.15.192:3300/0 conn(0x7f65e415a550 0x7f65e40cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:26.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.867+0000 7f65e91c0700 1 --2- 172.21.15.189:0/3483320028 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65e40b4ba0 0x7f65e406eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:26.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.867+0000 7f65e3fff700 1 -- 172.21.15.189:0/3483320028 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65e40b4ba0 msgr2=0x7f65e406eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:26.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.867+0000 7f65e3fff700 1 --2- 172.21.15.189:0/3483320028 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65e40b4ba0 0x7f65e406eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.867+0000 7f65e3fff700 1 -- 172.21.15.189:0/3483320028 >> v2:172.21.15.192:3300/0 conn(0x7f65e415a550 msgr2=0x7f65e40cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:26.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.867+0000 7f65e3fff700 1 --2- 172.21.15.189:0/3483320028 >> v2:172.21.15.192:3300/0 conn(0x7f65e415a550 0x7f65e40cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.867+0000 7f65e3fff700 1 -- 172.21.15.189:0/3483320028 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e415ac00 con 0x7f65e40cf900 2024-04-06T13:45:26.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.867+0000 7f65e89bf700 1 --2- 172.21.15.189:0/3483320028 >> v2:172.21.15.192:3300/0 conn(0x7f65e415a550 0x7f65e40cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:26.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.867+0000 7f65e3fff700 1 --2- 172.21.15.189:0/3483320028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65e40cf900 0x7f65e40b4660 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f65e4073470 tx=0x7f65d0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89afcec743a3f34c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:26.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.867+0000 7f65e37fe700 1 -- 172.21.15.189:0/3483320028 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d000b870 con 0x7f65e40cf900 2024-04-06T13:45:26.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.867+0000 7f65e37fe700 1 -- 172.21.15.189:0/3483320028 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d000b9d0 con 0x7f65e40cf900 2024-04-06T13:45:26.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.867+0000 7f65e37fe700 1 -- 172.21.15.189:0/3483320028 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d000bd60 con 0x7f65e40cf900 2024-04-06T13:45:26.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.868+0000 7f65eac23700 1 -- 172.21.15.189:0/3483320028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65e40cf900 msgr2=0x7f65e40b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:26.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.868+0000 7f65eac23700 1 --2- 172.21.15.189:0/3483320028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65e40cf900 0x7f65e40b4660 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f65e4073470 tx=0x7f65d0009bd0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.868+0000 7f65eac23700 1 -- 172.21.15.189:0/3483320028 shutdown_connections 2024-04-06T13:45:26.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.868+0000 7f65eac23700 1 --2- 172.21.15.189:0/3483320028 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65e40b4ba0 0x7f65e406eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.868+0000 7f65eac23700 1 --2- 172.21.15.189:0/3483320028 >> v2:172.21.15.192:3300/0 conn(0x7f65e415a550 0x7f65e40cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.868+0000 7f65eac23700 1 --2- 172.21.15.189:0/3483320028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65e40cf900 0x7f65e40b4660 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.868+0000 7f65eac23700 1 -- 172.21.15.189:0/3483320028 >> 172.21.15.189:0/3483320028 conn(0x7f65e40d00b0 msgr2=0x7f65e40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:26.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.868+0000 7f65eac23700 1 -- 172.21.15.189:0/3483320028 shutdown_connections 2024-04-06T13:45:26.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.868+0000 7f65eac23700 1 -- 172.21.15.189:0/3483320028 wait complete. 2024-04-06T13:45:26.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.869+0000 7f65eac23700 1 Processor -- start 2024-04-06T13:45:26.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.869+0000 7f65eac23700 1 -- start start 2024-04-06T13:45:26.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.870+0000 7f65eac23700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65e40b4ba0 0x7f65e4158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:26.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.870+0000 7f65eac23700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65e40cf900 0x7f65e41566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:26.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.870+0000 7f65eac23700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65e4148500 0x7f65e4147c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:26.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.870+0000 7f65eac23700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e414be30 con 0x7f65e40cf900 2024-04-06T13:45:26.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.870+0000 7f65eac23700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e415bcf0 con 0x7f65e4148500 2024-04-06T13:45:26.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.870+0000 7f65eac23700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e415bfd0 con 0x7f65e40b4ba0 2024-04-06T13:45:26.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.870+0000 7f65e3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65e40cf900 0x7f65e41566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:26.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.870+0000 7f65e3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65e40cf900 0x7f65e41566d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50880/0 (socket says 172.21.15.189:50880) 2024-04-06T13:45:26.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.870+0000 7f65e3fff700 1 -- 172.21.15.189:0/3193045457 learned_addr learned my addr 172.21.15.189:0/3193045457 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:26.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.871+0000 7f65e89bf700 1 --2- 172.21.15.189:0/3193045457 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65e40b4ba0 0x7f65e4158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:26.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.871+0000 7f65e91c0700 1 --2- 172.21.15.189:0/3193045457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65e4148500 0x7f65e4147c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:26.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.871+0000 7f65e3fff700 1 -- 172.21.15.189:0/3193045457 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65e40b4ba0 msgr2=0x7f65e4158160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:26.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.871+0000 7f65e3fff700 1 --2- 172.21.15.189:0/3193045457 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65e40b4ba0 0x7f65e4158160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.871+0000 7f65e3fff700 1 -- 172.21.15.189:0/3193045457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65e4148500 msgr2=0x7f65e4147c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:26.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.871+0000 7f65e3fff700 1 --2- 172.21.15.189:0/3193045457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65e4148500 0x7f65e4147c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:26.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.871+0000 7f65e3fff700 1 -- 172.21.15.189:0/3193045457 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d0009550 con 0x7f65e40cf900 2024-04-06T13:45:26.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.871+0000 7f65e3fff700 1 --2- 172.21.15.189:0/3193045457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65e40cf900 0x7f65e41566d0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f65d0009c10 tx=0x7f65d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:26.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.871+0000 7f65e17fa700 1 -- 172.21.15.189:0/3193045457 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d0004250 con 0x7f65e40cf900 2024-04-06T13:45:26.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.871+0000 7f65e17fa700 1 -- 172.21.15.189:0/3193045457 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d001f070 con 0x7f65e40cf900 2024-04-06T13:45:26.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.871+0000 7f65eac23700 1 -- 172.21.15.189:0/3193045457 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e415c2b0 con 0x7f65e40cf900 2024-04-06T13:45:26.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.871+0000 7f65e17fa700 1 -- 172.21.15.189:0/3193045457 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d00045d0 con 0x7f65e40cf900 2024-04-06T13:45:26.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.871+0000 7f65eac23700 1 -- 172.21.15.189:0/3193045457 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e4158760 con 0x7f65e40cf900 2024-04-06T13:45:26.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.872+0000 7f65eac23700 1 -- 172.21.15.189:0/3193045457 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e415be30 con 0x7f65e40cf900 2024-04-06T13:45:26.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.872+0000 7f65e17fa700 1 -- 172.21.15.189:0/3193045457 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 124) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f65d0021030 con 0x7f65e40cf900 2024-04-06T13:45:26.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.873+0000 7f65e17fa700 1 -- 172.21.15.189:0/3193045457 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 7047+0+0 (secure 0 0 0) 0x7f65d008a670 con 0x7f65e40cf900 2024-04-06T13:45:26.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:26.877+0000 7f65e17fa700 1 -- 172.21.15.189:0/3193045457 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f65e415be30 con 0x7f65e40cf900 2024-04-06T13:45:27.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.119+0000 7f65eac23700 1 -- 172.21.15.189:0/3193045457 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f65e415c7d0 con 0x7f65e40cf900 2024-04-06T13:45:27.120 INFO:tasks.ceph.mon.a.smithi189.stderr:2024-04-06T13:45:27.120+0000 7fa9fe1ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T13:45:27.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.131+0000 7f65e17fa700 1 -- 172.21.15.189:0/3193045457 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v86) v1 ==== 161+0+0 (secure 0 0 0) 0x7f65e415c7d0 con 0x7f65e40cf900 2024-04-06T13:45:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.133+0000 7f65eac23700 1 -- 172.21.15.189:0/3193045457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65e40cf900 msgr2=0x7f65e41566d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.133+0000 7f65eac23700 1 --2- 172.21.15.189:0/3193045457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65e40cf900 0x7f65e41566d0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f65d0009c10 tx=0x7f65d0004750 comp rx=0 tx=0).stop 2024-04-06T13:45:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.133+0000 7f65eac23700 1 -- 172.21.15.189:0/3193045457 shutdown_connections 2024-04-06T13:45:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.133+0000 7f65eac23700 1 --2- 172.21.15.189:0/3193045457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65e4148500 0x7f65e4147c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.133+0000 7f65eac23700 1 --2- 172.21.15.189:0/3193045457 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65e40b4ba0 0x7f65e4158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.133+0000 7f65eac23700 1 --2- 172.21.15.189:0/3193045457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65e40cf900 0x7f65e41566d0 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.133+0000 7f65eac23700 1 -- 172.21.15.189:0/3193045457 >> 172.21.15.189:0/3193045457 conn(0x7f65e40d00b0 msgr2=0x7f65e4158b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.133+0000 7f65eac23700 1 -- 172.21.15.189:0/3193045457 shutdown_connections 2024-04-06T13:45:27.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.133+0000 7f65eac23700 1 -- 172.21.15.189:0/3193045457 wait complete. 2024-04-06T13:45:27.135 INFO:teuthology.orchestra.run.smithi189.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-06T13:45:27.146 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-06T13:45:27.168 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T13:45:27.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.270+0000 7f706cf51700 1 Processor -- start 2024-04-06T13:45:27.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.270+0000 7f706cf51700 1 -- start start 2024-04-06T13:45:27.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.271+0000 7f706cf51700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7068146dd0 0x7f706806d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:27.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.271+0000 7f706cf51700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70680cedf0 0x7f70680cf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:27.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.271+0000 7f706cf51700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70680cf780 con 0x7f70680d17a0 2024-04-06T13:45:27.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.271+0000 7f706cf51700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f706806beb0 con 0x7f7068146dd0 2024-04-06T13:45:27.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.271+0000 7f706cf51700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706806c0a0 con 0x7f70680cedf0 2024-04-06T13:45:27.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.271+0000 7f7065d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7068146dd0 0x7f706806d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:27.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.272+0000 7f7065d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7068146dd0 0x7f706806d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50894/0 (socket says 172.21.15.189:50894) 2024-04-06T13:45:27.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.272+0000 7f7065d9b700 1 -- 172.21.15.189:0/3433988451 learned_addr learned my addr 172.21.15.189:0/3433988451 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:27.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.272+0000 7f7066d9d700 1 --2- 172.21.15.189:0/3433988451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70680cedf0 0x7f70680cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:27.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.272+0000 7f7065d9b700 1 -- 172.21.15.189:0/3433988451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70680cedf0 msgr2=0x7f70680cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:27.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.272+0000 7f7065d9b700 1 --2- 172.21.15.189:0/3433988451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70680cedf0 0x7f70680cf240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:27.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.272+0000 7f7065d9b700 1 -- 172.21.15.189:0/3433988451 >> v1:172.21.15.189:6789/0 conn(0x7f70680d17a0 legacy=0x7f706806b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:27.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.272+0000 7f7065d9b700 1 -- 172.21.15.189:0/3433988451 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706806c380 con 0x7f7068146dd0 2024-04-06T13:45:27.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.272+0000 7f7066d9d700 1 --2- 172.21.15.189:0/3433988451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70680cedf0 0x7f70680cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:45:27.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.272+0000 7f7065d9b700 1 --2- 172.21.15.189:0/3433988451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7068146dd0 0x7f706806d140 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f70580098f0 tx=0x7f7058009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425946f05d1c9c07 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:27.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.272+0000 7f706559a700 1 -- 172.21.15.189:0/3433988451 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705800b8a0 con 0x7f7068146dd0 2024-04-06T13:45:27.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.272+0000 7f706559a700 1 -- 172.21.15.189:0/3433988451 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705800ba00 con 0x7f7068146dd0 2024-04-06T13:45:27.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.272+0000 7f706559a700 1 -- 172.21.15.189:0/3433988451 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705800bd90 con 0x7f7068146dd0 2024-04-06T13:45:27.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.273+0000 7f706cf51700 1 -- 172.21.15.189:0/3433988451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7068146dd0 msgr2=0x7f706806d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:27.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.273+0000 7f706cf51700 1 --2- 172.21.15.189:0/3433988451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7068146dd0 0x7f706806d140 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f70580098f0 tx=0x7f7058009c00 comp rx=0 tx=0).stop 2024-04-06T13:45:27.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.273+0000 7f706cf51700 1 -- 172.21.15.189:0/3433988451 shutdown_connections 2024-04-06T13:45:27.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.273+0000 7f706cf51700 1 --2- 172.21.15.189:0/3433988451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70680cedf0 0x7f70680cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:27.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.273+0000 7f706cf51700 1 --2- 172.21.15.189:0/3433988451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7068146dd0 0x7f706806d140 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:27.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.273+0000 7f706cf51700 1 -- 172.21.15.189:0/3433988451 >> 172.21.15.189:0/3433988451 conn(0x7f70680b2db0 msgr2=0x7f70680b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:27.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.273+0000 7f706cf51700 1 -- 172.21.15.189:0/3433988451 shutdown_connections 2024-04-06T13:45:27.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.273+0000 7f706cf51700 1 -- 172.21.15.189:0/3433988451 wait complete. 2024-04-06T13:45:27.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.274+0000 7f706cf51700 1 Processor -- start 2024-04-06T13:45:27.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.274+0000 7f706cf51700 1 -- start start 2024-04-06T13:45:27.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.275+0000 7f706cf51700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70680cedf0 0x7f7068160110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:27.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.275+0000 7f706cf51700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70680d17a0 0x7f70680b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:27.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.275+0000 7f706cf51700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7068146dd0 0x7f70680c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:27.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.275+0000 7f706cf51700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70681f6260 con 0x7f70680d17a0 2024-04-06T13:45:27.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.275+0000 7f706cf51700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70681f6d70 con 0x7f70680cedf0 2024-04-06T13:45:27.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.275+0000 7f706cf51700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70681f6440 con 0x7f7068146dd0 2024-04-06T13:45:27.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.275+0000 7f7065d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70680d17a0 0x7f70680b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:27.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.275+0000 7f7065d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70680d17a0 0x7f70680b62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50902/0 (socket says 172.21.15.189:50902) 2024-04-06T13:45:27.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.275+0000 7f7065d9b700 1 -- 172.21.15.189:0/1904894225 learned_addr learned my addr 172.21.15.189:0/1904894225 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:27.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.275+0000 7f706659c700 1 --2- 172.21.15.189:0/1904894225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70680cedf0 0x7f7068160110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:27.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.275+0000 7f7066d9d700 1 --2- 172.21.15.189:0/1904894225 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7068146dd0 0x7f70680c4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:27.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.275+0000 7f7065d9b700 1 -- 172.21.15.189:0/1904894225 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7068146dd0 msgr2=0x7f70680c4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:27.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.275+0000 7f7065d9b700 1 --2- 172.21.15.189:0/1904894225 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7068146dd0 0x7f70680c4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:27.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.275+0000 7f7065d9b700 1 -- 172.21.15.189:0/1904894225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70680cedf0 msgr2=0x7f7068160110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:27.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.275+0000 7f7065d9b700 1 --2- 172.21.15.189:0/1904894225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70680cedf0 0x7f7068160110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:27.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.275+0000 7f7065d9b700 1 -- 172.21.15.189:0/1904894225 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7058009550 con 0x7f70680d17a0 2024-04-06T13:45:27.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.275+0000 7f7065d9b700 1 --2- 172.21.15.189:0/1904894225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70680d17a0 0x7f70680b62e0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f7058000c00 tx=0x7f7058004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:27.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.276+0000 7f7056ffd700 1 -- 172.21.15.189:0/1904894225 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7058004380 con 0x7f70680d17a0 2024-04-06T13:45:27.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.276+0000 7f7056ffd700 1 -- 172.21.15.189:0/1904894225 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705801f070 con 0x7f70680d17a0 2024-04-06T13:45:27.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.276+0000 7f706cf51700 1 -- 172.21.15.189:0/1904894225 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70681f6720 con 0x7f70680d17a0 2024-04-06T13:45:27.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.276+0000 7f7056ffd700 1 -- 172.21.15.189:0/1904894225 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70580036a0 con 0x7f70680d17a0 2024-04-06T13:45:27.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.276+0000 7f706cf51700 1 -- 172.21.15.189:0/1904894225 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7068160650 con 0x7f70680d17a0 2024-04-06T13:45:27.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.276+0000 7f706cf51700 1 -- 172.21.15.189:0/1904894225 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70680d0020 con 0x7f70680d17a0 2024-04-06T13:45:27.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.277+0000 7f7056ffd700 1 -- 172.21.15.189:0/1904894225 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 124) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f7058021030 con 0x7f70680d17a0 2024-04-06T13:45:27.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.277+0000 7f7056ffd700 1 -- 172.21.15.189:0/1904894225 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 7133+0+0 (secure 0 0 0) 0x7f705808a7e0 con 0x7f70680d17a0 2024-04-06T13:45:27.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.282+0000 7f7056ffd700 1 -- 172.21.15.189:0/1904894225 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7058055800 con 0x7f70680d17a0 2024-04-06T13:45:27.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.469+0000 7f7056ffd700 1 -- 172.21.15.189:0/1904894225 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 125) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f70580552b0 con 0x7f70680d17a0 2024-04-06T13:45:27.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.469+0000 7f7056ffd700 1 --2- 172.21.15.189:0/1904894225 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7048036150 0x7f7048072c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:27.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.471+0000 7f706659c700 1 --2- 172.21.15.189:0/1904894225 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7048036150 0x7f7048072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:27.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.472+0000 7f706659c700 1 --2- 172.21.15.189:0/1904894225 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7048036150 0x7f7048072c80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f705c007fd0 tx=0x7f705c009040 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:27.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:27.532+0000 7f706cf51700 1 -- 172.21.15.189:0/1904894225 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7068074990 con 0x7f70680d17a0 2024-04-06T13:45:28.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.132+0000 7f7056ffd700 1 -- 172.21.15.189:0/1904894225 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v87) v1 ==== 104+0+0 (secure 0 0 0) 0x7f705806d850 con 0x7f70680d17a0 2024-04-06T13:45:28.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.134+0000 7f706cf51700 1 -- 172.21.15.189:0/1904894225 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7048036150 msgr2=0x7f7048072c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:28.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.134+0000 7f706cf51700 1 --2- 172.21.15.189:0/1904894225 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7048036150 0x7f7048072c80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f705c007fd0 tx=0x7f705c009040 comp rx=0 tx=0).stop 2024-04-06T13:45:28.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.134+0000 7f706cf51700 1 -- 172.21.15.189:0/1904894225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70680d17a0 msgr2=0x7f70680b62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:28.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.134+0000 7f706cf51700 1 --2- 172.21.15.189:0/1904894225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70680d17a0 0x7f70680b62e0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f7058000c00 tx=0x7f7058004750 comp rx=0 tx=0).stop 2024-04-06T13:45:28.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.134+0000 7f706cf51700 1 -- 172.21.15.189:0/1904894225 shutdown_connections 2024-04-06T13:45:28.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.134+0000 7f706cf51700 1 --2- 172.21.15.189:0/1904894225 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7048036150 0x7f7048072c80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:28.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.135+0000 7f706cf51700 1 --2- 172.21.15.189:0/1904894225 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7068146dd0 0x7f70680c4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:28.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.135+0000 7f706cf51700 1 --2- 172.21.15.189:0/1904894225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70680cedf0 0x7f7068160110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:28.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.135+0000 7f706cf51700 1 --2- 172.21.15.189:0/1904894225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70680d17a0 0x7f70680b62e0 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:28.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.135+0000 7f706cf51700 1 -- 172.21.15.189:0/1904894225 >> 172.21.15.189:0/1904894225 conn(0x7f70680b2db0 msgr2=0x7f706815dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:28.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.135+0000 7f706cf51700 1 -- 172.21.15.189:0/1904894225 shutdown_connections 2024-04-06T13:45:28.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.135+0000 7f706cf51700 1 -- 172.21.15.189:0/1904894225 wait complete. 2024-04-06T13:45:28.150 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-06T13:45:28.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.278+0000 7f68f40b8700 1 Processor -- start 2024-04-06T13:45:28.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.278+0000 7f68f40b8700 1 -- start start 2024-04-06T13:45:28.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.279+0000 7f68f40b8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f68ec06ae40 0x7f68ec0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:28.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.279+0000 7f68f40b8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68ec0b3990 con 0x7f68ec15ddf0 2024-04-06T13:45:28.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.279+0000 7f68f40b8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68ec0b3060 con 0x7f68ec15d7b0 2024-04-06T13:45:28.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.279+0000 7f68f40b8700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68ec15e950 con 0x7f68ec06ae40 2024-04-06T13:45:28.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.279+0000 7f68f2655700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f68ec15ddf0 0x7f68ec15e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38488/0 (socket says 172.21.15.189:38488) 2024-04-06T13:45:28.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.279+0000 7f68f2655700 1 -- 172.21.15.189:0/1532337009 learned_addr learned my addr 172.21.15.189:0/1532337009 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:28.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.279+0000 7f68f0e52700 1 -- 172.21.15.189:0/1532337009 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881568463 0 0) 0x7f68ec0b3990 con 0x7f68ec15ddf0 2024-04-06T13:45:28.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.279+0000 7f68f0e52700 1 -- 172.21.15.189:0/1532337009 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d80035b0 con 0x7f68ec15ddf0 2024-04-06T13:45:28.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.279+0000 7f68f1653700 1 --2- 172.21.15.189:0/1532337009 >> v2:172.21.15.192:3300/0 conn(0x7f68ec06ae40 0x7f68ec0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:28.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.279+0000 7f68f0e52700 1 -- 172.21.15.189:0/1532337009 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2249164898 0 0) 0x7f68d80035b0 con 0x7f68ec15ddf0 2024-04-06T13:45:28.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.279+0000 7f68f0e52700 1 -- 172.21.15.189:0/1532337009 >> v2:172.21.15.192:3300/0 conn(0x7f68ec06ae40 msgr2=0x7f68ec0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:28.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.279+0000 7f68f0e52700 1 --2- 172.21.15.189:0/1532337009 >> v2:172.21.15.192:3300/0 conn(0x7f68ec06ae40 0x7f68ec0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:28.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.279+0000 7f68f0e52700 1 -- 172.21.15.189:0/1532337009 >> v1:172.21.15.192:6789/0 conn(0x7f68ec15d7b0 legacy=0x7f68ec06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:28.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.279+0000 7f68f0e52700 1 -- 172.21.15.189:0/1532337009 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ec0b3340 con 0x7f68ec15ddf0 2024-04-06T13:45:28.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.279+0000 7f68f0e52700 1 -- 172.21.15.189:0/1532337009 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f68e40019f0 con 0x7f68ec15ddf0 2024-04-06T13:45:28.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.279+0000 7f68f0e52700 1 -- 172.21.15.189:0/1532337009 >> v1:172.21.15.189:6789/0 conn(0x7f68ec15ddf0 legacy=0x7f68ec15e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:28.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.279+0000 7f68f0e52700 1 --2- 172.21.15.189:0/1532337009 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68d8003a00 0x7f68d8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:28.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.280+0000 7f68f0e52700 1 --2- 172.21.15.189:0/1532337009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68d8004850 0x7f68d8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:28.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.280+0000 7f68f0e52700 1 --2- 172.21.15.189:0/1532337009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d8005210 0x7f68d800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:28.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.280+0000 7f68f0e52700 1 -- 172.21.15.189:0/1532337009 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d800c180 con 0x7f68d8005210 2024-04-06T13:45:28.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.280+0000 7f68f0e52700 1 -- 172.21.15.189:0/1532337009 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d800c2f0 con 0x7f68d8004850 2024-04-06T13:45:28.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.280+0000 7f68f0e52700 1 -- 172.21.15.189:0/1532337009 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d800c4b0 con 0x7f68d8003a00 2024-04-06T13:45:28.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.280+0000 7f68f2655700 1 --2- 172.21.15.189:0/1532337009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d8005210 0x7f68d800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:28.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.280+0000 7f68f1653700 1 --2- 172.21.15.189:0/1532337009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68d8004850 0x7f68d8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:28.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.280+0000 7f68f1e54700 1 --2- 172.21.15.189:0/1532337009 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68d8003a00 0x7f68d8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:28.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.280+0000 7f68f2655700 1 -- 172.21.15.189:0/1532337009 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68d8003a00 msgr2=0x7f68d8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:28.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.280+0000 7f68f2655700 1 --2- 172.21.15.189:0/1532337009 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68d8003a00 0x7f68d8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:28.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.280+0000 7f68f2655700 1 -- 172.21.15.189:0/1532337009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68d8004850 msgr2=0x7f68d8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:28.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.280+0000 7f68f2655700 1 --2- 172.21.15.189:0/1532337009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68d8004850 0x7f68d8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:28.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.280+0000 7f68f2655700 1 -- 172.21.15.189:0/1532337009 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d80035b0 con 0x7f68d8005210 2024-04-06T13:45:28.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.280+0000 7f68f1e54700 1 --2- 172.21.15.189:0/1532337009 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68d8003a00 0x7f68d8003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:28.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.280+0000 7f68f2655700 1 -- 172.21.15.189:0/1532337009 reap_dead start 2024-04-06T13:45:28.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.280+0000 7f68f2655700 1 --2- 172.21.15.189:0/1532337009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d8005210 0x7f68d800ba10 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f68e400a830 tx=0x7f68d8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3e0f738bda4929 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:28.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.280+0000 7f68f0e52700 1 -- 172.21.15.189:0/1532337009 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e400a2f0 con 0x7f68d8005210 2024-04-06T13:45:28.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.280+0000 7f68f0e52700 1 -- 172.21.15.189:0/1532337009 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e4012600 con 0x7f68d8005210 2024-04-06T13:45:28.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.280+0000 7f68f0e52700 1 -- 172.21.15.189:0/1532337009 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e40129b0 con 0x7f68d8005210 2024-04-06T13:45:28.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.281+0000 7f68f40b8700 1 -- 172.21.15.189:0/1532337009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d8005210 msgr2=0x7f68d800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:28.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.281+0000 7f68f40b8700 1 --2- 172.21.15.189:0/1532337009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d8005210 0x7f68d800ba10 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f68e400a830 tx=0x7f68d8005880 comp rx=0 tx=0).stop 2024-04-06T13:45:28.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.281+0000 7f68f40b8700 1 -- 172.21.15.189:0/1532337009 shutdown_connections 2024-04-06T13:45:28.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.281+0000 7f68f40b8700 1 --2- 172.21.15.189:0/1532337009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d8005210 0x7f68d800ba10 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:28.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.281+0000 7f68f40b8700 1 -- 172.21.15.189:0/1532337009 >> 172.21.15.189:0/1532337009 conn(0x7f68ec0d00b0 msgr2=0x7f68ec0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:28.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.281+0000 7f68f40b8700 1 -- 172.21.15.189:0/1532337009 shutdown_connections 2024-04-06T13:45:28.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.281+0000 7f68f40b8700 1 -- 172.21.15.189:0/1532337009 wait complete. 2024-04-06T13:45:28.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.282+0000 7f68f40b8700 1 Processor -- start 2024-04-06T13:45:28.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.282+0000 7f68f40b8700 1 -- start start 2024-04-06T13:45:28.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.283+0000 7f68f40b8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d8005210 0x7f68ec142c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:28.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.283+0000 7f68f40b8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68ec0cc580 0x7f68ec0cca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:28.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.283+0000 7f68f40b8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68ec1f64c0 0x7f68ec1f6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:28.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.283+0000 7f68f40b8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec0ccf40 con 0x7f68d8005210 2024-04-06T13:45:28.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.283+0000 7f68f40b8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec145670 con 0x7f68ec1f64c0 2024-04-06T13:45:28.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.283+0000 7f68f40b8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec145f10 con 0x7f68ec0cc580 2024-04-06T13:45:28.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.283+0000 7f68f1e54700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d8005210 0x7f68ec142c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:28.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.283+0000 7f68f1e54700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d8005210 0x7f68ec142c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50922/0 (socket says 172.21.15.189:50922) 2024-04-06T13:45:28.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.283+0000 7f68f1e54700 1 -- 172.21.15.189:0/3428179829 learned_addr learned my addr 172.21.15.189:0/3428179829 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:28.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.283+0000 7f68f1e54700 1 -- 172.21.15.189:0/3428179829 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68ec0cc580 msgr2=0x7f68ec0cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:28.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.283+0000 7f68f1e54700 1 --2- 172.21.15.189:0/3428179829 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68ec0cc580 0x7f68ec0cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:28.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.283+0000 7f68f1e54700 1 -- 172.21.15.189:0/3428179829 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68ec1f64c0 msgr2=0x7f68ec1f6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:28.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.283+0000 7f68f1e54700 1 --2- 172.21.15.189:0/3428179829 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68ec1f64c0 0x7f68ec1f6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:28.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.283+0000 7f68f1e54700 1 -- 172.21.15.189:0/3428179829 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ec1f71f0 con 0x7f68d8005210 2024-04-06T13:45:28.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.284+0000 7f68f1e54700 1 --2- 172.21.15.189:0/3428179829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d8005210 0x7f68ec142c70 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f68dc00e5b0 tx=0x7f68dc00e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:28.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.284+0000 7f68e27fc700 1 -- 172.21.15.189:0/3428179829 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68dc0052e0 con 0x7f68d8005210 2024-04-06T13:45:28.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.284+0000 7f68e27fc700 1 -- 172.21.15.189:0/3428179829 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68dc005440 con 0x7f68d8005210 2024-04-06T13:45:28.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.284+0000 7f68f40b8700 1 -- 172.21.15.189:0/3428179829 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ec1458c0 con 0x7f68d8005210 2024-04-06T13:45:28.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.284+0000 7f68e27fc700 1 -- 172.21.15.189:0/3428179829 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68dc0038c0 con 0x7f68d8005210 2024-04-06T13:45:28.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.284+0000 7f68f40b8700 1 -- 172.21.15.189:0/3428179829 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ec1f6e80 con 0x7f68d8005210 2024-04-06T13:45:28.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.285+0000 7f68f40b8700 1 -- 172.21.15.189:0/3428179829 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ec067bb0 con 0x7f68d8005210 2024-04-06T13:45:28.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.285+0000 7f68e27fc700 1 -- 172.21.15.189:0/3428179829 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 125) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f68dc003a20 con 0x7f68d8005210 2024-04-06T13:45:28.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.285+0000 7f68e27fc700 1 --2- 172.21.15.189:0/3428179829 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f68d006aae0 0x7f68d006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:28.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.286+0000 7f68e27fc700 1 -- 172.21.15.189:0/3428179829 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 7133+0+0 (secure 0 0 0) 0x7f68dc088ad0 con 0x7f68d8005210 2024-04-06T13:45:28.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.286+0000 7f68f1653700 1 --2- 172.21.15.189:0/3428179829 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f68d006aae0 0x7f68d006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:28.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.286+0000 7f68f1653700 1 --2- 172.21.15.189:0/3428179829 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f68d006aae0 0x7f68d006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f68ec15cfa0 tx=0x7f68e80093c0 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:28.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.290+0000 7f68e27fc700 1 -- 172.21.15.189:0/3428179829 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f68dc053b10 con 0x7f68d8005210 2024-04-06T13:45:28.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:28.486+0000 7f68f40b8700 1 -- 172.21.15.189:0/3428179829 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f68ec0cfaa0 con 0x7f68d8005210 2024-04-06T13:45:29.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.481+0000 7f68e27fc700 1 -- 172.21.15.189:0/3428179829 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 126) v1 ==== 89191+0+0 (secure 0 0 0) 0x7f68dc003cd0 con 0x7f68d8005210 2024-04-06T13:45:29.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.482+0000 7f68e27fc700 1 -- 172.21.15.189:0/3428179829 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v199) v1 ==== 181+0+0 (secure 0 0 0) 0x7f68dc0709a0 con 0x7f68d8005210 2024-04-06T13:45:29.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.487+0000 7f68f40b8700 1 -- 172.21.15.189:0/3428179829 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f68d006aae0 msgr2=0x7f68d006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:29.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.487+0000 7f68f40b8700 1 --2- 172.21.15.189:0/3428179829 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f68d006aae0 0x7f68d006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f68ec15cfa0 tx=0x7f68e80093c0 comp rx=0 tx=0).stop 2024-04-06T13:45:29.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.487+0000 7f68f40b8700 1 -- 172.21.15.189:0/3428179829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d8005210 msgr2=0x7f68ec142c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:29.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.487+0000 7f68f40b8700 1 --2- 172.21.15.189:0/3428179829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d8005210 0x7f68ec142c70 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f68dc00e5b0 tx=0x7f68dc00e970 comp rx=0 tx=0).stop 2024-04-06T13:45:29.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.487+0000 7f68f40b8700 1 -- 172.21.15.189:0/3428179829 shutdown_connections 2024-04-06T13:45:29.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.487+0000 7f68f40b8700 1 --2- 172.21.15.189:0/3428179829 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f68d006aae0 0x7f68d006cfa0 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f68ec15cfa0 tx=0x7f68e80093c0 comp rx=0 tx=0).stop 2024-04-06T13:45:29.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.487+0000 7f68f40b8700 1 --2- 172.21.15.189:0/3428179829 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68ec1f64c0 0x7f68ec1f6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:29.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.487+0000 7f68f40b8700 1 --2- 172.21.15.189:0/3428179829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d8005210 0x7f68ec142c70 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:29.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.487+0000 7f68f40b8700 1 --2- 172.21.15.189:0/3428179829 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68ec0cc580 0x7f68ec0cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:29.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.487+0000 7f68f40b8700 1 -- 172.21.15.189:0/3428179829 >> 172.21.15.189:0/3428179829 conn(0x7f68ec0d00b0 msgr2=0x7f68ec06eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:29.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.488+0000 7f68f40b8700 1 -- 172.21.15.189:0/3428179829 shutdown_connections 2024-04-06T13:45:29.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.489+0000 7f68f40b8700 1 -- 172.21.15.189:0/3428179829 wait complete. 2024-04-06T13:45:29.491 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' removed 2024-04-06T13:45:29.506 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-06T13:45:29.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.637+0000 7fdbe0c24700 1 Processor -- start 2024-04-06T13:45:29.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.637+0000 7fdbe0c24700 1 -- start start 2024-04-06T13:45:29.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.638+0000 7fdbe0c24700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdbdc15d7b0 0x7fdbdc0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:29.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.638+0000 7fdbe0c24700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbdc06a730 0x7fdbdc06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:29.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.638+0000 7fdbe0c24700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbdc0b3990 con 0x7fdbdc15ddf0 2024-04-06T13:45:29.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.638+0000 7fdbe0c24700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbdc06b0f0 con 0x7fdbdc15d7b0 2024-04-06T13:45:29.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.638+0000 7fdbe0c24700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc0b3060 con 0x7fdbdc06a730 2024-04-06T13:45:29.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.638+0000 7fdbda59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdbdc15d7b0 0x7fdbdc0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:29.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.639+0000 7fdbda59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdbdc15d7b0 0x7fdbdc0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56372/0 (socket says 172.21.15.189:56372) 2024-04-06T13:45:29.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.639+0000 7fdbda59c700 1 -- 172.21.15.189:0/4180560215 learned_addr learned my addr 172.21.15.189:0/4180560215 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:29.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.639+0000 7fdbd9d9b700 1 --2- 172.21.15.189:0/4180560215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbdc06a730 0x7fdbdc06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:29.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.639+0000 7fdbda59c700 1 -- 172.21.15.189:0/4180560215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbdc06a730 msgr2=0x7fdbdc06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:29.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.639+0000 7fdbda59c700 1 --2- 172.21.15.189:0/4180560215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbdc06a730 0x7fdbdc06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:29.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.639+0000 7fdbda59c700 1 -- 172.21.15.189:0/4180560215 >> v1:172.21.15.192:6789/0 conn(0x7fdbdc15ddf0 legacy=0x7fdbdc15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:29.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.639+0000 7fdbda59c700 1 -- 172.21.15.189:0/4180560215 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbdc0b3340 con 0x7fdbdc15d7b0 2024-04-06T13:45:29.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.639+0000 7fdbda59c700 1 --2- 172.21.15.189:0/4180560215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbdc15d7b0 0x7fdbdc0b3e20 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fdbd00098f0 tx=0x7fdbd0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e10d563ee100a28c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:29.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.639+0000 7fdbd959a700 1 -- 172.21.15.189:0/4180560215 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd000b8a0 con 0x7fdbdc15d7b0 2024-04-06T13:45:29.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.639+0000 7fdbd959a700 1 -- 172.21.15.189:0/4180560215 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd000ba00 con 0x7fdbdc15d7b0 2024-04-06T13:45:29.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.639+0000 7fdbd959a700 1 -- 172.21.15.189:0/4180560215 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd000bd90 con 0x7fdbdc15d7b0 2024-04-06T13:45:29.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.640+0000 7fdbe0c24700 1 -- 172.21.15.189:0/4180560215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbdc15d7b0 msgr2=0x7fdbdc0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:29.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.640+0000 7fdbe0c24700 1 --2- 172.21.15.189:0/4180560215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbdc15d7b0 0x7fdbdc0b3e20 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fdbd00098f0 tx=0x7fdbd0009c00 comp rx=0 tx=0).stop 2024-04-06T13:45:29.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.640+0000 7fdbe0c24700 1 -- 172.21.15.189:0/4180560215 shutdown_connections 2024-04-06T13:45:29.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.640+0000 7fdbe0c24700 1 --2- 172.21.15.189:0/4180560215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbdc15d7b0 0x7fdbdc0b3e20 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:29.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.640+0000 7fdbe0c24700 1 --2- 172.21.15.189:0/4180560215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbdc06a730 0x7fdbdc06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:29.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.640+0000 7fdbe0c24700 1 -- 172.21.15.189:0/4180560215 >> 172.21.15.189:0/4180560215 conn(0x7fdbdc0d00b0 msgr2=0x7fdbdc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:29.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.640+0000 7fdbe0c24700 1 -- 172.21.15.189:0/4180560215 shutdown_connections 2024-04-06T13:45:29.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.640+0000 7fdbe0c24700 1 -- 172.21.15.189:0/4180560215 wait complete. 2024-04-06T13:45:29.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.641+0000 7fdbe0c24700 1 Processor -- start 2024-04-06T13:45:29.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.641+0000 7fdbe0c24700 1 -- start start 2024-04-06T13:45:29.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.642+0000 7fdbe0c24700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbdc06a730 0x7fdbdc1f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:29.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.642+0000 7fdbe0c24700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbdc15d7b0 0x7fdbdc1f6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:29.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.642+0000 7fdbe0c24700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbdc15ddf0 0x7fdbdc1f6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:29.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.642+0000 7fdbe0c24700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc148c90 con 0x7fdbdc15ddf0 2024-04-06T13:45:29.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.642+0000 7fdbe0c24700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc0b0d70 con 0x7fdbdc06a730 2024-04-06T13:45:29.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.642+0000 7fdbe0c24700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc0b1050 con 0x7fdbdc15d7b0 2024-04-06T13:45:29.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.642+0000 7fdbdad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbdc15ddf0 0x7fdbdc1f6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:29.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.642+0000 7fdbdad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbdc15ddf0 0x7fdbdc1f6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56380/0 (socket says 172.21.15.189:56380) 2024-04-06T13:45:29.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.642+0000 7fdbdad9d700 1 -- 172.21.15.189:0/2455751743 learned_addr learned my addr 172.21.15.189:0/2455751743 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:29.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.642+0000 7fdbd9d9b700 1 --2- 172.21.15.189:0/2455751743 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbdc15d7b0 0x7fdbdc1f6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:29.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.642+0000 7fdbda59c700 1 --2- 172.21.15.189:0/2455751743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbdc06a730 0x7fdbdc1f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:29.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.642+0000 7fdbdad9d700 1 -- 172.21.15.189:0/2455751743 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbdc15d7b0 msgr2=0x7fdbdc1f6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:29.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.642+0000 7fdbdad9d700 1 --2- 172.21.15.189:0/2455751743 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbdc15d7b0 0x7fdbdc1f6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:29.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.642+0000 7fdbdad9d700 1 -- 172.21.15.189:0/2455751743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbdc06a730 msgr2=0x7fdbdc1f28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:29.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.642+0000 7fdbdad9d700 1 --2- 172.21.15.189:0/2455751743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbdc06a730 0x7fdbdc1f28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:29.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.642+0000 7fdbdad9d700 1 -- 172.21.15.189:0/2455751743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd0009550 con 0x7fdbdc15ddf0 2024-04-06T13:45:29.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.642+0000 7fdbd9d9b700 1 --2- 172.21.15.189:0/2455751743 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbdc15d7b0 0x7fdbdc1f6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:29.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.642+0000 7fdbda59c700 1 --2- 172.21.15.189:0/2455751743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbdc06a730 0x7fdbdc1f28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:29.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.643+0000 7fdbdad9d700 1 --2- 172.21.15.189:0/2455751743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbdc15ddf0 0x7fdbdc1f6990 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fdbd400cb50 tx=0x7fdbd400cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:29.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.643+0000 7fdbcaffd700 1 -- 172.21.15.189:0/2455751743 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4003c00 con 0x7fdbdc15ddf0 2024-04-06T13:45:29.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.643+0000 7fdbcaffd700 1 -- 172.21.15.189:0/2455751743 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd4003d60 con 0x7fdbdc15ddf0 2024-04-06T13:45:29.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.643+0000 7fdbe0c24700 1 -- 172.21.15.189:0/2455751743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbdc0b1330 con 0x7fdbdc15ddf0 2024-04-06T13:45:29.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.643+0000 7fdbcaffd700 1 -- 172.21.15.189:0/2455751743 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd400ab70 con 0x7fdbdc15ddf0 2024-04-06T13:45:29.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.643+0000 7fdbe0c24700 1 -- 172.21.15.189:0/2455751743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbdc148460 con 0x7fdbdc15ddf0 2024-04-06T13:45:29.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.644+0000 7fdbe0c24700 1 -- 172.21.15.189:0/2455751743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbdc0b1850 con 0x7fdbdc15ddf0 2024-04-06T13:45:29.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.644+0000 7fdbcaffd700 1 -- 172.21.15.189:0/2455751743 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 126) v1 ==== 89191+0+0 (secure 0 0 0) 0x7fdbd400acd0 con 0x7fdbdc15ddf0 2024-04-06T13:45:29.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.644+0000 7fdbcaffd700 1 --2- 172.21.15.189:0/2455751743 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdbbc06ad20 0x7fdbbc06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:29.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.645+0000 7fdbcaffd700 1 -- 172.21.15.189:0/2455751743 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6695+0+0 (secure 0 0 0) 0x7fdbd4089780 con 0x7fdbdc15ddf0 2024-04-06T13:45:29.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.645+0000 7fdbda59c700 1 --2- 172.21.15.189:0/2455751743 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdbbc06ad20 0x7fdbbc06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:29.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.645+0000 7fdbda59c700 1 --2- 172.21.15.189:0/2455751743 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdbbc06ad20 0x7fdbbc06d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0009c40 tx=0x7fdbd0008030 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:29.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.649+0000 7fdbcaffd700 1 -- 172.21.15.189:0/2455751743 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdbdc0b1850 con 0x7fdbdc15ddf0 2024-04-06T13:45:29.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:29.841+0000 7fdbe0c24700 1 -- 172.21.15.189:0/2455751743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdbdc0b1850 con 0x7fdbdc15ddf0 2024-04-06T13:45:30.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.635+0000 7fdbcaffd700 1 -- 172.21.15.189:0/2455751743 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fdbd4054970 con 0x7fdbdc15ddf0 2024-04-06T13:45:30.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.638+0000 7fdbcaffd700 1 -- 172.21.15.189:0/2455751743 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v200) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdbd4083690 con 0x7fdbdc15ddf0 2024-04-06T13:45:30.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.643+0000 7fdbe0c24700 1 -- 172.21.15.189:0/2455751743 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdbbc06ad20 msgr2=0x7fdbbc06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:30.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.643+0000 7fdbe0c24700 1 --2- 172.21.15.189:0/2455751743 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdbbc06ad20 0x7fdbbc06d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0009c40 tx=0x7fdbd0008030 comp rx=0 tx=0).stop 2024-04-06T13:45:30.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.643+0000 7fdbe0c24700 1 -- 172.21.15.189:0/2455751743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbdc15ddf0 msgr2=0x7fdbdc1f6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:30.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.643+0000 7fdbe0c24700 1 --2- 172.21.15.189:0/2455751743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbdc15ddf0 0x7fdbdc1f6990 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fdbd400cb50 tx=0x7fdbd400cf10 comp rx=0 tx=0).stop 2024-04-06T13:45:30.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.643+0000 7fdbe0c24700 1 -- 172.21.15.189:0/2455751743 shutdown_connections 2024-04-06T13:45:30.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.643+0000 7fdbe0c24700 1 --2- 172.21.15.189:0/2455751743 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdbbc06ad20 0x7fdbbc06d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:30.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.643+0000 7fdbe0c24700 1 --2- 172.21.15.189:0/2455751743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbdc15ddf0 0x7fdbdc1f6990 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:30.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.643+0000 7fdbe0c24700 1 --2- 172.21.15.189:0/2455751743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbdc06a730 0x7fdbdc1f28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:30.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.643+0000 7fdbe0c24700 1 --2- 172.21.15.189:0/2455751743 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbdc15d7b0 0x7fdbdc1f6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:30.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.643+0000 7fdbe0c24700 1 -- 172.21.15.189:0/2455751743 >> 172.21.15.189:0/2455751743 conn(0x7fdbdc0d00b0 msgr2=0x7fdbdc0adbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:30.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.644+0000 7fdbe0c24700 1 -- 172.21.15.189:0/2455751743 shutdown_connections 2024-04-06T13:45:30.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.644+0000 7fdbe0c24700 1 -- 172.21.15.189:0/2455751743 wait complete. 2024-04-06T13:45:30.645 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' removed 2024-04-06T13:45:30.659 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-06T13:45:30.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.789+0000 7f9925a4f700 1 Processor -- start 2024-04-06T13:45:30.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.789+0000 7f9925a4f700 1 -- start start 2024-04-06T13:45:30.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.789+0000 7f9925a4f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f992015d470 0x7f99200d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:30.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.789+0000 7f9925a4f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f99200b6560 0x7f99200ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:30.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.789+0000 7f9925a4f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99200b90a0 con 0x7f99200d1500 2024-04-06T13:45:30.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.789+0000 7f9925a4f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99200b6a40 con 0x7f99200b6560 2024-04-06T13:45:30.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.789+0000 7f9925a4f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99200b6d20 con 0x7f992015d470 2024-04-06T13:45:30.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.790+0000 7f991e7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f99200b6560 0x7f99200ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:30.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.790+0000 7f991e7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f99200b6560 0x7f99200ce100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56396/0 (socket says 172.21.15.189:56396) 2024-04-06T13:45:30.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.790+0000 7f991e7fc700 1 -- 172.21.15.189:0/2748514980 learned_addr learned my addr 172.21.15.189:0/2748514980 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:30.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.790+0000 7f991effd700 1 --2- 172.21.15.189:0/2748514980 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f992015d470 0x7f99200d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:30.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.790+0000 7f991e7fc700 1 -- 172.21.15.189:0/2748514980 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f992015d470 msgr2=0x7f99200d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:30.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.790+0000 7f991e7fc700 1 --2- 172.21.15.189:0/2748514980 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f992015d470 0x7f99200d0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:30.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.790+0000 7f991e7fc700 1 -- 172.21.15.189:0/2748514980 >> v1:172.21.15.189:6789/0 conn(0x7f99200d1500 legacy=0x7f99201610d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:30.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.790+0000 7f991e7fc700 1 -- 172.21.15.189:0/2748514980 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99200b7000 con 0x7f99200b6560 2024-04-06T13:45:30.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.790+0000 7f991e7fc700 1 --2- 172.21.15.189:0/2748514980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99200b6560 0x7f99200ce100 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f99140098f0 tx=0x7f9914009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf7310dd3922ca3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:30.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.790+0000 7f991dffb700 1 -- 172.21.15.189:0/2748514980 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991400b8a0 con 0x7f99200b6560 2024-04-06T13:45:30.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.790+0000 7f991dffb700 1 -- 172.21.15.189:0/2748514980 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f991400ba00 con 0x7f99200b6560 2024-04-06T13:45:30.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.790+0000 7f991dffb700 1 -- 172.21.15.189:0/2748514980 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991400bd90 con 0x7f99200b6560 2024-04-06T13:45:30.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.791+0000 7f9925a4f700 1 -- 172.21.15.189:0/2748514980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99200b6560 msgr2=0x7f99200ce100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:30.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.791+0000 7f9925a4f700 1 --2- 172.21.15.189:0/2748514980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99200b6560 0x7f99200ce100 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f99140098f0 tx=0x7f9914009c00 comp rx=0 tx=0).stop 2024-04-06T13:45:30.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.791+0000 7f9925a4f700 1 -- 172.21.15.189:0/2748514980 shutdown_connections 2024-04-06T13:45:30.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.791+0000 7f9925a4f700 1 --2- 172.21.15.189:0/2748514980 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f992015d470 0x7f99200d0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:30.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.791+0000 7f9925a4f700 1 --2- 172.21.15.189:0/2748514980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99200b6560 0x7f99200ce100 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:30.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.791+0000 7f9925a4f700 1 -- 172.21.15.189:0/2748514980 >> 172.21.15.189:0/2748514980 conn(0x7f992006bbc0 msgr2=0x7f992006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:30.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.791+0000 7f9925a4f700 1 -- 172.21.15.189:0/2748514980 shutdown_connections 2024-04-06T13:45:30.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.791+0000 7f9925a4f700 1 -- 172.21.15.189:0/2748514980 wait complete. 2024-04-06T13:45:30.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.792+0000 7f9925a4f700 1 Processor -- start 2024-04-06T13:45:30.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.792+0000 7f9925a4f700 1 -- start start 2024-04-06T13:45:30.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.793+0000 7f9925a4f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99200b6560 0x7f99200c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:30.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.793+0000 7f9925a4f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99200d1500 0x7f99201f2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:30.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.793+0000 7f9925a4f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f992015d470 0x7f99200c7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:30.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.793+0000 7f9925a4f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99201f3dd0 con 0x7f99200d1500 2024-04-06T13:45:30.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.793+0000 7f9925a4f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99201f3810 con 0x7f992015d470 2024-04-06T13:45:30.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.793+0000 7f9925a4f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99201f3af0 con 0x7f99200b6560 2024-04-06T13:45:30.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.793+0000 7f991e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99200d1500 0x7f99201f2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:30.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.793+0000 7f991e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99200d1500 0x7f99201f2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56400/0 (socket says 172.21.15.189:56400) 2024-04-06T13:45:30.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.793+0000 7f991e7fc700 1 -- 172.21.15.189:0/3622150962 learned_addr learned my addr 172.21.15.189:0/3622150962 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:30.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.793+0000 7f991effd700 1 --2- 172.21.15.189:0/3622150962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99200b6560 0x7f99200c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:30.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.793+0000 7f991f7fe700 1 --2- 172.21.15.189:0/3622150962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f992015d470 0x7f99200c7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:30.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.793+0000 7f991e7fc700 1 -- 172.21.15.189:0/3622150962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99200b6560 msgr2=0x7f99200c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:30.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.793+0000 7f991e7fc700 1 --2- 172.21.15.189:0/3622150962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99200b6560 0x7f99200c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:30.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.793+0000 7f991e7fc700 1 -- 172.21.15.189:0/3622150962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f992015d470 msgr2=0x7f99200c7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:30.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.793+0000 7f991e7fc700 1 --2- 172.21.15.189:0/3622150962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f992015d470 0x7f99200c7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:30.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.793+0000 7f991e7fc700 1 -- 172.21.15.189:0/3622150962 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9914009550 con 0x7f99200d1500 2024-04-06T13:45:30.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.793+0000 7f991f7fe700 1 --2- 172.21.15.189:0/3622150962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f992015d470 0x7f99200c7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:30.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.794+0000 7f991e7fc700 1 --2- 172.21.15.189:0/3622150962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99200d1500 0x7f99201f2f80 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f9914009c40 tx=0x7f9914004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:30.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.794+0000 7f9907fff700 1 -- 172.21.15.189:0/3622150962 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9914004590 con 0x7f99200d1500 2024-04-06T13:45:30.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.794+0000 7f9907fff700 1 -- 172.21.15.189:0/3622150962 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f991401e070 con 0x7f99200d1500 2024-04-06T13:45:30.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.794+0000 7f9925a4f700 1 -- 172.21.15.189:0/3622150962 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99200c7000 con 0x7f99200d1500 2024-04-06T13:45:30.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.794+0000 7f9907fff700 1 -- 172.21.15.189:0/3622150962 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99140038e0 con 0x7f99200d1500 2024-04-06T13:45:30.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.794+0000 7f9925a4f700 1 -- 172.21.15.189:0/3622150962 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99200c9d30 con 0x7f99200d1500 2024-04-06T13:45:30.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.795+0000 7f9925a4f700 1 -- 172.21.15.189:0/3622150962 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9920067bf0 con 0x7f99200d1500 2024-04-06T13:45:30.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.795+0000 7f9907fff700 1 -- 172.21.15.189:0/3622150962 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f9914003a40 con 0x7f99200d1500 2024-04-06T13:45:30.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.796+0000 7f9907fff700 1 --2- 172.21.15.189:0/3622150962 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f990c06adb0 0x7f990c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:30.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.796+0000 7f9907fff700 1 -- 172.21.15.189:0/3622150962 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6281+0+0 (secure 0 0 0) 0x7f9914020070 con 0x7f99200d1500 2024-04-06T13:45:30.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.796+0000 7f991effd700 1 --2- 172.21.15.189:0/3622150962 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f990c06adb0 0x7f990c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:30.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.796+0000 7f991effd700 1 --2- 172.21.15.189:0/3622150962 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f990c06adb0 0x7f990c06d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9910006b20 tx=0x7f9910008040 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:30.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.800+0000 7f9907fff700 1 -- 172.21.15.189:0/3622150962 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9914055a30 con 0x7f99200d1500 2024-04-06T13:45:30.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.994+0000 7f9925a4f700 1 -- 172.21.15.189:0/3622150962 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f99200c7520 con 0x7f99200d1500 2024-04-06T13:45:30.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:30.998+0000 7f9907fff700 1 -- 172.21.15.189:0/3622150962 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v67) v1 ==== 104+0+0 (secure 0 0 0) 0x7f99200c7520 con 0x7f99200d1500 2024-04-06T13:45:31.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.000+0000 7f9925a4f700 1 -- 172.21.15.189:0/3622150962 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f990c06adb0 msgr2=0x7f990c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:31.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.000+0000 7f9925a4f700 1 --2- 172.21.15.189:0/3622150962 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f990c06adb0 0x7f990c06d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9910006b20 tx=0x7f9910008040 comp rx=0 tx=0).stop 2024-04-06T13:45:31.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.000+0000 7f9925a4f700 1 -- 172.21.15.189:0/3622150962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99200d1500 msgr2=0x7f99201f2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:31.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.000+0000 7f9925a4f700 1 --2- 172.21.15.189:0/3622150962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99200d1500 0x7f99201f2f80 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f9914009c40 tx=0x7f9914004750 comp rx=0 tx=0).stop 2024-04-06T13:45:31.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.000+0000 7f9925a4f700 1 -- 172.21.15.189:0/3622150962 shutdown_connections 2024-04-06T13:45:31.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.000+0000 7f9925a4f700 1 --2- 172.21.15.189:0/3622150962 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f990c06adb0 0x7f990c06d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.000+0000 7f9925a4f700 1 --2- 172.21.15.189:0/3622150962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f992015d470 0x7f99200c7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.000+0000 7f9925a4f700 1 --2- 172.21.15.189:0/3622150962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99200b6560 0x7f99200c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.000+0000 7f9925a4f700 1 --2- 172.21.15.189:0/3622150962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99200d1500 0x7f99201f2f80 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.000+0000 7f9925a4f700 1 -- 172.21.15.189:0/3622150962 >> 172.21.15.189:0/3622150962 conn(0x7f992006bbc0 msgr2=0x7f9920157bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:31.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.000+0000 7f9925a4f700 1 -- 172.21.15.189:0/3622150962 shutdown_connections 2024-04-06T13:45:31.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.001+0000 7f9925a4f700 1 -- 172.21.15.189:0/3622150962 wait complete. 2024-04-06T13:45:31.014 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-06T13:45:31.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.136+0000 7f5c24f40700 1 Processor -- start 2024-04-06T13:45:31.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.136+0000 7f5c24f40700 1 -- start start 2024-04-06T13:45:31.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.137+0000 7f5c24f40700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c2015d7b0 0x7f5c200b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.137+0000 7f5c24f40700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5c2006a730 0x7f5c2006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.137+0000 7f5c24f40700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5c2015ddf0 0x7f5c2015e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.137+0000 7f5c24f40700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c2006b0f0 con 0x7f5c2015ddf0 2024-04-06T13:45:31.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.137+0000 7f5c24f40700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c200b3990 con 0x7f5c2006a730 2024-04-06T13:45:31.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.137+0000 7f5c24f40700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c200b3060 con 0x7f5c2015d7b0 2024-04-06T13:45:31.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.137+0000 7f5c1ed9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5c2015ddf0 0x7f5c2015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:31.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.137+0000 7f5c1ed9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5c2015ddf0 0x7f5c2015e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56416/0 (socket says 172.21.15.189:56416) 2024-04-06T13:45:31.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.137+0000 7f5c1ed9d700 1 -- 172.21.15.189:0/970451857 learned_addr learned my addr 172.21.15.189:0/970451857 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:31.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.137+0000 7f5c1e59c700 1 --2- 172.21.15.189:0/970451857 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c2015d7b0 0x7f5c200b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:31.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.137+0000 7f5c1dd9b700 1 --2- 172.21.15.189:0/970451857 >> v2:172.21.15.192:3300/0 conn(0x7f5c2006a730 0x7f5c2006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:31.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.137+0000 7f5c1ed9d700 1 -- 172.21.15.189:0/970451857 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c2015d7b0 msgr2=0x7f5c200b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:31.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.137+0000 7f5c1ed9d700 1 --2- 172.21.15.189:0/970451857 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c2015d7b0 0x7f5c200b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.137+0000 7f5c1ed9d700 1 -- 172.21.15.189:0/970451857 >> v2:172.21.15.192:3300/0 conn(0x7f5c2006a730 msgr2=0x7f5c2006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:31.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.137+0000 7f5c1ed9d700 1 --2- 172.21.15.189:0/970451857 >> v2:172.21.15.192:3300/0 conn(0x7f5c2006a730 0x7f5c2006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.137+0000 7f5c1ed9d700 1 -- 172.21.15.189:0/970451857 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c200b3340 con 0x7f5c2015ddf0 2024-04-06T13:45:31.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.137+0000 7f5c1ed9d700 1 --2- 172.21.15.189:0/970451857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2015ddf0 0x7f5c2015e280 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f5c2006cb90 tx=0x7f5c1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d1e0909e958663 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:31.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.137+0000 7f5c1d59a700 1 -- 172.21.15.189:0/970451857 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1800e070 con 0x7f5c2015ddf0 2024-04-06T13:45:31.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.138+0000 7f5c1d59a700 1 -- 172.21.15.189:0/970451857 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c1800b9c0 con 0x7f5c2015ddf0 2024-04-06T13:45:31.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.138+0000 7f5c1d59a700 1 -- 172.21.15.189:0/970451857 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1800bd70 con 0x7f5c2015ddf0 2024-04-06T13:45:31.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.138+0000 7f5c24f40700 1 -- 172.21.15.189:0/970451857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2015ddf0 msgr2=0x7f5c2015e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:31.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.138+0000 7f5c24f40700 1 --2- 172.21.15.189:0/970451857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2015ddf0 0x7f5c2015e280 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f5c2006cb90 tx=0x7f5c1800b330 comp rx=0 tx=0).stop 2024-04-06T13:45:31.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.138+0000 7f5c24f40700 1 -- 172.21.15.189:0/970451857 shutdown_connections 2024-04-06T13:45:31.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.138+0000 7f5c24f40700 1 --2- 172.21.15.189:0/970451857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2015ddf0 0x7f5c2015e280 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.138+0000 7f5c24f40700 1 --2- 172.21.15.189:0/970451857 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c2015d7b0 0x7f5c200b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.138+0000 7f5c24f40700 1 --2- 172.21.15.189:0/970451857 >> v2:172.21.15.192:3300/0 conn(0x7f5c2006a730 0x7f5c2006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.138+0000 7f5c24f40700 1 -- 172.21.15.189:0/970451857 >> 172.21.15.189:0/970451857 conn(0x7f5c200d00b0 msgr2=0x7f5c200d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:31.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.138+0000 7f5c24f40700 1 -- 172.21.15.189:0/970451857 shutdown_connections 2024-04-06T13:45:31.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.138+0000 7f5c24f40700 1 -- 172.21.15.189:0/970451857 wait complete. 2024-04-06T13:45:31.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.140+0000 7f5c24f40700 1 Processor -- start 2024-04-06T13:45:31.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.140+0000 7f5c24f40700 1 -- start start 2024-04-06T13:45:31.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.140+0000 7f5c24f40700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c2006a730 0x7f5c20149ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.140+0000 7f5c24f40700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2015d7b0 0x7f5c2015b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.140+0000 7f5c24f40700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c2015ddf0 0x7f5c2015fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.140+0000 7f5c24f40700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c200cc510 con 0x7f5c2015d7b0 2024-04-06T13:45:31.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.140+0000 7f5c24f40700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2015c970 con 0x7f5c2006a730 2024-04-06T13:45:31.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.140+0000 7f5c24f40700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2015c0d0 con 0x7f5c2015ddf0 2024-04-06T13:45:31.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.140+0000 7f5c1dd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2015d7b0 0x7f5c2015b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:31.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.140+0000 7f5c1dd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2015d7b0 0x7f5c2015b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56432/0 (socket says 172.21.15.189:56432) 2024-04-06T13:45:31.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.141+0000 7f5c1dd9b700 1 -- 172.21.15.189:0/1132153573 learned_addr learned my addr 172.21.15.189:0/1132153573 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:31.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.141+0000 7f5c1ed9d700 1 --2- 172.21.15.189:0/1132153573 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c2015ddf0 0x7f5c2015fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:31.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.141+0000 7f5c1e59c700 1 --2- 172.21.15.189:0/1132153573 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c2006a730 0x7f5c20149ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:31.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.141+0000 7f5c1dd9b700 1 -- 172.21.15.189:0/1132153573 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c2015ddf0 msgr2=0x7f5c2015fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:31.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.141+0000 7f5c1dd9b700 1 --2- 172.21.15.189:0/1132153573 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c2015ddf0 0x7f5c2015fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.141+0000 7f5c1dd9b700 1 -- 172.21.15.189:0/1132153573 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c2006a730 msgr2=0x7f5c20149ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:31.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.141+0000 7f5c1dd9b700 1 --2- 172.21.15.189:0/1132153573 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c2006a730 0x7f5c20149ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.141+0000 7f5c1dd9b700 1 -- 172.21.15.189:0/1132153573 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c18009b30 con 0x7f5c2015d7b0 2024-04-06T13:45:31.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.141+0000 7f5c1e59c700 1 --2- 172.21.15.189:0/1132153573 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c2006a730 0x7f5c20149ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:31.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.141+0000 7f5c1dd9b700 1 --2- 172.21.15.189:0/1132153573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2015d7b0 0x7f5c2015b610 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f5c1400e810 tx=0x7f5c1400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:31.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.141+0000 7f5c0effd700 1 -- 172.21.15.189:0/1132153573 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c14013610 con 0x7f5c2015d7b0 2024-04-06T13:45:31.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.141+0000 7f5c0effd700 1 -- 172.21.15.189:0/1132153573 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c14013770 con 0x7f5c2015d7b0 2024-04-06T13:45:31.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.141+0000 7f5c24f40700 1 -- 172.21.15.189:0/1132153573 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c2015c3b0 con 0x7f5c2015d7b0 2024-04-06T13:45:31.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.141+0000 7f5c0effd700 1 -- 172.21.15.189:0/1132153573 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c14013b20 con 0x7f5c2015d7b0 2024-04-06T13:45:31.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.141+0000 7f5c24f40700 1 -- 172.21.15.189:0/1132153573 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c200cf8d0 con 0x7f5c2015d7b0 2024-04-06T13:45:31.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.142+0000 7f5c24f40700 1 -- 172.21.15.189:0/1132153573 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c20067bb0 con 0x7f5c2015d7b0 2024-04-06T13:45:31.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.143+0000 7f5c0effd700 1 -- 172.21.15.189:0/1132153573 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f5c14013c80 con 0x7f5c2015d7b0 2024-04-06T13:45:31.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.143+0000 7f5c0effd700 1 --2- 172.21.15.189:0/1132153573 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5c0006adb0 0x7f5c0006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.143+0000 7f5c0effd700 1 -- 172.21.15.189:0/1132153573 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6281+0+0 (secure 0 0 0) 0x7f5c14089880 con 0x7f5c2015d7b0 2024-04-06T13:45:31.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.143+0000 7f5c1e59c700 1 --2- 172.21.15.189:0/1132153573 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5c0006adb0 0x7f5c0006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:31.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.144+0000 7f5c1e59c700 1 --2- 172.21.15.189:0/1132153573 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5c0006adb0 0x7f5c0006d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5c10009970 tx=0x7f5c10008040 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:31.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.147+0000 7f5c0effd700 1 -- 172.21.15.189:0/1132153573 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5c14054690 con 0x7f5c2015d7b0 2024-04-06T13:45:31.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.339+0000 7f5c24f40700 1 -- 172.21.15.189:0/1132153573 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f5c20074990 con 0x7f5c2015d7b0 2024-04-06T13:45:31.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.343+0000 7f5c0effd700 1 -- 172.21.15.189:0/1132153573 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v68) v1 ==== 115+0+0 (secure 0 0 0) 0x7f5c14057cb0 con 0x7f5c2015d7b0 2024-04-06T13:45:31.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.346+0000 7f5c24f40700 1 -- 172.21.15.189:0/1132153573 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5c0006adb0 msgr2=0x7f5c0006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:31.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.346+0000 7f5c24f40700 1 --2- 172.21.15.189:0/1132153573 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5c0006adb0 0x7f5c0006d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5c10009970 tx=0x7f5c10008040 comp rx=0 tx=0).stop 2024-04-06T13:45:31.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.346+0000 7f5c24f40700 1 -- 172.21.15.189:0/1132153573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2015d7b0 msgr2=0x7f5c2015b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:31.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.346+0000 7f5c24f40700 1 --2- 172.21.15.189:0/1132153573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2015d7b0 0x7f5c2015b610 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f5c1400e810 tx=0x7f5c1400eb20 comp rx=0 tx=0).stop 2024-04-06T13:45:31.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.346+0000 7f5c24f40700 1 -- 172.21.15.189:0/1132153573 shutdown_connections 2024-04-06T13:45:31.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.346+0000 7f5c24f40700 1 --2- 172.21.15.189:0/1132153573 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5c0006adb0 0x7f5c0006d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.346+0000 7f5c24f40700 1 --2- 172.21.15.189:0/1132153573 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c2015ddf0 0x7f5c2015fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.346+0000 7f5c24f40700 1 --2- 172.21.15.189:0/1132153573 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c2006a730 0x7f5c20149ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.346+0000 7f5c24f40700 1 --2- 172.21.15.189:0/1132153573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2015d7b0 0x7f5c2015b610 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.346+0000 7f5c24f40700 1 -- 172.21.15.189:0/1132153573 >> 172.21.15.189:0/1132153573 conn(0x7f5c200d00b0 msgr2=0x7f5c200b15b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:31.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.346+0000 7f5c24f40700 1 -- 172.21.15.189:0/1132153573 shutdown_connections 2024-04-06T13:45:31.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.346+0000 7f5c24f40700 1 -- 172.21.15.189:0/1132153573 wait complete. 2024-04-06T13:45:31.359 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-06T13:45:31.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.481+0000 7f57e8dc8700 1 Processor -- start 2024-04-06T13:45:31.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.481+0000 7f57e8dc8700 1 -- start start 2024-04-06T13:45:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.482+0000 7f57e8dc8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57e415d7b0 0x7f57e40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.482+0000 7f57e8dc8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f57e406a730 0x7f57e406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.482+0000 7f57e8dc8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f57e415ddf0 0x7f57e415e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.482+0000 7f57e8dc8700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57e406b0f0 con 0x7f57e415ddf0 2024-04-06T13:45:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.482+0000 7f57e8dc8700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57e40b3990 con 0x7f57e406a730 2024-04-06T13:45:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.482+0000 7f57e8dc8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e40b3060 con 0x7f57e415d7b0 2024-04-06T13:45:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.482+0000 7f57e2d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f57e415ddf0 0x7f57e415e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.482+0000 7f57e2d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f57e415ddf0 0x7f57e415e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56440/0 (socket says 172.21.15.189:56440) 2024-04-06T13:45:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.482+0000 7f57e2d9d700 1 -- 172.21.15.189:0/602869287 learned_addr learned my addr 172.21.15.189:0/602869287 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.482+0000 7f57e259c700 1 --2- 172.21.15.189:0/602869287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57e415d7b0 0x7f57e40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.483+0000 7f57e2d9d700 1 -- 172.21.15.189:0/602869287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57e415d7b0 msgr2=0x7f57e40b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:31.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.483+0000 7f57e2d9d700 1 --2- 172.21.15.189:0/602869287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57e415d7b0 0x7f57e40b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.483+0000 7f57e2d9d700 1 -- 172.21.15.189:0/602869287 >> v2:172.21.15.192:3300/0 conn(0x7f57e406a730 msgr2=0x7f57e406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:31.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.483+0000 7f57e2d9d700 1 --2- 172.21.15.189:0/602869287 >> v2:172.21.15.192:3300/0 conn(0x7f57e406a730 0x7f57e406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.483+0000 7f57e2d9d700 1 -- 172.21.15.189:0/602869287 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e40b3340 con 0x7f57e415ddf0 2024-04-06T13:45:31.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.483+0000 7f57e2d9d700 1 --2- 172.21.15.189:0/602869287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57e415ddf0 0x7f57e415e280 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f57e406cb90 tx=0x7f57dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30850e711cdab4bb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:31.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.483+0000 7f57e159a700 1 -- 172.21.15.189:0/602869287 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc00e070 con 0x7f57e415ddf0 2024-04-06T13:45:31.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.483+0000 7f57e159a700 1 -- 172.21.15.189:0/602869287 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57dc00b9c0 con 0x7f57e415ddf0 2024-04-06T13:45:31.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.483+0000 7f57e159a700 1 -- 172.21.15.189:0/602869287 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc00bd70 con 0x7f57e415ddf0 2024-04-06T13:45:31.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.483+0000 7f57e8dc8700 1 -- 172.21.15.189:0/602869287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57e415ddf0 msgr2=0x7f57e415e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:31.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.483+0000 7f57e8dc8700 1 --2- 172.21.15.189:0/602869287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57e415ddf0 0x7f57e415e280 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f57e406cb90 tx=0x7f57dc00b330 comp rx=0 tx=0).stop 2024-04-06T13:45:31.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.483+0000 7f57e8dc8700 1 -- 172.21.15.189:0/602869287 shutdown_connections 2024-04-06T13:45:31.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.483+0000 7f57e8dc8700 1 --2- 172.21.15.189:0/602869287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57e415ddf0 0x7f57e415e280 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.483+0000 7f57e8dc8700 1 --2- 172.21.15.189:0/602869287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57e415d7b0 0x7f57e40b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.483+0000 7f57e8dc8700 1 --2- 172.21.15.189:0/602869287 >> v2:172.21.15.192:3300/0 conn(0x7f57e406a730 0x7f57e406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.483+0000 7f57e8dc8700 1 -- 172.21.15.189:0/602869287 >> 172.21.15.189:0/602869287 conn(0x7f57e40d00b0 msgr2=0x7f57e40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:31.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.484+0000 7f57e8dc8700 1 -- 172.21.15.189:0/602869287 shutdown_connections 2024-04-06T13:45:31.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.484+0000 7f57e8dc8700 1 -- 172.21.15.189:0/602869287 wait complete. 2024-04-06T13:45:31.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.485+0000 7f57e8dc8700 1 Processor -- start 2024-04-06T13:45:31.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.485+0000 7f57e8dc8700 1 -- start start 2024-04-06T13:45:31.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57e8dc8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57e406a730 0x7f57e4148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57e8dc8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57e415d7b0 0x7f57e414ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57e8dc8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57e415ddf0 0x7f57e415b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57e8dc8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e415b8f0 con 0x7f57e415d7b0 2024-04-06T13:45:31.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57e8dc8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e415bbd0 con 0x7f57e406a730 2024-04-06T13:45:31.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57e8dc8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e415beb0 con 0x7f57e415ddf0 2024-04-06T13:45:31.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57e1d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57e415d7b0 0x7f57e414ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:31.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57e1d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57e415d7b0 0x7f57e414ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56454/0 (socket says 172.21.15.189:56454) 2024-04-06T13:45:31.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57e1d9b700 1 -- 172.21.15.189:0/2768260478 learned_addr learned my addr 172.21.15.189:0/2768260478 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:31.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57e2d9d700 1 --2- 172.21.15.189:0/2768260478 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57e415ddf0 0x7f57e415b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:31.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57e259c700 1 --2- 172.21.15.189:0/2768260478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57e406a730 0x7f57e4148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:31.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57e1d9b700 1 -- 172.21.15.189:0/2768260478 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57e415ddf0 msgr2=0x7f57e415b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:31.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57e1d9b700 1 --2- 172.21.15.189:0/2768260478 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57e415ddf0 0x7f57e415b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57e1d9b700 1 -- 172.21.15.189:0/2768260478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57e406a730 msgr2=0x7f57e4148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:31.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57e1d9b700 1 --2- 172.21.15.189:0/2768260478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57e406a730 0x7f57e4148ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57e1d9b700 1 -- 172.21.15.189:0/2768260478 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57dc009b30 con 0x7f57e415d7b0 2024-04-06T13:45:31.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57e1d9b700 1 --2- 172.21.15.189:0/2768260478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57e415d7b0 0x7f57e414ef80 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f57d400b530 tx=0x7f57d400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:31.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57d2ffd700 1 -- 172.21.15.189:0/2768260478 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d4011830 con 0x7f57e415d7b0 2024-04-06T13:45:31.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57d2ffd700 1 -- 172.21.15.189:0/2768260478 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57d40119b0 con 0x7f57e415d7b0 2024-04-06T13:45:31.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57e8dc8700 1 -- 172.21.15.189:0/2768260478 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e414cbb0 con 0x7f57e415d7b0 2024-04-06T13:45:31.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57d2ffd700 1 -- 172.21.15.189:0/2768260478 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d4011d60 con 0x7f57e415d7b0 2024-04-06T13:45:31.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.486+0000 7f57e8dc8700 1 -- 172.21.15.189:0/2768260478 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e40c4f30 con 0x7f57e415d7b0 2024-04-06T13:45:31.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.487+0000 7f57e8dc8700 1 -- 172.21.15.189:0/2768260478 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e414ca10 con 0x7f57e415d7b0 2024-04-06T13:45:31.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.488+0000 7f57d2ffd700 1 -- 172.21.15.189:0/2768260478 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f57d401a400 con 0x7f57e415d7b0 2024-04-06T13:45:31.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.488+0000 7f57d2ffd700 1 --2- 172.21.15.189:0/2768260478 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f57c406adb0 0x7f57c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.488+0000 7f57d2ffd700 1 -- 172.21.15.189:0/2768260478 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6281+0+0 (secure 0 0 0) 0x7f57d4089690 con 0x7f57e415d7b0 2024-04-06T13:45:31.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.488+0000 7f57e259c700 1 --2- 172.21.15.189:0/2768260478 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f57c406adb0 0x7f57c406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:31.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.489+0000 7f57e259c700 1 --2- 172.21.15.189:0/2768260478 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f57c406adb0 0x7f57c406d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f57d800a990 tx=0x7f57d8008040 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:31.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.492+0000 7f57d2ffd700 1 -- 172.21.15.189:0/2768260478 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f57e414ca10 con 0x7f57e415d7b0 2024-04-06T13:45:31.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.688+0000 7f57e8dc8700 1 -- 172.21.15.189:0/2768260478 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f57e414ca10 con 0x7f57e415d7b0 2024-04-06T13:45:31.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.692+0000 7f57d2ffd700 1 -- 172.21.15.189:0/2768260478 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v69) v1 ==== 120+0+0 (secure 0 0 0) 0x7f57e414ca10 con 0x7f57e415d7b0 2024-04-06T13:45:31.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.694+0000 7f57e8dc8700 1 -- 172.21.15.189:0/2768260478 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f57c406adb0 msgr2=0x7f57c406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:31.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.694+0000 7f57e8dc8700 1 --2- 172.21.15.189:0/2768260478 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f57c406adb0 0x7f57c406d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f57d800a990 tx=0x7f57d8008040 comp rx=0 tx=0).stop 2024-04-06T13:45:31.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.694+0000 7f57e8dc8700 1 -- 172.21.15.189:0/2768260478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57e415d7b0 msgr2=0x7f57e414ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:31.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.694+0000 7f57e8dc8700 1 --2- 172.21.15.189:0/2768260478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57e415d7b0 0x7f57e414ef80 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f57d400b530 tx=0x7f57d400b8f0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.694+0000 7f57e8dc8700 1 -- 172.21.15.189:0/2768260478 shutdown_connections 2024-04-06T13:45:31.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.694+0000 7f57e8dc8700 1 --2- 172.21.15.189:0/2768260478 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f57c406adb0 0x7f57c406d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.694+0000 7f57e8dc8700 1 --2- 172.21.15.189:0/2768260478 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57e415ddf0 0x7f57e415b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.694+0000 7f57e8dc8700 1 --2- 172.21.15.189:0/2768260478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57e406a730 0x7f57e4148ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.694+0000 7f57e8dc8700 1 --2- 172.21.15.189:0/2768260478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57e415d7b0 0x7f57e414ef80 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.694+0000 7f57e8dc8700 1 -- 172.21.15.189:0/2768260478 >> 172.21.15.189:0/2768260478 conn(0x7f57e40d00b0 msgr2=0x7f57e406fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:31.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.694+0000 7f57e8dc8700 1 -- 172.21.15.189:0/2768260478 shutdown_connections 2024-04-06T13:45:31.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.694+0000 7f57e8dc8700 1 -- 172.21.15.189:0/2768260478 wait complete. 2024-04-06T13:45:31.707 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2024-04-06T13:45:31.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.828+0000 7f75b0cd0700 1 Processor -- start 2024-04-06T13:45:31.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.828+0000 7f75b0cd0700 1 -- start start 2024-04-06T13:45:31.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.829+0000 7f75b0cd0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f75ac0d17a0 0x7f75ac06b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.829+0000 7f75b0cd0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75ac0cedf0 0x7f75ac0cf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.829+0000 7f75b0cd0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75ac0cf780 con 0x7f75ac146dd0 2024-04-06T13:45:31.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.829+0000 7f75b0cd0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75ac06d850 con 0x7f75ac0d17a0 2024-04-06T13:45:31.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.829+0000 7f75b0cd0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac06c0a0 con 0x7f75ac0cedf0 2024-04-06T13:45:31.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.829+0000 7f75aa59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f75ac0d17a0 0x7f75ac06b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:31.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.829+0000 7f75aa59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f75ac0d17a0 0x7f75ac06b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56460/0 (socket says 172.21.15.189:56460) 2024-04-06T13:45:31.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.829+0000 7f75aa59c700 1 -- 172.21.15.189:0/3450682860 learned_addr learned my addr 172.21.15.189:0/3450682860 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:31.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.829+0000 7f75aad9d700 1 --2- 172.21.15.189:0/3450682860 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75ac0cedf0 0x7f75ac0cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:31.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.829+0000 7f75aa59c700 1 -- 172.21.15.189:0/3450682860 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75ac0cedf0 msgr2=0x7f75ac0cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:31.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.829+0000 7f75aa59c700 1 --2- 172.21.15.189:0/3450682860 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75ac0cedf0 0x7f75ac0cf240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.829+0000 7f75aa59c700 1 -- 172.21.15.189:0/3450682860 >> v1:172.21.15.192:6789/0 conn(0x7f75ac146dd0 legacy=0x7f75ac06d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:31.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.829+0000 7f75aa59c700 1 -- 172.21.15.189:0/3450682860 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ac06c380 con 0x7f75ac0d17a0 2024-04-06T13:45:31.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.829+0000 7f75aad9d700 1 --2- 172.21.15.189:0/3450682860 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75ac0cedf0 0x7f75ac0cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:45:31.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.829+0000 7f75aa59c700 1 --2- 172.21.15.189:0/3450682860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ac0d17a0 0x7f75ac06b7a0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f759c0098f0 tx=0x7f759c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff70726c613c7f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:31.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.829+0000 7f75a959a700 1 -- 172.21.15.189:0/3450682860 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c004230 con 0x7f75ac0d17a0 2024-04-06T13:45:31.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.829+0000 7f75a959a700 1 -- 172.21.15.189:0/3450682860 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759c004390 con 0x7f75ac0d17a0 2024-04-06T13:45:31.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.829+0000 7f75a959a700 1 -- 172.21.15.189:0/3450682860 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c0036a0 con 0x7f75ac0d17a0 2024-04-06T13:45:31.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.830+0000 7f75b0cd0700 1 -- 172.21.15.189:0/3450682860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ac0d17a0 msgr2=0x7f75ac06b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:31.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.830+0000 7f75b0cd0700 1 --2- 172.21.15.189:0/3450682860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ac0d17a0 0x7f75ac06b7a0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f759c0098f0 tx=0x7f759c009c00 comp rx=0 tx=0).stop 2024-04-06T13:45:31.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.830+0000 7f75b0cd0700 1 -- 172.21.15.189:0/3450682860 shutdown_connections 2024-04-06T13:45:31.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.830+0000 7f75b0cd0700 1 --2- 172.21.15.189:0/3450682860 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75ac0cedf0 0x7f75ac0cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.830+0000 7f75b0cd0700 1 --2- 172.21.15.189:0/3450682860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ac0d17a0 0x7f75ac06b7a0 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.830+0000 7f75b0cd0700 1 -- 172.21.15.189:0/3450682860 >> 172.21.15.189:0/3450682860 conn(0x7f75ac0b2db0 msgr2=0x7f75ac0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:31.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.830+0000 7f75b0cd0700 1 -- 172.21.15.189:0/3450682860 shutdown_connections 2024-04-06T13:45:31.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.830+0000 7f75b0cd0700 1 -- 172.21.15.189:0/3450682860 wait complete. 2024-04-06T13:45:31.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.831+0000 7f75b0cd0700 1 Processor -- start 2024-04-06T13:45:31.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.831+0000 7f75b0cd0700 1 -- start start 2024-04-06T13:45:31.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.832+0000 7f75b0cd0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75ac0cedf0 0x7f75ac0c6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.832+0000 7f75b0cd0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75ac0d17a0 0x7f75ac0c4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.832+0000 7f75b0cd0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ac146dd0 0x7f75ac0ae260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.832+0000 7f75b0cd0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac0c6dc0 con 0x7f75ac146dd0 2024-04-06T13:45:31.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.832+0000 7f75b0cd0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac0c57e0 con 0x7f75ac0cedf0 2024-04-06T13:45:31.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.832+0000 7f75b0cd0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac0c5ac0 con 0x7f75ac0d17a0 2024-04-06T13:45:31.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.832+0000 7f75aad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ac146dd0 0x7f75ac0ae260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:31.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.832+0000 7f75aad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ac146dd0 0x7f75ac0ae260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56462/0 (socket says 172.21.15.189:56462) 2024-04-06T13:45:31.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.832+0000 7f75aad9d700 1 -- 172.21.15.189:0/3256045910 learned_addr learned my addr 172.21.15.189:0/3256045910 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:31.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.832+0000 7f75a9d9b700 1 --2- 172.21.15.189:0/3256045910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75ac0d17a0 0x7f75ac0c4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:31.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.832+0000 7f75aa59c700 1 --2- 172.21.15.189:0/3256045910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75ac0cedf0 0x7f75ac0c6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:31.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.832+0000 7f75aad9d700 1 -- 172.21.15.189:0/3256045910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75ac0d17a0 msgr2=0x7f75ac0c4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:31.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.832+0000 7f75aad9d700 1 --2- 172.21.15.189:0/3256045910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75ac0d17a0 0x7f75ac0c4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.832+0000 7f75aad9d700 1 -- 172.21.15.189:0/3256045910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75ac0cedf0 msgr2=0x7f75ac0c6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:31.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.832+0000 7f75aad9d700 1 --2- 172.21.15.189:0/3256045910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75ac0cedf0 0x7f75ac0c6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:31.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.833+0000 7f75aad9d700 1 -- 172.21.15.189:0/3256045910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c009550 con 0x7f75ac146dd0 2024-04-06T13:45:31.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.833+0000 7f75aa59c700 1 --2- 172.21.15.189:0/3256045910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75ac0cedf0 0x7f75ac0c6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:31.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.833+0000 7f75aad9d700 1 --2- 172.21.15.189:0/3256045910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ac146dd0 0x7f75ac0ae260 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f75a400cd50 tx=0x7f75a400a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:31.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.833+0000 7f759affd700 1 -- 172.21.15.189:0/3256045910 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a400cea0 con 0x7f75ac146dd0 2024-04-06T13:45:31.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.833+0000 7f759affd700 1 -- 172.21.15.189:0/3256045910 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a400f480 con 0x7f75ac146dd0 2024-04-06T13:45:31.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.833+0000 7f75b0cd0700 1 -- 172.21.15.189:0/3256045910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75ac0c5da0 con 0x7f75ac146dd0 2024-04-06T13:45:31.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.833+0000 7f759affd700 1 -- 172.21.15.189:0/3256045910 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a400f810 con 0x7f75ac146dd0 2024-04-06T13:45:31.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.833+0000 7f75b0cd0700 1 -- 172.21.15.189:0/3256045910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ac0c6360 con 0x7f75ac146dd0 2024-04-06T13:45:31.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.834+0000 7f75b0cd0700 1 -- 172.21.15.189:0/3256045910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75ac0c6580 con 0x7f75ac146dd0 2024-04-06T13:45:31.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.834+0000 7f759affd700 1 -- 172.21.15.189:0/3256045910 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f75a400b040 con 0x7f75ac146dd0 2024-04-06T13:45:31.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.834+0000 7f759affd700 1 --2- 172.21.15.189:0/3256045910 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f758c06adb0 0x7f758c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:31.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.835+0000 7f759affd700 1 -- 172.21.15.189:0/3256045910 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6281+0+0 (secure 0 0 0) 0x7f75a4089f10 con 0x7f75ac146dd0 2024-04-06T13:45:31.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.835+0000 7f75aa59c700 1 --2- 172.21.15.189:0/3256045910 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f758c06adb0 0x7f758c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:31.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.835+0000 7f75aa59c700 1 --2- 172.21.15.189:0/3256045910 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f758c06adb0 0x7f758c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f759c009130 tx=0x7f759c008030 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:31.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:31.839+0000 7f759affd700 1 -- 172.21.15.189:0/3256045910 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f75a4054c90 con 0x7f75ac146dd0 2024-04-06T13:45:32.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.117+0000 7f75b0cd0700 1 -- 172.21.15.189:0/3256045910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7f75ac074990 con 0x7f75ac146dd0 2024-04-06T13:45:32.565 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:45:32.568 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:45:32.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.632+0000 7f759affd700 1 -- 172.21.15.189:0/3256045910 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v3031) v1 ==== 165+0+0 (secure 0 0 0) 0x7f75a40582b0 con 0x7f75ac146dd0 2024-04-06T13:45:32.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.634+0000 7f75b0cd0700 1 -- 172.21.15.189:0/3256045910 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f758c06adb0 msgr2=0x7f758c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:32.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.634+0000 7f75b0cd0700 1 --2- 172.21.15.189:0/3256045910 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f758c06adb0 0x7f758c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f759c009130 tx=0x7f759c008030 comp rx=0 tx=0).stop 2024-04-06T13:45:32.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.634+0000 7f75b0cd0700 1 -- 172.21.15.189:0/3256045910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ac146dd0 msgr2=0x7f75ac0ae260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:32.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.634+0000 7f75b0cd0700 1 --2- 172.21.15.189:0/3256045910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ac146dd0 0x7f75ac0ae260 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f75a400cd50 tx=0x7f75a400a560 comp rx=0 tx=0).stop 2024-04-06T13:45:32.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.634+0000 7f75b0cd0700 1 -- 172.21.15.189:0/3256045910 shutdown_connections 2024-04-06T13:45:32.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.634+0000 7f75b0cd0700 1 --2- 172.21.15.189:0/3256045910 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f758c06adb0 0x7f758c06d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:32.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.634+0000 7f75b0cd0700 1 --2- 172.21.15.189:0/3256045910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ac146dd0 0x7f75ac0ae260 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:32.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.634+0000 7f75b0cd0700 1 --2- 172.21.15.189:0/3256045910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75ac0cedf0 0x7f75ac0c6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:32.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.634+0000 7f75b0cd0700 1 --2- 172.21.15.189:0/3256045910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75ac0d17a0 0x7f75ac0c4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:32.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.634+0000 7f75b0cd0700 1 -- 172.21.15.189:0/3256045910 >> 172.21.15.189:0/3256045910 conn(0x7f75ac0b2db0 msgr2=0x7f75ac15e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:32.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.634+0000 7f75b0cd0700 1 -- 172.21.15.189:0/3256045910 shutdown_connections 2024-04-06T13:45:32.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.634+0000 7f75b0cd0700 1 -- 172.21.15.189:0/3256045910 wait complete. 2024-04-06T13:45:32.649 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2024-04-06T13:45:32.649 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2024-04-06T13:45:32.649 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-06T13:45:32.650 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2024-04-06T13:45:32.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.814+0000 7f0cd9ae1700 1 Processor -- start 2024-04-06T13:45:32.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.814+0000 7f0cd9ae1700 1 -- start start 2024-04-06T13:45:32.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.814+0000 7f0cd9ae1700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0cd406ba20 0x7f0cd4166270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:32.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.814+0000 7f0cd9ae1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd40669d0 con 0x7f0cd40cbc20 2024-04-06T13:45:32.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.814+0000 7f0cd9ae1700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd406ede0 con 0x7f0cd415e8a0 2024-04-06T13:45:32.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.814+0000 7f0cd9ae1700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd415fd00 con 0x7f0cd406ba20 2024-04-06T13:45:32.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd2ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0cd40cbc20 0x7f0cd415f5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48104/0 (socket says 172.21.15.189:48104) 2024-04-06T13:45:32.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd2ffd700 1 -- 172.21.15.189:0/1819099236 learned_addr learned my addr 172.21.15.189:0/1819099236 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:32.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd3fff700 1 --2- 172.21.15.189:0/1819099236 >> v2:172.21.15.192:3300/0 conn(0x7f0cd406ba20 0x7f0cd4166270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:32.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd27fc700 1 -- 172.21.15.189:0/1819099236 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775518534 0 0) 0x7f0cd40669d0 con 0x7f0cd40cbc20 2024-04-06T13:45:32.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd27fc700 1 -- 172.21.15.189:0/1819099236 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cbc0035b0 con 0x7f0cd40cbc20 2024-04-06T13:45:32.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd27fc700 1 -- 172.21.15.189:0/1819099236 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3470035273 0 0) 0x7f0cbc0035b0 con 0x7f0cd40cbc20 2024-04-06T13:45:32.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd27fc700 1 -- 172.21.15.189:0/1819099236 >> v2:172.21.15.192:3300/0 conn(0x7f0cd406ba20 msgr2=0x7f0cd4166270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:32.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd27fc700 1 --2- 172.21.15.189:0/1819099236 >> v2:172.21.15.192:3300/0 conn(0x7f0cd406ba20 0x7f0cd4166270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:32.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd27fc700 1 -- 172.21.15.189:0/1819099236 >> v1:172.21.15.192:6789/0 conn(0x7f0cd415e8a0 legacy=0x7f0cd415eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:32.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd27fc700 1 -- 172.21.15.189:0/1819099236 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd415ec80 con 0x7f0cd40cbc20 2024-04-06T13:45:32.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd27fc700 1 -- 172.21.15.189:0/1819099236 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f0cc8001670 con 0x7f0cd40cbc20 2024-04-06T13:45:32.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd27fc700 1 -- 172.21.15.189:0/1819099236 >> v1:172.21.15.189:6789/0 conn(0x7f0cd40cbc20 legacy=0x7f0cd415f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:32.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd27fc700 1 --2- 172.21.15.189:0/1819099236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cbc003a00 0x7f0cbc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:32.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd27fc700 1 --2- 172.21.15.189:0/1819099236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cbc004850 0x7f0cbc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:32.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd27fc700 1 --2- 172.21.15.189:0/1819099236 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cbc005330 0x7f0cbc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:32.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd27fc700 1 -- 172.21.15.189:0/1819099236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd415fd00 con 0x7f0cbc004850 2024-04-06T13:45:32.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd27fc700 1 -- 172.21.15.189:0/1819099236 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc00c200 con 0x7f0cbc003a00 2024-04-06T13:45:32.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd27fc700 1 -- 172.21.15.189:0/1819099236 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc00c3c0 con 0x7f0cbc005330 2024-04-06T13:45:32.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd2ffd700 1 --2- 172.21.15.189:0/1819099236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cbc004850 0x7f0cbc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:32.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd37fe700 1 --2- 172.21.15.189:0/1819099236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cbc003a00 0x7f0cbc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:32.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.815+0000 7f0cd3fff700 1 --2- 172.21.15.189:0/1819099236 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cbc005330 0x7f0cbc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:32.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.816+0000 7f0cd2ffd700 1 -- 172.21.15.189:0/1819099236 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cbc005330 msgr2=0x7f0cbc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:32.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.816+0000 7f0cd2ffd700 1 --2- 172.21.15.189:0/1819099236 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cbc005330 0x7f0cbc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:32.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.816+0000 7f0cd2ffd700 1 -- 172.21.15.189:0/1819099236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cbc003a00 msgr2=0x7f0cbc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:32.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.816+0000 7f0cd2ffd700 1 --2- 172.21.15.189:0/1819099236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cbc003a00 0x7f0cbc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:32.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.816+0000 7f0cd2ffd700 1 -- 172.21.15.189:0/1819099236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cbc0035b0 con 0x7f0cbc004850 2024-04-06T13:45:32.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.816+0000 7f0cd3fff700 1 --2- 172.21.15.189:0/1819099236 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cbc005330 0x7f0cbc00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:32.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.816+0000 7f0cd3fff700 1 -- 172.21.15.189:0/1819099236 reap_dead start 2024-04-06T13:45:32.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.816+0000 7f0cd2ffd700 1 --2- 172.21.15.189:0/1819099236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cbc004850 0x7f0cbc004cd0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f0cc800c210 tx=0x7f0cbc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c5c156df1973331 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:32.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.816+0000 7f0cd27fc700 1 -- 172.21.15.189:0/1819099236 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8007c10 con 0x7f0cbc004850 2024-04-06T13:45:32.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.816+0000 7f0cd27fc700 1 -- 172.21.15.189:0/1819099236 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc800cdd0 con 0x7f0cbc004850 2024-04-06T13:45:32.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.816+0000 7f0cd27fc700 1 -- 172.21.15.189:0/1819099236 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8005ba0 con 0x7f0cbc004850 2024-04-06T13:45:32.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.816+0000 7f0cd9ae1700 1 -- 172.21.15.189:0/1819099236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cbc004850 msgr2=0x7f0cbc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:32.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.816+0000 7f0cd9ae1700 1 --2- 172.21.15.189:0/1819099236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cbc004850 0x7f0cbc004cd0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f0cc800c210 tx=0x7f0cbc005880 comp rx=0 tx=0).stop 2024-04-06T13:45:32.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.816+0000 7f0cd9ae1700 1 -- 172.21.15.189:0/1819099236 shutdown_connections 2024-04-06T13:45:32.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.816+0000 7f0cd9ae1700 1 --2- 172.21.15.189:0/1819099236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cbc004850 0x7f0cbc004cd0 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:32.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.816+0000 7f0cd9ae1700 1 -- 172.21.15.189:0/1819099236 >> 172.21.15.189:0/1819099236 conn(0x7f0cd40cc290 msgr2=0x7f0cd40cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:32.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.816+0000 7f0cd9ae1700 1 -- 172.21.15.189:0/1819099236 shutdown_connections 2024-04-06T13:45:32.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.816+0000 7f0cd9ae1700 1 -- 172.21.15.189:0/1819099236 wait complete. 2024-04-06T13:45:32.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.818+0000 7f0cd9ae1700 1 Processor -- start 2024-04-06T13:45:32.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.818+0000 7f0cd9ae1700 1 -- start start 2024-04-06T13:45:32.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.818+0000 7f0cd9ae1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cbc004850 0x7f0cd41f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:32.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.818+0000 7f0cd9ae1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cd40ced70 0x7f0cd415aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:32.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cd9ae1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cd40c8b30 0x7f0cd40c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:32.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cd9ae1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd40c6660 con 0x7f0cd40ced70 2024-04-06T13:45:32.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cd9ae1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd40c94f0 con 0x7f0cd40c8b30 2024-04-06T13:45:32.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cd9ae1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd40ae3a0 con 0x7f0cbc004850 2024-04-06T13:45:32.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cd2ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cd40ced70 0x7f0cd415aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:32.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cd2ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cd40ced70 0x7f0cd415aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56474/0 (socket says 172.21.15.189:56474) 2024-04-06T13:45:32.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cd2ffd700 1 -- 172.21.15.189:0/71471553 learned_addr learned my addr 172.21.15.189:0/71471553 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:32.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cd37fe700 1 --2- 172.21.15.189:0/71471553 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cbc004850 0x7f0cd41f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:32.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cd3fff700 1 --2- 172.21.15.189:0/71471553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cd40c8b30 0x7f0cd40c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:32.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cd2ffd700 1 -- 172.21.15.189:0/71471553 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cbc004850 msgr2=0x7f0cd41f6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:32.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cd2ffd700 1 --2- 172.21.15.189:0/71471553 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cbc004850 0x7f0cd41f6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:32.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cd2ffd700 1 -- 172.21.15.189:0/71471553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cd40c8b30 msgr2=0x7f0cd40c8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:32.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cd2ffd700 1 --2- 172.21.15.189:0/71471553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cd40c8b30 0x7f0cd40c8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:32.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cd2ffd700 1 -- 172.21.15.189:0/71471553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd40ad540 con 0x7f0cd40ced70 2024-04-06T13:45:32.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cd2ffd700 1 --2- 172.21.15.189:0/71471553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cd40ced70 0x7f0cd415aa40 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f0cc800cfc0 tx=0x7f0cc8001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:32.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cbbfff700 1 -- 172.21.15.189:0/71471553 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8001430 con 0x7f0cd40ced70 2024-04-06T13:45:32.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cbbfff700 1 -- 172.21.15.189:0/71471553 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc8001590 con 0x7f0cd40ced70 2024-04-06T13:45:32.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cd9ae1700 1 -- 172.21.15.189:0/71471553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd40ad790 con 0x7f0cd40ced70 2024-04-06T13:45:32.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cbbfff700 1 -- 172.21.15.189:0/71471553 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8007310 con 0x7f0cd40ced70 2024-04-06T13:45:32.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.819+0000 7f0cd9ae1700 1 -- 172.21.15.189:0/71471553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd40add50 con 0x7f0cd40ced70 2024-04-06T13:45:32.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.820+0000 7f0cd9ae1700 1 -- 172.21.15.189:0/71471553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd4072140 con 0x7f0cd40ced70 2024-04-06T13:45:32.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.821+0000 7f0cbbfff700 1 -- 172.21.15.189:0/71471553 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0cc8007470 con 0x7f0cd40ced70 2024-04-06T13:45:32.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.821+0000 7f0cbbfff700 1 --2- 172.21.15.189:0/71471553 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0cc006adb0 0x7f0cc006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:32.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.821+0000 7f0cbbfff700 1 -- 172.21.15.189:0/71471553 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6281+0+0 (secure 0 0 0) 0x7f0cc808a850 con 0x7f0cd40ced70 2024-04-06T13:45:32.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.821+0000 7f0cd37fe700 1 --2- 172.21.15.189:0/71471553 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0cc006adb0 0x7f0cc006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:32.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.821+0000 7f0cd37fe700 1 --2- 172.21.15.189:0/71471553 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0cc006adb0 0x7f0cc006d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4000c00 tx=0x7f0cc4009040 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:32.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:32.826+0000 7f0cbbfff700 1 -- 172.21.15.189:0/71471553 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0cc80557b0 con 0x7f0cd40ced70 2024-04-06T13:45:33.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.104+0000 7f0cd9ae1700 1 -- 172.21.15.189:0/71471553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f0cd4074990 con 0x7f0cd40ced70 2024-04-06T13:45:33.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.633+0000 7f0cbbfff700 1 -- 172.21.15.189:0/71471553 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3032) v1 ==== 166+0+0 (secure 0 0 0) 0x7f0cc8058dd0 con 0x7f0cd40ced70 2024-04-06T13:45:33.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.635+0000 7f0cd9ae1700 1 -- 172.21.15.189:0/71471553 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0cc006adb0 msgr2=0x7f0cc006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:33.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.635+0000 7f0cd9ae1700 1 --2- 172.21.15.189:0/71471553 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0cc006adb0 0x7f0cc006d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4000c00 tx=0x7f0cc4009040 comp rx=0 tx=0).stop 2024-04-06T13:45:33.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.635+0000 7f0cd9ae1700 1 -- 172.21.15.189:0/71471553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cd40ced70 msgr2=0x7f0cd415aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:33.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.635+0000 7f0cd9ae1700 1 --2- 172.21.15.189:0/71471553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cd40ced70 0x7f0cd415aa40 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f0cc800cfc0 tx=0x7f0cc8001a00 comp rx=0 tx=0).stop 2024-04-06T13:45:33.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.635+0000 7f0cd9ae1700 1 -- 172.21.15.189:0/71471553 shutdown_connections 2024-04-06T13:45:33.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.635+0000 7f0cd9ae1700 1 --2- 172.21.15.189:0/71471553 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0cc006adb0 0x7f0cc006d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:33.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.635+0000 7f0cd9ae1700 1 --2- 172.21.15.189:0/71471553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cd40c8b30 0x7f0cd40c8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:33.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.635+0000 7f0cd9ae1700 1 --2- 172.21.15.189:0/71471553 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cbc004850 0x7f0cd41f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:33.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.635+0000 7f0cd9ae1700 1 --2- 172.21.15.189:0/71471553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cd40ced70 0x7f0cd415aa40 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:33.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.635+0000 7f0cd9ae1700 1 -- 172.21.15.189:0/71471553 >> 172.21.15.189:0/71471553 conn(0x7f0cd40cc290 msgr2=0x7f0cd406d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:33.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.636+0000 7f0cd9ae1700 1 -- 172.21.15.189:0/71471553 shutdown_connections 2024-04-06T13:45:33.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.636+0000 7f0cd9ae1700 1 -- 172.21.15.189:0/71471553 wait complete. 2024-04-06T13:45:33.649 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-06T13:45:33.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.774+0000 7f05b2fc0700 1 Processor -- start 2024-04-06T13:45:33.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.774+0000 7f05b2fc0700 1 -- start start 2024-04-06T13:45:33.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.775+0000 7f05b2fc0700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f05ac15d7b0 0x7f05ac0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:33.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.775+0000 7f05b2fc0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ac06a730 0x7f05ac06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:33.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.775+0000 7f05b2fc0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f05ac15ddf0 0x7f05ac15e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:33.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.775+0000 7f05b2fc0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05ac06b0f0 con 0x7f05ac15ddf0 2024-04-06T13:45:33.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.775+0000 7f05b2fc0700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05ac0b3990 con 0x7f05ac15d7b0 2024-04-06T13:45:33.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.775+0000 7f05b2fc0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac0b3060 con 0x7f05ac06a730 2024-04-06T13:45:33.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.775+0000 7f05b155d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f05ac15ddf0 0x7f05ac15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:33.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.775+0000 7f05b155d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f05ac15ddf0 0x7f05ac15e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56490/0 (socket says 172.21.15.189:56490) 2024-04-06T13:45:33.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.775+0000 7f05b155d700 1 -- 172.21.15.189:0/394866010 learned_addr learned my addr 172.21.15.189:0/394866010 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:33.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.775+0000 7f05abfff700 1 --2- 172.21.15.189:0/394866010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ac06a730 0x7f05ac06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:33.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.775+0000 7f05b0d5c700 1 --2- 172.21.15.189:0/394866010 >> v2:172.21.15.192:3300/0 conn(0x7f05ac15d7b0 0x7f05ac0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:33.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.775+0000 7f05b155d700 1 -- 172.21.15.189:0/394866010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ac06a730 msgr2=0x7f05ac06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:33.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.775+0000 7f05b155d700 1 --2- 172.21.15.189:0/394866010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ac06a730 0x7f05ac06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:33.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.775+0000 7f05b155d700 1 -- 172.21.15.189:0/394866010 >> v2:172.21.15.192:3300/0 conn(0x7f05ac15d7b0 msgr2=0x7f05ac0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:33.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.775+0000 7f05b155d700 1 --2- 172.21.15.189:0/394866010 >> v2:172.21.15.192:3300/0 conn(0x7f05ac15d7b0 0x7f05ac0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:33.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.775+0000 7f05b155d700 1 -- 172.21.15.189:0/394866010 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac0b3340 con 0x7f05ac15ddf0 2024-04-06T13:45:33.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.775+0000 7f05abfff700 1 --2- 172.21.15.189:0/394866010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ac06a730 0x7f05ac06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:45:33.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.776+0000 7f05b155d700 1 --2- 172.21.15.189:0/394866010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05ac15ddf0 0x7f05ac15e280 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f05ac0b6200 tx=0x7f05a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b67eb6dd9836fce9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:33.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.776+0000 7f05ab7fe700 1 -- 172.21.15.189:0/394866010 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a400e070 con 0x7f05ac15ddf0 2024-04-06T13:45:33.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.776+0000 7f05ab7fe700 1 -- 172.21.15.189:0/394866010 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a400b9c0 con 0x7f05ac15ddf0 2024-04-06T13:45:33.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.776+0000 7f05ab7fe700 1 -- 172.21.15.189:0/394866010 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a400bd70 con 0x7f05ac15ddf0 2024-04-06T13:45:33.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.776+0000 7f05b2fc0700 1 -- 172.21.15.189:0/394866010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05ac15ddf0 msgr2=0x7f05ac15e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:33.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.776+0000 7f05b2fc0700 1 --2- 172.21.15.189:0/394866010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05ac15ddf0 0x7f05ac15e280 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f05ac0b6200 tx=0x7f05a400b330 comp rx=0 tx=0).stop 2024-04-06T13:45:33.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.776+0000 7f05b2fc0700 1 -- 172.21.15.189:0/394866010 shutdown_connections 2024-04-06T13:45:33.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.776+0000 7f05b2fc0700 1 --2- 172.21.15.189:0/394866010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05ac15ddf0 0x7f05ac15e280 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:33.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.776+0000 7f05b2fc0700 1 --2- 172.21.15.189:0/394866010 >> v2:172.21.15.192:3300/0 conn(0x7f05ac15d7b0 0x7f05ac0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:33.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.776+0000 7f05b2fc0700 1 --2- 172.21.15.189:0/394866010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ac06a730 0x7f05ac06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:33.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.776+0000 7f05b2fc0700 1 -- 172.21.15.189:0/394866010 >> 172.21.15.189:0/394866010 conn(0x7f05ac0d00b0 msgr2=0x7f05ac0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:33.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.776+0000 7f05b2fc0700 1 -- 172.21.15.189:0/394866010 shutdown_connections 2024-04-06T13:45:33.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.776+0000 7f05b2fc0700 1 -- 172.21.15.189:0/394866010 wait complete. 2024-04-06T13:45:33.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.777+0000 7f05b2fc0700 1 Processor -- start 2024-04-06T13:45:33.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.778+0000 7f05b2fc0700 1 -- start start 2024-04-06T13:45:33.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.778+0000 7f05b2fc0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05ac06a730 0x7f05ac0adc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:33.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.778+0000 7f05b2fc0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05ac15d7b0 0x7f05ac0b1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:33.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.778+0000 7f05b2fc0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ac15ddf0 0x7f05ac0cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:33.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.778+0000 7f05b2fc0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac0cd790 con 0x7f05ac06a730 2024-04-06T13:45:33.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.778+0000 7f05b2fc0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac0aee30 con 0x7f05ac15d7b0 2024-04-06T13:45:33.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.778+0000 7f05b2fc0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac0cc410 con 0x7f05ac15ddf0 2024-04-06T13:45:33.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.779+0000 7f05b0d5c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05ac06a730 0x7f05ac0adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:33.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.779+0000 7f05b0d5c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05ac06a730 0x7f05ac0adc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56500/0 (socket says 172.21.15.189:56500) 2024-04-06T13:45:33.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.779+0000 7f05b0d5c700 1 -- 172.21.15.189:0/4148703482 learned_addr learned my addr 172.21.15.189:0/4148703482 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:33.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.779+0000 7f05b0d5c700 1 -- 172.21.15.189:0/4148703482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ac15ddf0 msgr2=0x7f05ac0cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:33.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.779+0000 7f05abfff700 1 --2- 172.21.15.189:0/4148703482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05ac15d7b0 0x7f05ac0b1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:33.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.779+0000 7f05b155d700 1 --2- 172.21.15.189:0/4148703482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ac15ddf0 0x7f05ac0cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:33.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.779+0000 7f05b0d5c700 1 --2- 172.21.15.189:0/4148703482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ac15ddf0 0x7f05ac0cb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:33.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.779+0000 7f05b0d5c700 1 -- 172.21.15.189:0/4148703482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05ac15d7b0 msgr2=0x7f05ac0b1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:33.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.779+0000 7f05b0d5c700 1 --2- 172.21.15.189:0/4148703482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05ac15d7b0 0x7f05ac0b1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:33.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.779+0000 7f05b0d5c700 1 -- 172.21.15.189:0/4148703482 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a4009b30 con 0x7f05ac06a730 2024-04-06T13:45:33.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.779+0000 7f05abfff700 1 --2- 172.21.15.189:0/4148703482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05ac15d7b0 0x7f05ac0b1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:45:33.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.779+0000 7f05b155d700 1 --2- 172.21.15.189:0/4148703482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ac15ddf0 0x7f05ac0cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:45:33.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.779+0000 7f05b0d5c700 1 --2- 172.21.15.189:0/4148703482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05ac06a730 0x7f05ac0adc10 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f05a000c7f0 tx=0x7f05a000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.779+0000 7f05a97fa700 1 -- 172.21.15.189:0/4148703482 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a0011800 con 0x7f05ac06a730 2024-04-06T13:45:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.779+0000 7f05a97fa700 1 -- 172.21.15.189:0/4148703482 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a0011960 con 0x7f05ac06a730 2024-04-06T13:45:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.779+0000 7f05b2fc0700 1 -- 172.21.15.189:0/4148703482 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ac0cc6f0 con 0x7f05ac06a730 2024-04-06T13:45:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.779+0000 7f05a97fa700 1 -- 172.21.15.189:0/4148703482 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a0011cf0 con 0x7f05ac06a730 2024-04-06T13:45:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.779+0000 7f05b2fc0700 1 -- 172.21.15.189:0/4148703482 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05ac0cccb0 con 0x7f05ac06a730 2024-04-06T13:45:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.780+0000 7f05b2fc0700 1 -- 172.21.15.189:0/4148703482 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05ac067bb0 con 0x7f05ac06a730 2024-04-06T13:45:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.781+0000 7f05a97fa700 1 -- 172.21.15.189:0/4148703482 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f05a0018360 con 0x7f05ac06a730 2024-04-06T13:45:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.781+0000 7f05a97fa700 1 --2- 172.21.15.189:0/4148703482 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f059006adb0 0x7f059006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.781+0000 7f05abfff700 1 --2- 172.21.15.189:0/4148703482 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f059006adb0 0x7f059006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.781+0000 7f05a97fa700 1 -- 172.21.15.189:0/4148703482 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6281+0+0 (secure 0 0 0) 0x7f05a00896f0 con 0x7f05ac06a730 2024-04-06T13:45:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.782+0000 7f05abfff700 1 --2- 172.21.15.189:0/4148703482 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f059006adb0 0x7f059006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f05ac1f5f60 tx=0x7f0598008040 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:33.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.786+0000 7f05a97fa700 1 -- 172.21.15.189:0/4148703482 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f05a0054480 con 0x7f05ac06a730 2024-04-06T13:45:33.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.979+0000 7f05b2fc0700 1 -- 172.21.15.189:0/4148703482 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f05ac0ceb10 con 0x7f05ac06a730 2024-04-06T13:45:33.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.982+0000 7f05a97fa700 1 -- 172.21.15.189:0/4148703482 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v70)=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7f05a0057aa0 con 0x7f05ac06a730 2024-04-06T13:45:33.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.984+0000 7f05b2fc0700 1 -- 172.21.15.189:0/4148703482 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f059006adb0 msgr2=0x7f059006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:33.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.984+0000 7f05b2fc0700 1 --2- 172.21.15.189:0/4148703482 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f059006adb0 0x7f059006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f05ac1f5f60 tx=0x7f0598008040 comp rx=0 tx=0).stop 2024-04-06T13:45:33.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.984+0000 7f05b2fc0700 1 -- 172.21.15.189:0/4148703482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05ac06a730 msgr2=0x7f05ac0adc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:33.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.984+0000 7f05b2fc0700 1 --2- 172.21.15.189:0/4148703482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05ac06a730 0x7f05ac0adc10 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f05a000c7f0 tx=0x7f05a000cbb0 comp rx=0 tx=0).stop 2024-04-06T13:45:33.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.984+0000 7f05b2fc0700 1 -- 172.21.15.189:0/4148703482 shutdown_connections 2024-04-06T13:45:33.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.984+0000 7f05b2fc0700 1 --2- 172.21.15.189:0/4148703482 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f059006adb0 0x7f059006d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:33.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.984+0000 7f05b2fc0700 1 --2- 172.21.15.189:0/4148703482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05ac15ddf0 0x7f05ac0cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:33.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.984+0000 7f05b2fc0700 1 --2- 172.21.15.189:0/4148703482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05ac06a730 0x7f05ac0adc10 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:33.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.984+0000 7f05b2fc0700 1 --2- 172.21.15.189:0/4148703482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05ac15d7b0 0x7f05ac0b1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:33.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.984+0000 7f05b2fc0700 1 -- 172.21.15.189:0/4148703482 >> 172.21.15.189:0/4148703482 conn(0x7f05ac0d00b0 msgr2=0x7f05ac06eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:33.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.984+0000 7f05b2fc0700 1 -- 172.21.15.189:0/4148703482 shutdown_connections 2024-04-06T13:45:33.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:33.985+0000 7f05b2fc0700 1 -- 172.21.15.189:0/4148703482 wait complete. 2024-04-06T13:45:33.997 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T13:45:33.998 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:45:34.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:45:34.078 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T13:45:34.079 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:45:34.079 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:45:34.096 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:45:34.096 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:45:34.096 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:45:34.116 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:45:34.116 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:45:34.116 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:45:34.154 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:45:34.155 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:45:34.156 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:45:34.176 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:45:34.176 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-06T13:45:34.176 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:45:34.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.316+0000 7ff4418c2700 1 Processor -- start 2024-04-06T13:45:34.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.316+0000 7ff4418c2700 1 -- start start 2024-04-06T13:45:34.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.317+0000 7ff4418c2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff43c15d590 0x7ff43c0b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:34.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.317+0000 7ff4418c2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff43c0d3890 0x7ff43c0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:34.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.317+0000 7ff4418c2700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43c0b5cc0 con 0x7ff43c0b4ed0 2024-04-06T13:45:34.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.317+0000 7ff4418c2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff43c0b5890 con 0x7ff43c0d3890 2024-04-06T13:45:34.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.317+0000 7ff4418c2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c0d1500 con 0x7ff43c15d590 2024-04-06T13:45:34.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.317+0000 7ff43affd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff43c15d590 0x7ff43c0b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:34.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.317+0000 7ff43b7fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff43c0d3890 0x7ff43c0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:34.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.317+0000 7ff43b7fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff43c0d3890 0x7ff43c0b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:48052/0 (socket says 172.21.15.189:48052) 2024-04-06T13:45:34.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.317+0000 7ff43a7fc700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7ff43c0b4ed0 0x7ff43c0b6580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:35964/0 (socket says 172.21.15.189:35964) 2024-04-06T13:45:34.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.317+0000 7ff43b7fe700 1 -- 172.21.15.189:0/4089849018 learned_addr learned my addr 172.21.15.189:0/4089849018 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:34.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.317+0000 7ff43affd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff43c15d590 0x7ff43c0b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:51920/0 (socket says 172.21.15.189:51920) 2024-04-06T13:45:34.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.318+0000 7ff439ffb700 1 -- 172.21.15.189:0/4089849018 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675455194 0 0) 0x7ff43c0b5cc0 con 0x7ff43c0b4ed0 2024-04-06T13:45:34.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.318+0000 7ff439ffb700 1 -- 172.21.15.189:0/4089849018 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4240035b0 con 0x7ff43c0b4ed0 2024-04-06T13:45:34.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.318+0000 7ff43affd700 1 -- 172.21.15.189:0/4089849018 >> v2:172.21.15.192:3300/0 conn(0x7ff43c0d3890 msgr2=0x7ff43c0b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:34.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.318+0000 7ff43affd700 1 --2- 172.21.15.189:0/4089849018 >> v2:172.21.15.192:3300/0 conn(0x7ff43c0d3890 0x7ff43c0b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:34.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.318+0000 7ff43affd700 1 -- 172.21.15.189:0/4089849018 >> v1:172.21.15.192:6789/0 conn(0x7ff43c0b4ed0 legacy=0x7ff43c0b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:34.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.318+0000 7ff43affd700 1 -- 172.21.15.189:0/4089849018 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43c0d17e0 con 0x7ff43c15d590 2024-04-06T13:45:34.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.318+0000 7ff43b7fe700 1 --2- 172.21.15.189:0/4089849018 >> v2:172.21.15.192:3300/0 conn(0x7ff43c0d3890 0x7ff43c0b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:45:34.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.318+0000 7ff43affd700 1 --2- 172.21.15.189:0/4089849018 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff43c15d590 0x7ff43c0b8d70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff430009870 tx=0x7ff430009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=997f1882e84a87dc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:34.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.318+0000 7ff439ffb700 1 -- 172.21.15.189:0/4089849018 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff430011830 con 0x7ff43c15d590 2024-04-06T13:45:34.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.318+0000 7ff439ffb700 1 -- 172.21.15.189:0/4089849018 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff430011990 con 0x7ff43c15d590 2024-04-06T13:45:34.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.318+0000 7ff439ffb700 1 -- 172.21.15.189:0/4089849018 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff430011d20 con 0x7ff43c15d590 2024-04-06T13:45:34.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.319+0000 7ff4418c2700 1 -- 172.21.15.189:0/4089849018 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff43c15d590 msgr2=0x7ff43c0b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:34.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.319+0000 7ff4418c2700 1 --2- 172.21.15.189:0/4089849018 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff43c15d590 0x7ff43c0b8d70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff430009870 tx=0x7ff430009b80 comp rx=0 tx=0).stop 2024-04-06T13:45:34.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.319+0000 7ff4418c2700 1 -- 172.21.15.189:0/4089849018 shutdown_connections 2024-04-06T13:45:34.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.319+0000 7ff4418c2700 1 --2- 172.21.15.189:0/4089849018 >> v2:172.21.15.192:3300/0 conn(0x7ff43c0d3890 0x7ff43c0b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:34.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.319+0000 7ff4418c2700 1 --2- 172.21.15.189:0/4089849018 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff43c15d590 0x7ff43c0b8d70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:34.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.319+0000 7ff4418c2700 1 -- 172.21.15.189:0/4089849018 >> 172.21.15.189:0/4089849018 conn(0x7ff43c06bbc0 msgr2=0x7ff43c06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:34.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.319+0000 7ff4418c2700 1 -- 172.21.15.189:0/4089849018 shutdown_connections 2024-04-06T13:45:34.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.319+0000 7ff4418c2700 1 -- 172.21.15.189:0/4089849018 wait complete. 2024-04-06T13:45:34.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.320+0000 7ff4418c2700 1 Processor -- start 2024-04-06T13:45:34.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.320+0000 7ff4418c2700 1 -- start start 2024-04-06T13:45:34.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.321+0000 7ff4418c2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff43c0b4ed0 0x7ff43c160140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:34.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.321+0000 7ff4418c2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff43c0d3890 0x7ff43c0ca520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:34.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.321+0000 7ff4418c2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff43c15d590 0x7ff43c14fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:34.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.321+0000 7ff4418c2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c1524e0 con 0x7ff43c0b4ed0 2024-04-06T13:45:34.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.321+0000 7ff4418c2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c1500f0 con 0x7ff43c15d590 2024-04-06T13:45:34.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.321+0000 7ff4418c2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c153dd0 con 0x7ff43c0d3890 2024-04-06T13:45:34.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.321+0000 7ff43affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff43c0b4ed0 0x7ff43c160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:34.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.321+0000 7ff43a7fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff43c0d3890 0x7ff43c0ca520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:34.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.321+0000 7ff43affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff43c0b4ed0 0x7ff43c160140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56516/0 (socket says 172.21.15.189:56516) 2024-04-06T13:45:34.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.321+0000 7ff43affd700 1 -- 172.21.15.189:0/3571848796 learned_addr learned my addr 172.21.15.189:0/3571848796 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:34.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.321+0000 7ff43b7fe700 1 --2- 172.21.15.189:0/3571848796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff43c15d590 0x7ff43c14fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:34.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.321+0000 7ff43affd700 1 -- 172.21.15.189:0/3571848796 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff43c0d3890 msgr2=0x7ff43c0ca520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:34.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.321+0000 7ff43affd700 1 --2- 172.21.15.189:0/3571848796 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff43c0d3890 0x7ff43c0ca520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:34.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.321+0000 7ff43affd700 1 -- 172.21.15.189:0/3571848796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff43c15d590 msgr2=0x7ff43c14fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:34.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.321+0000 7ff43affd700 1 --2- 172.21.15.189:0/3571848796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff43c15d590 0x7ff43c14fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:34.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.321+0000 7ff43affd700 1 -- 172.21.15.189:0/3571848796 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff434009b30 con 0x7ff43c0b4ed0 2024-04-06T13:45:34.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.321+0000 7ff43a7fc700 1 --2- 172.21.15.189:0/3571848796 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff43c0d3890 0x7ff43c0ca520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:45:34.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.321+0000 7ff43affd700 1 --2- 172.21.15.189:0/3571848796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff43c0b4ed0 0x7ff43c160140 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7ff430009840 tx=0x7ff430004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:34.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.322+0000 7ff423fff700 1 -- 172.21.15.189:0/3571848796 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4300036a0 con 0x7ff43c0b4ed0 2024-04-06T13:45:34.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.322+0000 7ff423fff700 1 -- 172.21.15.189:0/3571848796 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff43001f070 con 0x7ff43c0b4ed0 2024-04-06T13:45:34.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.322+0000 7ff4418c2700 1 -- 172.21.15.189:0/3571848796 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4300094a0 con 0x7ff43c0b4ed0 2024-04-06T13:45:34.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.322+0000 7ff423fff700 1 -- 172.21.15.189:0/3571848796 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff430003a60 con 0x7ff43c0b4ed0 2024-04-06T13:45:34.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.322+0000 7ff4418c2700 1 -- 172.21.15.189:0/3571848796 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff43c151e90 con 0x7ff43c0b4ed0 2024-04-06T13:45:34.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.323+0000 7ff4418c2700 1 -- 172.21.15.189:0/3571848796 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff43c074990 con 0x7ff43c0b4ed0 2024-04-06T13:45:34.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.323+0000 7ff423fff700 1 -- 172.21.15.189:0/3571848796 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7ff430003bc0 con 0x7ff43c0b4ed0 2024-04-06T13:45:34.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.323+0000 7ff423fff700 1 --2- 172.21.15.189:0/3571848796 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff42806adb0 0x7ff42806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:34.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.323+0000 7ff423fff700 1 -- 172.21.15.189:0/3571848796 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6281+0+0 (secure 0 0 0) 0x7ff430021070 con 0x7ff43c0b4ed0 2024-04-06T13:45:34.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.324+0000 7ff43a7fc700 1 --2- 172.21.15.189:0/3571848796 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff42806adb0 0x7ff42806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:34.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.324+0000 7ff43a7fc700 1 --2- 172.21.15.189:0/3571848796 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff42806adb0 0x7ff42806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff43c159e70 tx=0x7ff42c007dc0 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:34.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.328+0000 7ff423fff700 1 -- 172.21.15.189:0/3571848796 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff430055cd0 con 0x7ff43c0b4ed0 2024-04-06T13:45:34.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.569+0000 7ff4418c2700 1 -- 172.21.15.189:0/3571848796 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff43c0b4090 con 0x7ff43c0b4ed0 2024-04-06T13:45:34.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.570+0000 7ff423fff700 1 -- 172.21.15.189:0/3571848796 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3537 (secure 0 0 0) 0x7ff43002bdf0 con 0x7ff43c0b4ed0 2024-04-06T13:45:34.571 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:34.571 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":87,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8141,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2979280081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2979280081},{"type":"v1","addr":"172.21.15.192:6835","nonce":2979280081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":8477,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/1077133594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":1077133594},{"type":"v1","addr":"172.21.15.192:6834","nonce":1077133594}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":10125,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/286991908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":286991908},{"type":"v1","addr":"172.21.15.189:6836","nonce":286991908}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":10128,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/704323171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":704323171},{"type":"v1","addr":"172.21.15.189:6837","nonce":704323171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[]} 2024-04-06T13:45:34.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.572+0000 7ff4418c2700 1 -- 172.21.15.189:0/3571848796 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff42806adb0 msgr2=0x7ff42806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:34.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.572+0000 7ff4418c2700 1 --2- 172.21.15.189:0/3571848796 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff42806adb0 0x7ff42806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff43c159e70 tx=0x7ff42c007dc0 comp rx=0 tx=0).stop 2024-04-06T13:45:34.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.572+0000 7ff4418c2700 1 -- 172.21.15.189:0/3571848796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff43c0b4ed0 msgr2=0x7ff43c160140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:34.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.572+0000 7ff4418c2700 1 --2- 172.21.15.189:0/3571848796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff43c0b4ed0 0x7ff43c160140 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7ff430009840 tx=0x7ff430004750 comp rx=0 tx=0).stop 2024-04-06T13:45:34.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.572+0000 7ff4418c2700 1 -- 172.21.15.189:0/3571848796 shutdown_connections 2024-04-06T13:45:34.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.572+0000 7ff4418c2700 1 --2- 172.21.15.189:0/3571848796 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff42806adb0 0x7ff42806d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:34.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.572+0000 7ff4418c2700 1 --2- 172.21.15.189:0/3571848796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff43c15d590 0x7ff43c14fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:34.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.572+0000 7ff4418c2700 1 --2- 172.21.15.189:0/3571848796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff43c0b4ed0 0x7ff43c160140 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:34.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.572+0000 7ff4418c2700 1 --2- 172.21.15.189:0/3571848796 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff43c0d3890 0x7ff43c0ca520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:34.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.572+0000 7ff4418c2700 1 -- 172.21.15.189:0/3571848796 >> 172.21.15.189:0/3571848796 conn(0x7ff43c06bbc0 msgr2=0x7ff43c0b2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:34.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.572+0000 7ff4418c2700 1 -- 172.21.15.189:0/3571848796 shutdown_connections 2024-04-06T13:45:34.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.572+0000 7ff4418c2700 1 -- 172.21.15.189:0/3571848796 wait complete. 2024-04-06T13:45:34.575 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 87 2024-04-06T13:45:34.585 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-06T13:45:34.586 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-06T13:45:34.586 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-06T13:45:34.586 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:45:34.586 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-06T13:45:34.586 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-06T13:45:34.586 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-06T13:45:34.587 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:45:34.587 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-06T13:45:34.587 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-06T13:45:34.587 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-06T13:45:34.587 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:45:34.587 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-06T13:45:34.588 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-06T13:45:34.588 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-06T13:45:34.588 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:45:34.595 INFO:tasks.ceph.mds.a:Stopped 2024-04-06T13:45:34.595 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-06T13:45:34.596 INFO:tasks.ceph.mds.c:Stopped 2024-04-06T13:45:34.596 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-06T13:45:34.598 INFO:tasks.ceph.mds.b:Stopped 2024-04-06T13:45:34.598 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-06T13:45:34.600 INFO:tasks.ceph.mds.d:Stopped 2024-04-06T13:45:34.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-06T13:45:34.638 INFO:tasks.ceph.mds.a:Started 2024-04-06T13:45:34.638 DEBUG:teuthology.parallel:result is None 2024-04-06T13:45:34.639 INFO:tasks.ceph.mds.c:Started 2024-04-06T13:45:34.639 DEBUG:teuthology.parallel:result is None 2024-04-06T13:45:34.641 INFO:tasks.ceph.mds.b:Started 2024-04-06T13:45:34.641 DEBUG:teuthology.parallel:result is None 2024-04-06T13:45:34.642 INFO:tasks.ceph.mds.d:Started 2024-04-06T13:45:34.642 DEBUG:teuthology.parallel:result is None 2024-04-06T13:45:34.642 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-06T13:45:34.718 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2024-04-06T13:45:34.718 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T13:45:34.729 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T13:45:34.730 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2024-04-06T13:45:34.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.773+0000 7f482ab44700 1 Processor -- start 2024-04-06T13:45:34.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.773+0000 7f482ab44700 1 -- start start 2024-04-06T13:45:34.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.773+0000 7f482ab44700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48240b4660 0x7f48240b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:34.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.773+0000 7f482ab44700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f48240b2fc0 0x7f48240b3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:34.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.773+0000 7f482ab44700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48240cdbe0 con 0x7f482415a550 2024-04-06T13:45:34.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.773+0000 7f482ab44700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48240b5000 con 0x7f48240b2fc0 2024-04-06T13:45:34.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.773+0000 7f482ab44700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482415a940 con 0x7f48240b4660 2024-04-06T13:45:34.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.774+0000 7f48290e1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f48240b2fc0 0x7f48240b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:34.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.774+0000 7f48288e0700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f482415a550 0x7f48240cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48116/0 (socket says 172.21.15.189:48116) 2024-04-06T13:45:34.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.774+0000 7f48290e1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f48240b2fc0 0x7f48240b3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56576/0 (socket says 172.21.15.189:56576) 2024-04-06T13:45:34.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.774+0000 7f48288e0700 1 -- 172.21.15.189:0/2437842926 learned_addr learned my addr 172.21.15.189:0/2437842926 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:34.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.774+0000 7f4823fff700 1 --2- 172.21.15.189:0/2437842926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48240b4660 0x7f48240b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:34.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.774+0000 7f48290e1700 1 -- 172.21.15.189:0/2437842926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48240b4660 msgr2=0x7f48240b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:34.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.774+0000 7f48290e1700 1 --2- 172.21.15.189:0/2437842926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48240b4660 0x7f48240b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:34.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.774+0000 7f48290e1700 1 -- 172.21.15.189:0/2437842926 >> v1:172.21.15.189:6789/0 conn(0x7f482415a550 legacy=0x7f48240cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:34.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.774+0000 7f48290e1700 1 -- 172.21.15.189:0/2437842926 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482415ac00 con 0x7f48240b2fc0 2024-04-06T13:45:34.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.774+0000 7f48237fe700 1 -- 172.21.15.189:0/2437842926 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873103083 0 0) 0x7f48240cdbe0 con 0x7f482415a550 2024-04-06T13:45:34.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.774+0000 7f4823fff700 1 --2- 172.21.15.189:0/2437842926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48240b4660 0x7f48240b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:34.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.774+0000 7f48290e1700 1 --2- 172.21.15.189:0/2437842926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48240b2fc0 0x7f48240b3430 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f481c00bfa0 tx=0x7f481c00d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd66e283a5db330 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:34.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.774+0000 7f48237fe700 1 -- 172.21.15.189:0/2437842926 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481c010070 con 0x7f48240b2fc0 2024-04-06T13:45:34.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.774+0000 7f48237fe700 1 -- 172.21.15.189:0/2437842926 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481c00b2e0 con 0x7f48240b2fc0 2024-04-06T13:45:34.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.774+0000 7f48237fe700 1 -- 172.21.15.189:0/2437842926 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481c00dcf0 con 0x7f48240b2fc0 2024-04-06T13:45:34.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.775+0000 7f482ab44700 1 -- 172.21.15.189:0/2437842926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48240b2fc0 msgr2=0x7f48240b3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:34.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.775+0000 7f482ab44700 1 --2- 172.21.15.189:0/2437842926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48240b2fc0 0x7f48240b3430 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f481c00bfa0 tx=0x7f481c00d330 comp rx=0 tx=0).stop 2024-04-06T13:45:34.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.775+0000 7f482ab44700 1 -- 172.21.15.189:0/2437842926 shutdown_connections 2024-04-06T13:45:34.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.775+0000 7f482ab44700 1 --2- 172.21.15.189:0/2437842926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48240b2fc0 0x7f48240b3430 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:34.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.775+0000 7f482ab44700 1 --2- 172.21.15.189:0/2437842926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48240b4660 0x7f48240b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:34.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.775+0000 7f482ab44700 1 -- 172.21.15.189:0/2437842926 >> 172.21.15.189:0/2437842926 conn(0x7f48240d00b0 msgr2=0x7f48240d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:34.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.775+0000 7f482ab44700 1 -- 172.21.15.189:0/2437842926 shutdown_connections 2024-04-06T13:45:34.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.775+0000 7f482ab44700 1 -- 172.21.15.189:0/2437842926 wait complete. 2024-04-06T13:45:34.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.776+0000 7f482ab44700 1 Processor -- start 2024-04-06T13:45:34.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.776+0000 7f482ab44700 1 -- start start 2024-04-06T13:45:34.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.777+0000 7f482ab44700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48240b2fc0 0x7f482415b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.777+0000 7f482ab44700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48240b4660 0x7f482415cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.777+0000 7f482ab44700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f482414b1d0 0x7f4824157fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.777+0000 7f482ab44700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482415bea0 con 0x7f48240b2fc0 2024-04-06T13:45:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.777+0000 7f482ab44700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482415c180 con 0x7f48240b4660 2024-04-06T13:45:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.777+0000 7f482ab44700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482415c460 con 0x7f482414b1d0 2024-04-06T13:45:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.777+0000 7f48288e0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48240b2fc0 0x7f482415b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.777+0000 7f48288e0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48240b2fc0 0x7f482415b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56592/0 (socket says 172.21.15.189:56592) 2024-04-06T13:45:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.777+0000 7f48288e0700 1 -- 172.21.15.189:0/4257682764 learned_addr learned my addr 172.21.15.189:0/4257682764 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.777+0000 7f48290e1700 1 --2- 172.21.15.189:0/4257682764 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f482414b1d0 0x7f4824157fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.777+0000 7f4823fff700 1 --2- 172.21.15.189:0/4257682764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48240b4660 0x7f482415cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.778+0000 7f48288e0700 1 -- 172.21.15.189:0/4257682764 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f482414b1d0 msgr2=0x7f4824157fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.778+0000 7f48288e0700 1 --2- 172.21.15.189:0/4257682764 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f482414b1d0 0x7f4824157fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.778+0000 7f48288e0700 1 -- 172.21.15.189:0/4257682764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48240b4660 msgr2=0x7f482415cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.778+0000 7f48288e0700 1 --2- 172.21.15.189:0/4257682764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48240b4660 0x7f482415cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.778+0000 7f48288e0700 1 -- 172.21.15.189:0/4257682764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c00bc80 con 0x7f48240b2fc0 2024-04-06T13:45:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.778+0000 7f48288e0700 1 --2- 172.21.15.189:0/4257682764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48240b2fc0 0x7f482415b8c0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f48180011e0 tx=0x7f481800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.778+0000 7f48217fa700 1 -- 172.21.15.189:0/4257682764 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4818004c80 con 0x7f48240b2fc0 2024-04-06T13:45:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.778+0000 7f48217fa700 1 -- 172.21.15.189:0/4257682764 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4818004e00 con 0x7f48240b2fc0 2024-04-06T13:45:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.778+0000 7f48217fa700 1 -- 172.21.15.189:0/4257682764 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4818007310 con 0x7f48240b2fc0 2024-04-06T13:45:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.778+0000 7f482ab44700 1 -- 172.21.15.189:0/4257682764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f482415c740 con 0x7f48240b2fc0 2024-04-06T13:45:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.778+0000 7f482ab44700 1 -- 172.21.15.189:0/4257682764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f482414b680 con 0x7f48240b2fc0 2024-04-06T13:45:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.779+0000 7f482ab44700 1 -- 172.21.15.189:0/4257682764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f482415cc60 con 0x7f48240b2fc0 2024-04-06T13:45:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.779+0000 7f48217fa700 1 -- 172.21.15.189:0/4257682764 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f4818007470 con 0x7f48240b2fc0 2024-04-06T13:45:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.780+0000 7f48217fa700 1 --2- 172.21.15.189:0/4257682764 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f480806adb0 0x7f480806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.780+0000 7f48217fa700 1 -- 172.21.15.189:0/4257682764 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 6281+0+0 (secure 0 0 0) 0x7f4818089350 con 0x7f48240b2fc0 2024-04-06T13:45:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.780+0000 7f4823fff700 1 --2- 172.21.15.189:0/4257682764 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f480806adb0 0x7f480806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.780+0000 7f4823fff700 1 --2- 172.21.15.189:0/4257682764 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f480806adb0 0x7f480806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f482406b710 tx=0x7f4810008040 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:34.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.784+0000 7f48217fa700 1 -- 172.21.15.189:0/4257682764 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f482415cc60 con 0x7f48240b2fc0 2024-04-06T13:45:34.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:34.991+0000 7f482ab44700 1 -- 172.21.15.189:0/4257682764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f4824067bb0 con 0x7f48240b2fc0 2024-04-06T13:45:35.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:35.990+0000 7f48217fa700 1 -- 172.21.15.189:0/4257682764 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v205) v1 ==== 97+0+0 (secure 0 0 0) 0x7f48180541e0 con 0x7f48240b2fc0 2024-04-06T13:45:35.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:35.993+0000 7f482ab44700 1 -- 172.21.15.189:0/4257682764 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f480806adb0 msgr2=0x7f480806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:35.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:35.993+0000 7f482ab44700 1 --2- 172.21.15.189:0/4257682764 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f480806adb0 0x7f480806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f482406b710 tx=0x7f4810008040 comp rx=0 tx=0).stop 2024-04-06T13:45:35.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:35.993+0000 7f482ab44700 1 -- 172.21.15.189:0/4257682764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48240b2fc0 msgr2=0x7f482415b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:35.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:35.993+0000 7f482ab44700 1 --2- 172.21.15.189:0/4257682764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48240b2fc0 0x7f482415b8c0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f48180011e0 tx=0x7f481800ca30 comp rx=0 tx=0).stop 2024-04-06T13:45:35.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:35.993+0000 7f482ab44700 1 -- 172.21.15.189:0/4257682764 shutdown_connections 2024-04-06T13:45:35.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:35.993+0000 7f482ab44700 1 --2- 172.21.15.189:0/4257682764 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f480806adb0 0x7f480806d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:35.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:35.993+0000 7f482ab44700 1 --2- 172.21.15.189:0/4257682764 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f482414b1d0 0x7f4824157fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:35.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:35.993+0000 7f482ab44700 1 --2- 172.21.15.189:0/4257682764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48240b2fc0 0x7f482415b8c0 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:35.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:35.993+0000 7f482ab44700 1 --2- 172.21.15.189:0/4257682764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48240b4660 0x7f482415cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:35.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:35.993+0000 7f482ab44700 1 -- 172.21.15.189:0/4257682764 >> 172.21.15.189:0/4257682764 conn(0x7f48240d00b0 msgr2=0x7f482406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:35.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:35.993+0000 7f482ab44700 1 -- 172.21.15.189:0/4257682764 shutdown_connections 2024-04-06T13:45:35.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:35.993+0000 7f482ab44700 1 -- 172.21.15.189:0/4257682764 wait complete. 2024-04-06T13:45:35.995 INFO:teuthology.orchestra.run.smithi189.stderr: removed all blocklist entries 2024-04-06T13:45:36.009 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-06T13:45:36.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.136+0000 7f2cbe365700 1 Processor -- start 2024-04-06T13:45:36.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.136+0000 7f2cbe365700 1 -- start start 2024-04-06T13:45:36.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.137+0000 7f2cbe365700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2cb80d17a0 0x7f2cb806b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:36.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.137+0000 7f2cbe365700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2cb80cedf0 0x7f2cb80cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:36.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.137+0000 7f2cbe365700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb80cf730 con 0x7f2cb8146dd0 2024-04-06T13:45:36.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.137+0000 7f2cbe365700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb806d850 con 0x7f2cb80d17a0 2024-04-06T13:45:36.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.137+0000 7f2cbe365700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb806c0a0 con 0x7f2cb80cedf0 2024-04-06T13:45:36.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.137+0000 7f2cb7fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2cb80d17a0 0x7f2cb806b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:36.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.137+0000 7f2cb77fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2cb8146dd0 0x7f2cb806d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48132/0 (socket says 172.21.15.189:48132) 2024-04-06T13:45:36.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.137+0000 7f2cb77fe700 1 -- 172.21.15.189:0/1055339137 learned_addr learned my addr 172.21.15.189:0/1055339137 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:36.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.137+0000 7f2cbc902700 1 --2- 172.21.15.189:0/1055339137 >> v2:172.21.15.192:3300/0 conn(0x7f2cb80cedf0 0x7f2cb80cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:36.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.137+0000 7f2cb7fff700 1 -- 172.21.15.189:0/1055339137 >> v2:172.21.15.192:3300/0 conn(0x7f2cb80cedf0 msgr2=0x7f2cb80cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:36.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.137+0000 7f2cb7fff700 1 --2- 172.21.15.189:0/1055339137 >> v2:172.21.15.192:3300/0 conn(0x7f2cb80cedf0 0x7f2cb80cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.137+0000 7f2cb7fff700 1 -- 172.21.15.189:0/1055339137 >> v1:172.21.15.189:6789/0 conn(0x7f2cb8146dd0 legacy=0x7f2cb806d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:36.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.137+0000 7f2cb7fff700 1 -- 172.21.15.189:0/1055339137 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb806c380 con 0x7f2cb80d17a0 2024-04-06T13:45:36.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.137+0000 7f2cb6ffd700 1 -- 172.21.15.189:0/1055339137 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448390952 0 0) 0x7f2cb80cf730 con 0x7f2cb8146dd0 2024-04-06T13:45:36.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.137+0000 7f2cb7fff700 1 --2- 172.21.15.189:0/1055339137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb80d17a0 0x7f2cb806b7a0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8160fb0 tx=0x7f2cac009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42b1d6e248891cb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:36.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.137+0000 7f2cb6ffd700 1 -- 172.21.15.189:0/1055339137 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac011830 con 0x7f2cb80d17a0 2024-04-06T13:45:36.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.138+0000 7f2cb6ffd700 1 -- 172.21.15.189:0/1055339137 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cac011990 con 0x7f2cb80d17a0 2024-04-06T13:45:36.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.138+0000 7f2cb6ffd700 1 -- 172.21.15.189:0/1055339137 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac011d20 con 0x7f2cb80d17a0 2024-04-06T13:45:36.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.138+0000 7f2cbe365700 1 -- 172.21.15.189:0/1055339137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb80d17a0 msgr2=0x7f2cb806b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:36.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.138+0000 7f2cbe365700 1 --2- 172.21.15.189:0/1055339137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb80d17a0 0x7f2cb806b7a0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8160fb0 tx=0x7f2cac009b20 comp rx=0 tx=0).stop 2024-04-06T13:45:36.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.138+0000 7f2cbe365700 1 -- 172.21.15.189:0/1055339137 shutdown_connections 2024-04-06T13:45:36.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.138+0000 7f2cbe365700 1 --2- 172.21.15.189:0/1055339137 >> v2:172.21.15.192:3300/0 conn(0x7f2cb80cedf0 0x7f2cb80cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.138+0000 7f2cbe365700 1 --2- 172.21.15.189:0/1055339137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb80d17a0 0x7f2cb806b7a0 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.138+0000 7f2cbe365700 1 -- 172.21.15.189:0/1055339137 >> 172.21.15.189:0/1055339137 conn(0x7f2cb80b2db0 msgr2=0x7f2cb80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:36.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.138+0000 7f2cbe365700 1 -- 172.21.15.189:0/1055339137 shutdown_connections 2024-04-06T13:45:36.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.138+0000 7f2cbe365700 1 -- 172.21.15.189:0/1055339137 wait complete. 2024-04-06T13:45:36.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.139+0000 7f2cbe365700 1 Processor -- start 2024-04-06T13:45:36.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.139+0000 7f2cbe365700 1 -- start start 2024-04-06T13:45:36.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.140+0000 7f2cbe365700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb80cedf0 0x7f2cb80aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:36.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.140+0000 7f2cbe365700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cb80d17a0 0x7f2cb80b01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:36.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.140+0000 7f2cbe365700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cb8146dd0 0x7f2cb815e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:36.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.140+0000 7f2cbe365700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb81f2b40 con 0x7f2cb80cedf0 2024-04-06T13:45:36.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.140+0000 7f2cbe365700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb80af1a0 con 0x7f2cb8146dd0 2024-04-06T13:45:36.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.140+0000 7f2cbe365700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb81f5ef0 con 0x7f2cb80d17a0 2024-04-06T13:45:36.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.140+0000 7f2cb7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb80cedf0 0x7f2cb80aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:36.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.140+0000 7f2cb7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb80cedf0 0x7f2cb80aec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56612/0 (socket says 172.21.15.189:56612) 2024-04-06T13:45:36.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.140+0000 7f2cb7fff700 1 -- 172.21.15.189:0/652477433 learned_addr learned my addr 172.21.15.189:0/652477433 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:36.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.141+0000 7f2cbc902700 1 --2- 172.21.15.189:0/652477433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cb8146dd0 0x7f2cb815e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:36.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.141+0000 7f2cb77fe700 1 --2- 172.21.15.189:0/652477433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cb80d17a0 0x7f2cb80b01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:36.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.141+0000 7f2cb7fff700 1 -- 172.21.15.189:0/652477433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cb80d17a0 msgr2=0x7f2cb80b01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:36.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.141+0000 7f2cb7fff700 1 --2- 172.21.15.189:0/652477433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cb80d17a0 0x7f2cb80b01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.141+0000 7f2cb7fff700 1 -- 172.21.15.189:0/652477433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cb8146dd0 msgr2=0x7f2cb815e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:36.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.141+0000 7f2cb7fff700 1 --2- 172.21.15.189:0/652477433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cb8146dd0 0x7f2cb815e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.141+0000 7f2cb7fff700 1 -- 172.21.15.189:0/652477433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cac0094a0 con 0x7f2cb80cedf0 2024-04-06T13:45:36.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.141+0000 7f2cb7fff700 1 --2- 172.21.15.189:0/652477433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb80cedf0 0x7f2cb80aec60 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f2cac014040 tx=0x7f2cac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:36.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.141+0000 7f2cb4ff9700 1 -- 172.21.15.189:0/652477433 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac0038a0 con 0x7f2cb80cedf0 2024-04-06T13:45:36.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.141+0000 7f2cb4ff9700 1 -- 172.21.15.189:0/652477433 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cac01e070 con 0x7f2cb80cedf0 2024-04-06T13:45:36.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.141+0000 7f2cbe365700 1 -- 172.21.15.189:0/652477433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb80b06f0 con 0x7f2cb80cedf0 2024-04-06T13:45:36.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.141+0000 7f2cb4ff9700 1 -- 172.21.15.189:0/652477433 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac003c20 con 0x7f2cb80cedf0 2024-04-06T13:45:36.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.141+0000 7f2cbe365700 1 -- 172.21.15.189:0/652477433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb80af670 con 0x7f2cb80cedf0 2024-04-06T13:45:36.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.142+0000 7f2cbe365700 1 -- 172.21.15.189:0/652477433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb8074990 con 0x7f2cb80cedf0 2024-04-06T13:45:36.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.142+0000 7f2cb4ff9700 1 -- 172.21.15.189:0/652477433 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f2cac02d450 con 0x7f2cb80cedf0 2024-04-06T13:45:36.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.143+0000 7f2cb4ff9700 1 --2- 172.21.15.189:0/652477433 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2ca406adb0 0x7f2ca406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:36.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.143+0000 7f2cb4ff9700 1 -- 172.21.15.189:0/652477433 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f2cac08a120 con 0x7f2cb80cedf0 2024-04-06T13:45:36.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.143+0000 7f2cb77fe700 1 --2- 172.21.15.189:0/652477433 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2ca406adb0 0x7f2ca406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:36.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.143+0000 7f2cb77fe700 1 --2- 172.21.15.189:0/652477433 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2ca406adb0 0x7f2ca406d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2cb815d0c0 tx=0x7f2ca8007be0 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:36.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.147+0000 7f2cb4ff9700 1 -- 172.21.15.189:0/652477433 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2cac055440 con 0x7f2cb80cedf0 2024-04-06T13:45:36.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.420+0000 7f2cbe365700 1 -- 172.21.15.189:0/652477433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f2cb80b06f0 con 0x7f2cb80cedf0 2024-04-06T13:45:36.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.420+0000 7f2cb4ff9700 1 -- 172.21.15.189:0/652477433 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v49) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f2cb80b06f0 con 0x7f2cb80cedf0 2024-04-06T13:45:36.421 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:36.421 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:45:36.421 INFO:teuthology.orchestra.run.smithi189.stdout: "auth_dump": [ 2024-04-06T13:45:36.421 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:36.422 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.a", 2024-04-06T13:45:36.422 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmTjYMNBAA5voWGV4Pa7pE27VogrdipQ==", 2024-04-06T13:45:36.422 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:45:36.422 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:45:36.422 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:45:36.422 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:45:36.422 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:45:36.422 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.422 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:45:36.423 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:36.423 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.b", 2024-04-06T13:45:36.423 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm9A1OBRAAnHzubCvRUX8UdW8hmDzKbQ==", 2024-04-06T13:45:36.423 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:45:36.423 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:45:36.423 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:45:36.423 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:45:36.423 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:45:36.423 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.423 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:45:36.423 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:36.423 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.c", 2024-04-06T13:45:36.423 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmz0hhOxAAD3/GINNmGoFk+oPyiHHJXg==", 2024-04-06T13:45:36.423 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:45:36.423 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:45:36.424 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:45:36.424 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:45:36.424 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:45:36.424 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.424 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:45:36.424 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:36.424 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.d", 2024-04-06T13:45:36.424 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmhoAeDRAAoQC0vicWumBRTwpi3PtYaw==", 2024-04-06T13:45:36.424 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:45:36.424 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:45:36.424 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:45:36.424 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:45:36.424 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:45:36.424 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.424 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:45:36.425 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:36.425 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.0", 2024-04-06T13:45:36.425 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDkOxFmkxLWJhAAFXewCNOFX7dhAZVgUmewpQ==", 2024-04-06T13:45:36.425 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:45:36.425 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:45:36.425 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:45:36.425 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:45:36.425 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.425 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:45:36.425 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:36.425 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.1", 2024-04-06T13:45:36.425 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDnOxFmHYqWJRAABlsVCiv3e7YJMbfodJTWag==", 2024-04-06T13:45:36.425 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:45:36.425 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:45:36.426 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:45:36.426 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:45:36.426 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.426 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:45:36.426 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:36.426 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.2", 2024-04-06T13:45:36.426 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDqOxFmkhcvJBAA8NMEpHVGbGALjV9RISKP8g==", 2024-04-06T13:45:36.426 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:45:36.426 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:45:36.426 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:45:36.426 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:45:36.426 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.426 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:45:36.426 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:36.427 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.3", 2024-04-06T13:45:36.427 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDtOxFmDGzhIxAAamBoAbE8+lAel4os67meyQ==", 2024-04-06T13:45:36.427 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:45:36.427 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:45:36.427 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:45:36.427 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:45:36.427 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.427 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:45:36.427 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:36.427 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.4", 2024-04-06T13:45:36.427 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDzOxFmobZDJxAA9l8wdS81Eyw6+17I5ac6bw==", 2024-04-06T13:45:36.427 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:45:36.427 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:45:36.427 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:45:36.427 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:45:36.428 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.428 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:45:36.428 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:36.428 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.5", 2024-04-06T13:45:36.428 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD2OxFmi+ruJRAASlQ0o9jX/wA7U5AosnL+Wg==", 2024-04-06T13:45:36.428 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:45:36.428 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:45:36.428 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:45:36.428 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:45:36.428 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.428 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:45:36.428 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:36.428 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.6", 2024-04-06T13:45:36.428 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD5OxFmbwuuJBAA0wqJRmnJyskQPZJ7/LnpvA==", 2024-04-06T13:45:36.428 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:45:36.429 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:45:36.429 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:45:36.429 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:45:36.429 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.429 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:45:36.429 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:36.429 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.7", 2024-04-06T13:45:36.429 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD8OxFmDmRgIxAAzyW9AGAbmb67ZvB5qunwLQ==", 2024-04-06T13:45:36.429 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:45:36.429 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:45:36.429 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:45:36.429 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:45:36.429 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.429 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:45:36.430 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:36.430 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.0", 2024-04-06T13:45:36.430 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm0ByJERAAoEV8Yfc09+Go6u95XafElQ==", 2024-04-06T13:45:36.430 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:45:36.430 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:45:36.430 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow r", 2024-04-06T13:45:36.430 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-06T13:45:36.430 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.430 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:45:36.430 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:36.430 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.1", 2024-04-06T13:45:36.430 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmSV5lGBAAavDEo2LGy0IHpWSAuitHXg==", 2024-04-06T13:45:36.430 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:45:36.430 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:45:36.431 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:45:36.431 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:45:36.431 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:45:36.431 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.431 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:45:36.431 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:36.431 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.2", 2024-04-06T13:45:36.431 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm/Gz5FBAA3xX/rsAyWZ1JoUVlQlBIvQ==", 2024-04-06T13:45:36.431 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:45:36.431 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:45:36.431 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:45:36.431 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:45:36.431 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:45:36.431 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.431 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:45:36.432 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:36.432 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.3", 2024-04-06T13:45:36.432 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmzra7GxAADbnzIvWnRTo65LvPwtxWrg==", 2024-04-06T13:45:36.432 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:45:36.432 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:45:36.432 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:45:36.432 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:45:36.432 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:45:36.432 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.432 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:45:36.432 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:36.432 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.admin", 2024-04-06T13:45:36.432 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmxG+wCBAANg5kt6TOUQdj3/X0Mqgx1g==", 2024-04-06T13:45:36.432 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:45:36.433 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:45:36.433 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:45:36.433 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:45:36.433 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:45:36.433 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.433 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:45:36.433 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:36.433 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.x", 2024-04-06T13:45:36.433 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmPLyBMBAAYfhkI9Z1CPlf1Tijm+L7xA==", 2024-04-06T13:45:36.433 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:45:36.433 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:45:36.433 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T13:45:36.433 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:45:36.433 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.433 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:45:36.433 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:36.434 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.y", 2024-04-06T13:45:36.434 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFm1nHJHhAAQWRe8cU8duWuUJiYaDeTaQ==", 2024-04-06T13:45:36.434 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:45:36.434 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:45:36.434 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T13:45:36.434 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:45:36.434 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.434 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:36.434 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-04-06T13:45:36.434 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:45:36.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.422+0000 7f2cbe365700 1 -- 172.21.15.189:0/652477433 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2ca406adb0 msgr2=0x7f2ca406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:36.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.422+0000 7f2cbe365700 1 --2- 172.21.15.189:0/652477433 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2ca406adb0 0x7f2ca406d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2cb815d0c0 tx=0x7f2ca8007be0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.422+0000 7f2cbe365700 1 -- 172.21.15.189:0/652477433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb80cedf0 msgr2=0x7f2cb80aec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:36.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.422+0000 7f2cbe365700 1 --2- 172.21.15.189:0/652477433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb80cedf0 0x7f2cb80aec60 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f2cac014040 tx=0x7f2cac004750 comp rx=0 tx=0).stop 2024-04-06T13:45:36.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.422+0000 7f2cbe365700 1 -- 172.21.15.189:0/652477433 shutdown_connections 2024-04-06T13:45:36.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.422+0000 7f2cbe365700 1 --2- 172.21.15.189:0/652477433 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2ca406adb0 0x7f2ca406d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.422+0000 7f2cbe365700 1 --2- 172.21.15.189:0/652477433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cb8146dd0 0x7f2cb815e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.422+0000 7f2cbe365700 1 --2- 172.21.15.189:0/652477433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb80cedf0 0x7f2cb80aec60 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.422+0000 7f2cbe365700 1 --2- 172.21.15.189:0/652477433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cb80d17a0 0x7f2cb80b01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.422+0000 7f2cbe365700 1 -- 172.21.15.189:0/652477433 >> 172.21.15.189:0/652477433 conn(0x7f2cb80b2db0 msgr2=0x7f2cb806d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:36.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.422+0000 7f2cbe365700 1 -- 172.21.15.189:0/652477433 shutdown_connections 2024-04-06T13:45:36.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.422+0000 7f2cbe365700 1 -- 172.21.15.189:0/652477433 wait complete. 2024-04-06T13:45:36.437 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:45:36.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.559+0000 7efc19b17700 1 Processor -- start 2024-04-06T13:45:36.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.559+0000 7efc19b17700 1 -- start start 2024-04-06T13:45:36.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.560+0000 7efc19b17700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efc1415d7b0 0x7efc140b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:36.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.560+0000 7efc19b17700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc1415e500 0x7efc1406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:36.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.560+0000 7efc19b17700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc140b3990 con 0x7efc1406a730 2024-04-06T13:45:36.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.560+0000 7efc19b17700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc1406b0f0 con 0x7efc1415d7b0 2024-04-06T13:45:36.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.560+0000 7efc19b17700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc140b3060 con 0x7efc1415e500 2024-04-06T13:45:36.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.560+0000 7efc137fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efc1415d7b0 0x7efc140b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:36.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.561+0000 7efc137fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efc1415d7b0 0x7efc140b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56620/0 (socket says 172.21.15.189:56620) 2024-04-06T13:45:36.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.561+0000 7efc137fe700 1 -- 172.21.15.189:0/3270383985 learned_addr learned my addr 172.21.15.189:0/3270383985 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:36.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.561+0000 7efc13fff700 1 --2- 172.21.15.189:0/3270383985 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc1415e500 0x7efc1406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:36.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.561+0000 7efc137fe700 1 -- 172.21.15.189:0/3270383985 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc1415e500 msgr2=0x7efc1406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:36.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.561+0000 7efc137fe700 1 --2- 172.21.15.189:0/3270383985 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc1415e500 0x7efc1406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.561+0000 7efc137fe700 1 -- 172.21.15.189:0/3270383985 >> v1:172.21.15.192:6789/0 conn(0x7efc1406a730 legacy=0x7efc1415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:36.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.561+0000 7efc137fe700 1 -- 172.21.15.189:0/3270383985 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc140b3340 con 0x7efc1415d7b0 2024-04-06T13:45:36.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.561+0000 7efc137fe700 1 --2- 172.21.15.189:0/3270383985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1415d7b0 0x7efc140b3e20 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7efc080098f0 tx=0x7efc08009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11913a1ae4b6f07a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:36.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.561+0000 7efc127fc700 1 -- 172.21.15.189:0/3270383985 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc08011830 con 0x7efc1415d7b0 2024-04-06T13:45:36.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.561+0000 7efc127fc700 1 -- 172.21.15.189:0/3270383985 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc08011990 con 0x7efc1415d7b0 2024-04-06T13:45:36.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.561+0000 7efc127fc700 1 -- 172.21.15.189:0/3270383985 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc08011d20 con 0x7efc1415d7b0 2024-04-06T13:45:36.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.562+0000 7efc19b17700 1 -- 172.21.15.189:0/3270383985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1415d7b0 msgr2=0x7efc140b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:36.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.562+0000 7efc19b17700 1 --2- 172.21.15.189:0/3270383985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1415d7b0 0x7efc140b3e20 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7efc080098f0 tx=0x7efc08009c00 comp rx=0 tx=0).stop 2024-04-06T13:45:36.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.562+0000 7efc19b17700 1 -- 172.21.15.189:0/3270383985 shutdown_connections 2024-04-06T13:45:36.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.562+0000 7efc19b17700 1 --2- 172.21.15.189:0/3270383985 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc1415e500 0x7efc1406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.562+0000 7efc19b17700 1 --2- 172.21.15.189:0/3270383985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1415d7b0 0x7efc140b3e20 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.562+0000 7efc19b17700 1 -- 172.21.15.189:0/3270383985 >> 172.21.15.189:0/3270383985 conn(0x7efc140d00b0 msgr2=0x7efc140d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:36.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.562+0000 7efc19b17700 1 -- 172.21.15.189:0/3270383985 shutdown_connections 2024-04-06T13:45:36.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.562+0000 7efc19b17700 1 -- 172.21.15.189:0/3270383985 wait complete. 2024-04-06T13:45:36.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.563+0000 7efc19b17700 1 Processor -- start 2024-04-06T13:45:36.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.563+0000 7efc19b17700 1 -- start start 2024-04-06T13:45:36.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.564+0000 7efc19b17700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc1406a730 0x7efc141f4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:36.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.564+0000 7efc19b17700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc1415d7b0 0x7efc14146a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:36.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.564+0000 7efc19b17700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1415e500 0x7efc140ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:36.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.564+0000 7efc19b17700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1415ad90 con 0x7efc1415e500 2024-04-06T13:45:36.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.564+0000 7efc19b17700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc14160490 con 0x7efc1406a730 2024-04-06T13:45:36.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.564+0000 7efc19b17700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1415afe0 con 0x7efc1415d7b0 2024-04-06T13:45:36.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.564+0000 7efc13fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1415e500 0x7efc140ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:36.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.564+0000 7efc13fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1415e500 0x7efc140ccd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56626/0 (socket says 172.21.15.189:56626) 2024-04-06T13:45:36.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.564+0000 7efc13fff700 1 -- 172.21.15.189:0/2381403039 learned_addr learned my addr 172.21.15.189:0/2381403039 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:36.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.564+0000 7efc137fe700 1 --2- 172.21.15.189:0/2381403039 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc1406a730 0x7efc141f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:36.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.564+0000 7efc12ffd700 1 --2- 172.21.15.189:0/2381403039 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc1415d7b0 0x7efc14146a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:36.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.564+0000 7efc13fff700 1 -- 172.21.15.189:0/2381403039 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc1415d7b0 msgr2=0x7efc14146a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:36.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.564+0000 7efc13fff700 1 --2- 172.21.15.189:0/2381403039 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc1415d7b0 0x7efc14146a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.564+0000 7efc13fff700 1 -- 172.21.15.189:0/2381403039 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc1406a730 msgr2=0x7efc141f4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:36.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.564+0000 7efc13fff700 1 --2- 172.21.15.189:0/2381403039 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc1406a730 0x7efc141f4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.564+0000 7efc13fff700 1 -- 172.21.15.189:0/2381403039 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc08009550 con 0x7efc1415e500 2024-04-06T13:45:36.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.564+0000 7efc13fff700 1 --2- 172.21.15.189:0/2381403039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1415e500 0x7efc140ccd90 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7efc0c00efc0 tx=0x7efc0c00c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:36.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.565+0000 7efbfbfff700 1 -- 172.21.15.189:0/2381403039 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c00cad0 con 0x7efc1415e500 2024-04-06T13:45:36.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.565+0000 7efbfbfff700 1 -- 172.21.15.189:0/2381403039 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0c00f040 con 0x7efc1415e500 2024-04-06T13:45:36.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.565+0000 7efc19b17700 1 -- 172.21.15.189:0/2381403039 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1415b2c0 con 0x7efc1415e500 2024-04-06T13:45:36.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.565+0000 7efbfbfff700 1 -- 172.21.15.189:0/2381403039 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c00ce70 con 0x7efc1415e500 2024-04-06T13:45:36.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.565+0000 7efc19b17700 1 -- 172.21.15.189:0/2381403039 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc14146f80 con 0x7efc1415e500 2024-04-06T13:45:36.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.565+0000 7efc19b17700 1 -- 172.21.15.189:0/2381403039 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1415b7e0 con 0x7efc1415e500 2024-04-06T13:45:36.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.566+0000 7efbfbfff700 1 -- 172.21.15.189:0/2381403039 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7efc0c014480 con 0x7efc1415e500 2024-04-06T13:45:36.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.566+0000 7efbfbfff700 1 --2- 172.21.15.189:0/2381403039 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efc0006adb0 0x7efc0006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:36.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.566+0000 7efbfbfff700 1 -- 172.21.15.189:0/2381403039 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5249+0+0 (secure 0 0 0) 0x7efc0c089be0 con 0x7efc1415e500 2024-04-06T13:45:36.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.566+0000 7efc137fe700 1 --2- 172.21.15.189:0/2381403039 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efc0006adb0 0x7efc0006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:36.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.567+0000 7efc137fe700 1 --2- 172.21.15.189:0/2381403039 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efc0006adb0 0x7efc0006d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efc08009c40 tx=0x7efc08008030 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:36.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.571+0000 7efbfbfff700 1 -- 172.21.15.189:0/2381403039 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efc1415b7e0 con 0x7efc1415e500 2024-04-06T13:45:36.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.770+0000 7efc19b17700 1 -- 172.21.15.189:0/2381403039 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc1415b7e0 con 0x7efc1415e500 2024-04-06T13:45:36.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.771+0000 7efbfbfff700 1 -- 172.21.15.189:0/2381403039 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v205) v1 ==== 74+0+12768 (secure 0 0 0) 0x7efc1415b7e0 con 0x7efc1415e500 2024-04-06T13:45:36.772 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:36.772 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":205,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:45:35.982997+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:45:36.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.773+0000 7efc19b17700 1 -- 172.21.15.189:0/2381403039 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efc0006adb0 msgr2=0x7efc0006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:36.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.773+0000 7efc19b17700 1 --2- 172.21.15.189:0/2381403039 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efc0006adb0 0x7efc0006d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efc08009c40 tx=0x7efc08008030 comp rx=0 tx=0).stop 2024-04-06T13:45:36.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.773+0000 7efc19b17700 1 -- 172.21.15.189:0/2381403039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1415e500 msgr2=0x7efc140ccd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:36.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.773+0000 7efc19b17700 1 --2- 172.21.15.189:0/2381403039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1415e500 0x7efc140ccd90 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7efc0c00efc0 tx=0x7efc0c00c0c0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.773+0000 7efc19b17700 1 -- 172.21.15.189:0/2381403039 shutdown_connections 2024-04-06T13:45:36.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.773+0000 7efc19b17700 1 --2- 172.21.15.189:0/2381403039 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efc0006adb0 0x7efc0006d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.773+0000 7efc19b17700 1 --2- 172.21.15.189:0/2381403039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1415e500 0x7efc140ccd90 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.773+0000 7efc19b17700 1 --2- 172.21.15.189:0/2381403039 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc1406a730 0x7efc141f4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.773+0000 7efc19b17700 1 --2- 172.21.15.189:0/2381403039 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc1415d7b0 0x7efc14146a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.773+0000 7efc19b17700 1 -- 172.21.15.189:0/2381403039 >> 172.21.15.189:0/2381403039 conn(0x7efc140d00b0 msgr2=0x7efc1406eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:36.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.773+0000 7efc19b17700 1 -- 172.21.15.189:0/2381403039 shutdown_connections 2024-04-06T13:45:36.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.773+0000 7efc19b17700 1 -- 172.21.15.189:0/2381403039 wait complete. 2024-04-06T13:45:36.787 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:45:36.787 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:45:36.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.910+0000 7f2f0ab7a700 1 Processor -- start 2024-04-06T13:45:36.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.910+0000 7f2f0ab7a700 1 -- start start 2024-04-06T13:45:36.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.911+0000 7f2f0ab7a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2f0415d7b0 0x7f2f040b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:36.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.911+0000 7f2f0ab7a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f040b3990 con 0x7f2f0406a730 2024-04-06T13:45:36.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.911+0000 7f2f0ab7a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f040b3060 con 0x7f2f0415e500 2024-04-06T13:45:36.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.911+0000 7f2f0ab7a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f0415e950 con 0x7f2f0415d7b0 2024-04-06T13:45:36.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.911+0000 7f2f08916700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2f0415d7b0 0x7f2f040b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:36.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.911+0000 7f2f03fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2f0406a730 0x7f2f0415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48138/0 (socket says 172.21.15.189:48138) 2024-04-06T13:45:36.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.911+0000 7f2f03fff700 1 -- 172.21.15.189:0/2206329501 learned_addr learned my addr 172.21.15.189:0/2206329501 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:36.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.911+0000 7f2f08916700 1 -- 172.21.15.189:0/2206329501 >> v1:172.21.15.192:6789/0 conn(0x7f2f0415e500 legacy=0x7f2f040717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:36.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.911+0000 7f2f08916700 1 -- 172.21.15.189:0/2206329501 >> v1:172.21.15.189:6789/0 conn(0x7f2f0406a730 legacy=0x7f2f0415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:36.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.911+0000 7f2f037fe700 1 -- 172.21.15.189:0/2206329501 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258808667 0 0) 0x7f2f040b3990 con 0x7f2f0406a730 2024-04-06T13:45:36.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.911+0000 7f2f08916700 1 -- 172.21.15.189:0/2206329501 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f040b3340 con 0x7f2f0415d7b0 2024-04-06T13:45:36.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.911+0000 7f2f08916700 1 --2- 172.21.15.189:0/2206329501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f0415d7b0 0x7f2f040b3e20 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8009840 tx=0x7f2ef8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21127c3904e9b8b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:36.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.911+0000 7f2f037fe700 1 -- 172.21.15.189:0/2206329501 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8011830 con 0x7f2f0415d7b0 2024-04-06T13:45:36.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.912+0000 7f2f037fe700 1 -- 172.21.15.189:0/2206329501 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef8011990 con 0x7f2f0415d7b0 2024-04-06T13:45:36.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.912+0000 7f2f037fe700 1 -- 172.21.15.189:0/2206329501 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8011d20 con 0x7f2f0415d7b0 2024-04-06T13:45:36.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.912+0000 7f2f0ab7a700 1 -- 172.21.15.189:0/2206329501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f0415d7b0 msgr2=0x7f2f040b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:36.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.912+0000 7f2f0ab7a700 1 --2- 172.21.15.189:0/2206329501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f0415d7b0 0x7f2f040b3e20 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8009840 tx=0x7f2ef8009b50 comp rx=0 tx=0).stop 2024-04-06T13:45:36.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.912+0000 7f2f0ab7a700 1 -- 172.21.15.189:0/2206329501 shutdown_connections 2024-04-06T13:45:36.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.912+0000 7f2f0ab7a700 1 --2- 172.21.15.189:0/2206329501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f0415d7b0 0x7f2f040b3e20 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.912+0000 7f2f0ab7a700 1 -- 172.21.15.189:0/2206329501 >> 172.21.15.189:0/2206329501 conn(0x7f2f040d00b0 msgr2=0x7f2f040d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:36.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.912+0000 7f2f0ab7a700 1 -- 172.21.15.189:0/2206329501 shutdown_connections 2024-04-06T13:45:36.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.912+0000 7f2f0ab7a700 1 -- 172.21.15.189:0/2206329501 wait complete. 2024-04-06T13:45:36.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.914+0000 7f2f0ab7a700 1 Processor -- start 2024-04-06T13:45:36.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.914+0000 7f2f0ab7a700 1 -- start start 2024-04-06T13:45:36.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f0ab7a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f0406a730 0x7f2f041508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:36.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f0ab7a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f0415d7b0 0x7f2f0415fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:36.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f0ab7a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f0415e500 0x7f2f04160250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:36.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f0ab7a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f04155cc0 con 0x7f2f0415d7b0 2024-04-06T13:45:36.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f0ab7a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f04155fa0 con 0x7f2f0415e500 2024-04-06T13:45:36.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f0ab7a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f04156280 con 0x7f2f0406a730 2024-04-06T13:45:36.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f03fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f0415d7b0 0x7f2f0415fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:36.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f03fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f0415d7b0 0x7f2f0415fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56634/0 (socket says 172.21.15.189:56634) 2024-04-06T13:45:36.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f03fff700 1 -- 172.21.15.189:0/391431789 learned_addr learned my addr 172.21.15.189:0/391431789 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:36.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f08916700 1 --2- 172.21.15.189:0/391431789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f0406a730 0x7f2f041508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:36.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f09117700 1 --2- 172.21.15.189:0/391431789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f0415e500 0x7f2f04160250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:36.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f03fff700 1 -- 172.21.15.189:0/391431789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f0406a730 msgr2=0x7f2f041508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:36.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f03fff700 1 --2- 172.21.15.189:0/391431789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f0406a730 0x7f2f041508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f03fff700 1 -- 172.21.15.189:0/391431789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f0415e500 msgr2=0x7f2f04160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:36.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f03fff700 1 --2- 172.21.15.189:0/391431789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f0415e500 0x7f2f04160250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:36.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f03fff700 1 -- 172.21.15.189:0/391431789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef80094a0 con 0x7f2f0415d7b0 2024-04-06T13:45:36.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f03fff700 1 --2- 172.21.15.189:0/391431789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f0415d7b0 0x7f2f0415fd10 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f2ef000c520 tx=0x7f2ef000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:36.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f017fa700 1 -- 172.21.15.189:0/391431789 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef0004c80 con 0x7f2f0415d7b0 2024-04-06T13:45:36.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f017fa700 1 -- 172.21.15.189:0/391431789 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef0004e00 con 0x7f2f0415d7b0 2024-04-06T13:45:36.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f0ab7a700 1 -- 172.21.15.189:0/391431789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f04156560 con 0x7f2f0415d7b0 2024-04-06T13:45:36.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.915+0000 7f2f017fa700 1 -- 172.21.15.189:0/391431789 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef0007310 con 0x7f2f0415d7b0 2024-04-06T13:45:36.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.916+0000 7f2f0ab7a700 1 -- 172.21.15.189:0/391431789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f04156d60 con 0x7f2f0415d7b0 2024-04-06T13:45:36.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.917+0000 7f2f0ab7a700 1 -- 172.21.15.189:0/391431789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f04157040 con 0x7f2f0415d7b0 2024-04-06T13:45:36.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.917+0000 7f2f017fa700 1 -- 172.21.15.189:0/391431789 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f2ef0007470 con 0x7f2f0415d7b0 2024-04-06T13:45:36.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.917+0000 7f2f017fa700 1 --2- 172.21.15.189:0/391431789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2ee806adb0 0x7f2ee806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:36.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.917+0000 7f2f08916700 1 --2- 172.21.15.189:0/391431789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2ee806adb0 0x7f2ee806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:36.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.917+0000 7f2f017fa700 1 -- 172.21.15.189:0/391431789 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f2ef0088e70 con 0x7f2f0415d7b0 2024-04-06T13:45:36.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.918+0000 7f2f08916700 1 --2- 172.21.15.189:0/391431789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2ee806adb0 0x7f2ee806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2ef80090b0 tx=0x7f2ef801c040 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:36.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:36.922+0000 7f2f017fa700 1 -- 172.21.15.189:0/391431789 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2ef0054190 con 0x7f2f0415d7b0 2024-04-06T13:45:37.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.117+0000 7f2f0ab7a700 1 -- 172.21.15.189:0/391431789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2f04156a80 con 0x7f2f0415d7b0 2024-04-06T13:45:37.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.117+0000 7f2f017fa700 1 -- 172.21.15.189:0/391431789 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v205) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2f04156a80 con 0x7f2f0415d7b0 2024-04-06T13:45:37.118 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:45:37.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.119+0000 7f2f0ab7a700 1 -- 172.21.15.189:0/391431789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2ee806adb0 msgr2=0x7f2ee806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:37.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.119+0000 7f2f0ab7a700 1 --2- 172.21.15.189:0/391431789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2ee806adb0 0x7f2ee806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2ef80090b0 tx=0x7f2ef801c040 comp rx=0 tx=0).stop 2024-04-06T13:45:37.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.119+0000 7f2f0ab7a700 1 -- 172.21.15.189:0/391431789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f0415d7b0 msgr2=0x7f2f0415fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:37.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.119+0000 7f2f0ab7a700 1 --2- 172.21.15.189:0/391431789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f0415d7b0 0x7f2f0415fd10 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f2ef000c520 tx=0x7f2ef000c830 comp rx=0 tx=0).stop 2024-04-06T13:45:37.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.119+0000 7f2f0ab7a700 1 -- 172.21.15.189:0/391431789 shutdown_connections 2024-04-06T13:45:37.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.119+0000 7f2f0ab7a700 1 --2- 172.21.15.189:0/391431789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2ee806adb0 0x7f2ee806d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.119+0000 7f2f0ab7a700 1 --2- 172.21.15.189:0/391431789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f0415e500 0x7f2f04160250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.119+0000 7f2f0ab7a700 1 --2- 172.21.15.189:0/391431789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f0406a730 0x7f2f041508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.119+0000 7f2f0ab7a700 1 --2- 172.21.15.189:0/391431789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f0415d7b0 0x7f2f0415fd10 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.119+0000 7f2f0ab7a700 1 -- 172.21.15.189:0/391431789 >> 172.21.15.189:0/391431789 conn(0x7f2f040d00b0 msgr2=0x7f2f040ad240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:37.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.119+0000 7f2f0ab7a700 1 -- 172.21.15.189:0/391431789 shutdown_connections 2024-04-06T13:45:37.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.119+0000 7f2f0ab7a700 1 -- 172.21.15.189:0/391431789 wait complete. 2024-04-06T13:45:37.133 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-06T13:45:37.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.254+0000 7f8bd069b700 1 Processor -- start 2024-04-06T13:45:37.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.254+0000 7f8bd069b700 1 -- start start 2024-04-06T13:45:37.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.255+0000 7f8bd069b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8bc80b39d0 0x7f8bc80af040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:37.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.255+0000 7f8bd069b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bc80723f0 0x7f8bc80728a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:37.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.255+0000 7f8bd069b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc8071950 con 0x7f8bc815d7b0 2024-04-06T13:45:37.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.255+0000 7f8bd069b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc80b4170 con 0x7f8bc80b39d0 2024-04-06T13:45:37.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.255+0000 7f8bd069b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc80b3e90 con 0x7f8bc80723f0 2024-04-06T13:45:37.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.255+0000 7f8bcdc36700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8bc80b39d0 0x7f8bc80af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:37.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.255+0000 7f8bce437700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8bc815d7b0 0x7f8bc80ac8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48144/0 (socket says 172.21.15.189:48144) 2024-04-06T13:45:37.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.255+0000 7f8bce437700 1 -- 172.21.15.189:0/2625472842 learned_addr learned my addr 172.21.15.189:0/2625472842 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:37.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.255+0000 7f8bcec38700 1 --2- 172.21.15.189:0/2625472842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bc80723f0 0x7f8bc80728a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:37.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.255+0000 7f8bcdc36700 1 -- 172.21.15.189:0/2625472842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bc80723f0 msgr2=0x7f8bc80728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:37.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.255+0000 7f8bcdc36700 1 --2- 172.21.15.189:0/2625472842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bc80723f0 0x7f8bc80728a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.255+0000 7f8bcdc36700 1 -- 172.21.15.189:0/2625472842 >> v1:172.21.15.189:6789/0 conn(0x7f8bc815d7b0 legacy=0x7f8bc80ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:37.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.255+0000 7f8bcdc36700 1 -- 172.21.15.189:0/2625472842 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc8071bf0 con 0x7f8bc80b39d0 2024-04-06T13:45:37.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.255+0000 7f8bcd435700 1 -- 172.21.15.189:0/2625472842 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674064915 0 0) 0x7f8bc8071950 con 0x7f8bc815d7b0 2024-04-06T13:45:37.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.255+0000 7f8bcdc36700 1 --2- 172.21.15.189:0/2625472842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc80b39d0 0x7f8bc80af040 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009840 tx=0x7f8bb8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60a02293dffd247e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:37.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.256+0000 7f8bcd435700 1 -- 172.21.15.189:0/2625472842 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8011830 con 0x7f8bc80b39d0 2024-04-06T13:45:37.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.256+0000 7f8bcd435700 1 -- 172.21.15.189:0/2625472842 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb8011990 con 0x7f8bc80b39d0 2024-04-06T13:45:37.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.256+0000 7f8bcd435700 1 -- 172.21.15.189:0/2625472842 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8011d20 con 0x7f8bc80b39d0 2024-04-06T13:45:37.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.256+0000 7f8bd069b700 1 -- 172.21.15.189:0/2625472842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc80b39d0 msgr2=0x7f8bc80af040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:37.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.256+0000 7f8bd069b700 1 --2- 172.21.15.189:0/2625472842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc80b39d0 0x7f8bc80af040 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009840 tx=0x7f8bb8009b50 comp rx=0 tx=0).stop 2024-04-06T13:45:37.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.256+0000 7f8bd069b700 1 -- 172.21.15.189:0/2625472842 shutdown_connections 2024-04-06T13:45:37.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.256+0000 7f8bd069b700 1 --2- 172.21.15.189:0/2625472842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bc80723f0 0x7f8bc80728a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.256+0000 7f8bd069b700 1 --2- 172.21.15.189:0/2625472842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc80b39d0 0x7f8bc80af040 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.256+0000 7f8bd069b700 1 -- 172.21.15.189:0/2625472842 >> 172.21.15.189:0/2625472842 conn(0x7f8bc80d00b0 msgr2=0x7f8bc80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:37.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.256+0000 7f8bd069b700 1 -- 172.21.15.189:0/2625472842 shutdown_connections 2024-04-06T13:45:37.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.256+0000 7f8bd069b700 1 -- 172.21.15.189:0/2625472842 wait complete. 2024-04-06T13:45:37.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.257+0000 7f8bd069b700 1 Processor -- start 2024-04-06T13:45:37.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.257+0000 7f8bd069b700 1 -- start start 2024-04-06T13:45:37.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.258+0000 7f8bd069b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc80723f0 0x7f8bc81590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:37.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.258+0000 7f8bd069b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8bc80b39d0 0x7f8bc814c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:37.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.258+0000 7f8bd069b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bc815e8e0 0x7f8bc815c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:37.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.258+0000 7f8bd069b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc8159970 con 0x7f8bc80723f0 2024-04-06T13:45:37.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.258+0000 7f8bd069b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc81f5e90 con 0x7f8bc80b39d0 2024-04-06T13:45:37.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.258+0000 7f8bd069b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc81f6730 con 0x7f8bc815e8e0 2024-04-06T13:45:37.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.258+0000 7f8bce437700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc80723f0 0x7f8bc81590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:37.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.258+0000 7f8bce437700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc80723f0 0x7f8bc81590b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56652/0 (socket says 172.21.15.189:56652) 2024-04-06T13:45:37.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.258+0000 7f8bce437700 1 -- 172.21.15.189:0/772320454 learned_addr learned my addr 172.21.15.189:0/772320454 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:37.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.258+0000 7f8bcec38700 1 --2- 172.21.15.189:0/772320454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bc815e8e0 0x7f8bc815c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:37.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.258+0000 7f8bcdc36700 1 --2- 172.21.15.189:0/772320454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8bc80b39d0 0x7f8bc814c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:37.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.259+0000 7f8bce437700 1 -- 172.21.15.189:0/772320454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bc815e8e0 msgr2=0x7f8bc815c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:37.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.259+0000 7f8bce437700 1 --2- 172.21.15.189:0/772320454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bc815e8e0 0x7f8bc815c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.259+0000 7f8bce437700 1 -- 172.21.15.189:0/772320454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8bc80b39d0 msgr2=0x7f8bc814c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:37.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.259+0000 7f8bce437700 1 --2- 172.21.15.189:0/772320454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8bc80b39d0 0x7f8bc814c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.259+0000 7f8bce437700 1 -- 172.21.15.189:0/772320454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb80094a0 con 0x7f8bc80723f0 2024-04-06T13:45:37.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.259+0000 7f8bcdc36700 1 --2- 172.21.15.189:0/772320454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8bc80b39d0 0x7f8bc814c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:37.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.259+0000 7f8bce437700 1 --2- 172.21.15.189:0/772320454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc80723f0 0x7f8bc81590b0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400c750 tx=0x7f8bc400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:37.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.259+0000 7f8bbeffd700 1 -- 172.21.15.189:0/772320454 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4004c80 con 0x7f8bc80723f0 2024-04-06T13:45:37.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.259+0000 7f8bbeffd700 1 -- 172.21.15.189:0/772320454 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc4004e00 con 0x7f8bc80723f0 2024-04-06T13:45:37.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.259+0000 7f8bbeffd700 1 -- 172.21.15.189:0/772320454 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4007310 con 0x7f8bc80723f0 2024-04-06T13:45:37.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.259+0000 7f8bd069b700 1 -- 172.21.15.189:0/772320454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc814ca60 con 0x7f8bc80723f0 2024-04-06T13:45:37.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.259+0000 7f8bd069b700 1 -- 172.21.15.189:0/772320454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc8159690 con 0x7f8bc80723f0 2024-04-06T13:45:37.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.260+0000 7f8bd069b700 1 -- 172.21.15.189:0/772320454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc8067bb0 con 0x7f8bc80723f0 2024-04-06T13:45:37.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.260+0000 7f8bbeffd700 1 -- 172.21.15.189:0/772320454 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f8bc4007470 con 0x7f8bc80723f0 2024-04-06T13:45:37.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.261+0000 7f8bbeffd700 1 --2- 172.21.15.189:0/772320454 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8bac06adb0 0x7f8bac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:37.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.261+0000 7f8bbeffd700 1 -- 172.21.15.189:0/772320454 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8bc4089060 con 0x7f8bc80723f0 2024-04-06T13:45:37.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.261+0000 7f8bcdc36700 1 --2- 172.21.15.189:0/772320454 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8bac06adb0 0x7f8bac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:37.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.261+0000 7f8bcdc36700 1 --2- 172.21.15.189:0/772320454 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8bac06adb0 0x7f8bac06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8bb80090b0 tx=0x7f8bb8008030 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:37.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.265+0000 7f8bbeffd700 1 -- 172.21.15.189:0/772320454 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8bc40542f0 con 0x7f8bc80723f0 2024-04-06T13:45:37.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.462+0000 7f8bd069b700 1 -- 172.21.15.189:0/772320454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8bc80d0e00 con 0x7f8bc80723f0 2024-04-06T13:45:37.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.463+0000 7f8bbeffd700 1 -- 172.21.15.189:0/772320454 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v205) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8bc4057910 con 0x7f8bc80723f0 2024-04-06T13:45:37.465 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:37.465 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T13:45:37.465 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:37.465 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 1, 2024-04-06T13:45:37.466 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": ".mgr" 2024-04-06T13:45:37.466 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:37.466 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T13:45:37.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.465+0000 7f8bd069b700 1 -- 172.21.15.189:0/772320454 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8bac06adb0 msgr2=0x7f8bac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:37.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.465+0000 7f8bd069b700 1 --2- 172.21.15.189:0/772320454 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8bac06adb0 0x7f8bac06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8bb80090b0 tx=0x7f8bb8008030 comp rx=0 tx=0).stop 2024-04-06T13:45:37.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.465+0000 7f8bd069b700 1 -- 172.21.15.189:0/772320454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc80723f0 msgr2=0x7f8bc81590b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:37.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.465+0000 7f8bd069b700 1 --2- 172.21.15.189:0/772320454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc80723f0 0x7f8bc81590b0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400c750 tx=0x7f8bc400cb10 comp rx=0 tx=0).stop 2024-04-06T13:45:37.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.465+0000 7f8bd069b700 1 -- 172.21.15.189:0/772320454 shutdown_connections 2024-04-06T13:45:37.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.465+0000 7f8bd069b700 1 --2- 172.21.15.189:0/772320454 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8bac06adb0 0x7f8bac06d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.465+0000 7f8bd069b700 1 --2- 172.21.15.189:0/772320454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8bc815e8e0 0x7f8bc815c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.465+0000 7f8bd069b700 1 --2- 172.21.15.189:0/772320454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bc80723f0 0x7f8bc81590b0 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.465+0000 7f8bd069b700 1 --2- 172.21.15.189:0/772320454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8bc80b39d0 0x7f8bc814c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.465+0000 7f8bd069b700 1 -- 172.21.15.189:0/772320454 >> 172.21.15.189:0/772320454 conn(0x7f8bc80d00b0 msgr2=0x7f8bc80ac8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:37.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.465+0000 7f8bd069b700 1 -- 172.21.15.189:0/772320454 shutdown_connections 2024-04-06T13:45:37.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.465+0000 7f8bd069b700 1 -- 172.21.15.189:0/772320454 wait complete. 2024-04-06T13:45:37.479 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-06T13:45:37.479 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-06T13:45:37.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.606+0000 7f660f0f8700 1 Processor -- start 2024-04-06T13:45:37.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.606+0000 7f660f0f8700 1 -- start start 2024-04-06T13:45:37.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.606+0000 7f660f0f8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f66080d17a0 0x7f660806b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:37.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.606+0000 7f660f0f8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6608146dd0 0x7f660806d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:37.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.606+0000 7f660f0f8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f66080cedf0 0x7f66080cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:37.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.606+0000 7f660f0f8700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66080cf760 con 0x7f66080cedf0 2024-04-06T13:45:37.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.606+0000 7f660f0f8700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f660806c0a0 con 0x7f66080d17a0 2024-04-06T13:45:37.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.606+0000 7f660f0f8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f660806c380 con 0x7f6608146dd0 2024-04-06T13:45:37.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.607+0000 7f660d695700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f66080cedf0 0x7f66080cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:37.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.607+0000 7f660d695700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f66080cedf0 0x7f66080cf220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56660/0 (socket says 172.21.15.189:56660) 2024-04-06T13:45:37.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.607+0000 7f660d695700 1 -- 172.21.15.189:0/28653722 learned_addr learned my addr 172.21.15.189:0/28653722 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:37.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.607+0000 7f6607fff700 1 --2- 172.21.15.189:0/28653722 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6608146dd0 0x7f660806d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:37.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.607+0000 7f660ce94700 1 --2- 172.21.15.189:0/28653722 >> v2:172.21.15.192:3300/0 conn(0x7f66080d17a0 0x7f660806b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:37.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.607+0000 7f660d695700 1 -- 172.21.15.189:0/28653722 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6608146dd0 msgr2=0x7f660806d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:37.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.607+0000 7f660d695700 1 --2- 172.21.15.189:0/28653722 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6608146dd0 0x7f660806d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.607+0000 7f660d695700 1 -- 172.21.15.189:0/28653722 >> v2:172.21.15.192:3300/0 conn(0x7f66080d17a0 msgr2=0x7f660806b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:37.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.607+0000 7f660d695700 1 --2- 172.21.15.189:0/28653722 >> v2:172.21.15.192:3300/0 conn(0x7f66080d17a0 0x7f660806b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.607+0000 7f660d695700 1 -- 172.21.15.189:0/28653722 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660806c660 con 0x7f66080cedf0 2024-04-06T13:45:37.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.607+0000 7f660d695700 1 --2- 172.21.15.189:0/28653722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66080cedf0 0x7f66080cf220 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f660804fb40 tx=0x7f660000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a8f23fcaccca418 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:37.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.607+0000 7f66077fe700 1 -- 172.21.15.189:0/28653722 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660000e070 con 0x7f66080cedf0 2024-04-06T13:45:37.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.607+0000 7f66077fe700 1 -- 172.21.15.189:0/28653722 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660000b910 con 0x7f66080cedf0 2024-04-06T13:45:37.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.607+0000 7f66077fe700 1 -- 172.21.15.189:0/28653722 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660000bcc0 con 0x7f66080cedf0 2024-04-06T13:45:37.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.608+0000 7f660f0f8700 1 -- 172.21.15.189:0/28653722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66080cedf0 msgr2=0x7f66080cf220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:37.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.608+0000 7f660f0f8700 1 --2- 172.21.15.189:0/28653722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66080cedf0 0x7f66080cf220 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f660804fb40 tx=0x7f660000b330 comp rx=0 tx=0).stop 2024-04-06T13:45:37.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.608+0000 7f660f0f8700 1 -- 172.21.15.189:0/28653722 shutdown_connections 2024-04-06T13:45:37.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.608+0000 7f660f0f8700 1 --2- 172.21.15.189:0/28653722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66080cedf0 0x7f66080cf220 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.608+0000 7f660f0f8700 1 --2- 172.21.15.189:0/28653722 >> v2:172.21.15.192:3300/0 conn(0x7f66080d17a0 0x7f660806b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.608+0000 7f660f0f8700 1 --2- 172.21.15.189:0/28653722 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6608146dd0 0x7f660806d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.608+0000 7f660f0f8700 1 -- 172.21.15.189:0/28653722 >> 172.21.15.189:0/28653722 conn(0x7f66080b2db0 msgr2=0x7f66080b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:37.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.608+0000 7f660f0f8700 1 -- 172.21.15.189:0/28653722 shutdown_connections 2024-04-06T13:45:37.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.608+0000 7f660f0f8700 1 -- 172.21.15.189:0/28653722 wait complete. 2024-04-06T13:45:37.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.609+0000 7f660f0f8700 1 Processor -- start 2024-04-06T13:45:37.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.609+0000 7f660f0f8700 1 -- start start 2024-04-06T13:45:37.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.610+0000 7f660f0f8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66080cedf0 0x7f66080b0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:37.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.610+0000 7f660f0f8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66080d17a0 0x7f660815a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:37.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.610+0000 7f660f0f8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6608146dd0 0x7f66080c3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:37.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.610+0000 7f660f0f8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66080b0740 con 0x7f66080d17a0 2024-04-06T13:45:37.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.610+0000 7f660f0f8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6608157290 con 0x7f6608146dd0 2024-04-06T13:45:37.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.610+0000 7f660f0f8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66080af160 con 0x7f66080cedf0 2024-04-06T13:45:37.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.610+0000 7f660ce94700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66080cedf0 0x7f66080b0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:37.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.610+0000 7f660ce94700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66080cedf0 0x7f66080b0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:52036/0 (socket says 172.21.15.189:52036) 2024-04-06T13:45:37.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.610+0000 7f660ce94700 1 -- 172.21.15.189:0/2765623279 learned_addr learned my addr 172.21.15.189:0/2765623279 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:37.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.610+0000 7f660d695700 1 --2- 172.21.15.189:0/2765623279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6608146dd0 0x7f66080c3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:37.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.610+0000 7f6607fff700 1 --2- 172.21.15.189:0/2765623279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66080d17a0 0x7f660815a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:37.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.610+0000 7f6607fff700 1 -- 172.21.15.189:0/2765623279 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66080cedf0 msgr2=0x7f66080b0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:37.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.610+0000 7f6607fff700 1 --2- 172.21.15.189:0/2765623279 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66080cedf0 0x7f66080b0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.610+0000 7f6607fff700 1 -- 172.21.15.189:0/2765623279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6608146dd0 msgr2=0x7f66080c3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:37.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.610+0000 7f6607fff700 1 --2- 172.21.15.189:0/2765623279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6608146dd0 0x7f66080c3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:37.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.610+0000 7f6607fff700 1 -- 172.21.15.189:0/2765623279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6600009b30 con 0x7f66080d17a0 2024-04-06T13:45:37.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.611+0000 7f6607fff700 1 --2- 172.21.15.189:0/2765623279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66080d17a0 0x7f660815a370 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f65f400e900 tx=0x7f65f400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:37.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.611+0000 7f66057fa700 1 -- 172.21.15.189:0/2765623279 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f4013680 con 0x7f66080d17a0 2024-04-06T13:45:37.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.611+0000 7f66057fa700 1 -- 172.21.15.189:0/2765623279 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f40137e0 con 0x7f66080d17a0 2024-04-06T13:45:37.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.611+0000 7f660f0f8700 1 -- 172.21.15.189:0/2765623279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66080af440 con 0x7f66080d17a0 2024-04-06T13:45:37.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.611+0000 7f66057fa700 1 -- 172.21.15.189:0/2765623279 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f4013b50 con 0x7f66080d17a0 2024-04-06T13:45:37.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.611+0000 7f660f0f8700 1 -- 172.21.15.189:0/2765623279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66080afa00 con 0x7f66080d17a0 2024-04-06T13:45:37.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.612+0000 7f660f0f8700 1 -- 172.21.15.189:0/2765623279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6608074990 con 0x7f66080d17a0 2024-04-06T13:45:37.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.612+0000 7f66057fa700 1 -- 172.21.15.189:0/2765623279 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f65f4013cb0 con 0x7f66080d17a0 2024-04-06T13:45:37.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.613+0000 7f66057fa700 1 --2- 172.21.15.189:0/2765623279 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f65ec06ade0 0x7f65ec06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:37.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.613+0000 7f66057fa700 1 -- 172.21.15.189:0/2765623279 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f65f40894d0 con 0x7f66080d17a0 2024-04-06T13:45:37.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.613+0000 7f660ce94700 1 --2- 172.21.15.189:0/2765623279 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f65ec06ade0 0x7f65ec06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:37.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.613+0000 7f660ce94700 1 --2- 172.21.15.189:0/2765623279 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f65ec06ade0 0x7f65ec06d2a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f65fc009c00 tx=0x7f65fc009210 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:37.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.617+0000 7f66057fa700 1 -- 172.21.15.189:0/2765623279 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f65f40546c0 con 0x7f66080d17a0 2024-04-06T13:45:37.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:37.815+0000 7f660f0f8700 1 -- 172.21.15.189:0/2765623279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f66080b3900 con 0x7f66080d17a0 2024-04-06T13:45:38.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.651+0000 7f66057fa700 1 -- 172.21.15.189:0/2765623279 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v206) v1 ==== 138+0+0 (secure 0 0 0) 0x7f65f4057ce0 con 0x7f66080d17a0 2024-04-06T13:45:38.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.656+0000 7f660f0f8700 1 -- 172.21.15.189:0/2765623279 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f65ec06ade0 msgr2=0x7f65ec06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:38.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.656+0000 7f660f0f8700 1 --2- 172.21.15.189:0/2765623279 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f65ec06ade0 0x7f65ec06d2a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f65fc009c00 tx=0x7f65fc009210 comp rx=0 tx=0).stop 2024-04-06T13:45:38.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.656+0000 7f660f0f8700 1 -- 172.21.15.189:0/2765623279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66080d17a0 msgr2=0x7f660815a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:38.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.656+0000 7f660f0f8700 1 --2- 172.21.15.189:0/2765623279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66080d17a0 0x7f660815a370 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f65f400e900 tx=0x7f65f400ec10 comp rx=0 tx=0).stop 2024-04-06T13:45:38.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.656+0000 7f660f0f8700 1 -- 172.21.15.189:0/2765623279 shutdown_connections 2024-04-06T13:45:38.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.656+0000 7f660f0f8700 1 --2- 172.21.15.189:0/2765623279 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f65ec06ade0 0x7f65ec06d2a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:38.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.656+0000 7f660f0f8700 1 --2- 172.21.15.189:0/2765623279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6608146dd0 0x7f66080c3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:38.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.656+0000 7f660f0f8700 1 --2- 172.21.15.189:0/2765623279 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66080cedf0 0x7f66080b0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:38.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.656+0000 7f660f0f8700 1 --2- 172.21.15.189:0/2765623279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66080d17a0 0x7f660815a370 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:38.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.656+0000 7f660f0f8700 1 -- 172.21.15.189:0/2765623279 >> 172.21.15.189:0/2765623279 conn(0x7f66080b2db0 msgr2=0x7f66080b3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:38.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.656+0000 7f660f0f8700 1 -- 172.21.15.189:0/2765623279 shutdown_connections 2024-04-06T13:45:38.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.656+0000 7f660f0f8700 1 -- 172.21.15.189:0/2765623279 wait complete. 2024-04-06T13:45:38.658 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' created 2024-04-06T13:45:38.671 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-06T13:45:38.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.802+0000 7f7ac4c5f700 1 Processor -- start 2024-04-06T13:45:38.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.802+0000 7f7ac4c5f700 1 -- start start 2024-04-06T13:45:38.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.803+0000 7f7ac4c5f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7ac00b3e20 0x7f7ac015e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:38.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.803+0000 7f7ac4c5f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ac00b3990 con 0x7f7ac006ae40 2024-04-06T13:45:38.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.803+0000 7f7ac4c5f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ac00b3060 con 0x7f7ac015d7b0 2024-04-06T13:45:38.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.803+0000 7f7ac4c5f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac00b4230 con 0x7f7ac00b3e20 2024-04-06T13:45:38.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.803+0000 7f7abed9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7ac00b3e20 0x7f7ac015e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:38.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.803+0000 7f7abed9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7ac00b3e20 0x7f7ac015e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56674/0 (socket says 172.21.15.189:56674) 2024-04-06T13:45:38.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.803+0000 7f7abdd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7ac006ae40 0x7f7ac015ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48154/0 (socket says 172.21.15.189:48154) 2024-04-06T13:45:38.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.803+0000 7f7abed9d700 1 -- 172.21.15.189:0/2302173056 learned_addr learned my addr 172.21.15.189:0/2302173056 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:38.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.803+0000 7f7abed9d700 1 -- 172.21.15.189:0/2302173056 >> v1:172.21.15.192:6789/0 conn(0x7f7ac015d7b0 legacy=0x7f7ac006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:38.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.803+0000 7f7abed9d700 1 -- 172.21.15.189:0/2302173056 >> v1:172.21.15.189:6789/0 conn(0x7f7ac006ae40 legacy=0x7f7ac015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:38.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.803+0000 7f7abed9d700 1 -- 172.21.15.189:0/2302173056 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac00b3340 con 0x7f7ac00b3e20 2024-04-06T13:45:38.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.803+0000 7f7abd59a700 1 -- 172.21.15.189:0/2302173056 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497517812 0 0) 0x7f7ac00b3990 con 0x7f7ac006ae40 2024-04-06T13:45:38.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.803+0000 7f7abed9d700 1 --2- 172.21.15.189:0/2302173056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ac00b3e20 0x7f7ac015e500 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8009e50 tx=0x7f7ab800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5dd1717f4bd60cc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:38.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.803+0000 7f7abd59a700 1 -- 172.21.15.189:0/2302173056 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab800e070 con 0x7f7ac00b3e20 2024-04-06T13:45:38.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.803+0000 7f7abd59a700 1 -- 172.21.15.189:0/2302173056 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab8003a20 con 0x7f7ac00b3e20 2024-04-06T13:45:38.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.803+0000 7f7abd59a700 1 -- 172.21.15.189:0/2302173056 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8003dd0 con 0x7f7ac00b3e20 2024-04-06T13:45:38.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.804+0000 7f7ac4c5f700 1 -- 172.21.15.189:0/2302173056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ac00b3e20 msgr2=0x7f7ac015e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:38.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.804+0000 7f7ac4c5f700 1 --2- 172.21.15.189:0/2302173056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ac00b3e20 0x7f7ac015e500 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8009e50 tx=0x7f7ab800b330 comp rx=0 tx=0).stop 2024-04-06T13:45:38.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.804+0000 7f7ac4c5f700 1 -- 172.21.15.189:0/2302173056 shutdown_connections 2024-04-06T13:45:38.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.804+0000 7f7ac4c5f700 1 --2- 172.21.15.189:0/2302173056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ac00b3e20 0x7f7ac015e500 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:38.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.804+0000 7f7ac4c5f700 1 -- 172.21.15.189:0/2302173056 >> 172.21.15.189:0/2302173056 conn(0x7f7ac00d00b0 msgr2=0x7f7ac00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:38.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.804+0000 7f7ac4c5f700 1 -- 172.21.15.189:0/2302173056 shutdown_connections 2024-04-06T13:45:38.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.804+0000 7f7ac4c5f700 1 -- 172.21.15.189:0/2302173056 wait complete. 2024-04-06T13:45:38.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.805+0000 7f7ac4c5f700 1 Processor -- start 2024-04-06T13:45:38.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.805+0000 7f7ac4c5f700 1 -- start start 2024-04-06T13:45:38.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.806+0000 7f7ac4c5f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ac006ae40 0x7f7ac01f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:38.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.806+0000 7f7ac4c5f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ac00b3e20 0x7f7ac015c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:38.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.806+0000 7f7ac4c5f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ac015d7b0 0x7f7ac015c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:38.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.806+0000 7f7ac4c5f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac014c510 con 0x7f7ac015d7b0 2024-04-06T13:45:38.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.806+0000 7f7ac4c5f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac0149410 con 0x7f7ac006ae40 2024-04-06T13:45:38.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.806+0000 7f7ac4c5f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac014b890 con 0x7f7ac00b3e20 2024-04-06T13:45:38.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.806+0000 7f7abed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ac015d7b0 0x7f7ac015c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:38.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.806+0000 7f7abed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ac015d7b0 0x7f7ac015c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56680/0 (socket says 172.21.15.189:56680) 2024-04-06T13:45:38.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.806+0000 7f7abed9d700 1 -- 172.21.15.189:0/416552428 learned_addr learned my addr 172.21.15.189:0/416552428 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:38.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.806+0000 7f7abe59c700 1 --2- 172.21.15.189:0/416552428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ac006ae40 0x7f7ac01f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:38.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.806+0000 7f7abdd9b700 1 --2- 172.21.15.189:0/416552428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ac00b3e20 0x7f7ac015c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:38.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.807+0000 7f7abed9d700 1 -- 172.21.15.189:0/416552428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ac00b3e20 msgr2=0x7f7ac015c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:38.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.807+0000 7f7abed9d700 1 --2- 172.21.15.189:0/416552428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ac00b3e20 0x7f7ac015c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:38.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.807+0000 7f7abed9d700 1 -- 172.21.15.189:0/416552428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ac006ae40 msgr2=0x7f7ac01f25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:38.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.807+0000 7f7abed9d700 1 --2- 172.21.15.189:0/416552428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ac006ae40 0x7f7ac01f25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:38.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.807+0000 7f7abed9d700 1 -- 172.21.15.189:0/416552428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab8009b30 con 0x7f7ac015d7b0 2024-04-06T13:45:38.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.807+0000 7f7abdd9b700 1 --2- 172.21.15.189:0/416552428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ac00b3e20 0x7f7ac015c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:38.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.807+0000 7f7abed9d700 1 --2- 172.21.15.189:0/416552428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ac015d7b0 0x7f7ac015c6c0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f7ac006c820 tx=0x7f7ab8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:38.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.807+0000 7f7aaeffd700 1 -- 172.21.15.189:0/416552428 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab80092e0 con 0x7f7ac015d7b0 2024-04-06T13:45:38.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.807+0000 7f7aaeffd700 1 -- 172.21.15.189:0/416552428 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab8004240 con 0x7f7ac015d7b0 2024-04-06T13:45:38.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.807+0000 7f7aaeffd700 1 -- 172.21.15.189:0/416552428 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8004610 con 0x7f7ac015d7b0 2024-04-06T13:45:38.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.807+0000 7f7ac4c5f700 1 -- 172.21.15.189:0/416552428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac014bb70 con 0x7f7ac015d7b0 2024-04-06T13:45:38.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.807+0000 7f7ac4c5f700 1 -- 172.21.15.189:0/416552428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac014c130 con 0x7f7ac015d7b0 2024-04-06T13:45:38.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.808+0000 7f7ac4c5f700 1 -- 172.21.15.189:0/416552428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac0067bb0 con 0x7f7ac015d7b0 2024-04-06T13:45:38.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.809+0000 7f7aaeffd700 1 -- 172.21.15.189:0/416552428 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7ab8019070 con 0x7f7ac015d7b0 2024-04-06T13:45:38.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.809+0000 7f7aaeffd700 1 --2- 172.21.15.189:0/416552428 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7aa006adb0 0x7f7aa006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:38.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.809+0000 7f7aaeffd700 1 -- 172.21.15.189:0/416552428 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f7ab808a420 con 0x7f7ac015d7b0 2024-04-06T13:45:38.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.809+0000 7f7abe59c700 1 --2- 172.21.15.189:0/416552428 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7aa006adb0 0x7f7aa006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:38.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.810+0000 7f7abe59c700 1 --2- 172.21.15.189:0/416552428 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7aa006adb0 0x7f7aa006d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4009960 tx=0x7f7ab40090e0 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:38.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:38.814+0000 7f7aaeffd700 1 -- 172.21.15.189:0/416552428 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7ab8058f50 con 0x7f7ac015d7b0 2024-04-06T13:45:39.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.008+0000 7f7ac4c5f700 1 -- 172.21.15.189:0/416552428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7ac014c650 con 0x7f7ac015d7b0 2024-04-06T13:45:39.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.655+0000 7f7aaeffd700 1 -- 172.21.15.189:0/416552428 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v207) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7ab8058ae0 con 0x7f7ac015d7b0 2024-04-06T13:45:39.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.662+0000 7f7ac4c5f700 1 -- 172.21.15.189:0/416552428 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7aa006adb0 msgr2=0x7f7aa006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:39.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.662+0000 7f7ac4c5f700 1 --2- 172.21.15.189:0/416552428 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7aa006adb0 0x7f7aa006d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4009960 tx=0x7f7ab40090e0 comp rx=0 tx=0).stop 2024-04-06T13:45:39.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.662+0000 7f7ac4c5f700 1 -- 172.21.15.189:0/416552428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ac015d7b0 msgr2=0x7f7ac015c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:39.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.662+0000 7f7ac4c5f700 1 --2- 172.21.15.189:0/416552428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ac015d7b0 0x7f7ac015c6c0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f7ac006c820 tx=0x7f7ab8004ad0 comp rx=0 tx=0).stop 2024-04-06T13:45:39.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.662+0000 7f7ac4c5f700 1 -- 172.21.15.189:0/416552428 shutdown_connections 2024-04-06T13:45:39.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.662+0000 7f7ac4c5f700 1 --2- 172.21.15.189:0/416552428 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7aa006adb0 0x7f7aa006d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:39.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.662+0000 7f7ac4c5f700 1 --2- 172.21.15.189:0/416552428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ac015d7b0 0x7f7ac015c6c0 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:39.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.662+0000 7f7ac4c5f700 1 --2- 172.21.15.189:0/416552428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ac006ae40 0x7f7ac01f25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:39.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.662+0000 7f7ac4c5f700 1 --2- 172.21.15.189:0/416552428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ac00b3e20 0x7f7ac015c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:39.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.662+0000 7f7ac4c5f700 1 -- 172.21.15.189:0/416552428 >> 172.21.15.189:0/416552428 conn(0x7f7ac00d00b0 msgr2=0x7f7ac00ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:39.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.663+0000 7f7ac4c5f700 1 -- 172.21.15.189:0/416552428 shutdown_connections 2024-04-06T13:45:39.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.664+0000 7f7ac4c5f700 1 -- 172.21.15.189:0/416552428 wait complete. 2024-04-06T13:45:39.677 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' created 2024-04-06T13:45:39.691 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-06T13:45:39.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.821+0000 7fcaf7078700 1 Processor -- start 2024-04-06T13:45:39.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.821+0000 7fcaf7078700 1 -- start start 2024-04-06T13:45:39.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.821+0000 7fcaf7078700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcaf00cd3e0 0x7fcaf00cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:39.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.821+0000 7fcaf7078700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcaf006b8c0 0x7fcaf006dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:39.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.821+0000 7fcaf7078700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf006ceb0 con 0x7fcaf015d7c0 2024-04-06T13:45:39.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.821+0000 7fcaf7078700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaf00cdfa0 con 0x7fcaf00cd3e0 2024-04-06T13:45:39.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.821+0000 7fcaf7078700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf006bcf0 con 0x7fcaf006b8c0 2024-04-06T13:45:39.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.822+0000 7fcaf4e14700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcaf00cd3e0 0x7fcaf00cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:39.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.822+0000 7fcaf4e14700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcaf00cd3e0 0x7fcaf00cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48094/0 (socket says 172.21.15.189:48094) 2024-04-06T13:45:39.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.822+0000 7fcaf4e14700 1 -- 172.21.15.189:0/1575722456 learned_addr learned my addr 172.21.15.189:0/1575722456 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:39.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.822+0000 7fcaf5615700 1 --2- 172.21.15.189:0/1575722456 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcaf006b8c0 0x7fcaf006dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:39.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.822+0000 7fcaf4e14700 1 -- 172.21.15.189:0/1575722456 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcaf006b8c0 msgr2=0x7fcaf006dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:39.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.822+0000 7fcaf4e14700 1 --2- 172.21.15.189:0/1575722456 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcaf006b8c0 0x7fcaf006dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:39.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.822+0000 7fcaf4e14700 1 -- 172.21.15.189:0/1575722456 >> v1:172.21.15.192:6789/0 conn(0x7fcaf015d7c0 legacy=0x7fcaf006b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:39.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.822+0000 7fcaf4e14700 1 -- 172.21.15.189:0/1575722456 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf006d750 con 0x7fcaf00cd3e0 2024-04-06T13:45:39.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.822+0000 7fcaf4e14700 1 --2- 172.21.15.189:0/1575722456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcaf00cd3e0 0x7fcaf00cd7c0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7fcae40098f0 tx=0x7fcae4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64ccd1fa72a9918d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:39.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.822+0000 7fcaef7fe700 1 -- 172.21.15.189:0/1575722456 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae4004230 con 0x7fcaf00cd3e0 2024-04-06T13:45:39.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.822+0000 7fcaef7fe700 1 -- 172.21.15.189:0/1575722456 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae4004390 con 0x7fcaf00cd3e0 2024-04-06T13:45:39.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.822+0000 7fcaef7fe700 1 -- 172.21.15.189:0/1575722456 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae40036a0 con 0x7fcaf00cd3e0 2024-04-06T13:45:39.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.823+0000 7fcaf7078700 1 -- 172.21.15.189:0/1575722456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcaf00cd3e0 msgr2=0x7fcaf00cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:39.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.823+0000 7fcaf7078700 1 --2- 172.21.15.189:0/1575722456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcaf00cd3e0 0x7fcaf00cd7c0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7fcae40098f0 tx=0x7fcae4009c00 comp rx=0 tx=0).stop 2024-04-06T13:45:39.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.823+0000 7fcaf7078700 1 -- 172.21.15.189:0/1575722456 shutdown_connections 2024-04-06T13:45:39.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.823+0000 7fcaf7078700 1 --2- 172.21.15.189:0/1575722456 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcaf006b8c0 0x7fcaf006dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:39.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.823+0000 7fcaf7078700 1 --2- 172.21.15.189:0/1575722456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcaf00cd3e0 0x7fcaf00cd7c0 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:39.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.823+0000 7fcaf7078700 1 -- 172.21.15.189:0/1575722456 >> 172.21.15.189:0/1575722456 conn(0x7fcaf00cc290 msgr2=0x7fcaf00cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:39.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.823+0000 7fcaf7078700 1 -- 172.21.15.189:0/1575722456 shutdown_connections 2024-04-06T13:45:39.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.823+0000 7fcaf7078700 1 -- 172.21.15.189:0/1575722456 wait complete. 2024-04-06T13:45:39.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.824+0000 7fcaf7078700 1 Processor -- start 2024-04-06T13:45:39.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.824+0000 7fcaf7078700 1 -- start start 2024-04-06T13:45:39.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.825+0000 7fcaf7078700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcaf006b8c0 0x7fcaf01f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:39.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.825+0000 7fcaf7078700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcaf00cd3e0 0x7fcaf0146db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:39.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.825+0000 7fcaf7078700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcaf015d7c0 0x7fcaf00c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:39.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.825+0000 7fcaf7078700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf00c4a70 con 0x7fcaf00cd3e0 2024-04-06T13:45:39.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.825+0000 7fcaf7078700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf00c6160 con 0x7fcaf015d7c0 2024-04-06T13:45:39.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.825+0000 7fcaf7078700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf00c3df0 con 0x7fcaf006b8c0 2024-04-06T13:45:39.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.825+0000 7fcaeffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcaf00cd3e0 0x7fcaf0146db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:39.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.825+0000 7fcaf4e14700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcaf006b8c0 0x7fcaf01f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:39.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.825+0000 7fcaeffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcaf00cd3e0 0x7fcaf0146db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48100/0 (socket says 172.21.15.189:48100) 2024-04-06T13:45:39.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.825+0000 7fcaeffff700 1 -- 172.21.15.189:0/1892063752 learned_addr learned my addr 172.21.15.189:0/1892063752 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:39.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.825+0000 7fcaf5615700 1 --2- 172.21.15.189:0/1892063752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcaf015d7c0 0x7fcaf00c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:39.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.825+0000 7fcaeffff700 1 -- 172.21.15.189:0/1892063752 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcaf006b8c0 msgr2=0x7fcaf01f7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:39.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.825+0000 7fcaeffff700 1 --2- 172.21.15.189:0/1892063752 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcaf006b8c0 0x7fcaf01f7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:39.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.825+0000 7fcaeffff700 1 -- 172.21.15.189:0/1892063752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcaf015d7c0 msgr2=0x7fcaf00c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:39.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.825+0000 7fcaeffff700 1 --2- 172.21.15.189:0/1892063752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcaf015d7c0 0x7fcaf00c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:39.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.825+0000 7fcaeffff700 1 -- 172.21.15.189:0/1892063752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae4009550 con 0x7fcaf00cd3e0 2024-04-06T13:45:39.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.825+0000 7fcaeffff700 1 --2- 172.21.15.189:0/1892063752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcaf00cd3e0 0x7fcaf0146db0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7fcadc00c4a0 tx=0x7fcadc00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:39.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.826+0000 7fcaed7fa700 1 -- 172.21.15.189:0/1892063752 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcadc0054e0 con 0x7fcaf00cd3e0 2024-04-06T13:45:39.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.826+0000 7fcaed7fa700 1 -- 172.21.15.189:0/1892063752 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcadc003680 con 0x7fcaf00cd3e0 2024-04-06T13:45:39.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.826+0000 7fcaed7fa700 1 -- 172.21.15.189:0/1892063752 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcadc003a30 con 0x7fcaf00cd3e0 2024-04-06T13:45:39.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.826+0000 7fcaf7078700 1 -- 172.21.15.189:0/1892063752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf00c40d0 con 0x7fcaf00cd3e0 2024-04-06T13:45:39.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.826+0000 7fcaf7078700 1 -- 172.21.15.189:0/1892063752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf00c4690 con 0x7fcaf00cd3e0 2024-04-06T13:45:39.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.827+0000 7fcaf7078700 1 -- 172.21.15.189:0/1892063752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf00c4820 con 0x7fcaf00cd3e0 2024-04-06T13:45:39.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.827+0000 7fcaed7fa700 1 -- 172.21.15.189:0/1892063752 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fcadc003b90 con 0x7fcaf00cd3e0 2024-04-06T13:45:39.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.828+0000 7fcaed7fa700 1 --2- 172.21.15.189:0/1892063752 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcad406adb0 0x7fcad406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:39.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.828+0000 7fcaed7fa700 1 -- 172.21.15.189:0/1892063752 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fcadc0885e0 con 0x7fcaf00cd3e0 2024-04-06T13:45:39.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.828+0000 7fcaf4e14700 1 --2- 172.21.15.189:0/1892063752 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcad406adb0 0x7fcad406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:39.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.828+0000 7fcaf4e14700 1 --2- 172.21.15.189:0/1892063752 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcad406adb0 0x7fcad406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcae4009130 tx=0x7fcae4008030 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:39.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:39.832+0000 7fcaed7fa700 1 -- 172.21.15.189:0/1892063752 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fcadc0536f0 con 0x7fcaf00cd3e0 2024-04-06T13:45:40.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.070+0000 7fcaf7078700 1 -- 172.21.15.189:0/1892063752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fcaf0074990 con 0x7fcaf00cd3e0 2024-04-06T13:45:40.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.075+0000 7fcaed7fa700 1 -- 172.21.15.189:0/1892063752 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 25 and data pool 26 v89) v1 ==== 174+0+0 (secure 0 0 0) 0x7fcadc056d10 con 0x7fcaf00cd3e0 2024-04-06T13:45:40.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.083+0000 7fcaf7078700 1 -- 172.21.15.189:0/1892063752 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcad406adb0 msgr2=0x7fcad406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.083+0000 7fcaf7078700 1 --2- 172.21.15.189:0/1892063752 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcad406adb0 0x7fcad406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcae4009130 tx=0x7fcae4008030 comp rx=0 tx=0).stop 2024-04-06T13:45:40.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.083+0000 7fcaf7078700 1 -- 172.21.15.189:0/1892063752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcaf00cd3e0 msgr2=0x7fcaf0146db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.083+0000 7fcaf7078700 1 --2- 172.21.15.189:0/1892063752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcaf00cd3e0 0x7fcaf0146db0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7fcadc00c4a0 tx=0x7fcadc00c7b0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.086+0000 7fcaf7078700 1 -- 172.21.15.189:0/1892063752 shutdown_connections 2024-04-06T13:45:40.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.086+0000 7fcaf7078700 1 --2- 172.21.15.189:0/1892063752 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcad406adb0 0x7fcad406d270 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcae4009130 tx=0x7fcae4008030 comp rx=0 tx=0).stop 2024-04-06T13:45:40.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.086+0000 7fcaf7078700 1 --2- 172.21.15.189:0/1892063752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcaf015d7c0 0x7fcaf00c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.086+0000 7fcaf7078700 1 --2- 172.21.15.189:0/1892063752 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcaf006b8c0 0x7fcaf01f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.086+0000 7fcaf7078700 1 --2- 172.21.15.189:0/1892063752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcaf00cd3e0 0x7fcaf0146db0 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.086+0000 7fcaf7078700 1 -- 172.21.15.189:0/1892063752 >> 172.21.15.189:0/1892063752 conn(0x7fcaf00cc290 msgr2=0x7fcaf0157f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:40.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.094+0000 7fcaf7078700 1 -- 172.21.15.189:0/1892063752 shutdown_connections 2024-04-06T13:45:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.094+0000 7fcaf7078700 1 -- 172.21.15.189:0/1892063752 wait complete. 2024-04-06T13:45:40.104 INFO:teuthology.orchestra.run.smithi189.stderr:new fs with metadata pool 25 and data pool 26 2024-04-06T13:45:40.113 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:45:40.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.234+0000 7fce96859700 1 Processor -- start 2024-04-06T13:45:40.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.234+0000 7fce96859700 1 -- start start 2024-04-06T13:45:40.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.235+0000 7fce96859700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce900d0a60 0x7fce900d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:40.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.235+0000 7fce96859700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce900cdf80 con 0x7fce900d1af0 2024-04-06T13:45:40.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.235+0000 7fce96859700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce900d3230 con 0x7fce9015d7e0 2024-04-06T13:45:40.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.235+0000 7fce96859700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce900d1f80 con 0x7fce900d0a60 2024-04-06T13:45:40.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.235+0000 7fce94df6700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fce900d1af0 0x7fce900b66b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55890/0 (socket says 172.21.15.189:55890) 2024-04-06T13:45:40.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.235+0000 7fce94df6700 1 -- 172.21.15.189:0/3093111278 learned_addr learned my addr 172.21.15.189:0/3093111278 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:40.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.235+0000 7fce8f7fe700 1 --2- 172.21.15.189:0/3093111278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce900d0a60 0x7fce900d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:40.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.235+0000 7fce8effd700 1 -- 172.21.15.189:0/3093111278 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738423567 0 0) 0x7fce900cdf80 con 0x7fce900d1af0 2024-04-06T13:45:40.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.235+0000 7fce8effd700 1 -- 172.21.15.189:0/3093111278 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce780035b0 con 0x7fce900d1af0 2024-04-06T13:45:40.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.235+0000 7fce8effd700 1 -- 172.21.15.189:0/3093111278 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160637746 0 0) 0x7fce84001110 con 0x7fce9015d7e0 2024-04-06T13:45:40.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.235+0000 7fce8effd700 1 -- 172.21.15.189:0/3093111278 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce900cdf80 con 0x7fce9015d7e0 2024-04-06T13:45:40.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.235+0000 7fce8effd700 1 -- 172.21.15.189:0/3093111278 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3609230561 0 0) 0x7fce780035b0 con 0x7fce900d1af0 2024-04-06T13:45:40.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.235+0000 7fce8effd700 1 -- 172.21.15.189:0/3093111278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce900d0a60 msgr2=0x7fce900d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:40.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.235+0000 7fce8effd700 1 --2- 172.21.15.189:0/3093111278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce900d0a60 0x7fce900d15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.235+0000 7fce8effd700 1 -- 172.21.15.189:0/3093111278 >> v1:172.21.15.192:6789/0 conn(0x7fce9015d7e0 legacy=0x7fce900d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.235+0000 7fce8effd700 1 -- 172.21.15.189:0/3093111278 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce900d2da0 con 0x7fce900d1af0 2024-04-06T13:45:40.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.235+0000 7fce8effd700 1 -- 172.21.15.189:0/3093111278 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fce880019f0 con 0x7fce900d1af0 2024-04-06T13:45:40.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.236+0000 7fce8effd700 1 -- 172.21.15.189:0/3093111278 >> v1:172.21.15.189:6789/0 conn(0x7fce900d1af0 legacy=0x7fce900b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.236+0000 7fce8effd700 1 --2- 172.21.15.189:0/3093111278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce900d0a60 0x7fce78003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:40.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.236+0000 7fce8effd700 1 --2- 172.21.15.189:0/3093111278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce78004120 0x7fce78004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:40.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.236+0000 7fce8effd700 1 --2- 172.21.15.189:0/3093111278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce78004d90 0x7fce78005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:40.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.236+0000 7fce8effd700 1 -- 172.21.15.189:0/3093111278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce900d1f80 con 0x7fce78004d90 2024-04-06T13:45:40.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.236+0000 7fce8effd700 1 -- 172.21.15.189:0/3093111278 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7800baf0 con 0x7fce78004120 2024-04-06T13:45:40.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.236+0000 7fce8effd700 1 -- 172.21.15.189:0/3093111278 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce7800bcb0 con 0x7fce900d0a60 2024-04-06T13:45:40.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.236+0000 7fce94df6700 1 --2- 172.21.15.189:0/3093111278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce78004d90 0x7fce78005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:40.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.236+0000 7fce8ffff700 1 --2- 172.21.15.189:0/3093111278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce900d0a60 0x7fce78003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:40.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.236+0000 7fce8f7fe700 1 --2- 172.21.15.189:0/3093111278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce78004120 0x7fce78004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:40.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.236+0000 7fce94df6700 1 -- 172.21.15.189:0/3093111278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce900d0a60 msgr2=0x7fce78003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:40.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.236+0000 7fce94df6700 1 --2- 172.21.15.189:0/3093111278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce900d0a60 0x7fce78003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.236+0000 7fce94df6700 1 -- 172.21.15.189:0/3093111278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce78004120 msgr2=0x7fce78004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:40.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.236+0000 7fce94df6700 1 --2- 172.21.15.189:0/3093111278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce78004120 0x7fce78004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.236+0000 7fce94df6700 1 -- 172.21.15.189:0/3093111278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce780035b0 con 0x7fce78004d90 2024-04-06T13:45:40.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.236+0000 7fce94df6700 1 --2- 172.21.15.189:0/3093111278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce78004d90 0x7fce78005210 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7fce8800a7a0 tx=0x7fce78005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a12e7c3be1050e38 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:40.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.236+0000 7fce8effd700 1 -- 172.21.15.189:0/3093111278 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce88012600 con 0x7fce78004d90 2024-04-06T13:45:40.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.236+0000 7fce8effd700 1 -- 172.21.15.189:0/3093111278 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce88012780 con 0x7fce78004d90 2024-04-06T13:45:40.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.236+0000 7fce8effd700 1 -- 172.21.15.189:0/3093111278 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce88012b30 con 0x7fce78004d90 2024-04-06T13:45:40.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.237+0000 7fce96859700 1 -- 172.21.15.189:0/3093111278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce78004d90 msgr2=0x7fce78005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.237+0000 7fce96859700 1 --2- 172.21.15.189:0/3093111278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce78004d90 0x7fce78005210 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7fce8800a7a0 tx=0x7fce78005880 comp rx=0 tx=0).stop 2024-04-06T13:45:40.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.237+0000 7fce94df6700 1 -- 172.21.15.189:0/3093111278 reap_dead start 2024-04-06T13:45:40.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.237+0000 7fce96859700 1 -- 172.21.15.189:0/3093111278 shutdown_connections 2024-04-06T13:45:40.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.237+0000 7fce96859700 1 -- 172.21.15.189:0/3093111278 >> 172.21.15.189:0/3093111278 conn(0x7fce90072250 msgr2=0x7fce90072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:40.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.237+0000 7fce96859700 1 -- 172.21.15.189:0/3093111278 shutdown_connections 2024-04-06T13:45:40.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.237+0000 7fce96859700 1 -- 172.21.15.189:0/3093111278 wait complete. 2024-04-06T13:45:40.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.238+0000 7fce96859700 1 Processor -- start 2024-04-06T13:45:40.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.238+0000 7fce96859700 1 -- start start 2024-04-06T13:45:40.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.239+0000 7fce96859700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce78004d90 0x7fce90149af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:40.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.239+0000 7fce96859700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce900b4980 0x7fce900b4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:40.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.239+0000 7fce96859700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce900c6da0 0x7fce900c7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:40.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.239+0000 7fce96859700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce900b5340 con 0x7fce78004d90 2024-04-06T13:45:40.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.239+0000 7fce96859700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9014a030 con 0x7fce900c6da0 2024-04-06T13:45:40.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.239+0000 7fce96859700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce9014a450 con 0x7fce900b4980 2024-04-06T13:45:40.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.239+0000 7fce8ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce78004d90 0x7fce90149af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:40.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.239+0000 7fce8ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce78004d90 0x7fce90149af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48108/0 (socket says 172.21.15.189:48108) 2024-04-06T13:45:40.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.239+0000 7fce8ffff700 1 -- 172.21.15.189:0/3240646746 learned_addr learned my addr 172.21.15.189:0/3240646746 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:40.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.239+0000 7fce94df6700 1 --2- 172.21.15.189:0/3240646746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce900c6da0 0x7fce900c7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:40.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.239+0000 7fce8f7fe700 1 --2- 172.21.15.189:0/3240646746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce900b4980 0x7fce900b4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:40.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.239+0000 7fce8ffff700 1 -- 172.21.15.189:0/3240646746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce900b4980 msgr2=0x7fce900b4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.239+0000 7fce8ffff700 1 --2- 172.21.15.189:0/3240646746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce900b4980 0x7fce900b4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.239+0000 7fce8ffff700 1 -- 172.21.15.189:0/3240646746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce900c6da0 msgr2=0x7fce900c7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.239+0000 7fce8ffff700 1 --2- 172.21.15.189:0/3240646746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce900c6da0 0x7fce900c7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.239+0000 7fce8ffff700 1 -- 172.21.15.189:0/3240646746 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce9014a730 con 0x7fce78004d90 2024-04-06T13:45:40.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.240+0000 7fce8ffff700 1 --2- 172.21.15.189:0/3240646746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce78004d90 0x7fce90149af0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fce8400f7e0 tx=0x7fce8400faf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:40.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.240+0000 7fce8cff9700 1 -- 172.21.15.189:0/3240646746 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce840052e0 con 0x7fce78004d90 2024-04-06T13:45:40.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.240+0000 7fce8cff9700 1 -- 172.21.15.189:0/3240646746 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce84005460 con 0x7fce78004d90 2024-04-06T13:45:40.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.240+0000 7fce8cff9700 1 -- 172.21.15.189:0/3240646746 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce840038c0 con 0x7fce78004d90 2024-04-06T13:45:40.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.240+0000 7fce96859700 1 -- 172.21.15.189:0/3240646746 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce9014aa10 con 0x7fce78004d90 2024-04-06T13:45:40.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.240+0000 7fce96859700 1 -- 172.21.15.189:0/3240646746 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce900c7b10 con 0x7fce78004d90 2024-04-06T13:45:40.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.241+0000 7fce96859700 1 -- 172.21.15.189:0/3240646746 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce9014af30 con 0x7fce78004d90 2024-04-06T13:45:40.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.241+0000 7fce8cff9700 1 -- 172.21.15.189:0/3240646746 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fce84003a20 con 0x7fce78004d90 2024-04-06T13:45:40.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.242+0000 7fce8cff9700 1 --2- 172.21.15.189:0/3240646746 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fce7c06ade0 0x7fce7c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:40.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.242+0000 7fce8cff9700 1 -- 172.21.15.189:0/3240646746 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fce84089240 con 0x7fce78004d90 2024-04-06T13:45:40.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.242+0000 7fce8f7fe700 1 --2- 172.21.15.189:0/3240646746 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fce7c06ade0 0x7fce7c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:40.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.242+0000 7fce8f7fe700 1 --2- 172.21.15.189:0/3240646746 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fce7c06ade0 0x7fce7c06d2a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fce900cd7e0 tx=0x7fce80009210 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:40.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.246+0000 7fce8cff9700 1 -- 172.21.15.189:0/3240646746 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fce9014af30 con 0x7fce78004d90 2024-04-06T13:45:40.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.440+0000 7fce96859700 1 -- 172.21.15.189:0/3240646746 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce900d3230 con 0x7fce78004d90 2024-04-06T13:45:40.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.441+0000 7fce8cff9700 1 -- 172.21.15.189:0/3240646746 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v208) v1 ==== 74+0+16429 (secure 0 0 0) 0x7fce900d3230 con 0x7fce78004d90 2024-04-06T13:45:40.442 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:40.443 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":208,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:45:40.071945+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:45:37.837008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-04-06T13:45:39.030714+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:45:40.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.444+0000 7fce96859700 1 -- 172.21.15.189:0/3240646746 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fce7c06ade0 msgr2=0x7fce7c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.444+0000 7fce96859700 1 --2- 172.21.15.189:0/3240646746 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fce7c06ade0 0x7fce7c06d2a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fce900cd7e0 tx=0x7fce80009210 comp rx=0 tx=0).stop 2024-04-06T13:45:40.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.444+0000 7fce96859700 1 -- 172.21.15.189:0/3240646746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce78004d90 msgr2=0x7fce90149af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.444+0000 7fce96859700 1 --2- 172.21.15.189:0/3240646746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce78004d90 0x7fce90149af0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fce8400f7e0 tx=0x7fce8400faf0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.444+0000 7fce96859700 1 -- 172.21.15.189:0/3240646746 shutdown_connections 2024-04-06T13:45:40.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.444+0000 7fce96859700 1 --2- 172.21.15.189:0/3240646746 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fce7c06ade0 0x7fce7c06d2a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.444+0000 7fce96859700 1 --2- 172.21.15.189:0/3240646746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce900c6da0 0x7fce900c7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.444+0000 7fce96859700 1 --2- 172.21.15.189:0/3240646746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce78004d90 0x7fce90149af0 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.444+0000 7fce96859700 1 --2- 172.21.15.189:0/3240646746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce900b4980 0x7fce900b4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.444+0000 7fce96859700 1 -- 172.21.15.189:0/3240646746 >> 172.21.15.189:0/3240646746 conn(0x7fce90072250 msgr2=0x7fce900ce110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:40.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.444+0000 7fce96859700 1 -- 172.21.15.189:0/3240646746 shutdown_connections 2024-04-06T13:45:40.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.444+0000 7fce96859700 1 -- 172.21.15.189:0/3240646746 wait complete. 2024-04-06T13:45:40.458 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:45:40.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.580+0000 7f832e393700 1 Processor -- start 2024-04-06T13:45:40.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.580+0000 7f832e393700 1 -- start start 2024-04-06T13:45:40.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.580+0000 7f832e393700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8328146dd0 0x7f832806d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:40.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.580+0000 7f832e393700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f832806c0a0 con 0x7f83280d17a0 2024-04-06T13:45:40.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.580+0000 7f832e393700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f832806c380 con 0x7f83280cedf0 2024-04-06T13:45:40.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.580+0000 7f832e393700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f832806beb0 con 0x7f8328146dd0 2024-04-06T13:45:40.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.581+0000 7f8327fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f83280d17a0 0x7f832806b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55896/0 (socket says 172.21.15.189:55896) 2024-04-06T13:45:40.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.581+0000 7f8327fff700 1 -- 172.21.15.189:0/3371489075 learned_addr learned my addr 172.21.15.189:0/3371489075 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:40.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.581+0000 7f83277fe700 1 --2- 172.21.15.189:0/3371489075 >> v2:172.21.15.192:3300/0 conn(0x7f8328146dd0 0x7f832806d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:40.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.581+0000 7f8326ffd700 1 -- 172.21.15.189:0/3371489075 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784281512 0 0) 0x7f832806c0a0 con 0x7f83280d17a0 2024-04-06T13:45:40.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.581+0000 7f8326ffd700 1 -- 172.21.15.189:0/3371489075 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83100035b0 con 0x7f83280d17a0 2024-04-06T13:45:40.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.581+0000 7f8326ffd700 1 -- 172.21.15.189:0/3371489075 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507954715 0 0) 0x7f832806c380 con 0x7f83280cedf0 2024-04-06T13:45:40.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.581+0000 7f8326ffd700 1 -- 172.21.15.189:0/3371489075 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f832806c0a0 con 0x7f83280cedf0 2024-04-06T13:45:40.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.581+0000 7f83277fe700 1 -- 172.21.15.189:0/3371489075 >> v1:172.21.15.192:6789/0 conn(0x7f83280cedf0 legacy=0x7f83280cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.581+0000 7f8326ffd700 1 -- 172.21.15.189:0/3371489075 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1913313728 0 0) 0x7f83100035b0 con 0x7f83280d17a0 2024-04-06T13:45:40.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.581+0000 7f83277fe700 1 -- 172.21.15.189:0/3371489075 >> v1:172.21.15.189:6789/0 conn(0x7f83280d17a0 legacy=0x7f832806b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.581+0000 7f83277fe700 1 -- 172.21.15.189:0/3371489075 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832806c660 con 0x7f8328146dd0 2024-04-06T13:45:40.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.581+0000 7f83277fe700 1 --2- 172.21.15.189:0/3371489075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8328146dd0 0x7f832806d140 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f831c009840 tx=0x7f831c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1168c13523264bd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:40.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.581+0000 7f8326ffd700 1 -- 172.21.15.189:0/3371489075 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8318001670 con 0x7f8328146dd0 2024-04-06T13:45:40.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.582+0000 7f8326ffd700 1 -- 172.21.15.189:0/3371489075 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831c004030 con 0x7f8328146dd0 2024-04-06T13:45:40.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.582+0000 7f8326ffd700 1 -- 172.21.15.189:0/3371489075 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c0043a0 con 0x7f8328146dd0 2024-04-06T13:45:40.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.582+0000 7f832e393700 1 -- 172.21.15.189:0/3371489075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8328146dd0 msgr2=0x7f832806d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.582+0000 7f832e393700 1 --2- 172.21.15.189:0/3371489075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8328146dd0 0x7f832806d140 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f831c009840 tx=0x7f831c009b50 comp rx=0 tx=0).stop 2024-04-06T13:45:40.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.582+0000 7f832e393700 1 -- 172.21.15.189:0/3371489075 shutdown_connections 2024-04-06T13:45:40.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.582+0000 7f832e393700 1 --2- 172.21.15.189:0/3371489075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8328146dd0 0x7f832806d140 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.582+0000 7f832e393700 1 -- 172.21.15.189:0/3371489075 >> 172.21.15.189:0/3371489075 conn(0x7f83280b2db0 msgr2=0x7f83280b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:40.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.582+0000 7f832e393700 1 -- 172.21.15.189:0/3371489075 shutdown_connections 2024-04-06T13:45:40.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.582+0000 7f832e393700 1 -- 172.21.15.189:0/3371489075 wait complete. 2024-04-06T13:45:40.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.583+0000 7f832e393700 1 Processor -- start 2024-04-06T13:45:40.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.583+0000 7f832e393700 1 -- start start 2024-04-06T13:45:40.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.584+0000 7f832e393700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83280cedf0 0x7f83281f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:40.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.584+0000 7f832e393700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83280d17a0 0x7f83280ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:40.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.584+0000 7f832e393700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8328146dd0 0x7f83281f2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:40.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.584+0000 7f832e393700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83280acb00 con 0x7f8328146dd0 2024-04-06T13:45:40.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.584+0000 7f832e393700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83281f30d0 con 0x7f83280d17a0 2024-04-06T13:45:40.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.584+0000 7f832e393700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83281f3350 con 0x7f83280cedf0 2024-04-06T13:45:40.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.584+0000 7f832c930700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8328146dd0 0x7f83281f2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:40.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.584+0000 7f832c930700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8328146dd0 0x7f83281f2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48110/0 (socket says 172.21.15.189:48110) 2024-04-06T13:45:40.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.584+0000 7f832c930700 1 -- 172.21.15.189:0/1032404927 learned_addr learned my addr 172.21.15.189:0/1032404927 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:40.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.584+0000 7f8327fff700 1 --2- 172.21.15.189:0/1032404927 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83280cedf0 0x7f83281f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:40.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.585+0000 7f83277fe700 1 --2- 172.21.15.189:0/1032404927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83280d17a0 0x7f83280ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:40.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.585+0000 7f832c930700 1 -- 172.21.15.189:0/1032404927 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83280cedf0 msgr2=0x7f83281f3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.585+0000 7f832c930700 1 --2- 172.21.15.189:0/1032404927 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83280cedf0 0x7f83281f3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.585+0000 7f832c930700 1 -- 172.21.15.189:0/1032404927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83280d17a0 msgr2=0x7f83280ae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.585+0000 7f832c930700 1 --2- 172.21.15.189:0/1032404927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83280d17a0 0x7f83280ae3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.585+0000 7f832c930700 1 -- 172.21.15.189:0/1032404927 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831c0094a0 con 0x7f8328146dd0 2024-04-06T13:45:40.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.585+0000 7f832c930700 1 --2- 172.21.15.189:0/1032404927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8328146dd0 0x7f83281f2b90 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f832000cbc0 tx=0x7f832000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:40.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.585+0000 7f8324ff9700 1 -- 172.21.15.189:0/1032404927 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8320003c00 con 0x7f8328146dd0 2024-04-06T13:45:40.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.585+0000 7f8324ff9700 1 -- 172.21.15.189:0/1032404927 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8320003d60 con 0x7f8328146dd0 2024-04-06T13:45:40.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.585+0000 7f8324ff9700 1 -- 172.21.15.189:0/1032404927 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832000abb0 con 0x7f8328146dd0 2024-04-06T13:45:40.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.585+0000 7f832e393700 1 -- 172.21.15.189:0/1032404927 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83281f3630 con 0x7f8328146dd0 2024-04-06T13:45:40.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.585+0000 7f832e393700 1 -- 172.21.15.189:0/1032404927 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8328158f70 con 0x7f8328146dd0 2024-04-06T13:45:40.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.586+0000 7f832e393700 1 -- 172.21.15.189:0/1032404927 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83281f3b50 con 0x7f8328146dd0 2024-04-06T13:45:40.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.586+0000 7f8324ff9700 1 -- 172.21.15.189:0/1032404927 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f832000ad10 con 0x7f8328146dd0 2024-04-06T13:45:40.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.587+0000 7f8324ff9700 1 --2- 172.21.15.189:0/1032404927 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f831406adb0 0x7f831406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:40.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.587+0000 7f8324ff9700 1 -- 172.21.15.189:0/1032404927 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f83200896c0 con 0x7f8328146dd0 2024-04-06T13:45:40.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.587+0000 7f8327fff700 1 --2- 172.21.15.189:0/1032404927 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f831406adb0 0x7f831406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:40.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.587+0000 7f8327fff700 1 --2- 172.21.15.189:0/1032404927 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f831406adb0 0x7f831406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f83280c7cc0 tx=0x7f8318001be0 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:40.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.591+0000 7f8324ff9700 1 -- 172.21.15.189:0/1032404927 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f83281f3b50 con 0x7f8328146dd0 2024-04-06T13:45:40.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.788+0000 7f832e393700 1 -- 172.21.15.189:0/1032404927 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83280ac8d0 con 0x7f8328146dd0 2024-04-06T13:45:40.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.789+0000 7f8324ff9700 1 -- 172.21.15.189:0/1032404927 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v208) v1 ==== 74+0+16429 (secure 0 0 0) 0x7f83280ac8d0 con 0x7f8328146dd0 2024-04-06T13:45:40.791 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:40.791 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":208,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:45:40.071945+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:45:37.837008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-04-06T13:45:39.030714+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:45:40.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.792+0000 7f832e393700 1 -- 172.21.15.189:0/1032404927 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f831406adb0 msgr2=0x7f831406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.792+0000 7f832e393700 1 --2- 172.21.15.189:0/1032404927 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f831406adb0 0x7f831406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f83280c7cc0 tx=0x7f8318001be0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.792+0000 7f832e393700 1 -- 172.21.15.189:0/1032404927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8328146dd0 msgr2=0x7f83281f2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.792+0000 7f832e393700 1 --2- 172.21.15.189:0/1032404927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8328146dd0 0x7f83281f2b90 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f832000cbc0 tx=0x7f832000cf80 comp rx=0 tx=0).stop 2024-04-06T13:45:40.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.792+0000 7f832e393700 1 -- 172.21.15.189:0/1032404927 shutdown_connections 2024-04-06T13:45:40.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.792+0000 7f832e393700 1 --2- 172.21.15.189:0/1032404927 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f831406adb0 0x7f831406d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.792+0000 7f832e393700 1 --2- 172.21.15.189:0/1032404927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8328146dd0 0x7f83281f2b90 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.792+0000 7f832e393700 1 --2- 172.21.15.189:0/1032404927 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83280cedf0 0x7f83281f3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.792+0000 7f832e393700 1 --2- 172.21.15.189:0/1032404927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83280d17a0 0x7f83280ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.792+0000 7f832e393700 1 -- 172.21.15.189:0/1032404927 >> 172.21.15.189:0/1032404927 conn(0x7f83280b2db0 msgr2=0x7f83280b11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:40.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.792+0000 7f832e393700 1 -- 172.21.15.189:0/1032404927 shutdown_connections 2024-04-06T13:45:40.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.792+0000 7f832e393700 1 -- 172.21.15.189:0/1032404927 wait complete. 2024-04-06T13:45:40.806 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-06T13:45:40.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.930+0000 7f69b0668700 1 Processor -- start 2024-04-06T13:45:40.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.930+0000 7f69b0668700 1 -- start start 2024-04-06T13:45:40.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.931+0000 7f69b0668700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f69a80cb870 0x7f69a815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:40.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.931+0000 7f69b0668700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f69a80cfa00 0x7f69a80cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:40.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.931+0000 7f69b0668700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a80cbd80 con 0x7f69a80d11b0 2024-04-06T13:45:40.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.931+0000 7f69b0668700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a80ccdc0 con 0x7f69a80cb870 2024-04-06T13:45:40.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.931+0000 7f69b0668700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a80d0b50 con 0x7f69a80cfa00 2024-04-06T13:45:40.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.931+0000 7f69adc03700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f69a80cb870 0x7f69a815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:40.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.931+0000 7f69aec05700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f69a80cfa00 0x7f69a80cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:40.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.931+0000 7f69adc03700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f69a80cb870 0x7f69a815c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48122/0 (socket says 172.21.15.189:48122) 2024-04-06T13:45:40.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.931+0000 7f69adc03700 1 -- 172.21.15.189:0/31783937 learned_addr learned my addr 172.21.15.189:0/31783937 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:40.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.931+0000 7f69adc03700 1 -- 172.21.15.189:0/31783937 >> v2:172.21.15.192:3300/0 conn(0x7f69a80cfa00 msgr2=0x7f69a80cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:40.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.931+0000 7f69adc03700 1 --2- 172.21.15.189:0/31783937 >> v2:172.21.15.192:3300/0 conn(0x7f69a80cfa00 0x7f69a80cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.931+0000 7f69adc03700 1 -- 172.21.15.189:0/31783937 >> v1:172.21.15.192:6789/0 conn(0x7f69a80d11b0 legacy=0x7f69a80d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.931+0000 7f69adc03700 1 -- 172.21.15.189:0/31783937 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a80cef20 con 0x7f69a80cb870 2024-04-06T13:45:40.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.931+0000 7f69adc03700 1 --2- 172.21.15.189:0/31783937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69a80cb870 0x7f69a815c280 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f69a815ff70 tx=0x7f69a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=612ad82482fed3f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:40.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.932+0000 7f69ad402700 1 -- 172.21.15.189:0/31783937 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a4011830 con 0x7f69a80cb870 2024-04-06T13:45:40.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.932+0000 7f69ad402700 1 -- 172.21.15.189:0/31783937 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a4011990 con 0x7f69a80cb870 2024-04-06T13:45:40.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.932+0000 7f69ad402700 1 -- 172.21.15.189:0/31783937 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a4011d20 con 0x7f69a80cb870 2024-04-06T13:45:40.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.932+0000 7f69b0668700 1 -- 172.21.15.189:0/31783937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69a80cb870 msgr2=0x7f69a815c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.932+0000 7f69b0668700 1 --2- 172.21.15.189:0/31783937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69a80cb870 0x7f69a815c280 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f69a815ff70 tx=0x7f69a4009bd0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.932+0000 7f69b0668700 1 -- 172.21.15.189:0/31783937 shutdown_connections 2024-04-06T13:45:40.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.932+0000 7f69b0668700 1 --2- 172.21.15.189:0/31783937 >> v2:172.21.15.192:3300/0 conn(0x7f69a80cfa00 0x7f69a80cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.932+0000 7f69b0668700 1 --2- 172.21.15.189:0/31783937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69a80cb870 0x7f69a815c280 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.932+0000 7f69b0668700 1 -- 172.21.15.189:0/31783937 >> 172.21.15.189:0/31783937 conn(0x7f69a80b2db0 msgr2=0x7f69a80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:40.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.932+0000 7f69b0668700 1 -- 172.21.15.189:0/31783937 shutdown_connections 2024-04-06T13:45:40.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.932+0000 7f69b0668700 1 -- 172.21.15.189:0/31783937 wait complete. 2024-04-06T13:45:40.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.934+0000 7f69b0668700 1 Processor -- start 2024-04-06T13:45:40.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.934+0000 7f69b0668700 1 -- start start 2024-04-06T13:45:40.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.934+0000 7f69b0668700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69a80cb870 0x7f69a81f3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:40.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.934+0000 7f69b0668700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69a80cfa00 0x7f69a8151040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:40.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.934+0000 7f69b0668700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69a80d11b0 0x7f69a815f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:40.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.934+0000 7f69b0668700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a815dbe0 con 0x7f69a80cfa00 2024-04-06T13:45:40.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.934+0000 7f69b0668700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a815dec0 con 0x7f69a80d11b0 2024-04-06T13:45:40.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.934+0000 7f69b0668700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a815e1a0 con 0x7f69a80cb870 2024-04-06T13:45:40.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.935+0000 7f69adc03700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69a80cfa00 0x7f69a8151040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:40.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.935+0000 7f69adc03700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69a80cfa00 0x7f69a8151040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48124/0 (socket says 172.21.15.189:48124) 2024-04-06T13:45:40.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.935+0000 7f69adc03700 1 -- 172.21.15.189:0/3974629537 learned_addr learned my addr 172.21.15.189:0/3974629537 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:40.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.935+0000 7f69adc03700 1 -- 172.21.15.189:0/3974629537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69a80cb870 msgr2=0x7f69a81f3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.935+0000 7f69adc03700 1 --2- 172.21.15.189:0/3974629537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69a80cb870 0x7f69a81f3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.935+0000 7f69adc03700 1 -- 172.21.15.189:0/3974629537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69a80d11b0 msgr2=0x7f69a815f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:40.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.935+0000 7f69adc03700 1 --2- 172.21.15.189:0/3974629537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69a80d11b0 0x7f69a815f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:40.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.935+0000 7f69adc03700 1 -- 172.21.15.189:0/3974629537 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a4009550 con 0x7f69a80cfa00 2024-04-06T13:45:40.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.935+0000 7f69adc03700 1 --2- 172.21.15.189:0/3974629537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69a80cfa00 0x7f69a8151040 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f69a4009c10 tx=0x7f69a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:40.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.935+0000 7f699effd700 1 -- 172.21.15.189:0/3974629537 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a40036a0 con 0x7f69a80cfa00 2024-04-06T13:45:40.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.935+0000 7f699effd700 1 -- 172.21.15.189:0/3974629537 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a401e070 con 0x7f69a80cfa00 2024-04-06T13:45:40.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.935+0000 7f69b0668700 1 -- 172.21.15.189:0/3974629537 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a815e480 con 0x7f69a80cfa00 2024-04-06T13:45:40.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.935+0000 7f699effd700 1 -- 172.21.15.189:0/3974629537 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a4003a40 con 0x7f69a80cfa00 2024-04-06T13:45:40.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.935+0000 7f69b0668700 1 -- 172.21.15.189:0/3974629537 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a8151580 con 0x7f69a80cfa00 2024-04-06T13:45:40.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.936+0000 7f69b0668700 1 -- 172.21.15.189:0/3974629537 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a815e9a0 con 0x7f69a80cfa00 2024-04-06T13:45:40.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.937+0000 7f699effd700 1 -- 172.21.15.189:0/3974629537 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f69a4003ba0 con 0x7f69a80cfa00 2024-04-06T13:45:40.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.937+0000 7f699effd700 1 --2- 172.21.15.189:0/3974629537 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f698c06adb0 0x7f698c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:40.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.937+0000 7f699effd700 1 -- 172.21.15.189:0/3974629537 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f69a408a660 con 0x7f69a80cfa00 2024-04-06T13:45:40.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.937+0000 7f69ae404700 1 --2- 172.21.15.189:0/3974629537 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f698c06adb0 0x7f698c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:40.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.938+0000 7f69ae404700 1 --2- 172.21.15.189:0/3974629537 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f698c06adb0 0x7f698c06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6998000f60 tx=0x7f6998007a40 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:40.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:40.942+0000 7f699effd700 1 -- 172.21.15.189:0/3974629537 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f69a815e9a0 con 0x7f69a80cfa00 2024-04-06T13:45:41.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:41.182+0000 7f69b0668700 1 -- 172.21.15.189:0/3974629537 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f69a815e9a0 con 0x7f69a80cfa00 2024-04-06T13:45:42.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.092+0000 7f699effd700 1 -- 172.21.15.189:0/3974629537 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v92) v1 ==== 118+0+0 (secure 0 0 0) 0x7f69a815e9a0 con 0x7f69a80cfa00 2024-04-06T13:45:42.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.095+0000 7f69b0668700 1 -- 172.21.15.189:0/3974629537 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f698c06adb0 msgr2=0x7f698c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:42.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.095+0000 7f69b0668700 1 --2- 172.21.15.189:0/3974629537 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f698c06adb0 0x7f698c06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6998000f60 tx=0x7f6998007a40 comp rx=0 tx=0).stop 2024-04-06T13:45:42.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.095+0000 7f69b0668700 1 -- 172.21.15.189:0/3974629537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69a80cfa00 msgr2=0x7f69a8151040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:42.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.095+0000 7f69b0668700 1 --2- 172.21.15.189:0/3974629537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69a80cfa00 0x7f69a8151040 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f69a4009c10 tx=0x7f69a4004750 comp rx=0 tx=0).stop 2024-04-06T13:45:42.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.095+0000 7f69b0668700 1 -- 172.21.15.189:0/3974629537 shutdown_connections 2024-04-06T13:45:42.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.095+0000 7f69b0668700 1 --2- 172.21.15.189:0/3974629537 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f698c06adb0 0x7f698c06d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:42.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.095+0000 7f69b0668700 1 --2- 172.21.15.189:0/3974629537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69a80d11b0 0x7f69a815f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:42.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.095+0000 7f69b0668700 1 --2- 172.21.15.189:0/3974629537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69a80cb870 0x7f69a81f3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:42.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.095+0000 7f69b0668700 1 --2- 172.21.15.189:0/3974629537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69a80cfa00 0x7f69a8151040 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:42.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.095+0000 7f69b0668700 1 -- 172.21.15.189:0/3974629537 >> 172.21.15.189:0/3974629537 conn(0x7f69a80b2db0 msgr2=0x7f69a806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:42.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.095+0000 7f69b0668700 1 -- 172.21.15.189:0/3974629537 shutdown_connections 2024-04-06T13:45:42.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.095+0000 7f69b0668700 1 -- 172.21.15.189:0/3974629537 wait complete. 2024-04-06T13:45:42.111 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-06T13:45:42.111 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-06T13:45:42.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.241+0000 7faa652aa700 1 Processor -- start 2024-04-06T13:45:42.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.241+0000 7faa652aa700 1 -- start start 2024-04-06T13:45:42.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.242+0000 7faa652aa700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7faa6006b8c0 0x7faa60168420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:42.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.242+0000 7faa652aa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6006bcd0 con 0x7faa6015d7c0 2024-04-06T13:45:42.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.242+0000 7faa652aa700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6006ceb0 con 0x7faa600cd3e0 2024-04-06T13:45:42.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.242+0000 7faa652aa700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa600cdfa0 con 0x7faa6006b8c0 2024-04-06T13:45:42.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.242+0000 7faa5e7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7faa6015d7c0 0x7faa6006b1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55910/0 (socket says 172.21.15.189:55910) 2024-04-06T13:45:42.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.242+0000 7faa5e7fc700 1 -- 172.21.15.189:0/2762070883 learned_addr learned my addr 172.21.15.189:0/2762070883 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:42.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5dffb700 1 -- 172.21.15.189:0/2762070883 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331827460 0 0) 0x7faa6006bcd0 con 0x7faa6015d7c0 2024-04-06T13:45:42.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5dffb700 1 -- 172.21.15.189:0/2762070883 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa480035b0 con 0x7faa6015d7c0 2024-04-06T13:45:42.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5f7fe700 1 --2- 172.21.15.189:0/2762070883 >> v2:172.21.15.192:3300/0 conn(0x7faa6006b8c0 0x7faa60168420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:42.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5dffb700 1 -- 172.21.15.189:0/2762070883 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 777653086 0 0) 0x7faa480035b0 con 0x7faa6015d7c0 2024-04-06T13:45:42.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5dffb700 1 -- 172.21.15.189:0/2762070883 >> v2:172.21.15.192:3300/0 conn(0x7faa6006b8c0 msgr2=0x7faa60168420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:42.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5dffb700 1 --2- 172.21.15.189:0/2762070883 >> v2:172.21.15.192:3300/0 conn(0x7faa6006b8c0 0x7faa60168420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:42.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5dffb700 1 -- 172.21.15.189:0/2762070883 >> v1:172.21.15.192:6789/0 conn(0x7faa600cd3e0 legacy=0x7faa600cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:42.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5dffb700 1 -- 172.21.15.189:0/2762070883 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6006d750 con 0x7faa6015d7c0 2024-04-06T13:45:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5dffb700 1 -- 172.21.15.189:0/2762070883 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7faa50001670 con 0x7faa6015d7c0 2024-04-06T13:45:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5dffb700 1 -- 172.21.15.189:0/2762070883 >> v1:172.21.15.189:6789/0 conn(0x7faa6015d7c0 legacy=0x7faa6006b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5dffb700 1 --2- 172.21.15.189:0/2762070883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa48003a00 0x7faa48003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5dffb700 1 --2- 172.21.15.189:0/2762070883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa48004850 0x7faa48004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5dffb700 1 --2- 172.21.15.189:0/2762070883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa48005330 0x7faa4800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5dffb700 1 -- 172.21.15.189:0/2762070883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa600cdfa0 con 0x7faa48003a00 2024-04-06T13:45:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5dffb700 1 -- 172.21.15.189:0/2762070883 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4800c200 con 0x7faa48005330 2024-04-06T13:45:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5dffb700 1 -- 172.21.15.189:0/2762070883 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa4800c3c0 con 0x7faa48004850 2024-04-06T13:45:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5f7fe700 1 --2- 172.21.15.189:0/2762070883 >> v2:172.21.15.192:3300/0 conn(0x7faa6006b8c0 0x7faa60168420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5effd700 1 --2- 172.21.15.189:0/2762070883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa48003a00 0x7faa48003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5e7fc700 1 --2- 172.21.15.189:0/2762070883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa48004850 0x7faa48004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5f7fe700 1 --2- 172.21.15.189:0/2762070883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa48005330 0x7faa4800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5effd700 1 -- 172.21.15.189:0/2762070883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa48004850 msgr2=0x7faa48004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5effd700 1 --2- 172.21.15.189:0/2762070883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa48004850 0x7faa48004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5effd700 1 -- 172.21.15.189:0/2762070883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa48005330 msgr2=0x7faa4800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5effd700 1 --2- 172.21.15.189:0/2762070883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa48005330 0x7faa4800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5f7fe700 1 -- 172.21.15.189:0/2762070883 reap_dead start 2024-04-06T13:45:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5effd700 1 -- 172.21.15.189:0/2762070883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa480035b0 con 0x7faa48003a00 2024-04-06T13:45:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.243+0000 7faa5e7fc700 1 --2- 172.21.15.189:0/2762070883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa48004850 0x7faa48004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.244+0000 7faa5effd700 1 --2- 172.21.15.189:0/2762070883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa48003a00 0x7faa48003e20 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7faa54007fd0 tx=0x7faa48005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e0ee05aad2ef71 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.244+0000 7faa5dffb700 1 -- 172.21.15.189:0/2762070883 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa540052d0 con 0x7faa48003a00 2024-04-06T13:45:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.244+0000 7faa5dffb700 1 -- 172.21.15.189:0/2762070883 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa54005450 con 0x7faa48003a00 2024-04-06T13:45:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.244+0000 7faa5dffb700 1 -- 172.21.15.189:0/2762070883 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa540038c0 con 0x7faa48003a00 2024-04-06T13:45:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.244+0000 7faa652aa700 1 -- 172.21.15.189:0/2762070883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa48003a00 msgr2=0x7faa48003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.244+0000 7faa652aa700 1 --2- 172.21.15.189:0/2762070883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa48003a00 0x7faa48003e20 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7faa54007fd0 tx=0x7faa48005880 comp rx=0 tx=0).stop 2024-04-06T13:45:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.244+0000 7faa652aa700 1 -- 172.21.15.189:0/2762070883 shutdown_connections 2024-04-06T13:45:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.244+0000 7faa652aa700 1 --2- 172.21.15.189:0/2762070883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa48003a00 0x7faa48003e20 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:42.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.244+0000 7faa652aa700 1 -- 172.21.15.189:0/2762070883 >> 172.21.15.189:0/2762070883 conn(0x7faa600cc290 msgr2=0x7faa600cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:42.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.244+0000 7faa652aa700 1 -- 172.21.15.189:0/2762070883 shutdown_connections 2024-04-06T13:45:42.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.244+0000 7faa652aa700 1 -- 172.21.15.189:0/2762070883 wait complete. 2024-04-06T13:45:42.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.246+0000 7faa652aa700 1 Processor -- start 2024-04-06T13:45:42.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.246+0000 7faa652aa700 1 -- start start 2024-04-06T13:45:42.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.246+0000 7faa652aa700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa48003a00 0x7faa600c5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:42.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.246+0000 7faa652aa700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa60146bb0 0x7faa6015dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:42.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.246+0000 7faa652aa700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa6015e130 0x7faa600c3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:42.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.246+0000 7faa652aa700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6015e5b0 con 0x7faa60146bb0 2024-04-06T13:45:42.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.246+0000 7faa652aa700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa601601f0 con 0x7faa6015e130 2024-04-06T13:45:42.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.246+0000 7faa652aa700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa6015f8c0 con 0x7faa48003a00 2024-04-06T13:45:42.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.246+0000 7faa5e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa60146bb0 0x7faa6015dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:42.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.247+0000 7faa5e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa60146bb0 0x7faa6015dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48142/0 (socket says 172.21.15.189:48142) 2024-04-06T13:45:42.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.247+0000 7faa5e7fc700 1 -- 172.21.15.189:0/2736321873 learned_addr learned my addr 172.21.15.189:0/2736321873 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:42.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.247+0000 7faa5effd700 1 --2- 172.21.15.189:0/2736321873 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa48003a00 0x7faa600c5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:42.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.247+0000 7faa5f7fe700 1 --2- 172.21.15.189:0/2736321873 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa6015e130 0x7faa600c3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:42.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.247+0000 7faa5e7fc700 1 -- 172.21.15.189:0/2736321873 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa48003a00 msgr2=0x7faa600c5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:42.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.247+0000 7faa5e7fc700 1 --2- 172.21.15.189:0/2736321873 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa48003a00 0x7faa600c5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:42.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.247+0000 7faa5e7fc700 1 -- 172.21.15.189:0/2736321873 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa6015e130 msgr2=0x7faa600c3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:42.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.247+0000 7faa5e7fc700 1 --2- 172.21.15.189:0/2736321873 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa6015e130 0x7faa600c3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:42.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.247+0000 7faa5e7fc700 1 -- 172.21.15.189:0/2736321873 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6015fba0 con 0x7faa60146bb0 2024-04-06T13:45:42.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.247+0000 7faa5effd700 1 --2- 172.21.15.189:0/2736321873 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa48003a00 0x7faa600c5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:45:42.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.247+0000 7faa5e7fc700 1 --2- 172.21.15.189:0/2736321873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa60146bb0 0x7faa6015dbf0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7faa5000e7b0 tx=0x7faa5000eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:42.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.247+0000 7faa477fe700 1 -- 172.21.15.189:0/2736321873 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa50012590 con 0x7faa60146bb0 2024-04-06T13:45:42.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.247+0000 7faa477fe700 1 -- 172.21.15.189:0/2736321873 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa500126f0 con 0x7faa60146bb0 2024-04-06T13:45:42.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.247+0000 7faa652aa700 1 -- 172.21.15.189:0/2736321873 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa6015fe80 con 0x7faa60146bb0 2024-04-06T13:45:42.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.247+0000 7faa477fe700 1 -- 172.21.15.189:0/2736321873 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa50012a80 con 0x7faa60146bb0 2024-04-06T13:45:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.247+0000 7faa652aa700 1 -- 172.21.15.189:0/2736321873 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa600b46a0 con 0x7faa60146bb0 2024-04-06T13:45:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.248+0000 7faa652aa700 1 -- 172.21.15.189:0/2736321873 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa60160560 con 0x7faa60146bb0 2024-04-06T13:45:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.249+0000 7faa477fe700 1 -- 172.21.15.189:0/2736321873 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7faa50012be0 con 0x7faa60146bb0 2024-04-06T13:45:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.249+0000 7faa477fe700 1 --2- 172.21.15.189:0/2736321873 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faa4c06adb0 0x7faa4c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.249+0000 7faa477fe700 1 -- 172.21.15.189:0/2736321873 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7faa50089330 con 0x7faa60146bb0 2024-04-06T13:45:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.249+0000 7faa5effd700 1 --2- 172.21.15.189:0/2736321873 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faa4c06adb0 0x7faa4c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.249+0000 7faa5effd700 1 --2- 172.21.15.189:0/2736321873 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faa4c06adb0 0x7faa4c06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faa54001eb0 tx=0x7faa54009030 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:42.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.253+0000 7faa477fe700 1 -- 172.21.15.189:0/2736321873 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7faa50054300 con 0x7faa60146bb0 2024-04-06T13:45:42.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:42.492+0000 7faa652aa700 1 -- 172.21.15.189:0/2736321873 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7faa60074990 con 0x7faa60146bb0 2024-04-06T13:45:43.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.102+0000 7faa477fe700 1 -- 172.21.15.189:0/2736321873 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v93) v1 ==== 122+0+0 (secure 0 0 0) 0x7faa50057920 con 0x7faa60146bb0 2024-04-06T13:45:43.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.106+0000 7faa652aa700 1 -- 172.21.15.189:0/2736321873 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faa4c06adb0 msgr2=0x7faa4c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:43.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.106+0000 7faa652aa700 1 --2- 172.21.15.189:0/2736321873 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faa4c06adb0 0x7faa4c06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faa54001eb0 tx=0x7faa54009030 comp rx=0 tx=0).stop 2024-04-06T13:45:43.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.106+0000 7faa652aa700 1 -- 172.21.15.189:0/2736321873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa60146bb0 msgr2=0x7faa6015dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:43.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.106+0000 7faa652aa700 1 --2- 172.21.15.189:0/2736321873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa60146bb0 0x7faa6015dbf0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7faa5000e7b0 tx=0x7faa5000eb70 comp rx=0 tx=0).stop 2024-04-06T13:45:43.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.106+0000 7faa652aa700 1 -- 172.21.15.189:0/2736321873 shutdown_connections 2024-04-06T13:45:43.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.106+0000 7faa652aa700 1 --2- 172.21.15.189:0/2736321873 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faa4c06adb0 0x7faa4c06d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.106+0000 7faa652aa700 1 --2- 172.21.15.189:0/2736321873 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa6015e130 0x7faa600c3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.106+0000 7faa652aa700 1 --2- 172.21.15.189:0/2736321873 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa48003a00 0x7faa600c5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.106+0000 7faa652aa700 1 --2- 172.21.15.189:0/2736321873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa60146bb0 0x7faa6015dbf0 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.106+0000 7faa652aa700 1 -- 172.21.15.189:0/2736321873 >> 172.21.15.189:0/2736321873 conn(0x7faa600cc290 msgr2=0x7faa600b0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:43.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.106+0000 7faa652aa700 1 -- 172.21.15.189:0/2736321873 shutdown_connections 2024-04-06T13:45:43.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.106+0000 7faa652aa700 1 -- 172.21.15.189:0/2736321873 wait complete. 2024-04-06T13:45:43.120 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-06T13:45:43.121 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:45:43.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.245+0000 7f71636be700 1 Processor -- start 2024-04-06T13:45:43.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.246+0000 7f71636be700 1 -- start start 2024-04-06T13:45:43.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.246+0000 7f71636be700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f715c0b4660 0x7f715c0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.246+0000 7f71636be700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f715c0b2fc0 0x7f715c0b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.246+0000 7f71636be700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c0b5030 con 0x7f715c15a550 2024-04-06T13:45:43.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.246+0000 7f71636be700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f715c0cdbe0 con 0x7f715c0b4660 2024-04-06T13:45:43.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.246+0000 7f71636be700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f715c15a940 con 0x7f715c0b2fc0 2024-04-06T13:45:43.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.247+0000 7f7160c59700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f715c0b4660 0x7f715c0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:43.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.247+0000 7f7160c59700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f715c0b4660 0x7f715c0b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48156/0 (socket says 172.21.15.189:48156) 2024-04-06T13:45:43.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.247+0000 7f7160c59700 1 -- 172.21.15.189:0/3086603246 learned_addr learned my addr 172.21.15.189:0/3086603246 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:43.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.247+0000 7f7161c5b700 1 --2- 172.21.15.189:0/3086603246 >> v2:172.21.15.192:3300/0 conn(0x7f715c0b2fc0 0x7f715c0b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:43.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.247+0000 7f7160c59700 1 -- 172.21.15.189:0/3086603246 >> v2:172.21.15.192:3300/0 conn(0x7f715c0b2fc0 msgr2=0x7f715c0b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:43.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.247+0000 7f7160c59700 1 --2- 172.21.15.189:0/3086603246 >> v2:172.21.15.192:3300/0 conn(0x7f715c0b2fc0 0x7f715c0b33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.247+0000 7f7160c59700 1 -- 172.21.15.189:0/3086603246 >> v1:172.21.15.192:6789/0 conn(0x7f715c15a550 legacy=0x7f715c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:43.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.247+0000 7f7160c59700 1 -- 172.21.15.189:0/3086603246 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715c15ac00 con 0x7f715c0b4660 2024-04-06T13:45:43.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.247+0000 7f7161c5b700 1 --2- 172.21.15.189:0/3086603246 >> v2:172.21.15.192:3300/0 conn(0x7f715c0b2fc0 0x7f715c0b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:45:43.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.247+0000 7f7160c59700 1 --2- 172.21.15.189:0/3086603246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f715c0b4660 0x7f715c0b4af0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f715c072830 tx=0x7f7158009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75052345f48ac483 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:43.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.247+0000 7f714ffff700 1 -- 172.21.15.189:0/3086603246 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7158004230 con 0x7f715c0b4660 2024-04-06T13:45:43.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.247+0000 7f714ffff700 1 -- 172.21.15.189:0/3086603246 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7158004390 con 0x7f715c0b4660 2024-04-06T13:45:43.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.247+0000 7f714ffff700 1 -- 172.21.15.189:0/3086603246 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71580036a0 con 0x7f715c0b4660 2024-04-06T13:45:43.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.248+0000 7f71636be700 1 -- 172.21.15.189:0/3086603246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f715c0b4660 msgr2=0x7f715c0b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:43.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.248+0000 7f71636be700 1 --2- 172.21.15.189:0/3086603246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f715c0b4660 0x7f715c0b4af0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f715c072830 tx=0x7f7158009bd0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.248+0000 7f71636be700 1 -- 172.21.15.189:0/3086603246 shutdown_connections 2024-04-06T13:45:43.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.248+0000 7f71636be700 1 --2- 172.21.15.189:0/3086603246 >> v2:172.21.15.192:3300/0 conn(0x7f715c0b2fc0 0x7f715c0b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.248+0000 7f71636be700 1 --2- 172.21.15.189:0/3086603246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f715c0b4660 0x7f715c0b4af0 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.248+0000 7f71636be700 1 -- 172.21.15.189:0/3086603246 >> 172.21.15.189:0/3086603246 conn(0x7f715c0d00b0 msgr2=0x7f715c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:43.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.248+0000 7f71636be700 1 -- 172.21.15.189:0/3086603246 shutdown_connections 2024-04-06T13:45:43.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.248+0000 7f71636be700 1 -- 172.21.15.189:0/3086603246 wait complete. 2024-04-06T13:45:43.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.249+0000 7f71636be700 1 Processor -- start 2024-04-06T13:45:43.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.249+0000 7f71636be700 1 -- start start 2024-04-06T13:45:43.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.250+0000 7f71636be700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f715c0b2fc0 0x7f715c1ee8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.250+0000 7f71636be700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f715c0b4660 0x7f715c1f1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.250+0000 7f71636be700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f715c156030 0x7f715c0a8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.250+0000 7f71636be700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c156880 con 0x7f715c0b2fc0 2024-04-06T13:45:43.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.250+0000 7f71636be700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c1f1c70 con 0x7f715c0b4660 2024-04-06T13:45:43.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.250+0000 7f71636be700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715c0afdd0 con 0x7f715c156030 2024-04-06T13:45:43.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.250+0000 7f716145a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f715c0b2fc0 0x7f715c1ee8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:43.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.250+0000 7f716145a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f715c0b2fc0 0x7f715c1ee8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48158/0 (socket says 172.21.15.189:48158) 2024-04-06T13:45:43.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.250+0000 7f716145a700 1 -- 172.21.15.189:0/3491834585 learned_addr learned my addr 172.21.15.189:0/3491834585 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:43.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.251+0000 7f716145a700 1 -- 172.21.15.189:0/3491834585 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f715c156030 msgr2=0x7f715c0a8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:43.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.251+0000 7f716145a700 1 --2- 172.21.15.189:0/3491834585 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f715c156030 0x7f715c0a8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.251+0000 7f716145a700 1 -- 172.21.15.189:0/3491834585 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f715c0b4660 msgr2=0x7f715c1f1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:43.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.251+0000 7f716145a700 1 --2- 172.21.15.189:0/3491834585 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f715c0b4660 0x7f715c1f1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.251+0000 7f716145a700 1 -- 172.21.15.189:0/3491834585 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7158009550 con 0x7f715c0b2fc0 2024-04-06T13:45:43.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.251+0000 7f716145a700 1 --2- 172.21.15.189:0/3491834585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f715c0b2fc0 0x7f715c1ee8e0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f715000c500 tx=0x7f715000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:43.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.251+0000 7f714dffb700 1 -- 172.21.15.189:0/3491834585 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7150011790 con 0x7f715c0b2fc0 2024-04-06T13:45:43.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.251+0000 7f714dffb700 1 -- 172.21.15.189:0/3491834585 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7150011930 con 0x7f715c0b2fc0 2024-04-06T13:45:43.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.251+0000 7f71636be700 1 -- 172.21.15.189:0/3491834585 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f715c1565a0 con 0x7f715c0b2fc0 2024-04-06T13:45:43.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.251+0000 7f714dffb700 1 -- 172.21.15.189:0/3491834585 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7150011ce0 con 0x7f715c0b2fc0 2024-04-06T13:45:43.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.251+0000 7f71636be700 1 -- 172.21.15.189:0/3491834585 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f715c0b0350 con 0x7f715c0b2fc0 2024-04-06T13:45:43.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.252+0000 7f71636be700 1 -- 172.21.15.189:0/3491834585 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f715c0afc30 con 0x7f715c0b2fc0 2024-04-06T13:45:43.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.252+0000 7f714dffb700 1 -- 172.21.15.189:0/3491834585 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f715001c460 con 0x7f715c0b2fc0 2024-04-06T13:45:43.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.253+0000 7f714dffb700 1 --2- 172.21.15.189:0/3491834585 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f714006ade0 0x7f714006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.253+0000 7f7160c59700 1 --2- 172.21.15.189:0/3491834585 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f714006ade0 0x7f714006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:43.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.253+0000 7f714dffb700 1 -- 172.21.15.189:0/3491834585 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7150089460 con 0x7f715c0b2fc0 2024-04-06T13:45:43.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.253+0000 7f7160c59700 1 --2- 172.21.15.189:0/3491834585 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f714006ade0 0x7f714006d2a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7158000c00 tx=0x7f7158010000 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:43.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.258+0000 7f714dffb700 1 -- 172.21.15.189:0/3491834585 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f715c0afc30 con 0x7f715c0b2fc0 2024-04-06T13:45:43.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.497+0000 7f71636be700 1 -- 172.21.15.189:0/3491834585 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f715c0afc30 con 0x7f715c0b2fc0 2024-04-06T13:45:43.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.498+0000 7f714dffb700 1 -- 172.21.15.189:0/3491834585 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+4750 (secure 0 0 0) 0x7f715c0afc30 con 0x7f715c0b2fc0 2024-04-06T13:45:43.499 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:43.499 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":93,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8525,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3302262361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3302262361},{"type":"v1","addr":"172.21.15.189:6837","nonce":3302262361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":8534,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/229523550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":229523550},{"type":"v1","addr":"172.21.15.192:6834","nonce":229523550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":9739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2560949338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2560949338},{"type":"v1","addr":"172.21.15.192:6835","nonce":2560949338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:45:40.072274+0000","modified":"2024-04-06T13:45:43.092032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":0,"incarnation":89,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/3619731181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3619731181},{"type":"v1","addr":"172.21.15.189:6836","nonce":3619731181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-04-06T13:45:43.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.501+0000 7f71636be700 1 -- 172.21.15.189:0/3491834585 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f714006ade0 msgr2=0x7f714006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:43.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.501+0000 7f71636be700 1 --2- 172.21.15.189:0/3491834585 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f714006ade0 0x7f714006d2a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7158000c00 tx=0x7f7158010000 comp rx=0 tx=0).stop 2024-04-06T13:45:43.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.501+0000 7f71636be700 1 -- 172.21.15.189:0/3491834585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f715c0b2fc0 msgr2=0x7f715c1ee8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:43.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.501+0000 7f71636be700 1 --2- 172.21.15.189:0/3491834585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f715c0b2fc0 0x7f715c1ee8e0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f715000c500 tx=0x7f715000c8c0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.501+0000 7f71636be700 1 -- 172.21.15.189:0/3491834585 shutdown_connections 2024-04-06T13:45:43.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.501+0000 7f71636be700 1 --2- 172.21.15.189:0/3491834585 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f714006ade0 0x7f714006d2a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.501+0000 7f71636be700 1 --2- 172.21.15.189:0/3491834585 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f715c156030 0x7f715c0a8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.501+0000 7f71636be700 1 --2- 172.21.15.189:0/3491834585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f715c0b2fc0 0x7f715c1ee8e0 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.501+0000 7f71636be700 1 --2- 172.21.15.189:0/3491834585 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f715c0b4660 0x7f715c1f1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.501+0000 7f71636be700 1 -- 172.21.15.189:0/3491834585 >> 172.21.15.189:0/3491834585 conn(0x7f715c0d00b0 msgr2=0x7f715c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:43.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.501+0000 7f71636be700 1 -- 172.21.15.189:0/3491834585 shutdown_connections 2024-04-06T13:45:43.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.501+0000 7f71636be700 1 -- 172.21.15.189:0/3491834585 wait complete. 2024-04-06T13:45:43.502 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 93 2024-04-06T13:45:43.513 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:45:43.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeb116d700 1 Processor -- start 2024-04-06T13:45:43.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeb116d700 1 -- start start 2024-04-06T13:45:43.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeb116d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdeac0d17a0 0x7fdeac06b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeb116d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdeac146dd0 0x7fdeac06d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeb116d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdeac0cedf0 0x7fdeac0cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeb116d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeac0cf760 con 0x7fdeac146dd0 2024-04-06T13:45:43.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeb116d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeac06c0a0 con 0x7fdeac0cedf0 2024-04-06T13:45:43.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeb116d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac06c380 con 0x7fdeac0d17a0 2024-04-06T13:45:43.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeaa59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdeac146dd0 0x7fdeac06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:43.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeaa59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdeac146dd0 0x7fdeac06d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48160/0 (socket says 172.21.15.189:48160) 2024-04-06T13:45:43.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeaa59c700 1 -- 172.21.15.189:0/4204526299 learned_addr learned my addr 172.21.15.189:0/4204526299 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:43.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeaad9d700 1 --2- 172.21.15.189:0/4204526299 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdeac0d17a0 0x7fdeac06b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:43.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeab59e700 1 --2- 172.21.15.189:0/4204526299 >> v2:172.21.15.192:3300/0 conn(0x7fdeac0cedf0 0x7fdeac0cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:43.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeaa59c700 1 -- 172.21.15.189:0/4204526299 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdeac0d17a0 msgr2=0x7fdeac06b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:43.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeaa59c700 1 --2- 172.21.15.189:0/4204526299 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdeac0d17a0 0x7fdeac06b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeaa59c700 1 -- 172.21.15.189:0/4204526299 >> v2:172.21.15.192:3300/0 conn(0x7fdeac0cedf0 msgr2=0x7fdeac0cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:43.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeaa59c700 1 --2- 172.21.15.189:0/4204526299 >> v2:172.21.15.192:3300/0 conn(0x7fdeac0cedf0 0x7fdeac0cf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeaa59c700 1 -- 172.21.15.189:0/4204526299 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeac06c660 con 0x7fdeac146dd0 2024-04-06T13:45:43.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeab59e700 1 --2- 172.21.15.189:0/4204526299 >> v2:172.21.15.192:3300/0 conn(0x7fdeac0cedf0 0x7fdeac0cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:45:43.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdeaa59c700 1 --2- 172.21.15.189:0/4204526299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdeac146dd0 0x7fdeac06d140 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7fdeac162120 tx=0x7fdea0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2a9d59a6d694726 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:43.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdea9d9b700 1 -- 172.21.15.189:0/4204526299 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea000b870 con 0x7fdeac146dd0 2024-04-06T13:45:43.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdea9d9b700 1 -- 172.21.15.189:0/4204526299 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea000b9d0 con 0x7fdeac146dd0 2024-04-06T13:45:43.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.635+0000 7fdea9d9b700 1 -- 172.21.15.189:0/4204526299 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea000bd60 con 0x7fdeac146dd0 2024-04-06T13:45:43.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.636+0000 7fdeb116d700 1 -- 172.21.15.189:0/4204526299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdeac146dd0 msgr2=0x7fdeac06d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:43.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.636+0000 7fdeb116d700 1 --2- 172.21.15.189:0/4204526299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdeac146dd0 0x7fdeac06d140 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7fdeac162120 tx=0x7fdea0009bd0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.636+0000 7fdeb116d700 1 -- 172.21.15.189:0/4204526299 shutdown_connections 2024-04-06T13:45:43.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.636+0000 7fdeb116d700 1 --2- 172.21.15.189:0/4204526299 >> v2:172.21.15.192:3300/0 conn(0x7fdeac0cedf0 0x7fdeac0cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.636+0000 7fdeb116d700 1 --2- 172.21.15.189:0/4204526299 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdeac0d17a0 0x7fdeac06b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.636+0000 7fdeb116d700 1 --2- 172.21.15.189:0/4204526299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdeac146dd0 0x7fdeac06d140 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.636+0000 7fdeb116d700 1 -- 172.21.15.189:0/4204526299 >> 172.21.15.189:0/4204526299 conn(0x7fdeac0b2db0 msgr2=0x7fdeac0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:43.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.636+0000 7fdeb116d700 1 -- 172.21.15.189:0/4204526299 shutdown_connections 2024-04-06T13:45:43.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.636+0000 7fdeb116d700 1 -- 172.21.15.189:0/4204526299 wait complete. 2024-04-06T13:45:43.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeb116d700 1 Processor -- start 2024-04-06T13:45:43.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeb116d700 1 -- start start 2024-04-06T13:45:43.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeb116d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdeac0cedf0 0x7fdeac0c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeb116d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdeac0d17a0 0x7fdeac158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeb116d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdeac146dd0 0x7fdeac0c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeb116d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac0c66b0 con 0x7fdeac0cedf0 2024-04-06T13:45:43.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeb116d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac0c5da0 con 0x7fdeac0d17a0 2024-04-06T13:45:43.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeb116d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac0b5220 con 0x7fdeac146dd0 2024-04-06T13:45:43.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeaad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdeac0cedf0 0x7fdeac0c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:43.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeaad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdeac0cedf0 0x7fdeac0c77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48172/0 (socket says 172.21.15.189:48172) 2024-04-06T13:45:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeaad9d700 1 -- 172.21.15.189:0/3744568221 learned_addr learned my addr 172.21.15.189:0/3744568221 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeaa59c700 1 --2- 172.21.15.189:0/3744568221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdeac0d17a0 0x7fdeac158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeab59e700 1 --2- 172.21.15.189:0/3744568221 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdeac146dd0 0x7fdeac0c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeaad9d700 1 -- 172.21.15.189:0/3744568221 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdeac146dd0 msgr2=0x7fdeac0c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeaad9d700 1 --2- 172.21.15.189:0/3744568221 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdeac146dd0 0x7fdeac0c6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeaad9d700 1 -- 172.21.15.189:0/3744568221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdeac0d17a0 msgr2=0x7fdeac158720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeaad9d700 1 --2- 172.21.15.189:0/3744568221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdeac0d17a0 0x7fdeac158720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeaad9d700 1 -- 172.21.15.189:0/3744568221 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea0009550 con 0x7fdeac0cedf0 2024-04-06T13:45:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeaad9d700 1 --2- 172.21.15.189:0/3744568221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdeac0cedf0 0x7fdeac0c77e0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fde9c00d800 tx=0x7fde9c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fde937fe700 1 -- 172.21.15.189:0/3744568221 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c0125f0 con 0x7fdeac0cedf0 2024-04-06T13:45:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fde937fe700 1 -- 172.21.15.189:0/3744568221 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde9c012750 con 0x7fdeac0cedf0 2024-04-06T13:45:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fde937fe700 1 -- 172.21.15.189:0/3744568221 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c012b00 con 0x7fdeac0cedf0 2024-04-06T13:45:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.639+0000 7fdeb116d700 1 -- 172.21.15.189:0/3744568221 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeac0c5ff0 con 0x7fdeac0cedf0 2024-04-06T13:45:43.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.640+0000 7fdeb116d700 1 -- 172.21.15.189:0/3744568221 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeac158c60 con 0x7fdeac0cedf0 2024-04-06T13:45:43.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.640+0000 7fdeb116d700 1 -- 172.21.15.189:0/3744568221 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeac06cbe0 con 0x7fdeac0cedf0 2024-04-06T13:45:43.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.641+0000 7fde937fe700 1 -- 172.21.15.189:0/3744568221 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fde9c012c60 con 0x7fdeac0cedf0 2024-04-06T13:45:43.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.641+0000 7fde937fe700 1 --2- 172.21.15.189:0/3744568221 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fde9806adb0 0x7fde9806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.641+0000 7fde937fe700 1 -- 172.21.15.189:0/3744568221 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fde9c089680 con 0x7fdeac0cedf0 2024-04-06T13:45:43.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.641+0000 7fdeaa59c700 1 --2- 172.21.15.189:0/3744568221 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fde9806adb0 0x7fde9806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:43.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.642+0000 7fdeaa59c700 1 --2- 172.21.15.189:0/3744568221 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fde9806adb0 0x7fde9806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdea0000c00 tx=0x7fdea0008030 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:43.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.646+0000 7fde937fe700 1 -- 172.21.15.189:0/3744568221 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fde9c0545b0 con 0x7fdeac0cedf0 2024-04-06T13:45:43.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.842+0000 7fdeb116d700 1 -- 172.21.15.189:0/3744568221 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdeac074990 con 0x7fdeac0cedf0 2024-04-06T13:45:43.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.843+0000 7fde937fe700 1 -- 172.21.15.189:0/3744568221 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) v1 ==== 74+0+16416 (secure 0 0 0) 0x7fde9c057bd0 con 0x7fdeac0cedf0 2024-04-06T13:45:43.844 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:43.844 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":209,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:45:41.074380+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:45:37.837008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-04-06T13:45:39.030714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:45:43.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.846+0000 7fdeb116d700 1 -- 172.21.15.189:0/3744568221 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fde9806adb0 msgr2=0x7fde9806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:43.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.846+0000 7fdeb116d700 1 --2- 172.21.15.189:0/3744568221 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fde9806adb0 0x7fde9806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdea0000c00 tx=0x7fdea0008030 comp rx=0 tx=0).stop 2024-04-06T13:45:43.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.846+0000 7fdeb116d700 1 -- 172.21.15.189:0/3744568221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdeac0cedf0 msgr2=0x7fdeac0c77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:43.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.846+0000 7fdeb116d700 1 --2- 172.21.15.189:0/3744568221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdeac0cedf0 0x7fdeac0c77e0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fde9c00d800 tx=0x7fde9c00db10 comp rx=0 tx=0).stop 2024-04-06T13:45:43.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.846+0000 7fdeb116d700 1 -- 172.21.15.189:0/3744568221 shutdown_connections 2024-04-06T13:45:43.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.846+0000 7fdeb116d700 1 --2- 172.21.15.189:0/3744568221 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fde9806adb0 0x7fde9806d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.846+0000 7fdeb116d700 1 --2- 172.21.15.189:0/3744568221 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdeac146dd0 0x7fdeac0c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.846+0000 7fdeb116d700 1 --2- 172.21.15.189:0/3744568221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdeac0cedf0 0x7fdeac0c77e0 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.846+0000 7fdeb116d700 1 --2- 172.21.15.189:0/3744568221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdeac0d17a0 0x7fdeac158720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.846+0000 7fdeb116d700 1 -- 172.21.15.189:0/3744568221 >> 172.21.15.189:0/3744568221 conn(0x7fdeac0b2db0 msgr2=0x7fdeac15e840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:43.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.846+0000 7fdeb116d700 1 -- 172.21.15.189:0/3744568221 shutdown_connections 2024-04-06T13:45:43.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.846+0000 7fdeb116d700 1 -- 172.21.15.189:0/3744568221 wait complete. 2024-04-06T13:45:43.859 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-06T13:45:43.859 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:45:43.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.981+0000 7f6a10624700 1 Processor -- start 2024-04-06T13:45:43.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.981+0000 7f6a10624700 1 -- start start 2024-04-06T13:45:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a10624700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a08158fe0 0x7f6a0806a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a10624700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6a0806ac70 0x7f6a0815bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a10624700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6a0815c100 0x7f6a080717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a10624700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a0806b0f0 con 0x7f6a0815c100 2024-04-06T13:45:43.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a10624700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a0815f9e0 con 0x7f6a0806ac70 2024-04-06T13:45:43.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a10624700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0815f0d0 con 0x7f6a08158fe0 2024-04-06T13:45:43.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a0ebc1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6a0815c100 0x7f6a080717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:43.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a0ebc1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6a0815c100 0x7f6a080717b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48178/0 (socket says 172.21.15.189:48178) 2024-04-06T13:45:43.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a0ebc1700 1 -- 172.21.15.189:0/1670128005 learned_addr learned my addr 172.21.15.189:0/1670128005 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:43.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a0dbbf700 1 --2- 172.21.15.189:0/1670128005 >> v2:172.21.15.192:3300/0 conn(0x7f6a0806ac70 0x7f6a0815bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:43.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a0e3c0700 1 --2- 172.21.15.189:0/1670128005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a08158fe0 0x7f6a0806a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:43.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a0ebc1700 1 -- 172.21.15.189:0/1670128005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a08158fe0 msgr2=0x7f6a0806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:43.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a0ebc1700 1 --2- 172.21.15.189:0/1670128005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a08158fe0 0x7f6a0806a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a0ebc1700 1 -- 172.21.15.189:0/1670128005 >> v2:172.21.15.192:3300/0 conn(0x7f6a0806ac70 msgr2=0x7f6a0815bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:43.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a0ebc1700 1 --2- 172.21.15.189:0/1670128005 >> v2:172.21.15.192:3300/0 conn(0x7f6a0806ac70 0x7f6a0815bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a0ebc1700 1 -- 172.21.15.189:0/1670128005 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0815c8f0 con 0x7f6a0815c100 2024-04-06T13:45:43.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a0ebc1700 1 --2- 172.21.15.189:0/1670128005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a0815c100 0x7f6a080717b0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f6a0806cb90 tx=0x7f6a0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a890991d3828d37a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:43.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a0d3be700 1 -- 172.21.15.189:0/1670128005 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0000e070 con 0x7f6a0815c100 2024-04-06T13:45:43.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a0d3be700 1 -- 172.21.15.189:0/1670128005 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0000b910 con 0x7f6a0815c100 2024-04-06T13:45:43.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a0d3be700 1 -- 172.21.15.189:0/1670128005 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0000bcc0 con 0x7f6a0815c100 2024-04-06T13:45:43.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a10624700 1 -- 172.21.15.189:0/1670128005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a0815c100 msgr2=0x7f6a080717b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:43.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a10624700 1 --2- 172.21.15.189:0/1670128005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a0815c100 0x7f6a080717b0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f6a0806cb90 tx=0x7f6a0000b330 comp rx=0 tx=0).stop 2024-04-06T13:45:43.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a10624700 1 -- 172.21.15.189:0/1670128005 shutdown_connections 2024-04-06T13:45:43.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a10624700 1 --2- 172.21.15.189:0/1670128005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a0815c100 0x7f6a080717b0 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a10624700 1 --2- 172.21.15.189:0/1670128005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a08158fe0 0x7f6a0806a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a10624700 1 --2- 172.21.15.189:0/1670128005 >> v2:172.21.15.192:3300/0 conn(0x7f6a0806ac70 0x7f6a0815bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a10624700 1 -- 172.21.15.189:0/1670128005 >> 172.21.15.189:0/1670128005 conn(0x7f6a080d00b0 msgr2=0x7f6a080d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:43.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a10624700 1 -- 172.21.15.189:0/1670128005 shutdown_connections 2024-04-06T13:45:43.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.982+0000 7f6a10624700 1 -- 172.21.15.189:0/1670128005 wait complete. 2024-04-06T13:45:43.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.984+0000 7f6a10624700 1 Processor -- start 2024-04-06T13:45:43.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.984+0000 7f6a10624700 1 -- start start 2024-04-06T13:45:43.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.985+0000 7f6a10624700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a0806ac70 0x7f6a0815aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.985+0000 7f6a10624700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a08158fe0 0x7f6a0813c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.985+0000 7f6a10624700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a0815c100 0x7f6a0813e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.985+0000 7f6a10624700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0815b020 con 0x7f6a08158fe0 2024-04-06T13:45:43.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.985+0000 7f6a10624700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0813c960 con 0x7f6a0806ac70 2024-04-06T13:45:43.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.985+0000 7f6a10624700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a08159a40 con 0x7f6a0815c100 2024-04-06T13:45:43.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.985+0000 7f6a0dbbf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a08158fe0 0x7f6a0813c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:43.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.985+0000 7f6a0dbbf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a08158fe0 0x7f6a0813c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48182/0 (socket says 172.21.15.189:48182) 2024-04-06T13:45:43.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.985+0000 7f6a0dbbf700 1 -- 172.21.15.189:0/2192312741 learned_addr learned my addr 172.21.15.189:0/2192312741 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:43.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.985+0000 7f6a0ebc1700 1 --2- 172.21.15.189:0/2192312741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a0815c100 0x7f6a0813e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:43.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.985+0000 7f6a0e3c0700 1 --2- 172.21.15.189:0/2192312741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a0806ac70 0x7f6a0815aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:43.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.985+0000 7f6a0dbbf700 1 -- 172.21.15.189:0/2192312741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a0815c100 msgr2=0x7f6a0813e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:43.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.985+0000 7f6a0dbbf700 1 --2- 172.21.15.189:0/2192312741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a0815c100 0x7f6a0813e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.985+0000 7f6a0dbbf700 1 -- 172.21.15.189:0/2192312741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a0806ac70 msgr2=0x7f6a0815aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:43.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.985+0000 7f6a0dbbf700 1 --2- 172.21.15.189:0/2192312741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a0806ac70 0x7f6a0815aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:43.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.985+0000 7f6a0dbbf700 1 -- 172.21.15.189:0/2192312741 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a00009b30 con 0x7f6a08158fe0 2024-04-06T13:45:43.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.985+0000 7f6a0dbbf700 1 --2- 172.21.15.189:0/2192312741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a08158fe0 0x7f6a0813c420 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f69f800e930 tx=0x7f69f800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:43.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.986+0000 7f69feffd700 1 -- 172.21.15.189:0/2192312741 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f8013680 con 0x7f6a08158fe0 2024-04-06T13:45:43.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.986+0000 7f69feffd700 1 -- 172.21.15.189:0/2192312741 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f80137e0 con 0x7f6a08158fe0 2024-04-06T13:45:43.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.986+0000 7f6a10624700 1 -- 172.21.15.189:0/2192312741 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a08159d20 con 0x7f6a08158fe0 2024-04-06T13:45:43.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.986+0000 7f69feffd700 1 -- 172.21.15.189:0/2192312741 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f8013b70 con 0x7f6a08158fe0 2024-04-06T13:45:43.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.986+0000 7f6a10624700 1 -- 172.21.15.189:0/2192312741 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a0815a2e0 con 0x7f6a08158fe0 2024-04-06T13:45:43.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.987+0000 7f6a10624700 1 -- 172.21.15.189:0/2192312741 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a080d1050 con 0x7f6a08158fe0 2024-04-06T13:45:43.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.987+0000 7f69feffd700 1 -- 172.21.15.189:0/2192312741 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f69f8013cd0 con 0x7f6a08158fe0 2024-04-06T13:45:43.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.988+0000 7f69feffd700 1 --2- 172.21.15.189:0/2192312741 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f69ec06adb0 0x7f69ec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:43.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.988+0000 7f6a0e3c0700 1 --2- 172.21.15.189:0/2192312741 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f69ec06adb0 0x7f69ec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:43.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.988+0000 7f69feffd700 1 -- 172.21.15.189:0/2192312741 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f69f8089650 con 0x7f6a08158fe0 2024-04-06T13:45:43.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.988+0000 7f6a0e3c0700 1 --2- 172.21.15.189:0/2192312741 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f69ec06adb0 0x7f69ec06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6a04009be0 tx=0x7f6a04009210 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:43.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:43.992+0000 7f69feffd700 1 -- 172.21.15.189:0/2192312741 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f69f80544f0 con 0x7f6a08158fe0 2024-04-06T13:45:44.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.186+0000 7f6a10624700 1 -- 172.21.15.189:0/2192312741 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6a0815f670 con 0x7f69ec06adb0 2024-04-06T13:45:44.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.194+0000 7f69feffd700 1 -- 172.21.15.189:0/2192312741 <== mgr.10494 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290384 (secure 0 0 0) 0x7f6a04011860 con 0x7f69ec06adb0 2024-04-06T13:45:44.200 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:44.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.198+0000 7f6a10624700 1 -- 172.21.15.189:0/2192312741 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f69ec06adb0 msgr2=0x7f69ec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:44.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.198+0000 7f6a10624700 1 --2- 172.21.15.189:0/2192312741 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f69ec06adb0 0x7f69ec06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6a04009be0 tx=0x7f6a04009210 comp rx=0 tx=0).stop 2024-04-06T13:45:44.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.198+0000 7f6a10624700 1 -- 172.21.15.189:0/2192312741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a08158fe0 msgr2=0x7f6a0813c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:44.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.198+0000 7f6a10624700 1 --2- 172.21.15.189:0/2192312741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a08158fe0 0x7f6a0813c420 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f69f800e930 tx=0x7f69f800ec40 comp rx=0 tx=0).stop 2024-04-06T13:45:44.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.198+0000 7f6a10624700 1 -- 172.21.15.189:0/2192312741 shutdown_connections 2024-04-06T13:45:44.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.198+0000 7f6a10624700 1 --2- 172.21.15.189:0/2192312741 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f69ec06adb0 0x7f69ec06d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:44.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.198+0000 7f6a10624700 1 --2- 172.21.15.189:0/2192312741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6a0815c100 0x7f6a0813e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:44.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.198+0000 7f6a10624700 1 --2- 172.21.15.189:0/2192312741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a0806ac70 0x7f6a0815aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:44.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.198+0000 7f6a10624700 1 --2- 172.21.15.189:0/2192312741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a08158fe0 0x7f6a0813c420 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:44.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.198+0000 7f6a10624700 1 -- 172.21.15.189:0/2192312741 >> 172.21.15.189:0/2192312741 conn(0x7f6a080d00b0 msgr2=0x7f6a08144910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:44.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.198+0000 7f6a10624700 1 -- 172.21.15.189:0/2192312741 shutdown_connections 2024-04-06T13:45:44.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.198+0000 7f6a10624700 1 -- 172.21.15.189:0/2192312741 wait complete. 2024-04-06T13:45:44.203 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:45:44.211 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-06T13:45:42.478528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":423968,"kb_used_data":21660,"kb_used_omap":12,"kb_used_meta":402163,"kb_avail":754550752,"statfs":{"total":773094113280,"available":772659970048,"internally_reserved":0,"allocated":22179840,"data_stored":11834173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":411815480},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000649"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775504+0000","last_change":"2024-04-06T13:45:40.089525+0000","last_active":"2024-04-06T13:45:41.775504+0000","last_peered":"2024-04-06T13:45:41.775504+0000","last_clean":"2024-04-06T13:45:41.775504+0000","last_became_active":"2024-04-06T13:45:40.089241+0000","last_became_peered":"2024-04-06T13:45:40.089241+0000","last_unstale":"2024-04-06T13:45:41.775504+0000","last_undegraded":"2024-04-06T13:45:41.775504+0000","last_fullsized":"2024-04-06T13:45:41.775504+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:43:35.880636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775498+0000","last_change":"2024-04-06T13:45:40.100325+0000","last_active":"2024-04-06T13:45:41.775498+0000","last_peered":"2024-04-06T13:45:41.775498+0000","last_clean":"2024-04-06T13:45:41.775498+0000","last_became_active":"2024-04-06T13:45:40.100060+0000","last_became_peered":"2024-04-06T13:45:40.100060+0000","last_unstale":"2024-04-06T13:45:41.775498+0000","last_undegraded":"2024-04-06T13:45:41.775498+0000","last_fullsized":"2024-04-06T13:45:41.775498+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:06:05.815120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774972+0000","last_change":"2024-04-06T13:45:40.094182+0000","last_active":"2024-04-06T13:45:41.774972+0000","last_peered":"2024-04-06T13:45:41.774972+0000","last_clean":"2024-04-06T13:45:41.774972+0000","last_became_active":"2024-04-06T13:45:40.093003+0000","last_became_peered":"2024-04-06T13:45:40.093003+0000","last_unstale":"2024-04-06T13:45:41.774972+0000","last_undegraded":"2024-04-06T13:45:41.774972+0000","last_fullsized":"2024-04-06T13:45:41.774972+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:41:55.909045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775160+0000","last_change":"2024-04-06T13:45:40.091827+0000","last_active":"2024-04-06T13:45:41.775160+0000","last_peered":"2024-04-06T13:45:41.775160+0000","last_clean":"2024-04-06T13:45:41.775160+0000","last_became_active":"2024-04-06T13:45:40.091564+0000","last_became_peered":"2024-04-06T13:45:40.091564+0000","last_unstale":"2024-04-06T13:45:41.775160+0000","last_undegraded":"2024-04-06T13:45:41.775160+0000","last_fullsized":"2024-04-06T13:45:41.775160+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:24:57.492435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085481+0000","last_change":"2024-04-06T13:45:40.102544+0000","last_active":"2024-04-06T13:45:41.085481+0000","last_peered":"2024-04-06T13:45:41.085481+0000","last_clean":"2024-04-06T13:45:41.085481+0000","last_became_active":"2024-04-06T13:45:40.102295+0000","last_became_peered":"2024-04-06T13:45:40.102295+0000","last_unstale":"2024-04-06T13:45:41.085481+0000","last_undegraded":"2024-04-06T13:45:41.085481+0000","last_fullsized":"2024-04-06T13:45:41.085481+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:16:21.249070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085498+0000","last_change":"2024-04-06T13:45:40.105204+0000","last_active":"2024-04-06T13:45:41.085498+0000","last_peered":"2024-04-06T13:45:41.085498+0000","last_clean":"2024-04-06T13:45:41.085498+0000","last_became_active":"2024-04-06T13:45:40.104960+0000","last_became_peered":"2024-04-06T13:45:40.104960+0000","last_unstale":"2024-04-06T13:45:41.085498+0000","last_undegraded":"2024-04-06T13:45:41.085498+0000","last_fullsized":"2024-04-06T13:45:41.085498+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:10:28.482639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774357+0000","last_change":"2024-04-06T13:45:40.094696+0000","last_active":"2024-04-06T13:45:41.774357+0000","last_peered":"2024-04-06T13:45:41.774357+0000","last_clean":"2024-04-06T13:45:41.774357+0000","last_became_active":"2024-04-06T13:45:40.093318+0000","last_became_peered":"2024-04-06T13:45:40.093318+0000","last_unstale":"2024-04-06T13:45:41.774357+0000","last_undegraded":"2024-04-06T13:45:41.774357+0000","last_fullsized":"2024-04-06T13:45:41.774357+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:42:02.447546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.084671+0000","last_change":"2024-04-06T13:45:40.091160+0000","last_active":"2024-04-06T13:45:41.084671+0000","last_peered":"2024-04-06T13:45:41.084671+0000","last_clean":"2024-04-06T13:45:41.084671+0000","last_became_active":"2024-04-06T13:45:40.090702+0000","last_became_peered":"2024-04-06T13:45:40.090702+0000","last_unstale":"2024-04-06T13:45:41.084671+0000","last_undegraded":"2024-04-06T13:45:41.084671+0000","last_fullsized":"2024-04-06T13:45:41.084671+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:26:02.582066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775556+0000","last_change":"2024-04-06T13:45:40.100626+0000","last_active":"2024-04-06T13:45:41.775556+0000","last_peered":"2024-04-06T13:45:41.775556+0000","last_clean":"2024-04-06T13:45:41.775556+0000","last_became_active":"2024-04-06T13:45:40.096229+0000","last_became_peered":"2024-04-06T13:45:40.096229+0000","last_unstale":"2024-04-06T13:45:41.775556+0000","last_undegraded":"2024-04-06T13:45:41.775556+0000","last_fullsized":"2024-04-06T13:45:41.775556+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:17:31.461140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085931+0000","last_change":"2024-04-06T13:45:40.102825+0000","last_active":"2024-04-06T13:45:41.085931+0000","last_peered":"2024-04-06T13:45:41.085931+0000","last_clean":"2024-04-06T13:45:41.085931+0000","last_became_active":"2024-04-06T13:45:40.102315+0000","last_became_peered":"2024-04-06T13:45:40.102315+0000","last_unstale":"2024-04-06T13:45:41.085931+0000","last_undegraded":"2024-04-06T13:45:41.085931+0000","last_fullsized":"2024-04-06T13:45:41.085931+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:30:54.881116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775100+0000","last_change":"2024-04-06T13:45:40.096421+0000","last_active":"2024-04-06T13:45:41.775100+0000","last_peered":"2024-04-06T13:45:41.775100+0000","last_clean":"2024-04-06T13:45:41.775100+0000","last_became_active":"2024-04-06T13:45:40.096061+0000","last_became_peered":"2024-04-06T13:45:40.096061+0000","last_unstale":"2024-04-06T13:45:41.775100+0000","last_undegraded":"2024-04-06T13:45:41.775100+0000","last_fullsized":"2024-04-06T13:45:41.775100+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:42:18.183369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775216+0000","last_change":"2024-04-06T13:45:40.088645+0000","last_active":"2024-04-06T13:45:41.775216+0000","last_peered":"2024-04-06T13:45:41.775216+0000","last_clean":"2024-04-06T13:45:41.775216+0000","last_became_active":"2024-04-06T13:45:40.088410+0000","last_became_peered":"2024-04-06T13:45:40.088410+0000","last_unstale":"2024-04-06T13:45:41.775216+0000","last_undegraded":"2024-04-06T13:45:41.775216+0000","last_fullsized":"2024-04-06T13:45:41.775216+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:05:37.331539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086466+0000","last_change":"2024-04-06T13:45:40.083080+0000","last_active":"2024-04-06T13:45:41.086466+0000","last_peered":"2024-04-06T13:45:41.086466+0000","last_clean":"2024-04-06T13:45:41.086466+0000","last_became_active":"2024-04-06T13:45:40.082696+0000","last_became_peered":"2024-04-06T13:45:40.082696+0000","last_unstale":"2024-04-06T13:45:41.086466+0000","last_undegraded":"2024-04-06T13:45:41.086466+0000","last_fullsized":"2024-04-06T13:45:41.086466+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:49:57.256838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774711+0000","last_change":"2024-04-06T13:45:40.094634+0000","last_active":"2024-04-06T13:45:41.774711+0000","last_peered":"2024-04-06T13:45:41.774711+0000","last_clean":"2024-04-06T13:45:41.774711+0000","last_became_active":"2024-04-06T13:45:40.093724+0000","last_became_peered":"2024-04-06T13:45:40.093724+0000","last_unstale":"2024-04-06T13:45:41.774711+0000","last_undegraded":"2024-04-06T13:45:41.774711+0000","last_fullsized":"2024-04-06T13:45:41.774711+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:25:25.061230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775300+0000","last_change":"2024-04-06T13:45:40.089229+0000","last_active":"2024-04-06T13:45:41.775300+0000","last_peered":"2024-04-06T13:45:41.775300+0000","last_clean":"2024-04-06T13:45:41.775300+0000","last_became_active":"2024-04-06T13:45:40.087921+0000","last_became_peered":"2024-04-06T13:45:40.087921+0000","last_unstale":"2024-04-06T13:45:41.775300+0000","last_undegraded":"2024-04-06T13:45:41.775300+0000","last_fullsized":"2024-04-06T13:45:41.775300+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:51:32.416220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085426+0000","last_change":"2024-04-06T13:45:40.103933+0000","last_active":"2024-04-06T13:45:41.085426+0000","last_peered":"2024-04-06T13:45:41.085426+0000","last_clean":"2024-04-06T13:45:41.085426+0000","last_became_active":"2024-04-06T13:45:40.103694+0000","last_became_peered":"2024-04-06T13:45:40.103694+0000","last_unstale":"2024-04-06T13:45:41.085426+0000","last_undegraded":"2024-04-06T13:45:41.085426+0000","last_fullsized":"2024-04-06T13:45:41.085426+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:25.834350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085952+0000","last_change":"2024-04-06T13:45:40.099989+0000","last_active":"2024-04-06T13:45:41.085952+0000","last_peered":"2024-04-06T13:45:41.085952+0000","last_clean":"2024-04-06T13:45:41.085952+0000","last_became_active":"2024-04-06T13:45:40.099736+0000","last_became_peered":"2024-04-06T13:45:40.099736+0000","last_unstale":"2024-04-06T13:45:41.085952+0000","last_undegraded":"2024-04-06T13:45:41.085952+0000","last_fullsized":"2024-04-06T13:45:41.085952+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:58:36.474241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086796+0000","last_change":"2024-04-06T13:45:40.087268+0000","last_active":"2024-04-06T13:45:41.086796+0000","last_peered":"2024-04-06T13:45:41.086796+0000","last_clean":"2024-04-06T13:45:41.086796+0000","last_became_active":"2024-04-06T13:45:40.086959+0000","last_became_peered":"2024-04-06T13:45:40.086959+0000","last_unstale":"2024-04-06T13:45:41.086796+0000","last_undegraded":"2024-04-06T13:45:41.086796+0000","last_fullsized":"2024-04-06T13:45:41.086796+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:18:59.672479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086720+0000","last_change":"2024-04-06T13:45:40.092669+0000","last_active":"2024-04-06T13:45:41.086720+0000","last_peered":"2024-04-06T13:45:41.086720+0000","last_clean":"2024-04-06T13:45:41.086720+0000","last_became_active":"2024-04-06T13:45:40.092467+0000","last_became_peered":"2024-04-06T13:45:40.092467+0000","last_unstale":"2024-04-06T13:45:41.086720+0000","last_undegraded":"2024-04-06T13:45:41.086720+0000","last_fullsized":"2024-04-06T13:45:41.086720+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:25:14.399262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085462+0000","last_change":"2024-04-06T13:45:40.100609+0000","last_active":"2024-04-06T13:45:41.085462+0000","last_peered":"2024-04-06T13:45:41.085462+0000","last_clean":"2024-04-06T13:45:41.085462+0000","last_became_active":"2024-04-06T13:45:40.100358+0000","last_became_peered":"2024-04-06T13:45:40.100358+0000","last_unstale":"2024-04-06T13:45:41.085462+0000","last_undegraded":"2024-04-06T13:45:41.085462+0000","last_fullsized":"2024-04-06T13:45:41.085462+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:50:14.349390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775744+0000","last_change":"2024-04-06T13:45:40.087005+0000","last_active":"2024-04-06T13:45:41.775744+0000","last_peered":"2024-04-06T13:45:41.775744+0000","last_clean":"2024-04-06T13:45:41.775744+0000","last_became_active":"2024-04-06T13:45:40.086732+0000","last_became_peered":"2024-04-06T13:45:40.086732+0000","last_unstale":"2024-04-06T13:45:41.775744+0000","last_undegraded":"2024-04-06T13:45:41.775744+0000","last_fullsized":"2024-04-06T13:45:41.775744+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:38:59.616950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774728+0000","last_change":"2024-04-06T13:45:40.087356+0000","last_active":"2024-04-06T13:45:41.774728+0000","last_peered":"2024-04-06T13:45:41.774728+0000","last_clean":"2024-04-06T13:45:41.774728+0000","last_became_active":"2024-04-06T13:45:40.087103+0000","last_became_peered":"2024-04-06T13:45:40.087103+0000","last_unstale":"2024-04-06T13:45:41.774728+0000","last_undegraded":"2024-04-06T13:45:41.774728+0000","last_fullsized":"2024-04-06T13:45:41.774728+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:10:26.253958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.084551+0000","last_change":"2024-04-06T13:45:40.090129+0000","last_active":"2024-04-06T13:45:41.084551+0000","last_peered":"2024-04-06T13:45:41.084551+0000","last_clean":"2024-04-06T13:45:41.084551+0000","last_became_active":"2024-04-06T13:45:40.089740+0000","last_became_peered":"2024-04-06T13:45:40.089740+0000","last_unstale":"2024-04-06T13:45:41.084551+0000","last_undegraded":"2024-04-06T13:45:41.084551+0000","last_fullsized":"2024-04-06T13:45:41.084551+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:04:59.118082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774369+0000","last_change":"2024-04-06T13:45:40.089599+0000","last_active":"2024-04-06T13:45:41.774369+0000","last_peered":"2024-04-06T13:45:41.774369+0000","last_clean":"2024-04-06T13:45:41.774369+0000","last_became_active":"2024-04-06T13:45:40.089194+0000","last_became_peered":"2024-04-06T13:45:40.089194+0000","last_unstale":"2024-04-06T13:45:41.774369+0000","last_undegraded":"2024-04-06T13:45:41.774369+0000","last_fullsized":"2024-04-06T13:45:41.774369+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:01:22.608442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774821+0000","last_change":"2024-04-06T13:45:40.097563+0000","last_active":"2024-04-06T13:45:41.774821+0000","last_peered":"2024-04-06T13:45:41.774821+0000","last_clean":"2024-04-06T13:45:41.774821+0000","last_became_active":"2024-04-06T13:45:40.095657+0000","last_became_peered":"2024-04-06T13:45:40.095657+0000","last_unstale":"2024-04-06T13:45:41.774821+0000","last_undegraded":"2024-04-06T13:45:41.774821+0000","last_fullsized":"2024-04-06T13:45:41.774821+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:29:31.066605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775323+0000","last_change":"2024-04-06T13:45:40.100417+0000","last_active":"2024-04-06T13:45:41.775323+0000","last_peered":"2024-04-06T13:45:41.775323+0000","last_clean":"2024-04-06T13:45:41.775323+0000","last_became_active":"2024-04-06T13:45:40.100105+0000","last_became_peered":"2024-04-06T13:45:40.100105+0000","last_unstale":"2024-04-06T13:45:41.775323+0000","last_undegraded":"2024-04-06T13:45:41.775323+0000","last_fullsized":"2024-04-06T13:45:41.775323+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:55:17.346196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.084770+0000","last_change":"2024-04-06T13:45:40.089311+0000","last_active":"2024-04-06T13:45:41.084770+0000","last_peered":"2024-04-06T13:45:41.084770+0000","last_clean":"2024-04-06T13:45:41.084770+0000","last_became_active":"2024-04-06T13:45:40.088982+0000","last_became_peered":"2024-04-06T13:45:40.088982+0000","last_unstale":"2024-04-06T13:45:41.084770+0000","last_undegraded":"2024-04-06T13:45:41.084770+0000","last_fullsized":"2024-04-06T13:45:41.084770+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:23:00.349095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085572+0000","last_change":"2024-04-06T13:45:40.103916+0000","last_active":"2024-04-06T13:45:41.085572+0000","last_peered":"2024-04-06T13:45:41.085572+0000","last_clean":"2024-04-06T13:45:41.085572+0000","last_became_active":"2024-04-06T13:45:40.103733+0000","last_became_peered":"2024-04-06T13:45:40.103733+0000","last_unstale":"2024-04-06T13:45:41.085572+0000","last_undegraded":"2024-04-06T13:45:41.085572+0000","last_fullsized":"2024-04-06T13:45:41.085572+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:44.542777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774986+0000","last_change":"2024-04-06T13:45:40.101201+0000","last_active":"2024-04-06T13:45:41.774986+0000","last_peered":"2024-04-06T13:45:41.774986+0000","last_clean":"2024-04-06T13:45:41.774986+0000","last_became_active":"2024-04-06T13:45:40.100774+0000","last_became_peered":"2024-04-06T13:45:40.100774+0000","last_unstale":"2024-04-06T13:45:41.774986+0000","last_undegraded":"2024-04-06T13:45:41.774986+0000","last_fullsized":"2024-04-06T13:45:41.774986+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:20:45.306030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775602+0000","last_change":"2024-04-06T13:45:40.092867+0000","last_active":"2024-04-06T13:45:41.775602+0000","last_peered":"2024-04-06T13:45:41.775602+0000","last_clean":"2024-04-06T13:45:41.775602+0000","last_became_active":"2024-04-06T13:45:40.092514+0000","last_became_peered":"2024-04-06T13:45:40.092514+0000","last_unstale":"2024-04-06T13:45:41.775602+0000","last_undegraded":"2024-04-06T13:45:41.775602+0000","last_fullsized":"2024-04-06T13:45:41.775602+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:12.970828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774546+0000","last_change":"2024-04-06T13:45:40.093371+0000","last_active":"2024-04-06T13:45:41.774546+0000","last_peered":"2024-04-06T13:45:41.774546+0000","last_clean":"2024-04-06T13:45:41.774546+0000","last_became_active":"2024-04-06T13:45:40.093120+0000","last_became_peered":"2024-04-06T13:45:40.093120+0000","last_unstale":"2024-04-06T13:45:41.774546+0000","last_undegraded":"2024-04-06T13:45:41.774546+0000","last_fullsized":"2024-04-06T13:45:41.774546+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:05:09.968925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085510+0000","last_change":"2024-04-06T13:45:40.099053+0000","last_active":"2024-04-06T13:45:41.085510+0000","last_peered":"2024-04-06T13:45:41.085510+0000","last_clean":"2024-04-06T13:45:41.085510+0000","last_became_active":"2024-04-06T13:45:40.098844+0000","last_became_peered":"2024-04-06T13:45:40.098844+0000","last_unstale":"2024-04-06T13:45:41.085510+0000","last_undegraded":"2024-04-06T13:45:41.085510+0000","last_fullsized":"2024-04-06T13:45:41.085510+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:55:35.985540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.084929+0000","last_change":"2024-04-06T13:45:40.091890+0000","last_active":"2024-04-06T13:45:41.084929+0000","last_peered":"2024-04-06T13:45:41.084929+0000","last_clean":"2024-04-06T13:45:41.084929+0000","last_became_active":"2024-04-06T13:45:40.091562+0000","last_became_peered":"2024-04-06T13:45:40.091562+0000","last_unstale":"2024-04-06T13:45:41.084929+0000","last_undegraded":"2024-04-06T13:45:41.084929+0000","last_fullsized":"2024-04-06T13:45:41.084929+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:54:01.774660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"208'4","reported_seq":15,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774901+0000","last_change":"2024-04-06T13:45:39.672115+0000","last_active":"2024-04-06T13:45:41.774901+0000","last_peered":"2024-04-06T13:45:41.774901+0000","last_clean":"2024-04-06T13:45:41.774901+0000","last_became_active":"2024-04-06T13:45:39.671638+0000","last_became_peered":"2024-04-06T13:45:39.671638+0000","last_unstale":"2024-04-06T13:45:41.774901+0000","last_undegraded":"2024-04-06T13:45:41.774901+0000","last_fullsized":"2024-04-06T13:45:41.774901+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:50:03.038639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775304+0000","last_change":"2024-04-06T13:45:40.093142+0000","last_active":"2024-04-06T13:45:41.775304+0000","last_peered":"2024-04-06T13:45:41.775304+0000","last_clean":"2024-04-06T13:45:41.775304+0000","last_became_active":"2024-04-06T13:45:40.092478+0000","last_became_peered":"2024-04-06T13:45:40.092478+0000","last_unstale":"2024-04-06T13:45:41.775304+0000","last_undegraded":"2024-04-06T13:45:41.775304+0000","last_fullsized":"2024-04-06T13:45:41.775304+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:54:45.772130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"208'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086023+0000","last_change":"2024-04-06T13:45:39.680145+0000","last_active":"2024-04-06T13:45:41.086023+0000","last_peered":"2024-04-06T13:45:41.086023+0000","last_clean":"2024-04-06T13:45:41.086023+0000","last_became_active":"2024-04-06T13:45:39.679798+0000","last_became_peered":"2024-04-06T13:45:39.679798+0000","last_unstale":"2024-04-06T13:45:41.086023+0000","last_undegraded":"2024-04-06T13:45:41.086023+0000","last_fullsized":"2024-04-06T13:45:41.086023+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:58:18.377227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775361+0000","last_change":"2024-04-06T13:45:40.099397+0000","last_active":"2024-04-06T13:45:41.775361+0000","last_peered":"2024-04-06T13:45:41.775361+0000","last_clean":"2024-04-06T13:45:41.775361+0000","last_became_active":"2024-04-06T13:45:40.098838+0000","last_became_peered":"2024-04-06T13:45:40.098838+0000","last_unstale":"2024-04-06T13:45:41.775361+0000","last_undegraded":"2024-04-06T13:45:41.775361+0000","last_fullsized":"2024-04-06T13:45:41.775361+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:21:30.034786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"208'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775172+0000","last_change":"2024-04-06T13:45:39.666757+0000","last_active":"2024-04-06T13:45:41.775172+0000","last_peered":"2024-04-06T13:45:41.775172+0000","last_clean":"2024-04-06T13:45:41.775172+0000","last_became_active":"2024-04-06T13:45:39.666468+0000","last_became_peered":"2024-04-06T13:45:39.666468+0000","last_unstale":"2024-04-06T13:45:41.775172+0000","last_undegraded":"2024-04-06T13:45:41.775172+0000","last_fullsized":"2024-04-06T13:45:41.775172+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:16.205165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774820+0000","last_change":"2024-04-06T13:45:40.098735+0000","last_active":"2024-04-06T13:45:41.774820+0000","last_peered":"2024-04-06T13:45:41.774820+0000","last_clean":"2024-04-06T13:45:41.774820+0000","last_became_active":"2024-04-06T13:45:40.098454+0000","last_became_peered":"2024-04-06T13:45:40.098454+0000","last_unstale":"2024-04-06T13:45:41.774820+0000","last_undegraded":"2024-04-06T13:45:41.774820+0000","last_fullsized":"2024-04-06T13:45:41.774820+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:27:32.798856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"208'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086718+0000","last_change":"2024-04-06T13:45:39.665692+0000","last_active":"2024-04-06T13:45:41.086718+0000","last_peered":"2024-04-06T13:45:41.086718+0000","last_clean":"2024-04-06T13:45:41.086718+0000","last_became_active":"2024-04-06T13:45:39.665169+0000","last_became_peered":"2024-04-06T13:45:39.665169+0000","last_unstale":"2024-04-06T13:45:41.086718+0000","last_undegraded":"2024-04-06T13:45:41.086718+0000","last_fullsized":"2024-04-06T13:45:41.086718+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:21:29.030323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086312+0000","last_change":"2024-04-06T13:45:40.095571+0000","last_active":"2024-04-06T13:45:41.086312+0000","last_peered":"2024-04-06T13:45:41.086312+0000","last_clean":"2024-04-06T13:45:41.086312+0000","last_became_active":"2024-04-06T13:45:40.095280+0000","last_became_peered":"2024-04-06T13:45:40.095280+0000","last_unstale":"2024-04-06T13:45:41.086312+0000","last_undegraded":"2024-04-06T13:45:41.086312+0000","last_fullsized":"2024-04-06T13:45:41.086312+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:07:36.838483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"208'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085642+0000","last_change":"2024-04-06T13:45:39.679077+0000","last_active":"2024-04-06T13:45:41.085642+0000","last_peered":"2024-04-06T13:45:41.085642+0000","last_clean":"2024-04-06T13:45:41.085642+0000","last_became_active":"2024-04-06T13:45:39.678763+0000","last_became_peered":"2024-04-06T13:45:39.678763+0000","last_unstale":"2024-04-06T13:45:41.085642+0000","last_undegraded":"2024-04-06T13:45:41.085642+0000","last_fullsized":"2024-04-06T13:45:41.085642+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:23:07.225232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775395+0000","last_change":"2024-04-06T13:45:40.095694+0000","last_active":"2024-04-06T13:45:41.775395+0000","last_peered":"2024-04-06T13:45:41.775395+0000","last_clean":"2024-04-06T13:45:41.775395+0000","last_became_active":"2024-04-06T13:45:40.095463+0000","last_became_peered":"2024-04-06T13:45:40.095463+0000","last_unstale":"2024-04-06T13:45:41.775395+0000","last_undegraded":"2024-04-06T13:45:41.775395+0000","last_fullsized":"2024-04-06T13:45:41.775395+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:47:54.692873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"208'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774772+0000","last_change":"2024-04-06T13:45:39.681133+0000","last_active":"2024-04-06T13:45:41.774772+0000","last_peered":"2024-04-06T13:45:41.774772+0000","last_clean":"2024-04-06T13:45:41.774772+0000","last_became_active":"2024-04-06T13:45:39.680833+0000","last_became_peered":"2024-04-06T13:45:39.680833+0000","last_unstale":"2024-04-06T13:45:41.774772+0000","last_undegraded":"2024-04-06T13:45:41.774772+0000","last_fullsized":"2024-04-06T13:45:41.774772+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:22:59.344632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774822+0000","last_change":"2024-04-06T13:45:40.090838+0000","last_active":"2024-04-06T13:45:41.774822+0000","last_peered":"2024-04-06T13:45:41.774822+0000","last_clean":"2024-04-06T13:45:41.774822+0000","last_became_active":"2024-04-06T13:45:40.090616+0000","last_became_peered":"2024-04-06T13:45:40.090616+0000","last_unstale":"2024-04-06T13:45:41.774822+0000","last_undegraded":"2024-04-06T13:45:41.774822+0000","last_fullsized":"2024-04-06T13:45:41.774822+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:16.638956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086513+0000","last_change":"2024-04-06T13:45:39.661817+0000","last_active":"2024-04-06T13:45:41.086513+0000","last_peered":"2024-04-06T13:45:41.086513+0000","last_clean":"2024-04-06T13:45:41.086513+0000","last_became_active":"2024-04-06T13:45:39.661528+0000","last_became_peered":"2024-04-06T13:45:39.661528+0000","last_unstale":"2024-04-06T13:45:41.086513+0000","last_undegraded":"2024-04-06T13:45:41.086513+0000","last_fullsized":"2024-04-06T13:45:41.086513+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:23:08.710841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085030+0000","last_change":"2024-04-06T13:45:40.088375+0000","last_active":"2024-04-06T13:45:41.085030+0000","last_peered":"2024-04-06T13:45:41.085030+0000","last_clean":"2024-04-06T13:45:41.085030+0000","last_became_active":"2024-04-06T13:45:40.088114+0000","last_became_peered":"2024-04-06T13:45:40.088114+0000","last_unstale":"2024-04-06T13:45:41.085030+0000","last_undegraded":"2024-04-06T13:45:41.085030+0000","last_fullsized":"2024-04-06T13:45:41.085030+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:49:48.207025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086058+0000","last_change":"2024-04-06T13:45:39.680180+0000","last_active":"2024-04-06T13:45:41.086058+0000","last_peered":"2024-04-06T13:45:41.086058+0000","last_clean":"2024-04-06T13:45:41.086058+0000","last_became_active":"2024-04-06T13:45:39.679894+0000","last_became_peered":"2024-04-06T13:45:39.679894+0000","last_unstale":"2024-04-06T13:45:41.086058+0000","last_undegraded":"2024-04-06T13:45:41.086058+0000","last_fullsized":"2024-04-06T13:45:41.086058+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:05:36.327076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085766+0000","last_change":"2024-04-06T13:45:40.101167+0000","last_active":"2024-04-06T13:45:41.085766+0000","last_peered":"2024-04-06T13:45:41.085766+0000","last_clean":"2024-04-06T13:45:41.085766+0000","last_became_active":"2024-04-06T13:45:40.100847+0000","last_became_peered":"2024-04-06T13:45:40.100847+0000","last_unstale":"2024-04-06T13:45:41.085766+0000","last_undegraded":"2024-04-06T13:45:41.085766+0000","last_fullsized":"2024-04-06T13:45:41.085766+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:13:50.892939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775663+0000","last_change":"2024-04-06T13:45:39.681395+0000","last_active":"2024-04-06T13:45:41.775663+0000","last_peered":"2024-04-06T13:45:41.775663+0000","last_clean":"2024-04-06T13:45:41.775663+0000","last_became_active":"2024-04-06T13:45:39.681107+0000","last_became_peered":"2024-04-06T13:45:39.681107+0000","last_unstale":"2024-04-06T13:45:41.775663+0000","last_undegraded":"2024-04-06T13:45:41.775663+0000","last_fullsized":"2024-04-06T13:45:41.775663+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:02:47.403764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086852+0000","last_change":"2024-04-06T13:45:40.097418+0000","last_active":"2024-04-06T13:45:41.086852+0000","last_peered":"2024-04-06T13:45:41.086852+0000","last_clean":"2024-04-06T13:45:41.086852+0000","last_became_active":"2024-04-06T13:45:40.097229+0000","last_became_peered":"2024-04-06T13:45:40.097229+0000","last_unstale":"2024-04-06T13:45:41.086852+0000","last_undegraded":"2024-04-06T13:45:41.086852+0000","last_fullsized":"2024-04-06T13:45:41.086852+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:59:03.379200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"208'3","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774740+0000","last_change":"2024-04-06T13:45:39.681177+0000","last_active":"2024-04-06T13:45:41.774740+0000","last_peered":"2024-04-06T13:45:41.774740+0000","last_clean":"2024-04-06T13:45:41.774740+0000","last_became_active":"2024-04-06T13:45:39.680881+0000","last_became_peered":"2024-04-06T13:45:39.680881+0000","last_unstale":"2024-04-06T13:45:41.774740+0000","last_undegraded":"2024-04-06T13:45:41.774740+0000","last_fullsized":"2024-04-06T13:45:41.774740+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:21:12.767786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775610+0000","last_change":"2024-04-06T13:45:40.101751+0000","last_active":"2024-04-06T13:45:41.775610+0000","last_peered":"2024-04-06T13:45:41.775610+0000","last_clean":"2024-04-06T13:45:41.775610+0000","last_became_active":"2024-04-06T13:45:40.101491+0000","last_became_peered":"2024-04-06T13:45:40.101491+0000","last_unstale":"2024-04-06T13:45:41.775610+0000","last_undegraded":"2024-04-06T13:45:41.775610+0000","last_fullsized":"2024-04-06T13:45:41.775610+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:20:52.927579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"208'6","reported_seq":17,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086795+0000","last_change":"2024-04-06T13:45:39.667906+0000","last_active":"2024-04-06T13:45:41.086795+0000","last_peered":"2024-04-06T13:45:41.086795+0000","last_clean":"2024-04-06T13:45:41.086795+0000","last_became_active":"2024-04-06T13:45:39.667632+0000","last_became_peered":"2024-04-06T13:45:39.667632+0000","last_unstale":"2024-04-06T13:45:41.086795+0000","last_undegraded":"2024-04-06T13:45:41.086795+0000","last_fullsized":"2024-04-06T13:45:41.086795+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:05:01.859909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774233+0000","last_change":"2024-04-06T13:45:40.087438+0000","last_active":"2024-04-06T13:45:41.774233+0000","last_peered":"2024-04-06T13:45:41.774233+0000","last_clean":"2024-04-06T13:45:41.774233+0000","last_became_active":"2024-04-06T13:45:40.087173+0000","last_became_peered":"2024-04-06T13:45:40.087173+0000","last_unstale":"2024-04-06T13:45:41.774233+0000","last_undegraded":"2024-04-06T13:45:41.774233+0000","last_fullsized":"2024-04-06T13:45:41.774233+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:17.209628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"208'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086590+0000","last_change":"2024-04-06T13:45:39.665242+0000","last_active":"2024-04-06T13:45:41.086590+0000","last_peered":"2024-04-06T13:45:41.086590+0000","last_clean":"2024-04-06T13:45:41.086590+0000","last_became_active":"2024-04-06T13:45:39.664913+0000","last_became_peered":"2024-04-06T13:45:39.664913+0000","last_unstale":"2024-04-06T13:45:41.086590+0000","last_undegraded":"2024-04-06T13:45:41.086590+0000","last_fullsized":"2024-04-06T13:45:41.086590+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:02:56.300763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085383+0000","last_change":"2024-04-06T13:45:40.098012+0000","last_active":"2024-04-06T13:45:41.085383+0000","last_peered":"2024-04-06T13:45:41.085383+0000","last_clean":"2024-04-06T13:45:41.085383+0000","last_became_active":"2024-04-06T13:45:40.097768+0000","last_became_peered":"2024-04-06T13:45:40.097768+0000","last_unstale":"2024-04-06T13:45:41.085383+0000","last_undegraded":"2024-04-06T13:45:41.085383+0000","last_fullsized":"2024-04-06T13:45:41.085383+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:51:11.592246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775005+0000","last_change":"2024-04-06T13:45:39.675786+0000","last_active":"2024-04-06T13:45:41.775005+0000","last_peered":"2024-04-06T13:45:41.775005+0000","last_clean":"2024-04-06T13:45:41.775005+0000","last_became_active":"2024-04-06T13:45:39.675545+0000","last_became_peered":"2024-04-06T13:45:39.675545+0000","last_unstale":"2024-04-06T13:45:41.775005+0000","last_undegraded":"2024-04-06T13:45:41.775005+0000","last_fullsized":"2024-04-06T13:45:41.775005+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:55:16.341733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086561+0000","last_change":"2024-04-06T13:45:40.085958+0000","last_active":"2024-04-06T13:45:41.086561+0000","last_peered":"2024-04-06T13:45:41.086561+0000","last_clean":"2024-04-06T13:45:41.086561+0000","last_became_active":"2024-04-06T13:45:40.085677+0000","last_became_peered":"2024-04-06T13:45:40.085677+0000","last_unstale":"2024-04-06T13:45:41.086561+0000","last_undegraded":"2024-04-06T13:45:41.086561+0000","last_fullsized":"2024-04-06T13:45:41.086561+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:58:46.123989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774435+0000","last_change":"2024-04-06T13:45:39.667680+0000","last_active":"2024-04-06T13:45:41.774435+0000","last_peered":"2024-04-06T13:45:41.774435+0000","last_clean":"2024-04-06T13:45:41.774435+0000","last_became_active":"2024-04-06T13:45:39.666889+0000","last_became_peered":"2024-04-06T13:45:39.666889+0000","last_unstale":"2024-04-06T13:45:41.774435+0000","last_undegraded":"2024-04-06T13:45:41.774435+0000","last_fullsized":"2024-04-06T13:45:41.774435+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:01:09.108164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086543+0000","last_change":"2024-04-06T13:45:40.088259+0000","last_active":"2024-04-06T13:45:41.086543+0000","last_peered":"2024-04-06T13:45:41.086543+0000","last_clean":"2024-04-06T13:45:41.086543+0000","last_became_active":"2024-04-06T13:45:40.087981+0000","last_became_peered":"2024-04-06T13:45:40.087981+0000","last_unstale":"2024-04-06T13:45:41.086543+0000","last_undegraded":"2024-04-06T13:45:41.086543+0000","last_fullsized":"2024-04-06T13:45:41.086543+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:04:59.810704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"208'8","reported_seq":19,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774832+0000","last_change":"2024-04-06T13:45:39.671256+0000","last_active":"2024-04-06T13:45:41.774832+0000","last_peered":"2024-04-06T13:45:41.774832+0000","last_clean":"2024-04-06T13:45:41.774832+0000","last_became_active":"2024-04-06T13:45:39.670957+0000","last_became_peered":"2024-04-06T13:45:39.670957+0000","last_unstale":"2024-04-06T13:45:41.774832+0000","last_undegraded":"2024-04-06T13:45:41.774832+0000","last_fullsized":"2024-04-06T13:45:41.774832+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:05:08.964462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775059+0000","last_change":"2024-04-06T13:45:40.093107+0000","last_active":"2024-04-06T13:45:41.775059+0000","last_peered":"2024-04-06T13:45:41.775059+0000","last_clean":"2024-04-06T13:45:41.775059+0000","last_became_active":"2024-04-06T13:45:40.092815+0000","last_became_peered":"2024-04-06T13:45:40.092815+0000","last_unstale":"2024-04-06T13:45:41.775059+0000","last_undegraded":"2024-04-06T13:45:41.775059+0000","last_fullsized":"2024-04-06T13:45:41.775059+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:35:13.586257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"208'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775404+0000","last_change":"2024-04-06T13:45:39.669733+0000","last_active":"2024-04-06T13:45:41.775404+0000","last_peered":"2024-04-06T13:45:41.775404+0000","last_clean":"2024-04-06T13:45:41.775404+0000","last_became_active":"2024-04-06T13:45:39.669394+0000","last_became_peered":"2024-04-06T13:45:39.669394+0000","last_unstale":"2024-04-06T13:45:41.775404+0000","last_undegraded":"2024-04-06T13:45:41.775404+0000","last_fullsized":"2024-04-06T13:45:41.775404+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:36:00.946626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085797+0000","last_change":"2024-04-06T13:45:40.098243+0000","last_active":"2024-04-06T13:45:41.085797+0000","last_peered":"2024-04-06T13:45:41.085797+0000","last_clean":"2024-04-06T13:45:41.085797+0000","last_became_active":"2024-04-06T13:45:40.097989+0000","last_became_peered":"2024-04-06T13:45:40.097989+0000","last_unstale":"2024-04-06T13:45:41.085797+0000","last_undegraded":"2024-04-06T13:45:41.085797+0000","last_fullsized":"2024-04-06T13:45:41.085797+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:31:51.711889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"208'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775503+0000","last_change":"2024-04-06T13:45:39.674397+0000","last_active":"2024-04-06T13:45:41.775503+0000","last_peered":"2024-04-06T13:45:41.775503+0000","last_clean":"2024-04-06T13:45:41.775503+0000","last_became_active":"2024-04-06T13:45:39.674055+0000","last_became_peered":"2024-04-06T13:45:39.674055+0000","last_unstale":"2024-04-06T13:45:41.775503+0000","last_undegraded":"2024-04-06T13:45:41.775503+0000","last_fullsized":"2024-04-06T13:45:41.775503+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:04:58.113619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775432+0000","last_change":"2024-04-06T13:45:40.100362+0000","last_active":"2024-04-06T13:45:41.775432+0000","last_peered":"2024-04-06T13:45:41.775432+0000","last_clean":"2024-04-06T13:45:41.775432+0000","last_became_active":"2024-04-06T13:45:40.100081+0000","last_became_peered":"2024-04-06T13:45:40.100081+0000","last_unstale":"2024-04-06T13:45:41.775432+0000","last_undegraded":"2024-04-06T13:45:41.775432+0000","last_fullsized":"2024-04-06T13:45:41.775432+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:02:48.408226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"208'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775065+0000","last_change":"2024-04-06T13:45:39.680555+0000","last_active":"2024-04-06T13:45:41.775065+0000","last_peered":"2024-04-06T13:45:41.775065+0000","last_clean":"2024-04-06T13:45:41.775065+0000","last_became_active":"2024-04-06T13:45:39.680250+0000","last_became_peered":"2024-04-06T13:45:39.680250+0000","last_unstale":"2024-04-06T13:45:41.775065+0000","last_undegraded":"2024-04-06T13:45:41.775065+0000","last_fullsized":"2024-04-06T13:45:41.775065+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:25.490671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775676+0000","last_change":"2024-04-06T13:45:40.094003+0000","last_active":"2024-04-06T13:45:41.775676+0000","last_peered":"2024-04-06T13:45:41.775676+0000","last_clean":"2024-04-06T13:45:41.775676+0000","last_became_active":"2024-04-06T13:45:40.093766+0000","last_became_peered":"2024-04-06T13:45:40.093766+0000","last_unstale":"2024-04-06T13:45:41.775676+0000","last_undegraded":"2024-04-06T13:45:41.775676+0000","last_fullsized":"2024-04-06T13:45:41.775676+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:30:37.037455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775145+0000","last_change":"2024-04-06T13:45:39.670822+0000","last_active":"2024-04-06T13:45:41.775145+0000","last_peered":"2024-04-06T13:45:41.775145+0000","last_clean":"2024-04-06T13:45:41.775145+0000","last_became_active":"2024-04-06T13:45:39.670564+0000","last_became_peered":"2024-04-06T13:45:39.670564+0000","last_unstale":"2024-04-06T13:45:41.775145+0000","last_undegraded":"2024-04-06T13:45:41.775145+0000","last_fullsized":"2024-04-06T13:45:41.775145+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:42:01.443083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774476+0000","last_change":"2024-04-06T13:45:40.094718+0000","last_active":"2024-04-06T13:45:41.774476+0000","last_peered":"2024-04-06T13:45:41.774476+0000","last_clean":"2024-04-06T13:45:41.774476+0000","last_became_active":"2024-04-06T13:45:40.094486+0000","last_became_peered":"2024-04-06T13:45:40.094486+0000","last_unstale":"2024-04-06T13:45:41.774476+0000","last_undegraded":"2024-04-06T13:45:41.774476+0000","last_fullsized":"2024-04-06T13:45:41.774476+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:23:00.349095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"208'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086661+0000","last_change":"2024-04-06T13:45:39.679842+0000","last_active":"2024-04-06T13:45:41.086661+0000","last_peered":"2024-04-06T13:45:41.086661+0000","last_clean":"2024-04-06T13:45:41.086661+0000","last_became_active":"2024-04-06T13:45:39.679543+0000","last_became_peered":"2024-04-06T13:45:39.679543+0000","last_unstale":"2024-04-06T13:45:41.086661+0000","last_undegraded":"2024-04-06T13:45:41.086661+0000","last_fullsized":"2024-04-06T13:45:41.086661+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:11:58.314869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774371+0000","last_change":"2024-04-06T13:45:40.091910+0000","last_active":"2024-04-06T13:45:41.774371+0000","last_peered":"2024-04-06T13:45:41.774371+0000","last_clean":"2024-04-06T13:45:41.774371+0000","last_became_active":"2024-04-06T13:45:40.091677+0000","last_became_peered":"2024-04-06T13:45:40.091677+0000","last_unstale":"2024-04-06T13:45:41.774371+0000","last_undegraded":"2024-04-06T13:45:41.774371+0000","last_fullsized":"2024-04-06T13:45:41.774371+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:11:43.111082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774856+0000","last_change":"2024-04-06T13:45:39.672126+0000","last_active":"2024-04-06T13:45:41.774856+0000","last_peered":"2024-04-06T13:45:41.774856+0000","last_clean":"2024-04-06T13:45:41.774856+0000","last_became_active":"2024-04-06T13:45:39.671829+0000","last_became_peered":"2024-04-06T13:45:39.671829+0000","last_unstale":"2024-04-06T13:45:41.774856+0000","last_undegraded":"2024-04-06T13:45:41.774856+0000","last_fullsized":"2024-04-06T13:45:41.774856+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:03:24.392911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774624+0000","last_change":"2024-04-06T13:45:40.092393+0000","last_active":"2024-04-06T13:45:41.774624+0000","last_peered":"2024-04-06T13:45:41.774624+0000","last_clean":"2024-04-06T13:45:41.774624+0000","last_became_active":"2024-04-06T13:45:40.091375+0000","last_became_peered":"2024-04-06T13:45:40.091375+0000","last_unstale":"2024-04-06T13:45:41.774624+0000","last_undegraded":"2024-04-06T13:45:41.774624+0000","last_fullsized":"2024-04-06T13:45:41.774624+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:58:11.892572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774442+0000","last_change":"2024-04-06T13:45:39.676166+0000","last_active":"2024-04-06T13:45:41.774442+0000","last_peered":"2024-04-06T13:45:41.774442+0000","last_clean":"2024-04-06T13:45:41.774442+0000","last_became_active":"2024-04-06T13:45:39.675914+0000","last_became_peered":"2024-04-06T13:45:39.675914+0000","last_unstale":"2024-04-06T13:45:41.774442+0000","last_undegraded":"2024-04-06T13:45:41.774442+0000","last_fullsized":"2024-04-06T13:45:41.774442+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:26:57.018766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085555+0000","last_change":"2024-04-06T13:45:40.097373+0000","last_active":"2024-04-06T13:45:41.085555+0000","last_peered":"2024-04-06T13:45:41.085555+0000","last_clean":"2024-04-06T13:45:41.085555+0000","last_became_active":"2024-04-06T13:45:40.097124+0000","last_became_peered":"2024-04-06T13:45:40.097124+0000","last_unstale":"2024-04-06T13:45:41.085555+0000","last_undegraded":"2024-04-06T13:45:41.085555+0000","last_fullsized":"2024-04-06T13:45:41.085555+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:13:09.133530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774615+0000","last_change":"2024-04-06T13:45:39.663422+0000","last_active":"2024-04-06T13:45:41.774615+0000","last_peered":"2024-04-06T13:45:41.774615+0000","last_clean":"2024-04-06T13:45:41.774615+0000","last_became_active":"2024-04-06T13:45:39.663168+0000","last_became_peered":"2024-04-06T13:45:39.663168+0000","last_unstale":"2024-04-06T13:45:41.774615+0000","last_undegraded":"2024-04-06T13:45:41.774615+0000","last_fullsized":"2024-04-06T13:45:41.774615+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:08:46.166704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775122+0000","last_change":"2024-04-06T13:45:40.100105+0000","last_active":"2024-04-06T13:45:41.775122+0000","last_peered":"2024-04-06T13:45:41.775122+0000","last_clean":"2024-04-06T13:45:41.775122+0000","last_became_active":"2024-04-06T13:45:40.099733+0000","last_became_peered":"2024-04-06T13:45:40.099733+0000","last_unstale":"2024-04-06T13:45:41.775122+0000","last_undegraded":"2024-04-06T13:45:41.775122+0000","last_fullsized":"2024-04-06T13:45:41.775122+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:36:01.951089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"208'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774978+0000","last_change":"2024-04-06T13:45:39.671913+0000","last_active":"2024-04-06T13:45:41.774978+0000","last_peered":"2024-04-06T13:45:41.774978+0000","last_clean":"2024-04-06T13:45:41.774978+0000","last_became_active":"2024-04-06T13:45:39.671222+0000","last_became_peered":"2024-04-06T13:45:39.671222+0000","last_unstale":"2024-04-06T13:45:41.774978+0000","last_undegraded":"2024-04-06T13:45:41.774978+0000","last_fullsized":"2024-04-06T13:45:41.774978+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:51:31.411757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775705+0000","last_change":"2024-04-06T13:45:40.096319+0000","last_active":"2024-04-06T13:45:41.775705+0000","last_peered":"2024-04-06T13:45:41.775705+0000","last_clean":"2024-04-06T13:45:41.775705+0000","last_became_active":"2024-04-06T13:45:40.096095+0000","last_became_peered":"2024-04-06T13:45:40.096095+0000","last_unstale":"2024-04-06T13:45:41.775705+0000","last_undegraded":"2024-04-06T13:45:41.775705+0000","last_fullsized":"2024-04-06T13:45:41.775705+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:22:14.398303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775345+0000","last_change":"2024-04-06T13:45:39.675889+0000","last_active":"2024-04-06T13:45:41.775345+0000","last_peered":"2024-04-06T13:45:41.775345+0000","last_clean":"2024-04-06T13:45:41.775345+0000","last_became_active":"2024-04-06T13:45:39.675594+0000","last_became_peered":"2024-04-06T13:45:39.675594+0000","last_unstale":"2024-04-06T13:45:41.775345+0000","last_undegraded":"2024-04-06T13:45:41.775345+0000","last_fullsized":"2024-04-06T13:45:41.775345+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:14:48.751801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.084974+0000","last_change":"2024-04-06T13:45:40.087121+0000","last_active":"2024-04-06T13:45:41.084974+0000","last_peered":"2024-04-06T13:45:41.084974+0000","last_clean":"2024-04-06T13:45:41.084974+0000","last_became_active":"2024-04-06T13:45:40.086842+0000","last_became_peered":"2024-04-06T13:45:40.086842+0000","last_unstale":"2024-04-06T13:45:41.084974+0000","last_undegraded":"2024-04-06T13:45:41.084974+0000","last_fullsized":"2024-04-06T13:45:41.084974+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:01:10.112627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085904+0000","last_change":"2024-04-06T13:45:39.680563+0000","last_active":"2024-04-06T13:45:41.085904+0000","last_peered":"2024-04-06T13:45:41.085904+0000","last_clean":"2024-04-06T13:45:41.085904+0000","last_became_active":"2024-04-06T13:45:39.679462+0000","last_became_peered":"2024-04-06T13:45:39.679462+0000","last_unstale":"2024-04-06T13:45:41.085904+0000","last_undegraded":"2024-04-06T13:45:41.085904+0000","last_fullsized":"2024-04-06T13:45:41.085904+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:26:01.577603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774909+0000","last_change":"2024-04-06T13:45:40.099989+0000","last_active":"2024-04-06T13:45:41.774909+0000","last_peered":"2024-04-06T13:45:41.774909+0000","last_clean":"2024-04-06T13:45:41.774909+0000","last_became_active":"2024-04-06T13:45:40.099769+0000","last_became_peered":"2024-04-06T13:45:40.099769+0000","last_unstale":"2024-04-06T13:45:41.774909+0000","last_undegraded":"2024-04-06T13:45:41.774909+0000","last_fullsized":"2024-04-06T13:45:41.774909+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:54:30.364282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085738+0000","last_change":"2024-04-06T13:45:39.679145+0000","last_active":"2024-04-06T13:45:41.085738+0000","last_peered":"2024-04-06T13:45:41.085738+0000","last_clean":"2024-04-06T13:45:41.085738+0000","last_became_active":"2024-04-06T13:45:39.678837+0000","last_became_peered":"2024-04-06T13:45:39.678837+0000","last_unstale":"2024-04-06T13:45:41.085738+0000","last_undegraded":"2024-04-06T13:45:41.085738+0000","last_fullsized":"2024-04-06T13:45:41.085738+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:36:13.699413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774564+0000","last_change":"2024-04-06T13:45:40.091811+0000","last_active":"2024-04-06T13:45:41.774564+0000","last_peered":"2024-04-06T13:45:41.774564+0000","last_clean":"2024-04-06T13:45:41.774564+0000","last_became_active":"2024-04-06T13:45:40.091484+0000","last_became_peered":"2024-04-06T13:45:40.091484+0000","last_unstale":"2024-04-06T13:45:41.774564+0000","last_undegraded":"2024-04-06T13:45:41.774564+0000","last_fullsized":"2024-04-06T13:45:41.774564+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:36:37.635662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.084641+0000","last_change":"2024-04-06T13:45:39.666180+0000","last_active":"2024-04-06T13:45:41.084641+0000","last_peered":"2024-04-06T13:45:41.084641+0000","last_clean":"2024-04-06T13:45:41.084641+0000","last_became_active":"2024-04-06T13:45:39.665857+0000","last_became_peered":"2024-04-06T13:45:39.665857+0000","last_unstale":"2024-04-06T13:45:41.084641+0000","last_undegraded":"2024-04-06T13:45:41.084641+0000","last_fullsized":"2024-04-06T13:45:41.084641+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:43:34.876173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086452+0000","last_change":"2024-04-06T13:45:40.087647+0000","last_active":"2024-04-06T13:45:41.086452+0000","last_peered":"2024-04-06T13:45:41.086452+0000","last_clean":"2024-04-06T13:45:41.086452+0000","last_became_active":"2024-04-06T13:45:40.087119+0000","last_became_peered":"2024-04-06T13:45:40.087119+0000","last_unstale":"2024-04-06T13:45:41.086452+0000","last_undegraded":"2024-04-06T13:45:41.086452+0000","last_fullsized":"2024-04-06T13:45:41.086452+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:45:23.813731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.084497+0000","last_change":"2024-04-06T13:45:39.679474+0000","last_active":"2024-04-06T13:45:41.084497+0000","last_peered":"2024-04-06T13:45:41.084497+0000","last_clean":"2024-04-06T13:45:41.084497+0000","last_became_active":"2024-04-06T13:45:39.679220+0000","last_became_peered":"2024-04-06T13:45:39.679220+0000","last_unstale":"2024-04-06T13:45:41.084497+0000","last_undegraded":"2024-04-06T13:45:41.084497+0000","last_fullsized":"2024-04-06T13:45:41.084497+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:17:30.456677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774915+0000","last_change":"2024-04-06T13:45:40.095592+0000","last_active":"2024-04-06T13:45:41.774915+0000","last_peered":"2024-04-06T13:45:41.774915+0000","last_clean":"2024-04-06T13:45:41.774915+0000","last_became_active":"2024-04-06T13:45:40.095138+0000","last_became_peered":"2024-04-06T13:45:40.095138+0000","last_unstale":"2024-04-06T13:45:41.774915+0000","last_undegraded":"2024-04-06T13:45:41.774915+0000","last_fullsized":"2024-04-06T13:45:41.774915+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:02:59.095200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774562+0000","last_change":"2024-04-06T13:45:39.663365+0000","last_active":"2024-04-06T13:45:41.774562+0000","last_peered":"2024-04-06T13:45:41.774562+0000","last_clean":"2024-04-06T13:45:41.774562+0000","last_became_active":"2024-04-06T13:45:39.663090+0000","last_became_peered":"2024-04-06T13:45:39.663090+0000","last_unstale":"2024-04-06T13:45:41.774562+0000","last_undegraded":"2024-04-06T13:45:41.774562+0000","last_fullsized":"2024-04-06T13:45:41.774562+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:06:43.020693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.084554+0000","last_change":"2024-04-06T13:45:40.089447+0000","last_active":"2024-04-06T13:45:41.084554+0000","last_peered":"2024-04-06T13:45:41.084554+0000","last_clean":"2024-04-06T13:45:41.084554+0000","last_became_active":"2024-04-06T13:45:40.089143+0000","last_became_peered":"2024-04-06T13:45:40.089143+0000","last_unstale":"2024-04-06T13:45:41.084554+0000","last_undegraded":"2024-04-06T13:45:41.084554+0000","last_fullsized":"2024-04-06T13:45:41.084554+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:50:04.043102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774720+0000","last_change":"2024-04-06T13:45:39.668112+0000","last_active":"2024-04-06T13:45:41.774720+0000","last_peered":"2024-04-06T13:45:41.774720+0000","last_clean":"2024-04-06T13:45:41.774720+0000","last_became_active":"2024-04-06T13:45:39.667831+0000","last_became_peered":"2024-04-06T13:45:39.667831+0000","last_unstale":"2024-04-06T13:45:41.774720+0000","last_undegraded":"2024-04-06T13:45:41.774720+0000","last_fullsized":"2024-04-06T13:45:41.774720+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:46:27.010137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774839+0000","last_change":"2024-04-06T13:45:40.094387+0000","last_active":"2024-04-06T13:45:41.774839+0000","last_peered":"2024-04-06T13:45:41.774839+0000","last_clean":"2024-04-06T13:45:41.774839+0000","last_became_active":"2024-04-06T13:45:40.094113+0000","last_became_peered":"2024-04-06T13:45:40.094113+0000","last_unstale":"2024-04-06T13:45:41.774839+0000","last_undegraded":"2024-04-06T13:45:41.774839+0000","last_fullsized":"2024-04-06T13:45:41.774839+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:28:35.241968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774374+0000","last_change":"2024-04-06T13:45:39.667029+0000","last_active":"2024-04-06T13:45:41.774374+0000","last_peered":"2024-04-06T13:45:41.774374+0000","last_clean":"2024-04-06T13:45:41.774374+0000","last_became_active":"2024-04-06T13:45:39.666367+0000","last_became_peered":"2024-04-06T13:45:39.666367+0000","last_unstale":"2024-04-06T13:45:41.774374+0000","last_undegraded":"2024-04-06T13:45:41.774374+0000","last_fullsized":"2024-04-06T13:45:41.774374+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:49:47.202562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"198'326","reported_seq":1140,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-04-06T13:45:40.080815+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:45:40.080815+0000","last_peered":"2024-04-06T13:45:40.080815+0000","last_clean":"2024-04-06T13:45:40.080815+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:45:40.080815+0000","last_undegraded":"2024-04-06T13:45:40.080815+0000","last_fullsized":"2024-04-06T13:45:40.080815+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575964,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52308,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319532,"statfs":{"total":96636764160,"available":96583200768,"internally_reserved":0,"allocated":3215360,"data_stored":1923767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61099999999999999,"15min":0.60699999999999998},"min":{"1min":0.47199999999999998,"5min":0.38700000000000001,"15min":0.312},"max":{"1min":0.70799999999999996,"5min":0.79700000000000004,"15min":0.84899999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60199999999999998,"15min":0.60299999999999998},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.35199999999999998},"max":{"1min":0.72899999999999998,"5min":0.80800000000000005,"15min":0.82899999999999996},"last":0.56599999999999995}]},{"osd":1,"last update":"Sat Apr 6 13:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58999999999999997,"15min":0.59699999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.5},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.46000000000000002,"5min":0.41999999999999998,"15min":0.33000000000000002},"max":{"1min":0.754,"5min":0.83399999999999996,"15min":0.85299999999999998},"last":0.51900000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.47399999999999998,"5min":0.41299999999999998,"15min":0.36599999999999999},"max":{"1min":0.78600000000000003,"5min":0.82699999999999996,"15min":0.83899999999999997},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60799999999999998,"15min":0.60999999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.72999999999999998,"5min":0.79700000000000004,"15min":0.81699999999999995},"last":0.60599999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.48299999999999998,"5min":0.432,"15min":0.35999999999999999},"max":{"1min":0.76700000000000002,"5min":0.81100000000000005,"15min":0.86199999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60999999999999999,"15min":0.61499999999999999},"min":{"1min":0.42099999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.74099999999999999,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.67000000000000004}]},{"osd":4,"last update":"Sat Apr 6 13:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.30299999999999999},"max":{"1min":0.61799999999999999,"5min":0.76000000000000001,"15min":0.81599999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.36799999999999999},"max":{"1min":0.65900000000000003,"5min":0.74299999999999999,"15min":0.82699999999999996},"last":0.58199999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.48999999999999999,"15min":0.497},"min":{"1min":0.318,"5min":0.318,"15min":0.308},"max":{"1min":0.69299999999999995,"5min":0.76600000000000001,"15min":0.78900000000000003},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57699999999999996,"15min":0.57999999999999996},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.33600000000000002},"max":{"1min":0.68400000000000005,"5min":0.82099999999999995,"15min":0.85399999999999998},"last":0.68600000000000005}]},{"osd":6,"last update":"Sat Apr 6 13:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.379,"5min":0.374,"15min":0.34899999999999998},"max":{"1min":0.70599999999999996,"5min":0.81200000000000006,"15min":0.879},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.40100000000000002,"5min":0.377,"15min":0.34799999999999998},"max":{"1min":0.70199999999999996,"5min":0.84899999999999998,"15min":0.89800000000000002},"last":0.64800000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575963,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37764,"kb_used_data":2548,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334076,"statfs":{"total":96636764160,"available":96598093824,"internally_reserved":0,"allocated":2609152,"data_stored":1325758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:44:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.373,"15min":0.32000000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.83799999999999997},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56599999999999995,"15min":0.57199999999999995},"min":{"1min":0.442,"5min":0.38700000000000001,"15min":0.37},"max":{"1min":0.68500000000000005,"5min":0.81999999999999995,"15min":0.82399999999999995},"last":0.497}]},{"osd":1,"last update":"Sat Apr 6 13:44:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.628,"15min":0.63100000000000001},"min":{"1min":0.52600000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.78900000000000003,"5min":0.83699999999999997,"15min":0.85799999999999998},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64800000000000002,"15min":0.65100000000000002},"min":{"1min":0.495,"5min":0.378,"15min":0.378},"max":{"1min":0.752,"5min":0.81399999999999995,"15min":0.82699999999999996},"last":0.57999999999999996}]},{"osd":2,"last update":"Sat Apr 6 13:44:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60299999999999998,"15min":0.59399999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.42299999999999999},"max":{"1min":0.73099999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58999999999999997,"15min":0.59599999999999997},"min":{"1min":0.39200000000000002,"5min":0.375,"15min":0.35799999999999998},"max":{"1min":0.70999999999999996,"5min":0.78600000000000003,"15min":0.82599999999999996},"last":0.51200000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:44:50 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61299999999999999,"15min":0.61099999999999999},"min":{"1min":0.42199999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.75600000000000001,"5min":0.77900000000000003,"15min":0.82399999999999995},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64200000000000002,"15min":0.65000000000000002},"min":{"1min":0.45900000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77200000000000002,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.56899999999999995}]},{"osd":4,"last update":"Sat Apr 6 13:44:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53900000000000003,"15min":0.52800000000000002},"min":{"1min":0.36899999999999999,"5min":0.33800000000000002,"15min":0.30099999999999999},"max":{"1min":0.70699999999999996,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.54000000000000004,"15min":0.55100000000000005},"min":{"1min":0.34000000000000002,"5min":0.33800000000000002,"15min":0.28899999999999998},"max":{"1min":0.82099999999999995,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.58999999999999997}]},{"osd":5,"last update":"Sat Apr 6 13:44:50 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50800000000000001,"15min":0.51200000000000001},"min":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.27100000000000002},"max":{"1min":0.68400000000000005,"5min":0.78000000000000003,"15min":0.78400000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.28999999999999998},"max":{"1min":0.73899999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.45600000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:44:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54400000000000004,"15min":0.55100000000000005},"min":{"1min":0.40500000000000003,"5min":0.39500000000000002,"15min":0.312},"max":{"1min":0.72099999999999997,"5min":0.82599999999999996,"15min":0.84799999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.41899999999999998,"5min":0.36799999999999999,"15min":0.32500000000000001},"max":{"1min":0.77600000000000002,"5min":0.82499999999999996,"15min":0.86199999999999999},"last":0.55300000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59908,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311932,"statfs":{"total":96636764160,"available":96575418368,"internally_reserved":0,"allocated":2605056,"data_stored":1325688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39000000000000001,"15min":0.39600000000000002},"min":{"1min":0.29899999999999999,"5min":0.25,"15min":0.25},"max":{"1min":0.53100000000000003,"5min":0.56499999999999995,"15min":0.64500000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.433,"15min":0.433},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.254},"max":{"1min":0.56100000000000005,"5min":0.60999999999999999,"15min":0.69399999999999995},"last":0.40400000000000003}]},{"osd":2,"last update":"Sat Apr 6 13:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42299999999999999,"15min":0.42999999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.54600000000000004,"5min":0.61799999999999999,"15min":0.65900000000000003},"last":0.504},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.46800000000000003,"15min":0.46899999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.29399999999999998},"max":{"1min":0.64300000000000002,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.48899999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.33700000000000002,"5min":0.29799999999999999,"15min":0.28799999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.439,"15min":0.44400000000000001},"min":{"1min":0.30599999999999999,"5min":0.29699999999999999,"15min":0.26300000000000001},"max":{"1min":0.59099999999999997,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.42499999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63700000000000001,"15min":0.65300000000000002},"min":{"1min":0.47599999999999998,"5min":0.433,"15min":0.33800000000000002},"max":{"1min":0.872,"5min":0.95299999999999996,"15min":0.97399999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64300000000000002,"15min":0.66400000000000003},"min":{"1min":0.49199999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85999999999999999,"5min":0.97599999999999998,"15min":0.995},"last":0.70799999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.65800000000000003},"min":{"1min":0.48599999999999999,"5min":0.442,"15min":0.40799999999999997},"max":{"1min":0.77500000000000002,"5min":0.96899999999999997,"15min":1.073},"last":0.63},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66200000000000003,"15min":0.67700000000000005},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.81599999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.58599999999999997}]},{"osd":6,"last update":"Sat Apr 6 13:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64000000000000001,"15min":0.66700000000000004},"min":{"1min":0.47599999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.84599999999999997,"5min":0.90900000000000003,"15min":1},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.67300000000000004,"15min":0.69199999999999995},"min":{"1min":0.55500000000000005,"5min":0.48699999999999999,"15min":0.46800000000000003},"max":{"1min":0.82799999999999996,"5min":0.93999999999999995,"15min":1.036},"last":0.66900000000000004}]},{"osd":7,"last update":"Sat Apr 6 13:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.69399999999999995,"15min":0.70199999999999996},"min":{"1min":0.51900000000000002,"5min":0.50900000000000001,"15min":0.501},"max":{"1min":0.90200000000000002,"5min":0.98999999999999999,"15min":1.0649999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66000000000000003,"15min":0.67900000000000005},"min":{"1min":0.46700000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.86899999999999999,"5min":0.90000000000000002,"15min":1.056},"last":0.64500000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575962,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42468,"kb_used_data":3152,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329372,"statfs":{"total":96636764160,"available":96593276928,"internally_reserved":0,"allocated":3227648,"data_stored":1923476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.40000000000000002,"15min":0.40100000000000002},"min":{"1min":0.29299999999999998,"5min":0.26800000000000002,"15min":0.23799999999999999},"max":{"1min":0.56100000000000005,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.251},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.41999999999999998,"15min":0.42699999999999999},"min":{"1min":0.28199999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.58599999999999997,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.33400000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.42899999999999999,"15min":0.436},"min":{"1min":0.311,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.65400000000000003,"5min":0.67900000000000005,"15min":0.70099999999999996},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44400000000000001,"15min":0.45500000000000002},"min":{"1min":0.32500000000000001,"5min":0.316,"15min":0.315},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.27100000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.432,"15min":0.436},"min":{"1min":0.33500000000000002,"5min":0.28899999999999998,"15min":0.26800000000000002},"max":{"1min":0.63300000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.48299999999999998,"15min":0.48799999999999999},"min":{"1min":0.38,"5min":0.307,"15min":0.307},"max":{"1min":0.71899999999999997,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.34799999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.84899999999999998,"5min":0.92500000000000004,"15min":1.0329999999999999},"last":0.37},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.64900000000000002,"15min":0.65800000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.41999999999999998},"max":{"1min":0.94399999999999995,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.43099999999999999}]},{"osd":5,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63300000000000001,"15min":0.65000000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.40600000000000003},"max":{"1min":0.876,"5min":0.876,"15min":1.0009999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66900000000000004,"15min":0.67800000000000005},"min":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.92800000000000005,"5min":0.96199999999999997,"15min":1.052},"last":0.378}]},{"osd":6,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.67700000000000005,"15min":0.68700000000000006},"min":{"1min":0.48099999999999998,"5min":0.435,"15min":0.435},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.439},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67300000000000004,"15min":0.68400000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.90500000000000003,"5min":0.96399999999999997,"15min":1.085},"last":0.39000000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.69299999999999995,"15min":0.70299999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.01,"5min":1.01,"15min":1.121},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67800000000000005,"15min":0.68500000000000005},"min":{"1min":0.51100000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.89500000000000002,"5min":0.97699999999999998,"15min":1.0229999999999999},"last":0.36099999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575964,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59920,"kb_used_data":2560,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311920,"statfs":{"total":96636764160,"available":96575406080,"internally_reserved":0,"allocated":2621440,"data_stored":1332962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.38600000000000001,"15min":0.38800000000000001},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.52700000000000002,"5min":0.58399999999999996,"15min":0.59099999999999997},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.433,"15min":0.42899999999999999},"min":{"1min":0.30599999999999999,"5min":0.27000000000000002,"15min":0.249},"max":{"1min":0.57799999999999996,"5min":0.64100000000000001,"15min":1.044},"last":0.38800000000000001}]},{"osd":1,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.41199999999999998,"15min":0.41299999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.255},"max":{"1min":0.52900000000000003,"5min":0.56399999999999995,"15min":0.753},"last":0.313},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44600000000000001,"15min":0.45100000000000001},"min":{"1min":0.29299999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.60799999999999998,"5min":0.627,"15min":0.82199999999999995},"last":0.41999999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.47399999999999998,"15min":0.46600000000000003},"min":{"1min":0.34300000000000003,"5min":0.31900000000000001,"15min":0.27800000000000002},"max":{"1min":0.61499999999999999,"5min":0.69699999999999995,"15min":0.70299999999999996},"last":0.372},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45900000000000002,"15min":0.45400000000000001},"min":{"1min":0.33200000000000002,"5min":0.312,"15min":0.30299999999999999},"max":{"1min":0.57899999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.32700000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65500000000000003,"15min":0.64600000000000002},"min":{"1min":0.47399999999999998,"5min":0.41799999999999998,"15min":0.376},"max":{"1min":0.83099999999999996,"5min":0.91800000000000004,"15min":0.93400000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.68100000000000005,"15min":0.67100000000000004},"min":{"1min":0.504,"5min":0.42499999999999999,"15min":0.41899999999999998},"max":{"1min":0.83299999999999996,"5min":0.97599999999999998,"15min":1.0229999999999999},"last":0.58099999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68600000000000005,"15min":0.67100000000000004},"min":{"1min":0.46500000000000002,"5min":0.46000000000000002,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.68799999999999994,"15min":0.67200000000000004},"min":{"1min":0.50800000000000001,"5min":0.46200000000000002,"15min":0.42899999999999999},"max":{"1min":0.80300000000000005,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.54300000000000004}]},{"osd":6,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67900000000000005,"15min":0.67800000000000005},"min":{"1min":0.47599999999999998,"5min":0.46700000000000003,"15min":0.433},"max":{"1min":0.88600000000000001,"5min":1.016,"15min":1.091},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.71799999999999997,"15min":0.70199999999999996},"min":{"1min":0.56799999999999995,"5min":0.50600000000000001,"15min":0.41299999999999998},"max":{"1min":0.89600000000000002,"5min":0.95899999999999996,"15min":0.97699999999999998},"last":0.59699999999999998}]},{"osd":7,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.69999999999999996,"15min":0.69599999999999995},"min":{"1min":0.54500000000000004,"5min":0.48199999999999998,"15min":0.47999999999999998},"max":{"1min":0.86699999999999999,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72199999999999998,"15min":0.71499999999999997},"min":{"1min":0.58499999999999996,"5min":0.45900000000000002,"15min":0.443},"max":{"1min":0.871,"5min":0.99099999999999999,"15min":1.0089999999999999},"last":0.56899999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575963,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47636,"kb_used_data":2568,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324204,"statfs":{"total":96636764160,"available":96587984896,"internally_reserved":0,"allocated":2629632,"data_stored":1333559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.38600000000000001,"15min":0.39000000000000001},"min":{"1min":0.27200000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.502,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.41499999999999998,"15min":0.41599999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.58599999999999997,"5min":0.622,"15min":0.66700000000000004},"last":0.374}]},{"osd":1,"last update":"Sat Apr 6 13:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.41499999999999998,"15min":0.40899999999999997},"min":{"1min":0.36199999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.58899999999999997,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.436,"15min":0.44700000000000001},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.60599999999999998,"5min":0.66000000000000003,"15min":0.75800000000000001},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48899999999999999,"15min":0.49099999999999999},"min":{"1min":0.39600000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.60399999999999998,"5min":0.68999999999999995,"15min":0.70299999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.50600000000000001,"15min":0.50800000000000001},"min":{"1min":0.39100000000000001,"5min":0.33000000000000002,"15min":0.32900000000000001},"max":{"1min":0.64200000000000002,"5min":0.747,"15min":0.79300000000000004},"last":0.44500000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.64200000000000002,"15min":0.64500000000000002},"min":{"1min":0.504,"5min":0.40300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88200000000000001,"5min":1.002,"15min":1.018},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.64300000000000002,"15min":0.65400000000000003},"min":{"1min":0.5,"5min":0.375,"15min":0.375},"max":{"1min":0.79600000000000004,"5min":0.98599999999999999,"15min":0.995},"last":0.72899999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68400000000000005,"15min":0.69099999999999995},"min":{"1min":0.57699999999999996,"5min":0.46200000000000002,"15min":0.45900000000000002},"max":{"1min":0.879,"5min":0.97299999999999998,"15min":0.999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.68899999999999995,"15min":0.69899999999999995},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.46600000000000003},"max":{"1min":0.92300000000000004,"5min":0.92800000000000005,"15min":0.996},"last":0.76900000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.66100000000000003,"15min":0.67100000000000004},"min":{"1min":0.55800000000000005,"5min":0.443,"15min":0.443},"max":{"1min":0.83699999999999997,"5min":0.95199999999999996,"15min":0.97999999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.68600000000000005,"15min":0.68899999999999995},"min":{"1min":0.55900000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91600000000000004,"5min":0.96199999999999997,"15min":0.98799999999999999},"last":0.74299999999999999}]},{"osd":7,"last update":"Sat Apr 6 13:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.67600000000000005,"15min":0.68300000000000005},"min":{"1min":0.51300000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.94999999999999996,"5min":0.96199999999999997,"15min":0.97899999999999998},"last":0.625},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.69999999999999996,"15min":0.70499999999999996},"min":{"1min":0.60399999999999998,"5min":0.48499999999999999,"15min":0.47899999999999998},"max":{"1min":0.85999999999999999,"5min":0.98799999999999999,"15min":1.024},"last":0.68500000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64036,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307804,"statfs":{"total":96636764160,"available":96571191296,"internally_reserved":0,"allocated":2637824,"data_stored":1334997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58599999999999997,"15min":0.59799999999999998},"min":{"1min":0.51600000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.80900000000000005,"5min":0.876,"15min":0.88500000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.46100000000000002,"5min":0.40600000000000003,"15min":0.34799999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.68999999999999995}]},{"osd":1,"last update":"Sat Apr 6 13:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63600000000000001,"15min":0.64500000000000002},"min":{"1min":0.41199999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77900000000000003,"5min":0.84899999999999998,"15min":0.873},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64100000000000001,"15min":0.63200000000000001},"min":{"1min":0.54000000000000004,"5min":0.436,"15min":0.39700000000000002},"max":{"1min":0.81699999999999995,"5min":0.82299999999999995,"15min":0.86299999999999999},"last":0.55100000000000005}]},{"osd":2,"last update":"Sat Apr 6 13:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59999999999999998,"15min":0.60399999999999998},"min":{"1min":0.49299999999999999,"5min":0.41299999999999998,"15min":0.35399999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.877},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63200000000000001,"15min":0.621},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.38700000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.90400000000000003},"last":0.53200000000000003}]},{"osd":3,"last update":"Sat Apr 6 13:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.45500000000000002,"5min":0.41699999999999998,"15min":0.39900000000000002},"max":{"1min":0.83899999999999997,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63500000000000001,"15min":0.63100000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.40400000000000003},"max":{"1min":0.876,"5min":0.876,"15min":0.91100000000000003},"last":0.56399999999999995}]},{"osd":5,"last update":"Sat Apr 6 13:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.505,"15min":0.496},"min":{"1min":0.35399999999999998,"5min":0.33200000000000002,"15min":0.32800000000000001},"max":{"1min":0.68799999999999994,"5min":0.749,"15min":0.77700000000000002},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53700000000000003,"15min":0.52600000000000002},"min":{"1min":0.40300000000000002,"5min":0.379,"15min":0.34499999999999997},"max":{"1min":0.73799999999999999,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.51700000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53600000000000003,"15min":0.52800000000000002},"min":{"1min":0.36699999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75700000000000001,"5min":0.78800000000000003,"15min":0.83899999999999997},"last":0.498},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.38700000000000001,"5min":0.373,"15min":0.34399999999999997},"max":{"1min":0.78500000000000003,"5min":0.82399999999999995,"15min":0.84199999999999997},"last":0.58999999999999997}]},{"osd":7,"last update":"Sat Apr 6 13:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59899999999999998,"15min":0.60399999999999998},"min":{"1min":0.41599999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.86499999999999999,"5min":0.878,"15min":0.88100000000000001},"last":0.64100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59928,"kb_used_data":2572,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311912,"statfs":{"total":96636764160,"available":96575397888,"internally_reserved":0,"allocated":2633728,"data_stored":1333966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:44:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.59699999999999998,"15min":0.58999999999999997},"min":{"1min":0.372,"5min":0.34000000000000002,"15min":0.27600000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.60399999999999998,"15min":0.59699999999999998},"min":{"1min":0.47799999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.89200000000000002},"last":0.54200000000000004}]},{"osd":1,"last update":"Sat Apr 6 13:44:40 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60499999999999998,"15min":0.59399999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.32600000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81399999999999995},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59399999999999997,"15min":0.58899999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.311},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.83199999999999996},"last":0.505}]},{"osd":2,"last update":"Sat Apr 6 13:44:40 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61199999999999999,"15min":0.60499999999999998},"min":{"1min":0.45500000000000002,"5min":0.36399999999999999,"15min":0.312},"max":{"1min":0.78800000000000003,"5min":0.84099999999999997,"15min":0.84399999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.60899999999999999,"15min":0.59999999999999998},"min":{"1min":0.46800000000000003,"5min":0.39700000000000002,"15min":0.34699999999999998},"max":{"1min":0.72799999999999998,"5min":0.79500000000000004,"15min":0.82199999999999995},"last":0.56299999999999994}]},{"osd":3,"last update":"Sat Apr 6 13:44:40 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61399999999999999,"15min":0.60099999999999998},"min":{"1min":0.498,"5min":0.41099999999999998,"15min":0.28799999999999998},"max":{"1min":0.76700000000000002,"5min":0.79000000000000004,"15min":0.94499999999999995},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.63300000000000001,"15min":0.63500000000000001},"min":{"1min":0.40300000000000002,"5min":0.39000000000000001,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":0.84499999999999997,"15min":0.91500000000000004},"last":0.47599999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:44:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.50600000000000001,"15min":0.5},"min":{"1min":0.374,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.76400000000000001,"5min":0.76500000000000001,"15min":1.0369999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58199999999999996,"15min":0.56899999999999995},"min":{"1min":0.39000000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":1.1639999999999999},"last":0.66300000000000003}]},{"osd":6,"last update":"Sat Apr 6 13:44:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52300000000000002,"15min":0.51700000000000002},"min":{"1min":0.33800000000000002,"5min":0.28899999999999998,"15min":0.27900000000000003},"max":{"1min":0.74399999999999999,"5min":0.755,"15min":1.0169999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53000000000000003,"15min":0.51900000000000002},"min":{"1min":0.35399999999999998,"5min":0.32300000000000001,"15min":0.30399999999999999},"max":{"1min":0.77800000000000002,"5min":0.877,"15min":1.0009999999999999},"last":0.68899999999999995}]},{"osd":7,"last update":"Sat Apr 6 13:44:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54700000000000004,"15min":0.53600000000000003},"min":{"1min":0.39300000000000002,"5min":0.33600000000000002,"15min":0.32500000000000001},"max":{"1min":0.75600000000000001,"5min":0.80300000000000005,"15min":0.84199999999999997},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.64900000000000002,"15min":0.64000000000000001},"min":{"1min":0.498,"5min":0.38300000000000001,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.96599999999999997},"last":0.70099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:45:44.217 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:45:44.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.374+0000 7f7e0a6c1700 1 Processor -- start 2024-04-06T13:45:44.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.374+0000 7f7e0a6c1700 1 -- start start 2024-04-06T13:45:44.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e0a6c1700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7e0406ae40 0x7f7e040b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:44.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e0a6c1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e040b3990 con 0x7f7e0415d7b0 2024-04-06T13:45:44.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e0a6c1700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e040b3060 con 0x7f7e0415ddf0 2024-04-06T13:45:44.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e0a6c1700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e0415e950 con 0x7f7e0406ae40 2024-04-06T13:45:44.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e03fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7e0415d7b0 0x7f7e0406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55920/0 (socket says 172.21.15.189:55920) 2024-04-06T13:45:44.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e03fff700 1 -- 172.21.15.189:0/1197901633 learned_addr learned my addr 172.21.15.189:0/1197901633 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:44.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e02ffd700 1 -- 172.21.15.189:0/1197901633 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463929803 0 0) 0x7f7e040b3990 con 0x7f7e0415d7b0 2024-04-06T13:45:44.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e02ffd700 1 -- 172.21.15.189:0/1197901633 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dec0035b0 con 0x7f7e0415d7b0 2024-04-06T13:45:44.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e037fe700 1 --2- 172.21.15.189:0/1197901633 >> v2:172.21.15.192:3300/0 conn(0x7f7e0406ae40 0x7f7e040b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:44.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e02ffd700 1 -- 172.21.15.189:0/1197901633 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3148364917 0 0) 0x7f7dec0035b0 con 0x7f7e0415d7b0 2024-04-06T13:45:44.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e02ffd700 1 -- 172.21.15.189:0/1197901633 >> v2:172.21.15.192:3300/0 conn(0x7f7e0406ae40 msgr2=0x7f7e040b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:44.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e02ffd700 1 --2- 172.21.15.189:0/1197901633 >> v2:172.21.15.192:3300/0 conn(0x7f7e0406ae40 0x7f7e040b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:44.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e02ffd700 1 -- 172.21.15.189:0/1197901633 >> v1:172.21.15.192:6789/0 conn(0x7f7e0415ddf0 legacy=0x7f7e0415e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:44.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e02ffd700 1 -- 172.21.15.189:0/1197901633 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e040b3340 con 0x7f7e0415d7b0 2024-04-06T13:45:44.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e02ffd700 1 -- 172.21.15.189:0/1197901633 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f7df8001670 con 0x7f7e0415d7b0 2024-04-06T13:45:44.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e02ffd700 1 -- 172.21.15.189:0/1197901633 >> v1:172.21.15.189:6789/0 conn(0x7f7e0415d7b0 legacy=0x7f7e0406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:44.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e02ffd700 1 --2- 172.21.15.189:0/1197901633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dec003a00 0x7f7dec003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e02ffd700 1 --2- 172.21.15.189:0/1197901633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7dec004850 0x7f7dec004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e02ffd700 1 --2- 172.21.15.189:0/1197901633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dec005210 0x7f7dec00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e02ffd700 1 -- 172.21.15.189:0/1197901633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec00c180 con 0x7f7dec004850 2024-04-06T13:45:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e02ffd700 1 -- 172.21.15.189:0/1197901633 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec00c2f0 con 0x7f7dec003a00 2024-04-06T13:45:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e02ffd700 1 -- 172.21.15.189:0/1197901633 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec00c4b0 con 0x7f7dec005210 2024-04-06T13:45:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e037fe700 1 --2- 172.21.15.189:0/1197901633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7dec004850 0x7f7dec004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e03fff700 1 --2- 172.21.15.189:0/1197901633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dec003a00 0x7f7dec003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e08c5e700 1 --2- 172.21.15.189:0/1197901633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dec005210 0x7f7dec00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e037fe700 1 -- 172.21.15.189:0/1197901633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dec005210 msgr2=0x7f7dec00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e037fe700 1 --2- 172.21.15.189:0/1197901633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dec005210 0x7f7dec00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e037fe700 1 -- 172.21.15.189:0/1197901633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dec003a00 msgr2=0x7f7dec003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e037fe700 1 --2- 172.21.15.189:0/1197901633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dec003a00 0x7f7dec003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e037fe700 1 -- 172.21.15.189:0/1197901633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dec0035b0 con 0x7f7dec004850 2024-04-06T13:45:44.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e08c5e700 1 --2- 172.21.15.189:0/1197901633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dec005210 0x7f7dec00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:44.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e08c5e700 1 -- 172.21.15.189:0/1197901633 reap_dead start 2024-04-06T13:45:44.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e037fe700 1 --2- 172.21.15.189:0/1197901633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7dec004850 0x7f7dec004cd0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009b30 tx=0x7f7dec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55b99b2dc210c3d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:44.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e02ffd700 1 -- 172.21.15.189:0/1197901633 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df400e6c0 con 0x7f7dec004850 2024-04-06T13:45:44.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e02ffd700 1 -- 172.21.15.189:0/1197901633 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df400e820 con 0x7f7dec004850 2024-04-06T13:45:44.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.375+0000 7f7e02ffd700 1 -- 172.21.15.189:0/1197901633 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df400eb90 con 0x7f7dec004850 2024-04-06T13:45:44.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.376+0000 7f7e0a6c1700 1 -- 172.21.15.189:0/1197901633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7dec004850 msgr2=0x7f7dec004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:44.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.376+0000 7f7e0a6c1700 1 --2- 172.21.15.189:0/1197901633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7dec004850 0x7f7dec004cd0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009b30 tx=0x7f7dec005880 comp rx=0 tx=0).stop 2024-04-06T13:45:44.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.376+0000 7f7e0a6c1700 1 -- 172.21.15.189:0/1197901633 shutdown_connections 2024-04-06T13:45:44.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.376+0000 7f7e0a6c1700 1 --2- 172.21.15.189:0/1197901633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7dec004850 0x7f7dec004cd0 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:44.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.376+0000 7f7e0a6c1700 1 -- 172.21.15.189:0/1197901633 >> 172.21.15.189:0/1197901633 conn(0x7f7e040d00b0 msgr2=0x7f7e040d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:44.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.376+0000 7f7e0a6c1700 1 -- 172.21.15.189:0/1197901633 shutdown_connections 2024-04-06T13:45:44.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.376+0000 7f7e0a6c1700 1 -- 172.21.15.189:0/1197901633 wait complete. 2024-04-06T13:45:44.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.377+0000 7f7e0a6c1700 1 Processor -- start 2024-04-06T13:45:44.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.377+0000 7f7e0a6c1700 1 -- start start 2024-04-06T13:45:44.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.378+0000 7f7e0a6c1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dec004850 0x7f7e0415fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:44.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.378+0000 7f7e0a6c1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e041f6c40 0x7f7e040ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:44.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.378+0000 7f7e0a6c1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e040af300 0x7f7e04151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:44.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.378+0000 7f7e0a6c1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e040ce090 con 0x7f7e041f6c40 2024-04-06T13:45:44.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.378+0000 7f7e0a6c1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e041f70c0 con 0x7f7e040af300 2024-04-06T13:45:44.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.378+0000 7f7e0a6c1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e04160140 con 0x7f7dec004850 2024-04-06T13:45:44.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.378+0000 7f7e037fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e041f6c40 0x7f7e040ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:44.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.378+0000 7f7e037fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e041f6c40 0x7f7e040ce4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48188/0 (socket says 172.21.15.189:48188) 2024-04-06T13:45:44.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.378+0000 7f7e037fe700 1 -- 172.21.15.189:0/3351091427 learned_addr learned my addr 172.21.15.189:0/3351091427 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:44.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.378+0000 7f7e03fff700 1 --2- 172.21.15.189:0/3351091427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dec004850 0x7f7e0415fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:44.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.378+0000 7f7e08c5e700 1 --2- 172.21.15.189:0/3351091427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e040af300 0x7f7e04151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:44.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.378+0000 7f7e037fe700 1 -- 172.21.15.189:0/3351091427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dec004850 msgr2=0x7f7e0415fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:44.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.378+0000 7f7e037fe700 1 --2- 172.21.15.189:0/3351091427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dec004850 0x7f7e0415fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:44.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.378+0000 7f7e037fe700 1 -- 172.21.15.189:0/3351091427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e040af300 msgr2=0x7f7e04151850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:44.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.378+0000 7f7e037fe700 1 --2- 172.21.15.189:0/3351091427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e040af300 0x7f7e04151850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:44.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.378+0000 7f7e037fe700 1 -- 172.21.15.189:0/3351091427 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e040cd6f0 con 0x7f7e041f6c40 2024-04-06T13:45:44.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.379+0000 7f7e037fe700 1 --2- 172.21.15.189:0/3351091427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e041f6c40 0x7f7e040ce4b0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00c2c0 tx=0x7f7df40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:44.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.379+0000 7f7e00ff9700 1 -- 172.21.15.189:0/3351091427 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df400e6c0 con 0x7f7e041f6c40 2024-04-06T13:45:44.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.379+0000 7f7e00ff9700 1 -- 172.21.15.189:0/3351091427 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df4013c00 con 0x7f7e041f6c40 2024-04-06T13:45:44.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.379+0000 7f7e0a6c1700 1 -- 172.21.15.189:0/3351091427 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e040cd9d0 con 0x7f7e041f6c40 2024-04-06T13:45:44.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.379+0000 7f7e00ff9700 1 -- 172.21.15.189:0/3351091427 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df401c450 con 0x7f7e041f6c40 2024-04-06T13:45:44.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.379+0000 7f7e0a6c1700 1 -- 172.21.15.189:0/3351091427 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e040ce9f0 con 0x7f7e041f6c40 2024-04-06T13:45:44.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.380+0000 7f7e0a6c1700 1 -- 172.21.15.189:0/3351091427 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e04067bb0 con 0x7f7e041f6c40 2024-04-06T13:45:44.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.380+0000 7f7e00ff9700 1 -- 172.21.15.189:0/3351091427 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7df4024070 con 0x7f7e041f6c40 2024-04-06T13:45:44.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.380+0000 7f7e00ff9700 1 --2- 172.21.15.189:0/3351091427 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7df006adb0 0x7f7df006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:44.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.381+0000 7f7e00ff9700 1 -- 172.21.15.189:0/3351091427 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7df408ac60 con 0x7f7e041f6c40 2024-04-06T13:45:44.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.381+0000 7f7e03fff700 1 --2- 172.21.15.189:0/3351091427 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7df006adb0 0x7f7df006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:44.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.381+0000 7f7e03fff700 1 --2- 172.21.15.189:0/3351091427 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7df006adb0 0x7f7df006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7df8000fb0 tx=0x7f7df8009040 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:44.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.386+0000 7f7e00ff9700 1 -- 172.21.15.189:0/3351091427 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7df4055d90 con 0x7f7e041f6c40 2024-04-06T13:45:44.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.578+0000 7f7e0a6c1700 1 -- 172.21.15.189:0/3351091427 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7e0415eb40 con 0x7f7df006adb0 2024-04-06T13:45:44.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.586+0000 7f7e00ff9700 1 -- 172.21.15.189:0/3351091427 <== mgr.10494 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290384 (secure 0 0 0) 0x7f7e0415eb40 con 0x7f7df006adb0 2024-04-06T13:45:44.589 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:44.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.589+0000 7f7e0a6c1700 1 -- 172.21.15.189:0/3351091427 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7df006adb0 msgr2=0x7f7df006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:44.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.589+0000 7f7e0a6c1700 1 --2- 172.21.15.189:0/3351091427 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7df006adb0 0x7f7df006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7df8000fb0 tx=0x7f7df8009040 comp rx=0 tx=0).stop 2024-04-06T13:45:44.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.589+0000 7f7e0a6c1700 1 -- 172.21.15.189:0/3351091427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e041f6c40 msgr2=0x7f7e040ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:44.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.589+0000 7f7e0a6c1700 1 --2- 172.21.15.189:0/3351091427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e041f6c40 0x7f7e040ce4b0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00c2c0 tx=0x7f7df40047c0 comp rx=0 tx=0).stop 2024-04-06T13:45:44.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.589+0000 7f7e0a6c1700 1 -- 172.21.15.189:0/3351091427 shutdown_connections 2024-04-06T13:45:44.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.589+0000 7f7e0a6c1700 1 --2- 172.21.15.189:0/3351091427 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7df006adb0 0x7f7df006d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:44.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.589+0000 7f7e0a6c1700 1 --2- 172.21.15.189:0/3351091427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e040af300 0x7f7e04151850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:44.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.589+0000 7f7e0a6c1700 1 --2- 172.21.15.189:0/3351091427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dec004850 0x7f7e0415fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:44.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.589+0000 7f7e0a6c1700 1 --2- 172.21.15.189:0/3351091427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e041f6c40 0x7f7e040ce4b0 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:44.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.589+0000 7f7e0a6c1700 1 -- 172.21.15.189:0/3351091427 >> 172.21.15.189:0/3351091427 conn(0x7f7e040d00b0 msgr2=0x7f7e0406fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:44.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.589+0000 7f7e0a6c1700 1 -- 172.21.15.189:0/3351091427 shutdown_connections 2024-04-06T13:45:44.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.589+0000 7f7e0a6c1700 1 -- 172.21.15.189:0/3351091427 wait complete. 2024-04-06T13:45:44.593 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:45:44.601 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-06T13:45:44.478993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":423968,"kb_used_data":21660,"kb_used_omap":12,"kb_used_meta":402163,"kb_avail":754550752,"statfs":{"total":773094113280,"available":772659970048,"internally_reserved":0,"allocated":22179840,"data_stored":11834173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":411815480},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.828961"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775504+0000","last_change":"2024-04-06T13:45:40.089525+0000","last_active":"2024-04-06T13:45:41.775504+0000","last_peered":"2024-04-06T13:45:41.775504+0000","last_clean":"2024-04-06T13:45:41.775504+0000","last_became_active":"2024-04-06T13:45:40.089241+0000","last_became_peered":"2024-04-06T13:45:40.089241+0000","last_unstale":"2024-04-06T13:45:41.775504+0000","last_undegraded":"2024-04-06T13:45:41.775504+0000","last_fullsized":"2024-04-06T13:45:41.775504+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:43:35.880636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775498+0000","last_change":"2024-04-06T13:45:40.100325+0000","last_active":"2024-04-06T13:45:41.775498+0000","last_peered":"2024-04-06T13:45:41.775498+0000","last_clean":"2024-04-06T13:45:41.775498+0000","last_became_active":"2024-04-06T13:45:40.100060+0000","last_became_peered":"2024-04-06T13:45:40.100060+0000","last_unstale":"2024-04-06T13:45:41.775498+0000","last_undegraded":"2024-04-06T13:45:41.775498+0000","last_fullsized":"2024-04-06T13:45:41.775498+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:06:05.815120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774972+0000","last_change":"2024-04-06T13:45:40.094182+0000","last_active":"2024-04-06T13:45:41.774972+0000","last_peered":"2024-04-06T13:45:41.774972+0000","last_clean":"2024-04-06T13:45:41.774972+0000","last_became_active":"2024-04-06T13:45:40.093003+0000","last_became_peered":"2024-04-06T13:45:40.093003+0000","last_unstale":"2024-04-06T13:45:41.774972+0000","last_undegraded":"2024-04-06T13:45:41.774972+0000","last_fullsized":"2024-04-06T13:45:41.774972+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:41:55.909045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775160+0000","last_change":"2024-04-06T13:45:40.091827+0000","last_active":"2024-04-06T13:45:41.775160+0000","last_peered":"2024-04-06T13:45:41.775160+0000","last_clean":"2024-04-06T13:45:41.775160+0000","last_became_active":"2024-04-06T13:45:40.091564+0000","last_became_peered":"2024-04-06T13:45:40.091564+0000","last_unstale":"2024-04-06T13:45:41.775160+0000","last_undegraded":"2024-04-06T13:45:41.775160+0000","last_fullsized":"2024-04-06T13:45:41.775160+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:24:57.492435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085481+0000","last_change":"2024-04-06T13:45:40.102544+0000","last_active":"2024-04-06T13:45:41.085481+0000","last_peered":"2024-04-06T13:45:41.085481+0000","last_clean":"2024-04-06T13:45:41.085481+0000","last_became_active":"2024-04-06T13:45:40.102295+0000","last_became_peered":"2024-04-06T13:45:40.102295+0000","last_unstale":"2024-04-06T13:45:41.085481+0000","last_undegraded":"2024-04-06T13:45:41.085481+0000","last_fullsized":"2024-04-06T13:45:41.085481+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:16:21.249070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085498+0000","last_change":"2024-04-06T13:45:40.105204+0000","last_active":"2024-04-06T13:45:41.085498+0000","last_peered":"2024-04-06T13:45:41.085498+0000","last_clean":"2024-04-06T13:45:41.085498+0000","last_became_active":"2024-04-06T13:45:40.104960+0000","last_became_peered":"2024-04-06T13:45:40.104960+0000","last_unstale":"2024-04-06T13:45:41.085498+0000","last_undegraded":"2024-04-06T13:45:41.085498+0000","last_fullsized":"2024-04-06T13:45:41.085498+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:10:28.482639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774357+0000","last_change":"2024-04-06T13:45:40.094696+0000","last_active":"2024-04-06T13:45:41.774357+0000","last_peered":"2024-04-06T13:45:41.774357+0000","last_clean":"2024-04-06T13:45:41.774357+0000","last_became_active":"2024-04-06T13:45:40.093318+0000","last_became_peered":"2024-04-06T13:45:40.093318+0000","last_unstale":"2024-04-06T13:45:41.774357+0000","last_undegraded":"2024-04-06T13:45:41.774357+0000","last_fullsized":"2024-04-06T13:45:41.774357+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:42:02.447546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.084671+0000","last_change":"2024-04-06T13:45:40.091160+0000","last_active":"2024-04-06T13:45:41.084671+0000","last_peered":"2024-04-06T13:45:41.084671+0000","last_clean":"2024-04-06T13:45:41.084671+0000","last_became_active":"2024-04-06T13:45:40.090702+0000","last_became_peered":"2024-04-06T13:45:40.090702+0000","last_unstale":"2024-04-06T13:45:41.084671+0000","last_undegraded":"2024-04-06T13:45:41.084671+0000","last_fullsized":"2024-04-06T13:45:41.084671+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:26:02.582066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775556+0000","last_change":"2024-04-06T13:45:40.100626+0000","last_active":"2024-04-06T13:45:41.775556+0000","last_peered":"2024-04-06T13:45:41.775556+0000","last_clean":"2024-04-06T13:45:41.775556+0000","last_became_active":"2024-04-06T13:45:40.096229+0000","last_became_peered":"2024-04-06T13:45:40.096229+0000","last_unstale":"2024-04-06T13:45:41.775556+0000","last_undegraded":"2024-04-06T13:45:41.775556+0000","last_fullsized":"2024-04-06T13:45:41.775556+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:17:31.461140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085931+0000","last_change":"2024-04-06T13:45:40.102825+0000","last_active":"2024-04-06T13:45:41.085931+0000","last_peered":"2024-04-06T13:45:41.085931+0000","last_clean":"2024-04-06T13:45:41.085931+0000","last_became_active":"2024-04-06T13:45:40.102315+0000","last_became_peered":"2024-04-06T13:45:40.102315+0000","last_unstale":"2024-04-06T13:45:41.085931+0000","last_undegraded":"2024-04-06T13:45:41.085931+0000","last_fullsized":"2024-04-06T13:45:41.085931+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:30:54.881116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775100+0000","last_change":"2024-04-06T13:45:40.096421+0000","last_active":"2024-04-06T13:45:41.775100+0000","last_peered":"2024-04-06T13:45:41.775100+0000","last_clean":"2024-04-06T13:45:41.775100+0000","last_became_active":"2024-04-06T13:45:40.096061+0000","last_became_peered":"2024-04-06T13:45:40.096061+0000","last_unstale":"2024-04-06T13:45:41.775100+0000","last_undegraded":"2024-04-06T13:45:41.775100+0000","last_fullsized":"2024-04-06T13:45:41.775100+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:42:18.183369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775216+0000","last_change":"2024-04-06T13:45:40.088645+0000","last_active":"2024-04-06T13:45:41.775216+0000","last_peered":"2024-04-06T13:45:41.775216+0000","last_clean":"2024-04-06T13:45:41.775216+0000","last_became_active":"2024-04-06T13:45:40.088410+0000","last_became_peered":"2024-04-06T13:45:40.088410+0000","last_unstale":"2024-04-06T13:45:41.775216+0000","last_undegraded":"2024-04-06T13:45:41.775216+0000","last_fullsized":"2024-04-06T13:45:41.775216+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:05:37.331539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086466+0000","last_change":"2024-04-06T13:45:40.083080+0000","last_active":"2024-04-06T13:45:41.086466+0000","last_peered":"2024-04-06T13:45:41.086466+0000","last_clean":"2024-04-06T13:45:41.086466+0000","last_became_active":"2024-04-06T13:45:40.082696+0000","last_became_peered":"2024-04-06T13:45:40.082696+0000","last_unstale":"2024-04-06T13:45:41.086466+0000","last_undegraded":"2024-04-06T13:45:41.086466+0000","last_fullsized":"2024-04-06T13:45:41.086466+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:49:57.256838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774711+0000","last_change":"2024-04-06T13:45:40.094634+0000","last_active":"2024-04-06T13:45:41.774711+0000","last_peered":"2024-04-06T13:45:41.774711+0000","last_clean":"2024-04-06T13:45:41.774711+0000","last_became_active":"2024-04-06T13:45:40.093724+0000","last_became_peered":"2024-04-06T13:45:40.093724+0000","last_unstale":"2024-04-06T13:45:41.774711+0000","last_undegraded":"2024-04-06T13:45:41.774711+0000","last_fullsized":"2024-04-06T13:45:41.774711+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:25:25.061230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775300+0000","last_change":"2024-04-06T13:45:40.089229+0000","last_active":"2024-04-06T13:45:41.775300+0000","last_peered":"2024-04-06T13:45:41.775300+0000","last_clean":"2024-04-06T13:45:41.775300+0000","last_became_active":"2024-04-06T13:45:40.087921+0000","last_became_peered":"2024-04-06T13:45:40.087921+0000","last_unstale":"2024-04-06T13:45:41.775300+0000","last_undegraded":"2024-04-06T13:45:41.775300+0000","last_fullsized":"2024-04-06T13:45:41.775300+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:51:32.416220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085426+0000","last_change":"2024-04-06T13:45:40.103933+0000","last_active":"2024-04-06T13:45:41.085426+0000","last_peered":"2024-04-06T13:45:41.085426+0000","last_clean":"2024-04-06T13:45:41.085426+0000","last_became_active":"2024-04-06T13:45:40.103694+0000","last_became_peered":"2024-04-06T13:45:40.103694+0000","last_unstale":"2024-04-06T13:45:41.085426+0000","last_undegraded":"2024-04-06T13:45:41.085426+0000","last_fullsized":"2024-04-06T13:45:41.085426+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:07:25.834350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085952+0000","last_change":"2024-04-06T13:45:40.099989+0000","last_active":"2024-04-06T13:45:41.085952+0000","last_peered":"2024-04-06T13:45:41.085952+0000","last_clean":"2024-04-06T13:45:41.085952+0000","last_became_active":"2024-04-06T13:45:40.099736+0000","last_became_peered":"2024-04-06T13:45:40.099736+0000","last_unstale":"2024-04-06T13:45:41.085952+0000","last_undegraded":"2024-04-06T13:45:41.085952+0000","last_fullsized":"2024-04-06T13:45:41.085952+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:58:36.474241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086796+0000","last_change":"2024-04-06T13:45:40.087268+0000","last_active":"2024-04-06T13:45:41.086796+0000","last_peered":"2024-04-06T13:45:41.086796+0000","last_clean":"2024-04-06T13:45:41.086796+0000","last_became_active":"2024-04-06T13:45:40.086959+0000","last_became_peered":"2024-04-06T13:45:40.086959+0000","last_unstale":"2024-04-06T13:45:41.086796+0000","last_undegraded":"2024-04-06T13:45:41.086796+0000","last_fullsized":"2024-04-06T13:45:41.086796+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:18:59.672479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086720+0000","last_change":"2024-04-06T13:45:40.092669+0000","last_active":"2024-04-06T13:45:41.086720+0000","last_peered":"2024-04-06T13:45:41.086720+0000","last_clean":"2024-04-06T13:45:41.086720+0000","last_became_active":"2024-04-06T13:45:40.092467+0000","last_became_peered":"2024-04-06T13:45:40.092467+0000","last_unstale":"2024-04-06T13:45:41.086720+0000","last_undegraded":"2024-04-06T13:45:41.086720+0000","last_fullsized":"2024-04-06T13:45:41.086720+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:25:14.399262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085462+0000","last_change":"2024-04-06T13:45:40.100609+0000","last_active":"2024-04-06T13:45:41.085462+0000","last_peered":"2024-04-06T13:45:41.085462+0000","last_clean":"2024-04-06T13:45:41.085462+0000","last_became_active":"2024-04-06T13:45:40.100358+0000","last_became_peered":"2024-04-06T13:45:40.100358+0000","last_unstale":"2024-04-06T13:45:41.085462+0000","last_undegraded":"2024-04-06T13:45:41.085462+0000","last_fullsized":"2024-04-06T13:45:41.085462+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:50:14.349390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775744+0000","last_change":"2024-04-06T13:45:40.087005+0000","last_active":"2024-04-06T13:45:41.775744+0000","last_peered":"2024-04-06T13:45:41.775744+0000","last_clean":"2024-04-06T13:45:41.775744+0000","last_became_active":"2024-04-06T13:45:40.086732+0000","last_became_peered":"2024-04-06T13:45:40.086732+0000","last_unstale":"2024-04-06T13:45:41.775744+0000","last_undegraded":"2024-04-06T13:45:41.775744+0000","last_fullsized":"2024-04-06T13:45:41.775744+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:38:59.616950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774728+0000","last_change":"2024-04-06T13:45:40.087356+0000","last_active":"2024-04-06T13:45:41.774728+0000","last_peered":"2024-04-06T13:45:41.774728+0000","last_clean":"2024-04-06T13:45:41.774728+0000","last_became_active":"2024-04-06T13:45:40.087103+0000","last_became_peered":"2024-04-06T13:45:40.087103+0000","last_unstale":"2024-04-06T13:45:41.774728+0000","last_undegraded":"2024-04-06T13:45:41.774728+0000","last_fullsized":"2024-04-06T13:45:41.774728+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:10:26.253958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.084551+0000","last_change":"2024-04-06T13:45:40.090129+0000","last_active":"2024-04-06T13:45:41.084551+0000","last_peered":"2024-04-06T13:45:41.084551+0000","last_clean":"2024-04-06T13:45:41.084551+0000","last_became_active":"2024-04-06T13:45:40.089740+0000","last_became_peered":"2024-04-06T13:45:40.089740+0000","last_unstale":"2024-04-06T13:45:41.084551+0000","last_undegraded":"2024-04-06T13:45:41.084551+0000","last_fullsized":"2024-04-06T13:45:41.084551+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:04:59.118082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774369+0000","last_change":"2024-04-06T13:45:40.089599+0000","last_active":"2024-04-06T13:45:41.774369+0000","last_peered":"2024-04-06T13:45:41.774369+0000","last_clean":"2024-04-06T13:45:41.774369+0000","last_became_active":"2024-04-06T13:45:40.089194+0000","last_became_peered":"2024-04-06T13:45:40.089194+0000","last_unstale":"2024-04-06T13:45:41.774369+0000","last_undegraded":"2024-04-06T13:45:41.774369+0000","last_fullsized":"2024-04-06T13:45:41.774369+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:01:22.608442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774821+0000","last_change":"2024-04-06T13:45:40.097563+0000","last_active":"2024-04-06T13:45:41.774821+0000","last_peered":"2024-04-06T13:45:41.774821+0000","last_clean":"2024-04-06T13:45:41.774821+0000","last_became_active":"2024-04-06T13:45:40.095657+0000","last_became_peered":"2024-04-06T13:45:40.095657+0000","last_unstale":"2024-04-06T13:45:41.774821+0000","last_undegraded":"2024-04-06T13:45:41.774821+0000","last_fullsized":"2024-04-06T13:45:41.774821+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:29:31.066605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775323+0000","last_change":"2024-04-06T13:45:40.100417+0000","last_active":"2024-04-06T13:45:41.775323+0000","last_peered":"2024-04-06T13:45:41.775323+0000","last_clean":"2024-04-06T13:45:41.775323+0000","last_became_active":"2024-04-06T13:45:40.100105+0000","last_became_peered":"2024-04-06T13:45:40.100105+0000","last_unstale":"2024-04-06T13:45:41.775323+0000","last_undegraded":"2024-04-06T13:45:41.775323+0000","last_fullsized":"2024-04-06T13:45:41.775323+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:55:17.346196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.084770+0000","last_change":"2024-04-06T13:45:40.089311+0000","last_active":"2024-04-06T13:45:41.084770+0000","last_peered":"2024-04-06T13:45:41.084770+0000","last_clean":"2024-04-06T13:45:41.084770+0000","last_became_active":"2024-04-06T13:45:40.088982+0000","last_became_peered":"2024-04-06T13:45:40.088982+0000","last_unstale":"2024-04-06T13:45:41.084770+0000","last_undegraded":"2024-04-06T13:45:41.084770+0000","last_fullsized":"2024-04-06T13:45:41.084770+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:23:00.349095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085572+0000","last_change":"2024-04-06T13:45:40.103916+0000","last_active":"2024-04-06T13:45:41.085572+0000","last_peered":"2024-04-06T13:45:41.085572+0000","last_clean":"2024-04-06T13:45:41.085572+0000","last_became_active":"2024-04-06T13:45:40.103733+0000","last_became_peered":"2024-04-06T13:45:40.103733+0000","last_unstale":"2024-04-06T13:45:41.085572+0000","last_undegraded":"2024-04-06T13:45:41.085572+0000","last_fullsized":"2024-04-06T13:45:41.085572+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:44.542777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774986+0000","last_change":"2024-04-06T13:45:40.101201+0000","last_active":"2024-04-06T13:45:41.774986+0000","last_peered":"2024-04-06T13:45:41.774986+0000","last_clean":"2024-04-06T13:45:41.774986+0000","last_became_active":"2024-04-06T13:45:40.100774+0000","last_became_peered":"2024-04-06T13:45:40.100774+0000","last_unstale":"2024-04-06T13:45:41.774986+0000","last_undegraded":"2024-04-06T13:45:41.774986+0000","last_fullsized":"2024-04-06T13:45:41.774986+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:20:45.306030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775602+0000","last_change":"2024-04-06T13:45:40.092867+0000","last_active":"2024-04-06T13:45:41.775602+0000","last_peered":"2024-04-06T13:45:41.775602+0000","last_clean":"2024-04-06T13:45:41.775602+0000","last_became_active":"2024-04-06T13:45:40.092514+0000","last_became_peered":"2024-04-06T13:45:40.092514+0000","last_unstale":"2024-04-06T13:45:41.775602+0000","last_undegraded":"2024-04-06T13:45:41.775602+0000","last_fullsized":"2024-04-06T13:45:41.775602+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:12.970828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774546+0000","last_change":"2024-04-06T13:45:40.093371+0000","last_active":"2024-04-06T13:45:41.774546+0000","last_peered":"2024-04-06T13:45:41.774546+0000","last_clean":"2024-04-06T13:45:41.774546+0000","last_became_active":"2024-04-06T13:45:40.093120+0000","last_became_peered":"2024-04-06T13:45:40.093120+0000","last_unstale":"2024-04-06T13:45:41.774546+0000","last_undegraded":"2024-04-06T13:45:41.774546+0000","last_fullsized":"2024-04-06T13:45:41.774546+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:05:09.968925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085510+0000","last_change":"2024-04-06T13:45:40.099053+0000","last_active":"2024-04-06T13:45:41.085510+0000","last_peered":"2024-04-06T13:45:41.085510+0000","last_clean":"2024-04-06T13:45:41.085510+0000","last_became_active":"2024-04-06T13:45:40.098844+0000","last_became_peered":"2024-04-06T13:45:40.098844+0000","last_unstale":"2024-04-06T13:45:41.085510+0000","last_undegraded":"2024-04-06T13:45:41.085510+0000","last_fullsized":"2024-04-06T13:45:41.085510+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:55:35.985540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.084929+0000","last_change":"2024-04-06T13:45:40.091890+0000","last_active":"2024-04-06T13:45:41.084929+0000","last_peered":"2024-04-06T13:45:41.084929+0000","last_clean":"2024-04-06T13:45:41.084929+0000","last_became_active":"2024-04-06T13:45:40.091562+0000","last_became_peered":"2024-04-06T13:45:40.091562+0000","last_unstale":"2024-04-06T13:45:41.084929+0000","last_undegraded":"2024-04-06T13:45:41.084929+0000","last_fullsized":"2024-04-06T13:45:41.084929+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:54:01.774660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"208'4","reported_seq":15,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774901+0000","last_change":"2024-04-06T13:45:39.672115+0000","last_active":"2024-04-06T13:45:41.774901+0000","last_peered":"2024-04-06T13:45:41.774901+0000","last_clean":"2024-04-06T13:45:41.774901+0000","last_became_active":"2024-04-06T13:45:39.671638+0000","last_became_peered":"2024-04-06T13:45:39.671638+0000","last_unstale":"2024-04-06T13:45:41.774901+0000","last_undegraded":"2024-04-06T13:45:41.774901+0000","last_fullsized":"2024-04-06T13:45:41.774901+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:50:03.038639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775304+0000","last_change":"2024-04-06T13:45:40.093142+0000","last_active":"2024-04-06T13:45:41.775304+0000","last_peered":"2024-04-06T13:45:41.775304+0000","last_clean":"2024-04-06T13:45:41.775304+0000","last_became_active":"2024-04-06T13:45:40.092478+0000","last_became_peered":"2024-04-06T13:45:40.092478+0000","last_unstale":"2024-04-06T13:45:41.775304+0000","last_undegraded":"2024-04-06T13:45:41.775304+0000","last_fullsized":"2024-04-06T13:45:41.775304+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:54:45.772130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"208'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086023+0000","last_change":"2024-04-06T13:45:39.680145+0000","last_active":"2024-04-06T13:45:41.086023+0000","last_peered":"2024-04-06T13:45:41.086023+0000","last_clean":"2024-04-06T13:45:41.086023+0000","last_became_active":"2024-04-06T13:45:39.679798+0000","last_became_peered":"2024-04-06T13:45:39.679798+0000","last_unstale":"2024-04-06T13:45:41.086023+0000","last_undegraded":"2024-04-06T13:45:41.086023+0000","last_fullsized":"2024-04-06T13:45:41.086023+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:58:18.377227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775361+0000","last_change":"2024-04-06T13:45:40.099397+0000","last_active":"2024-04-06T13:45:41.775361+0000","last_peered":"2024-04-06T13:45:41.775361+0000","last_clean":"2024-04-06T13:45:41.775361+0000","last_became_active":"2024-04-06T13:45:40.098838+0000","last_became_peered":"2024-04-06T13:45:40.098838+0000","last_unstale":"2024-04-06T13:45:41.775361+0000","last_undegraded":"2024-04-06T13:45:41.775361+0000","last_fullsized":"2024-04-06T13:45:41.775361+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:21:30.034786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"208'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775172+0000","last_change":"2024-04-06T13:45:39.666757+0000","last_active":"2024-04-06T13:45:41.775172+0000","last_peered":"2024-04-06T13:45:41.775172+0000","last_clean":"2024-04-06T13:45:41.775172+0000","last_became_active":"2024-04-06T13:45:39.666468+0000","last_became_peered":"2024-04-06T13:45:39.666468+0000","last_unstale":"2024-04-06T13:45:41.775172+0000","last_undegraded":"2024-04-06T13:45:41.775172+0000","last_fullsized":"2024-04-06T13:45:41.775172+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:16.205165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774820+0000","last_change":"2024-04-06T13:45:40.098735+0000","last_active":"2024-04-06T13:45:41.774820+0000","last_peered":"2024-04-06T13:45:41.774820+0000","last_clean":"2024-04-06T13:45:41.774820+0000","last_became_active":"2024-04-06T13:45:40.098454+0000","last_became_peered":"2024-04-06T13:45:40.098454+0000","last_unstale":"2024-04-06T13:45:41.774820+0000","last_undegraded":"2024-04-06T13:45:41.774820+0000","last_fullsized":"2024-04-06T13:45:41.774820+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:27:32.798856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"208'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086718+0000","last_change":"2024-04-06T13:45:39.665692+0000","last_active":"2024-04-06T13:45:41.086718+0000","last_peered":"2024-04-06T13:45:41.086718+0000","last_clean":"2024-04-06T13:45:41.086718+0000","last_became_active":"2024-04-06T13:45:39.665169+0000","last_became_peered":"2024-04-06T13:45:39.665169+0000","last_unstale":"2024-04-06T13:45:41.086718+0000","last_undegraded":"2024-04-06T13:45:41.086718+0000","last_fullsized":"2024-04-06T13:45:41.086718+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:21:29.030323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086312+0000","last_change":"2024-04-06T13:45:40.095571+0000","last_active":"2024-04-06T13:45:41.086312+0000","last_peered":"2024-04-06T13:45:41.086312+0000","last_clean":"2024-04-06T13:45:41.086312+0000","last_became_active":"2024-04-06T13:45:40.095280+0000","last_became_peered":"2024-04-06T13:45:40.095280+0000","last_unstale":"2024-04-06T13:45:41.086312+0000","last_undegraded":"2024-04-06T13:45:41.086312+0000","last_fullsized":"2024-04-06T13:45:41.086312+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:07:36.838483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"208'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085642+0000","last_change":"2024-04-06T13:45:39.679077+0000","last_active":"2024-04-06T13:45:41.085642+0000","last_peered":"2024-04-06T13:45:41.085642+0000","last_clean":"2024-04-06T13:45:41.085642+0000","last_became_active":"2024-04-06T13:45:39.678763+0000","last_became_peered":"2024-04-06T13:45:39.678763+0000","last_unstale":"2024-04-06T13:45:41.085642+0000","last_undegraded":"2024-04-06T13:45:41.085642+0000","last_fullsized":"2024-04-06T13:45:41.085642+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:23:07.225232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775395+0000","last_change":"2024-04-06T13:45:40.095694+0000","last_active":"2024-04-06T13:45:41.775395+0000","last_peered":"2024-04-06T13:45:41.775395+0000","last_clean":"2024-04-06T13:45:41.775395+0000","last_became_active":"2024-04-06T13:45:40.095463+0000","last_became_peered":"2024-04-06T13:45:40.095463+0000","last_unstale":"2024-04-06T13:45:41.775395+0000","last_undegraded":"2024-04-06T13:45:41.775395+0000","last_fullsized":"2024-04-06T13:45:41.775395+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:47:54.692873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"208'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774772+0000","last_change":"2024-04-06T13:45:39.681133+0000","last_active":"2024-04-06T13:45:41.774772+0000","last_peered":"2024-04-06T13:45:41.774772+0000","last_clean":"2024-04-06T13:45:41.774772+0000","last_became_active":"2024-04-06T13:45:39.680833+0000","last_became_peered":"2024-04-06T13:45:39.680833+0000","last_unstale":"2024-04-06T13:45:41.774772+0000","last_undegraded":"2024-04-06T13:45:41.774772+0000","last_fullsized":"2024-04-06T13:45:41.774772+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:22:59.344632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774822+0000","last_change":"2024-04-06T13:45:40.090838+0000","last_active":"2024-04-06T13:45:41.774822+0000","last_peered":"2024-04-06T13:45:41.774822+0000","last_clean":"2024-04-06T13:45:41.774822+0000","last_became_active":"2024-04-06T13:45:40.090616+0000","last_became_peered":"2024-04-06T13:45:40.090616+0000","last_unstale":"2024-04-06T13:45:41.774822+0000","last_undegraded":"2024-04-06T13:45:41.774822+0000","last_fullsized":"2024-04-06T13:45:41.774822+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:16.638956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086513+0000","last_change":"2024-04-06T13:45:39.661817+0000","last_active":"2024-04-06T13:45:41.086513+0000","last_peered":"2024-04-06T13:45:41.086513+0000","last_clean":"2024-04-06T13:45:41.086513+0000","last_became_active":"2024-04-06T13:45:39.661528+0000","last_became_peered":"2024-04-06T13:45:39.661528+0000","last_unstale":"2024-04-06T13:45:41.086513+0000","last_undegraded":"2024-04-06T13:45:41.086513+0000","last_fullsized":"2024-04-06T13:45:41.086513+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:23:08.710841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085030+0000","last_change":"2024-04-06T13:45:40.088375+0000","last_active":"2024-04-06T13:45:41.085030+0000","last_peered":"2024-04-06T13:45:41.085030+0000","last_clean":"2024-04-06T13:45:41.085030+0000","last_became_active":"2024-04-06T13:45:40.088114+0000","last_became_peered":"2024-04-06T13:45:40.088114+0000","last_unstale":"2024-04-06T13:45:41.085030+0000","last_undegraded":"2024-04-06T13:45:41.085030+0000","last_fullsized":"2024-04-06T13:45:41.085030+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:49:48.207025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086058+0000","last_change":"2024-04-06T13:45:39.680180+0000","last_active":"2024-04-06T13:45:41.086058+0000","last_peered":"2024-04-06T13:45:41.086058+0000","last_clean":"2024-04-06T13:45:41.086058+0000","last_became_active":"2024-04-06T13:45:39.679894+0000","last_became_peered":"2024-04-06T13:45:39.679894+0000","last_unstale":"2024-04-06T13:45:41.086058+0000","last_undegraded":"2024-04-06T13:45:41.086058+0000","last_fullsized":"2024-04-06T13:45:41.086058+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:05:36.327076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085766+0000","last_change":"2024-04-06T13:45:40.101167+0000","last_active":"2024-04-06T13:45:41.085766+0000","last_peered":"2024-04-06T13:45:41.085766+0000","last_clean":"2024-04-06T13:45:41.085766+0000","last_became_active":"2024-04-06T13:45:40.100847+0000","last_became_peered":"2024-04-06T13:45:40.100847+0000","last_unstale":"2024-04-06T13:45:41.085766+0000","last_undegraded":"2024-04-06T13:45:41.085766+0000","last_fullsized":"2024-04-06T13:45:41.085766+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:13:50.892939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775663+0000","last_change":"2024-04-06T13:45:39.681395+0000","last_active":"2024-04-06T13:45:41.775663+0000","last_peered":"2024-04-06T13:45:41.775663+0000","last_clean":"2024-04-06T13:45:41.775663+0000","last_became_active":"2024-04-06T13:45:39.681107+0000","last_became_peered":"2024-04-06T13:45:39.681107+0000","last_unstale":"2024-04-06T13:45:41.775663+0000","last_undegraded":"2024-04-06T13:45:41.775663+0000","last_fullsized":"2024-04-06T13:45:41.775663+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:02:47.403764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086852+0000","last_change":"2024-04-06T13:45:40.097418+0000","last_active":"2024-04-06T13:45:41.086852+0000","last_peered":"2024-04-06T13:45:41.086852+0000","last_clean":"2024-04-06T13:45:41.086852+0000","last_became_active":"2024-04-06T13:45:40.097229+0000","last_became_peered":"2024-04-06T13:45:40.097229+0000","last_unstale":"2024-04-06T13:45:41.086852+0000","last_undegraded":"2024-04-06T13:45:41.086852+0000","last_fullsized":"2024-04-06T13:45:41.086852+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:59:03.379200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"208'3","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774740+0000","last_change":"2024-04-06T13:45:39.681177+0000","last_active":"2024-04-06T13:45:41.774740+0000","last_peered":"2024-04-06T13:45:41.774740+0000","last_clean":"2024-04-06T13:45:41.774740+0000","last_became_active":"2024-04-06T13:45:39.680881+0000","last_became_peered":"2024-04-06T13:45:39.680881+0000","last_unstale":"2024-04-06T13:45:41.774740+0000","last_undegraded":"2024-04-06T13:45:41.774740+0000","last_fullsized":"2024-04-06T13:45:41.774740+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:21:12.767786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775610+0000","last_change":"2024-04-06T13:45:40.101751+0000","last_active":"2024-04-06T13:45:41.775610+0000","last_peered":"2024-04-06T13:45:41.775610+0000","last_clean":"2024-04-06T13:45:41.775610+0000","last_became_active":"2024-04-06T13:45:40.101491+0000","last_became_peered":"2024-04-06T13:45:40.101491+0000","last_unstale":"2024-04-06T13:45:41.775610+0000","last_undegraded":"2024-04-06T13:45:41.775610+0000","last_fullsized":"2024-04-06T13:45:41.775610+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:20:52.927579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"208'6","reported_seq":17,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086795+0000","last_change":"2024-04-06T13:45:39.667906+0000","last_active":"2024-04-06T13:45:41.086795+0000","last_peered":"2024-04-06T13:45:41.086795+0000","last_clean":"2024-04-06T13:45:41.086795+0000","last_became_active":"2024-04-06T13:45:39.667632+0000","last_became_peered":"2024-04-06T13:45:39.667632+0000","last_unstale":"2024-04-06T13:45:41.086795+0000","last_undegraded":"2024-04-06T13:45:41.086795+0000","last_fullsized":"2024-04-06T13:45:41.086795+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:05:01.859909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774233+0000","last_change":"2024-04-06T13:45:40.087438+0000","last_active":"2024-04-06T13:45:41.774233+0000","last_peered":"2024-04-06T13:45:41.774233+0000","last_clean":"2024-04-06T13:45:41.774233+0000","last_became_active":"2024-04-06T13:45:40.087173+0000","last_became_peered":"2024-04-06T13:45:40.087173+0000","last_unstale":"2024-04-06T13:45:41.774233+0000","last_undegraded":"2024-04-06T13:45:41.774233+0000","last_fullsized":"2024-04-06T13:45:41.774233+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:17.209628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"208'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086590+0000","last_change":"2024-04-06T13:45:39.665242+0000","last_active":"2024-04-06T13:45:41.086590+0000","last_peered":"2024-04-06T13:45:41.086590+0000","last_clean":"2024-04-06T13:45:41.086590+0000","last_became_active":"2024-04-06T13:45:39.664913+0000","last_became_peered":"2024-04-06T13:45:39.664913+0000","last_unstale":"2024-04-06T13:45:41.086590+0000","last_undegraded":"2024-04-06T13:45:41.086590+0000","last_fullsized":"2024-04-06T13:45:41.086590+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:02:56.300763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085383+0000","last_change":"2024-04-06T13:45:40.098012+0000","last_active":"2024-04-06T13:45:41.085383+0000","last_peered":"2024-04-06T13:45:41.085383+0000","last_clean":"2024-04-06T13:45:41.085383+0000","last_became_active":"2024-04-06T13:45:40.097768+0000","last_became_peered":"2024-04-06T13:45:40.097768+0000","last_unstale":"2024-04-06T13:45:41.085383+0000","last_undegraded":"2024-04-06T13:45:41.085383+0000","last_fullsized":"2024-04-06T13:45:41.085383+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:51:11.592246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775005+0000","last_change":"2024-04-06T13:45:39.675786+0000","last_active":"2024-04-06T13:45:41.775005+0000","last_peered":"2024-04-06T13:45:41.775005+0000","last_clean":"2024-04-06T13:45:41.775005+0000","last_became_active":"2024-04-06T13:45:39.675545+0000","last_became_peered":"2024-04-06T13:45:39.675545+0000","last_unstale":"2024-04-06T13:45:41.775005+0000","last_undegraded":"2024-04-06T13:45:41.775005+0000","last_fullsized":"2024-04-06T13:45:41.775005+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:55:16.341733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086561+0000","last_change":"2024-04-06T13:45:40.085958+0000","last_active":"2024-04-06T13:45:41.086561+0000","last_peered":"2024-04-06T13:45:41.086561+0000","last_clean":"2024-04-06T13:45:41.086561+0000","last_became_active":"2024-04-06T13:45:40.085677+0000","last_became_peered":"2024-04-06T13:45:40.085677+0000","last_unstale":"2024-04-06T13:45:41.086561+0000","last_undegraded":"2024-04-06T13:45:41.086561+0000","last_fullsized":"2024-04-06T13:45:41.086561+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:58:46.123989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774435+0000","last_change":"2024-04-06T13:45:39.667680+0000","last_active":"2024-04-06T13:45:41.774435+0000","last_peered":"2024-04-06T13:45:41.774435+0000","last_clean":"2024-04-06T13:45:41.774435+0000","last_became_active":"2024-04-06T13:45:39.666889+0000","last_became_peered":"2024-04-06T13:45:39.666889+0000","last_unstale":"2024-04-06T13:45:41.774435+0000","last_undegraded":"2024-04-06T13:45:41.774435+0000","last_fullsized":"2024-04-06T13:45:41.774435+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:01:09.108164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086543+0000","last_change":"2024-04-06T13:45:40.088259+0000","last_active":"2024-04-06T13:45:41.086543+0000","last_peered":"2024-04-06T13:45:41.086543+0000","last_clean":"2024-04-06T13:45:41.086543+0000","last_became_active":"2024-04-06T13:45:40.087981+0000","last_became_peered":"2024-04-06T13:45:40.087981+0000","last_unstale":"2024-04-06T13:45:41.086543+0000","last_undegraded":"2024-04-06T13:45:41.086543+0000","last_fullsized":"2024-04-06T13:45:41.086543+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:04:59.810704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"208'8","reported_seq":19,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774832+0000","last_change":"2024-04-06T13:45:39.671256+0000","last_active":"2024-04-06T13:45:41.774832+0000","last_peered":"2024-04-06T13:45:41.774832+0000","last_clean":"2024-04-06T13:45:41.774832+0000","last_became_active":"2024-04-06T13:45:39.670957+0000","last_became_peered":"2024-04-06T13:45:39.670957+0000","last_unstale":"2024-04-06T13:45:41.774832+0000","last_undegraded":"2024-04-06T13:45:41.774832+0000","last_fullsized":"2024-04-06T13:45:41.774832+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:05:08.964462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775059+0000","last_change":"2024-04-06T13:45:40.093107+0000","last_active":"2024-04-06T13:45:41.775059+0000","last_peered":"2024-04-06T13:45:41.775059+0000","last_clean":"2024-04-06T13:45:41.775059+0000","last_became_active":"2024-04-06T13:45:40.092815+0000","last_became_peered":"2024-04-06T13:45:40.092815+0000","last_unstale":"2024-04-06T13:45:41.775059+0000","last_undegraded":"2024-04-06T13:45:41.775059+0000","last_fullsized":"2024-04-06T13:45:41.775059+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:35:13.586257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"208'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775404+0000","last_change":"2024-04-06T13:45:39.669733+0000","last_active":"2024-04-06T13:45:41.775404+0000","last_peered":"2024-04-06T13:45:41.775404+0000","last_clean":"2024-04-06T13:45:41.775404+0000","last_became_active":"2024-04-06T13:45:39.669394+0000","last_became_peered":"2024-04-06T13:45:39.669394+0000","last_unstale":"2024-04-06T13:45:41.775404+0000","last_undegraded":"2024-04-06T13:45:41.775404+0000","last_fullsized":"2024-04-06T13:45:41.775404+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:36:00.946626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085797+0000","last_change":"2024-04-06T13:45:40.098243+0000","last_active":"2024-04-06T13:45:41.085797+0000","last_peered":"2024-04-06T13:45:41.085797+0000","last_clean":"2024-04-06T13:45:41.085797+0000","last_became_active":"2024-04-06T13:45:40.097989+0000","last_became_peered":"2024-04-06T13:45:40.097989+0000","last_unstale":"2024-04-06T13:45:41.085797+0000","last_undegraded":"2024-04-06T13:45:41.085797+0000","last_fullsized":"2024-04-06T13:45:41.085797+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:31:51.711889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"208'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775503+0000","last_change":"2024-04-06T13:45:39.674397+0000","last_active":"2024-04-06T13:45:41.775503+0000","last_peered":"2024-04-06T13:45:41.775503+0000","last_clean":"2024-04-06T13:45:41.775503+0000","last_became_active":"2024-04-06T13:45:39.674055+0000","last_became_peered":"2024-04-06T13:45:39.674055+0000","last_unstale":"2024-04-06T13:45:41.775503+0000","last_undegraded":"2024-04-06T13:45:41.775503+0000","last_fullsized":"2024-04-06T13:45:41.775503+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:04:58.113619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775432+0000","last_change":"2024-04-06T13:45:40.100362+0000","last_active":"2024-04-06T13:45:41.775432+0000","last_peered":"2024-04-06T13:45:41.775432+0000","last_clean":"2024-04-06T13:45:41.775432+0000","last_became_active":"2024-04-06T13:45:40.100081+0000","last_became_peered":"2024-04-06T13:45:40.100081+0000","last_unstale":"2024-04-06T13:45:41.775432+0000","last_undegraded":"2024-04-06T13:45:41.775432+0000","last_fullsized":"2024-04-06T13:45:41.775432+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:02:48.408226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"208'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775065+0000","last_change":"2024-04-06T13:45:39.680555+0000","last_active":"2024-04-06T13:45:41.775065+0000","last_peered":"2024-04-06T13:45:41.775065+0000","last_clean":"2024-04-06T13:45:41.775065+0000","last_became_active":"2024-04-06T13:45:39.680250+0000","last_became_peered":"2024-04-06T13:45:39.680250+0000","last_unstale":"2024-04-06T13:45:41.775065+0000","last_undegraded":"2024-04-06T13:45:41.775065+0000","last_fullsized":"2024-04-06T13:45:41.775065+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:25.490671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775676+0000","last_change":"2024-04-06T13:45:40.094003+0000","last_active":"2024-04-06T13:45:41.775676+0000","last_peered":"2024-04-06T13:45:41.775676+0000","last_clean":"2024-04-06T13:45:41.775676+0000","last_became_active":"2024-04-06T13:45:40.093766+0000","last_became_peered":"2024-04-06T13:45:40.093766+0000","last_unstale":"2024-04-06T13:45:41.775676+0000","last_undegraded":"2024-04-06T13:45:41.775676+0000","last_fullsized":"2024-04-06T13:45:41.775676+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:30:37.037455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775145+0000","last_change":"2024-04-06T13:45:39.670822+0000","last_active":"2024-04-06T13:45:41.775145+0000","last_peered":"2024-04-06T13:45:41.775145+0000","last_clean":"2024-04-06T13:45:41.775145+0000","last_became_active":"2024-04-06T13:45:39.670564+0000","last_became_peered":"2024-04-06T13:45:39.670564+0000","last_unstale":"2024-04-06T13:45:41.775145+0000","last_undegraded":"2024-04-06T13:45:41.775145+0000","last_fullsized":"2024-04-06T13:45:41.775145+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:42:01.443083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774476+0000","last_change":"2024-04-06T13:45:40.094718+0000","last_active":"2024-04-06T13:45:41.774476+0000","last_peered":"2024-04-06T13:45:41.774476+0000","last_clean":"2024-04-06T13:45:41.774476+0000","last_became_active":"2024-04-06T13:45:40.094486+0000","last_became_peered":"2024-04-06T13:45:40.094486+0000","last_unstale":"2024-04-06T13:45:41.774476+0000","last_undegraded":"2024-04-06T13:45:41.774476+0000","last_fullsized":"2024-04-06T13:45:41.774476+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:23:00.349095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"208'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086661+0000","last_change":"2024-04-06T13:45:39.679842+0000","last_active":"2024-04-06T13:45:41.086661+0000","last_peered":"2024-04-06T13:45:41.086661+0000","last_clean":"2024-04-06T13:45:41.086661+0000","last_became_active":"2024-04-06T13:45:39.679543+0000","last_became_peered":"2024-04-06T13:45:39.679543+0000","last_unstale":"2024-04-06T13:45:41.086661+0000","last_undegraded":"2024-04-06T13:45:41.086661+0000","last_fullsized":"2024-04-06T13:45:41.086661+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:11:58.314869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774371+0000","last_change":"2024-04-06T13:45:40.091910+0000","last_active":"2024-04-06T13:45:41.774371+0000","last_peered":"2024-04-06T13:45:41.774371+0000","last_clean":"2024-04-06T13:45:41.774371+0000","last_became_active":"2024-04-06T13:45:40.091677+0000","last_became_peered":"2024-04-06T13:45:40.091677+0000","last_unstale":"2024-04-06T13:45:41.774371+0000","last_undegraded":"2024-04-06T13:45:41.774371+0000","last_fullsized":"2024-04-06T13:45:41.774371+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:11:43.111082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774856+0000","last_change":"2024-04-06T13:45:39.672126+0000","last_active":"2024-04-06T13:45:41.774856+0000","last_peered":"2024-04-06T13:45:41.774856+0000","last_clean":"2024-04-06T13:45:41.774856+0000","last_became_active":"2024-04-06T13:45:39.671829+0000","last_became_peered":"2024-04-06T13:45:39.671829+0000","last_unstale":"2024-04-06T13:45:41.774856+0000","last_undegraded":"2024-04-06T13:45:41.774856+0000","last_fullsized":"2024-04-06T13:45:41.774856+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:03:24.392911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774624+0000","last_change":"2024-04-06T13:45:40.092393+0000","last_active":"2024-04-06T13:45:41.774624+0000","last_peered":"2024-04-06T13:45:41.774624+0000","last_clean":"2024-04-06T13:45:41.774624+0000","last_became_active":"2024-04-06T13:45:40.091375+0000","last_became_peered":"2024-04-06T13:45:40.091375+0000","last_unstale":"2024-04-06T13:45:41.774624+0000","last_undegraded":"2024-04-06T13:45:41.774624+0000","last_fullsized":"2024-04-06T13:45:41.774624+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:58:11.892572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774442+0000","last_change":"2024-04-06T13:45:39.676166+0000","last_active":"2024-04-06T13:45:41.774442+0000","last_peered":"2024-04-06T13:45:41.774442+0000","last_clean":"2024-04-06T13:45:41.774442+0000","last_became_active":"2024-04-06T13:45:39.675914+0000","last_became_peered":"2024-04-06T13:45:39.675914+0000","last_unstale":"2024-04-06T13:45:41.774442+0000","last_undegraded":"2024-04-06T13:45:41.774442+0000","last_fullsized":"2024-04-06T13:45:41.774442+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:26:57.018766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085555+0000","last_change":"2024-04-06T13:45:40.097373+0000","last_active":"2024-04-06T13:45:41.085555+0000","last_peered":"2024-04-06T13:45:41.085555+0000","last_clean":"2024-04-06T13:45:41.085555+0000","last_became_active":"2024-04-06T13:45:40.097124+0000","last_became_peered":"2024-04-06T13:45:40.097124+0000","last_unstale":"2024-04-06T13:45:41.085555+0000","last_undegraded":"2024-04-06T13:45:41.085555+0000","last_fullsized":"2024-04-06T13:45:41.085555+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:13:09.133530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774615+0000","last_change":"2024-04-06T13:45:39.663422+0000","last_active":"2024-04-06T13:45:41.774615+0000","last_peered":"2024-04-06T13:45:41.774615+0000","last_clean":"2024-04-06T13:45:41.774615+0000","last_became_active":"2024-04-06T13:45:39.663168+0000","last_became_peered":"2024-04-06T13:45:39.663168+0000","last_unstale":"2024-04-06T13:45:41.774615+0000","last_undegraded":"2024-04-06T13:45:41.774615+0000","last_fullsized":"2024-04-06T13:45:41.774615+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:08:46.166704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775122+0000","last_change":"2024-04-06T13:45:40.100105+0000","last_active":"2024-04-06T13:45:41.775122+0000","last_peered":"2024-04-06T13:45:41.775122+0000","last_clean":"2024-04-06T13:45:41.775122+0000","last_became_active":"2024-04-06T13:45:40.099733+0000","last_became_peered":"2024-04-06T13:45:40.099733+0000","last_unstale":"2024-04-06T13:45:41.775122+0000","last_undegraded":"2024-04-06T13:45:41.775122+0000","last_fullsized":"2024-04-06T13:45:41.775122+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:36:01.951089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"208'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774978+0000","last_change":"2024-04-06T13:45:39.671913+0000","last_active":"2024-04-06T13:45:41.774978+0000","last_peered":"2024-04-06T13:45:41.774978+0000","last_clean":"2024-04-06T13:45:41.774978+0000","last_became_active":"2024-04-06T13:45:39.671222+0000","last_became_peered":"2024-04-06T13:45:39.671222+0000","last_unstale":"2024-04-06T13:45:41.774978+0000","last_undegraded":"2024-04-06T13:45:41.774978+0000","last_fullsized":"2024-04-06T13:45:41.774978+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:51:31.411757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775705+0000","last_change":"2024-04-06T13:45:40.096319+0000","last_active":"2024-04-06T13:45:41.775705+0000","last_peered":"2024-04-06T13:45:41.775705+0000","last_clean":"2024-04-06T13:45:41.775705+0000","last_became_active":"2024-04-06T13:45:40.096095+0000","last_became_peered":"2024-04-06T13:45:40.096095+0000","last_unstale":"2024-04-06T13:45:41.775705+0000","last_undegraded":"2024-04-06T13:45:41.775705+0000","last_fullsized":"2024-04-06T13:45:41.775705+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:22:14.398303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.775345+0000","last_change":"2024-04-06T13:45:39.675889+0000","last_active":"2024-04-06T13:45:41.775345+0000","last_peered":"2024-04-06T13:45:41.775345+0000","last_clean":"2024-04-06T13:45:41.775345+0000","last_became_active":"2024-04-06T13:45:39.675594+0000","last_became_peered":"2024-04-06T13:45:39.675594+0000","last_unstale":"2024-04-06T13:45:41.775345+0000","last_undegraded":"2024-04-06T13:45:41.775345+0000","last_fullsized":"2024-04-06T13:45:41.775345+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:14:48.751801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.084974+0000","last_change":"2024-04-06T13:45:40.087121+0000","last_active":"2024-04-06T13:45:41.084974+0000","last_peered":"2024-04-06T13:45:41.084974+0000","last_clean":"2024-04-06T13:45:41.084974+0000","last_became_active":"2024-04-06T13:45:40.086842+0000","last_became_peered":"2024-04-06T13:45:40.086842+0000","last_unstale":"2024-04-06T13:45:41.084974+0000","last_undegraded":"2024-04-06T13:45:41.084974+0000","last_fullsized":"2024-04-06T13:45:41.084974+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:01:10.112627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085904+0000","last_change":"2024-04-06T13:45:39.680563+0000","last_active":"2024-04-06T13:45:41.085904+0000","last_peered":"2024-04-06T13:45:41.085904+0000","last_clean":"2024-04-06T13:45:41.085904+0000","last_became_active":"2024-04-06T13:45:39.679462+0000","last_became_peered":"2024-04-06T13:45:39.679462+0000","last_unstale":"2024-04-06T13:45:41.085904+0000","last_undegraded":"2024-04-06T13:45:41.085904+0000","last_fullsized":"2024-04-06T13:45:41.085904+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:26:01.577603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774909+0000","last_change":"2024-04-06T13:45:40.099989+0000","last_active":"2024-04-06T13:45:41.774909+0000","last_peered":"2024-04-06T13:45:41.774909+0000","last_clean":"2024-04-06T13:45:41.774909+0000","last_became_active":"2024-04-06T13:45:40.099769+0000","last_became_peered":"2024-04-06T13:45:40.099769+0000","last_unstale":"2024-04-06T13:45:41.774909+0000","last_undegraded":"2024-04-06T13:45:41.774909+0000","last_fullsized":"2024-04-06T13:45:41.774909+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:54:30.364282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.085738+0000","last_change":"2024-04-06T13:45:39.679145+0000","last_active":"2024-04-06T13:45:41.085738+0000","last_peered":"2024-04-06T13:45:41.085738+0000","last_clean":"2024-04-06T13:45:41.085738+0000","last_became_active":"2024-04-06T13:45:39.678837+0000","last_became_peered":"2024-04-06T13:45:39.678837+0000","last_unstale":"2024-04-06T13:45:41.085738+0000","last_undegraded":"2024-04-06T13:45:41.085738+0000","last_fullsized":"2024-04-06T13:45:41.085738+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:36:13.699413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774564+0000","last_change":"2024-04-06T13:45:40.091811+0000","last_active":"2024-04-06T13:45:41.774564+0000","last_peered":"2024-04-06T13:45:41.774564+0000","last_clean":"2024-04-06T13:45:41.774564+0000","last_became_active":"2024-04-06T13:45:40.091484+0000","last_became_peered":"2024-04-06T13:45:40.091484+0000","last_unstale":"2024-04-06T13:45:41.774564+0000","last_undegraded":"2024-04-06T13:45:41.774564+0000","last_fullsized":"2024-04-06T13:45:41.774564+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:36:37.635662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.084641+0000","last_change":"2024-04-06T13:45:39.666180+0000","last_active":"2024-04-06T13:45:41.084641+0000","last_peered":"2024-04-06T13:45:41.084641+0000","last_clean":"2024-04-06T13:45:41.084641+0000","last_became_active":"2024-04-06T13:45:39.665857+0000","last_became_peered":"2024-04-06T13:45:39.665857+0000","last_unstale":"2024-04-06T13:45:41.084641+0000","last_undegraded":"2024-04-06T13:45:41.084641+0000","last_fullsized":"2024-04-06T13:45:41.084641+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:43:34.876173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.086452+0000","last_change":"2024-04-06T13:45:40.087647+0000","last_active":"2024-04-06T13:45:41.086452+0000","last_peered":"2024-04-06T13:45:41.086452+0000","last_clean":"2024-04-06T13:45:41.086452+0000","last_became_active":"2024-04-06T13:45:40.087119+0000","last_became_peered":"2024-04-06T13:45:40.087119+0000","last_unstale":"2024-04-06T13:45:41.086452+0000","last_undegraded":"2024-04-06T13:45:41.086452+0000","last_fullsized":"2024-04-06T13:45:41.086452+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:45:23.813731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.084497+0000","last_change":"2024-04-06T13:45:39.679474+0000","last_active":"2024-04-06T13:45:41.084497+0000","last_peered":"2024-04-06T13:45:41.084497+0000","last_clean":"2024-04-06T13:45:41.084497+0000","last_became_active":"2024-04-06T13:45:39.679220+0000","last_became_peered":"2024-04-06T13:45:39.679220+0000","last_unstale":"2024-04-06T13:45:41.084497+0000","last_undegraded":"2024-04-06T13:45:41.084497+0000","last_fullsized":"2024-04-06T13:45:41.084497+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:17:30.456677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774915+0000","last_change":"2024-04-06T13:45:40.095592+0000","last_active":"2024-04-06T13:45:41.774915+0000","last_peered":"2024-04-06T13:45:41.774915+0000","last_clean":"2024-04-06T13:45:41.774915+0000","last_became_active":"2024-04-06T13:45:40.095138+0000","last_became_peered":"2024-04-06T13:45:40.095138+0000","last_unstale":"2024-04-06T13:45:41.774915+0000","last_undegraded":"2024-04-06T13:45:41.774915+0000","last_fullsized":"2024-04-06T13:45:41.774915+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:02:59.095200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774562+0000","last_change":"2024-04-06T13:45:39.663365+0000","last_active":"2024-04-06T13:45:41.774562+0000","last_peered":"2024-04-06T13:45:41.774562+0000","last_clean":"2024-04-06T13:45:41.774562+0000","last_became_active":"2024-04-06T13:45:39.663090+0000","last_became_peered":"2024-04-06T13:45:39.663090+0000","last_unstale":"2024-04-06T13:45:41.774562+0000","last_undegraded":"2024-04-06T13:45:41.774562+0000","last_fullsized":"2024-04-06T13:45:41.774562+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:06:43.020693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.084554+0000","last_change":"2024-04-06T13:45:40.089447+0000","last_active":"2024-04-06T13:45:41.084554+0000","last_peered":"2024-04-06T13:45:41.084554+0000","last_clean":"2024-04-06T13:45:41.084554+0000","last_became_active":"2024-04-06T13:45:40.089143+0000","last_became_peered":"2024-04-06T13:45:40.089143+0000","last_unstale":"2024-04-06T13:45:41.084554+0000","last_undegraded":"2024-04-06T13:45:41.084554+0000","last_fullsized":"2024-04-06T13:45:41.084554+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:50:04.043102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774720+0000","last_change":"2024-04-06T13:45:39.668112+0000","last_active":"2024-04-06T13:45:41.774720+0000","last_peered":"2024-04-06T13:45:41.774720+0000","last_clean":"2024-04-06T13:45:41.774720+0000","last_became_active":"2024-04-06T13:45:39.667831+0000","last_became_peered":"2024-04-06T13:45:39.667831+0000","last_unstale":"2024-04-06T13:45:41.774720+0000","last_undegraded":"2024-04-06T13:45:41.774720+0000","last_fullsized":"2024-04-06T13:45:41.774720+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:46:27.010137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":10,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774839+0000","last_change":"2024-04-06T13:45:40.094387+0000","last_active":"2024-04-06T13:45:41.774839+0000","last_peered":"2024-04-06T13:45:41.774839+0000","last_clean":"2024-04-06T13:45:41.774839+0000","last_became_active":"2024-04-06T13:45:40.094113+0000","last_became_peered":"2024-04-06T13:45:40.094113+0000","last_unstale":"2024-04-06T13:45:41.774839+0000","last_undegraded":"2024-04-06T13:45:41.774839+0000","last_fullsized":"2024-04-06T13:45:41.774839+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:39.649788+0000","last_clean_scrub_stamp":"2024-04-06T13:45:39.649788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:28:35.241968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-06T13:45:41.774374+0000","last_change":"2024-04-06T13:45:39.667029+0000","last_active":"2024-04-06T13:45:41.774374+0000","last_peered":"2024-04-06T13:45:41.774374+0000","last_clean":"2024-04-06T13:45:41.774374+0000","last_became_active":"2024-04-06T13:45:39.666367+0000","last_became_peered":"2024-04-06T13:45:39.666367+0000","last_unstale":"2024-04-06T13:45:41.774374+0000","last_undegraded":"2024-04-06T13:45:41.774374+0000","last_fullsized":"2024-04-06T13:45:41.774374+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:45:38.645325+0000","last_clean_scrub_stamp":"2024-04-06T13:45:38.645325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:49:47.202562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"198'326","reported_seq":1140,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-04-06T13:45:40.080815+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:45:40.080815+0000","last_peered":"2024-04-06T13:45:40.080815+0000","last_clean":"2024-04-06T13:45:40.080815+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:45:40.080815+0000","last_undegraded":"2024-04-06T13:45:40.080815+0000","last_fullsized":"2024-04-06T13:45:40.080815+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575964,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52308,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319532,"statfs":{"total":96636764160,"available":96583200768,"internally_reserved":0,"allocated":3215360,"data_stored":1923767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61099999999999999,"15min":0.60699999999999998},"min":{"1min":0.47199999999999998,"5min":0.38700000000000001,"15min":0.312},"max":{"1min":0.70799999999999996,"5min":0.79700000000000004,"15min":0.84899999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60199999999999998,"15min":0.60299999999999998},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.35199999999999998},"max":{"1min":0.72899999999999998,"5min":0.80800000000000005,"15min":0.82899999999999996},"last":0.56599999999999995}]},{"osd":1,"last update":"Sat Apr 6 13:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58999999999999997,"15min":0.59699999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.5},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.46000000000000002,"5min":0.41999999999999998,"15min":0.33000000000000002},"max":{"1min":0.754,"5min":0.83399999999999996,"15min":0.85299999999999998},"last":0.51900000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.47399999999999998,"5min":0.41299999999999998,"15min":0.36599999999999999},"max":{"1min":0.78600000000000003,"5min":0.82699999999999996,"15min":0.83899999999999997},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60799999999999998,"15min":0.60999999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.72999999999999998,"5min":0.79700000000000004,"15min":0.81699999999999995},"last":0.60599999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.48299999999999998,"5min":0.432,"15min":0.35999999999999999},"max":{"1min":0.76700000000000002,"5min":0.81100000000000005,"15min":0.86199999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60999999999999999,"15min":0.61499999999999999},"min":{"1min":0.42099999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.74099999999999999,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.67000000000000004}]},{"osd":4,"last update":"Sat Apr 6 13:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.30299999999999999},"max":{"1min":0.61799999999999999,"5min":0.76000000000000001,"15min":0.81599999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.36799999999999999},"max":{"1min":0.65900000000000003,"5min":0.74299999999999999,"15min":0.82699999999999996},"last":0.58199999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.48999999999999999,"15min":0.497},"min":{"1min":0.318,"5min":0.318,"15min":0.308},"max":{"1min":0.69299999999999995,"5min":0.76600000000000001,"15min":0.78900000000000003},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57699999999999996,"15min":0.57999999999999996},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.33600000000000002},"max":{"1min":0.68400000000000005,"5min":0.82099999999999995,"15min":0.85399999999999998},"last":0.68600000000000005}]},{"osd":6,"last update":"Sat Apr 6 13:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.379,"5min":0.374,"15min":0.34899999999999998},"max":{"1min":0.70599999999999996,"5min":0.81200000000000006,"15min":0.879},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.40100000000000002,"5min":0.377,"15min":0.34799999999999998},"max":{"1min":0.70199999999999996,"5min":0.84899999999999998,"15min":0.89800000000000002},"last":0.64800000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575963,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37764,"kb_used_data":2548,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334076,"statfs":{"total":96636764160,"available":96598093824,"internally_reserved":0,"allocated":2609152,"data_stored":1325758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:44:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.373,"15min":0.32000000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.83799999999999997},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56599999999999995,"15min":0.57199999999999995},"min":{"1min":0.442,"5min":0.38700000000000001,"15min":0.37},"max":{"1min":0.68500000000000005,"5min":0.81999999999999995,"15min":0.82399999999999995},"last":0.497}]},{"osd":1,"last update":"Sat Apr 6 13:44:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.628,"15min":0.63100000000000001},"min":{"1min":0.52600000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.78900000000000003,"5min":0.83699999999999997,"15min":0.85799999999999998},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64800000000000002,"15min":0.65100000000000002},"min":{"1min":0.495,"5min":0.378,"15min":0.378},"max":{"1min":0.752,"5min":0.81399999999999995,"15min":0.82699999999999996},"last":0.57999999999999996}]},{"osd":2,"last update":"Sat Apr 6 13:44:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60299999999999998,"15min":0.59399999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.42299999999999999},"max":{"1min":0.73099999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58999999999999997,"15min":0.59599999999999997},"min":{"1min":0.39200000000000002,"5min":0.375,"15min":0.35799999999999998},"max":{"1min":0.70999999999999996,"5min":0.78600000000000003,"15min":0.82599999999999996},"last":0.51200000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:44:50 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61299999999999999,"15min":0.61099999999999999},"min":{"1min":0.42199999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.75600000000000001,"5min":0.77900000000000003,"15min":0.82399999999999995},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64200000000000002,"15min":0.65000000000000002},"min":{"1min":0.45900000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77200000000000002,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.56899999999999995}]},{"osd":4,"last update":"Sat Apr 6 13:44:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53900000000000003,"15min":0.52800000000000002},"min":{"1min":0.36899999999999999,"5min":0.33800000000000002,"15min":0.30099999999999999},"max":{"1min":0.70699999999999996,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.54000000000000004,"15min":0.55100000000000005},"min":{"1min":0.34000000000000002,"5min":0.33800000000000002,"15min":0.28899999999999998},"max":{"1min":0.82099999999999995,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.58999999999999997}]},{"osd":5,"last update":"Sat Apr 6 13:44:50 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50800000000000001,"15min":0.51200000000000001},"min":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.27100000000000002},"max":{"1min":0.68400000000000005,"5min":0.78000000000000003,"15min":0.78400000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.28999999999999998},"max":{"1min":0.73899999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.45600000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:44:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54400000000000004,"15min":0.55100000000000005},"min":{"1min":0.40500000000000003,"5min":0.39500000000000002,"15min":0.312},"max":{"1min":0.72099999999999997,"5min":0.82599999999999996,"15min":0.84799999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.41899999999999998,"5min":0.36799999999999999,"15min":0.32500000000000001},"max":{"1min":0.77600000000000002,"5min":0.82499999999999996,"15min":0.86199999999999999},"last":0.55300000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59908,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311932,"statfs":{"total":96636764160,"available":96575418368,"internally_reserved":0,"allocated":2605056,"data_stored":1325688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39000000000000001,"15min":0.39600000000000002},"min":{"1min":0.29899999999999999,"5min":0.25,"15min":0.25},"max":{"1min":0.53100000000000003,"5min":0.56499999999999995,"15min":0.64500000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.433,"15min":0.433},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.254},"max":{"1min":0.56100000000000005,"5min":0.60999999999999999,"15min":0.69399999999999995},"last":0.40400000000000003}]},{"osd":2,"last update":"Sat Apr 6 13:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42299999999999999,"15min":0.42999999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.54600000000000004,"5min":0.61799999999999999,"15min":0.65900000000000003},"last":0.504},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.46800000000000003,"15min":0.46899999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.29399999999999998},"max":{"1min":0.64300000000000002,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.48899999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.33700000000000002,"5min":0.29799999999999999,"15min":0.28799999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.439,"15min":0.44400000000000001},"min":{"1min":0.30599999999999999,"5min":0.29699999999999999,"15min":0.26300000000000001},"max":{"1min":0.59099999999999997,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.42499999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63700000000000001,"15min":0.65300000000000002},"min":{"1min":0.47599999999999998,"5min":0.433,"15min":0.33800000000000002},"max":{"1min":0.872,"5min":0.95299999999999996,"15min":0.97399999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64300000000000002,"15min":0.66400000000000003},"min":{"1min":0.49199999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85999999999999999,"5min":0.97599999999999998,"15min":0.995},"last":0.70799999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.65800000000000003},"min":{"1min":0.48599999999999999,"5min":0.442,"15min":0.40799999999999997},"max":{"1min":0.77500000000000002,"5min":0.96899999999999997,"15min":1.073},"last":0.63},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66200000000000003,"15min":0.67700000000000005},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.81599999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.58599999999999997}]},{"osd":6,"last update":"Sat Apr 6 13:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64000000000000001,"15min":0.66700000000000004},"min":{"1min":0.47599999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.84599999999999997,"5min":0.90900000000000003,"15min":1},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.67300000000000004,"15min":0.69199999999999995},"min":{"1min":0.55500000000000005,"5min":0.48699999999999999,"15min":0.46800000000000003},"max":{"1min":0.82799999999999996,"5min":0.93999999999999995,"15min":1.036},"last":0.66900000000000004}]},{"osd":7,"last update":"Sat Apr 6 13:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.69399999999999995,"15min":0.70199999999999996},"min":{"1min":0.51900000000000002,"5min":0.50900000000000001,"15min":0.501},"max":{"1min":0.90200000000000002,"5min":0.98999999999999999,"15min":1.0649999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66000000000000003,"15min":0.67900000000000005},"min":{"1min":0.46700000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.86899999999999999,"5min":0.90000000000000002,"15min":1.056},"last":0.64500000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575962,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42468,"kb_used_data":3152,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329372,"statfs":{"total":96636764160,"available":96593276928,"internally_reserved":0,"allocated":3227648,"data_stored":1923476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.40000000000000002,"15min":0.40100000000000002},"min":{"1min":0.29299999999999998,"5min":0.26800000000000002,"15min":0.23799999999999999},"max":{"1min":0.56100000000000005,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.251},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.41999999999999998,"15min":0.42699999999999999},"min":{"1min":0.28199999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.58599999999999997,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.33400000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.42899999999999999,"15min":0.436},"min":{"1min":0.311,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.65400000000000003,"5min":0.67900000000000005,"15min":0.70099999999999996},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44400000000000001,"15min":0.45500000000000002},"min":{"1min":0.32500000000000001,"5min":0.316,"15min":0.315},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.27100000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.432,"15min":0.436},"min":{"1min":0.33500000000000002,"5min":0.28899999999999998,"15min":0.26800000000000002},"max":{"1min":0.63300000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.48299999999999998,"15min":0.48799999999999999},"min":{"1min":0.38,"5min":0.307,"15min":0.307},"max":{"1min":0.71899999999999997,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.34799999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.84899999999999998,"5min":0.92500000000000004,"15min":1.0329999999999999},"last":0.37},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.64900000000000002,"15min":0.65800000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.41999999999999998},"max":{"1min":0.94399999999999995,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.43099999999999999}]},{"osd":5,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63300000000000001,"15min":0.65000000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.40600000000000003},"max":{"1min":0.876,"5min":0.876,"15min":1.0009999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66900000000000004,"15min":0.67800000000000005},"min":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.92800000000000005,"5min":0.96199999999999997,"15min":1.052},"last":0.378}]},{"osd":6,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.67700000000000005,"15min":0.68700000000000006},"min":{"1min":0.48099999999999998,"5min":0.435,"15min":0.435},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.439},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67300000000000004,"15min":0.68400000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.90500000000000003,"5min":0.96399999999999997,"15min":1.085},"last":0.39000000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.69299999999999995,"15min":0.70299999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.01,"5min":1.01,"15min":1.121},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67800000000000005,"15min":0.68500000000000005},"min":{"1min":0.51100000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.89500000000000002,"5min":0.97699999999999998,"15min":1.0229999999999999},"last":0.36099999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575964,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59920,"kb_used_data":2560,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311920,"statfs":{"total":96636764160,"available":96575406080,"internally_reserved":0,"allocated":2621440,"data_stored":1332962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.38600000000000001,"15min":0.38800000000000001},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.52700000000000002,"5min":0.58399999999999996,"15min":0.59099999999999997},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.433,"15min":0.42899999999999999},"min":{"1min":0.30599999999999999,"5min":0.27000000000000002,"15min":0.249},"max":{"1min":0.57799999999999996,"5min":0.64100000000000001,"15min":1.044},"last":0.38800000000000001}]},{"osd":1,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.41199999999999998,"15min":0.41299999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.255},"max":{"1min":0.52900000000000003,"5min":0.56399999999999995,"15min":0.753},"last":0.313},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44600000000000001,"15min":0.45100000000000001},"min":{"1min":0.29299999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.60799999999999998,"5min":0.627,"15min":0.82199999999999995},"last":0.41999999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.47399999999999998,"15min":0.46600000000000003},"min":{"1min":0.34300000000000003,"5min":0.31900000000000001,"15min":0.27800000000000002},"max":{"1min":0.61499999999999999,"5min":0.69699999999999995,"15min":0.70299999999999996},"last":0.372},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45900000000000002,"15min":0.45400000000000001},"min":{"1min":0.33200000000000002,"5min":0.312,"15min":0.30299999999999999},"max":{"1min":0.57899999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.32700000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65500000000000003,"15min":0.64600000000000002},"min":{"1min":0.47399999999999998,"5min":0.41799999999999998,"15min":0.376},"max":{"1min":0.83099999999999996,"5min":0.91800000000000004,"15min":0.93400000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.68100000000000005,"15min":0.67100000000000004},"min":{"1min":0.504,"5min":0.42499999999999999,"15min":0.41899999999999998},"max":{"1min":0.83299999999999996,"5min":0.97599999999999998,"15min":1.0229999999999999},"last":0.58099999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68600000000000005,"15min":0.67100000000000004},"min":{"1min":0.46500000000000002,"5min":0.46000000000000002,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.68799999999999994,"15min":0.67200000000000004},"min":{"1min":0.50800000000000001,"5min":0.46200000000000002,"15min":0.42899999999999999},"max":{"1min":0.80300000000000005,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.54300000000000004}]},{"osd":6,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67900000000000005,"15min":0.67800000000000005},"min":{"1min":0.47599999999999998,"5min":0.46700000000000003,"15min":0.433},"max":{"1min":0.88600000000000001,"5min":1.016,"15min":1.091},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.71799999999999997,"15min":0.70199999999999996},"min":{"1min":0.56799999999999995,"5min":0.50600000000000001,"15min":0.41299999999999998},"max":{"1min":0.89600000000000002,"5min":0.95899999999999996,"15min":0.97699999999999998},"last":0.59699999999999998}]},{"osd":7,"last update":"Sat Apr 6 13:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.69999999999999996,"15min":0.69599999999999995},"min":{"1min":0.54500000000000004,"5min":0.48199999999999998,"15min":0.47999999999999998},"max":{"1min":0.86699999999999999,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72199999999999998,"15min":0.71499999999999997},"min":{"1min":0.58499999999999996,"5min":0.45900000000000002,"15min":0.443},"max":{"1min":0.871,"5min":0.99099999999999999,"15min":1.0089999999999999},"last":0.56899999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575963,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47636,"kb_used_data":2568,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324204,"statfs":{"total":96636764160,"available":96587984896,"internally_reserved":0,"allocated":2629632,"data_stored":1333559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.38600000000000001,"15min":0.39000000000000001},"min":{"1min":0.27200000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.502,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.41499999999999998,"15min":0.41599999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.58599999999999997,"5min":0.622,"15min":0.66700000000000004},"last":0.374}]},{"osd":1,"last update":"Sat Apr 6 13:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.41499999999999998,"15min":0.40899999999999997},"min":{"1min":0.36199999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.58899999999999997,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.436,"15min":0.44700000000000001},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.60599999999999998,"5min":0.66000000000000003,"15min":0.75800000000000001},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48899999999999999,"15min":0.49099999999999999},"min":{"1min":0.39600000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.60399999999999998,"5min":0.68999999999999995,"15min":0.70299999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.50600000000000001,"15min":0.50800000000000001},"min":{"1min":0.39100000000000001,"5min":0.33000000000000002,"15min":0.32900000000000001},"max":{"1min":0.64200000000000002,"5min":0.747,"15min":0.79300000000000004},"last":0.44500000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.64200000000000002,"15min":0.64500000000000002},"min":{"1min":0.504,"5min":0.40300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88200000000000001,"5min":1.002,"15min":1.018},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.64300000000000002,"15min":0.65400000000000003},"min":{"1min":0.5,"5min":0.375,"15min":0.375},"max":{"1min":0.79600000000000004,"5min":0.98599999999999999,"15min":0.995},"last":0.72899999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68400000000000005,"15min":0.69099999999999995},"min":{"1min":0.57699999999999996,"5min":0.46200000000000002,"15min":0.45900000000000002},"max":{"1min":0.879,"5min":0.97299999999999998,"15min":0.999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.68899999999999995,"15min":0.69899999999999995},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.46600000000000003},"max":{"1min":0.92300000000000004,"5min":0.92800000000000005,"15min":0.996},"last":0.76900000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.66100000000000003,"15min":0.67100000000000004},"min":{"1min":0.55800000000000005,"5min":0.443,"15min":0.443},"max":{"1min":0.83699999999999997,"5min":0.95199999999999996,"15min":0.97999999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.68600000000000005,"15min":0.68899999999999995},"min":{"1min":0.55900000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91600000000000004,"5min":0.96199999999999997,"15min":0.98799999999999999},"last":0.74299999999999999}]},{"osd":7,"last update":"Sat Apr 6 13:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.67600000000000005,"15min":0.68300000000000005},"min":{"1min":0.51300000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.94999999999999996,"5min":0.96199999999999997,"15min":0.97899999999999998},"last":0.625},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.69999999999999996,"15min":0.70499999999999996},"min":{"1min":0.60399999999999998,"5min":0.48499999999999999,"15min":0.47899999999999998},"max":{"1min":0.85999999999999999,"5min":0.98799999999999999,"15min":1.024},"last":0.68500000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64036,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307804,"statfs":{"total":96636764160,"available":96571191296,"internally_reserved":0,"allocated":2637824,"data_stored":1334997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58599999999999997,"15min":0.59799999999999998},"min":{"1min":0.51600000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.80900000000000005,"5min":0.876,"15min":0.88500000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.46100000000000002,"5min":0.40600000000000003,"15min":0.34799999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.68999999999999995}]},{"osd":1,"last update":"Sat Apr 6 13:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63600000000000001,"15min":0.64500000000000002},"min":{"1min":0.41199999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77900000000000003,"5min":0.84899999999999998,"15min":0.873},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64100000000000001,"15min":0.63200000000000001},"min":{"1min":0.54000000000000004,"5min":0.436,"15min":0.39700000000000002},"max":{"1min":0.81699999999999995,"5min":0.82299999999999995,"15min":0.86299999999999999},"last":0.55100000000000005}]},{"osd":2,"last update":"Sat Apr 6 13:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59999999999999998,"15min":0.60399999999999998},"min":{"1min":0.49299999999999999,"5min":0.41299999999999998,"15min":0.35399999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.877},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63200000000000001,"15min":0.621},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.38700000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.90400000000000003},"last":0.53200000000000003}]},{"osd":3,"last update":"Sat Apr 6 13:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.45500000000000002,"5min":0.41699999999999998,"15min":0.39900000000000002},"max":{"1min":0.83899999999999997,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63500000000000001,"15min":0.63100000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.40400000000000003},"max":{"1min":0.876,"5min":0.876,"15min":0.91100000000000003},"last":0.56399999999999995}]},{"osd":5,"last update":"Sat Apr 6 13:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.505,"15min":0.496},"min":{"1min":0.35399999999999998,"5min":0.33200000000000002,"15min":0.32800000000000001},"max":{"1min":0.68799999999999994,"5min":0.749,"15min":0.77700000000000002},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53700000000000003,"15min":0.52600000000000002},"min":{"1min":0.40300000000000002,"5min":0.379,"15min":0.34499999999999997},"max":{"1min":0.73799999999999999,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.51700000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53600000000000003,"15min":0.52800000000000002},"min":{"1min":0.36699999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75700000000000001,"5min":0.78800000000000003,"15min":0.83899999999999997},"last":0.498},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.38700000000000001,"5min":0.373,"15min":0.34399999999999997},"max":{"1min":0.78500000000000003,"5min":0.82399999999999995,"15min":0.84199999999999997},"last":0.58999999999999997}]},{"osd":7,"last update":"Sat Apr 6 13:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59899999999999998,"15min":0.60399999999999998},"min":{"1min":0.41599999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.86499999999999999,"5min":0.878,"15min":0.88100000000000001},"last":0.64100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59928,"kb_used_data":2572,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311912,"statfs":{"total":96636764160,"available":96575397888,"internally_reserved":0,"allocated":2633728,"data_stored":1333966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:44:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.59699999999999998,"15min":0.58999999999999997},"min":{"1min":0.372,"5min":0.34000000000000002,"15min":0.27600000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.60399999999999998,"15min":0.59699999999999998},"min":{"1min":0.47799999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.89200000000000002},"last":0.54200000000000004}]},{"osd":1,"last update":"Sat Apr 6 13:44:40 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60499999999999998,"15min":0.59399999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.32600000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81399999999999995},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59399999999999997,"15min":0.58899999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.311},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.83199999999999996},"last":0.505}]},{"osd":2,"last update":"Sat Apr 6 13:44:40 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61199999999999999,"15min":0.60499999999999998},"min":{"1min":0.45500000000000002,"5min":0.36399999999999999,"15min":0.312},"max":{"1min":0.78800000000000003,"5min":0.84099999999999997,"15min":0.84399999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.60899999999999999,"15min":0.59999999999999998},"min":{"1min":0.46800000000000003,"5min":0.39700000000000002,"15min":0.34699999999999998},"max":{"1min":0.72799999999999998,"5min":0.79500000000000004,"15min":0.82199999999999995},"last":0.56299999999999994}]},{"osd":3,"last update":"Sat Apr 6 13:44:40 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61399999999999999,"15min":0.60099999999999998},"min":{"1min":0.498,"5min":0.41099999999999998,"15min":0.28799999999999998},"max":{"1min":0.76700000000000002,"5min":0.79000000000000004,"15min":0.94499999999999995},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.63300000000000001,"15min":0.63500000000000001},"min":{"1min":0.40300000000000002,"5min":0.39000000000000001,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":0.84499999999999997,"15min":0.91500000000000004},"last":0.47599999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:44:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.50600000000000001,"15min":0.5},"min":{"1min":0.374,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.76400000000000001,"5min":0.76500000000000001,"15min":1.0369999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58199999999999996,"15min":0.56899999999999995},"min":{"1min":0.39000000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":1.1639999999999999},"last":0.66300000000000003}]},{"osd":6,"last update":"Sat Apr 6 13:44:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52300000000000002,"15min":0.51700000000000002},"min":{"1min":0.33800000000000002,"5min":0.28899999999999998,"15min":0.27900000000000003},"max":{"1min":0.74399999999999999,"5min":0.755,"15min":1.0169999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53000000000000003,"15min":0.51900000000000002},"min":{"1min":0.35399999999999998,"5min":0.32300000000000001,"15min":0.30399999999999999},"max":{"1min":0.77800000000000002,"5min":0.877,"15min":1.0009999999999999},"last":0.68899999999999995}]},{"osd":7,"last update":"Sat Apr 6 13:44:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54700000000000004,"15min":0.53600000000000003},"min":{"1min":0.39300000000000002,"5min":0.33600000000000002,"15min":0.32500000000000001},"max":{"1min":0.75600000000000001,"5min":0.80300000000000005,"15min":0.84199999999999997},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.64900000000000002,"15min":0.64000000000000001},"min":{"1min":0.498,"5min":0.38300000000000001,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.96599999999999997},"last":0.70099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:45:44.610 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-06T13:45:44.611 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-06T13:45:44.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.773+0000 7f1bbbb90700 1 Processor -- start 2024-04-06T13:45:44.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.773+0000 7f1bbbb90700 1 -- start start 2024-04-06T13:45:44.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.774+0000 7f1bbbb90700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1bb406ae40 0x7f1bb40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:44.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.774+0000 7f1bbbb90700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb40b3990 con 0x7f1bb415ddf0 2024-04-06T13:45:44.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.774+0000 7f1bbbb90700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb40b3060 con 0x7f1bb415d7b0 2024-04-06T13:45:44.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.774+0000 7f1bbbb90700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bb415e950 con 0x7f1bb406ae40 2024-04-06T13:45:44.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.774+0000 7f1bb912b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1bb406ae40 0x7f1bb40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:44.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.774+0000 7f1bb912b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1bb406ae40 0x7f1bb40b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48192/0 (socket says 172.21.15.189:48192) 2024-04-06T13:45:44.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.774+0000 7f1bb912b700 1 -- 172.21.15.189:0/446607998 learned_addr learned my addr 172.21.15.189:0/446607998 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:44.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.774+0000 7f1bba12d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1bb415ddf0 0x7f1bb415e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55936/0 (socket says 172.21.15.189:55936) 2024-04-06T13:45:44.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.774+0000 7f1bb912b700 1 -- 172.21.15.189:0/446607998 >> v1:172.21.15.192:6789/0 conn(0x7f1bb415d7b0 legacy=0x7f1bb406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:44.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.774+0000 7f1bb912b700 1 -- 172.21.15.189:0/446607998 >> v1:172.21.15.189:6789/0 conn(0x7f1bb415ddf0 legacy=0x7f1bb415e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:44.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.774+0000 7f1bb912b700 1 -- 172.21.15.189:0/446607998 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb40b3340 con 0x7f1bb406ae40 2024-04-06T13:45:44.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.774+0000 7f1bb892a700 1 -- 172.21.15.189:0/446607998 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013100177 0 0) 0x7f1bb40b3990 con 0x7f1bb415ddf0 2024-04-06T13:45:44.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.775+0000 7f1bb912b700 1 --2- 172.21.15.189:0/446607998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb406ae40 0x7f1bb40b3e20 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f1bb00098f0 tx=0x7f1bb0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d8bd8d31a614799 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:44.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.775+0000 7f1bb892a700 1 -- 172.21.15.189:0/446607998 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0004230 con 0x7f1bb406ae40 2024-04-06T13:45:44.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.775+0000 7f1bb892a700 1 -- 172.21.15.189:0/446607998 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb0004390 con 0x7f1bb406ae40 2024-04-06T13:45:44.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.775+0000 7f1bb892a700 1 -- 172.21.15.189:0/446607998 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb00036a0 con 0x7f1bb406ae40 2024-04-06T13:45:44.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.775+0000 7f1bbbb90700 1 -- 172.21.15.189:0/446607998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb406ae40 msgr2=0x7f1bb40b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:44.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.775+0000 7f1bbbb90700 1 --2- 172.21.15.189:0/446607998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb406ae40 0x7f1bb40b3e20 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f1bb00098f0 tx=0x7f1bb0009c00 comp rx=0 tx=0).stop 2024-04-06T13:45:44.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.775+0000 7f1bbbb90700 1 -- 172.21.15.189:0/446607998 shutdown_connections 2024-04-06T13:45:44.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.775+0000 7f1bbbb90700 1 --2- 172.21.15.189:0/446607998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb406ae40 0x7f1bb40b3e20 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:44.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.775+0000 7f1bbbb90700 1 -- 172.21.15.189:0/446607998 >> 172.21.15.189:0/446607998 conn(0x7f1bb40d00b0 msgr2=0x7f1bb40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:44.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.775+0000 7f1bbbb90700 1 -- 172.21.15.189:0/446607998 shutdown_connections 2024-04-06T13:45:44.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.776+0000 7f1bbbb90700 1 -- 172.21.15.189:0/446607998 wait complete. 2024-04-06T13:45:44.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.777+0000 7f1bbbb90700 1 Processor -- start 2024-04-06T13:45:44.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.777+0000 7f1bbbb90700 1 -- start start 2024-04-06T13:45:44.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.777+0000 7f1bbbb90700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bb406ae40 0x7f1bb40b1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:44.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.777+0000 7f1bbbb90700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bb415d7b0 0x7f1bb41f5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:44.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.777+0000 7f1bbbb90700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb415ddf0 0x7f1bb40cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:44.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.777+0000 7f1bbbb90700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb4154210 con 0x7f1bb415ddf0 2024-04-06T13:45:44.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.777+0000 7f1bbbb90700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb41f6ed0 con 0x7f1bb406ae40 2024-04-06T13:45:44.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.777+0000 7f1bbbb90700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb40cc770 con 0x7f1bb415d7b0 2024-04-06T13:45:44.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.777+0000 7f1bba12d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb415ddf0 0x7f1bb40cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:44.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.777+0000 7f1bba12d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb415ddf0 0x7f1bb40cd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48196/0 (socket says 172.21.15.189:48196) 2024-04-06T13:45:44.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.778+0000 7f1bba12d700 1 -- 172.21.15.189:0/3581177090 learned_addr learned my addr 172.21.15.189:0/3581177090 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:44.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.778+0000 7f1bb992c700 1 --2- 172.21.15.189:0/3581177090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bb406ae40 0x7f1bb40b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:44.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.778+0000 7f1bba12d700 1 -- 172.21.15.189:0/3581177090 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bb415d7b0 msgr2=0x7f1bb41f5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:44.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.778+0000 7f1bb912b700 1 --2- 172.21.15.189:0/3581177090 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bb415d7b0 0x7f1bb41f5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:44.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.778+0000 7f1bba12d700 1 --2- 172.21.15.189:0/3581177090 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bb415d7b0 0x7f1bb41f5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:44.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.778+0000 7f1bba12d700 1 -- 172.21.15.189:0/3581177090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bb406ae40 msgr2=0x7f1bb40b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:44.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.778+0000 7f1bba12d700 1 --2- 172.21.15.189:0/3581177090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bb406ae40 0x7f1bb40b1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:44.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.778+0000 7f1bba12d700 1 -- 172.21.15.189:0/3581177090 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb0009550 con 0x7f1bb415ddf0 2024-04-06T13:45:44.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.778+0000 7f1bb992c700 1 --2- 172.21.15.189:0/3581177090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bb406ae40 0x7f1bb40b1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:45:44.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.778+0000 7f1bba12d700 1 --2- 172.21.15.189:0/3581177090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb415ddf0 0x7f1bb40cd720 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f1bac00cb60 tx=0x7f1bac00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:44.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.778+0000 7f1baa7fc700 1 -- 172.21.15.189:0/3581177090 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac00ab50 con 0x7f1bb415ddf0 2024-04-06T13:45:44.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.778+0000 7f1baa7fc700 1 -- 172.21.15.189:0/3581177090 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bac00acb0 con 0x7f1bb415ddf0 2024-04-06T13:45:44.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.778+0000 7f1bbbb90700 1 -- 172.21.15.189:0/3581177090 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb41f6630 con 0x7f1bb415ddf0 2024-04-06T13:45:44.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.778+0000 7f1baa7fc700 1 -- 172.21.15.189:0/3581177090 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac007310 con 0x7f1bb415ddf0 2024-04-06T13:45:44.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.778+0000 7f1bbbb90700 1 -- 172.21.15.189:0/3581177090 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb41f6bf0 con 0x7f1bb415ddf0 2024-04-06T13:45:44.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.779+0000 7f1bbbb90700 1 -- 172.21.15.189:0/3581177090 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb4067bb0 con 0x7f1bb415ddf0 2024-04-06T13:45:44.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.780+0000 7f1baa7fc700 1 -- 172.21.15.189:0/3581177090 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f1bac007470 con 0x7f1bb415ddf0 2024-04-06T13:45:44.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.780+0000 7f1baa7fc700 1 --2- 172.21.15.189:0/3581177090 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1b9806adb0 0x7f1b9806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:44.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.780+0000 7f1bb992c700 1 --2- 172.21.15.189:0/3581177090 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1b9806adb0 0x7f1b9806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:44.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.780+0000 7f1baa7fc700 1 -- 172.21.15.189:0/3581177090 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1bac089460 con 0x7f1bb415ddf0 2024-04-06T13:45:44.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.781+0000 7f1bb992c700 1 --2- 172.21.15.189:0/3581177090 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1b9806adb0 0x7f1b9806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4003f40 tx=0x7f1ba4009040 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:44.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:44.785+0000 7f1baa7fc700 1 -- 172.21.15.189:0/3581177090 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1bac0543a0 con 0x7f1bb415ddf0 2024-04-06T13:45:45.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.055+0000 7f1bbbb90700 1 -- 172.21.15.189:0/3581177090 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f1bb40cc8e0 con 0x7f1bb415ddf0 2024-04-06T13:45:45.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.059+0000 7f1baa7fc700 1 -- 172.21.15.189:0/3581177090 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v50) v1 ==== 190+0+0 (secure 0 0 0) 0x7f1bac0579c0 con 0x7f1bb415ddf0 2024-04-06T13:45:45.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.061+0000 7f1bbbb90700 1 -- 172.21.15.189:0/3581177090 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1b9806adb0 msgr2=0x7f1b9806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:45.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.061+0000 7f1bbbb90700 1 --2- 172.21.15.189:0/3581177090 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1b9806adb0 0x7f1b9806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4003f40 tx=0x7f1ba4009040 comp rx=0 tx=0).stop 2024-04-06T13:45:45.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.061+0000 7f1bbbb90700 1 -- 172.21.15.189:0/3581177090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb415ddf0 msgr2=0x7f1bb40cd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:45.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.061+0000 7f1bbbb90700 1 --2- 172.21.15.189:0/3581177090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb415ddf0 0x7f1bb40cd720 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f1bac00cb60 tx=0x7f1bac00cf20 comp rx=0 tx=0).stop 2024-04-06T13:45:45.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.061+0000 7f1bbbb90700 1 -- 172.21.15.189:0/3581177090 shutdown_connections 2024-04-06T13:45:45.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.061+0000 7f1bbbb90700 1 --2- 172.21.15.189:0/3581177090 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1b9806adb0 0x7f1b9806d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.061+0000 7f1bbbb90700 1 --2- 172.21.15.189:0/3581177090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb415ddf0 0x7f1bb40cd720 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.061+0000 7f1bbbb90700 1 --2- 172.21.15.189:0/3581177090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bb406ae40 0x7f1bb40b1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.061+0000 7f1bbbb90700 1 --2- 172.21.15.189:0/3581177090 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bb415d7b0 0x7f1bb41f5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.061+0000 7f1bbbb90700 1 -- 172.21.15.189:0/3581177090 >> 172.21.15.189:0/3581177090 conn(0x7f1bb40d00b0 msgr2=0x7f1bb40ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:45.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.061+0000 7f1bbbb90700 1 -- 172.21.15.189:0/3581177090 shutdown_connections 2024-04-06T13:45:45.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.061+0000 7f1bbbb90700 1 -- 172.21.15.189:0/3581177090 wait complete. 2024-04-06T13:45:45.062 INFO:teuthology.orchestra.run.smithi189.stderr:updated caps for client.0 2024-04-06T13:45:45.073 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:45:45.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.195+0000 7f96e7ef9700 1 Processor -- start 2024-04-06T13:45:45.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.195+0000 7f96e7ef9700 1 -- start start 2024-04-06T13:45:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.196+0000 7f96e7ef9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f96e015a550 0x7f96e00cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.196+0000 7f96e7ef9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96e00cdbe0 con 0x7f96e00b4d70 2024-04-06T13:45:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.196+0000 7f96e7ef9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96e015a940 con 0x7f96e00cf900 2024-04-06T13:45:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.196+0000 7f96e7ef9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96e015ac00 con 0x7f96e015a550 2024-04-06T13:45:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.196+0000 7f96e6496700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f96e00b4d70 0x7f96e00b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55938/0 (socket says 172.21.15.189:55938) 2024-04-06T13:45:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.196+0000 7f96e6496700 1 -- 172.21.15.189:0/3342429267 learned_addr learned my addr 172.21.15.189:0/3342429267 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.196+0000 7f96e5c95700 1 --2- 172.21.15.189:0/3342429267 >> v2:172.21.15.192:3300/0 conn(0x7f96e015a550 0x7f96e00cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.196+0000 7f96e4c93700 1 -- 172.21.15.189:0/3342429267 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802334286 0 0) 0x7f96e00cdbe0 con 0x7f96e00b4d70 2024-04-06T13:45:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.196+0000 7f96e4c93700 1 -- 172.21.15.189:0/3342429267 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96cc0035b0 con 0x7f96e00b4d70 2024-04-06T13:45:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.196+0000 7f96e4c93700 1 -- 172.21.15.189:0/3342429267 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134690441 0 0) 0x7f96e015a940 con 0x7f96e00cf900 2024-04-06T13:45:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.196+0000 7f96e4c93700 1 -- 172.21.15.189:0/3342429267 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96e00cdbe0 con 0x7f96e00cf900 2024-04-06T13:45:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.196+0000 7f96e4c93700 1 -- 172.21.15.189:0/3342429267 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3980183518 0 0) 0x7f96cc0035b0 con 0x7f96e00b4d70 2024-04-06T13:45:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.196+0000 7f96e4c93700 1 -- 172.21.15.189:0/3342429267 >> v2:172.21.15.192:3300/0 conn(0x7f96e015a550 msgr2=0x7f96e00cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:45.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.196+0000 7f96e4c93700 1 --2- 172.21.15.189:0/3342429267 >> v2:172.21.15.192:3300/0 conn(0x7f96e015a550 0x7f96e00cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.196+0000 7f96e4c93700 1 -- 172.21.15.189:0/3342429267 >> v1:172.21.15.192:6789/0 conn(0x7f96e00cf900 legacy=0x7f96e00b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.196+0000 7f96e4c93700 1 -- 172.21.15.189:0/3342429267 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e015aee0 con 0x7f96e00b4d70 2024-04-06T13:45:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.196+0000 7f96e4c93700 1 -- 172.21.15.189:0/3342429267 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f96d80019f0 con 0x7f96e00b4d70 2024-04-06T13:45:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e4c93700 1 -- 172.21.15.189:0/3342429267 >> v1:172.21.15.189:6789/0 conn(0x7f96e00b4d70 legacy=0x7f96e00b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e4c93700 1 --2- 172.21.15.189:0/3342429267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96cc003be0 0x7f96cc003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e4c93700 1 --2- 172.21.15.189:0/3342429267 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96cc004850 0x7f96cc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e4c93700 1 --2- 172.21.15.189:0/3342429267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96cc005210 0x7f96cc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e4c93700 1 -- 172.21.15.189:0/3342429267 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e015ac00 con 0x7f96cc005210 2024-04-06T13:45:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e4c93700 1 -- 172.21.15.189:0/3342429267 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc00c0e0 con 0x7f96cc003be0 2024-04-06T13:45:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e4c93700 1 -- 172.21.15.189:0/3342429267 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc00c2a0 con 0x7f96cc004850 2024-04-06T13:45:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e6496700 1 --2- 172.21.15.189:0/3342429267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96cc005210 0x7f96cc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e5c95700 1 --2- 172.21.15.189:0/3342429267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96cc003be0 0x7f96cc003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e5494700 1 --2- 172.21.15.189:0/3342429267 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96cc004850 0x7f96cc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e6496700 1 -- 172.21.15.189:0/3342429267 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96cc004850 msgr2=0x7f96cc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:45.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e6496700 1 --2- 172.21.15.189:0/3342429267 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96cc004850 0x7f96cc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e6496700 1 -- 172.21.15.189:0/3342429267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96cc003be0 msgr2=0x7f96cc003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:45.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e6496700 1 --2- 172.21.15.189:0/3342429267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96cc003be0 0x7f96cc003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e6496700 1 -- 172.21.15.189:0/3342429267 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96cc0035b0 con 0x7f96cc005210 2024-04-06T13:45:45.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e5494700 1 --2- 172.21.15.189:0/3342429267 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96cc004850 0x7f96cc004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:45.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e6496700 1 -- 172.21.15.189:0/3342429267 reap_dead start 2024-04-06T13:45:45.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e6496700 1 --2- 172.21.15.189:0/3342429267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96cc005210 0x7f96cc00ba10 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f96d800a700 tx=0x7f96cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=242fe4aef7de501 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:45.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e4c93700 1 -- 172.21.15.189:0/3342429267 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d80072d0 con 0x7f96cc005210 2024-04-06T13:45:45.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e4c93700 1 -- 172.21.15.189:0/3342429267 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d8007470 con 0x7f96cc005210 2024-04-06T13:45:45.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.197+0000 7f96e4c93700 1 -- 172.21.15.189:0/3342429267 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d8007820 con 0x7f96cc005210 2024-04-06T13:45:45.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.198+0000 7f96e7ef9700 1 -- 172.21.15.189:0/3342429267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96cc005210 msgr2=0x7f96cc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:45.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.198+0000 7f96e7ef9700 1 --2- 172.21.15.189:0/3342429267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96cc005210 0x7f96cc00ba10 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f96d800a700 tx=0x7f96cc005880 comp rx=0 tx=0).stop 2024-04-06T13:45:45.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.198+0000 7f96e7ef9700 1 -- 172.21.15.189:0/3342429267 shutdown_connections 2024-04-06T13:45:45.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.198+0000 7f96e7ef9700 1 --2- 172.21.15.189:0/3342429267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96cc005210 0x7f96cc00ba10 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.198+0000 7f96e7ef9700 1 -- 172.21.15.189:0/3342429267 >> 172.21.15.189:0/3342429267 conn(0x7f96e00d00b0 msgr2=0x7f96e00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:45.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.198+0000 7f96e7ef9700 1 -- 172.21.15.189:0/3342429267 shutdown_connections 2024-04-06T13:45:45.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.198+0000 7f96e7ef9700 1 -- 172.21.15.189:0/3342429267 wait complete. 2024-04-06T13:45:45.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.199+0000 7f96e7ef9700 1 Processor -- start 2024-04-06T13:45:45.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.199+0000 7f96e7ef9700 1 -- start start 2024-04-06T13:45:45.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.200+0000 7f96e7ef9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96cc005210 0x7f96e01411a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:45.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.200+0000 7f96e7ef9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96e01528e0 0x7f96e01f04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:45.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.200+0000 7f96e7ef9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96e0153e20 0x7f96e006a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:45.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.200+0000 7f96e7ef9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e01416e0 con 0x7f96e0153e20 2024-04-06T13:45:45.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.200+0000 7f96e7ef9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e0152d80 con 0x7f96e01528e0 2024-04-06T13:45:45.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.200+0000 7f96e7ef9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96e0153060 con 0x7f96cc005210 2024-04-06T13:45:45.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.200+0000 7f96e6496700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96e0153e20 0x7f96e006a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:45.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.200+0000 7f96e6496700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96e0153e20 0x7f96e006a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48210/0 (socket says 172.21.15.189:48210) 2024-04-06T13:45:45.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.200+0000 7f96e6496700 1 -- 172.21.15.189:0/2353214913 learned_addr learned my addr 172.21.15.189:0/2353214913 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:45.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.201+0000 7f96e5494700 1 --2- 172.21.15.189:0/2353214913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96e01528e0 0x7f96e01f04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:45.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.201+0000 7f96e5c95700 1 --2- 172.21.15.189:0/2353214913 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96cc005210 0x7f96e01411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:45.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.201+0000 7f96e6496700 1 -- 172.21.15.189:0/2353214913 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96cc005210 msgr2=0x7f96e01411a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:45.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.201+0000 7f96e6496700 1 --2- 172.21.15.189:0/2353214913 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96cc005210 0x7f96e01411a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.201+0000 7f96e6496700 1 -- 172.21.15.189:0/2353214913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96e01528e0 msgr2=0x7f96e01f04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:45.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.201+0000 7f96e6496700 1 --2- 172.21.15.189:0/2353214913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96e01528e0 0x7f96e01f04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.201+0000 7f96e6496700 1 -- 172.21.15.189:0/2353214913 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e0153340 con 0x7f96e0153e20 2024-04-06T13:45:45.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.201+0000 7f96e5494700 1 --2- 172.21.15.189:0/2353214913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96e01528e0 0x7f96e01f04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:45.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.201+0000 7f96e6496700 1 --2- 172.21.15.189:0/2353214913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96e0153e20 0x7f96e006a730 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f96d8002230 tx=0x7f96d8001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:45.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.201+0000 7f96d67fc700 1 -- 172.21.15.189:0/2353214913 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d8001250 con 0x7f96e0153e20 2024-04-06T13:45:45.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.201+0000 7f96d67fc700 1 -- 172.21.15.189:0/2353214913 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d8001980 con 0x7f96e0153e20 2024-04-06T13:45:45.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.201+0000 7f96d67fc700 1 -- 172.21.15.189:0/2353214913 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d801d610 con 0x7f96e0153e20 2024-04-06T13:45:45.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.201+0000 7f96e7ef9700 1 -- 172.21.15.189:0/2353214913 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96e0153620 con 0x7f96e0153e20 2024-04-06T13:45:45.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.201+0000 7f96e7ef9700 1 -- 172.21.15.189:0/2353214913 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e01f0a30 con 0x7f96e0153e20 2024-04-06T13:45:45.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.202+0000 7f96e7ef9700 1 -- 172.21.15.189:0/2353214913 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e0153b40 con 0x7f96e0153e20 2024-04-06T13:45:45.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.202+0000 7f96d67fc700 1 -- 172.21.15.189:0/2353214913 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f96d801a070 con 0x7f96e0153e20 2024-04-06T13:45:45.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.203+0000 7f96d67fc700 1 --2- 172.21.15.189:0/2353214913 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f96c406adb0 0x7f96c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:45.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.203+0000 7f96d67fc700 1 -- 172.21.15.189:0/2353214913 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f96d808a840 con 0x7f96e0153e20 2024-04-06T13:45:45.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.203+0000 7f96e5c95700 1 --2- 172.21.15.189:0/2353214913 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f96c406adb0 0x7f96c406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:45.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.203+0000 7f96e5c95700 1 --2- 172.21.15.189:0/2353214913 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f96c406adb0 0x7f96c406d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f96dc009be0 tx=0x7f96dc009210 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:45.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.207+0000 7f96d67fc700 1 -- 172.21.15.189:0/2353214913 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f96e0153b40 con 0x7f96e0153e20 2024-04-06T13:45:45.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.445+0000 7f96e7ef9700 1 -- 172.21.15.189:0/2353214913 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96e0153b40 con 0x7f96e0153e20 2024-04-06T13:45:45.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.445+0000 7f96d67fc700 1 -- 172.21.15.189:0/2353214913 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+4750 (secure 0 0 0) 0x7f96e0153b40 con 0x7f96e0153e20 2024-04-06T13:45:45.445 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:45.445 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":93,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8525,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3302262361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3302262361},{"type":"v1","addr":"172.21.15.189:6837","nonce":3302262361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":8534,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/229523550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":229523550},{"type":"v1","addr":"172.21.15.192:6834","nonce":229523550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":9739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2560949338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2560949338},{"type":"v1","addr":"172.21.15.192:6835","nonce":2560949338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:45:40.072274+0000","modified":"2024-04-06T13:45:43.092032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":0,"incarnation":89,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/3619731181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3619731181},{"type":"v1","addr":"172.21.15.189:6836","nonce":3619731181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-04-06T13:45:45.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.447+0000 7f96e7ef9700 1 -- 172.21.15.189:0/2353214913 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f96c406adb0 msgr2=0x7f96c406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:45.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.447+0000 7f96e7ef9700 1 --2- 172.21.15.189:0/2353214913 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f96c406adb0 0x7f96c406d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f96dc009be0 tx=0x7f96dc009210 comp rx=0 tx=0).stop 2024-04-06T13:45:45.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.447+0000 7f96e7ef9700 1 -- 172.21.15.189:0/2353214913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96e0153e20 msgr2=0x7f96e006a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:45.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.447+0000 7f96e7ef9700 1 --2- 172.21.15.189:0/2353214913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96e0153e20 0x7f96e006a730 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f96d8002230 tx=0x7f96d8001cb0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.447+0000 7f96e7ef9700 1 -- 172.21.15.189:0/2353214913 shutdown_connections 2024-04-06T13:45:45.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.447+0000 7f96e7ef9700 1 --2- 172.21.15.189:0/2353214913 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f96c406adb0 0x7f96c406d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.447+0000 7f96e7ef9700 1 --2- 172.21.15.189:0/2353214913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96e0153e20 0x7f96e006a730 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.447+0000 7f96e7ef9700 1 --2- 172.21.15.189:0/2353214913 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96cc005210 0x7f96e01411a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.447+0000 7f96e7ef9700 1 --2- 172.21.15.189:0/2353214913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96e01528e0 0x7f96e01f04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.447+0000 7f96e7ef9700 1 -- 172.21.15.189:0/2353214913 >> 172.21.15.189:0/2353214913 conn(0x7f96e00d00b0 msgr2=0x7f96e0072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:45.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.447+0000 7f96e7ef9700 1 -- 172.21.15.189:0/2353214913 shutdown_connections 2024-04-06T13:45:45.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.447+0000 7f96e7ef9700 1 -- 172.21.15.189:0/2353214913 wait complete. 2024-04-06T13:45:45.448 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 93 2024-04-06T13:45:45.642 INFO:tasks.ceph.mds.a.smithi189.stderr:2024-04-06T13:45:45.642+0000 7f71700ee700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T13:45:45.862 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 93, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-06T13:45:40.072274+0000', 'modified': '2024-04-06T13:45:43.092032+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 0, 'incarnation': 89, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.189:6836/3619731181', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 3619731181}, {'type': 'v1', 'addr': '172.21.15.189:6836', 'nonce': 3619731181}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-06T13:45:45.862 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-06T13:45:45.862 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-06T13:45:45.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.988+0000 7fa1b03f3700 1 Processor -- start 2024-04-06T13:45:45.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.989+0000 7fa1b03f3700 1 -- start start 2024-04-06T13:45:45.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.989+0000 7fa1b03f3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa1a815e8a0 0x7fa1a80cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:45.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.989+0000 7fa1b03f3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa1a815eee0 0x7fa1a815f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:45.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.989+0000 7fa1b03f3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1a815f8a0 0x7fa1a806ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:45.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.989+0000 7fa1b03f3700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a815fd10 con 0x7fa1a815e8a0 2024-04-06T13:45:45.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.989+0000 7fa1b03f3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a80669d0 con 0x7fa1a815eee0 2024-04-06T13:45:45.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.989+0000 7fa1b03f3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a806ede0 con 0x7fa1a815f8a0 2024-04-06T13:45:45.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.990+0000 7fa1ae18f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa1a815e8a0 0x7fa1a80cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:45.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.990+0000 7fa1ae18f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa1a815e8a0 0x7fa1a80cbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48214/0 (socket says 172.21.15.189:48214) 2024-04-06T13:45:45.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.990+0000 7fa1ae18f700 1 -- 172.21.15.189:0/1380528283 learned_addr learned my addr 172.21.15.189:0/1380528283 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:45.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.990+0000 7fa1ae990700 1 --2- 172.21.15.189:0/1380528283 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1a815f8a0 0x7fa1a806ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:45.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.990+0000 7fa1ad98e700 1 --2- 172.21.15.189:0/1380528283 >> v2:172.21.15.192:3300/0 conn(0x7fa1a815eee0 0x7fa1a815f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:45.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.990+0000 7fa1ae18f700 1 -- 172.21.15.189:0/1380528283 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1a815f8a0 msgr2=0x7fa1a806ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:45.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.990+0000 7fa1ae18f700 1 --2- 172.21.15.189:0/1380528283 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1a815f8a0 0x7fa1a806ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.990+0000 7fa1ae18f700 1 -- 172.21.15.189:0/1380528283 >> v2:172.21.15.192:3300/0 conn(0x7fa1a815eee0 msgr2=0x7fa1a815f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:45.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.990+0000 7fa1ae18f700 1 --2- 172.21.15.189:0/1380528283 >> v2:172.21.15.192:3300/0 conn(0x7fa1a815eee0 0x7fa1a815f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.990+0000 7fa1ae18f700 1 -- 172.21.15.189:0/1380528283 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a815ec80 con 0x7fa1a815e8a0 2024-04-06T13:45:45.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.990+0000 7fa1ae18f700 1 --2- 172.21.15.189:0/1380528283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1a815e8a0 0x7fa1a80cbc20 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fa1a80723a0 tx=0x7fa1a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0880440819885b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:45.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.990+0000 7fa1ad18d700 1 -- 172.21.15.189:0/1380528283 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a400b870 con 0x7fa1a815e8a0 2024-04-06T13:45:45.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.990+0000 7fa1ad18d700 1 -- 172.21.15.189:0/1380528283 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a400b9d0 con 0x7fa1a815e8a0 2024-04-06T13:45:45.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.990+0000 7fa1ad18d700 1 -- 172.21.15.189:0/1380528283 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a400bd60 con 0x7fa1a815e8a0 2024-04-06T13:45:45.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.991+0000 7fa1b03f3700 1 -- 172.21.15.189:0/1380528283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1a815e8a0 msgr2=0x7fa1a80cbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:45.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.991+0000 7fa1b03f3700 1 --2- 172.21.15.189:0/1380528283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1a815e8a0 0x7fa1a80cbc20 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fa1a80723a0 tx=0x7fa1a4009bd0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.991+0000 7fa1b03f3700 1 -- 172.21.15.189:0/1380528283 shutdown_connections 2024-04-06T13:45:45.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.991+0000 7fa1b03f3700 1 --2- 172.21.15.189:0/1380528283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1a815e8a0 0x7fa1a80cbc20 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.991+0000 7fa1b03f3700 1 --2- 172.21.15.189:0/1380528283 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1a815f8a0 0x7fa1a806ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.991+0000 7fa1b03f3700 1 --2- 172.21.15.189:0/1380528283 >> v2:172.21.15.192:3300/0 conn(0x7fa1a815eee0 0x7fa1a815f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.991+0000 7fa1b03f3700 1 -- 172.21.15.189:0/1380528283 >> 172.21.15.189:0/1380528283 conn(0x7fa1a80cc290 msgr2=0x7fa1a80cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:45.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.991+0000 7fa1b03f3700 1 -- 172.21.15.189:0/1380528283 shutdown_connections 2024-04-06T13:45:45.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.991+0000 7fa1b03f3700 1 -- 172.21.15.189:0/1380528283 wait complete. 2024-04-06T13:45:45.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.992+0000 7fa1b03f3700 1 Processor -- start 2024-04-06T13:45:45.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.992+0000 7fa1b03f3700 1 -- start start 2024-04-06T13:45:45.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.993+0000 7fa1b03f3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1a815e8a0 0x7fa1a80afa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:45.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.993+0000 7fa1b03f3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1a815eee0 0x7fa1a80c57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:45.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.993+0000 7fa1b03f3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1a815f8a0 0x7fa1a8152940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:45.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.993+0000 7fa1b03f3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a80ad810 con 0x7fa1a815f8a0 2024-04-06T13:45:45.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.993+0000 7fa1b03f3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a80acc00 con 0x7fa1a815eee0 2024-04-06T13:45:45.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.993+0000 7fa1b03f3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a80acee0 con 0x7fa1a815e8a0 2024-04-06T13:45:45.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.993+0000 7fa1ae990700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1a815f8a0 0x7fa1a8152940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:45.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.993+0000 7fa1ae990700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1a815f8a0 0x7fa1a8152940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48220/0 (socket says 172.21.15.189:48220) 2024-04-06T13:45:45.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.993+0000 7fa1ae990700 1 -- 172.21.15.189:0/3263139620 learned_addr learned my addr 172.21.15.189:0/3263139620 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:45.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.993+0000 7fa1ad98e700 1 --2- 172.21.15.189:0/3263139620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1a815eee0 0x7fa1a80c57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:45.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.993+0000 7fa1ae18f700 1 --2- 172.21.15.189:0/3263139620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1a815e8a0 0x7fa1a80afa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:45.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.993+0000 7fa1ae990700 1 -- 172.21.15.189:0/3263139620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1a815e8a0 msgr2=0x7fa1a80afa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:45.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.993+0000 7fa1ae990700 1 --2- 172.21.15.189:0/3263139620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1a815e8a0 0x7fa1a80afa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.993+0000 7fa1ae990700 1 -- 172.21.15.189:0/3263139620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1a815eee0 msgr2=0x7fa1a80c57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:45.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.993+0000 7fa1ae990700 1 --2- 172.21.15.189:0/3263139620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1a815eee0 0x7fa1a80c57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:45.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.993+0000 7fa1ae990700 1 -- 172.21.15.189:0/3263139620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a4009550 con 0x7fa1a815f8a0 2024-04-06T13:45:45.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.993+0000 7fa1ae990700 1 --2- 172.21.15.189:0/3263139620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1a815f8a0 0x7fa1a8152940 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0007e40 tx=0x7fa1a000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:45.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.994+0000 7fa19affd700 1 -- 172.21.15.189:0/3263139620 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a000ed30 con 0x7fa1a815f8a0 2024-04-06T13:45:45.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.994+0000 7fa19affd700 1 -- 172.21.15.189:0/3263139620 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a0014420 con 0x7fa1a815f8a0 2024-04-06T13:45:45.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.994+0000 7fa19affd700 1 -- 172.21.15.189:0/3263139620 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a00147d0 con 0x7fa1a815f8a0 2024-04-06T13:45:45.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.994+0000 7fa1b03f3700 1 -- 172.21.15.189:0/3263139620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a80ad1c0 con 0x7fa1a815f8a0 2024-04-06T13:45:45.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.994+0000 7fa1b03f3700 1 -- 172.21.15.189:0/3263139620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a80ac8d0 con 0x7fa1a815f8a0 2024-04-06T13:45:45.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.995+0000 7fa19affd700 1 -- 172.21.15.189:0/3263139620 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fa1a0014930 con 0x7fa1a815f8a0 2024-04-06T13:45:45.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.996+0000 7fa19affd700 1 --2- 172.21.15.189:0/3263139620 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa18c06adb0 0x7fa18c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:45.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.996+0000 7fa1ae18f700 1 --2- 172.21.15.189:0/3263139620 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa18c06adb0 0x7fa18c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:45.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.996+0000 7fa19affd700 1 -- 172.21.15.189:0/3263139620 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa1a0089d30 con 0x7fa1a815f8a0 2024-04-06T13:45:45.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.996+0000 7fa1ae18f700 1 --2- 172.21.15.189:0/3263139620 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa18c06adb0 0x7fa18c06d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4014040 tx=0x7fa1a4008030 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:46.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.997+0000 7fa1bc5a7b80 1 Processor -- start 2024-04-06T13:45:46.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.998+0000 7fa1bc5a7b80 1 -- start start 2024-04-06T13:45:46.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.998+0000 7fa1bc5a7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 0x55c22a946a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.998+0000 7fa1bc5a7b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a947fc0 0x55c22a91e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.998+0000 7fa1bc5a7b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a91d980 0x55c22a91eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.998+0000 7fa1bc5a7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c22a877940 con 0x55c22a942c20 2024-04-06T13:45:46.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.998+0000 7fa1bc5a7b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c22a875300 con 0x55c22a947fc0 2024-04-06T13:45:46.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.998+0000 7fa1bc5a7b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c22a870bf0 con 0x55c22a91d980 2024-04-06T13:45:46.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.998+0000 7fa1ae18f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 0x55c22a946a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.999+0000 7fa1ae18f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 0x55c22a946a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48222/0 (socket says 172.21.15.189:48222) 2024-04-06T13:45:46.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.999+0000 7fa1ae18f700 1 -- 172.21.15.189:0/1284233051 learned_addr learned my addr 172.21.15.189:0/1284233051 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:46.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.999+0000 7fa1ae990700 1 --2- 172.21.15.189:0/1284233051 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a91d980 0x55c22a91eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.999+0000 7fa1ad98e700 1 --2- 172.21.15.189:0/1284233051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a947fc0 0x55c22a91e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.999+0000 7fa1ae18f700 1 -- 172.21.15.189:0/1284233051 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a91d980 msgr2=0x55c22a91eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:46.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.999+0000 7fa1ae18f700 1 --2- 172.21.15.189:0/1284233051 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a91d980 0x55c22a91eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.999+0000 7fa1ae18f700 1 -- 172.21.15.189:0/1284233051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a947fc0 msgr2=0x55c22a91e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:46.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.999+0000 7fa1ae18f700 1 --2- 172.21.15.189:0/1284233051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a947fc0 0x55c22a91e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.999+0000 7fa1ae18f700 1 -- 172.21.15.189:0/1284233051 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c22a69eb10 con 0x55c22a942c20 2024-04-06T13:45:46.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.999+0000 7fa1ae18f700 1 --2- 172.21.15.189:0/1284233051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 0x55c22a946a60 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4009fd0 tx=0x7fa1a4010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9260fe73f2a50d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:46.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.999+0000 7fa18bfff700 1 -- 172.21.15.189:0/1284233051 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a401f070 con 0x55c22a942c20 2024-04-06T13:45:46.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.999+0000 7fa18bfff700 1 -- 172.21.15.189:0/1284233051 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a40278c0 con 0x55c22a942c20 2024-04-06T13:45:46.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:45.999+0000 7fa18bfff700 1 -- 172.21.15.189:0/1284233051 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a4027c50 con 0x55c22a942c20 2024-04-06T13:45:46.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.000+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/1284233051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 msgr2=0x55c22a946a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:46.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.000+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/1284233051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 0x55c22a946a60 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4009fd0 tx=0x7fa1a4010000 comp rx=0 tx=0).stop 2024-04-06T13:45:46.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.000+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/1284233051 shutdown_connections 2024-04-06T13:45:46.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.000+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/1284233051 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a91d980 0x55c22a91eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.000+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/1284233051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 0x55c22a946a60 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.000+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/1284233051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a947fc0 0x55c22a91e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.000+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/1284233051 >> 172.21.15.189:0/1284233051 conn(0x55c22a87e650 msgr2=0x55c22a9653e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:46.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.000+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/1284233051 shutdown_connections 2024-04-06T13:45:46.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.000+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/1284233051 wait complete. 2024-04-06T13:45:46.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.000+0000 7fa1bc5a7b80 1 -- start start 2024-04-06T13:45:46.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.001+0000 7fa1bc5a7b80 1 Processor -- start 2024-04-06T13:45:46.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.001+0000 7fa1bc5a7b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a91d980 0x55c22a95f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.001+0000 7fa1bc5a7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 0x55c22a932b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.001+0000 7fa1bc5a7b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a947fc0 0x55c22a965610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1bc5a7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c22a80cd30 con 0x55c22a942c20 2024-04-06T13:45:46.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1bc5a7b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c22a80c9d0 con 0x55c22a947fc0 2024-04-06T13:45:46.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1bc5a7b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c22a6f4650 con 0x55c22a91d980 2024-04-06T13:45:46.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1bc5a7b80 10 client.0 _ll_register_callbacks cb 0x55c22a816420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T13:45:46.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1ad98e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 0x55c22a932b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1ad98e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 0x55c22a932b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48224/0 (socket says 172.21.15.189:48224) 2024-04-06T13:45:46.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1ad98e700 1 -- 172.21.15.189:0/3261512742 learned_addr learned my addr 172.21.15.189:0/3261512742 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:46.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1ae990700 1 --2- 172.21.15.189:0/3261512742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a947fc0 0x55c22a965610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1ae18f700 1 --2- 172.21.15.189:0/3261512742 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a91d980 0x55c22a95f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1ad98e700 1 -- 172.21.15.189:0/3261512742 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a91d980 msgr2=0x55c22a95f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:46.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1ad98e700 1 --2- 172.21.15.189:0/3261512742 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a91d980 0x55c22a95f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1ad98e700 1 -- 172.21.15.189:0/3261512742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a947fc0 msgr2=0x55c22a965610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:46.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1ad98e700 1 --2- 172.21.15.189:0/3261512742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a947fc0 0x55c22a965610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1ad98e700 1 -- 172.21.15.189:0/3261512742 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a815ec80 con 0x55c22a942c20 2024-04-06T13:45:46.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1ad98e700 1 --2- 172.21.15.189:0/3261512742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 0x55c22a932b10 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7fa19c0145d0 tx=0x7fa19c0148e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e94ecc6bc365db06 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:46.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1897fa700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T13:45:46.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1897fa700 1 -- 172.21.15.189:0/3261512742 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c004440 con 0x55c22a942c20 2024-04-06T13:45:46.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1897fa700 1 -- 172.21.15.189:0/3261512742 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa19c01a030 con 0x55c22a942c20 2024-04-06T13:45:46.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/3261512742 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c22a831330 con 0x55c22a942c20 2024-04-06T13:45:46.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1897fa700 1 -- 172.21.15.189:0/3261512742 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa19c0045a0 con 0x55c22a942c20 2024-04-06T13:45:46.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1bc5a7b80 10 client.10872 fetch_fsmap learned FSMap version 93 2024-04-06T13:45:46.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/3261512742 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=2+}) v3 -- 0x55c22a836e50 con 0x55c22a942c20 2024-04-06T13:45:46.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.002+0000 7fa1897fa700 1 -- 172.21.15.189:0/3261512742 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c22a831330 con 0x55c22a942c20 2024-04-06T13:45:46.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.003+0000 7fa1897fa700 1 -- 172.21.15.189:0/3261512742 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 93) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa19c0036a0 con 0x55c22a942c20 2024-04-06T13:45:46.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.003+0000 7fa1bc5a7b80 10 client.10872 fetch_fsmap finished waiting for FSMap version 93 2024-04-06T13:45:46.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.003+0000 7fa1bc5a7b80 10 client.10872 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-06T13:45:46.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.003+0000 7fa1bc5a7b80 20 client.10872 populate_metadata read hostname 'smithi189' 2024-04-06T13:45:46.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.003+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/3261512742 >> [v2:172.21.15.189:6834/3619731181,v1:172.21.15.189:6836/3619731181] conn(0x55c22a965b50 0x55c22a930d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.003+0000 7fa1bc5a7b80 4 client.10872 mds_command: new command op to 10716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-06T13:45:46.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.003+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/3261512742 --> [v2:172.21.15.189:6834/3619731181,v1:172.21.15.189:6836/3619731181] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c22a8696b0 con 0x55c22a965b50 2024-04-06T13:45:46.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.003+0000 7fa1ae18f700 1 --2- 172.21.15.189:0/3261512742 >> [v2:172.21.15.189:6834/3619731181,v1:172.21.15.189:6836/3619731181] conn(0x55c22a965b50 0x55c22a930d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.003+0000 7fa1897fa700 10 client.10872 ms_handle_connect on v2:172.21.15.189:6834/3619731181 2024-04-06T13:45:46.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.003+0000 7fa1ae18f700 1 --2- 172.21.15.189:0/3261512742 >> [v2:172.21.15.189:6834/3619731181,v1:172.21.15.189:6836/3619731181] conn(0x55c22a965b50 0x55c22a930d70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=316887d645babd7c server_cookie=65cacf75b4cac91f in_seq=0 out_seq=0 2024-04-06T13:45:46.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.003+0000 7fa1897fa700 1 -- 172.21.15.189:0/3261512742 <== mds.0 v2:172.21.15.189:6834/3619731181 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c22a8696b0 con 0x55c22a965b50 2024-04-06T13:45:46.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.004+0000 7fa1897fa700 10 client.10872 handle_command_reply: tid=0 2024-04-06T13:45:46.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.004+0000 7fa1bc5a7b80 1 client.10872 shutdown 2024-04-06T13:45:46.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.004+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/3261512742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 msgr2=0x55c22a932b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:46.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.004+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/3261512742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 0x55c22a932b10 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7fa19c0145d0 tx=0x7fa19c0148e0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.004+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/3261512742 shutdown_connections 2024-04-06T13:45:46.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.004+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/3261512742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a947fc0 0x55c22a965610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.004+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/3261512742 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a91d980 0x55c22a95f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.004+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/3261512742 >> [v2:172.21.15.189:6834/3619731181,v1:172.21.15.189:6836/3619731181] conn(0x55c22a965b50 0x55c22a930d70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.004+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/3261512742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 0x55c22a932b10 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.004+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/3261512742 >> 172.21.15.189:0/3261512742 conn(0x55c22a87e650 msgr2=0x55c22a9211a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:46.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.004+0000 7fa1897fa700 0 client.10872 ms_handle_reset on v2:172.21.15.189:6834/3619731181 2024-04-06T13:45:46.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.004+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/3261512742 shutdown_connections 2024-04-06T13:45:46.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.004+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/3261512742 wait complete. 2024-04-06T13:45:46.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.004+0000 7fa1bc5a7b80 20 client.10872 trim_cache size 0 max 16384 2024-04-06T13:45:46.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.021+0000 7fa1bc5a7b80 1 Processor -- start 2024-04-06T13:45:46.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.021+0000 7fa1bc5a7b80 1 -- start start 2024-04-06T13:45:46.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.022+0000 7fa1bc5a7b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a91d980 0x55c22a91b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.022+0000 7fa1bc5a7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 0x55c22a946b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.022+0000 7fa1bc5a7b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a947fc0 0x55c22a94a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.022+0000 7fa1bc5a7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c22a977cf0 con 0x55c22a942c20 2024-04-06T13:45:46.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.022+0000 7fa1bc5a7b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c22a9878d0 con 0x55c22a91d980 2024-04-06T13:45:46.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.022+0000 7fa1bc5a7b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c22a91a880 con 0x55c22a947fc0 2024-04-06T13:45:46.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.022+0000 7fa1ad98e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 0x55c22a946b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.022+0000 7fa1ad98e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 0x55c22a946b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48226/0 (socket says 172.21.15.189:48226) 2024-04-06T13:45:46.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.022+0000 7fa1ad98e700 1 -- 172.21.15.189:0/1468109521 learned_addr learned my addr 172.21.15.189:0/1468109521 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:46.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.022+0000 7fa1ae18f700 1 --2- 172.21.15.189:0/1468109521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a91d980 0x55c22a91b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.022+0000 7fa1ae990700 1 --2- 172.21.15.189:0/1468109521 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a947fc0 0x55c22a94a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.022+0000 7fa1ad98e700 1 -- 172.21.15.189:0/1468109521 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a947fc0 msgr2=0x55c22a94a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:46.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.022+0000 7fa1ad98e700 1 --2- 172.21.15.189:0/1468109521 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a947fc0 0x55c22a94a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.022+0000 7fa1ad98e700 1 -- 172.21.15.189:0/1468109521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a91d980 msgr2=0x55c22a91b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:46.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.022+0000 7fa1ad98e700 1 --2- 172.21.15.189:0/1468109521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a91d980 0x55c22a91b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.022+0000 7fa1ad98e700 1 -- 172.21.15.189:0/1468109521 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a815b6f0 con 0x55c22a942c20 2024-04-06T13:45:46.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.023+0000 7fa1ad98e700 1 --2- 172.21.15.189:0/1468109521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 0x55c22a946b50 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7fa19c000c00 tx=0x7fa19c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f963ad7218954ae6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:46.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.023+0000 7fa198da5700 1 -- 172.21.15.189:0/1468109521 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c003c00 con 0x55c22a942c20 2024-04-06T13:45:46.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.023+0000 7fa198da5700 1 -- 172.21.15.189:0/1468109521 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19c003e10 con 0x55c22a942c20 2024-04-06T13:45:46.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.023+0000 7fa198da5700 1 -- 172.21.15.189:0/1468109521 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c004240 con 0x55c22a942c20 2024-04-06T13:45:46.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.023+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/1468109521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 msgr2=0x55c22a946b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:46.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.023+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/1468109521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 0x55c22a946b50 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7fa19c000c00 tx=0x7fa19c019000 comp rx=0 tx=0).stop 2024-04-06T13:45:46.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.023+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/1468109521 shutdown_connections 2024-04-06T13:45:46.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.023+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/1468109521 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a947fc0 0x55c22a94a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.023+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/1468109521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a91d980 0x55c22a91b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.023+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/1468109521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a942c20 0x55c22a946b50 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.023+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/1468109521 >> 172.21.15.189:0/1468109521 conn(0x55c22a87e650 msgr2=0x55c22a9165c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:46.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.023+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/1468109521 shutdown_connections 2024-04-06T13:45:46.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.023+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/1468109521 wait complete. 2024-04-06T13:45:46.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.024+0000 7fa1bc5a7b80 1 -- start start 2024-04-06T13:45:46.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.024+0000 7fa1bc5a7b80 1 Processor -- start 2024-04-06T13:45:46.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1bc5a7b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a91d980 0x55c22a9cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1bc5a7b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a942c20 0x55c22a945070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1bc5a7b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a947fc0 0x55c22a9455b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1bc5a7b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c22a91f9f0 con 0x55c22a947fc0 2024-04-06T13:45:46.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1bc5a7b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c22a941a10 con 0x55c22a91d980 2024-04-06T13:45:46.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1bc5a7b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c22a932d50 con 0x55c22a942c20 2024-04-06T13:45:46.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1bc5a7b80 10 client.0 _ll_register_callbacks cb 0x55c22a816420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T13:45:46.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1ae990700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a947fc0 0x55c22a9455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1ae990700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a947fc0 0x55c22a9455b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48230/0 (socket says 172.21.15.189:48230) 2024-04-06T13:45:46.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1ae990700 1 -- 172.21.15.189:0/2408596738 learned_addr learned my addr 172.21.15.189:0/2408596738 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:46.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1ae18f700 1 --2- 172.21.15.189:0/2408596738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a91d980 0x55c22a9cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1ad98e700 1 --2- 172.21.15.189:0/2408596738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a942c20 0x55c22a945070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1ae990700 1 -- 172.21.15.189:0/2408596738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a942c20 msgr2=0x55c22a945070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:46.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1ae990700 1 --2- 172.21.15.189:0/2408596738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a942c20 0x55c22a945070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1ae990700 1 -- 172.21.15.189:0/2408596738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a91d980 msgr2=0x55c22a9cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:46.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1ae990700 1 --2- 172.21.15.189:0/2408596738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a91d980 0x55c22a9cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1ae990700 1 -- 172.21.15.189:0/2408596738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c22a836e50 con 0x55c22a947fc0 2024-04-06T13:45:46.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1ad98e700 1 --2- 172.21.15.189:0/2408596738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a942c20 0x55c22a945070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:46.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1ae18f700 1 --2- 172.21.15.189:0/2408596738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a91d980 0x55c22a9cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:46.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1ae990700 1 --2- 172.21.15.189:0/2408596738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a947fc0 0x55c22a9455b0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fa1a000e710 tx=0x7fa1a00590e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92989589f4a339ce server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:46.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.025+0000 7fa1897fa700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T13:45:46.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.026+0000 7fa1897fa700 1 -- 172.21.15.189:0/2408596738 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0014de0 con 0x55c22a947fc0 2024-04-06T13:45:46.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.026+0000 7fa1897fa700 1 -- 172.21.15.189:0/2408596738 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa1a0060c80 con 0x55c22a947fc0 2024-04-06T13:45:46.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.026+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/2408596738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c22a920d50 con 0x55c22a947fc0 2024-04-06T13:45:46.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.026+0000 7fa1897fa700 1 -- 172.21.15.189:0/2408596738 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1a0058af0 con 0x55c22a947fc0 2024-04-06T13:45:46.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.026+0000 7fa1bc5a7b80 10 client.10878 fetch_fsmap learned FSMap version 93 2024-04-06T13:45:46.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.026+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/2408596738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=2+}) v3 -- 0x55c22a6f6c80 con 0x55c22a947fc0 2024-04-06T13:45:46.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.026+0000 7fa1897fa700 1 -- 172.21.15.189:0/2408596738 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c22a920d50 con 0x55c22a947fc0 2024-04-06T13:45:46.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.026+0000 7fa1897fa700 1 -- 172.21.15.189:0/2408596738 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 93) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa1a0057d80 con 0x55c22a947fc0 2024-04-06T13:45:46.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.026+0000 7fa1bc5a7b80 10 client.10878 fetch_fsmap finished waiting for FSMap version 93 2024-04-06T13:45:46.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.026+0000 7fa1bc5a7b80 10 client.10878 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-06T13:45:46.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.026+0000 7fa1bc5a7b80 20 client.10878 populate_metadata read hostname 'smithi189' 2024-04-06T13:45:46.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.026+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/2408596738 >> [v2:172.21.15.189:6834/3619731181,v1:172.21.15.189:6836/3619731181] conn(0x55c22a968dd0 0x55c22a9691b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.026+0000 7fa1bc5a7b80 4 client.10878 mds_command: new command op to 10716 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-06T13:45:46.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.026+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/2408596738 --> [v2:172.21.15.189:6834/3619731181,v1:172.21.15.189:6836/3619731181] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fa1a40278c0 con 0x55c22a968dd0 2024-04-06T13:45:46.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.026+0000 7fa1ae18f700 1 --2- 172.21.15.189:0/2408596738 >> [v2:172.21.15.189:6834/3619731181,v1:172.21.15.189:6836/3619731181] conn(0x55c22a968dd0 0x55c22a9691b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.026+0000 7fa1897fa700 10 client.10878 ms_handle_connect on v2:172.21.15.189:6834/3619731181 2024-04-06T13:45:46.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.026+0000 7fa1ae18f700 1 --2- 172.21.15.189:0/2408596738 >> [v2:172.21.15.189:6834/3619731181,v1:172.21.15.189:6836/3619731181] conn(0x55c22a968dd0 0x55c22a9691b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e214b263f148354c server_cookie=2800a9bae70c6d63 in_seq=0 out_seq=0 2024-04-06T13:45:46.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.027+0000 7fa1897fa700 1 -- 172.21.15.189:0/2408596738 <== mds.0 v2:172.21.15.189:6834/3619731181 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x7fa1a40278c0 con 0x55c22a968dd0 2024-04-06T13:45:46.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.027+0000 7fa1897fa700 10 client.10878 handle_command_reply: tid=0 2024-04-06T13:45:46.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.027+0000 7fa1bc5a7b80 1 client.10878 shutdown 2024-04-06T13:45:46.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.027+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/2408596738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a947fc0 msgr2=0x55c22a9455b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:46.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.027+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/2408596738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a947fc0 0x55c22a9455b0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fa1a000e710 tx=0x7fa1a00590e0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.027+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/2408596738 shutdown_connections 2024-04-06T13:45:46.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.027+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/2408596738 >> [v2:172.21.15.189:6834/3619731181,v1:172.21.15.189:6836/3619731181] conn(0x55c22a968dd0 0x55c22a9691b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.027+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/2408596738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22a947fc0 0x55c22a9455b0 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.027+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/2408596738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c22a91d980 0x55c22a9cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.027+0000 7fa1897fa700 0 client.10878 ms_handle_reset on v2:172.21.15.189:6834/3619731181 2024-04-06T13:45:46.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.027+0000 7fa1bc5a7b80 1 --2- 172.21.15.189:0/2408596738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c22a942c20 0x55c22a945070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.027+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/2408596738 >> 172.21.15.189:0/2408596738 conn(0x55c22a87e650 msgr2=0x55c22a943210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:46.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.027+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/2408596738 shutdown_connections 2024-04-06T13:45:46.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.027+0000 7fa1bc5a7b80 1 -- 172.21.15.189:0/2408596738 wait complete. 2024-04-06T13:45:46.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.027+0000 7fa1bc5a7b80 20 client.10878 trim_cache size 0 max 16384 2024-04-06T13:45:46.034 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:45:46.034 INFO:teuthology.orchestra.run.smithi189.stdout: "cluster_fsid": "e30041de-e9a3-4351-8ca7-29d7f3e3728c", 2024-04-06T13:45:46.034 INFO:teuthology.orchestra.run.smithi189.stdout: "whoami": 0, 2024-04-06T13:45:46.034 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 10716, 2024-04-06T13:45:46.034 INFO:teuthology.orchestra.run.smithi189.stdout: "want_state": "up:active", 2024-04-06T13:45:46.034 INFO:teuthology.orchestra.run.smithi189.stdout: "state": "up:active", 2024-04-06T13:45:46.034 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs", 2024-04-06T13:45:46.034 INFO:teuthology.orchestra.run.smithi189.stdout: "rank_uptime": 5.9530162679999998, 2024-04-06T13:45:46.034 INFO:teuthology.orchestra.run.smithi189.stdout: "mdsmap_epoch": 93, 2024-04-06T13:45:46.034 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch": 209, 2024-04-06T13:45:46.034 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch_barrier": 209, 2024-04-06T13:45:46.035 INFO:teuthology.orchestra.run.smithi189.stdout: "uptime": 11.306603555000001 2024-04-06T13:45:46.035 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:45:46.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.027+0000 7fa1b03f3700 1 -- 172.21.15.189:0/3263139620 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa18c06adb0 msgr2=0x7fa18c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:46.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.027+0000 7fa1b03f3700 1 --2- 172.21.15.189:0/3263139620 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa18c06adb0 0x7fa18c06d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4014040 tx=0x7fa1a4008030 comp rx=0 tx=0).stop 2024-04-06T13:45:46.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.027+0000 7fa1b03f3700 1 -- 172.21.15.189:0/3263139620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1a815f8a0 msgr2=0x7fa1a8152940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:46.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.027+0000 7fa1b03f3700 1 --2- 172.21.15.189:0/3263139620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1a815f8a0 0x7fa1a8152940 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0007e40 tx=0x7fa1a000e3c0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.028+0000 7fa1b03f3700 1 -- 172.21.15.189:0/3263139620 shutdown_connections 2024-04-06T13:45:46.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.028+0000 7fa1b03f3700 1 --2- 172.21.15.189:0/3263139620 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa18c06adb0 0x7fa18c06d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.028+0000 7fa1b03f3700 1 --2- 172.21.15.189:0/3263139620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1a815f8a0 0x7fa1a8152940 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.028+0000 7fa1b03f3700 1 --2- 172.21.15.189:0/3263139620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1a815e8a0 0x7fa1a80afa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.028+0000 7fa1b03f3700 1 --2- 172.21.15.189:0/3263139620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1a815eee0 0x7fa1a80c57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.028+0000 7fa1b03f3700 1 -- 172.21.15.189:0/3263139620 >> 172.21.15.189:0/3263139620 conn(0x7fa1a80cc290 msgr2=0x7fa1a815c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:46.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.028+0000 7fa1b03f3700 1 -- 172.21.15.189:0/3263139620 shutdown_connections 2024-04-06T13:45:46.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.028+0000 7fa1b03f3700 1 -- 172.21.15.189:0/3263139620 wait complete. 2024-04-06T13:45:46.040 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T13:45:46.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.163+0000 7f6632341700 1 Processor -- start 2024-04-06T13:45:46.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.163+0000 7f6632341700 1 -- start start 2024-04-06T13:45:46.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.164+0000 7f6632341700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f662c06ae40 0x7f662c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.164+0000 7f6632341700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f662c15ddf0 0x7f662c15e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.164+0000 7f6632341700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f662c0b3990 con 0x7f662c15d7b0 2024-04-06T13:45:46.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.164+0000 7f6632341700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f662c0b3060 con 0x7f662c06ae40 2024-04-06T13:45:46.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.164+0000 7f6632341700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f662c0b3340 con 0x7f662c15ddf0 2024-04-06T13:45:46.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.164+0000 7f662bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f662c15d7b0 0x7f662c06a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55946/0 (socket says 172.21.15.189:55946) 2024-04-06T13:45:46.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.164+0000 7f662bfff700 1 -- 172.21.15.189:0/2750524186 learned_addr learned my addr 172.21.15.189:0/2750524186 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:46.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.164+0000 7f662b7fe700 1 --2- 172.21.15.189:0/2750524186 >> v2:172.21.15.192:3300/0 conn(0x7f662c06ae40 0x7f662c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.164+0000 7f66308de700 1 --2- 172.21.15.189:0/2750524186 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f662c15ddf0 0x7f662c15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.164+0000 7f662affd700 1 -- 172.21.15.189:0/2750524186 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369536664 0 0) 0x7f662c0b3990 con 0x7f662c15d7b0 2024-04-06T13:45:46.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.164+0000 7f662affd700 1 -- 172.21.15.189:0/2750524186 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66140035b0 con 0x7f662c15d7b0 2024-04-06T13:45:46.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.164+0000 7f662affd700 1 -- 172.21.15.189:0/2750524186 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2664896010 0 0) 0x7f66140035b0 con 0x7f662c15d7b0 2024-04-06T13:45:46.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662affd700 1 -- 172.21.15.189:0/2750524186 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f662c15ddf0 msgr2=0x7f662c15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:46.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662affd700 1 --2- 172.21.15.189:0/2750524186 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f662c15ddf0 0x7f662c15e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662affd700 1 -- 172.21.15.189:0/2750524186 >> v2:172.21.15.192:3300/0 conn(0x7f662c06ae40 msgr2=0x7f662c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:46.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662affd700 1 --2- 172.21.15.189:0/2750524186 >> v2:172.21.15.192:3300/0 conn(0x7f662c06ae40 0x7f662c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662affd700 1 -- 172.21.15.189:0/2750524186 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662c0b3620 con 0x7f662c15d7b0 2024-04-06T13:45:46.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662affd700 1 -- 172.21.15.189:0/2750524186 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f6620001670 con 0x7f662c15d7b0 2024-04-06T13:45:46.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662b7fe700 1 --2- 172.21.15.189:0/2750524186 >> v2:172.21.15.192:3300/0 conn(0x7f662c06ae40 0x7f662c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:45:46.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662affd700 1 -- 172.21.15.189:0/2750524186 >> v1:172.21.15.189:6789/0 conn(0x7f662c15d7b0 legacy=0x7f662c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:46.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662affd700 1 --2- 172.21.15.189:0/2750524186 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6614003a00 0x7f6614003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662affd700 1 --2- 172.21.15.189:0/2750524186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66140043f0 0x7f66140048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662affd700 1 --2- 172.21.15.189:0/2750524186 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6614004fe0 0x7f661400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662affd700 1 -- 172.21.15.189:0/2750524186 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662c0b3060 con 0x7f66140043f0 2024-04-06T13:45:46.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662affd700 1 -- 172.21.15.189:0/2750524186 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662c0b3340 con 0x7f6614004fe0 2024-04-06T13:45:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662affd700 1 -- 172.21.15.189:0/2750524186 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661400bf80 con 0x7f6614003a00 2024-04-06T13:45:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f66308de700 1 --2- 172.21.15.189:0/2750524186 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f662c15ddf0 0x7f662c15e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:45:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662b7fe700 1 --2- 172.21.15.189:0/2750524186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66140043f0 0x7f66140048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662bfff700 1 --2- 172.21.15.189:0/2750524186 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6614003a00 0x7f6614003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f66308de700 1 --2- 172.21.15.189:0/2750524186 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6614004fe0 0x7f661400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662b7fe700 1 -- 172.21.15.189:0/2750524186 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6614003a00 msgr2=0x7f6614003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662b7fe700 1 --2- 172.21.15.189:0/2750524186 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6614003a00 0x7f6614003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662b7fe700 1 -- 172.21.15.189:0/2750524186 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6614004fe0 msgr2=0x7f661400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662b7fe700 1 --2- 172.21.15.189:0/2750524186 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6614004fe0 0x7f661400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662b7fe700 1 -- 172.21.15.189:0/2750524186 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66140035b0 con 0x7f66140043f0 2024-04-06T13:45:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662b7fe700 1 --2- 172.21.15.189:0/2750524186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66140043f0 0x7f66140048d0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f661c009d00 tx=0x7f6614005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=490203b2227a78f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662affd700 1 -- 172.21.15.189:0/2750524186 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c009e50 con 0x7f66140043f0 2024-04-06T13:45:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662affd700 1 -- 172.21.15.189:0/2750524186 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661c004030 con 0x7f66140043f0 2024-04-06T13:45:46.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.165+0000 7f662affd700 1 -- 172.21.15.189:0/2750524186 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c0043e0 con 0x7f66140043f0 2024-04-06T13:45:46.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.166+0000 7f6632341700 1 -- 172.21.15.189:0/2750524186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66140043f0 msgr2=0x7f66140048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:46.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.166+0000 7f6632341700 1 --2- 172.21.15.189:0/2750524186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66140043f0 0x7f66140048d0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f661c009d00 tx=0x7f6614005880 comp rx=0 tx=0).stop 2024-04-06T13:45:46.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.166+0000 7f66308de700 1 -- 172.21.15.189:0/2750524186 reap_dead start 2024-04-06T13:45:46.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.166+0000 7f6632341700 1 -- 172.21.15.189:0/2750524186 shutdown_connections 2024-04-06T13:45:46.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.166+0000 7f6632341700 1 -- 172.21.15.189:0/2750524186 >> 172.21.15.189:0/2750524186 conn(0x7f662c0d00b0 msgr2=0x7f662c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:46.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.166+0000 7f6632341700 1 -- 172.21.15.189:0/2750524186 shutdown_connections 2024-04-06T13:45:46.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.166+0000 7f6632341700 1 -- 172.21.15.189:0/2750524186 wait complete. 2024-04-06T13:45:46.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.167+0000 7f6632341700 1 Processor -- start 2024-04-06T13:45:46.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.167+0000 7f6632341700 1 -- start start 2024-04-06T13:45:46.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.168+0000 7f6632341700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66140043f0 0x7f662c14a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.168+0000 7f6632341700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f662c1583f0 0x7f662c158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.168+0000 7f6632341700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f662c0aebf0 0x7f662c0af070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.168+0000 7f6632341700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662c14a670 con 0x7f662c0aebf0 2024-04-06T13:45:46.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.168+0000 7f6632341700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662c158db0 con 0x7f662c1583f0 2024-04-06T13:45:46.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.168+0000 7f6632341700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f662c0af940 con 0x7f66140043f0 2024-04-06T13:45:46.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.168+0000 7f66308de700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f662c0aebf0 0x7f662c0af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.168+0000 7f66308de700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f662c0aebf0 0x7f662c0af070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48256/0 (socket says 172.21.15.189:48256) 2024-04-06T13:45:46.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.168+0000 7f66308de700 1 -- 172.21.15.189:0/2553974374 learned_addr learned my addr 172.21.15.189:0/2553974374 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:46.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.168+0000 7f662b7fe700 1 --2- 172.21.15.189:0/2553974374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f662c1583f0 0x7f662c158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.168+0000 7f662bfff700 1 --2- 172.21.15.189:0/2553974374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66140043f0 0x7f662c14a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.168+0000 7f66308de700 1 -- 172.21.15.189:0/2553974374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66140043f0 msgr2=0x7f662c14a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:46.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.168+0000 7f66308de700 1 --2- 172.21.15.189:0/2553974374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66140043f0 0x7f662c14a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.168+0000 7f66308de700 1 -- 172.21.15.189:0/2553974374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f662c1583f0 msgr2=0x7f662c158870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:46.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.168+0000 7f66308de700 1 --2- 172.21.15.189:0/2553974374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f662c1583f0 0x7f662c158870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.168+0000 7f66308de700 1 -- 172.21.15.189:0/2553974374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662c0afc20 con 0x7f662c0aebf0 2024-04-06T13:45:46.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.168+0000 7f662b7fe700 1 --2- 172.21.15.189:0/2553974374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f662c1583f0 0x7f662c158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:45:46.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.168+0000 7f662bfff700 1 --2- 172.21.15.189:0/2553974374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66140043f0 0x7f662c14a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:45:46.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.168+0000 7f66308de700 1 --2- 172.21.15.189:0/2553974374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f662c0aebf0 0x7f662c0af070 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f662c06c7d0 tx=0x7f6624011790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:46.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.169+0000 7f6628ff9700 1 -- 172.21.15.189:0/2553974374 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662400e070 con 0x7f662c0aebf0 2024-04-06T13:45:46.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.169+0000 7f6628ff9700 1 -- 172.21.15.189:0/2553974374 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6624003a20 con 0x7f662c0aebf0 2024-04-06T13:45:46.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.169+0000 7f6632341700 1 -- 172.21.15.189:0/2553974374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f662c0aff00 con 0x7f662c0aebf0 2024-04-06T13:45:46.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.169+0000 7f6628ff9700 1 -- 172.21.15.189:0/2553974374 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6624009ea0 con 0x7f662c0aebf0 2024-04-06T13:45:46.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.169+0000 7f6632341700 1 -- 172.21.15.189:0/2553974374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f662c0b04c0 con 0x7f662c0aebf0 2024-04-06T13:45:46.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.169+0000 7f6632341700 1 -- 172.21.15.189:0/2553974374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f662c0b06e0 con 0x7f662c0aebf0 2024-04-06T13:45:46.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.170+0000 7f6628ff9700 1 -- 172.21.15.189:0/2553974374 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f6624003bd0 con 0x7f662c0aebf0 2024-04-06T13:45:46.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.171+0000 7f6628ff9700 1 --2- 172.21.15.189:0/2553974374 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f661806adb0 0x7f661806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:46.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.171+0000 7f662bfff700 1 --2- 172.21.15.189:0/2553974374 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f661806adb0 0x7f661806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:46.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.171+0000 7f6628ff9700 1 -- 172.21.15.189:0/2553974374 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f662408a970 con 0x7f662c0aebf0 2024-04-06T13:45:46.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.171+0000 7f662bfff700 1 --2- 172.21.15.189:0/2553974374 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f661806adb0 0x7f661806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6620000c00 tx=0x7f6620009040 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:46.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.175+0000 7f6628ff9700 1 -- 172.21.15.189:0/2553974374 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f66240556f0 con 0x7f662c0aebf0 2024-04-06T13:45:46.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.415+0000 7f6632341700 1 -- 172.21.15.189:0/2553974374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f662c073200 con 0x7f662c0aebf0 2024-04-06T13:45:46.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.415+0000 7f6628ff9700 1 -- 172.21.15.189:0/2553974374 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v93) v1 ==== 53+0+73 (secure 0 0 0) 0x7f662c073200 con 0x7f662c0aebf0 2024-04-06T13:45:46.416 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-06T13:45:46.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.418+0000 7f6632341700 1 -- 172.21.15.189:0/2553974374 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f661806adb0 msgr2=0x7f661806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:46.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.418+0000 7f6632341700 1 --2- 172.21.15.189:0/2553974374 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f661806adb0 0x7f661806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6620000c00 tx=0x7f6620009040 comp rx=0 tx=0).stop 2024-04-06T13:45:46.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.418+0000 7f6632341700 1 -- 172.21.15.189:0/2553974374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f662c0aebf0 msgr2=0x7f662c0af070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:46.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.418+0000 7f6632341700 1 --2- 172.21.15.189:0/2553974374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f662c0aebf0 0x7f662c0af070 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f662c06c7d0 tx=0x7f6624011790 comp rx=0 tx=0).stop 2024-04-06T13:45:46.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.418+0000 7f6632341700 1 -- 172.21.15.189:0/2553974374 shutdown_connections 2024-04-06T13:45:46.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.418+0000 7f6632341700 1 --2- 172.21.15.189:0/2553974374 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f661806adb0 0x7f661806d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.418+0000 7f6632341700 1 --2- 172.21.15.189:0/2553974374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f662c0aebf0 0x7f662c0af070 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.418+0000 7f6632341700 1 --2- 172.21.15.189:0/2553974374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66140043f0 0x7f662c14a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.418+0000 7f6632341700 1 --2- 172.21.15.189:0/2553974374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f662c1583f0 0x7f662c158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:46.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.418+0000 7f6632341700 1 -- 172.21.15.189:0/2553974374 >> 172.21.15.189:0/2553974374 conn(0x7f662c0d00b0 msgr2=0x7f662c15c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:46.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.418+0000 7f6632341700 1 -- 172.21.15.189:0/2553974374 shutdown_connections 2024-04-06T13:45:46.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:46.418+0000 7f6632341700 1 -- 172.21.15.189:0/2553974374 wait complete. 2024-04-06T13:45:46.430 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-06T13:45:46.430 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T13:45:46.430 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-06T13:45:46.430 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-06T13:45:46.430 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T13:45:46.430 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-06T13:45:46.430 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-06T13:45:46.430 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T13:45:46.431 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:45:46.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:46.431 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-06T13:45:46.448 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T13:45:46.448 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T13:45:46.448 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T13:45:46.448 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:45:46.448 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T13:45:46.449 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:45:46.449 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:45:46.449 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:45:46.449 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-06T13:45:46.449 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:45:46.449 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:45:46.449 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-06T13:45:46.449 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:45:46.449 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:45:46.449 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T13:45:46.449 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:45:46.449 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T13:45:46.449 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 36743sec preferred_lft 36743sec 2024-04-06T13:45:46.449 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::8592:df2c:92e2:dace/64 scope link noprefixroute 2024-04-06T13:45:46.450 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:45:46.450 INFO:teuthology.orchestra.run.smithi189.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T13:45:46.450 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff 2024-04-06T13:45:46.450 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-06T13:45:46.450 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:45:46.450 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::88e:b6ff:fe6f:1cb0/64 scope link 2024-04-06T13:45:46.450 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:45:46.450 INFO:teuthology.orchestra.run.smithi189.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T13:45:46.450 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 62:a5:8d:c5:1a:9c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T13:45:46.450 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::60a5:8dff:fec5:1a9c/64 scope link 2024-04-06T13:45:46.450 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:45:46.450 INFO:teuthology.orchestra.run.smithi189.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T13:45:46.450 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-06T13:45:46.450 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::1c7e:c6ff:febb:1dab/64 scope link 2024-04-06T13:45:46.450 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:45:46.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:46.451 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T13:45:46.514 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-06T13:45:46.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:46.516 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-06T13:45:46.575 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-06T13:45:46.575 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T13:45:46.575 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T13:45:46.575 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T13:45:46.575 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T13:45:46.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:46.576 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T13:45:46.576 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T13:45:46.576 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T13:45:46.576 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T13:45:46.576 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T13:45:46.576 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T13:45:46.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:46.681 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T13:45:46.698 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-06T13:45:46.698 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-06T13:45:46.699 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:45:46.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:46.699 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-04-06T13:45:46.766 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T13:45:46.766 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-06T13:45:46.823 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:45:46.824 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T13:45:46.824 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-06T13:45:46.880 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T13:45:46.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:46.881 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:45:46.966 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:45:46.967 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:45:46.967 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:45:46.976 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:45:46.976 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:45:46.976 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:45:46.976 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T13:45:46.976 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T13:45:46.994 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:45:46.995 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:45:46.995 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:45:46.995 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T13:45:46.995 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T13:45:46.995 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790" 2024-04-06T13:45:46.995 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2024-04-06T13:45:46.995 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e30041de-e9a3-4351-8ca7-29d7f3e3728c" will pass to kernel 2024-04-06T13:45:46.998 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T13:45:46.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:46.999 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-06T13:45:47.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:47.027 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T13:45:47.109 INFO:teuthology.orchestra.run.smithi189.stdout:client10887 2024-04-06T13:45:47.109 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:45:47.109 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:45:47.170 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-06T13:45:47.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.334+0000 7fcde1138700 1 Processor -- start 2024-04-06T13:45:47.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.334+0000 7fcde1138700 1 -- start start 2024-04-06T13:45:47.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.335+0000 7fcde1138700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcddc159b20 0x7fcddc06b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:47.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.335+0000 7fcde1138700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcddc0cdc60 0x7fcddc0ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:47.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.335+0000 7fcde1138700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcddc06a8d0 con 0x7fcddc06e470 2024-04-06T13:45:47.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.335+0000 7fcde1138700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcddc0ce620 con 0x7fcddc0cdc60 2024-04-06T13:45:47.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.335+0000 7fcde1138700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc159f00 con 0x7fcddc159b20 2024-04-06T13:45:47.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.335+0000 7fcddb59e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcddc06e470 0x7fcddc06e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55962/0 (socket says 172.21.15.189:55962) 2024-04-06T13:45:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.335+0000 7fcddb59e700 1 -- 172.21.15.189:0/1081231964 learned_addr learned my addr 172.21.15.189:0/1081231964 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.335+0000 7fcddad9d700 1 --2- 172.21.15.189:0/1081231964 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcddc159b20 0x7fcddc06b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.335+0000 7fcdda59c700 1 --2- 172.21.15.189:0/1081231964 >> v2:172.21.15.192:3300/0 conn(0x7fcddc0cdc60 0x7fcddc0ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.335+0000 7fcdd9d9b700 1 -- 172.21.15.189:0/1081231964 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955532772 0 0) 0x7fcddc06a8d0 con 0x7fcddc06e470 2024-04-06T13:45:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.335+0000 7fcdd9d9b700 1 -- 172.21.15.189:0/1081231964 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdc40035b0 con 0x7fcddc06e470 2024-04-06T13:45:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.335+0000 7fcdd9d9b700 1 -- 172.21.15.189:0/1081231964 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3957340727 0 0) 0x7fcdc40035b0 con 0x7fcddc06e470 2024-04-06T13:45:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.335+0000 7fcdd9d9b700 1 -- 172.21.15.189:0/1081231964 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcddc159b20 msgr2=0x7fcddc06b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.335+0000 7fcdd9d9b700 1 --2- 172.21.15.189:0/1081231964 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcddc159b20 0x7fcddc06b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.335+0000 7fcdd9d9b700 1 -- 172.21.15.189:0/1081231964 >> v2:172.21.15.192:3300/0 conn(0x7fcddc0cdc60 msgr2=0x7fcddc0ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.335+0000 7fcdd9d9b700 1 --2- 172.21.15.189:0/1081231964 >> v2:172.21.15.192:3300/0 conn(0x7fcddc0cdc60 0x7fcddc0ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.335+0000 7fcdd9d9b700 1 -- 172.21.15.189:0/1081231964 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcddc06abb0 con 0x7fcddc06e470 2024-04-06T13:45:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.335+0000 7fcdd9d9b700 1 -- 172.21.15.189:0/1081231964 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fcdd40019f0 con 0x7fcddc06e470 2024-04-06T13:45:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.336+0000 7fcdd9d9b700 1 -- 172.21.15.189:0/1081231964 >> v1:172.21.15.189:6789/0 conn(0x7fcddc06e470 legacy=0x7fcddc06e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.336+0000 7fcdd9d9b700 1 --2- 172.21.15.189:0/1081231964 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcddc159b20 0x7fcdc4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:47.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.336+0000 7fcdd9d9b700 1 --2- 172.21.15.189:0/1081231964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcdc4003f80 0x7fcdc4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:47.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.336+0000 7fcdd9d9b700 1 --2- 172.21.15.189:0/1081231964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdc4004d90 0x7fcdc4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:47.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.336+0000 7fcdd9d9b700 1 -- 172.21.15.189:0/1081231964 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc0ce620 con 0x7fcdc4004d90 2024-04-06T13:45:47.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.336+0000 7fcdd9d9b700 1 -- 172.21.15.189:0/1081231964 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc159f00 con 0x7fcdc4003f80 2024-04-06T13:45:47.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.336+0000 7fcdd9d9b700 1 -- 172.21.15.189:0/1081231964 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc400bb10 con 0x7fcddc159b20 2024-04-06T13:45:47.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.336+0000 7fcddb59e700 1 --2- 172.21.15.189:0/1081231964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdc4004d90 0x7fcdc4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:47.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.336+0000 7fcddad9d700 1 --2- 172.21.15.189:0/1081231964 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcddc159b20 0x7fcdc4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:47.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.336+0000 7fcdda59c700 1 --2- 172.21.15.189:0/1081231964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcdc4003f80 0x7fcdc4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:47.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.336+0000 7fcddb59e700 1 -- 172.21.15.189:0/1081231964 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcddc159b20 msgr2=0x7fcdc4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:47.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.336+0000 7fcddb59e700 1 --2- 172.21.15.189:0/1081231964 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcddc159b20 0x7fcdc4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:47.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.336+0000 7fcddb59e700 1 -- 172.21.15.189:0/1081231964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcdc4003f80 msgr2=0x7fcdc4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:47.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.336+0000 7fcddb59e700 1 --2- 172.21.15.189:0/1081231964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcdc4003f80 0x7fcdc4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:47.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.336+0000 7fcddb59e700 1 -- 172.21.15.189:0/1081231964 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc40035b0 con 0x7fcdc4004d90 2024-04-06T13:45:47.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.336+0000 7fcddb59e700 1 --2- 172.21.15.189:0/1081231964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdc4004d90 0x7fcdc4005210 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400a890 tx=0x7fcdc4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f177cf201bb5c17 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:47.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.336+0000 7fcdd9d9b700 1 -- 172.21.15.189:0/1081231964 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4012600 con 0x7fcdc4004d90 2024-04-06T13:45:47.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.336+0000 7fcdd9d9b700 1 -- 172.21.15.189:0/1081231964 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd4012760 con 0x7fcdc4004d90 2024-04-06T13:45:47.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.336+0000 7fcdd9d9b700 1 -- 172.21.15.189:0/1081231964 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4012b10 con 0x7fcdc4004d90 2024-04-06T13:45:47.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.337+0000 7fcde1138700 1 -- 172.21.15.189:0/1081231964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdc4004d90 msgr2=0x7fcdc4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:47.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.337+0000 7fcde1138700 1 --2- 172.21.15.189:0/1081231964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdc4004d90 0x7fcdc4005210 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400a890 tx=0x7fcdc4005880 comp rx=0 tx=0).stop 2024-04-06T13:45:47.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.337+0000 7fcddb59e700 1 -- 172.21.15.189:0/1081231964 reap_dead start 2024-04-06T13:45:47.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.337+0000 7fcde1138700 1 -- 172.21.15.189:0/1081231964 shutdown_connections 2024-04-06T13:45:47.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.337+0000 7fcde1138700 1 -- 172.21.15.189:0/1081231964 >> 172.21.15.189:0/1081231964 conn(0x7fcddc0cc290 msgr2=0x7fcddc0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:47.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.337+0000 7fcde1138700 1 -- 172.21.15.189:0/1081231964 shutdown_connections 2024-04-06T13:45:47.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.337+0000 7fcde1138700 1 -- 172.21.15.189:0/1081231964 wait complete. 2024-04-06T13:45:47.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.338+0000 7fcde1138700 1 Processor -- start 2024-04-06T13:45:47.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.338+0000 7fcde1138700 1 -- start start 2024-04-06T13:45:47.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.339+0000 7fcde1138700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdc4004d90 0x7fcddc0ae730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:47.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.339+0000 7fcde1138700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcddc0c3320 0x7fcddc0b2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:47.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.339+0000 7fcde1138700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcddc14f630 0x7fcddc0c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:47.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.339+0000 7fcde1138700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc0c37a0 con 0x7fcdc4004d90 2024-04-06T13:45:47.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.339+0000 7fcde1138700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc0b2850 con 0x7fcddc14f630 2024-04-06T13:45:47.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.339+0000 7fcde1138700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc0b2a60 con 0x7fcddc0c3320 2024-04-06T13:45:47.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.339+0000 7fcddad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdc4004d90 0x7fcddc0ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:47.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.339+0000 7fcddad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdc4004d90 0x7fcddc0ae730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48274/0 (socket says 172.21.15.189:48274) 2024-04-06T13:45:47.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.339+0000 7fcddad9d700 1 -- 172.21.15.189:0/2082490802 learned_addr learned my addr 172.21.15.189:0/2082490802 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:47.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.339+0000 7fcddb59e700 1 --2- 172.21.15.189:0/2082490802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcddc14f630 0x7fcddc0c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:47.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.339+0000 7fcdda59c700 1 --2- 172.21.15.189:0/2082490802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcddc0c3320 0x7fcddc0b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:47.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.339+0000 7fcddad9d700 1 -- 172.21.15.189:0/2082490802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcddc0c3320 msgr2=0x7fcddc0b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:47.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.339+0000 7fcddad9d700 1 --2- 172.21.15.189:0/2082490802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcddc0c3320 0x7fcddc0b2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:47.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.339+0000 7fcddad9d700 1 -- 172.21.15.189:0/2082490802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcddc14f630 msgr2=0x7fcddc0c8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:47.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.339+0000 7fcddad9d700 1 --2- 172.21.15.189:0/2082490802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcddc14f630 0x7fcddc0c8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:47.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.339+0000 7fcddad9d700 1 -- 172.21.15.189:0/2082490802 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcddc0b2d40 con 0x7fcdc4004d90 2024-04-06T13:45:47.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.339+0000 7fcdda59c700 1 --2- 172.21.15.189:0/2082490802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcddc0c3320 0x7fcddc0b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:47.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.339+0000 7fcddb59e700 1 --2- 172.21.15.189:0/2082490802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcddc14f630 0x7fcddc0c8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:47.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.340+0000 7fcddad9d700 1 --2- 172.21.15.189:0/2082490802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdc4004d90 0x7fcddc0ae730 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00046e0 tx=0x7fcdd0003cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:47.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.340+0000 7fcdc37fe700 1 -- 172.21.15.189:0/2082490802 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0012040 con 0x7fcdc4004d90 2024-04-06T13:45:47.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.340+0000 7fcdc37fe700 1 -- 172.21.15.189:0/2082490802 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd000dcf0 con 0x7fcdc4004d90 2024-04-06T13:45:47.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.340+0000 7fcde1138700 1 -- 172.21.15.189:0/2082490802 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcddc0b3020 con 0x7fcdc4004d90 2024-04-06T13:45:47.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.340+0000 7fcdc37fe700 1 -- 172.21.15.189:0/2082490802 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0018670 con 0x7fcdc4004d90 2024-04-06T13:45:47.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.340+0000 7fcde1138700 1 -- 172.21.15.189:0/2082490802 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcddc0b35e0 con 0x7fcdc4004d90 2024-04-06T13:45:47.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.341+0000 7fcde1138700 1 -- 172.21.15.189:0/2082490802 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcddc074990 con 0x7fcdc4004d90 2024-04-06T13:45:47.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.341+0000 7fcdc37fe700 1 -- 172.21.15.189:0/2082490802 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fcdd00187d0 con 0x7fcdc4004d90 2024-04-06T13:45:47.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.342+0000 7fcdc37fe700 1 --2- 172.21.15.189:0/2082490802 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcdc806adb0 0x7fcdc806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:47.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.342+0000 7fcdc37fe700 1 -- 172.21.15.189:0/2082490802 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcdd008c360 con 0x7fcdc4004d90 2024-04-06T13:45:47.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.342+0000 7fcdda59c700 1 --2- 172.21.15.189:0/2082490802 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcdc806adb0 0x7fcdc806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:47.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.342+0000 7fcdda59c700 1 --2- 172.21.15.189:0/2082490802 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcdc806adb0 0x7fcdc806d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009c60 tx=0x7fcdcc008040 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:47.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.346+0000 7fcdc37fe700 1 -- 172.21.15.189:0/2082490802 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fcdd0057190 con 0x7fcdc4004d90 2024-04-06T13:45:47.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.568+0000 7fcde1138700 1 -- 172.21.15.189:0/2082490802 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fcddc06b450 con 0x7fcdc806adb0 2024-04-06T13:45:47.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.569+0000 7fcdc37fe700 1 -- 172.21.15.189:0/2082490802 <== mgr.10494 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fcdcc009da0 con 0x7fcdc806adb0 2024-04-06T13:45:47.569 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T13:45:47.570 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:45:47.570 INFO:teuthology.orchestra.run.smithi189.stdout: "name": "cephfs" 2024-04-06T13:45:47.570 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:45:47.570 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T13:45:47.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.572+0000 7fcde1138700 1 -- 172.21.15.189:0/2082490802 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcdc806adb0 msgr2=0x7fcdc806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:47.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.572+0000 7fcde1138700 1 --2- 172.21.15.189:0/2082490802 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcdc806adb0 0x7fcdc806d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009c60 tx=0x7fcdcc008040 comp rx=0 tx=0).stop 2024-04-06T13:45:47.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.572+0000 7fcde1138700 1 -- 172.21.15.189:0/2082490802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdc4004d90 msgr2=0x7fcddc0ae730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:47.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.572+0000 7fcde1138700 1 --2- 172.21.15.189:0/2082490802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdc4004d90 0x7fcddc0ae730 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00046e0 tx=0x7fcdd0003cb0 comp rx=0 tx=0).stop 2024-04-06T13:45:47.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.572+0000 7fcde1138700 1 -- 172.21.15.189:0/2082490802 shutdown_connections 2024-04-06T13:45:47.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.572+0000 7fcde1138700 1 --2- 172.21.15.189:0/2082490802 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcdc806adb0 0x7fcdc806d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:47.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.572+0000 7fcde1138700 1 --2- 172.21.15.189:0/2082490802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcddc14f630 0x7fcddc0c8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:47.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.572+0000 7fcde1138700 1 --2- 172.21.15.189:0/2082490802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdc4004d90 0x7fcddc0ae730 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:47.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.572+0000 7fcde1138700 1 --2- 172.21.15.189:0/2082490802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcddc0c3320 0x7fcddc0b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:47.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.572+0000 7fcde1138700 1 -- 172.21.15.189:0/2082490802 >> 172.21.15.189:0/2082490802 conn(0x7fcddc0cc290 msgr2=0x7fcddc0cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:47.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.572+0000 7fcde1138700 1 -- 172.21.15.189:0/2082490802 shutdown_connections 2024-04-06T13:45:47.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.572+0000 7fcde1138700 1 -- 172.21.15.189:0/2082490802 wait complete. 2024-04-06T13:45:47.584 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-06T13:45:47.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.710+0000 7f54ae70c700 1 Processor -- start 2024-04-06T13:45:47.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.710+0000 7f54ae70c700 1 -- start start 2024-04-06T13:45:47.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.711+0000 7f54ae70c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a815d7b0 0x7f54a80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:47.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.711+0000 7f54ae70c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a80b3990 con 0x7f54a815e500 2024-04-06T13:45:47.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.711+0000 7f54ae70c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a80b3060 con 0x7f54a806a730 2024-04-06T13:45:47.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.711+0000 7f54ae70c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a80b3340 con 0x7f54a815d7b0 2024-04-06T13:45:47.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.711+0000 7f54acca9700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f54a815e500 0x7f54a80717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55974/0 (socket says 172.21.15.189:55974) 2024-04-06T13:45:47.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.711+0000 7f54acca9700 1 -- 172.21.15.189:0/2610497339 learned_addr learned my addr 172.21.15.189:0/2610497339 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:47.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.711+0000 7f54a7fff700 1 --2- 172.21.15.189:0/2610497339 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a815d7b0 0x7f54a80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:47.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.711+0000 7f54a6ffd700 1 -- 172.21.15.189:0/2610497339 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972345269 0 0) 0x7f54a80b3990 con 0x7f54a815e500 2024-04-06T13:45:47.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.711+0000 7f54a6ffd700 1 -- 172.21.15.189:0/2610497339 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54900035b0 con 0x7f54a815e500 2024-04-06T13:45:47.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.711+0000 7f54a6ffd700 1 -- 172.21.15.189:0/2610497339 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713937908 0 0) 0x7f54a80b3060 con 0x7f54a806a730 2024-04-06T13:45:47.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.711+0000 7f54a6ffd700 1 -- 172.21.15.189:0/2610497339 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a80b3990 con 0x7f54a806a730 2024-04-06T13:45:47.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.711+0000 7f54a7fff700 1 -- 172.21.15.189:0/2610497339 >> v1:172.21.15.192:6789/0 conn(0x7f54a806a730 legacy=0x7f54a815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:47.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.711+0000 7f54a7fff700 1 -- 172.21.15.189:0/2610497339 >> v1:172.21.15.189:6789/0 conn(0x7f54a815e500 legacy=0x7f54a80717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:47.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.711+0000 7f54a7fff700 1 -- 172.21.15.189:0/2610497339 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a80b3620 con 0x7f54a815d7b0 2024-04-06T13:45:47.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.712+0000 7f54a7fff700 1 --2- 172.21.15.189:0/2610497339 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a815d7b0 0x7f54a80b3e20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f549c009870 tx=0x7f549c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b4e7215890e5860 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:47.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.712+0000 7f54a6ffd700 1 -- 172.21.15.189:0/2610497339 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c011830 con 0x7f54a815d7b0 2024-04-06T13:45:47.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.712+0000 7f54a6ffd700 1 -- 172.21.15.189:0/2610497339 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549c011990 con 0x7f54a815d7b0 2024-04-06T13:45:47.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.712+0000 7f54a6ffd700 1 -- 172.21.15.189:0/2610497339 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c011d20 con 0x7f54a815d7b0 2024-04-06T13:45:47.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.712+0000 7f54ae70c700 1 -- 172.21.15.189:0/2610497339 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a815d7b0 msgr2=0x7f54a80b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:47.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.712+0000 7f54ae70c700 1 --2- 172.21.15.189:0/2610497339 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a815d7b0 0x7f54a80b3e20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f549c009870 tx=0x7f549c009b80 comp rx=0 tx=0).stop 2024-04-06T13:45:47.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.712+0000 7f54ae70c700 1 -- 172.21.15.189:0/2610497339 shutdown_connections 2024-04-06T13:45:47.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.712+0000 7f54ae70c700 1 --2- 172.21.15.189:0/2610497339 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a815d7b0 0x7f54a80b3e20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:47.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.712+0000 7f54ae70c700 1 -- 172.21.15.189:0/2610497339 >> 172.21.15.189:0/2610497339 conn(0x7f54a80d00b0 msgr2=0x7f54a80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:47.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.712+0000 7f54ae70c700 1 -- 172.21.15.189:0/2610497339 shutdown_connections 2024-04-06T13:45:47.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.712+0000 7f54ae70c700 1 -- 172.21.15.189:0/2610497339 wait complete. 2024-04-06T13:45:47.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.714+0000 7f54ae70c700 1 Processor -- start 2024-04-06T13:45:47.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.714+0000 7f54ae70c700 1 -- start start 2024-04-06T13:45:47.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.714+0000 7f54ae70c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a806a730 0x7f54a81f2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:47.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.714+0000 7f54ae70c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54a815d7b0 0x7f54a815bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:47.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.714+0000 7f54ae70c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54a815e500 0x7f54a815c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:47.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.714+0000 7f54ae70c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a8150480 con 0x7f54a815d7b0 2024-04-06T13:45:47.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.714+0000 7f54ae70c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a81f3db0 con 0x7f54a815e500 2024-04-06T13:45:47.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.714+0000 7f54ae70c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a815a480 con 0x7f54a806a730 2024-04-06T13:45:47.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.715+0000 7f54a77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54a815d7b0 0x7f54a815bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:47.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.715+0000 7f54a77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54a815d7b0 0x7f54a815bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48280/0 (socket says 172.21.15.189:48280) 2024-04-06T13:45:47.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.715+0000 7f54a77fe700 1 -- 172.21.15.189:0/742808952 learned_addr learned my addr 172.21.15.189:0/742808952 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:47.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.715+0000 7f54a7fff700 1 --2- 172.21.15.189:0/742808952 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a806a730 0x7f54a81f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:47.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.715+0000 7f54acca9700 1 --2- 172.21.15.189:0/742808952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54a815e500 0x7f54a815c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:47.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.715+0000 7f54a77fe700 1 -- 172.21.15.189:0/742808952 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a806a730 msgr2=0x7f54a81f2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:47.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.715+0000 7f54a77fe700 1 --2- 172.21.15.189:0/742808952 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a806a730 0x7f54a81f2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:47.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.715+0000 7f54a77fe700 1 -- 172.21.15.189:0/742808952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54a815e500 msgr2=0x7f54a815c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:47.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.715+0000 7f54a77fe700 1 --2- 172.21.15.189:0/742808952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54a815e500 0x7f54a815c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:47.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.715+0000 7f54a77fe700 1 -- 172.21.15.189:0/742808952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f549c0094a0 con 0x7f54a815d7b0 2024-04-06T13:45:47.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.715+0000 7f54a77fe700 1 --2- 172.21.15.189:0/742808952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54a815d7b0 0x7f54a815bc60 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f549800c500 tx=0x7f549800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:47.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.715+0000 7f54a4ff9700 1 -- 172.21.15.189:0/742808952 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54980052d0 con 0x7f54a815d7b0 2024-04-06T13:45:47.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.715+0000 7f54a4ff9700 1 -- 172.21.15.189:0/742808952 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5498005450 con 0x7f54a815d7b0 2024-04-06T13:45:47.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.715+0000 7f54a4ff9700 1 -- 172.21.15.189:0/742808952 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54980038c0 con 0x7f54a815d7b0 2024-04-06T13:45:47.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.715+0000 7f54ae70c700 1 -- 172.21.15.189:0/742808952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a815a760 con 0x7f54a815d7b0 2024-04-06T13:45:47.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.715+0000 7f54ae70c700 1 -- 172.21.15.189:0/742808952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a815ad20 con 0x7f54a815d7b0 2024-04-06T13:45:47.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.716+0000 7f54ae70c700 1 -- 172.21.15.189:0/742808952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a8067bb0 con 0x7f54a815d7b0 2024-04-06T13:45:47.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.717+0000 7f54a4ff9700 1 -- 172.21.15.189:0/742808952 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 127) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f5498003a20 con 0x7f54a815d7b0 2024-04-06T13:45:47.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.717+0000 7f54a4ff9700 1 --2- 172.21.15.189:0/742808952 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f549406adb0 0x7f549406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:47.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.717+0000 7f54a4ff9700 1 -- 172.21.15.189:0/742808952 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5498088bb0 con 0x7f54a815d7b0 2024-04-06T13:45:47.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.717+0000 7f54a7fff700 1 --2- 172.21.15.189:0/742808952 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f549406adb0 0x7f549406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:47.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.718+0000 7f54a7fff700 1 --2- 172.21.15.189:0/742808952 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f549406adb0 0x7f549406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f549c0097c0 tx=0x7f549c01c040 comp rx=0 tx=0).ready entity=mgr.10494 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:47.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.722+0000 7f54a4ff9700 1 -- 172.21.15.189:0/742808952 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5498053a70 con 0x7f54a815d7b0 2024-04-06T13:45:47.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:47.974+0000 7f54ae70c700 1 -- 172.21.15.189:0/742808952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f54a8074990 con 0x7f54a815d7b0 2024-04-06T13:45:48.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.573+0000 7f54a4ff9700 1 -- 172.21.15.189:0/742808952 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 128) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f54980760f0 con 0x7f54a815d7b0 2024-04-06T13:45:48.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.574+0000 7f54a4ff9700 1 -- 172.21.15.189:0/742808952 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v128) v1 ==== 92+0+0 (secure 0 0 0) 0x7f5498083e50 con 0x7f54a815d7b0 2024-04-06T13:45:48.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.578+0000 7f54ae70c700 1 -- 172.21.15.189:0/742808952 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f549406adb0 msgr2=0x7f549406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:48.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.578+0000 7f54ae70c700 1 --2- 172.21.15.189:0/742808952 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f549406adb0 0x7f549406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f549c0097c0 tx=0x7f549c01c040 comp rx=0 tx=0).stop 2024-04-06T13:45:48.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.578+0000 7f54ae70c700 1 -- 172.21.15.189:0/742808952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54a815d7b0 msgr2=0x7f54a815bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:48.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.578+0000 7f54ae70c700 1 --2- 172.21.15.189:0/742808952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54a815d7b0 0x7f54a815bc60 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f549800c500 tx=0x7f549800c8c0 comp rx=0 tx=0).stop 2024-04-06T13:45:48.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.578+0000 7f54ae70c700 1 -- 172.21.15.189:0/742808952 shutdown_connections 2024-04-06T13:45:48.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.578+0000 7f54ae70c700 1 --2- 172.21.15.189:0/742808952 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f549406adb0 0x7f549406d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:48.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.578+0000 7f54ae70c700 1 --2- 172.21.15.189:0/742808952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54a815e500 0x7f54a815c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:48.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.578+0000 7f54ae70c700 1 --2- 172.21.15.189:0/742808952 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54a806a730 0x7f54a81f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:48.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.578+0000 7f54ae70c700 1 --2- 172.21.15.189:0/742808952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54a815d7b0 0x7f54a815bc60 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:48.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.578+0000 7f54ae70c700 1 -- 172.21.15.189:0/742808952 >> 172.21.15.189:0/742808952 conn(0x7f54a80d00b0 msgr2=0x7f54a806fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:48.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.578+0000 7f54ae70c700 1 -- 172.21.15.189:0/742808952 shutdown_connections 2024-04-06T13:45:48.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.578+0000 7f54ae70c700 1 -- 172.21.15.189:0/742808952 wait complete. 2024-04-06T13:45:48.597 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-06T13:45:48.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.731+0000 7f35674a4700 1 Processor -- start 2024-04-06T13:45:48.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.731+0000 7f35674a4700 1 -- start start 2024-04-06T13:45:48.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.732+0000 7f35674a4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35600cb870 0x7f356015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:48.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.732+0000 7f35674a4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f35600cfa00 0x7f35600cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:48.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.732+0000 7f35674a4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35600cbd80 con 0x7f35600d11b0 2024-04-06T13:45:48.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.732+0000 7f35674a4700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35600ccdc0 con 0x7f35600cfa00 2024-04-06T13:45:48.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.732+0000 7f35674a4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35600d0b50 con 0x7f35600cb870 2024-04-06T13:45:48.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.732+0000 7f3565a41700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f35600cfa00 0x7f35600cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:48.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.732+0000 7f3565240700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f35600d11b0 0x7f35600d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55990/0 (socket says 172.21.15.189:55990) 2024-04-06T13:45:48.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.732+0000 7f3565240700 1 -- 172.21.15.189:0/2800747340 learned_addr learned my addr 172.21.15.189:0/2800747340 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:48.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.732+0000 7f3564a3f700 1 --2- 172.21.15.189:0/2800747340 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35600cb870 0x7f356015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:48.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.732+0000 7f3565a41700 1 -- 172.21.15.189:0/2800747340 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35600cb870 msgr2=0x7f356015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:48.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.732+0000 7f3565a41700 1 --2- 172.21.15.189:0/2800747340 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35600cb870 0x7f356015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:48.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.732+0000 7f3565a41700 1 -- 172.21.15.189:0/2800747340 >> v1:172.21.15.189:6789/0 conn(0x7f35600d11b0 legacy=0x7f35600d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:48.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.732+0000 7f3557fff700 1 -- 172.21.15.189:0/2800747340 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931457439 0 0) 0x7f35600cbd80 con 0x7f35600d11b0 2024-04-06T13:45:48.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.732+0000 7f3565a41700 1 -- 172.21.15.189:0/2800747340 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35600cef20 con 0x7f35600cfa00 2024-04-06T13:45:48.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.732+0000 7f3565a41700 1 --2- 172.21.15.189:0/2800747340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35600cfa00 0x7f35600cd560 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f3558009d90 tx=0x7f355800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=696546df4a8c4dd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:48.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.733+0000 7f3557fff700 1 -- 172.21.15.189:0/2800747340 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355800e070 con 0x7f35600cfa00 2024-04-06T13:45:48.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.733+0000 7f3557fff700 1 -- 172.21.15.189:0/2800747340 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355800bb20 con 0x7f35600cfa00 2024-04-06T13:45:48.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.733+0000 7f3557fff700 1 -- 172.21.15.189:0/2800747340 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3558012430 con 0x7f35600cfa00 2024-04-06T13:45:48.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.733+0000 7f35674a4700 1 -- 172.21.15.189:0/2800747340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35600cfa00 msgr2=0x7f35600cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:48.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.733+0000 7f35674a4700 1 --2- 172.21.15.189:0/2800747340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35600cfa00 0x7f35600cd560 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f3558009d90 tx=0x7f355800b330 comp rx=0 tx=0).stop 2024-04-06T13:45:48.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.733+0000 7f35674a4700 1 -- 172.21.15.189:0/2800747340 shutdown_connections 2024-04-06T13:45:48.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.733+0000 7f35674a4700 1 --2- 172.21.15.189:0/2800747340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35600cfa00 0x7f35600cd560 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:48.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.733+0000 7f35674a4700 1 --2- 172.21.15.189:0/2800747340 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35600cb870 0x7f356015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:48.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.733+0000 7f35674a4700 1 -- 172.21.15.189:0/2800747340 >> 172.21.15.189:0/2800747340 conn(0x7f35600b2db0 msgr2=0x7f35600b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:48.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.733+0000 7f35674a4700 1 -- 172.21.15.189:0/2800747340 shutdown_connections 2024-04-06T13:45:48.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.733+0000 7f35674a4700 1 -- 172.21.15.189:0/2800747340 wait complete. 2024-04-06T13:45:48.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.735+0000 7f35674a4700 1 Processor -- start 2024-04-06T13:45:48.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.735+0000 7f35674a4700 1 -- start start 2024-04-06T13:45:48.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.735+0000 7f35674a4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35600cb870 0x7f356014eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:48.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.735+0000 7f35674a4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35600cfa00 0x7f35600b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:48.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.735+0000 7f35674a4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35600d11b0 0x7f35600b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:48.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.735+0000 7f35674a4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356014f0b0 con 0x7f35600cb870 2024-04-06T13:45:48.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.735+0000 7f35674a4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35600c8390 con 0x7f35600cfa00 2024-04-06T13:45:48.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.735+0000 7f35674a4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35600c8670 con 0x7f35600d11b0 2024-04-06T13:45:48.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.736+0000 7f3565240700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35600cb870 0x7f356014eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:48.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.736+0000 7f3565240700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35600cb870 0x7f356014eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48328/0 (socket says 172.21.15.189:48328) 2024-04-06T13:45:48.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.736+0000 7f3565240700 1 -- 172.21.15.189:0/4162564237 learned_addr learned my addr 172.21.15.189:0/4162564237 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:48.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.736+0000 7f3564a3f700 1 --2- 172.21.15.189:0/4162564237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35600cfa00 0x7f35600b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:48.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.736+0000 7f3565a41700 1 --2- 172.21.15.189:0/4162564237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35600d11b0 0x7f35600b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:48.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.736+0000 7f3565240700 1 -- 172.21.15.189:0/4162564237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35600d11b0 msgr2=0x7f35600b6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:48.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.736+0000 7f3565240700 1 --2- 172.21.15.189:0/4162564237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35600d11b0 0x7f35600b6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:48.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.736+0000 7f3565240700 1 -- 172.21.15.189:0/4162564237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35600cfa00 msgr2=0x7f35600b5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:48.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.736+0000 7f3565240700 1 --2- 172.21.15.189:0/4162564237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35600cfa00 0x7f35600b5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:48.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.736+0000 7f3565240700 1 -- 172.21.15.189:0/4162564237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3558009a70 con 0x7f35600cb870 2024-04-06T13:45:48.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.736+0000 7f3565240700 1 --2- 172.21.15.189:0/4162564237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35600cb870 0x7f356014eb70 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f355c00c550 tx=0x7f355c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:48.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.736+0000 7f3555ffb700 1 -- 172.21.15.189:0/4162564237 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c004c80 con 0x7f35600cb870 2024-04-06T13:45:48.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.736+0000 7f3555ffb700 1 -- 172.21.15.189:0/4162564237 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355c004e00 con 0x7f35600cb870 2024-04-06T13:45:48.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.736+0000 7f3555ffb700 1 -- 172.21.15.189:0/4162564237 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c007310 con 0x7f35600cb870 2024-04-06T13:45:48.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.736+0000 7f35674a4700 1 -- 172.21.15.189:0/4162564237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35600c8950 con 0x7f35600cb870 2024-04-06T13:45:48.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.736+0000 7f35674a4700 1 -- 172.21.15.189:0/4162564237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35600c8f10 con 0x7f35600cb870 2024-04-06T13:45:48.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.738+0000 7f3555ffb700 1 -- 172.21.15.189:0/4162564237 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 128) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f355c007470 con 0x7f35600cb870 2024-04-06T13:45:48.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.738+0000 7f35674a4700 1 -- 172.21.15.189:0/4162564237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3560074990 con 0x7f35600cb870 2024-04-06T13:45:48.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.738+0000 7f3555ffb700 1 --2- 172.21.15.189:0/4162564237 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f354406ae00 0x7f354406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:48.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.738+0000 7f3564a3f700 1 -- 172.21.15.189:0/4162564237 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f354406ae00 msgr2=0x7f354406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:48.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.738+0000 7f3555ffb700 1 -- 172.21.15.189:0/4162564237 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f355c0891d0 con 0x7f35600cb870 2024-04-06T13:45:48.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.738+0000 7f3564a3f700 1 --2- 172.21.15.189:0/4162564237 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f354406ae00 0x7f354406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:45:48.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.743+0000 7f3555ffb700 1 -- 172.21.15.189:0/4162564237 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f355c054110 con 0x7f35600cb870 2024-04-06T13:45:48.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.939+0000 7f3564a3f700 1 -- 172.21.15.189:0/4162564237 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f354406ae00 msgr2=0x7f354406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:48.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.939+0000 7f3564a3f700 1 --2- 172.21.15.189:0/4162564237 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f354406ae00 0x7f354406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:45:48.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.946+0000 7f35674a4700 1 -- 172.21.15.189:0/4162564237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f35600b0a00 con 0x7f35600cb870 2024-04-06T13:45:48.948 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:48.947+0000 7f6a1cbf5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:45:48.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.950+0000 7f3555ffb700 1 -- 172.21.15.189:0/4162564237 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v71)=0 v71) v1 ==== 138+0+0 (secure 0 0 0) 0x7f355c057730 con 0x7f35600cb870 2024-04-06T13:45:48.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.954+0000 7f35674a4700 1 -- 172.21.15.189:0/4162564237 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f354406ae00 msgr2=0x7f354406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:45:48.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.954+0000 7f35674a4700 1 --2- 172.21.15.189:0/4162564237 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f354406ae00 0x7f354406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:48.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.954+0000 7f35674a4700 1 -- 172.21.15.189:0/4162564237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35600cb870 msgr2=0x7f356014eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:48.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.954+0000 7f35674a4700 1 --2- 172.21.15.189:0/4162564237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35600cb870 0x7f356014eb70 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f355c00c550 tx=0x7f355c00c860 comp rx=0 tx=0).stop 2024-04-06T13:45:48.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.954+0000 7f35674a4700 1 -- 172.21.15.189:0/4162564237 shutdown_connections 2024-04-06T13:45:48.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.954+0000 7f35674a4700 1 --2- 172.21.15.189:0/4162564237 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f354406ae00 0x7f354406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:48.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.954+0000 7f35674a4700 1 --2- 172.21.15.189:0/4162564237 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35600d11b0 0x7f35600b6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:48.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.954+0000 7f35674a4700 1 --2- 172.21.15.189:0/4162564237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35600cb870 0x7f356014eb70 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:48.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.954+0000 7f35674a4700 1 --2- 172.21.15.189:0/4162564237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35600cfa00 0x7f35600b5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:48.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.954+0000 7f35674a4700 1 -- 172.21.15.189:0/4162564237 >> 172.21.15.189:0/4162564237 conn(0x7f35600b2db0 msgr2=0x7f356015aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:48.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.954+0000 7f35674a4700 1 -- 172.21.15.189:0/4162564237 shutdown_connections 2024-04-06T13:45:48.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:48.954+0000 7f35674a4700 1 -- 172.21.15.189:0/4162564237 wait complete. 2024-04-06T13:45:48.967 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-06T13:45:48.984 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:48.984+0000 7fa00db13200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:45:49.027 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:49.026+0000 7f6a1cbf5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:45:49.067 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:49.067+0000 7fa00db13200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:45:49.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.095+0000 7fa4c5d9a700 1 Processor -- start 2024-04-06T13:45:49.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.095+0000 7fa4c5d9a700 1 -- start start 2024-04-06T13:45:49.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.095+0000 7fa4c5d9a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4c006ba20 0x7fa4c0166270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:49.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.095+0000 7fa4c5d9a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c00669d0 con 0x7fa4c015e8a0 2024-04-06T13:45:49.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.095+0000 7fa4c5d9a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c006ede0 con 0x7fa4c00cbc20 2024-04-06T13:45:49.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.095+0000 7fa4c5d9a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c015fd00 con 0x7fa4c006ba20 2024-04-06T13:45:49.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.096+0000 7fa4bf7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa4c015e8a0 0x7fa4c015eee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55992/0 (socket says 172.21.15.189:55992) 2024-04-06T13:45:49.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.096+0000 7fa4bf7fe700 1 -- 172.21.15.189:0/1069221663 learned_addr learned my addr 172.21.15.189:0/1069221663 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:49.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.096+0000 7fa4bffff700 1 --2- 172.21.15.189:0/1069221663 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4c006ba20 0x7fa4c0166270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:49.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.096+0000 7fa4beffd700 1 -- 172.21.15.189:0/1069221663 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770272867 0 0) 0x7fa4c00669d0 con 0x7fa4c015e8a0 2024-04-06T13:45:49.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.096+0000 7fa4beffd700 1 -- 172.21.15.189:0/1069221663 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4ac0035b0 con 0x7fa4c015e8a0 2024-04-06T13:45:49.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.096+0000 7fa4beffd700 1 -- 172.21.15.189:0/1069221663 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1650805570 0 0) 0x7fa4c006ede0 con 0x7fa4c00cbc20 2024-04-06T13:45:49.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.096+0000 7fa4beffd700 1 -- 172.21.15.189:0/1069221663 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4c00669d0 con 0x7fa4c00cbc20 2024-04-06T13:45:49.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.096+0000 7fa4bffff700 1 -- 172.21.15.189:0/1069221663 >> v1:172.21.15.192:6789/0 conn(0x7fa4c00cbc20 legacy=0x7fa4c015f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:49.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.096+0000 7fa4bffff700 1 -- 172.21.15.189:0/1069221663 >> v1:172.21.15.189:6789/0 conn(0x7fa4c015e8a0 legacy=0x7fa4c015eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:49.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.096+0000 7fa4bffff700 1 -- 172.21.15.189:0/1069221663 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c015ec80 con 0x7fa4c006ba20 2024-04-06T13:45:49.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.096+0000 7fa4bffff700 1 --2- 172.21.15.189:0/1069221663 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4c006ba20 0x7fa4c0166270 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009dc0 tx=0x7fa4b800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39d50ea8be1a8a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:49.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.097+0000 7fa4beffd700 1 -- 172.21.15.189:0/1069221663 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b800e070 con 0x7fa4c006ba20 2024-04-06T13:45:49.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.097+0000 7fa4beffd700 1 -- 172.21.15.189:0/1069221663 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b800bb20 con 0x7fa4c006ba20 2024-04-06T13:45:49.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.097+0000 7fa4beffd700 1 -- 172.21.15.189:0/1069221663 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8012430 con 0x7fa4c006ba20 2024-04-06T13:45:49.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.097+0000 7fa4c5d9a700 1 -- 172.21.15.189:0/1069221663 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4c006ba20 msgr2=0x7fa4c0166270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:49.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.097+0000 7fa4c5d9a700 1 --2- 172.21.15.189:0/1069221663 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4c006ba20 0x7fa4c0166270 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009dc0 tx=0x7fa4b800b330 comp rx=0 tx=0).stop 2024-04-06T13:45:49.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.097+0000 7fa4c5d9a700 1 -- 172.21.15.189:0/1069221663 shutdown_connections 2024-04-06T13:45:49.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.097+0000 7fa4c5d9a700 1 --2- 172.21.15.189:0/1069221663 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4c006ba20 0x7fa4c0166270 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:49.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.097+0000 7fa4c5d9a700 1 -- 172.21.15.189:0/1069221663 >> 172.21.15.189:0/1069221663 conn(0x7fa4c00cc290 msgr2=0x7fa4c00cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:49.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.097+0000 7fa4c5d9a700 1 -- 172.21.15.189:0/1069221663 shutdown_connections 2024-04-06T13:45:49.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.097+0000 7fa4c5d9a700 1 -- 172.21.15.189:0/1069221663 wait complete. 2024-04-06T13:45:49.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.098+0000 7fa4c5d9a700 1 Processor -- start 2024-04-06T13:45:49.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.099+0000 7fa4c5d9a700 1 -- start start 2024-04-06T13:45:49.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.099+0000 7fa4c5d9a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4c006ba20 0x7fa4c00c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:49.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.099+0000 7fa4c5d9a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4c00cbc20 0x7fa4c00ce510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:49.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.099+0000 7fa4c5d9a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4c015e8a0 0x7fa4c00b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:49.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.099+0000 7fa4c5d9a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c00ad850 con 0x7fa4c015e8a0 2024-04-06T13:45:49.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.099+0000 7fa4c5d9a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c00adb30 con 0x7fa4c00cbc20 2024-04-06T13:45:49.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.099+0000 7fa4c5d9a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c00ade10 con 0x7fa4c006ba20 2024-04-06T13:45:49.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.099+0000 7fa4bffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4c015e8a0 0x7fa4c00b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:49.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.099+0000 7fa4b6ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4c00cbc20 0x7fa4c00ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:49.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.099+0000 7fa4bf7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4c006ba20 0x7fa4c00c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:49.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.100+0000 7fa4bf7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4c006ba20 0x7fa4c00c94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:50034/0 (socket says 172.21.15.189:50034) 2024-04-06T13:45:49.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.100+0000 7fa4bffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4c015e8a0 0x7fa4c00b3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48336/0 (socket says 172.21.15.189:48336) 2024-04-06T13:45:49.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.100+0000 7fa4b6ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4c00cbc20 0x7fa4c00ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:44928/0 (socket says 172.21.15.189:44928) 2024-04-06T13:45:49.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.100+0000 7fa4bf7fe700 1 -- 172.21.15.189:0/3035294022 learned_addr learned my addr 172.21.15.189:0/3035294022 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:49.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.100+0000 7fa4bf7fe700 1 -- 172.21.15.189:0/3035294022 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4c00cbc20 msgr2=0x7fa4c00ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:49.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.100+0000 7fa4bf7fe700 1 --2- 172.21.15.189:0/3035294022 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4c00cbc20 0x7fa4c00ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:49.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.100+0000 7fa4bf7fe700 1 -- 172.21.15.189:0/3035294022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4c015e8a0 msgr2=0x7fa4c00b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:49.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.100+0000 7fa4bf7fe700 1 --2- 172.21.15.189:0/3035294022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4c015e8a0 0x7fa4c00b3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:49.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.100+0000 7fa4bf7fe700 1 -- 172.21.15.189:0/3035294022 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b8009a70 con 0x7fa4c006ba20 2024-04-06T13:45:49.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.100+0000 7fa4bffff700 1 --2- 172.21.15.189:0/3035294022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4c015e8a0 0x7fa4c00b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:45:49.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.100+0000 7fa4bf7fe700 1 --2- 172.21.15.189:0/3035294022 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4c006ba20 0x7fa4c00c94d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0001eb0 tx=0x7fa4b000c7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:49.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.100+0000 7fa4bcff9700 1 -- 172.21.15.189:0/3035294022 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0005d70 con 0x7fa4c006ba20 2024-04-06T13:45:49.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.100+0000 7fa4bcff9700 1 -- 172.21.15.189:0/3035294022 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b00070d0 con 0x7fa4c006ba20 2024-04-06T13:45:49.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.100+0000 7fa4c5d9a700 1 -- 172.21.15.189:0/3035294022 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c00ae0f0 con 0x7fa4c006ba20 2024-04-06T13:45:49.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.100+0000 7fa4bcff9700 1 -- 172.21.15.189:0/3035294022 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0007440 con 0x7fa4c006ba20 2024-04-06T13:45:49.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.100+0000 7fa4c5d9a700 1 -- 172.21.15.189:0/3035294022 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c00c7230 con 0x7fa4c006ba20 2024-04-06T13:45:49.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.101+0000 7fa4c5d9a700 1 -- 172.21.15.189:0/3035294022 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c0072140 con 0x7fa4c006ba20 2024-04-06T13:45:49.103 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:49.099+0000 7f6a1cbf5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:45:49.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.102+0000 7fa4bcff9700 1 -- 172.21.15.189:0/3035294022 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 128) v1 ==== 89264+0+0 (secure 0 0 0) 0x7fa4b00075a0 con 0x7fa4c006ba20 2024-04-06T13:45:49.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.102+0000 7fa4bcff9700 1 --2- 172.21.15.189:0/3035294022 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa4a006ae00 0x7fa4a006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:49.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.102+0000 7fa4b6ffd700 1 -- 172.21.15.189:0/3035294022 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa4a006ae00 msgr2=0x7fa4a006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:49.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.102+0000 7fa4bcff9700 1 -- 172.21.15.189:0/3035294022 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa4b0088910 con 0x7fa4c006ba20 2024-04-06T13:45:49.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.102+0000 7fa4b6ffd700 1 --2- 172.21.15.189:0/3035294022 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa4a006ae00 0x7fa4a006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:45:49.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.107+0000 7fa4bcff9700 1 -- 172.21.15.189:0/3035294022 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa4b00538e0 con 0x7fa4c006ba20 2024-04-06T13:45:49.145 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:49.145+0000 7fa00db13200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:45:49.167 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:49.167+0000 7f6a1cbf5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:45:49.214 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:49.215+0000 7fa00db13200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:45:49.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.303+0000 7fa4b6ffd700 1 -- 172.21.15.189:0/3035294022 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa4a006ae00 msgr2=0x7fa4a006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:49.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.303+0000 7fa4b6ffd700 1 --2- 172.21.15.189:0/3035294022 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa4a006ae00 0x7fa4a006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:45:49.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.313+0000 7fa4c5d9a700 1 -- 172.21.15.189:0/3035294022 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fa4c00adf50 con 0x7fa4c006ba20 2024-04-06T13:45:49.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.318+0000 7fa4bcff9700 1 -- 172.21.15.189:0/3035294022 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v72)=0 v72) v1 ==== 133+0+0 (secure 0 0 0) 0x7fa4c00adf50 con 0x7fa4c006ba20 2024-04-06T13:45:49.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.320+0000 7fa4c5d9a700 1 -- 172.21.15.189:0/3035294022 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa4a006ae00 msgr2=0x7fa4a006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:45:49.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.320+0000 7fa4c5d9a700 1 --2- 172.21.15.189:0/3035294022 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa4a006ae00 0x7fa4a006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:49.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.320+0000 7fa4c5d9a700 1 -- 172.21.15.189:0/3035294022 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4c006ba20 msgr2=0x7fa4c00c94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:49.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.320+0000 7fa4c5d9a700 1 --2- 172.21.15.189:0/3035294022 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4c006ba20 0x7fa4c00c94d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0001eb0 tx=0x7fa4b000c7b0 comp rx=0 tx=0).stop 2024-04-06T13:45:49.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.320+0000 7fa4c5d9a700 1 -- 172.21.15.189:0/3035294022 shutdown_connections 2024-04-06T13:45:49.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.320+0000 7fa4c5d9a700 1 --2- 172.21.15.189:0/3035294022 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa4a006ae00 0x7fa4a006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:49.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.321+0000 7fa4c5d9a700 1 --2- 172.21.15.189:0/3035294022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4c015e8a0 0x7fa4c00b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:49.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.321+0000 7fa4c5d9a700 1 --2- 172.21.15.189:0/3035294022 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4c006ba20 0x7fa4c00c94d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:49.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.321+0000 7fa4c5d9a700 1 --2- 172.21.15.189:0/3035294022 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4c00cbc20 0x7fa4c00ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:49.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.321+0000 7fa4c5d9a700 1 -- 172.21.15.189:0/3035294022 >> 172.21.15.189:0/3035294022 conn(0x7fa4c00cc290 msgr2=0x7fa4c015eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:49.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.321+0000 7fa4c5d9a700 1 -- 172.21.15.189:0/3035294022 shutdown_connections 2024-04-06T13:45:49.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.321+0000 7fa4c5d9a700 1 -- 172.21.15.189:0/3035294022 wait complete. 2024-04-06T13:45:49.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:49.333 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-06T13:45:49.343 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:49.342+0000 7f6a1cbf5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:45:49.397 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-04-06T13:45:49.399 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:49.400+0000 7fa00db13200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:45:49.498 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:49.497+0000 7f6a1cbf5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:45:49.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.526+0000 7fb9db75c700 1 Processor -- start 2024-04-06T13:45:49.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.527+0000 7fb9db75c700 1 -- start start 2024-04-06T13:45:49.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.528+0000 7fb9db75c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb9d415d7b0 0x7fb9d40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:49.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.528+0000 7fb9db75c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9d415e500 0x7fb9d406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:49.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.528+0000 7fb9db75c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d40b3990 con 0x7fb9d406a730 2024-04-06T13:45:49.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.528+0000 7fb9db75c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9d406b0f0 con 0x7fb9d415d7b0 2024-04-06T13:45:49.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.528+0000 7fb9db75c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d40b3060 con 0x7fb9d415e500 2024-04-06T13:45:49.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.528+0000 7fb9d94f8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb9d415d7b0 0x7fb9d40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:49.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.528+0000 7fb9d8cf7700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb9d406a730 0x7fb9d415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54974/0 (socket says 172.21.15.189:54974) 2024-04-06T13:45:49.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.528+0000 7fb9d8cf7700 1 -- 172.21.15.189:0/3749845751 learned_addr learned my addr 172.21.15.189:0/3749845751 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:49.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.528+0000 7fb9d9cf9700 1 --2- 172.21.15.189:0/3749845751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9d415e500 0x7fb9d406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:49.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.528+0000 7fb9d94f8700 1 -- 172.21.15.189:0/3749845751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9d415e500 msgr2=0x7fb9d406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:49.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.528+0000 7fb9d94f8700 1 --2- 172.21.15.189:0/3749845751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9d415e500 0x7fb9d406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:49.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.528+0000 7fb9d94f8700 1 -- 172.21.15.189:0/3749845751 >> v1:172.21.15.189:6789/0 conn(0x7fb9d406a730 legacy=0x7fb9d415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:49.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.528+0000 7fb9d94f8700 1 -- 172.21.15.189:0/3749845751 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d40b3340 con 0x7fb9d415d7b0 2024-04-06T13:45:49.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.528+0000 7fb9d9cf9700 1 --2- 172.21.15.189:0/3749845751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9d415e500 0x7fb9d406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:49.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.529+0000 7fb9d94f8700 1 --2- 172.21.15.189:0/3749845751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9d415d7b0 0x7fb9d40b3e20 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0009840 tx=0x7fb9d0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b601b0b979f5334 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:49.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.529+0000 7fb9cbfff700 1 -- 172.21.15.189:0/3749845751 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0004230 con 0x7fb9d415d7b0 2024-04-06T13:45:49.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.529+0000 7fb9cbfff700 1 -- 172.21.15.189:0/3749845751 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d0004390 con 0x7fb9d415d7b0 2024-04-06T13:45:49.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.529+0000 7fb9cbfff700 1 -- 172.21.15.189:0/3749845751 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d00036a0 con 0x7fb9d415d7b0 2024-04-06T13:45:49.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.529+0000 7fb9db75c700 1 -- 172.21.15.189:0/3749845751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9d415d7b0 msgr2=0x7fb9d40b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:49.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.529+0000 7fb9db75c700 1 --2- 172.21.15.189:0/3749845751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9d415d7b0 0x7fb9d40b3e20 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0009840 tx=0x7fb9d0009b50 comp rx=0 tx=0).stop 2024-04-06T13:45:49.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.529+0000 7fb9db75c700 1 -- 172.21.15.189:0/3749845751 shutdown_connections 2024-04-06T13:45:49.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.529+0000 7fb9db75c700 1 --2- 172.21.15.189:0/3749845751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9d415e500 0x7fb9d406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:49.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.529+0000 7fb9db75c700 1 --2- 172.21.15.189:0/3749845751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9d415d7b0 0x7fb9d40b3e20 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:49.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.529+0000 7fb9db75c700 1 -- 172.21.15.189:0/3749845751 >> 172.21.15.189:0/3749845751 conn(0x7fb9d40d00b0 msgr2=0x7fb9d40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:49.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.530+0000 7fb9db75c700 1 -- 172.21.15.189:0/3749845751 shutdown_connections 2024-04-06T13:45:49.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.530+0000 7fb9db75c700 1 -- 172.21.15.189:0/3749845751 wait complete. 2024-04-06T13:45:49.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.531+0000 7fb9db75c700 1 Processor -- start 2024-04-06T13:45:49.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.531+0000 7fb9db75c700 1 -- start start 2024-04-06T13:45:49.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.531+0000 7fb9db75c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9d406a730 0x7fb9d415b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:49.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.531+0000 7fb9db75c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9d415d7b0 0x7fb9d415b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:49.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.531+0000 7fb9db75c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9d415e500 0x7fb9d41f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:49.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.531+0000 7fb9db75c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d415be90 con 0x7fb9d415d7b0 2024-04-06T13:45:49.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.531+0000 7fb9db75c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d41518b0 con 0x7fb9d406a730 2024-04-06T13:45:49.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.531+0000 7fb9db75c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d4150fc0 con 0x7fb9d415e500 2024-04-06T13:45:49.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.532+0000 7fb9d8cf7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9d415d7b0 0x7fb9d415b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:49.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.532+0000 7fb9d8cf7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9d415d7b0 0x7fb9d415b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51574/0 (socket says 172.21.15.189:51574) 2024-04-06T13:45:49.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.532+0000 7fb9d8cf7700 1 -- 172.21.15.189:0/1810241912 learned_addr learned my addr 172.21.15.189:0/1810241912 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:49.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.532+0000 7fb9d9cf9700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9d415e500 0x7fb9d41f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:49.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.532+0000 7fb9d94f8700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9d406a730 0x7fb9d415b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:49.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.532+0000 7fb9d8cf7700 1 -- 172.21.15.189:0/1810241912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9d415e500 msgr2=0x7fb9d41f24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:49.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.532+0000 7fb9d8cf7700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9d415e500 0x7fb9d41f24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:49.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.532+0000 7fb9d8cf7700 1 -- 172.21.15.189:0/1810241912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9d406a730 msgr2=0x7fb9d415b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:49.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.532+0000 7fb9d8cf7700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9d406a730 0x7fb9d415b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:49.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.532+0000 7fb9d8cf7700 1 -- 172.21.15.189:0/1810241912 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d00094a0 con 0x7fb9d415d7b0 2024-04-06T13:45:49.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.532+0000 7fb9d8cf7700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9d415d7b0 0x7fb9d415b950 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7fb9c400c780 tx=0x7fb9c400ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:49.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.532+0000 7fb9c9ffb700 1 -- 172.21.15.189:0/1810241912 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4004c80 con 0x7fb9d415d7b0 2024-04-06T13:45:49.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.532+0000 7fb9c9ffb700 1 -- 172.21.15.189:0/1810241912 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c4004de0 con 0x7fb9d415d7b0 2024-04-06T13:45:49.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.532+0000 7fb9db75c700 1 -- 172.21.15.189:0/1810241912 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9d4151260 con 0x7fb9d415d7b0 2024-04-06T13:45:49.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.532+0000 7fb9c9ffb700 1 -- 172.21.15.189:0/1810241912 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4007310 con 0x7fb9d415d7b0 2024-04-06T13:45:49.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.532+0000 7fb9db75c700 1 -- 172.21.15.189:0/1810241912 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9d415c0c0 con 0x7fb9d415d7b0 2024-04-06T13:45:49.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.534+0000 7fb9db75c700 1 -- 172.21.15.189:0/1810241912 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9d4151a40 con 0x7fb9d415d7b0 2024-04-06T13:45:49.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.534+0000 7fb9c9ffb700 1 -- 172.21.15.189:0/1810241912 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 128) v1 ==== 89264+0+0 (secure 0 0 0) 0x7fb9c4007470 con 0x7fb9d415d7b0 2024-04-06T13:45:49.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.534+0000 7fb9c9ffb700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb9b806ae00 0x7fb9b806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:49.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.535+0000 7fb9d94f8700 1 -- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb9b806ae00 msgr2=0x7fb9b806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:49.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.535+0000 7fb9c9ffb700 1 -- 172.21.15.189:0/1810241912 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb9c40893e0 con 0x7fb9d415d7b0 2024-04-06T13:45:49.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.535+0000 7fb9d94f8700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb9b806ae00 0x7fb9b806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:45:49.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.539+0000 7fb9c9ffb700 1 -- 172.21.15.189:0/1810241912 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb9c4054320 con 0x7fb9d415d7b0 2024-04-06T13:45:49.568 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:49.568+0000 7fa00db13200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:45:49.609 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:49.608+0000 7f6a1cbf5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:45:49.681 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:49.681+0000 7fa00db13200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:45:49.689 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:49.688+0000 7f6a1cbf5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:45:49.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.736+0000 7fb9d94f8700 1 -- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb9b806ae00 msgr2=0x7fb9b806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:49.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.736+0000 7fb9d94f8700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb9b806ae00 0x7fb9b806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:45:49.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:49.762+0000 7fb9db75c700 1 -- 172.21.15.189:0/1810241912 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9d415eb40 con 0x7fb9b806ae00 2024-04-06T13:45:49.763 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:49.764+0000 7fa00db13200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:45:49.764 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:49.763+0000 7f6a1cbf5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:45:49.839 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:49.840+0000 7fa00db13200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:45:49.967 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:49.966+0000 7f6a1cbf5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:45:50.043 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:50.042+0000 7f6a1cbf5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:45:50.054 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:50.044+0000 7fa00db13200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:45:50.120 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:50.120+0000 7fa00db13200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:45:50.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:50.137+0000 7fb9d94f8700 1 -- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb9b806ae00 msgr2=0x7fb9b806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:50.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:50.137+0000 7fb9d94f8700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb9b806ae00 0x7fb9b806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-06T13:45:50.361 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:50.360+0000 7f6a1cbf5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:45:50.441 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:50.442+0000 7fa00db13200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:45:50.679 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:50.678+0000 7f6a1cbf5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:45:50.763 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:50.763+0000 7fa00db13200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:45:50.837 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:50.836+0000 7f6a1cbf5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:45:50.909 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:50.908+0000 7f6a1cbf5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:45:50.922 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:50.922+0000 7fa00db13200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:45:50.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:50.938+0000 7fb9d94f8700 1 -- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb9b806ae00 msgr2=0x7fb9b806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:50.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:50.938+0000 7fb9d94f8700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb9b806ae00 0x7fb9b806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-06T13:45:50.996 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:50.996+0000 7fa00db13200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:45:51.527 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:51.526+0000 7f6a1cbf5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:45:51.604 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:51.603+0000 7f6a1cbf5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:45:51.616 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:51.616+0000 7fa00db13200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:45:51.694 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:51.695+0000 7fa00db13200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:45:51.729 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:51.728+0000 7f6a1cbf5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:45:51.821 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:51.821+0000 7fa00db13200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:45:52.237 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:52.236+0000 7f6a1cbf5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:45:52.333 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:52.334+0000 7fa00db13200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:45:52.390 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:52.389+0000 7f6a1cbf5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:45:52.488 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:52.487+0000 7fa00db13200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:45:52.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:52.539+0000 7fb9d94f8700 1 -- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb9b806ae00 msgr2=0x7fb9b806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:45:52.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:52.539+0000 7fb9d94f8700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb9b806ae00 0x7fb9b806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-06T13:45:52.554 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:52.544+0000 7f6a1cbf5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:45:52.646 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:52.646+0000 7fa00db13200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:45:53.265 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:53.263+0000 7f6a1cbf5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:45:53.375 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:53.375+0000 7fa00db13200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:45:53.504 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:53.503+0000 7f6a1cbf5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:45:53.579 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:53.578+0000 7f6a1cbf5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:45:53.620 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:53.620+0000 7fa00db13200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:45:53.697 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:53.697+0000 7fa00db13200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:45:53.954 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:45:53.951+0000 7f6a1cbf5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:45:54.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:54.023+0000 7fb9c9ffb700 1 -- 172.21.15.189:0/1810241912 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 129) v1 ==== 89264+0+0 (secure 0 0 0) 0x7fb9c4087770 con 0x7fb9d415d7b0 2024-04-06T13:45:54.078 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:54.078+0000 7fa00db13200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:45:54.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:54.104+0000 7fb9c9ffb700 1 -- 172.21.15.189:0/1810241912 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mgrmap(e 130) v1 ==== 88873+0+0 (secure 0 0 0) 0x7fb9c40546f0 con 0x7fb9d415d7b0 2024-04-06T13:45:54.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:54.104+0000 7fb9c9ffb700 1 -- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb9b806ae00 msgr2=0x7fb9b806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:45:54.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:54.104+0000 7fb9c9ffb700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb9b806ae00 0x7fb9b806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:54.239 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:54.239+0000 7f9f83a27700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:54.239 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:54.239+0000 7f9f83a27700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:54.239 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:54.239+0000 7f9f83a27700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:54.239 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:54.239+0000 7f9f83a27700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:54.239 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:54.239+0000 7f9f83a27700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:54.239 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:54.239+0000 7f9f80a21700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:54.239 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:54.239+0000 7f9f80a21700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:54.239 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:54.239+0000 7f9f80a21700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:54.239 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:54.239+0000 7f9f80a21700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:54.239 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:45:54.239+0000 7f9f80a21700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:45:55.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.108+0000 7fb9c9ffb700 1 -- 172.21.15.189:0/1810241912 <== mon.0 v2:172.21.15.189:3300/0 9 ==== mgrmap(e 131) v1 ==== 89000+0+0 (secure 0 0 0) 0x7fb9c4054fc0 con 0x7fb9d415d7b0 2024-04-06T13:45:55.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.108+0000 7fb9c9ffb700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb9b806ae00 0x7fb9b806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:55.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.108+0000 7fb9c9ffb700 1 -- 172.21.15.189:0/1810241912 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9b801cd40 con 0x7fb9b806ae00 2024-04-06T13:45:55.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.109+0000 7fb9d94f8700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb9b806ae00 0x7fb9b806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:55.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.113+0000 7fb9d94f8700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb9b806ae00 0x7fb9b806d2c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb9d00090b0 tx=0x7fb9d0008030 comp rx=0 tx=0).ready entity=mgr.10917 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:55.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.170+0000 7fb9c9ffb700 1 -- 172.21.15.189:0/1810241912 <== mgr.10917 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb9b801cd40 con 0x7fb9b806ae00 2024-04-06T13:45:55.170 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path snap_test_dir1 2024-04-06T13:45:55.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.173+0000 7fb9db75c700 1 -- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb9b806ae00 msgr2=0x7fb9b806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:55.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.173+0000 7fb9db75c700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb9b806ae00 0x7fb9b806d2c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb9d00090b0 tx=0x7fb9d0008030 comp rx=0 tx=0).stop 2024-04-06T13:45:55.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.173+0000 7fb9db75c700 1 -- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9d415d7b0 msgr2=0x7fb9d415b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:55.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.173+0000 7fb9db75c700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9d415d7b0 0x7fb9d415b950 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7fb9c400c780 tx=0x7fb9c400ca90 comp rx=0 tx=0).stop 2024-04-06T13:45:55.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.173+0000 7fb9db75c700 1 -- 172.21.15.189:0/1810241912 shutdown_connections 2024-04-06T13:45:55.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.173+0000 7fb9db75c700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb9b806ae00 0x7fb9b806d2c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.173+0000 7fb9db75c700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9d415e500 0x7fb9d41f24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.173+0000 7fb9db75c700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9d406a730 0x7fb9d415b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.173+0000 7fb9db75c700 1 --2- 172.21.15.189:0/1810241912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9d415d7b0 0x7fb9d415b950 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.173+0000 7fb9db75c700 1 -- 172.21.15.189:0/1810241912 >> 172.21.15.189:0/1810241912 conn(0x7fb9d40d00b0 msgr2=0x7fb9d41506b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:55.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.173+0000 7fb9db75c700 1 -- 172.21.15.189:0/1810241912 shutdown_connections 2024-04-06T13:45:55.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.173+0000 7fb9db75c700 1 -- 172.21.15.189:0/1810241912 wait complete. 2024-04-06T13:45:55.187 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-04-06T13:45:55.187 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2024-04-06T13:45:55.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.314+0000 7f009c195700 1 Processor -- start 2024-04-06T13:45:55.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.314+0000 7f009c195700 1 -- start start 2024-04-06T13:45:55.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.315+0000 7f009c195700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f00940d17a0 0x7f009406b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:55.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.315+0000 7f009c195700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009406d850 con 0x7f0094146dd0 2024-04-06T13:45:55.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.315+0000 7f009c195700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009406c0a0 con 0x7f00940cedf0 2024-04-06T13:45:55.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.315+0000 7f009c195700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f009406c380 con 0x7f00940d17a0 2024-04-06T13:45:55.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.315+0000 7f0099730700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0094146dd0 0x7f009406d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54984/0 (socket says 172.21.15.189:54984) 2024-04-06T13:45:55.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.315+0000 7f0099730700 1 -- 172.21.15.189:0/3938874741 learned_addr learned my addr 172.21.15.189:0/3938874741 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:45:55.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.315+0000 7f0099f31700 1 --2- 172.21.15.189:0/3938874741 >> v2:172.21.15.192:3300/0 conn(0x7f00940d17a0 0x7f009406b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:55.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.315+0000 7f0098f2f700 1 -- 172.21.15.189:0/3938874741 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773893460 0 0) 0x7f009406d850 con 0x7f0094146dd0 2024-04-06T13:45:55.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.315+0000 7f0098f2f700 1 -- 172.21.15.189:0/3938874741 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00800035b0 con 0x7f0094146dd0 2024-04-06T13:45:55.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.315+0000 7f0098f2f700 1 -- 172.21.15.189:0/3938874741 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1042059026 0 0) 0x7f00800035b0 con 0x7f0094146dd0 2024-04-06T13:45:55.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0098f2f700 1 -- 172.21.15.189:0/3938874741 >> v2:172.21.15.192:3300/0 conn(0x7f00940d17a0 msgr2=0x7f009406b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:55.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0098f2f700 1 --2- 172.21.15.189:0/3938874741 >> v2:172.21.15.192:3300/0 conn(0x7f00940d17a0 0x7f009406b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0098f2f700 1 -- 172.21.15.189:0/3938874741 >> v1:172.21.15.192:6789/0 conn(0x7f00940cedf0 legacy=0x7f00940cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:55.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0098f2f700 1 -- 172.21.15.189:0/3938874741 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009406c660 con 0x7f0094146dd0 2024-04-06T13:45:55.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0098f2f700 1 -- 172.21.15.189:0/3938874741 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f0084001670 con 0x7f0094146dd0 2024-04-06T13:45:55.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0098f2f700 1 -- 172.21.15.189:0/3938874741 >> v1:172.21.15.189:6789/0 conn(0x7f0094146dd0 legacy=0x7f009406d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:55.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0098f2f700 1 --2- 172.21.15.189:0/3938874741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0080003a00 0x7f0080003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:55.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0098f2f700 1 --2- 172.21.15.189:0/3938874741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0080004400 0x7f00800048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:55.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0098f2f700 1 --2- 172.21.15.189:0/3938874741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00800050d0 0x7f008000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:55.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0098f2f700 1 -- 172.21.15.189:0/3938874741 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009406c380 con 0x7f0080004400 2024-04-06T13:45:55.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0098f2f700 1 -- 172.21.15.189:0/3938874741 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008000c030 con 0x7f00800050d0 2024-04-06T13:45:55.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0098f2f700 1 -- 172.21.15.189:0/3938874741 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f008000c1f0 con 0x7f0080003a00 2024-04-06T13:45:55.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0099730700 1 --2- 172.21.15.189:0/3938874741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0080004400 0x7f00800048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:55.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0099f31700 1 --2- 172.21.15.189:0/3938874741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0080003a00 0x7f0080003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:55.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f009a732700 1 --2- 172.21.15.189:0/3938874741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00800050d0 0x7f008000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:55.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0099730700 1 -- 172.21.15.189:0/3938874741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0080003a00 msgr2=0x7f0080003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:55.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0099730700 1 --2- 172.21.15.189:0/3938874741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0080003a00 0x7f0080003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0099730700 1 -- 172.21.15.189:0/3938874741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00800050d0 msgr2=0x7f008000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:55.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0099730700 1 --2- 172.21.15.189:0/3938874741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00800050d0 0x7f008000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f009a732700 1 -- 172.21.15.189:0/3938874741 reap_dead start 2024-04-06T13:45:55.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0099730700 1 -- 172.21.15.189:0/3938874741 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00800035b0 con 0x7f0080004400 2024-04-06T13:45:55.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0099730700 1 --2- 172.21.15.189:0/3938874741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0080004400 0x7f00800048d0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f00941471e0 tx=0x7f0080005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c956b3cb935bf server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:55.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0098f2f700 1 -- 172.21.15.189:0/3938874741 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008400ce00 con 0x7f0080004400 2024-04-06T13:45:55.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0098f2f700 1 -- 172.21.15.189:0/3938874741 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0084005960 con 0x7f0080004400 2024-04-06T13:45:55.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.316+0000 7f0098f2f700 1 -- 172.21.15.189:0/3938874741 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0084005d10 con 0x7f0080004400 2024-04-06T13:45:55.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.317+0000 7f009c195700 1 -- 172.21.15.189:0/3938874741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0080004400 msgr2=0x7f00800048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:55.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.317+0000 7f009c195700 1 --2- 172.21.15.189:0/3938874741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0080004400 0x7f00800048d0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f00941471e0 tx=0x7f0080005880 comp rx=0 tx=0).stop 2024-04-06T13:45:55.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.317+0000 7f009c195700 1 -- 172.21.15.189:0/3938874741 shutdown_connections 2024-04-06T13:45:55.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.317+0000 7f009c195700 1 --2- 172.21.15.189:0/3938874741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0080004400 0x7f00800048d0 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.317+0000 7f009c195700 1 -- 172.21.15.189:0/3938874741 >> 172.21.15.189:0/3938874741 conn(0x7f00940b2db0 msgr2=0x7f00940b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:55.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.317+0000 7f009c195700 1 -- 172.21.15.189:0/3938874741 shutdown_connections 2024-04-06T13:45:55.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.317+0000 7f009c195700 1 -- 172.21.15.189:0/3938874741 wait complete. 2024-04-06T13:45:55.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.318+0000 7f009c195700 1 Processor -- start 2024-04-06T13:45:55.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.318+0000 7f009c195700 1 -- start start 2024-04-06T13:45:55.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.319+0000 7f009c195700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0080004400 0x7f00941f2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:55.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.319+0000 7f009c195700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f009415bd80 0x7f00940afe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:55.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.319+0000 7f009c195700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f009415c200 0x7f00940b0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:55.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.319+0000 7f009c195700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00941f2cb0 con 0x7f009415c200 2024-04-06T13:45:55.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.319+0000 7f009c195700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00940c69a0 con 0x7f0080004400 2024-04-06T13:45:55.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.319+0000 7f009c195700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00940c6c80 con 0x7f009415bd80 2024-04-06T13:45:55.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.319+0000 7f009a732700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f009415c200 0x7f00940b0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:55.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.319+0000 7f009a732700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f009415c200 0x7f00940b0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51712/0 (socket says 172.21.15.189:51712) 2024-04-06T13:45:55.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.319+0000 7f009a732700 1 -- 172.21.15.189:0/4041428303 learned_addr learned my addr 172.21.15.189:0/4041428303 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:55.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.319+0000 7f0099730700 1 --2- 172.21.15.189:0/4041428303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f009415bd80 0x7f00940afe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:55.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.319+0000 7f0099f31700 1 --2- 172.21.15.189:0/4041428303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0080004400 0x7f00941f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:55.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.319+0000 7f009a732700 1 -- 172.21.15.189:0/4041428303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f009415bd80 msgr2=0x7f00940afe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:55.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.319+0000 7f009a732700 1 --2- 172.21.15.189:0/4041428303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f009415bd80 0x7f00940afe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.319+0000 7f009a732700 1 -- 172.21.15.189:0/4041428303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0080004400 msgr2=0x7f00941f2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:55.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.319+0000 7f009a732700 1 --2- 172.21.15.189:0/4041428303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0080004400 0x7f00941f2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.319+0000 7f009a732700 1 -- 172.21.15.189:0/4041428303 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00940c6f60 con 0x7f009415c200 2024-04-06T13:45:55.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.319+0000 7f009a732700 1 --2- 172.21.15.189:0/4041428303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f009415c200 0x7f00940b0390 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f00940ced00 tx=0x7f008c00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:55.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.320+0000 7f008a7fc700 1 -- 172.21.15.189:0/4041428303 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c003c10 con 0x7f009415c200 2024-04-06T13:45:55.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.320+0000 7f008a7fc700 1 -- 172.21.15.189:0/4041428303 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008c003d70 con 0x7f009415c200 2024-04-06T13:45:55.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.320+0000 7f008a7fc700 1 -- 172.21.15.189:0/4041428303 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c00db90 con 0x7f009415c200 2024-04-06T13:45:55.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.320+0000 7f009c195700 1 -- 172.21.15.189:0/4041428303 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00940ad480 con 0x7f009415c200 2024-04-06T13:45:55.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.320+0000 7f009c195700 1 -- 172.21.15.189:0/4041428303 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f009415dec0 con 0x7f009415c200 2024-04-06T13:45:55.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.321+0000 7f009c195700 1 -- 172.21.15.189:0/4041428303 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f009415c930 con 0x7f009415c200 2024-04-06T13:45:55.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.321+0000 7f008a7fc700 1 -- 172.21.15.189:0/4041428303 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 131) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f008c00dcf0 con 0x7f009415c200 2024-04-06T13:45:55.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.321+0000 7f008a7fc700 1 --2- 172.21.15.189:0/4041428303 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f007806ab30 0x7f007806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:55.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.321+0000 7f008a7fc700 1 -- 172.21.15.189:0/4041428303 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f008c0895c0 con 0x7f009415c200 2024-04-06T13:45:55.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.322+0000 7f0099f31700 1 --2- 172.21.15.189:0/4041428303 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f007806ab30 0x7f007806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:55.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.322+0000 7f0099f31700 1 --2- 172.21.15.189:0/4041428303 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f007806ab30 0x7f007806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0090009ba0 tx=0x7f0090008040 comp rx=0 tx=0).ready entity=mgr.10917 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:55.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.326+0000 7f008a7fc700 1 -- 172.21.15.189:0/4041428303 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f008c054800 con 0x7f009415c200 2024-04-06T13:45:55.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.546+0000 7f009c195700 1 -- 172.21.15.189:0/4041428303 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f00940d1b80 con 0x7f007806ab30 2024-04-06T13:45:55.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.592+0000 7f008a7fc700 1 -- 172.21.15.189:0/4041428303 <== mgr.10917 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7f00900097f0 con 0x7f007806ab30 2024-04-06T13:45:55.592 INFO:teuthology.orchestra.run.smithi189.stdout:Retention added to path snap_test_dir1 2024-04-06T13:45:55.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.594+0000 7f009c195700 1 -- 172.21.15.189:0/4041428303 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f007806ab30 msgr2=0x7f007806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.594+0000 7f009c195700 1 --2- 172.21.15.189:0/4041428303 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f007806ab30 0x7f007806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0090009ba0 tx=0x7f0090008040 comp rx=0 tx=0).stop 2024-04-06T13:45:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.594+0000 7f009c195700 1 -- 172.21.15.189:0/4041428303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f009415c200 msgr2=0x7f00940b0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.594+0000 7f009c195700 1 --2- 172.21.15.189:0/4041428303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f009415c200 0x7f00940b0390 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f00940ced00 tx=0x7f008c00d430 comp rx=0 tx=0).stop 2024-04-06T13:45:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.594+0000 7f009c195700 1 -- 172.21.15.189:0/4041428303 shutdown_connections 2024-04-06T13:45:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.594+0000 7f009c195700 1 --2- 172.21.15.189:0/4041428303 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f007806ab30 0x7f007806cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.594+0000 7f009c195700 1 --2- 172.21.15.189:0/4041428303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f009415c200 0x7f00940b0390 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.594+0000 7f009c195700 1 --2- 172.21.15.189:0/4041428303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0080004400 0x7f00941f2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.594+0000 7f009c195700 1 --2- 172.21.15.189:0/4041428303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f009415bd80 0x7f00940afe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.594+0000 7f009c195700 1 -- 172.21.15.189:0/4041428303 >> 172.21.15.189:0/4041428303 conn(0x7f00940b2db0 msgr2=0x7f009406d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.594+0000 7f009c195700 1 -- 172.21.15.189:0/4041428303 shutdown_connections 2024-04-06T13:45:55.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.594+0000 7f009c195700 1 -- 172.21.15.189:0/4041428303 wait complete. 2024-04-06T13:45:55.606 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2024-04-06T13:45:55.607 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-04-06-13_46_00 in ~5s... 2024-04-06T13:45:55.607 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2024-04-06T13:45:55.607 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-04-06T13:45:55.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.734+0000 7fd9960b4700 1 Processor -- start 2024-04-06T13:45:55.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.734+0000 7fd9960b4700 1 -- start start 2024-04-06T13:45:55.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.735+0000 7fd9960b4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd9900d17a0 0x7fd99006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:55.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.735+0000 7fd9960b4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd9900cedf0 0x7fd9900cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:55.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.735+0000 7fd9960b4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9900cf730 con 0x7fd990146dd0 2024-04-06T13:45:55.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.735+0000 7fd9960b4700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd99006d850 con 0x7fd9900d17a0 2024-04-06T13:45:55.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.735+0000 7fd9960b4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd99006c0a0 con 0x7fd9900cedf0 2024-04-06T13:45:55.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.735+0000 7fd98f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd9900d17a0 0x7fd99006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:55.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.735+0000 7fd98f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd9900d17a0 0x7fd99006b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51728/0 (socket says 172.21.15.189:51728) 2024-04-06T13:45:55.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.735+0000 7fd98f7fe700 1 -- 172.21.15.189:0/3827763925 learned_addr learned my addr 172.21.15.189:0/3827763925 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:55.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.736+0000 7fd98ffff700 1 --2- 172.21.15.189:0/3827763925 >> v2:172.21.15.192:3300/0 conn(0x7fd9900cedf0 0x7fd9900cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:55.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.736+0000 7fd98f7fe700 1 -- 172.21.15.189:0/3827763925 >> v2:172.21.15.192:3300/0 conn(0x7fd9900cedf0 msgr2=0x7fd9900cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:45:55.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.736+0000 7fd98f7fe700 1 --2- 172.21.15.189:0/3827763925 >> v2:172.21.15.192:3300/0 conn(0x7fd9900cedf0 0x7fd9900cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.736+0000 7fd98f7fe700 1 -- 172.21.15.189:0/3827763925 >> v1:172.21.15.192:6789/0 conn(0x7fd990146dd0 legacy=0x7fd99006d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.736+0000 7fd98f7fe700 1 -- 172.21.15.189:0/3827763925 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99006c380 con 0x7fd9900d17a0 2024-04-06T13:45:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.736+0000 7fd98f7fe700 1 --2- 172.21.15.189:0/3827763925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9900d17a0 0x7fd99006b7a0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fd990160fb0 tx=0x7fd984009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b459151b378ff2a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.736+0000 7fd98e7fc700 1 -- 172.21.15.189:0/3827763925 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98400b870 con 0x7fd9900d17a0 2024-04-06T13:45:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.736+0000 7fd98e7fc700 1 -- 172.21.15.189:0/3827763925 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd98400b9d0 con 0x7fd9900d17a0 2024-04-06T13:45:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.736+0000 7fd98e7fc700 1 -- 172.21.15.189:0/3827763925 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98400bd60 con 0x7fd9900d17a0 2024-04-06T13:45:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.737+0000 7fd9960b4700 1 -- 172.21.15.189:0/3827763925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9900d17a0 msgr2=0x7fd99006b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.737+0000 7fd9960b4700 1 --2- 172.21.15.189:0/3827763925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9900d17a0 0x7fd99006b7a0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fd990160fb0 tx=0x7fd984009bd0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.737+0000 7fd9960b4700 1 -- 172.21.15.189:0/3827763925 shutdown_connections 2024-04-06T13:45:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.737+0000 7fd9960b4700 1 --2- 172.21.15.189:0/3827763925 >> v2:172.21.15.192:3300/0 conn(0x7fd9900cedf0 0x7fd9900cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.737+0000 7fd9960b4700 1 --2- 172.21.15.189:0/3827763925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9900d17a0 0x7fd99006b7a0 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.737+0000 7fd9960b4700 1 -- 172.21.15.189:0/3827763925 >> 172.21.15.189:0/3827763925 conn(0x7fd9900b2db0 msgr2=0x7fd9900b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.737+0000 7fd9960b4700 1 -- 172.21.15.189:0/3827763925 shutdown_connections 2024-04-06T13:45:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.737+0000 7fd9960b4700 1 -- 172.21.15.189:0/3827763925 wait complete. 2024-04-06T13:45:55.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.738+0000 7fd9960b4700 1 Processor -- start 2024-04-06T13:45:55.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.738+0000 7fd9960b4700 1 -- start start 2024-04-06T13:45:55.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd9960b4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9900cedf0 0x7fd990156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:55.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd9960b4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9900d17a0 0x7fd9901570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:55.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd9960b4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd990146dd0 0x7fd9900c3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:55.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd9960b4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99015a6f0 con 0x7fd990146dd0 2024-04-06T13:45:55.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd9960b4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9901e6ec0 con 0x7fd9900cedf0 2024-04-06T13:45:55.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd9960b4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9900c5b20 con 0x7fd9900d17a0 2024-04-06T13:45:55.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd98ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd990146dd0 0x7fd9900c3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:55.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd98ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd990146dd0 0x7fd9900c3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51734/0 (socket says 172.21.15.189:51734) 2024-04-06T13:45:55.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd98ffff700 1 -- 172.21.15.189:0/3858390617 learned_addr learned my addr 172.21.15.189:0/3858390617 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:45:55.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd98f7fe700 1 --2- 172.21.15.189:0/3858390617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9900cedf0 0x7fd990156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:55.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd98effd700 1 --2- 172.21.15.189:0/3858390617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9900d17a0 0x7fd9901570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:55.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd98ffff700 1 -- 172.21.15.189:0/3858390617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9900d17a0 msgr2=0x7fd9901570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:55.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd98ffff700 1 --2- 172.21.15.189:0/3858390617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9900d17a0 0x7fd9901570e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd98ffff700 1 -- 172.21.15.189:0/3858390617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9900cedf0 msgr2=0x7fd990156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:55.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd98ffff700 1 --2- 172.21.15.189:0/3858390617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9900cedf0 0x7fd990156ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd98ffff700 1 -- 172.21.15.189:0/3858390617 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd984009550 con 0x7fd990146dd0 2024-04-06T13:45:55.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd98f7fe700 1 --2- 172.21.15.189:0/3858390617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9900cedf0 0x7fd990156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:45:55.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd98ffff700 1 --2- 172.21.15.189:0/3858390617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd990146dd0 0x7fd9900c3860 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7fd988007e60 tx=0x7fd98800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:55.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd977fff700 1 -- 172.21.15.189:0/3858390617 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98800ede0 con 0x7fd990146dd0 2024-04-06T13:45:55.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd977fff700 1 -- 172.21.15.189:0/3858390617 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd988014420 con 0x7fd990146dd0 2024-04-06T13:45:55.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd9960b4700 1 -- 172.21.15.189:0/3858390617 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd990159e50 con 0x7fd990146dd0 2024-04-06T13:45:55.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd977fff700 1 -- 172.21.15.189:0/3858390617 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9880147d0 con 0x7fd990146dd0 2024-04-06T13:45:55.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.739+0000 7fd9960b4700 1 -- 172.21.15.189:0/3858390617 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd99015a410 con 0x7fd990146dd0 2024-04-06T13:45:55.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.740+0000 7fd9960b4700 1 -- 172.21.15.189:0/3858390617 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd990074990 con 0x7fd990146dd0 2024-04-06T13:45:55.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.741+0000 7fd977fff700 1 -- 172.21.15.189:0/3858390617 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 131) v1 ==== 89000+0+0 (secure 0 0 0) 0x7fd988014930 con 0x7fd990146dd0 2024-04-06T13:45:55.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.741+0000 7fd977fff700 1 --2- 172.21.15.189:0/3858390617 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd97c06ab30 0x7fd97c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:45:55.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.741+0000 7fd977fff700 1 -- 172.21.15.189:0/3858390617 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6445+0+0 (secure 0 0 0) 0x7fd9880899e0 con 0x7fd990146dd0 2024-04-06T13:45:55.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.741+0000 7fd98f7fe700 1 --2- 172.21.15.189:0/3858390617 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd97c06ab30 0x7fd97c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:45:55.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.742+0000 7fd98f7fe700 1 --2- 172.21.15.189:0/3858390617 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd97c06ab30 0x7fd97c06cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd984014040 tx=0x7fd984008030 comp rx=0 tx=0).ready entity=mgr.10917 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:45:55.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.746+0000 7fd977fff700 1 -- 172.21.15.189:0/3858390617 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd988054c20 con 0x7fd990146dd0 2024-04-06T13:45:55.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.965+0000 7fd9960b4700 1 -- 172.21.15.189:0/3858390617 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd99006cc20 con 0x7fd97c06ab30 2024-04-06T13:45:55.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.973+0000 7fd977fff700 1 -- 172.21.15.189:0/3858390617 <== mgr.10917 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7fd98400bbc0 con 0x7fd97c06ab30 2024-04-06T13:45:55.974 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:45:55.974 INFO:teuthology.orchestra.run.smithi189.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-04-06T13:45:55.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.976+0000 7fd9960b4700 1 -- 172.21.15.189:0/3858390617 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd97c06ab30 msgr2=0x7fd97c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:55.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.976+0000 7fd9960b4700 1 --2- 172.21.15.189:0/3858390617 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd97c06ab30 0x7fd97c06cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd984014040 tx=0x7fd984008030 comp rx=0 tx=0).stop 2024-04-06T13:45:55.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.976+0000 7fd9960b4700 1 -- 172.21.15.189:0/3858390617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd990146dd0 msgr2=0x7fd9900c3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:45:55.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.976+0000 7fd9960b4700 1 --2- 172.21.15.189:0/3858390617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd990146dd0 0x7fd9900c3860 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7fd988007e60 tx=0x7fd98800e3c0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.976+0000 7fd9960b4700 1 -- 172.21.15.189:0/3858390617 shutdown_connections 2024-04-06T13:45:55.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.976+0000 7fd9960b4700 1 --2- 172.21.15.189:0/3858390617 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd97c06ab30 0x7fd97c06cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.976+0000 7fd9960b4700 1 --2- 172.21.15.189:0/3858390617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd990146dd0 0x7fd9900c3860 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.976+0000 7fd9960b4700 1 --2- 172.21.15.189:0/3858390617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9900cedf0 0x7fd990156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.976+0000 7fd9960b4700 1 --2- 172.21.15.189:0/3858390617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9900d17a0 0x7fd9901570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:45:55.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.976+0000 7fd9960b4700 1 -- 172.21.15.189:0/3858390617 >> 172.21.15.189:0/3858390617 conn(0x7fd9900b2db0 msgr2=0x7fd9900b2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:45:55.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.976+0000 7fd9960b4700 1 -- 172.21.15.189:0/3858390617 shutdown_connections 2024-04-06T13:45:55.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:45:55.976+0000 7fd9960b4700 1 -- 172.21.15.189:0/3858390617 wait complete. 2024-04-06T13:45:55.988 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-04-06T13:45:55.988 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2024-04-06T13:45:55.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:55.989 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:45:56.006 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T13:45:56.007 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:45:56.007 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:45:57.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:57.008 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:45:57.027 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T13:45:57.028 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:45:57.028 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:45:58.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:58.029 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:45:58.051 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T13:45:58.051 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:45:58.051 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:45:59.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:45:59.052 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:45:59.074 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T13:45:59.074 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:45:59.074 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:00.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:00.075 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:00.098 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-06T13:46:00.098 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:00.098 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:00.601 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:46:00.601+0000 7f9f72a05700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:46:00.601 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:46:00.601+0000 7f9f72a05700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:46:00.601 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:46:00.601+0000 7f9f72a05700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:46:00.601 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:46:00.601+0000 7f9f72a05700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:46:00.602 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:46:00.601+0000 7f9f72a05700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:46:01.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:01.099 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:01.123 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:01.123 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:01.123 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:01.123 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:01.124 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-04-06-13_46_00_UTC'] 2024-04-06T13:46:01.124 DEBUG:tasks.cephfs.test_snap_schedules:exec=1712411155.607073, now = 1712411161.124148, timo = 5 2024-04-06T13:46:02.125 DEBUG:tasks.cephfs.test_snap_schedules:expecting removal of snap snap_test_dir1/.snap/scheduled-2024-04-06-13_46_00 in ~65s... 2024-04-06T13:46:02.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:02.125 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:02.146 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:02.147 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:02.147 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:02.147 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:02.632 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:46:02.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:46:03.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:03.148 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:03.171 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:03.172 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:03.172 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:03.172 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:04.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:04.173 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:04.198 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:04.199 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:04.199 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:04.199 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:05.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:05.200 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:05.222 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:05.223 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:05.223 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:05.223 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:06.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:06.224 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:06.246 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:06.246 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:06.246 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:06.246 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:07.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:07.247 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:07.270 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:07.271 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:07.271 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:07.271 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:08.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:08.273 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:08.295 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:08.375 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:08.375 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:08.375 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:09.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:09.376 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:09.399 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:09.399 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:09.399 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:09.399 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:10.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:10.400 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:10.422 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:10.423 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:10.423 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:10.423 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:11.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:11.424 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:11.447 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:11.447 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:11.447 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:11.447 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:12.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:12.449 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:12.470 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:12.471 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:12.471 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:12.471 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:13.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:13.472 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:13.495 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:13.495 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:13.495 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:13.495 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:14.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:14.496 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:14.518 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:14.519 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:14.519 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:14.519 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:15.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:15.520 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:15.542 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:15.543 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:15.543 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:15.543 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:16.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:16.544 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:16.565 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:16.566 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:16.566 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:16.567 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:17.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:17.568 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:17.589 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:17.590 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:17.590 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:17.590 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:18.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:18.591 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:18.628 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:18.629 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:18.629 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:18.629 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:19.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:19.630 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:19.652 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:19.653 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:19.653 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:19.653 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:20.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:20.654 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:20.677 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:20.677 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:20.677 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:20.677 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:21.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:21.678 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:21.701 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:21.701 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:21.701 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:21.701 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:22.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:22.703 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:22.725 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:22.725 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:22.725 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:22.725 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:23.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:23.726 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:23.749 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:23.749 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:23.749 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:23.749 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:24.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:24.751 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:24.772 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:24.773 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:24.773 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:24.773 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:25.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:25.774 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:25.796 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:25.797 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:25.797 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:25.797 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:26.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:26.798 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:26.820 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:26.820 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:26.820 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:26.821 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:27.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:27.822 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:27.844 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:27.844 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:27.844 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:27.844 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:28.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:28.845 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:28.867 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:28.867 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:28.868 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:28.868 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:29.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:29.868 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:29.890 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:29.891 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:29.891 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:29.891 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:30.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:30.893 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:30.915 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:30.915 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:30.915 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:30.915 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:31.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:31.916 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:31.938 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:31.938 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:31.938 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:31.939 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:32.759 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:46:32.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:46:32.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:32.939 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:32.960 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:32.960 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:32.960 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:32.960 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:33.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:33.961 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:33.984 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:33.984 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:33.984 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:33.984 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:34.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:34.985 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:35.009 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:35.009 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:35.009 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:35.009 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:36.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:36.010 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:36.032 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:36.033 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:36.033 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:36.033 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:37.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:37.034 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:37.056 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:37.056 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:37.056 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:37.056 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:38.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:38.057 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:38.079 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:38.079 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:38.079 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:38.079 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:39.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:39.080 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:39.102 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:39.103 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:39.103 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:39.103 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:40.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:40.104 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:40.126 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:40.127 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:40.127 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:40.127 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:41.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:41.128 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:41.149 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:41.150 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:41.150 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:41.150 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:42.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:42.151 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:42.173 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:42.173 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:42.173 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:42.173 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:43.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:43.175 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:43.196 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:43.197 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:43.197 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:43.197 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:44.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:44.198 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:44.219 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:44.219 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:44.220 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:44.220 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:45.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:45.221 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:45.243 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:45.243 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:45.243 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:45.243 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:46.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:46.245 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:46.267 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:46.268 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:46.268 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:46.268 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:47.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:47.269 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:47.291 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:47.291 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:47.291 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:47.291 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:48.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:48.292 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:48.314 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:48.315 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:48.315 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:48.315 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:49.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:49.316 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:49.337 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:49.338 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:49.338 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:49.338 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:50.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:50.339 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:50.361 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:50.362 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:50.362 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:50.362 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:51.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:51.363 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:51.385 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:51.386 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:51.386 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:51.386 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:52.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:52.388 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:52.409 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:52.410 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:52.410 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:52.410 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:53.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:53.412 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:53.434 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:53.434 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:53.434 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:53.434 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:54.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:54.435 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:54.457 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:54.457 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:54.458 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:54.458 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:55.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:55.459 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:55.481 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:55.482 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:55.482 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:55.482 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:56.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:56.483 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:56.505 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:56.505 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:56.505 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:56.505 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:57.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:57.507 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:57.530 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:57.530 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:57.530 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:57.530 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:58.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:58.532 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:58.554 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:58.554 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:58.554 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:58.554 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:46:59.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:46:59.556 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:46:59.578 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:46:59.578 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:46:59.578 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:46:59.578 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:47:00.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:00.579 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:47:00.601 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_46_00_UTC 2024-04-06T13:47:00.601 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:47:00.602 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-06T13:47:00.602 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-06T13:47:01.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:01.603 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:47:01.625 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_47_00_UTC 2024-04-06T13:47:01.625 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-06-13_47_00_UTC'} 2024-04-06T13:47:01.626 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-04-06-13_47_00_UTC'} 2024-04-06T13:47:01.626 INFO:tasks.cephfs.test_snap_schedules:removed: {'scheduled-2024-04-06-13_46_00_UTC'} 2024-04-06T13:47:01.626 DEBUG:tasks.cephfs.test_snap_schedules:snapshots removed=['scheduled-2024-04-06-13_46_00_UTC'] 2024-04-06T13:47:01.626 DEBUG:tasks.cephfs.test_snap_schedules:exec=1712411155.607073, now = 1712411221.6263297, timo = 65 2024-04-06T13:47:02.627 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-04-06T13:47:02.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.775+0000 7f7e0e17a700 1 Processor -- start 2024-04-06T13:47:02.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.775+0000 7f7e0e17a700 1 -- start start 2024-04-06T13:47:02.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.776+0000 7f7e0e17a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7e0815d7c0 0x7f7e0806b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:02.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.776+0000 7f7e0e17a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0806ceb0 con 0x7f7e0806b6f0 2024-04-06T13:47:02.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.776+0000 7f7e0e17a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0806d750 con 0x7f7e080cd3e0 2024-04-06T13:47:02.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.776+0000 7f7e0e17a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e080cdfa0 con 0x7f7e0815d7c0 2024-04-06T13:47:02.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.776+0000 7f7e07fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7e0806b6f0 0x7f7e0806dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42644/0 (socket says 172.21.15.189:42644) 2024-04-06T13:47:02.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.776+0000 7f7e07fff700 1 -- 172.21.15.189:0/1568002994 learned_addr learned my addr 172.21.15.189:0/1568002994 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:02.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.776+0000 7f7e06ffd700 1 --2- 172.21.15.189:0/1568002994 >> v2:172.21.15.192:3300/0 conn(0x7f7e0815d7c0 0x7f7e0806b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:02.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.776+0000 7f7e067fc700 1 -- 172.21.15.189:0/1568002994 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356039934 0 0) 0x7f7e0806ceb0 con 0x7f7e0806b6f0 2024-04-06T13:47:02.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.776+0000 7f7e067fc700 1 -- 172.21.15.189:0/1568002994 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df00035b0 con 0x7f7e0806b6f0 2024-04-06T13:47:02.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e067fc700 1 -- 172.21.15.189:0/1568002994 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733178693 0 0) 0x7f7e0806d750 con 0x7f7e080cd3e0 2024-04-06T13:47:02.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e067fc700 1 -- 172.21.15.189:0/1568002994 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e0806ceb0 con 0x7f7e080cd3e0 2024-04-06T13:47:02.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e067fc700 1 -- 172.21.15.189:0/1568002994 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3557371784 0 0) 0x7f7df00035b0 con 0x7f7e0806b6f0 2024-04-06T13:47:02.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e067fc700 1 -- 172.21.15.189:0/1568002994 >> v2:172.21.15.192:3300/0 conn(0x7f7e0815d7c0 msgr2=0x7f7e0806b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:02.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e067fc700 1 --2- 172.21.15.189:0/1568002994 >> v2:172.21.15.192:3300/0 conn(0x7f7e0815d7c0 0x7f7e0806b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:02.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e067fc700 1 -- 172.21.15.189:0/1568002994 >> v1:172.21.15.192:6789/0 conn(0x7f7e080cd3e0 legacy=0x7f7e080cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:02.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e067fc700 1 -- 172.21.15.189:0/1568002994 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0806bf00 con 0x7f7e0806b6f0 2024-04-06T13:47:02.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e067fc700 1 -- 172.21.15.189:0/1568002994 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f7e000019f0 con 0x7f7e0806b6f0 2024-04-06T13:47:02.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e067fc700 1 -- 172.21.15.189:0/1568002994 >> v1:172.21.15.189:6789/0 conn(0x7f7e0806b6f0 legacy=0x7f7e0806dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:02.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e067fc700 1 --2- 172.21.15.189:0/1568002994 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7df0003b90 0x7f7df0003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:02.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e067fc700 1 --2- 172.21.15.189:0/1568002994 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7df0004850 0x7f7df0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:02.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e067fc700 1 --2- 172.21.15.189:0/1568002994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7df0005210 0x7f7df000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e067fc700 1 -- 172.21.15.189:0/1568002994 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e080cdfa0 con 0x7f7df0005210 2024-04-06T13:47:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e067fc700 1 -- 172.21.15.189:0/1568002994 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df000c140 con 0x7f7df0003b90 2024-04-06T13:47:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e067fc700 1 -- 172.21.15.189:0/1568002994 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df000c300 con 0x7f7df0004850 2024-04-06T13:47:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e077fe700 1 --2- 172.21.15.189:0/1568002994 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7df0003b90 0x7f7df0003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e07fff700 1 --2- 172.21.15.189:0/1568002994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7df0005210 0x7f7df000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e06ffd700 1 --2- 172.21.15.189:0/1568002994 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7df0004850 0x7f7df0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e07fff700 1 -- 172.21.15.189:0/1568002994 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7df0004850 msgr2=0x7f7df0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e07fff700 1 --2- 172.21.15.189:0/1568002994 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7df0004850 0x7f7df0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e07fff700 1 -- 172.21.15.189:0/1568002994 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7df0003b90 msgr2=0x7f7df0003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e07fff700 1 --2- 172.21.15.189:0/1568002994 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7df0003b90 0x7f7df0003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e07fff700 1 -- 172.21.15.189:0/1568002994 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df00035b0 con 0x7f7df0005210 2024-04-06T13:47:02.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e077fe700 1 --2- 172.21.15.189:0/1568002994 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7df0003b90 0x7f7df0003f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:02.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e07fff700 1 -- 172.21.15.189:0/1568002994 reap_dead start 2024-04-06T13:47:02.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e07fff700 1 --2- 172.21.15.189:0/1568002994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7df0005210 0x7f7df000ba10 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f7e0000a7a0 tx=0x7f7df0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95347e997df78976 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:02.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.777+0000 7f7e067fc700 1 -- 172.21.15.189:0/1568002994 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00012600 con 0x7f7df0005210 2024-04-06T13:47:02.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.778+0000 7f7e067fc700 1 -- 172.21.15.189:0/1568002994 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e00012760 con 0x7f7df0005210 2024-04-06T13:47:02.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.778+0000 7f7e067fc700 1 -- 172.21.15.189:0/1568002994 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00012af0 con 0x7f7df0005210 2024-04-06T13:47:02.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.778+0000 7f7e0e17a700 1 -- 172.21.15.189:0/1568002994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7df0005210 msgr2=0x7f7df000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:02.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.778+0000 7f7e0e17a700 1 --2- 172.21.15.189:0/1568002994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7df0005210 0x7f7df000ba10 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f7e0000a7a0 tx=0x7f7df0005880 comp rx=0 tx=0).stop 2024-04-06T13:47:02.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.778+0000 7f7e0e17a700 1 -- 172.21.15.189:0/1568002994 shutdown_connections 2024-04-06T13:47:02.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.778+0000 7f7e0e17a700 1 --2- 172.21.15.189:0/1568002994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7df0005210 0x7f7df000ba10 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:02.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.778+0000 7f7e0e17a700 1 -- 172.21.15.189:0/1568002994 >> 172.21.15.189:0/1568002994 conn(0x7f7e080cc290 msgr2=0x7f7e080cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:02.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.778+0000 7f7e0e17a700 1 -- 172.21.15.189:0/1568002994 shutdown_connections 2024-04-06T13:47:02.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.778+0000 7f7e0e17a700 1 -- 172.21.15.189:0/1568002994 wait complete. 2024-04-06T13:47:02.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.779+0000 7f7e0e17a700 1 Processor -- start 2024-04-06T13:47:02.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.780+0000 7f7e0e17a700 1 -- start start 2024-04-06T13:47:02.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.780+0000 7f7e0e17a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7df0005210 0x7f7e0815ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:02.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.780+0000 7f7e0e17a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e08159860 0x7f7e080ae550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:02.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.780+0000 7f7e0e17a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e0815b8d0 0x7f7e0815bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:02.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.780+0000 7f7e0e17a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e080bc6b0 con 0x7f7df0005210 2024-04-06T13:47:02.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.780+0000 7f7e0e17a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0815c280 con 0x7f7e0815b8d0 2024-04-06T13:47:02.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.780+0000 7f7e0e17a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0815a900 con 0x7f7e08159860 2024-04-06T13:47:02.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.780+0000 7f7e077fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7df0005210 0x7f7e0815ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:02.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.781+0000 7f7e077fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7df0005210 0x7f7e0815ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46956/0 (socket says 172.21.15.189:46956) 2024-04-06T13:47:02.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.781+0000 7f7e077fe700 1 -- 172.21.15.189:0/2041215041 learned_addr learned my addr 172.21.15.189:0/2041215041 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:02.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.781+0000 7f7e07fff700 1 --2- 172.21.15.189:0/2041215041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e0815b8d0 0x7f7e0815bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:02.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.781+0000 7f7e06ffd700 1 --2- 172.21.15.189:0/2041215041 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e08159860 0x7f7e080ae550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:02.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.781+0000 7f7e077fe700 1 -- 172.21.15.189:0/2041215041 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e08159860 msgr2=0x7f7e080ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:02.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.781+0000 7f7e077fe700 1 --2- 172.21.15.189:0/2041215041 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e08159860 0x7f7e080ae550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:02.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.781+0000 7f7e077fe700 1 -- 172.21.15.189:0/2041215041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e0815b8d0 msgr2=0x7f7e0815bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:02.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.781+0000 7f7e077fe700 1 --2- 172.21.15.189:0/2041215041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e0815b8d0 0x7f7e0815bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:02.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.781+0000 7f7e077fe700 1 -- 172.21.15.189:0/2041215041 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e081577b0 con 0x7f7df0005210 2024-04-06T13:47:02.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.781+0000 7f7e077fe700 1 --2- 172.21.15.189:0/2041215041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7df0005210 0x7f7e0815ad90 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00f8c0 tx=0x7f7dfc00fc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:02.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.781+0000 7f7deffff700 1 -- 172.21.15.189:0/2041215041 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc00fec0 con 0x7f7df0005210 2024-04-06T13:47:02.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.781+0000 7f7deffff700 1 -- 172.21.15.189:0/2041215041 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dfc0052e0 con 0x7f7df0005210 2024-04-06T13:47:02.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.781+0000 7f7e0e17a700 1 -- 172.21.15.189:0/2041215041 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e08159cf0 con 0x7f7df0005210 2024-04-06T13:47:02.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.781+0000 7f7deffff700 1 -- 172.21.15.189:0/2041215041 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc003680 con 0x7f7df0005210 2024-04-06T13:47:02.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.781+0000 7f7e0e17a700 1 -- 172.21.15.189:0/2041215041 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e0815a2b0 con 0x7f7df0005210 2024-04-06T13:47:02.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.782+0000 7f7e0e17a700 1 -- 172.21.15.189:0/2041215041 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e08074990 con 0x7f7df0005210 2024-04-06T13:47:02.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.783+0000 7f7deffff700 1 -- 172.21.15.189:0/2041215041 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 134) v1 ==== 89378+0+0 (secure 0 0 0) 0x7f7dfc0037e0 con 0x7f7df0005210 2024-04-06T13:47:02.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.783+0000 7f7deffff700 1 --2- 172.21.15.189:0/2041215041 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7df406af20 0x7f7df406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:02.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.783+0000 7f7deffff700 1 -- 172.21.15.189:0/2041215041 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7dfc089b60 con 0x7f7df0005210 2024-04-06T13:47:02.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.783+0000 7f7e06ffd700 1 --2- 172.21.15.189:0/2041215041 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7df406af20 0x7f7df406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:02.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.783+0000 7f7e06ffd700 1 --2- 172.21.15.189:0/2041215041 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7df406af20 0x7f7df406d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e080af070 tx=0x7f7df8008040 comp rx=0 tx=0).ready entity=mgr.10917 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:02.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:02.788+0000 7f7deffff700 1 -- 172.21.15.189:0/2041215041 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7dfc0546a0 con 0x7f7df0005210 2024-04-06T13:47:02.888 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:47:02.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:47:03.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.012+0000 7f7e0e17a700 1 -- 172.21.15.189:0/2041215041 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e0806d3e0 con 0x7f7df406af20 2024-04-06T13:47:03.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.072+0000 7f7deffff700 1 -- 172.21.15.189:0/2041215041 <== mgr.10917 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f7df8014810 con 0x7f7df406af20 2024-04-06T13:47:03.073 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule removed for path snap_test_dir1 2024-04-06T13:47:03.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.074+0000 7f7e0e17a700 1 -- 172.21.15.189:0/2041215041 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7df406af20 msgr2=0x7f7df406d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:03.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.074+0000 7f7e0e17a700 1 --2- 172.21.15.189:0/2041215041 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7df406af20 0x7f7df406d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e080af070 tx=0x7f7df8008040 comp rx=0 tx=0).stop 2024-04-06T13:47:03.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.074+0000 7f7e0e17a700 1 -- 172.21.15.189:0/2041215041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7df0005210 msgr2=0x7f7e0815ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:03.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.074+0000 7f7e0e17a700 1 --2- 172.21.15.189:0/2041215041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7df0005210 0x7f7e0815ad90 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00f8c0 tx=0x7f7dfc00fc80 comp rx=0 tx=0).stop 2024-04-06T13:47:03.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.074+0000 7f7e0e17a700 1 -- 172.21.15.189:0/2041215041 shutdown_connections 2024-04-06T13:47:03.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.074+0000 7f7e0e17a700 1 --2- 172.21.15.189:0/2041215041 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7df406af20 0x7f7df406d3e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:03.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.074+0000 7f7e0e17a700 1 --2- 172.21.15.189:0/2041215041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e0815b8d0 0x7f7e0815bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:03.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.074+0000 7f7e0e17a700 1 --2- 172.21.15.189:0/2041215041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7df0005210 0x7f7e0815ad90 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:03.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.074+0000 7f7e0e17a700 1 --2- 172.21.15.189:0/2041215041 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e08159860 0x7f7e080ae550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:03.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.074+0000 7f7e0e17a700 1 -- 172.21.15.189:0/2041215041 >> 172.21.15.189:0/2041215041 conn(0x7f7e080cc290 msgr2=0x7f7e0815fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:03.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.074+0000 7f7e0e17a700 1 -- 172.21.15.189:0/2041215041 shutdown_connections 2024-04-06T13:47:03.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.074+0000 7f7e0e17a700 1 -- 172.21.15.189:0/2041215041 wait complete. 2024-04-06T13:47:03.087 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-04-06T13:47:03.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:03.088 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:47:03.105 INFO:teuthology.orchestra.run.smithi189.stdout:scheduled-2024-04-06-13_47_00_UTC 2024-04-06T13:47:03.105 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-06-13_47_00_UTC 2024-04-06T13:47:03.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:03.106 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-06-13_47_00_UTC) 2024-04-06T13:47:03.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:03.168 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-06T13:47:03.227 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-06T13:47:03.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.389+0000 7f2e26f7c700 1 Processor -- start 2024-04-06T13:47:03.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.389+0000 7f2e26f7c700 1 -- start start 2024-04-06T13:47:03.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.389+0000 7f2e26f7c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2e2015d7b0 0x7f2e200b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:03.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.389+0000 7f2e26f7c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2e2015e500 0x7f2e2006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:03.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.390+0000 7f2e26f7c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e200b3990 con 0x7f2e2006a730 2024-04-06T13:47:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.390+0000 7f2e26f7c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e2015e950 con 0x7f2e2015e500 2024-04-06T13:47:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.390+0000 7f2e26f7c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e2006b0f0 con 0x7f2e2015d7b0 2024-04-06T13:47:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.390+0000 7f2e25519700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2e2015e500 0x7f2e2006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.390+0000 7f2e25519700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2e2015e500 0x7f2e2006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46966/0 (socket says 172.21.15.189:46966) 2024-04-06T13:47:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.390+0000 7f2e25519700 1 -- 172.21.15.189:0/3696459451 learned_addr learned my addr 172.21.15.189:0/3696459451 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.390+0000 7f2e24d18700 1 --2- 172.21.15.189:0/3696459451 >> v2:172.21.15.192:3300/0 conn(0x7f2e2015d7b0 0x7f2e200b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.390+0000 7f2e25519700 1 -- 172.21.15.189:0/3696459451 >> v2:172.21.15.192:3300/0 conn(0x7f2e2015d7b0 msgr2=0x7f2e200b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.390+0000 7f2e25519700 1 --2- 172.21.15.189:0/3696459451 >> v2:172.21.15.192:3300/0 conn(0x7f2e2015d7b0 0x7f2e200b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.390+0000 7f2e25519700 1 -- 172.21.15.189:0/3696459451 >> v1:172.21.15.192:6789/0 conn(0x7f2e2006a730 legacy=0x7f2e2015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.390+0000 7f2e25519700 1 -- 172.21.15.189:0/3696459451 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e200b3060 con 0x7f2e2015e500 2024-04-06T13:47:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.390+0000 7f2e24d18700 1 --2- 172.21.15.189:0/3696459451 >> v2:172.21.15.192:3300/0 conn(0x7f2e2015d7b0 0x7f2e200b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.390+0000 7f2e25519700 1 --2- 172.21.15.189:0/3696459451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e2015e500 0x7f2e2006abb0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f2e200b6200 tx=0x7f2e1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e20743a50e41dc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.390+0000 7f2e1f7fe700 1 -- 172.21.15.189:0/3696459451 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1800e070 con 0x7f2e2015e500 2024-04-06T13:47:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.390+0000 7f2e1f7fe700 1 -- 172.21.15.189:0/3696459451 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e18003a20 con 0x7f2e2015e500 2024-04-06T13:47:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.390+0000 7f2e1f7fe700 1 -- 172.21.15.189:0/3696459451 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18003dd0 con 0x7f2e2015e500 2024-04-06T13:47:03.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.391+0000 7f2e26f7c700 1 -- 172.21.15.189:0/3696459451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e2015e500 msgr2=0x7f2e2006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:03.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.391+0000 7f2e26f7c700 1 --2- 172.21.15.189:0/3696459451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e2015e500 0x7f2e2006abb0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f2e200b6200 tx=0x7f2e1800b330 comp rx=0 tx=0).stop 2024-04-06T13:47:03.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.391+0000 7f2e26f7c700 1 -- 172.21.15.189:0/3696459451 shutdown_connections 2024-04-06T13:47:03.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.391+0000 7f2e26f7c700 1 --2- 172.21.15.189:0/3696459451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e2015e500 0x7f2e2006abb0 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:03.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.391+0000 7f2e26f7c700 1 --2- 172.21.15.189:0/3696459451 >> v2:172.21.15.192:3300/0 conn(0x7f2e2015d7b0 0x7f2e200b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:03.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.391+0000 7f2e26f7c700 1 -- 172.21.15.189:0/3696459451 >> 172.21.15.189:0/3696459451 conn(0x7f2e200d00b0 msgr2=0x7f2e200d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:03.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.391+0000 7f2e26f7c700 1 -- 172.21.15.189:0/3696459451 shutdown_connections 2024-04-06T13:47:03.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.391+0000 7f2e26f7c700 1 -- 172.21.15.189:0/3696459451 wait complete. 2024-04-06T13:47:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.392+0000 7f2e26f7c700 1 Processor -- start 2024-04-06T13:47:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.392+0000 7f2e26f7c700 1 -- start start 2024-04-06T13:47:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.393+0000 7f2e26f7c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e2006a730 0x7f2e20150090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.393+0000 7f2e26f7c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e2015d7b0 0x7f2e2015ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.393+0000 7f2e26f7c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e2015e500 0x7f2e200ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.393+0000 7f2e26f7c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2015c4f0 con 0x7f2e2015e500 2024-04-06T13:47:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.393+0000 7f2e26f7c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2015b8e0 con 0x7f2e2006a730 2024-04-06T13:47:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.393+0000 7f2e26f7c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2015bbc0 con 0x7f2e2015d7b0 2024-04-06T13:47:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.393+0000 7f2e25519700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e2015e500 0x7f2e200ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.393+0000 7f2e25519700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e2015e500 0x7f2e200ccf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46970/0 (socket says 172.21.15.189:46970) 2024-04-06T13:47:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.393+0000 7f2e25519700 1 -- 172.21.15.189:0/926229789 learned_addr learned my addr 172.21.15.189:0/926229789 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.393+0000 7f2e24d18700 1 --2- 172.21.15.189:0/926229789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e2006a730 0x7f2e20150090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.393+0000 7f2e1ffff700 1 --2- 172.21.15.189:0/926229789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e2015d7b0 0x7f2e2015ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.393+0000 7f2e25519700 1 -- 172.21.15.189:0/926229789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e2015d7b0 msgr2=0x7f2e2015ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:03.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.393+0000 7f2e25519700 1 --2- 172.21.15.189:0/926229789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e2015d7b0 0x7f2e2015ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:03.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.393+0000 7f2e25519700 1 -- 172.21.15.189:0/926229789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e2006a730 msgr2=0x7f2e20150090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:03.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.393+0000 7f2e25519700 1 --2- 172.21.15.189:0/926229789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e2006a730 0x7f2e20150090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:03.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.393+0000 7f2e25519700 1 -- 172.21.15.189:0/926229789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e18009b30 con 0x7f2e2015e500 2024-04-06T13:47:03.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.394+0000 7f2e25519700 1 --2- 172.21.15.189:0/926229789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e2015e500 0x7f2e200ccf50 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f2e18015040 tx=0x7f2e18004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:03.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.394+0000 7f2e1d7fa700 1 -- 172.21.15.189:0/926229789 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18004440 con 0x7f2e2015e500 2024-04-06T13:47:03.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.394+0000 7f2e1d7fa700 1 -- 172.21.15.189:0/926229789 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e180045a0 con 0x7f2e2015e500 2024-04-06T13:47:03.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.394+0000 7f2e26f7c700 1 -- 172.21.15.189:0/926229789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e2015bea0 con 0x7f2e2015e500 2024-04-06T13:47:03.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.394+0000 7f2e1d7fa700 1 -- 172.21.15.189:0/926229789 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18004950 con 0x7f2e2015e500 2024-04-06T13:47:03.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.394+0000 7f2e26f7c700 1 -- 172.21.15.189:0/926229789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e201505d0 con 0x7f2e2015e500 2024-04-06T13:47:03.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.395+0000 7f2e26f7c700 1 -- 172.21.15.189:0/926229789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e20067bb0 con 0x7f2e2015e500 2024-04-06T13:47:03.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.395+0000 7f2e1d7fa700 1 -- 172.21.15.189:0/926229789 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 134) v1 ==== 89378+0+0 (secure 0 0 0) 0x7f2e18012430 con 0x7f2e2015e500 2024-04-06T13:47:03.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.395+0000 7f2e1d7fa700 1 --2- 172.21.15.189:0/926229789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2e0406af20 0x7f2e0406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:03.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.396+0000 7f2e1d7fa700 1 -- 172.21.15.189:0/926229789 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2e1808b190 con 0x7f2e2015e500 2024-04-06T13:47:03.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.396+0000 7f2e24d18700 1 --2- 172.21.15.189:0/926229789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2e0406af20 0x7f2e0406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:03.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.396+0000 7f2e24d18700 1 --2- 172.21.15.189:0/926229789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2e0406af20 0x7f2e0406d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2e14006fd0 tx=0x7f2e14008040 comp rx=0 tx=0).ready entity=mgr.10917 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:03.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.400+0000 7f2e1d7fa700 1 -- 172.21.15.189:0/926229789 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2e18055bd0 con 0x7f2e2015e500 2024-04-06T13:47:03.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:03.652+0000 7f2e26f7c700 1 -- 172.21.15.189:0/926229789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f2e20074990 con 0x7f2e2015e500 2024-04-06T13:47:04.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.590+0000 7f2e1d7fa700 1 -- 172.21.15.189:0/926229789 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 135) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f2e18055fa0 con 0x7f2e2015e500 2024-04-06T13:47:04.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.592+0000 7f2e1d7fa700 1 -- 172.21.15.189:0/926229789 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v135) v1 ==== 93+0+0 (secure 0 0 0) 0x7f2e1806d6d0 con 0x7f2e2015e500 2024-04-06T13:47:04.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.596+0000 7f2e26f7c700 1 -- 172.21.15.189:0/926229789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2e0406af20 msgr2=0x7f2e0406d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:04.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.596+0000 7f2e26f7c700 1 --2- 172.21.15.189:0/926229789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2e0406af20 0x7f2e0406d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2e14006fd0 tx=0x7f2e14008040 comp rx=0 tx=0).stop 2024-04-06T13:47:04.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.596+0000 7f2e26f7c700 1 -- 172.21.15.189:0/926229789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e2015e500 msgr2=0x7f2e200ccf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:04.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.596+0000 7f2e26f7c700 1 --2- 172.21.15.189:0/926229789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e2015e500 0x7f2e200ccf50 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f2e18015040 tx=0x7f2e18004ad0 comp rx=0 tx=0).stop 2024-04-06T13:47:04.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.596+0000 7f2e26f7c700 1 -- 172.21.15.189:0/926229789 shutdown_connections 2024-04-06T13:47:04.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.596+0000 7f2e26f7c700 1 --2- 172.21.15.189:0/926229789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2e0406af20 0x7f2e0406d3e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:04.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.596+0000 7f2e26f7c700 1 --2- 172.21.15.189:0/926229789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e2015e500 0x7f2e200ccf50 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:04.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.596+0000 7f2e26f7c700 1 --2- 172.21.15.189:0/926229789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e2006a730 0x7f2e20150090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:04.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.596+0000 7f2e26f7c700 1 --2- 172.21.15.189:0/926229789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e2015d7b0 0x7f2e2015ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:04.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.596+0000 7f2e26f7c700 1 -- 172.21.15.189:0/926229789 >> 172.21.15.189:0/926229789 conn(0x7f2e200d00b0 msgr2=0x7f2e2006eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:04.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.596+0000 7f2e26f7c700 1 -- 172.21.15.189:0/926229789 shutdown_connections 2024-04-06T13:47:04.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.596+0000 7f2e26f7c700 1 -- 172.21.15.189:0/926229789 wait complete. 2024-04-06T13:47:04.613 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T13:47:04.613 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:47:04.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:47:04.703 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T13:47:04.704 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:47:04.704 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:47:04.722 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:47:04.722 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:47:04.781 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:47:04.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.945+0000 7f232b0c3700 1 Processor -- start 2024-04-06T13:47:04.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.945+0000 7f232b0c3700 1 -- start start 2024-04-06T13:47:04.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.946+0000 7f232b0c3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f23240b2cd0 0x7f232406c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:04.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.946+0000 7f232b0c3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f23240b2680 0x7f232406ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:04.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.946+0000 7f232b0c3700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232415d8d0 con 0x7f232415d040 2024-04-06T13:47:04.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.946+0000 7f232b0c3700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23240cba40 con 0x7f23240b2cd0 2024-04-06T13:47:04.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.946+0000 7f232b0c3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f232415d580 con 0x7f23240b2680 2024-04-06T13:47:04.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.946+0000 7f2323fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f23240b2cd0 0x7f232406c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:04.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.946+0000 7f2323fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f23240b2cd0 0x7f232406c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46996/0 (socket says 172.21.15.189:46996) 2024-04-06T13:47:04.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.946+0000 7f2323fff700 1 -- 172.21.15.189:0/2058373564 learned_addr learned my addr 172.21.15.189:0/2058373564 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:04.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.946+0000 7f2329660700 1 --2- 172.21.15.189:0/2058373564 >> v2:172.21.15.192:3300/0 conn(0x7f23240b2680 0x7f232406ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:04.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.946+0000 7f2323fff700 1 -- 172.21.15.189:0/2058373564 >> v2:172.21.15.192:3300/0 conn(0x7f23240b2680 msgr2=0x7f232406ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:04.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.946+0000 7f2323fff700 1 --2- 172.21.15.189:0/2058373564 >> v2:172.21.15.192:3300/0 conn(0x7f23240b2680 0x7f232406ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:04.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.946+0000 7f2323fff700 1 -- 172.21.15.189:0/2058373564 >> v1:172.21.15.192:6789/0 conn(0x7f232415d040 legacy=0x7f232415dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:04.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.946+0000 7f2323fff700 1 -- 172.21.15.189:0/2058373564 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232415e470 con 0x7f23240b2cd0 2024-04-06T13:47:04.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.947+0000 7f2323fff700 1 --2- 172.21.15.189:0/2058373564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23240b2cd0 0x7f232406c4e0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f23240cef50 tx=0x7f2310009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e1ed6737ca87f07 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:04.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.947+0000 7f23237fe700 1 -- 172.21.15.189:0/2058373564 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2310011830 con 0x7f23240b2cd0 2024-04-06T13:47:04.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.947+0000 7f23237fe700 1 -- 172.21.15.189:0/2058373564 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2310011990 con 0x7f23240b2cd0 2024-04-06T13:47:04.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.947+0000 7f23237fe700 1 -- 172.21.15.189:0/2058373564 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2310011d20 con 0x7f23240b2cd0 2024-04-06T13:47:04.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.947+0000 7f232b0c3700 1 -- 172.21.15.189:0/2058373564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23240b2cd0 msgr2=0x7f232406c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:04.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.947+0000 7f232b0c3700 1 --2- 172.21.15.189:0/2058373564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23240b2cd0 0x7f232406c4e0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f23240cef50 tx=0x7f2310009bd0 comp rx=0 tx=0).stop 2024-04-06T13:47:04.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.947+0000 7f232b0c3700 1 -- 172.21.15.189:0/2058373564 shutdown_connections 2024-04-06T13:47:04.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.947+0000 7f232b0c3700 1 --2- 172.21.15.189:0/2058373564 >> v2:172.21.15.192:3300/0 conn(0x7f23240b2680 0x7f232406ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:04.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.947+0000 7f232b0c3700 1 --2- 172.21.15.189:0/2058373564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23240b2cd0 0x7f232406c4e0 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:04.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.947+0000 7f232b0c3700 1 -- 172.21.15.189:0/2058373564 >> 172.21.15.189:0/2058373564 conn(0x7f232406a730 msgr2=0x7f232406ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:04.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.948+0000 7f232b0c3700 1 -- 172.21.15.189:0/2058373564 shutdown_connections 2024-04-06T13:47:04.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.948+0000 7f232b0c3700 1 -- 172.21.15.189:0/2058373564 wait complete. 2024-04-06T13:47:04.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.949+0000 7f232b0c3700 1 Processor -- start 2024-04-06T13:47:04.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.949+0000 7f232b0c3700 1 -- start start 2024-04-06T13:47:04.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f232b0c3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23240b2680 0x7f232415b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:04.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f232b0c3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23240b2cd0 0x7f23240af720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:04.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f232b0c3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f232415d040 0x7f23240afc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:04.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f232b0c3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23241534a0 con 0x7f23240b2cd0 2024-04-06T13:47:04.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f232b0c3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2324148e10 con 0x7f23240b2680 2024-04-06T13:47:04.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f232b0c3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23241490f0 con 0x7f232415d040 2024-04-06T13:47:04.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f2323fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23240b2cd0 0x7f23240af720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:04.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f2323fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23240b2cd0 0x7f23240af720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47002/0 (socket says 172.21.15.189:47002) 2024-04-06T13:47:04.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f2323fff700 1 -- 172.21.15.189:0/3248602030 learned_addr learned my addr 172.21.15.189:0/3248602030 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:04.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f2328e5f700 1 --2- 172.21.15.189:0/3248602030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23240b2680 0x7f232415b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:04.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f2329660700 1 --2- 172.21.15.189:0/3248602030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f232415d040 0x7f23240afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:04.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f2323fff700 1 -- 172.21.15.189:0/3248602030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f232415d040 msgr2=0x7f23240afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:04.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f2323fff700 1 --2- 172.21.15.189:0/3248602030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f232415d040 0x7f23240afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:04.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f2323fff700 1 -- 172.21.15.189:0/3248602030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23240b2680 msgr2=0x7f232415b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:04.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f2323fff700 1 --2- 172.21.15.189:0/3248602030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23240b2680 0x7f232415b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:04.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f2323fff700 1 -- 172.21.15.189:0/3248602030 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2310009550 con 0x7f23240b2cd0 2024-04-06T13:47:04.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f2323fff700 1 --2- 172.21.15.189:0/3248602030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23240b2cd0 0x7f23240af720 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f2310000c00 tx=0x7f2310004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:04.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f23217fa700 1 -- 172.21.15.189:0/3248602030 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2310004380 con 0x7f23240b2cd0 2024-04-06T13:47:04.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f23217fa700 1 -- 172.21.15.189:0/3248602030 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231001f070 con 0x7f23240b2cd0 2024-04-06T13:47:04.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f232b0c3700 1 -- 172.21.15.189:0/3248602030 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23241493d0 con 0x7f23240b2cd0 2024-04-06T13:47:04.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f23217fa700 1 -- 172.21.15.189:0/3248602030 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23100036a0 con 0x7f23240b2cd0 2024-04-06T13:47:04.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.950+0000 7f232b0c3700 1 -- 172.21.15.189:0/3248602030 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23241560a0 con 0x7f23240b2cd0 2024-04-06T13:47:04.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.951+0000 7f232b0c3700 1 -- 172.21.15.189:0/3248602030 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2324073270 con 0x7f23240b2cd0 2024-04-06T13:47:04.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.952+0000 7f23217fa700 1 -- 172.21.15.189:0/3248602030 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 135) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f2310021030 con 0x7f23240b2cd0 2024-04-06T13:47:04.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.952+0000 7f23217fa700 1 --2- 172.21.15.189:0/3248602030 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f230806aed0 0x7f230806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:04.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.952+0000 7f2328e5f700 1 -- 172.21.15.189:0/3248602030 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f230806aed0 msgr2=0x7f230806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:04.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.952+0000 7f23217fa700 1 -- 172.21.15.189:0/3248602030 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f231008ae00 con 0x7f23240b2cd0 2024-04-06T13:47:04.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.952+0000 7f2328e5f700 1 --2- 172.21.15.189:0/3248602030 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f230806aed0 0x7f230806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:04.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:04.957+0000 7f23217fa700 1 -- 172.21.15.189:0/3248602030 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2310055950 con 0x7f23240b2cd0 2024-04-06T13:47:04.978 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:04.978+0000 7fce45124200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:47:04.998 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:04.997+0000 7fbb7ac74200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:47:05.055 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:05.055+0000 7fce45124200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:47:05.077 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:05.077+0000 7fbb7ac74200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:47:05.127 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:05.127+0000 7fce45124200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:47:05.153 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:05.152+0000 7fbb7ac74200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:47:05.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.153+0000 7f2328e5f700 1 -- 172.21.15.189:0/3248602030 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f230806aed0 msgr2=0x7f230806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:05.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.153+0000 7f2328e5f700 1 --2- 172.21.15.189:0/3248602030 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f230806aed0 0x7f230806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:05.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.160+0000 7f232b0c3700 1 -- 172.21.15.189:0/3248602030 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2324074990 con 0x7f23240b2cd0 2024-04-06T13:47:05.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.161+0000 7f23217fa700 1 -- 172.21.15.189:0/3248602030 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v212) v1 ==== 74+0+17117 (secure 0 0 0) 0x7f2310058f70 con 0x7f23240b2cd0 2024-04-06T13:47:05.162 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:05.162 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":212,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:46:55.602408+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:45:37.837008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-04-06T13:45:39.030714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.22","mappings":[{"from":1,"to":0}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]},{"pgid":"26.2c","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/509530111":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/3787698124":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/2802982514":"2024-04-07T13:45:54.091329+0000","172.21.15.189:6833/111704":"2024-04-07T13:45:54.091329+0000","172.21.15.189:6832/111704":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/606015831":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/4174593093":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/3225575705":"2024-04-07T13:45:54.091329+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:47:05.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.163+0000 7f232b0c3700 1 -- 172.21.15.189:0/3248602030 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f230806aed0 msgr2=0x7f230806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:05.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.163+0000 7f232b0c3700 1 --2- 172.21.15.189:0/3248602030 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f230806aed0 0x7f230806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.163+0000 7f232b0c3700 1 -- 172.21.15.189:0/3248602030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23240b2cd0 msgr2=0x7f23240af720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:05.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.163+0000 7f232b0c3700 1 --2- 172.21.15.189:0/3248602030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23240b2cd0 0x7f23240af720 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f2310000c00 tx=0x7f2310004750 comp rx=0 tx=0).stop 2024-04-06T13:47:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.163+0000 7f232b0c3700 1 -- 172.21.15.189:0/3248602030 shutdown_connections 2024-04-06T13:47:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.163+0000 7f232b0c3700 1 --2- 172.21.15.189:0/3248602030 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f230806aed0 0x7f230806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.163+0000 7f232b0c3700 1 --2- 172.21.15.189:0/3248602030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f232415d040 0x7f23240afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.163+0000 7f232b0c3700 1 --2- 172.21.15.189:0/3248602030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23240b2680 0x7f232415b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.163+0000 7f232b0c3700 1 --2- 172.21.15.189:0/3248602030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23240b2cd0 0x7f23240af720 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.163+0000 7f232b0c3700 1 -- 172.21.15.189:0/3248602030 >> 172.21.15.189:0/3248602030 conn(0x7f232406a730 msgr2=0x7f2324071bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.163+0000 7f232b0c3700 1 -- 172.21.15.189:0/3248602030 shutdown_connections 2024-04-06T13:47:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.163+0000 7f232b0c3700 1 -- 172.21.15.189:0/3248602030 wait complete. 2024-04-06T13:47:05.177 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T13:45:37.837008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '208', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T13:45:39.030714+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '209', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:47:05.178 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:47:05.194 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:05.194+0000 7fce45124200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:47:05.229 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:05.228+0000 7fbb7ac74200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:47:05.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.306+0000 7fb866658700 1 Processor -- start 2024-04-06T13:47:05.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.306+0000 7fb866658700 1 -- start start 2024-04-06T13:47:05.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.307+0000 7fb866658700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb86015d7e0 0x7fb8600d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:05.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.307+0000 7fb866658700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb8600d1cc0 0x7fb8600b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:05.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.307+0000 7fb866658700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8600cdf80 con 0x7fb8600d0890 2024-04-06T13:47:05.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.307+0000 7fb866658700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8600d0d10 con 0x7fb8600d1cc0 2024-04-06T13:47:05.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.307+0000 7fb866658700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8600d3230 con 0x7fb86015d7e0 2024-04-06T13:47:05.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.307+0000 7fb85f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb8600d0890 0x7fb8600d15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42660/0 (socket says 172.21.15.189:42660) 2024-04-06T13:47:05.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.307+0000 7fb85f7fe700 1 -- 172.21.15.189:0/2165561626 learned_addr learned my addr 172.21.15.189:0/2165561626 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:05.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.307+0000 7fb85ffff700 1 --2- 172.21.15.189:0/2165561626 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb86015d7e0 0x7fb8600d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:05.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.307+0000 7fb864bf5700 1 --2- 172.21.15.189:0/2165561626 >> v2:172.21.15.192:3300/0 conn(0x7fb8600d1cc0 0x7fb8600b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:05.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.308+0000 7fb85effd700 1 -- 172.21.15.189:0/2165561626 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927953015 0 0) 0x7fb8600cdf80 con 0x7fb8600d0890 2024-04-06T13:47:05.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.308+0000 7fb85effd700 1 -- 172.21.15.189:0/2165561626 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8480035b0 con 0x7fb8600d0890 2024-04-06T13:47:05.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.308+0000 7fb85ffff700 1 -- 172.21.15.189:0/2165561626 >> v2:172.21.15.192:3300/0 conn(0x7fb8600d1cc0 msgr2=0x7fb8600b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:05.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.308+0000 7fb85ffff700 1 --2- 172.21.15.189:0/2165561626 >> v2:172.21.15.192:3300/0 conn(0x7fb8600d1cc0 0x7fb8600b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.308+0000 7fb85effd700 1 -- 172.21.15.189:0/2165561626 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3557982132 0 0) 0x7fb8480035b0 con 0x7fb8600d0890 2024-04-06T13:47:05.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.308+0000 7fb85ffff700 1 -- 172.21.15.189:0/2165561626 >> v1:172.21.15.189:6789/0 conn(0x7fb8600d0890 legacy=0x7fb8600d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:05.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.308+0000 7fb85ffff700 1 -- 172.21.15.189:0/2165561626 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8600d2da0 con 0x7fb86015d7e0 2024-04-06T13:47:05.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.308+0000 7fb864bf5700 1 --2- 172.21.15.189:0/2165561626 >> v2:172.21.15.192:3300/0 conn(0x7fb8600d1cc0 0x7fb8600b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:05.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.308+0000 7fb85ffff700 1 --2- 172.21.15.189:0/2165561626 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb86015d7e0 0x7fb8600d0350 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb854009840 tx=0x7fb854009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=878f10f5e50b1fb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:05.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.308+0000 7fb85effd700 1 -- 172.21.15.189:0/2165561626 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb850001670 con 0x7fb86015d7e0 2024-04-06T13:47:05.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.308+0000 7fb85effd700 1 -- 172.21.15.189:0/2165561626 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb854004030 con 0x7fb86015d7e0 2024-04-06T13:47:05.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.308+0000 7fb85effd700 1 -- 172.21.15.189:0/2165561626 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8540043c0 con 0x7fb86015d7e0 2024-04-06T13:47:05.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.309+0000 7fb866658700 1 -- 172.21.15.189:0/2165561626 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb86015d7e0 msgr2=0x7fb8600d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:05.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.309+0000 7fb866658700 1 --2- 172.21.15.189:0/2165561626 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb86015d7e0 0x7fb8600d0350 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb854009840 tx=0x7fb854009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:05.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.309+0000 7fb866658700 1 -- 172.21.15.189:0/2165561626 shutdown_connections 2024-04-06T13:47:05.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.309+0000 7fb866658700 1 --2- 172.21.15.189:0/2165561626 >> v2:172.21.15.192:3300/0 conn(0x7fb8600d1cc0 0x7fb8600b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.309+0000 7fb866658700 1 --2- 172.21.15.189:0/2165561626 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb86015d7e0 0x7fb8600d0350 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.309+0000 7fb866658700 1 -- 172.21.15.189:0/2165561626 >> 172.21.15.189:0/2165561626 conn(0x7fb860072250 msgr2=0x7fb860072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:05.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.309+0000 7fb866658700 1 -- 172.21.15.189:0/2165561626 shutdown_connections 2024-04-06T13:47:05.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.309+0000 7fb866658700 1 -- 172.21.15.189:0/2165561626 wait complete. 2024-04-06T13:47:05.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.310+0000 7fb866658700 1 Processor -- start 2024-04-06T13:47:05.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.310+0000 7fb866658700 1 -- start start 2024-04-06T13:47:05.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.311+0000 7fb866658700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8600d0890 0x7fb86014e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:05.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.311+0000 7fb866658700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8600d1cc0 0x7fb8600c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:05.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.311+0000 7fb866658700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb86015d7e0 0x7fb8600cc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:05.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.311+0000 7fb866658700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8600c7360 con 0x7fb86015d7e0 2024-04-06T13:47:05.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.311+0000 7fb866658700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8600c7640 con 0x7fb8600d0890 2024-04-06T13:47:05.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.311+0000 7fb866658700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8600c7920 con 0x7fb8600d1cc0 2024-04-06T13:47:05.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.311+0000 7fb864bf5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb86015d7e0 0x7fb8600cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:05.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.311+0000 7fb85ffff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8600d0890 0x7fb86014e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:05.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.311+0000 7fb864bf5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb86015d7e0 0x7fb8600cc690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47010/0 (socket says 172.21.15.189:47010) 2024-04-06T13:47:05.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.311+0000 7fb864bf5700 1 -- 172.21.15.189:0/1890030597 learned_addr learned my addr 172.21.15.189:0/1890030597 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:05.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.311+0000 7fb85f7fe700 1 --2- 172.21.15.189:0/1890030597 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8600d1cc0 0x7fb8600c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:05.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.312+0000 7fb864bf5700 1 -- 172.21.15.189:0/1890030597 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8600d1cc0 msgr2=0x7fb8600c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:05.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.312+0000 7fb864bf5700 1 --2- 172.21.15.189:0/1890030597 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8600d1cc0 0x7fb8600c6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.312+0000 7fb864bf5700 1 -- 172.21.15.189:0/1890030597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8600d0890 msgr2=0x7fb86014e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:05.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.312+0000 7fb864bf5700 1 --2- 172.21.15.189:0/1890030597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8600d0890 0x7fb86014e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.312+0000 7fb864bf5700 1 -- 172.21.15.189:0/1890030597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb858009a70 con 0x7fb86015d7e0 2024-04-06T13:47:05.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.312+0000 7fb85ffff700 1 --2- 172.21.15.189:0/1890030597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8600d0890 0x7fb86014e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:05.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.312+0000 7fb864bf5700 1 --2- 172.21.15.189:0/1890030597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb86015d7e0 0x7fb8600cc690 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fb85800efc0 tx=0x7fb85800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:05.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.312+0000 7fb85cff9700 1 -- 172.21.15.189:0/1890030597 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85800ca90 con 0x7fb86015d7e0 2024-04-06T13:47:05.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.312+0000 7fb85cff9700 1 -- 172.21.15.189:0/1890030597 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85800f040 con 0x7fb86015d7e0 2024-04-06T13:47:05.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.312+0000 7fb866658700 1 -- 172.21.15.189:0/1890030597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8540094a0 con 0x7fb86015d7e0 2024-04-06T13:47:05.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.312+0000 7fb85cff9700 1 -- 172.21.15.189:0/1890030597 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85800ce30 con 0x7fb86015d7e0 2024-04-06T13:47:05.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.312+0000 7fb866658700 1 -- 172.21.15.189:0/1890030597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8600c7ee0 con 0x7fb86015d7e0 2024-04-06T13:47:05.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.313+0000 7fb866658700 1 -- 172.21.15.189:0/1890030597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb860074990 con 0x7fb86015d7e0 2024-04-06T13:47:05.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.314+0000 7fb85cff9700 1 -- 172.21.15.189:0/1890030597 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 135) v1 ==== 89361+0+0 (secure 0 0 0) 0x7fb858014480 con 0x7fb86015d7e0 2024-04-06T13:47:05.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.314+0000 7fb85cff9700 1 --2- 172.21.15.189:0/1890030597 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb84c06aed0 0x7fb84c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:05.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.314+0000 7fb85ffff700 1 -- 172.21.15.189:0/1890030597 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb84c06aed0 msgr2=0x7fb84c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:05.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.314+0000 7fb85cff9700 1 -- 172.21.15.189:0/1890030597 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb85808a230 con 0x7fb86015d7e0 2024-04-06T13:47:05.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.314+0000 7fb85ffff700 1 --2- 172.21.15.189:0/1890030597 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb84c06aed0 0x7fb84c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:05.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.318+0000 7fb85cff9700 1 -- 172.21.15.189:0/1890030597 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb858054cb0 con 0x7fb86015d7e0 2024-04-06T13:47:05.371 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:05.371+0000 7fce45124200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:47:05.414 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:05.413+0000 7fbb7ac74200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:47:05.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.515+0000 7fb85ffff700 1 -- 172.21.15.189:0/1890030597 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb84c06aed0 msgr2=0x7fb84c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:05.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.515+0000 7fb85ffff700 1 --2- 172.21.15.189:0/1890030597 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb84c06aed0 0x7fb84c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:05.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.517+0000 7fb866658700 1 -- 172.21.15.189:0/1890030597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb8600c8430 con 0x7fb86015d7e0 2024-04-06T13:47:05.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.518+0000 7fb85cff9700 1 -- 172.21.15.189:0/1890030597 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v212) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb858009cd0 con 0x7fb86015d7e0 2024-04-06T13:47:05.519 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:47:05.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.520+0000 7fb866658700 1 -- 172.21.15.189:0/1890030597 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb84c06aed0 msgr2=0x7fb84c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:05.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.520+0000 7fb866658700 1 --2- 172.21.15.189:0/1890030597 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb84c06aed0 0x7fb84c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.520+0000 7fb866658700 1 -- 172.21.15.189:0/1890030597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb86015d7e0 msgr2=0x7fb8600cc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:05.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.520+0000 7fb866658700 1 --2- 172.21.15.189:0/1890030597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb86015d7e0 0x7fb8600cc690 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fb85800efc0 tx=0x7fb85800c3a0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.520+0000 7fb866658700 1 -- 172.21.15.189:0/1890030597 shutdown_connections 2024-04-06T13:47:05.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.520+0000 7fb866658700 1 --2- 172.21.15.189:0/1890030597 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb84c06aed0 0x7fb84c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.520+0000 7fb866658700 1 --2- 172.21.15.189:0/1890030597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb86015d7e0 0x7fb8600cc690 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.520+0000 7fb866658700 1 --2- 172.21.15.189:0/1890030597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8600d0890 0x7fb86014e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.520+0000 7fb866658700 1 --2- 172.21.15.189:0/1890030597 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8600d1cc0 0x7fb8600c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.520+0000 7fb866658700 1 -- 172.21.15.189:0/1890030597 >> 172.21.15.189:0/1890030597 conn(0x7fb860072250 msgr2=0x7fb8600cdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:05.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.520+0000 7fb866658700 1 -- 172.21.15.189:0/1890030597 shutdown_connections 2024-04-06T13:47:05.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.520+0000 7fb866658700 1 -- 172.21.15.189:0/1890030597 wait complete. 2024-04-06T13:47:05.527 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:05.527+0000 7fce45124200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:47:05.534 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T13:47:05.577 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:05.577+0000 7fbb7ac74200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:47:05.642 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:05.642+0000 7fce45124200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:47:05.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.664+0000 7f18128bc700 1 Processor -- start 2024-04-06T13:47:05.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.664+0000 7f18128bc700 1 -- start start 2024-04-06T13:47:05.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.665+0000 7f18128bc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f180c15d590 0x7f180c0b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:05.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.665+0000 7f18128bc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f180c0d3890 0x7f180c0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:05.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.665+0000 7f18128bc700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f180c0b5cc0 con 0x7f180c0b4ed0 2024-04-06T13:47:05.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.665+0000 7f18128bc700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f180c0b5890 con 0x7f180c0d3890 2024-04-06T13:47:05.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.665+0000 7f18128bc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180c0d1500 con 0x7f180c15d590 2024-04-06T13:47:05.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.665+0000 7f180b7fe700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f180c0b4ed0 0x7f180c0b6580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:59602/0 (socket says 172.21.15.189:59602) 2024-04-06T13:47:05.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.665+0000 7f1810e59700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f180c0d3890 0x7f180c0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:05.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.665+0000 7f180b7fe700 1 -- 172.21.15.189:0/4272359649 learned_addr learned my addr 172.21.15.189:0/4272359649 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:05.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.665+0000 7f180bfff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f180c15d590 0x7f180c0b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:05.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.665+0000 7f1810e59700 1 -- 172.21.15.189:0/4272359649 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f180c15d590 msgr2=0x7f180c0b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:05.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.665+0000 7f1810e59700 1 --2- 172.21.15.189:0/4272359649 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f180c15d590 0x7f180c0b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.665+0000 7f1810e59700 1 -- 172.21.15.189:0/4272359649 >> v1:172.21.15.192:6789/0 conn(0x7f180c0b4ed0 legacy=0x7f180c0b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:05.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.665+0000 7f1810e59700 1 -- 172.21.15.189:0/4272359649 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180c0d17e0 con 0x7f180c0d3890 2024-04-06T13:47:05.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.665+0000 7f180affd700 1 -- 172.21.15.189:0/4272359649 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290528424 0 0) 0x7f180c0b5cc0 con 0x7f180c0b4ed0 2024-04-06T13:47:05.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.666+0000 7f1810e59700 1 --2- 172.21.15.189:0/4272359649 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f180c0d3890 0x7f180c0b5350 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f1804009dc0 tx=0x7f180400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3edffffc4cbef848 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:05.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.666+0000 7f180affd700 1 -- 172.21.15.189:0/4272359649 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180400e070 con 0x7f180c0d3890 2024-04-06T13:47:05.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.666+0000 7f180affd700 1 -- 172.21.15.189:0/4272359649 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180400bae0 con 0x7f180c0d3890 2024-04-06T13:47:05.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.666+0000 7f180affd700 1 -- 172.21.15.189:0/4272359649 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180400be90 con 0x7f180c0d3890 2024-04-06T13:47:05.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.666+0000 7f18128bc700 1 -- 172.21.15.189:0/4272359649 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f180c0d3890 msgr2=0x7f180c0b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:05.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.666+0000 7f18128bc700 1 --2- 172.21.15.189:0/4272359649 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f180c0d3890 0x7f180c0b5350 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f1804009dc0 tx=0x7f180400b330 comp rx=0 tx=0).stop 2024-04-06T13:47:05.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.666+0000 7f18128bc700 1 -- 172.21.15.189:0/4272359649 shutdown_connections 2024-04-06T13:47:05.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.666+0000 7f18128bc700 1 --2- 172.21.15.189:0/4272359649 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f180c0d3890 0x7f180c0b5350 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.666+0000 7f18128bc700 1 --2- 172.21.15.189:0/4272359649 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f180c15d590 0x7f180c0b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.666+0000 7f18128bc700 1 -- 172.21.15.189:0/4272359649 >> 172.21.15.189:0/4272359649 conn(0x7f180c06bbc0 msgr2=0x7f180c06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:05.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.666+0000 7f18128bc700 1 -- 172.21.15.189:0/4272359649 shutdown_connections 2024-04-06T13:47:05.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.666+0000 7f18128bc700 1 -- 172.21.15.189:0/4272359649 wait complete. 2024-04-06T13:47:05.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.668+0000 7f18128bc700 1 Processor -- start 2024-04-06T13:47:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.668+0000 7f18128bc700 1 -- start start 2024-04-06T13:47:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.669+0000 7f18128bc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f180c0b4ed0 0x7f180c0cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.669+0000 7f18128bc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f180c0d3890 0x7f180c14aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.669+0000 7f18128bc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f180c15d590 0x7f180c15dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.669+0000 7f18128bc700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180c0cbb40 con 0x7f180c0b4ed0 2024-04-06T13:47:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.669+0000 7f18128bc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180c0caf30 con 0x7f180c15d590 2024-04-06T13:47:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.669+0000 7f18128bc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180c0cb210 con 0x7f180c0d3890 2024-04-06T13:47:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.669+0000 7f180bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f180c0b4ed0 0x7f180c0cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.669+0000 7f180b7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f180c0d3890 0x7f180c14aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.669+0000 7f180bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f180c0b4ed0 0x7f180c0cccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47022/0 (socket says 172.21.15.189:47022) 2024-04-06T13:47:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.669+0000 7f180bfff700 1 -- 172.21.15.189:0/1822973325 learned_addr learned my addr 172.21.15.189:0/1822973325 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.669+0000 7f1810e59700 1 --2- 172.21.15.189:0/1822973325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f180c15d590 0x7f180c15dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.669+0000 7f180bfff700 1 -- 172.21.15.189:0/1822973325 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f180c0d3890 msgr2=0x7f180c14aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.669+0000 7f180bfff700 1 --2- 172.21.15.189:0/1822973325 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f180c0d3890 0x7f180c14aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.669+0000 7f180bfff700 1 -- 172.21.15.189:0/1822973325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f180c15d590 msgr2=0x7f180c15dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.669+0000 7f180bfff700 1 --2- 172.21.15.189:0/1822973325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f180c15d590 0x7f180c15dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.669+0000 7f180bfff700 1 -- 172.21.15.189:0/1822973325 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1804009a70 con 0x7f180c0b4ed0 2024-04-06T13:47:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.669+0000 7f180b7fe700 1 --2- 172.21.15.189:0/1822973325 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f180c0d3890 0x7f180c14aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.669+0000 7f1810e59700 1 --2- 172.21.15.189:0/1822973325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f180c15d590 0x7f180c15dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.670+0000 7f180bfff700 1 --2- 172.21.15.189:0/1822973325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f180c0b4ed0 0x7f180c0cccf0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7f17fc00e870 tx=0x7f17fc00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.670+0000 7f1808ff9700 1 -- 172.21.15.189:0/1822973325 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17fc009590 con 0x7f180c0b4ed0 2024-04-06T13:47:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.670+0000 7f1808ff9700 1 -- 172.21.15.189:0/1822973325 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17fc013680 con 0x7f180c0b4ed0 2024-04-06T13:47:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.670+0000 7f18128bc700 1 -- 172.21.15.189:0/1822973325 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f180c0cb4f0 con 0x7f180c0b4ed0 2024-04-06T13:47:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.670+0000 7f1808ff9700 1 -- 172.21.15.189:0/1822973325 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17fc0139f0 con 0x7f180c0b4ed0 2024-04-06T13:47:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.670+0000 7f18128bc700 1 -- 172.21.15.189:0/1822973325 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f180c0cd230 con 0x7f180c0b4ed0 2024-04-06T13:47:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.671+0000 7f18128bc700 1 -- 172.21.15.189:0/1822973325 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f180c074990 con 0x7f180c0b4ed0 2024-04-06T13:47:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.671+0000 7f1808ff9700 1 -- 172.21.15.189:0/1822973325 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 135) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f17fc013b50 con 0x7f180c0b4ed0 2024-04-06T13:47:05.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.672+0000 7f1808ff9700 1 --2- 172.21.15.189:0/1822973325 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f17f806af00 0x7f17f806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:05.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.672+0000 7f180b7fe700 1 -- 172.21.15.189:0/1822973325 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f17f806af00 msgr2=0x7f17f806d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:05.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.672+0000 7f1808ff9700 1 -- 172.21.15.189:0/1822973325 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6620+0+0 (secure 0 0 0) 0x7f17fc0899c0 con 0x7f180c0b4ed0 2024-04-06T13:47:05.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.672+0000 7f180b7fe700 1 --2- 172.21.15.189:0/1822973325 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f17f806af00 0x7f17f806d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:05.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.676+0000 7f1808ff9700 1 -- 172.21.15.189:0/1822973325 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f17fc054410 con 0x7f180c0b4ed0 2024-04-06T13:47:05.696 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:05.696+0000 7fbb7ac74200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:47:05.724 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:05.724+0000 7fce45124200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:47:05.780 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:05.780+0000 7fbb7ac74200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:47:05.798 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:05.798+0000 7fce45124200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:47:05.863 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:05.862+0000 7fbb7ac74200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:47:05.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.873+0000 7f180b7fe700 1 -- 172.21.15.189:0/1822973325 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f17f806af00 msgr2=0x7f17f806d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:05.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.873+0000 7f180b7fe700 1 --2- 172.21.15.189:0/1822973325 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f17f806af00 0x7f17f806d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:05.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.887+0000 7f18128bc700 1 -- 172.21.15.189:0/1822973325 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f180c15ca90 con 0x7f180c0b4ed0 2024-04-06T13:47:05.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.887+0000 7f1808ff9700 1 -- 172.21.15.189:0/1822973325 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v213) v1 ==== 104+0+11 (secure 0 0 0) 0x7f17fc057a30 con 0x7f180c0b4ed0 2024-04-06T13:47:05.888 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:47:05.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.890+0000 7f18128bc700 1 -- 172.21.15.189:0/1822973325 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f17f806af00 msgr2=0x7f17f806d3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:05.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.890+0000 7f18128bc700 1 --2- 172.21.15.189:0/1822973325 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f17f806af00 0x7f17f806d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.890+0000 7f18128bc700 1 -- 172.21.15.189:0/1822973325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f180c0b4ed0 msgr2=0x7f180c0cccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:05.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.890+0000 7f18128bc700 1 --2- 172.21.15.189:0/1822973325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f180c0b4ed0 0x7f180c0cccf0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7f17fc00e870 tx=0x7f17fc00eb80 comp rx=0 tx=0).stop 2024-04-06T13:47:05.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.890+0000 7f18128bc700 1 -- 172.21.15.189:0/1822973325 shutdown_connections 2024-04-06T13:47:05.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.890+0000 7f18128bc700 1 --2- 172.21.15.189:0/1822973325 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f17f806af00 0x7f17f806d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.890+0000 7f18128bc700 1 --2- 172.21.15.189:0/1822973325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f180c15d590 0x7f180c15dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.890+0000 7f18128bc700 1 --2- 172.21.15.189:0/1822973325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f180c0b4ed0 0x7f180c0cccf0 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.890+0000 7f18128bc700 1 --2- 172.21.15.189:0/1822973325 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f180c0d3890 0x7f180c14aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:05.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.890+0000 7f18128bc700 1 -- 172.21.15.189:0/1822973325 >> 172.21.15.189:0/1822973325 conn(0x7f180c06bbc0 msgr2=0x7f180c0ce360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:05.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.890+0000 7f18128bc700 1 -- 172.21.15.189:0/1822973325 shutdown_connections 2024-04-06T13:47:05.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:05.890+0000 7f18128bc700 1 -- 172.21.15.189:0/1822973325 wait complete. 2024-04-06T13:47:05.903 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T13:47:06.006 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:06.005+0000 7fce45124200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:47:06.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.042+0000 7fecc6e7f700 1 Processor -- start 2024-04-06T13:47:06.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.042+0000 7fecc6e7f700 1 -- start start 2024-04-06T13:47:06.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.044+0000 7fecc6e7f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fecc0072650 0x7fecc006da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:06.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.044+0000 7fecc6e7f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc00b1fb0 con 0x7fecc006df60 2024-04-06T13:47:06.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.044+0000 7fecc6e7f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc006e360 con 0x7fecc0161a30 2024-04-06T13:47:06.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.044+0000 7fecc6e7f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecc00cba40 con 0x7fecc0072650 2024-04-06T13:47:06.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.044+0000 7fecc4c1b700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fecc0161a30 0x7fecc006eb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:59608/0 (socket says 172.21.15.189:59608) 2024-04-06T13:47:06.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.044+0000 7fecc541c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fecc006df60 0x7fecc016b670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42672/0 (socket says 172.21.15.189:42672) 2024-04-06T13:47:06.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.044+0000 7fecc4c1b700 1 -- 172.21.15.189:0/3203143128 learned_addr learned my addr 172.21.15.189:0/3203143128 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:06.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.044+0000 7fecbffff700 1 --2- 172.21.15.189:0/3203143128 >> v2:172.21.15.189:3300/0 conn(0x7fecc0072650 0x7fecc006da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:06.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.044+0000 7fecbf7fe700 1 -- 172.21.15.189:0/3203143128 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970053636 0 0) 0x7fecc006e360 con 0x7fecc0161a30 2024-04-06T13:47:06.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.044+0000 7fecbf7fe700 1 -- 172.21.15.189:0/3203143128 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb00035b0 con 0x7fecc0161a30 2024-04-06T13:47:06.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.045+0000 7fecbf7fe700 1 -- 172.21.15.189:0/3203143128 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085287444 0 0) 0x7fecc00b1fb0 con 0x7fecc006df60 2024-04-06T13:47:06.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.045+0000 7fecbf7fe700 1 -- 172.21.15.189:0/3203143128 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc006e360 con 0x7fecc006df60 2024-04-06T13:47:06.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.045+0000 7fecbf7fe700 1 -- 172.21.15.189:0/3203143128 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1138701537 0 0) 0x7fecb00035b0 con 0x7fecc0161a30 2024-04-06T13:47:06.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.045+0000 7fecbffff700 1 -- 172.21.15.189:0/3203143128 >> v1:172.21.15.192:6789/0 conn(0x7fecc0161a30 legacy=0x7fecc006eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:06.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.045+0000 7fecbffff700 1 -- 172.21.15.189:0/3203143128 >> v1:172.21.15.189:6789/0 conn(0x7fecc006df60 legacy=0x7fecc016b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:06.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.045+0000 7fecbffff700 1 -- 172.21.15.189:0/3203143128 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc00b2c60 con 0x7fecc0072650 2024-04-06T13:47:06.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.045+0000 7fecbffff700 1 --2- 172.21.15.189:0/3203143128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc0072650 0x7fecc006da20 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fecac009a20 tx=0x7fecac009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a509e76c464bdfb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:06.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.045+0000 7fecbf7fe700 1 -- 172.21.15.189:0/3203143128 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb4001670 con 0x7fecc0072650 2024-04-06T13:47:06.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.045+0000 7fecbf7fe700 1 -- 172.21.15.189:0/3203143128 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecac009ec0 con 0x7fecc0072650 2024-04-06T13:47:06.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.046+0000 7fecc6e7f700 1 -- 172.21.15.189:0/3203143128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc0072650 msgr2=0x7fecc006da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:06.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.046+0000 7fecc6e7f700 1 --2- 172.21.15.189:0/3203143128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc0072650 0x7fecc006da20 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fecac009a20 tx=0x7fecac009d30 comp rx=0 tx=0).stop 2024-04-06T13:47:06.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.046+0000 7fecc6e7f700 1 -- 172.21.15.189:0/3203143128 shutdown_connections 2024-04-06T13:47:06.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.046+0000 7fecc6e7f700 1 --2- 172.21.15.189:0/3203143128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc0072650 0x7fecc006da20 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.046+0000 7fecc6e7f700 1 -- 172.21.15.189:0/3203143128 >> 172.21.15.189:0/3203143128 conn(0x7fecc00cefa0 msgr2=0x7fecc00cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:06.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.046+0000 7fecc6e7f700 1 -- 172.21.15.189:0/3203143128 shutdown_connections 2024-04-06T13:47:06.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.046+0000 7fecc6e7f700 1 -- 172.21.15.189:0/3203143128 wait complete. 2024-04-06T13:47:06.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.048+0000 7fecc6e7f700 1 Processor -- start 2024-04-06T13:47:06.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.049+0000 7fecc6e7f700 1 -- start start 2024-04-06T13:47:06.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.050+0000 7fecc6e7f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc006df60 0x7fecc01472d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:06.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.050+0000 7fecc6e7f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecc0161a30 0x7fecc0156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:06.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.050+0000 7fecc6e7f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecc0156f40 0x7fecc015baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:06.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.050+0000 7fecc6e7f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc01579a0 con 0x7fecc006df60 2024-04-06T13:47:06.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.050+0000 7fecc6e7f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc0159580 con 0x7fecc0161a30 2024-04-06T13:47:06.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.050+0000 7fecc6e7f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc01573e0 con 0x7fecc0156f40 2024-04-06T13:47:06.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.050+0000 7fecc4c1b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc006df60 0x7fecc01472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:06.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.050+0000 7fecbffff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecc0161a30 0x7fecc0156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:06.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.050+0000 7fecc541c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecc0156f40 0x7fecc015baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:06.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.050+0000 7fecc4c1b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc006df60 0x7fecc01472d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47040/0 (socket says 172.21.15.189:47040) 2024-04-06T13:47:06.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.050+0000 7fecbffff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecc0161a30 0x7fecc0156a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:53312/0 (socket says 172.21.15.189:53312) 2024-04-06T13:47:06.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.050+0000 7fecc541c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecc0156f40 0x7fecc015baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:36688/0 (socket says 172.21.15.189:36688) 2024-04-06T13:47:06.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.050+0000 7fecc4c1b700 1 -- 172.21.15.189:0/2533088834 learned_addr learned my addr 172.21.15.189:0/2533088834 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:06.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.051+0000 7fecbffff700 1 -- 172.21.15.189:0/2533088834 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecc0156f40 msgr2=0x7fecc015baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:06.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.051+0000 7fecbffff700 1 --2- 172.21.15.189:0/2533088834 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecc0156f40 0x7fecc015baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.051+0000 7fecbffff700 1 -- 172.21.15.189:0/2533088834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc006df60 msgr2=0x7fecc01472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:06.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.051+0000 7fecbffff700 1 --2- 172.21.15.189:0/2533088834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc006df60 0x7fecc01472d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.051+0000 7fecbffff700 1 -- 172.21.15.189:0/2533088834 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecac0094a0 con 0x7fecc0161a30 2024-04-06T13:47:06.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.051+0000 7fecc541c700 1 --2- 172.21.15.189:0/2533088834 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecc0156f40 0x7fecc015baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:06.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.051+0000 7fecbffff700 1 --2- 172.21.15.189:0/2533088834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecc0161a30 0x7fecc0156a00 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fecac015000 tx=0x7fecac004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:06.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.051+0000 7fecbd7fa700 1 -- 172.21.15.189:0/2533088834 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecac00b5d0 con 0x7fecc0161a30 2024-04-06T13:47:06.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.051+0000 7fecc6e7f700 1 -- 172.21.15.189:0/2533088834 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc0072390 con 0x7fecc0161a30 2024-04-06T13:47:06.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.051+0000 7fecbd7fa700 1 -- 172.21.15.189:0/2533088834 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecac01e070 con 0x7fecc0161a30 2024-04-06T13:47:06.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.051+0000 7fecbd7fa700 1 -- 172.21.15.189:0/2533088834 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecac00b950 con 0x7fecc0161a30 2024-04-06T13:47:06.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.051+0000 7fecc6e7f700 1 -- 172.21.15.189:0/2533088834 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc0159210 con 0x7fecc0161a30 2024-04-06T13:47:06.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.052+0000 7fecc6e7f700 1 -- 172.21.15.189:0/2533088834 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca00052f0 con 0x7fecc0161a30 2024-04-06T13:47:06.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.053+0000 7fecbd7fa700 1 -- 172.21.15.189:0/2533088834 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 135) v1 ==== 89361+0+0 (secure 0 0 0) 0x7fecac00bab0 con 0x7fecc0161a30 2024-04-06T13:47:06.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.054+0000 7fecbd7fa700 1 --2- 172.21.15.189:0/2533088834 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feca406aed0 0x7feca406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:06.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.054+0000 7fecc4c1b700 1 -- 172.21.15.189:0/2533088834 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feca406aed0 msgr2=0x7feca406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:06.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.054+0000 7fecbd7fa700 1 -- 172.21.15.189:0/2533088834 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6620+0+0 (secure 0 0 0) 0x7fecac08a810 con 0x7fecc0161a30 2024-04-06T13:47:06.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.054+0000 7fecc4c1b700 1 --2- 172.21.15.189:0/2533088834 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feca406aed0 0x7feca406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:06.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.058+0000 7fecbd7fa700 1 -- 172.21.15.189:0/2533088834 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fecac0552f0 con 0x7fecc0161a30 2024-04-06T13:47:06.079 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:06.078+0000 7fbb7ac74200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:47:06.082 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:06.082+0000 7fce45124200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:47:06.155 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:06.154+0000 7fbb7ac74200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:47:06.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.255+0000 7fecc4c1b700 1 -- 172.21.15.189:0/2533088834 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feca406aed0 msgr2=0x7feca406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:06.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.255+0000 7fecc4c1b700 1 --2- 172.21.15.189:0/2533088834 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feca406aed0 0x7feca406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:06.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.275+0000 7fecc6e7f700 1 -- 172.21.15.189:0/2533088834 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7feca0005b20 con 0x7fecc0161a30 2024-04-06T13:47:06.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.276+0000 7fecbd7fa700 1 -- 172.21.15.189:0/2533088834 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v213) v1 ==== 100+0+11 (secure 0 0 0) 0x7fecac058910 con 0x7fecc0161a30 2024-04-06T13:47:06.277 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:47:06.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.278+0000 7fecc6e7f700 1 -- 172.21.15.189:0/2533088834 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feca406aed0 msgr2=0x7feca406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:06.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.278+0000 7fecc6e7f700 1 --2- 172.21.15.189:0/2533088834 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feca406aed0 0x7feca406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.278+0000 7fecc6e7f700 1 -- 172.21.15.189:0/2533088834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecc0161a30 msgr2=0x7fecc0156a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:06.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.278+0000 7fecc6e7f700 1 --2- 172.21.15.189:0/2533088834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecc0161a30 0x7fecc0156a00 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fecac015000 tx=0x7fecac004750 comp rx=0 tx=0).stop 2024-04-06T13:47:06.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.278+0000 7fecc6e7f700 1 -- 172.21.15.189:0/2533088834 shutdown_connections 2024-04-06T13:47:06.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.278+0000 7fecc6e7f700 1 --2- 172.21.15.189:0/2533088834 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feca406aed0 0x7feca406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.278+0000 7fecc6e7f700 1 --2- 172.21.15.189:0/2533088834 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecc0156f40 0x7fecc015baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.278+0000 7fecc6e7f700 1 --2- 172.21.15.189:0/2533088834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc006df60 0x7fecc01472d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.278+0000 7fecc6e7f700 1 --2- 172.21.15.189:0/2533088834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecc0161a30 0x7fecc0156a00 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.278+0000 7fecc6e7f700 1 -- 172.21.15.189:0/2533088834 >> 172.21.15.189:0/2533088834 conn(0x7fecc00cefa0 msgr2=0x7fecc00cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:06.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.278+0000 7fecc6e7f700 1 -- 172.21.15.189:0/2533088834 shutdown_connections 2024-04-06T13:47:06.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.278+0000 7fecc6e7f700 1 -- 172.21.15.189:0/2533088834 wait complete. 2024-04-06T13:47:06.293 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T13:47:06.401 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:06.401+0000 7fce45124200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:47:06.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.419+0000 7f56165b7700 1 Processor -- start 2024-04-06T13:47:06.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.419+0000 7f56165b7700 1 -- start start 2024-04-06T13:47:06.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.420+0000 7f56165b7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f56100d11b0 0x7f56100d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:06.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.420+0000 7f56165b7700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f56100cb870 0x7f561015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:06.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.420+0000 7f56165b7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56100cfa00 0x7f56100cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:06.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.420+0000 7f56165b7700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56100ccdc0 con 0x7f56100d11b0 2024-04-06T13:47:06.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.420+0000 7f56165b7700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56100d0b50 con 0x7f56100cb870 2024-04-06T13:47:06.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.420+0000 7f56165b7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56100d1ad0 con 0x7f56100cfa00 2024-04-06T13:47:06.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.420+0000 7f560ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f56100d11b0 0x7f56100d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:06.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.420+0000 7f560ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f56100d11b0 0x7f56100d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47046/0 (socket says 172.21.15.189:47046) 2024-04-06T13:47:06.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.420+0000 7f560ffff700 1 -- 172.21.15.189:0/2201732746 learned_addr learned my addr 172.21.15.189:0/2201732746 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:06.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.420+0000 7f560f7fe700 1 --2- 172.21.15.189:0/2201732746 >> v2:172.21.15.192:3300/0 conn(0x7f56100cb870 0x7f561015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:06.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.420+0000 7f5614b54700 1 --2- 172.21.15.189:0/2201732746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56100cfa00 0x7f56100cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:06.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.420+0000 7f560ffff700 1 -- 172.21.15.189:0/2201732746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56100cfa00 msgr2=0x7f56100cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:06.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.420+0000 7f560ffff700 1 --2- 172.21.15.189:0/2201732746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56100cfa00 0x7f56100cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.420+0000 7f560ffff700 1 -- 172.21.15.189:0/2201732746 >> v2:172.21.15.192:3300/0 conn(0x7f56100cb870 msgr2=0x7f561015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:06.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.420+0000 7f560ffff700 1 --2- 172.21.15.189:0/2201732746 >> v2:172.21.15.192:3300/0 conn(0x7f56100cb870 0x7f561015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.420+0000 7f560ffff700 1 -- 172.21.15.189:0/2201732746 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56100cbd80 con 0x7f56100d11b0 2024-04-06T13:47:06.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.421+0000 7f560ffff700 1 --2- 172.21.15.189:0/2201732746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56100d11b0 0x7f56100d1590 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f561004fb40 tx=0x7f5604009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba910542239f5949 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:06.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.421+0000 7f560effd700 1 -- 172.21.15.189:0/2201732746 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5604011830 con 0x7f56100d11b0 2024-04-06T13:47:06.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.421+0000 7f560effd700 1 -- 172.21.15.189:0/2201732746 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5604011990 con 0x7f56100d11b0 2024-04-06T13:47:06.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.421+0000 7f560effd700 1 -- 172.21.15.189:0/2201732746 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5604011d20 con 0x7f56100d11b0 2024-04-06T13:47:06.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.421+0000 7f56165b7700 1 -- 172.21.15.189:0/2201732746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56100d11b0 msgr2=0x7f56100d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:06.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.421+0000 7f56165b7700 1 --2- 172.21.15.189:0/2201732746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56100d11b0 0x7f56100d1590 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f561004fb40 tx=0x7f5604009bd0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.421+0000 7f56165b7700 1 -- 172.21.15.189:0/2201732746 shutdown_connections 2024-04-06T13:47:06.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.421+0000 7f56165b7700 1 --2- 172.21.15.189:0/2201732746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56100d11b0 0x7f56100d1590 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.421+0000 7f56165b7700 1 --2- 172.21.15.189:0/2201732746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56100cfa00 0x7f56100cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.421+0000 7f56165b7700 1 --2- 172.21.15.189:0/2201732746 >> v2:172.21.15.192:3300/0 conn(0x7f56100cb870 0x7f561015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.421+0000 7f56165b7700 1 -- 172.21.15.189:0/2201732746 >> 172.21.15.189:0/2201732746 conn(0x7f56100b2db0 msgr2=0x7f56100b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:06.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.421+0000 7f56165b7700 1 -- 172.21.15.189:0/2201732746 shutdown_connections 2024-04-06T13:47:06.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.421+0000 7f56165b7700 1 -- 172.21.15.189:0/2201732746 wait complete. 2024-04-06T13:47:06.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.423+0000 7f56165b7700 1 Processor -- start 2024-04-06T13:47:06.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.423+0000 7f56165b7700 1 -- start start 2024-04-06T13:47:06.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.424+0000 7f56165b7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56100cb870 0x7f56101557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:06.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.424+0000 7f56165b7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56100cfa00 0x7f56101588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:06.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.424+0000 7f56165b7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56100d11b0 0x7f56100c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:06.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.424+0000 7f56165b7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5610155d20 con 0x7f56100cfa00 2024-04-06T13:47:06.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.424+0000 7f56165b7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56101575e0 con 0x7f56100d11b0 2024-04-06T13:47:06.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.424+0000 7f56165b7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5610158e10 con 0x7f56100cb870 2024-04-06T13:47:06.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.424+0000 7f560f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56100cfa00 0x7f56101588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:06.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.424+0000 7f560f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56100cfa00 0x7f56101588d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47062/0 (socket says 172.21.15.189:47062) 2024-04-06T13:47:06.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.424+0000 7f560f7fe700 1 -- 172.21.15.189:0/848370549 learned_addr learned my addr 172.21.15.189:0/848370549 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:06.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.424+0000 7f560ffff700 1 --2- 172.21.15.189:0/848370549 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56100cb870 0x7f56101557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:06.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.424+0000 7f5614b54700 1 --2- 172.21.15.189:0/848370549 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56100d11b0 0x7f56100c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:06.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.424+0000 7f560f7fe700 1 -- 172.21.15.189:0/848370549 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56100cb870 msgr2=0x7f56101557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:06.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.424+0000 7f560f7fe700 1 --2- 172.21.15.189:0/848370549 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56100cb870 0x7f56101557e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.424+0000 7f560f7fe700 1 -- 172.21.15.189:0/848370549 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56100d11b0 msgr2=0x7f56100c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:06.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.424+0000 7f560f7fe700 1 --2- 172.21.15.189:0/848370549 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56100d11b0 0x7f56100c98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.424+0000 7f560f7fe700 1 -- 172.21.15.189:0/848370549 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5604009550 con 0x7f56100cfa00 2024-04-06T13:47:06.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.425+0000 7f560f7fe700 1 --2- 172.21.15.189:0/848370549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56100cfa00 0x7f56101588d0 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f560000e870 tx=0x7f560000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:06.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.425+0000 7f560cff9700 1 -- 172.21.15.189:0/848370549 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600013680 con 0x7f56100cfa00 2024-04-06T13:47:06.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.425+0000 7f560cff9700 1 -- 172.21.15.189:0/848370549 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56000137e0 con 0x7f56100cfa00 2024-04-06T13:47:06.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.425+0000 7f56165b7700 1 -- 172.21.15.189:0/848370549 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5610157830 con 0x7f56100cfa00 2024-04-06T13:47:06.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.425+0000 7f560cff9700 1 -- 172.21.15.189:0/848370549 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600013b70 con 0x7f56100cfa00 2024-04-06T13:47:06.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.425+0000 7f56165b7700 1 -- 172.21.15.189:0/848370549 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5610157df0 con 0x7f56100cfa00 2024-04-06T13:47:06.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.426+0000 7f56165b7700 1 -- 172.21.15.189:0/848370549 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5610074990 con 0x7f56100cfa00 2024-04-06T13:47:06.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.427+0000 7f560cff9700 1 -- 172.21.15.189:0/848370549 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 135) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f5600013cd0 con 0x7f56100cfa00 2024-04-06T13:47:06.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.427+0000 7f560cff9700 1 --2- 172.21.15.189:0/848370549 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f55fc06aed0 0x7f55fc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:06.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.427+0000 7f560ffff700 1 -- 172.21.15.189:0/848370549 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f55fc06aed0 msgr2=0x7f55fc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:06.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.427+0000 7f560cff9700 1 -- 172.21.15.189:0/848370549 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6620+0+0 (secure 0 0 0) 0x7f5600089760 con 0x7f56100cfa00 2024-04-06T13:47:06.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.427+0000 7f560ffff700 1 --2- 172.21.15.189:0/848370549 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f55fc06aed0 0x7f55fc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:06.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.431+0000 7f560cff9700 1 -- 172.21.15.189:0/848370549 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f56000542e0 con 0x7f56100cfa00 2024-04-06T13:47:06.495 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:06.493+0000 7fbb7ac74200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:47:06.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.628+0000 7f560ffff700 1 -- 172.21.15.189:0/848370549 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f55fc06aed0 msgr2=0x7f55fc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:06.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.628+0000 7f560ffff700 1 --2- 172.21.15.189:0/848370549 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f55fc06aed0 0x7f55fc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:06.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.654+0000 7f56165b7700 1 -- 172.21.15.189:0/848370549 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f56100cb320 con 0x7f56100cfa00 2024-04-06T13:47:06.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.654+0000 7f560cff9700 1 -- 172.21.15.189:0/848370549 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v214) v1 ==== 80+0+468 (secure 0 0 0) 0x7f5600057900 con 0x7f56100cfa00 2024-04-06T13:47:06.663 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/509530111 2024-04-07T13:45:54.091329+0000 2024-04-06T13:47:06.663 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/3787698124 2024-04-07T13:45:54.091329+0000 2024-04-06T13:47:06.663 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2802982514 2024-04-07T13:45:54.091329+0000 2024-04-06T13:47:06.664 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6833/111704 2024-04-07T13:45:54.091329+0000 2024-04-06T13:47:06.664 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6832/111704 2024-04-07T13:45:54.091329+0000 2024-04-06T13:47:06.664 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/606015831 2024-04-07T13:45:54.091329+0000 2024-04-06T13:47:06.664 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/4174593093 2024-04-07T13:45:54.091329+0000 2024-04-06T13:47:06.664 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/3225575705 2024-04-07T13:45:54.091329+0000 2024-04-06T13:47:06.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.656+0000 7f56165b7700 1 -- 172.21.15.189:0/848370549 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f55fc06aed0 msgr2=0x7f55fc06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:06.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.656+0000 7f56165b7700 1 --2- 172.21.15.189:0/848370549 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f55fc06aed0 0x7f55fc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.656+0000 7f56165b7700 1 -- 172.21.15.189:0/848370549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56100cfa00 msgr2=0x7f56101588d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:06.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.656+0000 7f56165b7700 1 --2- 172.21.15.189:0/848370549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56100cfa00 0x7f56101588d0 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f560000e870 tx=0x7f560000eb80 comp rx=0 tx=0).stop 2024-04-06T13:47:06.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.656+0000 7f56165b7700 1 -- 172.21.15.189:0/848370549 shutdown_connections 2024-04-06T13:47:06.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.656+0000 7f56165b7700 1 --2- 172.21.15.189:0/848370549 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f55fc06aed0 0x7f55fc06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.656+0000 7f56165b7700 1 --2- 172.21.15.189:0/848370549 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56100d11b0 0x7f56100c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.656+0000 7f56165b7700 1 --2- 172.21.15.189:0/848370549 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56100cb870 0x7f56101557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.656+0000 7f56165b7700 1 --2- 172.21.15.189:0/848370549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56100cfa00 0x7f56101588d0 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.656+0000 7f56165b7700 1 -- 172.21.15.189:0/848370549 >> 172.21.15.189:0/848370549 conn(0x7f56100b2db0 msgr2=0x7f56100b6060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:06.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.656+0000 7f56165b7700 1 -- 172.21.15.189:0/848370549 shutdown_connections 2024-04-06T13:47:06.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.656+0000 7f56165b7700 1 -- 172.21.15.189:0/848370549 wait complete. 2024-04-06T13:47:06.666 INFO:teuthology.orchestra.run.smithi189.stderr:listed 8 entries 2024-04-06T13:47:06.670 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-06T13:47:06.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:06.671 DEBUG:teuthology.orchestra.run.smithi189:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-06T13:47:06.725 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:06.725+0000 7fce45124200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:47:06.735 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T13:47:06.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:06.736 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-06T13:47:06.757 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:47:06.757 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:47:06.775 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:47:06.775 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:47:06.775 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:47:06.834 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:47:06.835 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:47:06.835 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:47:06.837 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:06.837+0000 7fbb7ac74200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:47:06.852 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:47:06.853 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:47:06.891 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:06.891+0000 7fce45124200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:47:06.966 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:06.964+0000 7fce45124200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:47:06.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.982+0000 7f81d2057700 1 Processor -- start 2024-04-06T13:47:06.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.982+0000 7f81d2057700 1 -- start start 2024-04-06T13:47:06.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.983+0000 7f81d2057700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81cc15d7e0 0x7f81cc0d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:06.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.983+0000 7f81d2057700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc0cdf80 con 0x7f81cc0d0890 2024-04-06T13:47:06.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.983+0000 7f81d2057700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc0d3230 con 0x7f81cc0d1cc0 2024-04-06T13:47:06.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.983+0000 7f81d2057700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc0d0d10 con 0x7f81cc15d7e0 2024-04-06T13:47:06.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.983+0000 7f81caffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f81cc0d0890 0x7f81cc0d15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42682/0 (socket says 172.21.15.189:42682) 2024-04-06T13:47:06.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.983+0000 7f81caffd700 1 -- 172.21.15.189:0/2937285403 learned_addr learned my addr 172.21.15.189:0/2937285403 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:06.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.983+0000 7f81cb7fe700 1 --2- 172.21.15.189:0/2937285403 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81cc15d7e0 0x7f81cc0d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:06.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.983+0000 7f81ca7fc700 1 -- 172.21.15.189:0/2937285403 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910531912 0 0) 0x7f81cc0cdf80 con 0x7f81cc0d0890 2024-04-06T13:47:06.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.984+0000 7f81ca7fc700 1 -- 172.21.15.189:0/2937285403 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b40035b0 con 0x7f81cc0d0890 2024-04-06T13:47:06.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.984+0000 7f81ca7fc700 1 -- 172.21.15.189:0/2937285403 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529577187 0 0) 0x7f81c4001110 con 0x7f81cc0d1cc0 2024-04-06T13:47:06.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.984+0000 7f81ca7fc700 1 -- 172.21.15.189:0/2937285403 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc0cdf80 con 0x7f81cc0d1cc0 2024-04-06T13:47:06.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.984+0000 7f81cb7fe700 1 -- 172.21.15.189:0/2937285403 >> v1:172.21.15.192:6789/0 conn(0x7f81cc0d1cc0 legacy=0x7f81cc0b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:06.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.984+0000 7f81cb7fe700 1 -- 172.21.15.189:0/2937285403 >> v1:172.21.15.189:6789/0 conn(0x7f81cc0d0890 legacy=0x7f81cc0d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:06.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.984+0000 7f81cb7fe700 1 -- 172.21.15.189:0/2937285403 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc0d2da0 con 0x7f81cc15d7e0 2024-04-06T13:47:06.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.984+0000 7f81cb7fe700 1 --2- 172.21.15.189:0/2937285403 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81cc15d7e0 0x7f81cc0d0350 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009dc0 tx=0x7f81bc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=846be5377d7dfbeb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:06.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.984+0000 7f81ca7fc700 1 -- 172.21.15.189:0/2937285403 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc00e070 con 0x7f81cc15d7e0 2024-04-06T13:47:06.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.984+0000 7f81ca7fc700 1 -- 172.21.15.189:0/2937285403 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81bc00bb20 con 0x7f81cc15d7e0 2024-04-06T13:47:06.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.984+0000 7f81ca7fc700 1 -- 172.21.15.189:0/2937285403 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc012430 con 0x7f81cc15d7e0 2024-04-06T13:47:06.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.985+0000 7f81d2057700 1 -- 172.21.15.189:0/2937285403 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81cc15d7e0 msgr2=0x7f81cc0d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:06.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.985+0000 7f81d2057700 1 --2- 172.21.15.189:0/2937285403 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81cc15d7e0 0x7f81cc0d0350 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009dc0 tx=0x7f81bc00b330 comp rx=0 tx=0).stop 2024-04-06T13:47:06.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.985+0000 7f81d2057700 1 -- 172.21.15.189:0/2937285403 shutdown_connections 2024-04-06T13:47:06.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.985+0000 7f81d2057700 1 --2- 172.21.15.189:0/2937285403 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81cc15d7e0 0x7f81cc0d0350 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:06.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.985+0000 7f81d2057700 1 -- 172.21.15.189:0/2937285403 >> 172.21.15.189:0/2937285403 conn(0x7f81cc072250 msgr2=0x7f81cc072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:06.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.985+0000 7f81d2057700 1 -- 172.21.15.189:0/2937285403 shutdown_connections 2024-04-06T13:47:06.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.985+0000 7f81d2057700 1 -- 172.21.15.189:0/2937285403 wait complete. 2024-04-06T13:47:06.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.986+0000 7f81d2057700 1 Processor -- start 2024-04-06T13:47:06.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.986+0000 7f81d2057700 1 -- start start 2024-04-06T13:47:06.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81d2057700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81cc0d0890 0x7f81cc0ab4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:06.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81d2057700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81cc0d1cc0 0x7f81cc0c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:06.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81d2057700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81cc15d7e0 0x7f81cc0ae9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:06.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81d2057700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc15c0a0 con 0x7f81cc15d7e0 2024-04-06T13:47:06.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81d2057700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc0aa410 con 0x7f81cc0d0890 2024-04-06T13:47:06.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81d2057700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc0aa6f0 con 0x7f81cc0d1cc0 2024-04-06T13:47:06.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81caffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81cc0d1cc0 0x7f81cc0c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:07.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81cbfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81cc15d7e0 0x7f81cc0ae9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:07.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81cb7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81cc0d0890 0x7f81cc0ab4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:07.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81caffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81cc0d1cc0 0x7f81cc0c8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:36724/0 (socket says 172.21.15.189:36724) 2024-04-06T13:47:07.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81cb7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81cc0d0890 0x7f81cc0ab4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:53334/0 (socket says 172.21.15.189:53334) 2024-04-06T13:47:07.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81caffd700 1 -- 172.21.15.189:0/3868417728 learned_addr learned my addr 172.21.15.189:0/3868417728 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:07.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81cbfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81cc15d7e0 0x7f81cc0ae9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47066/0 (socket says 172.21.15.189:47066) 2024-04-06T13:47:07.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81cbfff700 1 -- 172.21.15.189:0/3868417728 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81cc0d1cc0 msgr2=0x7f81cc0c8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:07.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81cbfff700 1 --2- 172.21.15.189:0/3868417728 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81cc0d1cc0 0x7f81cc0c8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81cbfff700 1 -- 172.21.15.189:0/3868417728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81cc0d0890 msgr2=0x7f81cc0ab4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:07.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81cbfff700 1 --2- 172.21.15.189:0/3868417728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81cc0d0890 0x7f81cc0ab4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81cbfff700 1 -- 172.21.15.189:0/3868417728 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81bc009a70 con 0x7f81cc15d7e0 2024-04-06T13:47:07.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81cb7fe700 1 --2- 172.21.15.189:0/3868417728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81cc0d0890 0x7f81cc0ab4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:07.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81caffd700 1 --2- 172.21.15.189:0/3868417728 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81cc0d1cc0 0x7f81cc0c8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:07.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.987+0000 7f81cbfff700 1 --2- 172.21.15.189:0/3868417728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81cc15d7e0 0x7f81cc0ae9d0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f81c400c700 tx=0x7f81c400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:07.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.988+0000 7f81b3fff700 1 -- 172.21.15.189:0/3868417728 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c40052d0 con 0x7f81cc15d7e0 2024-04-06T13:47:07.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.988+0000 7f81b3fff700 1 -- 172.21.15.189:0/3868417728 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c4005430 con 0x7f81cc15d7e0 2024-04-06T13:47:07.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.988+0000 7f81d2057700 1 -- 172.21.15.189:0/3868417728 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81cc0aa9d0 con 0x7f81cc15d7e0 2024-04-06T13:47:07.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.988+0000 7f81b3fff700 1 -- 172.21.15.189:0/3868417728 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c40038c0 con 0x7f81cc15d7e0 2024-04-06T13:47:07.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.988+0000 7f81d2057700 1 -- 172.21.15.189:0/3868417728 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81cc0aaf90 con 0x7f81cc15d7e0 2024-04-06T13:47:07.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.989+0000 7f81d2057700 1 -- 172.21.15.189:0/3868417728 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81cc074990 con 0x7f81cc15d7e0 2024-04-06T13:47:07.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.989+0000 7f81b3fff700 1 -- 172.21.15.189:0/3868417728 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 135) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f81c4003a20 con 0x7f81cc15d7e0 2024-04-06T13:47:07.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.990+0000 7f81b3fff700 1 --2- 172.21.15.189:0/3868417728 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f81b806aed0 0x7f81b806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:07.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.990+0000 7f81cb7fe700 1 -- 172.21.15.189:0/3868417728 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f81b806aed0 msgr2=0x7f81b806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:07.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.990+0000 7f81b3fff700 1 -- 172.21.15.189:0/3868417728 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6604+0+0 (secure 0 0 0) 0x7f81c4057a80 con 0x7f81cc15d7e0 2024-04-06T13:47:07.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.990+0000 7f81cb7fe700 1 --2- 172.21.15.189:0/3868417728 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f81b806aed0 0x7f81b806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:07.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:06.995+0000 7f81b3fff700 1 -- 172.21.15.189:0/3868417728 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f81c40536c0 con 0x7f81cc15d7e0 2024-04-06T13:47:07.017 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:07.012+0000 7fbb7ac74200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:47:07.089 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:07.089+0000 7fbb7ac74200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:47:07.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.191+0000 7f81cb7fe700 1 -- 172.21.15.189:0/3868417728 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f81b806aed0 msgr2=0x7f81b806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:07.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.191+0000 7f81cb7fe700 1 --2- 172.21.15.189:0/3868417728 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f81b806aed0 0x7f81b806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:07.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.251+0000 7f81d2057700 1 -- 172.21.15.189:0/3868417728 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81cc0d3210 con 0x7f81cc15d7e0 2024-04-06T13:47:07.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.252+0000 7f81b3fff700 1 -- 172.21.15.189:0/3868417728 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+4750 (secure 0 0 0) 0x7f81c4017020 con 0x7f81cc15d7e0 2024-04-06T13:47:07.253 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:07.253 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":93,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8525,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3302262361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3302262361},{"type":"v1","addr":"172.21.15.189:6837","nonce":3302262361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":8534,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/229523550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":229523550},{"type":"v1","addr":"172.21.15.192:6834","nonce":229523550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":9739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2560949338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2560949338},{"type":"v1","addr":"172.21.15.192:6835","nonce":2560949338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:45:40.072274+0000","modified":"2024-04-06T13:45:43.092032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":0,"incarnation":89,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/3619731181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3619731181},{"type":"v1","addr":"172.21.15.189:6836","nonce":3619731181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-04-06T13:47:07.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.254+0000 7f81d2057700 1 -- 172.21.15.189:0/3868417728 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f81b806aed0 msgr2=0x7f81b806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:07.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.254+0000 7f81d2057700 1 --2- 172.21.15.189:0/3868417728 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f81b806aed0 0x7f81b806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.254+0000 7f81d2057700 1 -- 172.21.15.189:0/3868417728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81cc15d7e0 msgr2=0x7f81cc0ae9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:07.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.254+0000 7f81d2057700 1 --2- 172.21.15.189:0/3868417728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81cc15d7e0 0x7f81cc0ae9d0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f81c400c700 tx=0x7f81c400ca10 comp rx=0 tx=0).stop 2024-04-06T13:47:07.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.254+0000 7f81d2057700 1 -- 172.21.15.189:0/3868417728 shutdown_connections 2024-04-06T13:47:07.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.254+0000 7f81d2057700 1 --2- 172.21.15.189:0/3868417728 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f81b806aed0 0x7f81b806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.254+0000 7f81d2057700 1 --2- 172.21.15.189:0/3868417728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81cc15d7e0 0x7f81cc0ae9d0 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.254+0000 7f81d2057700 1 --2- 172.21.15.189:0/3868417728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81cc0d0890 0x7f81cc0ab4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.254+0000 7f81d2057700 1 --2- 172.21.15.189:0/3868417728 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81cc0d1cc0 0x7f81cc0c8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.254+0000 7f81d2057700 1 -- 172.21.15.189:0/3868417728 >> 172.21.15.189:0/3868417728 conn(0x7f81cc072250 msgr2=0x7f81cc0ce110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:07.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.254+0000 7f81d2057700 1 -- 172.21.15.189:0/3868417728 shutdown_connections 2024-04-06T13:47:07.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.254+0000 7f81d2057700 1 -- 172.21.15.189:0/3868417728 wait complete. 2024-04-06T13:47:07.257 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 93 2024-04-06T13:47:07.269 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:47:07.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.396+0000 7f9fc80a2700 1 Processor -- start 2024-04-06T13:47:07.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.396+0000 7f9fc80a2700 1 -- start start 2024-04-06T13:47:07.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.397+0000 7f9fc80a2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9fc0160b60 0x7f9fc00b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:07.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.397+0000 7f9fc80a2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fc0143470 0x7f9fc00cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:07.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.397+0000 7f9fc80a2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc015eb40 con 0x7f9fc015db10 2024-04-06T13:47:07.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.397+0000 7f9fc80a2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fc0160f40 con 0x7f9fc0160b60 2024-04-06T13:47:07.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.397+0000 7f9fc80a2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc00d0bf0 con 0x7f9fc0143470 2024-04-06T13:47:07.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.397+0000 7f9fc563d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9fc015db10 0x7f9fc0142d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42692/0 (socket says 172.21.15.189:42692) 2024-04-06T13:47:07.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.397+0000 7f9fc563d700 1 -- 172.21.15.189:0/2586253355 learned_addr learned my addr 172.21.15.189:0/2586253355 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:07.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.398+0000 7f9fc663f700 1 --2- 172.21.15.189:0/2586253355 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fc0143470 0x7f9fc00cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:07.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.398+0000 7f9fc5e3e700 1 --2- 172.21.15.189:0/2586253355 >> v2:172.21.15.192:3300/0 conn(0x7f9fc0160b60 0x7f9fc00b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:07.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.398+0000 7f9fc4e3c700 1 -- 172.21.15.189:0/2586253355 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788278114 0 0) 0x7f9fbc001290 con 0x7f9fc015db10 2024-04-06T13:47:07.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.398+0000 7f9fc4e3c700 1 -- 172.21.15.189:0/2586253355 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fac0035b0 con 0x7f9fc015db10 2024-04-06T13:47:07.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.398+0000 7f9fc663f700 1 -- 172.21.15.189:0/2586253355 >> v2:172.21.15.192:3300/0 conn(0x7f9fc0160b60 msgr2=0x7f9fc00b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:07.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.398+0000 7f9fc663f700 1 --2- 172.21.15.189:0/2586253355 >> v2:172.21.15.192:3300/0 conn(0x7f9fc0160b60 0x7f9fc00b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.398+0000 7f9fc663f700 1 -- 172.21.15.189:0/2586253355 >> v1:172.21.15.189:6789/0 conn(0x7f9fc015db10 legacy=0x7f9fc0142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:07.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.398+0000 7f9fc663f700 1 -- 172.21.15.189:0/2586253355 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc015dfc0 con 0x7f9fc0143470 2024-04-06T13:47:07.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.398+0000 7f9fc663f700 1 --2- 172.21.15.189:0/2586253355 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fc0143470 0x7f9fc00cd890 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8009d90 tx=0x7f9fb800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fc0fd82cf85afde server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:07.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.398+0000 7f9fc4e3c700 1 -- 172.21.15.189:0/2586253355 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb800e070 con 0x7f9fc0143470 2024-04-06T13:47:07.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.398+0000 7f9fc4e3c700 1 -- 172.21.15.189:0/2586253355 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb800bb20 con 0x7f9fc0143470 2024-04-06T13:47:07.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.398+0000 7f9fc4e3c700 1 -- 172.21.15.189:0/2586253355 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb8012430 con 0x7f9fc0143470 2024-04-06T13:47:07.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.399+0000 7f9fc80a2700 1 -- 172.21.15.189:0/2586253355 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fc0143470 msgr2=0x7f9fc00cd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:07.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.399+0000 7f9fc80a2700 1 --2- 172.21.15.189:0/2586253355 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fc0143470 0x7f9fc00cd890 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8009d90 tx=0x7f9fb800b330 comp rx=0 tx=0).stop 2024-04-06T13:47:07.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.399+0000 7f9fc80a2700 1 -- 172.21.15.189:0/2586253355 shutdown_connections 2024-04-06T13:47:07.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.399+0000 7f9fc80a2700 1 --2- 172.21.15.189:0/2586253355 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fc0143470 0x7f9fc00cd890 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.399+0000 7f9fc80a2700 1 --2- 172.21.15.189:0/2586253355 >> v2:172.21.15.192:3300/0 conn(0x7f9fc0160b60 0x7f9fc00b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.399+0000 7f9fc80a2700 1 -- 172.21.15.189:0/2586253355 >> 172.21.15.189:0/2586253355 conn(0x7f9fc006a730 msgr2=0x7f9fc006ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:07.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.399+0000 7f9fc80a2700 1 -- 172.21.15.189:0/2586253355 shutdown_connections 2024-04-06T13:47:07.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.399+0000 7f9fc80a2700 1 -- 172.21.15.189:0/2586253355 wait complete. 2024-04-06T13:47:07.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.400+0000 7f9fc80a2700 1 Processor -- start 2024-04-06T13:47:07.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.400+0000 7f9fc80a2700 1 -- start start 2024-04-06T13:47:07.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.401+0000 7f9fc80a2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fc0143470 0x7f9fc00cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:07.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.401+0000 7f9fc80a2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fc015db10 0x7f9fc00c6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:07.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.401+0000 7f9fc80a2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fc0160b60 0x7f9fc00c74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:07.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.401+0000 7f9fc80a2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc00cb1e0 con 0x7f9fc015db10 2024-04-06T13:47:07.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.401+0000 7f9fc80a2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc00c9840 con 0x7f9fc0143470 2024-04-06T13:47:07.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.401+0000 7f9fc80a2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc00c9c60 con 0x7f9fc0160b60 2024-04-06T13:47:07.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.401+0000 7f9fc563d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fc015db10 0x7f9fc00c6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:07.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.401+0000 7f9fc563d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fc015db10 0x7f9fc00c6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47072/0 (socket says 172.21.15.189:47072) 2024-04-06T13:47:07.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.401+0000 7f9fc563d700 1 -- 172.21.15.189:0/9204424 learned_addr learned my addr 172.21.15.189:0/9204424 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:07.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.401+0000 7f9fc5e3e700 1 --2- 172.21.15.189:0/9204424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fc0143470 0x7f9fc00cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:07.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.401+0000 7f9fc663f700 1 --2- 172.21.15.189:0/9204424 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fc0160b60 0x7f9fc00c74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:07.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.402+0000 7f9fc563d700 1 -- 172.21.15.189:0/9204424 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fc0160b60 msgr2=0x7f9fc00c74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:07.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.402+0000 7f9fc563d700 1 --2- 172.21.15.189:0/9204424 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fc0160b60 0x7f9fc00c74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.402+0000 7f9fc563d700 1 -- 172.21.15.189:0/9204424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fc0143470 msgr2=0x7f9fc00cab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:07.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.402+0000 7f9fc563d700 1 --2- 172.21.15.189:0/9204424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fc0143470 0x7f9fc00cab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.402+0000 7f9fc563d700 1 -- 172.21.15.189:0/9204424 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb8009a70 con 0x7f9fc015db10 2024-04-06T13:47:07.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.402+0000 7f9fc5e3e700 1 --2- 172.21.15.189:0/9204424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fc0143470 0x7f9fc00cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:07.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.402+0000 7f9fc663f700 1 --2- 172.21.15.189:0/9204424 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fc0160b60 0x7f9fc00c74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:07.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.402+0000 7f9fc563d700 1 --2- 172.21.15.189:0/9204424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fc015db10 0x7f9fc00c6fb0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc00c700 tx=0x7f9fbc00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:07.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.402+0000 7f9fb67fc700 1 -- 172.21.15.189:0/9204424 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc005c40 con 0x7f9fc015db10 2024-04-06T13:47:07.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.402+0000 7f9fb67fc700 1 -- 172.21.15.189:0/9204424 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fbc005da0 con 0x7f9fc015db10 2024-04-06T13:47:07.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.402+0000 7f9fc80a2700 1 -- 172.21.15.189:0/9204424 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fc015d280 con 0x7f9fc015db10 2024-04-06T13:47:07.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.402+0000 7f9fb67fc700 1 -- 172.21.15.189:0/9204424 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc0072f0 con 0x7f9fc015db10 2024-04-06T13:47:07.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.402+0000 7f9fc80a2700 1 -- 172.21.15.189:0/9204424 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fc00c9180 con 0x7f9fc015db10 2024-04-06T13:47:07.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.403+0000 7f9fc80a2700 1 -- 172.21.15.189:0/9204424 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fc0067bf0 con 0x7f9fc015db10 2024-04-06T13:47:07.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.404+0000 7f9fb67fc700 1 -- 172.21.15.189:0/9204424 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 135) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f9fbc007450 con 0x7f9fc015db10 2024-04-06T13:47:07.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.404+0000 7f9fb67fc700 1 --2- 172.21.15.189:0/9204424 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9fa406af00 0x7f9fa406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:07.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.404+0000 7f9fc5e3e700 1 -- 172.21.15.189:0/9204424 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9fa406af00 msgr2=0x7f9fa406d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:07.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.404+0000 7f9fb67fc700 1 -- 172.21.15.189:0/9204424 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6604+0+0 (secure 0 0 0) 0x7f9fbc0892c0 con 0x7f9fc015db10 2024-04-06T13:47:07.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.404+0000 7f9fc5e3e700 1 --2- 172.21.15.189:0/9204424 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9fa406af00 0x7f9fa406d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:07.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.408+0000 7f9fb67fc700 1 -- 172.21.15.189:0/9204424 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9fbc053da0 con 0x7f9fc015db10 2024-04-06T13:47:07.581 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:07.581+0000 7fce45124200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:47:07.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.606+0000 7f9fc5e3e700 1 -- 172.21.15.189:0/9204424 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9fa406af00 msgr2=0x7f9fa406d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:07.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.606+0000 7f9fc5e3e700 1 --2- 172.21.15.189:0/9204424 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9fa406af00 0x7f9fa406d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:07.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.612+0000 7f9fc80a2700 1 -- 172.21.15.189:0/9204424 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fc00c9980 con 0x7f9fc015db10 2024-04-06T13:47:07.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.614+0000 7f9fb67fc700 1 -- 172.21.15.189:0/9204424 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v214) v1 ==== 74+0+17252 (secure 0 0 0) 0x7f9fc00c9980 con 0x7f9fc015db10 2024-04-06T13:47:07.615 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:07.615 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":214,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:47:06.599488+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:45:37.837008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-04-06T13:45:39.030714+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":214,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.22","mappings":[{"from":1,"to":0}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]},{"pgid":"26.2c","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/509530111":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/3787698124":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/2802982514":"2024-04-07T13:45:54.091329+0000","172.21.15.189:6833/111704":"2024-04-07T13:45:54.091329+0000","172.21.15.189:6832/111704":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/606015831":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/4174593093":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/3225575705":"2024-04-07T13:45:54.091329+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":26,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:47:07.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.616+0000 7f9fc80a2700 1 -- 172.21.15.189:0/9204424 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9fa406af00 msgr2=0x7f9fa406d3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:07.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.616+0000 7f9fc80a2700 1 --2- 172.21.15.189:0/9204424 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9fa406af00 0x7f9fa406d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.616+0000 7f9fc80a2700 1 -- 172.21.15.189:0/9204424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fc015db10 msgr2=0x7f9fc00c6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:07.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.616+0000 7f9fc80a2700 1 --2- 172.21.15.189:0/9204424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fc015db10 0x7f9fc00c6fb0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc00c700 tx=0x7f9fbc00ca10 comp rx=0 tx=0).stop 2024-04-06T13:47:07.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.616+0000 7f9fc80a2700 1 -- 172.21.15.189:0/9204424 shutdown_connections 2024-04-06T13:47:07.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.616+0000 7f9fc80a2700 1 --2- 172.21.15.189:0/9204424 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9fa406af00 0x7f9fa406d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.616+0000 7f9fc80a2700 1 --2- 172.21.15.189:0/9204424 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fc0160b60 0x7f9fc00c74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.616+0000 7f9fc80a2700 1 --2- 172.21.15.189:0/9204424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fc0143470 0x7f9fc00cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.616+0000 7f9fc80a2700 1 --2- 172.21.15.189:0/9204424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fc015db10 0x7f9fc00c6fb0 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.616+0000 7f9fc80a2700 1 -- 172.21.15.189:0/9204424 >> 172.21.15.189:0/9204424 conn(0x7f9fc006a730 msgr2=0x7f9fc0161080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:07.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.616+0000 7f9fc80a2700 1 -- 172.21.15.189:0/9204424 shutdown_connections 2024-04-06T13:47:07.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.616+0000 7f9fc80a2700 1 -- 172.21.15.189:0/9204424 wait complete. 2024-04-06T13:47:07.633 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T13:45:37.837008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '208', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T13:45:39.030714+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '214', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 214, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:47:07.634 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:47:07.674 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:07.661+0000 7fce45124200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:47:07.752 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:07.751+0000 7fbb7ac74200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:47:07.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.764+0000 7f3025f37700 1 Processor -- start 2024-04-06T13:47:07.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.764+0000 7f3025f37700 1 -- start start 2024-04-06T13:47:07.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.765+0000 7f3025f37700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f30200d11b0 0x7f30200d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:07.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.765+0000 7f3025f37700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f30200cb870 0x7f302015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:07.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.765+0000 7f3025f37700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30200d1ad0 con 0x7f30200cfa00 2024-04-06T13:47:07.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.765+0000 7f3025f37700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30200ccdc0 con 0x7f30200d11b0 2024-04-06T13:47:07.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.765+0000 7f3025f37700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30200d0b50 con 0x7f30200cb870 2024-04-06T13:47:07.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.765+0000 7f301f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f30200d11b0 0x7f30200d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:07.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.765+0000 7f301f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f30200d11b0 0x7f30200d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47082/0 (socket says 172.21.15.189:47082) 2024-04-06T13:47:07.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.765+0000 7f301f7fe700 1 -- 172.21.15.189:0/3620625862 learned_addr learned my addr 172.21.15.189:0/3620625862 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:07.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.765+0000 7f3017fff700 1 --2- 172.21.15.189:0/3620625862 >> v2:172.21.15.192:3300/0 conn(0x7f30200cb870 0x7f302015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:07.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.765+0000 7f301f7fe700 1 -- 172.21.15.189:0/3620625862 >> v2:172.21.15.192:3300/0 conn(0x7f30200cb870 msgr2=0x7f302015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:07.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.765+0000 7f301f7fe700 1 --2- 172.21.15.189:0/3620625862 >> v2:172.21.15.192:3300/0 conn(0x7f30200cb870 0x7f302015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.765+0000 7f301f7fe700 1 -- 172.21.15.189:0/3620625862 >> v1:172.21.15.192:6789/0 conn(0x7f30200cfa00 legacy=0x7f30200cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:07.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.765+0000 7f301f7fe700 1 -- 172.21.15.189:0/3620625862 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30200cbd80 con 0x7f30200d11b0 2024-04-06T13:47:07.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.765+0000 7f301f7fe700 1 --2- 172.21.15.189:0/3620625862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30200d11b0 0x7f30200d1590 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f302004fb40 tx=0x7f3010009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=571b73f6ba484a59 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:07.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.765+0000 7f301effd700 1 -- 172.21.15.189:0/3620625862 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3010011830 con 0x7f30200d11b0 2024-04-06T13:47:07.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.766+0000 7f301effd700 1 -- 172.21.15.189:0/3620625862 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3010011990 con 0x7f30200d11b0 2024-04-06T13:47:07.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.766+0000 7f301effd700 1 -- 172.21.15.189:0/3620625862 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3010011d20 con 0x7f30200d11b0 2024-04-06T13:47:07.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.766+0000 7f3025f37700 1 -- 172.21.15.189:0/3620625862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30200d11b0 msgr2=0x7f30200d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:07.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.766+0000 7f3025f37700 1 --2- 172.21.15.189:0/3620625862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30200d11b0 0x7f30200d1590 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f302004fb40 tx=0x7f3010009bd0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.766+0000 7f3025f37700 1 -- 172.21.15.189:0/3620625862 shutdown_connections 2024-04-06T13:47:07.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.766+0000 7f3025f37700 1 --2- 172.21.15.189:0/3620625862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30200d11b0 0x7f30200d1590 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.766+0000 7f3025f37700 1 --2- 172.21.15.189:0/3620625862 >> v2:172.21.15.192:3300/0 conn(0x7f30200cb870 0x7f302015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.766+0000 7f3025f37700 1 -- 172.21.15.189:0/3620625862 >> 172.21.15.189:0/3620625862 conn(0x7f30200b2db0 msgr2=0x7f30200b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:07.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.766+0000 7f3025f37700 1 -- 172.21.15.189:0/3620625862 shutdown_connections 2024-04-06T13:47:07.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.766+0000 7f3025f37700 1 -- 172.21.15.189:0/3620625862 wait complete. 2024-04-06T13:47:07.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.767+0000 7f3025f37700 1 Processor -- start 2024-04-06T13:47:07.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.767+0000 7f3025f37700 1 -- start start 2024-04-06T13:47:07.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.768+0000 7f3025f37700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f30200cb870 0x7f3020152050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:07.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.768+0000 7f3025f37700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30200cfa00 0x7f30201e6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:07.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.768+0000 7f3025f37700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30200d11b0 0x7f30201e6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:07.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.768+0000 7f3025f37700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302015e790 con 0x7f30200d11b0 2024-04-06T13:47:07.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.768+0000 7f3025f37700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302015db80 con 0x7f30200cfa00 2024-04-06T13:47:07.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.768+0000 7f3025f37700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302015de60 con 0x7f30200cb870 2024-04-06T13:47:07.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.768+0000 7f301ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30200d11b0 0x7f30201e6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:07.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.768+0000 7f301ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30200d11b0 0x7f30201e6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47086/0 (socket says 172.21.15.189:47086) 2024-04-06T13:47:07.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.768+0000 7f301ffff700 1 -- 172.21.15.189:0/3586512609 learned_addr learned my addr 172.21.15.189:0/3586512609 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:07.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.768+0000 7f3017fff700 1 --2- 172.21.15.189:0/3586512609 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30200cfa00 0x7f30201e6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:07.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.768+0000 7f301f7fe700 1 --2- 172.21.15.189:0/3586512609 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f30200cb870 0x7f3020152050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:07.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.769+0000 7f301ffff700 1 -- 172.21.15.189:0/3586512609 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f30200cb870 msgr2=0x7f3020152050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:07.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.769+0000 7f301ffff700 1 --2- 172.21.15.189:0/3586512609 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f30200cb870 0x7f3020152050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.769+0000 7f301ffff700 1 -- 172.21.15.189:0/3586512609 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30200cfa00 msgr2=0x7f30201e6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:07.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.769+0000 7f301ffff700 1 --2- 172.21.15.189:0/3586512609 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30200cfa00 0x7f30201e6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.769+0000 7f301ffff700 1 -- 172.21.15.189:0/3586512609 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3010009550 con 0x7f30200d11b0 2024-04-06T13:47:07.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.769+0000 7f301ffff700 1 --2- 172.21.15.189:0/3586512609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30200d11b0 0x7f30201e6840 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f301800cb50 tx=0x7f301800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:07.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.769+0000 7f301cff9700 1 -- 172.21.15.189:0/3586512609 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3018003c00 con 0x7f30200d11b0 2024-04-06T13:47:07.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.769+0000 7f301cff9700 1 -- 172.21.15.189:0/3586512609 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3018003d60 con 0x7f30200d11b0 2024-04-06T13:47:07.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.769+0000 7f3025f37700 1 -- 172.21.15.189:0/3586512609 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f302015e140 con 0x7f30200d11b0 2024-04-06T13:47:07.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.769+0000 7f301cff9700 1 -- 172.21.15.189:0/3586512609 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301800aa20 con 0x7f30200d11b0 2024-04-06T13:47:07.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.769+0000 7f3025f37700 1 -- 172.21.15.189:0/3586512609 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3020152590 con 0x7f30200d11b0 2024-04-06T13:47:07.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.770+0000 7f3025f37700 1 -- 172.21.15.189:0/3586512609 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3020074990 con 0x7f30200d11b0 2024-04-06T13:47:07.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.771+0000 7f301cff9700 1 -- 172.21.15.189:0/3586512609 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 135) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f301800ab80 con 0x7f30200d11b0 2024-04-06T13:47:07.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.771+0000 7f301cff9700 1 --2- 172.21.15.189:0/3586512609 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f300006aed0 0x7f300006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:07.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.771+0000 7f301f7fe700 1 -- 172.21.15.189:0/3586512609 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f300006aed0 msgr2=0x7f300006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:07.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.771+0000 7f301cff9700 1 -- 172.21.15.189:0/3586512609 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6604+0+0 (secure 0 0 0) 0x7f3018089ad0 con 0x7f30200d11b0 2024-04-06T13:47:07.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.771+0000 7f301f7fe700 1 --2- 172.21.15.189:0/3586512609 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f300006aed0 0x7f300006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:07.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.776+0000 7f301cff9700 1 -- 172.21.15.189:0/3586512609 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3018054600 con 0x7f30200d11b0 2024-04-06T13:47:07.789 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:07.788+0000 7fce45124200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:47:07.840 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:07.835+0000 7fbb7ac74200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:47:07.975 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:07.965+0000 7fbb7ac74200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:47:07.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.972+0000 7f301f7fe700 1 -- 172.21.15.189:0/3586512609 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f300006aed0 msgr2=0x7f300006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:07.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.972+0000 7f301f7fe700 1 --2- 172.21.15.189:0/3586512609 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f300006aed0 0x7f300006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:07.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.979+0000 7f3025f37700 1 -- 172.21.15.189:0/3586512609 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f302015e8d0 con 0x7f30200d11b0 2024-04-06T13:47:07.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.979+0000 7f301cff9700 1 -- 172.21.15.189:0/3586512609 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v214) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3018057c20 con 0x7f30200d11b0 2024-04-06T13:47:07.981 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:47:07.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.981+0000 7f3025f37700 1 -- 172.21.15.189:0/3586512609 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f300006aed0 msgr2=0x7f300006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:07.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.981+0000 7f3025f37700 1 --2- 172.21.15.189:0/3586512609 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f300006aed0 0x7f300006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.981+0000 7f3025f37700 1 -- 172.21.15.189:0/3586512609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30200d11b0 msgr2=0x7f30201e6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:07.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.981+0000 7f3025f37700 1 --2- 172.21.15.189:0/3586512609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30200d11b0 0x7f30201e6840 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f301800cb50 tx=0x7f301800ce60 comp rx=0 tx=0).stop 2024-04-06T13:47:07.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.982+0000 7f3025f37700 1 -- 172.21.15.189:0/3586512609 shutdown_connections 2024-04-06T13:47:07.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.982+0000 7f3025f37700 1 --2- 172.21.15.189:0/3586512609 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f300006aed0 0x7f300006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.982+0000 7f3025f37700 1 --2- 172.21.15.189:0/3586512609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30200d11b0 0x7f30201e6840 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.982+0000 7f3025f37700 1 --2- 172.21.15.189:0/3586512609 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f30200cb870 0x7f3020152050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.982+0000 7f3025f37700 1 --2- 172.21.15.189:0/3586512609 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30200cfa00 0x7f30201e6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:07.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.982+0000 7f3025f37700 1 -- 172.21.15.189:0/3586512609 >> 172.21.15.189:0/3586512609 conn(0x7f30200b2db0 msgr2=0x7f30200b5fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:07.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.982+0000 7f3025f37700 1 -- 172.21.15.189:0/3586512609 shutdown_connections 2024-04-06T13:47:07.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:07.982+0000 7f3025f37700 1 -- 172.21.15.189:0/3586512609 wait complete. 2024-04-06T13:47:07.996 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T13:47:08.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.130+0000 7fd378fb9700 1 Processor -- start 2024-04-06T13:47:08.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.130+0000 7fd378fb9700 1 -- start start 2024-04-06T13:47:08.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.131+0000 7fd378fb9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd37415d040 0x7fd37415dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.131+0000 7fd378fb9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd37415e2a0 0x7fd37406c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.131+0000 7fd378fb9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3740b2cd0 0x7fd37406ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.131+0000 7fd378fb9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3740cba40 con 0x7fd37415e2a0 2024-04-06T13:47:08.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.131+0000 7fd378fb9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd37415d8d0 con 0x7fd37415d040 2024-04-06T13:47:08.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.131+0000 7fd378fb9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd37415d580 con 0x7fd3740b2cd0 2024-04-06T13:47:08.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.131+0000 7fd371d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd37415e2a0 0x7fd37406c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:08.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.131+0000 7fd371d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd37415e2a0 0x7fd37406c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47090/0 (socket says 172.21.15.189:47090) 2024-04-06T13:47:08.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.131+0000 7fd371d9b700 1 -- 172.21.15.189:0/3434140771 learned_addr learned my addr 172.21.15.189:0/3434140771 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:08.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.131+0000 7fd37259c700 1 --2- 172.21.15.189:0/3434140771 >> v2:172.21.15.192:3300/0 conn(0x7fd37415d040 0x7fd37415dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:08.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.131+0000 7fd372d9d700 1 --2- 172.21.15.189:0/3434140771 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3740b2cd0 0x7fd37406ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:08.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.132+0000 7fd371d9b700 1 -- 172.21.15.189:0/3434140771 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3740b2cd0 msgr2=0x7fd37406ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:08.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.132+0000 7fd371d9b700 1 --2- 172.21.15.189:0/3434140771 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3740b2cd0 0x7fd37406ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.132+0000 7fd371d9b700 1 -- 172.21.15.189:0/3434140771 >> v2:172.21.15.192:3300/0 conn(0x7fd37415d040 msgr2=0x7fd37415dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:08.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.132+0000 7fd371d9b700 1 --2- 172.21.15.189:0/3434140771 >> v2:172.21.15.192:3300/0 conn(0x7fd37415d040 0x7fd37415dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.132+0000 7fd371d9b700 1 -- 172.21.15.189:0/3434140771 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3740b3240 con 0x7fd37415e2a0 2024-04-06T13:47:08.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.132+0000 7fd371d9b700 1 --2- 172.21.15.189:0/3434140771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd37415e2a0 0x7fd37406c4e0 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7fd37406ea40 tx=0x7fd368009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff87d4fd5f452206 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:08.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.132+0000 7fd37159a700 1 -- 172.21.15.189:0/3434140771 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd368011830 con 0x7fd37415e2a0 2024-04-06T13:47:08.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.132+0000 7fd37159a700 1 -- 172.21.15.189:0/3434140771 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd368011990 con 0x7fd37415e2a0 2024-04-06T13:47:08.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.132+0000 7fd37159a700 1 -- 172.21.15.189:0/3434140771 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd368011d20 con 0x7fd37415e2a0 2024-04-06T13:47:08.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.133+0000 7fd378fb9700 1 -- 172.21.15.189:0/3434140771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd37415e2a0 msgr2=0x7fd37406c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:08.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.133+0000 7fd378fb9700 1 --2- 172.21.15.189:0/3434140771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd37415e2a0 0x7fd37406c4e0 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7fd37406ea40 tx=0x7fd368009bd0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.133+0000 7fd378fb9700 1 -- 172.21.15.189:0/3434140771 shutdown_connections 2024-04-06T13:47:08.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.133+0000 7fd378fb9700 1 --2- 172.21.15.189:0/3434140771 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3740b2cd0 0x7fd37406ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.133+0000 7fd378fb9700 1 --2- 172.21.15.189:0/3434140771 >> v2:172.21.15.192:3300/0 conn(0x7fd37415d040 0x7fd37415dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.133+0000 7fd378fb9700 1 --2- 172.21.15.189:0/3434140771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd37415e2a0 0x7fd37406c4e0 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.133+0000 7fd378fb9700 1 -- 172.21.15.189:0/3434140771 >> 172.21.15.189:0/3434140771 conn(0x7fd37406a730 msgr2=0x7fd37406ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:08.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.133+0000 7fd378fb9700 1 -- 172.21.15.189:0/3434140771 shutdown_connections 2024-04-06T13:47:08.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.133+0000 7fd378fb9700 1 -- 172.21.15.189:0/3434140771 wait complete. 2024-04-06T13:47:08.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.134+0000 7fd378fb9700 1 Processor -- start 2024-04-06T13:47:08.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.134+0000 7fd378fb9700 1 -- start start 2024-04-06T13:47:08.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.135+0000 7fd378fb9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3740b2cd0 0x7fd374137500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.135+0000 7fd378fb9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd37415d040 0x7fd374139d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.135+0000 7fd378fb9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd37413a260 0x7fd374137c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.135+0000 7fd378fb9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37413a710 con 0x7fd37415d040 2024-04-06T13:47:08.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.135+0000 7fd378fb9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd374137a40 con 0x7fd37413a260 2024-04-06T13:47:08.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.135+0000 7fd378fb9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3741388e0 con 0x7fd3740b2cd0 2024-04-06T13:47:08.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.135+0000 7fd371d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd37415d040 0x7fd374139d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:08.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.135+0000 7fd371d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd37415d040 0x7fd374139d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47096/0 (socket says 172.21.15.189:47096) 2024-04-06T13:47:08.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.135+0000 7fd371d9b700 1 -- 172.21.15.189:0/2578962262 learned_addr learned my addr 172.21.15.189:0/2578962262 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:08.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.135+0000 7fd37259c700 1 --2- 172.21.15.189:0/2578962262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3740b2cd0 0x7fd374137500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:08.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.135+0000 7fd372d9d700 1 --2- 172.21.15.189:0/2578962262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd37413a260 0x7fd374137c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:08.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.136+0000 7fd371d9b700 1 -- 172.21.15.189:0/2578962262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3740b2cd0 msgr2=0x7fd374137500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:08.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.136+0000 7fd371d9b700 1 --2- 172.21.15.189:0/2578962262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3740b2cd0 0x7fd374137500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.136+0000 7fd371d9b700 1 -- 172.21.15.189:0/2578962262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd37413a260 msgr2=0x7fd374137c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:08.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.136+0000 7fd371d9b700 1 --2- 172.21.15.189:0/2578962262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd37413a260 0x7fd374137c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.136+0000 7fd371d9b700 1 -- 172.21.15.189:0/2578962262 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd368009550 con 0x7fd37415d040 2024-04-06T13:47:08.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.136+0000 7fd371d9b700 1 --2- 172.21.15.189:0/2578962262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd37415d040 0x7fd374139d20 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7fd368009c10 tx=0x7fd368004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:08.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.136+0000 7fd362ffd700 1 -- 172.21.15.189:0/2578962262 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd368004450 con 0x7fd37415d040 2024-04-06T13:47:08.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.136+0000 7fd362ffd700 1 -- 172.21.15.189:0/2578962262 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36801e070 con 0x7fd37415d040 2024-04-06T13:47:08.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.136+0000 7fd378fb9700 1 -- 172.21.15.189:0/2578962262 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd374138bc0 con 0x7fd37415d040 2024-04-06T13:47:08.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.136+0000 7fd362ffd700 1 -- 172.21.15.189:0/2578962262 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3680038e0 con 0x7fd37415d040 2024-04-06T13:47:08.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.136+0000 7fd378fb9700 1 -- 172.21.15.189:0/2578962262 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd374139180 con 0x7fd37415d040 2024-04-06T13:47:08.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.137+0000 7fd378fb9700 1 -- 172.21.15.189:0/2578962262 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd374073270 con 0x7fd37415d040 2024-04-06T13:47:08.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.138+0000 7fd362ffd700 1 -- 172.21.15.189:0/2578962262 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 135) v1 ==== 89361+0+0 (secure 0 0 0) 0x7fd368003a40 con 0x7fd37415d040 2024-04-06T13:47:08.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.138+0000 7fd362ffd700 1 --2- 172.21.15.189:0/2578962262 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd35406aed0 0x7fd35406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.139+0000 7fd37259c700 1 -- 172.21.15.189:0/2578962262 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd35406aed0 msgr2=0x7fd35406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:08.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.139+0000 7fd362ffd700 1 -- 172.21.15.189:0/2578962262 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6604+0+0 (secure 0 0 0) 0x7fd36808a730 con 0x7fd37415d040 2024-04-06T13:47:08.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.139+0000 7fd37259c700 1 --2- 172.21.15.189:0/2578962262 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd35406aed0 0x7fd35406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:08.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.143+0000 7fd362ffd700 1 -- 172.21.15.189:0/2578962262 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd368055310 con 0x7fd37415d040 2024-04-06T13:47:08.299 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:08.298+0000 7fce45124200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:47:08.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.340+0000 7fd37259c700 1 -- 172.21.15.189:0/2578962262 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd35406aed0 msgr2=0x7fd35406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:08.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.340+0000 7fd37259c700 1 --2- 172.21.15.189:0/2578962262 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd35406aed0 0x7fd35406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:08.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.342+0000 7fd378fb9700 1 -- 172.21.15.189:0/2578962262 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd374074990 con 0x7fd37415d040 2024-04-06T13:47:08.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.343+0000 7fd362ffd700 1 -- 172.21.15.189:0/2578962262 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v214) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd368058930 con 0x7fd37415d040 2024-04-06T13:47:08.344 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:47:08.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.345+0000 7fd378fb9700 1 -- 172.21.15.189:0/2578962262 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd35406aed0 msgr2=0x7fd35406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:08.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.345+0000 7fd378fb9700 1 --2- 172.21.15.189:0/2578962262 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd35406aed0 0x7fd35406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.345+0000 7fd378fb9700 1 -- 172.21.15.189:0/2578962262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd37415d040 msgr2=0x7fd374139d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:08.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.345+0000 7fd378fb9700 1 --2- 172.21.15.189:0/2578962262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd37415d040 0x7fd374139d20 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7fd368009c10 tx=0x7fd368004750 comp rx=0 tx=0).stop 2024-04-06T13:47:08.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.345+0000 7fd378fb9700 1 -- 172.21.15.189:0/2578962262 shutdown_connections 2024-04-06T13:47:08.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.345+0000 7fd378fb9700 1 --2- 172.21.15.189:0/2578962262 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd35406aed0 0x7fd35406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.345+0000 7fd378fb9700 1 --2- 172.21.15.189:0/2578962262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd37413a260 0x7fd374137c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.345+0000 7fd378fb9700 1 --2- 172.21.15.189:0/2578962262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3740b2cd0 0x7fd374137500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.345+0000 7fd378fb9700 1 --2- 172.21.15.189:0/2578962262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd37415d040 0x7fd374139d20 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.345+0000 7fd378fb9700 1 -- 172.21.15.189:0/2578962262 >> 172.21.15.189:0/2578962262 conn(0x7fd37406a730 msgr2=0x7fd3740b0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:08.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.345+0000 7fd378fb9700 1 -- 172.21.15.189:0/2578962262 shutdown_connections 2024-04-06T13:47:08.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.345+0000 7fd378fb9700 1 -- 172.21.15.189:0/2578962262 wait complete. 2024-04-06T13:47:08.359 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T13:47:08.452 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:08.452+0000 7fce45124200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:47:08.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.496+0000 7fa4a7c54700 1 Processor -- start 2024-04-06T13:47:08.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.496+0000 7fa4a7c54700 1 -- start start 2024-04-06T13:47:08.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a7c54700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa4a00cd770 0x7fa4a00b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a7c54700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4a00b3200 0x7fa4a00b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a7c54700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a00b0e50 con 0x7fa4a0159f10 2024-04-06T13:47:08.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a7c54700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a00b5430 con 0x7fa4a00cd770 2024-04-06T13:47:08.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a7c54700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a00b2680 con 0x7fa4a00b3200 2024-04-06T13:47:08.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a59f0700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa4a0159f10 0x7fa4a00cd060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42694/0 (socket says 172.21.15.189:42694) 2024-04-06T13:47:08.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a59f0700 1 -- 172.21.15.189:0/2079246202 learned_addr learned my addr 172.21.15.189:0/2079246202 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:08.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a61f1700 1 --2- 172.21.15.189:0/2079246202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4a00b3200 0x7fa4a00b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:08.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a51ef700 1 --2- 172.21.15.189:0/2079246202 >> v2:172.21.15.192:3300/0 conn(0x7fa4a00cd770 0x7fa4a00b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:08.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a49ee700 1 -- 172.21.15.189:0/2079246202 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752101298 0 0) 0x7fa4a00b0e50 con 0x7fa4a0159f10 2024-04-06T13:47:08.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a49ee700 1 -- 172.21.15.189:0/2079246202 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa48c0035b0 con 0x7fa4a0159f10 2024-04-06T13:47:08.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a49ee700 1 -- 172.21.15.189:0/2079246202 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2133666355 0 0) 0x7fa48c0035b0 con 0x7fa4a0159f10 2024-04-06T13:47:08.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a49ee700 1 -- 172.21.15.189:0/2079246202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4a00b3200 msgr2=0x7fa4a00b36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:08.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a49ee700 1 --2- 172.21.15.189:0/2079246202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4a00b3200 0x7fa4a00b36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a49ee700 1 -- 172.21.15.189:0/2079246202 >> v2:172.21.15.192:3300/0 conn(0x7fa4a00cd770 msgr2=0x7fa4a00b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:08.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a49ee700 1 --2- 172.21.15.189:0/2079246202 >> v2:172.21.15.192:3300/0 conn(0x7fa4a00cd770 0x7fa4a00b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a49ee700 1 -- 172.21.15.189:0/2079246202 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a00b3bf0 con 0x7fa4a0159f10 2024-04-06T13:47:08.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a49ee700 1 -- 172.21.15.189:0/2079246202 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fa49c001670 con 0x7fa4a0159f10 2024-04-06T13:47:08.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a51ef700 1 --2- 172.21.15.189:0/2079246202 >> v2:172.21.15.192:3300/0 conn(0x7fa4a00cd770 0x7fa4a00b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:08.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a61f1700 1 --2- 172.21.15.189:0/2079246202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4a00b3200 0x7fa4a00b36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:08.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a49ee700 1 -- 172.21.15.189:0/2079246202 >> v1:172.21.15.189:6789/0 conn(0x7fa4a0159f10 legacy=0x7fa4a00cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:08.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a49ee700 1 --2- 172.21.15.189:0/2079246202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa48c003a00 0x7fa48c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a49ee700 1 --2- 172.21.15.189:0/2079246202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4a00b3200 0x7fa48c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.497+0000 7fa4a49ee700 1 --2- 172.21.15.189:0/2079246202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa48c004d90 0x7fa48c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.498+0000 7fa4a49ee700 1 -- 172.21.15.189:0/2079246202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a00b5430 con 0x7fa48c003a00 2024-04-06T13:47:08.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.498+0000 7fa4a49ee700 1 -- 172.21.15.189:0/2079246202 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a00b2680 con 0x7fa48c004d90 2024-04-06T13:47:08.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.498+0000 7fa4a49ee700 1 -- 172.21.15.189:0/2079246202 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c00cb20 con 0x7fa4a00b3200 2024-04-06T13:47:08.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.498+0000 7fa4a59f0700 1 --2- 172.21.15.189:0/2079246202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa48c003a00 0x7fa48c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:08.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.498+0000 7fa4a51ef700 1 --2- 172.21.15.189:0/2079246202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4a00b3200 0x7fa48c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:08.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.498+0000 7fa4a61f1700 1 --2- 172.21.15.189:0/2079246202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa48c004d90 0x7fa48c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:08.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.498+0000 7fa4a59f0700 1 -- 172.21.15.189:0/2079246202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4a00b3200 msgr2=0x7fa48c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:08.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.498+0000 7fa4a59f0700 1 --2- 172.21.15.189:0/2079246202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4a00b3200 0x7fa48c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.498+0000 7fa4a59f0700 1 -- 172.21.15.189:0/2079246202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa48c004d90 msgr2=0x7fa48c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:08.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.498+0000 7fa4a59f0700 1 --2- 172.21.15.189:0/2079246202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa48c004d90 0x7fa48c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.498+0000 7fa4a59f0700 1 -- 172.21.15.189:0/2079246202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa498009a70 con 0x7fa48c003a00 2024-04-06T13:47:08.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.498+0000 7fa4a59f0700 1 --2- 172.21.15.189:0/2079246202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa48c003a00 0x7fa48c003e20 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00c320 tx=0x7fa48c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b619ef55e91398 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:08.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.498+0000 7fa4a49ee700 1 -- 172.21.15.189:0/2079246202 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c00ce60 con 0x7fa48c003a00 2024-04-06T13:47:08.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.498+0000 7fa4a49ee700 1 -- 172.21.15.189:0/2079246202 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49c005960 con 0x7fa48c003a00 2024-04-06T13:47:08.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.498+0000 7fa4a49ee700 1 -- 172.21.15.189:0/2079246202 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c005cf0 con 0x7fa48c003a00 2024-04-06T13:47:08.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.499+0000 7fa4a7c54700 1 -- 172.21.15.189:0/2079246202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa48c003a00 msgr2=0x7fa48c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:08.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.499+0000 7fa4a7c54700 1 --2- 172.21.15.189:0/2079246202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa48c003a00 0x7fa48c003e20 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00c320 tx=0x7fa48c005880 comp rx=0 tx=0).stop 2024-04-06T13:47:08.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.499+0000 7fa4a61f1700 1 -- 172.21.15.189:0/2079246202 reap_dead start 2024-04-06T13:47:08.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.499+0000 7fa4a7c54700 1 -- 172.21.15.189:0/2079246202 shutdown_connections 2024-04-06T13:47:08.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.499+0000 7fa4a7c54700 1 -- 172.21.15.189:0/2079246202 >> 172.21.15.189:0/2079246202 conn(0x7fa4a00b0fe0 msgr2=0x7fa4a00b13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:08.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.499+0000 7fa4a7c54700 1 -- 172.21.15.189:0/2079246202 shutdown_connections 2024-04-06T13:47:08.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.499+0000 7fa4a7c54700 1 -- 172.21.15.189:0/2079246202 wait complete. 2024-04-06T13:47:08.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.500+0000 7fa4a7c54700 1 Processor -- start 2024-04-06T13:47:08.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.500+0000 7fa4a7c54700 1 -- start start 2024-04-06T13:47:08.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.501+0000 7fa4a7c54700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa48c003a00 0x7fa4a0154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.501+0000 7fa4a7c54700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a015c500 0x7fa4a00993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.501+0000 7fa4a7c54700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4a015c980 0x7fa4a0099910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.501+0000 7fa4a7c54700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a01f1240 con 0x7fa4a015c500 2024-04-06T13:47:08.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.501+0000 7fa4a7c54700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a01f1520 con 0x7fa4a015c980 2024-04-06T13:47:08.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.501+0000 7fa4a7c54700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a01f1800 con 0x7fa48c003a00 2024-04-06T13:47:08.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.501+0000 7fa4a51ef700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a015c500 0x7fa4a00993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:08.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.501+0000 7fa4a51ef700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a015c500 0x7fa4a00993d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47110/0 (socket says 172.21.15.189:47110) 2024-04-06T13:47:08.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.501+0000 7fa4a51ef700 1 -- 172.21.15.189:0/2305002856 learned_addr learned my addr 172.21.15.189:0/2305002856 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:08.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.501+0000 7fa4a59f0700 1 --2- 172.21.15.189:0/2305002856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa48c003a00 0x7fa4a0154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:08.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.501+0000 7fa4a61f1700 1 --2- 172.21.15.189:0/2305002856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4a015c980 0x7fa4a0099910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:08.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.501+0000 7fa4a51ef700 1 -- 172.21.15.189:0/2305002856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa48c003a00 msgr2=0x7fa4a0154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:08.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.501+0000 7fa4a51ef700 1 --2- 172.21.15.189:0/2305002856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa48c003a00 0x7fa4a0154300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.501+0000 7fa4a51ef700 1 -- 172.21.15.189:0/2305002856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4a015c980 msgr2=0x7fa4a0099910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:08.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.501+0000 7fa4a51ef700 1 --2- 172.21.15.189:0/2305002856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4a015c980 0x7fa4a0099910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.501+0000 7fa4a51ef700 1 -- 172.21.15.189:0/2305002856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a01f1ae0 con 0x7fa4a015c500 2024-04-06T13:47:08.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.501+0000 7fa4a51ef700 1 --2- 172.21.15.189:0/2305002856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a015c500 0x7fa4a00993d0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7fa490011cd0 tx=0x7fa490009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:08.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.502+0000 7fa4967fc700 1 -- 172.21.15.189:0/2305002856 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49000fa00 con 0x7fa4a015c500 2024-04-06T13:47:08.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.502+0000 7fa4967fc700 1 -- 172.21.15.189:0/2305002856 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49000fb60 con 0x7fa4a015c500 2024-04-06T13:47:08.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.502+0000 7fa4a7c54700 1 -- 172.21.15.189:0/2305002856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a01f1dc0 con 0x7fa4a015c500 2024-04-06T13:47:08.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.502+0000 7fa4967fc700 1 -- 172.21.15.189:0/2305002856 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa490004030 con 0x7fa4a015c500 2024-04-06T13:47:08.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.502+0000 7fa4a7c54700 1 -- 172.21.15.189:0/2305002856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a0154a20 con 0x7fa4a015c500 2024-04-06T13:47:08.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.503+0000 7fa4a7c54700 1 -- 172.21.15.189:0/2305002856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a0074990 con 0x7fa4a015c500 2024-04-06T13:47:08.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.504+0000 7fa4967fc700 1 -- 172.21.15.189:0/2305002856 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 135) v1 ==== 89361+0+0 (secure 0 0 0) 0x7fa490004190 con 0x7fa4a015c500 2024-04-06T13:47:08.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.504+0000 7fa4967fc700 1 --2- 172.21.15.189:0/2305002856 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa48406aed0 0x7fa48406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.504+0000 7fa4a59f0700 1 -- 172.21.15.189:0/2305002856 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa48406aed0 msgr2=0x7fa48406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:08.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.504+0000 7fa4967fc700 1 -- 172.21.15.189:0/2305002856 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6604+0+0 (secure 0 0 0) 0x7fa49008a2f0 con 0x7fa4a015c500 2024-04-06T13:47:08.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.504+0000 7fa4a59f0700 1 --2- 172.21.15.189:0/2305002856 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa48406aed0 0x7fa48406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:08.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.508+0000 7fa4967fc700 1 -- 172.21.15.189:0/2305002856 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa490054ed0 con 0x7fa4a015c500 2024-04-06T13:47:08.517 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:08.510+0000 7fbb7ac74200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:47:08.608 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:08.607+0000 7fce45124200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:47:08.675 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:08.674+0000 7fbb7ac74200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:47:08.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.705+0000 7fa4a59f0700 1 -- 172.21.15.189:0/2305002856 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa48406aed0 msgr2=0x7fa48406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:08.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.705+0000 7fa4a59f0700 1 --2- 172.21.15.189:0/2305002856 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa48406aed0 0x7fa48406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:08.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.711+0000 7fa4a7c54700 1 -- 172.21.15.189:0/2305002856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa4a00b5210 con 0x7fa4a015c500 2024-04-06T13:47:08.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.711+0000 7fa4967fc700 1 -- 172.21.15.189:0/2305002856 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v214) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa490022020 con 0x7fa4a015c500 2024-04-06T13:47:08.712 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:47:08.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.713+0000 7fa4a7c54700 1 -- 172.21.15.189:0/2305002856 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa48406aed0 msgr2=0x7fa48406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:08.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.713+0000 7fa4a7c54700 1 --2- 172.21.15.189:0/2305002856 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa48406aed0 0x7fa48406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.713+0000 7fa4a7c54700 1 -- 172.21.15.189:0/2305002856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a015c500 msgr2=0x7fa4a00993d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.713+0000 7fa4a7c54700 1 --2- 172.21.15.189:0/2305002856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a015c500 0x7fa4a00993d0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7fa490011cd0 tx=0x7fa490009ed0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.713+0000 7fa4a7c54700 1 -- 172.21.15.189:0/2305002856 shutdown_connections 2024-04-06T13:47:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.713+0000 7fa4a7c54700 1 --2- 172.21.15.189:0/2305002856 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa48406aed0 0x7fa48406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.713+0000 7fa4a7c54700 1 --2- 172.21.15.189:0/2305002856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4a015c980 0x7fa4a0099910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.713+0000 7fa4a7c54700 1 --2- 172.21.15.189:0/2305002856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa48c003a00 0x7fa4a0154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.713+0000 7fa4a7c54700 1 --2- 172.21.15.189:0/2305002856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a015c500 0x7fa4a00993d0 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.713+0000 7fa4a7c54700 1 -- 172.21.15.189:0/2305002856 >> 172.21.15.189:0/2305002856 conn(0x7fa4a00b0fe0 msgr2=0x7fa4a0158460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.713+0000 7fa4a7c54700 1 -- 172.21.15.189:0/2305002856 shutdown_connections 2024-04-06T13:47:08.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.713+0000 7fa4a7c54700 1 -- 172.21.15.189:0/2305002856 wait complete. 2024-04-06T13:47:08.728 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:47:08.845 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:08.844+0000 7fbb7ac74200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:47:08.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.857+0000 7f1b9c87b700 1 Processor -- start 2024-04-06T13:47:08.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.857+0000 7f1b9c87b700 1 -- start start 2024-04-06T13:47:08.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.858+0000 7f1b9c87b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1b980d0a60 0x7f1b980d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.858+0000 7f1b9c87b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b980cdf80 con 0x7f1b980d1af0 2024-04-06T13:47:08.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.858+0000 7f1b9c87b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b980d1f60 con 0x7f1b9815d7e0 2024-04-06T13:47:08.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.858+0000 7f1b9c87b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b980d3230 con 0x7f1b980d0a60 2024-04-06T13:47:08.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.858+0000 7f1b95d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1b980d0a60 0x7f1b980d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:08.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.858+0000 7f1b96d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1b980d1af0 0x7f1b980b66b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42704/0 (socket says 172.21.15.189:42704) 2024-04-06T13:47:08.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.858+0000 7f1b96d9d700 1 -- 172.21.15.189:0/835331360 learned_addr learned my addr 172.21.15.189:0/835331360 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:08.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.858+0000 7f1b95d9b700 1 -- 172.21.15.189:0/835331360 >> v1:172.21.15.192:6789/0 conn(0x7f1b9815d7e0 legacy=0x7f1b980d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:08.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.858+0000 7f1b95d9b700 1 -- 172.21.15.189:0/835331360 >> v1:172.21.15.189:6789/0 conn(0x7f1b980d1af0 legacy=0x7f1b980b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:08.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.858+0000 7f1b95d9b700 1 -- 172.21.15.189:0/835331360 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b980d2da0 con 0x7f1b980d0a60 2024-04-06T13:47:08.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.858+0000 7f1b9559a700 1 -- 172.21.15.189:0/835331360 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470099406 0 0) 0x7f1b980cdf80 con 0x7f1b980d1af0 2024-04-06T13:47:08.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.858+0000 7f1b95d9b700 1 --2- 172.21.15.189:0/835331360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b980d0a60 0x7f1b980d15b0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c009840 tx=0x7f1b8c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10dc3a7791945ffd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:08.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.859+0000 7f1b9559a700 1 -- 172.21.15.189:0/835331360 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c011830 con 0x7f1b980d0a60 2024-04-06T13:47:08.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.859+0000 7f1b9559a700 1 -- 172.21.15.189:0/835331360 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8c011990 con 0x7f1b980d0a60 2024-04-06T13:47:08.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.859+0000 7f1b9559a700 1 -- 172.21.15.189:0/835331360 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c011d20 con 0x7f1b980d0a60 2024-04-06T13:47:08.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.859+0000 7f1b9c87b700 1 -- 172.21.15.189:0/835331360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b980d0a60 msgr2=0x7f1b980d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:08.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.859+0000 7f1b9c87b700 1 --2- 172.21.15.189:0/835331360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b980d0a60 0x7f1b980d15b0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c009840 tx=0x7f1b8c009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:08.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.860+0000 7f1b9c87b700 1 -- 172.21.15.189:0/835331360 shutdown_connections 2024-04-06T13:47:08.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.860+0000 7f1b9c87b700 1 --2- 172.21.15.189:0/835331360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b980d0a60 0x7f1b980d15b0 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.860+0000 7f1b9c87b700 1 -- 172.21.15.189:0/835331360 >> 172.21.15.189:0/835331360 conn(0x7f1b98072250 msgr2=0x7f1b98072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:08.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.860+0000 7f1b9c87b700 1 -- 172.21.15.189:0/835331360 shutdown_connections 2024-04-06T13:47:08.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.860+0000 7f1b9c87b700 1 -- 172.21.15.189:0/835331360 wait complete. 2024-04-06T13:47:08.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.861+0000 7f1b9c87b700 1 Processor -- start 2024-04-06T13:47:08.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.861+0000 7f1b9c87b700 1 -- start start 2024-04-06T13:47:08.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b9c87b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b980d0a60 0x7f1b9815c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b9c87b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b980d1af0 0x7f1b981e5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b9c87b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b9815d7e0 0x7f1b981e6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b9c87b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b981f3620 con 0x7f1b9815d7e0 2024-04-06T13:47:08.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b9c87b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b980cc840 con 0x7f1b980d1af0 2024-04-06T13:47:08.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b9c87b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b980ccb20 con 0x7f1b980d0a60 2024-04-06T13:47:08.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b96d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b9815d7e0 0x7f1b981e6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:08.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b96d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b9815d7e0 0x7f1b981e6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47116/0 (socket says 172.21.15.189:47116) 2024-04-06T13:47:08.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b96d9d700 1 -- 172.21.15.189:0/4034656541 learned_addr learned my addr 172.21.15.189:0/4034656541 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:08.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b9659c700 1 --2- 172.21.15.189:0/4034656541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b980d0a60 0x7f1b9815c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:08.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b95d9b700 1 --2- 172.21.15.189:0/4034656541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b980d1af0 0x7f1b981e5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:08.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b96d9d700 1 -- 172.21.15.189:0/4034656541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b980d0a60 msgr2=0x7f1b9815c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:08.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b96d9d700 1 --2- 172.21.15.189:0/4034656541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b980d0a60 0x7f1b9815c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b96d9d700 1 -- 172.21.15.189:0/4034656541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b980d1af0 msgr2=0x7f1b981e5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:08.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b96d9d700 1 --2- 172.21.15.189:0/4034656541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b980d1af0 0x7f1b981e5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:08.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b96d9d700 1 -- 172.21.15.189:0/4034656541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8c0094a0 con 0x7f1b9815d7e0 2024-04-06T13:47:08.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b96d9d700 1 --2- 172.21.15.189:0/4034656541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b9815d7e0 0x7f1b981e6130 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f1b9000cca0 tx=0x7f1b9000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:08.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b86ffd700 1 -- 172.21.15.189:0/4034656541 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9000ab70 con 0x7f1b9815d7e0 2024-04-06T13:47:08.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b86ffd700 1 -- 172.21.15.189:0/4034656541 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b9000acd0 con 0x7f1b9815d7e0 2024-04-06T13:47:08.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b9c87b700 1 -- 172.21.15.189:0/4034656541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b980cce00 con 0x7f1b9815d7e0 2024-04-06T13:47:08.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b86ffd700 1 -- 172.21.15.189:0/4034656541 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b90007310 con 0x7f1b9815d7e0 2024-04-06T13:47:08.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.862+0000 7f1b9c87b700 1 -- 172.21.15.189:0/4034656541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b9815c840 con 0x7f1b9815d7e0 2024-04-06T13:47:08.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.864+0000 7f1b9c87b700 1 -- 172.21.15.189:0/4034656541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b981f3790 con 0x7f1b9815d7e0 2024-04-06T13:47:08.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.864+0000 7f1b86ffd700 1 -- 172.21.15.189:0/4034656541 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 135) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f1b90007470 con 0x7f1b9815d7e0 2024-04-06T13:47:08.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.864+0000 7f1b86ffd700 1 --2- 172.21.15.189:0/4034656541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1b7806aed0 0x7f1b7806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:08.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.864+0000 7f1b9659c700 1 -- 172.21.15.189:0/4034656541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1b7806aed0 msgr2=0x7f1b7806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:08.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.864+0000 7f1b86ffd700 1 -- 172.21.15.189:0/4034656541 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6604+0+0 (secure 0 0 0) 0x7f1b900898b0 con 0x7f1b9815d7e0 2024-04-06T13:47:08.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.864+0000 7f1b9659c700 1 --2- 172.21.15.189:0/4034656541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1b7806aed0 0x7f1b7806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:08.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:08.869+0000 7f1b86ffd700 1 -- 172.21.15.189:0/4034656541 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1b900543b0 con 0x7f1b9815d7e0 2024-04-06T13:47:09.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.065+0000 7f1b9659c700 1 -- 172.21.15.189:0/4034656541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1b7806aed0 msgr2=0x7f1b7806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:09.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.065+0000 7f1b9659c700 1 --2- 172.21.15.189:0/4034656541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1b7806aed0 0x7f1b7806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:09.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.116+0000 7f1b9c87b700 1 -- 172.21.15.189:0/4034656541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b98074990 con 0x7f1b9815d7e0 2024-04-06T13:47:09.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.117+0000 7f1b86ffd700 1 -- 172.21.15.189:0/4034656541 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+4750 (secure 0 0 0) 0x7f1b900579d0 con 0x7f1b9815d7e0 2024-04-06T13:47:09.119 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:09.119 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":93,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8525,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3302262361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3302262361},{"type":"v1","addr":"172.21.15.189:6837","nonce":3302262361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":8534,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/229523550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":229523550},{"type":"v1","addr":"172.21.15.192:6834","nonce":229523550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":9739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2560949338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2560949338},{"type":"v1","addr":"172.21.15.192:6835","nonce":2560949338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:45:40.072274+0000","modified":"2024-04-06T13:45:43.092032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":0,"incarnation":89,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/3619731181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3619731181},{"type":"v1","addr":"172.21.15.189:6836","nonce":3619731181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-04-06T13:47:09.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.119+0000 7f1b9c87b700 1 -- 172.21.15.189:0/4034656541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1b7806aed0 msgr2=0x7f1b7806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:09.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.119+0000 7f1b9c87b700 1 --2- 172.21.15.189:0/4034656541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1b7806aed0 0x7f1b7806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.119+0000 7f1b9c87b700 1 -- 172.21.15.189:0/4034656541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b9815d7e0 msgr2=0x7f1b981e6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:09.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.119+0000 7f1b9c87b700 1 --2- 172.21.15.189:0/4034656541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b9815d7e0 0x7f1b981e6130 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f1b9000cca0 tx=0x7f1b9000a060 comp rx=0 tx=0).stop 2024-04-06T13:47:09.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.119+0000 7f1b9c87b700 1 -- 172.21.15.189:0/4034656541 shutdown_connections 2024-04-06T13:47:09.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.119+0000 7f1b9c87b700 1 --2- 172.21.15.189:0/4034656541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1b7806aed0 0x7f1b7806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.119+0000 7f1b9c87b700 1 --2- 172.21.15.189:0/4034656541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b9815d7e0 0x7f1b981e6130 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.119+0000 7f1b9c87b700 1 --2- 172.21.15.189:0/4034656541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b980d0a60 0x7f1b9815c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.119+0000 7f1b9c87b700 1 --2- 172.21.15.189:0/4034656541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b980d1af0 0x7f1b981e5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.119+0000 7f1b9c87b700 1 -- 172.21.15.189:0/4034656541 >> 172.21.15.189:0/4034656541 conn(0x7f1b98072250 msgr2=0x7f1b980b86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:09.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.119+0000 7f1b9c87b700 1 -- 172.21.15.189:0/4034656541 shutdown_connections 2024-04-06T13:47:09.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.119+0000 7f1b9c87b700 1 -- 172.21.15.189:0/4034656541 wait complete. 2024-04-06T13:47:09.123 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 93 2024-04-06T13:47:09.133 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:47:09.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.259+0000 7f2a6f2f9700 1 Processor -- start 2024-04-06T13:47:09.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.260+0000 7f2a6f2f9700 1 -- start start 2024-04-06T13:47:09.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.260+0000 7f2a6f2f9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a6806ae40 0x7f2a6815ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:09.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.260+0000 7f2a6f2f9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2a6815fcd0 0x7f2a6815bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:09.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.260+0000 7f2a6f2f9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a6815c4c0 con 0x7f2a68144f40 2024-04-06T13:47:09.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.260+0000 7f2a6f2f9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a680cfcb0 con 0x7f2a6815fcd0 2024-04-06T13:47:09.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.260+0000 7f2a6f2f9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6815ca20 con 0x7f2a6806ae40 2024-04-06T13:47:09.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.261+0000 7f2a6d896700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2a6815fcd0 0x7f2a6815bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:09.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.261+0000 7f2a6d095700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2a68144f40 0x7f2a6806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42712/0 (socket says 172.21.15.189:42712) 2024-04-06T13:47:09.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.261+0000 7f2a6d095700 1 -- 172.21.15.189:0/3191994797 learned_addr learned my addr 172.21.15.189:0/3191994797 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:09.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.261+0000 7f2a6c894700 1 --2- 172.21.15.189:0/3191994797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a6806ae40 0x7f2a6815ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:09.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.261+0000 7f2a6d896700 1 -- 172.21.15.189:0/3191994797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a6806ae40 msgr2=0x7f2a6815ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:09.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.261+0000 7f2a6d896700 1 --2- 172.21.15.189:0/3191994797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a6806ae40 0x7f2a6815ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.261+0000 7f2a6d896700 1 -- 172.21.15.189:0/3191994797 >> v1:172.21.15.189:6789/0 conn(0x7f2a68144f40 legacy=0x7f2a6806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:09.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.261+0000 7f2a5ffff700 1 -- 172.21.15.189:0/3191994797 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639631574 0 0) 0x7f2a6815c4c0 con 0x7f2a68144f40 2024-04-06T13:47:09.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.261+0000 7f2a6d896700 1 -- 172.21.15.189:0/3191994797 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a6815c740 con 0x7f2a6815fcd0 2024-04-06T13:47:09.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.261+0000 7f2a6d896700 1 --2- 172.21.15.189:0/3191994797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a6815fcd0 0x7f2a6815bf80 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7f2a60009d90 tx=0x7f2a6000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c157e01c5b1872e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:09.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.261+0000 7f2a5ffff700 1 -- 172.21.15.189:0/3191994797 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6000e070 con 0x7f2a6815fcd0 2024-04-06T13:47:09.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.261+0000 7f2a5ffff700 1 -- 172.21.15.189:0/3191994797 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6000bb20 con 0x7f2a6815fcd0 2024-04-06T13:47:09.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.261+0000 7f2a5ffff700 1 -- 172.21.15.189:0/3191994797 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a60012430 con 0x7f2a6815fcd0 2024-04-06T13:47:09.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.262+0000 7f2a6f2f9700 1 -- 172.21.15.189:0/3191994797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a6815fcd0 msgr2=0x7f2a6815bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:09.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.262+0000 7f2a6f2f9700 1 --2- 172.21.15.189:0/3191994797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a6815fcd0 0x7f2a6815bf80 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7f2a60009d90 tx=0x7f2a6000b330 comp rx=0 tx=0).stop 2024-04-06T13:47:09.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.262+0000 7f2a6f2f9700 1 -- 172.21.15.189:0/3191994797 shutdown_connections 2024-04-06T13:47:09.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.262+0000 7f2a6f2f9700 1 --2- 172.21.15.189:0/3191994797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a6815fcd0 0x7f2a6815bf80 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.262+0000 7f2a6f2f9700 1 --2- 172.21.15.189:0/3191994797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a6806ae40 0x7f2a6815ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.262+0000 7f2a6f2f9700 1 -- 172.21.15.189:0/3191994797 >> 172.21.15.189:0/3191994797 conn(0x7f2a680d00b0 msgr2=0x7f2a680d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:09.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.262+0000 7f2a6f2f9700 1 -- 172.21.15.189:0/3191994797 shutdown_connections 2024-04-06T13:47:09.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.262+0000 7f2a6f2f9700 1 -- 172.21.15.189:0/3191994797 wait complete. 2024-04-06T13:47:09.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.263+0000 7f2a6f2f9700 1 Processor -- start 2024-04-06T13:47:09.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.263+0000 7f2a6f2f9700 1 -- start start 2024-04-06T13:47:09.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.264+0000 7f2a6f2f9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a6806ae40 0x7f2a680b3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:09.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.264+0000 7f2a6f2f9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a68144f40 0x7f2a68143390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:09.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.264+0000 7f2a6f2f9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a6815fcd0 0x7f2a68159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:09.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.264+0000 7f2a6f2f9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a681438d0 con 0x7f2a6806ae40 2024-04-06T13:47:09.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.264+0000 7f2a6f2f9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6813eb50 con 0x7f2a6815fcd0 2024-04-06T13:47:09.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.264+0000 7f2a6f2f9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a681425d0 con 0x7f2a68144f40 2024-04-06T13:47:09.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.264+0000 7f2a6d095700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a6806ae40 0x7f2a680b3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:09.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.264+0000 7f2a6d095700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a6806ae40 0x7f2a680b3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47134/0 (socket says 172.21.15.189:47134) 2024-04-06T13:47:09.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.264+0000 7f2a6d095700 1 -- 172.21.15.189:0/1547038925 learned_addr learned my addr 172.21.15.189:0/1547038925 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:09.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.264+0000 7f2a6c894700 1 --2- 172.21.15.189:0/1547038925 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a68144f40 0x7f2a68143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:09.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.264+0000 7f2a6d896700 1 --2- 172.21.15.189:0/1547038925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a6815fcd0 0x7f2a68159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:09.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.264+0000 7f2a6d095700 1 -- 172.21.15.189:0/1547038925 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a68144f40 msgr2=0x7f2a68143390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:09.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.264+0000 7f2a6d095700 1 --2- 172.21.15.189:0/1547038925 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a68144f40 0x7f2a68143390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.264+0000 7f2a6d095700 1 -- 172.21.15.189:0/1547038925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a6815fcd0 msgr2=0x7f2a68159420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:09.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.264+0000 7f2a6d095700 1 --2- 172.21.15.189:0/1547038925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a6815fcd0 0x7f2a68159420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.264+0000 7f2a6d095700 1 -- 172.21.15.189:0/1547038925 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a60009a70 con 0x7f2a6806ae40 2024-04-06T13:47:09.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.265+0000 7f2a6d095700 1 --2- 172.21.15.189:0/1547038925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a6806ae40 0x7f2a680b3ad0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f2a640011e0 tx=0x7f2a6400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:09.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.265+0000 7f2a5dffb700 1 -- 172.21.15.189:0/1547038925 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64004c80 con 0x7f2a6806ae40 2024-04-06T13:47:09.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.265+0000 7f2a5dffb700 1 -- 172.21.15.189:0/1547038925 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a64004e00 con 0x7f2a6806ae40 2024-04-06T13:47:09.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.265+0000 7f2a5dffb700 1 -- 172.21.15.189:0/1547038925 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64007310 con 0x7f2a6806ae40 2024-04-06T13:47:09.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.265+0000 7f2a6f2f9700 1 -- 172.21.15.189:0/1547038925 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a681428b0 con 0x7f2a6806ae40 2024-04-06T13:47:09.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.265+0000 7f2a6f2f9700 1 -- 172.21.15.189:0/1547038925 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a68142e70 con 0x7f2a6806ae40 2024-04-06T13:47:09.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.266+0000 7f2a6f2f9700 1 -- 172.21.15.189:0/1547038925 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a68067bb0 con 0x7f2a6806ae40 2024-04-06T13:47:09.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.267+0000 7f2a5dffb700 1 -- 172.21.15.189:0/1547038925 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 135) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f2a64007470 con 0x7f2a6806ae40 2024-04-06T13:47:09.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.267+0000 7f2a5dffb700 1 --2- 172.21.15.189:0/1547038925 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2a4c06af00 0x7f2a4c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:09.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.267+0000 7f2a6c894700 1 -- 172.21.15.189:0/1547038925 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2a4c06af00 msgr2=0x7f2a4c06d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:09.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.267+0000 7f2a5dffb700 1 -- 172.21.15.189:0/1547038925 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6604+0+0 (secure 0 0 0) 0x7f2a64089710 con 0x7f2a6806ae40 2024-04-06T13:47:09.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.267+0000 7f2a6c894700 1 --2- 172.21.15.189:0/1547038925 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2a4c06af00 0x7f2a4c06d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:09.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.276+0000 7f2a5dffb700 1 -- 172.21.15.189:0/1547038925 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2a64054220 con 0x7f2a6806ae40 2024-04-06T13:47:09.328 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:09.328+0000 7fce45124200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:47:09.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.468+0000 7f2a6c894700 1 -- 172.21.15.189:0/1547038925 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2a4c06af00 msgr2=0x7f2a4c06d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:09.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.468+0000 7f2a6c894700 1 --2- 172.21.15.189:0/1547038925 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2a4c06af00 0x7f2a4c06d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:09.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.479+0000 7f2a6f2f9700 1 -- 172.21.15.189:0/1547038925 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a68143090 con 0x7f2a6806ae40 2024-04-06T13:47:09.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.480+0000 7f2a5dffb700 1 -- 172.21.15.189:0/1547038925 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v214) v1 ==== 74+0+17252 (secure 0 0 0) 0x7f2a64057840 con 0x7f2a6806ae40 2024-04-06T13:47:09.482 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:09.482 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":214,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:47:06.599488+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:45:37.837008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-04-06T13:45:39.030714+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":214,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.22","mappings":[{"from":1,"to":0}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]},{"pgid":"26.2c","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/509530111":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/3787698124":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/2802982514":"2024-04-07T13:45:54.091329+0000","172.21.15.189:6833/111704":"2024-04-07T13:45:54.091329+0000","172.21.15.189:6832/111704":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/606015831":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/4174593093":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/3225575705":"2024-04-07T13:45:54.091329+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":26,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:47:09.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.483+0000 7f2a6f2f9700 1 -- 172.21.15.189:0/1547038925 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2a4c06af00 msgr2=0x7f2a4c06d3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:09.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.483+0000 7f2a6f2f9700 1 --2- 172.21.15.189:0/1547038925 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2a4c06af00 0x7f2a4c06d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.483+0000 7f2a6f2f9700 1 -- 172.21.15.189:0/1547038925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a6806ae40 msgr2=0x7f2a680b3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:09.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.483+0000 7f2a6f2f9700 1 --2- 172.21.15.189:0/1547038925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a6806ae40 0x7f2a680b3ad0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f2a640011e0 tx=0x7f2a6400ca30 comp rx=0 tx=0).stop 2024-04-06T13:47:09.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.483+0000 7f2a6f2f9700 1 -- 172.21.15.189:0/1547038925 shutdown_connections 2024-04-06T13:47:09.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.483+0000 7f2a6f2f9700 1 --2- 172.21.15.189:0/1547038925 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2a4c06af00 0x7f2a4c06d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.483+0000 7f2a6f2f9700 1 --2- 172.21.15.189:0/1547038925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a6815fcd0 0x7f2a68159420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.483+0000 7f2a6f2f9700 1 --2- 172.21.15.189:0/1547038925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a6806ae40 0x7f2a680b3ad0 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.483+0000 7f2a6f2f9700 1 --2- 172.21.15.189:0/1547038925 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a68144f40 0x7f2a68143390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.483+0000 7f2a6f2f9700 1 -- 172.21.15.189:0/1547038925 >> 172.21.15.189:0/1547038925 conn(0x7f2a680d00b0 msgr2=0x7f2a6806de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:09.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.483+0000 7f2a6f2f9700 1 -- 172.21.15.189:0/1547038925 shutdown_connections 2024-04-06T13:47:09.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.483+0000 7f2a6f2f9700 1 -- 172.21.15.189:0/1547038925 wait complete. 2024-04-06T13:47:09.497 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-06T13:47:09.498 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:47:09.569 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:09.569+0000 7fce45124200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:47:09.630 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:09.629+0000 7fbb7ac74200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:47:09.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.640+0000 7f36edd9a700 1 Processor -- start 2024-04-06T13:47:09.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.640+0000 7f36edd9a700 1 -- start start 2024-04-06T13:47:09.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.641+0000 7f36edd9a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f36e80cb870 0x7f36e815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:09.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.641+0000 7f36edd9a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f36e80cfa00 0x7f36e80cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:09.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.641+0000 7f36edd9a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e80cbd80 con 0x7f36e80d11b0 2024-04-06T13:47:09.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.641+0000 7f36edd9a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36e80ccdc0 con 0x7f36e80cfa00 2024-04-06T13:47:09.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.641+0000 7f36edd9a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36e80d0b50 con 0x7f36e80cb870 2024-04-06T13:47:09.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.641+0000 7f36e7fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f36e80cfa00 0x7f36e80cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:09.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.641+0000 7f36e7fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f36e80cfa00 0x7f36e80cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38336/0 (socket says 172.21.15.189:38336) 2024-04-06T13:47:09.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.641+0000 7f36e7fff700 1 -- 172.21.15.189:0/1506125704 learned_addr learned my addr 172.21.15.189:0/1506125704 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:09.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.641+0000 7f36e6ffd700 1 --2- 172.21.15.189:0/1506125704 >> v2:172.21.15.192:3300/0 conn(0x7f36e80cb870 0x7f36e815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:09.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.641+0000 7f36e7fff700 1 -- 172.21.15.189:0/1506125704 >> v2:172.21.15.192:3300/0 conn(0x7f36e80cb870 msgr2=0x7f36e815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:09.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.641+0000 7f36e7fff700 1 --2- 172.21.15.189:0/1506125704 >> v2:172.21.15.192:3300/0 conn(0x7f36e80cb870 0x7f36e815c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.641+0000 7f36e7fff700 1 -- 172.21.15.189:0/1506125704 >> v1:172.21.15.192:6789/0 conn(0x7f36e80d11b0 legacy=0x7f36e80d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:09.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.641+0000 7f36e67fc700 1 -- 172.21.15.189:0/1506125704 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124484765 0 0) 0x7f36e80cbd80 con 0x7f36e80d11b0 2024-04-06T13:47:09.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.641+0000 7f36e7fff700 1 -- 172.21.15.189:0/1506125704 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e80cef20 con 0x7f36e80cfa00 2024-04-06T13:47:09.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.642+0000 7f36e7fff700 1 --2- 172.21.15.189:0/1506125704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36e80cfa00 0x7f36e80cd560 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f36e804fb40 tx=0x7f36e0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=112fc79beaf4da60 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:09.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.642+0000 7f36e67fc700 1 -- 172.21.15.189:0/1506125704 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e0011830 con 0x7f36e80cfa00 2024-04-06T13:47:09.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.642+0000 7f36e67fc700 1 -- 172.21.15.189:0/1506125704 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e0011990 con 0x7f36e80cfa00 2024-04-06T13:47:09.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.642+0000 7f36e67fc700 1 -- 172.21.15.189:0/1506125704 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e0011d20 con 0x7f36e80cfa00 2024-04-06T13:47:09.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.642+0000 7f36edd9a700 1 -- 172.21.15.189:0/1506125704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36e80cfa00 msgr2=0x7f36e80cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:09.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.642+0000 7f36edd9a700 1 --2- 172.21.15.189:0/1506125704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36e80cfa00 0x7f36e80cd560 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f36e804fb40 tx=0x7f36e0009bd0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.642+0000 7f36edd9a700 1 -- 172.21.15.189:0/1506125704 shutdown_connections 2024-04-06T13:47:09.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.642+0000 7f36edd9a700 1 --2- 172.21.15.189:0/1506125704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36e80cfa00 0x7f36e80cd560 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.642+0000 7f36edd9a700 1 --2- 172.21.15.189:0/1506125704 >> v2:172.21.15.192:3300/0 conn(0x7f36e80cb870 0x7f36e815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.642+0000 7f36edd9a700 1 -- 172.21.15.189:0/1506125704 >> 172.21.15.189:0/1506125704 conn(0x7f36e80b2db0 msgr2=0x7f36e80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:09.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.642+0000 7f36edd9a700 1 -- 172.21.15.189:0/1506125704 shutdown_connections 2024-04-06T13:47:09.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.642+0000 7f36edd9a700 1 -- 172.21.15.189:0/1506125704 wait complete. 2024-04-06T13:47:09.648 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:09.647+0000 7fce45124200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:47:09.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.644+0000 7f36edd9a700 1 Processor -- start 2024-04-06T13:47:09.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.644+0000 7f36edd9a700 1 -- start start 2024-04-06T13:47:09.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.645+0000 7f36edd9a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f36e80cb870 0x7f36e80b6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:09.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.645+0000 7f36edd9a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36e80cfa00 0x7f36e8157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:09.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.645+0000 7f36edd9a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36e80d11b0 0x7f36e8158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:09.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.645+0000 7f36edd9a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e815aa90 con 0x7f36e80cfa00 2024-04-06T13:47:09.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.645+0000 7f36edd9a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e8156390 con 0x7f36e80d11b0 2024-04-06T13:47:09.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.645+0000 7f36edd9a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e80b6ec0 con 0x7f36e80cb870 2024-04-06T13:47:09.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.645+0000 7f36e6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36e80cfa00 0x7f36e8157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:09.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.645+0000 7f36e6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36e80cfa00 0x7f36e8157f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38338/0 (socket says 172.21.15.189:38338) 2024-04-06T13:47:09.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.645+0000 7f36e77fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f36e80cb870 0x7f36e80b6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:09.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.645+0000 7f36e6ffd700 1 -- 172.21.15.189:0/1224764223 learned_addr learned my addr 172.21.15.189:0/1224764223 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:09.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.645+0000 7f36e7fff700 1 --2- 172.21.15.189:0/1224764223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36e80d11b0 0x7f36e8158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:09.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.645+0000 7f36e6ffd700 1 -- 172.21.15.189:0/1224764223 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f36e80cb870 msgr2=0x7f36e80b6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:09.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.645+0000 7f36e6ffd700 1 --2- 172.21.15.189:0/1224764223 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f36e80cb870 0x7f36e80b6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.645+0000 7f36e6ffd700 1 -- 172.21.15.189:0/1224764223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36e80d11b0 msgr2=0x7f36e8158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:09.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.645+0000 7f36e6ffd700 1 --2- 172.21.15.189:0/1224764223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36e80d11b0 0x7f36e8158450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.645+0000 7f36e6ffd700 1 -- 172.21.15.189:0/1224764223 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e0009550 con 0x7f36e80cfa00 2024-04-06T13:47:09.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.645+0000 7f36e6ffd700 1 --2- 172.21.15.189:0/1224764223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36e80cfa00 0x7f36e8157f10 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f36dc00e840 tx=0x7f36dc00eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:09.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.646+0000 7f36cffff700 1 -- 172.21.15.189:0/1224764223 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36dc009590 con 0x7f36e80cfa00 2024-04-06T13:47:09.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.646+0000 7f36cffff700 1 -- 172.21.15.189:0/1224764223 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36dc013680 con 0x7f36e80cfa00 2024-04-06T13:47:09.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.646+0000 7f36edd9a700 1 -- 172.21.15.189:0/1224764223 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e814c440 con 0x7f36e80cfa00 2024-04-06T13:47:09.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.646+0000 7f36cffff700 1 -- 172.21.15.189:0/1224764223 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36dc013a30 con 0x7f36e80cfa00 2024-04-06T13:47:09.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.646+0000 7f36edd9a700 1 -- 172.21.15.189:0/1224764223 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e80b5070 con 0x7f36e80cfa00 2024-04-06T13:47:09.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.647+0000 7f36edd9a700 1 -- 172.21.15.189:0/1224764223 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e8156520 con 0x7f36e80cfa00 2024-04-06T13:47:09.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.647+0000 7f36cffff700 1 -- 172.21.15.189:0/1224764223 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 135) v1 ==== 89361+0+0 (secure 0 0 0) 0x7f36dc013b90 con 0x7f36e80cfa00 2024-04-06T13:47:09.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.648+0000 7f36cffff700 1 --2- 172.21.15.189:0/1224764223 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f36d406aed0 0x7f36d406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:09.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.648+0000 7f36e77fe700 1 -- 172.21.15.189:0/1224764223 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f36d406aed0 msgr2=0x7f36d406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:09.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.648+0000 7f36cffff700 1 -- 172.21.15.189:0/1224764223 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6604+0+0 (secure 0 0 0) 0x7f36dc0899b0 con 0x7f36e80cfa00 2024-04-06T13:47:09.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.648+0000 7f36e77fe700 1 --2- 172.21.15.189:0/1224764223 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f36d406aed0 0x7f36d406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:09.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.652+0000 7f36cffff700 1 -- 172.21.15.189:0/1224764223 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f36dc054410 con 0x7f36e80cfa00 2024-04-06T13:47:09.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.849+0000 7f36e77fe700 1 -- 172.21.15.189:0/1224764223 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f36d406aed0 msgr2=0x7f36d406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:09.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.849+0000 7f36e77fe700 1 --2- 172.21.15.189:0/1224764223 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f36d406aed0 0x7f36d406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:09.888 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:09.887+0000 7fbb7ac74200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:47:09.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.904+0000 7f36edd9a700 1 -- 172.21.15.189:0/1224764223 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36e8074990 con 0x7f36e80cfa00 2024-04-06T13:47:09.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.905+0000 7f36cffff700 1 -- 172.21.15.189:0/1224764223 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+4750 (secure 0 0 0) 0x7f36dc057a30 con 0x7f36e80cfa00 2024-04-06T13:47:09.906 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:09.906 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":93,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8525,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3302262361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3302262361},{"type":"v1","addr":"172.21.15.189:6837","nonce":3302262361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":8534,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/229523550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":229523550},{"type":"v1","addr":"172.21.15.192:6834","nonce":229523550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":9739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2560949338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2560949338},{"type":"v1","addr":"172.21.15.192:6835","nonce":2560949338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:45:40.072274+0000","modified":"2024-04-06T13:45:43.092032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":0,"incarnation":89,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/3619731181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3619731181},{"type":"v1","addr":"172.21.15.189:6836","nonce":3619731181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-04-06T13:47:09.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.907+0000 7f36edd9a700 1 -- 172.21.15.189:0/1224764223 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f36d406aed0 msgr2=0x7f36d406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:09.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.907+0000 7f36edd9a700 1 --2- 172.21.15.189:0/1224764223 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f36d406aed0 0x7f36d406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.907+0000 7f36edd9a700 1 -- 172.21.15.189:0/1224764223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36e80cfa00 msgr2=0x7f36e8157f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:09.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.907+0000 7f36edd9a700 1 --2- 172.21.15.189:0/1224764223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36e80cfa00 0x7f36e8157f10 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f36dc00e840 tx=0x7f36dc00eb50 comp rx=0 tx=0).stop 2024-04-06T13:47:09.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.907+0000 7f36edd9a700 1 -- 172.21.15.189:0/1224764223 shutdown_connections 2024-04-06T13:47:09.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.907+0000 7f36edd9a700 1 --2- 172.21.15.189:0/1224764223 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f36d406aed0 0x7f36d406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.907+0000 7f36edd9a700 1 --2- 172.21.15.189:0/1224764223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36e80d11b0 0x7f36e8158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.907+0000 7f36edd9a700 1 --2- 172.21.15.189:0/1224764223 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f36e80cb870 0x7f36e80b6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.907+0000 7f36edd9a700 1 --2- 172.21.15.189:0/1224764223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36e80cfa00 0x7f36e8157f10 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:09.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.907+0000 7f36edd9a700 1 -- 172.21.15.189:0/1224764223 >> 172.21.15.189:0/1224764223 conn(0x7f36e80b2db0 msgr2=0x7f36e8159b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:09.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.907+0000 7f36edd9a700 1 -- 172.21.15.189:0/1224764223 shutdown_connections 2024-04-06T13:47:09.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:09.907+0000 7f36edd9a700 1 -- 172.21.15.189:0/1224764223 wait complete. 2024-04-06T13:47:09.910 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 93 2024-04-06T13:47:09.921 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:47:09.970 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:09.969+0000 7fbb7ac74200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:47:10.021 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:10.020+0000 7fce45124200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:47:10.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.046+0000 7fbc7d53c700 1 Processor -- start 2024-04-06T13:47:10.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.046+0000 7fbc7d53c700 1 -- start start 2024-04-06T13:47:10.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.047+0000 7fbc7d53c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbc780b4ed0 0x7fbc780b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:10.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.047+0000 7fbc7d53c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc780d2880 0x7fbc780d2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:10.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.047+0000 7fbc7d53c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc780b5cc0 con 0x7fbc7815d590 2024-04-06T13:47:10.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.047+0000 7fbc7d53c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc780b58c0 con 0x7fbc780b4ed0 2024-04-06T13:47:10.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.047+0000 7fbc7d53c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc780d1500 con 0x7fbc780d2880 2024-04-06T13:47:10.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.047+0000 7fbc76ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbc7815d590 0x7fbc780b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42784/0 (socket says 172.21.15.189:42784) 2024-04-06T13:47:10.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.047+0000 7fbc76ffd700 1 -- 172.21.15.189:0/525035311 learned_addr learned my addr 172.21.15.189:0/525035311 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:10.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.047+0000 7fbc777fe700 1 --2- 172.21.15.189:0/525035311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc780d2880 0x7fbc780d2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:10.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.047+0000 7fbc767fc700 1 --2- 172.21.15.189:0/525035311 >> v2:172.21.15.192:3300/0 conn(0x7fbc780b4ed0 0x7fbc780b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:10.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.047+0000 7fbc75ffb700 1 -- 172.21.15.189:0/525035311 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637027235 0 0) 0x7fbc780b5cc0 con 0x7fbc7815d590 2024-04-06T13:47:10.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.047+0000 7fbc75ffb700 1 -- 172.21.15.189:0/525035311 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc600035b0 con 0x7fbc7815d590 2024-04-06T13:47:10.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.047+0000 7fbc75ffb700 1 -- 172.21.15.189:0/525035311 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3557172824 0 0) 0x7fbc600035b0 con 0x7fbc7815d590 2024-04-06T13:47:10.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.047+0000 7fbc777fe700 1 -- 172.21.15.189:0/525035311 >> v2:172.21.15.192:3300/0 conn(0x7fbc780b4ed0 msgr2=0x7fbc780b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:10.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.047+0000 7fbc777fe700 1 --2- 172.21.15.189:0/525035311 >> v2:172.21.15.192:3300/0 conn(0x7fbc780b4ed0 0x7fbc780b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.047+0000 7fbc777fe700 1 -- 172.21.15.189:0/525035311 >> v1:172.21.15.189:6789/0 conn(0x7fbc7815d590 legacy=0x7fbc780b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:10.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.047+0000 7fbc777fe700 1 -- 172.21.15.189:0/525035311 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc780d17e0 con 0x7fbc780d2880 2024-04-06T13:47:10.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.047+0000 7fbc767fc700 1 --2- 172.21.15.189:0/525035311 >> v2:172.21.15.192:3300/0 conn(0x7fbc780b4ed0 0x7fbc780b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:10.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.048+0000 7fbc777fe700 1 --2- 172.21.15.189:0/525035311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc780d2880 0x7fbc780d2cf0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbc70009d90 tx=0x7fbc7000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6743616654530ba6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:10.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.048+0000 7fbc75ffb700 1 -- 172.21.15.189:0/525035311 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7000e070 con 0x7fbc780d2880 2024-04-06T13:47:10.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.048+0000 7fbc75ffb700 1 -- 172.21.15.189:0/525035311 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc70003a20 con 0x7fbc780d2880 2024-04-06T13:47:10.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.048+0000 7fbc75ffb700 1 -- 172.21.15.189:0/525035311 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc68001670 con 0x7fbc780d2880 2024-04-06T13:47:10.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.048+0000 7fbc7d53c700 1 -- 172.21.15.189:0/525035311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc780d2880 msgr2=0x7fbc780d2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:10.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.048+0000 7fbc7d53c700 1 --2- 172.21.15.189:0/525035311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc780d2880 0x7fbc780d2cf0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbc70009d90 tx=0x7fbc7000b330 comp rx=0 tx=0).stop 2024-04-06T13:47:10.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.048+0000 7fbc7d53c700 1 -- 172.21.15.189:0/525035311 shutdown_connections 2024-04-06T13:47:10.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.048+0000 7fbc7d53c700 1 --2- 172.21.15.189:0/525035311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc780d2880 0x7fbc780d2cf0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.048+0000 7fbc7d53c700 1 --2- 172.21.15.189:0/525035311 >> v2:172.21.15.192:3300/0 conn(0x7fbc780b4ed0 0x7fbc780b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.048+0000 7fbc7d53c700 1 -- 172.21.15.189:0/525035311 >> 172.21.15.189:0/525035311 conn(0x7fbc7806bbc0 msgr2=0x7fbc7806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:10.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.048+0000 7fbc7d53c700 1 -- 172.21.15.189:0/525035311 shutdown_connections 2024-04-06T13:47:10.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.048+0000 7fbc7d53c700 1 -- 172.21.15.189:0/525035311 wait complete. 2024-04-06T13:47:10.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.050+0000 7fbc7d53c700 1 Processor -- start 2024-04-06T13:47:10.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.050+0000 7fbc7d53c700 1 -- start start 2024-04-06T13:47:10.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.050+0000 7fbc7d53c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc780b4ed0 0x7fbc7814dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:10.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.050+0000 7fbc7d53c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc780d2880 0x7fbc780ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:10.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.050+0000 7fbc7d53c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc7815d590 0x7fbc7815a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:10.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.050+0000 7fbc7d53c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7814d3e0 con 0x7fbc780d2880 2024-04-06T13:47:10.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.050+0000 7fbc7d53c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7815ad60 con 0x7fbc7815d590 2024-04-06T13:47:10.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.050+0000 7fbc7d53c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7815e640 con 0x7fbc780b4ed0 2024-04-06T13:47:10.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.051+0000 7fbc767fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc780d2880 0x7fbc780ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:10.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.051+0000 7fbc767fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc780d2880 0x7fbc780ccdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38340/0 (socket says 172.21.15.189:38340) 2024-04-06T13:47:10.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.051+0000 7fbc767fc700 1 -- 172.21.15.189:0/1237635966 learned_addr learned my addr 172.21.15.189:0/1237635966 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:10.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.051+0000 7fbc76ffd700 1 --2- 172.21.15.189:0/1237635966 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc780b4ed0 0x7fbc7814dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:10.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.051+0000 7fbc777fe700 1 --2- 172.21.15.189:0/1237635966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc7815d590 0x7fbc7815a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:10.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.051+0000 7fbc767fc700 1 -- 172.21.15.189:0/1237635966 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc780b4ed0 msgr2=0x7fbc7814dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:10.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.051+0000 7fbc767fc700 1 --2- 172.21.15.189:0/1237635966 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc780b4ed0 0x7fbc7814dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.051+0000 7fbc767fc700 1 -- 172.21.15.189:0/1237635966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc7815d590 msgr2=0x7fbc7815a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:10.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.051+0000 7fbc767fc700 1 --2- 172.21.15.189:0/1237635966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc7815d590 0x7fbc7815a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.051+0000 7fbc767fc700 1 -- 172.21.15.189:0/1237635966 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6c0094a0 con 0x7fbc780d2880 2024-04-06T13:47:10.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.051+0000 7fbc76ffd700 1 --2- 172.21.15.189:0/1237635966 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc780b4ed0 0x7fbc7814dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:10.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.051+0000 7fbc767fc700 1 --2- 172.21.15.189:0/1237635966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc780d2880 0x7fbc780ccdc0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00ebe0 tx=0x7fbc6c009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:10.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.051+0000 7fbc5f7fe700 1 -- 172.21.15.189:0/1237635966 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c00c910 con 0x7fbc780d2880 2024-04-06T13:47:10.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.051+0000 7fbc5f7fe700 1 -- 172.21.15.189:0/1237635966 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc6c00ca70 con 0x7fbc780d2880 2024-04-06T13:47:10.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.051+0000 7fbc7d53c700 1 -- 172.21.15.189:0/1237635966 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc70009a70 con 0x7fbc780d2880 2024-04-06T13:47:10.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.051+0000 7fbc5f7fe700 1 -- 172.21.15.189:0/1237635966 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c00cde0 con 0x7fbc780d2880 2024-04-06T13:47:10.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.051+0000 7fbc7d53c700 1 -- 172.21.15.189:0/1237635966 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc7815ec00 con 0x7fbc780d2880 2024-04-06T13:47:10.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.052+0000 7fbc7d53c700 1 -- 172.21.15.189:0/1237635966 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc780d2060 con 0x7fbc780d2880 2024-04-06T13:47:10.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.053+0000 7fbc5f7fe700 1 -- 172.21.15.189:0/1237635966 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 135) v1 ==== 89361+0+0 (secure 0 0 0) 0x7fbc6c013470 con 0x7fbc780d2880 2024-04-06T13:47:10.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.053+0000 7fbc5f7fe700 1 --2- 172.21.15.189:0/1237635966 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbc6406aed0 0x7fbc6406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:10.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.053+0000 7fbc76ffd700 1 -- 172.21.15.189:0/1237635966 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbc6406aed0 msgr2=0x7fbc6406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:10.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.053+0000 7fbc5f7fe700 1 -- 172.21.15.189:0/1237635966 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6604+0+0 (secure 0 0 0) 0x7fbc6c089d60 con 0x7fbc780d2880 2024-04-06T13:47:10.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.053+0000 7fbc76ffd700 1 --2- 172.21.15.189:0/1237635966 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbc6406aed0 0x7fbc6406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:10.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.058+0000 7fbc5f7fe700 1 -- 172.21.15.189:0/1237635966 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbc780d2060 con 0x7fbc780d2880 2024-04-06T13:47:10.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.254+0000 7fbc76ffd700 1 -- 172.21.15.189:0/1237635966 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbc6406aed0 msgr2=0x7fbc6406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:10.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.254+0000 7fbc76ffd700 1 --2- 172.21.15.189:0/1237635966 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbc6406aed0 0x7fbc6406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:10.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.308+0000 7fbc7d53c700 1 -- 172.21.15.189:0/1237635966 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc780d2060 con 0x7fbc780d2880 2024-04-06T13:47:10.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.309+0000 7fbc5f7fe700 1 -- 172.21.15.189:0/1237635966 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+4750 (secure 0 0 0) 0x7fbc780d2060 con 0x7fbc780d2880 2024-04-06T13:47:10.310 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:10.310 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":93,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8525,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3302262361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3302262361},{"type":"v1","addr":"172.21.15.189:6837","nonce":3302262361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":8534,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/229523550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":229523550},{"type":"v1","addr":"172.21.15.192:6834","nonce":229523550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":9739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2560949338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2560949338},{"type":"v1","addr":"172.21.15.192:6835","nonce":2560949338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:45:40.072274+0000","modified":"2024-04-06T13:45:43.092032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":0,"incarnation":89,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6836/3619731181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3619731181},{"type":"v1","addr":"172.21.15.189:6836","nonce":3619731181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-04-06T13:47:10.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.313+0000 7fbc7d53c700 1 -- 172.21.15.189:0/1237635966 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbc6406aed0 msgr2=0x7fbc6406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:10.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.313+0000 7fbc7d53c700 1 --2- 172.21.15.189:0/1237635966 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbc6406aed0 0x7fbc6406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.313+0000 7fbc7d53c700 1 -- 172.21.15.189:0/1237635966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc780d2880 msgr2=0x7fbc780ccdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:10.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.313+0000 7fbc7d53c700 1 --2- 172.21.15.189:0/1237635966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc780d2880 0x7fbc780ccdc0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00ebe0 tx=0x7fbc6c009ec0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.313+0000 7fbc7d53c700 1 -- 172.21.15.189:0/1237635966 shutdown_connections 2024-04-06T13:47:10.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.313+0000 7fbc7d53c700 1 --2- 172.21.15.189:0/1237635966 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbc6406aed0 0x7fbc6406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.313+0000 7fbc7d53c700 1 --2- 172.21.15.189:0/1237635966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc7815d590 0x7fbc7815a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.313+0000 7fbc7d53c700 1 --2- 172.21.15.189:0/1237635966 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbc780b4ed0 0x7fbc7814dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.313+0000 7fbc7d53c700 1 --2- 172.21.15.189:0/1237635966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc780d2880 0x7fbc780ccdc0 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.313+0000 7fbc7d53c700 1 -- 172.21.15.189:0/1237635966 >> 172.21.15.189:0/1237635966 conn(0x7fbc7806bbc0 msgr2=0x7fbc7815bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:10.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.313+0000 7fbc7d53c700 1 -- 172.21.15.189:0/1237635966 shutdown_connections 2024-04-06T13:47:10.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.313+0000 7fbc7d53c700 1 -- 172.21.15.189:0/1237635966 wait complete. 2024-04-06T13:47:10.315 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 93 2024-04-06T13:47:10.331 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:47:10.377 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:10.376+0000 7fbb7ac74200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:47:10.459 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:10.458+0000 7fbaf6353700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:10.459 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:10.458+0000 7fbaf6353700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:10.459 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:10.458+0000 7fbaf6353700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:10.459 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:10.458+0000 7fbaf6353700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:10.459 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:10.458+0000 7fbaf6353700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:10.460 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:10.458+0000 7fbaf334d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:10.460 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:10.458+0000 7fbaf334d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:10.460 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:10.458+0000 7fbaf334d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:10.460 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:10.458+0000 7fbaf334d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:10.460 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:10.458+0000 7fbaf334d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:10.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.470+0000 7ffb350d4700 1 Processor -- start 2024-04-06T13:47:10.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.470+0000 7ffb350d4700 1 -- start start 2024-04-06T13:47:10.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.470+0000 7ffb350d4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ffb300d11b0 0x7ffb300d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:10.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.470+0000 7ffb350d4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffb300cb870 0x7ffb3015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:10.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.470+0000 7ffb350d4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb300cfa00 0x7ffb300cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:10.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.470+0000 7ffb350d4700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb300ccdc0 con 0x7ffb300cb870 2024-04-06T13:47:10.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.470+0000 7ffb350d4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb300d0b50 con 0x7ffb300d11b0 2024-04-06T13:47:10.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.470+0000 7ffb350d4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb300d1ad0 con 0x7ffb300cfa00 2024-04-06T13:47:10.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.470+0000 7ffb2e59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffb300cb870 0x7ffb3015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:10.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.471+0000 7ffb2e59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffb300cb870 0x7ffb3015c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38440/0 (socket says 172.21.15.189:38440) 2024-04-06T13:47:10.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.471+0000 7ffb2e59c700 1 -- 172.21.15.189:0/2622533915 learned_addr learned my addr 172.21.15.189:0/2622533915 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:10.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.471+0000 7ffb2ed9d700 1 --2- 172.21.15.189:0/2622533915 >> v2:172.21.15.192:3300/0 conn(0x7ffb300d11b0 0x7ffb300d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:10.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.471+0000 7ffb2f59e700 1 --2- 172.21.15.189:0/2622533915 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb300cfa00 0x7ffb300cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:10.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.471+0000 7ffb2e59c700 1 -- 172.21.15.189:0/2622533915 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb300cfa00 msgr2=0x7ffb300cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:10.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.471+0000 7ffb2e59c700 1 --2- 172.21.15.189:0/2622533915 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb300cfa00 0x7ffb300cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.471+0000 7ffb2e59c700 1 -- 172.21.15.189:0/2622533915 >> v2:172.21.15.192:3300/0 conn(0x7ffb300d11b0 msgr2=0x7ffb300d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:10.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.471+0000 7ffb2e59c700 1 --2- 172.21.15.189:0/2622533915 >> v2:172.21.15.192:3300/0 conn(0x7ffb300d11b0 0x7ffb300d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.471+0000 7ffb2e59c700 1 -- 172.21.15.189:0/2622533915 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb300cbd80 con 0x7ffb300cb870 2024-04-06T13:47:10.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.471+0000 7ffb2f59e700 1 --2- 172.21.15.189:0/2622533915 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb300cfa00 0x7ffb300cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:10.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.471+0000 7ffb2ed9d700 1 --2- 172.21.15.189:0/2622533915 >> v2:172.21.15.192:3300/0 conn(0x7ffb300d11b0 0x7ffb300d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:10.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.471+0000 7ffb2e59c700 1 --2- 172.21.15.189:0/2622533915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb300cb870 0x7ffb3015c280 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7ffb300ced00 tx=0x7ffb20009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8a6c03c949647c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:10.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.471+0000 7ffb2dd9b700 1 -- 172.21.15.189:0/2622533915 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2000b870 con 0x7ffb300cb870 2024-04-06T13:47:10.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.471+0000 7ffb2dd9b700 1 -- 172.21.15.189:0/2622533915 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb2000b9d0 con 0x7ffb300cb870 2024-04-06T13:47:10.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.471+0000 7ffb2dd9b700 1 -- 172.21.15.189:0/2622533915 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2000bd60 con 0x7ffb300cb870 2024-04-06T13:47:10.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.472+0000 7ffb350d4700 1 -- 172.21.15.189:0/2622533915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb300cb870 msgr2=0x7ffb3015c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:10.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.472+0000 7ffb350d4700 1 --2- 172.21.15.189:0/2622533915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb300cb870 0x7ffb3015c280 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7ffb300ced00 tx=0x7ffb20009bd0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.472+0000 7ffb350d4700 1 -- 172.21.15.189:0/2622533915 shutdown_connections 2024-04-06T13:47:10.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.472+0000 7ffb350d4700 1 --2- 172.21.15.189:0/2622533915 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb300cfa00 0x7ffb300cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.472+0000 7ffb350d4700 1 --2- 172.21.15.189:0/2622533915 >> v2:172.21.15.192:3300/0 conn(0x7ffb300d11b0 0x7ffb300d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.472+0000 7ffb350d4700 1 --2- 172.21.15.189:0/2622533915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb300cb870 0x7ffb3015c280 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.472+0000 7ffb350d4700 1 -- 172.21.15.189:0/2622533915 >> 172.21.15.189:0/2622533915 conn(0x7ffb300b2db0 msgr2=0x7ffb300b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:10.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.472+0000 7ffb350d4700 1 -- 172.21.15.189:0/2622533915 shutdown_connections 2024-04-06T13:47:10.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.472+0000 7ffb350d4700 1 -- 172.21.15.189:0/2622533915 wait complete. 2024-04-06T13:47:10.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.473+0000 7ffb350d4700 1 Processor -- start 2024-04-06T13:47:10.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.473+0000 7ffb350d4700 1 -- start start 2024-04-06T13:47:10.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.474+0000 7ffb350d4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb300cb870 0x7ffb301478d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:10.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.474+0000 7ffb350d4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb300cfa00 0x7ffb3015cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:10.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.474+0000 7ffb350d4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb300d11b0 0x7ffb300c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:10.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.474+0000 7ffb350d4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3015d550 con 0x7ffb300cfa00 2024-04-06T13:47:10.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.474+0000 7ffb350d4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb30157ec0 con 0x7ffb300d11b0 2024-04-06T13:47:10.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.474+0000 7ffb350d4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb301581a0 con 0x7ffb300cb870 2024-04-06T13:47:10.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.474+0000 7ffb2e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb300cfa00 0x7ffb3015cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:10.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.474+0000 7ffb2e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb300cfa00 0x7ffb3015cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38450/0 (socket says 172.21.15.189:38450) 2024-04-06T13:47:10.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.474+0000 7ffb2e59c700 1 -- 172.21.15.189:0/1894132454 learned_addr learned my addr 172.21.15.189:0/1894132454 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:10.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.474+0000 7ffb2ed9d700 1 --2- 172.21.15.189:0/1894132454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb300cb870 0x7ffb301478d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:10.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.474+0000 7ffb2f59e700 1 --2- 172.21.15.189:0/1894132454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb300d11b0 0x7ffb300c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:10.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.474+0000 7ffb2e59c700 1 -- 172.21.15.189:0/1894132454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb300cb870 msgr2=0x7ffb301478d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:10.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.474+0000 7ffb2e59c700 1 --2- 172.21.15.189:0/1894132454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb300cb870 0x7ffb301478d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.474+0000 7ffb2e59c700 1 -- 172.21.15.189:0/1894132454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb300d11b0 msgr2=0x7ffb300c84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:10.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.474+0000 7ffb2e59c700 1 --2- 172.21.15.189:0/1894132454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb300d11b0 0x7ffb300c84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.474+0000 7ffb2e59c700 1 -- 172.21.15.189:0/1894132454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb20009550 con 0x7ffb300cfa00 2024-04-06T13:47:10.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.474+0000 7ffb2e59c700 1 --2- 172.21.15.189:0/1894132454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb300cfa00 0x7ffb3015cbc0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7ffb20014040 tx=0x7ffb20004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:10.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.475+0000 7ffb177fe700 1 -- 172.21.15.189:0/1894132454 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb20004250 con 0x7ffb300cfa00 2024-04-06T13:47:10.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.475+0000 7ffb177fe700 1 -- 172.21.15.189:0/1894132454 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb2001e070 con 0x7ffb300cfa00 2024-04-06T13:47:10.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.475+0000 7ffb350d4700 1 -- 172.21.15.189:0/1894132454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb30158480 con 0x7ffb300cfa00 2024-04-06T13:47:10.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.475+0000 7ffb177fe700 1 -- 172.21.15.189:0/1894132454 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb200045b0 con 0x7ffb300cfa00 2024-04-06T13:47:10.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.475+0000 7ffb350d4700 1 -- 172.21.15.189:0/1894132454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb3015d100 con 0x7ffb300cfa00 2024-04-06T13:47:10.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.476+0000 7ffb350d4700 1 -- 172.21.15.189:0/1894132454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb3015d6c0 con 0x7ffb300cfa00 2024-04-06T13:47:10.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.476+0000 7ffb177fe700 1 -- 172.21.15.189:0/1894132454 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 136) v1 ==== 88856+0+0 (secure 0 0 0) 0x7ffb20020030 con 0x7ffb300cfa00 2024-04-06T13:47:10.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.477+0000 7ffb177fe700 1 -- 172.21.15.189:0/1894132454 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6904+0+0 (secure 0 0 0) 0x7ffb2008a0e0 con 0x7ffb300cfa00 2024-04-06T13:47:10.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.481+0000 7ffb177fe700 1 -- 172.21.15.189:0/1894132454 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ffb3015d6c0 con 0x7ffb300cfa00 2024-04-06T13:47:10.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.680+0000 7ffb350d4700 1 -- 172.21.15.189:0/1894132454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb3015d690 con 0x7ffb300cfa00 2024-04-06T13:47:10.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.681+0000 7ffb177fe700 1 -- 172.21.15.189:0/1894132454 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v215) v1 ==== 74+0+17689 (secure 0 0 0) 0x7ffb20055260 con 0x7ffb300cfa00 2024-04-06T13:47:10.682 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:10.682 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":215,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:47:10.390500+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:45:37.837008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-04-06T13:45:39.030714+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":214,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.22","mappings":[{"from":1,"to":0}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]},{"pgid":"26.2c","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/2259780054":"2024-04-07T13:47:10.390431+0000","172.21.15.189:0/2233985408":"2024-04-07T13:47:10.390431+0000","172.21.15.189:0/3129045456":"2024-04-07T13:47:10.390431+0000","172.21.15.189:0/3783774588":"2024-04-07T13:47:10.390431+0000","172.21.15.189:0/4098791028":"2024-04-07T13:47:10.390431+0000","172.21.15.189:0/2794240278":"2024-04-07T13:47:10.390431+0000","172.21.15.189:0/509530111":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/3787698124":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/2802982514":"2024-04-07T13:45:54.091329+0000","172.21.15.189:6833/111704":"2024-04-07T13:45:54.091329+0000","172.21.15.189:6832/111704":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/606015831":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/2678793686":"2024-04-07T13:47:10.390431+0000","172.21.15.189:0/4174593093":"2024-04-07T13:45:54.091329+0000","172.21.15.189:0/4162202349":"2024-04-07T13:47:10.390431+0000","172.21.15.189:0/3225575705":"2024-04-07T13:45:54.091329+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:47:10.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.683+0000 7ffb350d4700 1 -- 172.21.15.189:0/1894132454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb300cfa00 msgr2=0x7ffb3015cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:10.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.683+0000 7ffb350d4700 1 --2- 172.21.15.189:0/1894132454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb300cfa00 0x7ffb3015cbc0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7ffb20014040 tx=0x7ffb20004750 comp rx=0 tx=0).stop 2024-04-06T13:47:10.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.683+0000 7ffb350d4700 1 -- 172.21.15.189:0/1894132454 shutdown_connections 2024-04-06T13:47:10.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.683+0000 7ffb350d4700 1 --2- 172.21.15.189:0/1894132454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb300d11b0 0x7ffb300c84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.683+0000 7ffb350d4700 1 --2- 172.21.15.189:0/1894132454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb300cb870 0x7ffb301478d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.683+0000 7ffb350d4700 1 --2- 172.21.15.189:0/1894132454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb300cfa00 0x7ffb3015cbc0 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.683+0000 7ffb350d4700 1 -- 172.21.15.189:0/1894132454 >> 172.21.15.189:0/1894132454 conn(0x7ffb300b2db0 msgr2=0x7ffb300cd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:10.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.683+0000 7ffb350d4700 1 -- 172.21.15.189:0/1894132454 shutdown_connections 2024-04-06T13:47:10.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.683+0000 7ffb350d4700 1 -- 172.21.15.189:0/1894132454 wait complete. 2024-04-06T13:47:10.697 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-06T13:47:10.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.819+0000 7f02a1df1700 1 Processor -- start 2024-04-06T13:47:10.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.819+0000 7f02a1df1700 1 -- start start 2024-04-06T13:47:10.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.820+0000 7f02a1df1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f029c144f40 0x7f029c06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:10.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.820+0000 7f02a1df1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029c15c740 con 0x7f029c15fcd0 2024-04-06T13:47:10.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.820+0000 7f02a1df1700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029c15f1f0 con 0x7f029c15c150 2024-04-06T13:47:10.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.820+0000 7f02a1df1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029c15c560 con 0x7f029c144f40 2024-04-06T13:47:10.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.820+0000 7f029affd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f029c15fcd0 0x7f029c15ba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42798/0 (socket says 172.21.15.189:42798) 2024-04-06T13:47:10.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.820+0000 7f029affd700 1 -- 172.21.15.189:0/3479357042 learned_addr learned my addr 172.21.15.189:0/3479357042 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:10.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.820+0000 7f029a7fc700 1 -- 172.21.15.189:0/3479357042 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739541401 0 0) 0x7f029c15c740 con 0x7f029c15fcd0 2024-04-06T13:47:10.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.820+0000 7f029b7fe700 1 --2- 172.21.15.189:0/3479357042 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f029c144f40 0x7f029c06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:10.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.820+0000 7f029a7fc700 1 -- 172.21.15.189:0/3479357042 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02880035b0 con 0x7f029c15fcd0 2024-04-06T13:47:10.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.820+0000 7f029a7fc700 1 -- 172.21.15.189:0/3479357042 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3758890445 0 0) 0x7f02880035b0 con 0x7f029c15fcd0 2024-04-06T13:47:10.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.820+0000 7f029a7fc700 1 -- 172.21.15.189:0/3479357042 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f029c144f40 msgr2=0x7f029c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:10.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.820+0000 7f029a7fc700 1 --2- 172.21.15.189:0/3479357042 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f029c144f40 0x7f029c06a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.820+0000 7f029a7fc700 1 -- 172.21.15.189:0/3479357042 >> v1:172.21.15.192:6789/0 conn(0x7f029c15c150 legacy=0x7f029c0ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:10.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.820+0000 7f029a7fc700 1 -- 172.21.15.189:0/3479357042 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c15f4d0 con 0x7f029c15fcd0 2024-04-06T13:47:10.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.820+0000 7f029a7fc700 1 -- 172.21.15.189:0/3479357042 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f028c001670 con 0x7f029c15fcd0 2024-04-06T13:47:10.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.820+0000 7f029a7fc700 1 -- 172.21.15.189:0/3479357042 >> v1:172.21.15.189:6789/0 conn(0x7f029c15fcd0 legacy=0x7f029c15ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:10.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.821+0000 7f029a7fc700 1 --2- 172.21.15.189:0/3479357042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0288003a30 0x7f0288003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:10.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.821+0000 7f029a7fc700 1 --2- 172.21.15.189:0/3479357042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0288004850 0x7f0288004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:10.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.821+0000 7f029a7fc700 1 --2- 172.21.15.189:0/3479357042 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f029c144f40 0x7f0288005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:10.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.821+0000 7f029a7fc700 1 -- 172.21.15.189:0/3479357042 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c15c560 con 0x7f0288004850 2024-04-06T13:47:10.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.821+0000 7f029a7fc700 1 -- 172.21.15.189:0/3479357042 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028800bb40 con 0x7f0288003a30 2024-04-06T13:47:10.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.821+0000 7f029a7fc700 1 -- 172.21.15.189:0/3479357042 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f028800bd00 con 0x7f029c144f40 2024-04-06T13:47:10.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.821+0000 7f029affd700 1 --2- 172.21.15.189:0/3479357042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0288004850 0x7f0288004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:10.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.821+0000 7f029b7fe700 1 --2- 172.21.15.189:0/3479357042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0288003a30 0x7f0288003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:10.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.821+0000 7f029bfff700 1 --2- 172.21.15.189:0/3479357042 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f029c144f40 0x7f0288005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:10.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.821+0000 7f029affd700 1 -- 172.21.15.189:0/3479357042 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f029c144f40 msgr2=0x7f0288005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:10.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.821+0000 7f029affd700 1 --2- 172.21.15.189:0/3479357042 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f029c144f40 0x7f0288005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.821+0000 7f029affd700 1 -- 172.21.15.189:0/3479357042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0288003a30 msgr2=0x7f0288003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:10.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.821+0000 7f029affd700 1 --2- 172.21.15.189:0/3479357042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0288003a30 0x7f0288003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.821+0000 7f029affd700 1 -- 172.21.15.189:0/3479357042 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02880035b0 con 0x7f0288004850 2024-04-06T13:47:10.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.821+0000 7f029affd700 1 --2- 172.21.15.189:0/3479357042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0288004850 0x7f0288004cd0 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f028c00c240 tx=0x7f0288005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82d198451183a4ad server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:10.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.821+0000 7f029a7fc700 1 -- 172.21.15.189:0/3479357042 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c007c10 con 0x7f0288004850 2024-04-06T13:47:10.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.821+0000 7f029a7fc700 1 -- 172.21.15.189:0/3479357042 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028c00cde0 con 0x7f0288004850 2024-04-06T13:47:10.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.821+0000 7f029a7fc700 1 -- 172.21.15.189:0/3479357042 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c005ba0 con 0x7f0288004850 2024-04-06T13:47:10.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.822+0000 7f02a1df1700 1 -- 172.21.15.189:0/3479357042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0288004850 msgr2=0x7f0288004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:10.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.822+0000 7f02a1df1700 1 --2- 172.21.15.189:0/3479357042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0288004850 0x7f0288004cd0 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f028c00c240 tx=0x7f0288005880 comp rx=0 tx=0).stop 2024-04-06T13:47:10.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.822+0000 7f029bfff700 1 -- 172.21.15.189:0/3479357042 reap_dead start 2024-04-06T13:47:10.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.822+0000 7f02a1df1700 1 -- 172.21.15.189:0/3479357042 shutdown_connections 2024-04-06T13:47:10.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.822+0000 7f02a1df1700 1 -- 172.21.15.189:0/3479357042 >> 172.21.15.189:0/3479357042 conn(0x7f029c0d00b0 msgr2=0x7f029c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:10.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.822+0000 7f02a1df1700 1 -- 172.21.15.189:0/3479357042 shutdown_connections 2024-04-06T13:47:10.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.822+0000 7f02a1df1700 1 -- 172.21.15.189:0/3479357042 wait complete. 2024-04-06T13:47:10.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.823+0000 7f02a1df1700 1 Processor -- start 2024-04-06T13:47:10.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.823+0000 7f02a1df1700 1 -- start start 2024-04-06T13:47:10.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.824+0000 7f02a1df1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0288004850 0x7f029c136580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:10.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.824+0000 7f02a1df1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f029c1f6c00 0x7f029c1389b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:10.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.824+0000 7f02a1df1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f029c13f4a0 0x7f029c15a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:10.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.824+0000 7f02a1df1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c1f7080 con 0x7f029c13f4a0 2024-04-06T13:47:10.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.824+0000 7f02a1df1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c1f6160 con 0x7f029c1f6c00 2024-04-06T13:47:10.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.824+0000 7f02a1df1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029c1392e0 con 0x7f0288004850 2024-04-06T13:47:10.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.824+0000 7f029bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f029c13f4a0 0x7f029c15a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:10.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.824+0000 7f029bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f029c13f4a0 0x7f029c15a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38460/0 (socket says 172.21.15.189:38460) 2024-04-06T13:47:10.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.824+0000 7f029bfff700 1 -- 172.21.15.189:0/1470390413 learned_addr learned my addr 172.21.15.189:0/1470390413 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:10.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.824+0000 7f029affd700 1 --2- 172.21.15.189:0/1470390413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f029c1f6c00 0x7f029c1389b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:10.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.824+0000 7f029b7fe700 1 --2- 172.21.15.189:0/1470390413 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0288004850 0x7f029c136580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:10.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.824+0000 7f029bfff700 1 -- 172.21.15.189:0/1470390413 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0288004850 msgr2=0x7f029c136580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:10.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.824+0000 7f029bfff700 1 --2- 172.21.15.189:0/1470390413 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0288004850 0x7f029c136580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.824+0000 7f029bfff700 1 -- 172.21.15.189:0/1470390413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f029c1f6c00 msgr2=0x7f029c1389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:10.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.824+0000 7f029bfff700 1 --2- 172.21.15.189:0/1470390413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f029c1f6c00 0x7f029c1389b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:10.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.824+0000 7f029bfff700 1 -- 172.21.15.189:0/1470390413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c1f6400 con 0x7f029c13f4a0 2024-04-06T13:47:10.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.824+0000 7f029affd700 1 --2- 172.21.15.189:0/1470390413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f029c1f6c00 0x7f029c1389b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:10.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.824+0000 7f029bfff700 1 --2- 172.21.15.189:0/1470390413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f029c13f4a0 0x7f029c15a6a0 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f029c067550 tx=0x7f029400d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:10.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.825+0000 7f0293fff700 1 -- 172.21.15.189:0/1470390413 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0294003c10 con 0x7f029c13f4a0 2024-04-06T13:47:10.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.825+0000 7f0293fff700 1 -- 172.21.15.189:0/1470390413 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0294003d90 con 0x7f029c13f4a0 2024-04-06T13:47:10.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.825+0000 7f02a1df1700 1 -- 172.21.15.189:0/1470390413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f029c1f66e0 con 0x7f029c13f4a0 2024-04-06T13:47:10.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.825+0000 7f0293fff700 1 -- 172.21.15.189:0/1470390413 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029400d950 con 0x7f029c13f4a0 2024-04-06T13:47:10.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.825+0000 7f02a1df1700 1 -- 172.21.15.189:0/1470390413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029c138ef0 con 0x7f029c13f4a0 2024-04-06T13:47:10.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.826+0000 7f02a1df1700 1 -- 172.21.15.189:0/1470390413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f029c067bb0 con 0x7f029c13f4a0 2024-04-06T13:47:10.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.826+0000 7f0293fff700 1 -- 172.21.15.189:0/1470390413 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 136) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f029400dab0 con 0x7f029c13f4a0 2024-04-06T13:47:10.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.826+0000 7f0293fff700 1 -- 172.21.15.189:0/1470390413 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6904+0+0 (secure 0 0 0) 0x7f02940893b0 con 0x7f029c13f4a0 2024-04-06T13:47:10.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:10.831+0000 7f0293fff700 1 -- 172.21.15.189:0/1470390413 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0294054530 con 0x7f029c13f4a0 2024-04-06T13:47:11.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.064+0000 7f02a1df1700 1 -- 172.21.15.189:0/1470390413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f029c1f68d0 con 0x7f029c13f4a0 2024-04-06T13:47:11.065 INFO:tasks.ceph.mon.a.smithi189.stderr:2024-04-06T13:47:11.065+0000 7fa9fe1ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T13:47:11.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.075+0000 7f0293fff700 1 -- 172.21.15.189:0/1470390413 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v94) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0294057b50 con 0x7f029c13f4a0 2024-04-06T13:47:11.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.078+0000 7f02a1df1700 1 -- 172.21.15.189:0/1470390413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f029c13f4a0 msgr2=0x7f029c15a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:11.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.078+0000 7f02a1df1700 1 --2- 172.21.15.189:0/1470390413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f029c13f4a0 0x7f029c15a6a0 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f029c067550 tx=0x7f029400d150 comp rx=0 tx=0).stop 2024-04-06T13:47:11.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.078+0000 7f02a1df1700 1 -- 172.21.15.189:0/1470390413 shutdown_connections 2024-04-06T13:47:11.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.078+0000 7f02a1df1700 1 --2- 172.21.15.189:0/1470390413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f029c13f4a0 0x7f029c15a6a0 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:11.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.078+0000 7f02a1df1700 1 --2- 172.21.15.189:0/1470390413 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0288004850 0x7f029c136580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:11.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.078+0000 7f02a1df1700 1 --2- 172.21.15.189:0/1470390413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f029c1f6c00 0x7f029c1389b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:11.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.078+0000 7f02a1df1700 1 -- 172.21.15.189:0/1470390413 >> 172.21.15.189:0/1470390413 conn(0x7f029c0d00b0 msgr2=0x7f029c06f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:11.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.078+0000 7f02a1df1700 1 -- 172.21.15.189:0/1470390413 shutdown_connections 2024-04-06T13:47:11.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.078+0000 7f02a1df1700 1 -- 172.21.15.189:0/1470390413 wait complete. 2024-04-06T13:47:11.081 INFO:teuthology.orchestra.run.smithi189.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-06T13:47:11.095 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-06T13:47:11.122 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2024-04-06T13:47:11.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.221+0000 7f347f723700 1 Processor -- start 2024-04-06T13:47:11.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.221+0000 7f347f723700 1 -- start start 2024-04-06T13:47:11.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.222+0000 7f347f723700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f347815e8a0 0x7f34780cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:11.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.222+0000 7f347f723700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34780669d0 con 0x7f347815eee0 2024-04-06T13:47:11.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.222+0000 7f347f723700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347806ede0 con 0x7f347815fa70 2024-04-06T13:47:11.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.222+0000 7f347f723700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f347815ec80 con 0x7f347815e8a0 2024-04-06T13:47:11.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.222+0000 7f347d4bf700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f347815e8a0 0x7f34780cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:11.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.222+0000 7f347ccbe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f347815eee0 0x7f347815f360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42814/0 (socket says 172.21.15.189:42814) 2024-04-06T13:47:11.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.222+0000 7f347d4bf700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f347815e8a0 0x7f34780cbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38480/0 (socket says 172.21.15.189:38480) 2024-04-06T13:47:11.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.222+0000 7f347ccbe700 1 -- 172.21.15.189:0/1021115514 learned_addr learned my addr 172.21.15.189:0/1021115514 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:11.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.222+0000 7f347d4bf700 1 -- 172.21.15.189:0/1021115514 >> v1:172.21.15.192:6789/0 conn(0x7f347815fa70 legacy=0x7f347806aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:11.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.222+0000 7f347d4bf700 1 -- 172.21.15.189:0/1021115514 >> v1:172.21.15.189:6789/0 conn(0x7f347815eee0 legacy=0x7f347815f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:11.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.222+0000 7f346ffff700 1 -- 172.21.15.189:0/1021115514 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392155808 0 0) 0x7f34780669d0 con 0x7f347815eee0 2024-04-06T13:47:11.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.222+0000 7f347d4bf700 1 -- 172.21.15.189:0/1021115514 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34780666f0 con 0x7f347815e8a0 2024-04-06T13:47:11.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.222+0000 7f347d4bf700 1 --2- 172.21.15.189:0/1021115514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f347815e8a0 0x7f34780cbc20 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f3474009840 tx=0x7f3474009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7add06709e3175f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:11.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.223+0000 7f346ffff700 1 -- 172.21.15.189:0/1021115514 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474011830 con 0x7f347815e8a0 2024-04-06T13:47:11.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.223+0000 7f346ffff700 1 -- 172.21.15.189:0/1021115514 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3474011990 con 0x7f347815e8a0 2024-04-06T13:47:11.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.223+0000 7f346ffff700 1 -- 172.21.15.189:0/1021115514 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474011d20 con 0x7f347815e8a0 2024-04-06T13:47:11.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.223+0000 7f347f723700 1 -- 172.21.15.189:0/1021115514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f347815e8a0 msgr2=0x7f34780cbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:11.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.223+0000 7f347f723700 1 --2- 172.21.15.189:0/1021115514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f347815e8a0 0x7f34780cbc20 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f3474009840 tx=0x7f3474009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:11.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.223+0000 7f347f723700 1 -- 172.21.15.189:0/1021115514 shutdown_connections 2024-04-06T13:47:11.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.223+0000 7f347f723700 1 --2- 172.21.15.189:0/1021115514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f347815e8a0 0x7f34780cbc20 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:11.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.223+0000 7f347f723700 1 -- 172.21.15.189:0/1021115514 >> 172.21.15.189:0/1021115514 conn(0x7f34780cc290 msgr2=0x7f34780cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:11.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.223+0000 7f347f723700 1 -- 172.21.15.189:0/1021115514 shutdown_connections 2024-04-06T13:47:11.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.223+0000 7f347f723700 1 -- 172.21.15.189:0/1021115514 wait complete. 2024-04-06T13:47:11.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.224+0000 7f347f723700 1 Processor -- start 2024-04-06T13:47:11.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.224+0000 7f347f723700 1 -- start start 2024-04-06T13:47:11.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.225+0000 7f347f723700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f347815e8a0 0x7f34781f2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:11.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.225+0000 7f347f723700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f347815eee0 0x7f34780b2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:11.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.225+0000 7f347f723700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f347815fa70 0x7f34780b3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:11.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.225+0000 7f347f723700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34781f39e0 con 0x7f347815fa70 2024-04-06T13:47:11.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.225+0000 7f347f723700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34780b3650 con 0x7f347815eee0 2024-04-06T13:47:11.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.225+0000 7f347f723700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34781f3550 con 0x7f347815e8a0 2024-04-06T13:47:11.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.225+0000 7f347dcc0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f347815fa70 0x7f34780b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:11.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.225+0000 7f347dcc0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f347815fa70 0x7f34780b3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38484/0 (socket says 172.21.15.189:38484) 2024-04-06T13:47:11.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.225+0000 7f347dcc0700 1 -- 172.21.15.189:0/3050897870 learned_addr learned my addr 172.21.15.189:0/3050897870 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:11.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.225+0000 7f347d4bf700 1 --2- 172.21.15.189:0/3050897870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f347815e8a0 0x7f34781f2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:11.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.225+0000 7f347ccbe700 1 --2- 172.21.15.189:0/3050897870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f347815eee0 0x7f34780b2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:11.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.226+0000 7f347dcc0700 1 -- 172.21.15.189:0/3050897870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f347815e8a0 msgr2=0x7f34781f2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:11.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.226+0000 7f347dcc0700 1 --2- 172.21.15.189:0/3050897870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f347815e8a0 0x7f34781f2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:11.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.226+0000 7f347dcc0700 1 -- 172.21.15.189:0/3050897870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f347815eee0 msgr2=0x7f34780b2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:11.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.226+0000 7f347dcc0700 1 --2- 172.21.15.189:0/3050897870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f347815eee0 0x7f34780b2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:11.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.226+0000 7f347dcc0700 1 -- 172.21.15.189:0/3050897870 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34740094a0 con 0x7f347815fa70 2024-04-06T13:47:11.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.226+0000 7f347dcc0700 1 --2- 172.21.15.189:0/3050897870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f347815fa70 0x7f34780b3020 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7f347000cba0 tx=0x7f347000ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:11.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.226+0000 7f346dffb700 1 -- 172.21.15.189:0/3050897870 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3470003c00 con 0x7f347815fa70 2024-04-06T13:47:11.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.226+0000 7f346dffb700 1 -- 172.21.15.189:0/3050897870 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3470003d60 con 0x7f347815fa70 2024-04-06T13:47:11.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.226+0000 7f347f723700 1 -- 172.21.15.189:0/3050897870 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3478158670 con 0x7f347815fa70 2024-04-06T13:47:11.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.226+0000 7f346dffb700 1 -- 172.21.15.189:0/3050897870 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347000aa60 con 0x7f347815fa70 2024-04-06T13:47:11.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.226+0000 7f347f723700 1 -- 172.21.15.189:0/3050897870 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34781f2f00 con 0x7f347815fa70 2024-04-06T13:47:11.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.227+0000 7f347f723700 1 -- 172.21.15.189:0/3050897870 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3478072140 con 0x7f347815fa70 2024-04-06T13:47:11.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.228+0000 7f346dffb700 1 -- 172.21.15.189:0/3050897870 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 136) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f347000abc0 con 0x7f347815fa70 2024-04-06T13:47:11.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.228+0000 7f346dffb700 1 -- 172.21.15.189:0/3050897870 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6990+0+0 (secure 0 0 0) 0x7f34700891c0 con 0x7f347815fa70 2024-04-06T13:47:11.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.232+0000 7f346dffb700 1 -- 172.21.15.189:0/3050897870 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f34700542f0 con 0x7f347815fa70 2024-04-06T13:47:11.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.401+0000 7f346dffb700 1 -- 172.21.15.189:0/3050897870 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 137) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f3470053da0 con 0x7f347815fa70 2024-04-06T13:47:11.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.402+0000 7f346dffb700 1 --2- 172.21.15.189:0/3050897870 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f345c036150 0x7f345c072d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:11.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.402+0000 7f347d4bf700 1 --2- 172.21.15.189:0/3050897870 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f345c036150 0x7f345c072d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:11.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.404+0000 7f347d4bf700 1 --2- 172.21.15.189:0/3050897870 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f345c036150 0x7f345c072d20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f34740090b0 tx=0x7f347401c040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:11.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:11.478+0000 7f347f723700 1 -- 172.21.15.189:0/3050897870 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f34781f36c0 con 0x7f347815fa70 2024-04-06T13:47:12.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.077+0000 7f346dffb700 1 -- 172.21.15.189:0/3050897870 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v95) v1 ==== 104+0+0 (secure 0 0 0) 0x7f34700719e0 con 0x7f347815fa70 2024-04-06T13:47:12.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.080+0000 7f347f723700 1 -- 172.21.15.189:0/3050897870 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f345c036150 msgr2=0x7f345c072d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:12.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.080+0000 7f347f723700 1 --2- 172.21.15.189:0/3050897870 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f345c036150 0x7f345c072d20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f34740090b0 tx=0x7f347401c040 comp rx=0 tx=0).stop 2024-04-06T13:47:12.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.080+0000 7f347f723700 1 -- 172.21.15.189:0/3050897870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f347815fa70 msgr2=0x7f34780b3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:12.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.080+0000 7f347f723700 1 --2- 172.21.15.189:0/3050897870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f347815fa70 0x7f34780b3020 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7f347000cba0 tx=0x7f347000ceb0 comp rx=0 tx=0).stop 2024-04-06T13:47:12.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.080+0000 7f347f723700 1 -- 172.21.15.189:0/3050897870 shutdown_connections 2024-04-06T13:47:12.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.080+0000 7f347f723700 1 --2- 172.21.15.189:0/3050897870 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f345c036150 0x7f345c072d20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:12.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.080+0000 7f347f723700 1 --2- 172.21.15.189:0/3050897870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f347815fa70 0x7f34780b3020 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:12.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.080+0000 7f347f723700 1 --2- 172.21.15.189:0/3050897870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f347815e8a0 0x7f34781f2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:12.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.080+0000 7f347f723700 1 --2- 172.21.15.189:0/3050897870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f347815eee0 0x7f34780b2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:12.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.080+0000 7f347f723700 1 -- 172.21.15.189:0/3050897870 >> 172.21.15.189:0/3050897870 conn(0x7f34780cc290 msgr2=0x7f34780b07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:12.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.080+0000 7f347f723700 1 -- 172.21.15.189:0/3050897870 shutdown_connections 2024-04-06T13:47:12.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.080+0000 7f347f723700 1 -- 172.21.15.189:0/3050897870 wait complete. 2024-04-06T13:47:12.095 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-06T13:47:12.264 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.262+0000 7fa733ae5700 1 Processor -- start 2024-04-06T13:47:12.264 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.263+0000 7fa733ae5700 1 -- start start 2024-04-06T13:47:12.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.264+0000 7fa733ae5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa72c146dd0 0x7fa72c06d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:12.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.264+0000 7fa733ae5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72c06c0a0 con 0x7fa72c0d17a0 2024-04-06T13:47:12.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.264+0000 7fa733ae5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72c06c380 con 0x7fa72c0cedf0 2024-04-06T13:47:12.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.264+0000 7fa733ae5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa72c06beb0 con 0x7fa72c146dd0 2024-04-06T13:47:12.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.264+0000 7fa731080700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa72c146dd0 0x7fa72c06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:12.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.264+0000 7fa731881700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa72c0d17a0 0x7fa72c06b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42818/0 (socket says 172.21.15.189:42818) 2024-04-06T13:47:12.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.264+0000 7fa731881700 1 -- 172.21.15.189:0/391178933 learned_addr learned my addr 172.21.15.189:0/391178933 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:12.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.264+0000 7fa731080700 1 -- 172.21.15.189:0/391178933 >> v1:172.21.15.192:6789/0 conn(0x7fa72c0cedf0 legacy=0x7fa72c0cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:12.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.264+0000 7fa731080700 1 -- 172.21.15.189:0/391178933 >> v1:172.21.15.189:6789/0 conn(0x7fa72c0d17a0 legacy=0x7fa72c06b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:12.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.264+0000 7fa731080700 1 -- 172.21.15.189:0/391178933 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72c06c660 con 0x7fa72c146dd0 2024-04-06T13:47:12.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.264+0000 7fa731080700 1 --2- 172.21.15.189:0/391178933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa72c146dd0 0x7fa72c06d140 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7fa728009840 tx=0x7fa728009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98be72dd9bd6718 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:12.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.264+0000 7fa73087f700 1 -- 172.21.15.189:0/391178933 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa728004230 con 0x7fa72c146dd0 2024-04-06T13:47:12.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.264+0000 7fa73087f700 1 -- 172.21.15.189:0/391178933 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa728004390 con 0x7fa72c146dd0 2024-04-06T13:47:12.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.264+0000 7fa73087f700 1 -- 172.21.15.189:0/391178933 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7280036a0 con 0x7fa72c146dd0 2024-04-06T13:47:12.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.265+0000 7fa733ae5700 1 -- 172.21.15.189:0/391178933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa72c146dd0 msgr2=0x7fa72c06d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:12.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.265+0000 7fa733ae5700 1 --2- 172.21.15.189:0/391178933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa72c146dd0 0x7fa72c06d140 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7fa728009840 tx=0x7fa728009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:12.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.265+0000 7fa733ae5700 1 -- 172.21.15.189:0/391178933 shutdown_connections 2024-04-06T13:47:12.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.265+0000 7fa733ae5700 1 --2- 172.21.15.189:0/391178933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa72c146dd0 0x7fa72c06d140 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:12.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.265+0000 7fa733ae5700 1 -- 172.21.15.189:0/391178933 >> 172.21.15.189:0/391178933 conn(0x7fa72c0b2db0 msgr2=0x7fa72c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:12.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.265+0000 7fa733ae5700 1 -- 172.21.15.189:0/391178933 shutdown_connections 2024-04-06T13:47:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.265+0000 7fa733ae5700 1 -- 172.21.15.189:0/391178933 wait complete. 2024-04-06T13:47:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.266+0000 7fa733ae5700 1 Processor -- start 2024-04-06T13:47:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.266+0000 7fa733ae5700 1 -- start start 2024-04-06T13:47:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.267+0000 7fa733ae5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa72c0cedf0 0x7fa72c1f6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.267+0000 7fa733ae5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa72c0d17a0 0x7fa72c0cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.267+0000 7fa733ae5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa72c146dd0 0x7fa72c160050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.267+0000 7fa733ae5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c1f3bc0 con 0x7fa72c0cedf0 2024-04-06T13:47:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.267+0000 7fa733ae5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c1f7400 con 0x7fa72c0d17a0 2024-04-06T13:47:12.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.267+0000 7fa733ae5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c0b0740 con 0x7fa72c146dd0 2024-04-06T13:47:12.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.267+0000 7fa731881700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa72c0cedf0 0x7fa72c1f6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:12.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.267+0000 7fa731881700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa72c0cedf0 0x7fa72c1f6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38498/0 (socket says 172.21.15.189:38498) 2024-04-06T13:47:12.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.267+0000 7fa731881700 1 -- 172.21.15.189:0/4042648991 learned_addr learned my addr 172.21.15.189:0/4042648991 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:12.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.267+0000 7fa731080700 1 --2- 172.21.15.189:0/4042648991 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa72c0d17a0 0x7fa72c0cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:12.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.267+0000 7fa732082700 1 --2- 172.21.15.189:0/4042648991 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa72c146dd0 0x7fa72c160050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:12.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.267+0000 7fa731881700 1 -- 172.21.15.189:0/4042648991 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa72c146dd0 msgr2=0x7fa72c160050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:12.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.267+0000 7fa731881700 1 --2- 172.21.15.189:0/4042648991 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa72c146dd0 0x7fa72c160050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:12.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.267+0000 7fa731881700 1 -- 172.21.15.189:0/4042648991 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa72c0d17a0 msgr2=0x7fa72c0cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:12.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.267+0000 7fa731881700 1 --2- 172.21.15.189:0/4042648991 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa72c0d17a0 0x7fa72c0cdc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:12.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.267+0000 7fa731881700 1 -- 172.21.15.189:0/4042648991 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7280094a0 con 0x7fa72c0cedf0 2024-04-06T13:47:12.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.267+0000 7fa731881700 1 --2- 172.21.15.189:0/4042648991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa72c0cedf0 0x7fa72c1f6ec0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7fa72000c4c0 tx=0x7fa72000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:12.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.268+0000 7fa71e7fc700 1 -- 172.21.15.189:0/4042648991 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa720005d70 con 0x7fa72c0cedf0 2024-04-06T13:47:12.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.268+0000 7fa71e7fc700 1 -- 172.21.15.189:0/4042648991 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7200070d0 con 0x7fa72c0cedf0 2024-04-06T13:47:12.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.268+0000 7fa733ae5700 1 -- 172.21.15.189:0/4042648991 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa72c1f3220 con 0x7fa72c0cedf0 2024-04-06T13:47:12.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.268+0000 7fa71e7fc700 1 -- 172.21.15.189:0/4042648991 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa720007460 con 0x7fa72c0cedf0 2024-04-06T13:47:12.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.268+0000 7fa733ae5700 1 -- 172.21.15.189:0/4042648991 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa72c1f35b0 con 0x7fa72c0cedf0 2024-04-06T13:47:12.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.269+0000 7fa733ae5700 1 -- 172.21.15.189:0/4042648991 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa72c074990 con 0x7fa72c0cedf0 2024-04-06T13:47:12.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.269+0000 7fa71e7fc700 1 -- 172.21.15.189:0/4042648991 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 137) v1 ==== 88983+0+0 (secure 0 0 0) 0x7fa7200075c0 con 0x7fa72c0cedf0 2024-04-06T13:47:12.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.269+0000 7fa71e7fc700 1 --2- 172.21.15.189:0/4042648991 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa71006aae0 0x7fa71006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:12.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.270+0000 7fa71e7fc700 1 -- 172.21.15.189:0/4042648991 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6990+0+0 (secure 0 0 0) 0x7fa7200888d0 con 0x7fa72c0cedf0 2024-04-06T13:47:12.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.270+0000 7fa731080700 1 --2- 172.21.15.189:0/4042648991 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa71006aae0 0x7fa71006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:12.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.270+0000 7fa731080700 1 --2- 172.21.15.189:0/4042648991 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa71006aae0 0x7fa71006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa72c1471e0 tx=0x7fa72801c040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:12.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.274+0000 7fa71e7fc700 1 -- 172.21.15.189:0/4042648991 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa7200538f0 con 0x7fa72c0cedf0 2024-04-06T13:47:12.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.481+0000 7fa733ae5700 1 -- 172.21.15.189:0/4042648991 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa72c0d1e20 con 0x7fa72c0cedf0 2024-04-06T13:47:12.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.581+0000 7fa71e7fc700 1 -- 172.21.15.189:0/4042648991 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v217) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa7200170d0 con 0x7fa72c0cedf0 2024-04-06T13:47:12.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.586+0000 7fa733ae5700 1 -- 172.21.15.189:0/4042648991 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa71006aae0 msgr2=0x7fa71006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:12.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.586+0000 7fa733ae5700 1 --2- 172.21.15.189:0/4042648991 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa71006aae0 0x7fa71006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa72c1471e0 tx=0x7fa72801c040 comp rx=0 tx=0).stop 2024-04-06T13:47:12.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.586+0000 7fa733ae5700 1 -- 172.21.15.189:0/4042648991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa72c0cedf0 msgr2=0x7fa72c1f6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:12.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.586+0000 7fa733ae5700 1 --2- 172.21.15.189:0/4042648991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa72c0cedf0 0x7fa72c1f6ec0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7fa72000c4c0 tx=0x7fa72000c880 comp rx=0 tx=0).stop 2024-04-06T13:47:12.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.586+0000 7fa733ae5700 1 -- 172.21.15.189:0/4042648991 shutdown_connections 2024-04-06T13:47:12.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.586+0000 7fa733ae5700 1 --2- 172.21.15.189:0/4042648991 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa71006aae0 0x7fa71006cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:12.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.586+0000 7fa733ae5700 1 --2- 172.21.15.189:0/4042648991 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa72c146dd0 0x7fa72c160050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:12.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.586+0000 7fa733ae5700 1 --2- 172.21.15.189:0/4042648991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa72c0cedf0 0x7fa72c1f6ec0 secure :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7fa72000c4c0 tx=0x7fa72000c880 comp rx=0 tx=0).stop 2024-04-06T13:47:12.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.586+0000 7fa733ae5700 1 --2- 172.21.15.189:0/4042648991 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa72c0d17a0 0x7fa72c0cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:12.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.586+0000 7fa733ae5700 1 -- 172.21.15.189:0/4042648991 >> 172.21.15.189:0/4042648991 conn(0x7fa72c0b2db0 msgr2=0x7fa72c0b5030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:12.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.587+0000 7fa733ae5700 1 -- 172.21.15.189:0/4042648991 shutdown_connections 2024-04-06T13:47:12.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.587+0000 7fa733ae5700 1 -- 172.21.15.189:0/4042648991 wait complete. 2024-04-06T13:47:12.589 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' removed 2024-04-06T13:47:12.601 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-06T13:47:12.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.725+0000 7fc04131d700 1 Processor -- start 2024-04-06T13:47:12.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.725+0000 7fc04131d700 1 -- start start 2024-04-06T13:47:12.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.726+0000 7fc04131d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc03c15a550 0x7fc03c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:12.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.726+0000 7fc04131d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc03c0cf900 0x7fc03c0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:12.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.726+0000 7fc04131d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c0cdbe0 con 0x7fc03c0b4ba0 2024-04-06T13:47:12.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.726+0000 7fc04131d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc03c15a940 con 0x7fc03c15a550 2024-04-06T13:47:12.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.726+0000 7fc04131d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc03c15ac00 con 0x7fc03c0cf900 2024-04-06T13:47:12.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.726+0000 7fc03affd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc03c15a550 0x7fc03c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:12.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.726+0000 7fc03affd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc03c15a550 0x7fc03c0cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38526/0 (socket says 172.21.15.189:38526) 2024-04-06T13:47:12.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.726+0000 7fc03affd700 1 -- 172.21.15.189:0/3875671401 learned_addr learned my addr 172.21.15.189:0/3875671401 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:12.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.727+0000 7fc03affd700 1 -- 172.21.15.189:0/3875671401 >> v2:172.21.15.192:3300/0 conn(0x7fc03c0cf900 msgr2=0x7fc03c0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:12.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.727+0000 7fc03affd700 1 --2- 172.21.15.189:0/3875671401 >> v2:172.21.15.192:3300/0 conn(0x7fc03c0cf900 0x7fc03c0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:12.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.727+0000 7fc03affd700 1 -- 172.21.15.189:0/3875671401 >> v1:172.21.15.189:6789/0 conn(0x7fc03c0b4ba0 legacy=0x7fc03c06eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:12.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.727+0000 7fc03affd700 1 -- 172.21.15.189:0/3875671401 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c15aee0 con 0x7fc03c15a550 2024-04-06T13:47:12.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.727+0000 7fc03affd700 1 --2- 172.21.15.189:0/3875671401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc03c15a550 0x7fc03c0cf3c0 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7fc03c0b2d40 tx=0x7fc030009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38517f03fb8e8039 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:12.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.727+0000 7fc039ffb700 1 -- 172.21.15.189:0/3875671401 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03000b870 con 0x7fc03c15a550 2024-04-06T13:47:12.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.727+0000 7fc039ffb700 1 -- 172.21.15.189:0/3875671401 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03000b9d0 con 0x7fc03c15a550 2024-04-06T13:47:12.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.727+0000 7fc039ffb700 1 -- 172.21.15.189:0/3875671401 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03000bd60 con 0x7fc03c15a550 2024-04-06T13:47:12.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.727+0000 7fc04131d700 1 -- 172.21.15.189:0/3875671401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc03c15a550 msgr2=0x7fc03c0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:12.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.727+0000 7fc04131d700 1 --2- 172.21.15.189:0/3875671401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc03c15a550 0x7fc03c0cf3c0 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7fc03c0b2d40 tx=0x7fc030009bd0 comp rx=0 tx=0).stop 2024-04-06T13:47:12.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.728+0000 7fc04131d700 1 -- 172.21.15.189:0/3875671401 shutdown_connections 2024-04-06T13:47:12.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.728+0000 7fc04131d700 1 --2- 172.21.15.189:0/3875671401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc03c15a550 0x7fc03c0cf3c0 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:12.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.728+0000 7fc04131d700 1 --2- 172.21.15.189:0/3875671401 >> v2:172.21.15.192:3300/0 conn(0x7fc03c0cf900 0x7fc03c0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:12.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.728+0000 7fc04131d700 1 -- 172.21.15.189:0/3875671401 >> 172.21.15.189:0/3875671401 conn(0x7fc03c0d00b0 msgr2=0x7fc03c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:12.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.728+0000 7fc04131d700 1 -- 172.21.15.189:0/3875671401 shutdown_connections 2024-04-06T13:47:12.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.728+0000 7fc04131d700 1 -- 172.21.15.189:0/3875671401 wait complete. 2024-04-06T13:47:12.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.729+0000 7fc04131d700 1 Processor -- start 2024-04-06T13:47:12.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.729+0000 7fc04131d700 1 -- start start 2024-04-06T13:47:12.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc04131d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc03c0b4ba0 0x7fc03c14c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:12.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc04131d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc03c0cf900 0x7fc03c0b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:12.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc04131d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc03c14f730 0x7fc03c0ae840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:12.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc04131d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c146930 con 0x7fc03c0b4ba0 2024-04-06T13:47:12.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc04131d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c14cf30 con 0x7fc03c14f730 2024-04-06T13:47:12.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc04131d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c14d210 con 0x7fc03c0cf900 2024-04-06T13:47:12.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc03affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc03c0b4ba0 0x7fc03c14c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:12.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc03affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc03c0b4ba0 0x7fc03c14c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38530/0 (socket says 172.21.15.189:38530) 2024-04-06T13:47:12.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc03affd700 1 -- 172.21.15.189:0/3023366759 learned_addr learned my addr 172.21.15.189:0/3023366759 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:12.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc03affd700 1 -- 172.21.15.189:0/3023366759 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc03c0cf900 msgr2=0x7fc03c0b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:12.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc03affd700 1 --2- 172.21.15.189:0/3023366759 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc03c0cf900 0x7fc03c0b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:12.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc03affd700 1 -- 172.21.15.189:0/3023366759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc03c14f730 msgr2=0x7fc03c0ae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:12.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc03affd700 1 --2- 172.21.15.189:0/3023366759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc03c14f730 0x7fc03c0ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:12.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc03affd700 1 -- 172.21.15.189:0/3023366759 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc030009550 con 0x7fc03c0b4ba0 2024-04-06T13:47:12.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc03affd700 1 --2- 172.21.15.189:0/3023366759 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc03c0b4ba0 0x7fc03c14c5e0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7fc030014040 tx=0x7fc030004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:12.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc0237fe700 1 -- 172.21.15.189:0/3023366759 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc030004520 con 0x7fc03c0b4ba0 2024-04-06T13:47:12.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc0237fe700 1 -- 172.21.15.189:0/3023366759 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03001e070 con 0x7fc03c0b4ba0 2024-04-06T13:47:12.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc04131d700 1 -- 172.21.15.189:0/3023366759 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc03c14d4f0 con 0x7fc03c0b4ba0 2024-04-06T13:47:12.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc0237fe700 1 -- 172.21.15.189:0/3023366759 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0300038c0 con 0x7fc03c0b4ba0 2024-04-06T13:47:12.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.730+0000 7fc04131d700 1 -- 172.21.15.189:0/3023366759 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03c14cb20 con 0x7fc03c0b4ba0 2024-04-06T13:47:12.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.731+0000 7fc04131d700 1 -- 172.21.15.189:0/3023366759 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03c067bb0 con 0x7fc03c0b4ba0 2024-04-06T13:47:12.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.732+0000 7fc0237fe700 1 -- 172.21.15.189:0/3023366759 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 137) v1 ==== 88983+0+0 (secure 0 0 0) 0x7fc030003a20 con 0x7fc03c0b4ba0 2024-04-06T13:47:12.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.732+0000 7fc0237fe700 1 --2- 172.21.15.189:0/3023366759 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc02806aae0 0x7fc02806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:12.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.732+0000 7fc0237fe700 1 -- 172.21.15.189:0/3023366759 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6552+0+0 (secure 0 0 0) 0x7fc030020070 con 0x7fc03c0b4ba0 2024-04-06T13:47:12.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.732+0000 7fc03a7fc700 1 --2- 172.21.15.189:0/3023366759 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc02806aae0 0x7fc02806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:12.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.732+0000 7fc03a7fc700 1 --2- 172.21.15.189:0/3023366759 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc02806aae0 0x7fc02806cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc03c149810 tx=0x7fc02c006d20 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:12.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.737+0000 7fc0237fe700 1 -- 172.21.15.189:0/3023366759 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc030055b50 con 0x7fc03c0b4ba0 2024-04-06T13:47:12.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:12.931+0000 7fc04131d700 1 -- 172.21.15.189:0/3023366759 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc03c0b51c0 con 0x7fc03c0b4ba0 2024-04-06T13:47:13.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.409+0000 7fc0237fe700 1 -- 172.21.15.189:0/3023366759 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 138) v1 ==== 89191+0+0 (secure 0 0 0) 0x7fc030055f20 con 0x7fc03c0b4ba0 2024-04-06T13:47:13.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.618+0000 7fc0237fe700 1 -- 172.21.15.189:0/3023366759 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v218) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc03006e7a0 con 0x7fc03c0b4ba0 2024-04-06T13:47:13.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.623+0000 7fc04131d700 1 -- 172.21.15.189:0/3023366759 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc02806aae0 msgr2=0x7fc02806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:13.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.623+0000 7fc04131d700 1 --2- 172.21.15.189:0/3023366759 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc02806aae0 0x7fc02806cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc03c149810 tx=0x7fc02c006d20 comp rx=0 tx=0).stop 2024-04-06T13:47:13.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.623+0000 7fc04131d700 1 -- 172.21.15.189:0/3023366759 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc03c0b4ba0 msgr2=0x7fc03c14c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:13.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.623+0000 7fc04131d700 1 --2- 172.21.15.189:0/3023366759 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc03c0b4ba0 0x7fc03c14c5e0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7fc030014040 tx=0x7fc030004750 comp rx=0 tx=0).stop 2024-04-06T13:47:13.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.623+0000 7fc04131d700 1 -- 172.21.15.189:0/3023366759 shutdown_connections 2024-04-06T13:47:13.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.623+0000 7fc04131d700 1 --2- 172.21.15.189:0/3023366759 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc02806aae0 0x7fc02806cfa0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc03c149810 tx=0x7fc02c006d20 comp rx=0 tx=0).stop 2024-04-06T13:47:13.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.623+0000 7fc04131d700 1 --2- 172.21.15.189:0/3023366759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc03c14f730 0x7fc03c0ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:13.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.623+0000 7fc04131d700 1 --2- 172.21.15.189:0/3023366759 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc03c0b4ba0 0x7fc03c14c5e0 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:13.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.623+0000 7fc04131d700 1 --2- 172.21.15.189:0/3023366759 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc03c0cf900 0x7fc03c0b1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:13.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.623+0000 7fc04131d700 1 -- 172.21.15.189:0/3023366759 >> 172.21.15.189:0/3023366759 conn(0x7fc03c0d00b0 msgr2=0x7fc03c151ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:13.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.624+0000 7fc04131d700 1 -- 172.21.15.189:0/3023366759 shutdown_connections 2024-04-06T13:47:13.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.625+0000 7fc04131d700 1 -- 172.21.15.189:0/3023366759 wait complete. 2024-04-06T13:47:13.628 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' removed 2024-04-06T13:47:13.643 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-06T13:47:13.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.770+0000 7f201a273700 1 Processor -- start 2024-04-06T13:47:13.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.770+0000 7f201a273700 1 -- start start 2024-04-06T13:47:13.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.771+0000 7f201a273700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f201406ae40 0x7f20140b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:13.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.771+0000 7f201a273700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20140b3990 con 0x7f201415ddf0 2024-04-06T13:47:13.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.771+0000 7f201a273700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20140b3060 con 0x7f201415d7b0 2024-04-06T13:47:13.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.771+0000 7f201a273700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f201415e950 con 0x7f201406ae40 2024-04-06T13:47:13.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.771+0000 7f20137fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f201406ae40 0x7f20140b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:13.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.771+0000 7f20137fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f201406ae40 0x7f20140b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38534/0 (socket says 172.21.15.189:38534) 2024-04-06T13:47:13.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.771+0000 7f2018810700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f201415ddf0 0x7f201415e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42826/0 (socket says 172.21.15.189:42826) 2024-04-06T13:47:13.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.771+0000 7f20137fe700 1 -- 172.21.15.189:0/1511460122 learned_addr learned my addr 172.21.15.189:0/1511460122 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:13.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.771+0000 7f20137fe700 1 -- 172.21.15.189:0/1511460122 >> v1:172.21.15.192:6789/0 conn(0x7f201415d7b0 legacy=0x7f201406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:13.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.771+0000 7f20137fe700 1 -- 172.21.15.189:0/1511460122 >> v1:172.21.15.189:6789/0 conn(0x7f201415ddf0 legacy=0x7f201415e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:13.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.771+0000 7f20137fe700 1 -- 172.21.15.189:0/1511460122 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20140b3340 con 0x7f201406ae40 2024-04-06T13:47:13.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.771+0000 7f20137fe700 1 --2- 172.21.15.189:0/1511460122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f201406ae40 0x7f20140b3e20 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f20040098f0 tx=0x7f2004009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e97cf4326f182fbc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:13.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.771+0000 7f2012ffd700 1 -- 172.21.15.189:0/1511460122 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2004004230 con 0x7f201406ae40 2024-04-06T13:47:13.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.772+0000 7f2012ffd700 1 -- 172.21.15.189:0/1511460122 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2004004390 con 0x7f201406ae40 2024-04-06T13:47:13.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.772+0000 7f2012ffd700 1 -- 172.21.15.189:0/1511460122 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20040036a0 con 0x7f201406ae40 2024-04-06T13:47:13.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.772+0000 7f201a273700 1 -- 172.21.15.189:0/1511460122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f201406ae40 msgr2=0x7f20140b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:13.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.772+0000 7f201a273700 1 --2- 172.21.15.189:0/1511460122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f201406ae40 0x7f20140b3e20 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f20040098f0 tx=0x7f2004009c00 comp rx=0 tx=0).stop 2024-04-06T13:47:13.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.772+0000 7f201a273700 1 -- 172.21.15.189:0/1511460122 shutdown_connections 2024-04-06T13:47:13.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.772+0000 7f201a273700 1 --2- 172.21.15.189:0/1511460122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f201406ae40 0x7f20140b3e20 unknown :-1 s=CLOSED pgs=2346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:13.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.772+0000 7f201a273700 1 -- 172.21.15.189:0/1511460122 >> 172.21.15.189:0/1511460122 conn(0x7f20140d00b0 msgr2=0x7f20140d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:13.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.772+0000 7f201a273700 1 -- 172.21.15.189:0/1511460122 shutdown_connections 2024-04-06T13:47:13.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.772+0000 7f201a273700 1 -- 172.21.15.189:0/1511460122 wait complete. 2024-04-06T13:47:13.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.773+0000 7f201a273700 1 Processor -- start 2024-04-06T13:47:13.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.773+0000 7f201a273700 1 -- start start 2024-04-06T13:47:13.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.774+0000 7f201a273700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f201406ae40 0x7f20140aebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:13.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.774+0000 7f201a273700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f201415d7b0 0x7f20141455e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:13.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.774+0000 7f201a273700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f201415ddf0 0x7f20141422d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:13.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.774+0000 7f201a273700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20140af280 con 0x7f201415d7b0 2024-04-06T13:47:13.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.774+0000 7f201a273700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201415c140 con 0x7f201415ddf0 2024-04-06T13:47:13.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.774+0000 7f201a273700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2014145b20 con 0x7f201406ae40 2024-04-06T13:47:13.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.774+0000 7f20137fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f201415d7b0 0x7f20141455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:13.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.774+0000 7f20137fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f201415d7b0 0x7f20141455e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38540/0 (socket says 172.21.15.189:38540) 2024-04-06T13:47:13.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.774+0000 7f20137fe700 1 -- 172.21.15.189:0/2484844069 learned_addr learned my addr 172.21.15.189:0/2484844069 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:13.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.774+0000 7f2018810700 1 --2- 172.21.15.189:0/2484844069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f201415ddf0 0x7f20141422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:13.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.774+0000 7f2013fff700 1 --2- 172.21.15.189:0/2484844069 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f201406ae40 0x7f20140aebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:13.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.775+0000 7f20137fe700 1 -- 172.21.15.189:0/2484844069 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f201406ae40 msgr2=0x7f20140aebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:13.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.775+0000 7f20137fe700 1 --2- 172.21.15.189:0/2484844069 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f201406ae40 0x7f20140aebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:13.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.775+0000 7f20137fe700 1 -- 172.21.15.189:0/2484844069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f201415ddf0 msgr2=0x7f20141422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:13.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.775+0000 7f20137fe700 1 --2- 172.21.15.189:0/2484844069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f201415ddf0 0x7f20141422d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:13.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.775+0000 7f20137fe700 1 -- 172.21.15.189:0/2484844069 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2004009550 con 0x7f201415d7b0 2024-04-06T13:47:13.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.775+0000 7f2018810700 1 --2- 172.21.15.189:0/2484844069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f201415ddf0 0x7f20141422d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:13.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.775+0000 7f20137fe700 1 --2- 172.21.15.189:0/2484844069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f201415d7b0 0x7f20141455e0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f2004000c00 tx=0x7f2004004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:13.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.775+0000 7f2010ff9700 1 -- 172.21.15.189:0/2484844069 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2004003d70 con 0x7f201415d7b0 2024-04-06T13:47:13.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.775+0000 7f2010ff9700 1 -- 172.21.15.189:0/2484844069 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200401f070 con 0x7f201415d7b0 2024-04-06T13:47:13.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.775+0000 7f201a273700 1 -- 172.21.15.189:0/2484844069 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f201415c5d0 con 0x7f201415d7b0 2024-04-06T13:47:13.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.775+0000 7f2010ff9700 1 -- 172.21.15.189:0/2484844069 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200400b8a0 con 0x7f201415d7b0 2024-04-06T13:47:13.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.775+0000 7f201a273700 1 -- 172.21.15.189:0/2484844069 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f201415b810 con 0x7f201415d7b0 2024-04-06T13:47:13.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.776+0000 7f201a273700 1 -- 172.21.15.189:0/2484844069 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2014067bb0 con 0x7f201415d7b0 2024-04-06T13:47:13.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.776+0000 7f2010ff9700 1 -- 172.21.15.189:0/2484844069 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 138) v1 ==== 89191+0+0 (secure 0 0 0) 0x7f2004021030 con 0x7f201415d7b0 2024-04-06T13:47:13.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.777+0000 7f2010ff9700 1 --2- 172.21.15.189:0/2484844069 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ff406ad20 0x7f1ff406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:13.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.777+0000 7f2010ff9700 1 -- 172.21.15.189:0/2484844069 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6023+0+0 (secure 0 0 0) 0x7f200408ab20 con 0x7f201415d7b0 2024-04-06T13:47:13.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.777+0000 7f2013fff700 1 --2- 172.21.15.189:0/2484844069 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ff406ad20 0x7f1ff406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:13.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.777+0000 7f2013fff700 1 --2- 172.21.15.189:0/2484844069 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ff406ad20 0x7f1ff406d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc004f30 tx=0x7f1ffc009040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:13.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.782+0000 7f2010ff9700 1 -- 172.21.15.189:0/2484844069 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2004055c60 con 0x7f201415d7b0 2024-04-06T13:47:13.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.975+0000 7f201a273700 1 -- 172.21.15.189:0/2484844069 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f2014067bb0 con 0x7f201415d7b0 2024-04-06T13:47:13.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.978+0000 7f2010ff9700 1 -- 172.21.15.189:0/2484844069 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v73) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2004059280 con 0x7f201415d7b0 2024-04-06T13:47:13.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.981+0000 7f201a273700 1 -- 172.21.15.189:0/2484844069 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ff406ad20 msgr2=0x7f1ff406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:13.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.981+0000 7f201a273700 1 --2- 172.21.15.189:0/2484844069 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ff406ad20 0x7f1ff406d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc004f30 tx=0x7f1ffc009040 comp rx=0 tx=0).stop 2024-04-06T13:47:13.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.981+0000 7f201a273700 1 -- 172.21.15.189:0/2484844069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f201415d7b0 msgr2=0x7f20141455e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:13.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.981+0000 7f201a273700 1 --2- 172.21.15.189:0/2484844069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f201415d7b0 0x7f20141455e0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f2004000c00 tx=0x7f2004004750 comp rx=0 tx=0).stop 2024-04-06T13:47:13.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.981+0000 7f201a273700 1 -- 172.21.15.189:0/2484844069 shutdown_connections 2024-04-06T13:47:13.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.981+0000 7f201a273700 1 --2- 172.21.15.189:0/2484844069 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ff406ad20 0x7f1ff406d1e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:13.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.981+0000 7f201a273700 1 --2- 172.21.15.189:0/2484844069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f201415ddf0 0x7f20141422d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:13.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.981+0000 7f201a273700 1 --2- 172.21.15.189:0/2484844069 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f201406ae40 0x7f20140aebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:13.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.981+0000 7f201a273700 1 --2- 172.21.15.189:0/2484844069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f201415d7b0 0x7f20141455e0 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:13.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.981+0000 7f201a273700 1 -- 172.21.15.189:0/2484844069 >> 172.21.15.189:0/2484844069 conn(0x7f20140d00b0 msgr2=0x7f201415ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:13.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.981+0000 7f201a273700 1 -- 172.21.15.189:0/2484844069 shutdown_connections 2024-04-06T13:47:13.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:13.981+0000 7f201a273700 1 -- 172.21.15.189:0/2484844069 wait complete. 2024-04-06T13:47:13.994 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-06T13:47:14.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.117+0000 7fe5aa25a700 1 Processor -- start 2024-04-06T13:47:14.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.117+0000 7fe5aa25a700 1 -- start start 2024-04-06T13:47:14.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.119+0000 7fe5aa25a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe5a40d0a60 0x7fe5a40d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.119+0000 7fe5aa25a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5a40d1af0 0x7fe5a40b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.119+0000 7fe5aa25a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a40cdf80 con 0x7fe5a415d7e0 2024-04-06T13:47:14.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.119+0000 7fe5aa25a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a40d1f80 con 0x7fe5a40d0a60 2024-04-06T13:47:14.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.119+0000 7fe5aa25a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a40d3230 con 0x7fe5a40d1af0 2024-04-06T13:47:14.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.119+0000 7fe5a2ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe5a40d0a60 0x7fe5a40d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.119+0000 7fe5a2ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe5a40d0a60 0x7fe5a40d15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38552/0 (socket says 172.21.15.189:38552) 2024-04-06T13:47:14.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.119+0000 7fe5a2ffd700 1 -- 172.21.15.189:0/4090237734 learned_addr learned my addr 172.21.15.189:0/4090237734 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:14.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.119+0000 7fe5a3fff700 1 --2- 172.21.15.189:0/4090237734 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5a40d1af0 0x7fe5a40b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.119+0000 7fe5a2ffd700 1 -- 172.21.15.189:0/4090237734 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5a40d1af0 msgr2=0x7fe5a40b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:14.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.119+0000 7fe5a2ffd700 1 --2- 172.21.15.189:0/4090237734 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5a40d1af0 0x7fe5a40b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.119+0000 7fe5a2ffd700 1 -- 172.21.15.189:0/4090237734 >> v1:172.21.15.192:6789/0 conn(0x7fe5a415d7e0 legacy=0x7fe5a40d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:14.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.119+0000 7fe5a2ffd700 1 -- 172.21.15.189:0/4090237734 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a40d2da0 con 0x7fe5a40d0a60 2024-04-06T13:47:14.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.119+0000 7fe5a3fff700 1 --2- 172.21.15.189:0/4090237734 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5a40d1af0 0x7fe5a40b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:14.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.119+0000 7fe5a2ffd700 1 --2- 172.21.15.189:0/4090237734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5a40d0a60 0x7fe5a40d15b0 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7fe5940098f0 tx=0x7fe594009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7be469edb2910c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:14.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.119+0000 7fe5a27fc700 1 -- 172.21.15.189:0/4090237734 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59400b8a0 con 0x7fe5a40d0a60 2024-04-06T13:47:14.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.119+0000 7fe5a27fc700 1 -- 172.21.15.189:0/4090237734 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59400ba00 con 0x7fe5a40d0a60 2024-04-06T13:47:14.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.119+0000 7fe5a27fc700 1 -- 172.21.15.189:0/4090237734 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59400bd90 con 0x7fe5a40d0a60 2024-04-06T13:47:14.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.120+0000 7fe5aa25a700 1 -- 172.21.15.189:0/4090237734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5a40d0a60 msgr2=0x7fe5a40d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:14.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.120+0000 7fe5aa25a700 1 --2- 172.21.15.189:0/4090237734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5a40d0a60 0x7fe5a40d15b0 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7fe5940098f0 tx=0x7fe594009c00 comp rx=0 tx=0).stop 2024-04-06T13:47:14.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.120+0000 7fe5aa25a700 1 -- 172.21.15.189:0/4090237734 shutdown_connections 2024-04-06T13:47:14.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.120+0000 7fe5aa25a700 1 --2- 172.21.15.189:0/4090237734 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5a40d1af0 0x7fe5a40b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.120+0000 7fe5aa25a700 1 --2- 172.21.15.189:0/4090237734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5a40d0a60 0x7fe5a40d15b0 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.120+0000 7fe5aa25a700 1 -- 172.21.15.189:0/4090237734 >> 172.21.15.189:0/4090237734 conn(0x7fe5a4072250 msgr2=0x7fe5a4072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:14.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.120+0000 7fe5aa25a700 1 -- 172.21.15.189:0/4090237734 shutdown_connections 2024-04-06T13:47:14.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.120+0000 7fe5aa25a700 1 -- 172.21.15.189:0/4090237734 wait complete. 2024-04-06T13:47:14.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.122+0000 7fe5aa25a700 1 Processor -- start 2024-04-06T13:47:14.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.122+0000 7fe5aa25a700 1 -- start start 2024-04-06T13:47:14.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe5aa25a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5a40d0a60 0x7fe5a4157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe5aa25a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5a40d1af0 0x7fe5a40b4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe5aa25a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5a415d7e0 0x7fe5a40b4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe5aa25a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a40cac90 con 0x7fe5a415d7e0 2024-04-06T13:47:14.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe5aa25a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a4157ab0 con 0x7fe5a40d1af0 2024-04-06T13:47:14.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe5aa25a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a40c7a00 con 0x7fe5a40d0a60 2024-04-06T13:47:14.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe5a3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5a415d7e0 0x7fe5a40b4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe5a3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5a415d7e0 0x7fe5a40b4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38566/0 (socket says 172.21.15.189:38566) 2024-04-06T13:47:14.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe5a3fff700 1 -- 172.21.15.189:0/1163279161 learned_addr learned my addr 172.21.15.189:0/1163279161 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:14.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe5a2ffd700 1 --2- 172.21.15.189:0/1163279161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5a40d1af0 0x7fe5a40b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe5a37fe700 1 --2- 172.21.15.189:0/1163279161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5a40d0a60 0x7fe5a4157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe5a3fff700 1 -- 172.21.15.189:0/1163279161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5a40d0a60 msgr2=0x7fe5a4157210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:14.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe5a3fff700 1 --2- 172.21.15.189:0/1163279161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5a40d0a60 0x7fe5a4157210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe5a3fff700 1 -- 172.21.15.189:0/1163279161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5a40d1af0 msgr2=0x7fe5a40b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:14.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe5a3fff700 1 --2- 172.21.15.189:0/1163279161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5a40d1af0 0x7fe5a40b4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe5a3fff700 1 -- 172.21.15.189:0/1163279161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe594009550 con 0x7fe5a415d7e0 2024-04-06T13:47:14.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe5a3fff700 1 --2- 172.21.15.189:0/1163279161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5a415d7e0 0x7fe5a40b4790 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7fe59c009fa0 tx=0x7fe59c00d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:14.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe58bfff700 1 -- 172.21.15.189:0/1163279161 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c003a20 con 0x7fe5a415d7e0 2024-04-06T13:47:14.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe58bfff700 1 -- 172.21.15.189:0/1163279161 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59c00f040 con 0x7fe5a415d7e0 2024-04-06T13:47:14.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe5aa25a700 1 -- 172.21.15.189:0/1163279161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a40c7720 con 0x7fe5a415d7e0 2024-04-06T13:47:14.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.123+0000 7fe58bfff700 1 -- 172.21.15.189:0/1163279161 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c003de0 con 0x7fe5a415d7e0 2024-04-06T13:47:14.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.124+0000 7fe5aa25a700 1 -- 172.21.15.189:0/1163279161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a40ca8b0 con 0x7fe5a415d7e0 2024-04-06T13:47:14.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.125+0000 7fe5aa25a700 1 -- 172.21.15.189:0/1163279161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a40cadd0 con 0x7fe5a415d7e0 2024-04-06T13:47:14.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.125+0000 7fe58bfff700 1 -- 172.21.15.189:0/1163279161 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 138) v1 ==== 89191+0+0 (secure 0 0 0) 0x7fe59c004030 con 0x7fe5a415d7e0 2024-04-06T13:47:14.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.125+0000 7fe58bfff700 1 --2- 172.21.15.189:0/1163279161 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe59006ad20 0x7fe59006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.125+0000 7fe5a37fe700 1 --2- 172.21.15.189:0/1163279161 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe59006ad20 0x7fe59006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.125+0000 7fe58bfff700 1 -- 172.21.15.189:0/1163279161 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6023+0+0 (secure 0 0 0) 0x7fe59c089730 con 0x7fe5a415d7e0 2024-04-06T13:47:14.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.126+0000 7fe5a37fe700 1 --2- 172.21.15.189:0/1163279161 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe59006ad20 0x7fe59006d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe598000f60 tx=0x7fe598009040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:14.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.130+0000 7fe58bfff700 1 -- 172.21.15.189:0/1163279161 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe59c054760 con 0x7fe5a415d7e0 2024-04-06T13:47:14.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.325+0000 7fe5aa25a700 1 -- 172.21.15.189:0/1163279161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fe5a4074990 con 0x7fe5a415d7e0 2024-04-06T13:47:14.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.329+0000 7fe58bfff700 1 -- 172.21.15.189:0/1163279161 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v74) v1 ==== 115+0+0 (secure 0 0 0) 0x7fe59c05e020 con 0x7fe5a415d7e0 2024-04-06T13:47:14.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.331+0000 7fe5aa25a700 1 -- 172.21.15.189:0/1163279161 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe59006ad20 msgr2=0x7fe59006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:14.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.331+0000 7fe5aa25a700 1 --2- 172.21.15.189:0/1163279161 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe59006ad20 0x7fe59006d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe598000f60 tx=0x7fe598009040 comp rx=0 tx=0).stop 2024-04-06T13:47:14.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.331+0000 7fe5aa25a700 1 -- 172.21.15.189:0/1163279161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5a415d7e0 msgr2=0x7fe5a40b4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:14.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.331+0000 7fe5aa25a700 1 --2- 172.21.15.189:0/1163279161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5a415d7e0 0x7fe5a40b4790 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7fe59c009fa0 tx=0x7fe59c00d460 comp rx=0 tx=0).stop 2024-04-06T13:47:14.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.332+0000 7fe5aa25a700 1 -- 172.21.15.189:0/1163279161 shutdown_connections 2024-04-06T13:47:14.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.332+0000 7fe5aa25a700 1 --2- 172.21.15.189:0/1163279161 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe59006ad20 0x7fe59006d1e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.332+0000 7fe5aa25a700 1 --2- 172.21.15.189:0/1163279161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5a415d7e0 0x7fe5a40b4790 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.332+0000 7fe5aa25a700 1 --2- 172.21.15.189:0/1163279161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5a40d0a60 0x7fe5a4157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.332+0000 7fe5aa25a700 1 --2- 172.21.15.189:0/1163279161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5a40d1af0 0x7fe5a40b4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.332+0000 7fe5aa25a700 1 -- 172.21.15.189:0/1163279161 >> 172.21.15.189:0/1163279161 conn(0x7fe5a4072250 msgr2=0x7fe5a415f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:14.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.332+0000 7fe5aa25a700 1 -- 172.21.15.189:0/1163279161 shutdown_connections 2024-04-06T13:47:14.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.332+0000 7fe5aa25a700 1 -- 172.21.15.189:0/1163279161 wait complete. 2024-04-06T13:47:14.346 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-06T13:47:14.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.476+0000 7f6714eb5700 1 Processor -- start 2024-04-06T13:47:14.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.476+0000 7f6714eb5700 1 -- start start 2024-04-06T13:47:14.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f6714eb5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f67100d17a0 0x7f671006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f6714eb5700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6710146dd0 0x7f671006d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f6714eb5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67100cedf0 0x7f67100cf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f6714eb5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67100cf780 con 0x7f67100d17a0 2024-04-06T13:47:14.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f6714eb5700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f671006c0a0 con 0x7f6710146dd0 2024-04-06T13:47:14.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f6714eb5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671006c380 con 0x7f67100cedf0 2024-04-06T13:47:14.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f670e59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f67100d17a0 0x7f671006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f670e59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f67100d17a0 0x7f671006b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38578/0 (socket says 172.21.15.189:38578) 2024-04-06T13:47:14.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f670e59c700 1 -- 172.21.15.189:0/1413743619 learned_addr learned my addr 172.21.15.189:0/1413743619 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:14.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f670dd9b700 1 --2- 172.21.15.189:0/1413743619 >> v2:172.21.15.192:3300/0 conn(0x7f6710146dd0 0x7f671006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f670ed9d700 1 --2- 172.21.15.189:0/1413743619 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67100cedf0 0x7f67100cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f670e59c700 1 -- 172.21.15.189:0/1413743619 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67100cedf0 msgr2=0x7f67100cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:14.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f670e59c700 1 --2- 172.21.15.189:0/1413743619 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67100cedf0 0x7f67100cf240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f670e59c700 1 -- 172.21.15.189:0/1413743619 >> v2:172.21.15.192:3300/0 conn(0x7f6710146dd0 msgr2=0x7f671006d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:14.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f670e59c700 1 --2- 172.21.15.189:0/1413743619 >> v2:172.21.15.192:3300/0 conn(0x7f6710146dd0 0x7f671006d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f670e59c700 1 -- 172.21.15.189:0/1413743619 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671006c660 con 0x7f67100d17a0 2024-04-06T13:47:14.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f670dd9b700 1 --2- 172.21.15.189:0/1413743619 >> v2:172.21.15.192:3300/0 conn(0x7f6710146dd0 0x7f671006d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:14.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f670ed9d700 1 --2- 172.21.15.189:0/1413743619 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67100cedf0 0x7f67100cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:14.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f670e59c700 1 --2- 172.21.15.189:0/1413743619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67100d17a0 0x7f671006b7a0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7f67100d2300 tx=0x7f6700009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba06063edaf567c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:14.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f670d59a700 1 -- 172.21.15.189:0/1413743619 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670000b870 con 0x7f67100d17a0 2024-04-06T13:47:14.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f670d59a700 1 -- 172.21.15.189:0/1413743619 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670000b9d0 con 0x7f67100d17a0 2024-04-06T13:47:14.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.477+0000 7f670d59a700 1 -- 172.21.15.189:0/1413743619 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670000bd60 con 0x7f67100d17a0 2024-04-06T13:47:14.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.478+0000 7f6714eb5700 1 -- 172.21.15.189:0/1413743619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67100d17a0 msgr2=0x7f671006b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:14.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.478+0000 7f6714eb5700 1 --2- 172.21.15.189:0/1413743619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67100d17a0 0x7f671006b7a0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7f67100d2300 tx=0x7f6700009bd0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.478+0000 7f6714eb5700 1 -- 172.21.15.189:0/1413743619 shutdown_connections 2024-04-06T13:47:14.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.478+0000 7f6714eb5700 1 --2- 172.21.15.189:0/1413743619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67100d17a0 0x7f671006b7a0 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.478+0000 7f6714eb5700 1 --2- 172.21.15.189:0/1413743619 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67100cedf0 0x7f67100cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.478+0000 7f6714eb5700 1 --2- 172.21.15.189:0/1413743619 >> v2:172.21.15.192:3300/0 conn(0x7f6710146dd0 0x7f671006d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.478+0000 7f6714eb5700 1 -- 172.21.15.189:0/1413743619 >> 172.21.15.189:0/1413743619 conn(0x7f67100b2db0 msgr2=0x7f67100b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:14.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.478+0000 7f6714eb5700 1 -- 172.21.15.189:0/1413743619 shutdown_connections 2024-04-06T13:47:14.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.478+0000 7f6714eb5700 1 -- 172.21.15.189:0/1413743619 wait complete. 2024-04-06T13:47:14.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.479+0000 7f6714eb5700 1 Processor -- start 2024-04-06T13:47:14.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.479+0000 7f6714eb5700 1 -- start start 2024-04-06T13:47:14.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.480+0000 7f6714eb5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67100cedf0 0x7f67101f5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.480+0000 7f6714eb5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67100d17a0 0x7f67100aed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.480+0000 7f6714eb5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6710146dd0 0x7f67100af2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.480+0000 7f6714eb5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67100ad6b0 con 0x7f6710146dd0 2024-04-06T13:47:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.480+0000 7f6714eb5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67100ad990 con 0x7f67100d17a0 2024-04-06T13:47:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.480+0000 7f6714eb5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67100adc70 con 0x7f67100cedf0 2024-04-06T13:47:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.480+0000 7f670ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6710146dd0 0x7f67100af2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.480+0000 7f670ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6710146dd0 0x7f67100af2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38590/0 (socket says 172.21.15.189:38590) 2024-04-06T13:47:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.480+0000 7f670ed9d700 1 -- 172.21.15.189:0/2533201950 learned_addr learned my addr 172.21.15.189:0/2533201950 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.480+0000 7f670dd9b700 1 --2- 172.21.15.189:0/2533201950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67100d17a0 0x7f67100aed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.480+0000 7f670e59c700 1 --2- 172.21.15.189:0/2533201950 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67100cedf0 0x7f67101f5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.480+0000 7f670ed9d700 1 -- 172.21.15.189:0/2533201950 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67100cedf0 msgr2=0x7f67101f5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.480+0000 7f670ed9d700 1 --2- 172.21.15.189:0/2533201950 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67100cedf0 0x7f67101f5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.480+0000 7f670ed9d700 1 -- 172.21.15.189:0/2533201950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67100d17a0 msgr2=0x7f67100aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.480+0000 7f670ed9d700 1 --2- 172.21.15.189:0/2533201950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67100d17a0 0x7f67100aed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.480+0000 7f670ed9d700 1 -- 172.21.15.189:0/2533201950 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6700009550 con 0x7f6710146dd0 2024-04-06T13:47:14.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.481+0000 7f670ed9d700 1 --2- 172.21.15.189:0/2533201950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6710146dd0 0x7f67100af2c0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7f670800cd50 tx=0x7f670800a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:14.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.481+0000 7f66feffd700 1 -- 172.21.15.189:0/2533201950 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670800f480 con 0x7f6710146dd0 2024-04-06T13:47:14.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.481+0000 7f66feffd700 1 -- 172.21.15.189:0/2533201950 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670800f600 con 0x7f6710146dd0 2024-04-06T13:47:14.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.481+0000 7f6714eb5700 1 -- 172.21.15.189:0/2533201950 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67100adf50 con 0x7f6710146dd0 2024-04-06T13:47:14.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.481+0000 7f66feffd700 1 -- 172.21.15.189:0/2533201950 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670800f9b0 con 0x7f6710146dd0 2024-04-06T13:47:14.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.481+0000 7f6714eb5700 1 -- 172.21.15.189:0/2533201950 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67101f5fc0 con 0x7f6710146dd0 2024-04-06T13:47:14.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.482+0000 7f6714eb5700 1 -- 172.21.15.189:0/2533201950 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67100ae470 con 0x7f6710146dd0 2024-04-06T13:47:14.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.482+0000 7f66feffd700 1 -- 172.21.15.189:0/2533201950 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 138) v1 ==== 89191+0+0 (secure 0 0 0) 0x7f670800b040 con 0x7f6710146dd0 2024-04-06T13:47:14.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.483+0000 7f66feffd700 1 --2- 172.21.15.189:0/2533201950 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f66f006ad20 0x7f66f006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.483+0000 7f66feffd700 1 -- 172.21.15.189:0/2533201950 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6023+0+0 (secure 0 0 0) 0x7f6708089d70 con 0x7f6710146dd0 2024-04-06T13:47:14.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.483+0000 7f670e59c700 1 --2- 172.21.15.189:0/2533201950 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f66f006ad20 0x7f66f006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.483+0000 7f670e59c700 1 --2- 172.21.15.189:0/2533201950 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f66f006ad20 0x7f66f006d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f67100b08e0 tx=0x7f6700008030 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:14.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.487+0000 7f66feffd700 1 -- 172.21.15.189:0/2533201950 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6708054d10 con 0x7f6710146dd0 2024-04-06T13:47:14.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.680+0000 7f6714eb5700 1 -- 172.21.15.189:0/2533201950 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f67100d0020 con 0x7f6710146dd0 2024-04-06T13:47:14.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.684+0000 7f66feffd700 1 -- 172.21.15.189:0/2533201950 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v75) v1 ==== 120+0+0 (secure 0 0 0) 0x7f6708058330 con 0x7f6710146dd0 2024-04-06T13:47:14.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.686+0000 7f6714eb5700 1 -- 172.21.15.189:0/2533201950 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f66f006ad20 msgr2=0x7f66f006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:14.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.686+0000 7f6714eb5700 1 --2- 172.21.15.189:0/2533201950 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f66f006ad20 0x7f66f006d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f67100b08e0 tx=0x7f6700008030 comp rx=0 tx=0).stop 2024-04-06T13:47:14.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.686+0000 7f6714eb5700 1 -- 172.21.15.189:0/2533201950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6710146dd0 msgr2=0x7f67100af2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:14.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.686+0000 7f6714eb5700 1 --2- 172.21.15.189:0/2533201950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6710146dd0 0x7f67100af2c0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7f670800cd50 tx=0x7f670800a560 comp rx=0 tx=0).stop 2024-04-06T13:47:14.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.686+0000 7f6714eb5700 1 -- 172.21.15.189:0/2533201950 shutdown_connections 2024-04-06T13:47:14.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.686+0000 7f6714eb5700 1 --2- 172.21.15.189:0/2533201950 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f66f006ad20 0x7f66f006d1e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.686+0000 7f6714eb5700 1 --2- 172.21.15.189:0/2533201950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6710146dd0 0x7f67100af2c0 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.686+0000 7f6714eb5700 1 --2- 172.21.15.189:0/2533201950 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67100cedf0 0x7f67101f5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.686+0000 7f6714eb5700 1 --2- 172.21.15.189:0/2533201950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67100d17a0 0x7f67100aed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.686+0000 7f6714eb5700 1 -- 172.21.15.189:0/2533201950 >> 172.21.15.189:0/2533201950 conn(0x7f67100b2db0 msgr2=0x7f67100b5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:14.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.687+0000 7f6714eb5700 1 -- 172.21.15.189:0/2533201950 shutdown_connections 2024-04-06T13:47:14.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.687+0000 7f6714eb5700 1 -- 172.21.15.189:0/2533201950 wait complete. 2024-04-06T13:47:14.700 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2024-04-06T13:47:14.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.822+0000 7fb3340a1700 1 Processor -- start 2024-04-06T13:47:14.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.822+0000 7fb3340a1700 1 -- start start 2024-04-06T13:47:14.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.823+0000 7fb3340a1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb32c15d7b0 0x7fb32c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.823+0000 7fb3340a1700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb32c06a730 0x7fb32c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.823+0000 7fb3340a1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb32c15ddf0 0x7fb32c15e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.823+0000 7fb3340a1700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb32c06b0f0 con 0x7fb32c15d7b0 2024-04-06T13:47:14.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.823+0000 7fb3340a1700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb32c0b3990 con 0x7fb32c06a730 2024-04-06T13:47:14.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.823+0000 7fb3340a1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c0b3060 con 0x7fb32c15ddf0 2024-04-06T13:47:14.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.823+0000 7fb331e3d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb32c15d7b0 0x7fb32c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.823+0000 7fb331e3d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb32c15d7b0 0x7fb32c0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38604/0 (socket says 172.21.15.189:38604) 2024-04-06T13:47:14.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.823+0000 7fb331e3d700 1 -- 172.21.15.189:0/1979604035 learned_addr learned my addr 172.21.15.189:0/1979604035 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:14.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.823+0000 7fb33163c700 1 --2- 172.21.15.189:0/1979604035 >> v2:172.21.15.192:3300/0 conn(0x7fb32c06a730 0x7fb32c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.823+0000 7fb33263e700 1 --2- 172.21.15.189:0/1979604035 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb32c15ddf0 0x7fb32c15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.823+0000 7fb331e3d700 1 -- 172.21.15.189:0/1979604035 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb32c15ddf0 msgr2=0x7fb32c15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:14.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.823+0000 7fb331e3d700 1 --2- 172.21.15.189:0/1979604035 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb32c15ddf0 0x7fb32c15e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.823+0000 7fb331e3d700 1 -- 172.21.15.189:0/1979604035 >> v2:172.21.15.192:3300/0 conn(0x7fb32c06a730 msgr2=0x7fb32c06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:14.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.823+0000 7fb331e3d700 1 --2- 172.21.15.189:0/1979604035 >> v2:172.21.15.192:3300/0 conn(0x7fb32c06a730 0x7fb32c06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.823+0000 7fb331e3d700 1 -- 172.21.15.189:0/1979604035 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c0b3340 con 0x7fb32c15d7b0 2024-04-06T13:47:14.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.823+0000 7fb33163c700 1 --2- 172.21.15.189:0/1979604035 >> v2:172.21.15.192:3300/0 conn(0x7fb32c06a730 0x7fb32c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:14.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.823+0000 7fb33263e700 1 --2- 172.21.15.189:0/1979604035 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb32c15ddf0 0x7fb32c15e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:14.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.823+0000 7fb331e3d700 1 --2- 172.21.15.189:0/1979604035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c15d7b0 0x7fb32c0b3e20 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0afc00 tx=0x7fb328009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=736f539c55f8ff64 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:14.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.824+0000 7fb330e3b700 1 -- 172.21.15.189:0/1979604035 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32800b870 con 0x7fb32c15d7b0 2024-04-06T13:47:14.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.824+0000 7fb330e3b700 1 -- 172.21.15.189:0/1979604035 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32800b9d0 con 0x7fb32c15d7b0 2024-04-06T13:47:14.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.824+0000 7fb330e3b700 1 -- 172.21.15.189:0/1979604035 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32800bd60 con 0x7fb32c15d7b0 2024-04-06T13:47:14.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.824+0000 7fb3340a1700 1 -- 172.21.15.189:0/1979604035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c15d7b0 msgr2=0x7fb32c0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:14.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.824+0000 7fb3340a1700 1 --2- 172.21.15.189:0/1979604035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c15d7b0 0x7fb32c0b3e20 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0afc00 tx=0x7fb328009bd0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.824+0000 7fb3340a1700 1 -- 172.21.15.189:0/1979604035 shutdown_connections 2024-04-06T13:47:14.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.824+0000 7fb3340a1700 1 --2- 172.21.15.189:0/1979604035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c15d7b0 0x7fb32c0b3e20 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.824+0000 7fb3340a1700 1 --2- 172.21.15.189:0/1979604035 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb32c15ddf0 0x7fb32c15e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.824+0000 7fb3340a1700 1 --2- 172.21.15.189:0/1979604035 >> v2:172.21.15.192:3300/0 conn(0x7fb32c06a730 0x7fb32c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.824+0000 7fb3340a1700 1 -- 172.21.15.189:0/1979604035 >> 172.21.15.189:0/1979604035 conn(0x7fb32c0d00b0 msgr2=0x7fb32c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:14.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.824+0000 7fb3340a1700 1 -- 172.21.15.189:0/1979604035 shutdown_connections 2024-04-06T13:47:14.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.824+0000 7fb3340a1700 1 -- 172.21.15.189:0/1979604035 wait complete. 2024-04-06T13:47:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.825+0000 7fb3340a1700 1 Processor -- start 2024-04-06T13:47:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.825+0000 7fb3340a1700 1 -- start start 2024-04-06T13:47:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.826+0000 7fb3340a1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb32c06a730 0x7fb32c14f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.826+0000 7fb3340a1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c15d7b0 0x7fb32c1f28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.826+0000 7fb3340a1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb32c15ddf0 0x7fb32c0cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.826+0000 7fb3340a1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c157a00 con 0x7fb32c15d7b0 2024-04-06T13:47:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.826+0000 7fb3340a1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c157ca0 con 0x7fb32c06a730 2024-04-06T13:47:14.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.826+0000 7fb3340a1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c157f80 con 0x7fb32c15ddf0 2024-04-06T13:47:14.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.826+0000 7fb33163c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c15d7b0 0x7fb32c1f28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.826+0000 7fb33163c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c15d7b0 0x7fb32c1f28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38618/0 (socket says 172.21.15.189:38618) 2024-04-06T13:47:14.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.826+0000 7fb33163c700 1 -- 172.21.15.189:0/2520184149 learned_addr learned my addr 172.21.15.189:0/2520184149 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:14.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.826+0000 7fb331e3d700 1 --2- 172.21.15.189:0/2520184149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb32c06a730 0x7fb32c14f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.826+0000 7fb33263e700 1 --2- 172.21.15.189:0/2520184149 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb32c15ddf0 0x7fb32c0cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.826+0000 7fb33163c700 1 -- 172.21.15.189:0/2520184149 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb32c15ddf0 msgr2=0x7fb32c0cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:14.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.826+0000 7fb33163c700 1 --2- 172.21.15.189:0/2520184149 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb32c15ddf0 0x7fb32c0cebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.826+0000 7fb33163c700 1 -- 172.21.15.189:0/2520184149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb32c06a730 msgr2=0x7fb32c14f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:14.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.827+0000 7fb33163c700 1 --2- 172.21.15.189:0/2520184149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb32c06a730 0x7fb32c14f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:14.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.827+0000 7fb33163c700 1 -- 172.21.15.189:0/2520184149 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb328009550 con 0x7fb32c15d7b0 2024-04-06T13:47:14.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.827+0000 7fb33163c700 1 --2- 172.21.15.189:0/2520184149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c15d7b0 0x7fb32c1f28c0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7fb31c00c750 tx=0x7fb31c00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:14.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.827+0000 7fb3227fc700 1 -- 172.21.15.189:0/2520184149 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c011800 con 0x7fb32c15d7b0 2024-04-06T13:47:14.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.827+0000 7fb3227fc700 1 -- 172.21.15.189:0/2520184149 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31c011960 con 0x7fb32c15d7b0 2024-04-06T13:47:14.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.827+0000 7fb3340a1700 1 -- 172.21.15.189:0/2520184149 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb32c158260 con 0x7fb32c15d7b0 2024-04-06T13:47:14.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.827+0000 7fb3227fc700 1 -- 172.21.15.189:0/2520184149 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c011cd0 con 0x7fb32c15d7b0 2024-04-06T13:47:14.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.827+0000 7fb3340a1700 1 -- 172.21.15.189:0/2520184149 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb32c1585f0 con 0x7fb32c15d7b0 2024-04-06T13:47:14.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.828+0000 7fb3340a1700 1 -- 172.21.15.189:0/2520184149 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb32c158780 con 0x7fb32c15d7b0 2024-04-06T13:47:14.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.829+0000 7fb3227fc700 1 -- 172.21.15.189:0/2520184149 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 138) v1 ==== 89191+0+0 (secure 0 0 0) 0x7fb31c023480 con 0x7fb32c15d7b0 2024-04-06T13:47:14.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.829+0000 7fb3227fc700 1 --2- 172.21.15.189:0/2520184149 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb31006ad20 0x7fb31006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:14.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.829+0000 7fb3227fc700 1 -- 172.21.15.189:0/2520184149 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6023+0+0 (secure 0 0 0) 0x7fb31c089760 con 0x7fb32c15d7b0 2024-04-06T13:47:14.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.829+0000 7fb331e3d700 1 --2- 172.21.15.189:0/2520184149 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb31006ad20 0x7fb31006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:14.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.830+0000 7fb331e3d700 1 --2- 172.21.15.189:0/2520184149 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb31006ad20 0x7fb31006d1e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb328000c00 tx=0x7fb328008030 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:14.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:14.833+0000 7fb3227fc700 1 -- 172.21.15.189:0/2520184149 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb31c054810 con 0x7fb32c15d7b0 2024-04-06T13:47:15.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.107+0000 7fb3340a1700 1 -- 172.21.15.189:0/2520184149 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7fb32c158780 con 0x7fb32c15d7b0 2024-04-06T13:47:15.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.337+0000 7fb3227fc700 1 -- 172.21.15.189:0/2520184149 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fb31c054be0 con 0x7fb32c15d7b0 2024-04-06T13:47:15.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.632+0000 7fb3227fc700 1 -- 172.21.15.189:0/2520184149 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3098) v1 ==== 163+0+0 (secure 0 0 0) 0x7fb31c0758b0 con 0x7fb32c15d7b0 2024-04-06T13:47:15.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.634+0000 7fb3340a1700 1 -- 172.21.15.189:0/2520184149 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb31006ad20 msgr2=0x7fb31006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:15.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.634+0000 7fb3340a1700 1 --2- 172.21.15.189:0/2520184149 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb31006ad20 0x7fb31006d1e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb328000c00 tx=0x7fb328008030 comp rx=0 tx=0).stop 2024-04-06T13:47:15.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.634+0000 7fb3340a1700 1 -- 172.21.15.189:0/2520184149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c15d7b0 msgr2=0x7fb32c1f28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:15.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.634+0000 7fb3340a1700 1 --2- 172.21.15.189:0/2520184149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c15d7b0 0x7fb32c1f28c0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7fb31c00c750 tx=0x7fb31c00cb10 comp rx=0 tx=0).stop 2024-04-06T13:47:15.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.635+0000 7fb3340a1700 1 -- 172.21.15.189:0/2520184149 shutdown_connections 2024-04-06T13:47:15.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.635+0000 7fb3340a1700 1 --2- 172.21.15.189:0/2520184149 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb31006ad20 0x7fb31006d1e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:15.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.635+0000 7fb3340a1700 1 --2- 172.21.15.189:0/2520184149 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb32c15ddf0 0x7fb32c0cebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:15.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.635+0000 7fb3340a1700 1 --2- 172.21.15.189:0/2520184149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb32c06a730 0x7fb32c14f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:15.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.635+0000 7fb3340a1700 1 --2- 172.21.15.189:0/2520184149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c15d7b0 0x7fb32c1f28c0 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:15.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.635+0000 7fb3340a1700 1 -- 172.21.15.189:0/2520184149 >> 172.21.15.189:0/2520184149 conn(0x7fb32c0d00b0 msgr2=0x7fb32c0723d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:15.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.635+0000 7fb3340a1700 1 -- 172.21.15.189:0/2520184149 shutdown_connections 2024-04-06T13:47:15.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.635+0000 7fb3340a1700 1 -- 172.21.15.189:0/2520184149 wait complete. 2024-04-06T13:47:15.648 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... ok 2024-04-06T13:47:15.648 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command succes with --fs argument for non-default fs 2024-04-06T13:47:15.648 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2024-04-06T13:47:15.649 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2024-04-06T13:47:15.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.770+0000 7f39f0bef700 1 Processor -- start 2024-04-06T13:47:15.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.770+0000 7f39f0bef700 1 -- start start 2024-04-06T13:47:15.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.770+0000 7f39f0bef700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f39ec0d17a0 0x7f39ec06b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:15.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.770+0000 7f39f0bef700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39ec146dd0 0x7f39ec06d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:15.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.770+0000 7f39f0bef700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f39ec0cedf0 0x7f39ec0cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:15.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.770+0000 7f39f0bef700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39ec0cf760 con 0x7f39ec0d17a0 2024-04-06T13:47:15.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.770+0000 7f39f0bef700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39ec06c0a0 con 0x7f39ec0cedf0 2024-04-06T13:47:15.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.770+0000 7f39f0bef700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39ec06c380 con 0x7f39ec146dd0 2024-04-06T13:47:15.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.771+0000 7f39ea59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f39ec0d17a0 0x7f39ec06b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:15.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.771+0000 7f39ea59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f39ec0d17a0 0x7f39ec06b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38622/0 (socket says 172.21.15.189:38622) 2024-04-06T13:47:15.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.771+0000 7f39ea59c700 1 -- 172.21.15.189:0/461018636 learned_addr learned my addr 172.21.15.189:0/461018636 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:15.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.771+0000 7f39ea59c700 1 -- 172.21.15.189:0/461018636 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39ec146dd0 msgr2=0x7f39ec06d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:15.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.771+0000 7f39ea59c700 1 --2- 172.21.15.189:0/461018636 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39ec146dd0 0x7f39ec06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:15.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.771+0000 7f39ea59c700 1 -- 172.21.15.189:0/461018636 >> v2:172.21.15.192:3300/0 conn(0x7f39ec0cedf0 msgr2=0x7f39ec0cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:15.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.771+0000 7f39ead9d700 1 --2- 172.21.15.189:0/461018636 >> v2:172.21.15.192:3300/0 conn(0x7f39ec0cedf0 0x7f39ec0cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:15.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.771+0000 7f39ea59c700 1 --2- 172.21.15.189:0/461018636 >> v2:172.21.15.192:3300/0 conn(0x7f39ec0cedf0 0x7f39ec0cf220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:15.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.771+0000 7f39ea59c700 1 -- 172.21.15.189:0/461018636 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ec06c660 con 0x7f39ec0d17a0 2024-04-06T13:47:15.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.771+0000 7f39ea59c700 1 --2- 172.21.15.189:0/461018636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39ec0d17a0 0x7f39ec06b7a0 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f39ec162120 tx=0x7f39e0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ebff98ee03e9e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:15.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.771+0000 7f39e959a700 1 -- 172.21.15.189:0/461018636 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e000b870 con 0x7f39ec0d17a0 2024-04-06T13:47:15.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.771+0000 7f39e959a700 1 -- 172.21.15.189:0/461018636 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e000b9d0 con 0x7f39ec0d17a0 2024-04-06T13:47:15.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.771+0000 7f39e959a700 1 -- 172.21.15.189:0/461018636 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e000bd60 con 0x7f39ec0d17a0 2024-04-06T13:47:15.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.772+0000 7f39f0bef700 1 -- 172.21.15.189:0/461018636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39ec0d17a0 msgr2=0x7f39ec06b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:15.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.772+0000 7f39f0bef700 1 --2- 172.21.15.189:0/461018636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39ec0d17a0 0x7f39ec06b7a0 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f39ec162120 tx=0x7f39e0009bd0 comp rx=0 tx=0).stop 2024-04-06T13:47:15.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.772+0000 7f39f0bef700 1 -- 172.21.15.189:0/461018636 shutdown_connections 2024-04-06T13:47:15.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.772+0000 7f39f0bef700 1 --2- 172.21.15.189:0/461018636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39ec0d17a0 0x7f39ec06b7a0 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:15.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.772+0000 7f39f0bef700 1 --2- 172.21.15.189:0/461018636 >> v2:172.21.15.192:3300/0 conn(0x7f39ec0cedf0 0x7f39ec0cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:15.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.772+0000 7f39f0bef700 1 --2- 172.21.15.189:0/461018636 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39ec146dd0 0x7f39ec06d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:15.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.772+0000 7f39f0bef700 1 -- 172.21.15.189:0/461018636 >> 172.21.15.189:0/461018636 conn(0x7f39ec0b2db0 msgr2=0x7f39ec0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:15.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.772+0000 7f39f0bef700 1 -- 172.21.15.189:0/461018636 shutdown_connections 2024-04-06T13:47:15.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.772+0000 7f39f0bef700 1 -- 172.21.15.189:0/461018636 wait complete. 2024-04-06T13:47:15.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.773+0000 7f39f0bef700 1 Processor -- start 2024-04-06T13:47:15.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.773+0000 7f39f0bef700 1 -- start start 2024-04-06T13:47:15.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.774+0000 7f39f0bef700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39ec0cedf0 0x7f39ec0c4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:15.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.774+0000 7f39f0bef700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39ec0d17a0 0x7f39ec0c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:15.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.774+0000 7f39f0bef700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39ec146dd0 0x7f39ec0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:15.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.774+0000 7f39f0bef700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ec0c77f0 con 0x7f39ec146dd0 2024-04-06T13:47:15.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.774+0000 7f39f0bef700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ec0c7ad0 con 0x7f39ec0cedf0 2024-04-06T13:47:15.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.774+0000 7f39f0bef700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39ec0aeaf0 con 0x7f39ec0d17a0 2024-04-06T13:47:15.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.774+0000 7f39ead9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39ec146dd0 0x7f39ec0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:15.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.774+0000 7f39ead9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39ec146dd0 0x7f39ec0ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38624/0 (socket says 172.21.15.189:38624) 2024-04-06T13:47:15.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.774+0000 7f39ead9d700 1 -- 172.21.15.189:0/2656629315 learned_addr learned my addr 172.21.15.189:0/2656629315 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:15.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.774+0000 7f39ea59c700 1 --2- 172.21.15.189:0/2656629315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39ec0cedf0 0x7f39ec0c4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:15.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.774+0000 7f39e9d9b700 1 --2- 172.21.15.189:0/2656629315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39ec0d17a0 0x7f39ec0c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:15.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.774+0000 7f39ead9d700 1 -- 172.21.15.189:0/2656629315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39ec0d17a0 msgr2=0x7f39ec0c6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:15.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.774+0000 7f39ead9d700 1 --2- 172.21.15.189:0/2656629315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39ec0d17a0 0x7f39ec0c6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:15.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.774+0000 7f39ead9d700 1 -- 172.21.15.189:0/2656629315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39ec0cedf0 msgr2=0x7f39ec0c4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:15.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.774+0000 7f39ead9d700 1 --2- 172.21.15.189:0/2656629315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39ec0cedf0 0x7f39ec0c4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:15.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.774+0000 7f39ead9d700 1 -- 172.21.15.189:0/2656629315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e0009550 con 0x7f39ec146dd0 2024-04-06T13:47:15.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.774+0000 7f39e9d9b700 1 --2- 172.21.15.189:0/2656629315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39ec0d17a0 0x7f39ec0c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:15.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.775+0000 7f39ead9d700 1 --2- 172.21.15.189:0/2656629315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39ec146dd0 0x7f39ec0ac8d0 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f39e400cc30 tx=0x7f39e400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:15.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.775+0000 7f39daffd700 1 -- 172.21.15.189:0/2656629315 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e400ab00 con 0x7f39ec146dd0 2024-04-06T13:47:15.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.775+0000 7f39daffd700 1 -- 172.21.15.189:0/2656629315 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e400ac60 con 0x7f39ec146dd0 2024-04-06T13:47:15.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.775+0000 7f39f0bef700 1 -- 172.21.15.189:0/2656629315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39ec0aedd0 con 0x7f39ec146dd0 2024-04-06T13:47:15.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.775+0000 7f39daffd700 1 -- 172.21.15.189:0/2656629315 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e4003a20 con 0x7f39ec146dd0 2024-04-06T13:47:15.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.775+0000 7f39f0bef700 1 -- 172.21.15.189:0/2656629315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39ec0ae7e0 con 0x7f39ec146dd0 2024-04-06T13:47:15.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.776+0000 7f39f0bef700 1 -- 172.21.15.189:0/2656629315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39ec146490 con 0x7f39ec146dd0 2024-04-06T13:47:15.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.776+0000 7f39daffd700 1 -- 172.21.15.189:0/2656629315 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f39e400f3f0 con 0x7f39ec146dd0 2024-04-06T13:47:15.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.776+0000 7f39daffd700 1 --2- 172.21.15.189:0/2656629315 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f39cc06adb0 0x7f39cc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:15.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.777+0000 7f39daffd700 1 -- 172.21.15.189:0/2656629315 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6023+0+0 (secure 0 0 0) 0x7f39e40897e0 con 0x7f39ec146dd0 2024-04-06T13:47:15.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.777+0000 7f39ea59c700 1 --2- 172.21.15.189:0/2656629315 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f39cc06adb0 0x7f39cc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:15.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.777+0000 7f39ea59c700 1 --2- 172.21.15.189:0/2656629315 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f39cc06adb0 0x7f39cc06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f39e0014040 tx=0x7f39e0008030 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:15.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:15.781+0000 7f39daffd700 1 -- 172.21.15.189:0/2656629315 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f39e4054660 con 0x7f39ec146dd0 2024-04-06T13:47:16.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.059+0000 7f39f0bef700 1 -- 172.21.15.189:0/2656629315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7f39ec0aefc0 con 0x7f39ec146dd0 2024-04-06T13:47:16.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.636+0000 7f39daffd700 1 -- 172.21.15.189:0/2656629315 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3099) v1 ==== 189+0+0 (secure 0 0 0) 0x7f39e401d070 con 0x7f39ec146dd0 2024-04-06T13:47:16.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.639+0000 7f39f0bef700 1 -- 172.21.15.189:0/2656629315 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f39cc06adb0 msgr2=0x7f39cc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:16.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.639+0000 7f39f0bef700 1 --2- 172.21.15.189:0/2656629315 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f39cc06adb0 0x7f39cc06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f39e0014040 tx=0x7f39e0008030 comp rx=0 tx=0).stop 2024-04-06T13:47:16.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.639+0000 7f39f0bef700 1 -- 172.21.15.189:0/2656629315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39ec146dd0 msgr2=0x7f39ec0ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:16.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.639+0000 7f39f0bef700 1 --2- 172.21.15.189:0/2656629315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39ec146dd0 0x7f39ec0ac8d0 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f39e400cc30 tx=0x7f39e400cf40 comp rx=0 tx=0).stop 2024-04-06T13:47:16.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.639+0000 7f39f0bef700 1 -- 172.21.15.189:0/2656629315 shutdown_connections 2024-04-06T13:47:16.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.639+0000 7f39f0bef700 1 --2- 172.21.15.189:0/2656629315 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f39cc06adb0 0x7f39cc06d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:16.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.639+0000 7f39f0bef700 1 --2- 172.21.15.189:0/2656629315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39ec146dd0 0x7f39ec0ac8d0 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:16.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.639+0000 7f39f0bef700 1 --2- 172.21.15.189:0/2656629315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39ec0cedf0 0x7f39ec0c4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:16.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.639+0000 7f39f0bef700 1 --2- 172.21.15.189:0/2656629315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39ec0d17a0 0x7f39ec0c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:16.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.639+0000 7f39f0bef700 1 -- 172.21.15.189:0/2656629315 >> 172.21.15.189:0/2656629315 conn(0x7f39ec0b2db0 msgr2=0x7f39ec15df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:16.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.639+0000 7f39f0bef700 1 -- 172.21.15.189:0/2656629315 shutdown_connections 2024-04-06T13:47:16.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.639+0000 7f39f0bef700 1 -- 172.21.15.189:0/2656629315 wait complete. 2024-04-06T13:47:16.653 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-06T13:47:16.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.777+0000 7f186a665700 1 Processor -- start 2024-04-06T13:47:16.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.777+0000 7f186a665700 1 -- start start 2024-04-06T13:47:16.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.778+0000 7f186a665700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f186415d7b0 0x7f18640b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:16.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.778+0000 7f186a665700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f186406a730 0x7f186406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:16.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.778+0000 7f186a665700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18640b3990 con 0x7f186415ddf0 2024-04-06T13:47:16.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.778+0000 7f186a665700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f186406b0f0 con 0x7f186415d7b0 2024-04-06T13:47:16.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.778+0000 7f186a665700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18640b3060 con 0x7f186406a730 2024-04-06T13:47:16.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1868c02700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f186415ddf0 0x7f186415e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42838/0 (socket says 172.21.15.189:42838) 2024-04-06T13:47:16.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1868c02700 1 -- 172.21.15.189:0/1160867124 learned_addr learned my addr 172.21.15.189:0/1160867124 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:16.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f18637fe700 1 --2- 172.21.15.189:0/1160867124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f186406a730 0x7f186406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:16.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1863fff700 1 --2- 172.21.15.189:0/1160867124 >> v2:172.21.15.192:3300/0 conn(0x7f186415d7b0 0x7f18640b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:16.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1862ffd700 1 -- 172.21.15.189:0/1160867124 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091503829 0 0) 0x7f18640b3990 con 0x7f186415ddf0 2024-04-06T13:47:16.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1862ffd700 1 -- 172.21.15.189:0/1160867124 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f184c0035b0 con 0x7f186415ddf0 2024-04-06T13:47:16.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1862ffd700 1 -- 172.21.15.189:0/1160867124 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2086813132 0 0) 0x7f184c0035b0 con 0x7f186415ddf0 2024-04-06T13:47:16.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1862ffd700 1 -- 172.21.15.189:0/1160867124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f186406a730 msgr2=0x7f186406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:16.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1862ffd700 1 --2- 172.21.15.189:0/1160867124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f186406a730 0x7f186406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:16.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1862ffd700 1 -- 172.21.15.189:0/1160867124 >> v2:172.21.15.192:3300/0 conn(0x7f186415d7b0 msgr2=0x7f18640b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:16.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1862ffd700 1 --2- 172.21.15.189:0/1160867124 >> v2:172.21.15.192:3300/0 conn(0x7f186415d7b0 0x7f18640b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:16.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1862ffd700 1 -- 172.21.15.189:0/1160867124 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18640b3340 con 0x7f186415ddf0 2024-04-06T13:47:16.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1862ffd700 1 -- 172.21.15.189:0/1160867124 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f185c0019f0 con 0x7f186415ddf0 2024-04-06T13:47:16.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1862ffd700 1 -- 172.21.15.189:0/1160867124 >> v1:172.21.15.189:6789/0 conn(0x7f186415ddf0 legacy=0x7f186415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:16.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1862ffd700 1 --2- 172.21.15.189:0/1160867124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f184c003a00 0x7f184c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:16.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1862ffd700 1 --2- 172.21.15.189:0/1160867124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f184c004850 0x7f184c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:16.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1862ffd700 1 --2- 172.21.15.189:0/1160867124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f184c005210 0x7f184c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:16.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1862ffd700 1 -- 172.21.15.189:0/1160867124 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186406b0f0 con 0x7f184c005210 2024-04-06T13:47:16.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1862ffd700 1 -- 172.21.15.189:0/1160867124 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18640b3060 con 0x7f184c003a00 2024-04-06T13:47:16.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1862ffd700 1 -- 172.21.15.189:0/1160867124 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184c00bfa0 con 0x7f184c004850 2024-04-06T13:47:16.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f18637fe700 1 --2- 172.21.15.189:0/1160867124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f186406a730 0x7f186406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:16.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1863fff700 1 --2- 172.21.15.189:0/1160867124 >> v2:172.21.15.192:3300/0 conn(0x7f186415d7b0 0x7f18640b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:16.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1868c02700 1 --2- 172.21.15.189:0/1160867124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f184c005210 0x7f184c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:16.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f18637fe700 1 --2- 172.21.15.189:0/1160867124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f184c004850 0x7f184c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:16.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.779+0000 7f1863fff700 1 --2- 172.21.15.189:0/1160867124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f184c003a00 0x7f184c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:16.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.780+0000 7f1868c02700 1 -- 172.21.15.189:0/1160867124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f184c004850 msgr2=0x7f184c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:16.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.780+0000 7f1868c02700 1 --2- 172.21.15.189:0/1160867124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f184c004850 0x7f184c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:16.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.780+0000 7f1868c02700 1 -- 172.21.15.189:0/1160867124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f184c003a00 msgr2=0x7f184c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:16.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.780+0000 7f1868c02700 1 --2- 172.21.15.189:0/1160867124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f184c003a00 0x7f184c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:16.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.780+0000 7f1868c02700 1 -- 172.21.15.189:0/1160867124 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184c0035b0 con 0x7f184c005210 2024-04-06T13:47:16.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.780+0000 7f1868c02700 1 --2- 172.21.15.189:0/1160867124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f184c005210 0x7f184c00ba10 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7f185c00a890 tx=0x7f184c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=565bc7ff6de0ed89 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:16.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.780+0000 7f1862ffd700 1 -- 172.21.15.189:0/1160867124 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c012600 con 0x7f184c005210 2024-04-06T13:47:16.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.780+0000 7f1862ffd700 1 -- 172.21.15.189:0/1160867124 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185c012780 con 0x7f184c005210 2024-04-06T13:47:16.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.780+0000 7f1862ffd700 1 -- 172.21.15.189:0/1160867124 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c012b30 con 0x7f184c005210 2024-04-06T13:47:16.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.780+0000 7f186a665700 1 -- 172.21.15.189:0/1160867124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f184c005210 msgr2=0x7f184c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:16.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.780+0000 7f186a665700 1 --2- 172.21.15.189:0/1160867124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f184c005210 0x7f184c00ba10 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7f185c00a890 tx=0x7f184c005880 comp rx=0 tx=0).stop 2024-04-06T13:47:16.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.780+0000 7f1868c02700 1 -- 172.21.15.189:0/1160867124 reap_dead start 2024-04-06T13:47:16.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.780+0000 7f186a665700 1 -- 172.21.15.189:0/1160867124 shutdown_connections 2024-04-06T13:47:16.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.780+0000 7f186a665700 1 -- 172.21.15.189:0/1160867124 >> 172.21.15.189:0/1160867124 conn(0x7f18640d00b0 msgr2=0x7f18640d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:16.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.780+0000 7f186a665700 1 -- 172.21.15.189:0/1160867124 shutdown_connections 2024-04-06T13:47:16.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.780+0000 7f186a665700 1 -- 172.21.15.189:0/1160867124 wait complete. 2024-04-06T13:47:16.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.782+0000 7f186a665700 1 Processor -- start 2024-04-06T13:47:16.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.782+0000 7f186a665700 1 -- start start 2024-04-06T13:47:16.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f186a665700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f186414fa10 0x7f18641582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:16.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f186a665700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18640ceec0 0x7f18640ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:16.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f186a665700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18640cd030 0x7f18641f6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:16.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f186a665700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18640cd4b0 con 0x7f18640cd030 2024-04-06T13:47:16.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f186a665700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18641589a0 con 0x7f186414fa10 2024-04-06T13:47:16.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f186a665700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1864158c80 con 0x7f18640ceec0 2024-04-06T13:47:16.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f1868c02700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18640cd030 0x7f18641f6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:16.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f1868c02700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18640cd030 0x7f18641f6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38644/0 (socket says 172.21.15.189:38644) 2024-04-06T13:47:16.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f1868c02700 1 -- 172.21.15.189:0/2852755319 learned_addr learned my addr 172.21.15.189:0/2852755319 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:16.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f18637fe700 1 --2- 172.21.15.189:0/2852755319 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18640ceec0 0x7f18640ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:16.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f1863fff700 1 --2- 172.21.15.189:0/2852755319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f186414fa10 0x7f18641582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:16.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f1868c02700 1 -- 172.21.15.189:0/2852755319 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18640ceec0 msgr2=0x7f18640ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:16.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f1868c02700 1 --2- 172.21.15.189:0/2852755319 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18640ceec0 0x7f18640ccaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:16.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f1868c02700 1 -- 172.21.15.189:0/2852755319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f186414fa10 msgr2=0x7f18641582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:16.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f1868c02700 1 --2- 172.21.15.189:0/2852755319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f186414fa10 0x7f18641582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:16.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f1868c02700 1 -- 172.21.15.189:0/2852755319 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1864158f60 con 0x7f18640cd030 2024-04-06T13:47:16.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f1868c02700 1 --2- 172.21.15.189:0/2852755319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18640cd030 0x7f18641f6250 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f185c00a7b0 tx=0x7f185c001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:16.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f1860ff9700 1 -- 172.21.15.189:0/2852755319 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c01e630 con 0x7f18640cd030 2024-04-06T13:47:16.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f1860ff9700 1 -- 172.21.15.189:0/2852755319 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185c01e7b0 con 0x7f18640cd030 2024-04-06T13:47:16.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f1860ff9700 1 -- 172.21.15.189:0/2852755319 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c01eb60 con 0x7f18640cd030 2024-04-06T13:47:16.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f186a665700 1 -- 172.21.15.189:0/2852755319 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1864159240 con 0x7f18640cd030 2024-04-06T13:47:16.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.783+0000 7f186a665700 1 -- 172.21.15.189:0/2852755319 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18640cf340 con 0x7f18640cd030 2024-04-06T13:47:16.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.784+0000 7f186a665700 1 -- 172.21.15.189:0/2852755319 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1864074990 con 0x7f18640cd030 2024-04-06T13:47:16.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.785+0000 7f1860ff9700 1 -- 172.21.15.189:0/2852755319 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f185c01ecc0 con 0x7f18640cd030 2024-04-06T13:47:16.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.785+0000 7f1860ff9700 1 --2- 172.21.15.189:0/2852755319 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f185006adb0 0x7f185006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:16.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.786+0000 7f1860ff9700 1 -- 172.21.15.189:0/2852755319 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6023+0+0 (secure 0 0 0) 0x7f185c08b5e0 con 0x7f18640cd030 2024-04-06T13:47:16.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.786+0000 7f1863fff700 1 --2- 172.21.15.189:0/2852755319 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f185006adb0 0x7f185006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:16.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.786+0000 7f1863fff700 1 --2- 172.21.15.189:0/2852755319 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f185006adb0 0x7f185006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1858009470 tx=0x7f1858008040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:16.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.789+0000 7f1860ff9700 1 -- 172.21.15.189:0/2852755319 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f185c0567c0 con 0x7f18640cd030 2024-04-06T13:47:16.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.982+0000 7f186a665700 1 -- 172.21.15.189:0/2852755319 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f186406f970 con 0x7f18640cd030 2024-04-06T13:47:16.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.987+0000 7f1860ff9700 1 -- 172.21.15.189:0/2852755319 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v76)=0 v76) v1 ==== 122+0+0 (secure 0 0 0) 0x7f185c059de0 con 0x7f18640cd030 2024-04-06T13:47:16.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.990+0000 7f186a665700 1 -- 172.21.15.189:0/2852755319 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f185006adb0 msgr2=0x7f185006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:16.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.990+0000 7f186a665700 1 --2- 172.21.15.189:0/2852755319 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f185006adb0 0x7f185006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1858009470 tx=0x7f1858008040 comp rx=0 tx=0).stop 2024-04-06T13:47:16.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.990+0000 7f186a665700 1 -- 172.21.15.189:0/2852755319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18640cd030 msgr2=0x7f18641f6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:16.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.990+0000 7f186a665700 1 --2- 172.21.15.189:0/2852755319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18640cd030 0x7f18641f6250 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f185c00a7b0 tx=0x7f185c001d80 comp rx=0 tx=0).stop 2024-04-06T13:47:16.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.990+0000 7f186a665700 1 -- 172.21.15.189:0/2852755319 shutdown_connections 2024-04-06T13:47:16.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.990+0000 7f186a665700 1 --2- 172.21.15.189:0/2852755319 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f185006adb0 0x7f185006d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:16.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.990+0000 7f186a665700 1 --2- 172.21.15.189:0/2852755319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18640cd030 0x7f18641f6250 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:16.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.990+0000 7f186a665700 1 --2- 172.21.15.189:0/2852755319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f186414fa10 0x7f18641582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:16.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.990+0000 7f186a665700 1 --2- 172.21.15.189:0/2852755319 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f18640ceec0 0x7f18640ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:16.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.990+0000 7f186a665700 1 -- 172.21.15.189:0/2852755319 >> 172.21.15.189:0/2852755319 conn(0x7f18640d00b0 msgr2=0x7f186406eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:16.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.990+0000 7f186a665700 1 -- 172.21.15.189:0/2852755319 shutdown_connections 2024-04-06T13:47:16.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:16.990+0000 7f186a665700 1 -- 172.21.15.189:0/2852755319 wait complete. 2024-04-06T13:47:17.003 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T13:47:17.003 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:47:17.046 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:47:17.101 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T13:47:17.102 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:47:17.102 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:47:17.121 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:47:17.121 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:47:17.121 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:47:17.141 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:47:17.141 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:47:17.141 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:47:17.178 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:47:17.178 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:47:17.178 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:47:17.201 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:47:17.201 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-06T13:47:17.202 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:47:17.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.337+0000 7fd4665c3700 1 Processor -- start 2024-04-06T13:47:17.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.338+0000 7fd4665c3700 1 -- start start 2024-04-06T13:47:17.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.338+0000 7fd4665c3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd46015d7e0 0x7fd4600d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:17.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.338+0000 7fd4665c3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4600cdf80 con 0x7fd4600d1cc0 2024-04-06T13:47:17.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.338+0000 7fd4665c3700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4600d0cf0 con 0x7fd4600d0890 2024-04-06T13:47:17.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.338+0000 7fd4665c3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4600d3230 con 0x7fd46015d7e0 2024-04-06T13:47:17.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.339+0000 7fd464b60700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd4600d1cc0 0x7fd4600b66b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42854/0 (socket says 172.21.15.189:42854) 2024-04-06T13:47:17.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.339+0000 7fd464b60700 1 -- 172.21.15.189:0/1963520882 learned_addr learned my addr 172.21.15.189:0/1963520882 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:17.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.339+0000 7fd45ffff700 1 --2- 172.21.15.189:0/1963520882 >> v2:172.21.15.192:3300/0 conn(0x7fd46015d7e0 0x7fd4600d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:17.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.339+0000 7fd45effd700 1 -- 172.21.15.189:0/1963520882 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972661665 0 0) 0x7fd4600cdf80 con 0x7fd4600d1cc0 2024-04-06T13:47:17.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.339+0000 7fd45effd700 1 -- 172.21.15.189:0/1963520882 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd44c0035b0 con 0x7fd4600d1cc0 2024-04-06T13:47:17.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.339+0000 7fd45ffff700 1 -- 172.21.15.189:0/1963520882 >> v1:172.21.15.192:6789/0 conn(0x7fd4600d0890 legacy=0x7fd4600d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:17.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.339+0000 7fd45effd700 1 -- 172.21.15.189:0/1963520882 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1300208237 0 0) 0x7fd44c0035b0 con 0x7fd4600d1cc0 2024-04-06T13:47:17.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.339+0000 7fd45ffff700 1 -- 172.21.15.189:0/1963520882 >> v1:172.21.15.189:6789/0 conn(0x7fd4600d1cc0 legacy=0x7fd4600b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:17.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.339+0000 7fd45ffff700 1 -- 172.21.15.189:0/1963520882 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4600d2da0 con 0x7fd46015d7e0 2024-04-06T13:47:17.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.339+0000 7fd45ffff700 1 --2- 172.21.15.189:0/1963520882 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd46015d7e0 0x7fd4600d0350 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd450009840 tx=0x7fd450009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6a9f85627398999 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:17.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.339+0000 7fd45effd700 1 -- 172.21.15.189:0/1963520882 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4580019f0 con 0x7fd46015d7e0 2024-04-06T13:47:17.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.340+0000 7fd45effd700 1 -- 172.21.15.189:0/1963520882 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd450004030 con 0x7fd46015d7e0 2024-04-06T13:47:17.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.340+0000 7fd45effd700 1 -- 172.21.15.189:0/1963520882 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4500043a0 con 0x7fd46015d7e0 2024-04-06T13:47:17.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.340+0000 7fd4665c3700 1 -- 172.21.15.189:0/1963520882 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd46015d7e0 msgr2=0x7fd4600d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:17.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.340+0000 7fd4665c3700 1 --2- 172.21.15.189:0/1963520882 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd46015d7e0 0x7fd4600d0350 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd450009840 tx=0x7fd450009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:17.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.340+0000 7fd4665c3700 1 -- 172.21.15.189:0/1963520882 shutdown_connections 2024-04-06T13:47:17.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.340+0000 7fd4665c3700 1 --2- 172.21.15.189:0/1963520882 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd46015d7e0 0x7fd4600d0350 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:17.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.340+0000 7fd4665c3700 1 -- 172.21.15.189:0/1963520882 >> 172.21.15.189:0/1963520882 conn(0x7fd460072250 msgr2=0x7fd460072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:17.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.340+0000 7fd4665c3700 1 -- 172.21.15.189:0/1963520882 shutdown_connections 2024-04-06T13:47:17.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.340+0000 7fd4665c3700 1 -- 172.21.15.189:0/1963520882 wait complete. 2024-04-06T13:47:17.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.341+0000 7fd4665c3700 1 Processor -- start 2024-04-06T13:47:17.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.342+0000 7fd4665c3700 1 -- start start 2024-04-06T13:47:17.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.342+0000 7fd4665c3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4600d0890 0x7fd4601f3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:17.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.342+0000 7fd4665c3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4600d1cc0 0x7fd4600cba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:17.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.342+0000 7fd4665c3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd46015d7e0 0x7fd4600cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:17.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.342+0000 7fd4665c3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4600ca7f0 con 0x7fd46015d7e0 2024-04-06T13:47:17.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.342+0000 7fd4665c3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4600cb5e0 con 0x7fd4600d0890 2024-04-06T13:47:17.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.342+0000 7fd4665c3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4601f3980 con 0x7fd4600d1cc0 2024-04-06T13:47:17.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.342+0000 7fd464b60700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd46015d7e0 0x7fd4600cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:17.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.342+0000 7fd464b60700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd46015d7e0 0x7fd4600cc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38654/0 (socket says 172.21.15.189:38654) 2024-04-06T13:47:17.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.342+0000 7fd464b60700 1 -- 172.21.15.189:0/2462420478 learned_addr learned my addr 172.21.15.189:0/2462420478 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:17.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.343+0000 7fd45ffff700 1 --2- 172.21.15.189:0/2462420478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4600d0890 0x7fd4601f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:17.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.343+0000 7fd45f7fe700 1 --2- 172.21.15.189:0/2462420478 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4600d1cc0 0x7fd4600cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:17.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.343+0000 7fd464b60700 1 -- 172.21.15.189:0/2462420478 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4600d1cc0 msgr2=0x7fd4600cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:17.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.343+0000 7fd464b60700 1 --2- 172.21.15.189:0/2462420478 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4600d1cc0 0x7fd4600cba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:17.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.343+0000 7fd464b60700 1 -- 172.21.15.189:0/2462420478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4600d0890 msgr2=0x7fd4601f3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:17.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.343+0000 7fd464b60700 1 --2- 172.21.15.189:0/2462420478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4600d0890 0x7fd4601f3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:17.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.343+0000 7fd464b60700 1 -- 172.21.15.189:0/2462420478 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4500094a0 con 0x7fd46015d7e0 2024-04-06T13:47:17.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.343+0000 7fd45ffff700 1 --2- 172.21.15.189:0/2462420478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4600d0890 0x7fd4601f3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:17.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.343+0000 7fd464b60700 1 --2- 172.21.15.189:0/2462420478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd46015d7e0 0x7fd4600cc5d0 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7fd45800cc00 tx=0x7fd45800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:17.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.343+0000 7fd45cff9700 1 -- 172.21.15.189:0/2462420478 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45800ab80 con 0x7fd46015d7e0 2024-04-06T13:47:17.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.343+0000 7fd45cff9700 1 -- 172.21.15.189:0/2462420478 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45800ace0 con 0x7fd46015d7e0 2024-04-06T13:47:17.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.343+0000 7fd4665c3700 1 -- 172.21.15.189:0/2462420478 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4600cbfb0 con 0x7fd46015d7e0 2024-04-06T13:47:17.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.343+0000 7fd45cff9700 1 -- 172.21.15.189:0/2462420478 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd458007310 con 0x7fd46015d7e0 2024-04-06T13:47:17.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.343+0000 7fd4665c3700 1 -- 172.21.15.189:0/2462420478 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4600cacb0 con 0x7fd46015d7e0 2024-04-06T13:47:17.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.344+0000 7fd4665c3700 1 -- 172.21.15.189:0/2462420478 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4600d0700 con 0x7fd46015d7e0 2024-04-06T13:47:17.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.344+0000 7fd45cff9700 1 -- 172.21.15.189:0/2462420478 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fd458007470 con 0x7fd46015d7e0 2024-04-06T13:47:17.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.345+0000 7fd45cff9700 1 --2- 172.21.15.189:0/2462420478 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd44006adb0 0x7fd44006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:17.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.345+0000 7fd45cff9700 1 -- 172.21.15.189:0/2462420478 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6023+0+0 (secure 0 0 0) 0x7fd458089120 con 0x7fd46015d7e0 2024-04-06T13:47:17.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.345+0000 7fd45ffff700 1 --2- 172.21.15.189:0/2462420478 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd44006adb0 0x7fd44006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:17.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.345+0000 7fd45ffff700 1 --2- 172.21.15.189:0/2462420478 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd44006adb0 0x7fd44006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd46015be30 tx=0x7fd450008000 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:17.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.349+0000 7fd45cff9700 1 -- 172.21.15.189:0/2462420478 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd4580540c0 con 0x7fd46015d7e0 2024-04-06T13:47:17.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.588+0000 7fd4665c3700 1 -- 172.21.15.189:0/2462420478 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd460074990 con 0x7fd46015d7e0 2024-04-06T13:47:17.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.588+0000 7fd45cff9700 1 -- 172.21.15.189:0/2462420478 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+3539 (secure 0 0 0) 0x7fd458018df0 con 0x7fd46015d7e0 2024-04-06T13:47:17.589 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:17.590 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":95,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8525,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3302262361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3302262361},{"type":"v1","addr":"172.21.15.189:6837","nonce":3302262361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":8534,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/229523550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":229523550},{"type":"v1","addr":"172.21.15.192:6834","nonce":229523550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":9739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2560949338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":2560949338},{"type":"v1","addr":"172.21.15.192:6835","nonce":2560949338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":11163,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3611313963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3611313963},{"type":"v1","addr":"172.21.15.189:6836","nonce":3611313963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":95}],"filesystems":[]} 2024-04-06T13:47:17.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.590+0000 7fd4665c3700 1 -- 172.21.15.189:0/2462420478 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd44006adb0 msgr2=0x7fd44006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:17.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.590+0000 7fd4665c3700 1 --2- 172.21.15.189:0/2462420478 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd44006adb0 0x7fd44006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd46015be30 tx=0x7fd450008000 comp rx=0 tx=0).stop 2024-04-06T13:47:17.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.590+0000 7fd4665c3700 1 -- 172.21.15.189:0/2462420478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd46015d7e0 msgr2=0x7fd4600cc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:17.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.590+0000 7fd4665c3700 1 --2- 172.21.15.189:0/2462420478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd46015d7e0 0x7fd4600cc5d0 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7fd45800cc00 tx=0x7fd45800a060 comp rx=0 tx=0).stop 2024-04-06T13:47:17.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.591+0000 7fd4665c3700 1 -- 172.21.15.189:0/2462420478 shutdown_connections 2024-04-06T13:47:17.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.591+0000 7fd4665c3700 1 --2- 172.21.15.189:0/2462420478 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd44006adb0 0x7fd44006d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:17.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.591+0000 7fd4665c3700 1 --2- 172.21.15.189:0/2462420478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd46015d7e0 0x7fd4600cc5d0 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:17.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.591+0000 7fd4665c3700 1 --2- 172.21.15.189:0/2462420478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4600d0890 0x7fd4601f3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:17.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.591+0000 7fd4665c3700 1 --2- 172.21.15.189:0/2462420478 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4600d1cc0 0x7fd4600cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:17.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.591+0000 7fd4665c3700 1 -- 172.21.15.189:0/2462420478 >> 172.21.15.189:0/2462420478 conn(0x7fd460072250 msgr2=0x7fd4600b66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:17.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.591+0000 7fd4665c3700 1 -- 172.21.15.189:0/2462420478 shutdown_connections 2024-04-06T13:47:17.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.591+0000 7fd4665c3700 1 -- 172.21.15.189:0/2462420478 wait complete. 2024-04-06T13:47:17.593 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 95 2024-04-06T13:47:17.604 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-06T13:47:17.604 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-06T13:47:17.604 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-06T13:47:17.605 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:47:17.605 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-06T13:47:17.605 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-06T13:47:17.605 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-06T13:47:17.605 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:47:17.605 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-06T13:47:17.605 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-06T13:47:17.606 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-06T13:47:17.606 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:47:17.606 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-06T13:47:17.606 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-06T13:47:17.606 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-06T13:47:17.606 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:47:17.613 INFO:tasks.ceph.mds.a:Stopped 2024-04-06T13:47:17.614 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-06T13:47:17.614 INFO:tasks.ceph.mds.c:Stopped 2024-04-06T13:47:17.615 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-06T13:47:17.618 INFO:tasks.ceph.mds.b:Stopped 2024-04-06T13:47:17.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-06T13:47:17.619 INFO:tasks.ceph.mds.d:Stopped 2024-04-06T13:47:17.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-06T13:47:17.657 INFO:tasks.ceph.mds.a:Started 2024-04-06T13:47:17.657 DEBUG:teuthology.parallel:result is None 2024-04-06T13:47:17.658 INFO:tasks.ceph.mds.c:Started 2024-04-06T13:47:17.658 DEBUG:teuthology.parallel:result is None 2024-04-06T13:47:17.660 INFO:tasks.ceph.mds.b:Started 2024-04-06T13:47:17.660 DEBUG:teuthology.parallel:result is None 2024-04-06T13:47:17.661 INFO:tasks.ceph.mds.d:Started 2024-04-06T13:47:17.661 DEBUG:teuthology.parallel:result is None 2024-04-06T13:47:17.661 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-06T13:47:17.737 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T13:47:17.738 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2024-04-06T13:47:17.748 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2024-04-06T13:47:17.749 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T13:47:17.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.792+0000 7fd550f88700 1 Processor -- start 2024-04-06T13:47:17.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.792+0000 7fd550f88700 1 -- start start 2024-04-06T13:47:17.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.793+0000 7fd550f88700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd54c0b4660 0x7fd54c0b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:17.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.793+0000 7fd550f88700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd54c0b2fc0 0x7fd54c0b3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:17.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.793+0000 7fd550f88700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c0cdbe0 con 0x7fd54c15a550 2024-04-06T13:47:17.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.793+0000 7fd550f88700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54c0b5000 con 0x7fd54c0b2fc0 2024-04-06T13:47:17.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.793+0000 7fd550f88700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c15a940 con 0x7fd54c0b4660 2024-04-06T13:47:17.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.793+0000 7fd54ad9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd54c0b2fc0 0x7fd54c0b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:17.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.793+0000 7fd54a59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd54c15a550 0x7fd54c0cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42862/0 (socket says 172.21.15.189:42862) 2024-04-06T13:47:17.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.793+0000 7fd54a59c700 1 -- 172.21.15.189:0/2975464908 learned_addr learned my addr 172.21.15.189:0/2975464908 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:17.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.793+0000 7fd549d9b700 1 --2- 172.21.15.189:0/2975464908 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd54c0b4660 0x7fd54c0b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:17.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.793+0000 7fd54ad9d700 1 -- 172.21.15.189:0/2975464908 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd54c0b4660 msgr2=0x7fd54c0b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:17.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.793+0000 7fd54ad9d700 1 --2- 172.21.15.189:0/2975464908 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd54c0b4660 0x7fd54c0b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:17.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.793+0000 7fd54ad9d700 1 -- 172.21.15.189:0/2975464908 >> v1:172.21.15.189:6789/0 conn(0x7fd54c15a550 legacy=0x7fd54c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:17.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.793+0000 7fd54959a700 1 -- 172.21.15.189:0/2975464908 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141435559 0 0) 0x7fd54c0cdbe0 con 0x7fd54c15a550 2024-04-06T13:47:17.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.793+0000 7fd54ad9d700 1 -- 172.21.15.189:0/2975464908 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c15ac00 con 0x7fd54c0b2fc0 2024-04-06T13:47:17.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.793+0000 7fd54ad9d700 1 --2- 172.21.15.189:0/2975464908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd54c0b2fc0 0x7fd54c0b3430 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7fd544009d90 tx=0x7fd54400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa3f1ae881057c87 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:17.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.793+0000 7fd54959a700 1 -- 172.21.15.189:0/2975464908 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54400e070 con 0x7fd54c0b2fc0 2024-04-06T13:47:17.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.794+0000 7fd54959a700 1 -- 172.21.15.189:0/2975464908 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54400bb20 con 0x7fd54c0b2fc0 2024-04-06T13:47:17.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.794+0000 7fd54959a700 1 -- 172.21.15.189:0/2975464908 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd544012430 con 0x7fd54c0b2fc0 2024-04-06T13:47:17.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.794+0000 7fd550f88700 1 -- 172.21.15.189:0/2975464908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd54c0b2fc0 msgr2=0x7fd54c0b3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:17.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.794+0000 7fd550f88700 1 --2- 172.21.15.189:0/2975464908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd54c0b2fc0 0x7fd54c0b3430 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7fd544009d90 tx=0x7fd54400b330 comp rx=0 tx=0).stop 2024-04-06T13:47:17.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.794+0000 7fd550f88700 1 -- 172.21.15.189:0/2975464908 shutdown_connections 2024-04-06T13:47:17.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.794+0000 7fd550f88700 1 --2- 172.21.15.189:0/2975464908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd54c0b2fc0 0x7fd54c0b3430 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:17.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.794+0000 7fd550f88700 1 --2- 172.21.15.189:0/2975464908 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd54c0b4660 0x7fd54c0b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:17.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.794+0000 7fd550f88700 1 -- 172.21.15.189:0/2975464908 >> 172.21.15.189:0/2975464908 conn(0x7fd54c0d00b0 msgr2=0x7fd54c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:17.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.794+0000 7fd550f88700 1 -- 172.21.15.189:0/2975464908 shutdown_connections 2024-04-06T13:47:17.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.794+0000 7fd550f88700 1 -- 172.21.15.189:0/2975464908 wait complete. 2024-04-06T13:47:17.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.795+0000 7fd550f88700 1 Processor -- start 2024-04-06T13:47:17.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.795+0000 7fd550f88700 1 -- start start 2024-04-06T13:47:17.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.796+0000 7fd550f88700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd54c0b2fc0 0x7fd54c149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:17.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.796+0000 7fd550f88700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd54c0b4660 0x7fd54c156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:17.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.796+0000 7fd550f88700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd54c15d2e0 0x7fd54c157470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:17.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.796+0000 7fd550f88700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0b17d0 con 0x7fd54c0b4660 2024-04-06T13:47:17.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.796+0000 7fd550f88700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c1579b0 con 0x7fd54c0b2fc0 2024-04-06T13:47:17.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.796+0000 7fd550f88700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0b0e70 con 0x7fd54c15d2e0 2024-04-06T13:47:17.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.796+0000 7fd549d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd54c0b4660 0x7fd54c156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:17.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.796+0000 7fd549d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd54c0b4660 0x7fd54c156f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38718/0 (socket says 172.21.15.189:38718) 2024-04-06T13:47:17.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.796+0000 7fd549d9b700 1 -- 172.21.15.189:0/3150215948 learned_addr learned my addr 172.21.15.189:0/3150215948 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:17.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.796+0000 7fd54a59c700 1 --2- 172.21.15.189:0/3150215948 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd54c0b2fc0 0x7fd54c149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:17.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.796+0000 7fd54ad9d700 1 --2- 172.21.15.189:0/3150215948 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd54c15d2e0 0x7fd54c157470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:17.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.797+0000 7fd549d9b700 1 -- 172.21.15.189:0/3150215948 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd54c15d2e0 msgr2=0x7fd54c157470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:17.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.797+0000 7fd549d9b700 1 --2- 172.21.15.189:0/3150215948 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd54c15d2e0 0x7fd54c157470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:17.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.797+0000 7fd549d9b700 1 -- 172.21.15.189:0/3150215948 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd54c0b2fc0 msgr2=0x7fd54c149ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:17.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.797+0000 7fd549d9b700 1 --2- 172.21.15.189:0/3150215948 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd54c0b2fc0 0x7fd54c149ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:17.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.797+0000 7fd549d9b700 1 -- 172.21.15.189:0/3150215948 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd544009a70 con 0x7fd54c0b4660 2024-04-06T13:47:17.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.797+0000 7fd549d9b700 1 --2- 172.21.15.189:0/3150215948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd54c0b4660 0x7fd54c156f30 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7fd54000ea00 tx=0x7fd54000ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:17.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.797+0000 7fd53affd700 1 -- 172.21.15.189:0/3150215948 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd540009730 con 0x7fd54c0b4660 2024-04-06T13:47:17.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.797+0000 7fd53affd700 1 -- 172.21.15.189:0/3150215948 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54000eea0 con 0x7fd54c0b4660 2024-04-06T13:47:17.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.797+0000 7fd53affd700 1 -- 172.21.15.189:0/3150215948 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54000ca70 con 0x7fd54c0b4660 2024-04-06T13:47:17.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.797+0000 7fd550f88700 1 -- 172.21.15.189:0/3150215948 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd54c15d7e0 con 0x7fd54c0b4660 2024-04-06T13:47:17.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.797+0000 7fd550f88700 1 -- 172.21.15.189:0/3150215948 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54c0b13f0 con 0x7fd54c0b4660 2024-04-06T13:47:17.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.798+0000 7fd550f88700 1 -- 172.21.15.189:0/3150215948 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54c0b1910 con 0x7fd54c0b4660 2024-04-06T13:47:17.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.798+0000 7fd53affd700 1 -- 172.21.15.189:0/3150215948 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fd54000cbf0 con 0x7fd54c0b4660 2024-04-06T13:47:17.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.799+0000 7fd53affd700 1 --2- 172.21.15.189:0/3150215948 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd52c06adb0 0x7fd52c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:17.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.799+0000 7fd53affd700 1 -- 172.21.15.189:0/3150215948 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6023+0+0 (secure 0 0 0) 0x7fd5400895e0 con 0x7fd54c0b4660 2024-04-06T13:47:17.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.799+0000 7fd54a59c700 1 --2- 172.21.15.189:0/3150215948 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd52c06adb0 0x7fd52c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:17.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.799+0000 7fd54a59c700 1 --2- 172.21.15.189:0/3150215948 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd52c06adb0 0x7fd52c06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd53c007fd0 tx=0x7fd53c007de0 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:17.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:17.803+0000 7fd53affd700 1 -- 172.21.15.189:0/3150215948 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd54c0b1910 con 0x7fd54c0b4660 2024-04-06T13:47:18.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.014+0000 7fd550f88700 1 -- 172.21.15.189:0/3150215948 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd54c0b1910 con 0x7fd54c0b4660 2024-04-06T13:47:18.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.786+0000 7fd53affd700 1 -- 172.21.15.189:0/3150215948 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v223) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd54c0b1910 con 0x7fd54c0b4660 2024-04-06T13:47:18.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.789+0000 7fd550f88700 1 -- 172.21.15.189:0/3150215948 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd52c06adb0 msgr2=0x7fd52c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:18.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.789+0000 7fd550f88700 1 --2- 172.21.15.189:0/3150215948 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd52c06adb0 0x7fd52c06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd53c007fd0 tx=0x7fd53c007de0 comp rx=0 tx=0).stop 2024-04-06T13:47:18.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.789+0000 7fd550f88700 1 -- 172.21.15.189:0/3150215948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd54c0b4660 msgr2=0x7fd54c156f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:18.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.789+0000 7fd550f88700 1 --2- 172.21.15.189:0/3150215948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd54c0b4660 0x7fd54c156f30 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7fd54000ea00 tx=0x7fd54000ed10 comp rx=0 tx=0).stop 2024-04-06T13:47:18.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.790+0000 7fd550f88700 1 -- 172.21.15.189:0/3150215948 shutdown_connections 2024-04-06T13:47:18.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.790+0000 7fd550f88700 1 --2- 172.21.15.189:0/3150215948 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd52c06adb0 0x7fd52c06d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:18.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.790+0000 7fd550f88700 1 --2- 172.21.15.189:0/3150215948 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd54c15d2e0 0x7fd54c157470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:18.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.790+0000 7fd550f88700 1 --2- 172.21.15.189:0/3150215948 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd54c0b2fc0 0x7fd54c149ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:18.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.790+0000 7fd550f88700 1 --2- 172.21.15.189:0/3150215948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd54c0b4660 0x7fd54c156f30 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:18.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.790+0000 7fd550f88700 1 -- 172.21.15.189:0/3150215948 >> 172.21.15.189:0/3150215948 conn(0x7fd54c0d00b0 msgr2=0x7fd54c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:18.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.790+0000 7fd550f88700 1 -- 172.21.15.189:0/3150215948 shutdown_connections 2024-04-06T13:47:18.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.790+0000 7fd550f88700 1 -- 172.21.15.189:0/3150215948 wait complete. 2024-04-06T13:47:18.792 INFO:teuthology.orchestra.run.smithi189.stderr: removed all blocklist entries 2024-04-06T13:47:18.806 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-06T13:47:18.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.932+0000 7fd879592700 1 Processor -- start 2024-04-06T13:47:18.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.932+0000 7fd879592700 1 -- start start 2024-04-06T13:47:18.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd879592700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd87415db10 0x7fd8740b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:18.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd879592700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87415eb40 con 0x7fd874160b60 2024-04-06T13:47:18.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd879592700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8740d0bf0 con 0x7fd874143470 2024-04-06T13:47:18.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd879592700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd874160f40 con 0x7fd87415db10 2024-04-06T13:47:18.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd872ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd874160b60 0x7fd874142d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42878/0 (socket says 172.21.15.189:42878) 2024-04-06T13:47:18.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd872ffd700 1 -- 172.21.15.189:0/3098337165 learned_addr learned my addr 172.21.15.189:0/3098337165 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:18.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd871ffb700 1 -- 172.21.15.189:0/3098337165 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106987766 0 0) 0x7fd864001110 con 0x7fd874160b60 2024-04-06T13:47:18.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd8727fc700 1 --2- 172.21.15.189:0/3098337165 >> v2:172.21.15.192:3300/0 conn(0x7fd87415db10 0x7fd8740b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:18.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd871ffb700 1 -- 172.21.15.189:0/3098337165 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd85c0035b0 con 0x7fd874160b60 2024-04-06T13:47:18.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd871ffb700 1 -- 172.21.15.189:0/3098337165 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 991887989 0 0) 0x7fd85c0035b0 con 0x7fd874160b60 2024-04-06T13:47:18.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd871ffb700 1 -- 172.21.15.189:0/3098337165 >> v2:172.21.15.192:3300/0 conn(0x7fd87415db10 msgr2=0x7fd8740b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:18.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd871ffb700 1 --2- 172.21.15.189:0/3098337165 >> v2:172.21.15.192:3300/0 conn(0x7fd87415db10 0x7fd8740b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:18.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd871ffb700 1 -- 172.21.15.189:0/3098337165 >> v1:172.21.15.192:6789/0 conn(0x7fd874143470 legacy=0x7fd8740cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:18.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd871ffb700 1 -- 172.21.15.189:0/3098337165 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87415dfc0 con 0x7fd874160b60 2024-04-06T13:47:18.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd871ffb700 1 -- 172.21.15.189:0/3098337165 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fd864001800 con 0x7fd874160b60 2024-04-06T13:47:18.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd871ffb700 1 -- 172.21.15.189:0/3098337165 >> v1:172.21.15.189:6789/0 conn(0x7fd874160b60 legacy=0x7fd874142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:18.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd871ffb700 1 --2- 172.21.15.189:0/3098337165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd85c003a50 0x7fd85c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:18.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd871ffb700 1 --2- 172.21.15.189:0/3098337165 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd85c004850 0x7fd85c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:18.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd871ffb700 1 --2- 172.21.15.189:0/3098337165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd85c005210 0x7fd85c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:18.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd871ffb700 1 -- 172.21.15.189:0/3098337165 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd874160f40 con 0x7fd85c003a50 2024-04-06T13:47:18.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd871ffb700 1 -- 172.21.15.189:0/3098337165 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c00c110 con 0x7fd85c005210 2024-04-06T13:47:18.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd871ffb700 1 -- 172.21.15.189:0/3098337165 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c00c2d0 con 0x7fd85c004850 2024-04-06T13:47:18.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.933+0000 7fd872ffd700 1 --2- 172.21.15.189:0/3098337165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd85c003a50 0x7fd85c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:18.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.934+0000 7fd8727fc700 1 --2- 172.21.15.189:0/3098337165 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd85c004850 0x7fd85c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:18.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.934+0000 7fd8737fe700 1 --2- 172.21.15.189:0/3098337165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd85c005210 0x7fd85c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:18.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.934+0000 7fd872ffd700 1 -- 172.21.15.189:0/3098337165 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd85c004850 msgr2=0x7fd85c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:18.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.934+0000 7fd872ffd700 1 --2- 172.21.15.189:0/3098337165 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd85c004850 0x7fd85c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:18.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.934+0000 7fd872ffd700 1 -- 172.21.15.189:0/3098337165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd85c005210 msgr2=0x7fd85c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:18.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.934+0000 7fd872ffd700 1 --2- 172.21.15.189:0/3098337165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd85c005210 0x7fd85c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:18.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.934+0000 7fd8737fe700 1 -- 172.21.15.189:0/3098337165 reap_dead start 2024-04-06T13:47:18.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.934+0000 7fd872ffd700 1 -- 172.21.15.189:0/3098337165 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85c0035b0 con 0x7fd85c003a50 2024-04-06T13:47:18.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.934+0000 7fd872ffd700 1 --2- 172.21.15.189:0/3098337165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd85c003a50 0x7fd85c003e70 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7fd864007fd0 tx=0x7fd85c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c010c5f99a2b3c1f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:18.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.934+0000 7fd871ffb700 1 -- 172.21.15.189:0/3098337165 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86400ccd0 con 0x7fd85c003a50 2024-04-06T13:47:18.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.934+0000 7fd871ffb700 1 -- 172.21.15.189:0/3098337165 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86400ce30 con 0x7fd85c003a50 2024-04-06T13:47:18.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.934+0000 7fd871ffb700 1 -- 172.21.15.189:0/3098337165 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd864005b80 con 0x7fd85c003a50 2024-04-06T13:47:18.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.934+0000 7fd879592700 1 -- 172.21.15.189:0/3098337165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd85c003a50 msgr2=0x7fd85c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:18.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.934+0000 7fd879592700 1 --2- 172.21.15.189:0/3098337165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd85c003a50 0x7fd85c003e70 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7fd864007fd0 tx=0x7fd85c005880 comp rx=0 tx=0).stop 2024-04-06T13:47:18.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.935+0000 7fd879592700 1 -- 172.21.15.189:0/3098337165 shutdown_connections 2024-04-06T13:47:18.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.935+0000 7fd879592700 1 --2- 172.21.15.189:0/3098337165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd85c003a50 0x7fd85c003e70 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:18.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.935+0000 7fd879592700 1 -- 172.21.15.189:0/3098337165 >> 172.21.15.189:0/3098337165 conn(0x7fd87406a730 msgr2=0x7fd87406ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:18.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.935+0000 7fd879592700 1 -- 172.21.15.189:0/3098337165 shutdown_connections 2024-04-06T13:47:18.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.935+0000 7fd879592700 1 -- 172.21.15.189:0/3098337165 wait complete. 2024-04-06T13:47:18.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.936+0000 7fd879592700 1 Processor -- start 2024-04-06T13:47:18.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.936+0000 7fd879592700 1 -- start start 2024-04-06T13:47:18.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd879592700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd85c003a50 0x7fd8740b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:18.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd879592700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8740c7670 0x7fd87415c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:18.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd879592700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8740b57c0 0x7fd8740b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:18.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd879592700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8740b6180 con 0x7fd8740c7670 2024-04-06T13:47:18.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd879592700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8740b4a90 con 0x7fd85c003a50 2024-04-06T13:47:18.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd879592700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87415cda0 con 0x7fd8740b57c0 2024-04-06T13:47:18.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd8727fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8740c7670 0x7fd87415c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:18.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd8727fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8740c7670 0x7fd87415c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38736/0 (socket says 172.21.15.189:38736) 2024-04-06T13:47:18.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd8727fc700 1 -- 172.21.15.189:0/2732925497 learned_addr learned my addr 172.21.15.189:0/2732925497 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:18.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd8737fe700 1 --2- 172.21.15.189:0/2732925497 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8740b57c0 0x7fd8740b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:18.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd872ffd700 1 --2- 172.21.15.189:0/2732925497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd85c003a50 0x7fd8740b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:18.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd8727fc700 1 -- 172.21.15.189:0/2732925497 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8740b57c0 msgr2=0x7fd8740b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:18.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd8727fc700 1 --2- 172.21.15.189:0/2732925497 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8740b57c0 0x7fd8740b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:18.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd8727fc700 1 -- 172.21.15.189:0/2732925497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd85c003a50 msgr2=0x7fd8740b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:18.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd8727fc700 1 --2- 172.21.15.189:0/2732925497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd85c003a50 0x7fd8740b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:18.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd8727fc700 1 -- 172.21.15.189:0/2732925497 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8740b3e80 con 0x7fd8740c7670 2024-04-06T13:47:18.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd8727fc700 1 --2- 172.21.15.189:0/2732925497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8740c7670 0x7fd87415c860 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7fd868011920 tx=0x7fd868011c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:18.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd85b7fe700 1 -- 172.21.15.189:0/2732925497 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868005c30 con 0x7fd8740c7670 2024-04-06T13:47:18.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd85b7fe700 1 -- 172.21.15.189:0/2732925497 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd868005d90 con 0x7fd8740c7670 2024-04-06T13:47:18.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd85b7fe700 1 -- 172.21.15.189:0/2732925497 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868006290 con 0x7fd8740c7670 2024-04-06T13:47:18.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.937+0000 7fd879592700 1 -- 172.21.15.189:0/2732925497 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8740b4160 con 0x7fd8740c7670 2024-04-06T13:47:18.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.938+0000 7fd879592700 1 -- 172.21.15.189:0/2732925497 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8740b4720 con 0x7fd8740c7670 2024-04-06T13:47:18.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.939+0000 7fd879592700 1 -- 172.21.15.189:0/2732925497 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd874074990 con 0x7fd8740c7670 2024-04-06T13:47:18.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.939+0000 7fd85b7fe700 1 -- 172.21.15.189:0/2732925497 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fd8680063f0 con 0x7fd8740c7670 2024-04-06T13:47:18.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.939+0000 7fd85b7fe700 1 --2- 172.21.15.189:0/2732925497 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd86006ade0 0x7fd86006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:18.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.939+0000 7fd872ffd700 1 --2- 172.21.15.189:0/2732925497 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd86006ade0 0x7fd86006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:18.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.939+0000 7fd85b7fe700 1 -- 172.21.15.189:0/2732925497 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fd868089740 con 0x7fd8740c7670 2024-04-06T13:47:18.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.940+0000 7fd872ffd700 1 --2- 172.21.15.189:0/2732925497 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd86006ade0 0x7fd86006d2a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd864000c00 tx=0x7fd864007000 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:18.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:18.944+0000 7fd85b7fe700 1 -- 172.21.15.189:0/2732925497 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd8680549d0 con 0x7fd8740c7670 2024-04-06T13:47:19.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.215+0000 7fd879592700 1 -- 172.21.15.189:0/2732925497 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd874146f10 con 0x7fd8740c7670 2024-04-06T13:47:19.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.215+0000 7fd85b7fe700 1 -- 172.21.15.189:0/2732925497 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v50) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fd868021090 con 0x7fd8740c7670 2024-04-06T13:47:19.217 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:19.217 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:47:19.217 INFO:teuthology.orchestra.run.smithi189.stdout: "auth_dump": [ 2024-04-06T13:47:19.217 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:19.218 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.a", 2024-04-06T13:47:19.218 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmTjYMNBAA5voWGV4Pa7pE27VogrdipQ==", 2024-04-06T13:47:19.218 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:47:19.218 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:47:19.218 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:47:19.219 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:47:19.219 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:47:19.219 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.219 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:19.219 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:19.219 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.b", 2024-04-06T13:47:19.220 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm9A1OBRAAnHzubCvRUX8UdW8hmDzKbQ==", 2024-04-06T13:47:19.220 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:47:19.220 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:47:19.220 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:47:19.220 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:47:19.220 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:47:19.221 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.221 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:19.221 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:19.221 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.c", 2024-04-06T13:47:19.221 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmz0hhOxAAD3/GINNmGoFk+oPyiHHJXg==", 2024-04-06T13:47:19.221 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:47:19.222 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:47:19.222 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:47:19.222 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:47:19.222 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:47:19.222 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.222 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:19.223 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:19.223 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.d", 2024-04-06T13:47:19.223 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmhoAeDRAAoQC0vicWumBRTwpi3PtYaw==", 2024-04-06T13:47:19.223 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:47:19.223 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:47:19.224 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:47:19.224 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:47:19.225 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:47:19.225 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.225 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:19.225 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:19.225 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.0", 2024-04-06T13:47:19.226 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDkOxFmkxLWJhAAFXewCNOFX7dhAZVgUmewpQ==", 2024-04-06T13:47:19.226 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:47:19.226 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:47:19.226 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:47:19.226 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:47:19.226 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.227 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:19.227 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:19.227 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.1", 2024-04-06T13:47:19.227 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDnOxFmHYqWJRAABlsVCiv3e7YJMbfodJTWag==", 2024-04-06T13:47:19.227 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:47:19.227 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:47:19.228 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:47:19.228 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:47:19.228 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.228 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:19.228 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:19.228 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.2", 2024-04-06T13:47:19.229 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDqOxFmkhcvJBAA8NMEpHVGbGALjV9RISKP8g==", 2024-04-06T13:47:19.229 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:47:19.229 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:47:19.229 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:47:19.229 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:47:19.229 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.229 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:19.229 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:19.229 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.3", 2024-04-06T13:47:19.230 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDtOxFmDGzhIxAAamBoAbE8+lAel4os67meyQ==", 2024-04-06T13:47:19.230 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:47:19.230 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:47:19.230 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:47:19.230 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:47:19.230 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.231 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:19.231 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:19.231 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.4", 2024-04-06T13:47:19.231 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDzOxFmobZDJxAA9l8wdS81Eyw6+17I5ac6bw==", 2024-04-06T13:47:19.231 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:47:19.231 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:47:19.231 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:47:19.232 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:47:19.232 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.232 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:19.232 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:19.232 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.5", 2024-04-06T13:47:19.232 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD2OxFmi+ruJRAASlQ0o9jX/wA7U5AosnL+Wg==", 2024-04-06T13:47:19.232 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:47:19.232 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:47:19.233 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:47:19.233 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:47:19.233 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.233 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:19.233 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:19.233 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.6", 2024-04-06T13:47:19.233 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD5OxFmbwuuJBAA0wqJRmnJyskQPZJ7/LnpvA==", 2024-04-06T13:47:19.233 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:47:19.233 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:47:19.234 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:47:19.234 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:47:19.234 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.234 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:19.234 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:19.234 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.7", 2024-04-06T13:47:19.234 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD8OxFmDmRgIxAAzyW9AGAbmb67ZvB5qunwLQ==", 2024-04-06T13:47:19.235 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:47:19.235 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:47:19.235 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:47:19.235 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:47:19.235 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.236 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:19.236 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:19.236 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.0", 2024-04-06T13:47:19.236 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm0ByJERAAoEV8Yfc09+Go6u95XafElQ==", 2024-04-06T13:47:19.236 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:47:19.236 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:47:19.236 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow r", 2024-04-06T13:47:19.236 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-06T13:47:19.236 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.236 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:19.236 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:19.236 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.1", 2024-04-06T13:47:19.236 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmSV5lGBAAavDEo2LGy0IHpWSAuitHXg==", 2024-04-06T13:47:19.237 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:47:19.237 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:47:19.237 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:47:19.237 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:47:19.237 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:47:19.237 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.237 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:19.237 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:19.237 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.2", 2024-04-06T13:47:19.237 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm/Gz5FBAA3xX/rsAyWZ1JoUVlQlBIvQ==", 2024-04-06T13:47:19.237 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:47:19.237 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:47:19.237 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:47:19.237 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:47:19.237 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:47:19.237 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.238 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:19.238 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:19.238 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.3", 2024-04-06T13:47:19.238 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmzra7GxAADbnzIvWnRTo65LvPwtxWrg==", 2024-04-06T13:47:19.238 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:47:19.238 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:47:19.238 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:47:19.238 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:47:19.238 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:47:19.238 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.238 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:19.238 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:19.238 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.admin", 2024-04-06T13:47:19.238 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmxG+wCBAANg5kt6TOUQdj3/X0Mqgx1g==", 2024-04-06T13:47:19.239 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:47:19.239 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:47:19.239 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:47:19.239 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:47:19.242 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:47:19.242 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.242 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:19.242 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:19.242 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.x", 2024-04-06T13:47:19.242 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmPLyBMBAAYfhkI9Z1CPlf1Tijm+L7xA==", 2024-04-06T13:47:19.242 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:47:19.242 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:47:19.242 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T13:47:19.242 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:47:19.242 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.242 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:19.242 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:19.243 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.y", 2024-04-06T13:47:19.243 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFm1nHJHhAAQWRe8cU8duWuUJiYaDeTaQ==", 2024-04-06T13:47:19.243 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:47:19.243 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:47:19.243 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T13:47:19.243 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:47:19.243 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.243 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:19.243 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-04-06T13:47:19.243 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:47:19.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.217+0000 7fd879592700 1 -- 172.21.15.189:0/2732925497 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd86006ade0 msgr2=0x7fd86006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:19.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.217+0000 7fd879592700 1 --2- 172.21.15.189:0/2732925497 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd86006ade0 0x7fd86006d2a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd864000c00 tx=0x7fd864007000 comp rx=0 tx=0).stop 2024-04-06T13:47:19.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.217+0000 7fd879592700 1 -- 172.21.15.189:0/2732925497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8740c7670 msgr2=0x7fd87415c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:19.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.217+0000 7fd879592700 1 --2- 172.21.15.189:0/2732925497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8740c7670 0x7fd87415c860 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7fd868011920 tx=0x7fd868011c30 comp rx=0 tx=0).stop 2024-04-06T13:47:19.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.217+0000 7fd879592700 1 -- 172.21.15.189:0/2732925497 shutdown_connections 2024-04-06T13:47:19.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.217+0000 7fd879592700 1 --2- 172.21.15.189:0/2732925497 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd86006ade0 0x7fd86006d2a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.217+0000 7fd879592700 1 --2- 172.21.15.189:0/2732925497 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8740b57c0 0x7fd8740b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.217+0000 7fd879592700 1 --2- 172.21.15.189:0/2732925497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd85c003a50 0x7fd8740b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.217+0000 7fd879592700 1 --2- 172.21.15.189:0/2732925497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8740c7670 0x7fd87415c860 unknown :-1 s=CLOSED pgs=2373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.217+0000 7fd879592700 1 -- 172.21.15.189:0/2732925497 >> 172.21.15.189:0/2732925497 conn(0x7fd87406a730 msgr2=0x7fd87415fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:19.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.217+0000 7fd879592700 1 -- 172.21.15.189:0/2732925497 shutdown_connections 2024-04-06T13:47:19.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.217+0000 7fd879592700 1 -- 172.21.15.189:0/2732925497 wait complete. 2024-04-06T13:47:19.247 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:47:19.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.368+0000 7f96a09a0700 1 Processor -- start 2024-04-06T13:47:19.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.368+0000 7f96a09a0700 1 -- start start 2024-04-06T13:47:19.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.369+0000 7f96a09a0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c158fe0 0x7f969c06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:19.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.369+0000 7f96a09a0700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f969c15c2d0 0x7f969c0ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.369+0000 7f96a09a0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c15f9e0 con 0x7f969c06ac70 2024-04-06T13:47:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.369+0000 7f96a09a0700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f969c06b0f0 con 0x7f969c15c2d0 2024-04-06T13:47:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.369+0000 7f96a09a0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969c15f0d0 con 0x7f969c158fe0 2024-04-06T13:47:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.369+0000 7f9699d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f969c06ac70 0x7f969c15bbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58426/0 (socket says 172.21.15.189:58426) 2024-04-06T13:47:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.369+0000 7f9699d9b700 1 -- 172.21.15.189:0/618952144 learned_addr learned my addr 172.21.15.189:0/618952144 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.369+0000 7f969959a700 1 -- 172.21.15.189:0/618952144 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2759809796 0 0) 0x7f969c15f9e0 con 0x7f969c06ac70 2024-04-06T13:47:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.369+0000 7f969959a700 1 -- 172.21.15.189:0/618952144 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96840035b0 con 0x7f969c06ac70 2024-04-06T13:47:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.369+0000 7f969a59c700 1 --2- 172.21.15.189:0/618952144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c158fe0 0x7f969c06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.369+0000 7f969ad9d700 1 --2- 172.21.15.189:0/618952144 >> v2:172.21.15.192:3300/0 conn(0x7f969c15c2d0 0x7f969c0ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.369+0000 7f969959a700 1 -- 172.21.15.189:0/618952144 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1564106581 0 0) 0x7f96840035b0 con 0x7f969c06ac70 2024-04-06T13:47:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f969959a700 1 -- 172.21.15.189:0/618952144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c158fe0 msgr2=0x7f969c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f969959a700 1 --2- 172.21.15.189:0/618952144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c158fe0 0x7f969c06a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f969959a700 1 -- 172.21.15.189:0/618952144 >> v2:172.21.15.192:3300/0 conn(0x7f969c15c2d0 msgr2=0x7f969c0ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:19.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f969959a700 1 --2- 172.21.15.189:0/618952144 >> v2:172.21.15.192:3300/0 conn(0x7f969c15c2d0 0x7f969c0ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f969959a700 1 -- 172.21.15.189:0/618952144 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c15c8f0 con 0x7f969c06ac70 2024-04-06T13:47:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f969959a700 1 -- 172.21.15.189:0/618952144 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f968c001670 con 0x7f969c06ac70 2024-04-06T13:47:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f969959a700 1 -- 172.21.15.189:0/618952144 >> v1:172.21.15.189:6789/0 conn(0x7f969c06ac70 legacy=0x7f969c15bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f969959a700 1 --2- 172.21.15.189:0/618952144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9684003a00 0x7f9684003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f969959a700 1 --2- 172.21.15.189:0/618952144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9684004850 0x7f9684004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f969959a700 1 --2- 172.21.15.189:0/618952144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c158fe0 0x7f9684005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f969959a700 1 -- 172.21.15.189:0/618952144 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c06b0f0 con 0x7f9684004850 2024-04-06T13:47:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f969959a700 1 -- 172.21.15.189:0/618952144 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c15f0d0 con 0x7f9684003a00 2024-04-06T13:47:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f969959a700 1 -- 172.21.15.189:0/618952144 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f968400ba10 con 0x7f969c158fe0 2024-04-06T13:47:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f9699d9b700 1 --2- 172.21.15.189:0/618952144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9684004850 0x7f9684004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f969a59c700 1 --2- 172.21.15.189:0/618952144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9684003a00 0x7f9684003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f969ad9d700 1 --2- 172.21.15.189:0/618952144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c158fe0 0x7f9684005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f9699d9b700 1 -- 172.21.15.189:0/618952144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c158fe0 msgr2=0x7f9684005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f9699d9b700 1 --2- 172.21.15.189:0/618952144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c158fe0 0x7f9684005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f9699d9b700 1 -- 172.21.15.189:0/618952144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9684003a00 msgr2=0x7f9684003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:19.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f9699d9b700 1 --2- 172.21.15.189:0/618952144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9684003a00 0x7f9684003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f9699d9b700 1 -- 172.21.15.189:0/618952144 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96840035b0 con 0x7f9684004850 2024-04-06T13:47:19.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f9699d9b700 1 --2- 172.21.15.189:0/618952144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9684004850 0x7f9684004cd0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f968c00c320 tx=0x7f9684005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8813d7557cef5657 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:19.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f969959a700 1 -- 172.21.15.189:0/618952144 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c00ce20 con 0x7f9684004850 2024-04-06T13:47:19.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f969959a700 1 -- 172.21.15.189:0/618952144 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968c005960 con 0x7f9684004850 2024-04-06T13:47:19.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.370+0000 7f969959a700 1 -- 172.21.15.189:0/618952144 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c005d10 con 0x7f9684004850 2024-04-06T13:47:19.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.371+0000 7f96a09a0700 1 -- 172.21.15.189:0/618952144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9684004850 msgr2=0x7f9684004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:19.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.371+0000 7f96a09a0700 1 --2- 172.21.15.189:0/618952144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9684004850 0x7f9684004cd0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f968c00c320 tx=0x7f9684005880 comp rx=0 tx=0).stop 2024-04-06T13:47:19.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.371+0000 7f969ad9d700 1 -- 172.21.15.189:0/618952144 reap_dead start 2024-04-06T13:47:19.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.371+0000 7f96a09a0700 1 -- 172.21.15.189:0/618952144 shutdown_connections 2024-04-06T13:47:19.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.371+0000 7f96a09a0700 1 -- 172.21.15.189:0/618952144 >> 172.21.15.189:0/618952144 conn(0x7f969c0d00b0 msgr2=0x7f969c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:19.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.371+0000 7f96a09a0700 1 -- 172.21.15.189:0/618952144 shutdown_connections 2024-04-06T13:47:19.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.371+0000 7f96a09a0700 1 -- 172.21.15.189:0/618952144 wait complete. 2024-04-06T13:47:19.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.372+0000 7f96a09a0700 1 Processor -- start 2024-04-06T13:47:19.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.372+0000 7f96a09a0700 1 -- start start 2024-04-06T13:47:19.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.373+0000 7f96a09a0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9684004850 0x7f969c141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:19.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.373+0000 7f96a09a0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f969c1f3c10 0x7f969c1f1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:19.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.373+0000 7f96a09a0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c1f4c70 0x7f969c15dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:19.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.373+0000 7f96a09a0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c1f4070 con 0x7f969c1f3c10 2024-04-06T13:47:19.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.373+0000 7f96a09a0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c1f5100 con 0x7f9684004850 2024-04-06T13:47:19.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.373+0000 7f96a09a0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969c141a90 con 0x7f969c1f4c70 2024-04-06T13:47:19.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.373+0000 7f9699d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f969c1f3c10 0x7f969c1f1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:19.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.373+0000 7f9699d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f969c1f3c10 0x7f969c1f1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54894/0 (socket says 172.21.15.189:54894) 2024-04-06T13:47:19.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.373+0000 7f9699d9b700 1 -- 172.21.15.189:0/1458605168 learned_addr learned my addr 172.21.15.189:0/1458605168 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:19.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.373+0000 7f969a59c700 1 --2- 172.21.15.189:0/1458605168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9684004850 0x7f969c141400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:19.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.373+0000 7f969ad9d700 1 --2- 172.21.15.189:0/1458605168 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c1f4c70 0x7f969c15dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:19.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.374+0000 7f9699d9b700 1 -- 172.21.15.189:0/1458605168 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c1f4c70 msgr2=0x7f969c15dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:19.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.374+0000 7f9699d9b700 1 --2- 172.21.15.189:0/1458605168 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c1f4c70 0x7f969c15dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.374+0000 7f9699d9b700 1 -- 172.21.15.189:0/1458605168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9684004850 msgr2=0x7f969c141400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:19.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.374+0000 7f9699d9b700 1 --2- 172.21.15.189:0/1458605168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9684004850 0x7f969c141400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.374+0000 7f9699d9b700 1 -- 172.21.15.189:0/1458605168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c141d70 con 0x7f969c1f3c10 2024-04-06T13:47:19.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.374+0000 7f9699d9b700 1 --2- 172.21.15.189:0/1458605168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f969c1f3c10 0x7f969c1f1dd0 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f968c008fd0 tx=0x7f969c0d1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:19.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.374+0000 7f968affd700 1 -- 172.21.15.189:0/1458605168 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c004a80 con 0x7f969c1f3c10 2024-04-06T13:47:19.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.374+0000 7f968affd700 1 -- 172.21.15.189:0/1458605168 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968c004c00 con 0x7f969c1f3c10 2024-04-06T13:47:19.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.374+0000 7f96a09a0700 1 -- 172.21.15.189:0/1458605168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f969c142050 con 0x7f969c1f3c10 2024-04-06T13:47:19.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.374+0000 7f968affd700 1 -- 172.21.15.189:0/1458605168 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c004db0 con 0x7f969c1f3c10 2024-04-06T13:47:19.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.374+0000 7f96a09a0700 1 -- 172.21.15.189:0/1458605168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f969c13de80 con 0x7f969c1f3c10 2024-04-06T13:47:19.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.375+0000 7f96a09a0700 1 -- 172.21.15.189:0/1458605168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f969c067bb0 con 0x7f969c1f3c10 2024-04-06T13:47:19.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.375+0000 7f968affd700 1 -- 172.21.15.189:0/1458605168 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f968c0070d0 con 0x7f969c1f3c10 2024-04-06T13:47:19.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.376+0000 7f968affd700 1 --2- 172.21.15.189:0/1458605168 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f967c06adb0 0x7f967c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:19.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.376+0000 7f969a59c700 1 --2- 172.21.15.189:0/1458605168 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f967c06adb0 0x7f967c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:19.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.376+0000 7f968affd700 1 -- 172.21.15.189:0/1458605168 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f968c08a770 con 0x7f969c1f3c10 2024-04-06T13:47:19.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.376+0000 7f969a59c700 1 --2- 172.21.15.189:0/1458605168 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f967c06adb0 0x7f967c06d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f969c0cfb20 tx=0x7f9690009210 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:19.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.380+0000 7f968affd700 1 -- 172.21.15.189:0/1458605168 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f968c055bf0 con 0x7f969c1f3c10 2024-04-06T13:47:19.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.576+0000 7f96a09a0700 1 -- 172.21.15.189:0/1458605168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f969c074990 con 0x7f969c1f3c10 2024-04-06T13:47:19.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.577+0000 7f968affd700 1 -- 172.21.15.189:0/1458605168 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v223) v1 ==== 74+0+12768 (secure 0 0 0) 0x7f968c007da0 con 0x7f969c1f3c10 2024-04-06T13:47:19.577 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:19.578 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":223,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:47:18.781046+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:47:19.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.579+0000 7f96a09a0700 1 -- 172.21.15.189:0/1458605168 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f967c06adb0 msgr2=0x7f967c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:19.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.579+0000 7f96a09a0700 1 --2- 172.21.15.189:0/1458605168 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f967c06adb0 0x7f967c06d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f969c0cfb20 tx=0x7f9690009210 comp rx=0 tx=0).stop 2024-04-06T13:47:19.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.579+0000 7f96a09a0700 1 -- 172.21.15.189:0/1458605168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f969c1f3c10 msgr2=0x7f969c1f1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:19.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.579+0000 7f96a09a0700 1 --2- 172.21.15.189:0/1458605168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f969c1f3c10 0x7f969c1f1dd0 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f968c008fd0 tx=0x7f969c0d1040 comp rx=0 tx=0).stop 2024-04-06T13:47:19.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.579+0000 7f96a09a0700 1 -- 172.21.15.189:0/1458605168 shutdown_connections 2024-04-06T13:47:19.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.579+0000 7f96a09a0700 1 --2- 172.21.15.189:0/1458605168 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f967c06adb0 0x7f967c06d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.579+0000 7f96a09a0700 1 --2- 172.21.15.189:0/1458605168 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f969c1f4c70 0x7f969c15dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.579+0000 7f96a09a0700 1 --2- 172.21.15.189:0/1458605168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9684004850 0x7f969c141400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.579+0000 7f96a09a0700 1 --2- 172.21.15.189:0/1458605168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f969c1f3c10 0x7f969c1f1dd0 unknown :-1 s=CLOSED pgs=2376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.579+0000 7f96a09a0700 1 -- 172.21.15.189:0/1458605168 >> 172.21.15.189:0/1458605168 conn(0x7f969c0d00b0 msgr2=0x7f969c143ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:19.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.579+0000 7f96a09a0700 1 -- 172.21.15.189:0/1458605168 shutdown_connections 2024-04-06T13:47:19.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.579+0000 7f96a09a0700 1 -- 172.21.15.189:0/1458605168 wait complete. 2024-04-06T13:47:19.592 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:47:19.593 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:47:19.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.719+0000 7f7ee2b07700 1 Processor -- start 2024-04-06T13:47:19.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.719+0000 7f7ee2b07700 1 -- start start 2024-04-06T13:47:19.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.720+0000 7f7ee2b07700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7edc06ae40 0x7f7edc0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:19.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.720+0000 7f7ee2b07700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7edc0b3990 con 0x7f7edc15d7b0 2024-04-06T13:47:19.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.720+0000 7f7ee2b07700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7edc0b3060 con 0x7f7edc15ddf0 2024-04-06T13:47:19.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.720+0000 7f7ee2b07700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7edc15e950 con 0x7f7edc06ae40 2024-04-06T13:47:19.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.720+0000 7f7ee08a3700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7edc15d7b0 0x7f7edc06a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58434/0 (socket says 172.21.15.189:58434) 2024-04-06T13:47:19.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.720+0000 7f7ee08a3700 1 -- 172.21.15.189:0/1606379168 learned_addr learned my addr 172.21.15.189:0/1606379168 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:19.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.720+0000 7f7edb7fe700 1 -- 172.21.15.189:0/1606379168 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646711081 0 0) 0x7f7edc0b3990 con 0x7f7edc15d7b0 2024-04-06T13:47:19.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.720+0000 7f7edb7fe700 1 -- 172.21.15.189:0/1606379168 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ecc0035b0 con 0x7f7edc15d7b0 2024-04-06T13:47:19.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.720+0000 7f7edbfff700 1 --2- 172.21.15.189:0/1606379168 >> v2:172.21.15.192:3300/0 conn(0x7f7edc06ae40 0x7f7edc0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:19.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.720+0000 7f7edb7fe700 1 -- 172.21.15.189:0/1606379168 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2743142290 0 0) 0x7f7ecc0035b0 con 0x7f7edc15d7b0 2024-04-06T13:47:19.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.720+0000 7f7edb7fe700 1 -- 172.21.15.189:0/1606379168 >> v2:172.21.15.192:3300/0 conn(0x7f7edc06ae40 msgr2=0x7f7edc0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:19.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.720+0000 7f7edb7fe700 1 --2- 172.21.15.189:0/1606379168 >> v2:172.21.15.192:3300/0 conn(0x7f7edc06ae40 0x7f7edc0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.720+0000 7f7edb7fe700 1 -- 172.21.15.189:0/1606379168 >> v1:172.21.15.192:6789/0 conn(0x7f7edc15ddf0 legacy=0x7f7edc15e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:19.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.720+0000 7f7edb7fe700 1 -- 172.21.15.189:0/1606379168 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7edc0b3340 con 0x7f7edc15d7b0 2024-04-06T13:47:19.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.720+0000 7f7edb7fe700 1 -- 172.21.15.189:0/1606379168 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f7ed0001670 con 0x7f7edc15d7b0 2024-04-06T13:47:19.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.720+0000 7f7edb7fe700 1 -- 172.21.15.189:0/1606379168 >> v1:172.21.15.189:6789/0 conn(0x7f7edc15d7b0 legacy=0x7f7edc06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:19.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.721+0000 7f7edb7fe700 1 --2- 172.21.15.189:0/1606379168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ecc003a00 0x7f7ecc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:19.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.721+0000 7f7edb7fe700 1 --2- 172.21.15.189:0/1606379168 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ecc004850 0x7f7ecc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:19.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.721+0000 7f7edb7fe700 1 --2- 172.21.15.189:0/1606379168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ecc005210 0x7f7ecc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:19.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.721+0000 7f7edb7fe700 1 -- 172.21.15.189:0/1606379168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc00c110 con 0x7f7ecc005210 2024-04-06T13:47:19.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.721+0000 7f7edb7fe700 1 -- 172.21.15.189:0/1606379168 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc00c280 con 0x7f7ecc003a00 2024-04-06T13:47:19.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.721+0000 7f7edb7fe700 1 -- 172.21.15.189:0/1606379168 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc00c440 con 0x7f7ecc004850 2024-04-06T13:47:19.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.721+0000 7f7ee10a4700 1 --2- 172.21.15.189:0/1606379168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ecc005210 0x7f7ecc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:19.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.721+0000 7f7ee08a3700 1 --2- 172.21.15.189:0/1606379168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ecc003a00 0x7f7ecc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:19.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.721+0000 7f7edbfff700 1 --2- 172.21.15.189:0/1606379168 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ecc004850 0x7f7ecc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:19.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.721+0000 7f7ee10a4700 1 -- 172.21.15.189:0/1606379168 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ecc004850 msgr2=0x7f7ecc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:19.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.721+0000 7f7ee10a4700 1 --2- 172.21.15.189:0/1606379168 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ecc004850 0x7f7ecc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.721+0000 7f7ee10a4700 1 -- 172.21.15.189:0/1606379168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ecc003a00 msgr2=0x7f7ecc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:19.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.721+0000 7f7ee10a4700 1 --2- 172.21.15.189:0/1606379168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ecc003a00 0x7f7ecc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.721+0000 7f7ee10a4700 1 -- 172.21.15.189:0/1606379168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ecc0035b0 con 0x7f7ecc005210 2024-04-06T13:47:19.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.721+0000 7f7ee10a4700 1 -- 172.21.15.189:0/1606379168 reap_dead start 2024-04-06T13:47:19.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.721+0000 7f7ee10a4700 1 --2- 172.21.15.189:0/1606379168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ecc005210 0x7f7ecc00ba10 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f7ed40049e0 tx=0x7f7ecc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c3b7e0b3adbc8a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:19.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.721+0000 7f7edb7fe700 1 -- 172.21.15.189:0/1606379168 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed4004450 con 0x7f7ecc005210 2024-04-06T13:47:19.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.721+0000 7f7edb7fe700 1 -- 172.21.15.189:0/1606379168 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed400cc50 con 0x7f7ecc005210 2024-04-06T13:47:19.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.721+0000 7f7edb7fe700 1 -- 172.21.15.189:0/1606379168 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed4004a10 con 0x7f7ecc005210 2024-04-06T13:47:19.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.722+0000 7f7ee2b07700 1 -- 172.21.15.189:0/1606379168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ecc005210 msgr2=0x7f7ecc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:19.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.722+0000 7f7ee2b07700 1 --2- 172.21.15.189:0/1606379168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ecc005210 0x7f7ecc00ba10 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f7ed40049e0 tx=0x7f7ecc005880 comp rx=0 tx=0).stop 2024-04-06T13:47:19.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.722+0000 7f7ee2b07700 1 -- 172.21.15.189:0/1606379168 shutdown_connections 2024-04-06T13:47:19.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.722+0000 7f7ee2b07700 1 --2- 172.21.15.189:0/1606379168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ecc005210 0x7f7ecc00ba10 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.722+0000 7f7ee2b07700 1 -- 172.21.15.189:0/1606379168 >> 172.21.15.189:0/1606379168 conn(0x7f7edc0d00b0 msgr2=0x7f7edc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:19.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.722+0000 7f7ee2b07700 1 -- 172.21.15.189:0/1606379168 shutdown_connections 2024-04-06T13:47:19.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.722+0000 7f7ee2b07700 1 -- 172.21.15.189:0/1606379168 wait complete. 2024-04-06T13:47:19.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.723+0000 7f7ee2b07700 1 Processor -- start 2024-04-06T13:47:19.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.723+0000 7f7ee2b07700 1 -- start start 2024-04-06T13:47:19.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.724+0000 7f7ee2b07700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ecc005210 0x7f7edc0aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:19.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.724+0000 7f7ee2b07700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7edc0ceec0 0x7f7edc1542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:19.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.724+0000 7f7ee2b07700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7edc0b0470 0x7f7edc0b0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:19.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.724+0000 7f7ee2b07700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc154830 con 0x7f7ecc005210 2024-04-06T13:47:19.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.724+0000 7f7ee2b07700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc0cf3f0 con 0x7f7edc0b0470 2024-04-06T13:47:19.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.724+0000 7f7ee2b07700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc0b1140 con 0x7f7edc0ceec0 2024-04-06T13:47:19.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.724+0000 7f7ee08a3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ecc005210 0x7f7edc0aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:19.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.724+0000 7f7ee08a3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ecc005210 0x7f7edc0aeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54916/0 (socket says 172.21.15.189:54916) 2024-04-06T13:47:19.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.724+0000 7f7ee08a3700 1 -- 172.21.15.189:0/3252991393 learned_addr learned my addr 172.21.15.189:0/3252991393 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:19.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.724+0000 7f7edbfff700 1 --2- 172.21.15.189:0/3252991393 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7edc0ceec0 0x7f7edc1542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:19.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.724+0000 7f7ee10a4700 1 --2- 172.21.15.189:0/3252991393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7edc0b0470 0x7f7edc0b0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:19.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.724+0000 7f7ee08a3700 1 -- 172.21.15.189:0/3252991393 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7edc0ceec0 msgr2=0x7f7edc1542f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:19.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.724+0000 7f7ee08a3700 1 --2- 172.21.15.189:0/3252991393 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7edc0ceec0 0x7f7edc1542f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.724+0000 7f7ee08a3700 1 -- 172.21.15.189:0/3252991393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7edc0b0470 msgr2=0x7f7edc0b0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:19.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.724+0000 7f7ee08a3700 1 --2- 172.21.15.189:0/3252991393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7edc0b0470 0x7f7edc0b0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.724+0000 7f7ee08a3700 1 -- 172.21.15.189:0/3252991393 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7edc0b1420 con 0x7f7ecc005210 2024-04-06T13:47:19.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.724+0000 7f7ee10a4700 1 --2- 172.21.15.189:0/3252991393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7edc0b0470 0x7f7edc0b0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:19.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.725+0000 7f7ee08a3700 1 --2- 172.21.15.189:0/3252991393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ecc005210 0x7f7edc0aeb80 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0007290 tx=0x7f7ed0007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:19.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.725+0000 7f7ed97fa700 1 -- 172.21.15.189:0/3252991393 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed000f6c0 con 0x7f7ecc005210 2024-04-06T13:47:19.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.725+0000 7f7ed97fa700 1 -- 172.21.15.189:0/3252991393 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed000f840 con 0x7f7ecc005210 2024-04-06T13:47:19.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.725+0000 7f7ee2b07700 1 -- 172.21.15.189:0/3252991393 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7edc0b1700 con 0x7f7ecc005210 2024-04-06T13:47:19.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.725+0000 7f7ed97fa700 1 -- 172.21.15.189:0/3252991393 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed000fbf0 con 0x7f7ecc005210 2024-04-06T13:47:19.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.725+0000 7f7ee2b07700 1 -- 172.21.15.189:0/3252991393 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7edc0b0e60 con 0x7f7ecc005210 2024-04-06T13:47:19.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.726+0000 7f7ee2b07700 1 -- 172.21.15.189:0/3252991393 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7edc067bb0 con 0x7f7ecc005210 2024-04-06T13:47:19.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.726+0000 7f7ed97fa700 1 -- 172.21.15.189:0/3252991393 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7ed000fd50 con 0x7f7ecc005210 2024-04-06T13:47:19.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.727+0000 7f7ed97fa700 1 --2- 172.21.15.189:0/3252991393 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7ec006adb0 0x7f7ec006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:19.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.727+0000 7f7ed97fa700 1 -- 172.21.15.189:0/3252991393 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f7ed0089220 con 0x7f7ecc005210 2024-04-06T13:47:19.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.727+0000 7f7edbfff700 1 --2- 172.21.15.189:0/3252991393 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7ec006adb0 0x7f7ec006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:19.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.727+0000 7f7edbfff700 1 --2- 172.21.15.189:0/3252991393 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7ec006adb0 0x7f7ec006d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7edc14bde0 tx=0x7f7ec8009210 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:19.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.731+0000 7f7ed97fa700 1 -- 172.21.15.189:0/3252991393 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7ed0054570 con 0x7f7ecc005210 2024-04-06T13:47:19.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.924+0000 7f7ee2b07700 1 -- 172.21.15.189:0/3252991393 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7edc074990 con 0x7f7ecc005210 2024-04-06T13:47:19.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.925+0000 7f7ed97fa700 1 -- 172.21.15.189:0/3252991393 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v223) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7ed0057b90 con 0x7f7ecc005210 2024-04-06T13:47:19.926 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:47:19.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.927+0000 7f7ee2b07700 1 -- 172.21.15.189:0/3252991393 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7ec006adb0 msgr2=0x7f7ec006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:19.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.927+0000 7f7ee2b07700 1 --2- 172.21.15.189:0/3252991393 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7ec006adb0 0x7f7ec006d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7edc14bde0 tx=0x7f7ec8009210 comp rx=0 tx=0).stop 2024-04-06T13:47:19.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.927+0000 7f7ee2b07700 1 -- 172.21.15.189:0/3252991393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ecc005210 msgr2=0x7f7edc0aeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:19.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.927+0000 7f7ee2b07700 1 --2- 172.21.15.189:0/3252991393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ecc005210 0x7f7edc0aeb80 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0007290 tx=0x7f7ed0007650 comp rx=0 tx=0).stop 2024-04-06T13:47:19.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.927+0000 7f7ee2b07700 1 -- 172.21.15.189:0/3252991393 shutdown_connections 2024-04-06T13:47:19.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.927+0000 7f7ee2b07700 1 --2- 172.21.15.189:0/3252991393 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7ec006adb0 0x7f7ec006d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.927+0000 7f7ee2b07700 1 --2- 172.21.15.189:0/3252991393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7edc0b0470 0x7f7edc0b0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.927+0000 7f7ee2b07700 1 --2- 172.21.15.189:0/3252991393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ecc005210 0x7f7edc0aeb80 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.927+0000 7f7ee2b07700 1 --2- 172.21.15.189:0/3252991393 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7edc0ceec0 0x7f7edc1542f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:19.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.927+0000 7f7ee2b07700 1 -- 172.21.15.189:0/3252991393 >> 172.21.15.189:0/3252991393 conn(0x7f7edc0d00b0 msgr2=0x7f7edc0ada90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:19.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.927+0000 7f7ee2b07700 1 -- 172.21.15.189:0/3252991393 shutdown_connections 2024-04-06T13:47:19.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:19.927+0000 7f7ee2b07700 1 -- 172.21.15.189:0/3252991393 wait complete. 2024-04-06T13:47:19.940 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-06T13:47:20.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.063+0000 7fec45014700 1 Processor -- start 2024-04-06T13:47:20.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.063+0000 7fec45014700 1 -- start start 2024-04-06T13:47:20.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.064+0000 7fec45014700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fec400b4660 0x7fec400b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.064+0000 7fec45014700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec400b5030 con 0x7fec400b2fc0 2024-04-06T13:47:20.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.064+0000 7fec45014700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec400cdbe0 con 0x7fec4015a550 2024-04-06T13:47:20.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.064+0000 7fec45014700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec4015a940 con 0x7fec400b4660 2024-04-06T13:47:20.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.064+0000 7fec3e59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fec400b4660 0x7fec400b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.064+0000 7fec3f59e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fec400b2fc0 0x7fec400b33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58444/0 (socket says 172.21.15.189:58444) 2024-04-06T13:47:20.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.064+0000 7fec3f59e700 1 -- 172.21.15.189:0/1052431463 learned_addr learned my addr 172.21.15.189:0/1052431463 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:20.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.064+0000 7fec3e59c700 1 -- 172.21.15.189:0/1052431463 >> v1:172.21.15.192:6789/0 conn(0x7fec4015a550 legacy=0x7fec400cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:20.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.064+0000 7fec3e59c700 1 -- 172.21.15.189:0/1052431463 >> v1:172.21.15.189:6789/0 conn(0x7fec400b2fc0 legacy=0x7fec400b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:20.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.064+0000 7fec3e59c700 1 -- 172.21.15.189:0/1052431463 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4015ac00 con 0x7fec400b4660 2024-04-06T13:47:20.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.064+0000 7fec3dd9b700 1 -- 172.21.15.189:0/1052431463 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054139650 0 0) 0x7fec400b5030 con 0x7fec400b2fc0 2024-04-06T13:47:20.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.064+0000 7fec3e59c700 1 --2- 172.21.15.189:0/1052431463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec400b4660 0x7fec400b4af0 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7fec34009840 tx=0x7fec34009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e2e34317e122412 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:20.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.064+0000 7fec3dd9b700 1 -- 172.21.15.189:0/1052431463 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec34004230 con 0x7fec400b4660 2024-04-06T13:47:20.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.064+0000 7fec3dd9b700 1 -- 172.21.15.189:0/1052431463 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec34004390 con 0x7fec400b4660 2024-04-06T13:47:20.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.064+0000 7fec3dd9b700 1 -- 172.21.15.189:0/1052431463 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec340036a0 con 0x7fec400b4660 2024-04-06T13:47:20.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.065+0000 7fec45014700 1 -- 172.21.15.189:0/1052431463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec400b4660 msgr2=0x7fec400b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:20.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.065+0000 7fec45014700 1 --2- 172.21.15.189:0/1052431463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec400b4660 0x7fec400b4af0 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7fec34009840 tx=0x7fec34009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:20.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.065+0000 7fec45014700 1 -- 172.21.15.189:0/1052431463 shutdown_connections 2024-04-06T13:47:20.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.065+0000 7fec45014700 1 --2- 172.21.15.189:0/1052431463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec400b4660 0x7fec400b4af0 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.065+0000 7fec45014700 1 -- 172.21.15.189:0/1052431463 >> 172.21.15.189:0/1052431463 conn(0x7fec400d00b0 msgr2=0x7fec400d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:20.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.065+0000 7fec45014700 1 -- 172.21.15.189:0/1052431463 shutdown_connections 2024-04-06T13:47:20.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.065+0000 7fec45014700 1 -- 172.21.15.189:0/1052431463 wait complete. 2024-04-06T13:47:20.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.066+0000 7fec45014700 1 Processor -- start 2024-04-06T13:47:20.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.066+0000 7fec45014700 1 -- start start 2024-04-06T13:47:20.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.067+0000 7fec45014700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec400b2fc0 0x7fec4015c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.067+0000 7fec45014700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec400b4660 0x7fec4014d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.067+0000 7fec45014700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec401f0040 0x7fec401f3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.067+0000 7fec45014700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4014f0e0 con 0x7fec400b2fc0 2024-04-06T13:47:20.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.067+0000 7fec45014700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec401446a0 con 0x7fec401f0040 2024-04-06T13:47:20.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.067+0000 7fec45014700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec4014edb0 con 0x7fec400b4660 2024-04-06T13:47:20.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.067+0000 7fec3ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec400b2fc0 0x7fec4015c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.067+0000 7fec3ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec400b2fc0 0x7fec4015c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54936/0 (socket says 172.21.15.189:54936) 2024-04-06T13:47:20.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.067+0000 7fec3ed9d700 1 -- 172.21.15.189:0/531970310 learned_addr learned my addr 172.21.15.189:0/531970310 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:20.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.068+0000 7fec3e59c700 1 --2- 172.21.15.189:0/531970310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec400b4660 0x7fec4014d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.068+0000 7fec3f59e700 1 --2- 172.21.15.189:0/531970310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec401f0040 0x7fec401f3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.068+0000 7fec3ed9d700 1 -- 172.21.15.189:0/531970310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec400b4660 msgr2=0x7fec4014d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:20.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.068+0000 7fec3ed9d700 1 --2- 172.21.15.189:0/531970310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec400b4660 0x7fec4014d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.068+0000 7fec3ed9d700 1 -- 172.21.15.189:0/531970310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec401f0040 msgr2=0x7fec401f3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:20.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.068+0000 7fec3ed9d700 1 --2- 172.21.15.189:0/531970310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec401f0040 0x7fec401f3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.068+0000 7fec3ed9d700 1 -- 172.21.15.189:0/531970310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec340094a0 con 0x7fec400b2fc0 2024-04-06T13:47:20.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.068+0000 7fec3ed9d700 1 --2- 172.21.15.189:0/531970310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec400b2fc0 0x7fec4015c3d0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7fec3000b4d0 tx=0x7fec3000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:20.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.068+0000 7fec277fe700 1 -- 172.21.15.189:0/531970310 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec30011830 con 0x7fec400b2fc0 2024-04-06T13:47:20.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.068+0000 7fec277fe700 1 -- 172.21.15.189:0/531970310 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec30011990 con 0x7fec400b2fc0 2024-04-06T13:47:20.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.068+0000 7fec45014700 1 -- 172.21.15.189:0/531970310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec4015d300 con 0x7fec400b2fc0 2024-04-06T13:47:20.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.068+0000 7fec277fe700 1 -- 172.21.15.189:0/531970310 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec30011d40 con 0x7fec400b2fc0 2024-04-06T13:47:20.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.068+0000 7fec45014700 1 -- 172.21.15.189:0/531970310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec4015cd40 con 0x7fec400b2fc0 2024-04-06T13:47:20.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.069+0000 7fec45014700 1 -- 172.21.15.189:0/531970310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec40067bb0 con 0x7fec400b2fc0 2024-04-06T13:47:20.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.069+0000 7fec277fe700 1 -- 172.21.15.189:0/531970310 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fec3001a400 con 0x7fec400b2fc0 2024-04-06T13:47:20.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.070+0000 7fec277fe700 1 --2- 172.21.15.189:0/531970310 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fec2c06adb0 0x7fec2c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.070+0000 7fec277fe700 1 -- 172.21.15.189:0/531970310 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fec30088f60 con 0x7fec400b2fc0 2024-04-06T13:47:20.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.070+0000 7fec3e59c700 1 --2- 172.21.15.189:0/531970310 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fec2c06adb0 0x7fec2c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.070+0000 7fec3e59c700 1 --2- 172.21.15.189:0/531970310 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fec2c06adb0 0x7fec2c06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fec34000c00 tx=0x7fec3401c040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:20.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.074+0000 7fec277fe700 1 -- 172.21.15.189:0/531970310 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fec30054280 con 0x7fec400b2fc0 2024-04-06T13:47:20.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.270+0000 7fec45014700 1 -- 172.21.15.189:0/531970310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fec40074990 con 0x7fec400b2fc0 2024-04-06T13:47:20.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.271+0000 7fec277fe700 1 -- 172.21.15.189:0/531970310 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v223) v1 ==== 84+0+65 (secure 0 0 0) 0x7fec30015060 con 0x7fec400b2fc0 2024-04-06T13:47:20.272 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:20.272 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T13:47:20.272 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:20.272 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 1, 2024-04-06T13:47:20.272 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": ".mgr" 2024-04-06T13:47:20.272 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:20.272 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T13:47:20.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.273+0000 7fec45014700 1 -- 172.21.15.189:0/531970310 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fec2c06adb0 msgr2=0x7fec2c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:20.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.273+0000 7fec45014700 1 --2- 172.21.15.189:0/531970310 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fec2c06adb0 0x7fec2c06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fec34000c00 tx=0x7fec3401c040 comp rx=0 tx=0).stop 2024-04-06T13:47:20.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.273+0000 7fec45014700 1 -- 172.21.15.189:0/531970310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec400b2fc0 msgr2=0x7fec4015c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:20.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.273+0000 7fec45014700 1 --2- 172.21.15.189:0/531970310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec400b2fc0 0x7fec4015c3d0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7fec3000b4d0 tx=0x7fec3000b890 comp rx=0 tx=0).stop 2024-04-06T13:47:20.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.273+0000 7fec45014700 1 -- 172.21.15.189:0/531970310 shutdown_connections 2024-04-06T13:47:20.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.273+0000 7fec45014700 1 --2- 172.21.15.189:0/531970310 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fec2c06adb0 0x7fec2c06d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.273+0000 7fec45014700 1 --2- 172.21.15.189:0/531970310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec401f0040 0x7fec401f3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.273+0000 7fec45014700 1 --2- 172.21.15.189:0/531970310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec400b2fc0 0x7fec4015c3d0 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.273+0000 7fec45014700 1 --2- 172.21.15.189:0/531970310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec400b4660 0x7fec4014d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.273+0000 7fec45014700 1 -- 172.21.15.189:0/531970310 >> 172.21.15.189:0/531970310 conn(0x7fec400d00b0 msgr2=0x7fec401582c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:20.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.273+0000 7fec45014700 1 -- 172.21.15.189:0/531970310 shutdown_connections 2024-04-06T13:47:20.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.273+0000 7fec45014700 1 -- 172.21.15.189:0/531970310 wait complete. 2024-04-06T13:47:20.287 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-06T13:47:20.287 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-06T13:47:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.412+0000 7fbd60453700 1 Processor -- start 2024-04-06T13:47:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.412+0000 7fbd60453700 1 -- start start 2024-04-06T13:47:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd60453700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd5815d7b0 0x7fbd580b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd60453700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbd5806a730 0x7fbd5806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd60453700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd580b3990 con 0x7fbd5815ddf0 2024-04-06T13:47:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd60453700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd5806b0f0 con 0x7fbd5806a730 2024-04-06T13:47:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd60453700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd580b3060 con 0x7fbd5815d7b0 2024-04-06T13:47:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd5e9f0700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbd5815ddf0 0x7fbd5815e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58460/0 (socket says 172.21.15.189:58460) 2024-04-06T13:47:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd5e9f0700 1 -- 172.21.15.189:0/1525078011 learned_addr learned my addr 172.21.15.189:0/1525078011 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd5d1ed700 1 -- 172.21.15.189:0/1525078011 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718042578 0 0) 0x7fbd580b3990 con 0x7fbd5815ddf0 2024-04-06T13:47:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd5e1ef700 1 --2- 172.21.15.189:0/1525078011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd5815d7b0 0x7fbd580b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd5d9ee700 1 --2- 172.21.15.189:0/1525078011 >> v2:172.21.15.192:3300/0 conn(0x7fbd5806a730 0x7fbd5806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd5d1ed700 1 -- 172.21.15.189:0/1525078011 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd440035b0 con 0x7fbd5815ddf0 2024-04-06T13:47:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd5d1ed700 1 -- 172.21.15.189:0/1525078011 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 351899699 0 0) 0x7fbd440035b0 con 0x7fbd5815ddf0 2024-04-06T13:47:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd5d1ed700 1 -- 172.21.15.189:0/1525078011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd5815d7b0 msgr2=0x7fbd580b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd5d1ed700 1 --2- 172.21.15.189:0/1525078011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd5815d7b0 0x7fbd580b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd5d1ed700 1 -- 172.21.15.189:0/1525078011 >> v2:172.21.15.192:3300/0 conn(0x7fbd5806a730 msgr2=0x7fbd5806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd5d1ed700 1 --2- 172.21.15.189:0/1525078011 >> v2:172.21.15.192:3300/0 conn(0x7fbd5806a730 0x7fbd5806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd5d1ed700 1 -- 172.21.15.189:0/1525078011 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd580b3340 con 0x7fbd5815ddf0 2024-04-06T13:47:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd5d1ed700 1 -- 172.21.15.189:0/1525078011 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fbd500019f0 con 0x7fbd5815ddf0 2024-04-06T13:47:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd5d1ed700 1 -- 172.21.15.189:0/1525078011 >> v1:172.21.15.189:6789/0 conn(0x7fbd5815ddf0 legacy=0x7fbd5815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd5d1ed700 1 --2- 172.21.15.189:0/1525078011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd44003a00 0x7fbd44003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.413+0000 7fbd5d1ed700 1 --2- 172.21.15.189:0/1525078011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd44004850 0x7fbd44004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.414+0000 7fbd5d1ed700 1 --2- 172.21.15.189:0/1525078011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd5815d7b0 0x7fbd44005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.414+0000 7fbd5d1ed700 1 -- 172.21.15.189:0/1525078011 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5806b0f0 con 0x7fbd44003a00 2024-04-06T13:47:20.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.414+0000 7fbd5d1ed700 1 -- 172.21.15.189:0/1525078011 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd580b3060 con 0x7fbd44004850 2024-04-06T13:47:20.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.414+0000 7fbd5d1ed700 1 -- 172.21.15.189:0/1525078011 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4400bac0 con 0x7fbd5815d7b0 2024-04-06T13:47:20.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.414+0000 7fbd5d9ee700 1 --2- 172.21.15.189:0/1525078011 >> v2:172.21.15.192:3300/0 conn(0x7fbd5806a730 0x7fbd5806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:20.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.414+0000 7fbd5e1ef700 1 --2- 172.21.15.189:0/1525078011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd44003a00 0x7fbd44003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.414+0000 7fbd5e9f0700 1 --2- 172.21.15.189:0/1525078011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd5815d7b0 0x7fbd44005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.414+0000 7fbd5d9ee700 1 --2- 172.21.15.189:0/1525078011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd44004850 0x7fbd44004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.414+0000 7fbd5e1ef700 1 -- 172.21.15.189:0/1525078011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd5815d7b0 msgr2=0x7fbd44005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:20.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.414+0000 7fbd5e1ef700 1 --2- 172.21.15.189:0/1525078011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd5815d7b0 0x7fbd44005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.414+0000 7fbd5e1ef700 1 -- 172.21.15.189:0/1525078011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd44004850 msgr2=0x7fbd44004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:20.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.414+0000 7fbd5e1ef700 1 --2- 172.21.15.189:0/1525078011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd44004850 0x7fbd44004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.414+0000 7fbd5e1ef700 1 -- 172.21.15.189:0/1525078011 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd440035b0 con 0x7fbd44003a00 2024-04-06T13:47:20.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.414+0000 7fbd5e1ef700 1 --2- 172.21.15.189:0/1525078011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd44003a00 0x7fbd44003e20 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7fbd54009b30 tx=0x7fbd44005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70f1d1db644bb80c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:20.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.414+0000 7fbd5d1ed700 1 -- 172.21.15.189:0/1525078011 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd54009ea0 con 0x7fbd44003a00 2024-04-06T13:47:20.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.414+0000 7fbd5d1ed700 1 -- 172.21.15.189:0/1525078011 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5400e6c0 con 0x7fbd44003a00 2024-04-06T13:47:20.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.414+0000 7fbd5d1ed700 1 -- 172.21.15.189:0/1525078011 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5400ea30 con 0x7fbd44003a00 2024-04-06T13:47:20.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.415+0000 7fbd60453700 1 -- 172.21.15.189:0/1525078011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd44003a00 msgr2=0x7fbd44003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:20.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.415+0000 7fbd60453700 1 --2- 172.21.15.189:0/1525078011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd44003a00 0x7fbd44003e20 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7fbd54009b30 tx=0x7fbd44005880 comp rx=0 tx=0).stop 2024-04-06T13:47:20.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.415+0000 7fbd5e9f0700 1 -- 172.21.15.189:0/1525078011 reap_dead start 2024-04-06T13:47:20.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.415+0000 7fbd60453700 1 -- 172.21.15.189:0/1525078011 shutdown_connections 2024-04-06T13:47:20.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.415+0000 7fbd60453700 1 -- 172.21.15.189:0/1525078011 >> 172.21.15.189:0/1525078011 conn(0x7fbd580d00b0 msgr2=0x7fbd580d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:20.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.415+0000 7fbd60453700 1 -- 172.21.15.189:0/1525078011 shutdown_connections 2024-04-06T13:47:20.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.415+0000 7fbd60453700 1 -- 172.21.15.189:0/1525078011 wait complete. 2024-04-06T13:47:20.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.416+0000 7fbd60453700 1 Processor -- start 2024-04-06T13:47:20.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.416+0000 7fbd60453700 1 -- start start 2024-04-06T13:47:20.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.417+0000 7fbd60453700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd44003a00 0x7fbd58158e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.417+0000 7fbd60453700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd581f2dd0 0x7fbd580c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.417+0000 7fbd60453700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd581f3250 0x7fbd580c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.417+0000 7fbd60453700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd580c62a0 con 0x7fbd581f3250 2024-04-06T13:47:20.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.417+0000 7fbd60453700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd580c8840 con 0x7fbd581f2dd0 2024-04-06T13:47:20.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.417+0000 7fbd60453700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd580c5be0 con 0x7fbd44003a00 2024-04-06T13:47:20.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.417+0000 7fbd5e9f0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd581f3250 0x7fbd580c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.417+0000 7fbd5e9f0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd581f3250 0x7fbd580c8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54946/0 (socket says 172.21.15.189:54946) 2024-04-06T13:47:20.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.417+0000 7fbd5e9f0700 1 -- 172.21.15.189:0/1488166117 learned_addr learned my addr 172.21.15.189:0/1488166117 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:20.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.417+0000 7fbd5e1ef700 1 --2- 172.21.15.189:0/1488166117 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd44003a00 0x7fbd58158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.417+0000 7fbd5d9ee700 1 --2- 172.21.15.189:0/1488166117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd581f2dd0 0x7fbd580c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.417+0000 7fbd5e9f0700 1 -- 172.21.15.189:0/1488166117 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd44003a00 msgr2=0x7fbd58158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:20.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.417+0000 7fbd5e9f0700 1 --2- 172.21.15.189:0/1488166117 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd44003a00 0x7fbd58158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.417+0000 7fbd5e9f0700 1 -- 172.21.15.189:0/1488166117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd581f2dd0 msgr2=0x7fbd580c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:20.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.417+0000 7fbd5e9f0700 1 --2- 172.21.15.189:0/1488166117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd581f2dd0 0x7fbd580c7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.417+0000 7fbd5e9f0700 1 -- 172.21.15.189:0/1488166117 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd580c5ec0 con 0x7fbd581f3250 2024-04-06T13:47:20.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.417+0000 7fbd5e9f0700 1 --2- 172.21.15.189:0/1488166117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd581f3250 0x7fbd580c8230 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7fbd5806d2c0 tx=0x7fbd5000a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:20.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.418+0000 7fbd4effd700 1 -- 172.21.15.189:0/1488166117 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5000ad00 con 0x7fbd581f3250 2024-04-06T13:47:20.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.418+0000 7fbd4effd700 1 -- 172.21.15.189:0/1488166117 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5000ae60 con 0x7fbd581f3250 2024-04-06T13:47:20.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.418+0000 7fbd60453700 1 -- 172.21.15.189:0/1488166117 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd580c58e0 con 0x7fbd581f3250 2024-04-06T13:47:20.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.418+0000 7fbd4effd700 1 -- 172.21.15.189:0/1488166117 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50013630 con 0x7fbd581f3250 2024-04-06T13:47:20.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.418+0000 7fbd60453700 1 -- 172.21.15.189:0/1488166117 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd581593c0 con 0x7fbd581f3250 2024-04-06T13:47:20.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.419+0000 7fbd60453700 1 -- 172.21.15.189:0/1488166117 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd580c63e0 con 0x7fbd581f3250 2024-04-06T13:47:20.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.419+0000 7fbd4effd700 1 -- 172.21.15.189:0/1488166117 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fbd50013790 con 0x7fbd581f3250 2024-04-06T13:47:20.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.420+0000 7fbd4effd700 1 --2- 172.21.15.189:0/1488166117 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbd3c06adb0 0x7fbd3c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.420+0000 7fbd4effd700 1 -- 172.21.15.189:0/1488166117 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fbd50089e20 con 0x7fbd581f3250 2024-04-06T13:47:20.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.420+0000 7fbd5e1ef700 1 --2- 172.21.15.189:0/1488166117 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbd3c06adb0 0x7fbd3c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.420+0000 7fbd5e1ef700 1 --2- 172.21.15.189:0/1488166117 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbd3c06adb0 0x7fbd3c06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbd540090b0 tx=0x7fbd54008030 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:20.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.424+0000 7fbd4effd700 1 -- 172.21.15.189:0/1488166117 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbd580c63e0 con 0x7fbd581f3250 2024-04-06T13:47:20.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.621+0000 7fbd60453700 1 -- 172.21.15.189:0/1488166117 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fbd580c63e0 con 0x7fbd581f3250 2024-04-06T13:47:20.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.696+0000 7fbd4effd700 1 -- 172.21.15.189:0/1488166117 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v224) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbd580c63e0 con 0x7fbd581f3250 2024-04-06T13:47:20.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.698+0000 7fbd60453700 1 -- 172.21.15.189:0/1488166117 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbd3c06adb0 msgr2=0x7fbd3c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:20.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.698+0000 7fbd60453700 1 --2- 172.21.15.189:0/1488166117 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbd3c06adb0 0x7fbd3c06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbd540090b0 tx=0x7fbd54008030 comp rx=0 tx=0).stop 2024-04-06T13:47:20.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.698+0000 7fbd60453700 1 -- 172.21.15.189:0/1488166117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd581f3250 msgr2=0x7fbd580c8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:20.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.698+0000 7fbd60453700 1 --2- 172.21.15.189:0/1488166117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd581f3250 0x7fbd580c8230 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7fbd5806d2c0 tx=0x7fbd5000a640 comp rx=0 tx=0).stop 2024-04-06T13:47:20.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.698+0000 7fbd60453700 1 -- 172.21.15.189:0/1488166117 shutdown_connections 2024-04-06T13:47:20.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.698+0000 7fbd60453700 1 --2- 172.21.15.189:0/1488166117 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbd3c06adb0 0x7fbd3c06d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.698+0000 7fbd60453700 1 --2- 172.21.15.189:0/1488166117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd581f3250 0x7fbd580c8230 unknown :-1 s=CLOSED pgs=2385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.698+0000 7fbd60453700 1 --2- 172.21.15.189:0/1488166117 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd44003a00 0x7fbd58158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.698+0000 7fbd60453700 1 --2- 172.21.15.189:0/1488166117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd581f2dd0 0x7fbd580c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.698+0000 7fbd60453700 1 -- 172.21.15.189:0/1488166117 >> 172.21.15.189:0/1488166117 conn(0x7fbd580d00b0 msgr2=0x7fbd580ad370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:20.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.698+0000 7fbd60453700 1 -- 172.21.15.189:0/1488166117 shutdown_connections 2024-04-06T13:47:20.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.699+0000 7fbd60453700 1 -- 172.21.15.189:0/1488166117 wait complete. 2024-04-06T13:47:20.701 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' created 2024-04-06T13:47:20.716 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-06T13:47:20.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.844+0000 7f765be27700 1 Processor -- start 2024-04-06T13:47:20.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.845+0000 7f765be27700 1 -- start start 2024-04-06T13:47:20.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.845+0000 7f765be27700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7654146dd0 0x7f765406d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.845+0000 7f765be27700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f76540cedf0 0x7f76540cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.845+0000 7f765be27700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76540cf730 con 0x7f76540d17a0 2024-04-06T13:47:20.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.845+0000 7f765be27700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f765406beb0 con 0x7f7654146dd0 2024-04-06T13:47:20.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.845+0000 7f765be27700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f765406c0a0 con 0x7f76540cedf0 2024-04-06T13:47:20.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.846+0000 7f76593c2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7654146dd0 0x7f765406d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.846+0000 7f76593c2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7654146dd0 0x7f765406d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54962/0 (socket says 172.21.15.189:54962) 2024-04-06T13:47:20.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.846+0000 7f76593c2700 1 -- 172.21.15.189:0/1443374972 learned_addr learned my addr 172.21.15.189:0/1443374972 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:20.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.846+0000 7f765a3c4700 1 --2- 172.21.15.189:0/1443374972 >> v2:172.21.15.192:3300/0 conn(0x7f76540cedf0 0x7f76540cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.846+0000 7f76593c2700 1 -- 172.21.15.189:0/1443374972 >> v2:172.21.15.192:3300/0 conn(0x7f76540cedf0 msgr2=0x7f76540cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:20.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.846+0000 7f76593c2700 1 --2- 172.21.15.189:0/1443374972 >> v2:172.21.15.192:3300/0 conn(0x7f76540cedf0 0x7f76540cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.846+0000 7f76593c2700 1 -- 172.21.15.189:0/1443374972 >> v1:172.21.15.192:6789/0 conn(0x7f76540d17a0 legacy=0x7f765406b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:20.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.846+0000 7f76593c2700 1 -- 172.21.15.189:0/1443374972 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765406c380 con 0x7f7654146dd0 2024-04-06T13:47:20.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.846+0000 7f76593c2700 1 --2- 172.21.15.189:0/1443374972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7654146dd0 0x7f765406d140 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7f7654160fb0 tx=0x7f7644009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93927a729d025f83 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:20.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.846+0000 7f7658bc1700 1 -- 172.21.15.189:0/1443374972 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644004230 con 0x7f7654146dd0 2024-04-06T13:47:20.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.846+0000 7f7658bc1700 1 -- 172.21.15.189:0/1443374972 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7644004390 con 0x7f7654146dd0 2024-04-06T13:47:20.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.846+0000 7f7658bc1700 1 -- 172.21.15.189:0/1443374972 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76440036a0 con 0x7f7654146dd0 2024-04-06T13:47:20.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.847+0000 7f765be27700 1 -- 172.21.15.189:0/1443374972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7654146dd0 msgr2=0x7f765406d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:20.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.847+0000 7f765be27700 1 --2- 172.21.15.189:0/1443374972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7654146dd0 0x7f765406d140 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7f7654160fb0 tx=0x7f7644009bd0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.847+0000 7f765be27700 1 -- 172.21.15.189:0/1443374972 shutdown_connections 2024-04-06T13:47:20.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.847+0000 7f765be27700 1 --2- 172.21.15.189:0/1443374972 >> v2:172.21.15.192:3300/0 conn(0x7f76540cedf0 0x7f76540cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.847+0000 7f765be27700 1 --2- 172.21.15.189:0/1443374972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7654146dd0 0x7f765406d140 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.847+0000 7f765be27700 1 -- 172.21.15.189:0/1443374972 >> 172.21.15.189:0/1443374972 conn(0x7f76540b2db0 msgr2=0x7f76540b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:20.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.847+0000 7f765be27700 1 -- 172.21.15.189:0/1443374972 shutdown_connections 2024-04-06T13:47:20.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.847+0000 7f765be27700 1 -- 172.21.15.189:0/1443374972 wait complete. 2024-04-06T13:47:20.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.848+0000 7f765be27700 1 Processor -- start 2024-04-06T13:47:20.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.848+0000 7f765be27700 1 -- start start 2024-04-06T13:47:20.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f765be27700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76540cedf0 0x7f76541f2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f765be27700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76540d17a0 0x7f76540c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f765be27700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7654146dd0 0x7f76540aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f765be27700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76540ae660 con 0x7f76540d17a0 2024-04-06T13:47:20.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f765be27700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76540ada50 con 0x7f76540cedf0 2024-04-06T13:47:20.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f765be27700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76540add30 con 0x7f7654146dd0 2024-04-06T13:47:20.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f76593c2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76540d17a0 0x7f76540c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f76593c2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76540d17a0 0x7f76540c8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54972/0 (socket says 172.21.15.189:54972) 2024-04-06T13:47:20.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f76593c2700 1 -- 172.21.15.189:0/3408919770 learned_addr learned my addr 172.21.15.189:0/3408919770 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:20.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f7659bc3700 1 --2- 172.21.15.189:0/3408919770 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76540cedf0 0x7f76541f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f765a3c4700 1 --2- 172.21.15.189:0/3408919770 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7654146dd0 0x7f76540aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f76593c2700 1 -- 172.21.15.189:0/3408919770 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7654146dd0 msgr2=0x7f76540aeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:20.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f76593c2700 1 --2- 172.21.15.189:0/3408919770 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7654146dd0 0x7f76540aeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f76593c2700 1 -- 172.21.15.189:0/3408919770 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76540cedf0 msgr2=0x7f76541f2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:20.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f76593c2700 1 --2- 172.21.15.189:0/3408919770 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76540cedf0 0x7f76541f2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:20.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f76593c2700 1 -- 172.21.15.189:0/3408919770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7644009550 con 0x7f76540d17a0 2024-04-06T13:47:20.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f76593c2700 1 --2- 172.21.15.189:0/3408919770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76540d17a0 0x7f76540c8d00 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f7644009130 tx=0x7f7644004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:20.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f764a7fc700 1 -- 172.21.15.189:0/3408919770 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644003b90 con 0x7f76540d17a0 2024-04-06T13:47:20.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f764a7fc700 1 -- 172.21.15.189:0/3408919770 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764401f070 con 0x7f76540d17a0 2024-04-06T13:47:20.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f765be27700 1 -- 172.21.15.189:0/3408919770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76540ae010 con 0x7f76540d17a0 2024-04-06T13:47:20.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.849+0000 7f764a7fc700 1 -- 172.21.15.189:0/3408919770 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644011630 con 0x7f76540d17a0 2024-04-06T13:47:20.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.850+0000 7f765be27700 1 -- 172.21.15.189:0/3408919770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76540c4680 con 0x7f76540d17a0 2024-04-06T13:47:20.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.850+0000 7f765be27700 1 -- 172.21.15.189:0/3408919770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76540ae7d0 con 0x7f76540d17a0 2024-04-06T13:47:20.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.851+0000 7f764a7fc700 1 -- 172.21.15.189:0/3408919770 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7644021030 con 0x7f76540d17a0 2024-04-06T13:47:20.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.852+0000 7f764a7fc700 1 --2- 172.21.15.189:0/3408919770 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f763806adb0 0x7f763806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:20.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.852+0000 7f7659bc3700 1 --2- 172.21.15.189:0/3408919770 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f763806adb0 0x7f763806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:20.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.852+0000 7f764a7fc700 1 -- 172.21.15.189:0/3408919770 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f764408a920 con 0x7f76540d17a0 2024-04-06T13:47:20.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.852+0000 7f7659bc3700 1 --2- 172.21.15.189:0/3408919770 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f763806adb0 0x7f763806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7650009960 tx=0x7f76500090e0 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:20.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:20.856+0000 7f764a7fc700 1 -- 172.21.15.189:0/3408919770 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7644055b70 con 0x7f76540d17a0 2024-04-06T13:47:21.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.052+0000 7f765be27700 1 -- 172.21.15.189:0/3408919770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7654074990 con 0x7f76540d17a0 2024-04-06T13:47:21.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.699+0000 7f764a7fc700 1 -- 172.21.15.189:0/3408919770 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v225) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7644059190 con 0x7f76540d17a0 2024-04-06T13:47:21.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.710+0000 7f765be27700 1 -- 172.21.15.189:0/3408919770 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f763806adb0 msgr2=0x7f763806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:21.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.710+0000 7f765be27700 1 --2- 172.21.15.189:0/3408919770 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f763806adb0 0x7f763806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7650009960 tx=0x7f76500090e0 comp rx=0 tx=0).stop 2024-04-06T13:47:21.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.710+0000 7f765be27700 1 -- 172.21.15.189:0/3408919770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76540d17a0 msgr2=0x7f76540c8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:21.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.710+0000 7f765be27700 1 --2- 172.21.15.189:0/3408919770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76540d17a0 0x7f76540c8d00 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f7644009130 tx=0x7f7644004750 comp rx=0 tx=0).stop 2024-04-06T13:47:21.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.711+0000 7f765be27700 1 -- 172.21.15.189:0/3408919770 shutdown_connections 2024-04-06T13:47:21.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.711+0000 7f765be27700 1 --2- 172.21.15.189:0/3408919770 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f763806adb0 0x7f763806d270 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7650009960 tx=0x7f76500090e0 comp rx=0 tx=0).stop 2024-04-06T13:47:21.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.711+0000 7f765be27700 1 --2- 172.21.15.189:0/3408919770 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7654146dd0 0x7f76540aeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:21.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.711+0000 7f765be27700 1 --2- 172.21.15.189:0/3408919770 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76540cedf0 0x7f76541f2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:21.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.711+0000 7f765be27700 1 --2- 172.21.15.189:0/3408919770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76540d17a0 0x7f76540c8d00 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:21.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.711+0000 7f765be27700 1 -- 172.21.15.189:0/3408919770 >> 172.21.15.189:0/3408919770 conn(0x7f76540b2db0 msgr2=0x7f765406b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:21.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.711+0000 7f765be27700 1 -- 172.21.15.189:0/3408919770 shutdown_connections 2024-04-06T13:47:21.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.712+0000 7f765be27700 1 -- 172.21.15.189:0/3408919770 wait complete. 2024-04-06T13:47:21.714 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' created 2024-04-06T13:47:21.732 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-06T13:47:21.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.860+0000 7f6fbefea700 1 Processor -- start 2024-04-06T13:47:21.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.860+0000 7f6fbefea700 1 -- start start 2024-04-06T13:47:21.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.861+0000 7f6fbefea700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6fb8159b20 0x7f6fb806b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:21.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.861+0000 7f6fbefea700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb806a8d0 con 0x7f6fb806eb80 2024-04-06T13:47:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.861+0000 7f6fbefea700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb806abb0 con 0x7f6fb80cdc60 2024-04-06T13:47:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.861+0000 7f6fbefea700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb8159f00 con 0x7f6fb8159b20 2024-04-06T13:47:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.861+0000 7f6fbcd86700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6fb8159b20 0x7f6fb806b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.861+0000 7f6fbd587700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6fb806eb80 0x7f6fb80ca240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58462/0 (socket says 172.21.15.189:58462) 2024-04-06T13:47:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.861+0000 7f6fbcd86700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6fb8159b20 0x7f6fb806b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54978/0 (socket says 172.21.15.189:54978) 2024-04-06T13:47:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.861+0000 7f6fbd587700 1 -- 172.21.15.189:0/275269138 learned_addr learned my addr 172.21.15.189:0/275269138 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.861+0000 7f6fbcd86700 1 -- 172.21.15.189:0/275269138 >> v1:172.21.15.192:6789/0 conn(0x7f6fb80cdc60 legacy=0x7f6fb806e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.861+0000 7f6fbcd86700 1 -- 172.21.15.189:0/275269138 >> v1:172.21.15.189:6789/0 conn(0x7f6fb806eb80 legacy=0x7f6fb80ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.861+0000 7f6fbcd86700 1 -- 172.21.15.189:0/275269138 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb806ae90 con 0x7f6fb8159b20 2024-04-06T13:47:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.861+0000 7f6fb77fe700 1 -- 172.21.15.189:0/275269138 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405794894 0 0) 0x7f6fb806a8d0 con 0x7f6fb806eb80 2024-04-06T13:47:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.861+0000 7f6fbcd86700 1 --2- 172.21.15.189:0/275269138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb8159b20 0x7f6fb806b970 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f6fac00a9e0 tx=0x7f6fac00acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32cc81d955908cc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.862+0000 7f6fb77fe700 1 -- 172.21.15.189:0/275269138 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac00cb30 con 0x7f6fb8159b20 2024-04-06T13:47:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.862+0000 7f6fb77fe700 1 -- 172.21.15.189:0/275269138 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fac00cc90 con 0x7f6fb8159b20 2024-04-06T13:47:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.862+0000 7f6fb77fe700 1 -- 172.21.15.189:0/275269138 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac004030 con 0x7f6fb8159b20 2024-04-06T13:47:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.862+0000 7f6fbefea700 1 -- 172.21.15.189:0/275269138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb8159b20 msgr2=0x7f6fb806b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:21.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.862+0000 7f6fbefea700 1 --2- 172.21.15.189:0/275269138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb8159b20 0x7f6fb806b970 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f6fac00a9e0 tx=0x7f6fac00acf0 comp rx=0 tx=0).stop 2024-04-06T13:47:21.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.862+0000 7f6fbefea700 1 -- 172.21.15.189:0/275269138 shutdown_connections 2024-04-06T13:47:21.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.862+0000 7f6fbefea700 1 --2- 172.21.15.189:0/275269138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb8159b20 0x7f6fb806b970 unknown :-1 s=CLOSED pgs=2389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:21.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.862+0000 7f6fbefea700 1 -- 172.21.15.189:0/275269138 >> 172.21.15.189:0/275269138 conn(0x7f6fb80cc290 msgr2=0x7f6fb80cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:21.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.862+0000 7f6fbefea700 1 -- 172.21.15.189:0/275269138 shutdown_connections 2024-04-06T13:47:21.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.862+0000 7f6fbefea700 1 -- 172.21.15.189:0/275269138 wait complete. 2024-04-06T13:47:21.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.863+0000 7f6fbefea700 1 Processor -- start 2024-04-06T13:47:21.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.863+0000 7f6fbefea700 1 -- start start 2024-04-06T13:47:21.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.864+0000 7f6fbefea700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb806eb80 0x7f6fb80ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:21.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.864+0000 7f6fbefea700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fb80cdc60 0x7f6fb80b2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:21.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.864+0000 7f6fbefea700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fb8159b20 0x7f6fb80b2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:21.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.864+0000 7f6fbefea700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80ad320 con 0x7f6fb806eb80 2024-04-06T13:47:21.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.864+0000 7f6fbefea700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80ad600 con 0x7f6fb8159b20 2024-04-06T13:47:21.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.864+0000 7f6fbefea700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80ad8e0 con 0x7f6fb80cdc60 2024-04-06T13:47:21.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.864+0000 7f6fbcd86700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb806eb80 0x7f6fb80ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:21.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.864+0000 7f6fbcd86700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb806eb80 0x7f6fb80ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54994/0 (socket says 172.21.15.189:54994) 2024-04-06T13:47:21.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.864+0000 7f6fbcd86700 1 -- 172.21.15.189:0/4050089471 learned_addr learned my addr 172.21.15.189:0/4050089471 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:21.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.865+0000 7f6fbcd86700 1 -- 172.21.15.189:0/4050089471 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fb80cdc60 msgr2=0x7f6fb80b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:21.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.865+0000 7f6fbcd86700 1 --2- 172.21.15.189:0/4050089471 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fb80cdc60 0x7f6fb80b2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:21.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.865+0000 7f6fbd587700 1 --2- 172.21.15.189:0/4050089471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fb8159b20 0x7f6fb80b2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:21.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.865+0000 7f6fbcd86700 1 -- 172.21.15.189:0/4050089471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fb8159b20 msgr2=0x7f6fb80b2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:21.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.865+0000 7f6fbcd86700 1 --2- 172.21.15.189:0/4050089471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fb8159b20 0x7f6fb80b2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:21.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.865+0000 7f6fbcd86700 1 -- 172.21.15.189:0/4050089471 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac00a640 con 0x7f6fb806eb80 2024-04-06T13:47:21.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.865+0000 7f6fbcd86700 1 --2- 172.21.15.189:0/4050089471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb806eb80 0x7f6fb80ac8d0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f6fac006ec0 tx=0x7f6fac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:21.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.865+0000 7f6fb57fa700 1 -- 172.21.15.189:0/4050089471 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac01d070 con 0x7f6fb806eb80 2024-04-06T13:47:21.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.865+0000 7f6fb57fa700 1 -- 172.21.15.189:0/4050089471 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fac003ab0 con 0x7f6fb806eb80 2024-04-06T13:47:21.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.865+0000 7f6fbefea700 1 -- 172.21.15.189:0/4050089471 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb80adbc0 con 0x7f6fb806eb80 2024-04-06T13:47:21.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.865+0000 7f6fb57fa700 1 -- 172.21.15.189:0/4050089471 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac003e40 con 0x7f6fb806eb80 2024-04-06T13:47:21.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.865+0000 7f6fbefea700 1 -- 172.21.15.189:0/4050089471 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb815b740 con 0x7f6fb806eb80 2024-04-06T13:47:21.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.866+0000 7f6fbefea700 1 -- 172.21.15.189:0/4050089471 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb815b8d0 con 0x7f6fb806eb80 2024-04-06T13:47:21.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.867+0000 7f6fb57fa700 1 -- 172.21.15.189:0/4050089471 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f6fac0183f0 con 0x7f6fb806eb80 2024-04-06T13:47:21.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.867+0000 7f6fb57fa700 1 --2- 172.21.15.189:0/4050089471 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f9c06adb0 0x7f6f9c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:21.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.867+0000 7f6fb57fa700 1 -- 172.21.15.189:0/4050089471 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f6fac08b0c0 con 0x7f6fb806eb80 2024-04-06T13:47:21.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.867+0000 7f6fb7fff700 1 --2- 172.21.15.189:0/4050089471 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f9c06adb0 0x7f6f9c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:21.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.868+0000 7f6fb7fff700 1 --2- 172.21.15.189:0/4050089471 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f9c06adb0 0x7f6f9c06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6fa40096f0 tx=0x7f6fa4006c50 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:21.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:21.872+0000 7f6fb57fa700 1 -- 172.21.15.189:0/4050089471 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6fac0560e0 con 0x7f6fb806eb80 2024-04-06T13:47:22.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.114+0000 7f6fbefea700 1 -- 172.21.15.189:0/4050089471 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6fb80ae080 con 0x7f6fb806eb80 2024-04-06T13:47:22.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.121+0000 7f6fb57fa700 1 -- 172.21.15.189:0/4050089471 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 27 and data pool 28 v97) v1 ==== 174+0+0 (secure 0 0 0) 0x7f6fac059700 con 0x7f6fb806eb80 2024-04-06T13:47:22.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.128+0000 7f6fbefea700 1 -- 172.21.15.189:0/4050089471 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f9c06adb0 msgr2=0x7f6f9c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:22.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.128+0000 7f6fbefea700 1 --2- 172.21.15.189:0/4050089471 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f9c06adb0 0x7f6f9c06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6fa40096f0 tx=0x7f6fa4006c50 comp rx=0 tx=0).stop 2024-04-06T13:47:22.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.128+0000 7f6fbefea700 1 -- 172.21.15.189:0/4050089471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb806eb80 msgr2=0x7f6fb80ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:22.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.128+0000 7f6fbefea700 1 --2- 172.21.15.189:0/4050089471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb806eb80 0x7f6fb80ac8d0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f6fac006ec0 tx=0x7f6fac004750 comp rx=0 tx=0).stop 2024-04-06T13:47:22.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.128+0000 7f6fbefea700 1 -- 172.21.15.189:0/4050089471 shutdown_connections 2024-04-06T13:47:22.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.128+0000 7f6fbefea700 1 --2- 172.21.15.189:0/4050089471 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f9c06adb0 0x7f6f9c06d270 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6fa40096f0 tx=0x7f6fa4006c50 comp rx=0 tx=0).stop 2024-04-06T13:47:22.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.128+0000 7f6fbefea700 1 --2- 172.21.15.189:0/4050089471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fb8159b20 0x7f6fb80b2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.128+0000 7f6fbefea700 1 --2- 172.21.15.189:0/4050089471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb806eb80 0x7f6fb80ac8d0 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.128+0000 7f6fbefea700 1 --2- 172.21.15.189:0/4050089471 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fb80cdc60 0x7f6fb80b2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.128+0000 7f6fbefea700 1 -- 172.21.15.189:0/4050089471 >> 172.21.15.189:0/4050089471 conn(0x7f6fb80cc290 msgr2=0x7f6fb80af2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:22.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.130+0000 7f6fbefea700 1 -- 172.21.15.189:0/4050089471 shutdown_connections 2024-04-06T13:47:22.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.130+0000 7f6fbefea700 1 -- 172.21.15.189:0/4050089471 wait complete. 2024-04-06T13:47:22.133 INFO:teuthology.orchestra.run.smithi189.stderr:new fs with metadata pool 27 and data pool 28 2024-04-06T13:47:22.152 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:47:22.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.274+0000 7f1caea3b700 1 Processor -- start 2024-04-06T13:47:22.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.275+0000 7f1caea3b700 1 -- start start 2024-04-06T13:47:22.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.275+0000 7f1caea3b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ca80d11b0 0x7f1ca80d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.275+0000 7f1caea3b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1ca80cb870 0x7f1ca815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.275+0000 7f1caea3b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca80d1ad0 con 0x7f1ca80cfa00 2024-04-06T13:47:22.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.275+0000 7f1caea3b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ca80ccdc0 con 0x7f1ca80cb870 2024-04-06T13:47:22.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.275+0000 7f1caea3b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca80d0b50 con 0x7f1ca80d11b0 2024-04-06T13:47:22.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.275+0000 7f1ca77fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1ca80cb870 0x7f1ca815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:22.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.276+0000 7f1ca77fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1ca80cb870 0x7f1ca815c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55000/0 (socket says 172.21.15.189:55000) 2024-04-06T13:47:22.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.276+0000 7f1ca77fe700 1 -- 172.21.15.189:0/402240313 learned_addr learned my addr 172.21.15.189:0/402240313 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:22.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.276+0000 7f1ca7fff700 1 --2- 172.21.15.189:0/402240313 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ca80d11b0 0x7f1ca80d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:22.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.276+0000 7f1ca77fe700 1 -- 172.21.15.189:0/402240313 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ca80d11b0 msgr2=0x7f1ca80d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:22.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.276+0000 7f1ca77fe700 1 --2- 172.21.15.189:0/402240313 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ca80d11b0 0x7f1ca80d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.276+0000 7f1ca77fe700 1 -- 172.21.15.189:0/402240313 >> v1:172.21.15.192:6789/0 conn(0x7f1ca80cfa00 legacy=0x7f1ca80cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:22.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.276+0000 7f1ca77fe700 1 -- 172.21.15.189:0/402240313 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca80cbd80 con 0x7f1ca80cb870 2024-04-06T13:47:22.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.276+0000 7f1ca77fe700 1 --2- 172.21.15.189:0/402240313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ca80cb870 0x7f1ca815c280 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f1c980098f0 tx=0x7f1c98009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a30f6c4e06fe987 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:22.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.276+0000 7f1ca6ffd700 1 -- 172.21.15.189:0/402240313 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9800b8a0 con 0x7f1ca80cb870 2024-04-06T13:47:22.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.276+0000 7f1ca6ffd700 1 -- 172.21.15.189:0/402240313 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9800ba00 con 0x7f1ca80cb870 2024-04-06T13:47:22.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.276+0000 7f1ca6ffd700 1 -- 172.21.15.189:0/402240313 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9800bd90 con 0x7f1ca80cb870 2024-04-06T13:47:22.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.277+0000 7f1caea3b700 1 -- 172.21.15.189:0/402240313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ca80cb870 msgr2=0x7f1ca815c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:22.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.277+0000 7f1caea3b700 1 --2- 172.21.15.189:0/402240313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ca80cb870 0x7f1ca815c280 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f1c980098f0 tx=0x7f1c98009c00 comp rx=0 tx=0).stop 2024-04-06T13:47:22.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.277+0000 7f1caea3b700 1 -- 172.21.15.189:0/402240313 shutdown_connections 2024-04-06T13:47:22.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.277+0000 7f1caea3b700 1 --2- 172.21.15.189:0/402240313 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ca80d11b0 0x7f1ca80d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.277+0000 7f1caea3b700 1 --2- 172.21.15.189:0/402240313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ca80cb870 0x7f1ca815c280 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.277+0000 7f1caea3b700 1 -- 172.21.15.189:0/402240313 >> 172.21.15.189:0/402240313 conn(0x7f1ca80b2db0 msgr2=0x7f1ca80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:22.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.277+0000 7f1caea3b700 1 -- 172.21.15.189:0/402240313 shutdown_connections 2024-04-06T13:47:22.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.277+0000 7f1caea3b700 1 -- 172.21.15.189:0/402240313 wait complete. 2024-04-06T13:47:22.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.278+0000 7f1caea3b700 1 Processor -- start 2024-04-06T13:47:22.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.278+0000 7f1caea3b700 1 -- start start 2024-04-06T13:47:22.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.279+0000 7f1caea3b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ca80cb870 0x7f1ca8155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.279+0000 7f1caea3b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ca80cfa00 0x7f1ca80c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.279+0000 7f1caea3b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ca80d11b0 0x7f1ca80c5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.279+0000 7f1caea3b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca8155d40 con 0x7f1ca80cb870 2024-04-06T13:47:22.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.279+0000 7f1caea3b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca80c58c0 con 0x7f1ca80cfa00 2024-04-06T13:47:22.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.279+0000 7f1caea3b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca80c8ad0 con 0x7f1ca80d11b0 2024-04-06T13:47:22.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.279+0000 7f1ca7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ca80cb870 0x7f1ca8155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:22.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.279+0000 7f1ca7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ca80cb870 0x7f1ca8155800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55006/0 (socket says 172.21.15.189:55006) 2024-04-06T13:47:22.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.279+0000 7f1ca7fff700 1 -- 172.21.15.189:0/1877061641 learned_addr learned my addr 172.21.15.189:0/1877061641 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:22.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.279+0000 7f1cacfd8700 1 --2- 172.21.15.189:0/1877061641 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ca80d11b0 0x7f1ca80c5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:22.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.279+0000 7f1ca77fe700 1 --2- 172.21.15.189:0/1877061641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ca80cfa00 0x7f1ca80c4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:22.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.279+0000 7f1ca7fff700 1 -- 172.21.15.189:0/1877061641 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ca80d11b0 msgr2=0x7f1ca80c5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:22.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.279+0000 7f1ca7fff700 1 --2- 172.21.15.189:0/1877061641 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ca80d11b0 0x7f1ca80c5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.279+0000 7f1ca7fff700 1 -- 172.21.15.189:0/1877061641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ca80cfa00 msgr2=0x7f1ca80c4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:22.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.279+0000 7f1ca7fff700 1 --2- 172.21.15.189:0/1877061641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ca80cfa00 0x7f1ca80c4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.279+0000 7f1ca7fff700 1 -- 172.21.15.189:0/1877061641 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c98009550 con 0x7f1ca80cb870 2024-04-06T13:47:22.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.279+0000 7f1ca7fff700 1 --2- 172.21.15.189:0/1877061641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ca80cb870 0x7f1ca8155800 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00d800 tx=0x7f1c9c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:22.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.280+0000 7f1ca4ff9700 1 -- 172.21.15.189:0/1877061641 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c0125f0 con 0x7f1ca80cb870 2024-04-06T13:47:22.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.280+0000 7f1ca4ff9700 1 -- 172.21.15.189:0/1877061641 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9c012750 con 0x7f1ca80cb870 2024-04-06T13:47:22.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.280+0000 7f1caea3b700 1 -- 172.21.15.189:0/1877061641 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca80c8db0 con 0x7f1ca80cb870 2024-04-06T13:47:22.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.280+0000 7f1ca4ff9700 1 -- 172.21.15.189:0/1877061641 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c012ae0 con 0x7f1ca80cb870 2024-04-06T13:47:22.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.280+0000 7f1caea3b700 1 -- 172.21.15.189:0/1877061641 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca80c9370 con 0x7f1ca80cb870 2024-04-06T13:47:22.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.281+0000 7f1caea3b700 1 -- 172.21.15.189:0/1877061641 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca8074990 con 0x7f1ca80cb870 2024-04-06T13:47:22.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.281+0000 7f1ca4ff9700 1 -- 172.21.15.189:0/1877061641 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f1c9c012c40 con 0x7f1ca80cb870 2024-04-06T13:47:22.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.281+0000 7f1ca4ff9700 1 --2- 172.21.15.189:0/1877061641 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1c9406adb0 0x7f1c9406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.281+0000 7f1ca4ff9700 1 -- 172.21.15.189:0/1877061641 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1c9c0896e0 con 0x7f1ca80cb870 2024-04-06T13:47:22.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.281+0000 7f1ca77fe700 1 --2- 172.21.15.189:0/1877061641 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1c9406adb0 0x7f1c9406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:22.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.282+0000 7f1ca77fe700 1 --2- 172.21.15.189:0/1877061641 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1c9406adb0 0x7f1c9406d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80cbcb0 tx=0x7f1c98008030 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:22.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.286+0000 7f1ca4ff9700 1 -- 172.21.15.189:0/1877061641 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1c9c054590 con 0x7f1ca80cb870 2024-04-06T13:47:22.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.483+0000 7f1caea3b700 1 -- 172.21.15.189:0/1877061641 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ca80c9560 con 0x7f1ca80cb870 2024-04-06T13:47:22.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.485+0000 7f1ca4ff9700 1 -- 172.21.15.189:0/1877061641 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v226) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f1c9c057bb0 con 0x7f1ca80cb870 2024-04-06T13:47:22.486 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:22.486 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":226,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:47:22.116389+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:47:20.644275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-04-06T13:47:21.075143+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:47:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.487+0000 7f1caea3b700 1 -- 172.21.15.189:0/1877061641 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1c9406adb0 msgr2=0x7f1c9406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.487+0000 7f1caea3b700 1 --2- 172.21.15.189:0/1877061641 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1c9406adb0 0x7f1c9406d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80cbcb0 tx=0x7f1c98008030 comp rx=0 tx=0).stop 2024-04-06T13:47:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.487+0000 7f1caea3b700 1 -- 172.21.15.189:0/1877061641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ca80cb870 msgr2=0x7f1ca8155800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.487+0000 7f1caea3b700 1 --2- 172.21.15.189:0/1877061641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ca80cb870 0x7f1ca8155800 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00d800 tx=0x7f1c9c00db10 comp rx=0 tx=0).stop 2024-04-06T13:47:22.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.487+0000 7f1caea3b700 1 -- 172.21.15.189:0/1877061641 shutdown_connections 2024-04-06T13:47:22.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.487+0000 7f1caea3b700 1 --2- 172.21.15.189:0/1877061641 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1c9406adb0 0x7f1c9406d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.487+0000 7f1caea3b700 1 --2- 172.21.15.189:0/1877061641 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ca80d11b0 0x7f1ca80c5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.487+0000 7f1caea3b700 1 --2- 172.21.15.189:0/1877061641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ca80cb870 0x7f1ca8155800 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.487+0000 7f1caea3b700 1 --2- 172.21.15.189:0/1877061641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ca80cfa00 0x7f1ca80c4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.487+0000 7f1caea3b700 1 -- 172.21.15.189:0/1877061641 >> 172.21.15.189:0/1877061641 conn(0x7f1ca80b2db0 msgr2=0x7f1ca815a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:22.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.487+0000 7f1caea3b700 1 -- 172.21.15.189:0/1877061641 shutdown_connections 2024-04-06T13:47:22.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.487+0000 7f1caea3b700 1 -- 172.21.15.189:0/1877061641 wait complete. 2024-04-06T13:47:22.501 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:47:22.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.622+0000 7f4deb10c700 1 Processor -- start 2024-04-06T13:47:22.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.623+0000 7f4deb10c700 1 -- start start 2024-04-06T13:47:22.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.623+0000 7f4deb10c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4de406ba20 0x7f4de4166270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.623+0000 7f4deb10c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de40669d0 con 0x7f4de40cbc20 2024-04-06T13:47:22.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.623+0000 7f4deb10c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de406ede0 con 0x7f4de415e8a0 2024-04-06T13:47:22.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.623+0000 7f4deb10c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de415fd00 con 0x7f4de406ba20 2024-04-06T13:47:22.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.623+0000 7f4de3fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4de40cbc20 0x7f4de415f5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58470/0 (socket says 172.21.15.189:58470) 2024-04-06T13:47:22.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.623+0000 7f4de3fff700 1 -- 172.21.15.189:0/1495998692 learned_addr learned my addr 172.21.15.189:0/1495998692 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:22.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de96a9700 1 --2- 172.21.15.189:0/1495998692 >> v2:172.21.15.192:3300/0 conn(0x7f4de406ba20 0x7f4de4166270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:22.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de37fe700 1 -- 172.21.15.189:0/1495998692 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147259010 0 0) 0x7f4de40669d0 con 0x7f4de40cbc20 2024-04-06T13:47:22.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de37fe700 1 -- 172.21.15.189:0/1495998692 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd40035b0 con 0x7f4de40cbc20 2024-04-06T13:47:22.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de37fe700 1 -- 172.21.15.189:0/1495998692 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2484607409 0 0) 0x7f4dd40035b0 con 0x7f4de40cbc20 2024-04-06T13:47:22.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de37fe700 1 -- 172.21.15.189:0/1495998692 >> v2:172.21.15.192:3300/0 conn(0x7f4de406ba20 msgr2=0x7f4de4166270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:22.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de37fe700 1 --2- 172.21.15.189:0/1495998692 >> v2:172.21.15.192:3300/0 conn(0x7f4de406ba20 0x7f4de4166270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de37fe700 1 -- 172.21.15.189:0/1495998692 >> v1:172.21.15.192:6789/0 conn(0x7f4de415e8a0 legacy=0x7f4de415eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:22.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de37fe700 1 -- 172.21.15.189:0/1495998692 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de415ec80 con 0x7f4de40cbc20 2024-04-06T13:47:22.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de37fe700 1 -- 172.21.15.189:0/1495998692 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f4dd0001670 con 0x7f4de40cbc20 2024-04-06T13:47:22.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de37fe700 1 -- 172.21.15.189:0/1495998692 >> v1:172.21.15.189:6789/0 conn(0x7f4de40cbc20 legacy=0x7f4de415f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:22.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de37fe700 1 --2- 172.21.15.189:0/1495998692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd4003a00 0x7f4dd4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de37fe700 1 --2- 172.21.15.189:0/1495998692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd4004850 0x7f4dd4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de37fe700 1 --2- 172.21.15.189:0/1495998692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd4005330 0x7f4dd400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de37fe700 1 -- 172.21.15.189:0/1495998692 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de415fd00 con 0x7f4dd4004850 2024-04-06T13:47:22.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de37fe700 1 -- 172.21.15.189:0/1495998692 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd400c200 con 0x7f4dd4005330 2024-04-06T13:47:22.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de37fe700 1 -- 172.21.15.189:0/1495998692 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd400c3c0 con 0x7f4dd4003a00 2024-04-06T13:47:22.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de3fff700 1 --2- 172.21.15.189:0/1495998692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd4004850 0x7f4dd4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:22.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de96a9700 1 --2- 172.21.15.189:0/1495998692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd4005330 0x7f4dd400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:22.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de8ea8700 1 --2- 172.21.15.189:0/1495998692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd4003a00 0x7f4dd4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:22.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de3fff700 1 -- 172.21.15.189:0/1495998692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd4003a00 msgr2=0x7f4dd4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:22.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de3fff700 1 --2- 172.21.15.189:0/1495998692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd4003a00 0x7f4dd4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de3fff700 1 -- 172.21.15.189:0/1495998692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd4005330 msgr2=0x7f4dd400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:22.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.624+0000 7f4de3fff700 1 --2- 172.21.15.189:0/1495998692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd4005330 0x7f4dd400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.625+0000 7f4de96a9700 1 -- 172.21.15.189:0/1495998692 reap_dead start 2024-04-06T13:47:22.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.625+0000 7f4de3fff700 1 -- 172.21.15.189:0/1495998692 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd40035b0 con 0x7f4dd4004850 2024-04-06T13:47:22.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.625+0000 7f4de8ea8700 1 --2- 172.21.15.189:0/1495998692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd4003a00 0x7f4dd4003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:22.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.625+0000 7f4de3fff700 1 --2- 172.21.15.189:0/1495998692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd4004850 0x7f4dd4004cd0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f4dd000c210 tx=0x7f4dd4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=560c0d420a45f9f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:22.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.625+0000 7f4de37fe700 1 -- 172.21.15.189:0/1495998692 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0007c10 con 0x7f4dd4004850 2024-04-06T13:47:22.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.625+0000 7f4de37fe700 1 -- 172.21.15.189:0/1495998692 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd000cd30 con 0x7f4dd4004850 2024-04-06T13:47:22.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.625+0000 7f4de37fe700 1 -- 172.21.15.189:0/1495998692 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd000cea0 con 0x7f4dd4004850 2024-04-06T13:47:22.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.625+0000 7f4deb10c700 1 -- 172.21.15.189:0/1495998692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd4004850 msgr2=0x7f4dd4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:22.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.625+0000 7f4deb10c700 1 --2- 172.21.15.189:0/1495998692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd4004850 0x7f4dd4004cd0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f4dd000c210 tx=0x7f4dd4005880 comp rx=0 tx=0).stop 2024-04-06T13:47:22.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.625+0000 7f4deb10c700 1 -- 172.21.15.189:0/1495998692 shutdown_connections 2024-04-06T13:47:22.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.625+0000 7f4deb10c700 1 --2- 172.21.15.189:0/1495998692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd4004850 0x7f4dd4004cd0 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.625+0000 7f4deb10c700 1 -- 172.21.15.189:0/1495998692 >> 172.21.15.189:0/1495998692 conn(0x7f4de40cc290 msgr2=0x7f4de40cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:22.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.625+0000 7f4deb10c700 1 -- 172.21.15.189:0/1495998692 shutdown_connections 2024-04-06T13:47:22.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.625+0000 7f4deb10c700 1 -- 172.21.15.189:0/1495998692 wait complete. 2024-04-06T13:47:22.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.626+0000 7f4deb10c700 1 Processor -- start 2024-04-06T13:47:22.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.627+0000 7f4deb10c700 1 -- start start 2024-04-06T13:47:22.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.627+0000 7f4deb10c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd4004850 0x7f4de414f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.627+0000 7f4deb10c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4de40c3d10 0x7f4de40c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.627+0000 7f4deb10c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4de40c59d0 0x7f4de41f3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.627+0000 7f4deb10c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de40cd560 con 0x7f4de40c3d10 2024-04-06T13:47:22.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.627+0000 7f4deb10c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de40c5ee0 con 0x7f4dd4004850 2024-04-06T13:47:22.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.627+0000 7f4deb10c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de40cd820 con 0x7f4de40c59d0 2024-04-06T13:47:22.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.628+0000 7f4de3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4de40c3d10 0x7f4de40c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:22.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.628+0000 7f4de3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4de40c3d10 0x7f4de40c5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55022/0 (socket says 172.21.15.189:55022) 2024-04-06T13:47:22.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.628+0000 7f4de3fff700 1 -- 172.21.15.189:0/200862234 learned_addr learned my addr 172.21.15.189:0/200862234 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:22.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.628+0000 7f4de3fff700 1 -- 172.21.15.189:0/200862234 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4de40c59d0 msgr2=0x7f4de41f3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:22.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.628+0000 7f4de3fff700 1 --2- 172.21.15.189:0/200862234 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4de40c59d0 0x7f4de41f3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.628+0000 7f4de3fff700 1 -- 172.21.15.189:0/200862234 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd4004850 msgr2=0x7f4de414f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:22.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.628+0000 7f4de3fff700 1 --2- 172.21.15.189:0/200862234 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd4004850 0x7f4de414f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.628+0000 7f4de3fff700 1 -- 172.21.15.189:0/200862234 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de40cdb00 con 0x7f4de40c3d10 2024-04-06T13:47:22.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.628+0000 7f4de3fff700 1 --2- 172.21.15.189:0/200862234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4de40c3d10 0x7f4de40c5490 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f4dd000c800 tx=0x7f4dd0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:22.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.628+0000 7f4de17fa700 1 -- 172.21.15.189:0/200862234 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd000cbd0 con 0x7f4de40c3d10 2024-04-06T13:47:22.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.628+0000 7f4de17fa700 1 -- 172.21.15.189:0/200862234 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd0004c80 con 0x7f4de40c3d10 2024-04-06T13:47:22.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.628+0000 7f4deb10c700 1 -- 172.21.15.189:0/200862234 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de414fda0 con 0x7f4de40c3d10 2024-04-06T13:47:22.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.628+0000 7f4de17fa700 1 -- 172.21.15.189:0/200862234 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd00070d0 con 0x7f4de40c3d10 2024-04-06T13:47:22.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.628+0000 7f4deb10c700 1 -- 172.21.15.189:0/200862234 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de41f3980 con 0x7f4de40c3d10 2024-04-06T13:47:22.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.629+0000 7f4deb10c700 1 -- 172.21.15.189:0/200862234 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de41f3ba0 con 0x7f4de40c3d10 2024-04-06T13:47:22.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.630+0000 7f4de17fa700 1 -- 172.21.15.189:0/200862234 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f4dd0007230 con 0x7f4de40c3d10 2024-04-06T13:47:22.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.630+0000 7f4de17fa700 1 --2- 172.21.15.189:0/200862234 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4dc806adb0 0x7f4dc806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.630+0000 7f4de17fa700 1 -- 172.21.15.189:0/200862234 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4dd0089e00 con 0x7f4de40c3d10 2024-04-06T13:47:22.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.630+0000 7f4de8ea8700 1 --2- 172.21.15.189:0/200862234 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4dc806adb0 0x7f4dc806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:22.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.630+0000 7f4de8ea8700 1 --2- 172.21.15.189:0/200862234 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4dc806adb0 0x7f4dc806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4de4068450 tx=0x7f4dd8009040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:22.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.635+0000 7f4de17fa700 1 -- 172.21.15.189:0/200862234 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4dd0054dd0 con 0x7f4de40c3d10 2024-04-06T13:47:22.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.832+0000 7f4deb10c700 1 -- 172.21.15.189:0/200862234 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4de4072140 con 0x7f4de40c3d10 2024-04-06T13:47:22.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.834+0000 7f4de17fa700 1 -- 172.21.15.189:0/200862234 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v226) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f4dd0017090 con 0x7f4de40c3d10 2024-04-06T13:47:22.835 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:22.835 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":226,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:47:22.116389+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:47:20.644275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-04-06T13:47:21.075143+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:47:22.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.836+0000 7f4deb10c700 1 -- 172.21.15.189:0/200862234 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4dc806adb0 msgr2=0x7f4dc806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:22.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.836+0000 7f4deb10c700 1 --2- 172.21.15.189:0/200862234 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4dc806adb0 0x7f4dc806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4de4068450 tx=0x7f4dd8009040 comp rx=0 tx=0).stop 2024-04-06T13:47:22.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.836+0000 7f4deb10c700 1 -- 172.21.15.189:0/200862234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4de40c3d10 msgr2=0x7f4de40c5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:22.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.836+0000 7f4deb10c700 1 --2- 172.21.15.189:0/200862234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4de40c3d10 0x7f4de40c5490 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f4dd000c800 tx=0x7f4dd0001a00 comp rx=0 tx=0).stop 2024-04-06T13:47:22.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.836+0000 7f4deb10c700 1 -- 172.21.15.189:0/200862234 shutdown_connections 2024-04-06T13:47:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.836+0000 7f4deb10c700 1 --2- 172.21.15.189:0/200862234 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4dc806adb0 0x7f4dc806d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.836+0000 7f4deb10c700 1 --2- 172.21.15.189:0/200862234 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4de40c59d0 0x7f4de41f3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.836+0000 7f4deb10c700 1 --2- 172.21.15.189:0/200862234 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd4004850 0x7f4de414f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.836+0000 7f4deb10c700 1 --2- 172.21.15.189:0/200862234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4de40c3d10 0x7f4de40c5490 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.836+0000 7f4deb10c700 1 -- 172.21.15.189:0/200862234 >> 172.21.15.189:0/200862234 conn(0x7f4de40cc290 msgr2=0x7f4de406cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.836+0000 7f4deb10c700 1 -- 172.21.15.189:0/200862234 shutdown_connections 2024-04-06T13:47:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.836+0000 7f4deb10c700 1 -- 172.21.15.189:0/200862234 wait complete. 2024-04-06T13:47:22.850 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-06T13:47:22.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.974+0000 7fd78a15f700 1 Processor -- start 2024-04-06T13:47:22.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.974+0000 7fd78a15f700 1 -- start start 2024-04-06T13:47:22.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.975+0000 7fd78a15f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd78415d7b0 0x7fd7840b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.975+0000 7fd78a15f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd78415e500 0x7fd78406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.975+0000 7fd78a15f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7840b3990 con 0x7fd78406a730 2024-04-06T13:47:22.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.975+0000 7fd78a15f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd78415e950 con 0x7fd78415d7b0 2024-04-06T13:47:22.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.975+0000 7fd78a15f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd78406b0f0 con 0x7fd78415e500 2024-04-06T13:47:22.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.975+0000 7fd7837fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd78415d7b0 0x7fd7840b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:22.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.975+0000 7fd782ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd78406a730 0x7fd78415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58480/0 (socket says 172.21.15.189:58480) 2024-04-06T13:47:22.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.975+0000 7fd782ffd700 1 -- 172.21.15.189:0/1896451914 learned_addr learned my addr 172.21.15.189:0/1896451914 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:22.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.975+0000 7fd783fff700 1 --2- 172.21.15.189:0/1896451914 >> v2:172.21.15.192:3300/0 conn(0x7fd78415e500 0x7fd78406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:22.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.975+0000 7fd7837fe700 1 -- 172.21.15.189:0/1896451914 >> v2:172.21.15.192:3300/0 conn(0x7fd78415e500 msgr2=0x7fd78406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:22.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.975+0000 7fd7837fe700 1 --2- 172.21.15.189:0/1896451914 >> v2:172.21.15.192:3300/0 conn(0x7fd78415e500 0x7fd78406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.975+0000 7fd7837fe700 1 -- 172.21.15.189:0/1896451914 >> v1:172.21.15.189:6789/0 conn(0x7fd78406a730 legacy=0x7fd78415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:22.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.975+0000 7fd7827fc700 1 -- 172.21.15.189:0/1896451914 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949612385 0 0) 0x7fd7840b3990 con 0x7fd78406a730 2024-04-06T13:47:22.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.975+0000 7fd7837fe700 1 -- 172.21.15.189:0/1896451914 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7840b3060 con 0x7fd78415d7b0 2024-04-06T13:47:22.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.975+0000 7fd783fff700 1 --2- 172.21.15.189:0/1896451914 >> v2:172.21.15.192:3300/0 conn(0x7fd78415e500 0x7fd78406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:22.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.976+0000 7fd7837fe700 1 --2- 172.21.15.189:0/1896451914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd78415d7b0 0x7fd7840b3e20 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7fd7840673e0 tx=0x7fd76c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fb397cf309a89a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:22.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.976+0000 7fd7827fc700 1 -- 172.21.15.189:0/1896451914 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c011830 con 0x7fd78415d7b0 2024-04-06T13:47:22.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.976+0000 7fd7827fc700 1 -- 172.21.15.189:0/1896451914 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd76c011990 con 0x7fd78415d7b0 2024-04-06T13:47:22.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.976+0000 7fd7827fc700 1 -- 172.21.15.189:0/1896451914 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c011d20 con 0x7fd78415d7b0 2024-04-06T13:47:22.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.976+0000 7fd78a15f700 1 -- 172.21.15.189:0/1896451914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd78415d7b0 msgr2=0x7fd7840b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:22.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.976+0000 7fd78a15f700 1 --2- 172.21.15.189:0/1896451914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd78415d7b0 0x7fd7840b3e20 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7fd7840673e0 tx=0x7fd76c009b20 comp rx=0 tx=0).stop 2024-04-06T13:47:22.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.976+0000 7fd78a15f700 1 -- 172.21.15.189:0/1896451914 shutdown_connections 2024-04-06T13:47:22.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.976+0000 7fd78a15f700 1 --2- 172.21.15.189:0/1896451914 >> v2:172.21.15.192:3300/0 conn(0x7fd78415e500 0x7fd78406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.976+0000 7fd78a15f700 1 --2- 172.21.15.189:0/1896451914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd78415d7b0 0x7fd7840b3e20 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.976+0000 7fd78a15f700 1 -- 172.21.15.189:0/1896451914 >> 172.21.15.189:0/1896451914 conn(0x7fd7840d00b0 msgr2=0x7fd7840d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:22.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.976+0000 7fd78a15f700 1 -- 172.21.15.189:0/1896451914 shutdown_connections 2024-04-06T13:47:22.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.976+0000 7fd78a15f700 1 -- 172.21.15.189:0/1896451914 wait complete. 2024-04-06T13:47:22.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.978+0000 7fd78a15f700 1 Processor -- start 2024-04-06T13:47:22.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.978+0000 7fd78a15f700 1 -- start start 2024-04-06T13:47:22.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.979+0000 7fd78a15f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd78406a730 0x7fd78414d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.979+0000 7fd78a15f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd78415d7b0 0x7fd7840ced90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.979+0000 7fd78a15f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd78415e500 0x7fd78414dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.979+0000 7fd78a15f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78415b570 con 0x7fd78406a730 2024-04-06T13:47:22.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.979+0000 7fd78a15f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7840cd4d0 con 0x7fd78415e500 2024-04-06T13:47:22.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.979+0000 7fd78a15f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78415b7c0 con 0x7fd78415d7b0 2024-04-06T13:47:22.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.979+0000 7fd7837fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd78406a730 0x7fd78414d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:22.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.979+0000 7fd7837fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd78406a730 0x7fd78414d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55042/0 (socket says 172.21.15.189:55042) 2024-04-06T13:47:22.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.979+0000 7fd7837fe700 1 -- 172.21.15.189:0/4282397656 learned_addr learned my addr 172.21.15.189:0/4282397656 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:22.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.979+0000 7fd782ffd700 1 --2- 172.21.15.189:0/4282397656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd78415d7b0 0x7fd7840ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:22.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.979+0000 7fd783fff700 1 --2- 172.21.15.189:0/4282397656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd78415e500 0x7fd78414dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:22.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.979+0000 7fd7837fe700 1 -- 172.21.15.189:0/4282397656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd78415d7b0 msgr2=0x7fd7840ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:22.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.979+0000 7fd7837fe700 1 --2- 172.21.15.189:0/4282397656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd78415d7b0 0x7fd7840ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.979+0000 7fd7837fe700 1 -- 172.21.15.189:0/4282397656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd78415e500 msgr2=0x7fd78414dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:22.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.979+0000 7fd7837fe700 1 --2- 172.21.15.189:0/4282397656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd78415e500 0x7fd78414dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:22.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.979+0000 7fd7837fe700 1 -- 172.21.15.189:0/4282397656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76c0094a0 con 0x7fd78406a730 2024-04-06T13:47:22.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.980+0000 7fd7837fe700 1 --2- 172.21.15.189:0/4282397656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd78406a730 0x7fd78414d530 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7fd76c000c00 tx=0x7fd76c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:22.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.980+0000 7fd77bfff700 1 -- 172.21.15.189:0/4282397656 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c0038a0 con 0x7fd78406a730 2024-04-06T13:47:22.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.980+0000 7fd77bfff700 1 -- 172.21.15.189:0/4282397656 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd76c01f070 con 0x7fd78406a730 2024-04-06T13:47:22.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.980+0000 7fd78a15f700 1 -- 172.21.15.189:0/4282397656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd78415baa0 con 0x7fd78406a730 2024-04-06T13:47:22.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.980+0000 7fd77bfff700 1 -- 172.21.15.189:0/4282397656 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c003c40 con 0x7fd78406a730 2024-04-06T13:47:22.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.980+0000 7fd78a15f700 1 -- 172.21.15.189:0/4282397656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7840cf2d0 con 0x7fd78406a730 2024-04-06T13:47:22.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.981+0000 7fd78a15f700 1 -- 172.21.15.189:0/4282397656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd784067bb0 con 0x7fd78406a730 2024-04-06T13:47:22.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.981+0000 7fd77bfff700 1 -- 172.21.15.189:0/4282397656 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fd76c026480 con 0x7fd78406a730 2024-04-06T13:47:22.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.982+0000 7fd77bfff700 1 --2- 172.21.15.189:0/4282397656 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd76406adb0 0x7fd76406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:22.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.982+0000 7fd782ffd700 1 --2- 172.21.15.189:0/4282397656 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd76406adb0 0x7fd76406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:22.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.982+0000 7fd77bfff700 1 -- 172.21.15.189:0/4282397656 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd76c021070 con 0x7fd78406a730 2024-04-06T13:47:22.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.982+0000 7fd782ffd700 1 --2- 172.21.15.189:0/4282397656 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd76406adb0 0x7fd76406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd7841f5fd0 tx=0x7fd774007be0 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:22.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:22.986+0000 7fd77bfff700 1 -- 172.21.15.189:0/4282397656 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd76c0560f0 con 0x7fd78406a730 2024-04-06T13:47:23.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:23.231+0000 7fd78a15f700 1 -- 172.21.15.189:0/4282397656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd784074990 con 0x7fd78406a730 2024-04-06T13:47:23.649 INFO:tasks.ceph.mds.b.smithi192.stderr:2024-04-06T13:47:23.648+0000 7f030e357700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T13:47:24.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.130+0000 7fd77bfff700 1 -- 172.21.15.189:0/4282397656 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v99) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd76c02b0b0 con 0x7fd78406a730 2024-04-06T13:47:24.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.132+0000 7fd78a15f700 1 -- 172.21.15.189:0/4282397656 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd76406adb0 msgr2=0x7fd76406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:24.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.132+0000 7fd78a15f700 1 --2- 172.21.15.189:0/4282397656 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd76406adb0 0x7fd76406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd7841f5fd0 tx=0x7fd774007be0 comp rx=0 tx=0).stop 2024-04-06T13:47:24.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.132+0000 7fd78a15f700 1 -- 172.21.15.189:0/4282397656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd78406a730 msgr2=0x7fd78414d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:24.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.132+0000 7fd78a15f700 1 --2- 172.21.15.189:0/4282397656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd78406a730 0x7fd78414d530 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7fd76c000c00 tx=0x7fd76c004750 comp rx=0 tx=0).stop 2024-04-06T13:47:24.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.133+0000 7fd78a15f700 1 -- 172.21.15.189:0/4282397656 shutdown_connections 2024-04-06T13:47:24.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.133+0000 7fd78a15f700 1 --2- 172.21.15.189:0/4282397656 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd76406adb0 0x7fd76406d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:24.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.133+0000 7fd78a15f700 1 --2- 172.21.15.189:0/4282397656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd78415e500 0x7fd78414dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:24.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.133+0000 7fd78a15f700 1 --2- 172.21.15.189:0/4282397656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd78406a730 0x7fd78414d530 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:24.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.133+0000 7fd78a15f700 1 --2- 172.21.15.189:0/4282397656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd78415d7b0 0x7fd7840ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:24.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.133+0000 7fd78a15f700 1 -- 172.21.15.189:0/4282397656 >> 172.21.15.189:0/4282397656 conn(0x7fd7840d00b0 msgr2=0x7fd78406fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:24.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.133+0000 7fd78a15f700 1 -- 172.21.15.189:0/4282397656 shutdown_connections 2024-04-06T13:47:24.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.133+0000 7fd78a15f700 1 -- 172.21.15.189:0/4282397656 wait complete. 2024-04-06T13:47:24.149 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-06T13:47:24.149 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-06T13:47:24.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.279+0000 7f686bfed700 1 Processor -- start 2024-04-06T13:47:24.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.279+0000 7f686bfed700 1 -- start start 2024-04-06T13:47:24.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.280+0000 7f686bfed700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f686406b8c0 0x7f6864168420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:24.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.280+0000 7f686bfed700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686406bcd0 con 0x7f68640cd3e0 2024-04-06T13:47:24.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.280+0000 7f686bfed700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686406ceb0 con 0x7f686415d7c0 2024-04-06T13:47:24.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.280+0000 7f686bfed700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68640cdfa0 con 0x7f686406b8c0 2024-04-06T13:47:24.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.280+0000 7f686a58a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f686406b8c0 0x7f6864168420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:24.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.280+0000 7f6869d89700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f68640cd3e0 0x7f68640cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58484/0 (socket says 172.21.15.189:58484) 2024-04-06T13:47:24.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.280+0000 7f6869d89700 1 -- 172.21.15.189:0/1326177297 learned_addr learned my addr 172.21.15.189:0/1326177297 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:24.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.281+0000 7f686a58a700 1 -- 172.21.15.189:0/1326177297 >> v1:172.21.15.192:6789/0 conn(0x7f686415d7c0 legacy=0x7f686406b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:24.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.281+0000 7f686a58a700 1 -- 172.21.15.189:0/1326177297 >> v1:172.21.15.189:6789/0 conn(0x7f68640cd3e0 legacy=0x7f68640cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:24.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.281+0000 7f686a58a700 1 -- 172.21.15.189:0/1326177297 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686406d750 con 0x7f686406b8c0 2024-04-06T13:47:24.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.281+0000 7f6868d87700 1 -- 172.21.15.189:0/1326177297 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 168039012 0 0) 0x7f686406bcd0 con 0x7f68640cd3e0 2024-04-06T13:47:24.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.281+0000 7f686a58a700 1 --2- 172.21.15.189:0/1326177297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f686406b8c0 0x7f6864168420 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f685c009d90 tx=0x7f685c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4f9b81f598425c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:24.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.281+0000 7f6868d87700 1 -- 172.21.15.189:0/1326177297 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c00e070 con 0x7f686406b8c0 2024-04-06T13:47:24.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.281+0000 7f6868d87700 1 -- 172.21.15.189:0/1326177297 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685c00bb90 con 0x7f686406b8c0 2024-04-06T13:47:24.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.281+0000 7f6868d87700 1 -- 172.21.15.189:0/1326177297 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c003a20 con 0x7f686406b8c0 2024-04-06T13:47:24.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.281+0000 7f686bfed700 1 -- 172.21.15.189:0/1326177297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f686406b8c0 msgr2=0x7f6864168420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:24.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.281+0000 7f686bfed700 1 --2- 172.21.15.189:0/1326177297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f686406b8c0 0x7f6864168420 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f685c009d90 tx=0x7f685c00b330 comp rx=0 tx=0).stop 2024-04-06T13:47:24.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.282+0000 7f686bfed700 1 -- 172.21.15.189:0/1326177297 shutdown_connections 2024-04-06T13:47:24.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.282+0000 7f686bfed700 1 --2- 172.21.15.189:0/1326177297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f686406b8c0 0x7f6864168420 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:24.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.282+0000 7f686bfed700 1 -- 172.21.15.189:0/1326177297 >> 172.21.15.189:0/1326177297 conn(0x7f68640cc290 msgr2=0x7f68640cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:24.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.282+0000 7f686bfed700 1 -- 172.21.15.189:0/1326177297 shutdown_connections 2024-04-06T13:47:24.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.282+0000 7f686bfed700 1 -- 172.21.15.189:0/1326177297 wait complete. 2024-04-06T13:47:24.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.283+0000 7f686bfed700 1 Processor -- start 2024-04-06T13:47:24.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.283+0000 7f686bfed700 1 -- start start 2024-04-06T13:47:24.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.283+0000 7f686bfed700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f686406b8c0 0x7f68641f6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:24.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.283+0000 7f686bfed700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68640cd3e0 0x7f6864160470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:24.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.283+0000 7f686bfed700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f686415d7c0 0x7f6864154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:24.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f686bfed700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6864156540 con 0x7f686415d7c0 2024-04-06T13:47:24.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f686bfed700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6864153cc0 con 0x7f68640cd3e0 2024-04-06T13:47:24.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f686bfed700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6864153fa0 con 0x7f686406b8c0 2024-04-06T13:47:24.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f686a58a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f686415d7c0 0x7f6864154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:24.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f686a58a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f686415d7c0 0x7f6864154d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55064/0 (socket says 172.21.15.189:55064) 2024-04-06T13:47:24.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f686a58a700 1 -- 172.21.15.189:0/3217329977 learned_addr learned my addr 172.21.15.189:0/3217329977 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:24.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f6869588700 1 --2- 172.21.15.189:0/3217329977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68640cd3e0 0x7f6864160470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:24.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f6869d89700 1 --2- 172.21.15.189:0/3217329977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f686406b8c0 0x7f68641f6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:24.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f686a58a700 1 -- 172.21.15.189:0/3217329977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f686406b8c0 msgr2=0x7f68641f6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:24.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f686a58a700 1 --2- 172.21.15.189:0/3217329977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f686406b8c0 0x7f68641f6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:24.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f686a58a700 1 -- 172.21.15.189:0/3217329977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68640cd3e0 msgr2=0x7f6864160470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:24.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f686a58a700 1 --2- 172.21.15.189:0/3217329977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68640cd3e0 0x7f6864160470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:24.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f686a58a700 1 -- 172.21.15.189:0/3217329977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f685c009a70 con 0x7f686415d7c0 2024-04-06T13:47:24.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f6869d89700 1 --2- 172.21.15.189:0/3217329977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f686406b8c0 0x7f68641f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:24.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f686a58a700 1 --2- 172.21.15.189:0/3217329977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f686415d7c0 0x7f6864154d60 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f6864073430 tx=0x7f685c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:24.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f685a7fc700 1 -- 172.21.15.189:0/3217329977 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c012630 con 0x7f686415d7c0 2024-04-06T13:47:24.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f685a7fc700 1 -- 172.21.15.189:0/3217329977 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685c012790 con 0x7f686415d7c0 2024-04-06T13:47:24.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f686bfed700 1 -- 172.21.15.189:0/3217329977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6864154280 con 0x7f686415d7c0 2024-04-06T13:47:24.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f685a7fc700 1 -- 172.21.15.189:0/3217329977 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c012b40 con 0x7f686415d7c0 2024-04-06T13:47:24.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.284+0000 7f686bfed700 1 -- 172.21.15.189:0/3217329977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6864154840 con 0x7f686415d7c0 2024-04-06T13:47:24.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.285+0000 7f686bfed700 1 -- 172.21.15.189:0/3217329977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6864074990 con 0x7f686415d7c0 2024-04-06T13:47:24.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.286+0000 7f685a7fc700 1 -- 172.21.15.189:0/3217329977 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f685c012ca0 con 0x7f686415d7c0 2024-04-06T13:47:24.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.286+0000 7f685a7fc700 1 --2- 172.21.15.189:0/3217329977 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f684806adb0 0x7f684806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:24.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.286+0000 7f685a7fc700 1 -- 172.21.15.189:0/3217329977 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f685c08be00 con 0x7f686415d7c0 2024-04-06T13:47:24.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.286+0000 7f6869d89700 1 --2- 172.21.15.189:0/3217329977 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f684806adb0 0x7f684806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:24.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.287+0000 7f6869d89700 1 --2- 172.21.15.189:0/3217329977 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f684806adb0 0x7f684806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6854000f60 tx=0x7f6854009040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:24.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.291+0000 7f685a7fc700 1 -- 172.21.15.189:0/3217329977 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f685c056cc0 con 0x7f686415d7c0 2024-04-06T13:47:24.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:24.530+0000 7f686bfed700 1 -- 172.21.15.189:0/3217329977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6864066980 con 0x7f686415d7c0 2024-04-06T13:47:25.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.135+0000 7f685a7fc700 1 -- 172.21.15.189:0/3217329977 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v100) v1 ==== 122+0+0 (secure 0 0 0) 0x7f685c05a2e0 con 0x7f686415d7c0 2024-04-06T13:47:25.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.138+0000 7f686bfed700 1 -- 172.21.15.189:0/3217329977 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f684806adb0 msgr2=0x7f684806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:25.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.138+0000 7f686bfed700 1 --2- 172.21.15.189:0/3217329977 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f684806adb0 0x7f684806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6854000f60 tx=0x7f6854009040 comp rx=0 tx=0).stop 2024-04-06T13:47:25.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.138+0000 7f686bfed700 1 -- 172.21.15.189:0/3217329977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f686415d7c0 msgr2=0x7f6864154d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:25.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.138+0000 7f686bfed700 1 --2- 172.21.15.189:0/3217329977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f686415d7c0 0x7f6864154d60 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f6864073430 tx=0x7f685c004ad0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.138+0000 7f686bfed700 1 -- 172.21.15.189:0/3217329977 shutdown_connections 2024-04-06T13:47:25.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.138+0000 7f686bfed700 1 --2- 172.21.15.189:0/3217329977 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f684806adb0 0x7f684806d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.138+0000 7f686bfed700 1 --2- 172.21.15.189:0/3217329977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f686415d7c0 0x7f6864154d60 unknown :-1 s=CLOSED pgs=2401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.138+0000 7f686bfed700 1 --2- 172.21.15.189:0/3217329977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f686406b8c0 0x7f68641f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.138+0000 7f686bfed700 1 --2- 172.21.15.189:0/3217329977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68640cd3e0 0x7f6864160470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.138+0000 7f686bfed700 1 -- 172.21.15.189:0/3217329977 >> 172.21.15.189:0/3217329977 conn(0x7f68640cc290 msgr2=0x7f68640ae140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:25.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.138+0000 7f686bfed700 1 -- 172.21.15.189:0/3217329977 shutdown_connections 2024-04-06T13:47:25.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.138+0000 7f686bfed700 1 -- 172.21.15.189:0/3217329977 wait complete. 2024-04-06T13:47:25.154 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-06T13:47:25.154 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:47:25.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.287+0000 7fabb5a75700 1 Processor -- start 2024-04-06T13:47:25.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.287+0000 7fabb5a75700 1 -- start start 2024-04-06T13:47:25.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.287+0000 7fabb5a75700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fabb00d17a0 0x7fabb006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:25.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.287+0000 7fabb5a75700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fabb0146dd0 0x7fabb006d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:25.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.288+0000 7fabb5a75700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb006c0a0 con 0x7fabb00cedf0 2024-04-06T13:47:25.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.288+0000 7fabb5a75700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb006c380 con 0x7fabb0146dd0 2024-04-06T13:47:25.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.288+0000 7fabb5a75700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb006c660 con 0x7fabb00d17a0 2024-04-06T13:47:25.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.288+0000 7fabaffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fabb00cedf0 0x7fabb01441f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58490/0 (socket says 172.21.15.189:58490) 2024-04-06T13:47:25.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.288+0000 7fabaffff700 1 -- 172.21.15.189:0/225254905 learned_addr learned my addr 172.21.15.189:0/225254905 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:25.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.288+0000 7faba7fff700 1 --2- 172.21.15.189:0/225254905 >> v2:172.21.15.192:3300/0 conn(0x7fabb0146dd0 0x7fabb006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:25.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.288+0000 7fabaf7fe700 1 --2- 172.21.15.189:0/225254905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fabb00d17a0 0x7fabb006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:25.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.288+0000 7fabaeffd700 1 -- 172.21.15.189:0/225254905 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593575752 0 0) 0x7fabb006c0a0 con 0x7fabb00cedf0 2024-04-06T13:47:25.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.288+0000 7fabaeffd700 1 -- 172.21.15.189:0/225254905 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab9c0035b0 con 0x7fabb00cedf0 2024-04-06T13:47:25.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.288+0000 7faba7fff700 1 -- 172.21.15.189:0/225254905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fabb00d17a0 msgr2=0x7fabb006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:25.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.288+0000 7faba7fff700 1 --2- 172.21.15.189:0/225254905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fabb00d17a0 0x7fabb006b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.288+0000 7faba7fff700 1 -- 172.21.15.189:0/225254905 >> v1:172.21.15.189:6789/0 conn(0x7fabb00cedf0 legacy=0x7fabb01441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:25.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.288+0000 7faba7fff700 1 -- 172.21.15.189:0/225254905 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb006c940 con 0x7fabb0146dd0 2024-04-06T13:47:25.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.288+0000 7faba7fff700 1 --2- 172.21.15.189:0/225254905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabb0146dd0 0x7fabb006d140 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fab98009840 tx=0x7fab98009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbf9b8319f74a90e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:25.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.289+0000 7fabaeffd700 1 -- 172.21.15.189:0/225254905 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab98011830 con 0x7fabb0146dd0 2024-04-06T13:47:25.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.289+0000 7fabaeffd700 1 -- 172.21.15.189:0/225254905 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab98011990 con 0x7fabb0146dd0 2024-04-06T13:47:25.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.289+0000 7fabaeffd700 1 -- 172.21.15.189:0/225254905 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab98011d20 con 0x7fabb0146dd0 2024-04-06T13:47:25.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.289+0000 7fabb5a75700 1 -- 172.21.15.189:0/225254905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabb0146dd0 msgr2=0x7fabb006d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:25.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.289+0000 7fabb5a75700 1 --2- 172.21.15.189:0/225254905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabb0146dd0 0x7fabb006d140 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fab98009840 tx=0x7fab98009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:25.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.289+0000 7fabb5a75700 1 -- 172.21.15.189:0/225254905 shutdown_connections 2024-04-06T13:47:25.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.289+0000 7fabb5a75700 1 --2- 172.21.15.189:0/225254905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fabb00d17a0 0x7fabb006b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.289+0000 7fabb5a75700 1 --2- 172.21.15.189:0/225254905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabb0146dd0 0x7fabb006d140 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.289+0000 7fabb5a75700 1 -- 172.21.15.189:0/225254905 >> 172.21.15.189:0/225254905 conn(0x7fabb00b2db0 msgr2=0x7fabb00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:25.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.289+0000 7fabb5a75700 1 -- 172.21.15.189:0/225254905 shutdown_connections 2024-04-06T13:47:25.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.289+0000 7fabb5a75700 1 -- 172.21.15.189:0/225254905 wait complete. 2024-04-06T13:47:25.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.290+0000 7fabb5a75700 1 Processor -- start 2024-04-06T13:47:25.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.291+0000 7fabb5a75700 1 -- start start 2024-04-06T13:47:25.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.291+0000 7fabb5a75700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fabb00cedf0 0x7fabb015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:25.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.291+0000 7fabb5a75700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb00d17a0 0x7fabb00b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:25.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.291+0000 7fabb5a75700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabb0146dd0 0x7fabb00afe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:25.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.291+0000 7fabb5a75700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb00af0e0 con 0x7fabb00d17a0 2024-04-06T13:47:25.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.291+0000 7fabb5a75700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb00af360 con 0x7fabb0146dd0 2024-04-06T13:47:25.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.291+0000 7fabb5a75700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb01f7040 con 0x7fabb00cedf0 2024-04-06T13:47:25.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.291+0000 7faba7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb00d17a0 0x7fabb00b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:25.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.291+0000 7faba7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb00d17a0 0x7fabb00b5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55076/0 (socket says 172.21.15.189:55076) 2024-04-06T13:47:25.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.291+0000 7faba7fff700 1 -- 172.21.15.189:0/1100549689 learned_addr learned my addr 172.21.15.189:0/1100549689 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:25.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.292+0000 7fabaf7fe700 1 --2- 172.21.15.189:0/1100549689 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fabb00cedf0 0x7fabb015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:25.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.292+0000 7faba7fff700 1 -- 172.21.15.189:0/1100549689 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fabb00cedf0 msgr2=0x7fabb015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.292+0000 7fabaffff700 1 --2- 172.21.15.189:0/1100549689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabb0146dd0 0x7fabb00afe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.292+0000 7faba7fff700 1 --2- 172.21.15.189:0/1100549689 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fabb00cedf0 0x7fabb015a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.292+0000 7faba7fff700 1 -- 172.21.15.189:0/1100549689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabb0146dd0 msgr2=0x7fabb00afe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.292+0000 7faba7fff700 1 --2- 172.21.15.189:0/1100549689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabb0146dd0 0x7fabb00afe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.292+0000 7faba7fff700 1 -- 172.21.15.189:0/1100549689 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab980094a0 con 0x7fabb00d17a0 2024-04-06T13:47:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.292+0000 7fabaffff700 1 --2- 172.21.15.189:0/1100549689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabb0146dd0 0x7fabb00afe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.292+0000 7faba7fff700 1 --2- 172.21.15.189:0/1100549689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb00d17a0 0x7fabb00b5180 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7fab980090b0 tx=0x7fab98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.292+0000 7fabacff9700 1 -- 172.21.15.189:0/1100549689 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab980036a0 con 0x7fabb00d17a0 2024-04-06T13:47:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.292+0000 7fabacff9700 1 -- 172.21.15.189:0/1100549689 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9801f070 con 0x7fabb00d17a0 2024-04-06T13:47:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.292+0000 7fabb5a75700 1 -- 172.21.15.189:0/1100549689 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb00af620 con 0x7fabb00d17a0 2024-04-06T13:47:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.292+0000 7fabacff9700 1 -- 172.21.15.189:0/1100549689 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab98003a40 con 0x7fabb00d17a0 2024-04-06T13:47:25.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.292+0000 7fabb5a75700 1 -- 172.21.15.189:0/1100549689 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb015aef0 con 0x7fabb00d17a0 2024-04-06T13:47:25.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.293+0000 7fabb5a75700 1 -- 172.21.15.189:0/1100549689 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb00afb40 con 0x7fabb00d17a0 2024-04-06T13:47:25.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.293+0000 7fabacff9700 1 -- 172.21.15.189:0/1100549689 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fab98003ba0 con 0x7fabb00d17a0 2024-04-06T13:47:25.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.294+0000 7fabacff9700 1 --2- 172.21.15.189:0/1100549689 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fab9006adb0 0x7fab9006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:25.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.294+0000 7fabacff9700 1 -- 172.21.15.189:0/1100549689 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fab98021070 con 0x7fabb00d17a0 2024-04-06T13:47:25.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.294+0000 7fabaf7fe700 1 --2- 172.21.15.189:0/1100549689 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fab9006adb0 0x7fab9006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:25.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.294+0000 7fabaf7fe700 1 --2- 172.21.15.189:0/1100549689 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fab9006adb0 0x7fab9006d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faba0009ab0 tx=0x7faba0008040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:25.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.298+0000 7fabacff9700 1 -- 172.21.15.189:0/1100549689 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fabb00afb40 con 0x7fabb00d17a0 2024-04-06T13:47:25.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.539+0000 7fabb5a75700 1 -- 172.21.15.189:0/1100549689 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabb00afb40 con 0x7fabb00d17a0 2024-04-06T13:47:25.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.540+0000 7fabacff9700 1 -- 172.21.15.189:0/1100549689 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+4754 (secure 0 0 0) 0x7fabb00afb40 con 0x7fabb00d17a0 2024-04-06T13:47:25.540 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:25.540 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":100,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8885,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1481817364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1481817364},{"type":"v1","addr":"172.21.15.189:6837","nonce":1481817364}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":11244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/4021875447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":4021875447},{"type":"v1","addr":"172.21.15.189:6836","nonce":4021875447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":11262,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/4066431971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4066431971},{"type":"v1","addr":"172.21.15.192:6834","nonce":4066431971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:47:22.116705+0000","modified":"2024-04-06T13:47:25.129787+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/694564673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":694564673},{"type":"v1","addr":"172.21.15.192:6835","nonce":694564673}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-04-06T13:47:25.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.542+0000 7fabb5a75700 1 -- 172.21.15.189:0/1100549689 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fab9006adb0 msgr2=0x7fab9006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:25.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.542+0000 7fabb5a75700 1 --2- 172.21.15.189:0/1100549689 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fab9006adb0 0x7fab9006d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faba0009ab0 tx=0x7faba0008040 comp rx=0 tx=0).stop 2024-04-06T13:47:25.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.542+0000 7fabb5a75700 1 -- 172.21.15.189:0/1100549689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb00d17a0 msgr2=0x7fabb00b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:25.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.542+0000 7fabb5a75700 1 --2- 172.21.15.189:0/1100549689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb00d17a0 0x7fabb00b5180 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7fab980090b0 tx=0x7fab98004750 comp rx=0 tx=0).stop 2024-04-06T13:47:25.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.542+0000 7fabb5a75700 1 -- 172.21.15.189:0/1100549689 shutdown_connections 2024-04-06T13:47:25.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.542+0000 7fabb5a75700 1 --2- 172.21.15.189:0/1100549689 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fab9006adb0 0x7fab9006d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.542+0000 7fabb5a75700 1 --2- 172.21.15.189:0/1100549689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabb0146dd0 0x7fabb00afe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.542+0000 7fabb5a75700 1 --2- 172.21.15.189:0/1100549689 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fabb00cedf0 0x7fabb015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.542+0000 7fabb5a75700 1 --2- 172.21.15.189:0/1100549689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb00d17a0 0x7fabb00b5180 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.542+0000 7fabb5a75700 1 -- 172.21.15.189:0/1100549689 >> 172.21.15.189:0/1100549689 conn(0x7fabb00b2db0 msgr2=0x7fabb015cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:25.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.542+0000 7fabb5a75700 1 -- 172.21.15.189:0/1100549689 shutdown_connections 2024-04-06T13:47:25.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.542+0000 7fabb5a75700 1 -- 172.21.15.189:0/1100549689 wait complete. 2024-04-06T13:47:25.543 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 100 2024-04-06T13:47:25.554 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:47:25.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.675+0000 7f7870e5a700 1 Processor -- start 2024-04-06T13:47:25.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.675+0000 7f7870e5a700 1 -- start start 2024-04-06T13:47:25.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.676+0000 7f7870e5a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f786c15e8a0 0x7f786c0cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:25.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.676+0000 7f7870e5a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f786c15eee0 0x7f786c15f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.676+0000 7f7870e5a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f786c0669d0 con 0x7f786c15f8a0 2024-04-06T13:47:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.676+0000 7f7870e5a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f786c15fd30 con 0x7f786c15eee0 2024-04-06T13:47:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.676+0000 7f7870e5a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786c06ede0 con 0x7f786c15e8a0 2024-04-06T13:47:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.676+0000 7f786ad9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f786c15f8a0 0x7f786c06ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58496/0 (socket says 172.21.15.189:58496) 2024-04-06T13:47:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786ad9d700 1 -- 172.21.15.189:0/2621146584 learned_addr learned my addr 172.21.15.189:0/2621146584 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f7869d9b700 1 --2- 172.21.15.189:0/2621146584 >> v2:172.21.15.192:3300/0 conn(0x7f786c15eee0 0x7f786c15f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786a59c700 1 --2- 172.21.15.189:0/2621146584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f786c15e8a0 0x7f786c0cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786959a700 1 -- 172.21.15.189:0/2621146584 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000019697 0 0) 0x7f786c0669d0 con 0x7f786c15f8a0 2024-04-06T13:47:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786959a700 1 -- 172.21.15.189:0/2621146584 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78540035b0 con 0x7f786c15f8a0 2024-04-06T13:47:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786959a700 1 -- 172.21.15.189:0/2621146584 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2778820129 0 0) 0x7f78540035b0 con 0x7f786c15f8a0 2024-04-06T13:47:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786959a700 1 -- 172.21.15.189:0/2621146584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f786c15e8a0 msgr2=0x7f786c0cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786959a700 1 --2- 172.21.15.189:0/2621146584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f786c15e8a0 0x7f786c0cbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786959a700 1 -- 172.21.15.189:0/2621146584 >> v2:172.21.15.192:3300/0 conn(0x7f786c15eee0 msgr2=0x7f786c15f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:25.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786959a700 1 --2- 172.21.15.189:0/2621146584 >> v2:172.21.15.192:3300/0 conn(0x7f786c15eee0 0x7f786c15f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786959a700 1 -- 172.21.15.189:0/2621146584 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c15ec80 con 0x7f786c15f8a0 2024-04-06T13:47:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786959a700 1 -- 172.21.15.189:0/2621146584 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f78640019f0 con 0x7f786c15f8a0 2024-04-06T13:47:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f7869d9b700 1 --2- 172.21.15.189:0/2621146584 >> v2:172.21.15.192:3300/0 conn(0x7f786c15eee0 0x7f786c15f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786959a700 1 -- 172.21.15.189:0/2621146584 >> v1:172.21.15.189:6789/0 conn(0x7f786c15f8a0 legacy=0x7f786c06ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786959a700 1 --2- 172.21.15.189:0/2621146584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f786c15e8a0 0x7f7854003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786959a700 1 --2- 172.21.15.189:0/2621146584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7854003f80 0x7f7854004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786959a700 1 --2- 172.21.15.189:0/2621146584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7854004d90 0x7f7854005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786959a700 1 -- 172.21.15.189:0/2621146584 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c15fd30 con 0x7f7854003f80 2024-04-06T13:47:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786959a700 1 -- 172.21.15.189:0/2621146584 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c06ede0 con 0x7f7854004d90 2024-04-06T13:47:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786959a700 1 -- 172.21.15.189:0/2621146584 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785400ba60 con 0x7f786c15e8a0 2024-04-06T13:47:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f7869d9b700 1 --2- 172.21.15.189:0/2621146584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7854003f80 0x7f7854004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786a59c700 1 --2- 172.21.15.189:0/2621146584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f786c15e8a0 0x7f7854003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f786ad9d700 1 --2- 172.21.15.189:0/2621146584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7854004d90 0x7f7854005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:25.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.677+0000 7f7869d9b700 1 -- 172.21.15.189:0/2621146584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f786c15e8a0 msgr2=0x7f7854003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:25.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.678+0000 7f7869d9b700 1 --2- 172.21.15.189:0/2621146584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f786c15e8a0 0x7f7854003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.678+0000 7f7869d9b700 1 -- 172.21.15.189:0/2621146584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7854004d90 msgr2=0x7f7854005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:25.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.678+0000 7f7869d9b700 1 --2- 172.21.15.189:0/2621146584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7854004d90 0x7f7854005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.678+0000 7f7869d9b700 1 -- 172.21.15.189:0/2621146584 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78540035b0 con 0x7f7854003f80 2024-04-06T13:47:25.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.678+0000 7f7869d9b700 1 --2- 172.21.15.189:0/2621146584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7854003f80 0x7f7854004850 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7f7860009d60 tx=0x7f7854005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec51946e0ec6a94 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:25.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.678+0000 7f786959a700 1 -- 172.21.15.189:0/2621146584 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786000e9a0 con 0x7f7854003f80 2024-04-06T13:47:25.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.678+0000 7f786959a700 1 -- 172.21.15.189:0/2621146584 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786000eb00 con 0x7f7854003f80 2024-04-06T13:47:25.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.678+0000 7f786959a700 1 -- 172.21.15.189:0/2621146584 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786000eeb0 con 0x7f7854003f80 2024-04-06T13:47:25.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.678+0000 7f7870e5a700 1 -- 172.21.15.189:0/2621146584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7854003f80 msgr2=0x7f7854004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:25.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.678+0000 7f7870e5a700 1 --2- 172.21.15.189:0/2621146584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7854003f80 0x7f7854004850 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7f7860009d60 tx=0x7f7854005880 comp rx=0 tx=0).stop 2024-04-06T13:47:25.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.678+0000 7f786ad9d700 1 -- 172.21.15.189:0/2621146584 reap_dead start 2024-04-06T13:47:25.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.678+0000 7f7870e5a700 1 -- 172.21.15.189:0/2621146584 shutdown_connections 2024-04-06T13:47:25.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.678+0000 7f7870e5a700 1 -- 172.21.15.189:0/2621146584 >> 172.21.15.189:0/2621146584 conn(0x7f786c0cc290 msgr2=0x7f786c0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:25.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.678+0000 7f7870e5a700 1 -- 172.21.15.189:0/2621146584 shutdown_connections 2024-04-06T13:47:25.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.678+0000 7f7870e5a700 1 -- 172.21.15.189:0/2621146584 wait complete. 2024-04-06T13:47:25.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.680+0000 7f7870e5a700 1 Processor -- start 2024-04-06T13:47:25.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.680+0000 7f7870e5a700 1 -- start start 2024-04-06T13:47:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f7870e5a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7854003f80 0x7f786c0c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:25.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f7870e5a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f786c14eac0 0x7f786c0c32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f7870e5a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f786c0c3820 0x7f786c0acf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f7870e5a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c0c3ca0 con 0x7f786c14eac0 2024-04-06T13:47:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f7870e5a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c0ce9b0 con 0x7f786c0c3820 2024-04-06T13:47:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f7870e5a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786c15bf00 con 0x7f7854003f80 2024-04-06T13:47:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f7869d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f786c14eac0 0x7f786c0c32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f7869d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f786c14eac0 0x7f786c0c32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55104/0 (socket says 172.21.15.189:55104) 2024-04-06T13:47:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f7869d9b700 1 -- 172.21.15.189:0/4278006664 learned_addr learned my addr 172.21.15.189:0/4278006664 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f786ad9d700 1 --2- 172.21.15.189:0/4278006664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f786c0c3820 0x7f786c0acf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f786a59c700 1 --2- 172.21.15.189:0/4278006664 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7854003f80 0x7f786c0c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f7869d9b700 1 -- 172.21.15.189:0/4278006664 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7854003f80 msgr2=0x7f786c0c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f7869d9b700 1 --2- 172.21.15.189:0/4278006664 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7854003f80 0x7f786c0c5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f7869d9b700 1 -- 172.21.15.189:0/4278006664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f786c0c3820 msgr2=0x7f786c0acf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:25.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f7869d9b700 1 --2- 172.21.15.189:0/4278006664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f786c0c3820 0x7f786c0acf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f7869d9b700 1 -- 172.21.15.189:0/4278006664 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c15b980 con 0x7f786c14eac0 2024-04-06T13:47:25.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f786ad9d700 1 --2- 172.21.15.189:0/4278006664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f786c0c3820 0x7f786c0acf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:25.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f7869d9b700 1 --2- 172.21.15.189:0/4278006664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f786c14eac0 0x7f786c0c32e0 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f786000e790 tx=0x7f78600047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:25.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f785affd700 1 -- 172.21.15.189:0/4278006664 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7860021070 con 0x7f786c14eac0 2024-04-06T13:47:25.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f785affd700 1 -- 172.21.15.189:0/4278006664 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786000ee90 con 0x7f786c14eac0 2024-04-06T13:47:25.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f7870e5a700 1 -- 172.21.15.189:0/4278006664 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f786c0aca70 con 0x7f786c14eac0 2024-04-06T13:47:25.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f785affd700 1 -- 172.21.15.189:0/4278006664 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7860006e60 con 0x7f786c14eac0 2024-04-06T13:47:25.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.681+0000 7f7870e5a700 1 -- 172.21.15.189:0/4278006664 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f786c156d50 con 0x7f786c14eac0 2024-04-06T13:47:25.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.682+0000 7f7870e5a700 1 -- 172.21.15.189:0/4278006664 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f786c0acda0 con 0x7f786c14eac0 2024-04-06T13:47:25.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.683+0000 7f785affd700 1 -- 172.21.15.189:0/4278006664 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7860013470 con 0x7f786c14eac0 2024-04-06T13:47:25.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.683+0000 7f785affd700 1 --2- 172.21.15.189:0/4278006664 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f784c06adb0 0x7f784c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:25.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.683+0000 7f785affd700 1 -- 172.21.15.189:0/4278006664 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f786008af00 con 0x7f786c14eac0 2024-04-06T13:47:25.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.683+0000 7f786a59c700 1 --2- 172.21.15.189:0/4278006664 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f784c06adb0 0x7f784c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:25.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.684+0000 7f786a59c700 1 --2- 172.21.15.189:0/4278006664 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f784c06adb0 0x7f784c06d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f785c009ac0 tx=0x7f785c008040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:25.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.688+0000 7f785affd700 1 -- 172.21.15.189:0/4278006664 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7860055ed0 con 0x7f786c14eac0 2024-04-06T13:47:25.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.885+0000 7f7870e5a700 1 -- 172.21.15.189:0/4278006664 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f786c072140 con 0x7f786c14eac0 2024-04-06T13:47:25.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.886+0000 7f785affd700 1 -- 172.21.15.189:0/4278006664 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v227) v1 ==== 74+0+16480 (secure 0 0 0) 0x7f786002b090 con 0x7f786c14eac0 2024-04-06T13:47:25.886 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:25.886 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":227,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:47:23.118493+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:47:20.644275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-04-06T13:47:21.075143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:47:25.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.888+0000 7f7870e5a700 1 -- 172.21.15.189:0/4278006664 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f784c06adb0 msgr2=0x7f784c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:25.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.888+0000 7f7870e5a700 1 --2- 172.21.15.189:0/4278006664 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f784c06adb0 0x7f784c06d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f785c009ac0 tx=0x7f785c008040 comp rx=0 tx=0).stop 2024-04-06T13:47:25.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.888+0000 7f7870e5a700 1 -- 172.21.15.189:0/4278006664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f786c14eac0 msgr2=0x7f786c0c32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:25.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.888+0000 7f7870e5a700 1 --2- 172.21.15.189:0/4278006664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f786c14eac0 0x7f786c0c32e0 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f786000e790 tx=0x7f78600047c0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.888+0000 7f7870e5a700 1 -- 172.21.15.189:0/4278006664 shutdown_connections 2024-04-06T13:47:25.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.888+0000 7f7870e5a700 1 --2- 172.21.15.189:0/4278006664 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f784c06adb0 0x7f784c06d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.888+0000 7f7870e5a700 1 --2- 172.21.15.189:0/4278006664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f786c0c3820 0x7f786c0acf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.888+0000 7f7870e5a700 1 --2- 172.21.15.189:0/4278006664 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7854003f80 0x7f786c0c5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.888+0000 7f7870e5a700 1 --2- 172.21.15.189:0/4278006664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f786c14eac0 0x7f786c0c32e0 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:25.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.888+0000 7f7870e5a700 1 -- 172.21.15.189:0/4278006664 >> 172.21.15.189:0/4278006664 conn(0x7f786c0cc290 msgr2=0x7f786c06a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:25.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.888+0000 7f7870e5a700 1 -- 172.21.15.189:0/4278006664 shutdown_connections 2024-04-06T13:47:25.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:25.888+0000 7f7870e5a700 1 -- 172.21.15.189:0/4278006664 wait complete. 2024-04-06T13:47:25.901 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-06T13:47:25.901 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:47:26.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.024+0000 7fbee000a700 1 Processor -- start 2024-04-06T13:47:26.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.024+0000 7fbee000a700 1 -- start start 2024-04-06T13:47:26.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.024+0000 7fbee000a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbed815d590 0x7fbed80b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:26.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.025+0000 7fbee000a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbed80d3890 0x7fbed80b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:26.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.025+0000 7fbee000a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed80b5cc0 con 0x7fbed80b4ed0 2024-04-06T13:47:26.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.025+0000 7fbee000a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbed80b58c0 con 0x7fbed815d590 2024-04-06T13:47:26.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.025+0000 7fbee000a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed80d1500 con 0x7fbed80d3890 2024-04-06T13:47:26.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.025+0000 7fbeddda6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbed815d590 0x7fbed80b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:26.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.025+0000 7fbedd5a5700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbed80b4ed0 0x7fbed80b6580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58506/0 (socket says 172.21.15.189:58506) 2024-04-06T13:47:26.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.025+0000 7fbedd5a5700 1 -- 172.21.15.189:0/2061406522 learned_addr learned my addr 172.21.15.189:0/2061406522 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:26.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.025+0000 7fbede5a7700 1 --2- 172.21.15.189:0/2061406522 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbed80d3890 0x7fbed80b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:26.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.025+0000 7fbeddda6700 1 -- 172.21.15.189:0/2061406522 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbed80d3890 msgr2=0x7fbed80b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:26.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.025+0000 7fbeddda6700 1 --2- 172.21.15.189:0/2061406522 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbed80d3890 0x7fbed80b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.025+0000 7fbeddda6700 1 -- 172.21.15.189:0/2061406522 >> v1:172.21.15.189:6789/0 conn(0x7fbed80b4ed0 legacy=0x7fbed80b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:26.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.025+0000 7fbeddda6700 1 -- 172.21.15.189:0/2061406522 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed80d17e0 con 0x7fbed815d590 2024-04-06T13:47:26.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.025+0000 7fbedcda4700 1 -- 172.21.15.189:0/2061406522 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591840453 0 0) 0x7fbed80b5cc0 con 0x7fbed80b4ed0 2024-04-06T13:47:26.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.025+0000 7fbede5a7700 1 --2- 172.21.15.189:0/2061406522 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbed80d3890 0x7fbed80b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:26.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.025+0000 7fbeddda6700 1 --2- 172.21.15.189:0/2061406522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed815d590 0x7fbed80b8d70 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7fbec8009d90 tx=0x7fbec800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07e1b91cc93148e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:26.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.025+0000 7fbedcda4700 1 -- 172.21.15.189:0/2061406522 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec800e070 con 0x7fbed815d590 2024-04-06T13:47:26.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.025+0000 7fbedcda4700 1 -- 172.21.15.189:0/2061406522 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec800bbd0 con 0x7fbed815d590 2024-04-06T13:47:26.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.025+0000 7fbedcda4700 1 -- 172.21.15.189:0/2061406522 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec8012430 con 0x7fbed815d590 2024-04-06T13:47:26.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.026+0000 7fbee000a700 1 -- 172.21.15.189:0/2061406522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed815d590 msgr2=0x7fbed80b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:26.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.026+0000 7fbee000a700 1 --2- 172.21.15.189:0/2061406522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed815d590 0x7fbed80b8d70 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7fbec8009d90 tx=0x7fbec800b330 comp rx=0 tx=0).stop 2024-04-06T13:47:26.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.026+0000 7fbee000a700 1 -- 172.21.15.189:0/2061406522 shutdown_connections 2024-04-06T13:47:26.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.026+0000 7fbee000a700 1 --2- 172.21.15.189:0/2061406522 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbed80d3890 0x7fbed80b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.026+0000 7fbee000a700 1 --2- 172.21.15.189:0/2061406522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed815d590 0x7fbed80b8d70 unknown :-1 s=CLOSED pgs=2408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.026+0000 7fbee000a700 1 -- 172.21.15.189:0/2061406522 >> 172.21.15.189:0/2061406522 conn(0x7fbed806bbc0 msgr2=0x7fbed806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:26.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.026+0000 7fbee000a700 1 -- 172.21.15.189:0/2061406522 shutdown_connections 2024-04-06T13:47:26.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.026+0000 7fbee000a700 1 -- 172.21.15.189:0/2061406522 wait complete. 2024-04-06T13:47:26.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.027+0000 7fbee000a700 1 Processor -- start 2024-04-06T13:47:26.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.028+0000 7fbee000a700 1 -- start start 2024-04-06T13:47:26.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.028+0000 7fbee000a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbed80b4ed0 0x7fbed8146890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:26.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.028+0000 7fbee000a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbed80d3890 0x7fbed815d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:26.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.028+0000 7fbee000a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed815d590 0x7fbed80cac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:26.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.028+0000 7fbee000a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed815f400 con 0x7fbed815d590 2024-04-06T13:47:26.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.028+0000 7fbee000a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed815deb0 con 0x7fbed80b4ed0 2024-04-06T13:47:26.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.028+0000 7fbee000a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed80cbab0 con 0x7fbed80d3890 2024-04-06T13:47:26.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.029+0000 7fbede5a7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed815d590 0x7fbed80cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:26.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.029+0000 7fbede5a7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed815d590 0x7fbed80cac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55120/0 (socket says 172.21.15.189:55120) 2024-04-06T13:47:26.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.029+0000 7fbede5a7700 1 -- 172.21.15.189:0/3396665567 learned_addr learned my addr 172.21.15.189:0/3396665567 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:26.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.029+0000 7fbeddda6700 1 --2- 172.21.15.189:0/3396665567 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbed80b4ed0 0x7fbed8146890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:26.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.029+0000 7fbedd5a5700 1 --2- 172.21.15.189:0/3396665567 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbed80d3890 0x7fbed815d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:26.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.029+0000 7fbede5a7700 1 -- 172.21.15.189:0/3396665567 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbed80d3890 msgr2=0x7fbed815d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:26.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.029+0000 7fbede5a7700 1 --2- 172.21.15.189:0/3396665567 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbed80d3890 0x7fbed815d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.029+0000 7fbede5a7700 1 -- 172.21.15.189:0/3396665567 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbed80b4ed0 msgr2=0x7fbed8146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:26.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.029+0000 7fbede5a7700 1 --2- 172.21.15.189:0/3396665567 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbed80b4ed0 0x7fbed8146890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.029+0000 7fbede5a7700 1 -- 172.21.15.189:0/3396665567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec8009a70 con 0x7fbed815d590 2024-04-06T13:47:26.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.029+0000 7fbedd5a5700 1 --2- 172.21.15.189:0/3396665567 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbed80d3890 0x7fbed815d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:26.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.029+0000 7fbede5a7700 1 --2- 172.21.15.189:0/3396665567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed815d590 0x7fbed80cac90 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7fbed000c7f0 tx=0x7fbed000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:26.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.029+0000 7fbed67fc700 1 -- 172.21.15.189:0/3396665567 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed0011800 con 0x7fbed815d590 2024-04-06T13:47:26.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.029+0000 7fbed67fc700 1 -- 172.21.15.189:0/3396665567 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed0011960 con 0x7fbed815d590 2024-04-06T13:47:26.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.029+0000 7fbee000a700 1 -- 172.21.15.189:0/3396665567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbed80ca520 con 0x7fbed815d590 2024-04-06T13:47:26.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.029+0000 7fbed67fc700 1 -- 172.21.15.189:0/3396665567 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed0011d10 con 0x7fbed815d590 2024-04-06T13:47:26.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.029+0000 7fbee000a700 1 -- 172.21.15.189:0/3396665567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed80ca800 con 0x7fbed815d590 2024-04-06T13:47:26.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.030+0000 7fbee000a700 1 -- 172.21.15.189:0/3396665567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed80ca990 con 0x7fbed815d590 2024-04-06T13:47:26.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.031+0000 7fbed67fc700 1 -- 172.21.15.189:0/3396665567 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fbed0022350 con 0x7fbed815d590 2024-04-06T13:47:26.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.031+0000 7fbed67fc700 1 --2- 172.21.15.189:0/3396665567 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbec406adb0 0x7fbec406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:26.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.031+0000 7fbed67fc700 1 -- 172.21.15.189:0/3396665567 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbed0089790 con 0x7fbed815d590 2024-04-06T13:47:26.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.031+0000 7fbeddda6700 1 --2- 172.21.15.189:0/3396665567 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbec406adb0 0x7fbec406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:26.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.032+0000 7fbeddda6700 1 --2- 172.21.15.189:0/3396665567 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbec406adb0 0x7fbec406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbec8014040 tx=0x7fbec8011000 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:26.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.035+0000 7fbed67fc700 1 -- 172.21.15.189:0/3396665567 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbed00545d0 con 0x7fbed815d590 2024-04-06T13:47:26.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.229+0000 7fbee000a700 1 -- 172.21.15.189:0/3396665567 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbed80d2080 con 0x7fbec406adb0 2024-04-06T13:47:26.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.237+0000 7fbed67fc700 1 -- 172.21.15.189:0/3396665567 <== mgr.11028 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+279074 (secure 0 0 0) 0x7fbec8003d80 con 0x7fbec406adb0 2024-04-06T13:47:26.243 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:26.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.240+0000 7fbee000a700 1 -- 172.21.15.189:0/3396665567 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbec406adb0 msgr2=0x7fbec406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:26.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.240+0000 7fbee000a700 1 --2- 172.21.15.189:0/3396665567 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbec406adb0 0x7fbec406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbec8014040 tx=0x7fbec8011000 comp rx=0 tx=0).stop 2024-04-06T13:47:26.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.240+0000 7fbee000a700 1 -- 172.21.15.189:0/3396665567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed815d590 msgr2=0x7fbed80cac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:26.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.240+0000 7fbee000a700 1 --2- 172.21.15.189:0/3396665567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed815d590 0x7fbed80cac90 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7fbed000c7f0 tx=0x7fbed000cbb0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.240+0000 7fbee000a700 1 -- 172.21.15.189:0/3396665567 shutdown_connections 2024-04-06T13:47:26.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.240+0000 7fbee000a700 1 --2- 172.21.15.189:0/3396665567 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbec406adb0 0x7fbec406d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.240+0000 7fbee000a700 1 --2- 172.21.15.189:0/3396665567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed815d590 0x7fbed80cac90 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.240+0000 7fbee000a700 1 --2- 172.21.15.189:0/3396665567 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbed80b4ed0 0x7fbed8146890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.240+0000 7fbee000a700 1 --2- 172.21.15.189:0/3396665567 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbed80d3890 0x7fbed815d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.240+0000 7fbee000a700 1 -- 172.21.15.189:0/3396665567 >> 172.21.15.189:0/3396665567 conn(0x7fbed806bbc0 msgr2=0x7fbed80b2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:26.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.240+0000 7fbee000a700 1 -- 172.21.15.189:0/3396665567 shutdown_connections 2024-04-06T13:47:26.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.240+0000 7fbee000a700 1 -- 172.21.15.189:0/3396665567 wait complete. 2024-04-06T13:47:26.247 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:47:26.253 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-04-06T13:47:24.409210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":413100,"kb_used_data":23100,"kb_used_omap":12,"kb_used_meta":389875,"kb_avail":754561620,"statfs":{"total":773094113280,"available":772671098880,"internally_reserved":0,"allocated":23654400,"data_stored":12638241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":399232568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000700"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.701903+0000","last_change":"2024-04-06T13:47:20.701903+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.701903+0000","last_undegraded":"2024-04-06T13:47:20.701903+0000","last_fullsized":"2024-04-06T13:47:20.701903+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:55.001531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.706285+0000","last_change":"2024-04-06T13:47:20.706285+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.706285+0000","last_undegraded":"2024-04-06T13:47:20.706285+0000","last_fullsized":"2024-04-06T13:47:20.706285+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:30:58.296350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.706684+0000","last_change":"2024-04-06T13:47:20.706684+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.706684+0000","last_undegraded":"2024-04-06T13:47:20.706684+0000","last_fullsized":"2024-04-06T13:47:20.706684+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:09.423492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.708043+0000","last_change":"2024-04-06T13:47:20.708043+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.708043+0000","last_undegraded":"2024-04-06T13:47:20.708043+0000","last_fullsized":"2024-04-06T13:47:20.708043+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:43:51.273765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.703113+0000","last_change":"2024-04-06T13:47:20.703113+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.703113+0000","last_undegraded":"2024-04-06T13:47:20.703113+0000","last_fullsized":"2024-04-06T13:47:20.703113+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:25.144968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.703093+0000","last_change":"2024-04-06T13:47:20.703093+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.703093+0000","last_undegraded":"2024-04-06T13:47:20.703093+0000","last_fullsized":"2024-04-06T13:47:20.703093+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:29:38.052685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.706820+0000","last_change":"2024-04-06T13:47:20.706820+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.706820+0000","last_undegraded":"2024-04-06T13:47:20.706820+0000","last_fullsized":"2024-04-06T13:47:20.706820+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:02:18.580630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.708971+0000","last_change":"2024-04-06T13:47:20.708971+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.708971+0000","last_undegraded":"2024-04-06T13:47:20.708971+0000","last_fullsized":"2024-04-06T13:47:20.708971+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:30:35.611598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.703253+0000","last_change":"2024-04-06T13:47:20.703253+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.703253+0000","last_undegraded":"2024-04-06T13:47:20.703253+0000","last_fullsized":"2024-04-06T13:47:20.703253+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:14:53.056289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.709362+0000","last_change":"2024-04-06T13:47:20.709362+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.709362+0000","last_undegraded":"2024-04-06T13:47:20.709362+0000","last_fullsized":"2024-04-06T13:47:20.709362+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:33:33.280270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.703476+0000","last_change":"2024-04-06T13:47:20.703476+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.703476+0000","last_undegraded":"2024-04-06T13:47:20.703476+0000","last_fullsized":"2024-04-06T13:47:20.703476+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:43:36.814977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.706204+0000","last_change":"2024-04-06T13:47:20.706204+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.706204+0000","last_undegraded":"2024-04-06T13:47:20.706204+0000","last_fullsized":"2024-04-06T13:47:20.706204+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:00:20.959541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.706695+0000","last_change":"2024-04-06T13:47:20.706695+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.706695+0000","last_undegraded":"2024-04-06T13:47:20.706695+0000","last_fullsized":"2024-04-06T13:47:20.706695+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:15:13.775616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.708278+0000","last_change":"2024-04-06T13:47:20.708278+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.708278+0000","last_undegraded":"2024-04-06T13:47:20.708278+0000","last_fullsized":"2024-04-06T13:47:20.708278+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:27:30.927130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.709464+0000","last_change":"2024-04-06T13:47:20.709464+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.709464+0000","last_undegraded":"2024-04-06T13:47:20.709464+0000","last_fullsized":"2024-04-06T13:47:20.709464+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:06:40.698606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.705613+0000","last_change":"2024-04-06T13:47:20.705613+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.705613+0000","last_undegraded":"2024-04-06T13:47:20.705613+0000","last_fullsized":"2024-04-06T13:47:20.705613+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:54:05.540582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.705682+0000","last_change":"2024-04-06T13:47:20.705682+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.705682+0000","last_undegraded":"2024-04-06T13:47:20.705682+0000","last_fullsized":"2024-04-06T13:47:20.705682+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:11:25.850855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.706244+0000","last_change":"2024-04-06T13:47:20.706244+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.706244+0000","last_undegraded":"2024-04-06T13:47:20.706244+0000","last_fullsized":"2024-04-06T13:47:20.706244+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:52:28.469203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.708659+0000","last_change":"2024-04-06T13:47:20.708659+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.708659+0000","last_undegraded":"2024-04-06T13:47:20.708659+0000","last_fullsized":"2024-04-06T13:47:20.708659+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:49:15.850828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.709818+0000","last_change":"2024-04-06T13:47:20.709818+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.709818+0000","last_undegraded":"2024-04-06T13:47:20.709818+0000","last_fullsized":"2024-04-06T13:47:20.709818+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:09.996520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.707218+0000","last_change":"2024-04-06T13:47:20.707218+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.707218+0000","last_undegraded":"2024-04-06T13:47:20.707218+0000","last_fullsized":"2024-04-06T13:47:20.707218+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:10:24.669252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.701897+0000","last_change":"2024-04-06T13:47:20.701897+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.701897+0000","last_undegraded":"2024-04-06T13:47:20.701897+0000","last_fullsized":"2024-04-06T13:47:20.701897+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:11:25.850855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.705597+0000","last_change":"2024-04-06T13:47:20.705597+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.705597+0000","last_undegraded":"2024-04-06T13:47:20.705597+0000","last_fullsized":"2024-04-06T13:47:20.705597+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:34:40.754044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.708139+0000","last_change":"2024-04-06T13:47:20.708139+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.708139+0000","last_undegraded":"2024-04-06T13:47:20.708139+0000","last_fullsized":"2024-04-06T13:47:20.708139+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:45:09.803960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.704702+0000","last_change":"2024-04-06T13:47:20.704702+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.704702+0000","last_undegraded":"2024-04-06T13:47:20.704702+0000","last_fullsized":"2024-04-06T13:47:20.704702+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:13:14.840162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.701897+0000","last_change":"2024-04-06T13:47:20.701897+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.701897+0000","last_undegraded":"2024-04-06T13:47:20.701897+0000","last_fullsized":"2024-04-06T13:47:20.701897+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:34:40.754044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.705617+0000","last_change":"2024-04-06T13:47:20.705617+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.705617+0000","last_undegraded":"2024-04-06T13:47:20.705617+0000","last_fullsized":"2024-04-06T13:47:20.705617+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:54:07.189978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.704723+0000","last_change":"2024-04-06T13:47:20.704723+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.704723+0000","last_undegraded":"2024-04-06T13:47:20.704723+0000","last_fullsized":"2024-04-06T13:47:20.704723+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:53:48.795224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.708112+0000","last_change":"2024-04-06T13:47:20.708112+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.708112+0000","last_undegraded":"2024-04-06T13:47:20.708112+0000","last_fullsized":"2024-04-06T13:47:20.708112+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:38:23.226877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.705720+0000","last_change":"2024-04-06T13:47:20.705720+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.705720+0000","last_undegraded":"2024-04-06T13:47:20.705720+0000","last_fullsized":"2024-04-06T13:47:20.705720+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:55.001531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.708370+0000","last_change":"2024-04-06T13:47:20.708370+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.708370+0000","last_undegraded":"2024-04-06T13:47:20.708370+0000","last_fullsized":"2024-04-06T13:47:20.708370+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:56:43.873348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:47:21.696243+0000","last_change":"2024-04-06T13:47:21.696243+0000","last_active":"2024-04-06T13:47:21.696243+0000","last_peered":"2024-04-06T13:47:21.696243+0000","last_clean":"2024-04-06T13:47:21.696243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:21.696243+0000","last_undegraded":"2024-04-06T13:47:21.696243+0000","last_fullsized":"2024-04-06T13:47:21.696243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"creating+peering","last_fresh":"2024-04-06T13:47:20.709575+0000","last_change":"2024-04-06T13:47:20.709575+0000","last_active":"2024-04-06T13:47:20.694450+0000","last_peered":"2024-04-06T13:47:20.694450+0000","last_clean":"2024-04-06T13:47:20.694450+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:47:20.709575+0000","last_undegraded":"2024-04-06T13:47:20.709575+0000","last_fullsized":"2024-04-06T13:47:20.709575+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:55:29.107060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"216'348","reported_seq":1234,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-06T13:47:18.788725+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:47:18.788725+0000","last_peered":"2024-04-06T13:47:18.788725+0000","last_clean":"2024-04-06T13:47:18.788725+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:47:18.788725+0000","last_undegraded":"2024-04-06T13:47:18.788725+0000","last_fullsized":"2024-04-06T13:47:18.788725+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575982,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56588,"kb_used_data":3328,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315252,"statfs":{"total":96636764160,"available":96578818048,"internally_reserved":0,"allocated":3407872,"data_stored":2024729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.751,"5min":0.78400000000000003,"15min":0.84899999999999998},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58499999999999996,"15min":0.59799999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76000000000000001,"5min":0.80800000000000005,"15min":0.82899999999999996},"last":0.67000000000000004}]},{"osd":1,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.57099999999999995,"15min":0.58799999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.66600000000000004,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59799999999999998,"15min":0.60499999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38900000000000001},"max":{"1min":0.752,"5min":0.83399999999999996,"15min":0.85299999999999998},"last":0.56699999999999995}]},{"osd":2,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.57899999999999996,"15min":0.59499999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67700000000000005,"5min":0.81699999999999995,"15min":0.83899999999999997},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.60299999999999998,"15min":0.60599999999999998},"min":{"1min":0.47199999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73699999999999999,"5min":0.79700000000000004,"15min":0.81699999999999995},"last":0.69999999999999996}]},{"osd":3,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.61599999999999999,"15min":0.61499999999999999},"min":{"1min":0.437,"5min":0.41199999999999998,"15min":0.38500000000000001},"max":{"1min":0.75600000000000001,"5min":0.81100000000000005,"15min":0.86199999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59799999999999998,"15min":0.60899999999999999},"min":{"1min":0.42399999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.70199999999999996,"5min":0.83299999999999996,"15min":0.84099999999999997},"last":0.64100000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.45900000000000002,"15min":0.46400000000000002},"min":{"1min":0.33300000000000002,"5min":0.307,"15min":0.30299999999999999},"max":{"1min":0.628,"5min":0.76000000000000001,"15min":0.81599999999999995},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.54600000000000004,"15min":0.54300000000000004},"min":{"1min":0.39300000000000002,"5min":0.38700000000000001,"15min":0.36799999999999999},"max":{"1min":0.68700000000000006,"5min":0.74299999999999999,"15min":0.82699999999999996},"last":0.65900000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.49299999999999999,"15min":0.495},"min":{"1min":0.36099999999999999,"5min":0.318,"15min":0.308},"max":{"1min":0.71999999999999997,"5min":0.77100000000000002,"15min":0.78300000000000003},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.46899999999999997,"5min":0.39200000000000002,"15min":0.33600000000000002},"max":{"1min":0.69799999999999995,"5min":0.82099999999999995,"15min":0.85399999999999998},"last":0.63100000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.39600000000000002,"5min":0.374,"15min":0.34899999999999998},"max":{"1min":0.67700000000000005,"5min":0.755,"15min":0.879},"last":0.503},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56899999999999995,"15min":0.58699999999999997},"min":{"1min":0.39200000000000002,"5min":0.378,"15min":0.34799999999999998},"max":{"1min":0.73299999999999998,"5min":0.84899999999999998,"15min":0.89800000000000002},"last":0.60999999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575981,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37956,"kb_used_data":2744,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94333884,"statfs":{"total":96636764160,"available":96597897216,"internally_reserved":0,"allocated":2809856,"data_stored":1433342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.45300000000000001,"5min":0.38200000000000001,"15min":0.32000000000000001},"max":{"1min":0.74399999999999999,"5min":0.79700000000000004,"15min":0.83799999999999997},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.48199999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76900000000000002,"5min":0.80000000000000004,"15min":0.81999999999999995},"last":0.495}]},{"osd":1,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.626,"15min":0.63},"min":{"1min":0.48999999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.83699999999999997,"15min":0.85799999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64200000000000002,"15min":0.65000000000000002},"min":{"1min":0.54600000000000004,"5min":0.45100000000000001,"15min":0.378},"max":{"1min":0.78900000000000003,"5min":0.81399999999999995,"15min":0.82699999999999996},"last":0.58299999999999996}]},{"osd":2,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.495,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.76700000000000002,"5min":0.82299999999999995,"15min":0.82899999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.59299999999999997},"min":{"1min":0.46400000000000002,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.755,"5min":0.78600000000000003,"15min":0.80700000000000005},"last":0.51800000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77500000000000002,"5min":0.80100000000000005,"15min":0.82399999999999995},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64600000000000002,"15min":0.65000000000000002},"min":{"1min":0.51500000000000001,"5min":0.45900000000000002,"15min":0.36099999999999999},"max":{"1min":0.78800000000000003,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.51200000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54500000000000004,"15min":0.53100000000000003},"min":{"1min":0.38100000000000001,"5min":0.36899999999999999,"15min":0.33800000000000002},"max":{"1min":0.72199999999999998,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53500000000000003,"15min":0.54500000000000004},"min":{"1min":0.371,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68799999999999994,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.373}]},{"osd":5,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.504,"15min":0.50900000000000001},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":0.60799999999999998,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53100000000000003,"15min":0.53400000000000003},"min":{"1min":0.35499999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74099999999999999,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.41299999999999998}]},{"osd":7,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71999999999999997,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56699999999999995,"15min":0.57199999999999995},"min":{"1min":0.435,"5min":0.36799999999999999,"15min":0.32800000000000001},"max":{"1min":0.70099999999999996,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.47599999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575980,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60100,"kb_used_data":2744,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311740,"statfs":{"total":96636764160,"available":96575221760,"internally_reserved":0,"allocated":2809856,"data_stored":1433342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.248},{"interface":"front","average":{"1min":0.443,"5min":0.432,"15min":0.432},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25700000000000001},"max":{"1min":0.55300000000000005,"5min":0.60999999999999999,"15min":0.69399999999999995},"last":0.35199999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.42999999999999999,"15min":0.42899999999999999},"min":{"1min":0.30299999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46800000000000003,"15min":0.46899999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.29399999999999998},"max":{"1min":0.69499999999999995,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.32100000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.28799999999999998},"max":{"1min":0.67800000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.26300000000000001},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.66100000000000003},"last":0.308}]},{"osd":4,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.64000000000000001,"15min":0.64900000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.33800000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.97399999999999998},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.65600000000000003,"15min":0.66000000000000003},"min":{"1min":0.45000000000000001,"5min":0.44500000000000001,"15min":0.44400000000000001},"max":{"1min":0.877,"5min":0.94599999999999995,"15min":0.995},"last":0.61399999999999999}]},{"osd":5,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.65000000000000002,"15min":0.65600000000000003},"min":{"1min":0.51300000000000001,"5min":0.41299999999999998,"15min":0.40799999999999997},"max":{"1min":0.91800000000000004,"5min":0.96899999999999997,"15min":1.073},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.66800000000000004,"15min":0.67500000000000004},"min":{"1min":0.45500000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.90500000000000003,"5min":0.96499999999999997,"15min":1.0209999999999999},"last":0.65300000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65100000000000002,"15min":0.66200000000000003},"min":{"1min":0.47699999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":1},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.51400000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":1.036},"last":0.63900000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.69799999999999995,"15min":0.69999999999999996},"min":{"1min":0.50600000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89600000000000002,"5min":0.94899999999999995,"15min":1.0649999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.66500000000000004,"15min":0.67300000000000004},"min":{"1min":0.49299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":1.056},"last":0.59399999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575980,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42636,"kb_used_data":3324,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329204,"statfs":{"total":96636764160,"available":96593104896,"internally_reserved":0,"allocated":3403776,"data_stored":2023710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.40200000000000002,"15min":0.39900000000000002},"min":{"1min":0.315,"5min":0.24299999999999999,"15min":0.23799999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.41499999999999998,"15min":0.42299999999999999},"min":{"1min":0.32400000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55800000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.28399999999999997}]},{"osd":2,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.42299999999999999,"15min":0.432},"min":{"1min":0.32100000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.55900000000000005,"5min":0.67900000000000005,"15min":0.70099999999999996},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.441,"15min":0.45200000000000001},"min":{"1min":0.35899999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60799999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.38600000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.42999999999999999,"15min":0.434},"min":{"1min":0.34399999999999997,"5min":0.27400000000000002,"15min":0.26800000000000002},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.48299999999999998,"15min":0.48599999999999999},"min":{"1min":0.39700000000000002,"5min":0.314,"15min":0.307},"max":{"1min":0.64100000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.32700000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.63100000000000001,"15min":0.64800000000000002},"min":{"1min":0.48299999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":1.0329999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.63900000000000001,"15min":0.65100000000000002},"min":{"1min":0.504,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.621}]},{"osd":5,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.628,"15min":0.64800000000000002},"min":{"1min":0.5,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.87,"5min":0.876,"15min":1.0009999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.66300000000000003,"15min":0.67100000000000004},"min":{"1min":0.48599999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.89500000000000002,"5min":0.92800000000000005,"15min":1.052},"last":0.58599999999999997}]},{"osd":6,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67000000000000004,"15min":0.68100000000000005},"min":{"1min":0.52200000000000002,"5min":0.439,"15min":0.435},"max":{"1min":0.94399999999999995,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.66300000000000003,"15min":0.67900000000000005},"min":{"1min":0.49199999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.89500000000000002,"5min":0.96399999999999997,"15min":1.085},"last":0.63600000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.68500000000000005,"15min":0.70099999999999996},"min":{"1min":0.54000000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91200000000000003,"5min":1.01,"15min":1.121},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.67000000000000004,"15min":0.68100000000000005},"min":{"1min":0.55700000000000005,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":1.0229999999999999},"last":0.70699999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575982,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60104,"kb_used_data":2744,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311736,"statfs":{"total":96636764160,"available":96575217664,"internally_reserved":0,"allocated":2809856,"data_stored":1433342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.38900000000000001,"15min":0.38800000000000001},"min":{"1min":0.307,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42199999999999999,"15min":0.42699999999999999},"min":{"1min":0.28100000000000003,"5min":0.27300000000000002,"15min":0.249},"max":{"1min":0.57099999999999995,"5min":0.58099999999999996,"15min":1.044},"last":0.42599999999999999}]},{"osd":1,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40300000000000002,"15min":0.41199999999999998},"min":{"1min":0.28899999999999998,"5min":0.25700000000000001,"15min":0.255},"max":{"1min":0.52300000000000002,"5min":0.56399999999999995,"15min":0.753},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.44500000000000001,"15min":0.45200000000000001},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.28399999999999997},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.82199999999999995},"last":0.35299999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.46000000000000002,"15min":0.46700000000000003},"min":{"1min":0.36499999999999999,"5min":0.32900000000000001,"15min":0.27800000000000002},"max":{"1min":0.63900000000000001,"5min":0.69699999999999995,"15min":0.70299999999999996},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.45200000000000001,"15min":0.45500000000000002},"min":{"1min":0.308,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.59799999999999998,"5min":0.64500000000000002,"15min":0.68600000000000005},"last":0.36799999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.64000000000000001,"15min":0.64500000000000002},"min":{"1min":0.47199999999999998,"5min":0.42199999999999999,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66400000000000003,"15min":0.66800000000000004},"min":{"1min":0.44500000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.55200000000000005}]},{"osd":5,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66800000000000004,"15min":0.67000000000000004},"min":{"1min":0.46100000000000002,"5min":0.46000000000000002,"15min":0.433},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":1.0149999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.66900000000000004,"15min":0.67200000000000004},"min":{"1min":0.52500000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.76600000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.67300000000000004,"15min":0.67800000000000005},"min":{"1min":0.46999999999999997,"5min":0.46700000000000003,"15min":0.44600000000000001},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":1.091},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69999999999999996,"15min":0.70199999999999996},"min":{"1min":0.50600000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.96099999999999997},"last":0.753}]},{"osd":7,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68300000000000005,"15min":0.69199999999999995},"min":{"1min":0.5,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92500000000000004,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.70399999999999996,"15min":0.71599999999999997},"min":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.71499999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575981,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47800,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324040,"statfs":{"total":96636764160,"available":96587816960,"internally_reserved":0,"allocated":2789376,"data_stored":1422073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39000000000000001,"15min":0.38900000000000001},"min":{"1min":0.26000000000000001,"5min":0.255,"15min":0.23499999999999999},"max":{"1min":0.55200000000000005,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41199999999999998,"15min":0.41299999999999998},"min":{"1min":0.28799999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.56899999999999995,"5min":0.622,"15min":0.66700000000000004},"last":0.28599999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40699999999999997,"15min":0.40600000000000003},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.65700000000000003},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.438,"15min":0.44400000000000001},"min":{"1min":0.32800000000000001,"5min":0.27500000000000002,"15min":0.26600000000000001},"max":{"1min":0.63,"5min":0.65400000000000003,"15min":0.75800000000000001},"last":0.26700000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48299999999999998,"15min":0.48599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.5,"15min":0.503},"min":{"1min":0.38600000000000001,"5min":0.34599999999999997,"15min":0.32900000000000001},"max":{"1min":0.67800000000000005,"5min":0.69299999999999995,"15min":0.79300000000000004},"last":0.42099999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.439,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.876,"5min":1.002,"15min":1.018},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66100000000000003,"15min":0.65300000000000002},"min":{"1min":0.497,"5min":0.45900000000000002,"15min":0.375},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.57999999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.51900000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.95699999999999996,"5min":0.97299999999999998,"15min":0.999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.46200000000000002,"5min":0.437,"15min":0.437},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.79300000000000004}]},{"osd":6,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67500000000000004,"15min":0.66900000000000004},"min":{"1min":0.41499999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.68600000000000005,"15min":0.68400000000000005},"min":{"1min":0.48799999999999999,"5min":0.46700000000000003,"15min":0.41099999999999998},"max":{"1min":0.88800000000000001,"5min":0.96199999999999997,"15min":0.98799999999999999},"last":0.77500000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68500000000000005,"15min":0.68000000000000005},"min":{"1min":0.46100000000000002,"5min":0.432,"15min":0.40999999999999998},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70299999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.024},"last":0.80300000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575981,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47816,"kb_used_data":2748,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324024,"statfs":{"total":96636764160,"available":96587800576,"internally_reserved":0,"allocated":2813952,"data_stored":1434361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.373,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.76400000000000001,"5min":0.876,"15min":0.88500000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.60499999999999998,"15min":0.60299999999999998},"min":{"1min":0.40200000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77900000000000003,"5min":1.202,"15min":1.202},"last":0.61799999999999999}]},{"osd":1,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63,"15min":0.64400000000000002},"min":{"1min":0.439,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.82999999999999996,"5min":1.256,"15min":1.256},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64000000000000001,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80900000000000005,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.52700000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.42499999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.81799999999999995,"5min":0.84599999999999997,"15min":0.877},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.627,"15min":0.621},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88500000000000001,"5min":1.294,"15min":1.294},"last":0.54100000000000004}]},{"osd":3,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.44900000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.75,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46899999999999997,"5min":0.373,"15min":0.373},"max":{"1min":0.78000000000000003,"5min":0.90200000000000002,"15min":0.91100000000000003},"last":0.55600000000000005}]},{"osd":5,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.56200000000000006,"15min":0.51600000000000001},"min":{"1min":0.35299999999999998,"5min":0.33200000000000002,"15min":0.32800000000000001},"max":{"1min":0.69799999999999995,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.57899999999999996,"15min":0.54300000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68600000000000005,"5min":4.931,"15min":4.931},"last":0.46600000000000003}]},{"osd":6,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54000000000000004,"15min":0.53000000000000003},"min":{"1min":0.34100000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72399999999999998,"5min":0.83499999999999996,"15min":0.83899999999999997},"last":0.434},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57899999999999996,"15min":0.56999999999999995},"min":{"1min":0.41599999999999998,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":0.751,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.58099999999999996}]},{"osd":7,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.60899999999999999,"15min":0.58299999999999996},"min":{"1min":0.36899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.65400000000000003,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.60399999999999998,"15min":0.60199999999999998},"min":{"1min":0.437,"5min":0.40799999999999997,"15min":0.373},"max":{"1min":0.748,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.63}]}]},{"osd":5,"up_from":13,"seq":55834575981,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60100,"kb_used_data":2744,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311740,"statfs":{"total":96636764160,"available":96575221760,"internally_reserved":0,"allocated":2809856,"data_stored":1433342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.46400000000000002,"5min":0.34000000000000002,"15min":0.33000000000000002},"max":{"1min":0.70199999999999996,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58999999999999997,"15min":0.59499999999999997},"min":{"1min":0.39700000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.73999999999999999,"5min":0.84699999999999998,"15min":0.89200000000000002},"last":0.63900000000000001}]},{"osd":1,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.41199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72699999999999998,"5min":0.81299999999999994,"15min":0.81399999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58599999999999997,"15min":0.58699999999999997},"min":{"1min":0.43099999999999999,"5min":0.39400000000000002,"15min":0.37},"max":{"1min":0.70999999999999996,"5min":0.871,"15min":0.871},"last":0.61199999999999999}]},{"osd":2,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60599999999999998,"15min":0.60099999999999998},"min":{"1min":0.39400000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.70999999999999996,"5min":0.79700000000000004,"15min":0.84399999999999997},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.43099999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.67900000000000005,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.73099999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60499999999999998,"15min":0.59899999999999998},"min":{"1min":0.44700000000000001,"5min":0.40999999999999998,"15min":0.373},"max":{"1min":0.754,"5min":0.81100000000000005,"15min":0.94499999999999995},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63600000000000001,"15min":0.63700000000000001},"min":{"1min":0.48499999999999999,"5min":0.40300000000000002,"15min":0.39000000000000001},"max":{"1min":0.81399999999999995,"5min":0.88200000000000001,"15min":0.91500000000000004},"last":0.65700000000000003}]},{"osd":4,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52500000000000002,"15min":0.505},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.27400000000000002},"max":{"1min":0.746,"5min":0.85699999999999998,"15min":1.0369999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60499999999999998,"15min":0.57699999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.308},"max":{"1min":0.78000000000000003,"5min":0.88800000000000001,"15min":1.1639999999999999},"last":0.76500000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54000000000000004,"15min":0.52400000000000002},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.27900000000000003},"max":{"1min":0.73799999999999999,"5min":0.94999999999999996,"15min":1.0169999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54600000000000004,"15min":0.52800000000000002},"min":{"1min":0.34399999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70999999999999996,"5min":0.878,"15min":1.0009999999999999},"last":0.71199999999999997}]},{"osd":7,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56100000000000005,"15min":0.54500000000000004},"min":{"1min":0.374,"5min":0.33600000000000002,"15min":0.32900000000000001},"max":{"1min":0.70899999999999996,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.66500000000000004,"15min":0.64700000000000002},"min":{"1min":0.45600000000000002,"5min":0.433,"15min":0.35199999999999998},"max":{"1min":0.84799999999999998,"5min":0.92300000000000004,"15min":0.96599999999999997},"last":0.82199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:47:26.258 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:47:26.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.415+0000 7f3a5b44f700 1 Processor -- start 2024-04-06T13:47:26.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.415+0000 7f3a5b44f700 1 -- start start 2024-04-06T13:47:26.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.416+0000 7f3a5b44f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a5415e8a0 0x7f3a540cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:26.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.416+0000 7f3a5b44f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3a5415eee0 0x7f3a5415f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:26.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.416+0000 7f3a5b44f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a540669d0 con 0x7f3a5415f8a0 2024-04-06T13:47:26.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.416+0000 7f3a5b44f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a5415fd30 con 0x7f3a5415eee0 2024-04-06T13:47:26.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.416+0000 7f3a5b44f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5406ede0 con 0x7f3a5415e8a0 2024-04-06T13:47:26.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.416+0000 7f3a589ea700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3a5415eee0 0x7f3a5415f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:26.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.416+0000 7f3a599ec700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3a5415f8a0 0x7f3a5406ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58518/0 (socket says 172.21.15.189:58518) 2024-04-06T13:47:26.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.416+0000 7f3a599ec700 1 -- 172.21.15.189:0/3840593927 learned_addr learned my addr 172.21.15.189:0/3840593927 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:26.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.417+0000 7f3a591eb700 1 --2- 172.21.15.189:0/3840593927 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a5415e8a0 0x7f3a540cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:26.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.417+0000 7f3a589ea700 1 -- 172.21.15.189:0/3840593927 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a5415e8a0 msgr2=0x7f3a540cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:26.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.417+0000 7f3a589ea700 1 --2- 172.21.15.189:0/3840593927 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a5415e8a0 0x7f3a540cbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.417+0000 7f3a589ea700 1 -- 172.21.15.189:0/3840593927 >> v1:172.21.15.189:6789/0 conn(0x7f3a5415f8a0 legacy=0x7f3a5406ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:26.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.417+0000 7f3a4bfff700 1 -- 172.21.15.189:0/3840593927 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999623431 0 0) 0x7f3a540669d0 con 0x7f3a5415f8a0 2024-04-06T13:47:26.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.417+0000 7f3a589ea700 1 -- 172.21.15.189:0/3840593927 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5415ec80 con 0x7f3a5415eee0 2024-04-06T13:47:26.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.417+0000 7f3a591eb700 1 --2- 172.21.15.189:0/3840593927 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a5415e8a0 0x7f3a540cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:26.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.417+0000 7f3a589ea700 1 --2- 172.21.15.189:0/3840593927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a5415eee0 0x7f3a5415f360 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009840 tx=0x7f3a50009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9432f7b0ebb097d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:26.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.417+0000 7f3a4bfff700 1 -- 172.21.15.189:0/3840593927 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50011830 con 0x7f3a5415eee0 2024-04-06T13:47:26.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.417+0000 7f3a4bfff700 1 -- 172.21.15.189:0/3840593927 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a50011990 con 0x7f3a5415eee0 2024-04-06T13:47:26.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.417+0000 7f3a4bfff700 1 -- 172.21.15.189:0/3840593927 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50011d20 con 0x7f3a5415eee0 2024-04-06T13:47:26.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.417+0000 7f3a5b44f700 1 -- 172.21.15.189:0/3840593927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a5415eee0 msgr2=0x7f3a5415f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:26.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.417+0000 7f3a5b44f700 1 --2- 172.21.15.189:0/3840593927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a5415eee0 0x7f3a5415f360 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009840 tx=0x7f3a50009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:26.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.418+0000 7f3a5b44f700 1 -- 172.21.15.189:0/3840593927 shutdown_connections 2024-04-06T13:47:26.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.418+0000 7f3a5b44f700 1 --2- 172.21.15.189:0/3840593927 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a5415e8a0 0x7f3a540cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.418+0000 7f3a5b44f700 1 --2- 172.21.15.189:0/3840593927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a5415eee0 0x7f3a5415f360 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.418+0000 7f3a5b44f700 1 -- 172.21.15.189:0/3840593927 >> 172.21.15.189:0/3840593927 conn(0x7f3a540cc290 msgr2=0x7f3a540cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:26.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.418+0000 7f3a5b44f700 1 -- 172.21.15.189:0/3840593927 shutdown_connections 2024-04-06T13:47:26.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.418+0000 7f3a5b44f700 1 -- 172.21.15.189:0/3840593927 wait complete. 2024-04-06T13:47:26.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.419+0000 7f3a5b44f700 1 Processor -- start 2024-04-06T13:47:26.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.419+0000 7f3a5b44f700 1 -- start start 2024-04-06T13:47:26.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a5b44f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a5415e8a0 0x7f3a54156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:26.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a5b44f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a5415eee0 0x7f3a541520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:26.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a5b44f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a5415f8a0 0x7f3a54152d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:26.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a5b44f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a541501b0 con 0x7f3a5415e8a0 2024-04-06T13:47:26.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a5b44f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a541525f0 con 0x7f3a5415f8a0 2024-04-06T13:47:26.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a5b44f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a54157530 con 0x7f3a5415eee0 2024-04-06T13:47:26.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a591eb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a5415e8a0 0x7f3a54156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:26.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a591eb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a5415e8a0 0x7f3a54156a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55144/0 (socket says 172.21.15.189:55144) 2024-04-06T13:47:26.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a591eb700 1 -- 172.21.15.189:0/2881677528 learned_addr learned my addr 172.21.15.189:0/2881677528 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:26.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a589ea700 1 --2- 172.21.15.189:0/2881677528 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a5415eee0 0x7f3a541520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:26.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a599ec700 1 --2- 172.21.15.189:0/2881677528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a5415f8a0 0x7f3a54152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:26.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a591eb700 1 -- 172.21.15.189:0/2881677528 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a5415eee0 msgr2=0x7f3a541520b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:26.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a591eb700 1 --2- 172.21.15.189:0/2881677528 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a5415eee0 0x7f3a541520b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a591eb700 1 -- 172.21.15.189:0/2881677528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a5415f8a0 msgr2=0x7f3a54152d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:26.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a591eb700 1 --2- 172.21.15.189:0/2881677528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a5415f8a0 0x7f3a54152d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a591eb700 1 -- 172.21.15.189:0/2881677528 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a500094a0 con 0x7f3a5415e8a0 2024-04-06T13:47:26.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a599ec700 1 --2- 172.21.15.189:0/2881677528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a5415f8a0 0x7f3a54152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:26.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a591eb700 1 --2- 172.21.15.189:0/2881677528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a5415e8a0 0x7f3a54156a80 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400c750 tx=0x7f3a4400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:26.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a49ffb700 1 -- 172.21.15.189:0/2881677528 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44011800 con 0x7f3a5415e8a0 2024-04-06T13:47:26.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a49ffb700 1 -- 172.21.15.189:0/2881677528 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a44011960 con 0x7f3a5415e8a0 2024-04-06T13:47:26.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a5b44f700 1 -- 172.21.15.189:0/2881677528 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a54157810 con 0x7f3a5415e8a0 2024-04-06T13:47:26.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.420+0000 7f3a49ffb700 1 -- 172.21.15.189:0/2881677528 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44011cd0 con 0x7f3a5415e8a0 2024-04-06T13:47:26.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.421+0000 7f3a5b44f700 1 -- 172.21.15.189:0/2881677528 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a54153280 con 0x7f3a5415e8a0 2024-04-06T13:47:26.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.422+0000 7f3a5b44f700 1 -- 172.21.15.189:0/2881677528 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a54072140 con 0x7f3a5415e8a0 2024-04-06T13:47:26.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.422+0000 7f3a49ffb700 1 -- 172.21.15.189:0/2881677528 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f3a44021350 con 0x7f3a5415e8a0 2024-04-06T13:47:26.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.422+0000 7f3a49ffb700 1 --2- 172.21.15.189:0/2881677528 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3a3806adb0 0x7f3a3806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:26.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.422+0000 7f3a49ffb700 1 -- 172.21.15.189:0/2881677528 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3a440896c0 con 0x7f3a5415e8a0 2024-04-06T13:47:26.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.422+0000 7f3a589ea700 1 --2- 172.21.15.189:0/2881677528 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3a3806adb0 0x7f3a3806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:26.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.423+0000 7f3a589ea700 1 --2- 172.21.15.189:0/2881677528 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3a3806adb0 0x7f3a3806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3a500090b0 tx=0x7f3a50008030 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:26.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.427+0000 7f3a49ffb700 1 -- 172.21.15.189:0/2881677528 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3a44054580 con 0x7f3a5415e8a0 2024-04-06T13:47:26.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.620+0000 7f3a5b44f700 1 -- 172.21.15.189:0/2881677528 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3a5406ea70 con 0x7f3a3806adb0 2024-04-06T13:47:26.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.628+0000 7f3a49ffb700 1 -- 172.21.15.189:0/2881677528 <== mgr.11028 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290015 (secure 0 0 0) 0x7f3a50011750 con 0x7f3a3806adb0 2024-04-06T13:47:26.631 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:26.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.631+0000 7f3a5b44f700 1 -- 172.21.15.189:0/2881677528 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3a3806adb0 msgr2=0x7f3a3806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:26.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.631+0000 7f3a5b44f700 1 --2- 172.21.15.189:0/2881677528 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3a3806adb0 0x7f3a3806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3a500090b0 tx=0x7f3a50008030 comp rx=0 tx=0).stop 2024-04-06T13:47:26.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.631+0000 7f3a5b44f700 1 -- 172.21.15.189:0/2881677528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a5415e8a0 msgr2=0x7f3a54156a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:26.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.631+0000 7f3a5b44f700 1 --2- 172.21.15.189:0/2881677528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a5415e8a0 0x7f3a54156a80 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400c750 tx=0x7f3a4400cb10 comp rx=0 tx=0).stop 2024-04-06T13:47:26.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.631+0000 7f3a5b44f700 1 -- 172.21.15.189:0/2881677528 shutdown_connections 2024-04-06T13:47:26.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.631+0000 7f3a5b44f700 1 --2- 172.21.15.189:0/2881677528 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3a3806adb0 0x7f3a3806d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.631+0000 7f3a5b44f700 1 --2- 172.21.15.189:0/2881677528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a5415f8a0 0x7f3a54152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.631+0000 7f3a5b44f700 1 --2- 172.21.15.189:0/2881677528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a5415e8a0 0x7f3a54156a80 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.631+0000 7f3a5b44f700 1 --2- 172.21.15.189:0/2881677528 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3a5415eee0 0x7f3a541520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.631+0000 7f3a5b44f700 1 -- 172.21.15.189:0/2881677528 >> 172.21.15.189:0/2881677528 conn(0x7f3a540cc290 msgr2=0x7f3a5415d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:26.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.631+0000 7f3a5b44f700 1 -- 172.21.15.189:0/2881677528 shutdown_connections 2024-04-06T13:47:26.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.631+0000 7f3a5b44f700 1 -- 172.21.15.189:0/2881677528 wait complete. 2024-04-06T13:47:26.635 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:47:26.643 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-06T13:47:26.409514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":624,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":387,"ondisk_log_size":387,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":159,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":413508,"kb_used_data":23476,"kb_used_omap":12,"kb_used_meta":389875,"kb_avail":754561212,"statfs":{"total":773094113280,"available":772670681088,"internally_reserved":0,"allocated":24039424,"data_stored":12819461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":399232568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.712976"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.227617+0000","last_change":"2024-04-06T13:47:22.143038+0000","last_active":"2024-04-06T13:47:23.227617+0000","last_peered":"2024-04-06T13:47:23.227617+0000","last_clean":"2024-04-06T13:47:23.227617+0000","last_became_active":"2024-04-06T13:47:22.142498+0000","last_became_peered":"2024-04-06T13:47:22.142498+0000","last_unstale":"2024-04-06T13:47:23.227617+0000","last_undegraded":"2024-04-06T13:47:23.227617+0000","last_fullsized":"2024-04-06T13:47:23.227617+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:40.999324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.227636+0000","last_change":"2024-04-06T13:47:22.148394+0000","last_active":"2024-04-06T13:47:23.227636+0000","last_peered":"2024-04-06T13:47:23.227636+0000","last_clean":"2024-04-06T13:47:23.227636+0000","last_became_active":"2024-04-06T13:47:22.148136+0000","last_became_peered":"2024-04-06T13:47:22.148136+0000","last_unstale":"2024-04-06T13:47:23.227636+0000","last_undegraded":"2024-04-06T13:47:23.227636+0000","last_fullsized":"2024-04-06T13:47:23.227636+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:06:09.829859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.226767+0000","last_change":"2024-04-06T13:47:22.137179+0000","last_active":"2024-04-06T13:47:23.226767+0000","last_peered":"2024-04-06T13:47:23.226767+0000","last_clean":"2024-04-06T13:47:23.226767+0000","last_became_active":"2024-04-06T13:47:22.136779+0000","last_became_peered":"2024-04-06T13:47:22.136779+0000","last_unstale":"2024-04-06T13:47:23.226767+0000","last_undegraded":"2024-04-06T13:47:23.226767+0000","last_fullsized":"2024-04-06T13:47:23.226767+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:25:40.923568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.133842+0000","last_change":"2024-04-06T13:47:22.132869+0000","last_active":"2024-04-06T13:47:23.133842+0000","last_peered":"2024-04-06T13:47:23.133842+0000","last_clean":"2024-04-06T13:47:23.133842+0000","last_became_active":"2024-04-06T13:47:22.132613+0000","last_became_peered":"2024-04-06T13:47:22.132613+0000","last_unstale":"2024-04-06T13:47:23.133842+0000","last_undegraded":"2024-04-06T13:47:23.133842+0000","last_fullsized":"2024-04-06T13:47:23.133842+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:38:24.228670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.227668+0000","last_change":"2024-04-06T13:47:22.137418+0000","last_active":"2024-04-06T13:47:23.227668+0000","last_peered":"2024-04-06T13:47:23.227668+0000","last_clean":"2024-04-06T13:47:23.227668+0000","last_became_active":"2024-04-06T13:47:22.137176+0000","last_became_peered":"2024-04-06T13:47:22.137176+0000","last_unstale":"2024-04-06T13:47:23.227668+0000","last_undegraded":"2024-04-06T13:47:23.227668+0000","last_fullsized":"2024-04-06T13:47:23.227668+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:40:38.388468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.132908+0000","last_change":"2024-04-06T13:47:22.130575+0000","last_active":"2024-04-06T13:47:23.132908+0000","last_peered":"2024-04-06T13:47:23.132908+0000","last_clean":"2024-04-06T13:47:23.132908+0000","last_became_active":"2024-04-06T13:47:22.130287+0000","last_became_peered":"2024-04-06T13:47:22.130287+0000","last_unstale":"2024-04-06T13:47:23.132908+0000","last_undegraded":"2024-04-06T13:47:23.132908+0000","last_fullsized":"2024-04-06T13:47:23.132908+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:33:34.282063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.227323+0000","last_change":"2024-04-06T13:47:22.136209+0000","last_active":"2024-04-06T13:47:23.227323+0000","last_peered":"2024-04-06T13:47:23.227323+0000","last_clean":"2024-04-06T13:47:23.227323+0000","last_became_active":"2024-04-06T13:47:22.135366+0000","last_became_peered":"2024-04-06T13:47:22.135366+0000","last_unstale":"2024-04-06T13:47:23.227323+0000","last_undegraded":"2024-04-06T13:47:23.227323+0000","last_fullsized":"2024-04-06T13:47:23.227323+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:14:48.802617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.132661+0000","last_change":"2024-04-06T13:47:22.146702+0000","last_active":"2024-04-06T13:47:23.132661+0000","last_peered":"2024-04-06T13:47:23.132661+0000","last_clean":"2024-04-06T13:47:23.132661+0000","last_became_active":"2024-04-06T13:47:22.139334+0000","last_became_peered":"2024-04-06T13:47:22.139334+0000","last_unstale":"2024-04-06T13:47:23.132661+0000","last_undegraded":"2024-04-06T13:47:23.132661+0000","last_fullsized":"2024-04-06T13:47:23.132661+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:36:19.531363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.129965+0000","last_change":"2024-04-06T13:47:22.147771+0000","last_active":"2024-04-06T13:47:23.129965+0000","last_peered":"2024-04-06T13:47:23.129965+0000","last_clean":"2024-04-06T13:47:23.129965+0000","last_became_active":"2024-04-06T13:47:22.147512+0000","last_became_peered":"2024-04-06T13:47:22.147512+0000","last_unstale":"2024-04-06T13:47:23.129965+0000","last_undegraded":"2024-04-06T13:47:23.129965+0000","last_fullsized":"2024-04-06T13:47:23.129965+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:25:04.287164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.133579+0000","last_change":"2024-04-06T13:47:22.140321+0000","last_active":"2024-04-06T13:47:23.133579+0000","last_peered":"2024-04-06T13:47:23.133579+0000","last_clean":"2024-04-06T13:47:23.133579+0000","last_became_active":"2024-04-06T13:47:22.140027+0000","last_became_peered":"2024-04-06T13:47:22.140027+0000","last_unstale":"2024-04-06T13:47:23.133579+0000","last_undegraded":"2024-04-06T13:47:23.133579+0000","last_fullsized":"2024-04-06T13:47:23.133579+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:49:16.852621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.226858+0000","last_change":"2024-04-06T13:47:22.147317+0000","last_active":"2024-04-06T13:47:23.226858+0000","last_peered":"2024-04-06T13:47:23.226858+0000","last_clean":"2024-04-06T13:47:23.226858+0000","last_became_active":"2024-04-06T13:47:22.146941+0000","last_became_peered":"2024-04-06T13:47:22.146941+0000","last_unstale":"2024-04-06T13:47:23.226858+0000","last_undegraded":"2024-04-06T13:47:23.226858+0000","last_fullsized":"2024-04-06T13:47:23.226858+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:55:01.812146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.227636+0000","last_change":"2024-04-06T13:47:22.139348+0000","last_active":"2024-04-06T13:47:23.227636+0000","last_peered":"2024-04-06T13:47:23.227636+0000","last_clean":"2024-04-06T13:47:23.227636+0000","last_became_active":"2024-04-06T13:47:22.138953+0000","last_became_peered":"2024-04-06T13:47:22.138953+0000","last_unstale":"2024-04-06T13:47:23.227636+0000","last_undegraded":"2024-04-06T13:47:23.227636+0000","last_fullsized":"2024-04-06T13:47:23.227636+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:51:33.543781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.133527+0000","last_change":"2024-04-06T13:47:22.150089+0000","last_active":"2024-04-06T13:47:23.133527+0000","last_peered":"2024-04-06T13:47:23.133527+0000","last_clean":"2024-04-06T13:47:23.133527+0000","last_became_active":"2024-04-06T13:47:22.149867+0000","last_became_peered":"2024-04-06T13:47:22.149867+0000","last_unstale":"2024-04-06T13:47:23.133527+0000","last_undegraded":"2024-04-06T13:47:23.133527+0000","last_fullsized":"2024-04-06T13:47:23.133527+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:55:30.108853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.133280+0000","last_change":"2024-04-06T13:47:22.139295+0000","last_active":"2024-04-06T13:47:23.133280+0000","last_peered":"2024-04-06T13:47:23.133280+0000","last_clean":"2024-04-06T13:47:23.133280+0000","last_became_active":"2024-04-06T13:47:22.138518+0000","last_became_peered":"2024-04-06T13:47:22.138518+0000","last_unstale":"2024-04-06T13:47:23.133280+0000","last_undegraded":"2024-04-06T13:47:23.133280+0000","last_fullsized":"2024-04-06T13:47:23.133280+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:03:39.808430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.227187+0000","last_change":"2024-04-06T13:47:22.139469+0000","last_active":"2024-04-06T13:47:23.227187+0000","last_peered":"2024-04-06T13:47:23.227187+0000","last_clean":"2024-04-06T13:47:23.227187+0000","last_became_active":"2024-04-06T13:47:22.139177+0000","last_became_peered":"2024-04-06T13:47:22.139177+0000","last_unstale":"2024-04-06T13:47:23.227187+0000","last_undegraded":"2024-04-06T13:47:23.227187+0000","last_fullsized":"2024-04-06T13:47:23.227187+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:12:44.053539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.228070+0000","last_change":"2024-04-06T13:47:22.142723+0000","last_active":"2024-04-06T13:47:23.228070+0000","last_peered":"2024-04-06T13:47:23.228070+0000","last_clean":"2024-04-06T13:47:23.228070+0000","last_became_active":"2024-04-06T13:47:22.142456+0000","last_became_peered":"2024-04-06T13:47:22.142456+0000","last_unstale":"2024-04-06T13:47:23.228070+0000","last_undegraded":"2024-04-06T13:47:23.228070+0000","last_fullsized":"2024-04-06T13:47:23.228070+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:24:10.160194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.130059+0000","last_change":"2024-04-06T13:47:22.148647+0000","last_active":"2024-04-06T13:47:23.130059+0000","last_peered":"2024-04-06T13:47:23.130059+0000","last_clean":"2024-04-06T13:47:23.130059+0000","last_became_active":"2024-04-06T13:47:22.148404+0000","last_became_peered":"2024-04-06T13:47:22.148404+0000","last_unstale":"2024-04-06T13:47:23.130059+0000","last_undegraded":"2024-04-06T13:47:23.130059+0000","last_fullsized":"2024-04-06T13:47:23.130059+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:38:04.045660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.129891+0000","last_change":"2024-04-06T13:47:22.133302+0000","last_active":"2024-04-06T13:47:23.129891+0000","last_peered":"2024-04-06T13:47:23.129891+0000","last_clean":"2024-04-06T13:47:23.129891+0000","last_became_active":"2024-04-06T13:47:22.132902+0000","last_became_peered":"2024-04-06T13:47:22.132902+0000","last_unstale":"2024-04-06T13:47:23.129891+0000","last_undegraded":"2024-04-06T13:47:23.129891+0000","last_fullsized":"2024-04-06T13:47:23.129891+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:55:56.178154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.133498+0000","last_change":"2024-04-06T13:47:22.145887+0000","last_active":"2024-04-06T13:47:23.133498+0000","last_peered":"2024-04-06T13:47:23.133498+0000","last_clean":"2024-04-06T13:47:23.133498+0000","last_became_active":"2024-04-06T13:47:22.141744+0000","last_became_peered":"2024-04-06T13:47:22.141744+0000","last_unstale":"2024-04-06T13:47:23.133498+0000","last_undegraded":"2024-04-06T13:47:23.133498+0000","last_fullsized":"2024-04-06T13:47:23.133498+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:40:34.810688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.133141+0000","last_change":"2024-04-06T13:47:22.140812+0000","last_active":"2024-04-06T13:47:23.133141+0000","last_peered":"2024-04-06T13:47:23.133141+0000","last_clean":"2024-04-06T13:47:23.133141+0000","last_became_active":"2024-04-06T13:47:22.139010+0000","last_became_peered":"2024-04-06T13:47:22.139010+0000","last_unstale":"2024-04-06T13:47:23.133141+0000","last_undegraded":"2024-04-06T13:47:23.133141+0000","last_fullsized":"2024-04-06T13:47:23.133141+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:50:16.774388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.133177+0000","last_change":"2024-04-06T13:47:22.141672+0000","last_active":"2024-04-06T13:47:23.133177+0000","last_peered":"2024-04-06T13:47:23.133177+0000","last_clean":"2024-04-06T13:47:23.133177+0000","last_became_active":"2024-04-06T13:47:22.141369+0000","last_became_peered":"2024-04-06T13:47:22.141369+0000","last_unstale":"2024-04-06T13:47:23.133177+0000","last_undegraded":"2024-04-06T13:47:23.133177+0000","last_fullsized":"2024-04-06T13:47:23.133177+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:27:56.909726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.133393+0000","last_change":"2024-04-06T13:47:22.139139+0000","last_active":"2024-04-06T13:47:23.133393+0000","last_peered":"2024-04-06T13:47:23.133393+0000","last_clean":"2024-04-06T13:47:23.133393+0000","last_became_active":"2024-04-06T13:47:22.138505+0000","last_became_peered":"2024-04-06T13:47:22.138505+0000","last_unstale":"2024-04-06T13:47:23.133393+0000","last_undegraded":"2024-04-06T13:47:23.133393+0000","last_fullsized":"2024-04-06T13:47:23.133393+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:07:04.253792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.227137+0000","last_change":"2024-04-06T13:47:22.139078+0000","last_active":"2024-04-06T13:47:23.227137+0000","last_peered":"2024-04-06T13:47:23.227137+0000","last_clean":"2024-04-06T13:47:23.227137+0000","last_became_active":"2024-04-06T13:47:22.138825+0000","last_became_peered":"2024-04-06T13:47:22.138825+0000","last_unstale":"2024-04-06T13:47:23.227137+0000","last_undegraded":"2024-04-06T13:47:23.227137+0000","last_fullsized":"2024-04-06T13:47:23.227137+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:14:17.994649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.226870+0000","last_change":"2024-04-06T13:47:22.139336+0000","last_active":"2024-04-06T13:47:23.226870+0000","last_peered":"2024-04-06T13:47:23.226870+0000","last_clean":"2024-04-06T13:47:23.226870+0000","last_became_active":"2024-04-06T13:47:22.139092+0000","last_became_peered":"2024-04-06T13:47:22.139092+0000","last_unstale":"2024-04-06T13:47:23.226870+0000","last_undegraded":"2024-04-06T13:47:23.226870+0000","last_fullsized":"2024-04-06T13:47:23.226870+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:55:13.445775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.130096+0000","last_change":"2024-04-06T13:47:22.133305+0000","last_active":"2024-04-06T13:47:23.130096+0000","last_peered":"2024-04-06T13:47:23.130096+0000","last_clean":"2024-04-06T13:47:23.130096+0000","last_became_active":"2024-04-06T13:47:22.132897+0000","last_became_peered":"2024-04-06T13:47:22.132897+0000","last_unstale":"2024-04-06T13:47:23.130096+0000","last_undegraded":"2024-04-06T13:47:23.130096+0000","last_fullsized":"2024-04-06T13:47:23.130096+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:10:25.671045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.227154+0000","last_change":"2024-04-06T13:47:22.144335+0000","last_active":"2024-04-06T13:47:23.227154+0000","last_peered":"2024-04-06T13:47:23.227154+0000","last_clean":"2024-04-06T13:47:23.227154+0000","last_became_active":"2024-04-06T13:47:22.144006+0000","last_became_peered":"2024-04-06T13:47:22.144006+0000","last_unstale":"2024-04-06T13:47:23.227154+0000","last_undegraded":"2024-04-06T13:47:23.227154+0000","last_fullsized":"2024-04-06T13:47:23.227154+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:02:19.582423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.226907+0000","last_change":"2024-04-06T13:47:22.147177+0000","last_active":"2024-04-06T13:47:23.226907+0000","last_peered":"2024-04-06T13:47:23.226907+0000","last_clean":"2024-04-06T13:47:23.226907+0000","last_became_active":"2024-04-06T13:47:22.146634+0000","last_became_peered":"2024-04-06T13:47:22.146634+0000","last_unstale":"2024-04-06T13:47:23.226907+0000","last_undegraded":"2024-04-06T13:47:23.226907+0000","last_fullsized":"2024-04-06T13:47:23.226907+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:16:29.805273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.227706+0000","last_change":"2024-04-06T13:47:22.139194+0000","last_active":"2024-04-06T13:47:23.227706+0000","last_peered":"2024-04-06T13:47:23.227706+0000","last_clean":"2024-04-06T13:47:23.227706+0000","last_became_active":"2024-04-06T13:47:22.138677+0000","last_became_peered":"2024-04-06T13:47:22.138677+0000","last_unstale":"2024-04-06T13:47:23.227706+0000","last_undegraded":"2024-04-06T13:47:23.227706+0000","last_fullsized":"2024-04-06T13:47:23.227706+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:36:38.719202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.129657+0000","last_change":"2024-04-06T13:47:22.139159+0000","last_active":"2024-04-06T13:47:23.129657+0000","last_peered":"2024-04-06T13:47:23.129657+0000","last_clean":"2024-04-06T13:47:23.129657+0000","last_became_active":"2024-04-06T13:47:22.138916+0000","last_became_peered":"2024-04-06T13:47:22.138916+0000","last_unstale":"2024-04-06T13:47:23.129657+0000","last_undegraded":"2024-04-06T13:47:23.129657+0000","last_fullsized":"2024-04-06T13:47:23.129657+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:58:23.128672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.226964+0000","last_change":"2024-04-06T13:47:22.138788+0000","last_active":"2024-04-06T13:47:23.226964+0000","last_peered":"2024-04-06T13:47:23.226964+0000","last_clean":"2024-04-06T13:47:23.226964+0000","last_became_active":"2024-04-06T13:47:22.138480+0000","last_became_peered":"2024-04-06T13:47:22.138480+0000","last_unstale":"2024-04-06T13:47:23.226964+0000","last_undegraded":"2024-04-06T13:47:23.226964+0000","last_fullsized":"2024-04-06T13:47:23.226964+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:40.999324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.133210+0000","last_change":"2024-04-06T13:47:22.141102+0000","last_active":"2024-04-06T13:47:23.133210+0000","last_peered":"2024-04-06T13:47:23.133210+0000","last_clean":"2024-04-06T13:47:23.133210+0000","last_became_active":"2024-04-06T13:47:22.140831+0000","last_became_peered":"2024-04-06T13:47:22.140831+0000","last_unstale":"2024-04-06T13:47:23.133210+0000","last_undegraded":"2024-04-06T13:47:23.133210+0000","last_fullsized":"2024-04-06T13:47:23.133210+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:27:31.928923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.132653+0000","last_change":"2024-04-06T13:47:22.132331+0000","last_active":"2024-04-06T13:47:23.132653+0000","last_peered":"2024-04-06T13:47:23.132653+0000","last_clean":"2024-04-06T13:47:23.132653+0000","last_became_active":"2024-04-06T13:47:22.132068+0000","last_became_peered":"2024-04-06T13:47:22.132068+0000","last_unstale":"2024-04-06T13:47:23.132653+0000","last_undegraded":"2024-04-06T13:47:23.132653+0000","last_fullsized":"2024-04-06T13:47:23.132653+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:45:10.805753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.227145+0000","last_change":"2024-04-06T13:47:22.139965+0000","last_active":"2024-04-06T13:47:23.227145+0000","last_peered":"2024-04-06T13:47:23.227145+0000","last_clean":"2024-04-06T13:47:23.227145+0000","last_became_active":"2024-04-06T13:47:22.139645+0000","last_became_peered":"2024-04-06T13:47:22.139645+0000","last_unstale":"2024-04-06T13:47:23.227145+0000","last_undegraded":"2024-04-06T13:47:23.227145+0000","last_fullsized":"2024-04-06T13:47:23.227145+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:32:56.671578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"226'4","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.149938+0000","last_change":"2024-04-06T13:47:21.722633+0000","last_active":"2024-04-06T13:47:22.149938+0000","last_peered":"2024-04-06T13:47:22.149938+0000","last_clean":"2024-04-06T13:47:22.149938+0000","last_became_active":"2024-04-06T13:47:21.722356+0000","last_became_peered":"2024-04-06T13:47:21.722356+0000","last_unstale":"2024-04-06T13:47:22.149938+0000","last_undegraded":"2024-04-06T13:47:22.149938+0000","last_fullsized":"2024-04-06T13:47:22.149938+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:55.001531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.226715+0000","last_change":"2024-04-06T13:47:22.136299+0000","last_active":"2024-04-06T13:47:23.226715+0000","last_peered":"2024-04-06T13:47:23.226715+0000","last_clean":"2024-04-06T13:47:23.226715+0000","last_became_active":"2024-04-06T13:47:22.136046+0000","last_became_peered":"2024-04-06T13:47:22.136046+0000","last_unstale":"2024-04-06T13:47:23.226715+0000","last_undegraded":"2024-04-06T13:47:23.226715+0000","last_fullsized":"2024-04-06T13:47:23.226715+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:19:51.727577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"226'1","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.142032+0000","last_change":"2024-04-06T13:47:21.721394+0000","last_active":"2024-04-06T13:47:22.142032+0000","last_peered":"2024-04-06T13:47:22.142032+0000","last_clean":"2024-04-06T13:47:22.142032+0000","last_became_active":"2024-04-06T13:47:21.721093+0000","last_became_peered":"2024-04-06T13:47:21.721093+0000","last_unstale":"2024-04-06T13:47:22.142032+0000","last_undegraded":"2024-04-06T13:47:22.142032+0000","last_fullsized":"2024-04-06T13:47:22.142032+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:30:58.296350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.133387+0000","last_change":"2024-04-06T13:47:22.139445+0000","last_active":"2024-04-06T13:47:23.133387+0000","last_peered":"2024-04-06T13:47:23.133387+0000","last_clean":"2024-04-06T13:47:23.133387+0000","last_became_active":"2024-04-06T13:47:22.138882+0000","last_became_peered":"2024-04-06T13:47:22.138882+0000","last_unstale":"2024-04-06T13:47:23.133387+0000","last_undegraded":"2024-04-06T13:47:23.133387+0000","last_fullsized":"2024-04-06T13:47:23.133387+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:45:57.790454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"226'2","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.146203+0000","last_change":"2024-04-06T13:47:21.714467+0000","last_active":"2024-04-06T13:47:22.146203+0000","last_peered":"2024-04-06T13:47:22.146203+0000","last_clean":"2024-04-06T13:47:22.146203+0000","last_became_active":"2024-04-06T13:47:21.714208+0000","last_became_peered":"2024-04-06T13:47:21.714208+0000","last_unstale":"2024-04-06T13:47:22.146203+0000","last_undegraded":"2024-04-06T13:47:22.146203+0000","last_fullsized":"2024-04-06T13:47:22.146203+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:09.423492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.227844+0000","last_change":"2024-04-06T13:47:22.146835+0000","last_active":"2024-04-06T13:47:23.227844+0000","last_peered":"2024-04-06T13:47:23.227844+0000","last_clean":"2024-04-06T13:47:23.227844+0000","last_became_active":"2024-04-06T13:47:22.146565+0000","last_became_peered":"2024-04-06T13:47:22.146565+0000","last_unstale":"2024-04-06T13:47:23.227844+0000","last_undegraded":"2024-04-06T13:47:23.227844+0000","last_fullsized":"2024-04-06T13:47:23.227844+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:10.026234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"226'2","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.149882+0000","last_change":"2024-04-06T13:47:21.713494+0000","last_active":"2024-04-06T13:47:22.149882+0000","last_peered":"2024-04-06T13:47:22.149882+0000","last_clean":"2024-04-06T13:47:22.149882+0000","last_became_active":"2024-04-06T13:47:21.713155+0000","last_became_peered":"2024-04-06T13:47:21.713155+0000","last_unstale":"2024-04-06T13:47:22.149882+0000","last_undegraded":"2024-04-06T13:47:22.149882+0000","last_fullsized":"2024-04-06T13:47:22.149882+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:43:51.273765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.227248+0000","last_change":"2024-04-06T13:47:22.139248+0000","last_active":"2024-04-06T13:47:23.227248+0000","last_peered":"2024-04-06T13:47:23.227248+0000","last_clean":"2024-04-06T13:47:23.227248+0000","last_became_active":"2024-04-06T13:47:22.139085+0000","last_became_peered":"2024-04-06T13:47:22.139085+0000","last_unstale":"2024-04-06T13:47:23.227248+0000","last_undegraded":"2024-04-06T13:47:23.227248+0000","last_fullsized":"2024-04-06T13:47:23.227248+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:41:53.649355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"226'1","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.142561+0000","last_change":"2024-04-06T13:47:21.714435+0000","last_active":"2024-04-06T13:47:22.142561+0000","last_peered":"2024-04-06T13:47:22.142561+0000","last_clean":"2024-04-06T13:47:22.142561+0000","last_became_active":"2024-04-06T13:47:21.714154+0000","last_became_peered":"2024-04-06T13:47:21.714154+0000","last_unstale":"2024-04-06T13:47:22.142561+0000","last_undegraded":"2024-04-06T13:47:22.142561+0000","last_fullsized":"2024-04-06T13:47:22.142561+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:25.144968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.227257+0000","last_change":"2024-04-06T13:47:22.141742+0000","last_active":"2024-04-06T13:47:23.227257+0000","last_peered":"2024-04-06T13:47:23.227257+0000","last_clean":"2024-04-06T13:47:23.227257+0000","last_became_active":"2024-04-06T13:47:22.141302+0000","last_became_peered":"2024-04-06T13:47:22.141302+0000","last_unstale":"2024-04-06T13:47:23.227257+0000","last_undegraded":"2024-04-06T13:47:23.227257+0000","last_fullsized":"2024-04-06T13:47:23.227257+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:24.049383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.121498+0000","last_change":"2024-04-06T13:47:21.713514+0000","last_active":"2024-04-06T13:47:22.121498+0000","last_peered":"2024-04-06T13:47:22.121498+0000","last_clean":"2024-04-06T13:47:22.121498+0000","last_became_active":"2024-04-06T13:47:21.713230+0000","last_became_peered":"2024-04-06T13:47:21.713230+0000","last_unstale":"2024-04-06T13:47:22.121498+0000","last_undegraded":"2024-04-06T13:47:22.121498+0000","last_fullsized":"2024-04-06T13:47:22.121498+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:29:38.052685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.130149+0000","last_change":"2024-04-06T13:47:22.140985+0000","last_active":"2024-04-06T13:47:23.130149+0000","last_peered":"2024-04-06T13:47:23.130149+0000","last_clean":"2024-04-06T13:47:23.130149+0000","last_became_active":"2024-04-06T13:47:22.140751+0000","last_became_peered":"2024-04-06T13:47:22.140751+0000","last_unstale":"2024-04-06T13:47:23.130149+0000","last_undegraded":"2024-04-06T13:47:23.130149+0000","last_fullsized":"2024-04-06T13:47:23.130149+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:20:03.901368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.125241+0000","last_change":"2024-04-06T13:47:21.715138+0000","last_active":"2024-04-06T13:47:22.125241+0000","last_peered":"2024-04-06T13:47:22.125241+0000","last_clean":"2024-04-06T13:47:22.125241+0000","last_became_active":"2024-04-06T13:47:21.714839+0000","last_became_peered":"2024-04-06T13:47:21.714839+0000","last_unstale":"2024-04-06T13:47:22.125241+0000","last_undegraded":"2024-04-06T13:47:22.125241+0000","last_fullsized":"2024-04-06T13:47:22.125241+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:02:18.580630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.226905+0000","last_change":"2024-04-06T13:47:22.132142+0000","last_active":"2024-04-06T13:47:23.226905+0000","last_peered":"2024-04-06T13:47:23.226905+0000","last_clean":"2024-04-06T13:47:23.226905+0000","last_became_active":"2024-04-06T13:47:22.131874+0000","last_became_peered":"2024-04-06T13:47:22.131874+0000","last_unstale":"2024-04-06T13:47:23.226905+0000","last_undegraded":"2024-04-06T13:47:23.226905+0000","last_fullsized":"2024-04-06T13:47:23.226905+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:14:26.082176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.126813+0000","last_change":"2024-04-06T13:47:21.722601+0000","last_active":"2024-04-06T13:47:22.126813+0000","last_peered":"2024-04-06T13:47:22.126813+0000","last_clean":"2024-04-06T13:47:22.126813+0000","last_became_active":"2024-04-06T13:47:21.722328+0000","last_became_peered":"2024-04-06T13:47:21.722328+0000","last_unstale":"2024-04-06T13:47:22.126813+0000","last_undegraded":"2024-04-06T13:47:22.126813+0000","last_fullsized":"2024-04-06T13:47:22.126813+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:30:35.611598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.226611+0000","last_change":"2024-04-06T13:47:22.136160+0000","last_active":"2024-04-06T13:47:23.226611+0000","last_peered":"2024-04-06T13:47:23.226611+0000","last_clean":"2024-04-06T13:47:23.226611+0000","last_became_active":"2024-04-06T13:47:22.135869+0000","last_became_peered":"2024-04-06T13:47:22.135869+0000","last_unstale":"2024-04-06T13:47:23.226611+0000","last_undegraded":"2024-04-06T13:47:23.226611+0000","last_fullsized":"2024-04-06T13:47:23.226611+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:05:17.896292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.124941+0000","last_change":"2024-04-06T13:47:21.722544+0000","last_active":"2024-04-06T13:47:22.124941+0000","last_peered":"2024-04-06T13:47:22.124941+0000","last_clean":"2024-04-06T13:47:22.124941+0000","last_became_active":"2024-04-06T13:47:21.722242+0000","last_became_peered":"2024-04-06T13:47:21.722242+0000","last_unstale":"2024-04-06T13:47:22.124941+0000","last_undegraded":"2024-04-06T13:47:22.124941+0000","last_fullsized":"2024-04-06T13:47:22.124941+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:14:53.056289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.129605+0000","last_change":"2024-04-06T13:47:22.145873+0000","last_active":"2024-04-06T13:47:23.129605+0000","last_peered":"2024-04-06T13:47:23.129605+0000","last_clean":"2024-04-06T13:47:23.129605+0000","last_became_active":"2024-04-06T13:47:22.145602+0000","last_became_peered":"2024-04-06T13:47:22.145602+0000","last_unstale":"2024-04-06T13:47:23.129605+0000","last_undegraded":"2024-04-06T13:47:23.129605+0000","last_fullsized":"2024-04-06T13:47:23.129605+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:24:02.415549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"226'3","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.150417+0000","last_change":"2024-04-06T13:47:21.714316+0000","last_active":"2024-04-06T13:47:22.150417+0000","last_peered":"2024-04-06T13:47:22.150417+0000","last_clean":"2024-04-06T13:47:22.150417+0000","last_became_active":"2024-04-06T13:47:21.713975+0000","last_became_peered":"2024-04-06T13:47:21.713975+0000","last_unstale":"2024-04-06T13:47:22.150417+0000","last_undegraded":"2024-04-06T13:47:22.150417+0000","last_fullsized":"2024-04-06T13:47:22.150417+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:33:33.280270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.227762+0000","last_change":"2024-04-06T13:47:22.144290+0000","last_active":"2024-04-06T13:47:23.227762+0000","last_peered":"2024-04-06T13:47:23.227762+0000","last_clean":"2024-04-06T13:47:23.227762+0000","last_became_active":"2024-04-06T13:47:22.144023+0000","last_became_peered":"2024-04-06T13:47:22.144023+0000","last_unstale":"2024-04-06T13:47:23.227762+0000","last_undegraded":"2024-04-06T13:47:23.227762+0000","last_fullsized":"2024-04-06T13:47:23.227762+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:52:25.950818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"226'6","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.152806+0000","last_change":"2024-04-06T13:47:21.722629+0000","last_active":"2024-04-06T13:47:22.152806+0000","last_peered":"2024-04-06T13:47:22.152806+0000","last_clean":"2024-04-06T13:47:22.152806+0000","last_became_active":"2024-04-06T13:47:21.722378+0000","last_became_peered":"2024-04-06T13:47:21.722378+0000","last_unstale":"2024-04-06T13:47:22.152806+0000","last_undegraded":"2024-04-06T13:47:22.152806+0000","last_fullsized":"2024-04-06T13:47:22.152806+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:43:36.814977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.227798+0000","last_change":"2024-04-06T13:47:22.143097+0000","last_active":"2024-04-06T13:47:23.227798+0000","last_peered":"2024-04-06T13:47:23.227798+0000","last_clean":"2024-04-06T13:47:23.227798+0000","last_became_active":"2024-04-06T13:47:22.141937+0000","last_became_peered":"2024-04-06T13:47:22.141937+0000","last_unstale":"2024-04-06T13:47:23.227798+0000","last_undegraded":"2024-04-06T13:47:23.227798+0000","last_fullsized":"2024-04-06T13:47:23.227798+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:36:26.092264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"226'2","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.144030+0000","last_change":"2024-04-06T13:47:21.719461+0000","last_active":"2024-04-06T13:47:22.144030+0000","last_peered":"2024-04-06T13:47:22.144030+0000","last_clean":"2024-04-06T13:47:22.144030+0000","last_became_active":"2024-04-06T13:47:21.719215+0000","last_became_peered":"2024-04-06T13:47:21.719215+0000","last_unstale":"2024-04-06T13:47:22.144030+0000","last_undegraded":"2024-04-06T13:47:22.144030+0000","last_fullsized":"2024-04-06T13:47:22.144030+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:00:20.959541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.226951+0000","last_change":"2024-04-06T13:47:22.137019+0000","last_active":"2024-04-06T13:47:23.226951+0000","last_peered":"2024-04-06T13:47:23.226951+0000","last_clean":"2024-04-06T13:47:23.226951+0000","last_became_active":"2024-04-06T13:47:22.136423+0000","last_became_peered":"2024-04-06T13:47:22.136423+0000","last_unstale":"2024-04-06T13:47:23.226951+0000","last_undegraded":"2024-04-06T13:47:23.226951+0000","last_fullsized":"2024-04-06T13:47:23.226951+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:32:52.203373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.125740+0000","last_change":"2024-04-06T13:47:21.709366+0000","last_active":"2024-04-06T13:47:22.125740+0000","last_peered":"2024-04-06T13:47:22.125740+0000","last_clean":"2024-04-06T13:47:22.125740+0000","last_became_active":"2024-04-06T13:47:21.709118+0000","last_became_peered":"2024-04-06T13:47:21.709118+0000","last_unstale":"2024-04-06T13:47:22.125740+0000","last_undegraded":"2024-04-06T13:47:22.125740+0000","last_fullsized":"2024-04-06T13:47:22.125740+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:15:13.775616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.129974+0000","last_change":"2024-04-06T13:47:22.140481+0000","last_active":"2024-04-06T13:47:23.129974+0000","last_peered":"2024-04-06T13:47:23.129974+0000","last_clean":"2024-04-06T13:47:23.129974+0000","last_became_active":"2024-04-06T13:47:22.140216+0000","last_became_peered":"2024-04-06T13:47:22.140216+0000","last_unstale":"2024-04-06T13:47:23.129974+0000","last_undegraded":"2024-04-06T13:47:23.129974+0000","last_fullsized":"2024-04-06T13:47:23.129974+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:10.425285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.125916+0000","last_change":"2024-04-06T13:47:21.722302+0000","last_active":"2024-04-06T13:47:22.125916+0000","last_peered":"2024-04-06T13:47:22.125916+0000","last_clean":"2024-04-06T13:47:22.125916+0000","last_became_active":"2024-04-06T13:47:21.721839+0000","last_became_peered":"2024-04-06T13:47:21.721839+0000","last_unstale":"2024-04-06T13:47:22.125916+0000","last_undegraded":"2024-04-06T13:47:22.125916+0000","last_fullsized":"2024-04-06T13:47:22.125916+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:27:30.927130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.133889+0000","last_change":"2024-04-06T13:47:22.137131+0000","last_active":"2024-04-06T13:47:23.133889+0000","last_peered":"2024-04-06T13:47:23.133889+0000","last_clean":"2024-04-06T13:47:23.133889+0000","last_became_active":"2024-04-06T13:47:22.136845+0000","last_became_peered":"2024-04-06T13:47:22.136845+0000","last_unstale":"2024-04-06T13:47:23.133889+0000","last_undegraded":"2024-04-06T13:47:23.133889+0000","last_fullsized":"2024-04-06T13:47:23.133889+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:58:56.639338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"226'8","reported_seq":19,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.150561+0000","last_change":"2024-04-06T13:47:21.721909+0000","last_active":"2024-04-06T13:47:22.150561+0000","last_peered":"2024-04-06T13:47:22.150561+0000","last_clean":"2024-04-06T13:47:22.150561+0000","last_became_active":"2024-04-06T13:47:21.721654+0000","last_became_peered":"2024-04-06T13:47:21.721654+0000","last_unstale":"2024-04-06T13:47:22.150561+0000","last_undegraded":"2024-04-06T13:47:22.150561+0000","last_fullsized":"2024-04-06T13:47:22.150561+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:06:40.698606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.132883+0000","last_change":"2024-04-06T13:47:22.147277+0000","last_active":"2024-04-06T13:47:23.132883+0000","last_peered":"2024-04-06T13:47:23.132883+0000","last_clean":"2024-04-06T13:47:23.132883+0000","last_became_active":"2024-04-06T13:47:22.146965+0000","last_became_peered":"2024-04-06T13:47:22.146965+0000","last_unstale":"2024-04-06T13:47:23.132883+0000","last_undegraded":"2024-04-06T13:47:23.132883+0000","last_fullsized":"2024-04-06T13:47:23.132883+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:30:03.035386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"226'1","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.140230+0000","last_change":"2024-04-06T13:47:21.722333+0000","last_active":"2024-04-06T13:47:22.140230+0000","last_peered":"2024-04-06T13:47:22.140230+0000","last_clean":"2024-04-06T13:47:22.140230+0000","last_became_active":"2024-04-06T13:47:21.722021+0000","last_became_peered":"2024-04-06T13:47:21.722021+0000","last_unstale":"2024-04-06T13:47:22.140230+0000","last_undegraded":"2024-04-06T13:47:22.140230+0000","last_fullsized":"2024-04-06T13:47:22.140230+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:54:05.540582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.226603+0000","last_change":"2024-04-06T13:47:22.135011+0000","last_active":"2024-04-06T13:47:23.226603+0000","last_peered":"2024-04-06T13:47:23.226603+0000","last_clean":"2024-04-06T13:47:23.226603+0000","last_became_active":"2024-04-06T13:47:22.134193+0000","last_became_peered":"2024-04-06T13:47:22.134193+0000","last_unstale":"2024-04-06T13:47:23.226603+0000","last_undegraded":"2024-04-06T13:47:23.226603+0000","last_fullsized":"2024-04-06T13:47:23.226603+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:15:14.777409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"226'2","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.148391+0000","last_change":"2024-04-06T13:47:21.723380+0000","last_active":"2024-04-06T13:47:22.148391+0000","last_peered":"2024-04-06T13:47:22.148391+0000","last_clean":"2024-04-06T13:47:22.148391+0000","last_became_active":"2024-04-06T13:47:21.723089+0000","last_became_peered":"2024-04-06T13:47:21.723089+0000","last_unstale":"2024-04-06T13:47:22.148391+0000","last_undegraded":"2024-04-06T13:47:22.148391+0000","last_fullsized":"2024-04-06T13:47:22.148391+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:11:25.850855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.226941+0000","last_change":"2024-04-06T13:47:22.143187+0000","last_active":"2024-04-06T13:47:23.226941+0000","last_peered":"2024-04-06T13:47:23.226941+0000","last_clean":"2024-04-06T13:47:23.226941+0000","last_became_active":"2024-04-06T13:47:22.142878+0000","last_became_peered":"2024-04-06T13:47:22.142878+0000","last_unstale":"2024-04-06T13:47:23.226941+0000","last_undegraded":"2024-04-06T13:47:23.226941+0000","last_fullsized":"2024-04-06T13:47:23.226941+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:27:49.346320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"226'1","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.151540+0000","last_change":"2024-04-06T13:47:21.714087+0000","last_active":"2024-04-06T13:47:22.151540+0000","last_peered":"2024-04-06T13:47:22.151540+0000","last_clean":"2024-04-06T13:47:22.151540+0000","last_became_active":"2024-04-06T13:47:21.713790+0000","last_became_peered":"2024-04-06T13:47:21.713790+0000","last_unstale":"2024-04-06T13:47:22.151540+0000","last_undegraded":"2024-04-06T13:47:22.151540+0000","last_fullsized":"2024-04-06T13:47:22.151540+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:52:28.469203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.133302+0000","last_change":"2024-04-06T13:47:22.139250+0000","last_active":"2024-04-06T13:47:23.133302+0000","last_peered":"2024-04-06T13:47:23.133302+0000","last_clean":"2024-04-06T13:47:23.133302+0000","last_became_active":"2024-04-06T13:47:22.138989+0000","last_became_peered":"2024-04-06T13:47:22.138989+0000","last_unstale":"2024-04-06T13:47:23.133302+0000","last_undegraded":"2024-04-06T13:47:23.133302+0000","last_fullsized":"2024-04-06T13:47:23.133302+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:48:14.103260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.128885+0000","last_change":"2024-04-06T13:47:21.721929+0000","last_active":"2024-04-06T13:47:22.128885+0000","last_peered":"2024-04-06T13:47:22.128885+0000","last_clean":"2024-04-06T13:47:22.128885+0000","last_became_active":"2024-04-06T13:47:21.721667+0000","last_became_peered":"2024-04-06T13:47:21.721667+0000","last_unstale":"2024-04-06T13:47:22.128885+0000","last_undegraded":"2024-04-06T13:47:22.128885+0000","last_fullsized":"2024-04-06T13:47:22.128885+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:49:15.850828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.227889+0000","last_change":"2024-04-06T13:47:22.143638+0000","last_active":"2024-04-06T13:47:23.227889+0000","last_peered":"2024-04-06T13:47:23.227889+0000","last_clean":"2024-04-06T13:47:23.227889+0000","last_became_active":"2024-04-06T13:47:22.143386+0000","last_became_peered":"2024-04-06T13:47:22.143386+0000","last_unstale":"2024-04-06T13:47:23.227889+0000","last_undegraded":"2024-04-06T13:47:23.227889+0000","last_fullsized":"2024-04-06T13:47:23.227889+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:16:28.885120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"226'2","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.139269+0000","last_change":"2024-04-06T13:47:21.709408+0000","last_active":"2024-04-06T13:47:22.139269+0000","last_peered":"2024-04-06T13:47:22.139269+0000","last_clean":"2024-04-06T13:47:22.139269+0000","last_became_active":"2024-04-06T13:47:21.708906+0000","last_became_peered":"2024-04-06T13:47:21.708906+0000","last_unstale":"2024-04-06T13:47:22.139269+0000","last_undegraded":"2024-04-06T13:47:22.139269+0000","last_fullsized":"2024-04-06T13:47:22.139269+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:09.996520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.133868+0000","last_change":"2024-04-06T13:47:22.141810+0000","last_active":"2024-04-06T13:47:23.133868+0000","last_peered":"2024-04-06T13:47:23.133868+0000","last_clean":"2024-04-06T13:47:23.133868+0000","last_became_active":"2024-04-06T13:47:22.141394+0000","last_became_peered":"2024-04-06T13:47:22.141394+0000","last_unstale":"2024-04-06T13:47:23.133868+0000","last_undegraded":"2024-04-06T13:47:23.133868+0000","last_fullsized":"2024-04-06T13:47:23.133868+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:58:43.501343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.137553+0000","last_change":"2024-04-06T13:47:21.712418+0000","last_active":"2024-04-06T13:47:22.137553+0000","last_peered":"2024-04-06T13:47:22.137553+0000","last_clean":"2024-04-06T13:47:22.137553+0000","last_became_active":"2024-04-06T13:47:21.712149+0000","last_became_peered":"2024-04-06T13:47:21.712149+0000","last_unstale":"2024-04-06T13:47:22.137553+0000","last_undegraded":"2024-04-06T13:47:22.137553+0000","last_fullsized":"2024-04-06T13:47:22.137553+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:10:24.669252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.130038+0000","last_change":"2024-04-06T13:47:22.138293+0000","last_active":"2024-04-06T13:47:23.130038+0000","last_peered":"2024-04-06T13:47:23.130038+0000","last_clean":"2024-04-06T13:47:23.130038+0000","last_became_active":"2024-04-06T13:47:22.138051+0000","last_became_peered":"2024-04-06T13:47:22.138051+0000","last_unstale":"2024-04-06T13:47:23.130038+0000","last_undegraded":"2024-04-06T13:47:23.130038+0000","last_fullsized":"2024-04-06T13:47:23.130038+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:52:29.470996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.128039+0000","last_change":"2024-04-06T13:47:21.720187+0000","last_active":"2024-04-06T13:47:22.128039+0000","last_peered":"2024-04-06T13:47:22.128039+0000","last_clean":"2024-04-06T13:47:22.128039+0000","last_became_active":"2024-04-06T13:47:21.719896+0000","last_became_peered":"2024-04-06T13:47:21.719896+0000","last_unstale":"2024-04-06T13:47:22.128039+0000","last_undegraded":"2024-04-06T13:47:22.128039+0000","last_fullsized":"2024-04-06T13:47:22.128039+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:11:25.850855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.133577+0000","last_change":"2024-04-06T13:47:22.141028+0000","last_active":"2024-04-06T13:47:23.133577+0000","last_peered":"2024-04-06T13:47:23.133577+0000","last_clean":"2024-04-06T13:47:23.133577+0000","last_became_active":"2024-04-06T13:47:22.140749+0000","last_became_peered":"2024-04-06T13:47:22.140749+0000","last_unstale":"2024-04-06T13:47:23.133577+0000","last_undegraded":"2024-04-06T13:47:23.133577+0000","last_fullsized":"2024-04-06T13:47:23.133577+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:06:22.598094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.129283+0000","last_change":"2024-04-06T13:47:21.718440+0000","last_active":"2024-04-06T13:47:22.129283+0000","last_peered":"2024-04-06T13:47:22.129283+0000","last_clean":"2024-04-06T13:47:22.129283+0000","last_became_active":"2024-04-06T13:47:21.718175+0000","last_became_peered":"2024-04-06T13:47:21.718175+0000","last_unstale":"2024-04-06T13:47:22.129283+0000","last_undegraded":"2024-04-06T13:47:22.129283+0000","last_fullsized":"2024-04-06T13:47:22.129283+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:34:40.754044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.132782+0000","last_change":"2024-04-06T13:47:22.147695+0000","last_active":"2024-04-06T13:47:23.132782+0000","last_peered":"2024-04-06T13:47:23.132782+0000","last_clean":"2024-04-06T13:47:23.132782+0000","last_became_active":"2024-04-06T13:47:22.147127+0000","last_became_peered":"2024-04-06T13:47:22.147127+0000","last_unstale":"2024-04-06T13:47:23.132782+0000","last_undegraded":"2024-04-06T13:47:23.132782+0000","last_fullsized":"2024-04-06T13:47:23.132782+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:03:24.991063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"226'2","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.150356+0000","last_change":"2024-04-06T13:47:21.719079+0000","last_active":"2024-04-06T13:47:22.150356+0000","last_peered":"2024-04-06T13:47:22.150356+0000","last_clean":"2024-04-06T13:47:22.150356+0000","last_became_active":"2024-04-06T13:47:21.718773+0000","last_became_peered":"2024-04-06T13:47:21.718773+0000","last_unstale":"2024-04-06T13:47:22.150356+0000","last_undegraded":"2024-04-06T13:47:22.150356+0000","last_fullsized":"2024-04-06T13:47:22.150356+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:45:09.803960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.226704+0000","last_change":"2024-04-06T13:47:22.141257+0000","last_active":"2024-04-06T13:47:23.226704+0000","last_peered":"2024-04-06T13:47:23.226704+0000","last_clean":"2024-04-06T13:47:23.226704+0000","last_became_active":"2024-04-06T13:47:22.141005+0000","last_became_peered":"2024-04-06T13:47:22.141005+0000","last_unstale":"2024-04-06T13:47:23.226704+0000","last_undegraded":"2024-04-06T13:47:23.226704+0000","last_fullsized":"2024-04-06T13:47:23.226704+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:15.316599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"226'1","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.755176+0000","last_change":"2024-04-06T13:47:21.725237+0000","last_active":"2024-04-06T13:47:22.755176+0000","last_peered":"2024-04-06T13:47:22.755176+0000","last_clean":"2024-04-06T13:47:22.755176+0000","last_became_active":"2024-04-06T13:47:21.724977+0000","last_became_peered":"2024-04-06T13:47:21.724977+0000","last_unstale":"2024-04-06T13:47:22.755176+0000","last_undegraded":"2024-04-06T13:47:22.755176+0000","last_fullsized":"2024-04-06T13:47:22.755176+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:13:14.840162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.129649+0000","last_change":"2024-04-06T13:47:22.136494+0000","last_active":"2024-04-06T13:47:23.129649+0000","last_peered":"2024-04-06T13:47:23.129649+0000","last_clean":"2024-04-06T13:47:23.129649+0000","last_became_active":"2024-04-06T13:47:22.136225+0000","last_became_peered":"2024-04-06T13:47:22.136225+0000","last_unstale":"2024-04-06T13:47:23.129649+0000","last_undegraded":"2024-04-06T13:47:23.129649+0000","last_fullsized":"2024-04-06T13:47:23.129649+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:00:21.961334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.127615+0000","last_change":"2024-04-06T13:47:21.711878+0000","last_active":"2024-04-06T13:47:22.127615+0000","last_peered":"2024-04-06T13:47:22.127615+0000","last_clean":"2024-04-06T13:47:22.127615+0000","last_became_active":"2024-04-06T13:47:21.711481+0000","last_became_peered":"2024-04-06T13:47:21.711481+0000","last_unstale":"2024-04-06T13:47:22.127615+0000","last_undegraded":"2024-04-06T13:47:22.127615+0000","last_fullsized":"2024-04-06T13:47:22.127615+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:34:40.754044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.226855+0000","last_change":"2024-04-06T13:47:22.138308+0000","last_active":"2024-04-06T13:47:23.226855+0000","last_peered":"2024-04-06T13:47:23.226855+0000","last_clean":"2024-04-06T13:47:23.226855+0000","last_became_active":"2024-04-06T13:47:22.138038+0000","last_became_peered":"2024-04-06T13:47:22.138038+0000","last_unstale":"2024-04-06T13:47:23.226855+0000","last_undegraded":"2024-04-06T13:47:23.226855+0000","last_fullsized":"2024-04-06T13:47:23.226855+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:23:02.601223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.129478+0000","last_change":"2024-04-06T13:47:21.721811+0000","last_active":"2024-04-06T13:47:22.129478+0000","last_peered":"2024-04-06T13:47:22.129478+0000","last_clean":"2024-04-06T13:47:22.129478+0000","last_became_active":"2024-04-06T13:47:21.721528+0000","last_became_peered":"2024-04-06T13:47:21.721528+0000","last_unstale":"2024-04-06T13:47:22.129478+0000","last_undegraded":"2024-04-06T13:47:22.129478+0000","last_fullsized":"2024-04-06T13:47:22.129478+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:54:07.189978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.133263+0000","last_change":"2024-04-06T13:47:22.138725+0000","last_active":"2024-04-06T13:47:23.133263+0000","last_peered":"2024-04-06T13:47:23.133263+0000","last_clean":"2024-04-06T13:47:23.133263+0000","last_became_active":"2024-04-06T13:47:22.131665+0000","last_became_peered":"2024-04-06T13:47:22.131665+0000","last_unstale":"2024-04-06T13:47:23.133263+0000","last_undegraded":"2024-04-06T13:47:23.133263+0000","last_fullsized":"2024-04-06T13:47:23.133263+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:58:16.851577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.125072+0000","last_change":"2024-04-06T13:47:21.721538+0000","last_active":"2024-04-06T13:47:22.125072+0000","last_peered":"2024-04-06T13:47:22.125072+0000","last_clean":"2024-04-06T13:47:22.125072+0000","last_became_active":"2024-04-06T13:47:21.721255+0000","last_became_peered":"2024-04-06T13:47:21.721255+0000","last_unstale":"2024-04-06T13:47:22.125072+0000","last_undegraded":"2024-04-06T13:47:22.125072+0000","last_fullsized":"2024-04-06T13:47:22.125072+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:53:48.795224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.133873+0000","last_change":"2024-04-06T13:47:22.151110+0000","last_active":"2024-04-06T13:47:23.133873+0000","last_peered":"2024-04-06T13:47:23.133873+0000","last_clean":"2024-04-06T13:47:23.133873+0000","last_became_active":"2024-04-06T13:47:22.150885+0000","last_became_peered":"2024-04-06T13:47:22.150885+0000","last_unstale":"2024-04-06T13:47:23.133873+0000","last_undegraded":"2024-04-06T13:47:23.133873+0000","last_fullsized":"2024-04-06T13:47:23.133873+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:43:52.275558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.127025+0000","last_change":"2024-04-06T13:47:21.711931+0000","last_active":"2024-04-06T13:47:22.127025+0000","last_peered":"2024-04-06T13:47:22.127025+0000","last_clean":"2024-04-06T13:47:22.127025+0000","last_became_active":"2024-04-06T13:47:21.711592+0000","last_became_peered":"2024-04-06T13:47:21.711592+0000","last_unstale":"2024-04-06T13:47:22.127025+0000","last_undegraded":"2024-04-06T13:47:22.127025+0000","last_fullsized":"2024-04-06T13:47:22.127025+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:38:23.226877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.133941+0000","last_change":"2024-04-06T13:47:22.150739+0000","last_active":"2024-04-06T13:47:23.133941+0000","last_peered":"2024-04-06T13:47:23.133941+0000","last_clean":"2024-04-06T13:47:23.133941+0000","last_became_active":"2024-04-06T13:47:22.150305+0000","last_became_peered":"2024-04-06T13:47:22.150305+0000","last_unstale":"2024-04-06T13:47:23.133941+0000","last_undegraded":"2024-04-06T13:47:23.133941+0000","last_fullsized":"2024-04-06T13:47:23.133941+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:18:06.419840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.132223+0000","last_change":"2024-04-06T13:47:21.720234+0000","last_active":"2024-04-06T13:47:22.132223+0000","last_peered":"2024-04-06T13:47:22.132223+0000","last_clean":"2024-04-06T13:47:22.132223+0000","last_became_active":"2024-04-06T13:47:21.719765+0000","last_became_peered":"2024-04-06T13:47:21.719765+0000","last_unstale":"2024-04-06T13:47:22.132223+0000","last_undegraded":"2024-04-06T13:47:22.132223+0000","last_fullsized":"2024-04-06T13:47:22.132223+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:55.001531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.227244+0000","last_change":"2024-04-06T13:47:22.145029+0000","last_active":"2024-04-06T13:47:23.227244+0000","last_peered":"2024-04-06T13:47:23.227244+0000","last_clean":"2024-04-06T13:47:23.227244+0000","last_became_active":"2024-04-06T13:47:22.144765+0000","last_became_peered":"2024-04-06T13:47:22.144765+0000","last_unstale":"2024-04-06T13:47:23.227244+0000","last_undegraded":"2024-04-06T13:47:23.227244+0000","last_fullsized":"2024-04-06T13:47:23.227244+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:13:00.983141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.129040+0000","last_change":"2024-04-06T13:47:21.721753+0000","last_active":"2024-04-06T13:47:22.129040+0000","last_peered":"2024-04-06T13:47:22.129040+0000","last_clean":"2024-04-06T13:47:22.129040+0000","last_became_active":"2024-04-06T13:47:21.721452+0000","last_became_peered":"2024-04-06T13:47:21.721452+0000","last_unstale":"2024-04-06T13:47:22.129040+0000","last_undegraded":"2024-04-06T13:47:22.129040+0000","last_fullsized":"2024-04-06T13:47:22.129040+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:56:43.873348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-06T13:47:23.227053+0000","last_change":"2024-04-06T13:47:22.146992+0000","last_active":"2024-04-06T13:47:23.227053+0000","last_peered":"2024-04-06T13:47:23.227053+0000","last_clean":"2024-04-06T13:47:23.227053+0000","last_became_active":"2024-04-06T13:47:22.146728+0000","last_became_peered":"2024-04-06T13:47:22.146728+0000","last_unstale":"2024-04-06T13:47:23.227053+0000","last_undegraded":"2024-04-06T13:47:23.227053+0000","last_fullsized":"2024-04-06T13:47:23.227053+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:12:44.053539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"226'1","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.150058+0000","last_change":"2024-04-06T13:47:21.714987+0000","last_active":"2024-04-06T13:47:22.150058+0000","last_peered":"2024-04-06T13:47:22.150058+0000","last_clean":"2024-04-06T13:47:22.150058+0000","last_became_active":"2024-04-06T13:47:21.713909+0000","last_became_peered":"2024-04-06T13:47:21.713909+0000","last_unstale":"2024-04-06T13:47:22.150058+0000","last_undegraded":"2024-04-06T13:47:22.150058+0000","last_fullsized":"2024-04-06T13:47:22.150058+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:55:29.107060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"216'348","reported_seq":1237,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-06T13:47:22.121819+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:47:22.121819+0000","last_peered":"2024-04-06T13:47:22.121819+0000","last_clean":"2024-04-06T13:47:22.121819+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:47:22.121819+0000","last_undegraded":"2024-04-06T13:47:22.121819+0000","last_fullsized":"2024-04-06T13:47:22.121819+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575983,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56644,"kb_used_data":3380,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315196,"statfs":{"total":96636764160,"available":96578760704,"internally_reserved":0,"allocated":3461120,"data_stored":2046889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.751,"5min":0.78400000000000003,"15min":0.84899999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58499999999999996,"15min":0.59799999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76000000000000001,"5min":0.80800000000000005,"15min":0.82899999999999996},"last":0.49299999999999999}]},{"osd":1,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.57099999999999995,"15min":0.58799999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.66600000000000004,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.432},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59799999999999998,"15min":0.60499999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38900000000000001},"max":{"1min":0.752,"5min":0.83399999999999996,"15min":0.85299999999999998},"last":0.40899999999999997}]},{"osd":2,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.57899999999999996,"15min":0.59499999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67700000000000005,"5min":0.81699999999999995,"15min":0.83899999999999997},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.60299999999999998,"15min":0.60599999999999998},"min":{"1min":0.47199999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73699999999999999,"5min":0.79700000000000004,"15min":0.81699999999999995},"last":0.40100000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.61599999999999999,"15min":0.61499999999999999},"min":{"1min":0.437,"5min":0.41199999999999998,"15min":0.38500000000000001},"max":{"1min":0.75600000000000001,"5min":0.81100000000000005,"15min":0.86199999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59799999999999998,"15min":0.60899999999999999},"min":{"1min":0.42399999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.70199999999999996,"5min":0.83299999999999996,"15min":0.84099999999999997},"last":0.47699999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.45900000000000002,"15min":0.46400000000000002},"min":{"1min":0.33300000000000002,"5min":0.307,"15min":0.30299999999999999},"max":{"1min":0.628,"5min":0.76000000000000001,"15min":0.81599999999999995},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.54600000000000004,"15min":0.54300000000000004},"min":{"1min":0.39300000000000002,"5min":0.38700000000000001,"15min":0.36799999999999999},"max":{"1min":0.68700000000000006,"5min":0.74299999999999999,"15min":0.82699999999999996},"last":0.38200000000000001}]},{"osd":5,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.49299999999999999,"15min":0.495},"min":{"1min":0.36099999999999999,"5min":0.318,"15min":0.308},"max":{"1min":0.71999999999999997,"5min":0.77100000000000002,"15min":0.78300000000000003},"last":0.315},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.46899999999999997,"5min":0.39200000000000002,"15min":0.33600000000000002},"max":{"1min":0.69799999999999995,"5min":0.82099999999999995,"15min":0.85399999999999998},"last":0.46100000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.39600000000000002,"5min":0.374,"15min":0.34899999999999998},"max":{"1min":0.67700000000000005,"5min":0.755,"15min":0.879},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56899999999999995,"15min":0.58699999999999997},"min":{"1min":0.39200000000000002,"5min":0.378,"15min":0.34799999999999998},"max":{"1min":0.73299999999999998,"5min":0.84899999999999998,"15min":0.89800000000000002},"last":0.33500000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575982,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38012,"kb_used_data":2796,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94333828,"statfs":{"total":96636764160,"available":96597839872,"internally_reserved":0,"allocated":2863104,"data_stored":1456014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.45300000000000001,"5min":0.38200000000000001,"15min":0.32000000000000001},"max":{"1min":0.74399999999999999,"5min":0.79700000000000004,"15min":0.83799999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.48199999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76900000000000002,"5min":0.80000000000000004,"15min":0.81999999999999995},"last":0.47399999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.626,"15min":0.63},"min":{"1min":0.48999999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.83699999999999997,"15min":0.85799999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64200000000000002,"15min":0.65000000000000002},"min":{"1min":0.54600000000000004,"5min":0.45100000000000001,"15min":0.378},"max":{"1min":0.78900000000000003,"5min":0.81399999999999995,"15min":0.82699999999999996},"last":0.48999999999999999}]},{"osd":2,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.495,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.76700000000000002,"5min":0.82299999999999995,"15min":0.82899999999999996},"last":0.377},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.59299999999999997},"min":{"1min":0.46400000000000002,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.755,"5min":0.78600000000000003,"15min":0.80700000000000005},"last":0.45800000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77500000000000002,"5min":0.80100000000000005,"15min":0.82399999999999995},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64600000000000002,"15min":0.65000000000000002},"min":{"1min":0.51500000000000001,"5min":0.45900000000000002,"15min":0.36099999999999999},"max":{"1min":0.78800000000000003,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.499}]},{"osd":4,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54500000000000004,"15min":0.53100000000000003},"min":{"1min":0.38100000000000001,"5min":0.36899999999999999,"15min":0.33800000000000002},"max":{"1min":0.72199999999999998,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53500000000000003,"15min":0.54500000000000004},"min":{"1min":0.371,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68799999999999994,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.39400000000000002}]},{"osd":5,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.504,"15min":0.50900000000000001},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":0.60799999999999998,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.439},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53100000000000003,"15min":0.53400000000000003},"min":{"1min":0.35499999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74099999999999999,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.52500000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71999999999999997,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56699999999999995,"15min":0.57199999999999995},"min":{"1min":0.435,"5min":0.36799999999999999,"15min":0.32800000000000001},"max":{"1min":0.70099999999999996,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.40899999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575981,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60136,"kb_used_data":2780,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311704,"statfs":{"total":96636764160,"available":96575184896,"internally_reserved":0,"allocated":2846720,"data_stored":1453868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.248},{"interface":"front","average":{"1min":0.443,"5min":0.432,"15min":0.432},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25700000000000001},"max":{"1min":0.55300000000000005,"5min":0.60999999999999999,"15min":0.69399999999999995},"last":0.35199999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.42999999999999999,"15min":0.42899999999999999},"min":{"1min":0.30299999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46800000000000003,"15min":0.46899999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.29399999999999998},"max":{"1min":0.69499999999999995,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.32100000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.28799999999999998},"max":{"1min":0.67800000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.26300000000000001},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.66100000000000003},"last":0.308}]},{"osd":4,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.64000000000000001,"15min":0.64900000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.33800000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.97399999999999998},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.65600000000000003,"15min":0.66000000000000003},"min":{"1min":0.45000000000000001,"5min":0.44500000000000001,"15min":0.44400000000000001},"max":{"1min":0.877,"5min":0.94599999999999995,"15min":0.995},"last":0.61399999999999999}]},{"osd":5,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.65000000000000002,"15min":0.65600000000000003},"min":{"1min":0.51300000000000001,"5min":0.41299999999999998,"15min":0.40799999999999997},"max":{"1min":0.91800000000000004,"5min":0.96899999999999997,"15min":1.073},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.66800000000000004,"15min":0.67500000000000004},"min":{"1min":0.45500000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.90500000000000003,"5min":0.96499999999999997,"15min":1.0209999999999999},"last":0.65300000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65100000000000002,"15min":0.66200000000000003},"min":{"1min":0.47699999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":1},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.51400000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":1.036},"last":0.63900000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.69799999999999995,"15min":0.69999999999999996},"min":{"1min":0.50600000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89600000000000002,"5min":0.94899999999999995,"15min":1.0649999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.66500000000000004,"15min":0.67300000000000004},"min":{"1min":0.49299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":1.056},"last":0.59399999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575981,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42680,"kb_used_data":3364,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329160,"statfs":{"total":96636764160,"available":96593059840,"internally_reserved":0,"allocated":3444736,"data_stored":2044326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.40200000000000002,"15min":0.39900000000000002},"min":{"1min":0.315,"5min":0.24299999999999999,"15min":0.23799999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.41499999999999998,"15min":0.42299999999999999},"min":{"1min":0.32400000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55800000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.28000000000000003}]},{"osd":2,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.42299999999999999,"15min":0.432},"min":{"1min":0.32100000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.55900000000000005,"5min":0.67900000000000005,"15min":0.70099999999999996},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.441,"15min":0.45200000000000001},"min":{"1min":0.35899999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60799999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.36399999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.42999999999999999,"15min":0.434},"min":{"1min":0.34399999999999997,"5min":0.27400000000000002,"15min":0.26800000000000002},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.48299999999999998,"15min":0.48599999999999999},"min":{"1min":0.39700000000000002,"5min":0.314,"15min":0.307},"max":{"1min":0.64100000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.39000000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.63100000000000001,"15min":0.64800000000000002},"min":{"1min":0.48299999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":1.0329999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.63900000000000001,"15min":0.65100000000000002},"min":{"1min":0.504,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.75}]},{"osd":5,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.628,"15min":0.64800000000000002},"min":{"1min":0.5,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.87,"5min":0.876,"15min":1.0009999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.66300000000000003,"15min":0.67100000000000004},"min":{"1min":0.48599999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.89500000000000002,"5min":0.92800000000000005,"15min":1.052},"last":0.74199999999999999}]},{"osd":6,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67000000000000004,"15min":0.68100000000000005},"min":{"1min":0.52200000000000002,"5min":0.439,"15min":0.435},"max":{"1min":0.94399999999999995,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.66300000000000003,"15min":0.67900000000000005},"min":{"1min":0.49199999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.89500000000000002,"5min":0.96399999999999997,"15min":1.085},"last":0.76800000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.68500000000000005,"15min":0.70099999999999996},"min":{"1min":0.54000000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91200000000000003,"5min":1.01,"15min":1.121},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.67000000000000004,"15min":0.68100000000000005},"min":{"1min":0.55700000000000005,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":1.0229999999999999},"last":0.65700000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575983,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60144,"kb_used_data":2784,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311696,"statfs":{"total":96636764160,"available":96575176704,"internally_reserved":0,"allocated":2850816,"data_stored":1453958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.38900000000000001,"15min":0.38800000000000001},"min":{"1min":0.307,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42199999999999999,"15min":0.42699999999999999},"min":{"1min":0.28100000000000003,"5min":0.27300000000000002,"15min":0.249},"max":{"1min":0.57099999999999995,"5min":0.58099999999999996,"15min":1.044},"last":0.35399999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40300000000000002,"15min":0.41199999999999998},"min":{"1min":0.28899999999999998,"5min":0.25700000000000001,"15min":0.255},"max":{"1min":0.52300000000000002,"5min":0.56399999999999995,"15min":0.753},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.44500000000000001,"15min":0.45200000000000001},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.28399999999999997},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.82199999999999995},"last":0.38900000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.46000000000000002,"15min":0.46700000000000003},"min":{"1min":0.36499999999999999,"5min":0.32900000000000001,"15min":0.27800000000000002},"max":{"1min":0.63900000000000001,"5min":0.69699999999999995,"15min":0.70299999999999996},"last":0.318},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.45200000000000001,"15min":0.45500000000000002},"min":{"1min":0.308,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.59799999999999998,"5min":0.64500000000000002,"15min":0.68600000000000005},"last":0.32800000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.64000000000000001,"15min":0.64500000000000002},"min":{"1min":0.47199999999999998,"5min":0.42199999999999999,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66400000000000003,"15min":0.66800000000000004},"min":{"1min":0.44500000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.77100000000000002}]},{"osd":5,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66800000000000004,"15min":0.67000000000000004},"min":{"1min":0.46100000000000002,"5min":0.46000000000000002,"15min":0.433},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":1.0149999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.66900000000000004,"15min":0.67200000000000004},"min":{"1min":0.52500000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.68400000000000005}]},{"osd":6,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.67300000000000004,"15min":0.67800000000000005},"min":{"1min":0.46999999999999997,"5min":0.46700000000000003,"15min":0.44600000000000001},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":1.091},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69999999999999996,"15min":0.70199999999999996},"min":{"1min":0.50600000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.96099999999999997},"last":0.56200000000000006}]},{"osd":7,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68300000000000005,"15min":0.69199999999999995},"min":{"1min":0.5,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92500000000000004,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.70399999999999996,"15min":0.71599999999999997},"min":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.67100000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575982,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47872,"kb_used_data":2796,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323968,"statfs":{"total":96636764160,"available":96587743232,"internally_reserved":0,"allocated":2863104,"data_stored":1455013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39000000000000001,"15min":0.38900000000000001},"min":{"1min":0.26000000000000001,"5min":0.255,"15min":0.23499999999999999},"max":{"1min":0.55200000000000005,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41199999999999998,"15min":0.41299999999999998},"min":{"1min":0.28799999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.56899999999999995,"5min":0.622,"15min":0.66700000000000004},"last":0.41099999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40699999999999997,"15min":0.40600000000000003},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.65700000000000003},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.438,"15min":0.44400000000000001},"min":{"1min":0.32800000000000001,"5min":0.27500000000000002,"15min":0.26600000000000001},"max":{"1min":0.63,"5min":0.65400000000000003,"15min":0.75800000000000001},"last":0.44500000000000001}]},{"osd":2,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48299999999999998,"15min":0.48599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.432},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.5,"15min":0.503},"min":{"1min":0.38600000000000001,"5min":0.34599999999999997,"15min":0.32900000000000001},"max":{"1min":0.67800000000000005,"5min":0.69299999999999995,"15min":0.79300000000000004},"last":0.499}]},{"osd":4,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.439,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.876,"5min":1.002,"15min":1.018},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66100000000000003,"15min":0.65300000000000002},"min":{"1min":0.497,"5min":0.45900000000000002,"15min":0.375},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.57299999999999995}]},{"osd":5,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.51900000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.95699999999999996,"5min":0.97299999999999998,"15min":0.999},"last":0.62},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.46200000000000002,"5min":0.437,"15min":0.437},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.71099999999999997}]},{"osd":6,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67500000000000004,"15min":0.66900000000000004},"min":{"1min":0.41499999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.68600000000000005,"15min":0.68400000000000005},"min":{"1min":0.48799999999999999,"5min":0.46700000000000003,"15min":0.41099999999999998},"max":{"1min":0.88800000000000001,"5min":0.96199999999999997,"15min":0.98799999999999999},"last":0.67200000000000004}]},{"osd":7,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68500000000000005,"15min":0.68000000000000005},"min":{"1min":0.46100000000000002,"5min":0.432,"15min":0.40999999999999998},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70299999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.024},"last":0.68600000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575982,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47864,"kb_used_data":2788,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323976,"statfs":{"total":96636764160,"available":96587751424,"internally_reserved":0,"allocated":2854912,"data_stored":1455469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.373,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.76400000000000001,"5min":0.876,"15min":0.88500000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.60499999999999998,"15min":0.60299999999999998},"min":{"1min":0.40200000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77900000000000003,"5min":1.202,"15min":1.202},"last":0.59699999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63,"15min":0.64400000000000002},"min":{"1min":0.439,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.82999999999999996,"5min":1.256,"15min":1.256},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64000000000000001,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80900000000000005,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.627}]},{"osd":2,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.42499999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.81799999999999995,"5min":0.84599999999999997,"15min":0.877},"last":0.502},{"interface":"front","average":{"1min":0.625,"5min":0.627,"15min":0.621},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88500000000000001,"5min":1.294,"15min":1.294},"last":0.54100000000000004}]},{"osd":3,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.44900000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.75,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46899999999999997,"5min":0.373,"15min":0.373},"max":{"1min":0.78000000000000003,"5min":0.90200000000000002,"15min":0.91100000000000003},"last":0.56299999999999994}]},{"osd":5,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.56200000000000006,"15min":0.51600000000000001},"min":{"1min":0.35299999999999998,"5min":0.33200000000000002,"15min":0.32800000000000001},"max":{"1min":0.69799999999999995,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.57899999999999996,"15min":0.54300000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68600000000000005,"5min":4.931,"15min":4.931},"last":0.72599999999999998}]},{"osd":6,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54000000000000004,"15min":0.53000000000000003},"min":{"1min":0.34100000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72399999999999998,"5min":0.83499999999999996,"15min":0.83899999999999997},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57899999999999996,"15min":0.56999999999999995},"min":{"1min":0.41599999999999998,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":0.751,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.68100000000000005}]},{"osd":7,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.60899999999999999,"15min":0.58299999999999996},"min":{"1min":0.36899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.65400000000000003,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.751},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.60399999999999998,"15min":0.60199999999999998},"min":{"1min":0.437,"5min":0.40799999999999997,"15min":0.373},"max":{"1min":0.748,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.61299999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575982,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60156,"kb_used_data":2788,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311684,"statfs":{"total":96636764160,"available":96575164416,"internally_reserved":0,"allocated":2854912,"data_stored":1453924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.46400000000000002,"5min":0.34000000000000002,"15min":0.33000000000000002},"max":{"1min":0.70199999999999996,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.504},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58999999999999997,"15min":0.59499999999999997},"min":{"1min":0.39700000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.73999999999999999,"5min":0.84699999999999998,"15min":0.89200000000000002},"last":0.54100000000000004}]},{"osd":1,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.41199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72699999999999998,"5min":0.81299999999999994,"15min":0.81399999999999995},"last":0.434},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58599999999999997,"15min":0.58699999999999997},"min":{"1min":0.43099999999999999,"5min":0.39400000000000002,"15min":0.37},"max":{"1min":0.70999999999999996,"5min":0.871,"15min":0.871},"last":0.57099999999999995}]},{"osd":2,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60599999999999998,"15min":0.60099999999999998},"min":{"1min":0.39400000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.70999999999999996,"5min":0.79700000000000004,"15min":0.84399999999999997},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.43099999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.67900000000000005,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.47599999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60499999999999998,"15min":0.59899999999999998},"min":{"1min":0.44700000000000001,"5min":0.40999999999999998,"15min":0.373},"max":{"1min":0.754,"5min":0.81100000000000005,"15min":0.94499999999999995},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63600000000000001,"15min":0.63700000000000001},"min":{"1min":0.48499999999999999,"5min":0.40300000000000002,"15min":0.39000000000000001},"max":{"1min":0.81399999999999995,"5min":0.88200000000000001,"15min":0.91500000000000004},"last":0.45700000000000002}]},{"osd":4,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52500000000000002,"15min":0.505},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.27400000000000002},"max":{"1min":0.746,"5min":0.85699999999999998,"15min":1.0369999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60499999999999998,"15min":0.57699999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.308},"max":{"1min":0.78000000000000003,"5min":0.88800000000000001,"15min":1.1639999999999999},"last":0.59899999999999998}]},{"osd":6,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54000000000000004,"15min":0.52400000000000002},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.27900000000000003},"max":{"1min":0.73799999999999999,"5min":0.94999999999999996,"15min":1.0169999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54600000000000004,"15min":0.52800000000000002},"min":{"1min":0.34399999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70999999999999996,"5min":0.878,"15min":1.0009999999999999},"last":0.48499999999999999}]},{"osd":7,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56100000000000005,"15min":0.54500000000000004},"min":{"1min":0.374,"5min":0.33600000000000002,"15min":0.32900000000000001},"max":{"1min":0.70899999999999996,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.66500000000000004,"15min":0.64700000000000002},"min":{"1min":0.45600000000000002,"5min":0.433,"15min":0.35199999999999998},"max":{"1min":0.84799999999999998,"5min":0.92300000000000004,"15min":0.96599999999999997},"last":0.624}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:47:26.649 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-06T13:47:26.649 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-06T13:47:26.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.813+0000 7f827f990700 1 Processor -- start 2024-04-06T13:47:26.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.813+0000 7f827f990700 1 -- start start 2024-04-06T13:47:26.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.814+0000 7f827f990700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f827815e8a0 0x7f82780cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:26.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.814+0000 7f827f990700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f827815fa70 0x7f827806aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:26.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.814+0000 7f827f990700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82780669d0 con 0x7f827815eee0 2024-04-06T13:47:26.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.814+0000 7f827f990700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f827806ede0 con 0x7f827815e8a0 2024-04-06T13:47:26.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.814+0000 7f827f990700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f827815ec80 con 0x7f827815fa70 2024-04-06T13:47:26.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.814+0000 7f827d72c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f827815e8a0 0x7f82780cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:26.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.814+0000 7f827cf2b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f827815eee0 0x7f827815f360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58534/0 (socket says 172.21.15.189:58534) 2024-04-06T13:47:26.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.814+0000 7f827cf2b700 1 -- 172.21.15.189:0/4206616005 learned_addr learned my addr 172.21.15.189:0/4206616005 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:26.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.814+0000 7f827df2d700 1 --2- 172.21.15.189:0/4206616005 >> v2:172.21.15.192:3300/0 conn(0x7f827815fa70 0x7f827806aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:26.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.814+0000 7f827d72c700 1 -- 172.21.15.189:0/4206616005 >> v2:172.21.15.192:3300/0 conn(0x7f827815fa70 msgr2=0x7f827806aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:26.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.814+0000 7f827d72c700 1 --2- 172.21.15.189:0/4206616005 >> v2:172.21.15.192:3300/0 conn(0x7f827815fa70 0x7f827806aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.814+0000 7f827d72c700 1 -- 172.21.15.189:0/4206616005 >> v1:172.21.15.189:6789/0 conn(0x7f827815eee0 legacy=0x7f827815f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:26.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.814+0000 7f827d72c700 1 -- 172.21.15.189:0/4206616005 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82780666f0 con 0x7f827815e8a0 2024-04-06T13:47:26.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.814+0000 7f826ffff700 1 -- 172.21.15.189:0/4206616005 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040783616 0 0) 0x7f82780669d0 con 0x7f827815eee0 2024-04-06T13:47:26.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.815+0000 7f827d72c700 1 --2- 172.21.15.189:0/4206616005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f827815e8a0 0x7f82780cbc20 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f82780cce40 tx=0x7f8268009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a64920737cd8066 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:26.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.815+0000 7f826ffff700 1 -- 172.21.15.189:0/4206616005 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8268011830 con 0x7f827815e8a0 2024-04-06T13:47:26.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.815+0000 7f826ffff700 1 -- 172.21.15.189:0/4206616005 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8268011990 con 0x7f827815e8a0 2024-04-06T13:47:26.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.815+0000 7f826ffff700 1 -- 172.21.15.189:0/4206616005 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8268011d20 con 0x7f827815e8a0 2024-04-06T13:47:26.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.815+0000 7f827f990700 1 -- 172.21.15.189:0/4206616005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f827815e8a0 msgr2=0x7f82780cbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:26.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.815+0000 7f827f990700 1 --2- 172.21.15.189:0/4206616005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f827815e8a0 0x7f82780cbc20 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f82780cce40 tx=0x7f8268009b20 comp rx=0 tx=0).stop 2024-04-06T13:47:26.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.815+0000 7f827f990700 1 -- 172.21.15.189:0/4206616005 shutdown_connections 2024-04-06T13:47:26.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.815+0000 7f827f990700 1 --2- 172.21.15.189:0/4206616005 >> v2:172.21.15.192:3300/0 conn(0x7f827815fa70 0x7f827806aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.815+0000 7f827f990700 1 --2- 172.21.15.189:0/4206616005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f827815e8a0 0x7f82780cbc20 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.815+0000 7f827f990700 1 -- 172.21.15.189:0/4206616005 >> 172.21.15.189:0/4206616005 conn(0x7f82780cc290 msgr2=0x7f82780cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:26.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.815+0000 7f827f990700 1 -- 172.21.15.189:0/4206616005 shutdown_connections 2024-04-06T13:47:26.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.815+0000 7f827f990700 1 -- 172.21.15.189:0/4206616005 wait complete. 2024-04-06T13:47:26.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.817+0000 7f827f990700 1 Processor -- start 2024-04-06T13:47:26.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.817+0000 7f827f990700 1 -- start start 2024-04-06T13:47:26.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.817+0000 7f827f990700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f827815e8a0 0x7f82780af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:26.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.817+0000 7f827f990700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f827815eee0 0x7f8278152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:26.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.817+0000 7f827f990700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f827815fa70 0x7f827815a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:26.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.817+0000 7f827f990700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82780ae450 con 0x7f827815fa70 2024-04-06T13:47:26.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.817+0000 7f827f990700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82780adf20 con 0x7f827815e8a0 2024-04-06T13:47:26.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.817+0000 7f827f990700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82780b07c0 con 0x7f827815eee0 2024-04-06T13:47:26.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.817+0000 7f827df2d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f827815fa70 0x7f827815a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:26.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.817+0000 7f827df2d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f827815fa70 0x7f827815a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55172/0 (socket says 172.21.15.189:55172) 2024-04-06T13:47:26.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.817+0000 7f827df2d700 1 -- 172.21.15.189:0/1544305637 learned_addr learned my addr 172.21.15.189:0/1544305637 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:26.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.818+0000 7f827d72c700 1 --2- 172.21.15.189:0/1544305637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f827815e8a0 0x7f82780af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:26.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.818+0000 7f827cf2b700 1 --2- 172.21.15.189:0/1544305637 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f827815eee0 0x7f8278152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:26.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.818+0000 7f827df2d700 1 -- 172.21.15.189:0/1544305637 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f827815eee0 msgr2=0x7f8278152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:26.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.818+0000 7f827df2d700 1 --2- 172.21.15.189:0/1544305637 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f827815eee0 0x7f8278152ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.818+0000 7f827df2d700 1 -- 172.21.15.189:0/1544305637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f827815e8a0 msgr2=0x7f82780af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:26.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.818+0000 7f827df2d700 1 --2- 172.21.15.189:0/1544305637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f827815e8a0 0x7f82780af940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:26.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.818+0000 7f827df2d700 1 -- 172.21.15.189:0/1544305637 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82680094a0 con 0x7f827815fa70 2024-04-06T13:47:26.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.818+0000 7f827df2d700 1 --2- 172.21.15.189:0/1544305637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f827815fa70 0x7f827815a670 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f8270009d30 tx=0x7f827000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:26.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.818+0000 7f826dffb700 1 -- 172.21.15.189:0/1544305637 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270009ba0 con 0x7f827815fa70 2024-04-06T13:47:26.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.818+0000 7f826dffb700 1 -- 172.21.15.189:0/1544305637 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827000f040 con 0x7f827815fa70 2024-04-06T13:47:26.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.818+0000 7f827f990700 1 -- 172.21.15.189:0/1544305637 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82780ae170 con 0x7f827815fa70 2024-04-06T13:47:26.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.818+0000 7f826dffb700 1 -- 172.21.15.189:0/1544305637 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270014480 con 0x7f827815fa70 2024-04-06T13:47:26.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.818+0000 7f827f990700 1 -- 172.21.15.189:0/1544305637 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82780c4620 con 0x7f827815fa70 2024-04-06T13:47:26.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.819+0000 7f827f990700 1 -- 172.21.15.189:0/1544305637 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8278072140 con 0x7f827815fa70 2024-04-06T13:47:26.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.820+0000 7f826dffb700 1 -- 172.21.15.189:0/1544305637 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f82700145e0 con 0x7f827815fa70 2024-04-06T13:47:26.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.820+0000 7f826dffb700 1 --2- 172.21.15.189:0/1544305637 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f825c06adb0 0x7f825c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:26.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.820+0000 7f826dffb700 1 -- 172.21.15.189:0/1544305637 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f827008a070 con 0x7f827815fa70 2024-04-06T13:47:26.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.820+0000 7f827d72c700 1 --2- 172.21.15.189:0/1544305637 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f825c06adb0 0x7f825c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:26.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.820+0000 7f827d72c700 1 --2- 172.21.15.189:0/1544305637 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f825c06adb0 0x7f825c06d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f82680090b0 tx=0x7f8268008030 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:26.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:26.824+0000 7f826dffb700 1 -- 172.21.15.189:0/1544305637 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8270054ea0 con 0x7f827815fa70 2024-04-06T13:47:27.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.100+0000 7f827f990700 1 -- 172.21.15.189:0/1544305637 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f8278074990 con 0x7f827815fa70 2024-04-06T13:47:27.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.103+0000 7f826dffb700 1 -- 172.21.15.189:0/1544305637 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v51) v1 ==== 190+0+0 (secure 0 0 0) 0x7f82700584c0 con 0x7f827815fa70 2024-04-06T13:47:27.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.106+0000 7f827f990700 1 -- 172.21.15.189:0/1544305637 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f825c06adb0 msgr2=0x7f825c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:27.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.106+0000 7f827f990700 1 --2- 172.21.15.189:0/1544305637 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f825c06adb0 0x7f825c06d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f82680090b0 tx=0x7f8268008030 comp rx=0 tx=0).stop 2024-04-06T13:47:27.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.106+0000 7f827f990700 1 -- 172.21.15.189:0/1544305637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f827815fa70 msgr2=0x7f827815a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:27.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.106+0000 7f827f990700 1 --2- 172.21.15.189:0/1544305637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f827815fa70 0x7f827815a670 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f8270009d30 tx=0x7f827000c3a0 comp rx=0 tx=0).stop 2024-04-06T13:47:27.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.106+0000 7f827f990700 1 -- 172.21.15.189:0/1544305637 shutdown_connections 2024-04-06T13:47:27.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.106+0000 7f827f990700 1 --2- 172.21.15.189:0/1544305637 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f825c06adb0 0x7f825c06d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:27.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.106+0000 7f827f990700 1 --2- 172.21.15.189:0/1544305637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f827815fa70 0x7f827815a670 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:27.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.106+0000 7f827f990700 1 --2- 172.21.15.189:0/1544305637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f827815e8a0 0x7f82780af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:27.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.106+0000 7f827f990700 1 --2- 172.21.15.189:0/1544305637 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f827815eee0 0x7f8278152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:27.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.106+0000 7f827f990700 1 -- 172.21.15.189:0/1544305637 >> 172.21.15.189:0/1544305637 conn(0x7f82780cc290 msgr2=0x7f8278159760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:27.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.106+0000 7f827f990700 1 -- 172.21.15.189:0/1544305637 shutdown_connections 2024-04-06T13:47:27.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.106+0000 7f827f990700 1 -- 172.21.15.189:0/1544305637 wait complete. 2024-04-06T13:47:27.107 INFO:teuthology.orchestra.run.smithi189.stderr:updated caps for client.0 2024-04-06T13:47:27.119 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:47:27.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.242+0000 7f6fb62e5700 1 Processor -- start 2024-04-06T13:47:27.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.242+0000 7f6fb62e5700 1 -- start start 2024-04-06T13:47:27.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.243+0000 7f6fb62e5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6fb015d7b0 0x7f6fb00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:27.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.243+0000 7f6fb62e5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fb015e500 0x7f6fb006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:27.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.243+0000 7f6fb62e5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb00b3990 con 0x7f6fb006a730 2024-04-06T13:47:27.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.243+0000 7f6fb62e5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb006b0f0 con 0x7f6fb015d7b0 2024-04-06T13:47:27.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.243+0000 7f6fb62e5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00b3060 con 0x7f6fb015e500 2024-04-06T13:47:27.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.244+0000 7f6faffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6fb015d7b0 0x7f6fb00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:27.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.244+0000 7f6faf7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6fb006a730 0x7f6fb015ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58550/0 (socket says 172.21.15.189:58550) 2024-04-06T13:47:27.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.244+0000 7f6faf7fe700 1 -- 172.21.15.189:0/655178608 learned_addr learned my addr 172.21.15.189:0/655178608 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:27.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.244+0000 7f6fb4882700 1 --2- 172.21.15.189:0/655178608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fb015e500 0x7f6fb006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:27.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.244+0000 7f6faffff700 1 -- 172.21.15.189:0/655178608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fb015e500 msgr2=0x7f6fb006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:27.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.244+0000 7f6faffff700 1 --2- 172.21.15.189:0/655178608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fb015e500 0x7f6fb006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:27.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.244+0000 7f6faffff700 1 -- 172.21.15.189:0/655178608 >> v1:172.21.15.189:6789/0 conn(0x7f6fb006a730 legacy=0x7f6fb015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:27.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.244+0000 7f6faffff700 1 -- 172.21.15.189:0/655178608 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb00b3340 con 0x7f6fb015d7b0 2024-04-06T13:47:27.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.244+0000 7f6faeffd700 1 -- 172.21.15.189:0/655178608 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527317470 0 0) 0x7f6fb00b3990 con 0x7f6fb006a730 2024-04-06T13:47:27.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.244+0000 7f6faffff700 1 --2- 172.21.15.189:0/655178608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb015d7b0 0x7f6fb00b3e20 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f6f98009840 tx=0x7f6f98009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd41628caa5879ac server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:27.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.244+0000 7f6faeffd700 1 -- 172.21.15.189:0/655178608 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f98011830 con 0x7f6fb015d7b0 2024-04-06T13:47:27.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.244+0000 7f6faeffd700 1 -- 172.21.15.189:0/655178608 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f98011990 con 0x7f6fb015d7b0 2024-04-06T13:47:27.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.244+0000 7f6faeffd700 1 -- 172.21.15.189:0/655178608 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f98011d20 con 0x7f6fb015d7b0 2024-04-06T13:47:27.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.245+0000 7f6fb62e5700 1 -- 172.21.15.189:0/655178608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb015d7b0 msgr2=0x7f6fb00b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:27.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.245+0000 7f6fb62e5700 1 --2- 172.21.15.189:0/655178608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb015d7b0 0x7f6fb00b3e20 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f6f98009840 tx=0x7f6f98009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:27.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.245+0000 7f6fb62e5700 1 -- 172.21.15.189:0/655178608 shutdown_connections 2024-04-06T13:47:27.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.245+0000 7f6fb62e5700 1 --2- 172.21.15.189:0/655178608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fb015e500 0x7f6fb006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:27.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.245+0000 7f6fb62e5700 1 --2- 172.21.15.189:0/655178608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb015d7b0 0x7f6fb00b3e20 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:27.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.245+0000 7f6fb62e5700 1 -- 172.21.15.189:0/655178608 >> 172.21.15.189:0/655178608 conn(0x7f6fb00d00b0 msgr2=0x7f6fb00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:27.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.245+0000 7f6fb62e5700 1 -- 172.21.15.189:0/655178608 shutdown_connections 2024-04-06T13:47:27.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.245+0000 7f6fb62e5700 1 -- 172.21.15.189:0/655178608 wait complete. 2024-04-06T13:47:27.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.246+0000 7f6fb62e5700 1 Processor -- start 2024-04-06T13:47:27.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.246+0000 7f6fb62e5700 1 -- start start 2024-04-06T13:47:27.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.247+0000 7f6fb62e5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb006a730 0x7f6fb01584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:27.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.247+0000 7f6fb62e5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fb015d7b0 0x7f6fb0158e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:27.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.247+0000 7f6fb62e5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fb015e500 0x7f6fb01593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:27.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.247+0000 7f6fb62e5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00b0640 con 0x7f6fb006a730 2024-04-06T13:47:27.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.247+0000 7f6fb62e5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00cf4b0 con 0x7f6fb015e500 2024-04-06T13:47:27.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.247+0000 7f6fb62e5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00b0850 con 0x7f6fb015d7b0 2024-04-06T13:47:27.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.247+0000 7f6faffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb006a730 0x7f6fb01584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:27.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.247+0000 7f6faffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb006a730 0x7f6fb01584d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55190/0 (socket says 172.21.15.189:55190) 2024-04-06T13:47:27.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.247+0000 7f6faffff700 1 -- 172.21.15.189:0/891434996 learned_addr learned my addr 172.21.15.189:0/891434996 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:27.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.247+0000 7f6faf7fe700 1 --2- 172.21.15.189:0/891434996 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fb015d7b0 0x7f6fb0158e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:27.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.247+0000 7f6fb4882700 1 --2- 172.21.15.189:0/891434996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fb015e500 0x7f6fb01593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:27.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.247+0000 7f6faffff700 1 -- 172.21.15.189:0/891434996 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fb015d7b0 msgr2=0x7f6fb0158e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:27.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.247+0000 7f6faffff700 1 --2- 172.21.15.189:0/891434996 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fb015d7b0 0x7f6fb0158e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:27.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.247+0000 7f6faffff700 1 -- 172.21.15.189:0/891434996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fb015e500 msgr2=0x7f6fb01593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:27.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.247+0000 7f6faffff700 1 --2- 172.21.15.189:0/891434996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fb015e500 0x7f6fb01593a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:27.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.247+0000 7f6faffff700 1 -- 172.21.15.189:0/891434996 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f980094a0 con 0x7f6fb006a730 2024-04-06T13:47:27.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.247+0000 7f6faffff700 1 --2- 172.21.15.189:0/891434996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb006a730 0x7f6fb01584d0 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f6f98009b90 tx=0x7f6f98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:27.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.248+0000 7f6facff9700 1 -- 172.21.15.189:0/891434996 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f980036a0 con 0x7f6fb006a730 2024-04-06T13:47:27.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.248+0000 7f6facff9700 1 -- 172.21.15.189:0/891434996 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9801f070 con 0x7f6fb006a730 2024-04-06T13:47:27.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.248+0000 7f6fb62e5700 1 -- 172.21.15.189:0/891434996 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb00b0b30 con 0x7f6fb006a730 2024-04-06T13:47:27.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.248+0000 7f6facff9700 1 -- 172.21.15.189:0/891434996 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f98003a40 con 0x7f6fb006a730 2024-04-06T13:47:27.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.248+0000 7f6fb62e5700 1 -- 172.21.15.189:0/891434996 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb00b10f0 con 0x7f6fb006a730 2024-04-06T13:47:27.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.249+0000 7f6fb62e5700 1 -- 172.21.15.189:0/891434996 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb0067bb0 con 0x7f6fb006a730 2024-04-06T13:47:27.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.249+0000 7f6facff9700 1 -- 172.21.15.189:0/891434996 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f6f98003ba0 con 0x7f6fb006a730 2024-04-06T13:47:27.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.249+0000 7f6facff9700 1 --2- 172.21.15.189:0/891434996 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f9006adb0 0x7f6f9006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:27.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.250+0000 7f6facff9700 1 -- 172.21.15.189:0/891434996 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6f98021070 con 0x7f6fb006a730 2024-04-06T13:47:27.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.250+0000 7f6faf7fe700 1 --2- 172.21.15.189:0/891434996 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f9006adb0 0x7f6f9006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:27.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.250+0000 7f6faf7fe700 1 --2- 172.21.15.189:0/891434996 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f9006adb0 0x7f6f9006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6fb00cd030 tx=0x7f6fa0009040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:27.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.254+0000 7f6facff9700 1 -- 172.21.15.189:0/891434996 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6f98055f60 con 0x7f6fb006a730 2024-04-06T13:47:27.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.493+0000 7f6fb62e5700 1 -- 172.21.15.189:0/891434996 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fb0074990 con 0x7f6fb006a730 2024-04-06T13:47:27.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.493+0000 7f6facff9700 1 -- 172.21.15.189:0/891434996 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+4754 (secure 0 0 0) 0x7f6f9802b090 con 0x7f6fb006a730 2024-04-06T13:47:27.493 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:27.493 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":100,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8885,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1481817364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1481817364},{"type":"v1","addr":"172.21.15.189:6837","nonce":1481817364}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":11244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/4021875447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":4021875447},{"type":"v1","addr":"172.21.15.189:6836","nonce":4021875447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":11262,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/4066431971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4066431971},{"type":"v1","addr":"172.21.15.192:6834","nonce":4066431971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:47:22.116705+0000","modified":"2024-04-06T13:47:25.129787+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/694564673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":694564673},{"type":"v1","addr":"172.21.15.192:6835","nonce":694564673}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-04-06T13:47:27.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.496+0000 7f6fb62e5700 1 -- 172.21.15.189:0/891434996 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f9006adb0 msgr2=0x7f6f9006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:27.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.496+0000 7f6fb62e5700 1 --2- 172.21.15.189:0/891434996 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f9006adb0 0x7f6f9006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6fb00cd030 tx=0x7f6fa0009040 comp rx=0 tx=0).stop 2024-04-06T13:47:27.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.496+0000 7f6fb62e5700 1 -- 172.21.15.189:0/891434996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb006a730 msgr2=0x7f6fb01584d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:27.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.496+0000 7f6fb62e5700 1 --2- 172.21.15.189:0/891434996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb006a730 0x7f6fb01584d0 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f6f98009b90 tx=0x7f6f98004750 comp rx=0 tx=0).stop 2024-04-06T13:47:27.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.496+0000 7f6fb62e5700 1 -- 172.21.15.189:0/891434996 shutdown_connections 2024-04-06T13:47:27.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.496+0000 7f6fb62e5700 1 --2- 172.21.15.189:0/891434996 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f9006adb0 0x7f6f9006d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:27.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.496+0000 7f6fb62e5700 1 --2- 172.21.15.189:0/891434996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fb015e500 0x7f6fb01593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:27.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.496+0000 7f6fb62e5700 1 --2- 172.21.15.189:0/891434996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fb006a730 0x7f6fb01584d0 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:27.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.496+0000 7f6fb62e5700 1 --2- 172.21.15.189:0/891434996 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6fb015d7b0 0x7f6fb0158e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:27.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.496+0000 7f6fb62e5700 1 -- 172.21.15.189:0/891434996 >> 172.21.15.189:0/891434996 conn(0x7f6fb00d00b0 msgr2=0x7f6fb006ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:27.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.496+0000 7f6fb62e5700 1 -- 172.21.15.189:0/891434996 shutdown_connections 2024-04-06T13:47:27.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:27.496+0000 7f6fb62e5700 1 -- 172.21.15.189:0/891434996 wait complete. 2024-04-06T13:47:27.497 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 100 2024-04-06T13:47:27.910 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-06T13:47:22.116705+0000', 'modified': '2024-04-06T13:47:25.129787+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11265': {'gid': 11265, 'name': 'b', 'rank': 0, 'incarnation': 97, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.192:6835/694564673', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6833', 'nonce': 694564673}, {'type': 'v1', 'addr': '172.21.15.192:6835', 'nonce': 694564673}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-06T13:47:27.911 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-06T13:47:27.911 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-06T13:47:28.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.037+0000 7fecb30ff700 1 Processor -- start 2024-04-06T13:47:28.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.037+0000 7fecb30ff700 1 -- start start 2024-04-06T13:47:28.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.037+0000 7fecb30ff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecac159b20 0x7fecac06b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.037+0000 7fecb30ff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fecac0cdc60 0x7fecac0ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.037+0000 7fecb30ff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecac06a8d0 con 0x7fecac06e470 2024-04-06T13:47:28.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.037+0000 7fecb30ff700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecac0ce620 con 0x7fecac0cdc60 2024-04-06T13:47:28.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.037+0000 7fecb30ff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecac159f00 con 0x7fecac159b20 2024-04-06T13:47:28.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecb169c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fecac06e470 0x7fecac06e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58562/0 (socket says 172.21.15.189:58562) 2024-04-06T13:47:28.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecb169c700 1 -- 172.21.15.189:0/2329150835 learned_addr learned my addr 172.21.15.189:0/2329150835 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:28.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecab7fe700 1 -- 172.21.15.189:0/2329150835 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198515960 0 0) 0x7fecac06a8d0 con 0x7fecac06e470 2024-04-06T13:47:28.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecab7fe700 1 -- 172.21.15.189:0/2329150835 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec9c0035b0 con 0x7fecac06e470 2024-04-06T13:47:28.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecabfff700 1 --2- 172.21.15.189:0/2329150835 >> v2:172.21.15.192:3300/0 conn(0x7fecac0cdc60 0x7fecac0ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecb0e9b700 1 --2- 172.21.15.189:0/2329150835 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecac159b20 0x7fecac06b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecab7fe700 1 -- 172.21.15.189:0/2329150835 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3390885523 0 0) 0x7fec9c0035b0 con 0x7fecac06e470 2024-04-06T13:47:28.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecab7fe700 1 -- 172.21.15.189:0/2329150835 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecac159b20 msgr2=0x7fecac06b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:28.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecab7fe700 1 --2- 172.21.15.189:0/2329150835 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecac159b20 0x7fecac06b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecab7fe700 1 -- 172.21.15.189:0/2329150835 >> v2:172.21.15.192:3300/0 conn(0x7fecac0cdc60 msgr2=0x7fecac0ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:28.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecab7fe700 1 --2- 172.21.15.189:0/2329150835 >> v2:172.21.15.192:3300/0 conn(0x7fecac0cdc60 0x7fecac0ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecab7fe700 1 -- 172.21.15.189:0/2329150835 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecac06abb0 con 0x7fecac06e470 2024-04-06T13:47:28.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecab7fe700 1 -- 172.21.15.189:0/2329150835 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7feca40019f0 con 0x7fecac06e470 2024-04-06T13:47:28.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecab7fe700 1 -- 172.21.15.189:0/2329150835 >> v1:172.21.15.189:6789/0 conn(0x7fecac06e470 legacy=0x7fecac06e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:28.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecab7fe700 1 --2- 172.21.15.189:0/2329150835 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecac159b20 0x7fec9c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecab7fe700 1 --2- 172.21.15.189:0/2329150835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec9c003f80 0x7fec9c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecab7fe700 1 --2- 172.21.15.189:0/2329150835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec9c004d90 0x7fec9c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecab7fe700 1 -- 172.21.15.189:0/2329150835 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac0ce620 con 0x7fec9c003f80 2024-04-06T13:47:28.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecab7fe700 1 -- 172.21.15.189:0/2329150835 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac159f00 con 0x7fec9c004d90 2024-04-06T13:47:28.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecab7fe700 1 -- 172.21.15.189:0/2329150835 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c00ba60 con 0x7fecac159b20 2024-04-06T13:47:28.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecabfff700 1 --2- 172.21.15.189:0/2329150835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec9c003f80 0x7fec9c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecb0e9b700 1 --2- 172.21.15.189:0/2329150835 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecac159b20 0x7fec9c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.038+0000 7fecb169c700 1 --2- 172.21.15.189:0/2329150835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec9c004d90 0x7fec9c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.039+0000 7fecabfff700 1 -- 172.21.15.189:0/2329150835 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecac159b20 msgr2=0x7fec9c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:28.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.039+0000 7fecabfff700 1 --2- 172.21.15.189:0/2329150835 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecac159b20 0x7fec9c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.039+0000 7fecabfff700 1 -- 172.21.15.189:0/2329150835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec9c004d90 msgr2=0x7fec9c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:28.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.039+0000 7fecabfff700 1 --2- 172.21.15.189:0/2329150835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec9c004d90 0x7fec9c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.039+0000 7fecabfff700 1 -- 172.21.15.189:0/2329150835 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9c0035b0 con 0x7fec9c003f80 2024-04-06T13:47:28.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.039+0000 7fecb169c700 1 --2- 172.21.15.189:0/2329150835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec9c004d90 0x7fec9c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:28.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.039+0000 7fecabfff700 1 --2- 172.21.15.189:0/2329150835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec9c003f80 0x7fec9c004850 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7fec98009b90 tx=0x7fec9c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed0643af3875e2d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:28.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.039+0000 7fecab7fe700 1 -- 172.21.15.189:0/2329150835 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9800e6c0 con 0x7fec9c003f80 2024-04-06T13:47:28.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.039+0000 7fecab7fe700 1 -- 172.21.15.189:0/2329150835 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9800e820 con 0x7fec9c003f80 2024-04-06T13:47:28.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.039+0000 7fecab7fe700 1 -- 172.21.15.189:0/2329150835 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9800eb90 con 0x7fec9c003f80 2024-04-06T13:47:28.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.039+0000 7fecb30ff700 1 -- 172.21.15.189:0/2329150835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec9c003f80 msgr2=0x7fec9c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:28.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.039+0000 7fecb30ff700 1 --2- 172.21.15.189:0/2329150835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec9c003f80 0x7fec9c004850 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7fec98009b90 tx=0x7fec9c005880 comp rx=0 tx=0).stop 2024-04-06T13:47:28.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.039+0000 7fecb169c700 1 -- 172.21.15.189:0/2329150835 reap_dead start 2024-04-06T13:47:28.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.039+0000 7fecb30ff700 1 -- 172.21.15.189:0/2329150835 shutdown_connections 2024-04-06T13:47:28.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.039+0000 7fecb30ff700 1 -- 172.21.15.189:0/2329150835 >> 172.21.15.189:0/2329150835 conn(0x7fecac0cc290 msgr2=0x7fecac0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:28.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.039+0000 7fecb30ff700 1 -- 172.21.15.189:0/2329150835 shutdown_connections 2024-04-06T13:47:28.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.039+0000 7fecb30ff700 1 -- 172.21.15.189:0/2329150835 wait complete. 2024-04-06T13:47:28.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.041+0000 7fecb30ff700 1 Processor -- start 2024-04-06T13:47:28.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.041+0000 7fecb30ff700 1 -- start start 2024-04-06T13:47:28.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.041+0000 7fecb30ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec9c003f80 0x7fecac158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.041+0000 7fecb30ff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecac15b490 0x7fecac0c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7fecb30ff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecac0c88a0 0x7fecac0c35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7fecb30ff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac15b070 con 0x7fecac15b490 2024-04-06T13:47:28.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7fecb30ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac0c8d50 con 0x7fec9c003f80 2024-04-06T13:47:28.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7fecb30ff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecac15a3f0 con 0x7fecac0c88a0 2024-04-06T13:47:28.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7fecabfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecac15b490 0x7fecac0c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7fecabfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecac15b490 0x7fecac0c8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55200/0 (socket says 172.21.15.189:55200) 2024-04-06T13:47:28.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7fecabfff700 1 -- 172.21.15.189:0/1165076810 learned_addr learned my addr 172.21.15.189:0/1165076810 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:28.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7fecb169c700 1 --2- 172.21.15.189:0/1165076810 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecac0c88a0 0x7fecac0c35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7fecb0e9b700 1 --2- 172.21.15.189:0/1165076810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec9c003f80 0x7fecac158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7fecabfff700 1 -- 172.21.15.189:0/1165076810 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecac0c88a0 msgr2=0x7fecac0c35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:28.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7fecabfff700 1 --2- 172.21.15.189:0/1165076810 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecac0c88a0 0x7fecac0c35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7fecabfff700 1 -- 172.21.15.189:0/1165076810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec9c003f80 msgr2=0x7fecac158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:28.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7fecabfff700 1 --2- 172.21.15.189:0/1165076810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec9c003f80 0x7fecac158150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7fecabfff700 1 -- 172.21.15.189:0/1165076810 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecac15a6d0 con 0x7fecac15b490 2024-04-06T13:47:28.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7fecb0e9b700 1 --2- 172.21.15.189:0/1165076810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec9c003f80 0x7fecac158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:28.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7fecabfff700 1 --2- 172.21.15.189:0/1165076810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecac15b490 0x7fecac0c8360 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7fec9c004790 tx=0x7fec980047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:28.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7feca97fa700 1 -- 172.21.15.189:0/1165076810 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9800e6c0 con 0x7fecac15b490 2024-04-06T13:47:28.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7feca97fa700 1 -- 172.21.15.189:0/1165076810 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec98003ab0 con 0x7fecac15b490 2024-04-06T13:47:28.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7fecb30ff700 1 -- 172.21.15.189:0/1165076810 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecac15a9b0 con 0x7fecac15b490 2024-04-06T13:47:28.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7feca97fa700 1 -- 172.21.15.189:0/1165076810 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec98003e60 con 0x7fecac15b490 2024-04-06T13:47:28.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.042+0000 7fecb30ff700 1 -- 172.21.15.189:0/1165076810 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecac15b910 con 0x7fecac15b490 2024-04-06T13:47:28.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.044+0000 7feca97fa700 1 -- 172.21.15.189:0/1165076810 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fec9801e660 con 0x7fecac15b490 2024-04-06T13:47:28.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.044+0000 7feca97fa700 1 --2- 172.21.15.189:0/1165076810 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fec9006adb0 0x7fec9006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.044+0000 7feca97fa700 1 -- 172.21.15.189:0/1165076810 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fec9808d320 con 0x7fecac15b490 2024-04-06T13:47:28.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.044+0000 7fecb0e9b700 1 --2- 172.21.15.189:0/1165076810 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fec9006adb0 0x7fec9006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.045+0000 7fecb0e9b700 1 --2- 172.21.15.189:0/1165076810 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fec9006adb0 0x7fec9006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feca000ac80 tx=0x7feca0008040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:28.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.046+0000 7fecbf2b3b80 1 Processor -- start 2024-04-06T13:47:28.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.046+0000 7fecbf2b3b80 1 -- start start 2024-04-06T13:47:28.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.047+0000 7fecbf2b3b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 0x5608a22a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.047+0000 7fecbf2b3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 0x5608a227d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.047+0000 7fecbf2b3b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a227c980 0x5608a227db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.047+0000 7fecbf2b3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608a21d6940 con 0x5608a22a6fc0 2024-04-06T13:47:28.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.047+0000 7fecbf2b3b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608a21d4300 con 0x5608a227c980 2024-04-06T13:47:28.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.047+0000 7fecbf2b3b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x5608a21cfbf0 con 0x5608a22a1c20 2024-04-06T13:47:28.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.047+0000 7fecabfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 0x5608a227d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.047+0000 7fecabfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 0x5608a227d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55202/0 (socket says 172.21.15.189:55202) 2024-04-06T13:47:28.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.047+0000 7fecabfff700 1 -- 172.21.15.189:0/4234132479 learned_addr learned my addr 172.21.15.189:0/4234132479 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:28.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.047+0000 7fecb169c700 1 --2- 172.21.15.189:0/4234132479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a227c980 0x5608a227db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.047+0000 7fecb0e9b700 1 --2- 172.21.15.189:0/4234132479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 0x5608a22a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.047+0000 7fecabfff700 1 -- 172.21.15.189:0/4234132479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 msgr2=0x5608a22a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:28.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.047+0000 7fecabfff700 1 --2- 172.21.15.189:0/4234132479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 0x5608a22a5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.047+0000 7fecabfff700 1 -- 172.21.15.189:0/4234132479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a227c980 msgr2=0x5608a227db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:28.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.047+0000 7fecabfff700 1 --2- 172.21.15.189:0/4234132479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a227c980 0x5608a227db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.047+0000 7fecabfff700 1 -- 172.21.15.189:0/4234132479 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5608a1ffdb10 con 0x5608a22a6fc0 2024-04-06T13:47:28.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.048+0000 7fecabfff700 1 --2- 172.21.15.189:0/4234132479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 0x5608a227d5f0 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7fec98000c00 tx=0x7fec98058540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd49cbb41a9b056 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:28.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.048+0000 7fec967fc700 1 -- 172.21.15.189:0/4234132479 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9805ff00 con 0x5608a22a6fc0 2024-04-06T13:47:28.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.048+0000 7fec967fc700 1 -- 172.21.15.189:0/4234132479 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9805f070 con 0x5608a22a6fc0 2024-04-06T13:47:28.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.048+0000 7fec967fc700 1 -- 172.21.15.189:0/4234132479 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec980644c0 con 0x5608a22a6fc0 2024-04-06T13:47:28.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.048+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/4234132479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 msgr2=0x5608a227d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:28.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.048+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/4234132479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 0x5608a227d5f0 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7fec98000c00 tx=0x7fec98058540 comp rx=0 tx=0).stop 2024-04-06T13:47:28.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.048+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/4234132479 shutdown_connections 2024-04-06T13:47:28.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.048+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/4234132479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a227c980 0x5608a227db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.048+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/4234132479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 0x5608a22a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.048+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/4234132479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 0x5608a227d5f0 unknown :-1 s=CLOSED pgs=2422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.048+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/4234132479 >> 172.21.15.189:0/4234132479 conn(0x5608a21dd650 msgr2=0x5608a22c43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:28.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.048+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/4234132479 shutdown_connections 2024-04-06T13:47:28.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.048+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/4234132479 wait complete. 2024-04-06T13:47:28.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.049+0000 7fecbf2b3b80 1 -- start start 2024-04-06T13:47:28.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.049+0000 7fecbf2b3b80 1 Processor -- start 2024-04-06T13:47:28.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.050+0000 7fecbf2b3b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a227c980 0x5608a22a9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.050+0000 7fecbf2b3b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 0x5608a22ac900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.050+0000 7fecbf2b3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 0x5608a22ace40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.050+0000 7fecbf2b3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608a216bd30 con 0x5608a22a6fc0 2024-04-06T13:47:28.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.050+0000 7fecbf2b3b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608a216b9d0 con 0x5608a227c980 2024-04-06T13:47:28.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.050+0000 7fecbf2b3b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x5608a2053650 con 0x5608a22a1c20 2024-04-06T13:47:28.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fecbf2b3b80 10 client.0 _ll_register_callbacks cb 0x5608a2175420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T13:47:28.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fecb169c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 0x5608a22ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fecb169c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 0x5608a22ace40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55216/0 (socket says 172.21.15.189:55216) 2024-04-06T13:47:28.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fecb169c700 1 -- 172.21.15.189:0/3746143058 learned_addr learned my addr 172.21.15.189:0/3746143058 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:28.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fecabfff700 1 --2- 172.21.15.189:0/3746143058 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 0x5608a22ac900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fecb0e9b700 1 --2- 172.21.15.189:0/3746143058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a227c980 0x5608a22a9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fecb169c700 1 -- 172.21.15.189:0/3746143058 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 msgr2=0x5608a22ac900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:28.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fecb169c700 1 --2- 172.21.15.189:0/3746143058 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 0x5608a22ac900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fecb169c700 1 -- 172.21.15.189:0/3746143058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a227c980 msgr2=0x5608a22a9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:28.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fecb169c700 1 --2- 172.21.15.189:0/3746143058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a227c980 0x5608a22a9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fecb169c700 1 -- 172.21.15.189:0/3746143058 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecac15b910 con 0x5608a22a6fc0 2024-04-06T13:47:28.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fecb0e9b700 1 --2- 172.21.15.189:0/3746143058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a227c980 0x5608a22a9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:28.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fecabfff700 1 --2- 172.21.15.189:0/3746143058 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 0x5608a22ac900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:28.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fecb169c700 1 --2- 172.21.15.189:0/3746143058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 0x5608a22ace40 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7fecac072380 tx=0x7feca40079e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf0b25a86d80ed5c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:28.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fec8b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T13:47:28.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fec8b7fe700 1 -- 172.21.15.189:0/3746143058 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca40128b0 con 0x5608a22a6fc0 2024-04-06T13:47:28.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/3746143058 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5608a2190330 con 0x5608a22a6fc0 2024-04-06T13:47:28.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fec8b7fe700 1 -- 172.21.15.189:0/3746143058 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(227..227 src has 1..227) v4 ==== 6101+0+0 (secure 0 0 0) 0x7feca4012a10 con 0x5608a22a6fc0 2024-04-06T13:47:28.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fec8b7fe700 1 -- 172.21.15.189:0/3746143058 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) v2 ==== 24+0+0 (secure 0 0 0) 0x7feca4012c10 con 0x5608a22a6fc0 2024-04-06T13:47:28.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fecbf2b3b80 10 client.11412 fetch_fsmap learned FSMap version 100 2024-04-06T13:47:28.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/3746143058 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=2+}) v3 -- 0x5608a2195e50 con 0x5608a22a6fc0 2024-04-06T13:47:28.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.051+0000 7fec8b7fe700 1 -- 172.21.15.189:0/3746143058 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5608a2190330 con 0x5608a22a6fc0 2024-04-06T13:47:28.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.052+0000 7fec8b7fe700 1 -- 172.21.15.189:0/3746143058 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 100) v1 ==== 3114+0+0 (secure 0 0 0) 0x7feca4016480 con 0x5608a22a6fc0 2024-04-06T13:47:28.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.052+0000 7fecbf2b3b80 10 client.11412 fetch_fsmap finished waiting for FSMap version 100 2024-04-06T13:47:28.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.052+0000 7fecbf2b3b80 10 client.11412 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-06T13:47:28.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.052+0000 7fecbf2b3b80 20 client.11412 populate_metadata read hostname 'smithi189' 2024-04-06T13:47:28.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.052+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/3746143058 >> [v2:172.21.15.192:6833/694564673,v1:172.21.15.192:6835/694564673] conn(0x5608a22aac00 0x5608a22aafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.052+0000 7fecbf2b3b80 4 client.11412 mds_command: new command op to 11265 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-06T13:47:28.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.052+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/3746143058 --> [v2:172.21.15.192:6833/694564673,v1:172.21.15.192:6835/694564673] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5608a21c86b0 con 0x5608a22aac00 2024-04-06T13:47:28.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.052+0000 7fecb0e9b700 1 --2- 172.21.15.189:0/3746143058 >> [v2:172.21.15.192:6833/694564673,v1:172.21.15.192:6835/694564673] conn(0x5608a22aac00 0x5608a22aafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.052+0000 7fec8b7fe700 10 client.11412 ms_handle_connect on v2:172.21.15.192:6833/694564673 2024-04-06T13:47:28.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.052+0000 7fecb0e9b700 1 --2- 172.21.15.189:0/3746143058 >> [v2:172.21.15.192:6833/694564673,v1:172.21.15.192:6835/694564673] conn(0x5608a22aac00 0x5608a22aafe0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d55f340ee52c24b1 server_cookie=339ea8629c07cc38 in_seq=0 out_seq=0 2024-04-06T13:47:28.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.053+0000 7fec8b7fe700 1 -- 172.21.15.189:0/3746143058 <== mds.0 v2:172.21.15.192:6833/694564673 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5608a21c86b0 con 0x5608a22aac00 2024-04-06T13:47:28.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.053+0000 7fec8b7fe700 10 client.11412 handle_command_reply: tid=0 2024-04-06T13:47:28.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.053+0000 7fecbf2b3b80 1 client.11412 shutdown 2024-04-06T13:47:28.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.053+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/3746143058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 msgr2=0x5608a22ace40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:28.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.053+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/3746143058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 0x5608a22ace40 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7fecac072380 tx=0x7feca40079e0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.053+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/3746143058 shutdown_connections 2024-04-06T13:47:28.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.053+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/3746143058 >> [v2:172.21.15.192:6833/694564673,v1:172.21.15.192:6835/694564673] conn(0x5608a22aac00 0x5608a22aafe0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.053+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/3746143058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 0x5608a22ace40 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.053+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/3746143058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a227c980 0x5608a22a9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.053+0000 7fec8b7fe700 0 client.11412 ms_handle_reset on v2:172.21.15.192:6833/694564673 2024-04-06T13:47:28.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.053+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/3746143058 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 0x5608a22ac900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.053+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/3746143058 >> 172.21.15.189:0/3746143058 conn(0x5608a21dd650 msgr2=0x5608a22a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:28.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.053+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/3746143058 shutdown_connections 2024-04-06T13:47:28.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.053+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/3746143058 wait complete. 2024-04-06T13:47:28.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.053+0000 7fecbf2b3b80 20 client.11412 trim_cache size 0 max 16384 2024-04-06T13:47:28.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.071+0000 7fecbf2b3b80 1 Processor -- start 2024-04-06T13:47:28.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.071+0000 7fecbf2b3b80 1 -- start start 2024-04-06T13:47:28.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fecbf2b3b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a227c980 0x5608a22c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fecbf2b3b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 0x5608a2279120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fecbf2b3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 0x5608a229f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fecbf2b3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608a22a73a0 con 0x5608a22a6fc0 2024-04-06T13:47:28.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fecbf2b3b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608a227bf00 con 0x5608a227c980 2024-04-06T13:47:28.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fecbf2b3b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x5608a2279660 con 0x5608a22a1c20 2024-04-06T13:47:28.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fecb169c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 0x5608a229f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fecb169c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 0x5608a229f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55218/0 (socket says 172.21.15.189:55218) 2024-04-06T13:47:28.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fecb169c700 1 -- 172.21.15.189:0/1611262434 learned_addr learned my addr 172.21.15.189:0/1611262434 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:28.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fecb0e9b700 1 --2- 172.21.15.189:0/1611262434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a227c980 0x5608a22c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fecabfff700 1 --2- 172.21.15.189:0/1611262434 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 0x5608a2279120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fecb169c700 1 -- 172.21.15.189:0/1611262434 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 msgr2=0x5608a2279120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:28.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fecb169c700 1 --2- 172.21.15.189:0/1611262434 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 0x5608a2279120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fecb169c700 1 -- 172.21.15.189:0/1611262434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a227c980 msgr2=0x5608a22c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:28.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fecb169c700 1 --2- 172.21.15.189:0/1611262434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a227c980 0x5608a22c4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fecb169c700 1 -- 172.21.15.189:0/1611262434 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5608a2055c80 con 0x5608a22a6fc0 2024-04-06T13:47:28.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fecb0e9b700 1 --2- 172.21.15.189:0/1611262434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a227c980 0x5608a22c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:28.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fecabfff700 1 --2- 172.21.15.189:0/1611262434 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 0x5608a2279120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:28.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fecb169c700 1 --2- 172.21.15.189:0/1611262434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 0x5608a229f5f0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7feca4001e20 tx=0x7feca400e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d96247cc2129d616 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:28.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fec96ffd700 1 -- 172.21.15.189:0/1611262434 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca4016990 con 0x5608a22a6fc0 2024-04-06T13:47:28.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fec96ffd700 1 -- 172.21.15.189:0/1611262434 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca4016480 con 0x5608a22a6fc0 2024-04-06T13:47:28.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.072+0000 7fec96ffd700 1 -- 172.21.15.189:0/1611262434 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca4016d70 con 0x5608a22a6fc0 2024-04-06T13:47:28.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.073+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/1611262434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 msgr2=0x5608a229f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:28.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.073+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/1611262434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 0x5608a229f5f0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7feca4001e20 tx=0x7feca400e000 comp rx=0 tx=0).stop 2024-04-06T13:47:28.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.073+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/1611262434 shutdown_connections 2024-04-06T13:47:28.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.073+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/1611262434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a22a6fc0 0x5608a229f5f0 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.073+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/1611262434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a227c980 0x5608a22c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.073+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/1611262434 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 0x5608a2279120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.073+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/1611262434 >> 172.21.15.189:0/1611262434 conn(0x5608a21dd650 msgr2=0x5608a22bdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:28.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.073+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/1611262434 shutdown_connections 2024-04-06T13:47:28.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.073+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/1611262434 wait complete. 2024-04-06T13:47:28.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.073+0000 7fecbf2b3b80 1 -- start start 2024-04-06T13:47:28.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.074+0000 7fecbf2b3b80 1 Processor -- start 2024-04-06T13:47:28.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.074+0000 7fecbf2b3b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a227c980 0x5608a2332b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.074+0000 7fecbf2b3b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 0x5608a22abfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.074+0000 7fecbf2b3b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a22a6fc0 0x5608a22ac4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.074+0000 7fecbf2b3b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608a22a5b10 con 0x5608a227c980 2024-04-06T13:47:28.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.074+0000 7fecbf2b3b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608a22a9760 con 0x5608a22a6fc0 2024-04-06T13:47:28.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.074+0000 7fecbf2b3b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x5608a22794a0 con 0x5608a22a1c20 2024-04-06T13:47:28.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.074+0000 7fecbf2b3b80 10 client.0 _ll_register_callbacks cb 0x7feca40073b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T13:47:28.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.074+0000 7fecb0e9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a227c980 0x5608a2332b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fecb0e9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a227c980 0x5608a2332b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55224/0 (socket says 172.21.15.189:55224) 2024-04-06T13:47:28.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fecb0e9b700 1 -- 172.21.15.189:0/3828696115 learned_addr learned my addr 172.21.15.189:0/3828696115 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:28.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fecb169c700 1 --2- 172.21.15.189:0/3828696115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a22a6fc0 0x5608a22ac4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fecabfff700 1 --2- 172.21.15.189:0/3828696115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 0x5608a22abfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fecb0e9b700 1 -- 172.21.15.189:0/3828696115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 msgr2=0x5608a22abfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:28.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fecb0e9b700 1 --2- 172.21.15.189:0/3828696115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 0x5608a22abfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fecb0e9b700 1 -- 172.21.15.189:0/3828696115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a22a6fc0 msgr2=0x5608a22ac4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:28.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fecb0e9b700 1 --2- 172.21.15.189:0/3828696115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a22a6fc0 0x5608a22ac4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fecb0e9b700 1 -- 172.21.15.189:0/3828696115 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5608a2195e50 con 0x5608a227c980 2024-04-06T13:47:28.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fecb0e9b700 1 --2- 172.21.15.189:0/3828696115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a227c980 0x5608a2332b20 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x5608a229ce40 tx=0x7feca0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9be9f829cba1a59d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:28.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fec8b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T13:47:28.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fec8b7fe700 1 -- 172.21.15.189:0/3828696115 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca0027040 con 0x5608a227c980 2024-04-06T13:47:28.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fec8b7fe700 1 -- 172.21.15.189:0/3828696115 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(227..227 src has 1..227) v4 ==== 6101+0+0 (secure 0 0 0) 0x7feca001a070 con 0x5608a227c980 2024-04-06T13:47:28.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/3828696115 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fec9805f070 con 0x5608a227c980 2024-04-06T13:47:28.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fec8b7fe700 1 -- 172.21.15.189:0/3828696115 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) v2 ==== 24+0+0 (secure 0 0 0) 0x7feca0029590 con 0x5608a227c980 2024-04-06T13:47:28.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fecbf2b3b80 10 client.11418 fetch_fsmap learned FSMap version 100 2024-04-06T13:47:28.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/3828696115 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=2+}) v3 -- 0x5608a1ffbfe0 con 0x5608a227c980 2024-04-06T13:47:28.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fec8b7fe700 1 -- 172.21.15.189:0/3828696115 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9805f070 con 0x5608a227c980 2024-04-06T13:47:28.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fec8b7fe700 1 -- 172.21.15.189:0/3828696115 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 100) v1 ==== 3114+0+0 (secure 0 0 0) 0x7feca0014c90 con 0x5608a227c980 2024-04-06T13:47:28.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fecbf2b3b80 10 client.11418 fetch_fsmap finished waiting for FSMap version 100 2024-04-06T13:47:28.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.075+0000 7fecbf2b3b80 10 client.11418 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-06T13:47:28.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.076+0000 7fecbf2b3b80 20 client.11418 populate_metadata read hostname 'smithi189' 2024-04-06T13:47:28.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.076+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/3828696115 >> [v2:172.21.15.192:6833/694564673,v1:172.21.15.192:6835/694564673] conn(0x5608a22a4d20 0x5608a22a5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.076+0000 7fecbf2b3b80 4 client.11418 mds_command: new command op to 11265 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-06T13:47:28.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.076+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/3828696115 --> [v2:172.21.15.192:6833/694564673,v1:172.21.15.192:6835/694564673] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5608a201eca0 con 0x5608a22a4d20 2024-04-06T13:47:28.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.076+0000 7fecabfff700 1 --2- 172.21.15.189:0/3828696115 >> [v2:172.21.15.192:6833/694564673,v1:172.21.15.192:6835/694564673] conn(0x5608a22a4d20 0x5608a22a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.076+0000 7fec8b7fe700 10 client.11418 ms_handle_connect on v2:172.21.15.192:6833/694564673 2024-04-06T13:47:28.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.076+0000 7fecabfff700 1 --2- 172.21.15.189:0/3828696115 >> [v2:172.21.15.192:6833/694564673,v1:172.21.15.192:6835/694564673] conn(0x5608a22a4d20 0x5608a22a5100 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19cb73c0110b119c server_cookie=30c1272a6ee0c718 in_seq=0 out_seq=0 2024-04-06T13:47:28.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.076+0000 7fec8b7fe700 1 -- 172.21.15.189:0/3828696115 <== mds.0 v2:172.21.15.192:6833/694564673 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x5608a201eca0 con 0x5608a22a4d20 2024-04-06T13:47:28.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.076+0000 7fec8b7fe700 10 client.11418 handle_command_reply: tid=0 2024-04-06T13:47:28.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.076+0000 7fecbf2b3b80 1 client.11418 shutdown 2024-04-06T13:47:28.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.076+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/3828696115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a227c980 msgr2=0x5608a2332b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:28.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.076+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/3828696115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a227c980 0x5608a2332b20 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x5608a229ce40 tx=0x7feca0004750 comp rx=0 tx=0).stop 2024-04-06T13:47:28.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.076+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/3828696115 shutdown_connections 2024-04-06T13:47:28.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.076+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/3828696115 >> [v2:172.21.15.192:6833/694564673,v1:172.21.15.192:6835/694564673] conn(0x5608a22a4d20 0x5608a22a5100 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.076+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/3828696115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5608a22a6fc0 0x5608a22ac4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.076+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/3828696115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608a227c980 0x5608a2332b20 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.076+0000 7fecbf2b3b80 1 --2- 172.21.15.189:0/3828696115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x5608a22a1c20 0x5608a22abfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.076+0000 7fec8b7fe700 0 client.11418 ms_handle_reset on v2:172.21.15.192:6833/694564673 2024-04-06T13:47:28.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.076+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/3828696115 >> 172.21.15.189:0/3828696115 conn(0x5608a21dd650 msgr2=0x5608a22bdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:28.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.077+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/3828696115 shutdown_connections 2024-04-06T13:47:28.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.077+0000 7fecbf2b3b80 1 -- 172.21.15.189:0/3828696115 wait complete. 2024-04-06T13:47:28.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.077+0000 7fecbf2b3b80 20 client.11418 trim_cache size 0 max 16384 2024-04-06T13:47:28.083 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:47:28.083 INFO:teuthology.orchestra.run.smithi189.stdout: "cluster_fsid": "e30041de-e9a3-4351-8ca7-29d7f3e3728c", 2024-04-06T13:47:28.084 INFO:teuthology.orchestra.run.smithi189.stdout: "whoami": 0, 2024-04-06T13:47:28.084 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 11265, 2024-04-06T13:47:28.084 INFO:teuthology.orchestra.run.smithi189.stdout: "want_state": "up:active", 2024-04-06T13:47:28.084 INFO:teuthology.orchestra.run.smithi189.stdout: "state": "up:active", 2024-04-06T13:47:28.084 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs", 2024-04-06T13:47:28.084 INFO:teuthology.orchestra.run.smithi189.stdout: "rank_uptime": 5.9584483280000002, 2024-04-06T13:47:28.084 INFO:teuthology.orchestra.run.smithi189.stdout: "mdsmap_epoch": 100, 2024-04-06T13:47:28.084 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch": 227, 2024-04-06T13:47:28.084 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch_barrier": 227, 2024-04-06T13:47:28.084 INFO:teuthology.orchestra.run.smithi189.stdout: "uptime": 10.325953500000001 2024-04-06T13:47:28.084 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:47:28.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.077+0000 7fecb30ff700 1 -- 172.21.15.189:0/1165076810 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fec9006adb0 msgr2=0x7fec9006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:28.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.077+0000 7fecb30ff700 1 --2- 172.21.15.189:0/1165076810 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fec9006adb0 0x7fec9006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feca000ac80 tx=0x7feca0008040 comp rx=0 tx=0).stop 2024-04-06T13:47:28.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.077+0000 7fecb30ff700 1 -- 172.21.15.189:0/1165076810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecac15b490 msgr2=0x7fecac0c8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:28.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.077+0000 7fecb30ff700 1 --2- 172.21.15.189:0/1165076810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecac15b490 0x7fecac0c8360 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7fec9c004790 tx=0x7fec980047c0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.077+0000 7fecb30ff700 1 -- 172.21.15.189:0/1165076810 shutdown_connections 2024-04-06T13:47:28.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.077+0000 7fecb30ff700 1 --2- 172.21.15.189:0/1165076810 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fec9006adb0 0x7fec9006d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.077+0000 7fecb30ff700 1 --2- 172.21.15.189:0/1165076810 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecac0c88a0 0x7fecac0c35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.077+0000 7fecb30ff700 1 --2- 172.21.15.189:0/1165076810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec9c003f80 0x7fecac158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.077+0000 7fecb30ff700 1 --2- 172.21.15.189:0/1165076810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecac15b490 0x7fecac0c8360 unknown :-1 s=CLOSED pgs=2421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.077+0000 7fecb30ff700 1 -- 172.21.15.189:0/1165076810 >> 172.21.15.189:0/1165076810 conn(0x7fecac0cc290 msgr2=0x7fecac0b0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:28.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.077+0000 7fecb30ff700 1 -- 172.21.15.189:0/1165076810 shutdown_connections 2024-04-06T13:47:28.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.077+0000 7fecb30ff700 1 -- 172.21.15.189:0/1165076810 wait complete. 2024-04-06T13:47:28.089 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T13:47:28.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.212+0000 7f56f7dc5700 1 Processor -- start 2024-04-06T13:47:28.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.213+0000 7f56f7dc5700 1 -- start start 2024-04-06T13:47:28.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.213+0000 7f56f7dc5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56f015d7b0 0x7f56f00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.213+0000 7f56f7dc5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f56f015e500 0x7f56f006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.213+0000 7f56f7dc5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f00b3990 con 0x7f56f006a730 2024-04-06T13:47:28.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.213+0000 7f56f7dc5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f006b0f0 con 0x7f56f015e500 2024-04-06T13:47:28.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.213+0000 7f56f7dc5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f00b3060 con 0x7f56f015d7b0 2024-04-06T13:47:28.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.214+0000 7f56f6362700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f56f015e500 0x7f56f006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.214+0000 7f56f5360700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f56f006a730 0x7f56f015ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58572/0 (socket says 172.21.15.189:58572) 2024-04-06T13:47:28.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.214+0000 7f56f5360700 1 -- 172.21.15.189:0/1385106547 learned_addr learned my addr 172.21.15.189:0/1385106547 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:28.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.214+0000 7f56f5b61700 1 --2- 172.21.15.189:0/1385106547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56f015d7b0 0x7f56f00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.214+0000 7f56f6362700 1 -- 172.21.15.189:0/1385106547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56f015d7b0 msgr2=0x7f56f00b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:28.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.214+0000 7f56f6362700 1 --2- 172.21.15.189:0/1385106547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56f015d7b0 0x7f56f00b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.214+0000 7f56f6362700 1 -- 172.21.15.189:0/1385106547 >> v1:172.21.15.189:6789/0 conn(0x7f56f006a730 legacy=0x7f56f015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:28.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.214+0000 7f56f6362700 1 -- 172.21.15.189:0/1385106547 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f00b3340 con 0x7f56f015e500 2024-04-06T13:47:28.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.214+0000 7f56f4b5f700 1 -- 172.21.15.189:0/1385106547 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3127252206 0 0) 0x7f56f00b3990 con 0x7f56f006a730 2024-04-06T13:47:28.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.214+0000 7f56f6362700 1 --2- 172.21.15.189:0/1385106547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f015e500 0x7f56f006abb0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f56e8009d90 tx=0x7f56e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fe3b0ba3b40e7c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:28.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.214+0000 7f56f4b5f700 1 -- 172.21.15.189:0/1385106547 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e800e070 con 0x7f56f015e500 2024-04-06T13:47:28.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.214+0000 7f56f4b5f700 1 -- 172.21.15.189:0/1385106547 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e800bbd0 con 0x7f56f015e500 2024-04-06T13:47:28.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.214+0000 7f56f4b5f700 1 -- 172.21.15.189:0/1385106547 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e8012430 con 0x7f56f015e500 2024-04-06T13:47:28.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.215+0000 7f56f7dc5700 1 -- 172.21.15.189:0/1385106547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f015e500 msgr2=0x7f56f006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:28.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.215+0000 7f56f7dc5700 1 --2- 172.21.15.189:0/1385106547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f015e500 0x7f56f006abb0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f56e8009d90 tx=0x7f56e800b330 comp rx=0 tx=0).stop 2024-04-06T13:47:28.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.215+0000 7f56f7dc5700 1 -- 172.21.15.189:0/1385106547 shutdown_connections 2024-04-06T13:47:28.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.215+0000 7f56f7dc5700 1 --2- 172.21.15.189:0/1385106547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f015e500 0x7f56f006abb0 unknown :-1 s=CLOSED pgs=2427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.215+0000 7f56f7dc5700 1 --2- 172.21.15.189:0/1385106547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56f015d7b0 0x7f56f00b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.215+0000 7f56f7dc5700 1 -- 172.21.15.189:0/1385106547 >> 172.21.15.189:0/1385106547 conn(0x7f56f00d00b0 msgr2=0x7f56f00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:28.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.215+0000 7f56f7dc5700 1 -- 172.21.15.189:0/1385106547 shutdown_connections 2024-04-06T13:47:28.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.215+0000 7f56f7dc5700 1 -- 172.21.15.189:0/1385106547 wait complete. 2024-04-06T13:47:28.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.216+0000 7f56f7dc5700 1 Processor -- start 2024-04-06T13:47:28.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.216+0000 7f56f7dc5700 1 -- start start 2024-04-06T13:47:28.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.217+0000 7f56f7dc5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56f006a730 0x7f56f0148df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.217+0000 7f56f7dc5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f015d7b0 0x7f56f0158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.217+0000 7f56f7dc5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56f015e500 0x7f56f00cc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.217+0000 7f56f7dc5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f01591f0 con 0x7f56f015d7b0 2024-04-06T13:47:28.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.217+0000 7f56f7dc5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f00cc9a0 con 0x7f56f015e500 2024-04-06T13:47:28.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.217+0000 7f56f7dc5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f00cf350 con 0x7f56f006a730 2024-04-06T13:47:28.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.217+0000 7f56f5360700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f015d7b0 0x7f56f0158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.217+0000 7f56f5360700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f015d7b0 0x7f56f0158cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55246/0 (socket says 172.21.15.189:55246) 2024-04-06T13:47:28.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.217+0000 7f56f5360700 1 -- 172.21.15.189:0/2596682028 learned_addr learned my addr 172.21.15.189:0/2596682028 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:28.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.218+0000 7f56f5360700 1 -- 172.21.15.189:0/2596682028 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56f006a730 msgr2=0x7f56f0148df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:28.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.218+0000 7f56f5360700 1 --2- 172.21.15.189:0/2596682028 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56f006a730 0x7f56f0148df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.218+0000 7f56f5360700 1 -- 172.21.15.189:0/2596682028 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56f015e500 msgr2=0x7f56f00cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:28.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.218+0000 7f56f5360700 1 --2- 172.21.15.189:0/2596682028 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56f015e500 0x7f56f00cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.218+0000 7f56f5360700 1 -- 172.21.15.189:0/2596682028 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e8009a70 con 0x7f56f015d7b0 2024-04-06T13:47:28.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.218+0000 7f56f5360700 1 --2- 172.21.15.189:0/2596682028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f015d7b0 0x7f56f0158cb0 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7f56ec0011e0 tx=0x7f56ec00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:28.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.218+0000 7f56e27fc700 1 -- 172.21.15.189:0/2596682028 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec004c80 con 0x7f56f015d7b0 2024-04-06T13:47:28.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.218+0000 7f56e27fc700 1 -- 172.21.15.189:0/2596682028 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ec004e00 con 0x7f56f015d7b0 2024-04-06T13:47:28.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.218+0000 7f56f7dc5700 1 -- 172.21.15.189:0/2596682028 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f00cf070 con 0x7f56f015d7b0 2024-04-06T13:47:28.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.218+0000 7f56e27fc700 1 -- 172.21.15.189:0/2596682028 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec007310 con 0x7f56f015d7b0 2024-04-06T13:47:28.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.218+0000 7f56f7dc5700 1 -- 172.21.15.189:0/2596682028 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f01584b0 con 0x7f56f015d7b0 2024-04-06T13:47:28.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.219+0000 7f56f7dc5700 1 -- 172.21.15.189:0/2596682028 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f0067bb0 con 0x7f56f015d7b0 2024-04-06T13:47:28.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.219+0000 7f56e27fc700 1 -- 172.21.15.189:0/2596682028 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f56ec007470 con 0x7f56f015d7b0 2024-04-06T13:47:28.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.220+0000 7f56e27fc700 1 --2- 172.21.15.189:0/2596682028 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f56d406adb0 0x7f56d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:28.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.220+0000 7f56e27fc700 1 -- 172.21.15.189:0/2596682028 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f56ec089440 con 0x7f56f015d7b0 2024-04-06T13:47:28.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.220+0000 7f56f5b61700 1 --2- 172.21.15.189:0/2596682028 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f56d406adb0 0x7f56d406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:28.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.220+0000 7f56f5b61700 1 --2- 172.21.15.189:0/2596682028 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f56d406adb0 0x7f56d406d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f56e40097f0 tx=0x7f56e4008040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:28.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.225+0000 7f56e27fc700 1 -- 172.21.15.189:0/2596682028 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f56ec0542f0 con 0x7f56f015d7b0 2024-04-06T13:47:28.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.463+0000 7f56f7dc5700 1 -- 172.21.15.189:0/2596682028 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f56f0074990 con 0x7f56f015d7b0 2024-04-06T13:47:28.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.463+0000 7f56e27fc700 1 -- 172.21.15.189:0/2596682028 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v100) v1 ==== 53+0+73 (secure 0 0 0) 0x7f56ec057910 con 0x7f56f015d7b0 2024-04-06T13:47:28.463 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-06T13:47:28.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.465+0000 7f56f7dc5700 1 -- 172.21.15.189:0/2596682028 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f56d406adb0 msgr2=0x7f56d406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:28.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.465+0000 7f56f7dc5700 1 --2- 172.21.15.189:0/2596682028 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f56d406adb0 0x7f56d406d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f56e40097f0 tx=0x7f56e4008040 comp rx=0 tx=0).stop 2024-04-06T13:47:28.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.465+0000 7f56f7dc5700 1 -- 172.21.15.189:0/2596682028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f015d7b0 msgr2=0x7f56f0158cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:28.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.465+0000 7f56f7dc5700 1 --2- 172.21.15.189:0/2596682028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f015d7b0 0x7f56f0158cb0 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7f56ec0011e0 tx=0x7f56ec00cae0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.465+0000 7f56f7dc5700 1 -- 172.21.15.189:0/2596682028 shutdown_connections 2024-04-06T13:47:28.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.465+0000 7f56f7dc5700 1 --2- 172.21.15.189:0/2596682028 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f56d406adb0 0x7f56d406d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.465+0000 7f56f7dc5700 1 --2- 172.21.15.189:0/2596682028 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56f015e500 0x7f56f00cc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.465+0000 7f56f7dc5700 1 --2- 172.21.15.189:0/2596682028 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56f006a730 0x7f56f0148df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.465+0000 7f56f7dc5700 1 --2- 172.21.15.189:0/2596682028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f015d7b0 0x7f56f0158cb0 unknown :-1 s=CLOSED pgs=2428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:28.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.465+0000 7f56f7dc5700 1 -- 172.21.15.189:0/2596682028 >> 172.21.15.189:0/2596682028 conn(0x7f56f00d00b0 msgr2=0x7f56f00ae1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:28.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.465+0000 7f56f7dc5700 1 -- 172.21.15.189:0/2596682028 shutdown_connections 2024-04-06T13:47:28.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:28.465+0000 7f56f7dc5700 1 -- 172.21.15.189:0/2596682028 wait complete. 2024-04-06T13:47:28.478 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-06T13:47:28.478 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T13:47:28.478 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-06T13:47:28.478 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-06T13:47:28.478 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T13:47:28.479 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-06T13:47:28.479 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-06T13:47:28.479 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T13:47:28.479 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:47:28.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:28.479 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-06T13:47:28.499 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T13:47:28.500 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T13:47:28.500 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T13:47:28.500 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:47:28.500 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T13:47:28.500 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:47:28.501 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:47:28.501 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:47:28.501 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-06T13:47:28.501 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:47:28.501 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:47:28.502 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-06T13:47:28.502 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:47:28.502 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:47:28.502 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T13:47:28.502 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:47:28.503 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T13:47:28.503 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 36641sec preferred_lft 36641sec 2024-04-06T13:47:28.503 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::8592:df2c:92e2:dace/64 scope link noprefixroute 2024-04-06T13:47:28.503 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:47:28.503 INFO:teuthology.orchestra.run.smithi189.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T13:47:28.503 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff 2024-04-06T13:47:28.503 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-06T13:47:28.504 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:47:28.504 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::88e:b6ff:fe6f:1cb0/64 scope link 2024-04-06T13:47:28.504 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:47:28.504 INFO:teuthology.orchestra.run.smithi189.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T13:47:28.504 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 62:a5:8d:c5:1a:9c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T13:47:28.504 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::60a5:8dff:fec5:1a9c/64 scope link 2024-04-06T13:47:28.504 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:47:28.505 INFO:teuthology.orchestra.run.smithi189.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T13:47:28.505 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-06T13:47:28.505 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::1c7e:c6ff:febb:1dab/64 scope link 2024-04-06T13:47:28.505 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:47:28.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:28.507 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T13:47:28.565 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-06T13:47:28.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:28.567 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-06T13:47:28.627 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-06T13:47:28.627 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T13:47:28.628 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T13:47:28.628 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T13:47:28.628 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T13:47:28.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:28.629 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T13:47:28.629 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T13:47:28.629 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T13:47:28.629 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T13:47:28.629 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T13:47:28.629 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T13:47:28.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:28.734 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T13:47:28.751 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-06T13:47:28.752 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-06T13:47:28.752 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:47:28.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:28.752 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-04-06T13:47:28.819 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T13:47:28.819 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-06T13:47:28.879 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:47:28.879 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T13:47:28.879 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-06T13:47:28.937 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T13:47:28.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:28.937 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:47:29.232 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:47:29.232 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:47:29.232 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:47:29.242 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:47:29.242 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:47:29.242 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:47:29.242 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T13:47:29.242 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T13:47:29.262 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:47:29.262 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:47:29.262 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:47:29.262 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T13:47:29.262 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T13:47:29.263 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790" 2024-04-06T13:47:29.263 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2024-04-06T13:47:29.263 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e30041de-e9a3-4351-8ca7-29d7f3e3728c" will pass to kernel 2024-04-06T13:47:29.264 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T13:47:29.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:29.264 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-06T13:47:29.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:29.296 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T13:47:29.387 INFO:teuthology.orchestra.run.smithi189.stdout:client8969 2024-04-06T13:47:29.387 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:47:29.387 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:47:29.408 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-04-06T13:47:29.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.575+0000 7ff15b3f4700 1 Processor -- start 2024-04-06T13:47:29.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.575+0000 7ff15b3f4700 1 -- start start 2024-04-06T13:47:29.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.576+0000 7ff15b3f4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1540cea80 0x7ff1540cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:29.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.576+0000 7ff15b3f4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff1540cdae0 0x7ff15406ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:29.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.576+0000 7ff15b3f4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15415a830 con 0x7ff1540aec70 2024-04-06T13:47:29.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.576+0000 7ff15b3f4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1540cd920 con 0x7ff1540cdae0 2024-04-06T13:47:29.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.576+0000 7ff15b3f4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540cc050 con 0x7ff1540cea80 2024-04-06T13:47:29.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.576+0000 7ff159190700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff1540aec70 0x7ff1540ce370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55104/0 (socket says 172.21.15.189:55104) 2024-04-06T13:47:29.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.576+0000 7ff159190700 1 -- 172.21.15.189:0/178741618 learned_addr learned my addr 172.21.15.189:0/178741618 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:29.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.576+0000 7ff15898f700 1 --2- 172.21.15.189:0/178741618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1540cea80 0x7ff1540cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:29.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.576+0000 7ff159991700 1 --2- 172.21.15.189:0/178741618 >> v2:172.21.15.192:3300/0 conn(0x7ff1540cdae0 0x7ff15406ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:29.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.576+0000 7ff14bfff700 1 -- 172.21.15.189:0/178741618 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307821311 0 0) 0x7ff15415a830 con 0x7ff1540aec70 2024-04-06T13:47:29.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.576+0000 7ff14bfff700 1 -- 172.21.15.189:0/178741618 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1400035b0 con 0x7ff1540aec70 2024-04-06T13:47:29.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.576+0000 7ff14bfff700 1 -- 172.21.15.189:0/178741618 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 526158324 0 0) 0x7ff1400035b0 con 0x7ff1540aec70 2024-04-06T13:47:29.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.576+0000 7ff14bfff700 1 -- 172.21.15.189:0/178741618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1540cea80 msgr2=0x7ff1540cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:29.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.576+0000 7ff14bfff700 1 --2- 172.21.15.189:0/178741618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1540cea80 0x7ff1540cd3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:29.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff14bfff700 1 -- 172.21.15.189:0/178741618 >> v2:172.21.15.192:3300/0 conn(0x7ff1540cdae0 msgr2=0x7ff15406ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:29.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff14bfff700 1 --2- 172.21.15.189:0/178741618 >> v2:172.21.15.192:3300/0 conn(0x7ff1540cdae0 0x7ff15406ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:29.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff14bfff700 1 -- 172.21.15.189:0/178741618 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1540666f0 con 0x7ff1540aec70 2024-04-06T13:47:29.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff14bfff700 1 -- 172.21.15.189:0/178741618 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7ff150001670 con 0x7ff1540aec70 2024-04-06T13:47:29.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff14bfff700 1 -- 172.21.15.189:0/178741618 >> v1:172.21.15.189:6789/0 conn(0x7ff1540aec70 legacy=0x7ff1540ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:29.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff14bfff700 1 --2- 172.21.15.189:0/178741618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1540cea80 0x7ff140003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:29.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff14bfff700 1 --2- 172.21.15.189:0/178741618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff140003f80 0x7ff140004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:29.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff14bfff700 1 --2- 172.21.15.189:0/178741618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff140004d90 0x7ff140005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:29.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff14bfff700 1 -- 172.21.15.189:0/178741618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540cd920 con 0x7ff140003f80 2024-04-06T13:47:29.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff14bfff700 1 -- 172.21.15.189:0/178741618 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540cc050 con 0x7ff140004d90 2024-04-06T13:47:29.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff14bfff700 1 -- 172.21.15.189:0/178741618 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14000ba60 con 0x7ff1540cea80 2024-04-06T13:47:29.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff15898f700 1 --2- 172.21.15.189:0/178741618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff140003f80 0x7ff140004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:29.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff159190700 1 --2- 172.21.15.189:0/178741618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1540cea80 0x7ff140003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:29.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff159991700 1 --2- 172.21.15.189:0/178741618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff140004d90 0x7ff140005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:29.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff15898f700 1 -- 172.21.15.189:0/178741618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1540cea80 msgr2=0x7ff140003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:29.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff15898f700 1 --2- 172.21.15.189:0/178741618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1540cea80 0x7ff140003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:29.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff15898f700 1 -- 172.21.15.189:0/178741618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff140004d90 msgr2=0x7ff140005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:29.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff15898f700 1 --2- 172.21.15.189:0/178741618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff140004d90 0x7ff140005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:29.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff15898f700 1 -- 172.21.15.189:0/178741618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1400035b0 con 0x7ff140003f80 2024-04-06T13:47:29.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff15898f700 1 --2- 172.21.15.189:0/178741618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff140003f80 0x7ff140004850 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7ff1440096f0 tx=0x7ff140005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe51c789dcc322e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:29.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff14bfff700 1 -- 172.21.15.189:0/178741618 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14400e6c0 con 0x7ff140003f80 2024-04-06T13:47:29.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff14bfff700 1 -- 172.21.15.189:0/178741618 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14400e820 con 0x7ff140003f80 2024-04-06T13:47:29.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.577+0000 7ff14bfff700 1 -- 172.21.15.189:0/178741618 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14400ebb0 con 0x7ff140003f80 2024-04-06T13:47:29.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.578+0000 7ff15b3f4700 1 -- 172.21.15.189:0/178741618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff140003f80 msgr2=0x7ff140004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:29.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.578+0000 7ff15b3f4700 1 --2- 172.21.15.189:0/178741618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff140003f80 0x7ff140004850 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7ff1440096f0 tx=0x7ff140005880 comp rx=0 tx=0).stop 2024-04-06T13:47:29.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.578+0000 7ff159991700 1 -- 172.21.15.189:0/178741618 reap_dead start 2024-04-06T13:47:29.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.578+0000 7ff15b3f4700 1 -- 172.21.15.189:0/178741618 shutdown_connections 2024-04-06T13:47:29.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.578+0000 7ff15b3f4700 1 -- 172.21.15.189:0/178741618 >> 172.21.15.189:0/178741618 conn(0x7ff1540cc290 msgr2=0x7ff1540cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:29.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.578+0000 7ff15b3f4700 1 -- 172.21.15.189:0/178741618 shutdown_connections 2024-04-06T13:47:29.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.578+0000 7ff15b3f4700 1 -- 172.21.15.189:0/178741618 wait complete. 2024-04-06T13:47:29.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.579+0000 7ff15b3f4700 1 Processor -- start 2024-04-06T13:47:29.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.579+0000 7ff15b3f4700 1 -- start start 2024-04-06T13:47:29.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.580+0000 7ff15b3f4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff140003f80 0x7ff1541f3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:29.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.580+0000 7ff15b3f4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff154158d20 0x7ff154152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:29.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.580+0000 7ff15b3f4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff15406e1f0 0x7ff1541f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:29.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.580+0000 7ff15b3f4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff154155950 con 0x7ff15406e1f0 2024-04-06T13:47:29.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.580+0000 7ff15b3f4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1541594e0 con 0x7ff140003f80 2024-04-06T13:47:29.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.580+0000 7ff15b3f4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540b4720 con 0x7ff154158d20 2024-04-06T13:47:29.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.580+0000 7ff159991700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff15406e1f0 0x7ff1541f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:29.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.580+0000 7ff159991700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff15406e1f0 0x7ff1541f6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54830/0 (socket says 172.21.15.189:54830) 2024-04-06T13:47:29.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.580+0000 7ff159991700 1 -- 172.21.15.189:0/601460877 learned_addr learned my addr 172.21.15.189:0/601460877 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:29.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.580+0000 7ff159190700 1 --2- 172.21.15.189:0/601460877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff140003f80 0x7ff1541f3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:29.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.580+0000 7ff15898f700 1 --2- 172.21.15.189:0/601460877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff154158d20 0x7ff154152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:29.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.580+0000 7ff159991700 1 -- 172.21.15.189:0/601460877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff154158d20 msgr2=0x7ff154152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:29.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.580+0000 7ff159991700 1 --2- 172.21.15.189:0/601460877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff154158d20 0x7ff154152bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:29.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.580+0000 7ff159991700 1 -- 172.21.15.189:0/601460877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff140003f80 msgr2=0x7ff1541f3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:29.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.580+0000 7ff159991700 1 --2- 172.21.15.189:0/601460877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff140003f80 0x7ff1541f3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:29.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.580+0000 7ff159991700 1 -- 172.21.15.189:0/601460877 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1540b49c0 con 0x7ff15406e1f0 2024-04-06T13:47:29.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.580+0000 7ff159190700 1 --2- 172.21.15.189:0/601460877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff140003f80 0x7ff1541f3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:29.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.580+0000 7ff159991700 1 --2- 172.21.15.189:0/601460877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff15406e1f0 0x7ff1541f6b30 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7ff15406cc10 tx=0x7ff14c011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:29.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.581+0000 7ff149ffb700 1 -- 172.21.15.189:0/601460877 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c00e040 con 0x7ff15406e1f0 2024-04-06T13:47:29.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.581+0000 7ff149ffb700 1 -- 172.21.15.189:0/601460877 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14c00f040 con 0x7ff15406e1f0 2024-04-06T13:47:29.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.581+0000 7ff15b3f4700 1 -- 172.21.15.189:0/601460877 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1540b4ca0 con 0x7ff15406e1f0 2024-04-06T13:47:29.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.581+0000 7ff149ffb700 1 -- 172.21.15.189:0/601460877 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c00fdb0 con 0x7ff15406e1f0 2024-04-06T13:47:29.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.581+0000 7ff15b3f4700 1 -- 172.21.15.189:0/601460877 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1541591a0 con 0x7ff15406e1f0 2024-04-06T13:47:29.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.582+0000 7ff15b3f4700 1 -- 172.21.15.189:0/601460877 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1541f6850 con 0x7ff15406e1f0 2024-04-06T13:47:29.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.582+0000 7ff149ffb700 1 -- 172.21.15.189:0/601460877 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7ff14c015430 con 0x7ff15406e1f0 2024-04-06T13:47:29.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.583+0000 7ff149ffb700 1 --2- 172.21.15.189:0/601460877 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff13806ade0 0x7ff13806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:29.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.583+0000 7ff149ffb700 1 -- 172.21.15.189:0/601460877 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff14c089fc0 con 0x7ff15406e1f0 2024-04-06T13:47:29.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.583+0000 7ff159190700 1 --2- 172.21.15.189:0/601460877 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff13806ade0 0x7ff13806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:29.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.583+0000 7ff159190700 1 --2- 172.21.15.189:0/601460877 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff13806ade0 0x7ff13806d2a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff150000c00 tx=0x7ff150009040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:29.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.587+0000 7ff149ffb700 1 -- 172.21.15.189:0/601460877 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff1541f6850 con 0x7ff15406e1f0 2024-04-06T13:47:29.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:29.835+0000 7ff15b3f4700 1 -- 172.21.15.189:0/601460877 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff1541f6850 con 0x7ff15406e1f0 2024-04-06T13:47:30.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.156+0000 7ff149ffb700 1 -- 172.21.15.189:0/601460877 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v101) v1 ==== 136+0+0 (secure 0 0 0) 0x7ff1541f6850 con 0x7ff15406e1f0 2024-04-06T13:47:30.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.158+0000 7ff15b3f4700 1 -- 172.21.15.189:0/601460877 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff13806ade0 msgr2=0x7ff13806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:30.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.158+0000 7ff15b3f4700 1 --2- 172.21.15.189:0/601460877 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff13806ade0 0x7ff13806d2a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff150000c00 tx=0x7ff150009040 comp rx=0 tx=0).stop 2024-04-06T13:47:30.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.158+0000 7ff15b3f4700 1 -- 172.21.15.189:0/601460877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff15406e1f0 msgr2=0x7ff1541f6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:30.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.158+0000 7ff15b3f4700 1 --2- 172.21.15.189:0/601460877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff15406e1f0 0x7ff1541f6b30 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7ff15406cc10 tx=0x7ff14c011f80 comp rx=0 tx=0).stop 2024-04-06T13:47:30.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.158+0000 7ff15b3f4700 1 -- 172.21.15.189:0/601460877 shutdown_connections 2024-04-06T13:47:30.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.158+0000 7ff15b3f4700 1 --2- 172.21.15.189:0/601460877 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff13806ade0 0x7ff13806d2a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.158+0000 7ff15b3f4700 1 --2- 172.21.15.189:0/601460877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff15406e1f0 0x7ff1541f6b30 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.158+0000 7ff15b3f4700 1 --2- 172.21.15.189:0/601460877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff140003f80 0x7ff1541f3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.158+0000 7ff15b3f4700 1 --2- 172.21.15.189:0/601460877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff154158d20 0x7ff154152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.158+0000 7ff15b3f4700 1 -- 172.21.15.189:0/601460877 >> 172.21.15.189:0/601460877 conn(0x7ff1540cc290 msgr2=0x7ff1540afc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:30.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.158+0000 7ff15b3f4700 1 -- 172.21.15.189:0/601460877 shutdown_connections 2024-04-06T13:47:30.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.158+0000 7ff15b3f4700 1 -- 172.21.15.189:0/601460877 wait complete. 2024-04-06T13:47:30.200 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:47:30.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.322+0000 7f61b88ec700 1 Processor -- start 2024-04-06T13:47:30.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.322+0000 7f61b88ec700 1 -- start start 2024-04-06T13:47:30.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.323+0000 7f61b88ec700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b415d7b0 0x7f61b40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:30.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.323+0000 7f61b88ec700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f61b406a730 0x7f61b406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:30.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.323+0000 7f61b88ec700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b40b3990 con 0x7f61b415ddf0 2024-04-06T13:47:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.323+0000 7f61b88ec700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b406b0f0 con 0x7f61b406a730 2024-04-06T13:47:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.323+0000 7f61b88ec700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40b3060 con 0x7f61b415d7b0 2024-04-06T13:47:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.324+0000 7f61b259c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b415d7b0 0x7f61b40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.324+0000 7f61b1d9b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f61b406a730 0x7f61b406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.324+0000 7f61b1d9b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f61b406a730 0x7f61b406abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:33000/0 (socket says 172.21.15.189:33000) 2024-04-06T13:47:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.324+0000 7f61b259c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b415d7b0 0x7f61b40b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:38316/0 (socket says 172.21.15.189:38316) 2024-04-06T13:47:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.324+0000 7f61b1d9b700 1 -- 172.21.15.189:0/2579300459 learned_addr learned my addr 172.21.15.189:0/2579300459 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.324+0000 7f61b2d9d700 1 --1- 172.21.15.189:0/2579300459 >> v1:172.21.15.192:6789/0 conn(0x7f61b415ddf0 0x7f61b415e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:41486/0 (socket says 172.21.15.189:41486) 2024-04-06T13:47:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.324+0000 7f61b1d9b700 1 -- 172.21.15.189:0/2579300459 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b415d7b0 msgr2=0x7f61b40b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.324+0000 7f61b1d9b700 1 --2- 172.21.15.189:0/2579300459 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b415d7b0 0x7f61b40b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.324+0000 7f61b1d9b700 1 -- 172.21.15.189:0/2579300459 >> v1:172.21.15.192:6789/0 conn(0x7f61b415ddf0 legacy=0x7f61b415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.324+0000 7f61b1d9b700 1 -- 172.21.15.189:0/2579300459 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b40b3340 con 0x7f61b406a730 2024-04-06T13:47:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.324+0000 7f61b259c700 1 --2- 172.21.15.189:0/2579300459 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b415d7b0 0x7f61b40b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:30.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.324+0000 7f61b1d9b700 1 --2- 172.21.15.189:0/2579300459 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61b406a730 0x7f61b406abb0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f61a40098f0 tx=0x7f61a4009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d49c218eda514ae9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:30.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.325+0000 7f61b159a700 1 -- 172.21.15.189:0/2579300459 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a4011830 con 0x7f61b406a730 2024-04-06T13:47:30.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.325+0000 7f61b159a700 1 -- 172.21.15.189:0/2579300459 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a4011990 con 0x7f61b406a730 2024-04-06T13:47:30.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.325+0000 7f61b159a700 1 -- 172.21.15.189:0/2579300459 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a4011d20 con 0x7f61b406a730 2024-04-06T13:47:30.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.325+0000 7f61b88ec700 1 -- 172.21.15.189:0/2579300459 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61b406a730 msgr2=0x7f61b406abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:30.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.325+0000 7f61b88ec700 1 --2- 172.21.15.189:0/2579300459 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61b406a730 0x7f61b406abb0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f61a40098f0 tx=0x7f61a4009c00 comp rx=0 tx=0).stop 2024-04-06T13:47:30.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.325+0000 7f61b88ec700 1 -- 172.21.15.189:0/2579300459 shutdown_connections 2024-04-06T13:47:30.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.325+0000 7f61b88ec700 1 --2- 172.21.15.189:0/2579300459 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b415d7b0 0x7f61b40b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.325+0000 7f61b88ec700 1 --2- 172.21.15.189:0/2579300459 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61b406a730 0x7f61b406abb0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.325+0000 7f61b88ec700 1 -- 172.21.15.189:0/2579300459 >> 172.21.15.189:0/2579300459 conn(0x7f61b40d00b0 msgr2=0x7f61b40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:30.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.325+0000 7f61b88ec700 1 -- 172.21.15.189:0/2579300459 shutdown_connections 2024-04-06T13:47:30.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.325+0000 7f61b88ec700 1 -- 172.21.15.189:0/2579300459 wait complete. 2024-04-06T13:47:30.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.327+0000 7f61b88ec700 1 Processor -- start 2024-04-06T13:47:30.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.327+0000 7f61b88ec700 1 -- start start 2024-04-06T13:47:30.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.327+0000 7f61b88ec700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b406a730 0x7f61b40af090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:30.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.327+0000 7f61b88ec700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b415d7b0 0x7f61b41f6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:30.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.327+0000 7f61b88ec700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61b415ddf0 0x7f61b4146630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:30.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.327+0000 7f61b88ec700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b415af70 con 0x7f61b406a730 2024-04-06T13:47:30.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.327+0000 7f61b88ec700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b41f71c0 con 0x7f61b415ddf0 2024-04-06T13:47:30.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.327+0000 7f61b88ec700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b41f74a0 con 0x7f61b415d7b0 2024-04-06T13:47:30.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.327+0000 7f61b259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b406a730 0x7f61b40af090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:30.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.327+0000 7f61b259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b406a730 0x7f61b40af090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54844/0 (socket says 172.21.15.189:54844) 2024-04-06T13:47:30.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.327+0000 7f61b259c700 1 -- 172.21.15.189:0/1719633232 learned_addr learned my addr 172.21.15.189:0/1719633232 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:30.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.328+0000 7f61b1d9b700 1 --2- 172.21.15.189:0/1719633232 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b415d7b0 0x7f61b41f6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:30.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.328+0000 7f61b2d9d700 1 --2- 172.21.15.189:0/1719633232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61b415ddf0 0x7f61b4146630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:30.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.328+0000 7f61b259c700 1 -- 172.21.15.189:0/1719633232 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b415d7b0 msgr2=0x7f61b41f6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:30.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.328+0000 7f61b259c700 1 --2- 172.21.15.189:0/1719633232 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b415d7b0 0x7f61b41f6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.328+0000 7f61b259c700 1 -- 172.21.15.189:0/1719633232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61b415ddf0 msgr2=0x7f61b4146630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:30.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.328+0000 7f61b259c700 1 --2- 172.21.15.189:0/1719633232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61b415ddf0 0x7f61b4146630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.328+0000 7f61b259c700 1 -- 172.21.15.189:0/1719633232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619c0094a0 con 0x7f61b406a730 2024-04-06T13:47:30.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.328+0000 7f61b2d9d700 1 --2- 172.21.15.189:0/1719633232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61b415ddf0 0x7f61b4146630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:30.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.328+0000 7f61b259c700 1 --2- 172.21.15.189:0/1719633232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b406a730 0x7f61b40af090 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f619c00ebe0 tx=0x7f619c009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:30.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.328+0000 7f61aaffd700 1 -- 172.21.15.189:0/1719633232 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619c00c910 con 0x7f61b406a730 2024-04-06T13:47:30.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.328+0000 7f61aaffd700 1 -- 172.21.15.189:0/1719633232 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619c00ca70 con 0x7f61b406a730 2024-04-06T13:47:30.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.328+0000 7f61aaffd700 1 -- 172.21.15.189:0/1719633232 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619c00cde0 con 0x7f61b406a730 2024-04-06T13:47:30.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.328+0000 7f61b88ec700 1 -- 172.21.15.189:0/1719633232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a4009550 con 0x7f61b406a730 2024-04-06T13:47:30.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.328+0000 7f61b88ec700 1 -- 172.21.15.189:0/1719633232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b415b4f0 con 0x7f61b406a730 2024-04-06T13:47:30.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.329+0000 7f61b88ec700 1 -- 172.21.15.189:0/1719633232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b415b680 con 0x7f61b406a730 2024-04-06T13:47:30.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.330+0000 7f61aaffd700 1 -- 172.21.15.189:0/1719633232 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f619c013470 con 0x7f61b406a730 2024-04-06T13:47:30.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.330+0000 7f61aaffd700 1 --2- 172.21.15.189:0/1719633232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f619406adb0 0x7f619406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:30.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.330+0000 7f61aaffd700 1 -- 172.21.15.189:0/1719633232 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f619c089880 con 0x7f61b406a730 2024-04-06T13:47:30.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.330+0000 7f61b1d9b700 1 --2- 172.21.15.189:0/1719633232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f619406adb0 0x7f619406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:30.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.330+0000 7f61b1d9b700 1 --2- 172.21.15.189:0/1719633232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f619406adb0 0x7f619406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009c40 tx=0x7f61a4008030 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:30.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.334+0000 7f61aaffd700 1 -- 172.21.15.189:0/1719633232 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f619c054740 con 0x7f61b406a730 2024-04-06T13:47:30.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.528+0000 7f61b88ec700 1 -- 172.21.15.189:0/1719633232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61b4067bb0 con 0x7f61b406a730 2024-04-06T13:47:30.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.529+0000 7f61aaffd700 1 -- 172.21.15.189:0/1719633232 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v227) v1 ==== 74+0+16480 (secure 0 0 0) 0x7f619c057d60 con 0x7f61b406a730 2024-04-06T13:47:30.530 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:30.530 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":227,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:47:23.118493+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:47:20.644275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-04-06T13:47:21.075143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:47:30.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.532+0000 7f61b88ec700 1 -- 172.21.15.189:0/1719633232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f619406adb0 msgr2=0x7f619406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:30.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.532+0000 7f61b88ec700 1 --2- 172.21.15.189:0/1719633232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f619406adb0 0x7f619406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009c40 tx=0x7f61a4008030 comp rx=0 tx=0).stop 2024-04-06T13:47:30.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.532+0000 7f61b88ec700 1 -- 172.21.15.189:0/1719633232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b406a730 msgr2=0x7f61b40af090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:30.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.532+0000 7f61b88ec700 1 --2- 172.21.15.189:0/1719633232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b406a730 0x7f61b40af090 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f619c00ebe0 tx=0x7f619c009ec0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.532+0000 7f61b88ec700 1 -- 172.21.15.189:0/1719633232 shutdown_connections 2024-04-06T13:47:30.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.532+0000 7f61b88ec700 1 --2- 172.21.15.189:0/1719633232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f619406adb0 0x7f619406d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.532+0000 7f61b88ec700 1 --2- 172.21.15.189:0/1719633232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61b415ddf0 0x7f61b4146630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.532+0000 7f61b88ec700 1 --2- 172.21.15.189:0/1719633232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b406a730 0x7f61b40af090 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.532+0000 7f61b88ec700 1 --2- 172.21.15.189:0/1719633232 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61b415d7b0 0x7f61b41f6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.532+0000 7f61b88ec700 1 -- 172.21.15.189:0/1719633232 >> 172.21.15.189:0/1719633232 conn(0x7f61b40d00b0 msgr2=0x7f61b40ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:30.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.532+0000 7f61b88ec700 1 -- 172.21.15.189:0/1719633232 shutdown_connections 2024-04-06T13:47:30.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.532+0000 7f61b88ec700 1 -- 172.21.15.189:0/1719633232 wait complete. 2024-04-06T13:47:30.546 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T13:47:20.644275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '226', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T13:47:21.075143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '227', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:47:30.546 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:47:30.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.671+0000 7f6ae531e700 1 Processor -- start 2024-04-06T13:47:30.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.671+0000 7f6ae531e700 1 -- start start 2024-04-06T13:47:30.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.672+0000 7f6ae531e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6ae0144f40 0x7f6ae006a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:30.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.672+0000 7f6ae531e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae015c740 con 0x7f6ae015fcd0 2024-04-06T13:47:30.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.672+0000 7f6ae531e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae015f1f0 con 0x7f6ae015c150 2024-04-06T13:47:30.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.672+0000 7f6ae531e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ae015ca20 con 0x7f6ae0144f40 2024-04-06T13:47:30.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.672+0000 7f6adeffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6ae0144f40 0x7f6ae006a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:30.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.672+0000 7f6ade7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6ae015fcd0 0x7f6ae015ba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55114/0 (socket says 172.21.15.189:55114) 2024-04-06T13:47:30.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.672+0000 7f6ade7fc700 1 -- 172.21.15.189:0/1910350286 learned_addr learned my addr 172.21.15.189:0/1910350286 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:30.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.673+0000 7f6adeffd700 1 -- 172.21.15.189:0/1910350286 >> v1:172.21.15.192:6789/0 conn(0x7f6ae015c150 legacy=0x7f6ae00ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:30.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.673+0000 7f6adeffd700 1 -- 172.21.15.189:0/1910350286 >> v1:172.21.15.189:6789/0 conn(0x7f6ae015fcd0 legacy=0x7f6ae015ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:30.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.673+0000 7f6adeffd700 1 -- 172.21.15.189:0/1910350286 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae015f4d0 con 0x7f6ae0144f40 2024-04-06T13:47:30.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.673+0000 7f6addffb700 1 -- 172.21.15.189:0/1910350286 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390582038 0 0) 0x7f6ae015c740 con 0x7f6ae015fcd0 2024-04-06T13:47:30.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.673+0000 7f6adeffd700 1 --2- 172.21.15.189:0/1910350286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ae0144f40 0x7f6ae006a730 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009840 tx=0x7f6ad4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d66b7070f5c2bab server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:30.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.673+0000 7f6addffb700 1 -- 172.21.15.189:0/1910350286 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4004230 con 0x7f6ae0144f40 2024-04-06T13:47:30.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.673+0000 7f6addffb700 1 -- 172.21.15.189:0/1910350286 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad4004390 con 0x7f6ae0144f40 2024-04-06T13:47:30.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.673+0000 7f6addffb700 1 -- 172.21.15.189:0/1910350286 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad40036a0 con 0x7f6ae0144f40 2024-04-06T13:47:30.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.673+0000 7f6ae531e700 1 -- 172.21.15.189:0/1910350286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ae0144f40 msgr2=0x7f6ae006a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:30.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.673+0000 7f6ae531e700 1 --2- 172.21.15.189:0/1910350286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ae0144f40 0x7f6ae006a730 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009840 tx=0x7f6ad4009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:30.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.673+0000 7f6ae531e700 1 -- 172.21.15.189:0/1910350286 shutdown_connections 2024-04-06T13:47:30.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.674+0000 7f6ae531e700 1 --2- 172.21.15.189:0/1910350286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ae0144f40 0x7f6ae006a730 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.674+0000 7f6ae531e700 1 -- 172.21.15.189:0/1910350286 >> 172.21.15.189:0/1910350286 conn(0x7f6ae00d00b0 msgr2=0x7f6ae00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:30.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.674+0000 7f6ae531e700 1 -- 172.21.15.189:0/1910350286 shutdown_connections 2024-04-06T13:47:30.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.674+0000 7f6ae531e700 1 -- 172.21.15.189:0/1910350286 wait complete. 2024-04-06T13:47:30.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.675+0000 7f6ae531e700 1 Processor -- start 2024-04-06T13:47:30.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.675+0000 7f6ae531e700 1 -- start start 2024-04-06T13:47:30.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6ae531e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ae0144f40 0x7f6ae013eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:30.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6ae531e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ae015c150 0x7f6ae00cb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:30.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6ae531e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ae015fcd0 0x7f6ae00b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:30.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6ae531e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae013f0c0 con 0x7f6ae015fcd0 2024-04-06T13:47:30.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6ae531e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae0142250 con 0x7f6ae0144f40 2024-04-06T13:47:30.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6ae531e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae00cbc80 con 0x7f6ae015c150 2024-04-06T13:47:30.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6adf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ae015fcd0 0x7f6ae00b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:30.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6adf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ae015fcd0 0x7f6ae00b30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54862/0 (socket says 172.21.15.189:54862) 2024-04-06T13:47:30.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6adf7fe700 1 -- 172.21.15.189:0/1771184269 learned_addr learned my addr 172.21.15.189:0/1771184269 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:30.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6adeffd700 1 --2- 172.21.15.189:0/1771184269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ae0144f40 0x7f6ae013eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:30.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6ade7fc700 1 --2- 172.21.15.189:0/1771184269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ae015c150 0x7f6ae00cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:30.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6adf7fe700 1 -- 172.21.15.189:0/1771184269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ae015c150 msgr2=0x7f6ae00cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:30.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6adf7fe700 1 --2- 172.21.15.189:0/1771184269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ae015c150 0x7f6ae00cb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6adf7fe700 1 -- 172.21.15.189:0/1771184269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ae0144f40 msgr2=0x7f6ae013eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:30.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6adf7fe700 1 --2- 172.21.15.189:0/1771184269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ae0144f40 0x7f6ae013eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6adf7fe700 1 -- 172.21.15.189:0/1771184269 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad40094a0 con 0x7f6ae015fcd0 2024-04-06T13:47:30.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6adf7fe700 1 --2- 172.21.15.189:0/1771184269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ae015fcd0 0x7f6ae00b30c0 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f6ad800cc10 tx=0x7f6ad800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:30.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6ac77fe700 1 -- 172.21.15.189:0/1771184269 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad800abb0 con 0x7f6ae015fcd0 2024-04-06T13:47:30.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6ac77fe700 1 -- 172.21.15.189:0/1771184269 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad800ad10 con 0x7f6ae015fcd0 2024-04-06T13:47:30.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6ac77fe700 1 -- 172.21.15.189:0/1771184269 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8012660 con 0x7f6ae015fcd0 2024-04-06T13:47:30.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.676+0000 7f6ae531e700 1 -- 172.21.15.189:0/1771184269 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae0142510 con 0x7f6ae015fcd0 2024-04-06T13:47:30.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.677+0000 7f6ae531e700 1 -- 172.21.15.189:0/1771184269 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae0142ad0 con 0x7f6ae015fcd0 2024-04-06T13:47:30.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.677+0000 7f6ae531e700 1 -- 172.21.15.189:0/1771184269 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae0067bb0 con 0x7f6ae015fcd0 2024-04-06T13:47:30.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.678+0000 7f6ac77fe700 1 -- 172.21.15.189:0/1771184269 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f6ad80127c0 con 0x7f6ae015fcd0 2024-04-06T13:47:30.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.678+0000 7f6ac77fe700 1 --2- 172.21.15.189:0/1771184269 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6acc06adb0 0x7f6acc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:30.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.678+0000 7f6ac77fe700 1 -- 172.21.15.189:0/1771184269 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6ad8089d60 con 0x7f6ae015fcd0 2024-04-06T13:47:30.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.678+0000 7f6adeffd700 1 --2- 172.21.15.189:0/1771184269 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6acc06adb0 0x7f6acc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:30.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.679+0000 7f6adeffd700 1 --2- 172.21.15.189:0/1771184269 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6acc06adb0 0x7f6acc06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0144b10 tx=0x7f6ad401c040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:30.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.683+0000 7f6ac77fe700 1 -- 172.21.15.189:0/1771184269 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6ad8054d30 con 0x7f6ae015fcd0 2024-04-06T13:47:30.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.875+0000 7f6ae531e700 1 -- 172.21.15.189:0/1771184269 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6ae0142cf0 con 0x7f6ae015fcd0 2024-04-06T13:47:30.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.876+0000 7f6ac77fe700 1 -- 172.21.15.189:0/1771184269 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v227) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6ad8016070 con 0x7f6ae015fcd0 2024-04-06T13:47:30.876 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:47:30.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.878+0000 7f6ae531e700 1 -- 172.21.15.189:0/1771184269 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6acc06adb0 msgr2=0x7f6acc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:30.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.878+0000 7f6ae531e700 1 --2- 172.21.15.189:0/1771184269 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6acc06adb0 0x7f6acc06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0144b10 tx=0x7f6ad401c040 comp rx=0 tx=0).stop 2024-04-06T13:47:30.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.878+0000 7f6ae531e700 1 -- 172.21.15.189:0/1771184269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ae015fcd0 msgr2=0x7f6ae00b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:30.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.878+0000 7f6ae531e700 1 --2- 172.21.15.189:0/1771184269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ae015fcd0 0x7f6ae00b30c0 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f6ad800cc10 tx=0x7f6ad800a060 comp rx=0 tx=0).stop 2024-04-06T13:47:30.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.878+0000 7f6ae531e700 1 -- 172.21.15.189:0/1771184269 shutdown_connections 2024-04-06T13:47:30.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.878+0000 7f6ae531e700 1 --2- 172.21.15.189:0/1771184269 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6acc06adb0 0x7f6acc06d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.878+0000 7f6ae531e700 1 --2- 172.21.15.189:0/1771184269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ae015fcd0 0x7f6ae00b30c0 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.878+0000 7f6ae531e700 1 --2- 172.21.15.189:0/1771184269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ae0144f40 0x7f6ae013eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.878+0000 7f6ae531e700 1 --2- 172.21.15.189:0/1771184269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ae015c150 0x7f6ae00cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:30.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.878+0000 7f6ae531e700 1 -- 172.21.15.189:0/1771184269 >> 172.21.15.189:0/1771184269 conn(0x7f6ae00d00b0 msgr2=0x7f6ae00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:30.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.878+0000 7f6ae531e700 1 -- 172.21.15.189:0/1771184269 shutdown_connections 2024-04-06T13:47:30.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:30.878+0000 7f6ae531e700 1 -- 172.21.15.189:0/1771184269 wait complete. 2024-04-06T13:47:30.890 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T13:47:31.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.016+0000 7f68a626a700 1 Processor -- start 2024-04-06T13:47:31.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.016+0000 7f68a626a700 1 -- start start 2024-04-06T13:47:31.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.017+0000 7f68a626a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68a00d17a0 0x7f68a006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.017+0000 7f68a626a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a006d850 con 0x7f68a0146dd0 2024-04-06T13:47:31.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.017+0000 7f68a626a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a006c0a0 con 0x7f68a00cedf0 2024-04-06T13:47:31.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.017+0000 7f68a626a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a006c380 con 0x7f68a00d17a0 2024-04-06T13:47:31.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.017+0000 7f689f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f68a0146dd0 0x7f68a006d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55120/0 (socket says 172.21.15.189:55120) 2024-04-06T13:47:31.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.017+0000 7f689f7fe700 1 -- 172.21.15.189:0/1252615379 learned_addr learned my addr 172.21.15.189:0/1252615379 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:31.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.017+0000 7f689effd700 1 -- 172.21.15.189:0/1252615379 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750731310 0 0) 0x7f68a006d850 con 0x7f68a0146dd0 2024-04-06T13:47:31.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.017+0000 7f689ffff700 1 --2- 172.21.15.189:0/1252615379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68a00d17a0 0x7f68a006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.017+0000 7f689effd700 1 -- 172.21.15.189:0/1252615379 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68880035b0 con 0x7f68a0146dd0 2024-04-06T13:47:31.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f689effd700 1 -- 172.21.15.189:0/1252615379 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 909524259 0 0) 0x7f68880035b0 con 0x7f68a0146dd0 2024-04-06T13:47:31.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f689effd700 1 -- 172.21.15.189:0/1252615379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68a00d17a0 msgr2=0x7f68a006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:31.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f689effd700 1 --2- 172.21.15.189:0/1252615379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68a00d17a0 0x7f68a006b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f689effd700 1 -- 172.21.15.189:0/1252615379 >> v1:172.21.15.192:6789/0 conn(0x7f68a00cedf0 legacy=0x7f68a01441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f689effd700 1 -- 172.21.15.189:0/1252615379 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a006c660 con 0x7f68a0146dd0 2024-04-06T13:47:31.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f689effd700 1 -- 172.21.15.189:0/1252615379 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f6890001670 con 0x7f68a0146dd0 2024-04-06T13:47:31.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f689effd700 1 -- 172.21.15.189:0/1252615379 >> v1:172.21.15.189:6789/0 conn(0x7f68a0146dd0 legacy=0x7f68a006d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f689effd700 1 --2- 172.21.15.189:0/1252615379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6888003a30 0x7f6888003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f689effd700 1 --2- 172.21.15.189:0/1252615379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68a00d17a0 0x7f6888004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f689effd700 1 --2- 172.21.15.189:0/1252615379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6888004d90 0x7f6888005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f689effd700 1 -- 172.21.15.189:0/1252615379 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a006c380 con 0x7f6888004d90 2024-04-06T13:47:31.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f689effd700 1 -- 172.21.15.189:0/1252615379 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688800bac0 con 0x7f6888003a30 2024-04-06T13:47:31.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f689effd700 1 -- 172.21.15.189:0/1252615379 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f688800bc80 con 0x7f68a00d17a0 2024-04-06T13:47:31.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f68a4807700 1 --2- 172.21.15.189:0/1252615379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6888004d90 0x7f6888005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f689ffff700 1 --2- 172.21.15.189:0/1252615379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6888003a30 0x7f6888003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f689f7fe700 1 --2- 172.21.15.189:0/1252615379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68a00d17a0 0x7f6888004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f68a4807700 1 -- 172.21.15.189:0/1252615379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68a00d17a0 msgr2=0x7f6888004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:31.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f68a4807700 1 --2- 172.21.15.189:0/1252615379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68a00d17a0 0x7f6888004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f68a4807700 1 -- 172.21.15.189:0/1252615379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6888003a30 msgr2=0x7f6888003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:31.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f68a4807700 1 --2- 172.21.15.189:0/1252615379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6888003a30 0x7f6888003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f68a4807700 1 -- 172.21.15.189:0/1252615379 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68880035b0 con 0x7f6888004d90 2024-04-06T13:47:31.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f68a4807700 1 --2- 172.21.15.189:0/1252615379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6888004d90 0x7f6888005210 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f689800c160 tx=0x7f6888005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b77bf49378b0ad server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:31.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f689effd700 1 -- 172.21.15.189:0/1252615379 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6898001db0 con 0x7f6888004d90 2024-04-06T13:47:31.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f689effd700 1 -- 172.21.15.189:0/1252615379 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689800cca0 con 0x7f6888004d90 2024-04-06T13:47:31.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.018+0000 7f689effd700 1 -- 172.21.15.189:0/1252615379 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6898003c40 con 0x7f6888004d90 2024-04-06T13:47:31.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.019+0000 7f68a626a700 1 -- 172.21.15.189:0/1252615379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6888004d90 msgr2=0x7f6888005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.019+0000 7f68a626a700 1 --2- 172.21.15.189:0/1252615379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6888004d90 0x7f6888005210 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f689800c160 tx=0x7f6888005880 comp rx=0 tx=0).stop 2024-04-06T13:47:31.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.019+0000 7f68a4807700 1 -- 172.21.15.189:0/1252615379 reap_dead start 2024-04-06T13:47:31.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.019+0000 7f68a626a700 1 -- 172.21.15.189:0/1252615379 shutdown_connections 2024-04-06T13:47:31.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.019+0000 7f68a626a700 1 -- 172.21.15.189:0/1252615379 >> 172.21.15.189:0/1252615379 conn(0x7f68a00b2db0 msgr2=0x7f68a00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:31.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.019+0000 7f68a626a700 1 -- 172.21.15.189:0/1252615379 shutdown_connections 2024-04-06T13:47:31.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.019+0000 7f68a626a700 1 -- 172.21.15.189:0/1252615379 wait complete. 2024-04-06T13:47:31.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.020+0000 7f68a626a700 1 Processor -- start 2024-04-06T13:47:31.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.021+0000 7f68a626a700 1 -- start start 2024-04-06T13:47:31.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f68a626a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a015a560 0x7f68a00b6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f68a626a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68a00b6bb0 0x7f68a00c5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f68a626a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68a00c4790 0x7f68a00c57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f68a626a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a0160440 con 0x7f68a015a560 2024-04-06T13:47:31.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f68a626a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a015ac90 con 0x7f68a00c4790 2024-04-06T13:47:31.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f68a626a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a015af70 con 0x7f68a00b6bb0 2024-04-06T13:47:31.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f689ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a015a560 0x7f68a00b6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f689ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a015a560 0x7f68a00b6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54882/0 (socket says 172.21.15.189:54882) 2024-04-06T13:47:31.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f689ffff700 1 -- 172.21.15.189:0/320807690 learned_addr learned my addr 172.21.15.189:0/320807690 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:31.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f68a4807700 1 --2- 172.21.15.189:0/320807690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68a00c4790 0x7f68a00c57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f689f7fe700 1 --2- 172.21.15.189:0/320807690 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68a00b6bb0 0x7f68a00c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f689ffff700 1 -- 172.21.15.189:0/320807690 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68a00b6bb0 msgr2=0x7f68a00c5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f689ffff700 1 --2- 172.21.15.189:0/320807690 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68a00b6bb0 0x7f68a00c5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f689ffff700 1 -- 172.21.15.189:0/320807690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68a00c4790 msgr2=0x7f68a00c57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f689ffff700 1 --2- 172.21.15.189:0/320807690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68a00c4790 0x7f68a00c57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f689ffff700 1 -- 172.21.15.189:0/320807690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a00b6150 con 0x7f68a015a560 2024-04-06T13:47:31.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f689ffff700 1 --2- 172.21.15.189:0/320807690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a015a560 0x7f68a00b6670 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7f6894011aa0 tx=0x7f6894011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:31.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f689cff9700 1 -- 172.21.15.189:0/320807690 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689400f980 con 0x7f68a015a560 2024-04-06T13:47:31.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f689cff9700 1 -- 172.21.15.189:0/320807690 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689400fae0 con 0x7f68a015a560 2024-04-06T13:47:31.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f68a626a700 1 -- 172.21.15.189:0/320807690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a015a940 con 0x7f68a015a560 2024-04-06T13:47:31.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f689cff9700 1 -- 172.21.15.189:0/320807690 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689400fe90 con 0x7f68a015a560 2024-04-06T13:47:31.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.022+0000 7f68a626a700 1 -- 172.21.15.189:0/320807690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a00b0210 con 0x7f68a015a560 2024-04-06T13:47:31.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.023+0000 7f68a626a700 1 -- 172.21.15.189:0/320807690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a0144880 con 0x7f68a015a560 2024-04-06T13:47:31.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.024+0000 7f689cff9700 1 -- 172.21.15.189:0/320807690 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f6894014440 con 0x7f68a015a560 2024-04-06T13:47:31.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.024+0000 7f689cff9700 1 --2- 172.21.15.189:0/320807690 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f688c06adb0 0x7f688c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.024+0000 7f689cff9700 1 -- 172.21.15.189:0/320807690 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6894089e10 con 0x7f68a015a560 2024-04-06T13:47:31.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.024+0000 7f689f7fe700 1 --2- 172.21.15.189:0/320807690 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f688c06adb0 0x7f688c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.025+0000 7f689f7fe700 1 --2- 172.21.15.189:0/320807690 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f688c06adb0 0x7f688c06d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6890000c00 tx=0x7f6890009040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:31.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.029+0000 7f689cff9700 1 -- 172.21.15.189:0/320807690 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6894054e10 con 0x7f68a015a560 2024-04-06T13:47:31.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.223+0000 7f68a626a700 1 -- 172.21.15.189:0/320807690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f68a0074990 con 0x7f68a015a560 2024-04-06T13:47:31.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.224+0000 7f689cff9700 1 -- 172.21.15.189:0/320807690 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v227) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6894058430 con 0x7f68a015a560 2024-04-06T13:47:31.224 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:47:31.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.226+0000 7f68a626a700 1 -- 172.21.15.189:0/320807690 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f688c06adb0 msgr2=0x7f688c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.226+0000 7f68a626a700 1 --2- 172.21.15.189:0/320807690 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f688c06adb0 0x7f688c06d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6890000c00 tx=0x7f6890009040 comp rx=0 tx=0).stop 2024-04-06T13:47:31.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.226+0000 7f68a626a700 1 -- 172.21.15.189:0/320807690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a015a560 msgr2=0x7f68a00b6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.226+0000 7f68a626a700 1 --2- 172.21.15.189:0/320807690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a015a560 0x7f68a00b6670 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7f6894011aa0 tx=0x7f6894011e60 comp rx=0 tx=0).stop 2024-04-06T13:47:31.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.226+0000 7f68a626a700 1 -- 172.21.15.189:0/320807690 shutdown_connections 2024-04-06T13:47:31.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.226+0000 7f68a626a700 1 --2- 172.21.15.189:0/320807690 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f688c06adb0 0x7f688c06d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.226+0000 7f68a626a700 1 --2- 172.21.15.189:0/320807690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68a00c4790 0x7f68a00c57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.226+0000 7f68a626a700 1 --2- 172.21.15.189:0/320807690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a015a560 0x7f68a00b6670 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.226+0000 7f68a626a700 1 --2- 172.21.15.189:0/320807690 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68a00b6bb0 0x7f68a00c5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.226+0000 7f68a626a700 1 -- 172.21.15.189:0/320807690 >> 172.21.15.189:0/320807690 conn(0x7f68a00b2db0 msgr2=0x7f68a00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:31.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.226+0000 7f68a626a700 1 -- 172.21.15.189:0/320807690 shutdown_connections 2024-04-06T13:47:31.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.226+0000 7f68a626a700 1 -- 172.21.15.189:0/320807690 wait complete. 2024-04-06T13:47:31.238 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T13:47:31.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.364+0000 7f0dfacc2700 1 Processor -- start 2024-04-06T13:47:31.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.364+0000 7f0dfacc2700 1 -- start start 2024-04-06T13:47:31.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.365+0000 7f0dfacc2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0df40d17a0 0x7f0df406b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.365+0000 7f0dfacc2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0df4146dd0 0x7f0df406d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.365+0000 7f0dfacc2700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df406c0a0 con 0x7f0df40cedf0 2024-04-06T13:47:31.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.365+0000 7f0dfacc2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0df406c380 con 0x7f0df4146dd0 2024-04-06T13:47:31.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.365+0000 7f0dfacc2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df406c660 con 0x7f0df40d17a0 2024-04-06T13:47:31.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.365+0000 7f0df8a5e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0df40d17a0 0x7f0df406b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.365+0000 7f0df3fff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0df4146dd0 0x7f0df406d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.366+0000 7f0df8a5e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0df40d17a0 0x7f0df406b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:38356/0 (socket says 172.21.15.189:38356) 2024-04-06T13:47:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.366+0000 7f0df8a5e700 1 -- 172.21.15.189:0/3274544148 learned_addr learned my addr 172.21.15.189:0/3274544148 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.366+0000 7f0df8a5e700 1 -- 172.21.15.189:0/3274544148 >> v2:172.21.15.192:3300/0 conn(0x7f0df4146dd0 msgr2=0x7f0df406d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.366+0000 7f0df8a5e700 1 --2- 172.21.15.189:0/3274544148 >> v2:172.21.15.192:3300/0 conn(0x7f0df4146dd0 0x7f0df406d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.366+0000 7f0df8a5e700 1 -- 172.21.15.189:0/3274544148 >> v1:172.21.15.192:6789/0 conn(0x7f0df40cedf0 legacy=0x7f0df41441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.366+0000 7f0df8a5e700 1 -- 172.21.15.189:0/3274544148 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df406c940 con 0x7f0df40d17a0 2024-04-06T13:47:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.366+0000 7f0df8a5e700 1 --2- 172.21.15.189:0/3274544148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0df40d17a0 0x7f0df406b7a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0de80098f0 tx=0x7f0de8009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63c50ccaac848a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.366+0000 7f0df37fe700 1 -- 172.21.15.189:0/3274544148 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de8011830 con 0x7f0df40d17a0 2024-04-06T13:47:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.366+0000 7f0df37fe700 1 -- 172.21.15.189:0/3274544148 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de8011990 con 0x7f0df40d17a0 2024-04-06T13:47:31.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.366+0000 7f0df37fe700 1 -- 172.21.15.189:0/3274544148 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de8011d20 con 0x7f0df40d17a0 2024-04-06T13:47:31.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.367+0000 7f0dfacc2700 1 -- 172.21.15.189:0/3274544148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0df40d17a0 msgr2=0x7f0df406b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.367+0000 7f0dfacc2700 1 --2- 172.21.15.189:0/3274544148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0df40d17a0 0x7f0df406b7a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0de80098f0 tx=0x7f0de8009c00 comp rx=0 tx=0).stop 2024-04-06T13:47:31.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.367+0000 7f0dfacc2700 1 -- 172.21.15.189:0/3274544148 shutdown_connections 2024-04-06T13:47:31.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.367+0000 7f0dfacc2700 1 --2- 172.21.15.189:0/3274544148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0df40d17a0 0x7f0df406b7a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.367+0000 7f0dfacc2700 1 --2- 172.21.15.189:0/3274544148 >> v2:172.21.15.192:3300/0 conn(0x7f0df4146dd0 0x7f0df406d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.367+0000 7f0dfacc2700 1 -- 172.21.15.189:0/3274544148 >> 172.21.15.189:0/3274544148 conn(0x7f0df40b2db0 msgr2=0x7f0df40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:31.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.367+0000 7f0dfacc2700 1 -- 172.21.15.189:0/3274544148 shutdown_connections 2024-04-06T13:47:31.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.367+0000 7f0dfacc2700 1 -- 172.21.15.189:0/3274544148 wait complete. 2024-04-06T13:47:31.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.368+0000 7f0dfacc2700 1 Processor -- start 2024-04-06T13:47:31.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.368+0000 7f0dfacc2700 1 -- start start 2024-04-06T13:47:31.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.369+0000 7f0dfacc2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0df40cedf0 0x7f0df415a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.369+0000 7f0dfacc2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0df40d17a0 0x7f0df40b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.369+0000 7f0dfacc2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0df4146dd0 0x7f0df40b5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.369+0000 7f0dfacc2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df40b3650 con 0x7f0df40cedf0 2024-04-06T13:47:31.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.369+0000 7f0dfacc2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df40b6ce0 con 0x7f0df40d17a0 2024-04-06T13:47:31.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.369+0000 7f0dfacc2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df40c6040 con 0x7f0df4146dd0 2024-04-06T13:47:31.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.369+0000 7f0df8a5e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0df40cedf0 0x7f0df415a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.369+0000 7f0df8a5e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0df40cedf0 0x7f0df415a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54892/0 (socket says 172.21.15.189:54892) 2024-04-06T13:47:31.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.369+0000 7f0df8a5e700 1 -- 172.21.15.189:0/4132052107 learned_addr learned my addr 172.21.15.189:0/4132052107 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:31.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.369+0000 7f0df925f700 1 --2- 172.21.15.189:0/4132052107 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0df4146dd0 0x7f0df40b5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.369+0000 7f0df3fff700 1 --2- 172.21.15.189:0/4132052107 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0df40d17a0 0x7f0df40b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.369+0000 7f0df8a5e700 1 -- 172.21.15.189:0/4132052107 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0df4146dd0 msgr2=0x7f0df40b5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.369+0000 7f0df8a5e700 1 --2- 172.21.15.189:0/4132052107 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0df4146dd0 0x7f0df40b5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.369+0000 7f0df8a5e700 1 -- 172.21.15.189:0/4132052107 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0df40d17a0 msgr2=0x7f0df40b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.369+0000 7f0df8a5e700 1 --2- 172.21.15.189:0/4132052107 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0df40d17a0 0x7f0df40b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.369+0000 7f0df8a5e700 1 -- 172.21.15.189:0/4132052107 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de8009550 con 0x7f0df40cedf0 2024-04-06T13:47:31.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.369+0000 7f0df3fff700 1 --2- 172.21.15.189:0/4132052107 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0df40d17a0 0x7f0df40b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:31.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.370+0000 7f0df8a5e700 1 --2- 172.21.15.189:0/4132052107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0df40cedf0 0x7f0df415a060 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f0de8000c00 tx=0x7f0de8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:31.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.370+0000 7f0df17fa700 1 -- 172.21.15.189:0/4132052107 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de8004570 con 0x7f0df40cedf0 2024-04-06T13:47:31.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.370+0000 7f0df17fa700 1 -- 172.21.15.189:0/4132052107 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de801f070 con 0x7f0df40cedf0 2024-04-06T13:47:31.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.370+0000 7f0dfacc2700 1 -- 172.21.15.189:0/4132052107 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df40c6320 con 0x7f0df40cedf0 2024-04-06T13:47:31.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.370+0000 7f0df17fa700 1 -- 172.21.15.189:0/4132052107 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de80038c0 con 0x7f0df40cedf0 2024-04-06T13:47:31.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.370+0000 7f0dfacc2700 1 -- 172.21.15.189:0/4132052107 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df40c6b20 con 0x7f0df40cedf0 2024-04-06T13:47:31.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.371+0000 7f0dfacc2700 1 -- 172.21.15.189:0/4132052107 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df4074990 con 0x7f0df40cedf0 2024-04-06T13:47:31.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.371+0000 7f0df17fa700 1 -- 172.21.15.189:0/4132052107 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0de8021030 con 0x7f0df40cedf0 2024-04-06T13:47:31.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.371+0000 7f0df17fa700 1 --2- 172.21.15.189:0/4132052107 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0dd806adb0 0x7f0dd806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.372+0000 7f0df17fa700 1 -- 172.21.15.189:0/4132052107 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0de808ab80 con 0x7f0df40cedf0 2024-04-06T13:47:31.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.372+0000 7f0df3fff700 1 --2- 172.21.15.189:0/4132052107 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0dd806adb0 0x7f0dd806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.372+0000 7f0df3fff700 1 --2- 172.21.15.189:0/4132052107 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0dd806adb0 0x7f0dd806d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0df40d13d0 tx=0x7f0de0008040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:31.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.376+0000 7f0df17fa700 1 -- 172.21.15.189:0/4132052107 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0de8055c70 con 0x7f0df40cedf0 2024-04-06T13:47:31.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.570+0000 7f0dfacc2700 1 -- 172.21.15.189:0/4132052107 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0df40b7160 con 0x7f0df40cedf0 2024-04-06T13:47:31.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.571+0000 7f0df17fa700 1 -- 172.21.15.189:0/4132052107 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v227) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0de8015020 con 0x7f0df40cedf0 2024-04-06T13:47:31.572 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:47:31.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.573+0000 7f0dfacc2700 1 -- 172.21.15.189:0/4132052107 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0dd806adb0 msgr2=0x7f0dd806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.573+0000 7f0dfacc2700 1 --2- 172.21.15.189:0/4132052107 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0dd806adb0 0x7f0dd806d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0df40d13d0 tx=0x7f0de0008040 comp rx=0 tx=0).stop 2024-04-06T13:47:31.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.573+0000 7f0dfacc2700 1 -- 172.21.15.189:0/4132052107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0df40cedf0 msgr2=0x7f0df415a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.573+0000 7f0dfacc2700 1 --2- 172.21.15.189:0/4132052107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0df40cedf0 0x7f0df415a060 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f0de8000c00 tx=0x7f0de8004750 comp rx=0 tx=0).stop 2024-04-06T13:47:31.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.573+0000 7f0dfacc2700 1 -- 172.21.15.189:0/4132052107 shutdown_connections 2024-04-06T13:47:31.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.573+0000 7f0dfacc2700 1 --2- 172.21.15.189:0/4132052107 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0dd806adb0 0x7f0dd806d270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.573+0000 7f0dfacc2700 1 --2- 172.21.15.189:0/4132052107 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0df4146dd0 0x7f0df40b5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.573+0000 7f0dfacc2700 1 --2- 172.21.15.189:0/4132052107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0df40cedf0 0x7f0df415a060 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.573+0000 7f0dfacc2700 1 --2- 172.21.15.189:0/4132052107 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0df40d17a0 0x7f0df40b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.573+0000 7f0dfacc2700 1 -- 172.21.15.189:0/4132052107 >> 172.21.15.189:0/4132052107 conn(0x7f0df40b2db0 msgr2=0x7f0df415dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:31.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.573+0000 7f0dfacc2700 1 -- 172.21.15.189:0/4132052107 shutdown_connections 2024-04-06T13:47:31.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.573+0000 7f0dfacc2700 1 -- 172.21.15.189:0/4132052107 wait complete. 2024-04-06T13:47:31.587 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-06T13:47:31.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.709+0000 7f702b44d700 1 Processor -- start 2024-04-06T13:47:31.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.709+0000 7f702b44d700 1 -- start start 2024-04-06T13:47:31.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.710+0000 7f702b44d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f702406ae40 0x7f70240b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.710+0000 7f702b44d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70240b3990 con 0x7f702415ddf0 2024-04-06T13:47:31.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.710+0000 7f702b44d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70240b3060 con 0x7f702415d7b0 2024-04-06T13:47:31.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.710+0000 7f702b44d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f702415e950 con 0x7f702406ae40 2024-04-06T13:47:31.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.710+0000 7f70299ea700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f702415ddf0 0x7f702415e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55132/0 (socket says 172.21.15.189:55132) 2024-04-06T13:47:31.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.710+0000 7f70299ea700 1 -- 172.21.15.189:0/331774799 learned_addr learned my addr 172.21.15.189:0/331774799 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:31.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.710+0000 7f70289e8700 1 --2- 172.21.15.189:0/331774799 >> v2:172.21.15.192:3300/0 conn(0x7f702406ae40 0x7f70240b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.710+0000 7f701bfff700 1 -- 172.21.15.189:0/331774799 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837460579 0 0) 0x7f70240b3990 con 0x7f702415ddf0 2024-04-06T13:47:31.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.710+0000 7f701bfff700 1 -- 172.21.15.189:0/331774799 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70100035b0 con 0x7f702415ddf0 2024-04-06T13:47:31.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.710+0000 7f701bfff700 1 -- 172.21.15.189:0/331774799 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1157137711 0 0) 0x7f70100035b0 con 0x7f702415ddf0 2024-04-06T13:47:31.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.710+0000 7f701bfff700 1 -- 172.21.15.189:0/331774799 >> v2:172.21.15.192:3300/0 conn(0x7f702406ae40 msgr2=0x7f70240b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:31.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.710+0000 7f701bfff700 1 --2- 172.21.15.189:0/331774799 >> v2:172.21.15.192:3300/0 conn(0x7f702406ae40 0x7f70240b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.710+0000 7f701bfff700 1 -- 172.21.15.189:0/331774799 >> v1:172.21.15.192:6789/0 conn(0x7f702415d7b0 legacy=0x7f702406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.710+0000 7f701bfff700 1 -- 172.21.15.189:0/331774799 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70240b3340 con 0x7f702415ddf0 2024-04-06T13:47:31.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.710+0000 7f701bfff700 1 -- 172.21.15.189:0/331774799 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f701c0019f0 con 0x7f702415ddf0 2024-04-06T13:47:31.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f701bfff700 1 -- 172.21.15.189:0/331774799 >> v1:172.21.15.189:6789/0 conn(0x7f702415ddf0 legacy=0x7f702415e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f701bfff700 1 --2- 172.21.15.189:0/331774799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7010003a00 0x7f7010003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f701bfff700 1 --2- 172.21.15.189:0/331774799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7010004850 0x7f7010004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f701bfff700 1 --2- 172.21.15.189:0/331774799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7010005210 0x7f701000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f701bfff700 1 -- 172.21.15.189:0/331774799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701000c180 con 0x7f7010004850 2024-04-06T13:47:31.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f701bfff700 1 -- 172.21.15.189:0/331774799 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701000c2f0 con 0x7f7010003a00 2024-04-06T13:47:31.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f701bfff700 1 -- 172.21.15.189:0/331774799 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701000c4b0 con 0x7f7010005210 2024-04-06T13:47:31.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f70289e8700 1 --2- 172.21.15.189:0/331774799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7010004850 0x7f7010004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f70291e9700 1 --2- 172.21.15.189:0/331774799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7010003a00 0x7f7010003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f70299ea700 1 --2- 172.21.15.189:0/331774799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7010005210 0x7f701000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f70289e8700 1 -- 172.21.15.189:0/331774799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7010005210 msgr2=0x7f701000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:31.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f70289e8700 1 --2- 172.21.15.189:0/331774799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7010005210 0x7f701000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f70289e8700 1 -- 172.21.15.189:0/331774799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7010003a00 msgr2=0x7f7010003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:31.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f70289e8700 1 --2- 172.21.15.189:0/331774799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7010003a00 0x7f7010003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f70289e8700 1 -- 172.21.15.189:0/331774799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70100035b0 con 0x7f7010004850 2024-04-06T13:47:31.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f70291e9700 1 --2- 172.21.15.189:0/331774799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7010003a00 0x7f7010003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:31.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f70299ea700 1 -- 172.21.15.189:0/331774799 reap_dead start 2024-04-06T13:47:31.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f70289e8700 1 --2- 172.21.15.189:0/331774799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7010004850 0x7f7010004cd0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7f7014009b30 tx=0x7f7010005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b1128eabb0f074 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:31.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f701bfff700 1 -- 172.21.15.189:0/331774799 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701400e6c0 con 0x7f7010004850 2024-04-06T13:47:31.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f701bfff700 1 -- 172.21.15.189:0/331774799 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f701400e820 con 0x7f7010004850 2024-04-06T13:47:31.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.711+0000 7f701bfff700 1 -- 172.21.15.189:0/331774799 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701400eb90 con 0x7f7010004850 2024-04-06T13:47:31.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.712+0000 7f702b44d700 1 -- 172.21.15.189:0/331774799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7010004850 msgr2=0x7f7010004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.712+0000 7f702b44d700 1 --2- 172.21.15.189:0/331774799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7010004850 0x7f7010004cd0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7f7014009b30 tx=0x7f7010005880 comp rx=0 tx=0).stop 2024-04-06T13:47:31.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.712+0000 7f702b44d700 1 -- 172.21.15.189:0/331774799 shutdown_connections 2024-04-06T13:47:31.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.712+0000 7f702b44d700 1 --2- 172.21.15.189:0/331774799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7010004850 0x7f7010004cd0 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.712+0000 7f702b44d700 1 -- 172.21.15.189:0/331774799 >> 172.21.15.189:0/331774799 conn(0x7f70240d00b0 msgr2=0x7f70240d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:31.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.712+0000 7f702b44d700 1 -- 172.21.15.189:0/331774799 shutdown_connections 2024-04-06T13:47:31.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.712+0000 7f702b44d700 1 -- 172.21.15.189:0/331774799 wait complete. 2024-04-06T13:47:31.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.713+0000 7f702b44d700 1 Processor -- start 2024-04-06T13:47:31.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.713+0000 7f702b44d700 1 -- start start 2024-04-06T13:47:31.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f702b44d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7010004850 0x7f70241537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f702b44d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70240cd370 0x7f70240cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f702b44d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70240ceb20 0x7f70240cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f702b44d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702415eef0 con 0x7f70240cd370 2024-04-06T13:47:31.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f702b44d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70241552a0 con 0x7f7010004850 2024-04-06T13:47:31.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f702b44d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7024153d00 con 0x7f70240ceb20 2024-04-06T13:47:31.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f70289e8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70240cd370 0x7f70240cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f70289e8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70240cd370 0x7f70240cc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54904/0 (socket says 172.21.15.189:54904) 2024-04-06T13:47:31.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f70289e8700 1 -- 172.21.15.189:0/823295293 learned_addr learned my addr 172.21.15.189:0/823295293 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:31.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f70291e9700 1 --2- 172.21.15.189:0/823295293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7010004850 0x7f70241537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f70299ea700 1 --2- 172.21.15.189:0/823295293 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70240ceb20 0x7f70240cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f70289e8700 1 -- 172.21.15.189:0/823295293 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70240ceb20 msgr2=0x7f70240cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f70289e8700 1 --2- 172.21.15.189:0/823295293 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70240ceb20 0x7f70240cefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f70289e8700 1 -- 172.21.15.189:0/823295293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7010004850 msgr2=0x7f70241537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f70289e8700 1 --2- 172.21.15.189:0/823295293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7010004850 0x7f70241537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f70289e8700 1 -- 172.21.15.189:0/823295293 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7024154a00 con 0x7f70240cd370 2024-04-06T13:47:31.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f70289e8700 1 --2- 172.21.15.189:0/823295293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70240cd370 0x7f70240cc580 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f701000c2c0 tx=0x7f70140047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:31.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f7019ffb700 1 -- 172.21.15.189:0/823295293 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701400e6c0 con 0x7f70240cd370 2024-04-06T13:47:31.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f7019ffb700 1 -- 172.21.15.189:0/823295293 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7014013aa0 con 0x7f70240cd370 2024-04-06T13:47:31.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f702b44d700 1 -- 172.21.15.189:0/823295293 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7024154ce0 con 0x7f70240cd370 2024-04-06T13:47:31.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.714+0000 7f7019ffb700 1 -- 172.21.15.189:0/823295293 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7014013e50 con 0x7f70240cd370 2024-04-06T13:47:31.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.715+0000 7f702b44d700 1 -- 172.21.15.189:0/823295293 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70240cf4e0 con 0x7f70240cd370 2024-04-06T13:47:31.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.715+0000 7f702b44d700 1 -- 172.21.15.189:0/823295293 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7024067bb0 con 0x7f70240cd370 2024-04-06T13:47:31.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.717+0000 7f7019ffb700 1 -- 172.21.15.189:0/823295293 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7014024070 con 0x7f70240cd370 2024-04-06T13:47:31.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.717+0000 7f7019ffb700 1 --2- 172.21.15.189:0/823295293 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f700806adb0 0x7f700806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:31.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.717+0000 7f7019ffb700 1 -- 172.21.15.189:0/823295293 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f701408acb0 con 0x7f70240cd370 2024-04-06T13:47:31.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.717+0000 7f70291e9700 1 --2- 172.21.15.189:0/823295293 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f700806adb0 0x7f700806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:31.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.718+0000 7f70291e9700 1 --2- 172.21.15.189:0/823295293 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f700806adb0 0x7f700806d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7020000c00 tx=0x7f7020009040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:31.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.721+0000 7f7019ffb700 1 -- 172.21.15.189:0/823295293 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7014055e80 con 0x7f70240cd370 2024-04-06T13:47:31.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.917+0000 7f702b44d700 1 -- 172.21.15.189:0/823295293 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7024074990 con 0x7f70240cd370 2024-04-06T13:47:31.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.917+0000 7f7019ffb700 1 -- 172.21.15.189:0/823295293 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v227) v1 ==== 84+0+209 (secure 0 0 0) 0x7f7014017090 con 0x7f70240cd370 2024-04-06T13:47:31.919 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:31.919 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T13:47:31.919 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:31.919 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 1, 2024-04-06T13:47:31.920 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": ".mgr" 2024-04-06T13:47:31.920 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:31.920 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:31.920 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 27, 2024-04-06T13:47:31.920 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": "cephfs_metadata" 2024-04-06T13:47:31.920 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:31.920 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:31.920 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 28, 2024-04-06T13:47:31.920 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": "cephfs_data" 2024-04-06T13:47:31.920 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:31.920 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T13:47:31.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.919+0000 7f702b44d700 1 -- 172.21.15.189:0/823295293 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f700806adb0 msgr2=0x7f700806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.919+0000 7f702b44d700 1 --2- 172.21.15.189:0/823295293 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f700806adb0 0x7f700806d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7020000c00 tx=0x7f7020009040 comp rx=0 tx=0).stop 2024-04-06T13:47:31.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.919+0000 7f702b44d700 1 -- 172.21.15.189:0/823295293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70240cd370 msgr2=0x7f70240cc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:31.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.919+0000 7f702b44d700 1 --2- 172.21.15.189:0/823295293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70240cd370 0x7f70240cc580 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f701000c2c0 tx=0x7f70140047c0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.919+0000 7f702b44d700 1 -- 172.21.15.189:0/823295293 shutdown_connections 2024-04-06T13:47:31.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.919+0000 7f702b44d700 1 --2- 172.21.15.189:0/823295293 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f700806adb0 0x7f700806d270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.919+0000 7f702b44d700 1 --2- 172.21.15.189:0/823295293 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70240ceb20 0x7f70240cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.919+0000 7f702b44d700 1 --2- 172.21.15.189:0/823295293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7010004850 0x7f70241537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.919+0000 7f702b44d700 1 --2- 172.21.15.189:0/823295293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70240cd370 0x7f70240cc580 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:31.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.920+0000 7f702b44d700 1 -- 172.21.15.189:0/823295293 >> 172.21.15.189:0/823295293 conn(0x7f70240d00b0 msgr2=0x7f70240717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:31.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.920+0000 7f702b44d700 1 -- 172.21.15.189:0/823295293 shutdown_connections 2024-04-06T13:47:31.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:31.920+0000 7f702b44d700 1 -- 172.21.15.189:0/823295293 wait complete. 2024-04-06T13:47:31.933 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2024-04-06T13:47:31.934 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2024-04-06T13:47:32.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.061+0000 7fe09ade0700 1 Processor -- start 2024-04-06T13:47:32.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.061+0000 7fe09ade0700 1 -- start start 2024-04-06T13:47:32.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.062+0000 7fe09ade0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe0940d11b0 0x7fe0940d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:32.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.062+0000 7fe09ade0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0940d1ad0 con 0x7fe0940cfa00 2024-04-06T13:47:32.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.062+0000 7fe09ade0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0940cbd80 con 0x7fe0940cb870 2024-04-06T13:47:32.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.062+0000 7fe09ade0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940ccdc0 con 0x7fe0940d11b0 2024-04-06T13:47:32.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.062+0000 7fe09937d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe0940cfa00 0x7fe0940cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55140/0 (socket says 172.21.15.189:55140) 2024-04-06T13:47:32.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.062+0000 7fe09937d700 1 -- 172.21.15.189:0/3632265321 learned_addr learned my addr 172.21.15.189:0/3632265321 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:32.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.062+0000 7fe098b7c700 1 --2- 172.21.15.189:0/3632265321 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe0940d11b0 0x7fe0940d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:32.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.062+0000 7fe0937fe700 1 -- 172.21.15.189:0/3632265321 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087608911 0 0) 0x7fe0940d1ad0 con 0x7fe0940cfa00 2024-04-06T13:47:32.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.062+0000 7fe0937fe700 1 -- 172.21.15.189:0/3632265321 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0840035b0 con 0x7fe0940cfa00 2024-04-06T13:47:32.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.062+0000 7fe0937fe700 1 -- 172.21.15.189:0/3632265321 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996094776 0 0) 0x7fe0940cbd80 con 0x7fe0940cb870 2024-04-06T13:47:32.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.062+0000 7fe098b7c700 1 -- 172.21.15.189:0/3632265321 >> v1:172.21.15.192:6789/0 conn(0x7fe0940cb870 legacy=0x7fe09415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:32.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.062+0000 7fe098b7c700 1 -- 172.21.15.189:0/3632265321 >> v1:172.21.15.189:6789/0 conn(0x7fe0940cfa00 legacy=0x7fe0940cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:32.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.062+0000 7fe098b7c700 1 -- 172.21.15.189:0/3632265321 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0940cef20 con 0x7fe0940d11b0 2024-04-06T13:47:32.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.063+0000 7fe098b7c700 1 --2- 172.21.15.189:0/3632265321 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe0940d11b0 0x7fe0940d1590 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe088009870 tx=0x7fe088009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77a253a8a5bd6cd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:32.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.063+0000 7fe0937fe700 1 -- 172.21.15.189:0/3632265321 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe088011830 con 0x7fe0940d11b0 2024-04-06T13:47:32.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.063+0000 7fe0937fe700 1 -- 172.21.15.189:0/3632265321 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe088011990 con 0x7fe0940d11b0 2024-04-06T13:47:32.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.063+0000 7fe0937fe700 1 -- 172.21.15.189:0/3632265321 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe088011d00 con 0x7fe0940d11b0 2024-04-06T13:47:32.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.063+0000 7fe09ade0700 1 -- 172.21.15.189:0/3632265321 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe0940d11b0 msgr2=0x7fe0940d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:32.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.063+0000 7fe09ade0700 1 --2- 172.21.15.189:0/3632265321 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe0940d11b0 0x7fe0940d1590 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe088009870 tx=0x7fe088009b80 comp rx=0 tx=0).stop 2024-04-06T13:47:32.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.063+0000 7fe09ade0700 1 -- 172.21.15.189:0/3632265321 shutdown_connections 2024-04-06T13:47:32.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.063+0000 7fe09ade0700 1 --2- 172.21.15.189:0/3632265321 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe0940d11b0 0x7fe0940d1590 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:32.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.063+0000 7fe09ade0700 1 -- 172.21.15.189:0/3632265321 >> 172.21.15.189:0/3632265321 conn(0x7fe0940b2db0 msgr2=0x7fe0940b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:32.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.063+0000 7fe09ade0700 1 -- 172.21.15.189:0/3632265321 shutdown_connections 2024-04-06T13:47:32.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.063+0000 7fe09ade0700 1 -- 172.21.15.189:0/3632265321 wait complete. 2024-04-06T13:47:32.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.064+0000 7fe09ade0700 1 Processor -- start 2024-04-06T13:47:32.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.065+0000 7fe09ade0700 1 -- start start 2024-04-06T13:47:32.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.065+0000 7fe09ade0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0940cb870 0x7fe0941f3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:32.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.065+0000 7fe09ade0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0940cfa00 0x7fe094155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:32.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.065+0000 7fe09ade0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe0940d11b0 0x7fe0940b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:32.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.065+0000 7fe09ade0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0941f37e0 con 0x7fe0940cb870 2024-04-06T13:47:32.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.065+0000 7fe09ade0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0941f3ac0 con 0x7fe0940cfa00 2024-04-06T13:47:32.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.065+0000 7fe09ade0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0941f3da0 con 0x7fe0940d11b0 2024-04-06T13:47:32.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.065+0000 7fe098b7c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0940cb870 0x7fe0941f3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.066+0000 7fe098b7c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0940cb870 0x7fe0941f3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54906/0 (socket says 172.21.15.189:54906) 2024-04-06T13:47:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.066+0000 7fe098b7c700 1 -- 172.21.15.189:0/3566627055 learned_addr learned my addr 172.21.15.189:0/3566627055 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.066+0000 7fe09937d700 1 --2- 172.21.15.189:0/3566627055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe0940d11b0 0x7fe0940b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.066+0000 7fe098b7c700 1 -- 172.21.15.189:0/3566627055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe0940d11b0 msgr2=0x7fe0940b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.066+0000 7fe093fff700 1 --2- 172.21.15.189:0/3566627055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0940cfa00 0x7fe094155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.066+0000 7fe098b7c700 1 --2- 172.21.15.189:0/3566627055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe0940d11b0 0x7fe0940b3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.066+0000 7fe098b7c700 1 -- 172.21.15.189:0/3566627055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0940cfa00 msgr2=0x7fe094155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.066+0000 7fe098b7c700 1 --2- 172.21.15.189:0/3566627055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0940cfa00 0x7fe094155700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.066+0000 7fe098b7c700 1 -- 172.21.15.189:0/3566627055 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0880094a0 con 0x7fe0940cb870 2024-04-06T13:47:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.066+0000 7fe09937d700 1 --2- 172.21.15.189:0/3566627055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe0940d11b0 0x7fe0940b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.066+0000 7fe093fff700 1 --2- 172.21.15.189:0/3566627055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0940cfa00 0x7fe094155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.066+0000 7fe098b7c700 1 --2- 172.21.15.189:0/3566627055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0940cb870 0x7fe0941f3190 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7fe0880090b0 tx=0x7fe088004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:32.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.066+0000 7fe0917fa700 1 -- 172.21.15.189:0/3566627055 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0880036a0 con 0x7fe0940cb870 2024-04-06T13:47:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.066+0000 7fe0917fa700 1 -- 172.21.15.189:0/3566627055 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe088003800 con 0x7fe0940cb870 2024-04-06T13:47:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.066+0000 7fe09ade0700 1 -- 172.21.15.189:0/3566627055 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0941f4080 con 0x7fe0940cb870 2024-04-06T13:47:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.066+0000 7fe0917fa700 1 -- 172.21.15.189:0/3566627055 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe088003b70 con 0x7fe0940cb870 2024-04-06T13:47:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.066+0000 7fe09ade0700 1 -- 172.21.15.189:0/3566627055 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe094155c40 con 0x7fe0940cb870 2024-04-06T13:47:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.067+0000 7fe09ade0700 1 -- 172.21.15.189:0/3566627055 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe094074990 con 0x7fe0940cb870 2024-04-06T13:47:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.068+0000 7fe0917fa700 1 -- 172.21.15.189:0/3566627055 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fe088003cd0 con 0x7fe0940cb870 2024-04-06T13:47:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.068+0000 7fe0917fa700 1 --2- 172.21.15.189:0/3566627055 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe07806adb0 0x7fe07806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.068+0000 7fe093fff700 1 --2- 172.21.15.189:0/3566627055 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe07806adb0 0x7fe07806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.068+0000 7fe0917fa700 1 -- 172.21.15.189:0/3566627055 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe08808aae0 con 0x7fe0940cb870 2024-04-06T13:47:32.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.068+0000 7fe093fff700 1 --2- 172.21.15.189:0/3566627055 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe07806adb0 0x7fe07806d270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe0940c7c50 tx=0x7fe080009040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:32.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.073+0000 7fe0917fa700 1 -- 172.21.15.189:0/3566627055 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe088055a30 con 0x7fe0940cb870 2024-04-06T13:47:32.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:32.271+0000 7fe09ade0700 1 -- 172.21.15.189:0/3566627055 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe0940b0a00 con 0x7fe0940cb870 2024-04-06T13:47:32.944 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:47:32.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:47:33.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.180+0000 7fe0917fa700 1 -- 172.21.15.189:0/3566627055 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v228) v1 ==== 144+0+0 (secure 0 0 0) 0x7fe088059050 con 0x7fe0940cb870 2024-04-06T13:47:33.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.184+0000 7fe09ade0700 1 -- 172.21.15.189:0/3566627055 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe07806adb0 msgr2=0x7fe07806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:33.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.184+0000 7fe09ade0700 1 --2- 172.21.15.189:0/3566627055 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe07806adb0 0x7fe07806d270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe0940c7c50 tx=0x7fe080009040 comp rx=0 tx=0).stop 2024-04-06T13:47:33.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.184+0000 7fe09ade0700 1 -- 172.21.15.189:0/3566627055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0940cb870 msgr2=0x7fe0941f3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:33.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.184+0000 7fe09ade0700 1 --2- 172.21.15.189:0/3566627055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0940cb870 0x7fe0941f3190 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7fe0880090b0 tx=0x7fe088004750 comp rx=0 tx=0).stop 2024-04-06T13:47:33.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.184+0000 7fe09ade0700 1 -- 172.21.15.189:0/3566627055 shutdown_connections 2024-04-06T13:47:33.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.184+0000 7fe09ade0700 1 --2- 172.21.15.189:0/3566627055 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe07806adb0 0x7fe07806d270 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:33.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.184+0000 7fe09ade0700 1 --2- 172.21.15.189:0/3566627055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe0940d11b0 0x7fe0940b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:33.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.184+0000 7fe09ade0700 1 --2- 172.21.15.189:0/3566627055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0940cb870 0x7fe0941f3190 unknown :-1 s=CLOSED pgs=2444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:33.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.184+0000 7fe09ade0700 1 --2- 172.21.15.189:0/3566627055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0940cfa00 0x7fe094155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:33.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.184+0000 7fe09ade0700 1 -- 172.21.15.189:0/3566627055 >> 172.21.15.189:0/3566627055 conn(0x7fe0940b2db0 msgr2=0x7fe094159d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:33.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.185+0000 7fe09ade0700 1 -- 172.21.15.189:0/3566627055 shutdown_connections 2024-04-06T13:47:33.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.185+0000 7fe09ade0700 1 -- 172.21.15.189:0/3566627055 wait complete. 2024-04-06T13:47:33.190 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'backup_fs_metadata' created 2024-04-06T13:47:33.200 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-06T13:47:33.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.337+0000 7fbe68a44700 1 Processor -- start 2024-04-06T13:47:33.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.337+0000 7fbe68a44700 1 -- start start 2024-04-06T13:47:33.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.338+0000 7fbe68a44700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe6415a550 0x7fbe640cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:33.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.338+0000 7fbe68a44700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbe640b4d70 0x7fbe640b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:33.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.338+0000 7fbe68a44700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe640cdbe0 con 0x7fbe640cf900 2024-04-06T13:47:33.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.338+0000 7fbe68a44700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe6415a940 con 0x7fbe640b4d70 2024-04-06T13:47:33.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.338+0000 7fbe68a44700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6415ac00 con 0x7fbe6415a550 2024-04-06T13:47:33.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.338+0000 7fbe62d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbe640b4d70 0x7fbe640b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:33.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.338+0000 7fbe61d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbe640cf900 0x7fbe640b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55154/0 (socket says 172.21.15.189:55154) 2024-04-06T13:47:33.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.338+0000 7fbe61d9b700 1 -- 172.21.15.189:0/3367475038 learned_addr learned my addr 172.21.15.189:0/3367475038 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:33.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.338+0000 7fbe62d9d700 1 -- 172.21.15.189:0/3367475038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe6415a550 msgr2=0x7fbe640cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:33.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.338+0000 7fbe62d9d700 1 --2- 172.21.15.189:0/3367475038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe6415a550 0x7fbe640cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:33.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.338+0000 7fbe62d9d700 1 -- 172.21.15.189:0/3367475038 >> v1:172.21.15.189:6789/0 conn(0x7fbe640cf900 legacy=0x7fbe640b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:33.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.338+0000 7fbe62d9d700 1 -- 172.21.15.189:0/3367475038 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6415aee0 con 0x7fbe640b4d70 2024-04-06T13:47:33.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.338+0000 7fbe6159a700 1 -- 172.21.15.189:0/3367475038 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343030349 0 0) 0x7fbe640cdbe0 con 0x7fbe640cf900 2024-04-06T13:47:33.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.338+0000 7fbe62d9d700 1 --2- 172.21.15.189:0/3367475038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe640b4d70 0x7fbe640b2fc0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c009d90 tx=0x7fbe5c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a663fca3434d5d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:33.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.339+0000 7fbe6159a700 1 -- 172.21.15.189:0/3367475038 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c00e070 con 0x7fbe640b4d70 2024-04-06T13:47:33.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.339+0000 7fbe6159a700 1 -- 172.21.15.189:0/3367475038 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5c00bbd0 con 0x7fbe640b4d70 2024-04-06T13:47:33.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.339+0000 7fbe6159a700 1 -- 172.21.15.189:0/3367475038 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c012430 con 0x7fbe640b4d70 2024-04-06T13:47:33.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.339+0000 7fbe68a44700 1 -- 172.21.15.189:0/3367475038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe640b4d70 msgr2=0x7fbe640b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:33.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.339+0000 7fbe68a44700 1 --2- 172.21.15.189:0/3367475038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe640b4d70 0x7fbe640b2fc0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c009d90 tx=0x7fbe5c00b330 comp rx=0 tx=0).stop 2024-04-06T13:47:33.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.339+0000 7fbe68a44700 1 -- 172.21.15.189:0/3367475038 shutdown_connections 2024-04-06T13:47:33.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.339+0000 7fbe68a44700 1 --2- 172.21.15.189:0/3367475038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe640b4d70 0x7fbe640b2fc0 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:33.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.339+0000 7fbe68a44700 1 --2- 172.21.15.189:0/3367475038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe6415a550 0x7fbe640cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:33.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.339+0000 7fbe68a44700 1 -- 172.21.15.189:0/3367475038 >> 172.21.15.189:0/3367475038 conn(0x7fbe640d00b0 msgr2=0x7fbe640d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:33.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.339+0000 7fbe68a44700 1 -- 172.21.15.189:0/3367475038 shutdown_connections 2024-04-06T13:47:33.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.339+0000 7fbe68a44700 1 -- 172.21.15.189:0/3367475038 wait complete. 2024-04-06T13:47:33.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.341+0000 7fbe68a44700 1 Processor -- start 2024-04-06T13:47:33.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.341+0000 7fbe68a44700 1 -- start start 2024-04-06T13:47:33.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.341+0000 7fbe68a44700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe640b4d70 0x7fbe640aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:33.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.341+0000 7fbe68a44700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe640cf900 0x7fbe6415d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:33.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.341+0000 7fbe68a44700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe6414afb0 0x7fbe640ad490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:33.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.341+0000 7fbe68a44700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6415d840 con 0x7fbe640cf900 2024-04-06T13:47:33.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.341+0000 7fbe68a44700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe640aeb10 con 0x7fbe6414afb0 2024-04-06T13:47:33.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.341+0000 7fbe68a44700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe640adf00 con 0x7fbe640b4d70 2024-04-06T13:47:33.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.341+0000 7fbe61d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe640cf900 0x7fbe6415d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:33.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.341+0000 7fbe61d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe640cf900 0x7fbe6415d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54930/0 (socket says 172.21.15.189:54930) 2024-04-06T13:47:33.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.342+0000 7fbe61d9b700 1 -- 172.21.15.189:0/2253355606 learned_addr learned my addr 172.21.15.189:0/2253355606 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:33.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.342+0000 7fbe6259c700 1 --2- 172.21.15.189:0/2253355606 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe640b4d70 0x7fbe640aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:33.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.342+0000 7fbe62d9d700 1 --2- 172.21.15.189:0/2253355606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe6414afb0 0x7fbe640ad490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:33.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.342+0000 7fbe61d9b700 1 -- 172.21.15.189:0/2253355606 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe640b4d70 msgr2=0x7fbe640aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:33.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.342+0000 7fbe61d9b700 1 --2- 172.21.15.189:0/2253355606 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe640b4d70 0x7fbe640aefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:33.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.342+0000 7fbe61d9b700 1 -- 172.21.15.189:0/2253355606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe6414afb0 msgr2=0x7fbe640ad490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:33.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.342+0000 7fbe61d9b700 1 --2- 172.21.15.189:0/2253355606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe6414afb0 0x7fbe640ad490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:33.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.342+0000 7fbe61d9b700 1 -- 172.21.15.189:0/2253355606 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5c009a70 con 0x7fbe640cf900 2024-04-06T13:47:33.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.342+0000 7fbe62d9d700 1 --2- 172.21.15.189:0/2253355606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe6414afb0 0x7fbe640ad490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:33.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.342+0000 7fbe61d9b700 1 --2- 172.21.15.189:0/2253355606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe640cf900 0x7fbe6415d300 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7fbe580011e0 tx=0x7fbe5800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:33.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.342+0000 7fbe52ffd700 1 -- 172.21.15.189:0/2253355606 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58004c80 con 0x7fbe640cf900 2024-04-06T13:47:33.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.342+0000 7fbe52ffd700 1 -- 172.21.15.189:0/2253355606 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe58004e00 con 0x7fbe640cf900 2024-04-06T13:47:33.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.342+0000 7fbe52ffd700 1 -- 172.21.15.189:0/2253355606 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58007310 con 0x7fbe640cf900 2024-04-06T13:47:33.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.342+0000 7fbe68a44700 1 -- 172.21.15.189:0/2253355606 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe640ae1e0 con 0x7fbe640cf900 2024-04-06T13:47:33.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.342+0000 7fbe68a44700 1 -- 172.21.15.189:0/2253355606 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe640ae7a0 con 0x7fbe640cf900 2024-04-06T13:47:33.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.343+0000 7fbe68a44700 1 -- 172.21.15.189:0/2253355606 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe64067bb0 con 0x7fbe640cf900 2024-04-06T13:47:33.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.344+0000 7fbe52ffd700 1 -- 172.21.15.189:0/2253355606 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fbe58007470 con 0x7fbe640cf900 2024-04-06T13:47:33.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.344+0000 7fbe52ffd700 1 --2- 172.21.15.189:0/2253355606 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbe4406adb0 0x7fbe4406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:33.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.344+0000 7fbe52ffd700 1 -- 172.21.15.189:0/2253355606 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 6474+0+0 (secure 0 0 0) 0x7fbe58089590 con 0x7fbe640cf900 2024-04-06T13:47:33.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.344+0000 7fbe6259c700 1 --2- 172.21.15.189:0/2253355606 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbe4406adb0 0x7fbe4406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:33.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.344+0000 7fbe6259c700 1 --2- 172.21.15.189:0/2253355606 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbe4406adb0 0x7fbe4406d270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbe54006fd0 tx=0x7fbe54008040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:33.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.349+0000 7fbe52ffd700 1 -- 172.21.15.189:0/2253355606 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbe58054360 con 0x7fbe640cf900 2024-04-06T13:47:33.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:33.545+0000 7fbe68a44700 1 -- 172.21.15.189:0/2253355606 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fbe64074080 con 0x7fbe640cf900 2024-04-06T13:47:34.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.181+0000 7fbe52ffd700 1 -- 172.21.15.189:0/2253355606 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v229) v1 ==== 176+0+0 (secure 0 0 0) 0x7fbe58057980 con 0x7fbe640cf900 2024-04-06T13:47:34.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.191+0000 7fbe68a44700 1 -- 172.21.15.189:0/2253355606 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbe4406adb0 msgr2=0x7fbe4406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:34.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.191+0000 7fbe68a44700 1 --2- 172.21.15.189:0/2253355606 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbe4406adb0 0x7fbe4406d270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbe54006fd0 tx=0x7fbe54008040 comp rx=0 tx=0).stop 2024-04-06T13:47:34.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.191+0000 7fbe68a44700 1 -- 172.21.15.189:0/2253355606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe640cf900 msgr2=0x7fbe6415d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:34.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.191+0000 7fbe68a44700 1 --2- 172.21.15.189:0/2253355606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe640cf900 0x7fbe6415d300 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7fbe580011e0 tx=0x7fbe5800cae0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.191+0000 7fbe68a44700 1 -- 172.21.15.189:0/2253355606 shutdown_connections 2024-04-06T13:47:34.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.191+0000 7fbe68a44700 1 --2- 172.21.15.189:0/2253355606 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbe4406adb0 0x7fbe4406d270 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbe54006fd0 tx=0x7fbe54008040 comp rx=0 tx=0).stop 2024-04-06T13:47:34.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.191+0000 7fbe68a44700 1 --2- 172.21.15.189:0/2253355606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe6414afb0 0x7fbe640ad490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.192+0000 7fbe68a44700 1 --2- 172.21.15.189:0/2253355606 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe640b4d70 0x7fbe640aefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.192+0000 7fbe68a44700 1 --2- 172.21.15.189:0/2253355606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe640cf900 0x7fbe6415d300 unknown :-1 s=CLOSED pgs=2447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.192+0000 7fbe68a44700 1 -- 172.21.15.189:0/2253355606 >> 172.21.15.189:0/2253355606 conn(0x7fbe640d00b0 msgr2=0x7fbe640b3900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:34.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.192+0000 7fbe68a44700 1 -- 172.21.15.189:0/2253355606 shutdown_connections 2024-04-06T13:47:34.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.193+0000 7fbe68a44700 1 -- 172.21.15.189:0/2253355606 wait complete. 2024-04-06T13:47:34.195 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'backup_fs_data' created 2024-04-06T13:47:34.208 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2024-04-06T13:47:34.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.334+0000 7fdf5c261700 1 Processor -- start 2024-04-06T13:47:34.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.334+0000 7fdf5c261700 1 -- start start 2024-04-06T13:47:34.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.335+0000 7fdf5c261700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdf5415d7b0 0x7fdf540b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:34.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.335+0000 7fdf5c261700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdf5406a730 0x7fdf5406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:34.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.335+0000 7fdf5c261700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf5415ddf0 0x7fdf5415e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:34.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.335+0000 7fdf5c261700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf5406b0f0 con 0x7fdf5415d7b0 2024-04-06T13:47:34.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.335+0000 7fdf5c261700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf540b3990 con 0x7fdf5406a730 2024-04-06T13:47:34.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.335+0000 7fdf5c261700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf540b3060 con 0x7fdf5415ddf0 2024-04-06T13:47:34.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.335+0000 7fdf59ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdf5415d7b0 0x7fdf540b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:34.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.335+0000 7fdf59ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdf5415d7b0 0x7fdf540b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54944/0 (socket says 172.21.15.189:54944) 2024-04-06T13:47:34.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.335+0000 7fdf59ffd700 1 -- 172.21.15.189:0/2507635472 learned_addr learned my addr 172.21.15.189:0/2507635472 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:34.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.336+0000 7fdf5a7fe700 1 --2- 172.21.15.189:0/2507635472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf5415ddf0 0x7fdf5415e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:34.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.336+0000 7fdf597fc700 1 --2- 172.21.15.189:0/2507635472 >> v2:172.21.15.192:3300/0 conn(0x7fdf5406a730 0x7fdf5406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:34.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.336+0000 7fdf59ffd700 1 -- 172.21.15.189:0/2507635472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf5415ddf0 msgr2=0x7fdf5415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:34.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.336+0000 7fdf59ffd700 1 --2- 172.21.15.189:0/2507635472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf5415ddf0 0x7fdf5415e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.336+0000 7fdf59ffd700 1 -- 172.21.15.189:0/2507635472 >> v2:172.21.15.192:3300/0 conn(0x7fdf5406a730 msgr2=0x7fdf5406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:34.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.336+0000 7fdf59ffd700 1 --2- 172.21.15.189:0/2507635472 >> v2:172.21.15.192:3300/0 conn(0x7fdf5406a730 0x7fdf5406abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.336+0000 7fdf59ffd700 1 -- 172.21.15.189:0/2507635472 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf540b3340 con 0x7fdf5415d7b0 2024-04-06T13:47:34.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.336+0000 7fdf59ffd700 1 --2- 172.21.15.189:0/2507635472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf5415d7b0 0x7fdf540b3e20 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7fdf540afc00 tx=0x7fdf44009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=736595b356f9df00 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:34.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.336+0000 7fdf58ffb700 1 -- 172.21.15.189:0/2507635472 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4400b870 con 0x7fdf5415d7b0 2024-04-06T13:47:34.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.336+0000 7fdf58ffb700 1 -- 172.21.15.189:0/2507635472 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4400b9d0 con 0x7fdf5415d7b0 2024-04-06T13:47:34.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.336+0000 7fdf58ffb700 1 -- 172.21.15.189:0/2507635472 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4400bd60 con 0x7fdf5415d7b0 2024-04-06T13:47:34.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.336+0000 7fdf5c261700 1 -- 172.21.15.189:0/2507635472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf5415d7b0 msgr2=0x7fdf540b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:34.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.336+0000 7fdf5c261700 1 --2- 172.21.15.189:0/2507635472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf5415d7b0 0x7fdf540b3e20 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7fdf540afc00 tx=0x7fdf44009bd0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.336+0000 7fdf5c261700 1 -- 172.21.15.189:0/2507635472 shutdown_connections 2024-04-06T13:47:34.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.336+0000 7fdf5c261700 1 --2- 172.21.15.189:0/2507635472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf5415d7b0 0x7fdf540b3e20 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.336+0000 7fdf5c261700 1 --2- 172.21.15.189:0/2507635472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf5415ddf0 0x7fdf5415e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.336+0000 7fdf5c261700 1 --2- 172.21.15.189:0/2507635472 >> v2:172.21.15.192:3300/0 conn(0x7fdf5406a730 0x7fdf5406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.337+0000 7fdf5c261700 1 -- 172.21.15.189:0/2507635472 >> 172.21.15.189:0/2507635472 conn(0x7fdf540d00b0 msgr2=0x7fdf540d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:34.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.337+0000 7fdf5c261700 1 -- 172.21.15.189:0/2507635472 shutdown_connections 2024-04-06T13:47:34.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.337+0000 7fdf5c261700 1 -- 172.21.15.189:0/2507635472 wait complete. 2024-04-06T13:47:34.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.338+0000 7fdf5c261700 1 Processor -- start 2024-04-06T13:47:34.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.338+0000 7fdf5c261700 1 -- start start 2024-04-06T13:47:34.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.338+0000 7fdf5c261700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf5406a730 0x7fdf5414fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:34.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.338+0000 7fdf5c261700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf5415d7b0 0x7fdf540ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:34.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.338+0000 7fdf5c261700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf5415ddf0 0x7fdf540acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:34.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.339+0000 7fdf5c261700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5415a3e0 con 0x7fdf5415d7b0 2024-04-06T13:47:34.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.339+0000 7fdf5c261700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf54150e20 con 0x7fdf5415ddf0 2024-04-06T13:47:34.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.339+0000 7fdf5c261700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5415a630 con 0x7fdf5406a730 2024-04-06T13:47:34.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.339+0000 7fdf597fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf5415d7b0 0x7fdf540ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:34.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.339+0000 7fdf597fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf5415d7b0 0x7fdf540ccea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54958/0 (socket says 172.21.15.189:54958) 2024-04-06T13:47:34.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.339+0000 7fdf597fc700 1 -- 172.21.15.189:0/455698185 learned_addr learned my addr 172.21.15.189:0/455698185 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:34.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.339+0000 7fdf5a7fe700 1 --2- 172.21.15.189:0/455698185 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf5415ddf0 0x7fdf540acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:34.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.339+0000 7fdf597fc700 1 -- 172.21.15.189:0/455698185 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf5406a730 msgr2=0x7fdf5414fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:34.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.339+0000 7fdf59ffd700 1 --2- 172.21.15.189:0/455698185 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf5406a730 0x7fdf5414fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:34.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.339+0000 7fdf597fc700 1 --2- 172.21.15.189:0/455698185 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf5406a730 0x7fdf5414fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.339+0000 7fdf597fc700 1 -- 172.21.15.189:0/455698185 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf5415ddf0 msgr2=0x7fdf540acb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:34.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.339+0000 7fdf597fc700 1 --2- 172.21.15.189:0/455698185 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf5415ddf0 0x7fdf540acb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.339+0000 7fdf597fc700 1 -- 172.21.15.189:0/455698185 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf44009550 con 0x7fdf5415d7b0 2024-04-06T13:47:34.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.339+0000 7fdf597fc700 1 --2- 172.21.15.189:0/455698185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf5415d7b0 0x7fdf540ccea0 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7fdf5000c7f0 tx=0x7fdf5000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:34.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.339+0000 7fdf4a7fc700 1 -- 172.21.15.189:0/455698185 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf50011800 con 0x7fdf5415d7b0 2024-04-06T13:47:34.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.339+0000 7fdf4a7fc700 1 -- 172.21.15.189:0/455698185 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf50011960 con 0x7fdf5415d7b0 2024-04-06T13:47:34.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.339+0000 7fdf5c261700 1 -- 172.21.15.189:0/455698185 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf5415a910 con 0x7fdf5415d7b0 2024-04-06T13:47:34.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.339+0000 7fdf4a7fc700 1 -- 172.21.15.189:0/455698185 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf50011cf0 con 0x7fdf5415d7b0 2024-04-06T13:47:34.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.339+0000 7fdf5c261700 1 -- 172.21.15.189:0/455698185 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf5415aed0 con 0x7fdf5415d7b0 2024-04-06T13:47:34.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.340+0000 7fdf5c261700 1 -- 172.21.15.189:0/455698185 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf54067bb0 con 0x7fdf5415d7b0 2024-04-06T13:47:34.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.341+0000 7fdf4a7fc700 1 -- 172.21.15.189:0/455698185 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fdf50018360 con 0x7fdf5415d7b0 2024-04-06T13:47:34.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.341+0000 7fdf4a7fc700 1 --2- 172.21.15.189:0/455698185 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdf3806adb0 0x7fdf3806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:34.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.341+0000 7fdf4a7fc700 1 -- 172.21.15.189:0/455698185 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6859+0+0 (secure 0 0 0) 0x7fdf50089a40 con 0x7fdf5415d7b0 2024-04-06T13:47:34.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.341+0000 7fdf59ffd700 1 --2- 172.21.15.189:0/455698185 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdf3806adb0 0x7fdf3806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:34.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.342+0000 7fdf59ffd700 1 --2- 172.21.15.189:0/455698185 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdf3806adb0 0x7fdf3806d270 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdf44009130 tx=0x7fdf44008030 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:34.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.346+0000 7fdf4a7fc700 1 -- 172.21.15.189:0/455698185 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdf5008be60 con 0x7fdf5415d7b0 2024-04-06T13:47:34.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.583+0000 7fdf5c261700 1 -- 172.21.15.189:0/455698185 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7fdf54074990 con 0x7fdf5415d7b0 2024-04-06T13:47:34.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.588+0000 7fdf4a7fc700 1 -- 172.21.15.189:0/455698185 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 29 and data pool 30 v102) v1 ==== 183+0+0 (secure 0 0 0) 0x7fdf50058020 con 0x7fdf5415d7b0 2024-04-06T13:47:34.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.596+0000 7fdf5c261700 1 -- 172.21.15.189:0/455698185 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdf3806adb0 msgr2=0x7fdf3806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:34.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.596+0000 7fdf5c261700 1 --2- 172.21.15.189:0/455698185 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdf3806adb0 0x7fdf3806d270 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdf44009130 tx=0x7fdf44008030 comp rx=0 tx=0).stop 2024-04-06T13:47:34.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.596+0000 7fdf5c261700 1 -- 172.21.15.189:0/455698185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf5415d7b0 msgr2=0x7fdf540ccea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:34.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.596+0000 7fdf5c261700 1 --2- 172.21.15.189:0/455698185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf5415d7b0 0x7fdf540ccea0 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7fdf5000c7f0 tx=0x7fdf5000cbb0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.596+0000 7fdf5c261700 1 -- 172.21.15.189:0/455698185 shutdown_connections 2024-04-06T13:47:34.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.596+0000 7fdf5c261700 1 --2- 172.21.15.189:0/455698185 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdf3806adb0 0x7fdf3806d270 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdf44009130 tx=0x7fdf44008030 comp rx=0 tx=0).stop 2024-04-06T13:47:34.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.596+0000 7fdf5c261700 1 --2- 172.21.15.189:0/455698185 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf5415ddf0 0x7fdf540acb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.596+0000 7fdf5c261700 1 --2- 172.21.15.189:0/455698185 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf5406a730 0x7fdf5414fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.596+0000 7fdf5c261700 1 --2- 172.21.15.189:0/455698185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf5415d7b0 0x7fdf540ccea0 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.596+0000 7fdf5c261700 1 -- 172.21.15.189:0/455698185 >> 172.21.15.189:0/455698185 conn(0x7fdf540d00b0 msgr2=0x7fdf540ae0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:34.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.603+0000 7fdf5c261700 1 -- 172.21.15.189:0/455698185 shutdown_connections 2024-04-06T13:47:34.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.604+0000 7fdf5c261700 1 -- 172.21.15.189:0/455698185 wait complete. 2024-04-06T13:47:34.606 INFO:teuthology.orchestra.run.smithi189.stderr:new fs with metadata pool 29 and data pool 30 2024-04-06T13:47:34.621 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:47:34.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.743+0000 7fab71353700 1 Processor -- start 2024-04-06T13:47:34.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.743+0000 7fab71353700 1 -- start start 2024-04-06T13:47:34.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.744+0000 7fab71353700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fab6c15d5e0 0x7fab6c06d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:34.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.744+0000 7fab71353700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab6c0aede0 0x7fab6c0b2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:34.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.744+0000 7fab71353700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab6c06ae20 con 0x7fab6c0ac8c0 2024-04-06T13:47:34.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.744+0000 7fab71353700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab6c15d9c0 con 0x7fab6c15d5e0 2024-04-06T13:47:34.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.744+0000 7fab71353700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6c06e250 con 0x7fab6c0aede0 2024-04-06T13:47:34.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.744+0000 7fab6affd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fab6c15d5e0 0x7fab6c06d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:34.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.744+0000 7fab6b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fab6c0ac8c0 0x7fab6c06b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55160/0 (socket says 172.21.15.189:55160) 2024-04-06T13:47:34.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.744+0000 7fab6b7fe700 1 -- 172.21.15.189:0/2124464497 learned_addr learned my addr 172.21.15.189:0/2124464497 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:34.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.744+0000 7fab6a7fc700 1 --2- 172.21.15.189:0/2124464497 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab6c0aede0 0x7fab6c0b2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:34.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.744+0000 7fab6affd700 1 -- 172.21.15.189:0/2124464497 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab6c0aede0 msgr2=0x7fab6c0b2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:34.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.744+0000 7fab6affd700 1 --2- 172.21.15.189:0/2124464497 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab6c0aede0 0x7fab6c0b2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.744+0000 7fab6affd700 1 -- 172.21.15.189:0/2124464497 >> v1:172.21.15.189:6789/0 conn(0x7fab6c0ac8c0 legacy=0x7fab6c06b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:34.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.744+0000 7fab69ffb700 1 -- 172.21.15.189:0/2124464497 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795009913 0 0) 0x7fab6c06ae20 con 0x7fab6c0ac8c0 2024-04-06T13:47:34.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.744+0000 7fab6affd700 1 -- 172.21.15.189:0/2124464497 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6c06df70 con 0x7fab6c15d5e0 2024-04-06T13:47:34.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.744+0000 7fab6affd700 1 --2- 172.21.15.189:0/2124464497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab6c15d5e0 0x7fab6c06d720 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7fab60009840 tx=0x7fab60009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f857687f03c10295 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:34.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.745+0000 7fab69ffb700 1 -- 172.21.15.189:0/2124464497 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab60011830 con 0x7fab6c15d5e0 2024-04-06T13:47:34.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.745+0000 7fab69ffb700 1 -- 172.21.15.189:0/2124464497 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab60011990 con 0x7fab6c15d5e0 2024-04-06T13:47:34.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.745+0000 7fab69ffb700 1 -- 172.21.15.189:0/2124464497 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab60011d20 con 0x7fab6c15d5e0 2024-04-06T13:47:34.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.745+0000 7fab71353700 1 -- 172.21.15.189:0/2124464497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab6c15d5e0 msgr2=0x7fab6c06d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:34.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.745+0000 7fab71353700 1 --2- 172.21.15.189:0/2124464497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab6c15d5e0 0x7fab6c06d720 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7fab60009840 tx=0x7fab60009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:34.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.745+0000 7fab71353700 1 -- 172.21.15.189:0/2124464497 shutdown_connections 2024-04-06T13:47:34.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.745+0000 7fab71353700 1 --2- 172.21.15.189:0/2124464497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab6c15d5e0 0x7fab6c06d720 unknown :-1 s=CLOSED pgs=2451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.745+0000 7fab71353700 1 --2- 172.21.15.189:0/2124464497 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab6c0aede0 0x7fab6c0b2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.745+0000 7fab71353700 1 -- 172.21.15.189:0/2124464497 >> 172.21.15.189:0/2124464497 conn(0x7fab6c0b1d90 msgr2=0x7fab6c0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:34.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.745+0000 7fab71353700 1 -- 172.21.15.189:0/2124464497 shutdown_connections 2024-04-06T13:47:34.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.745+0000 7fab71353700 1 -- 172.21.15.189:0/2124464497 wait complete. 2024-04-06T13:47:34.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.746+0000 7fab71353700 1 Processor -- start 2024-04-06T13:47:34.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.747+0000 7fab71353700 1 -- start start 2024-04-06T13:47:34.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.747+0000 7fab71353700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab6c0ac8c0 0x7fab6c155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:34.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.747+0000 7fab71353700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab6c0aede0 0x7fab6c1f6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:34.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.747+0000 7fab71353700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab6c15d5e0 0x7fab6c157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:34.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.747+0000 7fab71353700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6c152c10 con 0x7fab6c15d5e0 2024-04-06T13:47:34.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.747+0000 7fab71353700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6c1f7300 con 0x7fab6c0ac8c0 2024-04-06T13:47:34.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.747+0000 7fab71353700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6c151f90 con 0x7fab6c0aede0 2024-04-06T13:47:34.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.747+0000 7fab6b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab6c15d5e0 0x7fab6c157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:34.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.747+0000 7fab6b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab6c15d5e0 0x7fab6c157fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54974/0 (socket says 172.21.15.189:54974) 2024-04-06T13:47:34.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.747+0000 7fab6b7fe700 1 -- 172.21.15.189:0/2606964132 learned_addr learned my addr 172.21.15.189:0/2606964132 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:34.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.747+0000 7fab6a7fc700 1 --2- 172.21.15.189:0/2606964132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab6c0aede0 0x7fab6c1f6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:34.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.748+0000 7fab6affd700 1 --2- 172.21.15.189:0/2606964132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab6c0ac8c0 0x7fab6c155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:34.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.748+0000 7fab6b7fe700 1 -- 172.21.15.189:0/2606964132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab6c0aede0 msgr2=0x7fab6c1f6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:34.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.748+0000 7fab6b7fe700 1 --2- 172.21.15.189:0/2606964132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab6c0aede0 0x7fab6c1f6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.748+0000 7fab6b7fe700 1 -- 172.21.15.189:0/2606964132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab6c0ac8c0 msgr2=0x7fab6c155490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:34.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.748+0000 7fab6b7fe700 1 --2- 172.21.15.189:0/2606964132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab6c0ac8c0 0x7fab6c155490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.748+0000 7fab6b7fe700 1 -- 172.21.15.189:0/2606964132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab600094a0 con 0x7fab6c15d5e0 2024-04-06T13:47:34.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.748+0000 7fab6affd700 1 --2- 172.21.15.189:0/2606964132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab6c0ac8c0 0x7fab6c155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:34.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.748+0000 7fab6b7fe700 1 --2- 172.21.15.189:0/2606964132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab6c15d5e0 0x7fab6c157fe0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7fab6400cca0 tx=0x7fab6400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:34.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.748+0000 7fab537fe700 1 -- 172.21.15.189:0/2606964132 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6400ac20 con 0x7fab6c15d5e0 2024-04-06T13:47:34.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.748+0000 7fab537fe700 1 -- 172.21.15.189:0/2606964132 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6400ad80 con 0x7fab6c15d5e0 2024-04-06T13:47:34.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.748+0000 7fab71353700 1 -- 172.21.15.189:0/2606964132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab6c152270 con 0x7fab6c15d5e0 2024-04-06T13:47:34.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.748+0000 7fab537fe700 1 -- 172.21.15.189:0/2606964132 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab64007310 con 0x7fab6c15d5e0 2024-04-06T13:47:34.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.748+0000 7fab71353700 1 -- 172.21.15.189:0/2606964132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab6c152830 con 0x7fab6c15d5e0 2024-04-06T13:47:34.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.749+0000 7fab71353700 1 -- 172.21.15.189:0/2606964132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab6c152d50 con 0x7fab6c15d5e0 2024-04-06T13:47:34.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.749+0000 7fab537fe700 1 -- 172.21.15.189:0/2606964132 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fab64007470 con 0x7fab6c15d5e0 2024-04-06T13:47:34.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.750+0000 7fab537fe700 1 --2- 172.21.15.189:0/2606964132 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fab5806adb0 0x7fab5806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:34.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.750+0000 7fab537fe700 1 -- 172.21.15.189:0/2606964132 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6965+0+0 (secure 0 0 0) 0x7fab64089840 con 0x7fab6c15d5e0 2024-04-06T13:47:34.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.750+0000 7fab6affd700 1 --2- 172.21.15.189:0/2606964132 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fab5806adb0 0x7fab5806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:34.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.750+0000 7fab6affd700 1 --2- 172.21.15.189:0/2606964132 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fab5806adb0 0x7fab5806d270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fab60000c00 tx=0x7fab6001c040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:34.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.754+0000 7fab537fe700 1 -- 172.21.15.189:0/2606964132 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fab64054420 con 0x7fab6c15d5e0 2024-04-06T13:47:34.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.951+0000 7fab71353700 1 -- 172.21.15.189:0/2606964132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab6c074990 con 0x7fab6c15d5e0 2024-04-06T13:47:34.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.952+0000 7fab537fe700 1 -- 172.21.15.189:0/2606964132 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v230) v1 ==== 74+0+20149 (secure 0 0 0) 0x7fab64057a40 con 0x7fab6c15d5e0 2024-04-06T13:47:34.952 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:34.953 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":230,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:47:34.583870+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:47:20.644275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-04-06T13:47:21.075143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:47:32.292623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:47:33.567052+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:47:34.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.954+0000 7fab71353700 1 -- 172.21.15.189:0/2606964132 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fab5806adb0 msgr2=0x7fab5806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:34.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.954+0000 7fab71353700 1 --2- 172.21.15.189:0/2606964132 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fab5806adb0 0x7fab5806d270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fab60000c00 tx=0x7fab6001c040 comp rx=0 tx=0).stop 2024-04-06T13:47:34.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.954+0000 7fab71353700 1 -- 172.21.15.189:0/2606964132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab6c15d5e0 msgr2=0x7fab6c157fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:34.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.954+0000 7fab71353700 1 --2- 172.21.15.189:0/2606964132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab6c15d5e0 0x7fab6c157fe0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7fab6400cca0 tx=0x7fab6400a060 comp rx=0 tx=0).stop 2024-04-06T13:47:34.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.955+0000 7fab71353700 1 -- 172.21.15.189:0/2606964132 shutdown_connections 2024-04-06T13:47:34.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.955+0000 7fab71353700 1 --2- 172.21.15.189:0/2606964132 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fab5806adb0 0x7fab5806d270 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.955+0000 7fab71353700 1 --2- 172.21.15.189:0/2606964132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab6c15d5e0 0x7fab6c157fe0 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.955+0000 7fab71353700 1 --2- 172.21.15.189:0/2606964132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab6c0ac8c0 0x7fab6c155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.955+0000 7fab71353700 1 --2- 172.21.15.189:0/2606964132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab6c0aede0 0x7fab6c1f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:34.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.955+0000 7fab71353700 1 -- 172.21.15.189:0/2606964132 >> 172.21.15.189:0/2606964132 conn(0x7fab6c0b1d90 msgr2=0x7fab6c0b0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:34.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.955+0000 7fab71353700 1 -- 172.21.15.189:0/2606964132 shutdown_connections 2024-04-06T13:47:34.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:34.955+0000 7fab71353700 1 -- 172.21.15.189:0/2606964132 wait complete. 2024-04-06T13:47:34.968 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:47:35.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.090+0000 7f01dac41700 1 Processor -- start 2024-04-06T13:47:35.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.090+0000 7f01dac41700 1 -- start start 2024-04-06T13:47:35.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.091+0000 7f01dac41700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f01d4146dd0 0x7f01d406d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:35.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.091+0000 7f01dac41700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f01d40cedf0 0x7f01d40cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:35.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.091+0000 7f01dac41700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d40cf730 con 0x7f01d40d17a0 2024-04-06T13:47:35.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.091+0000 7f01dac41700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d406beb0 con 0x7f01d4146dd0 2024-04-06T13:47:35.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.091+0000 7f01dac41700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d406c0a0 con 0x7f01d40cedf0 2024-04-06T13:47:35.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.091+0000 7f01d3fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f01d4146dd0 0x7f01d406d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:35.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.091+0000 7f01d89dd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f01d40d17a0 0x7f01d406b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55170/0 (socket says 172.21.15.189:55170) 2024-04-06T13:47:35.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.091+0000 7f01d89dd700 1 -- 172.21.15.189:0/2948889489 learned_addr learned my addr 172.21.15.189:0/2948889489 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:35.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.092+0000 7f01d91de700 1 --2- 172.21.15.189:0/2948889489 >> v2:172.21.15.192:3300/0 conn(0x7f01d40cedf0 0x7f01d40cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:35.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.092+0000 7f01d3fff700 1 -- 172.21.15.189:0/2948889489 >> v2:172.21.15.192:3300/0 conn(0x7f01d40cedf0 msgr2=0x7f01d40cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.092+0000 7f01d3fff700 1 --2- 172.21.15.189:0/2948889489 >> v2:172.21.15.192:3300/0 conn(0x7f01d40cedf0 0x7f01d40cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.092+0000 7f01d3fff700 1 -- 172.21.15.189:0/2948889489 >> v1:172.21.15.189:6789/0 conn(0x7f01d40d17a0 legacy=0x7f01d406b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.092+0000 7f01d3fff700 1 -- 172.21.15.189:0/2948889489 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d406c380 con 0x7f01d4146dd0 2024-04-06T13:47:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.092+0000 7f01d37fe700 1 -- 172.21.15.189:0/2948889489 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024623741 0 0) 0x7f01d40cf730 con 0x7f01d40d17a0 2024-04-06T13:47:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.092+0000 7f01d3fff700 1 --2- 172.21.15.189:0/2948889489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d4146dd0 0x7f01d406d140 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f01d4160fb0 tx=0x7f01c0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac0dda84d2d5f71f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:35.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.092+0000 7f01d37fe700 1 -- 172.21.15.189:0/2948889489 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c0011830 con 0x7f01d4146dd0 2024-04-06T13:47:35.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.092+0000 7f01d37fe700 1 -- 172.21.15.189:0/2948889489 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c0011990 con 0x7f01d4146dd0 2024-04-06T13:47:35.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.092+0000 7f01d37fe700 1 -- 172.21.15.189:0/2948889489 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c0011d20 con 0x7f01d4146dd0 2024-04-06T13:47:35.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.093+0000 7f01dac41700 1 -- 172.21.15.189:0/2948889489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d4146dd0 msgr2=0x7f01d406d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:35.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.093+0000 7f01dac41700 1 --2- 172.21.15.189:0/2948889489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d4146dd0 0x7f01d406d140 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f01d4160fb0 tx=0x7f01c0009b20 comp rx=0 tx=0).stop 2024-04-06T13:47:35.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.093+0000 7f01dac41700 1 -- 172.21.15.189:0/2948889489 shutdown_connections 2024-04-06T13:47:35.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.093+0000 7f01dac41700 1 --2- 172.21.15.189:0/2948889489 >> v2:172.21.15.192:3300/0 conn(0x7f01d40cedf0 0x7f01d40cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:35.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.093+0000 7f01dac41700 1 --2- 172.21.15.189:0/2948889489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d4146dd0 0x7f01d406d140 unknown :-1 s=CLOSED pgs=2454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:35.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.093+0000 7f01dac41700 1 -- 172.21.15.189:0/2948889489 >> 172.21.15.189:0/2948889489 conn(0x7f01d40b2db0 msgr2=0x7f01d40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:35.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.093+0000 7f01dac41700 1 -- 172.21.15.189:0/2948889489 shutdown_connections 2024-04-06T13:47:35.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.093+0000 7f01dac41700 1 -- 172.21.15.189:0/2948889489 wait complete. 2024-04-06T13:47:35.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.094+0000 7f01dac41700 1 Processor -- start 2024-04-06T13:47:35.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.094+0000 7f01dac41700 1 -- start start 2024-04-06T13:47:35.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01dac41700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01d40cedf0 0x7f01d40c73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:35.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01dac41700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d40d17a0 0x7f01d41f3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:35.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01dac41700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01d4146dd0 0x7f01d40af400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:35.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01dac41700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d40c6f90 con 0x7f01d40d17a0 2024-04-06T13:47:35.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01dac41700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d41f3e90 con 0x7f01d40cedf0 2024-04-06T13:47:35.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01dac41700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d40c6310 con 0x7f01d4146dd0 2024-04-06T13:47:35.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01d3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d40d17a0 0x7f01d41f3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:35.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01d3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d40d17a0 0x7f01d41f3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54988/0 (socket says 172.21.15.189:54988) 2024-04-06T13:47:35.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01d3fff700 1 -- 172.21.15.189:0/2403704349 learned_addr learned my addr 172.21.15.189:0/2403704349 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:35.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01d89dd700 1 --2- 172.21.15.189:0/2403704349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01d40cedf0 0x7f01d40c73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:35.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01d91de700 1 --2- 172.21.15.189:0/2403704349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01d4146dd0 0x7f01d40af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:35.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01d3fff700 1 -- 172.21.15.189:0/2403704349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01d4146dd0 msgr2=0x7f01d40af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:35.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01d3fff700 1 --2- 172.21.15.189:0/2403704349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01d4146dd0 0x7f01d40af400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:35.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01d3fff700 1 -- 172.21.15.189:0/2403704349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01d40cedf0 msgr2=0x7f01d40c73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:35.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01d3fff700 1 --2- 172.21.15.189:0/2403704349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01d40cedf0 0x7f01d40c73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:35.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01d3fff700 1 -- 172.21.15.189:0/2403704349 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c00094a0 con 0x7f01d40d17a0 2024-04-06T13:47:35.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01d89dd700 1 --2- 172.21.15.189:0/2403704349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01d40cedf0 0x7f01d40c73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:35.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01d3fff700 1 --2- 172.21.15.189:0/2403704349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d40d17a0 0x7f01d41f3950 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f01c00090b0 tx=0x7f01c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:35.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01d17fa700 1 -- 172.21.15.189:0/2403704349 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c00038a0 con 0x7f01d40d17a0 2024-04-06T13:47:35.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01d17fa700 1 -- 172.21.15.189:0/2403704349 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c001f070 con 0x7f01d40d17a0 2024-04-06T13:47:35.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01dac41700 1 -- 172.21.15.189:0/2403704349 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d40c65f0 con 0x7f01d40d17a0 2024-04-06T13:47:35.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.095+0000 7f01d17fa700 1 -- 172.21.15.189:0/2403704349 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c0003c20 con 0x7f01d40d17a0 2024-04-06T13:47:35.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.096+0000 7f01dac41700 1 -- 172.21.15.189:0/2403704349 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d40c6bb0 con 0x7f01d40d17a0 2024-04-06T13:47:35.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.096+0000 7f01dac41700 1 -- 172.21.15.189:0/2403704349 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d40c70d0 con 0x7f01d40d17a0 2024-04-06T13:47:35.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.097+0000 7f01d17fa700 1 -- 172.21.15.189:0/2403704349 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f01c001a430 con 0x7f01d40d17a0 2024-04-06T13:47:35.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.097+0000 7f01d17fa700 1 --2- 172.21.15.189:0/2403704349 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f01b806ade0 0x7f01b806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:35.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.097+0000 7f01d17fa700 1 -- 172.21.15.189:0/2403704349 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f01c0021070 con 0x7f01d40d17a0 2024-04-06T13:47:35.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.097+0000 7f01d89dd700 1 --2- 172.21.15.189:0/2403704349 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f01b806ade0 0x7f01b806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:35.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.098+0000 7f01d89dd700 1 --2- 172.21.15.189:0/2403704349 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f01b806ade0 0x7f01b806d2a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f01c8007fd0 tx=0x7f01c8009040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:35.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.102+0000 7f01d17fa700 1 -- 172.21.15.189:0/2403704349 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f01d40c70d0 con 0x7f01d40d17a0 2024-04-06T13:47:35.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.299+0000 7f01dac41700 1 -- 172.21.15.189:0/2403704349 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01d40c70d0 con 0x7f01d40d17a0 2024-04-06T13:47:35.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.300+0000 7f01d17fa700 1 -- 172.21.15.189:0/2403704349 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v230) v1 ==== 74+0+20149 (secure 0 0 0) 0x7f01d40c70d0 con 0x7f01d40d17a0 2024-04-06T13:47:35.300 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:35.301 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":230,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:47:34.583870+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:47:20.644275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-04-06T13:47:21.075143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:47:32.292623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:47:33.567052+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:47:35.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.303+0000 7f01dac41700 1 -- 172.21.15.189:0/2403704349 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f01b806ade0 msgr2=0x7f01b806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:35.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.303+0000 7f01dac41700 1 --2- 172.21.15.189:0/2403704349 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f01b806ade0 0x7f01b806d2a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f01c8007fd0 tx=0x7f01c8009040 comp rx=0 tx=0).stop 2024-04-06T13:47:35.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.303+0000 7f01dac41700 1 -- 172.21.15.189:0/2403704349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d40d17a0 msgr2=0x7f01d41f3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:35.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.303+0000 7f01dac41700 1 --2- 172.21.15.189:0/2403704349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d40d17a0 0x7f01d41f3950 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f01c00090b0 tx=0x7f01c0004750 comp rx=0 tx=0).stop 2024-04-06T13:47:35.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.303+0000 7f01dac41700 1 -- 172.21.15.189:0/2403704349 shutdown_connections 2024-04-06T13:47:35.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.303+0000 7f01dac41700 1 --2- 172.21.15.189:0/2403704349 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f01b806ade0 0x7f01b806d2a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:35.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.303+0000 7f01dac41700 1 --2- 172.21.15.189:0/2403704349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01d4146dd0 0x7f01d40af400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:35.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.303+0000 7f01dac41700 1 --2- 172.21.15.189:0/2403704349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01d40cedf0 0x7f01d40c73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:35.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.303+0000 7f01dac41700 1 --2- 172.21.15.189:0/2403704349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d40d17a0 0x7f01d41f3950 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:35.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.303+0000 7f01dac41700 1 -- 172.21.15.189:0/2403704349 >> 172.21.15.189:0/2403704349 conn(0x7f01d40b2db0 msgr2=0x7f01d4160fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:35.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.303+0000 7f01dac41700 1 -- 172.21.15.189:0/2403704349 shutdown_connections 2024-04-06T13:47:35.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.303+0000 7f01dac41700 1 -- 172.21.15.189:0/2403704349 wait complete. 2024-04-06T13:47:35.316 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2024-04-06T13:47:35.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.440+0000 7fbaf023f700 1 Processor -- start 2024-04-06T13:47:35.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.440+0000 7fbaf023f700 1 -- start start 2024-04-06T13:47:35.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.441+0000 7fbaf023f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbae815d7b0 0x7fbae80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:35.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.441+0000 7fbaf023f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbae806a730 0x7fbae806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:35.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.441+0000 7fbaf023f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae80b3990 con 0x7fbae815ddf0 2024-04-06T13:47:35.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.441+0000 7fbaf023f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae806b0f0 con 0x7fbae806a730 2024-04-06T13:47:35.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.441+0000 7fbaf023f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae80b3060 con 0x7fbae815d7b0 2024-04-06T13:47:35.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.441+0000 7fbaed7da700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbae806a730 0x7fbae806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:35.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.441+0000 7fbaed7da700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbae806a730 0x7fbae806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55002/0 (socket says 172.21.15.189:55002) 2024-04-06T13:47:35.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.441+0000 7fbaed7da700 1 -- 172.21.15.189:0/1142279143 learned_addr learned my addr 172.21.15.189:0/1142279143 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:35.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.441+0000 7fbaed7da700 1 -- 172.21.15.189:0/1142279143 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbae815d7b0 msgr2=0x7fbae80b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:35.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.441+0000 7fbaed7da700 1 --2- 172.21.15.189:0/1142279143 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbae815d7b0 0x7fbae80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:35.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.441+0000 7fbaed7da700 1 -- 172.21.15.189:0/1142279143 >> v1:172.21.15.192:6789/0 conn(0x7fbae815ddf0 legacy=0x7fbae815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:35.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.441+0000 7fbaed7da700 1 -- 172.21.15.189:0/1142279143 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae80b3340 con 0x7fbae806a730 2024-04-06T13:47:35.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.442+0000 7fbaed7da700 1 --2- 172.21.15.189:0/1142279143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbae806a730 0x7fbae806abb0 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7fbad80098f0 tx=0x7fbad8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2ca54bdb812161 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:35.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.442+0000 7fbaecfd9700 1 -- 172.21.15.189:0/1142279143 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad8004230 con 0x7fbae806a730 2024-04-06T13:47:35.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.442+0000 7fbaecfd9700 1 -- 172.21.15.189:0/1142279143 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad8004390 con 0x7fbae806a730 2024-04-06T13:47:35.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.442+0000 7fbaecfd9700 1 -- 172.21.15.189:0/1142279143 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad80036a0 con 0x7fbae806a730 2024-04-06T13:47:35.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.442+0000 7fbaf023f700 1 -- 172.21.15.189:0/1142279143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbae806a730 msgr2=0x7fbae806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:35.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.442+0000 7fbaf023f700 1 --2- 172.21.15.189:0/1142279143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbae806a730 0x7fbae806abb0 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7fbad80098f0 tx=0x7fbad8009c00 comp rx=0 tx=0).stop 2024-04-06T13:47:35.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.442+0000 7fbaf023f700 1 -- 172.21.15.189:0/1142279143 shutdown_connections 2024-04-06T13:47:35.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.442+0000 7fbaf023f700 1 --2- 172.21.15.189:0/1142279143 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbae815d7b0 0x7fbae80b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:35.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.442+0000 7fbaf023f700 1 --2- 172.21.15.189:0/1142279143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbae806a730 0x7fbae806abb0 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:35.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.442+0000 7fbaf023f700 1 -- 172.21.15.189:0/1142279143 >> 172.21.15.189:0/1142279143 conn(0x7fbae80d00b0 msgr2=0x7fbae80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:35.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.442+0000 7fbaf023f700 1 -- 172.21.15.189:0/1142279143 shutdown_connections 2024-04-06T13:47:35.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.442+0000 7fbaf023f700 1 -- 172.21.15.189:0/1142279143 wait complete. 2024-04-06T13:47:35.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.444+0000 7fbaf023f700 1 Processor -- start 2024-04-06T13:47:35.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.444+0000 7fbaf023f700 1 -- start start 2024-04-06T13:47:35.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.444+0000 7fbaf023f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbae806a730 0x7fbae80ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:35.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.444+0000 7fbaf023f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbae815d7b0 0x7fbae815f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:35.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.444+0000 7fbaf023f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbae815ddf0 0x7fbae814f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:35.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.444+0000 7fbaf023f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae815ae90 con 0x7fbae815d7b0 2024-04-06T13:47:35.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.444+0000 7fbaf023f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae80ceba0 con 0x7fbae815ddf0 2024-04-06T13:47:35.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.444+0000 7fbaf023f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae8160650 con 0x7fbae806a730 2024-04-06T13:47:35.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.444+0000 7fbaed7da700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbae815d7b0 0x7fbae815f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:35.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.444+0000 7fbaed7da700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbae815d7b0 0x7fbae815f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55016/0 (socket says 172.21.15.189:55016) 2024-04-06T13:47:35.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.444+0000 7fbaed7da700 1 -- 172.21.15.189:0/2946284848 learned_addr learned my addr 172.21.15.189:0/2946284848 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:35.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.445+0000 7fbaed7da700 1 -- 172.21.15.189:0/2946284848 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbae806a730 msgr2=0x7fbae80ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:35.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.445+0000 7fbaedfdb700 1 --2- 172.21.15.189:0/2946284848 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbae806a730 0x7fbae80ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:35.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.445+0000 7fbaee7dc700 1 --2- 172.21.15.189:0/2946284848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbae815ddf0 0x7fbae814f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:35.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.445+0000 7fbaed7da700 1 --2- 172.21.15.189:0/2946284848 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbae806a730 0x7fbae80ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:35.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.445+0000 7fbaed7da700 1 -- 172.21.15.189:0/2946284848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbae815ddf0 msgr2=0x7fbae814f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:35.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.445+0000 7fbaed7da700 1 --2- 172.21.15.189:0/2946284848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbae815ddf0 0x7fbae814f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:35.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.445+0000 7fbaed7da700 1 -- 172.21.15.189:0/2946284848 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad8009550 con 0x7fbae815d7b0 2024-04-06T13:47:35.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.445+0000 7fbaed7da700 1 --2- 172.21.15.189:0/2946284848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbae815d7b0 0x7fbae815f6d0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009130 tx=0x7fbad8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:35.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.445+0000 7fbade7fc700 1 -- 172.21.15.189:0/2946284848 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad8003e70 con 0x7fbae815d7b0 2024-04-06T13:47:35.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.445+0000 7fbade7fc700 1 -- 172.21.15.189:0/2946284848 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad801f070 con 0x7fbae815d7b0 2024-04-06T13:47:35.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.445+0000 7fbaf023f700 1 -- 172.21.15.189:0/2946284848 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae8160090 con 0x7fbae815d7b0 2024-04-06T13:47:35.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.445+0000 7fbade7fc700 1 -- 172.21.15.189:0/2946284848 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad800b8a0 con 0x7fbae815d7b0 2024-04-06T13:47:35.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.445+0000 7fbaf023f700 1 -- 172.21.15.189:0/2946284848 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae815aab0 con 0x7fbae815d7b0 2024-04-06T13:47:35.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.446+0000 7fbaf023f700 1 -- 172.21.15.189:0/2946284848 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae815afd0 con 0x7fbae815d7b0 2024-04-06T13:47:35.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.447+0000 7fbade7fc700 1 -- 172.21.15.189:0/2946284848 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fbad8021030 con 0x7fbae815d7b0 2024-04-06T13:47:35.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.447+0000 7fbade7fc700 1 --2- 172.21.15.189:0/2946284848 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbacc06adb0 0x7fbacc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:35.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.447+0000 7fbade7fc700 1 -- 172.21.15.189:0/2946284848 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6965+0+0 (secure 0 0 0) 0x7fbad808aff0 con 0x7fbae815d7b0 2024-04-06T13:47:35.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.447+0000 7fbaedfdb700 1 --2- 172.21.15.189:0/2946284848 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbacc06adb0 0x7fbacc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:35.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.448+0000 7fbaedfdb700 1 --2- 172.21.15.189:0/2946284848 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbacc06adb0 0x7fbacc06d270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbae4006fd0 tx=0x7fbae4008040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:35.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.452+0000 7fbade7fc700 1 -- 172.21.15.189:0/2946284848 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbae815afd0 con 0x7fbae815d7b0 2024-04-06T13:47:35.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:35.691+0000 7fbaf023f700 1 -- 172.21.15.189:0/2946284848 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fbae815afd0 con 0x7fbae815d7b0 2024-04-06T13:47:36.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.594+0000 7fbade7fc700 1 -- 172.21.15.189:0/2946284848 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v105) v1 ==== 121+0+0 (secure 0 0 0) 0x7fbae815afd0 con 0x7fbae815d7b0 2024-04-06T13:47:36.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.597+0000 7fbaf023f700 1 -- 172.21.15.189:0/2946284848 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbacc06adb0 msgr2=0x7fbacc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:36.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.597+0000 7fbaf023f700 1 --2- 172.21.15.189:0/2946284848 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbacc06adb0 0x7fbacc06d270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbae4006fd0 tx=0x7fbae4008040 comp rx=0 tx=0).stop 2024-04-06T13:47:36.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.597+0000 7fbaf023f700 1 -- 172.21.15.189:0/2946284848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbae815d7b0 msgr2=0x7fbae815f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:36.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.597+0000 7fbaf023f700 1 --2- 172.21.15.189:0/2946284848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbae815d7b0 0x7fbae815f6d0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009130 tx=0x7fbad8004750 comp rx=0 tx=0).stop 2024-04-06T13:47:36.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.597+0000 7fbaf023f700 1 -- 172.21.15.189:0/2946284848 shutdown_connections 2024-04-06T13:47:36.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.597+0000 7fbaf023f700 1 --2- 172.21.15.189:0/2946284848 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbacc06adb0 0x7fbacc06d270 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:36.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.597+0000 7fbaf023f700 1 --2- 172.21.15.189:0/2946284848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbae815ddf0 0x7fbae814f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:36.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.597+0000 7fbaf023f700 1 --2- 172.21.15.189:0/2946284848 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbae806a730 0x7fbae80ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:36.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.597+0000 7fbaf023f700 1 --2- 172.21.15.189:0/2946284848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbae815d7b0 0x7fbae815f6d0 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:36.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.597+0000 7fbaf023f700 1 -- 172.21.15.189:0/2946284848 >> 172.21.15.189:0/2946284848 conn(0x7fbae80d00b0 msgr2=0x7fbae80ad560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:36.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.597+0000 7fbaf023f700 1 -- 172.21.15.189:0/2946284848 shutdown_connections 2024-04-06T13:47:36.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.597+0000 7fbaf023f700 1 -- 172.21.15.189:0/2946284848 wait complete. 2024-04-06T13:47:36.612 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-06T13:47:36.613 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2024-04-06T13:47:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.743+0000 7f7949495700 1 Processor -- start 2024-04-06T13:47:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.743+0000 7f7949495700 1 -- start start 2024-04-06T13:47:36.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.744+0000 7f7949495700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f794415d7b0 0x7f79440b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:36.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.744+0000 7f7949495700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79440b3990 con 0x7f794415e500 2024-04-06T13:47:36.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.744+0000 7f7949495700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79440b3060 con 0x7f794406a730 2024-04-06T13:47:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.744+0000 7f7949495700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f794415e950 con 0x7f794415d7b0 2024-04-06T13:47:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.744+0000 7f7942ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f794415d7b0 0x7f79440b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.744+0000 7f79437fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f794415e500 0x7f79440717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55178/0 (socket says 172.21.15.189:55178) 2024-04-06T13:47:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.744+0000 7f79437fe700 1 -- 172.21.15.189:0/2976269231 learned_addr learned my addr 172.21.15.189:0/2976269231 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.745+0000 7f7942ffd700 1 -- 172.21.15.189:0/2976269231 >> v1:172.21.15.192:6789/0 conn(0x7f794406a730 legacy=0x7f794415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.745+0000 7f7942ffd700 1 -- 172.21.15.189:0/2976269231 >> v1:172.21.15.189:6789/0 conn(0x7f794415e500 legacy=0x7f79440717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.745+0000 7f7942ffd700 1 -- 172.21.15.189:0/2976269231 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79440b3340 con 0x7f794415d7b0 2024-04-06T13:47:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.745+0000 7f7942ffd700 1 --2- 172.21.15.189:0/2976269231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f794415d7b0 0x7f79440b3e20 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7f7938009840 tx=0x7f7938009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18db2cf9575f07c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.745+0000 7f7941ffb700 1 -- 172.21.15.189:0/2976269231 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7938004230 con 0x7f794415d7b0 2024-04-06T13:47:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.745+0000 7f7941ffb700 1 -- 172.21.15.189:0/2976269231 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7938004390 con 0x7f794415d7b0 2024-04-06T13:47:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.745+0000 7f7941ffb700 1 -- 172.21.15.189:0/2976269231 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79380036a0 con 0x7f794415d7b0 2024-04-06T13:47:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.745+0000 7f7949495700 1 -- 172.21.15.189:0/2976269231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f794415d7b0 msgr2=0x7f79440b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.745+0000 7f7949495700 1 --2- 172.21.15.189:0/2976269231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f794415d7b0 0x7f79440b3e20 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7f7938009840 tx=0x7f7938009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.745+0000 7f7949495700 1 -- 172.21.15.189:0/2976269231 shutdown_connections 2024-04-06T13:47:36.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.745+0000 7f7949495700 1 --2- 172.21.15.189:0/2976269231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f794415d7b0 0x7f79440b3e20 unknown :-1 s=CLOSED pgs=2459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:36.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.745+0000 7f7949495700 1 -- 172.21.15.189:0/2976269231 >> 172.21.15.189:0/2976269231 conn(0x7f79440d00b0 msgr2=0x7f79440d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:36.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.746+0000 7f7949495700 1 -- 172.21.15.189:0/2976269231 shutdown_connections 2024-04-06T13:47:36.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.746+0000 7f7949495700 1 -- 172.21.15.189:0/2976269231 wait complete. 2024-04-06T13:47:36.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.747+0000 7f7949495700 1 Processor -- start 2024-04-06T13:47:36.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.747+0000 7f7949495700 1 -- start start 2024-04-06T13:47:36.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.747+0000 7f7949495700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f794406a730 0x7f79440cf380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:36.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.747+0000 7f7949495700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f794415d7b0 0x7f79440cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:36.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.747+0000 7f7949495700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f794415e500 0x7f79441f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:36.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f7949495700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794415b140 con 0x7f794415d7b0 2024-04-06T13:47:36.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f7949495700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79440cca80 con 0x7f794415e500 2024-04-06T13:47:36.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f7949495700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794415b390 con 0x7f794406a730 2024-04-06T13:47:36.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f79427fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f794415d7b0 0x7f79440cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:36.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f79427fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f794415d7b0 0x7f79440cc540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55036/0 (socket says 172.21.15.189:55036) 2024-04-06T13:47:36.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f79427fc700 1 -- 172.21.15.189:0/2998766804 learned_addr learned my addr 172.21.15.189:0/2998766804 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:36.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f7942ffd700 1 --2- 172.21.15.189:0/2998766804 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f794406a730 0x7f79440cf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:36.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f79437fe700 1 --2- 172.21.15.189:0/2998766804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f794415e500 0x7f79441f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:36.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f79427fc700 1 -- 172.21.15.189:0/2998766804 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f794406a730 msgr2=0x7f79440cf380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:36.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f79427fc700 1 --2- 172.21.15.189:0/2998766804 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f794406a730 0x7f79440cf380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:36.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f79427fc700 1 -- 172.21.15.189:0/2998766804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f794415e500 msgr2=0x7f79441f1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:36.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f79427fc700 1 --2- 172.21.15.189:0/2998766804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f794415e500 0x7f79441f1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:36.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f79427fc700 1 -- 172.21.15.189:0/2998766804 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79380094a0 con 0x7f794415d7b0 2024-04-06T13:47:36.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f79437fe700 1 --2- 172.21.15.189:0/2998766804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f794415e500 0x7f79441f1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:36.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f79427fc700 1 --2- 172.21.15.189:0/2998766804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f794415d7b0 0x7f79440cc540 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7f793400c4a0 tx=0x7f793400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:36.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f792b7fe700 1 -- 172.21.15.189:0/2998766804 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79340054e0 con 0x7f794415d7b0 2024-04-06T13:47:36.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f792b7fe700 1 -- 172.21.15.189:0/2998766804 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7934003680 con 0x7f794415d7b0 2024-04-06T13:47:36.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f7949495700 1 -- 172.21.15.189:0/2998766804 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794415b670 con 0x7f794415d7b0 2024-04-06T13:47:36.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f792b7fe700 1 -- 172.21.15.189:0/2998766804 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7934003a30 con 0x7f794415d7b0 2024-04-06T13:47:36.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.748+0000 7f7949495700 1 -- 172.21.15.189:0/2998766804 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794415bc30 con 0x7f794415d7b0 2024-04-06T13:47:36.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.749+0000 7f7949495700 1 -- 172.21.15.189:0/2998766804 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7944067bb0 con 0x7f794415d7b0 2024-04-06T13:47:36.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.750+0000 7f792b7fe700 1 -- 172.21.15.189:0/2998766804 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7934003b90 con 0x7f794415d7b0 2024-04-06T13:47:36.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.750+0000 7f792b7fe700 1 --2- 172.21.15.189:0/2998766804 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f793006adb0 0x7f793006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:36.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.750+0000 7f792b7fe700 1 -- 172.21.15.189:0/2998766804 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f7934088a40 con 0x7f794415d7b0 2024-04-06T13:47:36.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.750+0000 7f7942ffd700 1 --2- 172.21.15.189:0/2998766804 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f793006adb0 0x7f793006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:36.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.751+0000 7f7942ffd700 1 --2- 172.21.15.189:0/2998766804 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f793006adb0 0x7f793006d270 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7938009b90 tx=0x7f7938008030 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:36.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.755+0000 7f792b7fe700 1 -- 172.21.15.189:0/2998766804 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f793408ae60 con 0x7f794415d7b0 2024-04-06T13:47:36.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:36.998+0000 7f7949495700 1 -- 172.21.15.189:0/2998766804 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7944074990 con 0x7f794415d7b0 2024-04-06T13:47:37.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.601+0000 7f792b7fe700 1 -- 172.21.15.189:0/2998766804 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v106) v1 ==== 125+0+0 (secure 0 0 0) 0x7f7934053620 con 0x7f794415d7b0 2024-04-06T13:47:37.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.604+0000 7f7949495700 1 -- 172.21.15.189:0/2998766804 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f793006adb0 msgr2=0x7f793006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:37.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.604+0000 7f7949495700 1 --2- 172.21.15.189:0/2998766804 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f793006adb0 0x7f793006d270 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7938009b90 tx=0x7f7938008030 comp rx=0 tx=0).stop 2024-04-06T13:47:37.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.604+0000 7f7949495700 1 -- 172.21.15.189:0/2998766804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f794415d7b0 msgr2=0x7f79440cc540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:37.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.604+0000 7f7949495700 1 --2- 172.21.15.189:0/2998766804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f794415d7b0 0x7f79440cc540 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7f793400c4a0 tx=0x7f793400c860 comp rx=0 tx=0).stop 2024-04-06T13:47:37.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.604+0000 7f7949495700 1 -- 172.21.15.189:0/2998766804 shutdown_connections 2024-04-06T13:47:37.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.604+0000 7f7949495700 1 --2- 172.21.15.189:0/2998766804 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f793006adb0 0x7f793006d270 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:37.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.604+0000 7f7949495700 1 --2- 172.21.15.189:0/2998766804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f794415e500 0x7f79441f1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:37.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.604+0000 7f7949495700 1 --2- 172.21.15.189:0/2998766804 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f794406a730 0x7f79440cf380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:37.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.604+0000 7f7949495700 1 --2- 172.21.15.189:0/2998766804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f794415d7b0 0x7f79440cc540 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:37.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.604+0000 7f7949495700 1 -- 172.21.15.189:0/2998766804 >> 172.21.15.189:0/2998766804 conn(0x7f79440d00b0 msgr2=0x7f794415ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:37.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.604+0000 7f7949495700 1 -- 172.21.15.189:0/2998766804 shutdown_connections 2024-04-06T13:47:37.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.604+0000 7f7949495700 1 -- 172.21.15.189:0/2998766804 wait complete. 2024-04-06T13:47:37.618 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2024-04-06T13:47:37.618 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:47:37.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.744+0000 7f01a1fae700 1 Processor -- start 2024-04-06T13:47:37.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.744+0000 7f01a1fae700 1 -- start start 2024-04-06T13:47:37.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.745+0000 7f01a1fae700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f019c144f40 0x7f019c06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.745+0000 7f01a1fae700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019c15c740 con 0x7f019c15c150 2024-04-06T13:47:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.745+0000 7f01a1fae700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019c15f1f0 con 0x7f019c15fcd0 2024-04-06T13:47:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.745+0000 7f01a1fae700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f019c15ca20 con 0x7f019c144f40 2024-04-06T13:47:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.745+0000 7f019bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f019c15c150 0x7f019c0ceca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55182/0 (socket says 172.21.15.189:55182) 2024-04-06T13:47:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.745+0000 7f019bfff700 1 -- 172.21.15.189:0/2085361457 learned_addr learned my addr 172.21.15.189:0/2085361457 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.745+0000 7f019a7fc700 1 -- 172.21.15.189:0/2085361457 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398521327 0 0) 0x7f019c15c740 con 0x7f019c15c150 2024-04-06T13:47:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.745+0000 7f019a7fc700 1 -- 172.21.15.189:0/2085361457 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01840035b0 con 0x7f019c15c150 2024-04-06T13:47:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.745+0000 7f019b7fe700 1 --2- 172.21.15.189:0/2085361457 >> v2:172.21.15.192:3300/0 conn(0x7f019c144f40 0x7f019c06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019a7fc700 1 -- 172.21.15.189:0/2085361457 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1889067067 0 0) 0x7f01840035b0 con 0x7f019c15c150 2024-04-06T13:47:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019a7fc700 1 -- 172.21.15.189:0/2085361457 >> v2:172.21.15.192:3300/0 conn(0x7f019c144f40 msgr2=0x7f019c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019a7fc700 1 --2- 172.21.15.189:0/2085361457 >> v2:172.21.15.192:3300/0 conn(0x7f019c144f40 0x7f019c06a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019a7fc700 1 -- 172.21.15.189:0/2085361457 >> v1:172.21.15.192:6789/0 conn(0x7f019c15fcd0 legacy=0x7f019c15ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019a7fc700 1 -- 172.21.15.189:0/2085361457 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c15f4d0 con 0x7f019c15c150 2024-04-06T13:47:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019a7fc700 1 -- 172.21.15.189:0/2085361457 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f01940019f0 con 0x7f019c15c150 2024-04-06T13:47:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019a7fc700 1 -- 172.21.15.189:0/2085361457 >> v1:172.21.15.189:6789/0 conn(0x7f019c15c150 legacy=0x7f019c0ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019a7fc700 1 --2- 172.21.15.189:0/2085361457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0184003a00 0x7f0184003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019a7fc700 1 --2- 172.21.15.189:0/2085361457 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0184004850 0x7f0184004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019a7fc700 1 --2- 172.21.15.189:0/2085361457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0184005330 0x7f018400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019a7fc700 1 -- 172.21.15.189:0/2085361457 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019c15ca20 con 0x7f0184003a00 2024-04-06T13:47:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019a7fc700 1 -- 172.21.15.189:0/2085361457 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018400c1e0 con 0x7f0184005330 2024-04-06T13:47:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019a7fc700 1 -- 172.21.15.189:0/2085361457 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018400c3a0 con 0x7f0184004850 2024-04-06T13:47:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019b7fe700 1 --2- 172.21.15.189:0/2085361457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0184003a00 0x7f0184003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019affd700 1 --2- 172.21.15.189:0/2085361457 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0184004850 0x7f0184004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019bfff700 1 --2- 172.21.15.189:0/2085361457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0184005330 0x7f018400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019b7fe700 1 -- 172.21.15.189:0/2085361457 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0184004850 msgr2=0x7f0184004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:37.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019b7fe700 1 --2- 172.21.15.189:0/2085361457 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0184004850 0x7f0184004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:37.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019b7fe700 1 -- 172.21.15.189:0/2085361457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0184005330 msgr2=0x7f018400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:37.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019b7fe700 1 --2- 172.21.15.189:0/2085361457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0184005330 0x7f018400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:37.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019bfff700 1 -- 172.21.15.189:0/2085361457 reap_dead start 2024-04-06T13:47:37.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019b7fe700 1 -- 172.21.15.189:0/2085361457 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01840035b0 con 0x7f0184003a00 2024-04-06T13:47:37.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019b7fe700 1 --2- 172.21.15.189:0/2085361457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0184003a00 0x7f0184003e00 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f0190009860 tx=0x7f0184005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=367411aa2297d44b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:37.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019a7fc700 1 -- 172.21.15.189:0/2085361457 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019000c070 con 0x7f0184003a00 2024-04-06T13:47:37.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019a7fc700 1 -- 172.21.15.189:0/2085361457 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0190013610 con 0x7f0184003a00 2024-04-06T13:47:37.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.746+0000 7f019a7fc700 1 -- 172.21.15.189:0/2085361457 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01900139a0 con 0x7f0184003a00 2024-04-06T13:47:37.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.747+0000 7f01a1fae700 1 -- 172.21.15.189:0/2085361457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0184003a00 msgr2=0x7f0184003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:37.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.747+0000 7f01a1fae700 1 --2- 172.21.15.189:0/2085361457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0184003a00 0x7f0184003e00 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f0190009860 tx=0x7f0184005880 comp rx=0 tx=0).stop 2024-04-06T13:47:37.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.747+0000 7f01a1fae700 1 -- 172.21.15.189:0/2085361457 shutdown_connections 2024-04-06T13:47:37.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.747+0000 7f01a1fae700 1 --2- 172.21.15.189:0/2085361457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0184003a00 0x7f0184003e00 unknown :-1 s=CLOSED pgs=2462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:37.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.747+0000 7f01a1fae700 1 -- 172.21.15.189:0/2085361457 >> 172.21.15.189:0/2085361457 conn(0x7f019c0d00b0 msgr2=0x7f019c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:37.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.747+0000 7f01a1fae700 1 -- 172.21.15.189:0/2085361457 shutdown_connections 2024-04-06T13:47:37.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.747+0000 7f01a1fae700 1 -- 172.21.15.189:0/2085361457 wait complete. 2024-04-06T13:47:37.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.748+0000 7f01a1fae700 1 Processor -- start 2024-04-06T13:47:37.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.748+0000 7f01a1fae700 1 -- start start 2024-04-06T13:47:37.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.749+0000 7f01a1fae700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0184003a00 0x7f019c1419b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:37.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.749+0000 7f01a1fae700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f019c1438c0 0x7f019c142340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:37.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.749+0000 7f01a1fae700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f019c15b130 0x7f019c142880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:37.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.749+0000 7f01a1fae700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019c141ef0 con 0x7f0184003a00 2024-04-06T13:47:37.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.749+0000 7f01a1fae700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019c0ca070 con 0x7f019c15b130 2024-04-06T13:47:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.749+0000 7f01a1fae700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019c0ca350 con 0x7f019c1438c0 2024-04-06T13:47:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.749+0000 7f019b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0184003a00 0x7f019c1419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.749+0000 7f019b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0184003a00 0x7f019c1419b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55066/0 (socket says 172.21.15.189:55066) 2024-04-06T13:47:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.749+0000 7f019b7fe700 1 -- 172.21.15.189:0/2024219055 learned_addr learned my addr 172.21.15.189:0/2024219055 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.749+0000 7f019affd700 1 --2- 172.21.15.189:0/2024219055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f019c1438c0 0x7f019c142340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.749+0000 7f019bfff700 1 --2- 172.21.15.189:0/2024219055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f019c15b130 0x7f019c142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.750+0000 7f019b7fe700 1 -- 172.21.15.189:0/2024219055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f019c1438c0 msgr2=0x7f019c142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.750+0000 7f019b7fe700 1 --2- 172.21.15.189:0/2024219055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f019c1438c0 0x7f019c142340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.750+0000 7f019b7fe700 1 -- 172.21.15.189:0/2024219055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f019c15b130 msgr2=0x7f019c142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.750+0000 7f019b7fe700 1 --2- 172.21.15.189:0/2024219055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f019c15b130 0x7f019c142880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.750+0000 7f019b7fe700 1 -- 172.21.15.189:0/2024219055 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c1430c0 con 0x7f0184003a00 2024-04-06T13:47:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.750+0000 7f019affd700 1 --2- 172.21.15.189:0/2024219055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f019c1438c0 0x7f019c142340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.750+0000 7f019bfff700 1 --2- 172.21.15.189:0/2024219055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f019c15b130 0x7f019c142880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.750+0000 7f019b7fe700 1 --2- 172.21.15.189:0/2024219055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0184003a00 0x7f019c1419b0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f0190016040 tx=0x7f01900047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.750+0000 7f0183fff700 1 -- 172.21.15.189:0/2024219055 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0190013c50 con 0x7f0184003a00 2024-04-06T13:47:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.750+0000 7f0183fff700 1 -- 172.21.15.189:0/2024219055 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0190013db0 con 0x7f0184003a00 2024-04-06T13:47:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.750+0000 7f0183fff700 1 -- 172.21.15.189:0/2024219055 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01900038e0 con 0x7f0184003a00 2024-04-06T13:47:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.750+0000 7f01a1fae700 1 -- 172.21.15.189:0/2024219055 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f019c1433a0 con 0x7f0184003a00 2024-04-06T13:47:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.750+0000 7f01a1fae700 1 -- 172.21.15.189:0/2024219055 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f019c0c9d20 con 0x7f0184003a00 2024-04-06T13:47:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.751+0000 7f01a1fae700 1 -- 172.21.15.189:0/2024219055 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f019c143710 con 0x7f0184003a00 2024-04-06T13:47:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.751+0000 7f0183fff700 1 -- 172.21.15.189:0/2024219055 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0190003a40 con 0x7f0184003a00 2024-04-06T13:47:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.752+0000 7f0183fff700 1 --2- 172.21.15.189:0/2024219055 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f018806adb0 0x7f018806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.752+0000 7f0183fff700 1 -- 172.21.15.189:0/2024219055 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f019008aac0 con 0x7f0184003a00 2024-04-06T13:47:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.752+0000 7f019affd700 1 --2- 172.21.15.189:0/2024219055 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f018806adb0 0x7f018806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:37.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.752+0000 7f019affd700 1 --2- 172.21.15.189:0/2024219055 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f018806adb0 0x7f018806d270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f019c0b1c20 tx=0x7f018c009040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:37.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.757+0000 7f0183fff700 1 -- 172.21.15.189:0/2024219055 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0190055620 con 0x7f0184003a00 2024-04-06T13:47:37.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.995+0000 7f01a1fae700 1 -- 172.21.15.189:0/2024219055 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f019c067bb0 con 0x7f0184003a00 2024-04-06T13:47:37.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.995+0000 7f0183fff700 1 -- 172.21.15.189:0/2024219055 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5971 (secure 0 0 0) 0x7f0190058c40 con 0x7f0184003a00 2024-04-06T13:47:37.996 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:37.996 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":106,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8885,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1481817364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1481817364},{"type":"v1","addr":"172.21.15.189:6837","nonce":1481817364}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":11244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/4021875447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":4021875447},{"type":"v1","addr":"172.21.15.189:6836","nonce":4021875447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:47:22.116705+0000","modified":"2024-04-06T13:47:25.129787+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/694564673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":694564673},{"type":"v1","addr":"172.21.15.192:6835","nonce":694564673}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:47:34.584208+0000","modified":"2024-04-06T13:47:37.592983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11262":{"gid":11262,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6834/4066431971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4066431971},{"type":"v1","addr":"172.21.15.192:6834","nonce":4066431971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-04-06T13:47:37.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.997+0000 7f01a1fae700 1 -- 172.21.15.189:0/2024219055 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f018806adb0 msgr2=0x7f018806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:37.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.997+0000 7f01a1fae700 1 --2- 172.21.15.189:0/2024219055 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f018806adb0 0x7f018806d270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f019c0b1c20 tx=0x7f018c009040 comp rx=0 tx=0).stop 2024-04-06T13:47:37.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.997+0000 7f01a1fae700 1 -- 172.21.15.189:0/2024219055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0184003a00 msgr2=0x7f019c1419b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:37.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.997+0000 7f01a1fae700 1 --2- 172.21.15.189:0/2024219055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0184003a00 0x7f019c1419b0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f0190016040 tx=0x7f01900047c0 comp rx=0 tx=0).stop 2024-04-06T13:47:37.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.997+0000 7f01a1fae700 1 -- 172.21.15.189:0/2024219055 shutdown_connections 2024-04-06T13:47:37.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.997+0000 7f01a1fae700 1 --2- 172.21.15.189:0/2024219055 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f018806adb0 0x7f018806d270 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:37.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.997+0000 7f01a1fae700 1 --2- 172.21.15.189:0/2024219055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f019c15b130 0x7f019c142880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:37.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.997+0000 7f01a1fae700 1 --2- 172.21.15.189:0/2024219055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0184003a00 0x7f019c1419b0 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:37.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.997+0000 7f01a1fae700 1 --2- 172.21.15.189:0/2024219055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f019c1438c0 0x7f019c142340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:37.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.997+0000 7f01a1fae700 1 -- 172.21.15.189:0/2024219055 >> 172.21.15.189:0/2024219055 conn(0x7f019c0d00b0 msgr2=0x7f019c06f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:37.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.997+0000 7f01a1fae700 1 -- 172.21.15.189:0/2024219055 shutdown_connections 2024-04-06T13:47:37.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:37.997+0000 7f01a1fae700 1 -- 172.21.15.189:0/2024219055 wait complete. 2024-04-06T13:47:37.999 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 106 2024-04-06T13:47:38.010 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:47:38.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.132+0000 7f739c712700 1 Processor -- start 2024-04-06T13:47:38.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.132+0000 7f739c712700 1 -- start start 2024-04-06T13:47:38.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.133+0000 7f739c712700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f739415a550 0x7f73940cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.133+0000 7f739c712700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f73940cf900 0x7f73940b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.133+0000 7f739c712700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73940cdbe0 con 0x7f73940b4ba0 2024-04-06T13:47:38.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.133+0000 7f739c712700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f739415a940 con 0x7f73940cf900 2024-04-06T13:47:38.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.133+0000 7f739c712700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f739415ac00 con 0x7f739415a550 2024-04-06T13:47:38.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.133+0000 7f7399cad700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f73940cf900 0x7f73940b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.133+0000 7f739acaf700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f73940b4ba0 0x7f739406eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55192/0 (socket says 172.21.15.189:55192) 2024-04-06T13:47:38.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.133+0000 7f739acaf700 1 -- 172.21.15.189:0/843733453 learned_addr learned my addr 172.21.15.189:0/843733453 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:38.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.134+0000 7f739a4ae700 1 --2- 172.21.15.189:0/843733453 >> v2:172.21.15.192:3300/0 conn(0x7f739415a550 0x7f73940cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.134+0000 7f7399cad700 1 -- 172.21.15.189:0/843733453 >> v2:172.21.15.192:3300/0 conn(0x7f739415a550 msgr2=0x7f73940cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:38.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.134+0000 7f7399cad700 1 --2- 172.21.15.189:0/843733453 >> v2:172.21.15.192:3300/0 conn(0x7f739415a550 0x7f73940cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.134+0000 7f7399cad700 1 -- 172.21.15.189:0/843733453 >> v1:172.21.15.189:6789/0 conn(0x7f73940b4ba0 legacy=0x7f739406eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:38.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.134+0000 7f7399cad700 1 -- 172.21.15.189:0/843733453 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739415aee0 con 0x7f73940cf900 2024-04-06T13:47:38.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.134+0000 7f73994ac700 1 -- 172.21.15.189:0/843733453 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678282794 0 0) 0x7f73940cdbe0 con 0x7f73940b4ba0 2024-04-06T13:47:38.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.134+0000 7f7399cad700 1 --2- 172.21.15.189:0/843733453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73940cf900 0x7f73940b4660 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7f7394073470 tx=0x7f7384009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce55c34e07bec2af server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:38.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.134+0000 7f73994ac700 1 -- 172.21.15.189:0/843733453 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7384011830 con 0x7f73940cf900 2024-04-06T13:47:38.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.134+0000 7f73994ac700 1 -- 172.21.15.189:0/843733453 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7384011990 con 0x7f73940cf900 2024-04-06T13:47:38.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.134+0000 7f73994ac700 1 -- 172.21.15.189:0/843733453 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7384011d20 con 0x7f73940cf900 2024-04-06T13:47:38.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.135+0000 7f739c712700 1 -- 172.21.15.189:0/843733453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73940cf900 msgr2=0x7f73940b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:38.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.135+0000 7f739c712700 1 --2- 172.21.15.189:0/843733453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73940cf900 0x7f73940b4660 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7f7394073470 tx=0x7f7384009b20 comp rx=0 tx=0).stop 2024-04-06T13:47:38.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.135+0000 7f739c712700 1 -- 172.21.15.189:0/843733453 shutdown_connections 2024-04-06T13:47:38.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.135+0000 7f739c712700 1 --2- 172.21.15.189:0/843733453 >> v2:172.21.15.192:3300/0 conn(0x7f739415a550 0x7f73940cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.135+0000 7f739c712700 1 --2- 172.21.15.189:0/843733453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73940cf900 0x7f73940b4660 unknown :-1 s=CLOSED pgs=2465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.135+0000 7f739c712700 1 -- 172.21.15.189:0/843733453 >> 172.21.15.189:0/843733453 conn(0x7f73940d00b0 msgr2=0x7f73940d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:38.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.135+0000 7f739c712700 1 -- 172.21.15.189:0/843733453 shutdown_connections 2024-04-06T13:47:38.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.135+0000 7f739c712700 1 -- 172.21.15.189:0/843733453 wait complete. 2024-04-06T13:47:38.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.136+0000 7f739c712700 1 Processor -- start 2024-04-06T13:47:38.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.136+0000 7f739c712700 1 -- start start 2024-04-06T13:47:38.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.137+0000 7f739c712700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73940b4ba0 0x7f739414ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.137+0000 7f739c712700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73940cf900 0x7f7394152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.137+0000 7f739c712700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7394159180 0x7f739414fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.137+0000 7f739c712700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739414e1e0 con 0x7f7394159180 2024-04-06T13:47:38.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.137+0000 7f739c712700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7394152c40 con 0x7f73940cf900 2024-04-06T13:47:38.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.137+0000 7f739c712700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f739414e480 con 0x7f73940b4ba0 2024-04-06T13:47:38.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.137+0000 7f739acaf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7394159180 0x7f739414fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.137+0000 7f739acaf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7394159180 0x7f739414fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55090/0 (socket says 172.21.15.189:55090) 2024-04-06T13:47:38.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.137+0000 7f739acaf700 1 -- 172.21.15.189:0/2865329634 learned_addr learned my addr 172.21.15.189:0/2865329634 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:38.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.137+0000 7f7399cad700 1 --2- 172.21.15.189:0/2865329634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73940cf900 0x7f7394152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.137+0000 7f739a4ae700 1 --2- 172.21.15.189:0/2865329634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73940b4ba0 0x7f739414ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.137+0000 7f739acaf700 1 -- 172.21.15.189:0/2865329634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73940b4ba0 msgr2=0x7f739414ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:38.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.137+0000 7f739acaf700 1 --2- 172.21.15.189:0/2865329634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73940b4ba0 0x7f739414ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.137+0000 7f739acaf700 1 -- 172.21.15.189:0/2865329634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73940cf900 msgr2=0x7f7394152300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:38.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.137+0000 7f739acaf700 1 --2- 172.21.15.189:0/2865329634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73940cf900 0x7f7394152300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.137+0000 7f739acaf700 1 -- 172.21.15.189:0/2865329634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73840094a0 con 0x7f7394159180 2024-04-06T13:47:38.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.137+0000 7f739acaf700 1 --2- 172.21.15.189:0/2865329634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7394159180 0x7f739414fc80 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f738c002230 tx=0x7f738c00cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:38.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.138+0000 7f738affd700 1 -- 172.21.15.189:0/2865329634 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c00aac0 con 0x7f7394159180 2024-04-06T13:47:38.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.138+0000 7f738affd700 1 -- 172.21.15.189:0/2865329634 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f738c00ac40 con 0x7f7394159180 2024-04-06T13:47:38.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.138+0000 7f739c712700 1 -- 172.21.15.189:0/2865329634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739414e760 con 0x7f7394159180 2024-04-06T13:47:38.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.138+0000 7f738affd700 1 -- 172.21.15.189:0/2865329634 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c0070d0 con 0x7f7394159180 2024-04-06T13:47:38.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.138+0000 7f739c712700 1 -- 172.21.15.189:0/2865329634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739414f4a0 con 0x7f7394159180 2024-04-06T13:47:38.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.139+0000 7f739c712700 1 -- 172.21.15.189:0/2865329634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f739414ec80 con 0x7f7394159180 2024-04-06T13:47:38.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.139+0000 7f738affd700 1 -- 172.21.15.189:0/2865329634 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f738c007250 con 0x7f7394159180 2024-04-06T13:47:38.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.140+0000 7f738affd700 1 --2- 172.21.15.189:0/2865329634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f737806adb0 0x7f737806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.140+0000 7f738affd700 1 -- 172.21.15.189:0/2865329634 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f738c089690 con 0x7f7394159180 2024-04-06T13:47:38.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.140+0000 7f739a4ae700 1 --2- 172.21.15.189:0/2865329634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f737806adb0 0x7f737806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.140+0000 7f739a4ae700 1 --2- 172.21.15.189:0/2865329634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f737806adb0 0x7f737806d270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f739414fb30 tx=0x7f7390009210 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:38.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.145+0000 7f738affd700 1 -- 172.21.15.189:0/2865329634 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f738c054300 con 0x7f7394159180 2024-04-06T13:47:38.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.342+0000 7f739c712700 1 -- 172.21.15.189:0/2865329634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7394067bb0 con 0x7f7394159180 2024-04-06T13:47:38.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.343+0000 7f738affd700 1 -- 172.21.15.189:0/2865329634 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v231) v1 ==== 74+0+20136 (secure 0 0 0) 0x7f738c057920 con 0x7f7394159180 2024-04-06T13:47:38.343 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:38.344 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":231,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:47:35.586023+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:47:20.644275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-04-06T13:47:21.075143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:47:32.292623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:47:33.567052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:47:38.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.345+0000 7f739c712700 1 -- 172.21.15.189:0/2865329634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f737806adb0 msgr2=0x7f737806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:38.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.345+0000 7f739c712700 1 --2- 172.21.15.189:0/2865329634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f737806adb0 0x7f737806d270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f739414fb30 tx=0x7f7390009210 comp rx=0 tx=0).stop 2024-04-06T13:47:38.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.345+0000 7f739c712700 1 -- 172.21.15.189:0/2865329634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7394159180 msgr2=0x7f739414fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:38.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.345+0000 7f739c712700 1 --2- 172.21.15.189:0/2865329634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7394159180 0x7f739414fc80 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f738c002230 tx=0x7f738c00cf50 comp rx=0 tx=0).stop 2024-04-06T13:47:38.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.346+0000 7f739c712700 1 -- 172.21.15.189:0/2865329634 shutdown_connections 2024-04-06T13:47:38.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.346+0000 7f739c712700 1 --2- 172.21.15.189:0/2865329634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f737806adb0 0x7f737806d270 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.346+0000 7f739c712700 1 --2- 172.21.15.189:0/2865329634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7394159180 0x7f739414fc80 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.346+0000 7f739c712700 1 --2- 172.21.15.189:0/2865329634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73940b4ba0 0x7f739414ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.346+0000 7f739c712700 1 --2- 172.21.15.189:0/2865329634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73940cf900 0x7f7394152300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.346+0000 7f739c712700 1 -- 172.21.15.189:0/2865329634 >> 172.21.15.189:0/2865329634 conn(0x7f73940d00b0 msgr2=0x7f73940b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.346+0000 7f739c712700 1 -- 172.21.15.189:0/2865329634 shutdown_connections 2024-04-06T13:47:38.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.346+0000 7f739c712700 1 -- 172.21.15.189:0/2865329634 wait complete. 2024-04-06T13:47:38.358 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-06T13:47:38.359 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:47:38.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.481+0000 7fe8ee8d5700 1 Processor -- start 2024-04-06T13:47:38.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.481+0000 7fe8ee8d5700 1 -- start start 2024-04-06T13:47:38.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.482+0000 7fe8ee8d5700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe8e8146dd0 0x7fe8e806d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.482+0000 7fe8ee8d5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8e80cedf0 0x7fe8e80cf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.482+0000 7fe8ee8d5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e80cf780 con 0x7fe8e80d17a0 2024-04-06T13:47:38.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.482+0000 7fe8ee8d5700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8e806beb0 con 0x7fe8e8146dd0 2024-04-06T13:47:38.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.482+0000 7fe8ee8d5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e806c0a0 con 0x7fe8e80cedf0 2024-04-06T13:47:38.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.482+0000 7fe8e7fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe8e80d17a0 0x7fe8e806b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55202/0 (socket says 172.21.15.189:55202) 2024-04-06T13:47:38.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.482+0000 7fe8e7fff700 1 -- 172.21.15.189:0/4256366723 learned_addr learned my addr 172.21.15.189:0/4256366723 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:38.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e6ffd700 1 -- 172.21.15.189:0/4256366723 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820972355 0 0) 0x7fe8d0001110 con 0x7fe8e80d17a0 2024-04-06T13:47:38.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e77fe700 1 --2- 172.21.15.189:0/4256366723 >> v2:172.21.15.192:3300/0 conn(0x7fe8e8146dd0 0x7fe8e806d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8ece72700 1 --2- 172.21.15.189:0/4256366723 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8e80cedf0 0x7fe8e80cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e6ffd700 1 -- 172.21.15.189:0/4256366723 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d40035b0 con 0x7fe8e80d17a0 2024-04-06T13:47:38.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e6ffd700 1 -- 172.21.15.189:0/4256366723 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 183950035 0 0) 0x7fe8d40035b0 con 0x7fe8e80d17a0 2024-04-06T13:47:38.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e6ffd700 1 -- 172.21.15.189:0/4256366723 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8e80cedf0 msgr2=0x7fe8e80cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:38.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e6ffd700 1 --2- 172.21.15.189:0/4256366723 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8e80cedf0 0x7fe8e80cf240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e6ffd700 1 -- 172.21.15.189:0/4256366723 >> v2:172.21.15.192:3300/0 conn(0x7fe8e8146dd0 msgr2=0x7fe8e806d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:38.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e6ffd700 1 --2- 172.21.15.189:0/4256366723 >> v2:172.21.15.192:3300/0 conn(0x7fe8e8146dd0 0x7fe8e806d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e6ffd700 1 -- 172.21.15.189:0/4256366723 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e806c380 con 0x7fe8e80d17a0 2024-04-06T13:47:38.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e6ffd700 1 -- 172.21.15.189:0/4256366723 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fe8d0001800 con 0x7fe8e80d17a0 2024-04-06T13:47:38.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e6ffd700 1 -- 172.21.15.189:0/4256366723 >> v1:172.21.15.189:6789/0 conn(0x7fe8e80d17a0 legacy=0x7fe8e806b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:38.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e6ffd700 1 --2- 172.21.15.189:0/4256366723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8d4003a00 0x7fe8d4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e6ffd700 1 --2- 172.21.15.189:0/4256366723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8d4004850 0x7fe8d4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e6ffd700 1 --2- 172.21.15.189:0/4256366723 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8e80cedf0 0x7fe8d4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e6ffd700 1 -- 172.21.15.189:0/4256366723 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e806c0a0 con 0x7fe8d4003a00 2024-04-06T13:47:38.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e6ffd700 1 -- 172.21.15.189:0/4256366723 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d400ba10 con 0x7fe8d4004850 2024-04-06T13:47:38.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e6ffd700 1 -- 172.21.15.189:0/4256366723 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d400bb50 con 0x7fe8e80cedf0 2024-04-06T13:47:38.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e7fff700 1 --2- 172.21.15.189:0/4256366723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8d4003a00 0x7fe8d4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e77fe700 1 --2- 172.21.15.189:0/4256366723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8d4004850 0x7fe8d4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8ece72700 1 --2- 172.21.15.189:0/4256366723 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8e80cedf0 0x7fe8d4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e7fff700 1 -- 172.21.15.189:0/4256366723 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8e80cedf0 msgr2=0x7fe8d4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:38.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e7fff700 1 --2- 172.21.15.189:0/4256366723 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8e80cedf0 0x7fe8d4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e7fff700 1 -- 172.21.15.189:0/4256366723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8d4004850 msgr2=0x7fe8d4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:38.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e7fff700 1 --2- 172.21.15.189:0/4256366723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8d4004850 0x7fe8d4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e7fff700 1 -- 172.21.15.189:0/4256366723 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d40035b0 con 0x7fe8d4003a00 2024-04-06T13:47:38.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e77fe700 1 --2- 172.21.15.189:0/4256366723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8d4004850 0x7fe8d4004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:38.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.483+0000 7fe8e7fff700 1 --2- 172.21.15.189:0/4256366723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8d4003a00 0x7fe8d4003e20 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7fe8d000c160 tx=0x7fe8d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84b90cf10f526c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:38.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.484+0000 7fe8e6ffd700 1 -- 172.21.15.189:0/4256366723 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0007d10 con 0x7fe8d4003a00 2024-04-06T13:47:38.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.484+0000 7fe8e6ffd700 1 -- 172.21.15.189:0/4256366723 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d000cca0 con 0x7fe8d4003a00 2024-04-06T13:47:38.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.484+0000 7fe8e6ffd700 1 -- 172.21.15.189:0/4256366723 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0005ba0 con 0x7fe8d4003a00 2024-04-06T13:47:38.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.484+0000 7fe8ee8d5700 1 -- 172.21.15.189:0/4256366723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8d4003a00 msgr2=0x7fe8d4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:38.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.484+0000 7fe8ee8d5700 1 --2- 172.21.15.189:0/4256366723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8d4003a00 0x7fe8d4003e20 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7fe8d000c160 tx=0x7fe8d4005880 comp rx=0 tx=0).stop 2024-04-06T13:47:38.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.484+0000 7fe8ece72700 1 -- 172.21.15.189:0/4256366723 reap_dead start 2024-04-06T13:47:38.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.484+0000 7fe8ee8d5700 1 -- 172.21.15.189:0/4256366723 shutdown_connections 2024-04-06T13:47:38.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.484+0000 7fe8ee8d5700 1 -- 172.21.15.189:0/4256366723 >> 172.21.15.189:0/4256366723 conn(0x7fe8e80b2db0 msgr2=0x7fe8e80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:38.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.484+0000 7fe8ee8d5700 1 -- 172.21.15.189:0/4256366723 shutdown_connections 2024-04-06T13:47:38.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.484+0000 7fe8ee8d5700 1 -- 172.21.15.189:0/4256366723 wait complete. 2024-04-06T13:47:38.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.485+0000 7fe8ee8d5700 1 Processor -- start 2024-04-06T13:47:38.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.485+0000 7fe8ee8d5700 1 -- start start 2024-04-06T13:47:38.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.486+0000 7fe8ee8d5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8d4003a00 0x7fe8e815dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.486+0000 7fe8ee8d5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8e80af090 0x7fe8e80af510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.486+0000 7fe8ee8d5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8e81f62c0 0x7fe8e81f6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.486+0000 7fe8ee8d5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e80c5da0 con 0x7fe8d4003a00 2024-04-06T13:47:38.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.486+0000 7fe8ee8d5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e80c6920 con 0x7fe8e80af090 2024-04-06T13:47:38.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.486+0000 7fe8ee8d5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e80c5ff0 con 0x7fe8e81f62c0 2024-04-06T13:47:38.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.486+0000 7fe8e7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8d4003a00 0x7fe8e815dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.486+0000 7fe8e7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8d4003a00 0x7fe8e815dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55110/0 (socket says 172.21.15.189:55110) 2024-04-06T13:47:38.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.486+0000 7fe8e7fff700 1 -- 172.21.15.189:0/1720079007 learned_addr learned my addr 172.21.15.189:0/1720079007 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:38.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.487+0000 7fe8e77fe700 1 --2- 172.21.15.189:0/1720079007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8e80af090 0x7fe8e80af510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.487+0000 7fe8ece72700 1 --2- 172.21.15.189:0/1720079007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8e81f62c0 0x7fe8e81f6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.487+0000 7fe8e7fff700 1 -- 172.21.15.189:0/1720079007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8e81f62c0 msgr2=0x7fe8e81f6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:38.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.487+0000 7fe8e7fff700 1 --2- 172.21.15.189:0/1720079007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8e81f62c0 0x7fe8e81f6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.487+0000 7fe8e7fff700 1 -- 172.21.15.189:0/1720079007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8e80af090 msgr2=0x7fe8e80af510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:38.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.487+0000 7fe8e7fff700 1 --2- 172.21.15.189:0/1720079007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8e80af090 0x7fe8e80af510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.487+0000 7fe8e7fff700 1 -- 172.21.15.189:0/1720079007 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e80c62d0 con 0x7fe8d4003a00 2024-04-06T13:47:38.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.487+0000 7fe8e77fe700 1 --2- 172.21.15.189:0/1720079007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8e80af090 0x7fe8e80af510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:38.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.487+0000 7fe8ece72700 1 --2- 172.21.15.189:0/1720079007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8e81f62c0 0x7fe8e81f6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:38.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.487+0000 7fe8e7fff700 1 --2- 172.21.15.189:0/1720079007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8d4003a00 0x7fe8e815dd40 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00019d0 tx=0x7fe8d0000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:38.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.487+0000 7fe8e4ff9700 1 -- 172.21.15.189:0/1720079007 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0004c80 con 0x7fe8d4003a00 2024-04-06T13:47:38.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.487+0000 7fe8e4ff9700 1 -- 172.21.15.189:0/1720079007 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d0004e00 con 0x7fe8d4003a00 2024-04-06T13:47:38.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.487+0000 7fe8ee8d5700 1 -- 172.21.15.189:0/1720079007 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e80c65b0 con 0x7fe8d4003a00 2024-04-06T13:47:38.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.487+0000 7fe8e4ff9700 1 -- 172.21.15.189:0/1720079007 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0007310 con 0x7fe8d4003a00 2024-04-06T13:47:38.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.487+0000 7fe8ee8d5700 1 -- 172.21.15.189:0/1720079007 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e80c6db0 con 0x7fe8d4003a00 2024-04-06T13:47:38.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.488+0000 7fe8ee8d5700 1 -- 172.21.15.189:0/1720079007 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e8074990 con 0x7fe8d4003a00 2024-04-06T13:47:38.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.488+0000 7fe8e4ff9700 1 -- 172.21.15.189:0/1720079007 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fe8d0019070 con 0x7fe8d4003a00 2024-04-06T13:47:38.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.489+0000 7fe8e4ff9700 1 --2- 172.21.15.189:0/1720079007 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe8c806adb0 0x7fe8c806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.489+0000 7fe8e4ff9700 1 -- 172.21.15.189:0/1720079007 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6965+0+0 (secure 0 0 0) 0x7fe8d008a500 con 0x7fe8d4003a00 2024-04-06T13:47:38.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.489+0000 7fe8e77fe700 1 --2- 172.21.15.189:0/1720079007 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe8c806adb0 0x7fe8c806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.489+0000 7fe8e77fe700 1 --2- 172.21.15.189:0/1720079007 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe8c806adb0 0x7fe8c806d270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8006d90 tx=0x7fe8d8008040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:38.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.493+0000 7fe8e4ff9700 1 -- 172.21.15.189:0/1720079007 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe8d0055170 con 0x7fe8d4003a00 2024-04-06T13:47:38.651 INFO:tasks.ceph.mds.d.smithi192.stderr:2024-04-06T13:47:38.649+0000 7fccfa933700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T13:47:38.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.686+0000 7fe8ee8d5700 1 -- 172.21.15.189:0/1720079007 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe8e806cc20 con 0x7fe8c806adb0 2024-04-06T13:47:38.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.700+0000 7fe8e4ff9700 1 -- 172.21.15.189:0/1720079007 <== mgr.11028 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528975 (secure 0 0 0) 0x7fe8d8011780 con 0x7fe8c806adb0 2024-04-06T13:47:38.707 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:38.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.704+0000 7fe8ee8d5700 1 -- 172.21.15.189:0/1720079007 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe8c806adb0 msgr2=0x7fe8c806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:38.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.704+0000 7fe8ee8d5700 1 --2- 172.21.15.189:0/1720079007 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe8c806adb0 0x7fe8c806d270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8006d90 tx=0x7fe8d8008040 comp rx=0 tx=0).stop 2024-04-06T13:47:38.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.704+0000 7fe8ee8d5700 1 -- 172.21.15.189:0/1720079007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8d4003a00 msgr2=0x7fe8e815dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:38.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.704+0000 7fe8ee8d5700 1 --2- 172.21.15.189:0/1720079007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8d4003a00 0x7fe8e815dd40 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00019d0 tx=0x7fe8d0000fe0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.704+0000 7fe8ee8d5700 1 -- 172.21.15.189:0/1720079007 shutdown_connections 2024-04-06T13:47:38.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.704+0000 7fe8ee8d5700 1 --2- 172.21.15.189:0/1720079007 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe8c806adb0 0x7fe8c806d270 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.704+0000 7fe8ee8d5700 1 --2- 172.21.15.189:0/1720079007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8e81f62c0 0x7fe8e81f6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.704+0000 7fe8ee8d5700 1 --2- 172.21.15.189:0/1720079007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8d4003a00 0x7fe8e815dd40 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.704+0000 7fe8ee8d5700 1 --2- 172.21.15.189:0/1720079007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8e80af090 0x7fe8e80af510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.704+0000 7fe8ee8d5700 1 -- 172.21.15.189:0/1720079007 >> 172.21.15.189:0/1720079007 conn(0x7fe8e80b2db0 msgr2=0x7fe8e815cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:38.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.705+0000 7fe8ee8d5700 1 -- 172.21.15.189:0/1720079007 shutdown_connections 2024-04-06T13:47:38.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.705+0000 7fe8ee8d5700 1 -- 172.21.15.189:0/1720079007 wait complete. 2024-04-06T13:47:38.714 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:47:38.717 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-04-06T13:47:38.412002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596341,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":670,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":427,"ondisk_log_size":427,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":299,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":418004,"kb_used_data":23876,"kb_used_omap":12,"kb_used_meta":393971,"kb_avail":754556716,"statfs":{"total":773094113280,"available":772666077184,"internally_reserved":0,"allocated":24449024,"data_stored":13037975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":403426872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3349,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.237104"},"pg_stats":[{"pgid":"1.0","version":"216'348","reported_seq":1241,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.596051+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:47:34.596051+0000","last_peered":"2024-04-06T13:47:34.596051+0000","last_clean":"2024-04-06T13:47:34.596051+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:47:34.596051+0000","last_undegraded":"2024-04-06T13:47:34.596051+0000","last_fullsized":"2024-04-06T13:47:34.596051+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.588028+0000","last_change":"2024-04-06T13:47:21.714987+0000","last_active":"2024-04-06T13:47:34.588028+0000","last_peered":"2024-04-06T13:47:34.588028+0000","last_clean":"2024-04-06T13:47:34.588028+0000","last_became_active":"2024-04-06T13:47:21.713909+0000","last_became_peered":"2024-04-06T13:47:21.713909+0000","last_unstale":"2024-04-06T13:47:34.588028+0000","last_undegraded":"2024-04-06T13:47:34.588028+0000","last_fullsized":"2024-04-06T13:47:34.588028+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:55:29.107060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.594293+0000","last_change":"2024-04-06T13:47:22.146992+0000","last_active":"2024-04-06T13:47:34.594293+0000","last_peered":"2024-04-06T13:47:34.594293+0000","last_clean":"2024-04-06T13:47:34.594293+0000","last_became_active":"2024-04-06T13:47:22.146728+0000","last_became_peered":"2024-04-06T13:47:22.146728+0000","last_unstale":"2024-04-06T13:47:34.594293+0000","last_undegraded":"2024-04-06T13:47:34.594293+0000","last_fullsized":"2024-04-06T13:47:34.594293+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:12:44.053539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594756+0000","last_change":"2024-04-06T13:47:34.190936+0000","last_active":"2024-04-06T13:47:35.594756+0000","last_peered":"2024-04-06T13:47:35.594756+0000","last_clean":"2024-04-06T13:47:35.594756+0000","last_became_active":"2024-04-06T13:47:34.190577+0000","last_became_peered":"2024-04-06T13:47:34.190577+0000","last_unstale":"2024-04-06T13:47:35.594756+0000","last_undegraded":"2024-04-06T13:47:35.594756+0000","last_fullsized":"2024-04-06T13:47:35.594756+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:26:11.066761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594387+0000","last_change":"2024-04-06T13:47:34.618206+0000","last_active":"2024-04-06T13:47:35.594387+0000","last_peered":"2024-04-06T13:47:35.594387+0000","last_clean":"2024-04-06T13:47:35.594387+0000","last_became_active":"2024-04-06T13:47:34.617974+0000","last_became_peered":"2024-04-06T13:47:34.617974+0000","last_unstale":"2024-04-06T13:47:35.594387+0000","last_undegraded":"2024-04-06T13:47:35.594387+0000","last_fullsized":"2024-04-06T13:47:35.594387+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:00.155859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590403+0000","last_change":"2024-04-06T13:47:21.721753+0000","last_active":"2024-04-06T13:47:34.590403+0000","last_peered":"2024-04-06T13:47:34.590403+0000","last_clean":"2024-04-06T13:47:34.590403+0000","last_became_active":"2024-04-06T13:47:21.721452+0000","last_became_peered":"2024-04-06T13:47:21.721452+0000","last_unstale":"2024-04-06T13:47:34.590403+0000","last_undegraded":"2024-04-06T13:47:34.590403+0000","last_fullsized":"2024-04-06T13:47:34.590403+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:56:43.873348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590945+0000","last_change":"2024-04-06T13:47:22.145029+0000","last_active":"2024-04-06T13:47:34.590945+0000","last_peered":"2024-04-06T13:47:34.590945+0000","last_clean":"2024-04-06T13:47:34.590945+0000","last_became_active":"2024-04-06T13:47:22.144765+0000","last_became_peered":"2024-04-06T13:47:22.144765+0000","last_unstale":"2024-04-06T13:47:34.590945+0000","last_undegraded":"2024-04-06T13:47:34.590945+0000","last_fullsized":"2024-04-06T13:47:34.590945+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:13:00.983141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120579+0000","last_change":"2024-04-06T13:47:34.195377+0000","last_active":"2024-04-06T13:47:36.120579+0000","last_peered":"2024-04-06T13:47:36.120579+0000","last_clean":"2024-04-06T13:47:36.120579+0000","last_became_active":"2024-04-06T13:47:34.195097+0000","last_became_peered":"2024-04-06T13:47:34.195097+0000","last_unstale":"2024-04-06T13:47:36.120579+0000","last_undegraded":"2024-04-06T13:47:36.120579+0000","last_fullsized":"2024-04-06T13:47:36.120579+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:20:15.370970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594819+0000","last_change":"2024-04-06T13:47:34.610261+0000","last_active":"2024-04-06T13:47:35.594819+0000","last_peered":"2024-04-06T13:47:35.594819+0000","last_clean":"2024-04-06T13:47:35.594819+0000","last_became_active":"2024-04-06T13:47:34.607361+0000","last_became_peered":"2024-04-06T13:47:34.607361+0000","last_unstale":"2024-04-06T13:47:35.594819+0000","last_undegraded":"2024-04-06T13:47:35.594819+0000","last_fullsized":"2024-04-06T13:47:35.594819+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:39:35.709825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590171+0000","last_change":"2024-04-06T13:47:21.720234+0000","last_active":"2024-04-06T13:47:34.590171+0000","last_peered":"2024-04-06T13:47:34.590171+0000","last_clean":"2024-04-06T13:47:34.590171+0000","last_became_active":"2024-04-06T13:47:21.719765+0000","last_became_peered":"2024-04-06T13:47:21.719765+0000","last_unstale":"2024-04-06T13:47:34.590171+0000","last_undegraded":"2024-04-06T13:47:34.590171+0000","last_fullsized":"2024-04-06T13:47:34.590171+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:55.001531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.598510+0000","last_change":"2024-04-06T13:47:22.150739+0000","last_active":"2024-04-06T13:47:34.598510+0000","last_peered":"2024-04-06T13:47:34.598510+0000","last_clean":"2024-04-06T13:47:34.598510+0000","last_became_active":"2024-04-06T13:47:22.150305+0000","last_became_peered":"2024-04-06T13:47:22.150305+0000","last_unstale":"2024-04-06T13:47:34.598510+0000","last_undegraded":"2024-04-06T13:47:34.598510+0000","last_fullsized":"2024-04-06T13:47:34.598510+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:18:06.419840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121086+0000","last_change":"2024-04-06T13:47:34.200780+0000","last_active":"2024-04-06T13:47:36.121086+0000","last_peered":"2024-04-06T13:47:36.121086+0000","last_clean":"2024-04-06T13:47:36.121086+0000","last_became_active":"2024-04-06T13:47:34.200474+0000","last_became_peered":"2024-04-06T13:47:34.200474+0000","last_unstale":"2024-04-06T13:47:36.121086+0000","last_undegraded":"2024-04-06T13:47:36.121086+0000","last_fullsized":"2024-04-06T13:47:36.121086+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:10:37.773559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120639+0000","last_change":"2024-04-06T13:47:34.598562+0000","last_active":"2024-04-06T13:47:36.120639+0000","last_peered":"2024-04-06T13:47:36.120639+0000","last_clean":"2024-04-06T13:47:36.120639+0000","last_became_active":"2024-04-06T13:47:34.598306+0000","last_became_peered":"2024-04-06T13:47:34.598306+0000","last_unstale":"2024-04-06T13:47:36.120639+0000","last_undegraded":"2024-04-06T13:47:36.120639+0000","last_fullsized":"2024-04-06T13:47:36.120639+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:55:26.384534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587897+0000","last_change":"2024-04-06T13:47:21.711931+0000","last_active":"2024-04-06T13:47:34.587897+0000","last_peered":"2024-04-06T13:47:34.587897+0000","last_clean":"2024-04-06T13:47:34.587897+0000","last_became_active":"2024-04-06T13:47:21.711592+0000","last_became_peered":"2024-04-06T13:47:21.711592+0000","last_unstale":"2024-04-06T13:47:34.587897+0000","last_undegraded":"2024-04-06T13:47:34.587897+0000","last_fullsized":"2024-04-06T13:47:34.587897+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:38:23.226877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.600111+0000","last_change":"2024-04-06T13:47:22.151110+0000","last_active":"2024-04-06T13:47:34.600111+0000","last_peered":"2024-04-06T13:47:34.600111+0000","last_clean":"2024-04-06T13:47:34.600111+0000","last_became_active":"2024-04-06T13:47:22.150885+0000","last_became_peered":"2024-04-06T13:47:22.150885+0000","last_unstale":"2024-04-06T13:47:34.600111+0000","last_undegraded":"2024-04-06T13:47:34.600111+0000","last_fullsized":"2024-04-06T13:47:34.600111+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:43:52.275558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121334+0000","last_change":"2024-04-06T13:47:34.200209+0000","last_active":"2024-04-06T13:47:36.121334+0000","last_peered":"2024-04-06T13:47:36.121334+0000","last_clean":"2024-04-06T13:47:36.121334+0000","last_became_active":"2024-04-06T13:47:34.198778+0000","last_became_peered":"2024-04-06T13:47:34.198778+0000","last_unstale":"2024-04-06T13:47:36.121334+0000","last_undegraded":"2024-04-06T13:47:36.121334+0000","last_fullsized":"2024-04-06T13:47:36.121334+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:02:31.052025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120410+0000","last_change":"2024-04-06T13:47:34.603150+0000","last_active":"2024-04-06T13:47:36.120410+0000","last_peered":"2024-04-06T13:47:36.120410+0000","last_clean":"2024-04-06T13:47:36.120410+0000","last_became_active":"2024-04-06T13:47:34.602899+0000","last_became_peered":"2024-04-06T13:47:34.602899+0000","last_unstale":"2024-04-06T13:47:36.120410+0000","last_undegraded":"2024-04-06T13:47:36.120410+0000","last_fullsized":"2024-04-06T13:47:36.120410+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:19:18.381473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597883+0000","last_change":"2024-04-06T13:47:21.721538+0000","last_active":"2024-04-06T13:47:34.597883+0000","last_peered":"2024-04-06T13:47:34.597883+0000","last_clean":"2024-04-06T13:47:34.597883+0000","last_became_active":"2024-04-06T13:47:21.721255+0000","last_became_peered":"2024-04-06T13:47:21.721255+0000","last_unstale":"2024-04-06T13:47:34.597883+0000","last_undegraded":"2024-04-06T13:47:34.597883+0000","last_fullsized":"2024-04-06T13:47:34.597883+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:53:48.795224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597888+0000","last_change":"2024-04-06T13:47:22.138725+0000","last_active":"2024-04-06T13:47:34.597888+0000","last_peered":"2024-04-06T13:47:34.597888+0000","last_clean":"2024-04-06T13:47:34.597888+0000","last_became_active":"2024-04-06T13:47:22.131665+0000","last_became_peered":"2024-04-06T13:47:22.131665+0000","last_unstale":"2024-04-06T13:47:34.597888+0000","last_undegraded":"2024-04-06T13:47:34.597888+0000","last_fullsized":"2024-04-06T13:47:34.597888+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:58:16.851577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120645+0000","last_change":"2024-04-06T13:47:34.188669+0000","last_active":"2024-04-06T13:47:36.120645+0000","last_peered":"2024-04-06T13:47:36.120645+0000","last_clean":"2024-04-06T13:47:36.120645+0000","last_became_active":"2024-04-06T13:47:34.188330+0000","last_became_peered":"2024-04-06T13:47:34.188330+0000","last_unstale":"2024-04-06T13:47:36.120645+0000","last_undegraded":"2024-04-06T13:47:36.120645+0000","last_fullsized":"2024-04-06T13:47:36.120645+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:21.894887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.595462+0000","last_change":"2024-04-06T13:47:34.606086+0000","last_active":"2024-04-06T13:47:35.595462+0000","last_peered":"2024-04-06T13:47:35.595462+0000","last_clean":"2024-04-06T13:47:35.595462+0000","last_became_active":"2024-04-06T13:47:34.605599+0000","last_became_peered":"2024-04-06T13:47:34.605599+0000","last_unstale":"2024-04-06T13:47:35.595462+0000","last_undegraded":"2024-04-06T13:47:35.595462+0000","last_fullsized":"2024-04-06T13:47:35.595462+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:51:34.385913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591573+0000","last_change":"2024-04-06T13:47:21.721811+0000","last_active":"2024-04-06T13:47:34.591573+0000","last_peered":"2024-04-06T13:47:34.591573+0000","last_clean":"2024-04-06T13:47:34.591573+0000","last_became_active":"2024-04-06T13:47:21.721528+0000","last_became_peered":"2024-04-06T13:47:21.721528+0000","last_unstale":"2024-04-06T13:47:34.591573+0000","last_undegraded":"2024-04-06T13:47:34.591573+0000","last_fullsized":"2024-04-06T13:47:34.591573+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:54:07.189978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591882+0000","last_change":"2024-04-06T13:47:22.138308+0000","last_active":"2024-04-06T13:47:34.591882+0000","last_peered":"2024-04-06T13:47:34.591882+0000","last_clean":"2024-04-06T13:47:34.591882+0000","last_became_active":"2024-04-06T13:47:22.138038+0000","last_became_peered":"2024-04-06T13:47:22.138038+0000","last_unstale":"2024-04-06T13:47:34.591882+0000","last_undegraded":"2024-04-06T13:47:34.591882+0000","last_fullsized":"2024-04-06T13:47:34.591882+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:23:02.601223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119872+0000","last_change":"2024-04-06T13:47:34.194775+0000","last_active":"2024-04-06T13:47:36.119872+0000","last_peered":"2024-04-06T13:47:36.119872+0000","last_clean":"2024-04-06T13:47:36.119872+0000","last_became_active":"2024-04-06T13:47:34.194481+0000","last_became_peered":"2024-04-06T13:47:34.194481+0000","last_unstale":"2024-04-06T13:47:36.119872+0000","last_undegraded":"2024-04-06T13:47:36.119872+0000","last_fullsized":"2024-04-06T13:47:36.119872+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:55:13.281748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593853+0000","last_change":"2024-04-06T13:47:34.613319+0000","last_active":"2024-04-06T13:47:35.593853+0000","last_peered":"2024-04-06T13:47:35.593853+0000","last_clean":"2024-04-06T13:47:35.593853+0000","last_became_active":"2024-04-06T13:47:34.613055+0000","last_became_peered":"2024-04-06T13:47:34.613055+0000","last_unstale":"2024-04-06T13:47:35.593853+0000","last_undegraded":"2024-04-06T13:47:35.593853+0000","last_fullsized":"2024-04-06T13:47:35.593853+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:04.023963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.596907+0000","last_change":"2024-04-06T13:47:21.711878+0000","last_active":"2024-04-06T13:47:34.596907+0000","last_peered":"2024-04-06T13:47:34.596907+0000","last_clean":"2024-04-06T13:47:34.596907+0000","last_became_active":"2024-04-06T13:47:21.711481+0000","last_became_peered":"2024-04-06T13:47:21.711481+0000","last_unstale":"2024-04-06T13:47:34.596907+0000","last_undegraded":"2024-04-06T13:47:34.596907+0000","last_fullsized":"2024-04-06T13:47:34.596907+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:34:40.754044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587817+0000","last_change":"2024-04-06T13:47:22.136494+0000","last_active":"2024-04-06T13:47:34.587817+0000","last_peered":"2024-04-06T13:47:34.587817+0000","last_clean":"2024-04-06T13:47:34.587817+0000","last_became_active":"2024-04-06T13:47:22.136225+0000","last_became_peered":"2024-04-06T13:47:22.136225+0000","last_unstale":"2024-04-06T13:47:34.587817+0000","last_undegraded":"2024-04-06T13:47:34.587817+0000","last_fullsized":"2024-04-06T13:47:34.587817+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:00:21.961334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120694+0000","last_change":"2024-04-06T13:47:34.202662+0000","last_active":"2024-04-06T13:47:36.120694+0000","last_peered":"2024-04-06T13:47:36.120694+0000","last_clean":"2024-04-06T13:47:36.120694+0000","last_became_active":"2024-04-06T13:47:34.201474+0000","last_became_peered":"2024-04-06T13:47:34.201474+0000","last_unstale":"2024-04-06T13:47:36.120694+0000","last_undegraded":"2024-04-06T13:47:36.120694+0000","last_fullsized":"2024-04-06T13:47:36.120694+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:42:05.118957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593666+0000","last_change":"2024-04-06T13:47:34.617304+0000","last_active":"2024-04-06T13:47:35.593666+0000","last_peered":"2024-04-06T13:47:35.593666+0000","last_clean":"2024-04-06T13:47:35.593666+0000","last_became_active":"2024-04-06T13:47:34.617119+0000","last_became_peered":"2024-04-06T13:47:34.617119+0000","last_unstale":"2024-04-06T13:47:35.593666+0000","last_undegraded":"2024-04-06T13:47:35.593666+0000","last_fullsized":"2024-04-06T13:47:35.593666+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:52:07.725370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.6","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597488+0000","last_change":"2024-04-06T13:47:21.725237+0000","last_active":"2024-04-06T13:47:34.597488+0000","last_peered":"2024-04-06T13:47:34.597488+0000","last_clean":"2024-04-06T13:47:34.597488+0000","last_became_active":"2024-04-06T13:47:21.724977+0000","last_became_peered":"2024-04-06T13:47:21.724977+0000","last_unstale":"2024-04-06T13:47:34.597488+0000","last_undegraded":"2024-04-06T13:47:34.597488+0000","last_fullsized":"2024-04-06T13:47:34.597488+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:13:14.840162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590344+0000","last_change":"2024-04-06T13:47:22.141257+0000","last_active":"2024-04-06T13:47:34.590344+0000","last_peered":"2024-04-06T13:47:34.590344+0000","last_clean":"2024-04-06T13:47:34.590344+0000","last_became_active":"2024-04-06T13:47:22.141005+0000","last_became_peered":"2024-04-06T13:47:22.141005+0000","last_unstale":"2024-04-06T13:47:34.590344+0000","last_undegraded":"2024-04-06T13:47:34.590344+0000","last_fullsized":"2024-04-06T13:47:34.590344+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:15.316599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121277+0000","last_change":"2024-04-06T13:47:34.200644+0000","last_active":"2024-04-06T13:47:36.121277+0000","last_peered":"2024-04-06T13:47:36.121277+0000","last_clean":"2024-04-06T13:47:36.121277+0000","last_became_active":"2024-04-06T13:47:34.200379+0000","last_became_peered":"2024-04-06T13:47:34.200379+0000","last_unstale":"2024-04-06T13:47:36.121277+0000","last_undegraded":"2024-04-06T13:47:36.121277+0000","last_fullsized":"2024-04-06T13:47:36.121277+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:29:46.861936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120258+0000","last_change":"2024-04-06T13:47:34.613965+0000","last_active":"2024-04-06T13:47:36.120258+0000","last_peered":"2024-04-06T13:47:36.120258+0000","last_clean":"2024-04-06T13:47:36.120258+0000","last_became_active":"2024-04-06T13:47:34.613692+0000","last_became_peered":"2024-04-06T13:47:34.613692+0000","last_unstale":"2024-04-06T13:47:36.120258+0000","last_undegraded":"2024-04-06T13:47:36.120258+0000","last_fullsized":"2024-04-06T13:47:36.120258+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:58:55.977176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.588174+0000","last_change":"2024-04-06T13:47:21.719079+0000","last_active":"2024-04-06T13:47:34.588174+0000","last_peered":"2024-04-06T13:47:34.588174+0000","last_clean":"2024-04-06T13:47:34.588174+0000","last_became_active":"2024-04-06T13:47:21.718773+0000","last_became_peered":"2024-04-06T13:47:21.718773+0000","last_unstale":"2024-04-06T13:47:34.588174+0000","last_undegraded":"2024-04-06T13:47:34.588174+0000","last_fullsized":"2024-04-06T13:47:34.588174+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:45:09.803960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597924+0000","last_change":"2024-04-06T13:47:22.147695+0000","last_active":"2024-04-06T13:47:34.597924+0000","last_peered":"2024-04-06T13:47:34.597924+0000","last_clean":"2024-04-06T13:47:34.597924+0000","last_became_active":"2024-04-06T13:47:22.147127+0000","last_became_peered":"2024-04-06T13:47:22.147127+0000","last_unstale":"2024-04-06T13:47:34.597924+0000","last_undegraded":"2024-04-06T13:47:34.597924+0000","last_fullsized":"2024-04-06T13:47:34.597924+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:03:24.991063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594311+0000","last_change":"2024-04-06T13:47:34.185950+0000","last_active":"2024-04-06T13:47:35.594311+0000","last_peered":"2024-04-06T13:47:35.594311+0000","last_clean":"2024-04-06T13:47:35.594311+0000","last_became_active":"2024-04-06T13:47:34.185647+0000","last_became_peered":"2024-04-06T13:47:34.185647+0000","last_unstale":"2024-04-06T13:47:35.594311+0000","last_undegraded":"2024-04-06T13:47:35.594311+0000","last_fullsized":"2024-04-06T13:47:35.594311+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:16:40.354721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119543+0000","last_change":"2024-04-06T13:47:34.612456+0000","last_active":"2024-04-06T13:47:36.119543+0000","last_peered":"2024-04-06T13:47:36.119543+0000","last_clean":"2024-04-06T13:47:36.119543+0000","last_became_active":"2024-04-06T13:47:34.612181+0000","last_became_peered":"2024-04-06T13:47:34.612181+0000","last_unstale":"2024-04-06T13:47:36.119543+0000","last_undegraded":"2024-04-06T13:47:36.119543+0000","last_fullsized":"2024-04-06T13:47:36.119543+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:27.792432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590099+0000","last_change":"2024-04-06T13:47:21.718440+0000","last_active":"2024-04-06T13:47:34.590099+0000","last_peered":"2024-04-06T13:47:34.590099+0000","last_clean":"2024-04-06T13:47:34.590099+0000","last_became_active":"2024-04-06T13:47:21.718175+0000","last_became_peered":"2024-04-06T13:47:21.718175+0000","last_unstale":"2024-04-06T13:47:34.590099+0000","last_undegraded":"2024-04-06T13:47:34.590099+0000","last_fullsized":"2024-04-06T13:47:34.590099+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:34:40.754044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.593523+0000","last_change":"2024-04-06T13:47:22.141028+0000","last_active":"2024-04-06T13:47:34.593523+0000","last_peered":"2024-04-06T13:47:34.593523+0000","last_clean":"2024-04-06T13:47:34.593523+0000","last_became_active":"2024-04-06T13:47:22.140749+0000","last_became_peered":"2024-04-06T13:47:22.140749+0000","last_unstale":"2024-04-06T13:47:34.593523+0000","last_undegraded":"2024-04-06T13:47:34.593523+0000","last_fullsized":"2024-04-06T13:47:34.593523+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:06:22.598094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120022+0000","last_change":"2024-04-06T13:47:34.194754+0000","last_active":"2024-04-06T13:47:36.120022+0000","last_peered":"2024-04-06T13:47:36.120022+0000","last_clean":"2024-04-06T13:47:36.120022+0000","last_became_active":"2024-04-06T13:47:34.194428+0000","last_became_peered":"2024-04-06T13:47:34.194428+0000","last_unstale":"2024-04-06T13:47:36.120022+0000","last_undegraded":"2024-04-06T13:47:36.120022+0000","last_fullsized":"2024-04-06T13:47:36.120022+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:21:48.835480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594687+0000","last_change":"2024-04-06T13:47:34.615494+0000","last_active":"2024-04-06T13:47:35.594687+0000","last_peered":"2024-04-06T13:47:35.594687+0000","last_clean":"2024-04-06T13:47:35.594687+0000","last_became_active":"2024-04-06T13:47:34.615234+0000","last_became_peered":"2024-04-06T13:47:34.615234+0000","last_unstale":"2024-04-06T13:47:35.594687+0000","last_undegraded":"2024-04-06T13:47:35.594687+0000","last_fullsized":"2024-04-06T13:47:35.594687+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:37.873504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.594288+0000","last_change":"2024-04-06T13:47:21.720187+0000","last_active":"2024-04-06T13:47:34.594288+0000","last_peered":"2024-04-06T13:47:34.594288+0000","last_clean":"2024-04-06T13:47:34.594288+0000","last_became_active":"2024-04-06T13:47:21.719896+0000","last_became_peered":"2024-04-06T13:47:21.719896+0000","last_unstale":"2024-04-06T13:47:34.594288+0000","last_undegraded":"2024-04-06T13:47:34.594288+0000","last_fullsized":"2024-04-06T13:47:34.594288+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:11:25.850855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587979+0000","last_change":"2024-04-06T13:47:22.138293+0000","last_active":"2024-04-06T13:47:34.587979+0000","last_peered":"2024-04-06T13:47:34.587979+0000","last_clean":"2024-04-06T13:47:34.587979+0000","last_became_active":"2024-04-06T13:47:22.138051+0000","last_became_peered":"2024-04-06T13:47:22.138051+0000","last_unstale":"2024-04-06T13:47:34.587979+0000","last_undegraded":"2024-04-06T13:47:34.587979+0000","last_fullsized":"2024-04-06T13:47:34.587979+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:52:29.470996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594443+0000","last_change":"2024-04-06T13:47:34.199667+0000","last_active":"2024-04-06T13:47:35.594443+0000","last_peered":"2024-04-06T13:47:35.594443+0000","last_clean":"2024-04-06T13:47:35.594443+0000","last_became_active":"2024-04-06T13:47:34.199405+0000","last_became_peered":"2024-04-06T13:47:34.199405+0000","last_unstale":"2024-04-06T13:47:35.594443+0000","last_undegraded":"2024-04-06T13:47:35.594443+0000","last_fullsized":"2024-04-06T13:47:35.594443+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:44:23.945875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120632+0000","last_change":"2024-04-06T13:47:34.613888+0000","last_active":"2024-04-06T13:47:36.120632+0000","last_peered":"2024-04-06T13:47:36.120632+0000","last_clean":"2024-04-06T13:47:36.120632+0000","last_became_active":"2024-04-06T13:47:34.613634+0000","last_became_peered":"2024-04-06T13:47:34.613634+0000","last_unstale":"2024-04-06T13:47:36.120632+0000","last_undegraded":"2024-04-06T13:47:36.120632+0000","last_fullsized":"2024-04-06T13:47:36.120632+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:30:31.417241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.595033+0000","last_change":"2024-04-06T13:47:21.712418+0000","last_active":"2024-04-06T13:47:34.595033+0000","last_peered":"2024-04-06T13:47:34.595033+0000","last_clean":"2024-04-06T13:47:34.595033+0000","last_became_active":"2024-04-06T13:47:21.712149+0000","last_became_peered":"2024-04-06T13:47:21.712149+0000","last_unstale":"2024-04-06T13:47:34.595033+0000","last_undegraded":"2024-04-06T13:47:34.595033+0000","last_fullsized":"2024-04-06T13:47:34.595033+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:10:24.669252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.599926+0000","last_change":"2024-04-06T13:47:22.141810+0000","last_active":"2024-04-06T13:47:34.599926+0000","last_peered":"2024-04-06T13:47:34.599926+0000","last_clean":"2024-04-06T13:47:34.599926+0000","last_became_active":"2024-04-06T13:47:22.141394+0000","last_became_peered":"2024-04-06T13:47:22.141394+0000","last_unstale":"2024-04-06T13:47:34.599926+0000","last_undegraded":"2024-04-06T13:47:34.599926+0000","last_fullsized":"2024-04-06T13:47:34.599926+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:58:43.501343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121016+0000","last_change":"2024-04-06T13:47:34.198480+0000","last_active":"2024-04-06T13:47:36.121016+0000","last_peered":"2024-04-06T13:47:36.121016+0000","last_clean":"2024-04-06T13:47:36.121016+0000","last_became_active":"2024-04-06T13:47:34.197845+0000","last_became_peered":"2024-04-06T13:47:34.197845+0000","last_unstale":"2024-04-06T13:47:36.121016+0000","last_undegraded":"2024-04-06T13:47:36.121016+0000","last_fullsized":"2024-04-06T13:47:36.121016+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:20:03.197179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121182+0000","last_change":"2024-04-06T13:47:34.614252+0000","last_active":"2024-04-06T13:47:36.121182+0000","last_peered":"2024-04-06T13:47:36.121182+0000","last_clean":"2024-04-06T13:47:36.121182+0000","last_became_active":"2024-04-06T13:47:34.613984+0000","last_became_peered":"2024-04-06T13:47:34.613984+0000","last_unstale":"2024-04-06T13:47:36.121182+0000","last_undegraded":"2024-04-06T13:47:36.121182+0000","last_fullsized":"2024-04-06T13:47:36.121182+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:49:31.353686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.b","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590538+0000","last_change":"2024-04-06T13:47:21.709408+0000","last_active":"2024-04-06T13:47:34.590538+0000","last_peered":"2024-04-06T13:47:34.590538+0000","last_clean":"2024-04-06T13:47:34.590538+0000","last_became_active":"2024-04-06T13:47:21.708906+0000","last_became_peered":"2024-04-06T13:47:21.708906+0000","last_unstale":"2024-04-06T13:47:34.590538+0000","last_undegraded":"2024-04-06T13:47:34.590538+0000","last_fullsized":"2024-04-06T13:47:34.590538+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:09.996520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.595599+0000","last_change":"2024-04-06T13:47:22.143638+0000","last_active":"2024-04-06T13:47:34.595599+0000","last_peered":"2024-04-06T13:47:34.595599+0000","last_clean":"2024-04-06T13:47:34.595599+0000","last_became_active":"2024-04-06T13:47:22.143386+0000","last_became_peered":"2024-04-06T13:47:22.143386+0000","last_unstale":"2024-04-06T13:47:34.595599+0000","last_undegraded":"2024-04-06T13:47:34.595599+0000","last_fullsized":"2024-04-06T13:47:34.595599+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:16:28.885120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120083+0000","last_change":"2024-04-06T13:47:34.195822+0000","last_active":"2024-04-06T13:47:36.120083+0000","last_peered":"2024-04-06T13:47:36.120083+0000","last_clean":"2024-04-06T13:47:36.120083+0000","last_became_active":"2024-04-06T13:47:34.195539+0000","last_became_peered":"2024-04-06T13:47:34.195539+0000","last_unstale":"2024-04-06T13:47:36.120083+0000","last_undegraded":"2024-04-06T13:47:36.120083+0000","last_fullsized":"2024-04-06T13:47:36.120083+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:52.468926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594485+0000","last_change":"2024-04-06T13:47:34.599198+0000","last_active":"2024-04-06T13:47:35.594485+0000","last_peered":"2024-04-06T13:47:35.594485+0000","last_clean":"2024-04-06T13:47:35.594485+0000","last_became_active":"2024-04-06T13:47:34.598897+0000","last_became_peered":"2024-04-06T13:47:34.598897+0000","last_unstale":"2024-04-06T13:47:35.594485+0000","last_undegraded":"2024-04-06T13:47:35.594485+0000","last_fullsized":"2024-04-06T13:47:35.594485+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:59:50.625262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591209+0000","last_change":"2024-04-06T13:47:21.721929+0000","last_active":"2024-04-06T13:47:34.591209+0000","last_peered":"2024-04-06T13:47:34.591209+0000","last_clean":"2024-04-06T13:47:34.591209+0000","last_became_active":"2024-04-06T13:47:21.721667+0000","last_became_peered":"2024-04-06T13:47:21.721667+0000","last_unstale":"2024-04-06T13:47:34.591209+0000","last_undegraded":"2024-04-06T13:47:34.591209+0000","last_fullsized":"2024-04-06T13:47:34.591209+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:49:15.850828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.602579+0000","last_change":"2024-04-06T13:47:22.139250+0000","last_active":"2024-04-06T13:47:34.602579+0000","last_peered":"2024-04-06T13:47:34.602579+0000","last_clean":"2024-04-06T13:47:34.602579+0000","last_became_active":"2024-04-06T13:47:22.138989+0000","last_became_peered":"2024-04-06T13:47:22.138989+0000","last_unstale":"2024-04-06T13:47:34.602579+0000","last_undegraded":"2024-04-06T13:47:34.602579+0000","last_fullsized":"2024-04-06T13:47:34.602579+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:48:14.103260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120427+0000","last_change":"2024-04-06T13:47:34.187384+0000","last_active":"2024-04-06T13:47:36.120427+0000","last_peered":"2024-04-06T13:47:36.120427+0000","last_clean":"2024-04-06T13:47:36.120427+0000","last_became_active":"2024-04-06T13:47:34.186741+0000","last_became_peered":"2024-04-06T13:47:34.186741+0000","last_unstale":"2024-04-06T13:47:36.120427+0000","last_undegraded":"2024-04-06T13:47:36.120427+0000","last_fullsized":"2024-04-06T13:47:36.120427+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:00:33.430936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594502+0000","last_change":"2024-04-06T13:47:34.604474+0000","last_active":"2024-04-06T13:47:35.594502+0000","last_peered":"2024-04-06T13:47:35.594502+0000","last_clean":"2024-04-06T13:47:35.594502+0000","last_became_active":"2024-04-06T13:47:34.603745+0000","last_became_peered":"2024-04-06T13:47:34.603745+0000","last_unstale":"2024-04-06T13:47:35.594502+0000","last_undegraded":"2024-04-06T13:47:35.594502+0000","last_fullsized":"2024-04-06T13:47:35.594502+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:38:37.608413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.d","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597603+0000","last_change":"2024-04-06T13:47:21.714087+0000","last_active":"2024-04-06T13:47:34.597603+0000","last_peered":"2024-04-06T13:47:34.597603+0000","last_clean":"2024-04-06T13:47:34.597603+0000","last_became_active":"2024-04-06T13:47:21.713790+0000","last_became_peered":"2024-04-06T13:47:21.713790+0000","last_unstale":"2024-04-06T13:47:34.597603+0000","last_undegraded":"2024-04-06T13:47:34.597603+0000","last_fullsized":"2024-04-06T13:47:34.597603+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:52:28.469203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.592773+0000","last_change":"2024-04-06T13:47:22.143187+0000","last_active":"2024-04-06T13:47:34.592773+0000","last_peered":"2024-04-06T13:47:34.592773+0000","last_clean":"2024-04-06T13:47:34.592773+0000","last_became_active":"2024-04-06T13:47:22.142878+0000","last_became_peered":"2024-04-06T13:47:22.142878+0000","last_unstale":"2024-04-06T13:47:34.592773+0000","last_undegraded":"2024-04-06T13:47:34.592773+0000","last_fullsized":"2024-04-06T13:47:34.592773+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:27:49.346320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120949+0000","last_change":"2024-04-06T13:47:34.195195+0000","last_active":"2024-04-06T13:47:36.120949+0000","last_peered":"2024-04-06T13:47:36.120949+0000","last_clean":"2024-04-06T13:47:36.120949+0000","last_became_active":"2024-04-06T13:47:34.194951+0000","last_became_peered":"2024-04-06T13:47:34.194951+0000","last_unstale":"2024-04-06T13:47:36.120949+0000","last_undegraded":"2024-04-06T13:47:36.120949+0000","last_fullsized":"2024-04-06T13:47:36.120949+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:12:55.523141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119895+0000","last_change":"2024-04-06T13:47:34.616419+0000","last_active":"2024-04-06T13:47:36.119895+0000","last_peered":"2024-04-06T13:47:36.119895+0000","last_clean":"2024-04-06T13:47:36.119895+0000","last_became_active":"2024-04-06T13:47:34.616079+0000","last_became_peered":"2024-04-06T13:47:34.616079+0000","last_unstale":"2024-04-06T13:47:36.119895+0000","last_undegraded":"2024-04-06T13:47:36.119895+0000","last_fullsized":"2024-04-06T13:47:36.119895+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:29:51.530310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.e","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.600642+0000","last_change":"2024-04-06T13:47:21.723380+0000","last_active":"2024-04-06T13:47:34.600642+0000","last_peered":"2024-04-06T13:47:34.600642+0000","last_clean":"2024-04-06T13:47:34.600642+0000","last_became_active":"2024-04-06T13:47:21.723089+0000","last_became_peered":"2024-04-06T13:47:21.723089+0000","last_unstale":"2024-04-06T13:47:34.600642+0000","last_undegraded":"2024-04-06T13:47:34.600642+0000","last_fullsized":"2024-04-06T13:47:34.600642+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:11:25.850855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591674+0000","last_change":"2024-04-06T13:47:22.135011+0000","last_active":"2024-04-06T13:47:34.591674+0000","last_peered":"2024-04-06T13:47:34.591674+0000","last_clean":"2024-04-06T13:47:34.591674+0000","last_became_active":"2024-04-06T13:47:22.134193+0000","last_became_peered":"2024-04-06T13:47:22.134193+0000","last_unstale":"2024-04-06T13:47:34.591674+0000","last_undegraded":"2024-04-06T13:47:34.591674+0000","last_fullsized":"2024-04-06T13:47:34.591674+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:15:14.777409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120008+0000","last_change":"2024-04-06T13:47:34.194432+0000","last_active":"2024-04-06T13:47:36.120008+0000","last_peered":"2024-04-06T13:47:36.120008+0000","last_clean":"2024-04-06T13:47:36.120008+0000","last_became_active":"2024-04-06T13:47:34.194153+0000","last_became_peered":"2024-04-06T13:47:34.194153+0000","last_unstale":"2024-04-06T13:47:36.120008+0000","last_undegraded":"2024-04-06T13:47:36.120008+0000","last_fullsized":"2024-04-06T13:47:36.120008+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:28:00.815922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594023+0000","last_change":"2024-04-06T13:47:34.618214+0000","last_active":"2024-04-06T13:47:35.594023+0000","last_peered":"2024-04-06T13:47:35.594023+0000","last_clean":"2024-04-06T13:47:35.594023+0000","last_became_active":"2024-04-06T13:47:34.617973+0000","last_became_peered":"2024-04-06T13:47:34.617973+0000","last_unstale":"2024-04-06T13:47:35.594023+0000","last_undegraded":"2024-04-06T13:47:35.594023+0000","last_fullsized":"2024-04-06T13:47:35.594023+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:01:34.086529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.f","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.601303+0000","last_change":"2024-04-06T13:47:21.722333+0000","last_active":"2024-04-06T13:47:34.601303+0000","last_peered":"2024-04-06T13:47:34.601303+0000","last_clean":"2024-04-06T13:47:34.601303+0000","last_became_active":"2024-04-06T13:47:21.722021+0000","last_became_peered":"2024-04-06T13:47:21.722021+0000","last_unstale":"2024-04-06T13:47:34.601303+0000","last_undegraded":"2024-04-06T13:47:34.601303+0000","last_fullsized":"2024-04-06T13:47:34.601303+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:54:05.540582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597829+0000","last_change":"2024-04-06T13:47:22.147277+0000","last_active":"2024-04-06T13:47:34.597829+0000","last_peered":"2024-04-06T13:47:34.597829+0000","last_clean":"2024-04-06T13:47:34.597829+0000","last_became_active":"2024-04-06T13:47:22.146965+0000","last_became_peered":"2024-04-06T13:47:22.146965+0000","last_unstale":"2024-04-06T13:47:34.597829+0000","last_undegraded":"2024-04-06T13:47:34.597829+0000","last_fullsized":"2024-04-06T13:47:34.597829+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:30:03.035386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120127+0000","last_change":"2024-04-06T13:47:34.195388+0000","last_active":"2024-04-06T13:47:36.120127+0000","last_peered":"2024-04-06T13:47:36.120127+0000","last_clean":"2024-04-06T13:47:36.120127+0000","last_became_active":"2024-04-06T13:47:34.195111+0000","last_became_peered":"2024-04-06T13:47:34.195111+0000","last_unstale":"2024-04-06T13:47:36.120127+0000","last_undegraded":"2024-04-06T13:47:36.120127+0000","last_fullsized":"2024-04-06T13:47:36.120127+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:10:37.140647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119882+0000","last_change":"2024-04-06T13:47:34.613796+0000","last_active":"2024-04-06T13:47:36.119882+0000","last_peered":"2024-04-06T13:47:36.119882+0000","last_clean":"2024-04-06T13:47:36.119882+0000","last_became_active":"2024-04-06T13:47:34.613561+0000","last_became_peered":"2024-04-06T13:47:34.613561+0000","last_unstale":"2024-04-06T13:47:36.119882+0000","last_undegraded":"2024-04-06T13:47:36.119882+0000","last_fullsized":"2024-04-06T13:47:36.119882+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:32:08.951962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.10","version":"226'8","reported_seq":23,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590323+0000","last_change":"2024-04-06T13:47:21.721909+0000","last_active":"2024-04-06T13:47:34.590323+0000","last_peered":"2024-04-06T13:47:34.590323+0000","last_clean":"2024-04-06T13:47:34.590323+0000","last_became_active":"2024-04-06T13:47:21.721654+0000","last_became_peered":"2024-04-06T13:47:21.721654+0000","last_unstale":"2024-04-06T13:47:34.590323+0000","last_undegraded":"2024-04-06T13:47:34.590323+0000","last_fullsized":"2024-04-06T13:47:34.590323+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:06:40.698606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.599341+0000","last_change":"2024-04-06T13:47:22.137131+0000","last_active":"2024-04-06T13:47:34.599341+0000","last_peered":"2024-04-06T13:47:34.599341+0000","last_clean":"2024-04-06T13:47:34.599341+0000","last_became_active":"2024-04-06T13:47:22.136845+0000","last_became_peered":"2024-04-06T13:47:22.136845+0000","last_unstale":"2024-04-06T13:47:34.599341+0000","last_undegraded":"2024-04-06T13:47:34.599341+0000","last_fullsized":"2024-04-06T13:47:34.599341+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:58:56.639338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121006+0000","last_change":"2024-04-06T13:47:34.199738+0000","last_active":"2024-04-06T13:47:36.121006+0000","last_peered":"2024-04-06T13:47:36.121006+0000","last_clean":"2024-04-06T13:47:36.121006+0000","last_became_active":"2024-04-06T13:47:34.199475+0000","last_became_peered":"2024-04-06T13:47:34.199475+0000","last_unstale":"2024-04-06T13:47:36.121006+0000","last_undegraded":"2024-04-06T13:47:36.121006+0000","last_fullsized":"2024-04-06T13:47:36.121006+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:16:41.274875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120418+0000","last_change":"2024-04-06T13:47:34.603287+0000","last_active":"2024-04-06T13:47:36.120418+0000","last_peered":"2024-04-06T13:47:36.120418+0000","last_clean":"2024-04-06T13:47:36.120418+0000","last_became_active":"2024-04-06T13:47:34.603009+0000","last_became_peered":"2024-04-06T13:47:34.603009+0000","last_unstale":"2024-04-06T13:47:36.120418+0000","last_undegraded":"2024-04-06T13:47:36.120418+0000","last_fullsized":"2024-04-06T13:47:36.120418+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:33:09.147411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587468+0000","last_change":"2024-04-06T13:47:21.722302+0000","last_active":"2024-04-06T13:47:34.587468+0000","last_peered":"2024-04-06T13:47:34.587468+0000","last_clean":"2024-04-06T13:47:34.587468+0000","last_became_active":"2024-04-06T13:47:21.721839+0000","last_became_peered":"2024-04-06T13:47:21.721839+0000","last_unstale":"2024-04-06T13:47:34.587468+0000","last_undegraded":"2024-04-06T13:47:34.587468+0000","last_fullsized":"2024-04-06T13:47:34.587468+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:27:30.927130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587825+0000","last_change":"2024-04-06T13:47:22.140481+0000","last_active":"2024-04-06T13:47:34.587825+0000","last_peered":"2024-04-06T13:47:34.587825+0000","last_clean":"2024-04-06T13:47:34.587825+0000","last_became_active":"2024-04-06T13:47:22.140216+0000","last_became_peered":"2024-04-06T13:47:22.140216+0000","last_unstale":"2024-04-06T13:47:34.587825+0000","last_undegraded":"2024-04-06T13:47:34.587825+0000","last_fullsized":"2024-04-06T13:47:34.587825+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:10.425285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.595864+0000","last_change":"2024-04-06T13:47:34.187197+0000","last_active":"2024-04-06T13:47:35.595864+0000","last_peered":"2024-04-06T13:47:35.595864+0000","last_clean":"2024-04-06T13:47:35.595864+0000","last_became_active":"2024-04-06T13:47:34.186401+0000","last_became_peered":"2024-04-06T13:47:34.186401+0000","last_unstale":"2024-04-06T13:47:35.595864+0000","last_undegraded":"2024-04-06T13:47:35.595864+0000","last_fullsized":"2024-04-06T13:47:35.595864+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:36:37.561866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120917+0000","last_change":"2024-04-06T13:47:34.611586+0000","last_active":"2024-04-06T13:47:36.120917+0000","last_peered":"2024-04-06T13:47:36.120917+0000","last_clean":"2024-04-06T13:47:36.120917+0000","last_became_active":"2024-04-06T13:47:34.611310+0000","last_became_peered":"2024-04-06T13:47:34.611310+0000","last_unstale":"2024-04-06T13:47:36.120917+0000","last_undegraded":"2024-04-06T13:47:36.120917+0000","last_fullsized":"2024-04-06T13:47:36.120917+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:24:11.679817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591343+0000","last_change":"2024-04-06T13:47:21.709366+0000","last_active":"2024-04-06T13:47:34.591343+0000","last_peered":"2024-04-06T13:47:34.591343+0000","last_clean":"2024-04-06T13:47:34.591343+0000","last_became_active":"2024-04-06T13:47:21.709118+0000","last_became_peered":"2024-04-06T13:47:21.709118+0000","last_unstale":"2024-04-06T13:47:34.591343+0000","last_undegraded":"2024-04-06T13:47:34.591343+0000","last_fullsized":"2024-04-06T13:47:34.591343+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:15:13.775616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591825+0000","last_change":"2024-04-06T13:47:22.137019+0000","last_active":"2024-04-06T13:47:34.591825+0000","last_peered":"2024-04-06T13:47:34.591825+0000","last_clean":"2024-04-06T13:47:34.591825+0000","last_became_active":"2024-04-06T13:47:22.136423+0000","last_became_peered":"2024-04-06T13:47:22.136423+0000","last_unstale":"2024-04-06T13:47:34.591825+0000","last_undegraded":"2024-04-06T13:47:34.591825+0000","last_fullsized":"2024-04-06T13:47:34.591825+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:32:52.203373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"230'6","reported_seq":17,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.595298+0000","last_change":"2024-04-06T13:47:34.197594+0000","last_active":"2024-04-06T13:47:35.595298+0000","last_peered":"2024-04-06T13:47:35.595298+0000","last_clean":"2024-04-06T13:47:35.595298+0000","last_became_active":"2024-04-06T13:47:34.197333+0000","last_became_peered":"2024-04-06T13:47:34.197333+0000","last_unstale":"2024-04-06T13:47:35.595298+0000","last_undegraded":"2024-04-06T13:47:35.595298+0000","last_fullsized":"2024-04-06T13:47:35.595298+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:18:17.889442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121415+0000","last_change":"2024-04-06T13:47:34.611400+0000","last_active":"2024-04-06T13:47:36.121415+0000","last_peered":"2024-04-06T13:47:36.121415+0000","last_clean":"2024-04-06T13:47:36.121415+0000","last_became_active":"2024-04-06T13:47:34.611073+0000","last_became_peered":"2024-04-06T13:47:34.611073+0000","last_unstale":"2024-04-06T13:47:36.121415+0000","last_undegraded":"2024-04-06T13:47:36.121415+0000","last_fullsized":"2024-04-06T13:47:36.121415+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:59:09.115171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.595424+0000","last_change":"2024-04-06T13:47:21.719461+0000","last_active":"2024-04-06T13:47:34.595424+0000","last_peered":"2024-04-06T13:47:34.595424+0000","last_clean":"2024-04-06T13:47:34.595424+0000","last_became_active":"2024-04-06T13:47:21.719215+0000","last_became_peered":"2024-04-06T13:47:21.719215+0000","last_unstale":"2024-04-06T13:47:34.595424+0000","last_undegraded":"2024-04-06T13:47:34.595424+0000","last_fullsized":"2024-04-06T13:47:34.595424+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:00:20.959541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.595429+0000","last_change":"2024-04-06T13:47:22.143097+0000","last_active":"2024-04-06T13:47:34.595429+0000","last_peered":"2024-04-06T13:47:34.595429+0000","last_clean":"2024-04-06T13:47:34.595429+0000","last_became_active":"2024-04-06T13:47:22.141937+0000","last_became_peered":"2024-04-06T13:47:22.141937+0000","last_unstale":"2024-04-06T13:47:34.595429+0000","last_undegraded":"2024-04-06T13:47:34.595429+0000","last_fullsized":"2024-04-06T13:47:34.595429+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:36:26.092264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"230'3","reported_seq":14,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.595391+0000","last_change":"2024-04-06T13:47:34.194044+0000","last_active":"2024-04-06T13:47:35.595391+0000","last_peered":"2024-04-06T13:47:35.595391+0000","last_clean":"2024-04-06T13:47:35.595391+0000","last_became_active":"2024-04-06T13:47:34.193760+0000","last_became_peered":"2024-04-06T13:47:34.193760+0000","last_unstale":"2024-04-06T13:47:35.595391+0000","last_undegraded":"2024-04-06T13:47:35.595391+0000","last_fullsized":"2024-04-06T13:47:35.595391+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:06:21.299461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594553+0000","last_change":"2024-04-06T13:47:34.602134+0000","last_active":"2024-04-06T13:47:35.594553+0000","last_peered":"2024-04-06T13:47:35.594553+0000","last_clean":"2024-04-06T13:47:35.594553+0000","last_became_active":"2024-04-06T13:47:34.601889+0000","last_became_peered":"2024-04-06T13:47:34.601889+0000","last_unstale":"2024-04-06T13:47:35.594553+0000","last_undegraded":"2024-04-06T13:47:35.594553+0000","last_fullsized":"2024-04-06T13:47:35.594553+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:53:41.874072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"227'8","reported_seq":23,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.600457+0000","last_change":"2024-04-06T13:47:21.722629+0000","last_active":"2024-04-06T13:47:34.600457+0000","last_peered":"2024-04-06T13:47:34.600457+0000","last_clean":"2024-04-06T13:47:34.600457+0000","last_became_active":"2024-04-06T13:47:21.722378+0000","last_became_peered":"2024-04-06T13:47:21.722378+0000","last_unstale":"2024-04-06T13:47:34.600457+0000","last_undegraded":"2024-04-06T13:47:34.600457+0000","last_fullsized":"2024-04-06T13:47:34.600457+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:43:36.814977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2813,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590621+0000","last_change":"2024-04-06T13:47:22.144290+0000","last_active":"2024-04-06T13:47:34.590621+0000","last_peered":"2024-04-06T13:47:34.590621+0000","last_clean":"2024-04-06T13:47:34.590621+0000","last_became_active":"2024-04-06T13:47:22.144023+0000","last_became_peered":"2024-04-06T13:47:22.144023+0000","last_unstale":"2024-04-06T13:47:34.590621+0000","last_undegraded":"2024-04-06T13:47:34.590621+0000","last_fullsized":"2024-04-06T13:47:34.590621+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:52:25.950818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119908+0000","last_change":"2024-04-06T13:47:34.190335+0000","last_active":"2024-04-06T13:47:36.119908+0000","last_peered":"2024-04-06T13:47:36.119908+0000","last_clean":"2024-04-06T13:47:36.119908+0000","last_became_active":"2024-04-06T13:47:34.190093+0000","last_became_peered":"2024-04-06T13:47:34.190093+0000","last_unstale":"2024-04-06T13:47:36.119908+0000","last_undegraded":"2024-04-06T13:47:36.119908+0000","last_fullsized":"2024-04-06T13:47:36.119908+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:10:33.916859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121064+0000","last_change":"2024-04-06T13:47:34.605328+0000","last_active":"2024-04-06T13:47:36.121064+0000","last_peered":"2024-04-06T13:47:36.121064+0000","last_clean":"2024-04-06T13:47:36.121064+0000","last_became_active":"2024-04-06T13:47:34.604770+0000","last_became_peered":"2024-04-06T13:47:34.604770+0000","last_unstale":"2024-04-06T13:47:36.121064+0000","last_undegraded":"2024-04-06T13:47:36.121064+0000","last_fullsized":"2024-04-06T13:47:36.121064+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:19:18.381473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.15","version":"226'3","reported_seq":18,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.588014+0000","last_change":"2024-04-06T13:47:21.714316+0000","last_active":"2024-04-06T13:47:34.588014+0000","last_peered":"2024-04-06T13:47:34.588014+0000","last_clean":"2024-04-06T13:47:34.588014+0000","last_became_active":"2024-04-06T13:47:21.713975+0000","last_became_peered":"2024-04-06T13:47:21.713975+0000","last_unstale":"2024-04-06T13:47:34.588014+0000","last_undegraded":"2024-04-06T13:47:34.588014+0000","last_fullsized":"2024-04-06T13:47:34.588014+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:33:33.280270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587735+0000","last_change":"2024-04-06T13:47:22.145873+0000","last_active":"2024-04-06T13:47:34.587735+0000","last_peered":"2024-04-06T13:47:34.587735+0000","last_clean":"2024-04-06T13:47:34.587735+0000","last_became_active":"2024-04-06T13:47:22.145602+0000","last_became_peered":"2024-04-06T13:47:22.145602+0000","last_unstale":"2024-04-06T13:47:34.587735+0000","last_undegraded":"2024-04-06T13:47:34.587735+0000","last_fullsized":"2024-04-06T13:47:34.587735+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:24:02.415549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120361+0000","last_change":"2024-04-06T13:47:34.202761+0000","last_active":"2024-04-06T13:47:36.120361+0000","last_peered":"2024-04-06T13:47:36.120361+0000","last_clean":"2024-04-06T13:47:36.120361+0000","last_became_active":"2024-04-06T13:47:34.202482+0000","last_became_peered":"2024-04-06T13:47:34.202482+0000","last_unstale":"2024-04-06T13:47:36.120361+0000","last_undegraded":"2024-04-06T13:47:36.120361+0000","last_fullsized":"2024-04-06T13:47:36.120361+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:15:26.247011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119533+0000","last_change":"2024-04-06T13:47:34.611811+0000","last_active":"2024-04-06T13:47:36.119533+0000","last_peered":"2024-04-06T13:47:36.119533+0000","last_clean":"2024-04-06T13:47:36.119533+0000","last_became_active":"2024-04-06T13:47:34.611383+0000","last_became_peered":"2024-04-06T13:47:34.611383+0000","last_unstale":"2024-04-06T13:47:36.119533+0000","last_undegraded":"2024-04-06T13:47:36.119533+0000","last_fullsized":"2024-04-06T13:47:36.119533+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:54:04.532210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.598422+0000","last_change":"2024-04-06T13:47:21.722544+0000","last_active":"2024-04-06T13:47:34.598422+0000","last_peered":"2024-04-06T13:47:34.598422+0000","last_clean":"2024-04-06T13:47:34.598422+0000","last_became_active":"2024-04-06T13:47:21.722242+0000","last_became_peered":"2024-04-06T13:47:21.722242+0000","last_unstale":"2024-04-06T13:47:34.598422+0000","last_undegraded":"2024-04-06T13:47:34.598422+0000","last_fullsized":"2024-04-06T13:47:34.598422+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:14:53.056289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590272+0000","last_change":"2024-04-06T13:47:22.136160+0000","last_active":"2024-04-06T13:47:34.590272+0000","last_peered":"2024-04-06T13:47:34.590272+0000","last_clean":"2024-04-06T13:47:34.590272+0000","last_became_active":"2024-04-06T13:47:22.135869+0000","last_became_peered":"2024-04-06T13:47:22.135869+0000","last_unstale":"2024-04-06T13:47:34.590272+0000","last_undegraded":"2024-04-06T13:47:34.590272+0000","last_fullsized":"2024-04-06T13:47:34.590272+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:05:17.896292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"230'8","reported_seq":19,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121500+0000","last_change":"2024-04-06T13:47:34.200748+0000","last_active":"2024-04-06T13:47:36.121500+0000","last_peered":"2024-04-06T13:47:36.121500+0000","last_clean":"2024-04-06T13:47:36.121500+0000","last_became_active":"2024-04-06T13:47:34.200487+0000","last_became_peered":"2024-04-06T13:47:34.200487+0000","last_unstale":"2024-04-06T13:47:36.121500+0000","last_undegraded":"2024-04-06T13:47:36.121500+0000","last_fullsized":"2024-04-06T13:47:36.121500+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:14:37.551778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120256+0000","last_change":"2024-04-06T13:47:34.601764+0000","last_active":"2024-04-06T13:47:36.120256+0000","last_peered":"2024-04-06T13:47:36.120256+0000","last_clean":"2024-04-06T13:47:36.120256+0000","last_became_active":"2024-04-06T13:47:34.601523+0000","last_became_peered":"2024-04-06T13:47:34.601523+0000","last_unstale":"2024-04-06T13:47:36.120256+0000","last_undegraded":"2024-04-06T13:47:36.120256+0000","last_fullsized":"2024-04-06T13:47:36.120256+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:06:22.305691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.593491+0000","last_change":"2024-04-06T13:47:21.722601+0000","last_active":"2024-04-06T13:47:34.593491+0000","last_peered":"2024-04-06T13:47:34.593491+0000","last_clean":"2024-04-06T13:47:34.593491+0000","last_became_active":"2024-04-06T13:47:21.722328+0000","last_became_peered":"2024-04-06T13:47:21.722328+0000","last_unstale":"2024-04-06T13:47:34.593491+0000","last_undegraded":"2024-04-06T13:47:34.593491+0000","last_fullsized":"2024-04-06T13:47:34.593491+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:30:35.611598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.594614+0000","last_change":"2024-04-06T13:47:22.132142+0000","last_active":"2024-04-06T13:47:34.594614+0000","last_peered":"2024-04-06T13:47:34.594614+0000","last_clean":"2024-04-06T13:47:34.594614+0000","last_became_active":"2024-04-06T13:47:22.131874+0000","last_became_peered":"2024-04-06T13:47:22.131874+0000","last_unstale":"2024-04-06T13:47:34.594614+0000","last_undegraded":"2024-04-06T13:47:34.594614+0000","last_fullsized":"2024-04-06T13:47:34.594614+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:14:26.082176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593478+0000","last_change":"2024-04-06T13:47:34.191145+0000","last_active":"2024-04-06T13:47:35.593478+0000","last_peered":"2024-04-06T13:47:35.593478+0000","last_clean":"2024-04-06T13:47:35.593478+0000","last_became_active":"2024-04-06T13:47:34.190823+0000","last_became_peered":"2024-04-06T13:47:34.190823+0000","last_unstale":"2024-04-06T13:47:35.593478+0000","last_undegraded":"2024-04-06T13:47:35.593478+0000","last_fullsized":"2024-04-06T13:47:35.593478+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:55:13.281748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593885+0000","last_change":"2024-04-06T13:47:34.603387+0000","last_active":"2024-04-06T13:47:35.593885+0000","last_peered":"2024-04-06T13:47:35.593885+0000","last_clean":"2024-04-06T13:47:35.593885+0000","last_became_active":"2024-04-06T13:47:34.603141+0000","last_became_peered":"2024-04-06T13:47:34.603141+0000","last_unstale":"2024-04-06T13:47:35.593885+0000","last_undegraded":"2024-04-06T13:47:35.593885+0000","last_fullsized":"2024-04-06T13:47:35.593885+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:00.587625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590221+0000","last_change":"2024-04-06T13:47:21.715138+0000","last_active":"2024-04-06T13:47:34.590221+0000","last_peered":"2024-04-06T13:47:34.590221+0000","last_clean":"2024-04-06T13:47:34.590221+0000","last_became_active":"2024-04-06T13:47:21.714839+0000","last_became_peered":"2024-04-06T13:47:21.714839+0000","last_unstale":"2024-04-06T13:47:34.590221+0000","last_undegraded":"2024-04-06T13:47:34.590221+0000","last_fullsized":"2024-04-06T13:47:34.590221+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:02:18.580630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.588090+0000","last_change":"2024-04-06T13:47:22.140985+0000","last_active":"2024-04-06T13:47:34.588090+0000","last_peered":"2024-04-06T13:47:34.588090+0000","last_clean":"2024-04-06T13:47:34.588090+0000","last_became_active":"2024-04-06T13:47:22.140751+0000","last_became_peered":"2024-04-06T13:47:22.140751+0000","last_unstale":"2024-04-06T13:47:34.588090+0000","last_undegraded":"2024-04-06T13:47:34.588090+0000","last_fullsized":"2024-04-06T13:47:34.588090+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:20:03.901368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120753+0000","last_change":"2024-04-06T13:47:34.198817+0000","last_active":"2024-04-06T13:47:36.120753+0000","last_peered":"2024-04-06T13:47:36.120753+0000","last_clean":"2024-04-06T13:47:36.120753+0000","last_became_active":"2024-04-06T13:47:34.198515+0000","last_became_peered":"2024-04-06T13:47:34.198515+0000","last_unstale":"2024-04-06T13:47:36.120753+0000","last_undegraded":"2024-04-06T13:47:36.120753+0000","last_fullsized":"2024-04-06T13:47:36.120753+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:32:41.940060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594168+0000","last_change":"2024-04-06T13:47:34.599400+0000","last_active":"2024-04-06T13:47:35.594168+0000","last_peered":"2024-04-06T13:47:35.594168+0000","last_clean":"2024-04-06T13:47:35.594168+0000","last_became_active":"2024-04-06T13:47:34.599155+0000","last_became_peered":"2024-04-06T13:47:34.599155+0000","last_unstale":"2024-04-06T13:47:35.594168+0000","last_undegraded":"2024-04-06T13:47:35.594168+0000","last_fullsized":"2024-04-06T13:47:35.594168+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:05:31.482786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.596154+0000","last_change":"2024-04-06T13:47:21.713514+0000","last_active":"2024-04-06T13:47:34.596154+0000","last_peered":"2024-04-06T13:47:34.596154+0000","last_clean":"2024-04-06T13:47:34.596154+0000","last_became_active":"2024-04-06T13:47:21.713230+0000","last_became_peered":"2024-04-06T13:47:21.713230+0000","last_unstale":"2024-04-06T13:47:34.596154+0000","last_undegraded":"2024-04-06T13:47:34.596154+0000","last_fullsized":"2024-04-06T13:47:34.596154+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:29:38.052685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.596525+0000","last_change":"2024-04-06T13:47:22.141742+0000","last_active":"2024-04-06T13:47:34.596525+0000","last_peered":"2024-04-06T13:47:34.596525+0000","last_clean":"2024-04-06T13:47:34.596525+0000","last_became_active":"2024-04-06T13:47:22.141302+0000","last_became_peered":"2024-04-06T13:47:22.141302+0000","last_unstale":"2024-04-06T13:47:34.596525+0000","last_undegraded":"2024-04-06T13:47:34.596525+0000","last_fullsized":"2024-04-06T13:47:34.596525+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:24.049383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"230'4","reported_seq":15,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120441+0000","last_change":"2024-04-06T13:47:34.190504+0000","last_active":"2024-04-06T13:47:36.120441+0000","last_peered":"2024-04-06T13:47:36.120441+0000","last_clean":"2024-04-06T13:47:36.120441+0000","last_became_active":"2024-04-06T13:47:34.189775+0000","last_became_peered":"2024-04-06T13:47:34.189775+0000","last_unstale":"2024-04-06T13:47:36.120441+0000","last_undegraded":"2024-04-06T13:47:36.120441+0000","last_fullsized":"2024-04-06T13:47:36.120441+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:23:14.070825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594110+0000","last_change":"2024-04-06T13:47:34.608005+0000","last_active":"2024-04-06T13:47:35.594110+0000","last_peered":"2024-04-06T13:47:35.594110+0000","last_clean":"2024-04-06T13:47:35.594110+0000","last_became_active":"2024-04-06T13:47:34.607728+0000","last_became_peered":"2024-04-06T13:47:34.607728+0000","last_unstale":"2024-04-06T13:47:35.594110+0000","last_undegraded":"2024-04-06T13:47:35.594110+0000","last_fullsized":"2024-04-06T13:47:35.594110+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:47:13.207002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1b","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.598875+0000","last_change":"2024-04-06T13:47:21.714435+0000","last_active":"2024-04-06T13:47:34.598875+0000","last_peered":"2024-04-06T13:47:34.598875+0000","last_clean":"2024-04-06T13:47:34.598875+0000","last_became_active":"2024-04-06T13:47:21.714154+0000","last_became_peered":"2024-04-06T13:47:21.714154+0000","last_unstale":"2024-04-06T13:47:34.598875+0000","last_undegraded":"2024-04-06T13:47:34.598875+0000","last_fullsized":"2024-04-06T13:47:34.598875+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:25.144968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.596726+0000","last_change":"2024-04-06T13:47:22.139248+0000","last_active":"2024-04-06T13:47:34.596726+0000","last_peered":"2024-04-06T13:47:34.596726+0000","last_clean":"2024-04-06T13:47:34.596726+0000","last_became_active":"2024-04-06T13:47:22.139085+0000","last_became_peered":"2024-04-06T13:47:22.139085+0000","last_unstale":"2024-04-06T13:47:34.596726+0000","last_undegraded":"2024-04-06T13:47:34.596726+0000","last_fullsized":"2024-04-06T13:47:34.596726+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:41:53.649355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.595182+0000","last_change":"2024-04-06T13:47:34.196041+0000","last_active":"2024-04-06T13:47:35.595182+0000","last_peered":"2024-04-06T13:47:35.595182+0000","last_clean":"2024-04-06T13:47:35.595182+0000","last_became_active":"2024-04-06T13:47:34.195744+0000","last_became_peered":"2024-04-06T13:47:34.195744+0000","last_unstale":"2024-04-06T13:47:35.595182+0000","last_undegraded":"2024-04-06T13:47:35.595182+0000","last_fullsized":"2024-04-06T13:47:35.595182+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:58:54.970945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120640+0000","last_change":"2024-04-06T13:47:34.612984+0000","last_active":"2024-04-06T13:47:36.120640+0000","last_peered":"2024-04-06T13:47:36.120640+0000","last_clean":"2024-04-06T13:47:36.120640+0000","last_became_active":"2024-04-06T13:47:34.612623+0000","last_became_peered":"2024-04-06T13:47:34.612623+0000","last_unstale":"2024-04-06T13:47:36.120640+0000","last_undegraded":"2024-04-06T13:47:36.120640+0000","last_fullsized":"2024-04-06T13:47:36.120640+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:25:53.399400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1c","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.588368+0000","last_change":"2024-04-06T13:47:21.713494+0000","last_active":"2024-04-06T13:47:34.588368+0000","last_peered":"2024-04-06T13:47:34.588368+0000","last_clean":"2024-04-06T13:47:34.588368+0000","last_became_active":"2024-04-06T13:47:21.713155+0000","last_became_peered":"2024-04-06T13:47:21.713155+0000","last_unstale":"2024-04-06T13:47:34.588368+0000","last_undegraded":"2024-04-06T13:47:34.588368+0000","last_fullsized":"2024-04-06T13:47:34.588368+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:43:51.273765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590695+0000","last_change":"2024-04-06T13:47:22.146835+0000","last_active":"2024-04-06T13:47:34.590695+0000","last_peered":"2024-04-06T13:47:34.590695+0000","last_clean":"2024-04-06T13:47:34.590695+0000","last_became_active":"2024-04-06T13:47:22.146565+0000","last_became_peered":"2024-04-06T13:47:22.146565+0000","last_unstale":"2024-04-06T13:47:34.590695+0000","last_undegraded":"2024-04-06T13:47:34.590695+0000","last_fullsized":"2024-04-06T13:47:34.590695+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:10.026234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594904+0000","last_change":"2024-04-06T13:47:34.193924+0000","last_active":"2024-04-06T13:47:35.594904+0000","last_peered":"2024-04-06T13:47:35.594904+0000","last_clean":"2024-04-06T13:47:35.594904+0000","last_became_active":"2024-04-06T13:47:34.193475+0000","last_became_peered":"2024-04-06T13:47:34.193475+0000","last_unstale":"2024-04-06T13:47:35.594904+0000","last_undegraded":"2024-04-06T13:47:35.594904+0000","last_fullsized":"2024-04-06T13:47:35.594904+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:33:08.141180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121335+0000","last_change":"2024-04-06T13:47:34.612503+0000","last_active":"2024-04-06T13:47:36.121335+0000","last_peered":"2024-04-06T13:47:36.121335+0000","last_clean":"2024-04-06T13:47:36.121335+0000","last_became_active":"2024-04-06T13:47:34.609994+0000","last_became_peered":"2024-04-06T13:47:34.609994+0000","last_unstale":"2024-04-06T13:47:36.121335+0000","last_undegraded":"2024-04-06T13:47:36.121335+0000","last_fullsized":"2024-04-06T13:47:36.121335+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:15:06.533915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597510+0000","last_change":"2024-04-06T13:47:21.714467+0000","last_active":"2024-04-06T13:47:34.597510+0000","last_peered":"2024-04-06T13:47:34.597510+0000","last_clean":"2024-04-06T13:47:34.597510+0000","last_became_active":"2024-04-06T13:47:21.714208+0000","last_became_peered":"2024-04-06T13:47:21.714208+0000","last_unstale":"2024-04-06T13:47:34.597510+0000","last_undegraded":"2024-04-06T13:47:34.597510+0000","last_fullsized":"2024-04-06T13:47:34.597510+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:09.423492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597362+0000","last_change":"2024-04-06T13:47:22.139445+0000","last_active":"2024-04-06T13:47:34.597362+0000","last_peered":"2024-04-06T13:47:34.597362+0000","last_clean":"2024-04-06T13:47:34.597362+0000","last_became_active":"2024-04-06T13:47:22.138882+0000","last_became_peered":"2024-04-06T13:47:22.138882+0000","last_unstale":"2024-04-06T13:47:34.597362+0000","last_undegraded":"2024-04-06T13:47:34.597362+0000","last_fullsized":"2024-04-06T13:47:34.597362+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:45:57.790454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120282+0000","last_change":"2024-04-06T13:47:34.199071+0000","last_active":"2024-04-06T13:47:36.120282+0000","last_peered":"2024-04-06T13:47:36.120282+0000","last_clean":"2024-04-06T13:47:36.120282+0000","last_became_active":"2024-04-06T13:47:34.198865+0000","last_became_peered":"2024-04-06T13:47:34.198865+0000","last_unstale":"2024-04-06T13:47:36.120282+0000","last_undegraded":"2024-04-06T13:47:36.120282+0000","last_fullsized":"2024-04-06T13:47:36.120282+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:52:40.940598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593654+0000","last_change":"2024-04-06T13:47:34.604731+0000","last_active":"2024-04-06T13:47:35.593654+0000","last_peered":"2024-04-06T13:47:35.593654+0000","last_clean":"2024-04-06T13:47:35.593654+0000","last_became_active":"2024-04-06T13:47:34.604478+0000","last_became_peered":"2024-04-06T13:47:34.604478+0000","last_unstale":"2024-04-06T13:47:35.593654+0000","last_undegraded":"2024-04-06T13:47:35.593654+0000","last_fullsized":"2024-04-06T13:47:35.593654+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:56:51.094042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1e","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.594143+0000","last_change":"2024-04-06T13:47:21.721394+0000","last_active":"2024-04-06T13:47:34.594143+0000","last_peered":"2024-04-06T13:47:34.594143+0000","last_clean":"2024-04-06T13:47:34.594143+0000","last_became_active":"2024-04-06T13:47:21.721093+0000","last_became_peered":"2024-04-06T13:47:21.721093+0000","last_unstale":"2024-04-06T13:47:34.594143+0000","last_undegraded":"2024-04-06T13:47:34.594143+0000","last_fullsized":"2024-04-06T13:47:34.594143+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:30:58.296350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591369+0000","last_change":"2024-04-06T13:47:22.136299+0000","last_active":"2024-04-06T13:47:34.591369+0000","last_peered":"2024-04-06T13:47:34.591369+0000","last_clean":"2024-04-06T13:47:34.591369+0000","last_became_active":"2024-04-06T13:47:22.136046+0000","last_became_peered":"2024-04-06T13:47:22.136046+0000","last_unstale":"2024-04-06T13:47:34.591369+0000","last_undegraded":"2024-04-06T13:47:34.591369+0000","last_fullsized":"2024-04-06T13:47:34.591369+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:19:51.727577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593715+0000","last_change":"2024-04-06T13:47:34.184612+0000","last_active":"2024-04-06T13:47:35.593715+0000","last_peered":"2024-04-06T13:47:35.593715+0000","last_clean":"2024-04-06T13:47:35.593715+0000","last_became_active":"2024-04-06T13:47:34.184034+0000","last_became_peered":"2024-04-06T13:47:34.184034+0000","last_unstale":"2024-04-06T13:47:35.593715+0000","last_undegraded":"2024-04-06T13:47:35.593715+0000","last_fullsized":"2024-04-06T13:47:35.593715+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:36.867273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119714+0000","last_change":"2024-04-06T13:47:34.600193+0000","last_active":"2024-04-06T13:47:36.119714+0000","last_peered":"2024-04-06T13:47:36.119714+0000","last_clean":"2024-04-06T13:47:36.119714+0000","last_became_active":"2024-04-06T13:47:34.599954+0000","last_became_peered":"2024-04-06T13:47:34.599954+0000","last_unstale":"2024-04-06T13:47:36.119714+0000","last_undegraded":"2024-04-06T13:47:36.119714+0000","last_fullsized":"2024-04-06T13:47:36.119714+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:19:08.553269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1f","version":"226'4","reported_seq":19,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.596866+0000","last_change":"2024-04-06T13:47:21.722633+0000","last_active":"2024-04-06T13:47:34.596866+0000","last_peered":"2024-04-06T13:47:34.596866+0000","last_clean":"2024-04-06T13:47:34.596866+0000","last_became_active":"2024-04-06T13:47:21.722356+0000","last_became_peered":"2024-04-06T13:47:21.722356+0000","last_unstale":"2024-04-06T13:47:34.596866+0000","last_undegraded":"2024-04-06T13:47:34.596866+0000","last_fullsized":"2024-04-06T13:47:34.596866+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:55.001531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.595629+0000","last_change":"2024-04-06T13:47:22.139965+0000","last_active":"2024-04-06T13:47:34.595629+0000","last_peered":"2024-04-06T13:47:34.595629+0000","last_clean":"2024-04-06T13:47:34.595629+0000","last_became_active":"2024-04-06T13:47:22.139645+0000","last_became_peered":"2024-04-06T13:47:22.139645+0000","last_unstale":"2024-04-06T13:47:34.595629+0000","last_undegraded":"2024-04-06T13:47:34.595629+0000","last_fullsized":"2024-04-06T13:47:34.595629+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:32:56.671578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119644+0000","last_change":"2024-04-06T13:47:34.199704+0000","last_active":"2024-04-06T13:47:36.119644+0000","last_peered":"2024-04-06T13:47:36.119644+0000","last_clean":"2024-04-06T13:47:36.119644+0000","last_became_active":"2024-04-06T13:47:34.199440+0000","last_became_peered":"2024-04-06T13:47:34.199440+0000","last_unstale":"2024-04-06T13:47:36.119644+0000","last_undegraded":"2024-04-06T13:47:36.119644+0000","last_fullsized":"2024-04-06T13:47:36.119644+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:13:12.452743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120301+0000","last_change":"2024-04-06T13:47:34.603979+0000","last_active":"2024-04-06T13:47:36.120301+0000","last_peered":"2024-04-06T13:47:36.120301+0000","last_clean":"2024-04-06T13:47:36.120301+0000","last_became_active":"2024-04-06T13:47:34.603726+0000","last_became_peered":"2024-04-06T13:47:34.603726+0000","last_unstale":"2024-04-06T13:47:36.120301+0000","last_undegraded":"2024-04-06T13:47:36.120301+0000","last_fullsized":"2024-04-06T13:47:36.120301+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:14:30.470482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.595529+0000","last_change":"2024-04-06T13:47:22.132331+0000","last_active":"2024-04-06T13:47:34.595529+0000","last_peered":"2024-04-06T13:47:34.595529+0000","last_clean":"2024-04-06T13:47:34.595529+0000","last_became_active":"2024-04-06T13:47:22.132068+0000","last_became_peered":"2024-04-06T13:47:22.132068+0000","last_unstale":"2024-04-06T13:47:34.595529+0000","last_undegraded":"2024-04-06T13:47:34.595529+0000","last_fullsized":"2024-04-06T13:47:34.595529+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:45:10.805753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593609+0000","last_change":"2024-04-06T13:47:34.616158+0000","last_active":"2024-04-06T13:47:35.593609+0000","last_peered":"2024-04-06T13:47:35.593609+0000","last_clean":"2024-04-06T13:47:35.593609+0000","last_became_active":"2024-04-06T13:47:34.615519+0000","last_became_peered":"2024-04-06T13:47:34.615519+0000","last_unstale":"2024-04-06T13:47:35.593609+0000","last_undegraded":"2024-04-06T13:47:35.593609+0000","last_fullsized":"2024-04-06T13:47:35.593609+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:46:21.375825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.595689+0000","last_change":"2024-04-06T13:47:22.141102+0000","last_active":"2024-04-06T13:47:34.595689+0000","last_peered":"2024-04-06T13:47:34.595689+0000","last_clean":"2024-04-06T13:47:34.595689+0000","last_became_active":"2024-04-06T13:47:22.140831+0000","last_became_peered":"2024-04-06T13:47:22.140831+0000","last_unstale":"2024-04-06T13:47:34.595689+0000","last_undegraded":"2024-04-06T13:47:34.595689+0000","last_fullsized":"2024-04-06T13:47:34.595689+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:27:31.928923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120955+0000","last_change":"2024-04-06T13:47:34.611700+0000","last_active":"2024-04-06T13:47:36.120955+0000","last_peered":"2024-04-06T13:47:36.120955+0000","last_clean":"2024-04-06T13:47:36.120955+0000","last_became_active":"2024-04-06T13:47:34.611406+0000","last_became_peered":"2024-04-06T13:47:34.611406+0000","last_unstale":"2024-04-06T13:47:36.120955+0000","last_undegraded":"2024-04-06T13:47:36.120955+0000","last_fullsized":"2024-04-06T13:47:36.120955+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:32:07.103859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.596434+0000","last_change":"2024-04-06T13:47:22.138788+0000","last_active":"2024-04-06T13:47:34.596434+0000","last_peered":"2024-04-06T13:47:34.596434+0000","last_clean":"2024-04-06T13:47:34.596434+0000","last_became_active":"2024-04-06T13:47:22.138480+0000","last_became_peered":"2024-04-06T13:47:22.138480+0000","last_unstale":"2024-04-06T13:47:34.596434+0000","last_undegraded":"2024-04-06T13:47:34.596434+0000","last_fullsized":"2024-04-06T13:47:34.596434+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:40.999324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593525+0000","last_change":"2024-04-06T13:47:34.618097+0000","last_active":"2024-04-06T13:47:35.593525+0000","last_peered":"2024-04-06T13:47:35.593525+0000","last_clean":"2024-04-06T13:47:35.593525+0000","last_became_active":"2024-04-06T13:47:34.617829+0000","last_became_peered":"2024-04-06T13:47:34.617829+0000","last_unstale":"2024-04-06T13:47:35.593525+0000","last_undegraded":"2024-04-06T13:47:35.593525+0000","last_fullsized":"2024-04-06T13:47:35.593525+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:01:17.865971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587883+0000","last_change":"2024-04-06T13:47:22.139159+0000","last_active":"2024-04-06T13:47:34.587883+0000","last_peered":"2024-04-06T13:47:34.587883+0000","last_clean":"2024-04-06T13:47:34.587883+0000","last_became_active":"2024-04-06T13:47:22.138916+0000","last_became_peered":"2024-04-06T13:47:22.138916+0000","last_unstale":"2024-04-06T13:47:34.587883+0000","last_undegraded":"2024-04-06T13:47:34.587883+0000","last_fullsized":"2024-04-06T13:47:34.587883+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:58:23.128672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119480+0000","last_change":"2024-04-06T13:47:34.603181+0000","last_active":"2024-04-06T13:47:36.119480+0000","last_peered":"2024-04-06T13:47:36.119480+0000","last_clean":"2024-04-06T13:47:36.119480+0000","last_became_active":"2024-04-06T13:47:34.602932+0000","last_became_peered":"2024-04-06T13:47:34.602932+0000","last_unstale":"2024-04-06T13:47:36.119480+0000","last_undegraded":"2024-04-06T13:47:36.119480+0000","last_fullsized":"2024-04-06T13:47:36.119480+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:33:04.679206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594463+0000","last_change":"2024-04-06T13:47:34.614325+0000","last_active":"2024-04-06T13:47:35.594463+0000","last_peered":"2024-04-06T13:47:35.594463+0000","last_clean":"2024-04-06T13:47:35.594463+0000","last_became_active":"2024-04-06T13:47:34.613988+0000","last_became_peered":"2024-04-06T13:47:34.613988+0000","last_unstale":"2024-04-06T13:47:35.594463+0000","last_undegraded":"2024-04-06T13:47:35.594463+0000","last_fullsized":"2024-04-06T13:47:35.594463+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:06:12.976308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591125+0000","last_change":"2024-04-06T13:47:22.139194+0000","last_active":"2024-04-06T13:47:34.591125+0000","last_peered":"2024-04-06T13:47:34.591125+0000","last_clean":"2024-04-06T13:47:34.591125+0000","last_became_active":"2024-04-06T13:47:22.138677+0000","last_became_peered":"2024-04-06T13:47:22.138677+0000","last_unstale":"2024-04-06T13:47:34.591125+0000","last_undegraded":"2024-04-06T13:47:34.591125+0000","last_fullsized":"2024-04-06T13:47:34.591125+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:36:38.719202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.595821+0000","last_change":"2024-04-06T13:47:34.608117+0000","last_active":"2024-04-06T13:47:35.595821+0000","last_peered":"2024-04-06T13:47:35.595821+0000","last_clean":"2024-04-06T13:47:35.595821+0000","last_became_active":"2024-04-06T13:47:34.607867+0000","last_became_peered":"2024-04-06T13:47:34.607867+0000","last_unstale":"2024-04-06T13:47:35.595821+0000","last_undegraded":"2024-04-06T13:47:35.595821+0000","last_fullsized":"2024-04-06T13:47:35.595821+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:44:43.484577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591899+0000","last_change":"2024-04-06T13:47:22.147177+0000","last_active":"2024-04-06T13:47:34.591899+0000","last_peered":"2024-04-06T13:47:34.591899+0000","last_clean":"2024-04-06T13:47:34.591899+0000","last_became_active":"2024-04-06T13:47:22.146634+0000","last_became_peered":"2024-04-06T13:47:22.146634+0000","last_unstale":"2024-04-06T13:47:34.591899+0000","last_undegraded":"2024-04-06T13:47:34.591899+0000","last_fullsized":"2024-04-06T13:47:34.591899+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:16:29.805273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.594048+0000","last_change":"2024-04-06T13:47:22.144335+0000","last_active":"2024-04-06T13:47:34.594048+0000","last_peered":"2024-04-06T13:47:34.594048+0000","last_clean":"2024-04-06T13:47:34.594048+0000","last_became_active":"2024-04-06T13:47:22.144006+0000","last_became_peered":"2024-04-06T13:47:22.144006+0000","last_unstale":"2024-04-06T13:47:34.594048+0000","last_undegraded":"2024-04-06T13:47:34.594048+0000","last_fullsized":"2024-04-06T13:47:34.594048+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:02:19.582423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120763+0000","last_change":"2024-04-06T13:47:34.613172+0000","last_active":"2024-04-06T13:47:36.120763+0000","last_peered":"2024-04-06T13:47:36.120763+0000","last_clean":"2024-04-06T13:47:36.120763+0000","last_became_active":"2024-04-06T13:47:34.612958+0000","last_became_peered":"2024-04-06T13:47:34.612958+0000","last_unstale":"2024-04-06T13:47:36.120763+0000","last_undegraded":"2024-04-06T13:47:36.120763+0000","last_fullsized":"2024-04-06T13:47:36.120763+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:18:18.895673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120281+0000","last_change":"2024-04-06T13:47:34.611841+0000","last_active":"2024-04-06T13:47:36.120281+0000","last_peered":"2024-04-06T13:47:36.120281+0000","last_clean":"2024-04-06T13:47:36.120281+0000","last_became_active":"2024-04-06T13:47:34.611230+0000","last_became_peered":"2024-04-06T13:47:34.611230+0000","last_unstale":"2024-04-06T13:47:36.120281+0000","last_undegraded":"2024-04-06T13:47:36.120281+0000","last_fullsized":"2024-04-06T13:47:36.120281+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:55:42.584686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587853+0000","last_change":"2024-04-06T13:47:22.133305+0000","last_active":"2024-04-06T13:47:34.587853+0000","last_peered":"2024-04-06T13:47:34.587853+0000","last_clean":"2024-04-06T13:47:34.587853+0000","last_became_active":"2024-04-06T13:47:22.132897+0000","last_became_peered":"2024-04-06T13:47:22.132897+0000","last_unstale":"2024-04-06T13:47:34.587853+0000","last_undegraded":"2024-04-06T13:47:34.587853+0000","last_fullsized":"2024-04-06T13:47:34.587853+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:10:25.671045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120175+0000","last_change":"2024-04-06T13:47:34.614504+0000","last_active":"2024-04-06T13:47:36.120175+0000","last_peered":"2024-04-06T13:47:36.120175+0000","last_clean":"2024-04-06T13:47:36.120175+0000","last_became_active":"2024-04-06T13:47:34.614254+0000","last_became_peered":"2024-04-06T13:47:34.614254+0000","last_unstale":"2024-04-06T13:47:36.120175+0000","last_undegraded":"2024-04-06T13:47:36.120175+0000","last_fullsized":"2024-04-06T13:47:36.120175+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:52:34.891491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590235+0000","last_change":"2024-04-06T13:47:22.139336+0000","last_active":"2024-04-06T13:47:34.590235+0000","last_peered":"2024-04-06T13:47:34.590235+0000","last_clean":"2024-04-06T13:47:34.590235+0000","last_became_active":"2024-04-06T13:47:22.139092+0000","last_became_peered":"2024-04-06T13:47:22.139092+0000","last_unstale":"2024-04-06T13:47:34.590235+0000","last_undegraded":"2024-04-06T13:47:34.590235+0000","last_fullsized":"2024-04-06T13:47:34.590235+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:55:13.445775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593910+0000","last_change":"2024-04-06T13:47:34.602857+0000","last_active":"2024-04-06T13:47:35.593910+0000","last_peered":"2024-04-06T13:47:35.593910+0000","last_clean":"2024-04-06T13:47:35.593910+0000","last_became_active":"2024-04-06T13:47:34.602608+0000","last_became_peered":"2024-04-06T13:47:34.602608+0000","last_unstale":"2024-04-06T13:47:35.593910+0000","last_undegraded":"2024-04-06T13:47:35.593910+0000","last_fullsized":"2024-04-06T13:47:35.593910+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:09:41.768103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597775+0000","last_change":"2024-04-06T13:47:22.139078+0000","last_active":"2024-04-06T13:47:34.597775+0000","last_peered":"2024-04-06T13:47:34.597775+0000","last_clean":"2024-04-06T13:47:34.597775+0000","last_became_active":"2024-04-06T13:47:22.138825+0000","last_became_peered":"2024-04-06T13:47:22.138825+0000","last_unstale":"2024-04-06T13:47:34.597775+0000","last_undegraded":"2024-04-06T13:47:34.597775+0000","last_fullsized":"2024-04-06T13:47:34.597775+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:14:17.994649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594332+0000","last_change":"2024-04-06T13:47:34.602833+0000","last_active":"2024-04-06T13:47:35.594332+0000","last_peered":"2024-04-06T13:47:35.594332+0000","last_clean":"2024-04-06T13:47:35.594332+0000","last_became_active":"2024-04-06T13:47:34.602554+0000","last_became_peered":"2024-04-06T13:47:34.602554+0000","last_unstale":"2024-04-06T13:47:35.594332+0000","last_undegraded":"2024-04-06T13:47:35.594332+0000","last_fullsized":"2024-04-06T13:47:35.594332+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:11:42.747001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597461+0000","last_change":"2024-04-06T13:47:22.139139+0000","last_active":"2024-04-06T13:47:34.597461+0000","last_peered":"2024-04-06T13:47:34.597461+0000","last_clean":"2024-04-06T13:47:34.597461+0000","last_became_active":"2024-04-06T13:47:22.138505+0000","last_became_peered":"2024-04-06T13:47:22.138505+0000","last_unstale":"2024-04-06T13:47:34.597461+0000","last_undegraded":"2024-04-06T13:47:34.597461+0000","last_fullsized":"2024-04-06T13:47:34.597461+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:07:04.253792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119850+0000","last_change":"2024-04-06T13:47:34.596373+0000","last_active":"2024-04-06T13:47:36.119850+0000","last_peered":"2024-04-06T13:47:36.119850+0000","last_clean":"2024-04-06T13:47:36.119850+0000","last_became_active":"2024-04-06T13:47:34.596027+0000","last_became_peered":"2024-04-06T13:47:34.596027+0000","last_unstale":"2024-04-06T13:47:36.119850+0000","last_undegraded":"2024-04-06T13:47:36.119850+0000","last_fullsized":"2024-04-06T13:47:36.119850+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:13:13.458973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.602762+0000","last_change":"2024-04-06T13:47:22.141672+0000","last_active":"2024-04-06T13:47:34.602762+0000","last_peered":"2024-04-06T13:47:34.602762+0000","last_clean":"2024-04-06T13:47:34.602762+0000","last_became_active":"2024-04-06T13:47:22.141369+0000","last_became_peered":"2024-04-06T13:47:22.141369+0000","last_unstale":"2024-04-06T13:47:34.602762+0000","last_undegraded":"2024-04-06T13:47:34.602762+0000","last_fullsized":"2024-04-06T13:47:34.602762+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:27:56.909726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594378+0000","last_change":"2024-04-06T13:47:34.613931+0000","last_active":"2024-04-06T13:47:35.594378+0000","last_peered":"2024-04-06T13:47:35.594378+0000","last_clean":"2024-04-06T13:47:35.594378+0000","last_became_active":"2024-04-06T13:47:34.613197+0000","last_became_peered":"2024-04-06T13:47:34.613197+0000","last_unstale":"2024-04-06T13:47:35.594378+0000","last_undegraded":"2024-04-06T13:47:35.594378+0000","last_fullsized":"2024-04-06T13:47:35.594378+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:45:59.689026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.598146+0000","last_change":"2024-04-06T13:47:22.140812+0000","last_active":"2024-04-06T13:47:34.598146+0000","last_peered":"2024-04-06T13:47:34.598146+0000","last_clean":"2024-04-06T13:47:34.598146+0000","last_became_active":"2024-04-06T13:47:22.139010+0000","last_became_peered":"2024-04-06T13:47:22.139010+0000","last_unstale":"2024-04-06T13:47:34.598146+0000","last_undegraded":"2024-04-06T13:47:34.598146+0000","last_fullsized":"2024-04-06T13:47:34.598146+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:50:16.774388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120995+0000","last_change":"2024-04-06T13:47:34.606907+0000","last_active":"2024-04-06T13:47:36.120995+0000","last_peered":"2024-04-06T13:47:36.120995+0000","last_clean":"2024-04-06T13:47:36.120995+0000","last_became_active":"2024-04-06T13:47:34.606638+0000","last_became_peered":"2024-04-06T13:47:34.606638+0000","last_unstale":"2024-04-06T13:47:36.120995+0000","last_undegraded":"2024-04-06T13:47:36.120995+0000","last_fullsized":"2024-04-06T13:47:36.120995+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:33:04.679206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.592276+0000","last_change":"2024-04-06T13:47:22.145887+0000","last_active":"2024-04-06T13:47:34.592276+0000","last_peered":"2024-04-06T13:47:34.592276+0000","last_clean":"2024-04-06T13:47:34.592276+0000","last_became_active":"2024-04-06T13:47:22.141744+0000","last_became_peered":"2024-04-06T13:47:22.141744+0000","last_unstale":"2024-04-06T13:47:34.592276+0000","last_undegraded":"2024-04-06T13:47:34.592276+0000","last_fullsized":"2024-04-06T13:47:34.592276+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:40:34.810688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.595318+0000","last_change":"2024-04-06T13:47:34.607104+0000","last_active":"2024-04-06T13:47:35.595318+0000","last_peered":"2024-04-06T13:47:35.595318+0000","last_clean":"2024-04-06T13:47:35.595318+0000","last_became_active":"2024-04-06T13:47:34.606862+0000","last_became_peered":"2024-04-06T13:47:34.606862+0000","last_unstale":"2024-04-06T13:47:35.595318+0000","last_undegraded":"2024-04-06T13:47:35.595318+0000","last_fullsized":"2024-04-06T13:47:35.595318+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:48:41.996008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587816+0000","last_change":"2024-04-06T13:47:22.133302+0000","last_active":"2024-04-06T13:47:34.587816+0000","last_peered":"2024-04-06T13:47:34.587816+0000","last_clean":"2024-04-06T13:47:34.587816+0000","last_became_active":"2024-04-06T13:47:22.132902+0000","last_became_peered":"2024-04-06T13:47:22.132902+0000","last_unstale":"2024-04-06T13:47:34.587816+0000","last_undegraded":"2024-04-06T13:47:34.587816+0000","last_fullsized":"2024-04-06T13:47:34.587816+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:55:56.178154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.595060+0000","last_change":"2024-04-06T13:47:34.610455+0000","last_active":"2024-04-06T13:47:35.595060+0000","last_peered":"2024-04-06T13:47:35.595060+0000","last_clean":"2024-04-06T13:47:35.595060+0000","last_became_active":"2024-04-06T13:47:34.608587+0000","last_became_peered":"2024-04-06T13:47:34.608587+0000","last_unstale":"2024-04-06T13:47:35.595060+0000","last_undegraded":"2024-04-06T13:47:35.595060+0000","last_fullsized":"2024-04-06T13:47:35.595060+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:10:34.923090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587775+0000","last_change":"2024-04-06T13:47:22.148647+0000","last_active":"2024-04-06T13:47:34.587775+0000","last_peered":"2024-04-06T13:47:34.587775+0000","last_clean":"2024-04-06T13:47:34.587775+0000","last_became_active":"2024-04-06T13:47:22.148404+0000","last_became_peered":"2024-04-06T13:47:22.148404+0000","last_unstale":"2024-04-06T13:47:34.587775+0000","last_undegraded":"2024-04-06T13:47:34.587775+0000","last_fullsized":"2024-04-06T13:47:34.587775+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:38:04.045660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121608+0000","last_change":"2024-04-06T13:47:34.612490+0000","last_active":"2024-04-06T13:47:36.121608+0000","last_peered":"2024-04-06T13:47:36.121608+0000","last_clean":"2024-04-06T13:47:36.121608+0000","last_became_active":"2024-04-06T13:47:34.609937+0000","last_became_peered":"2024-04-06T13:47:34.609937+0000","last_unstale":"2024-04-06T13:47:36.121608+0000","last_undegraded":"2024-04-06T13:47:36.121608+0000","last_fullsized":"2024-04-06T13:47:36.121608+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:55:14.287979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590142+0000","last_change":"2024-04-06T13:47:22.142723+0000","last_active":"2024-04-06T13:47:34.590142+0000","last_peered":"2024-04-06T13:47:34.590142+0000","last_clean":"2024-04-06T13:47:34.590142+0000","last_became_active":"2024-04-06T13:47:22.142456+0000","last_became_peered":"2024-04-06T13:47:22.142456+0000","last_unstale":"2024-04-06T13:47:34.590142+0000","last_undegraded":"2024-04-06T13:47:34.590142+0000","last_fullsized":"2024-04-06T13:47:34.590142+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:24:10.160194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593391+0000","last_change":"2024-04-06T13:47:34.614963+0000","last_active":"2024-04-06T13:47:35.593391+0000","last_peered":"2024-04-06T13:47:35.593391+0000","last_clean":"2024-04-06T13:47:35.593391+0000","last_became_active":"2024-04-06T13:47:34.614666+0000","last_became_peered":"2024-04-06T13:47:34.614666+0000","last_unstale":"2024-04-06T13:47:35.593391+0000","last_undegraded":"2024-04-06T13:47:35.593391+0000","last_fullsized":"2024-04-06T13:47:35.593391+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:42:37.496332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.594188+0000","last_change":"2024-04-06T13:47:22.139469+0000","last_active":"2024-04-06T13:47:34.594188+0000","last_peered":"2024-04-06T13:47:34.594188+0000","last_clean":"2024-04-06T13:47:34.594188+0000","last_became_active":"2024-04-06T13:47:22.139177+0000","last_became_peered":"2024-04-06T13:47:22.139177+0000","last_unstale":"2024-04-06T13:47:34.594188+0000","last_undegraded":"2024-04-06T13:47:34.594188+0000","last_fullsized":"2024-04-06T13:47:34.594188+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:12:44.053539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593424+0000","last_change":"2024-04-06T13:47:34.602271+0000","last_active":"2024-04-06T13:47:35.593424+0000","last_peered":"2024-04-06T13:47:35.593424+0000","last_clean":"2024-04-06T13:47:35.593424+0000","last_became_active":"2024-04-06T13:47:34.601348+0000","last_became_peered":"2024-04-06T13:47:34.601348+0000","last_unstale":"2024-04-06T13:47:35.593424+0000","last_undegraded":"2024-04-06T13:47:35.593424+0000","last_fullsized":"2024-04-06T13:47:35.593424+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:27:02.196316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597547+0000","last_change":"2024-04-06T13:47:22.139295+0000","last_active":"2024-04-06T13:47:34.597547+0000","last_peered":"2024-04-06T13:47:34.597547+0000","last_clean":"2024-04-06T13:47:34.597547+0000","last_became_active":"2024-04-06T13:47:22.138518+0000","last_became_peered":"2024-04-06T13:47:22.138518+0000","last_unstale":"2024-04-06T13:47:34.597547+0000","last_undegraded":"2024-04-06T13:47:34.597547+0000","last_fullsized":"2024-04-06T13:47:34.597547+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:03:39.808430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119223+0000","last_change":"2024-04-06T13:47:34.601270+0000","last_active":"2024-04-06T13:47:36.119223+0000","last_peered":"2024-04-06T13:47:36.119223+0000","last_clean":"2024-04-06T13:47:36.119223+0000","last_became_active":"2024-04-06T13:47:34.601018+0000","last_became_peered":"2024-04-06T13:47:34.601018+0000","last_unstale":"2024-04-06T13:47:36.119223+0000","last_undegraded":"2024-04-06T13:47:36.119223+0000","last_fullsized":"2024-04-06T13:47:36.119223+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:36:38.568097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.600213+0000","last_change":"2024-04-06T13:47:22.150089+0000","last_active":"2024-04-06T13:47:34.600213+0000","last_peered":"2024-04-06T13:47:34.600213+0000","last_clean":"2024-04-06T13:47:34.600213+0000","last_became_active":"2024-04-06T13:47:22.149867+0000","last_became_peered":"2024-04-06T13:47:22.149867+0000","last_unstale":"2024-04-06T13:47:34.600213+0000","last_undegraded":"2024-04-06T13:47:34.600213+0000","last_fullsized":"2024-04-06T13:47:34.600213+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:55:30.108853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594327+0000","last_change":"2024-04-06T13:47:34.605543+0000","last_active":"2024-04-06T13:47:35.594327+0000","last_peered":"2024-04-06T13:47:35.594327+0000","last_clean":"2024-04-06T13:47:35.594327+0000","last_became_active":"2024-04-06T13:47:34.605284+0000","last_became_peered":"2024-04-06T13:47:34.605284+0000","last_unstale":"2024-04-06T13:47:35.594327+0000","last_undegraded":"2024-04-06T13:47:35.594327+0000","last_fullsized":"2024-04-06T13:47:35.594327+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:48:50.376524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591044+0000","last_change":"2024-04-06T13:47:22.139348+0000","last_active":"2024-04-06T13:47:34.591044+0000","last_peered":"2024-04-06T13:47:34.591044+0000","last_clean":"2024-04-06T13:47:34.591044+0000","last_became_active":"2024-04-06T13:47:22.138953+0000","last_became_peered":"2024-04-06T13:47:22.138953+0000","last_unstale":"2024-04-06T13:47:34.591044+0000","last_undegraded":"2024-04-06T13:47:34.591044+0000","last_fullsized":"2024-04-06T13:47:34.591044+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:51:33.543781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120352+0000","last_change":"2024-04-06T13:47:34.599654+0000","last_active":"2024-04-06T13:47:36.120352+0000","last_peered":"2024-04-06T13:47:36.120352+0000","last_clean":"2024-04-06T13:47:36.120352+0000","last_became_active":"2024-04-06T13:47:34.599387+0000","last_became_peered":"2024-04-06T13:47:34.599387+0000","last_unstale":"2024-04-06T13:47:36.120352+0000","last_undegraded":"2024-04-06T13:47:36.120352+0000","last_fullsized":"2024-04-06T13:47:36.120352+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:25:53.399400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.592045+0000","last_change":"2024-04-06T13:47:22.147317+0000","last_active":"2024-04-06T13:47:34.592045+0000","last_peered":"2024-04-06T13:47:34.592045+0000","last_clean":"2024-04-06T13:47:34.592045+0000","last_became_active":"2024-04-06T13:47:22.146941+0000","last_became_peered":"2024-04-06T13:47:22.146941+0000","last_unstale":"2024-04-06T13:47:34.592045+0000","last_undegraded":"2024-04-06T13:47:34.592045+0000","last_fullsized":"2024-04-06T13:47:34.592045+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:55:01.812146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121220+0000","last_change":"2024-04-06T13:47:34.617247+0000","last_active":"2024-04-06T13:47:36.121220+0000","last_peered":"2024-04-06T13:47:36.121220+0000","last_clean":"2024-04-06T13:47:36.121220+0000","last_became_active":"2024-04-06T13:47:34.616973+0000","last_became_peered":"2024-04-06T13:47:34.616973+0000","last_unstale":"2024-04-06T13:47:36.121220+0000","last_undegraded":"2024-04-06T13:47:36.121220+0000","last_fullsized":"2024-04-06T13:47:36.121220+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:13:13.458973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597215+0000","last_change":"2024-04-06T13:47:22.140321+0000","last_active":"2024-04-06T13:47:34.597215+0000","last_peered":"2024-04-06T13:47:34.597215+0000","last_clean":"2024-04-06T13:47:34.597215+0000","last_became_active":"2024-04-06T13:47:22.140027+0000","last_became_peered":"2024-04-06T13:47:22.140027+0000","last_unstale":"2024-04-06T13:47:34.597215+0000","last_undegraded":"2024-04-06T13:47:34.597215+0000","last_fullsized":"2024-04-06T13:47:34.597215+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:49:16.852621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120163+0000","last_change":"2024-04-06T13:47:34.612019+0000","last_active":"2024-04-06T13:47:36.120163+0000","last_peered":"2024-04-06T13:47:36.120163+0000","last_clean":"2024-04-06T13:47:36.120163+0000","last_became_active":"2024-04-06T13:47:34.611550+0000","last_became_peered":"2024-04-06T13:47:34.611550+0000","last_unstale":"2024-04-06T13:47:36.120163+0000","last_undegraded":"2024-04-06T13:47:36.120163+0000","last_fullsized":"2024-04-06T13:47:36.120163+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:33:46.757895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587686+0000","last_change":"2024-04-06T13:47:22.147771+0000","last_active":"2024-04-06T13:47:34.587686+0000","last_peered":"2024-04-06T13:47:34.587686+0000","last_clean":"2024-04-06T13:47:34.587686+0000","last_became_active":"2024-04-06T13:47:22.147512+0000","last_became_peered":"2024-04-06T13:47:22.147512+0000","last_unstale":"2024-04-06T13:47:34.587686+0000","last_undegraded":"2024-04-06T13:47:34.587686+0000","last_fullsized":"2024-04-06T13:47:34.587686+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:25:04.287164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593592+0000","last_change":"2024-04-06T13:47:34.615987+0000","last_active":"2024-04-06T13:47:35.593592+0000","last_peered":"2024-04-06T13:47:35.593592+0000","last_clean":"2024-04-06T13:47:35.593592+0000","last_became_active":"2024-04-06T13:47:34.615180+0000","last_became_peered":"2024-04-06T13:47:34.615180+0000","last_unstale":"2024-04-06T13:47:35.593592+0000","last_undegraded":"2024-04-06T13:47:35.593592+0000","last_fullsized":"2024-04-06T13:47:35.593592+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:28:50.227790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597728+0000","last_change":"2024-04-06T13:47:22.146702+0000","last_active":"2024-04-06T13:47:34.597728+0000","last_peered":"2024-04-06T13:47:34.597728+0000","last_clean":"2024-04-06T13:47:34.597728+0000","last_became_active":"2024-04-06T13:47:22.139334+0000","last_became_peered":"2024-04-06T13:47:22.139334+0000","last_unstale":"2024-04-06T13:47:34.597728+0000","last_undegraded":"2024-04-06T13:47:34.597728+0000","last_fullsized":"2024-04-06T13:47:34.597728+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:36:19.531363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594785+0000","last_change":"2024-04-06T13:47:34.607813+0000","last_active":"2024-04-06T13:47:35.594785+0000","last_peered":"2024-04-06T13:47:35.594785+0000","last_clean":"2024-04-06T13:47:35.594785+0000","last_became_active":"2024-04-06T13:47:34.607531+0000","last_became_peered":"2024-04-06T13:47:34.607531+0000","last_unstale":"2024-04-06T13:47:35.594785+0000","last_undegraded":"2024-04-06T13:47:35.594785+0000","last_fullsized":"2024-04-06T13:47:35.594785+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:21.552700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591158+0000","last_change":"2024-04-06T13:47:22.136209+0000","last_active":"2024-04-06T13:47:34.591158+0000","last_peered":"2024-04-06T13:47:34.591158+0000","last_clean":"2024-04-06T13:47:34.591158+0000","last_became_active":"2024-04-06T13:47:22.135366+0000","last_became_peered":"2024-04-06T13:47:22.135366+0000","last_unstale":"2024-04-06T13:47:34.591158+0000","last_undegraded":"2024-04-06T13:47:34.591158+0000","last_fullsized":"2024-04-06T13:47:34.591158+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:14:48.802617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593242+0000","last_change":"2024-04-06T13:47:34.605433+0000","last_active":"2024-04-06T13:47:35.593242+0000","last_peered":"2024-04-06T13:47:35.593242+0000","last_clean":"2024-04-06T13:47:35.593242+0000","last_became_active":"2024-04-06T13:47:34.605066+0000","last_became_peered":"2024-04-06T13:47:34.605066+0000","last_unstale":"2024-04-06T13:47:35.593242+0000","last_undegraded":"2024-04-06T13:47:35.593242+0000","last_fullsized":"2024-04-06T13:47:35.593242+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:06:24.637773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.598140+0000","last_change":"2024-04-06T13:47:22.130575+0000","last_active":"2024-04-06T13:47:34.598140+0000","last_peered":"2024-04-06T13:47:34.598140+0000","last_clean":"2024-04-06T13:47:34.598140+0000","last_became_active":"2024-04-06T13:47:22.130287+0000","last_became_peered":"2024-04-06T13:47:22.130287+0000","last_unstale":"2024-04-06T13:47:34.598140+0000","last_undegraded":"2024-04-06T13:47:34.598140+0000","last_fullsized":"2024-04-06T13:47:34.598140+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:33:34.282063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593808+0000","last_change":"2024-04-06T13:47:34.602926+0000","last_active":"2024-04-06T13:47:35.593808+0000","last_peered":"2024-04-06T13:47:35.593808+0000","last_clean":"2024-04-06T13:47:35.593808+0000","last_became_active":"2024-04-06T13:47:34.602664+0000","last_became_peered":"2024-04-06T13:47:34.602664+0000","last_unstale":"2024-04-06T13:47:35.593808+0000","last_undegraded":"2024-04-06T13:47:35.593808+0000","last_fullsized":"2024-04-06T13:47:35.593808+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:32:42.519117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591141+0000","last_change":"2024-04-06T13:47:22.137418+0000","last_active":"2024-04-06T13:47:34.591141+0000","last_peered":"2024-04-06T13:47:34.591141+0000","last_clean":"2024-04-06T13:47:34.591141+0000","last_became_active":"2024-04-06T13:47:22.137176+0000","last_became_peered":"2024-04-06T13:47:22.137176+0000","last_unstale":"2024-04-06T13:47:34.591141+0000","last_undegraded":"2024-04-06T13:47:34.591141+0000","last_fullsized":"2024-04-06T13:47:34.591141+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:40:38.388468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120863+0000","last_change":"2024-04-06T13:47:34.596811+0000","last_active":"2024-04-06T13:47:36.120863+0000","last_peered":"2024-04-06T13:47:36.120863+0000","last_clean":"2024-04-06T13:47:36.120863+0000","last_became_active":"2024-04-06T13:47:34.596567+0000","last_became_peered":"2024-04-06T13:47:34.596567+0000","last_unstale":"2024-04-06T13:47:36.120863+0000","last_undegraded":"2024-04-06T13:47:36.120863+0000","last_fullsized":"2024-04-06T13:47:36.120863+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:16:21.941269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.600260+0000","last_change":"2024-04-06T13:47:22.132869+0000","last_active":"2024-04-06T13:47:34.600260+0000","last_peered":"2024-04-06T13:47:34.600260+0000","last_clean":"2024-04-06T13:47:34.600260+0000","last_became_active":"2024-04-06T13:47:22.132613+0000","last_became_peered":"2024-04-06T13:47:22.132613+0000","last_unstale":"2024-04-06T13:47:34.600260+0000","last_undegraded":"2024-04-06T13:47:34.600260+0000","last_fullsized":"2024-04-06T13:47:34.600260+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:38:24.228670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.595770+0000","last_change":"2024-04-06T13:47:34.608817+0000","last_active":"2024-04-06T13:47:35.595770+0000","last_peered":"2024-04-06T13:47:35.595770+0000","last_clean":"2024-04-06T13:47:35.595770+0000","last_became_active":"2024-04-06T13:47:34.607073+0000","last_became_peered":"2024-04-06T13:47:34.607073+0000","last_unstale":"2024-04-06T13:47:35.595770+0000","last_undegraded":"2024-04-06T13:47:35.595770+0000","last_fullsized":"2024-04-06T13:47:35.595770+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:44:32.385590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591973+0000","last_change":"2024-04-06T13:47:22.137179+0000","last_active":"2024-04-06T13:47:34.591973+0000","last_peered":"2024-04-06T13:47:34.591973+0000","last_clean":"2024-04-06T13:47:34.591973+0000","last_became_active":"2024-04-06T13:47:22.136779+0000","last_became_peered":"2024-04-06T13:47:22.136779+0000","last_unstale":"2024-04-06T13:47:34.591973+0000","last_undegraded":"2024-04-06T13:47:34.591973+0000","last_fullsized":"2024-04-06T13:47:34.591973+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:25:40.923568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121720+0000","last_change":"2024-04-06T13:47:34.612281+0000","last_active":"2024-04-06T13:47:36.121720+0000","last_peered":"2024-04-06T13:47:36.121720+0000","last_clean":"2024-04-06T13:47:36.121720+0000","last_became_active":"2024-04-06T13:47:34.611954+0000","last_became_peered":"2024-04-06T13:47:34.611954+0000","last_unstale":"2024-04-06T13:47:36.121720+0000","last_undegraded":"2024-04-06T13:47:36.121720+0000","last_fullsized":"2024-04-06T13:47:36.121720+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:38:36.704502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597173+0000","last_change":"2024-04-06T13:47:22.148394+0000","last_active":"2024-04-06T13:47:34.597173+0000","last_peered":"2024-04-06T13:47:34.597173+0000","last_clean":"2024-04-06T13:47:34.597173+0000","last_became_active":"2024-04-06T13:47:22.148136+0000","last_became_peered":"2024-04-06T13:47:22.148136+0000","last_unstale":"2024-04-06T13:47:34.597173+0000","last_undegraded":"2024-04-06T13:47:34.597173+0000","last_fullsized":"2024-04-06T13:47:34.597173+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:06:09.829859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594166+0000","last_change":"2024-04-06T13:47:34.602824+0000","last_active":"2024-04-06T13:47:35.594166+0000","last_peered":"2024-04-06T13:47:35.594166+0000","last_clean":"2024-04-06T13:47:35.594166+0000","last_became_active":"2024-04-06T13:47:34.602555+0000","last_became_peered":"2024-04-06T13:47:34.602555+0000","last_unstale":"2024-04-06T13:47:35.594166+0000","last_undegraded":"2024-04-06T13:47:35.594166+0000","last_fullsized":"2024-04-06T13:47:35.594166+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:56:48.838776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.601204+0000","last_change":"2024-04-06T13:47:22.143038+0000","last_active":"2024-04-06T13:47:34.601204+0000","last_peered":"2024-04-06T13:47:34.601204+0000","last_clean":"2024-04-06T13:47:34.601204+0000","last_became_active":"2024-04-06T13:47:22.142498+0000","last_became_peered":"2024-04-06T13:47:22.142498+0000","last_unstale":"2024-04-06T13:47:34.601204+0000","last_undegraded":"2024-04-06T13:47:34.601204+0000","last_fullsized":"2024-04-06T13:47:34.601204+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:40.999324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3611,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575985,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56704,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315136,"statfs":{"total":96636764160,"available":96578699264,"internally_reserved":0,"allocated":3518464,"data_stored":2079001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.751,"5min":0.78400000000000003,"15min":0.84899999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58499999999999996,"15min":0.59799999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76000000000000001,"5min":0.80800000000000005,"15min":0.82899999999999996},"last":0.42699999999999999}]},{"osd":1,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.57099999999999995,"15min":0.58799999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.66600000000000004,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59799999999999998,"15min":0.60499999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38900000000000001},"max":{"1min":0.752,"5min":0.83399999999999996,"15min":0.85299999999999998},"last":0.52200000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.57899999999999996,"15min":0.59499999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67700000000000005,"5min":0.81699999999999995,"15min":0.83899999999999997},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.60299999999999998,"15min":0.60599999999999998},"min":{"1min":0.47199999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73699999999999999,"5min":0.79700000000000004,"15min":0.81699999999999995},"last":0.442}]},{"osd":3,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.61599999999999999,"15min":0.61499999999999999},"min":{"1min":0.437,"5min":0.41199999999999998,"15min":0.38500000000000001},"max":{"1min":0.75600000000000001,"5min":0.81100000000000005,"15min":0.86199999999999999},"last":0.496},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59799999999999998,"15min":0.60899999999999999},"min":{"1min":0.42399999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.70199999999999996,"5min":0.83299999999999996,"15min":0.84099999999999997},"last":0.48799999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.45900000000000002,"15min":0.46400000000000002},"min":{"1min":0.33300000000000002,"5min":0.307,"15min":0.30299999999999999},"max":{"1min":0.628,"5min":0.76000000000000001,"15min":0.81599999999999995},"last":0.311},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.54600000000000004,"15min":0.54300000000000004},"min":{"1min":0.39300000000000002,"5min":0.38700000000000001,"15min":0.36799999999999999},"max":{"1min":0.68700000000000006,"5min":0.74299999999999999,"15min":0.82699999999999996},"last":0.45400000000000001}]},{"osd":5,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.49299999999999999,"15min":0.495},"min":{"1min":0.36099999999999999,"5min":0.318,"15min":0.308},"max":{"1min":0.71999999999999997,"5min":0.77100000000000002,"15min":0.78300000000000003},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.46899999999999997,"5min":0.39200000000000002,"15min":0.33600000000000002},"max":{"1min":0.69799999999999995,"5min":0.82099999999999995,"15min":0.85399999999999998},"last":0.35599999999999998}]},{"osd":6,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.39600000000000002,"5min":0.374,"15min":0.34899999999999998},"max":{"1min":0.67700000000000005,"5min":0.755,"15min":0.879},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56899999999999995,"15min":0.58699999999999997},"min":{"1min":0.39200000000000002,"5min":0.378,"15min":0.34799999999999998},"max":{"1min":0.73299999999999998,"5min":0.84899999999999998,"15min":0.89800000000000002},"last":0.40899999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575984,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38072,"kb_used_data":2856,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94333768,"statfs":{"total":96636764160,"available":96597778432,"internally_reserved":0,"allocated":2924544,"data_stored":1481585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.45300000000000001,"5min":0.38200000000000001,"15min":0.32000000000000001},"max":{"1min":0.74399999999999999,"5min":0.79700000000000004,"15min":0.83799999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.48199999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76900000000000002,"5min":0.80000000000000004,"15min":0.81999999999999995},"last":0.54100000000000004}]},{"osd":1,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.626,"15min":0.63},"min":{"1min":0.48999999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.83699999999999997,"15min":0.85799999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64200000000000002,"15min":0.65000000000000002},"min":{"1min":0.54600000000000004,"5min":0.45100000000000001,"15min":0.378},"max":{"1min":0.78900000000000003,"5min":0.81399999999999995,"15min":0.82699999999999996},"last":0.52500000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.495,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.76700000000000002,"5min":0.82299999999999995,"15min":0.82899999999999996},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.59299999999999997},"min":{"1min":0.46400000000000002,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.755,"5min":0.78600000000000003,"15min":0.80700000000000005},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77500000000000002,"5min":0.80100000000000005,"15min":0.82399999999999995},"last":0.495},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64600000000000002,"15min":0.65000000000000002},"min":{"1min":0.51500000000000001,"5min":0.45900000000000002,"15min":0.36099999999999999},"max":{"1min":0.78800000000000003,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.63900000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54500000000000004,"15min":0.53100000000000003},"min":{"1min":0.38100000000000001,"5min":0.36899999999999999,"15min":0.33800000000000002},"max":{"1min":0.72199999999999998,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53500000000000003,"15min":0.54500000000000004},"min":{"1min":0.371,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68799999999999994,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.59599999999999997}]},{"osd":5,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.504,"15min":0.50900000000000001},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":0.60799999999999998,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53100000000000003,"15min":0.53400000000000003},"min":{"1min":0.35499999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74099999999999999,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.625}]},{"osd":7,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71999999999999997,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56699999999999995,"15min":0.57199999999999995},"min":{"1min":0.435,"5min":0.36799999999999999,"15min":0.32800000000000001},"max":{"1min":0.70099999999999996,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.57799999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575983,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60180,"kb_used_data":2816,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311660,"statfs":{"total":96636764160,"available":96575139840,"internally_reserved":0,"allocated":2883584,"data_stored":1476500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.443,"5min":0.432,"15min":0.432},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25700000000000001},"max":{"1min":0.55300000000000005,"5min":0.60999999999999999,"15min":0.69399999999999995},"last":0.22800000000000001}]},{"osd":2,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.42999999999999999,"15min":0.42899999999999999},"min":{"1min":0.30299999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46800000000000003,"15min":0.46899999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.29399999999999998},"max":{"1min":0.69499999999999995,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.28699999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.28799999999999998},"max":{"1min":0.67800000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.26300000000000001},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.66100000000000003},"last":0.313}]},{"osd":4,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.64000000000000001,"15min":0.64900000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.33800000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.97399999999999998},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.65600000000000003,"15min":0.66000000000000003},"min":{"1min":0.45000000000000001,"5min":0.44500000000000001,"15min":0.44400000000000001},"max":{"1min":0.877,"5min":0.94599999999999995,"15min":0.995},"last":0.59699999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.65000000000000002,"15min":0.65600000000000003},"min":{"1min":0.51300000000000001,"5min":0.41299999999999998,"15min":0.40799999999999997},"max":{"1min":0.91800000000000004,"5min":0.96899999999999997,"15min":1.073},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.66800000000000004,"15min":0.67500000000000004},"min":{"1min":0.45500000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.90500000000000003,"5min":0.96499999999999997,"15min":1.0209999999999999},"last":0.54700000000000004}]},{"osd":6,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65100000000000002,"15min":0.66200000000000003},"min":{"1min":0.47699999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":1},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.51400000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":1.036},"last":0.624}]},{"osd":7,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.69799999999999995,"15min":0.69999999999999996},"min":{"1min":0.50600000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89600000000000002,"5min":0.94899999999999995,"15min":1.0649999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.66500000000000004,"15min":0.67300000000000004},"min":{"1min":0.49299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":1.056},"last":0.629}]}]},{"osd":0,"up_from":13,"seq":55834575983,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42728,"kb_used_data":3412,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329112,"statfs":{"total":96636764160,"available":96593010688,"internally_reserved":0,"allocated":3493888,"data_stored":2074517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.40200000000000002,"15min":0.39900000000000002},"min":{"1min":0.315,"5min":0.24299999999999999,"15min":0.23799999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.41499999999999998,"15min":0.42299999999999999},"min":{"1min":0.32400000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55800000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.33100000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.42299999999999999,"15min":0.432},"min":{"1min":0.32100000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.55900000000000005,"5min":0.67900000000000005,"15min":0.70099999999999996},"last":0.317},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.441,"15min":0.45200000000000001},"min":{"1min":0.35899999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60799999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.30299999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.42999999999999999,"15min":0.434},"min":{"1min":0.34399999999999997,"5min":0.27400000000000002,"15min":0.26800000000000002},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.48299999999999998,"15min":0.48599999999999999},"min":{"1min":0.39700000000000002,"5min":0.314,"15min":0.307},"max":{"1min":0.64100000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.35299999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.63100000000000001,"15min":0.64800000000000002},"min":{"1min":0.48299999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":1.0329999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.63900000000000001,"15min":0.65100000000000002},"min":{"1min":0.504,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.45700000000000002}]},{"osd":5,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.628,"15min":0.64800000000000002},"min":{"1min":0.5,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.87,"5min":0.876,"15min":1.0009999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.66300000000000003,"15min":0.67100000000000004},"min":{"1min":0.48599999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.89500000000000002,"5min":0.92800000000000005,"15min":1.052},"last":0.47999999999999998}]},{"osd":6,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67000000000000004,"15min":0.68100000000000005},"min":{"1min":0.52200000000000002,"5min":0.439,"15min":0.435},"max":{"1min":0.94399999999999995,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.66300000000000003,"15min":0.67900000000000005},"min":{"1min":0.49199999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.89500000000000002,"5min":0.96399999999999997,"15min":1.085},"last":0.40799999999999997}]},{"osd":7,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.68500000000000005,"15min":0.70099999999999996},"min":{"1min":0.54000000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91200000000000003,"5min":1.01,"15min":1.121},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.67000000000000004,"15min":0.68100000000000005},"min":{"1min":0.55700000000000005,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":1.0229999999999999},"last":0.49399999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575985,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60204,"kb_used_data":2840,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311636,"statfs":{"total":96636764160,"available":96575115264,"internally_reserved":0,"allocated":2908160,"data_stored":1484765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.38900000000000001,"15min":0.38800000000000001},"min":{"1min":0.307,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42199999999999999,"15min":0.42699999999999999},"min":{"1min":0.28100000000000003,"5min":0.27300000000000002,"15min":0.249},"max":{"1min":0.57099999999999995,"5min":0.58099999999999996,"15min":1.044},"last":0.40200000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40300000000000002,"15min":0.41199999999999998},"min":{"1min":0.28899999999999998,"5min":0.25700000000000001,"15min":0.255},"max":{"1min":0.52300000000000002,"5min":0.56399999999999995,"15min":0.753},"last":0.437},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.44500000000000001,"15min":0.45200000000000001},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.28399999999999997},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.82199999999999995},"last":0.51300000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.46000000000000002,"15min":0.46700000000000003},"min":{"1min":0.36499999999999999,"5min":0.32900000000000001,"15min":0.27800000000000002},"max":{"1min":0.63900000000000001,"5min":0.69699999999999995,"15min":0.70299999999999996},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.45200000000000001,"15min":0.45500000000000002},"min":{"1min":0.308,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.59799999999999998,"5min":0.64500000000000002,"15min":0.68600000000000005},"last":0.504}]},{"osd":4,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.64000000000000001,"15min":0.64500000000000002},"min":{"1min":0.47199999999999998,"5min":0.42199999999999999,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66400000000000003,"15min":0.66800000000000004},"min":{"1min":0.44500000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.88900000000000001}]},{"osd":5,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66800000000000004,"15min":0.67000000000000004},"min":{"1min":0.46100000000000002,"5min":0.46000000000000002,"15min":0.433},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":1.0149999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.66900000000000004,"15min":0.67200000000000004},"min":{"1min":0.52500000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":1.028}]},{"osd":6,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.67300000000000004,"15min":0.67800000000000005},"min":{"1min":0.46999999999999997,"5min":0.46700000000000003,"15min":0.44600000000000001},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":1.091},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69999999999999996,"15min":0.70199999999999996},"min":{"1min":0.50600000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.96099999999999997},"last":0.95699999999999996}]},{"osd":7,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68300000000000005,"15min":0.69199999999999995},"min":{"1min":0.5,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92500000000000004,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.70399999999999996,"15min":0.71599999999999997},"min":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.85299999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575984,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47912,"kb_used_data":2836,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323928,"statfs":{"total":96636764160,"available":96587702272,"internally_reserved":0,"allocated":2904064,"data_stored":1477667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39000000000000001,"15min":0.38900000000000001},"min":{"1min":0.26000000000000001,"5min":0.255,"15min":0.23499999999999999},"max":{"1min":0.55200000000000005,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41199999999999998,"15min":0.41299999999999998},"min":{"1min":0.28799999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.56899999999999995,"5min":0.622,"15min":0.66700000000000004},"last":0.439}]},{"osd":1,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40699999999999997,"15min":0.40600000000000003},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.65700000000000003},"last":0.379},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.438,"15min":0.44400000000000001},"min":{"1min":0.32800000000000001,"5min":0.27500000000000002,"15min":0.26600000000000001},"max":{"1min":0.63,"5min":0.65400000000000003,"15min":0.75800000000000001},"last":0.45400000000000001}]},{"osd":2,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48299999999999998,"15min":0.48599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.5,"15min":0.503},"min":{"1min":0.38600000000000001,"5min":0.34599999999999997,"15min":0.32900000000000001},"max":{"1min":0.67800000000000005,"5min":0.69299999999999995,"15min":0.79300000000000004},"last":0.40999999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.439,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.876,"5min":1.002,"15min":1.018},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66100000000000003,"15min":0.65300000000000002},"min":{"1min":0.497,"5min":0.45900000000000002,"15min":0.375},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.51900000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.95699999999999996,"5min":0.97299999999999998,"15min":0.999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.46200000000000002,"5min":0.437,"15min":0.437},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.60899999999999999}]},{"osd":6,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67500000000000004,"15min":0.66900000000000004},"min":{"1min":0.41499999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.68600000000000005,"15min":0.68400000000000005},"min":{"1min":0.48799999999999999,"5min":0.46700000000000003,"15min":0.41099999999999998},"max":{"1min":0.88800000000000001,"5min":0.96199999999999997,"15min":0.98799999999999999},"last":0.56299999999999994}]},{"osd":7,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68500000000000005,"15min":0.68000000000000005},"min":{"1min":0.46100000000000002,"5min":0.432,"15min":0.40999999999999998},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70299999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.024},"last":0.50800000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575984,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47900,"kb_used_data":2828,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323940,"statfs":{"total":96636764160,"available":96587714560,"internally_reserved":0,"allocated":2895872,"data_stored":1478191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.373,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.76400000000000001,"5min":0.876,"15min":0.88500000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.60499999999999998,"15min":0.60299999999999998},"min":{"1min":0.40200000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77900000000000003,"5min":1.202,"15min":1.202},"last":0.39300000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63,"15min":0.64400000000000002},"min":{"1min":0.439,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.82999999999999996,"5min":1.256,"15min":1.256},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64000000000000001,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80900000000000005,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.47699999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.42499999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.81799999999999995,"5min":0.84599999999999997,"15min":0.877},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.627,"15min":0.621},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88500000000000001,"5min":1.294,"15min":1.294},"last":0.46400000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.44900000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.75,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46899999999999997,"5min":0.373,"15min":0.373},"max":{"1min":0.78000000000000003,"5min":0.90200000000000002,"15min":0.91100000000000003},"last":0.48799999999999999}]},{"osd":5,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.56200000000000006,"15min":0.51600000000000001},"min":{"1min":0.35299999999999998,"5min":0.33200000000000002,"15min":0.32800000000000001},"max":{"1min":0.69799999999999995,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.57899999999999996,"15min":0.54300000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68600000000000005,"5min":4.931,"15min":4.931},"last":0.435}]},{"osd":6,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54000000000000004,"15min":0.53000000000000003},"min":{"1min":0.34100000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72399999999999998,"5min":0.83499999999999996,"15min":0.83899999999999997},"last":0.375},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57899999999999996,"15min":0.56999999999999995},"min":{"1min":0.41599999999999998,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":0.751,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.5}]},{"osd":7,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.60899999999999999,"15min":0.58299999999999996},"min":{"1min":0.36899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.65400000000000003,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.60399999999999998,"15min":0.60199999999999998},"min":{"1min":0.437,"5min":0.40799999999999997,"15min":0.373},"max":{"1min":0.748,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.51300000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575984,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64304,"kb_used_data":2852,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307536,"statfs":{"total":96636764160,"available":96570916864,"internally_reserved":0,"allocated":2920448,"data_stored":1485749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.46400000000000002,"5min":0.34000000000000002,"15min":0.33000000000000002},"max":{"1min":0.70199999999999996,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58999999999999997,"15min":0.59499999999999997},"min":{"1min":0.39700000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.73999999999999999,"5min":0.84699999999999998,"15min":0.89200000000000002},"last":0.747}]},{"osd":1,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.41199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72699999999999998,"5min":0.81299999999999994,"15min":0.81399999999999995},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58599999999999997,"15min":0.58699999999999997},"min":{"1min":0.43099999999999999,"5min":0.39400000000000002,"15min":0.37},"max":{"1min":0.70999999999999996,"5min":0.871,"15min":0.871},"last":0.52700000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60599999999999998,"15min":0.60099999999999998},"min":{"1min":0.39400000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.70999999999999996,"5min":0.79700000000000004,"15min":0.84399999999999997},"last":0.441},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.43099999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.67900000000000005,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.63600000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60499999999999998,"15min":0.59899999999999998},"min":{"1min":0.44700000000000001,"5min":0.40999999999999998,"15min":0.373},"max":{"1min":0.754,"5min":0.81100000000000005,"15min":0.94499999999999995},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63600000000000001,"15min":0.63700000000000001},"min":{"1min":0.48499999999999999,"5min":0.40300000000000002,"15min":0.39000000000000001},"max":{"1min":0.81399999999999995,"5min":0.88200000000000001,"15min":0.91500000000000004},"last":0.65400000000000003}]},{"osd":4,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52500000000000002,"15min":0.505},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.27400000000000002},"max":{"1min":0.746,"5min":0.85699999999999998,"15min":1.0369999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60499999999999998,"15min":0.57699999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.308},"max":{"1min":0.78000000000000003,"5min":0.88800000000000001,"15min":1.1639999999999999},"last":0.76500000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54000000000000004,"15min":0.52400000000000002},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.27900000000000003},"max":{"1min":0.73799999999999999,"5min":0.94999999999999996,"15min":1.0169999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54600000000000004,"15min":0.52800000000000002},"min":{"1min":0.34399999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70999999999999996,"5min":0.878,"15min":1.0009999999999999},"last":0.81899999999999995}]},{"osd":7,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56100000000000005,"15min":0.54500000000000004},"min":{"1min":0.374,"5min":0.33600000000000002,"15min":0.32900000000000001},"max":{"1min":0.70899999999999996,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.66500000000000004,"15min":0.64700000000000002},"min":{"1min":0.45600000000000002,"5min":0.433,"15min":0.35199999999999998},"max":{"1min":0.84799999999999998,"5min":0.92300000000000004,"15min":0.96599999999999997},"last":0.80800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:47:38.726 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:47:38.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.879+0000 7fedcb73c700 1 Processor -- start 2024-04-06T13:47:38.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.880+0000 7fedcb73c700 1 -- start start 2024-04-06T13:47:38.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.880+0000 7fedcb73c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fedc415d7b0 0x7fedc40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.880+0000 7fedcb73c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fedc406a730 0x7fedc406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.880+0000 7fedcb73c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc40b3990 con 0x7fedc415ddf0 2024-04-06T13:47:38.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.880+0000 7fedcb73c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedc406b0f0 con 0x7fedc415d7b0 2024-04-06T13:47:38.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.880+0000 7fedcb73c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc40b3060 con 0x7fedc406a730 2024-04-06T13:47:38.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.881+0000 7fedc94d8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fedc415d7b0 0x7fedc40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.881+0000 7fedc9cd9700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fedc415ddf0 0x7fedc415e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55208/0 (socket says 172.21.15.189:55208) 2024-04-06T13:47:38.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.881+0000 7fedc9cd9700 1 -- 172.21.15.189:0/1575845647 learned_addr learned my addr 172.21.15.189:0/1575845647 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:38.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.881+0000 7fedc94d8700 1 -- 172.21.15.189:0/1575845647 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fedc406a730 msgr2=0x7fedc406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:38.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.881+0000 7fedc94d8700 1 --2- 172.21.15.189:0/1575845647 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fedc406a730 0x7fedc406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.881+0000 7fedc94d8700 1 -- 172.21.15.189:0/1575845647 >> v1:172.21.15.189:6789/0 conn(0x7fedc415ddf0 legacy=0x7fedc415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:38.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.881+0000 7fedc94d8700 1 -- 172.21.15.189:0/1575845647 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc40b3340 con 0x7fedc415d7b0 2024-04-06T13:47:38.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.881+0000 7fedb7fff700 1 -- 172.21.15.189:0/1575845647 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725115444 0 0) 0x7fedc40b3990 con 0x7fedc415ddf0 2024-04-06T13:47:38.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.881+0000 7fedc94d8700 1 --2- 172.21.15.189:0/1575845647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedc415d7b0 0x7fedc40b3e20 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7fedc0009840 tx=0x7fedc0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aec2e04b3d1df721 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:38.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.881+0000 7fedb7fff700 1 -- 172.21.15.189:0/1575845647 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc0011830 con 0x7fedc415d7b0 2024-04-06T13:47:38.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.881+0000 7fedb7fff700 1 -- 172.21.15.189:0/1575845647 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc0011990 con 0x7fedc415d7b0 2024-04-06T13:47:38.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.881+0000 7fedb7fff700 1 -- 172.21.15.189:0/1575845647 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc0011d20 con 0x7fedc415d7b0 2024-04-06T13:47:38.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.882+0000 7fedcb73c700 1 -- 172.21.15.189:0/1575845647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedc415d7b0 msgr2=0x7fedc40b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:38.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.882+0000 7fedcb73c700 1 --2- 172.21.15.189:0/1575845647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedc415d7b0 0x7fedc40b3e20 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7fedc0009840 tx=0x7fedc0009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:38.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.882+0000 7fedcb73c700 1 -- 172.21.15.189:0/1575845647 shutdown_connections 2024-04-06T13:47:38.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.882+0000 7fedcb73c700 1 --2- 172.21.15.189:0/1575845647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedc415d7b0 0x7fedc40b3e20 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.882+0000 7fedcb73c700 1 --2- 172.21.15.189:0/1575845647 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fedc406a730 0x7fedc406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.882+0000 7fedcb73c700 1 -- 172.21.15.189:0/1575845647 >> 172.21.15.189:0/1575845647 conn(0x7fedc40d00b0 msgr2=0x7fedc40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:38.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.882+0000 7fedcb73c700 1 -- 172.21.15.189:0/1575845647 shutdown_connections 2024-04-06T13:47:38.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.882+0000 7fedcb73c700 1 -- 172.21.15.189:0/1575845647 wait complete. 2024-04-06T13:47:38.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.883+0000 7fedcb73c700 1 Processor -- start 2024-04-06T13:47:38.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.883+0000 7fedcb73c700 1 -- start start 2024-04-06T13:47:38.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedcb73c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedc406a730 0x7fedc41f6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedcb73c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedc415d7b0 0x7fedc41f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedcb73c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fedc415ddf0 0x7fedc4152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedcb73c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc4154e20 con 0x7fedc415d7b0 2024-04-06T13:47:38.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedcb73c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc415b450 con 0x7fedc406a730 2024-04-06T13:47:38.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedcb73c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc41f3610 con 0x7fedc415ddf0 2024-04-06T13:47:38.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedc8cd7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedc415d7b0 0x7fedc41f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedc8cd7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedc415d7b0 0x7fedc41f2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55114/0 (socket says 172.21.15.189:55114) 2024-04-06T13:47:38.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedc8cd7700 1 -- 172.21.15.189:0/3359221985 learned_addr learned my addr 172.21.15.189:0/3359221985 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:38.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedc94d8700 1 --2- 172.21.15.189:0/3359221985 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedc406a730 0x7fedc41f6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedc9cd9700 1 --2- 172.21.15.189:0/3359221985 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fedc415ddf0 0x7fedc4152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedc8cd7700 1 -- 172.21.15.189:0/3359221985 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fedc415ddf0 msgr2=0x7fedc4152eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:38.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedc8cd7700 1 --2- 172.21.15.189:0/3359221985 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fedc415ddf0 0x7fedc4152eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedc8cd7700 1 -- 172.21.15.189:0/3359221985 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedc406a730 msgr2=0x7fedc41f6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:38.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedc8cd7700 1 --2- 172.21.15.189:0/3359221985 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedc406a730 0x7fedc41f6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:38.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedc8cd7700 1 -- 172.21.15.189:0/3359221985 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc00094a0 con 0x7fedc415d7b0 2024-04-06T13:47:38.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedc8cd7700 1 --2- 172.21.15.189:0/3359221985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedc415d7b0 0x7fedc41f2dc0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7fedb800b530 tx=0x7fedb800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:38.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedb5ffb700 1 -- 172.21.15.189:0/3359221985 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb8011830 con 0x7fedc415d7b0 2024-04-06T13:47:38.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedb5ffb700 1 -- 172.21.15.189:0/3359221985 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb8011990 con 0x7fedc415d7b0 2024-04-06T13:47:38.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedb5ffb700 1 -- 172.21.15.189:0/3359221985 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb8011d40 con 0x7fedc415d7b0 2024-04-06T13:47:38.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.884+0000 7fedcb73c700 1 -- 172.21.15.189:0/3359221985 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc41f38f0 con 0x7fedc415d7b0 2024-04-06T13:47:38.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.885+0000 7fedcb73c700 1 -- 172.21.15.189:0/3359221985 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc415b710 con 0x7fedc415d7b0 2024-04-06T13:47:38.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.886+0000 7fedcb73c700 1 -- 172.21.15.189:0/3359221985 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc4067bb0 con 0x7fedc415d7b0 2024-04-06T13:47:38.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.886+0000 7fedb5ffb700 1 -- 172.21.15.189:0/3359221985 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fedb801a400 con 0x7fedc415d7b0 2024-04-06T13:47:38.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.886+0000 7fedb5ffb700 1 --2- 172.21.15.189:0/3359221985 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feda806adb0 0x7feda806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:38.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.886+0000 7fedb5ffb700 1 -- 172.21.15.189:0/3359221985 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6965+0+0 (secure 0 0 0) 0x7fedb8089910 con 0x7fedc415d7b0 2024-04-06T13:47:38.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.886+0000 7fedc94d8700 1 --2- 172.21.15.189:0/3359221985 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feda806adb0 0x7feda806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:38.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.887+0000 7fedc94d8700 1 --2- 172.21.15.189:0/3359221985 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feda806adb0 0x7feda806d270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fedc00090b0 tx=0x7fedc0008030 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:38.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:38.891+0000 7fedb5ffb700 1 -- 172.21.15.189:0/3359221985 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fedb80545b0 con 0x7fedc415d7b0 2024-04-06T13:47:39.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.083+0000 7fedcb73c700 1 -- 172.21.15.189:0/3359221985 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fedc415eb40 con 0x7feda806adb0 2024-04-06T13:47:39.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.096+0000 7fedb5ffb700 1 -- 172.21.15.189:0/3359221985 <== mgr.11028 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528975 (secure 0 0 0) 0x7fedc0011750 con 0x7feda806adb0 2024-04-06T13:47:39.103 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:39.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.101+0000 7fedcb73c700 1 -- 172.21.15.189:0/3359221985 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feda806adb0 msgr2=0x7feda806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:39.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.101+0000 7fedcb73c700 1 --2- 172.21.15.189:0/3359221985 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feda806adb0 0x7feda806d270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fedc00090b0 tx=0x7fedc0008030 comp rx=0 tx=0).stop 2024-04-06T13:47:39.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.101+0000 7fedcb73c700 1 -- 172.21.15.189:0/3359221985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedc415d7b0 msgr2=0x7fedc41f2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:39.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.101+0000 7fedcb73c700 1 --2- 172.21.15.189:0/3359221985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedc415d7b0 0x7fedc41f2dc0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7fedb800b530 tx=0x7fedb800b840 comp rx=0 tx=0).stop 2024-04-06T13:47:39.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.101+0000 7fedcb73c700 1 -- 172.21.15.189:0/3359221985 shutdown_connections 2024-04-06T13:47:39.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.101+0000 7fedcb73c700 1 --2- 172.21.15.189:0/3359221985 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feda806adb0 0x7feda806d270 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:39.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.101+0000 7fedcb73c700 1 --2- 172.21.15.189:0/3359221985 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fedc415ddf0 0x7fedc4152eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:39.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.101+0000 7fedcb73c700 1 --2- 172.21.15.189:0/3359221985 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedc406a730 0x7fedc41f6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:39.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.101+0000 7fedcb73c700 1 --2- 172.21.15.189:0/3359221985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedc415d7b0 0x7fedc41f2dc0 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:39.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.101+0000 7fedcb73c700 1 -- 172.21.15.189:0/3359221985 >> 172.21.15.189:0/3359221985 conn(0x7fedc40d00b0 msgr2=0x7fedc4071cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:39.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.101+0000 7fedcb73c700 1 -- 172.21.15.189:0/3359221985 shutdown_connections 2024-04-06T13:47:39.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.101+0000 7fedcb73c700 1 -- 172.21.15.189:0/3359221985 wait complete. 2024-04-06T13:47:39.110 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:47:39.113 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-04-06T13:47:38.412002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596341,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":670,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":427,"ondisk_log_size":427,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":299,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":418004,"kb_used_data":23876,"kb_used_omap":12,"kb_used_meta":393971,"kb_avail":754556716,"statfs":{"total":773094113280,"available":772666077184,"internally_reserved":0,"allocated":24449024,"data_stored":13037975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":403426872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3349,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.237104"},"pg_stats":[{"pgid":"1.0","version":"216'348","reported_seq":1241,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.596051+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:47:34.596051+0000","last_peered":"2024-04-06T13:47:34.596051+0000","last_clean":"2024-04-06T13:47:34.596051+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:47:34.596051+0000","last_undegraded":"2024-04-06T13:47:34.596051+0000","last_fullsized":"2024-04-06T13:47:34.596051+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.588028+0000","last_change":"2024-04-06T13:47:21.714987+0000","last_active":"2024-04-06T13:47:34.588028+0000","last_peered":"2024-04-06T13:47:34.588028+0000","last_clean":"2024-04-06T13:47:34.588028+0000","last_became_active":"2024-04-06T13:47:21.713909+0000","last_became_peered":"2024-04-06T13:47:21.713909+0000","last_unstale":"2024-04-06T13:47:34.588028+0000","last_undegraded":"2024-04-06T13:47:34.588028+0000","last_fullsized":"2024-04-06T13:47:34.588028+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:55:29.107060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.594293+0000","last_change":"2024-04-06T13:47:22.146992+0000","last_active":"2024-04-06T13:47:34.594293+0000","last_peered":"2024-04-06T13:47:34.594293+0000","last_clean":"2024-04-06T13:47:34.594293+0000","last_became_active":"2024-04-06T13:47:22.146728+0000","last_became_peered":"2024-04-06T13:47:22.146728+0000","last_unstale":"2024-04-06T13:47:34.594293+0000","last_undegraded":"2024-04-06T13:47:34.594293+0000","last_fullsized":"2024-04-06T13:47:34.594293+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:12:44.053539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594756+0000","last_change":"2024-04-06T13:47:34.190936+0000","last_active":"2024-04-06T13:47:35.594756+0000","last_peered":"2024-04-06T13:47:35.594756+0000","last_clean":"2024-04-06T13:47:35.594756+0000","last_became_active":"2024-04-06T13:47:34.190577+0000","last_became_peered":"2024-04-06T13:47:34.190577+0000","last_unstale":"2024-04-06T13:47:35.594756+0000","last_undegraded":"2024-04-06T13:47:35.594756+0000","last_fullsized":"2024-04-06T13:47:35.594756+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:26:11.066761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594387+0000","last_change":"2024-04-06T13:47:34.618206+0000","last_active":"2024-04-06T13:47:35.594387+0000","last_peered":"2024-04-06T13:47:35.594387+0000","last_clean":"2024-04-06T13:47:35.594387+0000","last_became_active":"2024-04-06T13:47:34.617974+0000","last_became_peered":"2024-04-06T13:47:34.617974+0000","last_unstale":"2024-04-06T13:47:35.594387+0000","last_undegraded":"2024-04-06T13:47:35.594387+0000","last_fullsized":"2024-04-06T13:47:35.594387+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:00.155859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590403+0000","last_change":"2024-04-06T13:47:21.721753+0000","last_active":"2024-04-06T13:47:34.590403+0000","last_peered":"2024-04-06T13:47:34.590403+0000","last_clean":"2024-04-06T13:47:34.590403+0000","last_became_active":"2024-04-06T13:47:21.721452+0000","last_became_peered":"2024-04-06T13:47:21.721452+0000","last_unstale":"2024-04-06T13:47:34.590403+0000","last_undegraded":"2024-04-06T13:47:34.590403+0000","last_fullsized":"2024-04-06T13:47:34.590403+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:56:43.873348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590945+0000","last_change":"2024-04-06T13:47:22.145029+0000","last_active":"2024-04-06T13:47:34.590945+0000","last_peered":"2024-04-06T13:47:34.590945+0000","last_clean":"2024-04-06T13:47:34.590945+0000","last_became_active":"2024-04-06T13:47:22.144765+0000","last_became_peered":"2024-04-06T13:47:22.144765+0000","last_unstale":"2024-04-06T13:47:34.590945+0000","last_undegraded":"2024-04-06T13:47:34.590945+0000","last_fullsized":"2024-04-06T13:47:34.590945+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:13:00.983141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120579+0000","last_change":"2024-04-06T13:47:34.195377+0000","last_active":"2024-04-06T13:47:36.120579+0000","last_peered":"2024-04-06T13:47:36.120579+0000","last_clean":"2024-04-06T13:47:36.120579+0000","last_became_active":"2024-04-06T13:47:34.195097+0000","last_became_peered":"2024-04-06T13:47:34.195097+0000","last_unstale":"2024-04-06T13:47:36.120579+0000","last_undegraded":"2024-04-06T13:47:36.120579+0000","last_fullsized":"2024-04-06T13:47:36.120579+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:20:15.370970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594819+0000","last_change":"2024-04-06T13:47:34.610261+0000","last_active":"2024-04-06T13:47:35.594819+0000","last_peered":"2024-04-06T13:47:35.594819+0000","last_clean":"2024-04-06T13:47:35.594819+0000","last_became_active":"2024-04-06T13:47:34.607361+0000","last_became_peered":"2024-04-06T13:47:34.607361+0000","last_unstale":"2024-04-06T13:47:35.594819+0000","last_undegraded":"2024-04-06T13:47:35.594819+0000","last_fullsized":"2024-04-06T13:47:35.594819+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:39:35.709825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590171+0000","last_change":"2024-04-06T13:47:21.720234+0000","last_active":"2024-04-06T13:47:34.590171+0000","last_peered":"2024-04-06T13:47:34.590171+0000","last_clean":"2024-04-06T13:47:34.590171+0000","last_became_active":"2024-04-06T13:47:21.719765+0000","last_became_peered":"2024-04-06T13:47:21.719765+0000","last_unstale":"2024-04-06T13:47:34.590171+0000","last_undegraded":"2024-04-06T13:47:34.590171+0000","last_fullsized":"2024-04-06T13:47:34.590171+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:55.001531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.598510+0000","last_change":"2024-04-06T13:47:22.150739+0000","last_active":"2024-04-06T13:47:34.598510+0000","last_peered":"2024-04-06T13:47:34.598510+0000","last_clean":"2024-04-06T13:47:34.598510+0000","last_became_active":"2024-04-06T13:47:22.150305+0000","last_became_peered":"2024-04-06T13:47:22.150305+0000","last_unstale":"2024-04-06T13:47:34.598510+0000","last_undegraded":"2024-04-06T13:47:34.598510+0000","last_fullsized":"2024-04-06T13:47:34.598510+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:18:06.419840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121086+0000","last_change":"2024-04-06T13:47:34.200780+0000","last_active":"2024-04-06T13:47:36.121086+0000","last_peered":"2024-04-06T13:47:36.121086+0000","last_clean":"2024-04-06T13:47:36.121086+0000","last_became_active":"2024-04-06T13:47:34.200474+0000","last_became_peered":"2024-04-06T13:47:34.200474+0000","last_unstale":"2024-04-06T13:47:36.121086+0000","last_undegraded":"2024-04-06T13:47:36.121086+0000","last_fullsized":"2024-04-06T13:47:36.121086+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:10:37.773559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120639+0000","last_change":"2024-04-06T13:47:34.598562+0000","last_active":"2024-04-06T13:47:36.120639+0000","last_peered":"2024-04-06T13:47:36.120639+0000","last_clean":"2024-04-06T13:47:36.120639+0000","last_became_active":"2024-04-06T13:47:34.598306+0000","last_became_peered":"2024-04-06T13:47:34.598306+0000","last_unstale":"2024-04-06T13:47:36.120639+0000","last_undegraded":"2024-04-06T13:47:36.120639+0000","last_fullsized":"2024-04-06T13:47:36.120639+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:55:26.384534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587897+0000","last_change":"2024-04-06T13:47:21.711931+0000","last_active":"2024-04-06T13:47:34.587897+0000","last_peered":"2024-04-06T13:47:34.587897+0000","last_clean":"2024-04-06T13:47:34.587897+0000","last_became_active":"2024-04-06T13:47:21.711592+0000","last_became_peered":"2024-04-06T13:47:21.711592+0000","last_unstale":"2024-04-06T13:47:34.587897+0000","last_undegraded":"2024-04-06T13:47:34.587897+0000","last_fullsized":"2024-04-06T13:47:34.587897+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:38:23.226877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.600111+0000","last_change":"2024-04-06T13:47:22.151110+0000","last_active":"2024-04-06T13:47:34.600111+0000","last_peered":"2024-04-06T13:47:34.600111+0000","last_clean":"2024-04-06T13:47:34.600111+0000","last_became_active":"2024-04-06T13:47:22.150885+0000","last_became_peered":"2024-04-06T13:47:22.150885+0000","last_unstale":"2024-04-06T13:47:34.600111+0000","last_undegraded":"2024-04-06T13:47:34.600111+0000","last_fullsized":"2024-04-06T13:47:34.600111+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:43:52.275558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121334+0000","last_change":"2024-04-06T13:47:34.200209+0000","last_active":"2024-04-06T13:47:36.121334+0000","last_peered":"2024-04-06T13:47:36.121334+0000","last_clean":"2024-04-06T13:47:36.121334+0000","last_became_active":"2024-04-06T13:47:34.198778+0000","last_became_peered":"2024-04-06T13:47:34.198778+0000","last_unstale":"2024-04-06T13:47:36.121334+0000","last_undegraded":"2024-04-06T13:47:36.121334+0000","last_fullsized":"2024-04-06T13:47:36.121334+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:02:31.052025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120410+0000","last_change":"2024-04-06T13:47:34.603150+0000","last_active":"2024-04-06T13:47:36.120410+0000","last_peered":"2024-04-06T13:47:36.120410+0000","last_clean":"2024-04-06T13:47:36.120410+0000","last_became_active":"2024-04-06T13:47:34.602899+0000","last_became_peered":"2024-04-06T13:47:34.602899+0000","last_unstale":"2024-04-06T13:47:36.120410+0000","last_undegraded":"2024-04-06T13:47:36.120410+0000","last_fullsized":"2024-04-06T13:47:36.120410+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:19:18.381473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597883+0000","last_change":"2024-04-06T13:47:21.721538+0000","last_active":"2024-04-06T13:47:34.597883+0000","last_peered":"2024-04-06T13:47:34.597883+0000","last_clean":"2024-04-06T13:47:34.597883+0000","last_became_active":"2024-04-06T13:47:21.721255+0000","last_became_peered":"2024-04-06T13:47:21.721255+0000","last_unstale":"2024-04-06T13:47:34.597883+0000","last_undegraded":"2024-04-06T13:47:34.597883+0000","last_fullsized":"2024-04-06T13:47:34.597883+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:53:48.795224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597888+0000","last_change":"2024-04-06T13:47:22.138725+0000","last_active":"2024-04-06T13:47:34.597888+0000","last_peered":"2024-04-06T13:47:34.597888+0000","last_clean":"2024-04-06T13:47:34.597888+0000","last_became_active":"2024-04-06T13:47:22.131665+0000","last_became_peered":"2024-04-06T13:47:22.131665+0000","last_unstale":"2024-04-06T13:47:34.597888+0000","last_undegraded":"2024-04-06T13:47:34.597888+0000","last_fullsized":"2024-04-06T13:47:34.597888+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:58:16.851577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120645+0000","last_change":"2024-04-06T13:47:34.188669+0000","last_active":"2024-04-06T13:47:36.120645+0000","last_peered":"2024-04-06T13:47:36.120645+0000","last_clean":"2024-04-06T13:47:36.120645+0000","last_became_active":"2024-04-06T13:47:34.188330+0000","last_became_peered":"2024-04-06T13:47:34.188330+0000","last_unstale":"2024-04-06T13:47:36.120645+0000","last_undegraded":"2024-04-06T13:47:36.120645+0000","last_fullsized":"2024-04-06T13:47:36.120645+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:21.894887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.595462+0000","last_change":"2024-04-06T13:47:34.606086+0000","last_active":"2024-04-06T13:47:35.595462+0000","last_peered":"2024-04-06T13:47:35.595462+0000","last_clean":"2024-04-06T13:47:35.595462+0000","last_became_active":"2024-04-06T13:47:34.605599+0000","last_became_peered":"2024-04-06T13:47:34.605599+0000","last_unstale":"2024-04-06T13:47:35.595462+0000","last_undegraded":"2024-04-06T13:47:35.595462+0000","last_fullsized":"2024-04-06T13:47:35.595462+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:51:34.385913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591573+0000","last_change":"2024-04-06T13:47:21.721811+0000","last_active":"2024-04-06T13:47:34.591573+0000","last_peered":"2024-04-06T13:47:34.591573+0000","last_clean":"2024-04-06T13:47:34.591573+0000","last_became_active":"2024-04-06T13:47:21.721528+0000","last_became_peered":"2024-04-06T13:47:21.721528+0000","last_unstale":"2024-04-06T13:47:34.591573+0000","last_undegraded":"2024-04-06T13:47:34.591573+0000","last_fullsized":"2024-04-06T13:47:34.591573+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:54:07.189978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591882+0000","last_change":"2024-04-06T13:47:22.138308+0000","last_active":"2024-04-06T13:47:34.591882+0000","last_peered":"2024-04-06T13:47:34.591882+0000","last_clean":"2024-04-06T13:47:34.591882+0000","last_became_active":"2024-04-06T13:47:22.138038+0000","last_became_peered":"2024-04-06T13:47:22.138038+0000","last_unstale":"2024-04-06T13:47:34.591882+0000","last_undegraded":"2024-04-06T13:47:34.591882+0000","last_fullsized":"2024-04-06T13:47:34.591882+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:23:02.601223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119872+0000","last_change":"2024-04-06T13:47:34.194775+0000","last_active":"2024-04-06T13:47:36.119872+0000","last_peered":"2024-04-06T13:47:36.119872+0000","last_clean":"2024-04-06T13:47:36.119872+0000","last_became_active":"2024-04-06T13:47:34.194481+0000","last_became_peered":"2024-04-06T13:47:34.194481+0000","last_unstale":"2024-04-06T13:47:36.119872+0000","last_undegraded":"2024-04-06T13:47:36.119872+0000","last_fullsized":"2024-04-06T13:47:36.119872+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:55:13.281748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593853+0000","last_change":"2024-04-06T13:47:34.613319+0000","last_active":"2024-04-06T13:47:35.593853+0000","last_peered":"2024-04-06T13:47:35.593853+0000","last_clean":"2024-04-06T13:47:35.593853+0000","last_became_active":"2024-04-06T13:47:34.613055+0000","last_became_peered":"2024-04-06T13:47:34.613055+0000","last_unstale":"2024-04-06T13:47:35.593853+0000","last_undegraded":"2024-04-06T13:47:35.593853+0000","last_fullsized":"2024-04-06T13:47:35.593853+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:04.023963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.596907+0000","last_change":"2024-04-06T13:47:21.711878+0000","last_active":"2024-04-06T13:47:34.596907+0000","last_peered":"2024-04-06T13:47:34.596907+0000","last_clean":"2024-04-06T13:47:34.596907+0000","last_became_active":"2024-04-06T13:47:21.711481+0000","last_became_peered":"2024-04-06T13:47:21.711481+0000","last_unstale":"2024-04-06T13:47:34.596907+0000","last_undegraded":"2024-04-06T13:47:34.596907+0000","last_fullsized":"2024-04-06T13:47:34.596907+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:34:40.754044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587817+0000","last_change":"2024-04-06T13:47:22.136494+0000","last_active":"2024-04-06T13:47:34.587817+0000","last_peered":"2024-04-06T13:47:34.587817+0000","last_clean":"2024-04-06T13:47:34.587817+0000","last_became_active":"2024-04-06T13:47:22.136225+0000","last_became_peered":"2024-04-06T13:47:22.136225+0000","last_unstale":"2024-04-06T13:47:34.587817+0000","last_undegraded":"2024-04-06T13:47:34.587817+0000","last_fullsized":"2024-04-06T13:47:34.587817+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:00:21.961334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120694+0000","last_change":"2024-04-06T13:47:34.202662+0000","last_active":"2024-04-06T13:47:36.120694+0000","last_peered":"2024-04-06T13:47:36.120694+0000","last_clean":"2024-04-06T13:47:36.120694+0000","last_became_active":"2024-04-06T13:47:34.201474+0000","last_became_peered":"2024-04-06T13:47:34.201474+0000","last_unstale":"2024-04-06T13:47:36.120694+0000","last_undegraded":"2024-04-06T13:47:36.120694+0000","last_fullsized":"2024-04-06T13:47:36.120694+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:42:05.118957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593666+0000","last_change":"2024-04-06T13:47:34.617304+0000","last_active":"2024-04-06T13:47:35.593666+0000","last_peered":"2024-04-06T13:47:35.593666+0000","last_clean":"2024-04-06T13:47:35.593666+0000","last_became_active":"2024-04-06T13:47:34.617119+0000","last_became_peered":"2024-04-06T13:47:34.617119+0000","last_unstale":"2024-04-06T13:47:35.593666+0000","last_undegraded":"2024-04-06T13:47:35.593666+0000","last_fullsized":"2024-04-06T13:47:35.593666+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:52:07.725370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.6","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597488+0000","last_change":"2024-04-06T13:47:21.725237+0000","last_active":"2024-04-06T13:47:34.597488+0000","last_peered":"2024-04-06T13:47:34.597488+0000","last_clean":"2024-04-06T13:47:34.597488+0000","last_became_active":"2024-04-06T13:47:21.724977+0000","last_became_peered":"2024-04-06T13:47:21.724977+0000","last_unstale":"2024-04-06T13:47:34.597488+0000","last_undegraded":"2024-04-06T13:47:34.597488+0000","last_fullsized":"2024-04-06T13:47:34.597488+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:13:14.840162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590344+0000","last_change":"2024-04-06T13:47:22.141257+0000","last_active":"2024-04-06T13:47:34.590344+0000","last_peered":"2024-04-06T13:47:34.590344+0000","last_clean":"2024-04-06T13:47:34.590344+0000","last_became_active":"2024-04-06T13:47:22.141005+0000","last_became_peered":"2024-04-06T13:47:22.141005+0000","last_unstale":"2024-04-06T13:47:34.590344+0000","last_undegraded":"2024-04-06T13:47:34.590344+0000","last_fullsized":"2024-04-06T13:47:34.590344+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:15.316599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121277+0000","last_change":"2024-04-06T13:47:34.200644+0000","last_active":"2024-04-06T13:47:36.121277+0000","last_peered":"2024-04-06T13:47:36.121277+0000","last_clean":"2024-04-06T13:47:36.121277+0000","last_became_active":"2024-04-06T13:47:34.200379+0000","last_became_peered":"2024-04-06T13:47:34.200379+0000","last_unstale":"2024-04-06T13:47:36.121277+0000","last_undegraded":"2024-04-06T13:47:36.121277+0000","last_fullsized":"2024-04-06T13:47:36.121277+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:29:46.861936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120258+0000","last_change":"2024-04-06T13:47:34.613965+0000","last_active":"2024-04-06T13:47:36.120258+0000","last_peered":"2024-04-06T13:47:36.120258+0000","last_clean":"2024-04-06T13:47:36.120258+0000","last_became_active":"2024-04-06T13:47:34.613692+0000","last_became_peered":"2024-04-06T13:47:34.613692+0000","last_unstale":"2024-04-06T13:47:36.120258+0000","last_undegraded":"2024-04-06T13:47:36.120258+0000","last_fullsized":"2024-04-06T13:47:36.120258+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:58:55.977176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.588174+0000","last_change":"2024-04-06T13:47:21.719079+0000","last_active":"2024-04-06T13:47:34.588174+0000","last_peered":"2024-04-06T13:47:34.588174+0000","last_clean":"2024-04-06T13:47:34.588174+0000","last_became_active":"2024-04-06T13:47:21.718773+0000","last_became_peered":"2024-04-06T13:47:21.718773+0000","last_unstale":"2024-04-06T13:47:34.588174+0000","last_undegraded":"2024-04-06T13:47:34.588174+0000","last_fullsized":"2024-04-06T13:47:34.588174+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:45:09.803960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597924+0000","last_change":"2024-04-06T13:47:22.147695+0000","last_active":"2024-04-06T13:47:34.597924+0000","last_peered":"2024-04-06T13:47:34.597924+0000","last_clean":"2024-04-06T13:47:34.597924+0000","last_became_active":"2024-04-06T13:47:22.147127+0000","last_became_peered":"2024-04-06T13:47:22.147127+0000","last_unstale":"2024-04-06T13:47:34.597924+0000","last_undegraded":"2024-04-06T13:47:34.597924+0000","last_fullsized":"2024-04-06T13:47:34.597924+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:03:24.991063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594311+0000","last_change":"2024-04-06T13:47:34.185950+0000","last_active":"2024-04-06T13:47:35.594311+0000","last_peered":"2024-04-06T13:47:35.594311+0000","last_clean":"2024-04-06T13:47:35.594311+0000","last_became_active":"2024-04-06T13:47:34.185647+0000","last_became_peered":"2024-04-06T13:47:34.185647+0000","last_unstale":"2024-04-06T13:47:35.594311+0000","last_undegraded":"2024-04-06T13:47:35.594311+0000","last_fullsized":"2024-04-06T13:47:35.594311+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:16:40.354721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119543+0000","last_change":"2024-04-06T13:47:34.612456+0000","last_active":"2024-04-06T13:47:36.119543+0000","last_peered":"2024-04-06T13:47:36.119543+0000","last_clean":"2024-04-06T13:47:36.119543+0000","last_became_active":"2024-04-06T13:47:34.612181+0000","last_became_peered":"2024-04-06T13:47:34.612181+0000","last_unstale":"2024-04-06T13:47:36.119543+0000","last_undegraded":"2024-04-06T13:47:36.119543+0000","last_fullsized":"2024-04-06T13:47:36.119543+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:27.792432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590099+0000","last_change":"2024-04-06T13:47:21.718440+0000","last_active":"2024-04-06T13:47:34.590099+0000","last_peered":"2024-04-06T13:47:34.590099+0000","last_clean":"2024-04-06T13:47:34.590099+0000","last_became_active":"2024-04-06T13:47:21.718175+0000","last_became_peered":"2024-04-06T13:47:21.718175+0000","last_unstale":"2024-04-06T13:47:34.590099+0000","last_undegraded":"2024-04-06T13:47:34.590099+0000","last_fullsized":"2024-04-06T13:47:34.590099+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:34:40.754044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.593523+0000","last_change":"2024-04-06T13:47:22.141028+0000","last_active":"2024-04-06T13:47:34.593523+0000","last_peered":"2024-04-06T13:47:34.593523+0000","last_clean":"2024-04-06T13:47:34.593523+0000","last_became_active":"2024-04-06T13:47:22.140749+0000","last_became_peered":"2024-04-06T13:47:22.140749+0000","last_unstale":"2024-04-06T13:47:34.593523+0000","last_undegraded":"2024-04-06T13:47:34.593523+0000","last_fullsized":"2024-04-06T13:47:34.593523+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:06:22.598094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120022+0000","last_change":"2024-04-06T13:47:34.194754+0000","last_active":"2024-04-06T13:47:36.120022+0000","last_peered":"2024-04-06T13:47:36.120022+0000","last_clean":"2024-04-06T13:47:36.120022+0000","last_became_active":"2024-04-06T13:47:34.194428+0000","last_became_peered":"2024-04-06T13:47:34.194428+0000","last_unstale":"2024-04-06T13:47:36.120022+0000","last_undegraded":"2024-04-06T13:47:36.120022+0000","last_fullsized":"2024-04-06T13:47:36.120022+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:21:48.835480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594687+0000","last_change":"2024-04-06T13:47:34.615494+0000","last_active":"2024-04-06T13:47:35.594687+0000","last_peered":"2024-04-06T13:47:35.594687+0000","last_clean":"2024-04-06T13:47:35.594687+0000","last_became_active":"2024-04-06T13:47:34.615234+0000","last_became_peered":"2024-04-06T13:47:34.615234+0000","last_unstale":"2024-04-06T13:47:35.594687+0000","last_undegraded":"2024-04-06T13:47:35.594687+0000","last_fullsized":"2024-04-06T13:47:35.594687+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:37.873504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.594288+0000","last_change":"2024-04-06T13:47:21.720187+0000","last_active":"2024-04-06T13:47:34.594288+0000","last_peered":"2024-04-06T13:47:34.594288+0000","last_clean":"2024-04-06T13:47:34.594288+0000","last_became_active":"2024-04-06T13:47:21.719896+0000","last_became_peered":"2024-04-06T13:47:21.719896+0000","last_unstale":"2024-04-06T13:47:34.594288+0000","last_undegraded":"2024-04-06T13:47:34.594288+0000","last_fullsized":"2024-04-06T13:47:34.594288+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:11:25.850855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587979+0000","last_change":"2024-04-06T13:47:22.138293+0000","last_active":"2024-04-06T13:47:34.587979+0000","last_peered":"2024-04-06T13:47:34.587979+0000","last_clean":"2024-04-06T13:47:34.587979+0000","last_became_active":"2024-04-06T13:47:22.138051+0000","last_became_peered":"2024-04-06T13:47:22.138051+0000","last_unstale":"2024-04-06T13:47:34.587979+0000","last_undegraded":"2024-04-06T13:47:34.587979+0000","last_fullsized":"2024-04-06T13:47:34.587979+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:52:29.470996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594443+0000","last_change":"2024-04-06T13:47:34.199667+0000","last_active":"2024-04-06T13:47:35.594443+0000","last_peered":"2024-04-06T13:47:35.594443+0000","last_clean":"2024-04-06T13:47:35.594443+0000","last_became_active":"2024-04-06T13:47:34.199405+0000","last_became_peered":"2024-04-06T13:47:34.199405+0000","last_unstale":"2024-04-06T13:47:35.594443+0000","last_undegraded":"2024-04-06T13:47:35.594443+0000","last_fullsized":"2024-04-06T13:47:35.594443+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:44:23.945875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120632+0000","last_change":"2024-04-06T13:47:34.613888+0000","last_active":"2024-04-06T13:47:36.120632+0000","last_peered":"2024-04-06T13:47:36.120632+0000","last_clean":"2024-04-06T13:47:36.120632+0000","last_became_active":"2024-04-06T13:47:34.613634+0000","last_became_peered":"2024-04-06T13:47:34.613634+0000","last_unstale":"2024-04-06T13:47:36.120632+0000","last_undegraded":"2024-04-06T13:47:36.120632+0000","last_fullsized":"2024-04-06T13:47:36.120632+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:30:31.417241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.595033+0000","last_change":"2024-04-06T13:47:21.712418+0000","last_active":"2024-04-06T13:47:34.595033+0000","last_peered":"2024-04-06T13:47:34.595033+0000","last_clean":"2024-04-06T13:47:34.595033+0000","last_became_active":"2024-04-06T13:47:21.712149+0000","last_became_peered":"2024-04-06T13:47:21.712149+0000","last_unstale":"2024-04-06T13:47:34.595033+0000","last_undegraded":"2024-04-06T13:47:34.595033+0000","last_fullsized":"2024-04-06T13:47:34.595033+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:10:24.669252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.599926+0000","last_change":"2024-04-06T13:47:22.141810+0000","last_active":"2024-04-06T13:47:34.599926+0000","last_peered":"2024-04-06T13:47:34.599926+0000","last_clean":"2024-04-06T13:47:34.599926+0000","last_became_active":"2024-04-06T13:47:22.141394+0000","last_became_peered":"2024-04-06T13:47:22.141394+0000","last_unstale":"2024-04-06T13:47:34.599926+0000","last_undegraded":"2024-04-06T13:47:34.599926+0000","last_fullsized":"2024-04-06T13:47:34.599926+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:58:43.501343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121016+0000","last_change":"2024-04-06T13:47:34.198480+0000","last_active":"2024-04-06T13:47:36.121016+0000","last_peered":"2024-04-06T13:47:36.121016+0000","last_clean":"2024-04-06T13:47:36.121016+0000","last_became_active":"2024-04-06T13:47:34.197845+0000","last_became_peered":"2024-04-06T13:47:34.197845+0000","last_unstale":"2024-04-06T13:47:36.121016+0000","last_undegraded":"2024-04-06T13:47:36.121016+0000","last_fullsized":"2024-04-06T13:47:36.121016+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:20:03.197179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121182+0000","last_change":"2024-04-06T13:47:34.614252+0000","last_active":"2024-04-06T13:47:36.121182+0000","last_peered":"2024-04-06T13:47:36.121182+0000","last_clean":"2024-04-06T13:47:36.121182+0000","last_became_active":"2024-04-06T13:47:34.613984+0000","last_became_peered":"2024-04-06T13:47:34.613984+0000","last_unstale":"2024-04-06T13:47:36.121182+0000","last_undegraded":"2024-04-06T13:47:36.121182+0000","last_fullsized":"2024-04-06T13:47:36.121182+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:49:31.353686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.b","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590538+0000","last_change":"2024-04-06T13:47:21.709408+0000","last_active":"2024-04-06T13:47:34.590538+0000","last_peered":"2024-04-06T13:47:34.590538+0000","last_clean":"2024-04-06T13:47:34.590538+0000","last_became_active":"2024-04-06T13:47:21.708906+0000","last_became_peered":"2024-04-06T13:47:21.708906+0000","last_unstale":"2024-04-06T13:47:34.590538+0000","last_undegraded":"2024-04-06T13:47:34.590538+0000","last_fullsized":"2024-04-06T13:47:34.590538+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:09.996520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.595599+0000","last_change":"2024-04-06T13:47:22.143638+0000","last_active":"2024-04-06T13:47:34.595599+0000","last_peered":"2024-04-06T13:47:34.595599+0000","last_clean":"2024-04-06T13:47:34.595599+0000","last_became_active":"2024-04-06T13:47:22.143386+0000","last_became_peered":"2024-04-06T13:47:22.143386+0000","last_unstale":"2024-04-06T13:47:34.595599+0000","last_undegraded":"2024-04-06T13:47:34.595599+0000","last_fullsized":"2024-04-06T13:47:34.595599+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:16:28.885120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120083+0000","last_change":"2024-04-06T13:47:34.195822+0000","last_active":"2024-04-06T13:47:36.120083+0000","last_peered":"2024-04-06T13:47:36.120083+0000","last_clean":"2024-04-06T13:47:36.120083+0000","last_became_active":"2024-04-06T13:47:34.195539+0000","last_became_peered":"2024-04-06T13:47:34.195539+0000","last_unstale":"2024-04-06T13:47:36.120083+0000","last_undegraded":"2024-04-06T13:47:36.120083+0000","last_fullsized":"2024-04-06T13:47:36.120083+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:52.468926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594485+0000","last_change":"2024-04-06T13:47:34.599198+0000","last_active":"2024-04-06T13:47:35.594485+0000","last_peered":"2024-04-06T13:47:35.594485+0000","last_clean":"2024-04-06T13:47:35.594485+0000","last_became_active":"2024-04-06T13:47:34.598897+0000","last_became_peered":"2024-04-06T13:47:34.598897+0000","last_unstale":"2024-04-06T13:47:35.594485+0000","last_undegraded":"2024-04-06T13:47:35.594485+0000","last_fullsized":"2024-04-06T13:47:35.594485+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:59:50.625262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591209+0000","last_change":"2024-04-06T13:47:21.721929+0000","last_active":"2024-04-06T13:47:34.591209+0000","last_peered":"2024-04-06T13:47:34.591209+0000","last_clean":"2024-04-06T13:47:34.591209+0000","last_became_active":"2024-04-06T13:47:21.721667+0000","last_became_peered":"2024-04-06T13:47:21.721667+0000","last_unstale":"2024-04-06T13:47:34.591209+0000","last_undegraded":"2024-04-06T13:47:34.591209+0000","last_fullsized":"2024-04-06T13:47:34.591209+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:49:15.850828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.602579+0000","last_change":"2024-04-06T13:47:22.139250+0000","last_active":"2024-04-06T13:47:34.602579+0000","last_peered":"2024-04-06T13:47:34.602579+0000","last_clean":"2024-04-06T13:47:34.602579+0000","last_became_active":"2024-04-06T13:47:22.138989+0000","last_became_peered":"2024-04-06T13:47:22.138989+0000","last_unstale":"2024-04-06T13:47:34.602579+0000","last_undegraded":"2024-04-06T13:47:34.602579+0000","last_fullsized":"2024-04-06T13:47:34.602579+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:48:14.103260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120427+0000","last_change":"2024-04-06T13:47:34.187384+0000","last_active":"2024-04-06T13:47:36.120427+0000","last_peered":"2024-04-06T13:47:36.120427+0000","last_clean":"2024-04-06T13:47:36.120427+0000","last_became_active":"2024-04-06T13:47:34.186741+0000","last_became_peered":"2024-04-06T13:47:34.186741+0000","last_unstale":"2024-04-06T13:47:36.120427+0000","last_undegraded":"2024-04-06T13:47:36.120427+0000","last_fullsized":"2024-04-06T13:47:36.120427+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:00:33.430936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594502+0000","last_change":"2024-04-06T13:47:34.604474+0000","last_active":"2024-04-06T13:47:35.594502+0000","last_peered":"2024-04-06T13:47:35.594502+0000","last_clean":"2024-04-06T13:47:35.594502+0000","last_became_active":"2024-04-06T13:47:34.603745+0000","last_became_peered":"2024-04-06T13:47:34.603745+0000","last_unstale":"2024-04-06T13:47:35.594502+0000","last_undegraded":"2024-04-06T13:47:35.594502+0000","last_fullsized":"2024-04-06T13:47:35.594502+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:38:37.608413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.d","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597603+0000","last_change":"2024-04-06T13:47:21.714087+0000","last_active":"2024-04-06T13:47:34.597603+0000","last_peered":"2024-04-06T13:47:34.597603+0000","last_clean":"2024-04-06T13:47:34.597603+0000","last_became_active":"2024-04-06T13:47:21.713790+0000","last_became_peered":"2024-04-06T13:47:21.713790+0000","last_unstale":"2024-04-06T13:47:34.597603+0000","last_undegraded":"2024-04-06T13:47:34.597603+0000","last_fullsized":"2024-04-06T13:47:34.597603+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:52:28.469203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.592773+0000","last_change":"2024-04-06T13:47:22.143187+0000","last_active":"2024-04-06T13:47:34.592773+0000","last_peered":"2024-04-06T13:47:34.592773+0000","last_clean":"2024-04-06T13:47:34.592773+0000","last_became_active":"2024-04-06T13:47:22.142878+0000","last_became_peered":"2024-04-06T13:47:22.142878+0000","last_unstale":"2024-04-06T13:47:34.592773+0000","last_undegraded":"2024-04-06T13:47:34.592773+0000","last_fullsized":"2024-04-06T13:47:34.592773+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:27:49.346320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120949+0000","last_change":"2024-04-06T13:47:34.195195+0000","last_active":"2024-04-06T13:47:36.120949+0000","last_peered":"2024-04-06T13:47:36.120949+0000","last_clean":"2024-04-06T13:47:36.120949+0000","last_became_active":"2024-04-06T13:47:34.194951+0000","last_became_peered":"2024-04-06T13:47:34.194951+0000","last_unstale":"2024-04-06T13:47:36.120949+0000","last_undegraded":"2024-04-06T13:47:36.120949+0000","last_fullsized":"2024-04-06T13:47:36.120949+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:12:55.523141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119895+0000","last_change":"2024-04-06T13:47:34.616419+0000","last_active":"2024-04-06T13:47:36.119895+0000","last_peered":"2024-04-06T13:47:36.119895+0000","last_clean":"2024-04-06T13:47:36.119895+0000","last_became_active":"2024-04-06T13:47:34.616079+0000","last_became_peered":"2024-04-06T13:47:34.616079+0000","last_unstale":"2024-04-06T13:47:36.119895+0000","last_undegraded":"2024-04-06T13:47:36.119895+0000","last_fullsized":"2024-04-06T13:47:36.119895+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:29:51.530310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.e","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.600642+0000","last_change":"2024-04-06T13:47:21.723380+0000","last_active":"2024-04-06T13:47:34.600642+0000","last_peered":"2024-04-06T13:47:34.600642+0000","last_clean":"2024-04-06T13:47:34.600642+0000","last_became_active":"2024-04-06T13:47:21.723089+0000","last_became_peered":"2024-04-06T13:47:21.723089+0000","last_unstale":"2024-04-06T13:47:34.600642+0000","last_undegraded":"2024-04-06T13:47:34.600642+0000","last_fullsized":"2024-04-06T13:47:34.600642+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:11:25.850855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591674+0000","last_change":"2024-04-06T13:47:22.135011+0000","last_active":"2024-04-06T13:47:34.591674+0000","last_peered":"2024-04-06T13:47:34.591674+0000","last_clean":"2024-04-06T13:47:34.591674+0000","last_became_active":"2024-04-06T13:47:22.134193+0000","last_became_peered":"2024-04-06T13:47:22.134193+0000","last_unstale":"2024-04-06T13:47:34.591674+0000","last_undegraded":"2024-04-06T13:47:34.591674+0000","last_fullsized":"2024-04-06T13:47:34.591674+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:15:14.777409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120008+0000","last_change":"2024-04-06T13:47:34.194432+0000","last_active":"2024-04-06T13:47:36.120008+0000","last_peered":"2024-04-06T13:47:36.120008+0000","last_clean":"2024-04-06T13:47:36.120008+0000","last_became_active":"2024-04-06T13:47:34.194153+0000","last_became_peered":"2024-04-06T13:47:34.194153+0000","last_unstale":"2024-04-06T13:47:36.120008+0000","last_undegraded":"2024-04-06T13:47:36.120008+0000","last_fullsized":"2024-04-06T13:47:36.120008+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:28:00.815922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594023+0000","last_change":"2024-04-06T13:47:34.618214+0000","last_active":"2024-04-06T13:47:35.594023+0000","last_peered":"2024-04-06T13:47:35.594023+0000","last_clean":"2024-04-06T13:47:35.594023+0000","last_became_active":"2024-04-06T13:47:34.617973+0000","last_became_peered":"2024-04-06T13:47:34.617973+0000","last_unstale":"2024-04-06T13:47:35.594023+0000","last_undegraded":"2024-04-06T13:47:35.594023+0000","last_fullsized":"2024-04-06T13:47:35.594023+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:01:34.086529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.f","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.601303+0000","last_change":"2024-04-06T13:47:21.722333+0000","last_active":"2024-04-06T13:47:34.601303+0000","last_peered":"2024-04-06T13:47:34.601303+0000","last_clean":"2024-04-06T13:47:34.601303+0000","last_became_active":"2024-04-06T13:47:21.722021+0000","last_became_peered":"2024-04-06T13:47:21.722021+0000","last_unstale":"2024-04-06T13:47:34.601303+0000","last_undegraded":"2024-04-06T13:47:34.601303+0000","last_fullsized":"2024-04-06T13:47:34.601303+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:54:05.540582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597829+0000","last_change":"2024-04-06T13:47:22.147277+0000","last_active":"2024-04-06T13:47:34.597829+0000","last_peered":"2024-04-06T13:47:34.597829+0000","last_clean":"2024-04-06T13:47:34.597829+0000","last_became_active":"2024-04-06T13:47:22.146965+0000","last_became_peered":"2024-04-06T13:47:22.146965+0000","last_unstale":"2024-04-06T13:47:34.597829+0000","last_undegraded":"2024-04-06T13:47:34.597829+0000","last_fullsized":"2024-04-06T13:47:34.597829+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:30:03.035386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120127+0000","last_change":"2024-04-06T13:47:34.195388+0000","last_active":"2024-04-06T13:47:36.120127+0000","last_peered":"2024-04-06T13:47:36.120127+0000","last_clean":"2024-04-06T13:47:36.120127+0000","last_became_active":"2024-04-06T13:47:34.195111+0000","last_became_peered":"2024-04-06T13:47:34.195111+0000","last_unstale":"2024-04-06T13:47:36.120127+0000","last_undegraded":"2024-04-06T13:47:36.120127+0000","last_fullsized":"2024-04-06T13:47:36.120127+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:10:37.140647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119882+0000","last_change":"2024-04-06T13:47:34.613796+0000","last_active":"2024-04-06T13:47:36.119882+0000","last_peered":"2024-04-06T13:47:36.119882+0000","last_clean":"2024-04-06T13:47:36.119882+0000","last_became_active":"2024-04-06T13:47:34.613561+0000","last_became_peered":"2024-04-06T13:47:34.613561+0000","last_unstale":"2024-04-06T13:47:36.119882+0000","last_undegraded":"2024-04-06T13:47:36.119882+0000","last_fullsized":"2024-04-06T13:47:36.119882+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:32:08.951962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.10","version":"226'8","reported_seq":23,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590323+0000","last_change":"2024-04-06T13:47:21.721909+0000","last_active":"2024-04-06T13:47:34.590323+0000","last_peered":"2024-04-06T13:47:34.590323+0000","last_clean":"2024-04-06T13:47:34.590323+0000","last_became_active":"2024-04-06T13:47:21.721654+0000","last_became_peered":"2024-04-06T13:47:21.721654+0000","last_unstale":"2024-04-06T13:47:34.590323+0000","last_undegraded":"2024-04-06T13:47:34.590323+0000","last_fullsized":"2024-04-06T13:47:34.590323+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:06:40.698606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.599341+0000","last_change":"2024-04-06T13:47:22.137131+0000","last_active":"2024-04-06T13:47:34.599341+0000","last_peered":"2024-04-06T13:47:34.599341+0000","last_clean":"2024-04-06T13:47:34.599341+0000","last_became_active":"2024-04-06T13:47:22.136845+0000","last_became_peered":"2024-04-06T13:47:22.136845+0000","last_unstale":"2024-04-06T13:47:34.599341+0000","last_undegraded":"2024-04-06T13:47:34.599341+0000","last_fullsized":"2024-04-06T13:47:34.599341+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:58:56.639338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121006+0000","last_change":"2024-04-06T13:47:34.199738+0000","last_active":"2024-04-06T13:47:36.121006+0000","last_peered":"2024-04-06T13:47:36.121006+0000","last_clean":"2024-04-06T13:47:36.121006+0000","last_became_active":"2024-04-06T13:47:34.199475+0000","last_became_peered":"2024-04-06T13:47:34.199475+0000","last_unstale":"2024-04-06T13:47:36.121006+0000","last_undegraded":"2024-04-06T13:47:36.121006+0000","last_fullsized":"2024-04-06T13:47:36.121006+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:16:41.274875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120418+0000","last_change":"2024-04-06T13:47:34.603287+0000","last_active":"2024-04-06T13:47:36.120418+0000","last_peered":"2024-04-06T13:47:36.120418+0000","last_clean":"2024-04-06T13:47:36.120418+0000","last_became_active":"2024-04-06T13:47:34.603009+0000","last_became_peered":"2024-04-06T13:47:34.603009+0000","last_unstale":"2024-04-06T13:47:36.120418+0000","last_undegraded":"2024-04-06T13:47:36.120418+0000","last_fullsized":"2024-04-06T13:47:36.120418+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:33:09.147411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587468+0000","last_change":"2024-04-06T13:47:21.722302+0000","last_active":"2024-04-06T13:47:34.587468+0000","last_peered":"2024-04-06T13:47:34.587468+0000","last_clean":"2024-04-06T13:47:34.587468+0000","last_became_active":"2024-04-06T13:47:21.721839+0000","last_became_peered":"2024-04-06T13:47:21.721839+0000","last_unstale":"2024-04-06T13:47:34.587468+0000","last_undegraded":"2024-04-06T13:47:34.587468+0000","last_fullsized":"2024-04-06T13:47:34.587468+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:27:30.927130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587825+0000","last_change":"2024-04-06T13:47:22.140481+0000","last_active":"2024-04-06T13:47:34.587825+0000","last_peered":"2024-04-06T13:47:34.587825+0000","last_clean":"2024-04-06T13:47:34.587825+0000","last_became_active":"2024-04-06T13:47:22.140216+0000","last_became_peered":"2024-04-06T13:47:22.140216+0000","last_unstale":"2024-04-06T13:47:34.587825+0000","last_undegraded":"2024-04-06T13:47:34.587825+0000","last_fullsized":"2024-04-06T13:47:34.587825+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:10.425285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.595864+0000","last_change":"2024-04-06T13:47:34.187197+0000","last_active":"2024-04-06T13:47:35.595864+0000","last_peered":"2024-04-06T13:47:35.595864+0000","last_clean":"2024-04-06T13:47:35.595864+0000","last_became_active":"2024-04-06T13:47:34.186401+0000","last_became_peered":"2024-04-06T13:47:34.186401+0000","last_unstale":"2024-04-06T13:47:35.595864+0000","last_undegraded":"2024-04-06T13:47:35.595864+0000","last_fullsized":"2024-04-06T13:47:35.595864+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:36:37.561866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120917+0000","last_change":"2024-04-06T13:47:34.611586+0000","last_active":"2024-04-06T13:47:36.120917+0000","last_peered":"2024-04-06T13:47:36.120917+0000","last_clean":"2024-04-06T13:47:36.120917+0000","last_became_active":"2024-04-06T13:47:34.611310+0000","last_became_peered":"2024-04-06T13:47:34.611310+0000","last_unstale":"2024-04-06T13:47:36.120917+0000","last_undegraded":"2024-04-06T13:47:36.120917+0000","last_fullsized":"2024-04-06T13:47:36.120917+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:24:11.679817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591343+0000","last_change":"2024-04-06T13:47:21.709366+0000","last_active":"2024-04-06T13:47:34.591343+0000","last_peered":"2024-04-06T13:47:34.591343+0000","last_clean":"2024-04-06T13:47:34.591343+0000","last_became_active":"2024-04-06T13:47:21.709118+0000","last_became_peered":"2024-04-06T13:47:21.709118+0000","last_unstale":"2024-04-06T13:47:34.591343+0000","last_undegraded":"2024-04-06T13:47:34.591343+0000","last_fullsized":"2024-04-06T13:47:34.591343+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:15:13.775616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591825+0000","last_change":"2024-04-06T13:47:22.137019+0000","last_active":"2024-04-06T13:47:34.591825+0000","last_peered":"2024-04-06T13:47:34.591825+0000","last_clean":"2024-04-06T13:47:34.591825+0000","last_became_active":"2024-04-06T13:47:22.136423+0000","last_became_peered":"2024-04-06T13:47:22.136423+0000","last_unstale":"2024-04-06T13:47:34.591825+0000","last_undegraded":"2024-04-06T13:47:34.591825+0000","last_fullsized":"2024-04-06T13:47:34.591825+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:32:52.203373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"230'6","reported_seq":17,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.595298+0000","last_change":"2024-04-06T13:47:34.197594+0000","last_active":"2024-04-06T13:47:35.595298+0000","last_peered":"2024-04-06T13:47:35.595298+0000","last_clean":"2024-04-06T13:47:35.595298+0000","last_became_active":"2024-04-06T13:47:34.197333+0000","last_became_peered":"2024-04-06T13:47:34.197333+0000","last_unstale":"2024-04-06T13:47:35.595298+0000","last_undegraded":"2024-04-06T13:47:35.595298+0000","last_fullsized":"2024-04-06T13:47:35.595298+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:18:17.889442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121415+0000","last_change":"2024-04-06T13:47:34.611400+0000","last_active":"2024-04-06T13:47:36.121415+0000","last_peered":"2024-04-06T13:47:36.121415+0000","last_clean":"2024-04-06T13:47:36.121415+0000","last_became_active":"2024-04-06T13:47:34.611073+0000","last_became_peered":"2024-04-06T13:47:34.611073+0000","last_unstale":"2024-04-06T13:47:36.121415+0000","last_undegraded":"2024-04-06T13:47:36.121415+0000","last_fullsized":"2024-04-06T13:47:36.121415+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:59:09.115171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.595424+0000","last_change":"2024-04-06T13:47:21.719461+0000","last_active":"2024-04-06T13:47:34.595424+0000","last_peered":"2024-04-06T13:47:34.595424+0000","last_clean":"2024-04-06T13:47:34.595424+0000","last_became_active":"2024-04-06T13:47:21.719215+0000","last_became_peered":"2024-04-06T13:47:21.719215+0000","last_unstale":"2024-04-06T13:47:34.595424+0000","last_undegraded":"2024-04-06T13:47:34.595424+0000","last_fullsized":"2024-04-06T13:47:34.595424+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:00:20.959541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.595429+0000","last_change":"2024-04-06T13:47:22.143097+0000","last_active":"2024-04-06T13:47:34.595429+0000","last_peered":"2024-04-06T13:47:34.595429+0000","last_clean":"2024-04-06T13:47:34.595429+0000","last_became_active":"2024-04-06T13:47:22.141937+0000","last_became_peered":"2024-04-06T13:47:22.141937+0000","last_unstale":"2024-04-06T13:47:34.595429+0000","last_undegraded":"2024-04-06T13:47:34.595429+0000","last_fullsized":"2024-04-06T13:47:34.595429+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:36:26.092264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"230'3","reported_seq":14,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.595391+0000","last_change":"2024-04-06T13:47:34.194044+0000","last_active":"2024-04-06T13:47:35.595391+0000","last_peered":"2024-04-06T13:47:35.595391+0000","last_clean":"2024-04-06T13:47:35.595391+0000","last_became_active":"2024-04-06T13:47:34.193760+0000","last_became_peered":"2024-04-06T13:47:34.193760+0000","last_unstale":"2024-04-06T13:47:35.595391+0000","last_undegraded":"2024-04-06T13:47:35.595391+0000","last_fullsized":"2024-04-06T13:47:35.595391+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:06:21.299461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594553+0000","last_change":"2024-04-06T13:47:34.602134+0000","last_active":"2024-04-06T13:47:35.594553+0000","last_peered":"2024-04-06T13:47:35.594553+0000","last_clean":"2024-04-06T13:47:35.594553+0000","last_became_active":"2024-04-06T13:47:34.601889+0000","last_became_peered":"2024-04-06T13:47:34.601889+0000","last_unstale":"2024-04-06T13:47:35.594553+0000","last_undegraded":"2024-04-06T13:47:35.594553+0000","last_fullsized":"2024-04-06T13:47:35.594553+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:53:41.874072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"227'8","reported_seq":23,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.600457+0000","last_change":"2024-04-06T13:47:21.722629+0000","last_active":"2024-04-06T13:47:34.600457+0000","last_peered":"2024-04-06T13:47:34.600457+0000","last_clean":"2024-04-06T13:47:34.600457+0000","last_became_active":"2024-04-06T13:47:21.722378+0000","last_became_peered":"2024-04-06T13:47:21.722378+0000","last_unstale":"2024-04-06T13:47:34.600457+0000","last_undegraded":"2024-04-06T13:47:34.600457+0000","last_fullsized":"2024-04-06T13:47:34.600457+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:43:36.814977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2813,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590621+0000","last_change":"2024-04-06T13:47:22.144290+0000","last_active":"2024-04-06T13:47:34.590621+0000","last_peered":"2024-04-06T13:47:34.590621+0000","last_clean":"2024-04-06T13:47:34.590621+0000","last_became_active":"2024-04-06T13:47:22.144023+0000","last_became_peered":"2024-04-06T13:47:22.144023+0000","last_unstale":"2024-04-06T13:47:34.590621+0000","last_undegraded":"2024-04-06T13:47:34.590621+0000","last_fullsized":"2024-04-06T13:47:34.590621+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:52:25.950818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119908+0000","last_change":"2024-04-06T13:47:34.190335+0000","last_active":"2024-04-06T13:47:36.119908+0000","last_peered":"2024-04-06T13:47:36.119908+0000","last_clean":"2024-04-06T13:47:36.119908+0000","last_became_active":"2024-04-06T13:47:34.190093+0000","last_became_peered":"2024-04-06T13:47:34.190093+0000","last_unstale":"2024-04-06T13:47:36.119908+0000","last_undegraded":"2024-04-06T13:47:36.119908+0000","last_fullsized":"2024-04-06T13:47:36.119908+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:10:33.916859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121064+0000","last_change":"2024-04-06T13:47:34.605328+0000","last_active":"2024-04-06T13:47:36.121064+0000","last_peered":"2024-04-06T13:47:36.121064+0000","last_clean":"2024-04-06T13:47:36.121064+0000","last_became_active":"2024-04-06T13:47:34.604770+0000","last_became_peered":"2024-04-06T13:47:34.604770+0000","last_unstale":"2024-04-06T13:47:36.121064+0000","last_undegraded":"2024-04-06T13:47:36.121064+0000","last_fullsized":"2024-04-06T13:47:36.121064+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:19:18.381473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.15","version":"226'3","reported_seq":18,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.588014+0000","last_change":"2024-04-06T13:47:21.714316+0000","last_active":"2024-04-06T13:47:34.588014+0000","last_peered":"2024-04-06T13:47:34.588014+0000","last_clean":"2024-04-06T13:47:34.588014+0000","last_became_active":"2024-04-06T13:47:21.713975+0000","last_became_peered":"2024-04-06T13:47:21.713975+0000","last_unstale":"2024-04-06T13:47:34.588014+0000","last_undegraded":"2024-04-06T13:47:34.588014+0000","last_fullsized":"2024-04-06T13:47:34.588014+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:33:33.280270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587735+0000","last_change":"2024-04-06T13:47:22.145873+0000","last_active":"2024-04-06T13:47:34.587735+0000","last_peered":"2024-04-06T13:47:34.587735+0000","last_clean":"2024-04-06T13:47:34.587735+0000","last_became_active":"2024-04-06T13:47:22.145602+0000","last_became_peered":"2024-04-06T13:47:22.145602+0000","last_unstale":"2024-04-06T13:47:34.587735+0000","last_undegraded":"2024-04-06T13:47:34.587735+0000","last_fullsized":"2024-04-06T13:47:34.587735+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:24:02.415549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120361+0000","last_change":"2024-04-06T13:47:34.202761+0000","last_active":"2024-04-06T13:47:36.120361+0000","last_peered":"2024-04-06T13:47:36.120361+0000","last_clean":"2024-04-06T13:47:36.120361+0000","last_became_active":"2024-04-06T13:47:34.202482+0000","last_became_peered":"2024-04-06T13:47:34.202482+0000","last_unstale":"2024-04-06T13:47:36.120361+0000","last_undegraded":"2024-04-06T13:47:36.120361+0000","last_fullsized":"2024-04-06T13:47:36.120361+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:15:26.247011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119533+0000","last_change":"2024-04-06T13:47:34.611811+0000","last_active":"2024-04-06T13:47:36.119533+0000","last_peered":"2024-04-06T13:47:36.119533+0000","last_clean":"2024-04-06T13:47:36.119533+0000","last_became_active":"2024-04-06T13:47:34.611383+0000","last_became_peered":"2024-04-06T13:47:34.611383+0000","last_unstale":"2024-04-06T13:47:36.119533+0000","last_undegraded":"2024-04-06T13:47:36.119533+0000","last_fullsized":"2024-04-06T13:47:36.119533+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:54:04.532210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.598422+0000","last_change":"2024-04-06T13:47:21.722544+0000","last_active":"2024-04-06T13:47:34.598422+0000","last_peered":"2024-04-06T13:47:34.598422+0000","last_clean":"2024-04-06T13:47:34.598422+0000","last_became_active":"2024-04-06T13:47:21.722242+0000","last_became_peered":"2024-04-06T13:47:21.722242+0000","last_unstale":"2024-04-06T13:47:34.598422+0000","last_undegraded":"2024-04-06T13:47:34.598422+0000","last_fullsized":"2024-04-06T13:47:34.598422+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:14:53.056289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590272+0000","last_change":"2024-04-06T13:47:22.136160+0000","last_active":"2024-04-06T13:47:34.590272+0000","last_peered":"2024-04-06T13:47:34.590272+0000","last_clean":"2024-04-06T13:47:34.590272+0000","last_became_active":"2024-04-06T13:47:22.135869+0000","last_became_peered":"2024-04-06T13:47:22.135869+0000","last_unstale":"2024-04-06T13:47:34.590272+0000","last_undegraded":"2024-04-06T13:47:34.590272+0000","last_fullsized":"2024-04-06T13:47:34.590272+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:05:17.896292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"230'8","reported_seq":19,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121500+0000","last_change":"2024-04-06T13:47:34.200748+0000","last_active":"2024-04-06T13:47:36.121500+0000","last_peered":"2024-04-06T13:47:36.121500+0000","last_clean":"2024-04-06T13:47:36.121500+0000","last_became_active":"2024-04-06T13:47:34.200487+0000","last_became_peered":"2024-04-06T13:47:34.200487+0000","last_unstale":"2024-04-06T13:47:36.121500+0000","last_undegraded":"2024-04-06T13:47:36.121500+0000","last_fullsized":"2024-04-06T13:47:36.121500+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:14:37.551778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120256+0000","last_change":"2024-04-06T13:47:34.601764+0000","last_active":"2024-04-06T13:47:36.120256+0000","last_peered":"2024-04-06T13:47:36.120256+0000","last_clean":"2024-04-06T13:47:36.120256+0000","last_became_active":"2024-04-06T13:47:34.601523+0000","last_became_peered":"2024-04-06T13:47:34.601523+0000","last_unstale":"2024-04-06T13:47:36.120256+0000","last_undegraded":"2024-04-06T13:47:36.120256+0000","last_fullsized":"2024-04-06T13:47:36.120256+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:06:22.305691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.593491+0000","last_change":"2024-04-06T13:47:21.722601+0000","last_active":"2024-04-06T13:47:34.593491+0000","last_peered":"2024-04-06T13:47:34.593491+0000","last_clean":"2024-04-06T13:47:34.593491+0000","last_became_active":"2024-04-06T13:47:21.722328+0000","last_became_peered":"2024-04-06T13:47:21.722328+0000","last_unstale":"2024-04-06T13:47:34.593491+0000","last_undegraded":"2024-04-06T13:47:34.593491+0000","last_fullsized":"2024-04-06T13:47:34.593491+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:30:35.611598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.594614+0000","last_change":"2024-04-06T13:47:22.132142+0000","last_active":"2024-04-06T13:47:34.594614+0000","last_peered":"2024-04-06T13:47:34.594614+0000","last_clean":"2024-04-06T13:47:34.594614+0000","last_became_active":"2024-04-06T13:47:22.131874+0000","last_became_peered":"2024-04-06T13:47:22.131874+0000","last_unstale":"2024-04-06T13:47:34.594614+0000","last_undegraded":"2024-04-06T13:47:34.594614+0000","last_fullsized":"2024-04-06T13:47:34.594614+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:14:26.082176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593478+0000","last_change":"2024-04-06T13:47:34.191145+0000","last_active":"2024-04-06T13:47:35.593478+0000","last_peered":"2024-04-06T13:47:35.593478+0000","last_clean":"2024-04-06T13:47:35.593478+0000","last_became_active":"2024-04-06T13:47:34.190823+0000","last_became_peered":"2024-04-06T13:47:34.190823+0000","last_unstale":"2024-04-06T13:47:35.593478+0000","last_undegraded":"2024-04-06T13:47:35.593478+0000","last_fullsized":"2024-04-06T13:47:35.593478+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:55:13.281748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593885+0000","last_change":"2024-04-06T13:47:34.603387+0000","last_active":"2024-04-06T13:47:35.593885+0000","last_peered":"2024-04-06T13:47:35.593885+0000","last_clean":"2024-04-06T13:47:35.593885+0000","last_became_active":"2024-04-06T13:47:34.603141+0000","last_became_peered":"2024-04-06T13:47:34.603141+0000","last_unstale":"2024-04-06T13:47:35.593885+0000","last_undegraded":"2024-04-06T13:47:35.593885+0000","last_fullsized":"2024-04-06T13:47:35.593885+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:00.587625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590221+0000","last_change":"2024-04-06T13:47:21.715138+0000","last_active":"2024-04-06T13:47:34.590221+0000","last_peered":"2024-04-06T13:47:34.590221+0000","last_clean":"2024-04-06T13:47:34.590221+0000","last_became_active":"2024-04-06T13:47:21.714839+0000","last_became_peered":"2024-04-06T13:47:21.714839+0000","last_unstale":"2024-04-06T13:47:34.590221+0000","last_undegraded":"2024-04-06T13:47:34.590221+0000","last_fullsized":"2024-04-06T13:47:34.590221+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:02:18.580630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.588090+0000","last_change":"2024-04-06T13:47:22.140985+0000","last_active":"2024-04-06T13:47:34.588090+0000","last_peered":"2024-04-06T13:47:34.588090+0000","last_clean":"2024-04-06T13:47:34.588090+0000","last_became_active":"2024-04-06T13:47:22.140751+0000","last_became_peered":"2024-04-06T13:47:22.140751+0000","last_unstale":"2024-04-06T13:47:34.588090+0000","last_undegraded":"2024-04-06T13:47:34.588090+0000","last_fullsized":"2024-04-06T13:47:34.588090+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:20:03.901368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120753+0000","last_change":"2024-04-06T13:47:34.198817+0000","last_active":"2024-04-06T13:47:36.120753+0000","last_peered":"2024-04-06T13:47:36.120753+0000","last_clean":"2024-04-06T13:47:36.120753+0000","last_became_active":"2024-04-06T13:47:34.198515+0000","last_became_peered":"2024-04-06T13:47:34.198515+0000","last_unstale":"2024-04-06T13:47:36.120753+0000","last_undegraded":"2024-04-06T13:47:36.120753+0000","last_fullsized":"2024-04-06T13:47:36.120753+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:32:41.940060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594168+0000","last_change":"2024-04-06T13:47:34.599400+0000","last_active":"2024-04-06T13:47:35.594168+0000","last_peered":"2024-04-06T13:47:35.594168+0000","last_clean":"2024-04-06T13:47:35.594168+0000","last_became_active":"2024-04-06T13:47:34.599155+0000","last_became_peered":"2024-04-06T13:47:34.599155+0000","last_unstale":"2024-04-06T13:47:35.594168+0000","last_undegraded":"2024-04-06T13:47:35.594168+0000","last_fullsized":"2024-04-06T13:47:35.594168+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:05:31.482786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":15,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.596154+0000","last_change":"2024-04-06T13:47:21.713514+0000","last_active":"2024-04-06T13:47:34.596154+0000","last_peered":"2024-04-06T13:47:34.596154+0000","last_clean":"2024-04-06T13:47:34.596154+0000","last_became_active":"2024-04-06T13:47:21.713230+0000","last_became_peered":"2024-04-06T13:47:21.713230+0000","last_unstale":"2024-04-06T13:47:34.596154+0000","last_undegraded":"2024-04-06T13:47:34.596154+0000","last_fullsized":"2024-04-06T13:47:34.596154+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:29:38.052685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.596525+0000","last_change":"2024-04-06T13:47:22.141742+0000","last_active":"2024-04-06T13:47:34.596525+0000","last_peered":"2024-04-06T13:47:34.596525+0000","last_clean":"2024-04-06T13:47:34.596525+0000","last_became_active":"2024-04-06T13:47:22.141302+0000","last_became_peered":"2024-04-06T13:47:22.141302+0000","last_unstale":"2024-04-06T13:47:34.596525+0000","last_undegraded":"2024-04-06T13:47:34.596525+0000","last_fullsized":"2024-04-06T13:47:34.596525+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:24.049383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"230'4","reported_seq":15,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120441+0000","last_change":"2024-04-06T13:47:34.190504+0000","last_active":"2024-04-06T13:47:36.120441+0000","last_peered":"2024-04-06T13:47:36.120441+0000","last_clean":"2024-04-06T13:47:36.120441+0000","last_became_active":"2024-04-06T13:47:34.189775+0000","last_became_peered":"2024-04-06T13:47:34.189775+0000","last_unstale":"2024-04-06T13:47:36.120441+0000","last_undegraded":"2024-04-06T13:47:36.120441+0000","last_fullsized":"2024-04-06T13:47:36.120441+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:23:14.070825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594110+0000","last_change":"2024-04-06T13:47:34.608005+0000","last_active":"2024-04-06T13:47:35.594110+0000","last_peered":"2024-04-06T13:47:35.594110+0000","last_clean":"2024-04-06T13:47:35.594110+0000","last_became_active":"2024-04-06T13:47:34.607728+0000","last_became_peered":"2024-04-06T13:47:34.607728+0000","last_unstale":"2024-04-06T13:47:35.594110+0000","last_undegraded":"2024-04-06T13:47:35.594110+0000","last_fullsized":"2024-04-06T13:47:35.594110+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:47:13.207002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1b","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.598875+0000","last_change":"2024-04-06T13:47:21.714435+0000","last_active":"2024-04-06T13:47:34.598875+0000","last_peered":"2024-04-06T13:47:34.598875+0000","last_clean":"2024-04-06T13:47:34.598875+0000","last_became_active":"2024-04-06T13:47:21.714154+0000","last_became_peered":"2024-04-06T13:47:21.714154+0000","last_unstale":"2024-04-06T13:47:34.598875+0000","last_undegraded":"2024-04-06T13:47:34.598875+0000","last_fullsized":"2024-04-06T13:47:34.598875+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:25.144968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.596726+0000","last_change":"2024-04-06T13:47:22.139248+0000","last_active":"2024-04-06T13:47:34.596726+0000","last_peered":"2024-04-06T13:47:34.596726+0000","last_clean":"2024-04-06T13:47:34.596726+0000","last_became_active":"2024-04-06T13:47:22.139085+0000","last_became_peered":"2024-04-06T13:47:22.139085+0000","last_unstale":"2024-04-06T13:47:34.596726+0000","last_undegraded":"2024-04-06T13:47:34.596726+0000","last_fullsized":"2024-04-06T13:47:34.596726+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:41:53.649355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"230'2","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.595182+0000","last_change":"2024-04-06T13:47:34.196041+0000","last_active":"2024-04-06T13:47:35.595182+0000","last_peered":"2024-04-06T13:47:35.595182+0000","last_clean":"2024-04-06T13:47:35.595182+0000","last_became_active":"2024-04-06T13:47:34.195744+0000","last_became_peered":"2024-04-06T13:47:34.195744+0000","last_unstale":"2024-04-06T13:47:35.595182+0000","last_undegraded":"2024-04-06T13:47:35.595182+0000","last_fullsized":"2024-04-06T13:47:35.595182+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:58:54.970945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120640+0000","last_change":"2024-04-06T13:47:34.612984+0000","last_active":"2024-04-06T13:47:36.120640+0000","last_peered":"2024-04-06T13:47:36.120640+0000","last_clean":"2024-04-06T13:47:36.120640+0000","last_became_active":"2024-04-06T13:47:34.612623+0000","last_became_peered":"2024-04-06T13:47:34.612623+0000","last_unstale":"2024-04-06T13:47:36.120640+0000","last_undegraded":"2024-04-06T13:47:36.120640+0000","last_fullsized":"2024-04-06T13:47:36.120640+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:25:53.399400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1c","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.588368+0000","last_change":"2024-04-06T13:47:21.713494+0000","last_active":"2024-04-06T13:47:34.588368+0000","last_peered":"2024-04-06T13:47:34.588368+0000","last_clean":"2024-04-06T13:47:34.588368+0000","last_became_active":"2024-04-06T13:47:21.713155+0000","last_became_peered":"2024-04-06T13:47:21.713155+0000","last_unstale":"2024-04-06T13:47:34.588368+0000","last_undegraded":"2024-04-06T13:47:34.588368+0000","last_fullsized":"2024-04-06T13:47:34.588368+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:43:51.273765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590695+0000","last_change":"2024-04-06T13:47:22.146835+0000","last_active":"2024-04-06T13:47:34.590695+0000","last_peered":"2024-04-06T13:47:34.590695+0000","last_clean":"2024-04-06T13:47:34.590695+0000","last_became_active":"2024-04-06T13:47:22.146565+0000","last_became_peered":"2024-04-06T13:47:22.146565+0000","last_unstale":"2024-04-06T13:47:34.590695+0000","last_undegraded":"2024-04-06T13:47:34.590695+0000","last_fullsized":"2024-04-06T13:47:34.590695+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:01:10.026234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594904+0000","last_change":"2024-04-06T13:47:34.193924+0000","last_active":"2024-04-06T13:47:35.594904+0000","last_peered":"2024-04-06T13:47:35.594904+0000","last_clean":"2024-04-06T13:47:35.594904+0000","last_became_active":"2024-04-06T13:47:34.193475+0000","last_became_peered":"2024-04-06T13:47:34.193475+0000","last_unstale":"2024-04-06T13:47:35.594904+0000","last_undegraded":"2024-04-06T13:47:35.594904+0000","last_fullsized":"2024-04-06T13:47:35.594904+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:33:08.141180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121335+0000","last_change":"2024-04-06T13:47:34.612503+0000","last_active":"2024-04-06T13:47:36.121335+0000","last_peered":"2024-04-06T13:47:36.121335+0000","last_clean":"2024-04-06T13:47:36.121335+0000","last_became_active":"2024-04-06T13:47:34.609994+0000","last_became_peered":"2024-04-06T13:47:34.609994+0000","last_unstale":"2024-04-06T13:47:36.121335+0000","last_undegraded":"2024-04-06T13:47:36.121335+0000","last_fullsized":"2024-04-06T13:47:36.121335+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:15:06.533915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"226'2","reported_seq":17,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597510+0000","last_change":"2024-04-06T13:47:21.714467+0000","last_active":"2024-04-06T13:47:34.597510+0000","last_peered":"2024-04-06T13:47:34.597510+0000","last_clean":"2024-04-06T13:47:34.597510+0000","last_became_active":"2024-04-06T13:47:21.714208+0000","last_became_peered":"2024-04-06T13:47:21.714208+0000","last_unstale":"2024-04-06T13:47:34.597510+0000","last_undegraded":"2024-04-06T13:47:34.597510+0000","last_fullsized":"2024-04-06T13:47:34.597510+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:09.423492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597362+0000","last_change":"2024-04-06T13:47:22.139445+0000","last_active":"2024-04-06T13:47:34.597362+0000","last_peered":"2024-04-06T13:47:34.597362+0000","last_clean":"2024-04-06T13:47:34.597362+0000","last_became_active":"2024-04-06T13:47:22.138882+0000","last_became_peered":"2024-04-06T13:47:22.138882+0000","last_unstale":"2024-04-06T13:47:34.597362+0000","last_undegraded":"2024-04-06T13:47:34.597362+0000","last_fullsized":"2024-04-06T13:47:34.597362+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:45:57.790454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"230'1","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120282+0000","last_change":"2024-04-06T13:47:34.199071+0000","last_active":"2024-04-06T13:47:36.120282+0000","last_peered":"2024-04-06T13:47:36.120282+0000","last_clean":"2024-04-06T13:47:36.120282+0000","last_became_active":"2024-04-06T13:47:34.198865+0000","last_became_peered":"2024-04-06T13:47:34.198865+0000","last_unstale":"2024-04-06T13:47:36.120282+0000","last_undegraded":"2024-04-06T13:47:36.120282+0000","last_fullsized":"2024-04-06T13:47:36.120282+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:52:40.940598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593654+0000","last_change":"2024-04-06T13:47:34.604731+0000","last_active":"2024-04-06T13:47:35.593654+0000","last_peered":"2024-04-06T13:47:35.593654+0000","last_clean":"2024-04-06T13:47:35.593654+0000","last_became_active":"2024-04-06T13:47:34.604478+0000","last_became_peered":"2024-04-06T13:47:34.604478+0000","last_unstale":"2024-04-06T13:47:35.593654+0000","last_undegraded":"2024-04-06T13:47:35.593654+0000","last_fullsized":"2024-04-06T13:47:35.593654+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:56:51.094042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1e","version":"226'1","reported_seq":16,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.594143+0000","last_change":"2024-04-06T13:47:21.721394+0000","last_active":"2024-04-06T13:47:34.594143+0000","last_peered":"2024-04-06T13:47:34.594143+0000","last_clean":"2024-04-06T13:47:34.594143+0000","last_became_active":"2024-04-06T13:47:21.721093+0000","last_became_peered":"2024-04-06T13:47:21.721093+0000","last_unstale":"2024-04-06T13:47:34.594143+0000","last_undegraded":"2024-04-06T13:47:34.594143+0000","last_fullsized":"2024-04-06T13:47:34.594143+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:30:58.296350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591369+0000","last_change":"2024-04-06T13:47:22.136299+0000","last_active":"2024-04-06T13:47:34.591369+0000","last_peered":"2024-04-06T13:47:34.591369+0000","last_clean":"2024-04-06T13:47:34.591369+0000","last_became_active":"2024-04-06T13:47:22.136046+0000","last_became_peered":"2024-04-06T13:47:22.136046+0000","last_unstale":"2024-04-06T13:47:34.591369+0000","last_undegraded":"2024-04-06T13:47:34.591369+0000","last_fullsized":"2024-04-06T13:47:34.591369+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:19:51.727577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593715+0000","last_change":"2024-04-06T13:47:34.184612+0000","last_active":"2024-04-06T13:47:35.593715+0000","last_peered":"2024-04-06T13:47:35.593715+0000","last_clean":"2024-04-06T13:47:35.593715+0000","last_became_active":"2024-04-06T13:47:34.184034+0000","last_became_peered":"2024-04-06T13:47:34.184034+0000","last_unstale":"2024-04-06T13:47:35.593715+0000","last_undegraded":"2024-04-06T13:47:35.593715+0000","last_fullsized":"2024-04-06T13:47:35.593715+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:36.867273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119714+0000","last_change":"2024-04-06T13:47:34.600193+0000","last_active":"2024-04-06T13:47:36.119714+0000","last_peered":"2024-04-06T13:47:36.119714+0000","last_clean":"2024-04-06T13:47:36.119714+0000","last_became_active":"2024-04-06T13:47:34.599954+0000","last_became_peered":"2024-04-06T13:47:34.599954+0000","last_unstale":"2024-04-06T13:47:36.119714+0000","last_undegraded":"2024-04-06T13:47:36.119714+0000","last_fullsized":"2024-04-06T13:47:36.119714+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:19:08.553269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1f","version":"226'4","reported_seq":19,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.596866+0000","last_change":"2024-04-06T13:47:21.722633+0000","last_active":"2024-04-06T13:47:34.596866+0000","last_peered":"2024-04-06T13:47:34.596866+0000","last_clean":"2024-04-06T13:47:34.596866+0000","last_became_active":"2024-04-06T13:47:21.722356+0000","last_became_peered":"2024-04-06T13:47:21.722356+0000","last_unstale":"2024-04-06T13:47:34.596866+0000","last_undegraded":"2024-04-06T13:47:34.596866+0000","last_fullsized":"2024-04-06T13:47:34.596866+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:20.694450+0000","last_clean_scrub_stamp":"2024-04-06T13:47:20.694450+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:55.001531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.595629+0000","last_change":"2024-04-06T13:47:22.139965+0000","last_active":"2024-04-06T13:47:34.595629+0000","last_peered":"2024-04-06T13:47:34.595629+0000","last_clean":"2024-04-06T13:47:34.595629+0000","last_became_active":"2024-04-06T13:47:22.139645+0000","last_became_peered":"2024-04-06T13:47:22.139645+0000","last_unstale":"2024-04-06T13:47:34.595629+0000","last_undegraded":"2024-04-06T13:47:34.595629+0000","last_fullsized":"2024-04-06T13:47:34.595629+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:32:56.671578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119644+0000","last_change":"2024-04-06T13:47:34.199704+0000","last_active":"2024-04-06T13:47:36.119644+0000","last_peered":"2024-04-06T13:47:36.119644+0000","last_clean":"2024-04-06T13:47:36.119644+0000","last_became_active":"2024-04-06T13:47:34.199440+0000","last_became_peered":"2024-04-06T13:47:34.199440+0000","last_unstale":"2024-04-06T13:47:36.119644+0000","last_undegraded":"2024-04-06T13:47:36.119644+0000","last_fullsized":"2024-04-06T13:47:36.119644+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:33.165845+0000","last_clean_scrub_stamp":"2024-04-06T13:47:33.165845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:13:12.452743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120301+0000","last_change":"2024-04-06T13:47:34.603979+0000","last_active":"2024-04-06T13:47:36.120301+0000","last_peered":"2024-04-06T13:47:36.120301+0000","last_clean":"2024-04-06T13:47:36.120301+0000","last_became_active":"2024-04-06T13:47:34.603726+0000","last_became_peered":"2024-04-06T13:47:34.603726+0000","last_unstale":"2024-04-06T13:47:36.120301+0000","last_undegraded":"2024-04-06T13:47:36.120301+0000","last_fullsized":"2024-04-06T13:47:36.120301+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:14:30.470482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.595529+0000","last_change":"2024-04-06T13:47:22.132331+0000","last_active":"2024-04-06T13:47:34.595529+0000","last_peered":"2024-04-06T13:47:34.595529+0000","last_clean":"2024-04-06T13:47:34.595529+0000","last_became_active":"2024-04-06T13:47:22.132068+0000","last_became_peered":"2024-04-06T13:47:22.132068+0000","last_unstale":"2024-04-06T13:47:34.595529+0000","last_undegraded":"2024-04-06T13:47:34.595529+0000","last_fullsized":"2024-04-06T13:47:34.595529+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:45:10.805753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593609+0000","last_change":"2024-04-06T13:47:34.616158+0000","last_active":"2024-04-06T13:47:35.593609+0000","last_peered":"2024-04-06T13:47:35.593609+0000","last_clean":"2024-04-06T13:47:35.593609+0000","last_became_active":"2024-04-06T13:47:34.615519+0000","last_became_peered":"2024-04-06T13:47:34.615519+0000","last_unstale":"2024-04-06T13:47:35.593609+0000","last_undegraded":"2024-04-06T13:47:35.593609+0000","last_fullsized":"2024-04-06T13:47:35.593609+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:46:21.375825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.595689+0000","last_change":"2024-04-06T13:47:22.141102+0000","last_active":"2024-04-06T13:47:34.595689+0000","last_peered":"2024-04-06T13:47:34.595689+0000","last_clean":"2024-04-06T13:47:34.595689+0000","last_became_active":"2024-04-06T13:47:22.140831+0000","last_became_peered":"2024-04-06T13:47:22.140831+0000","last_unstale":"2024-04-06T13:47:34.595689+0000","last_undegraded":"2024-04-06T13:47:34.595689+0000","last_fullsized":"2024-04-06T13:47:34.595689+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:27:31.928923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120955+0000","last_change":"2024-04-06T13:47:34.611700+0000","last_active":"2024-04-06T13:47:36.120955+0000","last_peered":"2024-04-06T13:47:36.120955+0000","last_clean":"2024-04-06T13:47:36.120955+0000","last_became_active":"2024-04-06T13:47:34.611406+0000","last_became_peered":"2024-04-06T13:47:34.611406+0000","last_unstale":"2024-04-06T13:47:36.120955+0000","last_undegraded":"2024-04-06T13:47:36.120955+0000","last_fullsized":"2024-04-06T13:47:36.120955+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:32:07.103859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.596434+0000","last_change":"2024-04-06T13:47:22.138788+0000","last_active":"2024-04-06T13:47:34.596434+0000","last_peered":"2024-04-06T13:47:34.596434+0000","last_clean":"2024-04-06T13:47:34.596434+0000","last_became_active":"2024-04-06T13:47:22.138480+0000","last_became_peered":"2024-04-06T13:47:22.138480+0000","last_unstale":"2024-04-06T13:47:34.596434+0000","last_undegraded":"2024-04-06T13:47:34.596434+0000","last_fullsized":"2024-04-06T13:47:34.596434+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:40.999324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593525+0000","last_change":"2024-04-06T13:47:34.618097+0000","last_active":"2024-04-06T13:47:35.593525+0000","last_peered":"2024-04-06T13:47:35.593525+0000","last_clean":"2024-04-06T13:47:35.593525+0000","last_became_active":"2024-04-06T13:47:34.617829+0000","last_became_peered":"2024-04-06T13:47:34.617829+0000","last_unstale":"2024-04-06T13:47:35.593525+0000","last_undegraded":"2024-04-06T13:47:35.593525+0000","last_fullsized":"2024-04-06T13:47:35.593525+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:01:17.865971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587883+0000","last_change":"2024-04-06T13:47:22.139159+0000","last_active":"2024-04-06T13:47:34.587883+0000","last_peered":"2024-04-06T13:47:34.587883+0000","last_clean":"2024-04-06T13:47:34.587883+0000","last_became_active":"2024-04-06T13:47:22.138916+0000","last_became_peered":"2024-04-06T13:47:22.138916+0000","last_unstale":"2024-04-06T13:47:34.587883+0000","last_undegraded":"2024-04-06T13:47:34.587883+0000","last_fullsized":"2024-04-06T13:47:34.587883+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:58:23.128672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119480+0000","last_change":"2024-04-06T13:47:34.603181+0000","last_active":"2024-04-06T13:47:36.119480+0000","last_peered":"2024-04-06T13:47:36.119480+0000","last_clean":"2024-04-06T13:47:36.119480+0000","last_became_active":"2024-04-06T13:47:34.602932+0000","last_became_peered":"2024-04-06T13:47:34.602932+0000","last_unstale":"2024-04-06T13:47:36.119480+0000","last_undegraded":"2024-04-06T13:47:36.119480+0000","last_fullsized":"2024-04-06T13:47:36.119480+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:33:04.679206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594463+0000","last_change":"2024-04-06T13:47:34.614325+0000","last_active":"2024-04-06T13:47:35.594463+0000","last_peered":"2024-04-06T13:47:35.594463+0000","last_clean":"2024-04-06T13:47:35.594463+0000","last_became_active":"2024-04-06T13:47:34.613988+0000","last_became_peered":"2024-04-06T13:47:34.613988+0000","last_unstale":"2024-04-06T13:47:35.594463+0000","last_undegraded":"2024-04-06T13:47:35.594463+0000","last_fullsized":"2024-04-06T13:47:35.594463+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:06:12.976308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591125+0000","last_change":"2024-04-06T13:47:22.139194+0000","last_active":"2024-04-06T13:47:34.591125+0000","last_peered":"2024-04-06T13:47:34.591125+0000","last_clean":"2024-04-06T13:47:34.591125+0000","last_became_active":"2024-04-06T13:47:22.138677+0000","last_became_peered":"2024-04-06T13:47:22.138677+0000","last_unstale":"2024-04-06T13:47:34.591125+0000","last_undegraded":"2024-04-06T13:47:34.591125+0000","last_fullsized":"2024-04-06T13:47:34.591125+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:36:38.719202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.595821+0000","last_change":"2024-04-06T13:47:34.608117+0000","last_active":"2024-04-06T13:47:35.595821+0000","last_peered":"2024-04-06T13:47:35.595821+0000","last_clean":"2024-04-06T13:47:35.595821+0000","last_became_active":"2024-04-06T13:47:34.607867+0000","last_became_peered":"2024-04-06T13:47:34.607867+0000","last_unstale":"2024-04-06T13:47:35.595821+0000","last_undegraded":"2024-04-06T13:47:35.595821+0000","last_fullsized":"2024-04-06T13:47:35.595821+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:44:43.484577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591899+0000","last_change":"2024-04-06T13:47:22.147177+0000","last_active":"2024-04-06T13:47:34.591899+0000","last_peered":"2024-04-06T13:47:34.591899+0000","last_clean":"2024-04-06T13:47:34.591899+0000","last_became_active":"2024-04-06T13:47:22.146634+0000","last_became_peered":"2024-04-06T13:47:22.146634+0000","last_unstale":"2024-04-06T13:47:34.591899+0000","last_undegraded":"2024-04-06T13:47:34.591899+0000","last_fullsized":"2024-04-06T13:47:34.591899+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:16:29.805273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.594048+0000","last_change":"2024-04-06T13:47:22.144335+0000","last_active":"2024-04-06T13:47:34.594048+0000","last_peered":"2024-04-06T13:47:34.594048+0000","last_clean":"2024-04-06T13:47:34.594048+0000","last_became_active":"2024-04-06T13:47:22.144006+0000","last_became_peered":"2024-04-06T13:47:22.144006+0000","last_unstale":"2024-04-06T13:47:34.594048+0000","last_undegraded":"2024-04-06T13:47:34.594048+0000","last_fullsized":"2024-04-06T13:47:34.594048+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:02:19.582423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120763+0000","last_change":"2024-04-06T13:47:34.613172+0000","last_active":"2024-04-06T13:47:36.120763+0000","last_peered":"2024-04-06T13:47:36.120763+0000","last_clean":"2024-04-06T13:47:36.120763+0000","last_became_active":"2024-04-06T13:47:34.612958+0000","last_became_peered":"2024-04-06T13:47:34.612958+0000","last_unstale":"2024-04-06T13:47:36.120763+0000","last_undegraded":"2024-04-06T13:47:36.120763+0000","last_fullsized":"2024-04-06T13:47:36.120763+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:18:18.895673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120281+0000","last_change":"2024-04-06T13:47:34.611841+0000","last_active":"2024-04-06T13:47:36.120281+0000","last_peered":"2024-04-06T13:47:36.120281+0000","last_clean":"2024-04-06T13:47:36.120281+0000","last_became_active":"2024-04-06T13:47:34.611230+0000","last_became_peered":"2024-04-06T13:47:34.611230+0000","last_unstale":"2024-04-06T13:47:36.120281+0000","last_undegraded":"2024-04-06T13:47:36.120281+0000","last_fullsized":"2024-04-06T13:47:36.120281+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:55:42.584686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587853+0000","last_change":"2024-04-06T13:47:22.133305+0000","last_active":"2024-04-06T13:47:34.587853+0000","last_peered":"2024-04-06T13:47:34.587853+0000","last_clean":"2024-04-06T13:47:34.587853+0000","last_became_active":"2024-04-06T13:47:22.132897+0000","last_became_peered":"2024-04-06T13:47:22.132897+0000","last_unstale":"2024-04-06T13:47:34.587853+0000","last_undegraded":"2024-04-06T13:47:34.587853+0000","last_fullsized":"2024-04-06T13:47:34.587853+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:10:25.671045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120175+0000","last_change":"2024-04-06T13:47:34.614504+0000","last_active":"2024-04-06T13:47:36.120175+0000","last_peered":"2024-04-06T13:47:36.120175+0000","last_clean":"2024-04-06T13:47:36.120175+0000","last_became_active":"2024-04-06T13:47:34.614254+0000","last_became_peered":"2024-04-06T13:47:34.614254+0000","last_unstale":"2024-04-06T13:47:36.120175+0000","last_undegraded":"2024-04-06T13:47:36.120175+0000","last_fullsized":"2024-04-06T13:47:36.120175+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:52:34.891491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590235+0000","last_change":"2024-04-06T13:47:22.139336+0000","last_active":"2024-04-06T13:47:34.590235+0000","last_peered":"2024-04-06T13:47:34.590235+0000","last_clean":"2024-04-06T13:47:34.590235+0000","last_became_active":"2024-04-06T13:47:22.139092+0000","last_became_peered":"2024-04-06T13:47:22.139092+0000","last_unstale":"2024-04-06T13:47:34.590235+0000","last_undegraded":"2024-04-06T13:47:34.590235+0000","last_fullsized":"2024-04-06T13:47:34.590235+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:55:13.445775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593910+0000","last_change":"2024-04-06T13:47:34.602857+0000","last_active":"2024-04-06T13:47:35.593910+0000","last_peered":"2024-04-06T13:47:35.593910+0000","last_clean":"2024-04-06T13:47:35.593910+0000","last_became_active":"2024-04-06T13:47:34.602608+0000","last_became_peered":"2024-04-06T13:47:34.602608+0000","last_unstale":"2024-04-06T13:47:35.593910+0000","last_undegraded":"2024-04-06T13:47:35.593910+0000","last_fullsized":"2024-04-06T13:47:35.593910+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:09:41.768103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597775+0000","last_change":"2024-04-06T13:47:22.139078+0000","last_active":"2024-04-06T13:47:34.597775+0000","last_peered":"2024-04-06T13:47:34.597775+0000","last_clean":"2024-04-06T13:47:34.597775+0000","last_became_active":"2024-04-06T13:47:22.138825+0000","last_became_peered":"2024-04-06T13:47:22.138825+0000","last_unstale":"2024-04-06T13:47:34.597775+0000","last_undegraded":"2024-04-06T13:47:34.597775+0000","last_fullsized":"2024-04-06T13:47:34.597775+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:14:17.994649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594332+0000","last_change":"2024-04-06T13:47:34.602833+0000","last_active":"2024-04-06T13:47:35.594332+0000","last_peered":"2024-04-06T13:47:35.594332+0000","last_clean":"2024-04-06T13:47:35.594332+0000","last_became_active":"2024-04-06T13:47:34.602554+0000","last_became_peered":"2024-04-06T13:47:34.602554+0000","last_unstale":"2024-04-06T13:47:35.594332+0000","last_undegraded":"2024-04-06T13:47:35.594332+0000","last_fullsized":"2024-04-06T13:47:35.594332+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:11:42.747001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597461+0000","last_change":"2024-04-06T13:47:22.139139+0000","last_active":"2024-04-06T13:47:34.597461+0000","last_peered":"2024-04-06T13:47:34.597461+0000","last_clean":"2024-04-06T13:47:34.597461+0000","last_became_active":"2024-04-06T13:47:22.138505+0000","last_became_peered":"2024-04-06T13:47:22.138505+0000","last_unstale":"2024-04-06T13:47:34.597461+0000","last_undegraded":"2024-04-06T13:47:34.597461+0000","last_fullsized":"2024-04-06T13:47:34.597461+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:07:04.253792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119850+0000","last_change":"2024-04-06T13:47:34.596373+0000","last_active":"2024-04-06T13:47:36.119850+0000","last_peered":"2024-04-06T13:47:36.119850+0000","last_clean":"2024-04-06T13:47:36.119850+0000","last_became_active":"2024-04-06T13:47:34.596027+0000","last_became_peered":"2024-04-06T13:47:34.596027+0000","last_unstale":"2024-04-06T13:47:36.119850+0000","last_undegraded":"2024-04-06T13:47:36.119850+0000","last_fullsized":"2024-04-06T13:47:36.119850+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:13:13.458973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.602762+0000","last_change":"2024-04-06T13:47:22.141672+0000","last_active":"2024-04-06T13:47:34.602762+0000","last_peered":"2024-04-06T13:47:34.602762+0000","last_clean":"2024-04-06T13:47:34.602762+0000","last_became_active":"2024-04-06T13:47:22.141369+0000","last_became_peered":"2024-04-06T13:47:22.141369+0000","last_unstale":"2024-04-06T13:47:34.602762+0000","last_undegraded":"2024-04-06T13:47:34.602762+0000","last_fullsized":"2024-04-06T13:47:34.602762+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:27:56.909726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594378+0000","last_change":"2024-04-06T13:47:34.613931+0000","last_active":"2024-04-06T13:47:35.594378+0000","last_peered":"2024-04-06T13:47:35.594378+0000","last_clean":"2024-04-06T13:47:35.594378+0000","last_became_active":"2024-04-06T13:47:34.613197+0000","last_became_peered":"2024-04-06T13:47:34.613197+0000","last_unstale":"2024-04-06T13:47:35.594378+0000","last_undegraded":"2024-04-06T13:47:35.594378+0000","last_fullsized":"2024-04-06T13:47:35.594378+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:45:59.689026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.598146+0000","last_change":"2024-04-06T13:47:22.140812+0000","last_active":"2024-04-06T13:47:34.598146+0000","last_peered":"2024-04-06T13:47:34.598146+0000","last_clean":"2024-04-06T13:47:34.598146+0000","last_became_active":"2024-04-06T13:47:22.139010+0000","last_became_peered":"2024-04-06T13:47:22.139010+0000","last_unstale":"2024-04-06T13:47:34.598146+0000","last_undegraded":"2024-04-06T13:47:34.598146+0000","last_fullsized":"2024-04-06T13:47:34.598146+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:50:16.774388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120995+0000","last_change":"2024-04-06T13:47:34.606907+0000","last_active":"2024-04-06T13:47:36.120995+0000","last_peered":"2024-04-06T13:47:36.120995+0000","last_clean":"2024-04-06T13:47:36.120995+0000","last_became_active":"2024-04-06T13:47:34.606638+0000","last_became_peered":"2024-04-06T13:47:34.606638+0000","last_unstale":"2024-04-06T13:47:36.120995+0000","last_undegraded":"2024-04-06T13:47:36.120995+0000","last_fullsized":"2024-04-06T13:47:36.120995+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:33:04.679206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.592276+0000","last_change":"2024-04-06T13:47:22.145887+0000","last_active":"2024-04-06T13:47:34.592276+0000","last_peered":"2024-04-06T13:47:34.592276+0000","last_clean":"2024-04-06T13:47:34.592276+0000","last_became_active":"2024-04-06T13:47:22.141744+0000","last_became_peered":"2024-04-06T13:47:22.141744+0000","last_unstale":"2024-04-06T13:47:34.592276+0000","last_undegraded":"2024-04-06T13:47:34.592276+0000","last_fullsized":"2024-04-06T13:47:34.592276+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:40:34.810688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.595318+0000","last_change":"2024-04-06T13:47:34.607104+0000","last_active":"2024-04-06T13:47:35.595318+0000","last_peered":"2024-04-06T13:47:35.595318+0000","last_clean":"2024-04-06T13:47:35.595318+0000","last_became_active":"2024-04-06T13:47:34.606862+0000","last_became_peered":"2024-04-06T13:47:34.606862+0000","last_unstale":"2024-04-06T13:47:35.595318+0000","last_undegraded":"2024-04-06T13:47:35.595318+0000","last_fullsized":"2024-04-06T13:47:35.595318+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:48:41.996008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587816+0000","last_change":"2024-04-06T13:47:22.133302+0000","last_active":"2024-04-06T13:47:34.587816+0000","last_peered":"2024-04-06T13:47:34.587816+0000","last_clean":"2024-04-06T13:47:34.587816+0000","last_became_active":"2024-04-06T13:47:22.132902+0000","last_became_peered":"2024-04-06T13:47:22.132902+0000","last_unstale":"2024-04-06T13:47:34.587816+0000","last_undegraded":"2024-04-06T13:47:34.587816+0000","last_fullsized":"2024-04-06T13:47:34.587816+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:55:56.178154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.595060+0000","last_change":"2024-04-06T13:47:34.610455+0000","last_active":"2024-04-06T13:47:35.595060+0000","last_peered":"2024-04-06T13:47:35.595060+0000","last_clean":"2024-04-06T13:47:35.595060+0000","last_became_active":"2024-04-06T13:47:34.608587+0000","last_became_peered":"2024-04-06T13:47:34.608587+0000","last_unstale":"2024-04-06T13:47:35.595060+0000","last_undegraded":"2024-04-06T13:47:35.595060+0000","last_fullsized":"2024-04-06T13:47:35.595060+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:10:34.923090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587775+0000","last_change":"2024-04-06T13:47:22.148647+0000","last_active":"2024-04-06T13:47:34.587775+0000","last_peered":"2024-04-06T13:47:34.587775+0000","last_clean":"2024-04-06T13:47:34.587775+0000","last_became_active":"2024-04-06T13:47:22.148404+0000","last_became_peered":"2024-04-06T13:47:22.148404+0000","last_unstale":"2024-04-06T13:47:34.587775+0000","last_undegraded":"2024-04-06T13:47:34.587775+0000","last_fullsized":"2024-04-06T13:47:34.587775+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:38:04.045660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121608+0000","last_change":"2024-04-06T13:47:34.612490+0000","last_active":"2024-04-06T13:47:36.121608+0000","last_peered":"2024-04-06T13:47:36.121608+0000","last_clean":"2024-04-06T13:47:36.121608+0000","last_became_active":"2024-04-06T13:47:34.609937+0000","last_became_peered":"2024-04-06T13:47:34.609937+0000","last_unstale":"2024-04-06T13:47:36.121608+0000","last_undegraded":"2024-04-06T13:47:36.121608+0000","last_fullsized":"2024-04-06T13:47:36.121608+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:55:14.287979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.590142+0000","last_change":"2024-04-06T13:47:22.142723+0000","last_active":"2024-04-06T13:47:34.590142+0000","last_peered":"2024-04-06T13:47:34.590142+0000","last_clean":"2024-04-06T13:47:34.590142+0000","last_became_active":"2024-04-06T13:47:22.142456+0000","last_became_peered":"2024-04-06T13:47:22.142456+0000","last_unstale":"2024-04-06T13:47:34.590142+0000","last_undegraded":"2024-04-06T13:47:34.590142+0000","last_fullsized":"2024-04-06T13:47:34.590142+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:24:10.160194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593391+0000","last_change":"2024-04-06T13:47:34.614963+0000","last_active":"2024-04-06T13:47:35.593391+0000","last_peered":"2024-04-06T13:47:35.593391+0000","last_clean":"2024-04-06T13:47:35.593391+0000","last_became_active":"2024-04-06T13:47:34.614666+0000","last_became_peered":"2024-04-06T13:47:34.614666+0000","last_unstale":"2024-04-06T13:47:35.593391+0000","last_undegraded":"2024-04-06T13:47:35.593391+0000","last_fullsized":"2024-04-06T13:47:35.593391+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:42:37.496332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.594188+0000","last_change":"2024-04-06T13:47:22.139469+0000","last_active":"2024-04-06T13:47:34.594188+0000","last_peered":"2024-04-06T13:47:34.594188+0000","last_clean":"2024-04-06T13:47:34.594188+0000","last_became_active":"2024-04-06T13:47:22.139177+0000","last_became_peered":"2024-04-06T13:47:22.139177+0000","last_unstale":"2024-04-06T13:47:34.594188+0000","last_undegraded":"2024-04-06T13:47:34.594188+0000","last_fullsized":"2024-04-06T13:47:34.594188+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:12:44.053539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593424+0000","last_change":"2024-04-06T13:47:34.602271+0000","last_active":"2024-04-06T13:47:35.593424+0000","last_peered":"2024-04-06T13:47:35.593424+0000","last_clean":"2024-04-06T13:47:35.593424+0000","last_became_active":"2024-04-06T13:47:34.601348+0000","last_became_peered":"2024-04-06T13:47:34.601348+0000","last_unstale":"2024-04-06T13:47:35.593424+0000","last_undegraded":"2024-04-06T13:47:35.593424+0000","last_fullsized":"2024-04-06T13:47:35.593424+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:27:02.196316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597547+0000","last_change":"2024-04-06T13:47:22.139295+0000","last_active":"2024-04-06T13:47:34.597547+0000","last_peered":"2024-04-06T13:47:34.597547+0000","last_clean":"2024-04-06T13:47:34.597547+0000","last_became_active":"2024-04-06T13:47:22.138518+0000","last_became_peered":"2024-04-06T13:47:22.138518+0000","last_unstale":"2024-04-06T13:47:34.597547+0000","last_undegraded":"2024-04-06T13:47:34.597547+0000","last_fullsized":"2024-04-06T13:47:34.597547+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:03:39.808430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.119223+0000","last_change":"2024-04-06T13:47:34.601270+0000","last_active":"2024-04-06T13:47:36.119223+0000","last_peered":"2024-04-06T13:47:36.119223+0000","last_clean":"2024-04-06T13:47:36.119223+0000","last_became_active":"2024-04-06T13:47:34.601018+0000","last_became_peered":"2024-04-06T13:47:34.601018+0000","last_unstale":"2024-04-06T13:47:36.119223+0000","last_undegraded":"2024-04-06T13:47:36.119223+0000","last_fullsized":"2024-04-06T13:47:36.119223+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:36:38.568097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.600213+0000","last_change":"2024-04-06T13:47:22.150089+0000","last_active":"2024-04-06T13:47:34.600213+0000","last_peered":"2024-04-06T13:47:34.600213+0000","last_clean":"2024-04-06T13:47:34.600213+0000","last_became_active":"2024-04-06T13:47:22.149867+0000","last_became_peered":"2024-04-06T13:47:22.149867+0000","last_unstale":"2024-04-06T13:47:34.600213+0000","last_undegraded":"2024-04-06T13:47:34.600213+0000","last_fullsized":"2024-04-06T13:47:34.600213+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:55:30.108853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594327+0000","last_change":"2024-04-06T13:47:34.605543+0000","last_active":"2024-04-06T13:47:35.594327+0000","last_peered":"2024-04-06T13:47:35.594327+0000","last_clean":"2024-04-06T13:47:35.594327+0000","last_became_active":"2024-04-06T13:47:34.605284+0000","last_became_peered":"2024-04-06T13:47:34.605284+0000","last_unstale":"2024-04-06T13:47:35.594327+0000","last_undegraded":"2024-04-06T13:47:35.594327+0000","last_fullsized":"2024-04-06T13:47:35.594327+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:48:50.376524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591044+0000","last_change":"2024-04-06T13:47:22.139348+0000","last_active":"2024-04-06T13:47:34.591044+0000","last_peered":"2024-04-06T13:47:34.591044+0000","last_clean":"2024-04-06T13:47:34.591044+0000","last_became_active":"2024-04-06T13:47:22.138953+0000","last_became_peered":"2024-04-06T13:47:22.138953+0000","last_unstale":"2024-04-06T13:47:34.591044+0000","last_undegraded":"2024-04-06T13:47:34.591044+0000","last_fullsized":"2024-04-06T13:47:34.591044+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:51:33.543781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120352+0000","last_change":"2024-04-06T13:47:34.599654+0000","last_active":"2024-04-06T13:47:36.120352+0000","last_peered":"2024-04-06T13:47:36.120352+0000","last_clean":"2024-04-06T13:47:36.120352+0000","last_became_active":"2024-04-06T13:47:34.599387+0000","last_became_peered":"2024-04-06T13:47:34.599387+0000","last_unstale":"2024-04-06T13:47:36.120352+0000","last_undegraded":"2024-04-06T13:47:36.120352+0000","last_fullsized":"2024-04-06T13:47:36.120352+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:25:53.399400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.592045+0000","last_change":"2024-04-06T13:47:22.147317+0000","last_active":"2024-04-06T13:47:34.592045+0000","last_peered":"2024-04-06T13:47:34.592045+0000","last_clean":"2024-04-06T13:47:34.592045+0000","last_became_active":"2024-04-06T13:47:22.146941+0000","last_became_peered":"2024-04-06T13:47:22.146941+0000","last_unstale":"2024-04-06T13:47:34.592045+0000","last_undegraded":"2024-04-06T13:47:34.592045+0000","last_fullsized":"2024-04-06T13:47:34.592045+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:55:01.812146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121220+0000","last_change":"2024-04-06T13:47:34.617247+0000","last_active":"2024-04-06T13:47:36.121220+0000","last_peered":"2024-04-06T13:47:36.121220+0000","last_clean":"2024-04-06T13:47:36.121220+0000","last_became_active":"2024-04-06T13:47:34.616973+0000","last_became_peered":"2024-04-06T13:47:34.616973+0000","last_unstale":"2024-04-06T13:47:36.121220+0000","last_undegraded":"2024-04-06T13:47:36.121220+0000","last_fullsized":"2024-04-06T13:47:36.121220+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:13:13.458973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597215+0000","last_change":"2024-04-06T13:47:22.140321+0000","last_active":"2024-04-06T13:47:34.597215+0000","last_peered":"2024-04-06T13:47:34.597215+0000","last_clean":"2024-04-06T13:47:34.597215+0000","last_became_active":"2024-04-06T13:47:22.140027+0000","last_became_peered":"2024-04-06T13:47:22.140027+0000","last_unstale":"2024-04-06T13:47:34.597215+0000","last_undegraded":"2024-04-06T13:47:34.597215+0000","last_fullsized":"2024-04-06T13:47:34.597215+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:49:16.852621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120163+0000","last_change":"2024-04-06T13:47:34.612019+0000","last_active":"2024-04-06T13:47:36.120163+0000","last_peered":"2024-04-06T13:47:36.120163+0000","last_clean":"2024-04-06T13:47:36.120163+0000","last_became_active":"2024-04-06T13:47:34.611550+0000","last_became_peered":"2024-04-06T13:47:34.611550+0000","last_unstale":"2024-04-06T13:47:36.120163+0000","last_undegraded":"2024-04-06T13:47:36.120163+0000","last_fullsized":"2024-04-06T13:47:36.120163+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:33:46.757895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.587686+0000","last_change":"2024-04-06T13:47:22.147771+0000","last_active":"2024-04-06T13:47:34.587686+0000","last_peered":"2024-04-06T13:47:34.587686+0000","last_clean":"2024-04-06T13:47:34.587686+0000","last_became_active":"2024-04-06T13:47:22.147512+0000","last_became_peered":"2024-04-06T13:47:22.147512+0000","last_unstale":"2024-04-06T13:47:34.587686+0000","last_undegraded":"2024-04-06T13:47:34.587686+0000","last_fullsized":"2024-04-06T13:47:34.587686+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:25:04.287164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593592+0000","last_change":"2024-04-06T13:47:34.615987+0000","last_active":"2024-04-06T13:47:35.593592+0000","last_peered":"2024-04-06T13:47:35.593592+0000","last_clean":"2024-04-06T13:47:35.593592+0000","last_became_active":"2024-04-06T13:47:34.615180+0000","last_became_peered":"2024-04-06T13:47:34.615180+0000","last_unstale":"2024-04-06T13:47:35.593592+0000","last_undegraded":"2024-04-06T13:47:35.593592+0000","last_fullsized":"2024-04-06T13:47:35.593592+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:28:50.227790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597728+0000","last_change":"2024-04-06T13:47:22.146702+0000","last_active":"2024-04-06T13:47:34.597728+0000","last_peered":"2024-04-06T13:47:34.597728+0000","last_clean":"2024-04-06T13:47:34.597728+0000","last_became_active":"2024-04-06T13:47:22.139334+0000","last_became_peered":"2024-04-06T13:47:22.139334+0000","last_unstale":"2024-04-06T13:47:34.597728+0000","last_undegraded":"2024-04-06T13:47:34.597728+0000","last_fullsized":"2024-04-06T13:47:34.597728+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:36:19.531363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594785+0000","last_change":"2024-04-06T13:47:34.607813+0000","last_active":"2024-04-06T13:47:35.594785+0000","last_peered":"2024-04-06T13:47:35.594785+0000","last_clean":"2024-04-06T13:47:35.594785+0000","last_became_active":"2024-04-06T13:47:34.607531+0000","last_became_peered":"2024-04-06T13:47:34.607531+0000","last_unstale":"2024-04-06T13:47:35.594785+0000","last_undegraded":"2024-04-06T13:47:35.594785+0000","last_fullsized":"2024-04-06T13:47:35.594785+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:21.552700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591158+0000","last_change":"2024-04-06T13:47:22.136209+0000","last_active":"2024-04-06T13:47:34.591158+0000","last_peered":"2024-04-06T13:47:34.591158+0000","last_clean":"2024-04-06T13:47:34.591158+0000","last_became_active":"2024-04-06T13:47:22.135366+0000","last_became_peered":"2024-04-06T13:47:22.135366+0000","last_unstale":"2024-04-06T13:47:34.591158+0000","last_undegraded":"2024-04-06T13:47:34.591158+0000","last_fullsized":"2024-04-06T13:47:34.591158+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:14:48.802617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593242+0000","last_change":"2024-04-06T13:47:34.605433+0000","last_active":"2024-04-06T13:47:35.593242+0000","last_peered":"2024-04-06T13:47:35.593242+0000","last_clean":"2024-04-06T13:47:35.593242+0000","last_became_active":"2024-04-06T13:47:34.605066+0000","last_became_peered":"2024-04-06T13:47:34.605066+0000","last_unstale":"2024-04-06T13:47:35.593242+0000","last_undegraded":"2024-04-06T13:47:35.593242+0000","last_fullsized":"2024-04-06T13:47:35.593242+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:06:24.637773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.598140+0000","last_change":"2024-04-06T13:47:22.130575+0000","last_active":"2024-04-06T13:47:34.598140+0000","last_peered":"2024-04-06T13:47:34.598140+0000","last_clean":"2024-04-06T13:47:34.598140+0000","last_became_active":"2024-04-06T13:47:22.130287+0000","last_became_peered":"2024-04-06T13:47:22.130287+0000","last_unstale":"2024-04-06T13:47:34.598140+0000","last_undegraded":"2024-04-06T13:47:34.598140+0000","last_fullsized":"2024-04-06T13:47:34.598140+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:33:34.282063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.593808+0000","last_change":"2024-04-06T13:47:34.602926+0000","last_active":"2024-04-06T13:47:35.593808+0000","last_peered":"2024-04-06T13:47:35.593808+0000","last_clean":"2024-04-06T13:47:35.593808+0000","last_became_active":"2024-04-06T13:47:34.602664+0000","last_became_peered":"2024-04-06T13:47:34.602664+0000","last_unstale":"2024-04-06T13:47:35.593808+0000","last_undegraded":"2024-04-06T13:47:35.593808+0000","last_fullsized":"2024-04-06T13:47:35.593808+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:32:42.519117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591141+0000","last_change":"2024-04-06T13:47:22.137418+0000","last_active":"2024-04-06T13:47:34.591141+0000","last_peered":"2024-04-06T13:47:34.591141+0000","last_clean":"2024-04-06T13:47:34.591141+0000","last_became_active":"2024-04-06T13:47:22.137176+0000","last_became_peered":"2024-04-06T13:47:22.137176+0000","last_unstale":"2024-04-06T13:47:34.591141+0000","last_undegraded":"2024-04-06T13:47:34.591141+0000","last_fullsized":"2024-04-06T13:47:34.591141+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:40:38.388468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.120863+0000","last_change":"2024-04-06T13:47:34.596811+0000","last_active":"2024-04-06T13:47:36.120863+0000","last_peered":"2024-04-06T13:47:36.120863+0000","last_clean":"2024-04-06T13:47:36.120863+0000","last_became_active":"2024-04-06T13:47:34.596567+0000","last_became_peered":"2024-04-06T13:47:34.596567+0000","last_unstale":"2024-04-06T13:47:36.120863+0000","last_undegraded":"2024-04-06T13:47:36.120863+0000","last_fullsized":"2024-04-06T13:47:36.120863+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:16:21.941269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.600260+0000","last_change":"2024-04-06T13:47:22.132869+0000","last_active":"2024-04-06T13:47:34.600260+0000","last_peered":"2024-04-06T13:47:34.600260+0000","last_clean":"2024-04-06T13:47:34.600260+0000","last_became_active":"2024-04-06T13:47:22.132613+0000","last_became_peered":"2024-04-06T13:47:22.132613+0000","last_unstale":"2024-04-06T13:47:34.600260+0000","last_undegraded":"2024-04-06T13:47:34.600260+0000","last_fullsized":"2024-04-06T13:47:34.600260+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:38:24.228670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.595770+0000","last_change":"2024-04-06T13:47:34.608817+0000","last_active":"2024-04-06T13:47:35.595770+0000","last_peered":"2024-04-06T13:47:35.595770+0000","last_clean":"2024-04-06T13:47:35.595770+0000","last_became_active":"2024-04-06T13:47:34.607073+0000","last_became_peered":"2024-04-06T13:47:34.607073+0000","last_unstale":"2024-04-06T13:47:35.595770+0000","last_undegraded":"2024-04-06T13:47:35.595770+0000","last_fullsized":"2024-04-06T13:47:35.595770+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:44:32.385590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.591973+0000","last_change":"2024-04-06T13:47:22.137179+0000","last_active":"2024-04-06T13:47:34.591973+0000","last_peered":"2024-04-06T13:47:34.591973+0000","last_clean":"2024-04-06T13:47:34.591973+0000","last_became_active":"2024-04-06T13:47:22.136779+0000","last_became_peered":"2024-04-06T13:47:22.136779+0000","last_unstale":"2024-04-06T13:47:34.591973+0000","last_undegraded":"2024-04-06T13:47:34.591973+0000","last_fullsized":"2024-04-06T13:47:34.591973+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:25:40.923568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:36.121720+0000","last_change":"2024-04-06T13:47:34.612281+0000","last_active":"2024-04-06T13:47:36.121720+0000","last_peered":"2024-04-06T13:47:36.121720+0000","last_clean":"2024-04-06T13:47:36.121720+0000","last_became_active":"2024-04-06T13:47:34.611954+0000","last_became_peered":"2024-04-06T13:47:34.611954+0000","last_unstale":"2024-04-06T13:47:36.121720+0000","last_undegraded":"2024-04-06T13:47:36.121720+0000","last_fullsized":"2024-04-06T13:47:36.121720+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:38:36.704502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.597173+0000","last_change":"2024-04-06T13:47:22.148394+0000","last_active":"2024-04-06T13:47:34.597173+0000","last_peered":"2024-04-06T13:47:34.597173+0000","last_clean":"2024-04-06T13:47:34.597173+0000","last_became_active":"2024-04-06T13:47:22.148136+0000","last_became_peered":"2024-04-06T13:47:22.148136+0000","last_unstale":"2024-04-06T13:47:34.597173+0000","last_undegraded":"2024-04-06T13:47:34.597173+0000","last_fullsized":"2024-04-06T13:47:34.597173+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:06:09.829859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-06T13:47:35.594166+0000","last_change":"2024-04-06T13:47:34.602824+0000","last_active":"2024-04-06T13:47:35.594166+0000","last_peered":"2024-04-06T13:47:35.594166+0000","last_clean":"2024-04-06T13:47:35.594166+0000","last_became_active":"2024-04-06T13:47:34.602555+0000","last_became_peered":"2024-04-06T13:47:34.602555+0000","last_unstale":"2024-04-06T13:47:35.594166+0000","last_undegraded":"2024-04-06T13:47:35.594166+0000","last_fullsized":"2024-04-06T13:47:35.594166+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:34.172076+0000","last_clean_scrub_stamp":"2024-04-06T13:47:34.172076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:56:48.838776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":14,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T13:47:34.601204+0000","last_change":"2024-04-06T13:47:22.143038+0000","last_active":"2024-04-06T13:47:34.601204+0000","last_peered":"2024-04-06T13:47:34.601204+0000","last_clean":"2024-04-06T13:47:34.601204+0000","last_became_active":"2024-04-06T13:47:22.142498+0000","last_became_peered":"2024-04-06T13:47:22.142498+0000","last_unstale":"2024-04-06T13:47:34.601204+0000","last_undegraded":"2024-04-06T13:47:34.601204+0000","last_fullsized":"2024-04-06T13:47:34.601204+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:47:21.696243+0000","last_clean_scrub_stamp":"2024-04-06T13:47:21.696243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:40.999324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3611,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575985,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56704,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315136,"statfs":{"total":96636764160,"available":96578699264,"internally_reserved":0,"allocated":3518464,"data_stored":2079001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.751,"5min":0.78400000000000003,"15min":0.84899999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58499999999999996,"15min":0.59799999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76000000000000001,"5min":0.80800000000000005,"15min":0.82899999999999996},"last":0.42699999999999999}]},{"osd":1,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.57099999999999995,"15min":0.58799999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.66600000000000004,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59799999999999998,"15min":0.60499999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38900000000000001},"max":{"1min":0.752,"5min":0.83399999999999996,"15min":0.85299999999999998},"last":0.52200000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.57899999999999996,"15min":0.59499999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67700000000000005,"5min":0.81699999999999995,"15min":0.83899999999999997},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.60299999999999998,"15min":0.60599999999999998},"min":{"1min":0.47199999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73699999999999999,"5min":0.79700000000000004,"15min":0.81699999999999995},"last":0.442}]},{"osd":3,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.61599999999999999,"15min":0.61499999999999999},"min":{"1min":0.437,"5min":0.41199999999999998,"15min":0.38500000000000001},"max":{"1min":0.75600000000000001,"5min":0.81100000000000005,"15min":0.86199999999999999},"last":0.496},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59799999999999998,"15min":0.60899999999999999},"min":{"1min":0.42399999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.70199999999999996,"5min":0.83299999999999996,"15min":0.84099999999999997},"last":0.48799999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.45900000000000002,"15min":0.46400000000000002},"min":{"1min":0.33300000000000002,"5min":0.307,"15min":0.30299999999999999},"max":{"1min":0.628,"5min":0.76000000000000001,"15min":0.81599999999999995},"last":0.311},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.54600000000000004,"15min":0.54300000000000004},"min":{"1min":0.39300000000000002,"5min":0.38700000000000001,"15min":0.36799999999999999},"max":{"1min":0.68700000000000006,"5min":0.74299999999999999,"15min":0.82699999999999996},"last":0.45400000000000001}]},{"osd":5,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.49299999999999999,"15min":0.495},"min":{"1min":0.36099999999999999,"5min":0.318,"15min":0.308},"max":{"1min":0.71999999999999997,"5min":0.77100000000000002,"15min":0.78300000000000003},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.46899999999999997,"5min":0.39200000000000002,"15min":0.33600000000000002},"max":{"1min":0.69799999999999995,"5min":0.82099999999999995,"15min":0.85399999999999998},"last":0.35599999999999998}]},{"osd":6,"last update":"Sat Apr 6 13:47:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.39600000000000002,"5min":0.374,"15min":0.34899999999999998},"max":{"1min":0.67700000000000005,"5min":0.755,"15min":0.879},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56899999999999995,"15min":0.58699999999999997},"min":{"1min":0.39200000000000002,"5min":0.378,"15min":0.34799999999999998},"max":{"1min":0.73299999999999998,"5min":0.84899999999999998,"15min":0.89800000000000002},"last":0.40899999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575984,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38072,"kb_used_data":2856,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94333768,"statfs":{"total":96636764160,"available":96597778432,"internally_reserved":0,"allocated":2924544,"data_stored":1481585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.45300000000000001,"5min":0.38200000000000001,"15min":0.32000000000000001},"max":{"1min":0.74399999999999999,"5min":0.79700000000000004,"15min":0.83799999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.48199999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76900000000000002,"5min":0.80000000000000004,"15min":0.81999999999999995},"last":0.54100000000000004}]},{"osd":1,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.626,"15min":0.63},"min":{"1min":0.48999999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.83699999999999997,"15min":0.85799999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64200000000000002,"15min":0.65000000000000002},"min":{"1min":0.54600000000000004,"5min":0.45100000000000001,"15min":0.378},"max":{"1min":0.78900000000000003,"5min":0.81399999999999995,"15min":0.82699999999999996},"last":0.52500000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.495,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.76700000000000002,"5min":0.82299999999999995,"15min":0.82899999999999996},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.59299999999999997},"min":{"1min":0.46400000000000002,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.755,"5min":0.78600000000000003,"15min":0.80700000000000005},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77500000000000002,"5min":0.80100000000000005,"15min":0.82399999999999995},"last":0.495},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64600000000000002,"15min":0.65000000000000002},"min":{"1min":0.51500000000000001,"5min":0.45900000000000002,"15min":0.36099999999999999},"max":{"1min":0.78800000000000003,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.63900000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54500000000000004,"15min":0.53100000000000003},"min":{"1min":0.38100000000000001,"5min":0.36899999999999999,"15min":0.33800000000000002},"max":{"1min":0.72199999999999998,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53500000000000003,"15min":0.54500000000000004},"min":{"1min":0.371,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68799999999999994,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.59599999999999997}]},{"osd":5,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.504,"15min":0.50900000000000001},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":0.60799999999999998,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53100000000000003,"15min":0.53400000000000003},"min":{"1min":0.35499999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74099999999999999,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.625}]},{"osd":7,"last update":"Sat Apr 6 13:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71999999999999997,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56699999999999995,"15min":0.57199999999999995},"min":{"1min":0.435,"5min":0.36799999999999999,"15min":0.32800000000000001},"max":{"1min":0.70099999999999996,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.57799999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575983,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60180,"kb_used_data":2816,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311660,"statfs":{"total":96636764160,"available":96575139840,"internally_reserved":0,"allocated":2883584,"data_stored":1476500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.443,"5min":0.432,"15min":0.432},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25700000000000001},"max":{"1min":0.55300000000000005,"5min":0.60999999999999999,"15min":0.69399999999999995},"last":0.22800000000000001}]},{"osd":2,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.42999999999999999,"15min":0.42899999999999999},"min":{"1min":0.30299999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46800000000000003,"15min":0.46899999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.29399999999999998},"max":{"1min":0.69499999999999995,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.28699999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.28799999999999998},"max":{"1min":0.67800000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.26300000000000001},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.66100000000000003},"last":0.313}]},{"osd":4,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.64000000000000001,"15min":0.64900000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.33800000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.97399999999999998},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.65600000000000003,"15min":0.66000000000000003},"min":{"1min":0.45000000000000001,"5min":0.44500000000000001,"15min":0.44400000000000001},"max":{"1min":0.877,"5min":0.94599999999999995,"15min":0.995},"last":0.59699999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.65000000000000002,"15min":0.65600000000000003},"min":{"1min":0.51300000000000001,"5min":0.41299999999999998,"15min":0.40799999999999997},"max":{"1min":0.91800000000000004,"5min":0.96899999999999997,"15min":1.073},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.66800000000000004,"15min":0.67500000000000004},"min":{"1min":0.45500000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.90500000000000003,"5min":0.96499999999999997,"15min":1.0209999999999999},"last":0.54700000000000004}]},{"osd":6,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65100000000000002,"15min":0.66200000000000003},"min":{"1min":0.47699999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":1},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.51400000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":1.036},"last":0.624}]},{"osd":7,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.69799999999999995,"15min":0.69999999999999996},"min":{"1min":0.50600000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89600000000000002,"5min":0.94899999999999995,"15min":1.0649999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.66500000000000004,"15min":0.67300000000000004},"min":{"1min":0.49299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":1.056},"last":0.629}]}]},{"osd":0,"up_from":13,"seq":55834575983,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42728,"kb_used_data":3412,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329112,"statfs":{"total":96636764160,"available":96593010688,"internally_reserved":0,"allocated":3493888,"data_stored":2074517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.40200000000000002,"15min":0.39900000000000002},"min":{"1min":0.315,"5min":0.24299999999999999,"15min":0.23799999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.41499999999999998,"15min":0.42299999999999999},"min":{"1min":0.32400000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55800000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.33100000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.42299999999999999,"15min":0.432},"min":{"1min":0.32100000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.55900000000000005,"5min":0.67900000000000005,"15min":0.70099999999999996},"last":0.317},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.441,"15min":0.45200000000000001},"min":{"1min":0.35899999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60799999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.30299999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.42999999999999999,"15min":0.434},"min":{"1min":0.34399999999999997,"5min":0.27400000000000002,"15min":0.26800000000000002},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.48299999999999998,"15min":0.48599999999999999},"min":{"1min":0.39700000000000002,"5min":0.314,"15min":0.307},"max":{"1min":0.64100000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.35299999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.63100000000000001,"15min":0.64800000000000002},"min":{"1min":0.48299999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":1.0329999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.63900000000000001,"15min":0.65100000000000002},"min":{"1min":0.504,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.45700000000000002}]},{"osd":5,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.628,"15min":0.64800000000000002},"min":{"1min":0.5,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.87,"5min":0.876,"15min":1.0009999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.66300000000000003,"15min":0.67100000000000004},"min":{"1min":0.48599999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.89500000000000002,"5min":0.92800000000000005,"15min":1.052},"last":0.47999999999999998}]},{"osd":6,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67000000000000004,"15min":0.68100000000000005},"min":{"1min":0.52200000000000002,"5min":0.439,"15min":0.435},"max":{"1min":0.94399999999999995,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.66300000000000003,"15min":0.67900000000000005},"min":{"1min":0.49199999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.89500000000000002,"5min":0.96399999999999997,"15min":1.085},"last":0.40799999999999997}]},{"osd":7,"last update":"Sat Apr 6 13:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.68500000000000005,"15min":0.70099999999999996},"min":{"1min":0.54000000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91200000000000003,"5min":1.01,"15min":1.121},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.67000000000000004,"15min":0.68100000000000005},"min":{"1min":0.55700000000000005,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":1.0229999999999999},"last":0.49399999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575985,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60204,"kb_used_data":2840,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311636,"statfs":{"total":96636764160,"available":96575115264,"internally_reserved":0,"allocated":2908160,"data_stored":1484765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.38900000000000001,"15min":0.38800000000000001},"min":{"1min":0.307,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42199999999999999,"15min":0.42699999999999999},"min":{"1min":0.28100000000000003,"5min":0.27300000000000002,"15min":0.249},"max":{"1min":0.57099999999999995,"5min":0.58099999999999996,"15min":1.044},"last":0.40200000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40300000000000002,"15min":0.41199999999999998},"min":{"1min":0.28899999999999998,"5min":0.25700000000000001,"15min":0.255},"max":{"1min":0.52300000000000002,"5min":0.56399999999999995,"15min":0.753},"last":0.437},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.44500000000000001,"15min":0.45200000000000001},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.28399999999999997},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.82199999999999995},"last":0.51300000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.46000000000000002,"15min":0.46700000000000003},"min":{"1min":0.36499999999999999,"5min":0.32900000000000001,"15min":0.27800000000000002},"max":{"1min":0.63900000000000001,"5min":0.69699999999999995,"15min":0.70299999999999996},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.45200000000000001,"15min":0.45500000000000002},"min":{"1min":0.308,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.59799999999999998,"5min":0.64500000000000002,"15min":0.68600000000000005},"last":0.504}]},{"osd":4,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.64000000000000001,"15min":0.64500000000000002},"min":{"1min":0.47199999999999998,"5min":0.42199999999999999,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66400000000000003,"15min":0.66800000000000004},"min":{"1min":0.44500000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.88900000000000001}]},{"osd":5,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66800000000000004,"15min":0.67000000000000004},"min":{"1min":0.46100000000000002,"5min":0.46000000000000002,"15min":0.433},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":1.0149999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.66900000000000004,"15min":0.67200000000000004},"min":{"1min":0.52500000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":1.028}]},{"osd":6,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.67300000000000004,"15min":0.67800000000000005},"min":{"1min":0.46999999999999997,"5min":0.46700000000000003,"15min":0.44600000000000001},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":1.091},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69999999999999996,"15min":0.70199999999999996},"min":{"1min":0.50600000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.96099999999999997},"last":0.95699999999999996}]},{"osd":7,"last update":"Sat Apr 6 13:46:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68300000000000005,"15min":0.69199999999999995},"min":{"1min":0.5,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92500000000000004,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.70399999999999996,"15min":0.71599999999999997},"min":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.85299999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575984,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47912,"kb_used_data":2836,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323928,"statfs":{"total":96636764160,"available":96587702272,"internally_reserved":0,"allocated":2904064,"data_stored":1477667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39000000000000001,"15min":0.38900000000000001},"min":{"1min":0.26000000000000001,"5min":0.255,"15min":0.23499999999999999},"max":{"1min":0.55200000000000005,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41199999999999998,"15min":0.41299999999999998},"min":{"1min":0.28799999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.56899999999999995,"5min":0.622,"15min":0.66700000000000004},"last":0.439}]},{"osd":1,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40699999999999997,"15min":0.40600000000000003},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.65700000000000003},"last":0.379},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.438,"15min":0.44400000000000001},"min":{"1min":0.32800000000000001,"5min":0.27500000000000002,"15min":0.26600000000000001},"max":{"1min":0.63,"5min":0.65400000000000003,"15min":0.75800000000000001},"last":0.45400000000000001}]},{"osd":2,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48299999999999998,"15min":0.48599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.5,"15min":0.503},"min":{"1min":0.38600000000000001,"5min":0.34599999999999997,"15min":0.32900000000000001},"max":{"1min":0.67800000000000005,"5min":0.69299999999999995,"15min":0.79300000000000004},"last":0.40999999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.439,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.876,"5min":1.002,"15min":1.018},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66100000000000003,"15min":0.65300000000000002},"min":{"1min":0.497,"5min":0.45900000000000002,"15min":0.375},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.51900000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.95699999999999996,"5min":0.97299999999999998,"15min":0.999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.46200000000000002,"5min":0.437,"15min":0.437},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.60899999999999999}]},{"osd":6,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67500000000000004,"15min":0.66900000000000004},"min":{"1min":0.41499999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.68600000000000005,"15min":0.68400000000000005},"min":{"1min":0.48799999999999999,"5min":0.46700000000000003,"15min":0.41099999999999998},"max":{"1min":0.88800000000000001,"5min":0.96199999999999997,"15min":0.98799999999999999},"last":0.56299999999999994}]},{"osd":7,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68500000000000005,"15min":0.68000000000000005},"min":{"1min":0.46100000000000002,"5min":0.432,"15min":0.40999999999999998},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70299999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.024},"last":0.50800000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575984,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47900,"kb_used_data":2828,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323940,"statfs":{"total":96636764160,"available":96587714560,"internally_reserved":0,"allocated":2895872,"data_stored":1478191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.373,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.76400000000000001,"5min":0.876,"15min":0.88500000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.60499999999999998,"15min":0.60299999999999998},"min":{"1min":0.40200000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77900000000000003,"5min":1.202,"15min":1.202},"last":0.39300000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63,"15min":0.64400000000000002},"min":{"1min":0.439,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.82999999999999996,"5min":1.256,"15min":1.256},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64000000000000001,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80900000000000005,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.47699999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.42499999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.81799999999999995,"5min":0.84599999999999997,"15min":0.877},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.627,"15min":0.621},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88500000000000001,"5min":1.294,"15min":1.294},"last":0.46400000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.44900000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.75,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46899999999999997,"5min":0.373,"15min":0.373},"max":{"1min":0.78000000000000003,"5min":0.90200000000000002,"15min":0.91100000000000003},"last":0.48799999999999999}]},{"osd":5,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.56200000000000006,"15min":0.51600000000000001},"min":{"1min":0.35299999999999998,"5min":0.33200000000000002,"15min":0.32800000000000001},"max":{"1min":0.69799999999999995,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.57899999999999996,"15min":0.54300000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68600000000000005,"5min":4.931,"15min":4.931},"last":0.435}]},{"osd":6,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54000000000000004,"15min":0.53000000000000003},"min":{"1min":0.34100000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72399999999999998,"5min":0.83499999999999996,"15min":0.83899999999999997},"last":0.375},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57899999999999996,"15min":0.56999999999999995},"min":{"1min":0.41599999999999998,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":0.751,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.5}]},{"osd":7,"last update":"Sat Apr 6 13:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.60899999999999999,"15min":0.58299999999999996},"min":{"1min":0.36899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.65400000000000003,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.60399999999999998,"15min":0.60199999999999998},"min":{"1min":0.437,"5min":0.40799999999999997,"15min":0.373},"max":{"1min":0.748,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.51300000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575984,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64304,"kb_used_data":2852,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307536,"statfs":{"total":96636764160,"available":96570916864,"internally_reserved":0,"allocated":2920448,"data_stored":1485749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.46400000000000002,"5min":0.34000000000000002,"15min":0.33000000000000002},"max":{"1min":0.70199999999999996,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58999999999999997,"15min":0.59499999999999997},"min":{"1min":0.39700000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.73999999999999999,"5min":0.84699999999999998,"15min":0.89200000000000002},"last":0.747}]},{"osd":1,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.41199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72699999999999998,"5min":0.81299999999999994,"15min":0.81399999999999995},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58599999999999997,"15min":0.58699999999999997},"min":{"1min":0.43099999999999999,"5min":0.39400000000000002,"15min":0.37},"max":{"1min":0.70999999999999996,"5min":0.871,"15min":0.871},"last":0.52700000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60599999999999998,"15min":0.60099999999999998},"min":{"1min":0.39400000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.70999999999999996,"5min":0.79700000000000004,"15min":0.84399999999999997},"last":0.441},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.43099999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.67900000000000005,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.63600000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60499999999999998,"15min":0.59899999999999998},"min":{"1min":0.44700000000000001,"5min":0.40999999999999998,"15min":0.373},"max":{"1min":0.754,"5min":0.81100000000000005,"15min":0.94499999999999995},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63600000000000001,"15min":0.63700000000000001},"min":{"1min":0.48499999999999999,"5min":0.40300000000000002,"15min":0.39000000000000001},"max":{"1min":0.81399999999999995,"5min":0.88200000000000001,"15min":0.91500000000000004},"last":0.65400000000000003}]},{"osd":4,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52500000000000002,"15min":0.505},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.27400000000000002},"max":{"1min":0.746,"5min":0.85699999999999998,"15min":1.0369999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60499999999999998,"15min":0.57699999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.308},"max":{"1min":0.78000000000000003,"5min":0.88800000000000001,"15min":1.1639999999999999},"last":0.76500000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54000000000000004,"15min":0.52400000000000002},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.27900000000000003},"max":{"1min":0.73799999999999999,"5min":0.94999999999999996,"15min":1.0169999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54600000000000004,"15min":0.52800000000000002},"min":{"1min":0.34399999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70999999999999996,"5min":0.878,"15min":1.0009999999999999},"last":0.81899999999999995}]},{"osd":7,"last update":"Sat Apr 6 13:46:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56100000000000005,"15min":0.54500000000000004},"min":{"1min":0.374,"5min":0.33600000000000002,"15min":0.32900000000000001},"max":{"1min":0.70899999999999996,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.66500000000000004,"15min":0.64700000000000002},"min":{"1min":0.45600000000000002,"5min":0.433,"15min":0.35199999999999998},"max":{"1min":0.84799999999999998,"5min":0.92300000000000004,"15min":0.96599999999999997},"last":0.80800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:47:39.122 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-06T13:47:39.122 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:47:39.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.276+0000 7fec7d48e700 1 Processor -- start 2024-04-06T13:47:39.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.276+0000 7fec7d48e700 1 -- start start 2024-04-06T13:47:39.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.276+0000 7fec7d48e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fec780b8d70 0x7fec7815dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:39.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.276+0000 7fec7d48e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec7815eac0 con 0x7fec780ce730 2024-04-06T13:47:39.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.276+0000 7fec7d48e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec7815e220 con 0x7fec7815d6f0 2024-04-06T13:47:39.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.276+0000 7fec7d48e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec7815e010 con 0x7fec780b8d70 2024-04-06T13:47:39.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.277+0000 7fec777fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fec780b8d70 0x7fec7815dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:39.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.277+0000 7fec767fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fec780ce730 0x7fec780d1e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55220/0 (socket says 172.21.15.189:55220) 2024-04-06T13:47:39.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.277+0000 7fec767fc700 1 -- 172.21.15.189:0/3289846081 learned_addr learned my addr 172.21.15.189:0/3289846081 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:39.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.277+0000 7fec777fe700 1 -- 172.21.15.189:0/3289846081 >> v1:172.21.15.192:6789/0 conn(0x7fec7815d6f0 legacy=0x7fec780ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:39.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.277+0000 7fec777fe700 1 -- 172.21.15.189:0/3289846081 >> v1:172.21.15.189:6789/0 conn(0x7fec780ce730 legacy=0x7fec780d1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:39.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.277+0000 7fec777fe700 1 -- 172.21.15.189:0/3289846081 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7815e500 con 0x7fec780b8d70 2024-04-06T13:47:39.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.277+0000 7fec75ffb700 1 -- 172.21.15.189:0/3289846081 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659240808 0 0) 0x7fec7815eac0 con 0x7fec780ce730 2024-04-06T13:47:39.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.277+0000 7fec777fe700 1 --2- 172.21.15.189:0/3289846081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec780b8d70 0x7fec7815dad0 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7fec70009d90 tx=0x7fec7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e4d82bb6c4e1d6f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:39.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.277+0000 7fec75ffb700 1 -- 172.21.15.189:0/3289846081 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7000e070 con 0x7fec780b8d70 2024-04-06T13:47:39.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.277+0000 7fec75ffb700 1 -- 172.21.15.189:0/3289846081 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7000bbd0 con 0x7fec780b8d70 2024-04-06T13:47:39.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.277+0000 7fec75ffb700 1 -- 172.21.15.189:0/3289846081 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70012430 con 0x7fec780b8d70 2024-04-06T13:47:39.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.278+0000 7fec7d48e700 1 -- 172.21.15.189:0/3289846081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec780b8d70 msgr2=0x7fec7815dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:39.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.278+0000 7fec7d48e700 1 --2- 172.21.15.189:0/3289846081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec780b8d70 0x7fec7815dad0 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7fec70009d90 tx=0x7fec7000b330 comp rx=0 tx=0).stop 2024-04-06T13:47:39.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.278+0000 7fec7d48e700 1 -- 172.21.15.189:0/3289846081 shutdown_connections 2024-04-06T13:47:39.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.278+0000 7fec7d48e700 1 --2- 172.21.15.189:0/3289846081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec780b8d70 0x7fec7815dad0 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:39.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.278+0000 7fec7d48e700 1 -- 172.21.15.189:0/3289846081 >> 172.21.15.189:0/3289846081 conn(0x7fec78072490 msgr2=0x7fec7806b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:39.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.278+0000 7fec7d48e700 1 -- 172.21.15.189:0/3289846081 shutdown_connections 2024-04-06T13:47:39.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.278+0000 7fec7d48e700 1 -- 172.21.15.189:0/3289846081 wait complete. 2024-04-06T13:47:39.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.279+0000 7fec7d48e700 1 Processor -- start 2024-04-06T13:47:39.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.279+0000 7fec7d48e700 1 -- start start 2024-04-06T13:47:39.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.280+0000 7fec7d48e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec780b8d70 0x7fec78143330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:39.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.280+0000 7fec7d48e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec780ce730 0x7fec78157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:39.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.280+0000 7fec7d48e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec7815d6f0 0x7fec781585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:39.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.280+0000 7fec7d48e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec78149aa0 con 0x7fec7815d6f0 2024-04-06T13:47:39.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.280+0000 7fec7d48e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec780b4770 con 0x7fec780ce730 2024-04-06T13:47:39.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.280+0000 7fec7d48e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec780b4a50 con 0x7fec780b8d70 2024-04-06T13:47:39.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.280+0000 7fec777fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec7815d6f0 0x7fec781585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:39.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.280+0000 7fec777fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec7815d6f0 0x7fec781585d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:55128/0 (socket says 172.21.15.189:55128) 2024-04-06T13:47:39.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.280+0000 7fec777fe700 1 -- 172.21.15.189:0/3735024710 learned_addr learned my addr 172.21.15.189:0/3735024710 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:39.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.280+0000 7fec76ffd700 1 --2- 172.21.15.189:0/3735024710 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec780b8d70 0x7fec78143330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:39.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.280+0000 7fec767fc700 1 --2- 172.21.15.189:0/3735024710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec780ce730 0x7fec78157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:39.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.280+0000 7fec777fe700 1 -- 172.21.15.189:0/3735024710 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec780b8d70 msgr2=0x7fec78143330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:39.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.280+0000 7fec777fe700 1 --2- 172.21.15.189:0/3735024710 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec780b8d70 0x7fec78143330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:39.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.280+0000 7fec777fe700 1 -- 172.21.15.189:0/3735024710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec780ce730 msgr2=0x7fec78157c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:39.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.280+0000 7fec777fe700 1 --2- 172.21.15.189:0/3735024710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec780ce730 0x7fec78157c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:39.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.280+0000 7fec777fe700 1 -- 172.21.15.189:0/3735024710 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec70009a70 con 0x7fec7815d6f0 2024-04-06T13:47:39.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.280+0000 7fec76ffd700 1 --2- 172.21.15.189:0/3735024710 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec780b8d70 0x7fec78143330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:39.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.280+0000 7fec777fe700 1 --2- 172.21.15.189:0/3735024710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec7815d6f0 0x7fec781585d0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fec70014040 tx=0x7fec70004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:39.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.281+0000 7fec6f7fe700 1 -- 172.21.15.189:0/3735024710 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7001b8a0 con 0x7fec7815d6f0 2024-04-06T13:47:39.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.281+0000 7fec6f7fe700 1 -- 172.21.15.189:0/3735024710 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec70020070 con 0x7fec7815d6f0 2024-04-06T13:47:39.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.281+0000 7fec6f7fe700 1 -- 172.21.15.189:0/3735024710 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7001bc40 con 0x7fec7815d6f0 2024-04-06T13:47:39.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.281+0000 7fec7d48e700 1 -- 172.21.15.189:0/3735024710 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec780b4d30 con 0x7fec7815d6f0 2024-04-06T13:47:39.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.281+0000 7fec7d48e700 1 -- 172.21.15.189:0/3735024710 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec780b52f0 con 0x7fec7815d6f0 2024-04-06T13:47:39.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.281+0000 7fec7d48e700 1 -- 172.21.15.189:0/3735024710 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec78074970 con 0x7fec7815d6f0 2024-04-06T13:47:39.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.282+0000 7fec6f7fe700 1 -- 172.21.15.189:0/3735024710 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fec70026400 con 0x7fec7815d6f0 2024-04-06T13:47:39.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.282+0000 7fec6f7fe700 1 --2- 172.21.15.189:0/3735024710 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fec5806adb0 0x7fec5806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:39.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.283+0000 7fec76ffd700 1 --2- 172.21.15.189:0/3735024710 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fec5806adb0 0x7fec5806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:39.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.283+0000 7fec6f7fe700 1 -- 172.21.15.189:0/3735024710 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6965+0+0 (secure 0 0 0) 0x7fec7008b850 con 0x7fec7815d6f0 2024-04-06T13:47:39.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.283+0000 7fec76ffd700 1 --2- 172.21.15.189:0/3735024710 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fec5806adb0 0x7fec5806d270 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fec68000f60 tx=0x7fec68009040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:39.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.287+0000 7fec6f7fe700 1 -- 172.21.15.189:0/3735024710 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fec70059f50 con 0x7fec7815d6f0 2024-04-06T13:47:39.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.528+0000 7fec7d48e700 1 -- 172.21.15.189:0/3735024710 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec7815c4f0 con 0x7fec7815d6f0 2024-04-06T13:47:39.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.528+0000 7fec6f7fe700 1 -- 172.21.15.189:0/3735024710 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5971 (secure 0 0 0) 0x7fec70059ae0 con 0x7fec7815d6f0 2024-04-06T13:47:39.528 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:39.528 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":106,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8885,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1481817364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1481817364},{"type":"v1","addr":"172.21.15.189:6837","nonce":1481817364}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":11244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/4021875447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":4021875447},{"type":"v1","addr":"172.21.15.189:6836","nonce":4021875447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:47:22.116705+0000","modified":"2024-04-06T13:47:25.129787+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/694564673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":694564673},{"type":"v1","addr":"172.21.15.192:6835","nonce":694564673}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:47:34.584208+0000","modified":"2024-04-06T13:47:37.592983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11262":{"gid":11262,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6834/4066431971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4066431971},{"type":"v1","addr":"172.21.15.192:6834","nonce":4066431971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-04-06T13:47:39.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.530+0000 7fec7d48e700 1 -- 172.21.15.189:0/3735024710 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fec5806adb0 msgr2=0x7fec5806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:39.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.530+0000 7fec7d48e700 1 --2- 172.21.15.189:0/3735024710 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fec5806adb0 0x7fec5806d270 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fec68000f60 tx=0x7fec68009040 comp rx=0 tx=0).stop 2024-04-06T13:47:39.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.530+0000 7fec7d48e700 1 -- 172.21.15.189:0/3735024710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec7815d6f0 msgr2=0x7fec781585d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:39.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.530+0000 7fec7d48e700 1 --2- 172.21.15.189:0/3735024710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec7815d6f0 0x7fec781585d0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fec70014040 tx=0x7fec70004ad0 comp rx=0 tx=0).stop 2024-04-06T13:47:39.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.530+0000 7fec7d48e700 1 -- 172.21.15.189:0/3735024710 shutdown_connections 2024-04-06T13:47:39.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.530+0000 7fec7d48e700 1 --2- 172.21.15.189:0/3735024710 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fec5806adb0 0x7fec5806d270 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:39.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.530+0000 7fec7d48e700 1 --2- 172.21.15.189:0/3735024710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec7815d6f0 0x7fec781585d0 unknown :-1 s=CLOSED pgs=2475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:39.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.530+0000 7fec7d48e700 1 --2- 172.21.15.189:0/3735024710 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec780b8d70 0x7fec78143330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:39.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.530+0000 7fec7d48e700 1 --2- 172.21.15.189:0/3735024710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec780ce730 0x7fec78157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:39.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.530+0000 7fec7d48e700 1 -- 172.21.15.189:0/3735024710 >> 172.21.15.189:0/3735024710 conn(0x7fec78072490 msgr2=0x7fec7806b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:39.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.530+0000 7fec7d48e700 1 -- 172.21.15.189:0/3735024710 shutdown_connections 2024-04-06T13:47:39.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:39.530+0000 7fec7d48e700 1 -- 172.21.15.189:0/3735024710 wait complete. 2024-04-06T13:47:39.532 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 106 2024-04-06T13:47:39.945 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-06T13:47:34.584208+0000', 'modified': '2024-04-06T13:47:37.592983+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11262}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11262': {'gid': 11262, 'name': 'd', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.192:6834/4066431971', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6832', 'nonce': 4066431971}, {'type': 'v1', 'addr': '172.21.15.192:6834', 'nonce': 4066431971}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-06T13:47:39.945 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-06T13:47:39.945 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-06T13:47:40.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.070+0000 7f05c589b700 1 Processor -- start 2024-04-06T13:47:40.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.070+0000 7f05c589b700 1 -- start start 2024-04-06T13:47:40.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.070+0000 7f05c589b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f05c00b4ed0 0x7f05c00b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.070+0000 7f05c589b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05c00d2880 0x7f05c00d2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.070+0000 7f05c589b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c00b5cc0 con 0x7f05c015d590 2024-04-06T13:47:40.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.070+0000 7f05c589b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c00b58c0 con 0x7f05c00b4ed0 2024-04-06T13:47:40.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.070+0000 7f05c589b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c00d1500 con 0x7f05c00d2880 2024-04-06T13:47:40.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.071+0000 7f05be7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f05c00b4ed0 0x7f05c00b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.071+0000 7f05beffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f05c015d590 0x7f05c00b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58178/0 (socket says 172.21.15.189:58178) 2024-04-06T13:47:40.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.071+0000 7f05beffd700 1 -- 172.21.15.189:0/4167952221 learned_addr learned my addr 172.21.15.189:0/4167952221 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:40.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.071+0000 7f05bf7fe700 1 --2- 172.21.15.189:0/4167952221 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05c00d2880 0x7f05c00d2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.071+0000 7f05be7fc700 1 -- 172.21.15.189:0/4167952221 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05c00d2880 msgr2=0x7f05c00d2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:40.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.071+0000 7f05be7fc700 1 --2- 172.21.15.189:0/4167952221 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05c00d2880 0x7f05c00d2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.071+0000 7f05be7fc700 1 -- 172.21.15.189:0/4167952221 >> v1:172.21.15.189:6789/0 conn(0x7f05c015d590 legacy=0x7f05c00b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:40.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.071+0000 7f05be7fc700 1 -- 172.21.15.189:0/4167952221 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c00d17e0 con 0x7f05c00b4ed0 2024-04-06T13:47:40.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.071+0000 7f05bdffb700 1 -- 172.21.15.189:0/4167952221 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700449393 0 0) 0x7f05c00b5cc0 con 0x7f05c015d590 2024-04-06T13:47:40.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.071+0000 7f05bf7fe700 1 --2- 172.21.15.189:0/4167952221 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05c00d2880 0x7f05c00d2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:40.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.071+0000 7f05be7fc700 1 --2- 172.21.15.189:0/4167952221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05c00b4ed0 0x7f05c00b5380 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009840 tx=0x7f05b4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63cc23924f70dc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:40.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.071+0000 7f05bdffb700 1 -- 172.21.15.189:0/4167952221 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4011830 con 0x7f05c00b4ed0 2024-04-06T13:47:40.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.071+0000 7f05bdffb700 1 -- 172.21.15.189:0/4167952221 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b4011990 con 0x7f05c00b4ed0 2024-04-06T13:47:40.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.071+0000 7f05bdffb700 1 -- 172.21.15.189:0/4167952221 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4011d20 con 0x7f05c00b4ed0 2024-04-06T13:47:40.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.072+0000 7f05c589b700 1 -- 172.21.15.189:0/4167952221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05c00b4ed0 msgr2=0x7f05c00b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:40.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.072+0000 7f05c589b700 1 --2- 172.21.15.189:0/4167952221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05c00b4ed0 0x7f05c00b5380 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009840 tx=0x7f05b4009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:40.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.072+0000 7f05c589b700 1 -- 172.21.15.189:0/4167952221 shutdown_connections 2024-04-06T13:47:40.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.072+0000 7f05c589b700 1 --2- 172.21.15.189:0/4167952221 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05c00d2880 0x7f05c00d2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.072+0000 7f05c589b700 1 --2- 172.21.15.189:0/4167952221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05c00b4ed0 0x7f05c00b5380 unknown :-1 s=CLOSED pgs=2477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.072+0000 7f05c589b700 1 -- 172.21.15.189:0/4167952221 >> 172.21.15.189:0/4167952221 conn(0x7f05c006bbc0 msgr2=0x7f05c006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:40.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.072+0000 7f05c589b700 1 -- 172.21.15.189:0/4167952221 shutdown_connections 2024-04-06T13:47:40.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.072+0000 7f05c589b700 1 -- 172.21.15.189:0/4167952221 wait complete. 2024-04-06T13:47:40.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.073+0000 7f05c589b700 1 Processor -- start 2024-04-06T13:47:40.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.074+0000 7f05c589b700 1 -- start start 2024-04-06T13:47:40.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.074+0000 7f05c589b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05c00b4ed0 0x7f05c015aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.074+0000 7f05c589b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05c00d2880 0x7f05c014fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.074+0000 7f05c589b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05c015d590 0x7f05c0153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.075+0000 7f05c589b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c00ca6a0 con 0x7f05c015d590 2024-04-06T13:47:40.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.075+0000 7f05c589b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c00ca980 con 0x7f05c00b4ed0 2024-04-06T13:47:40.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.075+0000 7f05c589b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c00cac60 con 0x7f05c00d2880 2024-04-06T13:47:40.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.075+0000 7f05bf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05c015d590 0x7f05c0153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.075+0000 7f05bf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05c015d590 0x7f05c0153260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44440/0 (socket says 172.21.15.189:44440) 2024-04-06T13:47:40.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.075+0000 7f05bf7fe700 1 -- 172.21.15.189:0/1235647851 learned_addr learned my addr 172.21.15.189:0/1235647851 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:40.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.075+0000 7f05be7fc700 1 --2- 172.21.15.189:0/1235647851 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05c00d2880 0x7f05c014fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.075+0000 7f05beffd700 1 --2- 172.21.15.189:0/1235647851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05c00b4ed0 0x7f05c015aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.075+0000 7f05bf7fe700 1 -- 172.21.15.189:0/1235647851 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05c00d2880 msgr2=0x7f05c014fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:40.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.075+0000 7f05bf7fe700 1 --2- 172.21.15.189:0/1235647851 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05c00d2880 0x7f05c014fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.075+0000 7f05bf7fe700 1 -- 172.21.15.189:0/1235647851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05c00b4ed0 msgr2=0x7f05c015aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:40.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.075+0000 7f05bf7fe700 1 --2- 172.21.15.189:0/1235647851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05c00b4ed0 0x7f05c015aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.075+0000 7f05bf7fe700 1 -- 172.21.15.189:0/1235647851 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b40094a0 con 0x7f05c015d590 2024-04-06T13:47:40.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.075+0000 7f05bf7fe700 1 --2- 172.21.15.189:0/1235647851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05c015d590 0x7f05c0153260 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7f05b800cef0 tx=0x7f05b800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:40.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.075+0000 7f05a7fff700 1 -- 172.21.15.189:0/1235647851 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b800adb0 con 0x7f05c015d590 2024-04-06T13:47:40.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.075+0000 7f05a7fff700 1 -- 172.21.15.189:0/1235647851 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b8015440 con 0x7f05c015d590 2024-04-06T13:47:40.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.075+0000 7f05a7fff700 1 -- 172.21.15.189:0/1235647851 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b80157f0 con 0x7f05c015d590 2024-04-06T13:47:40.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.075+0000 7f05c589b700 1 -- 172.21.15.189:0/1235647851 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c00caf40 con 0x7f05c015d590 2024-04-06T13:47:40.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.075+0000 7f05c589b700 1 -- 172.21.15.189:0/1235647851 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c01537a0 con 0x7f05c015d590 2024-04-06T13:47:40.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.077+0000 7f05a7fff700 1 -- 172.21.15.189:0/1235647851 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f05b800d430 con 0x7f05c015d590 2024-04-06T13:47:40.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.077+0000 7f05a7fff700 1 --2- 172.21.15.189:0/1235647851 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f05ac06adb0 0x7f05ac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.077+0000 7f05a7fff700 1 -- 172.21.15.189:0/1235647851 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f05b8089e60 con 0x7f05c015d590 2024-04-06T13:47:40.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.077+0000 7f05beffd700 1 --2- 172.21.15.189:0/1235647851 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f05ac06adb0 0x7f05ac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.078+0000 7f05beffd700 1 --2- 172.21.15.189:0/1235647851 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f05ac06adb0 0x7f05ac06d270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f05b0000f60 tx=0x7f05b0007c90 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:40.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.079+0000 7f05d1a4fb80 1 Processor -- start 2024-04-06T13:47:40.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.079+0000 7f05d1a4fb80 1 -- start start 2024-04-06T13:47:40.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.080+0000 7f05d1a4fb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b8009c20 0x55a0b800da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.080+0000 7f05d1a4fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b800efc0 0x55a0b7fe55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.080+0000 7f05d1a4fb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b7fe4980 0x55a0b7fe5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.080+0000 7f05d1a4fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b7f3e940 con 0x55a0b800efc0 2024-04-06T13:47:40.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.080+0000 7f05d1a4fb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b7f3c300 con 0x55a0b8009c20 2024-04-06T13:47:40.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.080+0000 7f05d1a4fb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0b7f37bf0 con 0x55a0b7fe4980 2024-04-06T13:47:40.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.080+0000 7f05be7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b800efc0 0x55a0b7fe55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.080+0000 7f05be7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b800efc0 0x55a0b7fe55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44454/0 (socket says 172.21.15.189:44454) 2024-04-06T13:47:40.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.080+0000 7f05be7fc700 1 -- 172.21.15.189:0/3423092746 learned_addr learned my addr 172.21.15.189:0/3423092746 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:40.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.080+0000 7f05bf7fe700 1 --2- 172.21.15.189:0/3423092746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b7fe4980 0x55a0b7fe5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.080+0000 7f05beffd700 1 --2- 172.21.15.189:0/3423092746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b8009c20 0x55a0b800da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.081+0000 7f05be7fc700 1 -- 172.21.15.189:0/3423092746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b7fe4980 msgr2=0x55a0b7fe5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:40.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.081+0000 7f05be7fc700 1 --2- 172.21.15.189:0/3423092746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b7fe4980 0x55a0b7fe5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.081+0000 7f05be7fc700 1 -- 172.21.15.189:0/3423092746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b8009c20 msgr2=0x55a0b800da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:40.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.081+0000 7f05be7fc700 1 --2- 172.21.15.189:0/3423092746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b8009c20 0x55a0b800da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.081+0000 7f05be7fc700 1 -- 172.21.15.189:0/3423092746 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0b7d65b10 con 0x55a0b800efc0 2024-04-06T13:47:40.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.081+0000 7f05bf7fe700 1 --2- 172.21.15.189:0/3423092746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b7fe4980 0x55a0b7fe5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.081+0000 7f05be7fc700 1 --2- 172.21.15.189:0/3423092746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b800efc0 0x55a0b7fe55f0 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7f05b4013000 tx=0x7f05b4010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e64a0eb2f6f19719 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.081+0000 7f05a57fa700 1 -- 172.21.15.189:0/3423092746 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4020070 con 0x55a0b800efc0 2024-04-06T13:47:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.081+0000 7f05a57fa700 1 -- 172.21.15.189:0/3423092746 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b4011bc0 con 0x55a0b800efc0 2024-04-06T13:47:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.081+0000 7f05a57fa700 1 -- 172.21.15.189:0/3423092746 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4004260 con 0x55a0b800efc0 2024-04-06T13:47:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.082+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/3423092746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b800efc0 msgr2=0x55a0b7fe55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.082+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/3423092746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b800efc0 0x55a0b7fe55f0 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7f05b4013000 tx=0x7f05b4010000 comp rx=0 tx=0).stop 2024-04-06T13:47:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.082+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/3423092746 shutdown_connections 2024-04-06T13:47:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.082+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/3423092746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b7fe4980 0x55a0b7fe5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.082+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/3423092746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b8009c20 0x55a0b800da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.082+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/3423092746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b800efc0 0x55a0b7fe55f0 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.082+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/3423092746 >> 172.21.15.189:0/3423092746 conn(0x55a0b7f45650 msgr2=0x55a0b802c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:40.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.082+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/3423092746 shutdown_connections 2024-04-06T13:47:40.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.082+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/3423092746 wait complete. 2024-04-06T13:47:40.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.082+0000 7f05d1a4fb80 1 -- start start 2024-04-06T13:47:40.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.082+0000 7f05d1a4fb80 1 Processor -- start 2024-04-06T13:47:40.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.083+0000 7f05d1a4fb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b7fe4980 0x55a0b8009060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.083+0000 7f05d1a4fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b8009c20 0x55a0b803e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.083+0000 7f05d1a4fb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b800efc0 0x55a0b803cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.083+0000 7f05d1a4fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b7ed3d30 con 0x55a0b8009c20 2024-04-06T13:47:40.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.083+0000 7f05d1a4fb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b7ed39d0 con 0x55a0b7fe4980 2024-04-06T13:47:40.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.083+0000 7f05d1a4fb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0b7dbb650 con 0x55a0b800efc0 2024-04-06T13:47:40.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.083+0000 7f05d1a4fb80 10 client.0 _ll_register_callbacks cb 0x55a0b7edd420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T13:47:40.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.083+0000 7f05be7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b8009c20 0x55a0b803e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.083+0000 7f05be7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b8009c20 0x55a0b803e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44468/0 (socket says 172.21.15.189:44468) 2024-04-06T13:47:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.083+0000 7f05be7fc700 1 -- 172.21.15.189:0/674273310 learned_addr learned my addr 172.21.15.189:0/674273310 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.083+0000 7f05beffd700 1 --2- 172.21.15.189:0/674273310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b7fe4980 0x55a0b8009060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.083+0000 7f05bf7fe700 1 --2- 172.21.15.189:0/674273310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b800efc0 0x55a0b803cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f05be7fc700 1 -- 172.21.15.189:0/674273310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b800efc0 msgr2=0x55a0b803cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f05be7fc700 1 --2- 172.21.15.189:0/674273310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b800efc0 0x55a0b803cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f05be7fc700 1 -- 172.21.15.189:0/674273310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b7fe4980 msgr2=0x55a0b8009060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f05be7fc700 1 --2- 172.21.15.189:0/674273310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b7fe4980 0x55a0b8009060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f05be7fc700 1 -- 172.21.15.189:0/674273310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c00d17e0 con 0x55a0b8009c20 2024-04-06T13:47:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f05be7fc700 1 --2- 172.21.15.189:0/674273310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b8009c20 0x55a0b803e8c0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009b90 tx=0x7f05b40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f324256bc79ce0c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f059e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T13:47:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f059e7fc700 1 -- 172.21.15.189:0/674273310 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4020040 con 0x55a0b8009c20 2024-04-06T13:47:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f059e7fc700 1 -- 172.21.15.189:0/674273310 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(231..231 src has 1..231) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f05b4004480 con 0x55a0b8009c20 2024-04-06T13:47:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/674273310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a0b7ef8330 con 0x55a0b8009c20 2024-04-06T13:47:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f059e7fc700 1 -- 172.21.15.189:0/674273310 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05b40036a0 con 0x55a0b8009c20 2024-04-06T13:47:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f05d1a4fb80 10 client.11568 fetch_fsmap learned FSMap version 106 2024-04-06T13:47:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/674273310 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=2+}) v3 -- 0x55a0b7efde50 con 0x55a0b8009c20 2024-04-06T13:47:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f059e7fc700 1 -- 172.21.15.189:0/674273310 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a0b7ef8330 con 0x55a0b8009c20 2024-04-06T13:47:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f059e7fc700 1 -- 172.21.15.189:0/674273310 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 106) v1 ==== 3681+0+0 (secure 0 0 0) 0x7f05b402d020 con 0x55a0b8009c20 2024-04-06T13:47:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f05d1a4fb80 10 client.11568 fetch_fsmap finished waiting for FSMap version 106 2024-04-06T13:47:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f05d1a4fb80 10 client.11568 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-06T13:47:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f05d1a4fb80 20 client.11568 populate_metadata read hostname 'smithi189' 2024-04-06T13:47:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/674273310 >> [v2:172.21.15.192:6832/4066431971,v1:172.21.15.192:6834/4066431971] conn(0x55a0b8023750 0x55a0b8023b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f05d1a4fb80 4 client.11568 mds_command: new command op to 11262 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-06T13:47:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.084+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/674273310 --> [v2:172.21.15.192:6832/4066431971,v1:172.21.15.192:6834/4066431971] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a0b7f306b0 con 0x55a0b8023750 2024-04-06T13:47:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.085+0000 7f05beffd700 1 --2- 172.21.15.189:0/674273310 >> [v2:172.21.15.192:6832/4066431971,v1:172.21.15.192:6834/4066431971] conn(0x55a0b8023750 0x55a0b8023b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.085+0000 7f059e7fc700 10 client.11568 ms_handle_connect on v2:172.21.15.192:6832/4066431971 2024-04-06T13:47:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.085+0000 7f05beffd700 1 --2- 172.21.15.189:0/674273310 >> [v2:172.21.15.192:6832/4066431971,v1:172.21.15.192:6834/4066431971] conn(0x55a0b8023750 0x55a0b8023b30 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dfbe7011206eb58 server_cookie=74f9e67cb130575e in_seq=0 out_seq=0 2024-04-06T13:47:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.086+0000 7f059e7fc700 1 -- 172.21.15.189:0/674273310 <== mds.0 v2:172.21.15.192:6832/4066431971 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a0b7f306b0 con 0x55a0b8023750 2024-04-06T13:47:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.086+0000 7f059e7fc700 10 client.11568 handle_command_reply: tid=0 2024-04-06T13:47:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.086+0000 7f05d1a4fb80 1 client.11568 shutdown 2024-04-06T13:47:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.086+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/674273310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b8009c20 msgr2=0x55a0b803e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.086+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/674273310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b8009c20 0x55a0b803e8c0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009b90 tx=0x7f05b40047c0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.086+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/674273310 shutdown_connections 2024-04-06T13:47:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.086+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/674273310 >> [v2:172.21.15.192:6832/4066431971,v1:172.21.15.192:6834/4066431971] conn(0x55a0b8023750 0x55a0b8023b30 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.086+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/674273310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b800efc0 0x55a0b803cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.086+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/674273310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b7fe4980 0x55a0b8009060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.086+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/674273310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b8009c20 0x55a0b803e8c0 unknown :-1 s=CLOSED pgs=2480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.086+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/674273310 >> 172.21.15.189:0/674273310 conn(0x55a0b7f45650 msgr2=0x55a0b7fe5560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.086+0000 7f059e7fc700 0 client.11568 ms_handle_reset on v2:172.21.15.192:6832/4066431971 2024-04-06T13:47:40.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.086+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/674273310 shutdown_connections 2024-04-06T13:47:40.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.086+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/674273310 wait complete. 2024-04-06T13:47:40.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.086+0000 7f05d1a4fb80 20 client.11568 trim_cache size 0 max 16384 2024-04-06T13:47:40.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.103+0000 7f05d1a4fb80 1 Processor -- start 2024-04-06T13:47:40.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.104+0000 7f05d1a4fb80 1 -- start start 2024-04-06T13:47:40.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.104+0000 7f05d1a4fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b7fe4980 0x55a0b8011ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.104+0000 7f05d1a4fb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b8009c20 0x55a0b8091800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.104+0000 7f05d1a4fb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b800efc0 0x55a0b8014590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.104+0000 7f05d1a4fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b7fe5640 con 0x55a0b7fe4980 2024-04-06T13:47:40.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.104+0000 7f05d1a4fb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b802bd40 con 0x55a0b800efc0 2024-04-06T13:47:40.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.104+0000 7f05d1a4fb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0b802cc30 con 0x55a0b8009c20 2024-04-06T13:47:40.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.104+0000 7f05beffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b7fe4980 0x55a0b8011ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.104+0000 7f05beffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b7fe4980 0x55a0b8011ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44478/0 (socket says 172.21.15.189:44478) 2024-04-06T13:47:40.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.105+0000 7f05beffd700 1 -- 172.21.15.189:0/201881287 learned_addr learned my addr 172.21.15.189:0/201881287 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:40.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.105+0000 7f05be7fc700 1 --2- 172.21.15.189:0/201881287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b8009c20 0x55a0b8091800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.105+0000 7f05bf7fe700 1 --2- 172.21.15.189:0/201881287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b800efc0 0x55a0b8014590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.105+0000 7f05beffd700 1 -- 172.21.15.189:0/201881287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b8009c20 msgr2=0x55a0b8091800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:40.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.105+0000 7f05beffd700 1 --2- 172.21.15.189:0/201881287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b8009c20 0x55a0b8091800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.105+0000 7f05beffd700 1 -- 172.21.15.189:0/201881287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b800efc0 msgr2=0x55a0b8014590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:40.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.105+0000 7f05beffd700 1 --2- 172.21.15.189:0/201881287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b800efc0 0x55a0b8014590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.105+0000 7f05beffd700 1 -- 172.21.15.189:0/201881287 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0b7dbdc80 con 0x55a0b7fe4980 2024-04-06T13:47:40.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.105+0000 7f05beffd700 1 --2- 172.21.15.189:0/201881287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b7fe4980 0x55a0b8011ad0 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7f05b0008fd0 tx=0x7f05b0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9820e3224dbeb710 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:40.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.105+0000 7f05a5ffb700 1 -- 172.21.15.189:0/201881287 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b0015a80 con 0x55a0b7fe4980 2024-04-06T13:47:40.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.105+0000 7f05a5ffb700 1 -- 172.21.15.189:0/201881287 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b0015be0 con 0x55a0b7fe4980 2024-04-06T13:47:40.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.105+0000 7f05a5ffb700 1 -- 172.21.15.189:0/201881287 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b0015d90 con 0x55a0b7fe4980 2024-04-06T13:47:40.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.106+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/201881287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b7fe4980 msgr2=0x55a0b8011ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:40.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.106+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/201881287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b7fe4980 0x55a0b8011ad0 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7f05b0008fd0 tx=0x7f05b0012000 comp rx=0 tx=0).stop 2024-04-06T13:47:40.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.106+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/201881287 shutdown_connections 2024-04-06T13:47:40.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.106+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/201881287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b800efc0 0x55a0b8014590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.106+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/201881287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b7fe4980 0x55a0b8011ad0 unknown :-1 s=CLOSED pgs=2481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.106+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/201881287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b8009c20 0x55a0b8091800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.106+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/201881287 >> 172.21.15.189:0/201881287 conn(0x55a0b7f45650 msgr2=0x55a0b804e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:40.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.106+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/201881287 shutdown_connections 2024-04-06T13:47:40.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.106+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/201881287 wait complete. 2024-04-06T13:47:40.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.106+0000 7f05d1a4fb80 1 -- start start 2024-04-06T13:47:40.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.106+0000 7f05d1a4fb80 1 Processor -- start 2024-04-06T13:47:40.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.107+0000 7f05d1a4fb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b7fe4980 0x55a0b8005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.107+0000 7f05d1a4fb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b8009c20 0x55a0b7fe6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.107+0000 7f05d1a4fb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b800efc0 0x55a0b8092410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.107+0000 7f05d1a4fb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b8091ff0 con 0x55a0b800efc0 2024-04-06T13:47:40.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.107+0000 7f05d1a4fb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b8006150 con 0x55a0b8009c20 2024-04-06T13:47:40.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.107+0000 7f05d1a4fb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0b808ed50 con 0x55a0b7fe4980 2024-04-06T13:47:40.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.107+0000 7f05d1a4fb80 10 client.0 _ll_register_callbacks cb 0x55a0b7edd420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T13:47:40.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.107+0000 7f05bf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b800efc0 0x55a0b8092410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.107+0000 7f05beffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b7fe4980 0x55a0b8005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.107+0000 7f05bf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b800efc0 0x55a0b8092410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44486/0 (socket says 172.21.15.189:44486) 2024-04-06T13:47:40.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.107+0000 7f05beffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b7fe4980 0x55a0b8005b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:50956/0 (socket says 172.21.15.189:50956) 2024-04-06T13:47:40.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.107+0000 7f05bf7fe700 1 -- 172.21.15.189:0/2428280799 learned_addr learned my addr 172.21.15.189:0/2428280799 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:40.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.107+0000 7f05be7fc700 1 --2- 172.21.15.189:0/2428280799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b8009c20 0x55a0b7fe6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.107+0000 7f05bf7fe700 1 -- 172.21.15.189:0/2428280799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b7fe4980 msgr2=0x55a0b8005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:40.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.107+0000 7f05bf7fe700 1 --2- 172.21.15.189:0/2428280799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b7fe4980 0x55a0b8005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.107+0000 7f05bf7fe700 1 -- 172.21.15.189:0/2428280799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b8009c20 msgr2=0x55a0b7fe6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:40.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.107+0000 7f05bf7fe700 1 --2- 172.21.15.189:0/2428280799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b8009c20 0x55a0b7fe6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.107+0000 7f05bf7fe700 1 -- 172.21.15.189:0/2428280799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b001d040 con 0x55a0b800efc0 2024-04-06T13:47:40.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.107+0000 7f05be7fc700 1 --2- 172.21.15.189:0/2428280799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b8009c20 0x55a0b7fe6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:40.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.108+0000 7f05bf7fe700 1 --2- 172.21.15.189:0/2428280799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b800efc0 0x55a0b8092410 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7f05b8054c40 tx=0x7f05b8058790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc81123909bc941 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:40.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.108+0000 7f059e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T13:47:40.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.108+0000 7f059e7fc700 1 -- 172.21.15.189:0/2428280799 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8060b80 con 0x55a0b800efc0 2024-04-06T13:47:40.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.108+0000 7f059e7fc700 1 -- 172.21.15.189:0/2428280799 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(231..231 src has 1..231) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f05b80606f0 con 0x55a0b800efc0 2024-04-06T13:47:40.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.108+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/2428280799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05b4011bc0 con 0x55a0b800efc0 2024-04-06T13:47:40.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.108+0000 7f059e7fc700 1 -- 172.21.15.189:0/2428280799 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05b80608b0 con 0x55a0b800efc0 2024-04-06T13:47:40.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.108+0000 7f05d1a4fb80 10 client.11574 fetch_fsmap learned FSMap version 106 2024-04-06T13:47:40.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.108+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/2428280799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=2+}) v3 -- 0x55a0b7d63fe0 con 0x55a0b800efc0 2024-04-06T13:47:40.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.108+0000 7f059e7fc700 1 -- 172.21.15.189:0/2428280799 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b4011bc0 con 0x55a0b800efc0 2024-04-06T13:47:40.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.108+0000 7f059e7fc700 1 -- 172.21.15.189:0/2428280799 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 106) v1 ==== 3681+0+0 (secure 0 0 0) 0x7f05b8081d20 con 0x55a0b800efc0 2024-04-06T13:47:40.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.108+0000 7f05d1a4fb80 10 client.11574 fetch_fsmap finished waiting for FSMap version 106 2024-04-06T13:47:40.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.108+0000 7f05d1a4fb80 10 client.11574 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-06T13:47:40.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.108+0000 7f05d1a4fb80 20 client.11574 populate_metadata read hostname 'smithi189' 2024-04-06T13:47:40.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.108+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/2428280799 >> [v2:172.21.15.192:6832/4066431971,v1:172.21.15.192:6834/4066431971] conn(0x55a0b80448a0 0x55a0b8044c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.108+0000 7f05d1a4fb80 4 client.11574 mds_command: new command op to 11262 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-06T13:47:40.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.108+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/2428280799 --> [v2:172.21.15.192:6832/4066431971,v1:172.21.15.192:6834/4066431971] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a0b7d86ca0 con 0x55a0b80448a0 2024-04-06T13:47:40.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.108+0000 7f05beffd700 1 --2- 172.21.15.189:0/2428280799 >> [v2:172.21.15.192:6832/4066431971,v1:172.21.15.192:6834/4066431971] conn(0x55a0b80448a0 0x55a0b8044c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.109+0000 7f059e7fc700 10 client.11574 ms_handle_connect on v2:172.21.15.192:6832/4066431971 2024-04-06T13:47:40.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.109+0000 7f05beffd700 1 --2- 172.21.15.189:0/2428280799 >> [v2:172.21.15.192:6832/4066431971,v1:172.21.15.192:6834/4066431971] conn(0x55a0b80448a0 0x55a0b8044c80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b75328eb8bfd1fcf server_cookie=93b37fab0c39cb7a in_seq=0 out_seq=0 2024-04-06T13:47:40.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.109+0000 7f059e7fc700 1 -- 172.21.15.189:0/2428280799 <== mds.0 v2:172.21.15.192:6832/4066431971 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+338 (crc 0 0 0) 0x55a0b7d86ca0 con 0x55a0b80448a0 2024-04-06T13:47:40.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.109+0000 7f059e7fc700 10 client.11574 handle_command_reply: tid=0 2024-04-06T13:47:40.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.109+0000 7f05d1a4fb80 1 client.11574 shutdown 2024-04-06T13:47:40.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.109+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/2428280799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b800efc0 msgr2=0x55a0b8092410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:40.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.109+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/2428280799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b800efc0 0x55a0b8092410 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7f05b8054c40 tx=0x7f05b8058790 comp rx=0 tx=0).stop 2024-04-06T13:47:40.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.109+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/2428280799 shutdown_connections 2024-04-06T13:47:40.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.109+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/2428280799 >> [v2:172.21.15.192:6832/4066431971,v1:172.21.15.192:6834/4066431971] conn(0x55a0b80448a0 0x55a0b8044c80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.109+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/2428280799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b800efc0 0x55a0b8092410 unknown :-1 s=CLOSED pgs=2482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.109+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/2428280799 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55a0b7fe4980 0x55a0b8005b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.109+0000 7f059e7fc700 0 client.11574 ms_handle_reset on v2:172.21.15.192:6832/4066431971 2024-04-06T13:47:40.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.109+0000 7f05d1a4fb80 1 --2- 172.21.15.189:0/2428280799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0b8009c20 0x55a0b7fe6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.109+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/2428280799 >> 172.21.15.189:0/2428280799 conn(0x55a0b7f45650 msgr2=0x55a0b7fdd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:40.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.109+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/2428280799 shutdown_connections 2024-04-06T13:47:40.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.109+0000 7f05d1a4fb80 1 -- 172.21.15.189:0/2428280799 wait complete. 2024-04-06T13:47:40.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.109+0000 7f05d1a4fb80 20 client.11574 trim_cache size 0 max 16384 2024-04-06T13:47:40.118 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:47:40.118 INFO:teuthology.orchestra.run.smithi189.stdout: "cluster_fsid": "e30041de-e9a3-4351-8ca7-29d7f3e3728c", 2024-04-06T13:47:40.118 INFO:teuthology.orchestra.run.smithi189.stdout: "whoami": 0, 2024-04-06T13:47:40.118 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 11262, 2024-04-06T13:47:40.118 INFO:teuthology.orchestra.run.smithi189.stdout: "want_state": "up:active", 2024-04-06T13:47:40.118 INFO:teuthology.orchestra.run.smithi189.stdout: "state": "up:active", 2024-04-06T13:47:40.118 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "backup_fs", 2024-04-06T13:47:40.118 INFO:teuthology.orchestra.run.smithi189.stdout: "rank_uptime": 5.5235950850000002, 2024-04-06T13:47:40.118 INFO:teuthology.orchestra.run.smithi189.stdout: "mdsmap_epoch": 106, 2024-04-06T13:47:40.118 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch": 231, 2024-04-06T13:47:40.118 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch_barrier": 231, 2024-04-06T13:47:40.119 INFO:teuthology.orchestra.run.smithi189.stdout: "uptime": 22.359066292000001 2024-04-06T13:47:40.119 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:47:40.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.110+0000 7f05c589b700 1 -- 172.21.15.189:0/1235647851 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f05ac06adb0 msgr2=0x7f05ac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:40.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.110+0000 7f05c589b700 1 --2- 172.21.15.189:0/1235647851 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f05ac06adb0 0x7f05ac06d270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f05b0000f60 tx=0x7f05b0007c90 comp rx=0 tx=0).stop 2024-04-06T13:47:40.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.110+0000 7f05c589b700 1 -- 172.21.15.189:0/1235647851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05c015d590 msgr2=0x7f05c0153260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:40.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.110+0000 7f05c589b700 1 --2- 172.21.15.189:0/1235647851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05c015d590 0x7f05c0153260 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7f05b800cef0 tx=0x7f05b800cf20 comp rx=0 tx=0).stop 2024-04-06T13:47:40.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.110+0000 7f05c589b700 1 -- 172.21.15.189:0/1235647851 shutdown_connections 2024-04-06T13:47:40.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.110+0000 7f05c589b700 1 --2- 172.21.15.189:0/1235647851 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f05ac06adb0 0x7f05ac06d270 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.110+0000 7f05c589b700 1 --2- 172.21.15.189:0/1235647851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05c015d590 0x7f05c0153260 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.110+0000 7f05c589b700 1 --2- 172.21.15.189:0/1235647851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05c00b4ed0 0x7f05c015aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.110+0000 7f05c589b700 1 --2- 172.21.15.189:0/1235647851 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05c00d2880 0x7f05c014fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.110+0000 7f05c589b700 1 -- 172.21.15.189:0/1235647851 >> 172.21.15.189:0/1235647851 conn(0x7f05c006bbc0 msgr2=0x7f05c015c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:40.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.110+0000 7f05c589b700 1 -- 172.21.15.189:0/1235647851 shutdown_connections 2024-04-06T13:47:40.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.110+0000 7f05c589b700 1 -- 172.21.15.189:0/1235647851 wait complete. 2024-04-06T13:47:40.122 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-06T13:47:40.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.245+0000 7f0665e00700 1 Processor -- start 2024-04-06T13:47:40.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.245+0000 7f0665e00700 1 -- start start 2024-04-06T13:47:40.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.246+0000 7f0665e00700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f066006ae40 0x7f066015ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.246+0000 7f0665e00700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f066015fcd0 0x7f066015bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.246+0000 7f0665e00700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066015c4c0 con 0x7f0660144f40 2024-04-06T13:47:40.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.246+0000 7f0665e00700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06600cfcb0 con 0x7f066006ae40 2024-04-06T13:47:40.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.246+0000 7f0665e00700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f066015ca20 con 0x7f066015fcd0 2024-04-06T13:47:40.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.246+0000 7f065effd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f066006ae40 0x7f066015ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.246+0000 7f065effd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f066006ae40 0x7f066015ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44502/0 (socket says 172.21.15.189:44502) 2024-04-06T13:47:40.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.246+0000 7f065effd700 1 -- 172.21.15.189:0/4264230443 learned_addr learned my addr 172.21.15.189:0/4264230443 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:40.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.246+0000 7f065ffff700 1 --2- 172.21.15.189:0/4264230443 >> v2:172.21.15.192:3300/0 conn(0x7f066015fcd0 0x7f066015bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.246+0000 7f065effd700 1 -- 172.21.15.189:0/4264230443 >> v2:172.21.15.192:3300/0 conn(0x7f066015fcd0 msgr2=0x7f066015bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:40.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.246+0000 7f065effd700 1 --2- 172.21.15.189:0/4264230443 >> v2:172.21.15.192:3300/0 conn(0x7f066015fcd0 0x7f066015bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.246+0000 7f065effd700 1 -- 172.21.15.189:0/4264230443 >> v1:172.21.15.192:6789/0 conn(0x7f0660144f40 legacy=0x7f066006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:40.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.246+0000 7f065effd700 1 -- 172.21.15.189:0/4264230443 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066015c740 con 0x7f066006ae40 2024-04-06T13:47:40.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.246+0000 7f065effd700 1 --2- 172.21.15.189:0/4264230443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f066006ae40 0x7f066015ba40 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f06600b5630 tx=0x7f0650009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6a43bf23629e8ab server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:40.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.246+0000 7f065e7fc700 1 -- 172.21.15.189:0/4264230443 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0650004230 con 0x7f066006ae40 2024-04-06T13:47:40.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.247+0000 7f065e7fc700 1 -- 172.21.15.189:0/4264230443 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0650004390 con 0x7f066006ae40 2024-04-06T13:47:40.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.247+0000 7f065e7fc700 1 -- 172.21.15.189:0/4264230443 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06500036a0 con 0x7f066006ae40 2024-04-06T13:47:40.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.247+0000 7f0665e00700 1 -- 172.21.15.189:0/4264230443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f066006ae40 msgr2=0x7f066015ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:40.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.247+0000 7f0665e00700 1 --2- 172.21.15.189:0/4264230443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f066006ae40 0x7f066015ba40 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f06600b5630 tx=0x7f0650009bd0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.247+0000 7f0665e00700 1 -- 172.21.15.189:0/4264230443 shutdown_connections 2024-04-06T13:47:40.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.247+0000 7f0665e00700 1 --2- 172.21.15.189:0/4264230443 >> v2:172.21.15.192:3300/0 conn(0x7f066015fcd0 0x7f066015bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.247+0000 7f0665e00700 1 --2- 172.21.15.189:0/4264230443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f066006ae40 0x7f066015ba40 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.247+0000 7f0665e00700 1 -- 172.21.15.189:0/4264230443 >> 172.21.15.189:0/4264230443 conn(0x7f06600d00b0 msgr2=0x7f06600d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:40.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.247+0000 7f0665e00700 1 -- 172.21.15.189:0/4264230443 shutdown_connections 2024-04-06T13:47:40.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.247+0000 7f0665e00700 1 -- 172.21.15.189:0/4264230443 wait complete. 2024-04-06T13:47:40.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.248+0000 7f0665e00700 1 Processor -- start 2024-04-06T13:47:40.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.249+0000 7f0665e00700 1 -- start start 2024-04-06T13:47:40.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.249+0000 7f0665e00700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f066006ae40 0x7f06600c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.249+0000 7f0665e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0660144f40 0x7f06600c6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.249+0000 7f0665e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f066015fcd0 0x7f06601416a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.249+0000 7f0665e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06600c4f30 con 0x7f0660144f40 2024-04-06T13:47:40.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.249+0000 7f0665e00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06601f1430 con 0x7f066015fcd0 2024-04-06T13:47:40.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.249+0000 7f0665e00700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06600c5140 con 0x7f066006ae40 2024-04-06T13:47:40.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.249+0000 7f065effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0660144f40 0x7f06600c6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.249+0000 7f065effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0660144f40 0x7f06600c6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44510/0 (socket says 172.21.15.189:44510) 2024-04-06T13:47:40.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.249+0000 7f065effd700 1 -- 172.21.15.189:0/3762480889 learned_addr learned my addr 172.21.15.189:0/3762480889 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:40.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.250+0000 7f065f7fe700 1 --2- 172.21.15.189:0/3762480889 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f066006ae40 0x7f06600c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.250+0000 7f065ffff700 1 --2- 172.21.15.189:0/3762480889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f066015fcd0 0x7f06601416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.250+0000 7f065effd700 1 -- 172.21.15.189:0/3762480889 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f066006ae40 msgr2=0x7f06600c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:40.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.250+0000 7f065effd700 1 --2- 172.21.15.189:0/3762480889 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f066006ae40 0x7f06600c5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.250+0000 7f065effd700 1 -- 172.21.15.189:0/3762480889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f066015fcd0 msgr2=0x7f06601416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:40.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.250+0000 7f065effd700 1 --2- 172.21.15.189:0/3762480889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f066015fcd0 0x7f06601416a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.250+0000 7f065effd700 1 -- 172.21.15.189:0/3762480889 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0650009550 con 0x7f0660144f40 2024-04-06T13:47:40.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.250+0000 7f065effd700 1 --2- 172.21.15.189:0/3762480889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0660144f40 0x7f06600c6a40 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f0650009130 tx=0x7f0650004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:40.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.250+0000 7f0647fff700 1 -- 172.21.15.189:0/3762480889 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0650003dc0 con 0x7f0660144f40 2024-04-06T13:47:40.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.250+0000 7f0647fff700 1 -- 172.21.15.189:0/3762480889 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065001f070 con 0x7f0660144f40 2024-04-06T13:47:40.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.250+0000 7f0665e00700 1 -- 172.21.15.189:0/3762480889 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06600c5420 con 0x7f0660144f40 2024-04-06T13:47:40.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.250+0000 7f0647fff700 1 -- 172.21.15.189:0/3762480889 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0650011870 con 0x7f0660144f40 2024-04-06T13:47:40.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.250+0000 7f0665e00700 1 -- 172.21.15.189:0/3762480889 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06600c6f80 con 0x7f0660144f40 2024-04-06T13:47:40.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.251+0000 7f0665e00700 1 -- 172.21.15.189:0/3762480889 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0660067bb0 con 0x7f0660144f40 2024-04-06T13:47:40.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.252+0000 7f0647fff700 1 -- 172.21.15.189:0/3762480889 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0650021030 con 0x7f0660144f40 2024-04-06T13:47:40.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.252+0000 7f0647fff700 1 --2- 172.21.15.189:0/3762480889 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f064c06adb0 0x7f064c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.252+0000 7f0647fff700 1 -- 172.21.15.189:0/3762480889 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f065008afa0 con 0x7f0660144f40 2024-04-06T13:47:40.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.252+0000 7f065f7fe700 1 --2- 172.21.15.189:0/3762480889 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f064c06adb0 0x7f064c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.253+0000 7f065f7fe700 1 --2- 172.21.15.189:0/3762480889 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f064c06adb0 0x7f064c06d270 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0654001670 tx=0x7f0654009040 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:40.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.257+0000 7f0647fff700 1 -- 172.21.15.189:0/3762480889 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f06500be650 con 0x7f0660144f40 2024-04-06T13:47:40.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.481+0000 7f0665e00700 1 -- 172.21.15.189:0/3762480889 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f066015f7b0 con 0x7f064c06adb0 2024-04-06T13:47:40.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.482+0000 7f0647fff700 1 -- 172.21.15.189:0/3762480889 <== mgr.11028 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7f06540050d0 con 0x7f064c06adb0 2024-04-06T13:47:40.482 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T13:47:40.482 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:40.482 INFO:teuthology.orchestra.run.smithi189.stdout: "name": "cephfs" 2024-04-06T13:47:40.482 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:47:40.483 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:47:40.483 INFO:teuthology.orchestra.run.smithi189.stdout: "name": "backup_fs" 2024-04-06T13:47:40.483 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:47:40.483 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T13:47:40.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.484+0000 7f0665e00700 1 -- 172.21.15.189:0/3762480889 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f064c06adb0 msgr2=0x7f064c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:40.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.484+0000 7f0665e00700 1 --2- 172.21.15.189:0/3762480889 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f064c06adb0 0x7f064c06d270 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0654001670 tx=0x7f0654009040 comp rx=0 tx=0).stop 2024-04-06T13:47:40.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.484+0000 7f0665e00700 1 -- 172.21.15.189:0/3762480889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0660144f40 msgr2=0x7f06600c6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:40.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.484+0000 7f0665e00700 1 --2- 172.21.15.189:0/3762480889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0660144f40 0x7f06600c6a40 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f0650009130 tx=0x7f0650004750 comp rx=0 tx=0).stop 2024-04-06T13:47:40.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.485+0000 7f0665e00700 1 -- 172.21.15.189:0/3762480889 shutdown_connections 2024-04-06T13:47:40.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.485+0000 7f0665e00700 1 --2- 172.21.15.189:0/3762480889 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f064c06adb0 0x7f064c06d270 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.485+0000 7f0665e00700 1 --2- 172.21.15.189:0/3762480889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f066015fcd0 0x7f06601416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.485+0000 7f0665e00700 1 --2- 172.21.15.189:0/3762480889 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f066006ae40 0x7f06600c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.485+0000 7f0665e00700 1 --2- 172.21.15.189:0/3762480889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0660144f40 0x7f06600c6a40 unknown :-1 s=CLOSED pgs=2484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.485+0000 7f0665e00700 1 -- 172.21.15.189:0/3762480889 >> 172.21.15.189:0/3762480889 conn(0x7f06600d00b0 msgr2=0x7f066006ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:40.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.485+0000 7f0665e00700 1 -- 172.21.15.189:0/3762480889 shutdown_connections 2024-04-06T13:47:40.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.485+0000 7f0665e00700 1 -- 172.21.15.189:0/3762480889 wait complete. 2024-04-06T13:47:40.497 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-06T13:47:40.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.624+0000 7f1ec48a0700 1 Processor -- start 2024-04-06T13:47:40.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.624+0000 7f1ec48a0700 1 -- start start 2024-04-06T13:47:40.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.625+0000 7f1ec48a0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1ec015d7b0 0x7f1ec00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.625+0000 7f1ec48a0700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1ec015e500 0x7f1ec006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.625+0000 7f1ec48a0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec00b3990 con 0x7f1ec006a730 2024-04-06T13:47:40.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.625+0000 7f1ec48a0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec015e950 con 0x7f1ec015d7b0 2024-04-06T13:47:40.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.625+0000 7f1ec48a0700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec006b0f0 con 0x7f1ec015e500 2024-04-06T13:47:40.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.625+0000 7f1ebe59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1ec015d7b0 0x7f1ec00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.625+0000 7f1ebe59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1ec015d7b0 0x7f1ec00b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44520/0 (socket says 172.21.15.189:44520) 2024-04-06T13:47:40.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.625+0000 7f1ebe59c700 1 -- 172.21.15.189:0/1274656070 learned_addr learned my addr 172.21.15.189:0/1274656070 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:40.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.626+0000 7f1ebed9d700 1 --2- 172.21.15.189:0/1274656070 >> v2:172.21.15.192:3300/0 conn(0x7f1ec015e500 0x7f1ec006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.626+0000 7f1ebe59c700 1 -- 172.21.15.189:0/1274656070 >> v2:172.21.15.192:3300/0 conn(0x7f1ec015e500 msgr2=0x7f1ec006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:40.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.626+0000 7f1ebe59c700 1 --2- 172.21.15.189:0/1274656070 >> v2:172.21.15.192:3300/0 conn(0x7f1ec015e500 0x7f1ec006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.626+0000 7f1ebe59c700 1 -- 172.21.15.189:0/1274656070 >> v1:172.21.15.192:6789/0 conn(0x7f1ec006a730 legacy=0x7f1ec015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:40.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.626+0000 7f1ebe59c700 1 -- 172.21.15.189:0/1274656070 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec00b3060 con 0x7f1ec015d7b0 2024-04-06T13:47:40.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.626+0000 7f1ebed9d700 1 --2- 172.21.15.189:0/1274656070 >> v2:172.21.15.192:3300/0 conn(0x7f1ec015e500 0x7f1ec006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:40.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.626+0000 7f1ebe59c700 1 --2- 172.21.15.189:0/1274656070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ec015d7b0 0x7f1ec00b3e20 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00673e0 tx=0x7f1eb4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa1879b0315af88a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:40.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.626+0000 7f1ebd59a700 1 -- 172.21.15.189:0/1274656070 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4004230 con 0x7f1ec015d7b0 2024-04-06T13:47:40.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.626+0000 7f1ebd59a700 1 -- 172.21.15.189:0/1274656070 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb4004390 con 0x7f1ec015d7b0 2024-04-06T13:47:40.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.626+0000 7f1ebd59a700 1 -- 172.21.15.189:0/1274656070 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb40036a0 con 0x7f1ec015d7b0 2024-04-06T13:47:40.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.626+0000 7f1ec48a0700 1 -- 172.21.15.189:0/1274656070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ec015d7b0 msgr2=0x7f1ec00b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:40.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.626+0000 7f1ec48a0700 1 --2- 172.21.15.189:0/1274656070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ec015d7b0 0x7f1ec00b3e20 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00673e0 tx=0x7f1eb4009bd0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.627+0000 7f1ec48a0700 1 -- 172.21.15.189:0/1274656070 shutdown_connections 2024-04-06T13:47:40.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.627+0000 7f1ec48a0700 1 --2- 172.21.15.189:0/1274656070 >> v2:172.21.15.192:3300/0 conn(0x7f1ec015e500 0x7f1ec006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.627+0000 7f1ec48a0700 1 --2- 172.21.15.189:0/1274656070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ec015d7b0 0x7f1ec00b3e20 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.627+0000 7f1ec48a0700 1 -- 172.21.15.189:0/1274656070 >> 172.21.15.189:0/1274656070 conn(0x7f1ec00d00b0 msgr2=0x7f1ec00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:40.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.627+0000 7f1ec48a0700 1 -- 172.21.15.189:0/1274656070 shutdown_connections 2024-04-06T13:47:40.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.627+0000 7f1ec48a0700 1 -- 172.21.15.189:0/1274656070 wait complete. 2024-04-06T13:47:40.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.628+0000 7f1ec48a0700 1 Processor -- start 2024-04-06T13:47:40.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.628+0000 7f1ec48a0700 1 -- start start 2024-04-06T13:47:40.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1ec48a0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ec006a730 0x7f1ec015b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1ec48a0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ec015d7b0 0x7f1ec015f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1ec48a0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ec015e500 0x7f1ec015fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1ec48a0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec0151930 con 0x7f1ec015e500 2024-04-06T13:47:40.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1ec48a0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec0152790 con 0x7f1ec015d7b0 2024-04-06T13:47:40.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1ec48a0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec0151b80 con 0x7f1ec006a730 2024-04-06T13:47:40.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1ebed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ec015e500 0x7f1ec015fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1ebed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ec015e500 0x7f1ec015fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44536/0 (socket says 172.21.15.189:44536) 2024-04-06T13:47:40.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1ebed9d700 1 -- 172.21.15.189:0/2585812934 learned_addr learned my addr 172.21.15.189:0/2585812934 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:40.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1ebdd9b700 1 --2- 172.21.15.189:0/2585812934 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ec015d7b0 0x7f1ec015f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1ebe59c700 1 --2- 172.21.15.189:0/2585812934 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ec006a730 0x7f1ec015b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1ebed9d700 1 -- 172.21.15.189:0/2585812934 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ec006a730 msgr2=0x7f1ec015b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:40.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1ebed9d700 1 --2- 172.21.15.189:0/2585812934 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ec006a730 0x7f1ec015b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1ebed9d700 1 -- 172.21.15.189:0/2585812934 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ec015d7b0 msgr2=0x7f1ec015f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:40.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1ebed9d700 1 --2- 172.21.15.189:0/2585812934 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ec015d7b0 0x7f1ec015f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:40.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1ebed9d700 1 -- 172.21.15.189:0/2585812934 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb4009550 con 0x7f1ec015e500 2024-04-06T13:47:40.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1ebed9d700 1 --2- 172.21.15.189:0/2585812934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ec015e500 0x7f1ec015fa70 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800cd50 tx=0x7f1eb800a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:40.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1eaeffd700 1 -- 172.21.15.189:0/2585812934 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb800cea0 con 0x7f1ec015e500 2024-04-06T13:47:40.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1eaeffd700 1 -- 172.21.15.189:0/2585812934 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb800f480 con 0x7f1ec015e500 2024-04-06T13:47:40.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1ec48a0700 1 -- 172.21.15.189:0/2585812934 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec0151e60 con 0x7f1ec015e500 2024-04-06T13:47:40.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.629+0000 7f1eaeffd700 1 -- 172.21.15.189:0/2585812934 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb800f830 con 0x7f1ec015e500 2024-04-06T13:47:40.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.630+0000 7f1ec48a0700 1 -- 172.21.15.189:0/2585812934 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec0152420 con 0x7f1ec015e500 2024-04-06T13:47:40.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.630+0000 7f1ec48a0700 1 -- 172.21.15.189:0/2585812934 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec00b3600 con 0x7f1ec015e500 2024-04-06T13:47:40.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.631+0000 7f1eaeffd700 1 -- 172.21.15.189:0/2585812934 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 139) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f1eb800b040 con 0x7f1ec015e500 2024-04-06T13:47:40.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.631+0000 7f1eaeffd700 1 --2- 172.21.15.189:0/2585812934 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ea006adb0 0x7f1ea006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:40.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.631+0000 7f1eaeffd700 1 -- 172.21.15.189:0/2585812934 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f1eb808a1c0 con 0x7f1ec015e500 2024-04-06T13:47:40.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.631+0000 7f1ebe59c700 1 --2- 172.21.15.189:0/2585812934 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ea006adb0 0x7f1ea006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:40.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.632+0000 7f1ebe59c700 1 --2- 172.21.15.189:0/2585812934 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ea006adb0 0x7f1ea006d270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4009130 tx=0x7f1eb4008030 comp rx=0 tx=0).ready entity=mgr.11028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:40.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.636+0000 7f1eaeffd700 1 -- 172.21.15.189:0/2585812934 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1ec00b3600 con 0x7f1ec015e500 2024-04-06T13:47:40.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:40.887+0000 7f1ec48a0700 1 -- 172.21.15.189:0/2585812934 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f1ec00b3600 con 0x7f1ec015e500 2024-04-06T13:47:41.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.623+0000 7f1eaeffd700 1 -- 172.21.15.189:0/2585812934 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 140) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f1eb80881f0 con 0x7f1ec015e500 2024-04-06T13:47:41.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.625+0000 7f1eaeffd700 1 -- 172.21.15.189:0/2585812934 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v140) v1 ==== 92+0+0 (secure 0 0 0) 0x7f1eb8071c10 con 0x7f1ec015e500 2024-04-06T13:47:41.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.627+0000 7f1ec48a0700 1 -- 172.21.15.189:0/2585812934 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ea006adb0 msgr2=0x7f1ea006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:41.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.627+0000 7f1ec48a0700 1 --2- 172.21.15.189:0/2585812934 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ea006adb0 0x7f1ea006d270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4009130 tx=0x7f1eb4008030 comp rx=0 tx=0).stop 2024-04-06T13:47:41.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.627+0000 7f1ec48a0700 1 -- 172.21.15.189:0/2585812934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ec015e500 msgr2=0x7f1ec015fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:41.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.627+0000 7f1ec48a0700 1 --2- 172.21.15.189:0/2585812934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ec015e500 0x7f1ec015fa70 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800cd50 tx=0x7f1eb800a560 comp rx=0 tx=0).stop 2024-04-06T13:47:41.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.627+0000 7f1ec48a0700 1 -- 172.21.15.189:0/2585812934 shutdown_connections 2024-04-06T13:47:41.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.627+0000 7f1ec48a0700 1 --2- 172.21.15.189:0/2585812934 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1ea006adb0 0x7f1ea006d270 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:41.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.627+0000 7f1ec48a0700 1 --2- 172.21.15.189:0/2585812934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ec015e500 0x7f1ec015fa70 unknown :-1 s=CLOSED pgs=2486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:41.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.627+0000 7f1ec48a0700 1 --2- 172.21.15.189:0/2585812934 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ec006a730 0x7f1ec015b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:41.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.627+0000 7f1ec48a0700 1 --2- 172.21.15.189:0/2585812934 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ec015d7b0 0x7f1ec015f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:41.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.627+0000 7f1ec48a0700 1 -- 172.21.15.189:0/2585812934 >> 172.21.15.189:0/2585812934 conn(0x7f1ec00d00b0 msgr2=0x7f1ec006eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:41.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.628+0000 7f1ec48a0700 1 -- 172.21.15.189:0/2585812934 shutdown_connections 2024-04-06T13:47:41.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.628+0000 7f1ec48a0700 1 -- 172.21.15.189:0/2585812934 wait complete. 2024-04-06T13:47:41.642 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-06T13:47:41.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.773+0000 7ff944ce4700 1 Processor -- start 2024-04-06T13:47:41.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.773+0000 7ff944ce4700 1 -- start start 2024-04-06T13:47:41.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.774+0000 7ff944ce4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff9400b4ed0 0x7ff9400b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:41.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.774+0000 7ff944ce4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff9400d2880 0x7ff9400d2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:41.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.774+0000 7ff944ce4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9400b5cc0 con 0x7ff94015d590 2024-04-06T13:47:41.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.774+0000 7ff944ce4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9400b58c0 con 0x7ff9400b4ed0 2024-04-06T13:47:41.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.774+0000 7ff944ce4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9400d1500 con 0x7ff9400d2880 2024-04-06T13:47:41.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.774+0000 7ff93e59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff94015d590 0x7ff9400b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58194/0 (socket says 172.21.15.189:58194) 2024-04-06T13:47:41.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.774+0000 7ff93e59c700 1 -- 172.21.15.189:0/4151710930 learned_addr learned my addr 172.21.15.189:0/4151710930 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:41.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.774+0000 7ff93ed9d700 1 --2- 172.21.15.189:0/4151710930 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff9400d2880 0x7ff9400d2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:41.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.774+0000 7ff93dd9b700 1 --2- 172.21.15.189:0/4151710930 >> v2:172.21.15.192:3300/0 conn(0x7ff9400b4ed0 0x7ff9400b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:41.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.774+0000 7ff93d59a700 1 -- 172.21.15.189:0/4151710930 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039949130 0 0) 0x7ff9400b5cc0 con 0x7ff94015d590 2024-04-06T13:47:41.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.774+0000 7ff93d59a700 1 -- 172.21.15.189:0/4151710930 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9280035b0 con 0x7ff94015d590 2024-04-06T13:47:41.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.774+0000 7ff93dd9b700 1 -- 172.21.15.189:0/4151710930 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff9400d2880 msgr2=0x7ff9400d2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:41.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.774+0000 7ff93dd9b700 1 --2- 172.21.15.189:0/4151710930 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff9400d2880 0x7ff9400d2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:41.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.775+0000 7ff93dd9b700 1 -- 172.21.15.189:0/4151710930 >> v1:172.21.15.189:6789/0 conn(0x7ff94015d590 legacy=0x7ff9400b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:41.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.775+0000 7ff93dd9b700 1 -- 172.21.15.189:0/4151710930 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9400d17e0 con 0x7ff9400b4ed0 2024-04-06T13:47:41.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.775+0000 7ff93ed9d700 1 --2- 172.21.15.189:0/4151710930 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff9400d2880 0x7ff9400d2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:41.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.775+0000 7ff93dd9b700 1 --2- 172.21.15.189:0/4151710930 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9400b4ed0 0x7ff9400b5380 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff934009840 tx=0x7ff934009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7654aca408d39507 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:41.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.775+0000 7ff93d59a700 1 -- 172.21.15.189:0/4151710930 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff934011830 con 0x7ff9400b4ed0 2024-04-06T13:47:41.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.775+0000 7ff93d59a700 1 -- 172.21.15.189:0/4151710930 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff934011990 con 0x7ff9400b4ed0 2024-04-06T13:47:41.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.775+0000 7ff93d59a700 1 -- 172.21.15.189:0/4151710930 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff934011d20 con 0x7ff9400b4ed0 2024-04-06T13:47:41.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.775+0000 7ff944ce4700 1 -- 172.21.15.189:0/4151710930 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9400b4ed0 msgr2=0x7ff9400b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:41.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.775+0000 7ff944ce4700 1 --2- 172.21.15.189:0/4151710930 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9400b4ed0 0x7ff9400b5380 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff934009840 tx=0x7ff934009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:41.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.776+0000 7ff944ce4700 1 -- 172.21.15.189:0/4151710930 shutdown_connections 2024-04-06T13:47:41.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.776+0000 7ff944ce4700 1 --2- 172.21.15.189:0/4151710930 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff9400d2880 0x7ff9400d2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:41.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.776+0000 7ff944ce4700 1 --2- 172.21.15.189:0/4151710930 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9400b4ed0 0x7ff9400b5380 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:41.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.776+0000 7ff944ce4700 1 -- 172.21.15.189:0/4151710930 >> 172.21.15.189:0/4151710930 conn(0x7ff94006bbc0 msgr2=0x7ff94006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:41.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.776+0000 7ff944ce4700 1 -- 172.21.15.189:0/4151710930 shutdown_connections 2024-04-06T13:47:41.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.776+0000 7ff944ce4700 1 -- 172.21.15.189:0/4151710930 wait complete. 2024-04-06T13:47:41.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.777+0000 7ff944ce4700 1 Processor -- start 2024-04-06T13:47:41.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.777+0000 7ff944ce4700 1 -- start start 2024-04-06T13:47:41.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff944ce4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9400b4ed0 0x7ff940160370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:41.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff944ce4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff9400d2880 0x7ff94015bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:41.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff944ce4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff94015d590 0x7ff94014f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:41.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff944ce4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9400cb960 con 0x7ff94015d590 2024-04-06T13:47:41.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff944ce4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94015c0c0 con 0x7ff9400b4ed0 2024-04-06T13:47:41.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff944ce4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94015c320 con 0x7ff9400d2880 2024-04-06T13:47:41.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff93ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff94015d590 0x7ff94014f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:41.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff93e59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9400b4ed0 0x7ff940160370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:41.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff93dd9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff9400d2880 0x7ff94015bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:41.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff93ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff94015d590 0x7ff94014f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44560/0 (socket says 172.21.15.189:44560) 2024-04-06T13:47:41.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff93dd9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff9400d2880 0x7ff94015bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:51004/0 (socket says 172.21.15.189:51004) 2024-04-06T13:47:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff93ed9d700 1 -- 172.21.15.189:0/1361620150 learned_addr learned my addr 172.21.15.189:0/1361620150 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff93e59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9400b4ed0 0x7ff940160370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:57252/0 (socket says 172.21.15.189:57252) 2024-04-06T13:47:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff93ed9d700 1 -- 172.21.15.189:0/1361620150 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff9400d2880 msgr2=0x7ff94015bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff93ed9d700 1 --2- 172.21.15.189:0/1361620150 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff9400d2880 0x7ff94015bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff93ed9d700 1 -- 172.21.15.189:0/1361620150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9400b4ed0 msgr2=0x7ff940160370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff93ed9d700 1 --2- 172.21.15.189:0/1361620150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9400b4ed0 0x7ff940160370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff93ed9d700 1 -- 172.21.15.189:0/1361620150 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff938009a70 con 0x7ff94015d590 2024-04-06T13:47:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff93dd9b700 1 --2- 172.21.15.189:0/1361620150 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff9400d2880 0x7ff94015bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff93ed9d700 1 --2- 172.21.15.189:0/1361620150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff94015d590 0x7ff94014f920 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7ff93800efc0 tx=0x7ff93800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.778+0000 7ff92effd700 1 -- 172.21.15.189:0/1361620150 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93800ca90 con 0x7ff94015d590 2024-04-06T13:47:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.779+0000 7ff92effd700 1 -- 172.21.15.189:0/1361620150 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93800f040 con 0x7ff94015d590 2024-04-06T13:47:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.779+0000 7ff944ce4700 1 -- 172.21.15.189:0/1361620150 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9340094a0 con 0x7ff94015d590 2024-04-06T13:47:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.779+0000 7ff92effd700 1 -- 172.21.15.189:0/1361620150 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93800ce10 con 0x7ff94015d590 2024-04-06T13:47:41.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.779+0000 7ff944ce4700 1 -- 172.21.15.189:0/1361620150 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94015c8e0 con 0x7ff94015d590 2024-04-06T13:47:41.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.779+0000 7ff944ce4700 1 -- 172.21.15.189:0/1361620150 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9400cbaa0 con 0x7ff94015d590 2024-04-06T13:47:41.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.780+0000 7ff92effd700 1 -- 172.21.15.189:0/1361620150 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 140) v1 ==== 89264+0+0 (secure 0 0 0) 0x7ff938014480 con 0x7ff94015d590 2024-04-06T13:47:41.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.780+0000 7ff92effd700 1 --2- 172.21.15.189:0/1361620150 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff92006ae00 0x7ff92006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:41.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.780+0000 7ff93e59c700 1 -- 172.21.15.189:0/1361620150 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff92006ae00 msgr2=0x7ff92006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:41.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.781+0000 7ff92effd700 1 -- 172.21.15.189:0/1361620150 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6965+0+0 (secure 0 0 0) 0x7ff93808a0e0 con 0x7ff94015d590 2024-04-06T13:47:41.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.781+0000 7ff93e59c700 1 --2- 172.21.15.189:0/1361620150 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff92006ae00 0x7ff92006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:41.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.785+0000 7ff92effd700 1 -- 172.21.15.189:0/1361620150 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff9400cbaa0 con 0x7ff94015d590 2024-04-06T13:47:41.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.982+0000 7ff93e59c700 1 -- 172.21.15.189:0/1361620150 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff92006ae00 msgr2=0x7ff92006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:41.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.982+0000 7ff93e59c700 1 --2- 172.21.15.189:0/1361620150 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff92006ae00 0x7ff92006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:41.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.989+0000 7ff944ce4700 1 -- 172.21.15.189:0/1361620150 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7ff9400cbaa0 con 0x7ff94015d590 2024-04-06T13:47:41.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.994+0000 7ff92effd700 1 -- 172.21.15.189:0/1361620150 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v77)=0 v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff9400cbaa0 con 0x7ff94015d590 2024-04-06T13:47:41.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.997+0000 7ff944ce4700 1 -- 172.21.15.189:0/1361620150 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff92006ae00 msgr2=0x7ff92006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:41.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.997+0000 7ff944ce4700 1 --2- 172.21.15.189:0/1361620150 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff92006ae00 0x7ff92006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:41.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.997+0000 7ff944ce4700 1 -- 172.21.15.189:0/1361620150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff94015d590 msgr2=0x7ff94014f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:41.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.997+0000 7ff944ce4700 1 --2- 172.21.15.189:0/1361620150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff94015d590 0x7ff94014f920 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7ff93800efc0 tx=0x7ff93800c3a0 comp rx=0 tx=0).stop 2024-04-06T13:47:41.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.997+0000 7ff944ce4700 1 -- 172.21.15.189:0/1361620150 shutdown_connections 2024-04-06T13:47:41.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.997+0000 7ff944ce4700 1 --2- 172.21.15.189:0/1361620150 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff92006ae00 0x7ff92006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:41.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.997+0000 7ff944ce4700 1 --2- 172.21.15.189:0/1361620150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff94015d590 0x7ff94014f920 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:41.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.997+0000 7ff944ce4700 1 --2- 172.21.15.189:0/1361620150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9400b4ed0 0x7ff940160370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:41.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.997+0000 7ff944ce4700 1 --2- 172.21.15.189:0/1361620150 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff9400d2880 0x7ff94015bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:41.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.997+0000 7ff944ce4700 1 -- 172.21.15.189:0/1361620150 >> 172.21.15.189:0/1361620150 conn(0x7ff94006bbc0 msgr2=0x7ff94006bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:41.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.997+0000 7ff944ce4700 1 -- 172.21.15.189:0/1361620150 shutdown_connections 2024-04-06T13:47:41.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:41.997+0000 7ff944ce4700 1 -- 172.21.15.189:0/1361620150 wait complete. 2024-04-06T13:47:42.006 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:42.007+0000 7f844accb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:47:42.009 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-06T13:47:42.023 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:42.023+0000 7fb42f122200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:47:42.084 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:42.084+0000 7f844accb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:47:42.106 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:42.107+0000 7fb42f122200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:47:42.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.137+0000 7fa3ee1f3700 1 Processor -- start 2024-04-06T13:47:42.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.137+0000 7fa3ee1f3700 1 -- start start 2024-04-06T13:47:42.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.138+0000 7fa3ee1f3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3e80d0a60 0x7fa3e80d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:42.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.138+0000 7fa3ee1f3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e80cdf80 con 0x7fa3e80d1af0 2024-04-06T13:47:42.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.138+0000 7fa3ee1f3700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e80d3230 con 0x7fa3e815d7e0 2024-04-06T13:47:42.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.138+0000 7fa3ee1f3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e80d1f80 con 0x7fa3e80d0a60 2024-04-06T13:47:42.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.138+0000 7fa3e7fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa3e80d1af0 0x7fa3e80b66b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58200/0 (socket says 172.21.15.189:58200) 2024-04-06T13:47:42.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.138+0000 7fa3e7fff700 1 -- 172.21.15.189:0/1996566867 learned_addr learned my addr 172.21.15.189:0/1996566867 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:42.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.138+0000 7fa3e6ffd700 1 --2- 172.21.15.189:0/1996566867 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3e80d0a60 0x7fa3e80d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:42.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.138+0000 7fa3e67fc700 1 -- 172.21.15.189:0/1996566867 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019864109 0 0) 0x7fa3e80cdf80 con 0x7fa3e80d1af0 2024-04-06T13:47:42.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.138+0000 7fa3e67fc700 1 -- 172.21.15.189:0/1996566867 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3d40035b0 con 0x7fa3e80d1af0 2024-04-06T13:47:42.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.139+0000 7fa3e67fc700 1 -- 172.21.15.189:0/1996566867 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899125855 0 0) 0x7fa3d8001490 con 0x7fa3e815d7e0 2024-04-06T13:47:42.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.139+0000 7fa3e67fc700 1 -- 172.21.15.189:0/1996566867 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3e80cdf80 con 0x7fa3e815d7e0 2024-04-06T13:47:42.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.139+0000 7fa3e6ffd700 1 -- 172.21.15.189:0/1996566867 >> v1:172.21.15.192:6789/0 conn(0x7fa3e815d7e0 legacy=0x7fa3e80d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:42.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.139+0000 7fa3e6ffd700 1 -- 172.21.15.189:0/1996566867 >> v1:172.21.15.189:6789/0 conn(0x7fa3e80d1af0 legacy=0x7fa3e80b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:42.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.139+0000 7fa3e6ffd700 1 -- 172.21.15.189:0/1996566867 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e80d2da0 con 0x7fa3e80d0a60 2024-04-06T13:47:42.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.139+0000 7fa3e6ffd700 1 --2- 172.21.15.189:0/1996566867 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3e80d0a60 0x7fa3e80d15b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0009870 tx=0x7fa3d0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a9dee23628453e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:42.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.139+0000 7fa3e67fc700 1 -- 172.21.15.189:0/1996566867 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0004230 con 0x7fa3e80d0a60 2024-04-06T13:47:42.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.139+0000 7fa3e67fc700 1 -- 172.21.15.189:0/1996566867 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d0004390 con 0x7fa3e80d0a60 2024-04-06T13:47:42.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.139+0000 7fa3e67fc700 1 -- 172.21.15.189:0/1996566867 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d00036a0 con 0x7fa3e80d0a60 2024-04-06T13:47:42.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.140+0000 7fa3ee1f3700 1 -- 172.21.15.189:0/1996566867 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3e80d0a60 msgr2=0x7fa3e80d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:42.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.140+0000 7fa3ee1f3700 1 --2- 172.21.15.189:0/1996566867 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3e80d0a60 0x7fa3e80d15b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0009870 tx=0x7fa3d0009b80 comp rx=0 tx=0).stop 2024-04-06T13:47:42.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.140+0000 7fa3ee1f3700 1 -- 172.21.15.189:0/1996566867 shutdown_connections 2024-04-06T13:47:42.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.140+0000 7fa3ee1f3700 1 --2- 172.21.15.189:0/1996566867 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3e80d0a60 0x7fa3e80d15b0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:42.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.140+0000 7fa3ee1f3700 1 -- 172.21.15.189:0/1996566867 >> 172.21.15.189:0/1996566867 conn(0x7fa3e8072250 msgr2=0x7fa3e8072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:42.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.140+0000 7fa3ee1f3700 1 -- 172.21.15.189:0/1996566867 shutdown_connections 2024-04-06T13:47:42.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.140+0000 7fa3ee1f3700 1 -- 172.21.15.189:0/1996566867 wait complete. 2024-04-06T13:47:42.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.141+0000 7fa3ee1f3700 1 Processor -- start 2024-04-06T13:47:42.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.141+0000 7fa3ee1f3700 1 -- start start 2024-04-06T13:47:42.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.142+0000 7fa3ee1f3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e80d0a60 0x7fa3e815ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:42.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.142+0000 7fa3ee1f3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3e80d1af0 0x7fa3e81f29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:42.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.142+0000 7fa3ee1f3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3e815d7e0 0x7fa3e80cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:42.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.142+0000 7fa3ee1f3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e80b4cc0 con 0x7fa3e80d0a60 2024-04-06T13:47:42.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.142+0000 7fa3ee1f3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e80a9f40 con 0x7fa3e815d7e0 2024-04-06T13:47:42.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.142+0000 7fa3ee1f3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e80ca640 con 0x7fa3e80d1af0 2024-04-06T13:47:42.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.142+0000 7fa3e77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e80d0a60 0x7fa3e815ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:42.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.142+0000 7fa3e6ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3e80d1af0 0x7fa3e81f29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:42.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.142+0000 7fa3e7fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3e815d7e0 0x7fa3e80cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:42.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.142+0000 7fa3e77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e80d0a60 0x7fa3e815ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44570/0 (socket says 172.21.15.189:44570) 2024-04-06T13:47:42.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.142+0000 7fa3e6ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3e80d1af0 0x7fa3e81f29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:51014/0 (socket says 172.21.15.189:51014) 2024-04-06T13:47:42.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.142+0000 7fa3e77fe700 1 -- 172.21.15.189:0/2818040604 learned_addr learned my addr 172.21.15.189:0/2818040604 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:42.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.142+0000 7fa3e6ffd700 1 -- 172.21.15.189:0/2818040604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3e815d7e0 msgr2=0x7fa3e80cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:42.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.142+0000 7fa3e6ffd700 1 --2- 172.21.15.189:0/2818040604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3e815d7e0 0x7fa3e80cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:42.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.142+0000 7fa3e6ffd700 1 -- 172.21.15.189:0/2818040604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e80d0a60 msgr2=0x7fa3e815ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:42.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.142+0000 7fa3e6ffd700 1 --2- 172.21.15.189:0/2818040604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e80d0a60 0x7fa3e815ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:42.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.142+0000 7fa3e6ffd700 1 -- 172.21.15.189:0/2818040604 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d00094a0 con 0x7fa3e80d1af0 2024-04-06T13:47:42.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.142+0000 7fa3e7fff700 1 --2- 172.21.15.189:0/2818040604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3e815d7e0 0x7fa3e80cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:42.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.142+0000 7fa3e6ffd700 1 --2- 172.21.15.189:0/2818040604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3e80d1af0 0x7fa3e81f29a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa3e806c960 tx=0x7fa3d0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:42.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.143+0000 7fa3dffff700 1 -- 172.21.15.189:0/2818040604 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0003d20 con 0x7fa3e80d1af0 2024-04-06T13:47:42.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.143+0000 7fa3dffff700 1 -- 172.21.15.189:0/2818040604 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d001e070 con 0x7fa3e80d1af0 2024-04-06T13:47:42.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.143+0000 7fa3ee1f3700 1 -- 172.21.15.189:0/2818040604 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e80ca920 con 0x7fa3e80d1af0 2024-04-06T13:47:42.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.143+0000 7fa3dffff700 1 -- 172.21.15.189:0/2818040604 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0011870 con 0x7fa3e80d1af0 2024-04-06T13:47:42.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.143+0000 7fa3ee1f3700 1 -- 172.21.15.189:0/2818040604 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e80caee0 con 0x7fa3e80d1af0 2024-04-06T13:47:42.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.144+0000 7fa3ee1f3700 1 -- 172.21.15.189:0/2818040604 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e80cb0d0 con 0x7fa3e80d1af0 2024-04-06T13:47:42.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.144+0000 7fa3dffff700 1 -- 172.21.15.189:0/2818040604 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 140) v1 ==== 89264+0+0 (secure 0 0 0) 0x7fa3d00119d0 con 0x7fa3e80d1af0 2024-04-06T13:47:42.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.144+0000 7fa3dffff700 1 --2- 172.21.15.189:0/2818040604 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa3c806ae00 0x7fa3c806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:42.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.144+0000 7fa3e77fe700 1 -- 172.21.15.189:0/2818040604 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa3c806ae00 msgr2=0x7fa3c806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:42.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.144+0000 7fa3dffff700 1 -- 172.21.15.189:0/2818040604 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6965+0+0 (secure 0 0 0) 0x7fa3d008a870 con 0x7fa3e80d1af0 2024-04-06T13:47:42.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.144+0000 7fa3e77fe700 1 --2- 172.21.15.189:0/2818040604 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa3c806ae00 0x7fa3c806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:42.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.149+0000 7fa3dffff700 1 -- 172.21.15.189:0/2818040604 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa3d00554e0 con 0x7fa3e80d1af0 2024-04-06T13:47:42.157 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:42.157+0000 7f844accb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:47:42.184 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:42.184+0000 7fb42f122200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:47:42.227 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:42.227+0000 7f844accb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:47:42.256 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:42.256+0000 7fb42f122200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:47:42.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.345+0000 7fa3e77fe700 1 -- 172.21.15.189:0/2818040604 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa3c806ae00 msgr2=0x7fa3c806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:42.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.345+0000 7fa3e77fe700 1 --2- 172.21.15.189:0/2818040604 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa3c806ae00 0x7fa3c806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:42.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.356+0000 7fa3ee1f3700 1 -- 172.21.15.189:0/2818040604 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fa3e8074990 con 0x7fa3e80d1af0 2024-04-06T13:47:42.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.361+0000 7fa3dffff700 1 -- 172.21.15.189:0/2818040604 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v78)=0 v78) v1 ==== 133+0+0 (secure 0 0 0) 0x7fa3d00200c0 con 0x7fa3e80d1af0 2024-04-06T13:47:42.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.363+0000 7fa3ee1f3700 1 -- 172.21.15.189:0/2818040604 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa3c806ae00 msgr2=0x7fa3c806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:42.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.363+0000 7fa3ee1f3700 1 --2- 172.21.15.189:0/2818040604 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa3c806ae00 0x7fa3c806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:42.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.363+0000 7fa3ee1f3700 1 -- 172.21.15.189:0/2818040604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3e80d1af0 msgr2=0x7fa3e81f29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:42.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.363+0000 7fa3ee1f3700 1 --2- 172.21.15.189:0/2818040604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3e80d1af0 0x7fa3e81f29a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa3e806c960 tx=0x7fa3d0004750 comp rx=0 tx=0).stop 2024-04-06T13:47:42.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.363+0000 7fa3ee1f3700 1 -- 172.21.15.189:0/2818040604 shutdown_connections 2024-04-06T13:47:42.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.363+0000 7fa3ee1f3700 1 --2- 172.21.15.189:0/2818040604 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa3c806ae00 0x7fa3c806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:42.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.363+0000 7fa3ee1f3700 1 --2- 172.21.15.189:0/2818040604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3e815d7e0 0x7fa3e80cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:42.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.363+0000 7fa3ee1f3700 1 --2- 172.21.15.189:0/2818040604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e80d0a60 0x7fa3e815ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:42.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.363+0000 7fa3ee1f3700 1 --2- 172.21.15.189:0/2818040604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3e80d1af0 0x7fa3e81f29a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:42.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.363+0000 7fa3ee1f3700 1 -- 172.21.15.189:0/2818040604 >> 172.21.15.189:0/2818040604 conn(0x7fa3e8072250 msgr2=0x7fa3e80b84a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:42.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.363+0000 7fa3ee1f3700 1 -- 172.21.15.189:0/2818040604 shutdown_connections 2024-04-06T13:47:42.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.363+0000 7fa3ee1f3700 1 -- 172.21.15.189:0/2818040604 wait complete. 2024-04-06T13:47:42.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:42.377 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2024-04-06T13:47:42.403 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:42.403+0000 7f844accb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:47:42.437 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-04-06T13:47:42.440 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:42.440+0000 7fb42f122200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:47:42.561 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:42.561+0000 7f844accb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:47:42.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.568+0000 7efecfb83700 1 Processor -- start 2024-04-06T13:47:42.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.568+0000 7efecfb83700 1 -- start start 2024-04-06T13:47:42.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.570+0000 7efecfb83700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efec8158fe0 0x7efec806a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:42.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.570+0000 7efecfb83700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efec806ac70 0x7efec815bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:42.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.570+0000 7efecfb83700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efec815c100 0x7efec80717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:42.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.570+0000 7efecfb83700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efec806b0f0 con 0x7efec815c100 2024-04-06T13:47:42.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.570+0000 7efecfb83700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efec815f9e0 con 0x7efec8158fe0 2024-04-06T13:47:42.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.570+0000 7efecfb83700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec815f0d0 con 0x7efec806ac70 2024-04-06T13:47:42.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.570+0000 7efece120700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efec815c100 0x7efec80717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:42.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.570+0000 7efece120700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efec815c100 0x7efec80717b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44582/0 (socket says 172.21.15.189:44582) 2024-04-06T13:47:42.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.570+0000 7efece120700 1 -- 172.21.15.189:0/588600264 learned_addr learned my addr 172.21.15.189:0/588600264 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:42.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.570+0000 7efecd11e700 1 --2- 172.21.15.189:0/588600264 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efec806ac70 0x7efec815bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:42.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.570+0000 7efecd91f700 1 --2- 172.21.15.189:0/588600264 >> v2:172.21.15.192:3300/0 conn(0x7efec8158fe0 0x7efec806a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:42.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.570+0000 7efece120700 1 -- 172.21.15.189:0/588600264 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efec806ac70 msgr2=0x7efec815bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:42.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.570+0000 7efece120700 1 --2- 172.21.15.189:0/588600264 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efec806ac70 0x7efec815bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:42.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.570+0000 7efece120700 1 -- 172.21.15.189:0/588600264 >> v2:172.21.15.192:3300/0 conn(0x7efec8158fe0 msgr2=0x7efec806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:42.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.570+0000 7efece120700 1 --2- 172.21.15.189:0/588600264 >> v2:172.21.15.192:3300/0 conn(0x7efec8158fe0 0x7efec806a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:42.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.570+0000 7efece120700 1 -- 172.21.15.189:0/588600264 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec815c8f0 con 0x7efec815c100 2024-04-06T13:47:42.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.570+0000 7efecd91f700 1 --2- 172.21.15.189:0/588600264 >> v2:172.21.15.192:3300/0 conn(0x7efec8158fe0 0x7efec806a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:42.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.571+0000 7efece120700 1 --2- 172.21.15.189:0/588600264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efec815c100 0x7efec80717b0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7efec8073470 tx=0x7efec000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b04dfc3edea1e93f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:42.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.571+0000 7efecc91d700 1 -- 172.21.15.189:0/588600264 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec000e070 con 0x7efec815c100 2024-04-06T13:47:42.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.571+0000 7efecc91d700 1 -- 172.21.15.189:0/588600264 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec000b910 con 0x7efec815c100 2024-04-06T13:47:42.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.571+0000 7efecc91d700 1 -- 172.21.15.189:0/588600264 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec000bcc0 con 0x7efec815c100 2024-04-06T13:47:42.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.572+0000 7efecfb83700 1 -- 172.21.15.189:0/588600264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efec815c100 msgr2=0x7efec80717b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:42.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.572+0000 7efecfb83700 1 --2- 172.21.15.189:0/588600264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efec815c100 0x7efec80717b0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7efec8073470 tx=0x7efec000b330 comp rx=0 tx=0).stop 2024-04-06T13:47:42.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.572+0000 7efecfb83700 1 -- 172.21.15.189:0/588600264 shutdown_connections 2024-04-06T13:47:42.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.572+0000 7efecfb83700 1 --2- 172.21.15.189:0/588600264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efec815c100 0x7efec80717b0 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:42.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.572+0000 7efecfb83700 1 --2- 172.21.15.189:0/588600264 >> v2:172.21.15.192:3300/0 conn(0x7efec8158fe0 0x7efec806a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:42.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.572+0000 7efecfb83700 1 --2- 172.21.15.189:0/588600264 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efec806ac70 0x7efec815bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:42.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.572+0000 7efecfb83700 1 -- 172.21.15.189:0/588600264 >> 172.21.15.189:0/588600264 conn(0x7efec80d00b0 msgr2=0x7efec80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:42.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.572+0000 7efecfb83700 1 -- 172.21.15.189:0/588600264 shutdown_connections 2024-04-06T13:47:42.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.572+0000 7efecfb83700 1 -- 172.21.15.189:0/588600264 wait complete. 2024-04-06T13:47:42.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.573+0000 7efecfb83700 1 Processor -- start 2024-04-06T13:47:42.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.573+0000 7efecfb83700 1 -- start start 2024-04-06T13:47:42.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efecfb83700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efec806ac70 0x7efec80cf920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:42.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efecfb83700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efec8158fe0 0x7efec8157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:42.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efecfb83700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efec815c100 0x7efec815dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:42.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efecfb83700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec815a860 con 0x7efec806ac70 2024-04-06T13:47:42.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efecfb83700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec80cfe60 con 0x7efec8158fe0 2024-04-06T13:47:42.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efecfb83700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec81f0cf0 con 0x7efec815c100 2024-04-06T13:47:42.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efecd91f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efec806ac70 0x7efec80cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:42.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efecd91f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efec806ac70 0x7efec80cf920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44594/0 (socket says 172.21.15.189:44594) 2024-04-06T13:47:42.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efecd91f700 1 -- 172.21.15.189:0/1792213333 learned_addr learned my addr 172.21.15.189:0/1792213333 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:42.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efece120700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efec815c100 0x7efec815dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:42.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efecd11e700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efec8158fe0 0x7efec8157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:42.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efecd91f700 1 -- 172.21.15.189:0/1792213333 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efec815c100 msgr2=0x7efec815dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:42.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efecd91f700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efec815c100 0x7efec815dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:42.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efecd91f700 1 -- 172.21.15.189:0/1792213333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efec8158fe0 msgr2=0x7efec8157cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:42.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efecd91f700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efec8158fe0 0x7efec8157cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:42.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efecd91f700 1 -- 172.21.15.189:0/1792213333 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec40094a0 con 0x7efec806ac70 2024-04-06T13:47:42.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efecd91f700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efec806ac70 0x7efec80cf920 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7efec400ebe0 tx=0x7efec400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:42.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efebe7fc700 1 -- 172.21.15.189:0/1792213333 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec400c850 con 0x7efec806ac70 2024-04-06T13:47:42.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efebe7fc700 1 -- 172.21.15.189:0/1792213333 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec400c9b0 con 0x7efec806ac70 2024-04-06T13:47:42.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efecfb83700 1 -- 172.21.15.189:0/1792213333 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec0009b30 con 0x7efec806ac70 2024-04-06T13:47:42.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.574+0000 7efebe7fc700 1 -- 172.21.15.189:0/1792213333 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec400cd40 con 0x7efec806ac70 2024-04-06T13:47:42.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.575+0000 7efecfb83700 1 -- 172.21.15.189:0/1792213333 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec815a580 con 0x7efec806ac70 2024-04-06T13:47:42.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.575+0000 7efecfb83700 1 -- 172.21.15.189:0/1792213333 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec813ab60 con 0x7efec806ac70 2024-04-06T13:47:42.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.577+0000 7efebe7fc700 1 -- 172.21.15.189:0/1792213333 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 140) v1 ==== 89264+0+0 (secure 0 0 0) 0x7efec4013470 con 0x7efec806ac70 2024-04-06T13:47:42.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.577+0000 7efebe7fc700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efeac06ae00 0x7efeac06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:42.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.577+0000 7efecd11e700 1 -- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efeac06ae00 msgr2=0x7efeac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:42.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.577+0000 7efebe7fc700 1 -- 172.21.15.189:0/1792213333 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6965+0+0 (secure 0 0 0) 0x7efec408a270 con 0x7efec806ac70 2024-04-06T13:47:42.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.577+0000 7efecd11e700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efeac06ae00 0x7efeac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:42.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.584+0000 7efebe7fc700 1 -- 172.21.15.189:0/1792213333 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efec4054e30 con 0x7efec806ac70 2024-04-06T13:47:42.612 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:42.612+0000 7fb42f122200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:47:42.676 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:42.676+0000 7f844accb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:47:42.727 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:42.727+0000 7fb42f122200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:47:42.758 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:42.758+0000 7f844accb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:47:42.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.778+0000 7efecd11e700 1 -- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efeac06ae00 msgr2=0x7efeac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:42.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.778+0000 7efecd11e700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efeac06ae00 0x7efeac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:42.811 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:42.811+0000 7fb42f122200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:47:42.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:42.819+0000 7efecfb83700 1 -- 172.21.15.189:0/1792213333 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7efec81593c0 con 0x7efeac06ae00 2024-04-06T13:47:42.832 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:42.832+0000 7f844accb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:47:42.886 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:42.887+0000 7fb42f122200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:47:43.036 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:43.036+0000 7f844accb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:47:43.089 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:43.089+0000 7fb42f122200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:47:43.113 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:43.113+0000 7f844accb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:47:43.165 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:43.165+0000 7fb42f122200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:47:43.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:43.179+0000 7efecd11e700 1 -- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efeac06ae00 msgr2=0x7efeac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:43.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:43.179+0000 7efecd11e700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efeac06ae00 0x7efeac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-06T13:47:43.433 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:43.433+0000 7f844accb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:47:43.483 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:43.483+0000 7fb42f122200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:47:43.753 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:43.753+0000 7f844accb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:47:43.803 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:43.803+0000 7fb42f122200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:47:43.913 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:43.913+0000 7f844accb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:47:43.967 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:43.964+0000 7fb42f122200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:47:43.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:43.980+0000 7efecd11e700 1 -- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efeac06ae00 msgr2=0x7efeac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:43.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:43.980+0000 7efecd11e700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efeac06ae00 0x7efeac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-06T13:47:43.985 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:43.985+0000 7f844accb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:47:44.037 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:44.037+0000 7fb42f122200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:47:44.599 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:44.599+0000 7f844accb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:47:44.654 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:44.654+0000 7fb42f122200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:47:44.677 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:44.677+0000 7f844accb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:47:44.734 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:44.734+0000 7fb42f122200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:47:44.803 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:44.802+0000 7f844accb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:47:44.859 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:44.859+0000 7fb42f122200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:47:45.313 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:45.313+0000 7f844accb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:47:45.370 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:45.370+0000 7fb42f122200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:47:45.467 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:45.467+0000 7f844accb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:47:45.525 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:45.525+0000 7fb42f122200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:47:45.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:45.581+0000 7efecd11e700 1 -- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efeac06ae00 msgr2=0x7efeac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:45.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:45.581+0000 7efecd11e700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efeac06ae00 0x7efeac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-06T13:47:45.623 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:45.623+0000 7f844accb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:47:45.683 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:45.683+0000 7fb42f122200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:47:46.353 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:46.353+0000 7f844accb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:47:46.413 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:46.413+0000 7fb42f122200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:47:46.598 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:46.597+0000 7f844accb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:47:46.660 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:46.660+0000 7fb42f122200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:47:46.674 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:46.674+0000 7f844accb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:47:46.738 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:46.738+0000 7fb42f122200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:47:47.050 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:47.050+0000 7f844accb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:47:47.114 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.114+0000 7fb42f122200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:47:47.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:47.120+0000 7efebe7fc700 1 -- 172.21.15.189:0/1792213333 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 141) v1 ==== 89264+0+0 (secure 0 0 0) 0x7efec4055200 con 0x7efec806ac70 2024-04-06T13:47:47.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:47.135+0000 7efebe7fc700 1 -- 172.21.15.189:0/1792213333 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mgrmap(e 142) v1 ==== 88873+0+0 (secure 0 0 0) 0x7efec408abc0 con 0x7efec806ac70 2024-04-06T13:47:47.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:47.135+0000 7efebe7fc700 1 -- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efeac06ae00 msgr2=0x7efeac06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:47.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:47.135+0000 7efebe7fc700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efeac06ae00 0x7efeac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:47.367 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.367+0000 7fb3a4835700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:47.368 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.367+0000 7fb3a4835700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:47.368 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.367+0000 7fb3a4835700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:47.368 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.367+0000 7fb3a4835700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:47.368 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.367+0000 7fb3a4835700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:47.368 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.367+0000 7fb3a182f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:47.368 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.367+0000 7fb3a182f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:47.368 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.367+0000 7fb3a182f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:47.368 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.367+0000 7fb3a182f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:47.368 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.367+0000 7fb3a182f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:47.379 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.379+0000 7fb3a182f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:47.379 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.379+0000 7fb3a182f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:47.379 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.379+0000 7fb3a182f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:47.379 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.379+0000 7fb3a182f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:47.379 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.379+0000 7fb3a182f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:47.379 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.379+0000 7fb3a4835700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:47.380 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.379+0000 7fb3a4835700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:47.380 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.379+0000 7fb3a4835700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:47.380 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.379+0000 7fb3a4835700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:47.380 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:47.379+0000 7fb3a4835700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:48.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.142+0000 7efebe7fc700 1 -- 172.21.15.189:0/1792213333 <== mon.0 v2:172.21.15.189:3300/0 9 ==== mgrmap(e 143) v1 ==== 89000+0+0 (secure 0 0 0) 0x7efec4073fe0 con 0x7efec806ac70 2024-04-06T13:47:48.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.142+0000 7efebe7fc700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efeac06ae00 0x7efeac06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:48.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.142+0000 7efebe7fc700 1 -- 172.21.15.189:0/1792213333 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7efeac01cf00 con 0x7efeac06ae00 2024-04-06T13:47:48.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.144+0000 7efecd11e700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efeac06ae00 0x7efeac06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:48.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.146+0000 7efecd11e700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efeac06ae00 0x7efeac06d2c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efec8143190 tx=0x7efeb8009210 comp rx=0 tx=0).ready entity=mgr.9047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:48.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.207+0000 7efebe7fc700 1 -- 172.21.15.189:0/1792213333 <== mgr.9047 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+53 (secure 0 0 0) 0x7efeac01cf00 con 0x7efeac06ae00 2024-04-06T13:47:48.207 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path mandatory_fs_argument_test_dir 2024-04-06T13:47:48.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.210+0000 7efecfb83700 1 -- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efeac06ae00 msgr2=0x7efeac06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:48.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.210+0000 7efecfb83700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efeac06ae00 0x7efeac06d2c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efec8143190 tx=0x7efeb8009210 comp rx=0 tx=0).stop 2024-04-06T13:47:48.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.210+0000 7efecfb83700 1 -- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efec806ac70 msgr2=0x7efec80cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:48.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.210+0000 7efecfb83700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efec806ac70 0x7efec80cf920 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7efec400ebe0 tx=0x7efec400eef0 comp rx=0 tx=0).stop 2024-04-06T13:47:48.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.210+0000 7efecfb83700 1 -- 172.21.15.189:0/1792213333 shutdown_connections 2024-04-06T13:47:48.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.210+0000 7efecfb83700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efeac06ae00 0x7efeac06d2c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:48.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.210+0000 7efecfb83700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efec815c100 0x7efec815dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:48.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.210+0000 7efecfb83700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efec806ac70 0x7efec80cf920 unknown :-1 s=CLOSED pgs=2492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:48.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.210+0000 7efecfb83700 1 --2- 172.21.15.189:0/1792213333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efec8158fe0 0x7efec8157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:48.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.210+0000 7efecfb83700 1 -- 172.21.15.189:0/1792213333 >> 172.21.15.189:0/1792213333 conn(0x7efec80d00b0 msgr2=0x7efec806ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:48.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.210+0000 7efecfb83700 1 -- 172.21.15.189:0/1792213333 shutdown_connections 2024-04-06T13:47:48.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.210+0000 7efecfb83700 1 -- 172.21.15.189:0/1792213333 wait complete. 2024-04-06T13:47:48.223 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path mandatory_fs_argument_test_dir 2024-04-06T13:47:48.223 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-04-06T13:47:48.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.353+0000 7f9d114ee700 1 Processor -- start 2024-04-06T13:47:48.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.353+0000 7f9d114ee700 1 -- start start 2024-04-06T13:47:48.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.354+0000 7f9d114ee700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9d0c0b5380 0x7f9d0c0d2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:48.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.354+0000 7f9d114ee700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c0b5cc0 con 0x7f9d0c0b4ed0 2024-04-06T13:47:48.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.354+0000 7f9d114ee700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c0b57d0 con 0x7f9d0c15d590 2024-04-06T13:47:48.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.354+0000 7f9d114ee700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d0c0d1500 con 0x7f9d0c0b5380 2024-04-06T13:47:48.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.354+0000 7f9d0a7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9d0c0b4ed0 0x7f9d0c0d2880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58216/0 (socket says 172.21.15.189:58216) 2024-04-06T13:47:48.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.354+0000 7f9d0a7fc700 1 -- 172.21.15.189:0/1455241419 learned_addr learned my addr 172.21.15.189:0/1455241419 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:48.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.354+0000 7f9d0b7fe700 1 --2- 172.21.15.189:0/1455241419 >> v2:172.21.15.192:3300/0 conn(0x7f9d0c0b5380 0x7f9d0c0d2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:48.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.354+0000 7f9d09ffb700 1 -- 172.21.15.189:0/1455241419 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139211002 0 0) 0x7f9d0c0b5cc0 con 0x7f9d0c0b4ed0 2024-04-06T13:47:48.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.354+0000 7f9d09ffb700 1 -- 172.21.15.189:0/1455241419 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cf40035b0 con 0x7f9d0c0b4ed0 2024-04-06T13:47:48.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.354+0000 7f9d09ffb700 1 -- 172.21.15.189:0/1455241419 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160561474 0 0) 0x7f9d0c0b57d0 con 0x7f9d0c15d590 2024-04-06T13:47:48.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.354+0000 7f9d09ffb700 1 -- 172.21.15.189:0/1455241419 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d0c0b5cc0 con 0x7f9d0c15d590 2024-04-06T13:47:48.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.354+0000 7f9d09ffb700 1 -- 172.21.15.189:0/1455241419 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3538141168 0 0) 0x7f9cf40035b0 con 0x7f9d0c0b4ed0 2024-04-06T13:47:48.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.354+0000 7f9d0b7fe700 1 -- 172.21.15.189:0/1455241419 >> v1:172.21.15.192:6789/0 conn(0x7f9d0c15d590 legacy=0x7f9d0c0b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:48.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.354+0000 7f9d0b7fe700 1 -- 172.21.15.189:0/1455241419 >> v1:172.21.15.189:6789/0 conn(0x7f9d0c0b4ed0 legacy=0x7f9d0c0d2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:48.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.354+0000 7f9d0b7fe700 1 -- 172.21.15.189:0/1455241419 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0c0d17e0 con 0x7f9d0c0b5380 2024-04-06T13:47:48.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.354+0000 7f9d0b7fe700 1 --2- 172.21.15.189:0/1455241419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d0c0b5380 0x7f9d0c0d2f90 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009d90 tx=0x7f9d0400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd417219d251ac8c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:48.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.355+0000 7f9d09ffb700 1 -- 172.21.15.189:0/1455241419 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0400e070 con 0x7f9d0c0b5380 2024-04-06T13:47:48.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.355+0000 7f9d09ffb700 1 -- 172.21.15.189:0/1455241419 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d04003a20 con 0x7f9d0c0b5380 2024-04-06T13:47:48.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.355+0000 7f9d09ffb700 1 -- 172.21.15.189:0/1455241419 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00001670 con 0x7f9d0c0b5380 2024-04-06T13:47:48.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.355+0000 7f9d114ee700 1 -- 172.21.15.189:0/1455241419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d0c0b5380 msgr2=0x7f9d0c0d2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:48.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.355+0000 7f9d114ee700 1 --2- 172.21.15.189:0/1455241419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d0c0b5380 0x7f9d0c0d2f90 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009d90 tx=0x7f9d0400b330 comp rx=0 tx=0).stop 2024-04-06T13:47:48.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.355+0000 7f9d114ee700 1 -- 172.21.15.189:0/1455241419 shutdown_connections 2024-04-06T13:47:48.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.355+0000 7f9d114ee700 1 --2- 172.21.15.189:0/1455241419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d0c0b5380 0x7f9d0c0d2f90 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:48.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.355+0000 7f9d114ee700 1 -- 172.21.15.189:0/1455241419 >> 172.21.15.189:0/1455241419 conn(0x7f9d0c06bbc0 msgr2=0x7f9d0c06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:48.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.355+0000 7f9d114ee700 1 -- 172.21.15.189:0/1455241419 shutdown_connections 2024-04-06T13:47:48.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.355+0000 7f9d114ee700 1 -- 172.21.15.189:0/1455241419 wait complete. 2024-04-06T13:47:48.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.357+0000 7f9d114ee700 1 Processor -- start 2024-04-06T13:47:48.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.357+0000 7f9d114ee700 1 -- start start 2024-04-06T13:47:48.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.358+0000 7f9d114ee700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d0c0b4ed0 0x7f9d0c15e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:48.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.358+0000 7f9d114ee700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d0c0b5380 0x7f9d0c1f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:48.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.358+0000 7f9d114ee700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d0c15d590 0x7f9d0c15a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:48.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.358+0000 7f9d114ee700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c15d970 con 0x7f9d0c15d590 2024-04-06T13:47:48.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.358+0000 7f9d114ee700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c15dbc0 con 0x7f9d0c0b4ed0 2024-04-06T13:47:48.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.358+0000 7f9d114ee700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c15dea0 con 0x7f9d0c0b5380 2024-04-06T13:47:48.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.358+0000 7f9d0b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d0c15d590 0x7f9d0c15a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:48.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.358+0000 7f9d0b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d0c15d590 0x7f9d0c15a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44748/0 (socket says 172.21.15.189:44748) 2024-04-06T13:47:48.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.358+0000 7f9d0b7fe700 1 -- 172.21.15.189:0/4264505449 learned_addr learned my addr 172.21.15.189:0/4264505449 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:48.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.358+0000 7f9d0a7fc700 1 --2- 172.21.15.189:0/4264505449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d0c0b5380 0x7f9d0c1f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:48.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.358+0000 7f9d0affd700 1 --2- 172.21.15.189:0/4264505449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d0c0b4ed0 0x7f9d0c15e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:48.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.358+0000 7f9d0b7fe700 1 -- 172.21.15.189:0/4264505449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d0c0b5380 msgr2=0x7f9d0c1f22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:48.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.358+0000 7f9d0b7fe700 1 --2- 172.21.15.189:0/4264505449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d0c0b5380 0x7f9d0c1f22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:48.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.358+0000 7f9d0b7fe700 1 -- 172.21.15.189:0/4264505449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d0c0b4ed0 msgr2=0x7f9d0c15e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:48.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.358+0000 7f9d0b7fe700 1 --2- 172.21.15.189:0/4264505449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d0c0b4ed0 0x7f9d0c15e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:48.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.358+0000 7f9d0b7fe700 1 -- 172.21.15.189:0/4264505449 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d04009a70 con 0x7f9d0c15d590 2024-04-06T13:47:48.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.358+0000 7f9d0b7fe700 1 --2- 172.21.15.189:0/4264505449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d0c15d590 0x7f9d0c15a440 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400b370 tx=0x7f9d04004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:48.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.359+0000 7f9cf37fe700 1 -- 172.21.15.189:0/4264505449 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04004440 con 0x7f9d0c15d590 2024-04-06T13:47:48.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.359+0000 7f9cf37fe700 1 -- 172.21.15.189:0/4264505449 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d040045a0 con 0x7f9d0c15d590 2024-04-06T13:47:48.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.359+0000 7f9d114ee700 1 -- 172.21.15.189:0/4264505449 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0c15e180 con 0x7f9d0c15d590 2024-04-06T13:47:48.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.359+0000 7f9cf37fe700 1 -- 172.21.15.189:0/4264505449 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04004950 con 0x7f9d0c15d590 2024-04-06T13:47:48.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.359+0000 7f9d114ee700 1 -- 172.21.15.189:0/4264505449 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d0c15b790 con 0x7f9d0c15d590 2024-04-06T13:47:48.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.360+0000 7f9d114ee700 1 -- 172.21.15.189:0/4264505449 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d0c15b920 con 0x7f9d0c15d590 2024-04-06T13:47:48.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.360+0000 7f9cf37fe700 1 -- 172.21.15.189:0/4264505449 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 143) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f9d0401d470 con 0x7f9d0c15d590 2024-04-06T13:47:48.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.360+0000 7f9cf37fe700 1 --2- 172.21.15.189:0/4264505449 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9cf806ab30 0x7f9cf806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:48.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.361+0000 7f9cf37fe700 1 -- 172.21.15.189:0/4264505449 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7f9d0408ad60 con 0x7f9d0c15d590 2024-04-06T13:47:48.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.361+0000 7f9d0affd700 1 --2- 172.21.15.189:0/4264505449 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9cf806ab30 0x7f9cf806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:48.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.361+0000 7f9d0affd700 1 --2- 172.21.15.189:0/4264505449 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9cf806ab30 0x7f9cf806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc0011e0 tx=0x7f9cfc009040 comp rx=0 tx=0).ready entity=mgr.9047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:48.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.365+0000 7f9cf37fe700 1 -- 172.21.15.189:0/4264505449 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9d04059690 con 0x7f9d0c15d590 2024-04-06T13:47:48.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.586+0000 7f9d114ee700 1 -- 172.21.15.189:0/4264505449 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d0c15dd30 con 0x7f9cf806ab30 2024-04-06T13:47:48.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.632+0000 7f9cf37fe700 1 -- 172.21.15.189:0/4264505449 <== mgr.9047 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f9d0c15dd30 con 0x7f9cf806ab30 2024-04-06T13:47:48.634 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule activated for path mandatory_fs_argument_test_dir 2024-04-06T13:47:48.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.634+0000 7f9d114ee700 1 -- 172.21.15.189:0/4264505449 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9cf806ab30 msgr2=0x7f9cf806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:48.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.634+0000 7f9d114ee700 1 --2- 172.21.15.189:0/4264505449 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9cf806ab30 0x7f9cf806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc0011e0 tx=0x7f9cfc009040 comp rx=0 tx=0).stop 2024-04-06T13:47:48.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.634+0000 7f9d114ee700 1 -- 172.21.15.189:0/4264505449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d0c15d590 msgr2=0x7f9d0c15a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:48.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.634+0000 7f9d114ee700 1 --2- 172.21.15.189:0/4264505449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d0c15d590 0x7f9d0c15a440 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400b370 tx=0x7f9d04004ad0 comp rx=0 tx=0).stop 2024-04-06T13:47:48.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.634+0000 7f9d114ee700 1 -- 172.21.15.189:0/4264505449 shutdown_connections 2024-04-06T13:47:48.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.634+0000 7f9d114ee700 1 --2- 172.21.15.189:0/4264505449 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9cf806ab30 0x7f9cf806cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:48.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.634+0000 7f9d114ee700 1 --2- 172.21.15.189:0/4264505449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d0c15d590 0x7f9d0c15a440 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:48.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.634+0000 7f9d114ee700 1 --2- 172.21.15.189:0/4264505449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d0c0b4ed0 0x7f9d0c15e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:48.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.634+0000 7f9d114ee700 1 --2- 172.21.15.189:0/4264505449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d0c0b5380 0x7f9d0c1f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:48.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.635+0000 7f9d114ee700 1 -- 172.21.15.189:0/4264505449 >> 172.21.15.189:0/4264505449 conn(0x7f9d0c06bbc0 msgr2=0x7f9d0c0b3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:48.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.635+0000 7f9d114ee700 1 -- 172.21.15.189:0/4264505449 shutdown_connections 2024-04-06T13:47:48.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.635+0000 7f9d114ee700 1 -- 172.21.15.189:0/4264505449 wait complete. 2024-04-06T13:47:48.648 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path mandatory_fs_argument_test_dir 2024-04-06T13:47:48.648 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-04-06T13:47:48.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.778+0000 7f28daf74700 1 Processor -- start 2024-04-06T13:47:48.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.778+0000 7f28daf74700 1 -- start start 2024-04-06T13:47:48.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.779+0000 7f28daf74700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f28d40b3e20 0x7f28d415e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:48.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.779+0000 7f28daf74700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d40b3990 con 0x7f28d406ae40 2024-04-06T13:47:48.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.779+0000 7f28daf74700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d40b3060 con 0x7f28d415d7b0 2024-04-06T13:47:48.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.779+0000 7f28daf74700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28d40b4230 con 0x7f28d40b3e20 2024-04-06T13:47:48.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d3fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f28d406ae40 0x7f28d415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58232/0 (socket says 172.21.15.189:58232) 2024-04-06T13:47:48.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d3fff700 1 -- 172.21.15.189:0/2876549406 learned_addr learned my addr 172.21.15.189:0/2876549406 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:48.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d37fe700 1 -- 172.21.15.189:0/2876549406 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319492048 0 0) 0x7f28d40b3990 con 0x7f28d406ae40 2024-04-06T13:47:48.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d37fe700 1 -- 172.21.15.189:0/2876549406 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28c40035b0 con 0x7f28d406ae40 2024-04-06T13:47:48.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d9511700 1 --2- 172.21.15.189:0/2876549406 >> v2:172.21.15.192:3300/0 conn(0x7f28d40b3e20 0x7f28d415e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:48.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d37fe700 1 -- 172.21.15.189:0/2876549406 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3165852860 0 0) 0x7f28c40035b0 con 0x7f28d406ae40 2024-04-06T13:47:48.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d37fe700 1 -- 172.21.15.189:0/2876549406 >> v2:172.21.15.192:3300/0 conn(0x7f28d40b3e20 msgr2=0x7f28d415e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:48.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d37fe700 1 --2- 172.21.15.189:0/2876549406 >> v2:172.21.15.192:3300/0 conn(0x7f28d40b3e20 0x7f28d415e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:48.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d37fe700 1 -- 172.21.15.189:0/2876549406 >> v1:172.21.15.192:6789/0 conn(0x7f28d415d7b0 legacy=0x7f28d406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:48.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d37fe700 1 -- 172.21.15.189:0/2876549406 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d40b3340 con 0x7f28d406ae40 2024-04-06T13:47:48.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d37fe700 1 -- 172.21.15.189:0/2876549406 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f28c0001670 con 0x7f28d406ae40 2024-04-06T13:47:48.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d37fe700 1 -- 172.21.15.189:0/2876549406 >> v1:172.21.15.189:6789/0 conn(0x7f28d406ae40 legacy=0x7f28d415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:48.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d37fe700 1 --2- 172.21.15.189:0/2876549406 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28c4003a00 0x7f28c4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:48.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d37fe700 1 --2- 172.21.15.189:0/2876549406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c4004850 0x7f28c4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:48.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d37fe700 1 --2- 172.21.15.189:0/2876549406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28c4005210 0x7f28c400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:48.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d37fe700 1 -- 172.21.15.189:0/2876549406 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d40b4230 con 0x7f28c4004850 2024-04-06T13:47:48.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d37fe700 1 -- 172.21.15.189:0/2876549406 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c400c1b0 con 0x7f28c4005210 2024-04-06T13:47:48.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d37fe700 1 -- 172.21.15.189:0/2876549406 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c400c370 con 0x7f28c4003a00 2024-04-06T13:47:48.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d3fff700 1 --2- 172.21.15.189:0/2876549406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c4004850 0x7f28c4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:48.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d9511700 1 --2- 172.21.15.189:0/2876549406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28c4005210 0x7f28c400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:48.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.780+0000 7f28d8d10700 1 --2- 172.21.15.189:0/2876549406 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28c4003a00 0x7f28c4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:48.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.781+0000 7f28d3fff700 1 -- 172.21.15.189:0/2876549406 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28c4003a00 msgr2=0x7f28c4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:48.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.781+0000 7f28d3fff700 1 --2- 172.21.15.189:0/2876549406 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28c4003a00 0x7f28c4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:48.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.781+0000 7f28d3fff700 1 -- 172.21.15.189:0/2876549406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28c4005210 msgr2=0x7f28c400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:48.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.781+0000 7f28d3fff700 1 --2- 172.21.15.189:0/2876549406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28c4005210 0x7f28c400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:48.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.781+0000 7f28d3fff700 1 -- 172.21.15.189:0/2876549406 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c40035b0 con 0x7f28c4004850 2024-04-06T13:47:48.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.781+0000 7f28d9511700 1 -- 172.21.15.189:0/2876549406 reap_dead start 2024-04-06T13:47:48.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.781+0000 7f28d3fff700 1 --2- 172.21.15.189:0/2876549406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c4004850 0x7f28c4004cd0 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f28c000c240 tx=0x7f28c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c84136dd41f46b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:48.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.781+0000 7f28d37fe700 1 -- 172.21.15.189:0/2876549406 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0007c10 con 0x7f28c4004850 2024-04-06T13:47:48.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.781+0000 7f28d37fe700 1 -- 172.21.15.189:0/2876549406 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c000cd60 con 0x7f28c4004850 2024-04-06T13:47:48.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.781+0000 7f28d37fe700 1 -- 172.21.15.189:0/2876549406 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0005ba0 con 0x7f28c4004850 2024-04-06T13:47:48.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.781+0000 7f28daf74700 1 -- 172.21.15.189:0/2876549406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c4004850 msgr2=0x7f28c4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:48.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.781+0000 7f28daf74700 1 --2- 172.21.15.189:0/2876549406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c4004850 0x7f28c4004cd0 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f28c000c240 tx=0x7f28c4005880 comp rx=0 tx=0).stop 2024-04-06T13:47:48.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.781+0000 7f28daf74700 1 -- 172.21.15.189:0/2876549406 shutdown_connections 2024-04-06T13:47:48.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.781+0000 7f28daf74700 1 --2- 172.21.15.189:0/2876549406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c4004850 0x7f28c4004cd0 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:48.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.781+0000 7f28daf74700 1 -- 172.21.15.189:0/2876549406 >> 172.21.15.189:0/2876549406 conn(0x7f28d40d00b0 msgr2=0x7f28d40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:48.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.781+0000 7f28daf74700 1 -- 172.21.15.189:0/2876549406 shutdown_connections 2024-04-06T13:47:48.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.781+0000 7f28daf74700 1 -- 172.21.15.189:0/2876549406 wait complete. 2024-04-06T13:47:48.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.783+0000 7f28daf74700 1 Processor -- start 2024-04-06T13:47:48.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.783+0000 7f28daf74700 1 -- start start 2024-04-06T13:47:48.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.784+0000 7f28daf74700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28c4004850 0x7f28d40ced60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:48.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.784+0000 7f28daf74700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28d40b0790 0x7f28d40aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:48.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.784+0000 7f28daf74700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28d40c50f0 0x7f28d40af500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:48.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.784+0000 7f28daf74700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d40cc630 con 0x7f28d40b0790 2024-04-06T13:47:48.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.784+0000 7f28daf74700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d40cc910 con 0x7f28c4004850 2024-04-06T13:47:48.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.784+0000 7f28daf74700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d40ccbf0 con 0x7f28d40c50f0 2024-04-06T13:47:48.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.784+0000 7f28d3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28d40b0790 0x7f28d40aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:48.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.784+0000 7f28d3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28d40b0790 0x7f28d40aefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44772/0 (socket says 172.21.15.189:44772) 2024-04-06T13:47:48.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.784+0000 7f28d3fff700 1 -- 172.21.15.189:0/1193324067 learned_addr learned my addr 172.21.15.189:0/1193324067 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:48.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.784+0000 7f28d8d10700 1 --2- 172.21.15.189:0/1193324067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28c4004850 0x7f28d40ced60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:48.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.784+0000 7f28d9511700 1 --2- 172.21.15.189:0/1193324067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28d40c50f0 0x7f28d40af500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:48.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.784+0000 7f28d3fff700 1 -- 172.21.15.189:0/1193324067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28d40c50f0 msgr2=0x7f28d40af500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:48.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.784+0000 7f28d3fff700 1 --2- 172.21.15.189:0/1193324067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28d40c50f0 0x7f28d40af500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:48.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.784+0000 7f28d3fff700 1 -- 172.21.15.189:0/1193324067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28c4004850 msgr2=0x7f28d40ced60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:48.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.784+0000 7f28d3fff700 1 --2- 172.21.15.189:0/1193324067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28c4004850 0x7f28d40ced60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:48.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.784+0000 7f28d3fff700 1 -- 172.21.15.189:0/1193324067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d40cced0 con 0x7f28d40b0790 2024-04-06T13:47:48.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.784+0000 7f28d9511700 1 --2- 172.21.15.189:0/1193324067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28d40c50f0 0x7f28d40af500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:48.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.784+0000 7f28d8d10700 1 --2- 172.21.15.189:0/1193324067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28c4004850 0x7f28d40ced60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:48.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.785+0000 7f28d3fff700 1 --2- 172.21.15.189:0/1193324067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28d40b0790 0x7f28d40aefc0 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f28c000c830 tx=0x7f28c0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:48.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.785+0000 7f28d17fa700 1 -- 172.21.15.189:0/1193324067 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0000ed0 con 0x7f28d40b0790 2024-04-06T13:47:48.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.785+0000 7f28d17fa700 1 -- 172.21.15.189:0/1193324067 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c00070d0 con 0x7f28d40b0790 2024-04-06T13:47:48.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.785+0000 7f28d17fa700 1 -- 172.21.15.189:0/1193324067 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0007480 con 0x7f28d40b0790 2024-04-06T13:47:48.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.785+0000 7f28daf74700 1 -- 172.21.15.189:0/1193324067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d40cd1b0 con 0x7f28d40b0790 2024-04-06T13:47:48.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.785+0000 7f28daf74700 1 -- 172.21.15.189:0/1193324067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d40afa40 con 0x7f28d40b0790 2024-04-06T13:47:48.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.786+0000 7f28daf74700 1 -- 172.21.15.189:0/1193324067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d4067bb0 con 0x7f28d40b0790 2024-04-06T13:47:48.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.786+0000 7f28d17fa700 1 -- 172.21.15.189:0/1193324067 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 143) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f28c00075e0 con 0x7f28d40b0790 2024-04-06T13:47:48.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.787+0000 7f28d17fa700 1 --2- 172.21.15.189:0/1193324067 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f28b806ab30 0x7f28b806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:48.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.787+0000 7f28d8d10700 1 --2- 172.21.15.189:0/1193324067 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f28b806ab30 0x7f28b806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:48.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.787+0000 7f28d17fa700 1 -- 172.21.15.189:0/1193324067 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7f28c008a8c0 con 0x7f28d40b0790 2024-04-06T13:47:48.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.787+0000 7f28d8d10700 1 --2- 172.21.15.189:0/1193324067 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f28b806ab30 0x7f28b806cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f28c8007fd0 tx=0x7f28c8009040 comp rx=0 tx=0).ready entity=mgr.9047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:48.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:48.791+0000 7f28d17fa700 1 -- 172.21.15.189:0/1193324067 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f28c00557a0 con 0x7f28d40b0790 2024-04-06T13:47:49.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.015+0000 7f28daf74700 1 -- 172.21.15.189:0/1193324067 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f28d40cca80 con 0x7f28b806ab30 2024-04-06T13:47:49.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.069+0000 7f28d17fa700 1 -- 172.21.15.189:0/1193324067 <== mgr.9047 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f28d40cca80 con 0x7f28b806ab30 2024-04-06T13:47:49.070 INFO:teuthology.orchestra.run.smithi189.stdout:Retention added to path mandatory_fs_argument_test_dir 2024-04-06T13:47:49.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.071+0000 7f28daf74700 1 -- 172.21.15.189:0/1193324067 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f28b806ab30 msgr2=0x7f28b806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:49.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.071+0000 7f28daf74700 1 --2- 172.21.15.189:0/1193324067 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f28b806ab30 0x7f28b806cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f28c8007fd0 tx=0x7f28c8009040 comp rx=0 tx=0).stop 2024-04-06T13:47:49.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.071+0000 7f28daf74700 1 -- 172.21.15.189:0/1193324067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28d40b0790 msgr2=0x7f28d40aefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:49.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.071+0000 7f28daf74700 1 --2- 172.21.15.189:0/1193324067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28d40b0790 0x7f28d40aefc0 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f28c000c830 tx=0x7f28c0001a00 comp rx=0 tx=0).stop 2024-04-06T13:47:49.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.071+0000 7f28daf74700 1 -- 172.21.15.189:0/1193324067 shutdown_connections 2024-04-06T13:47:49.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.071+0000 7f28daf74700 1 --2- 172.21.15.189:0/1193324067 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f28b806ab30 0x7f28b806cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.071+0000 7f28daf74700 1 --2- 172.21.15.189:0/1193324067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28d40c50f0 0x7f28d40af500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.071+0000 7f28daf74700 1 --2- 172.21.15.189:0/1193324067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28c4004850 0x7f28d40ced60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.071+0000 7f28daf74700 1 --2- 172.21.15.189:0/1193324067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28d40b0790 0x7f28d40aefc0 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.071+0000 7f28daf74700 1 -- 172.21.15.189:0/1193324067 >> 172.21.15.189:0/1193324067 conn(0x7f28d40d00b0 msgr2=0x7f28d40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:49.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.071+0000 7f28daf74700 1 -- 172.21.15.189:0/1193324067 shutdown_connections 2024-04-06T13:47:49.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.071+0000 7f28daf74700 1 -- 172.21.15.189:0/1193324067 wait complete. 2024-04-06T13:47:49.084 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path mandatory_fs_argument_test_dir 2024-04-06T13:47:49.084 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list mandatory_fs_argument_test_dir --fs backup_fs --format json 2024-04-06T13:47:49.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.215+0000 7f3aba1cf700 1 Processor -- start 2024-04-06T13:47:49.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.215+0000 7f3aba1cf700 1 -- start start 2024-04-06T13:47:49.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.216+0000 7f3aba1cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ab4158fe0 0x7f3ab406a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.216+0000 7f3aba1cf700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3ab406ac70 0x7f3ab415bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.216+0000 7f3aba1cf700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab415f9e0 con 0x7f3ab415c100 2024-04-06T13:47:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.216+0000 7f3aba1cf700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab406b0f0 con 0x7f3ab406ac70 2024-04-06T13:47:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.216+0000 7f3aba1cf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab415f0d0 con 0x7f3ab4158fe0 2024-04-06T13:47:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.216+0000 7f3ab2ffd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3ab406ac70 0x7f3ab415bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.216+0000 7f3ab3fff700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f3ab415c100 0x7f3ab40717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:50728/0 (socket says 172.21.15.189:50728) 2024-04-06T13:47:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.216+0000 7f3ab3fff700 1 -- 172.21.15.189:0/304457974 learned_addr learned my addr 172.21.15.189:0/304457974 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.216+0000 7f3ab37fe700 1 --2- 172.21.15.189:0/304457974 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ab4158fe0 0x7f3ab406a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:49.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.216+0000 7f3ab2ffd700 1 -- 172.21.15.189:0/304457974 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ab4158fe0 msgr2=0x7f3ab406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.216+0000 7f3ab2ffd700 1 --2- 172.21.15.189:0/304457974 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ab4158fe0 0x7f3ab406a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.216+0000 7f3ab2ffd700 1 -- 172.21.15.189:0/304457974 >> v1:172.21.15.192:6789/0 conn(0x7f3ab415c100 legacy=0x7f3ab40717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.216+0000 7f3ab2ffd700 1 -- 172.21.15.189:0/304457974 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab415c8f0 con 0x7f3ab406ac70 2024-04-06T13:47:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.216+0000 7f3ab2ffd700 1 --2- 172.21.15.189:0/304457974 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ab406ac70 0x7f3ab415bbc0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4009840 tx=0x7f3aa4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8779275e28678b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.217+0000 7f3ab27fc700 1 -- 172.21.15.189:0/304457974 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4011830 con 0x7f3ab406ac70 2024-04-06T13:47:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.217+0000 7f3ab27fc700 1 -- 172.21.15.189:0/304457974 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa4011990 con 0x7f3ab406ac70 2024-04-06T13:47:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.217+0000 7f3ab27fc700 1 -- 172.21.15.189:0/304457974 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4011d20 con 0x7f3ab406ac70 2024-04-06T13:47:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.217+0000 7f3aba1cf700 1 -- 172.21.15.189:0/304457974 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ab406ac70 msgr2=0x7f3ab415bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.217+0000 7f3aba1cf700 1 --2- 172.21.15.189:0/304457974 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ab406ac70 0x7f3ab415bbc0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4009840 tx=0x7f3aa4009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.217+0000 7f3aba1cf700 1 -- 172.21.15.189:0/304457974 shutdown_connections 2024-04-06T13:47:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.217+0000 7f3aba1cf700 1 --2- 172.21.15.189:0/304457974 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ab4158fe0 0x7f3ab406a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.217+0000 7f3aba1cf700 1 --2- 172.21.15.189:0/304457974 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ab406ac70 0x7f3ab415bbc0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.217+0000 7f3aba1cf700 1 -- 172.21.15.189:0/304457974 >> 172.21.15.189:0/304457974 conn(0x7f3ab40d00b0 msgr2=0x7f3ab40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:49.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.217+0000 7f3aba1cf700 1 -- 172.21.15.189:0/304457974 shutdown_connections 2024-04-06T13:47:49.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.217+0000 7f3aba1cf700 1 -- 172.21.15.189:0/304457974 wait complete. 2024-04-06T13:47:49.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.218+0000 7f3aba1cf700 1 Processor -- start 2024-04-06T13:47:49.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.219+0000 7f3aba1cf700 1 -- start start 2024-04-06T13:47:49.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.219+0000 7f3aba1cf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab406ac70 0x7f3ab40a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:49.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.219+0000 7f3aba1cf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ab4158fe0 0x7f3ab415dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:49.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.219+0000 7f3aba1cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ab415c100 0x7f3ab415e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:49.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.219+0000 7f3aba1cf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab41f0ae0 con 0x7f3ab406ac70 2024-04-06T13:47:49.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.219+0000 7f3aba1cf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab40adc60 con 0x7f3ab4158fe0 2024-04-06T13:47:49.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.219+0000 7f3aba1cf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab40a8e20 con 0x7f3ab415c100 2024-04-06T13:47:49.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.219+0000 7f3ab37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab406ac70 0x7f3ab40a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:49.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.220+0000 7f3ab37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab406ac70 0x7f3ab40a84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44804/0 (socket says 172.21.15.189:44804) 2024-04-06T13:47:49.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.220+0000 7f3ab37fe700 1 -- 172.21.15.189:0/3685614217 learned_addr learned my addr 172.21.15.189:0/3685614217 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:49.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.220+0000 7f3ab3fff700 1 --2- 172.21.15.189:0/3685614217 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ab415c100 0x7f3ab415e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:49.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.220+0000 7f3ab2ffd700 1 --2- 172.21.15.189:0/3685614217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ab4158fe0 0x7f3ab415dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:49.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.220+0000 7f3ab37fe700 1 -- 172.21.15.189:0/3685614217 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ab415c100 msgr2=0x7f3ab415e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:49.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.220+0000 7f3ab37fe700 1 --2- 172.21.15.189:0/3685614217 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ab415c100 0x7f3ab415e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.220+0000 7f3ab37fe700 1 -- 172.21.15.189:0/3685614217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ab4158fe0 msgr2=0x7f3ab415dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:49.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.220+0000 7f3ab37fe700 1 --2- 172.21.15.189:0/3685614217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ab4158fe0 0x7f3ab415dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.220+0000 7f3ab37fe700 1 -- 172.21.15.189:0/3685614217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa40094a0 con 0x7f3ab406ac70 2024-04-06T13:47:49.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.220+0000 7f3ab3fff700 1 --2- 172.21.15.189:0/3685614217 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ab415c100 0x7f3ab415e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:49.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.220+0000 7f3ab2ffd700 1 --2- 172.21.15.189:0/3685614217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ab4158fe0 0x7f3ab415dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:49.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.220+0000 7f3ab37fe700 1 --2- 172.21.15.189:0/3685614217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab406ac70 0x7f3ab40a84b0 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c00e9b0 tx=0x7f3a9c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:49.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.220+0000 7f3aabfff700 1 -- 172.21.15.189:0/3685614217 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c009710 con 0x7f3ab406ac70 2024-04-06T13:47:49.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.220+0000 7f3aabfff700 1 -- 172.21.15.189:0/3685614217 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9c00ee50 con 0x7f3ab406ac70 2024-04-06T13:47:49.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.220+0000 7f3aba1cf700 1 -- 172.21.15.189:0/3685614217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab40a9100 con 0x7f3ab406ac70 2024-04-06T13:47:49.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.220+0000 7f3aabfff700 1 -- 172.21.15.189:0/3685614217 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c00ca50 con 0x7f3ab406ac70 2024-04-06T13:47:49.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.220+0000 7f3aba1cf700 1 -- 172.21.15.189:0/3685614217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab40ad8c0 con 0x7f3ab406ac70 2024-04-06T13:47:49.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.221+0000 7f3aba1cf700 1 -- 172.21.15.189:0/3685614217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab40adf40 con 0x7f3ab406ac70 2024-04-06T13:47:49.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.222+0000 7f3aabfff700 1 -- 172.21.15.189:0/3685614217 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 143) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f3a9c00cbb0 con 0x7f3ab406ac70 2024-04-06T13:47:49.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.222+0000 7f3aabfff700 1 --2- 172.21.15.189:0/3685614217 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3a9406ab30 0x7f3a9406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:49.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.222+0000 7f3aabfff700 1 -- 172.21.15.189:0/3685614217 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7f3a9c089590 con 0x7f3ab406ac70 2024-04-06T13:47:49.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.222+0000 7f3ab2ffd700 1 --2- 172.21.15.189:0/3685614217 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3a9406ab30 0x7f3a9406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:49.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.222+0000 7f3ab2ffd700 1 --2- 172.21.15.189:0/3685614217 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3a9406ab30 0x7f3a9406cff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4000c00 tx=0x7f3aa401c040 comp rx=0 tx=0).ready entity=mgr.9047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:49.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.226+0000 7f3aabfff700 1 -- 172.21.15.189:0/3685614217 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3a9c054470 con 0x7f3ab406ac70 2024-04-06T13:47:49.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.447+0000 7f3aba1cf700 1 -- 172.21.15.189:0/3685614217 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3ab415f670 con 0x7f3a9406ab30 2024-04-06T13:47:49.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.453+0000 7f3aabfff700 1 -- 172.21.15.189:0/3685614217 <== mgr.9047 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+88 (secure 0 0 0) 0x7f3aa4011c00 con 0x7f3a9406ab30 2024-04-06T13:47:49.454 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:49.454 INFO:teuthology.orchestra.run.smithi189.stdout:{"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2024-04-06T13:47:49.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.455+0000 7f3aba1cf700 1 -- 172.21.15.189:0/3685614217 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3a9406ab30 msgr2=0x7f3a9406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:49.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.455+0000 7f3aba1cf700 1 --2- 172.21.15.189:0/3685614217 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3a9406ab30 0x7f3a9406cff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4000c00 tx=0x7f3aa401c040 comp rx=0 tx=0).stop 2024-04-06T13:47:49.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.455+0000 7f3aba1cf700 1 -- 172.21.15.189:0/3685614217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab406ac70 msgr2=0x7f3ab40a84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:49.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.455+0000 7f3aba1cf700 1 --2- 172.21.15.189:0/3685614217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab406ac70 0x7f3ab40a84b0 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c00e9b0 tx=0x7f3a9c00ecc0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.455+0000 7f3aba1cf700 1 -- 172.21.15.189:0/3685614217 shutdown_connections 2024-04-06T13:47:49.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.455+0000 7f3aba1cf700 1 --2- 172.21.15.189:0/3685614217 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3a9406ab30 0x7f3a9406cff0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.455+0000 7f3aba1cf700 1 --2- 172.21.15.189:0/3685614217 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ab415c100 0x7f3ab415e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.455+0000 7f3aba1cf700 1 --2- 172.21.15.189:0/3685614217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab406ac70 0x7f3ab40a84b0 unknown :-1 s=CLOSED pgs=2514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.455+0000 7f3aba1cf700 1 --2- 172.21.15.189:0/3685614217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ab4158fe0 0x7f3ab415dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.455+0000 7f3aba1cf700 1 -- 172.21.15.189:0/3685614217 >> 172.21.15.189:0/3685614217 conn(0x7f3ab40d00b0 msgr2=0x7f3ab406fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:49.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.455+0000 7f3aba1cf700 1 -- 172.21.15.189:0/3685614217 shutdown_connections 2024-04-06T13:47:49.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.455+0000 7f3aba1cf700 1 -- 172.21.15.189:0/3685614217 wait complete. 2024-04-06T13:47:49.469 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2024-04-06T13:47:49.469 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status mandatory_fs_argument_test_dir --fs backup_fs --format json 2024-04-06T13:47:49.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.593+0000 7fdf29ef9700 1 Processor -- start 2024-04-06T13:47:49.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.593+0000 7fdf29ef9700 1 -- start start 2024-04-06T13:47:49.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.594+0000 7fdf29ef9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdf24144f40 0x7fdf2406a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:49.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.594+0000 7fdf29ef9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2415c740 con 0x7fdf2415c150 2024-04-06T13:47:49.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.594+0000 7fdf29ef9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2415f1f0 con 0x7fdf2415fcd0 2024-04-06T13:47:49.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.594+0000 7fdf29ef9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf2415ca20 con 0x7fdf24144f40 2024-04-06T13:47:49.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.594+0000 7fdf237fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdf24144f40 0x7fdf2406a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:49.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.594+0000 7fdf23fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdf2415c150 0x7fdf240ceca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58252/0 (socket says 172.21.15.189:58252) 2024-04-06T13:47:49.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.594+0000 7fdf23fff700 1 -- 172.21.15.189:0/3421509868 learned_addr learned my addr 172.21.15.189:0/3421509868 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:49.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.594+0000 7fdf237fe700 1 -- 172.21.15.189:0/3421509868 >> v1:172.21.15.192:6789/0 conn(0x7fdf2415fcd0 legacy=0x7fdf2415ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:49.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.594+0000 7fdf237fe700 1 -- 172.21.15.189:0/3421509868 >> v1:172.21.15.189:6789/0 conn(0x7fdf2415c150 legacy=0x7fdf240ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:49.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.594+0000 7fdf227fc700 1 -- 172.21.15.189:0/3421509868 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603489697 0 0) 0x7fdf2415c740 con 0x7fdf2415c150 2024-04-06T13:47:49.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.594+0000 7fdf237fe700 1 -- 172.21.15.189:0/3421509868 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2415f4d0 con 0x7fdf24144f40 2024-04-06T13:47:49.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.595+0000 7fdf237fe700 1 --2- 172.21.15.189:0/3421509868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf24144f40 0x7fdf2406a730 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7fdf18009840 tx=0x7fdf18009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a813cb73b6d7a37b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:49.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.595+0000 7fdf227fc700 1 -- 172.21.15.189:0/3421509868 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf18011830 con 0x7fdf24144f40 2024-04-06T13:47:49.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.595+0000 7fdf227fc700 1 -- 172.21.15.189:0/3421509868 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf18011990 con 0x7fdf24144f40 2024-04-06T13:47:49.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.595+0000 7fdf227fc700 1 -- 172.21.15.189:0/3421509868 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf18011d20 con 0x7fdf24144f40 2024-04-06T13:47:49.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.595+0000 7fdf29ef9700 1 -- 172.21.15.189:0/3421509868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf24144f40 msgr2=0x7fdf2406a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:49.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.595+0000 7fdf29ef9700 1 --2- 172.21.15.189:0/3421509868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf24144f40 0x7fdf2406a730 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7fdf18009840 tx=0x7fdf18009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:49.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.595+0000 7fdf29ef9700 1 -- 172.21.15.189:0/3421509868 shutdown_connections 2024-04-06T13:47:49.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.595+0000 7fdf29ef9700 1 --2- 172.21.15.189:0/3421509868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf24144f40 0x7fdf2406a730 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.595+0000 7fdf29ef9700 1 -- 172.21.15.189:0/3421509868 >> 172.21.15.189:0/3421509868 conn(0x7fdf240d00b0 msgr2=0x7fdf240d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:49.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.595+0000 7fdf29ef9700 1 -- 172.21.15.189:0/3421509868 shutdown_connections 2024-04-06T13:47:49.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.595+0000 7fdf29ef9700 1 -- 172.21.15.189:0/3421509868 wait complete. 2024-04-06T13:47:49.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.596+0000 7fdf29ef9700 1 Processor -- start 2024-04-06T13:47:49.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.597+0000 7fdf29ef9700 1 -- start start 2024-04-06T13:47:49.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.597+0000 7fdf29ef9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf24144f40 0x7fdf2414ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:49.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.597+0000 7fdf29ef9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf2415c150 0x7fdf241f23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:49.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.597+0000 7fdf29ef9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf2415fcd0 0x7fdf240719c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:49.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.597+0000 7fdf29ef9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf24138fa0 con 0x7fdf2415fcd0 2024-04-06T13:47:49.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.597+0000 7fdf29ef9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf24136880 con 0x7fdf24144f40 2024-04-06T13:47:49.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.597+0000 7fdf29ef9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2414f390 con 0x7fdf2415c150 2024-04-06T13:47:49.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.598+0000 7fdf23fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf2415fcd0 0x7fdf240719c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:49.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.598+0000 7fdf23fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf2415fcd0 0x7fdf240719c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38068/0 (socket says 172.21.15.189:38068) 2024-04-06T13:47:49.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.598+0000 7fdf23fff700 1 -- 172.21.15.189:0/4062968341 learned_addr learned my addr 172.21.15.189:0/4062968341 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:49.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.598+0000 7fdf237fe700 1 --2- 172.21.15.189:0/4062968341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf24144f40 0x7fdf2414ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:49.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.598+0000 7fdf22ffd700 1 --2- 172.21.15.189:0/4062968341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf2415c150 0x7fdf241f23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:49.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.598+0000 7fdf23fff700 1 -- 172.21.15.189:0/4062968341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf2415c150 msgr2=0x7fdf241f23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:49.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.598+0000 7fdf23fff700 1 --2- 172.21.15.189:0/4062968341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf2415c150 0x7fdf241f23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.598+0000 7fdf23fff700 1 -- 172.21.15.189:0/4062968341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf24144f40 msgr2=0x7fdf2414ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:49.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.598+0000 7fdf23fff700 1 --2- 172.21.15.189:0/4062968341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf24144f40 0x7fdf2414ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.598+0000 7fdf23fff700 1 -- 172.21.15.189:0/4062968341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf180094a0 con 0x7fdf2415fcd0 2024-04-06T13:47:49.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.598+0000 7fdf237fe700 1 --2- 172.21.15.189:0/4062968341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf24144f40 0x7fdf2414ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:49.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.598+0000 7fdf22ffd700 1 --2- 172.21.15.189:0/4062968341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf2415c150 0x7fdf241f23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:49.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.598+0000 7fdf23fff700 1 --2- 172.21.15.189:0/4062968341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf2415fcd0 0x7fdf240719c0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c002230 tx=0x7fdf1c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:49.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.598+0000 7fdf0bfff700 1 -- 172.21.15.189:0/4062968341 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c00abc0 con 0x7fdf2415fcd0 2024-04-06T13:47:49.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.598+0000 7fdf0bfff700 1 -- 172.21.15.189:0/4062968341 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1c00ad40 con 0x7fdf2415fcd0 2024-04-06T13:47:49.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.598+0000 7fdf29ef9700 1 -- 172.21.15.189:0/4062968341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf24138320 con 0x7fdf2415fcd0 2024-04-06T13:47:49.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.598+0000 7fdf0bfff700 1 -- 172.21.15.189:0/4062968341 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c007310 con 0x7fdf2415fcd0 2024-04-06T13:47:49.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.598+0000 7fdf29ef9700 1 -- 172.21.15.189:0/4062968341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf241388e0 con 0x7fdf2415fcd0 2024-04-06T13:47:49.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.599+0000 7fdf29ef9700 1 -- 172.21.15.189:0/4062968341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf24067bb0 con 0x7fdf2415fcd0 2024-04-06T13:47:49.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.600+0000 7fdf0bfff700 1 -- 172.21.15.189:0/4062968341 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 143) v1 ==== 89000+0+0 (secure 0 0 0) 0x7fdf1c007470 con 0x7fdf2415fcd0 2024-04-06T13:47:49.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.600+0000 7fdf0bfff700 1 --2- 172.21.15.189:0/4062968341 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdf1006ab30 0x7fdf1006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:49.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.600+0000 7fdf0bfff700 1 -- 172.21.15.189:0/4062968341 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7fdf1c089460 con 0x7fdf2415fcd0 2024-04-06T13:47:49.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.600+0000 7fdf237fe700 1 --2- 172.21.15.189:0/4062968341 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdf1006ab30 0x7fdf1006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:49.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.601+0000 7fdf237fe700 1 --2- 172.21.15.189:0/4062968341 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdf1006ab30 0x7fdf1006cff0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdf240b40f0 tx=0x7fdf1801c040 comp rx=0 tx=0).ready entity=mgr.9047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:49.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.604+0000 7fdf0bfff700 1 -- 172.21.15.189:0/4062968341 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdf1c054340 con 0x7fdf2415fcd0 2024-04-06T13:47:49.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.821+0000 7fdf29ef9700 1 -- 172.21.15.189:0/4062968341 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdf24074990 con 0x7fdf1006ab30 2024-04-06T13:47:49.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.827+0000 7fdf0bfff700 1 -- 172.21.15.189:0/4062968341 <== mgr.9047 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+353 (secure 0 0 0) 0x7fdf24074990 con 0x7fdf1006ab30 2024-04-06T13:47:49.828 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:49.828 INFO:teuthology.orchestra.run.smithi189.stdout:[{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T13:47:48", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-04-06T13:47:49.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.830+0000 7fdf29ef9700 1 -- 172.21.15.189:0/4062968341 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdf1006ab30 msgr2=0x7fdf1006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:49.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.830+0000 7fdf29ef9700 1 --2- 172.21.15.189:0/4062968341 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdf1006ab30 0x7fdf1006cff0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdf240b40f0 tx=0x7fdf1801c040 comp rx=0 tx=0).stop 2024-04-06T13:47:49.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.830+0000 7fdf29ef9700 1 -- 172.21.15.189:0/4062968341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf2415fcd0 msgr2=0x7fdf240719c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:49.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.830+0000 7fdf29ef9700 1 --2- 172.21.15.189:0/4062968341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf2415fcd0 0x7fdf240719c0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c002230 tx=0x7fdf1c00a060 comp rx=0 tx=0).stop 2024-04-06T13:47:49.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.830+0000 7fdf29ef9700 1 -- 172.21.15.189:0/4062968341 shutdown_connections 2024-04-06T13:47:49.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.830+0000 7fdf29ef9700 1 --2- 172.21.15.189:0/4062968341 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdf1006ab30 0x7fdf1006cff0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.830+0000 7fdf29ef9700 1 --2- 172.21.15.189:0/4062968341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf2415fcd0 0x7fdf240719c0 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.830+0000 7fdf29ef9700 1 --2- 172.21.15.189:0/4062968341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf24144f40 0x7fdf2414ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.830+0000 7fdf29ef9700 1 --2- 172.21.15.189:0/4062968341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdf2415c150 0x7fdf241f23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.830+0000 7fdf29ef9700 1 -- 172.21.15.189:0/4062968341 >> 172.21.15.189:0/4062968341 conn(0x7fdf240d00b0 msgr2=0x7fdf2415e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:49.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.830+0000 7fdf29ef9700 1 -- 172.21.15.189:0/4062968341 shutdown_connections 2024-04-06T13:47:49.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.830+0000 7fdf29ef9700 1 -- 172.21.15.189:0/4062968341 wait complete. 2024-04-06T13:47:49.843 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2024-04-06T00:00:00", "created": "2024-04-06T13:47:48", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-04-06T13:47:49.843 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-04-06T13:47:49.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.970+0000 7f9fa5542700 1 Processor -- start 2024-04-06T13:47:49.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.971+0000 7f9fa5542700 1 -- start start 2024-04-06T13:47:49.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.972+0000 7f9fa5542700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9fa00d11b0 0x7f9fa00d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:49.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.972+0000 7f9fa5542700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fa00cfa00 0x7f9fa00cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:49.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.972+0000 7f9fa5542700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa00d1ad0 con 0x7f9fa00cb870 2024-04-06T13:47:49.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.972+0000 7f9fa5542700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fa00ccdc0 con 0x7f9fa00d11b0 2024-04-06T13:47:49.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.972+0000 7f9fa5542700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa00d0b50 con 0x7f9fa00cfa00 2024-04-06T13:47:49.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.972+0000 7f9f9effd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9fa00d11b0 0x7f9fa00d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:49.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.972+0000 7f9f9e7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9fa00cb870 0x7f9fa015c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58258/0 (socket says 172.21.15.189:58258) 2024-04-06T13:47:49.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.972+0000 7f9f9e7fc700 1 -- 172.21.15.189:0/1117539520 learned_addr learned my addr 172.21.15.189:0/1117539520 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:49.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.972+0000 7f9f9f7fe700 1 --2- 172.21.15.189:0/1117539520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fa00cfa00 0x7f9fa00cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:49.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.972+0000 7f9f9effd700 1 -- 172.21.15.189:0/1117539520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fa00cfa00 msgr2=0x7f9fa00cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:49.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.972+0000 7f9f9effd700 1 --2- 172.21.15.189:0/1117539520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fa00cfa00 0x7f9fa00cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.972+0000 7f9f9effd700 1 -- 172.21.15.189:0/1117539520 >> v1:172.21.15.189:6789/0 conn(0x7f9fa00cb870 legacy=0x7f9fa015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:49.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.972+0000 7f9f9effd700 1 -- 172.21.15.189:0/1117539520 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa00cbd80 con 0x7f9fa00d11b0 2024-04-06T13:47:49.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.972+0000 7f9f9dffb700 1 -- 172.21.15.189:0/1117539520 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202186754 0 0) 0x7f9fa00d1ad0 con 0x7f9fa00cb870 2024-04-06T13:47:49.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.972+0000 7f9f9effd700 1 --2- 172.21.15.189:0/1117539520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fa00d11b0 0x7f9fa00d1590 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f9f94009840 tx=0x7f9f94009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d1294304934bb87 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:49.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.972+0000 7f9f9dffb700 1 -- 172.21.15.189:0/1117539520 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f94011830 con 0x7f9fa00d11b0 2024-04-06T13:47:49.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.972+0000 7f9f9dffb700 1 -- 172.21.15.189:0/1117539520 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f94011990 con 0x7f9fa00d11b0 2024-04-06T13:47:49.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.973+0000 7f9f9dffb700 1 -- 172.21.15.189:0/1117539520 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f94011d20 con 0x7f9fa00d11b0 2024-04-06T13:47:49.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.973+0000 7f9fa5542700 1 -- 172.21.15.189:0/1117539520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fa00d11b0 msgr2=0x7f9fa00d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:49.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.973+0000 7f9fa5542700 1 --2- 172.21.15.189:0/1117539520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fa00d11b0 0x7f9fa00d1590 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f9f94009840 tx=0x7f9f94009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:49.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.973+0000 7f9fa5542700 1 -- 172.21.15.189:0/1117539520 shutdown_connections 2024-04-06T13:47:49.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.973+0000 7f9fa5542700 1 --2- 172.21.15.189:0/1117539520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fa00cfa00 0x7f9fa00cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.973+0000 7f9fa5542700 1 --2- 172.21.15.189:0/1117539520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fa00d11b0 0x7f9fa00d1590 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.973+0000 7f9fa5542700 1 -- 172.21.15.189:0/1117539520 >> 172.21.15.189:0/1117539520 conn(0x7f9fa00b2db0 msgr2=0x7f9fa00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:49.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.973+0000 7f9fa5542700 1 -- 172.21.15.189:0/1117539520 shutdown_connections 2024-04-06T13:47:49.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.973+0000 7f9fa5542700 1 -- 172.21.15.189:0/1117539520 wait complete. 2024-04-06T13:47:49.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.975+0000 7f9fa5542700 1 Processor -- start 2024-04-06T13:47:49.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.975+0000 7f9fa5542700 1 -- start start 2024-04-06T13:47:49.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9fa5542700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fa00cb870 0x7f9fa0155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:49.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9fa5542700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fa00cfa00 0x7f9fa00c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:49.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9fa5542700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa00d11b0 0x7f9fa0157e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:49.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9fa5542700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa00cb030 con 0x7f9fa00cfa00 2024-04-06T13:47:49.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9fa5542700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa00ca420 con 0x7f9fa00d11b0 2024-04-06T13:47:49.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9fa5542700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa00ca700 con 0x7f9fa00cb870 2024-04-06T13:47:49.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9f9e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fa00cfa00 0x7f9fa00c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:49.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9f9e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fa00cfa00 0x7f9fa00c8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38082/0 (socket says 172.21.15.189:38082) 2024-04-06T13:47:49.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9f9e7fc700 1 -- 172.21.15.189:0/3310669996 learned_addr learned my addr 172.21.15.189:0/3310669996 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:49.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9f9effd700 1 --2- 172.21.15.189:0/3310669996 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fa00cb870 0x7f9fa0155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:49.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9f9f7fe700 1 --2- 172.21.15.189:0/3310669996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa00d11b0 0x7f9fa0157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:49.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9f9e7fc700 1 -- 172.21.15.189:0/3310669996 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fa00cb870 msgr2=0x7f9fa0155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:49.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9f9e7fc700 1 --2- 172.21.15.189:0/3310669996 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fa00cb870 0x7f9fa0155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9f9e7fc700 1 -- 172.21.15.189:0/3310669996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa00d11b0 msgr2=0x7f9fa0157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:49.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9f9e7fc700 1 --2- 172.21.15.189:0/3310669996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa00d11b0 0x7f9fa0157e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:49.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9f9e7fc700 1 -- 172.21.15.189:0/3310669996 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f940094a0 con 0x7f9fa00cfa00 2024-04-06T13:47:49.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9f9f7fe700 1 --2- 172.21.15.189:0/3310669996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa00d11b0 0x7f9fa0157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:49.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9f9e7fc700 1 --2- 172.21.15.189:0/3310669996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fa00cfa00 0x7f9fa00c8d70 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000c520 tx=0x7f9f9000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:49.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9f877fe700 1 -- 172.21.15.189:0/3310669996 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f90004c80 con 0x7f9fa00cfa00 2024-04-06T13:47:49.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9f877fe700 1 -- 172.21.15.189:0/3310669996 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f90004de0 con 0x7f9fa00cfa00 2024-04-06T13:47:49.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9fa5542700 1 -- 172.21.15.189:0/3310669996 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa00ca9e0 con 0x7f9fa00cfa00 2024-04-06T13:47:49.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9f877fe700 1 -- 172.21.15.189:0/3310669996 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f900072d0 con 0x7f9fa00cfa00 2024-04-06T13:47:49.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.976+0000 7f9fa5542700 1 -- 172.21.15.189:0/3310669996 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa0155bb0 con 0x7f9fa00cfa00 2024-04-06T13:47:49.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.977+0000 7f9fa5542700 1 -- 172.21.15.189:0/3310669996 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa0074990 con 0x7f9fa00cfa00 2024-04-06T13:47:49.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.978+0000 7f9f877fe700 1 -- 172.21.15.189:0/3310669996 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 143) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f9f90007430 con 0x7f9fa00cfa00 2024-04-06T13:47:49.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.978+0000 7f9f877fe700 1 --2- 172.21.15.189:0/3310669996 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9f8c06ab30 0x7f9f8c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:49.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.979+0000 7f9f9effd700 1 --2- 172.21.15.189:0/3310669996 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9f8c06ab30 0x7f9f8c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:49.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.979+0000 7f9f877fe700 1 -- 172.21.15.189:0/3310669996 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7f9f900892d0 con 0x7f9fa00cfa00 2024-04-06T13:47:49.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.979+0000 7f9f9effd700 1 --2- 172.21.15.189:0/3310669996 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9f8c06ab30 0x7f9f8c06cff0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9f940090b0 tx=0x7f9f94008030 comp rx=0 tx=0).ready entity=mgr.9047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:49.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:49.983+0000 7f9f877fe700 1 -- 172.21.15.189:0/3310669996 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9f900540e0 con 0x7f9fa00cfa00 2024-04-06T13:47:50.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.146+0000 7f9f877fe700 1 -- 172.21.15.189:0/3310669996 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 144) v1 ==== 89367+0+0 (secure 0 0 0) 0x7f9f900544b0 con 0x7f9fa00cfa00 2024-04-06T13:47:50.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.210+0000 7f9fa5542700 1 -- 172.21.15.189:0/3310669996 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f9fa00cf4e0 con 0x7f9f8c06ab30 2024-04-06T13:47:50.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.265+0000 7f9f877fe700 1 -- 172.21.15.189:0/3310669996 <== mgr.9047 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f9f94011c00 con 0x7f9f8c06ab30 2024-04-06T13:47:50.267 INFO:teuthology.orchestra.run.smithi189.stdout:Retention removed from path mandatory_fs_argument_test_dir 2024-04-06T13:47:50.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.267+0000 7f9fa5542700 1 -- 172.21.15.189:0/3310669996 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9f8c06ab30 msgr2=0x7f9f8c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:50.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.267+0000 7f9fa5542700 1 --2- 172.21.15.189:0/3310669996 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9f8c06ab30 0x7f9f8c06cff0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9f940090b0 tx=0x7f9f94008030 comp rx=0 tx=0).stop 2024-04-06T13:47:50.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.267+0000 7f9fa5542700 1 -- 172.21.15.189:0/3310669996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fa00cfa00 msgr2=0x7f9fa00c8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:50.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.267+0000 7f9fa5542700 1 --2- 172.21.15.189:0/3310669996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fa00cfa00 0x7f9fa00c8d70 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000c520 tx=0x7f9f9000c830 comp rx=0 tx=0).stop 2024-04-06T13:47:50.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.268+0000 7f9fa5542700 1 -- 172.21.15.189:0/3310669996 shutdown_connections 2024-04-06T13:47:50.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.268+0000 7f9fa5542700 1 --2- 172.21.15.189:0/3310669996 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9f8c06ab30 0x7f9f8c06cff0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.268+0000 7f9fa5542700 1 --2- 172.21.15.189:0/3310669996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa00d11b0 0x7f9fa0157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.268+0000 7f9fa5542700 1 --2- 172.21.15.189:0/3310669996 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9fa00cb870 0x7f9fa0155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.268+0000 7f9fa5542700 1 --2- 172.21.15.189:0/3310669996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fa00cfa00 0x7f9fa00c8d70 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.268+0000 7f9fa5542700 1 -- 172.21.15.189:0/3310669996 >> 172.21.15.189:0/3310669996 conn(0x7f9fa00b2db0 msgr2=0x7f9fa015a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:50.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.268+0000 7f9fa5542700 1 -- 172.21.15.189:0/3310669996 shutdown_connections 2024-04-06T13:47:50.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.268+0000 7f9fa5542700 1 -- 172.21.15.189:0/3310669996 wait complete. 2024-04-06T13:47:50.281 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention removed from path mandatory_fs_argument_test_dir 2024-04-06T13:47:50.281 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-04-06T13:47:50.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.409+0000 7fbe789ca700 1 Processor -- start 2024-04-06T13:47:50.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.409+0000 7fbe789ca700 1 -- start start 2024-04-06T13:47:50.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.410+0000 7fbe789ca700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbe740cd3e0 0x7fbe740cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:50.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.410+0000 7fbe789ca700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe7406b8c0 0x7fbe7406dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:50.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.410+0000 7fbe789ca700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7406ceb0 con 0x7fbe7415d7c0 2024-04-06T13:47:50.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.410+0000 7fbe789ca700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe740cdfa0 con 0x7fbe740cd3e0 2024-04-06T13:47:50.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.410+0000 7fbe789ca700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7406bcf0 con 0x7fbe7406b8c0 2024-04-06T13:47:50.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.410+0000 7fbe72d9d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe7406b8c0 0x7fbe7406dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:50.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.410+0000 7fbe7259c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbe740cd3e0 0x7fbe740cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:50.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.410+0000 7fbe71d9b700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fbe7415d7c0 0x7fbe7406b1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:40456/0 (socket says 172.21.15.189:40456) 2024-04-06T13:47:50.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.410+0000 7fbe71d9b700 1 -- 172.21.15.189:0/1636159140 learned_addr learned my addr 172.21.15.189:0/1636159140 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:50.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.410+0000 7fbe7159a700 1 -- 172.21.15.189:0/1636159140 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995228538 0 0) 0x7fbe7406ceb0 con 0x7fbe7415d7c0 2024-04-06T13:47:50.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.410+0000 7fbe7159a700 1 -- 172.21.15.189:0/1636159140 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe5c0035b0 con 0x7fbe7415d7c0 2024-04-06T13:47:50.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.410+0000 7fbe72d9d700 1 -- 172.21.15.189:0/1636159140 >> v2:172.21.15.192:3300/0 conn(0x7fbe740cd3e0 msgr2=0x7fbe740cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:50.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.410+0000 7fbe72d9d700 1 --2- 172.21.15.189:0/1636159140 >> v2:172.21.15.192:3300/0 conn(0x7fbe740cd3e0 0x7fbe740cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.410+0000 7fbe72d9d700 1 -- 172.21.15.189:0/1636159140 >> v1:172.21.15.192:6789/0 conn(0x7fbe7415d7c0 legacy=0x7fbe7406b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:50.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.410+0000 7fbe72d9d700 1 -- 172.21.15.189:0/1636159140 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7406d750 con 0x7fbe7406b8c0 2024-04-06T13:47:50.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.410+0000 7fbe7259c700 1 --2- 172.21.15.189:0/1636159140 >> v2:172.21.15.192:3300/0 conn(0x7fbe740cd3e0 0x7fbe740cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:50.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.411+0000 7fbe72d9d700 1 --2- 172.21.15.189:0/1636159140 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe7406b8c0 0x7fbe7406dbe0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c009d90 tx=0x7fbe6c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ade5243605c6d92c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:50.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.411+0000 7fbe7159a700 1 -- 172.21.15.189:0/1636159140 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c00e070 con 0x7fbe7406b8c0 2024-04-06T13:47:50.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.411+0000 7fbe7159a700 1 -- 172.21.15.189:0/1636159140 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6c00bb20 con 0x7fbe7406b8c0 2024-04-06T13:47:50.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.411+0000 7fbe7159a700 1 -- 172.21.15.189:0/1636159140 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c00be90 con 0x7fbe7406b8c0 2024-04-06T13:47:50.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.411+0000 7fbe789ca700 1 -- 172.21.15.189:0/1636159140 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe7406b8c0 msgr2=0x7fbe7406dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:50.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.411+0000 7fbe789ca700 1 --2- 172.21.15.189:0/1636159140 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe7406b8c0 0x7fbe7406dbe0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c009d90 tx=0x7fbe6c00b330 comp rx=0 tx=0).stop 2024-04-06T13:47:50.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.411+0000 7fbe789ca700 1 -- 172.21.15.189:0/1636159140 shutdown_connections 2024-04-06T13:47:50.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.411+0000 7fbe789ca700 1 --2- 172.21.15.189:0/1636159140 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe7406b8c0 0x7fbe7406dbe0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.411+0000 7fbe789ca700 1 --2- 172.21.15.189:0/1636159140 >> v2:172.21.15.192:3300/0 conn(0x7fbe740cd3e0 0x7fbe740cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.411+0000 7fbe789ca700 1 -- 172.21.15.189:0/1636159140 >> 172.21.15.189:0/1636159140 conn(0x7fbe740cc290 msgr2=0x7fbe740cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:50.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.411+0000 7fbe789ca700 1 -- 172.21.15.189:0/1636159140 shutdown_connections 2024-04-06T13:47:50.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.411+0000 7fbe789ca700 1 -- 172.21.15.189:0/1636159140 wait complete. 2024-04-06T13:47:50.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.413+0000 7fbe789ca700 1 Processor -- start 2024-04-06T13:47:50.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.413+0000 7fbe789ca700 1 -- start start 2024-04-06T13:47:50.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.413+0000 7fbe789ca700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe7406b8c0 0x7fbe7415dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:50.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.413+0000 7fbe789ca700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe740cd3e0 0x7fbe74146bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:50.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe789ca700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe7415d7c0 0x7fbe74160360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:50.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe789ca700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe741470f0 con 0x7fbe7406b8c0 2024-04-06T13:47:50.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe789ca700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7415e1d0 con 0x7fbe740cd3e0 2024-04-06T13:47:50.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe789ca700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7415e4b0 con 0x7fbe7415d7c0 2024-04-06T13:47:50.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe7259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe7406b8c0 0x7fbe7415dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:50.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe7259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe7406b8c0 0x7fbe7415dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38090/0 (socket says 172.21.15.189:38090) 2024-04-06T13:47:50.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe7259c700 1 -- 172.21.15.189:0/3930732461 learned_addr learned my addr 172.21.15.189:0/3930732461 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:50.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe71d9b700 1 --2- 172.21.15.189:0/3930732461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe740cd3e0 0x7fbe74146bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:50.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe72d9d700 1 --2- 172.21.15.189:0/3930732461 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe7415d7c0 0x7fbe74160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:50.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe7259c700 1 -- 172.21.15.189:0/3930732461 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe7415d7c0 msgr2=0x7fbe74160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:50.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe7259c700 1 --2- 172.21.15.189:0/3930732461 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe7415d7c0 0x7fbe74160360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe7259c700 1 -- 172.21.15.189:0/3930732461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe740cd3e0 msgr2=0x7fbe74146bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:50.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe7259c700 1 --2- 172.21.15.189:0/3930732461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe740cd3e0 0x7fbe74146bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe7259c700 1 -- 172.21.15.189:0/3930732461 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe640094a0 con 0x7fbe7406b8c0 2024-04-06T13:47:50.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe7259c700 1 --2- 172.21.15.189:0/3930732461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe7406b8c0 0x7fbe7415dbf0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7fbe6400ebe0 tx=0x7fbe64009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:50.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe62ffd700 1 -- 172.21.15.189:0/3930732461 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6400c910 con 0x7fbe7406b8c0 2024-04-06T13:47:50.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe62ffd700 1 -- 172.21.15.189:0/3930732461 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6400ca70 con 0x7fbe7406b8c0 2024-04-06T13:47:50.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe789ca700 1 -- 172.21.15.189:0/3930732461 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe6c009a70 con 0x7fbe7406b8c0 2024-04-06T13:47:50.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe62ffd700 1 -- 172.21.15.189:0/3930732461 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6400cde0 con 0x7fbe7406b8c0 2024-04-06T13:47:50.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.414+0000 7fbe789ca700 1 -- 172.21.15.189:0/3930732461 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe7415ea70 con 0x7fbe7406b8c0 2024-04-06T13:47:50.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.415+0000 7fbe789ca700 1 -- 172.21.15.189:0/3930732461 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe74074990 con 0x7fbe7406b8c0 2024-04-06T13:47:50.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.416+0000 7fbe62ffd700 1 -- 172.21.15.189:0/3930732461 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 144) v1 ==== 89367+0+0 (secure 0 0 0) 0x7fbe64013470 con 0x7fbe7406b8c0 2024-04-06T13:47:50.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.416+0000 7fbe62ffd700 1 --2- 172.21.15.189:0/3930732461 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbe5406af20 0x7fbe5406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:50.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.416+0000 7fbe62ffd700 1 -- 172.21.15.189:0/3930732461 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7fbe6408a0c0 con 0x7fbe7406b8c0 2024-04-06T13:47:50.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.416+0000 7fbe71d9b700 1 --2- 172.21.15.189:0/3930732461 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbe5406af20 0x7fbe5406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:50.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.417+0000 7fbe71d9b700 1 --2- 172.21.15.189:0/3930732461 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbe5406af20 0x7fbe5406d3e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbe74156810 tx=0x7fbe68009040 comp rx=0 tx=0).ready entity=mgr.9047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:50.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.420+0000 7fbe62ffd700 1 -- 172.21.15.189:0/3930732461 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbe64054870 con 0x7fbe7406b8c0 2024-04-06T13:47:50.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.646+0000 7fbe789ca700 1 -- 172.21.15.189:0/3930732461 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe740b3490 con 0x7fbe5406af20 2024-04-06T13:47:50.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.703+0000 7fbe62ffd700 1 -- 172.21.15.189:0/3930732461 <== mgr.9047 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+61 (secure 0 0 0) 0x7fbe740b3490 con 0x7fbe5406af20 2024-04-06T13:47:50.705 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule deactivated for path mandatory_fs_argument_test_dir 2024-04-06T13:47:50.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.706+0000 7fbe789ca700 1 -- 172.21.15.189:0/3930732461 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbe5406af20 msgr2=0x7fbe5406d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:50.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.706+0000 7fbe789ca700 1 --2- 172.21.15.189:0/3930732461 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbe5406af20 0x7fbe5406d3e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbe74156810 tx=0x7fbe68009040 comp rx=0 tx=0).stop 2024-04-06T13:47:50.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.706+0000 7fbe789ca700 1 -- 172.21.15.189:0/3930732461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe7406b8c0 msgr2=0x7fbe7415dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:50.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.706+0000 7fbe789ca700 1 --2- 172.21.15.189:0/3930732461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe7406b8c0 0x7fbe7415dbf0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7fbe6400ebe0 tx=0x7fbe64009ec0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.706+0000 7fbe789ca700 1 -- 172.21.15.189:0/3930732461 shutdown_connections 2024-04-06T13:47:50.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.706+0000 7fbe789ca700 1 --2- 172.21.15.189:0/3930732461 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbe5406af20 0x7fbe5406d3e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.706+0000 7fbe789ca700 1 --2- 172.21.15.189:0/3930732461 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbe7415d7c0 0x7fbe74160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.706+0000 7fbe789ca700 1 --2- 172.21.15.189:0/3930732461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe7406b8c0 0x7fbe7415dbf0 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.706+0000 7fbe789ca700 1 --2- 172.21.15.189:0/3930732461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe740cd3e0 0x7fbe74146bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.706+0000 7fbe789ca700 1 -- 172.21.15.189:0/3930732461 >> 172.21.15.189:0/3930732461 conn(0x7fbe740cc290 msgr2=0x7fbe7415c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:50.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.706+0000 7fbe789ca700 1 -- 172.21.15.189:0/3930732461 shutdown_connections 2024-04-06T13:47:50.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.706+0000 7fbe789ca700 1 -- 172.21.15.189:0/3930732461 wait complete. 2024-04-06T13:47:50.720 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path mandatory_fs_argument_test_dir 2024-04-06T13:47:50.720 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-04-06T13:47:50.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.846+0000 7f327f9c6700 1 Processor -- start 2024-04-06T13:47:50.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.846+0000 7f327f9c6700 1 -- start start 2024-04-06T13:47:50.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.847+0000 7f327f9c6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f32780d11b0 0x7f32780d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:50.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.847+0000 7f327f9c6700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f32780cfa00 0x7f32780cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:50.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.847+0000 7f327f9c6700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32780d1ad0 con 0x7f32780cb870 2024-04-06T13:47:50.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.847+0000 7f327f9c6700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32780ccdc0 con 0x7f32780d11b0 2024-04-06T13:47:50.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.847+0000 7f327f9c6700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32780d0b50 con 0x7f32780cfa00 2024-04-06T13:47:50.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.847+0000 7f327d762700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f32780d11b0 0x7f32780d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:50.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.847+0000 7f327d762700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f32780d11b0 0x7f32780d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38106/0 (socket says 172.21.15.189:38106) 2024-04-06T13:47:50.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.847+0000 7f327d762700 1 -- 172.21.15.189:0/430890947 learned_addr learned my addr 172.21.15.189:0/430890947 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:50.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.847+0000 7f327df63700 1 --2- 172.21.15.189:0/430890947 >> v2:172.21.15.192:3300/0 conn(0x7f32780cfa00 0x7f32780cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:50.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.848+0000 7f327d762700 1 -- 172.21.15.189:0/430890947 >> v2:172.21.15.192:3300/0 conn(0x7f32780cfa00 msgr2=0x7f32780cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:50.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.848+0000 7f327d762700 1 --2- 172.21.15.189:0/430890947 >> v2:172.21.15.192:3300/0 conn(0x7f32780cfa00 0x7f32780cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.848+0000 7f327d762700 1 -- 172.21.15.189:0/430890947 >> v1:172.21.15.192:6789/0 conn(0x7f32780cb870 legacy=0x7f327815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:50.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.848+0000 7f327d762700 1 -- 172.21.15.189:0/430890947 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32780cbd80 con 0x7f32780d11b0 2024-04-06T13:47:50.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.848+0000 7f327d762700 1 --2- 172.21.15.189:0/430890947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32780d11b0 0x7f32780d1590 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f32780b0cd0 tx=0x7f3268009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985685344f2ec67b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:50.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.848+0000 7f326ffff700 1 -- 172.21.15.189:0/430890947 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326800b870 con 0x7f32780d11b0 2024-04-06T13:47:50.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.848+0000 7f326ffff700 1 -- 172.21.15.189:0/430890947 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326800b9d0 con 0x7f32780d11b0 2024-04-06T13:47:50.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.848+0000 7f326ffff700 1 -- 172.21.15.189:0/430890947 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326800bd60 con 0x7f32780d11b0 2024-04-06T13:47:50.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.848+0000 7f327f9c6700 1 -- 172.21.15.189:0/430890947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32780d11b0 msgr2=0x7f32780d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:50.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.848+0000 7f327f9c6700 1 --2- 172.21.15.189:0/430890947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32780d11b0 0x7f32780d1590 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f32780b0cd0 tx=0x7f3268009bd0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.849+0000 7f327f9c6700 1 -- 172.21.15.189:0/430890947 shutdown_connections 2024-04-06T13:47:50.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.849+0000 7f327f9c6700 1 --2- 172.21.15.189:0/430890947 >> v2:172.21.15.192:3300/0 conn(0x7f32780cfa00 0x7f32780cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.849+0000 7f327f9c6700 1 --2- 172.21.15.189:0/430890947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32780d11b0 0x7f32780d1590 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.849+0000 7f327f9c6700 1 -- 172.21.15.189:0/430890947 >> 172.21.15.189:0/430890947 conn(0x7f32780b2db0 msgr2=0x7f32780b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:50.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.849+0000 7f327f9c6700 1 -- 172.21.15.189:0/430890947 shutdown_connections 2024-04-06T13:47:50.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.849+0000 7f327f9c6700 1 -- 172.21.15.189:0/430890947 wait complete. 2024-04-06T13:47:50.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.850+0000 7f327f9c6700 1 Processor -- start 2024-04-06T13:47:50.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.850+0000 7f327f9c6700 1 -- start start 2024-04-06T13:47:50.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.851+0000 7f327f9c6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32780cb870 0x7f32781555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:50.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.851+0000 7f327f9c6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32780cfa00 0x7f32780ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:50.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.851+0000 7f327f9c6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32780d11b0 0x7f32780caa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:50.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.851+0000 7f327f9c6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3278157d00 con 0x7f32780cfa00 2024-04-06T13:47:50.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.851+0000 7f327f9c6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3278157f50 con 0x7f32780d11b0 2024-04-06T13:47:50.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.851+0000 7f327f9c6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3278158230 con 0x7f32780cb870 2024-04-06T13:47:50.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.851+0000 7f327cf61700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32780cfa00 0x7f32780ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:50.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.851+0000 7f327cf61700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32780cfa00 0x7f32780ca4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38114/0 (socket says 172.21.15.189:38114) 2024-04-06T13:47:50.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.851+0000 7f327cf61700 1 -- 172.21.15.189:0/559169256 learned_addr learned my addr 172.21.15.189:0/559169256 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:50.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.851+0000 7f327cf61700 1 -- 172.21.15.189:0/559169256 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32780cb870 msgr2=0x7f32781555d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:50.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.851+0000 7f327d762700 1 --2- 172.21.15.189:0/559169256 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32780cb870 0x7f32781555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:50.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.851+0000 7f327df63700 1 --2- 172.21.15.189:0/559169256 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32780d11b0 0x7f32780caa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:50.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.852+0000 7f327cf61700 1 --2- 172.21.15.189:0/559169256 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32780cb870 0x7f32781555d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.852+0000 7f327cf61700 1 -- 172.21.15.189:0/559169256 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32780d11b0 msgr2=0x7f32780caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:50.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.852+0000 7f327cf61700 1 --2- 172.21.15.189:0/559169256 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32780d11b0 0x7f32780caa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:50.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.852+0000 7f327cf61700 1 -- 172.21.15.189:0/559169256 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3268009550 con 0x7f32780cfa00 2024-04-06T13:47:50.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.852+0000 7f327d762700 1 --2- 172.21.15.189:0/559169256 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32780cb870 0x7f32781555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:50.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.852+0000 7f327df63700 1 --2- 172.21.15.189:0/559169256 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32780d11b0 0x7f32780caa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:50.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.852+0000 7f327cf61700 1 --2- 172.21.15.189:0/559169256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32780cfa00 0x7f32780ca4c0 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f327400c3c0 tx=0x7f327400c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:50.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.852+0000 7f326dffb700 1 -- 172.21.15.189:0/559169256 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32740052d0 con 0x7f32780cfa00 2024-04-06T13:47:50.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.852+0000 7f326dffb700 1 -- 172.21.15.189:0/559169256 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3274005430 con 0x7f32780cfa00 2024-04-06T13:47:50.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.852+0000 7f326dffb700 1 -- 172.21.15.189:0/559169256 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3274006aa0 con 0x7f32780cfa00 2024-04-06T13:47:50.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.852+0000 7f327f9c6700 1 -- 172.21.15.189:0/559169256 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3278158510 con 0x7f32780cfa00 2024-04-06T13:47:50.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.852+0000 7f327f9c6700 1 -- 172.21.15.189:0/559169256 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3278158ad0 con 0x7f32780cfa00 2024-04-06T13:47:50.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.853+0000 7f327f9c6700 1 -- 172.21.15.189:0/559169256 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3278158c60 con 0x7f32780cfa00 2024-04-06T13:47:50.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.853+0000 7f326dffb700 1 -- 172.21.15.189:0/559169256 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 144) v1 ==== 89367+0+0 (secure 0 0 0) 0x7f3274003680 con 0x7f32780cfa00 2024-04-06T13:47:50.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.854+0000 7f326dffb700 1 --2- 172.21.15.189:0/559169256 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f325c06af20 0x7f325c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:50.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.854+0000 7f326dffb700 1 -- 172.21.15.189:0/559169256 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7f3274088f50 con 0x7f32780cfa00 2024-04-06T13:47:50.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.854+0000 7f327d762700 1 --2- 172.21.15.189:0/559169256 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f325c06af20 0x7f325c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:50.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.854+0000 7f327d762700 1 --2- 172.21.15.189:0/559169256 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f325c06af20 0x7f325c06d3e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3268009130 tx=0x7f3268008030 comp rx=0 tx=0).ready entity=mgr.9047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:50.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:50.858+0000 7f326dffb700 1 -- 172.21.15.189:0/559169256 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3274053860 con 0x7f32780cfa00 2024-04-06T13:47:51.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.077+0000 7f327f9c6700 1 -- 172.21.15.189:0/559169256 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f32780cf4e0 con 0x7f325c06af20 2024-04-06T13:47:51.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.140+0000 7f326dffb700 1 -- 172.21.15.189:0/559169256 <== mgr.9047 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7f326800bb40 con 0x7f325c06af20 2024-04-06T13:47:51.145 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule removed for path mandatory_fs_argument_test_dir 2024-04-06T13:47:51.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.142+0000 7f327f9c6700 1 -- 172.21.15.189:0/559169256 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f325c06af20 msgr2=0x7f325c06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:51.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.142+0000 7f327f9c6700 1 --2- 172.21.15.189:0/559169256 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f325c06af20 0x7f325c06d3e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3268009130 tx=0x7f3268008030 comp rx=0 tx=0).stop 2024-04-06T13:47:51.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.142+0000 7f327f9c6700 1 -- 172.21.15.189:0/559169256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32780cfa00 msgr2=0x7f32780ca4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:51.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.142+0000 7f327f9c6700 1 --2- 172.21.15.189:0/559169256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32780cfa00 0x7f32780ca4c0 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f327400c3c0 tx=0x7f327400c780 comp rx=0 tx=0).stop 2024-04-06T13:47:51.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.143+0000 7f327f9c6700 1 -- 172.21.15.189:0/559169256 shutdown_connections 2024-04-06T13:47:51.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.143+0000 7f327f9c6700 1 --2- 172.21.15.189:0/559169256 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f325c06af20 0x7f325c06d3e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:51.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.143+0000 7f327f9c6700 1 --2- 172.21.15.189:0/559169256 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32780d11b0 0x7f32780caa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:51.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.143+0000 7f327f9c6700 1 --2- 172.21.15.189:0/559169256 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f32780cb870 0x7f32781555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:51.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.143+0000 7f327f9c6700 1 --2- 172.21.15.189:0/559169256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32780cfa00 0x7f32780ca4c0 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:51.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.143+0000 7f327f9c6700 1 -- 172.21.15.189:0/559169256 >> 172.21.15.189:0/559169256 conn(0x7f32780b2db0 msgr2=0x7f3278159f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:51.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.143+0000 7f327f9c6700 1 -- 172.21.15.189:0/559169256 shutdown_connections 2024-04-06T13:47:51.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.143+0000 7f327f9c6700 1 -- 172.21.15.189:0/559169256 wait complete. 2024-04-06T13:47:51.156 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path mandatory_fs_argument_test_dir 2024-04-06T13:47:51.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:51.157 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2024-04-06T13:47:51.179 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-06T13:47:51.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.347+0000 7fb89ecf8700 1 Processor -- start 2024-04-06T13:47:51.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.347+0000 7fb89ecf8700 1 -- start start 2024-04-06T13:47:51.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.348+0000 7fb89ecf8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb8980d11b0 0x7fb8980d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:51.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.348+0000 7fb89ecf8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8980cfa00 0x7fb8980cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:51.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.348+0000 7fb89ecf8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8980d1ad0 con 0x7fb8980cb870 2024-04-06T13:47:51.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.348+0000 7fb89ecf8700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8980ccdc0 con 0x7fb8980d11b0 2024-04-06T13:47:51.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.348+0000 7fb89ecf8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8980d0b50 con 0x7fb8980cfa00 2024-04-06T13:47:51.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.348+0000 7fb89ca94700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb8980d11b0 0x7fb8980d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:51.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.348+0000 7fb897fff700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fb8980cb870 0x7fb89815c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:40466/0 (socket says 172.21.15.189:40466) 2024-04-06T13:47:51.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.348+0000 7fb89d295700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8980cfa00 0x7fb8980cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:51.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.348+0000 7fb897fff700 1 -- 172.21.15.189:0/2644513724 learned_addr learned my addr 172.21.15.189:0/2644513724 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:51.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.349+0000 7fb89ca94700 1 -- 172.21.15.189:0/2644513724 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8980cfa00 msgr2=0x7fb8980cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:51.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.349+0000 7fb89ca94700 1 --2- 172.21.15.189:0/2644513724 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8980cfa00 0x7fb8980cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:51.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.349+0000 7fb89ca94700 1 -- 172.21.15.189:0/2644513724 >> v1:172.21.15.192:6789/0 conn(0x7fb8980cb870 legacy=0x7fb89815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:51.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.349+0000 7fb89ca94700 1 -- 172.21.15.189:0/2644513724 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8980cbd80 con 0x7fb8980d11b0 2024-04-06T13:47:51.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.349+0000 7fb89d295700 1 --2- 172.21.15.189:0/2644513724 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8980cfa00 0x7fb8980cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:51.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.349+0000 7fb89ca94700 1 --2- 172.21.15.189:0/2644513724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8980d11b0 0x7fb8980d1590 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fb88c009840 tx=0x7fb88c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30eda36d3f52fba4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:51.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.349+0000 7fb8977fe700 1 -- 172.21.15.189:0/2644513724 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c011830 con 0x7fb8980d11b0 2024-04-06T13:47:51.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.349+0000 7fb8977fe700 1 -- 172.21.15.189:0/2644513724 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88c011990 con 0x7fb8980d11b0 2024-04-06T13:47:51.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.349+0000 7fb8977fe700 1 -- 172.21.15.189:0/2644513724 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c011d20 con 0x7fb8980d11b0 2024-04-06T13:47:51.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.350+0000 7fb89ecf8700 1 -- 172.21.15.189:0/2644513724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8980d11b0 msgr2=0x7fb8980d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:51.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.350+0000 7fb89ecf8700 1 --2- 172.21.15.189:0/2644513724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8980d11b0 0x7fb8980d1590 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fb88c009840 tx=0x7fb88c009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:51.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.350+0000 7fb89ecf8700 1 -- 172.21.15.189:0/2644513724 shutdown_connections 2024-04-06T13:47:51.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.350+0000 7fb89ecf8700 1 --2- 172.21.15.189:0/2644513724 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8980cfa00 0x7fb8980cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:51.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.350+0000 7fb89ecf8700 1 --2- 172.21.15.189:0/2644513724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8980d11b0 0x7fb8980d1590 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:51.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.350+0000 7fb89ecf8700 1 -- 172.21.15.189:0/2644513724 >> 172.21.15.189:0/2644513724 conn(0x7fb8980b2db0 msgr2=0x7fb8980b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:51.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.350+0000 7fb89ecf8700 1 -- 172.21.15.189:0/2644513724 shutdown_connections 2024-04-06T13:47:51.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.350+0000 7fb89ecf8700 1 -- 172.21.15.189:0/2644513724 wait complete. 2024-04-06T13:47:51.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.351+0000 7fb89ecf8700 1 Processor -- start 2024-04-06T13:47:51.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.351+0000 7fb89ecf8700 1 -- start start 2024-04-06T13:47:51.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.352+0000 7fb89ecf8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8980cb870 0x7fb89815f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:51.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.352+0000 7fb89ecf8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8980cfa00 0x7fb8980c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:51.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.352+0000 7fb89ecf8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8980d11b0 0x7fb8981554d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:51.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.352+0000 7fb89ecf8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8980c9180 con 0x7fb8980d11b0 2024-04-06T13:47:51.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.352+0000 7fb89ecf8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8981f09e0 con 0x7fb8980cfa00 2024-04-06T13:47:51.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.352+0000 7fb89ecf8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8981f0ca0 con 0x7fb8980cb870 2024-04-06T13:47:51.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.352+0000 7fb89d295700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8980d11b0 0x7fb8981554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:51.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.352+0000 7fb89d295700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8980d11b0 0x7fb8981554d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38118/0 (socket says 172.21.15.189:38118) 2024-04-06T13:47:51.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.352+0000 7fb89d295700 1 -- 172.21.15.189:0/4163058631 learned_addr learned my addr 172.21.15.189:0/4163058631 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:51.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.352+0000 7fb897fff700 1 --2- 172.21.15.189:0/4163058631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8980cfa00 0x7fb8980c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:51.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.352+0000 7fb89ca94700 1 --2- 172.21.15.189:0/4163058631 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8980cb870 0x7fb89815f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:51.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.352+0000 7fb89d295700 1 -- 172.21.15.189:0/4163058631 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8980cb870 msgr2=0x7fb89815f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:51.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.352+0000 7fb89d295700 1 --2- 172.21.15.189:0/4163058631 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8980cb870 0x7fb89815f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:51.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.352+0000 7fb89d295700 1 -- 172.21.15.189:0/4163058631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8980cfa00 msgr2=0x7fb8980c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:51.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.352+0000 7fb89d295700 1 --2- 172.21.15.189:0/4163058631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8980cfa00 0x7fb8980c8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:51.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.352+0000 7fb89d295700 1 -- 172.21.15.189:0/4163058631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb890009a70 con 0x7fb8980d11b0 2024-04-06T13:47:51.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.352+0000 7fb89d295700 1 --2- 172.21.15.189:0/4163058631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8980d11b0 0x7fb8981554d0 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7fb89000efc0 tx=0x7fb89000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:51.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.353+0000 7fb8957fa700 1 -- 172.21.15.189:0/4163058631 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89000ca90 con 0x7fb8980d11b0 2024-04-06T13:47:51.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.353+0000 7fb8957fa700 1 -- 172.21.15.189:0/4163058631 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89000f040 con 0x7fb8980d11b0 2024-04-06T13:47:51.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.353+0000 7fb89ecf8700 1 -- 172.21.15.189:0/4163058631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb88c0094a0 con 0x7fb8980d11b0 2024-04-06T13:47:51.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.353+0000 7fb8957fa700 1 -- 172.21.15.189:0/4163058631 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89000ce10 con 0x7fb8980d11b0 2024-04-06T13:47:51.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.353+0000 7fb89ecf8700 1 -- 172.21.15.189:0/4163058631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8981f1260 con 0x7fb8980d11b0 2024-04-06T13:47:51.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.354+0000 7fb89ecf8700 1 -- 172.21.15.189:0/4163058631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb898074990 con 0x7fb8980d11b0 2024-04-06T13:47:51.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.354+0000 7fb8957fa700 1 -- 172.21.15.189:0/4163058631 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 145) v1 ==== 89423+0+0 (secure 0 0 0) 0x7fb890014480 con 0x7fb8980d11b0 2024-04-06T13:47:51.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.354+0000 7fb8957fa700 1 --2- 172.21.15.189:0/4163058631 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb87c06afb0 0x7fb87c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:51.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.355+0000 7fb8957fa700 1 -- 172.21.15.189:0/4163058631 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7fb89008a4c0 con 0x7fb8980d11b0 2024-04-06T13:47:51.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.355+0000 7fb89ca94700 1 --2- 172.21.15.189:0/4163058631 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb87c06afb0 0x7fb87c06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:51.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.355+0000 7fb89ca94700 1 --2- 172.21.15.189:0/4163058631 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb87c06afb0 0x7fb87c06d470 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb88c000c00 tx=0x7fb88c01c040 comp rx=0 tx=0).ready entity=mgr.9047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:51.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.359+0000 7fb8957fa700 1 -- 172.21.15.189:0/4163058631 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb890054c90 con 0x7fb8980d11b0 2024-04-06T13:47:51.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:51.610+0000 7fb89ecf8700 1 -- 172.21.15.189:0/4163058631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fb8980b0a00 con 0x7fb8980d11b0 2024-04-06T13:47:52.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.172+0000 7fb8957fa700 1 -- 172.21.15.189:0/4163058631 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 146) v1 ==== 89406+0+0 (secure 0 0 0) 0x7fb890055060 con 0x7fb8980d11b0 2024-04-06T13:47:52.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.173+0000 7fb8957fa700 1 -- 172.21.15.189:0/4163058631 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v146) v1 ==== 93+0+0 (secure 0 0 0) 0x7fb8900716b0 con 0x7fb8980d11b0 2024-04-06T13:47:52.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.175+0000 7fb89ecf8700 1 -- 172.21.15.189:0/4163058631 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb87c06afb0 msgr2=0x7fb87c06d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:52.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.175+0000 7fb89ecf8700 1 --2- 172.21.15.189:0/4163058631 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb87c06afb0 0x7fb87c06d470 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb88c000c00 tx=0x7fb88c01c040 comp rx=0 tx=0).stop 2024-04-06T13:47:52.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.175+0000 7fb89ecf8700 1 -- 172.21.15.189:0/4163058631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8980d11b0 msgr2=0x7fb8981554d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:52.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.175+0000 7fb89ecf8700 1 --2- 172.21.15.189:0/4163058631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8980d11b0 0x7fb8981554d0 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7fb89000efc0 tx=0x7fb89000c3a0 comp rx=0 tx=0).stop 2024-04-06T13:47:52.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.175+0000 7fb89ecf8700 1 -- 172.21.15.189:0/4163058631 shutdown_connections 2024-04-06T13:47:52.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.175+0000 7fb89ecf8700 1 --2- 172.21.15.189:0/4163058631 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb87c06afb0 0x7fb87c06d470 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:52.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.175+0000 7fb89ecf8700 1 --2- 172.21.15.189:0/4163058631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8980d11b0 0x7fb8981554d0 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:52.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.175+0000 7fb89ecf8700 1 --2- 172.21.15.189:0/4163058631 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb8980cb870 0x7fb89815f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:52.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.175+0000 7fb89ecf8700 1 --2- 172.21.15.189:0/4163058631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8980cfa00 0x7fb8980c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:52.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.175+0000 7fb89ecf8700 1 -- 172.21.15.189:0/4163058631 >> 172.21.15.189:0/4163058631 conn(0x7fb8980b2db0 msgr2=0x7fb8980b6630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:52.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.175+0000 7fb89ecf8700 1 -- 172.21.15.189:0/4163058631 shutdown_connections 2024-04-06T13:47:52.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.175+0000 7fb89ecf8700 1 -- 172.21.15.189:0/4163058631 wait complete. 2024-04-06T13:47:52.190 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T13:47:52.191 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:47:52.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:47:52.289 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T13:47:52.289 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:47:52.289 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:47:52.308 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:47:52.308 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:47:52.368 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:47:52.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.533+0000 7ff382b5b700 1 Processor -- start 2024-04-06T13:47:52.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.533+0000 7ff382b5b700 1 -- start start 2024-04-06T13:47:52.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.534+0000 7ff382b5b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff37c0b4af0 0x7ff37c0b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:52.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.534+0000 7ff382b5b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff37c0cdbe0 con 0x7ff37c15a550 2024-04-06T13:47:52.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.534+0000 7ff382b5b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff37c15a940 con 0x7ff37c0b4660 2024-04-06T13:47:52.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.534+0000 7ff382b5b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff37c15ac00 con 0x7ff37c0b4af0 2024-04-06T13:47:52.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.534+0000 7ff3810f8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff37c0b4af0 0x7ff37c0b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:52.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.534+0000 7ff3808f7700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff37c15a550 0x7ff37c0cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58274/0 (socket says 172.21.15.189:58274) 2024-04-06T13:47:52.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.534+0000 7ff3808f7700 1 -- 172.21.15.189:0/1776164324 learned_addr learned my addr 172.21.15.189:0/1776164324 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:52.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.535+0000 7ff3810f8700 1 -- 172.21.15.189:0/1776164324 >> v1:172.21.15.192:6789/0 conn(0x7ff37c0b4660 legacy=0x7ff37c0b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:52.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.535+0000 7ff3810f8700 1 -- 172.21.15.189:0/1776164324 >> v1:172.21.15.189:6789/0 conn(0x7ff37c15a550 legacy=0x7ff37c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:52.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.535+0000 7ff3810f8700 1 -- 172.21.15.189:0/1776164324 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff37c15aee0 con 0x7ff37c0b4af0 2024-04-06T13:47:52.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.535+0000 7ff37b7fe700 1 -- 172.21.15.189:0/1776164324 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336917683 0 0) 0x7ff37c0cdbe0 con 0x7ff37c15a550 2024-04-06T13:47:52.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.535+0000 7ff3810f8700 1 --2- 172.21.15.189:0/1776164324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff37c0b4af0 0x7ff37c0b36d0 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7ff374009d90 tx=0x7ff37400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd73857180c8609 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:52.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.535+0000 7ff37b7fe700 1 -- 172.21.15.189:0/1776164324 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37400e070 con 0x7ff37c0b4af0 2024-04-06T13:47:52.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.535+0000 7ff37b7fe700 1 -- 172.21.15.189:0/1776164324 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37400bbd0 con 0x7ff37c0b4af0 2024-04-06T13:47:52.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.535+0000 7ff37b7fe700 1 -- 172.21.15.189:0/1776164324 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374012430 con 0x7ff37c0b4af0 2024-04-06T13:47:52.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.535+0000 7ff382b5b700 1 -- 172.21.15.189:0/1776164324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff37c0b4af0 msgr2=0x7ff37c0b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:52.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.535+0000 7ff382b5b700 1 --2- 172.21.15.189:0/1776164324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff37c0b4af0 0x7ff37c0b36d0 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7ff374009d90 tx=0x7ff37400b330 comp rx=0 tx=0).stop 2024-04-06T13:47:52.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.536+0000 7ff382b5b700 1 -- 172.21.15.189:0/1776164324 shutdown_connections 2024-04-06T13:47:52.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.536+0000 7ff382b5b700 1 --2- 172.21.15.189:0/1776164324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff37c0b4af0 0x7ff37c0b36d0 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:52.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.536+0000 7ff382b5b700 1 -- 172.21.15.189:0/1776164324 >> 172.21.15.189:0/1776164324 conn(0x7ff37c0d00b0 msgr2=0x7ff37c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:52.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.536+0000 7ff382b5b700 1 -- 172.21.15.189:0/1776164324 shutdown_connections 2024-04-06T13:47:52.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.536+0000 7ff382b5b700 1 -- 172.21.15.189:0/1776164324 wait complete. 2024-04-06T13:47:52.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.537+0000 7ff382b5b700 1 Processor -- start 2024-04-06T13:47:52.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.537+0000 7ff382b5b700 1 -- start start 2024-04-06T13:47:52.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.539+0000 7ff382b5b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff37c0b4660 0x7ff37c1494f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:52.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.539+0000 7ff382b5b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff37c0b4af0 0x7ff37c0ac9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:52.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.539+0000 7ff382b5b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff37c1f6ec0 0x7ff37c06a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:52.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.539+0000 7ff382b5b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37c1f6aa0 con 0x7ff37c0b4af0 2024-04-06T13:47:52.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.539+0000 7ff382b5b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37c149a30 con 0x7ff37c1f6ec0 2024-04-06T13:47:52.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.539+0000 7ff382b5b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff37c0ad240 con 0x7ff37c0b4660 2024-04-06T13:47:52.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.539+0000 7ff37bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff37c0b4af0 0x7ff37c0ac9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:52.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.539+0000 7ff37bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff37c0b4af0 0x7ff37c0ac9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38154/0 (socket says 172.21.15.189:38154) 2024-04-06T13:47:52.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.539+0000 7ff37bfff700 1 -- 172.21.15.189:0/769147887 learned_addr learned my addr 172.21.15.189:0/769147887 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:52.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.539+0000 7ff3808f7700 1 --2- 172.21.15.189:0/769147887 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff37c0b4660 0x7ff37c1494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:52.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.539+0000 7ff3810f8700 1 --2- 172.21.15.189:0/769147887 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff37c1f6ec0 0x7ff37c06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:52.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.539+0000 7ff37bfff700 1 -- 172.21.15.189:0/769147887 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff37c0b4660 msgr2=0x7ff37c1494f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:52.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.539+0000 7ff37bfff700 1 --2- 172.21.15.189:0/769147887 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff37c0b4660 0x7ff37c1494f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:52.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.539+0000 7ff37bfff700 1 -- 172.21.15.189:0/769147887 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff37c1f6ec0 msgr2=0x7ff37c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:52.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.539+0000 7ff37bfff700 1 --2- 172.21.15.189:0/769147887 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff37c1f6ec0 0x7ff37c06a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:52.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.539+0000 7ff37bfff700 1 -- 172.21.15.189:0/769147887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff374009a70 con 0x7ff37c0b4af0 2024-04-06T13:47:52.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.540+0000 7ff37bfff700 1 --2- 172.21.15.189:0/769147887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff37c0b4af0 0x7ff37c0ac9e0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7ff36800c4e0 tx=0x7ff36800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:52.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.540+0000 7ff3797fa700 1 -- 172.21.15.189:0/769147887 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3680052d0 con 0x7ff37c0b4af0 2024-04-06T13:47:52.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.540+0000 7ff3797fa700 1 -- 172.21.15.189:0/769147887 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff368005430 con 0x7ff37c0b4af0 2024-04-06T13:47:52.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.540+0000 7ff3797fa700 1 -- 172.21.15.189:0/769147887 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3680038c0 con 0x7ff37c0b4af0 2024-04-06T13:47:52.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.540+0000 7ff382b5b700 1 -- 172.21.15.189:0/769147887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37c1f63e0 con 0x7ff37c0b4af0 2024-04-06T13:47:52.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.540+0000 7ff382b5b700 1 -- 172.21.15.189:0/769147887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37c0ad520 con 0x7ff37c0b4af0 2024-04-06T13:47:52.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.541+0000 7ff382b5b700 1 -- 172.21.15.189:0/769147887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff37c1f6be0 con 0x7ff37c0b4af0 2024-04-06T13:47:52.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.542+0000 7ff3797fa700 1 -- 172.21.15.189:0/769147887 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 146) v1 ==== 89406+0+0 (secure 0 0 0) 0x7ff368003a20 con 0x7ff37c0b4af0 2024-04-06T13:47:52.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.542+0000 7ff3797fa700 1 --2- 172.21.15.189:0/769147887 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff36006af60 0x7ff36006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:52.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.542+0000 7ff3808f7700 1 -- 172.21.15.189:0/769147887 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff36006af60 msgr2=0x7ff36006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:52.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.542+0000 7ff3797fa700 1 -- 172.21.15.189:0/769147887 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7ff3680890c0 con 0x7ff37c0b4af0 2024-04-06T13:47:52.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.542+0000 7ff3808f7700 1 --2- 172.21.15.189:0/769147887 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff36006af60 0x7ff36006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:52.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.547+0000 7ff3797fa700 1 -- 172.21.15.189:0/769147887 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff37c1f6be0 con 0x7ff37c0b4af0 2024-04-06T13:47:52.557 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:52.557+0000 7f1c6056b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:47:52.574 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:52.573+0000 7f7e16fe6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:47:52.637 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:52.637+0000 7f1c6056b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:47:52.655 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:52.654+0000 7f7e16fe6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:47:52.710 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:52.710+0000 7f1c6056b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:47:52.728 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:52.727+0000 7f7e16fe6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:47:52.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.743+0000 7ff3808f7700 1 -- 172.21.15.189:0/769147887 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff36006af60 msgr2=0x7ff36006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:52.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.743+0000 7ff3808f7700 1 --2- 172.21.15.189:0/769147887 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff36006af60 0x7ff36006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:52.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.750+0000 7ff382b5b700 1 -- 172.21.15.189:0/769147887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff37c1f6be0 con 0x7ff37c0b4af0 2024-04-06T13:47:52.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.752+0000 7ff3797fa700 1 -- 172.21.15.189:0/769147887 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) v1 ==== 74+0+20637 (secure 0 0 0) 0x7ff37c1f6be0 con 0x7ff37c0b4af0 2024-04-06T13:47:52.753 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:52.753 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":232,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:47:47.129326+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:47:20.644275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-04-06T13:47:21.075143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:47:32.292623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:47:33.567052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/3920072072":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/1878657401":"2024-04-07T13:47:47.129296+0000","172.21.15.189:6833/111704":"2024-04-07T13:47:47.129296+0000","172.21.15.189:6832/111704":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/2827130089":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/2716712615":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/1807905335":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/1173010452":"2024-04-07T13:47:47.129296+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:47:52.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.754+0000 7ff382b5b700 1 -- 172.21.15.189:0/769147887 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff36006af60 msgr2=0x7ff36006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:52.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.754+0000 7ff382b5b700 1 --2- 172.21.15.189:0/769147887 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff36006af60 0x7ff36006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:52.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.754+0000 7ff382b5b700 1 -- 172.21.15.189:0/769147887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff37c0b4af0 msgr2=0x7ff37c0ac9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:52.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.754+0000 7ff382b5b700 1 --2- 172.21.15.189:0/769147887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff37c0b4af0 0x7ff37c0ac9e0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7ff36800c4e0 tx=0x7ff36800c7f0 comp rx=0 tx=0).stop 2024-04-06T13:47:52.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.754+0000 7ff382b5b700 1 -- 172.21.15.189:0/769147887 shutdown_connections 2024-04-06T13:47:52.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.754+0000 7ff382b5b700 1 --2- 172.21.15.189:0/769147887 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff36006af60 0x7ff36006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:52.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.754+0000 7ff382b5b700 1 --2- 172.21.15.189:0/769147887 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff37c1f6ec0 0x7ff37c06a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:52.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.754+0000 7ff382b5b700 1 --2- 172.21.15.189:0/769147887 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff37c0b4660 0x7ff37c1494f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:52.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.754+0000 7ff382b5b700 1 --2- 172.21.15.189:0/769147887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff37c0b4af0 0x7ff37c0ac9e0 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:52.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.754+0000 7ff382b5b700 1 -- 172.21.15.189:0/769147887 >> 172.21.15.189:0/769147887 conn(0x7ff37c0d00b0 msgr2=0x7ff37c0ce630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:52.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.754+0000 7ff382b5b700 1 -- 172.21.15.189:0/769147887 shutdown_connections 2024-04-06T13:47:52.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.754+0000 7ff382b5b700 1 -- 172.21.15.189:0/769147887 wait complete. 2024-04-06T13:47:52.769 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T13:47:20.644275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '226', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T13:47:21.075143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '227', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-04-06T13:47:32.292623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '230', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2024-04-06T13:47:33.567052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '231', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:47:52.769 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:47:52.778 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:52.778+0000 7f1c6056b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:47:52.800 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:52.799+0000 7f7e16fe6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:47:52.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.899+0000 7f9de8124700 1 Processor -- start 2024-04-06T13:47:52.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.900+0000 7f9de8124700 1 -- start start 2024-04-06T13:47:52.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.901+0000 7f9de8124700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de00d11b0 0x7f9de00d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:52.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.901+0000 7f9de8124700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9de00cfa00 0x7f9de00cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:52.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.901+0000 7f9de8124700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de00d1ad0 con 0x7f9de00cb870 2024-04-06T13:47:52.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.901+0000 7f9de8124700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de00ccdc0 con 0x7f9de00cfa00 2024-04-06T13:47:52.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.901+0000 7f9de8124700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de00d0b50 con 0x7f9de00d11b0 2024-04-06T13:47:52.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.901+0000 7f9de66c1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9de00cfa00 0x7f9de00cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:52.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.901+0000 7f9de56bf700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9de00cb870 0x7f9de015c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58290/0 (socket says 172.21.15.189:58290) 2024-04-06T13:47:52.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.901+0000 7f9de56bf700 1 -- 172.21.15.189:0/189443113 learned_addr learned my addr 172.21.15.189:0/189443113 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:52.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.901+0000 7f9de5ec0700 1 --2- 172.21.15.189:0/189443113 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de00d11b0 0x7f9de00d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:52.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.901+0000 7f9de66c1700 1 -- 172.21.15.189:0/189443113 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de00d11b0 msgr2=0x7f9de00d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:52.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.901+0000 7f9de66c1700 1 --2- 172.21.15.189:0/189443113 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de00d11b0 0x7f9de00d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:52.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.901+0000 7f9de66c1700 1 -- 172.21.15.189:0/189443113 >> v1:172.21.15.189:6789/0 conn(0x7f9de00cb870 legacy=0x7f9de015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:52.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.901+0000 7f9de4ebe700 1 -- 172.21.15.189:0/189443113 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518421215 0 0) 0x7f9de00d1ad0 con 0x7f9de00cb870 2024-04-06T13:47:52.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.901+0000 7f9de66c1700 1 -- 172.21.15.189:0/189443113 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de00cbd80 con 0x7f9de00cfa00 2024-04-06T13:47:52.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.901+0000 7f9de66c1700 1 --2- 172.21.15.189:0/189443113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9de00cfa00 0x7f9de00cd560 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f9dd8009d90 tx=0x7f9dd800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=910bab23e5f544bb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:52.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.902+0000 7f9de4ebe700 1 -- 172.21.15.189:0/189443113 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd800e070 con 0x7f9de00cfa00 2024-04-06T13:47:52.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.902+0000 7f9de4ebe700 1 -- 172.21.15.189:0/189443113 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd800bb20 con 0x7f9de00cfa00 2024-04-06T13:47:52.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.902+0000 7f9de4ebe700 1 -- 172.21.15.189:0/189443113 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd8012430 con 0x7f9de00cfa00 2024-04-06T13:47:52.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.902+0000 7f9de8124700 1 -- 172.21.15.189:0/189443113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9de00cfa00 msgr2=0x7f9de00cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:52.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.902+0000 7f9de8124700 1 --2- 172.21.15.189:0/189443113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9de00cfa00 0x7f9de00cd560 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f9dd8009d90 tx=0x7f9dd800b330 comp rx=0 tx=0).stop 2024-04-06T13:47:52.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.902+0000 7f9de8124700 1 -- 172.21.15.189:0/189443113 shutdown_connections 2024-04-06T13:47:52.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.902+0000 7f9de8124700 1 --2- 172.21.15.189:0/189443113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9de00cfa00 0x7f9de00cd560 unknown :-1 s=CLOSED pgs=2534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:52.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.902+0000 7f9de8124700 1 --2- 172.21.15.189:0/189443113 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de00d11b0 0x7f9de00d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:52.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.902+0000 7f9de8124700 1 -- 172.21.15.189:0/189443113 >> 172.21.15.189:0/189443113 conn(0x7f9de00b2db0 msgr2=0x7f9de00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:52.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.902+0000 7f9de8124700 1 -- 172.21.15.189:0/189443113 shutdown_connections 2024-04-06T13:47:52.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.902+0000 7f9de8124700 1 -- 172.21.15.189:0/189443113 wait complete. 2024-04-06T13:47:52.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.904+0000 7f9de8124700 1 Processor -- start 2024-04-06T13:47:52.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.904+0000 7f9de8124700 1 -- start start 2024-04-06T13:47:52.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.904+0000 7f9de8124700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9de00cb870 0x7f9de00c5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:52.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.904+0000 7f9de8124700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9de00cfa00 0x7f9de0154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:52.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.904+0000 7f9de8124700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de00d11b0 0x7f9de0158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:52.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.904+0000 7f9de8124700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de0158100 con 0x7f9de00cfa00 2024-04-06T13:47:52.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.904+0000 7f9de8124700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de0157230 con 0x7f9de00cb870 2024-04-06T13:47:52.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.904+0000 7f9de8124700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de0157480 con 0x7f9de00d11b0 2024-04-06T13:47:52.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.904+0000 7f9de56bf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9de00cfa00 0x7f9de0154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:52.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.904+0000 7f9de56bf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9de00cfa00 0x7f9de0154e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38170/0 (socket says 172.21.15.189:38170) 2024-04-06T13:47:52.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.904+0000 7f9de56bf700 1 -- 172.21.15.189:0/1412976554 learned_addr learned my addr 172.21.15.189:0/1412976554 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:52.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.904+0000 7f9de66c1700 1 --2- 172.21.15.189:0/1412976554 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de00d11b0 0x7f9de0158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:52.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.904+0000 7f9de5ec0700 1 --2- 172.21.15.189:0/1412976554 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9de00cb870 0x7f9de00c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:52.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.905+0000 7f9de56bf700 1 -- 172.21.15.189:0/1412976554 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de00d11b0 msgr2=0x7f9de0158520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:52.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.905+0000 7f9de56bf700 1 --2- 172.21.15.189:0/1412976554 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de00d11b0 0x7f9de0158520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:52.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.905+0000 7f9de56bf700 1 -- 172.21.15.189:0/1412976554 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9de00cb870 msgr2=0x7f9de00c5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:52.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.905+0000 7f9de56bf700 1 --2- 172.21.15.189:0/1412976554 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9de00cb870 0x7f9de00c5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:52.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.905+0000 7f9de56bf700 1 -- 172.21.15.189:0/1412976554 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd8009a70 con 0x7f9de00cfa00 2024-04-06T13:47:52.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.905+0000 7f9de66c1700 1 --2- 172.21.15.189:0/1412976554 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de00d11b0 0x7f9de0158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:52.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.905+0000 7f9de56bf700 1 --2- 172.21.15.189:0/1412976554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9de00cfa00 0x7f9de0154e10 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f9dd000c540 tx=0x7f9dd000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:52.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.905+0000 7f9dd67fc700 1 -- 172.21.15.189:0/1412976554 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0004c80 con 0x7f9de00cfa00 2024-04-06T13:47:52.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.905+0000 7f9dd67fc700 1 -- 172.21.15.189:0/1412976554 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd0004de0 con 0x7f9de00cfa00 2024-04-06T13:47:52.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.905+0000 7f9de8124700 1 -- 172.21.15.189:0/1412976554 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de0157760 con 0x7f9de00cfa00 2024-04-06T13:47:52.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.905+0000 7f9dd67fc700 1 -- 172.21.15.189:0/1412976554 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0007310 con 0x7f9de00cfa00 2024-04-06T13:47:52.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.905+0000 7f9de8124700 1 -- 172.21.15.189:0/1412976554 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de0157d20 con 0x7f9de00cfa00 2024-04-06T13:47:52.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.906+0000 7f9de8124700 1 -- 172.21.15.189:0/1412976554 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de0158240 con 0x7f9de00cfa00 2024-04-06T13:47:52.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.907+0000 7f9dd67fc700 1 -- 172.21.15.189:0/1412976554 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 146) v1 ==== 89406+0+0 (secure 0 0 0) 0x7f9dd0007470 con 0x7f9de00cfa00 2024-04-06T13:47:52.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.907+0000 7f9dd67fc700 1 --2- 172.21.15.189:0/1412976554 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9dc406af60 0x7f9dc406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:52.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.908+0000 7f9de5ec0700 1 -- 172.21.15.189:0/1412976554 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9dc406af60 msgr2=0x7f9dc406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:52.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.908+0000 7f9dd67fc700 1 -- 172.21.15.189:0/1412976554 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7f9dd0089a60 con 0x7f9de00cfa00 2024-04-06T13:47:52.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.908+0000 7f9de5ec0700 1 --2- 172.21.15.189:0/1412976554 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9dc406af60 0x7f9dc406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:52.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:52.912+0000 7f9dd67fc700 1 -- 172.21.15.189:0/1412976554 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9de0158240 con 0x7f9de00cfa00 2024-04-06T13:47:52.955 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:52.955+0000 7f1c6056b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:47:52.986 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:52.984+0000 7f7e16fe6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:47:53.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.109+0000 7f9de5ec0700 1 -- 172.21.15.189:0/1412976554 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9dc406af60 msgr2=0x7f9dc406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:53.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.109+0000 7f9de5ec0700 1 --2- 172.21.15.189:0/1412976554 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9dc406af60 0x7f9dc406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:53.111 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:53.111+0000 7f1c6056b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:47:53.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.110+0000 7f9de8124700 1 -- 172.21.15.189:0/1412976554 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9de0158240 con 0x7f9de00cfa00 2024-04-06T13:47:53.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.111+0000 7f9dd67fc700 1 -- 172.21.15.189:0/1412976554 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v232) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9de0158240 con 0x7f9de00cfa00 2024-04-06T13:47:53.112 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:47:53.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.113+0000 7f9de8124700 1 -- 172.21.15.189:0/1412976554 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9dc406af60 msgr2=0x7f9dc406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:53.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.113+0000 7f9de8124700 1 --2- 172.21.15.189:0/1412976554 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9dc406af60 0x7f9dc406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.113+0000 7f9de8124700 1 -- 172.21.15.189:0/1412976554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9de00cfa00 msgr2=0x7f9de0154e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:53.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.113+0000 7f9de8124700 1 --2- 172.21.15.189:0/1412976554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9de00cfa00 0x7f9de0154e10 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f9dd000c540 tx=0x7f9dd000c850 comp rx=0 tx=0).stop 2024-04-06T13:47:53.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.113+0000 7f9de8124700 1 -- 172.21.15.189:0/1412976554 shutdown_connections 2024-04-06T13:47:53.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.113+0000 7f9de8124700 1 --2- 172.21.15.189:0/1412976554 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9dc406af60 0x7f9dc406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.113+0000 7f9de8124700 1 --2- 172.21.15.189:0/1412976554 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de00d11b0 0x7f9de0158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.113+0000 7f9de8124700 1 --2- 172.21.15.189:0/1412976554 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9de00cb870 0x7f9de00c5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.113+0000 7f9de8124700 1 --2- 172.21.15.189:0/1412976554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9de00cfa00 0x7f9de0154e10 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.113+0000 7f9de8124700 1 -- 172.21.15.189:0/1412976554 >> 172.21.15.189:0/1412976554 conn(0x7f9de00b2db0 msgr2=0x7f9de0159e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:53.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.113+0000 7f9de8124700 1 -- 172.21.15.189:0/1412976554 shutdown_connections 2024-04-06T13:47:53.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.113+0000 7f9de8124700 1 -- 172.21.15.189:0/1412976554 wait complete. 2024-04-06T13:47:53.127 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T13:47:53.144 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:53.143+0000 7f7e16fe6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:47:53.221 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:53.221+0000 7f1c6056b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:47:53.261 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:53.260+0000 7f7e16fe6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:47:53.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.275+0000 7f404700f700 1 Processor -- start 2024-04-06T13:47:53.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.275+0000 7f404700f700 1 -- start start 2024-04-06T13:47:53.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.275+0000 7f404700f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f40400b4ed0 0x7f40400b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:53.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.275+0000 7f404700f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40400b5cc0 con 0x7f404015d590 2024-04-06T13:47:53.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.275+0000 7f404700f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40400d1500 con 0x7f40400d2880 2024-04-06T13:47:53.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.275+0000 7f404700f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40400b58c0 con 0x7f40400b4ed0 2024-04-06T13:47:53.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.276+0000 7f4044dab700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f404015d590 0x7f40400b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58294/0 (socket says 172.21.15.189:58294) 2024-04-06T13:47:53.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.276+0000 7f4044dab700 1 -- 172.21.15.189:0/2957954285 learned_addr learned my addr 172.21.15.189:0/2957954285 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:53.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.276+0000 7f403ffff700 1 --2- 172.21.15.189:0/2957954285 >> v2:172.21.15.192:3300/0 conn(0x7f40400b4ed0 0x7f40400b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:53.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.276+0000 7f403f7fe700 1 -- 172.21.15.189:0/2957954285 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527101545 0 0) 0x7f40400b5cc0 con 0x7f404015d590 2024-04-06T13:47:53.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.276+0000 7f403f7fe700 1 -- 172.21.15.189:0/2957954285 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40300035b0 con 0x7f404015d590 2024-04-06T13:47:53.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.276+0000 7f403ffff700 1 -- 172.21.15.189:0/2957954285 >> v1:172.21.15.192:6789/0 conn(0x7f40400d2880 legacy=0x7f40400d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:53.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.276+0000 7f403ffff700 1 -- 172.21.15.189:0/2957954285 >> v1:172.21.15.189:6789/0 conn(0x7f404015d590 legacy=0x7f40400b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:53.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.276+0000 7f403ffff700 1 -- 172.21.15.189:0/2957954285 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40400d17e0 con 0x7f40400b4ed0 2024-04-06T13:47:53.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.276+0000 7f403ffff700 1 --2- 172.21.15.189:0/2957954285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40400b4ed0 0x7f40400b5380 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f402c009840 tx=0x7f402c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d7cab0cb1922d31 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:53.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.276+0000 7f403f7fe700 1 -- 172.21.15.189:0/2957954285 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c011830 con 0x7f40400b4ed0 2024-04-06T13:47:53.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.276+0000 7f403f7fe700 1 -- 172.21.15.189:0/2957954285 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402c011990 con 0x7f40400b4ed0 2024-04-06T13:47:53.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.276+0000 7f403f7fe700 1 -- 172.21.15.189:0/2957954285 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c011d20 con 0x7f40400b4ed0 2024-04-06T13:47:53.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.277+0000 7f404700f700 1 -- 172.21.15.189:0/2957954285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40400b4ed0 msgr2=0x7f40400b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:53.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.277+0000 7f404700f700 1 --2- 172.21.15.189:0/2957954285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40400b4ed0 0x7f40400b5380 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f402c009840 tx=0x7f402c009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:53.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.277+0000 7f404700f700 1 -- 172.21.15.189:0/2957954285 shutdown_connections 2024-04-06T13:47:53.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.277+0000 7f404700f700 1 --2- 172.21.15.189:0/2957954285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40400b4ed0 0x7f40400b5380 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.277+0000 7f404700f700 1 -- 172.21.15.189:0/2957954285 >> 172.21.15.189:0/2957954285 conn(0x7f404006bbc0 msgr2=0x7f404006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:53.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.277+0000 7f404700f700 1 -- 172.21.15.189:0/2957954285 shutdown_connections 2024-04-06T13:47:53.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.277+0000 7f404700f700 1 -- 172.21.15.189:0/2957954285 wait complete. 2024-04-06T13:47:53.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.278+0000 7f404700f700 1 Processor -- start 2024-04-06T13:47:53.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.278+0000 7f404700f700 1 -- start start 2024-04-06T13:47:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.279+0000 7f404700f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40400b4ed0 0x7f404014dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.279+0000 7f404700f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40400d2880 0x7f40400cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.279+0000 7f404700f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f404015d590 0x7f40401f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.279+0000 7f404700f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404014dff0 con 0x7f40400b4ed0 2024-04-06T13:47:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.279+0000 7f404700f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40400ca520 con 0x7f40400d2880 2024-04-06T13:47:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.279+0000 7f404700f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40400ccb20 con 0x7f404015d590 2024-04-06T13:47:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.279+0000 7f4044dab700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40400b4ed0 0x7f404014dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.279+0000 7f4044dab700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40400b4ed0 0x7f404014dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38184/0 (socket says 172.21.15.189:38184) 2024-04-06T13:47:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.279+0000 7f4044dab700 1 -- 172.21.15.189:0/2098369910 learned_addr learned my addr 172.21.15.189:0/2098369910 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.280+0000 7f403ffff700 1 --2- 172.21.15.189:0/2098369910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40400d2880 0x7f40400cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.280+0000 7f40455ac700 1 --2- 172.21.15.189:0/2098369910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f404015d590 0x7f40401f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.280+0000 7f4044dab700 1 -- 172.21.15.189:0/2098369910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f404015d590 msgr2=0x7f40401f42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.280+0000 7f4044dab700 1 --2- 172.21.15.189:0/2098369910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f404015d590 0x7f40401f42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.280+0000 7f4044dab700 1 -- 172.21.15.189:0/2098369910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40400d2880 msgr2=0x7f40400cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.280+0000 7f4044dab700 1 --2- 172.21.15.189:0/2098369910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40400d2880 0x7f40400cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.280+0000 7f4044dab700 1 -- 172.21.15.189:0/2098369910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402c0094a0 con 0x7f40400b4ed0 2024-04-06T13:47:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.280+0000 7f4044dab700 1 --2- 172.21.15.189:0/2098369910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40400b4ed0 0x7f404014dab0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f4034001eb0 tx=0x7f403400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.280+0000 7f403d7fa700 1 -- 172.21.15.189:0/2098369910 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4034005d70 con 0x7f40400b4ed0 2024-04-06T13:47:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.280+0000 7f403d7fa700 1 -- 172.21.15.189:0/2098369910 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40340070d0 con 0x7f40400b4ed0 2024-04-06T13:47:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.280+0000 7f404700f700 1 -- 172.21.15.189:0/2098369910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40400cce00 con 0x7f40400b4ed0 2024-04-06T13:47:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.280+0000 7f403d7fa700 1 -- 172.21.15.189:0/2098369910 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4034007460 con 0x7f40400b4ed0 2024-04-06T13:47:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.280+0000 7f404700f700 1 -- 172.21.15.189:0/2098369910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40400caaa0 con 0x7f40400b4ed0 2024-04-06T13:47:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.281+0000 7f404700f700 1 -- 172.21.15.189:0/2098369910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40400ccff0 con 0x7f40400b4ed0 2024-04-06T13:47:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.282+0000 7f403d7fa700 1 -- 172.21.15.189:0/2098369910 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 146) v1 ==== 89406+0+0 (secure 0 0 0) 0x7f40340075c0 con 0x7f40400b4ed0 2024-04-06T13:47:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.283+0000 7f403d7fa700 1 --2- 172.21.15.189:0/2098369910 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f402406af60 0x7f402406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.283+0000 7f403ffff700 1 -- 172.21.15.189:0/2098369910 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f402406af60 msgr2=0x7f402406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.283+0000 7f403d7fa700 1 -- 172.21.15.189:0/2098369910 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7f4034089110 con 0x7f40400b4ed0 2024-04-06T13:47:53.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.283+0000 7f403ffff700 1 --2- 172.21.15.189:0/2098369910 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f402406af60 0x7f402406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:53.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.287+0000 7f403d7fa700 1 -- 172.21.15.189:0/2098369910 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f40340539f0 con 0x7f40400b4ed0 2024-04-06T13:47:53.303 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:53.303+0000 7f1c6056b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:47:53.347 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:53.347+0000 7f7e16fe6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:47:53.377 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:53.377+0000 7f1c6056b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:47:53.425 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:53.424+0000 7f7e16fe6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:47:53.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.484+0000 7f403ffff700 1 -- 172.21.15.189:0/2098369910 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f402406af60 msgr2=0x7f402406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:53.485 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.484+0000 7f403ffff700 1 --2- 172.21.15.189:0/2098369910 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f402406af60 0x7f402406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:53.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.489+0000 7f404700f700 1 -- 172.21.15.189:0/2098369910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4040074990 con 0x7f40400b4ed0 2024-04-06T13:47:53.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.489+0000 7f403d7fa700 1 -- 172.21.15.189:0/2098369910 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v232) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4034057010 con 0x7f40400b4ed0 2024-04-06T13:47:53.491 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:47:53.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.491+0000 7f404700f700 1 -- 172.21.15.189:0/2098369910 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f402406af60 msgr2=0x7f402406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:53.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.491+0000 7f404700f700 1 --2- 172.21.15.189:0/2098369910 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f402406af60 0x7f402406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.491+0000 7f404700f700 1 -- 172.21.15.189:0/2098369910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40400b4ed0 msgr2=0x7f404014dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:53.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.491+0000 7f404700f700 1 --2- 172.21.15.189:0/2098369910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40400b4ed0 0x7f404014dab0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f4034001eb0 tx=0x7f403400c7b0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.492+0000 7f404700f700 1 -- 172.21.15.189:0/2098369910 shutdown_connections 2024-04-06T13:47:53.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.492+0000 7f404700f700 1 --2- 172.21.15.189:0/2098369910 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f402406af60 0x7f402406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.492+0000 7f404700f700 1 --2- 172.21.15.189:0/2098369910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f404015d590 0x7f40401f42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.492+0000 7f404700f700 1 --2- 172.21.15.189:0/2098369910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40400b4ed0 0x7f404014dab0 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.492+0000 7f404700f700 1 --2- 172.21.15.189:0/2098369910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40400d2880 0x7f40400cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.492+0000 7f404700f700 1 -- 172.21.15.189:0/2098369910 >> 172.21.15.189:0/2098369910 conn(0x7f404006bbc0 msgr2=0x7f40400d2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:53.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.492+0000 7f404700f700 1 -- 172.21.15.189:0/2098369910 shutdown_connections 2024-04-06T13:47:53.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.492+0000 7f404700f700 1 -- 172.21.15.189:0/2098369910 wait complete. 2024-04-06T13:47:53.507 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T13:47:53.580 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:53.580+0000 7f1c6056b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:47:53.642 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:53.637+0000 7f7e16fe6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:47:53.657 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:53.656+0000 7f1c6056b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:47:53.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.658+0000 7f531528c700 1 Processor -- start 2024-04-06T13:47:53.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.658+0000 7f531528c700 1 -- start start 2024-04-06T13:47:53.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.659+0000 7f531528c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f531015d7c0 0x7f531006b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:53.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.659+0000 7f531528c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f531006b6f0 0x7f531006dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:53.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.659+0000 7f531528c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531006ceb0 con 0x7f53100cd3e0 2024-04-06T13:47:53.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.659+0000 7f531528c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53100cdfa0 con 0x7f531006b6f0 2024-04-06T13:47:53.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.659+0000 7f531528c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f531006d750 con 0x7f531015d7c0 2024-04-06T13:47:53.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.659+0000 7f530f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f531006b6f0 0x7f531006dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:53.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.659+0000 7f530f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f531006b6f0 0x7f531006dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38186/0 (socket says 172.21.15.189:38186) 2024-04-06T13:47:53.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.659+0000 7f530f7fe700 1 -- 172.21.15.189:0/3023060020 learned_addr learned my addr 172.21.15.189:0/3023060020 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:53.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.659+0000 7f530e7fc700 1 --2- 172.21.15.189:0/3023060020 >> v2:172.21.15.192:3300/0 conn(0x7f531015d7c0 0x7f531006b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:53.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.660+0000 7f530f7fe700 1 -- 172.21.15.189:0/3023060020 >> v2:172.21.15.192:3300/0 conn(0x7f531015d7c0 msgr2=0x7f531006b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:53.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.660+0000 7f530f7fe700 1 --2- 172.21.15.189:0/3023060020 >> v2:172.21.15.192:3300/0 conn(0x7f531015d7c0 0x7f531006b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.660+0000 7f530f7fe700 1 -- 172.21.15.189:0/3023060020 >> v1:172.21.15.192:6789/0 conn(0x7f53100cd3e0 legacy=0x7f53100cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:53.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.660+0000 7f530dffb700 1 -- 172.21.15.189:0/3023060020 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842513390 0 0) 0x7f531006ceb0 con 0x7f53100cd3e0 2024-04-06T13:47:53.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.660+0000 7f530f7fe700 1 -- 172.21.15.189:0/3023060020 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531006bf00 con 0x7f531006b6f0 2024-04-06T13:47:53.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.660+0000 7f530e7fc700 1 --2- 172.21.15.189:0/3023060020 >> v2:172.21.15.192:3300/0 conn(0x7f531015d7c0 0x7f531006b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:53.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.660+0000 7f530f7fe700 1 --2- 172.21.15.189:0/3023060020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f531006b6f0 0x7f531006dbe0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f5310073390 tx=0x7f530800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32da6d3d790ac3bd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:53.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.660+0000 7f530dffb700 1 -- 172.21.15.189:0/3023060020 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530800e070 con 0x7f531006b6f0 2024-04-06T13:47:53.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.660+0000 7f530dffb700 1 -- 172.21.15.189:0/3023060020 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530800b9c0 con 0x7f531006b6f0 2024-04-06T13:47:53.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.660+0000 7f530dffb700 1 -- 172.21.15.189:0/3023060020 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530800bd70 con 0x7f531006b6f0 2024-04-06T13:47:53.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.660+0000 7f531528c700 1 -- 172.21.15.189:0/3023060020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f531006b6f0 msgr2=0x7f531006dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:53.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.660+0000 7f531528c700 1 --2- 172.21.15.189:0/3023060020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f531006b6f0 0x7f531006dbe0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f5310073390 tx=0x7f530800b330 comp rx=0 tx=0).stop 2024-04-06T13:47:53.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.661+0000 7f531528c700 1 -- 172.21.15.189:0/3023060020 shutdown_connections 2024-04-06T13:47:53.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.661+0000 7f531528c700 1 --2- 172.21.15.189:0/3023060020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f531006b6f0 0x7f531006dbe0 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.661+0000 7f531528c700 1 --2- 172.21.15.189:0/3023060020 >> v2:172.21.15.192:3300/0 conn(0x7f531015d7c0 0x7f531006b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.661+0000 7f531528c700 1 -- 172.21.15.189:0/3023060020 >> 172.21.15.189:0/3023060020 conn(0x7f53100cc290 msgr2=0x7f53100cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:53.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.661+0000 7f531528c700 1 -- 172.21.15.189:0/3023060020 shutdown_connections 2024-04-06T13:47:53.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.661+0000 7f531528c700 1 -- 172.21.15.189:0/3023060020 wait complete. 2024-04-06T13:47:53.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.662+0000 7f531528c700 1 Processor -- start 2024-04-06T13:47:53.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.662+0000 7f531528c700 1 -- start start 2024-04-06T13:47:53.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.663+0000 7f531528c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f531006b6f0 0x7f53100b3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:53.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.663+0000 7f531528c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53100cd3e0 0x7f53100b2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:53.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.663+0000 7f531528c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f531015d7c0 0x7f53100c4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:53.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.663+0000 7f531528c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53100b3420 con 0x7f53100cd3e0 2024-04-06T13:47:53.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.663+0000 7f531528c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53100b31f0 con 0x7f531015d7c0 2024-04-06T13:47:53.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.663+0000 7f531528c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531015bee0 con 0x7f531006b6f0 2024-04-06T13:47:53.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.663+0000 7f530e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53100cd3e0 0x7f53100b2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:53.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.663+0000 7f530e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53100cd3e0 0x7f53100b2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38202/0 (socket says 172.21.15.189:38202) 2024-04-06T13:47:53.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.663+0000 7f530e7fc700 1 -- 172.21.15.189:0/2491526912 learned_addr learned my addr 172.21.15.189:0/2491526912 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:53.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.663+0000 7f530effd700 1 --2- 172.21.15.189:0/2491526912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f531006b6f0 0x7f53100b3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:53.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.663+0000 7f530f7fe700 1 --2- 172.21.15.189:0/2491526912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f531015d7c0 0x7f53100c4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:53.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.663+0000 7f530effd700 1 -- 172.21.15.189:0/2491526912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f531015d7c0 msgr2=0x7f53100c4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:53.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.663+0000 7f530effd700 1 --2- 172.21.15.189:0/2491526912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f531015d7c0 0x7f53100c4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.663+0000 7f530effd700 1 -- 172.21.15.189:0/2491526912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53100cd3e0 msgr2=0x7f53100b2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:53.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.663+0000 7f530effd700 1 --2- 172.21.15.189:0/2491526912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53100cd3e0 0x7f53100b2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.663+0000 7f530effd700 1 -- 172.21.15.189:0/2491526912 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5308009b30 con 0x7f531006b6f0 2024-04-06T13:47:53.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.663+0000 7f530f7fe700 1 --2- 172.21.15.189:0/2491526912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f531015d7c0 0x7f53100c4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:53.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.663+0000 7f530e7fc700 1 --2- 172.21.15.189:0/2491526912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53100cd3e0 0x7f53100b2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:53.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.663+0000 7f530effd700 1 --2- 172.21.15.189:0/2491526912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f531006b6f0 0x7f53100b3780 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f530000c410 tx=0x7f530000c720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:53.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.664+0000 7f52f77fe700 1 -- 172.21.15.189:0/2491526912 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53000052d0 con 0x7f531006b6f0 2024-04-06T13:47:53.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.664+0000 7f52f77fe700 1 -- 172.21.15.189:0/2491526912 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5300005430 con 0x7f531006b6f0 2024-04-06T13:47:53.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.664+0000 7f531528c700 1 -- 172.21.15.189:0/2491526912 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53100c6190 con 0x7f531006b6f0 2024-04-06T13:47:53.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.664+0000 7f52f77fe700 1 -- 172.21.15.189:0/2491526912 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53000038a0 con 0x7f531006b6f0 2024-04-06T13:47:53.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.664+0000 7f531528c700 1 -- 172.21.15.189:0/2491526912 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f531015a490 con 0x7f531006b6f0 2024-04-06T13:47:53.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.665+0000 7f531528c700 1 -- 172.21.15.189:0/2491526912 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5310074990 con 0x7f531006b6f0 2024-04-06T13:47:53.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.665+0000 7f52f77fe700 1 -- 172.21.15.189:0/2491526912 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 146) v1 ==== 89406+0+0 (secure 0 0 0) 0x7f5300003c10 con 0x7f531006b6f0 2024-04-06T13:47:53.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.665+0000 7f52f77fe700 1 --2- 172.21.15.189:0/2491526912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f52fc06af60 0x7f52fc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:53.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.665+0000 7f530e7fc700 1 -- 172.21.15.189:0/2491526912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f52fc06af60 msgr2=0x7f52fc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:53.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.665+0000 7f52f77fe700 1 -- 172.21.15.189:0/2491526912 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7f53000891f0 con 0x7f531006b6f0 2024-04-06T13:47:53.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.665+0000 7f530e7fc700 1 --2- 172.21.15.189:0/2491526912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f52fc06af60 0x7f52fc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:53.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.670+0000 7f52f77fe700 1 -- 172.21.15.189:0/2491526912 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5300053ad0 con 0x7f531006b6f0 2024-04-06T13:47:53.718 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:53.717+0000 7f7e16fe6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:47:53.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.867+0000 7f530e7fc700 1 -- 172.21.15.189:0/2491526912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f52fc06af60 msgr2=0x7f52fc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:53.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.867+0000 7f530e7fc700 1 --2- 172.21.15.189:0/2491526912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f52fc06af60 0x7f52fc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:53.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.873+0000 7f531528c700 1 -- 172.21.15.189:0/2491526912 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f53100b3560 con 0x7f531006b6f0 2024-04-06T13:47:53.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.873+0000 7f52f77fe700 1 -- 172.21.15.189:0/2491526912 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v232) v1 ==== 100+0+11 (secure 0 0 0) 0x7f53000570f0 con 0x7f531006b6f0 2024-04-06T13:47:53.875 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:47:53.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.876+0000 7f531528c700 1 -- 172.21.15.189:0/2491526912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f52fc06af60 msgr2=0x7f52fc06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:53.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.876+0000 7f531528c700 1 --2- 172.21.15.189:0/2491526912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f52fc06af60 0x7f52fc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.876+0000 7f531528c700 1 -- 172.21.15.189:0/2491526912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f531006b6f0 msgr2=0x7f53100b3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:53.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.876+0000 7f531528c700 1 --2- 172.21.15.189:0/2491526912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f531006b6f0 0x7f53100b3780 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f530000c410 tx=0x7f530000c720 comp rx=0 tx=0).stop 2024-04-06T13:47:53.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.876+0000 7f531528c700 1 -- 172.21.15.189:0/2491526912 shutdown_connections 2024-04-06T13:47:53.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.876+0000 7f531528c700 1 --2- 172.21.15.189:0/2491526912 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f52fc06af60 0x7f52fc06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.876+0000 7f531528c700 1 --2- 172.21.15.189:0/2491526912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f531015d7c0 0x7f53100c4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.876+0000 7f531528c700 1 --2- 172.21.15.189:0/2491526912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f531006b6f0 0x7f53100b3780 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.876+0000 7f531528c700 1 --2- 172.21.15.189:0/2491526912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53100cd3e0 0x7f53100b2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:53.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.876+0000 7f531528c700 1 -- 172.21.15.189:0/2491526912 >> 172.21.15.189:0/2491526912 conn(0x7f53100cc290 msgr2=0x7f531015fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:53.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.876+0000 7f531528c700 1 -- 172.21.15.189:0/2491526912 shutdown_connections 2024-04-06T13:47:53.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:53.876+0000 7f531528c700 1 -- 172.21.15.189:0/2491526912 wait complete. 2024-04-06T13:47:53.891 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-04-06T13:47:53.978 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:53.978+0000 7f1c6056b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:47:54.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.020+0000 7f31292a4700 1 Processor -- start 2024-04-06T13:47:54.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.020+0000 7f31292a4700 1 -- start start 2024-04-06T13:47:54.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.020+0000 7f31292a4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f31240b3b80 0x7f3124071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:54.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.020+0000 7f31292a4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31240b1fb0 con 0x7f312415b510 2024-04-06T13:47:54.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.020+0000 7f31292a4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3124071700 con 0x7f312415b130 2024-04-06T13:47:54.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.020+0000 7f31292a4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31240b2200 con 0x7f31240b3b80 2024-04-06T13:47:54.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.021+0000 7f31237fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f312415b510 0x7f3124072320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58310/0 (socket says 172.21.15.189:58310) 2024-04-06T13:47:54.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.021+0000 7f31237fe700 1 -- 172.21.15.189:0/3610120161 learned_addr learned my addr 172.21.15.189:0/3610120161 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:54.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.021+0000 7f31227fc700 1 --2- 172.21.15.189:0/3610120161 >> v2:172.21.15.192:3300/0 conn(0x7f31240b3b80 0x7f3124071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:54.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.021+0000 7f3121ffb700 1 -- 172.21.15.189:0/3610120161 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249942722 0 0) 0x7f31240b1fb0 con 0x7f312415b510 2024-04-06T13:47:54.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.021+0000 7f3121ffb700 1 -- 172.21.15.189:0/3610120161 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f310c0035b0 con 0x7f312415b510 2024-04-06T13:47:54.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.021+0000 7f3121ffb700 1 -- 172.21.15.189:0/3610120161 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2795079428 0 0) 0x7f3124071700 con 0x7f312415b130 2024-04-06T13:47:54.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.021+0000 7f3121ffb700 1 -- 172.21.15.189:0/3610120161 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31240b1fb0 con 0x7f312415b130 2024-04-06T13:47:54.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.021+0000 7f31227fc700 1 -- 172.21.15.189:0/3610120161 >> v1:172.21.15.192:6789/0 conn(0x7f312415b130 legacy=0x7f31240b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:54.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.021+0000 7f31227fc700 1 -- 172.21.15.189:0/3610120161 >> v1:172.21.15.189:6789/0 conn(0x7f312415b510 legacy=0x7f3124072320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:54.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.021+0000 7f3121ffb700 1 -- 172.21.15.189:0/3610120161 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3544194839 0 0) 0x7f310c0035b0 con 0x7f312415b510 2024-04-06T13:47:54.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.021+0000 7f31227fc700 1 -- 172.21.15.189:0/3610120161 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31240b24e0 con 0x7f31240b3b80 2024-04-06T13:47:54.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.021+0000 7f31227fc700 1 --2- 172.21.15.189:0/3610120161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31240b3b80 0x7f3124071de0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f3118009870 tx=0x7f3118009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d214668d2a66479 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:54.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.021+0000 7f3121ffb700 1 -- 172.21.15.189:0/3610120161 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c0019f0 con 0x7f31240b3b80 2024-04-06T13:47:54.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.022+0000 7f3121ffb700 1 -- 172.21.15.189:0/3610120161 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3118004030 con 0x7f31240b3b80 2024-04-06T13:47:54.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.022+0000 7f3121ffb700 1 -- 172.21.15.189:0/3610120161 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31180043c0 con 0x7f31240b3b80 2024-04-06T13:47:54.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.022+0000 7f31292a4700 1 -- 172.21.15.189:0/3610120161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31240b3b80 msgr2=0x7f3124071de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:54.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.022+0000 7f31292a4700 1 --2- 172.21.15.189:0/3610120161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31240b3b80 0x7f3124071de0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f3118009870 tx=0x7f3118009b80 comp rx=0 tx=0).stop 2024-04-06T13:47:54.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.022+0000 7f31292a4700 1 -- 172.21.15.189:0/3610120161 shutdown_connections 2024-04-06T13:47:54.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.022+0000 7f31292a4700 1 --2- 172.21.15.189:0/3610120161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31240b3b80 0x7f3124071de0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.022+0000 7f31292a4700 1 -- 172.21.15.189:0/3610120161 >> 172.21.15.189:0/3610120161 conn(0x7f312406a730 msgr2=0x7f312406ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:54.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.022+0000 7f31292a4700 1 -- 172.21.15.189:0/3610120161 shutdown_connections 2024-04-06T13:47:54.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.022+0000 7f31292a4700 1 -- 172.21.15.189:0/3610120161 wait complete. 2024-04-06T13:47:54.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.024+0000 7f31292a4700 1 Processor -- start 2024-04-06T13:47:54.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.024+0000 7f31292a4700 1 -- start start 2024-04-06T13:47:54.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.024+0000 7f31292a4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31240b3b80 0x7f31241f0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:54.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.024+0000 7f31292a4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f312415b130 0x7f312413c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:54.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.024+0000 7f31292a4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f312415b510 0x7f3124157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:54.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.024+0000 7f31292a4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31241f5450 con 0x7f312415b130 2024-04-06T13:47:54.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.024+0000 7f31292a4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31241f1410 con 0x7f31240b3b80 2024-04-06T13:47:54.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.024+0000 7f31292a4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31241f48d0 con 0x7f312415b510 2024-04-06T13:47:54.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.024+0000 7f31227fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f312415b130 0x7f312413c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:54.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.024+0000 7f31227fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f312415b130 0x7f312413c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38214/0 (socket says 172.21.15.189:38214) 2024-04-06T13:47:54.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.024+0000 7f31227fc700 1 -- 172.21.15.189:0/222332611 learned_addr learned my addr 172.21.15.189:0/222332611 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:54.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.024+0000 7f31237fe700 1 --2- 172.21.15.189:0/222332611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f312415b510 0x7f3124157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:54.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.024+0000 7f3122ffd700 1 --2- 172.21.15.189:0/222332611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31240b3b80 0x7f31241f0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:54.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.025+0000 7f31227fc700 1 -- 172.21.15.189:0/222332611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f312415b510 msgr2=0x7f3124157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:54.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.025+0000 7f31227fc700 1 --2- 172.21.15.189:0/222332611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f312415b510 0x7f3124157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.025+0000 7f31227fc700 1 -- 172.21.15.189:0/222332611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31240b3b80 msgr2=0x7f31241f0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:54.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.025+0000 7f31227fc700 1 --2- 172.21.15.189:0/222332611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31240b3b80 0x7f31241f0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.025+0000 7f31227fc700 1 -- 172.21.15.189:0/222332611 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31180094a0 con 0x7f312415b130 2024-04-06T13:47:54.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.025+0000 7f31227fc700 1 --2- 172.21.15.189:0/222332611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f312415b130 0x7f312413c800 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f31180090b0 tx=0x7f3118004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:54.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.025+0000 7f310b7fe700 1 -- 172.21.15.189:0/222332611 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3118004570 con 0x7f312415b130 2024-04-06T13:47:54.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.025+0000 7f310b7fe700 1 -- 172.21.15.189:0/222332611 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3118011630 con 0x7f312415b130 2024-04-06T13:47:54.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.025+0000 7f31292a4700 1 -- 172.21.15.189:0/222332611 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31241f4bb0 con 0x7f312415b130 2024-04-06T13:47:54.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.025+0000 7f310b7fe700 1 -- 172.21.15.189:0/222332611 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31180119c0 con 0x7f312415b130 2024-04-06T13:47:54.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.025+0000 7f31292a4700 1 -- 172.21.15.189:0/222332611 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31241f5170 con 0x7f312415b130 2024-04-06T13:47:54.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.026+0000 7f31292a4700 1 -- 172.21.15.189:0/222332611 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3124073270 con 0x7f312415b130 2024-04-06T13:47:54.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.027+0000 7f310b7fe700 1 -- 172.21.15.189:0/222332611 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 146) v1 ==== 89406+0+0 (secure 0 0 0) 0x7f3118011b20 con 0x7f312415b130 2024-04-06T13:47:54.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.027+0000 7f310b7fe700 1 --2- 172.21.15.189:0/222332611 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f311006af60 0x7f311006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:54.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.027+0000 7f3122ffd700 1 -- 172.21.15.189:0/222332611 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f311006af60 msgr2=0x7f311006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:54.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.027+0000 7f310b7fe700 1 -- 172.21.15.189:0/222332611 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7f3118020030 con 0x7f312415b130 2024-04-06T13:47:54.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.027+0000 7f3122ffd700 1 --2- 172.21.15.189:0/222332611 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f311006af60 0x7f311006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:54.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.031+0000 7f310b7fe700 1 -- 172.21.15.189:0/222332611 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3118055610 con 0x7f312415b130 2024-04-06T13:47:54.057 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:54.057+0000 7f7e16fe6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:47:54.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.228+0000 7f3122ffd700 1 -- 172.21.15.189:0/222332611 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f311006af60 msgr2=0x7f311006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:54.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.228+0000 7f3122ffd700 1 --2- 172.21.15.189:0/222332611 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f311006af60 0x7f311006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:54.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.238+0000 7f31292a4700 1 -- 172.21.15.189:0/222332611 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3124074990 con 0x7f312415b130 2024-04-06T13:47:54.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.239+0000 7f310b7fe700 1 -- 172.21.15.189:0/222332611 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v232) v1 ==== 107+0+11 (secure 0 0 0) 0x7f311802a020 con 0x7f312415b130 2024-04-06T13:47:54.240 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:47:54.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.241+0000 7f31292a4700 1 -- 172.21.15.189:0/222332611 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f311006af60 msgr2=0x7f311006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:54.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.241+0000 7f31292a4700 1 --2- 172.21.15.189:0/222332611 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f311006af60 0x7f311006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.241+0000 7f31292a4700 1 -- 172.21.15.189:0/222332611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f312415b130 msgr2=0x7f312413c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:54.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.241+0000 7f31292a4700 1 --2- 172.21.15.189:0/222332611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f312415b130 0x7f312413c800 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f31180090b0 tx=0x7f3118004750 comp rx=0 tx=0).stop 2024-04-06T13:47:54.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.241+0000 7f31292a4700 1 -- 172.21.15.189:0/222332611 shutdown_connections 2024-04-06T13:47:54.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.241+0000 7f31292a4700 1 --2- 172.21.15.189:0/222332611 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f311006af60 0x7f311006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.241+0000 7f31292a4700 1 --2- 172.21.15.189:0/222332611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f312415b510 0x7f3124157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.241+0000 7f31292a4700 1 --2- 172.21.15.189:0/222332611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31240b3b80 0x7f31241f0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.241+0000 7f31292a4700 1 --2- 172.21.15.189:0/222332611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f312415b130 0x7f312413c800 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.241+0000 7f31292a4700 1 -- 172.21.15.189:0/222332611 >> 172.21.15.189:0/222332611 conn(0x7f312406a730 msgr2=0x7f312415e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:54.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.241+0000 7f31292a4700 1 -- 172.21.15.189:0/222332611 shutdown_connections 2024-04-06T13:47:54.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.241+0000 7f31292a4700 1 -- 172.21.15.189:0/222332611 wait complete. 2024-04-06T13:47:54.255 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-04-06T13:47:54.301 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:54.301+0000 7f1c6056b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:47:54.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.387+0000 7fbbd7b32700 1 Processor -- start 2024-04-06T13:47:54.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.387+0000 7fbbd7b32700 1 -- start start 2024-04-06T13:47:54.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.389+0000 7fbbd7b32700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbbd00d11b0 0x7fbbd00d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:54.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.389+0000 7fbbd7b32700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbbd00cfa00 0x7fbbd00cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:54.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.389+0000 7fbbd7b32700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd00d1ad0 con 0x7fbbd00cb870 2024-04-06T13:47:54.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.389+0000 7fbbd7b32700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbd00ccdc0 con 0x7fbbd00d11b0 2024-04-06T13:47:54.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.389+0000 7fbbd7b32700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbd00d0b50 con 0x7fbbd00cfa00 2024-04-06T13:47:54.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.389+0000 7fbbd58ce700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbbd00d11b0 0x7fbbd00d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:54.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.389+0000 7fbbd50cd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbbd00cb870 0x7fbbd015c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58320/0 (socket says 172.21.15.189:58320) 2024-04-06T13:47:54.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.389+0000 7fbbd58ce700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbbd00d11b0 0x7fbbd00d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38218/0 (socket says 172.21.15.189:38218) 2024-04-06T13:47:54.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.389+0000 7fbbd50cd700 1 -- 172.21.15.189:0/76786529 learned_addr learned my addr 172.21.15.189:0/76786529 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:54.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.389+0000 7fbbd60cf700 1 --2- 172.21.15.189:0/76786529 >> v2:172.21.15.192:3300/0 conn(0x7fbbd00cfa00 0x7fbbd00cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:54.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.389+0000 7fbbd58ce700 1 -- 172.21.15.189:0/76786529 >> v2:172.21.15.192:3300/0 conn(0x7fbbd00cfa00 msgr2=0x7fbbd00cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:54.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.389+0000 7fbbd58ce700 1 --2- 172.21.15.189:0/76786529 >> v2:172.21.15.192:3300/0 conn(0x7fbbd00cfa00 0x7fbbd00cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.389+0000 7fbbd58ce700 1 -- 172.21.15.189:0/76786529 >> v1:172.21.15.189:6789/0 conn(0x7fbbd00cb870 legacy=0x7fbbd015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:54.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.389+0000 7fbbd48cc700 1 -- 172.21.15.189:0/76786529 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070855101 0 0) 0x7fbbd00d1ad0 con 0x7fbbd00cb870 2024-04-06T13:47:54.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.389+0000 7fbbd58ce700 1 -- 172.21.15.189:0/76786529 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd00cbd80 con 0x7fbbd00d11b0 2024-04-06T13:47:54.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.389+0000 7fbbd58ce700 1 --2- 172.21.15.189:0/76786529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbd00d11b0 0x7fbbd00d1590 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7fbbd00b0cd0 tx=0x7fbbcc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eccca72b2cb71876 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:54.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.390+0000 7fbbd48cc700 1 -- 172.21.15.189:0/76786529 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc011830 con 0x7fbbd00d11b0 2024-04-06T13:47:54.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.390+0000 7fbbd48cc700 1 -- 172.21.15.189:0/76786529 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbcc011990 con 0x7fbbd00d11b0 2024-04-06T13:47:54.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.390+0000 7fbbd48cc700 1 -- 172.21.15.189:0/76786529 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc011d20 con 0x7fbbd00d11b0 2024-04-06T13:47:54.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.390+0000 7fbbd7b32700 1 -- 172.21.15.189:0/76786529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbd00d11b0 msgr2=0x7fbbd00d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:54.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.390+0000 7fbbd7b32700 1 --2- 172.21.15.189:0/76786529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbd00d11b0 0x7fbbd00d1590 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7fbbd00b0cd0 tx=0x7fbbcc009b20 comp rx=0 tx=0).stop 2024-04-06T13:47:54.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.391+0000 7fbbd7b32700 1 -- 172.21.15.189:0/76786529 shutdown_connections 2024-04-06T13:47:54.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.391+0000 7fbbd7b32700 1 --2- 172.21.15.189:0/76786529 >> v2:172.21.15.192:3300/0 conn(0x7fbbd00cfa00 0x7fbbd00cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.391+0000 7fbbd7b32700 1 --2- 172.21.15.189:0/76786529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbd00d11b0 0x7fbbd00d1590 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.391+0000 7fbbd7b32700 1 -- 172.21.15.189:0/76786529 >> 172.21.15.189:0/76786529 conn(0x7fbbd00b2db0 msgr2=0x7fbbd00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:54.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.391+0000 7fbbd7b32700 1 -- 172.21.15.189:0/76786529 shutdown_connections 2024-04-06T13:47:54.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.391+0000 7fbbd7b32700 1 -- 172.21.15.189:0/76786529 wait complete. 2024-04-06T13:47:54.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.392+0000 7fbbd7b32700 1 Processor -- start 2024-04-06T13:47:54.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.393+0000 7fbbd7b32700 1 -- start start 2024-04-06T13:47:54.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.394+0000 7fbbd7b32700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbd00cb870 0x7fbbd0152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:54.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.394+0000 7fbbd7b32700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbbd00cfa00 0x7fbbd006d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:54.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.394+0000 7fbbd7b32700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbd00d11b0 0x7fbbd015a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:54.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.394+0000 7fbbd7b32700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd006d890 con 0x7fbbd00cb870 2024-04-06T13:47:54.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.394+0000 7fbbd7b32700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd01530d0 con 0x7fbbd00d11b0 2024-04-06T13:47:54.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.394+0000 7fbbd7b32700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd006d610 con 0x7fbbd00cfa00 2024-04-06T13:47:54.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.394+0000 7fbbd58ce700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbd00cb870 0x7fbbd0152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:54.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.394+0000 7fbbd58ce700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbd00cb870 0x7fbbd0152b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38226/0 (socket says 172.21.15.189:38226) 2024-04-06T13:47:54.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.394+0000 7fbbd58ce700 1 -- 172.21.15.189:0/3021098792 learned_addr learned my addr 172.21.15.189:0/3021098792 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:54.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.394+0000 7fbbd60cf700 1 --2- 172.21.15.189:0/3021098792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbd00d11b0 0x7fbbd015a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:54.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.394+0000 7fbbd50cd700 1 --2- 172.21.15.189:0/3021098792 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbbd00cfa00 0x7fbbd006d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:54.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.394+0000 7fbbd58ce700 1 -- 172.21.15.189:0/3021098792 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbbd00cfa00 msgr2=0x7fbbd006d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:54.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.394+0000 7fbbd58ce700 1 --2- 172.21.15.189:0/3021098792 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbbd00cfa00 0x7fbbd006d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.394+0000 7fbbd58ce700 1 -- 172.21.15.189:0/3021098792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbd00d11b0 msgr2=0x7fbbd015a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:54.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.394+0000 7fbbd58ce700 1 --2- 172.21.15.189:0/3021098792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbd00d11b0 0x7fbbd015a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.394+0000 7fbbd58ce700 1 -- 172.21.15.189:0/3021098792 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbcc0094a0 con 0x7fbbd00cb870 2024-04-06T13:47:54.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.394+0000 7fbbd60cf700 1 --2- 172.21.15.189:0/3021098792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbd00d11b0 0x7fbbd015a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:54.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.394+0000 7fbbd58ce700 1 --2- 172.21.15.189:0/3021098792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbd00cb870 0x7fbbd0152b90 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc0090b0 tx=0x7fbbcc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:54.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.395+0000 7fbbc67fc700 1 -- 172.21.15.189:0/3021098792 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc0036a0 con 0x7fbbd00cb870 2024-04-06T13:47:54.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.395+0000 7fbbc67fc700 1 -- 172.21.15.189:0/3021098792 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbcc01f070 con 0x7fbbd00cb870 2024-04-06T13:47:54.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.395+0000 7fbbd7b32700 1 -- 172.21.15.189:0/3021098792 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbd006b970 con 0x7fbbd00cb870 2024-04-06T13:47:54.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.395+0000 7fbbc67fc700 1 -- 172.21.15.189:0/3021098792 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc003a20 con 0x7fbbd00cb870 2024-04-06T13:47:54.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.395+0000 7fbbd7b32700 1 -- 172.21.15.189:0/3021098792 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbd006bf30 con 0x7fbbd00cb870 2024-04-06T13:47:54.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.396+0000 7fbbd7b32700 1 -- 172.21.15.189:0/3021098792 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbd0074990 con 0x7fbbd00cb870 2024-04-06T13:47:54.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.396+0000 7fbbc67fc700 1 -- 172.21.15.189:0/3021098792 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 146) v1 ==== 89406+0+0 (secure 0 0 0) 0x7fbbcc003b80 con 0x7fbbd00cb870 2024-04-06T13:47:54.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.396+0000 7fbbc67fc700 1 --2- 172.21.15.189:0/3021098792 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbbb406af60 0x7fbbb406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:54.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.396+0000 7fbbd50cd700 1 -- 172.21.15.189:0/3021098792 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbbb406af60 msgr2=0x7fbbb406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:54.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.397+0000 7fbbc67fc700 1 -- 172.21.15.189:0/3021098792 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7fbbcc021070 con 0x7fbbd00cb870 2024-04-06T13:47:54.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.397+0000 7fbbd50cd700 1 --2- 172.21.15.189:0/3021098792 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbbb406af60 0x7fbbb406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:54.401 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:54.398+0000 7f7e16fe6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:47:54.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.401+0000 7fbbc67fc700 1 -- 172.21.15.189:0/3021098792 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbbcc055dd0 con 0x7fbbd00cb870 2024-04-06T13:47:54.459 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:54.459+0000 7f1c6056b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:47:54.532 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:54.532+0000 7f1c6056b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:47:54.564 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:54.563+0000 7f7e16fe6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:47:54.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.598+0000 7fbbd50cd700 1 -- 172.21.15.189:0/3021098792 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbbb406af60 msgr2=0x7fbbb406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:54.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.598+0000 7fbbd50cd700 1 --2- 172.21.15.189:0/3021098792 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbbb406af60 0x7fbbb406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:54.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.604+0000 7fbbd7b32700 1 -- 172.21.15.189:0/3021098792 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7fbbd0072cd0 con 0x7fbbd00cb870 2024-04-06T13:47:54.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.604+0000 7fbbc67fc700 1 -- 172.21.15.189:0/3021098792 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v232) v1 ==== 103+0+11 (secure 0 0 0) 0x7fbbcc0593f0 con 0x7fbbd00cb870 2024-04-06T13:47:54.605 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:47:54.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.607+0000 7fbbd7b32700 1 -- 172.21.15.189:0/3021098792 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbbb406af60 msgr2=0x7fbbb406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:54.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.607+0000 7fbbd7b32700 1 --2- 172.21.15.189:0/3021098792 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbbb406af60 0x7fbbb406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.607+0000 7fbbd7b32700 1 -- 172.21.15.189:0/3021098792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbd00cb870 msgr2=0x7fbbd0152b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:54.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.607+0000 7fbbd7b32700 1 --2- 172.21.15.189:0/3021098792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbd00cb870 0x7fbbd0152b90 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc0090b0 tx=0x7fbbcc004750 comp rx=0 tx=0).stop 2024-04-06T13:47:54.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.607+0000 7fbbd7b32700 1 -- 172.21.15.189:0/3021098792 shutdown_connections 2024-04-06T13:47:54.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.607+0000 7fbbd7b32700 1 --2- 172.21.15.189:0/3021098792 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbbb406af60 0x7fbbb406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.607+0000 7fbbd7b32700 1 --2- 172.21.15.189:0/3021098792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbd00d11b0 0x7fbbd015a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.607+0000 7fbbd7b32700 1 --2- 172.21.15.189:0/3021098792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbd00cb870 0x7fbbd0152b90 unknown :-1 s=CLOSED pgs=2543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.607+0000 7fbbd7b32700 1 --2- 172.21.15.189:0/3021098792 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbbd00cfa00 0x7fbbd006d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.607+0000 7fbbd7b32700 1 -- 172.21.15.189:0/3021098792 >> 172.21.15.189:0/3021098792 conn(0x7fbbd00b2db0 msgr2=0x7fbbd0159d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:54.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.607+0000 7fbbd7b32700 1 -- 172.21.15.189:0/3021098792 shutdown_connections 2024-04-06T13:47:54.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.607+0000 7fbbd7b32700 1 -- 172.21.15.189:0/3021098792 wait complete. 2024-04-06T13:47:54.622 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T13:47:54.642 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:54.642+0000 7f7e16fe6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:47:54.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.747+0000 7f01fed16700 1 Processor -- start 2024-04-06T13:47:54.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.747+0000 7f01fed16700 1 -- start start 2024-04-06T13:47:54.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.748+0000 7f01fed16700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f01f815d590 0x7f01f80b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:54.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.748+0000 7f01fed16700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f80d3890 0x7f01f80b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:54.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.748+0000 7f01fed16700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f80b5cc0 con 0x7f01f80b4ed0 2024-04-06T13:47:54.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.748+0000 7f01fed16700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f80b58c0 con 0x7f01f815d590 2024-04-06T13:47:54.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.748+0000 7f01fed16700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f80d1500 con 0x7f01f80d3890 2024-04-06T13:47:54.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.748+0000 7f01fcab2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f01f815d590 0x7f01f80b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:54.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.748+0000 7f01f7fff700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f01f80b4ed0 0x7f01f80b6580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:40512/0 (socket says 172.21.15.189:40512) 2024-04-06T13:47:54.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.748+0000 7f01fd2b3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f80d3890 0x7f01f80b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:54.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.748+0000 7f01f7fff700 1 -- 172.21.15.189:0/315097612 learned_addr learned my addr 172.21.15.189:0/315097612 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:54.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.748+0000 7f01fcab2700 1 -- 172.21.15.189:0/315097612 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f80d3890 msgr2=0x7f01f80b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:54.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.748+0000 7f01fcab2700 1 --2- 172.21.15.189:0/315097612 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f80d3890 0x7f01f80b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.748+0000 7f01fcab2700 1 -- 172.21.15.189:0/315097612 >> v1:172.21.15.192:6789/0 conn(0x7f01f80b4ed0 legacy=0x7f01f80b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:54.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.748+0000 7f01fcab2700 1 -- 172.21.15.189:0/315097612 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f80d17e0 con 0x7f01f815d590 2024-04-06T13:47:54.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.748+0000 7f01f77fe700 1 -- 172.21.15.189:0/315097612 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301517835 0 0) 0x7f01f80b5cc0 con 0x7f01f80b4ed0 2024-04-06T13:47:54.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.749+0000 7f01fcab2700 1 --2- 172.21.15.189:0/315097612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01f815d590 0x7f01f80b8d70 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009840 tx=0x7f01ec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1731917911efa17 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:54.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.749+0000 7f01f77fe700 1 -- 172.21.15.189:0/315097612 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ec011830 con 0x7f01f815d590 2024-04-06T13:47:54.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.749+0000 7f01f77fe700 1 -- 172.21.15.189:0/315097612 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01ec011990 con 0x7f01f815d590 2024-04-06T13:47:54.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.749+0000 7f01f77fe700 1 -- 172.21.15.189:0/315097612 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ec011d20 con 0x7f01f815d590 2024-04-06T13:47:54.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.749+0000 7f01fed16700 1 -- 172.21.15.189:0/315097612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01f815d590 msgr2=0x7f01f80b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:54.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.749+0000 7f01fed16700 1 --2- 172.21.15.189:0/315097612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01f815d590 0x7f01f80b8d70 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009840 tx=0x7f01ec009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:54.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.750+0000 7f01fed16700 1 -- 172.21.15.189:0/315097612 shutdown_connections 2024-04-06T13:47:54.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.750+0000 7f01fed16700 1 --2- 172.21.15.189:0/315097612 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f80d3890 0x7f01f80b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.750+0000 7f01fed16700 1 --2- 172.21.15.189:0/315097612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01f815d590 0x7f01f80b8d70 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.750+0000 7f01fed16700 1 -- 172.21.15.189:0/315097612 >> 172.21.15.189:0/315097612 conn(0x7f01f806bbc0 msgr2=0x7f01f806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:54.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.750+0000 7f01fed16700 1 -- 172.21.15.189:0/315097612 shutdown_connections 2024-04-06T13:47:54.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.750+0000 7f01fed16700 1 -- 172.21.15.189:0/315097612 wait complete. 2024-04-06T13:47:54.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.751+0000 7f01fed16700 1 Processor -- start 2024-04-06T13:47:54.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.751+0000 7f01fed16700 1 -- start start 2024-04-06T13:47:54.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01fed16700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01f80b4ed0 0x7f01f815e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:54.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01fed16700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01f80d3890 0x7f01f815bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:54.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01fed16700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f815d590 0x7f01f815c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:54.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01fed16700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f81604b0 con 0x7f01f80d3890 2024-04-06T13:47:54.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01fed16700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f81f2f40 con 0x7f01f80b4ed0 2024-04-06T13:47:54.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01fed16700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f815c790 con 0x7f01f815d590 2024-04-06T13:47:54.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01f7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01f80d3890 0x7f01f815bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:54.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01f7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01f80d3890 0x7f01f815bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38242/0 (socket says 172.21.15.189:38242) 2024-04-06T13:47:54.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01f7fff700 1 -- 172.21.15.189:0/128142634 learned_addr learned my addr 172.21.15.189:0/128142634 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:54.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01fd2b3700 1 --2- 172.21.15.189:0/128142634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f815d590 0x7f01f815c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:54.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01fcab2700 1 --2- 172.21.15.189:0/128142634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01f80b4ed0 0x7f01f815e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:54.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01f7fff700 1 -- 172.21.15.189:0/128142634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f815d590 msgr2=0x7f01f815c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:54.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01f7fff700 1 --2- 172.21.15.189:0/128142634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f815d590 0x7f01f815c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01f7fff700 1 -- 172.21.15.189:0/128142634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01f80b4ed0 msgr2=0x7f01f815e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:54.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01f7fff700 1 --2- 172.21.15.189:0/128142634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01f80b4ed0 0x7f01f815e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01f7fff700 1 -- 172.21.15.189:0/128142634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec0094a0 con 0x7f01f80d3890 2024-04-06T13:47:54.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01fcab2700 1 --2- 172.21.15.189:0/128142634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01f80b4ed0 0x7f01f815e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:54.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01f7fff700 1 --2- 172.21.15.189:0/128142634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01f80d3890 0x7f01f815bbb0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7f01e40011e0 tx=0x7f01e400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:54.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01f57fa700 1 -- 172.21.15.189:0/128142634 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e4004c80 con 0x7f01f80d3890 2024-04-06T13:47:54.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01f57fa700 1 -- 172.21.15.189:0/128142634 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e4004e00 con 0x7f01f80d3890 2024-04-06T13:47:54.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.752+0000 7f01fed16700 1 -- 172.21.15.189:0/128142634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f815ca70 con 0x7f01f80d3890 2024-04-06T13:47:54.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.753+0000 7f01f57fa700 1 -- 172.21.15.189:0/128142634 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e4007310 con 0x7f01f80d3890 2024-04-06T13:47:54.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.753+0000 7f01fed16700 1 -- 172.21.15.189:0/128142634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f80cb7b0 con 0x7f01f80d3890 2024-04-06T13:47:54.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.753+0000 7f01fed16700 1 -- 172.21.15.189:0/128142634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f8074990 con 0x7f01f80d3890 2024-04-06T13:47:54.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.754+0000 7f01f57fa700 1 -- 172.21.15.189:0/128142634 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 146) v1 ==== 89406+0+0 (secure 0 0 0) 0x7f01e4007470 con 0x7f01f80d3890 2024-04-06T13:47:54.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.754+0000 7f01f57fa700 1 --2- 172.21.15.189:0/128142634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f01dc06af60 0x7f01dc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:54.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.754+0000 7f01fcab2700 1 -- 172.21.15.189:0/128142634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f01dc06af60 msgr2=0x7f01dc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:54.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.754+0000 7f01f57fa700 1 -- 172.21.15.189:0/128142634 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7f01e4089960 con 0x7f01f80d3890 2024-04-06T13:47:54.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.754+0000 7f01fcab2700 1 --2- 172.21.15.189:0/128142634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f01dc06af60 0x7f01dc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:54.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.759+0000 7f01f57fa700 1 -- 172.21.15.189:0/128142634 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f01e4054240 con 0x7f01f80d3890 2024-04-06T13:47:54.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.955+0000 7f01fcab2700 1 -- 172.21.15.189:0/128142634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f01dc06af60 msgr2=0x7f01dc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:54.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.955+0000 7f01fcab2700 1 --2- 172.21.15.189:0/128142634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f01dc06af60 0x7f01dc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:54.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.976+0000 7f01fed16700 1 -- 172.21.15.189:0/128142634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f01f80ce6a0 con 0x7f01f80d3890 2024-04-06T13:47:54.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.977+0000 7f01f57fa700 1 -- 172.21.15.189:0/128142634 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v232) v1 ==== 80+0+470 (secure 0 0 0) 0x7f01e4057860 con 0x7f01f80d3890 2024-04-06T13:47:54.978 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/3920072072 2024-04-07T13:47:47.129296+0000 2024-04-06T13:47:54.978 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1878657401 2024-04-07T13:47:47.129296+0000 2024-04-06T13:47:54.978 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6833/111704 2024-04-07T13:47:47.129296+0000 2024-04-06T13:47:54.978 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6832/111704 2024-04-07T13:47:47.129296+0000 2024-04-06T13:47:54.978 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2827130089 2024-04-07T13:47:47.129296+0000 2024-04-06T13:47:54.978 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2716712615 2024-04-07T13:47:47.129296+0000 2024-04-06T13:47:54.978 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1807905335 2024-04-07T13:47:47.129296+0000 2024-04-06T13:47:54.978 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1173010452 2024-04-07T13:47:47.129296+0000 2024-04-06T13:47:54.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.979+0000 7f01fed16700 1 -- 172.21.15.189:0/128142634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f01dc06af60 msgr2=0x7f01dc06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:54.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.979+0000 7f01fed16700 1 --2- 172.21.15.189:0/128142634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f01dc06af60 0x7f01dc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.979+0000 7f01fed16700 1 -- 172.21.15.189:0/128142634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01f80d3890 msgr2=0x7f01f815bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:54.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.979+0000 7f01fed16700 1 --2- 172.21.15.189:0/128142634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01f80d3890 0x7f01f815bbb0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7f01e40011e0 tx=0x7f01e400ca30 comp rx=0 tx=0).stop 2024-04-06T13:47:54.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.979+0000 7f01fed16700 1 -- 172.21.15.189:0/128142634 shutdown_connections 2024-04-06T13:47:54.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.979+0000 7f01fed16700 1 --2- 172.21.15.189:0/128142634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f01dc06af60 0x7f01dc06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.979+0000 7f01fed16700 1 --2- 172.21.15.189:0/128142634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f815d590 0x7f01f815c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.979+0000 7f01fed16700 1 --2- 172.21.15.189:0/128142634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01f80b4ed0 0x7f01f815e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.979+0000 7f01fed16700 1 --2- 172.21.15.189:0/128142634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01f80d3890 0x7f01f815bbb0 unknown :-1 s=CLOSED pgs=2544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:54.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.979+0000 7f01fed16700 1 -- 172.21.15.189:0/128142634 >> 172.21.15.189:0/128142634 conn(0x7f01f806bbc0 msgr2=0x7f01f80cd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:54.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.979+0000 7f01fed16700 1 -- 172.21.15.189:0/128142634 shutdown_connections 2024-04-06T13:47:54.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:54.979+0000 7f01fed16700 1 -- 172.21.15.189:0/128142634 wait complete. 2024-04-06T13:47:54.981 INFO:teuthology.orchestra.run.smithi189.stderr:listed 8 entries 2024-04-06T13:47:54.992 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-06T13:47:54.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:54.993 DEBUG:teuthology.orchestra.run.smithi189:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-06T13:47:55.084 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T13:47:55.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:47:55.085 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-06T13:47:55.102 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:47:55.102 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:47:55.119 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:47:55.119 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:47:55.119 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:47:55.149 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:55.149+0000 7f1c6056b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:47:55.159 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:47:55.160 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:47:55.160 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:47:55.178 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:47:55.179 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:47:55.234 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:55.231+0000 7f1c6056b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:47:55.303 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:55.302+0000 7f7e16fe6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:47:55.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.324+0000 7f69b0b96700 1 Processor -- start 2024-04-06T13:47:55.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.324+0000 7f69b0b96700 1 -- start start 2024-04-06T13:47:55.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.325+0000 7f69b0b96700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69ac0b2cd0 0x7f69ac06c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:55.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.325+0000 7f69b0b96700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f69ac0b2680 0x7f69ac06ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:55.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.325+0000 7f69b0b96700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69ac15d8d0 con 0x7f69ac15d040 2024-04-06T13:47:55.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.325+0000 7f69b0b96700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69ac0cba40 con 0x7f69ac0b2680 2024-04-06T13:47:55.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.325+0000 7f69b0b96700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac15d580 con 0x7f69ac0b2cd0 2024-04-06T13:47:55.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.325+0000 7f69aa59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f69ac15d040 0x7f69ac15dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58322/0 (socket says 172.21.15.189:58322) 2024-04-06T13:47:55.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.325+0000 7f69aa59c700 1 -- 172.21.15.189:0/3673845199 learned_addr learned my addr 172.21.15.189:0/3673845199 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:55.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.325+0000 7f69a9d9b700 1 --2- 172.21.15.189:0/3673845199 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69ac0b2cd0 0x7f69ac06c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:55.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.325+0000 7f69aad9d700 1 --2- 172.21.15.189:0/3673845199 >> v2:172.21.15.192:3300/0 conn(0x7f69ac0b2680 0x7f69ac06ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:55.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.325+0000 7f69a959a700 1 -- 172.21.15.189:0/3673845199 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770552375 0 0) 0x7f69ac15d8d0 con 0x7f69ac15d040 2024-04-06T13:47:55.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.325+0000 7f69a959a700 1 -- 172.21.15.189:0/3673845199 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69940035b0 con 0x7f69ac15d040 2024-04-06T13:47:55.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.326+0000 7f69a9d9b700 1 -- 172.21.15.189:0/3673845199 >> v2:172.21.15.192:3300/0 conn(0x7f69ac0b2680 msgr2=0x7f69ac06ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:55.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.326+0000 7f69a9d9b700 1 --2- 172.21.15.189:0/3673845199 >> v2:172.21.15.192:3300/0 conn(0x7f69ac0b2680 0x7f69ac06ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.326+0000 7f69a9d9b700 1 -- 172.21.15.189:0/3673845199 >> v1:172.21.15.189:6789/0 conn(0x7f69ac15d040 legacy=0x7f69ac15dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:55.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.326+0000 7f69a9d9b700 1 -- 172.21.15.189:0/3673845199 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ac15e470 con 0x7f69ac0b2cd0 2024-04-06T13:47:55.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.326+0000 7f69a9d9b700 1 --2- 172.21.15.189:0/3673845199 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69ac0b2cd0 0x7f69ac06c4e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f69a0009840 tx=0x7f69a0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a772735bf836b6a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:55.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.326+0000 7f69a959a700 1 -- 172.21.15.189:0/3673845199 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a0011830 con 0x7f69ac0b2cd0 2024-04-06T13:47:55.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.326+0000 7f69a959a700 1 -- 172.21.15.189:0/3673845199 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a0011990 con 0x7f69ac0b2cd0 2024-04-06T13:47:55.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.326+0000 7f69a959a700 1 -- 172.21.15.189:0/3673845199 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a0011d20 con 0x7f69ac0b2cd0 2024-04-06T13:47:55.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.326+0000 7f69b0b96700 1 -- 172.21.15.189:0/3673845199 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69ac0b2cd0 msgr2=0x7f69ac06c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:55.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.326+0000 7f69b0b96700 1 --2- 172.21.15.189:0/3673845199 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69ac0b2cd0 0x7f69ac06c4e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f69a0009840 tx=0x7f69a0009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:55.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.327+0000 7f69b0b96700 1 -- 172.21.15.189:0/3673845199 shutdown_connections 2024-04-06T13:47:55.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.327+0000 7f69b0b96700 1 --2- 172.21.15.189:0/3673845199 >> v2:172.21.15.192:3300/0 conn(0x7f69ac0b2680 0x7f69ac06ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.327+0000 7f69b0b96700 1 --2- 172.21.15.189:0/3673845199 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69ac0b2cd0 0x7f69ac06c4e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.327+0000 7f69b0b96700 1 -- 172.21.15.189:0/3673845199 >> 172.21.15.189:0/3673845199 conn(0x7f69ac06a730 msgr2=0x7f69ac06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:55.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.327+0000 7f69b0b96700 1 -- 172.21.15.189:0/3673845199 shutdown_connections 2024-04-06T13:47:55.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.327+0000 7f69b0b96700 1 -- 172.21.15.189:0/3673845199 wait complete. 2024-04-06T13:47:55.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.328+0000 7f69b0b96700 1 Processor -- start 2024-04-06T13:47:55.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.328+0000 7f69b0b96700 1 -- start start 2024-04-06T13:47:55.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.329+0000 7f69b0b96700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69ac0b2680 0x7f69ac15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:55.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.329+0000 7f69b0b96700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69ac0b2cd0 0x7f69ac159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:55.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.329+0000 7f69b0b96700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69ac15d040 0x7f69ac1540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:55.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.329+0000 7f69b0b96700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac0ad820 con 0x7f69ac0b2cd0 2024-04-06T13:47:55.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.329+0000 7f69b0b96700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac0ada90 con 0x7f69ac0b2680 2024-04-06T13:47:55.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.329+0000 7f69b0b96700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac15b0c0 con 0x7f69ac15d040 2024-04-06T13:47:55.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.329+0000 7f69a9d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69ac0b2cd0 0x7f69ac159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:55.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.329+0000 7f69a9d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69ac0b2cd0 0x7f69ac159770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38258/0 (socket says 172.21.15.189:38258) 2024-04-06T13:47:55.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.329+0000 7f69a9d9b700 1 -- 172.21.15.189:0/3678034262 learned_addr learned my addr 172.21.15.189:0/3678034262 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:55.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.329+0000 7f69aa59c700 1 --2- 172.21.15.189:0/3678034262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69ac0b2680 0x7f69ac15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:55.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.329+0000 7f69aad9d700 1 --2- 172.21.15.189:0/3678034262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69ac15d040 0x7f69ac1540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:55.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.329+0000 7f69aa59c700 1 -- 172.21.15.189:0/3678034262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69ac15d040 msgr2=0x7f69ac1540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:55.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.329+0000 7f69aa59c700 1 --2- 172.21.15.189:0/3678034262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69ac15d040 0x7f69ac1540c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.329+0000 7f69aa59c700 1 -- 172.21.15.189:0/3678034262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69ac0b2cd0 msgr2=0x7f69ac159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:55.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.329+0000 7f69aa59c700 1 --2- 172.21.15.189:0/3678034262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69ac0b2cd0 0x7f69ac159770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.329+0000 7f69aa59c700 1 -- 172.21.15.189:0/3678034262 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a00094a0 con 0x7f69ac0b2680 2024-04-06T13:47:55.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.329+0000 7f69a9d9b700 1 --2- 172.21.15.189:0/3678034262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69ac0b2cd0 0x7f69ac159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:55.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.329+0000 7f69aad9d700 1 --2- 172.21.15.189:0/3678034262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69ac15d040 0x7f69ac1540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:55.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.330+0000 7f69aa59c700 1 --2- 172.21.15.189:0/3678034262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69ac0b2680 0x7f69ac15a9b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f699c00c460 tx=0x7f699c00c770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:55.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.330+0000 7f699affd700 1 -- 172.21.15.189:0/3678034262 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699c005d70 con 0x7f69ac0b2680 2024-04-06T13:47:55.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.330+0000 7f699affd700 1 -- 172.21.15.189:0/3678034262 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699c0070d0 con 0x7f69ac0b2680 2024-04-06T13:47:55.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.330+0000 7f699affd700 1 -- 172.21.15.189:0/3678034262 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699c007460 con 0x7f69ac0b2680 2024-04-06T13:47:55.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.330+0000 7f69b0b96700 1 -- 172.21.15.189:0/3678034262 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69ac15b330 con 0x7f69ac0b2680 2024-04-06T13:47:55.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.330+0000 7f69b0b96700 1 -- 172.21.15.189:0/3678034262 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69ac1f7370 con 0x7f69ac0b2680 2024-04-06T13:47:55.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.331+0000 7f69b0b96700 1 -- 172.21.15.189:0/3678034262 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69ac0add70 con 0x7f69ac0b2680 2024-04-06T13:47:55.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.331+0000 7f699affd700 1 -- 172.21.15.189:0/3678034262 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 146) v1 ==== 89406+0+0 (secure 0 0 0) 0x7f699c0075c0 con 0x7f69ac0b2680 2024-04-06T13:47:55.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.332+0000 7f699affd700 1 --2- 172.21.15.189:0/3678034262 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f698c06af60 0x7f698c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:55.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.332+0000 7f69a9d9b700 1 -- 172.21.15.189:0/3678034262 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f698c06af60 msgr2=0x7f698c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:55.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.332+0000 7f699affd700 1 -- 172.21.15.189:0/3678034262 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7f699c088ce0 con 0x7f69ac0b2680 2024-04-06T13:47:55.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.332+0000 7f69a9d9b700 1 --2- 172.21.15.189:0/3678034262 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f698c06af60 0x7f698c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:55.337 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.337+0000 7f699affd700 1 -- 172.21.15.189:0/3678034262 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f699c0535c0 con 0x7f69ac0b2680 2024-04-06T13:47:55.358 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:55.358+0000 7f1c6056b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:47:55.386 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:55.385+0000 7f7e16fe6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:47:55.517 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:55.516+0000 7f7e16fe6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:47:55.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.533+0000 7f69a9d9b700 1 -- 172.21.15.189:0/3678034262 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f698c06af60 msgr2=0x7f698c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:55.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.533+0000 7f69a9d9b700 1 --2- 172.21.15.189:0/3678034262 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f698c06af60 0x7f698c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.585+0000 7f69b0b96700 1 -- 172.21.15.189:0/3678034262 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69ac073270 con 0x7f69ac0b2680 2024-04-06T13:47:55.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.585+0000 7f699affd700 1 -- 172.21.15.189:0/3678034262 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5971 (secure 0 0 0) 0x7f699c056be0 con 0x7f69ac0b2680 2024-04-06T13:47:55.586 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:55.587 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":106,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8885,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1481817364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1481817364},{"type":"v1","addr":"172.21.15.189:6837","nonce":1481817364}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":11244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/4021875447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":4021875447},{"type":"v1","addr":"172.21.15.189:6836","nonce":4021875447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:47:22.116705+0000","modified":"2024-04-06T13:47:25.129787+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/694564673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":694564673},{"type":"v1","addr":"172.21.15.192:6835","nonce":694564673}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:47:34.584208+0000","modified":"2024-04-06T13:47:37.592983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11262":{"gid":11262,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6834/4066431971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4066431971},{"type":"v1","addr":"172.21.15.192:6834","nonce":4066431971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-04-06T13:47:55.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.588+0000 7f69b0b96700 1 -- 172.21.15.189:0/3678034262 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f698c06af60 msgr2=0x7f698c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:55.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.588+0000 7f69b0b96700 1 --2- 172.21.15.189:0/3678034262 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f698c06af60 0x7f698c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.588+0000 7f69b0b96700 1 -- 172.21.15.189:0/3678034262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69ac0b2680 msgr2=0x7f69ac15a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:55.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.588+0000 7f69b0b96700 1 --2- 172.21.15.189:0/3678034262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69ac0b2680 0x7f69ac15a9b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f699c00c460 tx=0x7f699c00c770 comp rx=0 tx=0).stop 2024-04-06T13:47:55.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.588+0000 7f69b0b96700 1 -- 172.21.15.189:0/3678034262 shutdown_connections 2024-04-06T13:47:55.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.588+0000 7f69b0b96700 1 --2- 172.21.15.189:0/3678034262 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f698c06af60 0x7f698c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.588+0000 7f69b0b96700 1 --2- 172.21.15.189:0/3678034262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69ac15d040 0x7f69ac1540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.588+0000 7f69b0b96700 1 --2- 172.21.15.189:0/3678034262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69ac0b2680 0x7f69ac15a9b0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.588+0000 7f69b0b96700 1 --2- 172.21.15.189:0/3678034262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69ac0b2cd0 0x7f69ac159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.588+0000 7f69b0b96700 1 -- 172.21.15.189:0/3678034262 >> 172.21.15.189:0/3678034262 conn(0x7f69ac06a730 msgr2=0x7f69ac1518b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:55.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.588+0000 7f69b0b96700 1 -- 172.21.15.189:0/3678034262 shutdown_connections 2024-04-06T13:47:55.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.588+0000 7f69b0b96700 1 -- 172.21.15.189:0/3678034262 wait complete. 2024-04-06T13:47:55.590 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 106 2024-04-06T13:47:55.602 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:47:55.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.730+0000 7f7c3e31b700 1 Processor -- start 2024-04-06T13:47:55.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.731+0000 7f7c3e31b700 1 -- start start 2024-04-06T13:47:55.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.731+0000 7f7c3e31b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7c3815d7e0 0x7f7c380d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:55.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.731+0000 7f7c3e31b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7c380d0890 0x7f7c380d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:55.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.731+0000 7f7c3e31b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c380d1af0 0x7f7c380b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:55.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.731+0000 7f7c3e31b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c380d1f80 con 0x7f7c380d0890 2024-04-06T13:47:55.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.731+0000 7f7c3e31b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c380cdf80 con 0x7f7c3815d7e0 2024-04-06T13:47:55.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.731+0000 7f7c3e31b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c380d0cf0 con 0x7f7c380d1af0 2024-04-06T13:47:55.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.732+0000 7f7c377fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7c380d0890 0x7f7c380d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:55.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.732+0000 7f7c377fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7c380d0890 0x7f7c380d15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38270/0 (socket says 172.21.15.189:38270) 2024-04-06T13:47:55.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.732+0000 7f7c377fe700 1 -- 172.21.15.189:0/899149167 learned_addr learned my addr 172.21.15.189:0/899149167 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:55.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.732+0000 7f7c3c8b8700 1 --2- 172.21.15.189:0/899149167 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c380d1af0 0x7f7c380b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:55.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.732+0000 7f7c37fff700 1 --2- 172.21.15.189:0/899149167 >> v2:172.21.15.192:3300/0 conn(0x7f7c3815d7e0 0x7f7c380d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:55.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.732+0000 7f7c377fe700 1 -- 172.21.15.189:0/899149167 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c380d1af0 msgr2=0x7f7c380b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:55.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.732+0000 7f7c377fe700 1 --2- 172.21.15.189:0/899149167 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c380d1af0 0x7f7c380b4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.732+0000 7f7c377fe700 1 -- 172.21.15.189:0/899149167 >> v2:172.21.15.192:3300/0 conn(0x7f7c3815d7e0 msgr2=0x7f7c380d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:55.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.732+0000 7f7c377fe700 1 --2- 172.21.15.189:0/899149167 >> v2:172.21.15.192:3300/0 conn(0x7f7c3815d7e0 0x7f7c380d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.732+0000 7f7c377fe700 1 -- 172.21.15.189:0/899149167 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c380d3230 con 0x7f7c380d0890 2024-04-06T13:47:55.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.732+0000 7f7c3c8b8700 1 --2- 172.21.15.189:0/899149167 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c380d1af0 0x7f7c380b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:55.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.732+0000 7f7c377fe700 1 --2- 172.21.15.189:0/899149167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c380d0890 0x7f7c380d15b0 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f7c3808c190 tx=0x7f7c2c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b8c7329f58c50e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:55.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.732+0000 7f7c36ffd700 1 -- 172.21.15.189:0/899149167 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c011830 con 0x7f7c380d0890 2024-04-06T13:47:55.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.732+0000 7f7c36ffd700 1 -- 172.21.15.189:0/899149167 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c2c011990 con 0x7f7c380d0890 2024-04-06T13:47:55.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.732+0000 7f7c36ffd700 1 -- 172.21.15.189:0/899149167 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c011d20 con 0x7f7c380d0890 2024-04-06T13:47:55.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.733+0000 7f7c3e31b700 1 -- 172.21.15.189:0/899149167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c380d0890 msgr2=0x7f7c380d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:55.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.733+0000 7f7c3e31b700 1 --2- 172.21.15.189:0/899149167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c380d0890 0x7f7c380d15b0 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f7c3808c190 tx=0x7f7c2c009bd0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.733+0000 7f7c3e31b700 1 -- 172.21.15.189:0/899149167 shutdown_connections 2024-04-06T13:47:55.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.733+0000 7f7c3e31b700 1 --2- 172.21.15.189:0/899149167 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c380d1af0 0x7f7c380b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.733+0000 7f7c3e31b700 1 --2- 172.21.15.189:0/899149167 >> v2:172.21.15.192:3300/0 conn(0x7f7c3815d7e0 0x7f7c380d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.733+0000 7f7c3e31b700 1 --2- 172.21.15.189:0/899149167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c380d0890 0x7f7c380d15b0 unknown :-1 s=CLOSED pgs=2546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.733+0000 7f7c3e31b700 1 -- 172.21.15.189:0/899149167 >> 172.21.15.189:0/899149167 conn(0x7f7c38072250 msgr2=0x7f7c38072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:55.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.733+0000 7f7c3e31b700 1 -- 172.21.15.189:0/899149167 shutdown_connections 2024-04-06T13:47:55.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.733+0000 7f7c3e31b700 1 -- 172.21.15.189:0/899149167 wait complete. 2024-04-06T13:47:55.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.734+0000 7f7c3e31b700 1 Processor -- start 2024-04-06T13:47:55.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.734+0000 7f7c3e31b700 1 -- start start 2024-04-06T13:47:55.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.735+0000 7f7c3e31b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c380d0890 0x7f7c38155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:55.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.735+0000 7f7c3e31b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c380d1af0 0x7f7c381506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:55.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.735+0000 7f7c3e31b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c3815d7e0 0x7f7c380cc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:55.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.735+0000 7f7c3e31b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c38150be0 con 0x7f7c380d1af0 2024-04-06T13:47:55.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.735+0000 7f7c3e31b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c380cd400 con 0x7f7c3815d7e0 2024-04-06T13:47:55.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.735+0000 7f7c3e31b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c381580b0 con 0x7f7c380d0890 2024-04-06T13:47:55.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.735+0000 7f7c377fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c380d1af0 0x7f7c381506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:55.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.735+0000 7f7c377fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c380d1af0 0x7f7c381506a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38286/0 (socket says 172.21.15.189:38286) 2024-04-06T13:47:55.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.735+0000 7f7c377fe700 1 -- 172.21.15.189:0/974863584 learned_addr learned my addr 172.21.15.189:0/974863584 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:55.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.735+0000 7f7c37fff700 1 --2- 172.21.15.189:0/974863584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c380d0890 0x7f7c38155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:55.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.735+0000 7f7c3c8b8700 1 --2- 172.21.15.189:0/974863584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c3815d7e0 0x7f7c380cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:55.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.735+0000 7f7c377fe700 1 -- 172.21.15.189:0/974863584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c380d0890 msgr2=0x7f7c38155080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:55.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.735+0000 7f7c377fe700 1 --2- 172.21.15.189:0/974863584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c380d0890 0x7f7c38155080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.735+0000 7f7c377fe700 1 -- 172.21.15.189:0/974863584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c3815d7e0 msgr2=0x7f7c380cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:55.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.735+0000 7f7c377fe700 1 --2- 172.21.15.189:0/974863584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c3815d7e0 0x7f7c380cc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.735+0000 7f7c377fe700 1 -- 172.21.15.189:0/974863584 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c30009a70 con 0x7f7c380d1af0 2024-04-06T13:47:55.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.736+0000 7f7c377fe700 1 --2- 172.21.15.189:0/974863584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c380d1af0 0x7f7c381506a0 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c009c10 tx=0x7f7c2c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:55.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.736+0000 7f7c34ff9700 1 -- 172.21.15.189:0/974863584 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c004450 con 0x7f7c380d1af0 2024-04-06T13:47:55.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.736+0000 7f7c34ff9700 1 -- 172.21.15.189:0/974863584 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c2c01e070 con 0x7f7c380d1af0 2024-04-06T13:47:55.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.736+0000 7f7c3e31b700 1 -- 172.21.15.189:0/974863584 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2c009550 con 0x7f7c380d1af0 2024-04-06T13:47:55.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.736+0000 7f7c34ff9700 1 -- 172.21.15.189:0/974863584 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c0038e0 con 0x7f7c380d1af0 2024-04-06T13:47:55.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.736+0000 7f7c3e31b700 1 -- 172.21.15.189:0/974863584 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c380ccc00 con 0x7f7c380d1af0 2024-04-06T13:47:55.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.737+0000 7f7c3e31b700 1 -- 172.21.15.189:0/974863584 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c38074990 con 0x7f7c380d1af0 2024-04-06T13:47:55.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.737+0000 7f7c34ff9700 1 -- 172.21.15.189:0/974863584 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 146) v1 ==== 89406+0+0 (secure 0 0 0) 0x7f7c2c003a40 con 0x7f7c380d1af0 2024-04-06T13:47:55.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.738+0000 7f7c34ff9700 1 --2- 172.21.15.189:0/974863584 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7c2406af60 0x7f7c2406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:55.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.738+0000 7f7c37fff700 1 -- 172.21.15.189:0/974863584 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7c2406af60 msgr2=0x7f7c2406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:55.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.738+0000 7f7c34ff9700 1 -- 172.21.15.189:0/974863584 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7f7c2c08a8d0 con 0x7f7c380d1af0 2024-04-06T13:47:55.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.738+0000 7f7c37fff700 1 --2- 172.21.15.189:0/974863584 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7c2406af60 0x7f7c2406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:55.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.742+0000 7f7c34ff9700 1 -- 172.21.15.189:0/974863584 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7c2c055170 con 0x7f7c380d1af0 2024-04-06T13:47:55.870 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:55.870+0000 7f1c6056b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:47:55.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.939+0000 7f7c37fff700 1 -- 172.21.15.189:0/974863584 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7c2406af60 msgr2=0x7f7c2406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:55.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.939+0000 7f7c37fff700 1 --2- 172.21.15.189:0/974863584 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7c2406af60 0x7f7c2406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:55.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.947+0000 7f7c3e31b700 1 -- 172.21.15.189:0/974863584 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c380d3230 con 0x7f7c380d1af0 2024-04-06T13:47:55.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.949+0000 7f7c34ff9700 1 -- 172.21.15.189:0/974863584 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) v1 ==== 74+0+20637 (secure 0 0 0) 0x7f7c380d3230 con 0x7f7c380d1af0 2024-04-06T13:47:55.950 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:55.950 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":232,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:47:47.129326+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:47:20.644275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-04-06T13:47:21.075143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:47:32.292623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:47:33.567052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/3920072072":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/1878657401":"2024-04-07T13:47:47.129296+0000","172.21.15.189:6833/111704":"2024-04-07T13:47:47.129296+0000","172.21.15.189:6832/111704":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/2827130089":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/2716712615":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/1807905335":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/1173010452":"2024-04-07T13:47:47.129296+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:47:55.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.951+0000 7f7c3e31b700 1 -- 172.21.15.189:0/974863584 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7c2406af60 msgr2=0x7f7c2406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:55.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.951+0000 7f7c3e31b700 1 --2- 172.21.15.189:0/974863584 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7c2406af60 0x7f7c2406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.951+0000 7f7c3e31b700 1 -- 172.21.15.189:0/974863584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c380d1af0 msgr2=0x7f7c381506a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:55.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.951+0000 7f7c3e31b700 1 --2- 172.21.15.189:0/974863584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c380d1af0 0x7f7c381506a0 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c009c10 tx=0x7f7c2c004750 comp rx=0 tx=0).stop 2024-04-06T13:47:55.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.952+0000 7f7c3e31b700 1 -- 172.21.15.189:0/974863584 shutdown_connections 2024-04-06T13:47:55.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.952+0000 7f7c3e31b700 1 --2- 172.21.15.189:0/974863584 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7c2406af60 0x7f7c2406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.952+0000 7f7c3e31b700 1 --2- 172.21.15.189:0/974863584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c3815d7e0 0x7f7c380cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.952+0000 7f7c3e31b700 1 --2- 172.21.15.189:0/974863584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7c380d0890 0x7f7c38155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.952+0000 7f7c3e31b700 1 --2- 172.21.15.189:0/974863584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c380d1af0 0x7f7c381506a0 unknown :-1 s=CLOSED pgs=2547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:55.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.952+0000 7f7c3e31b700 1 -- 172.21.15.189:0/974863584 >> 172.21.15.189:0/974863584 conn(0x7f7c38072250 msgr2=0x7f7c38160240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:55.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.952+0000 7f7c3e31b700 1 -- 172.21.15.189:0/974863584 shutdown_connections 2024-04-06T13:47:55.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:55.952+0000 7f7c3e31b700 1 -- 172.21.15.189:0/974863584 wait complete. 2024-04-06T13:47:55.966 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T13:47:20.644275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '226', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T13:47:21.075143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '227', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-04-06T13:47:32.292623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '230', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2024-04-06T13:47:33.567052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '231', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:47:55.966 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:47:56.024 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:56.024+0000 7f1c6056b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:47:56.056 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:56.056+0000 7f7e16fe6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:47:56.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.092+0000 7f533b2b6700 1 Processor -- start 2024-04-06T13:47:56.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.092+0000 7f533b2b6700 1 -- start start 2024-04-06T13:47:56.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.093+0000 7f533b2b6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f533415d590 0x7f53340b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.093+0000 7f533b2b6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53340d3890 0x7f53340b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.093+0000 7f533b2b6700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53340b5cc0 con 0x7f53340b4ed0 2024-04-06T13:47:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.093+0000 7f533b2b6700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53340b58c0 con 0x7f533415d590 2024-04-06T13:47:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.093+0000 7f533b2b6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53340d1500 con 0x7f53340d3890 2024-04-06T13:47:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.093+0000 7f5339052700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f533415d590 0x7f53340b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.093+0000 7f5339052700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f533415d590 0x7f53340b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38300/0 (socket says 172.21.15.189:38300) 2024-04-06T13:47:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.093+0000 7f5339052700 1 -- 172.21.15.189:0/2815277463 learned_addr learned my addr 172.21.15.189:0/2815277463 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.093+0000 7f5339853700 1 --2- 172.21.15.189:0/2815277463 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53340d3890 0x7f53340b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.093+0000 7f5339052700 1 -- 172.21.15.189:0/2815277463 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53340d3890 msgr2=0x7f53340b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.093+0000 7f5339052700 1 --2- 172.21.15.189:0/2815277463 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53340d3890 0x7f53340b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.093+0000 7f5339052700 1 -- 172.21.15.189:0/2815277463 >> v1:172.21.15.192:6789/0 conn(0x7f53340b4ed0 legacy=0x7f53340b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.093+0000 7f5339052700 1 -- 172.21.15.189:0/2815277463 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53340d17e0 con 0x7f533415d590 2024-04-06T13:47:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.094+0000 7f5339052700 1 --2- 172.21.15.189:0/2815277463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f533415d590 0x7f53340b8d70 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7f53300098f0 tx=0x7f5330009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f1fee6af63e6cd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.094+0000 7f532bfff700 1 -- 172.21.15.189:0/2815277463 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330011830 con 0x7f533415d590 2024-04-06T13:47:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.094+0000 7f532bfff700 1 -- 172.21.15.189:0/2815277463 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5330011990 con 0x7f533415d590 2024-04-06T13:47:56.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.094+0000 7f532bfff700 1 -- 172.21.15.189:0/2815277463 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330011d20 con 0x7f533415d590 2024-04-06T13:47:56.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.094+0000 7f533b2b6700 1 -- 172.21.15.189:0/2815277463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f533415d590 msgr2=0x7f53340b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:56.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.094+0000 7f533b2b6700 1 --2- 172.21.15.189:0/2815277463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f533415d590 0x7f53340b8d70 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7f53300098f0 tx=0x7f5330009c00 comp rx=0 tx=0).stop 2024-04-06T13:47:56.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.094+0000 7f533b2b6700 1 -- 172.21.15.189:0/2815277463 shutdown_connections 2024-04-06T13:47:56.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.094+0000 7f533b2b6700 1 --2- 172.21.15.189:0/2815277463 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53340d3890 0x7f53340b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.094+0000 7f533b2b6700 1 --2- 172.21.15.189:0/2815277463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f533415d590 0x7f53340b8d70 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.094+0000 7f533b2b6700 1 -- 172.21.15.189:0/2815277463 >> 172.21.15.189:0/2815277463 conn(0x7f533406bbc0 msgr2=0x7f533406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:56.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.094+0000 7f533b2b6700 1 -- 172.21.15.189:0/2815277463 shutdown_connections 2024-04-06T13:47:56.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.094+0000 7f533b2b6700 1 -- 172.21.15.189:0/2815277463 wait complete. 2024-04-06T13:47:56.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.096+0000 7f533b2b6700 1 Processor -- start 2024-04-06T13:47:56.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.096+0000 7f533b2b6700 1 -- start start 2024-04-06T13:47:56.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.096+0000 7f533b2b6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53340b4ed0 0x7f53340cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:56.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.096+0000 7f533b2b6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53340d3890 0x7f53340cb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:56.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.096+0000 7f533b2b6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f533415d590 0x7f533414f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:56.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.096+0000 7f533b2b6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533415aba0 con 0x7f533415d590 2024-04-06T13:47:56.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.096+0000 7f533b2b6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53340ccac0 con 0x7f53340b4ed0 2024-04-06T13:47:56.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.096+0000 7f533b2b6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53340ccda0 con 0x7f53340d3890 2024-04-06T13:47:56.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.096+0000 7f5339853700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f533415d590 0x7f533414f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:56.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.096+0000 7f5339853700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f533415d590 0x7f533414f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38314/0 (socket says 172.21.15.189:38314) 2024-04-06T13:47:56.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.096+0000 7f5339853700 1 -- 172.21.15.189:0/1960542484 learned_addr learned my addr 172.21.15.189:0/1960542484 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:56.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.096+0000 7f5339052700 1 --2- 172.21.15.189:0/1960542484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53340b4ed0 0x7f53340cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:56.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.096+0000 7f5338851700 1 --2- 172.21.15.189:0/1960542484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53340d3890 0x7f53340cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:56.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.097+0000 7f5339853700 1 -- 172.21.15.189:0/1960542484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53340d3890 msgr2=0x7f53340cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:56.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.097+0000 7f5339853700 1 --2- 172.21.15.189:0/1960542484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53340d3890 0x7f53340cb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.097+0000 7f5339853700 1 -- 172.21.15.189:0/1960542484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53340b4ed0 msgr2=0x7f53340cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:56.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.097+0000 7f5339853700 1 --2- 172.21.15.189:0/1960542484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53340b4ed0 0x7f53340cc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.097+0000 7f5339853700 1 -- 172.21.15.189:0/1960542484 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5330009550 con 0x7f533415d590 2024-04-06T13:47:56.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.097+0000 7f5339853700 1 --2- 172.21.15.189:0/1960542484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f533415d590 0x7f533414f990 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f532c00efc0 tx=0x7f532c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:56.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.097+0000 7f5329ffb700 1 -- 172.21.15.189:0/1960542484 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532c009ea0 con 0x7f533415d590 2024-04-06T13:47:56.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.097+0000 7f5329ffb700 1 -- 172.21.15.189:0/1960542484 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f532c00f040 con 0x7f533415d590 2024-04-06T13:47:56.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.097+0000 7f533b2b6700 1 -- 172.21.15.189:0/1960542484 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53340cd080 con 0x7f533415d590 2024-04-06T13:47:56.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.097+0000 7f5329ffb700 1 -- 172.21.15.189:0/1960542484 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532c00ced0 con 0x7f533415d590 2024-04-06T13:47:56.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.097+0000 7f533b2b6700 1 -- 172.21.15.189:0/1960542484 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53341500b0 con 0x7f533415d590 2024-04-06T13:47:56.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.098+0000 7f533b2b6700 1 -- 172.21.15.189:0/1960542484 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f533415a900 con 0x7f533415d590 2024-04-06T13:47:56.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.099+0000 7f5329ffb700 1 -- 172.21.15.189:0/1960542484 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 146) v1 ==== 89406+0+0 (secure 0 0 0) 0x7f532c014480 con 0x7f533415d590 2024-04-06T13:47:56.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.099+0000 7f5329ffb700 1 --2- 172.21.15.189:0/1960542484 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f531806af60 0x7f531806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:56.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.099+0000 7f5339052700 1 -- 172.21.15.189:0/1960542484 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f531806af60 msgr2=0x7f531806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:56.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.099+0000 7f5329ffb700 1 -- 172.21.15.189:0/1960542484 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7f532c08a910 con 0x7f533415d590 2024-04-06T13:47:56.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.099+0000 7f5339052700 1 --2- 172.21.15.189:0/1960542484 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f531806af60 0x7f531806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:56.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.103+0000 7f5329ffb700 1 -- 172.21.15.189:0/1960542484 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f533415a900 con 0x7f533415d590 2024-04-06T13:47:56.180 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:56.180+0000 7f1c6056b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:47:56.221 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:56.220+0000 7f7e16fe6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:47:56.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.300+0000 7f5339052700 1 -- 172.21.15.189:0/1960542484 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f531806af60 msgr2=0x7f531806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:56.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.300+0000 7f5339052700 1 --2- 172.21.15.189:0/1960542484 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f531806af60 0x7f531806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:56.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.306+0000 7f533b2b6700 1 -- 172.21.15.189:0/1960542484 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f533415ace0 con 0x7f533415d590 2024-04-06T13:47:56.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.306+0000 7f5329ffb700 1 -- 172.21.15.189:0/1960542484 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v232) v1 ==== 93+0+10 (secure 0 0 0) 0x7f533415ace0 con 0x7f533415d590 2024-04-06T13:47:56.307 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:47:56.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.309+0000 7f533b2b6700 1 -- 172.21.15.189:0/1960542484 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f531806af60 msgr2=0x7f531806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:56.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.309+0000 7f533b2b6700 1 --2- 172.21.15.189:0/1960542484 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f531806af60 0x7f531806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.309+0000 7f533b2b6700 1 -- 172.21.15.189:0/1960542484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f533415d590 msgr2=0x7f533414f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:56.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.309+0000 7f533b2b6700 1 --2- 172.21.15.189:0/1960542484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f533415d590 0x7f533414f990 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f532c00efc0 tx=0x7f532c00c3a0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.309+0000 7f533b2b6700 1 -- 172.21.15.189:0/1960542484 shutdown_connections 2024-04-06T13:47:56.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.309+0000 7f533b2b6700 1 --2- 172.21.15.189:0/1960542484 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f531806af60 0x7f531806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.309+0000 7f533b2b6700 1 --2- 172.21.15.189:0/1960542484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f533415d590 0x7f533414f990 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.309+0000 7f533b2b6700 1 --2- 172.21.15.189:0/1960542484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53340b4ed0 0x7f53340cc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.309+0000 7f533b2b6700 1 --2- 172.21.15.189:0/1960542484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f53340d3890 0x7f53340cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.309+0000 7f533b2b6700 1 -- 172.21.15.189:0/1960542484 >> 172.21.15.189:0/1960542484 conn(0x7f533406bbc0 msgr2=0x7f53340b6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:56.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.309+0000 7f533b2b6700 1 -- 172.21.15.189:0/1960542484 shutdown_connections 2024-04-06T13:47:56.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.309+0000 7f533b2b6700 1 -- 172.21.15.189:0/1960542484 wait complete. 2024-04-06T13:47:56.322 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T13:47:56.387 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:56.387+0000 7f7e16fe6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:47:56.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.450+0000 7f89fb4a5700 1 Processor -- start 2024-04-06T13:47:56.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.450+0000 7f89fb4a5700 1 -- start start 2024-04-06T13:47:56.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.451+0000 7f89fb4a5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89f40d0a60 0x7f89f40d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:56.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.451+0000 7f89fb4a5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f40cdf80 con 0x7f89f415d7e0 2024-04-06T13:47:56.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.451+0000 7f89fb4a5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f40d3230 con 0x7f89f40d1af0 2024-04-06T13:47:56.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.451+0000 7f89fb4a5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f40d1f80 con 0x7f89f40d0a60 2024-04-06T13:47:56.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.451+0000 7f89f9241700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f89f415d7e0 0x7f89f40d0350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58324/0 (socket says 172.21.15.189:58324) 2024-04-06T13:47:56.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.451+0000 7f89f9241700 1 -- 172.21.15.189:0/3081856685 learned_addr learned my addr 172.21.15.189:0/3081856685 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:56.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.451+0000 7f89f8a40700 1 --2- 172.21.15.189:0/3081856685 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89f40d0a60 0x7f89f40d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:56.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.452+0000 7f89ebfff700 1 -- 172.21.15.189:0/3081856685 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147817327 0 0) 0x7f89f40cdf80 con 0x7f89f415d7e0 2024-04-06T13:47:56.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.452+0000 7f89ebfff700 1 -- 172.21.15.189:0/3081856685 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e00035b0 con 0x7f89f415d7e0 2024-04-06T13:47:56.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.452+0000 7f89ebfff700 1 -- 172.21.15.189:0/3081856685 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398930974 0 0) 0x7f89ec001490 con 0x7f89f40d1af0 2024-04-06T13:47:56.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.452+0000 7f89ebfff700 1 -- 172.21.15.189:0/3081856685 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89f40cdf80 con 0x7f89f40d1af0 2024-04-06T13:47:56.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.452+0000 7f89f8a40700 1 -- 172.21.15.189:0/3081856685 >> v1:172.21.15.192:6789/0 conn(0x7f89f40d1af0 legacy=0x7f89f40b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:56.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.452+0000 7f89f8a40700 1 -- 172.21.15.189:0/3081856685 >> v1:172.21.15.189:6789/0 conn(0x7f89f415d7e0 legacy=0x7f89f40d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:56.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.452+0000 7f89f8a40700 1 -- 172.21.15.189:0/3081856685 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f40d2da0 con 0x7f89f40d0a60 2024-04-06T13:47:56.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.452+0000 7f89f8a40700 1 --2- 172.21.15.189:0/3081856685 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89f40d0a60 0x7f89f40d15b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009870 tx=0x7f89e4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6572fe042b3b2976 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:56.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.452+0000 7f89ebfff700 1 -- 172.21.15.189:0/3081856685 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e4011830 con 0x7f89f40d0a60 2024-04-06T13:47:56.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.452+0000 7f89ebfff700 1 -- 172.21.15.189:0/3081856685 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e4011990 con 0x7f89f40d0a60 2024-04-06T13:47:56.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.452+0000 7f89ebfff700 1 -- 172.21.15.189:0/3081856685 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e4011d20 con 0x7f89f40d0a60 2024-04-06T13:47:56.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.453+0000 7f89fb4a5700 1 -- 172.21.15.189:0/3081856685 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89f40d0a60 msgr2=0x7f89f40d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:56.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.453+0000 7f89fb4a5700 1 --2- 172.21.15.189:0/3081856685 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89f40d0a60 0x7f89f40d15b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009870 tx=0x7f89e4009b80 comp rx=0 tx=0).stop 2024-04-06T13:47:56.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.453+0000 7f89fb4a5700 1 -- 172.21.15.189:0/3081856685 shutdown_connections 2024-04-06T13:47:56.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.453+0000 7f89fb4a5700 1 --2- 172.21.15.189:0/3081856685 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89f40d0a60 0x7f89f40d15b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.453+0000 7f89fb4a5700 1 -- 172.21.15.189:0/3081856685 >> 172.21.15.189:0/3081856685 conn(0x7f89f4072250 msgr2=0x7f89f4072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:56.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.453+0000 7f89fb4a5700 1 -- 172.21.15.189:0/3081856685 shutdown_connections 2024-04-06T13:47:56.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.453+0000 7f89fb4a5700 1 -- 172.21.15.189:0/3081856685 wait complete. 2024-04-06T13:47:56.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.454+0000 7f89fb4a5700 1 Processor -- start 2024-04-06T13:47:56.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.454+0000 7f89fb4a5700 1 -- start start 2024-04-06T13:47:56.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.455+0000 7f89fb4a5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89f40d0a60 0x7f89f40ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:56.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.455+0000 7f89fb4a5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89f40d1af0 0x7f89f4152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:56.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.455+0000 7f89fb4a5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89f415d7e0 0x7f89f40b4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:56.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.455+0000 7f89fb4a5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f4152ea0 con 0x7f89f40d0a60 2024-04-06T13:47:56.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.455+0000 7f89fb4a5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f40b4a60 con 0x7f89f40d1af0 2024-04-06T13:47:56.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.455+0000 7f89fb4a5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f40b3c90 con 0x7f89f415d7e0 2024-04-06T13:47:56.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.455+0000 7f89f9241700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89f40d0a60 0x7f89f40ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.455+0000 7f89f9241700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89f40d0a60 0x7f89f40ca4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38322/0 (socket says 172.21.15.189:38322) 2024-04-06T13:47:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.455+0000 7f89f9241700 1 -- 172.21.15.189:0/2349910166 learned_addr learned my addr 172.21.15.189:0/2349910166 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.455+0000 7f89f9a42700 1 --2- 172.21.15.189:0/2349910166 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89f415d7e0 0x7f89f40b4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.455+0000 7f89f8a40700 1 --2- 172.21.15.189:0/2349910166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89f40d1af0 0x7f89f4152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.455+0000 7f89f9241700 1 -- 172.21.15.189:0/2349910166 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89f415d7e0 msgr2=0x7f89f40b4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.455+0000 7f89f9241700 1 --2- 172.21.15.189:0/2349910166 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89f415d7e0 0x7f89f40b4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.455+0000 7f89f9241700 1 -- 172.21.15.189:0/2349910166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89f40d1af0 msgr2=0x7f89f4152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.455+0000 7f89f9241700 1 --2- 172.21.15.189:0/2349910166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89f40d1af0 0x7f89f4152960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.455+0000 7f89f9241700 1 -- 172.21.15.189:0/2349910166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e40094a0 con 0x7f89f40d0a60 2024-04-06T13:47:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.456+0000 7f89f9241700 1 --2- 172.21.15.189:0/2349910166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89f40d0a60 0x7f89f40ca4a0 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f89f0001eb0 tx=0x7f89f000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.456+0000 7f89e9ffb700 1 -- 172.21.15.189:0/2349910166 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f0005d70 con 0x7f89f40d0a60 2024-04-06T13:47:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.456+0000 7f89e9ffb700 1 -- 172.21.15.189:0/2349910166 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f000ce50 con 0x7f89f40d0a60 2024-04-06T13:47:56.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.456+0000 7f89fb4a5700 1 -- 172.21.15.189:0/2349910166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f40b3f70 con 0x7f89f40d0a60 2024-04-06T13:47:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.456+0000 7f89e9ffb700 1 -- 172.21.15.189:0/2349910166 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f00072f0 con 0x7f89f40d0a60 2024-04-06T13:47:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.456+0000 7f89fb4a5700 1 -- 172.21.15.189:0/2349910166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f40c8310 con 0x7f89f40d0a60 2024-04-06T13:47:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.457+0000 7f89fb4a5700 1 -- 172.21.15.189:0/2349910166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f4074990 con 0x7f89f40d0a60 2024-04-06T13:47:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.458+0000 7f89e9ffb700 1 -- 172.21.15.189:0/2349910166 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 146) v1 ==== 89406+0+0 (secure 0 0 0) 0x7f89f0007450 con 0x7f89f40d0a60 2024-04-06T13:47:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.458+0000 7f89e9ffb700 1 --2- 172.21.15.189:0/2349910166 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f89d806af60 0x7f89d806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.458+0000 7f89f8a40700 1 -- 172.21.15.189:0/2349910166 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f89d806af60 msgr2=0x7f89d806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.458+0000 7f89e9ffb700 1 -- 172.21.15.189:0/2349910166 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7f89f0089080 con 0x7f89f40d0a60 2024-04-06T13:47:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.458+0000 7f89f8a40700 1 --2- 172.21.15.189:0/2349910166 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f89d806af60 0x7f89d806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:56.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.463+0000 7f89e9ffb700 1 -- 172.21.15.189:0/2349910166 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f89f0053960 con 0x7f89f40d0a60 2024-04-06T13:47:56.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.659+0000 7f89f8a40700 1 -- 172.21.15.189:0/2349910166 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f89d806af60 msgr2=0x7f89d806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:56.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.659+0000 7f89f8a40700 1 --2- 172.21.15.189:0/2349910166 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f89d806af60 0x7f89d806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:56.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.663+0000 7f89fb4a5700 1 -- 172.21.15.189:0/2349910166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f89f40b5ff0 con 0x7f89f40d0a60 2024-04-06T13:47:56.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.663+0000 7f89e9ffb700 1 -- 172.21.15.189:0/2349910166 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v232) v1 ==== 104+0+11 (secure 0 0 0) 0x7f89f0056f80 con 0x7f89f40d0a60 2024-04-06T13:47:56.664 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:47:56.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.665+0000 7f89fb4a5700 1 -- 172.21.15.189:0/2349910166 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f89d806af60 msgr2=0x7f89d806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:56.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.665+0000 7f89fb4a5700 1 --2- 172.21.15.189:0/2349910166 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f89d806af60 0x7f89d806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.665+0000 7f89fb4a5700 1 -- 172.21.15.189:0/2349910166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89f40d0a60 msgr2=0x7f89f40ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:56.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.665+0000 7f89fb4a5700 1 --2- 172.21.15.189:0/2349910166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89f40d0a60 0x7f89f40ca4a0 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f89f0001eb0 tx=0x7f89f000c7b0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.665+0000 7f89fb4a5700 1 -- 172.21.15.189:0/2349910166 shutdown_connections 2024-04-06T13:47:56.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.665+0000 7f89fb4a5700 1 --2- 172.21.15.189:0/2349910166 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f89d806af60 0x7f89d806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.666+0000 7f89fb4a5700 1 --2- 172.21.15.189:0/2349910166 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89f415d7e0 0x7f89f40b4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.666+0000 7f89fb4a5700 1 --2- 172.21.15.189:0/2349910166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89f40d0a60 0x7f89f40ca4a0 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.666+0000 7f89fb4a5700 1 --2- 172.21.15.189:0/2349910166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89f40d1af0 0x7f89f4152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.666+0000 7f89fb4a5700 1 -- 172.21.15.189:0/2349910166 >> 172.21.15.189:0/2349910166 conn(0x7f89f4072250 msgr2=0x7f89f415ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:56.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.666+0000 7f89fb4a5700 1 -- 172.21.15.189:0/2349910166 shutdown_connections 2024-04-06T13:47:56.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.666+0000 7f89fb4a5700 1 -- 172.21.15.189:0/2349910166 wait complete. 2024-04-06T13:47:56.679 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T13:47:56.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.805+0000 7f77fc2bf700 1 Processor -- start 2024-04-06T13:47:56.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.805+0000 7f77fc2bf700 1 -- start start 2024-04-06T13:47:56.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.806+0000 7f77fc2bf700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f77f40cb870 0x7f77f415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:56.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.806+0000 7f77fc2bf700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f77f40cfa00 0x7f77f40cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:56.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.806+0000 7f77fc2bf700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f40cbd80 con 0x7f77f40d11b0 2024-04-06T13:47:56.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.806+0000 7f77fc2bf700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77f40ccdc0 con 0x7f77f40cfa00 2024-04-06T13:47:56.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.806+0000 7f77fc2bf700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77f40d0b50 con 0x7f77f40cb870 2024-04-06T13:47:56.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.807+0000 7f77fa85c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f77f40cfa00 0x7f77f40cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:56.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.807+0000 7f77fa85c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f77f40cfa00 0x7f77f40cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38334/0 (socket says 172.21.15.189:38334) 2024-04-06T13:47:56.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.807+0000 7f77fa85c700 1 -- 172.21.15.189:0/3159889803 learned_addr learned my addr 172.21.15.189:0/3159889803 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:56.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.807+0000 7f77f985a700 1 --2- 172.21.15.189:0/3159889803 >> v2:172.21.15.192:3300/0 conn(0x7f77f40cb870 0x7f77f415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:56.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.807+0000 7f77fa85c700 1 -- 172.21.15.189:0/3159889803 >> v2:172.21.15.192:3300/0 conn(0x7f77f40cb870 msgr2=0x7f77f415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:56.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.807+0000 7f77fa85c700 1 --2- 172.21.15.189:0/3159889803 >> v2:172.21.15.192:3300/0 conn(0x7f77f40cb870 0x7f77f415c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.807+0000 7f77fa85c700 1 -- 172.21.15.189:0/3159889803 >> v1:172.21.15.192:6789/0 conn(0x7f77f40d11b0 legacy=0x7f77f40d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:56.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.807+0000 7f77fa85c700 1 -- 172.21.15.189:0/3159889803 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f40cef20 con 0x7f77f40cfa00 2024-04-06T13:47:56.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.807+0000 7f77f985a700 1 --2- 172.21.15.189:0/3159889803 >> v2:172.21.15.192:3300/0 conn(0x7f77f40cb870 0x7f77f415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:56.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.807+0000 7f77fa85c700 1 --2- 172.21.15.189:0/3159889803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77f40cfa00 0x7f77f40cd560 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f77f404fb40 tx=0x7f77ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7494cbc5574ab76b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:56.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.807+0000 7f77f9059700 1 -- 172.21.15.189:0/3159889803 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ec00e070 con 0x7f77f40cfa00 2024-04-06T13:47:56.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.807+0000 7f77f9059700 1 -- 172.21.15.189:0/3159889803 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77ec00b9c0 con 0x7f77f40cfa00 2024-04-06T13:47:56.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.807+0000 7f77f9059700 1 -- 172.21.15.189:0/3159889803 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ec00bd70 con 0x7f77f40cfa00 2024-04-06T13:47:56.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.808+0000 7f77fc2bf700 1 -- 172.21.15.189:0/3159889803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77f40cfa00 msgr2=0x7f77f40cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:56.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.808+0000 7f77fc2bf700 1 --2- 172.21.15.189:0/3159889803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77f40cfa00 0x7f77f40cd560 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f77f404fb40 tx=0x7f77ec00b330 comp rx=0 tx=0).stop 2024-04-06T13:47:56.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.808+0000 7f77fc2bf700 1 -- 172.21.15.189:0/3159889803 shutdown_connections 2024-04-06T13:47:56.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.808+0000 7f77fc2bf700 1 --2- 172.21.15.189:0/3159889803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77f40cfa00 0x7f77f40cd560 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.808+0000 7f77fc2bf700 1 --2- 172.21.15.189:0/3159889803 >> v2:172.21.15.192:3300/0 conn(0x7f77f40cb870 0x7f77f415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.808+0000 7f77fc2bf700 1 -- 172.21.15.189:0/3159889803 >> 172.21.15.189:0/3159889803 conn(0x7f77f40b2db0 msgr2=0x7f77f40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:56.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.808+0000 7f77fc2bf700 1 -- 172.21.15.189:0/3159889803 shutdown_connections 2024-04-06T13:47:56.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.808+0000 7f77fc2bf700 1 -- 172.21.15.189:0/3159889803 wait complete. 2024-04-06T13:47:56.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.809+0000 7f77fc2bf700 1 Processor -- start 2024-04-06T13:47:56.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.809+0000 7f77fc2bf700 1 -- start start 2024-04-06T13:47:56.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.810+0000 7f77fc2bf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77f40cb870 0x7f77f4158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:56.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.810+0000 7f77fc2bf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77f40cfa00 0x7f77f4151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:56.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.810+0000 7f77fc2bf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77f40d11b0 0x7f77f40b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:56.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.810+0000 7f77fc2bf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f41580e0 con 0x7f77f40cfa00 2024-04-06T13:47:56.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.810+0000 7f77fc2bf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f40c8d00 con 0x7f77f40d11b0 2024-04-06T13:47:56.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.810+0000 7f77fc2bf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77f4158ce0 con 0x7f77f40cb870 2024-04-06T13:47:56.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.810+0000 7f77f985a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77f40cfa00 0x7f77f4151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:56.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.810+0000 7f77f985a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77f40cfa00 0x7f77f4151850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38342/0 (socket says 172.21.15.189:38342) 2024-04-06T13:47:56.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.810+0000 7f77f985a700 1 -- 172.21.15.189:0/3499005076 learned_addr learned my addr 172.21.15.189:0/3499005076 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:56.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.810+0000 7f77fa85c700 1 --2- 172.21.15.189:0/3499005076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77f40d11b0 0x7f77f40b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:56.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.810+0000 7f77fa05b700 1 --2- 172.21.15.189:0/3499005076 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77f40cb870 0x7f77f4158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:56.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.810+0000 7f77f985a700 1 -- 172.21.15.189:0/3499005076 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77f40cb870 msgr2=0x7f77f4158600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:56.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.810+0000 7f77f985a700 1 --2- 172.21.15.189:0/3499005076 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77f40cb870 0x7f77f4158600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.810+0000 7f77f985a700 1 -- 172.21.15.189:0/3499005076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77f40d11b0 msgr2=0x7f77f40b6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:56.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.810+0000 7f77f985a700 1 --2- 172.21.15.189:0/3499005076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77f40d11b0 0x7f77f40b6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:56.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.810+0000 7f77f985a700 1 -- 172.21.15.189:0/3499005076 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ec009b30 con 0x7f77f40cfa00 2024-04-06T13:47:56.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.810+0000 7f77f985a700 1 --2- 172.21.15.189:0/3499005076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77f40cfa00 0x7f77f4151850 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7f77e400e810 tx=0x7f77e400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:56.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.810+0000 7f77eaffd700 1 -- 172.21.15.189:0/3499005076 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e400c030 con 0x7f77f40cfa00 2024-04-06T13:47:56.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.810+0000 7f77eaffd700 1 -- 172.21.15.189:0/3499005076 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e4004220 con 0x7f77f40cfa00 2024-04-06T13:47:56.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.811+0000 7f77fc2bf700 1 -- 172.21.15.189:0/3499005076 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f40c8f50 con 0x7f77f40cfa00 2024-04-06T13:47:56.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.811+0000 7f77eaffd700 1 -- 172.21.15.189:0/3499005076 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e4004590 con 0x7f77f40cfa00 2024-04-06T13:47:56.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.811+0000 7f77fc2bf700 1 -- 172.21.15.189:0/3499005076 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f40c9510 con 0x7f77f40cfa00 2024-04-06T13:47:56.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.812+0000 7f77fc2bf700 1 -- 172.21.15.189:0/3499005076 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f4074990 con 0x7f77f40cfa00 2024-04-06T13:47:56.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.812+0000 7f77eaffd700 1 -- 172.21.15.189:0/3499005076 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 146) v1 ==== 89406+0+0 (secure 0 0 0) 0x7f77e40036a0 con 0x7f77f40cfa00 2024-04-06T13:47:56.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.813+0000 7f77eaffd700 1 --2- 172.21.15.189:0/3499005076 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f77d806af60 0x7f77d806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:56.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.813+0000 7f77fa05b700 1 -- 172.21.15.189:0/3499005076 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f77d806af60 msgr2=0x7f77d806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:56.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.813+0000 7f77eaffd700 1 -- 172.21.15.189:0/3499005076 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7f77e4089ab0 con 0x7f77f40cfa00 2024-04-06T13:47:56.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.813+0000 7f77fa05b700 1 --2- 172.21.15.189:0/3499005076 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f77d806af60 0x7f77d806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:56.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:56.817+0000 7f77eaffd700 1 -- 172.21.15.189:0/3499005076 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f77e4054280 con 0x7f77f40cfa00 2024-04-06T13:47:56.906 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:56.906+0000 7f1c6056b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:47:57.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.014+0000 7f77fa05b700 1 -- 172.21.15.189:0/3499005076 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f77d806af60 msgr2=0x7f77d806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:57.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.014+0000 7f77fa05b700 1 --2- 172.21.15.189:0/3499005076 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f77d806af60 0x7f77d806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:57.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.020+0000 7f77fc2bf700 1 -- 172.21.15.189:0/3499005076 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f77f40b0a00 con 0x7f77f40cfa00 2024-04-06T13:47:57.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.021+0000 7f77eaffd700 1 -- 172.21.15.189:0/3499005076 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v232) v1 ==== 100+0+11 (secure 0 0 0) 0x7f77e40578a0 con 0x7f77f40cfa00 2024-04-06T13:47:57.021 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:47:57.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.023+0000 7f77fc2bf700 1 -- 172.21.15.189:0/3499005076 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f77d806af60 msgr2=0x7f77d806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:57.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.023+0000 7f77fc2bf700 1 --2- 172.21.15.189:0/3499005076 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f77d806af60 0x7f77d806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.023+0000 7f77fc2bf700 1 -- 172.21.15.189:0/3499005076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77f40cfa00 msgr2=0x7f77f4151850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:57.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.023+0000 7f77fc2bf700 1 --2- 172.21.15.189:0/3499005076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77f40cfa00 0x7f77f4151850 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7f77e400e810 tx=0x7f77e400eb20 comp rx=0 tx=0).stop 2024-04-06T13:47:57.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.023+0000 7f77fc2bf700 1 -- 172.21.15.189:0/3499005076 shutdown_connections 2024-04-06T13:47:57.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.023+0000 7f77fc2bf700 1 --2- 172.21.15.189:0/3499005076 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f77d806af60 0x7f77d806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.023+0000 7f77fc2bf700 1 --2- 172.21.15.189:0/3499005076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77f40d11b0 0x7f77f40b6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.023+0000 7f77fc2bf700 1 --2- 172.21.15.189:0/3499005076 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77f40cb870 0x7f77f4158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.023+0000 7f77fc2bf700 1 --2- 172.21.15.189:0/3499005076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77f40cfa00 0x7f77f4151850 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.023+0000 7f77fc2bf700 1 -- 172.21.15.189:0/3499005076 >> 172.21.15.189:0/3499005076 conn(0x7f77f40b2db0 msgr2=0x7f77f406f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:57.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.023+0000 7f77fc2bf700 1 -- 172.21.15.189:0/3499005076 shutdown_connections 2024-04-06T13:47:57.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.023+0000 7f77fc2bf700 1 -- 172.21.15.189:0/3499005076 wait complete. 2024-04-06T13:47:57.037 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-04-06T13:47:57.150 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:57.150+0000 7f1c6056b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:47:57.157 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.157+0000 7f7e16fe6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:47:57.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.166+0000 7f2c4c4e5700 1 Processor -- start 2024-04-06T13:47:57.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.166+0000 7f2c4c4e5700 1 -- start start 2024-04-06T13:47:57.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.167+0000 7f2c4c4e5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c4415d7e0 0x7f2c440d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:57.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.167+0000 7f2c4c4e5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2c440d0890 0x7f2c440d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:57.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.167+0000 7f2c4c4e5700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2c440d1af0 0x7f2c440b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:57.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.167+0000 7f2c4c4e5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c440d0d10 con 0x7f2c440d0890 2024-04-06T13:47:57.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.167+0000 7f2c4c4e5700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c440d1f80 con 0x7f2c440d1af0 2024-04-06T13:47:57.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.167+0000 7f2c4c4e5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c440cdf80 con 0x7f2c4415d7e0 2024-04-06T13:47:57.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.167+0000 7f2c49a80700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2c440d0890 0x7f2c440d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:57.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.167+0000 7f2c49a80700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2c440d0890 0x7f2c440d15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38346/0 (socket says 172.21.15.189:38346) 2024-04-06T13:47:57.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.167+0000 7f2c49a80700 1 -- 172.21.15.189:0/4087742683 learned_addr learned my addr 172.21.15.189:0/4087742683 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:57.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.167+0000 7f2c4a281700 1 --2- 172.21.15.189:0/4087742683 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c4415d7e0 0x7f2c440d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:57.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.167+0000 7f2c4aa82700 1 --2- 172.21.15.189:0/4087742683 >> v2:172.21.15.192:3300/0 conn(0x7f2c440d1af0 0x7f2c440b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:57.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.167+0000 7f2c49a80700 1 -- 172.21.15.189:0/4087742683 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c4415d7e0 msgr2=0x7f2c440d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:57.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.168+0000 7f2c49a80700 1 --2- 172.21.15.189:0/4087742683 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c4415d7e0 0x7f2c440d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.168+0000 7f2c49a80700 1 -- 172.21.15.189:0/4087742683 >> v2:172.21.15.192:3300/0 conn(0x7f2c440d1af0 msgr2=0x7f2c440b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:57.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.168+0000 7f2c49a80700 1 --2- 172.21.15.189:0/4087742683 >> v2:172.21.15.192:3300/0 conn(0x7f2c440d1af0 0x7f2c440b4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.168+0000 7f2c49a80700 1 -- 172.21.15.189:0/4087742683 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c440d3230 con 0x7f2c440d0890 2024-04-06T13:47:57.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.168+0000 7f2c49a80700 1 --2- 172.21.15.189:0/4087742683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c440d0890 0x7f2c440d15b0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f2c4408c730 tx=0x7f2c34009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c38f4247d6571840 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:57.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.168+0000 7f2c4927f700 1 -- 172.21.15.189:0/4087742683 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c34011830 con 0x7f2c440d0890 2024-04-06T13:47:57.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.168+0000 7f2c4927f700 1 -- 172.21.15.189:0/4087742683 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c34011990 con 0x7f2c440d0890 2024-04-06T13:47:57.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.168+0000 7f2c4927f700 1 -- 172.21.15.189:0/4087742683 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c34011d20 con 0x7f2c440d0890 2024-04-06T13:47:57.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.168+0000 7f2c4c4e5700 1 -- 172.21.15.189:0/4087742683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c440d0890 msgr2=0x7f2c440d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:57.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.168+0000 7f2c4c4e5700 1 --2- 172.21.15.189:0/4087742683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c440d0890 0x7f2c440d15b0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f2c4408c730 tx=0x7f2c34009bd0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.169+0000 7f2c4c4e5700 1 -- 172.21.15.189:0/4087742683 shutdown_connections 2024-04-06T13:47:57.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.169+0000 7f2c4c4e5700 1 --2- 172.21.15.189:0/4087742683 >> v2:172.21.15.192:3300/0 conn(0x7f2c440d1af0 0x7f2c440b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.169+0000 7f2c4c4e5700 1 --2- 172.21.15.189:0/4087742683 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c4415d7e0 0x7f2c440d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.169+0000 7f2c4c4e5700 1 --2- 172.21.15.189:0/4087742683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c440d0890 0x7f2c440d15b0 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.169+0000 7f2c4c4e5700 1 -- 172.21.15.189:0/4087742683 >> 172.21.15.189:0/4087742683 conn(0x7f2c44072250 msgr2=0x7f2c44072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:57.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.169+0000 7f2c4c4e5700 1 -- 172.21.15.189:0/4087742683 shutdown_connections 2024-04-06T13:47:57.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.169+0000 7f2c4c4e5700 1 -- 172.21.15.189:0/4087742683 wait complete. 2024-04-06T13:47:57.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.170+0000 7f2c4c4e5700 1 Processor -- start 2024-04-06T13:47:57.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.170+0000 7f2c4c4e5700 1 -- start start 2024-04-06T13:47:57.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c4c4e5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c440d0890 0x7f2c441519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:57.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c4c4e5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c440d1af0 0x7f2c440ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:57.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c4c4e5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c4415d7e0 0x7f2c440b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:57.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c4c4e5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c440cbdf0 con 0x7f2c440d1af0 2024-04-06T13:47:57.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c4c4e5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c440cb050 con 0x7f2c4415d7e0 2024-04-06T13:47:57.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c4c4e5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c440cb330 con 0x7f2c440d0890 2024-04-06T13:47:57.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c49a80700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c440d1af0 0x7f2c440ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:57.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c49a80700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c440d1af0 0x7f2c440ccde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38360/0 (socket says 172.21.15.189:38360) 2024-04-06T13:47:57.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c49a80700 1 -- 172.21.15.189:0/3708588369 learned_addr learned my addr 172.21.15.189:0/3708588369 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:57.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c4a281700 1 --2- 172.21.15.189:0/3708588369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c440d0890 0x7f2c441519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:57.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c4aa82700 1 --2- 172.21.15.189:0/3708588369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c4415d7e0 0x7f2c440b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:57.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c49a80700 1 -- 172.21.15.189:0/3708588369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c440d0890 msgr2=0x7f2c441519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:57.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c49a80700 1 --2- 172.21.15.189:0/3708588369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c440d0890 0x7f2c441519a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c49a80700 1 -- 172.21.15.189:0/3708588369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c4415d7e0 msgr2=0x7f2c440b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:57.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c49a80700 1 --2- 172.21.15.189:0/3708588369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c4415d7e0 0x7f2c440b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c49a80700 1 -- 172.21.15.189:0/3708588369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c34009550 con 0x7f2c440d1af0 2024-04-06T13:47:57.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c49a80700 1 --2- 172.21.15.189:0/3708588369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c440d1af0 0x7f2c440ccde0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f2c34009c10 tx=0x7f2c34004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:57.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c3affd700 1 -- 172.21.15.189:0/3708588369 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c34004450 con 0x7f2c440d1af0 2024-04-06T13:47:57.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c3affd700 1 -- 172.21.15.189:0/3708588369 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3401e070 con 0x7f2c440d1af0 2024-04-06T13:47:57.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c4c4e5700 1 -- 172.21.15.189:0/3708588369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c440cb610 con 0x7f2c440d1af0 2024-04-06T13:47:57.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c3affd700 1 -- 172.21.15.189:0/3708588369 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c340038e0 con 0x7f2c440d1af0 2024-04-06T13:47:57.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.171+0000 7f2c4c4e5700 1 -- 172.21.15.189:0/3708588369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c440b4440 con 0x7f2c440d1af0 2024-04-06T13:47:57.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.172+0000 7f2c4c4e5700 1 -- 172.21.15.189:0/3708588369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c440cbc60 con 0x7f2c440d1af0 2024-04-06T13:47:57.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.173+0000 7f2c3affd700 1 -- 172.21.15.189:0/3708588369 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 146) v1 ==== 89406+0+0 (secure 0 0 0) 0x7f2c34003a40 con 0x7f2c440d1af0 2024-04-06T13:47:57.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.173+0000 7f2c3affd700 1 --2- 172.21.15.189:0/3708588369 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2c2806af60 0x7f2c2806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:57.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.173+0000 7f2c4a281700 1 -- 172.21.15.189:0/3708588369 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2c2806af60 msgr2=0x7f2c2806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:57.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.173+0000 7f2c3affd700 1 -- 172.21.15.189:0/3708588369 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7f2c3408a8a0 con 0x7f2c440d1af0 2024-04-06T13:47:57.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.173+0000 7f2c4a281700 1 --2- 172.21.15.189:0/3708588369 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2c2806af60 0x7f2c2806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:57.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.178+0000 7f2c3affd700 1 -- 172.21.15.189:0/3708588369 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2c34055180 con 0x7f2c440d1af0 2024-04-06T13:47:57.228 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:57.228+0000 7f1c6056b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:47:57.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.374+0000 7f2c4a281700 1 -- 172.21.15.189:0/3708588369 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2c2806af60 msgr2=0x7f2c2806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:57.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.374+0000 7f2c4a281700 1 --2- 172.21.15.189:0/3708588369 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2c2806af60 0x7f2c2806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:57.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.383+0000 7f2c4c4e5700 1 -- 172.21.15.189:0/3708588369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2c440d3230 con 0x7f2c440d1af0 2024-04-06T13:47:57.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.384+0000 7f2c3affd700 1 -- 172.21.15.189:0/3708588369 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v232) v1 ==== 107+0+11 (secure 0 0 0) 0x7f2c440d3230 con 0x7f2c440d1af0 2024-04-06T13:47:57.385 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:47:57.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.386+0000 7f2c4c4e5700 1 -- 172.21.15.189:0/3708588369 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2c2806af60 msgr2=0x7f2c2806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:57.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.386+0000 7f2c4c4e5700 1 --2- 172.21.15.189:0/3708588369 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2c2806af60 0x7f2c2806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.386+0000 7f2c4c4e5700 1 -- 172.21.15.189:0/3708588369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c440d1af0 msgr2=0x7f2c440ccde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:57.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.386+0000 7f2c4c4e5700 1 --2- 172.21.15.189:0/3708588369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c440d1af0 0x7f2c440ccde0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f2c34009c10 tx=0x7f2c34004750 comp rx=0 tx=0).stop 2024-04-06T13:47:57.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.386+0000 7f2c4c4e5700 1 -- 172.21.15.189:0/3708588369 shutdown_connections 2024-04-06T13:47:57.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.386+0000 7f2c4c4e5700 1 --2- 172.21.15.189:0/3708588369 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2c2806af60 0x7f2c2806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.386+0000 7f2c4c4e5700 1 --2- 172.21.15.189:0/3708588369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c4415d7e0 0x7f2c440b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.386+0000 7f2c4c4e5700 1 --2- 172.21.15.189:0/3708588369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c440d0890 0x7f2c441519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.386+0000 7f2c4c4e5700 1 --2- 172.21.15.189:0/3708588369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c440d1af0 0x7f2c440ccde0 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.386+0000 7f2c4c4e5700 1 -- 172.21.15.189:0/3708588369 >> 172.21.15.189:0/3708588369 conn(0x7f2c44072250 msgr2=0x7f2c440b8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:57.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.386+0000 7f2c4c4e5700 1 -- 172.21.15.189:0/3708588369 shutdown_connections 2024-04-06T13:47:57.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.386+0000 7f2c4c4e5700 1 -- 172.21.15.189:0/3708588369 wait complete. 2024-04-06T13:47:57.400 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-04-06T13:47:57.416 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.416+0000 7f7e16fe6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:47:57.499 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.498+0000 7f7e16fe6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:47:57.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.526+0000 7fa9ef940700 1 Processor -- start 2024-04-06T13:47:57.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.526+0000 7fa9ef940700 1 -- start start 2024-04-06T13:47:57.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.527+0000 7fa9ef940700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9e8146dd0 0x7fa9e806d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:57.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.527+0000 7fa9ef940700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e806c0a0 con 0x7fa9e80d17a0 2024-04-06T13:47:57.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.527+0000 7fa9ef940700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e806c380 con 0x7fa9e80cedf0 2024-04-06T13:47:57.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.527+0000 7fa9ef940700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e806beb0 con 0x7fa9e8146dd0 2024-04-06T13:47:57.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.527+0000 7fa9ed6dc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa9e80d17a0 0x7fa9e806b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58332/0 (socket says 172.21.15.189:58332) 2024-04-06T13:47:57.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.527+0000 7fa9ed6dc700 1 -- 172.21.15.189:0/2168157736 learned_addr learned my addr 172.21.15.189:0/2168157736 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:57.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.527+0000 7fa9ecedb700 1 --2- 172.21.15.189:0/2168157736 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9e8146dd0 0x7fa9e806d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:57.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.528+0000 7fa9dffff700 1 -- 172.21.15.189:0/2168157736 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866812251 0 0) 0x7fa9e806c0a0 con 0x7fa9e80d17a0 2024-04-06T13:47:57.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.528+0000 7fa9dffff700 1 -- 172.21.15.189:0/2168157736 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d40035b0 con 0x7fa9e80d17a0 2024-04-06T13:47:57.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.528+0000 7fa9ecedb700 1 -- 172.21.15.189:0/2168157736 >> v1:172.21.15.192:6789/0 conn(0x7fa9e80cedf0 legacy=0x7fa9e81441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:57.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.528+0000 7fa9dffff700 1 -- 172.21.15.189:0/2168157736 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860007512 0 0) 0x7fa9e806c380 con 0x7fa9e80cedf0 2024-04-06T13:47:57.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.528+0000 7fa9ecedb700 1 -- 172.21.15.189:0/2168157736 >> v1:172.21.15.189:6789/0 conn(0x7fa9e80d17a0 legacy=0x7fa9e806b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:57.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.528+0000 7fa9ecedb700 1 -- 172.21.15.189:0/2168157736 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e806c660 con 0x7fa9e8146dd0 2024-04-06T13:47:57.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.528+0000 7fa9ecedb700 1 --2- 172.21.15.189:0/2168157736 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9e8146dd0 0x7fa9e806d140 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8009870 tx=0x7fa9d8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7985fe6731ec4551 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:57.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.528+0000 7fa9dffff700 1 -- 172.21.15.189:0/2168157736 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8011830 con 0x7fa9e8146dd0 2024-04-06T13:47:57.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.528+0000 7fa9dffff700 1 -- 172.21.15.189:0/2168157736 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d8011990 con 0x7fa9e8146dd0 2024-04-06T13:47:57.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.528+0000 7fa9dffff700 1 -- 172.21.15.189:0/2168157736 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8011d20 con 0x7fa9e8146dd0 2024-04-06T13:47:57.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.529+0000 7fa9ef940700 1 -- 172.21.15.189:0/2168157736 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9e8146dd0 msgr2=0x7fa9e806d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:57.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.529+0000 7fa9ef940700 1 --2- 172.21.15.189:0/2168157736 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9e8146dd0 0x7fa9e806d140 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8009870 tx=0x7fa9d8009b80 comp rx=0 tx=0).stop 2024-04-06T13:47:57.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.529+0000 7fa9ef940700 1 -- 172.21.15.189:0/2168157736 shutdown_connections 2024-04-06T13:47:57.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.529+0000 7fa9ef940700 1 --2- 172.21.15.189:0/2168157736 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9e8146dd0 0x7fa9e806d140 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.529+0000 7fa9ef940700 1 -- 172.21.15.189:0/2168157736 >> 172.21.15.189:0/2168157736 conn(0x7fa9e80b2db0 msgr2=0x7fa9e80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:57.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.529+0000 7fa9ef940700 1 -- 172.21.15.189:0/2168157736 shutdown_connections 2024-04-06T13:47:57.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.529+0000 7fa9ef940700 1 -- 172.21.15.189:0/2168157736 wait complete. 2024-04-06T13:47:57.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.530+0000 7fa9ef940700 1 Processor -- start 2024-04-06T13:47:57.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.530+0000 7fa9ef940700 1 -- start start 2024-04-06T13:47:57.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.531+0000 7fa9ef940700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9e80cedf0 0x7fa9e80cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:57.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.531+0000 7fa9ef940700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9e80d17a0 0x7fa9e80b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:57.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.531+0000 7fa9ef940700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9e8146dd0 0x7fa9e80c6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:57.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.531+0000 7fa9ef940700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e80c5480 con 0x7fa9e8146dd0 2024-04-06T13:47:57.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.531+0000 7fa9ef940700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e80c5760 con 0x7fa9e80d17a0 2024-04-06T13:47:57.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.531+0000 7fa9ef940700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e80c5a40 con 0x7fa9e80cedf0 2024-04-06T13:47:57.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.531+0000 7fa9ededd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9e8146dd0 0x7fa9e80c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:57.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.531+0000 7fa9ededd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9e8146dd0 0x7fa9e80c6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38364/0 (socket says 172.21.15.189:38364) 2024-04-06T13:47:57.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.531+0000 7fa9ededd700 1 -- 172.21.15.189:0/4198053383 learned_addr learned my addr 172.21.15.189:0/4198053383 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:57.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.531+0000 7fa9ed6dc700 1 --2- 172.21.15.189:0/4198053383 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9e80cedf0 0x7fa9e80cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:57.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.531+0000 7fa9ecedb700 1 --2- 172.21.15.189:0/4198053383 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9e80d17a0 0x7fa9e80b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:57.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.531+0000 7fa9ed6dc700 1 -- 172.21.15.189:0/4198053383 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9e80d17a0 msgr2=0x7fa9e80b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:57.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.531+0000 7fa9ed6dc700 1 --2- 172.21.15.189:0/4198053383 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9e80d17a0 0x7fa9e80b02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.531+0000 7fa9ed6dc700 1 -- 172.21.15.189:0/4198053383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9e8146dd0 msgr2=0x7fa9e80c6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:57.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.531+0000 7fa9ed6dc700 1 --2- 172.21.15.189:0/4198053383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9e8146dd0 0x7fa9e80c6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.531+0000 7fa9ed6dc700 1 -- 172.21.15.189:0/4198053383 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d80094a0 con 0x7fa9e80cedf0 2024-04-06T13:47:57.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.531+0000 7fa9ededd700 1 --2- 172.21.15.189:0/4198053383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9e8146dd0 0x7fa9e80c6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:57.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.532+0000 7fa9ed6dc700 1 --2- 172.21.15.189:0/4198053383 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9e80cedf0 0x7fa9e80cddc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa9e400c390 tx=0x7fa9e400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:57.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.532+0000 7fa9ddffb700 1 -- 172.21.15.189:0/4198053383 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4005b40 con 0x7fa9e80cedf0 2024-04-06T13:47:57.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.532+0000 7fa9ddffb700 1 -- 172.21.15.189:0/4198053383 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e4005ca0 con 0x7fa9e80cedf0 2024-04-06T13:47:57.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.532+0000 7fa9ef940700 1 -- 172.21.15.189:0/4198053383 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e80c5d20 con 0x7fa9e80cedf0 2024-04-06T13:47:57.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.532+0000 7fa9ddffb700 1 -- 172.21.15.189:0/4198053383 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4007310 con 0x7fa9e80cedf0 2024-04-06T13:47:57.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.532+0000 7fa9ef940700 1 -- 172.21.15.189:0/4198053383 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e80c4d50 con 0x7fa9e80cedf0 2024-04-06T13:47:57.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.533+0000 7fa9ef940700 1 -- 172.21.15.189:0/4198053383 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e80c6240 con 0x7fa9e80cedf0 2024-04-06T13:47:57.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.533+0000 7fa9ddffb700 1 -- 172.21.15.189:0/4198053383 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 146) v1 ==== 89406+0+0 (secure 0 0 0) 0x7fa9e4007470 con 0x7fa9e80cedf0 2024-04-06T13:47:57.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.534+0000 7fa9ddffb700 1 --2- 172.21.15.189:0/4198053383 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa9cc06af60 0x7fa9cc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:57.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.534+0000 7fa9ecedb700 1 -- 172.21.15.189:0/4198053383 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa9cc06af60 msgr2=0x7fa9cc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:57.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.534+0000 7fa9ddffb700 1 -- 172.21.15.189:0/4198053383 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7309+0+0 (secure 0 0 0) 0x7fa9e4089070 con 0x7fa9e80cedf0 2024-04-06T13:47:57.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.534+0000 7fa9ecedb700 1 --2- 172.21.15.189:0/4198053383 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa9cc06af60 0x7fa9cc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:47:57.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.538+0000 7fa9ddffb700 1 -- 172.21.15.189:0/4198053383 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa9e40538d0 con 0x7fa9e80cedf0 2024-04-06T13:47:57.607 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:47:57.606+0000 7f1c6056b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:47:57.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.735+0000 7fa9ecedb700 1 -- 172.21.15.189:0/4198053383 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa9cc06af60 msgr2=0x7fa9cc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:47:57.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.735+0000 7fa9ecedb700 1 --2- 172.21.15.189:0/4198053383 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa9cc06af60 0x7fa9cc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:47:57.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.743+0000 7fa9ef940700 1 -- 172.21.15.189:0/4198053383 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7fa9e8074990 con 0x7fa9e80cedf0 2024-04-06T13:47:57.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.744+0000 7fa9ddffb700 1 -- 172.21.15.189:0/4198053383 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v232) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa9e4056ef0 con 0x7fa9e80cedf0 2024-04-06T13:47:57.745 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:47:57.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.746+0000 7fa9ef940700 1 -- 172.21.15.189:0/4198053383 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa9cc06af60 msgr2=0x7fa9cc06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:47:57.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.746+0000 7fa9ef940700 1 --2- 172.21.15.189:0/4198053383 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa9cc06af60 0x7fa9cc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.746+0000 7fa9ef940700 1 -- 172.21.15.189:0/4198053383 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9e80cedf0 msgr2=0x7fa9e80cddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:57.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.746+0000 7fa9ef940700 1 --2- 172.21.15.189:0/4198053383 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9e80cedf0 0x7fa9e80cddc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa9e400c390 tx=0x7fa9e400c6a0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.746+0000 7fa9ef940700 1 -- 172.21.15.189:0/4198053383 shutdown_connections 2024-04-06T13:47:57.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.746+0000 7fa9ef940700 1 --2- 172.21.15.189:0/4198053383 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa9cc06af60 0x7fa9cc06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.746+0000 7fa9ef940700 1 --2- 172.21.15.189:0/4198053383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9e8146dd0 0x7fa9e80c6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.746+0000 7fa9ef940700 1 --2- 172.21.15.189:0/4198053383 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9e80cedf0 0x7fa9e80cddc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.746+0000 7fa9ef940700 1 --2- 172.21.15.189:0/4198053383 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9e80d17a0 0x7fa9e80b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.746+0000 7fa9ef940700 1 -- 172.21.15.189:0/4198053383 >> 172.21.15.189:0/4198053383 conn(0x7fa9e80b2db0 msgr2=0x7fa9e8160810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:57.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.746+0000 7fa9ef940700 1 -- 172.21.15.189:0/4198053383 shutdown_connections 2024-04-06T13:47:57.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.746+0000 7fa9ef940700 1 -- 172.21.15.189:0/4198053383 wait complete. 2024-04-06T13:47:57.761 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:47:57.895 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.894+0000 7f7e16fe6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:47:57.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.932+0000 7f6b52ef1700 1 Processor -- start 2024-04-06T13:47:57.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.932+0000 7f6b52ef1700 1 -- start start 2024-04-06T13:47:57.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.933+0000 7f6b52ef1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b4c160b60 0x7f6b4c0b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:57.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.933+0000 7f6b52ef1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6b4c143470 0x7f6b4c0cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:57.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.933+0000 7f6b52ef1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b4c15eb40 con 0x7f6b4c15db10 2024-04-06T13:47:57.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.933+0000 7f6b52ef1700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b4c160f40 con 0x7f6b4c143470 2024-04-06T13:47:57.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.933+0000 7f6b52ef1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c0d0bf0 con 0x7f6b4c160b60 2024-04-06T13:47:57.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.933+0000 7f6b5148e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6b4c143470 0x7f6b4c0cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:57.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.933+0000 7f6b4bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6b4c15db10 0x7f6b4c142d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58340/0 (socket says 172.21.15.189:58340) 2024-04-06T13:47:57.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.933+0000 7f6b4bfff700 1 -- 172.21.15.189:0/1062092515 learned_addr learned my addr 172.21.15.189:0/1062092515 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:57.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.933+0000 7f6b50c8d700 1 --2- 172.21.15.189:0/1062092515 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b4c160b60 0x7f6b4c0b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:57.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.933+0000 7f6b4b7fe700 1 -- 172.21.15.189:0/1062092515 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343580700 0 0) 0x7f6b38001290 con 0x7f6b4c15db10 2024-04-06T13:47:57.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.933+0000 7f6b4b7fe700 1 -- 172.21.15.189:0/1062092515 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b3c0035b0 con 0x7f6b4c15db10 2024-04-06T13:47:57.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.933+0000 7f6b50c8d700 1 -- 172.21.15.189:0/1062092515 >> v2:172.21.15.189:3300/0 conn(0x7f6b4c143470 msgr2=0x7f6b4c0cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:57.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.933+0000 7f6b50c8d700 1 --2- 172.21.15.189:0/1062092515 >> v2:172.21.15.189:3300/0 conn(0x7f6b4c143470 0x7f6b4c0cd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.933+0000 7f6b50c8d700 1 -- 172.21.15.189:0/1062092515 >> v1:172.21.15.189:6789/0 conn(0x7f6b4c15db10 legacy=0x7f6b4c142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:57.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.933+0000 7f6b50c8d700 1 -- 172.21.15.189:0/1062092515 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4c15dfc0 con 0x7f6b4c160b60 2024-04-06T13:47:57.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.933+0000 7f6b5148e700 1 --2- 172.21.15.189:0/1062092515 >> v2:172.21.15.189:3300/0 conn(0x7f6b4c143470 0x7f6b4c0cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:47:57.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.933+0000 7f6b50c8d700 1 --2- 172.21.15.189:0/1062092515 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b4c160b60 0x7f6b4c0b37f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6b40009840 tx=0x7f6b40009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8295cf1c6872bfd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:57.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.933+0000 7f6b4b7fe700 1 -- 172.21.15.189:0/1062092515 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b40011830 con 0x7f6b4c160b60 2024-04-06T13:47:57.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.933+0000 7f6b4b7fe700 1 -- 172.21.15.189:0/1062092515 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b40011990 con 0x7f6b4c160b60 2024-04-06T13:47:57.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.934+0000 7f6b4b7fe700 1 -- 172.21.15.189:0/1062092515 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b40011d20 con 0x7f6b4c160b60 2024-04-06T13:47:57.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.934+0000 7f6b52ef1700 1 -- 172.21.15.189:0/1062092515 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b4c160b60 msgr2=0x7f6b4c0b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:57.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.934+0000 7f6b52ef1700 1 --2- 172.21.15.189:0/1062092515 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b4c160b60 0x7f6b4c0b37f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6b40009840 tx=0x7f6b40009b50 comp rx=0 tx=0).stop 2024-04-06T13:47:57.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.934+0000 7f6b52ef1700 1 -- 172.21.15.189:0/1062092515 shutdown_connections 2024-04-06T13:47:57.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.934+0000 7f6b52ef1700 1 --2- 172.21.15.189:0/1062092515 >> v2:172.21.15.189:3300/0 conn(0x7f6b4c143470 0x7f6b4c0cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.934+0000 7f6b52ef1700 1 --2- 172.21.15.189:0/1062092515 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b4c160b60 0x7f6b4c0b37f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.934+0000 7f6b52ef1700 1 -- 172.21.15.189:0/1062092515 >> 172.21.15.189:0/1062092515 conn(0x7f6b4c06a730 msgr2=0x7f6b4c06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:57.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.934+0000 7f6b52ef1700 1 -- 172.21.15.189:0/1062092515 shutdown_connections 2024-04-06T13:47:57.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.934+0000 7f6b52ef1700 1 -- 172.21.15.189:0/1062092515 wait complete. 2024-04-06T13:47:57.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.935+0000 7f6b52ef1700 1 Processor -- start 2024-04-06T13:47:57.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.935+0000 7f6b52ef1700 1 -- start start 2024-04-06T13:47:57.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.936+0000 7f6b52ef1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b4c143470 0x7f6b4c15cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:57.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.936+0000 7f6b52ef1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b4c15db10 0x7f6b4c0b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:57.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.936+0000 7f6b52ef1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b4c160b60 0x7f6b4c1e9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:57.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.936+0000 7f6b52ef1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c0c51c0 con 0x7f6b4c15db10 2024-04-06T13:47:57.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.936+0000 7f6b52ef1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c0c54a0 con 0x7f6b4c160b60 2024-04-06T13:47:57.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.936+0000 7f6b52ef1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c0c5780 con 0x7f6b4c143470 2024-04-06T13:47:57.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.936+0000 7f6b4bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b4c15db10 0x7f6b4c0b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:57.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.936+0000 7f6b4bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b4c15db10 0x7f6b4c0b0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38400/0 (socket says 172.21.15.189:38400) 2024-04-06T13:47:57.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.936+0000 7f6b4bfff700 1 -- 172.21.15.189:0/375034346 learned_addr learned my addr 172.21.15.189:0/375034346 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:57.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.936+0000 7f6b5148e700 1 --2- 172.21.15.189:0/375034346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b4c160b60 0x7f6b4c1e9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:57.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.936+0000 7f6b50c8d700 1 --2- 172.21.15.189:0/375034346 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b4c143470 0x7f6b4c15cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:57.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.937+0000 7f6b4bfff700 1 -- 172.21.15.189:0/375034346 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b4c143470 msgr2=0x7f6b4c15cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:57.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.937+0000 7f6b4bfff700 1 --2- 172.21.15.189:0/375034346 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b4c143470 0x7f6b4c15cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.937+0000 7f6b4bfff700 1 -- 172.21.15.189:0/375034346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b4c160b60 msgr2=0x7f6b4c1e9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:57.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.937+0000 7f6b4bfff700 1 --2- 172.21.15.189:0/375034346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b4c160b60 0x7f6b4c1e9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:57.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.937+0000 7f6b4bfff700 1 -- 172.21.15.189:0/375034346 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b44009a70 con 0x7f6b4c15db10 2024-04-06T13:47:57.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.937+0000 7f6b5148e700 1 --2- 172.21.15.189:0/375034346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b4c160b60 0x7f6b4c1e9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:57.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.937+0000 7f6b4bfff700 1 --2- 172.21.15.189:0/375034346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b4c15db10 0x7f6b4c0b0280 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f6b380013f0 tx=0x7f6b3800ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:57.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.937+0000 7f6b497fa700 1 -- 172.21.15.189:0/375034346 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b38005c40 con 0x7f6b4c15db10 2024-04-06T13:47:57.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.937+0000 7f6b497fa700 1 -- 172.21.15.189:0/375034346 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b38005da0 con 0x7f6b4c15db10 2024-04-06T13:47:57.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.937+0000 7f6b52ef1700 1 -- 172.21.15.189:0/375034346 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b400094a0 con 0x7f6b4c15db10 2024-04-06T13:47:57.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.937+0000 7f6b497fa700 1 -- 172.21.15.189:0/375034346 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b380072d0 con 0x7f6b4c15db10 2024-04-06T13:47:57.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.937+0000 7f6b52ef1700 1 -- 172.21.15.189:0/375034346 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b4c0c5d40 con 0x7f6b4c15db10 2024-04-06T13:47:57.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.938+0000 7f6b52ef1700 1 -- 172.21.15.189:0/375034346 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b4c067bf0 con 0x7f6b4c15db10 2024-04-06T13:47:57.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.938+0000 7f6b497fa700 1 -- 172.21.15.189:0/375034346 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 147) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f6b38007430 con 0x7f6b4c15db10 2024-04-06T13:47:57.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.939+0000 7f6b497fa700 1 -- 172.21.15.189:0/375034346 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7696+0+0 (secure 0 0 0) 0x7f6b380887f0 con 0x7f6b4c15db10 2024-04-06T13:47:57.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:57.944+0000 7f6b497fa700 1 -- 172.21.15.189:0/375034346 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6b38053710 con 0x7f6b4c15db10 2024-04-06T13:47:57.983 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.981+0000 7f7d8f6ff700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:57.983 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.981+0000 7f7d8f6ff700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:57.983 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.981+0000 7f7d8f6ff700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:57.983 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.981+0000 7f7d8f6ff700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:57.983 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.981+0000 7f7d8f6ff700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:57.983 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.982+0000 7f7d92705700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:57.983 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.982+0000 7f7d92705700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:57.983 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.982+0000 7f7d92705700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:57.983 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.982+0000 7f7d92705700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:57.983 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.982+0000 7f7d92705700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:57.991 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.990+0000 7f7d92705700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:57.991 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.990+0000 7f7d92705700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:57.991 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.990+0000 7f7d92705700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:57.991 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.990+0000 7f7d92705700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:57.991 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.990+0000 7f7d92705700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:57.992 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.991+0000 7f7d8f6ff700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:57.992 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.991+0000 7f7d8f6ff700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:57.992 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.991+0000 7f7d8f6ff700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:57.992 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.991+0000 7f7d8f6ff700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:57.992 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:47:57.991+0000 7f7d8f6ff700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:47:58.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.202+0000 7f6b52ef1700 1 -- 172.21.15.189:0/375034346 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b4c15eb40 con 0x7f6b4c15db10 2024-04-06T13:47:58.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.203+0000 7f6b497fa700 1 -- 172.21.15.189:0/375034346 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5971 (secure 0 0 0) 0x7f6b4c15eb40 con 0x7f6b4c15db10 2024-04-06T13:47:58.204 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:58.204 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":106,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8885,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1481817364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1481817364},{"type":"v1","addr":"172.21.15.189:6837","nonce":1481817364}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":11244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/4021875447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":4021875447},{"type":"v1","addr":"172.21.15.189:6836","nonce":4021875447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:47:22.116705+0000","modified":"2024-04-06T13:47:25.129787+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/694564673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":694564673},{"type":"v1","addr":"172.21.15.192:6835","nonce":694564673}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:47:34.584208+0000","modified":"2024-04-06T13:47:37.592983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11262":{"gid":11262,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6834/4066431971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4066431971},{"type":"v1","addr":"172.21.15.192:6834","nonce":4066431971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-04-06T13:47:58.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.205+0000 7f6b52ef1700 1 -- 172.21.15.189:0/375034346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b4c15db10 msgr2=0x7f6b4c0b0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:58.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.205+0000 7f6b52ef1700 1 --2- 172.21.15.189:0/375034346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b4c15db10 0x7f6b4c0b0280 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f6b380013f0 tx=0x7f6b3800ca90 comp rx=0 tx=0).stop 2024-04-06T13:47:58.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.205+0000 7f6b52ef1700 1 -- 172.21.15.189:0/375034346 shutdown_connections 2024-04-06T13:47:58.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.205+0000 7f6b52ef1700 1 --2- 172.21.15.189:0/375034346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b4c160b60 0x7f6b4c1e9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.205+0000 7f6b52ef1700 1 --2- 172.21.15.189:0/375034346 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b4c143470 0x7f6b4c15cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.205+0000 7f6b52ef1700 1 --2- 172.21.15.189:0/375034346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b4c15db10 0x7f6b4c0b0280 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.205+0000 7f6b52ef1700 1 -- 172.21.15.189:0/375034346 >> 172.21.15.189:0/375034346 conn(0x7f6b4c06a730 msgr2=0x7f6b4c06ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:58.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.205+0000 7f6b52ef1700 1 -- 172.21.15.189:0/375034346 shutdown_connections 2024-04-06T13:47:58.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.205+0000 7f6b52ef1700 1 -- 172.21.15.189:0/375034346 wait complete. 2024-04-06T13:47:58.207 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 106 2024-04-06T13:47:58.219 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:47:58.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.339+0000 7f60ad66d700 1 Processor -- start 2024-04-06T13:47:58.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.339+0000 7f60ad66d700 1 -- start start 2024-04-06T13:47:58.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.340+0000 7f60ad66d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60a806ba20 0x7f60a8166270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:58.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.340+0000 7f60ad66d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a80669d0 con 0x7f60a80cbc20 2024-04-06T13:47:58.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.340+0000 7f60ad66d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a806ede0 con 0x7f60a815e8a0 2024-04-06T13:47:58.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.340+0000 7f60ad66d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a815fd00 con 0x7f60a806ba20 2024-04-06T13:47:58.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.340+0000 7f60a67fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f60a80cbc20 0x7f60a815f5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58348/0 (socket says 172.21.15.189:58348) 2024-04-06T13:47:58.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.340+0000 7f60a67fc700 1 -- 172.21.15.189:0/1020576124 learned_addr learned my addr 172.21.15.189:0/1020576124 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:58.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a77fe700 1 --2- 172.21.15.189:0/1020576124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60a806ba20 0x7f60a8166270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:58.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a5ffb700 1 -- 172.21.15.189:0/1020576124 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717077276 0 0) 0x7f60a80669d0 con 0x7f60a80cbc20 2024-04-06T13:47:58.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a5ffb700 1 -- 172.21.15.189:0/1020576124 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60940035b0 con 0x7f60a80cbc20 2024-04-06T13:47:58.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a5ffb700 1 -- 172.21.15.189:0/1020576124 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3166027498 0 0) 0x7f60940035b0 con 0x7f60a80cbc20 2024-04-06T13:47:58.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a5ffb700 1 -- 172.21.15.189:0/1020576124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60a806ba20 msgr2=0x7f60a8166270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:58.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a5ffb700 1 --2- 172.21.15.189:0/1020576124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60a806ba20 0x7f60a8166270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a5ffb700 1 -- 172.21.15.189:0/1020576124 >> v1:172.21.15.192:6789/0 conn(0x7f60a815e8a0 legacy=0x7f60a815eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:58.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a5ffb700 1 -- 172.21.15.189:0/1020576124 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a815ec80 con 0x7f60a80cbc20 2024-04-06T13:47:58.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a5ffb700 1 -- 172.21.15.189:0/1020576124 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f6098001670 con 0x7f60a80cbc20 2024-04-06T13:47:58.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a5ffb700 1 -- 172.21.15.189:0/1020576124 >> v1:172.21.15.189:6789/0 conn(0x7f60a80cbc20 legacy=0x7f60a815f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:58.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a5ffb700 1 --2- 172.21.15.189:0/1020576124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6094003a30 0x7f6094003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:58.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a5ffb700 1 --2- 172.21.15.189:0/1020576124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60a806ba20 0x7f6094004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:58.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a5ffb700 1 --2- 172.21.15.189:0/1020576124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6094004d90 0x7f6094005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:58.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a5ffb700 1 -- 172.21.15.189:0/1020576124 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a815fd00 con 0x7f6094004d90 2024-04-06T13:47:58.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a5ffb700 1 -- 172.21.15.189:0/1020576124 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609400bac0 con 0x7f6094003a30 2024-04-06T13:47:58.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a5ffb700 1 -- 172.21.15.189:0/1020576124 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f609400bc80 con 0x7f60a806ba20 2024-04-06T13:47:58.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a77fe700 1 --2- 172.21.15.189:0/1020576124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6094004d90 0x7f6094005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:58.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a6ffd700 1 --2- 172.21.15.189:0/1020576124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6094003a30 0x7f6094003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:58.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a67fc700 1 --2- 172.21.15.189:0/1020576124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60a806ba20 0x7f6094004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:58.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a77fe700 1 -- 172.21.15.189:0/1020576124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60a806ba20 msgr2=0x7f6094004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:58.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a77fe700 1 --2- 172.21.15.189:0/1020576124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60a806ba20 0x7f6094004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a77fe700 1 -- 172.21.15.189:0/1020576124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6094003a30 msgr2=0x7f6094003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:58.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a77fe700 1 --2- 172.21.15.189:0/1020576124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6094003a30 0x7f6094003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.341+0000 7f60a77fe700 1 -- 172.21.15.189:0/1020576124 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60940035b0 con 0x7f6094004d90 2024-04-06T13:47:58.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.342+0000 7f60a77fe700 1 --2- 172.21.15.189:0/1020576124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6094004d90 0x7f6094005210 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009fc0 tx=0x7f6094005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc57ce9c601fa3b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:58.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.342+0000 7f60a5ffb700 1 -- 172.21.15.189:0/1020576124 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a000ec30 con 0x7f6094004d90 2024-04-06T13:47:58.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.342+0000 7f60a5ffb700 1 -- 172.21.15.189:0/1020576124 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a000f040 con 0x7f6094004d90 2024-04-06T13:47:58.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.342+0000 7f60a5ffb700 1 -- 172.21.15.189:0/1020576124 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a0014480 con 0x7f6094004d90 2024-04-06T13:47:58.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.342+0000 7f60ad66d700 1 -- 172.21.15.189:0/1020576124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6094004d90 msgr2=0x7f6094005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:58.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.342+0000 7f60ad66d700 1 --2- 172.21.15.189:0/1020576124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6094004d90 0x7f6094005210 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009fc0 tx=0x7f6094005880 comp rx=0 tx=0).stop 2024-04-06T13:47:58.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.342+0000 7f60a77fe700 1 -- 172.21.15.189:0/1020576124 reap_dead start 2024-04-06T13:47:58.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.342+0000 7f60ad66d700 1 -- 172.21.15.189:0/1020576124 shutdown_connections 2024-04-06T13:47:58.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.342+0000 7f60ad66d700 1 -- 172.21.15.189:0/1020576124 >> 172.21.15.189:0/1020576124 conn(0x7f60a80cc290 msgr2=0x7f60a80cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:58.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.342+0000 7f60ad66d700 1 -- 172.21.15.189:0/1020576124 shutdown_connections 2024-04-06T13:47:58.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.342+0000 7f60ad66d700 1 -- 172.21.15.189:0/1020576124 wait complete. 2024-04-06T13:47:58.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.344+0000 7f60ad66d700 1 Processor -- start 2024-04-06T13:47:58.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.344+0000 7f60ad66d700 1 -- start start 2024-04-06T13:47:58.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.344+0000 7f60ad66d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60a814f510 0x7f60a80c3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:58.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.344+0000 7f60ad66d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60a814f8f0 0x7f60a80af810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:58.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.344+0000 7f60ad66d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60a80c36b0 0x7f60a80afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:58.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.344+0000 7f60ad66d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a80c4b10 con 0x7f60a814f510 2024-04-06T13:47:58.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.344+0000 7f60ad66d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a80c3c70 con 0x7f60a80c36b0 2024-04-06T13:47:58.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.344+0000 7f60ad66d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a80c3f50 con 0x7f60a814f8f0 2024-04-06T13:47:58.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.344+0000 7f60a6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60a814f510 0x7f60a80c3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:58.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.344+0000 7f60a6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60a814f510 0x7f60a80c3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38528/0 (socket says 172.21.15.189:38528) 2024-04-06T13:47:58.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.344+0000 7f60a6ffd700 1 -- 172.21.15.189:0/1048761309 learned_addr learned my addr 172.21.15.189:0/1048761309 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:58.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.345+0000 7f60a67fc700 1 --2- 172.21.15.189:0/1048761309 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60a814f8f0 0x7f60a80af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:58.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.345+0000 7f60a77fe700 1 --2- 172.21.15.189:0/1048761309 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60a80c36b0 0x7f60a80afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:58.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.345+0000 7f60a6ffd700 1 -- 172.21.15.189:0/1048761309 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60a814f8f0 msgr2=0x7f60a80af810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:58.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.345+0000 7f60a6ffd700 1 --2- 172.21.15.189:0/1048761309 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60a814f8f0 0x7f60a80af810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.345+0000 7f60a6ffd700 1 -- 172.21.15.189:0/1048761309 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60a80c36b0 msgr2=0x7f60a80afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:58.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.345+0000 7f60a6ffd700 1 --2- 172.21.15.189:0/1048761309 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60a80c36b0 0x7f60a80afd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.345+0000 7f60a6ffd700 1 -- 172.21.15.189:0/1048761309 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a80c4230 con 0x7f60a814f510 2024-04-06T13:47:58.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.345+0000 7f60a6ffd700 1 --2- 172.21.15.189:0/1048761309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60a814f510 0x7f60a80c3170 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f609000f820 tx=0x7f609000fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:58.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.345+0000 7f609f7fe700 1 -- 172.21.15.189:0/1048761309 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60900052e0 con 0x7f60a814f510 2024-04-06T13:47:58.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.345+0000 7f609f7fe700 1 -- 172.21.15.189:0/1048761309 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6090005440 con 0x7f60a814f510 2024-04-06T13:47:58.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.345+0000 7f609f7fe700 1 -- 172.21.15.189:0/1048761309 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60900038c0 con 0x7f60a814f510 2024-04-06T13:47:58.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.345+0000 7f60ad66d700 1 -- 172.21.15.189:0/1048761309 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60a80c47a0 con 0x7f60a814f510 2024-04-06T13:47:58.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.345+0000 7f60ad66d700 1 -- 172.21.15.189:0/1048761309 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a81f1110 con 0x7f60a814f510 2024-04-06T13:47:58.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.346+0000 7f60ad66d700 1 -- 172.21.15.189:0/1048761309 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60a8072140 con 0x7f60a814f510 2024-04-06T13:47:58.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.347+0000 7f609f7fe700 1 -- 172.21.15.189:0/1048761309 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 147) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f6090003a20 con 0x7f60a814f510 2024-04-06T13:47:58.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.347+0000 7f609f7fe700 1 -- 172.21.15.189:0/1048761309 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7696+0+0 (secure 0 0 0) 0x7f6090089360 con 0x7f60a814f510 2024-04-06T13:47:58.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.351+0000 7f609f7fe700 1 -- 172.21.15.189:0/1048761309 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f60900541d0 con 0x7f60a814f510 2024-04-06T13:47:58.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.548+0000 7f60ad66d700 1 -- 172.21.15.189:0/1048761309 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60a806ea50 con 0x7f60a814f510 2024-04-06T13:47:58.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.549+0000 7f609f7fe700 1 -- 172.21.15.189:0/1048761309 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v233) v1 ==== 74+0+21201 (secure 0 0 0) 0x7f60a806ea50 con 0x7f60a814f510 2024-04-06T13:47:58.550 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:58.550 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":233,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:47:57.909364+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:47:20.644275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-04-06T13:47:21.075143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:47:32.292623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:47:33.567052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/3017776380":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1992839362":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/579471530":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/4049616553":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/619264656":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1506469869":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1831656794":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1190635801":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/3920072072":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/1878657401":"2024-04-07T13:47:47.129296+0000","172.21.15.189:6833/111704":"2024-04-07T13:47:47.129296+0000","172.21.15.189:6832/111704":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/2827130089":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/2716712615":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/1807905335":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/914480885":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1173010452":"2024-04-07T13:47:47.129296+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:47:58.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.551+0000 7f60ad66d700 1 -- 172.21.15.189:0/1048761309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60a814f510 msgr2=0x7f60a80c3170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:58.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.551+0000 7f60ad66d700 1 --2- 172.21.15.189:0/1048761309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60a814f510 0x7f60a80c3170 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f609000f820 tx=0x7f609000fb30 comp rx=0 tx=0).stop 2024-04-06T13:47:58.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.551+0000 7f60ad66d700 1 -- 172.21.15.189:0/1048761309 shutdown_connections 2024-04-06T13:47:58.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.551+0000 7f60ad66d700 1 --2- 172.21.15.189:0/1048761309 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60a80c36b0 0x7f60a80afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.551+0000 7f60ad66d700 1 --2- 172.21.15.189:0/1048761309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60a814f510 0x7f60a80c3170 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.551+0000 7f60ad66d700 1 --2- 172.21.15.189:0/1048761309 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60a814f8f0 0x7f60a80af810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.551+0000 7f60ad66d700 1 -- 172.21.15.189:0/1048761309 >> 172.21.15.189:0/1048761309 conn(0x7f60a80cc290 msgr2=0x7f60a80c29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:58.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.551+0000 7f60ad66d700 1 -- 172.21.15.189:0/1048761309 shutdown_connections 2024-04-06T13:47:58.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.551+0000 7f60ad66d700 1 -- 172.21.15.189:0/1048761309 wait complete. 2024-04-06T13:47:58.565 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-06T13:47:58.565 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:47:58.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.688+0000 7ff35c845700 1 Processor -- start 2024-04-06T13:47:58.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.688+0000 7ff35c845700 1 -- start start 2024-04-06T13:47:58.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.689+0000 7ff35c845700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3580b4660 0x7ff3580b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:58.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.689+0000 7ff35c845700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff3580b2fc0 0x7ff3580b3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:58.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.689+0000 7ff35c845700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3580cdbe0 con 0x7ff35815a550 2024-04-06T13:47:58.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.689+0000 7ff35c845700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3580b5000 con 0x7ff3580b2fc0 2024-04-06T13:47:58.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.689+0000 7ff35c845700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35815a940 con 0x7ff3580b4660 2024-04-06T13:47:58.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.689+0000 7ff356d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff3580b2fc0 0x7ff3580b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:58.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.689+0000 7ff35659c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff35815a550 0x7ff3580cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58356/0 (socket says 172.21.15.189:58356) 2024-04-06T13:47:58.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.689+0000 7ff35659c700 1 -- 172.21.15.189:0/4178185414 learned_addr learned my addr 172.21.15.189:0/4178185414 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:47:58.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.689+0000 7ff355d9b700 1 --2- 172.21.15.189:0/4178185414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3580b4660 0x7ff3580b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:58.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.689+0000 7ff356d9d700 1 -- 172.21.15.189:0/4178185414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3580b4660 msgr2=0x7ff3580b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:58.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.689+0000 7ff356d9d700 1 --2- 172.21.15.189:0/4178185414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3580b4660 0x7ff3580b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.689+0000 7ff356d9d700 1 -- 172.21.15.189:0/4178185414 >> v1:172.21.15.189:6789/0 conn(0x7ff35815a550 legacy=0x7ff3580cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:58.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.689+0000 7ff356d9d700 1 -- 172.21.15.189:0/4178185414 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35815ac00 con 0x7ff3580b2fc0 2024-04-06T13:47:58.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.689+0000 7ff35559a700 1 -- 172.21.15.189:0/4178185414 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453400519 0 0) 0x7ff3580cdbe0 con 0x7ff35815a550 2024-04-06T13:47:58.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.689+0000 7ff356d9d700 1 --2- 172.21.15.189:0/4178185414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3580b2fc0 0x7ff3580b3430 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7ff350009d90 tx=0x7ff35000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d32cc55ef417fa6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:58.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.689+0000 7ff35559a700 1 -- 172.21.15.189:0/4178185414 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35000e070 con 0x7ff3580b2fc0 2024-04-06T13:47:58.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.689+0000 7ff35559a700 1 -- 172.21.15.189:0/4178185414 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35000bbd0 con 0x7ff3580b2fc0 2024-04-06T13:47:58.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.689+0000 7ff35559a700 1 -- 172.21.15.189:0/4178185414 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff350012430 con 0x7ff3580b2fc0 2024-04-06T13:47:58.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.690+0000 7ff35c845700 1 -- 172.21.15.189:0/4178185414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3580b2fc0 msgr2=0x7ff3580b3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:58.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.690+0000 7ff35c845700 1 --2- 172.21.15.189:0/4178185414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3580b2fc0 0x7ff3580b3430 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7ff350009d90 tx=0x7ff35000b330 comp rx=0 tx=0).stop 2024-04-06T13:47:58.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.690+0000 7ff35c845700 1 -- 172.21.15.189:0/4178185414 shutdown_connections 2024-04-06T13:47:58.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.690+0000 7ff35c845700 1 --2- 172.21.15.189:0/4178185414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3580b2fc0 0x7ff3580b3430 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.690+0000 7ff35c845700 1 --2- 172.21.15.189:0/4178185414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3580b4660 0x7ff3580b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.690+0000 7ff35c845700 1 -- 172.21.15.189:0/4178185414 >> 172.21.15.189:0/4178185414 conn(0x7ff3580d00b0 msgr2=0x7ff3580d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:58.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.690+0000 7ff35c845700 1 -- 172.21.15.189:0/4178185414 shutdown_connections 2024-04-06T13:47:58.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.690+0000 7ff35c845700 1 -- 172.21.15.189:0/4178185414 wait complete. 2024-04-06T13:47:58.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.691+0000 7ff35c845700 1 Processor -- start 2024-04-06T13:47:58.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.691+0000 7ff35c845700 1 -- start start 2024-04-06T13:47:58.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.692+0000 7ff35c845700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3580b2fc0 0x7ff35813c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:58.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.692+0000 7ff35c845700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3580b4660 0x7ff358157610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:58.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.692+0000 7ff35c845700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff358157b50 0x7ff358153390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:58.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.692+0000 7ff35c845700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3581f4c60 con 0x7ff358157b50 2024-04-06T13:47:58.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.692+0000 7ff35c845700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff358156850 con 0x7ff3580b4660 2024-04-06T13:47:58.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.692+0000 7ff35c845700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff358156b30 con 0x7ff3580b2fc0 2024-04-06T13:47:58.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.692+0000 7ff356d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff358157b50 0x7ff358153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:58.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.692+0000 7ff356d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff358157b50 0x7ff358153390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38550/0 (socket says 172.21.15.189:38550) 2024-04-06T13:47:58.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.692+0000 7ff356d9d700 1 -- 172.21.15.189:0/3965056132 learned_addr learned my addr 172.21.15.189:0/3965056132 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:58.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.692+0000 7ff355d9b700 1 --2- 172.21.15.189:0/3965056132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3580b4660 0x7ff358157610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:58.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.693+0000 7ff35659c700 1 --2- 172.21.15.189:0/3965056132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3580b2fc0 0x7ff35813c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:58.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.693+0000 7ff356d9d700 1 -- 172.21.15.189:0/3965056132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3580b2fc0 msgr2=0x7ff35813c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:58.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.693+0000 7ff356d9d700 1 --2- 172.21.15.189:0/3965056132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3580b2fc0 0x7ff35813c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.693+0000 7ff356d9d700 1 -- 172.21.15.189:0/3965056132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3580b4660 msgr2=0x7ff358157610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:58.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.693+0000 7ff356d9d700 1 --2- 172.21.15.189:0/3965056132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3580b4660 0x7ff358157610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.693+0000 7ff356d9d700 1 -- 172.21.15.189:0/3965056132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff350009a70 con 0x7ff358157b50 2024-04-06T13:47:58.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.693+0000 7ff355d9b700 1 --2- 172.21.15.189:0/3965056132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3580b4660 0x7ff358157610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:58.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.693+0000 7ff35659c700 1 --2- 172.21.15.189:0/3965056132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3580b2fc0 0x7ff35813c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:58.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.693+0000 7ff356d9d700 1 --2- 172.21.15.189:0/3965056132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff358157b50 0x7ff358153390 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7ff35806c820 tx=0x7ff350004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:58.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.693+0000 7ff346ffd700 1 -- 172.21.15.189:0/3965056132 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff350012430 con 0x7ff358157b50 2024-04-06T13:47:58.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.693+0000 7ff346ffd700 1 -- 172.21.15.189:0/3965056132 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff350020070 con 0x7ff358157b50 2024-04-06T13:47:58.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.693+0000 7ff35c845700 1 -- 172.21.15.189:0/3965056132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff358156e10 con 0x7ff358157b50 2024-04-06T13:47:58.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.693+0000 7ff346ffd700 1 -- 172.21.15.189:0/3965056132 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff350004280 con 0x7ff358157b50 2024-04-06T13:47:58.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.693+0000 7ff35c845700 1 -- 172.21.15.189:0/3965056132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35813cd20 con 0x7ff358157b50 2024-04-06T13:47:58.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.694+0000 7ff35c845700 1 -- 172.21.15.189:0/3965056132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff358157330 con 0x7ff358157b50 2024-04-06T13:47:58.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.694+0000 7ff346ffd700 1 -- 172.21.15.189:0/3965056132 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 147) v1 ==== 88856+0+0 (secure 0 0 0) 0x7ff3500043e0 con 0x7ff358157b50 2024-04-06T13:47:58.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.695+0000 7ff346ffd700 1 -- 172.21.15.189:0/3965056132 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7696+0+0 (secure 0 0 0) 0x7ff350022070 con 0x7ff358157b50 2024-04-06T13:47:58.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.699+0000 7ff346ffd700 1 -- 172.21.15.189:0/3965056132 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff358157330 con 0x7ff358157b50 2024-04-06T13:47:58.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.921+0000 7ff346ffd700 1 -- 172.21.15.189:0/3965056132 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 148) v1 ==== 88983+0+0 (secure 0 0 0) 0x7ff350056490 con 0x7ff358157b50 2024-04-06T13:47:58.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.921+0000 7ff346ffd700 1 --2- 172.21.15.189:0/3965056132 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff338036150 0x7ff3380751c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:58.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.922+0000 7ff35659c700 1 --2- 172.21.15.189:0/3965056132 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff338036150 0x7ff3380751c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:58.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.923+0000 7ff35659c700 1 --2- 172.21.15.189:0/3965056132 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff338036150 0x7ff3380751c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff348000f60 tx=0x7ff348009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:58.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.943+0000 7ff35c845700 1 -- 172.21.15.189:0/3965056132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff358157330 con 0x7ff358157b50 2024-04-06T13:47:58.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.944+0000 7ff346ffd700 1 -- 172.21.15.189:0/3965056132 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5971 (secure 0 0 0) 0x7ff358157330 con 0x7ff358157b50 2024-04-06T13:47:58.945 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:58.945 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":106,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8885,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1481817364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1481817364},{"type":"v1","addr":"172.21.15.189:6837","nonce":1481817364}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":11244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/4021875447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":4021875447},{"type":"v1","addr":"172.21.15.189:6836","nonce":4021875447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:47:22.116705+0000","modified":"2024-04-06T13:47:25.129787+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/694564673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":694564673},{"type":"v1","addr":"172.21.15.192:6835","nonce":694564673}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:47:34.584208+0000","modified":"2024-04-06T13:47:37.592983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11262":{"gid":11262,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6834/4066431971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4066431971},{"type":"v1","addr":"172.21.15.192:6834","nonce":4066431971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-04-06T13:47:58.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.946+0000 7ff35c845700 1 -- 172.21.15.189:0/3965056132 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff338036150 msgr2=0x7ff3380751c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:58.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.946+0000 7ff35c845700 1 --2- 172.21.15.189:0/3965056132 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff338036150 0x7ff3380751c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff348000f60 tx=0x7ff348009040 comp rx=0 tx=0).stop 2024-04-06T13:47:58.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.946+0000 7ff35c845700 1 -- 172.21.15.189:0/3965056132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff358157b50 msgr2=0x7ff358153390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:58.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.946+0000 7ff35c845700 1 --2- 172.21.15.189:0/3965056132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff358157b50 0x7ff358153390 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7ff35806c820 tx=0x7ff350004ad0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.946+0000 7ff35c845700 1 -- 172.21.15.189:0/3965056132 shutdown_connections 2024-04-06T13:47:58.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.946+0000 7ff35c845700 1 --2- 172.21.15.189:0/3965056132 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff338036150 0x7ff3380751c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.946+0000 7ff35c845700 1 --2- 172.21.15.189:0/3965056132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff358157b50 0x7ff358153390 unknown :-1 s=CLOSED pgs=2574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.946+0000 7ff35c845700 1 --2- 172.21.15.189:0/3965056132 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3580b2fc0 0x7ff35813c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.946+0000 7ff35c845700 1 --2- 172.21.15.189:0/3965056132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3580b4660 0x7ff358157610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:58.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.946+0000 7ff35c845700 1 -- 172.21.15.189:0/3965056132 >> 172.21.15.189:0/3965056132 conn(0x7ff3580d00b0 msgr2=0x7ff35806eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:58.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.946+0000 7ff35c845700 1 -- 172.21.15.189:0/3965056132 shutdown_connections 2024-04-06T13:47:58.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:58.946+0000 7ff35c845700 1 -- 172.21.15.189:0/3965056132 wait complete. 2024-04-06T13:47:58.948 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 106 2024-04-06T13:47:58.959 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:47:59.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.080+0000 7fba50a6c700 1 Processor -- start 2024-04-06T13:47:59.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.080+0000 7fba50a6c700 1 -- start start 2024-04-06T13:47:59.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.081+0000 7fba50a6c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fba4c0d11b0 0x7fba4c0d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:59.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.081+0000 7fba50a6c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fba4c0cb870 0x7fba4c15c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:59.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.081+0000 7fba50a6c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba4c0d1ad0 con 0x7fba4c0cfa00 2024-04-06T13:47:59.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.081+0000 7fba50a6c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba4c0ccdc0 con 0x7fba4c0d11b0 2024-04-06T13:47:59.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.081+0000 7fba50a6c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba4c0d0b50 con 0x7fba4c0cb870 2024-04-06T13:47:59.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.082+0000 7fba4a59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fba4c0d11b0 0x7fba4c0d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:59.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.082+0000 7fba4a59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fba4c0d11b0 0x7fba4c0d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38554/0 (socket says 172.21.15.189:38554) 2024-04-06T13:47:59.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.082+0000 7fba4a59c700 1 -- 172.21.15.189:0/387615641 learned_addr learned my addr 172.21.15.189:0/387615641 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:59.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.082+0000 7fba49d9b700 1 --2- 172.21.15.189:0/387615641 >> v2:172.21.15.192:3300/0 conn(0x7fba4c0cb870 0x7fba4c15c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:59.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.082+0000 7fba4a59c700 1 -- 172.21.15.189:0/387615641 >> v2:172.21.15.192:3300/0 conn(0x7fba4c0cb870 msgr2=0x7fba4c15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:59.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.082+0000 7fba4a59c700 1 --2- 172.21.15.189:0/387615641 >> v2:172.21.15.192:3300/0 conn(0x7fba4c0cb870 0x7fba4c15c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.082+0000 7fba4a59c700 1 -- 172.21.15.189:0/387615641 >> v1:172.21.15.192:6789/0 conn(0x7fba4c0cfa00 legacy=0x7fba4c0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:59.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.082+0000 7fba4a59c700 1 -- 172.21.15.189:0/387615641 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4c0cbd80 con 0x7fba4c0d11b0 2024-04-06T13:47:59.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.082+0000 7fba4a59c700 1 --2- 172.21.15.189:0/387615641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba4c0d11b0 0x7fba4c0d1590 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7fba4c04fb40 tx=0x7fba40009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec6b85695c94327 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:59.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.082+0000 7fba4959a700 1 -- 172.21.15.189:0/387615641 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4000b870 con 0x7fba4c0d11b0 2024-04-06T13:47:59.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.082+0000 7fba4959a700 1 -- 172.21.15.189:0/387615641 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4000b9d0 con 0x7fba4c0d11b0 2024-04-06T13:47:59.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.082+0000 7fba4959a700 1 -- 172.21.15.189:0/387615641 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4000bd60 con 0x7fba4c0d11b0 2024-04-06T13:47:59.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.083+0000 7fba50a6c700 1 -- 172.21.15.189:0/387615641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba4c0d11b0 msgr2=0x7fba4c0d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:59.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.083+0000 7fba50a6c700 1 --2- 172.21.15.189:0/387615641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba4c0d11b0 0x7fba4c0d1590 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7fba4c04fb40 tx=0x7fba40009bd0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.083+0000 7fba50a6c700 1 -- 172.21.15.189:0/387615641 shutdown_connections 2024-04-06T13:47:59.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.083+0000 7fba50a6c700 1 --2- 172.21.15.189:0/387615641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba4c0d11b0 0x7fba4c0d1590 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.083+0000 7fba50a6c700 1 --2- 172.21.15.189:0/387615641 >> v2:172.21.15.192:3300/0 conn(0x7fba4c0cb870 0x7fba4c15c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.083+0000 7fba50a6c700 1 -- 172.21.15.189:0/387615641 >> 172.21.15.189:0/387615641 conn(0x7fba4c0b2db0 msgr2=0x7fba4c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:59.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.083+0000 7fba50a6c700 1 -- 172.21.15.189:0/387615641 shutdown_connections 2024-04-06T13:47:59.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.083+0000 7fba50a6c700 1 -- 172.21.15.189:0/387615641 wait complete. 2024-04-06T13:47:59.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.084+0000 7fba50a6c700 1 Processor -- start 2024-04-06T13:47:59.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.084+0000 7fba50a6c700 1 -- start start 2024-04-06T13:47:59.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.085+0000 7fba50a6c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba4c0cb870 0x7fba4c15ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:59.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.085+0000 7fba50a6c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba4c0cfa00 0x7fba4c155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:59.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.085+0000 7fba50a6c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba4c0d11b0 0x7fba4c1577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:59.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.085+0000 7fba50a6c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c0ca4a0 con 0x7fba4c0cb870 2024-04-06T13:47:59.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.085+0000 7fba50a6c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c158de0 con 0x7fba4c0d11b0 2024-04-06T13:47:59.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.085+0000 7fba50a6c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c158b00 con 0x7fba4c0cfa00 2024-04-06T13:47:59.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.085+0000 7fba4a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba4c0cb870 0x7fba4c15ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:59.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.085+0000 7fba4a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba4c0cb870 0x7fba4c15ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38558/0 (socket says 172.21.15.189:38558) 2024-04-06T13:47:59.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.085+0000 7fba4a59c700 1 -- 172.21.15.189:0/3280523352 learned_addr learned my addr 172.21.15.189:0/3280523352 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:59.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.085+0000 7fba49d9b700 1 --2- 172.21.15.189:0/3280523352 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba4c0cfa00 0x7fba4c155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:59.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.085+0000 7fba4ad9d700 1 --2- 172.21.15.189:0/3280523352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba4c0d11b0 0x7fba4c1577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:59.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.085+0000 7fba4a59c700 1 -- 172.21.15.189:0/3280523352 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba4c0cfa00 msgr2=0x7fba4c155630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:59.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.085+0000 7fba4a59c700 1 --2- 172.21.15.189:0/3280523352 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba4c0cfa00 0x7fba4c155630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.085+0000 7fba4a59c700 1 -- 172.21.15.189:0/3280523352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba4c0d11b0 msgr2=0x7fba4c1577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:59.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.085+0000 7fba4a59c700 1 --2- 172.21.15.189:0/3280523352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba4c0d11b0 0x7fba4c1577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.085+0000 7fba4a59c700 1 -- 172.21.15.189:0/3280523352 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba40009550 con 0x7fba4c0cb870 2024-04-06T13:47:59.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.085+0000 7fba4ad9d700 1 --2- 172.21.15.189:0/3280523352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba4c0d11b0 0x7fba4c1577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:59.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.085+0000 7fba4a59c700 1 --2- 172.21.15.189:0/3280523352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba4c0cb870 0x7fba4c15ae90 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7fba4c0677e0 tx=0x7fba40004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:59.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.085+0000 7fba3affd700 1 -- 172.21.15.189:0/3280523352 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40004310 con 0x7fba4c0cb870 2024-04-06T13:47:59.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.086+0000 7fba3affd700 1 -- 172.21.15.189:0/3280523352 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4001f070 con 0x7fba4c0cb870 2024-04-06T13:47:59.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.086+0000 7fba3affd700 1 -- 172.21.15.189:0/3280523352 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba400036a0 con 0x7fba4c0cb870 2024-04-06T13:47:59.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.086+0000 7fba50a6c700 1 -- 172.21.15.189:0/3280523352 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba4c0ca6b0 con 0x7fba4c0cb870 2024-04-06T13:47:59.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.086+0000 7fba50a6c700 1 -- 172.21.15.189:0/3280523352 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba4c0cac70 con 0x7fba4c0cb870 2024-04-06T13:47:59.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.086+0000 7fba50a6c700 1 -- 172.21.15.189:0/3280523352 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba4c072d40 con 0x7fba4c0cb870 2024-04-06T13:47:59.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.087+0000 7fba3affd700 1 -- 172.21.15.189:0/3280523352 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 148) v1 ==== 88983+0+0 (secure 0 0 0) 0x7fba40021030 con 0x7fba4c0cb870 2024-04-06T13:47:59.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.088+0000 7fba3affd700 1 --2- 172.21.15.189:0/3280523352 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fba2c06aae0 0x7fba2c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:59.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.088+0000 7fba3affd700 1 -- 172.21.15.189:0/3280523352 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7696+0+0 (secure 0 0 0) 0x7fba4008adb0 con 0x7fba4c0cb870 2024-04-06T13:47:59.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.088+0000 7fba49d9b700 1 --2- 172.21.15.189:0/3280523352 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fba2c06aae0 0x7fba2c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:59.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.088+0000 7fba49d9b700 1 --2- 172.21.15.189:0/3280523352 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fba2c06aae0 0x7fba2c06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fba4c0ca290 tx=0x7fba3c008040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:59.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.092+0000 7fba3affd700 1 -- 172.21.15.189:0/3280523352 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fba40055ae0 con 0x7fba4c0cb870 2024-04-06T13:47:59.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.330+0000 7fba50a6c700 1 -- 172.21.15.189:0/3280523352 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba4c074990 con 0x7fba4c0cb870 2024-04-06T13:47:59.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.330+0000 7fba3affd700 1 -- 172.21.15.189:0/3280523352 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5971 (secure 0 0 0) 0x7fba40059100 con 0x7fba4c0cb870 2024-04-06T13:47:59.331 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:59.332 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":106,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8885,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1481817364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1481817364},{"type":"v1","addr":"172.21.15.189:6837","nonce":1481817364}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":11244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/4021875447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":4021875447},{"type":"v1","addr":"172.21.15.189:6836","nonce":4021875447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:47:22.116705+0000","modified":"2024-04-06T13:47:25.129787+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/694564673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":694564673},{"type":"v1","addr":"172.21.15.192:6835","nonce":694564673}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:47:34.584208+0000","modified":"2024-04-06T13:47:37.592983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11262":{"gid":11262,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6834/4066431971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4066431971},{"type":"v1","addr":"172.21.15.192:6834","nonce":4066431971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-04-06T13:47:59.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.333+0000 7fba50a6c700 1 -- 172.21.15.189:0/3280523352 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fba2c06aae0 msgr2=0x7fba2c06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:59.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.333+0000 7fba50a6c700 1 --2- 172.21.15.189:0/3280523352 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fba2c06aae0 0x7fba2c06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fba4c0ca290 tx=0x7fba3c008040 comp rx=0 tx=0).stop 2024-04-06T13:47:59.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.333+0000 7fba50a6c700 1 -- 172.21.15.189:0/3280523352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba4c0cb870 msgr2=0x7fba4c15ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:59.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.333+0000 7fba50a6c700 1 --2- 172.21.15.189:0/3280523352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba4c0cb870 0x7fba4c15ae90 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7fba4c0677e0 tx=0x7fba40004750 comp rx=0 tx=0).stop 2024-04-06T13:47:59.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.333+0000 7fba50a6c700 1 -- 172.21.15.189:0/3280523352 shutdown_connections 2024-04-06T13:47:59.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.333+0000 7fba50a6c700 1 --2- 172.21.15.189:0/3280523352 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fba2c06aae0 0x7fba2c06cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.333+0000 7fba50a6c700 1 --2- 172.21.15.189:0/3280523352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba4c0d11b0 0x7fba4c1577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.333+0000 7fba50a6c700 1 --2- 172.21.15.189:0/3280523352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba4c0cb870 0x7fba4c15ae90 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.333+0000 7fba50a6c700 1 --2- 172.21.15.189:0/3280523352 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba4c0cfa00 0x7fba4c155630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.333+0000 7fba50a6c700 1 -- 172.21.15.189:0/3280523352 >> 172.21.15.189:0/3280523352 conn(0x7fba4c0b2db0 msgr2=0x7fba4c0b5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:59.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.333+0000 7fba50a6c700 1 -- 172.21.15.189:0/3280523352 shutdown_connections 2024-04-06T13:47:59.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.333+0000 7fba50a6c700 1 -- 172.21.15.189:0/3280523352 wait complete. 2024-04-06T13:47:59.336 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 106 2024-04-06T13:47:59.347 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:47:59.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.467+0000 7f9b7c955700 1 Processor -- start 2024-04-06T13:47:59.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.467+0000 7f9b7c955700 1 -- start start 2024-04-06T13:47:59.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.468+0000 7f9b7c955700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9b780d0a60 0x7f9b780d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:59.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.468+0000 7f9b7c955700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b780d1af0 0x7f9b780b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:59.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.468+0000 7f9b7c955700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b780cdf80 con 0x7f9b7815d7e0 2024-04-06T13:47:59.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.468+0000 7f9b7c955700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b780d1f80 con 0x7f9b780d0a60 2024-04-06T13:47:59.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.468+0000 7f9b7c955700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b780d3230 con 0x7f9b780d1af0 2024-04-06T13:47:59.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.468+0000 7f9b75d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9b780d0a60 0x7f9b780d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:59.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.469+0000 7f9b75d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9b780d0a60 0x7f9b780d15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41602/0 (socket says 172.21.15.189:41602) 2024-04-06T13:47:59.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.469+0000 7f9b75d9b700 1 -- 172.21.15.189:0/2224913671 learned_addr learned my addr 172.21.15.189:0/2224913671 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:59.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.469+0000 7f9b76d9d700 1 --2- 172.21.15.189:0/2224913671 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b780d1af0 0x7f9b780b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:59.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.469+0000 7f9b75d9b700 1 -- 172.21.15.189:0/2224913671 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b780d1af0 msgr2=0x7f9b780b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:59.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.469+0000 7f9b75d9b700 1 --2- 172.21.15.189:0/2224913671 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b780d1af0 0x7f9b780b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.469+0000 7f9b75d9b700 1 -- 172.21.15.189:0/2224913671 >> v1:172.21.15.192:6789/0 conn(0x7f9b7815d7e0 legacy=0x7f9b780d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:59.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.469+0000 7f9b75d9b700 1 -- 172.21.15.189:0/2224913671 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b780d2da0 con 0x7f9b780d0a60 2024-04-06T13:47:59.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.469+0000 7f9b75d9b700 1 --2- 172.21.15.189:0/2224913671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b780d0a60 0x7f9b780d15b0 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7f9b600098f0 tx=0x7f9b60009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b224394e07eaf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:59.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.469+0000 7f9b7559a700 1 -- 172.21.15.189:0/2224913671 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6000b8a0 con 0x7f9b780d0a60 2024-04-06T13:47:59.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.469+0000 7f9b7559a700 1 -- 172.21.15.189:0/2224913671 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b6000ba00 con 0x7f9b780d0a60 2024-04-06T13:47:59.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.469+0000 7f9b7559a700 1 -- 172.21.15.189:0/2224913671 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6000bd90 con 0x7f9b780d0a60 2024-04-06T13:47:59.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.470+0000 7f9b7c955700 1 -- 172.21.15.189:0/2224913671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b780d0a60 msgr2=0x7f9b780d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:59.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.470+0000 7f9b7c955700 1 --2- 172.21.15.189:0/2224913671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b780d0a60 0x7f9b780d15b0 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7f9b600098f0 tx=0x7f9b60009c00 comp rx=0 tx=0).stop 2024-04-06T13:47:59.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.470+0000 7f9b7c955700 1 -- 172.21.15.189:0/2224913671 shutdown_connections 2024-04-06T13:47:59.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.470+0000 7f9b7c955700 1 --2- 172.21.15.189:0/2224913671 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b780d1af0 0x7f9b780b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.470+0000 7f9b7c955700 1 --2- 172.21.15.189:0/2224913671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b780d0a60 0x7f9b780d15b0 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.470+0000 7f9b7c955700 1 -- 172.21.15.189:0/2224913671 >> 172.21.15.189:0/2224913671 conn(0x7f9b78072250 msgr2=0x7f9b78072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:59.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.470+0000 7f9b7c955700 1 -- 172.21.15.189:0/2224913671 shutdown_connections 2024-04-06T13:47:59.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.470+0000 7f9b7c955700 1 -- 172.21.15.189:0/2224913671 wait complete. 2024-04-06T13:47:59.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.471+0000 7f9b7c955700 1 Processor -- start 2024-04-06T13:47:59.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.471+0000 7f9b7c955700 1 -- start start 2024-04-06T13:47:59.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.472+0000 7f9b7c955700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b780d0a60 0x7f9b780c7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:59.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.472+0000 7f9b7c955700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b780d1af0 0x7f9b78159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:59.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.472+0000 7f9b7c955700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b7815d7e0 0x7f9b780c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:59.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.472+0000 7f9b7c955700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b781f3b80 con 0x7f9b7815d7e0 2024-04-06T13:47:59.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.472+0000 7f9b7c955700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b780c74c0 con 0x7f9b780d1af0 2024-04-06T13:47:59.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.472+0000 7f9b7c955700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b780c6b90 con 0x7f9b780d0a60 2024-04-06T13:47:59.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.472+0000 7f9b76d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b7815d7e0 0x7f9b780c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:59.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.472+0000 7f9b76d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b7815d7e0 0x7f9b780c92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41616/0 (socket says 172.21.15.189:41616) 2024-04-06T13:47:59.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.472+0000 7f9b76d9d700 1 -- 172.21.15.189:0/3551110416 learned_addr learned my addr 172.21.15.189:0/3551110416 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:59.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.472+0000 7f9b7659c700 1 --2- 172.21.15.189:0/3551110416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b780d0a60 0x7f9b780c7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:59.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.472+0000 7f9b75d9b700 1 --2- 172.21.15.189:0/3551110416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b780d1af0 0x7f9b78159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:59.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.472+0000 7f9b76d9d700 1 -- 172.21.15.189:0/3551110416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b780d0a60 msgr2=0x7f9b780c7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:59.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.473+0000 7f9b76d9d700 1 --2- 172.21.15.189:0/3551110416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b780d0a60 0x7f9b780c7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.473+0000 7f9b76d9d700 1 -- 172.21.15.189:0/3551110416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b780d1af0 msgr2=0x7f9b78159840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:59.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.473+0000 7f9b76d9d700 1 --2- 172.21.15.189:0/3551110416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b780d1af0 0x7f9b78159840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.473+0000 7f9b76d9d700 1 -- 172.21.15.189:0/3551110416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b60009550 con 0x7f9b7815d7e0 2024-04-06T13:47:59.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.473+0000 7f9b76d9d700 1 --2- 172.21.15.189:0/3551110416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b7815d7e0 0x7f9b780c92e0 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f9b70007e40 tx=0x7f9b7000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:59.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.473+0000 7f9b6effd700 1 -- 172.21.15.189:0/3551110416 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7000eea0 con 0x7f9b7815d7e0 2024-04-06T13:47:59.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.473+0000 7f9b6effd700 1 -- 172.21.15.189:0/3551110416 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b70014420 con 0x7f9b7815d7e0 2024-04-06T13:47:59.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.473+0000 7f9b7c955700 1 -- 172.21.15.189:0/3551110416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b780c6e70 con 0x7f9b7815d7e0 2024-04-06T13:47:59.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.473+0000 7f9b6effd700 1 -- 172.21.15.189:0/3551110416 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b700147b0 con 0x7f9b7815d7e0 2024-04-06T13:47:59.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.473+0000 7f9b7c955700 1 -- 172.21.15.189:0/3551110416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b78159d80 con 0x7f9b7815d7e0 2024-04-06T13:47:59.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.474+0000 7f9b7c955700 1 -- 172.21.15.189:0/3551110416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b78159f10 con 0x7f9b7815d7e0 2024-04-06T13:47:59.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.474+0000 7f9b6effd700 1 -- 172.21.15.189:0/3551110416 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 148) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f9b70014910 con 0x7f9b7815d7e0 2024-04-06T13:47:59.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.475+0000 7f9b6effd700 1 --2- 172.21.15.189:0/3551110416 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9b5806aae0 0x7f9b5806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:59.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.475+0000 7f9b6effd700 1 -- 172.21.15.189:0/3551110416 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7696+0+0 (secure 0 0 0) 0x7f9b70089fe0 con 0x7f9b7815d7e0 2024-04-06T13:47:59.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.475+0000 7f9b7659c700 1 --2- 172.21.15.189:0/3551110416 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9b5806aae0 0x7f9b5806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:59.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.475+0000 7f9b7659c700 1 --2- 172.21.15.189:0/3551110416 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9b5806aae0 0x7f9b5806cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9b781f3d50 tx=0x7f9b68009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:59.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.479+0000 7f9b6effd700 1 -- 172.21.15.189:0/3551110416 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9b70054c90 con 0x7f9b7815d7e0 2024-04-06T13:47:59.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.674+0000 7f9b7c955700 1 -- 172.21.15.189:0/3551110416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b78073f90 con 0x7f9b7815d7e0 2024-04-06T13:47:59.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.675+0000 7f9b6effd700 1 -- 172.21.15.189:0/3551110416 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v233) v1 ==== 74+0+21201 (secure 0 0 0) 0x7f9b700582b0 con 0x7f9b7815d7e0 2024-04-06T13:47:59.676 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:47:59.677 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":233,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:47:57.909364+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:47:20.644275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-04-06T13:47:21.075143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:47:32.292623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:47:33.567052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/3017776380":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1992839362":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/579471530":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/4049616553":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/619264656":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1506469869":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1831656794":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1190635801":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/3920072072":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/1878657401":"2024-04-07T13:47:47.129296+0000","172.21.15.189:6833/111704":"2024-04-07T13:47:47.129296+0000","172.21.15.189:6832/111704":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/2827130089":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/2716712615":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/1807905335":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/914480885":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1173010452":"2024-04-07T13:47:47.129296+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:47:59.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.677+0000 7f9b7c955700 1 -- 172.21.15.189:0/3551110416 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9b5806aae0 msgr2=0x7f9b5806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:59.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.677+0000 7f9b7c955700 1 --2- 172.21.15.189:0/3551110416 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9b5806aae0 0x7f9b5806cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9b781f3d50 tx=0x7f9b68009040 comp rx=0 tx=0).stop 2024-04-06T13:47:59.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.677+0000 7f9b7c955700 1 -- 172.21.15.189:0/3551110416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b7815d7e0 msgr2=0x7f9b780c92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:59.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.677+0000 7f9b7c955700 1 --2- 172.21.15.189:0/3551110416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b7815d7e0 0x7f9b780c92e0 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f9b70007e40 tx=0x7f9b7000e3c0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.677+0000 7f9b7c955700 1 -- 172.21.15.189:0/3551110416 shutdown_connections 2024-04-06T13:47:59.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.677+0000 7f9b7c955700 1 --2- 172.21.15.189:0/3551110416 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9b5806aae0 0x7f9b5806cfa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.677+0000 7f9b7c955700 1 --2- 172.21.15.189:0/3551110416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b7815d7e0 0x7f9b780c92e0 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.677+0000 7f9b7c955700 1 --2- 172.21.15.189:0/3551110416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b780d0a60 0x7f9b780c7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.677+0000 7f9b7c955700 1 --2- 172.21.15.189:0/3551110416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b780d1af0 0x7f9b78159840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.677+0000 7f9b7c955700 1 -- 172.21.15.189:0/3551110416 >> 172.21.15.189:0/3551110416 conn(0x7f9b78072250 msgr2=0x7f9b780cd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:59.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.678+0000 7f9b7c955700 1 -- 172.21.15.189:0/3551110416 shutdown_connections 2024-04-06T13:47:59.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.678+0000 7f9b7c955700 1 -- 172.21.15.189:0/3551110416 wait complete. 2024-04-06T13:47:59.691 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-06T13:47:59.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.813+0000 7fac1c683700 1 Processor -- start 2024-04-06T13:47:59.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.813+0000 7fac1c683700 1 -- start start 2024-04-06T13:47:59.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.814+0000 7fac1c683700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac1415d7b0 0x7fac140b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:59.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.814+0000 7fac1c683700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fac1415e500 0x7fac1406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:59.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.814+0000 7fac1c683700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac140b3990 con 0x7fac1406a730 2024-04-06T13:47:59.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.814+0000 7fac1c683700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac1406b0f0 con 0x7fac1415e500 2024-04-06T13:47:59.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.814+0000 7fac1c683700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac140b3060 con 0x7fac1415d7b0 2024-04-06T13:47:59.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.814+0000 7fac1ac20700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fac1415e500 0x7fac1406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:59.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.814+0000 7fac1ac20700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fac1415e500 0x7fac1406abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41624/0 (socket says 172.21.15.189:41624) 2024-04-06T13:47:59.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.814+0000 7fac1ac20700 1 -- 172.21.15.189:0/1600568745 learned_addr learned my addr 172.21.15.189:0/1600568745 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:59.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.814+0000 7fac1ac20700 1 -- 172.21.15.189:0/1600568745 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac1415d7b0 msgr2=0x7fac140b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:47:59.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.814+0000 7fac1a41f700 1 --2- 172.21.15.189:0/1600568745 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac1415d7b0 0x7fac140b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:59.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.815+0000 7fac1ac20700 1 --2- 172.21.15.189:0/1600568745 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac1415d7b0 0x7fac140b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.815+0000 7fac1ac20700 1 -- 172.21.15.189:0/1600568745 >> v1:172.21.15.189:6789/0 conn(0x7fac1406a730 legacy=0x7fac1415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:59.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.815+0000 7fac1ac20700 1 -- 172.21.15.189:0/1600568745 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac140b3340 con 0x7fac1415e500 2024-04-06T13:47:59.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.815+0000 7fac1a41f700 1 --2- 172.21.15.189:0/1600568745 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac1415d7b0 0x7fac140b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:47:59.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.815+0000 7fac1ac20700 1 --2- 172.21.15.189:0/1600568745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac1415e500 0x7fac1406abb0 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7fac0c009e50 tx=0x7fac0c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee908e06ee83639 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:59.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.815+0000 7fac1941d700 1 -- 172.21.15.189:0/1600568745 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c00e070 con 0x7fac1415e500 2024-04-06T13:47:59.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.815+0000 7fac1941d700 1 -- 172.21.15.189:0/1600568745 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0c00b9c0 con 0x7fac1415e500 2024-04-06T13:47:59.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.815+0000 7fac1941d700 1 -- 172.21.15.189:0/1600568745 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c00bd70 con 0x7fac1415e500 2024-04-06T13:47:59.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.815+0000 7fac1c683700 1 -- 172.21.15.189:0/1600568745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac1415e500 msgr2=0x7fac1406abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:59.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.815+0000 7fac1c683700 1 --2- 172.21.15.189:0/1600568745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac1415e500 0x7fac1406abb0 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7fac0c009e50 tx=0x7fac0c00b330 comp rx=0 tx=0).stop 2024-04-06T13:47:59.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.815+0000 7fac1c683700 1 -- 172.21.15.189:0/1600568745 shutdown_connections 2024-04-06T13:47:59.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.815+0000 7fac1c683700 1 --2- 172.21.15.189:0/1600568745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac1415e500 0x7fac1406abb0 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.815+0000 7fac1c683700 1 --2- 172.21.15.189:0/1600568745 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac1415d7b0 0x7fac140b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.815+0000 7fac1c683700 1 -- 172.21.15.189:0/1600568745 >> 172.21.15.189:0/1600568745 conn(0x7fac140d00b0 msgr2=0x7fac140d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:47:59.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.816+0000 7fac1c683700 1 -- 172.21.15.189:0/1600568745 shutdown_connections 2024-04-06T13:47:59.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.816+0000 7fac1c683700 1 -- 172.21.15.189:0/1600568745 wait complete. 2024-04-06T13:47:59.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.817+0000 7fac1c683700 1 Processor -- start 2024-04-06T13:47:59.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.817+0000 7fac1c683700 1 -- start start 2024-04-06T13:47:59.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.817+0000 7fac1c683700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac1406a730 0x7fac1415eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:59.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.817+0000 7fac1c683700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac1415d7b0 0x7fac140ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:59.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.817+0000 7fac1c683700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac1415e500 0x7fac140cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:59.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.817+0000 7fac1c683700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac140ce480 con 0x7fac1415d7b0 2024-04-06T13:47:59.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.817+0000 7fac1c683700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac14150f20 con 0x7fac1415e500 2024-04-06T13:47:59.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.817+0000 7fac1c683700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac141501b0 con 0x7fac1406a730 2024-04-06T13:47:59.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.818+0000 7fac19c1e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac1415d7b0 0x7fac140ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:59.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.818+0000 7fac19c1e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac1415d7b0 0x7fac140ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41632/0 (socket says 172.21.15.189:41632) 2024-04-06T13:47:59.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.818+0000 7fac19c1e700 1 -- 172.21.15.189:0/1085380447 learned_addr learned my addr 172.21.15.189:0/1085380447 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:47:59.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.818+0000 7fac1ac20700 1 --2- 172.21.15.189:0/1085380447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac1415e500 0x7fac140cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:59.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.818+0000 7fac1a41f700 1 --2- 172.21.15.189:0/1085380447 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac1406a730 0x7fac1415eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:59.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.818+0000 7fac19c1e700 1 -- 172.21.15.189:0/1085380447 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac1406a730 msgr2=0x7fac1415eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:59.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.818+0000 7fac19c1e700 1 --2- 172.21.15.189:0/1085380447 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac1406a730 0x7fac1415eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.818+0000 7fac19c1e700 1 -- 172.21.15.189:0/1085380447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac1415e500 msgr2=0x7fac140cd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:47:59.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.818+0000 7fac19c1e700 1 --2- 172.21.15.189:0/1085380447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac1415e500 0x7fac140cd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:47:59.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.818+0000 7fac19c1e700 1 -- 172.21.15.189:0/1085380447 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c009b30 con 0x7fac1415d7b0 2024-04-06T13:47:59.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.818+0000 7fac1ac20700 1 --2- 172.21.15.189:0/1085380447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac1415e500 0x7fac140cd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:47:59.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.818+0000 7fac19c1e700 1 --2- 172.21.15.189:0/1085380447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac1415d7b0 0x7fac140ccb10 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7fac1000c4e0 tx=0x7fac1000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:59.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.818+0000 7fac0affd700 1 -- 172.21.15.189:0/1085380447 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac100052d0 con 0x7fac1415d7b0 2024-04-06T13:47:59.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.818+0000 7fac0affd700 1 -- 172.21.15.189:0/1085380447 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac10005430 con 0x7fac1415d7b0 2024-04-06T13:47:59.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.818+0000 7fac1c683700 1 -- 172.21.15.189:0/1085380447 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac1415f430 con 0x7fac1415d7b0 2024-04-06T13:47:59.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.818+0000 7fac0affd700 1 -- 172.21.15.189:0/1085380447 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac100038c0 con 0x7fac1415d7b0 2024-04-06T13:47:59.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.818+0000 7fac1c683700 1 -- 172.21.15.189:0/1085380447 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1415a530 con 0x7fac1415d7b0 2024-04-06T13:47:59.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.819+0000 7fac1c683700 1 -- 172.21.15.189:0/1085380447 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac14067bb0 con 0x7fac1415d7b0 2024-04-06T13:47:59.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.820+0000 7fac0affd700 1 -- 172.21.15.189:0/1085380447 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 148) v1 ==== 88983+0+0 (secure 0 0 0) 0x7fac10003a20 con 0x7fac1415d7b0 2024-04-06T13:47:59.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.820+0000 7fac0affd700 1 --2- 172.21.15.189:0/1085380447 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fabf806aae0 0x7fabf806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:47:59.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.820+0000 7fac0affd700 1 -- 172.21.15.189:0/1085380447 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7696+0+0 (secure 0 0 0) 0x7fac10088bc0 con 0x7fac1415d7b0 2024-04-06T13:47:59.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.820+0000 7fac1a41f700 1 --2- 172.21.15.189:0/1085380447 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fabf806aae0 0x7fabf806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:47:59.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.820+0000 7fac1a41f700 1 --2- 172.21.15.189:0/1085380447 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fabf806aae0 0x7fabf806cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fac04006fd0 tx=0x7fac04008040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:47:59.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:47:59.824+0000 7fac0affd700 1 -- 172.21.15.189:0/1085380447 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fac10053920 con 0x7fac1415d7b0 2024-04-06T13:48:00.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.071+0000 7fac1c683700 1 -- 172.21.15.189:0/1085380447 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fac14074990 con 0x7fac1415d7b0 2024-04-06T13:48:00.072 INFO:tasks.ceph.mon.a.smithi189.stderr:2024-04-06T13:48:00.072+0000 7fa9fe1ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T13:48:00.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.084+0000 7fac0affd700 1 -- 172.21.15.189:0/1085380447 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v107) v1 ==== 161+0+0 (secure 0 0 0) 0x7fac10056f40 con 0x7fac1415d7b0 2024-04-06T13:48:00.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.086+0000 7fac1c683700 1 -- 172.21.15.189:0/1085380447 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fabf806aae0 msgr2=0x7fabf806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:00.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.086+0000 7fac1c683700 1 --2- 172.21.15.189:0/1085380447 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fabf806aae0 0x7fabf806cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fac04006fd0 tx=0x7fac04008040 comp rx=0 tx=0).stop 2024-04-06T13:48:00.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.086+0000 7fac1c683700 1 -- 172.21.15.189:0/1085380447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac1415d7b0 msgr2=0x7fac140ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:00.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.086+0000 7fac1c683700 1 --2- 172.21.15.189:0/1085380447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac1415d7b0 0x7fac140ccb10 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7fac1000c4e0 tx=0x7fac1000c8a0 comp rx=0 tx=0).stop 2024-04-06T13:48:00.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.086+0000 7fac1c683700 1 -- 172.21.15.189:0/1085380447 shutdown_connections 2024-04-06T13:48:00.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.086+0000 7fac1c683700 1 --2- 172.21.15.189:0/1085380447 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fabf806aae0 0x7fabf806cfa0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:00.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.086+0000 7fac1c683700 1 --2- 172.21.15.189:0/1085380447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac1415e500 0x7fac140cd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:00.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.086+0000 7fac1c683700 1 --2- 172.21.15.189:0/1085380447 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac1406a730 0x7fac1415eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:00.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.086+0000 7fac1c683700 1 --2- 172.21.15.189:0/1085380447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac1415d7b0 0x7fac140ccb10 unknown :-1 s=CLOSED pgs=2581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:00.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.086+0000 7fac1c683700 1 -- 172.21.15.189:0/1085380447 >> 172.21.15.189:0/1085380447 conn(0x7fac140d00b0 msgr2=0x7fac140739d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:00.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.086+0000 7fac1c683700 1 -- 172.21.15.189:0/1085380447 shutdown_connections 2024-04-06T13:48:00.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.086+0000 7fac1c683700 1 -- 172.21.15.189:0/1085380447 wait complete. 2024-04-06T13:48:00.088 INFO:teuthology.orchestra.run.smithi189.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-06T13:48:00.088 INFO:tasks.ceph.mds.b.smithi192.stderr: -5663> 2024-04-06T13:47:23.648+0000 7f030e357700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T13:48:00.099 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-06T13:48:00.120 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T13:48:00.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.224+0000 7f9383a3d700 1 Processor -- start 2024-04-06T13:48:00.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.224+0000 7f9383a3d700 1 -- start start 2024-04-06T13:48:00.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.225+0000 7f9383a3d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f937c0cd3e0 0x7f937c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:00.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.225+0000 7f9383a3d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f937c06b8c0 0x7f937c06dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:00.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.225+0000 7f9383a3d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937c06ceb0 con 0x7f937c15d7c0 2024-04-06T13:48:00.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.225+0000 7f9383a3d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f937c0cdfa0 con 0x7f937c06b8c0 2024-04-06T13:48:00.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.225+0000 7f9383a3d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f937c06d750 con 0x7f937c0cd3e0 2024-04-06T13:48:00.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.225+0000 7f9381fda700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f937c06b8c0 0x7f937c06dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:00.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.225+0000 7f9381fda700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f937c06b8c0 0x7f937c06dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41654/0 (socket says 172.21.15.189:41654) 2024-04-06T13:48:00.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.225+0000 7f9381fda700 1 -- 172.21.15.189:0/4151301161 learned_addr learned my addr 172.21.15.189:0/4151301161 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:00.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.225+0000 7f9380fd8700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f937c15d7c0 0x7f937c06b1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50632/0 (socket says 172.21.15.189:50632) 2024-04-06T13:48:00.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.225+0000 7f93817d9700 1 --2- 172.21.15.189:0/4151301161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f937c0cd3e0 0x7f937c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:00.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.225+0000 7f9381fda700 1 -- 172.21.15.189:0/4151301161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f937c0cd3e0 msgr2=0x7f937c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:00.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.225+0000 7f9381fda700 1 --2- 172.21.15.189:0/4151301161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f937c0cd3e0 0x7f937c0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:00.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.225+0000 7f9381fda700 1 -- 172.21.15.189:0/4151301161 >> v1:172.21.15.189:6789/0 conn(0x7f937c15d7c0 legacy=0x7f937c06b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:00.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.225+0000 7f9373fff700 1 -- 172.21.15.189:0/4151301161 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84941933 0 0) 0x7f937c06ceb0 con 0x7f937c15d7c0 2024-04-06T13:48:00.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.225+0000 7f9381fda700 1 -- 172.21.15.189:0/4151301161 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937c0cdd00 con 0x7f937c06b8c0 2024-04-06T13:48:00.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.225+0000 7f93817d9700 1 --2- 172.21.15.189:0/4151301161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f937c0cd3e0 0x7f937c0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:00.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.225+0000 7f9381fda700 1 --2- 172.21.15.189:0/4151301161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f937c06b8c0 0x7f937c06dbe0 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f9374009e50 tx=0x7f937400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f372ba361490216 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:00.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.225+0000 7f9373fff700 1 -- 172.21.15.189:0/4151301161 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937400e070 con 0x7f937c06b8c0 2024-04-06T13:48:00.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.226+0000 7f9373fff700 1 -- 172.21.15.189:0/4151301161 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f937400b9c0 con 0x7f937c06b8c0 2024-04-06T13:48:00.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.226+0000 7f9373fff700 1 -- 172.21.15.189:0/4151301161 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937400bd70 con 0x7f937c06b8c0 2024-04-06T13:48:00.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.226+0000 7f9383a3d700 1 -- 172.21.15.189:0/4151301161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f937c06b8c0 msgr2=0x7f937c06dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:00.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.226+0000 7f9383a3d700 1 --2- 172.21.15.189:0/4151301161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f937c06b8c0 0x7f937c06dbe0 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f9374009e50 tx=0x7f937400b330 comp rx=0 tx=0).stop 2024-04-06T13:48:00.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.226+0000 7f9383a3d700 1 -- 172.21.15.189:0/4151301161 shutdown_connections 2024-04-06T13:48:00.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.226+0000 7f9383a3d700 1 --2- 172.21.15.189:0/4151301161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f937c06b8c0 0x7f937c06dbe0 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:00.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.226+0000 7f9383a3d700 1 --2- 172.21.15.189:0/4151301161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f937c0cd3e0 0x7f937c0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:00.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.226+0000 7f9383a3d700 1 -- 172.21.15.189:0/4151301161 >> 172.21.15.189:0/4151301161 conn(0x7f937c0cc290 msgr2=0x7f937c0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:00.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.226+0000 7f9383a3d700 1 -- 172.21.15.189:0/4151301161 shutdown_connections 2024-04-06T13:48:00.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.226+0000 7f9383a3d700 1 -- 172.21.15.189:0/4151301161 wait complete. 2024-04-06T13:48:00.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.227+0000 7f9383a3d700 1 Processor -- start 2024-04-06T13:48:00.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.227+0000 7f9383a3d700 1 -- start start 2024-04-06T13:48:00.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.228+0000 7f9383a3d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f937c06b8c0 0x7f937c0c5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:00.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.228+0000 7f9383a3d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f937c0cd3e0 0x7f937c15be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:00.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.228+0000 7f9383a3d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f937c15d7c0 0x7f937c15ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:00.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.228+0000 7f9383a3d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937c15c950 con 0x7f937c0cd3e0 2024-04-06T13:48:00.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.228+0000 7f9383a3d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937c15cc30 con 0x7f937c15d7c0 2024-04-06T13:48:00.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.228+0000 7f9383a3d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f937c0b3930 con 0x7f937c06b8c0 2024-04-06T13:48:00.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.228+0000 7f9380fd8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f937c0cd3e0 0x7f937c15be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:00.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.228+0000 7f9380fd8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f937c0cd3e0 0x7f937c15be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41668/0 (socket says 172.21.15.189:41668) 2024-04-06T13:48:00.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.228+0000 7f9380fd8700 1 -- 172.21.15.189:0/473288561 learned_addr learned my addr 172.21.15.189:0/473288561 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:00.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.228+0000 7f9381fda700 1 --2- 172.21.15.189:0/473288561 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f937c15d7c0 0x7f937c15ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:00.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.228+0000 7f93817d9700 1 --2- 172.21.15.189:0/473288561 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f937c06b8c0 0x7f937c0c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:00.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.228+0000 7f9380fd8700 1 -- 172.21.15.189:0/473288561 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f937c06b8c0 msgr2=0x7f937c0c5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:00.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.228+0000 7f9380fd8700 1 --2- 172.21.15.189:0/473288561 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f937c06b8c0 0x7f937c0c5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:00.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.228+0000 7f9380fd8700 1 -- 172.21.15.189:0/473288561 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f937c15d7c0 msgr2=0x7f937c15ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:00.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.228+0000 7f9380fd8700 1 --2- 172.21.15.189:0/473288561 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f937c15d7c0 0x7f937c15ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:00.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.228+0000 7f9380fd8700 1 -- 172.21.15.189:0/473288561 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9374009b30 con 0x7f937c0cd3e0 2024-04-06T13:48:00.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.229+0000 7f9381fda700 1 --2- 172.21.15.189:0/473288561 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f937c15d7c0 0x7f937c15ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:00.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.229+0000 7f9380fd8700 1 --2- 172.21.15.189:0/473288561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f937c0cd3e0 0x7f937c15be90 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f936c00c520 tx=0x7f936c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:00.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.229+0000 7f9371ffb700 1 -- 172.21.15.189:0/473288561 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c004c80 con 0x7f937c0cd3e0 2024-04-06T13:48:00.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.229+0000 7f9371ffb700 1 -- 172.21.15.189:0/473288561 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936c004de0 con 0x7f937c0cd3e0 2024-04-06T13:48:00.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.229+0000 7f9383a3d700 1 -- 172.21.15.189:0/473288561 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f937c0b3c10 con 0x7f937c0cd3e0 2024-04-06T13:48:00.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.229+0000 7f9371ffb700 1 -- 172.21.15.189:0/473288561 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c0072d0 con 0x7f937c0cd3e0 2024-04-06T13:48:00.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.229+0000 7f9383a3d700 1 -- 172.21.15.189:0/473288561 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f937c1595e0 con 0x7f937c0cd3e0 2024-04-06T13:48:00.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.230+0000 7f9383a3d700 1 -- 172.21.15.189:0/473288561 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f937c0b3790 con 0x7f937c0cd3e0 2024-04-06T13:48:00.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.230+0000 7f9371ffb700 1 -- 172.21.15.189:0/473288561 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 148) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f936c007430 con 0x7f937c0cd3e0 2024-04-06T13:48:00.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.231+0000 7f9371ffb700 1 --2- 172.21.15.189:0/473288561 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f936006aae0 0x7f936006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:00.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.231+0000 7f9371ffb700 1 -- 172.21.15.189:0/473288561 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7782+0+0 (secure 0 0 0) 0x7f936c089490 con 0x7f937c0cd3e0 2024-04-06T13:48:00.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.231+0000 7f93817d9700 1 --2- 172.21.15.189:0/473288561 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f936006aae0 0x7f936006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:00.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.231+0000 7f93817d9700 1 --2- 172.21.15.189:0/473288561 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f936006aae0 0x7f936006cfa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9378006fd0 tx=0x7f9378008040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:00.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.235+0000 7f9371ffb700 1 -- 172.21.15.189:0/473288561 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f937c0b3790 con 0x7f937c0cd3e0 2024-04-06T13:48:00.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.473+0000 7f9383a3d700 1 -- 172.21.15.189:0/473288561 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f937c0b3790 con 0x7f937c0cd3e0 2024-04-06T13:48:00.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:00.922+0000 7f9371ffb700 1 -- 172.21.15.189:0/473288561 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 149) v1 ==== 89293+0+0 (secure 0 0 0) 0x7f936c054560 con 0x7f937c0cd3e0 2024-04-06T13:48:01.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.085+0000 7f9371ffb700 1 -- 172.21.15.189:0/473288561 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v108) v1 ==== 104+0+0 (secure 0 0 0) 0x7f936c0737d0 con 0x7f937c0cd3e0 2024-04-06T13:48:01.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.087+0000 7f9383a3d700 1 -- 172.21.15.189:0/473288561 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f936006aae0 msgr2=0x7f936006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:01.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.087+0000 7f9383a3d700 1 --2- 172.21.15.189:0/473288561 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f936006aae0 0x7f936006cfa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9378006fd0 tx=0x7f9378008040 comp rx=0 tx=0).stop 2024-04-06T13:48:01.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.087+0000 7f9383a3d700 1 -- 172.21.15.189:0/473288561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f937c0cd3e0 msgr2=0x7f937c15be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:01.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.087+0000 7f9383a3d700 1 --2- 172.21.15.189:0/473288561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f937c0cd3e0 0x7f937c15be90 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f936c00c520 tx=0x7f936c00c8e0 comp rx=0 tx=0).stop 2024-04-06T13:48:01.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.087+0000 7f9383a3d700 1 -- 172.21.15.189:0/473288561 shutdown_connections 2024-04-06T13:48:01.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.087+0000 7f9383a3d700 1 --2- 172.21.15.189:0/473288561 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f936006aae0 0x7f936006cfa0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:01.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.087+0000 7f9383a3d700 1 --2- 172.21.15.189:0/473288561 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f937c15d7c0 0x7f937c15ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:01.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.087+0000 7f9383a3d700 1 --2- 172.21.15.189:0/473288561 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f937c06b8c0 0x7f937c0c5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:01.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.087+0000 7f9383a3d700 1 --2- 172.21.15.189:0/473288561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f937c0cd3e0 0x7f937c15be90 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:01.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.087+0000 7f9383a3d700 1 -- 172.21.15.189:0/473288561 >> 172.21.15.189:0/473288561 conn(0x7f937c0cc290 msgr2=0x7f937c0af2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:01.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.087+0000 7f9383a3d700 1 -- 172.21.15.189:0/473288561 shutdown_connections 2024-04-06T13:48:01.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.087+0000 7f9383a3d700 1 -- 172.21.15.189:0/473288561 wait complete. 2024-04-06T13:48:01.101 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-06T13:48:01.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.227+0000 7f8573502700 1 Processor -- start 2024-04-06T13:48:01.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.227+0000 7f8573502700 1 -- start start 2024-04-06T13:48:01.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f8573502700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f856c0b4af0 0x7f856c0b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:01.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f8573502700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f856c0cdbe0 con 0x7f856c15a550 2024-04-06T13:48:01.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f8573502700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f856c15a940 con 0x7f856c0b4660 2024-04-06T13:48:01.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f8573502700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f856c15ac00 con 0x7f856c0b4af0 2024-04-06T13:48:01.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f857129e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f856c15a550 0x7f856c0cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50648/0 (socket says 172.21.15.189:50648) 2024-04-06T13:48:01.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f857129e700 1 -- 172.21.15.189:0/648251286 learned_addr learned my addr 172.21.15.189:0/648251286 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:01.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f8563fff700 1 -- 172.21.15.189:0/648251286 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728791172 0 0) 0x7f856c0cdbe0 con 0x7f856c15a550 2024-04-06T13:48:01.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f8563fff700 1 -- 172.21.15.189:0/648251286 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85580035b0 con 0x7f856c15a550 2024-04-06T13:48:01.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f8571a9f700 1 --2- 172.21.15.189:0/648251286 >> v2:172.21.15.192:3300/0 conn(0x7f856c0b4af0 0x7f856c0b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:01.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f8563fff700 1 -- 172.21.15.189:0/648251286 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 84871017 0 0) 0x7f85580035b0 con 0x7f856c15a550 2024-04-06T13:48:01.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f8563fff700 1 -- 172.21.15.189:0/648251286 >> v2:172.21.15.192:3300/0 conn(0x7f856c0b4af0 msgr2=0x7f856c0b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:01.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f8563fff700 1 --2- 172.21.15.189:0/648251286 >> v2:172.21.15.192:3300/0 conn(0x7f856c0b4af0 0x7f856c0b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:01.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f8563fff700 1 -- 172.21.15.189:0/648251286 >> v1:172.21.15.192:6789/0 conn(0x7f856c0b4660 legacy=0x7f856c0b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:01.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f8563fff700 1 -- 172.21.15.189:0/648251286 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c15aee0 con 0x7f856c15a550 2024-04-06T13:48:01.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f8563fff700 1 -- 172.21.15.189:0/648251286 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f8568001670 con 0x7f856c15a550 2024-04-06T13:48:01.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f8571a9f700 1 --2- 172.21.15.189:0/648251286 >> v2:172.21.15.192:3300/0 conn(0x7f856c0b4af0 0x7f856c0b36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:01.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f8563fff700 1 -- 172.21.15.189:0/648251286 >> v1:172.21.15.189:6789/0 conn(0x7f856c15a550 legacy=0x7f856c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:01.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f8563fff700 1 --2- 172.21.15.189:0/648251286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8558003a00 0x7f8558003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:01.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f8563fff700 1 --2- 172.21.15.189:0/648251286 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8558004850 0x7f8558004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:01.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.228+0000 7f8563fff700 1 --2- 172.21.15.189:0/648251286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8558005210 0x7f855800dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:01.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.229+0000 7f8563fff700 1 -- 172.21.15.189:0/648251286 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c15ac00 con 0x7f8558005210 2024-04-06T13:48:01.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.229+0000 7f8563fff700 1 -- 172.21.15.189:0/648251286 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855800e370 con 0x7f8558003a00 2024-04-06T13:48:01.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.229+0000 7f8563fff700 1 -- 172.21.15.189:0/648251286 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f855800e530 con 0x7f8558004850 2024-04-06T13:48:01.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.229+0000 7f857129e700 1 --2- 172.21.15.189:0/648251286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8558003a00 0x7f8558003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:01.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.229+0000 7f8571a9f700 1 --2- 172.21.15.189:0/648251286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8558005210 0x7f855800dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:01.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.229+0000 7f8570a9d700 1 --2- 172.21.15.189:0/648251286 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8558004850 0x7f8558004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:01.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.229+0000 7f8571a9f700 1 -- 172.21.15.189:0/648251286 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8558004850 msgr2=0x7f8558004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:01.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.229+0000 7f8571a9f700 1 --2- 172.21.15.189:0/648251286 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8558004850 0x7f8558004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:01.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.229+0000 7f8571a9f700 1 -- 172.21.15.189:0/648251286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8558003a00 msgr2=0x7f8558003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:01.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.229+0000 7f8571a9f700 1 --2- 172.21.15.189:0/648251286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8558003a00 0x7f8558003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:01.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.229+0000 7f8571a9f700 1 -- 172.21.15.189:0/648251286 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85580035b0 con 0x7f8558005210 2024-04-06T13:48:01.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.229+0000 7f8570a9d700 1 --2- 172.21.15.189:0/648251286 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8558004850 0x7f8558004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:01.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.229+0000 7f8571a9f700 1 -- 172.21.15.189:0/648251286 reap_dead start 2024-04-06T13:48:01.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.229+0000 7f8571a9f700 1 --2- 172.21.15.189:0/648251286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8558005210 0x7f855800dc20 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f85640073e0 tx=0x7f8558005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=860423f9db9d67a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:01.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.229+0000 7f8563fff700 1 -- 172.21.15.189:0/648251286 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8564009490 con 0x7f8558005210 2024-04-06T13:48:01.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.229+0000 7f8563fff700 1 -- 172.21.15.189:0/648251286 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8564014400 con 0x7f8558005210 2024-04-06T13:48:01.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.229+0000 7f8563fff700 1 -- 172.21.15.189:0/648251286 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85640147b0 con 0x7f8558005210 2024-04-06T13:48:01.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.230+0000 7f8573502700 1 -- 172.21.15.189:0/648251286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8558005210 msgr2=0x7f855800dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:01.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.230+0000 7f8573502700 1 --2- 172.21.15.189:0/648251286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8558005210 0x7f855800dc20 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f85640073e0 tx=0x7f8558005880 comp rx=0 tx=0).stop 2024-04-06T13:48:01.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.230+0000 7f8573502700 1 -- 172.21.15.189:0/648251286 shutdown_connections 2024-04-06T13:48:01.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.230+0000 7f8573502700 1 --2- 172.21.15.189:0/648251286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8558005210 0x7f855800dc20 unknown :-1 s=CLOSED pgs=2588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:01.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.230+0000 7f8573502700 1 -- 172.21.15.189:0/648251286 >> 172.21.15.189:0/648251286 conn(0x7f856c0d00b0 msgr2=0x7f856c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:01.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.230+0000 7f8573502700 1 -- 172.21.15.189:0/648251286 shutdown_connections 2024-04-06T13:48:01.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.230+0000 7f8573502700 1 -- 172.21.15.189:0/648251286 wait complete. 2024-04-06T13:48:01.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.231+0000 7f8573502700 1 Processor -- start 2024-04-06T13:48:01.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.231+0000 7f8573502700 1 -- start start 2024-04-06T13:48:01.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.232+0000 7f8573502700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8558005210 0x7f856c0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:01.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.232+0000 7f8573502700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f856c155070 0x7f856c14b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:01.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.232+0000 7f8573502700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f856c0adf80 0x7f856c0ae410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:01.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.232+0000 7f8573502700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c1f64e0 con 0x7f8558005210 2024-04-06T13:48:01.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.232+0000 7f8573502700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0b0170 con 0x7f856c0adf80 2024-04-06T13:48:01.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.232+0000 7f8573502700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0b0450 con 0x7f856c155070 2024-04-06T13:48:01.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.232+0000 7f857129e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8558005210 0x7f856c0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:01.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.232+0000 7f857129e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8558005210 0x7f856c0afb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41682/0 (socket says 172.21.15.189:41682) 2024-04-06T13:48:01.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.232+0000 7f857129e700 1 -- 172.21.15.189:0/2288722172 learned_addr learned my addr 172.21.15.189:0/2288722172 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:01.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.232+0000 7f8570a9d700 1 --2- 172.21.15.189:0/2288722172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f856c155070 0x7f856c14b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:01.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.232+0000 7f8571a9f700 1 --2- 172.21.15.189:0/2288722172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f856c0adf80 0x7f856c0ae410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:01.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.232+0000 7f857129e700 1 -- 172.21.15.189:0/2288722172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f856c155070 msgr2=0x7f856c14b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:01.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.232+0000 7f857129e700 1 --2- 172.21.15.189:0/2288722172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f856c155070 0x7f856c14b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:01.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.232+0000 7f857129e700 1 -- 172.21.15.189:0/2288722172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f856c0adf80 msgr2=0x7f856c0ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:01.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.232+0000 7f857129e700 1 --2- 172.21.15.189:0/2288722172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f856c0adf80 0x7f856c0ae410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:01.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.232+0000 7f857129e700 1 -- 172.21.15.189:0/2288722172 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c0b0730 con 0x7f8558005210 2024-04-06T13:48:01.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.232+0000 7f857129e700 1 --2- 172.21.15.189:0/2288722172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8558005210 0x7f856c0afb90 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f85680072c0 tx=0x7f85680075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:01.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.232+0000 7f8561ffb700 1 -- 172.21.15.189:0/2288722172 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8568011580 con 0x7f8558005210 2024-04-06T13:48:01.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.232+0000 7f8561ffb700 1 -- 172.21.15.189:0/2288722172 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8568011700 con 0x7f8558005210 2024-04-06T13:48:01.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.233+0000 7f8573502700 1 -- 172.21.15.189:0/2288722172 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856c0b0a10 con 0x7f8558005210 2024-04-06T13:48:01.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.233+0000 7f8561ffb700 1 -- 172.21.15.189:0/2288722172 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8568011ab0 con 0x7f8558005210 2024-04-06T13:48:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.233+0000 7f8573502700 1 -- 172.21.15.189:0/2288722172 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f856c1554f0 con 0x7f8558005210 2024-04-06T13:48:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.234+0000 7f8573502700 1 -- 172.21.15.189:0/2288722172 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856c067bb0 con 0x7f8558005210 2024-04-06T13:48:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.234+0000 7f8561ffb700 1 -- 172.21.15.189:0/2288722172 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 149) v1 ==== 89293+0+0 (secure 0 0 0) 0x7f8568011c10 con 0x7f8558005210 2024-04-06T13:48:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.234+0000 7f8561ffb700 1 --2- 172.21.15.189:0/2288722172 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f855006ae40 0x7f855006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.235+0000 7f8570a9d700 1 --2- 172.21.15.189:0/2288722172 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f855006ae40 0x7f855006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.235+0000 7f8561ffb700 1 -- 172.21.15.189:0/2288722172 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7782+0+0 (secure 0 0 0) 0x7f8568089cb0 con 0x7f8558005210 2024-04-06T13:48:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.235+0000 7f8570a9d700 1 --2- 172.21.15.189:0/2288722172 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f855006ae40 0x7f855006d300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f855c000c00 tx=0x7f855c009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.239+0000 7f8561ffb700 1 -- 172.21.15.189:0/2288722172 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8568054480 con 0x7f8558005210 2024-04-06T13:48:01.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:01.434+0000 7f8573502700 1 -- 172.21.15.189:0/2288722172 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f856c074990 con 0x7f8558005210 2024-04-06T13:48:02.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.200+0000 7f8561ffb700 1 -- 172.21.15.189:0/2288722172 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f8568054850 con 0x7f8558005210 2024-04-06T13:48:02.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.202+0000 7f8561ffb700 1 -- 172.21.15.189:0/2288722172 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v235) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8568086b20 con 0x7f8558005210 2024-04-06T13:48:02.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.205+0000 7f8573502700 1 -- 172.21.15.189:0/2288722172 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f855006ae40 msgr2=0x7f855006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:02.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.205+0000 7f8573502700 1 --2- 172.21.15.189:0/2288722172 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f855006ae40 0x7f855006d300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f855c000c00 tx=0x7f855c009040 comp rx=0 tx=0).stop 2024-04-06T13:48:02.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.205+0000 7f8573502700 1 -- 172.21.15.189:0/2288722172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8558005210 msgr2=0x7f856c0afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:02.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.205+0000 7f8573502700 1 --2- 172.21.15.189:0/2288722172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8558005210 0x7f856c0afb90 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f85680072c0 tx=0x7f85680075d0 comp rx=0 tx=0).stop 2024-04-06T13:48:02.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.205+0000 7f8573502700 1 -- 172.21.15.189:0/2288722172 shutdown_connections 2024-04-06T13:48:02.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.205+0000 7f8573502700 1 --2- 172.21.15.189:0/2288722172 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f855006ae40 0x7f855006d300 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:02.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.205+0000 7f8573502700 1 --2- 172.21.15.189:0/2288722172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f856c0adf80 0x7f856c0ae410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:02.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.205+0000 7f8573502700 1 --2- 172.21.15.189:0/2288722172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8558005210 0x7f856c0afb90 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:02.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.205+0000 7f8573502700 1 --2- 172.21.15.189:0/2288722172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f856c155070 0x7f856c14b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:02.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.205+0000 7f8573502700 1 -- 172.21.15.189:0/2288722172 >> 172.21.15.189:0/2288722172 conn(0x7f856c0d00b0 msgr2=0x7f856c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:02.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.205+0000 7f8573502700 1 -- 172.21.15.189:0/2288722172 shutdown_connections 2024-04-06T13:48:02.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.206+0000 7f8573502700 1 -- 172.21.15.189:0/2288722172 wait complete. 2024-04-06T13:48:02.207 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' removed 2024-04-06T13:48:02.221 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-06T13:48:02.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.351+0000 7fa2eace5700 1 Processor -- start 2024-04-06T13:48:02.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.351+0000 7fa2eace5700 1 -- start start 2024-04-06T13:48:02.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2eace5700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa2e40b4af0 0x7fa2e40b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:02.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2eace5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e40cdbe0 con 0x7fa2e40b4660 2024-04-06T13:48:02.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2eace5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e415a940 con 0x7fa2e415a550 2024-04-06T13:48:02.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2eace5700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e415ac00 con 0x7fa2e40b4af0 2024-04-06T13:48:02.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e3fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa2e40b4660 0x7fa2e40b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50664/0 (socket says 172.21.15.189:50664) 2024-04-06T13:48:02.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e3fff700 1 -- 172.21.15.189:0/2775563274 learned_addr learned my addr 172.21.15.189:0/2775563274 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:02.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e9282700 1 --2- 172.21.15.189:0/2775563274 >> v2:172.21.15.192:3300/0 conn(0x7fa2e40b4af0 0x7fa2e40b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:02.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e37fe700 1 -- 172.21.15.189:0/2775563274 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534967590 0 0) 0x7fa2e40cdbe0 con 0x7fa2e40b4660 2024-04-06T13:48:02.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e37fe700 1 -- 172.21.15.189:0/2775563274 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2d40035b0 con 0x7fa2e40b4660 2024-04-06T13:48:02.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e37fe700 1 -- 172.21.15.189:0/2775563274 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2362016479 0 0) 0x7fa2d40035b0 con 0x7fa2e40b4660 2024-04-06T13:48:02.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e37fe700 1 -- 172.21.15.189:0/2775563274 >> v2:172.21.15.192:3300/0 conn(0x7fa2e40b4af0 msgr2=0x7fa2e40b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:02.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e37fe700 1 --2- 172.21.15.189:0/2775563274 >> v2:172.21.15.192:3300/0 conn(0x7fa2e40b4af0 0x7fa2e40b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:02.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e37fe700 1 -- 172.21.15.189:0/2775563274 >> v1:172.21.15.192:6789/0 conn(0x7fa2e415a550 legacy=0x7fa2e40cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:02.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e37fe700 1 -- 172.21.15.189:0/2775563274 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e415aee0 con 0x7fa2e40b4660 2024-04-06T13:48:02.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e37fe700 1 -- 172.21.15.189:0/2775563274 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fa2d0001670 con 0x7fa2e40b4660 2024-04-06T13:48:02.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e37fe700 1 -- 172.21.15.189:0/2775563274 >> v1:172.21.15.189:6789/0 conn(0x7fa2e40b4660 legacy=0x7fa2e40b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:02.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e37fe700 1 --2- 172.21.15.189:0/2775563274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2d4003a00 0x7fa2d4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:02.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e37fe700 1 --2- 172.21.15.189:0/2775563274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d4004400 0x7fa2d40048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:02.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e37fe700 1 --2- 172.21.15.189:0/2775563274 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d40050d0 0x7fa2d400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:02.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e37fe700 1 -- 172.21.15.189:0/2775563274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e415ac00 con 0x7fa2d4004400 2024-04-06T13:48:02.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e37fe700 1 -- 172.21.15.189:0/2775563274 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d400c030 con 0x7fa2d4003a00 2024-04-06T13:48:02.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e37fe700 1 -- 172.21.15.189:0/2775563274 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d400c1f0 con 0x7fa2d40050d0 2024-04-06T13:48:02.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e9282700 1 --2- 172.21.15.189:0/2775563274 >> v2:172.21.15.192:3300/0 conn(0x7fa2e40b4af0 0x7fa2e40b36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:02.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.352+0000 7fa2e3fff700 1 --2- 172.21.15.189:0/2775563274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d4004400 0x7fa2d40048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:02.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.353+0000 7fa2e8a81700 1 --2- 172.21.15.189:0/2775563274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2d4003a00 0x7fa2d4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:02.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.353+0000 7fa2e9282700 1 --2- 172.21.15.189:0/2775563274 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d40050d0 0x7fa2d400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:02.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.353+0000 7fa2e3fff700 1 -- 172.21.15.189:0/2775563274 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d40050d0 msgr2=0x7fa2d400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:02.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.353+0000 7fa2e3fff700 1 --2- 172.21.15.189:0/2775563274 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d40050d0 0x7fa2d400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:02.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.353+0000 7fa2e3fff700 1 -- 172.21.15.189:0/2775563274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2d4003a00 msgr2=0x7fa2d4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:02.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.353+0000 7fa2e3fff700 1 --2- 172.21.15.189:0/2775563274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2d4003a00 0x7fa2d4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:02.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.353+0000 7fa2e3fff700 1 -- 172.21.15.189:0/2775563274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d40035b0 con 0x7fa2d4004400 2024-04-06T13:48:02.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.353+0000 7fa2e9282700 1 -- 172.21.15.189:0/2775563274 reap_dead start 2024-04-06T13:48:02.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.353+0000 7fa2e3fff700 1 --2- 172.21.15.189:0/2775563274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d4004400 0x7fa2d40048d0 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0007f40 tx=0x7fa2d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a293b61298924b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:02.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.353+0000 7fa2e37fe700 1 -- 172.21.15.189:0/2775563274 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d000cd80 con 0x7fa2d4004400 2024-04-06T13:48:02.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.353+0000 7fa2e37fe700 1 -- 172.21.15.189:0/2775563274 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d0005960 con 0x7fa2d4004400 2024-04-06T13:48:02.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.353+0000 7fa2e37fe700 1 -- 172.21.15.189:0/2775563274 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0005d10 con 0x7fa2d4004400 2024-04-06T13:48:02.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.353+0000 7fa2eace5700 1 -- 172.21.15.189:0/2775563274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d4004400 msgr2=0x7fa2d40048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:02.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.353+0000 7fa2eace5700 1 --2- 172.21.15.189:0/2775563274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d4004400 0x7fa2d40048d0 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0007f40 tx=0x7fa2d4005880 comp rx=0 tx=0).stop 2024-04-06T13:48:02.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.354+0000 7fa2eace5700 1 -- 172.21.15.189:0/2775563274 shutdown_connections 2024-04-06T13:48:02.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.354+0000 7fa2eace5700 1 --2- 172.21.15.189:0/2775563274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d4004400 0x7fa2d40048d0 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:02.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.354+0000 7fa2eace5700 1 -- 172.21.15.189:0/2775563274 >> 172.21.15.189:0/2775563274 conn(0x7fa2e40d00b0 msgr2=0x7fa2e40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:02.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.354+0000 7fa2eace5700 1 -- 172.21.15.189:0/2775563274 shutdown_connections 2024-04-06T13:48:02.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.354+0000 7fa2eace5700 1 -- 172.21.15.189:0/2775563274 wait complete. 2024-04-06T13:48:02.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.355+0000 7fa2eace5700 1 Processor -- start 2024-04-06T13:48:02.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.355+0000 7fa2eace5700 1 -- start start 2024-04-06T13:48:02.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.356+0000 7fa2eace5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d4004400 0x7fa2e414bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:02.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.356+0000 7fa2eace5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2e414f420 0x7fa2e4158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:02.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.356+0000 7fa2eace5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2e414f8a0 0x7fa2e4159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:02.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.356+0000 7fa2eace5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e4153300 con 0x7fa2e414f420 2024-04-06T13:48:02.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.356+0000 7fa2eace5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e414c140 con 0x7fa2e414f8a0 2024-04-06T13:48:02.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.356+0000 7fa2eace5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e40b0900 con 0x7fa2d4004400 2024-04-06T13:48:02.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.356+0000 7fa2e3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2e414f420 0x7fa2e4158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:02.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.356+0000 7fa2e3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2e414f420 0x7fa2e4158b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41696/0 (socket says 172.21.15.189:41696) 2024-04-06T13:48:02.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.356+0000 7fa2e3fff700 1 -- 172.21.15.189:0/2476808816 learned_addr learned my addr 172.21.15.189:0/2476808816 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:02.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.356+0000 7fa2e8a81700 1 --2- 172.21.15.189:0/2476808816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d4004400 0x7fa2e414bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:02.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.356+0000 7fa2e9282700 1 --2- 172.21.15.189:0/2476808816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2e414f8a0 0x7fa2e4159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:02.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.357+0000 7fa2e3fff700 1 -- 172.21.15.189:0/2476808816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d4004400 msgr2=0x7fa2e414bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:02.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.357+0000 7fa2e3fff700 1 --2- 172.21.15.189:0/2476808816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d4004400 0x7fa2e414bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:02.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.357+0000 7fa2e3fff700 1 -- 172.21.15.189:0/2476808816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2e414f8a0 msgr2=0x7fa2e4159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:02.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.357+0000 7fa2e3fff700 1 --2- 172.21.15.189:0/2476808816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2e414f8a0 0x7fa2e4159090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:02.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.357+0000 7fa2e3fff700 1 -- 172.21.15.189:0/2476808816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e40b0be0 con 0x7fa2e414f420 2024-04-06T13:48:02.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.357+0000 7fa2e3fff700 1 --2- 172.21.15.189:0/2476808816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2e414f420 0x7fa2e4158b50 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0006d90 tx=0x7fa2d0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:02.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.357+0000 7fa2e17fa700 1 -- 172.21.15.189:0/2476808816 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0004a80 con 0x7fa2e414f420 2024-04-06T13:48:02.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.357+0000 7fa2e17fa700 1 -- 172.21.15.189:0/2476808816 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d0004be0 con 0x7fa2e414f420 2024-04-06T13:48:02.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.357+0000 7fa2e17fa700 1 -- 172.21.15.189:0/2476808816 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d00070d0 con 0x7fa2e414f420 2024-04-06T13:48:02.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.357+0000 7fa2eace5700 1 -- 172.21.15.189:0/2476808816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e40b0ec0 con 0x7fa2e414f420 2024-04-06T13:48:02.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.357+0000 7fa2eace5700 1 -- 172.21.15.189:0/2476808816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e40b04f0 con 0x7fa2e414f420 2024-04-06T13:48:02.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.358+0000 7fa2eace5700 1 -- 172.21.15.189:0/2476808816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e4067bb0 con 0x7fa2e414f420 2024-04-06T13:48:02.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.358+0000 7fa2e17fa700 1 -- 172.21.15.189:0/2476808816 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fa2d0007230 con 0x7fa2e414f420 2024-04-06T13:48:02.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.359+0000 7fa2e17fa700 1 --2- 172.21.15.189:0/2476808816 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa2c806aed0 0x7fa2c806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:02.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.359+0000 7fa2e17fa700 1 -- 172.21.15.189:0/2476808816 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 7344+0+0 (secure 0 0 0) 0x7fa2d008ad00 con 0x7fa2e414f420 2024-04-06T13:48:02.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.359+0000 7fa2e8a81700 1 --2- 172.21.15.189:0/2476808816 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa2c806aed0 0x7fa2c806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:02.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.359+0000 7fa2e8a81700 1 --2- 172.21.15.189:0/2476808816 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa2c806aed0 0x7fa2c806d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8007fd0 tx=0x7fa2d8009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:02.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.363+0000 7fa2e17fa700 1 -- 172.21.15.189:0/2476808816 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa2d0055530 con 0x7fa2e414f420 2024-04-06T13:48:02.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:02.556+0000 7fa2eace5700 1 -- 172.21.15.189:0/2476808816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa2e40b0680 con 0x7fa2e414f420 2024-04-06T13:48:03.021 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:48:03.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:48:03.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.195+0000 7fa2e17fa700 1 -- 172.21.15.189:0/2476808816 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v236) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa2d0007da0 con 0x7fa2e414f420 2024-04-06T13:48:03.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.198+0000 7fa2eace5700 1 -- 172.21.15.189:0/2476808816 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa2c806aed0 msgr2=0x7fa2c806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:03.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.198+0000 7fa2eace5700 1 --2- 172.21.15.189:0/2476808816 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa2c806aed0 0x7fa2c806d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8007fd0 tx=0x7fa2d8009040 comp rx=0 tx=0).stop 2024-04-06T13:48:03.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.198+0000 7fa2eace5700 1 -- 172.21.15.189:0/2476808816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2e414f420 msgr2=0x7fa2e4158b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:03.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.198+0000 7fa2eace5700 1 --2- 172.21.15.189:0/2476808816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2e414f420 0x7fa2e4158b50 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0006d90 tx=0x7fa2d0001a00 comp rx=0 tx=0).stop 2024-04-06T13:48:03.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.198+0000 7fa2eace5700 1 -- 172.21.15.189:0/2476808816 shutdown_connections 2024-04-06T13:48:03.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.198+0000 7fa2eace5700 1 --2- 172.21.15.189:0/2476808816 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa2c806aed0 0x7fa2c806d390 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.198+0000 7fa2eace5700 1 --2- 172.21.15.189:0/2476808816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2e414f8a0 0x7fa2e4159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.198+0000 7fa2eace5700 1 --2- 172.21.15.189:0/2476808816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d4004400 0x7fa2e414bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.198+0000 7fa2eace5700 1 --2- 172.21.15.189:0/2476808816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2e414f420 0x7fa2e4158b50 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.198+0000 7fa2eace5700 1 -- 172.21.15.189:0/2476808816 >> 172.21.15.189:0/2476808816 conn(0x7fa2e40d00b0 msgr2=0x7fa2e406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:03.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.198+0000 7fa2eace5700 1 -- 172.21.15.189:0/2476808816 shutdown_connections 2024-04-06T13:48:03.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.198+0000 7fa2eace5700 1 -- 172.21.15.189:0/2476808816 wait complete. 2024-04-06T13:48:03.202 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' removed 2024-04-06T13:48:03.215 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:48:03.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.344+0000 7fc6ec1f0700 1 Processor -- start 2024-04-06T13:48:03.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.345+0000 7fc6ec1f0700 1 -- start start 2024-04-06T13:48:03.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.345+0000 7fc6ec1f0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc6e40d17a0 0x7fc6e406b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:03.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.345+0000 7fc6ec1f0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e406d850 con 0x7fc6e40cedf0 2024-04-06T13:48:03.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.345+0000 7fc6ec1f0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e406c0a0 con 0x7fc6e4146dd0 2024-04-06T13:48:03.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.345+0000 7fc6ec1f0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6e406c380 con 0x7fc6e40d17a0 2024-04-06T13:48:03.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.346+0000 7fc6e9f8c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc6e40d17a0 0x7fc6e406b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:03.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.346+0000 7fc6ea78d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc6e40cedf0 0x7fc6e40cf1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50668/0 (socket says 172.21.15.189:50668) 2024-04-06T13:48:03.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.346+0000 7fc6e9f8c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc6e40d17a0 0x7fc6e406b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41698/0 (socket says 172.21.15.189:41698) 2024-04-06T13:48:03.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.346+0000 7fc6ea78d700 1 -- 172.21.15.189:0/3245821987 learned_addr learned my addr 172.21.15.189:0/3245821987 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:03.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.346+0000 7fc6e9f8c700 1 -- 172.21.15.189:0/3245821987 >> v1:172.21.15.192:6789/0 conn(0x7fc6e4146dd0 legacy=0x7fc6e406d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:03.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.346+0000 7fc6e9f8c700 1 -- 172.21.15.189:0/3245821987 >> v1:172.21.15.189:6789/0 conn(0x7fc6e40cedf0 legacy=0x7fc6e40cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:03.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.346+0000 7fc6e9f8c700 1 -- 172.21.15.189:0/3245821987 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e406c660 con 0x7fc6e40d17a0 2024-04-06T13:48:03.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.346+0000 7fc6e9f8c700 1 --2- 172.21.15.189:0/3245821987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6e40d17a0 0x7fc6e406b7a0 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009840 tx=0x7fc6d8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55cd7b7c7397c619 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:03.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.346+0000 7fc6e8f8a700 1 -- 172.21.15.189:0/3245821987 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8011830 con 0x7fc6e40d17a0 2024-04-06T13:48:03.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.346+0000 7fc6e8f8a700 1 -- 172.21.15.189:0/3245821987 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d8011990 con 0x7fc6e40d17a0 2024-04-06T13:48:03.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.346+0000 7fc6e8f8a700 1 -- 172.21.15.189:0/3245821987 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8011d20 con 0x7fc6e40d17a0 2024-04-06T13:48:03.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.347+0000 7fc6ec1f0700 1 -- 172.21.15.189:0/3245821987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6e40d17a0 msgr2=0x7fc6e406b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:03.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.347+0000 7fc6ec1f0700 1 --2- 172.21.15.189:0/3245821987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6e40d17a0 0x7fc6e406b7a0 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009840 tx=0x7fc6d8009b50 comp rx=0 tx=0).stop 2024-04-06T13:48:03.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.347+0000 7fc6ec1f0700 1 -- 172.21.15.189:0/3245821987 shutdown_connections 2024-04-06T13:48:03.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.347+0000 7fc6ec1f0700 1 --2- 172.21.15.189:0/3245821987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6e40d17a0 0x7fc6e406b7a0 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.347+0000 7fc6ec1f0700 1 -- 172.21.15.189:0/3245821987 >> 172.21.15.189:0/3245821987 conn(0x7fc6e40b2db0 msgr2=0x7fc6e40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:03.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.347+0000 7fc6ec1f0700 1 -- 172.21.15.189:0/3245821987 shutdown_connections 2024-04-06T13:48:03.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.347+0000 7fc6ec1f0700 1 -- 172.21.15.189:0/3245821987 wait complete. 2024-04-06T13:48:03.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.348+0000 7fc6ec1f0700 1 Processor -- start 2024-04-06T13:48:03.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.348+0000 7fc6ec1f0700 1 -- start start 2024-04-06T13:48:03.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.349+0000 7fc6ec1f0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6e40cedf0 0x7fc6e415ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:03.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.349+0000 7fc6ec1f0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6e40d17a0 0x7fc6e40aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:03.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.349+0000 7fc6ec1f0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6e4146dd0 0x7fc6e40af400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:03.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.349+0000 7fc6ec1f0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e415d3d0 con 0x7fc6e40cedf0 2024-04-06T13:48:03.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.349+0000 7fc6ec1f0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e40ad5f0 con 0x7fc6e40d17a0 2024-04-06T13:48:03.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.349+0000 7fc6ec1f0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e40ad8d0 con 0x7fc6e4146dd0 2024-04-06T13:48:03.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.349+0000 7fc6e9f8c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6e40cedf0 0x7fc6e415ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:03.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.349+0000 7fc6e9f8c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6e40cedf0 0x7fc6e415ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41712/0 (socket says 172.21.15.189:41712) 2024-04-06T13:48:03.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.349+0000 7fc6e9f8c700 1 -- 172.21.15.189:0/2641437904 learned_addr learned my addr 172.21.15.189:0/2641437904 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:03.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.349+0000 7fc6e978b700 1 --2- 172.21.15.189:0/2641437904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6e40d17a0 0x7fc6e40aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:03.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.349+0000 7fc6ea78d700 1 --2- 172.21.15.189:0/2641437904 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6e4146dd0 0x7fc6e40af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:03.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.349+0000 7fc6e9f8c700 1 -- 172.21.15.189:0/2641437904 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6e4146dd0 msgr2=0x7fc6e40af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:03.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.349+0000 7fc6e9f8c700 1 --2- 172.21.15.189:0/2641437904 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6e4146dd0 0x7fc6e40af400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.349+0000 7fc6e9f8c700 1 -- 172.21.15.189:0/2641437904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6e40d17a0 msgr2=0x7fc6e40aeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:03.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.349+0000 7fc6e9f8c700 1 --2- 172.21.15.189:0/2641437904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6e40d17a0 0x7fc6e40aeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.349+0000 7fc6e9f8c700 1 -- 172.21.15.189:0/2641437904 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d80094a0 con 0x7fc6e40cedf0 2024-04-06T13:48:03.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.349+0000 7fc6e978b700 1 --2- 172.21.15.189:0/2641437904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6e40d17a0 0x7fc6e40aeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:03.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.350+0000 7fc6e9f8c700 1 --2- 172.21.15.189:0/2641437904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6e40cedf0 0x7fc6e415ce90 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80090b0 tx=0x7fc6d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:03.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.350+0000 7fc6d67fc700 1 -- 172.21.15.189:0/2641437904 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d80038a0 con 0x7fc6e40cedf0 2024-04-06T13:48:03.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.350+0000 7fc6d67fc700 1 -- 172.21.15.189:0/2641437904 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d801f070 con 0x7fc6e40cedf0 2024-04-06T13:48:03.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.350+0000 7fc6d67fc700 1 -- 172.21.15.189:0/2641437904 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8003c60 con 0x7fc6e40cedf0 2024-04-06T13:48:03.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.350+0000 7fc6ec1f0700 1 -- 172.21.15.189:0/2641437904 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e40adbb0 con 0x7fc6e40cedf0 2024-04-06T13:48:03.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.350+0000 7fc6ec1f0700 1 -- 172.21.15.189:0/2641437904 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e40ae170 con 0x7fc6e40cedf0 2024-04-06T13:48:03.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.351+0000 7fc6ec1f0700 1 -- 172.21.15.189:0/2641437904 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e4074990 con 0x7fc6e40cedf0 2024-04-06T13:48:03.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.351+0000 7fc6d67fc700 1 -- 172.21.15.189:0/2641437904 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fc6d801a430 con 0x7fc6e40cedf0 2024-04-06T13:48:03.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.351+0000 7fc6d67fc700 1 --2- 172.21.15.189:0/2641437904 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc6c806aed0 0x7fc6c806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:03.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.352+0000 7fc6d67fc700 1 -- 172.21.15.189:0/2641437904 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6930+0+0 (secure 0 0 0) 0x7fc6d8021070 con 0x7fc6e40cedf0 2024-04-06T13:48:03.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.352+0000 7fc6e978b700 1 --2- 172.21.15.189:0/2641437904 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc6c806aed0 0x7fc6c806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:03.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.352+0000 7fc6e978b700 1 --2- 172.21.15.189:0/2641437904 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc6c806aed0 0x7fc6c806d390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0000f60 tx=0x7fc6e0009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:03.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.356+0000 7fc6d67fc700 1 -- 172.21.15.189:0/2641437904 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc6d8056120 con 0x7fc6e40cedf0 2024-04-06T13:48:03.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.549+0000 7fc6ec1f0700 1 -- 172.21.15.189:0/2641437904 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6e40d1e20 con 0x7fc6e40cedf0 2024-04-06T13:48:03.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.550+0000 7fc6d67fc700 1 -- 172.21.15.189:0/2641437904 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v236) v1 ==== 74+0+17619 (secure 0 0 0) 0x7fc6d8059740 con 0x7fc6e40cedf0 2024-04-06T13:48:03.551 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:03.551 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":236,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:48:03.192350+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:47:32.292623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:47:33.567052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.192:6833/694564673":"2024-04-07T13:48:00.073095+0000","172.21.15.189:0/3017776380":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1992839362":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/579471530":"2024-04-07T13:47:57.909304+0000","172.21.15.192:6835/694564673":"2024-04-07T13:48:00.073095+0000","172.21.15.189:0/4049616553":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/619264656":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1506469869":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1831656794":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1190635801":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/3920072072":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/1878657401":"2024-04-07T13:47:47.129296+0000","172.21.15.189:6833/111704":"2024-04-07T13:47:47.129296+0000","172.21.15.189:6832/111704":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/2827130089":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/2716712615":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/1807905335":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/914480885":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1173010452":"2024-04-07T13:47:47.129296+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:48:03.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.552+0000 7fc6ec1f0700 1 -- 172.21.15.189:0/2641437904 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc6c806aed0 msgr2=0x7fc6c806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:03.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.552+0000 7fc6ec1f0700 1 --2- 172.21.15.189:0/2641437904 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc6c806aed0 0x7fc6c806d390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0000f60 tx=0x7fc6e0009040 comp rx=0 tx=0).stop 2024-04-06T13:48:03.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.552+0000 7fc6ec1f0700 1 -- 172.21.15.189:0/2641437904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6e40cedf0 msgr2=0x7fc6e415ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:03.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.552+0000 7fc6ec1f0700 1 --2- 172.21.15.189:0/2641437904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6e40cedf0 0x7fc6e415ce90 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80090b0 tx=0x7fc6d8004750 comp rx=0 tx=0).stop 2024-04-06T13:48:03.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.552+0000 7fc6ec1f0700 1 -- 172.21.15.189:0/2641437904 shutdown_connections 2024-04-06T13:48:03.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.552+0000 7fc6ec1f0700 1 --2- 172.21.15.189:0/2641437904 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc6c806aed0 0x7fc6c806d390 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.552+0000 7fc6ec1f0700 1 --2- 172.21.15.189:0/2641437904 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6e4146dd0 0x7fc6e40af400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.552+0000 7fc6ec1f0700 1 --2- 172.21.15.189:0/2641437904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6e40cedf0 0x7fc6e415ce90 unknown :-1 s=CLOSED pgs=2595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.552+0000 7fc6ec1f0700 1 --2- 172.21.15.189:0/2641437904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6e40d17a0 0x7fc6e40aeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.552+0000 7fc6ec1f0700 1 -- 172.21.15.189:0/2641437904 >> 172.21.15.189:0/2641437904 conn(0x7fc6e40b2db0 msgr2=0x7fc6e40b2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:03.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.552+0000 7fc6ec1f0700 1 -- 172.21.15.189:0/2641437904 shutdown_connections 2024-04-06T13:48:03.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.552+0000 7fc6ec1f0700 1 -- 172.21.15.189:0/2641437904 wait complete. 2024-04-06T13:48:03.566 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-04-06T13:47:32.292623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '230', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2024-04-06T13:47:33.567052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '231', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:48:03.567 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:48:03.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.694+0000 7fd300ddf700 1 Processor -- start 2024-04-06T13:48:03.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.694+0000 7fd300ddf700 1 -- start start 2024-04-06T13:48:03.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.695+0000 7fd300ddf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2fc06ae40 0x7fd2fc0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:03.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.695+0000 7fd300ddf700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd2fc15ddf0 0x7fd2fc15e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:03.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.695+0000 7fd300ddf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2fc0b3990 con 0x7fd2fc15d7b0 2024-04-06T13:48:03.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.695+0000 7fd300ddf700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2fc0b3060 con 0x7fd2fc15ddf0 2024-04-06T13:48:03.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.695+0000 7fd300ddf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc0b3340 con 0x7fd2fc06ae40 2024-04-06T13:48:03.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.695+0000 7fd2fad9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd2fc15ddf0 0x7fd2fc15e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:03.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.695+0000 7fd2fa59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd2fc15d7b0 0x7fd2fc06a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50678/0 (socket says 172.21.15.189:50678) 2024-04-06T13:48:03.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.695+0000 7fd2fa59c700 1 -- 172.21.15.189:0/1667543866 learned_addr learned my addr 172.21.15.189:0/1667543866 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:03.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.695+0000 7fd2fad9d700 1 -- 172.21.15.189:0/1667543866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2fc06ae40 msgr2=0x7fd2fc0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:03.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.695+0000 7fd2f9d9b700 1 --2- 172.21.15.189:0/1667543866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2fc06ae40 0x7fd2fc0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:03.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.696+0000 7fd2fad9d700 1 --2- 172.21.15.189:0/1667543866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2fc06ae40 0x7fd2fc0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.696+0000 7fd2f959a700 1 -- 172.21.15.189:0/1667543866 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734621267 0 0) 0x7fd2fc0b3990 con 0x7fd2fc15d7b0 2024-04-06T13:48:03.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.696+0000 7fd2fad9d700 1 -- 172.21.15.189:0/1667543866 >> v1:172.21.15.189:6789/0 conn(0x7fd2fc15d7b0 legacy=0x7fd2fc06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:03.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.696+0000 7fd2fad9d700 1 -- 172.21.15.189:0/1667543866 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc0b3620 con 0x7fd2fc15ddf0 2024-04-06T13:48:03.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.696+0000 7fd2fad9d700 1 --2- 172.21.15.189:0/1667543866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2fc15ddf0 0x7fd2fc15e2a0 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4009d90 tx=0x7fd2f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a60d1434b127522 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:03.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.696+0000 7fd2f959a700 1 -- 172.21.15.189:0/1667543866 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f400e070 con 0x7fd2fc15ddf0 2024-04-06T13:48:03.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.696+0000 7fd2f959a700 1 -- 172.21.15.189:0/1667543866 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f400bbd0 con 0x7fd2fc15ddf0 2024-04-06T13:48:03.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.696+0000 7fd2f959a700 1 -- 172.21.15.189:0/1667543866 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4012430 con 0x7fd2fc15ddf0 2024-04-06T13:48:03.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.696+0000 7fd300ddf700 1 -- 172.21.15.189:0/1667543866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2fc15ddf0 msgr2=0x7fd2fc15e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:03.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.696+0000 7fd300ddf700 1 --2- 172.21.15.189:0/1667543866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2fc15ddf0 0x7fd2fc15e2a0 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4009d90 tx=0x7fd2f400b330 comp rx=0 tx=0).stop 2024-04-06T13:48:03.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.696+0000 7fd300ddf700 1 -- 172.21.15.189:0/1667543866 shutdown_connections 2024-04-06T13:48:03.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.696+0000 7fd300ddf700 1 --2- 172.21.15.189:0/1667543866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2fc15ddf0 0x7fd2fc15e2a0 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.696+0000 7fd300ddf700 1 --2- 172.21.15.189:0/1667543866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2fc06ae40 0x7fd2fc0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.696+0000 7fd300ddf700 1 -- 172.21.15.189:0/1667543866 >> 172.21.15.189:0/1667543866 conn(0x7fd2fc0d00b0 msgr2=0x7fd2fc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:03.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.697+0000 7fd300ddf700 1 -- 172.21.15.189:0/1667543866 shutdown_connections 2024-04-06T13:48:03.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.697+0000 7fd300ddf700 1 -- 172.21.15.189:0/1667543866 wait complete. 2024-04-06T13:48:03.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.698+0000 7fd300ddf700 1 Processor -- start 2024-04-06T13:48:03.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.698+0000 7fd300ddf700 1 -- start start 2024-04-06T13:48:03.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd300ddf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2fc06ae40 0x7fd2fc15c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:03.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd300ddf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2fc15d7b0 0x7fd2fc15c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:03.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd300ddf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2fc15ddf0 0x7fd2fc148e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:03.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd300ddf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc0cf550 con 0x7fd2fc06ae40 2024-04-06T13:48:03.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd300ddf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc14a800 con 0x7fd2fc15d7b0 2024-04-06T13:48:03.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd300ddf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc149b80 con 0x7fd2fc15ddf0 2024-04-06T13:48:03.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd2fa59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2fc06ae40 0x7fd2fc15c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:03.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd2fa59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2fc06ae40 0x7fd2fc15c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41726/0 (socket says 172.21.15.189:41726) 2024-04-06T13:48:03.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd2fa59c700 1 -- 172.21.15.189:0/1475093688 learned_addr learned my addr 172.21.15.189:0/1475093688 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:03.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd2fad9d700 1 --2- 172.21.15.189:0/1475093688 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2fc15ddf0 0x7fd2fc148e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:03.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd2f9d9b700 1 --2- 172.21.15.189:0/1475093688 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2fc15d7b0 0x7fd2fc15c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:03.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd2fa59c700 1 -- 172.21.15.189:0/1475093688 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2fc15ddf0 msgr2=0x7fd2fc148e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:03.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd2fa59c700 1 --2- 172.21.15.189:0/1475093688 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2fc15ddf0 0x7fd2fc148e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd2fa59c700 1 -- 172.21.15.189:0/1475093688 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2fc15d7b0 msgr2=0x7fd2fc15c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:03.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd2fa59c700 1 --2- 172.21.15.189:0/1475093688 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2fc15d7b0 0x7fd2fc15c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd2fa59c700 1 -- 172.21.15.189:0/1475093688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f4009a70 con 0x7fd2fc06ae40 2024-04-06T13:48:03.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd2fa59c700 1 --2- 172.21.15.189:0/1475093688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2fc06ae40 0x7fd2fc15c3a0 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7fd2f00011e0 tx=0x7fd2f000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:03.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd2eaffd700 1 -- 172.21.15.189:0/1475093688 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0004c80 con 0x7fd2fc06ae40 2024-04-06T13:48:03.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd2eaffd700 1 -- 172.21.15.189:0/1475093688 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f0004e00 con 0x7fd2fc06ae40 2024-04-06T13:48:03.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd300ddf700 1 -- 172.21.15.189:0/1475093688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2fc149e60 con 0x7fd2fc06ae40 2024-04-06T13:48:03.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd2eaffd700 1 -- 172.21.15.189:0/1475093688 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0007310 con 0x7fd2fc06ae40 2024-04-06T13:48:03.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.699+0000 7fd300ddf700 1 -- 172.21.15.189:0/1475093688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2fc14a420 con 0x7fd2fc06ae40 2024-04-06T13:48:03.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.700+0000 7fd300ddf700 1 -- 172.21.15.189:0/1475093688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2fc067bb0 con 0x7fd2fc06ae40 2024-04-06T13:48:03.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.701+0000 7fd2eaffd700 1 -- 172.21.15.189:0/1475093688 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fd2f0007470 con 0x7fd2fc06ae40 2024-04-06T13:48:03.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.701+0000 7fd2eaffd700 1 --2- 172.21.15.189:0/1475093688 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd2dc06aed0 0x7fd2dc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:03.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.701+0000 7fd2eaffd700 1 -- 172.21.15.189:0/1475093688 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6930+0+0 (secure 0 0 0) 0x7fd2f0089920 con 0x7fd2fc06ae40 2024-04-06T13:48:03.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.701+0000 7fd2f9d9b700 1 --2- 172.21.15.189:0/1475093688 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd2dc06aed0 0x7fd2dc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:03.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.702+0000 7fd2f9d9b700 1 --2- 172.21.15.189:0/1475093688 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd2dc06aed0 0x7fd2dc06d390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc1f60d0 tx=0x7fd2ec008040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:03.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.706+0000 7fd2eaffd700 1 -- 172.21.15.189:0/1475093688 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd2f0054360 con 0x7fd2fc06ae40 2024-04-06T13:48:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.909+0000 7fd300ddf700 1 -- 172.21.15.189:0/1475093688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd2fc074990 con 0x7fd2fc06ae40 2024-04-06T13:48:03.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.909+0000 7fd2eaffd700 1 -- 172.21.15.189:0/1475093688 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v236) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd2f0057980 con 0x7fd2fc06ae40 2024-04-06T13:48:03.911 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:48:03.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.911+0000 7fd300ddf700 1 -- 172.21.15.189:0/1475093688 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd2dc06aed0 msgr2=0x7fd2dc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:03.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.911+0000 7fd300ddf700 1 --2- 172.21.15.189:0/1475093688 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd2dc06aed0 0x7fd2dc06d390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc1f60d0 tx=0x7fd2ec008040 comp rx=0 tx=0).stop 2024-04-06T13:48:03.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.911+0000 7fd300ddf700 1 -- 172.21.15.189:0/1475093688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2fc06ae40 msgr2=0x7fd2fc15c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:03.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.911+0000 7fd300ddf700 1 --2- 172.21.15.189:0/1475093688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2fc06ae40 0x7fd2fc15c3a0 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7fd2f00011e0 tx=0x7fd2f000cae0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.911+0000 7fd300ddf700 1 -- 172.21.15.189:0/1475093688 shutdown_connections 2024-04-06T13:48:03.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.911+0000 7fd300ddf700 1 --2- 172.21.15.189:0/1475093688 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd2dc06aed0 0x7fd2dc06d390 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.911+0000 7fd300ddf700 1 --2- 172.21.15.189:0/1475093688 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2fc15ddf0 0x7fd2fc148e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.911+0000 7fd300ddf700 1 --2- 172.21.15.189:0/1475093688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2fc06ae40 0x7fd2fc15c3a0 unknown :-1 s=CLOSED pgs=2598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.911+0000 7fd300ddf700 1 --2- 172.21.15.189:0/1475093688 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2fc15d7b0 0x7fd2fc15c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:03.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.912+0000 7fd300ddf700 1 -- 172.21.15.189:0/1475093688 >> 172.21.15.189:0/1475093688 conn(0x7fd2fc0d00b0 msgr2=0x7fd2fc159510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:03.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.912+0000 7fd300ddf700 1 -- 172.21.15.189:0/1475093688 shutdown_connections 2024-04-06T13:48:03.913 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:03.912+0000 7fd300ddf700 1 -- 172.21.15.189:0/1475093688 wait complete. 2024-04-06T13:48:03.925 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-04-06T13:48:04.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.050+0000 7fcde7a52700 1 Processor -- start 2024-04-06T13:48:04.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.050+0000 7fcde7a52700 1 -- start start 2024-04-06T13:48:04.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.051+0000 7fcde7a52700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcde00d11b0 0x7fcde00d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:04.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.051+0000 7fcde7a52700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde00d1ad0 con 0x7fcde00cb870 2024-04-06T13:48:04.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.051+0000 7fcde7a52700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde00cbd80 con 0x7fcde00cfa00 2024-04-06T13:48:04.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.051+0000 7fcde7a52700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcde00ccdc0 con 0x7fcde00d11b0 2024-04-06T13:48:04.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.051+0000 7fcde57ee700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcde00d11b0 0x7fcde00d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:04.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.051+0000 7fcde4fed700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcde00cb870 0x7fcde015c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50686/0 (socket says 172.21.15.189:50686) 2024-04-06T13:48:04.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.051+0000 7fcde4fed700 1 -- 172.21.15.189:0/453360809 learned_addr learned my addr 172.21.15.189:0/453360809 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:04.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.051+0000 7fcde57ee700 1 -- 172.21.15.189:0/453360809 >> v1:172.21.15.192:6789/0 conn(0x7fcde00cfa00 legacy=0x7fcde00cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:04.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.051+0000 7fcde57ee700 1 -- 172.21.15.189:0/453360809 >> v1:172.21.15.189:6789/0 conn(0x7fcde00cb870 legacy=0x7fcde015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:04.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.051+0000 7fcde57ee700 1 -- 172.21.15.189:0/453360809 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde00cef20 con 0x7fcde00d11b0 2024-04-06T13:48:04.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.051+0000 7fcdd3fff700 1 -- 172.21.15.189:0/453360809 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859831 0 0) 0x7fcde00d1ad0 con 0x7fcde00cb870 2024-04-06T13:48:04.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.051+0000 7fcde57ee700 1 --2- 172.21.15.189:0/453360809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde00d11b0 0x7fcde00d1590 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7fcddc009840 tx=0x7fcddc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db3c008513ba524 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:04.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.051+0000 7fcdd3fff700 1 -- 172.21.15.189:0/453360809 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc011830 con 0x7fcde00d11b0 2024-04-06T13:48:04.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.051+0000 7fcdd3fff700 1 -- 172.21.15.189:0/453360809 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcddc011990 con 0x7fcde00d11b0 2024-04-06T13:48:04.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.051+0000 7fcdd3fff700 1 -- 172.21.15.189:0/453360809 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc011d20 con 0x7fcde00d11b0 2024-04-06T13:48:04.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.052+0000 7fcde7a52700 1 -- 172.21.15.189:0/453360809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde00d11b0 msgr2=0x7fcde00d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:04.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.052+0000 7fcde7a52700 1 --2- 172.21.15.189:0/453360809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde00d11b0 0x7fcde00d1590 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7fcddc009840 tx=0x7fcddc009b50 comp rx=0 tx=0).stop 2024-04-06T13:48:04.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.052+0000 7fcde7a52700 1 -- 172.21.15.189:0/453360809 shutdown_connections 2024-04-06T13:48:04.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.052+0000 7fcde7a52700 1 --2- 172.21.15.189:0/453360809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde00d11b0 0x7fcde00d1590 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.052+0000 7fcde7a52700 1 -- 172.21.15.189:0/453360809 >> 172.21.15.189:0/453360809 conn(0x7fcde00b2db0 msgr2=0x7fcde00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:04.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.052+0000 7fcde7a52700 1 -- 172.21.15.189:0/453360809 shutdown_connections 2024-04-06T13:48:04.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.052+0000 7fcde7a52700 1 -- 172.21.15.189:0/453360809 wait complete. 2024-04-06T13:48:04.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.053+0000 7fcde7a52700 1 Processor -- start 2024-04-06T13:48:04.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.053+0000 7fcde7a52700 1 -- start start 2024-04-06T13:48:04.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.054+0000 7fcde7a52700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde00cb870 0x7fcde01f3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:04.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.054+0000 7fcde7a52700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcde00cfa00 0x7fcde0157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:04.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.054+0000 7fcde7a52700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcde00d11b0 0x7fcde01550b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:04.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.054+0000 7fcde7a52700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde00c82f0 con 0x7fcde00cb870 2024-04-06T13:48:04.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.054+0000 7fcde7a52700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde01f3f70 con 0x7fcde00d11b0 2024-04-06T13:48:04.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.054+0000 7fcde7a52700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde01f4170 con 0x7fcde00cfa00 2024-04-06T13:48:04.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.054+0000 7fcde57ee700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde00cb870 0x7fcde01f3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:04.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.054+0000 7fcde57ee700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde00cb870 0x7fcde01f3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41736/0 (socket says 172.21.15.189:41736) 2024-04-06T13:48:04.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.054+0000 7fcde57ee700 1 -- 172.21.15.189:0/1763011252 learned_addr learned my addr 172.21.15.189:0/1763011252 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:04.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.054+0000 7fcde5fef700 1 --2- 172.21.15.189:0/1763011252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcde00d11b0 0x7fcde01550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:04.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.054+0000 7fcde4fed700 1 --2- 172.21.15.189:0/1763011252 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcde00cfa00 0x7fcde0157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:04.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.054+0000 7fcde57ee700 1 -- 172.21.15.189:0/1763011252 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcde00cfa00 msgr2=0x7fcde0157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:04.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.054+0000 7fcde57ee700 1 --2- 172.21.15.189:0/1763011252 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcde00cfa00 0x7fcde0157940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.054+0000 7fcde57ee700 1 -- 172.21.15.189:0/1763011252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcde00d11b0 msgr2=0x7fcde01550b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.054+0000 7fcde57ee700 1 --2- 172.21.15.189:0/1763011252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcde00d11b0 0x7fcde01550b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.054+0000 7fcde57ee700 1 -- 172.21.15.189:0/1763011252 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcddc0094a0 con 0x7fcde00cb870 2024-04-06T13:48:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.054+0000 7fcde4fed700 1 --2- 172.21.15.189:0/1763011252 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcde00cfa00 0x7fcde0157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.055+0000 7fcde57ee700 1 --2- 172.21.15.189:0/1763011252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde00cb870 0x7fcde01f3a30 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7fcddc014040 tx=0x7fcddc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.055+0000 7fcdd1ffb700 1 -- 172.21.15.189:0/1763011252 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc01a910 con 0x7fcde00cb870 2024-04-06T13:48:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.055+0000 7fcdd1ffb700 1 -- 172.21.15.189:0/1763011252 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcddc01aa70 con 0x7fcde00cb870 2024-04-06T13:48:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.055+0000 7fcde7a52700 1 -- 172.21.15.189:0/1763011252 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde00c8540 con 0x7fcde00cb870 2024-04-06T13:48:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.055+0000 7fcdd1ffb700 1 -- 172.21.15.189:0/1763011252 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc01ade0 con 0x7fcde00cb870 2024-04-06T13:48:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.055+0000 7fcde7a52700 1 -- 172.21.15.189:0/1763011252 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde00c8b00 con 0x7fcde00cb870 2024-04-06T13:48:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.056+0000 7fcde7a52700 1 -- 172.21.15.189:0/1763011252 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde0074990 con 0x7fcde00cb870 2024-04-06T13:48:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.056+0000 7fcdd1ffb700 1 -- 172.21.15.189:0/1763011252 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fcddc022070 con 0x7fcde00cb870 2024-04-06T13:48:04.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.057+0000 7fcdd1ffb700 1 --2- 172.21.15.189:0/1763011252 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcdc406aed0 0x7fcdc406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:04.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.057+0000 7fcde4fed700 1 --2- 172.21.15.189:0/1763011252 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcdc406aed0 0x7fcdc406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:04.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.057+0000 7fcdd1ffb700 1 -- 172.21.15.189:0/1763011252 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6930+0+0 (secure 0 0 0) 0x7fcddc08af30 con 0x7fcde00cb870 2024-04-06T13:48:04.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.057+0000 7fcde4fed700 1 --2- 172.21.15.189:0/1763011252 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcdc406aed0 0x7fcdc406d390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcde00d0950 tx=0x7fcdd4007ce0 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:04.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.061+0000 7fcdd1ffb700 1 -- 172.21.15.189:0/1763011252 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fcddc055a90 con 0x7fcde00cb870 2024-04-06T13:48:04.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.257+0000 7fcde7a52700 1 -- 172.21.15.189:0/1763011252 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcde015bb20 con 0x7fcde00cb870 2024-04-06T13:48:04.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.258+0000 7fcdd1ffb700 1 -- 172.21.15.189:0/1763011252 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v236) v1 ==== 107+0+11 (secure 0 0 0) 0x7fcddc029020 con 0x7fcde00cb870 2024-04-06T13:48:04.260 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:48:04.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.260+0000 7fcde7a52700 1 -- 172.21.15.189:0/1763011252 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcdc406aed0 msgr2=0x7fcdc406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:04.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.260+0000 7fcde7a52700 1 --2- 172.21.15.189:0/1763011252 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcdc406aed0 0x7fcdc406d390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcde00d0950 tx=0x7fcdd4007ce0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.260+0000 7fcde7a52700 1 -- 172.21.15.189:0/1763011252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde00cb870 msgr2=0x7fcde01f3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:04.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.260+0000 7fcde7a52700 1 --2- 172.21.15.189:0/1763011252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde00cb870 0x7fcde01f3a30 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7fcddc014040 tx=0x7fcddc004750 comp rx=0 tx=0).stop 2024-04-06T13:48:04.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.260+0000 7fcde7a52700 1 -- 172.21.15.189:0/1763011252 shutdown_connections 2024-04-06T13:48:04.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.260+0000 7fcde7a52700 1 --2- 172.21.15.189:0/1763011252 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcdc406aed0 0x7fcdc406d390 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.260+0000 7fcde7a52700 1 --2- 172.21.15.189:0/1763011252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcde00d11b0 0x7fcde01550b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.260+0000 7fcde7a52700 1 --2- 172.21.15.189:0/1763011252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde00cb870 0x7fcde01f3a30 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.260+0000 7fcde7a52700 1 --2- 172.21.15.189:0/1763011252 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcde00cfa00 0x7fcde0157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.260+0000 7fcde7a52700 1 -- 172.21.15.189:0/1763011252 >> 172.21.15.189:0/1763011252 conn(0x7fcde00b2db0 msgr2=0x7fcde006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:04.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.260+0000 7fcde7a52700 1 -- 172.21.15.189:0/1763011252 shutdown_connections 2024-04-06T13:48:04.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.260+0000 7fcde7a52700 1 -- 172.21.15.189:0/1763011252 wait complete. 2024-04-06T13:48:04.273 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-04-06T13:48:04.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.397+0000 7efca076a700 1 Processor -- start 2024-04-06T13:48:04.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.397+0000 7efca076a700 1 -- start start 2024-04-06T13:48:04.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.398+0000 7efca076a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc98158fe0 0x7efc9806a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:04.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.398+0000 7efca076a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efc9806ac70 0x7efc9815bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:04.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.398+0000 7efca076a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9815f9e0 con 0x7efc9815c100 2024-04-06T13:48:04.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.398+0000 7efca076a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc9806b0f0 con 0x7efc9806ac70 2024-04-06T13:48:04.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.398+0000 7efca076a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc9815f0d0 con 0x7efc98158fe0 2024-04-06T13:48:04.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.398+0000 7efc9dd05700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efc9806ac70 0x7efc9815bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:04.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.398+0000 7efc9dd05700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efc9806ac70 0x7efc9815bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41744/0 (socket says 172.21.15.189:41744) 2024-04-06T13:48:04.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.398+0000 7efc9dd05700 1 -- 172.21.15.189:0/1257722516 learned_addr learned my addr 172.21.15.189:0/1257722516 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:04.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.398+0000 7efc9e506700 1 --2- 172.21.15.189:0/1257722516 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc98158fe0 0x7efc9806a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:04.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.399+0000 7efc9dd05700 1 -- 172.21.15.189:0/1257722516 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc98158fe0 msgr2=0x7efc9806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:04.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.399+0000 7efc9dd05700 1 --2- 172.21.15.189:0/1257722516 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc98158fe0 0x7efc9806a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.399+0000 7efc9dd05700 1 -- 172.21.15.189:0/1257722516 >> v1:172.21.15.192:6789/0 conn(0x7efc9815c100 legacy=0x7efc980717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:04.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.399+0000 7efc9dd05700 1 -- 172.21.15.189:0/1257722516 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9815c8f0 con 0x7efc9806ac70 2024-04-06T13:48:04.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.399+0000 7efc9e506700 1 --2- 172.21.15.189:0/1257722516 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc98158fe0 0x7efc9806a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:04.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.399+0000 7efc9dd05700 1 --2- 172.21.15.189:0/1257722516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9806ac70 0x7efc9815bbc0 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7efc940098f0 tx=0x7efc94009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edc3671c3842e81d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:04.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.399+0000 7efc9d504700 1 -- 172.21.15.189:0/1257722516 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9400b8a0 con 0x7efc9806ac70 2024-04-06T13:48:04.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.399+0000 7efc9d504700 1 -- 172.21.15.189:0/1257722516 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9400ba00 con 0x7efc9806ac70 2024-04-06T13:48:04.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.399+0000 7efc9d504700 1 -- 172.21.15.189:0/1257722516 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9400bd90 con 0x7efc9806ac70 2024-04-06T13:48:04.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.399+0000 7efca076a700 1 -- 172.21.15.189:0/1257722516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9806ac70 msgr2=0x7efc9815bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:04.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.399+0000 7efca076a700 1 --2- 172.21.15.189:0/1257722516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9806ac70 0x7efc9815bbc0 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7efc940098f0 tx=0x7efc94009c00 comp rx=0 tx=0).stop 2024-04-06T13:48:04.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.399+0000 7efca076a700 1 -- 172.21.15.189:0/1257722516 shutdown_connections 2024-04-06T13:48:04.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.399+0000 7efca076a700 1 --2- 172.21.15.189:0/1257722516 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc98158fe0 0x7efc9806a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.399+0000 7efca076a700 1 --2- 172.21.15.189:0/1257722516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9806ac70 0x7efc9815bbc0 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.399+0000 7efca076a700 1 -- 172.21.15.189:0/1257722516 >> 172.21.15.189:0/1257722516 conn(0x7efc980d00b0 msgr2=0x7efc980d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:04.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.400+0000 7efca076a700 1 -- 172.21.15.189:0/1257722516 shutdown_connections 2024-04-06T13:48:04.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.400+0000 7efca076a700 1 -- 172.21.15.189:0/1257722516 wait complete. 2024-04-06T13:48:04.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.401+0000 7efca076a700 1 Processor -- start 2024-04-06T13:48:04.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.401+0000 7efca076a700 1 -- start start 2024-04-06T13:48:04.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.401+0000 7efca076a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9806ac70 0x7efc98157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:04.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.401+0000 7efca076a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc98158fe0 0x7efc9813eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:04.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.401+0000 7efca076a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc9815c100 0x7efc980ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:04.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.401+0000 7efca076a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc980b0840 con 0x7efc9806ac70 2024-04-06T13:48:04.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.401+0000 7efca076a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc980b0cd0 con 0x7efc9815c100 2024-04-06T13:48:04.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.401+0000 7efca076a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc980afc30 con 0x7efc98158fe0 2024-04-06T13:48:04.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.401+0000 7efc9e506700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9806ac70 0x7efc98157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:04.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.402+0000 7efc9e506700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9806ac70 0x7efc98157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41752/0 (socket says 172.21.15.189:41752) 2024-04-06T13:48:04.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.402+0000 7efc9e506700 1 -- 172.21.15.189:0/3484226282 learned_addr learned my addr 172.21.15.189:0/3484226282 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:04.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.402+0000 7efc9dd05700 1 --2- 172.21.15.189:0/3484226282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc98158fe0 0x7efc9813eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:04.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.402+0000 7efc9ed07700 1 --2- 172.21.15.189:0/3484226282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc9815c100 0x7efc980ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:04.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.402+0000 7efc9e506700 1 -- 172.21.15.189:0/3484226282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc98158fe0 msgr2=0x7efc9813eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:04.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.402+0000 7efc9e506700 1 --2- 172.21.15.189:0/3484226282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc98158fe0 0x7efc9813eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.402+0000 7efc9e506700 1 -- 172.21.15.189:0/3484226282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc9815c100 msgr2=0x7efc980ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:04.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.402+0000 7efc9e506700 1 --2- 172.21.15.189:0/3484226282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc9815c100 0x7efc980ccac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.402+0000 7efc9e506700 1 -- 172.21.15.189:0/3484226282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc94009550 con 0x7efc9806ac70 2024-04-06T13:48:04.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.402+0000 7efc9ed07700 1 --2- 172.21.15.189:0/3484226282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc9815c100 0x7efc980ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:04.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.402+0000 7efc9e506700 1 --2- 172.21.15.189:0/3484226282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9806ac70 0x7efc98157a90 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7efc8800e810 tx=0x7efc8800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:04.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.402+0000 7efc8effd700 1 -- 172.21.15.189:0/3484226282 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc88004230 con 0x7efc9806ac70 2024-04-06T13:48:04.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.402+0000 7efc8effd700 1 -- 172.21.15.189:0/3484226282 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc880043b0 con 0x7efc9806ac70 2024-04-06T13:48:04.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.402+0000 7efca076a700 1 -- 172.21.15.189:0/3484226282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc980aff10 con 0x7efc9806ac70 2024-04-06T13:48:04.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.402+0000 7efc8effd700 1 -- 172.21.15.189:0/3484226282 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc880038e0 con 0x7efc9806ac70 2024-04-06T13:48:04.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.402+0000 7efca076a700 1 -- 172.21.15.189:0/3484226282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc980b04d0 con 0x7efc9806ac70 2024-04-06T13:48:04.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.403+0000 7efca076a700 1 -- 172.21.15.189:0/3484226282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc980b0b20 con 0x7efc9806ac70 2024-04-06T13:48:04.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.404+0000 7efc8effd700 1 -- 172.21.15.189:0/3484226282 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7efc88003a40 con 0x7efc9806ac70 2024-04-06T13:48:04.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.404+0000 7efc8effd700 1 --2- 172.21.15.189:0/3484226282 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efc7c06aed0 0x7efc7c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:04.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.404+0000 7efc8effd700 1 -- 172.21.15.189:0/3484226282 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6930+0+0 (secure 0 0 0) 0x7efc88089760 con 0x7efc9806ac70 2024-04-06T13:48:04.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.404+0000 7efc9dd05700 1 --2- 172.21.15.189:0/3484226282 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efc7c06aed0 0x7efc7c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:04.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.404+0000 7efc9dd05700 1 --2- 172.21.15.189:0/3484226282 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efc7c06aed0 0x7efc7c06d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efc94014040 tx=0x7efc94008030 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:04.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.408+0000 7efc8effd700 1 -- 172.21.15.189:0/3484226282 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efc88054000 con 0x7efc9806ac70 2024-04-06T13:48:04.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.600+0000 7efca076a700 1 -- 172.21.15.189:0/3484226282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7efc98067bb0 con 0x7efc9806ac70 2024-04-06T13:48:04.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.600+0000 7efc8effd700 1 -- 172.21.15.189:0/3484226282 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v236) v1 ==== 103+0+11 (secure 0 0 0) 0x7efc880094c0 con 0x7efc9806ac70 2024-04-06T13:48:04.601 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:48:04.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.603+0000 7efca076a700 1 -- 172.21.15.189:0/3484226282 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efc7c06aed0 msgr2=0x7efc7c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:04.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.603+0000 7efca076a700 1 --2- 172.21.15.189:0/3484226282 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efc7c06aed0 0x7efc7c06d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efc94014040 tx=0x7efc94008030 comp rx=0 tx=0).stop 2024-04-06T13:48:04.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.603+0000 7efca076a700 1 -- 172.21.15.189:0/3484226282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9806ac70 msgr2=0x7efc98157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:04.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.603+0000 7efca076a700 1 --2- 172.21.15.189:0/3484226282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9806ac70 0x7efc98157a90 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7efc8800e810 tx=0x7efc8800eb20 comp rx=0 tx=0).stop 2024-04-06T13:48:04.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.603+0000 7efca076a700 1 -- 172.21.15.189:0/3484226282 shutdown_connections 2024-04-06T13:48:04.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.603+0000 7efca076a700 1 --2- 172.21.15.189:0/3484226282 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efc7c06aed0 0x7efc7c06d390 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.603+0000 7efca076a700 1 --2- 172.21.15.189:0/3484226282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc9815c100 0x7efc980ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.603+0000 7efca076a700 1 --2- 172.21.15.189:0/3484226282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9806ac70 0x7efc98157a90 unknown :-1 s=CLOSED pgs=2603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.603+0000 7efca076a700 1 --2- 172.21.15.189:0/3484226282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc98158fe0 0x7efc9813eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.603+0000 7efca076a700 1 -- 172.21.15.189:0/3484226282 >> 172.21.15.189:0/3484226282 conn(0x7efc980d00b0 msgr2=0x7efc981595d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:04.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.603+0000 7efca076a700 1 -- 172.21.15.189:0/3484226282 shutdown_connections 2024-04-06T13:48:04.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.603+0000 7efca076a700 1 -- 172.21.15.189:0/3484226282 wait complete. 2024-04-06T13:48:04.616 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:48:04.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.737+0000 7f9af358a700 1 Processor -- start 2024-04-06T13:48:04.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.737+0000 7f9af358a700 1 -- start start 2024-04-06T13:48:04.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.738+0000 7f9af358a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9aec06ae40 0x7f9aec0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:04.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.738+0000 7f9af358a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aec0b3990 con 0x7f9aec15d7b0 2024-04-06T13:48:04.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.738+0000 7f9af358a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aec0b3060 con 0x7f9aec15ddf0 2024-04-06T13:48:04.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.738+0000 7f9af358a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aec15e950 con 0x7f9aec06ae40 2024-04-06T13:48:04.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.738+0000 7f9af0b25700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9aec06ae40 0x7f9aec0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:04.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.738+0000 7f9af1326700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9aec15d7b0 0x7f9aec06a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50698/0 (socket says 172.21.15.189:50698) 2024-04-06T13:48:04.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.738+0000 7f9af0b25700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9aec06ae40 0x7f9aec0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41754/0 (socket says 172.21.15.189:41754) 2024-04-06T13:48:04.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.738+0000 7f9af1326700 1 -- 172.21.15.189:0/2568964089 learned_addr learned my addr 172.21.15.189:0/2568964089 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:04.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.738+0000 7f9af0b25700 1 -- 172.21.15.189:0/2568964089 >> v1:172.21.15.192:6789/0 conn(0x7f9aec15ddf0 legacy=0x7f9aec15e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:04.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.738+0000 7f9af0b25700 1 -- 172.21.15.189:0/2568964089 >> v1:172.21.15.189:6789/0 conn(0x7f9aec15d7b0 legacy=0x7f9aec06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:04.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.738+0000 7f9af0b25700 1 -- 172.21.15.189:0/2568964089 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aec0b3340 con 0x7f9aec06ae40 2024-04-06T13:48:04.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.738+0000 7f9ae3fff700 1 -- 172.21.15.189:0/2568964089 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766447404 0 0) 0x7f9aec0b3990 con 0x7f9aec15d7b0 2024-04-06T13:48:04.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.738+0000 7f9af0b25700 1 --2- 172.21.15.189:0/2568964089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9aec06ae40 0x7f9aec0b3e20 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009840 tx=0x7f9adc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e42b876c3b05338 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:04.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.739+0000 7f9ae3fff700 1 -- 172.21.15.189:0/2568964089 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc004230 con 0x7f9aec06ae40 2024-04-06T13:48:04.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.739+0000 7f9ae3fff700 1 -- 172.21.15.189:0/2568964089 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9adc004390 con 0x7f9aec06ae40 2024-04-06T13:48:04.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.739+0000 7f9ae3fff700 1 -- 172.21.15.189:0/2568964089 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc0036a0 con 0x7f9aec06ae40 2024-04-06T13:48:04.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.739+0000 7f9af358a700 1 -- 172.21.15.189:0/2568964089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9aec06ae40 msgr2=0x7f9aec0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:04.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.739+0000 7f9af358a700 1 --2- 172.21.15.189:0/2568964089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9aec06ae40 0x7f9aec0b3e20 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009840 tx=0x7f9adc009b50 comp rx=0 tx=0).stop 2024-04-06T13:48:04.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.739+0000 7f9af358a700 1 -- 172.21.15.189:0/2568964089 shutdown_connections 2024-04-06T13:48:04.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.739+0000 7f9af358a700 1 --2- 172.21.15.189:0/2568964089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9aec06ae40 0x7f9aec0b3e20 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.739+0000 7f9af358a700 1 -- 172.21.15.189:0/2568964089 >> 172.21.15.189:0/2568964089 conn(0x7f9aec0d00b0 msgr2=0x7f9aec0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:04.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.739+0000 7f9af358a700 1 -- 172.21.15.189:0/2568964089 shutdown_connections 2024-04-06T13:48:04.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.739+0000 7f9af358a700 1 -- 172.21.15.189:0/2568964089 wait complete. 2024-04-06T13:48:04.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.740+0000 7f9af358a700 1 Processor -- start 2024-04-06T13:48:04.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.740+0000 7f9af358a700 1 -- start start 2024-04-06T13:48:04.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.741+0000 7f9af358a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9aec15d7b0 0x7f9aec0af0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:04.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.741+0000 7f9af358a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9aec15ddf0 0x7f9aec15bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:04.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.741+0000 7f9af358a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9aec0afa50 0x7f9aec0afee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:04.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.741+0000 7f9af358a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aec0af5f0 con 0x7f9aec0afa50 2024-04-06T13:48:04.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.741+0000 7f9af358a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aec1f71a0 con 0x7f9aec15d7b0 2024-04-06T13:48:04.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.741+0000 7f9af358a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aec1f6620 con 0x7f9aec15ddf0 2024-04-06T13:48:04.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.741+0000 7f9af1b27700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9aec0afa50 0x7f9aec0afee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:04.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.741+0000 7f9af1b27700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9aec0afa50 0x7f9aec0afee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41756/0 (socket says 172.21.15.189:41756) 2024-04-06T13:48:04.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.741+0000 7f9af1b27700 1 -- 172.21.15.189:0/1808686282 learned_addr learned my addr 172.21.15.189:0/1808686282 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:04.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.741+0000 7f9af1326700 1 --2- 172.21.15.189:0/1808686282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9aec15d7b0 0x7f9aec0af0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:04.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.741+0000 7f9af0b25700 1 --2- 172.21.15.189:0/1808686282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9aec15ddf0 0x7f9aec15bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:04.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.741+0000 7f9af1b27700 1 -- 172.21.15.189:0/1808686282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9aec15ddf0 msgr2=0x7f9aec15bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:04.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.741+0000 7f9af1b27700 1 --2- 172.21.15.189:0/1808686282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9aec15ddf0 0x7f9aec15bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.742+0000 7f9af1b27700 1 -- 172.21.15.189:0/1808686282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9aec15d7b0 msgr2=0x7f9aec0af0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:04.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.742+0000 7f9af1b27700 1 --2- 172.21.15.189:0/1808686282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9aec15d7b0 0x7f9aec0af0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.742+0000 7f9af1b27700 1 -- 172.21.15.189:0/1808686282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9adc0094a0 con 0x7f9aec0afa50 2024-04-06T13:48:04.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.742+0000 7f9af1b27700 1 --2- 172.21.15.189:0/1808686282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9aec0afa50 0x7f9aec0afee0 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f9ae400b970 tx=0x7f9ae400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:04.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.742+0000 7f9ae1ffb700 1 -- 172.21.15.189:0/1808686282 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae400d860 con 0x7f9aec0afa50 2024-04-06T13:48:04.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.742+0000 7f9ae1ffb700 1 -- 172.21.15.189:0/1808686282 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae400d9c0 con 0x7f9aec0afa50 2024-04-06T13:48:04.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.742+0000 7f9af358a700 1 -- 172.21.15.189:0/1808686282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aec1f6900 con 0x7f9aec0afa50 2024-04-06T13:48:04.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.742+0000 7f9ae1ffb700 1 -- 172.21.15.189:0/1808686282 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae400dd70 con 0x7f9aec0afa50 2024-04-06T13:48:04.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.742+0000 7f9af358a700 1 -- 172.21.15.189:0/1808686282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aec1f6ec0 con 0x7f9aec0afa50 2024-04-06T13:48:04.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.743+0000 7f9af358a700 1 -- 172.21.15.189:0/1808686282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aec0b3e00 con 0x7f9aec0afa50 2024-04-06T13:48:04.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.743+0000 7f9ae1ffb700 1 -- 172.21.15.189:0/1808686282 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f9ae400e430 con 0x7f9aec0afa50 2024-04-06T13:48:04.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.744+0000 7f9ae1ffb700 1 --2- 172.21.15.189:0/1808686282 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9ad006aed0 0x7f9ad006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:04.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.744+0000 7f9ae1ffb700 1 -- 172.21.15.189:0/1808686282 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6930+0+0 (secure 0 0 0) 0x7f9ae4089df0 con 0x7f9aec0afa50 2024-04-06T13:48:04.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.744+0000 7f9af1326700 1 --2- 172.21.15.189:0/1808686282 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9ad006aed0 0x7f9ad006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:04.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.744+0000 7f9af1326700 1 --2- 172.21.15.189:0/1808686282 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9ad006aed0 0x7f9ad006d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8007b10 tx=0x7f9ae8007a60 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:04.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.748+0000 7f9ae1ffb700 1 -- 172.21.15.189:0/1808686282 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9ae4054720 con 0x7f9aec0afa50 2024-04-06T13:48:04.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.986+0000 7f9af358a700 1 -- 172.21.15.189:0/1808686282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9aec074990 con 0x7f9aec0afa50 2024-04-06T13:48:04.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.987+0000 7f9ae1ffb700 1 -- 172.21.15.189:0/1808686282 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+4759 (secure 0 0 0) 0x7f9ae401c020 con 0x7f9aec0afa50 2024-04-06T13:48:04.988 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:04.988 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":108,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8885,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1481817364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1481817364},{"type":"v1","addr":"172.21.15.189:6837","nonce":1481817364}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":10639,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/573594428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":573594428},{"type":"v1","addr":"172.21.15.192:6835","nonce":573594428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":108},{"gid":11244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/4021875447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":4021875447},{"type":"v1","addr":"172.21.15.189:6836","nonce":4021875447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:47:34.584208+0000","modified":"2024-04-06T13:47:37.592983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11262":{"gid":11262,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6834/4066431971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4066431971},{"type":"v1","addr":"172.21.15.192:6834","nonce":4066431971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-04-06T13:48:04.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.989+0000 7f9af358a700 1 -- 172.21.15.189:0/1808686282 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9ad006aed0 msgr2=0x7f9ad006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:04.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.989+0000 7f9af358a700 1 --2- 172.21.15.189:0/1808686282 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9ad006aed0 0x7f9ad006d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8007b10 tx=0x7f9ae8007a60 comp rx=0 tx=0).stop 2024-04-06T13:48:04.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.989+0000 7f9af358a700 1 -- 172.21.15.189:0/1808686282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9aec0afa50 msgr2=0x7f9aec0afee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:04.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.989+0000 7f9af358a700 1 --2- 172.21.15.189:0/1808686282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9aec0afa50 0x7f9aec0afee0 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f9ae400b970 tx=0x7f9ae400bc80 comp rx=0 tx=0).stop 2024-04-06T13:48:04.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.989+0000 7f9af358a700 1 -- 172.21.15.189:0/1808686282 shutdown_connections 2024-04-06T13:48:04.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.989+0000 7f9af358a700 1 --2- 172.21.15.189:0/1808686282 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9ad006aed0 0x7f9ad006d390 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.989+0000 7f9af358a700 1 --2- 172.21.15.189:0/1808686282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9aec0afa50 0x7f9aec0afee0 unknown :-1 s=CLOSED pgs=2606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.989+0000 7f9af358a700 1 --2- 172.21.15.189:0/1808686282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9aec15d7b0 0x7f9aec0af0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.989+0000 7f9af358a700 1 --2- 172.21.15.189:0/1808686282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9aec15ddf0 0x7f9aec15bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:04.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.989+0000 7f9af358a700 1 -- 172.21.15.189:0/1808686282 >> 172.21.15.189:0/1808686282 conn(0x7f9aec0d00b0 msgr2=0x7f9aec0ae060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:04.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.989+0000 7f9af358a700 1 -- 172.21.15.189:0/1808686282 shutdown_connections 2024-04-06T13:48:04.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:04.989+0000 7f9af358a700 1 -- 172.21.15.189:0/1808686282 wait complete. 2024-04-06T13:48:04.991 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 108 2024-04-06T13:48:05.003 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:48:05.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.123+0000 7f237e4d7700 1 Processor -- start 2024-04-06T13:48:05.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.123+0000 7f237e4d7700 1 -- start start 2024-04-06T13:48:05.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.124+0000 7f237e4d7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23780cb870 0x7f237815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:05.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.124+0000 7f237e4d7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f23780cfa00 0x7f23780cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:05.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.124+0000 7f237e4d7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23780cbd80 con 0x7f23780d11b0 2024-04-06T13:48:05.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.124+0000 7f237e4d7700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23780ccdc0 con 0x7f23780cfa00 2024-04-06T13:48:05.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.124+0000 7f237e4d7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23780d0b50 con 0x7f23780cb870 2024-04-06T13:48:05.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.124+0000 7f237ca74700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f23780cfa00 0x7f23780cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:05.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.124+0000 7f2377fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f23780d11b0 0x7f23780d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50712/0 (socket says 172.21.15.189:50712) 2024-04-06T13:48:05.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.124+0000 7f237ca74700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f23780cfa00 0x7f23780cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41764/0 (socket says 172.21.15.189:41764) 2024-04-06T13:48:05.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.124+0000 7f2377fff700 1 -- 172.21.15.189:0/2728355946 learned_addr learned my addr 172.21.15.189:0/2728355946 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:05.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.124+0000 7f23777fe700 1 --2- 172.21.15.189:0/2728355946 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23780cb870 0x7f237815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:05.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.124+0000 7f237ca74700 1 -- 172.21.15.189:0/2728355946 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23780cb870 msgr2=0x7f237815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:05.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.124+0000 7f237ca74700 1 --2- 172.21.15.189:0/2728355946 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23780cb870 0x7f237815c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.124+0000 7f237ca74700 1 -- 172.21.15.189:0/2728355946 >> v1:172.21.15.189:6789/0 conn(0x7f23780d11b0 legacy=0x7f23780d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:05.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.124+0000 7f237ca74700 1 -- 172.21.15.189:0/2728355946 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23780cef20 con 0x7f23780cfa00 2024-04-06T13:48:05.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.124+0000 7f2376ffd700 1 -- 172.21.15.189:0/2728355946 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645856028 0 0) 0x7f23780cbd80 con 0x7f23780d11b0 2024-04-06T13:48:05.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.124+0000 7f237ca74700 1 --2- 172.21.15.189:0/2728355946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23780cfa00 0x7f23780cd560 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f2370009d90 tx=0x7f237000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e4413e0f5db38eb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:05.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.125+0000 7f2376ffd700 1 -- 172.21.15.189:0/2728355946 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237000e070 con 0x7f23780cfa00 2024-04-06T13:48:05.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.125+0000 7f2376ffd700 1 -- 172.21.15.189:0/2728355946 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237000bbd0 con 0x7f23780cfa00 2024-04-06T13:48:05.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.125+0000 7f2376ffd700 1 -- 172.21.15.189:0/2728355946 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2370012430 con 0x7f23780cfa00 2024-04-06T13:48:05.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.125+0000 7f237e4d7700 1 -- 172.21.15.189:0/2728355946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23780cfa00 msgr2=0x7f23780cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:05.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.125+0000 7f237e4d7700 1 --2- 172.21.15.189:0/2728355946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23780cfa00 0x7f23780cd560 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f2370009d90 tx=0x7f237000b330 comp rx=0 tx=0).stop 2024-04-06T13:48:05.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.125+0000 7f237e4d7700 1 -- 172.21.15.189:0/2728355946 shutdown_connections 2024-04-06T13:48:05.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.125+0000 7f237e4d7700 1 --2- 172.21.15.189:0/2728355946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23780cfa00 0x7f23780cd560 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.125+0000 7f237e4d7700 1 --2- 172.21.15.189:0/2728355946 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23780cb870 0x7f237815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.125+0000 7f237e4d7700 1 -- 172.21.15.189:0/2728355946 >> 172.21.15.189:0/2728355946 conn(0x7f23780b2db0 msgr2=0x7f23780b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:05.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.125+0000 7f237e4d7700 1 -- 172.21.15.189:0/2728355946 shutdown_connections 2024-04-06T13:48:05.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.125+0000 7f237e4d7700 1 -- 172.21.15.189:0/2728355946 wait complete. 2024-04-06T13:48:05.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.126+0000 7f237e4d7700 1 Processor -- start 2024-04-06T13:48:05.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.127+0000 7f237e4d7700 1 -- start start 2024-04-06T13:48:05.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.127+0000 7f237e4d7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23780cb870 0x7f237806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:05.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.127+0000 7f237e4d7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23780cfa00 0x7f23780c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:05.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.127+0000 7f237e4d7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23780d11b0 0x7f237815ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:05.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.127+0000 7f237e4d7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237806d8d0 con 0x7f23780d11b0 2024-04-06T13:48:05.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.127+0000 7f237e4d7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23781f3ab0 con 0x7f23780cfa00 2024-04-06T13:48:05.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.127+0000 7f237e4d7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23781f2f30 con 0x7f23780cb870 2024-04-06T13:48:05.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.127+0000 7f237ca74700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23780d11b0 0x7f237815ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:05.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.127+0000 7f237ca74700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23780d11b0 0x7f237815ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41776/0 (socket says 172.21.15.189:41776) 2024-04-06T13:48:05.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.127+0000 7f237ca74700 1 -- 172.21.15.189:0/3650321001 learned_addr learned my addr 172.21.15.189:0/3650321001 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:05.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.128+0000 7f23777fe700 1 --2- 172.21.15.189:0/3650321001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23780cfa00 0x7f23780c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:05.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.128+0000 7f2377fff700 1 --2- 172.21.15.189:0/3650321001 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23780cb870 0x7f237806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:05.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.128+0000 7f237ca74700 1 -- 172.21.15.189:0/3650321001 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23780cb870 msgr2=0x7f237806d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:05.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.128+0000 7f237ca74700 1 --2- 172.21.15.189:0/3650321001 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23780cb870 0x7f237806d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.128+0000 7f237ca74700 1 -- 172.21.15.189:0/3650321001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23780cfa00 msgr2=0x7f23780c9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:05.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.128+0000 7f237ca74700 1 --2- 172.21.15.189:0/3650321001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23780cfa00 0x7f23780c9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.128+0000 7f237ca74700 1 -- 172.21.15.189:0/3650321001 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2370009a70 con 0x7f23780d11b0 2024-04-06T13:48:05.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.128+0000 7f23777fe700 1 --2- 172.21.15.189:0/3650321001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23780cfa00 0x7f23780c9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:05.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.128+0000 7f2377fff700 1 --2- 172.21.15.189:0/3650321001 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23780cb870 0x7f237806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:05.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.128+0000 7f237ca74700 1 --2- 172.21.15.189:0/3650321001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23780d11b0 0x7f237815ee70 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f23780677e0 tx=0x7f2370004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:05.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.128+0000 7f2374ff9700 1 -- 172.21.15.189:0/3650321001 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2370012430 con 0x7f23780d11b0 2024-04-06T13:48:05.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.128+0000 7f2374ff9700 1 -- 172.21.15.189:0/3650321001 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2370020070 con 0x7f23780d11b0 2024-04-06T13:48:05.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.128+0000 7f2374ff9700 1 -- 172.21.15.189:0/3650321001 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2370004280 con 0x7f23780d11b0 2024-04-06T13:48:05.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.128+0000 7f237e4d7700 1 -- 172.21.15.189:0/3650321001 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23781f3210 con 0x7f23780d11b0 2024-04-06T13:48:05.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.128+0000 7f237e4d7700 1 -- 172.21.15.189:0/3650321001 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23781f37d0 con 0x7f23780d11b0 2024-04-06T13:48:05.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.129+0000 7f237e4d7700 1 -- 172.21.15.189:0/3650321001 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23781f61d0 con 0x7f23780d11b0 2024-04-06T13:48:05.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.129+0000 7f2374ff9700 1 -- 172.21.15.189:0/3650321001 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f23700043e0 con 0x7f23780d11b0 2024-04-06T13:48:05.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.130+0000 7f2374ff9700 1 --2- 172.21.15.189:0/3650321001 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f236406aed0 0x7f236406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:05.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.130+0000 7f2374ff9700 1 -- 172.21.15.189:0/3650321001 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6930+0+0 (secure 0 0 0) 0x7f2370022070 con 0x7f23780d11b0 2024-04-06T13:48:05.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.130+0000 7f2377fff700 1 --2- 172.21.15.189:0/3650321001 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f236406aed0 0x7f236406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:05.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.130+0000 7f2377fff700 1 --2- 172.21.15.189:0/3650321001 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f236406aed0 0x7f236406d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f236c000f60 tx=0x7f236c009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:05.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.134+0000 7f2374ff9700 1 -- 172.21.15.189:0/3650321001 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f23781f61d0 con 0x7f23780d11b0 2024-04-06T13:48:05.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.330+0000 7f237e4d7700 1 -- 172.21.15.189:0/3650321001 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23781f61d0 con 0x7f23780d11b0 2024-04-06T13:48:05.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.331+0000 7f2374ff9700 1 -- 172.21.15.189:0/3650321001 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v236) v1 ==== 74+0+17619 (secure 0 0 0) 0x7f23781f61d0 con 0x7f23780d11b0 2024-04-06T13:48:05.332 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:05.332 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":236,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:48:03.192350+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:47:32.292623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:47:33.567052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.192:6833/694564673":"2024-04-07T13:48:00.073095+0000","172.21.15.189:0/3017776380":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1992839362":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/579471530":"2024-04-07T13:47:57.909304+0000","172.21.15.192:6835/694564673":"2024-04-07T13:48:00.073095+0000","172.21.15.189:0/4049616553":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/619264656":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1506469869":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1831656794":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1190635801":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/3920072072":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/1878657401":"2024-04-07T13:47:47.129296+0000","172.21.15.189:6833/111704":"2024-04-07T13:47:47.129296+0000","172.21.15.189:6832/111704":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/2827130089":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/2716712615":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/1807905335":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/914480885":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1173010452":"2024-04-07T13:47:47.129296+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:48:05.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.334+0000 7f237e4d7700 1 -- 172.21.15.189:0/3650321001 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f236406aed0 msgr2=0x7f236406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:05.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.334+0000 7f237e4d7700 1 --2- 172.21.15.189:0/3650321001 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f236406aed0 0x7f236406d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f236c000f60 tx=0x7f236c009040 comp rx=0 tx=0).stop 2024-04-06T13:48:05.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.334+0000 7f237e4d7700 1 -- 172.21.15.189:0/3650321001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23780d11b0 msgr2=0x7f237815ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:05.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.334+0000 7f237e4d7700 1 --2- 172.21.15.189:0/3650321001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23780d11b0 0x7f237815ee70 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f23780677e0 tx=0x7f2370004ad0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.334+0000 7f237e4d7700 1 -- 172.21.15.189:0/3650321001 shutdown_connections 2024-04-06T13:48:05.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.334+0000 7f237e4d7700 1 --2- 172.21.15.189:0/3650321001 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f236406aed0 0x7f236406d390 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.334+0000 7f237e4d7700 1 --2- 172.21.15.189:0/3650321001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23780d11b0 0x7f237815ee70 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.334+0000 7f237e4d7700 1 --2- 172.21.15.189:0/3650321001 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23780cb870 0x7f237806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.334+0000 7f237e4d7700 1 --2- 172.21.15.189:0/3650321001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23780cfa00 0x7f23780c9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.334+0000 7f237e4d7700 1 -- 172.21.15.189:0/3650321001 >> 172.21.15.189:0/3650321001 conn(0x7f23780b2db0 msgr2=0x7f23780b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:05.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.334+0000 7f237e4d7700 1 -- 172.21.15.189:0/3650321001 shutdown_connections 2024-04-06T13:48:05.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.334+0000 7f237e4d7700 1 -- 172.21.15.189:0/3650321001 wait complete. 2024-04-06T13:48:05.347 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2024-04-06T13:48:05.347 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:48:05.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.468+0000 7f09266ad700 1 Processor -- start 2024-04-06T13:48:05.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.468+0000 7f09266ad700 1 -- start start 2024-04-06T13:48:05.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.469+0000 7f09266ad700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f092015a550 0x7f09200cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:05.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.469+0000 7f09266ad700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09200b4d70 0x7f09200b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:05.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.470+0000 7f09266ad700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09200cdbe0 con 0x7f09200cf900 2024-04-06T13:48:05.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.470+0000 7f09266ad700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f092015a940 con 0x7f092015a550 2024-04-06T13:48:05.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.470+0000 7f09266ad700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f092015ac00 con 0x7f09200b4d70 2024-04-06T13:48:05.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.470+0000 7f091ffff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f092015a550 0x7f09200cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:05.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.470+0000 7f0924c4a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09200b4d70 0x7f09200b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:05.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.470+0000 7f091f7fe700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f09200cf900 0x7f09200b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:32770/0 (socket says 172.21.15.189:32770) 2024-04-06T13:48:05.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.470+0000 7f091f7fe700 1 -- 172.21.15.189:0/3194149071 learned_addr learned my addr 172.21.15.189:0/3194149071 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:05.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.470+0000 7f091ffff700 1 -- 172.21.15.189:0/3194149071 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09200b4d70 msgr2=0x7f09200b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:05.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.470+0000 7f091ffff700 1 --2- 172.21.15.189:0/3194149071 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09200b4d70 0x7f09200b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.470+0000 7f091ffff700 1 -- 172.21.15.189:0/3194149071 >> v1:172.21.15.192:6789/0 conn(0x7f09200cf900 legacy=0x7f09200b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:05.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.470+0000 7f091ffff700 1 -- 172.21.15.189:0/3194149071 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092015aee0 con 0x7f092015a550 2024-04-06T13:48:05.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.470+0000 7f0924c4a700 1 --2- 172.21.15.189:0/3194149071 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09200b4d70 0x7f09200b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:05.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.471+0000 7f091ffff700 1 --2- 172.21.15.189:0/3194149071 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f092015a550 0x7f09200cf3c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f0914009840 tx=0x7f0914009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e18cf491d3e8e00d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:05.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.471+0000 7f091effd700 1 -- 172.21.15.189:0/3194149071 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0914011830 con 0x7f092015a550 2024-04-06T13:48:05.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.471+0000 7f091effd700 1 -- 172.21.15.189:0/3194149071 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0914011990 con 0x7f092015a550 2024-04-06T13:48:05.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.471+0000 7f091effd700 1 -- 172.21.15.189:0/3194149071 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0914011d20 con 0x7f092015a550 2024-04-06T13:48:05.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.471+0000 7f09266ad700 1 -- 172.21.15.189:0/3194149071 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f092015a550 msgr2=0x7f09200cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:05.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.471+0000 7f09266ad700 1 --2- 172.21.15.189:0/3194149071 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f092015a550 0x7f09200cf3c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f0914009840 tx=0x7f0914009b50 comp rx=0 tx=0).stop 2024-04-06T13:48:05.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.471+0000 7f09266ad700 1 -- 172.21.15.189:0/3194149071 shutdown_connections 2024-04-06T13:48:05.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.471+0000 7f09266ad700 1 --2- 172.21.15.189:0/3194149071 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09200b4d70 0x7f09200b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.471+0000 7f09266ad700 1 --2- 172.21.15.189:0/3194149071 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f092015a550 0x7f09200cf3c0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.471+0000 7f09266ad700 1 -- 172.21.15.189:0/3194149071 >> 172.21.15.189:0/3194149071 conn(0x7f09200d00b0 msgr2=0x7f09200d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:05.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.471+0000 7f09266ad700 1 -- 172.21.15.189:0/3194149071 shutdown_connections 2024-04-06T13:48:05.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.471+0000 7f09266ad700 1 -- 172.21.15.189:0/3194149071 wait complete. 2024-04-06T13:48:05.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.473+0000 7f09266ad700 1 Processor -- start 2024-04-06T13:48:05.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.473+0000 7f09266ad700 1 -- start start 2024-04-06T13:48:05.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.474+0000 7f09266ad700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09200b4d70 0x7f092014fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:05.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.474+0000 7f09266ad700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09200cf900 0x7f09201586c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:05.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.474+0000 7f09266ad700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f092006a730 0x7f092006abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:05.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.474+0000 7f09266ad700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092006b120 con 0x7f09200cf900 2024-04-06T13:48:05.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.474+0000 7f09266ad700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0920150f50 con 0x7f09200b4d70 2024-04-06T13:48:05.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.474+0000 7f09266ad700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0920150340 con 0x7f092006a730 2024-04-06T13:48:05.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.474+0000 7f091f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09200cf900 0x7f09201586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:05.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.474+0000 7f091f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09200cf900 0x7f09201586c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41780/0 (socket says 172.21.15.189:41780) 2024-04-06T13:48:05.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.474+0000 7f091f7fe700 1 -- 172.21.15.189:0/807170056 learned_addr learned my addr 172.21.15.189:0/807170056 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:05.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.474+0000 7f091ffff700 1 --2- 172.21.15.189:0/807170056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09200b4d70 0x7f092014fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:05.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.474+0000 7f0924c4a700 1 --2- 172.21.15.189:0/807170056 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f092006a730 0x7f092006abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:05.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.474+0000 7f091f7fe700 1 -- 172.21.15.189:0/807170056 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f092006a730 msgr2=0x7f092006abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:05.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.474+0000 7f091f7fe700 1 --2- 172.21.15.189:0/807170056 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f092006a730 0x7f092006abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.474+0000 7f091f7fe700 1 -- 172.21.15.189:0/807170056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09200b4d70 msgr2=0x7f092014fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:05.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.474+0000 7f091f7fe700 1 --2- 172.21.15.189:0/807170056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09200b4d70 0x7f092014fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.474+0000 7f091f7fe700 1 -- 172.21.15.189:0/807170056 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0918009a70 con 0x7f09200cf900 2024-04-06T13:48:05.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.475+0000 7f091f7fe700 1 --2- 172.21.15.189:0/807170056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09200cf900 0x7f09201586c0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f091000c490 tx=0x7f091000c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:05.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.475+0000 7f091cff9700 1 -- 172.21.15.189:0/807170056 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091000cea0 con 0x7f09200cf900 2024-04-06T13:48:05.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.475+0000 7f091cff9700 1 -- 172.21.15.189:0/807170056 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0910005d70 con 0x7f09200cf900 2024-04-06T13:48:05.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.475+0000 7f09266ad700 1 -- 172.21.15.189:0/807170056 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09140094a0 con 0x7f09200cf900 2024-04-06T13:48:05.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.475+0000 7f091cff9700 1 -- 172.21.15.189:0/807170056 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09100072f0 con 0x7f09200cf900 2024-04-06T13:48:05.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.475+0000 7f09266ad700 1 -- 172.21.15.189:0/807170056 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0920150900 con 0x7f09200cf900 2024-04-06T13:48:05.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.476+0000 7f09266ad700 1 -- 172.21.15.189:0/807170056 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0920151100 con 0x7f09200cf900 2024-04-06T13:48:05.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.476+0000 7f091cff9700 1 -- 172.21.15.189:0/807170056 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f0910007450 con 0x7f09200cf900 2024-04-06T13:48:05.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.477+0000 7f091cff9700 1 --2- 172.21.15.189:0/807170056 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f090c06aed0 0x7f090c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:05.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.477+0000 7f091cff9700 1 -- 172.21.15.189:0/807170056 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6930+0+0 (secure 0 0 0) 0x7f0910088fc0 con 0x7f09200cf900 2024-04-06T13:48:05.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.477+0000 7f091ffff700 1 --2- 172.21.15.189:0/807170056 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f090c06aed0 0x7f090c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:05.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.477+0000 7f091ffff700 1 --2- 172.21.15.189:0/807170056 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f090c06aed0 0x7f090c06d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f09140090b0 tx=0x7f091401c040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:05.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.481+0000 7f091cff9700 1 -- 172.21.15.189:0/807170056 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f09100bc650 con 0x7f09200cf900 2024-04-06T13:48:05.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.718+0000 7f09266ad700 1 -- 172.21.15.189:0/807170056 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0920067bb0 con 0x7f09200cf900 2024-04-06T13:48:05.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.719+0000 7f091cff9700 1 -- 172.21.15.189:0/807170056 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+4759 (secure 0 0 0) 0x7f0910017020 con 0x7f09200cf900 2024-04-06T13:48:05.720 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:05.720 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":108,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8885,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1481817364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1481817364},{"type":"v1","addr":"172.21.15.189:6837","nonce":1481817364}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":10639,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/573594428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":573594428},{"type":"v1","addr":"172.21.15.192:6835","nonce":573594428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":108},{"gid":11244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/4021875447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":4021875447},{"type":"v1","addr":"172.21.15.189:6836","nonce":4021875447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:47:34.584208+0000","modified":"2024-04-06T13:47:37.592983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11262":{"gid":11262,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6834/4066431971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4066431971},{"type":"v1","addr":"172.21.15.192:6834","nonce":4066431971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-04-06T13:48:05.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.721+0000 7f09266ad700 1 -- 172.21.15.189:0/807170056 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f090c06aed0 msgr2=0x7f090c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:05.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.721+0000 7f09266ad700 1 --2- 172.21.15.189:0/807170056 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f090c06aed0 0x7f090c06d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f09140090b0 tx=0x7f091401c040 comp rx=0 tx=0).stop 2024-04-06T13:48:05.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.721+0000 7f09266ad700 1 -- 172.21.15.189:0/807170056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09200cf900 msgr2=0x7f09201586c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:05.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.721+0000 7f09266ad700 1 --2- 172.21.15.189:0/807170056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09200cf900 0x7f09201586c0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f091000c490 tx=0x7f091000c7a0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.721+0000 7f09266ad700 1 -- 172.21.15.189:0/807170056 shutdown_connections 2024-04-06T13:48:05.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.721+0000 7f09266ad700 1 --2- 172.21.15.189:0/807170056 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f090c06aed0 0x7f090c06d390 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.721+0000 7f09266ad700 1 --2- 172.21.15.189:0/807170056 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f092006a730 0x7f092006abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.721+0000 7f09266ad700 1 --2- 172.21.15.189:0/807170056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09200b4d70 0x7f092014fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.721+0000 7f09266ad700 1 --2- 172.21.15.189:0/807170056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09200cf900 0x7f09201586c0 unknown :-1 s=CLOSED pgs=2610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.721+0000 7f09266ad700 1 -- 172.21.15.189:0/807170056 >> 172.21.15.189:0/807170056 conn(0x7f09200d00b0 msgr2=0x7f09200b3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:05.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.721+0000 7f09266ad700 1 -- 172.21.15.189:0/807170056 shutdown_connections 2024-04-06T13:48:05.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.721+0000 7f09266ad700 1 -- 172.21.15.189:0/807170056 wait complete. 2024-04-06T13:48:05.723 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 108 2024-04-06T13:48:05.734 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:48:05.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.855+0000 7fa7497c4700 1 Processor -- start 2024-04-06T13:48:05.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.855+0000 7fa7497c4700 1 -- start start 2024-04-06T13:48:05.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.856+0000 7fa7497c4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa7440b4af0 0x7fa7440b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:05.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.856+0000 7fa7497c4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7440cdbe0 con 0x7fa7440b4660 2024-04-06T13:48:05.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.856+0000 7fa7497c4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa74415a940 con 0x7fa74415a550 2024-04-06T13:48:05.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.856+0000 7fa7497c4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa74415ac00 con 0x7fa7440b4af0 2024-04-06T13:48:05.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.856+0000 7fa7427fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa7440b4660 0x7fa7440b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50718/0 (socket says 172.21.15.189:50718) 2024-04-06T13:48:05.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.856+0000 7fa7427fc700 1 -- 172.21.15.189:0/572857806 learned_addr learned my addr 172.21.15.189:0/572857806 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:05.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.856+0000 7fa7437fe700 1 --2- 172.21.15.189:0/572857806 >> v2:172.21.15.192:3300/0 conn(0x7fa7440b4af0 0x7fa7440b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:05.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.857+0000 7fa741ffb700 1 -- 172.21.15.189:0/572857806 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076158119 0 0) 0x7fa7440cdbe0 con 0x7fa7440b4660 2024-04-06T13:48:05.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.857+0000 7fa741ffb700 1 -- 172.21.15.189:0/572857806 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa72c0035b0 con 0x7fa7440b4660 2024-04-06T13:48:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.857+0000 7fa741ffb700 1 -- 172.21.15.189:0/572857806 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 515281558 0 0) 0x7fa72c0035b0 con 0x7fa7440b4660 2024-04-06T13:48:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.857+0000 7fa7437fe700 1 -- 172.21.15.189:0/572857806 >> v1:172.21.15.192:6789/0 conn(0x7fa74415a550 legacy=0x7fa7440cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.857+0000 7fa7437fe700 1 -- 172.21.15.189:0/572857806 >> v1:172.21.15.189:6789/0 conn(0x7fa7440b4660 legacy=0x7fa7440b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.857+0000 7fa7437fe700 1 -- 172.21.15.189:0/572857806 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74415aee0 con 0x7fa7440b4af0 2024-04-06T13:48:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.857+0000 7fa7437fe700 1 --2- 172.21.15.189:0/572857806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7440b4af0 0x7fa7440b36d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009d90 tx=0x7fa73c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c5d9a9ef92f63e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.857+0000 7fa741ffb700 1 -- 172.21.15.189:0/572857806 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c00e070 con 0x7fa7440b4af0 2024-04-06T13:48:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.857+0000 7fa741ffb700 1 -- 172.21.15.189:0/572857806 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73c003a20 con 0x7fa7440b4af0 2024-04-06T13:48:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.857+0000 7fa741ffb700 1 -- 172.21.15.189:0/572857806 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa734001670 con 0x7fa7440b4af0 2024-04-06T13:48:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.858+0000 7fa7497c4700 1 -- 172.21.15.189:0/572857806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7440b4af0 msgr2=0x7fa7440b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.858+0000 7fa7497c4700 1 --2- 172.21.15.189:0/572857806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7440b4af0 0x7fa7440b36d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009d90 tx=0x7fa73c00b330 comp rx=0 tx=0).stop 2024-04-06T13:48:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.858+0000 7fa7497c4700 1 -- 172.21.15.189:0/572857806 shutdown_connections 2024-04-06T13:48:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.858+0000 7fa7497c4700 1 --2- 172.21.15.189:0/572857806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7440b4af0 0x7fa7440b36d0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.858+0000 7fa7497c4700 1 -- 172.21.15.189:0/572857806 >> 172.21.15.189:0/572857806 conn(0x7fa7440d00b0 msgr2=0x7fa7440d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.858+0000 7fa7497c4700 1 -- 172.21.15.189:0/572857806 shutdown_connections 2024-04-06T13:48:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.858+0000 7fa7497c4700 1 -- 172.21.15.189:0/572857806 wait complete. 2024-04-06T13:48:05.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.859+0000 7fa7497c4700 1 Processor -- start 2024-04-06T13:48:05.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.860+0000 7fa7497c4700 1 -- start start 2024-04-06T13:48:05.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.860+0000 7fa7497c4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7440b4660 0x7fa74414cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:05.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.860+0000 7fa7497c4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7440b4af0 0x7fa7440cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:05.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.860+0000 7fa7497c4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7440ce2e0 0x7fa7440cb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:05.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.860+0000 7fa7497c4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7441f39c0 con 0x7fa7440b4660 2024-04-06T13:48:05.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.860+0000 7fa7497c4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7440ce760 con 0x7fa7440b4af0 2024-04-06T13:48:05.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.860+0000 7fa7497c4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7441f2af0 con 0x7fa7440ce2e0 2024-04-06T13:48:05.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.861+0000 7fa742ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7440b4660 0x7fa74414cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:05.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.861+0000 7fa742ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7440b4660 0x7fa74414cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41784/0 (socket says 172.21.15.189:41784) 2024-04-06T13:48:05.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.861+0000 7fa742ffd700 1 -- 172.21.15.189:0/1734638245 learned_addr learned my addr 172.21.15.189:0/1734638245 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:05.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.861+0000 7fa7427fc700 1 --2- 172.21.15.189:0/1734638245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7440b4af0 0x7fa7440cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:05.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.861+0000 7fa7437fe700 1 --2- 172.21.15.189:0/1734638245 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7440ce2e0 0x7fa7440cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:05.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.861+0000 7fa742ffd700 1 -- 172.21.15.189:0/1734638245 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7440ce2e0 msgr2=0x7fa7440cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:05.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.861+0000 7fa742ffd700 1 --2- 172.21.15.189:0/1734638245 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7440ce2e0 0x7fa7440cb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.861+0000 7fa742ffd700 1 -- 172.21.15.189:0/1734638245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7440b4af0 msgr2=0x7fa7440cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:05.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.861+0000 7fa742ffd700 1 --2- 172.21.15.189:0/1734638245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7440b4af0 0x7fa7440cdda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:05.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.861+0000 7fa742ffd700 1 -- 172.21.15.189:0/1734638245 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73c009a70 con 0x7fa7440b4660 2024-04-06T13:48:05.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.861+0000 7fa7427fc700 1 --2- 172.21.15.189:0/1734638245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7440b4af0 0x7fa7440cdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:05.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.861+0000 7fa742ffd700 1 --2- 172.21.15.189:0/1734638245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7440b4660 0x7fa74414cb10 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7fa73800c550 tx=0x7fa73800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:05.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.861+0000 7fa72b7fe700 1 -- 172.21.15.189:0/1734638245 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7380054e0 con 0x7fa7440b4660 2024-04-06T13:48:05.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.861+0000 7fa72b7fe700 1 -- 172.21.15.189:0/1734638245 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa738003680 con 0x7fa7440b4660 2024-04-06T13:48:05.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.861+0000 7fa72b7fe700 1 -- 172.21.15.189:0/1734638245 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa738003a30 con 0x7fa7440b4660 2024-04-06T13:48:05.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.861+0000 7fa7497c4700 1 -- 172.21.15.189:0/1734638245 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7441f2d40 con 0x7fa7440b4660 2024-04-06T13:48:05.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.861+0000 7fa7497c4700 1 -- 172.21.15.189:0/1734638245 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7441f3300 con 0x7fa7440b4660 2024-04-06T13:48:05.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.862+0000 7fa7497c4700 1 -- 172.21.15.189:0/1734638245 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7441f3b00 con 0x7fa7440b4660 2024-04-06T13:48:05.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.863+0000 7fa72b7fe700 1 -- 172.21.15.189:0/1734638245 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fa738003b90 con 0x7fa7440b4660 2024-04-06T13:48:05.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.863+0000 7fa72b7fe700 1 --2- 172.21.15.189:0/1734638245 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa73006aed0 0x7fa73006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:05.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.863+0000 7fa72b7fe700 1 -- 172.21.15.189:0/1734638245 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6930+0+0 (secure 0 0 0) 0x7fa7380890e0 con 0x7fa7440b4660 2024-04-06T13:48:05.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.863+0000 7fa7427fc700 1 --2- 172.21.15.189:0/1734638245 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa73006aed0 0x7fa73006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:05.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.863+0000 7fa7427fc700 1 --2- 172.21.15.189:0/1734638245 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa73006aed0 0x7fa73006d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa734001eb0 tx=0x7fa734001ca0 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:05.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:05.868+0000 7fa72b7fe700 1 -- 172.21.15.189:0/1734638245 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa7441f3b00 con 0x7fa7440b4660 2024-04-06T13:48:06.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.107+0000 7fa7497c4700 1 -- 172.21.15.189:0/1734638245 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7441f3b00 con 0x7fa7440b4660 2024-04-06T13:48:06.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.108+0000 7fa72b7fe700 1 -- 172.21.15.189:0/1734638245 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+4759 (secure 0 0 0) 0x7fa7441f3b00 con 0x7fa7440b4660 2024-04-06T13:48:06.109 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:06.109 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":108,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8885,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1481817364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1481817364},{"type":"v1","addr":"172.21.15.189:6837","nonce":1481817364}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":10639,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/573594428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":573594428},{"type":"v1","addr":"172.21.15.192:6835","nonce":573594428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":108},{"gid":11244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/4021875447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":4021875447},{"type":"v1","addr":"172.21.15.189:6836","nonce":4021875447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:47:34.584208+0000","modified":"2024-04-06T13:47:37.592983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11262":{"gid":11262,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6834/4066431971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4066431971},{"type":"v1","addr":"172.21.15.192:6834","nonce":4066431971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-04-06T13:48:06.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.110+0000 7fa7497c4700 1 -- 172.21.15.189:0/1734638245 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa73006aed0 msgr2=0x7fa73006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:06.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.110+0000 7fa7497c4700 1 --2- 172.21.15.189:0/1734638245 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa73006aed0 0x7fa73006d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa734001eb0 tx=0x7fa734001ca0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.110+0000 7fa7497c4700 1 -- 172.21.15.189:0/1734638245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7440b4660 msgr2=0x7fa74414cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:06.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.110+0000 7fa7497c4700 1 --2- 172.21.15.189:0/1734638245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7440b4660 0x7fa74414cb10 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7fa73800c550 tx=0x7fa73800c910 comp rx=0 tx=0).stop 2024-04-06T13:48:06.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.110+0000 7fa7497c4700 1 -- 172.21.15.189:0/1734638245 shutdown_connections 2024-04-06T13:48:06.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.110+0000 7fa7497c4700 1 --2- 172.21.15.189:0/1734638245 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa73006aed0 0x7fa73006d390 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.110+0000 7fa7497c4700 1 --2- 172.21.15.189:0/1734638245 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa7440ce2e0 0x7fa7440cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.110+0000 7fa7497c4700 1 --2- 172.21.15.189:0/1734638245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7440b4660 0x7fa74414cb10 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.110+0000 7fa7497c4700 1 --2- 172.21.15.189:0/1734638245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7440b4af0 0x7fa7440cdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.110+0000 7fa7497c4700 1 -- 172.21.15.189:0/1734638245 >> 172.21.15.189:0/1734638245 conn(0x7fa7440d00b0 msgr2=0x7fa7441597d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:06.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.110+0000 7fa7497c4700 1 -- 172.21.15.189:0/1734638245 shutdown_connections 2024-04-06T13:48:06.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.110+0000 7fa7497c4700 1 -- 172.21.15.189:0/1734638245 wait complete. 2024-04-06T13:48:06.113 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 108 2024-04-06T13:48:06.123 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:48:06.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.247+0000 7f8db06fe700 1 Processor -- start 2024-04-06T13:48:06.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.247+0000 7f8db06fe700 1 -- start start 2024-04-06T13:48:06.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.248+0000 7f8db06fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8da806ae40 0x7f8da815bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:06.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.248+0000 7f8db06fe700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da815f9e0 con 0x7f8da815c100 2024-04-06T13:48:06.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.248+0000 7f8db06fe700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da815f0d0 con 0x7f8da8158fe0 2024-04-06T13:48:06.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.248+0000 7f8db06fe700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8da815c8f0 con 0x7f8da806ae40 2024-04-06T13:48:06.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.248+0000 7f8daec9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8da815c100 0x7f8da80ceca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50728/0 (socket says 172.21.15.189:50728) 2024-04-06T13:48:06.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.248+0000 7f8daec9b700 1 -- 172.21.15.189:0/2114434824 learned_addr learned my addr 172.21.15.189:0/2114434824 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:06.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.248+0000 7f8dadc99700 1 --2- 172.21.15.189:0/2114434824 >> v2:172.21.15.192:3300/0 conn(0x7f8da806ae40 0x7f8da815bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:06.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.248+0000 7f8dad498700 1 -- 172.21.15.189:0/2114434824 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184014702 0 0) 0x7f8da815f9e0 con 0x7f8da815c100 2024-04-06T13:48:06.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.248+0000 7f8dad498700 1 -- 172.21.15.189:0/2114434824 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d940035b0 con 0x7f8da815c100 2024-04-06T13:48:06.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.248+0000 7f8dad498700 1 -- 172.21.15.189:0/2114434824 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2797208882 0 0) 0x7f8d940035b0 con 0x7f8da815c100 2024-04-06T13:48:06.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.248+0000 7f8dad498700 1 -- 172.21.15.189:0/2114434824 >> v2:172.21.15.192:3300/0 conn(0x7f8da806ae40 msgr2=0x7f8da815bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:06.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.248+0000 7f8dad498700 1 --2- 172.21.15.189:0/2114434824 >> v2:172.21.15.192:3300/0 conn(0x7f8da806ae40 0x7f8da815bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.248+0000 7f8dad498700 1 -- 172.21.15.189:0/2114434824 >> v1:172.21.15.192:6789/0 conn(0x7f8da8158fe0 legacy=0x7f8da806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:06.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.248+0000 7f8dad498700 1 -- 172.21.15.189:0/2114434824 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da815f390 con 0x7f8da815c100 2024-04-06T13:48:06.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.248+0000 7f8dad498700 1 -- 172.21.15.189:0/2114434824 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f8da00019f0 con 0x7f8da815c100 2024-04-06T13:48:06.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.248+0000 7f8dad498700 1 -- 172.21.15.189:0/2114434824 >> v1:172.21.15.189:6789/0 conn(0x7f8da815c100 legacy=0x7f8da80ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:06.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.249+0000 7f8dad498700 1 --2- 172.21.15.189:0/2114434824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d94003a00 0x7f8d94003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:06.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.249+0000 7f8dad498700 1 --2- 172.21.15.189:0/2114434824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d94004400 0x7f8d940048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:06.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.249+0000 7f8dad498700 1 --2- 172.21.15.189:0/2114434824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d94005170 0x7f8d9400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:06.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.249+0000 7f8dad498700 1 -- 172.21.15.189:0/2114434824 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da815c8f0 con 0x7f8d94005170 2024-04-06T13:48:06.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.249+0000 7f8dad498700 1 -- 172.21.15.189:0/2114434824 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9400c0b0 con 0x7f8d94004400 2024-04-06T13:48:06.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.249+0000 7f8dad498700 1 -- 172.21.15.189:0/2114434824 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9400c270 con 0x7f8d94003a00 2024-04-06T13:48:06.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.249+0000 7f8daec9b700 1 --2- 172.21.15.189:0/2114434824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d94005170 0x7f8d9400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:06.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.249+0000 7f8dadc99700 1 --2- 172.21.15.189:0/2114434824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d94004400 0x7f8d940048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:06.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.249+0000 7f8dae49a700 1 --2- 172.21.15.189:0/2114434824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d94003a00 0x7f8d94003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:06.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.249+0000 7f8daec9b700 1 -- 172.21.15.189:0/2114434824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d94003a00 msgr2=0x7f8d94003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:06.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.249+0000 7f8daec9b700 1 --2- 172.21.15.189:0/2114434824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d94003a00 0x7f8d94003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.249+0000 7f8daec9b700 1 -- 172.21.15.189:0/2114434824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d94004400 msgr2=0x7f8d940048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:06.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.249+0000 7f8daec9b700 1 --2- 172.21.15.189:0/2114434824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d94004400 0x7f8d940048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.249+0000 7f8daec9b700 1 -- 172.21.15.189:0/2114434824 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d940035b0 con 0x7f8d94005170 2024-04-06T13:48:06.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.249+0000 7f8daec9b700 1 -- 172.21.15.189:0/2114434824 reap_dead start 2024-04-06T13:48:06.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.249+0000 7f8daec9b700 1 --2- 172.21.15.189:0/2114434824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d94005170 0x7f8d9400ba10 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f8da000a7d0 tx=0x7f8d94005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47d625db59475296 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:06.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.249+0000 7f8dad498700 1 -- 172.21.15.189:0/2114434824 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da0012600 con 0x7f8d94005170 2024-04-06T13:48:06.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.249+0000 7f8dad498700 1 -- 172.21.15.189:0/2114434824 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da00127a0 con 0x7f8d94005170 2024-04-06T13:48:06.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.249+0000 7f8dad498700 1 -- 172.21.15.189:0/2114434824 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da0012b50 con 0x7f8d94005170 2024-04-06T13:48:06.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.250+0000 7f8db06fe700 1 -- 172.21.15.189:0/2114434824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d94005170 msgr2=0x7f8d9400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:06.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.250+0000 7f8db06fe700 1 --2- 172.21.15.189:0/2114434824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d94005170 0x7f8d9400ba10 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f8da000a7d0 tx=0x7f8d94005880 comp rx=0 tx=0).stop 2024-04-06T13:48:06.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.250+0000 7f8db06fe700 1 -- 172.21.15.189:0/2114434824 shutdown_connections 2024-04-06T13:48:06.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.250+0000 7f8db06fe700 1 --2- 172.21.15.189:0/2114434824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d94005170 0x7f8d9400ba10 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.250+0000 7f8db06fe700 1 -- 172.21.15.189:0/2114434824 >> 172.21.15.189:0/2114434824 conn(0x7f8da80d00b0 msgr2=0x7f8da80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:06.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.250+0000 7f8db06fe700 1 -- 172.21.15.189:0/2114434824 shutdown_connections 2024-04-06T13:48:06.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.250+0000 7f8db06fe700 1 -- 172.21.15.189:0/2114434824 wait complete. 2024-04-06T13:48:06.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.251+0000 7f8db06fe700 1 Processor -- start 2024-04-06T13:48:06.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.251+0000 7f8db06fe700 1 -- start start 2024-04-06T13:48:06.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.252+0000 7f8db06fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d94005170 0x7f8da8157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:06.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.252+0000 7f8db06fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8da813aa10 0x7f8da813ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:06.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.252+0000 7f8db06fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8da813cc00 0x7f8da813d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:06.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.252+0000 7f8db06fe700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da81410f0 con 0x7f8da813aa10 2024-04-06T13:48:06.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.252+0000 7f8db06fe700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da81579d0 con 0x7f8d94005170 2024-04-06T13:48:06.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.252+0000 7f8db06fe700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da813b580 con 0x7f8da813cc00 2024-04-06T13:48:06.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.252+0000 7f8dadc99700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8da813aa10 0x7f8da813ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:06.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.252+0000 7f8dae49a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d94005170 0x7f8da8157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:06.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.252+0000 7f8dadc99700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8da813aa10 0x7f8da813ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41794/0 (socket says 172.21.15.189:41794) 2024-04-06T13:48:06.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.252+0000 7f8dadc99700 1 -- 172.21.15.189:0/4064896970 learned_addr learned my addr 172.21.15.189:0/4064896970 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:06.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.252+0000 7f8daec9b700 1 --2- 172.21.15.189:0/4064896970 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8da813cc00 0x7f8da813d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:06.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.252+0000 7f8dadc99700 1 -- 172.21.15.189:0/4064896970 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8da813cc00 msgr2=0x7f8da813d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:06.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.252+0000 7f8dadc99700 1 --2- 172.21.15.189:0/4064896970 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8da813cc00 0x7f8da813d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.252+0000 7f8dadc99700 1 -- 172.21.15.189:0/4064896970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d94005170 msgr2=0x7f8da8157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.252+0000 7f8dadc99700 1 --2- 172.21.15.189:0/4064896970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d94005170 0x7f8da8157490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.252+0000 7f8dadc99700 1 -- 172.21.15.189:0/4064896970 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da8141340 con 0x7f8da813aa10 2024-04-06T13:48:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.252+0000 7f8dae49a700 1 --2- 172.21.15.189:0/4064896970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d94005170 0x7f8da8157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.252+0000 7f8dadc99700 1 --2- 172.21.15.189:0/4064896970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8da813aa10 0x7f8da813ae90 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f8d980119a0 tx=0x7f8d98011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.253+0000 7f8d9effd700 1 -- 172.21.15.189:0/4064896970 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9800f780 con 0x7f8da813aa10 2024-04-06T13:48:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.253+0000 7f8d9effd700 1 -- 172.21.15.189:0/4064896970 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d9800f8e0 con 0x7f8da813aa10 2024-04-06T13:48:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.253+0000 7f8db06fe700 1 -- 172.21.15.189:0/4064896970 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da8141620 con 0x7f8da813aa10 2024-04-06T13:48:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.253+0000 7f8d9effd700 1 -- 172.21.15.189:0/4064896970 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9800fc70 con 0x7f8da813aa10 2024-04-06T13:48:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.253+0000 7f8db06fe700 1 -- 172.21.15.189:0/4064896970 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da81406b0 con 0x7f8da813aa10 2024-04-06T13:48:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.254+0000 7f8db06fe700 1 -- 172.21.15.189:0/4064896970 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da8141b40 con 0x7f8da813aa10 2024-04-06T13:48:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.254+0000 7f8d9effd700 1 -- 172.21.15.189:0/4064896970 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f8d98014440 con 0x7f8da813aa10 2024-04-06T13:48:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.254+0000 7f8d9effd700 1 --2- 172.21.15.189:0/4064896970 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8d8c06aed0 0x7f8d8c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.255+0000 7f8dae49a700 1 --2- 172.21.15.189:0/4064896970 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8d8c06aed0 0x7f8d8c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:06.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.255+0000 7f8d9effd700 1 -- 172.21.15.189:0/4064896970 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6930+0+0 (secure 0 0 0) 0x7f8d9808a280 con 0x7f8da813aa10 2024-04-06T13:48:06.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.255+0000 7f8dae49a700 1 --2- 172.21.15.189:0/4064896970 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8d8c06aed0 0x7f8d8c06d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8da4007fd0 tx=0x7f8da4009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:06.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.259+0000 7f8d9effd700 1 -- 172.21.15.189:0/4064896970 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8da8141b40 con 0x7f8da813aa10 2024-04-06T13:48:06.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.456+0000 7f8db06fe700 1 -- 172.21.15.189:0/4064896970 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8da8067bb0 con 0x7f8da813aa10 2024-04-06T13:48:06.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.457+0000 7f8d9effd700 1 -- 172.21.15.189:0/4064896970 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v236) v1 ==== 74+0+17619 (secure 0 0 0) 0x7f8d98054ba0 con 0x7f8da813aa10 2024-04-06T13:48:06.459 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:06.459 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":236,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:48:03.192350+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:47:32.292623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:47:33.567052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.192:6833/694564673":"2024-04-07T13:48:00.073095+0000","172.21.15.189:0/3017776380":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1992839362":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/579471530":"2024-04-07T13:47:57.909304+0000","172.21.15.192:6835/694564673":"2024-04-07T13:48:00.073095+0000","172.21.15.189:0/4049616553":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/619264656":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1506469869":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1831656794":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1190635801":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/3920072072":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/1878657401":"2024-04-07T13:47:47.129296+0000","172.21.15.189:6833/111704":"2024-04-07T13:47:47.129296+0000","172.21.15.189:6832/111704":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/2827130089":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/2716712615":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/1807905335":"2024-04-07T13:47:47.129296+0000","172.21.15.189:0/914480885":"2024-04-07T13:47:57.909304+0000","172.21.15.189:0/1173010452":"2024-04-07T13:47:47.129296+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:48:06.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.460+0000 7f8db06fe700 1 -- 172.21.15.189:0/4064896970 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8d8c06aed0 msgr2=0x7f8d8c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:06.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.460+0000 7f8db06fe700 1 --2- 172.21.15.189:0/4064896970 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8d8c06aed0 0x7f8d8c06d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8da4007fd0 tx=0x7f8da4009040 comp rx=0 tx=0).stop 2024-04-06T13:48:06.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.460+0000 7f8db06fe700 1 -- 172.21.15.189:0/4064896970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8da813aa10 msgr2=0x7f8da813ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:06.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.460+0000 7f8db06fe700 1 --2- 172.21.15.189:0/4064896970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8da813aa10 0x7f8da813ae90 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f8d980119a0 tx=0x7f8d98011cb0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.460+0000 7f8db06fe700 1 -- 172.21.15.189:0/4064896970 shutdown_connections 2024-04-06T13:48:06.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.460+0000 7f8db06fe700 1 --2- 172.21.15.189:0/4064896970 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8d8c06aed0 0x7f8d8c06d390 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.460+0000 7f8db06fe700 1 --2- 172.21.15.189:0/4064896970 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8da813cc00 0x7f8da813d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.460+0000 7f8db06fe700 1 --2- 172.21.15.189:0/4064896970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d94005170 0x7f8da8157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.460+0000 7f8db06fe700 1 --2- 172.21.15.189:0/4064896970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8da813aa10 0x7f8da813ae90 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.460+0000 7f8db06fe700 1 -- 172.21.15.189:0/4064896970 >> 172.21.15.189:0/4064896970 conn(0x7f8da80d00b0 msgr2=0x7f8da8158220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:06.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.460+0000 7f8db06fe700 1 -- 172.21.15.189:0/4064896970 shutdown_connections 2024-04-06T13:48:06.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.460+0000 7f8db06fe700 1 -- 172.21.15.189:0/4064896970 wait complete. 2024-04-06T13:48:06.473 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2024-04-06T13:48:06.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.600+0000 7fca18cbc700 1 Processor -- start 2024-04-06T13:48:06.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.600+0000 7fca18cbc700 1 -- start start 2024-04-06T13:48:06.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.601+0000 7fca18cbc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fca1415d7b0 0x7fca140b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:06.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.601+0000 7fca18cbc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca140b3990 con 0x7fca1415e500 2024-04-06T13:48:06.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.601+0000 7fca18cbc700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca140b3060 con 0x7fca1406a730 2024-04-06T13:48:06.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.601+0000 7fca18cbc700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca1415e950 con 0x7fca1415d7b0 2024-04-06T13:48:06.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.601+0000 7fca12d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fca1415e500 0x7fca140717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50732/0 (socket says 172.21.15.189:50732) 2024-04-06T13:48:06.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.601+0000 7fca12d9d700 1 -- 172.21.15.189:0/4227950621 learned_addr learned my addr 172.21.15.189:0/4227950621 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:06.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.601+0000 7fca1259c700 1 --2- 172.21.15.189:0/4227950621 >> v2:172.21.15.192:3300/0 conn(0x7fca1415d7b0 0x7fca140b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:06.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.602+0000 7fca1259c700 1 -- 172.21.15.189:0/4227950621 >> v1:172.21.15.192:6789/0 conn(0x7fca1406a730 legacy=0x7fca1415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:06.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.602+0000 7fca1259c700 1 -- 172.21.15.189:0/4227950621 >> v1:172.21.15.189:6789/0 conn(0x7fca1415e500 legacy=0x7fca140717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:06.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.602+0000 7fca1259c700 1 -- 172.21.15.189:0/4227950621 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca140b3340 con 0x7fca1415d7b0 2024-04-06T13:48:06.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.602+0000 7fca1259c700 1 --2- 172.21.15.189:0/4227950621 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca1415d7b0 0x7fca140b3e20 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fca04009840 tx=0x7fca04009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f26ee9a0d5a7a2e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:06.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.602+0000 7fca1159a700 1 -- 172.21.15.189:0/4227950621 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca04004230 con 0x7fca1415d7b0 2024-04-06T13:48:06.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.602+0000 7fca1159a700 1 -- 172.21.15.189:0/4227950621 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca04004390 con 0x7fca1415d7b0 2024-04-06T13:48:06.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.602+0000 7fca1159a700 1 -- 172.21.15.189:0/4227950621 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca040036a0 con 0x7fca1415d7b0 2024-04-06T13:48:06.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.603+0000 7fca18cbc700 1 -- 172.21.15.189:0/4227950621 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca1415d7b0 msgr2=0x7fca140b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:06.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.603+0000 7fca18cbc700 1 --2- 172.21.15.189:0/4227950621 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca1415d7b0 0x7fca140b3e20 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fca04009840 tx=0x7fca04009b50 comp rx=0 tx=0).stop 2024-04-06T13:48:06.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.603+0000 7fca18cbc700 1 -- 172.21.15.189:0/4227950621 shutdown_connections 2024-04-06T13:48:06.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.603+0000 7fca18cbc700 1 --2- 172.21.15.189:0/4227950621 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca1415d7b0 0x7fca140b3e20 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.603+0000 7fca18cbc700 1 -- 172.21.15.189:0/4227950621 >> 172.21.15.189:0/4227950621 conn(0x7fca140d00b0 msgr2=0x7fca140d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:06.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.603+0000 7fca18cbc700 1 -- 172.21.15.189:0/4227950621 shutdown_connections 2024-04-06T13:48:06.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.603+0000 7fca18cbc700 1 -- 172.21.15.189:0/4227950621 wait complete. 2024-04-06T13:48:06.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.605+0000 7fca18cbc700 1 Processor -- start 2024-04-06T13:48:06.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.605+0000 7fca18cbc700 1 -- start start 2024-04-06T13:48:06.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.605+0000 7fca18cbc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca1406a730 0x7fca1414a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:06.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.605+0000 7fca18cbc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fca1415d7b0 0x7fca140ce610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:06.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.606+0000 7fca18cbc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca1415e500 0x7fca14158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:06.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.606+0000 7fca18cbc700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca14154800 con 0x7fca1415e500 2024-04-06T13:48:06.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.606+0000 7fca18cbc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca141473f0 con 0x7fca1406a730 2024-04-06T13:48:06.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.606+0000 7fca18cbc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca140cee40 con 0x7fca1415d7b0 2024-04-06T13:48:06.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.606+0000 7fca12d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca1415e500 0x7fca14158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:06.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.606+0000 7fca1259c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca1406a730 0x7fca1414a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:06.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.606+0000 7fca12d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca1415e500 0x7fca14158cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41804/0 (socket says 172.21.15.189:41804) 2024-04-06T13:48:06.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.606+0000 7fca12d9d700 1 -- 172.21.15.189:0/3638234963 learned_addr learned my addr 172.21.15.189:0/3638234963 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:06.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.606+0000 7fca11d9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fca1415d7b0 0x7fca140ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:06.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.606+0000 7fca12d9d700 1 -- 172.21.15.189:0/3638234963 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fca1415d7b0 msgr2=0x7fca140ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:06.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.606+0000 7fca12d9d700 1 --2- 172.21.15.189:0/3638234963 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fca1415d7b0 0x7fca140ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.606+0000 7fca12d9d700 1 -- 172.21.15.189:0/3638234963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca1406a730 msgr2=0x7fca1414a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:06.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.606+0000 7fca12d9d700 1 --2- 172.21.15.189:0/3638234963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca1406a730 0x7fca1414a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.606+0000 7fca12d9d700 1 -- 172.21.15.189:0/3638234963 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca040094a0 con 0x7fca1415e500 2024-04-06T13:48:06.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.606+0000 7fca12d9d700 1 --2- 172.21.15.189:0/3638234963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca1415e500 0x7fca14158cc0 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7fca0c001f80 tx=0x7fca0c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:06.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.606+0000 7fca02ffd700 1 -- 172.21.15.189:0/3638234963 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c00acd0 con 0x7fca1415e500 2024-04-06T13:48:06.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.606+0000 7fca02ffd700 1 -- 172.21.15.189:0/3638234963 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0c00ae30 con 0x7fca1415e500 2024-04-06T13:48:06.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.606+0000 7fca18cbc700 1 -- 172.21.15.189:0/3638234963 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca140cf120 con 0x7fca1415e500 2024-04-06T13:48:06.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.606+0000 7fca02ffd700 1 -- 172.21.15.189:0/3638234963 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c0072f0 con 0x7fca1415e500 2024-04-06T13:48:06.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.607+0000 7fca18cbc700 1 -- 172.21.15.189:0/3638234963 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca140aef00 con 0x7fca1415e500 2024-04-06T13:48:06.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.608+0000 7fca18cbc700 1 -- 172.21.15.189:0/3638234963 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca14067bb0 con 0x7fca1415e500 2024-04-06T13:48:06.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.608+0000 7fca02ffd700 1 -- 172.21.15.189:0/3638234963 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fca0c007450 con 0x7fca1415e500 2024-04-06T13:48:06.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.608+0000 7fca02ffd700 1 --2- 172.21.15.189:0/3638234963 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc9f406aed0 0x7fc9f406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:06.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.609+0000 7fca02ffd700 1 -- 172.21.15.189:0/3638234963 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6930+0+0 (secure 0 0 0) 0x7fca0c0899f0 con 0x7fca1415e500 2024-04-06T13:48:06.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.609+0000 7fca1259c700 1 --2- 172.21.15.189:0/3638234963 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc9f406aed0 0x7fc9f406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:06.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.609+0000 7fca1259c700 1 --2- 172.21.15.189:0/3638234963 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc9f406aed0 0x7fc9f406d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fca04009b90 tx=0x7fca04008000 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:06.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.613+0000 7fca02ffd700 1 -- 172.21.15.189:0/3638234963 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fca0c0543a0 con 0x7fca1415e500 2024-04-06T13:48:06.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.852+0000 7fca18cbc700 1 -- 172.21.15.189:0/3638234963 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7fca14074990 con 0x7fca1415e500 2024-04-06T13:48:06.855 INFO:tasks.ceph.mon.a.smithi189.stderr:2024-04-06T13:48:06.853+0000 7fa9fe1ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T13:48:06.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.862+0000 7fca02ffd700 1 -- 172.21.15.189:0/3638234963 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v109) v1 ==== 167+0+0 (secure 0 0 0) 0x7fca0c0579c0 con 0x7fca1415e500 2024-04-06T13:48:06.865 INFO:tasks.ceph.mds.d.smithi192.stderr: -1730> 2024-04-06T13:47:38.649+0000 7fccfa933700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T13:48:06.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.864+0000 7fca18cbc700 1 -- 172.21.15.189:0/3638234963 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc9f406aed0 msgr2=0x7fc9f406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:06.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.864+0000 7fca18cbc700 1 --2- 172.21.15.189:0/3638234963 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc9f406aed0 0x7fc9f406d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fca04009b90 tx=0x7fca04008000 comp rx=0 tx=0).stop 2024-04-06T13:48:06.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.864+0000 7fca18cbc700 1 -- 172.21.15.189:0/3638234963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca1415e500 msgr2=0x7fca14158cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:06.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.864+0000 7fca18cbc700 1 --2- 172.21.15.189:0/3638234963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca1415e500 0x7fca14158cc0 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7fca0c001f80 tx=0x7fca0c00a340 comp rx=0 tx=0).stop 2024-04-06T13:48:06.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.864+0000 7fca18cbc700 1 -- 172.21.15.189:0/3638234963 shutdown_connections 2024-04-06T13:48:06.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.864+0000 7fca18cbc700 1 --2- 172.21.15.189:0/3638234963 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc9f406aed0 0x7fc9f406d390 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.864+0000 7fca18cbc700 1 --2- 172.21.15.189:0/3638234963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca1415e500 0x7fca14158cc0 unknown :-1 s=CLOSED pgs=2617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.864+0000 7fca18cbc700 1 --2- 172.21.15.189:0/3638234963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca1406a730 0x7fca1414a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.864+0000 7fca18cbc700 1 --2- 172.21.15.189:0/3638234963 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fca1415d7b0 0x7fca140ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:06.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.864+0000 7fca18cbc700 1 -- 172.21.15.189:0/3638234963 >> 172.21.15.189:0/3638234963 conn(0x7fca140d00b0 msgr2=0x7fca140ae1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:06.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.864+0000 7fca18cbc700 1 -- 172.21.15.189:0/3638234963 shutdown_connections 2024-04-06T13:48:06.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:06.864+0000 7fca18cbc700 1 -- 172.21.15.189:0/3638234963 wait complete. 2024-04-06T13:48:06.867 INFO:teuthology.orchestra.run.smithi189.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-06T13:48:06.878 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2024-04-06T13:48:06.892 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2024-04-06T13:48:07.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.002+0000 7fb4489b7700 1 Processor -- start 2024-04-06T13:48:07.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.002+0000 7fb4489b7700 1 -- start start 2024-04-06T13:48:07.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.003+0000 7fb4489b7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4440b4af0 0x7fb4440b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:07.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.003+0000 7fb4489b7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4440cdbe0 con 0x7fb44415a550 2024-04-06T13:48:07.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.003+0000 7fb4489b7700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44415a940 con 0x7fb4440b4660 2024-04-06T13:48:07.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.003+0000 7fb4489b7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb44415ac00 con 0x7fb4440b4af0 2024-04-06T13:48:07.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.003+0000 7fb44259c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb44415a550 0x7fb4440cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50734/0 (socket says 172.21.15.189:50734) 2024-04-06T13:48:07.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.003+0000 7fb44259c700 1 -- 172.21.15.189:0/2573837670 learned_addr learned my addr 172.21.15.189:0/2573837670 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:07.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.003+0000 7fb442d9d700 1 --2- 172.21.15.189:0/2573837670 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4440b4af0 0x7fb4440b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:07.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.003+0000 7fb44159a700 1 -- 172.21.15.189:0/2573837670 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506675811 0 0) 0x7fb4440cdbe0 con 0x7fb44415a550 2024-04-06T13:48:07.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.003+0000 7fb44159a700 1 -- 172.21.15.189:0/2573837670 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb42c0057c0 con 0x7fb44415a550 2024-04-06T13:48:07.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.003+0000 7fb44159a700 1 -- 172.21.15.189:0/2573837670 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558133478 0 0) 0x7fb44415a940 con 0x7fb4440b4660 2024-04-06T13:48:07.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.003+0000 7fb44159a700 1 -- 172.21.15.189:0/2573837670 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4440cdbe0 con 0x7fb4440b4660 2024-04-06T13:48:07.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.004+0000 7fb442d9d700 1 -- 172.21.15.189:0/2573837670 >> v1:172.21.15.192:6789/0 conn(0x7fb4440b4660 legacy=0x7fb4440b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:07.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.004+0000 7fb44159a700 1 -- 172.21.15.189:0/2573837670 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 942436961 0 0) 0x7fb42c0057c0 con 0x7fb44415a550 2024-04-06T13:48:07.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.004+0000 7fb442d9d700 1 -- 172.21.15.189:0/2573837670 >> v1:172.21.15.189:6789/0 conn(0x7fb44415a550 legacy=0x7fb4440cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:07.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.004+0000 7fb442d9d700 1 -- 172.21.15.189:0/2573837670 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44415aee0 con 0x7fb4440b4af0 2024-04-06T13:48:07.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.004+0000 7fb442d9d700 1 --2- 172.21.15.189:0/2573837670 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4440b4af0 0x7fb4440b36d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb43c009dc0 tx=0x7fb43c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9ac1348799f1c39 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:07.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.004+0000 7fb44159a700 1 -- 172.21.15.189:0/2573837670 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43c00e070 con 0x7fb4440b4af0 2024-04-06T13:48:07.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.004+0000 7fb44159a700 1 -- 172.21.15.189:0/2573837670 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43c003a20 con 0x7fb4440b4af0 2024-04-06T13:48:07.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.004+0000 7fb44159a700 1 -- 172.21.15.189:0/2573837670 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb438001670 con 0x7fb4440b4af0 2024-04-06T13:48:07.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.004+0000 7fb4489b7700 1 -- 172.21.15.189:0/2573837670 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4440b4af0 msgr2=0x7fb4440b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:07.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.004+0000 7fb4489b7700 1 --2- 172.21.15.189:0/2573837670 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4440b4af0 0x7fb4440b36d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb43c009dc0 tx=0x7fb43c00b330 comp rx=0 tx=0).stop 2024-04-06T13:48:07.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.005+0000 7fb4489b7700 1 -- 172.21.15.189:0/2573837670 shutdown_connections 2024-04-06T13:48:07.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.005+0000 7fb4489b7700 1 --2- 172.21.15.189:0/2573837670 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4440b4af0 0x7fb4440b36d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:07.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.005+0000 7fb4489b7700 1 -- 172.21.15.189:0/2573837670 >> 172.21.15.189:0/2573837670 conn(0x7fb4440d00b0 msgr2=0x7fb4440d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:07.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.005+0000 7fb4489b7700 1 -- 172.21.15.189:0/2573837670 shutdown_connections 2024-04-06T13:48:07.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.005+0000 7fb4489b7700 1 -- 172.21.15.189:0/2573837670 wait complete. 2024-04-06T13:48:07.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.006+0000 7fb4489b7700 1 Processor -- start 2024-04-06T13:48:07.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.006+0000 7fb4489b7700 1 -- start start 2024-04-06T13:48:07.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb4489b7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4440b4660 0x7fb444151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:07.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb4489b7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4440b4af0 0x7fb4440aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:07.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb4489b7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4441ee790 0x7fb44406d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:07.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb4489b7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb444155aa0 con 0x7fb4441ee790 2024-04-06T13:48:07.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb4489b7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb444155700 con 0x7fb4440b4660 2024-04-06T13:48:07.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb4489b7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4441520d0 con 0x7fb4440b4af0 2024-04-06T13:48:07.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb442d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4441ee790 0x7fb44406d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:07.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb442d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4441ee790 0x7fb44406d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41818/0 (socket says 172.21.15.189:41818) 2024-04-06T13:48:07.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb442d9d700 1 -- 172.21.15.189:0/2973044386 learned_addr learned my addr 172.21.15.189:0/2973044386 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:07.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb44259c700 1 --2- 172.21.15.189:0/2973044386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4440b4660 0x7fb444151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:07.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb441d9b700 1 --2- 172.21.15.189:0/2973044386 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4440b4af0 0x7fb4440aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:07.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb442d9d700 1 -- 172.21.15.189:0/2973044386 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4440b4af0 msgr2=0x7fb4440aeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:07.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb442d9d700 1 --2- 172.21.15.189:0/2973044386 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4440b4af0 0x7fb4440aeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:07.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb442d9d700 1 -- 172.21.15.189:0/2973044386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4440b4660 msgr2=0x7fb444151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:07.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb442d9d700 1 --2- 172.21.15.189:0/2973044386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4440b4660 0x7fb444151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:07.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb442d9d700 1 -- 172.21.15.189:0/2973044386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43c009a70 con 0x7fb4441ee790 2024-04-06T13:48:07.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb442d9d700 1 --2- 172.21.15.189:0/2973044386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4441ee790 0x7fb44406d610 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fb43c009d90 tx=0x7fb43c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:07.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb432ffd700 1 -- 172.21.15.189:0/2973044386 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43c0044c0 con 0x7fb4441ee790 2024-04-06T13:48:07.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb432ffd700 1 -- 172.21.15.189:0/2973044386 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43c004620 con 0x7fb4441ee790 2024-04-06T13:48:07.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb4489b7700 1 -- 172.21.15.189:0/2973044386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4441575a0 con 0x7fb4441ee790 2024-04-06T13:48:07.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb432ffd700 1 -- 172.21.15.189:0/2973044386 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43c01d470 con 0x7fb4441ee790 2024-04-06T13:48:07.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.007+0000 7fb4489b7700 1 -- 172.21.15.189:0/2973044386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb444157b60 con 0x7fb4441ee790 2024-04-06T13:48:07.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.009+0000 7fb4489b7700 1 -- 172.21.15.189:0/2973044386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb444067bb0 con 0x7fb4441ee790 2024-04-06T13:48:07.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.009+0000 7fb432ffd700 1 -- 172.21.15.189:0/2973044386 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fb43c01d5d0 con 0x7fb4441ee790 2024-04-06T13:48:07.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.009+0000 7fb432ffd700 1 --2- 172.21.15.189:0/2973044386 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb42406aed0 0x7fb42406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:07.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.009+0000 7fb432ffd700 1 -- 172.21.15.189:0/2973044386 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7016+0+0 (secure 0 0 0) 0x7fb43c08bab0 con 0x7fb4441ee790 2024-04-06T13:48:07.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.009+0000 7fb44259c700 1 --2- 172.21.15.189:0/2973044386 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb42406aed0 0x7fb42406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:07.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.010+0000 7fb44259c700 1 --2- 172.21.15.189:0/2973044386 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb42406aed0 0x7fb42406d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb438000f60 tx=0x7fb438001690 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:07.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.014+0000 7fb432ffd700 1 -- 172.21.15.189:0/2973044386 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb43c056270 con 0x7fb4441ee790 2024-04-06T13:48:07.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.257+0000 7fb4489b7700 1 -- 172.21.15.189:0/2973044386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb444074990 con 0x7fb4441ee790 2024-04-06T13:48:07.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.868+0000 7fb432ffd700 1 -- 172.21.15.189:0/2973044386 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v110) v1 ==== 107+0+0 (secure 0 0 0) 0x7fb43c024360 con 0x7fb4441ee790 2024-04-06T13:48:07.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.871+0000 7fb4489b7700 1 -- 172.21.15.189:0/2973044386 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb42406aed0 msgr2=0x7fb42406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:07.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.871+0000 7fb4489b7700 1 --2- 172.21.15.189:0/2973044386 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb42406aed0 0x7fb42406d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb438000f60 tx=0x7fb438001690 comp rx=0 tx=0).stop 2024-04-06T13:48:07.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.871+0000 7fb4489b7700 1 -- 172.21.15.189:0/2973044386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4441ee790 msgr2=0x7fb44406d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:07.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.871+0000 7fb4489b7700 1 --2- 172.21.15.189:0/2973044386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4441ee790 0x7fb44406d610 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fb43c009d90 tx=0x7fb43c004ad0 comp rx=0 tx=0).stop 2024-04-06T13:48:07.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.871+0000 7fb4489b7700 1 -- 172.21.15.189:0/2973044386 shutdown_connections 2024-04-06T13:48:07.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.871+0000 7fb4489b7700 1 --2- 172.21.15.189:0/2973044386 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb42406aed0 0x7fb42406d390 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:07.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.871+0000 7fb4489b7700 1 --2- 172.21.15.189:0/2973044386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4441ee790 0x7fb44406d610 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:07.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.871+0000 7fb4489b7700 1 --2- 172.21.15.189:0/2973044386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4440b4660 0x7fb444151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:07.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.871+0000 7fb4489b7700 1 --2- 172.21.15.189:0/2973044386 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4440b4af0 0x7fb4440aeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:07.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.871+0000 7fb4489b7700 1 -- 172.21.15.189:0/2973044386 >> 172.21.15.189:0/2973044386 conn(0x7fb4440d00b0 msgr2=0x7fb444152c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:07.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.871+0000 7fb4489b7700 1 -- 172.21.15.189:0/2973044386 shutdown_connections 2024-04-06T13:48:07.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:07.871+0000 7fb4489b7700 1 -- 172.21.15.189:0/2973044386 wait complete. 2024-04-06T13:48:07.886 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2024-04-06T13:48:08.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.016+0000 7f801691d700 1 Processor -- start 2024-04-06T13:48:08.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.016+0000 7f801691d700 1 -- start start 2024-04-06T13:48:08.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.017+0000 7f801691d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f801006ae40 0x7f80100b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:08.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.017+0000 7f801691d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f801015ddf0 0x7f801015e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:08.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.017+0000 7f801691d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80100b3990 con 0x7f801015d7b0 2024-04-06T13:48:08.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.017+0000 7f801691d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80100b3060 con 0x7f801006ae40 2024-04-06T13:48:08.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.017+0000 7f801691d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80100b3340 con 0x7f801015ddf0 2024-04-06T13:48:08.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.017+0000 7f800f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f801006ae40 0x7f80100b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:08.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.017+0000 7f800f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f801006ae40 0x7f80100b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41828/0 (socket says 172.21.15.189:41828) 2024-04-06T13:48:08.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.017+0000 7f800f7fe700 1 -- 172.21.15.189:0/1357553625 learned_addr learned my addr 172.21.15.189:0/1357553625 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:08.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.017+0000 7f8014eba700 1 --2- 172.21.15.189:0/1357553625 >> v2:172.21.15.192:3300/0 conn(0x7f801015ddf0 0x7f801015e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:08.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.017+0000 7f800f7fe700 1 -- 172.21.15.189:0/1357553625 >> v2:172.21.15.192:3300/0 conn(0x7f801015ddf0 msgr2=0x7f801015e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:08.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.017+0000 7f800f7fe700 1 --2- 172.21.15.189:0/1357553625 >> v2:172.21.15.192:3300/0 conn(0x7f801015ddf0 0x7f801015e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:08.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.017+0000 7f800f7fe700 1 -- 172.21.15.189:0/1357553625 >> v1:172.21.15.192:6789/0 conn(0x7f801015d7b0 legacy=0x7f801006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:08.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.017+0000 7f800f7fe700 1 -- 172.21.15.189:0/1357553625 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80100b3620 con 0x7f801006ae40 2024-04-06T13:48:08.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.018+0000 7f800f7fe700 1 --2- 172.21.15.189:0/1357553625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801006ae40 0x7f80100b3e20 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7f801006d2c0 tx=0x7f8000009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90a3520d62347d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:08.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.018+0000 7f800effd700 1 -- 172.21.15.189:0/1357553625 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8000011830 con 0x7f801006ae40 2024-04-06T13:48:08.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.018+0000 7f800effd700 1 -- 172.21.15.189:0/1357553625 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8000011990 con 0x7f801006ae40 2024-04-06T13:48:08.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.018+0000 7f800effd700 1 -- 172.21.15.189:0/1357553625 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8000011d20 con 0x7f801006ae40 2024-04-06T13:48:08.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.018+0000 7f801691d700 1 -- 172.21.15.189:0/1357553625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801006ae40 msgr2=0x7f80100b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:08.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.018+0000 7f801691d700 1 --2- 172.21.15.189:0/1357553625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801006ae40 0x7f80100b3e20 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7f801006d2c0 tx=0x7f8000009bd0 comp rx=0 tx=0).stop 2024-04-06T13:48:08.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.018+0000 7f801691d700 1 -- 172.21.15.189:0/1357553625 shutdown_connections 2024-04-06T13:48:08.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.018+0000 7f801691d700 1 --2- 172.21.15.189:0/1357553625 >> v2:172.21.15.192:3300/0 conn(0x7f801015ddf0 0x7f801015e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:08.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.018+0000 7f801691d700 1 --2- 172.21.15.189:0/1357553625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801006ae40 0x7f80100b3e20 unknown :-1 s=CLOSED pgs=2622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:08.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.018+0000 7f801691d700 1 -- 172.21.15.189:0/1357553625 >> 172.21.15.189:0/1357553625 conn(0x7f80100d00b0 msgr2=0x7f80100d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:08.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.018+0000 7f801691d700 1 -- 172.21.15.189:0/1357553625 shutdown_connections 2024-04-06T13:48:08.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.018+0000 7f801691d700 1 -- 172.21.15.189:0/1357553625 wait complete. 2024-04-06T13:48:08.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.019+0000 7f801691d700 1 Processor -- start 2024-04-06T13:48:08.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.020+0000 7f801691d700 1 -- start start 2024-04-06T13:48:08.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.020+0000 7f801691d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801006ae40 0x7f801014ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:08.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.020+0000 7f801691d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f801015d7b0 0x7f80100cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:08.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.020+0000 7f801691d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f801015ddf0 0x7f80100cca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:08.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.020+0000 7f801691d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80101573e0 con 0x7f801006ae40 2024-04-06T13:48:08.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.020+0000 7f801691d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8010156860 con 0x7f801015d7b0 2024-04-06T13:48:08.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.020+0000 7f801691d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8010156b40 con 0x7f801015ddf0 2024-04-06T13:48:08.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.021+0000 7f800ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801006ae40 0x7f801014ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:08.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.021+0000 7f800ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801006ae40 0x7f801014ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41840/0 (socket says 172.21.15.189:41840) 2024-04-06T13:48:08.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.021+0000 7f800ffff700 1 -- 172.21.15.189:0/2086032602 learned_addr learned my addr 172.21.15.189:0/2086032602 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:08.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.021+0000 7f800f7fe700 1 --2- 172.21.15.189:0/2086032602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f801015d7b0 0x7f80100cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:08.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.021+0000 7f8014eba700 1 --2- 172.21.15.189:0/2086032602 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f801015ddf0 0x7f80100cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:08.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.021+0000 7f800ffff700 1 -- 172.21.15.189:0/2086032602 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f801015ddf0 msgr2=0x7f80100cca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:08.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.021+0000 7f800ffff700 1 --2- 172.21.15.189:0/2086032602 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f801015ddf0 0x7f80100cca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:08.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.021+0000 7f800ffff700 1 -- 172.21.15.189:0/2086032602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f801015d7b0 msgr2=0x7f80100cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:08.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.021+0000 7f800ffff700 1 --2- 172.21.15.189:0/2086032602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f801015d7b0 0x7f80100cc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:08.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.021+0000 7f800ffff700 1 -- 172.21.15.189:0/2086032602 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8000009550 con 0x7f801006ae40 2024-04-06T13:48:08.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.021+0000 7f800ffff700 1 --2- 172.21.15.189:0/2086032602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801006ae40 0x7f801014ecc0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7f7ff800c4e0 tx=0x7f7ff800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:08.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.021+0000 7f800cff9700 1 -- 172.21.15.189:0/2086032602 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff80052d0 con 0x7f801006ae40 2024-04-06T13:48:08.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.021+0000 7f800cff9700 1 -- 172.21.15.189:0/2086032602 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff8005430 con 0x7f801006ae40 2024-04-06T13:48:08.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.021+0000 7f801691d700 1 -- 172.21.15.189:0/2086032602 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8010156e20 con 0x7f801006ae40 2024-04-06T13:48:08.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.021+0000 7f800cff9700 1 -- 172.21.15.189:0/2086032602 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff80038c0 con 0x7f801006ae40 2024-04-06T13:48:08.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.021+0000 7f801691d700 1 -- 172.21.15.189:0/2086032602 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f801015ce30 con 0x7f801006ae40 2024-04-06T13:48:08.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.022+0000 7f801691d700 1 -- 172.21.15.189:0/2086032602 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801015cfc0 con 0x7f801006ae40 2024-04-06T13:48:08.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.023+0000 7f800cff9700 1 -- 172.21.15.189:0/2086032602 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f7ff8003a20 con 0x7f801006ae40 2024-04-06T13:48:08.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.023+0000 7f800cff9700 1 --2- 172.21.15.189:0/2086032602 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7ff006aed0 0x7f7ff006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:08.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.023+0000 7f800cff9700 1 -- 172.21.15.189:0/2086032602 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7016+0+0 (secure 0 0 0) 0x7f7ff8088f60 con 0x7f801006ae40 2024-04-06T13:48:08.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.023+0000 7f800f7fe700 1 --2- 172.21.15.189:0/2086032602 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7ff006aed0 0x7f7ff006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:08.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.024+0000 7f800f7fe700 1 --2- 172.21.15.189:0/2086032602 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7ff006aed0 0x7f7ff006d390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f801006d2c0 tx=0x7f8000008030 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:08.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.028+0000 7f800cff9700 1 -- 172.21.15.189:0/2086032602 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7ff8053a20 con 0x7f801006ae40 2024-04-06T13:48:08.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.222+0000 7f801691d700 1 -- 172.21.15.189:0/2086032602 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8010067bb0 con 0x7f801006ae40 2024-04-06T13:48:08.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.874+0000 7f800cff9700 1 -- 172.21.15.189:0/2086032602 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v238) v1 ==== 190+0+0 (secure 0 0 0) 0x7f7ff80170e0 con 0x7f801006ae40 2024-04-06T13:48:08.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.877+0000 7f801691d700 1 -- 172.21.15.189:0/2086032602 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7ff006aed0 msgr2=0x7f7ff006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:08.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.877+0000 7f801691d700 1 --2- 172.21.15.189:0/2086032602 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7ff006aed0 0x7f7ff006d390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f801006d2c0 tx=0x7f8000008030 comp rx=0 tx=0).stop 2024-04-06T13:48:08.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.877+0000 7f801691d700 1 -- 172.21.15.189:0/2086032602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801006ae40 msgr2=0x7f801014ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:08.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.877+0000 7f801691d700 1 --2- 172.21.15.189:0/2086032602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801006ae40 0x7f801014ecc0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7f7ff800c4e0 tx=0x7f7ff800c7f0 comp rx=0 tx=0).stop 2024-04-06T13:48:08.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.877+0000 7f801691d700 1 -- 172.21.15.189:0/2086032602 shutdown_connections 2024-04-06T13:48:08.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.877+0000 7f801691d700 1 --2- 172.21.15.189:0/2086032602 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7ff006aed0 0x7f7ff006d390 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:08.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.877+0000 7f801691d700 1 --2- 172.21.15.189:0/2086032602 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f801015ddf0 0x7f80100cca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:08.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.877+0000 7f801691d700 1 --2- 172.21.15.189:0/2086032602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801006ae40 0x7f801014ecc0 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:08.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.877+0000 7f801691d700 1 --2- 172.21.15.189:0/2086032602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f801015d7b0 0x7f80100cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:08.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.877+0000 7f801691d700 1 -- 172.21.15.189:0/2086032602 >> 172.21.15.189:0/2086032602 conn(0x7f80100d00b0 msgr2=0x7f80100adc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:08.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.877+0000 7f801691d700 1 -- 172.21.15.189:0/2086032602 shutdown_connections 2024-04-06T13:48:08.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:08.877+0000 7f801691d700 1 -- 172.21.15.189:0/2086032602 wait complete. 2024-04-06T13:48:08.880 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'backup_fs_metadata' removed 2024-04-06T13:48:08.892 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2024-04-06T13:48:09.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.024+0000 7f41bf54b700 1 Processor -- start 2024-04-06T13:48:09.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.024+0000 7f41bf54b700 1 -- start start 2024-04-06T13:48:09.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.025+0000 7f41bf54b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f41b815a550 0x7f41b80cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:09.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.025+0000 7f41bf54b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b80cf900 0x7f41b80b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:09.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.025+0000 7f41bf54b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b80cdbe0 con 0x7f41b80b4ba0 2024-04-06T13:48:09.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.025+0000 7f41bf54b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b80b5010 con 0x7f41b815a550 2024-04-06T13:48:09.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.025+0000 7f41bf54b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b815a940 con 0x7f41b80cf900 2024-04-06T13:48:09.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.025+0000 7f41bd2e7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f41b815a550 0x7f41b80cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:09.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.025+0000 7f41bdae8700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f41b80b4ba0 0x7f41b806eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50748/0 (socket says 172.21.15.189:50748) 2024-04-06T13:48:09.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.025+0000 7f41bd2e7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f41b815a550 0x7f41b80cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41844/0 (socket says 172.21.15.189:41844) 2024-04-06T13:48:09.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.025+0000 7f41bdae8700 1 -- 172.21.15.189:0/3843452057 learned_addr learned my addr 172.21.15.189:0/3843452057 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:09.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.025+0000 7f41bcae6700 1 --2- 172.21.15.189:0/3843452057 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b80cf900 0x7f41b80b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:09.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.025+0000 7f41bd2e7700 1 -- 172.21.15.189:0/3843452057 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b80cf900 msgr2=0x7f41b80b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:09.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.025+0000 7f41bd2e7700 1 --2- 172.21.15.189:0/3843452057 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b80cf900 0x7f41b80b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:09.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.025+0000 7f41bd2e7700 1 -- 172.21.15.189:0/3843452057 >> v1:172.21.15.189:6789/0 conn(0x7f41b80b4ba0 legacy=0x7f41b806eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:09.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.025+0000 7f41affff700 1 -- 172.21.15.189:0/3843452057 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372558295 0 0) 0x7f41b80cdbe0 con 0x7f41b80b4ba0 2024-04-06T13:48:09.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.025+0000 7f41bd2e7700 1 -- 172.21.15.189:0/3843452057 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b815ac00 con 0x7f41b815a550 2024-04-06T13:48:09.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.025+0000 7f41bd2e7700 1 --2- 172.21.15.189:0/3843452057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41b815a550 0x7f41b80cf3c0 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009840 tx=0x7f41a8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=898455f4f83f906c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:09.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.026+0000 7f41affff700 1 -- 172.21.15.189:0/3843452057 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a8011830 con 0x7f41b815a550 2024-04-06T13:48:09.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.026+0000 7f41affff700 1 -- 172.21.15.189:0/3843452057 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a8011990 con 0x7f41b815a550 2024-04-06T13:48:09.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.026+0000 7f41affff700 1 -- 172.21.15.189:0/3843452057 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a8011d20 con 0x7f41b815a550 2024-04-06T13:48:09.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.026+0000 7f41bf54b700 1 -- 172.21.15.189:0/3843452057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41b815a550 msgr2=0x7f41b80cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:09.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.026+0000 7f41bf54b700 1 --2- 172.21.15.189:0/3843452057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41b815a550 0x7f41b80cf3c0 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009840 tx=0x7f41a8009b50 comp rx=0 tx=0).stop 2024-04-06T13:48:09.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.026+0000 7f41bf54b700 1 -- 172.21.15.189:0/3843452057 shutdown_connections 2024-04-06T13:48:09.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.026+0000 7f41bf54b700 1 --2- 172.21.15.189:0/3843452057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41b815a550 0x7f41b80cf3c0 unknown :-1 s=CLOSED pgs=2625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:09.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.026+0000 7f41bf54b700 1 --2- 172.21.15.189:0/3843452057 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b80cf900 0x7f41b80b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:09.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.026+0000 7f41bf54b700 1 -- 172.21.15.189:0/3843452057 >> 172.21.15.189:0/3843452057 conn(0x7f41b80d00b0 msgr2=0x7f41b80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:09.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.026+0000 7f41bf54b700 1 -- 172.21.15.189:0/3843452057 shutdown_connections 2024-04-06T13:48:09.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.026+0000 7f41bf54b700 1 -- 172.21.15.189:0/3843452057 wait complete. 2024-04-06T13:48:09.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.027+0000 7f41bf54b700 1 Processor -- start 2024-04-06T13:48:09.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.027+0000 7f41bf54b700 1 -- start start 2024-04-06T13:48:09.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.028+0000 7f41bf54b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41b80b4ba0 0x7f41b80b1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:09.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.028+0000 7f41bf54b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41b80cf900 0x7f41b81596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:09.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.028+0000 7f41bf54b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b8154610 0x7f41b8154a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:09.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.028+0000 7f41bf54b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b8154fd0 con 0x7f41b80cf900 2024-04-06T13:48:09.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.028+0000 7f41bf54b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b8159210 con 0x7f41b80b4ba0 2024-04-06T13:48:09.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.028+0000 7f41bf54b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b81f2740 con 0x7f41b8154610 2024-04-06T13:48:09.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.028+0000 7f41bcae6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41b80cf900 0x7f41b81596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:09.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.028+0000 7f41bcae6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41b80cf900 0x7f41b81596a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41846/0 (socket says 172.21.15.189:41846) 2024-04-06T13:48:09.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.028+0000 7f41bcae6700 1 -- 172.21.15.189:0/1006041701 learned_addr learned my addr 172.21.15.189:0/1006041701 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:09.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.028+0000 7f41bd2e7700 1 --2- 172.21.15.189:0/1006041701 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41b80b4ba0 0x7f41b80b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:09.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.028+0000 7f41bdae8700 1 --2- 172.21.15.189:0/1006041701 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b8154610 0x7f41b8154a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:09.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.029+0000 7f41bcae6700 1 -- 172.21.15.189:0/1006041701 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b8154610 msgr2=0x7f41b8154a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:09.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.029+0000 7f41bcae6700 1 --2- 172.21.15.189:0/1006041701 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b8154610 0x7f41b8154a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:09.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.029+0000 7f41bcae6700 1 -- 172.21.15.189:0/1006041701 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41b80b4ba0 msgr2=0x7f41b80b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:09.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.029+0000 7f41bcae6700 1 --2- 172.21.15.189:0/1006041701 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41b80b4ba0 0x7f41b80b1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:09.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.029+0000 7f41bcae6700 1 -- 172.21.15.189:0/1006041701 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a80094a0 con 0x7f41b80cf900 2024-04-06T13:48:09.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.029+0000 7f41bcae6700 1 --2- 172.21.15.189:0/1006041701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41b80cf900 0x7f41b81596a0 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f41b400e840 tx=0x7f41b400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:09.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.029+0000 7f41adffb700 1 -- 172.21.15.189:0/1006041701 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b4013610 con 0x7f41b80cf900 2024-04-06T13:48:09.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.029+0000 7f41adffb700 1 -- 172.21.15.189:0/1006041701 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b4013770 con 0x7f41b80cf900 2024-04-06T13:48:09.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.029+0000 7f41bf54b700 1 -- 172.21.15.189:0/1006041701 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b8158640 con 0x7f41b80cf900 2024-04-06T13:48:09.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.029+0000 7f41adffb700 1 -- 172.21.15.189:0/1006041701 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b4013b20 con 0x7f41b80cf900 2024-04-06T13:48:09.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.029+0000 7f41bf54b700 1 -- 172.21.15.189:0/1006041701 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b8158bc0 con 0x7f41b80cf900 2024-04-06T13:48:09.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.030+0000 7f41bf54b700 1 -- 172.21.15.189:0/1006041701 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b81593a0 con 0x7f41b80cf900 2024-04-06T13:48:09.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.030+0000 7f41adffb700 1 -- 172.21.15.189:0/1006041701 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f41b4013c80 con 0x7f41b80cf900 2024-04-06T13:48:09.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.031+0000 7f41adffb700 1 --2- 172.21.15.189:0/1006041701 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f419c06af00 0x7f419c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:09.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.031+0000 7f41adffb700 1 -- 172.21.15.189:0/1006041701 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6572+0+0 (secure 0 0 0) 0x7f41b4089800 con 0x7f41b80cf900 2024-04-06T13:48:09.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.031+0000 7f41bd2e7700 1 --2- 172.21.15.189:0/1006041701 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f419c06af00 0x7f419c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:09.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.031+0000 7f41bd2e7700 1 --2- 172.21.15.189:0/1006041701 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f419c06af00 0x7f419c06d3c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f41a80090b0 tx=0x7f41a8008030 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:09.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.035+0000 7f41adffb700 1 -- 172.21.15.189:0/1006041701 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f41b4054210 con 0x7f41b80cf900 2024-04-06T13:48:09.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:09.228+0000 7f41bf54b700 1 -- 172.21.15.189:0/1006041701 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f41b8067bb0 con 0x7f41b80cf900 2024-04-06T13:48:10.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.232+0000 7f41adffb700 1 -- 172.21.15.189:0/1006041701 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v239) v1 ==== 178+0+0 (secure 0 0 0) 0x7f41b4057830 con 0x7f41b80cf900 2024-04-06T13:48:10.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.236+0000 7f41bf54b700 1 -- 172.21.15.189:0/1006041701 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f419c06af00 msgr2=0x7f419c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:10.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.236+0000 7f41bf54b700 1 --2- 172.21.15.189:0/1006041701 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f419c06af00 0x7f419c06d3c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f41a80090b0 tx=0x7f41a8008030 comp rx=0 tx=0).stop 2024-04-06T13:48:10.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.236+0000 7f41bf54b700 1 -- 172.21.15.189:0/1006041701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41b80cf900 msgr2=0x7f41b81596a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:10.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.236+0000 7f41bf54b700 1 --2- 172.21.15.189:0/1006041701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41b80cf900 0x7f41b81596a0 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f41b400e840 tx=0x7f41b400ec00 comp rx=0 tx=0).stop 2024-04-06T13:48:10.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.236+0000 7f41bf54b700 1 -- 172.21.15.189:0/1006041701 shutdown_connections 2024-04-06T13:48:10.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.236+0000 7f41bf54b700 1 --2- 172.21.15.189:0/1006041701 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f419c06af00 0x7f419c06d3c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.236+0000 7f41bf54b700 1 --2- 172.21.15.189:0/1006041701 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b8154610 0x7f41b8154a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.236+0000 7f41bf54b700 1 --2- 172.21.15.189:0/1006041701 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41b80b4ba0 0x7f41b80b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.236+0000 7f41bf54b700 1 --2- 172.21.15.189:0/1006041701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41b80cf900 0x7f41b81596a0 unknown :-1 s=CLOSED pgs=2626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.236+0000 7f41bf54b700 1 -- 172.21.15.189:0/1006041701 >> 172.21.15.189:0/1006041701 conn(0x7f41b80d00b0 msgr2=0x7f41b80b3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:10.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.237+0000 7f41bf54b700 1 -- 172.21.15.189:0/1006041701 shutdown_connections 2024-04-06T13:48:10.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.237+0000 7f41bf54b700 1 -- 172.21.15.189:0/1006041701 wait complete. 2024-04-06T13:48:10.243 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'backup_fs_data' removed 2024-04-06T13:48:10.254 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-06T13:48:10.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.384+0000 7f8ba6b8c700 1 Processor -- start 2024-04-06T13:48:10.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.384+0000 7f8ba6b8c700 1 -- start start 2024-04-06T13:48:10.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.384+0000 7f8ba6b8c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ba015a550 0x7f8ba00cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:10.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.384+0000 7f8ba6b8c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8ba00cf900 0x7f8ba00b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:10.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.384+0000 7f8ba6b8c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8ba00b4ba0 0x7f8ba006eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:10.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.384+0000 7f8ba6b8c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba00cdbe0 con 0x7f8ba00b4ba0 2024-04-06T13:48:10.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.384+0000 7f8ba6b8c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba00b5010 con 0x7f8ba00cf900 2024-04-06T13:48:10.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.384+0000 7f8ba6b8c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba015a940 con 0x7f8ba015a550 2024-04-06T13:48:10.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.384+0000 7f8ba5129700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8ba00b4ba0 0x7f8ba006eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:10.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.385+0000 7f8ba5129700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8ba00b4ba0 0x7f8ba006eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51502/0 (socket says 172.21.15.189:51502) 2024-04-06T13:48:10.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.385+0000 7f8ba5129700 1 -- 172.21.15.189:0/613260828 learned_addr learned my addr 172.21.15.189:0/613260828 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:10.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.385+0000 7f8ba5129700 1 -- 172.21.15.189:0/613260828 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ba015a550 msgr2=0x7f8ba00cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:10.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.385+0000 7f8ba5129700 1 --2- 172.21.15.189:0/613260828 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ba015a550 0x7f8ba00cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.385+0000 7f8ba5129700 1 -- 172.21.15.189:0/613260828 >> v2:172.21.15.192:3300/0 conn(0x7f8ba00cf900 msgr2=0x7f8ba00b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:10.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.385+0000 7f8b9ffff700 1 --2- 172.21.15.189:0/613260828 >> v2:172.21.15.192:3300/0 conn(0x7f8ba00cf900 0x7f8ba00b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:10.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.385+0000 7f8ba5129700 1 --2- 172.21.15.189:0/613260828 >> v2:172.21.15.192:3300/0 conn(0x7f8ba00cf900 0x7f8ba00b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.385+0000 7f8ba5129700 1 -- 172.21.15.189:0/613260828 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba015ac00 con 0x7f8ba00b4ba0 2024-04-06T13:48:10.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.385+0000 7f8ba5129700 1 --2- 172.21.15.189:0/613260828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba00b4ba0 0x7f8ba006eb50 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f8ba006cb90 tx=0x7f8b9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8433c8f0b706eb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:10.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.385+0000 7f8b9f7fe700 1 -- 172.21.15.189:0/613260828 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9800bbc0 con 0x7f8ba00b4ba0 2024-04-06T13:48:10.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.385+0000 7f8b9f7fe700 1 -- 172.21.15.189:0/613260828 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9800bd20 con 0x7f8ba00b4ba0 2024-04-06T13:48:10.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.385+0000 7f8b9f7fe700 1 -- 172.21.15.189:0/613260828 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b980116a0 con 0x7f8ba00b4ba0 2024-04-06T13:48:10.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.386+0000 7f8ba6b8c700 1 -- 172.21.15.189:0/613260828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba00b4ba0 msgr2=0x7f8ba006eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:10.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.386+0000 7f8ba6b8c700 1 --2- 172.21.15.189:0/613260828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba00b4ba0 0x7f8ba006eb50 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f8ba006cb90 tx=0x7f8b9800b330 comp rx=0 tx=0).stop 2024-04-06T13:48:10.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.386+0000 7f8ba6b8c700 1 -- 172.21.15.189:0/613260828 shutdown_connections 2024-04-06T13:48:10.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.386+0000 7f8ba6b8c700 1 --2- 172.21.15.189:0/613260828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba00b4ba0 0x7f8ba006eb50 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.386+0000 7f8ba6b8c700 1 --2- 172.21.15.189:0/613260828 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ba015a550 0x7f8ba00cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.386+0000 7f8ba6b8c700 1 --2- 172.21.15.189:0/613260828 >> v2:172.21.15.192:3300/0 conn(0x7f8ba00cf900 0x7f8ba00b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.386+0000 7f8ba6b8c700 1 -- 172.21.15.189:0/613260828 >> 172.21.15.189:0/613260828 conn(0x7f8ba00d00b0 msgr2=0x7f8ba00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:10.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.386+0000 7f8ba6b8c700 1 -- 172.21.15.189:0/613260828 shutdown_connections 2024-04-06T13:48:10.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.386+0000 7f8ba6b8c700 1 -- 172.21.15.189:0/613260828 wait complete. 2024-04-06T13:48:10.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.387+0000 7f8ba6b8c700 1 Processor -- start 2024-04-06T13:48:10.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.387+0000 7f8ba6b8c700 1 -- start start 2024-04-06T13:48:10.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.388+0000 7f8ba6b8c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba00b4ba0 0x7f8ba0150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:10.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.388+0000 7f8ba6b8c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ba00cf900 0x7f8ba01567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:10.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.388+0000 7f8ba6b8c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ba01494f0 0x7f8ba015c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:10.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.388+0000 7f8ba6b8c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba0154510 con 0x7f8ba00b4ba0 2024-04-06T13:48:10.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.388+0000 7f8ba6b8c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba0149ee0 con 0x7f8ba00cf900 2024-04-06T13:48:10.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.388+0000 7f8ba6b8c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba0149980 con 0x7f8ba01494f0 2024-04-06T13:48:10.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.388+0000 7f8ba4928700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba00b4ba0 0x7f8ba0150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:10.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.388+0000 7f8ba4928700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba00b4ba0 0x7f8ba0150a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51512/0 (socket says 172.21.15.189:51512) 2024-04-06T13:48:10.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.388+0000 7f8ba4928700 1 -- 172.21.15.189:0/2092679081 learned_addr learned my addr 172.21.15.189:0/2092679081 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:10.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.388+0000 7f8b9ffff700 1 --2- 172.21.15.189:0/2092679081 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ba00cf900 0x7f8ba01567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:10.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.388+0000 7f8ba5129700 1 --2- 172.21.15.189:0/2092679081 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ba01494f0 0x7f8ba015c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:10.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.388+0000 7f8ba4928700 1 -- 172.21.15.189:0/2092679081 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ba01494f0 msgr2=0x7f8ba015c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:10.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.388+0000 7f8ba4928700 1 --2- 172.21.15.189:0/2092679081 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ba01494f0 0x7f8ba015c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.388+0000 7f8ba4928700 1 -- 172.21.15.189:0/2092679081 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ba00cf900 msgr2=0x7f8ba01567b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:10.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.388+0000 7f8ba4928700 1 --2- 172.21.15.189:0/2092679081 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ba00cf900 0x7f8ba01567b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.388+0000 7f8ba4928700 1 -- 172.21.15.189:0/2092679081 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b98009b30 con 0x7f8ba00b4ba0 2024-04-06T13:48:10.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.388+0000 7f8b9ffff700 1 --2- 172.21.15.189:0/2092679081 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ba00cf900 0x7f8ba01567b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:10.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.388+0000 7f8ba4928700 1 --2- 172.21.15.189:0/2092679081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba00b4ba0 0x7f8ba0150a10 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400b530 tx=0x7f8b9400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:10.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.389+0000 7f8b9d7fa700 1 -- 172.21.15.189:0/2092679081 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b94011830 con 0x7f8ba00b4ba0 2024-04-06T13:48:10.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.389+0000 7f8b9d7fa700 1 -- 172.21.15.189:0/2092679081 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b940119b0 con 0x7f8ba00b4ba0 2024-04-06T13:48:10.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.389+0000 7f8ba6b8c700 1 -- 172.21.15.189:0/2092679081 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba01ee690 con 0x7f8ba00b4ba0 2024-04-06T13:48:10.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.389+0000 7f8b9d7fa700 1 -- 172.21.15.189:0/2092679081 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b94011d60 con 0x7f8ba00b4ba0 2024-04-06T13:48:10.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.389+0000 7f8ba6b8c700 1 -- 172.21.15.189:0/2092679081 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba0153b70 con 0x7f8ba00b4ba0 2024-04-06T13:48:10.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.390+0000 7f8ba6b8c700 1 -- 172.21.15.189:0/2092679081 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba0067bb0 con 0x7f8ba00b4ba0 2024-04-06T13:48:10.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.390+0000 7f8b9d7fa700 1 -- 172.21.15.189:0/2092679081 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f8b9401a400 con 0x7f8ba00b4ba0 2024-04-06T13:48:10.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.390+0000 7f8b9d7fa700 1 --2- 172.21.15.189:0/2092679081 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8b8406aed0 0x7f8b8406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:10.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.391+0000 7f8b9d7fa700 1 -- 172.21.15.189:0/2092679081 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f8b94089ae0 con 0x7f8ba00b4ba0 2024-04-06T13:48:10.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.391+0000 7f8b9ffff700 1 --2- 172.21.15.189:0/2092679081 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8b8406aed0 0x7f8b8406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:10.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.391+0000 7f8b9ffff700 1 --2- 172.21.15.189:0/2092679081 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8b8406aed0 0x7f8b8406d390 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8ba015d810 tx=0x7f8b8c008040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:10.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.395+0000 7f8b9d7fa700 1 -- 172.21.15.189:0/2092679081 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8b94054670 con 0x7f8ba00b4ba0 2024-04-06T13:48:10.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.588+0000 7f8ba6b8c700 1 -- 172.21.15.189:0/2092679081 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f8ba0074990 con 0x7f8ba00b4ba0 2024-04-06T13:48:10.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.592+0000 7f8b9d7fa700 1 -- 172.21.15.189:0/2092679081 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v79) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8b94015020 con 0x7f8ba00b4ba0 2024-04-06T13:48:10.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.594+0000 7f8ba6b8c700 1 -- 172.21.15.189:0/2092679081 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8b8406aed0 msgr2=0x7f8b8406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:10.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.594+0000 7f8ba6b8c700 1 --2- 172.21.15.189:0/2092679081 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8b8406aed0 0x7f8b8406d390 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8ba015d810 tx=0x7f8b8c008040 comp rx=0 tx=0).stop 2024-04-06T13:48:10.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.594+0000 7f8ba6b8c700 1 -- 172.21.15.189:0/2092679081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba00b4ba0 msgr2=0x7f8ba0150a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:10.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.594+0000 7f8ba6b8c700 1 --2- 172.21.15.189:0/2092679081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba00b4ba0 0x7f8ba0150a10 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400b530 tx=0x7f8b9400b8f0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.594+0000 7f8ba6b8c700 1 -- 172.21.15.189:0/2092679081 shutdown_connections 2024-04-06T13:48:10.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.594+0000 7f8ba6b8c700 1 --2- 172.21.15.189:0/2092679081 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8b8406aed0 0x7f8b8406d390 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.594+0000 7f8ba6b8c700 1 --2- 172.21.15.189:0/2092679081 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ba01494f0 0x7f8ba015c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.594+0000 7f8ba6b8c700 1 --2- 172.21.15.189:0/2092679081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba00b4ba0 0x7f8ba0150a10 unknown :-1 s=CLOSED pgs=2628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.594+0000 7f8ba6b8c700 1 --2- 172.21.15.189:0/2092679081 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ba00cf900 0x7f8ba01567b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.594+0000 7f8ba6b8c700 1 -- 172.21.15.189:0/2092679081 >> 172.21.15.189:0/2092679081 conn(0x7f8ba00d00b0 msgr2=0x7f8ba00cc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:10.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.594+0000 7f8ba6b8c700 1 -- 172.21.15.189:0/2092679081 shutdown_connections 2024-04-06T13:48:10.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.594+0000 7f8ba6b8c700 1 -- 172.21.15.189:0/2092679081 wait complete. 2024-04-06T13:48:10.610 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-06T13:48:10.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.735+0000 7f43636cf700 1 Processor -- start 2024-04-06T13:48:10.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.736+0000 7f43636cf700 1 -- start start 2024-04-06T13:48:10.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.736+0000 7f43636cf700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f435c0d11b0 0x7f435c0d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:10.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.736+0000 7f43636cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f435c0cfa00 0x7f435c0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:10.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.736+0000 7f43636cf700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c0d1ad0 con 0x7f435c0cb870 2024-04-06T13:48:10.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.736+0000 7f43636cf700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f435c0ccdc0 con 0x7f435c0d11b0 2024-04-06T13:48:10.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.737+0000 7f43636cf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435c0d0b50 con 0x7f435c0cfa00 2024-04-06T13:48:10.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.737+0000 7f436146b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f435c0d11b0 0x7f435c0d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:10.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.737+0000 7f436146b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f435c0d11b0 0x7f435c0d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51520/0 (socket says 172.21.15.189:51520) 2024-04-06T13:48:10.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.737+0000 7f436146b700 1 -- 172.21.15.189:0/4052139149 learned_addr learned my addr 172.21.15.189:0/4052139149 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:10.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.737+0000 7f4361c6c700 1 --2- 172.21.15.189:0/4052139149 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f435c0cfa00 0x7f435c0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:10.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.737+0000 7f436146b700 1 -- 172.21.15.189:0/4052139149 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f435c0cfa00 msgr2=0x7f435c0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:10.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.737+0000 7f436146b700 1 --2- 172.21.15.189:0/4052139149 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f435c0cfa00 0x7f435c0cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.737+0000 7f436146b700 1 -- 172.21.15.189:0/4052139149 >> v1:172.21.15.192:6789/0 conn(0x7f435c0cb870 legacy=0x7f435c15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:10.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.737+0000 7f436146b700 1 -- 172.21.15.189:0/4052139149 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c0cbd80 con 0x7f435c0d11b0 2024-04-06T13:48:10.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.737+0000 7f436146b700 1 --2- 172.21.15.189:0/4052139149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f435c0d11b0 0x7f435c0d1590 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7f43500098f0 tx=0x7f4350009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adaff1aebd8ae74a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:10.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.737+0000 7f434ffff700 1 -- 172.21.15.189:0/4052139149 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4350004230 con 0x7f435c0d11b0 2024-04-06T13:48:10.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.737+0000 7f434ffff700 1 -- 172.21.15.189:0/4052139149 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4350004390 con 0x7f435c0d11b0 2024-04-06T13:48:10.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.737+0000 7f434ffff700 1 -- 172.21.15.189:0/4052139149 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43500036a0 con 0x7f435c0d11b0 2024-04-06T13:48:10.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.738+0000 7f43636cf700 1 -- 172.21.15.189:0/4052139149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f435c0d11b0 msgr2=0x7f435c0d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:10.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.738+0000 7f43636cf700 1 --2- 172.21.15.189:0/4052139149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f435c0d11b0 0x7f435c0d1590 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7f43500098f0 tx=0x7f4350009c00 comp rx=0 tx=0).stop 2024-04-06T13:48:10.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.738+0000 7f43636cf700 1 -- 172.21.15.189:0/4052139149 shutdown_connections 2024-04-06T13:48:10.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.738+0000 7f43636cf700 1 --2- 172.21.15.189:0/4052139149 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f435c0cfa00 0x7f435c0cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.738+0000 7f43636cf700 1 --2- 172.21.15.189:0/4052139149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f435c0d11b0 0x7f435c0d1590 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.738+0000 7f43636cf700 1 -- 172.21.15.189:0/4052139149 >> 172.21.15.189:0/4052139149 conn(0x7f435c0b2db0 msgr2=0x7f435c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:10.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.738+0000 7f43636cf700 1 -- 172.21.15.189:0/4052139149 shutdown_connections 2024-04-06T13:48:10.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.738+0000 7f43636cf700 1 -- 172.21.15.189:0/4052139149 wait complete. 2024-04-06T13:48:10.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.739+0000 7f43636cf700 1 Processor -- start 2024-04-06T13:48:10.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.740+0000 7f43636cf700 1 -- start start 2024-04-06T13:48:10.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.740+0000 7f43636cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f435c0cb870 0x7f435c0ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:10.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.740+0000 7f43636cf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f435c0cfa00 0x7f435c157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:10.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.740+0000 7f43636cf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f435c0d11b0 0x7f435c0b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:10.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.740+0000 7f43636cf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c157ac0 con 0x7f435c0d11b0 2024-04-06T13:48:10.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.740+0000 7f43636cf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c1f4370 con 0x7f435c0cfa00 2024-04-06T13:48:10.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.740+0000 7f43636cf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435c1f45c0 con 0x7f435c0cb870 2024-04-06T13:48:10.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.741+0000 7f4361c6c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f435c0d11b0 0x7f435c0b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:10.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.741+0000 7f4361c6c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f435c0d11b0 0x7f435c0b6350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51524/0 (socket says 172.21.15.189:51524) 2024-04-06T13:48:10.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.741+0000 7f4361c6c700 1 -- 172.21.15.189:0/2131241785 learned_addr learned my addr 172.21.15.189:0/2131241785 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:10.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.741+0000 7f436146b700 1 --2- 172.21.15.189:0/2131241785 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f435c0cb870 0x7f435c0ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:10.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.741+0000 7f4361c6c700 1 -- 172.21.15.189:0/2131241785 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f435c0cb870 msgr2=0x7f435c0ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:10.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.741+0000 7f4361c6c700 1 --2- 172.21.15.189:0/2131241785 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f435c0cb870 0x7f435c0ca4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.741+0000 7f4361c6c700 1 -- 172.21.15.189:0/2131241785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f435c0cfa00 msgr2=0x7f435c157580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:10.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.741+0000 7f4360c6a700 1 --2- 172.21.15.189:0/2131241785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f435c0cfa00 0x7f435c157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:10.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.741+0000 7f4361c6c700 1 --2- 172.21.15.189:0/2131241785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f435c0cfa00 0x7f435c157580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.741+0000 7f4361c6c700 1 -- 172.21.15.189:0/2131241785 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4350009550 con 0x7f435c0d11b0 2024-04-06T13:48:10.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.741+0000 7f4360c6a700 1 --2- 172.21.15.189:0/2131241785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f435c0cfa00 0x7f435c157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:10.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.741+0000 7f4361c6c700 1 --2- 172.21.15.189:0/2131241785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f435c0d11b0 0x7f435c0b6350 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f435400cc30 tx=0x7f435400a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:10.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.741+0000 7f434dffb700 1 -- 172.21.15.189:0/2131241785 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435400ab70 con 0x7f435c0d11b0 2024-04-06T13:48:10.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.741+0000 7f434dffb700 1 -- 172.21.15.189:0/2131241785 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435400acd0 con 0x7f435c0d11b0 2024-04-06T13:48:10.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.741+0000 7f43636cf700 1 -- 172.21.15.189:0/2131241785 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f435c1f48a0 con 0x7f435c0d11b0 2024-04-06T13:48:10.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.741+0000 7f434dffb700 1 -- 172.21.15.189:0/2131241785 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4354003c40 con 0x7f435c0d11b0 2024-04-06T13:48:10.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.741+0000 7f43636cf700 1 -- 172.21.15.189:0/2131241785 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f435c1f4e60 con 0x7f435c0d11b0 2024-04-06T13:48:10.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.742+0000 7f43636cf700 1 -- 172.21.15.189:0/2131241785 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f435c0c82d0 con 0x7f435c0d11b0 2024-04-06T13:48:10.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.743+0000 7f434dffb700 1 -- 172.21.15.189:0/2131241785 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f435400f3f0 con 0x7f435c0d11b0 2024-04-06T13:48:10.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.743+0000 7f434dffb700 1 --2- 172.21.15.189:0/2131241785 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f434006aed0 0x7f434006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:10.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.743+0000 7f436146b700 1 --2- 172.21.15.189:0/2131241785 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f434006aed0 0x7f434006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:10.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.743+0000 7f434dffb700 1 -- 172.21.15.189:0/2131241785 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f4354089c10 con 0x7f435c0d11b0 2024-04-06T13:48:10.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.744+0000 7f436146b700 1 --2- 172.21.15.189:0/2131241785 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f434006aed0 0x7f434006d390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4350009c40 tx=0x7f4350008030 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:10.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.748+0000 7f434dffb700 1 -- 172.21.15.189:0/2131241785 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f43540547c0 con 0x7f435c0d11b0 2024-04-06T13:48:10.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.944+0000 7f43636cf700 1 -- 172.21.15.189:0/2131241785 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f435c074990 con 0x7f435c0d11b0 2024-04-06T13:48:10.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.948+0000 7f434dffb700 1 -- 172.21.15.189:0/2131241785 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v80) v1 ==== 115+0+0 (secure 0 0 0) 0x7f435401d070 con 0x7f435c0d11b0 2024-04-06T13:48:10.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.951+0000 7f43636cf700 1 -- 172.21.15.189:0/2131241785 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f434006aed0 msgr2=0x7f434006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:10.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.951+0000 7f43636cf700 1 --2- 172.21.15.189:0/2131241785 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f434006aed0 0x7f434006d390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4350009c40 tx=0x7f4350008030 comp rx=0 tx=0).stop 2024-04-06T13:48:10.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.951+0000 7f43636cf700 1 -- 172.21.15.189:0/2131241785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f435c0d11b0 msgr2=0x7f435c0b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:10.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.951+0000 7f43636cf700 1 --2- 172.21.15.189:0/2131241785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f435c0d11b0 0x7f435c0b6350 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f435400cc30 tx=0x7f435400a1f0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.951+0000 7f43636cf700 1 -- 172.21.15.189:0/2131241785 shutdown_connections 2024-04-06T13:48:10.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.951+0000 7f43636cf700 1 --2- 172.21.15.189:0/2131241785 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f434006aed0 0x7f434006d390 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.951+0000 7f43636cf700 1 --2- 172.21.15.189:0/2131241785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f435c0d11b0 0x7f435c0b6350 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.951+0000 7f43636cf700 1 --2- 172.21.15.189:0/2131241785 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f435c0cb870 0x7f435c0ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.951+0000 7f43636cf700 1 --2- 172.21.15.189:0/2131241785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f435c0cfa00 0x7f435c157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:10.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.951+0000 7f43636cf700 1 -- 172.21.15.189:0/2131241785 >> 172.21.15.189:0/2131241785 conn(0x7f435c0b2db0 msgr2=0x7f435c0cd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:10.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.951+0000 7f43636cf700 1 -- 172.21.15.189:0/2131241785 shutdown_connections 2024-04-06T13:48:10.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:10.951+0000 7f43636cf700 1 -- 172.21.15.189:0/2131241785 wait complete. 2024-04-06T13:48:10.965 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-06T13:48:11.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.088+0000 7fda808c5700 1 Processor -- start 2024-04-06T13:48:11.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.088+0000 7fda808c5700 1 -- start start 2024-04-06T13:48:11.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.089+0000 7fda808c5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fda7c15d7b0 0x7fda7c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:11.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.089+0000 7fda808c5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda7c06a730 0x7fda7c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:11.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.089+0000 7fda808c5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda7c0b3990 con 0x7fda7c15ddf0 2024-04-06T13:48:11.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.089+0000 7fda808c5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda7c06b0f0 con 0x7fda7c15d7b0 2024-04-06T13:48:11.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.089+0000 7fda808c5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda7c0b3060 con 0x7fda7c06a730 2024-04-06T13:48:11.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.089+0000 7fda7a59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fda7c15d7b0 0x7fda7c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:11.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.090+0000 7fda7a59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fda7c15d7b0 0x7fda7c0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51526/0 (socket says 172.21.15.189:51526) 2024-04-06T13:48:11.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.090+0000 7fda7a59c700 1 -- 172.21.15.189:0/730923848 learned_addr learned my addr 172.21.15.189:0/730923848 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:11.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.090+0000 7fda79d9b700 1 --2- 172.21.15.189:0/730923848 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda7c06a730 0x7fda7c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:11.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.090+0000 7fda7a59c700 1 -- 172.21.15.189:0/730923848 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda7c06a730 msgr2=0x7fda7c06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:11.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.090+0000 7fda7a59c700 1 --2- 172.21.15.189:0/730923848 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda7c06a730 0x7fda7c06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:11.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.090+0000 7fda7a59c700 1 -- 172.21.15.189:0/730923848 >> v1:172.21.15.192:6789/0 conn(0x7fda7c15ddf0 legacy=0x7fda7c15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:11.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.090+0000 7fda7a59c700 1 -- 172.21.15.189:0/730923848 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7c0b3340 con 0x7fda7c15d7b0 2024-04-06T13:48:11.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.090+0000 7fda7a59c700 1 --2- 172.21.15.189:0/730923848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda7c15d7b0 0x7fda7c0b3e20 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7fda700098f0 tx=0x7fda70009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc2cfe9e6b4f8387 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:11.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.090+0000 7fda7959a700 1 -- 172.21.15.189:0/730923848 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda70011830 con 0x7fda7c15d7b0 2024-04-06T13:48:11.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.090+0000 7fda7959a700 1 -- 172.21.15.189:0/730923848 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda70011990 con 0x7fda7c15d7b0 2024-04-06T13:48:11.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.090+0000 7fda7959a700 1 -- 172.21.15.189:0/730923848 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda70011d20 con 0x7fda7c15d7b0 2024-04-06T13:48:11.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.091+0000 7fda808c5700 1 -- 172.21.15.189:0/730923848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda7c15d7b0 msgr2=0x7fda7c0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:11.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.091+0000 7fda808c5700 1 --2- 172.21.15.189:0/730923848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda7c15d7b0 0x7fda7c0b3e20 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7fda700098f0 tx=0x7fda70009c00 comp rx=0 tx=0).stop 2024-04-06T13:48:11.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.091+0000 7fda808c5700 1 -- 172.21.15.189:0/730923848 shutdown_connections 2024-04-06T13:48:11.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.091+0000 7fda808c5700 1 --2- 172.21.15.189:0/730923848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda7c15d7b0 0x7fda7c0b3e20 unknown :-1 s=CLOSED pgs=2631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:11.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.091+0000 7fda808c5700 1 --2- 172.21.15.189:0/730923848 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda7c06a730 0x7fda7c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:11.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.091+0000 7fda808c5700 1 -- 172.21.15.189:0/730923848 >> 172.21.15.189:0/730923848 conn(0x7fda7c0d00b0 msgr2=0x7fda7c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:11.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.091+0000 7fda808c5700 1 -- 172.21.15.189:0/730923848 shutdown_connections 2024-04-06T13:48:11.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.091+0000 7fda808c5700 1 -- 172.21.15.189:0/730923848 wait complete. 2024-04-06T13:48:11.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.092+0000 7fda808c5700 1 Processor -- start 2024-04-06T13:48:11.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.092+0000 7fda808c5700 1 -- start start 2024-04-06T13:48:11.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.093+0000 7fda808c5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda7c06a730 0x7fda7c1f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:11.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.093+0000 7fda808c5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda7c15d7b0 0x7fda7c1f6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:11.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.093+0000 7fda808c5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda7c15ddf0 0x7fda7c1f6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:11.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.093+0000 7fda808c5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7c148c90 con 0x7fda7c15ddf0 2024-04-06T13:48:11.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.093+0000 7fda808c5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7c0b0d70 con 0x7fda7c06a730 2024-04-06T13:48:11.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.093+0000 7fda808c5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda7c0b1050 con 0x7fda7c15d7b0 2024-04-06T13:48:11.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.093+0000 7fda7ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda7c15ddf0 0x7fda7c1f6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:11.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.093+0000 7fda7ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda7c15ddf0 0x7fda7c1f6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51536/0 (socket says 172.21.15.189:51536) 2024-04-06T13:48:11.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.093+0000 7fda7ad9d700 1 -- 172.21.15.189:0/1815062428 learned_addr learned my addr 172.21.15.189:0/1815062428 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:11.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.093+0000 7fda7a59c700 1 --2- 172.21.15.189:0/1815062428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda7c06a730 0x7fda7c1f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:11.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.093+0000 7fda79d9b700 1 --2- 172.21.15.189:0/1815062428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda7c15d7b0 0x7fda7c1f6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:11.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.093+0000 7fda7ad9d700 1 -- 172.21.15.189:0/1815062428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda7c15d7b0 msgr2=0x7fda7c1f6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:11.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.093+0000 7fda7ad9d700 1 --2- 172.21.15.189:0/1815062428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda7c15d7b0 0x7fda7c1f6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:11.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.093+0000 7fda7ad9d700 1 -- 172.21.15.189:0/1815062428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda7c06a730 msgr2=0x7fda7c1f28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:11.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.093+0000 7fda7ad9d700 1 --2- 172.21.15.189:0/1815062428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda7c06a730 0x7fda7c1f28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:11.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.093+0000 7fda7ad9d700 1 -- 172.21.15.189:0/1815062428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda70009550 con 0x7fda7c15ddf0 2024-04-06T13:48:11.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.094+0000 7fda7ad9d700 1 --2- 172.21.15.189:0/1815062428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda7c15ddf0 0x7fda7c1f6990 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7fda7400cb50 tx=0x7fda7400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:11.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.094+0000 7fda6affd700 1 -- 172.21.15.189:0/1815062428 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74003c00 con 0x7fda7c15ddf0 2024-04-06T13:48:11.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.094+0000 7fda6affd700 1 -- 172.21.15.189:0/1815062428 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda74003d60 con 0x7fda7c15ddf0 2024-04-06T13:48:11.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.094+0000 7fda808c5700 1 -- 172.21.15.189:0/1815062428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda7c0b1330 con 0x7fda7c15ddf0 2024-04-06T13:48:11.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.094+0000 7fda6affd700 1 -- 172.21.15.189:0/1815062428 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7400aa20 con 0x7fda7c15ddf0 2024-04-06T13:48:11.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.094+0000 7fda808c5700 1 -- 172.21.15.189:0/1815062428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda7c148460 con 0x7fda7c15ddf0 2024-04-06T13:48:11.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.095+0000 7fda808c5700 1 -- 172.21.15.189:0/1815062428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda7c0b1850 con 0x7fda7c15ddf0 2024-04-06T13:48:11.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.095+0000 7fda6affd700 1 -- 172.21.15.189:0/1815062428 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fda7400ab80 con 0x7fda7c15ddf0 2024-04-06T13:48:11.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.095+0000 7fda6affd700 1 --2- 172.21.15.189:0/1815062428 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fda5c06aed0 0x7fda5c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:11.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.096+0000 7fda6affd700 1 -- 172.21.15.189:0/1815062428 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fda740897b0 con 0x7fda7c15ddf0 2024-04-06T13:48:11.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.096+0000 7fda7a59c700 1 --2- 172.21.15.189:0/1815062428 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fda5c06aed0 0x7fda5c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:11.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.096+0000 7fda7a59c700 1 --2- 172.21.15.189:0/1815062428 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fda5c06aed0 0x7fda5c06d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fda70009c40 tx=0x7fda70008030 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:11.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.100+0000 7fda6affd700 1 -- 172.21.15.189:0/1815062428 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fda7c0b1850 con 0x7fda7c15ddf0 2024-04-06T13:48:11.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.294+0000 7fda808c5700 1 -- 172.21.15.189:0/1815062428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fda7c0b1850 con 0x7fda7c15ddf0 2024-04-06T13:48:11.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.300+0000 7fda6affd700 1 -- 172.21.15.189:0/1815062428 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v81) v1 ==== 120+0+0 (secure 0 0 0) 0x7fda7c0b1850 con 0x7fda7c15ddf0 2024-04-06T13:48:11.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.303+0000 7fda808c5700 1 -- 172.21.15.189:0/1815062428 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fda5c06aed0 msgr2=0x7fda5c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:11.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.303+0000 7fda808c5700 1 --2- 172.21.15.189:0/1815062428 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fda5c06aed0 0x7fda5c06d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fda70009c40 tx=0x7fda70008030 comp rx=0 tx=0).stop 2024-04-06T13:48:11.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.303+0000 7fda808c5700 1 -- 172.21.15.189:0/1815062428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda7c15ddf0 msgr2=0x7fda7c1f6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:11.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.303+0000 7fda808c5700 1 --2- 172.21.15.189:0/1815062428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda7c15ddf0 0x7fda7c1f6990 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7fda7400cb50 tx=0x7fda7400ce60 comp rx=0 tx=0).stop 2024-04-06T13:48:11.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.303+0000 7fda808c5700 1 -- 172.21.15.189:0/1815062428 shutdown_connections 2024-04-06T13:48:11.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.303+0000 7fda808c5700 1 --2- 172.21.15.189:0/1815062428 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fda5c06aed0 0x7fda5c06d390 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:11.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.303+0000 7fda808c5700 1 --2- 172.21.15.189:0/1815062428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda7c15ddf0 0x7fda7c1f6990 unknown :-1 s=CLOSED pgs=2632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:11.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.303+0000 7fda808c5700 1 --2- 172.21.15.189:0/1815062428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda7c06a730 0x7fda7c1f28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:11.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.303+0000 7fda808c5700 1 --2- 172.21.15.189:0/1815062428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda7c15d7b0 0x7fda7c1f6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:11.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.303+0000 7fda808c5700 1 -- 172.21.15.189:0/1815062428 >> 172.21.15.189:0/1815062428 conn(0x7fda7c0d00b0 msgr2=0x7fda7c0adbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:11.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.303+0000 7fda808c5700 1 -- 172.21.15.189:0/1815062428 shutdown_connections 2024-04-06T13:48:11.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.303+0000 7fda808c5700 1 -- 172.21.15.189:0/1815062428 wait complete. 2024-04-06T13:48:11.317 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2024-04-06T13:48:11.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.437+0000 7ffbe6398700 1 Processor -- start 2024-04-06T13:48:11.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.437+0000 7ffbe6398700 1 -- start start 2024-04-06T13:48:11.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.438+0000 7ffbe6398700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ffbe015d7b0 0x7ffbe00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:11.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.438+0000 7ffbe6398700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffbe015e500 0x7ffbe006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:11.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.438+0000 7ffbe6398700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe00b3990 con 0x7ffbe006a730 2024-04-06T13:48:11.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.438+0000 7ffbe6398700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbe015e950 con 0x7ffbe015e500 2024-04-06T13:48:11.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.438+0000 7ffbe6398700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbe006b0f0 con 0x7ffbe015d7b0 2024-04-06T13:48:11.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.438+0000 7ffbe4935700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffbe015e500 0x7ffbe006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:11.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.438+0000 7ffbe4935700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffbe015e500 0x7ffbe006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51540/0 (socket says 172.21.15.189:51540) 2024-04-06T13:48:11.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.438+0000 7ffbe4935700 1 -- 172.21.15.189:0/204633913 learned_addr learned my addr 172.21.15.189:0/204633913 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:11.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.438+0000 7ffbdf7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ffbe006a730 0x7ffbe015ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47964/0 (socket says 172.21.15.189:47964) 2024-04-06T13:48:11.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.439+0000 7ffbdffff700 1 --2- 172.21.15.189:0/204633913 >> v2:172.21.15.192:3300/0 conn(0x7ffbe015d7b0 0x7ffbe00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:11.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.439+0000 7ffbe4935700 1 -- 172.21.15.189:0/204633913 >> v2:172.21.15.192:3300/0 conn(0x7ffbe015d7b0 msgr2=0x7ffbe00b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:11.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.439+0000 7ffbe4935700 1 --2- 172.21.15.189:0/204633913 >> v2:172.21.15.192:3300/0 conn(0x7ffbe015d7b0 0x7ffbe00b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:11.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.439+0000 7ffbe4935700 1 -- 172.21.15.189:0/204633913 >> v1:172.21.15.189:6789/0 conn(0x7ffbe006a730 legacy=0x7ffbe015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:11.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.439+0000 7ffbdeffd700 1 -- 172.21.15.189:0/204633913 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4179996596 0 0) 0x7ffbe00b3990 con 0x7ffbe006a730 2024-04-06T13:48:11.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.439+0000 7ffbe4935700 1 -- 172.21.15.189:0/204633913 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe00b3060 con 0x7ffbe015e500 2024-04-06T13:48:11.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.439+0000 7ffbe4935700 1 --2- 172.21.15.189:0/204633913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbe015e500 0x7ffbe006abb0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7ffbe00b6200 tx=0x7ffbd800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=664a01ea70613e74 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:11.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.439+0000 7ffbdeffd700 1 -- 172.21.15.189:0/204633913 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd800e070 con 0x7ffbe015e500 2024-04-06T13:48:11.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.439+0000 7ffbdeffd700 1 -- 172.21.15.189:0/204633913 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd800b9c0 con 0x7ffbe015e500 2024-04-06T13:48:11.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.439+0000 7ffbdeffd700 1 -- 172.21.15.189:0/204633913 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd800bd70 con 0x7ffbe015e500 2024-04-06T13:48:11.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.439+0000 7ffbe6398700 1 -- 172.21.15.189:0/204633913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbe015e500 msgr2=0x7ffbe006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:11.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.439+0000 7ffbe6398700 1 --2- 172.21.15.189:0/204633913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbe015e500 0x7ffbe006abb0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7ffbe00b6200 tx=0x7ffbd800b330 comp rx=0 tx=0).stop 2024-04-06T13:48:11.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.439+0000 7ffbe6398700 1 -- 172.21.15.189:0/204633913 shutdown_connections 2024-04-06T13:48:11.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.439+0000 7ffbe6398700 1 --2- 172.21.15.189:0/204633913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbe015e500 0x7ffbe006abb0 unknown :-1 s=CLOSED pgs=2634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:11.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.439+0000 7ffbe6398700 1 --2- 172.21.15.189:0/204633913 >> v2:172.21.15.192:3300/0 conn(0x7ffbe015d7b0 0x7ffbe00b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:11.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.439+0000 7ffbe6398700 1 -- 172.21.15.189:0/204633913 >> 172.21.15.189:0/204633913 conn(0x7ffbe00d00b0 msgr2=0x7ffbe00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:11.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.440+0000 7ffbe6398700 1 -- 172.21.15.189:0/204633913 shutdown_connections 2024-04-06T13:48:11.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.440+0000 7ffbe6398700 1 -- 172.21.15.189:0/204633913 wait complete. 2024-04-06T13:48:11.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.441+0000 7ffbe6398700 1 Processor -- start 2024-04-06T13:48:11.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.441+0000 7ffbe6398700 1 -- start start 2024-04-06T13:48:11.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbe6398700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbe006a730 0x7ffbe0148710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:11.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbe6398700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbe015d7b0 0x7ffbe01f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:11.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbe6398700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbe015e500 0x7ffbe01f2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:11.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbe6398700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe015bb50 con 0x7ffbe006a730 2024-04-06T13:48:11.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbe6398700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe015cba0 con 0x7ffbe015d7b0 2024-04-06T13:48:11.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbe6398700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe015ac50 con 0x7ffbe015e500 2024-04-06T13:48:11.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbdffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbe006a730 0x7ffbe0148710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:11.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbdffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbe006a730 0x7ffbe0148710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51544/0 (socket says 172.21.15.189:51544) 2024-04-06T13:48:11.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbdffff700 1 -- 172.21.15.189:0/972245002 learned_addr learned my addr 172.21.15.189:0/972245002 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:11.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbdf7fe700 1 --2- 172.21.15.189:0/972245002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbe015d7b0 0x7ffbe01f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:11.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbe4935700 1 --2- 172.21.15.189:0/972245002 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbe015e500 0x7ffbe01f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:11.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbdffff700 1 -- 172.21.15.189:0/972245002 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbe015e500 msgr2=0x7ffbe01f2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:11.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbdffff700 1 --2- 172.21.15.189:0/972245002 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbe015e500 0x7ffbe01f2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:11.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbdffff700 1 -- 172.21.15.189:0/972245002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbe015d7b0 msgr2=0x7ffbe01f1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:11.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbdffff700 1 --2- 172.21.15.189:0/972245002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbe015d7b0 0x7ffbe01f1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:11.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbdffff700 1 -- 172.21.15.189:0/972245002 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd8009b30 con 0x7ffbe006a730 2024-04-06T13:48:11.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbdffff700 1 --2- 172.21.15.189:0/972245002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbe006a730 0x7ffbe0148710 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400e810 tx=0x7ffbd400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:11.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbdcff9700 1 -- 172.21.15.189:0/972245002 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4013610 con 0x7ffbe006a730 2024-04-06T13:48:11.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbdcff9700 1 -- 172.21.15.189:0/972245002 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd4013770 con 0x7ffbe006a730 2024-04-06T13:48:11.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbe6398700 1 -- 172.21.15.189:0/972245002 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe015aed0 con 0x7ffbe006a730 2024-04-06T13:48:11.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbdcff9700 1 -- 172.21.15.189:0/972245002 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4013b20 con 0x7ffbe006a730 2024-04-06T13:48:11.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.442+0000 7ffbe6398700 1 -- 172.21.15.189:0/972245002 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe015b490 con 0x7ffbe006a730 2024-04-06T13:48:11.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.443+0000 7ffbe6398700 1 -- 172.21.15.189:0/972245002 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe0067bb0 con 0x7ffbe006a730 2024-04-06T13:48:11.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.444+0000 7ffbdcff9700 1 -- 172.21.15.189:0/972245002 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7ffbd4013c80 con 0x7ffbe006a730 2024-04-06T13:48:11.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.444+0000 7ffbdcff9700 1 --2- 172.21.15.189:0/972245002 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ffbcc06aed0 0x7ffbcc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:11.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.444+0000 7ffbdcff9700 1 -- 172.21.15.189:0/972245002 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6152+0+0 (secure 0 0 0) 0x7ffbd4089710 con 0x7ffbe006a730 2024-04-06T13:48:11.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.444+0000 7ffbdf7fe700 1 --2- 172.21.15.189:0/972245002 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ffbcc06aed0 0x7ffbcc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:11.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.445+0000 7ffbdf7fe700 1 --2- 172.21.15.189:0/972245002 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ffbcc06aed0 0x7ffbcc06d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ffbe015de20 tx=0x7ffbd0007be0 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:11.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.449+0000 7ffbdcff9700 1 -- 172.21.15.189:0/972245002 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ffbd40542d0 con 0x7ffbe006a730 2024-04-06T13:48:11.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:11.724+0000 7ffbe6398700 1 -- 172.21.15.189:0/972245002 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7ffbe0074990 con 0x7ffbe006a730 2024-04-06T13:48:12.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.236+0000 7ffbdcff9700 1 -- 172.21.15.189:0/972245002 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3150) v1 ==== 186+0+0 (secure 0 0 0) 0x7ffbd40578f0 con 0x7ffbe006a730 2024-04-06T13:48:12.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.239+0000 7ffbe6398700 1 -- 172.21.15.189:0/972245002 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ffbcc06aed0 msgr2=0x7ffbcc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:12.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.239+0000 7ffbe6398700 1 --2- 172.21.15.189:0/972245002 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ffbcc06aed0 0x7ffbcc06d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ffbe015de20 tx=0x7ffbd0007be0 comp rx=0 tx=0).stop 2024-04-06T13:48:12.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.239+0000 7ffbe6398700 1 -- 172.21.15.189:0/972245002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbe006a730 msgr2=0x7ffbe0148710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:12.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.239+0000 7ffbe6398700 1 --2- 172.21.15.189:0/972245002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbe006a730 0x7ffbe0148710 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400e810 tx=0x7ffbd400ebd0 comp rx=0 tx=0).stop 2024-04-06T13:48:12.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.239+0000 7ffbe6398700 1 -- 172.21.15.189:0/972245002 shutdown_connections 2024-04-06T13:48:12.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.239+0000 7ffbe6398700 1 --2- 172.21.15.189:0/972245002 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ffbcc06aed0 0x7ffbcc06d390 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:12.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.239+0000 7ffbe6398700 1 --2- 172.21.15.189:0/972245002 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbe015e500 0x7ffbe01f2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:12.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.239+0000 7ffbe6398700 1 --2- 172.21.15.189:0/972245002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbe006a730 0x7ffbe0148710 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:12.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.239+0000 7ffbe6398700 1 --2- 172.21.15.189:0/972245002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbe015d7b0 0x7ffbe01f1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:12.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.239+0000 7ffbe6398700 1 -- 172.21.15.189:0/972245002 >> 172.21.15.189:0/972245002 conn(0x7ffbe00d00b0 msgr2=0x7ffbe00ad090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:12.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.239+0000 7ffbe6398700 1 -- 172.21.15.189:0/972245002 shutdown_connections 2024-04-06T13:48:12.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.239+0000 7ffbe6398700 1 -- 172.21.15.189:0/972245002 wait complete. 2024-04-06T13:48:12.254 INFO:tasks.cephfs_test_runner:Test command succes with --fs argument for non-default fs ... ok 2024-04-06T13:48:12.254 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command fails without --fs argument in presence of multiple fs 2024-04-06T13:48:12.254 INFO:tasks.cephfs_test_runner:test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2024-04-06T13:48:12.255 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2024-04-06T13:48:12.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.380+0000 7f0677b1c700 1 Processor -- start 2024-04-06T13:48:12.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.380+0000 7f0677b1c700 1 -- start start 2024-04-06T13:48:12.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.381+0000 7f0677b1c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f067015a550 0x7f06700cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:12.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.381+0000 7f0677b1c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f06700cf900 0x7f06700b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:12.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.381+0000 7f0677b1c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06700b4ba0 0x7f067006eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:12.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.381+0000 7f0677b1c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06700cdbe0 con 0x7f06700b4ba0 2024-04-06T13:48:12.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.381+0000 7f0677b1c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06700b5010 con 0x7f06700cf900 2024-04-06T13:48:12.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.381+0000 7f0677b1c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067015a940 con 0x7f067015a550 2024-04-06T13:48:12.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.381+0000 7f06760b9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06700b4ba0 0x7f067006eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:12.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.381+0000 7f06760b9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06700b4ba0 0x7f067006eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51546/0 (socket says 172.21.15.189:51546) 2024-04-06T13:48:12.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.381+0000 7f06760b9700 1 -- 172.21.15.189:0/2892493181 learned_addr learned my addr 172.21.15.189:0/2892493181 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:12.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.382+0000 7f06760b9700 1 -- 172.21.15.189:0/2892493181 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f067015a550 msgr2=0x7f06700cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:12.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.382+0000 7f06760b9700 1 --2- 172.21.15.189:0/2892493181 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f067015a550 0x7f06700cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:12.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.382+0000 7f06750b7700 1 --2- 172.21.15.189:0/2892493181 >> v2:172.21.15.192:3300/0 conn(0x7f06700cf900 0x7f06700b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:12.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.382+0000 7f06760b9700 1 -- 172.21.15.189:0/2892493181 >> v2:172.21.15.192:3300/0 conn(0x7f06700cf900 msgr2=0x7f06700b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:12.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.382+0000 7f06760b9700 1 --2- 172.21.15.189:0/2892493181 >> v2:172.21.15.192:3300/0 conn(0x7f06700cf900 0x7f06700b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:12.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.382+0000 7f06760b9700 1 -- 172.21.15.189:0/2892493181 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067015ac00 con 0x7f06700b4ba0 2024-04-06T13:48:12.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.382+0000 7f06760b9700 1 --2- 172.21.15.189:0/2892493181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06700b4ba0 0x7f067006eb50 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f067006cb90 tx=0x7f066800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e35c0ff7d51244 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:12.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.382+0000 7f06748b6700 1 -- 172.21.15.189:0/2892493181 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066800e070 con 0x7f06700b4ba0 2024-04-06T13:48:12.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.382+0000 7f06748b6700 1 -- 172.21.15.189:0/2892493181 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066800b9c0 con 0x7f06700b4ba0 2024-04-06T13:48:12.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.382+0000 7f06748b6700 1 -- 172.21.15.189:0/2892493181 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066800bd70 con 0x7f06700b4ba0 2024-04-06T13:48:12.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.382+0000 7f0677b1c700 1 -- 172.21.15.189:0/2892493181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06700b4ba0 msgr2=0x7f067006eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:12.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.382+0000 7f0677b1c700 1 --2- 172.21.15.189:0/2892493181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06700b4ba0 0x7f067006eb50 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f067006cb90 tx=0x7f066800b330 comp rx=0 tx=0).stop 2024-04-06T13:48:12.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.382+0000 7f0677b1c700 1 -- 172.21.15.189:0/2892493181 shutdown_connections 2024-04-06T13:48:12.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.382+0000 7f0677b1c700 1 --2- 172.21.15.189:0/2892493181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06700b4ba0 0x7f067006eb50 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:12.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.382+0000 7f0677b1c700 1 --2- 172.21.15.189:0/2892493181 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f067015a550 0x7f06700cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:12.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.382+0000 7f0677b1c700 1 --2- 172.21.15.189:0/2892493181 >> v2:172.21.15.192:3300/0 conn(0x7f06700cf900 0x7f06700b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:12.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.382+0000 7f0677b1c700 1 -- 172.21.15.189:0/2892493181 >> 172.21.15.189:0/2892493181 conn(0x7f06700d00b0 msgr2=0x7f06700d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:12.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.383+0000 7f0677b1c700 1 -- 172.21.15.189:0/2892493181 shutdown_connections 2024-04-06T13:48:12.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.383+0000 7f0677b1c700 1 -- 172.21.15.189:0/2892493181 wait complete. 2024-04-06T13:48:12.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.384+0000 7f0677b1c700 1 Processor -- start 2024-04-06T13:48:12.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.384+0000 7f0677b1c700 1 -- start start 2024-04-06T13:48:12.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.384+0000 7f0677b1c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06700b4ba0 0x7f067014d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:12.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.384+0000 7f0677b1c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06700cf900 0x7f0670150a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:12.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.384+0000 7f0677b1c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f067014ff10 0x7f0670150f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:12.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.384+0000 7f0677b1c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067015b810 con 0x7f06700cf900 2024-04-06T13:48:12.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.384+0000 7f0677b1c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067015ba60 con 0x7f067014ff10 2024-04-06T13:48:12.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.384+0000 7f0677b1c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067015bd40 con 0x7f06700b4ba0 2024-04-06T13:48:12.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.385+0000 7f06750b7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06700cf900 0x7f0670150a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:12.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.385+0000 7f06750b7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06700cf900 0x7f0670150a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51560/0 (socket says 172.21.15.189:51560) 2024-04-06T13:48:12.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.385+0000 7f06750b7700 1 -- 172.21.15.189:0/999075793 learned_addr learned my addr 172.21.15.189:0/999075793 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:12.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.385+0000 7f06750b7700 1 -- 172.21.15.189:0/999075793 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06700b4ba0 msgr2=0x7f067014d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:12.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.385+0000 7f06750b7700 1 --2- 172.21.15.189:0/999075793 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06700b4ba0 0x7f067014d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:12.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.385+0000 7f06750b7700 1 -- 172.21.15.189:0/999075793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f067014ff10 msgr2=0x7f0670150f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:12.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.385+0000 7f06750b7700 1 --2- 172.21.15.189:0/999075793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f067014ff10 0x7f0670150f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:12.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.385+0000 7f06750b7700 1 -- 172.21.15.189:0/999075793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0668009b30 con 0x7f06700cf900 2024-04-06T13:48:12.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.385+0000 7f06750b7700 1 --2- 172.21.15.189:0/999075793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06700cf900 0x7f0670150a20 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7f066c00eb40 tx=0x7f066c00ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:12.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.385+0000 7f06667fc700 1 -- 172.21.15.189:0/999075793 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c004220 con 0x7f06700cf900 2024-04-06T13:48:12.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.385+0000 7f06667fc700 1 -- 172.21.15.189:0/999075793 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066c0043a0 con 0x7f06700cf900 2024-04-06T13:48:12.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.385+0000 7f0677b1c700 1 -- 172.21.15.189:0/999075793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f067015c020 con 0x7f06700cf900 2024-04-06T13:48:12.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.385+0000 7f06667fc700 1 -- 172.21.15.189:0/999075793 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c007b30 con 0x7f06700cf900 2024-04-06T13:48:12.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.385+0000 7f0677b1c700 1 -- 172.21.15.189:0/999075793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f067014d9a0 con 0x7f06700cf900 2024-04-06T13:48:12.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.386+0000 7f0677b1c700 1 -- 172.21.15.189:0/999075793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f067015c540 con 0x7f06700cf900 2024-04-06T13:48:12.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.386+0000 7f06667fc700 1 -- 172.21.15.189:0/999075793 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f066c007c90 con 0x7f06700cf900 2024-04-06T13:48:12.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.387+0000 7f06667fc700 1 --2- 172.21.15.189:0/999075793 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f065406aed0 0x7f065406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:12.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.387+0000 7f06667fc700 1 -- 172.21.15.189:0/999075793 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f066c014070 con 0x7f06700cf900 2024-04-06T13:48:12.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.387+0000 7f06758b8700 1 --2- 172.21.15.189:0/999075793 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f065406aed0 0x7f065406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:12.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.387+0000 7f06758b8700 1 --2- 172.21.15.189:0/999075793 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f065406aed0 0x7f065406d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f06600097b0 tx=0x7f0660006d20 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:12.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.391+0000 7f06667fc700 1 -- 172.21.15.189:0/999075793 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f067015c540 con 0x7f06700cf900 2024-04-06T13:48:12.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:12.668+0000 7f0677b1c700 1 -- 172.21.15.189:0/999075793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7f067015c540 con 0x7f06700cf900 2024-04-06T13:48:13.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.238+0000 7f06667fc700 1 -- 172.21.15.189:0/999075793 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3151) v1 ==== 190+0+0 (secure 0 0 0) 0x7f067015c540 con 0x7f06700cf900 2024-04-06T13:48:13.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.241+0000 7f0677b1c700 1 -- 172.21.15.189:0/999075793 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f065406aed0 msgr2=0x7f065406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:13.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.241+0000 7f0677b1c700 1 --2- 172.21.15.189:0/999075793 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f065406aed0 0x7f065406d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f06600097b0 tx=0x7f0660006d20 comp rx=0 tx=0).stop 2024-04-06T13:48:13.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.241+0000 7f0677b1c700 1 -- 172.21.15.189:0/999075793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06700cf900 msgr2=0x7f0670150a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:13.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.241+0000 7f0677b1c700 1 --2- 172.21.15.189:0/999075793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06700cf900 0x7f0670150a20 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7f066c00eb40 tx=0x7f066c00ef00 comp rx=0 tx=0).stop 2024-04-06T13:48:13.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.241+0000 7f0677b1c700 1 -- 172.21.15.189:0/999075793 shutdown_connections 2024-04-06T13:48:13.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.241+0000 7f0677b1c700 1 --2- 172.21.15.189:0/999075793 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f065406aed0 0x7f065406d390 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:13.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.241+0000 7f0677b1c700 1 --2- 172.21.15.189:0/999075793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f067014ff10 0x7f0670150f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:13.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.241+0000 7f0677b1c700 1 --2- 172.21.15.189:0/999075793 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06700b4ba0 0x7f067014d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:13.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.241+0000 7f0677b1c700 1 --2- 172.21.15.189:0/999075793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06700cf900 0x7f0670150a20 unknown :-1 s=CLOSED pgs=2637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:13.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.241+0000 7f0677b1c700 1 -- 172.21.15.189:0/999075793 >> 172.21.15.189:0/999075793 conn(0x7f06700d00b0 msgr2=0x7f06701579b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:13.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.241+0000 7f0677b1c700 1 -- 172.21.15.189:0/999075793 shutdown_connections 2024-04-06T13:48:13.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.241+0000 7f0677b1c700 1 -- 172.21.15.189:0/999075793 wait complete. 2024-04-06T13:48:13.255 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-06T13:48:13.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.380+0000 7fddd45a9700 1 Processor -- start 2024-04-06T13:48:13.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.380+0000 7fddd45a9700 1 -- start start 2024-04-06T13:48:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.381+0000 7fddd45a9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fddcc0cea80 0x7fddcc0cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.381+0000 7fddd45a9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddcc0cdae0 0x7fddcc06ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.381+0000 7fddd45a9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddcc15a830 con 0x7fddcc0aec70 2024-04-06T13:48:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.381+0000 7fddd45a9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddcc0cd920 con 0x7fddcc0cea80 2024-04-06T13:48:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.381+0000 7fddd45a9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc0cc050 con 0x7fddcc0cdae0 2024-04-06T13:48:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.381+0000 7fddd1b44700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fddcc0cea80 0x7fddcc0cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.381+0000 7fddd1b44700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fddcc0cea80 0x7fddcc0cd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51566/0 (socket says 172.21.15.189:51566) 2024-04-06T13:48:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.381+0000 7fddd1b44700 1 -- 172.21.15.189:0/857930037 learned_addr learned my addr 172.21.15.189:0/857930037 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.381+0000 7fddd2b46700 1 --2- 172.21.15.189:0/857930037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddcc0cdae0 0x7fddcc06ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.381+0000 7fddd1b44700 1 -- 172.21.15.189:0/857930037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddcc0cdae0 msgr2=0x7fddcc06ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.381+0000 7fddd1b44700 1 --2- 172.21.15.189:0/857930037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddcc0cdae0 0x7fddcc06ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.381+0000 7fddd1b44700 1 -- 172.21.15.189:0/857930037 >> v1:172.21.15.192:6789/0 conn(0x7fddcc0aec70 legacy=0x7fddcc0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.381+0000 7fddd1b44700 1 -- 172.21.15.189:0/857930037 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddcc0666f0 con 0x7fddcc0cea80 2024-04-06T13:48:13.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.381+0000 7fddd2b46700 1 --2- 172.21.15.189:0/857930037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddcc0cdae0 0x7fddcc06ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:13.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.381+0000 7fddd1b44700 1 --2- 172.21.15.189:0/857930037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddcc0cea80 0x7fddcc0cd3e0 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7fddbc0098f0 tx=0x7fddbc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8af63c8a2c68454 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:13.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.382+0000 7fddd1343700 1 -- 172.21.15.189:0/857930037 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddbc004230 con 0x7fddcc0cea80 2024-04-06T13:48:13.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.382+0000 7fddd1343700 1 -- 172.21.15.189:0/857930037 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddbc004390 con 0x7fddcc0cea80 2024-04-06T13:48:13.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.382+0000 7fddd1343700 1 -- 172.21.15.189:0/857930037 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddbc0036a0 con 0x7fddcc0cea80 2024-04-06T13:48:13.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.382+0000 7fddd45a9700 1 -- 172.21.15.189:0/857930037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddcc0cea80 msgr2=0x7fddcc0cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:13.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.382+0000 7fddd45a9700 1 --2- 172.21.15.189:0/857930037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddcc0cea80 0x7fddcc0cd3e0 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7fddbc0098f0 tx=0x7fddbc009c00 comp rx=0 tx=0).stop 2024-04-06T13:48:13.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.382+0000 7fddd45a9700 1 -- 172.21.15.189:0/857930037 shutdown_connections 2024-04-06T13:48:13.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.382+0000 7fddd45a9700 1 --2- 172.21.15.189:0/857930037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddcc0cdae0 0x7fddcc06ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:13.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.382+0000 7fddd45a9700 1 --2- 172.21.15.189:0/857930037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddcc0cea80 0x7fddcc0cd3e0 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:13.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.382+0000 7fddd45a9700 1 -- 172.21.15.189:0/857930037 >> 172.21.15.189:0/857930037 conn(0x7fddcc0cc290 msgr2=0x7fddcc0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:13.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.382+0000 7fddd45a9700 1 -- 172.21.15.189:0/857930037 shutdown_connections 2024-04-06T13:48:13.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.382+0000 7fddd45a9700 1 -- 172.21.15.189:0/857930037 wait complete. 2024-04-06T13:48:13.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.383+0000 7fddd45a9700 1 Processor -- start 2024-04-06T13:48:13.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.384+0000 7fddd45a9700 1 -- start start 2024-04-06T13:48:13.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.384+0000 7fddd45a9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddcc0aec70 0x7fddcc1572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:13.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.384+0000 7fddd45a9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddcc0cdae0 0x7fddcc153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:13.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.384+0000 7fddd45a9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddcc0cea80 0x7fddcc0c5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:13.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.384+0000 7fddd45a9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc156020 con 0x7fddcc0cea80 2024-04-06T13:48:13.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.384+0000 7fddd45a9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc157810 con 0x7fddcc0cdae0 2024-04-06T13:48:13.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.384+0000 7fddd45a9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc156230 con 0x7fddcc0aec70 2024-04-06T13:48:13.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.384+0000 7fddd2b46700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddcc0cea80 0x7fddcc0c5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:13.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.384+0000 7fddd2b46700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddcc0cea80 0x7fddcc0c5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51582/0 (socket says 172.21.15.189:51582) 2024-04-06T13:48:13.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.384+0000 7fddd2b46700 1 -- 172.21.15.189:0/529717216 learned_addr learned my addr 172.21.15.189:0/529717216 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:13.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.385+0000 7fddd1b44700 1 --2- 172.21.15.189:0/529717216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddcc0cdae0 0x7fddcc153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:13.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.385+0000 7fddd2345700 1 --2- 172.21.15.189:0/529717216 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddcc0aec70 0x7fddcc1572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:13.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.385+0000 7fddd2b46700 1 -- 172.21.15.189:0/529717216 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddcc0aec70 msgr2=0x7fddcc1572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:13.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.385+0000 7fddd2b46700 1 --2- 172.21.15.189:0/529717216 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddcc0aec70 0x7fddcc1572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:13.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.385+0000 7fddd2b46700 1 -- 172.21.15.189:0/529717216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddcc0cdae0 msgr2=0x7fddcc153c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:13.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.385+0000 7fddd2b46700 1 --2- 172.21.15.189:0/529717216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddcc0cdae0 0x7fddcc153c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:13.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.385+0000 7fddd2b46700 1 -- 172.21.15.189:0/529717216 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddbc009550 con 0x7fddcc0cea80 2024-04-06T13:48:13.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.385+0000 7fddd2b46700 1 --2- 172.21.15.189:0/529717216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddcc0cea80 0x7fddcc0c5bd0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7fddc400cef0 tx=0x7fddc400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:13.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.385+0000 7fddc2ffd700 1 -- 172.21.15.189:0/529717216 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc400ae50 con 0x7fddcc0cea80 2024-04-06T13:48:13.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.385+0000 7fddc2ffd700 1 -- 172.21.15.189:0/529717216 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc4015440 con 0x7fddcc0cea80 2024-04-06T13:48:13.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.385+0000 7fddc2ffd700 1 -- 172.21.15.189:0/529717216 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc40157f0 con 0x7fddcc0cea80 2024-04-06T13:48:13.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.385+0000 7fddd45a9700 1 -- 172.21.15.189:0/529717216 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddcc156510 con 0x7fddcc0cea80 2024-04-06T13:48:13.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.385+0000 7fddd45a9700 1 -- 172.21.15.189:0/529717216 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddcc156db0 con 0x7fddcc0cea80 2024-04-06T13:48:13.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.386+0000 7fddd45a9700 1 -- 172.21.15.189:0/529717216 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddcc074990 con 0x7fddcc0cea80 2024-04-06T13:48:13.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.387+0000 7fddc2ffd700 1 -- 172.21.15.189:0/529717216 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fddc400d430 con 0x7fddcc0cea80 2024-04-06T13:48:13.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.387+0000 7fddc2ffd700 1 --2- 172.21.15.189:0/529717216 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fddb006aed0 0x7fddb006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:13.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.387+0000 7fddc2ffd700 1 -- 172.21.15.189:0/529717216 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fddc4089f60 con 0x7fddcc0cea80 2024-04-06T13:48:13.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.387+0000 7fddd2345700 1 --2- 172.21.15.189:0/529717216 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fddb006aed0 0x7fddb006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:13.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.387+0000 7fddd2345700 1 --2- 172.21.15.189:0/529717216 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fddb006aed0 0x7fddb006d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fddc8004f30 tx=0x7fddc8009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:13.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.391+0000 7fddc2ffd700 1 -- 172.21.15.189:0/529717216 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fddc4054c20 con 0x7fddcc0cea80 2024-04-06T13:48:13.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.584+0000 7fddd45a9700 1 -- 172.21.15.189:0/529717216 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fddcc0b5790 con 0x7fddcc0cea80 2024-04-06T13:48:13.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.588+0000 7fddc2ffd700 1 -- 172.21.15.189:0/529717216 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v82)=0 v82) v1 ==== 122+0+0 (secure 0 0 0) 0x7fddc401a020 con 0x7fddcc0cea80 2024-04-06T13:48:13.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.590+0000 7fddd45a9700 1 -- 172.21.15.189:0/529717216 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fddb006aed0 msgr2=0x7fddb006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:13.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.590+0000 7fddd45a9700 1 --2- 172.21.15.189:0/529717216 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fddb006aed0 0x7fddb006d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fddc8004f30 tx=0x7fddc8009040 comp rx=0 tx=0).stop 2024-04-06T13:48:13.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.590+0000 7fddd45a9700 1 -- 172.21.15.189:0/529717216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddcc0cea80 msgr2=0x7fddcc0c5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:13.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.590+0000 7fddd45a9700 1 --2- 172.21.15.189:0/529717216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddcc0cea80 0x7fddcc0c5bd0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7fddc400cef0 tx=0x7fddc400a340 comp rx=0 tx=0).stop 2024-04-06T13:48:13.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.590+0000 7fddd45a9700 1 -- 172.21.15.189:0/529717216 shutdown_connections 2024-04-06T13:48:13.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.590+0000 7fddd45a9700 1 --2- 172.21.15.189:0/529717216 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fddb006aed0 0x7fddb006d390 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:13.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.590+0000 7fddd45a9700 1 --2- 172.21.15.189:0/529717216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddcc0cea80 0x7fddcc0c5bd0 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:13.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.590+0000 7fddd45a9700 1 --2- 172.21.15.189:0/529717216 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddcc0aec70 0x7fddcc1572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:13.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.590+0000 7fddd45a9700 1 --2- 172.21.15.189:0/529717216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddcc0cdae0 0x7fddcc153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:13.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.590+0000 7fddd45a9700 1 -- 172.21.15.189:0/529717216 >> 172.21.15.189:0/529717216 conn(0x7fddcc0cc290 msgr2=0x7fddcc06ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:13.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.590+0000 7fddd45a9700 1 -- 172.21.15.189:0/529717216 shutdown_connections 2024-04-06T13:48:13.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.590+0000 7fddd45a9700 1 -- 172.21.15.189:0/529717216 wait complete. 2024-04-06T13:48:13.603 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T13:48:13.604 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:48:13.654 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:48:13.699 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T13:48:13.699 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:48:13.699 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:48:13.717 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:48:13.717 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:48:13.718 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:48:13.759 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:48:13.759 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:48:13.759 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:48:13.778 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:48:13.778 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:48:13.778 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:48:13.820 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:48:13.820 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-06T13:48:13.820 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:48:13.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.943+0000 7f707c834700 1 Processor -- start 2024-04-06T13:48:13.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.943+0000 7f707c834700 1 -- start start 2024-04-06T13:48:13.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.944+0000 7f707c834700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f707815d590 0x7f70780b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:13.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.944+0000 7f707c834700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f70780b4ed0 0x7f70780b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:13.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.944+0000 7f707c834700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70780b5cc0 con 0x7f70780b6580 2024-04-06T13:48:13.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.944+0000 7f707c834700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70780b5890 con 0x7f70780b4ed0 2024-04-06T13:48:13.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.944+0000 7f707c834700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70780d1500 con 0x7f707815d590 2024-04-06T13:48:13.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.944+0000 7f7075d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f70780b4ed0 0x7f70780b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:13.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.944+0000 7f7075d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f70780b4ed0 0x7f70780b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51596/0 (socket says 172.21.15.189:51596) 2024-04-06T13:48:13.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.944+0000 7f7075d9b700 1 -- 172.21.15.189:0/212890219 learned_addr learned my addr 172.21.15.189:0/212890219 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:13.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.944+0000 7f7075d9b700 1 -- 172.21.15.189:0/212890219 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f707815d590 msgr2=0x7f70780b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:13.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.944+0000 7f7075d9b700 1 --2- 172.21.15.189:0/212890219 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f707815d590 0x7f70780b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:13.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.944+0000 7f7075d9b700 1 -- 172.21.15.189:0/212890219 >> v1:172.21.15.192:6789/0 conn(0x7f70780b6580 legacy=0x7f70780b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:13.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.944+0000 7f7075d9b700 1 -- 172.21.15.189:0/212890219 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70780d17e0 con 0x7f70780b4ed0 2024-04-06T13:48:13.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.944+0000 7f7075d9b700 1 --2- 172.21.15.189:0/212890219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70780b4ed0 0x7f70780b5350 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7f706c009920 tx=0x7f706c009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c3b7499b60f41e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:13.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.944+0000 7f707559a700 1 -- 172.21.15.189:0/212890219 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c004230 con 0x7f70780b4ed0 2024-04-06T13:48:13.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.944+0000 7f707559a700 1 -- 172.21.15.189:0/212890219 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706c004390 con 0x7f70780b4ed0 2024-04-06T13:48:13.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.944+0000 7f707559a700 1 -- 172.21.15.189:0/212890219 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c009e70 con 0x7f70780b4ed0 2024-04-06T13:48:13.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.945+0000 7f707c834700 1 -- 172.21.15.189:0/212890219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70780b4ed0 msgr2=0x7f70780b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:13.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.945+0000 7f707c834700 1 --2- 172.21.15.189:0/212890219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70780b4ed0 0x7f70780b5350 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7f706c009920 tx=0x7f706c009c30 comp rx=0 tx=0).stop 2024-04-06T13:48:13.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.945+0000 7f707c834700 1 -- 172.21.15.189:0/212890219 shutdown_connections 2024-04-06T13:48:13.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.945+0000 7f707c834700 1 --2- 172.21.15.189:0/212890219 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f707815d590 0x7f70780b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:13.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.945+0000 7f707c834700 1 --2- 172.21.15.189:0/212890219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70780b4ed0 0x7f70780b5350 unknown :-1 s=CLOSED pgs=2640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:13.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.945+0000 7f707c834700 1 -- 172.21.15.189:0/212890219 >> 172.21.15.189:0/212890219 conn(0x7f707806bbc0 msgr2=0x7f707806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:13.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.945+0000 7f707c834700 1 -- 172.21.15.189:0/212890219 shutdown_connections 2024-04-06T13:48:13.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.945+0000 7f707c834700 1 -- 172.21.15.189:0/212890219 wait complete. 2024-04-06T13:48:13.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.946+0000 7f707c834700 1 Processor -- start 2024-04-06T13:48:13.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.946+0000 7f707c834700 1 -- start start 2024-04-06T13:48:13.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.947+0000 7f707c834700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70780b4ed0 0x7f70780cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:13.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.947+0000 7f707c834700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70780b6580 0x7f7078148740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:13.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.947+0000 7f707c834700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f707815d590 0x7f7078158610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:13.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.947+0000 7f707c834700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70780cbab0 con 0x7f70780b6580 2024-04-06T13:48:13.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.947+0000 7f707c834700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70780cbce0 con 0x7f70780b4ed0 2024-04-06T13:48:13.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.947+0000 7f707c834700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70780cbfc0 con 0x7f707815d590 2024-04-06T13:48:13.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.947+0000 7f7075d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70780b6580 0x7f7078148740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:13.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.947+0000 7f7075d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70780b6580 0x7f7078148740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51598/0 (socket says 172.21.15.189:51598) 2024-04-06T13:48:13.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.947+0000 7f7075d9b700 1 -- 172.21.15.189:0/3643843812 learned_addr learned my addr 172.21.15.189:0/3643843812 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:13.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.948+0000 7f7076d9d700 1 --2- 172.21.15.189:0/3643843812 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f707815d590 0x7f7078158610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:13.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.948+0000 7f707659c700 1 --2- 172.21.15.189:0/3643843812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70780b4ed0 0x7f70780cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:13.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.948+0000 7f7075d9b700 1 -- 172.21.15.189:0/3643843812 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f707815d590 msgr2=0x7f7078158610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:13.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.948+0000 7f7075d9b700 1 --2- 172.21.15.189:0/3643843812 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f707815d590 0x7f7078158610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:13.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.948+0000 7f7075d9b700 1 -- 172.21.15.189:0/3643843812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70780b4ed0 msgr2=0x7f70780cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:13.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.948+0000 7f7075d9b700 1 --2- 172.21.15.189:0/3643843812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70780b4ed0 0x7f70780cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:13.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.948+0000 7f7075d9b700 1 -- 172.21.15.189:0/3643843812 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c009550 con 0x7f70780b6580 2024-04-06T13:48:13.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.948+0000 7f7076d9d700 1 --2- 172.21.15.189:0/3643843812 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f707815d590 0x7f7078158610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:13.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.948+0000 7f7075d9b700 1 --2- 172.21.15.189:0/3643843812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70780b6580 0x7f7078148740 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f70780d41b0 tx=0x7f706c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:13.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.948+0000 7f7066ffd700 1 -- 172.21.15.189:0/3643843812 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c00bd50 con 0x7f70780b6580 2024-04-06T13:48:13.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.948+0000 7f7066ffd700 1 -- 172.21.15.189:0/3643843812 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706c01f070 con 0x7f70780b6580 2024-04-06T13:48:13.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.948+0000 7f707c834700 1 -- 172.21.15.189:0/3643843812 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70780cc2a0 con 0x7f70780b6580 2024-04-06T13:48:13.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.948+0000 7f7066ffd700 1 -- 172.21.15.189:0/3643843812 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c0038e0 con 0x7f70780b6580 2024-04-06T13:48:13.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.948+0000 7f707c834700 1 -- 172.21.15.189:0/3643843812 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70780cc860 con 0x7f70780b6580 2024-04-06T13:48:13.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.949+0000 7f707c834700 1 -- 172.21.15.189:0/3643843812 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7078074990 con 0x7f70780b6580 2024-04-06T13:48:13.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.949+0000 7f7066ffd700 1 -- 172.21.15.189:0/3643843812 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f706c021030 con 0x7f70780b6580 2024-04-06T13:48:13.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.950+0000 7f7066ffd700 1 --2- 172.21.15.189:0/3643843812 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f705806aed0 0x7f705806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:13.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.950+0000 7f7066ffd700 1 -- 172.21.15.189:0/3643843812 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f706c08af80 con 0x7f70780b6580 2024-04-06T13:48:13.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.950+0000 7f707659c700 1 --2- 172.21.15.189:0/3643843812 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f705806aed0 0x7f705806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:13.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.950+0000 7f707659c700 1 --2- 172.21.15.189:0/3643843812 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f705806aed0 0x7f705806d390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7068006fd0 tx=0x7f7068008040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:13.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:13.954+0000 7f7066ffd700 1 -- 172.21.15.189:0/3643843812 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f706c0be650 con 0x7f70780b6580 2024-04-06T13:48:14.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.193+0000 7f707c834700 1 -- 172.21.15.189:0/3643843812 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7078074990 con 0x7f70780b6580 2024-04-06T13:48:14.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.194+0000 7f7066ffd700 1 -- 172.21.15.189:0/3643843812 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+3544 (secure 0 0 0) 0x7f706c055da0 con 0x7f70780b6580 2024-04-06T13:48:14.195 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:14.195 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":110,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8885,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1481817364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1481817364},{"type":"v1","addr":"172.21.15.189:6837","nonce":1481817364}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":10639,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/573594428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":573594428},{"type":"v1","addr":"172.21.15.192:6835","nonce":573594428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":108},{"gid":10720,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/2681738491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2681738491},{"type":"v1","addr":"172.21.15.192:6834","nonce":2681738491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":110},{"gid":11244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/4021875447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":4021875447},{"type":"v1","addr":"172.21.15.189:6836","nonce":4021875447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[]} 2024-04-06T13:48:14.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.196+0000 7f707c834700 1 -- 172.21.15.189:0/3643843812 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f705806aed0 msgr2=0x7f705806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:14.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.196+0000 7f707c834700 1 --2- 172.21.15.189:0/3643843812 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f705806aed0 0x7f705806d390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7068006fd0 tx=0x7f7068008040 comp rx=0 tx=0).stop 2024-04-06T13:48:14.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.196+0000 7f707c834700 1 -- 172.21.15.189:0/3643843812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70780b6580 msgr2=0x7f7078148740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:14.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.196+0000 7f707c834700 1 --2- 172.21.15.189:0/3643843812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70780b6580 0x7f7078148740 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f70780d41b0 tx=0x7f706c004750 comp rx=0 tx=0).stop 2024-04-06T13:48:14.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.196+0000 7f707c834700 1 -- 172.21.15.189:0/3643843812 shutdown_connections 2024-04-06T13:48:14.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.196+0000 7f707c834700 1 --2- 172.21.15.189:0/3643843812 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f705806aed0 0x7f705806d390 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:14.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.196+0000 7f707c834700 1 --2- 172.21.15.189:0/3643843812 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f707815d590 0x7f7078158610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:14.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.196+0000 7f707c834700 1 --2- 172.21.15.189:0/3643843812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70780b4ed0 0x7f70780cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:14.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.196+0000 7f707c834700 1 --2- 172.21.15.189:0/3643843812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70780b6580 0x7f7078148740 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:14.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.196+0000 7f707c834700 1 -- 172.21.15.189:0/3643843812 >> 172.21.15.189:0/3643843812 conn(0x7f707806bbc0 msgr2=0x7f70780d33a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:14.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.196+0000 7f707c834700 1 -- 172.21.15.189:0/3643843812 shutdown_connections 2024-04-06T13:48:14.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.196+0000 7f707c834700 1 -- 172.21.15.189:0/3643843812 wait complete. 2024-04-06T13:48:14.198 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 110 2024-04-06T13:48:14.210 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-06T13:48:14.210 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-06T13:48:14.211 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-06T13:48:14.211 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:48:14.211 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-06T13:48:14.211 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-06T13:48:14.211 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-06T13:48:14.211 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:48:14.211 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-06T13:48:14.212 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-06T13:48:14.212 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-06T13:48:14.212 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:48:14.212 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-06T13:48:14.212 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-06T13:48:14.213 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-06T13:48:14.213 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:48:14.219 INFO:tasks.ceph.mds.a:Stopped 2024-04-06T13:48:14.220 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-06T13:48:14.221 INFO:tasks.ceph.mds.c:Stopped 2024-04-06T13:48:14.221 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-06T13:48:14.226 INFO:tasks.ceph.mds.b:Stopped 2024-04-06T13:48:14.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-06T13:48:14.227 INFO:tasks.ceph.mds.d:Stopped 2024-04-06T13:48:14.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-06T13:48:14.262 INFO:tasks.ceph.mds.a:Started 2024-04-06T13:48:14.263 DEBUG:teuthology.parallel:result is None 2024-04-06T13:48:14.263 INFO:tasks.ceph.mds.c:Started 2024-04-06T13:48:14.263 DEBUG:teuthology.parallel:result is None 2024-04-06T13:48:14.268 INFO:tasks.ceph.mds.b:Started 2024-04-06T13:48:14.268 DEBUG:teuthology.parallel:result is None 2024-04-06T13:48:14.269 INFO:tasks.ceph.mds.d:Started 2024-04-06T13:48:14.269 DEBUG:teuthology.parallel:result is None 2024-04-06T13:48:14.269 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-06T13:48:14.342 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2024-04-06T13:48:14.342 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T13:48:14.359 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2024-04-06T13:48:14.360 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T13:48:14.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.402+0000 7fd2b1e64700 1 Processor -- start 2024-04-06T13:48:14.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.402+0000 7fd2b1e64700 1 -- start start 2024-04-06T13:48:14.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.402+0000 7fd2b1e64700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2ac15b130 0x7fd2ac0b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.402+0000 7fd2b1e64700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd2ac15b510 0x7fd2ac0b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.402+0000 7fd2b1e64700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd2ac071de0 0x7fd2ac072250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.402+0000 7fd2b1e64700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ac0b1fb0 con 0x7fd2ac15b510 2024-04-06T13:48:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.402+0000 7fd2b1e64700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ac0b2200 con 0x7fd2ac071de0 2024-04-06T13:48:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.402+0000 7fd2b1e64700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac0b24e0 con 0x7fd2ac15b130 2024-04-06T13:48:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.403+0000 7fd2aaffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd2ac15b510 0x7fd2ac0b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.403+0000 7fd2aaffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd2ac15b510 0x7fd2ac0b39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51630/0 (socket says 172.21.15.189:51630) 2024-04-06T13:48:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.403+0000 7fd2aaffd700 1 -- 172.21.15.189:0/2716973512 learned_addr learned my addr 172.21.15.189:0/2716973512 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.403+0000 7fd2ab7fe700 1 --2- 172.21.15.189:0/2716973512 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2ac15b130 0x7fd2ac0b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.403+0000 7fd2abfff700 1 --2- 172.21.15.189:0/2716973512 >> v2:172.21.15.192:3300/0 conn(0x7fd2ac071de0 0x7fd2ac072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.403+0000 7fd2aaffd700 1 -- 172.21.15.189:0/2716973512 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2ac15b130 msgr2=0x7fd2ac0b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.403+0000 7fd2aaffd700 1 --2- 172.21.15.189:0/2716973512 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2ac15b130 0x7fd2ac0b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.403+0000 7fd2aaffd700 1 -- 172.21.15.189:0/2716973512 >> v2:172.21.15.192:3300/0 conn(0x7fd2ac071de0 msgr2=0x7fd2ac072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:14.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.403+0000 7fd2aaffd700 1 --2- 172.21.15.189:0/2716973512 >> v2:172.21.15.192:3300/0 conn(0x7fd2ac071de0 0x7fd2ac072250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.403+0000 7fd2aaffd700 1 -- 172.21.15.189:0/2716973512 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ac0b27c0 con 0x7fd2ac15b510 2024-04-06T13:48:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.403+0000 7fd2abfff700 1 --2- 172.21.15.189:0/2716973512 >> v2:172.21.15.192:3300/0 conn(0x7fd2ac071de0 0x7fd2ac072250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.403+0000 7fd2aaffd700 1 --2- 172.21.15.189:0/2716973512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2ac15b510 0x7fd2ac0b39b0 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac070680 tx=0x7fd2a0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=757f2601873581b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.403+0000 7fd2aa7fc700 1 -- 172.21.15.189:0/2716973512 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a000b870 con 0x7fd2ac15b510 2024-04-06T13:48:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.403+0000 7fd2aa7fc700 1 -- 172.21.15.189:0/2716973512 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a000b9d0 con 0x7fd2ac15b510 2024-04-06T13:48:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.403+0000 7fd2aa7fc700 1 -- 172.21.15.189:0/2716973512 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a000bd60 con 0x7fd2ac15b510 2024-04-06T13:48:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.404+0000 7fd2b1e64700 1 -- 172.21.15.189:0/2716973512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2ac15b510 msgr2=0x7fd2ac0b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.404+0000 7fd2b1e64700 1 --2- 172.21.15.189:0/2716973512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2ac15b510 0x7fd2ac0b39b0 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac070680 tx=0x7fd2a0009bd0 comp rx=0 tx=0).stop 2024-04-06T13:48:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.404+0000 7fd2b1e64700 1 -- 172.21.15.189:0/2716973512 shutdown_connections 2024-04-06T13:48:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.404+0000 7fd2b1e64700 1 --2- 172.21.15.189:0/2716973512 >> v2:172.21.15.192:3300/0 conn(0x7fd2ac071de0 0x7fd2ac072250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.404+0000 7fd2b1e64700 1 --2- 172.21.15.189:0/2716973512 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2ac15b130 0x7fd2ac0b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.404+0000 7fd2b1e64700 1 --2- 172.21.15.189:0/2716973512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2ac15b510 0x7fd2ac0b39b0 unknown :-1 s=CLOSED pgs=2646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.404+0000 7fd2b1e64700 1 -- 172.21.15.189:0/2716973512 >> 172.21.15.189:0/2716973512 conn(0x7fd2ac06a730 msgr2=0x7fd2ac06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.404+0000 7fd2b1e64700 1 -- 172.21.15.189:0/2716973512 shutdown_connections 2024-04-06T13:48:14.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.404+0000 7fd2b1e64700 1 -- 172.21.15.189:0/2716973512 wait complete. 2024-04-06T13:48:14.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.405+0000 7fd2b1e64700 1 Processor -- start 2024-04-06T13:48:14.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.405+0000 7fd2b1e64700 1 -- start start 2024-04-06T13:48:14.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.406+0000 7fd2b1e64700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2ac071de0 0x7fd2ac14f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:14.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.406+0000 7fd2b1e64700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2ac15b130 0x7fd2ac0cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:14.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.406+0000 7fd2b1e64700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2ac15b510 0x7fd2ac15c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:14.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.406+0000 7fd2b1e64700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac14f6f0 con 0x7fd2ac15b130 2024-04-06T13:48:14.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.406+0000 7fd2b1e64700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac0ceea0 con 0x7fd2ac15b510 2024-04-06T13:48:14.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.406+0000 7fd2b1e64700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac0cf180 con 0x7fd2ac071de0 2024-04-06T13:48:14.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.406+0000 7fd2aaffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2ac15b130 0x7fd2ac0cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:14.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.406+0000 7fd2aaffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2ac15b130 0x7fd2ac0cd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51636/0 (socket says 172.21.15.189:51636) 2024-04-06T13:48:14.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.406+0000 7fd2aaffd700 1 -- 172.21.15.189:0/1804843666 learned_addr learned my addr 172.21.15.189:0/1804843666 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:14.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.406+0000 7fd2ab7fe700 1 --2- 172.21.15.189:0/1804843666 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2ac071de0 0x7fd2ac14f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:14.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.406+0000 7fd2abfff700 1 --2- 172.21.15.189:0/1804843666 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2ac15b510 0x7fd2ac15c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:14.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.406+0000 7fd2aaffd700 1 -- 172.21.15.189:0/1804843666 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2ac071de0 msgr2=0x7fd2ac14f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:14.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.406+0000 7fd2aaffd700 1 --2- 172.21.15.189:0/1804843666 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2ac071de0 0x7fd2ac14f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:14.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.406+0000 7fd2aaffd700 1 -- 172.21.15.189:0/1804843666 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2ac15b510 msgr2=0x7fd2ac15c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:14.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.406+0000 7fd2aaffd700 1 --2- 172.21.15.189:0/1804843666 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2ac15b510 0x7fd2ac15c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:14.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.406+0000 7fd2aaffd700 1 -- 172.21.15.189:0/1804843666 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a0009550 con 0x7fd2ac15b130 2024-04-06T13:48:14.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.406+0000 7fd2aaffd700 1 --2- 172.21.15.189:0/1804843666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2ac15b130 0x7fd2ac0cd2f0 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0000c00 tx=0x7fd2a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:14.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.406+0000 7fd293fff700 1 -- 172.21.15.189:0/1804843666 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a0004480 con 0x7fd2ac15b130 2024-04-06T13:48:14.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.407+0000 7fd293fff700 1 -- 172.21.15.189:0/1804843666 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a00045e0 con 0x7fd2ac15b130 2024-04-06T13:48:14.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.407+0000 7fd2b1e64700 1 -- 172.21.15.189:0/1804843666 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ac0cf460 con 0x7fd2ac15b130 2024-04-06T13:48:14.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.407+0000 7fd293fff700 1 -- 172.21.15.189:0/1804843666 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a00038e0 con 0x7fd2ac15b130 2024-04-06T13:48:14.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.407+0000 7fd2b1e64700 1 -- 172.21.15.189:0/1804843666 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ac154f30 con 0x7fd2ac15b130 2024-04-06T13:48:14.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.408+0000 7fd2b1e64700 1 -- 172.21.15.189:0/1804843666 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ac073270 con 0x7fd2ac15b130 2024-04-06T13:48:14.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.408+0000 7fd293fff700 1 -- 172.21.15.189:0/1804843666 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fd2a0021070 con 0x7fd2ac15b130 2024-04-06T13:48:14.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.408+0000 7fd293fff700 1 --2- 172.21.15.189:0/1804843666 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd29806aed0 0x7fd29806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:14.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.408+0000 7fd293fff700 1 -- 172.21.15.189:0/1804843666 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fd2a008ade0 con 0x7fd2ac15b130 2024-04-06T13:48:14.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.408+0000 7fd2ab7fe700 1 --2- 172.21.15.189:0/1804843666 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd29806aed0 0x7fd29806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:14.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.409+0000 7fd2ab7fe700 1 --2- 172.21.15.189:0/1804843666 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd29806aed0 0x7fd29806d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0b1c40 tx=0x7fd29c008040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:14.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.413+0000 7fd293fff700 1 -- 172.21.15.189:0/1804843666 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd2a0055c50 con 0x7fd2ac15b130 2024-04-06T13:48:14.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:14.626+0000 7fd2b1e64700 1 -- 172.21.15.189:0/1804843666 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd2ac074990 con 0x7fd2ac15b130 2024-04-06T13:48:15.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.596+0000 7fd293fff700 1 -- 172.21.15.189:0/1804843666 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v244) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd2a0015020 con 0x7fd2ac15b130 2024-04-06T13:48:15.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.598+0000 7fd2b1e64700 1 -- 172.21.15.189:0/1804843666 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd29806aed0 msgr2=0x7fd29806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:15.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.598+0000 7fd2b1e64700 1 --2- 172.21.15.189:0/1804843666 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd29806aed0 0x7fd29806d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0b1c40 tx=0x7fd29c008040 comp rx=0 tx=0).stop 2024-04-06T13:48:15.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.598+0000 7fd2b1e64700 1 -- 172.21.15.189:0/1804843666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2ac15b130 msgr2=0x7fd2ac0cd2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:15.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.598+0000 7fd2b1e64700 1 --2- 172.21.15.189:0/1804843666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2ac15b130 0x7fd2ac0cd2f0 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0000c00 tx=0x7fd2a0004750 comp rx=0 tx=0).stop 2024-04-06T13:48:15.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.598+0000 7fd2b1e64700 1 -- 172.21.15.189:0/1804843666 shutdown_connections 2024-04-06T13:48:15.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.598+0000 7fd2b1e64700 1 --2- 172.21.15.189:0/1804843666 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd29806aed0 0x7fd29806d390 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:15.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.599+0000 7fd2b1e64700 1 --2- 172.21.15.189:0/1804843666 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2ac15b510 0x7fd2ac15c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:15.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.599+0000 7fd2b1e64700 1 --2- 172.21.15.189:0/1804843666 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2ac071de0 0x7fd2ac14f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:15.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.599+0000 7fd2b1e64700 1 --2- 172.21.15.189:0/1804843666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2ac15b130 0x7fd2ac0cd2f0 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:15.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.599+0000 7fd2b1e64700 1 -- 172.21.15.189:0/1804843666 >> 172.21.15.189:0/1804843666 conn(0x7fd2ac06a730 msgr2=0x7fd2ac070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:15.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.599+0000 7fd2b1e64700 1 -- 172.21.15.189:0/1804843666 shutdown_connections 2024-04-06T13:48:15.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.599+0000 7fd2b1e64700 1 -- 172.21.15.189:0/1804843666 wait complete. 2024-04-06T13:48:15.600 INFO:teuthology.orchestra.run.smithi189.stderr: removed all blocklist entries 2024-04-06T13:48:15.613 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-06T13:48:15.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.744+0000 7fcd3584c700 1 Processor -- start 2024-04-06T13:48:15.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.744+0000 7fcd3584c700 1 -- start start 2024-04-06T13:48:15.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.745+0000 7fcd3584c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd30144f40 0x7fcd3006a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:15.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.745+0000 7fcd3584c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcd3015c150 0x7fcd3006ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:15.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.745+0000 7fcd3584c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd3015c740 con 0x7fcd3015fcd0 2024-04-06T13:48:15.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.745+0000 7fcd3584c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd3015c560 con 0x7fcd3015c150 2024-04-06T13:48:15.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.745+0000 7fcd3584c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd300cfcb0 con 0x7fcd30144f40 2024-04-06T13:48:15.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.745+0000 7fcd2f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcd3015c150 0x7fcd3006ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:15.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.745+0000 7fcd2f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcd3015c150 0x7fcd3006ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51640/0 (socket says 172.21.15.189:51640) 2024-04-06T13:48:15.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.745+0000 7fcd2f7fe700 1 -- 172.21.15.189:0/195343466 learned_addr learned my addr 172.21.15.189:0/195343466 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:15.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.745+0000 7fcd2effd700 1 --2- 172.21.15.189:0/195343466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd30144f40 0x7fcd3006a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:15.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.746+0000 7fcd2f7fe700 1 -- 172.21.15.189:0/195343466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd30144f40 msgr2=0x7fcd3006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:15.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.746+0000 7fcd2f7fe700 1 --2- 172.21.15.189:0/195343466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd30144f40 0x7fcd3006a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:15.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.746+0000 7fcd2f7fe700 1 -- 172.21.15.189:0/195343466 >> v1:172.21.15.192:6789/0 conn(0x7fcd3015fcd0 legacy=0x7fcd3015ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:15.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.746+0000 7fcd2f7fe700 1 -- 172.21.15.189:0/195343466 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3015f1f0 con 0x7fcd3015c150 2024-04-06T13:48:15.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.746+0000 7fcd2f7fe700 1 --2- 172.21.15.189:0/195343466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd3015c150 0x7fcd3006ac70 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7fcd28009e50 tx=0x7fcd2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d64bb590afe5ac5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:15.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.746+0000 7fcd2dffb700 1 -- 172.21.15.189:0/195343466 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2800e070 con 0x7fcd3015c150 2024-04-06T13:48:15.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.746+0000 7fcd2dffb700 1 -- 172.21.15.189:0/195343466 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2800b910 con 0x7fcd3015c150 2024-04-06T13:48:15.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.746+0000 7fcd2dffb700 1 -- 172.21.15.189:0/195343466 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2800bcc0 con 0x7fcd3015c150 2024-04-06T13:48:15.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.746+0000 7fcd3584c700 1 -- 172.21.15.189:0/195343466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd3015c150 msgr2=0x7fcd3006ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:15.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.746+0000 7fcd3584c700 1 --2- 172.21.15.189:0/195343466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd3015c150 0x7fcd3006ac70 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7fcd28009e50 tx=0x7fcd2800b330 comp rx=0 tx=0).stop 2024-04-06T13:48:15.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.747+0000 7fcd3584c700 1 -- 172.21.15.189:0/195343466 shutdown_connections 2024-04-06T13:48:15.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.747+0000 7fcd3584c700 1 --2- 172.21.15.189:0/195343466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd3015c150 0x7fcd3006ac70 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:15.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.747+0000 7fcd3584c700 1 --2- 172.21.15.189:0/195343466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd30144f40 0x7fcd3006a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:15.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.747+0000 7fcd3584c700 1 -- 172.21.15.189:0/195343466 >> 172.21.15.189:0/195343466 conn(0x7fcd300d00b0 msgr2=0x7fcd300d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:15.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.747+0000 7fcd3584c700 1 -- 172.21.15.189:0/195343466 shutdown_connections 2024-04-06T13:48:15.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.747+0000 7fcd3584c700 1 -- 172.21.15.189:0/195343466 wait complete. 2024-04-06T13:48:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.748+0000 7fcd3584c700 1 Processor -- start 2024-04-06T13:48:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.748+0000 7fcd3584c700 1 -- start start 2024-04-06T13:48:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd3584c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd30144f40 0x7fcd300c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd3584c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd3015c150 0x7fcd30133980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd3584c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd3015fcd0 0x7fcd3006fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd3584c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd301f08c0 con 0x7fcd30144f40 2024-04-06T13:48:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd3584c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd30133ec0 con 0x7fcd3015fcd0 2024-04-06T13:48:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd3584c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd300c4f30 con 0x7fcd3015c150 2024-04-06T13:48:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd2effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd30144f40 0x7fcd300c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd2effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd30144f40 0x7fcd300c78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51646/0 (socket says 172.21.15.189:51646) 2024-04-06T13:48:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd2effd700 1 -- 172.21.15.189:0/1739502980 learned_addr learned my addr 172.21.15.189:0/1739502980 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd2f7fe700 1 --2- 172.21.15.189:0/1739502980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd3015fcd0 0x7fcd3006fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd2e7fc700 1 --2- 172.21.15.189:0/1739502980 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd3015c150 0x7fcd30133980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd2effd700 1 -- 172.21.15.189:0/1739502980 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd3015c150 msgr2=0x7fcd30133980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd2effd700 1 --2- 172.21.15.189:0/1739502980 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd3015c150 0x7fcd30133980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd2effd700 1 -- 172.21.15.189:0/1739502980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd3015fcd0 msgr2=0x7fcd3006fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd2effd700 1 --2- 172.21.15.189:0/1739502980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd3015fcd0 0x7fcd3006fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd2effd700 1 -- 172.21.15.189:0/1739502980 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd28009b30 con 0x7fcd30144f40 2024-04-06T13:48:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd2effd700 1 --2- 172.21.15.189:0/1739502980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd30144f40 0x7fcd300c78e0 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7fcd2400e9e0 tx=0x7fcd2400ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd17fff700 1 -- 172.21.15.189:0/1739502980 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd24009710 con 0x7fcd30144f40 2024-04-06T13:48:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd17fff700 1 -- 172.21.15.189:0/1739502980 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2400ee80 con 0x7fcd30144f40 2024-04-06T13:48:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd3584c700 1 -- 172.21.15.189:0/1739502980 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd30070230 con 0x7fcd30144f40 2024-04-06T13:48:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd17fff700 1 -- 172.21.15.189:0/1739502980 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2400ca50 con 0x7fcd30144f40 2024-04-06T13:48:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.749+0000 7fcd3584c700 1 -- 172.21.15.189:0/1739502980 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd300c84d0 con 0x7fcd30144f40 2024-04-06T13:48:15.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.750+0000 7fcd3584c700 1 -- 172.21.15.189:0/1739502980 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd300c5210 con 0x7fcd30144f40 2024-04-06T13:48:15.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.751+0000 7fcd17fff700 1 -- 172.21.15.189:0/1739502980 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fcd2400cbb0 con 0x7fcd30144f40 2024-04-06T13:48:15.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.751+0000 7fcd17fff700 1 --2- 172.21.15.189:0/1739502980 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcd1c06aed0 0x7fcd1c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:15.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.751+0000 7fcd17fff700 1 -- 172.21.15.189:0/1739502980 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fcd24089510 con 0x7fcd30144f40 2024-04-06T13:48:15.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.751+0000 7fcd2e7fc700 1 --2- 172.21.15.189:0/1739502980 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcd1c06aed0 0x7fcd1c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:15.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.752+0000 7fcd2e7fc700 1 --2- 172.21.15.189:0/1739502980 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcd1c06aed0 0x7fcd1c06d390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcd3015e310 tx=0x7fcd20009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:15.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:15.756+0000 7fcd17fff700 1 -- 172.21.15.189:0/1739502980 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fcd24054550 con 0x7fcd30144f40 2024-04-06T13:48:16.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.031+0000 7fcd3584c700 1 -- 172.21.15.189:0/1739502980 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fcd30067bb0 con 0x7fcd30144f40 2024-04-06T13:48:16.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.031+0000 7fcd17fff700 1 -- 172.21.15.189:0/1739502980 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v52) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fcd24057b70 con 0x7fcd30144f40 2024-04-06T13:48:16.031 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:16.031 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:48:16.031 INFO:teuthology.orchestra.run.smithi189.stdout: "auth_dump": [ 2024-04-06T13:48:16.031 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:16.031 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.a", 2024-04-06T13:48:16.031 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmTjYMNBAA5voWGV4Pa7pE27VogrdipQ==", 2024-04-06T13:48:16.032 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:48:16.032 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:48:16.032 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:48:16.032 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:48:16.032 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:48:16.032 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.032 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:16.032 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:16.032 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.b", 2024-04-06T13:48:16.032 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm9A1OBRAAnHzubCvRUX8UdW8hmDzKbQ==", 2024-04-06T13:48:16.032 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:48:16.032 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:48:16.032 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:48:16.032 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:48:16.033 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:48:16.033 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.033 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:16.033 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:16.033 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.c", 2024-04-06T13:48:16.033 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmz0hhOxAAD3/GINNmGoFk+oPyiHHJXg==", 2024-04-06T13:48:16.033 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:48:16.033 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:48:16.033 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:48:16.033 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:48:16.033 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:48:16.033 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.033 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:16.033 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:16.033 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.d", 2024-04-06T13:48:16.034 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmhoAeDRAAoQC0vicWumBRTwpi3PtYaw==", 2024-04-06T13:48:16.034 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:48:16.034 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:48:16.034 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:48:16.034 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:48:16.034 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:48:16.034 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.034 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:16.034 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:16.034 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.0", 2024-04-06T13:48:16.034 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDkOxFmkxLWJhAAFXewCNOFX7dhAZVgUmewpQ==", 2024-04-06T13:48:16.034 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:48:16.034 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:48:16.034 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:48:16.035 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:48:16.035 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.035 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:16.035 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:16.035 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.1", 2024-04-06T13:48:16.035 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDnOxFmHYqWJRAABlsVCiv3e7YJMbfodJTWag==", 2024-04-06T13:48:16.035 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:48:16.035 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:48:16.035 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:48:16.035 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:48:16.035 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.035 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:16.035 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:16.035 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.2", 2024-04-06T13:48:16.036 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDqOxFmkhcvJBAA8NMEpHVGbGALjV9RISKP8g==", 2024-04-06T13:48:16.036 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:48:16.036 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:48:16.036 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:48:16.036 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:48:16.036 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.036 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:16.036 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:16.036 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.3", 2024-04-06T13:48:16.036 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDtOxFmDGzhIxAAamBoAbE8+lAel4os67meyQ==", 2024-04-06T13:48:16.036 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:48:16.036 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:48:16.036 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:48:16.036 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:48:16.037 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.037 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:16.037 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:16.037 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.4", 2024-04-06T13:48:16.037 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDzOxFmobZDJxAA9l8wdS81Eyw6+17I5ac6bw==", 2024-04-06T13:48:16.037 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:48:16.037 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:48:16.037 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:48:16.037 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:48:16.037 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.037 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:16.037 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:16.037 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.5", 2024-04-06T13:48:16.037 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD2OxFmi+ruJRAASlQ0o9jX/wA7U5AosnL+Wg==", 2024-04-06T13:48:16.038 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:48:16.038 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:48:16.038 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:48:16.038 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:48:16.038 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.038 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:16.038 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:16.038 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.6", 2024-04-06T13:48:16.038 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD5OxFmbwuuJBAA0wqJRmnJyskQPZJ7/LnpvA==", 2024-04-06T13:48:16.038 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:48:16.038 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:48:16.038 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:48:16.038 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:48:16.038 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.039 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:16.039 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:16.039 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.7", 2024-04-06T13:48:16.039 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD8OxFmDmRgIxAAzyW9AGAbmb67ZvB5qunwLQ==", 2024-04-06T13:48:16.039 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:48:16.039 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:48:16.039 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:48:16.039 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:48:16.039 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.039 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:16.039 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:16.039 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.0", 2024-04-06T13:48:16.039 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm0ByJERAAoEV8Yfc09+Go6u95XafElQ==", 2024-04-06T13:48:16.039 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:48:16.039 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:48:16.040 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow r", 2024-04-06T13:48:16.040 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-06T13:48:16.040 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.040 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:16.040 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:16.040 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.1", 2024-04-06T13:48:16.040 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmSV5lGBAAavDEo2LGy0IHpWSAuitHXg==", 2024-04-06T13:48:16.040 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:48:16.040 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:48:16.040 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:48:16.040 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:48:16.040 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:48:16.040 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.040 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:16.041 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:16.041 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.2", 2024-04-06T13:48:16.041 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm/Gz5FBAA3xX/rsAyWZ1JoUVlQlBIvQ==", 2024-04-06T13:48:16.041 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:48:16.041 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:48:16.041 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:48:16.041 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:48:16.041 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:48:16.041 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.041 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:16.041 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:16.041 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.3", 2024-04-06T13:48:16.041 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmzra7GxAADbnzIvWnRTo65LvPwtxWrg==", 2024-04-06T13:48:16.041 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:48:16.041 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:48:16.042 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:48:16.042 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:48:16.042 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:48:16.042 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.042 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:16.042 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:16.042 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.admin", 2024-04-06T13:48:16.042 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmxG+wCBAANg5kt6TOUQdj3/X0Mqgx1g==", 2024-04-06T13:48:16.042 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:48:16.042 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:48:16.042 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:48:16.042 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:48:16.042 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:48:16.042 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.042 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:16.043 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:16.043 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.x", 2024-04-06T13:48:16.043 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmPLyBMBAAYfhkI9Z1CPlf1Tijm+L7xA==", 2024-04-06T13:48:16.043 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:48:16.043 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:48:16.043 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T13:48:16.043 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:48:16.043 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.043 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:16.043 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:16.043 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.y", 2024-04-06T13:48:16.043 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFm1nHJHhAAQWRe8cU8duWuUJiYaDeTaQ==", 2024-04-06T13:48:16.043 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:48:16.043 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:48:16.043 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T13:48:16.044 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:48:16.044 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.044 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:16.044 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-04-06T13:48:16.044 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:48:16.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.033+0000 7fcd3584c700 1 -- 172.21.15.189:0/1739502980 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcd1c06aed0 msgr2=0x7fcd1c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:16.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.033+0000 7fcd3584c700 1 --2- 172.21.15.189:0/1739502980 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcd1c06aed0 0x7fcd1c06d390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcd3015e310 tx=0x7fcd20009040 comp rx=0 tx=0).stop 2024-04-06T13:48:16.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.033+0000 7fcd3584c700 1 -- 172.21.15.189:0/1739502980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd30144f40 msgr2=0x7fcd300c78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:16.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.033+0000 7fcd3584c700 1 --2- 172.21.15.189:0/1739502980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd30144f40 0x7fcd300c78e0 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7fcd2400e9e0 tx=0x7fcd2400ecf0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.033+0000 7fcd3584c700 1 -- 172.21.15.189:0/1739502980 shutdown_connections 2024-04-06T13:48:16.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.033+0000 7fcd3584c700 1 --2- 172.21.15.189:0/1739502980 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcd1c06aed0 0x7fcd1c06d390 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.033+0000 7fcd3584c700 1 --2- 172.21.15.189:0/1739502980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd3015fcd0 0x7fcd3006fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.033+0000 7fcd3584c700 1 --2- 172.21.15.189:0/1739502980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd30144f40 0x7fcd300c78e0 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.033+0000 7fcd3584c700 1 --2- 172.21.15.189:0/1739502980 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd3015c150 0x7fcd30133980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.033+0000 7fcd3584c700 1 -- 172.21.15.189:0/1739502980 >> 172.21.15.189:0/1739502980 conn(0x7fcd300d00b0 msgr2=0x7fcd301441b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:16.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.033+0000 7fcd3584c700 1 -- 172.21.15.189:0/1739502980 shutdown_connections 2024-04-06T13:48:16.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.033+0000 7fcd3584c700 1 -- 172.21.15.189:0/1739502980 wait complete. 2024-04-06T13:48:16.047 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:48:16.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.172+0000 7fc018f54700 1 Processor -- start 2024-04-06T13:48:16.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.172+0000 7fc018f54700 1 -- start start 2024-04-06T13:48:16.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.173+0000 7fc018f54700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc01415d7b0 0x7fc0140af040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:16.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.173+0000 7fc018f54700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0140723f0 0x7fc0140728a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:16.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.173+0000 7fc018f54700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc014071950 con 0x7fc0140b39d0 2024-04-06T13:48:16.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.173+0000 7fc018f54700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0140b4170 con 0x7fc01415d7b0 2024-04-06T13:48:16.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.173+0000 7fc018f54700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0140b3e90 con 0x7fc0140723f0 2024-04-06T13:48:16.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.174+0000 7fc011d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc0140b39d0 0x7fc0140ac8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47988/0 (socket says 172.21.15.189:47988) 2024-04-06T13:48:16.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.174+0000 7fc011d9b700 1 -- 172.21.15.189:0/3536826477 learned_addr learned my addr 172.21.15.189:0/3536826477 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:16.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.174+0000 7fc012d9d700 1 --2- 172.21.15.189:0/3536826477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0140723f0 0x7fc0140728a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:16.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.174+0000 7fc01259c700 1 --2- 172.21.15.189:0/3536826477 >> v2:172.21.15.192:3300/0 conn(0x7fc01415d7b0 0x7fc0140af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:16.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.174+0000 7fc01159a700 1 -- 172.21.15.189:0/3536826477 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774012803 0 0) 0x7fc014071950 con 0x7fc0140b39d0 2024-04-06T13:48:16.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.174+0000 7fc01159a700 1 -- 172.21.15.189:0/3536826477 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0000035b0 con 0x7fc0140b39d0 2024-04-06T13:48:16.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.174+0000 7fc01159a700 1 -- 172.21.15.189:0/3536826477 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 643498445 0 0) 0x7fc0000035b0 con 0x7fc0140b39d0 2024-04-06T13:48:16.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.174+0000 7fc012d9d700 1 -- 172.21.15.189:0/3536826477 >> v2:172.21.15.192:3300/0 conn(0x7fc01415d7b0 msgr2=0x7fc0140af040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:16.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.174+0000 7fc012d9d700 1 --2- 172.21.15.189:0/3536826477 >> v2:172.21.15.192:3300/0 conn(0x7fc01415d7b0 0x7fc0140af040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.174+0000 7fc012d9d700 1 -- 172.21.15.189:0/3536826477 >> v1:172.21.15.189:6789/0 conn(0x7fc0140b39d0 legacy=0x7fc0140ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:16.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.174+0000 7fc012d9d700 1 -- 172.21.15.189:0/3536826477 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc014071bf0 con 0x7fc0140723f0 2024-04-06T13:48:16.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.174+0000 7fc01259c700 1 --2- 172.21.15.189:0/3536826477 >> v2:172.21.15.192:3300/0 conn(0x7fc01415d7b0 0x7fc0140af040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:16.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.174+0000 7fc012d9d700 1 --2- 172.21.15.189:0/3536826477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0140723f0 0x7fc0140728a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc00c009d90 tx=0x7fc00c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4beebbdf1d26bff2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:16.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.174+0000 7fc01159a700 1 -- 172.21.15.189:0/3536826477 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c00e070 con 0x7fc0140723f0 2024-04-06T13:48:16.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.174+0000 7fc01159a700 1 -- 172.21.15.189:0/3536826477 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00c003a20 con 0x7fc0140723f0 2024-04-06T13:48:16.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.174+0000 7fc01159a700 1 -- 172.21.15.189:0/3536826477 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc001670 con 0x7fc0140723f0 2024-04-06T13:48:16.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.175+0000 7fc018f54700 1 -- 172.21.15.189:0/3536826477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0140723f0 msgr2=0x7fc0140728a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:16.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.175+0000 7fc018f54700 1 --2- 172.21.15.189:0/3536826477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0140723f0 0x7fc0140728a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc00c009d90 tx=0x7fc00c00b330 comp rx=0 tx=0).stop 2024-04-06T13:48:16.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.175+0000 7fc018f54700 1 -- 172.21.15.189:0/3536826477 shutdown_connections 2024-04-06T13:48:16.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.175+0000 7fc018f54700 1 --2- 172.21.15.189:0/3536826477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0140723f0 0x7fc0140728a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.175+0000 7fc018f54700 1 --2- 172.21.15.189:0/3536826477 >> v2:172.21.15.192:3300/0 conn(0x7fc01415d7b0 0x7fc0140af040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.175+0000 7fc018f54700 1 -- 172.21.15.189:0/3536826477 >> 172.21.15.189:0/3536826477 conn(0x7fc0140d00b0 msgr2=0x7fc0140d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:16.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.175+0000 7fc018f54700 1 -- 172.21.15.189:0/3536826477 shutdown_connections 2024-04-06T13:48:16.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.175+0000 7fc018f54700 1 -- 172.21.15.189:0/3536826477 wait complete. 2024-04-06T13:48:16.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.176+0000 7fc018f54700 1 Processor -- start 2024-04-06T13:48:16.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.176+0000 7fc018f54700 1 -- start start 2024-04-06T13:48:16.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.177+0000 7fc018f54700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0140723f0 0x7fc01414c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:16.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.177+0000 7fc018f54700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0140b39d0 0x7fc0140cce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:16.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.177+0000 7fc018f54700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0140cd370 0x7fc01415f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:16.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.177+0000 7fc018f54700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc014159c30 con 0x7fc0140cd370 2024-04-06T13:48:16.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.177+0000 7fc018f54700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0140cd810 con 0x7fc0140723f0 2024-04-06T13:48:16.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.177+0000 7fc018f54700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc014159ed0 con 0x7fc0140b39d0 2024-04-06T13:48:16.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.177+0000 7fc012d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0140cd370 0x7fc01415f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:16.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.177+0000 7fc012d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0140cd370 0x7fc01415f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51656/0 (socket says 172.21.15.189:51656) 2024-04-06T13:48:16.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.177+0000 7fc012d9d700 1 -- 172.21.15.189:0/2984681733 learned_addr learned my addr 172.21.15.189:0/2984681733 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:16.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.177+0000 7fc011d9b700 1 --2- 172.21.15.189:0/2984681733 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0140b39d0 0x7fc0140cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:16.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.177+0000 7fc01259c700 1 --2- 172.21.15.189:0/2984681733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0140723f0 0x7fc01414c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:16.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.178+0000 7fc012d9d700 1 -- 172.21.15.189:0/2984681733 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0140b39d0 msgr2=0x7fc0140cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:16.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.178+0000 7fc012d9d700 1 --2- 172.21.15.189:0/2984681733 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0140b39d0 0x7fc0140cce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.178+0000 7fc012d9d700 1 -- 172.21.15.189:0/2984681733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0140723f0 msgr2=0x7fc01414c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:16.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.178+0000 7fc012d9d700 1 --2- 172.21.15.189:0/2984681733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0140723f0 0x7fc01414c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.178+0000 7fc012d9d700 1 -- 172.21.15.189:0/2984681733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0040094a0 con 0x7fc0140cd370 2024-04-06T13:48:16.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.178+0000 7fc012d9d700 1 --2- 172.21.15.189:0/2984681733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0140cd370 0x7fc01415f390 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7fc0140b3fd0 tx=0x7fc00c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:16.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.178+0000 7fc00affd700 1 -- 172.21.15.189:0/2984681733 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c009410 con 0x7fc0140cd370 2024-04-06T13:48:16.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.178+0000 7fc00affd700 1 -- 172.21.15.189:0/2984681733 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00c004460 con 0x7fc0140cd370 2024-04-06T13:48:16.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.178+0000 7fc00affd700 1 -- 172.21.15.189:0/2984681733 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c004810 con 0x7fc0140cd370 2024-04-06T13:48:16.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.178+0000 7fc018f54700 1 -- 172.21.15.189:0/2984681733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc00c009a70 con 0x7fc0140cd370 2024-04-06T13:48:16.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.178+0000 7fc018f54700 1 -- 172.21.15.189:0/2984681733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01415a490 con 0x7fc0140cd370 2024-04-06T13:48:16.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.179+0000 7fc018f54700 1 -- 172.21.15.189:0/2984681733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc014067bb0 con 0x7fc0140cd370 2024-04-06T13:48:16.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.179+0000 7fc00affd700 1 -- 172.21.15.189:0/2984681733 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fc00c012430 con 0x7fc0140cd370 2024-04-06T13:48:16.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.180+0000 7fc00affd700 1 --2- 172.21.15.189:0/2984681733 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbff406aed0 0x7fbff406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:16.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.180+0000 7fc00affd700 1 -- 172.21.15.189:0/2984681733 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc00c08b160 con 0x7fc0140cd370 2024-04-06T13:48:16.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.180+0000 7fc01259c700 1 --2- 172.21.15.189:0/2984681733 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbff406aed0 0x7fbff406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:16.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.180+0000 7fc01259c700 1 --2- 172.21.15.189:0/2984681733 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbff406aed0 0x7fbff406d390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc004009d70 tx=0x7fc004009210 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:16.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.184+0000 7fc00affd700 1 -- 172.21.15.189:0/2984681733 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc00c056120 con 0x7fc0140cd370 2024-04-06T13:48:16.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.382+0000 7fc018f54700 1 -- 172.21.15.189:0/2984681733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0140d0e00 con 0x7fc0140cd370 2024-04-06T13:48:16.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.383+0000 7fc00affd700 1 -- 172.21.15.189:0/2984681733 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v244) v1 ==== 74+0+12768 (secure 0 0 0) 0x7fc00c059740 con 0x7fc0140cd370 2024-04-06T13:48:16.383 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:16.383 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":244,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:48:15.591431+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:48:16.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.385+0000 7fc018f54700 1 -- 172.21.15.189:0/2984681733 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbff406aed0 msgr2=0x7fbff406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:16.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.385+0000 7fc018f54700 1 --2- 172.21.15.189:0/2984681733 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbff406aed0 0x7fbff406d390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc004009d70 tx=0x7fc004009210 comp rx=0 tx=0).stop 2024-04-06T13:48:16.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.385+0000 7fc018f54700 1 -- 172.21.15.189:0/2984681733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0140cd370 msgr2=0x7fc01415f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:16.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.385+0000 7fc018f54700 1 --2- 172.21.15.189:0/2984681733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0140cd370 0x7fc01415f390 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7fc0140b3fd0 tx=0x7fc00c004ad0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.385+0000 7fc018f54700 1 -- 172.21.15.189:0/2984681733 shutdown_connections 2024-04-06T13:48:16.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.385+0000 7fc018f54700 1 --2- 172.21.15.189:0/2984681733 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbff406aed0 0x7fbff406d390 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.385+0000 7fc018f54700 1 --2- 172.21.15.189:0/2984681733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0140cd370 0x7fc01415f390 unknown :-1 s=CLOSED pgs=2651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.385+0000 7fc018f54700 1 --2- 172.21.15.189:0/2984681733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0140723f0 0x7fc01414c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.385+0000 7fc018f54700 1 --2- 172.21.15.189:0/2984681733 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0140b39d0 0x7fc0140cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.385+0000 7fc018f54700 1 -- 172.21.15.189:0/2984681733 >> 172.21.15.189:0/2984681733 conn(0x7fc0140d00b0 msgr2=0x7fc0141580b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:16.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.385+0000 7fc018f54700 1 -- 172.21.15.189:0/2984681733 shutdown_connections 2024-04-06T13:48:16.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.385+0000 7fc018f54700 1 -- 172.21.15.189:0/2984681733 wait complete. 2024-04-06T13:48:16.398 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:48:16.398 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:48:16.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.524+0000 7f2702d9e700 1 Processor -- start 2024-04-06T13:48:16.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.524+0000 7f2702d9e700 1 -- start start 2024-04-06T13:48:16.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.525+0000 7f2702d9e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f26fc0b4660 0x7f26fc0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:16.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.525+0000 7f2702d9e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f26fc0b2fc0 0x7f26fc0b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.525+0000 7f2702d9e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26fc0b5030 con 0x7f26fc15a550 2024-04-06T13:48:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.525+0000 7f2702d9e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26fc0cdbe0 con 0x7f26fc0b2fc0 2024-04-06T13:48:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.525+0000 7f2702d9e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26fc15a940 con 0x7f26fc0b4660 2024-04-06T13:48:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.525+0000 7f270133b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f26fc0b2fc0 0x7f26fc0b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.525+0000 7f270133b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f26fc0b2fc0 0x7f26fc0b33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51658/0 (socket says 172.21.15.189:51658) 2024-04-06T13:48:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.525+0000 7f270133b700 1 -- 172.21.15.189:0/2395347635 learned_addr learned my addr 172.21.15.189:0/2395347635 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.525+0000 7f26fbfff700 1 --2- 172.21.15.189:0/2395347635 >> v2:172.21.15.192:3300/0 conn(0x7f26fc0b4660 0x7f26fc0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.525+0000 7f270133b700 1 -- 172.21.15.189:0/2395347635 >> v2:172.21.15.192:3300/0 conn(0x7f26fc0b4660 msgr2=0x7f26fc0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.525+0000 7f270133b700 1 --2- 172.21.15.189:0/2395347635 >> v2:172.21.15.192:3300/0 conn(0x7f26fc0b4660 0x7f26fc0b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.525+0000 7f270133b700 1 -- 172.21.15.189:0/2395347635 >> v1:172.21.15.192:6789/0 conn(0x7f26fc15a550 legacy=0x7f26fc0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.525+0000 7f270133b700 1 -- 172.21.15.189:0/2395347635 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26fc15ac00 con 0x7f26fc0b2fc0 2024-04-06T13:48:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.525+0000 7f26fbfff700 1 --2- 172.21.15.189:0/2395347635 >> v2:172.21.15.192:3300/0 conn(0x7f26fc0b4660 0x7f26fc0b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.525+0000 7f270133b700 1 --2- 172.21.15.189:0/2395347635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc0b2fc0 0x7f26fc0b33d0 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f26fc0b2d40 tx=0x7f26f400d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b67dd43f262ef4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.525+0000 7f26fb7fe700 1 -- 172.21.15.189:0/2395347635 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f4010070 con 0x7f26fc0b2fc0 2024-04-06T13:48:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.526+0000 7f26fb7fe700 1 -- 172.21.15.189:0/2395347635 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f400dd90 con 0x7f26fc0b2fc0 2024-04-06T13:48:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.526+0000 7f26fb7fe700 1 -- 172.21.15.189:0/2395347635 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f400b310 con 0x7f26fc0b2fc0 2024-04-06T13:48:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.526+0000 7f2702d9e700 1 -- 172.21.15.189:0/2395347635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc0b2fc0 msgr2=0x7f26fc0b33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.526+0000 7f2702d9e700 1 --2- 172.21.15.189:0/2395347635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc0b2fc0 0x7f26fc0b33d0 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f26fc0b2d40 tx=0x7f26f400d430 comp rx=0 tx=0).stop 2024-04-06T13:48:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.526+0000 7f2702d9e700 1 -- 172.21.15.189:0/2395347635 shutdown_connections 2024-04-06T13:48:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.526+0000 7f2702d9e700 1 --2- 172.21.15.189:0/2395347635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc0b2fc0 0x7f26fc0b33d0 unknown :-1 s=CLOSED pgs=2652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.526+0000 7f2702d9e700 1 --2- 172.21.15.189:0/2395347635 >> v2:172.21.15.192:3300/0 conn(0x7f26fc0b4660 0x7f26fc0b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.526+0000 7f2702d9e700 1 -- 172.21.15.189:0/2395347635 >> 172.21.15.189:0/2395347635 conn(0x7f26fc0d00b0 msgr2=0x7f26fc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.526+0000 7f2702d9e700 1 -- 172.21.15.189:0/2395347635 shutdown_connections 2024-04-06T13:48:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.526+0000 7f2702d9e700 1 -- 172.21.15.189:0/2395347635 wait complete. 2024-04-06T13:48:16.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.527+0000 7f2702d9e700 1 Processor -- start 2024-04-06T13:48:16.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.527+0000 7f2702d9e700 1 -- start start 2024-04-06T13:48:16.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.528+0000 7f2702d9e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc0b2fc0 0x7f26fc13b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.528+0000 7f2702d9e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26fc0b4660 0x7f26fc14ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.528+0000 7f2702d9e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26fc14d350 0x7f26fc153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.528+0000 7f2702d9e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc14d800 con 0x7f26fc0b2fc0 2024-04-06T13:48:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.528+0000 7f2702d9e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc1548b0 con 0x7f26fc0b4660 2024-04-06T13:48:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.528+0000 7f2702d9e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc154b90 con 0x7f26fc14d350 2024-04-06T13:48:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.528+0000 7f2700b3a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc0b2fc0 0x7f26fc13b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.528+0000 7f2700b3a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc0b2fc0 0x7f26fc13b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51660/0 (socket says 172.21.15.189:51660) 2024-04-06T13:48:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.528+0000 7f2700b3a700 1 -- 172.21.15.189:0/320435712 learned_addr learned my addr 172.21.15.189:0/320435712 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.528+0000 7f270133b700 1 --2- 172.21.15.189:0/320435712 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26fc14d350 0x7f26fc153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.528+0000 7f26fbfff700 1 --2- 172.21.15.189:0/320435712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26fc0b4660 0x7f26fc14ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.529+0000 7f2700b3a700 1 -- 172.21.15.189:0/320435712 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26fc14d350 msgr2=0x7f26fc153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.529+0000 7f2700b3a700 1 --2- 172.21.15.189:0/320435712 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26fc14d350 0x7f26fc153ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.529+0000 7f2700b3a700 1 -- 172.21.15.189:0/320435712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26fc0b4660 msgr2=0x7f26fc14ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.529+0000 7f2700b3a700 1 --2- 172.21.15.189:0/320435712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26fc0b4660 0x7f26fc14ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.529+0000 7f2700b3a700 1 -- 172.21.15.189:0/320435712 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f400bd40 con 0x7f26fc0b2fc0 2024-04-06T13:48:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.529+0000 7f26fbfff700 1 --2- 172.21.15.189:0/320435712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26fc0b4660 0x7f26fc14ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.529+0000 7f2700b3a700 1 --2- 172.21.15.189:0/320435712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc0b2fc0 0x7f26fc13b640 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f26f000c4e0 tx=0x7f26f000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.529+0000 7f26f97fa700 1 -- 172.21.15.189:0/320435712 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f00052d0 con 0x7f26fc0b2fc0 2024-04-06T13:48:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.529+0000 7f26f97fa700 1 -- 172.21.15.189:0/320435712 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f0005430 con 0x7f26fc0b2fc0 2024-04-06T13:48:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.529+0000 7f2702d9e700 1 -- 172.21.15.189:0/320435712 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26fc154e70 con 0x7f26fc0b2fc0 2024-04-06T13:48:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.529+0000 7f26f97fa700 1 -- 172.21.15.189:0/320435712 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f00038c0 con 0x7f26fc0b2fc0 2024-04-06T13:48:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.529+0000 7f2702d9e700 1 -- 172.21.15.189:0/320435712 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26fc154020 con 0x7f26fc0b2fc0 2024-04-06T13:48:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.530+0000 7f2702d9e700 1 -- 172.21.15.189:0/320435712 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26fc067bb0 con 0x7f26fc0b2fc0 2024-04-06T13:48:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.530+0000 7f26f97fa700 1 -- 172.21.15.189:0/320435712 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f26f0003a20 con 0x7f26fc0b2fc0 2024-04-06T13:48:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.531+0000 7f26f97fa700 1 --2- 172.21.15.189:0/320435712 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f26e006aed0 0x7f26e006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.531+0000 7f26f97fa700 1 -- 172.21.15.189:0/320435712 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f26f0088880 con 0x7f26fc0b2fc0 2024-04-06T13:48:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.531+0000 7f26fbfff700 1 --2- 172.21.15.189:0/320435712 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f26e006aed0 0x7f26e006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.531+0000 7f26fbfff700 1 --2- 172.21.15.189:0/320435712 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f26e006aed0 0x7f26e006d390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f26e80099f0 tx=0x7f26e8008040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:16.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.535+0000 7f26f97fa700 1 -- 172.21.15.189:0/320435712 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f26f0053a20 con 0x7f26fc0b2fc0 2024-04-06T13:48:16.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.729+0000 7f2702d9e700 1 -- 172.21.15.189:0/320435712 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f26fc074990 con 0x7f26fc0b2fc0 2024-04-06T13:48:16.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.730+0000 7f26f97fa700 1 -- 172.21.15.189:0/320435712 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v244) v1 ==== 93+0+10 (secure 0 0 0) 0x7f26f00170a0 con 0x7f26fc0b2fc0 2024-04-06T13:48:16.730 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:48:16.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.732+0000 7f2702d9e700 1 -- 172.21.15.189:0/320435712 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f26e006aed0 msgr2=0x7f26e006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:16.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.732+0000 7f2702d9e700 1 --2- 172.21.15.189:0/320435712 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f26e006aed0 0x7f26e006d390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f26e80099f0 tx=0x7f26e8008040 comp rx=0 tx=0).stop 2024-04-06T13:48:16.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.732+0000 7f2702d9e700 1 -- 172.21.15.189:0/320435712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc0b2fc0 msgr2=0x7f26fc13b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:16.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.732+0000 7f2702d9e700 1 --2- 172.21.15.189:0/320435712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc0b2fc0 0x7f26fc13b640 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f26f000c4e0 tx=0x7f26f000c7f0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.732+0000 7f2702d9e700 1 -- 172.21.15.189:0/320435712 shutdown_connections 2024-04-06T13:48:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.732+0000 7f2702d9e700 1 --2- 172.21.15.189:0/320435712 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f26e006aed0 0x7f26e006d390 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.732+0000 7f2702d9e700 1 --2- 172.21.15.189:0/320435712 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26fc14d350 0x7f26fc153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.732+0000 7f2702d9e700 1 --2- 172.21.15.189:0/320435712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc0b2fc0 0x7f26fc13b640 unknown :-1 s=CLOSED pgs=2653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.732+0000 7f2702d9e700 1 --2- 172.21.15.189:0/320435712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26fc0b4660 0x7f26fc14ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.732+0000 7f2702d9e700 1 -- 172.21.15.189:0/320435712 >> 172.21.15.189:0/320435712 conn(0x7f26fc0d00b0 msgr2=0x7f26fc155c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.732+0000 7f2702d9e700 1 -- 172.21.15.189:0/320435712 shutdown_connections 2024-04-06T13:48:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.732+0000 7f2702d9e700 1 -- 172.21.15.189:0/320435712 wait complete. 2024-04-06T13:48:16.745 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-06T13:48:16.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.872+0000 7fb700fa9700 1 Processor -- start 2024-04-06T13:48:16.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.872+0000 7fb700fa9700 1 -- start start 2024-04-06T13:48:16.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.873+0000 7fb700fa9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb6fc0cea80 0x7fb6fc0cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:16.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.873+0000 7fb700fa9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb6fc0cdae0 0x7fb6fc06ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:16.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.873+0000 7fb700fa9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6fc15a830 con 0x7fb6fc0aec70 2024-04-06T13:48:16.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.873+0000 7fb700fa9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6fc0cd920 con 0x7fb6fc0cea80 2024-04-06T13:48:16.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.873+0000 7fb700fa9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6fc0cc050 con 0x7fb6fc0cdae0 2024-04-06T13:48:16.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.873+0000 7fb6f9d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb6fc0cea80 0x7fb6fc0cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:16.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.873+0000 7fb6f9d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb6fc0cea80 0x7fb6fc0cd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51664/0 (socket says 172.21.15.189:51664) 2024-04-06T13:48:16.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.873+0000 7fb6f9d9b700 1 -- 172.21.15.189:0/2499086901 learned_addr learned my addr 172.21.15.189:0/2499086901 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:16.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.873+0000 7fb6fad9d700 1 --2- 172.21.15.189:0/2499086901 >> v2:172.21.15.192:3300/0 conn(0x7fb6fc0cdae0 0x7fb6fc06ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:16.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.873+0000 7fb6f9d9b700 1 -- 172.21.15.189:0/2499086901 >> v2:172.21.15.192:3300/0 conn(0x7fb6fc0cdae0 msgr2=0x7fb6fc06ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:16.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.873+0000 7fb6f9d9b700 1 --2- 172.21.15.189:0/2499086901 >> v2:172.21.15.192:3300/0 conn(0x7fb6fc0cdae0 0x7fb6fc06ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.873+0000 7fb6f9d9b700 1 -- 172.21.15.189:0/2499086901 >> v1:172.21.15.192:6789/0 conn(0x7fb6fc0aec70 legacy=0x7fb6fc0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:16.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.873+0000 7fb6f9d9b700 1 -- 172.21.15.189:0/2499086901 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6fc0666f0 con 0x7fb6fc0cea80 2024-04-06T13:48:16.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.874+0000 7fb6f9d9b700 1 --2- 172.21.15.189:0/2499086901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6fc0cea80 0x7fb6fc0cd3e0 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc06cc10 tx=0x7fb6ec00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7395cf19127df90 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:16.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.874+0000 7fb6f959a700 1 -- 172.21.15.189:0/2499086901 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec00dd30 con 0x7fb6fc0cea80 2024-04-06T13:48:16.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.874+0000 7fb6f959a700 1 -- 172.21.15.189:0/2499086901 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6ec00de90 con 0x7fb6fc0cea80 2024-04-06T13:48:16.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.874+0000 7fb6f959a700 1 -- 172.21.15.189:0/2499086901 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec010690 con 0x7fb6fc0cea80 2024-04-06T13:48:16.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.874+0000 7fb700fa9700 1 -- 172.21.15.189:0/2499086901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6fc0cea80 msgr2=0x7fb6fc0cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:16.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.874+0000 7fb700fa9700 1 --2- 172.21.15.189:0/2499086901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6fc0cea80 0x7fb6fc0cd3e0 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc06cc10 tx=0x7fb6ec00be70 comp rx=0 tx=0).stop 2024-04-06T13:48:16.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.874+0000 7fb700fa9700 1 -- 172.21.15.189:0/2499086901 shutdown_connections 2024-04-06T13:48:16.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.874+0000 7fb700fa9700 1 --2- 172.21.15.189:0/2499086901 >> v2:172.21.15.192:3300/0 conn(0x7fb6fc0cdae0 0x7fb6fc06ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.874+0000 7fb700fa9700 1 --2- 172.21.15.189:0/2499086901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6fc0cea80 0x7fb6fc0cd3e0 unknown :-1 s=CLOSED pgs=2654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.874+0000 7fb700fa9700 1 -- 172.21.15.189:0/2499086901 >> 172.21.15.189:0/2499086901 conn(0x7fb6fc0cc290 msgr2=0x7fb6fc0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:16.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.874+0000 7fb700fa9700 1 -- 172.21.15.189:0/2499086901 shutdown_connections 2024-04-06T13:48:16.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.874+0000 7fb700fa9700 1 -- 172.21.15.189:0/2499086901 wait complete. 2024-04-06T13:48:16.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.876+0000 7fb700fa9700 1 Processor -- start 2024-04-06T13:48:16.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.876+0000 7fb700fa9700 1 -- start start 2024-04-06T13:48:16.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.876+0000 7fb700fa9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6fc0aec70 0x7fb6fc151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:16.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.876+0000 7fb700fa9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6fc0cdae0 0x7fb6fc0a8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:16.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.876+0000 7fb700fa9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6fc0cea80 0x7fb6fc0b4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:16.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.876+0000 7fb700fa9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc0c5a40 con 0x7fb6fc0cdae0 2024-04-06T13:48:16.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.876+0000 7fb700fa9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc0c4e30 con 0x7fb6fc0cea80 2024-04-06T13:48:16.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.876+0000 7fb700fa9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc0c5110 con 0x7fb6fc0aec70 2024-04-06T13:48:16.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.877+0000 7fb6f9d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6fc0cdae0 0x7fb6fc0a8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:16.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.877+0000 7fb6f9d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6fc0cdae0 0x7fb6fc0a8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51672/0 (socket says 172.21.15.189:51672) 2024-04-06T13:48:16.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.877+0000 7fb6f9d9b700 1 -- 172.21.15.189:0/3897429789 learned_addr learned my addr 172.21.15.189:0/3897429789 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:16.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.877+0000 7fb6fa59c700 1 --2- 172.21.15.189:0/3897429789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6fc0aec70 0x7fb6fc151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:16.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.877+0000 7fb6fad9d700 1 --2- 172.21.15.189:0/3897429789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6fc0cea80 0x7fb6fc0b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:16.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.877+0000 7fb6f9d9b700 1 -- 172.21.15.189:0/3897429789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6fc0aec70 msgr2=0x7fb6fc151b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:16.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.877+0000 7fb6f9d9b700 1 --2- 172.21.15.189:0/3897429789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6fc0aec70 0x7fb6fc151b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.877+0000 7fb6f9d9b700 1 -- 172.21.15.189:0/3897429789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6fc0cea80 msgr2=0x7fb6fc0b4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:16.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.877+0000 7fb6f9d9b700 1 --2- 172.21.15.189:0/3897429789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6fc0cea80 0x7fb6fc0b4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.877+0000 7fb6f9d9b700 1 -- 172.21.15.189:0/3897429789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ec00b7f0 con 0x7fb6fc0cdae0 2024-04-06T13:48:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.877+0000 7fb6fad9d700 1 --2- 172.21.15.189:0/3897429789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6fc0cea80 0x7fb6fc0b4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.877+0000 7fb6f9d9b700 1 --2- 172.21.15.189:0/3897429789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6fc0cdae0 0x7fb6fc0a8cd0 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec000c00 tx=0x7fb6ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.877+0000 7fb6eaffd700 1 -- 172.21.15.189:0/3897429789 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec01f070 con 0x7fb6fc0cdae0 2024-04-06T13:48:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.877+0000 7fb6eaffd700 1 -- 172.21.15.189:0/3897429789 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6ec003ba0 con 0x7fb6fc0cdae0 2024-04-06T13:48:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.877+0000 7fb700fa9700 1 -- 172.21.15.189:0/3897429789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6fc0c53f0 con 0x7fb6fc0cdae0 2024-04-06T13:48:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.877+0000 7fb6eaffd700 1 -- 172.21.15.189:0/3897429789 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec010470 con 0x7fb6fc0cdae0 2024-04-06T13:48:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.877+0000 7fb700fa9700 1 -- 172.21.15.189:0/3897429789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6fc152070 con 0x7fb6fc0cdae0 2024-04-06T13:48:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.878+0000 7fb700fa9700 1 -- 172.21.15.189:0/3897429789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6fc074990 con 0x7fb6fc0cdae0 2024-04-06T13:48:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.879+0000 7fb6eaffd700 1 -- 172.21.15.189:0/3897429789 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fb6ec019400 con 0x7fb6fc0cdae0 2024-04-06T13:48:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.879+0000 7fb6eaffd700 1 --2- 172.21.15.189:0/3897429789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb6dc06aed0 0x7fb6dc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.879+0000 7fb6eaffd700 1 -- 172.21.15.189:0/3897429789 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb6ec08c9c0 con 0x7fb6fc0cdae0 2024-04-06T13:48:16.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.879+0000 7fb6fa59c700 1 --2- 172.21.15.189:0/3897429789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb6dc06aed0 0x7fb6dc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:16.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.880+0000 7fb6fa59c700 1 --2- 172.21.15.189:0/3897429789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb6dc06aed0 0x7fb6dc06d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0000f60 tx=0x7fb6f0009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:16.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:16.884+0000 7fb6eaffd700 1 -- 172.21.15.189:0/3897429789 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb6ec0578e0 con 0x7fb6fc0cdae0 2024-04-06T13:48:17.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.079+0000 7fb700fa9700 1 -- 172.21.15.189:0/3897429789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb6fc0c5b80 con 0x7fb6fc0cdae0 2024-04-06T13:48:17.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.080+0000 7fb6eaffd700 1 -- 172.21.15.189:0/3897429789 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v244) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb6ec05af00 con 0x7fb6fc0cdae0 2024-04-06T13:48:17.080 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:17.080 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T13:48:17.080 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:17.080 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 1, 2024-04-06T13:48:17.080 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": ".mgr" 2024-04-06T13:48:17.080 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:17.080 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T13:48:17.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.082+0000 7fb700fa9700 1 -- 172.21.15.189:0/3897429789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb6dc06aed0 msgr2=0x7fb6dc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:17.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.082+0000 7fb700fa9700 1 --2- 172.21.15.189:0/3897429789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb6dc06aed0 0x7fb6dc06d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0000f60 tx=0x7fb6f0009040 comp rx=0 tx=0).stop 2024-04-06T13:48:17.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.082+0000 7fb700fa9700 1 -- 172.21.15.189:0/3897429789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6fc0cdae0 msgr2=0x7fb6fc0a8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:17.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.082+0000 7fb700fa9700 1 --2- 172.21.15.189:0/3897429789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6fc0cdae0 0x7fb6fc0a8cd0 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec000c00 tx=0x7fb6ec004750 comp rx=0 tx=0).stop 2024-04-06T13:48:17.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.082+0000 7fb700fa9700 1 -- 172.21.15.189:0/3897429789 shutdown_connections 2024-04-06T13:48:17.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.082+0000 7fb700fa9700 1 --2- 172.21.15.189:0/3897429789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb6dc06aed0 0x7fb6dc06d390 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:17.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.082+0000 7fb700fa9700 1 --2- 172.21.15.189:0/3897429789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6fc0cea80 0x7fb6fc0b4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:17.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.082+0000 7fb700fa9700 1 --2- 172.21.15.189:0/3897429789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6fc0aec70 0x7fb6fc151b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:17.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.082+0000 7fb700fa9700 1 --2- 172.21.15.189:0/3897429789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6fc0cdae0 0x7fb6fc0a8cd0 unknown :-1 s=CLOSED pgs=2655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:17.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.082+0000 7fb700fa9700 1 -- 172.21.15.189:0/3897429789 >> 172.21.15.189:0/3897429789 conn(0x7fb6fc0cc290 msgr2=0x7fb6fc0cd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:17.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.082+0000 7fb700fa9700 1 -- 172.21.15.189:0/3897429789 shutdown_connections 2024-04-06T13:48:17.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.082+0000 7fb700fa9700 1 -- 172.21.15.189:0/3897429789 wait complete. 2024-04-06T13:48:17.094 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-06T13:48:17.095 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-06T13:48:17.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.221+0000 7f079e621700 1 Processor -- start 2024-04-06T13:48:17.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.221+0000 7f079e621700 1 -- start start 2024-04-06T13:48:17.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.222+0000 7f079e621700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f079815d7b0 0x7f07980b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:17.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.222+0000 7f079e621700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f079815e500 0x7f079806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:17.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.222+0000 7f079e621700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07980b3990 con 0x7f079806a730 2024-04-06T13:48:17.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.222+0000 7f079e621700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f079815e950 con 0x7f079815d7b0 2024-04-06T13:48:17.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.222+0000 7f079e621700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f079806b0f0 con 0x7f079815e500 2024-04-06T13:48:17.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.222+0000 7f0797fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f079815d7b0 0x7f07980b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:17.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.222+0000 7f0797fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f079815d7b0 0x7f07980b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51676/0 (socket says 172.21.15.189:51676) 2024-04-06T13:48:17.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.222+0000 7f0797fff700 1 -- 172.21.15.189:0/4260527732 learned_addr learned my addr 172.21.15.189:0/4260527732 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:17.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.222+0000 7f079cbbe700 1 --2- 172.21.15.189:0/4260527732 >> v2:172.21.15.192:3300/0 conn(0x7f079815e500 0x7f079806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:17.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.222+0000 7f0797fff700 1 -- 172.21.15.189:0/4260527732 >> v2:172.21.15.192:3300/0 conn(0x7f079815e500 msgr2=0x7f079806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:17.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.222+0000 7f0797fff700 1 --2- 172.21.15.189:0/4260527732 >> v2:172.21.15.192:3300/0 conn(0x7f079815e500 0x7f079806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:17.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.222+0000 7f0797fff700 1 -- 172.21.15.189:0/4260527732 >> v1:172.21.15.192:6789/0 conn(0x7f079806a730 legacy=0x7f079815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:17.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.222+0000 7f0797fff700 1 -- 172.21.15.189:0/4260527732 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07980b3060 con 0x7f079815d7b0 2024-04-06T13:48:17.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.223+0000 7f0797fff700 1 --2- 172.21.15.189:0/4260527732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079815d7b0 0x7f07980b3e20 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7f07980673e0 tx=0x7f0788009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3a5c6cd02be4bc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:17.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.223+0000 7f07977fe700 1 -- 172.21.15.189:0/4260527732 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0788004230 con 0x7f079815d7b0 2024-04-06T13:48:17.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.223+0000 7f07977fe700 1 -- 172.21.15.189:0/4260527732 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0788004390 con 0x7f079815d7b0 2024-04-06T13:48:17.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.223+0000 7f07977fe700 1 -- 172.21.15.189:0/4260527732 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07880036a0 con 0x7f079815d7b0 2024-04-06T13:48:17.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.223+0000 7f079e621700 1 -- 172.21.15.189:0/4260527732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079815d7b0 msgr2=0x7f07980b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:17.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.223+0000 7f079e621700 1 --2- 172.21.15.189:0/4260527732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079815d7b0 0x7f07980b3e20 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7f07980673e0 tx=0x7f0788009bd0 comp rx=0 tx=0).stop 2024-04-06T13:48:17.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.223+0000 7f079e621700 1 -- 172.21.15.189:0/4260527732 shutdown_connections 2024-04-06T13:48:17.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.223+0000 7f079e621700 1 --2- 172.21.15.189:0/4260527732 >> v2:172.21.15.192:3300/0 conn(0x7f079815e500 0x7f079806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:17.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.223+0000 7f079e621700 1 --2- 172.21.15.189:0/4260527732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079815d7b0 0x7f07980b3e20 unknown :-1 s=CLOSED pgs=2656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:17.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.223+0000 7f079e621700 1 -- 172.21.15.189:0/4260527732 >> 172.21.15.189:0/4260527732 conn(0x7f07980d00b0 msgr2=0x7f07980d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:17.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.223+0000 7f079e621700 1 -- 172.21.15.189:0/4260527732 shutdown_connections 2024-04-06T13:48:17.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.223+0000 7f079e621700 1 -- 172.21.15.189:0/4260527732 wait complete. 2024-04-06T13:48:17.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.225+0000 7f079e621700 1 Processor -- start 2024-04-06T13:48:17.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.225+0000 7f079e621700 1 -- start start 2024-04-06T13:48:17.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.225+0000 7f079e621700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079806a730 0x7f07981f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:17.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.225+0000 7f079e621700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f079815d7b0 0x7f079815c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:17.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.225+0000 7f079e621700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f079815e500 0x7f07981f6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:17.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.225+0000 7f079e621700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07981f6a40 con 0x7f079806a730 2024-04-06T13:48:17.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.225+0000 7f079e621700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07981f6710 con 0x7f079815d7b0 2024-04-06T13:48:17.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.225+0000 7f079e621700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07981f39e0 con 0x7f079815e500 2024-04-06T13:48:17.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.225+0000 7f0797fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079806a730 0x7f07981f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:17.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.225+0000 7f0797fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079806a730 0x7f07981f3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51682/0 (socket says 172.21.15.189:51682) 2024-04-06T13:48:17.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.225+0000 7f0797fff700 1 -- 172.21.15.189:0/745339249 learned_addr learned my addr 172.21.15.189:0/745339249 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:17.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.226+0000 7f0797fff700 1 -- 172.21.15.189:0/745339249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f079815e500 msgr2=0x7f07981f6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:17.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.226+0000 7f079cbbe700 1 --2- 172.21.15.189:0/745339249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f079815e500 0x7f07981f6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:17.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.226+0000 7f078ffff700 1 --2- 172.21.15.189:0/745339249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f079815d7b0 0x7f079815c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:17.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.226+0000 7f0797fff700 1 --2- 172.21.15.189:0/745339249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f079815e500 0x7f07981f6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:17.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.226+0000 7f0797fff700 1 -- 172.21.15.189:0/745339249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f079815d7b0 msgr2=0x7f079815c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:17.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.226+0000 7f0797fff700 1 --2- 172.21.15.189:0/745339249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f079815d7b0 0x7f079815c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:17.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.226+0000 7f0797fff700 1 -- 172.21.15.189:0/745339249 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0788009550 con 0x7f079806a730 2024-04-06T13:48:17.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.226+0000 7f0797fff700 1 --2- 172.21.15.189:0/745339249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079806a730 0x7f07981f3140 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f079806c820 tx=0x7f0788004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:17.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.226+0000 7f07957fa700 1 -- 172.21.15.189:0/745339249 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0788003d80 con 0x7f079806a730 2024-04-06T13:48:17.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.226+0000 7f07957fa700 1 -- 172.21.15.189:0/745339249 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078801f070 con 0x7f079806a730 2024-04-06T13:48:17.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.226+0000 7f07957fa700 1 -- 172.21.15.189:0/745339249 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0788011870 con 0x7f079806a730 2024-04-06T13:48:17.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.226+0000 7f079e621700 1 -- 172.21.15.189:0/745339249 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07981f3cc0 con 0x7f079806a730 2024-04-06T13:48:17.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.226+0000 7f079e621700 1 -- 172.21.15.189:0/745339249 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07981f3680 con 0x7f079806a730 2024-04-06T13:48:17.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.227+0000 7f079e621700 1 -- 172.21.15.189:0/745339249 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0798067bb0 con 0x7f079806a730 2024-04-06T13:48:17.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.227+0000 7f07957fa700 1 -- 172.21.15.189:0/745339249 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f0788021030 con 0x7f079806a730 2024-04-06T13:48:17.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.228+0000 7f07957fa700 1 --2- 172.21.15.189:0/745339249 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f077806aed0 0x7f077806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:17.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.228+0000 7f07957fa700 1 -- 172.21.15.189:0/745339249 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f078808ab20 con 0x7f079806a730 2024-04-06T13:48:17.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.228+0000 7f078ffff700 1 --2- 172.21.15.189:0/745339249 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f077806aed0 0x7f077806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:17.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.228+0000 7f078ffff700 1 --2- 172.21.15.189:0/745339249 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f077806aed0 0x7f077806d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0798068110 tx=0x7f0780009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:17.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.232+0000 7f07957fa700 1 -- 172.21.15.189:0/745339249 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0788055cc0 con 0x7f079806a730 2024-04-06T13:48:17.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:17.425+0000 7f079e621700 1 -- 172.21.15.189:0/745339249 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0798074990 con 0x7f079806a730 2024-04-06T13:48:18.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.257+0000 7f07957fa700 1 -- 172.21.15.189:0/745339249 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v245) v1 ==== 138+0+0 (secure 0 0 0) 0x7f07880290b0 con 0x7f079806a730 2024-04-06T13:48:18.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.261+0000 7f079e621700 1 -- 172.21.15.189:0/745339249 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f077806aed0 msgr2=0x7f077806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:18.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.261+0000 7f079e621700 1 --2- 172.21.15.189:0/745339249 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f077806aed0 0x7f077806d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0798068110 tx=0x7f0780009040 comp rx=0 tx=0).stop 2024-04-06T13:48:18.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.261+0000 7f079e621700 1 -- 172.21.15.189:0/745339249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079806a730 msgr2=0x7f07981f3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:18.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.261+0000 7f079e621700 1 --2- 172.21.15.189:0/745339249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079806a730 0x7f07981f3140 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f079806c820 tx=0x7f0788004750 comp rx=0 tx=0).stop 2024-04-06T13:48:18.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.262+0000 7f079e621700 1 -- 172.21.15.189:0/745339249 shutdown_connections 2024-04-06T13:48:18.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.262+0000 7f079e621700 1 --2- 172.21.15.189:0/745339249 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f077806aed0 0x7f077806d390 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:18.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.262+0000 7f079e621700 1 --2- 172.21.15.189:0/745339249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f079815e500 0x7f07981f6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:18.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.262+0000 7f079e621700 1 --2- 172.21.15.189:0/745339249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079806a730 0x7f07981f3140 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:18.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.262+0000 7f079e621700 1 --2- 172.21.15.189:0/745339249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f079815d7b0 0x7f079815c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:18.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.262+0000 7f079e621700 1 -- 172.21.15.189:0/745339249 >> 172.21.15.189:0/745339249 conn(0x7f07980d00b0 msgr2=0x7f07980af410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:18.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.262+0000 7f079e621700 1 -- 172.21.15.189:0/745339249 shutdown_connections 2024-04-06T13:48:18.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.262+0000 7f079e621700 1 -- 172.21.15.189:0/745339249 wait complete. 2024-04-06T13:48:18.267 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' created 2024-04-06T13:48:18.280 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-06T13:48:18.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.417+0000 7f339e1a8700 1 Processor -- start 2024-04-06T13:48:18.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.417+0000 7f339e1a8700 1 -- start start 2024-04-06T13:48:18.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.418+0000 7f339e1a8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f339815d610 0x7f33980b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:18.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.418+0000 7f339e1a8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f339806a730 0x7f339806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:18.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.418+0000 7f339e1a8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f339815d9f0 con 0x7f3398161100 2024-04-06T13:48:18.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.418+0000 7f339e1a8700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f339806b0f0 con 0x7f339815d610 2024-04-06T13:48:18.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.418+0000 7f339e1a8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33980d1040 con 0x7f339806a730 2024-04-06T13:48:18.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.418+0000 7f33977fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f339815d610 0x7f33980b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:18.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.418+0000 7f3396ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f339806a730 0x7f339806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:18.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.418+0000 7f3396ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f339806a730 0x7f339806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:52848/0 (socket says 172.21.15.189:52848) 2024-04-06T13:48:18.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.418+0000 7f3397fff700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f3398161100 0x7f3398161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:39638/0 (socket says 172.21.15.189:39638) 2024-04-06T13:48:18.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.418+0000 7f33977fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f339815d610 0x7f33980b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:49534/0 (socket says 172.21.15.189:49534) 2024-04-06T13:48:18.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.418+0000 7f3396ffd700 1 -- 172.21.15.189:0/2848025198 learned_addr learned my addr 172.21.15.189:0/2848025198 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:18.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.419+0000 7f3396ffd700 1 -- 172.21.15.189:0/2848025198 >> v2:172.21.15.192:3300/0 conn(0x7f339815d610 msgr2=0x7f33980b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:18.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.419+0000 7f3396ffd700 1 --2- 172.21.15.189:0/2848025198 >> v2:172.21.15.192:3300/0 conn(0x7f339815d610 0x7f33980b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:18.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.419+0000 7f3396ffd700 1 -- 172.21.15.189:0/2848025198 >> v1:172.21.15.192:6789/0 conn(0x7f3398161100 legacy=0x7f3398161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:18.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.419+0000 7f3396ffd700 1 -- 172.21.15.189:0/2848025198 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3398072760 con 0x7f339806a730 2024-04-06T13:48:18.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.419+0000 7f33967fc700 1 -- 172.21.15.189:0/2848025198 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548550910 0 0) 0x7f339815d9f0 con 0x7f3398161100 2024-04-06T13:48:18.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.419+0000 7f33977fe700 1 --2- 172.21.15.189:0/2848025198 >> v2:172.21.15.192:3300/0 conn(0x7f339815d610 0x7f33980b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:18.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.419+0000 7f3396ffd700 1 --2- 172.21.15.189:0/2848025198 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f339806a730 0x7f339806abb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f33880098f0 tx=0x7f3388009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30037423a56e714f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:18.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.419+0000 7f33967fc700 1 -- 172.21.15.189:0/2848025198 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3388011830 con 0x7f339806a730 2024-04-06T13:48:18.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.419+0000 7f33967fc700 1 -- 172.21.15.189:0/2848025198 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3388011990 con 0x7f339806a730 2024-04-06T13:48:18.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.419+0000 7f33967fc700 1 -- 172.21.15.189:0/2848025198 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3388011d20 con 0x7f339806a730 2024-04-06T13:48:18.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.420+0000 7f339e1a8700 1 -- 172.21.15.189:0/2848025198 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f339806a730 msgr2=0x7f339806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:18.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.420+0000 7f339e1a8700 1 --2- 172.21.15.189:0/2848025198 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f339806a730 0x7f339806abb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f33880098f0 tx=0x7f3388009c00 comp rx=0 tx=0).stop 2024-04-06T13:48:18.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.420+0000 7f339e1a8700 1 -- 172.21.15.189:0/2848025198 shutdown_connections 2024-04-06T13:48:18.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.420+0000 7f339e1a8700 1 --2- 172.21.15.189:0/2848025198 >> v2:172.21.15.192:3300/0 conn(0x7f339815d610 0x7f33980b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:18.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.420+0000 7f339e1a8700 1 --2- 172.21.15.189:0/2848025198 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f339806a730 0x7f339806abb0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:18.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.420+0000 7f339e1a8700 1 -- 172.21.15.189:0/2848025198 >> 172.21.15.189:0/2848025198 conn(0x7f33980d00b0 msgr2=0x7f33980d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:18.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.420+0000 7f339e1a8700 1 -- 172.21.15.189:0/2848025198 shutdown_connections 2024-04-06T13:48:18.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.420+0000 7f339e1a8700 1 -- 172.21.15.189:0/2848025198 wait complete. 2024-04-06T13:48:18.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.421+0000 7f339e1a8700 1 Processor -- start 2024-04-06T13:48:18.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.421+0000 7f339e1a8700 1 -- start start 2024-04-06T13:48:18.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.422+0000 7f339e1a8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f339806a730 0x7f339815e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:18.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.422+0000 7f339e1a8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f339815d610 0x7f339815f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:18.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.422+0000 7f339e1a8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3398161100 0x7f339814af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:18.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.422+0000 7f339e1a8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339815e350 con 0x7f339806a730 2024-04-06T13:48:18.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.422+0000 7f339e1a8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339815ed20 con 0x7f339815d610 2024-04-06T13:48:18.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.422+0000 7f339e1a8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339815da90 con 0x7f3398161100 2024-04-06T13:48:18.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.422+0000 7f33977fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f339806a730 0x7f339815e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:18.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.422+0000 7f3396ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f339815d610 0x7f339815f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:18.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.422+0000 7f33977fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f339806a730 0x7f339815e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51688/0 (socket says 172.21.15.189:51688) 2024-04-06T13:48:18.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.422+0000 7f33977fe700 1 -- 172.21.15.189:0/2239191429 learned_addr learned my addr 172.21.15.189:0/2239191429 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:18.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.422+0000 7f3397fff700 1 --2- 172.21.15.189:0/2239191429 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3398161100 0x7f339814af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:18.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.422+0000 7f3396ffd700 1 -- 172.21.15.189:0/2239191429 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3398161100 msgr2=0x7f339814af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:18.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.422+0000 7f3396ffd700 1 --2- 172.21.15.189:0/2239191429 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3398161100 0x7f339814af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:18.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.422+0000 7f3396ffd700 1 -- 172.21.15.189:0/2239191429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f339806a730 msgr2=0x7f339815e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:18.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.422+0000 7f3396ffd700 1 --2- 172.21.15.189:0/2239191429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f339806a730 0x7f339815e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:18.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.422+0000 7f3396ffd700 1 -- 172.21.15.189:0/2239191429 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33800094a0 con 0x7f339815d610 2024-04-06T13:48:18.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.422+0000 7f33977fe700 1 --2- 172.21.15.189:0/2239191429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f339806a730 0x7f339815e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:18.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.423+0000 7f3396ffd700 1 --2- 172.21.15.189:0/2239191429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f339815d610 0x7f339815f2c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f3388016000 tx=0x7f3388004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:18.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.423+0000 7f338ffff700 1 -- 172.21.15.189:0/2239191429 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3388004550 con 0x7f339815d610 2024-04-06T13:48:18.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.423+0000 7f338ffff700 1 -- 172.21.15.189:0/2239191429 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338801f070 con 0x7f339815d610 2024-04-06T13:48:18.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.423+0000 7f339e1a8700 1 -- 172.21.15.189:0/2239191429 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3388009550 con 0x7f339815d610 2024-04-06T13:48:18.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.423+0000 7f338ffff700 1 -- 172.21.15.189:0/2239191429 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33880038e0 con 0x7f339815d610 2024-04-06T13:48:18.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.423+0000 7f339e1a8700 1 -- 172.21.15.189:0/2239191429 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f339815dfe0 con 0x7f339815d610 2024-04-06T13:48:18.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.424+0000 7f339e1a8700 1 -- 172.21.15.189:0/2239191429 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f339815e4c0 con 0x7f339815d610 2024-04-06T13:48:18.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.424+0000 7f338ffff700 1 -- 172.21.15.189:0/2239191429 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f3388003a40 con 0x7f339815d610 2024-04-06T13:48:18.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.425+0000 7f338ffff700 1 --2- 172.21.15.189:0/2239191429 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f337806aed0 0x7f337806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:18.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.425+0000 7f338ffff700 1 -- 172.21.15.189:0/2239191429 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f3388021070 con 0x7f339815d610 2024-04-06T13:48:18.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.425+0000 7f33977fe700 1 --2- 172.21.15.189:0/2239191429 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f337806aed0 0x7f337806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:18.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.425+0000 7f33977fe700 1 --2- 172.21.15.189:0/2239191429 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f337806aed0 0x7f337806d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3380011530 tx=0x7f3380009210 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:18.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.429+0000 7f338ffff700 1 -- 172.21.15.189:0/2239191429 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3388056240 con 0x7f339815d610 2024-04-06T13:48:18.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:18.623+0000 7f339e1a8700 1 -- 172.21.15.189:0/2239191429 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f3398074990 con 0x7f339815d610 2024-04-06T13:48:19.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.272+0000 7f338ffff700 1 -- 172.21.15.189:0/2239191429 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v246) v1 ==== 170+0+0 (secure 0 0 0) 0x7f3388059860 con 0x7f339815d610 2024-04-06T13:48:19.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.279+0000 7f339e1a8700 1 -- 172.21.15.189:0/2239191429 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f337806aed0 msgr2=0x7f337806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:19.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.279+0000 7f339e1a8700 1 --2- 172.21.15.189:0/2239191429 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f337806aed0 0x7f337806d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3380011530 tx=0x7f3380009210 comp rx=0 tx=0).stop 2024-04-06T13:48:19.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.279+0000 7f339e1a8700 1 -- 172.21.15.189:0/2239191429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f339815d610 msgr2=0x7f339815f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:19.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.279+0000 7f339e1a8700 1 --2- 172.21.15.189:0/2239191429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f339815d610 0x7f339815f2c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f3388016000 tx=0x7f3388004750 comp rx=0 tx=0).stop 2024-04-06T13:48:19.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.280+0000 7f339e1a8700 1 -- 172.21.15.189:0/2239191429 shutdown_connections 2024-04-06T13:48:19.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.280+0000 7f339e1a8700 1 --2- 172.21.15.189:0/2239191429 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f337806aed0 0x7f337806d390 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:19.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.280+0000 7f339e1a8700 1 --2- 172.21.15.189:0/2239191429 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3398161100 0x7f339814af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:19.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.280+0000 7f339e1a8700 1 --2- 172.21.15.189:0/2239191429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f339806a730 0x7f339815e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:19.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.280+0000 7f339e1a8700 1 --2- 172.21.15.189:0/2239191429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f339815d610 0x7f339815f2c0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:19.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.280+0000 7f339e1a8700 1 -- 172.21.15.189:0/2239191429 >> 172.21.15.189:0/2239191429 conn(0x7f33980d00b0 msgr2=0x7f33980b1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:19.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.280+0000 7f339e1a8700 1 -- 172.21.15.189:0/2239191429 shutdown_connections 2024-04-06T13:48:19.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.282+0000 7f339e1a8700 1 -- 172.21.15.189:0/2239191429 wait complete. 2024-04-06T13:48:19.284 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' created 2024-04-06T13:48:19.296 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-06T13:48:19.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.422+0000 7f89b2fff700 1 Processor -- start 2024-04-06T13:48:19.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.422+0000 7f89b2fff700 1 -- start start 2024-04-06T13:48:19.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.423+0000 7f89b2fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f89ac15d7b0 0x7f89ac0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:19.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.423+0000 7f89b2fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89ac06a730 0x7f89ac06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:19.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.423+0000 7f89b2fff700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89ac0b3990 con 0x7f89ac15ddf0 2024-04-06T13:48:19.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.423+0000 7f89b2fff700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ac06b0f0 con 0x7f89ac15d7b0 2024-04-06T13:48:19.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.423+0000 7f89b2fff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac0b3060 con 0x7f89ac06a730 2024-04-06T13:48:19.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.423+0000 7f89b0d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f89ac15d7b0 0x7f89ac0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:19.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.423+0000 7f89b0d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f89ac15d7b0 0x7f89ac0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44962/0 (socket says 172.21.15.189:44962) 2024-04-06T13:48:19.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.423+0000 7f89b0d9b700 1 -- 172.21.15.189:0/277740686 learned_addr learned my addr 172.21.15.189:0/277740686 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:19.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.423+0000 7f89abfff700 1 --2- 172.21.15.189:0/277740686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89ac06a730 0x7f89ac06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:19.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.423+0000 7f89b0d9b700 1 -- 172.21.15.189:0/277740686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89ac06a730 msgr2=0x7f89ac06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:19.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.423+0000 7f89b0d9b700 1 --2- 172.21.15.189:0/277740686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89ac06a730 0x7f89ac06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:19.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.423+0000 7f89b0d9b700 1 -- 172.21.15.189:0/277740686 >> v1:172.21.15.192:6789/0 conn(0x7f89ac15ddf0 legacy=0x7f89ac15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:19.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.423+0000 7f89b0d9b700 1 -- 172.21.15.189:0/277740686 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ac0b3340 con 0x7f89ac15d7b0 2024-04-06T13:48:19.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.423+0000 7f89b0d9b700 1 --2- 172.21.15.189:0/277740686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89ac15d7b0 0x7f89ac0b3e20 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f89a00098f0 tx=0x7f89a0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46bc56c952db4041 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:19.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.424+0000 7f89ab7fe700 1 -- 172.21.15.189:0/277740686 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a0004230 con 0x7f89ac15d7b0 2024-04-06T13:48:19.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.424+0000 7f89ab7fe700 1 -- 172.21.15.189:0/277740686 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a0004390 con 0x7f89ac15d7b0 2024-04-06T13:48:19.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.424+0000 7f89ab7fe700 1 -- 172.21.15.189:0/277740686 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a00036a0 con 0x7f89ac15d7b0 2024-04-06T13:48:19.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.424+0000 7f89b2fff700 1 -- 172.21.15.189:0/277740686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89ac15d7b0 msgr2=0x7f89ac0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:19.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.424+0000 7f89b2fff700 1 --2- 172.21.15.189:0/277740686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89ac15d7b0 0x7f89ac0b3e20 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f89a00098f0 tx=0x7f89a0009c00 comp rx=0 tx=0).stop 2024-04-06T13:48:19.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.424+0000 7f89b2fff700 1 -- 172.21.15.189:0/277740686 shutdown_connections 2024-04-06T13:48:19.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.424+0000 7f89b2fff700 1 --2- 172.21.15.189:0/277740686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89ac15d7b0 0x7f89ac0b3e20 unknown :-1 s=CLOSED pgs=2658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:19.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.424+0000 7f89b2fff700 1 --2- 172.21.15.189:0/277740686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89ac06a730 0x7f89ac06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:19.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.424+0000 7f89b2fff700 1 -- 172.21.15.189:0/277740686 >> 172.21.15.189:0/277740686 conn(0x7f89ac0d00b0 msgr2=0x7f89ac0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:19.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.424+0000 7f89b2fff700 1 -- 172.21.15.189:0/277740686 shutdown_connections 2024-04-06T13:48:19.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.424+0000 7f89b2fff700 1 -- 172.21.15.189:0/277740686 wait complete. 2024-04-06T13:48:19.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.425+0000 7f89b2fff700 1 Processor -- start 2024-04-06T13:48:19.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.425+0000 7f89b2fff700 1 -- start start 2024-04-06T13:48:19.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.426+0000 7f89b2fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89ac06a730 0x7f89ac151710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:19.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.426+0000 7f89b2fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89ac15d7b0 0x7f89ac0cc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:19.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.426+0000 7f89b2fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89ac15ddf0 0x7f89ac0cca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:19.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.426+0000 7f89b2fff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac153200 con 0x7f89ac15d7b0 2024-04-06T13:48:19.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.426+0000 7f89b2fff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac1534e0 con 0x7f89ac15ddf0 2024-04-06T13:48:19.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.426+0000 7f89b2fff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac1537c0 con 0x7f89ac06a730 2024-04-06T13:48:19.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.426+0000 7f89abfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89ac15d7b0 0x7f89ac0cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:19.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.426+0000 7f89abfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89ac15d7b0 0x7f89ac0cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44966/0 (socket says 172.21.15.189:44966) 2024-04-06T13:48:19.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.426+0000 7f89abfff700 1 -- 172.21.15.189:0/3814133323 learned_addr learned my addr 172.21.15.189:0/3814133323 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:19.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.426+0000 7f89b0d9b700 1 --2- 172.21.15.189:0/3814133323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89ac06a730 0x7f89ac151710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:19.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.426+0000 7f89b159c700 1 --2- 172.21.15.189:0/3814133323 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89ac15ddf0 0x7f89ac0cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:19.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.427+0000 7f89abfff700 1 -- 172.21.15.189:0/3814133323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89ac06a730 msgr2=0x7f89ac151710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:19.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.427+0000 7f89abfff700 1 --2- 172.21.15.189:0/3814133323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89ac06a730 0x7f89ac151710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:19.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.427+0000 7f89abfff700 1 -- 172.21.15.189:0/3814133323 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89ac15ddf0 msgr2=0x7f89ac0cca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:19.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.427+0000 7f89abfff700 1 --2- 172.21.15.189:0/3814133323 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89ac15ddf0 0x7f89ac0cca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:19.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.427+0000 7f89abfff700 1 -- 172.21.15.189:0/3814133323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a0009550 con 0x7f89ac15d7b0 2024-04-06T13:48:19.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.427+0000 7f89abfff700 1 --2- 172.21.15.189:0/3814133323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89ac15d7b0 0x7f89ac0cc500 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f899800e810 tx=0x7f899800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:19.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.427+0000 7f89a97fa700 1 -- 172.21.15.189:0/3814133323 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899800c070 con 0x7f89ac15d7b0 2024-04-06T13:48:19.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.427+0000 7f89a97fa700 1 -- 172.21.15.189:0/3814133323 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8998013610 con 0x7f89ac15d7b0 2024-04-06T13:48:19.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.427+0000 7f89b2fff700 1 -- 172.21.15.189:0/3814133323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89ac153aa0 con 0x7f89ac15d7b0 2024-04-06T13:48:19.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.427+0000 7f89a97fa700 1 -- 172.21.15.189:0/3814133323 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89980139c0 con 0x7f89ac15d7b0 2024-04-06T13:48:19.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.427+0000 7f89b2fff700 1 -- 172.21.15.189:0/3814133323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89ac154330 con 0x7f89ac15d7b0 2024-04-06T13:48:19.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.428+0000 7f89b2fff700 1 -- 172.21.15.189:0/3814133323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89ac067bb0 con 0x7f89ac15d7b0 2024-04-06T13:48:19.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.429+0000 7f89a97fa700 1 -- 172.21.15.189:0/3814133323 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f8998013b20 con 0x7f89ac15d7b0 2024-04-06T13:48:19.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.429+0000 7f89a97fa700 1 --2- 172.21.15.189:0/3814133323 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f899006aed0 0x7f899006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:19.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.429+0000 7f89a97fa700 1 -- 172.21.15.189:0/3814133323 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f8998089730 con 0x7f89ac15d7b0 2024-04-06T13:48:19.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.429+0000 7f89b0d9b700 1 --2- 172.21.15.189:0/3814133323 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f899006aed0 0x7f899006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:19.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.429+0000 7f89b0d9b700 1 --2- 172.21.15.189:0/3814133323 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f899006aed0 0x7f899006d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f89a0009130 tx=0x7f89a0008030 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:19.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.433+0000 7f89a97fa700 1 -- 172.21.15.189:0/3814133323 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f89980542e0 con 0x7f89ac15d7b0 2024-04-06T13:48:19.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.673+0000 7f89b2fff700 1 -- 172.21.15.189:0/3814133323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f89ac074990 con 0x7f89ac15d7b0 2024-04-06T13:48:19.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.680+0000 7f89a97fa700 1 -- 172.21.15.189:0/3814133323 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 31 and data pool 32 v112) v1 ==== 174+0+0 (secure 0 0 0) 0x7f8998057900 con 0x7f89ac15d7b0 2024-04-06T13:48:19.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.686+0000 7f89b2fff700 1 -- 172.21.15.189:0/3814133323 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f899006aed0 msgr2=0x7f899006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:19.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.686+0000 7f89b2fff700 1 --2- 172.21.15.189:0/3814133323 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f899006aed0 0x7f899006d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f89a0009130 tx=0x7f89a0008030 comp rx=0 tx=0).stop 2024-04-06T13:48:19.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.687+0000 7f89b2fff700 1 -- 172.21.15.189:0/3814133323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89ac15d7b0 msgr2=0x7f89ac0cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:19.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.687+0000 7f89b2fff700 1 --2- 172.21.15.189:0/3814133323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89ac15d7b0 0x7f89ac0cc500 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f899800e810 tx=0x7f899800eb20 comp rx=0 tx=0).stop 2024-04-06T13:48:19.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.687+0000 7f89b2fff700 1 -- 172.21.15.189:0/3814133323 shutdown_connections 2024-04-06T13:48:19.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.687+0000 7f89b2fff700 1 --2- 172.21.15.189:0/3814133323 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f899006aed0 0x7f899006d390 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f89a0009130 tx=0x7f89a0008030 comp rx=0 tx=0).stop 2024-04-06T13:48:19.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.687+0000 7f89b2fff700 1 --2- 172.21.15.189:0/3814133323 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89ac15ddf0 0x7f89ac0cca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:19.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.687+0000 7f89b2fff700 1 --2- 172.21.15.189:0/3814133323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89ac06a730 0x7f89ac151710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:19.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.687+0000 7f89b2fff700 1 --2- 172.21.15.189:0/3814133323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89ac15d7b0 0x7f89ac0cc500 secure :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f899800e810 tx=0x7f899800eb20 comp rx=0 tx=0).stop 2024-04-06T13:48:19.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.687+0000 7f89b2fff700 1 -- 172.21.15.189:0/3814133323 >> 172.21.15.189:0/3814133323 conn(0x7f89ac0d00b0 msgr2=0x7f89ac0ace30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:19.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.689+0000 7f89b2fff700 1 -- 172.21.15.189:0/3814133323 shutdown_connections 2024-04-06T13:48:19.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.693+0000 7f89b2fff700 1 -- 172.21.15.189:0/3814133323 wait complete. 2024-04-06T13:48:19.696 INFO:teuthology.orchestra.run.smithi189.stderr:new fs with metadata pool 31 and data pool 32 2024-04-06T13:48:19.713 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:48:19.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.834+0000 7f99841e6700 1 Processor -- start 2024-04-06T13:48:19.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.835+0000 7f99841e6700 1 -- start start 2024-04-06T13:48:19.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.835+0000 7f99841e6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f997c15a550 0x7f997c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:19.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.835+0000 7f99841e6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f997c0cf900 0x7f997c0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:19.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.835+0000 7f99841e6700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f997c0cdbe0 con 0x7f997c0b4ba0 2024-04-06T13:48:19.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.836+0000 7f99841e6700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f997c0b5010 con 0x7f997c15a550 2024-04-06T13:48:19.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.836+0000 7f99841e6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997c15a940 con 0x7f997c0cf900 2024-04-06T13:48:19.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.836+0000 7f9981f82700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f997c15a550 0x7f997c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:19.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.836+0000 7f9981f82700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f997c15a550 0x7f997c0cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44974/0 (socket says 172.21.15.189:44974) 2024-04-06T13:48:19.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.836+0000 7f9981f82700 1 -- 172.21.15.189:0/1398791416 learned_addr learned my addr 172.21.15.189:0/1398791416 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:19.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.836+0000 7f9981781700 1 --2- 172.21.15.189:0/1398791416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f997c0cf900 0x7f997c0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:19.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.836+0000 7f9981f82700 1 -- 172.21.15.189:0/1398791416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f997c0cf900 msgr2=0x7f997c0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:19.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.836+0000 7f9981f82700 1 --2- 172.21.15.189:0/1398791416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f997c0cf900 0x7f997c0b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:19.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.836+0000 7f9981f82700 1 -- 172.21.15.189:0/1398791416 >> v1:172.21.15.192:6789/0 conn(0x7f997c0b4ba0 legacy=0x7f997c06eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:19.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.836+0000 7f9981f82700 1 -- 172.21.15.189:0/1398791416 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997c15ac00 con 0x7f997c15a550 2024-04-06T13:48:19.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.836+0000 7f9981f82700 1 --2- 172.21.15.189:0/1398791416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f997c15a550 0x7f997c0cf3c0 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7f99780098f0 tx=0x7f9978009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70319450cd3c0dbe server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:19.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.836+0000 7f9980f80700 1 -- 172.21.15.189:0/1398791416 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997800b8a0 con 0x7f997c15a550 2024-04-06T13:48:19.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.836+0000 7f9980f80700 1 -- 172.21.15.189:0/1398791416 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997800ba00 con 0x7f997c15a550 2024-04-06T13:48:19.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.836+0000 7f9980f80700 1 -- 172.21.15.189:0/1398791416 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997800bd90 con 0x7f997c15a550 2024-04-06T13:48:19.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.837+0000 7f99841e6700 1 -- 172.21.15.189:0/1398791416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f997c15a550 msgr2=0x7f997c0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:19.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.837+0000 7f99841e6700 1 --2- 172.21.15.189:0/1398791416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f997c15a550 0x7f997c0cf3c0 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7f99780098f0 tx=0x7f9978009c00 comp rx=0 tx=0).stop 2024-04-06T13:48:19.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.837+0000 7f99841e6700 1 -- 172.21.15.189:0/1398791416 shutdown_connections 2024-04-06T13:48:19.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.837+0000 7f99841e6700 1 --2- 172.21.15.189:0/1398791416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f997c15a550 0x7f997c0cf3c0 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:19.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.837+0000 7f99841e6700 1 --2- 172.21.15.189:0/1398791416 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f997c0cf900 0x7f997c0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:19.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.837+0000 7f99841e6700 1 -- 172.21.15.189:0/1398791416 >> 172.21.15.189:0/1398791416 conn(0x7f997c0d00b0 msgr2=0x7f997c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:19.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.837+0000 7f99841e6700 1 -- 172.21.15.189:0/1398791416 shutdown_connections 2024-04-06T13:48:19.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.837+0000 7f99841e6700 1 -- 172.21.15.189:0/1398791416 wait complete. 2024-04-06T13:48:19.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.838+0000 7f99841e6700 1 Processor -- start 2024-04-06T13:48:19.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.838+0000 7f99841e6700 1 -- start start 2024-04-06T13:48:19.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.839+0000 7f99841e6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f997c0b4ba0 0x7f997c14ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:19.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.839+0000 7f99841e6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f997c0cf900 0x7f997c0b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:19.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.839+0000 7f99841e6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f997c159220 0x7f997c1596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:19.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.839+0000 7f99841e6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997c14b2c0 con 0x7f997c159220 2024-04-06T13:48:19.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.839+0000 7f99841e6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997c149d90 con 0x7f997c0b4ba0 2024-04-06T13:48:19.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.839+0000 7f99841e6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997c14a8f0 con 0x7f997c0cf900 2024-04-06T13:48:19.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.839+0000 7f9982783700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f997c159220 0x7f997c1596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:19.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.839+0000 7f9981f82700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f997c0b4ba0 0x7f997c14ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:19.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.839+0000 7f9982783700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f997c159220 0x7f997c1596d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44976/0 (socket says 172.21.15.189:44976) 2024-04-06T13:48:19.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.839+0000 7f9982783700 1 -- 172.21.15.189:0/2178135834 learned_addr learned my addr 172.21.15.189:0/2178135834 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:19.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.839+0000 7f9981781700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f997c0cf900 0x7f997c0b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:19.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.839+0000 7f9982783700 1 -- 172.21.15.189:0/2178135834 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f997c0cf900 msgr2=0x7f997c0b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:19.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.839+0000 7f9982783700 1 --2- 172.21.15.189:0/2178135834 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f997c0cf900 0x7f997c0b0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:19.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.839+0000 7f9982783700 1 -- 172.21.15.189:0/2178135834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f997c0b4ba0 msgr2=0x7f997c14ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:19.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.839+0000 7f9982783700 1 --2- 172.21.15.189:0/2178135834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f997c0b4ba0 0x7f997c14ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:19.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.839+0000 7f9982783700 1 -- 172.21.15.189:0/2178135834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9978009550 con 0x7f997c159220 2024-04-06T13:48:19.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.839+0000 7f9982783700 1 --2- 172.21.15.189:0/2178135834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f997c159220 0x7f997c1596d0 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f997400cc80 tx=0x7f997400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:19.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.840+0000 7f99727fc700 1 -- 172.21.15.189:0/2178135834 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9974003c00 con 0x7f997c159220 2024-04-06T13:48:19.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.840+0000 7f99727fc700 1 -- 172.21.15.189:0/2178135834 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9974003d60 con 0x7f997c159220 2024-04-06T13:48:19.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.840+0000 7f99727fc700 1 -- 172.21.15.189:0/2178135834 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997400aaa0 con 0x7f997c159220 2024-04-06T13:48:19.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.840+0000 7f99841e6700 1 -- 172.21.15.189:0/2178135834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f997c159c10 con 0x7f997c159220 2024-04-06T13:48:19.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.840+0000 7f99841e6700 1 -- 172.21.15.189:0/2178135834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f997c14a2a0 con 0x7f997c159220 2024-04-06T13:48:19.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.841+0000 7f99841e6700 1 -- 172.21.15.189:0/2178135834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f997c067bb0 con 0x7f997c159220 2024-04-06T13:48:19.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.841+0000 7f99727fc700 1 -- 172.21.15.189:0/2178135834 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f997400ac00 con 0x7f997c159220 2024-04-06T13:48:19.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.841+0000 7f99727fc700 1 --2- 172.21.15.189:0/2178135834 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f996006aed0 0x7f996006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:19.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.841+0000 7f99727fc700 1 -- 172.21.15.189:0/2178135834 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f99740898a0 con 0x7f997c159220 2024-04-06T13:48:19.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.842+0000 7f9981f82700 1 --2- 172.21.15.189:0/2178135834 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f996006aed0 0x7f996006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:19.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.842+0000 7f9981f82700 1 --2- 172.21.15.189:0/2178135834 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f996006aed0 0x7f996006d390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9978014040 tx=0x7f9978008030 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:19.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:19.846+0000 7f99727fc700 1 -- 172.21.15.189:0/2178135834 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9974054590 con 0x7f997c159220 2024-04-06T13:48:20.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.043+0000 7f99841e6700 1 -- 172.21.15.189:0/2178135834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f997c074990 con 0x7f997c159220 2024-04-06T13:48:20.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.044+0000 7f99727fc700 1 -- 172.21.15.189:0/2178135834 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v247) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f9974057bb0 con 0x7f997c159220 2024-04-06T13:48:20.044 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:20.044 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":247,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:48:19.673880+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:48:17.447247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-04-06T13:48:18.646229+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:48:20.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.046+0000 7f99841e6700 1 -- 172.21.15.189:0/2178135834 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f996006aed0 msgr2=0x7f996006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:20.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.046+0000 7f99841e6700 1 --2- 172.21.15.189:0/2178135834 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f996006aed0 0x7f996006d390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9978014040 tx=0x7f9978008030 comp rx=0 tx=0).stop 2024-04-06T13:48:20.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.046+0000 7f99841e6700 1 -- 172.21.15.189:0/2178135834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f997c159220 msgr2=0x7f997c1596d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:20.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.046+0000 7f99841e6700 1 --2- 172.21.15.189:0/2178135834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f997c159220 0x7f997c1596d0 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f997400cc80 tx=0x7f997400cf90 comp rx=0 tx=0).stop 2024-04-06T13:48:20.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.046+0000 7f99841e6700 1 -- 172.21.15.189:0/2178135834 shutdown_connections 2024-04-06T13:48:20.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.046+0000 7f99841e6700 1 --2- 172.21.15.189:0/2178135834 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f996006aed0 0x7f996006d390 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:20.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.046+0000 7f99841e6700 1 --2- 172.21.15.189:0/2178135834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f997c159220 0x7f997c1596d0 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:20.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.046+0000 7f99841e6700 1 --2- 172.21.15.189:0/2178135834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f997c0b4ba0 0x7f997c14ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:20.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.046+0000 7f99841e6700 1 --2- 172.21.15.189:0/2178135834 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f997c0cf900 0x7f997c0b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:20.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.046+0000 7f99841e6700 1 -- 172.21.15.189:0/2178135834 >> 172.21.15.189:0/2178135834 conn(0x7f997c0d00b0 msgr2=0x7f997c06ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:20.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.046+0000 7f99841e6700 1 -- 172.21.15.189:0/2178135834 shutdown_connections 2024-04-06T13:48:20.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.046+0000 7f99841e6700 1 -- 172.21.15.189:0/2178135834 wait complete. 2024-04-06T13:48:20.059 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:48:20.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.181+0000 7fc9aac23700 1 Processor -- start 2024-04-06T13:48:20.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.181+0000 7fc9aac23700 1 -- start start 2024-04-06T13:48:20.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.181+0000 7fc9aac23700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc9a4158fe0 0x7fc9a406a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:20.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.181+0000 7fc9aac23700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc9a406ac70 0x7fc9a415bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:20.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.181+0000 7fc9aac23700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc9a415c100 0x7fc9a40717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:20.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.181+0000 7fc9aac23700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a406b0f0 con 0x7fc9a415c100 2024-04-06T13:48:20.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.182+0000 7fc9aac23700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a415f9e0 con 0x7fc9a406ac70 2024-04-06T13:48:20.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.182+0000 7fc9aac23700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a415f0d0 con 0x7fc9a4158fe0 2024-04-06T13:48:20.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.182+0000 7fc9a91c0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc9a415c100 0x7fc9a40717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:20.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.182+0000 7fc9a91c0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc9a415c100 0x7fc9a40717b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44984/0 (socket says 172.21.15.189:44984) 2024-04-06T13:48:20.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.182+0000 7fc9a91c0700 1 -- 172.21.15.189:0/2281833544 learned_addr learned my addr 172.21.15.189:0/2281833544 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:20.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.182+0000 7fc9a89bf700 1 --2- 172.21.15.189:0/2281833544 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc9a4158fe0 0x7fc9a406a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:20.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.182+0000 7fc9a3fff700 1 --2- 172.21.15.189:0/2281833544 >> v2:172.21.15.192:3300/0 conn(0x7fc9a406ac70 0x7fc9a415bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:20.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.182+0000 7fc9a91c0700 1 -- 172.21.15.189:0/2281833544 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc9a4158fe0 msgr2=0x7fc9a406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:20.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.182+0000 7fc9a91c0700 1 --2- 172.21.15.189:0/2281833544 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc9a4158fe0 0x7fc9a406a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:20.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.182+0000 7fc9a91c0700 1 -- 172.21.15.189:0/2281833544 >> v2:172.21.15.192:3300/0 conn(0x7fc9a406ac70 msgr2=0x7fc9a415bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:20.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.182+0000 7fc9a91c0700 1 --2- 172.21.15.189:0/2281833544 >> v2:172.21.15.192:3300/0 conn(0x7fc9a406ac70 0x7fc9a415bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:20.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.182+0000 7fc9a91c0700 1 -- 172.21.15.189:0/2281833544 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a415c8f0 con 0x7fc9a415c100 2024-04-06T13:48:20.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.182+0000 7fc9a3fff700 1 --2- 172.21.15.189:0/2281833544 >> v2:172.21.15.192:3300/0 conn(0x7fc9a406ac70 0x7fc9a415bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:20.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.182+0000 7fc9a91c0700 1 --2- 172.21.15.189:0/2281833544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a415c100 0x7fc9a40717b0 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7fc9a406cb90 tx=0x7fc99c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a19cb491ea7440 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:20.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.182+0000 7fc9a37fe700 1 -- 172.21.15.189:0/2281833544 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c00e070 con 0x7fc9a415c100 2024-04-06T13:48:20.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.182+0000 7fc9a37fe700 1 -- 172.21.15.189:0/2281833544 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99c00b9c0 con 0x7fc9a415c100 2024-04-06T13:48:20.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.182+0000 7fc9a37fe700 1 -- 172.21.15.189:0/2281833544 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c00bd70 con 0x7fc9a415c100 2024-04-06T13:48:20.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.183+0000 7fc9aac23700 1 -- 172.21.15.189:0/2281833544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a415c100 msgr2=0x7fc9a40717b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:20.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.183+0000 7fc9aac23700 1 --2- 172.21.15.189:0/2281833544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a415c100 0x7fc9a40717b0 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7fc9a406cb90 tx=0x7fc99c00b330 comp rx=0 tx=0).stop 2024-04-06T13:48:20.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.183+0000 7fc9aac23700 1 -- 172.21.15.189:0/2281833544 shutdown_connections 2024-04-06T13:48:20.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.183+0000 7fc9aac23700 1 --2- 172.21.15.189:0/2281833544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a415c100 0x7fc9a40717b0 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:20.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.183+0000 7fc9aac23700 1 --2- 172.21.15.189:0/2281833544 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc9a4158fe0 0x7fc9a406a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:20.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.183+0000 7fc9aac23700 1 --2- 172.21.15.189:0/2281833544 >> v2:172.21.15.192:3300/0 conn(0x7fc9a406ac70 0x7fc9a415bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:20.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.183+0000 7fc9aac23700 1 -- 172.21.15.189:0/2281833544 >> 172.21.15.189:0/2281833544 conn(0x7fc9a40d00b0 msgr2=0x7fc9a40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:20.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.183+0000 7fc9aac23700 1 -- 172.21.15.189:0/2281833544 shutdown_connections 2024-04-06T13:48:20.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.183+0000 7fc9aac23700 1 -- 172.21.15.189:0/2281833544 wait complete. 2024-04-06T13:48:20.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.184+0000 7fc9aac23700 1 Processor -- start 2024-04-06T13:48:20.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.184+0000 7fc9aac23700 1 -- start start 2024-04-06T13:48:20.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.185+0000 7fc9aac23700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9a406ac70 0x7fc9a413e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:20.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.185+0000 7fc9aac23700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc9a4158fe0 0x7fc9a41f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:20.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.185+0000 7fc9aac23700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a415c100 0x7fc9a4158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:20.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.185+0000 7fc9aac23700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a413e620 con 0x7fc9a415c100 2024-04-06T13:48:20.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.185+0000 7fc9aac23700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a41f43c0 con 0x7fc9a406ac70 2024-04-06T13:48:20.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.185+0000 7fc9aac23700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a415cd70 con 0x7fc9a4158fe0 2024-04-06T13:48:20.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.185+0000 7fc9a91c0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a415c100 0x7fc9a4158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:20.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.185+0000 7fc9a91c0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a415c100 0x7fc9a4158080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44998/0 (socket says 172.21.15.189:44998) 2024-04-06T13:48:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.185+0000 7fc9a91c0700 1 -- 172.21.15.189:0/589905300 learned_addr learned my addr 172.21.15.189:0/589905300 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.185+0000 7fc9a3fff700 1 --2- 172.21.15.189:0/589905300 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc9a4158fe0 0x7fc9a41f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.185+0000 7fc9a89bf700 1 --2- 172.21.15.189:0/589905300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9a406ac70 0x7fc9a413e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.185+0000 7fc9a91c0700 1 -- 172.21.15.189:0/589905300 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc9a4158fe0 msgr2=0x7fc9a41f3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.185+0000 7fc9a91c0700 1 --2- 172.21.15.189:0/589905300 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc9a4158fe0 0x7fc9a41f3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.185+0000 7fc9a91c0700 1 -- 172.21.15.189:0/589905300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9a406ac70 msgr2=0x7fc9a413e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.185+0000 7fc9a91c0700 1 --2- 172.21.15.189:0/589905300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9a406ac70 0x7fc9a413e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.185+0000 7fc9a91c0700 1 -- 172.21.15.189:0/589905300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99c009b30 con 0x7fc9a415c100 2024-04-06T13:48:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.186+0000 7fc9a91c0700 1 --2- 172.21.15.189:0/589905300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a415c100 0x7fc9a4158080 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00b730 tx=0x7fc99c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.186+0000 7fc9a17fa700 1 -- 172.21.15.189:0/589905300 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c0092e0 con 0x7fc9a415c100 2024-04-06T13:48:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.186+0000 7fc9a17fa700 1 -- 172.21.15.189:0/589905300 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99c004250 con 0x7fc9a415c100 2024-04-06T13:48:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.186+0000 7fc9aac23700 1 -- 172.21.15.189:0/589905300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a41f45f0 con 0x7fc9a415c100 2024-04-06T13:48:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.186+0000 7fc9a17fa700 1 -- 172.21.15.189:0/589905300 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c004600 con 0x7fc9a415c100 2024-04-06T13:48:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.186+0000 7fc9aac23700 1 -- 172.21.15.189:0/589905300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a41f4bb0 con 0x7fc9a415c100 2024-04-06T13:48:20.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.187+0000 7fc9aac23700 1 -- 172.21.15.189:0/589905300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a40d1050 con 0x7fc9a415c100 2024-04-06T13:48:20.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.187+0000 7fc9a17fa700 1 -- 172.21.15.189:0/589905300 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fc99c004760 con 0x7fc9a415c100 2024-04-06T13:48:20.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.188+0000 7fc9a17fa700 1 --2- 172.21.15.189:0/589905300 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc98806aed0 0x7fc98806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:20.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.188+0000 7fc9a17fa700 1 -- 172.21.15.189:0/589905300 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc99c08abf0 con 0x7fc9a415c100 2024-04-06T13:48:20.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.188+0000 7fc9a89bf700 1 --2- 172.21.15.189:0/589905300 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc98806aed0 0x7fc98806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:20.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.188+0000 7fc9a89bf700 1 --2- 172.21.15.189:0/589905300 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc98806aed0 0x7fc98806d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc998009930 tx=0x7fc998008040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:20.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.192+0000 7fc9a17fa700 1 -- 172.21.15.189:0/589905300 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc99c055910 con 0x7fc9a415c100 2024-04-06T13:48:20.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.390+0000 7fc9aac23700 1 -- 172.21.15.189:0/589905300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9a415cbd0 con 0x7fc9a415c100 2024-04-06T13:48:20.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.391+0000 7fc9a17fa700 1 -- 172.21.15.189:0/589905300 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v247) v1 ==== 74+0+16493 (secure 0 0 0) 0x7fc9a415cbd0 con 0x7fc9a415c100 2024-04-06T13:48:20.391 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:20.391 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":247,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:48:19.673880+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:48:17.447247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-04-06T13:48:18.646229+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:48:20.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.393+0000 7fc9aac23700 1 -- 172.21.15.189:0/589905300 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc98806aed0 msgr2=0x7fc98806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:20.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.393+0000 7fc9aac23700 1 --2- 172.21.15.189:0/589905300 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc98806aed0 0x7fc98806d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc998009930 tx=0x7fc998008040 comp rx=0 tx=0).stop 2024-04-06T13:48:20.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.393+0000 7fc9aac23700 1 -- 172.21.15.189:0/589905300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a415c100 msgr2=0x7fc9a4158080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:20.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.393+0000 7fc9aac23700 1 --2- 172.21.15.189:0/589905300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a415c100 0x7fc9a4158080 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00b730 tx=0x7fc99c004ad0 comp rx=0 tx=0).stop 2024-04-06T13:48:20.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.393+0000 7fc9aac23700 1 -- 172.21.15.189:0/589905300 shutdown_connections 2024-04-06T13:48:20.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.393+0000 7fc9aac23700 1 --2- 172.21.15.189:0/589905300 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc98806aed0 0x7fc98806d390 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:20.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.393+0000 7fc9aac23700 1 --2- 172.21.15.189:0/589905300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a415c100 0x7fc9a4158080 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:20.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.393+0000 7fc9aac23700 1 --2- 172.21.15.189:0/589905300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9a406ac70 0x7fc9a413e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:20.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.393+0000 7fc9aac23700 1 --2- 172.21.15.189:0/589905300 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc9a4158fe0 0x7fc9a41f3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:20.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.393+0000 7fc9aac23700 1 -- 172.21.15.189:0/589905300 >> 172.21.15.189:0/589905300 conn(0x7fc9a40d00b0 msgr2=0x7fc9a40ccac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:20.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.393+0000 7fc9aac23700 1 -- 172.21.15.189:0/589905300 shutdown_connections 2024-04-06T13:48:20.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.393+0000 7fc9aac23700 1 -- 172.21.15.189:0/589905300 wait complete. 2024-04-06T13:48:20.406 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-06T13:48:20.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.530+0000 7f9c1eee3700 1 Processor -- start 2024-04-06T13:48:20.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.530+0000 7f9c1eee3700 1 -- start start 2024-04-06T13:48:20.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.531+0000 7f9c1eee3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9c180cfa00 0x7f9c180cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:20.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.531+0000 7f9c1eee3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c180cbd80 con 0x7f9c180d11b0 2024-04-06T13:48:20.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.531+0000 7f9c1eee3700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c180cef20 con 0x7f9c180cb870 2024-04-06T13:48:20.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.531+0000 7f9c1eee3700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c180ccdc0 con 0x7f9c180cfa00 2024-04-06T13:48:20.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.531+0000 7f9c1d480700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9c180cfa00 0x7f9c180cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:20.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.531+0000 7f9c1cc7f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9c180d11b0 0x7f9c180d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33176/0 (socket says 172.21.15.189:33176) 2024-04-06T13:48:20.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.531+0000 7f9c1cc7f700 1 -- 172.21.15.189:0/2915159684 learned_addr learned my addr 172.21.15.189:0/2915159684 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:20.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.532+0000 7f9c1d480700 1 -- 172.21.15.189:0/2915159684 >> v1:172.21.15.192:6789/0 conn(0x7f9c180cb870 legacy=0x7f9c1815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:20.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.532+0000 7f9c1d480700 1 -- 172.21.15.189:0/2915159684 >> v1:172.21.15.189:6789/0 conn(0x7f9c180d11b0 legacy=0x7f9c180d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:20.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.532+0000 7f9c0f7fe700 1 -- 172.21.15.189:0/2915159684 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250192345 0 0) 0x7f9c180cbd80 con 0x7f9c180d11b0 2024-04-06T13:48:20.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.532+0000 7f9c1d480700 1 -- 172.21.15.189:0/2915159684 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c180cf200 con 0x7f9c180cfa00 2024-04-06T13:48:20.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.532+0000 7f9c1d480700 1 --2- 172.21.15.189:0/2915159684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c180cfa00 0x7f9c180cd560 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7f9c10009d90 tx=0x7f9c1000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ffad01896ff61df server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:20.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.532+0000 7f9c0f7fe700 1 -- 172.21.15.189:0/2915159684 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1000e070 con 0x7f9c180cfa00 2024-04-06T13:48:20.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.532+0000 7f9c0f7fe700 1 -- 172.21.15.189:0/2915159684 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1000bbd0 con 0x7f9c180cfa00 2024-04-06T13:48:20.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.532+0000 7f9c0f7fe700 1 -- 172.21.15.189:0/2915159684 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c10012430 con 0x7f9c180cfa00 2024-04-06T13:48:20.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.533+0000 7f9c1eee3700 1 -- 172.21.15.189:0/2915159684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c180cfa00 msgr2=0x7f9c180cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:20.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.533+0000 7f9c1eee3700 1 --2- 172.21.15.189:0/2915159684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c180cfa00 0x7f9c180cd560 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7f9c10009d90 tx=0x7f9c1000b330 comp rx=0 tx=0).stop 2024-04-06T13:48:20.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.533+0000 7f9c1eee3700 1 -- 172.21.15.189:0/2915159684 shutdown_connections 2024-04-06T13:48:20.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.533+0000 7f9c1eee3700 1 --2- 172.21.15.189:0/2915159684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c180cfa00 0x7f9c180cd560 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:20.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.533+0000 7f9c1eee3700 1 -- 172.21.15.189:0/2915159684 >> 172.21.15.189:0/2915159684 conn(0x7f9c180b2db0 msgr2=0x7f9c180b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:20.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.533+0000 7f9c1eee3700 1 -- 172.21.15.189:0/2915159684 shutdown_connections 2024-04-06T13:48:20.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.533+0000 7f9c1eee3700 1 -- 172.21.15.189:0/2915159684 wait complete. 2024-04-06T13:48:20.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.534+0000 7f9c1eee3700 1 Processor -- start 2024-04-06T13:48:20.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.534+0000 7f9c1eee3700 1 -- start start 2024-04-06T13:48:20.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.535+0000 7f9c1eee3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c180cb870 0x7f9c180c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:20.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.535+0000 7f9c1eee3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c180cfa00 0x7f9c18155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:20.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.535+0000 7f9c1eee3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c180d11b0 0x7f9c181579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:20.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.535+0000 7f9c1eee3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c180c5290 con 0x7f9c180cfa00 2024-04-06T13:48:20.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.535+0000 7f9c1eee3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c180c8fb0 con 0x7f9c180d11b0 2024-04-06T13:48:20.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.535+0000 7f9c1eee3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c180c4ef0 con 0x7f9c180cb870 2024-04-06T13:48:20.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.535+0000 7f9c0ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c180cfa00 0x7f9c18155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:20.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.535+0000 7f9c0ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c180cfa00 0x7f9c18155670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45008/0 (socket says 172.21.15.189:45008) 2024-04-06T13:48:20.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.535+0000 7f9c0ffff700 1 -- 172.21.15.189:0/4272534669 learned_addr learned my addr 172.21.15.189:0/4272534669 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:20.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.535+0000 7f9c1cc7f700 1 --2- 172.21.15.189:0/4272534669 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c180cb870 0x7f9c180c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:20.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.535+0000 7f9c1d480700 1 --2- 172.21.15.189:0/4272534669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c180d11b0 0x7f9c181579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:20.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.535+0000 7f9c0ffff700 1 -- 172.21.15.189:0/4272534669 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c180cb870 msgr2=0x7f9c180c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:20.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.535+0000 7f9c0ffff700 1 --2- 172.21.15.189:0/4272534669 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c180cb870 0x7f9c180c82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:20.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.535+0000 7f9c0ffff700 1 -- 172.21.15.189:0/4272534669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c180d11b0 msgr2=0x7f9c181579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:20.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.535+0000 7f9c0ffff700 1 --2- 172.21.15.189:0/4272534669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c180d11b0 0x7f9c181579e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:20.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.535+0000 7f9c0ffff700 1 -- 172.21.15.189:0/4272534669 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c10009a70 con 0x7f9c180cfa00 2024-04-06T13:48:20.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.535+0000 7f9c0ffff700 1 --2- 172.21.15.189:0/4272534669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c180cfa00 0x7f9c18155670 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f9c0800c450 tx=0x7f9c0800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:20.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.535+0000 7f9c0d7fa700 1 -- 172.21.15.189:0/4272534669 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c080052d0 con 0x7f9c180cfa00 2024-04-06T13:48:20.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.536+0000 7f9c0d7fa700 1 -- 172.21.15.189:0/4272534669 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c08005430 con 0x7f9c180cfa00 2024-04-06T13:48:20.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.536+0000 7f9c1eee3700 1 -- 172.21.15.189:0/4272534669 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c181e6600 con 0x7f9c180cfa00 2024-04-06T13:48:20.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.536+0000 7f9c0d7fa700 1 -- 172.21.15.189:0/4272534669 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c080038c0 con 0x7f9c180cfa00 2024-04-06T13:48:20.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.536+0000 7f9c1eee3700 1 -- 172.21.15.189:0/4272534669 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c180c8cd0 con 0x7f9c180cfa00 2024-04-06T13:48:20.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.536+0000 7f9c1eee3700 1 -- 172.21.15.189:0/4272534669 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c180c4d30 con 0x7f9c180cfa00 2024-04-06T13:48:20.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.537+0000 7f9c0d7fa700 1 -- 172.21.15.189:0/4272534669 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f9c08003a20 con 0x7f9c180cfa00 2024-04-06T13:48:20.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.537+0000 7f9c0d7fa700 1 --2- 172.21.15.189:0/4272534669 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9c0406aed0 0x7f9c0406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:20.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.538+0000 7f9c1cc7f700 1 --2- 172.21.15.189:0/4272534669 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9c0406aed0 0x7f9c0406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:20.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.538+0000 7f9c0d7fa700 1 -- 172.21.15.189:0/4272534669 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9c08088b00 con 0x7f9c180cfa00 2024-04-06T13:48:20.539 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.538+0000 7f9c1cc7f700 1 --2- 172.21.15.189:0/4272534669 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9c0406aed0 0x7f9c0406d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9c14007b10 tx=0x7f9c14007a60 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:20.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.542+0000 7f9c0d7fa700 1 -- 172.21.15.189:0/4272534669 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9c080538c0 con 0x7f9c180cfa00 2024-04-06T13:48:20.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:20.780+0000 7f9c1eee3700 1 -- 172.21.15.189:0/4272534669 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9c18074990 con 0x7f9c180cfa00 2024-04-06T13:48:21.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.684+0000 7f9c0d7fa700 1 -- 172.21.15.189:0/4272534669 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v115) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9c080170b0 con 0x7f9c180cfa00 2024-04-06T13:48:21.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.687+0000 7f9c1eee3700 1 -- 172.21.15.189:0/4272534669 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9c0406aed0 msgr2=0x7f9c0406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:21.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.687+0000 7f9c1eee3700 1 --2- 172.21.15.189:0/4272534669 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9c0406aed0 0x7f9c0406d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9c14007b10 tx=0x7f9c14007a60 comp rx=0 tx=0).stop 2024-04-06T13:48:21.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.687+0000 7f9c1eee3700 1 -- 172.21.15.189:0/4272534669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c180cfa00 msgr2=0x7f9c18155670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:21.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.687+0000 7f9c1eee3700 1 --2- 172.21.15.189:0/4272534669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c180cfa00 0x7f9c18155670 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f9c0800c450 tx=0x7f9c0800c760 comp rx=0 tx=0).stop 2024-04-06T13:48:21.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.687+0000 7f9c1eee3700 1 -- 172.21.15.189:0/4272534669 shutdown_connections 2024-04-06T13:48:21.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.687+0000 7f9c1eee3700 1 --2- 172.21.15.189:0/4272534669 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9c0406aed0 0x7f9c0406d390 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:21.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.687+0000 7f9c1eee3700 1 --2- 172.21.15.189:0/4272534669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c180d11b0 0x7f9c181579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:21.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.687+0000 7f9c1eee3700 1 --2- 172.21.15.189:0/4272534669 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c180cb870 0x7f9c180c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:21.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.687+0000 7f9c1eee3700 1 --2- 172.21.15.189:0/4272534669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c180cfa00 0x7f9c18155670 unknown :-1 s=CLOSED pgs=2666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:21.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.687+0000 7f9c1eee3700 1 -- 172.21.15.189:0/4272534669 >> 172.21.15.189:0/4272534669 conn(0x7f9c180b2db0 msgr2=0x7f9c180b5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:21.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.687+0000 7f9c1eee3700 1 -- 172.21.15.189:0/4272534669 shutdown_connections 2024-04-06T13:48:21.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.687+0000 7f9c1eee3700 1 -- 172.21.15.189:0/4272534669 wait complete. 2024-04-06T13:48:21.702 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-06T13:48:21.702 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-06T13:48:21.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.832+0000 7f152a556700 1 Processor -- start 2024-04-06T13:48:21.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.832+0000 7f152a556700 1 -- start start 2024-04-06T13:48:21.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.833+0000 7f152a556700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1524146dd0 0x7f152406d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:21.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.833+0000 7f152a556700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152406c0a0 con 0x7f15240cedf0 2024-04-06T13:48:21.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.833+0000 7f152a556700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152406c380 con 0x7f15240d17a0 2024-04-06T13:48:21.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.833+0000 7f152a556700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f152406beb0 con 0x7f1524146dd0 2024-04-06T13:48:21.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.833+0000 7f1528af3700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f15240cedf0 0x7f15241441f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33190/0 (socket says 172.21.15.189:33190) 2024-04-06T13:48:21.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.833+0000 7f1528af3700 1 -- 172.21.15.189:0/3803814576 learned_addr learned my addr 172.21.15.189:0/3803814576 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:21.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.833+0000 7f15237fe700 1 --2- 172.21.15.189:0/3803814576 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1524146dd0 0x7f152406d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:21.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.833+0000 7f1522ffd700 1 -- 172.21.15.189:0/3803814576 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664152046 0 0) 0x7f152406c0a0 con 0x7f15240cedf0 2024-04-06T13:48:21.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.833+0000 7f1522ffd700 1 -- 172.21.15.189:0/3803814576 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f150c0035b0 con 0x7f15240cedf0 2024-04-06T13:48:21.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.833+0000 7f1522ffd700 1 -- 172.21.15.189:0/3803814576 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082155480 0 0) 0x7f152406c380 con 0x7f15240d17a0 2024-04-06T13:48:21.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.833+0000 7f1522ffd700 1 -- 172.21.15.189:0/3803814576 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f152406c0a0 con 0x7f15240d17a0 2024-04-06T13:48:21.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.833+0000 7f1522ffd700 1 -- 172.21.15.189:0/3803814576 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2828670355 0 0) 0x7f150c0035b0 con 0x7f15240cedf0 2024-04-06T13:48:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.833+0000 7f1522ffd700 1 -- 172.21.15.189:0/3803814576 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1524146dd0 msgr2=0x7f152406d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.833+0000 7f1522ffd700 1 --2- 172.21.15.189:0/3803814576 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1524146dd0 0x7f152406d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.833+0000 7f1522ffd700 1 -- 172.21.15.189:0/3803814576 >> v1:172.21.15.192:6789/0 conn(0x7f15240d17a0 legacy=0x7f152406b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.833+0000 7f1522ffd700 1 -- 172.21.15.189:0/3803814576 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152406c660 con 0x7f15240cedf0 2024-04-06T13:48:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.833+0000 7f1522ffd700 1 -- 172.21.15.189:0/3803814576 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f151c0019f0 con 0x7f15240cedf0 2024-04-06T13:48:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.834+0000 7f1522ffd700 1 -- 172.21.15.189:0/3803814576 >> v1:172.21.15.189:6789/0 conn(0x7f15240cedf0 legacy=0x7f15241441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.834+0000 7f1522ffd700 1 --2- 172.21.15.189:0/3803814576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f150c003bc0 0x7f150c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.834+0000 7f1522ffd700 1 --2- 172.21.15.189:0/3803814576 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1524146dd0 0x7f150c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.834+0000 7f1522ffd700 1 --2- 172.21.15.189:0/3803814576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150c004d90 0x7f150c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.834+0000 7f1522ffd700 1 -- 172.21.15.189:0/3803814576 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150c00bac0 con 0x7f150c004d90 2024-04-06T13:48:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.834+0000 7f1522ffd700 1 -- 172.21.15.189:0/3803814576 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150c00bc30 con 0x7f150c003bc0 2024-04-06T13:48:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.834+0000 7f1522ffd700 1 -- 172.21.15.189:0/3803814576 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f150c00bdf0 con 0x7f1524146dd0 2024-04-06T13:48:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.834+0000 7f1528af3700 1 --2- 172.21.15.189:0/3803814576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150c004d90 0x7f150c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.834+0000 7f1523fff700 1 --2- 172.21.15.189:0/3803814576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f150c003bc0 0x7f150c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:21.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.834+0000 7f15237fe700 1 --2- 172.21.15.189:0/3803814576 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1524146dd0 0x7f150c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:21.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.834+0000 7f1528af3700 1 -- 172.21.15.189:0/3803814576 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1524146dd0 msgr2=0x7f150c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:21.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.834+0000 7f1528af3700 1 --2- 172.21.15.189:0/3803814576 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1524146dd0 0x7f150c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:21.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.834+0000 7f1528af3700 1 -- 172.21.15.189:0/3803814576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f150c003bc0 msgr2=0x7f150c003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:21.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.834+0000 7f1528af3700 1 --2- 172.21.15.189:0/3803814576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f150c003bc0 0x7f150c003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:21.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.834+0000 7f1528af3700 1 -- 172.21.15.189:0/3803814576 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150c0035b0 con 0x7f150c004d90 2024-04-06T13:48:21.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.834+0000 7f1528af3700 1 --2- 172.21.15.189:0/3803814576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150c004d90 0x7f150c005210 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f151c00a710 tx=0x7f150c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=511b6d967b030acc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:21.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.834+0000 7f1522ffd700 1 -- 172.21.15.189:0/3803814576 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151c0072d0 con 0x7f150c004d90 2024-04-06T13:48:21.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.834+0000 7f1522ffd700 1 -- 172.21.15.189:0/3803814576 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151c007430 con 0x7f150c004d90 2024-04-06T13:48:21.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.834+0000 7f1522ffd700 1 -- 172.21.15.189:0/3803814576 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151c0077e0 con 0x7f150c004d90 2024-04-06T13:48:21.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.835+0000 7f152a556700 1 -- 172.21.15.189:0/3803814576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150c004d90 msgr2=0x7f150c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:21.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.835+0000 7f152a556700 1 --2- 172.21.15.189:0/3803814576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150c004d90 0x7f150c005210 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f151c00a710 tx=0x7f150c005880 comp rx=0 tx=0).stop 2024-04-06T13:48:21.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.835+0000 7f1528af3700 1 -- 172.21.15.189:0/3803814576 reap_dead start 2024-04-06T13:48:21.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.835+0000 7f152a556700 1 -- 172.21.15.189:0/3803814576 shutdown_connections 2024-04-06T13:48:21.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.835+0000 7f152a556700 1 -- 172.21.15.189:0/3803814576 >> 172.21.15.189:0/3803814576 conn(0x7f15240b2db0 msgr2=0x7f15240b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:21.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.835+0000 7f152a556700 1 -- 172.21.15.189:0/3803814576 shutdown_connections 2024-04-06T13:48:21.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.835+0000 7f152a556700 1 -- 172.21.15.189:0/3803814576 wait complete. 2024-04-06T13:48:21.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.836+0000 7f152a556700 1 Processor -- start 2024-04-06T13:48:21.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.837+0000 7f152a556700 1 -- start start 2024-04-06T13:48:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f152a556700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1524159650 0x7f15240af170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f152a556700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1524159a30 0x7f15240afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f152a556700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1524161880 0x7f15240b0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f152a556700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15240af980 con 0x7f1524161880 2024-04-06T13:48:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f152a556700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15240af6b0 con 0x7f1524159650 2024-04-06T13:48:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f152a556700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f152415ec10 con 0x7f1524159a30 2024-04-06T13:48:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f1528af3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1524161880 0x7f15240b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f1528af3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1524161880 0x7f15240b0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45022/0 (socket says 172.21.15.189:45022) 2024-04-06T13:48:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f1528af3700 1 -- 172.21.15.189:0/2103259336 learned_addr learned my addr 172.21.15.189:0/2103259336 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f1523fff700 1 --2- 172.21.15.189:0/2103259336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1524159650 0x7f15240af170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f15237fe700 1 --2- 172.21.15.189:0/2103259336 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1524159a30 0x7f15240afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f1528af3700 1 -- 172.21.15.189:0/2103259336 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1524159a30 msgr2=0x7f15240afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:21.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f1528af3700 1 --2- 172.21.15.189:0/2103259336 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1524159a30 0x7f15240afd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f1528af3700 1 -- 172.21.15.189:0/2103259336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1524159650 msgr2=0x7f15240af170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f1528af3700 1 --2- 172.21.15.189:0/2103259336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1524159650 0x7f15240af170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f1528af3700 1 -- 172.21.15.189:0/2103259336 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152415eef0 con 0x7f1524161880 2024-04-06T13:48:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f1528af3700 1 --2- 172.21.15.189:0/2103259336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1524161880 0x7f15240b0290 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f151c002230 tx=0x7f151c001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f1520ff9700 1 -- 172.21.15.189:0/2103259336 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151c0079b0 con 0x7f1524161880 2024-04-06T13:48:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f1520ff9700 1 -- 172.21.15.189:0/2103259336 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151c007b10 con 0x7f1524161880 2024-04-06T13:48:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f1520ff9700 1 -- 172.21.15.189:0/2103259336 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151c012620 con 0x7f1524161880 2024-04-06T13:48:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f152a556700 1 -- 172.21.15.189:0/2103259336 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f152415f1d0 con 0x7f1524161880 2024-04-06T13:48:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.838+0000 7f152a556700 1 -- 172.21.15.189:0/2103259336 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f152415dad0 con 0x7f1524161880 2024-04-06T13:48:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.839+0000 7f152a556700 1 -- 172.21.15.189:0/2103259336 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f152415f6f0 con 0x7f1524161880 2024-04-06T13:48:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.840+0000 7f1520ff9700 1 -- 172.21.15.189:0/2103259336 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f151c01a070 con 0x7f1524161880 2024-04-06T13:48:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.840+0000 7f1520ff9700 1 --2- 172.21.15.189:0/2103259336 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f151006aed0 0x7f151006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.840+0000 7f1520ff9700 1 -- 172.21.15.189:0/2103259336 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f151c08b350 con 0x7f1524161880 2024-04-06T13:48:21.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.840+0000 7f1523fff700 1 --2- 172.21.15.189:0/2103259336 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f151006aed0 0x7f151006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:21.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.841+0000 7f1523fff700 1 --2- 172.21.15.189:0/2103259336 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f151006aed0 0x7f151006d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1518000c00 tx=0x7f1518009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:21.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:21.845+0000 7f1520ff9700 1 -- 172.21.15.189:0/2103259336 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f152415f6f0 con 0x7f1524161880 2024-04-06T13:48:22.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.085+0000 7f152a556700 1 -- 172.21.15.189:0/2103259336 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f152415f6f0 con 0x7f1524161880 2024-04-06T13:48:22.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.688+0000 7f1520ff9700 1 -- 172.21.15.189:0/2103259336 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v116) v1 ==== 122+0+0 (secure 0 0 0) 0x7f152415f6f0 con 0x7f1524161880 2024-04-06T13:48:22.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.691+0000 7f152a556700 1 -- 172.21.15.189:0/2103259336 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f151006aed0 msgr2=0x7f151006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:22.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.691+0000 7f152a556700 1 --2- 172.21.15.189:0/2103259336 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f151006aed0 0x7f151006d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1518000c00 tx=0x7f1518009040 comp rx=0 tx=0).stop 2024-04-06T13:48:22.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.691+0000 7f152a556700 1 -- 172.21.15.189:0/2103259336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1524161880 msgr2=0x7f15240b0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:22.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.691+0000 7f152a556700 1 --2- 172.21.15.189:0/2103259336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1524161880 0x7f15240b0290 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f151c002230 tx=0x7f151c001cb0 comp rx=0 tx=0).stop 2024-04-06T13:48:22.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.691+0000 7f152a556700 1 -- 172.21.15.189:0/2103259336 shutdown_connections 2024-04-06T13:48:22.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.691+0000 7f152a556700 1 --2- 172.21.15.189:0/2103259336 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f151006aed0 0x7f151006d390 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:22.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.691+0000 7f152a556700 1 --2- 172.21.15.189:0/2103259336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1524161880 0x7f15240b0290 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:22.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.691+0000 7f152a556700 1 --2- 172.21.15.189:0/2103259336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1524159650 0x7f15240af170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:22.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.691+0000 7f152a556700 1 --2- 172.21.15.189:0/2103259336 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1524159a30 0x7f15240afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:22.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.691+0000 7f152a556700 1 -- 172.21.15.189:0/2103259336 >> 172.21.15.189:0/2103259336 conn(0x7f15240b2db0 msgr2=0x7f1524145920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:22.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.691+0000 7f152a556700 1 -- 172.21.15.189:0/2103259336 shutdown_connections 2024-04-06T13:48:22.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.691+0000 7f152a556700 1 -- 172.21.15.189:0/2103259336 wait complete. 2024-04-06T13:48:22.704 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-06T13:48:22.705 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:48:22.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.829+0000 7f3ff85eb700 1 Processor -- start 2024-04-06T13:48:22.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.829+0000 7f3ff85eb700 1 -- start start 2024-04-06T13:48:22.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.830+0000 7f3ff85eb700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3ff00b4af0 0x7f3ff00b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:22.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.830+0000 7f3ff85eb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff00cdbe0 con 0x7f3ff015a550 2024-04-06T13:48:22.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.830+0000 7f3ff85eb700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff015a940 con 0x7f3ff00b4660 2024-04-06T13:48:22.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.830+0000 7f3ff85eb700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ff015ac00 con 0x7f3ff00b4af0 2024-04-06T13:48:22.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.830+0000 7f3ff6b88700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3ff00b4af0 0x7f3ff00b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:22.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.830+0000 7f3ff6387700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3ff015a550 0x7f3ff00cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33198/0 (socket says 172.21.15.189:33198) 2024-04-06T13:48:22.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.830+0000 7f3ff6387700 1 -- 172.21.15.189:0/2451990475 learned_addr learned my addr 172.21.15.189:0/2451990475 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:22.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.830+0000 7f3ff6b88700 1 -- 172.21.15.189:0/2451990475 >> v1:172.21.15.192:6789/0 conn(0x7f3ff00b4660 legacy=0x7f3ff00b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:22.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.830+0000 7f3ff6b88700 1 -- 172.21.15.189:0/2451990475 >> v1:172.21.15.189:6789/0 conn(0x7f3ff015a550 legacy=0x7f3ff00cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:22.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.830+0000 7f3ff6b88700 1 -- 172.21.15.189:0/2451990475 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff015aee0 con 0x7f3ff00b4af0 2024-04-06T13:48:22.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.830+0000 7f3ff5385700 1 -- 172.21.15.189:0/2451990475 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345367373 0 0) 0x7f3ff00cdbe0 con 0x7f3ff015a550 2024-04-06T13:48:22.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.830+0000 7f3ff6b88700 1 --2- 172.21.15.189:0/2451990475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ff00b4af0 0x7f3ff00b36d0 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8009d90 tx=0x7f3fe800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d91b69d5c92247b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:22.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.831+0000 7f3ff5385700 1 -- 172.21.15.189:0/2451990475 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe800e070 con 0x7f3ff00b4af0 2024-04-06T13:48:22.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.831+0000 7f3ff5385700 1 -- 172.21.15.189:0/2451990475 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe800bbd0 con 0x7f3ff00b4af0 2024-04-06T13:48:22.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.831+0000 7f3ff5385700 1 -- 172.21.15.189:0/2451990475 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe8012430 con 0x7f3ff00b4af0 2024-04-06T13:48:22.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.831+0000 7f3ff85eb700 1 -- 172.21.15.189:0/2451990475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ff00b4af0 msgr2=0x7f3ff00b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:22.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.831+0000 7f3ff85eb700 1 --2- 172.21.15.189:0/2451990475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ff00b4af0 0x7f3ff00b36d0 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8009d90 tx=0x7f3fe800b330 comp rx=0 tx=0).stop 2024-04-06T13:48:22.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.831+0000 7f3ff85eb700 1 -- 172.21.15.189:0/2451990475 shutdown_connections 2024-04-06T13:48:22.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.831+0000 7f3ff85eb700 1 --2- 172.21.15.189:0/2451990475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ff00b4af0 0x7f3ff00b36d0 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:22.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.831+0000 7f3ff85eb700 1 -- 172.21.15.189:0/2451990475 >> 172.21.15.189:0/2451990475 conn(0x7f3ff00d00b0 msgr2=0x7f3ff00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:22.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.831+0000 7f3ff85eb700 1 -- 172.21.15.189:0/2451990475 shutdown_connections 2024-04-06T13:48:22.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.831+0000 7f3ff85eb700 1 -- 172.21.15.189:0/2451990475 wait complete. 2024-04-06T13:48:22.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.832+0000 7f3ff85eb700 1 Processor -- start 2024-04-06T13:48:22.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.832+0000 7f3ff85eb700 1 -- start start 2024-04-06T13:48:22.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.833+0000 7f3ff85eb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ff00b4660 0x7f3ff0147a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:22.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.833+0000 7f3ff85eb700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ff00b4af0 0x7f3ff00ab160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:22.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.833+0000 7f3ff85eb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ff0145020 0x7f3ff01465c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:22.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.833+0000 7f3ff85eb700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff0158f00 con 0x7f3ff00b4660 2024-04-06T13:48:22.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.833+0000 7f3ff85eb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff0146130 con 0x7f3ff0145020 2024-04-06T13:48:22.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.833+0000 7f3ff85eb700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff00af350 con 0x7f3ff00b4af0 2024-04-06T13:48:22.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.833+0000 7f3ff6387700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ff00b4660 0x7f3ff0147a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:22.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.833+0000 7f3ff6387700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ff00b4660 0x7f3ff0147a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45050/0 (socket says 172.21.15.189:45050) 2024-04-06T13:48:22.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.833+0000 7f3ff6387700 1 -- 172.21.15.189:0/3371268059 learned_addr learned my addr 172.21.15.189:0/3371268059 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:22.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.833+0000 7f3ff6b88700 1 --2- 172.21.15.189:0/3371268059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ff0145020 0x7f3ff01465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:22.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.833+0000 7f3ff5b86700 1 --2- 172.21.15.189:0/3371268059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ff00b4af0 0x7f3ff00ab160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:22.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.834+0000 7f3ff6387700 1 -- 172.21.15.189:0/3371268059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ff00b4af0 msgr2=0x7f3ff00ab160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:22.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.834+0000 7f3ff6387700 1 --2- 172.21.15.189:0/3371268059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ff00b4af0 0x7f3ff00ab160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:22.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.834+0000 7f3ff6387700 1 -- 172.21.15.189:0/3371268059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ff0145020 msgr2=0x7f3ff01465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.834+0000 7f3ff6387700 1 --2- 172.21.15.189:0/3371268059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ff0145020 0x7f3ff01465c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.834+0000 7f3ff6387700 1 -- 172.21.15.189:0/3371268059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe8009a70 con 0x7f3ff00b4660 2024-04-06T13:48:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.834+0000 7f3ff6387700 1 --2- 172.21.15.189:0/3371268059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ff00b4660 0x7f3ff0147a70 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00c580 tx=0x7f3fec00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.834+0000 7f3fe6ffd700 1 -- 172.21.15.189:0/3371268059 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fec004c80 con 0x7f3ff00b4660 2024-04-06T13:48:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.834+0000 7f3fe6ffd700 1 -- 172.21.15.189:0/3371268059 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fec004de0 con 0x7f3ff00b4660 2024-04-06T13:48:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.834+0000 7f3ff85eb700 1 -- 172.21.15.189:0/3371268059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff0147fb0 con 0x7f3ff00b4660 2024-04-06T13:48:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.834+0000 7f3fe6ffd700 1 -- 172.21.15.189:0/3371268059 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fec007310 con 0x7f3ff00b4660 2024-04-06T13:48:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.834+0000 7f3ff85eb700 1 -- 172.21.15.189:0/3371268059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff0145800 con 0x7f3ff00b4660 2024-04-06T13:48:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.835+0000 7f3ff85eb700 1 -- 172.21.15.189:0/3371268059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ff0146410 con 0x7f3ff00b4660 2024-04-06T13:48:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.836+0000 7f3fe6ffd700 1 -- 172.21.15.189:0/3371268059 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f3fec007470 con 0x7f3ff00b4660 2024-04-06T13:48:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.836+0000 7f3fe6ffd700 1 --2- 172.21.15.189:0/3371268059 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3fd406aed0 0x7f3fd406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.836+0000 7f3fe6ffd700 1 -- 172.21.15.189:0/3371268059 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3fec089b30 con 0x7f3ff00b4660 2024-04-06T13:48:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.836+0000 7f3ff5b86700 1 --2- 172.21.15.189:0/3371268059 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3fd406aed0 0x7f3fd406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:22.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.836+0000 7f3ff5b86700 1 --2- 172.21.15.189:0/3371268059 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3fd406aed0 0x7f3fd406d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0158820 tx=0x7f3fe0009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:22.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:22.840+0000 7f3fe6ffd700 1 -- 172.21.15.189:0/3371268059 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3fec054990 con 0x7f3ff00b4660 2024-04-06T13:48:23.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.079+0000 7f3ff85eb700 1 -- 172.21.15.189:0/3371268059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ff0067bb0 con 0x7f3ff00b4660 2024-04-06T13:48:23.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.080+0000 7f3fe6ffd700 1 -- 172.21.15.189:0/3371268059 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+4753 (secure 0 0 0) 0x7f3fec057fb0 con 0x7f3ff00b4660 2024-04-06T13:48:23.080 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:23.080 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":116,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3323503154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3323503154},{"type":"v1","addr":"172.21.15.189:6836","nonce":3323503154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":9383,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3292345173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3292345173},{"type":"v1","addr":"172.21.15.189:6837","nonce":3292345173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":9395,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/455737532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":455737532},{"type":"v1","addr":"172.21.15.192:6835","nonce":455737532}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:48:19.674205+0000","modified":"2024-04-06T13:48:22.682911+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12108},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12108":{"gid":12108,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6833/726816746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":726816746},{"type":"v1","addr":"172.21.15.192:6833","nonce":726816746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-04-06T13:48:23.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.082+0000 7f3ff85eb700 1 -- 172.21.15.189:0/3371268059 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3fd406aed0 msgr2=0x7f3fd406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:23.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.082+0000 7f3ff85eb700 1 --2- 172.21.15.189:0/3371268059 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3fd406aed0 0x7f3fd406d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0158820 tx=0x7f3fe0009040 comp rx=0 tx=0).stop 2024-04-06T13:48:23.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.082+0000 7f3ff85eb700 1 -- 172.21.15.189:0/3371268059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ff00b4660 msgr2=0x7f3ff0147a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:23.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.082+0000 7f3ff85eb700 1 --2- 172.21.15.189:0/3371268059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ff00b4660 0x7f3ff0147a70 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00c580 tx=0x7f3fec00c940 comp rx=0 tx=0).stop 2024-04-06T13:48:23.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.082+0000 7f3ff85eb700 1 -- 172.21.15.189:0/3371268059 shutdown_connections 2024-04-06T13:48:23.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.082+0000 7f3ff85eb700 1 --2- 172.21.15.189:0/3371268059 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3fd406aed0 0x7f3fd406d390 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.082+0000 7f3ff85eb700 1 --2- 172.21.15.189:0/3371268059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ff0145020 0x7f3ff01465c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.082+0000 7f3ff85eb700 1 --2- 172.21.15.189:0/3371268059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ff00b4660 0x7f3ff0147a70 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.082+0000 7f3ff85eb700 1 --2- 172.21.15.189:0/3371268059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3ff00b4af0 0x7f3ff00ab160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.082+0000 7f3ff85eb700 1 -- 172.21.15.189:0/3371268059 >> 172.21.15.189:0/3371268059 conn(0x7f3ff00d00b0 msgr2=0x7f3ff00ce630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:23.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.082+0000 7f3ff85eb700 1 -- 172.21.15.189:0/3371268059 shutdown_connections 2024-04-06T13:48:23.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.082+0000 7f3ff85eb700 1 -- 172.21.15.189:0/3371268059 wait complete. 2024-04-06T13:48:23.083 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 116 2024-04-06T13:48:23.094 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:48:23.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.216+0000 7f8ff7c08700 1 Processor -- start 2024-04-06T13:48:23.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.216+0000 7f8ff7c08700 1 -- start start 2024-04-06T13:48:23.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.217+0000 7f8ff7c08700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8ff015d7b0 0x7f8ff00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.217+0000 7f8ff7c08700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff006a730 0x7f8ff006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.217+0000 7f8ff7c08700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff00b3990 con 0x7f8ff015ddf0 2024-04-06T13:48:23.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.217+0000 7f8ff7c08700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff006b0f0 con 0x7f8ff015d7b0 2024-04-06T13:48:23.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.217+0000 7f8ff7c08700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff00b3060 con 0x7f8ff006a730 2024-04-06T13:48:23.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.217+0000 7f8ff59a4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8ff015d7b0 0x7f8ff00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.217+0000 7f8ff61a5700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8ff015ddf0 0x7f8ff015e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33202/0 (socket says 172.21.15.189:33202) 2024-04-06T13:48:23.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.217+0000 7f8ff61a5700 1 -- 172.21.15.189:0/4080506254 learned_addr learned my addr 172.21.15.189:0/4080506254 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:23.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.217+0000 7f8ff51a3700 1 --2- 172.21.15.189:0/4080506254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff006a730 0x7f8ff006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.218+0000 7f8ff49a2700 1 -- 172.21.15.189:0/4080506254 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274827948 0 0) 0x7f8ff00b3990 con 0x7f8ff015ddf0 2024-04-06T13:48:23.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.218+0000 7f8ff49a2700 1 -- 172.21.15.189:0/4080506254 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fdc0035b0 con 0x7f8ff015ddf0 2024-04-06T13:48:23.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.218+0000 7f8ff59a4700 1 -- 172.21.15.189:0/4080506254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff006a730 msgr2=0x7f8ff006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:23.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.218+0000 7f8ff59a4700 1 --2- 172.21.15.189:0/4080506254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff006a730 0x7f8ff006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.218+0000 7f8ff59a4700 1 -- 172.21.15.189:0/4080506254 >> v1:172.21.15.189:6789/0 conn(0x7f8ff015ddf0 legacy=0x7f8ff015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:23.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.218+0000 7f8ff59a4700 1 -- 172.21.15.189:0/4080506254 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff00b3340 con 0x7f8ff015d7b0 2024-04-06T13:48:23.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.218+0000 7f8ff59a4700 1 --2- 172.21.15.189:0/4080506254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ff015d7b0 0x7f8ff00b3e20 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0009840 tx=0x7f8fe0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c31d01583e15c410 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:23.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.218+0000 7f8ff49a2700 1 -- 172.21.15.189:0/4080506254 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0011830 con 0x7f8ff015d7b0 2024-04-06T13:48:23.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.218+0000 7f8ff49a2700 1 -- 172.21.15.189:0/4080506254 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe0011990 con 0x7f8ff015d7b0 2024-04-06T13:48:23.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.218+0000 7f8ff49a2700 1 -- 172.21.15.189:0/4080506254 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0011d20 con 0x7f8ff015d7b0 2024-04-06T13:48:23.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.218+0000 7f8ff7c08700 1 -- 172.21.15.189:0/4080506254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ff015d7b0 msgr2=0x7f8ff00b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:23.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.218+0000 7f8ff7c08700 1 --2- 172.21.15.189:0/4080506254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ff015d7b0 0x7f8ff00b3e20 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0009840 tx=0x7f8fe0009b50 comp rx=0 tx=0).stop 2024-04-06T13:48:23.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.219+0000 7f8ff7c08700 1 -- 172.21.15.189:0/4080506254 shutdown_connections 2024-04-06T13:48:23.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.219+0000 7f8ff7c08700 1 --2- 172.21.15.189:0/4080506254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ff015d7b0 0x7f8ff00b3e20 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.219+0000 7f8ff7c08700 1 --2- 172.21.15.189:0/4080506254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff006a730 0x7f8ff006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.219+0000 7f8ff7c08700 1 -- 172.21.15.189:0/4080506254 >> 172.21.15.189:0/4080506254 conn(0x7f8ff00d00b0 msgr2=0x7f8ff00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:23.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.219+0000 7f8ff7c08700 1 -- 172.21.15.189:0/4080506254 shutdown_connections 2024-04-06T13:48:23.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.219+0000 7f8ff7c08700 1 -- 172.21.15.189:0/4080506254 wait complete. 2024-04-06T13:48:23.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.220+0000 7f8ff7c08700 1 Processor -- start 2024-04-06T13:48:23.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.220+0000 7f8ff7c08700 1 -- start start 2024-04-06T13:48:23.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.220+0000 7f8ff7c08700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ff006a730 0x7f8ff01f5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.220+0000 7f8ff7c08700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ff015d7b0 0x7f8ff015c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8ff7c08700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff015ddf0 0x7f8ff01f6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8ff7c08700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff01554d0 con 0x7f8ff006a730 2024-04-06T13:48:23.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8ff7c08700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff01f5f70 con 0x7f8ff015d7b0 2024-04-06T13:48:23.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8ff7c08700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff015ad80 con 0x7f8ff015ddf0 2024-04-06T13:48:23.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8ff59a4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ff006a730 0x7f8ff01f5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8ff59a4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ff006a730 0x7f8ff01f5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45058/0 (socket says 172.21.15.189:45058) 2024-04-06T13:48:23.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8ff59a4700 1 -- 172.21.15.189:0/2660487776 learned_addr learned my addr 172.21.15.189:0/2660487776 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:23.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8ff61a5700 1 --2- 172.21.15.189:0/2660487776 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff015ddf0 0x7f8ff01f6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8ff51a3700 1 --2- 172.21.15.189:0/2660487776 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ff015d7b0 0x7f8ff015c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8ff59a4700 1 -- 172.21.15.189:0/2660487776 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff015ddf0 msgr2=0x7f8ff01f6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:23.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8ff59a4700 1 --2- 172.21.15.189:0/2660487776 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff015ddf0 0x7f8ff01f6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8ff59a4700 1 -- 172.21.15.189:0/2660487776 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ff015d7b0 msgr2=0x7f8ff015c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:23.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8ff59a4700 1 --2- 172.21.15.189:0/2660487776 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ff015d7b0 0x7f8ff015c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8ff59a4700 1 -- 172.21.15.189:0/2660487776 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe00094a0 con 0x7f8ff006a730 2024-04-06T13:48:23.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8ff61a5700 1 --2- 172.21.15.189:0/2660487776 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff015ddf0 0x7f8ff01f6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:23.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8ff51a3700 1 --2- 172.21.15.189:0/2660487776 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ff015d7b0 0x7f8ff015c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:23.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8ff59a4700 1 --2- 172.21.15.189:0/2660487776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ff006a730 0x7f8ff01f5a30 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f8fe00090b0 tx=0x7f8fe0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:23.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8fe67fc700 1 -- 172.21.15.189:0/2660487776 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe00036a0 con 0x7f8ff006a730 2024-04-06T13:48:23.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8fe67fc700 1 -- 172.21.15.189:0/2660487776 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe001f070 con 0x7f8ff006a730 2024-04-06T13:48:23.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8ff7c08700 1 -- 172.21.15.189:0/2660487776 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff015afd0 con 0x7f8ff006a730 2024-04-06T13:48:23.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8fe67fc700 1 -- 172.21.15.189:0/2660487776 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0003a00 con 0x7f8ff006a730 2024-04-06T13:48:23.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.221+0000 7f8ff7c08700 1 -- 172.21.15.189:0/2660487776 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff015b590 con 0x7f8ff006a730 2024-04-06T13:48:23.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.222+0000 7f8ff7c08700 1 -- 172.21.15.189:0/2660487776 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff0067bb0 con 0x7f8ff006a730 2024-04-06T13:48:23.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.223+0000 7f8fe67fc700 1 -- 172.21.15.189:0/2660487776 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f8fe0003b60 con 0x7f8ff006a730 2024-04-06T13:48:23.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.223+0000 7f8fe67fc700 1 --2- 172.21.15.189:0/2660487776 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8fd406aed0 0x7f8fd406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.223+0000 7f8fe67fc700 1 -- 172.21.15.189:0/2660487776 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8fe0021070 con 0x7f8ff006a730 2024-04-06T13:48:23.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.223+0000 7f8ff51a3700 1 --2- 172.21.15.189:0/2660487776 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8fd406aed0 0x7f8fd406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.223+0000 7f8ff51a3700 1 --2- 172.21.15.189:0/2660487776 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8fd406aed0 0x7f8fd406d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8ff00cf2b0 tx=0x7f8fec008040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:23.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.227+0000 7f8fe67fc700 1 -- 172.21.15.189:0/2660487776 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8fe0055f30 con 0x7f8ff006a730 2024-04-06T13:48:23.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.423+0000 7f8ff7c08700 1 -- 172.21.15.189:0/2660487776 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ff0074990 con 0x7f8ff006a730 2024-04-06T13:48:23.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.424+0000 7f8fe67fc700 1 -- 172.21.15.189:0/2660487776 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v248) v1 ==== 74+0+16480 (secure 0 0 0) 0x7f8fe0059550 con 0x7f8ff006a730 2024-04-06T13:48:23.425 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:23.425 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":248,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:48:20.676026+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:48:17.447247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-04-06T13:48:18.646229+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:48:23.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.426+0000 7f8ff7c08700 1 -- 172.21.15.189:0/2660487776 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8fd406aed0 msgr2=0x7f8fd406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:23.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.426+0000 7f8ff7c08700 1 --2- 172.21.15.189:0/2660487776 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8fd406aed0 0x7f8fd406d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8ff00cf2b0 tx=0x7f8fec008040 comp rx=0 tx=0).stop 2024-04-06T13:48:23.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.426+0000 7f8ff7c08700 1 -- 172.21.15.189:0/2660487776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ff006a730 msgr2=0x7f8ff01f5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:23.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.426+0000 7f8ff7c08700 1 --2- 172.21.15.189:0/2660487776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ff006a730 0x7f8ff01f5a30 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f8fe00090b0 tx=0x7f8fe0004750 comp rx=0 tx=0).stop 2024-04-06T13:48:23.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.427+0000 7f8ff7c08700 1 -- 172.21.15.189:0/2660487776 shutdown_connections 2024-04-06T13:48:23.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.427+0000 7f8ff7c08700 1 --2- 172.21.15.189:0/2660487776 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8fd406aed0 0x7f8fd406d390 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.427+0000 7f8ff7c08700 1 --2- 172.21.15.189:0/2660487776 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ff015ddf0 0x7f8ff01f6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.427+0000 7f8ff7c08700 1 --2- 172.21.15.189:0/2660487776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ff006a730 0x7f8ff01f5a30 unknown :-1 s=CLOSED pgs=2675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.427+0000 7f8ff7c08700 1 --2- 172.21.15.189:0/2660487776 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ff015d7b0 0x7f8ff015c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.427+0000 7f8ff7c08700 1 -- 172.21.15.189:0/2660487776 >> 172.21.15.189:0/2660487776 conn(0x7f8ff00d00b0 msgr2=0x7f8ff0157110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:23.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.427+0000 7f8ff7c08700 1 -- 172.21.15.189:0/2660487776 shutdown_connections 2024-04-06T13:48:23.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.427+0000 7f8ff7c08700 1 -- 172.21.15.189:0/2660487776 wait complete. 2024-04-06T13:48:23.440 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-06T13:48:23.440 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:48:23.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.562+0000 7f7b64898700 1 Processor -- start 2024-04-06T13:48:23.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.562+0000 7f7b64898700 1 -- start start 2024-04-06T13:48:23.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.562+0000 7f7b64898700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7b6015e8a0 0x7f7b600cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.562+0000 7f7b64898700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7b6015eee0 0x7f7b6015f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.563+0000 7f7b64898700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6015fcd0 con 0x7f7b6015f8a0 2024-04-06T13:48:23.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.563+0000 7f7b64898700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b600669d0 con 0x7f7b6015e8a0 2024-04-06T13:48:23.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.563+0000 7f7b64898700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b6006ede0 con 0x7f7b6015eee0 2024-04-06T13:48:23.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.563+0000 7f7b5e59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7b6015e8a0 0x7f7b600cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.563+0000 7f7b5e59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7b6015e8a0 0x7f7b600cbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45062/0 (socket says 172.21.15.189:45062) 2024-04-06T13:48:23.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.563+0000 7f7b5e59c700 1 -- 172.21.15.189:0/2347121238 learned_addr learned my addr 172.21.15.189:0/2347121238 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:23.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.563+0000 7f7b5dd9b700 1 --2- 172.21.15.189:0/2347121238 >> v2:172.21.15.192:3300/0 conn(0x7f7b6015eee0 0x7f7b6015f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.563+0000 7f7b5e59c700 1 -- 172.21.15.189:0/2347121238 >> v2:172.21.15.192:3300/0 conn(0x7f7b6015eee0 msgr2=0x7f7b6015f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:23.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.563+0000 7f7b5e59c700 1 --2- 172.21.15.189:0/2347121238 >> v2:172.21.15.192:3300/0 conn(0x7f7b6015eee0 0x7f7b6015f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.563+0000 7f7b5e59c700 1 -- 172.21.15.189:0/2347121238 >> v1:172.21.15.192:6789/0 conn(0x7f7b6015f8a0 legacy=0x7f7b6006ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:23.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.563+0000 7f7b5e59c700 1 -- 172.21.15.189:0/2347121238 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6015ec80 con 0x7f7b6015e8a0 2024-04-06T13:48:23.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.563+0000 7f7b5dd9b700 1 --2- 172.21.15.189:0/2347121238 >> v2:172.21.15.192:3300/0 conn(0x7f7b6015eee0 0x7f7b6015f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:23.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.563+0000 7f7b5e59c700 1 --2- 172.21.15.189:0/2347121238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b6015e8a0 0x7f7b600cbc20 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f7b600723a0 tx=0x7f7b48009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=364fd2de0bfb492e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:23.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.563+0000 7f7b5d59a700 1 -- 172.21.15.189:0/2347121238 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b48004230 con 0x7f7b6015e8a0 2024-04-06T13:48:23.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.563+0000 7f7b5d59a700 1 -- 172.21.15.189:0/2347121238 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b48004390 con 0x7f7b6015e8a0 2024-04-06T13:48:23.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.563+0000 7f7b5d59a700 1 -- 172.21.15.189:0/2347121238 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b480036a0 con 0x7f7b6015e8a0 2024-04-06T13:48:23.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.564+0000 7f7b64898700 1 -- 172.21.15.189:0/2347121238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b6015e8a0 msgr2=0x7f7b600cbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:23.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.564+0000 7f7b64898700 1 --2- 172.21.15.189:0/2347121238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b6015e8a0 0x7f7b600cbc20 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f7b600723a0 tx=0x7f7b48009bd0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.564+0000 7f7b64898700 1 -- 172.21.15.189:0/2347121238 shutdown_connections 2024-04-06T13:48:23.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.564+0000 7f7b64898700 1 --2- 172.21.15.189:0/2347121238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b6015e8a0 0x7f7b600cbc20 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.564+0000 7f7b64898700 1 --2- 172.21.15.189:0/2347121238 >> v2:172.21.15.192:3300/0 conn(0x7f7b6015eee0 0x7f7b6015f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.564+0000 7f7b64898700 1 -- 172.21.15.189:0/2347121238 >> 172.21.15.189:0/2347121238 conn(0x7f7b600cc290 msgr2=0x7f7b600cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:23.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.564+0000 7f7b64898700 1 -- 172.21.15.189:0/2347121238 shutdown_connections 2024-04-06T13:48:23.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.564+0000 7f7b64898700 1 -- 172.21.15.189:0/2347121238 wait complete. 2024-04-06T13:48:23.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.565+0000 7f7b64898700 1 Processor -- start 2024-04-06T13:48:23.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.565+0000 7f7b64898700 1 -- start start 2024-04-06T13:48:23.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.566+0000 7f7b64898700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b6015e8a0 0x7f7b600c4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.566+0000 7f7b64898700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b6015eee0 0x7f7b600c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.566+0000 7f7b64898700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b6015f8a0 0x7f7b6006ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.566+0000 7f7b64898700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b600c6e20 con 0x7f7b6015f8a0 2024-04-06T13:48:23.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.566+0000 7f7b64898700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b600c7c40 con 0x7f7b6015e8a0 2024-04-06T13:48:23.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.566+0000 7f7b64898700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b600c7030 con 0x7f7b6015eee0 2024-04-06T13:48:23.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.566+0000 7f7b5ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b6015f8a0 0x7f7b6006ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.566+0000 7f7b5ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b6015f8a0 0x7f7b6006ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45066/0 (socket says 172.21.15.189:45066) 2024-04-06T13:48:23.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.566+0000 7f7b5ed9d700 1 -- 172.21.15.189:0/6398001 learned_addr learned my addr 172.21.15.189:0/6398001 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:23.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.566+0000 7f7b5dd9b700 1 --2- 172.21.15.189:0/6398001 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b6015eee0 0x7f7b600c68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.566+0000 7f7b5e59c700 1 --2- 172.21.15.189:0/6398001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b6015e8a0 0x7f7b600c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.566+0000 7f7b5ed9d700 1 -- 172.21.15.189:0/6398001 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b6015eee0 msgr2=0x7f7b600c68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:23.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.566+0000 7f7b5ed9d700 1 --2- 172.21.15.189:0/6398001 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b6015eee0 0x7f7b600c68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.566+0000 7f7b5ed9d700 1 -- 172.21.15.189:0/6398001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b6015e8a0 msgr2=0x7f7b600c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:23.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.566+0000 7f7b5ed9d700 1 --2- 172.21.15.189:0/6398001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b6015e8a0 0x7f7b600c4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.566+0000 7f7b5ed9d700 1 -- 172.21.15.189:0/6398001 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b48009550 con 0x7f7b6015f8a0 2024-04-06T13:48:23.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.567+0000 7f7b5ed9d700 1 --2- 172.21.15.189:0/6398001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b6015f8a0 0x7f7b6006ac80 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800b970 tx=0x7f7b5800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:23.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.567+0000 7f7b56ffd700 1 -- 172.21.15.189:0/6398001 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5800d860 con 0x7f7b6015f8a0 2024-04-06T13:48:23.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.567+0000 7f7b56ffd700 1 -- 172.21.15.189:0/6398001 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5800d9c0 con 0x7f7b6015f8a0 2024-04-06T13:48:23.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.567+0000 7f7b64898700 1 -- 172.21.15.189:0/6398001 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b600c7310 con 0x7f7b6015f8a0 2024-04-06T13:48:23.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.567+0000 7f7b56ffd700 1 -- 172.21.15.189:0/6398001 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5800dd70 con 0x7f7b6015f8a0 2024-04-06T13:48:23.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.567+0000 7f7b64898700 1 -- 172.21.15.189:0/6398001 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b600c78d0 con 0x7f7b6015f8a0 2024-04-06T13:48:23.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.568+0000 7f7b64898700 1 -- 172.21.15.189:0/6398001 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b60072140 con 0x7f7b6015f8a0 2024-04-06T13:48:23.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.568+0000 7f7b56ffd700 1 -- 172.21.15.189:0/6398001 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f7b5800e430 con 0x7f7b6015f8a0 2024-04-06T13:48:23.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.569+0000 7f7b56ffd700 1 --2- 172.21.15.189:0/6398001 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7b4006aed0 0x7f7b4006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.569+0000 7f7b56ffd700 1 -- 172.21.15.189:0/6398001 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7b58089ad0 con 0x7f7b6015f8a0 2024-04-06T13:48:23.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.569+0000 7f7b5e59c700 1 --2- 172.21.15.189:0/6398001 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7b4006aed0 0x7f7b4006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.569+0000 7f7b5e59c700 1 --2- 172.21.15.189:0/6398001 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7b4006aed0 0x7f7b4006d390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7b48009130 tx=0x7f7b48008030 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:23.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.573+0000 7f7b56ffd700 1 -- 172.21.15.189:0/6398001 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7b58054750 con 0x7f7b6015f8a0 2024-04-06T13:48:23.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.764+0000 7f7b64898700 1 -- 172.21.15.189:0/6398001 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7b6006ea70 con 0x7f7b4006aed0 2024-04-06T13:48:23.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.772+0000 7f7b56ffd700 1 -- 172.21.15.189:0/6398001 <== mgr.11733 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+279219 (secure 0 0 0) 0x7f7b48004500 con 0x7f7b4006aed0 2024-04-06T13:48:23.777 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:23.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.776+0000 7f7b64898700 1 -- 172.21.15.189:0/6398001 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7b4006aed0 msgr2=0x7f7b4006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:23.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.776+0000 7f7b64898700 1 --2- 172.21.15.189:0/6398001 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7b4006aed0 0x7f7b4006d390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7b48009130 tx=0x7f7b48008030 comp rx=0 tx=0).stop 2024-04-06T13:48:23.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.776+0000 7f7b64898700 1 -- 172.21.15.189:0/6398001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b6015f8a0 msgr2=0x7f7b6006ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:23.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.776+0000 7f7b64898700 1 --2- 172.21.15.189:0/6398001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b6015f8a0 0x7f7b6006ac80 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800b970 tx=0x7f7b5800bc80 comp rx=0 tx=0).stop 2024-04-06T13:48:23.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.776+0000 7f7b64898700 1 -- 172.21.15.189:0/6398001 shutdown_connections 2024-04-06T13:48:23.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.776+0000 7f7b64898700 1 --2- 172.21.15.189:0/6398001 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7b4006aed0 0x7f7b4006d390 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.776+0000 7f7b64898700 1 --2- 172.21.15.189:0/6398001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b6015f8a0 0x7f7b6006ac80 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.776+0000 7f7b64898700 1 --2- 172.21.15.189:0/6398001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b6015e8a0 0x7f7b600c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.776+0000 7f7b64898700 1 --2- 172.21.15.189:0/6398001 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b6015eee0 0x7f7b600c68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.776+0000 7f7b64898700 1 -- 172.21.15.189:0/6398001 >> 172.21.15.189:0/6398001 conn(0x7f7b600cc290 msgr2=0x7f7b6015a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:23.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.776+0000 7f7b64898700 1 -- 172.21.15.189:0/6398001 shutdown_connections 2024-04-06T13:48:23.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.776+0000 7f7b64898700 1 -- 172.21.15.189:0/6398001 wait complete. 2024-04-06T13:48:23.781 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:48:23.788 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-04-06T13:48:21.927356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":437368,"kb_used_data":25232,"kb_used_omap":12,"kb_used_meta":412019,"kb_avail":754537352,"statfs":{"total":773094113280,"available":772646248448,"internally_reserved":0,"allocated":25837568,"data_stored":13829150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":421908024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000633"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.266186+0000","last_change":"2024-04-06T13:48:18.266186+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.266186+0000","last_undegraded":"2024-04-06T13:48:18.266186+0000","last_fullsized":"2024-04-06T13:48:18.266186+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:54:56.369118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.265879+0000","last_change":"2024-04-06T13:48:18.265879+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.265879+0000","last_undegraded":"2024-04-06T13:48:18.265879+0000","last_fullsized":"2024-04-06T13:48:18.265879+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:43.765754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.264137+0000","last_change":"2024-04-06T13:48:18.264137+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.264137+0000","last_undegraded":"2024-04-06T13:48:18.264137+0000","last_fullsized":"2024-04-06T13:48:18.264137+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:30:23.776677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262863+0000","last_change":"2024-04-06T13:48:18.262863+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262863+0000","last_undegraded":"2024-04-06T13:48:18.262863+0000","last_fullsized":"2024-04-06T13:48:18.262863+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:18.513042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.263814+0000","last_change":"2024-04-06T13:48:18.263814+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.263814+0000","last_undegraded":"2024-04-06T13:48:18.263814+0000","last_fullsized":"2024-04-06T13:48:18.263814+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:49:23.791293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.261419+0000","last_change":"2024-04-06T13:48:18.261419+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.261419+0000","last_undegraded":"2024-04-06T13:48:18.261419+0000","last_fullsized":"2024-04-06T13:48:18.261419+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:42:35.927858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.265371+0000","last_change":"2024-04-06T13:48:18.265371+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.265371+0000","last_undegraded":"2024-04-06T13:48:18.265371+0000","last_fullsized":"2024-04-06T13:48:18.265371+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:02:13.092175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.263434+0000","last_change":"2024-04-06T13:48:18.263434+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.263434+0000","last_undegraded":"2024-04-06T13:48:18.263434+0000","last_fullsized":"2024-04-06T13:48:18.263434+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:00:55.130902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.260986+0000","last_change":"2024-04-06T13:48:18.260986+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.260986+0000","last_undegraded":"2024-04-06T13:48:18.260986+0000","last_fullsized":"2024-04-06T13:48:18.260986+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:31:43.320734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262594+0000","last_change":"2024-04-06T13:48:18.262594+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262594+0000","last_undegraded":"2024-04-06T13:48:18.262594+0000","last_fullsized":"2024-04-06T13:48:18.262594+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:06:05.488014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.265322+0000","last_change":"2024-04-06T13:48:18.265322+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.265322+0000","last_undegraded":"2024-04-06T13:48:18.265322+0000","last_fullsized":"2024-04-06T13:48:18.265322+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:56:55.759595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262385+0000","last_change":"2024-04-06T13:48:18.262385+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262385+0000","last_undegraded":"2024-04-06T13:48:18.262385+0000","last_fullsized":"2024-04-06T13:48:18.262385+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:31:24.701348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.261860+0000","last_change":"2024-04-06T13:48:18.261860+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.261860+0000","last_undegraded":"2024-04-06T13:48:18.261860+0000","last_fullsized":"2024-04-06T13:48:18.261860+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:35:17.934513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.263085+0000","last_change":"2024-04-06T13:48:18.263085+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.263085+0000","last_undegraded":"2024-04-06T13:48:18.263085+0000","last_fullsized":"2024-04-06T13:48:18.263085+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:02:23.692226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.259116+0000","last_change":"2024-04-06T13:48:18.259116+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.259116+0000","last_undegraded":"2024-04-06T13:48:18.259116+0000","last_fullsized":"2024-04-06T13:48:18.259116+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:31:15.493970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.265775+0000","last_change":"2024-04-06T13:48:18.265775+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.265775+0000","last_undegraded":"2024-04-06T13:48:18.265775+0000","last_fullsized":"2024-04-06T13:48:18.265775+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:49.041735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.259098+0000","last_change":"2024-04-06T13:48:18.259098+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.259098+0000","last_undegraded":"2024-04-06T13:48:18.259098+0000","last_fullsized":"2024-04-06T13:48:18.259098+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:11:29.081483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.265340+0000","last_change":"2024-04-06T13:48:18.265340+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.265340+0000","last_undegraded":"2024-04-06T13:48:18.265340+0000","last_fullsized":"2024-04-06T13:48:18.265340+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:48.100692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262279+0000","last_change":"2024-04-06T13:48:18.262279+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262279+0000","last_undegraded":"2024-04-06T13:48:18.262279+0000","last_fullsized":"2024-04-06T13:48:18.262279+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:55:10.767276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.260926+0000","last_change":"2024-04-06T13:48:18.260926+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.260926+0000","last_undegraded":"2024-04-06T13:48:18.260926+0000","last_fullsized":"2024-04-06T13:48:18.260926+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:19:26.867548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.264796+0000","last_change":"2024-04-06T13:48:18.264796+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.264796+0000","last_undegraded":"2024-04-06T13:48:18.264796+0000","last_fullsized":"2024-04-06T13:48:18.264796+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:10:25.844832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.265757+0000","last_change":"2024-04-06T13:48:18.265757+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.265757+0000","last_undegraded":"2024-04-06T13:48:18.265757+0000","last_fullsized":"2024-04-06T13:48:18.265757+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:35.170771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.263362+0000","last_change":"2024-04-06T13:48:18.263362+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.263362+0000","last_undegraded":"2024-04-06T13:48:18.263362+0000","last_fullsized":"2024-04-06T13:48:18.263362+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:46:06.202483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.260920+0000","last_change":"2024-04-06T13:48:18.260920+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.260920+0000","last_undegraded":"2024-04-06T13:48:18.260920+0000","last_fullsized":"2024-04-06T13:48:18.260920+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:54:44.219906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262063+0000","last_change":"2024-04-06T13:48:18.262063+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262063+0000","last_undegraded":"2024-04-06T13:48:18.262063+0000","last_fullsized":"2024-04-06T13:48:18.262063+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:18.981232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.263402+0000","last_change":"2024-04-06T13:48:18.263402+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.263402+0000","last_undegraded":"2024-04-06T13:48:18.263402+0000","last_fullsized":"2024-04-06T13:48:18.263402+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:23.871607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.263368+0000","last_change":"2024-04-06T13:48:18.263368+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.263368+0000","last_undegraded":"2024-04-06T13:48:18.263368+0000","last_fullsized":"2024-04-06T13:48:18.263368+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:15.157954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.264814+0000","last_change":"2024-04-06T13:48:18.264814+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.264814+0000","last_undegraded":"2024-04-06T13:48:18.264814+0000","last_fullsized":"2024-04-06T13:48:18.264814+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:34.453253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.261299+0000","last_change":"2024-04-06T13:48:18.261299+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.261299+0000","last_undegraded":"2024-04-06T13:48:18.261299+0000","last_fullsized":"2024-04-06T13:48:18.261299+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:51:53.597002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.260925+0000","last_change":"2024-04-06T13:48:18.260925+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.260925+0000","last_undegraded":"2024-04-06T13:48:18.260925+0000","last_fullsized":"2024-04-06T13:48:18.260925+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:42:14.962783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.264896+0000","last_change":"2024-04-06T13:48:18.264896+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.264896+0000","last_undegraded":"2024-04-06T13:48:18.264896+0000","last_fullsized":"2024-04-06T13:48:18.264896+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:44.664354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.261230+0000","last_change":"2024-04-06T13:48:18.261230+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.261230+0000","last_undegraded":"2024-04-06T13:48:18.261230+0000","last_fullsized":"2024-04-06T13:48:18.261230+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:54:25.950801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"233'370","reported_seq":1333,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-06T13:48:15.595868+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:48:15.595868+0000","last_peered":"2024-04-06T13:48:15.595868+0000","last_clean":"2024-04-06T13:48:15.595868+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:48:15.595868+0000","last_undegraded":"2024-04-06T13:48:15.595868+0000","last_fullsized":"2024-04-06T13:48:15.595868+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575992,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56856,"kb_used_data":3596,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314984,"statfs":{"total":96636764160,"available":96578543616,"internally_reserved":0,"allocated":3682304,"data_stored":2174385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57899999999999996,"15min":0.59499999999999997},"min":{"1min":0.35499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.72199999999999998,"5min":0.78400000000000003,"15min":0.82499999999999996},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56799999999999995,"15min":0.58999999999999997},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78500000000000003,"5min":0.80800000000000005,"15min":0.82899999999999996},"last":0.753}]},{"osd":1,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.55000000000000004,"15min":0.57899999999999996},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77200000000000002,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.57699999999999996,"15min":0.59599999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.69899999999999995,"5min":0.83399999999999996,"15min":0.85299999999999998},"last":0.78100000000000003}]},{"osd":2,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.56299999999999994,"15min":0.58499999999999996},"min":{"1min":0.33200000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69399999999999995,"5min":0.81699999999999995,"15min":0.83899999999999997},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.58599999999999997,"15min":0.60099999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69999999999999996,"5min":0.755,"15min":0.79700000000000004},"last":0.76900000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.59399999999999997,"15min":0.60599999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.67200000000000004,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58799999999999997,"15min":0.60299999999999998},"min":{"1min":0.39000000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78900000000000003,"5min":0.82099999999999995,"15min":0.84099999999999997},"last":0.66500000000000004}]},{"osd":4,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45800000000000002,"15min":0.46600000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.81599999999999995},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.36799999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.82699999999999996},"last":0.53500000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.498,"15min":0.498},"min":{"1min":0.315,"5min":0.315,"15min":0.308},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56899999999999995,"15min":0.57799999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33600000000000002},"max":{"1min":0.74099999999999999,"5min":0.80700000000000005,"15min":0.85399999999999998},"last":0.55200000000000005}]},{"osd":6,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.34899999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.879},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57099999999999995,"15min":0.58699999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81799999999999995,"5min":0.84899999999999998,"15min":0.89800000000000002},"last":0.57199999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575991,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56260,"kb_used_data":3000,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315580,"statfs":{"total":96636764160,"available":96579153920,"internally_reserved":0,"allocated":3072000,"data_stored":1576789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.53600000000000003,"15min":0.55700000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69399999999999995,"5min":0.79700000000000004,"15min":0.83799999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.54100000000000004,"15min":0.56499999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.62,"5min":0.76900000000000002,"15min":0.81999999999999995},"last":0.61899999999999999}]},{"osd":1,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.60399999999999998,"15min":0.624},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76000000000000001,"5min":0.81699999999999995,"15min":0.85799999999999998},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.622,"15min":0.64400000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.378},"max":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.82699999999999996},"last":0.67800000000000005}]},{"osd":2,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56999999999999995,"15min":0.59399999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72099999999999997,"5min":0.79000000000000004,"15min":0.82899999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.56299999999999994,"15min":0.58599999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.80700000000000005},"last":0.58499999999999996}]},{"osd":3,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.58599999999999997,"15min":0.60199999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.73199999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.61899999999999999,"15min":0.64300000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.36099999999999999},"max":{"1min":0.80200000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.68899999999999995}]},{"osd":4,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70599999999999996,"5min":0.72299999999999998,"15min":0.81399999999999995},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.51100000000000001,"15min":0.53800000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.66400000000000003,"5min":0.82099999999999995,"15min":0.84099999999999997},"last":0.70799999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.48899999999999999,"15min":0.50600000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.33300000000000002},"max":{"1min":0.63,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52000000000000002,"15min":0.53100000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67400000000000004,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.63500000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53900000000000003,"15min":0.54800000000000004},"min":{"1min":0.34899999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.67700000000000005,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55300000000000005,"15min":0.56899999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.32800000000000001},"max":{"1min":0.71199999999999997,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.54800000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575990,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60372,"kb_used_data":3012,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311468,"statfs":{"total":96636764160,"available":96574943232,"internally_reserved":0,"allocated":3084288,"data_stored":1582998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.38100000000000001,"15min":0.39100000000000001},"min":{"1min":0.248,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.378,"5min":0.621,"15min":0.621},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.41799999999999998,"15min":0.42699999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.501,"5min":0.60799999999999998,"15min":0.69399999999999995},"last":0.41499999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.41299999999999998,"15min":0.42399999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.46899999999999997,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.378,"5min":0.44900000000000001,"15min":0.46400000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.55400000000000005,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.46999999999999997}]},{"osd":3,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.46999999999999997,"15min":0.47999999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.52800000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.42699999999999999,"15min":0.439},"min":{"1min":0.28799999999999998,"5min":0.27600000000000002,"15min":0.26300000000000001},"max":{"1min":0.45500000000000002,"5min":0.63500000000000001,"15min":0.66100000000000003},"last":0.45500000000000002}]},{"osd":4,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.628,"15min":0.64400000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.33800000000000002},"max":{"1min":0.70099999999999996,"5min":0.89900000000000002,"15min":0.97399999999999998},"last":0.377},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.64400000000000002,"15min":0.65500000000000003},"min":{"1min":0.46100000000000002,"5min":0.44500000000000001,"15min":0.44400000000000001},"max":{"1min":0.73499999999999999,"5min":0.94599999999999995,"15min":0.995},"last":0.49099999999999999}]},{"osd":5,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.64100000000000001,"15min":0.65300000000000002},"min":{"1min":0.42999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73899999999999999,"5min":0.96899999999999997,"15min":1.073},"last":0.441},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.65700000000000003,"15min":0.66800000000000004},"min":{"1min":0.45200000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.72799999999999998,"5min":0.96499999999999997,"15min":1.0209999999999999},"last":0.54600000000000004}]},{"osd":6,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.46300000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73899999999999999,"5min":0.89200000000000002,"15min":1},"last":0.503},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.67100000000000004,"15min":0.68400000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.72499999999999998,"5min":0.94599999999999995,"15min":1.036},"last":0.35799999999999998}]},{"osd":7,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.69399999999999995,"15min":0.69499999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85399999999999998,"5min":0.94899999999999995,"15min":1.0649999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65700000000000003,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.76300000000000001,"5min":0.92900000000000005,"15min":1.056},"last":0.48299999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575990,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42900,"kb_used_data":3592,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94328940,"statfs":{"total":96636764160,"available":96592834560,"internally_reserved":0,"allocated":3678208,"data_stored":2173366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.38200000000000001,"15min":0.39300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.45100000000000001,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.39300000000000002,"15min":0.41699999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.48199999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.41399999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.40100000000000002,"15min":0.42499999999999999},"min":{"1min":0.26100000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49399999999999999,"5min":0.66700000000000004,"15min":0.70099999999999996},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.42499999999999999,"15min":0.443},"min":{"1min":0.28399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.47999999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.33300000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.40699999999999997,"15min":0.42499999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.50700000000000001,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.45700000000000002,"15min":0.47799999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.51900000000000002,"5min":0.72999999999999998,"15min":0.75700000000000001},"last":0.442}]},{"osd":4,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.60499999999999998,"15min":0.64100000000000001},"min":{"1min":0.39100000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.80900000000000005,"5min":0.93100000000000005,"15min":1.0329999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.61199999999999999,"15min":0.64300000000000002},"min":{"1min":0.372,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88500000000000001,"5min":0.999,"15min":0.999},"last":0.57899999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.60199999999999998,"15min":0.63900000000000001},"min":{"1min":0.39300000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.0009999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.63800000000000001,"15min":0.66300000000000003},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.749,"5min":0.92800000000000005,"15min":1.052},"last":0.501}]},{"osd":6,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.64500000000000002,"15min":0.67400000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.874,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.64200000000000002,"15min":0.66900000000000004},"min":{"1min":0.40500000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77500000000000002,"5min":0.96399999999999997,"15min":1.085},"last":0.67600000000000005}]},{"osd":7,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.66300000000000003,"15min":0.69099999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.80700000000000005,"5min":1.01,"15min":1.121},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.64700000000000002,"15min":0.67200000000000004},"min":{"1min":0.42199999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.93700000000000006,"5min":0.95799999999999996,"15min":1.0229999999999999},"last":0.69399999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575992,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60368,"kb_used_data":3012,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311472,"statfs":{"total":96636764160,"available":96574947328,"internally_reserved":0,"allocated":3084288,"data_stored":1582998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.376,"15min":0.38400000000000001},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.42299999999999999,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.40600000000000003,"15min":0.42199999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.249},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":1.044},"last":0.28799999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.38500000000000001,"15min":0.40400000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51100000000000001,"5min":0.54000000000000004,"15min":0.753},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.42799999999999999,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.82199999999999995},"last":0.36799999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.439,"15min":0.45800000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.53200000000000003,"5min":0.69699999999999995,"15min":0.70299999999999996},"last":0.315},{"interface":"front","average":{"1min":0.377,"5min":0.437,"15min":0.44900000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.504,"5min":0.64500000000000002,"15min":0.68600000000000005},"last":0.32800000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.621,"15min":0.63900000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.376},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.64400000000000002,"15min":0.66100000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.88900000000000001,"5min":1.022,"15min":1.022},"last":0.47199999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.64300000000000002,"15min":0.66300000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88100000000000001,"5min":0.98599999999999999,"15min":1.0149999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.65100000000000002,"15min":0.66700000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.56000000000000005}]},{"osd":6,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.65200000000000002,"15min":0.67200000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92000000000000004,"5min":0.97199999999999998,"15min":1.091},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.67600000000000005,"15min":0.69499999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.96099999999999997},"last":0.57699999999999996}]},{"osd":7,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.66600000000000004,"15min":0.68799999999999994},"min":{"1min":0.42699999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.95399999999999996},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.68300000000000005,"15min":0.70799999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.85299999999999998,"5min":1.01,"15min":1.01},"last":0.52000000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575991,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48084,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323756,"statfs":{"total":96636764160,"available":96587526144,"internally_reserved":0,"allocated":3088384,"data_stored":1584017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39000000000000001,"15min":0.38900000000000001},"min":{"1min":0.26000000000000001,"5min":0.255,"15min":0.23499999999999999},"max":{"1min":0.55200000000000005,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41199999999999998,"15min":0.41299999999999998},"min":{"1min":0.28799999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.56899999999999995,"5min":0.622,"15min":0.66700000000000004},"last":0.34000000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40699999999999997,"15min":0.40600000000000003},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.65700000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.438,"15min":0.44400000000000001},"min":{"1min":0.32800000000000001,"5min":0.27500000000000002,"15min":0.26600000000000001},"max":{"1min":0.63,"5min":0.65400000000000003,"15min":0.75800000000000001},"last":0.39900000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48299999999999998,"15min":0.48599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.5,"15min":0.503},"min":{"1min":0.38600000000000001,"5min":0.34599999999999997,"15min":0.32900000000000001},"max":{"1min":0.67800000000000005,"5min":0.69299999999999995,"15min":0.79300000000000004},"last":0.49199999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.439,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.876,"5min":1.002,"15min":1.018},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66100000000000003,"15min":0.65300000000000002},"min":{"1min":0.497,"5min":0.45900000000000002,"15min":0.375},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.60399999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.51900000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.95699999999999996,"5min":0.97299999999999998,"15min":0.999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.46200000000000002,"5min":0.437,"15min":0.437},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.61499999999999999}]},{"osd":6,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67500000000000004,"15min":0.66900000000000004},"min":{"1min":0.41499999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.63},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.68600000000000005,"15min":0.68400000000000005},"min":{"1min":0.48799999999999999,"5min":0.46700000000000003,"15min":0.41099999999999998},"max":{"1min":0.88800000000000001,"5min":0.96199999999999997,"15min":0.98799999999999999},"last":0.57399999999999995}]},{"osd":7,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68500000000000005,"15min":0.68000000000000005},"min":{"1min":0.46100000000000002,"5min":0.432,"15min":0.40999999999999998},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70299999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.024},"last":0.71199999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575991,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48076,"kb_used_data":3004,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323764,"statfs":{"total":96636764160,"available":96587534336,"internally_reserved":0,"allocated":3076096,"data_stored":1577808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55900000000000005,"15min":0.58699999999999997},"min":{"1min":0.40699999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.753,"5min":0.876,"15min":0.88500000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.39300000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81100000000000005,"5min":1.202,"15min":1.202},"last":0.48299999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.61299999999999999,"15min":0.64000000000000001},"min":{"1min":0.40799999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71699999999999997,"5min":1.256,"15min":1.256},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.624,"15min":0.63},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.75900000000000001,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.56499999999999995}]},{"osd":2,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.57999999999999996,"15min":0.59999999999999998},"min":{"1min":0.41599999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77500000000000002,"5min":0.84599999999999997,"15min":0.877},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.60599999999999998,"15min":0.61599999999999999},"min":{"1min":0.39600000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.97899999999999998,"5min":1.294,"15min":1.294},"last":0.41199999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84499999999999997,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.60899999999999999,"15min":0.626},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.78800000000000003,"5min":0.90200000000000002,"15min":0.91100000000000003},"last":0.55400000000000005}]},{"osd":5,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55500000000000005,"15min":0.52000000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.92800000000000005,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.56399999999999995,"15min":0.54100000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.95199999999999996,"5min":4.931,"15min":4.931},"last":0.63500000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53500000000000003,"15min":0.53300000000000003},"min":{"1min":0.34999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.505},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34699999999999998},"max":{"1min":1.0609999999999999,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.62}]},{"osd":7,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59499999999999997,"15min":0.58199999999999996},"min":{"1min":0.36899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.91000000000000003,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59399999999999997,"15min":0.60199999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.373},"max":{"1min":0.86899999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.53200000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575991,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64452,"kb_used_data":3000,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307388,"statfs":{"total":96636764160,"available":96570765312,"internally_reserved":0,"allocated":3072000,"data_stored":1576789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.57399999999999995,"15min":0.58499999999999996},"min":{"1min":0.36399999999999999,"5min":0.35899999999999999,"15min":0.33000000000000002},"max":{"1min":0.68000000000000005,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58099999999999996,"15min":0.59399999999999997},"min":{"1min":0.38100000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.76600000000000001,"5min":0.84699999999999998,"15min":0.89200000000000002},"last":0.40200000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.58099999999999996,"15min":0.59399999999999997},"min":{"1min":0.38400000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79100000000000004,"5min":0.81299999999999994,"15min":0.81399999999999995},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.439,"5min":0.39400000000000002,"15min":0.37},"max":{"1min":0.70299999999999996,"5min":0.871,"15min":0.871},"last":0.41599999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59299999999999997,"15min":0.59699999999999998},"min":{"1min":0.311,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69199999999999995,"5min":0.79700000000000004,"15min":0.84399999999999997},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58799999999999997,"15min":0.59599999999999997},"min":{"1min":0.44,"5min":0.378,"15min":0.378},"max":{"1min":0.73099999999999998,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.56599999999999995}]},{"osd":3,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.79800000000000004,"5min":0.81100000000000005,"15min":0.94499999999999995},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61699999999999999,"15min":0.63200000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.74099999999999999,"5min":0.88200000000000001,"15min":0.91500000000000004},"last":0.71199999999999997}]},{"osd":4,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52900000000000003,"15min":0.50700000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.27400000000000002},"max":{"1min":0.70999999999999996,"5min":0.85699999999999998,"15min":1.0369999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.59999999999999998,"15min":0.57699999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.308},"max":{"1min":0.77800000000000002,"5min":0.88800000000000001,"15min":1.1639999999999999},"last":0.65700000000000003}]},{"osd":6,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54300000000000004,"15min":0.52600000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.27900000000000003},"max":{"1min":0.89100000000000001,"5min":0.94999999999999996,"15min":1.0169999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54300000000000004,"15min":0.53100000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.32300000000000001},"max":{"1min":0.81899999999999995,"5min":0.878,"15min":1.0009999999999999},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56499999999999995,"15min":0.54800000000000004},"min":{"1min":0.39200000000000002,"5min":0.35799999999999998,"15min":0.32900000000000001},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.66400000000000003,"15min":0.64800000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.35199999999999998},"max":{"1min":0.85299999999999998,"5min":0.92300000000000004,"15min":0.96599999999999997},"last":0.68400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:48:23.794 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:48:23.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.952+0000 7f6980a0d700 1 Processor -- start 2024-04-06T13:48:23.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.952+0000 7f6980a0d700 1 -- start start 2024-04-06T13:48:23.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.953+0000 7f6980a0d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f697c15d040 0x7f697c15dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.953+0000 7f6980a0d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697c15d8d0 con 0x7f697c15e2a0 2024-04-06T13:48:23.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.953+0000 7f6980a0d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697c15d580 con 0x7f697c06cbf0 2024-04-06T13:48:23.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.953+0000 7f6980a0d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697c0cba40 con 0x7f697c15d040 2024-04-06T13:48:23.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.953+0000 7f6979d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f697c15e2a0 0x7f697c06c4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33214/0 (socket says 172.21.15.189:33214) 2024-04-06T13:48:23.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.953+0000 7f6979d9b700 1 -- 172.21.15.189:0/4088840604 learned_addr learned my addr 172.21.15.189:0/4088840604 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:23.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697959a700 1 -- 172.21.15.189:0/4088840604 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356151108 0 0) 0x7f697c15d8d0 con 0x7f697c15e2a0 2024-04-06T13:48:23.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697a59c700 1 --2- 172.21.15.189:0/4088840604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f697c15d040 0x7f697c15dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697959a700 1 -- 172.21.15.189:0/4088840604 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69640035b0 con 0x7f697c15e2a0 2024-04-06T13:48:23.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697959a700 1 -- 172.21.15.189:0/4088840604 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2542164826 0 0) 0x7f69640035b0 con 0x7f697c15e2a0 2024-04-06T13:48:23.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697959a700 1 -- 172.21.15.189:0/4088840604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f697c15d040 msgr2=0x7f697c15dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:23.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697959a700 1 --2- 172.21.15.189:0/4088840604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f697c15d040 0x7f697c15dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697959a700 1 -- 172.21.15.189:0/4088840604 >> v1:172.21.15.192:6789/0 conn(0x7f697c06cbf0 legacy=0x7f697c0cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:23.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697959a700 1 -- 172.21.15.189:0/4088840604 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697c0b3240 con 0x7f697c15e2a0 2024-04-06T13:48:23.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697959a700 1 -- 172.21.15.189:0/4088840604 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f696c001670 con 0x7f697c15e2a0 2024-04-06T13:48:23.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697959a700 1 -- 172.21.15.189:0/4088840604 >> v1:172.21.15.189:6789/0 conn(0x7f697c15e2a0 legacy=0x7f697c06c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:23.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697959a700 1 --2- 172.21.15.189:0/4088840604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6964003a30 0x7f6964003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697959a700 1 --2- 172.21.15.189:0/4088840604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f697c15d040 0x7f6964004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697959a700 1 --2- 172.21.15.189:0/4088840604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6964004d90 0x7f6964005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697959a700 1 -- 172.21.15.189:0/4088840604 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696400bac0 con 0x7f6964004d90 2024-04-06T13:48:23.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697959a700 1 -- 172.21.15.189:0/4088840604 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696400bc00 con 0x7f6964003a30 2024-04-06T13:48:23.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697959a700 1 -- 172.21.15.189:0/4088840604 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696400bdc0 con 0x7f697c15d040 2024-04-06T13:48:23.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697ad9d700 1 --2- 172.21.15.189:0/4088840604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6964004d90 0x7f6964005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697a59c700 1 --2- 172.21.15.189:0/4088840604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6964003a30 0x7f6964003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f6979d9b700 1 --2- 172.21.15.189:0/4088840604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f697c15d040 0x7f6964004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697ad9d700 1 -- 172.21.15.189:0/4088840604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f697c15d040 msgr2=0x7f6964004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:23.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697ad9d700 1 --2- 172.21.15.189:0/4088840604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f697c15d040 0x7f6964004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697ad9d700 1 -- 172.21.15.189:0/4088840604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6964003a30 msgr2=0x7f6964003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:23.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697ad9d700 1 --2- 172.21.15.189:0/4088840604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6964003a30 0x7f6964003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.954+0000 7f697ad9d700 1 -- 172.21.15.189:0/4088840604 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69640035b0 con 0x7f6964004d90 2024-04-06T13:48:23.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.955+0000 7f697ad9d700 1 --2- 172.21.15.189:0/4088840604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6964004d90 0x7f6964005210 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7f697c0b2c80 tx=0x7f6964005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33fbae88d1145b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:23.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.955+0000 7f697959a700 1 -- 172.21.15.189:0/4088840604 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6974001df0 con 0x7f6964004d90 2024-04-06T13:48:23.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.955+0000 7f697959a700 1 -- 172.21.15.189:0/4088840604 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f697400cd40 con 0x7f6964004d90 2024-04-06T13:48:23.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.955+0000 7f697959a700 1 -- 172.21.15.189:0/4088840604 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6974003c40 con 0x7f6964004d90 2024-04-06T13:48:23.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.955+0000 7f6980a0d700 1 -- 172.21.15.189:0/4088840604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6964004d90 msgr2=0x7f6964005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:23.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.955+0000 7f6980a0d700 1 --2- 172.21.15.189:0/4088840604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6964004d90 0x7f6964005210 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7f697c0b2c80 tx=0x7f6964005880 comp rx=0 tx=0).stop 2024-04-06T13:48:23.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.955+0000 7f697ad9d700 1 -- 172.21.15.189:0/4088840604 reap_dead start 2024-04-06T13:48:23.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.955+0000 7f6980a0d700 1 -- 172.21.15.189:0/4088840604 shutdown_connections 2024-04-06T13:48:23.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.955+0000 7f6980a0d700 1 -- 172.21.15.189:0/4088840604 >> 172.21.15.189:0/4088840604 conn(0x7f697c06a730 msgr2=0x7f697c06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:23.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.955+0000 7f6980a0d700 1 -- 172.21.15.189:0/4088840604 shutdown_connections 2024-04-06T13:48:23.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.955+0000 7f6980a0d700 1 -- 172.21.15.189:0/4088840604 wait complete. 2024-04-06T13:48:23.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.957+0000 7f6980a0d700 1 Processor -- start 2024-04-06T13:48:23.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.957+0000 7f6980a0d700 1 -- start start 2024-04-06T13:48:23.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.957+0000 7f6980a0d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6964004d90 0x7f697c146660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.957+0000 7f6980a0d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f697c149d30 0x7f697c0cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.957+0000 7f6980a0d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f697c167710 0x7f697c167bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.957+0000 7f6980a0d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697c0cfa20 con 0x7f697c167710 2024-04-06T13:48:23.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.957+0000 7f6980a0d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697c168100 con 0x7f697c149d30 2024-04-06T13:48:23.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.957+0000 7f6980a0d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697c0cd070 con 0x7f6964004d90 2024-04-06T13:48:23.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.958+0000 7f697ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f697c167710 0x7f697c167bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.958+0000 7f697ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f697c167710 0x7f697c167bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45096/0 (socket says 172.21.15.189:45096) 2024-04-06T13:48:23.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.958+0000 7f697ad9d700 1 -- 172.21.15.189:0/1407620642 learned_addr learned my addr 172.21.15.189:0/1407620642 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:23.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.958+0000 7f697a59c700 1 --2- 172.21.15.189:0/1407620642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6964004d90 0x7f697c146660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.958+0000 7f6979d9b700 1 --2- 172.21.15.189:0/1407620642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f697c149d30 0x7f697c0cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.958+0000 7f697ad9d700 1 -- 172.21.15.189:0/1407620642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6964004d90 msgr2=0x7f697c146660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:23.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.958+0000 7f697ad9d700 1 --2- 172.21.15.189:0/1407620642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6964004d90 0x7f697c146660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.958+0000 7f697ad9d700 1 -- 172.21.15.189:0/1407620642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f697c149d30 msgr2=0x7f697c0cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:23.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.958+0000 7f697ad9d700 1 --2- 172.21.15.189:0/1407620642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f697c149d30 0x7f697c0cc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:23.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.958+0000 7f697ad9d700 1 -- 172.21.15.189:0/1407620642 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697c146ba0 con 0x7f697c167710 2024-04-06T13:48:23.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.958+0000 7f697a59c700 1 --2- 172.21.15.189:0/1407620642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6964004d90 0x7f697c146660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:23.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.958+0000 7f6979d9b700 1 --2- 172.21.15.189:0/1407620642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f697c149d30 0x7f697c0cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:23.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.958+0000 7f697ad9d700 1 --2- 172.21.15.189:0/1407620642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f697c167710 0x7f697c167bc0 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f6974004000 tx=0x7f69740014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:23.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.958+0000 7f696affd700 1 -- 172.21.15.189:0/1407620642 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6974003e10 con 0x7f697c167710 2024-04-06T13:48:23.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.958+0000 7f696affd700 1 -- 172.21.15.189:0/1407620642 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69740050d0 con 0x7f697c167710 2024-04-06T13:48:23.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.958+0000 7f6980a0d700 1 -- 172.21.15.189:0/1407620642 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f697c0cf4a0 con 0x7f697c167710 2024-04-06T13:48:23.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.958+0000 7f696affd700 1 -- 172.21.15.189:0/1407620642 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6974005480 con 0x7f697c167710 2024-04-06T13:48:23.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.958+0000 7f6980a0d700 1 -- 172.21.15.189:0/1407620642 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f697c0ccd00 con 0x7f697c167710 2024-04-06T13:48:23.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.959+0000 7f6980a0d700 1 -- 172.21.15.189:0/1407620642 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f697c0ccb60 con 0x7f697c167710 2024-04-06T13:48:23.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.960+0000 7f696affd700 1 -- 172.21.15.189:0/1407620642 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f6974019070 con 0x7f697c167710 2024-04-06T13:48:23.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.960+0000 7f696affd700 1 --2- 172.21.15.189:0/1407620642 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f695c06aed0 0x7f695c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:23.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.960+0000 7f696affd700 1 -- 172.21.15.189:0/1407620642 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f697408aa20 con 0x7f697c167710 2024-04-06T13:48:23.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.960+0000 7f697a59c700 1 --2- 172.21.15.189:0/1407620642 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f695c06aed0 0x7f695c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:23.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.960+0000 7f697a59c700 1 --2- 172.21.15.189:0/1407620642 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f695c06aed0 0x7f695c06d390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6970009c60 tx=0x7f6970008040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:23.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:23.964+0000 7f696affd700 1 -- 172.21.15.189:0/1407620642 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f697c0ccb60 con 0x7f697c167710 2024-04-06T13:48:24.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.157+0000 7f6980a0d700 1 -- 172.21.15.189:0/1407620642 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f697c0ccb60 con 0x7f695c06aed0 2024-04-06T13:48:24.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.164+0000 7f696affd700 1 -- 172.21.15.189:0/1407620642 <== mgr.11733 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286244 (secure 0 0 0) 0x7f697c0ccb60 con 0x7f695c06aed0 2024-04-06T13:48:24.168 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:24.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.167+0000 7f6980a0d700 1 -- 172.21.15.189:0/1407620642 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f695c06aed0 msgr2=0x7f695c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:24.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.167+0000 7f6980a0d700 1 --2- 172.21.15.189:0/1407620642 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f695c06aed0 0x7f695c06d390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6970009c60 tx=0x7f6970008040 comp rx=0 tx=0).stop 2024-04-06T13:48:24.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.167+0000 7f6980a0d700 1 -- 172.21.15.189:0/1407620642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f697c167710 msgr2=0x7f697c167bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:24.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.168+0000 7f6980a0d700 1 --2- 172.21.15.189:0/1407620642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f697c167710 0x7f697c167bc0 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f6974004000 tx=0x7f69740014a0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.168+0000 7f6980a0d700 1 -- 172.21.15.189:0/1407620642 shutdown_connections 2024-04-06T13:48:24.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.168+0000 7f6980a0d700 1 --2- 172.21.15.189:0/1407620642 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f695c06aed0 0x7f695c06d390 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.168+0000 7f6980a0d700 1 --2- 172.21.15.189:0/1407620642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f697c167710 0x7f697c167bc0 unknown :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.168+0000 7f6980a0d700 1 --2- 172.21.15.189:0/1407620642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6964004d90 0x7f697c146660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.168+0000 7f6980a0d700 1 --2- 172.21.15.189:0/1407620642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f697c149d30 0x7f697c0cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.168+0000 7f6980a0d700 1 -- 172.21.15.189:0/1407620642 >> 172.21.15.189:0/1407620642 conn(0x7f697c06a730 msgr2=0x7f697c0b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:24.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.168+0000 7f6980a0d700 1 -- 172.21.15.189:0/1407620642 shutdown_connections 2024-04-06T13:48:24.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.168+0000 7f6980a0d700 1 -- 172.21.15.189:0/1407620642 wait complete. 2024-04-06T13:48:24.171 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:48:24.180 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-04-06T13:48:23.927698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590438,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":645,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":391,"ondisk_log_size":391,"up":148,"acting":148,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":77,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":455668,"kb_used_data":25460,"kb_used_omap":12,"kb_used_meta":430067,"kb_avail":754519052,"statfs":{"total":773094113280,"available":772627509248,"internally_reserved":0,"allocated":26071040,"data_stored":13946484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":440389176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.670934"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.900020+0000","last_change":"2024-04-06T13:48:19.704688+0000","last_active":"2024-04-06T13:48:20.900020+0000","last_peered":"2024-04-06T13:48:20.900020+0000","last_clean":"2024-04-06T13:48:20.900020+0000","last_became_active":"2024-04-06T13:48:19.704088+0000","last_became_peered":"2024-04-06T13:48:19.704088+0000","last_unstale":"2024-04-06T13:48:20.900020+0000","last_undegraded":"2024-04-06T13:48:20.900020+0000","last_fullsized":"2024-04-06T13:48:20.900020+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:49.108653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097823+0000","last_change":"2024-04-06T13:48:19.697461+0000","last_active":"2024-04-06T13:48:21.097823+0000","last_peered":"2024-04-06T13:48:21.097823+0000","last_clean":"2024-04-06T13:48:21.097823+0000","last_became_active":"2024-04-06T13:48:19.695459+0000","last_became_peered":"2024-04-06T13:48:19.695459+0000","last_unstale":"2024-04-06T13:48:21.097823+0000","last_undegraded":"2024-04-06T13:48:21.097823+0000","last_fullsized":"2024-04-06T13:48:21.097823+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:42:12.990594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098046+0000","last_change":"2024-04-06T13:48:19.690589+0000","last_active":"2024-04-06T13:48:21.098046+0000","last_peered":"2024-04-06T13:48:21.098046+0000","last_clean":"2024-04-06T13:48:21.098046+0000","last_became_active":"2024-04-06T13:48:19.690338+0000","last_became_peered":"2024-04-06T13:48:19.690338+0000","last_unstale":"2024-04-06T13:48:21.098046+0000","last_undegraded":"2024-04-06T13:48:21.098046+0000","last_fullsized":"2024-04-06T13:48:21.098046+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:51:42.234400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097813+0000","last_change":"2024-04-06T13:48:19.695665+0000","last_active":"2024-04-06T13:48:21.097813+0000","last_peered":"2024-04-06T13:48:21.097813+0000","last_clean":"2024-04-06T13:48:21.097813+0000","last_became_active":"2024-04-06T13:48:19.695362+0000","last_became_peered":"2024-04-06T13:48:19.695362+0000","last_unstale":"2024-04-06T13:48:21.097813+0000","last_undegraded":"2024-04-06T13:48:21.097813+0000","last_fullsized":"2024-04-06T13:48:21.097813+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:26:21.699641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098450+0000","last_change":"2024-04-06T13:48:19.700300+0000","last_active":"2024-04-06T13:48:21.098450+0000","last_peered":"2024-04-06T13:48:21.098450+0000","last_clean":"2024-04-06T13:48:21.098450+0000","last_became_active":"2024-04-06T13:48:19.700054+0000","last_became_peered":"2024-04-06T13:48:19.700054+0000","last_unstale":"2024-04-06T13:48:21.098450+0000","last_undegraded":"2024-04-06T13:48:21.098450+0000","last_fullsized":"2024-04-06T13:48:21.098450+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:30:14.717681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097799+0000","last_change":"2024-04-06T13:48:19.700151+0000","last_active":"2024-04-06T13:48:21.097799+0000","last_peered":"2024-04-06T13:48:21.097799+0000","last_clean":"2024-04-06T13:48:21.097799+0000","last_became_active":"2024-04-06T13:48:19.699882+0000","last_became_peered":"2024-04-06T13:48:19.699882+0000","last_unstale":"2024-04-06T13:48:21.097799+0000","last_undegraded":"2024-04-06T13:48:21.097799+0000","last_fullsized":"2024-04-06T13:48:21.097799+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:24.879568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683541+0000","last_change":"2024-04-06T13:48:19.690929+0000","last_active":"2024-04-06T13:48:20.683541+0000","last_peered":"2024-04-06T13:48:20.683541+0000","last_clean":"2024-04-06T13:48:20.683541+0000","last_became_active":"2024-04-06T13:48:19.690663+0000","last_became_peered":"2024-04-06T13:48:19.690663+0000","last_unstale":"2024-04-06T13:48:20.683541+0000","last_undegraded":"2024-04-06T13:48:20.683541+0000","last_fullsized":"2024-04-06T13:48:20.683541+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:52:01.901954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097559+0000","last_change":"2024-04-06T13:48:19.693212+0000","last_active":"2024-04-06T13:48:21.097559+0000","last_peered":"2024-04-06T13:48:21.097559+0000","last_clean":"2024-04-06T13:48:21.097559+0000","last_became_active":"2024-04-06T13:48:19.692957+0000","last_became_peered":"2024-04-06T13:48:19.692957+0000","last_unstale":"2024-04-06T13:48:21.097559+0000","last_undegraded":"2024-04-06T13:48:21.097559+0000","last_fullsized":"2024-04-06T13:48:21.097559+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:00:25.538231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097435+0000","last_change":"2024-04-06T13:48:19.700696+0000","last_active":"2024-04-06T13:48:21.097435+0000","last_peered":"2024-04-06T13:48:21.097435+0000","last_clean":"2024-04-06T13:48:21.097435+0000","last_became_active":"2024-04-06T13:48:19.700444+0000","last_became_peered":"2024-04-06T13:48:19.700444+0000","last_unstale":"2024-04-06T13:48:21.097435+0000","last_undegraded":"2024-04-06T13:48:21.097435+0000","last_fullsized":"2024-04-06T13:48:21.097435+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:46.094428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683996+0000","last_change":"2024-04-06T13:48:19.700988+0000","last_active":"2024-04-06T13:48:20.683996+0000","last_peered":"2024-04-06T13:48:20.683996+0000","last_clean":"2024-04-06T13:48:20.683996+0000","last_became_active":"2024-04-06T13:48:19.700718+0000","last_became_peered":"2024-04-06T13:48:19.700718+0000","last_unstale":"2024-04-06T13:48:20.683996+0000","last_undegraded":"2024-04-06T13:48:20.683996+0000","last_fullsized":"2024-04-06T13:48:20.683996+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:37:50.946459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098548+0000","last_change":"2024-04-06T13:48:19.700420+0000","last_active":"2024-04-06T13:48:21.098548+0000","last_peered":"2024-04-06T13:48:21.098548+0000","last_clean":"2024-04-06T13:48:21.098548+0000","last_became_active":"2024-04-06T13:48:19.698705+0000","last_became_peered":"2024-04-06T13:48:19.698705+0000","last_unstale":"2024-04-06T13:48:21.098548+0000","last_undegraded":"2024-04-06T13:48:21.098548+0000","last_fullsized":"2024-04-06T13:48:21.098548+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:29:58.555526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899939+0000","last_change":"2024-04-06T13:48:19.702407+0000","last_active":"2024-04-06T13:48:20.899939+0000","last_peered":"2024-04-06T13:48:20.899939+0000","last_clean":"2024-04-06T13:48:20.899939+0000","last_became_active":"2024-04-06T13:48:19.702139+0000","last_became_peered":"2024-04-06T13:48:19.702139+0000","last_unstale":"2024-04-06T13:48:20.899939+0000","last_undegraded":"2024-04-06T13:48:20.899939+0000","last_fullsized":"2024-04-06T13:48:20.899939+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:33.923465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098117+0000","last_change":"2024-04-06T13:48:19.692817+0000","last_active":"2024-04-06T13:48:21.098117+0000","last_peered":"2024-04-06T13:48:21.098117+0000","last_clean":"2024-04-06T13:48:21.098117+0000","last_became_active":"2024-04-06T13:48:19.692567+0000","last_became_peered":"2024-04-06T13:48:19.692567+0000","last_unstale":"2024-04-06T13:48:21.098117+0000","last_undegraded":"2024-04-06T13:48:21.098117+0000","last_fullsized":"2024-04-06T13:48:21.098117+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:22.432067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683285+0000","last_change":"2024-04-06T13:48:19.688698+0000","last_active":"2024-04-06T13:48:20.683285+0000","last_peered":"2024-04-06T13:48:20.683285+0000","last_clean":"2024-04-06T13:48:20.683285+0000","last_became_active":"2024-04-06T13:48:19.688377+0000","last_became_peered":"2024-04-06T13:48:19.688377+0000","last_unstale":"2024-04-06T13:48:20.683285+0000","last_undegraded":"2024-04-06T13:48:20.683285+0000","last_fullsized":"2024-04-06T13:48:20.683285+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:22.432067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683150+0000","last_change":"2024-04-06T13:48:19.698243+0000","last_active":"2024-04-06T13:48:20.683150+0000","last_peered":"2024-04-06T13:48:20.683150+0000","last_clean":"2024-04-06T13:48:20.683150+0000","last_became_active":"2024-04-06T13:48:19.697956+0000","last_became_peered":"2024-04-06T13:48:19.697956+0000","last_unstale":"2024-04-06T13:48:20.683150+0000","last_undegraded":"2024-04-06T13:48:20.683150+0000","last_fullsized":"2024-04-06T13:48:20.683150+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:26:21.699641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.900099+0000","last_change":"2024-04-06T13:48:19.694103+0000","last_active":"2024-04-06T13:48:20.900099+0000","last_peered":"2024-04-06T13:48:20.900099+0000","last_clean":"2024-04-06T13:48:20.900099+0000","last_became_active":"2024-04-06T13:48:19.693643+0000","last_became_peered":"2024-04-06T13:48:19.693643+0000","last_unstale":"2024-04-06T13:48:20.900099+0000","last_undegraded":"2024-04-06T13:48:20.900099+0000","last_fullsized":"2024-04-06T13:48:20.900099+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:00:35.709952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097980+0000","last_change":"2024-04-06T13:48:19.697522+0000","last_active":"2024-04-06T13:48:21.097980+0000","last_peered":"2024-04-06T13:48:21.097980+0000","last_clean":"2024-04-06T13:48:21.097980+0000","last_became_active":"2024-04-06T13:48:19.695505+0000","last_became_peered":"2024-04-06T13:48:19.695505+0000","last_unstale":"2024-04-06T13:48:21.097980+0000","last_undegraded":"2024-04-06T13:48:21.097980+0000","last_fullsized":"2024-04-06T13:48:21.097980+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:37:50.946459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097790+0000","last_change":"2024-04-06T13:48:19.694038+0000","last_active":"2024-04-06T13:48:21.097790+0000","last_peered":"2024-04-06T13:48:21.097790+0000","last_clean":"2024-04-06T13:48:21.097790+0000","last_became_active":"2024-04-06T13:48:19.693775+0000","last_became_peered":"2024-04-06T13:48:19.693775+0000","last_unstale":"2024-04-06T13:48:21.097790+0000","last_undegraded":"2024-04-06T13:48:21.097790+0000","last_fullsized":"2024-04-06T13:48:21.097790+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:38:06.251591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097641+0000","last_change":"2024-04-06T13:48:19.699860+0000","last_active":"2024-04-06T13:48:21.097641+0000","last_peered":"2024-04-06T13:48:21.097641+0000","last_clean":"2024-04-06T13:48:21.097641+0000","last_became_active":"2024-04-06T13:48:19.699567+0000","last_became_peered":"2024-04-06T13:48:19.699567+0000","last_unstale":"2024-04-06T13:48:21.097641+0000","last_undegraded":"2024-04-06T13:48:21.097641+0000","last_fullsized":"2024-04-06T13:48:21.097641+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:01:05.087706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899870+0000","last_change":"2024-04-06T13:48:19.697603+0000","last_active":"2024-04-06T13:48:20.899870+0000","last_peered":"2024-04-06T13:48:20.899870+0000","last_clean":"2024-04-06T13:48:20.899870+0000","last_became_active":"2024-04-06T13:48:19.697356+0000","last_became_peered":"2024-04-06T13:48:19.697356+0000","last_unstale":"2024-04-06T13:48:20.899870+0000","last_undegraded":"2024-04-06T13:48:20.899870+0000","last_fullsized":"2024-04-06T13:48:20.899870+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:44.773715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899804+0000","last_change":"2024-04-06T13:48:19.696446+0000","last_active":"2024-04-06T13:48:20.899804+0000","last_peered":"2024-04-06T13:48:20.899804+0000","last_clean":"2024-04-06T13:48:20.899804+0000","last_became_active":"2024-04-06T13:48:19.696090+0000","last_became_peered":"2024-04-06T13:48:19.696090+0000","last_unstale":"2024-04-06T13:48:20.899804+0000","last_undegraded":"2024-04-06T13:48:20.899804+0000","last_fullsized":"2024-04-06T13:48:20.899804+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:35.461214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.266186+0000","last_change":"2024-04-06T13:48:18.266186+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.266186+0000","last_undegraded":"2024-04-06T13:48:18.266186+0000","last_fullsized":"2024-04-06T13:48:18.266186+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:54:56.369118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1d","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.695623+0000","last_change":"2024-04-06T13:48:19.280859+0000","last_active":"2024-04-06T13:48:19.695623+0000","last_peered":"2024-04-06T13:48:19.695623+0000","last_clean":"2024-04-06T13:48:19.695623+0000","last_became_active":"2024-04-06T13:48:19.280512+0000","last_became_peered":"2024-04-06T13:48:19.280512+0000","last_unstale":"2024-04-06T13:48:19.695623+0000","last_undegraded":"2024-04-06T13:48:19.695623+0000","last_fullsized":"2024-04-06T13:48:19.695623+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:43.765754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1c","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.703978+0000","last_change":"2024-04-06T13:48:19.280934+0000","last_active":"2024-04-06T13:48:19.703978+0000","last_peered":"2024-04-06T13:48:19.703978+0000","last_clean":"2024-04-06T13:48:19.703978+0000","last_became_active":"2024-04-06T13:48:19.280677+0000","last_became_peered":"2024-04-06T13:48:19.280677+0000","last_unstale":"2024-04-06T13:48:19.703978+0000","last_undegraded":"2024-04-06T13:48:19.703978+0000","last_fullsized":"2024-04-06T13:48:19.703978+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:30:23.776677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098008+0000","last_change":"2024-04-06T13:48:19.695081+0000","last_active":"2024-04-06T13:48:21.098008+0000","last_peered":"2024-04-06T13:48:21.098008+0000","last_clean":"2024-04-06T13:48:21.098008+0000","last_became_active":"2024-04-06T13:48:19.694838+0000","last_became_peered":"2024-04-06T13:48:19.694838+0000","last_unstale":"2024-04-06T13:48:21.098008+0000","last_undegraded":"2024-04-06T13:48:21.098008+0000","last_fullsized":"2024-04-06T13:48:21.098008+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:37:44.732983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262863+0000","last_change":"2024-04-06T13:48:18.262863+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262863+0000","last_undegraded":"2024-04-06T13:48:18.262863+0000","last_fullsized":"2024-04-06T13:48:18.262863+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:18.513042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098240+0000","last_change":"2024-04-06T13:48:19.694881+0000","last_active":"2024-04-06T13:48:21.098240+0000","last_peered":"2024-04-06T13:48:21.098240+0000","last_clean":"2024-04-06T13:48:21.098240+0000","last_became_active":"2024-04-06T13:48:19.693362+0000","last_became_peered":"2024-04-06T13:48:19.693362+0000","last_unstale":"2024-04-06T13:48:21.098240+0000","last_undegraded":"2024-04-06T13:48:21.098240+0000","last_fullsized":"2024-04-06T13:48:21.098240+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:36:47.917157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.699585+0000","last_change":"2024-04-06T13:48:19.274234+0000","last_active":"2024-04-06T13:48:19.699585+0000","last_peered":"2024-04-06T13:48:19.699585+0000","last_clean":"2024-04-06T13:48:19.699585+0000","last_became_active":"2024-04-06T13:48:19.273998+0000","last_became_peered":"2024-04-06T13:48:19.273998+0000","last_unstale":"2024-04-06T13:48:19.699585+0000","last_undegraded":"2024-04-06T13:48:19.699585+0000","last_fullsized":"2024-04-06T13:48:19.699585+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:49:23.791293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098351+0000","last_change":"2024-04-06T13:48:19.694934+0000","last_active":"2024-04-06T13:48:21.098351+0000","last_peered":"2024-04-06T13:48:21.098351+0000","last_clean":"2024-04-06T13:48:21.098351+0000","last_became_active":"2024-04-06T13:48:19.692682+0000","last_became_peered":"2024-04-06T13:48:19.692682+0000","last_unstale":"2024-04-06T13:48:21.098351+0000","last_undegraded":"2024-04-06T13:48:21.098351+0000","last_fullsized":"2024-04-06T13:48:21.098351+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:40:43.027987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.684729+0000","last_change":"2024-04-06T13:48:19.279719+0000","last_active":"2024-04-06T13:48:19.684729+0000","last_peered":"2024-04-06T13:48:19.684729+0000","last_clean":"2024-04-06T13:48:19.684729+0000","last_became_active":"2024-04-06T13:48:19.279404+0000","last_became_peered":"2024-04-06T13:48:19.279404+0000","last_unstale":"2024-04-06T13:48:19.684729+0000","last_undegraded":"2024-04-06T13:48:19.684729+0000","last_fullsized":"2024-04-06T13:48:19.684729+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:42:35.927858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.265371+0000","last_change":"2024-04-06T13:48:18.265371+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.265371+0000","last_undegraded":"2024-04-06T13:48:18.265371+0000","last_fullsized":"2024-04-06T13:48:18.265371+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:02:13.092175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098078+0000","last_change":"2024-04-06T13:48:19.694595+0000","last_active":"2024-04-06T13:48:21.098078+0000","last_peered":"2024-04-06T13:48:21.098078+0000","last_clean":"2024-04-06T13:48:21.098078+0000","last_became_active":"2024-04-06T13:48:19.692548+0000","last_became_peered":"2024-04-06T13:48:19.692548+0000","last_unstale":"2024-04-06T13:48:21.098078+0000","last_undegraded":"2024-04-06T13:48:21.098078+0000","last_fullsized":"2024-04-06T13:48:21.098078+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:39:01.587642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.684653+0000","last_change":"2024-04-06T13:48:19.273745+0000","last_active":"2024-04-06T13:48:19.684653+0000","last_peered":"2024-04-06T13:48:19.684653+0000","last_clean":"2024-04-06T13:48:19.684653+0000","last_became_active":"2024-04-06T13:48:19.273478+0000","last_became_peered":"2024-04-06T13:48:19.273478+0000","last_unstale":"2024-04-06T13:48:19.684653+0000","last_undegraded":"2024-04-06T13:48:19.684653+0000","last_fullsized":"2024-04-06T13:48:19.684653+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:00:55.130902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683007+0000","last_change":"2024-04-06T13:48:19.688682+0000","last_active":"2024-04-06T13:48:20.683007+0000","last_peered":"2024-04-06T13:48:20.683007+0000","last_clean":"2024-04-06T13:48:20.683007+0000","last_became_active":"2024-04-06T13:48:19.688397+0000","last_became_peered":"2024-04-06T13:48:19.688397+0000","last_unstale":"2024-04-06T13:48:20.683007+0000","last_undegraded":"2024-04-06T13:48:20.683007+0000","last_fullsized":"2024-04-06T13:48:20.683007+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:39:01.587642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.683105+0000","last_change":"2024-04-06T13:48:19.278056+0000","last_active":"2024-04-06T13:48:19.683105+0000","last_peered":"2024-04-06T13:48:19.683105+0000","last_clean":"2024-04-06T13:48:19.683105+0000","last_became_active":"2024-04-06T13:48:19.277796+0000","last_became_peered":"2024-04-06T13:48:19.277796+0000","last_unstale":"2024-04-06T13:48:19.683105+0000","last_undegraded":"2024-04-06T13:48:19.683105+0000","last_fullsized":"2024-04-06T13:48:19.683105+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:31:43.320734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097587+0000","last_change":"2024-04-06T13:48:19.700325+0000","last_active":"2024-04-06T13:48:21.097587+0000","last_peered":"2024-04-06T13:48:21.097587+0000","last_clean":"2024-04-06T13:48:21.097587+0000","last_became_active":"2024-04-06T13:48:19.698751+0000","last_became_peered":"2024-04-06T13:48:19.698751+0000","last_unstale":"2024-04-06T13:48:21.097587+0000","last_undegraded":"2024-04-06T13:48:21.097587+0000","last_fullsized":"2024-04-06T13:48:21.097587+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:16.165915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262594+0000","last_change":"2024-04-06T13:48:18.262594+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262594+0000","last_undegraded":"2024-04-06T13:48:18.262594+0000","last_fullsized":"2024-04-06T13:48:18.262594+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:06:05.488014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097951+0000","last_change":"2024-04-06T13:48:19.692164+0000","last_active":"2024-04-06T13:48:21.097951+0000","last_peered":"2024-04-06T13:48:21.097951+0000","last_clean":"2024-04-06T13:48:21.097951+0000","last_became_active":"2024-04-06T13:48:19.691910+0000","last_became_peered":"2024-04-06T13:48:19.691910+0000","last_unstale":"2024-04-06T13:48:21.097951+0000","last_undegraded":"2024-04-06T13:48:21.097951+0000","last_fullsized":"2024-04-06T13:48:21.097951+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:19:27.875509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.265322+0000","last_change":"2024-04-06T13:48:18.265322+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.265322+0000","last_undegraded":"2024-04-06T13:48:18.265322+0000","last_fullsized":"2024-04-06T13:48:18.265322+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:56:55.759595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899896+0000","last_change":"2024-04-06T13:48:19.693858+0000","last_active":"2024-04-06T13:48:20.899896+0000","last_peered":"2024-04-06T13:48:20.899896+0000","last_clean":"2024-04-06T13:48:20.899896+0000","last_became_active":"2024-04-06T13:48:19.693515+0000","last_became_peered":"2024-04-06T13:48:19.693515+0000","last_unstale":"2024-04-06T13:48:20.899896+0000","last_undegraded":"2024-04-06T13:48:20.899896+0000","last_fullsized":"2024-04-06T13:48:20.899896+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:10:26.852793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262385+0000","last_change":"2024-04-06T13:48:18.262385+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262385+0000","last_undegraded":"2024-04-06T13:48:18.262385+0000","last_fullsized":"2024-04-06T13:48:18.262385+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:31:24.701348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098230+0000","last_change":"2024-04-06T13:48:19.694858+0000","last_active":"2024-04-06T13:48:21.098230+0000","last_peered":"2024-04-06T13:48:21.098230+0000","last_clean":"2024-04-06T13:48:21.098230+0000","last_became_active":"2024-04-06T13:48:19.693049+0000","last_became_peered":"2024-04-06T13:48:19.693049+0000","last_unstale":"2024-04-06T13:48:21.098230+0000","last_undegraded":"2024-04-06T13:48:21.098230+0000","last_fullsized":"2024-04-06T13:48:21.098230+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:19:00.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.261860+0000","last_change":"2024-04-06T13:48:18.261860+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.261860+0000","last_undegraded":"2024-04-06T13:48:18.261860+0000","last_fullsized":"2024-04-06T13:48:18.261860+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:35:17.934513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.263085+0000","last_change":"2024-04-06T13:48:18.263085+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.263085+0000","last_undegraded":"2024-04-06T13:48:18.263085+0000","last_fullsized":"2024-04-06T13:48:18.263085+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:02:23.692226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.10","version":"247'8","reported_seq":19,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.704884+0000","last_change":"2024-04-06T13:48:19.274645+0000","last_active":"2024-04-06T13:48:19.704884+0000","last_peered":"2024-04-06T13:48:19.704884+0000","last_clean":"2024-04-06T13:48:19.704884+0000","last_became_active":"2024-04-06T13:48:19.274308+0000","last_became_peered":"2024-04-06T13:48:19.274308+0000","last_unstale":"2024-04-06T13:48:19.704884+0000","last_undegraded":"2024-04-06T13:48:19.704884+0000","last_fullsized":"2024-04-06T13:48:19.704884+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:31:15.493970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.265775+0000","last_change":"2024-04-06T13:48:18.265775+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.265775+0000","last_undegraded":"2024-04-06T13:48:18.265775+0000","last_fullsized":"2024-04-06T13:48:18.265775+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:49.041735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.e","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.706958+0000","last_change":"2024-04-06T13:48:19.275494+0000","last_active":"2024-04-06T13:48:19.706958+0000","last_peered":"2024-04-06T13:48:19.706958+0000","last_clean":"2024-04-06T13:48:19.706958+0000","last_became_active":"2024-04-06T13:48:19.275194+0000","last_became_peered":"2024-04-06T13:48:19.275194+0000","last_unstale":"2024-04-06T13:48:19.706958+0000","last_undegraded":"2024-04-06T13:48:19.706958+0000","last_fullsized":"2024-04-06T13:48:19.706958+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:11:29.081483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899798+0000","last_change":"2024-04-06T13:48:19.702505+0000","last_active":"2024-04-06T13:48:20.899798+0000","last_peered":"2024-04-06T13:48:20.899798+0000","last_clean":"2024-04-06T13:48:20.899798+0000","last_became_active":"2024-04-06T13:48:19.702245+0000","last_became_peered":"2024-04-06T13:48:19.702245+0000","last_unstale":"2024-04-06T13:48:20.899798+0000","last_undegraded":"2024-04-06T13:48:20.899798+0000","last_fullsized":"2024-04-06T13:48:20.899798+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:45.672315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.695714+0000","last_change":"2024-04-06T13:48:19.281905+0000","last_active":"2024-04-06T13:48:19.695714+0000","last_peered":"2024-04-06T13:48:19.695714+0000","last_clean":"2024-04-06T13:48:19.695714+0000","last_became_active":"2024-04-06T13:48:19.281052+0000","last_became_peered":"2024-04-06T13:48:19.281052+0000","last_unstale":"2024-04-06T13:48:19.695714+0000","last_undegraded":"2024-04-06T13:48:19.695714+0000","last_fullsized":"2024-04-06T13:48:19.695714+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:48.100692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097580+0000","last_change":"2024-04-06T13:48:19.700428+0000","last_active":"2024-04-06T13:48:21.097580+0000","last_peered":"2024-04-06T13:48:21.097580+0000","last_clean":"2024-04-06T13:48:21.097580+0000","last_became_active":"2024-04-06T13:48:19.699896+0000","last_became_peered":"2024-04-06T13:48:19.699896+0000","last_unstale":"2024-04-06T13:48:21.097580+0000","last_undegraded":"2024-04-06T13:48:21.097580+0000","last_fullsized":"2024-04-06T13:48:21.097580+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:00:56.138863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262279+0000","last_change":"2024-04-06T13:48:18.262279+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262279+0000","last_undegraded":"2024-04-06T13:48:18.262279+0000","last_fullsized":"2024-04-06T13:48:18.262279+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:55:10.767276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097956+0000","last_change":"2024-04-06T13:48:19.697815+0000","last_active":"2024-04-06T13:48:21.097956+0000","last_peered":"2024-04-06T13:48:21.097956+0000","last_clean":"2024-04-06T13:48:21.097956+0000","last_became_active":"2024-04-06T13:48:19.695354+0000","last_became_peered":"2024-04-06T13:48:19.695354+0000","last_unstale":"2024-04-06T13:48:21.097956+0000","last_undegraded":"2024-04-06T13:48:21.097956+0000","last_fullsized":"2024-04-06T13:48:21.097956+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:38:12.538926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.700247+0000","last_change":"2024-04-06T13:48:19.278849+0000","last_active":"2024-04-06T13:48:19.700247+0000","last_peered":"2024-04-06T13:48:19.700247+0000","last_clean":"2024-04-06T13:48:19.700247+0000","last_became_active":"2024-04-06T13:48:19.278604+0000","last_became_peered":"2024-04-06T13:48:19.278604+0000","last_unstale":"2024-04-06T13:48:19.700247+0000","last_undegraded":"2024-04-06T13:48:19.700247+0000","last_fullsized":"2024-04-06T13:48:19.700247+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:19:26.867548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.900001+0000","last_change":"2024-04-06T13:48:19.696316+0000","last_active":"2024-04-06T13:48:20.900001+0000","last_peered":"2024-04-06T13:48:20.900001+0000","last_clean":"2024-04-06T13:48:20.900001+0000","last_became_active":"2024-04-06T13:48:19.696025+0000","last_became_peered":"2024-04-06T13:48:19.696025+0000","last_unstale":"2024-04-06T13:48:20.900001+0000","last_undegraded":"2024-04-06T13:48:20.900001+0000","last_fullsized":"2024-04-06T13:48:20.900001+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:33:27.603807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.680627+0000","last_change":"2024-04-06T13:48:19.282165+0000","last_active":"2024-04-06T13:48:19.680627+0000","last_peered":"2024-04-06T13:48:19.680627+0000","last_clean":"2024-04-06T13:48:19.680627+0000","last_became_active":"2024-04-06T13:48:19.281892+0000","last_became_peered":"2024-04-06T13:48:19.281892+0000","last_unstale":"2024-04-06T13:48:19.680627+0000","last_undegraded":"2024-04-06T13:48:19.680627+0000","last_fullsized":"2024-04-06T13:48:19.680627+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:10:25.844832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683665+0000","last_change":"2024-04-06T13:48:19.699890+0000","last_active":"2024-04-06T13:48:20.683665+0000","last_peered":"2024-04-06T13:48:20.683665+0000","last_clean":"2024-04-06T13:48:20.683665+0000","last_became_active":"2024-04-06T13:48:19.699566+0000","last_became_peered":"2024-04-06T13:48:19.699566+0000","last_unstale":"2024-04-06T13:48:20.683665+0000","last_undegraded":"2024-04-06T13:48:20.683665+0000","last_fullsized":"2024-04-06T13:48:20.683665+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:10:02.935661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.682503+0000","last_change":"2024-04-06T13:48:19.276781+0000","last_active":"2024-04-06T13:48:19.682503+0000","last_peered":"2024-04-06T13:48:19.682503+0000","last_clean":"2024-04-06T13:48:19.682503+0000","last_became_active":"2024-04-06T13:48:19.275438+0000","last_became_peered":"2024-04-06T13:48:19.275438+0000","last_unstale":"2024-04-06T13:48:19.682503+0000","last_undegraded":"2024-04-06T13:48:19.682503+0000","last_fullsized":"2024-04-06T13:48:19.682503+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:35.170771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683912+0000","last_change":"2024-04-06T13:48:19.685464+0000","last_active":"2024-04-06T13:48:20.683912+0000","last_peered":"2024-04-06T13:48:20.683912+0000","last_clean":"2024-04-06T13:48:20.683912+0000","last_became_active":"2024-04-06T13:48:19.685139+0000","last_became_peered":"2024-04-06T13:48:19.685139+0000","last_unstale":"2024-04-06T13:48:20.683912+0000","last_undegraded":"2024-04-06T13:48:20.683912+0000","last_fullsized":"2024-04-06T13:48:20.683912+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:19:00.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.683371+0000","last_change":"2024-04-06T13:48:19.279096+0000","last_active":"2024-04-06T13:48:19.683371+0000","last_peered":"2024-04-06T13:48:19.683371+0000","last_clean":"2024-04-06T13:48:19.683371+0000","last_became_active":"2024-04-06T13:48:19.278774+0000","last_became_peered":"2024-04-06T13:48:19.278774+0000","last_unstale":"2024-04-06T13:48:19.683371+0000","last_undegraded":"2024-04-06T13:48:19.683371+0000","last_fullsized":"2024-04-06T13:48:19.683371+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:46:06.202483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097785+0000","last_change":"2024-04-06T13:48:19.697387+0000","last_active":"2024-04-06T13:48:21.097785+0000","last_peered":"2024-04-06T13:48:21.097785+0000","last_clean":"2024-04-06T13:48:21.097785+0000","last_became_active":"2024-04-06T13:48:19.697096+0000","last_became_peered":"2024-04-06T13:48:19.697096+0000","last_unstale":"2024-04-06T13:48:21.097785+0000","last_undegraded":"2024-04-06T13:48:21.097785+0000","last_fullsized":"2024-04-06T13:48:21.097785+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:52:01.901954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.698698+0000","last_change":"2024-04-06T13:48:19.271906+0000","last_active":"2024-04-06T13:48:19.698698+0000","last_peered":"2024-04-06T13:48:19.698698+0000","last_clean":"2024-04-06T13:48:19.698698+0000","last_became_active":"2024-04-06T13:48:19.271643+0000","last_became_peered":"2024-04-06T13:48:19.271643+0000","last_unstale":"2024-04-06T13:48:19.698698+0000","last_undegraded":"2024-04-06T13:48:19.698698+0000","last_fullsized":"2024-04-06T13:48:19.698698+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:54:44.219906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262063+0000","last_change":"2024-04-06T13:48:18.262063+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262063+0000","last_undegraded":"2024-04-06T13:48:18.262063+0000","last_fullsized":"2024-04-06T13:48:18.262063+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:18.981232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097491+0000","last_change":"2024-04-06T13:48:19.694094+0000","last_active":"2024-04-06T13:48:21.097491+0000","last_peered":"2024-04-06T13:48:21.097491+0000","last_clean":"2024-04-06T13:48:21.097491+0000","last_became_active":"2024-04-06T13:48:19.692958+0000","last_became_peered":"2024-04-06T13:48:19.692958+0000","last_unstale":"2024-04-06T13:48:21.097491+0000","last_undegraded":"2024-04-06T13:48:21.097491+0000","last_fullsized":"2024-04-06T13:48:21.097491+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:04:10.547739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.684628+0000","last_change":"2024-04-06T13:48:19.272078+0000","last_active":"2024-04-06T13:48:19.684628+0000","last_peered":"2024-04-06T13:48:19.684628+0000","last_clean":"2024-04-06T13:48:19.684628+0000","last_became_active":"2024-04-06T13:48:19.271672+0000","last_became_peered":"2024-04-06T13:48:19.271672+0000","last_unstale":"2024-04-06T13:48:19.684628+0000","last_undegraded":"2024-04-06T13:48:19.684628+0000","last_fullsized":"2024-04-06T13:48:19.684628+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:23.871607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.685267+0000","last_change":"2024-04-06T13:48:19.278939+0000","last_active":"2024-04-06T13:48:19.685267+0000","last_peered":"2024-04-06T13:48:19.685267+0000","last_clean":"2024-04-06T13:48:19.685267+0000","last_became_active":"2024-04-06T13:48:19.278677+0000","last_became_peered":"2024-04-06T13:48:19.278677+0000","last_unstale":"2024-04-06T13:48:19.685267+0000","last_undegraded":"2024-04-06T13:48:19.685267+0000","last_fullsized":"2024-04-06T13:48:19.685267+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:15.157954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097917+0000","last_change":"2024-04-06T13:48:19.693016+0000","last_active":"2024-04-06T13:48:21.097917+0000","last_peered":"2024-04-06T13:48:21.097917+0000","last_clean":"2024-04-06T13:48:21.097917+0000","last_became_active":"2024-04-06T13:48:19.692760+0000","last_became_peered":"2024-04-06T13:48:19.692760+0000","last_unstale":"2024-04-06T13:48:21.097917+0000","last_undegraded":"2024-04-06T13:48:21.097917+0000","last_fullsized":"2024-04-06T13:48:21.097917+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:41:37.714397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.682132+0000","last_change":"2024-04-06T13:48:19.282189+0000","last_active":"2024-04-06T13:48:19.682132+0000","last_peered":"2024-04-06T13:48:19.682132+0000","last_clean":"2024-04-06T13:48:19.682132+0000","last_became_active":"2024-04-06T13:48:19.281907+0000","last_became_peered":"2024-04-06T13:48:19.281907+0000","last_unstale":"2024-04-06T13:48:19.682132+0000","last_undegraded":"2024-04-06T13:48:19.682132+0000","last_fullsized":"2024-04-06T13:48:19.682132+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:34.453253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.261299+0000","last_change":"2024-04-06T13:48:18.261299+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.261299+0000","last_undegraded":"2024-04-06T13:48:18.261299+0000","last_fullsized":"2024-04-06T13:48:18.261299+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:51:53.597002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899957+0000","last_change":"2024-04-06T13:48:19.704846+0000","last_active":"2024-04-06T13:48:20.899957+0000","last_peered":"2024-04-06T13:48:20.899957+0000","last_clean":"2024-04-06T13:48:20.899957+0000","last_became_active":"2024-04-06T13:48:19.704420+0000","last_became_peered":"2024-04-06T13:48:19.704420+0000","last_unstale":"2024-04-06T13:48:20.899957+0000","last_undegraded":"2024-04-06T13:48:20.899957+0000","last_fullsized":"2024-04-06T13:48:20.899957+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:36.178732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.684819+0000","last_change":"2024-04-06T13:48:19.277530+0000","last_active":"2024-04-06T13:48:19.684819+0000","last_peered":"2024-04-06T13:48:19.684819+0000","last_clean":"2024-04-06T13:48:19.684819+0000","last_became_active":"2024-04-06T13:48:19.276966+0000","last_became_peered":"2024-04-06T13:48:19.276966+0000","last_unstale":"2024-04-06T13:48:19.684819+0000","last_undegraded":"2024-04-06T13:48:19.684819+0000","last_fullsized":"2024-04-06T13:48:19.684819+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:42:14.962783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683820+0000","last_change":"2024-04-06T13:48:19.688861+0000","last_active":"2024-04-06T13:48:20.683820+0000","last_peered":"2024-04-06T13:48:20.683820+0000","last_clean":"2024-04-06T13:48:20.683820+0000","last_became_active":"2024-04-06T13:48:19.688496+0000","last_became_peered":"2024-04-06T13:48:19.688496+0000","last_unstale":"2024-04-06T13:48:20.683820+0000","last_undegraded":"2024-04-06T13:48:20.683820+0000","last_fullsized":"2024-04-06T13:48:20.683820+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:36:47.917157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.680854+0000","last_change":"2024-04-06T13:48:19.276467+0000","last_active":"2024-04-06T13:48:19.680854+0000","last_peered":"2024-04-06T13:48:19.680854+0000","last_clean":"2024-04-06T13:48:19.680854+0000","last_became_active":"2024-04-06T13:48:19.275988+0000","last_became_peered":"2024-04-06T13:48:19.275988+0000","last_unstale":"2024-04-06T13:48:19.680854+0000","last_undegraded":"2024-04-06T13:48:19.680854+0000","last_fullsized":"2024-04-06T13:48:19.680854+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:44.664354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1e","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.694613+0000","last_change":"2024-04-06T13:48:19.277300+0000","last_active":"2024-04-06T13:48:19.694613+0000","last_peered":"2024-04-06T13:48:19.694613+0000","last_clean":"2024-04-06T13:48:19.694613+0000","last_became_active":"2024-04-06T13:48:19.276795+0000","last_became_peered":"2024-04-06T13:48:19.276795+0000","last_unstale":"2024-04-06T13:48:19.694613+0000","last_undegraded":"2024-04-06T13:48:19.694613+0000","last_fullsized":"2024-04-06T13:48:19.694613+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:54:25.950801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"233'370","reported_seq":1333,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-06T13:48:15.595868+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:48:15.595868+0000","last_peered":"2024-04-06T13:48:15.595868+0000","last_clean":"2024-04-06T13:48:15.595868+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:48:15.595868+0000","last_undegraded":"2024-04-06T13:48:15.595868+0000","last_fullsized":"2024-04-06T13:48:15.595868+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":82,"acting":82,"num_store_stats":5},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":70,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21,"ondisk_log_size":21,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575992,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56856,"kb_used_data":3596,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314984,"statfs":{"total":96636764160,"available":96578543616,"internally_reserved":0,"allocated":3682304,"data_stored":2174385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57899999999999996,"15min":0.59499999999999997},"min":{"1min":0.35499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.72199999999999998,"5min":0.78400000000000003,"15min":0.82499999999999996},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56799999999999995,"15min":0.58999999999999997},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78500000000000003,"5min":0.80800000000000005,"15min":0.82899999999999996},"last":0.753}]},{"osd":1,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.55000000000000004,"15min":0.57899999999999996},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77200000000000002,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.57699999999999996,"15min":0.59599999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.69899999999999995,"5min":0.83399999999999996,"15min":0.85299999999999998},"last":0.78100000000000003}]},{"osd":2,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.56299999999999994,"15min":0.58499999999999996},"min":{"1min":0.33200000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69399999999999995,"5min":0.81699999999999995,"15min":0.83899999999999997},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.58599999999999997,"15min":0.60099999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69999999999999996,"5min":0.755,"15min":0.79700000000000004},"last":0.76900000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.59399999999999997,"15min":0.60599999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.67200000000000004,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58799999999999997,"15min":0.60299999999999998},"min":{"1min":0.39000000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78900000000000003,"5min":0.82099999999999995,"15min":0.84099999999999997},"last":0.66500000000000004}]},{"osd":4,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45800000000000002,"15min":0.46600000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.81599999999999995},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.36799999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.82699999999999996},"last":0.53500000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.498,"15min":0.498},"min":{"1min":0.315,"5min":0.315,"15min":0.308},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56899999999999995,"15min":0.57799999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33600000000000002},"max":{"1min":0.74099999999999999,"5min":0.80700000000000005,"15min":0.85399999999999998},"last":0.55200000000000005}]},{"osd":6,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.34899999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.879},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57099999999999995,"15min":0.58699999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81799999999999995,"5min":0.84899999999999998,"15min":0.89800000000000002},"last":0.57199999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575991,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56260,"kb_used_data":3000,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315580,"statfs":{"total":96636764160,"available":96579153920,"internally_reserved":0,"allocated":3072000,"data_stored":1576789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.53600000000000003,"15min":0.55700000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69399999999999995,"5min":0.79700000000000004,"15min":0.83799999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.54100000000000004,"15min":0.56499999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.62,"5min":0.76900000000000002,"15min":0.81999999999999995},"last":0.61899999999999999}]},{"osd":1,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.60399999999999998,"15min":0.624},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76000000000000001,"5min":0.81699999999999995,"15min":0.85799999999999998},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.622,"15min":0.64400000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.378},"max":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.82699999999999996},"last":0.67800000000000005}]},{"osd":2,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56999999999999995,"15min":0.59399999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72099999999999997,"5min":0.79000000000000004,"15min":0.82899999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.56299999999999994,"15min":0.58599999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.80700000000000005},"last":0.58499999999999996}]},{"osd":3,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.58599999999999997,"15min":0.60199999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.73199999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.61899999999999999,"15min":0.64300000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.36099999999999999},"max":{"1min":0.80200000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.68899999999999995}]},{"osd":4,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70599999999999996,"5min":0.72299999999999998,"15min":0.81399999999999995},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.51100000000000001,"15min":0.53800000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.66400000000000003,"5min":0.82099999999999995,"15min":0.84099999999999997},"last":0.70799999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.48899999999999999,"15min":0.50600000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.33300000000000002},"max":{"1min":0.63,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52000000000000002,"15min":0.53100000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67400000000000004,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.63500000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53900000000000003,"15min":0.54800000000000004},"min":{"1min":0.34899999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.67700000000000005,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55300000000000005,"15min":0.56899999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.32800000000000001},"max":{"1min":0.71199999999999997,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.54800000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575991,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60404,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311436,"statfs":{"total":96636764160,"available":96574910464,"internally_reserved":0,"allocated":3121152,"data_stored":1603524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.38100000000000001,"15min":0.39100000000000001},"min":{"1min":0.248,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.378,"5min":0.621,"15min":0.621},"last":0.441},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.41799999999999998,"15min":0.42699999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.501,"5min":0.60799999999999998,"15min":0.69399999999999995},"last":0.27500000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.41299999999999998,"15min":0.42399999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.46899999999999997,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.44900000000000001,"15min":0.46400000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.55400000000000005,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.34799999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.46999999999999997,"15min":0.47999999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.52800000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.42699999999999999,"15min":0.439},"min":{"1min":0.28799999999999998,"5min":0.27600000000000002,"15min":0.26300000000000001},"max":{"1min":0.45500000000000002,"5min":0.63500000000000001,"15min":0.66100000000000003},"last":0.33100000000000002}]},{"osd":4,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.628,"15min":0.64400000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.33800000000000002},"max":{"1min":0.70099999999999996,"5min":0.89900000000000002,"15min":0.97399999999999998},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.64400000000000002,"15min":0.65500000000000003},"min":{"1min":0.46100000000000002,"5min":0.44500000000000001,"15min":0.44400000000000001},"max":{"1min":0.73499999999999999,"5min":0.94599999999999995,"15min":0.995},"last":0.55400000000000005}]},{"osd":5,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.64100000000000001,"15min":0.65300000000000002},"min":{"1min":0.42999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73899999999999999,"5min":0.96899999999999997,"15min":1.073},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.65700000000000003,"15min":0.66800000000000004},"min":{"1min":0.45200000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.72799999999999998,"5min":0.96499999999999997,"15min":1.0209999999999999},"last":0.47899999999999998}]},{"osd":6,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.46300000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73899999999999999,"5min":0.89200000000000002,"15min":1},"last":0.503},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.67100000000000004,"15min":0.68400000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.72499999999999998,"5min":0.94599999999999995,"15min":1.036},"last":0.48799999999999999}]},{"osd":7,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.69399999999999995,"15min":0.69499999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85399999999999998,"5min":0.94899999999999995,"15min":1.0649999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65700000000000003,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.76300000000000001,"5min":0.92900000000000005,"15min":1.056},"last":0.46899999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575991,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61004,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310836,"statfs":{"total":96636764160,"available":96574296064,"internally_reserved":0,"allocated":3727360,"data_stored":2195456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.38200000000000001,"15min":0.39300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.45100000000000001,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.39300000000000002,"15min":0.41699999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.48199999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.41399999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.40100000000000002,"15min":0.42499999999999999},"min":{"1min":0.26100000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49399999999999999,"5min":0.66700000000000004,"15min":0.70099999999999996},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.42499999999999999,"15min":0.443},"min":{"1min":0.28399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.47999999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.33300000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.40699999999999997,"15min":0.42499999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.50700000000000001,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.45700000000000002,"15min":0.47799999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.51900000000000002,"5min":0.72999999999999998,"15min":0.75700000000000001},"last":0.442}]},{"osd":4,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.60499999999999998,"15min":0.64100000000000001},"min":{"1min":0.39100000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.80900000000000005,"5min":0.93100000000000005,"15min":1.0329999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.61199999999999999,"15min":0.64300000000000002},"min":{"1min":0.372,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88500000000000001,"5min":0.999,"15min":0.999},"last":0.57899999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.60199999999999998,"15min":0.63900000000000001},"min":{"1min":0.39300000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.0009999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.63800000000000001,"15min":0.66300000000000003},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.749,"5min":0.92800000000000005,"15min":1.052},"last":0.501}]},{"osd":6,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.64500000000000002,"15min":0.67400000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.874,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.64200000000000002,"15min":0.66900000000000004},"min":{"1min":0.40500000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77500000000000002,"5min":0.96399999999999997,"15min":1.085},"last":0.67600000000000005}]},{"osd":7,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.66300000000000003,"15min":0.69099999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.80700000000000005,"5min":1.01,"15min":1.121},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.64700000000000002,"15min":0.67200000000000004},"min":{"1min":0.42199999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.93700000000000006,"5min":0.95799999999999996,"15min":1.0229999999999999},"last":0.69399999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575993,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60420,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311420,"statfs":{"total":96636764160,"available":96574894080,"internally_reserved":0,"allocated":3129344,"data_stored":1604176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.376,"15min":0.38400000000000001},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.42299999999999999,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.40600000000000003,"15min":0.42199999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.249},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":1.044},"last":0.27900000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.38500000000000001,"15min":0.40400000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51100000000000001,"5min":0.54000000000000004,"15min":0.753},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.42799999999999999,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.82199999999999995},"last":0.35999999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.439,"15min":0.45800000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.53200000000000003,"5min":0.69699999999999995,"15min":0.70299999999999996},"last":0.374},{"interface":"front","average":{"1min":0.377,"5min":0.437,"15min":0.44900000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.504,"5min":0.64500000000000002,"15min":0.68600000000000005},"last":0.41499999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.621,"15min":0.63900000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.376},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.64400000000000002,"15min":0.66100000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.88900000000000001,"5min":1.022,"15min":1.022},"last":0.629}]},{"osd":5,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.64300000000000002,"15min":0.66300000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88100000000000001,"5min":0.98599999999999999,"15min":1.0149999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.65100000000000002,"15min":0.66700000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.65200000000000002,"15min":0.67200000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92000000000000004,"5min":0.97199999999999998,"15min":1.091},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.67600000000000005,"15min":0.69499999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.96099999999999997},"last":0.51200000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.66600000000000004,"15min":0.68799999999999994},"min":{"1min":0.42699999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.95399999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.68300000000000005,"15min":0.70799999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.85299999999999998,"5min":1.01,"15min":1.01},"last":0.53600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575991,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48084,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323756,"statfs":{"total":96636764160,"available":96587526144,"internally_reserved":0,"allocated":3088384,"data_stored":1584017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39000000000000001,"15min":0.38900000000000001},"min":{"1min":0.26000000000000001,"5min":0.255,"15min":0.23499999999999999},"max":{"1min":0.55200000000000005,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41199999999999998,"15min":0.41299999999999998},"min":{"1min":0.28799999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.56899999999999995,"5min":0.622,"15min":0.66700000000000004},"last":0.34000000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40699999999999997,"15min":0.40600000000000003},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.65700000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.438,"15min":0.44400000000000001},"min":{"1min":0.32800000000000001,"5min":0.27500000000000002,"15min":0.26600000000000001},"max":{"1min":0.63,"5min":0.65400000000000003,"15min":0.75800000000000001},"last":0.39900000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48299999999999998,"15min":0.48599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.5,"15min":0.503},"min":{"1min":0.38600000000000001,"5min":0.34599999999999997,"15min":0.32900000000000001},"max":{"1min":0.67800000000000005,"5min":0.69299999999999995,"15min":0.79300000000000004},"last":0.49199999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.439,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.876,"5min":1.002,"15min":1.018},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66100000000000003,"15min":0.65300000000000002},"min":{"1min":0.497,"5min":0.45900000000000002,"15min":0.375},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.60399999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.51900000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.95699999999999996,"5min":0.97299999999999998,"15min":0.999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.46200000000000002,"5min":0.437,"15min":0.437},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.61499999999999999}]},{"osd":6,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67500000000000004,"15min":0.66900000000000004},"min":{"1min":0.41499999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.63},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.68600000000000005,"15min":0.68400000000000005},"min":{"1min":0.48799999999999999,"5min":0.46700000000000003,"15min":0.41099999999999998},"max":{"1min":0.88800000000000001,"5min":0.96199999999999997,"15min":0.98799999999999999},"last":0.57399999999999995}]},{"osd":7,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68500000000000005,"15min":0.68000000000000005},"min":{"1min":0.46100000000000002,"5min":0.432,"15min":0.40999999999999998},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70299999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.024},"last":0.71199999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575992,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48132,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323708,"statfs":{"total":96636764160,"available":96587476992,"internally_reserved":0,"allocated":3129344,"data_stored":1604613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55900000000000005,"15min":0.58699999999999997},"min":{"1min":0.40699999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.753,"5min":0.876,"15min":0.88500000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.39300000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81100000000000005,"5min":1.202,"15min":1.202},"last":0.48299999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.61299999999999999,"15min":0.64000000000000001},"min":{"1min":0.40799999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71699999999999997,"5min":1.256,"15min":1.256},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.624,"15min":0.63},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.75900000000000001,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.56499999999999995}]},{"osd":2,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.57999999999999996,"15min":0.59999999999999998},"min":{"1min":0.41599999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77500000000000002,"5min":0.84599999999999997,"15min":0.877},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.60599999999999998,"15min":0.61599999999999999},"min":{"1min":0.39600000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.97899999999999998,"5min":1.294,"15min":1.294},"last":0.41199999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84499999999999997,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.60899999999999999,"15min":0.626},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.78800000000000003,"5min":0.90200000000000002,"15min":0.91100000000000003},"last":0.55400000000000005}]},{"osd":5,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55500000000000005,"15min":0.52000000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.92800000000000005,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.56399999999999995,"15min":0.54100000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.95199999999999996,"5min":4.931,"15min":4.931},"last":0.63500000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53500000000000003,"15min":0.53300000000000003},"min":{"1min":0.34999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.505},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34699999999999998},"max":{"1min":1.0609999999999999,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.62}]},{"osd":7,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59499999999999997,"15min":0.58199999999999996},"min":{"1min":0.36899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.91000000000000003,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59399999999999997,"15min":0.60199999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.373},"max":{"1min":0.86899999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.53200000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575992,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64508,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307332,"statfs":{"total":96636764160,"available":96570707968,"internally_reserved":0,"allocated":3121152,"data_stored":1603524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.57399999999999995,"15min":0.58499999999999996},"min":{"1min":0.36399999999999999,"5min":0.35899999999999999,"15min":0.33000000000000002},"max":{"1min":0.68000000000000005,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58099999999999996,"15min":0.59399999999999997},"min":{"1min":0.38100000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.76600000000000001,"5min":0.84699999999999998,"15min":0.89200000000000002},"last":0.63200000000000001}]},{"osd":1,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.58099999999999996,"15min":0.59399999999999997},"min":{"1min":0.38400000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79100000000000004,"5min":0.81299999999999994,"15min":0.81399999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.439,"5min":0.39400000000000002,"15min":0.37},"max":{"1min":0.70299999999999996,"5min":0.871,"15min":0.871},"last":0.59199999999999997}]},{"osd":2,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59299999999999997,"15min":0.59699999999999998},"min":{"1min":0.311,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69199999999999995,"5min":0.79700000000000004,"15min":0.84399999999999997},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58799999999999997,"15min":0.59599999999999997},"min":{"1min":0.44,"5min":0.378,"15min":0.378},"max":{"1min":0.73099999999999998,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.51700000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.79800000000000004,"5min":0.81100000000000005,"15min":0.94499999999999995},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61699999999999999,"15min":0.63200000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.74099999999999999,"5min":0.88200000000000001,"15min":0.91500000000000004},"last":0.61499999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52900000000000003,"15min":0.50700000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.27400000000000002},"max":{"1min":0.70999999999999996,"5min":0.85699999999999998,"15min":1.0369999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.59999999999999998,"15min":0.57699999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.308},"max":{"1min":0.77800000000000002,"5min":0.88800000000000001,"15min":1.1639999999999999},"last":0.70299999999999996}]},{"osd":6,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54300000000000004,"15min":0.52600000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.27900000000000003},"max":{"1min":0.89100000000000001,"5min":0.94999999999999996,"15min":1.0169999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54300000000000004,"15min":0.53100000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.32300000000000001},"max":{"1min":0.81899999999999995,"5min":0.878,"15min":1.0009999999999999},"last":0.68799999999999994}]},{"osd":7,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56499999999999995,"15min":0.54800000000000004},"min":{"1min":0.39200000000000002,"5min":0.35799999999999998,"15min":0.32900000000000001},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.66400000000000003,"15min":0.64800000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.35199999999999998},"max":{"1min":0.85299999999999998,"5min":0.92300000000000004,"15min":0.96599999999999997},"last":0.73199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:48:24.184 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:48:24.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.342+0000 7f7d2e346700 1 Processor -- start 2024-04-06T13:48:24.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.342+0000 7f7d2e346700 1 -- start start 2024-04-06T13:48:24.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.342+0000 7f7d2e346700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d280d11b0 0x7f7d280d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:24.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.342+0000 7f7d2e346700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7d280cb870 0x7f7d2815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:24.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.342+0000 7f7d2e346700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d280d1ad0 con 0x7f7d280cfa00 2024-04-06T13:48:24.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.342+0000 7f7d2e346700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d280ccdc0 con 0x7f7d280cb870 2024-04-06T13:48:24.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.342+0000 7f7d2e346700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d280d0b50 con 0x7f7d280d11b0 2024-04-06T13:48:24.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.343+0000 7f7d27fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d280d11b0 0x7f7d280d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:24.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.343+0000 7f7d277fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7d280cb870 0x7f7d2815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:24.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.343+0000 7f7d2c8e3700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f7d280cfa00 0x7f7d280cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:44276/0 (socket says 172.21.15.189:44276) 2024-04-06T13:48:24.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.343+0000 7f7d2c8e3700 1 -- 172.21.15.189:0/3340494121 learned_addr learned my addr 172.21.15.189:0/3340494121 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:24.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.343+0000 7f7d277fe700 1 -- 172.21.15.189:0/3340494121 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d280d11b0 msgr2=0x7f7d280d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:24.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.343+0000 7f7d277fe700 1 --2- 172.21.15.189:0/3340494121 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d280d11b0 0x7f7d280d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.343+0000 7f7d277fe700 1 -- 172.21.15.189:0/3340494121 >> v1:172.21.15.192:6789/0 conn(0x7f7d280cfa00 legacy=0x7f7d280cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:24.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.343+0000 7f7d277fe700 1 -- 172.21.15.189:0/3340494121 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d280cbd80 con 0x7f7d280cb870 2024-04-06T13:48:24.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.343+0000 7f7d26ffd700 1 -- 172.21.15.189:0/3340494121 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764073180 0 0) 0x7f7d280d1ad0 con 0x7f7d280cfa00 2024-04-06T13:48:24.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.343+0000 7f7d277fe700 1 --2- 172.21.15.189:0/3340494121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d280cb870 0x7f7d2815c280 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c009840 tx=0x7f7d1c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cad8d5fd983bf4a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:24.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.343+0000 7f7d26ffd700 1 -- 172.21.15.189:0/3340494121 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c011830 con 0x7f7d280cb870 2024-04-06T13:48:24.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.343+0000 7f7d26ffd700 1 -- 172.21.15.189:0/3340494121 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1c011990 con 0x7f7d280cb870 2024-04-06T13:48:24.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.343+0000 7f7d26ffd700 1 -- 172.21.15.189:0/3340494121 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c011d20 con 0x7f7d280cb870 2024-04-06T13:48:24.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.344+0000 7f7d2e346700 1 -- 172.21.15.189:0/3340494121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d280cb870 msgr2=0x7f7d2815c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:24.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.344+0000 7f7d2e346700 1 --2- 172.21.15.189:0/3340494121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d280cb870 0x7f7d2815c280 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c009840 tx=0x7f7d1c009b50 comp rx=0 tx=0).stop 2024-04-06T13:48:24.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.344+0000 7f7d2e346700 1 -- 172.21.15.189:0/3340494121 shutdown_connections 2024-04-06T13:48:24.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.344+0000 7f7d2e346700 1 --2- 172.21.15.189:0/3340494121 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d280d11b0 0x7f7d280d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.344+0000 7f7d2e346700 1 --2- 172.21.15.189:0/3340494121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d280cb870 0x7f7d2815c280 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.344+0000 7f7d2e346700 1 -- 172.21.15.189:0/3340494121 >> 172.21.15.189:0/3340494121 conn(0x7f7d280b2db0 msgr2=0x7f7d280b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:24.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.344+0000 7f7d2e346700 1 -- 172.21.15.189:0/3340494121 shutdown_connections 2024-04-06T13:48:24.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.344+0000 7f7d2e346700 1 -- 172.21.15.189:0/3340494121 wait complete. 2024-04-06T13:48:24.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.345+0000 7f7d2e346700 1 Processor -- start 2024-04-06T13:48:24.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.345+0000 7f7d2e346700 1 -- start start 2024-04-06T13:48:24.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.346+0000 7f7d2e346700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d280cb870 0x7f7d28157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:24.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.346+0000 7f7d2e346700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d280cfa00 0x7f7d2815c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:24.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.346+0000 7f7d2e346700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d280d11b0 0x7f7d280c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:24.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.346+0000 7f7d2e346700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d280c61e0 con 0x7f7d280cfa00 2024-04-06T13:48:24.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.346+0000 7f7d2e346700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d281e5040 con 0x7f7d280d11b0 2024-04-06T13:48:24.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.346+0000 7f7d2e346700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d280c6430 con 0x7f7d280cb870 2024-04-06T13:48:24.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.346+0000 7f7d277fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d280cfa00 0x7f7d2815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:24.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.346+0000 7f7d277fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d280cfa00 0x7f7d2815c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45098/0 (socket says 172.21.15.189:45098) 2024-04-06T13:48:24.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.346+0000 7f7d277fe700 1 -- 172.21.15.189:0/956175643 learned_addr learned my addr 172.21.15.189:0/956175643 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:24.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.346+0000 7f7d27fff700 1 --2- 172.21.15.189:0/956175643 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d280cb870 0x7f7d28157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:24.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.347+0000 7f7d2c8e3700 1 --2- 172.21.15.189:0/956175643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d280d11b0 0x7f7d280c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:24.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.347+0000 7f7d277fe700 1 -- 172.21.15.189:0/956175643 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d280cb870 msgr2=0x7f7d28157cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:24.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.347+0000 7f7d277fe700 1 --2- 172.21.15.189:0/956175643 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d280cb870 0x7f7d28157cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.347+0000 7f7d277fe700 1 -- 172.21.15.189:0/956175643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d280d11b0 msgr2=0x7f7d280c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:24.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.347+0000 7f7d277fe700 1 --2- 172.21.15.189:0/956175643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d280d11b0 0x7f7d280c71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.347+0000 7f7d277fe700 1 -- 172.21.15.189:0/956175643 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1c0094a0 con 0x7f7d280cfa00 2024-04-06T13:48:24.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.347+0000 7f7d277fe700 1 --2- 172.21.15.189:0/956175643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d280cfa00 0x7f7d2815c280 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c0090b0 tx=0x7f7d1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:24.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.347+0000 7f7d24ff9700 1 -- 172.21.15.189:0/956175643 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c0036a0 con 0x7f7d280cfa00 2024-04-06T13:48:24.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.347+0000 7f7d24ff9700 1 -- 172.21.15.189:0/956175643 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1c01e070 con 0x7f7d280cfa00 2024-04-06T13:48:24.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.347+0000 7f7d2e346700 1 -- 172.21.15.189:0/956175643 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d280c6710 con 0x7f7d280cfa00 2024-04-06T13:48:24.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.347+0000 7f7d24ff9700 1 -- 172.21.15.189:0/956175643 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c003a60 con 0x7f7d280cfa00 2024-04-06T13:48:24.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.347+0000 7f7d2e346700 1 -- 172.21.15.189:0/956175643 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d280c6cd0 con 0x7f7d280cfa00 2024-04-06T13:48:24.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.348+0000 7f7d0e7fc700 1 -- 172.21.15.189:0/956175643 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d000052f0 con 0x7f7d280cfa00 2024-04-06T13:48:24.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.348+0000 7f7d24ff9700 1 -- 172.21.15.189:0/956175643 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f7d1c003bc0 con 0x7f7d280cfa00 2024-04-06T13:48:24.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.349+0000 7f7d24ff9700 1 --2- 172.21.15.189:0/956175643 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7d1406aed0 0x7f7d1406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:24.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.349+0000 7f7d27fff700 1 --2- 172.21.15.189:0/956175643 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7d1406aed0 0x7f7d1406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:24.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.349+0000 7f7d24ff9700 1 -- 172.21.15.189:0/956175643 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7d1c08aa70 con 0x7f7d280cfa00 2024-04-06T13:48:24.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.349+0000 7f7d27fff700 1 --2- 172.21.15.189:0/956175643 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7d1406aed0 0x7f7d1406d390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7d18009ad0 tx=0x7f7d18008040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:24.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.353+0000 7f7d24ff9700 1 -- 172.21.15.189:0/956175643 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7d1c055750 con 0x7f7d280cfa00 2024-04-06T13:48:24.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.543+0000 7f7d0e7fc700 1 -- 172.21.15.189:0/956175643 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7d00000bc0 con 0x7f7d1406aed0 2024-04-06T13:48:24.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.551+0000 7f7d24ff9700 1 -- 172.21.15.189:0/956175643 <== mgr.11733 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286244 (secure 0 0 0) 0x7f7d00000bc0 con 0x7f7d1406aed0 2024-04-06T13:48:24.554 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:24.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.554+0000 7f7d0e7fc700 1 -- 172.21.15.189:0/956175643 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7d1406aed0 msgr2=0x7f7d1406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:24.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.554+0000 7f7d0e7fc700 1 --2- 172.21.15.189:0/956175643 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7d1406aed0 0x7f7d1406d390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7d18009ad0 tx=0x7f7d18008040 comp rx=0 tx=0).stop 2024-04-06T13:48:24.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.554+0000 7f7d0e7fc700 1 -- 172.21.15.189:0/956175643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d280cfa00 msgr2=0x7f7d2815c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:24.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.554+0000 7f7d0e7fc700 1 --2- 172.21.15.189:0/956175643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d280cfa00 0x7f7d2815c280 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c0090b0 tx=0x7f7d1c004750 comp rx=0 tx=0).stop 2024-04-06T13:48:24.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.554+0000 7f7d0e7fc700 1 -- 172.21.15.189:0/956175643 shutdown_connections 2024-04-06T13:48:24.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.554+0000 7f7d0e7fc700 1 --2- 172.21.15.189:0/956175643 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7d1406aed0 0x7f7d1406d390 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.554+0000 7f7d0e7fc700 1 --2- 172.21.15.189:0/956175643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d280d11b0 0x7f7d280c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.554+0000 7f7d0e7fc700 1 --2- 172.21.15.189:0/956175643 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d280cb870 0x7f7d28157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.554+0000 7f7d0e7fc700 1 --2- 172.21.15.189:0/956175643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d280cfa00 0x7f7d2815c280 unknown :-1 s=CLOSED pgs=2681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.554+0000 7f7d0e7fc700 1 -- 172.21.15.189:0/956175643 >> 172.21.15.189:0/956175643 conn(0x7f7d280b2db0 msgr2=0x7f7d2815a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:24.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.554+0000 7f7d0e7fc700 1 -- 172.21.15.189:0/956175643 shutdown_connections 2024-04-06T13:48:24.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.555+0000 7f7d0e7fc700 1 -- 172.21.15.189:0/956175643 wait complete. 2024-04-06T13:48:24.557 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:48:24.567 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-04-06T13:48:23.927698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590438,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":645,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":391,"ondisk_log_size":391,"up":148,"acting":148,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":77,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":455668,"kb_used_data":25460,"kb_used_omap":12,"kb_used_meta":430067,"kb_avail":754519052,"statfs":{"total":773094113280,"available":772627509248,"internally_reserved":0,"allocated":26071040,"data_stored":13946484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":440389176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.670934"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.900020+0000","last_change":"2024-04-06T13:48:19.704688+0000","last_active":"2024-04-06T13:48:20.900020+0000","last_peered":"2024-04-06T13:48:20.900020+0000","last_clean":"2024-04-06T13:48:20.900020+0000","last_became_active":"2024-04-06T13:48:19.704088+0000","last_became_peered":"2024-04-06T13:48:19.704088+0000","last_unstale":"2024-04-06T13:48:20.900020+0000","last_undegraded":"2024-04-06T13:48:20.900020+0000","last_fullsized":"2024-04-06T13:48:20.900020+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:49.108653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097823+0000","last_change":"2024-04-06T13:48:19.697461+0000","last_active":"2024-04-06T13:48:21.097823+0000","last_peered":"2024-04-06T13:48:21.097823+0000","last_clean":"2024-04-06T13:48:21.097823+0000","last_became_active":"2024-04-06T13:48:19.695459+0000","last_became_peered":"2024-04-06T13:48:19.695459+0000","last_unstale":"2024-04-06T13:48:21.097823+0000","last_undegraded":"2024-04-06T13:48:21.097823+0000","last_fullsized":"2024-04-06T13:48:21.097823+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:42:12.990594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098046+0000","last_change":"2024-04-06T13:48:19.690589+0000","last_active":"2024-04-06T13:48:21.098046+0000","last_peered":"2024-04-06T13:48:21.098046+0000","last_clean":"2024-04-06T13:48:21.098046+0000","last_became_active":"2024-04-06T13:48:19.690338+0000","last_became_peered":"2024-04-06T13:48:19.690338+0000","last_unstale":"2024-04-06T13:48:21.098046+0000","last_undegraded":"2024-04-06T13:48:21.098046+0000","last_fullsized":"2024-04-06T13:48:21.098046+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:51:42.234400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097813+0000","last_change":"2024-04-06T13:48:19.695665+0000","last_active":"2024-04-06T13:48:21.097813+0000","last_peered":"2024-04-06T13:48:21.097813+0000","last_clean":"2024-04-06T13:48:21.097813+0000","last_became_active":"2024-04-06T13:48:19.695362+0000","last_became_peered":"2024-04-06T13:48:19.695362+0000","last_unstale":"2024-04-06T13:48:21.097813+0000","last_undegraded":"2024-04-06T13:48:21.097813+0000","last_fullsized":"2024-04-06T13:48:21.097813+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:26:21.699641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098450+0000","last_change":"2024-04-06T13:48:19.700300+0000","last_active":"2024-04-06T13:48:21.098450+0000","last_peered":"2024-04-06T13:48:21.098450+0000","last_clean":"2024-04-06T13:48:21.098450+0000","last_became_active":"2024-04-06T13:48:19.700054+0000","last_became_peered":"2024-04-06T13:48:19.700054+0000","last_unstale":"2024-04-06T13:48:21.098450+0000","last_undegraded":"2024-04-06T13:48:21.098450+0000","last_fullsized":"2024-04-06T13:48:21.098450+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:30:14.717681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097799+0000","last_change":"2024-04-06T13:48:19.700151+0000","last_active":"2024-04-06T13:48:21.097799+0000","last_peered":"2024-04-06T13:48:21.097799+0000","last_clean":"2024-04-06T13:48:21.097799+0000","last_became_active":"2024-04-06T13:48:19.699882+0000","last_became_peered":"2024-04-06T13:48:19.699882+0000","last_unstale":"2024-04-06T13:48:21.097799+0000","last_undegraded":"2024-04-06T13:48:21.097799+0000","last_fullsized":"2024-04-06T13:48:21.097799+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:24.879568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683541+0000","last_change":"2024-04-06T13:48:19.690929+0000","last_active":"2024-04-06T13:48:20.683541+0000","last_peered":"2024-04-06T13:48:20.683541+0000","last_clean":"2024-04-06T13:48:20.683541+0000","last_became_active":"2024-04-06T13:48:19.690663+0000","last_became_peered":"2024-04-06T13:48:19.690663+0000","last_unstale":"2024-04-06T13:48:20.683541+0000","last_undegraded":"2024-04-06T13:48:20.683541+0000","last_fullsized":"2024-04-06T13:48:20.683541+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:52:01.901954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097559+0000","last_change":"2024-04-06T13:48:19.693212+0000","last_active":"2024-04-06T13:48:21.097559+0000","last_peered":"2024-04-06T13:48:21.097559+0000","last_clean":"2024-04-06T13:48:21.097559+0000","last_became_active":"2024-04-06T13:48:19.692957+0000","last_became_peered":"2024-04-06T13:48:19.692957+0000","last_unstale":"2024-04-06T13:48:21.097559+0000","last_undegraded":"2024-04-06T13:48:21.097559+0000","last_fullsized":"2024-04-06T13:48:21.097559+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:00:25.538231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097435+0000","last_change":"2024-04-06T13:48:19.700696+0000","last_active":"2024-04-06T13:48:21.097435+0000","last_peered":"2024-04-06T13:48:21.097435+0000","last_clean":"2024-04-06T13:48:21.097435+0000","last_became_active":"2024-04-06T13:48:19.700444+0000","last_became_peered":"2024-04-06T13:48:19.700444+0000","last_unstale":"2024-04-06T13:48:21.097435+0000","last_undegraded":"2024-04-06T13:48:21.097435+0000","last_fullsized":"2024-04-06T13:48:21.097435+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:46.094428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683996+0000","last_change":"2024-04-06T13:48:19.700988+0000","last_active":"2024-04-06T13:48:20.683996+0000","last_peered":"2024-04-06T13:48:20.683996+0000","last_clean":"2024-04-06T13:48:20.683996+0000","last_became_active":"2024-04-06T13:48:19.700718+0000","last_became_peered":"2024-04-06T13:48:19.700718+0000","last_unstale":"2024-04-06T13:48:20.683996+0000","last_undegraded":"2024-04-06T13:48:20.683996+0000","last_fullsized":"2024-04-06T13:48:20.683996+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:37:50.946459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098548+0000","last_change":"2024-04-06T13:48:19.700420+0000","last_active":"2024-04-06T13:48:21.098548+0000","last_peered":"2024-04-06T13:48:21.098548+0000","last_clean":"2024-04-06T13:48:21.098548+0000","last_became_active":"2024-04-06T13:48:19.698705+0000","last_became_peered":"2024-04-06T13:48:19.698705+0000","last_unstale":"2024-04-06T13:48:21.098548+0000","last_undegraded":"2024-04-06T13:48:21.098548+0000","last_fullsized":"2024-04-06T13:48:21.098548+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:29:58.555526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899939+0000","last_change":"2024-04-06T13:48:19.702407+0000","last_active":"2024-04-06T13:48:20.899939+0000","last_peered":"2024-04-06T13:48:20.899939+0000","last_clean":"2024-04-06T13:48:20.899939+0000","last_became_active":"2024-04-06T13:48:19.702139+0000","last_became_peered":"2024-04-06T13:48:19.702139+0000","last_unstale":"2024-04-06T13:48:20.899939+0000","last_undegraded":"2024-04-06T13:48:20.899939+0000","last_fullsized":"2024-04-06T13:48:20.899939+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:33.923465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098117+0000","last_change":"2024-04-06T13:48:19.692817+0000","last_active":"2024-04-06T13:48:21.098117+0000","last_peered":"2024-04-06T13:48:21.098117+0000","last_clean":"2024-04-06T13:48:21.098117+0000","last_became_active":"2024-04-06T13:48:19.692567+0000","last_became_peered":"2024-04-06T13:48:19.692567+0000","last_unstale":"2024-04-06T13:48:21.098117+0000","last_undegraded":"2024-04-06T13:48:21.098117+0000","last_fullsized":"2024-04-06T13:48:21.098117+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:22.432067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683285+0000","last_change":"2024-04-06T13:48:19.688698+0000","last_active":"2024-04-06T13:48:20.683285+0000","last_peered":"2024-04-06T13:48:20.683285+0000","last_clean":"2024-04-06T13:48:20.683285+0000","last_became_active":"2024-04-06T13:48:19.688377+0000","last_became_peered":"2024-04-06T13:48:19.688377+0000","last_unstale":"2024-04-06T13:48:20.683285+0000","last_undegraded":"2024-04-06T13:48:20.683285+0000","last_fullsized":"2024-04-06T13:48:20.683285+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:22.432067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683150+0000","last_change":"2024-04-06T13:48:19.698243+0000","last_active":"2024-04-06T13:48:20.683150+0000","last_peered":"2024-04-06T13:48:20.683150+0000","last_clean":"2024-04-06T13:48:20.683150+0000","last_became_active":"2024-04-06T13:48:19.697956+0000","last_became_peered":"2024-04-06T13:48:19.697956+0000","last_unstale":"2024-04-06T13:48:20.683150+0000","last_undegraded":"2024-04-06T13:48:20.683150+0000","last_fullsized":"2024-04-06T13:48:20.683150+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:26:21.699641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.900099+0000","last_change":"2024-04-06T13:48:19.694103+0000","last_active":"2024-04-06T13:48:20.900099+0000","last_peered":"2024-04-06T13:48:20.900099+0000","last_clean":"2024-04-06T13:48:20.900099+0000","last_became_active":"2024-04-06T13:48:19.693643+0000","last_became_peered":"2024-04-06T13:48:19.693643+0000","last_unstale":"2024-04-06T13:48:20.900099+0000","last_undegraded":"2024-04-06T13:48:20.900099+0000","last_fullsized":"2024-04-06T13:48:20.900099+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:00:35.709952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097980+0000","last_change":"2024-04-06T13:48:19.697522+0000","last_active":"2024-04-06T13:48:21.097980+0000","last_peered":"2024-04-06T13:48:21.097980+0000","last_clean":"2024-04-06T13:48:21.097980+0000","last_became_active":"2024-04-06T13:48:19.695505+0000","last_became_peered":"2024-04-06T13:48:19.695505+0000","last_unstale":"2024-04-06T13:48:21.097980+0000","last_undegraded":"2024-04-06T13:48:21.097980+0000","last_fullsized":"2024-04-06T13:48:21.097980+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:37:50.946459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097790+0000","last_change":"2024-04-06T13:48:19.694038+0000","last_active":"2024-04-06T13:48:21.097790+0000","last_peered":"2024-04-06T13:48:21.097790+0000","last_clean":"2024-04-06T13:48:21.097790+0000","last_became_active":"2024-04-06T13:48:19.693775+0000","last_became_peered":"2024-04-06T13:48:19.693775+0000","last_unstale":"2024-04-06T13:48:21.097790+0000","last_undegraded":"2024-04-06T13:48:21.097790+0000","last_fullsized":"2024-04-06T13:48:21.097790+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:38:06.251591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097641+0000","last_change":"2024-04-06T13:48:19.699860+0000","last_active":"2024-04-06T13:48:21.097641+0000","last_peered":"2024-04-06T13:48:21.097641+0000","last_clean":"2024-04-06T13:48:21.097641+0000","last_became_active":"2024-04-06T13:48:19.699567+0000","last_became_peered":"2024-04-06T13:48:19.699567+0000","last_unstale":"2024-04-06T13:48:21.097641+0000","last_undegraded":"2024-04-06T13:48:21.097641+0000","last_fullsized":"2024-04-06T13:48:21.097641+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:01:05.087706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899870+0000","last_change":"2024-04-06T13:48:19.697603+0000","last_active":"2024-04-06T13:48:20.899870+0000","last_peered":"2024-04-06T13:48:20.899870+0000","last_clean":"2024-04-06T13:48:20.899870+0000","last_became_active":"2024-04-06T13:48:19.697356+0000","last_became_peered":"2024-04-06T13:48:19.697356+0000","last_unstale":"2024-04-06T13:48:20.899870+0000","last_undegraded":"2024-04-06T13:48:20.899870+0000","last_fullsized":"2024-04-06T13:48:20.899870+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:44.773715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899804+0000","last_change":"2024-04-06T13:48:19.696446+0000","last_active":"2024-04-06T13:48:20.899804+0000","last_peered":"2024-04-06T13:48:20.899804+0000","last_clean":"2024-04-06T13:48:20.899804+0000","last_became_active":"2024-04-06T13:48:19.696090+0000","last_became_peered":"2024-04-06T13:48:19.696090+0000","last_unstale":"2024-04-06T13:48:20.899804+0000","last_undegraded":"2024-04-06T13:48:20.899804+0000","last_fullsized":"2024-04-06T13:48:20.899804+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:35.461214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.266186+0000","last_change":"2024-04-06T13:48:18.266186+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.266186+0000","last_undegraded":"2024-04-06T13:48:18.266186+0000","last_fullsized":"2024-04-06T13:48:18.266186+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:54:56.369118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1d","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.695623+0000","last_change":"2024-04-06T13:48:19.280859+0000","last_active":"2024-04-06T13:48:19.695623+0000","last_peered":"2024-04-06T13:48:19.695623+0000","last_clean":"2024-04-06T13:48:19.695623+0000","last_became_active":"2024-04-06T13:48:19.280512+0000","last_became_peered":"2024-04-06T13:48:19.280512+0000","last_unstale":"2024-04-06T13:48:19.695623+0000","last_undegraded":"2024-04-06T13:48:19.695623+0000","last_fullsized":"2024-04-06T13:48:19.695623+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:43.765754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1c","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.703978+0000","last_change":"2024-04-06T13:48:19.280934+0000","last_active":"2024-04-06T13:48:19.703978+0000","last_peered":"2024-04-06T13:48:19.703978+0000","last_clean":"2024-04-06T13:48:19.703978+0000","last_became_active":"2024-04-06T13:48:19.280677+0000","last_became_peered":"2024-04-06T13:48:19.280677+0000","last_unstale":"2024-04-06T13:48:19.703978+0000","last_undegraded":"2024-04-06T13:48:19.703978+0000","last_fullsized":"2024-04-06T13:48:19.703978+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:30:23.776677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098008+0000","last_change":"2024-04-06T13:48:19.695081+0000","last_active":"2024-04-06T13:48:21.098008+0000","last_peered":"2024-04-06T13:48:21.098008+0000","last_clean":"2024-04-06T13:48:21.098008+0000","last_became_active":"2024-04-06T13:48:19.694838+0000","last_became_peered":"2024-04-06T13:48:19.694838+0000","last_unstale":"2024-04-06T13:48:21.098008+0000","last_undegraded":"2024-04-06T13:48:21.098008+0000","last_fullsized":"2024-04-06T13:48:21.098008+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:37:44.732983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262863+0000","last_change":"2024-04-06T13:48:18.262863+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262863+0000","last_undegraded":"2024-04-06T13:48:18.262863+0000","last_fullsized":"2024-04-06T13:48:18.262863+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:18.513042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098240+0000","last_change":"2024-04-06T13:48:19.694881+0000","last_active":"2024-04-06T13:48:21.098240+0000","last_peered":"2024-04-06T13:48:21.098240+0000","last_clean":"2024-04-06T13:48:21.098240+0000","last_became_active":"2024-04-06T13:48:19.693362+0000","last_became_peered":"2024-04-06T13:48:19.693362+0000","last_unstale":"2024-04-06T13:48:21.098240+0000","last_undegraded":"2024-04-06T13:48:21.098240+0000","last_fullsized":"2024-04-06T13:48:21.098240+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:36:47.917157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.699585+0000","last_change":"2024-04-06T13:48:19.274234+0000","last_active":"2024-04-06T13:48:19.699585+0000","last_peered":"2024-04-06T13:48:19.699585+0000","last_clean":"2024-04-06T13:48:19.699585+0000","last_became_active":"2024-04-06T13:48:19.273998+0000","last_became_peered":"2024-04-06T13:48:19.273998+0000","last_unstale":"2024-04-06T13:48:19.699585+0000","last_undegraded":"2024-04-06T13:48:19.699585+0000","last_fullsized":"2024-04-06T13:48:19.699585+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:49:23.791293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098351+0000","last_change":"2024-04-06T13:48:19.694934+0000","last_active":"2024-04-06T13:48:21.098351+0000","last_peered":"2024-04-06T13:48:21.098351+0000","last_clean":"2024-04-06T13:48:21.098351+0000","last_became_active":"2024-04-06T13:48:19.692682+0000","last_became_peered":"2024-04-06T13:48:19.692682+0000","last_unstale":"2024-04-06T13:48:21.098351+0000","last_undegraded":"2024-04-06T13:48:21.098351+0000","last_fullsized":"2024-04-06T13:48:21.098351+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:40:43.027987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.684729+0000","last_change":"2024-04-06T13:48:19.279719+0000","last_active":"2024-04-06T13:48:19.684729+0000","last_peered":"2024-04-06T13:48:19.684729+0000","last_clean":"2024-04-06T13:48:19.684729+0000","last_became_active":"2024-04-06T13:48:19.279404+0000","last_became_peered":"2024-04-06T13:48:19.279404+0000","last_unstale":"2024-04-06T13:48:19.684729+0000","last_undegraded":"2024-04-06T13:48:19.684729+0000","last_fullsized":"2024-04-06T13:48:19.684729+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:42:35.927858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.265371+0000","last_change":"2024-04-06T13:48:18.265371+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.265371+0000","last_undegraded":"2024-04-06T13:48:18.265371+0000","last_fullsized":"2024-04-06T13:48:18.265371+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:02:13.092175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098078+0000","last_change":"2024-04-06T13:48:19.694595+0000","last_active":"2024-04-06T13:48:21.098078+0000","last_peered":"2024-04-06T13:48:21.098078+0000","last_clean":"2024-04-06T13:48:21.098078+0000","last_became_active":"2024-04-06T13:48:19.692548+0000","last_became_peered":"2024-04-06T13:48:19.692548+0000","last_unstale":"2024-04-06T13:48:21.098078+0000","last_undegraded":"2024-04-06T13:48:21.098078+0000","last_fullsized":"2024-04-06T13:48:21.098078+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:39:01.587642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.684653+0000","last_change":"2024-04-06T13:48:19.273745+0000","last_active":"2024-04-06T13:48:19.684653+0000","last_peered":"2024-04-06T13:48:19.684653+0000","last_clean":"2024-04-06T13:48:19.684653+0000","last_became_active":"2024-04-06T13:48:19.273478+0000","last_became_peered":"2024-04-06T13:48:19.273478+0000","last_unstale":"2024-04-06T13:48:19.684653+0000","last_undegraded":"2024-04-06T13:48:19.684653+0000","last_fullsized":"2024-04-06T13:48:19.684653+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:00:55.130902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683007+0000","last_change":"2024-04-06T13:48:19.688682+0000","last_active":"2024-04-06T13:48:20.683007+0000","last_peered":"2024-04-06T13:48:20.683007+0000","last_clean":"2024-04-06T13:48:20.683007+0000","last_became_active":"2024-04-06T13:48:19.688397+0000","last_became_peered":"2024-04-06T13:48:19.688397+0000","last_unstale":"2024-04-06T13:48:20.683007+0000","last_undegraded":"2024-04-06T13:48:20.683007+0000","last_fullsized":"2024-04-06T13:48:20.683007+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:39:01.587642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.683105+0000","last_change":"2024-04-06T13:48:19.278056+0000","last_active":"2024-04-06T13:48:19.683105+0000","last_peered":"2024-04-06T13:48:19.683105+0000","last_clean":"2024-04-06T13:48:19.683105+0000","last_became_active":"2024-04-06T13:48:19.277796+0000","last_became_peered":"2024-04-06T13:48:19.277796+0000","last_unstale":"2024-04-06T13:48:19.683105+0000","last_undegraded":"2024-04-06T13:48:19.683105+0000","last_fullsized":"2024-04-06T13:48:19.683105+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:31:43.320734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097587+0000","last_change":"2024-04-06T13:48:19.700325+0000","last_active":"2024-04-06T13:48:21.097587+0000","last_peered":"2024-04-06T13:48:21.097587+0000","last_clean":"2024-04-06T13:48:21.097587+0000","last_became_active":"2024-04-06T13:48:19.698751+0000","last_became_peered":"2024-04-06T13:48:19.698751+0000","last_unstale":"2024-04-06T13:48:21.097587+0000","last_undegraded":"2024-04-06T13:48:21.097587+0000","last_fullsized":"2024-04-06T13:48:21.097587+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:16.165915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262594+0000","last_change":"2024-04-06T13:48:18.262594+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262594+0000","last_undegraded":"2024-04-06T13:48:18.262594+0000","last_fullsized":"2024-04-06T13:48:18.262594+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:06:05.488014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097951+0000","last_change":"2024-04-06T13:48:19.692164+0000","last_active":"2024-04-06T13:48:21.097951+0000","last_peered":"2024-04-06T13:48:21.097951+0000","last_clean":"2024-04-06T13:48:21.097951+0000","last_became_active":"2024-04-06T13:48:19.691910+0000","last_became_peered":"2024-04-06T13:48:19.691910+0000","last_unstale":"2024-04-06T13:48:21.097951+0000","last_undegraded":"2024-04-06T13:48:21.097951+0000","last_fullsized":"2024-04-06T13:48:21.097951+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:19:27.875509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.265322+0000","last_change":"2024-04-06T13:48:18.265322+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.265322+0000","last_undegraded":"2024-04-06T13:48:18.265322+0000","last_fullsized":"2024-04-06T13:48:18.265322+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:56:55.759595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899896+0000","last_change":"2024-04-06T13:48:19.693858+0000","last_active":"2024-04-06T13:48:20.899896+0000","last_peered":"2024-04-06T13:48:20.899896+0000","last_clean":"2024-04-06T13:48:20.899896+0000","last_became_active":"2024-04-06T13:48:19.693515+0000","last_became_peered":"2024-04-06T13:48:19.693515+0000","last_unstale":"2024-04-06T13:48:20.899896+0000","last_undegraded":"2024-04-06T13:48:20.899896+0000","last_fullsized":"2024-04-06T13:48:20.899896+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:10:26.852793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262385+0000","last_change":"2024-04-06T13:48:18.262385+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262385+0000","last_undegraded":"2024-04-06T13:48:18.262385+0000","last_fullsized":"2024-04-06T13:48:18.262385+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:31:24.701348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098230+0000","last_change":"2024-04-06T13:48:19.694858+0000","last_active":"2024-04-06T13:48:21.098230+0000","last_peered":"2024-04-06T13:48:21.098230+0000","last_clean":"2024-04-06T13:48:21.098230+0000","last_became_active":"2024-04-06T13:48:19.693049+0000","last_became_peered":"2024-04-06T13:48:19.693049+0000","last_unstale":"2024-04-06T13:48:21.098230+0000","last_undegraded":"2024-04-06T13:48:21.098230+0000","last_fullsized":"2024-04-06T13:48:21.098230+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:19:00.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.261860+0000","last_change":"2024-04-06T13:48:18.261860+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.261860+0000","last_undegraded":"2024-04-06T13:48:18.261860+0000","last_fullsized":"2024-04-06T13:48:18.261860+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:35:17.934513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.263085+0000","last_change":"2024-04-06T13:48:18.263085+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.263085+0000","last_undegraded":"2024-04-06T13:48:18.263085+0000","last_fullsized":"2024-04-06T13:48:18.263085+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:02:23.692226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.10","version":"247'8","reported_seq":19,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.704884+0000","last_change":"2024-04-06T13:48:19.274645+0000","last_active":"2024-04-06T13:48:19.704884+0000","last_peered":"2024-04-06T13:48:19.704884+0000","last_clean":"2024-04-06T13:48:19.704884+0000","last_became_active":"2024-04-06T13:48:19.274308+0000","last_became_peered":"2024-04-06T13:48:19.274308+0000","last_unstale":"2024-04-06T13:48:19.704884+0000","last_undegraded":"2024-04-06T13:48:19.704884+0000","last_fullsized":"2024-04-06T13:48:19.704884+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:31:15.493970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.265775+0000","last_change":"2024-04-06T13:48:18.265775+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.265775+0000","last_undegraded":"2024-04-06T13:48:18.265775+0000","last_fullsized":"2024-04-06T13:48:18.265775+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:49.041735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.e","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.706958+0000","last_change":"2024-04-06T13:48:19.275494+0000","last_active":"2024-04-06T13:48:19.706958+0000","last_peered":"2024-04-06T13:48:19.706958+0000","last_clean":"2024-04-06T13:48:19.706958+0000","last_became_active":"2024-04-06T13:48:19.275194+0000","last_became_peered":"2024-04-06T13:48:19.275194+0000","last_unstale":"2024-04-06T13:48:19.706958+0000","last_undegraded":"2024-04-06T13:48:19.706958+0000","last_fullsized":"2024-04-06T13:48:19.706958+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:11:29.081483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899798+0000","last_change":"2024-04-06T13:48:19.702505+0000","last_active":"2024-04-06T13:48:20.899798+0000","last_peered":"2024-04-06T13:48:20.899798+0000","last_clean":"2024-04-06T13:48:20.899798+0000","last_became_active":"2024-04-06T13:48:19.702245+0000","last_became_peered":"2024-04-06T13:48:19.702245+0000","last_unstale":"2024-04-06T13:48:20.899798+0000","last_undegraded":"2024-04-06T13:48:20.899798+0000","last_fullsized":"2024-04-06T13:48:20.899798+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:45.672315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.695714+0000","last_change":"2024-04-06T13:48:19.281905+0000","last_active":"2024-04-06T13:48:19.695714+0000","last_peered":"2024-04-06T13:48:19.695714+0000","last_clean":"2024-04-06T13:48:19.695714+0000","last_became_active":"2024-04-06T13:48:19.281052+0000","last_became_peered":"2024-04-06T13:48:19.281052+0000","last_unstale":"2024-04-06T13:48:19.695714+0000","last_undegraded":"2024-04-06T13:48:19.695714+0000","last_fullsized":"2024-04-06T13:48:19.695714+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:48.100692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097580+0000","last_change":"2024-04-06T13:48:19.700428+0000","last_active":"2024-04-06T13:48:21.097580+0000","last_peered":"2024-04-06T13:48:21.097580+0000","last_clean":"2024-04-06T13:48:21.097580+0000","last_became_active":"2024-04-06T13:48:19.699896+0000","last_became_peered":"2024-04-06T13:48:19.699896+0000","last_unstale":"2024-04-06T13:48:21.097580+0000","last_undegraded":"2024-04-06T13:48:21.097580+0000","last_fullsized":"2024-04-06T13:48:21.097580+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:00:56.138863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262279+0000","last_change":"2024-04-06T13:48:18.262279+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262279+0000","last_undegraded":"2024-04-06T13:48:18.262279+0000","last_fullsized":"2024-04-06T13:48:18.262279+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:55:10.767276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097956+0000","last_change":"2024-04-06T13:48:19.697815+0000","last_active":"2024-04-06T13:48:21.097956+0000","last_peered":"2024-04-06T13:48:21.097956+0000","last_clean":"2024-04-06T13:48:21.097956+0000","last_became_active":"2024-04-06T13:48:19.695354+0000","last_became_peered":"2024-04-06T13:48:19.695354+0000","last_unstale":"2024-04-06T13:48:21.097956+0000","last_undegraded":"2024-04-06T13:48:21.097956+0000","last_fullsized":"2024-04-06T13:48:21.097956+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:38:12.538926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.700247+0000","last_change":"2024-04-06T13:48:19.278849+0000","last_active":"2024-04-06T13:48:19.700247+0000","last_peered":"2024-04-06T13:48:19.700247+0000","last_clean":"2024-04-06T13:48:19.700247+0000","last_became_active":"2024-04-06T13:48:19.278604+0000","last_became_peered":"2024-04-06T13:48:19.278604+0000","last_unstale":"2024-04-06T13:48:19.700247+0000","last_undegraded":"2024-04-06T13:48:19.700247+0000","last_fullsized":"2024-04-06T13:48:19.700247+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:19:26.867548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.900001+0000","last_change":"2024-04-06T13:48:19.696316+0000","last_active":"2024-04-06T13:48:20.900001+0000","last_peered":"2024-04-06T13:48:20.900001+0000","last_clean":"2024-04-06T13:48:20.900001+0000","last_became_active":"2024-04-06T13:48:19.696025+0000","last_became_peered":"2024-04-06T13:48:19.696025+0000","last_unstale":"2024-04-06T13:48:20.900001+0000","last_undegraded":"2024-04-06T13:48:20.900001+0000","last_fullsized":"2024-04-06T13:48:20.900001+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:33:27.603807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.680627+0000","last_change":"2024-04-06T13:48:19.282165+0000","last_active":"2024-04-06T13:48:19.680627+0000","last_peered":"2024-04-06T13:48:19.680627+0000","last_clean":"2024-04-06T13:48:19.680627+0000","last_became_active":"2024-04-06T13:48:19.281892+0000","last_became_peered":"2024-04-06T13:48:19.281892+0000","last_unstale":"2024-04-06T13:48:19.680627+0000","last_undegraded":"2024-04-06T13:48:19.680627+0000","last_fullsized":"2024-04-06T13:48:19.680627+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:10:25.844832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683665+0000","last_change":"2024-04-06T13:48:19.699890+0000","last_active":"2024-04-06T13:48:20.683665+0000","last_peered":"2024-04-06T13:48:20.683665+0000","last_clean":"2024-04-06T13:48:20.683665+0000","last_became_active":"2024-04-06T13:48:19.699566+0000","last_became_peered":"2024-04-06T13:48:19.699566+0000","last_unstale":"2024-04-06T13:48:20.683665+0000","last_undegraded":"2024-04-06T13:48:20.683665+0000","last_fullsized":"2024-04-06T13:48:20.683665+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:10:02.935661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.682503+0000","last_change":"2024-04-06T13:48:19.276781+0000","last_active":"2024-04-06T13:48:19.682503+0000","last_peered":"2024-04-06T13:48:19.682503+0000","last_clean":"2024-04-06T13:48:19.682503+0000","last_became_active":"2024-04-06T13:48:19.275438+0000","last_became_peered":"2024-04-06T13:48:19.275438+0000","last_unstale":"2024-04-06T13:48:19.682503+0000","last_undegraded":"2024-04-06T13:48:19.682503+0000","last_fullsized":"2024-04-06T13:48:19.682503+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:35.170771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683912+0000","last_change":"2024-04-06T13:48:19.685464+0000","last_active":"2024-04-06T13:48:20.683912+0000","last_peered":"2024-04-06T13:48:20.683912+0000","last_clean":"2024-04-06T13:48:20.683912+0000","last_became_active":"2024-04-06T13:48:19.685139+0000","last_became_peered":"2024-04-06T13:48:19.685139+0000","last_unstale":"2024-04-06T13:48:20.683912+0000","last_undegraded":"2024-04-06T13:48:20.683912+0000","last_fullsized":"2024-04-06T13:48:20.683912+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:19:00.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.683371+0000","last_change":"2024-04-06T13:48:19.279096+0000","last_active":"2024-04-06T13:48:19.683371+0000","last_peered":"2024-04-06T13:48:19.683371+0000","last_clean":"2024-04-06T13:48:19.683371+0000","last_became_active":"2024-04-06T13:48:19.278774+0000","last_became_peered":"2024-04-06T13:48:19.278774+0000","last_unstale":"2024-04-06T13:48:19.683371+0000","last_undegraded":"2024-04-06T13:48:19.683371+0000","last_fullsized":"2024-04-06T13:48:19.683371+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:46:06.202483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097785+0000","last_change":"2024-04-06T13:48:19.697387+0000","last_active":"2024-04-06T13:48:21.097785+0000","last_peered":"2024-04-06T13:48:21.097785+0000","last_clean":"2024-04-06T13:48:21.097785+0000","last_became_active":"2024-04-06T13:48:19.697096+0000","last_became_peered":"2024-04-06T13:48:19.697096+0000","last_unstale":"2024-04-06T13:48:21.097785+0000","last_undegraded":"2024-04-06T13:48:21.097785+0000","last_fullsized":"2024-04-06T13:48:21.097785+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:52:01.901954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.698698+0000","last_change":"2024-04-06T13:48:19.271906+0000","last_active":"2024-04-06T13:48:19.698698+0000","last_peered":"2024-04-06T13:48:19.698698+0000","last_clean":"2024-04-06T13:48:19.698698+0000","last_became_active":"2024-04-06T13:48:19.271643+0000","last_became_peered":"2024-04-06T13:48:19.271643+0000","last_unstale":"2024-04-06T13:48:19.698698+0000","last_undegraded":"2024-04-06T13:48:19.698698+0000","last_fullsized":"2024-04-06T13:48:19.698698+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:54:44.219906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262063+0000","last_change":"2024-04-06T13:48:18.262063+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262063+0000","last_undegraded":"2024-04-06T13:48:18.262063+0000","last_fullsized":"2024-04-06T13:48:18.262063+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:18.981232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097491+0000","last_change":"2024-04-06T13:48:19.694094+0000","last_active":"2024-04-06T13:48:21.097491+0000","last_peered":"2024-04-06T13:48:21.097491+0000","last_clean":"2024-04-06T13:48:21.097491+0000","last_became_active":"2024-04-06T13:48:19.692958+0000","last_became_peered":"2024-04-06T13:48:19.692958+0000","last_unstale":"2024-04-06T13:48:21.097491+0000","last_undegraded":"2024-04-06T13:48:21.097491+0000","last_fullsized":"2024-04-06T13:48:21.097491+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:04:10.547739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.684628+0000","last_change":"2024-04-06T13:48:19.272078+0000","last_active":"2024-04-06T13:48:19.684628+0000","last_peered":"2024-04-06T13:48:19.684628+0000","last_clean":"2024-04-06T13:48:19.684628+0000","last_became_active":"2024-04-06T13:48:19.271672+0000","last_became_peered":"2024-04-06T13:48:19.271672+0000","last_unstale":"2024-04-06T13:48:19.684628+0000","last_undegraded":"2024-04-06T13:48:19.684628+0000","last_fullsized":"2024-04-06T13:48:19.684628+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:23.871607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.685267+0000","last_change":"2024-04-06T13:48:19.278939+0000","last_active":"2024-04-06T13:48:19.685267+0000","last_peered":"2024-04-06T13:48:19.685267+0000","last_clean":"2024-04-06T13:48:19.685267+0000","last_became_active":"2024-04-06T13:48:19.278677+0000","last_became_peered":"2024-04-06T13:48:19.278677+0000","last_unstale":"2024-04-06T13:48:19.685267+0000","last_undegraded":"2024-04-06T13:48:19.685267+0000","last_fullsized":"2024-04-06T13:48:19.685267+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:15.157954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097917+0000","last_change":"2024-04-06T13:48:19.693016+0000","last_active":"2024-04-06T13:48:21.097917+0000","last_peered":"2024-04-06T13:48:21.097917+0000","last_clean":"2024-04-06T13:48:21.097917+0000","last_became_active":"2024-04-06T13:48:19.692760+0000","last_became_peered":"2024-04-06T13:48:19.692760+0000","last_unstale":"2024-04-06T13:48:21.097917+0000","last_undegraded":"2024-04-06T13:48:21.097917+0000","last_fullsized":"2024-04-06T13:48:21.097917+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:41:37.714397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.682132+0000","last_change":"2024-04-06T13:48:19.282189+0000","last_active":"2024-04-06T13:48:19.682132+0000","last_peered":"2024-04-06T13:48:19.682132+0000","last_clean":"2024-04-06T13:48:19.682132+0000","last_became_active":"2024-04-06T13:48:19.281907+0000","last_became_peered":"2024-04-06T13:48:19.281907+0000","last_unstale":"2024-04-06T13:48:19.682132+0000","last_undegraded":"2024-04-06T13:48:19.682132+0000","last_fullsized":"2024-04-06T13:48:19.682132+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:34.453253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.261299+0000","last_change":"2024-04-06T13:48:18.261299+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.261299+0000","last_undegraded":"2024-04-06T13:48:18.261299+0000","last_fullsized":"2024-04-06T13:48:18.261299+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:51:53.597002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899957+0000","last_change":"2024-04-06T13:48:19.704846+0000","last_active":"2024-04-06T13:48:20.899957+0000","last_peered":"2024-04-06T13:48:20.899957+0000","last_clean":"2024-04-06T13:48:20.899957+0000","last_became_active":"2024-04-06T13:48:19.704420+0000","last_became_peered":"2024-04-06T13:48:19.704420+0000","last_unstale":"2024-04-06T13:48:20.899957+0000","last_undegraded":"2024-04-06T13:48:20.899957+0000","last_fullsized":"2024-04-06T13:48:20.899957+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:36.178732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.684819+0000","last_change":"2024-04-06T13:48:19.277530+0000","last_active":"2024-04-06T13:48:19.684819+0000","last_peered":"2024-04-06T13:48:19.684819+0000","last_clean":"2024-04-06T13:48:19.684819+0000","last_became_active":"2024-04-06T13:48:19.276966+0000","last_became_peered":"2024-04-06T13:48:19.276966+0000","last_unstale":"2024-04-06T13:48:19.684819+0000","last_undegraded":"2024-04-06T13:48:19.684819+0000","last_fullsized":"2024-04-06T13:48:19.684819+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:42:14.962783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683820+0000","last_change":"2024-04-06T13:48:19.688861+0000","last_active":"2024-04-06T13:48:20.683820+0000","last_peered":"2024-04-06T13:48:20.683820+0000","last_clean":"2024-04-06T13:48:20.683820+0000","last_became_active":"2024-04-06T13:48:19.688496+0000","last_became_peered":"2024-04-06T13:48:19.688496+0000","last_unstale":"2024-04-06T13:48:20.683820+0000","last_undegraded":"2024-04-06T13:48:20.683820+0000","last_fullsized":"2024-04-06T13:48:20.683820+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:36:47.917157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.680854+0000","last_change":"2024-04-06T13:48:19.276467+0000","last_active":"2024-04-06T13:48:19.680854+0000","last_peered":"2024-04-06T13:48:19.680854+0000","last_clean":"2024-04-06T13:48:19.680854+0000","last_became_active":"2024-04-06T13:48:19.275988+0000","last_became_peered":"2024-04-06T13:48:19.275988+0000","last_unstale":"2024-04-06T13:48:19.680854+0000","last_undegraded":"2024-04-06T13:48:19.680854+0000","last_fullsized":"2024-04-06T13:48:19.680854+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:44.664354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1e","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.694613+0000","last_change":"2024-04-06T13:48:19.277300+0000","last_active":"2024-04-06T13:48:19.694613+0000","last_peered":"2024-04-06T13:48:19.694613+0000","last_clean":"2024-04-06T13:48:19.694613+0000","last_became_active":"2024-04-06T13:48:19.276795+0000","last_became_peered":"2024-04-06T13:48:19.276795+0000","last_unstale":"2024-04-06T13:48:19.694613+0000","last_undegraded":"2024-04-06T13:48:19.694613+0000","last_fullsized":"2024-04-06T13:48:19.694613+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:54:25.950801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"233'370","reported_seq":1333,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-06T13:48:15.595868+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:48:15.595868+0000","last_peered":"2024-04-06T13:48:15.595868+0000","last_clean":"2024-04-06T13:48:15.595868+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:48:15.595868+0000","last_undegraded":"2024-04-06T13:48:15.595868+0000","last_fullsized":"2024-04-06T13:48:15.595868+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":82,"acting":82,"num_store_stats":5},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":70,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21,"ondisk_log_size":21,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575992,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56856,"kb_used_data":3596,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314984,"statfs":{"total":96636764160,"available":96578543616,"internally_reserved":0,"allocated":3682304,"data_stored":2174385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57899999999999996,"15min":0.59499999999999997},"min":{"1min":0.35499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.72199999999999998,"5min":0.78400000000000003,"15min":0.82499999999999996},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56799999999999995,"15min":0.58999999999999997},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78500000000000003,"5min":0.80800000000000005,"15min":0.82899999999999996},"last":0.753}]},{"osd":1,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.55000000000000004,"15min":0.57899999999999996},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77200000000000002,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.57699999999999996,"15min":0.59599999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.69899999999999995,"5min":0.83399999999999996,"15min":0.85299999999999998},"last":0.78100000000000003}]},{"osd":2,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.56299999999999994,"15min":0.58499999999999996},"min":{"1min":0.33200000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69399999999999995,"5min":0.81699999999999995,"15min":0.83899999999999997},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.58599999999999997,"15min":0.60099999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69999999999999996,"5min":0.755,"15min":0.79700000000000004},"last":0.76900000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.59399999999999997,"15min":0.60599999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.67200000000000004,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58799999999999997,"15min":0.60299999999999998},"min":{"1min":0.39000000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78900000000000003,"5min":0.82099999999999995,"15min":0.84099999999999997},"last":0.66500000000000004}]},{"osd":4,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45800000000000002,"15min":0.46600000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.81599999999999995},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.36799999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.82699999999999996},"last":0.53500000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.498,"15min":0.498},"min":{"1min":0.315,"5min":0.315,"15min":0.308},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56899999999999995,"15min":0.57799999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33600000000000002},"max":{"1min":0.74099999999999999,"5min":0.80700000000000005,"15min":0.85399999999999998},"last":0.55200000000000005}]},{"osd":6,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.34899999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.879},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57099999999999995,"15min":0.58699999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81799999999999995,"5min":0.84899999999999998,"15min":0.89800000000000002},"last":0.57199999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575991,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56260,"kb_used_data":3000,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315580,"statfs":{"total":96636764160,"available":96579153920,"internally_reserved":0,"allocated":3072000,"data_stored":1576789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.53600000000000003,"15min":0.55700000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69399999999999995,"5min":0.79700000000000004,"15min":0.83799999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.54100000000000004,"15min":0.56499999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.62,"5min":0.76900000000000002,"15min":0.81999999999999995},"last":0.61899999999999999}]},{"osd":1,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.60399999999999998,"15min":0.624},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76000000000000001,"5min":0.81699999999999995,"15min":0.85799999999999998},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.622,"15min":0.64400000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.378},"max":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.82699999999999996},"last":0.67800000000000005}]},{"osd":2,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56999999999999995,"15min":0.59399999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72099999999999997,"5min":0.79000000000000004,"15min":0.82899999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.56299999999999994,"15min":0.58599999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.80700000000000005},"last":0.58499999999999996}]},{"osd":3,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.58599999999999997,"15min":0.60199999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.73199999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.61899999999999999,"15min":0.64300000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.36099999999999999},"max":{"1min":0.80200000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.68899999999999995}]},{"osd":4,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70599999999999996,"5min":0.72299999999999998,"15min":0.81399999999999995},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.51100000000000001,"15min":0.53800000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.66400000000000003,"5min":0.82099999999999995,"15min":0.84099999999999997},"last":0.70799999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.48899999999999999,"15min":0.50600000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.33300000000000002},"max":{"1min":0.63,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52000000000000002,"15min":0.53100000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67400000000000004,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.63500000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53900000000000003,"15min":0.54800000000000004},"min":{"1min":0.34899999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.67700000000000005,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55300000000000005,"15min":0.56899999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.32800000000000001},"max":{"1min":0.71199999999999997,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.54800000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575991,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60404,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311436,"statfs":{"total":96636764160,"available":96574910464,"internally_reserved":0,"allocated":3121152,"data_stored":1603524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.38100000000000001,"15min":0.39100000000000001},"min":{"1min":0.248,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.378,"5min":0.621,"15min":0.621},"last":0.441},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.41799999999999998,"15min":0.42699999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.501,"5min":0.60799999999999998,"15min":0.69399999999999995},"last":0.27500000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.41299999999999998,"15min":0.42399999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.46899999999999997,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.44900000000000001,"15min":0.46400000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.55400000000000005,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.34799999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.46999999999999997,"15min":0.47999999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.52800000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.42699999999999999,"15min":0.439},"min":{"1min":0.28799999999999998,"5min":0.27600000000000002,"15min":0.26300000000000001},"max":{"1min":0.45500000000000002,"5min":0.63500000000000001,"15min":0.66100000000000003},"last":0.33100000000000002}]},{"osd":4,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.628,"15min":0.64400000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.33800000000000002},"max":{"1min":0.70099999999999996,"5min":0.89900000000000002,"15min":0.97399999999999998},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.64400000000000002,"15min":0.65500000000000003},"min":{"1min":0.46100000000000002,"5min":0.44500000000000001,"15min":0.44400000000000001},"max":{"1min":0.73499999999999999,"5min":0.94599999999999995,"15min":0.995},"last":0.55400000000000005}]},{"osd":5,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.64100000000000001,"15min":0.65300000000000002},"min":{"1min":0.42999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73899999999999999,"5min":0.96899999999999997,"15min":1.073},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.65700000000000003,"15min":0.66800000000000004},"min":{"1min":0.45200000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.72799999999999998,"5min":0.96499999999999997,"15min":1.0209999999999999},"last":0.47899999999999998}]},{"osd":6,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.46300000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73899999999999999,"5min":0.89200000000000002,"15min":1},"last":0.503},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.67100000000000004,"15min":0.68400000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.72499999999999998,"5min":0.94599999999999995,"15min":1.036},"last":0.48799999999999999}]},{"osd":7,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.69399999999999995,"15min":0.69499999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85399999999999998,"5min":0.94899999999999995,"15min":1.0649999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65700000000000003,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.76300000000000001,"5min":0.92900000000000005,"15min":1.056},"last":0.46899999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575991,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61004,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310836,"statfs":{"total":96636764160,"available":96574296064,"internally_reserved":0,"allocated":3727360,"data_stored":2195456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.38200000000000001,"15min":0.39300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.45100000000000001,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.39300000000000002,"15min":0.41699999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.48199999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.41399999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.40100000000000002,"15min":0.42499999999999999},"min":{"1min":0.26100000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49399999999999999,"5min":0.66700000000000004,"15min":0.70099999999999996},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.42499999999999999,"15min":0.443},"min":{"1min":0.28399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.47999999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.33300000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.40699999999999997,"15min":0.42499999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.50700000000000001,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.45700000000000002,"15min":0.47799999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.51900000000000002,"5min":0.72999999999999998,"15min":0.75700000000000001},"last":0.442}]},{"osd":4,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.60499999999999998,"15min":0.64100000000000001},"min":{"1min":0.39100000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.80900000000000005,"5min":0.93100000000000005,"15min":1.0329999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.61199999999999999,"15min":0.64300000000000002},"min":{"1min":0.372,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88500000000000001,"5min":0.999,"15min":0.999},"last":0.57899999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.60199999999999998,"15min":0.63900000000000001},"min":{"1min":0.39300000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.0009999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.63800000000000001,"15min":0.66300000000000003},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.749,"5min":0.92800000000000005,"15min":1.052},"last":0.501}]},{"osd":6,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.64500000000000002,"15min":0.67400000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.874,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.64200000000000002,"15min":0.66900000000000004},"min":{"1min":0.40500000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77500000000000002,"5min":0.96399999999999997,"15min":1.085},"last":0.67600000000000005}]},{"osd":7,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.66300000000000003,"15min":0.69099999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.80700000000000005,"5min":1.01,"15min":1.121},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.64700000000000002,"15min":0.67200000000000004},"min":{"1min":0.42199999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.93700000000000006,"5min":0.95799999999999996,"15min":1.0229999999999999},"last":0.69399999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575993,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60420,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311420,"statfs":{"total":96636764160,"available":96574894080,"internally_reserved":0,"allocated":3129344,"data_stored":1604176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.376,"15min":0.38400000000000001},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.42299999999999999,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.40600000000000003,"15min":0.42199999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.249},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":1.044},"last":0.27900000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.38500000000000001,"15min":0.40400000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51100000000000001,"5min":0.54000000000000004,"15min":0.753},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.42799999999999999,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.82199999999999995},"last":0.35999999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.439,"15min":0.45800000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.53200000000000003,"5min":0.69699999999999995,"15min":0.70299999999999996},"last":0.374},{"interface":"front","average":{"1min":0.377,"5min":0.437,"15min":0.44900000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.504,"5min":0.64500000000000002,"15min":0.68600000000000005},"last":0.41499999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.621,"15min":0.63900000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.376},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.64400000000000002,"15min":0.66100000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.88900000000000001,"5min":1.022,"15min":1.022},"last":0.629}]},{"osd":5,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.64300000000000002,"15min":0.66300000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88100000000000001,"5min":0.98599999999999999,"15min":1.0149999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.65100000000000002,"15min":0.66700000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.65200000000000002,"15min":0.67200000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92000000000000004,"5min":0.97199999999999998,"15min":1.091},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.67600000000000005,"15min":0.69499999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.96099999999999997},"last":0.51200000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.66600000000000004,"15min":0.68799999999999994},"min":{"1min":0.42699999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.95399999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.68300000000000005,"15min":0.70799999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.85299999999999998,"5min":1.01,"15min":1.01},"last":0.53600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575991,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48084,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323756,"statfs":{"total":96636764160,"available":96587526144,"internally_reserved":0,"allocated":3088384,"data_stored":1584017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39000000000000001,"15min":0.38900000000000001},"min":{"1min":0.26000000000000001,"5min":0.255,"15min":0.23499999999999999},"max":{"1min":0.55200000000000005,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41199999999999998,"15min":0.41299999999999998},"min":{"1min":0.28799999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.56899999999999995,"5min":0.622,"15min":0.66700000000000004},"last":0.34000000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40699999999999997,"15min":0.40600000000000003},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.65700000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.438,"15min":0.44400000000000001},"min":{"1min":0.32800000000000001,"5min":0.27500000000000002,"15min":0.26600000000000001},"max":{"1min":0.63,"5min":0.65400000000000003,"15min":0.75800000000000001},"last":0.39900000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48299999999999998,"15min":0.48599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.5,"15min":0.503},"min":{"1min":0.38600000000000001,"5min":0.34599999999999997,"15min":0.32900000000000001},"max":{"1min":0.67800000000000005,"5min":0.69299999999999995,"15min":0.79300000000000004},"last":0.49199999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.439,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.876,"5min":1.002,"15min":1.018},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66100000000000003,"15min":0.65300000000000002},"min":{"1min":0.497,"5min":0.45900000000000002,"15min":0.375},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.60399999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.51900000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.95699999999999996,"5min":0.97299999999999998,"15min":0.999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.46200000000000002,"5min":0.437,"15min":0.437},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.61499999999999999}]},{"osd":6,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67500000000000004,"15min":0.66900000000000004},"min":{"1min":0.41499999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.63},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.68600000000000005,"15min":0.68400000000000005},"min":{"1min":0.48799999999999999,"5min":0.46700000000000003,"15min":0.41099999999999998},"max":{"1min":0.88800000000000001,"5min":0.96199999999999997,"15min":0.98799999999999999},"last":0.57399999999999995}]},{"osd":7,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68500000000000005,"15min":0.68000000000000005},"min":{"1min":0.46100000000000002,"5min":0.432,"15min":0.40999999999999998},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70299999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.024},"last":0.71199999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575992,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48132,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323708,"statfs":{"total":96636764160,"available":96587476992,"internally_reserved":0,"allocated":3129344,"data_stored":1604613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55900000000000005,"15min":0.58699999999999997},"min":{"1min":0.40699999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.753,"5min":0.876,"15min":0.88500000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.39300000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81100000000000005,"5min":1.202,"15min":1.202},"last":0.48299999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.61299999999999999,"15min":0.64000000000000001},"min":{"1min":0.40799999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71699999999999997,"5min":1.256,"15min":1.256},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.624,"15min":0.63},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.75900000000000001,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.56499999999999995}]},{"osd":2,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.57999999999999996,"15min":0.59999999999999998},"min":{"1min":0.41599999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77500000000000002,"5min":0.84599999999999997,"15min":0.877},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.60599999999999998,"15min":0.61599999999999999},"min":{"1min":0.39600000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.97899999999999998,"5min":1.294,"15min":1.294},"last":0.41199999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84499999999999997,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.60899999999999999,"15min":0.626},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.78800000000000003,"5min":0.90200000000000002,"15min":0.91100000000000003},"last":0.55400000000000005}]},{"osd":5,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55500000000000005,"15min":0.52000000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.92800000000000005,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.56399999999999995,"15min":0.54100000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.95199999999999996,"5min":4.931,"15min":4.931},"last":0.63500000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53500000000000003,"15min":0.53300000000000003},"min":{"1min":0.34999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.505},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34699999999999998},"max":{"1min":1.0609999999999999,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.62}]},{"osd":7,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59499999999999997,"15min":0.58199999999999996},"min":{"1min":0.36899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.91000000000000003,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59399999999999997,"15min":0.60199999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.373},"max":{"1min":0.86899999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.53200000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575992,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64508,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307332,"statfs":{"total":96636764160,"available":96570707968,"internally_reserved":0,"allocated":3121152,"data_stored":1603524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.57399999999999995,"15min":0.58499999999999996},"min":{"1min":0.36399999999999999,"5min":0.35899999999999999,"15min":0.33000000000000002},"max":{"1min":0.68000000000000005,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58099999999999996,"15min":0.59399999999999997},"min":{"1min":0.38100000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.76600000000000001,"5min":0.84699999999999998,"15min":0.89200000000000002},"last":0.63200000000000001}]},{"osd":1,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.58099999999999996,"15min":0.59399999999999997},"min":{"1min":0.38400000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79100000000000004,"5min":0.81299999999999994,"15min":0.81399999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.439,"5min":0.39400000000000002,"15min":0.37},"max":{"1min":0.70299999999999996,"5min":0.871,"15min":0.871},"last":0.59199999999999997}]},{"osd":2,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59299999999999997,"15min":0.59699999999999998},"min":{"1min":0.311,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69199999999999995,"5min":0.79700000000000004,"15min":0.84399999999999997},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58799999999999997,"15min":0.59599999999999997},"min":{"1min":0.44,"5min":0.378,"15min":0.378},"max":{"1min":0.73099999999999998,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.51700000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.79800000000000004,"5min":0.81100000000000005,"15min":0.94499999999999995},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61699999999999999,"15min":0.63200000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.74099999999999999,"5min":0.88200000000000001,"15min":0.91500000000000004},"last":0.61499999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52900000000000003,"15min":0.50700000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.27400000000000002},"max":{"1min":0.70999999999999996,"5min":0.85699999999999998,"15min":1.0369999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.59999999999999998,"15min":0.57699999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.308},"max":{"1min":0.77800000000000002,"5min":0.88800000000000001,"15min":1.1639999999999999},"last":0.70299999999999996}]},{"osd":6,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54300000000000004,"15min":0.52600000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.27900000000000003},"max":{"1min":0.89100000000000001,"5min":0.94999999999999996,"15min":1.0169999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54300000000000004,"15min":0.53100000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.32300000000000001},"max":{"1min":0.81899999999999995,"5min":0.878,"15min":1.0009999999999999},"last":0.68799999999999994}]},{"osd":7,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56499999999999995,"15min":0.54800000000000004},"min":{"1min":0.39200000000000002,"5min":0.35799999999999998,"15min":0.32900000000000001},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.66400000000000003,"15min":0.64800000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.35199999999999998},"max":{"1min":0.85299999999999998,"5min":0.92300000000000004,"15min":0.96599999999999997},"last":0.73199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:48:24.571 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.1f is not active+clean 2024-04-06T13:48:24.572 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.572 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.1b is not active+clean 2024-04-06T13:48:24.572 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.572 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.1a is not active+clean 2024-04-06T13:48:24.572 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.572 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.18 is not active+clean 2024-04-06T13:48:24.572 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.572 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.17 is not active+clean 2024-04-06T13:48:24.572 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.573 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.14 is not active+clean 2024-04-06T13:48:24.573 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.573 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.11 is not active+clean 2024-04-06T13:48:24.573 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.573 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.d is not active+clean 2024-04-06T13:48:24.573 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.573 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.9 is not active+clean 2024-04-06T13:48:24.573 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.573 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.5 is not active+clean 2024-04-06T13:48:24.573 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.573 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.4 is not active+clean 2024-04-06T13:48:24.574 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.574 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.0 is not active+clean 2024-04-06T13:48:24.574 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.574 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.1f is not active+clean 2024-04-06T13:48:24.574 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-04-06T13:48:18.266186+0000', 'last_change': '2024-04-06T13:48:18.266186+0000', 'last_active': '2024-04-06T13:48:18.248805+0000', 'last_peered': '2024-04-06T13:48:18.248805+0000', 'last_clean': '2024-04-06T13:48:18.248805+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:18.266186+0000', 'last_undegraded': '2024-04-06T13:48:18.266186+0000', 'last_fullsized': '2024-04-06T13:48:18.266186+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:54:56.369118+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-06T13:48:24.574 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.22 is not active+clean 2024-04-06T13:48:24.574 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.574 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.23 is not active+clean 2024-04-06T13:48:24.574 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.574 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.1b is not active+clean 2024-04-06T13:48:24.574 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-04-06T13:48:18.262863+0000', 'last_change': '2024-04-06T13:48:18.262863+0000', 'last_active': '2024-04-06T13:48:18.248805+0000', 'last_peered': '2024-04-06T13:48:18.248805+0000', 'last_clean': '2024-04-06T13:48:18.248805+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:18.262863+0000', 'last_undegraded': '2024-04-06T13:48:18.262863+0000', 'last_fullsized': '2024-04-06T13:48:18.262863+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T16:59:18.513042+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-06T13:48:24.575 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.27 is not active+clean 2024-04-06T13:48:24.575 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.575 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.18 is not active+clean 2024-04-06T13:48:24.575 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-04-06T13:48:18.265371+0000', 'last_change': '2024-04-06T13:48:18.265371+0000', 'last_active': '2024-04-06T13:48:18.248805+0000', 'last_peered': '2024-04-06T13:48:18.248805+0000', 'last_clean': '2024-04-06T13:48:18.248805+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:18.265371+0000', 'last_undegraded': '2024-04-06T13:48:18.265371+0000', 'last_fullsized': '2024-04-06T13:48:18.265371+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T14:02:13.092175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-06T13:48:24.575 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.15 is not active+clean 2024-04-06T13:48:24.575 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-04-06T13:48:18.262594+0000', 'last_change': '2024-04-06T13:48:18.262594+0000', 'last_active': '2024-04-06T13:48:18.248805+0000', 'last_peered': '2024-04-06T13:48:18.248805+0000', 'last_clean': '2024-04-06T13:48:18.248805+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:18.262594+0000', 'last_undegraded': '2024-04-06T13:48:18.262594+0000', 'last_fullsized': '2024-04-06T13:48:18.262594+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T16:06:05.488014+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-06T13:48:24.575 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.14 is not active+clean 2024-04-06T13:48:24.575 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-04-06T13:48:18.265322+0000', 'last_change': '2024-04-06T13:48:18.265322+0000', 'last_active': '2024-04-06T13:48:18.248805+0000', 'last_peered': '2024-04-06T13:48:18.248805+0000', 'last_clean': '2024-04-06T13:48:18.248805+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:18.265322+0000', 'last_undegraded': '2024-04-06T13:48:18.265322+0000', 'last_fullsized': '2024-04-06T13:48:18.265322+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T20:56:55.759595+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-06T13:48:24.575 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.13 is not active+clean 2024-04-06T13:48:24.575 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-04-06T13:48:18.262385+0000', 'last_change': '2024-04-06T13:48:18.262385+0000', 'last_active': '2024-04-06T13:48:18.248805+0000', 'last_peered': '2024-04-06T13:48:18.248805+0000', 'last_clean': '2024-04-06T13:48:18.248805+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:18.262385+0000', 'last_undegraded': '2024-04-06T13:48:18.262385+0000', 'last_fullsized': '2024-04-06T13:48:18.262385+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:31:24.701348+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-06T13:48:24.575 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.12 is not active+clean 2024-04-06T13:48:24.576 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-04-06T13:48:18.261860+0000', 'last_change': '2024-04-06T13:48:18.261860+0000', 'last_active': '2024-04-06T13:48:18.248805+0000', 'last_peered': '2024-04-06T13:48:18.248805+0000', 'last_clean': '2024-04-06T13:48:18.248805+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:18.261860+0000', 'last_undegraded': '2024-04-06T13:48:18.261860+0000', 'last_fullsized': '2024-04-06T13:48:18.261860+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T23:35:17.934513+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-06T13:48:24.576 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.2e is not active+clean 2024-04-06T13:48:24.576 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.576 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.11 is not active+clean 2024-04-06T13:48:24.576 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-04-06T13:48:18.263085+0000', 'last_change': '2024-04-06T13:48:18.263085+0000', 'last_active': '2024-04-06T13:48:18.248805+0000', 'last_peered': '2024-04-06T13:48:18.248805+0000', 'last_clean': '2024-04-06T13:48:18.248805+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:18.263085+0000', 'last_undegraded': '2024-04-06T13:48:18.263085+0000', 'last_fullsized': '2024-04-06T13:48:18.263085+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:02:23.692226+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-06T13:48:24.576 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.2f is not active+clean 2024-04-06T13:48:24.576 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.576 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.30 is not active+clean 2024-04-06T13:48:24.576 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.576 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.f is not active+clean 2024-04-06T13:48:24.576 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-04-06T13:48:18.265775+0000', 'last_change': '2024-04-06T13:48:18.265775+0000', 'last_active': '2024-04-06T13:48:18.248805+0000', 'last_peered': '2024-04-06T13:48:18.248805+0000', 'last_clean': '2024-04-06T13:48:18.248805+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:18.265775+0000', 'last_undegraded': '2024-04-06T13:48:18.265775+0000', 'last_fullsized': '2024-04-06T13:48:18.265775+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:46:49.041735+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-06T13:48:24.577 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.31 is not active+clean 2024-04-06T13:48:24.577 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.577 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.c is not active+clean 2024-04-06T13:48:24.577 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-04-06T13:48:18.262279+0000', 'last_change': '2024-04-06T13:48:18.262279+0000', 'last_active': '2024-04-06T13:48:18.248805+0000', 'last_peered': '2024-04-06T13:48:18.248805+0000', 'last_clean': '2024-04-06T13:48:18.248805+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:18.262279+0000', 'last_undegraded': '2024-04-06T13:48:18.262279+0000', 'last_fullsized': '2024-04-06T13:48:18.262279+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T14:55:10.767276+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-06T13:48:24.577 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.39 is not active+clean 2024-04-06T13:48:24.577 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.577 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.6 is not active+clean 2024-04-06T13:48:24.577 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-04-06T13:48:18.262063+0000', 'last_change': '2024-04-06T13:48:18.262063+0000', 'last_active': '2024-04-06T13:48:18.248805+0000', 'last_peered': '2024-04-06T13:48:18.248805+0000', 'last_clean': '2024-04-06T13:48:18.248805+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:18.262063+0000', 'last_undegraded': '2024-04-06T13:48:18.262063+0000', 'last_fullsized': '2024-04-06T13:48:18.262063+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T14:27:18.981232+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-06T13:48:24.577 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.3b is not active+clean 2024-04-06T13:48:24.577 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.577 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.3d is not active+clean 2024-04-06T13:48:24.577 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.578 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.2 is not active+clean 2024-04-06T13:48:24.578 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-04-06T13:48:18.261299+0000', 'last_change': '2024-04-06T13:48:18.261299+0000', 'last_active': '2024-04-06T13:48:18.248805+0000', 'last_peered': '2024-04-06T13:48:18.248805+0000', 'last_clean': '2024-04-06T13:48:18.248805+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:18.261299+0000', 'last_undegraded': '2024-04-06T13:48:18.261299+0000', 'last_fullsized': '2024-04-06T13:48:18.261299+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:18.248805+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T20:51:53.597002+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-06T13:48:24.578 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.21 is not active+clean 2024-04-06T13:48:24.578 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-06T13:48:19.256766+0000', 'last_change': '2024-04-06T13:48:19.256766+0000', 'last_active': '2024-04-06T13:48:19.256766+0000', 'last_peered': '2024-04-06T13:48:19.256766+0000', 'last_clean': '2024-04-06T13:48:19.256766+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-06T13:48:19.256766+0000', 'last_undegraded': '2024-04-06T13:48:19.256766+0000', 'last_fullsized': '2024-04-06T13:48:19.256766+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'last_clean_scrub_stamp': '2024-04-06T13:48:19.256766+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-06T13:48:24.578 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-06T13:48:24.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.729+0000 7f0112db4700 1 Processor -- start 2024-04-06T13:48:24.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.729+0000 7f0112db4700 1 -- start start 2024-04-06T13:48:24.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.730+0000 7f0112db4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f010c0cf500 0x7f010c0cf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:24.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.730+0000 7f0112db4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f010c0cd3f0 0x7f010c06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:24.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.730+0000 7f0112db4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f010c0cfe20 con 0x7f010c0cd870 2024-04-06T13:48:24.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.730+0000 7f0112db4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f010c06b340 con 0x7f010c0cf500 2024-04-06T13:48:24.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.730+0000 7f0112db4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010c0ccfd0 con 0x7f010c0cd3f0 2024-04-06T13:48:24.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.730+0000 7f0111351700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f010c0cd870 0x7f010c0ca7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33230/0 (socket says 172.21.15.189:33230) 2024-04-06T13:48:24.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.730+0000 7f0111351700 1 -- 172.21.15.189:0/4020831725 learned_addr learned my addr 172.21.15.189:0/4020831725 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:24.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.730+0000 7f010b7fe700 1 -- 172.21.15.189:0/4020831725 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382985829 0 0) 0x7f010c0cfe20 con 0x7f010c0cd870 2024-04-06T13:48:24.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f010b7fe700 1 -- 172.21.15.189:0/4020831725 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00fc0035b0 con 0x7f010c0cd870 2024-04-06T13:48:24.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f010bfff700 1 --2- 172.21.15.189:0/4020831725 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f010c0cd3f0 0x7f010c06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:24.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f0110b50700 1 --2- 172.21.15.189:0/4020831725 >> v2:172.21.15.192:3300/0 conn(0x7f010c0cf500 0x7f010c0cf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:24.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f010b7fe700 1 -- 172.21.15.189:0/4020831725 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3000142408 0 0) 0x7f00fc0035b0 con 0x7f010c0cd870 2024-04-06T13:48:24.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f010b7fe700 1 -- 172.21.15.189:0/4020831725 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f010c0cd3f0 msgr2=0x7f010c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:24.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f010b7fe700 1 --2- 172.21.15.189:0/4020831725 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f010c0cd3f0 0x7f010c06a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f010b7fe700 1 -- 172.21.15.189:0/4020831725 >> v2:172.21.15.192:3300/0 conn(0x7f010c0cf500 msgr2=0x7f010c0cf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:24.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f010b7fe700 1 --2- 172.21.15.189:0/4020831725 >> v2:172.21.15.192:3300/0 conn(0x7f010c0cf500 0x7f010c0cf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f010b7fe700 1 -- 172.21.15.189:0/4020831725 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010c0cc060 con 0x7f010c0cd870 2024-04-06T13:48:24.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f010b7fe700 1 -- 172.21.15.189:0/4020831725 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f01040019f0 con 0x7f010c0cd870 2024-04-06T13:48:24.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f010b7fe700 1 -- 172.21.15.189:0/4020831725 >> v1:172.21.15.189:6789/0 conn(0x7f010c0cd870 legacy=0x7f010c0ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:24.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f010b7fe700 1 --2- 172.21.15.189:0/4020831725 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f010c0cd3f0 0x7f00fc003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:24.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f010b7fe700 1 --2- 172.21.15.189:0/4020831725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00fc003f80 0x7f00fc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:24.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f010b7fe700 1 --2- 172.21.15.189:0/4020831725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00fc004d90 0x7f00fc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:24.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f010b7fe700 1 -- 172.21.15.189:0/4020831725 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010c06b340 con 0x7f00fc004d90 2024-04-06T13:48:24.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f010b7fe700 1 -- 172.21.15.189:0/4020831725 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010c0ccfd0 con 0x7f00fc003f80 2024-04-06T13:48:24.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f010b7fe700 1 -- 172.21.15.189:0/4020831725 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc00ba10 con 0x7f010c0cd3f0 2024-04-06T13:48:24.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f0111351700 1 --2- 172.21.15.189:0/4020831725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00fc004d90 0x7f00fc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:24.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f010bfff700 1 --2- 172.21.15.189:0/4020831725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00fc003f80 0x7f00fc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:24.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f0110b50700 1 --2- 172.21.15.189:0/4020831725 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f010c0cd3f0 0x7f00fc003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:24.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f0111351700 1 -- 172.21.15.189:0/4020831725 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f010c0cd3f0 msgr2=0x7f00fc003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:24.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f0111351700 1 --2- 172.21.15.189:0/4020831725 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f010c0cd3f0 0x7f00fc003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f0111351700 1 -- 172.21.15.189:0/4020831725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00fc003f80 msgr2=0x7f00fc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:24.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f0111351700 1 --2- 172.21.15.189:0/4020831725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00fc003f80 0x7f00fc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f0111351700 1 -- 172.21.15.189:0/4020831725 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00fc0035b0 con 0x7f00fc004d90 2024-04-06T13:48:24.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.731+0000 7f0111351700 1 --2- 172.21.15.189:0/4020831725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00fc004d90 0x7f00fc005210 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f010400a830 tx=0x7f00fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bbb48e41a70c01f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:24.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.732+0000 7f010b7fe700 1 -- 172.21.15.189:0/4020831725 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0104012600 con 0x7f00fc004d90 2024-04-06T13:48:24.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.732+0000 7f010b7fe700 1 -- 172.21.15.189:0/4020831725 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0104012760 con 0x7f00fc004d90 2024-04-06T13:48:24.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.732+0000 7f010b7fe700 1 -- 172.21.15.189:0/4020831725 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0104012b10 con 0x7f00fc004d90 2024-04-06T13:48:24.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.732+0000 7f0112db4700 1 -- 172.21.15.189:0/4020831725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00fc004d90 msgr2=0x7f00fc005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:24.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.732+0000 7f0112db4700 1 --2- 172.21.15.189:0/4020831725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00fc004d90 0x7f00fc005210 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f010400a830 tx=0x7f00fc005880 comp rx=0 tx=0).stop 2024-04-06T13:48:24.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.732+0000 7f0111351700 1 -- 172.21.15.189:0/4020831725 reap_dead start 2024-04-06T13:48:24.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.732+0000 7f0112db4700 1 -- 172.21.15.189:0/4020831725 shutdown_connections 2024-04-06T13:48:24.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.732+0000 7f0112db4700 1 -- 172.21.15.189:0/4020831725 >> 172.21.15.189:0/4020831725 conn(0x7f010c099a50 msgr2=0x7f010c0b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:24.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.732+0000 7f0112db4700 1 -- 172.21.15.189:0/4020831725 shutdown_connections 2024-04-06T13:48:24.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.732+0000 7f0112db4700 1 -- 172.21.15.189:0/4020831725 wait complete. 2024-04-06T13:48:24.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.733+0000 7f0112db4700 1 Processor -- start 2024-04-06T13:48:24.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.733+0000 7f0112db4700 1 -- start start 2024-04-06T13:48:24.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.734+0000 7f0112db4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00fc004d90 0x7f010c1f6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:24.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.734+0000 7f0112db4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f010c1e1020 0x7f010c0afc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:24.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.734+0000 7f0112db4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f010c1e1480 0x7f010c0b01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:24.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.734+0000 7f0112db4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010c0b2d60 con 0x7f00fc004d90 2024-04-06T13:48:24.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.734+0000 7f0112db4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010c1f73c0 con 0x7f010c1e1020 2024-04-06T13:48:24.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.734+0000 7f0112db4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010c0adef0 con 0x7f010c1e1480 2024-04-06T13:48:24.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.734+0000 7f0110b50700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00fc004d90 0x7f010c1f6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:24.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.734+0000 7f0110b50700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00fc004d90 0x7f010c1f6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45120/0 (socket says 172.21.15.189:45120) 2024-04-06T13:48:24.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.734+0000 7f0110b50700 1 -- 172.21.15.189:0/1704716060 learned_addr learned my addr 172.21.15.189:0/1704716060 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:24.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.735+0000 7f0111351700 1 --2- 172.21.15.189:0/1704716060 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f010c1e1480 0x7f010c0b01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:24.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.735+0000 7f010bfff700 1 --2- 172.21.15.189:0/1704716060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f010c1e1020 0x7f010c0afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:24.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.735+0000 7f0110b50700 1 -- 172.21.15.189:0/1704716060 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f010c1e1480 msgr2=0x7f010c0b01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:24.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.735+0000 7f0110b50700 1 --2- 172.21.15.189:0/1704716060 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f010c1e1480 0x7f010c0b01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.735+0000 7f0110b50700 1 -- 172.21.15.189:0/1704716060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f010c1e1020 msgr2=0x7f010c0afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:24.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.735+0000 7f0110b50700 1 --2- 172.21.15.189:0/1704716060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f010c1e1020 0x7f010c0afc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:24.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.735+0000 7f0110b50700 1 -- 172.21.15.189:0/1704716060 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010c0ae1d0 con 0x7f00fc004d90 2024-04-06T13:48:24.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.735+0000 7f0110b50700 1 --2- 172.21.15.189:0/1704716060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00fc004d90 0x7f010c1f6e80 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f0100011ad0 tx=0x7f0100011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:24.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.735+0000 7f01097fa700 1 -- 172.21.15.189:0/1704716060 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010000fa60 con 0x7f00fc004d90 2024-04-06T13:48:24.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.735+0000 7f01097fa700 1 -- 172.21.15.189:0/1704716060 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010000fbc0 con 0x7f00fc004d90 2024-04-06T13:48:24.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.735+0000 7f0112db4700 1 -- 172.21.15.189:0/1704716060 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f010c0b2fb0 con 0x7f00fc004d90 2024-04-06T13:48:24.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.735+0000 7f01097fa700 1 -- 172.21.15.189:0/1704716060 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0100014440 con 0x7f00fc004d90 2024-04-06T13:48:24.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.735+0000 7f0112db4700 1 -- 172.21.15.189:0/1704716060 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f010c0aec50 con 0x7f00fc004d90 2024-04-06T13:48:24.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.736+0000 7f0112db4700 1 -- 172.21.15.189:0/1704716060 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f010c067bb0 con 0x7f00fc004d90 2024-04-06T13:48:24.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.737+0000 7f01097fa700 1 -- 172.21.15.189:0/1704716060 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f01000145a0 con 0x7f00fc004d90 2024-04-06T13:48:24.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.737+0000 7f01097fa700 1 --2- 172.21.15.189:0/1704716060 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f00f006af00 0x7f00f006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:24.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.737+0000 7f010bfff700 1 --2- 172.21.15.189:0/1704716060 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f00f006af00 0x7f00f006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:24.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.737+0000 7f01097fa700 1 -- 172.21.15.189:0/1704716060 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f010008a180 con 0x7f00fc004d90 2024-04-06T13:48:24.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.737+0000 7f010bfff700 1 --2- 172.21.15.189:0/1704716060 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f00f006af00 0x7f00f006d3c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f010c072890 tx=0x7f00f8008040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:24.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:24.742+0000 7f01097fa700 1 -- 172.21.15.189:0/1704716060 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0100054e70 con 0x7f00fc004d90 2024-04-06T13:48:25.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.022+0000 7f0112db4700 1 -- 172.21.15.189:0/1704716060 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f010c0ae360 con 0x7f00fc004d90 2024-04-06T13:48:25.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.022+0000 7f01097fa700 1 -- 172.21.15.189:0/1704716060 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1180 (secure 0 0 0) 0x7f0100058490 con 0x7f00fc004d90 2024-04-06T13:48:25.022 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:25.022 INFO:teuthology.orchestra.run.smithi189.stdout:{"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":5798,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":248,"num_osds":8,"num_up_osds":8,"osd_up_since":1712405517,"num_in_osds":8,"osd_in_since":1712405509,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":62},{"state_name":"unknown","count":23},{"state_name":"creating+peering","count":12}],"num_pgs":97,"num_pools":3,"num_objects":12,"data_bytes":590438,"bytes_used":466604032,"bytes_avail":772627509248,"bytes_total":773094113280,"unknown_pgs_ratio":0.23711340129375458,"inactive_pgs_ratio":0.12371134012937546,"write_bytes_sec":180,"read_op_per_sec":0,"write_op_per_sec":4},"fsmap":{"epoch":116,"id":15,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":15,"rank":0,"name":"d","status":"up:active","gid":12108}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2024-04-06T13:46:56.125819+0000","services":{}},"progress_events":{}} 2024-04-06T13:48:25.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.024+0000 7f0112db4700 1 -- 172.21.15.189:0/1704716060 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f00f006af00 msgr2=0x7f00f006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:25.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.024+0000 7f0112db4700 1 --2- 172.21.15.189:0/1704716060 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f00f006af00 0x7f00f006d3c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f010c072890 tx=0x7f00f8008040 comp rx=0 tx=0).stop 2024-04-06T13:48:25.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.024+0000 7f0112db4700 1 -- 172.21.15.189:0/1704716060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00fc004d90 msgr2=0x7f010c1f6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:25.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.024+0000 7f0112db4700 1 --2- 172.21.15.189:0/1704716060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00fc004d90 0x7f010c1f6e80 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f0100011ad0 tx=0x7f0100011e90 comp rx=0 tx=0).stop 2024-04-06T13:48:25.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.024+0000 7f0112db4700 1 -- 172.21.15.189:0/1704716060 shutdown_connections 2024-04-06T13:48:25.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.024+0000 7f0112db4700 1 --2- 172.21.15.189:0/1704716060 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f00f006af00 0x7f00f006d3c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:25.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.024+0000 7f0112db4700 1 --2- 172.21.15.189:0/1704716060 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f010c1e1480 0x7f010c0b01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:25.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.024+0000 7f0112db4700 1 --2- 172.21.15.189:0/1704716060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00fc004d90 0x7f010c1f6e80 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:25.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.024+0000 7f0112db4700 1 --2- 172.21.15.189:0/1704716060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f010c1e1020 0x7f010c0afc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:25.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.024+0000 7f0112db4700 1 -- 172.21.15.189:0/1704716060 >> 172.21.15.189:0/1704716060 conn(0x7f010c099a50 msgr2=0x7f010c0b1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:25.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.024+0000 7f0112db4700 1 -- 172.21.15.189:0/1704716060 shutdown_connections 2024-04-06T13:48:25.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.025+0000 7f0112db4700 1 -- 172.21.15.189:0/1704716060 wait complete. 2024-04-06T13:48:25.037 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-06T13:48:25.037 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:48:25.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.159+0000 7f92d5efc700 1 Processor -- start 2024-04-06T13:48:25.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.159+0000 7f92d5efc700 1 -- start start 2024-04-06T13:48:25.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.160+0000 7f92d5efc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f92d015d7b0 0x7f92d00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:25.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.160+0000 7f92d5efc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f92d006a730 0x7f92d006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:25.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.160+0000 7f92d5efc700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d00b3990 con 0x7f92d015ddf0 2024-04-06T13:48:25.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.160+0000 7f92d5efc700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92d015e950 con 0x7f92d015d7b0 2024-04-06T13:48:25.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.160+0000 7f92d5efc700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92d006b0f0 con 0x7f92d006a730 2024-04-06T13:48:25.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.160+0000 7f92cf7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f92d015d7b0 0x7f92d00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:25.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.160+0000 7f92cf7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f92d015d7b0 0x7f92d00b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45122/0 (socket says 172.21.15.189:45122) 2024-04-06T13:48:25.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.160+0000 7f92cf7fe700 1 -- 172.21.15.189:0/1216866779 learned_addr learned my addr 172.21.15.189:0/1216866779 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:25.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.160+0000 7f92ceffd700 1 --2- 172.21.15.189:0/1216866779 >> v2:172.21.15.192:3300/0 conn(0x7f92d006a730 0x7f92d006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.161+0000 7f92cf7fe700 1 -- 172.21.15.189:0/1216866779 >> v2:172.21.15.192:3300/0 conn(0x7f92d006a730 msgr2=0x7f92d006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.161+0000 7f92cf7fe700 1 --2- 172.21.15.189:0/1216866779 >> v2:172.21.15.192:3300/0 conn(0x7f92d006a730 0x7f92d006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.161+0000 7f92cf7fe700 1 -- 172.21.15.189:0/1216866779 >> v1:172.21.15.192:6789/0 conn(0x7f92d015ddf0 legacy=0x7f92d015e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.161+0000 7f92cf7fe700 1 -- 172.21.15.189:0/1216866779 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d00b3060 con 0x7f92d015d7b0 2024-04-06T13:48:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.161+0000 7f92ceffd700 1 --2- 172.21.15.189:0/1216866779 >> v2:172.21.15.192:3300/0 conn(0x7f92d006a730 0x7f92d006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.161+0000 7f92cf7fe700 1 --2- 172.21.15.189:0/1216866779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92d015d7b0 0x7f92d00b3e20 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7f92d00afc00 tx=0x7f92c4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e5781562c80bc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.161+0000 7f92ce7fc700 1 -- 172.21.15.189:0/1216866779 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c400b870 con 0x7f92d015d7b0 2024-04-06T13:48:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.161+0000 7f92ce7fc700 1 -- 172.21.15.189:0/1216866779 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c400b9d0 con 0x7f92d015d7b0 2024-04-06T13:48:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.161+0000 7f92ce7fc700 1 -- 172.21.15.189:0/1216866779 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c400bd60 con 0x7f92d015d7b0 2024-04-06T13:48:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.161+0000 7f92d5efc700 1 -- 172.21.15.189:0/1216866779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92d015d7b0 msgr2=0x7f92d00b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.161+0000 7f92d5efc700 1 --2- 172.21.15.189:0/1216866779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92d015d7b0 0x7f92d00b3e20 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7f92d00afc00 tx=0x7f92c4009bd0 comp rx=0 tx=0).stop 2024-04-06T13:48:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.162+0000 7f92d5efc700 1 -- 172.21.15.189:0/1216866779 shutdown_connections 2024-04-06T13:48:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.162+0000 7f92d5efc700 1 --2- 172.21.15.189:0/1216866779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92d015d7b0 0x7f92d00b3e20 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:25.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.162+0000 7f92d5efc700 1 --2- 172.21.15.189:0/1216866779 >> v2:172.21.15.192:3300/0 conn(0x7f92d006a730 0x7f92d006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:25.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.162+0000 7f92d5efc700 1 -- 172.21.15.189:0/1216866779 >> 172.21.15.189:0/1216866779 conn(0x7f92d00d00b0 msgr2=0x7f92d00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:25.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.162+0000 7f92d5efc700 1 -- 172.21.15.189:0/1216866779 shutdown_connections 2024-04-06T13:48:25.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.162+0000 7f92d5efc700 1 -- 172.21.15.189:0/1216866779 wait complete. 2024-04-06T13:48:25.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.163+0000 7f92d5efc700 1 Processor -- start 2024-04-06T13:48:25.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.163+0000 7f92d5efc700 1 -- start start 2024-04-06T13:48:25.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.164+0000 7f92d5efc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92d006a730 0x7f92d0146ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:25.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.164+0000 7f92d5efc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92d015d7b0 0x7f92d015fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:25.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.164+0000 7f92d5efc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92d015ddf0 0x7f92d015ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:25.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.164+0000 7f92d5efc700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d015fa50 con 0x7f92d015d7b0 2024-04-06T13:48:25.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.164+0000 7f92d5efc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d01485f0 con 0x7f92d015ddf0 2024-04-06T13:48:25.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.164+0000 7f92d5efc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92d015ee40 con 0x7f92d006a730 2024-04-06T13:48:25.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.164+0000 7f92ceffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92d015d7b0 0x7f92d015fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:25.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.164+0000 7f92ceffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92d015d7b0 0x7f92d015fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45128/0 (socket says 172.21.15.189:45128) 2024-04-06T13:48:25.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.164+0000 7f92ceffd700 1 -- 172.21.15.189:0/2026340411 learned_addr learned my addr 172.21.15.189:0/2026340411 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:25.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.164+0000 7f92cffff700 1 --2- 172.21.15.189:0/2026340411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92d015ddf0 0x7f92d015ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:25.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.164+0000 7f92cf7fe700 1 --2- 172.21.15.189:0/2026340411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92d006a730 0x7f92d0146ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:25.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.164+0000 7f92ceffd700 1 -- 172.21.15.189:0/2026340411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92d006a730 msgr2=0x7f92d0146ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:25.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.164+0000 7f92ceffd700 1 --2- 172.21.15.189:0/2026340411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92d006a730 0x7f92d0146ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:25.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.164+0000 7f92ceffd700 1 -- 172.21.15.189:0/2026340411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92d015ddf0 msgr2=0x7f92d015ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:25.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.164+0000 7f92ceffd700 1 --2- 172.21.15.189:0/2026340411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92d015ddf0 0x7f92d015ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:25.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.164+0000 7f92ceffd700 1 -- 172.21.15.189:0/2026340411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c4009550 con 0x7f92d015d7b0 2024-04-06T13:48:25.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.164+0000 7f92ceffd700 1 --2- 172.21.15.189:0/2026340411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92d015d7b0 0x7f92d015fee0 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7f92c000e810 tx=0x7f92c000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:25.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.164+0000 7f92b7fff700 1 -- 172.21.15.189:0/2026340411 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c0013610 con 0x7f92d015d7b0 2024-04-06T13:48:25.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.164+0000 7f92b7fff700 1 -- 172.21.15.189:0/2026340411 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c0013770 con 0x7f92d015d7b0 2024-04-06T13:48:25.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.165+0000 7f92b7fff700 1 -- 172.21.15.189:0/2026340411 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c0013b20 con 0x7f92d015d7b0 2024-04-06T13:48:25.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.165+0000 7f92d5efc700 1 -- 172.21.15.189:0/2026340411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d015f120 con 0x7f92d015d7b0 2024-04-06T13:48:25.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.165+0000 7f92d5efc700 1 -- 172.21.15.189:0/2026340411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d015f6e0 con 0x7f92d015d7b0 2024-04-06T13:48:25.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.165+0000 7f92d5efc700 1 -- 172.21.15.189:0/2026340411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d015fc00 con 0x7f92d015d7b0 2024-04-06T13:48:25.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.166+0000 7f92b7fff700 1 -- 172.21.15.189:0/2026340411 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f92c0013c80 con 0x7f92d015d7b0 2024-04-06T13:48:25.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.166+0000 7f92b7fff700 1 --2- 172.21.15.189:0/2026340411 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f92bc06af00 0x7f92bc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:25.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.166+0000 7f92cf7fe700 1 --2- 172.21.15.189:0/2026340411 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f92bc06af00 0x7f92bc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:25.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.166+0000 7f92b7fff700 1 -- 172.21.15.189:0/2026340411 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f92c00894f0 con 0x7f92d015d7b0 2024-04-06T13:48:25.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.167+0000 7f92cf7fe700 1 --2- 172.21.15.189:0/2026340411 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f92bc06af00 0x7f92bc06d3c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f92c4000c00 tx=0x7f92c4008030 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:25.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.171+0000 7f92b7fff700 1 -- 172.21.15.189:0/2026340411 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f92d015fc00 con 0x7f92d015d7b0 2024-04-06T13:48:25.251 INFO:tasks.ceph.mds.d.smithi192.stderr:2024-04-06T13:48:25.251+0000 7f36f3503700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T13:48:25.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.359+0000 7f92d5efc700 1 -- 172.21.15.189:0/2026340411 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f92d00b3620 con 0x7f92bc06af00 2024-04-06T13:48:25.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.366+0000 7f92b7fff700 1 -- 172.21.15.189:0/2026340411 <== mgr.11733 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286244 (secure 0 0 0) 0x7f92c400bb40 con 0x7f92bc06af00 2024-04-06T13:48:25.370 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:25.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.370+0000 7f92d5efc700 1 -- 172.21.15.189:0/2026340411 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f92bc06af00 msgr2=0x7f92bc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:25.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.370+0000 7f92d5efc700 1 --2- 172.21.15.189:0/2026340411 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f92bc06af00 0x7f92bc06d3c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f92c4000c00 tx=0x7f92c4008030 comp rx=0 tx=0).stop 2024-04-06T13:48:25.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.370+0000 7f92d5efc700 1 -- 172.21.15.189:0/2026340411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92d015d7b0 msgr2=0x7f92d015fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:25.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.370+0000 7f92d5efc700 1 --2- 172.21.15.189:0/2026340411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92d015d7b0 0x7f92d015fee0 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7f92c000e810 tx=0x7f92c000eb20 comp rx=0 tx=0).stop 2024-04-06T13:48:25.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.370+0000 7f92d5efc700 1 -- 172.21.15.189:0/2026340411 shutdown_connections 2024-04-06T13:48:25.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.370+0000 7f92d5efc700 1 --2- 172.21.15.189:0/2026340411 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f92bc06af00 0x7f92bc06d3c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:25.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.370+0000 7f92d5efc700 1 --2- 172.21.15.189:0/2026340411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92d015ddf0 0x7f92d015ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:25.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.370+0000 7f92d5efc700 1 --2- 172.21.15.189:0/2026340411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92d006a730 0x7f92d0146ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:25.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.370+0000 7f92d5efc700 1 --2- 172.21.15.189:0/2026340411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92d015d7b0 0x7f92d015fee0 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:25.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.370+0000 7f92d5efc700 1 -- 172.21.15.189:0/2026340411 >> 172.21.15.189:0/2026340411 conn(0x7f92d00d00b0 msgr2=0x7f92d00afc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:25.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.370+0000 7f92d5efc700 1 -- 172.21.15.189:0/2026340411 shutdown_connections 2024-04-06T13:48:25.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:25.370+0000 7f92d5efc700 1 -- 172.21.15.189:0/2026340411 wait complete. 2024-04-06T13:48:25.373 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:48:25.382 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-04-06T13:48:23.927698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590438,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":645,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":391,"ondisk_log_size":391,"up":148,"acting":148,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":77,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":455668,"kb_used_data":25460,"kb_used_omap":12,"kb_used_meta":430067,"kb_avail":754519052,"statfs":{"total":773094113280,"available":772627509248,"internally_reserved":0,"allocated":26071040,"data_stored":13946484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":440389176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.670934"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.900020+0000","last_change":"2024-04-06T13:48:19.704688+0000","last_active":"2024-04-06T13:48:20.900020+0000","last_peered":"2024-04-06T13:48:20.900020+0000","last_clean":"2024-04-06T13:48:20.900020+0000","last_became_active":"2024-04-06T13:48:19.704088+0000","last_became_peered":"2024-04-06T13:48:19.704088+0000","last_unstale":"2024-04-06T13:48:20.900020+0000","last_undegraded":"2024-04-06T13:48:20.900020+0000","last_fullsized":"2024-04-06T13:48:20.900020+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:49.108653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097823+0000","last_change":"2024-04-06T13:48:19.697461+0000","last_active":"2024-04-06T13:48:21.097823+0000","last_peered":"2024-04-06T13:48:21.097823+0000","last_clean":"2024-04-06T13:48:21.097823+0000","last_became_active":"2024-04-06T13:48:19.695459+0000","last_became_peered":"2024-04-06T13:48:19.695459+0000","last_unstale":"2024-04-06T13:48:21.097823+0000","last_undegraded":"2024-04-06T13:48:21.097823+0000","last_fullsized":"2024-04-06T13:48:21.097823+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:42:12.990594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098046+0000","last_change":"2024-04-06T13:48:19.690589+0000","last_active":"2024-04-06T13:48:21.098046+0000","last_peered":"2024-04-06T13:48:21.098046+0000","last_clean":"2024-04-06T13:48:21.098046+0000","last_became_active":"2024-04-06T13:48:19.690338+0000","last_became_peered":"2024-04-06T13:48:19.690338+0000","last_unstale":"2024-04-06T13:48:21.098046+0000","last_undegraded":"2024-04-06T13:48:21.098046+0000","last_fullsized":"2024-04-06T13:48:21.098046+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:51:42.234400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097813+0000","last_change":"2024-04-06T13:48:19.695665+0000","last_active":"2024-04-06T13:48:21.097813+0000","last_peered":"2024-04-06T13:48:21.097813+0000","last_clean":"2024-04-06T13:48:21.097813+0000","last_became_active":"2024-04-06T13:48:19.695362+0000","last_became_peered":"2024-04-06T13:48:19.695362+0000","last_unstale":"2024-04-06T13:48:21.097813+0000","last_undegraded":"2024-04-06T13:48:21.097813+0000","last_fullsized":"2024-04-06T13:48:21.097813+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:26:21.699641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098450+0000","last_change":"2024-04-06T13:48:19.700300+0000","last_active":"2024-04-06T13:48:21.098450+0000","last_peered":"2024-04-06T13:48:21.098450+0000","last_clean":"2024-04-06T13:48:21.098450+0000","last_became_active":"2024-04-06T13:48:19.700054+0000","last_became_peered":"2024-04-06T13:48:19.700054+0000","last_unstale":"2024-04-06T13:48:21.098450+0000","last_undegraded":"2024-04-06T13:48:21.098450+0000","last_fullsized":"2024-04-06T13:48:21.098450+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:30:14.717681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097799+0000","last_change":"2024-04-06T13:48:19.700151+0000","last_active":"2024-04-06T13:48:21.097799+0000","last_peered":"2024-04-06T13:48:21.097799+0000","last_clean":"2024-04-06T13:48:21.097799+0000","last_became_active":"2024-04-06T13:48:19.699882+0000","last_became_peered":"2024-04-06T13:48:19.699882+0000","last_unstale":"2024-04-06T13:48:21.097799+0000","last_undegraded":"2024-04-06T13:48:21.097799+0000","last_fullsized":"2024-04-06T13:48:21.097799+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:24.879568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683541+0000","last_change":"2024-04-06T13:48:19.690929+0000","last_active":"2024-04-06T13:48:20.683541+0000","last_peered":"2024-04-06T13:48:20.683541+0000","last_clean":"2024-04-06T13:48:20.683541+0000","last_became_active":"2024-04-06T13:48:19.690663+0000","last_became_peered":"2024-04-06T13:48:19.690663+0000","last_unstale":"2024-04-06T13:48:20.683541+0000","last_undegraded":"2024-04-06T13:48:20.683541+0000","last_fullsized":"2024-04-06T13:48:20.683541+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:52:01.901954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097559+0000","last_change":"2024-04-06T13:48:19.693212+0000","last_active":"2024-04-06T13:48:21.097559+0000","last_peered":"2024-04-06T13:48:21.097559+0000","last_clean":"2024-04-06T13:48:21.097559+0000","last_became_active":"2024-04-06T13:48:19.692957+0000","last_became_peered":"2024-04-06T13:48:19.692957+0000","last_unstale":"2024-04-06T13:48:21.097559+0000","last_undegraded":"2024-04-06T13:48:21.097559+0000","last_fullsized":"2024-04-06T13:48:21.097559+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:00:25.538231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097435+0000","last_change":"2024-04-06T13:48:19.700696+0000","last_active":"2024-04-06T13:48:21.097435+0000","last_peered":"2024-04-06T13:48:21.097435+0000","last_clean":"2024-04-06T13:48:21.097435+0000","last_became_active":"2024-04-06T13:48:19.700444+0000","last_became_peered":"2024-04-06T13:48:19.700444+0000","last_unstale":"2024-04-06T13:48:21.097435+0000","last_undegraded":"2024-04-06T13:48:21.097435+0000","last_fullsized":"2024-04-06T13:48:21.097435+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:46.094428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683996+0000","last_change":"2024-04-06T13:48:19.700988+0000","last_active":"2024-04-06T13:48:20.683996+0000","last_peered":"2024-04-06T13:48:20.683996+0000","last_clean":"2024-04-06T13:48:20.683996+0000","last_became_active":"2024-04-06T13:48:19.700718+0000","last_became_peered":"2024-04-06T13:48:19.700718+0000","last_unstale":"2024-04-06T13:48:20.683996+0000","last_undegraded":"2024-04-06T13:48:20.683996+0000","last_fullsized":"2024-04-06T13:48:20.683996+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:37:50.946459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098548+0000","last_change":"2024-04-06T13:48:19.700420+0000","last_active":"2024-04-06T13:48:21.098548+0000","last_peered":"2024-04-06T13:48:21.098548+0000","last_clean":"2024-04-06T13:48:21.098548+0000","last_became_active":"2024-04-06T13:48:19.698705+0000","last_became_peered":"2024-04-06T13:48:19.698705+0000","last_unstale":"2024-04-06T13:48:21.098548+0000","last_undegraded":"2024-04-06T13:48:21.098548+0000","last_fullsized":"2024-04-06T13:48:21.098548+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:29:58.555526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899939+0000","last_change":"2024-04-06T13:48:19.702407+0000","last_active":"2024-04-06T13:48:20.899939+0000","last_peered":"2024-04-06T13:48:20.899939+0000","last_clean":"2024-04-06T13:48:20.899939+0000","last_became_active":"2024-04-06T13:48:19.702139+0000","last_became_peered":"2024-04-06T13:48:19.702139+0000","last_unstale":"2024-04-06T13:48:20.899939+0000","last_undegraded":"2024-04-06T13:48:20.899939+0000","last_fullsized":"2024-04-06T13:48:20.899939+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:33.923465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098117+0000","last_change":"2024-04-06T13:48:19.692817+0000","last_active":"2024-04-06T13:48:21.098117+0000","last_peered":"2024-04-06T13:48:21.098117+0000","last_clean":"2024-04-06T13:48:21.098117+0000","last_became_active":"2024-04-06T13:48:19.692567+0000","last_became_peered":"2024-04-06T13:48:19.692567+0000","last_unstale":"2024-04-06T13:48:21.098117+0000","last_undegraded":"2024-04-06T13:48:21.098117+0000","last_fullsized":"2024-04-06T13:48:21.098117+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:22.432067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683285+0000","last_change":"2024-04-06T13:48:19.688698+0000","last_active":"2024-04-06T13:48:20.683285+0000","last_peered":"2024-04-06T13:48:20.683285+0000","last_clean":"2024-04-06T13:48:20.683285+0000","last_became_active":"2024-04-06T13:48:19.688377+0000","last_became_peered":"2024-04-06T13:48:19.688377+0000","last_unstale":"2024-04-06T13:48:20.683285+0000","last_undegraded":"2024-04-06T13:48:20.683285+0000","last_fullsized":"2024-04-06T13:48:20.683285+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:22.432067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683150+0000","last_change":"2024-04-06T13:48:19.698243+0000","last_active":"2024-04-06T13:48:20.683150+0000","last_peered":"2024-04-06T13:48:20.683150+0000","last_clean":"2024-04-06T13:48:20.683150+0000","last_became_active":"2024-04-06T13:48:19.697956+0000","last_became_peered":"2024-04-06T13:48:19.697956+0000","last_unstale":"2024-04-06T13:48:20.683150+0000","last_undegraded":"2024-04-06T13:48:20.683150+0000","last_fullsized":"2024-04-06T13:48:20.683150+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:26:21.699641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.900099+0000","last_change":"2024-04-06T13:48:19.694103+0000","last_active":"2024-04-06T13:48:20.900099+0000","last_peered":"2024-04-06T13:48:20.900099+0000","last_clean":"2024-04-06T13:48:20.900099+0000","last_became_active":"2024-04-06T13:48:19.693643+0000","last_became_peered":"2024-04-06T13:48:19.693643+0000","last_unstale":"2024-04-06T13:48:20.900099+0000","last_undegraded":"2024-04-06T13:48:20.900099+0000","last_fullsized":"2024-04-06T13:48:20.900099+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:00:35.709952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097980+0000","last_change":"2024-04-06T13:48:19.697522+0000","last_active":"2024-04-06T13:48:21.097980+0000","last_peered":"2024-04-06T13:48:21.097980+0000","last_clean":"2024-04-06T13:48:21.097980+0000","last_became_active":"2024-04-06T13:48:19.695505+0000","last_became_peered":"2024-04-06T13:48:19.695505+0000","last_unstale":"2024-04-06T13:48:21.097980+0000","last_undegraded":"2024-04-06T13:48:21.097980+0000","last_fullsized":"2024-04-06T13:48:21.097980+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:37:50.946459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097790+0000","last_change":"2024-04-06T13:48:19.694038+0000","last_active":"2024-04-06T13:48:21.097790+0000","last_peered":"2024-04-06T13:48:21.097790+0000","last_clean":"2024-04-06T13:48:21.097790+0000","last_became_active":"2024-04-06T13:48:19.693775+0000","last_became_peered":"2024-04-06T13:48:19.693775+0000","last_unstale":"2024-04-06T13:48:21.097790+0000","last_undegraded":"2024-04-06T13:48:21.097790+0000","last_fullsized":"2024-04-06T13:48:21.097790+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:38:06.251591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097641+0000","last_change":"2024-04-06T13:48:19.699860+0000","last_active":"2024-04-06T13:48:21.097641+0000","last_peered":"2024-04-06T13:48:21.097641+0000","last_clean":"2024-04-06T13:48:21.097641+0000","last_became_active":"2024-04-06T13:48:19.699567+0000","last_became_peered":"2024-04-06T13:48:19.699567+0000","last_unstale":"2024-04-06T13:48:21.097641+0000","last_undegraded":"2024-04-06T13:48:21.097641+0000","last_fullsized":"2024-04-06T13:48:21.097641+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:01:05.087706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899870+0000","last_change":"2024-04-06T13:48:19.697603+0000","last_active":"2024-04-06T13:48:20.899870+0000","last_peered":"2024-04-06T13:48:20.899870+0000","last_clean":"2024-04-06T13:48:20.899870+0000","last_became_active":"2024-04-06T13:48:19.697356+0000","last_became_peered":"2024-04-06T13:48:19.697356+0000","last_unstale":"2024-04-06T13:48:20.899870+0000","last_undegraded":"2024-04-06T13:48:20.899870+0000","last_fullsized":"2024-04-06T13:48:20.899870+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:44.773715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899804+0000","last_change":"2024-04-06T13:48:19.696446+0000","last_active":"2024-04-06T13:48:20.899804+0000","last_peered":"2024-04-06T13:48:20.899804+0000","last_clean":"2024-04-06T13:48:20.899804+0000","last_became_active":"2024-04-06T13:48:19.696090+0000","last_became_peered":"2024-04-06T13:48:19.696090+0000","last_unstale":"2024-04-06T13:48:20.899804+0000","last_undegraded":"2024-04-06T13:48:20.899804+0000","last_fullsized":"2024-04-06T13:48:20.899804+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:35.461214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.266186+0000","last_change":"2024-04-06T13:48:18.266186+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.266186+0000","last_undegraded":"2024-04-06T13:48:18.266186+0000","last_fullsized":"2024-04-06T13:48:18.266186+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:54:56.369118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1d","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.695623+0000","last_change":"2024-04-06T13:48:19.280859+0000","last_active":"2024-04-06T13:48:19.695623+0000","last_peered":"2024-04-06T13:48:19.695623+0000","last_clean":"2024-04-06T13:48:19.695623+0000","last_became_active":"2024-04-06T13:48:19.280512+0000","last_became_peered":"2024-04-06T13:48:19.280512+0000","last_unstale":"2024-04-06T13:48:19.695623+0000","last_undegraded":"2024-04-06T13:48:19.695623+0000","last_fullsized":"2024-04-06T13:48:19.695623+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:43.765754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1c","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.703978+0000","last_change":"2024-04-06T13:48:19.280934+0000","last_active":"2024-04-06T13:48:19.703978+0000","last_peered":"2024-04-06T13:48:19.703978+0000","last_clean":"2024-04-06T13:48:19.703978+0000","last_became_active":"2024-04-06T13:48:19.280677+0000","last_became_peered":"2024-04-06T13:48:19.280677+0000","last_unstale":"2024-04-06T13:48:19.703978+0000","last_undegraded":"2024-04-06T13:48:19.703978+0000","last_fullsized":"2024-04-06T13:48:19.703978+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:30:23.776677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098008+0000","last_change":"2024-04-06T13:48:19.695081+0000","last_active":"2024-04-06T13:48:21.098008+0000","last_peered":"2024-04-06T13:48:21.098008+0000","last_clean":"2024-04-06T13:48:21.098008+0000","last_became_active":"2024-04-06T13:48:19.694838+0000","last_became_peered":"2024-04-06T13:48:19.694838+0000","last_unstale":"2024-04-06T13:48:21.098008+0000","last_undegraded":"2024-04-06T13:48:21.098008+0000","last_fullsized":"2024-04-06T13:48:21.098008+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:37:44.732983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262863+0000","last_change":"2024-04-06T13:48:18.262863+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262863+0000","last_undegraded":"2024-04-06T13:48:18.262863+0000","last_fullsized":"2024-04-06T13:48:18.262863+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:18.513042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098240+0000","last_change":"2024-04-06T13:48:19.694881+0000","last_active":"2024-04-06T13:48:21.098240+0000","last_peered":"2024-04-06T13:48:21.098240+0000","last_clean":"2024-04-06T13:48:21.098240+0000","last_became_active":"2024-04-06T13:48:19.693362+0000","last_became_peered":"2024-04-06T13:48:19.693362+0000","last_unstale":"2024-04-06T13:48:21.098240+0000","last_undegraded":"2024-04-06T13:48:21.098240+0000","last_fullsized":"2024-04-06T13:48:21.098240+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:36:47.917157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.699585+0000","last_change":"2024-04-06T13:48:19.274234+0000","last_active":"2024-04-06T13:48:19.699585+0000","last_peered":"2024-04-06T13:48:19.699585+0000","last_clean":"2024-04-06T13:48:19.699585+0000","last_became_active":"2024-04-06T13:48:19.273998+0000","last_became_peered":"2024-04-06T13:48:19.273998+0000","last_unstale":"2024-04-06T13:48:19.699585+0000","last_undegraded":"2024-04-06T13:48:19.699585+0000","last_fullsized":"2024-04-06T13:48:19.699585+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:49:23.791293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098351+0000","last_change":"2024-04-06T13:48:19.694934+0000","last_active":"2024-04-06T13:48:21.098351+0000","last_peered":"2024-04-06T13:48:21.098351+0000","last_clean":"2024-04-06T13:48:21.098351+0000","last_became_active":"2024-04-06T13:48:19.692682+0000","last_became_peered":"2024-04-06T13:48:19.692682+0000","last_unstale":"2024-04-06T13:48:21.098351+0000","last_undegraded":"2024-04-06T13:48:21.098351+0000","last_fullsized":"2024-04-06T13:48:21.098351+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:40:43.027987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.684729+0000","last_change":"2024-04-06T13:48:19.279719+0000","last_active":"2024-04-06T13:48:19.684729+0000","last_peered":"2024-04-06T13:48:19.684729+0000","last_clean":"2024-04-06T13:48:19.684729+0000","last_became_active":"2024-04-06T13:48:19.279404+0000","last_became_peered":"2024-04-06T13:48:19.279404+0000","last_unstale":"2024-04-06T13:48:19.684729+0000","last_undegraded":"2024-04-06T13:48:19.684729+0000","last_fullsized":"2024-04-06T13:48:19.684729+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:42:35.927858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.265371+0000","last_change":"2024-04-06T13:48:18.265371+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.265371+0000","last_undegraded":"2024-04-06T13:48:18.265371+0000","last_fullsized":"2024-04-06T13:48:18.265371+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:02:13.092175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098078+0000","last_change":"2024-04-06T13:48:19.694595+0000","last_active":"2024-04-06T13:48:21.098078+0000","last_peered":"2024-04-06T13:48:21.098078+0000","last_clean":"2024-04-06T13:48:21.098078+0000","last_became_active":"2024-04-06T13:48:19.692548+0000","last_became_peered":"2024-04-06T13:48:19.692548+0000","last_unstale":"2024-04-06T13:48:21.098078+0000","last_undegraded":"2024-04-06T13:48:21.098078+0000","last_fullsized":"2024-04-06T13:48:21.098078+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:39:01.587642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.684653+0000","last_change":"2024-04-06T13:48:19.273745+0000","last_active":"2024-04-06T13:48:19.684653+0000","last_peered":"2024-04-06T13:48:19.684653+0000","last_clean":"2024-04-06T13:48:19.684653+0000","last_became_active":"2024-04-06T13:48:19.273478+0000","last_became_peered":"2024-04-06T13:48:19.273478+0000","last_unstale":"2024-04-06T13:48:19.684653+0000","last_undegraded":"2024-04-06T13:48:19.684653+0000","last_fullsized":"2024-04-06T13:48:19.684653+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:00:55.130902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683007+0000","last_change":"2024-04-06T13:48:19.688682+0000","last_active":"2024-04-06T13:48:20.683007+0000","last_peered":"2024-04-06T13:48:20.683007+0000","last_clean":"2024-04-06T13:48:20.683007+0000","last_became_active":"2024-04-06T13:48:19.688397+0000","last_became_peered":"2024-04-06T13:48:19.688397+0000","last_unstale":"2024-04-06T13:48:20.683007+0000","last_undegraded":"2024-04-06T13:48:20.683007+0000","last_fullsized":"2024-04-06T13:48:20.683007+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:39:01.587642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.683105+0000","last_change":"2024-04-06T13:48:19.278056+0000","last_active":"2024-04-06T13:48:19.683105+0000","last_peered":"2024-04-06T13:48:19.683105+0000","last_clean":"2024-04-06T13:48:19.683105+0000","last_became_active":"2024-04-06T13:48:19.277796+0000","last_became_peered":"2024-04-06T13:48:19.277796+0000","last_unstale":"2024-04-06T13:48:19.683105+0000","last_undegraded":"2024-04-06T13:48:19.683105+0000","last_fullsized":"2024-04-06T13:48:19.683105+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:31:43.320734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097587+0000","last_change":"2024-04-06T13:48:19.700325+0000","last_active":"2024-04-06T13:48:21.097587+0000","last_peered":"2024-04-06T13:48:21.097587+0000","last_clean":"2024-04-06T13:48:21.097587+0000","last_became_active":"2024-04-06T13:48:19.698751+0000","last_became_peered":"2024-04-06T13:48:19.698751+0000","last_unstale":"2024-04-06T13:48:21.097587+0000","last_undegraded":"2024-04-06T13:48:21.097587+0000","last_fullsized":"2024-04-06T13:48:21.097587+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:16.165915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262594+0000","last_change":"2024-04-06T13:48:18.262594+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262594+0000","last_undegraded":"2024-04-06T13:48:18.262594+0000","last_fullsized":"2024-04-06T13:48:18.262594+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:06:05.488014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097951+0000","last_change":"2024-04-06T13:48:19.692164+0000","last_active":"2024-04-06T13:48:21.097951+0000","last_peered":"2024-04-06T13:48:21.097951+0000","last_clean":"2024-04-06T13:48:21.097951+0000","last_became_active":"2024-04-06T13:48:19.691910+0000","last_became_peered":"2024-04-06T13:48:19.691910+0000","last_unstale":"2024-04-06T13:48:21.097951+0000","last_undegraded":"2024-04-06T13:48:21.097951+0000","last_fullsized":"2024-04-06T13:48:21.097951+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:19:27.875509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.265322+0000","last_change":"2024-04-06T13:48:18.265322+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.265322+0000","last_undegraded":"2024-04-06T13:48:18.265322+0000","last_fullsized":"2024-04-06T13:48:18.265322+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:56:55.759595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899896+0000","last_change":"2024-04-06T13:48:19.693858+0000","last_active":"2024-04-06T13:48:20.899896+0000","last_peered":"2024-04-06T13:48:20.899896+0000","last_clean":"2024-04-06T13:48:20.899896+0000","last_became_active":"2024-04-06T13:48:19.693515+0000","last_became_peered":"2024-04-06T13:48:19.693515+0000","last_unstale":"2024-04-06T13:48:20.899896+0000","last_undegraded":"2024-04-06T13:48:20.899896+0000","last_fullsized":"2024-04-06T13:48:20.899896+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:10:26.852793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262385+0000","last_change":"2024-04-06T13:48:18.262385+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262385+0000","last_undegraded":"2024-04-06T13:48:18.262385+0000","last_fullsized":"2024-04-06T13:48:18.262385+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:31:24.701348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098230+0000","last_change":"2024-04-06T13:48:19.694858+0000","last_active":"2024-04-06T13:48:21.098230+0000","last_peered":"2024-04-06T13:48:21.098230+0000","last_clean":"2024-04-06T13:48:21.098230+0000","last_became_active":"2024-04-06T13:48:19.693049+0000","last_became_peered":"2024-04-06T13:48:19.693049+0000","last_unstale":"2024-04-06T13:48:21.098230+0000","last_undegraded":"2024-04-06T13:48:21.098230+0000","last_fullsized":"2024-04-06T13:48:21.098230+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:19:00.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.261860+0000","last_change":"2024-04-06T13:48:18.261860+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.261860+0000","last_undegraded":"2024-04-06T13:48:18.261860+0000","last_fullsized":"2024-04-06T13:48:18.261860+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:35:17.934513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.263085+0000","last_change":"2024-04-06T13:48:18.263085+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.263085+0000","last_undegraded":"2024-04-06T13:48:18.263085+0000","last_fullsized":"2024-04-06T13:48:18.263085+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:02:23.692226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.10","version":"247'8","reported_seq":19,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.704884+0000","last_change":"2024-04-06T13:48:19.274645+0000","last_active":"2024-04-06T13:48:19.704884+0000","last_peered":"2024-04-06T13:48:19.704884+0000","last_clean":"2024-04-06T13:48:19.704884+0000","last_became_active":"2024-04-06T13:48:19.274308+0000","last_became_peered":"2024-04-06T13:48:19.274308+0000","last_unstale":"2024-04-06T13:48:19.704884+0000","last_undegraded":"2024-04-06T13:48:19.704884+0000","last_fullsized":"2024-04-06T13:48:19.704884+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:31:15.493970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.265775+0000","last_change":"2024-04-06T13:48:18.265775+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.265775+0000","last_undegraded":"2024-04-06T13:48:18.265775+0000","last_fullsized":"2024-04-06T13:48:18.265775+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:49.041735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.e","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.706958+0000","last_change":"2024-04-06T13:48:19.275494+0000","last_active":"2024-04-06T13:48:19.706958+0000","last_peered":"2024-04-06T13:48:19.706958+0000","last_clean":"2024-04-06T13:48:19.706958+0000","last_became_active":"2024-04-06T13:48:19.275194+0000","last_became_peered":"2024-04-06T13:48:19.275194+0000","last_unstale":"2024-04-06T13:48:19.706958+0000","last_undegraded":"2024-04-06T13:48:19.706958+0000","last_fullsized":"2024-04-06T13:48:19.706958+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:11:29.081483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899798+0000","last_change":"2024-04-06T13:48:19.702505+0000","last_active":"2024-04-06T13:48:20.899798+0000","last_peered":"2024-04-06T13:48:20.899798+0000","last_clean":"2024-04-06T13:48:20.899798+0000","last_became_active":"2024-04-06T13:48:19.702245+0000","last_became_peered":"2024-04-06T13:48:19.702245+0000","last_unstale":"2024-04-06T13:48:20.899798+0000","last_undegraded":"2024-04-06T13:48:20.899798+0000","last_fullsized":"2024-04-06T13:48:20.899798+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:45.672315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.695714+0000","last_change":"2024-04-06T13:48:19.281905+0000","last_active":"2024-04-06T13:48:19.695714+0000","last_peered":"2024-04-06T13:48:19.695714+0000","last_clean":"2024-04-06T13:48:19.695714+0000","last_became_active":"2024-04-06T13:48:19.281052+0000","last_became_peered":"2024-04-06T13:48:19.281052+0000","last_unstale":"2024-04-06T13:48:19.695714+0000","last_undegraded":"2024-04-06T13:48:19.695714+0000","last_fullsized":"2024-04-06T13:48:19.695714+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:48.100692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097580+0000","last_change":"2024-04-06T13:48:19.700428+0000","last_active":"2024-04-06T13:48:21.097580+0000","last_peered":"2024-04-06T13:48:21.097580+0000","last_clean":"2024-04-06T13:48:21.097580+0000","last_became_active":"2024-04-06T13:48:19.699896+0000","last_became_peered":"2024-04-06T13:48:19.699896+0000","last_unstale":"2024-04-06T13:48:21.097580+0000","last_undegraded":"2024-04-06T13:48:21.097580+0000","last_fullsized":"2024-04-06T13:48:21.097580+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:00:56.138863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262279+0000","last_change":"2024-04-06T13:48:18.262279+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262279+0000","last_undegraded":"2024-04-06T13:48:18.262279+0000","last_fullsized":"2024-04-06T13:48:18.262279+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:55:10.767276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097956+0000","last_change":"2024-04-06T13:48:19.697815+0000","last_active":"2024-04-06T13:48:21.097956+0000","last_peered":"2024-04-06T13:48:21.097956+0000","last_clean":"2024-04-06T13:48:21.097956+0000","last_became_active":"2024-04-06T13:48:19.695354+0000","last_became_peered":"2024-04-06T13:48:19.695354+0000","last_unstale":"2024-04-06T13:48:21.097956+0000","last_undegraded":"2024-04-06T13:48:21.097956+0000","last_fullsized":"2024-04-06T13:48:21.097956+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:38:12.538926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.700247+0000","last_change":"2024-04-06T13:48:19.278849+0000","last_active":"2024-04-06T13:48:19.700247+0000","last_peered":"2024-04-06T13:48:19.700247+0000","last_clean":"2024-04-06T13:48:19.700247+0000","last_became_active":"2024-04-06T13:48:19.278604+0000","last_became_peered":"2024-04-06T13:48:19.278604+0000","last_unstale":"2024-04-06T13:48:19.700247+0000","last_undegraded":"2024-04-06T13:48:19.700247+0000","last_fullsized":"2024-04-06T13:48:19.700247+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:19:26.867548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.900001+0000","last_change":"2024-04-06T13:48:19.696316+0000","last_active":"2024-04-06T13:48:20.900001+0000","last_peered":"2024-04-06T13:48:20.900001+0000","last_clean":"2024-04-06T13:48:20.900001+0000","last_became_active":"2024-04-06T13:48:19.696025+0000","last_became_peered":"2024-04-06T13:48:19.696025+0000","last_unstale":"2024-04-06T13:48:20.900001+0000","last_undegraded":"2024-04-06T13:48:20.900001+0000","last_fullsized":"2024-04-06T13:48:20.900001+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:33:27.603807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.680627+0000","last_change":"2024-04-06T13:48:19.282165+0000","last_active":"2024-04-06T13:48:19.680627+0000","last_peered":"2024-04-06T13:48:19.680627+0000","last_clean":"2024-04-06T13:48:19.680627+0000","last_became_active":"2024-04-06T13:48:19.281892+0000","last_became_peered":"2024-04-06T13:48:19.281892+0000","last_unstale":"2024-04-06T13:48:19.680627+0000","last_undegraded":"2024-04-06T13:48:19.680627+0000","last_fullsized":"2024-04-06T13:48:19.680627+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:10:25.844832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683665+0000","last_change":"2024-04-06T13:48:19.699890+0000","last_active":"2024-04-06T13:48:20.683665+0000","last_peered":"2024-04-06T13:48:20.683665+0000","last_clean":"2024-04-06T13:48:20.683665+0000","last_became_active":"2024-04-06T13:48:19.699566+0000","last_became_peered":"2024-04-06T13:48:19.699566+0000","last_unstale":"2024-04-06T13:48:20.683665+0000","last_undegraded":"2024-04-06T13:48:20.683665+0000","last_fullsized":"2024-04-06T13:48:20.683665+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:10:02.935661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.682503+0000","last_change":"2024-04-06T13:48:19.276781+0000","last_active":"2024-04-06T13:48:19.682503+0000","last_peered":"2024-04-06T13:48:19.682503+0000","last_clean":"2024-04-06T13:48:19.682503+0000","last_became_active":"2024-04-06T13:48:19.275438+0000","last_became_peered":"2024-04-06T13:48:19.275438+0000","last_unstale":"2024-04-06T13:48:19.682503+0000","last_undegraded":"2024-04-06T13:48:19.682503+0000","last_fullsized":"2024-04-06T13:48:19.682503+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:35.170771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683912+0000","last_change":"2024-04-06T13:48:19.685464+0000","last_active":"2024-04-06T13:48:20.683912+0000","last_peered":"2024-04-06T13:48:20.683912+0000","last_clean":"2024-04-06T13:48:20.683912+0000","last_became_active":"2024-04-06T13:48:19.685139+0000","last_became_peered":"2024-04-06T13:48:19.685139+0000","last_unstale":"2024-04-06T13:48:20.683912+0000","last_undegraded":"2024-04-06T13:48:20.683912+0000","last_fullsized":"2024-04-06T13:48:20.683912+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:19:00.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.683371+0000","last_change":"2024-04-06T13:48:19.279096+0000","last_active":"2024-04-06T13:48:19.683371+0000","last_peered":"2024-04-06T13:48:19.683371+0000","last_clean":"2024-04-06T13:48:19.683371+0000","last_became_active":"2024-04-06T13:48:19.278774+0000","last_became_peered":"2024-04-06T13:48:19.278774+0000","last_unstale":"2024-04-06T13:48:19.683371+0000","last_undegraded":"2024-04-06T13:48:19.683371+0000","last_fullsized":"2024-04-06T13:48:19.683371+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:46:06.202483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097785+0000","last_change":"2024-04-06T13:48:19.697387+0000","last_active":"2024-04-06T13:48:21.097785+0000","last_peered":"2024-04-06T13:48:21.097785+0000","last_clean":"2024-04-06T13:48:21.097785+0000","last_became_active":"2024-04-06T13:48:19.697096+0000","last_became_peered":"2024-04-06T13:48:19.697096+0000","last_unstale":"2024-04-06T13:48:21.097785+0000","last_undegraded":"2024-04-06T13:48:21.097785+0000","last_fullsized":"2024-04-06T13:48:21.097785+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:52:01.901954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.698698+0000","last_change":"2024-04-06T13:48:19.271906+0000","last_active":"2024-04-06T13:48:19.698698+0000","last_peered":"2024-04-06T13:48:19.698698+0000","last_clean":"2024-04-06T13:48:19.698698+0000","last_became_active":"2024-04-06T13:48:19.271643+0000","last_became_peered":"2024-04-06T13:48:19.271643+0000","last_unstale":"2024-04-06T13:48:19.698698+0000","last_undegraded":"2024-04-06T13:48:19.698698+0000","last_fullsized":"2024-04-06T13:48:19.698698+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:54:44.219906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.262063+0000","last_change":"2024-04-06T13:48:18.262063+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.262063+0000","last_undegraded":"2024-04-06T13:48:18.262063+0000","last_fullsized":"2024-04-06T13:48:18.262063+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:18.981232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097491+0000","last_change":"2024-04-06T13:48:19.694094+0000","last_active":"2024-04-06T13:48:21.097491+0000","last_peered":"2024-04-06T13:48:21.097491+0000","last_clean":"2024-04-06T13:48:21.097491+0000","last_became_active":"2024-04-06T13:48:19.692958+0000","last_became_peered":"2024-04-06T13:48:19.692958+0000","last_unstale":"2024-04-06T13:48:21.097491+0000","last_undegraded":"2024-04-06T13:48:21.097491+0000","last_fullsized":"2024-04-06T13:48:21.097491+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:04:10.547739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.684628+0000","last_change":"2024-04-06T13:48:19.272078+0000","last_active":"2024-04-06T13:48:19.684628+0000","last_peered":"2024-04-06T13:48:19.684628+0000","last_clean":"2024-04-06T13:48:19.684628+0000","last_became_active":"2024-04-06T13:48:19.271672+0000","last_became_peered":"2024-04-06T13:48:19.271672+0000","last_unstale":"2024-04-06T13:48:19.684628+0000","last_undegraded":"2024-04-06T13:48:19.684628+0000","last_fullsized":"2024-04-06T13:48:19.684628+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:23.871607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.685267+0000","last_change":"2024-04-06T13:48:19.278939+0000","last_active":"2024-04-06T13:48:19.685267+0000","last_peered":"2024-04-06T13:48:19.685267+0000","last_clean":"2024-04-06T13:48:19.685267+0000","last_became_active":"2024-04-06T13:48:19.278677+0000","last_became_peered":"2024-04-06T13:48:19.278677+0000","last_unstale":"2024-04-06T13:48:19.685267+0000","last_undegraded":"2024-04-06T13:48:19.685267+0000","last_fullsized":"2024-04-06T13:48:19.685267+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:15.157954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097917+0000","last_change":"2024-04-06T13:48:19.693016+0000","last_active":"2024-04-06T13:48:21.097917+0000","last_peered":"2024-04-06T13:48:21.097917+0000","last_clean":"2024-04-06T13:48:21.097917+0000","last_became_active":"2024-04-06T13:48:19.692760+0000","last_became_peered":"2024-04-06T13:48:19.692760+0000","last_unstale":"2024-04-06T13:48:21.097917+0000","last_undegraded":"2024-04-06T13:48:21.097917+0000","last_fullsized":"2024-04-06T13:48:21.097917+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:41:37.714397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.682132+0000","last_change":"2024-04-06T13:48:19.282189+0000","last_active":"2024-04-06T13:48:19.682132+0000","last_peered":"2024-04-06T13:48:19.682132+0000","last_clean":"2024-04-06T13:48:19.682132+0000","last_became_active":"2024-04-06T13:48:19.281907+0000","last_became_peered":"2024-04-06T13:48:19.281907+0000","last_unstale":"2024-04-06T13:48:19.682132+0000","last_undegraded":"2024-04-06T13:48:19.682132+0000","last_fullsized":"2024-04-06T13:48:19.682132+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:34.453253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-06T13:48:18.261299+0000","last_change":"2024-04-06T13:48:18.261299+0000","last_active":"2024-04-06T13:48:18.248805+0000","last_peered":"2024-04-06T13:48:18.248805+0000","last_clean":"2024-04-06T13:48:18.248805+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:18.261299+0000","last_undegraded":"2024-04-06T13:48:18.261299+0000","last_fullsized":"2024-04-06T13:48:18.261299+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:51:53.597002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899957+0000","last_change":"2024-04-06T13:48:19.704846+0000","last_active":"2024-04-06T13:48:20.899957+0000","last_peered":"2024-04-06T13:48:20.899957+0000","last_clean":"2024-04-06T13:48:20.899957+0000","last_became_active":"2024-04-06T13:48:19.704420+0000","last_became_peered":"2024-04-06T13:48:19.704420+0000","last_unstale":"2024-04-06T13:48:20.899957+0000","last_undegraded":"2024-04-06T13:48:20.899957+0000","last_fullsized":"2024-04-06T13:48:20.899957+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:36.178732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.684819+0000","last_change":"2024-04-06T13:48:19.277530+0000","last_active":"2024-04-06T13:48:19.684819+0000","last_peered":"2024-04-06T13:48:19.684819+0000","last_clean":"2024-04-06T13:48:19.684819+0000","last_became_active":"2024-04-06T13:48:19.276966+0000","last_became_peered":"2024-04-06T13:48:19.276966+0000","last_unstale":"2024-04-06T13:48:19.684819+0000","last_undegraded":"2024-04-06T13:48:19.684819+0000","last_fullsized":"2024-04-06T13:48:19.684819+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:42:14.962783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683820+0000","last_change":"2024-04-06T13:48:19.688861+0000","last_active":"2024-04-06T13:48:20.683820+0000","last_peered":"2024-04-06T13:48:20.683820+0000","last_clean":"2024-04-06T13:48:20.683820+0000","last_became_active":"2024-04-06T13:48:19.688496+0000","last_became_peered":"2024-04-06T13:48:19.688496+0000","last_unstale":"2024-04-06T13:48:20.683820+0000","last_undegraded":"2024-04-06T13:48:20.683820+0000","last_fullsized":"2024-04-06T13:48:20.683820+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:36:47.917157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.680854+0000","last_change":"2024-04-06T13:48:19.276467+0000","last_active":"2024-04-06T13:48:19.680854+0000","last_peered":"2024-04-06T13:48:19.680854+0000","last_clean":"2024-04-06T13:48:19.680854+0000","last_became_active":"2024-04-06T13:48:19.275988+0000","last_became_peered":"2024-04-06T13:48:19.275988+0000","last_unstale":"2024-04-06T13:48:19.680854+0000","last_undegraded":"2024-04-06T13:48:19.680854+0000","last_fullsized":"2024-04-06T13:48:19.680854+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:44.664354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-06T13:48:19.256766+0000","last_change":"2024-04-06T13:48:19.256766+0000","last_active":"2024-04-06T13:48:19.256766+0000","last_peered":"2024-04-06T13:48:19.256766+0000","last_clean":"2024-04-06T13:48:19.256766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-06T13:48:19.256766+0000","last_undegraded":"2024-04-06T13:48:19.256766+0000","last_fullsized":"2024-04-06T13:48:19.256766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1e","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.694613+0000","last_change":"2024-04-06T13:48:19.277300+0000","last_active":"2024-04-06T13:48:19.694613+0000","last_peered":"2024-04-06T13:48:19.694613+0000","last_clean":"2024-04-06T13:48:19.694613+0000","last_became_active":"2024-04-06T13:48:19.276795+0000","last_became_peered":"2024-04-06T13:48:19.276795+0000","last_unstale":"2024-04-06T13:48:19.694613+0000","last_undegraded":"2024-04-06T13:48:19.694613+0000","last_fullsized":"2024-04-06T13:48:19.694613+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:54:25.950801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"233'370","reported_seq":1333,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-06T13:48:15.595868+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:48:15.595868+0000","last_peered":"2024-04-06T13:48:15.595868+0000","last_clean":"2024-04-06T13:48:15.595868+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:48:15.595868+0000","last_undegraded":"2024-04-06T13:48:15.595868+0000","last_fullsized":"2024-04-06T13:48:15.595868+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":82,"acting":82,"num_store_stats":5},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":70,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21,"ondisk_log_size":21,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575992,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56856,"kb_used_data":3596,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314984,"statfs":{"total":96636764160,"available":96578543616,"internally_reserved":0,"allocated":3682304,"data_stored":2174385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57899999999999996,"15min":0.59499999999999997},"min":{"1min":0.35499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.72199999999999998,"5min":0.78400000000000003,"15min":0.82499999999999996},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56799999999999995,"15min":0.58999999999999997},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78500000000000003,"5min":0.80800000000000005,"15min":0.82899999999999996},"last":0.753}]},{"osd":1,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.55000000000000004,"15min":0.57899999999999996},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77200000000000002,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.57699999999999996,"15min":0.59599999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.69899999999999995,"5min":0.83399999999999996,"15min":0.85299999999999998},"last":0.78100000000000003}]},{"osd":2,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.56299999999999994,"15min":0.58499999999999996},"min":{"1min":0.33200000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69399999999999995,"5min":0.81699999999999995,"15min":0.83899999999999997},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.58599999999999997,"15min":0.60099999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69999999999999996,"5min":0.755,"15min":0.79700000000000004},"last":0.76900000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.59399999999999997,"15min":0.60599999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.67200000000000004,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58799999999999997,"15min":0.60299999999999998},"min":{"1min":0.39000000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78900000000000003,"5min":0.82099999999999995,"15min":0.84099999999999997},"last":0.66500000000000004}]},{"osd":4,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45800000000000002,"15min":0.46600000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.81599999999999995},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.36799999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.82699999999999996},"last":0.53500000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.498,"15min":0.498},"min":{"1min":0.315,"5min":0.315,"15min":0.308},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56899999999999995,"15min":0.57799999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33600000000000002},"max":{"1min":0.74099999999999999,"5min":0.80700000000000005,"15min":0.85399999999999998},"last":0.55200000000000005}]},{"osd":6,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.34899999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.879},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57099999999999995,"15min":0.58699999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81799999999999995,"5min":0.84899999999999998,"15min":0.89800000000000002},"last":0.57199999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575991,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56260,"kb_used_data":3000,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315580,"statfs":{"total":96636764160,"available":96579153920,"internally_reserved":0,"allocated":3072000,"data_stored":1576789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.53600000000000003,"15min":0.55700000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69399999999999995,"5min":0.79700000000000004,"15min":0.83799999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.54100000000000004,"15min":0.56499999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.62,"5min":0.76900000000000002,"15min":0.81999999999999995},"last":0.61899999999999999}]},{"osd":1,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.60399999999999998,"15min":0.624},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76000000000000001,"5min":0.81699999999999995,"15min":0.85799999999999998},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.622,"15min":0.64400000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.378},"max":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.82699999999999996},"last":0.67800000000000005}]},{"osd":2,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56999999999999995,"15min":0.59399999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72099999999999997,"5min":0.79000000000000004,"15min":0.82899999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.56299999999999994,"15min":0.58599999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.80700000000000005},"last":0.58499999999999996}]},{"osd":3,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.58599999999999997,"15min":0.60199999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.73199999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.61899999999999999,"15min":0.64300000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.36099999999999999},"max":{"1min":0.80200000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.68899999999999995}]},{"osd":4,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70599999999999996,"5min":0.72299999999999998,"15min":0.81399999999999995},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.51100000000000001,"15min":0.53800000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.66400000000000003,"5min":0.82099999999999995,"15min":0.84099999999999997},"last":0.70799999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.48899999999999999,"15min":0.50600000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.33300000000000002},"max":{"1min":0.63,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52000000000000002,"15min":0.53100000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67400000000000004,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.63500000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53900000000000003,"15min":0.54800000000000004},"min":{"1min":0.34899999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.67700000000000005,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55300000000000005,"15min":0.56899999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.32800000000000001},"max":{"1min":0.71199999999999997,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.54800000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575991,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60404,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311436,"statfs":{"total":96636764160,"available":96574910464,"internally_reserved":0,"allocated":3121152,"data_stored":1603524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.38100000000000001,"15min":0.39100000000000001},"min":{"1min":0.248,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.378,"5min":0.621,"15min":0.621},"last":0.441},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.41799999999999998,"15min":0.42699999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.501,"5min":0.60799999999999998,"15min":0.69399999999999995},"last":0.27500000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.41299999999999998,"15min":0.42399999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.46899999999999997,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.44900000000000001,"15min":0.46400000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.55400000000000005,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.34799999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.46999999999999997,"15min":0.47999999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.52800000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.42699999999999999,"15min":0.439},"min":{"1min":0.28799999999999998,"5min":0.27600000000000002,"15min":0.26300000000000001},"max":{"1min":0.45500000000000002,"5min":0.63500000000000001,"15min":0.66100000000000003},"last":0.33100000000000002}]},{"osd":4,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.628,"15min":0.64400000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.33800000000000002},"max":{"1min":0.70099999999999996,"5min":0.89900000000000002,"15min":0.97399999999999998},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.64400000000000002,"15min":0.65500000000000003},"min":{"1min":0.46100000000000002,"5min":0.44500000000000001,"15min":0.44400000000000001},"max":{"1min":0.73499999999999999,"5min":0.94599999999999995,"15min":0.995},"last":0.55400000000000005}]},{"osd":5,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.64100000000000001,"15min":0.65300000000000002},"min":{"1min":0.42999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73899999999999999,"5min":0.96899999999999997,"15min":1.073},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.65700000000000003,"15min":0.66800000000000004},"min":{"1min":0.45200000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.72799999999999998,"5min":0.96499999999999997,"15min":1.0209999999999999},"last":0.47899999999999998}]},{"osd":6,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.46300000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73899999999999999,"5min":0.89200000000000002,"15min":1},"last":0.503},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.67100000000000004,"15min":0.68400000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.72499999999999998,"5min":0.94599999999999995,"15min":1.036},"last":0.48799999999999999}]},{"osd":7,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.69399999999999995,"15min":0.69499999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85399999999999998,"5min":0.94899999999999995,"15min":1.0649999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65700000000000003,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.76300000000000001,"5min":0.92900000000000005,"15min":1.056},"last":0.46899999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575991,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61004,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310836,"statfs":{"total":96636764160,"available":96574296064,"internally_reserved":0,"allocated":3727360,"data_stored":2195456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.38200000000000001,"15min":0.39300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.45100000000000001,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.39300000000000002,"15min":0.41699999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.48199999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.41399999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.40100000000000002,"15min":0.42499999999999999},"min":{"1min":0.26100000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49399999999999999,"5min":0.66700000000000004,"15min":0.70099999999999996},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.42499999999999999,"15min":0.443},"min":{"1min":0.28399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.47999999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.33300000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.40699999999999997,"15min":0.42499999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.50700000000000001,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.45700000000000002,"15min":0.47799999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.51900000000000002,"5min":0.72999999999999998,"15min":0.75700000000000001},"last":0.442}]},{"osd":4,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.60499999999999998,"15min":0.64100000000000001},"min":{"1min":0.39100000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.80900000000000005,"5min":0.93100000000000005,"15min":1.0329999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.61199999999999999,"15min":0.64300000000000002},"min":{"1min":0.372,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88500000000000001,"5min":0.999,"15min":0.999},"last":0.57899999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.60199999999999998,"15min":0.63900000000000001},"min":{"1min":0.39300000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.0009999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.63800000000000001,"15min":0.66300000000000003},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.749,"5min":0.92800000000000005,"15min":1.052},"last":0.501}]},{"osd":6,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.64500000000000002,"15min":0.67400000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.874,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.64200000000000002,"15min":0.66900000000000004},"min":{"1min":0.40500000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77500000000000002,"5min":0.96399999999999997,"15min":1.085},"last":0.67600000000000005}]},{"osd":7,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.66300000000000003,"15min":0.69099999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.80700000000000005,"5min":1.01,"15min":1.121},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.64700000000000002,"15min":0.67200000000000004},"min":{"1min":0.42199999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.93700000000000006,"5min":0.95799999999999996,"15min":1.0229999999999999},"last":0.69399999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575993,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60420,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311420,"statfs":{"total":96636764160,"available":96574894080,"internally_reserved":0,"allocated":3129344,"data_stored":1604176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.376,"15min":0.38400000000000001},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.42299999999999999,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.40600000000000003,"15min":0.42199999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.249},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":1.044},"last":0.27900000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.38500000000000001,"15min":0.40400000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51100000000000001,"5min":0.54000000000000004,"15min":0.753},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.42799999999999999,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.82199999999999995},"last":0.35999999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.439,"15min":0.45800000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.53200000000000003,"5min":0.69699999999999995,"15min":0.70299999999999996},"last":0.374},{"interface":"front","average":{"1min":0.377,"5min":0.437,"15min":0.44900000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.504,"5min":0.64500000000000002,"15min":0.68600000000000005},"last":0.41499999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.621,"15min":0.63900000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.376},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.64400000000000002,"15min":0.66100000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.88900000000000001,"5min":1.022,"15min":1.022},"last":0.629}]},{"osd":5,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.64300000000000002,"15min":0.66300000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88100000000000001,"5min":0.98599999999999999,"15min":1.0149999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.65100000000000002,"15min":0.66700000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.65200000000000002,"15min":0.67200000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92000000000000004,"5min":0.97199999999999998,"15min":1.091},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.67600000000000005,"15min":0.69499999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.96099999999999997},"last":0.51200000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.66600000000000004,"15min":0.68799999999999994},"min":{"1min":0.42699999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.95399999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.68300000000000005,"15min":0.70799999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.85299999999999998,"5min":1.01,"15min":1.01},"last":0.53600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575991,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48084,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323756,"statfs":{"total":96636764160,"available":96587526144,"internally_reserved":0,"allocated":3088384,"data_stored":1584017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39000000000000001,"15min":0.38900000000000001},"min":{"1min":0.26000000000000001,"5min":0.255,"15min":0.23499999999999999},"max":{"1min":0.55200000000000005,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41199999999999998,"15min":0.41299999999999998},"min":{"1min":0.28799999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.56899999999999995,"5min":0.622,"15min":0.66700000000000004},"last":0.34000000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40699999999999997,"15min":0.40600000000000003},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.65700000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.438,"15min":0.44400000000000001},"min":{"1min":0.32800000000000001,"5min":0.27500000000000002,"15min":0.26600000000000001},"max":{"1min":0.63,"5min":0.65400000000000003,"15min":0.75800000000000001},"last":0.39900000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48299999999999998,"15min":0.48599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.5,"15min":0.503},"min":{"1min":0.38600000000000001,"5min":0.34599999999999997,"15min":0.32900000000000001},"max":{"1min":0.67800000000000005,"5min":0.69299999999999995,"15min":0.79300000000000004},"last":0.49199999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.439,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.876,"5min":1.002,"15min":1.018},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66100000000000003,"15min":0.65300000000000002},"min":{"1min":0.497,"5min":0.45900000000000002,"15min":0.375},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.60399999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.51900000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.95699999999999996,"5min":0.97299999999999998,"15min":0.999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.46200000000000002,"5min":0.437,"15min":0.437},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.61499999999999999}]},{"osd":6,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67500000000000004,"15min":0.66900000000000004},"min":{"1min":0.41499999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.63},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.68600000000000005,"15min":0.68400000000000005},"min":{"1min":0.48799999999999999,"5min":0.46700000000000003,"15min":0.41099999999999998},"max":{"1min":0.88800000000000001,"5min":0.96199999999999997,"15min":0.98799999999999999},"last":0.57399999999999995}]},{"osd":7,"last update":"Sat Apr 6 13:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68500000000000005,"15min":0.68000000000000005},"min":{"1min":0.46100000000000002,"5min":0.432,"15min":0.40999999999999998},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70299999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.024},"last":0.71199999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575992,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48132,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323708,"statfs":{"total":96636764160,"available":96587476992,"internally_reserved":0,"allocated":3129344,"data_stored":1604613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55900000000000005,"15min":0.58699999999999997},"min":{"1min":0.40699999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.753,"5min":0.876,"15min":0.88500000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.39300000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81100000000000005,"5min":1.202,"15min":1.202},"last":0.48299999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.61299999999999999,"15min":0.64000000000000001},"min":{"1min":0.40799999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71699999999999997,"5min":1.256,"15min":1.256},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.624,"15min":0.63},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.75900000000000001,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.56499999999999995}]},{"osd":2,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.57999999999999996,"15min":0.59999999999999998},"min":{"1min":0.41599999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77500000000000002,"5min":0.84599999999999997,"15min":0.877},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.60599999999999998,"15min":0.61599999999999999},"min":{"1min":0.39600000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.97899999999999998,"5min":1.294,"15min":1.294},"last":0.41199999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84499999999999997,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.60899999999999999,"15min":0.626},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.78800000000000003,"5min":0.90200000000000002,"15min":0.91100000000000003},"last":0.55400000000000005}]},{"osd":5,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55500000000000005,"15min":0.52000000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.92800000000000005,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.56399999999999995,"15min":0.54100000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.95199999999999996,"5min":4.931,"15min":4.931},"last":0.63500000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53500000000000003,"15min":0.53300000000000003},"min":{"1min":0.34999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.505},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34699999999999998},"max":{"1min":1.0609999999999999,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.62}]},{"osd":7,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59499999999999997,"15min":0.58199999999999996},"min":{"1min":0.36899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.91000000000000003,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59399999999999997,"15min":0.60199999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.373},"max":{"1min":0.86899999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.53200000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575992,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64508,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307332,"statfs":{"total":96636764160,"available":96570707968,"internally_reserved":0,"allocated":3121152,"data_stored":1603524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.57399999999999995,"15min":0.58499999999999996},"min":{"1min":0.36399999999999999,"5min":0.35899999999999999,"15min":0.33000000000000002},"max":{"1min":0.68000000000000005,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58099999999999996,"15min":0.59399999999999997},"min":{"1min":0.38100000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.76600000000000001,"5min":0.84699999999999998,"15min":0.89200000000000002},"last":0.63200000000000001}]},{"osd":1,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.58099999999999996,"15min":0.59399999999999997},"min":{"1min":0.38400000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79100000000000004,"5min":0.81299999999999994,"15min":0.81399999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.439,"5min":0.39400000000000002,"15min":0.37},"max":{"1min":0.70299999999999996,"5min":0.871,"15min":0.871},"last":0.59199999999999997}]},{"osd":2,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59299999999999997,"15min":0.59699999999999998},"min":{"1min":0.311,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69199999999999995,"5min":0.79700000000000004,"15min":0.84399999999999997},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58799999999999997,"15min":0.59599999999999997},"min":{"1min":0.44,"5min":0.378,"15min":0.378},"max":{"1min":0.73099999999999998,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.51700000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.79800000000000004,"5min":0.81100000000000005,"15min":0.94499999999999995},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61699999999999999,"15min":0.63200000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.74099999999999999,"5min":0.88200000000000001,"15min":0.91500000000000004},"last":0.61499999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52900000000000003,"15min":0.50700000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.27400000000000002},"max":{"1min":0.70999999999999996,"5min":0.85699999999999998,"15min":1.0369999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.59999999999999998,"15min":0.57699999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.308},"max":{"1min":0.77800000000000002,"5min":0.88800000000000001,"15min":1.1639999999999999},"last":0.70299999999999996}]},{"osd":6,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54300000000000004,"15min":0.52600000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.27900000000000003},"max":{"1min":0.89100000000000001,"5min":0.94999999999999996,"15min":1.0169999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54300000000000004,"15min":0.53100000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.32300000000000001},"max":{"1min":0.81899999999999995,"5min":0.878,"15min":1.0009999999999999},"last":0.68799999999999994}]},{"osd":7,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56499999999999995,"15min":0.54800000000000004},"min":{"1min":0.39200000000000002,"5min":0.35799999999999998,"15min":0.32900000000000001},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.66400000000000003,"15min":0.64800000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.35199999999999998},"max":{"1min":0.85299999999999998,"5min":0.92300000000000004,"15min":0.96599999999999997},"last":0.73199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:48:28.388 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:48:28.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.522+0000 7efcb909b700 1 Processor -- start 2024-04-06T13:48:28.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.522+0000 7efcb909b700 1 -- start start 2024-04-06T13:48:28.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.523+0000 7efcb909b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcb406ae40 0x7efcb40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:28.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.523+0000 7efcb909b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb40b3990 con 0x7efcb415ddf0 2024-04-06T13:48:28.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.523+0000 7efcb909b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb40b3060 con 0x7efcb415d7b0 2024-04-06T13:48:28.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.523+0000 7efcb909b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb40b3340 con 0x7efcb406ae40 2024-04-06T13:48:28.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.523+0000 7efcb359e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efcb415ddf0 0x7efcb415e2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33242/0 (socket says 172.21.15.189:33242) 2024-04-06T13:48:28.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.523+0000 7efcb359e700 1 -- 172.21.15.189:0/3169961383 learned_addr learned my addr 172.21.15.189:0/3169961383 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:28.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb259c700 1 --2- 172.21.15.189:0/3169961383 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcb406ae40 0x7efcb40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:28.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb1d9b700 1 -- 172.21.15.189:0/3169961383 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511127129 0 0) 0x7efcb40b3990 con 0x7efcb415ddf0 2024-04-06T13:48:28.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb1d9b700 1 -- 172.21.15.189:0/3169961383 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc9c0035b0 con 0x7efcb415ddf0 2024-04-06T13:48:28.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb1d9b700 1 -- 172.21.15.189:0/3169961383 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3038249269 0 0) 0x7efc9c0035b0 con 0x7efcb415ddf0 2024-04-06T13:48:28.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb1d9b700 1 -- 172.21.15.189:0/3169961383 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcb406ae40 msgr2=0x7efcb40b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:28.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb1d9b700 1 --2- 172.21.15.189:0/3169961383 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcb406ae40 0x7efcb40b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:28.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb1d9b700 1 -- 172.21.15.189:0/3169961383 >> v1:172.21.15.192:6789/0 conn(0x7efcb415d7b0 legacy=0x7efcb406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:28.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb1d9b700 1 -- 172.21.15.189:0/3169961383 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb40b3620 con 0x7efcb415ddf0 2024-04-06T13:48:28.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb1d9b700 1 -- 172.21.15.189:0/3169961383 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7efcac0019f0 con 0x7efcb415ddf0 2024-04-06T13:48:28.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb1d9b700 1 -- 172.21.15.189:0/3169961383 >> v1:172.21.15.189:6789/0 conn(0x7efcb415ddf0 legacy=0x7efcb415e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:28.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb1d9b700 1 --2- 172.21.15.189:0/3169961383 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc9c003a30 0x7efc9c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:28.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb1d9b700 1 --2- 172.21.15.189:0/3169961383 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcb406ae40 0x7efc9c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:28.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb1d9b700 1 --2- 172.21.15.189:0/3169961383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9c004d90 0x7efc9c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:28.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb1d9b700 1 -- 172.21.15.189:0/3169961383 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb40b3340 con 0x7efc9c004d90 2024-04-06T13:48:28.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb1d9b700 1 -- 172.21.15.189:0/3169961383 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c00bb40 con 0x7efc9c003a30 2024-04-06T13:48:28.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb1d9b700 1 -- 172.21.15.189:0/3169961383 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c00bd00 con 0x7efcb406ae40 2024-04-06T13:48:28.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb359e700 1 --2- 172.21.15.189:0/3169961383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9c004d90 0x7efc9c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:28.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb2d9d700 1 --2- 172.21.15.189:0/3169961383 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc9c003a30 0x7efc9c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:28.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb259c700 1 --2- 172.21.15.189:0/3169961383 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcb406ae40 0x7efc9c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:28.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb359e700 1 -- 172.21.15.189:0/3169961383 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcb406ae40 msgr2=0x7efc9c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:28.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb359e700 1 --2- 172.21.15.189:0/3169961383 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcb406ae40 0x7efc9c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:28.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb359e700 1 -- 172.21.15.189:0/3169961383 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc9c003a30 msgr2=0x7efc9c003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:28.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb359e700 1 --2- 172.21.15.189:0/3169961383 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc9c003a30 0x7efc9c003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:28.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.524+0000 7efcb359e700 1 -- 172.21.15.189:0/3169961383 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9c0035b0 con 0x7efc9c004d90 2024-04-06T13:48:28.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.525+0000 7efcb359e700 1 --2- 172.21.15.189:0/3169961383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9c004d90 0x7efc9c005210 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7efcac00a830 tx=0x7efc9c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74ae15a6cf6bccb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:28.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.525+0000 7efcb1d9b700 1 -- 172.21.15.189:0/3169961383 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac00a2f0 con 0x7efc9c004d90 2024-04-06T13:48:28.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.525+0000 7efcb1d9b700 1 -- 172.21.15.189:0/3169961383 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcac012620 con 0x7efc9c004d90 2024-04-06T13:48:28.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.525+0000 7efcb1d9b700 1 -- 172.21.15.189:0/3169961383 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac0129d0 con 0x7efc9c004d90 2024-04-06T13:48:28.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.525+0000 7efcb909b700 1 -- 172.21.15.189:0/3169961383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9c004d90 msgr2=0x7efc9c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:28.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.525+0000 7efcb909b700 1 --2- 172.21.15.189:0/3169961383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9c004d90 0x7efc9c005210 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7efcac00a830 tx=0x7efc9c005880 comp rx=0 tx=0).stop 2024-04-06T13:48:28.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.525+0000 7efcb359e700 1 -- 172.21.15.189:0/3169961383 reap_dead start 2024-04-06T13:48:28.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.525+0000 7efcb909b700 1 -- 172.21.15.189:0/3169961383 shutdown_connections 2024-04-06T13:48:28.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.525+0000 7efcb909b700 1 -- 172.21.15.189:0/3169961383 >> 172.21.15.189:0/3169961383 conn(0x7efcb40d00b0 msgr2=0x7efcb40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:28.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.525+0000 7efcb909b700 1 -- 172.21.15.189:0/3169961383 shutdown_connections 2024-04-06T13:48:28.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.525+0000 7efcb909b700 1 -- 172.21.15.189:0/3169961383 wait complete. 2024-04-06T13:48:28.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.527+0000 7efcb909b700 1 Processor -- start 2024-04-06T13:48:28.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.527+0000 7efcb909b700 1 -- start start 2024-04-06T13:48:28.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb909b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb40af240 0x7efcb40b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:28.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb909b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcb414c670 0x7efcb40af620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:28.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb909b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcb414caf0 0x7efcb40ad360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:28.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb909b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb4155740 con 0x7efcb40af240 2024-04-06T13:48:28.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb909b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb4155d00 con 0x7efcb414c670 2024-04-06T13:48:28.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb909b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb40b0720 con 0x7efcb414caf0 2024-04-06T13:48:28.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb2d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb40af240 0x7efcb40b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:28.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb2d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb40af240 0x7efcb40b01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45142/0 (socket says 172.21.15.189:45142) 2024-04-06T13:48:28.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb2d9d700 1 -- 172.21.15.189:0/3990649265 learned_addr learned my addr 172.21.15.189:0/3990649265 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:28.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb359e700 1 --2- 172.21.15.189:0/3990649265 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcb414caf0 0x7efcb40ad360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:28.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb259c700 1 --2- 172.21.15.189:0/3990649265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcb414c670 0x7efcb40af620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:28.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb2d9d700 1 -- 172.21.15.189:0/3990649265 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcb414caf0 msgr2=0x7efcb40ad360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:28.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb2d9d700 1 --2- 172.21.15.189:0/3990649265 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcb414caf0 0x7efcb40ad360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:28.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb2d9d700 1 -- 172.21.15.189:0/3990649265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcb414c670 msgr2=0x7efcb40af620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:28.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb2d9d700 1 --2- 172.21.15.189:0/3990649265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcb414c670 0x7efcb40af620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:28.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb2d9d700 1 -- 172.21.15.189:0/3990649265 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb40b0980 con 0x7efcb40af240 2024-04-06T13:48:28.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb259c700 1 --2- 172.21.15.189:0/3990649265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcb414c670 0x7efcb40af620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:28.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb2d9d700 1 --2- 172.21.15.189:0/3990649265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb40af240 0x7efcb40b01e0 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7efca400f7f0 tx=0x7efca400fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:28.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efc9b7fe700 1 -- 172.21.15.189:0/3990649265 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca40052e0 con 0x7efcb40af240 2024-04-06T13:48:28.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efc9b7fe700 1 -- 172.21.15.189:0/3990649265 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca4005440 con 0x7efcb40af240 2024-04-06T13:48:28.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb909b700 1 -- 172.21.15.189:0/3990649265 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb40b0c60 con 0x7efcb40af240 2024-04-06T13:48:28.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efc9b7fe700 1 -- 172.21.15.189:0/3990649265 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca40038c0 con 0x7efcb40af240 2024-04-06T13:48:28.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.528+0000 7efcb909b700 1 -- 172.21.15.189:0/3990649265 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb40afb60 con 0x7efcb40af240 2024-04-06T13:48:28.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.529+0000 7efcb909b700 1 -- 172.21.15.189:0/3990649265 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb40b0ea0 con 0x7efcb40af240 2024-04-06T13:48:28.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.530+0000 7efc9b7fe700 1 -- 172.21.15.189:0/3990649265 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7efca4003a20 con 0x7efcb40af240 2024-04-06T13:48:28.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.530+0000 7efc9b7fe700 1 --2- 172.21.15.189:0/3990649265 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efca006aed0 0x7efca006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:28.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.530+0000 7efc9b7fe700 1 -- 172.21.15.189:0/3990649265 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7efca4089760 con 0x7efcb40af240 2024-04-06T13:48:28.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.530+0000 7efcb259c700 1 --2- 172.21.15.189:0/3990649265 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efca006aed0 0x7efca006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:28.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.530+0000 7efcb259c700 1 --2- 172.21.15.189:0/3990649265 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efca006aed0 0x7efca006d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efcb4070230 tx=0x7efca8008040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:28.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.535+0000 7efc9b7fe700 1 -- 172.21.15.189:0/3990649265 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efca4054450 con 0x7efcb40af240 2024-04-06T13:48:28.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.726+0000 7efcb909b700 1 -- 172.21.15.189:0/3990649265 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efcb4067bb0 con 0x7efca006aed0 2024-04-06T13:48:28.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.734+0000 7efc9b7fe700 1 -- 172.21.15.189:0/3990649265 <== mgr.11733 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290199 (secure 0 0 0) 0x7efcb4067bb0 con 0x7efca006aed0 2024-04-06T13:48:28.738 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:28.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.737+0000 7efcb909b700 1 -- 172.21.15.189:0/3990649265 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efca006aed0 msgr2=0x7efca006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:28.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.737+0000 7efcb909b700 1 --2- 172.21.15.189:0/3990649265 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efca006aed0 0x7efca006d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efcb4070230 tx=0x7efca8008040 comp rx=0 tx=0).stop 2024-04-06T13:48:28.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.737+0000 7efcb909b700 1 -- 172.21.15.189:0/3990649265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb40af240 msgr2=0x7efcb40b01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:28.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.738+0000 7efcb909b700 1 --2- 172.21.15.189:0/3990649265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb40af240 0x7efcb40b01e0 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7efca400f7f0 tx=0x7efca400fbb0 comp rx=0 tx=0).stop 2024-04-06T13:48:28.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.738+0000 7efcb909b700 1 -- 172.21.15.189:0/3990649265 shutdown_connections 2024-04-06T13:48:28.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.738+0000 7efcb909b700 1 --2- 172.21.15.189:0/3990649265 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efca006aed0 0x7efca006d390 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:28.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.738+0000 7efcb909b700 1 --2- 172.21.15.189:0/3990649265 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcb414caf0 0x7efcb40ad360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:28.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.738+0000 7efcb909b700 1 --2- 172.21.15.189:0/3990649265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb40af240 0x7efcb40b01e0 unknown :-1 s=CLOSED pgs=2689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:28.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.738+0000 7efcb909b700 1 --2- 172.21.15.189:0/3990649265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcb414c670 0x7efcb40af620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:28.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.738+0000 7efcb909b700 1 -- 172.21.15.189:0/3990649265 >> 172.21.15.189:0/3990649265 conn(0x7efcb40d00b0 msgr2=0x7efcb40ae0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:28.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.738+0000 7efcb909b700 1 -- 172.21.15.189:0/3990649265 shutdown_connections 2024-04-06T13:48:28.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.738+0000 7efcb909b700 1 -- 172.21.15.189:0/3990649265 wait complete. 2024-04-06T13:48:28.742 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:48:28.750 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-04-06T13:48:27.928534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":663,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":408,"ondisk_log_size":408,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":455864,"kb_used_data":25620,"kb_used_omap":12,"kb_used_meta":430067,"kb_avail":754518856,"statfs":{"total":773094113280,"available":772627308544,"internally_reserved":0,"allocated":26234880,"data_stored":14016709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":440389176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.250488"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.684635+0000","last_change":"2024-04-06T13:48:19.691215+0000","last_active":"2024-04-06T13:48:20.684635+0000","last_peered":"2024-04-06T13:48:20.684635+0000","last_clean":"2024-04-06T13:48:20.684635+0000","last_became_active":"2024-04-06T13:48:19.689693+0000","last_became_peered":"2024-04-06T13:48:19.689693+0000","last_unstale":"2024-04-06T13:48:20.684635+0000","last_undegraded":"2024-04-06T13:48:20.684635+0000","last_fullsized":"2024-04-06T13:48:20.684635+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:49:37.511498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.900020+0000","last_change":"2024-04-06T13:48:19.704688+0000","last_active":"2024-04-06T13:48:20.900020+0000","last_peered":"2024-04-06T13:48:20.900020+0000","last_clean":"2024-04-06T13:48:20.900020+0000","last_became_active":"2024-04-06T13:48:19.704088+0000","last_became_peered":"2024-04-06T13:48:19.704088+0000","last_unstale":"2024-04-06T13:48:20.900020+0000","last_undegraded":"2024-04-06T13:48:20.900020+0000","last_fullsized":"2024-04-06T13:48:20.900020+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:49.108653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097823+0000","last_change":"2024-04-06T13:48:19.697461+0000","last_active":"2024-04-06T13:48:21.097823+0000","last_peered":"2024-04-06T13:48:21.097823+0000","last_clean":"2024-04-06T13:48:21.097823+0000","last_became_active":"2024-04-06T13:48:19.695459+0000","last_became_peered":"2024-04-06T13:48:19.695459+0000","last_unstale":"2024-04-06T13:48:21.097823+0000","last_undegraded":"2024-04-06T13:48:21.097823+0000","last_fullsized":"2024-04-06T13:48:21.097823+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:42:12.990594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098046+0000","last_change":"2024-04-06T13:48:19.690589+0000","last_active":"2024-04-06T13:48:21.098046+0000","last_peered":"2024-04-06T13:48:21.098046+0000","last_clean":"2024-04-06T13:48:21.098046+0000","last_became_active":"2024-04-06T13:48:19.690338+0000","last_became_peered":"2024-04-06T13:48:19.690338+0000","last_unstale":"2024-04-06T13:48:21.098046+0000","last_undegraded":"2024-04-06T13:48:21.098046+0000","last_fullsized":"2024-04-06T13:48:21.098046+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:51:42.234400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098529+0000","last_change":"2024-04-06T13:48:19.698891+0000","last_active":"2024-04-06T13:48:21.098529+0000","last_peered":"2024-04-06T13:48:21.098529+0000","last_clean":"2024-04-06T13:48:21.098529+0000","last_became_active":"2024-04-06T13:48:19.698477+0000","last_became_peered":"2024-04-06T13:48:19.698477+0000","last_unstale":"2024-04-06T13:48:21.098529+0000","last_undegraded":"2024-04-06T13:48:21.098529+0000","last_fullsized":"2024-04-06T13:48:21.098529+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:02:19.171219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.684095+0000","last_change":"2024-04-06T13:48:19.684151+0000","last_active":"2024-04-06T13:48:20.684095+0000","last_peered":"2024-04-06T13:48:20.684095+0000","last_clean":"2024-04-06T13:48:20.684095+0000","last_became_active":"2024-04-06T13:48:19.683897+0000","last_became_peered":"2024-04-06T13:48:19.683897+0000","last_unstale":"2024-04-06T13:48:20.684095+0000","last_undegraded":"2024-04-06T13:48:20.684095+0000","last_fullsized":"2024-04-06T13:48:20.684095+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:47.270406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097813+0000","last_change":"2024-04-06T13:48:19.695665+0000","last_active":"2024-04-06T13:48:21.097813+0000","last_peered":"2024-04-06T13:48:21.097813+0000","last_clean":"2024-04-06T13:48:21.097813+0000","last_became_active":"2024-04-06T13:48:19.695362+0000","last_became_peered":"2024-04-06T13:48:19.695362+0000","last_unstale":"2024-04-06T13:48:21.097813+0000","last_undegraded":"2024-04-06T13:48:21.097813+0000","last_fullsized":"2024-04-06T13:48:21.097813+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:26:21.699641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097975+0000","last_change":"2024-04-06T13:48:19.704200+0000","last_active":"2024-04-06T13:48:21.097975+0000","last_peered":"2024-04-06T13:48:21.097975+0000","last_clean":"2024-04-06T13:48:21.097975+0000","last_became_active":"2024-04-06T13:48:19.703951+0000","last_became_peered":"2024-04-06T13:48:19.703951+0000","last_unstale":"2024-04-06T13:48:21.097975+0000","last_undegraded":"2024-04-06T13:48:21.097975+0000","last_fullsized":"2024-04-06T13:48:21.097975+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:31:20.020434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.684619+0000","last_change":"2024-04-06T13:48:19.705385+0000","last_active":"2024-04-06T13:48:20.684619+0000","last_peered":"2024-04-06T13:48:20.684619+0000","last_clean":"2024-04-06T13:48:20.684619+0000","last_became_active":"2024-04-06T13:48:19.705091+0000","last_became_peered":"2024-04-06T13:48:19.705091+0000","last_unstale":"2024-04-06T13:48:20.684619+0000","last_undegraded":"2024-04-06T13:48:20.684619+0000","last_fullsized":"2024-04-06T13:48:20.684619+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:23:39.345908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098450+0000","last_change":"2024-04-06T13:48:19.700300+0000","last_active":"2024-04-06T13:48:21.098450+0000","last_peered":"2024-04-06T13:48:21.098450+0000","last_clean":"2024-04-06T13:48:21.098450+0000","last_became_active":"2024-04-06T13:48:19.700054+0000","last_became_peered":"2024-04-06T13:48:19.700054+0000","last_unstale":"2024-04-06T13:48:21.098450+0000","last_undegraded":"2024-04-06T13:48:21.098450+0000","last_fullsized":"2024-04-06T13:48:21.098450+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:30:14.717681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097799+0000","last_change":"2024-04-06T13:48:19.700151+0000","last_active":"2024-04-06T13:48:21.097799+0000","last_peered":"2024-04-06T13:48:21.097799+0000","last_clean":"2024-04-06T13:48:21.097799+0000","last_became_active":"2024-04-06T13:48:19.699882+0000","last_became_peered":"2024-04-06T13:48:19.699882+0000","last_unstale":"2024-04-06T13:48:21.097799+0000","last_undegraded":"2024-04-06T13:48:21.097799+0000","last_fullsized":"2024-04-06T13:48:21.097799+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:24.879568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.684197+0000","last_change":"2024-04-06T13:48:19.683194+0000","last_active":"2024-04-06T13:48:20.684197+0000","last_peered":"2024-04-06T13:48:20.684197+0000","last_clean":"2024-04-06T13:48:20.684197+0000","last_became_active":"2024-04-06T13:48:19.682953+0000","last_became_peered":"2024-04-06T13:48:19.682953+0000","last_unstale":"2024-04-06T13:48:20.684197+0000","last_undegraded":"2024-04-06T13:48:20.684197+0000","last_fullsized":"2024-04-06T13:48:20.684197+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:33:52.351082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683541+0000","last_change":"2024-04-06T13:48:19.690929+0000","last_active":"2024-04-06T13:48:20.683541+0000","last_peered":"2024-04-06T13:48:20.683541+0000","last_clean":"2024-04-06T13:48:20.683541+0000","last_became_active":"2024-04-06T13:48:19.690663+0000","last_became_peered":"2024-04-06T13:48:19.690663+0000","last_unstale":"2024-04-06T13:48:20.683541+0000","last_undegraded":"2024-04-06T13:48:20.683541+0000","last_fullsized":"2024-04-06T13:48:20.683541+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:52:01.901954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097559+0000","last_change":"2024-04-06T13:48:19.693212+0000","last_active":"2024-04-06T13:48:21.097559+0000","last_peered":"2024-04-06T13:48:21.097559+0000","last_clean":"2024-04-06T13:48:21.097559+0000","last_became_active":"2024-04-06T13:48:19.692957+0000","last_became_peered":"2024-04-06T13:48:19.692957+0000","last_unstale":"2024-04-06T13:48:21.097559+0000","last_undegraded":"2024-04-06T13:48:21.097559+0000","last_fullsized":"2024-04-06T13:48:21.097559+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:00:25.538231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683869+0000","last_change":"2024-04-06T13:48:19.693508+0000","last_active":"2024-04-06T13:48:20.683869+0000","last_peered":"2024-04-06T13:48:20.683869+0000","last_clean":"2024-04-06T13:48:20.683869+0000","last_became_active":"2024-04-06T13:48:19.693250+0000","last_became_peered":"2024-04-06T13:48:19.693250+0000","last_unstale":"2024-04-06T13:48:20.683869+0000","last_undegraded":"2024-04-06T13:48:20.683869+0000","last_fullsized":"2024-04-06T13:48:20.683869+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:00:56.138863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097435+0000","last_change":"2024-04-06T13:48:19.700696+0000","last_active":"2024-04-06T13:48:21.097435+0000","last_peered":"2024-04-06T13:48:21.097435+0000","last_clean":"2024-04-06T13:48:21.097435+0000","last_became_active":"2024-04-06T13:48:19.700444+0000","last_became_peered":"2024-04-06T13:48:19.700444+0000","last_unstale":"2024-04-06T13:48:21.097435+0000","last_undegraded":"2024-04-06T13:48:21.097435+0000","last_fullsized":"2024-04-06T13:48:21.097435+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:46.094428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683996+0000","last_change":"2024-04-06T13:48:19.700988+0000","last_active":"2024-04-06T13:48:20.683996+0000","last_peered":"2024-04-06T13:48:20.683996+0000","last_clean":"2024-04-06T13:48:20.683996+0000","last_became_active":"2024-04-06T13:48:19.700718+0000","last_became_peered":"2024-04-06T13:48:19.700718+0000","last_unstale":"2024-04-06T13:48:20.683996+0000","last_undegraded":"2024-04-06T13:48:20.683996+0000","last_fullsized":"2024-04-06T13:48:20.683996+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:37:50.946459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098548+0000","last_change":"2024-04-06T13:48:19.700420+0000","last_active":"2024-04-06T13:48:21.098548+0000","last_peered":"2024-04-06T13:48:21.098548+0000","last_clean":"2024-04-06T13:48:21.098548+0000","last_became_active":"2024-04-06T13:48:19.698705+0000","last_became_peered":"2024-04-06T13:48:19.698705+0000","last_unstale":"2024-04-06T13:48:21.098548+0000","last_undegraded":"2024-04-06T13:48:21.098548+0000","last_fullsized":"2024-04-06T13:48:21.098548+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:29:58.555526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.684066+0000","last_change":"2024-04-06T13:48:19.695591+0000","last_active":"2024-04-06T13:48:20.684066+0000","last_peered":"2024-04-06T13:48:20.684066+0000","last_clean":"2024-04-06T13:48:20.684066+0000","last_became_active":"2024-04-06T13:48:19.695289+0000","last_became_peered":"2024-04-06T13:48:19.695289+0000","last_unstale":"2024-04-06T13:48:20.684066+0000","last_undegraded":"2024-04-06T13:48:20.684066+0000","last_fullsized":"2024-04-06T13:48:20.684066+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:16.165915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899939+0000","last_change":"2024-04-06T13:48:19.702407+0000","last_active":"2024-04-06T13:48:20.899939+0000","last_peered":"2024-04-06T13:48:20.899939+0000","last_clean":"2024-04-06T13:48:20.899939+0000","last_became_active":"2024-04-06T13:48:19.702139+0000","last_became_peered":"2024-04-06T13:48:19.702139+0000","last_unstale":"2024-04-06T13:48:20.899939+0000","last_undegraded":"2024-04-06T13:48:20.899939+0000","last_fullsized":"2024-04-06T13:48:20.899939+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:33.923465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098117+0000","last_change":"2024-04-06T13:48:19.692817+0000","last_active":"2024-04-06T13:48:21.098117+0000","last_peered":"2024-04-06T13:48:21.098117+0000","last_clean":"2024-04-06T13:48:21.098117+0000","last_became_active":"2024-04-06T13:48:19.692567+0000","last_became_peered":"2024-04-06T13:48:19.692567+0000","last_unstale":"2024-04-06T13:48:21.098117+0000","last_undegraded":"2024-04-06T13:48:21.098117+0000","last_fullsized":"2024-04-06T13:48:21.098117+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:22.432067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683285+0000","last_change":"2024-04-06T13:48:19.688698+0000","last_active":"2024-04-06T13:48:20.683285+0000","last_peered":"2024-04-06T13:48:20.683285+0000","last_clean":"2024-04-06T13:48:20.683285+0000","last_became_active":"2024-04-06T13:48:19.688377+0000","last_became_peered":"2024-04-06T13:48:19.688377+0000","last_unstale":"2024-04-06T13:48:20.683285+0000","last_undegraded":"2024-04-06T13:48:20.683285+0000","last_fullsized":"2024-04-06T13:48:20.683285+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:22.432067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683963+0000","last_change":"2024-04-06T13:48:19.693370+0000","last_active":"2024-04-06T13:48:20.683963+0000","last_peered":"2024-04-06T13:48:20.683963+0000","last_clean":"2024-04-06T13:48:20.683963+0000","last_became_active":"2024-04-06T13:48:19.692893+0000","last_became_peered":"2024-04-06T13:48:19.692893+0000","last_unstale":"2024-04-06T13:48:20.683963+0000","last_undegraded":"2024-04-06T13:48:20.683963+0000","last_fullsized":"2024-04-06T13:48:20.683963+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:24.879568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683150+0000","last_change":"2024-04-06T13:48:19.698243+0000","last_active":"2024-04-06T13:48:20.683150+0000","last_peered":"2024-04-06T13:48:20.683150+0000","last_clean":"2024-04-06T13:48:20.683150+0000","last_became_active":"2024-04-06T13:48:19.697956+0000","last_became_peered":"2024-04-06T13:48:19.697956+0000","last_unstale":"2024-04-06T13:48:20.683150+0000","last_undegraded":"2024-04-06T13:48:20.683150+0000","last_fullsized":"2024-04-06T13:48:20.683150+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:26:21.699641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.900099+0000","last_change":"2024-04-06T13:48:19.694103+0000","last_active":"2024-04-06T13:48:20.900099+0000","last_peered":"2024-04-06T13:48:20.900099+0000","last_clean":"2024-04-06T13:48:20.900099+0000","last_became_active":"2024-04-06T13:48:19.693643+0000","last_became_peered":"2024-04-06T13:48:19.693643+0000","last_unstale":"2024-04-06T13:48:20.900099+0000","last_undegraded":"2024-04-06T13:48:20.900099+0000","last_fullsized":"2024-04-06T13:48:20.900099+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:00:35.709952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097980+0000","last_change":"2024-04-06T13:48:19.697522+0000","last_active":"2024-04-06T13:48:21.097980+0000","last_peered":"2024-04-06T13:48:21.097980+0000","last_clean":"2024-04-06T13:48:21.097980+0000","last_became_active":"2024-04-06T13:48:19.695505+0000","last_became_peered":"2024-04-06T13:48:19.695505+0000","last_unstale":"2024-04-06T13:48:21.097980+0000","last_undegraded":"2024-04-06T13:48:21.097980+0000","last_fullsized":"2024-04-06T13:48:21.097980+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:37:50.946459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098781+0000","last_change":"2024-04-06T13:48:19.699786+0000","last_active":"2024-04-06T13:48:21.098781+0000","last_peered":"2024-04-06T13:48:21.098781+0000","last_clean":"2024-04-06T13:48:21.098781+0000","last_became_active":"2024-04-06T13:48:19.698719+0000","last_became_peered":"2024-04-06T13:48:19.698719+0000","last_unstale":"2024-04-06T13:48:21.098781+0000","last_undegraded":"2024-04-06T13:48:21.098781+0000","last_fullsized":"2024-04-06T13:48:21.098781+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:54:26.958761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.684281+0000","last_change":"2024-04-06T13:48:19.704381+0000","last_active":"2024-04-06T13:48:20.684281+0000","last_peered":"2024-04-06T13:48:20.684281+0000","last_clean":"2024-04-06T13:48:20.684281+0000","last_became_active":"2024-04-06T13:48:19.704140+0000","last_became_peered":"2024-04-06T13:48:19.704140+0000","last_unstale":"2024-04-06T13:48:20.684281+0000","last_undegraded":"2024-04-06T13:48:20.684281+0000","last_fullsized":"2024-04-06T13:48:20.684281+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:30:47.386830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097790+0000","last_change":"2024-04-06T13:48:19.694038+0000","last_active":"2024-04-06T13:48:21.097790+0000","last_peered":"2024-04-06T13:48:21.097790+0000","last_clean":"2024-04-06T13:48:21.097790+0000","last_became_active":"2024-04-06T13:48:19.693775+0000","last_became_peered":"2024-04-06T13:48:19.693775+0000","last_unstale":"2024-04-06T13:48:21.097790+0000","last_undegraded":"2024-04-06T13:48:21.097790+0000","last_fullsized":"2024-04-06T13:48:21.097790+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:38:06.251591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097641+0000","last_change":"2024-04-06T13:48:19.699860+0000","last_active":"2024-04-06T13:48:21.097641+0000","last_peered":"2024-04-06T13:48:21.097641+0000","last_clean":"2024-04-06T13:48:21.097641+0000","last_became_active":"2024-04-06T13:48:19.699567+0000","last_became_peered":"2024-04-06T13:48:19.699567+0000","last_unstale":"2024-04-06T13:48:21.097641+0000","last_undegraded":"2024-04-06T13:48:21.097641+0000","last_fullsized":"2024-04-06T13:48:21.097641+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:01:05.087706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899870+0000","last_change":"2024-04-06T13:48:19.697603+0000","last_active":"2024-04-06T13:48:20.899870+0000","last_peered":"2024-04-06T13:48:20.899870+0000","last_clean":"2024-04-06T13:48:20.899870+0000","last_became_active":"2024-04-06T13:48:19.697356+0000","last_became_peered":"2024-04-06T13:48:19.697356+0000","last_unstale":"2024-04-06T13:48:20.899870+0000","last_undegraded":"2024-04-06T13:48:20.899870+0000","last_fullsized":"2024-04-06T13:48:20.899870+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:44.773715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.684112+0000","last_change":"2024-04-06T13:48:19.705129+0000","last_active":"2024-04-06T13:48:20.684112+0000","last_peered":"2024-04-06T13:48:20.684112+0000","last_clean":"2024-04-06T13:48:20.684112+0000","last_became_active":"2024-04-06T13:48:19.704722+0000","last_became_peered":"2024-04-06T13:48:19.704722+0000","last_unstale":"2024-04-06T13:48:20.684112+0000","last_undegraded":"2024-04-06T13:48:20.684112+0000","last_fullsized":"2024-04-06T13:48:20.684112+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:18.449707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899804+0000","last_change":"2024-04-06T13:48:19.696446+0000","last_active":"2024-04-06T13:48:20.899804+0000","last_peered":"2024-04-06T13:48:20.899804+0000","last_clean":"2024-04-06T13:48:20.899804+0000","last_became_active":"2024-04-06T13:48:19.696090+0000","last_became_peered":"2024-04-06T13:48:19.696090+0000","last_unstale":"2024-04-06T13:48:20.899804+0000","last_undegraded":"2024-04-06T13:48:20.899804+0000","last_fullsized":"2024-04-06T13:48:20.899804+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:35.461214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"247'4","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.707430+0000","last_change":"2024-04-06T13:48:19.281764+0000","last_active":"2024-04-06T13:48:19.707430+0000","last_peered":"2024-04-06T13:48:19.707430+0000","last_clean":"2024-04-06T13:48:19.707430+0000","last_became_active":"2024-04-06T13:48:19.281501+0000","last_became_peered":"2024-04-06T13:48:19.281501+0000","last_unstale":"2024-04-06T13:48:19.707430+0000","last_undegraded":"2024-04-06T13:48:19.707430+0000","last_fullsized":"2024-04-06T13:48:19.707430+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:54:56.369118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.684207+0000","last_change":"2024-04-06T13:48:19.699483+0000","last_active":"2024-04-06T13:48:20.684207+0000","last_peered":"2024-04-06T13:48:20.684207+0000","last_clean":"2024-04-06T13:48:20.684207+0000","last_became_active":"2024-04-06T13:48:19.699209+0000","last_became_peered":"2024-04-06T13:48:19.699209+0000","last_unstale":"2024-04-06T13:48:20.684207+0000","last_undegraded":"2024-04-06T13:48:20.684207+0000","last_fullsized":"2024-04-06T13:48:20.684207+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:42:12.990594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.695623+0000","last_change":"2024-04-06T13:48:19.280859+0000","last_active":"2024-04-06T13:48:19.695623+0000","last_peered":"2024-04-06T13:48:19.695623+0000","last_clean":"2024-04-06T13:48:19.695623+0000","last_became_active":"2024-04-06T13:48:19.280512+0000","last_became_peered":"2024-04-06T13:48:19.280512+0000","last_unstale":"2024-04-06T13:48:19.695623+0000","last_undegraded":"2024-04-06T13:48:19.695623+0000","last_fullsized":"2024-04-06T13:48:19.695623+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:43.765754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.684232+0000","last_change":"2024-04-06T13:48:19.687508+0000","last_active":"2024-04-06T13:48:20.684232+0000","last_peered":"2024-04-06T13:48:20.684232+0000","last_clean":"2024-04-06T13:48:20.684232+0000","last_became_active":"2024-04-06T13:48:19.687246+0000","last_became_peered":"2024-04-06T13:48:19.687246+0000","last_unstale":"2024-04-06T13:48:20.684232+0000","last_undegraded":"2024-04-06T13:48:20.684232+0000","last_fullsized":"2024-04-06T13:48:20.684232+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:15:08.111319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.703978+0000","last_change":"2024-04-06T13:48:19.280934+0000","last_active":"2024-04-06T13:48:19.703978+0000","last_peered":"2024-04-06T13:48:19.703978+0000","last_clean":"2024-04-06T13:48:19.703978+0000","last_became_active":"2024-04-06T13:48:19.280677+0000","last_became_peered":"2024-04-06T13:48:19.280677+0000","last_unstale":"2024-04-06T13:48:19.703978+0000","last_undegraded":"2024-04-06T13:48:19.703978+0000","last_fullsized":"2024-04-06T13:48:19.703978+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:30:23.776677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098008+0000","last_change":"2024-04-06T13:48:19.695081+0000","last_active":"2024-04-06T13:48:21.098008+0000","last_peered":"2024-04-06T13:48:21.098008+0000","last_clean":"2024-04-06T13:48:21.098008+0000","last_became_active":"2024-04-06T13:48:19.694838+0000","last_became_peered":"2024-04-06T13:48:19.694838+0000","last_unstale":"2024-04-06T13:48:21.098008+0000","last_undegraded":"2024-04-06T13:48:21.098008+0000","last_fullsized":"2024-04-06T13:48:21.098008+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:37:44.732983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.705137+0000","last_change":"2024-04-06T13:48:19.282906+0000","last_active":"2024-04-06T13:48:19.705137+0000","last_peered":"2024-04-06T13:48:19.705137+0000","last_clean":"2024-04-06T13:48:19.705137+0000","last_became_active":"2024-04-06T13:48:19.282628+0000","last_became_peered":"2024-04-06T13:48:19.282628+0000","last_unstale":"2024-04-06T13:48:19.705137+0000","last_undegraded":"2024-04-06T13:48:19.705137+0000","last_fullsized":"2024-04-06T13:48:19.705137+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:18.513042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098240+0000","last_change":"2024-04-06T13:48:19.694881+0000","last_active":"2024-04-06T13:48:21.098240+0000","last_peered":"2024-04-06T13:48:21.098240+0000","last_clean":"2024-04-06T13:48:21.098240+0000","last_became_active":"2024-04-06T13:48:19.693362+0000","last_became_peered":"2024-04-06T13:48:19.693362+0000","last_unstale":"2024-04-06T13:48:21.098240+0000","last_undegraded":"2024-04-06T13:48:21.098240+0000","last_fullsized":"2024-04-06T13:48:21.098240+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:36:47.917157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.699585+0000","last_change":"2024-04-06T13:48:19.274234+0000","last_active":"2024-04-06T13:48:19.699585+0000","last_peered":"2024-04-06T13:48:19.699585+0000","last_clean":"2024-04-06T13:48:19.699585+0000","last_became_active":"2024-04-06T13:48:19.273998+0000","last_became_peered":"2024-04-06T13:48:19.273998+0000","last_unstale":"2024-04-06T13:48:19.699585+0000","last_undegraded":"2024-04-06T13:48:19.699585+0000","last_fullsized":"2024-04-06T13:48:19.699585+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:49:23.791293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098351+0000","last_change":"2024-04-06T13:48:19.694934+0000","last_active":"2024-04-06T13:48:21.098351+0000","last_peered":"2024-04-06T13:48:21.098351+0000","last_clean":"2024-04-06T13:48:21.098351+0000","last_became_active":"2024-04-06T13:48:19.692682+0000","last_became_peered":"2024-04-06T13:48:19.692682+0000","last_unstale":"2024-04-06T13:48:21.098351+0000","last_undegraded":"2024-04-06T13:48:21.098351+0000","last_fullsized":"2024-04-06T13:48:21.098351+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:40:43.027987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.684729+0000","last_change":"2024-04-06T13:48:19.279719+0000","last_active":"2024-04-06T13:48:19.684729+0000","last_peered":"2024-04-06T13:48:19.684729+0000","last_clean":"2024-04-06T13:48:19.684729+0000","last_became_active":"2024-04-06T13:48:19.279404+0000","last_became_peered":"2024-04-06T13:48:19.279404+0000","last_unstale":"2024-04-06T13:48:19.684729+0000","last_undegraded":"2024-04-06T13:48:19.684729+0000","last_fullsized":"2024-04-06T13:48:19.684729+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:42:35.927858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098843+0000","last_change":"2024-04-06T13:48:19.702348+0000","last_active":"2024-04-06T13:48:21.098843+0000","last_peered":"2024-04-06T13:48:21.098843+0000","last_clean":"2024-04-06T13:48:21.098843+0000","last_became_active":"2024-04-06T13:48:19.701891+0000","last_became_peered":"2024-04-06T13:48:19.701891+0000","last_unstale":"2024-04-06T13:48:21.098843+0000","last_undegraded":"2024-04-06T13:48:21.098843+0000","last_fullsized":"2024-04-06T13:48:21.098843+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:52:52.810060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.679191+0000","last_change":"2024-04-06T13:48:19.281924+0000","last_active":"2024-04-06T13:48:19.679191+0000","last_peered":"2024-04-06T13:48:19.679191+0000","last_clean":"2024-04-06T13:48:19.679191+0000","last_became_active":"2024-04-06T13:48:19.281609+0000","last_became_peered":"2024-04-06T13:48:19.281609+0000","last_unstale":"2024-04-06T13:48:19.679191+0000","last_undegraded":"2024-04-06T13:48:19.679191+0000","last_fullsized":"2024-04-06T13:48:19.679191+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:02:13.092175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098078+0000","last_change":"2024-04-06T13:48:19.694595+0000","last_active":"2024-04-06T13:48:21.098078+0000","last_peered":"2024-04-06T13:48:21.098078+0000","last_clean":"2024-04-06T13:48:21.098078+0000","last_became_active":"2024-04-06T13:48:19.692548+0000","last_became_peered":"2024-04-06T13:48:19.692548+0000","last_unstale":"2024-04-06T13:48:21.098078+0000","last_undegraded":"2024-04-06T13:48:21.098078+0000","last_fullsized":"2024-04-06T13:48:21.098078+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:39:01.587642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.684653+0000","last_change":"2024-04-06T13:48:19.273745+0000","last_active":"2024-04-06T13:48:19.684653+0000","last_peered":"2024-04-06T13:48:19.684653+0000","last_clean":"2024-04-06T13:48:19.684653+0000","last_became_active":"2024-04-06T13:48:19.273478+0000","last_became_peered":"2024-04-06T13:48:19.273478+0000","last_unstale":"2024-04-06T13:48:19.684653+0000","last_undegraded":"2024-04-06T13:48:19.684653+0000","last_fullsized":"2024-04-06T13:48:19.684653+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:00:55.130902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683007+0000","last_change":"2024-04-06T13:48:19.688682+0000","last_active":"2024-04-06T13:48:20.683007+0000","last_peered":"2024-04-06T13:48:20.683007+0000","last_clean":"2024-04-06T13:48:20.683007+0000","last_became_active":"2024-04-06T13:48:19.688397+0000","last_became_peered":"2024-04-06T13:48:19.688397+0000","last_unstale":"2024-04-06T13:48:20.683007+0000","last_undegraded":"2024-04-06T13:48:20.683007+0000","last_fullsized":"2024-04-06T13:48:20.683007+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:39:01.587642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.683105+0000","last_change":"2024-04-06T13:48:19.278056+0000","last_active":"2024-04-06T13:48:19.683105+0000","last_peered":"2024-04-06T13:48:19.683105+0000","last_clean":"2024-04-06T13:48:19.683105+0000","last_became_active":"2024-04-06T13:48:19.277796+0000","last_became_peered":"2024-04-06T13:48:19.277796+0000","last_unstale":"2024-04-06T13:48:19.683105+0000","last_undegraded":"2024-04-06T13:48:19.683105+0000","last_fullsized":"2024-04-06T13:48:19.683105+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:31:43.320734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097587+0000","last_change":"2024-04-06T13:48:19.700325+0000","last_active":"2024-04-06T13:48:21.097587+0000","last_peered":"2024-04-06T13:48:21.097587+0000","last_clean":"2024-04-06T13:48:21.097587+0000","last_became_active":"2024-04-06T13:48:19.698751+0000","last_became_peered":"2024-04-06T13:48:19.698751+0000","last_unstale":"2024-04-06T13:48:21.097587+0000","last_undegraded":"2024-04-06T13:48:21.097587+0000","last_fullsized":"2024-04-06T13:48:21.097587+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:16.165915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"247'3","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.699735+0000","last_change":"2024-04-06T13:48:19.283170+0000","last_active":"2024-04-06T13:48:19.699735+0000","last_peered":"2024-04-06T13:48:19.699735+0000","last_clean":"2024-04-06T13:48:19.699735+0000","last_became_active":"2024-04-06T13:48:19.282896+0000","last_became_peered":"2024-04-06T13:48:19.282896+0000","last_unstale":"2024-04-06T13:48:19.699735+0000","last_undegraded":"2024-04-06T13:48:19.699735+0000","last_fullsized":"2024-04-06T13:48:19.699735+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:06:05.488014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097951+0000","last_change":"2024-04-06T13:48:19.692164+0000","last_active":"2024-04-06T13:48:21.097951+0000","last_peered":"2024-04-06T13:48:21.097951+0000","last_clean":"2024-04-06T13:48:21.097951+0000","last_became_active":"2024-04-06T13:48:19.691910+0000","last_became_peered":"2024-04-06T13:48:19.691910+0000","last_unstale":"2024-04-06T13:48:21.097951+0000","last_undegraded":"2024-04-06T13:48:21.097951+0000","last_fullsized":"2024-04-06T13:48:21.097951+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:19:27.875509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"247'6","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.708346+0000","last_change":"2024-04-06T13:48:19.281276+0000","last_active":"2024-04-06T13:48:19.708346+0000","last_peered":"2024-04-06T13:48:19.708346+0000","last_clean":"2024-04-06T13:48:19.708346+0000","last_became_active":"2024-04-06T13:48:19.280081+0000","last_became_peered":"2024-04-06T13:48:19.280081+0000","last_unstale":"2024-04-06T13:48:19.708346+0000","last_undegraded":"2024-04-06T13:48:19.708346+0000","last_fullsized":"2024-04-06T13:48:19.708346+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:56:55.759595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899896+0000","last_change":"2024-04-06T13:48:19.693858+0000","last_active":"2024-04-06T13:48:20.899896+0000","last_peered":"2024-04-06T13:48:20.899896+0000","last_clean":"2024-04-06T13:48:20.899896+0000","last_became_active":"2024-04-06T13:48:19.693515+0000","last_became_peered":"2024-04-06T13:48:19.693515+0000","last_unstale":"2024-04-06T13:48:20.899896+0000","last_undegraded":"2024-04-06T13:48:20.899896+0000","last_fullsized":"2024-04-06T13:48:20.899896+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:10:26.852793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.706627+0000","last_change":"2024-04-06T13:48:19.283816+0000","last_active":"2024-04-06T13:48:19.706627+0000","last_peered":"2024-04-06T13:48:19.706627+0000","last_clean":"2024-04-06T13:48:19.706627+0000","last_became_active":"2024-04-06T13:48:19.283543+0000","last_became_peered":"2024-04-06T13:48:19.283543+0000","last_unstale":"2024-04-06T13:48:19.706627+0000","last_undegraded":"2024-04-06T13:48:19.706627+0000","last_fullsized":"2024-04-06T13:48:19.706627+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:31:24.701348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098230+0000","last_change":"2024-04-06T13:48:19.694858+0000","last_active":"2024-04-06T13:48:21.098230+0000","last_peered":"2024-04-06T13:48:21.098230+0000","last_clean":"2024-04-06T13:48:21.098230+0000","last_became_active":"2024-04-06T13:48:19.693049+0000","last_became_peered":"2024-04-06T13:48:19.693049+0000","last_unstale":"2024-04-06T13:48:21.098230+0000","last_undegraded":"2024-04-06T13:48:21.098230+0000","last_fullsized":"2024-04-06T13:48:21.098230+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:19:00.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.692150+0000","last_change":"2024-04-06T13:48:19.282855+0000","last_active":"2024-04-06T13:48:19.692150+0000","last_peered":"2024-04-06T13:48:19.692150+0000","last_clean":"2024-04-06T13:48:19.692150+0000","last_became_active":"2024-04-06T13:48:19.282570+0000","last_became_peered":"2024-04-06T13:48:19.282570+0000","last_unstale":"2024-04-06T13:48:19.692150+0000","last_undegraded":"2024-04-06T13:48:19.692150+0000","last_fullsized":"2024-04-06T13:48:19.692150+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:35:17.934513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683694+0000","last_change":"2024-04-06T13:48:19.693208+0000","last_active":"2024-04-06T13:48:20.683694+0000","last_peered":"2024-04-06T13:48:20.683694+0000","last_clean":"2024-04-06T13:48:20.683694+0000","last_became_active":"2024-04-06T13:48:19.692939+0000","last_became_peered":"2024-04-06T13:48:19.692939+0000","last_unstale":"2024-04-06T13:48:20.683694+0000","last_undegraded":"2024-04-06T13:48:20.683694+0000","last_fullsized":"2024-04-06T13:48:20.683694+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:46:07.210444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.693396+0000","last_change":"2024-04-06T13:48:19.283720+0000","last_active":"2024-04-06T13:48:19.693396+0000","last_peered":"2024-04-06T13:48:19.693396+0000","last_clean":"2024-04-06T13:48:19.693396+0000","last_became_active":"2024-04-06T13:48:19.283453+0000","last_became_peered":"2024-04-06T13:48:19.283453+0000","last_unstale":"2024-04-06T13:48:19.693396+0000","last_undegraded":"2024-04-06T13:48:19.693396+0000","last_fullsized":"2024-04-06T13:48:19.693396+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:02:23.692226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098759+0000","last_change":"2024-04-06T13:48:19.700890+0000","last_active":"2024-04-06T13:48:21.098759+0000","last_peered":"2024-04-06T13:48:21.098759+0000","last_clean":"2024-04-06T13:48:21.098759+0000","last_became_active":"2024-04-06T13:48:19.700513+0000","last_became_peered":"2024-04-06T13:48:19.700513+0000","last_unstale":"2024-04-06T13:48:21.098759+0000","last_undegraded":"2024-04-06T13:48:21.098759+0000","last_fullsized":"2024-04-06T13:48:21.098759+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:35:45.830191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"247'8","reported_seq":19,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.704884+0000","last_change":"2024-04-06T13:48:19.274645+0000","last_active":"2024-04-06T13:48:19.704884+0000","last_peered":"2024-04-06T13:48:19.704884+0000","last_clean":"2024-04-06T13:48:19.704884+0000","last_became_active":"2024-04-06T13:48:19.274308+0000","last_became_peered":"2024-04-06T13:48:19.274308+0000","last_unstale":"2024-04-06T13:48:19.704884+0000","last_undegraded":"2024-04-06T13:48:19.704884+0000","last_fullsized":"2024-04-06T13:48:19.704884+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:31:15.493970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097816+0000","last_change":"2024-04-06T13:48:19.700124+0000","last_active":"2024-04-06T13:48:21.097816+0000","last_peered":"2024-04-06T13:48:21.097816+0000","last_clean":"2024-04-06T13:48:21.097816+0000","last_became_active":"2024-04-06T13:48:19.699857+0000","last_became_peered":"2024-04-06T13:48:19.699857+0000","last_unstale":"2024-04-06T13:48:21.097816+0000","last_undegraded":"2024-04-06T13:48:21.097816+0000","last_fullsized":"2024-04-06T13:48:21.097816+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:44.773715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.705288+0000","last_change":"2024-04-06T13:48:19.284301+0000","last_active":"2024-04-06T13:48:19.705288+0000","last_peered":"2024-04-06T13:48:19.705288+0000","last_clean":"2024-04-06T13:48:19.705288+0000","last_became_active":"2024-04-06T13:48:19.284062+0000","last_became_peered":"2024-04-06T13:48:19.284062+0000","last_unstale":"2024-04-06T13:48:19.705288+0000","last_undegraded":"2024-04-06T13:48:19.705288+0000","last_fullsized":"2024-04-06T13:48:19.705288+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:49.041735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098299+0000","last_change":"2024-04-06T13:48:19.702274+0000","last_active":"2024-04-06T13:48:21.098299+0000","last_peered":"2024-04-06T13:48:21.098299+0000","last_clean":"2024-04-06T13:48:21.098299+0000","last_became_active":"2024-04-06T13:48:19.701964+0000","last_became_peered":"2024-04-06T13:48:19.701964+0000","last_unstale":"2024-04-06T13:48:21.098299+0000","last_undegraded":"2024-04-06T13:48:21.098299+0000","last_fullsized":"2024-04-06T13:48:21.098299+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:45.240549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.706958+0000","last_change":"2024-04-06T13:48:19.275494+0000","last_active":"2024-04-06T13:48:19.706958+0000","last_peered":"2024-04-06T13:48:19.706958+0000","last_clean":"2024-04-06T13:48:19.706958+0000","last_became_active":"2024-04-06T13:48:19.275194+0000","last_became_peered":"2024-04-06T13:48:19.275194+0000","last_unstale":"2024-04-06T13:48:19.706958+0000","last_undegraded":"2024-04-06T13:48:19.706958+0000","last_fullsized":"2024-04-06T13:48:19.706958+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:11:29.081483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899798+0000","last_change":"2024-04-06T13:48:19.702505+0000","last_active":"2024-04-06T13:48:20.899798+0000","last_peered":"2024-04-06T13:48:20.899798+0000","last_clean":"2024-04-06T13:48:20.899798+0000","last_became_active":"2024-04-06T13:48:19.702245+0000","last_became_peered":"2024-04-06T13:48:19.702245+0000","last_unstale":"2024-04-06T13:48:20.899798+0000","last_undegraded":"2024-04-06T13:48:20.899798+0000","last_fullsized":"2024-04-06T13:48:20.899798+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:45.672315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.695714+0000","last_change":"2024-04-06T13:48:19.281905+0000","last_active":"2024-04-06T13:48:19.695714+0000","last_peered":"2024-04-06T13:48:19.695714+0000","last_clean":"2024-04-06T13:48:19.695714+0000","last_became_active":"2024-04-06T13:48:19.281052+0000","last_became_peered":"2024-04-06T13:48:19.281052+0000","last_unstale":"2024-04-06T13:48:19.695714+0000","last_undegraded":"2024-04-06T13:48:19.695714+0000","last_fullsized":"2024-04-06T13:48:19.695714+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:48.100692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097580+0000","last_change":"2024-04-06T13:48:19.700428+0000","last_active":"2024-04-06T13:48:21.097580+0000","last_peered":"2024-04-06T13:48:21.097580+0000","last_clean":"2024-04-06T13:48:21.097580+0000","last_became_active":"2024-04-06T13:48:19.699896+0000","last_became_peered":"2024-04-06T13:48:19.699896+0000","last_unstale":"2024-04-06T13:48:21.097580+0000","last_undegraded":"2024-04-06T13:48:21.097580+0000","last_fullsized":"2024-04-06T13:48:21.097580+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:00:56.138863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.685335+0000","last_change":"2024-04-06T13:48:19.286358+0000","last_active":"2024-04-06T13:48:19.685335+0000","last_peered":"2024-04-06T13:48:19.685335+0000","last_clean":"2024-04-06T13:48:19.685335+0000","last_became_active":"2024-04-06T13:48:19.286136+0000","last_became_peered":"2024-04-06T13:48:19.286136+0000","last_unstale":"2024-04-06T13:48:19.685335+0000","last_undegraded":"2024-04-06T13:48:19.685335+0000","last_fullsized":"2024-04-06T13:48:19.685335+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:55:10.767276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097956+0000","last_change":"2024-04-06T13:48:19.697815+0000","last_active":"2024-04-06T13:48:21.097956+0000","last_peered":"2024-04-06T13:48:21.097956+0000","last_clean":"2024-04-06T13:48:21.097956+0000","last_became_active":"2024-04-06T13:48:19.695354+0000","last_became_peered":"2024-04-06T13:48:19.695354+0000","last_unstale":"2024-04-06T13:48:21.097956+0000","last_undegraded":"2024-04-06T13:48:21.097956+0000","last_fullsized":"2024-04-06T13:48:21.097956+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:38:12.538926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.700247+0000","last_change":"2024-04-06T13:48:19.278849+0000","last_active":"2024-04-06T13:48:19.700247+0000","last_peered":"2024-04-06T13:48:19.700247+0000","last_clean":"2024-04-06T13:48:19.700247+0000","last_became_active":"2024-04-06T13:48:19.278604+0000","last_became_peered":"2024-04-06T13:48:19.278604+0000","last_unstale":"2024-04-06T13:48:19.700247+0000","last_undegraded":"2024-04-06T13:48:19.700247+0000","last_fullsized":"2024-04-06T13:48:19.700247+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:19:26.867548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.900001+0000","last_change":"2024-04-06T13:48:19.696316+0000","last_active":"2024-04-06T13:48:20.900001+0000","last_peered":"2024-04-06T13:48:20.900001+0000","last_clean":"2024-04-06T13:48:20.900001+0000","last_became_active":"2024-04-06T13:48:19.696025+0000","last_became_peered":"2024-04-06T13:48:19.696025+0000","last_unstale":"2024-04-06T13:48:20.900001+0000","last_undegraded":"2024-04-06T13:48:20.900001+0000","last_fullsized":"2024-04-06T13:48:20.900001+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:33:27.603807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.680627+0000","last_change":"2024-04-06T13:48:19.282165+0000","last_active":"2024-04-06T13:48:19.680627+0000","last_peered":"2024-04-06T13:48:19.680627+0000","last_clean":"2024-04-06T13:48:19.680627+0000","last_became_active":"2024-04-06T13:48:19.281892+0000","last_became_peered":"2024-04-06T13:48:19.281892+0000","last_unstale":"2024-04-06T13:48:19.680627+0000","last_undegraded":"2024-04-06T13:48:19.680627+0000","last_fullsized":"2024-04-06T13:48:19.680627+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:10:25.844832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683665+0000","last_change":"2024-04-06T13:48:19.699890+0000","last_active":"2024-04-06T13:48:20.683665+0000","last_peered":"2024-04-06T13:48:20.683665+0000","last_clean":"2024-04-06T13:48:20.683665+0000","last_became_active":"2024-04-06T13:48:19.699566+0000","last_became_peered":"2024-04-06T13:48:19.699566+0000","last_unstale":"2024-04-06T13:48:20.683665+0000","last_undegraded":"2024-04-06T13:48:20.683665+0000","last_fullsized":"2024-04-06T13:48:20.683665+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:10:02.935661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.682503+0000","last_change":"2024-04-06T13:48:19.276781+0000","last_active":"2024-04-06T13:48:19.682503+0000","last_peered":"2024-04-06T13:48:19.682503+0000","last_clean":"2024-04-06T13:48:19.682503+0000","last_became_active":"2024-04-06T13:48:19.275438+0000","last_became_peered":"2024-04-06T13:48:19.275438+0000","last_unstale":"2024-04-06T13:48:19.682503+0000","last_undegraded":"2024-04-06T13:48:19.682503+0000","last_fullsized":"2024-04-06T13:48:19.682503+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:35.170771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683912+0000","last_change":"2024-04-06T13:48:19.685464+0000","last_active":"2024-04-06T13:48:20.683912+0000","last_peered":"2024-04-06T13:48:20.683912+0000","last_clean":"2024-04-06T13:48:20.683912+0000","last_became_active":"2024-04-06T13:48:19.685139+0000","last_became_peered":"2024-04-06T13:48:19.685139+0000","last_unstale":"2024-04-06T13:48:20.683912+0000","last_undegraded":"2024-04-06T13:48:20.683912+0000","last_fullsized":"2024-04-06T13:48:20.683912+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:19:00.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.683371+0000","last_change":"2024-04-06T13:48:19.279096+0000","last_active":"2024-04-06T13:48:19.683371+0000","last_peered":"2024-04-06T13:48:19.683371+0000","last_clean":"2024-04-06T13:48:19.683371+0000","last_became_active":"2024-04-06T13:48:19.278774+0000","last_became_peered":"2024-04-06T13:48:19.278774+0000","last_unstale":"2024-04-06T13:48:19.683371+0000","last_undegraded":"2024-04-06T13:48:19.683371+0000","last_fullsized":"2024-04-06T13:48:19.683371+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:46:06.202483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097785+0000","last_change":"2024-04-06T13:48:19.697387+0000","last_active":"2024-04-06T13:48:21.097785+0000","last_peered":"2024-04-06T13:48:21.097785+0000","last_clean":"2024-04-06T13:48:21.097785+0000","last_became_active":"2024-04-06T13:48:19.697096+0000","last_became_peered":"2024-04-06T13:48:19.697096+0000","last_unstale":"2024-04-06T13:48:21.097785+0000","last_undegraded":"2024-04-06T13:48:21.097785+0000","last_fullsized":"2024-04-06T13:48:21.097785+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:52:01.901954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.698698+0000","last_change":"2024-04-06T13:48:19.271906+0000","last_active":"2024-04-06T13:48:19.698698+0000","last_peered":"2024-04-06T13:48:19.698698+0000","last_clean":"2024-04-06T13:48:19.698698+0000","last_became_active":"2024-04-06T13:48:19.271643+0000","last_became_peered":"2024-04-06T13:48:19.271643+0000","last_unstale":"2024-04-06T13:48:19.698698+0000","last_undegraded":"2024-04-06T13:48:19.698698+0000","last_fullsized":"2024-04-06T13:48:19.698698+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:54:44.219906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683773+0000","last_change":"2024-04-06T13:48:19.697332+0000","last_active":"2024-04-06T13:48:20.683773+0000","last_peered":"2024-04-06T13:48:20.683773+0000","last_clean":"2024-04-06T13:48:20.683773+0000","last_became_active":"2024-04-06T13:48:19.697087+0000","last_became_peered":"2024-04-06T13:48:19.697087+0000","last_unstale":"2024-04-06T13:48:20.683773+0000","last_undegraded":"2024-04-06T13:48:20.683773+0000","last_fullsized":"2024-04-06T13:48:20.683773+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:49:24.799254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.693797+0000","last_change":"2024-04-06T13:48:19.279294+0000","last_active":"2024-04-06T13:48:19.693797+0000","last_peered":"2024-04-06T13:48:19.693797+0000","last_clean":"2024-04-06T13:48:19.693797+0000","last_became_active":"2024-04-06T13:48:19.279009+0000","last_became_peered":"2024-04-06T13:48:19.279009+0000","last_unstale":"2024-04-06T13:48:19.693797+0000","last_undegraded":"2024-04-06T13:48:19.693797+0000","last_fullsized":"2024-04-06T13:48:19.693797+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:18.981232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097491+0000","last_change":"2024-04-06T13:48:19.694094+0000","last_active":"2024-04-06T13:48:21.097491+0000","last_peered":"2024-04-06T13:48:21.097491+0000","last_clean":"2024-04-06T13:48:21.097491+0000","last_became_active":"2024-04-06T13:48:19.692958+0000","last_became_peered":"2024-04-06T13:48:19.692958+0000","last_unstale":"2024-04-06T13:48:21.097491+0000","last_undegraded":"2024-04-06T13:48:21.097491+0000","last_fullsized":"2024-04-06T13:48:21.097491+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:04:10.547739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.684628+0000","last_change":"2024-04-06T13:48:19.272078+0000","last_active":"2024-04-06T13:48:19.684628+0000","last_peered":"2024-04-06T13:48:19.684628+0000","last_clean":"2024-04-06T13:48:19.684628+0000","last_became_active":"2024-04-06T13:48:19.271672+0000","last_became_peered":"2024-04-06T13:48:19.271672+0000","last_unstale":"2024-04-06T13:48:19.684628+0000","last_undegraded":"2024-04-06T13:48:19.684628+0000","last_fullsized":"2024-04-06T13:48:19.684628+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:23.871607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.098451+0000","last_change":"2024-04-06T13:48:19.702071+0000","last_active":"2024-04-06T13:48:21.098451+0000","last_peered":"2024-04-06T13:48:21.098451+0000","last_clean":"2024-04-06T13:48:21.098451+0000","last_became_active":"2024-04-06T13:48:19.698755+0000","last_became_peered":"2024-04-06T13:48:19.698755+0000","last_unstale":"2024-04-06T13:48:21.098451+0000","last_undegraded":"2024-04-06T13:48:21.098451+0000","last_fullsized":"2024-04-06T13:48:21.098451+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:06:58.060998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.685267+0000","last_change":"2024-04-06T13:48:19.278939+0000","last_active":"2024-04-06T13:48:19.685267+0000","last_peered":"2024-04-06T13:48:19.685267+0000","last_clean":"2024-04-06T13:48:19.685267+0000","last_became_active":"2024-04-06T13:48:19.278677+0000","last_became_peered":"2024-04-06T13:48:19.278677+0000","last_unstale":"2024-04-06T13:48:19.685267+0000","last_undegraded":"2024-04-06T13:48:19.685267+0000","last_fullsized":"2024-04-06T13:48:19.685267+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:15.157954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:21.097917+0000","last_change":"2024-04-06T13:48:19.693016+0000","last_active":"2024-04-06T13:48:21.097917+0000","last_peered":"2024-04-06T13:48:21.097917+0000","last_clean":"2024-04-06T13:48:21.097917+0000","last_became_active":"2024-04-06T13:48:19.692760+0000","last_became_peered":"2024-04-06T13:48:19.692760+0000","last_unstale":"2024-04-06T13:48:21.097917+0000","last_undegraded":"2024-04-06T13:48:21.097917+0000","last_fullsized":"2024-04-06T13:48:21.097917+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:41:37.714397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.682132+0000","last_change":"2024-04-06T13:48:19.282189+0000","last_active":"2024-04-06T13:48:19.682132+0000","last_peered":"2024-04-06T13:48:19.682132+0000","last_clean":"2024-04-06T13:48:19.682132+0000","last_became_active":"2024-04-06T13:48:19.281907+0000","last_became_peered":"2024-04-06T13:48:19.281907+0000","last_unstale":"2024-04-06T13:48:19.682132+0000","last_undegraded":"2024-04-06T13:48:19.682132+0000","last_fullsized":"2024-04-06T13:48:19.682132+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:34.453253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.684495+0000","last_change":"2024-04-06T13:48:19.689941+0000","last_active":"2024-04-06T13:48:20.684495+0000","last_peered":"2024-04-06T13:48:20.684495+0000","last_clean":"2024-04-06T13:48:20.684495+0000","last_became_active":"2024-04-06T13:48:19.689689+0000","last_became_peered":"2024-04-06T13:48:19.689689+0000","last_unstale":"2024-04-06T13:48:20.684495+0000","last_undegraded":"2024-04-06T13:48:20.684495+0000","last_fullsized":"2024-04-06T13:48:20.684495+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:55:19.563924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.692252+0000","last_change":"2024-04-06T13:48:19.283271+0000","last_active":"2024-04-06T13:48:19.692252+0000","last_peered":"2024-04-06T13:48:19.692252+0000","last_clean":"2024-04-06T13:48:19.692252+0000","last_became_active":"2024-04-06T13:48:19.282980+0000","last_became_peered":"2024-04-06T13:48:19.282980+0000","last_unstale":"2024-04-06T13:48:19.692252+0000","last_undegraded":"2024-04-06T13:48:19.692252+0000","last_fullsized":"2024-04-06T13:48:19.692252+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:51:53.597002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.899957+0000","last_change":"2024-04-06T13:48:19.704846+0000","last_active":"2024-04-06T13:48:20.899957+0000","last_peered":"2024-04-06T13:48:20.899957+0000","last_clean":"2024-04-06T13:48:20.899957+0000","last_became_active":"2024-04-06T13:48:19.704420+0000","last_became_peered":"2024-04-06T13:48:19.704420+0000","last_unstale":"2024-04-06T13:48:20.899957+0000","last_undegraded":"2024-04-06T13:48:20.899957+0000","last_fullsized":"2024-04-06T13:48:20.899957+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:36.178732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.684819+0000","last_change":"2024-04-06T13:48:19.277530+0000","last_active":"2024-04-06T13:48:19.684819+0000","last_peered":"2024-04-06T13:48:19.684819+0000","last_clean":"2024-04-06T13:48:19.684819+0000","last_became_active":"2024-04-06T13:48:19.276966+0000","last_became_peered":"2024-04-06T13:48:19.276966+0000","last_unstale":"2024-04-06T13:48:19.684819+0000","last_undegraded":"2024-04-06T13:48:19.684819+0000","last_fullsized":"2024-04-06T13:48:19.684819+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:42:14.962783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.683820+0000","last_change":"2024-04-06T13:48:19.688861+0000","last_active":"2024-04-06T13:48:20.683820+0000","last_peered":"2024-04-06T13:48:20.683820+0000","last_clean":"2024-04-06T13:48:20.683820+0000","last_became_active":"2024-04-06T13:48:19.688496+0000","last_became_peered":"2024-04-06T13:48:19.688496+0000","last_unstale":"2024-04-06T13:48:20.683820+0000","last_undegraded":"2024-04-06T13:48:20.683820+0000","last_fullsized":"2024-04-06T13:48:20.683820+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:36:47.917157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.680854+0000","last_change":"2024-04-06T13:48:19.276467+0000","last_active":"2024-04-06T13:48:19.680854+0000","last_peered":"2024-04-06T13:48:19.680854+0000","last_clean":"2024-04-06T13:48:19.680854+0000","last_became_active":"2024-04-06T13:48:19.275988+0000","last_became_peered":"2024-04-06T13:48:19.275988+0000","last_unstale":"2024-04-06T13:48:19.680854+0000","last_undegraded":"2024-04-06T13:48:19.680854+0000","last_fullsized":"2024-04-06T13:48:19.680854+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:44.664354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-06T13:48:20.684150+0000","last_change":"2024-04-06T13:48:19.700894+0000","last_active":"2024-04-06T13:48:20.684150+0000","last_peered":"2024-04-06T13:48:20.684150+0000","last_clean":"2024-04-06T13:48:20.684150+0000","last_became_active":"2024-04-06T13:48:19.700666+0000","last_became_peered":"2024-04-06T13:48:19.700666+0000","last_unstale":"2024-04-06T13:48:20.684150+0000","last_undegraded":"2024-04-06T13:48:20.684150+0000","last_fullsized":"2024-04-06T13:48:20.684150+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:41:21.910614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.694613+0000","last_change":"2024-04-06T13:48:19.277300+0000","last_active":"2024-04-06T13:48:19.694613+0000","last_peered":"2024-04-06T13:48:19.694613+0000","last_clean":"2024-04-06T13:48:19.694613+0000","last_became_active":"2024-04-06T13:48:19.276795+0000","last_became_peered":"2024-04-06T13:48:19.276795+0000","last_unstale":"2024-04-06T13:48:19.694613+0000","last_undegraded":"2024-04-06T13:48:19.694613+0000","last_fullsized":"2024-04-06T13:48:19.694613+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:54:25.950801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"233'370","reported_seq":1336,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-06T13:48:19.680055+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:48:19.680055+0000","last_peered":"2024-04-06T13:48:19.680055+0000","last_clean":"2024-04-06T13:48:19.680055+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:48:19.680055+0000","last_undegraded":"2024-04-06T13:48:19.680055+0000","last_fullsized":"2024-04-06T13:48:19.680055+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575993,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56912,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314928,"statfs":{"total":96636764160,"available":96578486272,"internally_reserved":0,"allocated":3727360,"data_stored":2195035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57899999999999996,"15min":0.59499999999999997},"min":{"1min":0.35499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.72199999999999998,"5min":0.78400000000000003,"15min":0.82499999999999996},"last":0.379},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56799999999999995,"15min":0.58999999999999997},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78500000000000003,"5min":0.80800000000000005,"15min":0.82899999999999996},"last":0.39100000000000001}]},{"osd":1,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.55000000000000004,"15min":0.57899999999999996},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77200000000000002,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.57699999999999996,"15min":0.59599999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.69899999999999995,"5min":0.83399999999999996,"15min":0.85299999999999998},"last":0.46000000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.56299999999999994,"15min":0.58499999999999996},"min":{"1min":0.33200000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69399999999999995,"5min":0.81699999999999995,"15min":0.83899999999999997},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.58599999999999997,"15min":0.60099999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69999999999999996,"5min":0.755,"15min":0.79700000000000004},"last":0.53200000000000003}]},{"osd":3,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.59399999999999997,"15min":0.60599999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.67200000000000004,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58799999999999997,"15min":0.60299999999999998},"min":{"1min":0.39000000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78900000000000003,"5min":0.82099999999999995,"15min":0.84099999999999997},"last":0.49299999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45800000000000002,"15min":0.46600000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.81599999999999995},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.36799999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.82699999999999996},"last":0.52300000000000002}]},{"osd":5,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.498,"15min":0.498},"min":{"1min":0.315,"5min":0.315,"15min":0.308},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.443},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56899999999999995,"15min":0.57799999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33600000000000002},"max":{"1min":0.74099999999999999,"5min":0.80700000000000005,"15min":0.85399999999999998},"last":0.54400000000000004}]},{"osd":6,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.34899999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.879},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57099999999999995,"15min":0.58699999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81799999999999995,"5min":0.84899999999999998,"15min":0.89800000000000002},"last":0.55900000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575992,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56328,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315512,"statfs":{"total":96636764160,"available":96579084288,"internally_reserved":0,"allocated":3129344,"data_stored":1603636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.53600000000000003,"15min":0.55700000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69399999999999995,"5min":0.79700000000000004,"15min":0.83799999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.54100000000000004,"15min":0.56499999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.62,"5min":0.76900000000000002,"15min":0.81999999999999995},"last":0.55700000000000005}]},{"osd":1,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.60399999999999998,"15min":0.624},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76000000000000001,"5min":0.81699999999999995,"15min":0.85799999999999998},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.622,"15min":0.64400000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.378},"max":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.82699999999999996},"last":0.54600000000000004}]},{"osd":2,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56999999999999995,"15min":0.59399999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72099999999999997,"5min":0.79000000000000004,"15min":0.82899999999999996},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.56299999999999994,"15min":0.58599999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.80700000000000005},"last":0.59099999999999997}]},{"osd":3,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.58599999999999997,"15min":0.60199999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.73199999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.437},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.61899999999999999,"15min":0.64300000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.36099999999999999},"max":{"1min":0.80200000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.63300000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70599999999999996,"5min":0.72299999999999998,"15min":0.81399999999999995},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.51100000000000001,"15min":0.53800000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.66400000000000003,"5min":0.82099999999999995,"15min":0.84099999999999997},"last":0.51500000000000001}]},{"osd":5,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.48899999999999999,"15min":0.50600000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.33300000000000002},"max":{"1min":0.63,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52000000000000002,"15min":0.53100000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67400000000000004,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.496}]},{"osd":7,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53900000000000003,"15min":0.54800000000000004},"min":{"1min":0.34899999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.67700000000000005,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55300000000000005,"15min":0.56899999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.32800000000000001},"max":{"1min":0.71199999999999997,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.623}]}]},{"osd":1,"up_from":13,"seq":55834575991,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60404,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311436,"statfs":{"total":96636764160,"available":96574910464,"internally_reserved":0,"allocated":3121152,"data_stored":1603524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.38100000000000001,"15min":0.39100000000000001},"min":{"1min":0.248,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.378,"5min":0.621,"15min":0.621},"last":0.441},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.41799999999999998,"15min":0.42699999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.501,"5min":0.60799999999999998,"15min":0.69399999999999995},"last":0.27500000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.41299999999999998,"15min":0.42399999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.46899999999999997,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.44900000000000001,"15min":0.46400000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.55400000000000005,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.34799999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.46999999999999997,"15min":0.47999999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.52800000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.42699999999999999,"15min":0.439},"min":{"1min":0.28799999999999998,"5min":0.27600000000000002,"15min":0.26300000000000001},"max":{"1min":0.45500000000000002,"5min":0.63500000000000001,"15min":0.66100000000000003},"last":0.33100000000000002}]},{"osd":4,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.628,"15min":0.64400000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.33800000000000002},"max":{"1min":0.70099999999999996,"5min":0.89900000000000002,"15min":0.97399999999999998},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.64400000000000002,"15min":0.65500000000000003},"min":{"1min":0.46100000000000002,"5min":0.44500000000000001,"15min":0.44400000000000001},"max":{"1min":0.73499999999999999,"5min":0.94599999999999995,"15min":0.995},"last":0.55400000000000005}]},{"osd":5,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.64100000000000001,"15min":0.65300000000000002},"min":{"1min":0.42999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73899999999999999,"5min":0.96899999999999997,"15min":1.073},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.65700000000000003,"15min":0.66800000000000004},"min":{"1min":0.45200000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.72799999999999998,"5min":0.96499999999999997,"15min":1.0209999999999999},"last":0.47899999999999998}]},{"osd":6,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.46300000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73899999999999999,"5min":0.89200000000000002,"15min":1},"last":0.503},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.67100000000000004,"15min":0.68400000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.72499999999999998,"5min":0.94599999999999995,"15min":1.036},"last":0.48799999999999999}]},{"osd":7,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.69399999999999995,"15min":0.69499999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85399999999999998,"5min":0.94899999999999995,"15min":1.0649999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65700000000000003,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.76300000000000001,"5min":0.92900000000000005,"15min":1.056},"last":0.46899999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575991,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61004,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310836,"statfs":{"total":96636764160,"available":96574296064,"internally_reserved":0,"allocated":3727360,"data_stored":2195456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.38200000000000001,"15min":0.39300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.45100000000000001,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.39300000000000002,"15min":0.41699999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.48199999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.41399999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.40100000000000002,"15min":0.42499999999999999},"min":{"1min":0.26100000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49399999999999999,"5min":0.66700000000000004,"15min":0.70099999999999996},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.42499999999999999,"15min":0.443},"min":{"1min":0.28399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.47999999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.33300000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.40699999999999997,"15min":0.42499999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.50700000000000001,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.45700000000000002,"15min":0.47799999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.51900000000000002,"5min":0.72999999999999998,"15min":0.75700000000000001},"last":0.442}]},{"osd":4,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.60499999999999998,"15min":0.64100000000000001},"min":{"1min":0.39100000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.80900000000000005,"5min":0.93100000000000005,"15min":1.0329999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.61199999999999999,"15min":0.64300000000000002},"min":{"1min":0.372,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88500000000000001,"5min":0.999,"15min":0.999},"last":0.57899999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.60199999999999998,"15min":0.63900000000000001},"min":{"1min":0.39300000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.0009999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.63800000000000001,"15min":0.66300000000000003},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.749,"5min":0.92800000000000005,"15min":1.052},"last":0.501}]},{"osd":6,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.64500000000000002,"15min":0.67400000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.874,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.64200000000000002,"15min":0.66900000000000004},"min":{"1min":0.40500000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77500000000000002,"5min":0.96399999999999997,"15min":1.085},"last":0.67600000000000005}]},{"osd":7,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.66300000000000003,"15min":0.69099999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.80700000000000005,"5min":1.01,"15min":1.121},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.64700000000000002,"15min":0.67200000000000004},"min":{"1min":0.42199999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.93700000000000006,"5min":0.95799999999999996,"15min":1.0229999999999999},"last":0.69399999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575993,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60420,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311420,"statfs":{"total":96636764160,"available":96574894080,"internally_reserved":0,"allocated":3129344,"data_stored":1604176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.376,"15min":0.38400000000000001},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.42299999999999999,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.40600000000000003,"15min":0.42199999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.249},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":1.044},"last":0.27900000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.38500000000000001,"15min":0.40400000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51100000000000001,"5min":0.54000000000000004,"15min":0.753},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.42799999999999999,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.82199999999999995},"last":0.35999999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.439,"15min":0.45800000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.53200000000000003,"5min":0.69699999999999995,"15min":0.70299999999999996},"last":0.374},{"interface":"front","average":{"1min":0.377,"5min":0.437,"15min":0.44900000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.504,"5min":0.64500000000000002,"15min":0.68600000000000005},"last":0.41499999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.621,"15min":0.63900000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.376},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.64400000000000002,"15min":0.66100000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.88900000000000001,"5min":1.022,"15min":1.022},"last":0.629}]},{"osd":5,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.64300000000000002,"15min":0.66300000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88100000000000001,"5min":0.98599999999999999,"15min":1.0149999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.65100000000000002,"15min":0.66700000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.65200000000000002,"15min":0.67200000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92000000000000004,"5min":0.97199999999999998,"15min":1.091},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.67600000000000005,"15min":0.69499999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.96099999999999997},"last":0.51200000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.66600000000000004,"15min":0.68799999999999994},"min":{"1min":0.42699999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.95399999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.68300000000000005,"15min":0.70799999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.85299999999999998,"5min":1.01,"15min":1.01},"last":0.53600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575992,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48156,"kb_used_data":3076,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323684,"statfs":{"total":96636764160,"available":96587452416,"internally_reserved":0,"allocated":3149824,"data_stored":1606745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.37,"15min":0.38400000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.47399999999999998,"5min":0.55200000000000005,"15min":0.72799999999999998},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.39600000000000002,"15min":0.40799999999999997},"min":{"1min":0.25700000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.48799999999999999,"5min":0.58599999999999997,"15min":0.66700000000000004},"last":0.34300000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.39300000000000002,"15min":0.40100000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.46899999999999997,"5min":0.59499999999999997,"15min":0.65700000000000003},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.41699999999999998,"15min":0.437},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.502,"5min":0.63,"15min":0.75800000000000001},"last":0.38600000000000001}]},{"osd":2,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.46100000000000002,"15min":0.47999999999999998},"min":{"1min":0.32700000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.53900000000000003,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.376},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.47699999999999998,"15min":0.496},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.51900000000000002,"5min":0.67900000000000005,"15min":0.79300000000000004},"last":0.42499999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.622,"15min":0.63100000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76700000000000002,"5min":0.94099999999999995,"15min":1.018},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.628,"15min":0.64200000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.81100000000000005,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.80400000000000005}]},{"osd":5,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.66100000000000003,"15min":0.67600000000000005},"min":{"1min":0.44700000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.872,"5min":0.95699999999999996,"15min":0.999},"last":0.747},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.67900000000000005,"15min":0.68700000000000006},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.86299999999999999,"5min":1.036,"15min":1.036},"last":0.81499999999999995}]},{"osd":6,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.64100000000000001,"15min":0.66100000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.76300000000000001,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.66000000000000003,"15min":0.67600000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41099999999999998},"max":{"1min":0.82799999999999996,"5min":0.91600000000000004,"15min":0.98799999999999999},"last":0.78800000000000003}]},{"osd":7,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.65500000000000003,"15min":0.66800000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80200000000000005,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.67800000000000005,"15min":0.69499999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85199999999999998,"5min":1.0209999999999999,"15min":1.024},"last":0.77800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575992,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48132,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323708,"statfs":{"total":96636764160,"available":96587476992,"internally_reserved":0,"allocated":3129344,"data_stored":1604613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55900000000000005,"15min":0.58699999999999997},"min":{"1min":0.40699999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.753,"5min":0.876,"15min":0.88500000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.39300000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81100000000000005,"5min":1.202,"15min":1.202},"last":0.48299999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.61299999999999999,"15min":0.64000000000000001},"min":{"1min":0.40799999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71699999999999997,"5min":1.256,"15min":1.256},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.624,"15min":0.63},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.75900000000000001,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.56499999999999995}]},{"osd":2,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.57999999999999996,"15min":0.59999999999999998},"min":{"1min":0.41599999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77500000000000002,"5min":0.84599999999999997,"15min":0.877},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.60599999999999998,"15min":0.61599999999999999},"min":{"1min":0.39600000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.97899999999999998,"5min":1.294,"15min":1.294},"last":0.41199999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84499999999999997,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.60899999999999999,"15min":0.626},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.78800000000000003,"5min":0.90200000000000002,"15min":0.91100000000000003},"last":0.55400000000000005}]},{"osd":5,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55500000000000005,"15min":0.52000000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.92800000000000005,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.56399999999999995,"15min":0.54100000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.95199999999999996,"5min":4.931,"15min":4.931},"last":0.63500000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53500000000000003,"15min":0.53300000000000003},"min":{"1min":0.34999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.505},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34699999999999998},"max":{"1min":1.0609999999999999,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.62}]},{"osd":7,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59499999999999997,"15min":0.58199999999999996},"min":{"1min":0.36899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.91000000000000003,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59399999999999997,"15min":0.60199999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.373},"max":{"1min":0.86899999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.53200000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575992,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64508,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307332,"statfs":{"total":96636764160,"available":96570707968,"internally_reserved":0,"allocated":3121152,"data_stored":1603524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.57399999999999995,"15min":0.58499999999999996},"min":{"1min":0.36399999999999999,"5min":0.35899999999999999,"15min":0.33000000000000002},"max":{"1min":0.68000000000000005,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58099999999999996,"15min":0.59399999999999997},"min":{"1min":0.38100000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.76600000000000001,"5min":0.84699999999999998,"15min":0.89200000000000002},"last":0.63200000000000001}]},{"osd":1,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.58099999999999996,"15min":0.59399999999999997},"min":{"1min":0.38400000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79100000000000004,"5min":0.81299999999999994,"15min":0.81399999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.439,"5min":0.39400000000000002,"15min":0.37},"max":{"1min":0.70299999999999996,"5min":0.871,"15min":0.871},"last":0.59199999999999997}]},{"osd":2,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59299999999999997,"15min":0.59699999999999998},"min":{"1min":0.311,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69199999999999995,"5min":0.79700000000000004,"15min":0.84399999999999997},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58799999999999997,"15min":0.59599999999999997},"min":{"1min":0.44,"5min":0.378,"15min":0.378},"max":{"1min":0.73099999999999998,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.51700000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.79800000000000004,"5min":0.81100000000000005,"15min":0.94499999999999995},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61699999999999999,"15min":0.63200000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.74099999999999999,"5min":0.88200000000000001,"15min":0.91500000000000004},"last":0.61499999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52900000000000003,"15min":0.50700000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.27400000000000002},"max":{"1min":0.70999999999999996,"5min":0.85699999999999998,"15min":1.0369999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.59999999999999998,"15min":0.57699999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.308},"max":{"1min":0.77800000000000002,"5min":0.88800000000000001,"15min":1.1639999999999999},"last":0.70299999999999996}]},{"osd":6,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54300000000000004,"15min":0.52600000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.27900000000000003},"max":{"1min":0.89100000000000001,"5min":0.94999999999999996,"15min":1.0169999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54300000000000004,"15min":0.53100000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.32300000000000001},"max":{"1min":0.81899999999999995,"5min":0.878,"15min":1.0009999999999999},"last":0.68799999999999994}]},{"osd":7,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56499999999999995,"15min":0.54800000000000004},"min":{"1min":0.39200000000000002,"5min":0.35799999999999998,"15min":0.32900000000000001},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.66400000000000003,"15min":0.64800000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.35199999999999998},"max":{"1min":0.85299999999999998,"5min":0.92300000000000004,"15min":0.96599999999999997},"last":0.73199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:48:28.757 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-06T13:48:28.757 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-06T13:48:28.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.920+0000 7fd77b1d1700 1 Processor -- start 2024-04-06T13:48:28.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.920+0000 7fd77b1d1700 1 -- start start 2024-04-06T13:48:28.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.921+0000 7fd77b1d1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7740cdc60 0x7fd7740ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:28.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.921+0000 7fd77b1d1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd77406c080 con 0x7fd77406e470 2024-04-06T13:48:28.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.921+0000 7fd77b1d1700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd77406a8d0 con 0x7fd774159b20 2024-04-06T13:48:28.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.921+0000 7fd77b1d1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7740ce620 con 0x7fd7740cdc60 2024-04-06T13:48:28.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.921+0000 7fd77976e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd77406e470 0x7fd77406e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33254/0 (socket says 172.21.15.189:33254) 2024-04-06T13:48:28.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.921+0000 7fd77976e700 1 -- 172.21.15.189:0/1267174024 learned_addr learned my addr 172.21.15.189:0/1267174024 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:28.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.921+0000 7fd773fff700 1 --2- 172.21.15.189:0/1267174024 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7740cdc60 0x7fd7740ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:28.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.921+0000 7fd7737fe700 1 -- 172.21.15.189:0/1267174024 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965409037 0 0) 0x7fd77406c080 con 0x7fd77406e470 2024-04-06T13:48:28.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd7737fe700 1 -- 172.21.15.189:0/1267174024 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7640035b0 con 0x7fd77406e470 2024-04-06T13:48:28.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd7737fe700 1 -- 172.21.15.189:0/1267174024 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1819818083 0 0) 0x7fd7640035b0 con 0x7fd77406e470 2024-04-06T13:48:28.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd7737fe700 1 -- 172.21.15.189:0/1267174024 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7740cdc60 msgr2=0x7fd7740ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:28.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd7737fe700 1 --2- 172.21.15.189:0/1267174024 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7740cdc60 0x7fd7740ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:28.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd7737fe700 1 -- 172.21.15.189:0/1267174024 >> v1:172.21.15.192:6789/0 conn(0x7fd774159b20 legacy=0x7fd77406b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:28.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd7737fe700 1 -- 172.21.15.189:0/1267174024 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77406abb0 con 0x7fd77406e470 2024-04-06T13:48:28.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd7737fe700 1 -- 172.21.15.189:0/1267174024 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fd76c0019f0 con 0x7fd77406e470 2024-04-06T13:48:28.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd7737fe700 1 -- 172.21.15.189:0/1267174024 >> v1:172.21.15.189:6789/0 conn(0x7fd77406e470 legacy=0x7fd77406e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:28.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd7737fe700 1 --2- 172.21.15.189:0/1267174024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd764003a30 0x7fd764003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:28.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd7737fe700 1 --2- 172.21.15.189:0/1267174024 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7740cdc60 0x7fd764004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:28.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd7737fe700 1 --2- 172.21.15.189:0/1267174024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd764004d90 0x7fd764005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:28.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd7737fe700 1 -- 172.21.15.189:0/1267174024 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7740ce620 con 0x7fd764003a30 2024-04-06T13:48:28.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd7737fe700 1 -- 172.21.15.189:0/1267174024 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76400bb70 con 0x7fd764004d90 2024-04-06T13:48:28.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd7737fe700 1 -- 172.21.15.189:0/1267174024 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd76400bd30 con 0x7fd7740cdc60 2024-04-06T13:48:28.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd773fff700 1 --2- 172.21.15.189:0/1267174024 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7740cdc60 0x7fd764004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:28.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd778f6d700 1 --2- 172.21.15.189:0/1267174024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd764003a30 0x7fd764003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:28.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd77976e700 1 --2- 172.21.15.189:0/1267174024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd764004d90 0x7fd764005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:28.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd778f6d700 1 -- 172.21.15.189:0/1267174024 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7740cdc60 msgr2=0x7fd764004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:28.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd778f6d700 1 --2- 172.21.15.189:0/1267174024 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7740cdc60 0x7fd764004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:28.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd778f6d700 1 -- 172.21.15.189:0/1267174024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd764004d90 msgr2=0x7fd764005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:28.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd778f6d700 1 --2- 172.21.15.189:0/1267174024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd764004d90 0x7fd764005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:28.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd778f6d700 1 -- 172.21.15.189:0/1267174024 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7640035b0 con 0x7fd764003a30 2024-04-06T13:48:28.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.922+0000 7fd778f6d700 1 --2- 172.21.15.189:0/1267174024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd764003a30 0x7fd764003e50 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7fd768001800 tx=0x7fd764005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d869c0a6ac3e111 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:28.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.923+0000 7fd7737fe700 1 -- 172.21.15.189:0/1267174024 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7680012f0 con 0x7fd764003a30 2024-04-06T13:48:28.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.923+0000 7fd7737fe700 1 -- 172.21.15.189:0/1267174024 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7680052f0 con 0x7fd764003a30 2024-04-06T13:48:28.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.923+0000 7fd7737fe700 1 -- 172.21.15.189:0/1267174024 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd768006860 con 0x7fd764003a30 2024-04-06T13:48:28.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.923+0000 7fd77b1d1700 1 -- 172.21.15.189:0/1267174024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd764003a30 msgr2=0x7fd764003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:28.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.923+0000 7fd77b1d1700 1 --2- 172.21.15.189:0/1267174024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd764003a30 0x7fd764003e50 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7fd768001800 tx=0x7fd764005880 comp rx=0 tx=0).stop 2024-04-06T13:48:28.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.923+0000 7fd77976e700 1 -- 172.21.15.189:0/1267174024 reap_dead start 2024-04-06T13:48:28.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.923+0000 7fd77b1d1700 1 -- 172.21.15.189:0/1267174024 shutdown_connections 2024-04-06T13:48:28.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.923+0000 7fd77b1d1700 1 -- 172.21.15.189:0/1267174024 >> 172.21.15.189:0/1267174024 conn(0x7fd7740cc290 msgr2=0x7fd7740cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:28.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.923+0000 7fd77b1d1700 1 -- 172.21.15.189:0/1267174024 shutdown_connections 2024-04-06T13:48:28.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.923+0000 7fd77b1d1700 1 -- 172.21.15.189:0/1267174024 wait complete. 2024-04-06T13:48:28.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.925+0000 7fd77b1d1700 1 Processor -- start 2024-04-06T13:48:28.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.925+0000 7fd77b1d1700 1 -- start start 2024-04-06T13:48:28.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.925+0000 7fd77b1d1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd764003a30 0x7fd7740c3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:28.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.925+0000 7fd77b1d1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77414ac10 0x7fd7741475b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:28.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.925+0000 7fd77b1d1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd774147f00 0x7fd7741483b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:28.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.925+0000 7fd77b1d1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7740c66e0 con 0x7fd764003a30 2024-04-06T13:48:28.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.925+0000 7fd77b1d1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7740c6120 con 0x7fd77414ac10 2024-04-06T13:48:28.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.925+0000 7fd77b1d1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7740c6400 con 0x7fd774147f00 2024-04-06T13:48:28.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.926+0000 7fd778f6d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd764003a30 0x7fd7740c3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:28.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.926+0000 7fd773fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77414ac10 0x7fd7741475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:28.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.926+0000 7fd778f6d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd764003a30 0x7fd7740c3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45156/0 (socket says 172.21.15.189:45156) 2024-04-06T13:48:28.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.926+0000 7fd778f6d700 1 -- 172.21.15.189:0/1911987155 learned_addr learned my addr 172.21.15.189:0/1911987155 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:28.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.926+0000 7fd77976e700 1 --2- 172.21.15.189:0/1911987155 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd774147f00 0x7fd7741483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:28.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.926+0000 7fd778f6d700 1 -- 172.21.15.189:0/1911987155 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd774147f00 msgr2=0x7fd7741483b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:28.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.926+0000 7fd778f6d700 1 --2- 172.21.15.189:0/1911987155 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd774147f00 0x7fd7741483b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:28.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.926+0000 7fd778f6d700 1 -- 172.21.15.189:0/1911987155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77414ac10 msgr2=0x7fd7741475b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:28.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.926+0000 7fd778f6d700 1 --2- 172.21.15.189:0/1911987155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77414ac10 0x7fd7741475b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:28.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.926+0000 7fd778f6d700 1 -- 172.21.15.189:0/1911987155 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77415bc10 con 0x7fd764003a30 2024-04-06T13:48:28.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.926+0000 7fd773fff700 1 --2- 172.21.15.189:0/1911987155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77414ac10 0x7fd7741475b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:28.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.926+0000 7fd778f6d700 1 --2- 172.21.15.189:0/1911987155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd764003a30 0x7fd7740c3150 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7fd768004000 tx=0x7fd768001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:28.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.926+0000 7fd7717fa700 1 -- 172.21.15.189:0/1911987155 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd768006df0 con 0x7fd764003a30 2024-04-06T13:48:28.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.926+0000 7fd7717fa700 1 -- 172.21.15.189:0/1911987155 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd768007020 con 0x7fd764003a30 2024-04-06T13:48:28.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.926+0000 7fd77b1d1700 1 -- 172.21.15.189:0/1911987155 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7740c3690 con 0x7fd764003a30 2024-04-06T13:48:28.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.926+0000 7fd7717fa700 1 -- 172.21.15.189:0/1911987155 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7680073d0 con 0x7fd764003a30 2024-04-06T13:48:28.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.926+0000 7fd77b1d1700 1 -- 172.21.15.189:0/1911987155 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd77415c130 con 0x7fd764003a30 2024-04-06T13:48:28.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.927+0000 7fd77b1d1700 1 -- 172.21.15.189:0/1911987155 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd77415bf80 con 0x7fd764003a30 2024-04-06T13:48:28.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.928+0000 7fd7717fa700 1 -- 172.21.15.189:0/1911987155 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fd768007530 con 0x7fd764003a30 2024-04-06T13:48:28.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.928+0000 7fd7717fa700 1 --2- 172.21.15.189:0/1911987155 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd75806aed0 0x7fd75806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:28.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.928+0000 7fd773fff700 1 --2- 172.21.15.189:0/1911987155 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd75806aed0 0x7fd75806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:28.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.928+0000 7fd7717fa700 1 -- 172.21.15.189:0/1911987155 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd76808aa30 con 0x7fd764003a30 2024-04-06T13:48:28.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.929+0000 7fd773fff700 1 --2- 172.21.15.189:0/1911987155 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd75806aed0 0x7fd75806d390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd77414b090 tx=0x7fd760009210 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:28.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:28.933+0000 7fd7717fa700 1 -- 172.21.15.189:0/1911987155 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd768055880 con 0x7fd764003a30 2024-04-06T13:48:29.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.208+0000 7fd77b1d1700 1 -- 172.21.15.189:0/1911987155 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd77415ba70 con 0x7fd764003a30 2024-04-06T13:48:29.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.211+0000 7fd7717fa700 1 -- 172.21.15.189:0/1911987155 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v53) v1 ==== 190+0+0 (secure 0 0 0) 0x7fd77415ba70 con 0x7fd764003a30 2024-04-06T13:48:29.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.213+0000 7fd77b1d1700 1 -- 172.21.15.189:0/1911987155 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd75806aed0 msgr2=0x7fd75806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:29.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.213+0000 7fd77b1d1700 1 --2- 172.21.15.189:0/1911987155 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd75806aed0 0x7fd75806d390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd77414b090 tx=0x7fd760009210 comp rx=0 tx=0).stop 2024-04-06T13:48:29.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.213+0000 7fd77b1d1700 1 -- 172.21.15.189:0/1911987155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd764003a30 msgr2=0x7fd7740c3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:29.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.213+0000 7fd77b1d1700 1 --2- 172.21.15.189:0/1911987155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd764003a30 0x7fd7740c3150 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7fd768004000 tx=0x7fd768001120 comp rx=0 tx=0).stop 2024-04-06T13:48:29.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.214+0000 7fd77b1d1700 1 -- 172.21.15.189:0/1911987155 shutdown_connections 2024-04-06T13:48:29.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.214+0000 7fd77b1d1700 1 --2- 172.21.15.189:0/1911987155 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd75806aed0 0x7fd75806d390 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:29.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.214+0000 7fd77b1d1700 1 --2- 172.21.15.189:0/1911987155 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd774147f00 0x7fd7741483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:29.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.214+0000 7fd77b1d1700 1 --2- 172.21.15.189:0/1911987155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd764003a30 0x7fd7740c3150 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:29.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.214+0000 7fd77b1d1700 1 --2- 172.21.15.189:0/1911987155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77414ac10 0x7fd7741475b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:29.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.214+0000 7fd77b1d1700 1 -- 172.21.15.189:0/1911987155 >> 172.21.15.189:0/1911987155 conn(0x7fd7740cc290 msgr2=0x7fd774066550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:29.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.214+0000 7fd77b1d1700 1 -- 172.21.15.189:0/1911987155 shutdown_connections 2024-04-06T13:48:29.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.214+0000 7fd77b1d1700 1 -- 172.21.15.189:0/1911987155 wait complete. 2024-04-06T13:48:29.216 INFO:teuthology.orchestra.run.smithi189.stderr:updated caps for client.0 2024-04-06T13:48:29.226 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:48:29.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.348+0000 7fa066ec8700 1 Processor -- start 2024-04-06T13:48:29.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.348+0000 7fa066ec8700 1 -- start start 2024-04-06T13:48:29.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.349+0000 7fa066ec8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa06015a550 0x7fa0600cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:29.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.349+0000 7fa066ec8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa0600b4d70 0x7fa0600b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:29.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.349+0000 7fa066ec8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0600cdbe0 con 0x7fa0600cf900 2024-04-06T13:48:29.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.349+0000 7fa066ec8700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa06015a940 con 0x7fa0600b4d70 2024-04-06T13:48:29.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.349+0000 7fa066ec8700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa06015ac00 con 0x7fa06015a550 2024-04-06T13:48:29.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.349+0000 7fa065465700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa0600b4d70 0x7fa0600b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:29.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.349+0000 7fa05ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa0600cf900 0x7fa0600b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55766/0 (socket says 172.21.15.189:55766) 2024-04-06T13:48:29.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.349+0000 7fa065465700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa0600b4d70 0x7fa0600b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49238/0 (socket says 172.21.15.189:49238) 2024-04-06T13:48:29.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.349+0000 7fa05ffff700 1 -- 172.21.15.189:0/1299717328 learned_addr learned my addr 172.21.15.189:0/1299717328 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:29.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.349+0000 7fa064c64700 1 --2- 172.21.15.189:0/1299717328 >> v2:172.21.15.192:3300/0 conn(0x7fa06015a550 0x7fa0600cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:29.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.349+0000 7fa065465700 1 -- 172.21.15.189:0/1299717328 >> v2:172.21.15.192:3300/0 conn(0x7fa06015a550 msgr2=0x7fa0600cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:29.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.349+0000 7fa065465700 1 --2- 172.21.15.189:0/1299717328 >> v2:172.21.15.192:3300/0 conn(0x7fa06015a550 0x7fa0600cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:29.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.349+0000 7fa065465700 1 -- 172.21.15.189:0/1299717328 >> v1:172.21.15.189:6789/0 conn(0x7fa0600cf900 legacy=0x7fa0600b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:29.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.349+0000 7fa065465700 1 -- 172.21.15.189:0/1299717328 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06015aee0 con 0x7fa0600b4d70 2024-04-06T13:48:29.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.349+0000 7fa05f7fe700 1 -- 172.21.15.189:0/1299717328 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889066612 0 0) 0x7fa0600cdbe0 con 0x7fa0600cf900 2024-04-06T13:48:29.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.350+0000 7fa065465700 1 --2- 172.21.15.189:0/1299717328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0600b4d70 0x7fa0600b2fc0 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7fa060073470 tx=0x7fa05800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3842bef06b010007 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:29.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.350+0000 7fa05f7fe700 1 -- 172.21.15.189:0/1299717328 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05800e070 con 0x7fa0600b4d70 2024-04-06T13:48:29.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.350+0000 7fa05f7fe700 1 -- 172.21.15.189:0/1299717328 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05800bb20 con 0x7fa0600b4d70 2024-04-06T13:48:29.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.350+0000 7fa05f7fe700 1 -- 172.21.15.189:0/1299717328 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa058012430 con 0x7fa0600b4d70 2024-04-06T13:48:29.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.350+0000 7fa066ec8700 1 -- 172.21.15.189:0/1299717328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0600b4d70 msgr2=0x7fa0600b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:29.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.350+0000 7fa066ec8700 1 --2- 172.21.15.189:0/1299717328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0600b4d70 0x7fa0600b2fc0 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7fa060073470 tx=0x7fa05800b330 comp rx=0 tx=0).stop 2024-04-06T13:48:29.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.350+0000 7fa066ec8700 1 -- 172.21.15.189:0/1299717328 shutdown_connections 2024-04-06T13:48:29.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.350+0000 7fa066ec8700 1 --2- 172.21.15.189:0/1299717328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0600b4d70 0x7fa0600b2fc0 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:29.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.350+0000 7fa066ec8700 1 --2- 172.21.15.189:0/1299717328 >> v2:172.21.15.192:3300/0 conn(0x7fa06015a550 0x7fa0600cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:29.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.350+0000 7fa066ec8700 1 -- 172.21.15.189:0/1299717328 >> 172.21.15.189:0/1299717328 conn(0x7fa0600d00b0 msgr2=0x7fa0600d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:29.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.350+0000 7fa066ec8700 1 -- 172.21.15.189:0/1299717328 shutdown_connections 2024-04-06T13:48:29.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.350+0000 7fa066ec8700 1 -- 172.21.15.189:0/1299717328 wait complete. 2024-04-06T13:48:29.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.351+0000 7fa066ec8700 1 Processor -- start 2024-04-06T13:48:29.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.352+0000 7fa066ec8700 1 -- start start 2024-04-06T13:48:29.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.352+0000 7fa066ec8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0600b4d70 0x7fa060151950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:29.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.352+0000 7fa066ec8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0600cf900 0x7fa0601f0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:29.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.352+0000 7fa066ec8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0600a8ec0 0x7fa0600a9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:29.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.352+0000 7fa066ec8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa060156f30 con 0x7fa0600b4d70 2024-04-06T13:48:29.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.352+0000 7fa066ec8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0600a9d10 con 0x7fa0600cf900 2024-04-06T13:48:29.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.352+0000 7fa066ec8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa060151e90 con 0x7fa0600a8ec0 2024-04-06T13:48:29.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.352+0000 7fa064c64700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0600b4d70 0x7fa060151950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:29.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.353+0000 7fa064c64700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0600b4d70 0x7fa060151950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49248/0 (socket says 172.21.15.189:49248) 2024-04-06T13:48:29.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.353+0000 7fa064c64700 1 -- 172.21.15.189:0/207598160 learned_addr learned my addr 172.21.15.189:0/207598160 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:29.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.353+0000 7fa05ffff700 1 --2- 172.21.15.189:0/207598160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0600cf900 0x7fa0601f0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:29.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.353+0000 7fa065465700 1 --2- 172.21.15.189:0/207598160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0600a8ec0 0x7fa0600a9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:29.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.353+0000 7fa064c64700 1 -- 172.21.15.189:0/207598160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0600a8ec0 msgr2=0x7fa0600a9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:29.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.353+0000 7fa064c64700 1 --2- 172.21.15.189:0/207598160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0600a8ec0 0x7fa0600a9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:29.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.353+0000 7fa064c64700 1 -- 172.21.15.189:0/207598160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0600cf900 msgr2=0x7fa0601f0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:29.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.353+0000 7fa064c64700 1 --2- 172.21.15.189:0/207598160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0600cf900 0x7fa0601f0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:29.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.353+0000 7fa064c64700 1 -- 172.21.15.189:0/207598160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa058009a70 con 0x7fa0600b4d70 2024-04-06T13:48:29.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.353+0000 7fa064c64700 1 --2- 172.21.15.189:0/207598160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0600b4d70 0x7fa060151950 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7fa05400e870 tx=0x7fa05400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:29.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.353+0000 7fa05d7fa700 1 -- 172.21.15.189:0/207598160 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa054009590 con 0x7fa0600b4d70 2024-04-06T13:48:29.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.353+0000 7fa05d7fa700 1 -- 172.21.15.189:0/207598160 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05400ee70 con 0x7fa0600b4d70 2024-04-06T13:48:29.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.353+0000 7fa066ec8700 1 -- 172.21.15.189:0/207598160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa060156620 con 0x7fa0600b4d70 2024-04-06T13:48:29.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.353+0000 7fa05d7fa700 1 -- 172.21.15.189:0/207598160 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05400c9f0 con 0x7fa0600b4d70 2024-04-06T13:48:29.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.353+0000 7fa066ec8700 1 -- 172.21.15.189:0/207598160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0600a9a30 con 0x7fa0600b4d70 2024-04-06T13:48:29.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.354+0000 7fa066ec8700 1 -- 172.21.15.189:0/207598160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0601571a0 con 0x7fa0600b4d70 2024-04-06T13:48:29.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.355+0000 7fa05d7fa700 1 -- 172.21.15.189:0/207598160 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fa05400cb50 con 0x7fa0600b4d70 2024-04-06T13:48:29.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.355+0000 7fa05d7fa700 1 --2- 172.21.15.189:0/207598160 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa04406aed0 0x7fa04406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:29.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.355+0000 7fa05d7fa700 1 -- 172.21.15.189:0/207598160 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa054089780 con 0x7fa0600b4d70 2024-04-06T13:48:29.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.355+0000 7fa05ffff700 1 --2- 172.21.15.189:0/207598160 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa04406aed0 0x7fa04406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:29.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.355+0000 7fa05ffff700 1 --2- 172.21.15.189:0/207598160 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa04406aed0 0x7fa04406d390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa0600cc630 tx=0x7fa04c007be0 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:29.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.359+0000 7fa05d7fa700 1 -- 172.21.15.189:0/207598160 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa054054500 con 0x7fa0600b4d70 2024-04-06T13:48:29.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.598+0000 7fa066ec8700 1 -- 172.21.15.189:0/207598160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa060067bb0 con 0x7fa0600b4d70 2024-04-06T13:48:29.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.599+0000 7fa05d7fa700 1 -- 172.21.15.189:0/207598160 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+4753 (secure 0 0 0) 0x7fa054057b20 con 0x7fa0600b4d70 2024-04-06T13:48:29.599 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:29.600 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":116,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3323503154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3323503154},{"type":"v1","addr":"172.21.15.189:6836","nonce":3323503154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":9383,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3292345173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3292345173},{"type":"v1","addr":"172.21.15.189:6837","nonce":3292345173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":9395,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/455737532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":455737532},{"type":"v1","addr":"172.21.15.192:6835","nonce":455737532}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:48:19.674205+0000","modified":"2024-04-06T13:48:22.682911+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12108},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12108":{"gid":12108,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6833/726816746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":726816746},{"type":"v1","addr":"172.21.15.192:6833","nonce":726816746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-04-06T13:48:29.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.601+0000 7fa066ec8700 1 -- 172.21.15.189:0/207598160 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa04406aed0 msgr2=0x7fa04406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:29.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.601+0000 7fa066ec8700 1 --2- 172.21.15.189:0/207598160 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa04406aed0 0x7fa04406d390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa0600cc630 tx=0x7fa04c007be0 comp rx=0 tx=0).stop 2024-04-06T13:48:29.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.601+0000 7fa066ec8700 1 -- 172.21.15.189:0/207598160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0600b4d70 msgr2=0x7fa060151950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:29.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.601+0000 7fa066ec8700 1 --2- 172.21.15.189:0/207598160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0600b4d70 0x7fa060151950 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7fa05400e870 tx=0x7fa05400ec30 comp rx=0 tx=0).stop 2024-04-06T13:48:29.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.601+0000 7fa066ec8700 1 -- 172.21.15.189:0/207598160 shutdown_connections 2024-04-06T13:48:29.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.601+0000 7fa066ec8700 1 --2- 172.21.15.189:0/207598160 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa04406aed0 0x7fa04406d390 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:29.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.601+0000 7fa066ec8700 1 --2- 172.21.15.189:0/207598160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0600a8ec0 0x7fa0600a9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:29.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.601+0000 7fa066ec8700 1 --2- 172.21.15.189:0/207598160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0600b4d70 0x7fa060151950 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:29.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.601+0000 7fa066ec8700 1 --2- 172.21.15.189:0/207598160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0600cf900 0x7fa0601f0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:29.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.601+0000 7fa066ec8700 1 -- 172.21.15.189:0/207598160 >> 172.21.15.189:0/207598160 conn(0x7fa0600d00b0 msgr2=0x7fa0600b44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:29.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.601+0000 7fa066ec8700 1 -- 172.21.15.189:0/207598160 shutdown_connections 2024-04-06T13:48:29.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:29.601+0000 7fa066ec8700 1 -- 172.21.15.189:0/207598160 wait complete. 2024-04-06T13:48:29.603 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 116 2024-04-06T13:48:30.017 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-06T13:48:19.674205+0000', 'modified': '2024-04-06T13:48:22.682911+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12108}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12108': {'gid': 12108, 'name': 'd', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.192:6833/726816746', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6832', 'nonce': 726816746}, {'type': 'v1', 'addr': '172.21.15.192:6833', 'nonce': 726816746}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [32], 'metadata_pool': 31, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-06T13:48:30.018 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-06T13:48:30.018 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-06T13:48:30.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.144+0000 7f56a2cf8700 1 Processor -- start 2024-04-06T13:48:30.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.144+0000 7f56a2cf8700 1 -- start start 2024-04-06T13:48:30.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.145+0000 7f56a2cf8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f569c0cedf0 0x7f569c0cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.145+0000 7f56a2cf8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f569c0cf730 con 0x7f569c146dd0 2024-04-06T13:48:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.145+0000 7f56a2cf8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f569c06d850 con 0x7f569c0d17a0 2024-04-06T13:48:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.145+0000 7f56a2cf8700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f569c06beb0 con 0x7f569c0cedf0 2024-04-06T13:48:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.145+0000 7f569bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f569c146dd0 0x7f569c06d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55780/0 (socket says 172.21.15.189:55780) 2024-04-06T13:48:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.145+0000 7f569bfff700 1 -- 172.21.15.189:0/2827123999 learned_addr learned my addr 172.21.15.189:0/2827123999 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.145+0000 7f56a1295700 1 --2- 172.21.15.189:0/2827123999 >> v2:172.21.15.192:3300/0 conn(0x7f569c0cedf0 0x7f569c0cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.145+0000 7f569b7fe700 1 -- 172.21.15.189:0/2827123999 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087604216 0 0) 0x7f569c0cf730 con 0x7f569c146dd0 2024-04-06T13:48:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.145+0000 7f569b7fe700 1 -- 172.21.15.189:0/2827123999 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f568c0035b0 con 0x7f569c146dd0 2024-04-06T13:48:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.145+0000 7f56a1295700 1 -- 172.21.15.189:0/2827123999 >> v1:172.21.15.192:6789/0 conn(0x7f569c0d17a0 legacy=0x7f569c06b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.145+0000 7f56a1295700 1 -- 172.21.15.189:0/2827123999 >> v1:172.21.15.189:6789/0 conn(0x7f569c146dd0 legacy=0x7f569c06d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.145+0000 7f569b7fe700 1 -- 172.21.15.189:0/2827123999 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4116025054 0 0) 0x7f568c0035b0 con 0x7f569c146dd0 2024-04-06T13:48:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.145+0000 7f56a1295700 1 -- 172.21.15.189:0/2827123999 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569c06c0a0 con 0x7f569c0cedf0 2024-04-06T13:48:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.146+0000 7f56a1295700 1 --2- 172.21.15.189:0/2827123999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f569c0cedf0 0x7f569c0cf1f0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f5694009d90 tx=0x7f569400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53ac3ca052d10452 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.146+0000 7f569b7fe700 1 -- 172.21.15.189:0/2827123999 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569400e070 con 0x7f569c0cedf0 2024-04-06T13:48:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.146+0000 7f569b7fe700 1 -- 172.21.15.189:0/2827123999 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569400bb20 con 0x7f569c0cedf0 2024-04-06T13:48:30.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.146+0000 7f569b7fe700 1 -- 172.21.15.189:0/2827123999 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569400bed0 con 0x7f569c0cedf0 2024-04-06T13:48:30.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.146+0000 7f56a2cf8700 1 -- 172.21.15.189:0/2827123999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f569c0cedf0 msgr2=0x7f569c0cf1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:30.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.146+0000 7f56a2cf8700 1 --2- 172.21.15.189:0/2827123999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f569c0cedf0 0x7f569c0cf1f0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f5694009d90 tx=0x7f569400b330 comp rx=0 tx=0).stop 2024-04-06T13:48:30.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.146+0000 7f56a2cf8700 1 -- 172.21.15.189:0/2827123999 shutdown_connections 2024-04-06T13:48:30.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.146+0000 7f56a2cf8700 1 --2- 172.21.15.189:0/2827123999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f569c0cedf0 0x7f569c0cf1f0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.146+0000 7f56a2cf8700 1 -- 172.21.15.189:0/2827123999 >> 172.21.15.189:0/2827123999 conn(0x7f569c0b2db0 msgr2=0x7f569c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:30.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.146+0000 7f56a2cf8700 1 -- 172.21.15.189:0/2827123999 shutdown_connections 2024-04-06T13:48:30.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.146+0000 7f56a2cf8700 1 -- 172.21.15.189:0/2827123999 wait complete. 2024-04-06T13:48:30.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.148+0000 7f56a2cf8700 1 Processor -- start 2024-04-06T13:48:30.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.148+0000 7f56a2cf8700 1 -- start start 2024-04-06T13:48:30.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.148+0000 7f56a2cf8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f569c0cedf0 0x7f569c155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.148+0000 7f56a2cf8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f569c0d17a0 0x7f569c1560d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.148+0000 7f56a2cf8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f569c146dd0 0x7f569c0af8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.148+0000 7f56a2cf8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569c0aed90 con 0x7f569c146dd0 2024-04-06T13:48:30.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.148+0000 7f56a2cf8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569c1f2ff0 con 0x7f569c0d17a0 2024-04-06T13:48:30.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.148+0000 7f56a2cf8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f569c0ae140 con 0x7f569c0cedf0 2024-04-06T13:48:30.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.149+0000 7f56a1295700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f569c146dd0 0x7f569c0af8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.149+0000 7f56a1295700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f569c146dd0 0x7f569c0af8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49254/0 (socket says 172.21.15.189:49254) 2024-04-06T13:48:30.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.149+0000 7f56a1295700 1 -- 172.21.15.189:0/3470492206 learned_addr learned my addr 172.21.15.189:0/3470492206 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:30.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.149+0000 7f569bfff700 1 --2- 172.21.15.189:0/3470492206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f569c0d17a0 0x7f569c1560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.149+0000 7f56a0a94700 1 --2- 172.21.15.189:0/3470492206 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f569c0cedf0 0x7f569c155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.149+0000 7f56a1295700 1 -- 172.21.15.189:0/3470492206 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f569c0cedf0 msgr2=0x7f569c155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:30.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.149+0000 7f56a1295700 1 --2- 172.21.15.189:0/3470492206 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f569c0cedf0 0x7f569c155b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.149+0000 7f56a1295700 1 -- 172.21.15.189:0/3470492206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f569c0d17a0 msgr2=0x7f569c1560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:30.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.149+0000 7f56a1295700 1 --2- 172.21.15.189:0/3470492206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f569c0d17a0 0x7f569c1560d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.149+0000 7f56a1295700 1 -- 172.21.15.189:0/3470492206 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5694009a70 con 0x7f569c146dd0 2024-04-06T13:48:30.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.149+0000 7f56a1295700 1 --2- 172.21.15.189:0/3470492206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f569c146dd0 0x7f569c0af8e0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f569c0677e0 tx=0x7f5694004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:30.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.149+0000 7f56997fa700 1 -- 172.21.15.189:0/3470492206 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569401b810 con 0x7f569c146dd0 2024-04-06T13:48:30.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.149+0000 7f56997fa700 1 -- 172.21.15.189:0/3470492206 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5694020070 con 0x7f569c146dd0 2024-04-06T13:48:30.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.149+0000 7f56997fa700 1 -- 172.21.15.189:0/3470492206 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569401bbb0 con 0x7f569c146dd0 2024-04-06T13:48:30.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.149+0000 7f56a2cf8700 1 -- 172.21.15.189:0/3470492206 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f569c158420 con 0x7f569c146dd0 2024-04-06T13:48:30.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.149+0000 7f56a2cf8700 1 -- 172.21.15.189:0/3470492206 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f569c0ad980 con 0x7f569c146dd0 2024-04-06T13:48:30.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.151+0000 7f56997fa700 1 -- 172.21.15.189:0/3470492206 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f569401bd10 con 0x7f569c146dd0 2024-04-06T13:48:30.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.151+0000 7f56997fa700 1 --2- 172.21.15.189:0/3470492206 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f568006aed0 0x7f568006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.151+0000 7f56997fa700 1 -- 172.21.15.189:0/3470492206 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5694022070 con 0x7f569c146dd0 2024-04-06T13:48:30.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.151+0000 7f56a0a94700 1 --2- 172.21.15.189:0/3470492206 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f568006aed0 0x7f568006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.152+0000 7f56a0a94700 1 --2- 172.21.15.189:0/3470492206 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f568006aed0 0x7f568006d390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5690000f60 tx=0x7f5690009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:30.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.153+0000 7f56aeeacb80 1 Processor -- start 2024-04-06T13:48:30.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.153+0000 7f56aeeacb80 1 -- start start 2024-04-06T13:48:30.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.154+0000 7f56aeeacb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb75c20 0x55b5ccb79a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.154+0000 7f56aeeacb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb7afc0 0x55b5ccb515f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.154+0000 7f56aeeacb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b5ccb50980 0x55b5ccb51b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.154+0000 7f56aeeacb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ccaaa940 con 0x55b5ccb7afc0 2024-04-06T13:48:30.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.154+0000 7f56aeeacb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ccaa8300 con 0x55b5ccb75c20 2024-04-06T13:48:30.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.154+0000 7f56aeeacb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b5ccaa3bf0 con 0x55b5ccb50980 2024-04-06T13:48:30.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.154+0000 7f569bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb7afc0 0x55b5ccb515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.154+0000 7f569bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb7afc0 0x55b5ccb515f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49270/0 (socket says 172.21.15.189:49270) 2024-04-06T13:48:30.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.154+0000 7f569bfff700 1 -- 172.21.15.189:0/2538081982 learned_addr learned my addr 172.21.15.189:0/2538081982 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:30.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.154+0000 7f56a0a94700 1 --2- 172.21.15.189:0/2538081982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb75c20 0x55b5ccb79a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.154+0000 7f569bfff700 1 -- 172.21.15.189:0/2538081982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b5ccb50980 msgr2=0x55b5ccb51b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:30.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.154+0000 7f569bfff700 1 --2- 172.21.15.189:0/2538081982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b5ccb50980 0x55b5ccb51b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.154+0000 7f569bfff700 1 -- 172.21.15.189:0/2538081982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb75c20 msgr2=0x55b5ccb79a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:30.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.154+0000 7f569bfff700 1 --2- 172.21.15.189:0/2538081982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb75c20 0x55b5ccb79a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.154+0000 7f569bfff700 1 -- 172.21.15.189:0/2538081982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5cc8d1b10 con 0x55b5ccb7afc0 2024-04-06T13:48:30.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.155+0000 7f569bfff700 1 --2- 172.21.15.189:0/2538081982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb7afc0 0x55b5ccb515f0 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f5688000f60 tx=0x7f5688001b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32dee30bc29d7cf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:30.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.155+0000 7f56867fc700 1 -- 172.21.15.189:0/2538081982 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568800b070 con 0x55b5ccb7afc0 2024-04-06T13:48:30.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.155+0000 7f56867fc700 1 -- 172.21.15.189:0/2538081982 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5688005b40 con 0x55b5ccb7afc0 2024-04-06T13:48:30.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.155+0000 7f56867fc700 1 -- 172.21.15.189:0/2538081982 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56880070d0 con 0x55b5ccb7afc0 2024-04-06T13:48:30.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.155+0000 7f56aeeacb80 1 -- 172.21.15.189:0/2538081982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb7afc0 msgr2=0x55b5ccb515f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:30.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.155+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/2538081982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb7afc0 0x55b5ccb515f0 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f5688000f60 tx=0x7f5688001b90 comp rx=0 tx=0).stop 2024-04-06T13:48:30.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.155+0000 7f56aeeacb80 1 -- 172.21.15.189:0/2538081982 shutdown_connections 2024-04-06T13:48:30.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.155+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/2538081982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b5ccb50980 0x55b5ccb51b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.155+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/2538081982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb75c20 0x55b5ccb79a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.155+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/2538081982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb7afc0 0x55b5ccb515f0 unknown :-1 s=CLOSED pgs=2698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.155+0000 7f56aeeacb80 1 -- 172.21.15.189:0/2538081982 >> 172.21.15.189:0/2538081982 conn(0x55b5ccab1650 msgr2=0x55b5ccb983e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:30.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.155+0000 7f56aeeacb80 1 -- 172.21.15.189:0/2538081982 shutdown_connections 2024-04-06T13:48:30.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.155+0000 7f56aeeacb80 1 -- 172.21.15.189:0/2538081982 wait complete. 2024-04-06T13:48:30.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.156+0000 7f56aeeacb80 1 -- start start 2024-04-06T13:48:30.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.156+0000 7f56aeeacb80 1 Processor -- start 2024-04-06T13:48:30.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.157+0000 7f56aeeacb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb50980 0x55b5ccb704e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.157+0000 7f56aeeacb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b5ccb75c20 0x55b5ccb74a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.157+0000 7f56aeeacb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb7afc0 0x55b5ccb808a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.157+0000 7f56aeeacb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5cca3fd30 con 0x55b5ccb7afc0 2024-04-06T13:48:30.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.157+0000 7f56aeeacb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5cca3f9d0 con 0x55b5ccb50980 2024-04-06T13:48:30.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.157+0000 7f56aeeacb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b5cc927650 con 0x55b5ccb75c20 2024-04-06T13:48:30.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.157+0000 7f56aeeacb80 10 client.0 _ll_register_callbacks cb 0x55b5cca49420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T13:48:30.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.157+0000 7f56a1295700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb7afc0 0x55b5ccb808a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.157+0000 7f56a1295700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb7afc0 0x55b5ccb808a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49272/0 (socket says 172.21.15.189:49272) 2024-04-06T13:48:30.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.157+0000 7f56a1295700 1 -- 172.21.15.189:0/928655024 learned_addr learned my addr 172.21.15.189:0/928655024 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:30.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.157+0000 7f56a0a94700 1 --2- 172.21.15.189:0/928655024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb50980 0x55b5ccb704e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.157+0000 7f569bfff700 1 --2- 172.21.15.189:0/928655024 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b5ccb75c20 0x55b5ccb74a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.157+0000 7f56a1295700 1 -- 172.21.15.189:0/928655024 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b5ccb75c20 msgr2=0x55b5ccb74a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:30.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.157+0000 7f56a1295700 1 --2- 172.21.15.189:0/928655024 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b5ccb75c20 0x55b5ccb74a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.157+0000 7f56a1295700 1 -- 172.21.15.189:0/928655024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb50980 msgr2=0x55b5ccb704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:30.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.157+0000 7f56a1295700 1 --2- 172.21.15.189:0/928655024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb50980 0x55b5ccb704e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.157+0000 7f56a1295700 1 -- 172.21.15.189:0/928655024 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5688001510 con 0x55b5ccb7afc0 2024-04-06T13:48:30.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.157+0000 7f569bfff700 1 --2- 172.21.15.189:0/928655024 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b5ccb75c20 0x55b5ccb74a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:30.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.157+0000 7f56a0a94700 1 --2- 172.21.15.189:0/928655024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb50980 0x55b5ccb704e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:30.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.158+0000 7f56a1295700 1 --2- 172.21.15.189:0/928655024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb7afc0 0x55b5ccb808a0 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f569405a490 tx=0x7f569405e5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a34be4a009f0aaf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:30.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.158+0000 7f567b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T13:48:30.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.158+0000 7f567b7fe700 1 -- 172.21.15.189:0/928655024 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5694072170 con 0x55b5ccb7afc0 2024-04-06T13:48:30.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.158+0000 7f567b7fe700 1 -- 172.21.15.189:0/928655024 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f569405e170 con 0x55b5ccb7afc0 2024-04-06T13:48:30.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.158+0000 7f56aeeacb80 1 -- 172.21.15.189:0/928655024 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b5cca64330 con 0x55b5ccb7afc0 2024-04-06T13:48:30.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.158+0000 7f567b7fe700 1 -- 172.21.15.189:0/928655024 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) v2 ==== 24+0+0 (secure 0 0 0) 0x7f569405e330 con 0x55b5ccb7afc0 2024-04-06T13:48:30.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.158+0000 7f56aeeacb80 10 client.12261 fetch_fsmap learned FSMap version 116 2024-04-06T13:48:30.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.158+0000 7f56aeeacb80 1 -- 172.21.15.189:0/928655024 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=2+}) v3 -- 0x55b5cca69e50 con 0x55b5ccb7afc0 2024-04-06T13:48:30.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.158+0000 7f567b7fe700 1 -- 172.21.15.189:0/928655024 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b5cca64330 con 0x55b5ccb7afc0 2024-04-06T13:48:30.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.158+0000 7f567b7fe700 1 -- 172.21.15.189:0/928655024 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 116) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5694083d20 con 0x55b5ccb7afc0 2024-04-06T13:48:30.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.158+0000 7f56aeeacb80 10 client.12261 fetch_fsmap finished waiting for FSMap version 116 2024-04-06T13:48:30.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.158+0000 7f56aeeacb80 10 client.12261 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-06T13:48:30.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.158+0000 7f56aeeacb80 20 client.12261 populate_metadata read hostname 'smithi189' 2024-04-06T13:48:30.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.158+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/928655024 >> [v2:172.21.15.192:6832/726816746,v1:172.21.15.192:6833/726816746] conn(0x55b5ccbab5e0 0x55b5ccbab9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.158+0000 7f56aeeacb80 4 client.12261 mds_command: new command op to 12108 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-06T13:48:30.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.158+0000 7f56aeeacb80 1 -- 172.21.15.189:0/928655024 --> [v2:172.21.15.192:6832/726816746,v1:172.21.15.192:6833/726816746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b5cca9c6b0 con 0x55b5ccbab5e0 2024-04-06T13:48:30.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.158+0000 7f56a0a94700 1 --2- 172.21.15.189:0/928655024 >> [v2:172.21.15.192:6832/726816746,v1:172.21.15.192:6833/726816746] conn(0x55b5ccbab5e0 0x55b5ccbab9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.159+0000 7f567b7fe700 10 client.12261 ms_handle_connect on v2:172.21.15.192:6832/726816746 2024-04-06T13:48:30.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.159+0000 7f56a0a94700 1 --2- 172.21.15.189:0/928655024 >> [v2:172.21.15.192:6832/726816746,v1:172.21.15.192:6833/726816746] conn(0x55b5ccbab5e0 0x55b5ccbab9c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e712fbd58277ecf server_cookie=72e6373884cf9def in_seq=0 out_seq=0 2024-04-06T13:48:30.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.159+0000 7f567b7fe700 1 -- 172.21.15.189:0/928655024 <== mds.0 v2:172.21.15.192:6832/726816746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b5cca9c6b0 con 0x55b5ccbab5e0 2024-04-06T13:48:30.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.159+0000 7f567b7fe700 10 client.12261 handle_command_reply: tid=0 2024-04-06T13:48:30.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.159+0000 7f56aeeacb80 1 client.12261 shutdown 2024-04-06T13:48:30.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.160+0000 7f56aeeacb80 1 -- 172.21.15.189:0/928655024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb7afc0 msgr2=0x55b5ccb808a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:30.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.160+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/928655024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb7afc0 0x55b5ccb808a0 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f569405a490 tx=0x7f569405e5c0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.160+0000 7f56aeeacb80 1 -- 172.21.15.189:0/928655024 shutdown_connections 2024-04-06T13:48:30.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.160+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/928655024 >> [v2:172.21.15.192:6832/726816746,v1:172.21.15.192:6833/726816746] conn(0x55b5ccbab5e0 0x55b5ccbab9c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.160+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/928655024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb7afc0 0x55b5ccb808a0 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.160+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/928655024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb50980 0x55b5ccb704e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.160+0000 7f567b7fe700 0 client.12261 ms_handle_reset on v2:172.21.15.192:6832/726816746 2024-04-06T13:48:30.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.160+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/928655024 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b5ccb75c20 0x55b5ccb74a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.160+0000 7f56aeeacb80 1 -- 172.21.15.189:0/928655024 >> 172.21.15.189:0/928655024 conn(0x55b5ccab1650 msgr2=0x55b5ccb51bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:30.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.160+0000 7f56aeeacb80 1 -- 172.21.15.189:0/928655024 shutdown_connections 2024-04-06T13:48:30.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.160+0000 7f56aeeacb80 1 -- 172.21.15.189:0/928655024 wait complete. 2024-04-06T13:48:30.166 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.160+0000 7f56aeeacb80 20 client.12261 trim_cache size 0 max 16384 2024-04-06T13:48:30.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.177+0000 7f56aeeacb80 1 Processor -- start 2024-04-06T13:48:30.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.177+0000 7f56aeeacb80 1 -- start start 2024-04-06T13:48:30.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.178+0000 7f56aeeacb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb50980 0x55b5ccb52710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.178+0000 7f56aeeacb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b5ccb75c20 0x55b5ccb52c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.178+0000 7f56aeeacb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb7afc0 0x55b5ccb72ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.178+0000 7f56aeeacb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ccb70610 con 0x55b5ccb50980 2024-04-06T13:48:30.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.178+0000 7f56aeeacb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ccb91470 con 0x55b5ccb7afc0 2024-04-06T13:48:30.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.178+0000 7f56aeeacb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b5ccb98d50 con 0x55b5ccb75c20 2024-04-06T13:48:30.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.178+0000 7f56a0a94700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb50980 0x55b5ccb52710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.178+0000 7f56a0a94700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb50980 0x55b5ccb52710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49282/0 (socket says 172.21.15.189:49282) 2024-04-06T13:48:30.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.178+0000 7f56a0a94700 1 -- 172.21.15.189:0/705059816 learned_addr learned my addr 172.21.15.189:0/705059816 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:30.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.178+0000 7f56a0a94700 1 -- 172.21.15.189:0/705059816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b5ccb75c20 msgr2=0x55b5ccb52c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:30.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.178+0000 7f56a0a94700 1 --2- 172.21.15.189:0/705059816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b5ccb75c20 0x55b5ccb52c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.178+0000 7f56a0a94700 1 -- 172.21.15.189:0/705059816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb7afc0 msgr2=0x55b5ccb72ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:30.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.178+0000 7f56a0a94700 1 --2- 172.21.15.189:0/705059816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb7afc0 0x55b5ccb72ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.178+0000 7f56a0a94700 1 -- 172.21.15.189:0/705059816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5cc929c80 con 0x55b5ccb50980 2024-04-06T13:48:30.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.179+0000 7f56a0a94700 1 --2- 172.21.15.189:0/705059816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb50980 0x55b5ccb52710 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7f5690007e70 tx=0x7f5690013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c57a80d0c7ddc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:30.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.179+0000 7f5686ffd700 1 -- 172.21.15.189:0/705059816 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5690023d30 con 0x55b5ccb50980 2024-04-06T13:48:30.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.179+0000 7f5686ffd700 1 -- 172.21.15.189:0/705059816 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5690027070 con 0x55b5ccb50980 2024-04-06T13:48:30.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.179+0000 7f5686ffd700 1 -- 172.21.15.189:0/705059816 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5690023e90 con 0x55b5ccb50980 2024-04-06T13:48:30.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.179+0000 7f56aeeacb80 1 -- 172.21.15.189:0/705059816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb50980 msgr2=0x55b5ccb52710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:30.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.179+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/705059816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb50980 0x55b5ccb52710 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7f5690007e70 tx=0x7f5690013000 comp rx=0 tx=0).stop 2024-04-06T13:48:30.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.179+0000 7f56aeeacb80 1 -- 172.21.15.189:0/705059816 shutdown_connections 2024-04-06T13:48:30.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.179+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/705059816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb7afc0 0x55b5ccb72ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.179+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/705059816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb50980 0x55b5ccb52710 unknown :-1 s=CLOSED pgs=2700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.179+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/705059816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b5ccb75c20 0x55b5ccb52c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.179+0000 7f56aeeacb80 1 -- 172.21.15.189:0/705059816 >> 172.21.15.189:0/705059816 conn(0x55b5ccab1650 msgr2=0x55b5ccbba470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:30.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.179+0000 7f56aeeacb80 1 -- 172.21.15.189:0/705059816 shutdown_connections 2024-04-06T13:48:30.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.179+0000 7f56aeeacb80 1 -- 172.21.15.189:0/705059816 wait complete. 2024-04-06T13:48:30.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.180+0000 7f56aeeacb80 1 -- start start 2024-04-06T13:48:30.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.180+0000 7f56aeeacb80 1 Processor -- start 2024-04-06T13:48:30.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.181+0000 7f56aeeacb80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb50980 0x55b5ccb52710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.181+0000 7f56aeeacb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb75c20 0x55b5ccb52c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.181+0000 7f56aeeacb80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b5ccb7afc0 0x55b5ccb924c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.181+0000 7f56aeeacb80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ccb93370 con 0x55b5ccb50980 2024-04-06T13:48:30.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.181+0000 7f56aeeacb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ccb4cbf0 con 0x55b5ccb75c20 2024-04-06T13:48:30.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.181+0000 7f56aeeacb80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b5ccb4d120 con 0x55b5ccb7afc0 2024-04-06T13:48:30.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.181+0000 7f56aeeacb80 10 client.0 _ll_register_callbacks cb 0x55b5cca49420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T13:48:30.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.181+0000 7f56a0a94700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb50980 0x55b5ccb52710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.181+0000 7f56a0a94700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb50980 0x55b5ccb52710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49298/0 (socket says 172.21.15.189:49298) 2024-04-06T13:48:30.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.181+0000 7f56a0a94700 1 -- 172.21.15.189:0/4084125017 learned_addr learned my addr 172.21.15.189:0/4084125017 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:30.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.181+0000 7f56a1295700 1 --2- 172.21.15.189:0/4084125017 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b5ccb7afc0 0x55b5ccb924c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.181+0000 7f569bfff700 1 --2- 172.21.15.189:0/4084125017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb75c20 0x55b5ccb52c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.181+0000 7f56a0a94700 1 -- 172.21.15.189:0/4084125017 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b5ccb7afc0 msgr2=0x55b5ccb924c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:30.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.181+0000 7f56a0a94700 1 --2- 172.21.15.189:0/4084125017 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b5ccb7afc0 0x55b5ccb924c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.181+0000 7f56a0a94700 1 -- 172.21.15.189:0/4084125017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb75c20 msgr2=0x55b5ccb52c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:30.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.181+0000 7f56a0a94700 1 --2- 172.21.15.189:0/4084125017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb75c20 0x55b5ccb52c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.181+0000 7f56a0a94700 1 -- 172.21.15.189:0/4084125017 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f569001d040 con 0x55b5ccb50980 2024-04-06T13:48:30.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.181+0000 7f56a0a94700 1 --2- 172.21.15.189:0/4084125017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb50980 0x55b5ccb52710 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f5690009010 tx=0x7f5690000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2369136b01a944 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:30.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.181+0000 7f567b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T13:48:30.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.182+0000 7f567b7fe700 1 -- 172.21.15.189:0/4084125017 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5690031070 con 0x55b5ccb50980 2024-04-06T13:48:30.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.182+0000 7f567b7fe700 1 -- 172.21.15.189:0/4084125017 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f56900233b0 con 0x55b5ccb50980 2024-04-06T13:48:30.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.182+0000 7f56aeeacb80 1 -- 172.21.15.189:0/4084125017 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5688005b40 con 0x55b5ccb50980 2024-04-06T13:48:30.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.182+0000 7f567b7fe700 1 -- 172.21.15.189:0/4084125017 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5690023570 con 0x55b5ccb50980 2024-04-06T13:48:30.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.182+0000 7f56aeeacb80 10 client.12267 fetch_fsmap learned FSMap version 116 2024-04-06T13:48:30.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.182+0000 7f56aeeacb80 1 -- 172.21.15.189:0/4084125017 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=2+}) v3 -- 0x55b5cc8cffe0 con 0x55b5ccb50980 2024-04-06T13:48:30.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.182+0000 7f567b7fe700 1 -- 172.21.15.189:0/4084125017 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5688005b40 con 0x55b5ccb50980 2024-04-06T13:48:30.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.182+0000 7f567b7fe700 1 -- 172.21.15.189:0/4084125017 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 116) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5690003a00 con 0x55b5ccb50980 2024-04-06T13:48:30.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.182+0000 7f56aeeacb80 10 client.12267 fetch_fsmap finished waiting for FSMap version 116 2024-04-06T13:48:30.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.182+0000 7f56aeeacb80 10 client.12267 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-06T13:48:30.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.182+0000 7f56aeeacb80 20 client.12267 populate_metadata read hostname 'smithi189' 2024-04-06T13:48:30.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.182+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/4084125017 >> [v2:172.21.15.192:6832/726816746,v1:172.21.15.192:6833/726816746] conn(0x55b5ccb54490 0x55b5ccb92a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.182+0000 7f56aeeacb80 4 client.12267 mds_command: new command op to 12108 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-06T13:48:30.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.182+0000 7f56aeeacb80 1 -- 172.21.15.189:0/4084125017 --> [v2:172.21.15.192:6832/726816746,v1:172.21.15.192:6833/726816746] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b5cc8f2ca0 con 0x55b5ccb54490 2024-04-06T13:48:30.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.182+0000 7f569bfff700 1 --2- 172.21.15.189:0/4084125017 >> [v2:172.21.15.192:6832/726816746,v1:172.21.15.192:6833/726816746] conn(0x55b5ccb54490 0x55b5ccb92a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.183+0000 7f569bfff700 1 --2- 172.21.15.189:0/4084125017 >> [v2:172.21.15.192:6832/726816746,v1:172.21.15.192:6833/726816746] conn(0x55b5ccb54490 0x55b5ccb92a00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4ebef31c79d2559 server_cookie=25e52727b7473216 in_seq=0 out_seq=0 2024-04-06T13:48:30.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.183+0000 7f567b7fe700 10 client.12267 ms_handle_connect on v2:172.21.15.192:6832/726816746 2024-04-06T13:48:30.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.183+0000 7f567b7fe700 1 -- 172.21.15.189:0/4084125017 <== mds.0 v2:172.21.15.192:6832/726816746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55b5cc8f2ca0 con 0x55b5ccb54490 2024-04-06T13:48:30.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.183+0000 7f567b7fe700 10 client.12267 handle_command_reply: tid=0 2024-04-06T13:48:30.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.183+0000 7f56aeeacb80 1 client.12267 shutdown 2024-04-06T13:48:30.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.183+0000 7f56aeeacb80 1 -- 172.21.15.189:0/4084125017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb50980 msgr2=0x55b5ccb52710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:30.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.183+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/4084125017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb50980 0x55b5ccb52710 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f5690009010 tx=0x7f5690000ed0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.183+0000 7f56aeeacb80 1 -- 172.21.15.189:0/4084125017 shutdown_connections 2024-04-06T13:48:30.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.183+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/4084125017 >> [v2:172.21.15.192:6832/726816746,v1:172.21.15.192:6833/726816746] conn(0x55b5ccb54490 0x55b5ccb92a00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.183+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/4084125017 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b5ccb7afc0 0x55b5ccb924c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.183+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/4084125017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ccb50980 0x55b5ccb52710 unknown :-1 s=CLOSED pgs=2701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.183+0000 7f567b7fe700 0 client.12267 ms_handle_reset on v2:172.21.15.192:6832/726816746 2024-04-06T13:48:30.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.183+0000 7f56aeeacb80 1 --2- 172.21.15.189:0/4084125017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5ccb75c20 0x55b5ccb52c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.183+0000 7f56aeeacb80 1 -- 172.21.15.189:0/4084125017 >> 172.21.15.189:0/4084125017 conn(0x55b5ccab1650 msgr2=0x55b5ccb806b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:30.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.183+0000 7f56aeeacb80 1 -- 172.21.15.189:0/4084125017 shutdown_connections 2024-04-06T13:48:30.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.183+0000 7f56aeeacb80 1 -- 172.21.15.189:0/4084125017 wait complete. 2024-04-06T13:48:30.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.183+0000 7f56aeeacb80 20 client.12267 trim_cache size 0 max 16384 2024-04-06T13:48:30.190 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:48:30.190 INFO:teuthology.orchestra.run.smithi189.stdout: "cluster_fsid": "e30041de-e9a3-4351-8ca7-29d7f3e3728c", 2024-04-06T13:48:30.190 INFO:teuthology.orchestra.run.smithi189.stdout: "whoami": 0, 2024-04-06T13:48:30.190 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 12108, 2024-04-06T13:48:30.190 INFO:teuthology.orchestra.run.smithi189.stdout: "want_state": "up:active", 2024-04-06T13:48:30.190 INFO:teuthology.orchestra.run.smithi189.stdout: "state": "up:active", 2024-04-06T13:48:30.190 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs", 2024-04-06T13:48:30.190 INFO:teuthology.orchestra.run.smithi189.stdout: "rank_uptime": 10.507524521000001, 2024-04-06T13:48:30.190 INFO:teuthology.orchestra.run.smithi189.stdout: "mdsmap_epoch": 116, 2024-04-06T13:48:30.190 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch": 248, 2024-04-06T13:48:30.191 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch_barrier": 248, 2024-04-06T13:48:30.191 INFO:teuthology.orchestra.run.smithi189.stdout: "uptime": 15.821764701999999 2024-04-06T13:48:30.191 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:48:30.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.184+0000 7f56a2cf8700 1 -- 172.21.15.189:0/3470492206 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f568006aed0 msgr2=0x7f568006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:30.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.184+0000 7f56a2cf8700 1 --2- 172.21.15.189:0/3470492206 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f568006aed0 0x7f568006d390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5690000f60 tx=0x7f5690009040 comp rx=0 tx=0).stop 2024-04-06T13:48:30.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.184+0000 7f56a2cf8700 1 -- 172.21.15.189:0/3470492206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f569c146dd0 msgr2=0x7f569c0af8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:30.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.184+0000 7f56a2cf8700 1 --2- 172.21.15.189:0/3470492206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f569c146dd0 0x7f569c0af8e0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f569c0677e0 tx=0x7f5694004ad0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.184+0000 7f56a2cf8700 1 -- 172.21.15.189:0/3470492206 shutdown_connections 2024-04-06T13:48:30.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.184+0000 7f56a2cf8700 1 --2- 172.21.15.189:0/3470492206 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f568006aed0 0x7f568006d390 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.184+0000 7f56a2cf8700 1 --2- 172.21.15.189:0/3470492206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f569c146dd0 0x7f569c0af8e0 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.184+0000 7f56a2cf8700 1 --2- 172.21.15.189:0/3470492206 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f569c0cedf0 0x7f569c155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.184+0000 7f56a2cf8700 1 --2- 172.21.15.189:0/3470492206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f569c0d17a0 0x7f569c1560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.184+0000 7f56a2cf8700 1 -- 172.21.15.189:0/3470492206 >> 172.21.15.189:0/3470492206 conn(0x7f569c0b2db0 msgr2=0x7f569c0b5690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:30.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.184+0000 7f56a2cf8700 1 -- 172.21.15.189:0/3470492206 shutdown_connections 2024-04-06T13:48:30.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.184+0000 7f56a2cf8700 1 -- 172.21.15.189:0/3470492206 wait complete. 2024-04-06T13:48:30.196 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T13:48:30.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.318+0000 7f52dd298700 1 Processor -- start 2024-04-06T13:48:30.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.318+0000 7f52dd298700 1 -- start start 2024-04-06T13:48:30.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.319+0000 7f52dd298700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52d8143470 0x7f52d80b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.319+0000 7f52dd298700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d815eb40 con 0x7f52d815db10 2024-04-06T13:48:30.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.319+0000 7f52dd298700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d80d0bf0 con 0x7f52d8160b60 2024-04-06T13:48:30.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.319+0000 7f52dd298700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d8160f40 con 0x7f52d8143470 2024-04-06T13:48:30.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.319+0000 7f52d67fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f52d815db10 0x7f52d80cd890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55790/0 (socket says 172.21.15.189:55790) 2024-04-06T13:48:30.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.319+0000 7f52d67fc700 1 -- 172.21.15.189:0/1028309398 learned_addr learned my addr 172.21.15.189:0/1028309398 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:30.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.319+0000 7f52d77fe700 1 --2- 172.21.15.189:0/1028309398 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52d8143470 0x7f52d80b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.319+0000 7f52d5ffb700 1 -- 172.21.15.189:0/1028309398 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313818277 0 0) 0x7f52c8001290 con 0x7f52d815db10 2024-04-06T13:48:30.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.319+0000 7f52d5ffb700 1 -- 172.21.15.189:0/1028309398 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52c00035b0 con 0x7f52d815db10 2024-04-06T13:48:30.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.319+0000 7f52d5ffb700 1 -- 172.21.15.189:0/1028309398 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237301331 0 0) 0x7f52d80d0bf0 con 0x7f52d8160b60 2024-04-06T13:48:30.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.319+0000 7f52d5ffb700 1 -- 172.21.15.189:0/1028309398 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52c8001290 con 0x7f52d8160b60 2024-04-06T13:48:30.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.319+0000 7f52d5ffb700 1 -- 172.21.15.189:0/1028309398 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3764883525 0 0) 0x7f52c00035b0 con 0x7f52d815db10 2024-04-06T13:48:30.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.319+0000 7f52d77fe700 1 -- 172.21.15.189:0/1028309398 >> v1:172.21.15.192:6789/0 conn(0x7f52d8160b60 legacy=0x7f52d8142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:30.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.319+0000 7f52d77fe700 1 -- 172.21.15.189:0/1028309398 >> v1:172.21.15.189:6789/0 conn(0x7f52d815db10 legacy=0x7f52d80cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:30.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.319+0000 7f52d77fe700 1 -- 172.21.15.189:0/1028309398 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d815dfc0 con 0x7f52d8143470 2024-04-06T13:48:30.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.320+0000 7f52d77fe700 1 --2- 172.21.15.189:0/1028309398 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52d8143470 0x7f52d80b37f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f52d0009dc0 tx=0x7f52d000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e75057d2fa53cdc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:30.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.320+0000 7f52d5ffb700 1 -- 172.21.15.189:0/1028309398 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d000e070 con 0x7f52d8143470 2024-04-06T13:48:30.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.320+0000 7f52d5ffb700 1 -- 172.21.15.189:0/1028309398 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d0003a20 con 0x7f52d8143470 2024-04-06T13:48:30.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.320+0000 7f52d5ffb700 1 -- 172.21.15.189:0/1028309398 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8001980 con 0x7f52d8143470 2024-04-06T13:48:30.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.320+0000 7f52dd298700 1 -- 172.21.15.189:0/1028309398 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52d8143470 msgr2=0x7f52d80b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:30.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.320+0000 7f52dd298700 1 --2- 172.21.15.189:0/1028309398 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52d8143470 0x7f52d80b37f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f52d0009dc0 tx=0x7f52d000b330 comp rx=0 tx=0).stop 2024-04-06T13:48:30.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.320+0000 7f52dd298700 1 -- 172.21.15.189:0/1028309398 shutdown_connections 2024-04-06T13:48:30.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.320+0000 7f52dd298700 1 --2- 172.21.15.189:0/1028309398 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52d8143470 0x7f52d80b37f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.320+0000 7f52dd298700 1 -- 172.21.15.189:0/1028309398 >> 172.21.15.189:0/1028309398 conn(0x7f52d806a730 msgr2=0x7f52d806ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:30.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.320+0000 7f52dd298700 1 -- 172.21.15.189:0/1028309398 shutdown_connections 2024-04-06T13:48:30.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.320+0000 7f52dd298700 1 -- 172.21.15.189:0/1028309398 wait complete. 2024-04-06T13:48:30.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.322+0000 7f52dd298700 1 Processor -- start 2024-04-06T13:48:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.322+0000 7f52dd298700 1 -- start start 2024-04-06T13:48:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52dd298700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52d8143470 0x7f52d80cda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52dd298700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52d815db10 0x7f52d80b6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52dd298700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52d8160b60 0x7f52d815ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52dd298700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d80b67d0 con 0x7f52d8160b60 2024-04-06T13:48:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52dd298700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d80b5e00 con 0x7f52d8143470 2024-04-06T13:48:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52dd298700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d815d340 con 0x7f52d815db10 2024-04-06T13:48:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52d77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52d8160b60 0x7f52d815ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52d77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52d8160b60 0x7f52d815ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49314/0 (socket says 172.21.15.189:49314) 2024-04-06T13:48:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52d77fe700 1 -- 172.21.15.189:0/962201852 learned_addr learned my addr 172.21.15.189:0/962201852 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52d67fc700 1 --2- 172.21.15.189:0/962201852 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52d815db10 0x7f52d80b6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52d6ffd700 1 --2- 172.21.15.189:0/962201852 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52d8143470 0x7f52d80cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52d77fe700 1 -- 172.21.15.189:0/962201852 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52d815db10 msgr2=0x7f52d80b6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:30.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52d77fe700 1 --2- 172.21.15.189:0/962201852 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52d815db10 0x7f52d80b6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52d77fe700 1 -- 172.21.15.189:0/962201852 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52d8143470 msgr2=0x7f52d80cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:30.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52d77fe700 1 --2- 172.21.15.189:0/962201852 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52d8143470 0x7f52d80cda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52d77fe700 1 -- 172.21.15.189:0/962201852 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d0009a70 con 0x7f52d8160b60 2024-04-06T13:48:30.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52d67fc700 1 --2- 172.21.15.189:0/962201852 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52d815db10 0x7f52d80b6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:30.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52d77fe700 1 --2- 172.21.15.189:0/962201852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52d8160b60 0x7f52d815ce00 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f52d0004000 tx=0x7f52d0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:30.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52bf7fe700 1 -- 172.21.15.189:0/962201852 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d0004600 con 0x7f52d8160b60 2024-04-06T13:48:30.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52bf7fe700 1 -- 172.21.15.189:0/962201852 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d0004780 con 0x7f52d8160b60 2024-04-06T13:48:30.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52dd298700 1 -- 172.21.15.189:0/962201852 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d80b54d0 con 0x7f52d8160b60 2024-04-06T13:48:30.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52bf7fe700 1 -- 172.21.15.189:0/962201852 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d001d6b0 con 0x7f52d8160b60 2024-04-06T13:48:30.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.323+0000 7f52dd298700 1 -- 172.21.15.189:0/962201852 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d80b5a90 con 0x7f52d8160b60 2024-04-06T13:48:30.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.324+0000 7f52dd298700 1 -- 172.21.15.189:0/962201852 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d8074990 con 0x7f52d8160b60 2024-04-06T13:48:30.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.325+0000 7f52bf7fe700 1 -- 172.21.15.189:0/962201852 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f52d001d810 con 0x7f52d8160b60 2024-04-06T13:48:30.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.325+0000 7f52bf7fe700 1 --2- 172.21.15.189:0/962201852 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f52c406aed0 0x7f52c406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:30.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.325+0000 7f52bf7fe700 1 -- 172.21.15.189:0/962201852 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f52d008b6d0 con 0x7f52d8160b60 2024-04-06T13:48:30.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.325+0000 7f52d6ffd700 1 --2- 172.21.15.189:0/962201852 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f52c406aed0 0x7f52c406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:30.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.326+0000 7f52d6ffd700 1 --2- 172.21.15.189:0/962201852 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f52c406aed0 0x7f52c406d390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0015b0 tx=0x7f52cc009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:30.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.330+0000 7f52bf7fe700 1 -- 172.21.15.189:0/962201852 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f52d0059dd0 con 0x7f52d8160b60 2024-04-06T13:48:30.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.565+0000 7f52dd298700 1 -- 172.21.15.189:0/962201852 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f52d80b5f40 con 0x7f52d8160b60 2024-04-06T13:48:30.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.565+0000 7f52bf7fe700 1 -- 172.21.15.189:0/962201852 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v116) v1 ==== 53+0+73 (secure 0 0 0) 0x7f52d0059960 con 0x7f52d8160b60 2024-04-06T13:48:30.565 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-06T13:48:30.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.567+0000 7f52dd298700 1 -- 172.21.15.189:0/962201852 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f52c406aed0 msgr2=0x7f52c406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:30.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.567+0000 7f52dd298700 1 --2- 172.21.15.189:0/962201852 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f52c406aed0 0x7f52c406d390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0015b0 tx=0x7f52cc009040 comp rx=0 tx=0).stop 2024-04-06T13:48:30.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.567+0000 7f52dd298700 1 -- 172.21.15.189:0/962201852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52d8160b60 msgr2=0x7f52d815ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:30.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.567+0000 7f52dd298700 1 --2- 172.21.15.189:0/962201852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52d8160b60 0x7f52d815ce00 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f52d0004000 tx=0x7f52d0004ad0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.567+0000 7f52dd298700 1 -- 172.21.15.189:0/962201852 shutdown_connections 2024-04-06T13:48:30.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.567+0000 7f52dd298700 1 --2- 172.21.15.189:0/962201852 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f52c406aed0 0x7f52c406d390 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.567+0000 7f52dd298700 1 --2- 172.21.15.189:0/962201852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52d8160b60 0x7f52d815ce00 unknown :-1 s=CLOSED pgs=2703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.567+0000 7f52dd298700 1 --2- 172.21.15.189:0/962201852 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52d8143470 0x7f52d80cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.567+0000 7f52dd298700 1 --2- 172.21.15.189:0/962201852 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52d815db10 0x7f52d80b6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:30.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.567+0000 7f52dd298700 1 -- 172.21.15.189:0/962201852 >> 172.21.15.189:0/962201852 conn(0x7f52d806a730 msgr2=0x7f52d806bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:30.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.567+0000 7f52dd298700 1 -- 172.21.15.189:0/962201852 shutdown_connections 2024-04-06T13:48:30.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:30.567+0000 7f52dd298700 1 -- 172.21.15.189:0/962201852 wait complete. 2024-04-06T13:48:30.579 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-06T13:48:30.580 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T13:48:30.580 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-06T13:48:30.580 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-06T13:48:30.580 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T13:48:30.580 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-06T13:48:30.580 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-06T13:48:30.580 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T13:48:30.580 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:48:30.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:48:30.580 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-06T13:48:30.598 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T13:48:30.598 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T13:48:30.598 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T13:48:30.599 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:48:30.599 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T13:48:30.599 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:48:30.599 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:48:30.599 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:48:30.599 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-06T13:48:30.599 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:48:30.599 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:48:30.599 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-06T13:48:30.599 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:48:30.599 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:48:30.599 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T13:48:30.599 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:48:30.599 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T13:48:30.600 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 36579sec preferred_lft 36579sec 2024-04-06T13:48:30.600 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::8592:df2c:92e2:dace/64 scope link noprefixroute 2024-04-06T13:48:30.600 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:48:30.600 INFO:teuthology.orchestra.run.smithi189.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T13:48:30.600 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff 2024-04-06T13:48:30.600 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-06T13:48:30.600 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:48:30.600 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::88e:b6ff:fe6f:1cb0/64 scope link 2024-04-06T13:48:30.600 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:48:30.600 INFO:teuthology.orchestra.run.smithi189.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T13:48:30.600 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 62:a5:8d:c5:1a:9c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T13:48:30.600 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::60a5:8dff:fec5:1a9c/64 scope link 2024-04-06T13:48:30.601 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:48:30.601 INFO:teuthology.orchestra.run.smithi189.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T13:48:30.601 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-06T13:48:30.601 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::1c7e:c6ff:febb:1dab/64 scope link 2024-04-06T13:48:30.601 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:48:30.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:48:30.601 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T13:48:30.665 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-06T13:48:30.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:48:30.667 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-06T13:48:30.726 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-06T13:48:30.727 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T13:48:30.727 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T13:48:30.727 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T13:48:30.727 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T13:48:30.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:48:30.727 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T13:48:30.727 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T13:48:30.727 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T13:48:30.728 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T13:48:30.728 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T13:48:30.728 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T13:48:30.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:48:30.832 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T13:48:30.850 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-06T13:48:30.851 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-06T13:48:30.851 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:48:30.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:48:30.851 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-04-06T13:48:30.918 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T13:48:30.918 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-06T13:48:30.976 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:48:30.976 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T13:48:30.977 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-06T13:48:31.034 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T13:48:31.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:48:31.034 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:48:31.120 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:48:31.120 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:48:31.120 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:48:31.128 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:48:31.129 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:48:31.129 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:48:31.129 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T13:48:31.129 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T13:48:31.148 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:48:31.149 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:48:31.149 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:48:31.149 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T13:48:31.149 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T13:48:31.150 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790" 2024-04-06T13:48:31.150 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2024-04-06T13:48:31.150 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e30041de-e9a3-4351-8ca7-29d7f3e3728c" will pass to kernel 2024-04-06T13:48:31.152 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T13:48:31.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:48:31.153 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-06T13:48:31.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:48:31.182 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T13:48:31.271 INFO:teuthology.orchestra.run.smithi189.stdout:client9509 2024-04-06T13:48:31.271 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:48:31.271 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:48:31.290 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-04-06T13:48:31.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.457+0000 7fd84db41700 1 Processor -- start 2024-04-06T13:48:31.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.457+0000 7fd84db41700 1 -- start start 2024-04-06T13:48:31.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.458+0000 7fd84db41700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd84815a550 0x7fd8480cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:31.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.458+0000 7fd84db41700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd8480cf900 0x7fd8480b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:31.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.458+0000 7fd84db41700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8480cdbe0 con 0x7fd8480b4ba0 2024-04-06T13:48:31.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.458+0000 7fd84db41700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84815a940 con 0x7fd84815a550 2024-04-06T13:48:31.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.458+0000 7fd84db41700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84815ac00 con 0x7fd8480cf900 2024-04-06T13:48:31.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.458+0000 7fd8477fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd84815a550 0x7fd8480cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:31.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.458+0000 7fd847fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd8480b4ba0 0x7fd84806eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55798/0 (socket says 172.21.15.189:55798) 2024-04-06T13:48:31.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.458+0000 7fd8477fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd84815a550 0x7fd8480cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49324/0 (socket says 172.21.15.189:49324) 2024-04-06T13:48:31.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.458+0000 7fd847fff700 1 -- 172.21.15.189:0/1486110702 learned_addr learned my addr 172.21.15.189:0/1486110702 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:31.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.458+0000 7fd846ffd700 1 --2- 172.21.15.189:0/1486110702 >> v2:172.21.15.192:3300/0 conn(0x7fd8480cf900 0x7fd8480b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:31.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.458+0000 7fd8477fe700 1 -- 172.21.15.189:0/1486110702 >> v2:172.21.15.192:3300/0 conn(0x7fd8480cf900 msgr2=0x7fd8480b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:31.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.458+0000 7fd8477fe700 1 --2- 172.21.15.189:0/1486110702 >> v2:172.21.15.192:3300/0 conn(0x7fd8480cf900 0x7fd8480b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:31.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.458+0000 7fd8477fe700 1 -- 172.21.15.189:0/1486110702 >> v1:172.21.15.189:6789/0 conn(0x7fd8480b4ba0 legacy=0x7fd84806eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:31.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.458+0000 7fd8477fe700 1 -- 172.21.15.189:0/1486110702 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84815aee0 con 0x7fd84815a550 2024-04-06T13:48:31.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.458+0000 7fd8477fe700 1 --2- 172.21.15.189:0/1486110702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84815a550 0x7fd8480cf3c0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7fd8480b2d40 tx=0x7fd83c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b3feab4b5cdd35 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:31.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.459+0000 7fd8467fc700 1 -- 172.21.15.189:0/1486110702 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c011830 con 0x7fd84815a550 2024-04-06T13:48:31.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.459+0000 7fd8467fc700 1 -- 172.21.15.189:0/1486110702 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83c011990 con 0x7fd84815a550 2024-04-06T13:48:31.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.459+0000 7fd8467fc700 1 -- 172.21.15.189:0/1486110702 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c011d20 con 0x7fd84815a550 2024-04-06T13:48:31.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.459+0000 7fd84db41700 1 -- 172.21.15.189:0/1486110702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84815a550 msgr2=0x7fd8480cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:31.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.459+0000 7fd84db41700 1 --2- 172.21.15.189:0/1486110702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84815a550 0x7fd8480cf3c0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7fd8480b2d40 tx=0x7fd83c009b20 comp rx=0 tx=0).stop 2024-04-06T13:48:31.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.459+0000 7fd84db41700 1 -- 172.21.15.189:0/1486110702 shutdown_connections 2024-04-06T13:48:31.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.459+0000 7fd84db41700 1 --2- 172.21.15.189:0/1486110702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84815a550 0x7fd8480cf3c0 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:31.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.459+0000 7fd84db41700 1 --2- 172.21.15.189:0/1486110702 >> v2:172.21.15.192:3300/0 conn(0x7fd8480cf900 0x7fd8480b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:31.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.459+0000 7fd84db41700 1 -- 172.21.15.189:0/1486110702 >> 172.21.15.189:0/1486110702 conn(0x7fd8480d00b0 msgr2=0x7fd8480d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:31.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.459+0000 7fd84db41700 1 -- 172.21.15.189:0/1486110702 shutdown_connections 2024-04-06T13:48:31.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.459+0000 7fd84db41700 1 -- 172.21.15.189:0/1486110702 wait complete. 2024-04-06T13:48:31.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.460+0000 7fd84db41700 1 Processor -- start 2024-04-06T13:48:31.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.461+0000 7fd84db41700 1 -- start start 2024-04-06T13:48:31.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.461+0000 7fd84db41700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8480b4ba0 0x7fd8481f2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:31.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.461+0000 7fd84db41700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8480cf900 0x7fd8480c50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:31.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.461+0000 7fd84db41700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd84814fb20 0x7fd84815bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:31.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.461+0000 7fd84db41700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8481f3220 con 0x7fd8480b4ba0 2024-04-06T13:48:31.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.461+0000 7fd84db41700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8481f3500 con 0x7fd8480cf900 2024-04-06T13:48:31.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.461+0000 7fd84db41700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8481f37e0 con 0x7fd84814fb20 2024-04-06T13:48:31.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.461+0000 7fd8477fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8480b4ba0 0x7fd8481f2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:31.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.461+0000 7fd8477fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8480b4ba0 0x7fd8481f2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49334/0 (socket says 172.21.15.189:49334) 2024-04-06T13:48:31.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.461+0000 7fd8477fe700 1 -- 172.21.15.189:0/3756191 learned_addr learned my addr 172.21.15.189:0/3756191 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:31.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.462+0000 7fd847fff700 1 --2- 172.21.15.189:0/3756191 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd84814fb20 0x7fd84815bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:31.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.462+0000 7fd846ffd700 1 --2- 172.21.15.189:0/3756191 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8480cf900 0x7fd8480c50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:31.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.462+0000 7fd8477fe700 1 -- 172.21.15.189:0/3756191 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd84814fb20 msgr2=0x7fd84815bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:31.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.462+0000 7fd8477fe700 1 --2- 172.21.15.189:0/3756191 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd84814fb20 0x7fd84815bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:31.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.462+0000 7fd8477fe700 1 -- 172.21.15.189:0/3756191 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8480cf900 msgr2=0x7fd8480c50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:31.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.462+0000 7fd8477fe700 1 --2- 172.21.15.189:0/3756191 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8480cf900 0x7fd8480c50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:31.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.462+0000 7fd8477fe700 1 -- 172.21.15.189:0/3756191 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83c0094a0 con 0x7fd8480b4ba0 2024-04-06T13:48:31.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.462+0000 7fd847fff700 1 --2- 172.21.15.189:0/3756191 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd84814fb20 0x7fd84815bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:31.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.462+0000 7fd846ffd700 1 --2- 172.21.15.189:0/3756191 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8480cf900 0x7fd8480c50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:31.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.462+0000 7fd8477fe700 1 --2- 172.21.15.189:0/3756191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8480b4ba0 0x7fd8481f2c40 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7fd83c000c00 tx=0x7fd83c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:31.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.462+0000 7fd82ffff700 1 -- 172.21.15.189:0/3756191 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0036a0 con 0x7fd8480b4ba0 2024-04-06T13:48:31.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.462+0000 7fd82ffff700 1 -- 172.21.15.189:0/3756191 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83c01e070 con 0x7fd8480b4ba0 2024-04-06T13:48:31.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.462+0000 7fd84db41700 1 -- 172.21.15.189:0/3756191 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8481f3ac0 con 0x7fd8480b4ba0 2024-04-06T13:48:31.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.462+0000 7fd82ffff700 1 -- 172.21.15.189:0/3756191 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c003a20 con 0x7fd8480b4ba0 2024-04-06T13:48:31.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.462+0000 7fd84db41700 1 -- 172.21.15.189:0/3756191 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8480c5630 con 0x7fd8480b4ba0 2024-04-06T13:48:31.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.463+0000 7fd84db41700 1 -- 172.21.15.189:0/3756191 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8481f64f0 con 0x7fd8480b4ba0 2024-04-06T13:48:31.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.463+0000 7fd82ffff700 1 -- 172.21.15.189:0/3756191 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fd83c003b80 con 0x7fd8480b4ba0 2024-04-06T13:48:31.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.464+0000 7fd82ffff700 1 --2- 172.21.15.189:0/3756191 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd83406aed0 0x7fd83406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:31.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.464+0000 7fd82ffff700 1 -- 172.21.15.189:0/3756191 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd83c08a940 con 0x7fd8480b4ba0 2024-04-06T13:48:31.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.464+0000 7fd846ffd700 1 --2- 172.21.15.189:0/3756191 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd83406aed0 0x7fd83406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:31.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.464+0000 7fd846ffd700 1 --2- 172.21.15.189:0/3756191 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd83406aed0 0x7fd83406d390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd8480cca10 tx=0x7fd838008040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:31.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.468+0000 7fd82ffff700 1 -- 172.21.15.189:0/3756191 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd8481f64f0 con 0x7fd8480b4ba0 2024-04-06T13:48:31.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.708+0000 7fd84db41700 1 -- 172.21.15.189:0/3756191 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd8481f64f0 con 0x7fd8480b4ba0 2024-04-06T13:48:31.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.762+0000 7fd82ffff700 1 -- 172.21.15.189:0/3756191 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v117) v1 ==== 136+0+0 (secure 0 0 0) 0x7fd8481f64f0 con 0x7fd8480b4ba0 2024-04-06T13:48:31.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.765+0000 7fd84db41700 1 -- 172.21.15.189:0/3756191 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd83406aed0 msgr2=0x7fd83406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:31.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.765+0000 7fd84db41700 1 --2- 172.21.15.189:0/3756191 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd83406aed0 0x7fd83406d390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd8480cca10 tx=0x7fd838008040 comp rx=0 tx=0).stop 2024-04-06T13:48:31.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.765+0000 7fd84db41700 1 -- 172.21.15.189:0/3756191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8480b4ba0 msgr2=0x7fd8481f2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:31.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.765+0000 7fd84db41700 1 --2- 172.21.15.189:0/3756191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8480b4ba0 0x7fd8481f2c40 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7fd83c000c00 tx=0x7fd83c004750 comp rx=0 tx=0).stop 2024-04-06T13:48:31.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.765+0000 7fd84db41700 1 -- 172.21.15.189:0/3756191 shutdown_connections 2024-04-06T13:48:31.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.765+0000 7fd84db41700 1 --2- 172.21.15.189:0/3756191 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd83406aed0 0x7fd83406d390 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:31.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.765+0000 7fd84db41700 1 --2- 172.21.15.189:0/3756191 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd84814fb20 0x7fd84815bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:31.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.765+0000 7fd84db41700 1 --2- 172.21.15.189:0/3756191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8480b4ba0 0x7fd8481f2c40 unknown :-1 s=CLOSED pgs=2706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:31.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.765+0000 7fd84db41700 1 --2- 172.21.15.189:0/3756191 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8480cf900 0x7fd8480c50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:31.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.765+0000 7fd84db41700 1 -- 172.21.15.189:0/3756191 >> 172.21.15.189:0/3756191 conn(0x7fd8480d00b0 msgr2=0x7fd8480b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:31.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.765+0000 7fd84db41700 1 -- 172.21.15.189:0/3756191 shutdown_connections 2024-04-06T13:48:31.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.765+0000 7fd84db41700 1 -- 172.21.15.189:0/3756191 wait complete. 2024-04-06T13:48:31.778 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:48:31.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.901+0000 7fd57e0eb700 1 Processor -- start 2024-04-06T13:48:31.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.901+0000 7fd57e0eb700 1 -- start start 2024-04-06T13:48:31.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.901+0000 7fd57e0eb700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd57815d7b0 0x7fd5780b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:31.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.901+0000 7fd57e0eb700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd57806a730 0x7fd57806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:31.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.901+0000 7fd57e0eb700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5780b3990 con 0x7fd57815ddf0 2024-04-06T13:48:31.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.901+0000 7fd57e0eb700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd57815e950 con 0x7fd57815d7b0 2024-04-06T13:48:31.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.901+0000 7fd57e0eb700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd57806b0f0 con 0x7fd57806a730 2024-04-06T13:48:31.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.902+0000 7fd5777fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd57815d7b0 0x7fd5780b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:31.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.902+0000 7fd5777fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd57815d7b0 0x7fd5780b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49340/0 (socket says 172.21.15.189:49340) 2024-04-06T13:48:31.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.902+0000 7fd5777fe700 1 -- 172.21.15.189:0/2528496835 learned_addr learned my addr 172.21.15.189:0/2528496835 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:31.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.902+0000 7fd5777fe700 1 -- 172.21.15.189:0/2528496835 >> v2:172.21.15.192:3300/0 conn(0x7fd57806a730 msgr2=0x7fd57806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:31.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.902+0000 7fd5777fe700 1 --2- 172.21.15.189:0/2528496835 >> v2:172.21.15.192:3300/0 conn(0x7fd57806a730 0x7fd57806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:31.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.902+0000 7fd5777fe700 1 -- 172.21.15.189:0/2528496835 >> v1:172.21.15.192:6789/0 conn(0x7fd57815ddf0 legacy=0x7fd57815e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:31.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.902+0000 7fd5777fe700 1 -- 172.21.15.189:0/2528496835 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5780b3060 con 0x7fd57815d7b0 2024-04-06T13:48:31.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.902+0000 7fd5777fe700 1 --2- 172.21.15.189:0/2528496835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd57815d7b0 0x7fd5780b3e20 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7fd5780afc00 tx=0x7fd568009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3229385817c00830 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:31.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.902+0000 7fd5767fc700 1 -- 172.21.15.189:0/2528496835 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568004230 con 0x7fd57815d7b0 2024-04-06T13:48:31.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.902+0000 7fd5767fc700 1 -- 172.21.15.189:0/2528496835 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd568004390 con 0x7fd57815d7b0 2024-04-06T13:48:31.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.902+0000 7fd5767fc700 1 -- 172.21.15.189:0/2528496835 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5680036a0 con 0x7fd57815d7b0 2024-04-06T13:48:31.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.903+0000 7fd57e0eb700 1 -- 172.21.15.189:0/2528496835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd57815d7b0 msgr2=0x7fd5780b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:31.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.903+0000 7fd57e0eb700 1 --2- 172.21.15.189:0/2528496835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd57815d7b0 0x7fd5780b3e20 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7fd5780afc00 tx=0x7fd568009bd0 comp rx=0 tx=0).stop 2024-04-06T13:48:31.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.903+0000 7fd57e0eb700 1 -- 172.21.15.189:0/2528496835 shutdown_connections 2024-04-06T13:48:31.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.903+0000 7fd57e0eb700 1 --2- 172.21.15.189:0/2528496835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd57815d7b0 0x7fd5780b3e20 unknown :-1 s=CLOSED pgs=2707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:31.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.903+0000 7fd57e0eb700 1 --2- 172.21.15.189:0/2528496835 >> v2:172.21.15.192:3300/0 conn(0x7fd57806a730 0x7fd57806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:31.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.903+0000 7fd57e0eb700 1 -- 172.21.15.189:0/2528496835 >> 172.21.15.189:0/2528496835 conn(0x7fd5780d00b0 msgr2=0x7fd5780d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:31.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.903+0000 7fd57e0eb700 1 -- 172.21.15.189:0/2528496835 shutdown_connections 2024-04-06T13:48:31.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.903+0000 7fd57e0eb700 1 -- 172.21.15.189:0/2528496835 wait complete. 2024-04-06T13:48:31.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.904+0000 7fd57e0eb700 1 Processor -- start 2024-04-06T13:48:31.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.904+0000 7fd57e0eb700 1 -- start start 2024-04-06T13:48:31.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.905+0000 7fd57e0eb700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd57806a730 0x7fd57814fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:31.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.905+0000 7fd57e0eb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd57815d7b0 0x7fd5780cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:31.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.905+0000 7fd57e0eb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd57815ddf0 0x7fd57814ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:31.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.905+0000 7fd57e0eb700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5781f3ee0 con 0x7fd57815ddf0 2024-04-06T13:48:31.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.905+0000 7fd57e0eb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57815b5d0 con 0x7fd57815d7b0 2024-04-06T13:48:31.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.905+0000 7fd57e0eb700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57815b8b0 con 0x7fd57806a730 2024-04-06T13:48:31.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.905+0000 7fd577fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd57815ddf0 0x7fd57814ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:31.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.905+0000 7fd577fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd57815ddf0 0x7fd57814ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49356/0 (socket says 172.21.15.189:49356) 2024-04-06T13:48:31.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.905+0000 7fd577fff700 1 -- 172.21.15.189:0/312812851 learned_addr learned my addr 172.21.15.189:0/312812851 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:31.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.905+0000 7fd576ffd700 1 --2- 172.21.15.189:0/312812851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd57815d7b0 0x7fd5780cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:31.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.905+0000 7fd5777fe700 1 --2- 172.21.15.189:0/312812851 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd57806a730 0x7fd57814fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:31.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.905+0000 7fd577fff700 1 -- 172.21.15.189:0/312812851 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd57806a730 msgr2=0x7fd57814fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:31.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.905+0000 7fd577fff700 1 --2- 172.21.15.189:0/312812851 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd57806a730 0x7fd57814fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:31.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.905+0000 7fd577fff700 1 -- 172.21.15.189:0/312812851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd57815d7b0 msgr2=0x7fd5780cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:31.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.905+0000 7fd577fff700 1 --2- 172.21.15.189:0/312812851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd57815d7b0 0x7fd5780cce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:31.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.905+0000 7fd577fff700 1 -- 172.21.15.189:0/312812851 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd568009550 con 0x7fd57815ddf0 2024-04-06T13:48:31.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.905+0000 7fd5777fe700 1 --2- 172.21.15.189:0/312812851 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd57806a730 0x7fd57814fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:31.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.905+0000 7fd577fff700 1 --2- 172.21.15.189:0/312812851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd57815ddf0 0x7fd57814ed50 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7fd57000b990 tx=0x7fd57000bca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:31.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.906+0000 7fd55ffff700 1 -- 172.21.15.189:0/312812851 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57000c930 con 0x7fd57815ddf0 2024-04-06T13:48:31.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.906+0000 7fd55ffff700 1 -- 172.21.15.189:0/312812851 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57000ca90 con 0x7fd57815ddf0 2024-04-06T13:48:31.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.906+0000 7fd57e0eb700 1 -- 172.21.15.189:0/312812851 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57815bb90 con 0x7fd57815ddf0 2024-04-06T13:48:31.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.906+0000 7fd55ffff700 1 -- 172.21.15.189:0/312812851 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57000ce40 con 0x7fd57815ddf0 2024-04-06T13:48:31.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.906+0000 7fd57e0eb700 1 -- 172.21.15.189:0/312812851 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5780cd390 con 0x7fd57815ddf0 2024-04-06T13:48:31.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.907+0000 7fd57e0eb700 1 -- 172.21.15.189:0/312812851 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd578067bb0 con 0x7fd57815ddf0 2024-04-06T13:48:31.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.907+0000 7fd55ffff700 1 -- 172.21.15.189:0/312812851 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fd570006d50 con 0x7fd57815ddf0 2024-04-06T13:48:31.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.907+0000 7fd55ffff700 1 --2- 172.21.15.189:0/312812851 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd56406aed0 0x7fd56406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:31.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.908+0000 7fd55ffff700 1 -- 172.21.15.189:0/312812851 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd570089310 con 0x7fd57815ddf0 2024-04-06T13:48:31.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.908+0000 7fd5777fe700 1 --2- 172.21.15.189:0/312812851 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd56406aed0 0x7fd56406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:31.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.908+0000 7fd5777fe700 1 --2- 172.21.15.189:0/312812851 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd56406aed0 0x7fd56406d390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd568009130 tx=0x7fd568008030 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:31.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:31.912+0000 7fd55ffff700 1 -- 172.21.15.189:0/312812851 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd570054010 con 0x7fd57815ddf0 2024-04-06T13:48:32.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.109+0000 7fd57e0eb700 1 -- 172.21.15.189:0/312812851 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd578074990 con 0x7fd57815ddf0 2024-04-06T13:48:32.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.110+0000 7fd55ffff700 1 -- 172.21.15.189:0/312812851 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v248) v1 ==== 74+0+16480 (secure 0 0 0) 0x7fd570057630 con 0x7fd57815ddf0 2024-04-06T13:48:32.110 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:32.110 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":248,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:48:20.676026+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:48:17.447247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-04-06T13:48:18.646229+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:48:32.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.112+0000 7fd57e0eb700 1 -- 172.21.15.189:0/312812851 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd56406aed0 msgr2=0x7fd56406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.112+0000 7fd57e0eb700 1 --2- 172.21.15.189:0/312812851 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd56406aed0 0x7fd56406d390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd568009130 tx=0x7fd568008030 comp rx=0 tx=0).stop 2024-04-06T13:48:32.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.112+0000 7fd57e0eb700 1 -- 172.21.15.189:0/312812851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd57815ddf0 msgr2=0x7fd57814ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.112+0000 7fd57e0eb700 1 --2- 172.21.15.189:0/312812851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd57815ddf0 0x7fd57814ed50 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7fd57000b990 tx=0x7fd57000bca0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.112+0000 7fd57e0eb700 1 -- 172.21.15.189:0/312812851 shutdown_connections 2024-04-06T13:48:32.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.112+0000 7fd57e0eb700 1 --2- 172.21.15.189:0/312812851 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd56406aed0 0x7fd56406d390 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.112+0000 7fd57e0eb700 1 --2- 172.21.15.189:0/312812851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd57815ddf0 0x7fd57814ed50 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.112+0000 7fd57e0eb700 1 --2- 172.21.15.189:0/312812851 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd57806a730 0x7fd57814fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.112+0000 7fd57e0eb700 1 --2- 172.21.15.189:0/312812851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd57815d7b0 0x7fd5780cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.112+0000 7fd57e0eb700 1 -- 172.21.15.189:0/312812851 >> 172.21.15.189:0/312812851 conn(0x7fd5780d00b0 msgr2=0x7fd57806eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:32.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.112+0000 7fd57e0eb700 1 -- 172.21.15.189:0/312812851 shutdown_connections 2024-04-06T13:48:32.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.112+0000 7fd57e0eb700 1 -- 172.21.15.189:0/312812851 wait complete. 2024-04-06T13:48:32.125 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T13:48:17.447247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '247', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T13:48:18.646229+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:48:32.125 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:48:32.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.251+0000 7f42fedbf700 1 Processor -- start 2024-04-06T13:48:32.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.251+0000 7f42fedbf700 1 -- start start 2024-04-06T13:48:32.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.251+0000 7f42fedbf700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f42f815a550 0x7f42f80cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:32.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.252+0000 7f42fedbf700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f42f80b4d70 0x7f42f80b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:32.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.252+0000 7f42fedbf700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f80cdbe0 con 0x7f42f80cf900 2024-04-06T13:48:32.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.252+0000 7f42fedbf700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f815a940 con 0x7f42f815a550 2024-04-06T13:48:32.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.252+0000 7f42fedbf700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f815ac00 con 0x7f42f80b4d70 2024-04-06T13:48:32.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.252+0000 7f42fcb5b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f42f815a550 0x7f42f80cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:32.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.252+0000 7f42fcb5b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f42f815a550 0x7f42f80cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49360/0 (socket says 172.21.15.189:49360) 2024-04-06T13:48:32.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.252+0000 7f42fcb5b700 1 -- 172.21.15.189:0/3865746493 learned_addr learned my addr 172.21.15.189:0/3865746493 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:32.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.252+0000 7f42fd35c700 1 --2- 172.21.15.189:0/3865746493 >> v2:172.21.15.192:3300/0 conn(0x7f42f80b4d70 0x7f42f80b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:32.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.252+0000 7f42fcb5b700 1 -- 172.21.15.189:0/3865746493 >> v2:172.21.15.192:3300/0 conn(0x7f42f80b4d70 msgr2=0x7f42f80b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:32.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.252+0000 7f42fcb5b700 1 --2- 172.21.15.189:0/3865746493 >> v2:172.21.15.192:3300/0 conn(0x7f42f80b4d70 0x7f42f80b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.252+0000 7f42fcb5b700 1 -- 172.21.15.189:0/3865746493 >> v1:172.21.15.192:6789/0 conn(0x7f42f80cf900 legacy=0x7f42f80b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.252+0000 7f42fcb5b700 1 -- 172.21.15.189:0/3865746493 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f815aee0 con 0x7f42f815a550 2024-04-06T13:48:32.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.252+0000 7f42fcb5b700 1 --2- 172.21.15.189:0/3865746493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42f815a550 0x7f42f80cf3c0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f42f8072830 tx=0x7f42ec00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=154ca7ef6218713b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:32.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.252+0000 7f42f77fe700 1 -- 172.21.15.189:0/3865746493 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec00dc70 con 0x7f42f815a550 2024-04-06T13:48:32.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.252+0000 7f42f77fe700 1 -- 172.21.15.189:0/3865746493 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ec00ddd0 con 0x7f42f815a550 2024-04-06T13:48:32.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.252+0000 7f42f77fe700 1 -- 172.21.15.189:0/3865746493 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec010690 con 0x7f42f815a550 2024-04-06T13:48:32.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.253+0000 7f42fedbf700 1 -- 172.21.15.189:0/3865746493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42f815a550 msgr2=0x7f42f80cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.253+0000 7f42fedbf700 1 --2- 172.21.15.189:0/3865746493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42f815a550 0x7f42f80cf3c0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f42f8072830 tx=0x7f42ec00be70 comp rx=0 tx=0).stop 2024-04-06T13:48:32.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.253+0000 7f42fedbf700 1 -- 172.21.15.189:0/3865746493 shutdown_connections 2024-04-06T13:48:32.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.253+0000 7f42fedbf700 1 --2- 172.21.15.189:0/3865746493 >> v2:172.21.15.192:3300/0 conn(0x7f42f80b4d70 0x7f42f80b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.253+0000 7f42fedbf700 1 --2- 172.21.15.189:0/3865746493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42f815a550 0x7f42f80cf3c0 unknown :-1 s=CLOSED pgs=2709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.253+0000 7f42fedbf700 1 -- 172.21.15.189:0/3865746493 >> 172.21.15.189:0/3865746493 conn(0x7f42f80d00b0 msgr2=0x7f42f80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:32.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.253+0000 7f42fedbf700 1 -- 172.21.15.189:0/3865746493 shutdown_connections 2024-04-06T13:48:32.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.253+0000 7f42fedbf700 1 -- 172.21.15.189:0/3865746493 wait complete. 2024-04-06T13:48:32.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.254+0000 7f42fedbf700 1 Processor -- start 2024-04-06T13:48:32.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.254+0000 7f42fedbf700 1 -- start start 2024-04-06T13:48:32.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.255+0000 7f42fedbf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42f80b4d70 0x7f42f8154110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:32.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.255+0000 7f42fedbf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42f80cf900 0x7f42f80a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:32.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.255+0000 7f42fedbf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42f81529e0 0x7f42f8148630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:32.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.255+0000 7f42fedbf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f81578f0 con 0x7f42f80cf900 2024-04-06T13:48:32.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.255+0000 7f42fedbf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f8154960 con 0x7f42f80b4d70 2024-04-06T13:48:32.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.255+0000 7f42fedbf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f8154c40 con 0x7f42f81529e0 2024-04-06T13:48:32.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.255+0000 7f42f7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42f80cf900 0x7f42f80a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:32.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.255+0000 7f42f7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42f80cf900 0x7f42f80a8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49372/0 (socket says 172.21.15.189:49372) 2024-04-06T13:48:32.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.255+0000 7f42f7fff700 1 -- 172.21.15.189:0/2315733634 learned_addr learned my addr 172.21.15.189:0/2315733634 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:32.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.255+0000 7f42fd35c700 1 --2- 172.21.15.189:0/2315733634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42f81529e0 0x7f42f8148630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:32.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.255+0000 7f42fcb5b700 1 --2- 172.21.15.189:0/2315733634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42f80b4d70 0x7f42f8154110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:32.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.255+0000 7f42f7fff700 1 -- 172.21.15.189:0/2315733634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42f81529e0 msgr2=0x7f42f8148630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.255+0000 7f42f7fff700 1 --2- 172.21.15.189:0/2315733634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42f81529e0 0x7f42f8148630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.255+0000 7f42f7fff700 1 -- 172.21.15.189:0/2315733634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42f80b4d70 msgr2=0x7f42f8154110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.255+0000 7f42f7fff700 1 --2- 172.21.15.189:0/2315733634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42f80b4d70 0x7f42f8154110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.255+0000 7f42f7fff700 1 -- 172.21.15.189:0/2315733634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec00b7f0 con 0x7f42f80cf900 2024-04-06T13:48:32.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.256+0000 7f42f7fff700 1 --2- 172.21.15.189:0/2315733634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42f80cf900 0x7f42f80a8aa0 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7f42e400c4e0 tx=0x7f42e400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:32.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.256+0000 7f42f57fa700 1 -- 172.21.15.189:0/2315733634 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e40052d0 con 0x7f42f80cf900 2024-04-06T13:48:32.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.256+0000 7f42f57fa700 1 -- 172.21.15.189:0/2315733634 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e4005430 con 0x7f42f80cf900 2024-04-06T13:48:32.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.256+0000 7f42fedbf700 1 -- 172.21.15.189:0/2315733634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f8154f20 con 0x7f42f80cf900 2024-04-06T13:48:32.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.256+0000 7f42f57fa700 1 -- 172.21.15.189:0/2315733634 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e40038c0 con 0x7f42f80cf900 2024-04-06T13:48:32.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.256+0000 7f42fedbf700 1 -- 172.21.15.189:0/2315733634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f8154650 con 0x7f42f80cf900 2024-04-06T13:48:32.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.257+0000 7f42fedbf700 1 -- 172.21.15.189:0/2315733634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f8157dd0 con 0x7f42f80cf900 2024-04-06T13:48:32.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.257+0000 7f42f57fa700 1 -- 172.21.15.189:0/2315733634 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f42e4003a20 con 0x7f42f80cf900 2024-04-06T13:48:32.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.257+0000 7f42f57fa700 1 --2- 172.21.15.189:0/2315733634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42dc06aed0 0x7f42dc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:32.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.258+0000 7f42f57fa700 1 -- 172.21.15.189:0/2315733634 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f42e4088b60 con 0x7f42f80cf900 2024-04-06T13:48:32.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.258+0000 7f42fcb5b700 1 --2- 172.21.15.189:0/2315733634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42dc06aed0 0x7f42dc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:32.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.258+0000 7f42fcb5b700 1 --2- 172.21.15.189:0/2315733634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42dc06aed0 0x7f42dc06d390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00beb0 tx=0x7f42ec013000 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:32.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.262+0000 7f42f57fa700 1 -- 172.21.15.189:0/2315733634 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f42f8157dd0 con 0x7f42f80cf900 2024-04-06T13:48:32.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.456+0000 7f42fedbf700 1 -- 172.21.15.189:0/2315733634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f42f8157dd0 con 0x7f42f80cf900 2024-04-06T13:48:32.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.457+0000 7f42f57fa700 1 -- 172.21.15.189:0/2315733634 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v248) v1 ==== 93+0+10 (secure 0 0 0) 0x7f42f8157dd0 con 0x7f42f80cf900 2024-04-06T13:48:32.458 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:48:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.459+0000 7f42fedbf700 1 -- 172.21.15.189:0/2315733634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42dc06aed0 msgr2=0x7f42dc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.459+0000 7f42fedbf700 1 --2- 172.21.15.189:0/2315733634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42dc06aed0 0x7f42dc06d390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00beb0 tx=0x7f42ec013000 comp rx=0 tx=0).stop 2024-04-06T13:48:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.459+0000 7f42fedbf700 1 -- 172.21.15.189:0/2315733634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42f80cf900 msgr2=0x7f42f80a8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.459+0000 7f42fedbf700 1 --2- 172.21.15.189:0/2315733634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42f80cf900 0x7f42f80a8aa0 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7f42e400c4e0 tx=0x7f42e400c7f0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.459+0000 7f42fedbf700 1 -- 172.21.15.189:0/2315733634 shutdown_connections 2024-04-06T13:48:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.459+0000 7f42fedbf700 1 --2- 172.21.15.189:0/2315733634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42dc06aed0 0x7f42dc06d390 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.459+0000 7f42fedbf700 1 --2- 172.21.15.189:0/2315733634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42f81529e0 0x7f42f8148630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.459+0000 7f42fedbf700 1 --2- 172.21.15.189:0/2315733634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42f80b4d70 0x7f42f8154110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.459+0000 7f42fedbf700 1 --2- 172.21.15.189:0/2315733634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42f80cf900 0x7f42f80a8aa0 unknown :-1 s=CLOSED pgs=2710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.459+0000 7f42fedbf700 1 -- 172.21.15.189:0/2315733634 >> 172.21.15.189:0/2315733634 conn(0x7f42f80d00b0 msgr2=0x7f42f80b38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:32.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.459+0000 7f42fedbf700 1 -- 172.21.15.189:0/2315733634 shutdown_connections 2024-04-06T13:48:32.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.459+0000 7f42fedbf700 1 -- 172.21.15.189:0/2315733634 wait complete. 2024-04-06T13:48:32.472 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T13:48:32.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.597+0000 7f00b54e8700 1 Processor -- start 2024-04-06T13:48:32.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.597+0000 7f00b54e8700 1 -- start start 2024-04-06T13:48:32.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.598+0000 7f00b54e8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00b00ce730 0x7f00b015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:32.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.598+0000 7f00b54e8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b015eac0 con 0x7f00b00b8d70 2024-04-06T13:48:32.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.598+0000 7f00b54e8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b015e220 con 0x7f00b015d6f0 2024-04-06T13:48:32.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.598+0000 7f00b54e8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00b015e010 con 0x7f00b00ce730 2024-04-06T13:48:32.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.598+0000 7f00af7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f00b00b8d70 0x7f00b00d1e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55802/0 (socket says 172.21.15.189:55802) 2024-04-06T13:48:32.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.598+0000 7f00af7fe700 1 -- 172.21.15.189:0/3658048498 learned_addr learned my addr 172.21.15.189:0/3658048498 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:32.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.598+0000 7f00ae7fc700 1 --2- 172.21.15.189:0/3658048498 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00b00ce730 0x7f00b015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:32.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.598+0000 7f00ae7fc700 1 -- 172.21.15.189:0/3658048498 >> v1:172.21.15.192:6789/0 conn(0x7f00b015d6f0 legacy=0x7f00b00ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.598+0000 7f00ae7fc700 1 -- 172.21.15.189:0/3658048498 >> v1:172.21.15.189:6789/0 conn(0x7f00b00b8d70 legacy=0x7f00b00d1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.598+0000 7f00ae7fc700 1 -- 172.21.15.189:0/3658048498 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b015e500 con 0x7f00b00ce730 2024-04-06T13:48:32.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.599+0000 7f00ae7fc700 1 --2- 172.21.15.189:0/3658048498 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00b00ce730 0x7f00b015dad0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009870 tx=0x7f00a4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16813e2d242d58be server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:32.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.600+0000 7f00adffb700 1 -- 172.21.15.189:0/3658048498 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a4011830 con 0x7f00b00ce730 2024-04-06T13:48:32.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.600+0000 7f00adffb700 1 -- 172.21.15.189:0/3658048498 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a4011990 con 0x7f00b00ce730 2024-04-06T13:48:32.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.600+0000 7f00adffb700 1 -- 172.21.15.189:0/3658048498 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a4011d20 con 0x7f00b00ce730 2024-04-06T13:48:32.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.601+0000 7f00b54e8700 1 -- 172.21.15.189:0/3658048498 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00b00ce730 msgr2=0x7f00b015dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.601+0000 7f00b54e8700 1 --2- 172.21.15.189:0/3658048498 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00b00ce730 0x7f00b015dad0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009870 tx=0x7f00a4009b80 comp rx=0 tx=0).stop 2024-04-06T13:48:32.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.601+0000 7f00b54e8700 1 -- 172.21.15.189:0/3658048498 shutdown_connections 2024-04-06T13:48:32.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.601+0000 7f00b54e8700 1 --2- 172.21.15.189:0/3658048498 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00b00ce730 0x7f00b015dad0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.601+0000 7f00b54e8700 1 -- 172.21.15.189:0/3658048498 >> 172.21.15.189:0/3658048498 conn(0x7f00b0072490 msgr2=0x7f00b006b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:32.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.601+0000 7f00b54e8700 1 -- 172.21.15.189:0/3658048498 shutdown_connections 2024-04-06T13:48:32.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.601+0000 7f00b54e8700 1 -- 172.21.15.189:0/3658048498 wait complete. 2024-04-06T13:48:32.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.602+0000 7f00b54e8700 1 Processor -- start 2024-04-06T13:48:32.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.602+0000 7f00b54e8700 1 -- start start 2024-04-06T13:48:32.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.603+0000 7f00b54e8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00b00b8d70 0x7f00b01f6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:32.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.603+0000 7f00b54e8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b00ce730 0x7f00b00c8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:32.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.603+0000 7f00b54e8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00b015d6f0 0x7f00b00b5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:32.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.603+0000 7f00b54e8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b00c89b0 con 0x7f00b00b8d70 2024-04-06T13:48:32.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.603+0000 7f00b54e8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b00c8c10 con 0x7f00b00ce730 2024-04-06T13:48:32.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.603+0000 7f00b54e8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00b00c8ef0 con 0x7f00b015d6f0 2024-04-06T13:48:32.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.603+0000 7f00aeffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00b00b8d70 0x7f00b01f6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:32.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.603+0000 7f00aeffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00b00b8d70 0x7f00b01f6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49382/0 (socket says 172.21.15.189:49382) 2024-04-06T13:48:32.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.603+0000 7f00aeffd700 1 -- 172.21.15.189:0/1437807719 learned_addr learned my addr 172.21.15.189:0/1437807719 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:32.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.603+0000 7f00af7fe700 1 --2- 172.21.15.189:0/1437807719 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00b015d6f0 0x7f00b00b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:32.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.603+0000 7f00ae7fc700 1 --2- 172.21.15.189:0/1437807719 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b00ce730 0x7f00b00c8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:32.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.603+0000 7f00aeffd700 1 -- 172.21.15.189:0/1437807719 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00b015d6f0 msgr2=0x7f00b00b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.603+0000 7f00aeffd700 1 --2- 172.21.15.189:0/1437807719 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00b015d6f0 0x7f00b00b5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.603+0000 7f00aeffd700 1 -- 172.21.15.189:0/1437807719 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b00ce730 msgr2=0x7f00b00c8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.603+0000 7f00aeffd700 1 --2- 172.21.15.189:0/1437807719 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b00ce730 0x7f00b00c8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.603+0000 7f00aeffd700 1 -- 172.21.15.189:0/1437807719 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a40094a0 con 0x7f00b00b8d70 2024-04-06T13:48:32.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.603+0000 7f00aeffd700 1 --2- 172.21.15.189:0/1437807719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00b00b8d70 0x7f00b01f6c30 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f00a000c450 tx=0x7f00a000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:32.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.603+0000 7f00977fe700 1 -- 172.21.15.189:0/1437807719 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a00052d0 con 0x7f00b00b8d70 2024-04-06T13:48:32.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.604+0000 7f00977fe700 1 -- 172.21.15.189:0/1437807719 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a0005430 con 0x7f00b00b8d70 2024-04-06T13:48:32.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.604+0000 7f00977fe700 1 -- 172.21.15.189:0/1437807719 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a00038c0 con 0x7f00b00b8d70 2024-04-06T13:48:32.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.604+0000 7f00b54e8700 1 -- 172.21.15.189:0/1437807719 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b00b4c70 con 0x7f00b00b8d70 2024-04-06T13:48:32.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.604+0000 7f00b54e8700 1 -- 172.21.15.189:0/1437807719 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b00b4890 con 0x7f00b00b8d70 2024-04-06T13:48:32.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.604+0000 7f00b54e8700 1 -- 172.21.15.189:0/1437807719 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b00c6d80 con 0x7f00b00b8d70 2024-04-06T13:48:32.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.605+0000 7f00977fe700 1 -- 172.21.15.189:0/1437807719 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f00a0003a20 con 0x7f00b00b8d70 2024-04-06T13:48:32.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.606+0000 7f00977fe700 1 --2- 172.21.15.189:0/1437807719 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f009c06aed0 0x7f009c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:32.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.606+0000 7f00977fe700 1 -- 172.21.15.189:0/1437807719 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f00a0088f10 con 0x7f00b00b8d70 2024-04-06T13:48:32.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.606+0000 7f00ae7fc700 1 --2- 172.21.15.189:0/1437807719 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f009c06aed0 0x7f009c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:32.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.606+0000 7f00ae7fc700 1 --2- 172.21.15.189:0/1437807719 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f009c06aed0 0x7f009c06d390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f00a40097c0 tx=0x7f00a4008030 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:32.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.610+0000 7f00977fe700 1 -- 172.21.15.189:0/1437807719 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f00a0053d60 con 0x7f00b00b8d70 2024-04-06T13:48:32.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.803+0000 7f00b54e8700 1 -- 172.21.15.189:0/1437807719 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f00b0074990 con 0x7f00b00b8d70 2024-04-06T13:48:32.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.803+0000 7f00977fe700 1 -- 172.21.15.189:0/1437807719 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v248) v1 ==== 104+0+11 (secure 0 0 0) 0x7f00a0017020 con 0x7f00b00b8d70 2024-04-06T13:48:32.803 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:48:32.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.805+0000 7f00b54e8700 1 -- 172.21.15.189:0/1437807719 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f009c06aed0 msgr2=0x7f009c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.805+0000 7f00b54e8700 1 --2- 172.21.15.189:0/1437807719 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f009c06aed0 0x7f009c06d390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f00a40097c0 tx=0x7f00a4008030 comp rx=0 tx=0).stop 2024-04-06T13:48:32.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.805+0000 7f00b54e8700 1 -- 172.21.15.189:0/1437807719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00b00b8d70 msgr2=0x7f00b01f6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.805+0000 7f00b54e8700 1 --2- 172.21.15.189:0/1437807719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00b00b8d70 0x7f00b01f6c30 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f00a000c450 tx=0x7f00a000c760 comp rx=0 tx=0).stop 2024-04-06T13:48:32.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.805+0000 7f00b54e8700 1 -- 172.21.15.189:0/1437807719 shutdown_connections 2024-04-06T13:48:32.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.805+0000 7f00b54e8700 1 --2- 172.21.15.189:0/1437807719 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f009c06aed0 0x7f009c06d390 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.805+0000 7f00b54e8700 1 --2- 172.21.15.189:0/1437807719 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00b015d6f0 0x7f00b00b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.805+0000 7f00b54e8700 1 --2- 172.21.15.189:0/1437807719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00b00b8d70 0x7f00b01f6c30 unknown :-1 s=CLOSED pgs=2712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.805+0000 7f00b54e8700 1 --2- 172.21.15.189:0/1437807719 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b00ce730 0x7f00b00c8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.805+0000 7f00b54e8700 1 -- 172.21.15.189:0/1437807719 >> 172.21.15.189:0/1437807719 conn(0x7f00b0072490 msgr2=0x7f00b00b6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:32.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.806+0000 7f00b54e8700 1 -- 172.21.15.189:0/1437807719 shutdown_connections 2024-04-06T13:48:32.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.806+0000 7f00b54e8700 1 -- 172.21.15.189:0/1437807719 wait complete. 2024-04-06T13:48:32.818 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T13:48:32.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.946+0000 7f4561f6c700 1 Processor -- start 2024-04-06T13:48:32.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.946+0000 7f4561f6c700 1 -- start start 2024-04-06T13:48:32.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.947+0000 7f4561f6c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f455c0cb870 0x7f455c15c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:32.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.947+0000 7f4561f6c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f455c0cbd80 con 0x7f455c0cfa00 2024-04-06T13:48:32.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.947+0000 7f4561f6c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f455c0cef20 con 0x7f455c0d11b0 2024-04-06T13:48:32.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.947+0000 7f4561f6c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f455c0ccdc0 con 0x7f455c0cb870 2024-04-06T13:48:32.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.947+0000 7f455affd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f455c0cb870 0x7f455c15c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:32.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.947+0000 7f455affd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f455c0cb870 0x7f455c15c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49388/0 (socket says 172.21.15.189:49388) 2024-04-06T13:48:32.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.947+0000 7f455affd700 1 -- 172.21.15.189:0/4079338200 learned_addr learned my addr 172.21.15.189:0/4079338200 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:32.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.947+0000 7f455bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f455c0cfa00 0x7f455c0cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55810/0 (socket says 172.21.15.189:55810) 2024-04-06T13:48:32.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.948+0000 7f455affd700 1 -- 172.21.15.189:0/4079338200 >> v1:172.21.15.192:6789/0 conn(0x7f455c0d11b0 legacy=0x7f455c0d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.948+0000 7f455affd700 1 -- 172.21.15.189:0/4079338200 >> v1:172.21.15.189:6789/0 conn(0x7f455c0cfa00 legacy=0x7f455c0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.948+0000 7f455affd700 1 -- 172.21.15.189:0/4079338200 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455c0cf200 con 0x7f455c0cb870 2024-04-06T13:48:32.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.948+0000 7f455a7fc700 1 -- 172.21.15.189:0/4079338200 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3449631207 0 0) 0x7f455c0cbd80 con 0x7f455c0cfa00 2024-04-06T13:48:32.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.948+0000 7f455affd700 1 --2- 172.21.15.189:0/4079338200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f455c0cb870 0x7f455c15c280 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7f454c0098f0 tx=0x7f454c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e692754d98604f8c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:32.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.948+0000 7f455a7fc700 1 -- 172.21.15.189:0/4079338200 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c00b8a0 con 0x7f455c0cb870 2024-04-06T13:48:32.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.948+0000 7f455a7fc700 1 -- 172.21.15.189:0/4079338200 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f454c00ba00 con 0x7f455c0cb870 2024-04-06T13:48:32.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.948+0000 7f455a7fc700 1 -- 172.21.15.189:0/4079338200 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c00bd90 con 0x7f455c0cb870 2024-04-06T13:48:32.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.948+0000 7f4561f6c700 1 -- 172.21.15.189:0/4079338200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f455c0cb870 msgr2=0x7f455c15c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.948+0000 7f4561f6c700 1 --2- 172.21.15.189:0/4079338200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f455c0cb870 0x7f455c15c280 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7f454c0098f0 tx=0x7f454c009c00 comp rx=0 tx=0).stop 2024-04-06T13:48:32.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.948+0000 7f4561f6c700 1 -- 172.21.15.189:0/4079338200 shutdown_connections 2024-04-06T13:48:32.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.949+0000 7f4561f6c700 1 --2- 172.21.15.189:0/4079338200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f455c0cb870 0x7f455c15c280 unknown :-1 s=CLOSED pgs=2714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.949+0000 7f4561f6c700 1 -- 172.21.15.189:0/4079338200 >> 172.21.15.189:0/4079338200 conn(0x7f455c0b2db0 msgr2=0x7f455c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:32.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.949+0000 7f4561f6c700 1 -- 172.21.15.189:0/4079338200 shutdown_connections 2024-04-06T13:48:32.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.949+0000 7f4561f6c700 1 -- 172.21.15.189:0/4079338200 wait complete. 2024-04-06T13:48:32.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.950+0000 7f4561f6c700 1 Processor -- start 2024-04-06T13:48:32.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.950+0000 7f4561f6c700 1 -- start start 2024-04-06T13:48:32.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f4561f6c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f455c0cb870 0x7f455c1403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:32.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f4561f6c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f455c0cfa00 0x7f455c0b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:32.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f4561f6c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f455c0d11b0 0x7f455c150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:32.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f4561f6c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455c15e4d0 con 0x7f455c0d11b0 2024-04-06T13:48:32.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f4561f6c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455c15e7b0 con 0x7f455c0cfa00 2024-04-06T13:48:32.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f4561f6c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f455c15ea90 con 0x7f455c0cb870 2024-04-06T13:48:32.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f455bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f455c0d11b0 0x7f455c150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:32.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f455bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f455c0d11b0 0x7f455c150b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49394/0 (socket says 172.21.15.189:49394) 2024-04-06T13:48:32.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f455bfff700 1 -- 172.21.15.189:0/46800433 learned_addr learned my addr 172.21.15.189:0/46800433 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:32.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f455affd700 1 --2- 172.21.15.189:0/46800433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f455c0cfa00 0x7f455c0b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:32.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f455b7fe700 1 --2- 172.21.15.189:0/46800433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f455c0cb870 0x7f455c1403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:32.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f455bfff700 1 -- 172.21.15.189:0/46800433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f455c0cb870 msgr2=0x7f455c1403f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f455bfff700 1 --2- 172.21.15.189:0/46800433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f455c0cb870 0x7f455c1403f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f455bfff700 1 -- 172.21.15.189:0/46800433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f455c0cfa00 msgr2=0x7f455c0b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:32.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f455bfff700 1 --2- 172.21.15.189:0/46800433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f455c0cfa00 0x7f455c0b3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:32.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f455bfff700 1 -- 172.21.15.189:0/46800433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454c009550 con 0x7f455c0d11b0 2024-04-06T13:48:32.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f455affd700 1 --2- 172.21.15.189:0/46800433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f455c0cfa00 0x7f455c0b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:32.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f455bfff700 1 --2- 172.21.15.189:0/46800433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f455c0d11b0 0x7f455c150b00 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7f4554002230 tx=0x7f455400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:32.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f4553fff700 1 -- 172.21.15.189:0/46800433 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455400ac20 con 0x7f455c0d11b0 2024-04-06T13:48:32.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f4553fff700 1 -- 172.21.15.189:0/46800433 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f455400ada0 con 0x7f455c0d11b0 2024-04-06T13:48:32.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f4553fff700 1 -- 172.21.15.189:0/46800433 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554007310 con 0x7f455c0d11b0 2024-04-06T13:48:32.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f4561f6c700 1 -- 172.21.15.189:0/46800433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f455c15ed70 con 0x7f455c0d11b0 2024-04-06T13:48:32.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.951+0000 7f4561f6c700 1 -- 172.21.15.189:0/46800433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f455c151900 con 0x7f455c0d11b0 2024-04-06T13:48:32.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.952+0000 7f4561f6c700 1 -- 172.21.15.189:0/46800433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f455c15f290 con 0x7f455c0d11b0 2024-04-06T13:48:32.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.953+0000 7f4553fff700 1 -- 172.21.15.189:0/46800433 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f4554007470 con 0x7f455c0d11b0 2024-04-06T13:48:32.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.953+0000 7f4553fff700 1 --2- 172.21.15.189:0/46800433 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f453c06aed0 0x7f453c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:32.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.953+0000 7f455b7fe700 1 --2- 172.21.15.189:0/46800433 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f453c06aed0 0x7f453c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:32.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.953+0000 7f4553fff700 1 -- 172.21.15.189:0/46800433 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4554089470 con 0x7f455c0d11b0 2024-04-06T13:48:32.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.954+0000 7f455b7fe700 1 --2- 172.21.15.189:0/46800433 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f453c06aed0 0x7f453c06d390 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4544000f60 tx=0x7f4544009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:32.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:32.958+0000 7f4553fff700 1 -- 172.21.15.189:0/46800433 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f455c15f290 con 0x7f455c0d11b0 2024-04-06T13:48:33.091 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:48:33.093 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:48:33.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.156+0000 7f4561f6c700 1 -- 172.21.15.189:0/46800433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f455c15f290 con 0x7f455c0d11b0 2024-04-06T13:48:33.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.157+0000 7f4553fff700 1 -- 172.21.15.189:0/46800433 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v248) v1 ==== 100+0+11 (secure 0 0 0) 0x7f455c15f290 con 0x7f455c0d11b0 2024-04-06T13:48:33.157 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:48:33.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.159+0000 7f4561f6c700 1 -- 172.21.15.189:0/46800433 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f453c06aed0 msgr2=0x7f453c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:33.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.159+0000 7f4561f6c700 1 --2- 172.21.15.189:0/46800433 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f453c06aed0 0x7f453c06d390 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4544000f60 tx=0x7f4544009040 comp rx=0 tx=0).stop 2024-04-06T13:48:33.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.159+0000 7f4561f6c700 1 -- 172.21.15.189:0/46800433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f455c0d11b0 msgr2=0x7f455c150b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:33.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.159+0000 7f4561f6c700 1 --2- 172.21.15.189:0/46800433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f455c0d11b0 0x7f455c150b00 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7f4554002230 tx=0x7f455400a060 comp rx=0 tx=0).stop 2024-04-06T13:48:33.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.159+0000 7f4561f6c700 1 -- 172.21.15.189:0/46800433 shutdown_connections 2024-04-06T13:48:33.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.159+0000 7f4561f6c700 1 --2- 172.21.15.189:0/46800433 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f453c06aed0 0x7f453c06d390 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.159+0000 7f4561f6c700 1 --2- 172.21.15.189:0/46800433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f455c0d11b0 0x7f455c150b00 unknown :-1 s=CLOSED pgs=2715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.159+0000 7f4561f6c700 1 --2- 172.21.15.189:0/46800433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f455c0cb870 0x7f455c1403f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.159+0000 7f4561f6c700 1 --2- 172.21.15.189:0/46800433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f455c0cfa00 0x7f455c0b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.159+0000 7f4561f6c700 1 -- 172.21.15.189:0/46800433 >> 172.21.15.189:0/46800433 conn(0x7f455c0b2db0 msgr2=0x7f455c0b3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:33.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.159+0000 7f4561f6c700 1 -- 172.21.15.189:0/46800433 shutdown_connections 2024-04-06T13:48:33.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.159+0000 7f4561f6c700 1 -- 172.21.15.189:0/46800433 wait complete. 2024-04-06T13:48:33.172 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-06T13:48:33.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.334+0000 7f2467fe4700 1 Processor -- start 2024-04-06T13:48:33.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.334+0000 7f2467fe4700 1 -- start start 2024-04-06T13:48:33.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.335+0000 7f2467fe4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24600b4af0 0x7f24600b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:33.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.335+0000 7f2467fe4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24600cdbe0 con 0x7f246015a550 2024-04-06T13:48:33.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.335+0000 7f2467fe4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246015a940 con 0x7f24600b4660 2024-04-06T13:48:33.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.335+0000 7f2467fe4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f246015ac00 con 0x7f24600b4af0 2024-04-06T13:48:33.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.335+0000 7f2465d80700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f246015a550 0x7f24600cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55812/0 (socket says 172.21.15.189:55812) 2024-04-06T13:48:33.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.335+0000 7f2465d80700 1 -- 172.21.15.189:0/4233567448 learned_addr learned my addr 172.21.15.189:0/4233567448 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:33.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.335+0000 7f2464d7e700 1 -- 172.21.15.189:0/4233567448 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960952553 0 0) 0x7f24600cdbe0 con 0x7f246015a550 2024-04-06T13:48:33.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.335+0000 7f2464d7e700 1 -- 172.21.15.189:0/4233567448 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f244c0035b0 con 0x7f246015a550 2024-04-06T13:48:33.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.335+0000 7f2466581700 1 --2- 172.21.15.189:0/4233567448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24600b4af0 0x7f24600b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:33.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.335+0000 7f2464d7e700 1 -- 172.21.15.189:0/4233567448 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3159712578 0 0) 0x7f244c0035b0 con 0x7f246015a550 2024-04-06T13:48:33.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.335+0000 7f2464d7e700 1 -- 172.21.15.189:0/4233567448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24600b4af0 msgr2=0x7f24600b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:33.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.335+0000 7f2464d7e700 1 --2- 172.21.15.189:0/4233567448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24600b4af0 0x7f24600b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.335+0000 7f2464d7e700 1 -- 172.21.15.189:0/4233567448 >> v1:172.21.15.192:6789/0 conn(0x7f24600b4660 legacy=0x7f24600b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:33.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.335+0000 7f2464d7e700 1 -- 172.21.15.189:0/4233567448 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246015aee0 con 0x7f246015a550 2024-04-06T13:48:33.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.335+0000 7f2464d7e700 1 -- 172.21.15.189:0/4233567448 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f245c001670 con 0x7f246015a550 2024-04-06T13:48:33.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.335+0000 7f2464d7e700 1 -- 172.21.15.189:0/4233567448 >> v1:172.21.15.189:6789/0 conn(0x7f246015a550 legacy=0x7f24600cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:33.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.336+0000 7f2464d7e700 1 --2- 172.21.15.189:0/4233567448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f244c003a30 0x7f244c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:33.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.336+0000 7f2464d7e700 1 --2- 172.21.15.189:0/4233567448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f244c004850 0x7f244c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:33.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.336+0000 7f2464d7e700 1 --2- 172.21.15.189:0/4233567448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24600b4af0 0x7f244c005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:33.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.336+0000 7f2464d7e700 1 -- 172.21.15.189:0/4233567448 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246015ac00 con 0x7f244c003a30 2024-04-06T13:48:33.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.336+0000 7f2464d7e700 1 -- 172.21.15.189:0/4233567448 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c00bb40 con 0x7f244c004850 2024-04-06T13:48:33.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.336+0000 7f2464d7e700 1 -- 172.21.15.189:0/4233567448 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f244c00bd00 con 0x7f24600b4af0 2024-04-06T13:48:33.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.336+0000 7f2465d80700 1 --2- 172.21.15.189:0/4233567448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f244c003a30 0x7f244c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:33.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.336+0000 7f246557f700 1 --2- 172.21.15.189:0/4233567448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f244c004850 0x7f244c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:33.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.336+0000 7f2466581700 1 --2- 172.21.15.189:0/4233567448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24600b4af0 0x7f244c005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:33.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.336+0000 7f2465d80700 1 -- 172.21.15.189:0/4233567448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24600b4af0 msgr2=0x7f244c005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:33.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.336+0000 7f2465d80700 1 --2- 172.21.15.189:0/4233567448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24600b4af0 0x7f244c005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.336+0000 7f2465d80700 1 -- 172.21.15.189:0/4233567448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f244c004850 msgr2=0x7f244c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:33.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.336+0000 7f2465d80700 1 --2- 172.21.15.189:0/4233567448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f244c004850 0x7f244c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.336+0000 7f2465d80700 1 -- 172.21.15.189:0/4233567448 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f244c0035b0 con 0x7f244c003a30 2024-04-06T13:48:33.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.336+0000 7f2465d80700 1 --2- 172.21.15.189:0/4233567448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f244c003a30 0x7f244c003e50 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7f245c001050 tx=0x7f244c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45f88597e63472a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:33.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.336+0000 7f2464d7e700 1 -- 172.21.15.189:0/4233567448 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c00cd80 con 0x7f244c003a30 2024-04-06T13:48:33.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.336+0000 7f2464d7e700 1 -- 172.21.15.189:0/4233567448 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245c005960 con 0x7f244c003a30 2024-04-06T13:48:33.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.336+0000 7f2464d7e700 1 -- 172.21.15.189:0/4233567448 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c005d10 con 0x7f244c003a30 2024-04-06T13:48:33.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.337+0000 7f2467fe4700 1 -- 172.21.15.189:0/4233567448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f244c003a30 msgr2=0x7f244c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:33.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.337+0000 7f2467fe4700 1 --2- 172.21.15.189:0/4233567448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f244c003a30 0x7f244c003e50 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7f245c001050 tx=0x7f244c005880 comp rx=0 tx=0).stop 2024-04-06T13:48:33.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.337+0000 7f2466581700 1 -- 172.21.15.189:0/4233567448 reap_dead start 2024-04-06T13:48:33.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.337+0000 7f2467fe4700 1 -- 172.21.15.189:0/4233567448 shutdown_connections 2024-04-06T13:48:33.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.337+0000 7f2467fe4700 1 -- 172.21.15.189:0/4233567448 >> 172.21.15.189:0/4233567448 conn(0x7f24600d00b0 msgr2=0x7f24600d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:33.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.337+0000 7f2467fe4700 1 -- 172.21.15.189:0/4233567448 shutdown_connections 2024-04-06T13:48:33.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.337+0000 7f2467fe4700 1 -- 172.21.15.189:0/4233567448 wait complete. 2024-04-06T13:48:33.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.338+0000 7f2467fe4700 1 Processor -- start 2024-04-06T13:48:33.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.338+0000 7f2467fe4700 1 -- start start 2024-04-06T13:48:33.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f2467fe4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f244c003a30 0x7f246015b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:33.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f2467fe4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24600aef80 0x7f246014faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:33.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f2467fe4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24600b2cb0 0x7f24600b3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:33.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f2467fe4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24600b3670 con 0x7f244c003a30 2024-04-06T13:48:33.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f2467fe4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246014ffe0 con 0x7f24600b2cb0 2024-04-06T13:48:33.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f2467fe4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f246015cb20 con 0x7f24600aef80 2024-04-06T13:48:33.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f2465d80700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f244c003a30 0x7f246015b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:33.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f2465d80700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f244c003a30 0x7f246015b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49414/0 (socket says 172.21.15.189:49414) 2024-04-06T13:48:33.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f2465d80700 1 -- 172.21.15.189:0/3302783514 learned_addr learned my addr 172.21.15.189:0/3302783514 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:33.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f246557f700 1 --2- 172.21.15.189:0/3302783514 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24600aef80 0x7f246014faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:33.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f2466581700 1 --2- 172.21.15.189:0/3302783514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24600b2cb0 0x7f24600b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:33.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f2465d80700 1 -- 172.21.15.189:0/3302783514 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24600aef80 msgr2=0x7f246014faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:33.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f2465d80700 1 --2- 172.21.15.189:0/3302783514 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24600aef80 0x7f246014faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f2465d80700 1 -- 172.21.15.189:0/3302783514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24600b2cb0 msgr2=0x7f24600b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:33.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f2465d80700 1 --2- 172.21.15.189:0/3302783514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24600b2cb0 0x7f24600b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f2465d80700 1 -- 172.21.15.189:0/3302783514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24601563c0 con 0x7f244c003a30 2024-04-06T13:48:33.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f2465d80700 1 --2- 172.21.15.189:0/3302783514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f244c003a30 0x7f246015b930 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7f244c00bb10 tx=0x7f245c000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:33.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f24567fc700 1 -- 172.21.15.189:0/3302783514 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c007c00 con 0x7f244c003a30 2024-04-06T13:48:33.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f24567fc700 1 -- 172.21.15.189:0/3302783514 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245c0070d0 con 0x7f244c003a30 2024-04-06T13:48:33.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f2467fe4700 1 -- 172.21.15.189:0/3302783514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f246015bf10 con 0x7f244c003a30 2024-04-06T13:48:33.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f24567fc700 1 -- 172.21.15.189:0/3302783514 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c007460 con 0x7f244c003a30 2024-04-06T13:48:33.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.339+0000 7f2467fe4700 1 -- 172.21.15.189:0/3302783514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f246015c4d0 con 0x7f244c003a30 2024-04-06T13:48:33.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.340+0000 7f2467fe4700 1 -- 172.21.15.189:0/3302783514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f246015ce00 con 0x7f244c003a30 2024-04-06T13:48:33.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.341+0000 7f24567fc700 1 -- 172.21.15.189:0/3302783514 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f245c019030 con 0x7f244c003a30 2024-04-06T13:48:33.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.341+0000 7f24567fc700 1 --2- 172.21.15.189:0/3302783514 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f244406aed0 0x7f244406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:33.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.341+0000 7f24567fc700 1 -- 172.21.15.189:0/3302783514 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f245c08a7f0 con 0x7f244c003a30 2024-04-06T13:48:33.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.341+0000 7f246557f700 1 --2- 172.21.15.189:0/3302783514 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f244406aed0 0x7f244406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:33.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.342+0000 7f246557f700 1 --2- 172.21.15.189:0/3302783514 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f244406aed0 0x7f244406d390 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f24601502c0 tx=0x7f2450009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:33.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.346+0000 7f24567fc700 1 -- 172.21.15.189:0/3302783514 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f245c055570 con 0x7f244c003a30 2024-04-06T13:48:33.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.540+0000 7f2467fe4700 1 -- 172.21.15.189:0/3302783514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f24600af360 con 0x7f244c003a30 2024-04-06T13:48:33.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.540+0000 7f24567fc700 1 -- 172.21.15.189:0/3302783514 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v248) v1 ==== 84+0+209 (secure 0 0 0) 0x7f245c058b90 con 0x7f244c003a30 2024-04-06T13:48:33.541 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:33.541 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T13:48:33.541 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:33.541 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 1, 2024-04-06T13:48:33.541 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": ".mgr" 2024-04-06T13:48:33.541 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:33.541 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:33.541 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 31, 2024-04-06T13:48:33.541 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": "cephfs_metadata" 2024-04-06T13:48:33.542 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:33.542 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:33.542 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 32, 2024-04-06T13:48:33.542 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": "cephfs_data" 2024-04-06T13:48:33.542 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:33.542 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T13:48:33.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.542+0000 7f2467fe4700 1 -- 172.21.15.189:0/3302783514 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f244406aed0 msgr2=0x7f244406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:33.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.542+0000 7f2467fe4700 1 --2- 172.21.15.189:0/3302783514 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f244406aed0 0x7f244406d390 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f24601502c0 tx=0x7f2450009040 comp rx=0 tx=0).stop 2024-04-06T13:48:33.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.542+0000 7f2467fe4700 1 -- 172.21.15.189:0/3302783514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f244c003a30 msgr2=0x7f246015b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:33.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.542+0000 7f2467fe4700 1 --2- 172.21.15.189:0/3302783514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f244c003a30 0x7f246015b930 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7f244c00bb10 tx=0x7f245c000fe0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.543+0000 7f2467fe4700 1 -- 172.21.15.189:0/3302783514 shutdown_connections 2024-04-06T13:48:33.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.543+0000 7f2467fe4700 1 --2- 172.21.15.189:0/3302783514 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f244406aed0 0x7f244406d390 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.543+0000 7f2467fe4700 1 --2- 172.21.15.189:0/3302783514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24600b2cb0 0x7f24600b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.543+0000 7f2467fe4700 1 --2- 172.21.15.189:0/3302783514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f244c003a30 0x7f246015b930 unknown :-1 s=CLOSED pgs=2718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.543+0000 7f2467fe4700 1 --2- 172.21.15.189:0/3302783514 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24600aef80 0x7f246014faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.543+0000 7f2467fe4700 1 -- 172.21.15.189:0/3302783514 >> 172.21.15.189:0/3302783514 conn(0x7f24600d00b0 msgr2=0x7f24600717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:33.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.543+0000 7f2467fe4700 1 -- 172.21.15.189:0/3302783514 shutdown_connections 2024-04-06T13:48:33.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.543+0000 7f2467fe4700 1 -- 172.21.15.189:0/3302783514 wait complete. 2024-04-06T13:48:33.555 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2024-04-06T13:48:33.556 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2024-04-06T13:48:33.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.682+0000 7f3e13a8f700 1 Processor -- start 2024-04-06T13:48:33.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.682+0000 7f3e13a8f700 1 -- start start 2024-04-06T13:48:33.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.683+0000 7f3e13a8f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e0c15d7b0 0x7f3e0c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:33.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.683+0000 7f3e13a8f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3e0c15e500 0x7f3e0c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:33.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.683+0000 7f3e13a8f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e0c0b3990 con 0x7f3e0c06a730 2024-04-06T13:48:33.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.683+0000 7f3e13a8f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e0c06b0f0 con 0x7f3e0c15e500 2024-04-06T13:48:33.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.683+0000 7f3e13a8f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c0b3060 con 0x7f3e0c15d7b0 2024-04-06T13:48:33.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e1102a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3e0c06a730 0x7f3e0c15ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55824/0 (socket says 172.21.15.189:55824) 2024-04-06T13:48:33.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e1102a700 1 -- 172.21.15.189:0/1844022575 learned_addr learned my addr 172.21.15.189:0/1844022575 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:33.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e1202c700 1 --2- 172.21.15.189:0/1844022575 >> v2:172.21.15.192:3300/0 conn(0x7f3e0c15e500 0x7f3e0c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:33.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e1182b700 1 --2- 172.21.15.189:0/1844022575 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e0c15d7b0 0x7f3e0c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:33.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e10829700 1 -- 172.21.15.189:0/1844022575 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992550626 0 0) 0x7f3e0c0b3990 con 0x7f3e0c06a730 2024-04-06T13:48:33.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e10829700 1 -- 172.21.15.189:0/1844022575 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3df80035b0 con 0x7f3e0c06a730 2024-04-06T13:48:33.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e10829700 1 -- 172.21.15.189:0/1844022575 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2957847197 0 0) 0x7f3df80035b0 con 0x7f3e0c06a730 2024-04-06T13:48:33.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e10829700 1 -- 172.21.15.189:0/1844022575 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e0c15d7b0 msgr2=0x7f3e0c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:33.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e10829700 1 --2- 172.21.15.189:0/1844022575 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e0c15d7b0 0x7f3e0c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e10829700 1 -- 172.21.15.189:0/1844022575 >> v2:172.21.15.192:3300/0 conn(0x7f3e0c15e500 msgr2=0x7f3e0c06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:33.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e10829700 1 --2- 172.21.15.189:0/1844022575 >> v2:172.21.15.192:3300/0 conn(0x7f3e0c15e500 0x7f3e0c06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e10829700 1 -- 172.21.15.189:0/1844022575 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0c0b3340 con 0x7f3e0c06a730 2024-04-06T13:48:33.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e10829700 1 -- 172.21.15.189:0/1844022575 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f3e08001670 con 0x7f3e0c06a730 2024-04-06T13:48:33.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e10829700 1 -- 172.21.15.189:0/1844022575 >> v1:172.21.15.189:6789/0 conn(0x7f3e0c06a730 legacy=0x7f3e0c15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:33.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e10829700 1 --2- 172.21.15.189:0/1844022575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df8003a00 0x7f3df8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:33.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e10829700 1 --2- 172.21.15.189:0/1844022575 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e0c15d7b0 0x7f3df8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:33.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e10829700 1 --2- 172.21.15.189:0/1844022575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3df8004d90 0x7f3df8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:33.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e10829700 1 -- 172.21.15.189:0/1844022575 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c06b0f0 con 0x7f3df8003a00 2024-04-06T13:48:33.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e10829700 1 -- 172.21.15.189:0/1844022575 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c0b3060 con 0x7f3df8004d90 2024-04-06T13:48:33.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e10829700 1 -- 172.21.15.189:0/1844022575 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df800ba10 con 0x7f3e0c15d7b0 2024-04-06T13:48:33.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e1182b700 1 --2- 172.21.15.189:0/1844022575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df8003a00 0x7f3df8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:33.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e1102a700 1 --2- 172.21.15.189:0/1844022575 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e0c15d7b0 0x7f3df8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:33.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.684+0000 7f3e1202c700 1 --2- 172.21.15.189:0/1844022575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3df8004d90 0x7f3df8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:33.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.685+0000 7f3e1182b700 1 -- 172.21.15.189:0/1844022575 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e0c15d7b0 msgr2=0x7f3df8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:33.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.685+0000 7f3e1182b700 1 --2- 172.21.15.189:0/1844022575 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e0c15d7b0 0x7f3df8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.685+0000 7f3e1182b700 1 -- 172.21.15.189:0/1844022575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3df8004d90 msgr2=0x7f3df8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:33.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.685+0000 7f3e1182b700 1 --2- 172.21.15.189:0/1844022575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3df8004d90 0x7f3df8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.685+0000 7f3e1182b700 1 -- 172.21.15.189:0/1844022575 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df80035b0 con 0x7f3df8003a00 2024-04-06T13:48:33.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.685+0000 7f3e1182b700 1 --2- 172.21.15.189:0/1844022575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df8003a00 0x7f3df8003e20 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc009b60 tx=0x7f3df8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a803e4eb8f47681b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:33.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.685+0000 7f3e10829700 1 -- 172.21.15.189:0/1844022575 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc00e6c0 con 0x7f3df8003a00 2024-04-06T13:48:33.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.685+0000 7f3e10829700 1 -- 172.21.15.189:0/1844022575 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dfc00e820 con 0x7f3df8003a00 2024-04-06T13:48:33.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.685+0000 7f3e10829700 1 -- 172.21.15.189:0/1844022575 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc00eb90 con 0x7f3df8003a00 2024-04-06T13:48:33.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.685+0000 7f3e13a8f700 1 -- 172.21.15.189:0/1844022575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df8003a00 msgr2=0x7f3df8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:33.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.685+0000 7f3e13a8f700 1 --2- 172.21.15.189:0/1844022575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df8003a00 0x7f3df8003e20 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc009b60 tx=0x7f3df8005880 comp rx=0 tx=0).stop 2024-04-06T13:48:33.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.685+0000 7f3e1202c700 1 -- 172.21.15.189:0/1844022575 reap_dead start 2024-04-06T13:48:33.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.685+0000 7f3e13a8f700 1 -- 172.21.15.189:0/1844022575 shutdown_connections 2024-04-06T13:48:33.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.685+0000 7f3e13a8f700 1 -- 172.21.15.189:0/1844022575 >> 172.21.15.189:0/1844022575 conn(0x7f3e0c0d00b0 msgr2=0x7f3e0c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:33.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.686+0000 7f3e13a8f700 1 -- 172.21.15.189:0/1844022575 shutdown_connections 2024-04-06T13:48:33.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.686+0000 7f3e13a8f700 1 -- 172.21.15.189:0/1844022575 wait complete. 2024-04-06T13:48:33.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.687+0000 7f3e13a8f700 1 Processor -- start 2024-04-06T13:48:33.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.687+0000 7f3e13a8f700 1 -- start start 2024-04-06T13:48:33.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.687+0000 7f3e13a8f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df8003a00 0x7f3e0c1f3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:33.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.687+0000 7f3e13a8f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e0c153ae0 0x7f3e0c0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:33.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.687+0000 7f3e13a8f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e0c15fcd0 0x7f3e0c160150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:33.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.687+0000 7f3e13a8f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c0cd260 con 0x7f3df8003a00 2024-04-06T13:48:33.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.687+0000 7f3e13a8f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c1f28f0 con 0x7f3e0c15fcd0 2024-04-06T13:48:33.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.687+0000 7f3e13a8f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c160690 con 0x7f3e0c153ae0 2024-04-06T13:48:33.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.688+0000 7f3e1182b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df8003a00 0x7f3e0c1f3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:33.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.688+0000 7f3e1182b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df8003a00 0x7f3e0c1f3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49430/0 (socket says 172.21.15.189:49430) 2024-04-06T13:48:33.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.688+0000 7f3e1182b700 1 -- 172.21.15.189:0/2972456445 learned_addr learned my addr 172.21.15.189:0/2972456445 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:33.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.688+0000 7f3e1102a700 1 --2- 172.21.15.189:0/2972456445 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e0c153ae0 0x7f3e0c0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:33.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.688+0000 7f3e1202c700 1 --2- 172.21.15.189:0/2972456445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e0c15fcd0 0x7f3e0c160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:33.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.688+0000 7f3e1182b700 1 -- 172.21.15.189:0/2972456445 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e0c153ae0 msgr2=0x7f3e0c0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:33.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.688+0000 7f3e1182b700 1 --2- 172.21.15.189:0/2972456445 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e0c153ae0 0x7f3e0c0ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.688+0000 7f3e1182b700 1 -- 172.21.15.189:0/2972456445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e0c15fcd0 msgr2=0x7f3e0c160150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:33.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.688+0000 7f3e1182b700 1 --2- 172.21.15.189:0/2972456445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e0c15fcd0 0x7f3e0c160150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:33.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.688+0000 7f3e1182b700 1 -- 172.21.15.189:0/2972456445 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0c1f39b0 con 0x7f3df8003a00 2024-04-06T13:48:33.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.688+0000 7f3e1102a700 1 --2- 172.21.15.189:0/2972456445 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e0c153ae0 0x7f3e0c0ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:33.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.688+0000 7f3e1182b700 1 --2- 172.21.15.189:0/2972456445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df8003a00 0x7f3e0c1f3470 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0090b0 tx=0x7f3dfc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:33.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.688+0000 7f3e027fc700 1 -- 172.21.15.189:0/2972456445 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc021070 con 0x7f3df8003a00 2024-04-06T13:48:33.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.688+0000 7f3e027fc700 1 -- 172.21.15.189:0/2972456445 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dfc004590 con 0x7f3df8003a00 2024-04-06T13:48:33.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.688+0000 7f3e13a8f700 1 -- 172.21.15.189:0/2972456445 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e0c0cc5e0 con 0x7f3df8003a00 2024-04-06T13:48:33.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.688+0000 7f3e027fc700 1 -- 172.21.15.189:0/2972456445 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc00e6c0 con 0x7f3df8003a00 2024-04-06T13:48:33.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.688+0000 7f3e13a8f700 1 -- 172.21.15.189:0/2972456445 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e0c0ccba0 con 0x7f3df8003a00 2024-04-06T13:48:33.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.689+0000 7f3e13a8f700 1 -- 172.21.15.189:0/2972456445 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e0c067bb0 con 0x7f3df8003a00 2024-04-06T13:48:33.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.690+0000 7f3e027fc700 1 -- 172.21.15.189:0/2972456445 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f3dfc025070 con 0x7f3df8003a00 2024-04-06T13:48:33.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.690+0000 7f3e027fc700 1 --2- 172.21.15.189:0/2972456445 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3df006aed0 0x7f3df006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:33.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.690+0000 7f3e027fc700 1 -- 172.21.15.189:0/2972456445 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3dfc08a6b0 con 0x7f3df8003a00 2024-04-06T13:48:33.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.690+0000 7f3e1102a700 1 --2- 172.21.15.189:0/2972456445 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3df006aed0 0x7f3df006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:33.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.690+0000 7f3e1102a700 1 --2- 172.21.15.189:0/2972456445 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3df006aed0 0x7f3df006d390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c0cc410 tx=0x7f3e08009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:33.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.695+0000 7f3e027fc700 1 -- 172.21.15.189:0/2972456445 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3dfc055430 con 0x7f3df8003a00 2024-04-06T13:48:33.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:33.890+0000 7f3e13a8f700 1 -- 172.21.15.189:0/2972456445 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3e0c074990 con 0x7f3df8003a00 2024-04-06T13:48:34.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.781+0000 7f3e027fc700 1 -- 172.21.15.189:0/2972456445 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v249) v1 ==== 144+0+0 (secure 0 0 0) 0x7f3dfc058a50 con 0x7f3df8003a00 2024-04-06T13:48:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.789+0000 7f3e13a8f700 1 -- 172.21.15.189:0/2972456445 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3df006aed0 msgr2=0x7f3df006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.789+0000 7f3e13a8f700 1 --2- 172.21.15.189:0/2972456445 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3df006aed0 0x7f3df006d390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c0cc410 tx=0x7f3e08009040 comp rx=0 tx=0).stop 2024-04-06T13:48:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.789+0000 7f3e13a8f700 1 -- 172.21.15.189:0/2972456445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df8003a00 msgr2=0x7f3e0c1f3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.789+0000 7f3e13a8f700 1 --2- 172.21.15.189:0/2972456445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df8003a00 0x7f3e0c1f3470 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0090b0 tx=0x7f3dfc0047c0 comp rx=0 tx=0).stop 2024-04-06T13:48:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.790+0000 7f3e13a8f700 1 -- 172.21.15.189:0/2972456445 shutdown_connections 2024-04-06T13:48:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.790+0000 7f3e13a8f700 1 --2- 172.21.15.189:0/2972456445 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3df006aed0 0x7f3df006d390 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.790+0000 7f3e13a8f700 1 --2- 172.21.15.189:0/2972456445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e0c15fcd0 0x7f3e0c160150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.790+0000 7f3e13a8f700 1 --2- 172.21.15.189:0/2972456445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df8003a00 0x7f3e0c1f3470 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:34.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.790+0000 7f3e13a8f700 1 --2- 172.21.15.189:0/2972456445 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e0c153ae0 0x7f3e0c0ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:34.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.790+0000 7f3e13a8f700 1 -- 172.21.15.189:0/2972456445 >> 172.21.15.189:0/2972456445 conn(0x7f3e0c0d00b0 msgr2=0x7f3e0c0aeb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:34.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.791+0000 7f3e13a8f700 1 -- 172.21.15.189:0/2972456445 shutdown_connections 2024-04-06T13:48:34.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.791+0000 7f3e13a8f700 1 -- 172.21.15.189:0/2972456445 wait complete. 2024-04-06T13:48:34.792 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'backup_fs_metadata' created 2024-04-06T13:48:34.806 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-06T13:48:34.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.940+0000 7f857dac5700 1 Processor -- start 2024-04-06T13:48:34.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.940+0000 7f857dac5700 1 -- start start 2024-04-06T13:48:34.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.941+0000 7f857dac5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85780d17a0 0x7f857806b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:34.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.941+0000 7f857dac5700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f85780cedf0 0x7f85780cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:34.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.941+0000 7f857dac5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85780cf760 con 0x7f8578146dd0 2024-04-06T13:48:34.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.941+0000 7f857dac5700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f857806d850 con 0x7f85780cedf0 2024-04-06T13:48:34.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.941+0000 7f857dac5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f857806c0a0 con 0x7f85780d17a0 2024-04-06T13:48:34.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.941+0000 7f8576ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8578146dd0 0x7f857806d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55828/0 (socket says 172.21.15.189:55828) 2024-04-06T13:48:34.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.941+0000 7f8576ffd700 1 -- 172.21.15.189:0/370869889 learned_addr learned my addr 172.21.15.189:0/370869889 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:34.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.941+0000 7f8577fff700 1 --2- 172.21.15.189:0/370869889 >> v2:172.21.15.192:3300/0 conn(0x7f85780cedf0 0x7f85780cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:34.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.941+0000 7f85777fe700 1 --2- 172.21.15.189:0/370869889 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85780d17a0 0x7f857806b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:34.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.941+0000 7f85767fc700 1 -- 172.21.15.189:0/370869889 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761398211 0 0) 0x7f85780cf760 con 0x7f8578146dd0 2024-04-06T13:48:34.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.941+0000 7f85767fc700 1 -- 172.21.15.189:0/370869889 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85600035b0 con 0x7f8578146dd0 2024-04-06T13:48:34.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85767fc700 1 -- 172.21.15.189:0/370869889 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2113640291 0 0) 0x7f85600035b0 con 0x7f8578146dd0 2024-04-06T13:48:34.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85767fc700 1 -- 172.21.15.189:0/370869889 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85780d17a0 msgr2=0x7f857806b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:34.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85767fc700 1 --2- 172.21.15.189:0/370869889 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85780d17a0 0x7f857806b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:34.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85767fc700 1 -- 172.21.15.189:0/370869889 >> v2:172.21.15.192:3300/0 conn(0x7f85780cedf0 msgr2=0x7f85780cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:34.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85767fc700 1 --2- 172.21.15.189:0/370869889 >> v2:172.21.15.192:3300/0 conn(0x7f85780cedf0 0x7f85780cf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:34.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85767fc700 1 -- 172.21.15.189:0/370869889 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857806c380 con 0x7f8578146dd0 2024-04-06T13:48:34.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85767fc700 1 -- 172.21.15.189:0/370869889 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f8568001670 con 0x7f8578146dd0 2024-04-06T13:48:34.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f8577fff700 1 --2- 172.21.15.189:0/370869889 >> v2:172.21.15.192:3300/0 conn(0x7f85780cedf0 0x7f85780cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:34.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85767fc700 1 -- 172.21.15.189:0/370869889 >> v1:172.21.15.189:6789/0 conn(0x7f8578146dd0 legacy=0x7f857806d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:34.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85767fc700 1 --2- 172.21.15.189:0/370869889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8560003a00 0x7f8560003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:34.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85767fc700 1 --2- 172.21.15.189:0/370869889 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8560004850 0x7f8560004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:34.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85767fc700 1 --2- 172.21.15.189:0/370869889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8560005210 0x7f856000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:34.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85767fc700 1 -- 172.21.15.189:0/370869889 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857806d850 con 0x7f8560003a00 2024-04-06T13:48:34.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85767fc700 1 -- 172.21.15.189:0/370869889 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857806c0a0 con 0x7f8560005210 2024-04-06T13:48:34.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85767fc700 1 -- 172.21.15.189:0/370869889 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856000bf50 con 0x7f8560004850 2024-04-06T13:48:34.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85777fe700 1 --2- 172.21.15.189:0/370869889 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85780d17a0 0x7f857806b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:34.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85777fe700 1 --2- 172.21.15.189:0/370869889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8560003a00 0x7f8560003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:34.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f8577fff700 1 --2- 172.21.15.189:0/370869889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8560005210 0x7f856000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:34.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f8576ffd700 1 --2- 172.21.15.189:0/370869889 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8560004850 0x7f8560004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:34.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85777fe700 1 -- 172.21.15.189:0/370869889 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8560004850 msgr2=0x7f8560004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:34.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85777fe700 1 --2- 172.21.15.189:0/370869889 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8560004850 0x7f8560004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:34.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85777fe700 1 -- 172.21.15.189:0/370869889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8560005210 msgr2=0x7f856000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:34.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85777fe700 1 --2- 172.21.15.189:0/370869889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8560005210 0x7f856000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:34.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85777fe700 1 -- 172.21.15.189:0/370869889 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85600035b0 con 0x7f8560003a00 2024-04-06T13:48:34.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85777fe700 1 --2- 172.21.15.189:0/370869889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8560003a00 0x7f8560003e20 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f856c009ba0 tx=0x7f8560005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=572657664fd0171f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:34.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.942+0000 7f85767fc700 1 -- 172.21.15.189:0/370869889 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f856c004030 con 0x7f8560003a00 2024-04-06T13:48:34.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.943+0000 7f85767fc700 1 -- 172.21.15.189:0/370869889 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f856c004190 con 0x7f8560003a00 2024-04-06T13:48:34.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.943+0000 7f85767fc700 1 -- 172.21.15.189:0/370869889 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f856c004540 con 0x7f8560003a00 2024-04-06T13:48:34.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.943+0000 7f857dac5700 1 -- 172.21.15.189:0/370869889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8560003a00 msgr2=0x7f8560003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:34.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.943+0000 7f857dac5700 1 --2- 172.21.15.189:0/370869889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8560003a00 0x7f8560003e20 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f856c009ba0 tx=0x7f8560005880 comp rx=0 tx=0).stop 2024-04-06T13:48:34.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.943+0000 7f8577fff700 1 -- 172.21.15.189:0/370869889 reap_dead start 2024-04-06T13:48:34.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.943+0000 7f857dac5700 1 -- 172.21.15.189:0/370869889 shutdown_connections 2024-04-06T13:48:34.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.943+0000 7f857dac5700 1 -- 172.21.15.189:0/370869889 >> 172.21.15.189:0/370869889 conn(0x7f85780b2db0 msgr2=0x7f85780b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:34.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.943+0000 7f857dac5700 1 -- 172.21.15.189:0/370869889 shutdown_connections 2024-04-06T13:48:34.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.943+0000 7f857dac5700 1 -- 172.21.15.189:0/370869889 wait complete. 2024-04-06T13:48:34.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.944+0000 7f857dac5700 1 Processor -- start 2024-04-06T13:48:34.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.944+0000 7f857dac5700 1 -- start start 2024-04-06T13:48:34.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.945+0000 7f857dac5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8560003a00 0x7f85780ad030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:34.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.945+0000 7f857dac5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f857815aee0 0x7f85780b0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:34.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.945+0000 7f857dac5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85780b5eb0 0x7f85780b6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:34.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.945+0000 7f857dac5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85780b68a0 con 0x7f85780b5eb0 2024-04-06T13:48:34.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.945+0000 7f857dac5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85780c6170 con 0x7f8560003a00 2024-04-06T13:48:34.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.945+0000 7f857dac5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f857815b360 con 0x7f857815aee0 2024-04-06T13:48:34.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.945+0000 7f8577fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85780b5eb0 0x7f85780b6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:34.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.945+0000 7f8577fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85780b5eb0 0x7f85780b6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49440/0 (socket says 172.21.15.189:49440) 2024-04-06T13:48:34.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.945+0000 7f8577fff700 1 -- 172.21.15.189:0/412863615 learned_addr learned my addr 172.21.15.189:0/412863615 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:34.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.945+0000 7f85777fe700 1 --2- 172.21.15.189:0/412863615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8560003a00 0x7f85780ad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:34.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.945+0000 7f8576ffd700 1 --2- 172.21.15.189:0/412863615 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f857815aee0 0x7f85780b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:34.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.946+0000 7f8577fff700 1 -- 172.21.15.189:0/412863615 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f857815aee0 msgr2=0x7f85780b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:34.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.946+0000 7f8577fff700 1 --2- 172.21.15.189:0/412863615 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f857815aee0 0x7f85780b0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:34.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.946+0000 7f8577fff700 1 -- 172.21.15.189:0/412863615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8560003a00 msgr2=0x7f85780ad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:34.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.946+0000 7f8577fff700 1 --2- 172.21.15.189:0/412863615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8560003a00 0x7f85780ad030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:34.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.946+0000 7f8577fff700 1 -- 172.21.15.189:0/412863615 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85780ad610 con 0x7f85780b5eb0 2024-04-06T13:48:34.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.946+0000 7f85777fe700 1 --2- 172.21.15.189:0/412863615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8560003a00 0x7f85780ad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:34.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.946+0000 7f8577fff700 1 --2- 172.21.15.189:0/412863615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85780b5eb0 0x7f85780b6360 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f8578160fb0 tx=0x7f857000f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:34.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.946+0000 7f855ffff700 1 -- 172.21.15.189:0/412863615 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857000e040 con 0x7f85780b5eb0 2024-04-06T13:48:34.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.946+0000 7f855ffff700 1 -- 172.21.15.189:0/412863615 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f857000f040 con 0x7f85780b5eb0 2024-04-06T13:48:34.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.946+0000 7f855ffff700 1 -- 172.21.15.189:0/412863615 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8570003a20 con 0x7f85780b5eb0 2024-04-06T13:48:34.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.946+0000 7f857dac5700 1 -- 172.21.15.189:0/412863615 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85780ad8f0 con 0x7f85780b5eb0 2024-04-06T13:48:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.946+0000 7f857dac5700 1 -- 172.21.15.189:0/412863615 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85780c63c0 con 0x7f85780b5eb0 2024-04-06T13:48:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.947+0000 7f857dac5700 1 -- 172.21.15.189:0/412863615 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85780c65e0 con 0x7f85780b5eb0 2024-04-06T13:48:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.947+0000 7f855ffff700 1 -- 172.21.15.189:0/412863615 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f8570015430 con 0x7f85780b5eb0 2024-04-06T13:48:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.948+0000 7f855ffff700 1 --2- 172.21.15.189:0/412863615 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f856406aed0 0x7f856406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.948+0000 7f855ffff700 1 -- 172.21.15.189:0/412863615 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 6474+0+0 (secure 0 0 0) 0x7f857008a290 con 0x7f85780b5eb0 2024-04-06T13:48:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.948+0000 7f85777fe700 1 --2- 172.21.15.189:0/412863615 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f856406aed0 0x7f856406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.948+0000 7f85777fe700 1 --2- 172.21.15.189:0/412863615 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f856406aed0 0x7f856406d390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f856c00a010 tx=0x7f856c008070 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:34.953+0000 7f855ffff700 1 -- 172.21.15.189:0/412863615 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8570054d90 con 0x7f85780b5eb0 2024-04-06T13:48:35.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.148+0000 7f857dac5700 1 -- 172.21.15.189:0/412863615 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f85780ade10 con 0x7f85780b5eb0 2024-04-06T13:48:35.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.788+0000 7f855ffff700 1 -- 172.21.15.189:0/412863615 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v250) v1 ==== 176+0+0 (secure 0 0 0) 0x7f85700583b0 con 0x7f85780b5eb0 2024-04-06T13:48:35.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.793+0000 7f857dac5700 1 -- 172.21.15.189:0/412863615 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f856406aed0 msgr2=0x7f856406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:35.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.793+0000 7f857dac5700 1 --2- 172.21.15.189:0/412863615 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f856406aed0 0x7f856406d390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f856c00a010 tx=0x7f856c008070 comp rx=0 tx=0).stop 2024-04-06T13:48:35.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.793+0000 7f857dac5700 1 -- 172.21.15.189:0/412863615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85780b5eb0 msgr2=0x7f85780b6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:35.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.793+0000 7f857dac5700 1 --2- 172.21.15.189:0/412863615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85780b5eb0 0x7f85780b6360 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f8578160fb0 tx=0x7f857000f490 comp rx=0 tx=0).stop 2024-04-06T13:48:35.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.794+0000 7f857dac5700 1 -- 172.21.15.189:0/412863615 shutdown_connections 2024-04-06T13:48:35.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.794+0000 7f857dac5700 1 --2- 172.21.15.189:0/412863615 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f856406aed0 0x7f856406d390 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:35.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.794+0000 7f857dac5700 1 --2- 172.21.15.189:0/412863615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85780b5eb0 0x7f85780b6360 unknown :-1 s=CLOSED pgs=2724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:35.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.794+0000 7f857dac5700 1 --2- 172.21.15.189:0/412863615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8560003a00 0x7f85780ad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:35.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.794+0000 7f857dac5700 1 --2- 172.21.15.189:0/412863615 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f857815aee0 0x7f85780b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:35.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.794+0000 7f857dac5700 1 -- 172.21.15.189:0/412863615 >> 172.21.15.189:0/412863615 conn(0x7f85780b2db0 msgr2=0x7f85780b2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:35.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.798+0000 7f857dac5700 1 -- 172.21.15.189:0/412863615 shutdown_connections 2024-04-06T13:48:35.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.799+0000 7f857dac5700 1 -- 172.21.15.189:0/412863615 wait complete. 2024-04-06T13:48:35.808 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'backup_fs_data' created 2024-04-06T13:48:35.818 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2024-04-06T13:48:35.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.945+0000 7f308e1b0700 1 Processor -- start 2024-04-06T13:48:35.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.945+0000 7f308e1b0700 1 -- start start 2024-04-06T13:48:35.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.946+0000 7f308e1b0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f308806ae40 0x7f30880b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:35.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.946+0000 7f308e1b0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30880b3990 con 0x7f308815d7b0 2024-04-06T13:48:35.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.946+0000 7f308e1b0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30880b3060 con 0x7f308815ddf0 2024-04-06T13:48:35.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.946+0000 7f308e1b0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f308815e950 con 0x7f308806ae40 2024-04-06T13:48:35.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.946+0000 7f3086ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f308806ae40 0x7f30880b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:35.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.946+0000 7f30877fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f308815d7b0 0x7f308806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55844/0 (socket says 172.21.15.189:55844) 2024-04-06T13:48:35.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.946+0000 7f30877fe700 1 -- 172.21.15.189:0/135856487 learned_addr learned my addr 172.21.15.189:0/135856487 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:35.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.946+0000 7f3086ffd700 1 -- 172.21.15.189:0/135856487 >> v1:172.21.15.192:6789/0 conn(0x7f308815ddf0 legacy=0x7f308815e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:35.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.946+0000 7f3086ffd700 1 -- 172.21.15.189:0/135856487 >> v1:172.21.15.189:6789/0 conn(0x7f308815d7b0 legacy=0x7f308806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:35.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.946+0000 7f30867fc700 1 -- 172.21.15.189:0/135856487 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266122533 0 0) 0x7f30880b3990 con 0x7f308815d7b0 2024-04-06T13:48:35.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.946+0000 7f3086ffd700 1 -- 172.21.15.189:0/135856487 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30880b3340 con 0x7f308806ae40 2024-04-06T13:48:35.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.946+0000 7f3086ffd700 1 --2- 172.21.15.189:0/135856487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f308806ae40 0x7f30880b3e20 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f3078009840 tx=0x7f3078009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ebe4b9643d6b58 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:35.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.947+0000 7f30867fc700 1 -- 172.21.15.189:0/135856487 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078011830 con 0x7f308806ae40 2024-04-06T13:48:35.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.947+0000 7f30867fc700 1 -- 172.21.15.189:0/135856487 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3078011990 con 0x7f308806ae40 2024-04-06T13:48:35.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.947+0000 7f30867fc700 1 -- 172.21.15.189:0/135856487 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078011d20 con 0x7f308806ae40 2024-04-06T13:48:35.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.947+0000 7f308e1b0700 1 -- 172.21.15.189:0/135856487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f308806ae40 msgr2=0x7f30880b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:35.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.947+0000 7f308e1b0700 1 --2- 172.21.15.189:0/135856487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f308806ae40 0x7f30880b3e20 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f3078009840 tx=0x7f3078009b50 comp rx=0 tx=0).stop 2024-04-06T13:48:35.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.947+0000 7f308e1b0700 1 -- 172.21.15.189:0/135856487 shutdown_connections 2024-04-06T13:48:35.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.947+0000 7f308e1b0700 1 --2- 172.21.15.189:0/135856487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f308806ae40 0x7f30880b3e20 unknown :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:35.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.947+0000 7f308e1b0700 1 -- 172.21.15.189:0/135856487 >> 172.21.15.189:0/135856487 conn(0x7f30880d00b0 msgr2=0x7f30880d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:35.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.947+0000 7f308e1b0700 1 -- 172.21.15.189:0/135856487 shutdown_connections 2024-04-06T13:48:35.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.947+0000 7f308e1b0700 1 -- 172.21.15.189:0/135856487 wait complete. 2024-04-06T13:48:35.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.948+0000 7f308e1b0700 1 Processor -- start 2024-04-06T13:48:35.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.949+0000 7f308e1b0700 1 -- start start 2024-04-06T13:48:35.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.949+0000 7f308e1b0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f308806ae40 0x7f308815bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:35.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.949+0000 7f308e1b0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f308815d7b0 0x7f30880aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:35.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.949+0000 7f308e1b0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f308815ddf0 0x7f3088142480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:35.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.949+0000 7f308e1b0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30881429c0 con 0x7f308815d7b0 2024-04-06T13:48:35.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.949+0000 7f308e1b0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30880b1720 con 0x7f308806ae40 2024-04-06T13:48:35.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.949+0000 7f308e1b0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30880b1a00 con 0x7f308815ddf0 2024-04-06T13:48:35.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.950+0000 7f3086ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f308815d7b0 0x7f30880aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:35.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.950+0000 7f3086ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f308815d7b0 0x7f30880aeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49452/0 (socket says 172.21.15.189:49452) 2024-04-06T13:48:35.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.950+0000 7f3086ffd700 1 -- 172.21.15.189:0/3933057126 learned_addr learned my addr 172.21.15.189:0/3933057126 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:35.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.950+0000 7f3087fff700 1 --2- 172.21.15.189:0/3933057126 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f308815ddf0 0x7f3088142480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:35.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.950+0000 7f30877fe700 1 --2- 172.21.15.189:0/3933057126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f308806ae40 0x7f308815bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:35.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.950+0000 7f3086ffd700 1 -- 172.21.15.189:0/3933057126 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f308815ddf0 msgr2=0x7f3088142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:35.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.950+0000 7f3086ffd700 1 --2- 172.21.15.189:0/3933057126 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f308815ddf0 0x7f3088142480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:35.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.950+0000 7f3086ffd700 1 -- 172.21.15.189:0/3933057126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f308806ae40 msgr2=0x7f308815bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:35.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.950+0000 7f3086ffd700 1 --2- 172.21.15.189:0/3933057126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f308806ae40 0x7f308815bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:35.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.950+0000 7f3086ffd700 1 -- 172.21.15.189:0/3933057126 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30780094a0 con 0x7f308815d7b0 2024-04-06T13:48:35.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.950+0000 7f30877fe700 1 --2- 172.21.15.189:0/3933057126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f308806ae40 0x7f308815bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:35.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.950+0000 7f3087fff700 1 --2- 172.21.15.189:0/3933057126 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f308815ddf0 0x7f3088142480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:35.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.950+0000 7f3086ffd700 1 --2- 172.21.15.189:0/3933057126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f308815d7b0 0x7f30880aeb80 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7f30780090b0 tx=0x7f3078004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:35.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.950+0000 7f307ffff700 1 -- 172.21.15.189:0/3933057126 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307801a900 con 0x7f308815d7b0 2024-04-06T13:48:35.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.950+0000 7f307ffff700 1 -- 172.21.15.189:0/3933057126 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307801aa60 con 0x7f308815d7b0 2024-04-06T13:48:35.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.950+0000 7f308e1b0700 1 -- 172.21.15.189:0/3933057126 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f308815afd0 con 0x7f308815d7b0 2024-04-06T13:48:35.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.950+0000 7f307ffff700 1 -- 172.21.15.189:0/3933057126 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307801add0 con 0x7f308815d7b0 2024-04-06T13:48:35.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.950+0000 7f308e1b0700 1 -- 172.21.15.189:0/3933057126 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f308815b590 con 0x7f308815d7b0 2024-04-06T13:48:35.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.951+0000 7f308e1b0700 1 -- 172.21.15.189:0/3933057126 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3088067bb0 con 0x7f308815d7b0 2024-04-06T13:48:35.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.952+0000 7f307ffff700 1 -- 172.21.15.189:0/3933057126 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f3078022070 con 0x7f308815d7b0 2024-04-06T13:48:35.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.952+0000 7f307ffff700 1 --2- 172.21.15.189:0/3933057126 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f306806aed0 0x7f306806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:35.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.952+0000 7f307ffff700 1 -- 172.21.15.189:0/3933057126 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 6859+0+0 (secure 0 0 0) 0x7f307808aaa0 con 0x7f308815d7b0 2024-04-06T13:48:35.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.952+0000 7f30877fe700 1 --2- 172.21.15.189:0/3933057126 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f306806aed0 0x7f306806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:35.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.953+0000 7f30877fe700 1 --2- 172.21.15.189:0/3933057126 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f306806aed0 0x7f306806d390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3070000f60 tx=0x7f3070007ce0 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:35.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:35.957+0000 7f307ffff700 1 -- 172.21.15.189:0/3933057126 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3078055560 con 0x7f308815d7b0 2024-04-06T13:48:36.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.195+0000 7f308e1b0700 1 -- 172.21.15.189:0/3933057126 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7f3088074990 con 0x7f308815d7b0 2024-04-06T13:48:36.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.202+0000 7f307ffff700 1 -- 172.21.15.189:0/3933057126 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 33 and data pool 34 v118) v1 ==== 183+0+0 (secure 0 0 0) 0x7f3078058b80 con 0x7f308815d7b0 2024-04-06T13:48:36.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.220+0000 7f308e1b0700 1 -- 172.21.15.189:0/3933057126 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f306806aed0 msgr2=0x7f306806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:36.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.220+0000 7f308e1b0700 1 --2- 172.21.15.189:0/3933057126 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f306806aed0 0x7f306806d390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3070000f60 tx=0x7f3070007ce0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.220+0000 7f308e1b0700 1 -- 172.21.15.189:0/3933057126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f308815d7b0 msgr2=0x7f30880aeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:36.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.220+0000 7f308e1b0700 1 --2- 172.21.15.189:0/3933057126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f308815d7b0 0x7f30880aeb80 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7f30780090b0 tx=0x7f3078004750 comp rx=0 tx=0).stop 2024-04-06T13:48:36.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.225+0000 7f308e1b0700 1 -- 172.21.15.189:0/3933057126 shutdown_connections 2024-04-06T13:48:36.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.225+0000 7f308e1b0700 1 --2- 172.21.15.189:0/3933057126 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f306806aed0 0x7f306806d390 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.225+0000 7f308e1b0700 1 --2- 172.21.15.189:0/3933057126 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f308815ddf0 0x7f3088142480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.225+0000 7f308e1b0700 1 --2- 172.21.15.189:0/3933057126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f308806ae40 0x7f308815bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.225+0000 7f308e1b0700 1 --2- 172.21.15.189:0/3933057126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f308815d7b0 0x7f30880aeb80 unknown :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.225+0000 7f308e1b0700 1 -- 172.21.15.189:0/3933057126 >> 172.21.15.189:0/3933057126 conn(0x7f30880d00b0 msgr2=0x7f30880ae1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:36.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.225+0000 7f308e1b0700 1 -- 172.21.15.189:0/3933057126 shutdown_connections 2024-04-06T13:48:36.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.225+0000 7f308e1b0700 1 -- 172.21.15.189:0/3933057126 wait complete. 2024-04-06T13:48:36.231 INFO:teuthology.orchestra.run.smithi189.stderr:new fs with metadata pool 33 and data pool 34 2024-04-06T13:48:36.241 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:48:36.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.362+0000 7f2f97dbf700 1 Processor -- start 2024-04-06T13:48:36.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.362+0000 7f2f97dbf700 1 -- start start 2024-04-06T13:48:36.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.363+0000 7f2f97dbf700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2f9015d7b0 0x7f2f900b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:36.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.363+0000 7f2f97dbf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f9006a730 0x7f2f9006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:36.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.363+0000 7f2f97dbf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f900b3990 con 0x7f2f9015ddf0 2024-04-06T13:48:36.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.363+0000 7f2f97dbf700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f9006b0f0 con 0x7f2f9015d7b0 2024-04-06T13:48:36.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.363+0000 7f2f97dbf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f900b3060 con 0x7f2f9006a730 2024-04-06T13:48:36.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.363+0000 7f2f9635c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2f9015ddf0 0x7f2f9015e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55846/0 (socket says 172.21.15.189:55846) 2024-04-06T13:48:36.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.363+0000 7f2f9635c700 1 -- 172.21.15.189:0/407656350 learned_addr learned my addr 172.21.15.189:0/407656350 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:36.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f9535a700 1 --2- 172.21.15.189:0/407656350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f9006a730 0x7f2f9006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:36.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f94b59700 1 -- 172.21.15.189:0/407656350 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195152623 0 0) 0x7f2f900b3990 con 0x7f2f9015ddf0 2024-04-06T13:48:36.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f94b59700 1 -- 172.21.15.189:0/407656350 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f7c0035b0 con 0x7f2f9015ddf0 2024-04-06T13:48:36.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f95b5b700 1 --2- 172.21.15.189:0/407656350 >> v2:172.21.15.192:3300/0 conn(0x7f2f9015d7b0 0x7f2f900b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:36.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f94b59700 1 -- 172.21.15.189:0/407656350 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3049004493 0 0) 0x7f2f7c0035b0 con 0x7f2f9015ddf0 2024-04-06T13:48:36.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f94b59700 1 -- 172.21.15.189:0/407656350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f9006a730 msgr2=0x7f2f9006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:36.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f94b59700 1 --2- 172.21.15.189:0/407656350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f9006a730 0x7f2f9006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f94b59700 1 -- 172.21.15.189:0/407656350 >> v2:172.21.15.192:3300/0 conn(0x7f2f9015d7b0 msgr2=0x7f2f900b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:36.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f94b59700 1 --2- 172.21.15.189:0/407656350 >> v2:172.21.15.192:3300/0 conn(0x7f2f9015d7b0 0x7f2f900b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f94b59700 1 -- 172.21.15.189:0/407656350 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f900b3340 con 0x7f2f9015ddf0 2024-04-06T13:48:36.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f94b59700 1 -- 172.21.15.189:0/407656350 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f2f880019f0 con 0x7f2f9015ddf0 2024-04-06T13:48:36.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f94b59700 1 -- 172.21.15.189:0/407656350 >> v1:172.21.15.189:6789/0 conn(0x7f2f9015ddf0 legacy=0x7f2f9015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:36.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f94b59700 1 --2- 172.21.15.189:0/407656350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f7c003a00 0x7f2f7c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:36.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f94b59700 1 --2- 172.21.15.189:0/407656350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f7c004850 0x7f2f7c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:36.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f94b59700 1 --2- 172.21.15.189:0/407656350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f9006a730 0x7f2f7c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:36.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f94b59700 1 -- 172.21.15.189:0/407656350 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9006b0f0 con 0x7f2f7c003a00 2024-04-06T13:48:36.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f94b59700 1 -- 172.21.15.189:0/407656350 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f900b3060 con 0x7f2f7c004850 2024-04-06T13:48:36.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f94b59700 1 -- 172.21.15.189:0/407656350 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7c00ba10 con 0x7f2f9006a730 2024-04-06T13:48:36.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f95b5b700 1 --2- 172.21.15.189:0/407656350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f7c003a00 0x7f2f7c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:36.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f9535a700 1 --2- 172.21.15.189:0/407656350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f7c004850 0x7f2f7c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:36.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f9635c700 1 --2- 172.21.15.189:0/407656350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f9006a730 0x7f2f7c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:36.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f95b5b700 1 -- 172.21.15.189:0/407656350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f9006a730 msgr2=0x7f2f7c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:36.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f95b5b700 1 --2- 172.21.15.189:0/407656350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f9006a730 0x7f2f7c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f95b5b700 1 -- 172.21.15.189:0/407656350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f7c004850 msgr2=0x7f2f7c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:36.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f95b5b700 1 --2- 172.21.15.189:0/407656350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f7c004850 0x7f2f7c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.364+0000 7f2f95b5b700 1 -- 172.21.15.189:0/407656350 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7c0035b0 con 0x7f2f7c003a00 2024-04-06T13:48:36.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.365+0000 7f2f95b5b700 1 --2- 172.21.15.189:0/407656350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f7c003a00 0x7f2f7c003e20 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f2f80009b90 tx=0x7f2f7c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b37ce59d4b9cbd54 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:36.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.365+0000 7f2f94b59700 1 -- 172.21.15.189:0/407656350 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8000e6c0 con 0x7f2f7c003a00 2024-04-06T13:48:36.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.365+0000 7f2f94b59700 1 -- 172.21.15.189:0/407656350 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8000e820 con 0x7f2f7c003a00 2024-04-06T13:48:36.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.365+0000 7f2f94b59700 1 -- 172.21.15.189:0/407656350 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8000eb90 con 0x7f2f7c003a00 2024-04-06T13:48:36.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.365+0000 7f2f97dbf700 1 -- 172.21.15.189:0/407656350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f7c003a00 msgr2=0x7f2f7c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:36.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.365+0000 7f2f97dbf700 1 --2- 172.21.15.189:0/407656350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f7c003a00 0x7f2f7c003e20 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f2f80009b90 tx=0x7f2f7c005880 comp rx=0 tx=0).stop 2024-04-06T13:48:36.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.365+0000 7f2f9635c700 1 -- 172.21.15.189:0/407656350 reap_dead start 2024-04-06T13:48:36.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.365+0000 7f2f97dbf700 1 -- 172.21.15.189:0/407656350 shutdown_connections 2024-04-06T13:48:36.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.365+0000 7f2f97dbf700 1 -- 172.21.15.189:0/407656350 >> 172.21.15.189:0/407656350 conn(0x7f2f900d00b0 msgr2=0x7f2f900d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:36.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.365+0000 7f2f97dbf700 1 -- 172.21.15.189:0/407656350 shutdown_connections 2024-04-06T13:48:36.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.365+0000 7f2f97dbf700 1 -- 172.21.15.189:0/407656350 wait complete. 2024-04-06T13:48:36.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.366+0000 7f2f97dbf700 1 Processor -- start 2024-04-06T13:48:36.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.367+0000 7f2f97dbf700 1 -- start start 2024-04-06T13:48:36.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.367+0000 7f2f97dbf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f7c003a00 0x7f2f900ce350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:36.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.367+0000 7f2f97dbf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f900717b0 0x7f2f90071c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:36.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.367+0000 7f2f97dbf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f9015a430 0x7f2f9015a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:36.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.367+0000 7f2f97dbf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9015b180 con 0x7f2f7c003a00 2024-04-06T13:48:36.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.367+0000 7f2f97dbf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f90072170 con 0x7f2f9015a430 2024-04-06T13:48:36.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.367+0000 7f2f97dbf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f901f6110 con 0x7f2f900717b0 2024-04-06T13:48:36.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.367+0000 7f2f95b5b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f7c003a00 0x7f2f900ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:36.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.367+0000 7f2f95b5b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f7c003a00 0x7f2f900ce350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49466/0 (socket says 172.21.15.189:49466) 2024-04-06T13:48:36.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.367+0000 7f2f95b5b700 1 -- 172.21.15.189:0/328499077 learned_addr learned my addr 172.21.15.189:0/328499077 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:36.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.367+0000 7f2f9635c700 1 --2- 172.21.15.189:0/328499077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f9015a430 0x7f2f9015a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:36.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.368+0000 7f2f9535a700 1 --2- 172.21.15.189:0/328499077 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f900717b0 0x7f2f90071c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:36.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.368+0000 7f2f95b5b700 1 -- 172.21.15.189:0/328499077 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f900717b0 msgr2=0x7f2f90071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:36.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.368+0000 7f2f95b5b700 1 --2- 172.21.15.189:0/328499077 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f900717b0 0x7f2f90071c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.368+0000 7f2f95b5b700 1 -- 172.21.15.189:0/328499077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f9015a430 msgr2=0x7f2f9015a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:36.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.368+0000 7f2f95b5b700 1 --2- 172.21.15.189:0/328499077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f9015a430 0x7f2f9015a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.368+0000 7f2f95b5b700 1 -- 172.21.15.189:0/328499077 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f901f63f0 con 0x7f2f7c003a00 2024-04-06T13:48:36.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.368+0000 7f2f9635c700 1 --2- 172.21.15.189:0/328499077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f9015a430 0x7f2f9015a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:36.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.368+0000 7f2f95b5b700 1 --2- 172.21.15.189:0/328499077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f7c003a00 0x7f2f900ce350 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c004740 tx=0x7f2f800047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:36.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.368+0000 7f2f867fc700 1 -- 172.21.15.189:0/328499077 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f80021070 con 0x7f2f7c003a00 2024-04-06T13:48:36.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.368+0000 7f2f867fc700 1 -- 172.21.15.189:0/328499077 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f80004550 con 0x7f2f7c003a00 2024-04-06T13:48:36.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.368+0000 7f2f97dbf700 1 -- 172.21.15.189:0/328499077 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f901f66d0 con 0x7f2f7c003a00 2024-04-06T13:48:36.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.368+0000 7f2f867fc700 1 -- 172.21.15.189:0/328499077 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8000e6c0 con 0x7f2f7c003a00 2024-04-06T13:48:36.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.368+0000 7f2f97dbf700 1 -- 172.21.15.189:0/328499077 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f9015ae20 con 0x7f2f7c003a00 2024-04-06T13:48:36.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.369+0000 7f2f97dbf700 1 -- 172.21.15.189:0/328499077 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f901f6250 con 0x7f2f7c003a00 2024-04-06T13:48:36.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.369+0000 7f2f867fc700 1 -- 172.21.15.189:0/328499077 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f2f80025070 con 0x7f2f7c003a00 2024-04-06T13:48:36.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.370+0000 7f2f867fc700 1 --2- 172.21.15.189:0/328499077 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2f7406aed0 0x7f2f7406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:36.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.370+0000 7f2f867fc700 1 -- 172.21.15.189:0/328499077 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f2f8008ae60 con 0x7f2f7c003a00 2024-04-06T13:48:36.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.370+0000 7f2f9535a700 1 --2- 172.21.15.189:0/328499077 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2f7406aed0 0x7f2f7406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:36.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.370+0000 7f2f9535a700 1 --2- 172.21.15.189:0/328499077 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2f7406aed0 0x7f2f7406d390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2f90073e00 tx=0x7f2f8c008040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:36.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.374+0000 7f2f867fc700 1 -- 172.21.15.189:0/328499077 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2f901f6250 con 0x7f2f7c003a00 2024-04-06T13:48:36.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.572+0000 7f2f97dbf700 1 -- 172.21.15.189:0/328499077 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f901f6250 con 0x7f2f7c003a00 2024-04-06T13:48:36.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.573+0000 7f2f867fc700 1 -- 172.21.15.189:0/328499077 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v251) v1 ==== 74+0+20153 (secure 0 0 0) 0x7f2f901f6250 con 0x7f2f7c003a00 2024-04-06T13:48:36.574 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:36.574 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":251,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:48:36.196680+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:48:17.447247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-04-06T13:48:18.646229+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:48:33.912100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:48:35.170642+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:48:36.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.576+0000 7f2f97dbf700 1 -- 172.21.15.189:0/328499077 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2f7406aed0 msgr2=0x7f2f7406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:36.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.576+0000 7f2f97dbf700 1 --2- 172.21.15.189:0/328499077 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2f7406aed0 0x7f2f7406d390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2f90073e00 tx=0x7f2f8c008040 comp rx=0 tx=0).stop 2024-04-06T13:48:36.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.576+0000 7f2f97dbf700 1 -- 172.21.15.189:0/328499077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f7c003a00 msgr2=0x7f2f900ce350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:36.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.576+0000 7f2f97dbf700 1 --2- 172.21.15.189:0/328499077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f7c003a00 0x7f2f900ce350 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c004740 tx=0x7f2f800047c0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.576+0000 7f2f97dbf700 1 -- 172.21.15.189:0/328499077 shutdown_connections 2024-04-06T13:48:36.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.576+0000 7f2f97dbf700 1 --2- 172.21.15.189:0/328499077 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2f7406aed0 0x7f2f7406d390 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.576+0000 7f2f97dbf700 1 --2- 172.21.15.189:0/328499077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f9015a430 0x7f2f9015a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.576+0000 7f2f97dbf700 1 --2- 172.21.15.189:0/328499077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f7c003a00 0x7f2f900ce350 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.576+0000 7f2f97dbf700 1 --2- 172.21.15.189:0/328499077 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f900717b0 0x7f2f90071c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.576+0000 7f2f97dbf700 1 -- 172.21.15.189:0/328499077 >> 172.21.15.189:0/328499077 conn(0x7f2f900d00b0 msgr2=0x7f2f900ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:36.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.576+0000 7f2f97dbf700 1 -- 172.21.15.189:0/328499077 shutdown_connections 2024-04-06T13:48:36.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.576+0000 7f2f97dbf700 1 -- 172.21.15.189:0/328499077 wait complete. 2024-04-06T13:48:36.588 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:48:36.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.710+0000 7fe8b6dd7700 1 Processor -- start 2024-04-06T13:48:36.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.710+0000 7fe8b6dd7700 1 -- start start 2024-04-06T13:48:36.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.710+0000 7fe8b6dd7700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe8b00d11b0 0x7fe8b00d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:36.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.710+0000 7fe8b6dd7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b00d1ad0 con 0x7fe8b00cfa00 2024-04-06T13:48:36.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.710+0000 7fe8b6dd7700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b00cbd80 con 0x7fe8b00cb870 2024-04-06T13:48:36.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.710+0000 7fe8b6dd7700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8b00ccdc0 con 0x7fe8b00d11b0 2024-04-06T13:48:36.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8b5374700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe8b00cfa00 0x7fe8b00cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55858/0 (socket says 172.21.15.189:55858) 2024-04-06T13:48:36.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8b5374700 1 -- 172.21.15.189:0/184590816 learned_addr learned my addr 172.21.15.189:0/184590816 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:36.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8b4b73700 1 --2- 172.21.15.189:0/184590816 >> v2:172.21.15.192:3300/0 conn(0x7fe8b00d11b0 0x7fe8b00d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:36.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8af7fe700 1 -- 172.21.15.189:0/184590816 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087498931 0 0) 0x7fe8b00d1ad0 con 0x7fe8b00cfa00 2024-04-06T13:48:36.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8af7fe700 1 -- 172.21.15.189:0/184590816 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8a00035b0 con 0x7fe8b00cfa00 2024-04-06T13:48:36.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8af7fe700 1 -- 172.21.15.189:0/184590816 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204922519 0 0) 0x7fe8b00cbd80 con 0x7fe8b00cb870 2024-04-06T13:48:36.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8af7fe700 1 -- 172.21.15.189:0/184590816 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b00d1ad0 con 0x7fe8b00cb870 2024-04-06T13:48:36.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8af7fe700 1 -- 172.21.15.189:0/184590816 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4252242859 0 0) 0x7fe8a00035b0 con 0x7fe8b00cfa00 2024-04-06T13:48:36.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8af7fe700 1 -- 172.21.15.189:0/184590816 >> v2:172.21.15.192:3300/0 conn(0x7fe8b00d11b0 msgr2=0x7fe8b00d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:36.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8af7fe700 1 --2- 172.21.15.189:0/184590816 >> v2:172.21.15.192:3300/0 conn(0x7fe8b00d11b0 0x7fe8b00d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8af7fe700 1 -- 172.21.15.189:0/184590816 >> v1:172.21.15.192:6789/0 conn(0x7fe8b00cb870 legacy=0x7fe8b015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:36.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8af7fe700 1 -- 172.21.15.189:0/184590816 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b00cef20 con 0x7fe8b00cfa00 2024-04-06T13:48:36.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8af7fe700 1 -- 172.21.15.189:0/184590816 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fe8a80019f0 con 0x7fe8b00cfa00 2024-04-06T13:48:36.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8b4b73700 1 --2- 172.21.15.189:0/184590816 >> v2:172.21.15.192:3300/0 conn(0x7fe8b00d11b0 0x7fe8b00d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:36.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8af7fe700 1 -- 172.21.15.189:0/184590816 >> v1:172.21.15.189:6789/0 conn(0x7fe8b00cfa00 legacy=0x7fe8b00cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:36.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8af7fe700 1 --2- 172.21.15.189:0/184590816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a0003b90 0x7fe8a0003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:36.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8af7fe700 1 --2- 172.21.15.189:0/184590816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8a0004850 0x7fe8a0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:36.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8af7fe700 1 --2- 172.21.15.189:0/184590816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8a0005210 0x7fe8a000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:36.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8af7fe700 1 -- 172.21.15.189:0/184590816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a000c110 con 0x7fe8a0005210 2024-04-06T13:48:36.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8af7fe700 1 -- 172.21.15.189:0/184590816 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a000c280 con 0x7fe8a0003b90 2024-04-06T13:48:36.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8af7fe700 1 -- 172.21.15.189:0/184590816 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a000c440 con 0x7fe8a0004850 2024-04-06T13:48:36.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8b5374700 1 --2- 172.21.15.189:0/184590816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8a0005210 0x7fe8a000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:36.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8b4b73700 1 --2- 172.21.15.189:0/184590816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a0003b90 0x7fe8a0003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:36.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.711+0000 7fe8affff700 1 --2- 172.21.15.189:0/184590816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8a0004850 0x7fe8a0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:36.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.712+0000 7fe8b5374700 1 -- 172.21.15.189:0/184590816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8a0004850 msgr2=0x7fe8a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:36.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.712+0000 7fe8b5374700 1 --2- 172.21.15.189:0/184590816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8a0004850 0x7fe8a0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.712+0000 7fe8b5374700 1 -- 172.21.15.189:0/184590816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a0003b90 msgr2=0x7fe8a0003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:36.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.712+0000 7fe8b5374700 1 --2- 172.21.15.189:0/184590816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a0003b90 0x7fe8a0003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.712+0000 7fe8b5374700 1 -- 172.21.15.189:0/184590816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a00035b0 con 0x7fe8a0005210 2024-04-06T13:48:36.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.712+0000 7fe8b5374700 1 -- 172.21.15.189:0/184590816 reap_dead start 2024-04-06T13:48:36.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.712+0000 7fe8b5374700 1 --2- 172.21.15.189:0/184590816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8a0005210 0x7fe8a000ba10 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7fe8a800a7c0 tx=0x7fe8a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11d3d3ea8cd726e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:36.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.712+0000 7fe8af7fe700 1 -- 172.21.15.189:0/184590816 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a8012600 con 0x7fe8a0005210 2024-04-06T13:48:36.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.712+0000 7fe8af7fe700 1 -- 172.21.15.189:0/184590816 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8a8012760 con 0x7fe8a0005210 2024-04-06T13:48:36.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.712+0000 7fe8af7fe700 1 -- 172.21.15.189:0/184590816 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a8012b10 con 0x7fe8a0005210 2024-04-06T13:48:36.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.712+0000 7fe8b6dd7700 1 -- 172.21.15.189:0/184590816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8a0005210 msgr2=0x7fe8a000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:36.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.712+0000 7fe8b6dd7700 1 --2- 172.21.15.189:0/184590816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8a0005210 0x7fe8a000ba10 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7fe8a800a7c0 tx=0x7fe8a0005880 comp rx=0 tx=0).stop 2024-04-06T13:48:36.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.712+0000 7fe8b6dd7700 1 -- 172.21.15.189:0/184590816 shutdown_connections 2024-04-06T13:48:36.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.712+0000 7fe8b6dd7700 1 --2- 172.21.15.189:0/184590816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8a0005210 0x7fe8a000ba10 unknown :-1 s=CLOSED pgs=2732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.712+0000 7fe8b6dd7700 1 -- 172.21.15.189:0/184590816 >> 172.21.15.189:0/184590816 conn(0x7fe8b00b2db0 msgr2=0x7fe8b00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:36.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.712+0000 7fe8b6dd7700 1 -- 172.21.15.189:0/184590816 shutdown_connections 2024-04-06T13:48:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.712+0000 7fe8b6dd7700 1 -- 172.21.15.189:0/184590816 wait complete. 2024-04-06T13:48:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.714+0000 7fe8b6dd7700 1 Processor -- start 2024-04-06T13:48:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.714+0000 7fe8b6dd7700 1 -- start start 2024-04-06T13:48:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.715+0000 7fe8b6dd7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8a0005210 0x7fe8b01ee500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.715+0000 7fe8b6dd7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8b01f2040 0x7fe8b00c6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.715+0000 7fe8b6dd7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8b00c73f0 0x7fe8b01f1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.715+0000 7fe8b6dd7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b01f1ec0 con 0x7fe8a0005210 2024-04-06T13:48:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.715+0000 7fe8b6dd7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b00c7870 con 0x7fe8b01f2040 2024-04-06T13:48:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.715+0000 7fe8b6dd7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b00c4ef0 con 0x7fe8b00c73f0 2024-04-06T13:48:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.715+0000 7fe8b4b73700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8a0005210 0x7fe8b01ee500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.715+0000 7fe8b4b73700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8a0005210 0x7fe8b01ee500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49474/0 (socket says 172.21.15.189:49474) 2024-04-06T13:48:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.715+0000 7fe8b4b73700 1 -- 172.21.15.189:0/3096394131 learned_addr learned my addr 172.21.15.189:0/3096394131 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.715+0000 7fe8affff700 1 --2- 172.21.15.189:0/3096394131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8b01f2040 0x7fe8b00c6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:36.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.715+0000 7fe8b5374700 1 --2- 172.21.15.189:0/3096394131 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8b00c73f0 0x7fe8b01f1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.715+0000 7fe8b4b73700 1 -- 172.21.15.189:0/3096394131 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8b00c73f0 msgr2=0x7fe8b01f1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.715+0000 7fe8b4b73700 1 --2- 172.21.15.189:0/3096394131 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8b00c73f0 0x7fe8b01f1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.715+0000 7fe8b4b73700 1 -- 172.21.15.189:0/3096394131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8b01f2040 msgr2=0x7fe8b00c6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.715+0000 7fe8b4b73700 1 --2- 172.21.15.189:0/3096394131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8b01f2040 0x7fe8b00c6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.716+0000 7fe8b4b73700 1 -- 172.21.15.189:0/3096394131 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b00c51d0 con 0x7fe8a0005210 2024-04-06T13:48:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.716+0000 7fe8affff700 1 --2- 172.21.15.189:0/3096394131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8b01f2040 0x7fe8b00c6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.716+0000 7fe8b4b73700 1 --2- 172.21.15.189:0/3096394131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8a0005210 0x7fe8b01ee500 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4011ca0 tx=0x7fe8a400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.716+0000 7fe8ad7fa700 1 -- 172.21.15.189:0/3096394131 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a400fb30 con 0x7fe8a0005210 2024-04-06T13:48:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.716+0000 7fe8ad7fa700 1 -- 172.21.15.189:0/3096394131 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8a400fc90 con 0x7fe8a0005210 2024-04-06T13:48:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.716+0000 7fe8b6dd7700 1 -- 172.21.15.189:0/3096394131 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b00c54b0 con 0x7fe8a0005210 2024-04-06T13:48:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.716+0000 7fe8ad7fa700 1 -- 172.21.15.189:0/3096394131 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a4004020 con 0x7fe8a0005210 2024-04-06T13:48:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.716+0000 7fe8b6dd7700 1 -- 172.21.15.189:0/3096394131 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b01f24c0 con 0x7fe8a0005210 2024-04-06T13:48:36.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.717+0000 7fe8b6dd7700 1 -- 172.21.15.189:0/3096394131 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b00c5b00 con 0x7fe8a0005210 2024-04-06T13:48:36.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.717+0000 7fe8ad7fa700 1 -- 172.21.15.189:0/3096394131 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fe8a4004180 con 0x7fe8a0005210 2024-04-06T13:48:36.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.718+0000 7fe8ad7fa700 1 --2- 172.21.15.189:0/3096394131 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe89406aed0 0x7fe89406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:36.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.718+0000 7fe8ad7fa700 1 -- 172.21.15.189:0/3096394131 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 6965+0+0 (secure 0 0 0) 0x7fe8a408a240 con 0x7fe8a0005210 2024-04-06T13:48:36.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.718+0000 7fe8affff700 1 --2- 172.21.15.189:0/3096394131 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe89406aed0 0x7fe89406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:36.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.718+0000 7fe8affff700 1 --2- 172.21.15.189:0/3096394131 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe89406aed0 0x7fe89406d390 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe8b015fd40 tx=0x7fe89c009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:36.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.722+0000 7fe8ad7fa700 1 -- 172.21.15.189:0/3096394131 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe8a4054b30 con 0x7fe8a0005210 2024-04-06T13:48:36.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.923+0000 7fe8b6dd7700 1 -- 172.21.15.189:0/3096394131 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8b0074990 con 0x7fe8a0005210 2024-04-06T13:48:36.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.924+0000 7fe8ad7fa700 1 -- 172.21.15.189:0/3096394131 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v251) v1 ==== 74+0+20153 (secure 0 0 0) 0x7fe8a4058150 con 0x7fe8a0005210 2024-04-06T13:48:36.924 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:36.924 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":251,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:48:36.196680+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:48:17.447247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-04-06T13:48:18.646229+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:48:33.912100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:48:35.170642+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:48:36.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.927+0000 7fe8b6dd7700 1 -- 172.21.15.189:0/3096394131 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe89406aed0 msgr2=0x7fe89406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:36.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.927+0000 7fe8b6dd7700 1 --2- 172.21.15.189:0/3096394131 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe89406aed0 0x7fe89406d390 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe8b015fd40 tx=0x7fe89c009040 comp rx=0 tx=0).stop 2024-04-06T13:48:36.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.927+0000 7fe8b6dd7700 1 -- 172.21.15.189:0/3096394131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8a0005210 msgr2=0x7fe8b01ee500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:36.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.927+0000 7fe8b6dd7700 1 --2- 172.21.15.189:0/3096394131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8a0005210 0x7fe8b01ee500 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4011ca0 tx=0x7fe8a400f520 comp rx=0 tx=0).stop 2024-04-06T13:48:36.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.927+0000 7fe8b6dd7700 1 -- 172.21.15.189:0/3096394131 shutdown_connections 2024-04-06T13:48:36.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.927+0000 7fe8b6dd7700 1 --2- 172.21.15.189:0/3096394131 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe89406aed0 0x7fe89406d390 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.927+0000 7fe8b6dd7700 1 --2- 172.21.15.189:0/3096394131 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8b00c73f0 0x7fe8b01f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.927+0000 7fe8b6dd7700 1 --2- 172.21.15.189:0/3096394131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8a0005210 0x7fe8b01ee500 unknown :-1 s=CLOSED pgs=2733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.927+0000 7fe8b6dd7700 1 --2- 172.21.15.189:0/3096394131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8b01f2040 0x7fe8b00c6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:36.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.927+0000 7fe8b6dd7700 1 -- 172.21.15.189:0/3096394131 >> 172.21.15.189:0/3096394131 conn(0x7fe8b00b2db0 msgr2=0x7fe8b00b66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:36.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.927+0000 7fe8b6dd7700 1 -- 172.21.15.189:0/3096394131 shutdown_connections 2024-04-06T13:48:36.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:36.927+0000 7fe8b6dd7700 1 -- 172.21.15.189:0/3096394131 wait complete. 2024-04-06T13:48:36.940 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2024-04-06T13:48:37.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.065+0000 7f20d1e9a700 1 Processor -- start 2024-04-06T13:48:37.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.065+0000 7f20d1e9a700 1 -- start start 2024-04-06T13:48:37.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.066+0000 7f20d1e9a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f20cc15a550 0x7f20cc0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:37.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.066+0000 7f20d1e9a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20cc0cf900 0x7f20cc0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:37.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.066+0000 7f20d1e9a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f20cc0b4ba0 0x7f20cc06eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:37.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.066+0000 7f20d1e9a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20cc0cdbe0 con 0x7f20cc15a550 2024-04-06T13:48:37.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.066+0000 7f20d1e9a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20cc0b5010 con 0x7f20cc0b4ba0 2024-04-06T13:48:37.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.066+0000 7f20d1e9a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc15a940 con 0x7f20cc0cf900 2024-04-06T13:48:37.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.066+0000 7f20cb7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f20cc15a550 0x7f20cc0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:37.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.066+0000 7f20cb7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f20cc15a550 0x7f20cc0cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49482/0 (socket says 172.21.15.189:49482) 2024-04-06T13:48:37.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.066+0000 7f20cb7fe700 1 -- 172.21.15.189:0/2301430720 learned_addr learned my addr 172.21.15.189:0/2301430720 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:37.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.066+0000 7f20caffd700 1 --2- 172.21.15.189:0/2301430720 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20cc0cf900 0x7f20cc0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:37.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.066+0000 7f20cbfff700 1 --2- 172.21.15.189:0/2301430720 >> v2:172.21.15.192:3300/0 conn(0x7f20cc0b4ba0 0x7f20cc06eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:37.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.066+0000 7f20cb7fe700 1 -- 172.21.15.189:0/2301430720 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20cc0cf900 msgr2=0x7f20cc0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:37.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.066+0000 7f20cb7fe700 1 --2- 172.21.15.189:0/2301430720 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20cc0cf900 0x7f20cc0b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:37.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.066+0000 7f20cb7fe700 1 -- 172.21.15.189:0/2301430720 >> v2:172.21.15.192:3300/0 conn(0x7f20cc0b4ba0 msgr2=0x7f20cc06eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:37.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.066+0000 7f20cb7fe700 1 --2- 172.21.15.189:0/2301430720 >> v2:172.21.15.192:3300/0 conn(0x7f20cc0b4ba0 0x7f20cc06eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:37.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.066+0000 7f20cb7fe700 1 -- 172.21.15.189:0/2301430720 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc15ac00 con 0x7f20cc15a550 2024-04-06T13:48:37.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.067+0000 7f20cb7fe700 1 --2- 172.21.15.189:0/2301430720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20cc15a550 0x7f20cc0cf3c0 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f20cc0d1110 tx=0x7f20c0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d59b9818ddd22dd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:37.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.067+0000 7f20ca7fc700 1 -- 172.21.15.189:0/2301430720 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c0011830 con 0x7f20cc15a550 2024-04-06T13:48:37.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.067+0000 7f20ca7fc700 1 -- 172.21.15.189:0/2301430720 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c0011990 con 0x7f20cc15a550 2024-04-06T13:48:37.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.067+0000 7f20ca7fc700 1 -- 172.21.15.189:0/2301430720 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c0011d20 con 0x7f20cc15a550 2024-04-06T13:48:37.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.067+0000 7f20d1e9a700 1 -- 172.21.15.189:0/2301430720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20cc15a550 msgr2=0x7f20cc0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:37.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.067+0000 7f20d1e9a700 1 --2- 172.21.15.189:0/2301430720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20cc15a550 0x7f20cc0cf3c0 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f20cc0d1110 tx=0x7f20c0009bd0 comp rx=0 tx=0).stop 2024-04-06T13:48:37.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.067+0000 7f20d1e9a700 1 -- 172.21.15.189:0/2301430720 shutdown_connections 2024-04-06T13:48:37.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.067+0000 7f20d1e9a700 1 --2- 172.21.15.189:0/2301430720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20cc15a550 0x7f20cc0cf3c0 unknown :-1 s=CLOSED pgs=2734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:37.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.067+0000 7f20d1e9a700 1 --2- 172.21.15.189:0/2301430720 >> v2:172.21.15.192:3300/0 conn(0x7f20cc0b4ba0 0x7f20cc06eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:37.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.067+0000 7f20d1e9a700 1 --2- 172.21.15.189:0/2301430720 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20cc0cf900 0x7f20cc0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:37.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.067+0000 7f20d1e9a700 1 -- 172.21.15.189:0/2301430720 >> 172.21.15.189:0/2301430720 conn(0x7f20cc0d00b0 msgr2=0x7f20cc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:37.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.067+0000 7f20d1e9a700 1 -- 172.21.15.189:0/2301430720 shutdown_connections 2024-04-06T13:48:37.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.067+0000 7f20d1e9a700 1 -- 172.21.15.189:0/2301430720 wait complete. 2024-04-06T13:48:37.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.069+0000 7f20d1e9a700 1 Processor -- start 2024-04-06T13:48:37.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.069+0000 7f20d1e9a700 1 -- start start 2024-04-06T13:48:37.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.070+0000 7f20d1e9a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20cc0b4ba0 0x7f20cc145ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:37.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.070+0000 7f20d1e9a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20cc0cf900 0x7f20cc0b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:37.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.070+0000 7f20d1e9a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20cc141450 0x7f20cc0b11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:37.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.070+0000 7f20d1e9a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc151840 con 0x7f20cc0cf900 2024-04-06T13:48:37.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.070+0000 7f20d1e9a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc148830 con 0x7f20cc0b4ba0 2024-04-06T13:48:37.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.070+0000 7f20d1e9a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc145220 con 0x7f20cc141450 2024-04-06T13:48:37.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.070+0000 7f20caffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20cc0cf900 0x7f20cc0b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:37.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.070+0000 7f20caffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20cc0cf900 0x7f20cc0b0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49490/0 (socket says 172.21.15.189:49490) 2024-04-06T13:48:37.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.070+0000 7f20caffd700 1 -- 172.21.15.189:0/2971019482 learned_addr learned my addr 172.21.15.189:0/2971019482 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:37.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.070+0000 7f20cb7fe700 1 --2- 172.21.15.189:0/2971019482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20cc0b4ba0 0x7f20cc145ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:37.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.070+0000 7f20cbfff700 1 --2- 172.21.15.189:0/2971019482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20cc141450 0x7f20cc0b11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:37.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.070+0000 7f20caffd700 1 -- 172.21.15.189:0/2971019482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20cc141450 msgr2=0x7f20cc0b11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:37.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.070+0000 7f20caffd700 1 --2- 172.21.15.189:0/2971019482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20cc141450 0x7f20cc0b11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:37.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.070+0000 7f20caffd700 1 -- 172.21.15.189:0/2971019482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20cc0b4ba0 msgr2=0x7f20cc145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:37.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.070+0000 7f20caffd700 1 --2- 172.21.15.189:0/2971019482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20cc0b4ba0 0x7f20cc145ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:37.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.070+0000 7f20caffd700 1 -- 172.21.15.189:0/2971019482 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c0009550 con 0x7f20cc0cf900 2024-04-06T13:48:37.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.070+0000 7f20caffd700 1 --2- 172.21.15.189:0/2971019482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20cc0cf900 0x7f20cc0b0c80 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7f20bc00e9e0 tx=0x7f20bc00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:37.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.071+0000 7f20b3fff700 1 -- 172.21.15.189:0/2971019482 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc009710 con 0x7f20cc0cf900 2024-04-06T13:48:37.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.071+0000 7f20b3fff700 1 -- 172.21.15.189:0/2971019482 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20bc00ee80 con 0x7f20cc0cf900 2024-04-06T13:48:37.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.071+0000 7f20b3fff700 1 -- 172.21.15.189:0/2971019482 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc00c9f0 con 0x7f20cc0cf900 2024-04-06T13:48:37.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.071+0000 7f20d1e9a700 1 -- 172.21.15.189:0/2971019482 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20cc151560 con 0x7f20cc0cf900 2024-04-06T13:48:37.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.071+0000 7f20d1e9a700 1 -- 172.21.15.189:0/2971019482 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20cc1484c0 con 0x7f20cc0cf900 2024-04-06T13:48:37.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.072+0000 7f20d1e9a700 1 -- 172.21.15.189:0/2971019482 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20cc148680 con 0x7f20cc0cf900 2024-04-06T13:48:37.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.072+0000 7f20b3fff700 1 -- 172.21.15.189:0/2971019482 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f20bc00cb50 con 0x7f20cc0cf900 2024-04-06T13:48:37.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.073+0000 7f20b3fff700 1 --2- 172.21.15.189:0/2971019482 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f20b806aed0 0x7f20b806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:37.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.073+0000 7f20b3fff700 1 -- 172.21.15.189:0/2971019482 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f20bc089b20 con 0x7f20cc0cf900 2024-04-06T13:48:37.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.073+0000 7f20cb7fe700 1 --2- 172.21.15.189:0/2971019482 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f20b806aed0 0x7f20b806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:37.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.073+0000 7f20cb7fe700 1 --2- 172.21.15.189:0/2971019482 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f20b806aed0 0x7f20b806d390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f20c0009c10 tx=0x7f20c0008030 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:37.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.077+0000 7f20b3fff700 1 -- 172.21.15.189:0/2971019482 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f20bc054540 con 0x7f20cc0cf900 2024-04-06T13:48:37.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:37.319+0000 7f20d1e9a700 1 -- 172.21.15.189:0/2971019482 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f20cc0cd870 con 0x7f20cc0cf900 2024-04-06T13:48:38.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.208+0000 7f20b3fff700 1 -- 172.21.15.189:0/2971019482 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v121) v1 ==== 121+0+0 (secure 0 0 0) 0x7f20bc057b60 con 0x7f20cc0cf900 2024-04-06T13:48:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.211+0000 7f20d1e9a700 1 -- 172.21.15.189:0/2971019482 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f20b806aed0 msgr2=0x7f20b806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.211+0000 7f20d1e9a700 1 --2- 172.21.15.189:0/2971019482 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f20b806aed0 0x7f20b806d390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f20c0009c10 tx=0x7f20c0008030 comp rx=0 tx=0).stop 2024-04-06T13:48:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.211+0000 7f20d1e9a700 1 -- 172.21.15.189:0/2971019482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20cc0cf900 msgr2=0x7f20cc0b0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.211+0000 7f20d1e9a700 1 --2- 172.21.15.189:0/2971019482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20cc0cf900 0x7f20cc0b0c80 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7f20bc00e9e0 tx=0x7f20bc00ecf0 comp rx=0 tx=0).stop 2024-04-06T13:48:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.211+0000 7f20d1e9a700 1 -- 172.21.15.189:0/2971019482 shutdown_connections 2024-04-06T13:48:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.211+0000 7f20d1e9a700 1 --2- 172.21.15.189:0/2971019482 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f20b806aed0 0x7f20b806d390 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.211+0000 7f20d1e9a700 1 --2- 172.21.15.189:0/2971019482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20cc141450 0x7f20cc0b11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.211+0000 7f20d1e9a700 1 --2- 172.21.15.189:0/2971019482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20cc0b4ba0 0x7f20cc145ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.211+0000 7f20d1e9a700 1 --2- 172.21.15.189:0/2971019482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20cc0cf900 0x7f20cc0b0c80 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.211+0000 7f20d1e9a700 1 -- 172.21.15.189:0/2971019482 >> 172.21.15.189:0/2971019482 conn(0x7f20cc0d00b0 msgr2=0x7f20cc14f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.212+0000 7f20d1e9a700 1 -- 172.21.15.189:0/2971019482 shutdown_connections 2024-04-06T13:48:38.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.212+0000 7f20d1e9a700 1 -- 172.21.15.189:0/2971019482 wait complete. 2024-04-06T13:48:38.226 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-06T13:48:38.227 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2024-04-06T13:48:38.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.357+0000 7fd49ff87700 1 Processor -- start 2024-04-06T13:48:38.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.358+0000 7fd49ff87700 1 -- start start 2024-04-06T13:48:38.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.358+0000 7fd49ff87700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd4980cbc20 0x7fd49815f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:38.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.358+0000 7fd49ff87700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd49806aa10 0x7fd49806ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:38.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.358+0000 7fd49ff87700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4980669d0 con 0x7fd49815e8a0 2024-04-06T13:48:38.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.358+0000 7fd49ff87700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd49806ede0 con 0x7fd49806aa10 2024-04-06T13:48:38.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.358+0000 7fd49ff87700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd49815ec80 con 0x7fd4980cbc20 2024-04-06T13:48:38.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.359+0000 7fd49e524700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd49806aa10 0x7fd49806ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:38.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.359+0000 7fd49e524700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd49806aa10 0x7fd49806ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49498/0 (socket says 172.21.15.189:49498) 2024-04-06T13:48:38.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.359+0000 7fd49dd23700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd49815e8a0 0x7fd49815eee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55868/0 (socket says 172.21.15.189:55868) 2024-04-06T13:48:38.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.359+0000 7fd49e524700 1 -- 172.21.15.189:0/375822421 learned_addr learned my addr 172.21.15.189:0/375822421 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:38.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.359+0000 7fd49d522700 1 --2- 172.21.15.189:0/375822421 >> v2:172.21.15.192:3300/0 conn(0x7fd4980cbc20 0x7fd49815f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:38.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.359+0000 7fd49e524700 1 -- 172.21.15.189:0/375822421 >> v2:172.21.15.192:3300/0 conn(0x7fd4980cbc20 msgr2=0x7fd49815f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:38.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.359+0000 7fd49e524700 1 --2- 172.21.15.189:0/375822421 >> v2:172.21.15.192:3300/0 conn(0x7fd4980cbc20 0x7fd49815f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:38.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.359+0000 7fd49e524700 1 -- 172.21.15.189:0/375822421 >> v1:172.21.15.189:6789/0 conn(0x7fd49815e8a0 legacy=0x7fd49815eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:38.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.359+0000 7fd49e524700 1 -- 172.21.15.189:0/375822421 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4980666f0 con 0x7fd49806aa10 2024-04-06T13:48:38.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.359+0000 7fd49e524700 1 --2- 172.21.15.189:0/375822421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49806aa10 0x7fd49806ae40 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7fd4980723a0 tx=0x7fd48800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef8d5f3696d3a26 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:38.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.359+0000 7fd49cd21700 1 -- 172.21.15.189:0/375822421 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48800e070 con 0x7fd49806aa10 2024-04-06T13:48:38.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.359+0000 7fd49cd21700 1 -- 172.21.15.189:0/375822421 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48800b9c0 con 0x7fd49806aa10 2024-04-06T13:48:38.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.359+0000 7fd49cd21700 1 -- 172.21.15.189:0/375822421 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48800bd70 con 0x7fd49806aa10 2024-04-06T13:48:38.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.360+0000 7fd49ff87700 1 -- 172.21.15.189:0/375822421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49806aa10 msgr2=0x7fd49806ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:38.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.360+0000 7fd49ff87700 1 --2- 172.21.15.189:0/375822421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49806aa10 0x7fd49806ae40 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7fd4980723a0 tx=0x7fd48800b330 comp rx=0 tx=0).stop 2024-04-06T13:48:38.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.360+0000 7fd49ff87700 1 -- 172.21.15.189:0/375822421 shutdown_connections 2024-04-06T13:48:38.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.360+0000 7fd49ff87700 1 --2- 172.21.15.189:0/375822421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49806aa10 0x7fd49806ae40 unknown :-1 s=CLOSED pgs=2737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:38.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.360+0000 7fd49ff87700 1 --2- 172.21.15.189:0/375822421 >> v2:172.21.15.192:3300/0 conn(0x7fd4980cbc20 0x7fd49815f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:38.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.360+0000 7fd49ff87700 1 -- 172.21.15.189:0/375822421 >> 172.21.15.189:0/375822421 conn(0x7fd4980cc290 msgr2=0x7fd4980cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:38.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.360+0000 7fd49ff87700 1 -- 172.21.15.189:0/375822421 shutdown_connections 2024-04-06T13:48:38.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.360+0000 7fd49ff87700 1 -- 172.21.15.189:0/375822421 wait complete. 2024-04-06T13:48:38.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.361+0000 7fd49ff87700 1 Processor -- start 2024-04-06T13:48:38.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.361+0000 7fd49ff87700 1 -- start start 2024-04-06T13:48:38.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd49ff87700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49806aa10 0x7fd4980c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:38.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd49ff87700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4980cbc20 0x7fd4980ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:38.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd49ff87700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd49815e8a0 0x7fd4980b0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:38.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd49ff87700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4980c8c20 con 0x7fd49806aa10 2024-04-06T13:48:38.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd49ff87700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4980c8010 con 0x7fd49815e8a0 2024-04-06T13:48:38.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd49ff87700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4980c82f0 con 0x7fd4980cbc20 2024-04-06T13:48:38.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd49dd23700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49806aa10 0x7fd4980c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:38.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd49dd23700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49806aa10 0x7fd4980c90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49512/0 (socket says 172.21.15.189:49512) 2024-04-06T13:48:38.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd49dd23700 1 -- 172.21.15.189:0/4191392454 learned_addr learned my addr 172.21.15.189:0/4191392454 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:38.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd49e524700 1 --2- 172.21.15.189:0/4191392454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd49815e8a0 0x7fd4980b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:38.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd49d522700 1 --2- 172.21.15.189:0/4191392454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4980cbc20 0x7fd4980ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:38.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd49dd23700 1 -- 172.21.15.189:0/4191392454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4980cbc20 msgr2=0x7fd4980ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:38.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd49dd23700 1 --2- 172.21.15.189:0/4191392454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4980cbc20 0x7fd4980ce2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:38.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd49dd23700 1 -- 172.21.15.189:0/4191392454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd49815e8a0 msgr2=0x7fd4980b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:38.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd49dd23700 1 --2- 172.21.15.189:0/4191392454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd49815e8a0 0x7fd4980b0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:38.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd49dd23700 1 -- 172.21.15.189:0/4191392454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd488009b30 con 0x7fd49806aa10 2024-04-06T13:48:38.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd49d522700 1 --2- 172.21.15.189:0/4191392454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4980cbc20 0x7fd4980ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:38.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd49dd23700 1 --2- 172.21.15.189:0/4191392454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49806aa10 0x7fd4980c90b0 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7fd49000c500 tx=0x7fd49000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:38.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd4967fc700 1 -- 172.21.15.189:0/4191392454 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd490005cb0 con 0x7fd49806aa10 2024-04-06T13:48:38.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd4967fc700 1 -- 172.21.15.189:0/4191392454 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd490005e10 con 0x7fd49806aa10 2024-04-06T13:48:38.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd4967fc700 1 -- 172.21.15.189:0/4191392454 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4900072d0 con 0x7fd49806aa10 2024-04-06T13:48:38.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.362+0000 7fd49ff87700 1 -- 172.21.15.189:0/4191392454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4980c85d0 con 0x7fd49806aa10 2024-04-06T13:48:38.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.363+0000 7fd49ff87700 1 -- 172.21.15.189:0/4191392454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4980ce7e0 con 0x7fd49806aa10 2024-04-06T13:48:38.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.363+0000 7fd49ff87700 1 -- 172.21.15.189:0/4191392454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd498072140 con 0x7fd49806aa10 2024-04-06T13:48:38.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.364+0000 7fd4967fc700 1 -- 172.21.15.189:0/4191392454 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fd490007430 con 0x7fd49806aa10 2024-04-06T13:48:38.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.364+0000 7fd4967fc700 1 --2- 172.21.15.189:0/4191392454 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd48406aed0 0x7fd48406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:38.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.364+0000 7fd4967fc700 1 -- 172.21.15.189:0/4191392454 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6965+0+0 (secure 0 0 0) 0x7fd490088e20 con 0x7fd49806aa10 2024-04-06T13:48:38.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.364+0000 7fd49d522700 1 --2- 172.21.15.189:0/4191392454 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd48406aed0 0x7fd48406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:38.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.365+0000 7fd49d522700 1 --2- 172.21.15.189:0/4191392454 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd48406aed0 0x7fd48406d390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd4980c9880 tx=0x7fd48c008040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:38.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.369+0000 7fd4967fc700 1 -- 172.21.15.189:0/4191392454 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd4900538f0 con 0x7fd49806aa10 2024-04-06T13:48:38.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:38.605+0000 7fd49ff87700 1 -- 172.21.15.189:0/4191392454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd4980c8f00 con 0x7fd49806aa10 2024-04-06T13:48:39.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.210+0000 7fd4967fc700 1 -- 172.21.15.189:0/4191392454 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v122) v1 ==== 125+0+0 (secure 0 0 0) 0x7fd490017020 con 0x7fd49806aa10 2024-04-06T13:48:39.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.212+0000 7fd49ff87700 1 -- 172.21.15.189:0/4191392454 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd48406aed0 msgr2=0x7fd48406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:39.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.212+0000 7fd49ff87700 1 --2- 172.21.15.189:0/4191392454 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd48406aed0 0x7fd48406d390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd4980c9880 tx=0x7fd48c008040 comp rx=0 tx=0).stop 2024-04-06T13:48:39.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.212+0000 7fd49ff87700 1 -- 172.21.15.189:0/4191392454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49806aa10 msgr2=0x7fd4980c90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:39.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.212+0000 7fd49ff87700 1 --2- 172.21.15.189:0/4191392454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49806aa10 0x7fd4980c90b0 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7fd49000c500 tx=0x7fd49000c810 comp rx=0 tx=0).stop 2024-04-06T13:48:39.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.212+0000 7fd49ff87700 1 -- 172.21.15.189:0/4191392454 shutdown_connections 2024-04-06T13:48:39.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.212+0000 7fd49ff87700 1 --2- 172.21.15.189:0/4191392454 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd48406aed0 0x7fd48406d390 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.213+0000 7fd49ff87700 1 --2- 172.21.15.189:0/4191392454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd49815e8a0 0x7fd4980b0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.213+0000 7fd49ff87700 1 --2- 172.21.15.189:0/4191392454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49806aa10 0x7fd4980c90b0 unknown :-1 s=CLOSED pgs=2738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.213+0000 7fd49ff87700 1 --2- 172.21.15.189:0/4191392454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4980cbc20 0x7fd4980ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.213+0000 7fd49ff87700 1 -- 172.21.15.189:0/4191392454 >> 172.21.15.189:0/4191392454 conn(0x7fd4980cc290 msgr2=0x7fd49806cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:39.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.213+0000 7fd49ff87700 1 -- 172.21.15.189:0/4191392454 shutdown_connections 2024-04-06T13:48:39.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.213+0000 7fd49ff87700 1 -- 172.21.15.189:0/4191392454 wait complete. 2024-04-06T13:48:39.227 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2024-04-06T13:48:39.227 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:48:39.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.351+0000 7f6e8c535700 1 Processor -- start 2024-04-06T13:48:39.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.351+0000 7f6e8c535700 1 -- start start 2024-04-06T13:48:39.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.352+0000 7f6e8c535700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6e8415d7e0 0x7f6e840d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:39.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.352+0000 7f6e8c535700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e840d1cc0 0x7f6e840b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:39.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.352+0000 7f6e8c535700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e840cdf80 con 0x7f6e840d0890 2024-04-06T13:48:39.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.352+0000 7f6e8c535700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e840d3230 con 0x7f6e8415d7e0 2024-04-06T13:48:39.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.352+0000 7f6e8c535700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840d2da0 con 0x7f6e840d1cc0 2024-04-06T13:48:39.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.353+0000 7f6e8a2d1700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6e8415d7e0 0x7f6e840d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:39.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.353+0000 7f6e8aad2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e840d1cc0 0x7f6e840b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:39.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.353+0000 7f6e89ad0700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f6e840d0890 0x7f6e840d15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:41524/0 (socket says 172.21.15.189:41524) 2024-04-06T13:48:39.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.353+0000 7f6e89ad0700 1 -- 172.21.15.189:0/3628406353 learned_addr learned my addr 172.21.15.189:0/3628406353 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:39.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.353+0000 7f6e8a2d1700 1 -- 172.21.15.189:0/3628406353 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e840d1cc0 msgr2=0x7f6e840b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:39.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.353+0000 7f6e8a2d1700 1 --2- 172.21.15.189:0/3628406353 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e840d1cc0 0x7f6e840b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.353+0000 7f6e8a2d1700 1 -- 172.21.15.189:0/3628406353 >> v1:172.21.15.192:6789/0 conn(0x7f6e840d0890 legacy=0x7f6e840d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:39.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.353+0000 7f6e8a2d1700 1 -- 172.21.15.189:0/3628406353 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e840d0cf0 con 0x7f6e8415d7e0 2024-04-06T13:48:39.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.353+0000 7f6e8aad2700 1 --2- 172.21.15.189:0/3628406353 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e840d1cc0 0x7f6e840b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:39.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.353+0000 7f6e8a2d1700 1 --2- 172.21.15.189:0/3628406353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e8415d7e0 0x7f6e840d0350 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009840 tx=0x7f6e78009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84eb1151b5d209f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:39.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.353+0000 7f6e892cf700 1 -- 172.21.15.189:0/3628406353 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e78011830 con 0x7f6e8415d7e0 2024-04-06T13:48:39.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.353+0000 7f6e892cf700 1 -- 172.21.15.189:0/3628406353 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e78011990 con 0x7f6e8415d7e0 2024-04-06T13:48:39.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.353+0000 7f6e892cf700 1 -- 172.21.15.189:0/3628406353 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e78011d20 con 0x7f6e8415d7e0 2024-04-06T13:48:39.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.354+0000 7f6e8c535700 1 -- 172.21.15.189:0/3628406353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e8415d7e0 msgr2=0x7f6e840d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:39.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.354+0000 7f6e8c535700 1 --2- 172.21.15.189:0/3628406353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e8415d7e0 0x7f6e840d0350 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009840 tx=0x7f6e78009b50 comp rx=0 tx=0).stop 2024-04-06T13:48:39.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.354+0000 7f6e8c535700 1 -- 172.21.15.189:0/3628406353 shutdown_connections 2024-04-06T13:48:39.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.354+0000 7f6e8c535700 1 --2- 172.21.15.189:0/3628406353 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e840d1cc0 0x7f6e840b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.354+0000 7f6e8c535700 1 --2- 172.21.15.189:0/3628406353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e8415d7e0 0x7f6e840d0350 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.354+0000 7f6e8c535700 1 -- 172.21.15.189:0/3628406353 >> 172.21.15.189:0/3628406353 conn(0x7f6e84072250 msgr2=0x7f6e84072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.354+0000 7f6e8c535700 1 -- 172.21.15.189:0/3628406353 shutdown_connections 2024-04-06T13:48:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.354+0000 7f6e8c535700 1 -- 172.21.15.189:0/3628406353 wait complete. 2024-04-06T13:48:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.355+0000 7f6e8c535700 1 Processor -- start 2024-04-06T13:48:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.355+0000 7f6e8c535700 1 -- start start 2024-04-06T13:48:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.356+0000 7f6e8c535700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e840d0890 0x7f6e840c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.356+0000 7f6e8c535700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e840d1cc0 0x7f6e841600a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:39.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.356+0000 7f6e8c535700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e8415d7e0 0x7f6e841605e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.356+0000 7f6e8c535700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840cae70 con 0x7f6e840d0890 2024-04-06T13:48:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.356+0000 7f6e8c535700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840c90e0 con 0x7f6e840d1cc0 2024-04-06T13:48:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.356+0000 7f6e8c535700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840c93a0 con 0x7f6e8415d7e0 2024-04-06T13:48:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.356+0000 7f6e8a2d1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e840d0890 0x7f6e840c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.356+0000 7f6e8a2d1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e840d0890 0x7f6e840c8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34536/0 (socket says 172.21.15.189:34536) 2024-04-06T13:48:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.356+0000 7f6e8a2d1700 1 -- 172.21.15.189:0/2413413450 learned_addr learned my addr 172.21.15.189:0/2413413450 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.356+0000 7f6e89ad0700 1 --2- 172.21.15.189:0/2413413450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e840d1cc0 0x7f6e841600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.356+0000 7f6e8aad2700 1 --2- 172.21.15.189:0/2413413450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e8415d7e0 0x7f6e841605e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.356+0000 7f6e8a2d1700 1 -- 172.21.15.189:0/2413413450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e8415d7e0 msgr2=0x7f6e841605e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.356+0000 7f6e8a2d1700 1 --2- 172.21.15.189:0/2413413450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e8415d7e0 0x7f6e841605e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.356+0000 7f6e8a2d1700 1 -- 172.21.15.189:0/2413413450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e840d1cc0 msgr2=0x7f6e841600a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.356+0000 7f6e8a2d1700 1 --2- 172.21.15.189:0/2413413450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e840d1cc0 0x7f6e841600a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.356+0000 7f6e8a2d1700 1 -- 172.21.15.189:0/2413413450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7c009a70 con 0x7f6e840d0890 2024-04-06T13:48:39.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.357+0000 7f6e8a2d1700 1 --2- 172.21.15.189:0/2413413450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e840d0890 0x7f6e840c8ba0 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009fd0 tx=0x7f6e78004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:39.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.357+0000 7f6e76ffd700 1 -- 172.21.15.189:0/2413413450 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e780045d0 con 0x7f6e840d0890 2024-04-06T13:48:39.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.357+0000 7f6e76ffd700 1 -- 172.21.15.189:0/2413413450 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7801f070 con 0x7f6e840d0890 2024-04-06T13:48:39.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.357+0000 7f6e8c535700 1 -- 172.21.15.189:0/2413413450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e780094a0 con 0x7f6e840d0890 2024-04-06T13:48:39.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.357+0000 7f6e76ffd700 1 -- 172.21.15.189:0/2413413450 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e780038e0 con 0x7f6e840d0890 2024-04-06T13:48:39.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.357+0000 7f6e8c535700 1 -- 172.21.15.189:0/2413413450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e840c9960 con 0x7f6e840d0890 2024-04-06T13:48:39.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.358+0000 7f6e8c535700 1 -- 172.21.15.189:0/2413413450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e84074990 con 0x7f6e840d0890 2024-04-06T13:48:39.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.358+0000 7f6e76ffd700 1 -- 172.21.15.189:0/2413413450 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f6e78003a40 con 0x7f6e840d0890 2024-04-06T13:48:39.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.359+0000 7f6e76ffd700 1 --2- 172.21.15.189:0/2413413450 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6e6806aed0 0x7f6e6806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:39.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.359+0000 7f6e76ffd700 1 -- 172.21.15.189:0/2413413450 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f6e78021070 con 0x7f6e840d0890 2024-04-06T13:48:39.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.359+0000 7f6e89ad0700 1 --2- 172.21.15.189:0/2413413450 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6e6806aed0 0x7f6e6806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:39.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.359+0000 7f6e89ad0700 1 --2- 172.21.15.189:0/2413413450 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6e6806aed0 0x7f6e6806d390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6e840b8d20 tx=0x7f6e80009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:39.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.363+0000 7f6e76ffd700 1 -- 172.21.15.189:0/2413413450 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6e78055e30 con 0x7f6e840d0890 2024-04-06T13:48:39.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.601+0000 7f6e8c535700 1 -- 172.21.15.189:0/2413413450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e8415d000 con 0x7f6e840d0890 2024-04-06T13:48:39.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.602+0000 7f6e76ffd700 1 -- 172.21.15.189:0/2413413450 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5967 (secure 0 0 0) 0x7f6e78059450 con 0x7f6e840d0890 2024-04-06T13:48:39.603 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:39.603 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":122,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3323503154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3323503154},{"type":"v1","addr":"172.21.15.189:6836","nonce":3323503154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":9383,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3292345173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3292345173},{"type":"v1","addr":"172.21.15.189:6837","nonce":3292345173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:48:19.674205+0000","modified":"2024-04-06T13:48:22.682911+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12108},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12108":{"gid":12108,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6833/726816746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":726816746},{"type":"v1","addr":"172.21.15.192:6833","nonce":726816746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:48:36.197004+0000","modified":"2024-04-06T13:48:39.206755+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9395},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9395":{"gid":9395,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/455737532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":455737532},{"type":"v1","addr":"172.21.15.192:6835","nonce":455737532}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-04-06T13:48:39.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.604+0000 7f6e8c535700 1 -- 172.21.15.189:0/2413413450 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6e6806aed0 msgr2=0x7f6e6806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:39.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.604+0000 7f6e8c535700 1 --2- 172.21.15.189:0/2413413450 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6e6806aed0 0x7f6e6806d390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6e840b8d20 tx=0x7f6e80009040 comp rx=0 tx=0).stop 2024-04-06T13:48:39.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.604+0000 7f6e8c535700 1 -- 172.21.15.189:0/2413413450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e840d0890 msgr2=0x7f6e840c8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:39.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.604+0000 7f6e8c535700 1 --2- 172.21.15.189:0/2413413450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e840d0890 0x7f6e840c8ba0 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009fd0 tx=0x7f6e78004750 comp rx=0 tx=0).stop 2024-04-06T13:48:39.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.604+0000 7f6e8c535700 1 -- 172.21.15.189:0/2413413450 shutdown_connections 2024-04-06T13:48:39.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.604+0000 7f6e8c535700 1 --2- 172.21.15.189:0/2413413450 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6e6806aed0 0x7f6e6806d390 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.604+0000 7f6e8c535700 1 --2- 172.21.15.189:0/2413413450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e8415d7e0 0x7f6e841605e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.604+0000 7f6e8c535700 1 --2- 172.21.15.189:0/2413413450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e840d0890 0x7f6e840c8ba0 unknown :-1 s=CLOSED pgs=2739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.604+0000 7f6e8c535700 1 --2- 172.21.15.189:0/2413413450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e840d1cc0 0x7f6e841600a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.604+0000 7f6e8c535700 1 -- 172.21.15.189:0/2413413450 >> 172.21.15.189:0/2413413450 conn(0x7f6e84072250 msgr2=0x7f6e840b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:39.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.604+0000 7f6e8c535700 1 -- 172.21.15.189:0/2413413450 shutdown_connections 2024-04-06T13:48:39.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.604+0000 7f6e8c535700 1 -- 172.21.15.189:0/2413413450 wait complete. 2024-04-06T13:48:39.606 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 122 2024-04-06T13:48:39.617 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:48:39.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.739+0000 7f406604e700 1 Processor -- start 2024-04-06T13:48:39.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.740+0000 7f406604e700 1 -- start start 2024-04-06T13:48:39.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.740+0000 7f406604e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4060144f40 0x7f406006a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:39.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.740+0000 7f406604e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f406015fcd0 0x7f406006ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:39.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.740+0000 7f406604e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406015c740 con 0x7f406015ba40 2024-04-06T13:48:39.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.740+0000 7f406604e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f406015c560 con 0x7f406015fcd0 2024-04-06T13:48:39.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.740+0000 7f406604e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40600cfcb0 con 0x7f4060144f40 2024-04-06T13:48:39.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.741+0000 7f405f7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4060144f40 0x7f406006a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.741+0000 7f405effd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f406015fcd0 0x7f406006ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.741+0000 7f405ffff700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f406015ba40 0x7f406015be50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:41530/0 (socket says 172.21.15.189:41530) 2024-04-06T13:48:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.741+0000 7f405effd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f406015fcd0 0x7f406006ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:40322/0 (socket says 172.21.15.189:40322) 2024-04-06T13:48:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.741+0000 7f405ffff700 1 -- 172.21.15.189:0/3469606986 learned_addr learned my addr 172.21.15.189:0/3469606986 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.741+0000 7f405f7fe700 1 --2- 172.21.15.189:0/3469606986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4060144f40 0x7f406006a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:60968/0 (socket says 172.21.15.189:60968) 2024-04-06T13:48:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.741+0000 7f405effd700 1 -- 172.21.15.189:0/3469606986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4060144f40 msgr2=0x7f406006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.741+0000 7f405effd700 1 --2- 172.21.15.189:0/3469606986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4060144f40 0x7f406006a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.741+0000 7f405effd700 1 -- 172.21.15.189:0/3469606986 >> v1:172.21.15.192:6789/0 conn(0x7f406015ba40 legacy=0x7f406015be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.741+0000 7f405effd700 1 -- 172.21.15.189:0/3469606986 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406015f1f0 con 0x7f406015fcd0 2024-04-06T13:48:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.741+0000 7f405effd700 1 --2- 172.21.15.189:0/3469606986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f406015fcd0 0x7f406006ac70 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f4048009840 tx=0x7f4048009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25d2067768056d7d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.741+0000 7f405e7fc700 1 -- 172.21.15.189:0/3469606986 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4048011830 con 0x7f406015fcd0 2024-04-06T13:48:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.741+0000 7f405e7fc700 1 -- 172.21.15.189:0/3469606986 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4048011990 con 0x7f406015fcd0 2024-04-06T13:48:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.741+0000 7f405e7fc700 1 -- 172.21.15.189:0/3469606986 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4048011d20 con 0x7f406015fcd0 2024-04-06T13:48:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.742+0000 7f406604e700 1 -- 172.21.15.189:0/3469606986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f406015fcd0 msgr2=0x7f406006ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:39.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.742+0000 7f406604e700 1 --2- 172.21.15.189:0/3469606986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f406015fcd0 0x7f406006ac70 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f4048009840 tx=0x7f4048009b50 comp rx=0 tx=0).stop 2024-04-06T13:48:39.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.742+0000 7f406604e700 1 -- 172.21.15.189:0/3469606986 shutdown_connections 2024-04-06T13:48:39.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.742+0000 7f406604e700 1 --2- 172.21.15.189:0/3469606986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4060144f40 0x7f406006a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.742+0000 7f406604e700 1 --2- 172.21.15.189:0/3469606986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f406015fcd0 0x7f406006ac70 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.742+0000 7f406604e700 1 -- 172.21.15.189:0/3469606986 >> 172.21.15.189:0/3469606986 conn(0x7f40600d00b0 msgr2=0x7f40600d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:39.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.742+0000 7f406604e700 1 -- 172.21.15.189:0/3469606986 shutdown_connections 2024-04-06T13:48:39.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.742+0000 7f406604e700 1 -- 172.21.15.189:0/3469606986 wait complete. 2024-04-06T13:48:39.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.743+0000 7f406604e700 1 Processor -- start 2024-04-06T13:48:39.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.743+0000 7f406604e700 1 -- start start 2024-04-06T13:48:39.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.744+0000 7f406604e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4060144f40 0x7f40601f05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:39.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.744+0000 7f406604e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f406015ba40 0x7f40600cafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:39.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.744+0000 7f406604e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f406015fcd0 0x7f40600cb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:39.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.744+0000 7f406604e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40601f0b20 con 0x7f4060144f40 2024-04-06T13:48:39.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.744+0000 7f406604e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40600c6b80 con 0x7f406015fcd0 2024-04-06T13:48:39.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.744+0000 7f406604e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40600c7a10 con 0x7f406015ba40 2024-04-06T13:48:39.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.744+0000 7f405f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4060144f40 0x7f40601f05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:39.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.744+0000 7f405f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4060144f40 0x7f40601f05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34546/0 (socket says 172.21.15.189:34546) 2024-04-06T13:48:39.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.744+0000 7f405f7fe700 1 -- 172.21.15.189:0/600977975 learned_addr learned my addr 172.21.15.189:0/600977975 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:39.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.744+0000 7f405effd700 1 --2- 172.21.15.189:0/600977975 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f406015ba40 0x7f40600cafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:39.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.744+0000 7f405ffff700 1 --2- 172.21.15.189:0/600977975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f406015fcd0 0x7f40600cb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:39.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.745+0000 7f405f7fe700 1 -- 172.21.15.189:0/600977975 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f406015ba40 msgr2=0x7f40600cafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:39.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.745+0000 7f405f7fe700 1 --2- 172.21.15.189:0/600977975 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f406015ba40 0x7f40600cafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.745+0000 7f405f7fe700 1 -- 172.21.15.189:0/600977975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f406015fcd0 msgr2=0x7f40600cb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.745+0000 7f405f7fe700 1 --2- 172.21.15.189:0/600977975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f406015fcd0 0x7f40600cb520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.745+0000 7f405f7fe700 1 -- 172.21.15.189:0/600977975 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40480094a0 con 0x7f4060144f40 2024-04-06T13:48:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.745+0000 7f405f7fe700 1 --2- 172.21.15.189:0/600977975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4060144f40 0x7f40601f05e0 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7f405000e9e0 tx=0x7f405000eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.745+0000 7f4057fff700 1 -- 172.21.15.189:0/600977975 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4050009710 con 0x7f4060144f40 2024-04-06T13:48:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.745+0000 7f4057fff700 1 -- 172.21.15.189:0/600977975 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405000c7f0 con 0x7f4060144f40 2024-04-06T13:48:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.745+0000 7f406604e700 1 -- 172.21.15.189:0/600977975 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40600cba60 con 0x7f4060144f40 2024-04-06T13:48:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.745+0000 7f4057fff700 1 -- 172.21.15.189:0/600977975 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405000cb60 con 0x7f4060144f40 2024-04-06T13:48:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.745+0000 7f406604e700 1 -- 172.21.15.189:0/600977975 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40600c73c0 con 0x7f4060144f40 2024-04-06T13:48:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.746+0000 7f406604e700 1 -- 172.21.15.189:0/600977975 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4060067bb0 con 0x7f4060144f40 2024-04-06T13:48:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.746+0000 7f4057fff700 1 -- 172.21.15.189:0/600977975 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f405000ccc0 con 0x7f4060144f40 2024-04-06T13:48:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.747+0000 7f4057fff700 1 --2- 172.21.15.189:0/600977975 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f404006aed0 0x7f404006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.747+0000 7f405effd700 1 --2- 172.21.15.189:0/600977975 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f404006aed0 0x7f404006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.747+0000 7f4057fff700 1 -- 172.21.15.189:0/600977975 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f4050089ce0 con 0x7f4060144f40 2024-04-06T13:48:39.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.747+0000 7f405effd700 1 --2- 172.21.15.189:0/600977975 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f404006aed0 0x7f404006d390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4048000c00 tx=0x7f404801c040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:39.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.751+0000 7f4057fff700 1 -- 172.21.15.189:0/600977975 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4050054670 con 0x7f4060144f40 2024-04-06T13:48:39.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.948+0000 7f406604e700 1 -- 172.21.15.189:0/600977975 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4060074990 con 0x7f4060144f40 2024-04-06T13:48:39.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.949+0000 7f4057fff700 1 -- 172.21.15.189:0/600977975 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) v1 ==== 74+0+20140 (secure 0 0 0) 0x7f4050017090 con 0x7f4060144f40 2024-04-06T13:48:39.951 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:39.951 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":252,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:48:37.198984+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:48:17.447247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-04-06T13:48:18.646229+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:48:33.912100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:48:35.170642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:48:39.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.951+0000 7f406604e700 1 -- 172.21.15.189:0/600977975 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f404006aed0 msgr2=0x7f404006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:39.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.951+0000 7f406604e700 1 --2- 172.21.15.189:0/600977975 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f404006aed0 0x7f404006d390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4048000c00 tx=0x7f404801c040 comp rx=0 tx=0).stop 2024-04-06T13:48:39.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.951+0000 7f406604e700 1 -- 172.21.15.189:0/600977975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4060144f40 msgr2=0x7f40601f05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:39.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.951+0000 7f406604e700 1 --2- 172.21.15.189:0/600977975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4060144f40 0x7f40601f05e0 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7f405000e9e0 tx=0x7f405000eda0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.952+0000 7f406604e700 1 -- 172.21.15.189:0/600977975 shutdown_connections 2024-04-06T13:48:39.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.952+0000 7f406604e700 1 --2- 172.21.15.189:0/600977975 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f404006aed0 0x7f404006d390 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.952+0000 7f406604e700 1 --2- 172.21.15.189:0/600977975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f406015fcd0 0x7f40600cb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.952+0000 7f406604e700 1 --2- 172.21.15.189:0/600977975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4060144f40 0x7f40601f05e0 unknown :-1 s=CLOSED pgs=2740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.952+0000 7f406604e700 1 --2- 172.21.15.189:0/600977975 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f406015ba40 0x7f40600cafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:39.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.952+0000 7f406604e700 1 -- 172.21.15.189:0/600977975 >> 172.21.15.189:0/600977975 conn(0x7f40600d00b0 msgr2=0x7f40600ccc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:39.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.952+0000 7f406604e700 1 -- 172.21.15.189:0/600977975 shutdown_connections 2024-04-06T13:48:39.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:39.952+0000 7f406604e700 1 -- 172.21.15.189:0/600977975 wait complete. 2024-04-06T13:48:39.965 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-06T13:48:39.966 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:48:40.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.086+0000 7f2ad533a700 1 Processor -- start 2024-04-06T13:48:40.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.087+0000 7f2ad533a700 1 -- start start 2024-04-06T13:48:40.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.087+0000 7f2ad533a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2ad00cd3e0 0x7f2ad00cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:40.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.087+0000 7f2ad533a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ad006b8c0 0x7f2ad006dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:40.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.087+0000 7f2ad533a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad006ceb0 con 0x7f2ad015d7c0 2024-04-06T13:48:40.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.087+0000 7f2ad533a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ad00cdfa0 con 0x7f2ad00cd3e0 2024-04-06T13:48:40.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.087+0000 7f2ad533a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad006bcf0 con 0x7f2ad006b8c0 2024-04-06T13:48:40.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.087+0000 7f2ace7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2ad015d7c0 0x7f2ad006b1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56514/0 (socket says 172.21.15.189:56514) 2024-04-06T13:48:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.088+0000 7f2ace7fc700 1 -- 172.21.15.189:0/290057152 learned_addr learned my addr 172.21.15.189:0/290057152 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.088+0000 7f2aceffd700 1 --2- 172.21.15.189:0/290057152 >> v2:172.21.15.192:3300/0 conn(0x7f2ad00cd3e0 0x7f2ad00cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.088+0000 7f2acf7fe700 1 --2- 172.21.15.189:0/290057152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ad006b8c0 0x7f2ad006dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.088+0000 7f2acdffb700 1 -- 172.21.15.189:0/290057152 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 505342774 0 0) 0x7f2ad006ceb0 con 0x7f2ad015d7c0 2024-04-06T13:48:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.088+0000 7f2acdffb700 1 -- 172.21.15.189:0/290057152 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab80035b0 con 0x7f2ad015d7c0 2024-04-06T13:48:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.088+0000 7f2aceffd700 1 -- 172.21.15.189:0/290057152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ad006b8c0 msgr2=0x7f2ad006dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.088+0000 7f2aceffd700 1 --2- 172.21.15.189:0/290057152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ad006b8c0 0x7f2ad006dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.088+0000 7f2aceffd700 1 -- 172.21.15.189:0/290057152 >> v1:172.21.15.189:6789/0 conn(0x7f2ad015d7c0 legacy=0x7f2ad006b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.088+0000 7f2aceffd700 1 -- 172.21.15.189:0/290057152 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad006d750 con 0x7f2ad00cd3e0 2024-04-06T13:48:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.088+0000 7f2acf7fe700 1 --2- 172.21.15.189:0/290057152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ad006b8c0 0x7f2ad006dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.088+0000 7f2aceffd700 1 --2- 172.21.15.189:0/290057152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ad00cd3e0 0x7f2ad00cd7c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009840 tx=0x7f2ac4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72c32e658d2b17cd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:40.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.088+0000 7f2acdffb700 1 -- 172.21.15.189:0/290057152 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4011830 con 0x7f2ad00cd3e0 2024-04-06T13:48:40.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.088+0000 7f2acdffb700 1 -- 172.21.15.189:0/290057152 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac4011990 con 0x7f2ad00cd3e0 2024-04-06T13:48:40.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.088+0000 7f2acdffb700 1 -- 172.21.15.189:0/290057152 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4011d20 con 0x7f2ad00cd3e0 2024-04-06T13:48:40.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.089+0000 7f2ad533a700 1 -- 172.21.15.189:0/290057152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ad00cd3e0 msgr2=0x7f2ad00cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:40.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.089+0000 7f2ad533a700 1 --2- 172.21.15.189:0/290057152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ad00cd3e0 0x7f2ad00cd7c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009840 tx=0x7f2ac4009b50 comp rx=0 tx=0).stop 2024-04-06T13:48:40.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.089+0000 7f2ad533a700 1 -- 172.21.15.189:0/290057152 shutdown_connections 2024-04-06T13:48:40.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.089+0000 7f2ad533a700 1 --2- 172.21.15.189:0/290057152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ad006b8c0 0x7f2ad006dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.089+0000 7f2ad533a700 1 --2- 172.21.15.189:0/290057152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ad00cd3e0 0x7f2ad00cd7c0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.089+0000 7f2ad533a700 1 -- 172.21.15.189:0/290057152 >> 172.21.15.189:0/290057152 conn(0x7f2ad00cc290 msgr2=0x7f2ad00cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:40.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.089+0000 7f2ad533a700 1 -- 172.21.15.189:0/290057152 shutdown_connections 2024-04-06T13:48:40.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.089+0000 7f2ad533a700 1 -- 172.21.15.189:0/290057152 wait complete. 2024-04-06T13:48:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.090+0000 7f2ad533a700 1 Processor -- start 2024-04-06T13:48:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.090+0000 7f2ad533a700 1 -- start start 2024-04-06T13:48:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.091+0000 7f2ad533a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ad006b8c0 0x7f2ad01f3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.091+0000 7f2ad533a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ad00cd3e0 0x7f2ad01f6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.091+0000 7f2ad533a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ad015d7c0 0x7f2ad015f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.091+0000 7f2ad533a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad00c4370 con 0x7f2ad015d7c0 2024-04-06T13:48:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.091+0000 7f2ad533a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad01f3aa0 con 0x7f2ad006b8c0 2024-04-06T13:48:40.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.091+0000 7f2ad533a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad015fed0 con 0x7f2ad00cd3e0 2024-04-06T13:48:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.091+0000 7f2acf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ad015d7c0 0x7f2ad015f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.091+0000 7f2acf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ad015d7c0 0x7f2ad015f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34558/0 (socket says 172.21.15.189:34558) 2024-04-06T13:48:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.091+0000 7f2acf7fe700 1 -- 172.21.15.189:0/2177348965 learned_addr learned my addr 172.21.15.189:0/2177348965 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.091+0000 7f2ace7fc700 1 --2- 172.21.15.189:0/2177348965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ad00cd3e0 0x7f2ad01f6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.091+0000 7f2aceffd700 1 --2- 172.21.15.189:0/2177348965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ad006b8c0 0x7f2ad01f3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.091+0000 7f2acf7fe700 1 -- 172.21.15.189:0/2177348965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ad00cd3e0 msgr2=0x7f2ad01f6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.091+0000 7f2acf7fe700 1 --2- 172.21.15.189:0/2177348965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ad00cd3e0 0x7f2ad01f6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.091+0000 7f2acf7fe700 1 -- 172.21.15.189:0/2177348965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ad006b8c0 msgr2=0x7f2ad01f3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.092+0000 7f2acf7fe700 1 --2- 172.21.15.189:0/2177348965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ad006b8c0 0x7f2ad01f3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.092+0000 7f2acf7fe700 1 -- 172.21.15.189:0/2177348965 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac8009a70 con 0x7f2ad015d7c0 2024-04-06T13:48:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.092+0000 7f2aceffd700 1 --2- 172.21.15.189:0/2177348965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ad006b8c0 0x7f2ad01f3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.092+0000 7f2acf7fe700 1 --2- 172.21.15.189:0/2177348965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ad015d7c0 0x7f2ad015f250 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8009e60 tx=0x7f2ac800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.092+0000 7f2ab77fe700 1 -- 172.21.15.189:0/2177348965 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8009cd0 con 0x7f2ad015d7c0 2024-04-06T13:48:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.092+0000 7f2ab77fe700 1 -- 172.21.15.189:0/2177348965 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac800f040 con 0x7f2ad015d7c0 2024-04-06T13:48:40.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.092+0000 7f2ad533a700 1 -- 172.21.15.189:0/2177348965 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac40094a0 con 0x7f2ad015d7c0 2024-04-06T13:48:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.092+0000 7f2ab77fe700 1 -- 172.21.15.189:0/2177348965 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac800ce20 con 0x7f2ad015d7c0 2024-04-06T13:48:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.092+0000 7f2ad533a700 1 -- 172.21.15.189:0/2177348965 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad00c48a0 con 0x7f2ad015d7c0 2024-04-06T13:48:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.093+0000 7f2ad533a700 1 -- 172.21.15.189:0/2177348965 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad006d3a0 con 0x7f2ad015d7c0 2024-04-06T13:48:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.093+0000 7f2ab77fe700 1 -- 172.21.15.189:0/2177348965 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f2ac8014480 con 0x7f2ad015d7c0 2024-04-06T13:48:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.094+0000 7f2ab77fe700 1 --2- 172.21.15.189:0/2177348965 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2abc06aed0 0x7f2abc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.094+0000 7f2ab77fe700 1 -- 172.21.15.189:0/2177348965 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f2ac808a390 con 0x7f2ad015d7c0 2024-04-06T13:48:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.094+0000 7f2aceffd700 1 --2- 172.21.15.189:0/2177348965 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2abc06aed0 0x7f2abc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:40.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.094+0000 7f2aceffd700 1 --2- 172.21.15.189:0/2177348965 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2abc06aed0 0x7f2abc06d390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2ac40090b0 tx=0x7f2ac4008030 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:40.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.098+0000 7f2ab77fe700 1 -- 172.21.15.189:0/2177348965 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2ac8054ca0 con 0x7f2ad015d7c0 2024-04-06T13:48:40.252 INFO:tasks.ceph.mds.b.smithi192.stderr:2024-04-06T13:48:40.252+0000 7fe83f367700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T13:48:40.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.300+0000 7f2ad533a700 1 -- 172.21.15.189:0/2177348965 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ad006d100 con 0x7f2abc06aed0 2024-04-06T13:48:40.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.314+0000 7f2ab77fe700 1 -- 172.21.15.189:0/2177348965 <== mgr.11733 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+529167 (secure 0 0 0) 0x7f2ac4011b00 con 0x7f2abc06aed0 2024-04-06T13:48:40.316 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:40.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.318+0000 7f2ad533a700 1 -- 172.21.15.189:0/2177348965 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2abc06aed0 msgr2=0x7f2abc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:40.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.318+0000 7f2ad533a700 1 --2- 172.21.15.189:0/2177348965 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2abc06aed0 0x7f2abc06d390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2ac40090b0 tx=0x7f2ac4008030 comp rx=0 tx=0).stop 2024-04-06T13:48:40.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.318+0000 7f2ad533a700 1 -- 172.21.15.189:0/2177348965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ad015d7c0 msgr2=0x7f2ad015f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:40.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.318+0000 7f2ad533a700 1 --2- 172.21.15.189:0/2177348965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ad015d7c0 0x7f2ad015f250 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8009e60 tx=0x7f2ac800c3a0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.318+0000 7f2ad533a700 1 -- 172.21.15.189:0/2177348965 shutdown_connections 2024-04-06T13:48:40.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.318+0000 7f2ad533a700 1 --2- 172.21.15.189:0/2177348965 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2abc06aed0 0x7f2abc06d390 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.318+0000 7f2ad533a700 1 --2- 172.21.15.189:0/2177348965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ad015d7c0 0x7f2ad015f250 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.318+0000 7f2ad533a700 1 --2- 172.21.15.189:0/2177348965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ad006b8c0 0x7f2ad01f3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.318+0000 7f2ad533a700 1 --2- 172.21.15.189:0/2177348965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ad00cd3e0 0x7f2ad01f6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.318+0000 7f2ad533a700 1 -- 172.21.15.189:0/2177348965 >> 172.21.15.189:0/2177348965 conn(0x7f2ad00cc290 msgr2=0x7f2ad0160db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:40.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.318+0000 7f2ad533a700 1 -- 172.21.15.189:0/2177348965 shutdown_connections 2024-04-06T13:48:40.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.319+0000 7f2ad533a700 1 -- 172.21.15.189:0/2177348965 wait complete. 2024-04-06T13:48:40.330 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:48:40.332 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-04-06T13:48:39.931078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596341,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":710,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":449,"ondisk_log_size":449,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":288,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":457908,"kb_used_data":26032,"kb_used_omap":12,"kb_used_meta":431731,"kb_avail":754516812,"statfs":{"total":773094113280,"available":772625215488,"internally_reserved":0,"allocated":26656768,"data_stored":14242782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":442093112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3349,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.150915"},"pg_stats":[{"pgid":"34.1d","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.208437+0000","last_change":"2024-04-06T13:48:36.223675+0000","last_active":"2024-04-06T13:48:37.208437+0000","last_peered":"2024-04-06T13:48:37.208437+0000","last_clean":"2024-04-06T13:48:37.208437+0000","last_became_active":"2024-04-06T13:48:36.223406+0000","last_became_peered":"2024-04-06T13:48:36.223406+0000","last_unstale":"2024-04-06T13:48:37.208437+0000","last_undegraded":"2024-04-06T13:48:37.208437+0000","last_fullsized":"2024-04-06T13:48:37.208437+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:01:12.661900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"251'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204182+0000","last_change":"2024-04-06T13:48:35.801830+0000","last_active":"2024-04-06T13:48:38.204182+0000","last_peered":"2024-04-06T13:48:38.204182+0000","last_clean":"2024-04-06T13:48:38.204182+0000","last_became_active":"2024-04-06T13:48:35.801558+0000","last_became_peered":"2024-04-06T13:48:35.801558+0000","last_unstale":"2024-04-06T13:48:38.204182+0000","last_undegraded":"2024-04-06T13:48:38.204182+0000","last_fullsized":"2024-04-06T13:48:38.204182+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:35.037760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204196+0000","last_change":"2024-04-06T13:48:19.691215+0000","last_active":"2024-04-06T13:48:36.204196+0000","last_peered":"2024-04-06T13:48:36.204196+0000","last_clean":"2024-04-06T13:48:36.204196+0000","last_became_active":"2024-04-06T13:48:19.689693+0000","last_became_peered":"2024-04-06T13:48:19.689693+0000","last_unstale":"2024-04-06T13:48:36.204196+0000","last_undegraded":"2024-04-06T13:48:36.204196+0000","last_fullsized":"2024-04-06T13:48:36.204196+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:49:37.511498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204036+0000","last_change":"2024-04-06T13:48:36.216544+0000","last_active":"2024-04-06T13:48:38.204036+0000","last_peered":"2024-04-06T13:48:38.204036+0000","last_clean":"2024-04-06T13:48:38.204036+0000","last_became_active":"2024-04-06T13:48:36.216226+0000","last_became_peered":"2024-04-06T13:48:36.216226+0000","last_unstale":"2024-04-06T13:48:38.204036+0000","last_undegraded":"2024-04-06T13:48:38.204036+0000","last_fullsized":"2024-04-06T13:48:38.204036+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:41.402605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"251'4","reported_seq":15,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.208661+0000","last_change":"2024-04-06T13:48:35.801039+0000","last_active":"2024-04-06T13:48:37.208661+0000","last_peered":"2024-04-06T13:48:37.208661+0000","last_clean":"2024-04-06T13:48:37.208661+0000","last_became_active":"2024-04-06T13:48:35.800800+0000","last_became_peered":"2024-04-06T13:48:35.800800+0000","last_unstale":"2024-04-06T13:48:37.208661+0000","last_undegraded":"2024-04-06T13:48:37.208661+0000","last_fullsized":"2024-04-06T13:48:37.208661+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:35.037760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204220+0000","last_change":"2024-04-06T13:48:19.704688+0000","last_active":"2024-04-06T13:48:36.204220+0000","last_peered":"2024-04-06T13:48:36.204220+0000","last_clean":"2024-04-06T13:48:36.204220+0000","last_became_active":"2024-04-06T13:48:19.704088+0000","last_became_peered":"2024-04-06T13:48:19.704088+0000","last_unstale":"2024-04-06T13:48:36.204220+0000","last_undegraded":"2024-04-06T13:48:36.204220+0000","last_fullsized":"2024-04-06T13:48:36.204220+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:49.108653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205108+0000","last_change":"2024-04-06T13:48:36.226178+0000","last_active":"2024-04-06T13:48:38.205108+0000","last_peered":"2024-04-06T13:48:38.205108+0000","last_clean":"2024-04-06T13:48:38.205108+0000","last_became_active":"2024-04-06T13:48:36.225943+0000","last_became_peered":"2024-04-06T13:48:36.225943+0000","last_unstale":"2024-04-06T13:48:38.205108+0000","last_undegraded":"2024-04-06T13:48:38.205108+0000","last_fullsized":"2024-04-06T13:48:38.205108+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:05:37.001407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204208+0000","last_change":"2024-04-06T13:48:35.801138+0000","last_active":"2024-04-06T13:48:38.204208+0000","last_peered":"2024-04-06T13:48:38.204208+0000","last_clean":"2024-04-06T13:48:38.204208+0000","last_became_active":"2024-04-06T13:48:35.800885+0000","last_became_peered":"2024-04-06T13:48:35.800885+0000","last_unstale":"2024-04-06T13:48:38.204208+0000","last_undegraded":"2024-04-06T13:48:38.204208+0000","last_fullsized":"2024-04-06T13:48:38.204208+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:55:27.291993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206106+0000","last_change":"2024-04-06T13:48:19.697461+0000","last_active":"2024-04-06T13:48:36.206106+0000","last_peered":"2024-04-06T13:48:36.206106+0000","last_clean":"2024-04-06T13:48:36.206106+0000","last_became_active":"2024-04-06T13:48:19.695459+0000","last_became_peered":"2024-04-06T13:48:19.695459+0000","last_unstale":"2024-04-06T13:48:36.206106+0000","last_undegraded":"2024-04-06T13:48:36.206106+0000","last_fullsized":"2024-04-06T13:48:36.206106+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:42:12.990594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.208330+0000","last_change":"2024-04-06T13:48:36.213647+0000","last_active":"2024-04-06T13:48:37.208330+0000","last_peered":"2024-04-06T13:48:37.208330+0000","last_clean":"2024-04-06T13:48:37.208330+0000","last_became_active":"2024-04-06T13:48:36.213389+0000","last_became_peered":"2024-04-06T13:48:36.213389+0000","last_unstale":"2024-04-06T13:48:37.208330+0000","last_undegraded":"2024-04-06T13:48:37.208330+0000","last_fullsized":"2024-04-06T13:48:37.208330+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:32.688952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.206177+0000","last_change":"2024-04-06T13:48:35.804173+0000","last_active":"2024-04-06T13:48:38.206177+0000","last_peered":"2024-04-06T13:48:38.206177+0000","last_clean":"2024-04-06T13:48:38.206177+0000","last_became_active":"2024-04-06T13:48:35.803800+0000","last_became_peered":"2024-04-06T13:48:35.803800+0000","last_unstale":"2024-04-06T13:48:38.206177+0000","last_undegraded":"2024-04-06T13:48:38.206177+0000","last_fullsized":"2024-04-06T13:48:38.206177+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:38:28.055682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206402+0000","last_change":"2024-04-06T13:48:19.690589+0000","last_active":"2024-04-06T13:48:36.206402+0000","last_peered":"2024-04-06T13:48:36.206402+0000","last_clean":"2024-04-06T13:48:36.206402+0000","last_became_active":"2024-04-06T13:48:19.690338+0000","last_became_peered":"2024-04-06T13:48:19.690338+0000","last_unstale":"2024-04-06T13:48:36.206402+0000","last_undegraded":"2024-04-06T13:48:36.206402+0000","last_fullsized":"2024-04-06T13:48:36.206402+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:51:42.234400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204276+0000","last_change":"2024-04-06T13:48:36.218773+0000","last_active":"2024-04-06T13:48:38.204276+0000","last_peered":"2024-04-06T13:48:38.204276+0000","last_clean":"2024-04-06T13:48:38.204276+0000","last_became_active":"2024-04-06T13:48:36.218530+0000","last_became_peered":"2024-04-06T13:48:36.218530+0000","last_unstale":"2024-04-06T13:48:38.204276+0000","last_undegraded":"2024-04-06T13:48:38.204276+0000","last_fullsized":"2024-04-06T13:48:38.204276+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:16:51.992527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"251'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204961+0000","last_change":"2024-04-06T13:48:35.808825+0000","last_active":"2024-04-06T13:48:38.204961+0000","last_peered":"2024-04-06T13:48:38.204961+0000","last_clean":"2024-04-06T13:48:38.204961+0000","last_became_active":"2024-04-06T13:48:35.808542+0000","last_became_peered":"2024-04-06T13:48:35.808542+0000","last_unstale":"2024-04-06T13:48:38.204961+0000","last_undegraded":"2024-04-06T13:48:38.204961+0000","last_fullsized":"2024-04-06T13:48:38.204961+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:15:23.628075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.214984+0000","last_change":"2024-04-06T13:48:19.698891+0000","last_active":"2024-04-06T13:48:36.214984+0000","last_peered":"2024-04-06T13:48:36.214984+0000","last_clean":"2024-04-06T13:48:36.214984+0000","last_became_active":"2024-04-06T13:48:19.698477+0000","last_became_peered":"2024-04-06T13:48:19.698477+0000","last_unstale":"2024-04-06T13:48:36.214984+0000","last_undegraded":"2024-04-06T13:48:36.214984+0000","last_fullsized":"2024-04-06T13:48:36.214984+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:02:19.171219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203426+0000","last_change":"2024-04-06T13:48:36.215117+0000","last_active":"2024-04-06T13:48:38.203426+0000","last_peered":"2024-04-06T13:48:38.203426+0000","last_clean":"2024-04-06T13:48:38.203426+0000","last_became_active":"2024-04-06T13:48:36.214844+0000","last_became_peered":"2024-04-06T13:48:36.214844+0000","last_unstale":"2024-04-06T13:48:38.203426+0000","last_undegraded":"2024-04-06T13:48:38.203426+0000","last_fullsized":"2024-04-06T13:48:38.203426+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:00:12.108708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"251'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203983+0000","last_change":"2024-04-06T13:48:35.803442+0000","last_active":"2024-04-06T13:48:38.203983+0000","last_peered":"2024-04-06T13:48:38.203983+0000","last_clean":"2024-04-06T13:48:38.203983+0000","last_became_active":"2024-04-06T13:48:35.803182+0000","last_became_peered":"2024-04-06T13:48:35.803182+0000","last_unstale":"2024-04-06T13:48:38.203983+0000","last_undegraded":"2024-04-06T13:48:38.203983+0000","last_fullsized":"2024-04-06T13:48:38.203983+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:55:12.893835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.203677+0000","last_change":"2024-04-06T13:48:19.684151+0000","last_active":"2024-04-06T13:48:36.203677+0000","last_peered":"2024-04-06T13:48:36.203677+0000","last_clean":"2024-04-06T13:48:36.203677+0000","last_became_active":"2024-04-06T13:48:19.683897+0000","last_became_peered":"2024-04-06T13:48:19.683897+0000","last_unstale":"2024-04-06T13:48:36.203677+0000","last_undegraded":"2024-04-06T13:48:36.203677+0000","last_fullsized":"2024-04-06T13:48:36.203677+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:47.270406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212114+0000","last_change":"2024-04-06T13:48:36.211844+0000","last_active":"2024-04-06T13:48:37.212114+0000","last_peered":"2024-04-06T13:48:37.212114+0000","last_clean":"2024-04-06T13:48:37.212114+0000","last_became_active":"2024-04-06T13:48:36.210158+0000","last_became_peered":"2024-04-06T13:48:36.210158+0000","last_unstale":"2024-04-06T13:48:37.212114+0000","last_undegraded":"2024-04-06T13:48:37.212114+0000","last_fullsized":"2024-04-06T13:48:37.212114+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:16:24.874180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203419+0000","last_change":"2024-04-06T13:48:35.798263+0000","last_active":"2024-04-06T13:48:38.203419+0000","last_peered":"2024-04-06T13:48:38.203419+0000","last_clean":"2024-04-06T13:48:38.203419+0000","last_became_active":"2024-04-06T13:48:35.798010+0000","last_became_peered":"2024-04-06T13:48:35.798010+0000","last_unstale":"2024-04-06T13:48:38.203419+0000","last_undegraded":"2024-04-06T13:48:38.203419+0000","last_fullsized":"2024-04-06T13:48:38.203419+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:05:41.494271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.201592+0000","last_change":"2024-04-06T13:48:19.695665+0000","last_active":"2024-04-06T13:48:36.201592+0000","last_peered":"2024-04-06T13:48:36.201592+0000","last_clean":"2024-04-06T13:48:36.201592+0000","last_became_active":"2024-04-06T13:48:19.695362+0000","last_became_peered":"2024-04-06T13:48:19.695362+0000","last_unstale":"2024-04-06T13:48:36.201592+0000","last_undegraded":"2024-04-06T13:48:36.201592+0000","last_fullsized":"2024-04-06T13:48:36.201592+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:26:21.699641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203763+0000","last_change":"2024-04-06T13:48:36.224661+0000","last_active":"2024-04-06T13:48:38.203763+0000","last_peered":"2024-04-06T13:48:38.203763+0000","last_clean":"2024-04-06T13:48:38.203763+0000","last_became_active":"2024-04-06T13:48:36.220854+0000","last_became_peered":"2024-04-06T13:48:36.220854+0000","last_unstale":"2024-04-06T13:48:38.203763+0000","last_undegraded":"2024-04-06T13:48:38.203763+0000","last_fullsized":"2024-04-06T13:48:38.203763+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:38:22.774628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204518+0000","last_change":"2024-04-06T13:48:35.799944+0000","last_active":"2024-04-06T13:48:38.204518+0000","last_peered":"2024-04-06T13:48:38.204518+0000","last_clean":"2024-04-06T13:48:38.204518+0000","last_became_active":"2024-04-06T13:48:35.799661+0000","last_became_peered":"2024-04-06T13:48:35.799661+0000","last_unstale":"2024-04-06T13:48:38.204518+0000","last_undegraded":"2024-04-06T13:48:38.204518+0000","last_fullsized":"2024-04-06T13:48:38.204518+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:40:56.233528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213567+0000","last_change":"2024-04-06T13:48:19.704200+0000","last_active":"2024-04-06T13:48:36.213567+0000","last_peered":"2024-04-06T13:48:36.213567+0000","last_clean":"2024-04-06T13:48:36.213567+0000","last_became_active":"2024-04-06T13:48:19.703951+0000","last_became_peered":"2024-04-06T13:48:19.703951+0000","last_unstale":"2024-04-06T13:48:36.213567+0000","last_undegraded":"2024-04-06T13:48:36.213567+0000","last_fullsized":"2024-04-06T13:48:36.213567+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:31:20.020434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.208218+0000","last_change":"2024-04-06T13:48:36.221195+0000","last_active":"2024-04-06T13:48:37.208218+0000","last_peered":"2024-04-06T13:48:37.208218+0000","last_clean":"2024-04-06T13:48:37.208218+0000","last_became_active":"2024-04-06T13:48:36.220949+0000","last_became_peered":"2024-04-06T13:48:36.220949+0000","last_unstale":"2024-04-06T13:48:37.208218+0000","last_undegraded":"2024-04-06T13:48:37.208218+0000","last_fullsized":"2024-04-06T13:48:37.208218+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:41.402605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204287+0000","last_change":"2024-04-06T13:48:35.800025+0000","last_active":"2024-04-06T13:48:38.204287+0000","last_peered":"2024-04-06T13:48:38.204287+0000","last_clean":"2024-04-06T13:48:38.204287+0000","last_became_active":"2024-04-06T13:48:35.799760+0000","last_became_peered":"2024-04-06T13:48:35.799760+0000","last_unstale":"2024-04-06T13:48:38.204287+0000","last_undegraded":"2024-04-06T13:48:38.204287+0000","last_fullsized":"2024-04-06T13:48:38.204287+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:17:41.311632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204323+0000","last_change":"2024-04-06T13:48:19.705385+0000","last_active":"2024-04-06T13:48:36.204323+0000","last_peered":"2024-04-06T13:48:36.204323+0000","last_clean":"2024-04-06T13:48:36.204323+0000","last_became_active":"2024-04-06T13:48:19.705091+0000","last_became_peered":"2024-04-06T13:48:19.705091+0000","last_unstale":"2024-04-06T13:48:36.204323+0000","last_undegraded":"2024-04-06T13:48:36.204323+0000","last_fullsized":"2024-04-06T13:48:36.204323+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:23:39.345908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212151+0000","last_change":"2024-04-06T13:48:36.224819+0000","last_active":"2024-04-06T13:48:37.212151+0000","last_peered":"2024-04-06T13:48:37.212151+0000","last_clean":"2024-04-06T13:48:37.212151+0000","last_became_active":"2024-04-06T13:48:36.224546+0000","last_became_peered":"2024-04-06T13:48:36.224546+0000","last_unstale":"2024-04-06T13:48:37.212151+0000","last_undegraded":"2024-04-06T13:48:37.212151+0000","last_fullsized":"2024-04-06T13:48:37.212151+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:12:07.931133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212331+0000","last_change":"2024-04-06T13:48:35.808073+0000","last_active":"2024-04-06T13:48:37.212331+0000","last_peered":"2024-04-06T13:48:37.212331+0000","last_clean":"2024-04-06T13:48:37.212331+0000","last_became_active":"2024-04-06T13:48:35.807847+0000","last_became_peered":"2024-04-06T13:48:35.807847+0000","last_unstale":"2024-04-06T13:48:37.212331+0000","last_undegraded":"2024-04-06T13:48:37.212331+0000","last_fullsized":"2024-04-06T13:48:37.212331+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:26:50.278260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206598+0000","last_change":"2024-04-06T13:48:19.700300+0000","last_active":"2024-04-06T13:48:36.206598+0000","last_peered":"2024-04-06T13:48:36.206598+0000","last_clean":"2024-04-06T13:48:36.206598+0000","last_became_active":"2024-04-06T13:48:19.700054+0000","last_became_peered":"2024-04-06T13:48:19.700054+0000","last_unstale":"2024-04-06T13:48:36.206598+0000","last_undegraded":"2024-04-06T13:48:36.206598+0000","last_fullsized":"2024-04-06T13:48:36.206598+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:30:14.717681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.206279+0000","last_change":"2024-04-06T13:48:36.219005+0000","last_active":"2024-04-06T13:48:38.206279+0000","last_peered":"2024-04-06T13:48:38.206279+0000","last_clean":"2024-04-06T13:48:38.206279+0000","last_became_active":"2024-04-06T13:48:36.218703+0000","last_became_peered":"2024-04-06T13:48:36.218703+0000","last_unstale":"2024-04-06T13:48:38.206279+0000","last_undegraded":"2024-04-06T13:48:38.206279+0000","last_fullsized":"2024-04-06T13:48:38.206279+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:41:54.237434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"251'6","reported_seq":17,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.208150+0000","last_change":"2024-04-06T13:48:35.798075+0000","last_active":"2024-04-06T13:48:37.208150+0000","last_peered":"2024-04-06T13:48:37.208150+0000","last_clean":"2024-04-06T13:48:37.208150+0000","last_became_active":"2024-04-06T13:48:35.797577+0000","last_became_peered":"2024-04-06T13:48:35.797577+0000","last_unstale":"2024-04-06T13:48:37.208150+0000","last_undegraded":"2024-04-06T13:48:37.208150+0000","last_fullsized":"2024-04-06T13:48:37.208150+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:17:41.311632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206244+0000","last_change":"2024-04-06T13:48:19.700151+0000","last_active":"2024-04-06T13:48:36.206244+0000","last_peered":"2024-04-06T13:48:36.206244+0000","last_clean":"2024-04-06T13:48:36.206244+0000","last_became_active":"2024-04-06T13:48:19.699882+0000","last_became_peered":"2024-04-06T13:48:19.699882+0000","last_unstale":"2024-04-06T13:48:36.206244+0000","last_undegraded":"2024-04-06T13:48:36.206244+0000","last_fullsized":"2024-04-06T13:48:36.206244+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:24.879568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.213000+0000","last_change":"2024-04-06T13:48:36.225507+0000","last_active":"2024-04-06T13:48:37.213000+0000","last_peered":"2024-04-06T13:48:37.213000+0000","last_clean":"2024-04-06T13:48:37.213000+0000","last_became_active":"2024-04-06T13:48:36.225261+0000","last_became_peered":"2024-04-06T13:48:36.225261+0000","last_unstale":"2024-04-06T13:48:37.213000+0000","last_undegraded":"2024-04-06T13:48:37.213000+0000","last_fullsized":"2024-04-06T13:48:37.213000+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:24:22.367478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"251'3","reported_seq":14,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212628+0000","last_change":"2024-04-06T13:48:35.808132+0000","last_active":"2024-04-06T13:48:37.212628+0000","last_peered":"2024-04-06T13:48:37.212628+0000","last_clean":"2024-04-06T13:48:37.212628+0000","last_became_active":"2024-04-06T13:48:35.807646+0000","last_became_peered":"2024-04-06T13:48:35.807646+0000","last_unstale":"2024-04-06T13:48:37.212628+0000","last_undegraded":"2024-04-06T13:48:37.212628+0000","last_fullsized":"2024-04-06T13:48:37.212628+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:35:18.103406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.203137+0000","last_change":"2024-04-06T13:48:19.683194+0000","last_active":"2024-04-06T13:48:36.203137+0000","last_peered":"2024-04-06T13:48:36.203137+0000","last_clean":"2024-04-06T13:48:36.203137+0000","last_became_active":"2024-04-06T13:48:19.682953+0000","last_became_peered":"2024-04-06T13:48:19.682953+0000","last_unstale":"2024-04-06T13:48:36.203137+0000","last_undegraded":"2024-04-06T13:48:36.203137+0000","last_fullsized":"2024-04-06T13:48:36.203137+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:33:52.351082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205355+0000","last_change":"2024-04-06T13:48:36.220815+0000","last_active":"2024-04-06T13:48:38.205355+0000","last_peered":"2024-04-06T13:48:38.205355+0000","last_clean":"2024-04-06T13:48:38.205355+0000","last_became_active":"2024-04-06T13:48:36.220545+0000","last_became_peered":"2024-04-06T13:48:36.220545+0000","last_unstale":"2024-04-06T13:48:38.205355+0000","last_undegraded":"2024-04-06T13:48:38.205355+0000","last_fullsized":"2024-04-06T13:48:38.205355+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:00:42.061268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.211883+0000","last_change":"2024-04-06T13:48:35.807074+0000","last_active":"2024-04-06T13:48:37.211883+0000","last_peered":"2024-04-06T13:48:37.211883+0000","last_clean":"2024-04-06T13:48:37.211883+0000","last_became_active":"2024-04-06T13:48:35.806811+0000","last_became_peered":"2024-04-06T13:48:35.806811+0000","last_unstale":"2024-04-06T13:48:37.211883+0000","last_undegraded":"2024-04-06T13:48:37.211883+0000","last_fullsized":"2024-04-06T13:48:37.211883+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:28:11.587938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212271+0000","last_change":"2024-04-06T13:48:19.690929+0000","last_active":"2024-04-06T13:48:36.212271+0000","last_peered":"2024-04-06T13:48:36.212271+0000","last_clean":"2024-04-06T13:48:36.212271+0000","last_became_active":"2024-04-06T13:48:19.690663+0000","last_became_peered":"2024-04-06T13:48:19.690663+0000","last_unstale":"2024-04-06T13:48:36.212271+0000","last_undegraded":"2024-04-06T13:48:36.212271+0000","last_fullsized":"2024-04-06T13:48:36.212271+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:52:01.901954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203338+0000","last_change":"2024-04-06T13:48:36.217030+0000","last_active":"2024-04-06T13:48:38.203338+0000","last_peered":"2024-04-06T13:48:38.203338+0000","last_clean":"2024-04-06T13:48:38.203338+0000","last_became_active":"2024-04-06T13:48:36.216760+0000","last_became_peered":"2024-04-06T13:48:36.216760+0000","last_unstale":"2024-04-06T13:48:38.203338+0000","last_undegraded":"2024-04-06T13:48:38.203338+0000","last_fullsized":"2024-04-06T13:48:38.203338+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:15:24.634356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204288+0000","last_change":"2024-04-06T13:48:35.800985+0000","last_active":"2024-04-06T13:48:38.204288+0000","last_peered":"2024-04-06T13:48:38.204288+0000","last_clean":"2024-04-06T13:48:38.204288+0000","last_became_active":"2024-04-06T13:48:35.800742+0000","last_became_peered":"2024-04-06T13:48:35.800742+0000","last_unstale":"2024-04-06T13:48:38.204288+0000","last_undegraded":"2024-04-06T13:48:38.204288+0000","last_fullsized":"2024-04-06T13:48:38.204288+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:56:50.591253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.207498+0000","last_change":"2024-04-06T13:48:19.693212+0000","last_active":"2024-04-06T13:48:36.207498+0000","last_peered":"2024-04-06T13:48:36.207498+0000","last_clean":"2024-04-06T13:48:36.207498+0000","last_became_active":"2024-04-06T13:48:19.692957+0000","last_became_peered":"2024-04-06T13:48:19.692957+0000","last_unstale":"2024-04-06T13:48:36.207498+0000","last_undegraded":"2024-04-06T13:48:36.207498+0000","last_fullsized":"2024-04-06T13:48:36.207498+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:00:25.538231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203624+0000","last_change":"2024-04-06T13:48:36.220425+0000","last_active":"2024-04-06T13:48:38.203624+0000","last_peered":"2024-04-06T13:48:38.203624+0000","last_clean":"2024-04-06T13:48:38.203624+0000","last_became_active":"2024-04-06T13:48:36.219730+0000","last_became_peered":"2024-04-06T13:48:36.219730+0000","last_unstale":"2024-04-06T13:48:38.203624+0000","last_undegraded":"2024-04-06T13:48:38.203624+0000","last_fullsized":"2024-04-06T13:48:38.203624+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:41:38.433651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"251'8","reported_seq":19,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.211712+0000","last_change":"2024-04-06T13:48:35.806208+0000","last_active":"2024-04-06T13:48:37.211712+0000","last_peered":"2024-04-06T13:48:37.211712+0000","last_clean":"2024-04-06T13:48:37.211712+0000","last_became_active":"2024-04-06T13:48:35.805798+0000","last_became_peered":"2024-04-06T13:48:35.805798+0000","last_unstale":"2024-04-06T13:48:37.211712+0000","last_undegraded":"2024-04-06T13:48:37.211712+0000","last_fullsized":"2024-04-06T13:48:37.211712+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:43:51.499864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206417+0000","last_change":"2024-04-06T13:48:19.693508+0000","last_active":"2024-04-06T13:48:36.206417+0000","last_peered":"2024-04-06T13:48:36.206417+0000","last_clean":"2024-04-06T13:48:36.206417+0000","last_became_active":"2024-04-06T13:48:19.693250+0000","last_became_peered":"2024-04-06T13:48:19.693250+0000","last_unstale":"2024-04-06T13:48:36.206417+0000","last_undegraded":"2024-04-06T13:48:36.206417+0000","last_fullsized":"2024-04-06T13:48:36.206417+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:00:56.138863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203462+0000","last_change":"2024-04-06T13:48:36.218751+0000","last_active":"2024-04-06T13:48:38.203462+0000","last_peered":"2024-04-06T13:48:38.203462+0000","last_clean":"2024-04-06T13:48:38.203462+0000","last_became_active":"2024-04-06T13:48:36.217434+0000","last_became_peered":"2024-04-06T13:48:36.217434+0000","last_unstale":"2024-04-06T13:48:38.203462+0000","last_undegraded":"2024-04-06T13:48:38.203462+0000","last_fullsized":"2024-04-06T13:48:38.203462+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:49:54.034535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.211807+0000","last_change":"2024-04-06T13:48:35.803827+0000","last_active":"2024-04-06T13:48:37.211807+0000","last_peered":"2024-04-06T13:48:37.211807+0000","last_clean":"2024-04-06T13:48:37.211807+0000","last_became_active":"2024-04-06T13:48:35.803457+0000","last_became_peered":"2024-04-06T13:48:35.803457+0000","last_unstale":"2024-04-06T13:48:37.211807+0000","last_undegraded":"2024-04-06T13:48:37.211807+0000","last_fullsized":"2024-04-06T13:48:37.211807+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:50:30.176310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.205584+0000","last_change":"2024-04-06T13:48:19.700696+0000","last_active":"2024-04-06T13:48:36.205584+0000","last_peered":"2024-04-06T13:48:36.205584+0000","last_clean":"2024-04-06T13:48:36.205584+0000","last_became_active":"2024-04-06T13:48:19.700444+0000","last_became_peered":"2024-04-06T13:48:19.700444+0000","last_unstale":"2024-04-06T13:48:36.205584+0000","last_undegraded":"2024-04-06T13:48:36.205584+0000","last_fullsized":"2024-04-06T13:48:36.205584+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:46.094428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.206066+0000","last_change":"2024-04-06T13:48:36.226075+0000","last_active":"2024-04-06T13:48:38.206066+0000","last_peered":"2024-04-06T13:48:38.206066+0000","last_clean":"2024-04-06T13:48:38.206066+0000","last_became_active":"2024-04-06T13:48:36.225818+0000","last_became_peered":"2024-04-06T13:48:36.225818+0000","last_unstale":"2024-04-06T13:48:38.206066+0000","last_undegraded":"2024-04-06T13:48:38.206066+0000","last_fullsized":"2024-04-06T13:48:38.206066+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:06:46.118211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.208572+0000","last_change":"2024-04-06T13:48:35.800512+0000","last_active":"2024-04-06T13:48:37.208572+0000","last_peered":"2024-04-06T13:48:37.208572+0000","last_clean":"2024-04-06T13:48:37.208572+0000","last_became_active":"2024-04-06T13:48:35.799815+0000","last_became_peered":"2024-04-06T13:48:35.799815+0000","last_unstale":"2024-04-06T13:48:37.208572+0000","last_undegraded":"2024-04-06T13:48:37.208572+0000","last_fullsized":"2024-04-06T13:48:37.208572+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:55:27.291993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213820+0000","last_change":"2024-04-06T13:48:19.700988+0000","last_active":"2024-04-06T13:48:36.213820+0000","last_peered":"2024-04-06T13:48:36.213820+0000","last_clean":"2024-04-06T13:48:36.213820+0000","last_became_active":"2024-04-06T13:48:19.700718+0000","last_became_peered":"2024-04-06T13:48:19.700718+0000","last_unstale":"2024-04-06T13:48:36.213820+0000","last_undegraded":"2024-04-06T13:48:36.213820+0000","last_fullsized":"2024-04-06T13:48:36.213820+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:37:50.946459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205901+0000","last_change":"2024-04-06T13:48:36.218918+0000","last_active":"2024-04-06T13:48:38.205901+0000","last_peered":"2024-04-06T13:48:38.205901+0000","last_clean":"2024-04-06T13:48:38.205901+0000","last_became_active":"2024-04-06T13:48:36.218625+0000","last_became_peered":"2024-04-06T13:48:36.218625+0000","last_unstale":"2024-04-06T13:48:38.205901+0000","last_undegraded":"2024-04-06T13:48:38.205901+0000","last_fullsized":"2024-04-06T13:48:38.205901+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:38:01.256020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"251'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.213136+0000","last_change":"2024-04-06T13:48:35.804317+0000","last_active":"2024-04-06T13:48:37.213136+0000","last_peered":"2024-04-06T13:48:37.213136+0000","last_clean":"2024-04-06T13:48:37.213136+0000","last_became_active":"2024-04-06T13:48:35.804011+0000","last_became_peered":"2024-04-06T13:48:35.804011+0000","last_unstale":"2024-04-06T13:48:37.213136+0000","last_undegraded":"2024-04-06T13:48:37.213136+0000","last_fullsized":"2024-04-06T13:48:37.213136+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:02:56.857425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206649+0000","last_change":"2024-04-06T13:48:19.700420+0000","last_active":"2024-04-06T13:48:36.206649+0000","last_peered":"2024-04-06T13:48:36.206649+0000","last_clean":"2024-04-06T13:48:36.206649+0000","last_became_active":"2024-04-06T13:48:19.698705+0000","last_became_peered":"2024-04-06T13:48:19.698705+0000","last_unstale":"2024-04-06T13:48:36.206649+0000","last_undegraded":"2024-04-06T13:48:36.206649+0000","last_fullsized":"2024-04-06T13:48:36.206649+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:29:58.555526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.206191+0000","last_change":"2024-04-06T13:48:36.220887+0000","last_active":"2024-04-06T13:48:38.206191+0000","last_peered":"2024-04-06T13:48:38.206191+0000","last_clean":"2024-04-06T13:48:38.206191+0000","last_became_active":"2024-04-06T13:48:36.220628+0000","last_became_peered":"2024-04-06T13:48:36.220628+0000","last_unstale":"2024-04-06T13:48:38.206191+0000","last_undegraded":"2024-04-06T13:48:38.206191+0000","last_fullsized":"2024-04-06T13:48:38.206191+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:50:20.772122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.207927+0000","last_change":"2024-04-06T13:48:35.800121+0000","last_active":"2024-04-06T13:48:37.207927+0000","last_peered":"2024-04-06T13:48:37.207927+0000","last_clean":"2024-04-06T13:48:37.207927+0000","last_became_active":"2024-04-06T13:48:35.799487+0000","last_became_peered":"2024-04-06T13:48:35.799487+0000","last_unstale":"2024-04-06T13:48:37.207927+0000","last_undegraded":"2024-04-06T13:48:37.207927+0000","last_fullsized":"2024-04-06T13:48:37.207927+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:31:41.226066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.207264+0000","last_change":"2024-04-06T13:48:19.695591+0000","last_active":"2024-04-06T13:48:36.207264+0000","last_peered":"2024-04-06T13:48:36.207264+0000","last_clean":"2024-04-06T13:48:36.207264+0000","last_became_active":"2024-04-06T13:48:19.695289+0000","last_became_peered":"2024-04-06T13:48:19.695289+0000","last_unstale":"2024-04-06T13:48:36.207264+0000","last_undegraded":"2024-04-06T13:48:36.207264+0000","last_fullsized":"2024-04-06T13:48:36.207264+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:16.165915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212010+0000","last_change":"2024-04-06T13:48:36.218655+0000","last_active":"2024-04-06T13:48:37.212010+0000","last_peered":"2024-04-06T13:48:37.212010+0000","last_clean":"2024-04-06T13:48:37.212010+0000","last_became_active":"2024-04-06T13:48:36.218402+0000","last_became_peered":"2024-04-06T13:48:36.218402+0000","last_unstale":"2024-04-06T13:48:37.212010+0000","last_undegraded":"2024-04-06T13:48:37.212010+0000","last_fullsized":"2024-04-06T13:48:37.212010+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:43:51.718388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"251'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204574+0000","last_change":"2024-04-06T13:48:35.805712+0000","last_active":"2024-04-06T13:48:38.204574+0000","last_peered":"2024-04-06T13:48:38.204574+0000","last_clean":"2024-04-06T13:48:38.204574+0000","last_became_active":"2024-04-06T13:48:35.804958+0000","last_became_peered":"2024-04-06T13:48:35.804958+0000","last_unstale":"2024-04-06T13:48:38.204574+0000","last_undegraded":"2024-04-06T13:48:38.204574+0000","last_fullsized":"2024-04-06T13:48:38.204574+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:31.682672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.203754+0000","last_change":"2024-04-06T13:48:19.702407+0000","last_active":"2024-04-06T13:48:36.203754+0000","last_peered":"2024-04-06T13:48:36.203754+0000","last_clean":"2024-04-06T13:48:36.203754+0000","last_became_active":"2024-04-06T13:48:19.702139+0000","last_became_peered":"2024-04-06T13:48:19.702139+0000","last_unstale":"2024-04-06T13:48:36.203754+0000","last_undegraded":"2024-04-06T13:48:36.203754+0000","last_fullsized":"2024-04-06T13:48:36.203754+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:33.923465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205476+0000","last_change":"2024-04-06T13:48:36.220266+0000","last_active":"2024-04-06T13:48:38.205476+0000","last_peered":"2024-04-06T13:48:38.205476+0000","last_clean":"2024-04-06T13:48:38.205476+0000","last_became_active":"2024-04-06T13:48:36.220004+0000","last_became_peered":"2024-04-06T13:48:36.220004+0000","last_unstale":"2024-04-06T13:48:38.205476+0000","last_undegraded":"2024-04-06T13:48:38.205476+0000","last_fullsized":"2024-04-06T13:48:38.205476+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:24:40.860500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204908+0000","last_change":"2024-04-06T13:48:35.802080+0000","last_active":"2024-04-06T13:48:38.204908+0000","last_peered":"2024-04-06T13:48:38.204908+0000","last_clean":"2024-04-06T13:48:38.204908+0000","last_became_active":"2024-04-06T13:48:35.801820+0000","last_became_peered":"2024-04-06T13:48:35.801820+0000","last_unstale":"2024-04-06T13:48:38.204908+0000","last_undegraded":"2024-04-06T13:48:38.204908+0000","last_fullsized":"2024-04-06T13:48:38.204908+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:36:48.849028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.200902+0000","last_change":"2024-04-06T13:48:19.692817+0000","last_active":"2024-04-06T13:48:36.200902+0000","last_peered":"2024-04-06T13:48:36.200902+0000","last_clean":"2024-04-06T13:48:36.200902+0000","last_became_active":"2024-04-06T13:48:19.692567+0000","last_became_peered":"2024-04-06T13:48:19.692567+0000","last_unstale":"2024-04-06T13:48:36.200902+0000","last_undegraded":"2024-04-06T13:48:36.200902+0000","last_fullsized":"2024-04-06T13:48:36.200902+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:22.432067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203845+0000","last_change":"2024-04-06T13:48:36.227364+0000","last_active":"2024-04-06T13:48:38.203845+0000","last_peered":"2024-04-06T13:48:38.203845+0000","last_clean":"2024-04-06T13:48:38.203845+0000","last_became_active":"2024-04-06T13:48:36.227165+0000","last_became_peered":"2024-04-06T13:48:36.227165+0000","last_unstale":"2024-04-06T13:48:38.203845+0000","last_undegraded":"2024-04-06T13:48:38.203845+0000","last_fullsized":"2024-04-06T13:48:38.203845+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:55:53.565152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204189+0000","last_change":"2024-04-06T13:48:35.809018+0000","last_active":"2024-04-06T13:48:38.204189+0000","last_peered":"2024-04-06T13:48:38.204189+0000","last_clean":"2024-04-06T13:48:38.204189+0000","last_became_active":"2024-04-06T13:48:35.808774+0000","last_became_peered":"2024-04-06T13:48:35.808774+0000","last_unstale":"2024-04-06T13:48:38.204189+0000","last_undegraded":"2024-04-06T13:48:38.204189+0000","last_fullsized":"2024-04-06T13:48:38.204189+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:58:35.871449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212065+0000","last_change":"2024-04-06T13:48:19.688698+0000","last_active":"2024-04-06T13:48:36.212065+0000","last_peered":"2024-04-06T13:48:36.212065+0000","last_clean":"2024-04-06T13:48:36.212065+0000","last_became_active":"2024-04-06T13:48:19.688377+0000","last_became_peered":"2024-04-06T13:48:19.688377+0000","last_unstale":"2024-04-06T13:48:36.212065+0000","last_undegraded":"2024-04-06T13:48:36.212065+0000","last_fullsized":"2024-04-06T13:48:36.212065+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:22.432067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204432+0000","last_change":"2024-04-06T13:48:36.225373+0000","last_active":"2024-04-06T13:48:38.204432+0000","last_peered":"2024-04-06T13:48:38.204432+0000","last_clean":"2024-04-06T13:48:38.204432+0000","last_became_active":"2024-04-06T13:48:36.220506+0000","last_became_peered":"2024-04-06T13:48:36.220506+0000","last_unstale":"2024-04-06T13:48:38.204432+0000","last_undegraded":"2024-04-06T13:48:38.204432+0000","last_fullsized":"2024-04-06T13:48:38.204432+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:34:44.870588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.211789+0000","last_change":"2024-04-06T13:48:35.806027+0000","last_active":"2024-04-06T13:48:37.211789+0000","last_peered":"2024-04-06T13:48:37.211789+0000","last_clean":"2024-04-06T13:48:37.211789+0000","last_became_active":"2024-04-06T13:48:35.804529+0000","last_became_peered":"2024-04-06T13:48:35.804529+0000","last_unstale":"2024-04-06T13:48:37.211789+0000","last_undegraded":"2024-04-06T13:48:37.211789+0000","last_fullsized":"2024-04-06T13:48:37.211789+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:28:01.756284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206334+0000","last_change":"2024-04-06T13:48:19.693370+0000","last_active":"2024-04-06T13:48:36.206334+0000","last_peered":"2024-04-06T13:48:36.206334+0000","last_clean":"2024-04-06T13:48:36.206334+0000","last_became_active":"2024-04-06T13:48:19.692893+0000","last_became_peered":"2024-04-06T13:48:19.692893+0000","last_unstale":"2024-04-06T13:48:36.206334+0000","last_undegraded":"2024-04-06T13:48:36.206334+0000","last_fullsized":"2024-04-06T13:48:36.206334+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:24.879568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.211799+0000","last_change":"2024-04-06T13:48:36.211404+0000","last_active":"2024-04-06T13:48:37.211799+0000","last_peered":"2024-04-06T13:48:37.211799+0000","last_clean":"2024-04-06T13:48:37.211799+0000","last_became_active":"2024-04-06T13:48:36.211170+0000","last_became_peered":"2024-04-06T13:48:36.211170+0000","last_unstale":"2024-04-06T13:48:37.211799+0000","last_undegraded":"2024-04-06T13:48:37.211799+0000","last_fullsized":"2024-04-06T13:48:37.211799+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:25:46.842981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204236+0000","last_change":"2024-04-06T13:48:35.802498+0000","last_active":"2024-04-06T13:48:38.204236+0000","last_peered":"2024-04-06T13:48:38.204236+0000","last_clean":"2024-04-06T13:48:38.204236+0000","last_became_active":"2024-04-06T13:48:35.801896+0000","last_became_peered":"2024-04-06T13:48:35.801896+0000","last_unstale":"2024-04-06T13:48:38.204236+0000","last_undegraded":"2024-04-06T13:48:38.204236+0000","last_fullsized":"2024-04-06T13:48:38.204236+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:58:02.787162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212571+0000","last_change":"2024-04-06T13:48:19.698243+0000","last_active":"2024-04-06T13:48:36.212571+0000","last_peered":"2024-04-06T13:48:36.212571+0000","last_clean":"2024-04-06T13:48:36.212571+0000","last_became_active":"2024-04-06T13:48:19.697956+0000","last_became_peered":"2024-04-06T13:48:19.697956+0000","last_unstale":"2024-04-06T13:48:36.212571+0000","last_undegraded":"2024-04-06T13:48:36.212571+0000","last_fullsized":"2024-04-06T13:48:36.212571+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:26:21.699641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212570+0000","last_change":"2024-04-06T13:48:36.213811+0000","last_active":"2024-04-06T13:48:37.212570+0000","last_peered":"2024-04-06T13:48:37.212570+0000","last_clean":"2024-04-06T13:48:37.212570+0000","last_became_active":"2024-04-06T13:48:36.213507+0000","last_became_peered":"2024-04-06T13:48:36.213507+0000","last_unstale":"2024-04-06T13:48:37.212570+0000","last_undegraded":"2024-04-06T13:48:37.212570+0000","last_fullsized":"2024-04-06T13:48:37.212570+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:34.972744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.208483+0000","last_change":"2024-04-06T13:48:35.800677+0000","last_active":"2024-04-06T13:48:37.208483+0000","last_peered":"2024-04-06T13:48:37.208483+0000","last_clean":"2024-04-06T13:48:37.208483+0000","last_became_active":"2024-04-06T13:48:35.800240+0000","last_became_peered":"2024-04-06T13:48:35.800240+0000","last_unstale":"2024-04-06T13:48:37.208483+0000","last_undegraded":"2024-04-06T13:48:37.208483+0000","last_fullsized":"2024-04-06T13:48:37.208483+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:40:56.233528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.205136+0000","last_change":"2024-04-06T13:48:19.694103+0000","last_active":"2024-04-06T13:48:36.205136+0000","last_peered":"2024-04-06T13:48:36.205136+0000","last_clean":"2024-04-06T13:48:36.205136+0000","last_became_active":"2024-04-06T13:48:19.693643+0000","last_became_peered":"2024-04-06T13:48:19.693643+0000","last_unstale":"2024-04-06T13:48:36.205136+0000","last_undegraded":"2024-04-06T13:48:36.205136+0000","last_fullsized":"2024-04-06T13:48:36.205136+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:00:35.709952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212674+0000","last_change":"2024-04-06T13:48:36.227862+0000","last_active":"2024-04-06T13:48:37.212674+0000","last_peered":"2024-04-06T13:48:37.212674+0000","last_clean":"2024-04-06T13:48:37.212674+0000","last_became_active":"2024-04-06T13:48:36.226969+0000","last_became_peered":"2024-04-06T13:48:36.226969+0000","last_unstale":"2024-04-06T13:48:37.212674+0000","last_undegraded":"2024-04-06T13:48:37.212674+0000","last_fullsized":"2024-04-06T13:48:37.212674+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:31:03.909867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212588+0000","last_change":"2024-04-06T13:48:35.807813+0000","last_active":"2024-04-06T13:48:37.212588+0000","last_peered":"2024-04-06T13:48:37.212588+0000","last_clean":"2024-04-06T13:48:37.212588+0000","last_became_active":"2024-04-06T13:48:35.807534+0000","last_became_peered":"2024-04-06T13:48:35.807534+0000","last_unstale":"2024-04-06T13:48:37.212588+0000","last_undegraded":"2024-04-06T13:48:37.212588+0000","last_fullsized":"2024-04-06T13:48:37.212588+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:05:16.392498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.202812+0000","last_change":"2024-04-06T13:48:19.697522+0000","last_active":"2024-04-06T13:48:36.202812+0000","last_peered":"2024-04-06T13:48:36.202812+0000","last_clean":"2024-04-06T13:48:36.202812+0000","last_became_active":"2024-04-06T13:48:19.695505+0000","last_became_peered":"2024-04-06T13:48:19.695505+0000","last_unstale":"2024-04-06T13:48:36.202812+0000","last_undegraded":"2024-04-06T13:48:36.202812+0000","last_fullsized":"2024-04-06T13:48:36.202812+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:37:50.946459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.213080+0000","last_change":"2024-04-06T13:48:36.223153+0000","last_active":"2024-04-06T13:48:37.213080+0000","last_peered":"2024-04-06T13:48:37.213080+0000","last_clean":"2024-04-06T13:48:37.213080+0000","last_became_active":"2024-04-06T13:48:36.222855+0000","last_became_peered":"2024-04-06T13:48:36.222855+0000","last_unstale":"2024-04-06T13:48:37.213080+0000","last_undegraded":"2024-04-06T13:48:37.213080+0000","last_fullsized":"2024-04-06T13:48:37.213080+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:50:08.995699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204589+0000","last_change":"2024-04-06T13:48:35.802506+0000","last_active":"2024-04-06T13:48:38.204589+0000","last_peered":"2024-04-06T13:48:38.204589+0000","last_clean":"2024-04-06T13:48:38.204589+0000","last_became_active":"2024-04-06T13:48:35.802227+0000","last_became_peered":"2024-04-06T13:48:35.802227+0000","last_unstale":"2024-04-06T13:48:38.204589+0000","last_undegraded":"2024-04-06T13:48:38.204589+0000","last_fullsized":"2024-04-06T13:48:38.204589+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:01:11.655619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213149+0000","last_change":"2024-04-06T13:48:19.699786+0000","last_active":"2024-04-06T13:48:36.213149+0000","last_peered":"2024-04-06T13:48:36.213149+0000","last_clean":"2024-04-06T13:48:36.213149+0000","last_became_active":"2024-04-06T13:48:19.698719+0000","last_became_peered":"2024-04-06T13:48:19.698719+0000","last_unstale":"2024-04-06T13:48:36.213149+0000","last_undegraded":"2024-04-06T13:48:36.213149+0000","last_fullsized":"2024-04-06T13:48:36.213149+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:54:26.958761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203992+0000","last_change":"2024-04-06T13:48:36.221117+0000","last_active":"2024-04-06T13:48:38.203992+0000","last_peered":"2024-04-06T13:48:38.203992+0000","last_clean":"2024-04-06T13:48:38.203992+0000","last_became_active":"2024-04-06T13:48:36.219036+0000","last_became_peered":"2024-04-06T13:48:36.219036+0000","last_unstale":"2024-04-06T13:48:38.203992+0000","last_undegraded":"2024-04-06T13:48:38.203992+0000","last_fullsized":"2024-04-06T13:48:38.203992+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:24:40.860500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203784+0000","last_change":"2024-04-06T13:48:35.804169+0000","last_active":"2024-04-06T13:48:38.203784+0000","last_peered":"2024-04-06T13:48:38.203784+0000","last_clean":"2024-04-06T13:48:38.203784+0000","last_became_active":"2024-04-06T13:48:35.803914+0000","last_became_peered":"2024-04-06T13:48:35.803914+0000","last_unstale":"2024-04-06T13:48:38.203784+0000","last_undegraded":"2024-04-06T13:48:38.203784+0000","last_fullsized":"2024-04-06T13:48:38.203784+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:42:28.507350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.203178+0000","last_change":"2024-04-06T13:48:19.704381+0000","last_active":"2024-04-06T13:48:36.203178+0000","last_peered":"2024-04-06T13:48:36.203178+0000","last_clean":"2024-04-06T13:48:36.203178+0000","last_became_active":"2024-04-06T13:48:19.704140+0000","last_became_peered":"2024-04-06T13:48:19.704140+0000","last_unstale":"2024-04-06T13:48:36.203178+0000","last_undegraded":"2024-04-06T13:48:36.203178+0000","last_fullsized":"2024-04-06T13:48:36.203178+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:30:47.386830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204158+0000","last_change":"2024-04-06T13:48:36.229581+0000","last_active":"2024-04-06T13:48:38.204158+0000","last_peered":"2024-04-06T13:48:38.204158+0000","last_clean":"2024-04-06T13:48:38.204158+0000","last_became_active":"2024-04-06T13:48:36.229260+0000","last_became_peered":"2024-04-06T13:48:36.229260+0000","last_unstale":"2024-04-06T13:48:38.204158+0000","last_undegraded":"2024-04-06T13:48:38.204158+0000","last_fullsized":"2024-04-06T13:48:38.204158+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:15:06.292012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.207690+0000","last_change":"2024-04-06T13:48:35.810188+0000","last_active":"2024-04-06T13:48:37.207690+0000","last_peered":"2024-04-06T13:48:37.207690+0000","last_clean":"2024-04-06T13:48:37.207690+0000","last_became_active":"2024-04-06T13:48:35.809890+0000","last_became_peered":"2024-04-06T13:48:35.809890+0000","last_unstale":"2024-04-06T13:48:37.207690+0000","last_undegraded":"2024-04-06T13:48:37.207690+0000","last_fullsized":"2024-04-06T13:48:37.207690+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:38:28.055682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.208700+0000","last_change":"2024-04-06T13:48:19.694038+0000","last_active":"2024-04-06T13:48:36.208700+0000","last_peered":"2024-04-06T13:48:36.208700+0000","last_clean":"2024-04-06T13:48:36.208700+0000","last_became_active":"2024-04-06T13:48:19.693775+0000","last_became_peered":"2024-04-06T13:48:19.693775+0000","last_unstale":"2024-04-06T13:48:36.208700+0000","last_undegraded":"2024-04-06T13:48:36.208700+0000","last_fullsized":"2024-04-06T13:48:36.208700+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:38:06.251591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.211631+0000","last_change":"2024-04-06T13:48:36.217070+0000","last_active":"2024-04-06T13:48:37.211631+0000","last_peered":"2024-04-06T13:48:37.211631+0000","last_clean":"2024-04-06T13:48:37.211631+0000","last_became_active":"2024-04-06T13:48:36.216807+0000","last_became_peered":"2024-04-06T13:48:36.216807+0000","last_unstale":"2024-04-06T13:48:37.211631+0000","last_undegraded":"2024-04-06T13:48:37.211631+0000","last_fullsized":"2024-04-06T13:48:37.211631+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:54:08.390619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.208024+0000","last_change":"2024-04-06T13:48:35.799922+0000","last_active":"2024-04-06T13:48:37.208024+0000","last_peered":"2024-04-06T13:48:37.208024+0000","last_clean":"2024-04-06T13:48:37.208024+0000","last_became_active":"2024-04-06T13:48:35.799654+0000","last_became_peered":"2024-04-06T13:48:35.799654+0000","last_unstale":"2024-04-06T13:48:37.208024+0000","last_undegraded":"2024-04-06T13:48:37.208024+0000","last_fullsized":"2024-04-06T13:48:37.208024+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:57:01.308231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.207573+0000","last_change":"2024-04-06T13:48:19.699860+0000","last_active":"2024-04-06T13:48:36.207573+0000","last_peered":"2024-04-06T13:48:36.207573+0000","last_clean":"2024-04-06T13:48:36.207573+0000","last_became_active":"2024-04-06T13:48:19.699567+0000","last_became_peered":"2024-04-06T13:48:19.699567+0000","last_unstale":"2024-04-06T13:48:36.207573+0000","last_undegraded":"2024-04-06T13:48:36.207573+0000","last_fullsized":"2024-04-06T13:48:36.207573+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:01:05.087706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204718+0000","last_change":"2024-04-06T13:48:36.215179+0000","last_active":"2024-04-06T13:48:38.204718+0000","last_peered":"2024-04-06T13:48:38.204718+0000","last_clean":"2024-04-06T13:48:38.204718+0000","last_became_active":"2024-04-06T13:48:36.213976+0000","last_became_peered":"2024-04-06T13:48:36.213976+0000","last_unstale":"2024-04-06T13:48:38.204718+0000","last_undegraded":"2024-04-06T13:48:38.204718+0000","last_fullsized":"2024-04-06T13:48:38.204718+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:42:29.513631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.207773+0000","last_change":"2024-04-06T13:48:35.797269+0000","last_active":"2024-04-06T13:48:37.207773+0000","last_peered":"2024-04-06T13:48:37.207773+0000","last_clean":"2024-04-06T13:48:37.207773+0000","last_became_active":"2024-04-06T13:48:35.796938+0000","last_became_peered":"2024-04-06T13:48:35.796938+0000","last_unstale":"2024-04-06T13:48:37.207773+0000","last_undegraded":"2024-04-06T13:48:37.207773+0000","last_fullsized":"2024-04-06T13:48:37.207773+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:47:01.611185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.203008+0000","last_change":"2024-04-06T13:48:19.697603+0000","last_active":"2024-04-06T13:48:36.203008+0000","last_peered":"2024-04-06T13:48:36.203008+0000","last_clean":"2024-04-06T13:48:36.203008+0000","last_became_active":"2024-04-06T13:48:19.697356+0000","last_became_peered":"2024-04-06T13:48:19.697356+0000","last_unstale":"2024-04-06T13:48:36.203008+0000","last_undegraded":"2024-04-06T13:48:36.203008+0000","last_fullsized":"2024-04-06T13:48:36.203008+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:44.773715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204757+0000","last_change":"2024-04-06T13:48:36.214926+0000","last_active":"2024-04-06T13:48:38.204757+0000","last_peered":"2024-04-06T13:48:38.204757+0000","last_clean":"2024-04-06T13:48:38.204757+0000","last_became_active":"2024-04-06T13:48:36.213877+0000","last_became_peered":"2024-04-06T13:48:36.213877+0000","last_unstale":"2024-04-06T13:48:38.204757+0000","last_undegraded":"2024-04-06T13:48:38.204757+0000","last_fullsized":"2024-04-06T13:48:38.204757+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:14:55.492477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204092+0000","last_change":"2024-04-06T13:48:35.805462+0000","last_active":"2024-04-06T13:48:38.204092+0000","last_peered":"2024-04-06T13:48:38.204092+0000","last_clean":"2024-04-06T13:48:38.204092+0000","last_became_active":"2024-04-06T13:48:35.805205+0000","last_became_peered":"2024-04-06T13:48:35.805205+0000","last_unstale":"2024-04-06T13:48:38.204092+0000","last_undegraded":"2024-04-06T13:48:38.204092+0000","last_fullsized":"2024-04-06T13:48:38.204092+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:40.396325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.202358+0000","last_change":"2024-04-06T13:48:19.705129+0000","last_active":"2024-04-06T13:48:36.202358+0000","last_peered":"2024-04-06T13:48:36.202358+0000","last_clean":"2024-04-06T13:48:36.202358+0000","last_became_active":"2024-04-06T13:48:19.704722+0000","last_became_peered":"2024-04-06T13:48:19.704722+0000","last_unstale":"2024-04-06T13:48:36.202358+0000","last_undegraded":"2024-04-06T13:48:36.202358+0000","last_fullsized":"2024-04-06T13:48:36.202358+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:18.449707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212458+0000","last_change":"2024-04-06T13:48:36.225691+0000","last_active":"2024-04-06T13:48:37.212458+0000","last_peered":"2024-04-06T13:48:37.212458+0000","last_clean":"2024-04-06T13:48:37.212458+0000","last_became_active":"2024-04-06T13:48:36.225409+0000","last_became_peered":"2024-04-06T13:48:36.225409+0000","last_unstale":"2024-04-06T13:48:37.212458+0000","last_undegraded":"2024-04-06T13:48:37.212458+0000","last_fullsized":"2024-04-06T13:48:37.212458+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:36:41.699885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204692+0000","last_change":"2024-04-06T13:48:19.696446+0000","last_active":"2024-04-06T13:48:36.204692+0000","last_peered":"2024-04-06T13:48:36.204692+0000","last_clean":"2024-04-06T13:48:36.204692+0000","last_became_active":"2024-04-06T13:48:19.696090+0000","last_became_peered":"2024-04-06T13:48:19.696090+0000","last_unstale":"2024-04-06T13:48:36.204692+0000","last_undegraded":"2024-04-06T13:48:36.204692+0000","last_fullsized":"2024-04-06T13:48:36.204692+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:35.461214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"247'4","reported_seq":19,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204118+0000","last_change":"2024-04-06T13:48:19.281764+0000","last_active":"2024-04-06T13:48:36.204118+0000","last_peered":"2024-04-06T13:48:36.204118+0000","last_clean":"2024-04-06T13:48:36.204118+0000","last_became_active":"2024-04-06T13:48:19.281501+0000","last_became_peered":"2024-04-06T13:48:19.281501+0000","last_unstale":"2024-04-06T13:48:36.204118+0000","last_undegraded":"2024-04-06T13:48:36.204118+0000","last_fullsized":"2024-04-06T13:48:36.204118+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:54:56.369118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203369+0000","last_change":"2024-04-06T13:48:36.226891+0000","last_active":"2024-04-06T13:48:38.203369+0000","last_peered":"2024-04-06T13:48:38.203369+0000","last_clean":"2024-04-06T13:48:38.203369+0000","last_became_active":"2024-04-06T13:48:36.220372+0000","last_became_peered":"2024-04-06T13:48:36.220372+0000","last_unstale":"2024-04-06T13:48:38.203369+0000","last_undegraded":"2024-04-06T13:48:38.203369+0000","last_fullsized":"2024-04-06T13:48:38.203369+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:19:31.844777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206141+0000","last_change":"2024-04-06T13:48:19.699483+0000","last_active":"2024-04-06T13:48:36.206141+0000","last_peered":"2024-04-06T13:48:36.206141+0000","last_clean":"2024-04-06T13:48:36.206141+0000","last_became_active":"2024-04-06T13:48:19.699209+0000","last_became_peered":"2024-04-06T13:48:19.699209+0000","last_unstale":"2024-04-06T13:48:36.206141+0000","last_undegraded":"2024-04-06T13:48:36.206141+0000","last_fullsized":"2024-04-06T13:48:36.206141+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:42:12.990594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.214449+0000","last_change":"2024-04-06T13:48:19.280859+0000","last_active":"2024-04-06T13:48:36.214449+0000","last_peered":"2024-04-06T13:48:36.214449+0000","last_clean":"2024-04-06T13:48:36.214449+0000","last_became_active":"2024-04-06T13:48:19.280512+0000","last_became_peered":"2024-04-06T13:48:19.280512+0000","last_unstale":"2024-04-06T13:48:36.214449+0000","last_undegraded":"2024-04-06T13:48:36.214449+0000","last_fullsized":"2024-04-06T13:48:36.214449+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:43.765754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204376+0000","last_change":"2024-04-06T13:48:36.217497+0000","last_active":"2024-04-06T13:48:38.204376+0000","last_peered":"2024-04-06T13:48:38.204376+0000","last_clean":"2024-04-06T13:48:38.204376+0000","last_became_active":"2024-04-06T13:48:36.217261+0000","last_became_peered":"2024-04-06T13:48:36.217261+0000","last_unstale":"2024-04-06T13:48:38.204376+0000","last_undegraded":"2024-04-06T13:48:38.204376+0000","last_fullsized":"2024-04-06T13:48:38.204376+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:32:13.503864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.202674+0000","last_change":"2024-04-06T13:48:19.687508+0000","last_active":"2024-04-06T13:48:36.202674+0000","last_peered":"2024-04-06T13:48:36.202674+0000","last_clean":"2024-04-06T13:48:36.202674+0000","last_became_active":"2024-04-06T13:48:19.687246+0000","last_became_peered":"2024-04-06T13:48:19.687246+0000","last_unstale":"2024-04-06T13:48:36.202674+0000","last_undegraded":"2024-04-06T13:48:36.202674+0000","last_fullsized":"2024-04-06T13:48:36.202674+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:15:08.111319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206548+0000","last_change":"2024-04-06T13:48:19.280934+0000","last_active":"2024-04-06T13:48:36.206548+0000","last_peered":"2024-04-06T13:48:36.206548+0000","last_clean":"2024-04-06T13:48:36.206548+0000","last_became_active":"2024-04-06T13:48:19.280677+0000","last_became_peered":"2024-04-06T13:48:19.280677+0000","last_unstale":"2024-04-06T13:48:36.206548+0000","last_undegraded":"2024-04-06T13:48:36.206548+0000","last_fullsized":"2024-04-06T13:48:36.206548+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:30:23.776677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204905+0000","last_change":"2024-04-06T13:48:36.225949+0000","last_active":"2024-04-06T13:48:38.204905+0000","last_peered":"2024-04-06T13:48:38.204905+0000","last_clean":"2024-04-06T13:48:38.204905+0000","last_became_active":"2024-04-06T13:48:36.225375+0000","last_became_peered":"2024-04-06T13:48:36.225375+0000","last_unstale":"2024-04-06T13:48:38.204905+0000","last_undegraded":"2024-04-06T13:48:38.204905+0000","last_fullsized":"2024-04-06T13:48:38.204905+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:17:18.829437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206018+0000","last_change":"2024-04-06T13:48:19.695081+0000","last_active":"2024-04-06T13:48:36.206018+0000","last_peered":"2024-04-06T13:48:36.206018+0000","last_clean":"2024-04-06T13:48:36.206018+0000","last_became_active":"2024-04-06T13:48:19.694838+0000","last_became_peered":"2024-04-06T13:48:19.694838+0000","last_unstale":"2024-04-06T13:48:36.206018+0000","last_undegraded":"2024-04-06T13:48:36.206018+0000","last_fullsized":"2024-04-06T13:48:36.206018+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:37:44.732983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206452+0000","last_change":"2024-04-06T13:48:19.282906+0000","last_active":"2024-04-06T13:48:36.206452+0000","last_peered":"2024-04-06T13:48:36.206452+0000","last_clean":"2024-04-06T13:48:36.206452+0000","last_became_active":"2024-04-06T13:48:19.282628+0000","last_became_peered":"2024-04-06T13:48:19.282628+0000","last_unstale":"2024-04-06T13:48:36.206452+0000","last_undegraded":"2024-04-06T13:48:36.206452+0000","last_fullsized":"2024-04-06T13:48:36.206452+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:18.513042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204073+0000","last_change":"2024-04-06T13:48:36.221507+0000","last_active":"2024-04-06T13:48:38.204073+0000","last_peered":"2024-04-06T13:48:38.204073+0000","last_clean":"2024-04-06T13:48:38.204073+0000","last_became_active":"2024-04-06T13:48:36.221248+0000","last_became_peered":"2024-04-06T13:48:36.221248+0000","last_unstale":"2024-04-06T13:48:38.204073+0000","last_undegraded":"2024-04-06T13:48:38.204073+0000","last_fullsized":"2024-04-06T13:48:38.204073+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:06:46.118211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.202326+0000","last_change":"2024-04-06T13:48:19.694881+0000","last_active":"2024-04-06T13:48:36.202326+0000","last_peered":"2024-04-06T13:48:36.202326+0000","last_clean":"2024-04-06T13:48:36.202326+0000","last_became_active":"2024-04-06T13:48:19.693362+0000","last_became_peered":"2024-04-06T13:48:19.693362+0000","last_unstale":"2024-04-06T13:48:36.202326+0000","last_undegraded":"2024-04-06T13:48:36.202326+0000","last_fullsized":"2024-04-06T13:48:36.202326+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:36:47.917157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.207654+0000","last_change":"2024-04-06T13:48:19.274234+0000","last_active":"2024-04-06T13:48:36.207654+0000","last_peered":"2024-04-06T13:48:36.207654+0000","last_clean":"2024-04-06T13:48:36.207654+0000","last_became_active":"2024-04-06T13:48:19.273998+0000","last_became_peered":"2024-04-06T13:48:19.273998+0000","last_unstale":"2024-04-06T13:48:36.207654+0000","last_undegraded":"2024-04-06T13:48:36.207654+0000","last_fullsized":"2024-04-06T13:48:36.207654+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:49:23.791293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205696+0000","last_change":"2024-04-06T13:48:36.229400+0000","last_active":"2024-04-06T13:48:38.205696+0000","last_peered":"2024-04-06T13:48:38.205696+0000","last_clean":"2024-04-06T13:48:38.205696+0000","last_became_active":"2024-04-06T13:48:36.229163+0000","last_became_peered":"2024-04-06T13:48:36.229163+0000","last_unstale":"2024-04-06T13:48:38.205696+0000","last_undegraded":"2024-04-06T13:48:38.205696+0000","last_fullsized":"2024-04-06T13:48:38.205696+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:01:21.610743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206538+0000","last_change":"2024-04-06T13:48:19.694934+0000","last_active":"2024-04-06T13:48:36.206538+0000","last_peered":"2024-04-06T13:48:36.206538+0000","last_clean":"2024-04-06T13:48:36.206538+0000","last_became_active":"2024-04-06T13:48:19.692682+0000","last_became_peered":"2024-04-06T13:48:19.692682+0000","last_unstale":"2024-04-06T13:48:36.206538+0000","last_undegraded":"2024-04-06T13:48:36.206538+0000","last_fullsized":"2024-04-06T13:48:36.206538+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:40:43.027987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.205235+0000","last_change":"2024-04-06T13:48:19.279719+0000","last_active":"2024-04-06T13:48:36.205235+0000","last_peered":"2024-04-06T13:48:36.205235+0000","last_clean":"2024-04-06T13:48:36.205235+0000","last_became_active":"2024-04-06T13:48:19.279404+0000","last_became_peered":"2024-04-06T13:48:19.279404+0000","last_unstale":"2024-04-06T13:48:36.205235+0000","last_undegraded":"2024-04-06T13:48:36.205235+0000","last_fullsized":"2024-04-06T13:48:36.205235+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:42:35.927858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205956+0000","last_change":"2024-04-06T13:48:36.230010+0000","last_active":"2024-04-06T13:48:38.205956+0000","last_peered":"2024-04-06T13:48:38.205956+0000","last_clean":"2024-04-06T13:48:38.205956+0000","last_became_active":"2024-04-06T13:48:36.229745+0000","last_became_peered":"2024-04-06T13:48:36.229745+0000","last_unstale":"2024-04-06T13:48:38.205956+0000","last_undegraded":"2024-04-06T13:48:38.205956+0000","last_fullsized":"2024-04-06T13:48:38.205956+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:40:41.412534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212903+0000","last_change":"2024-04-06T13:48:19.702348+0000","last_active":"2024-04-06T13:48:36.212903+0000","last_peered":"2024-04-06T13:48:36.212903+0000","last_clean":"2024-04-06T13:48:36.212903+0000","last_became_active":"2024-04-06T13:48:19.701891+0000","last_became_peered":"2024-04-06T13:48:19.701891+0000","last_unstale":"2024-04-06T13:48:36.212903+0000","last_undegraded":"2024-04-06T13:48:36.212903+0000","last_fullsized":"2024-04-06T13:48:36.212903+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:52:52.810060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.202536+0000","last_change":"2024-04-06T13:48:19.281924+0000","last_active":"2024-04-06T13:48:36.202536+0000","last_peered":"2024-04-06T13:48:36.202536+0000","last_clean":"2024-04-06T13:48:36.202536+0000","last_became_active":"2024-04-06T13:48:19.281609+0000","last_became_peered":"2024-04-06T13:48:19.281609+0000","last_unstale":"2024-04-06T13:48:36.202536+0000","last_undegraded":"2024-04-06T13:48:36.202536+0000","last_fullsized":"2024-04-06T13:48:36.202536+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:02:13.092175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205733+0000","last_change":"2024-04-06T13:48:36.229390+0000","last_active":"2024-04-06T13:48:38.205733+0000","last_peered":"2024-04-06T13:48:38.205733+0000","last_clean":"2024-04-06T13:48:38.205733+0000","last_became_active":"2024-04-06T13:48:36.229111+0000","last_became_peered":"2024-04-06T13:48:36.229111+0000","last_unstale":"2024-04-06T13:48:38.205733+0000","last_undegraded":"2024-04-06T13:48:38.205733+0000","last_fullsized":"2024-04-06T13:48:38.205733+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:38:22.774628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.201909+0000","last_change":"2024-04-06T13:48:19.694595+0000","last_active":"2024-04-06T13:48:36.201909+0000","last_peered":"2024-04-06T13:48:36.201909+0000","last_clean":"2024-04-06T13:48:36.201909+0000","last_became_active":"2024-04-06T13:48:19.692548+0000","last_became_peered":"2024-04-06T13:48:19.692548+0000","last_unstale":"2024-04-06T13:48:36.201909+0000","last_undegraded":"2024-04-06T13:48:36.201909+0000","last_fullsized":"2024-04-06T13:48:36.201909+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:39:01.587642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.208178+0000","last_change":"2024-04-06T13:48:19.273745+0000","last_active":"2024-04-06T13:48:36.208178+0000","last_peered":"2024-04-06T13:48:36.208178+0000","last_clean":"2024-04-06T13:48:36.208178+0000","last_became_active":"2024-04-06T13:48:19.273478+0000","last_became_peered":"2024-04-06T13:48:19.273478+0000","last_unstale":"2024-04-06T13:48:36.208178+0000","last_undegraded":"2024-04-06T13:48:36.208178+0000","last_fullsized":"2024-04-06T13:48:36.208178+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:00:55.130902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203553+0000","last_change":"2024-04-06T13:48:36.220579+0000","last_active":"2024-04-06T13:48:38.203553+0000","last_peered":"2024-04-06T13:48:38.203553+0000","last_clean":"2024-04-06T13:48:38.203553+0000","last_became_active":"2024-04-06T13:48:36.220002+0000","last_became_peered":"2024-04-06T13:48:36.220002+0000","last_unstale":"2024-04-06T13:48:38.203553+0000","last_undegraded":"2024-04-06T13:48:38.203553+0000","last_fullsized":"2024-04-06T13:48:38.203553+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:34.972744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212322+0000","last_change":"2024-04-06T13:48:19.688682+0000","last_active":"2024-04-06T13:48:36.212322+0000","last_peered":"2024-04-06T13:48:36.212322+0000","last_clean":"2024-04-06T13:48:36.212322+0000","last_became_active":"2024-04-06T13:48:19.688397+0000","last_became_peered":"2024-04-06T13:48:19.688397+0000","last_unstale":"2024-04-06T13:48:36.212322+0000","last_undegraded":"2024-04-06T13:48:36.212322+0000","last_fullsized":"2024-04-06T13:48:36.212322+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:39:01.587642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206720+0000","last_change":"2024-04-06T13:48:19.278056+0000","last_active":"2024-04-06T13:48:36.206720+0000","last_peered":"2024-04-06T13:48:36.206720+0000","last_clean":"2024-04-06T13:48:36.206720+0000","last_became_active":"2024-04-06T13:48:19.277796+0000","last_became_peered":"2024-04-06T13:48:19.277796+0000","last_unstale":"2024-04-06T13:48:36.206720+0000","last_undegraded":"2024-04-06T13:48:36.206720+0000","last_fullsized":"2024-04-06T13:48:36.206720+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:31:43.320734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205161+0000","last_change":"2024-04-06T13:48:36.223212+0000","last_active":"2024-04-06T13:48:38.205161+0000","last_peered":"2024-04-06T13:48:38.205161+0000","last_clean":"2024-04-06T13:48:38.205161+0000","last_became_active":"2024-04-06T13:48:36.222954+0000","last_became_peered":"2024-04-06T13:48:36.222954+0000","last_unstale":"2024-04-06T13:48:38.205161+0000","last_undegraded":"2024-04-06T13:48:38.205161+0000","last_fullsized":"2024-04-06T13:48:38.205161+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:34:38.126512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.205572+0000","last_change":"2024-04-06T13:48:19.700325+0000","last_active":"2024-04-06T13:48:36.205572+0000","last_peered":"2024-04-06T13:48:36.205572+0000","last_clean":"2024-04-06T13:48:36.205572+0000","last_became_active":"2024-04-06T13:48:19.698751+0000","last_became_peered":"2024-04-06T13:48:19.698751+0000","last_unstale":"2024-04-06T13:48:36.205572+0000","last_undegraded":"2024-04-06T13:48:36.205572+0000","last_fullsized":"2024-04-06T13:48:36.205572+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:16.165915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"247'3","reported_seq":18,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213341+0000","last_change":"2024-04-06T13:48:19.283170+0000","last_active":"2024-04-06T13:48:36.213341+0000","last_peered":"2024-04-06T13:48:36.213341+0000","last_clean":"2024-04-06T13:48:36.213341+0000","last_became_active":"2024-04-06T13:48:19.282896+0000","last_became_peered":"2024-04-06T13:48:19.282896+0000","last_unstale":"2024-04-06T13:48:36.213341+0000","last_undegraded":"2024-04-06T13:48:36.213341+0000","last_fullsized":"2024-04-06T13:48:36.213341+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:06:05.488014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204391+0000","last_change":"2024-04-06T13:48:36.225295+0000","last_active":"2024-04-06T13:48:38.204391+0000","last_peered":"2024-04-06T13:48:38.204391+0000","last_clean":"2024-04-06T13:48:38.204391+0000","last_became_active":"2024-04-06T13:48:36.220340+0000","last_became_peered":"2024-04-06T13:48:36.220340+0000","last_unstale":"2024-04-06T13:48:38.204391+0000","last_undegraded":"2024-04-06T13:48:38.204391+0000","last_fullsized":"2024-04-06T13:48:38.204391+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:14:04.534874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.200896+0000","last_change":"2024-04-06T13:48:19.692164+0000","last_active":"2024-04-06T13:48:36.200896+0000","last_peered":"2024-04-06T13:48:36.200896+0000","last_clean":"2024-04-06T13:48:36.200896+0000","last_became_active":"2024-04-06T13:48:19.691910+0000","last_became_peered":"2024-04-06T13:48:19.691910+0000","last_unstale":"2024-04-06T13:48:36.200896+0000","last_undegraded":"2024-04-06T13:48:36.200896+0000","last_fullsized":"2024-04-06T13:48:36.200896+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:19:27.875509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"248'8","reported_seq":23,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.203561+0000","last_change":"2024-04-06T13:48:19.281276+0000","last_active":"2024-04-06T13:48:36.203561+0000","last_peered":"2024-04-06T13:48:36.203561+0000","last_clean":"2024-04-06T13:48:36.203561+0000","last_became_active":"2024-04-06T13:48:19.280081+0000","last_became_peered":"2024-04-06T13:48:19.280081+0000","last_unstale":"2024-04-06T13:48:36.203561+0000","last_undegraded":"2024-04-06T13:48:36.203561+0000","last_fullsized":"2024-04-06T13:48:36.203561+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:56:55.759595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2813,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.208268+0000","last_change":"2024-04-06T13:48:36.220050+0000","last_active":"2024-04-06T13:48:37.208268+0000","last_peered":"2024-04-06T13:48:37.208268+0000","last_clean":"2024-04-06T13:48:37.208268+0000","last_became_active":"2024-04-06T13:48:36.219803+0000","last_became_peered":"2024-04-06T13:48:36.219803+0000","last_unstale":"2024-04-06T13:48:37.208268+0000","last_undegraded":"2024-04-06T13:48:37.208268+0000","last_fullsized":"2024-04-06T13:48:37.208268+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:42:29.513631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.203704+0000","last_change":"2024-04-06T13:48:19.693858+0000","last_active":"2024-04-06T13:48:36.203704+0000","last_peered":"2024-04-06T13:48:36.203704+0000","last_clean":"2024-04-06T13:48:36.203704+0000","last_became_active":"2024-04-06T13:48:19.693515+0000","last_became_peered":"2024-04-06T13:48:19.693515+0000","last_unstale":"2024-04-06T13:48:36.203704+0000","last_undegraded":"2024-04-06T13:48:36.203704+0000","last_fullsized":"2024-04-06T13:48:36.203704+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:10:26.852793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206543+0000","last_change":"2024-04-06T13:48:19.283816+0000","last_active":"2024-04-06T13:48:36.206543+0000","last_peered":"2024-04-06T13:48:36.206543+0000","last_clean":"2024-04-06T13:48:36.206543+0000","last_became_active":"2024-04-06T13:48:19.283543+0000","last_became_peered":"2024-04-06T13:48:19.283543+0000","last_unstale":"2024-04-06T13:48:36.206543+0000","last_undegraded":"2024-04-06T13:48:36.206543+0000","last_fullsized":"2024-04-06T13:48:36.206543+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:31:24.701348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204733+0000","last_change":"2024-04-06T13:48:36.211366+0000","last_active":"2024-04-06T13:48:38.204733+0000","last_peered":"2024-04-06T13:48:38.204733+0000","last_clean":"2024-04-06T13:48:38.204733+0000","last_became_active":"2024-04-06T13:48:36.211180+0000","last_became_peered":"2024-04-06T13:48:36.211180+0000","last_unstale":"2024-04-06T13:48:38.204733+0000","last_undegraded":"2024-04-06T13:48:38.204733+0000","last_fullsized":"2024-04-06T13:48:38.204733+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:04:27.070776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.202245+0000","last_change":"2024-04-06T13:48:19.694858+0000","last_active":"2024-04-06T13:48:36.202245+0000","last_peered":"2024-04-06T13:48:36.202245+0000","last_clean":"2024-04-06T13:48:36.202245+0000","last_became_active":"2024-04-06T13:48:19.693049+0000","last_became_peered":"2024-04-06T13:48:19.693049+0000","last_unstale":"2024-04-06T13:48:36.202245+0000","last_undegraded":"2024-04-06T13:48:36.202245+0000","last_fullsized":"2024-04-06T13:48:36.202245+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:19:00.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213239+0000","last_change":"2024-04-06T13:48:19.282855+0000","last_active":"2024-04-06T13:48:36.213239+0000","last_peered":"2024-04-06T13:48:36.213239+0000","last_clean":"2024-04-06T13:48:36.213239+0000","last_became_active":"2024-04-06T13:48:19.282570+0000","last_became_peered":"2024-04-06T13:48:19.282570+0000","last_unstale":"2024-04-06T13:48:36.213239+0000","last_undegraded":"2024-04-06T13:48:36.213239+0000","last_fullsized":"2024-04-06T13:48:36.213239+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:35:17.934513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212985+0000","last_change":"2024-04-06T13:48:36.227966+0000","last_active":"2024-04-06T13:48:37.212985+0000","last_peered":"2024-04-06T13:48:37.212985+0000","last_clean":"2024-04-06T13:48:37.212985+0000","last_became_active":"2024-04-06T13:48:36.227294+0000","last_became_peered":"2024-04-06T13:48:36.227294+0000","last_unstale":"2024-04-06T13:48:37.212985+0000","last_undegraded":"2024-04-06T13:48:37.212985+0000","last_fullsized":"2024-04-06T13:48:37.212985+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:23:55.868945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.208874+0000","last_change":"2024-04-06T13:48:19.693208+0000","last_active":"2024-04-06T13:48:36.208874+0000","last_peered":"2024-04-06T13:48:36.208874+0000","last_clean":"2024-04-06T13:48:36.208874+0000","last_became_active":"2024-04-06T13:48:19.692939+0000","last_became_peered":"2024-04-06T13:48:19.692939+0000","last_unstale":"2024-04-06T13:48:36.208874+0000","last_undegraded":"2024-04-06T13:48:36.208874+0000","last_fullsized":"2024-04-06T13:48:36.208874+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:46:07.210444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212994+0000","last_change":"2024-04-06T13:48:19.283720+0000","last_active":"2024-04-06T13:48:36.212994+0000","last_peered":"2024-04-06T13:48:36.212994+0000","last_clean":"2024-04-06T13:48:36.212994+0000","last_became_active":"2024-04-06T13:48:19.283453+0000","last_became_peered":"2024-04-06T13:48:19.283453+0000","last_unstale":"2024-04-06T13:48:36.212994+0000","last_undegraded":"2024-04-06T13:48:36.212994+0000","last_fullsized":"2024-04-06T13:48:36.212994+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:02:23.692226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204247+0000","last_change":"2024-04-06T13:48:36.222844+0000","last_active":"2024-04-06T13:48:38.204247+0000","last_peered":"2024-04-06T13:48:38.204247+0000","last_clean":"2024-04-06T13:48:38.204247+0000","last_became_active":"2024-04-06T13:48:36.222573+0000","last_became_peered":"2024-04-06T13:48:36.222573+0000","last_unstale":"2024-04-06T13:48:38.204247+0000","last_undegraded":"2024-04-06T13:48:38.204247+0000","last_fullsized":"2024-04-06T13:48:38.204247+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:46:23.733481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212817+0000","last_change":"2024-04-06T13:48:19.700890+0000","last_active":"2024-04-06T13:48:36.212817+0000","last_peered":"2024-04-06T13:48:36.212817+0000","last_clean":"2024-04-06T13:48:36.212817+0000","last_became_active":"2024-04-06T13:48:19.700513+0000","last_became_peered":"2024-04-06T13:48:19.700513+0000","last_unstale":"2024-04-06T13:48:36.212817+0000","last_undegraded":"2024-04-06T13:48:36.212817+0000","last_fullsized":"2024-04-06T13:48:36.212817+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:35:45.830191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"247'8","reported_seq":23,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204608+0000","last_change":"2024-04-06T13:48:19.274645+0000","last_active":"2024-04-06T13:48:36.204608+0000","last_peered":"2024-04-06T13:48:36.204608+0000","last_clean":"2024-04-06T13:48:36.204608+0000","last_became_active":"2024-04-06T13:48:19.274308+0000","last_became_peered":"2024-04-06T13:48:19.274308+0000","last_unstale":"2024-04-06T13:48:36.204608+0000","last_undegraded":"2024-04-06T13:48:36.204608+0000","last_fullsized":"2024-04-06T13:48:36.204608+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:31:15.493970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205094+0000","last_change":"2024-04-06T13:48:36.214621+0000","last_active":"2024-04-06T13:48:38.205094+0000","last_peered":"2024-04-06T13:48:38.205094+0000","last_clean":"2024-04-06T13:48:38.205094+0000","last_became_active":"2024-04-06T13:48:36.213123+0000","last_became_peered":"2024-04-06T13:48:36.213123+0000","last_unstale":"2024-04-06T13:48:38.205094+0000","last_undegraded":"2024-04-06T13:48:38.205094+0000","last_fullsized":"2024-04-06T13:48:38.205094+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:39:13.811002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213492+0000","last_change":"2024-04-06T13:48:19.700124+0000","last_active":"2024-04-06T13:48:36.213492+0000","last_peered":"2024-04-06T13:48:36.213492+0000","last_clean":"2024-04-06T13:48:36.213492+0000","last_became_active":"2024-04-06T13:48:19.699857+0000","last_became_peered":"2024-04-06T13:48:19.699857+0000","last_unstale":"2024-04-06T13:48:36.213492+0000","last_undegraded":"2024-04-06T13:48:36.213492+0000","last_fullsized":"2024-04-06T13:48:36.213492+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:44.773715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204041+0000","last_change":"2024-04-06T13:48:19.284301+0000","last_active":"2024-04-06T13:48:36.204041+0000","last_peered":"2024-04-06T13:48:36.204041+0000","last_clean":"2024-04-06T13:48:36.204041+0000","last_became_active":"2024-04-06T13:48:19.284062+0000","last_became_peered":"2024-04-06T13:48:19.284062+0000","last_unstale":"2024-04-06T13:48:36.204041+0000","last_undegraded":"2024-04-06T13:48:36.204041+0000","last_fullsized":"2024-04-06T13:48:36.204041+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:49.041735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212045+0000","last_change":"2024-04-06T13:48:36.211538+0000","last_active":"2024-04-06T13:48:37.212045+0000","last_peered":"2024-04-06T13:48:37.212045+0000","last_clean":"2024-04-06T13:48:37.212045+0000","last_became_active":"2024-04-06T13:48:36.210725+0000","last_became_peered":"2024-04-06T13:48:36.210725+0000","last_unstale":"2024-04-06T13:48:37.212045+0000","last_undegraded":"2024-04-06T13:48:37.212045+0000","last_fullsized":"2024-04-06T13:48:37.212045+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:00:05.559228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212880+0000","last_change":"2024-04-06T13:48:19.702274+0000","last_active":"2024-04-06T13:48:36.212880+0000","last_peered":"2024-04-06T13:48:36.212880+0000","last_clean":"2024-04-06T13:48:36.212880+0000","last_became_active":"2024-04-06T13:48:19.701964+0000","last_became_peered":"2024-04-06T13:48:19.701964+0000","last_unstale":"2024-04-06T13:48:36.212880+0000","last_undegraded":"2024-04-06T13:48:36.212880+0000","last_fullsized":"2024-04-06T13:48:36.212880+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:45.240549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204097+0000","last_change":"2024-04-06T13:48:19.275494+0000","last_active":"2024-04-06T13:48:36.204097+0000","last_peered":"2024-04-06T13:48:36.204097+0000","last_clean":"2024-04-06T13:48:36.204097+0000","last_became_active":"2024-04-06T13:48:19.275194+0000","last_became_peered":"2024-04-06T13:48:19.275194+0000","last_unstale":"2024-04-06T13:48:36.204097+0000","last_undegraded":"2024-04-06T13:48:36.204097+0000","last_fullsized":"2024-04-06T13:48:36.204097+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:11:29.081483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204870+0000","last_change":"2024-04-06T13:48:36.216455+0000","last_active":"2024-04-06T13:48:38.204870+0000","last_peered":"2024-04-06T13:48:38.204870+0000","last_clean":"2024-04-06T13:48:38.204870+0000","last_became_active":"2024-04-06T13:48:36.215926+0000","last_became_peered":"2024-04-06T13:48:36.215926+0000","last_unstale":"2024-04-06T13:48:38.204870+0000","last_undegraded":"2024-04-06T13:48:38.204870+0000","last_fullsized":"2024-04-06T13:48:38.204870+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:32.688952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.203455+0000","last_change":"2024-04-06T13:48:19.702505+0000","last_active":"2024-04-06T13:48:36.203455+0000","last_peered":"2024-04-06T13:48:36.203455+0000","last_clean":"2024-04-06T13:48:36.203455+0000","last_became_active":"2024-04-06T13:48:19.702245+0000","last_became_peered":"2024-04-06T13:48:19.702245+0000","last_unstale":"2024-04-06T13:48:36.203455+0000","last_undegraded":"2024-04-06T13:48:36.203455+0000","last_fullsized":"2024-04-06T13:48:36.203455+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:45.672315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.214556+0000","last_change":"2024-04-06T13:48:19.281905+0000","last_active":"2024-04-06T13:48:36.214556+0000","last_peered":"2024-04-06T13:48:36.214556+0000","last_clean":"2024-04-06T13:48:36.214556+0000","last_became_active":"2024-04-06T13:48:19.281052+0000","last_became_peered":"2024-04-06T13:48:19.281052+0000","last_unstale":"2024-04-06T13:48:36.214556+0000","last_undegraded":"2024-04-06T13:48:36.214556+0000","last_fullsized":"2024-04-06T13:48:36.214556+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:48.100692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203933+0000","last_change":"2024-04-06T13:48:36.226131+0000","last_active":"2024-04-06T13:48:38.203933+0000","last_peered":"2024-04-06T13:48:38.203933+0000","last_clean":"2024-04-06T13:48:38.203933+0000","last_became_active":"2024-04-06T13:48:36.225885+0000","last_became_peered":"2024-04-06T13:48:36.225885+0000","last_unstale":"2024-04-06T13:48:38.203933+0000","last_undegraded":"2024-04-06T13:48:38.203933+0000","last_fullsized":"2024-04-06T13:48:38.203933+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:16:24.546765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.205615+0000","last_change":"2024-04-06T13:48:19.700428+0000","last_active":"2024-04-06T13:48:36.205615+0000","last_peered":"2024-04-06T13:48:36.205615+0000","last_clean":"2024-04-06T13:48:36.205615+0000","last_became_active":"2024-04-06T13:48:19.699896+0000","last_became_peered":"2024-04-06T13:48:19.699896+0000","last_unstale":"2024-04-06T13:48:36.205615+0000","last_undegraded":"2024-04-06T13:48:36.205615+0000","last_fullsized":"2024-04-06T13:48:36.205615+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:00:56.138863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206015+0000","last_change":"2024-04-06T13:48:19.286358+0000","last_active":"2024-04-06T13:48:36.206015+0000","last_peered":"2024-04-06T13:48:36.206015+0000","last_clean":"2024-04-06T13:48:36.206015+0000","last_became_active":"2024-04-06T13:48:19.286136+0000","last_became_peered":"2024-04-06T13:48:19.286136+0000","last_unstale":"2024-04-06T13:48:36.206015+0000","last_undegraded":"2024-04-06T13:48:36.206015+0000","last_fullsized":"2024-04-06T13:48:36.206015+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:55:10.767276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205017+0000","last_change":"2024-04-06T13:48:36.220625+0000","last_active":"2024-04-06T13:48:38.205017+0000","last_peered":"2024-04-06T13:48:38.205017+0000","last_clean":"2024-04-06T13:48:38.205017+0000","last_became_active":"2024-04-06T13:48:36.220345+0000","last_became_peered":"2024-04-06T13:48:36.220345+0000","last_unstale":"2024-04-06T13:48:38.205017+0000","last_undegraded":"2024-04-06T13:48:38.205017+0000","last_fullsized":"2024-04-06T13:48:38.205017+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:01:12.661900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206344+0000","last_change":"2024-04-06T13:48:19.697815+0000","last_active":"2024-04-06T13:48:36.206344+0000","last_peered":"2024-04-06T13:48:36.206344+0000","last_clean":"2024-04-06T13:48:36.206344+0000","last_became_active":"2024-04-06T13:48:19.695354+0000","last_became_peered":"2024-04-06T13:48:19.695354+0000","last_unstale":"2024-04-06T13:48:36.206344+0000","last_undegraded":"2024-04-06T13:48:36.206344+0000","last_fullsized":"2024-04-06T13:48:36.206344+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:38:12.538926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.205541+0000","last_change":"2024-04-06T13:48:19.278849+0000","last_active":"2024-04-06T13:48:36.205541+0000","last_peered":"2024-04-06T13:48:36.205541+0000","last_clean":"2024-04-06T13:48:36.205541+0000","last_became_active":"2024-04-06T13:48:19.278604+0000","last_became_peered":"2024-04-06T13:48:19.278604+0000","last_unstale":"2024-04-06T13:48:36.205541+0000","last_undegraded":"2024-04-06T13:48:36.205541+0000","last_fullsized":"2024-04-06T13:48:36.205541+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:19:26.867548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.213079+0000","last_change":"2024-04-06T13:48:36.223518+0000","last_active":"2024-04-06T13:48:37.213079+0000","last_peered":"2024-04-06T13:48:37.213079+0000","last_clean":"2024-04-06T13:48:37.213079+0000","last_became_active":"2024-04-06T13:48:36.223244+0000","last_became_peered":"2024-04-06T13:48:36.223244+0000","last_unstale":"2024-04-06T13:48:37.213079+0000","last_undegraded":"2024-04-06T13:48:37.213079+0000","last_fullsized":"2024-04-06T13:48:37.213079+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:41:38.433651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204402+0000","last_change":"2024-04-06T13:48:19.696316+0000","last_active":"2024-04-06T13:48:36.204402+0000","last_peered":"2024-04-06T13:48:36.204402+0000","last_clean":"2024-04-06T13:48:36.204402+0000","last_became_active":"2024-04-06T13:48:19.696025+0000","last_became_peered":"2024-04-06T13:48:19.696025+0000","last_unstale":"2024-04-06T13:48:36.204402+0000","last_undegraded":"2024-04-06T13:48:36.204402+0000","last_fullsized":"2024-04-06T13:48:36.204402+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:33:27.603807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212162+0000","last_change":"2024-04-06T13:48:19.282165+0000","last_active":"2024-04-06T13:48:36.212162+0000","last_peered":"2024-04-06T13:48:36.212162+0000","last_clean":"2024-04-06T13:48:36.212162+0000","last_became_active":"2024-04-06T13:48:19.281892+0000","last_became_peered":"2024-04-06T13:48:19.281892+0000","last_unstale":"2024-04-06T13:48:36.212162+0000","last_undegraded":"2024-04-06T13:48:36.212162+0000","last_fullsized":"2024-04-06T13:48:36.212162+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:10:25.844832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212938+0000","last_change":"2024-04-06T13:48:36.228092+0000","last_active":"2024-04-06T13:48:37.212938+0000","last_peered":"2024-04-06T13:48:37.212938+0000","last_clean":"2024-04-06T13:48:37.212938+0000","last_became_active":"2024-04-06T13:48:36.227645+0000","last_became_peered":"2024-04-06T13:48:36.227645+0000","last_unstale":"2024-04-06T13:48:37.212938+0000","last_undegraded":"2024-04-06T13:48:37.212938+0000","last_fullsized":"2024-04-06T13:48:37.212938+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:08.874119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.210661+0000","last_change":"2024-04-06T13:48:19.699890+0000","last_active":"2024-04-06T13:48:36.210661+0000","last_peered":"2024-04-06T13:48:36.210661+0000","last_clean":"2024-04-06T13:48:36.210661+0000","last_became_active":"2024-04-06T13:48:19.699566+0000","last_became_peered":"2024-04-06T13:48:19.699566+0000","last_unstale":"2024-04-06T13:48:36.210661+0000","last_undegraded":"2024-04-06T13:48:36.210661+0000","last_fullsized":"2024-04-06T13:48:36.210661+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:10:02.935661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212829+0000","last_change":"2024-04-06T13:48:19.276781+0000","last_active":"2024-04-06T13:48:36.212829+0000","last_peered":"2024-04-06T13:48:36.212829+0000","last_clean":"2024-04-06T13:48:36.212829+0000","last_became_active":"2024-04-06T13:48:19.275438+0000","last_became_peered":"2024-04-06T13:48:19.275438+0000","last_unstale":"2024-04-06T13:48:36.212829+0000","last_undegraded":"2024-04-06T13:48:36.212829+0000","last_fullsized":"2024-04-06T13:48:36.212829+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:35.170771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205845+0000","last_change":"2024-04-06T13:48:36.221932+0000","last_active":"2024-04-06T13:48:38.205845+0000","last_peered":"2024-04-06T13:48:38.205845+0000","last_clean":"2024-04-06T13:48:38.205845+0000","last_became_active":"2024-04-06T13:48:36.218607+0000","last_became_peered":"2024-04-06T13:48:36.218607+0000","last_unstale":"2024-04-06T13:48:38.205845+0000","last_undegraded":"2024-04-06T13:48:38.205845+0000","last_fullsized":"2024-04-06T13:48:38.205845+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:53:03.521273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213905+0000","last_change":"2024-04-06T13:48:19.685464+0000","last_active":"2024-04-06T13:48:36.213905+0000","last_peered":"2024-04-06T13:48:36.213905+0000","last_clean":"2024-04-06T13:48:36.213905+0000","last_became_active":"2024-04-06T13:48:19.685139+0000","last_became_peered":"2024-04-06T13:48:19.685139+0000","last_unstale":"2024-04-06T13:48:36.213905+0000","last_undegraded":"2024-04-06T13:48:36.213905+0000","last_fullsized":"2024-04-06T13:48:36.213905+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:19:00.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.207172+0000","last_change":"2024-04-06T13:48:19.279096+0000","last_active":"2024-04-06T13:48:36.207172+0000","last_peered":"2024-04-06T13:48:36.207172+0000","last_clean":"2024-04-06T13:48:36.207172+0000","last_became_active":"2024-04-06T13:48:19.278774+0000","last_became_peered":"2024-04-06T13:48:19.278774+0000","last_unstale":"2024-04-06T13:48:36.207172+0000","last_undegraded":"2024-04-06T13:48:36.207172+0000","last_fullsized":"2024-04-06T13:48:36.207172+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:46:06.202483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212409+0000","last_change":"2024-04-06T13:48:36.226888+0000","last_active":"2024-04-06T13:48:37.212409+0000","last_peered":"2024-04-06T13:48:37.212409+0000","last_clean":"2024-04-06T13:48:37.212409+0000","last_became_active":"2024-04-06T13:48:36.226621+0000","last_became_peered":"2024-04-06T13:48:36.226621+0000","last_unstale":"2024-04-06T13:48:37.212409+0000","last_undegraded":"2024-04-06T13:48:37.212409+0000","last_fullsized":"2024-04-06T13:48:37.212409+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:45:41.254229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.201824+0000","last_change":"2024-04-06T13:48:19.697387+0000","last_active":"2024-04-06T13:48:36.201824+0000","last_peered":"2024-04-06T13:48:36.201824+0000","last_clean":"2024-04-06T13:48:36.201824+0000","last_became_active":"2024-04-06T13:48:19.697096+0000","last_became_peered":"2024-04-06T13:48:19.697096+0000","last_unstale":"2024-04-06T13:48:36.201824+0000","last_undegraded":"2024-04-06T13:48:36.201824+0000","last_fullsized":"2024-04-06T13:48:36.201824+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:52:01.901954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206430+0000","last_change":"2024-04-06T13:48:19.271906+0000","last_active":"2024-04-06T13:48:36.206430+0000","last_peered":"2024-04-06T13:48:36.206430+0000","last_clean":"2024-04-06T13:48:36.206430+0000","last_became_active":"2024-04-06T13:48:19.271643+0000","last_became_peered":"2024-04-06T13:48:19.271643+0000","last_unstale":"2024-04-06T13:48:36.206430+0000","last_undegraded":"2024-04-06T13:48:36.206430+0000","last_fullsized":"2024-04-06T13:48:36.206430+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:54:44.219906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204091+0000","last_change":"2024-04-06T13:48:36.220263+0000","last_active":"2024-04-06T13:48:38.204091+0000","last_peered":"2024-04-06T13:48:38.204091+0000","last_clean":"2024-04-06T13:48:38.204091+0000","last_became_active":"2024-04-06T13:48:36.219217+0000","last_became_peered":"2024-04-06T13:48:36.219217+0000","last_unstale":"2024-04-06T13:48:38.204091+0000","last_undegraded":"2024-04-06T13:48:38.204091+0000","last_fullsized":"2024-04-06T13:48:38.204091+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:08.874119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206574+0000","last_change":"2024-04-06T13:48:19.697332+0000","last_active":"2024-04-06T13:48:36.206574+0000","last_peered":"2024-04-06T13:48:36.206574+0000","last_clean":"2024-04-06T13:48:36.206574+0000","last_became_active":"2024-04-06T13:48:19.697087+0000","last_became_peered":"2024-04-06T13:48:19.697087+0000","last_unstale":"2024-04-06T13:48:36.206574+0000","last_undegraded":"2024-04-06T13:48:36.206574+0000","last_fullsized":"2024-04-06T13:48:36.206574+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:49:24.799254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"248'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213141+0000","last_change":"2024-04-06T13:48:19.279294+0000","last_active":"2024-04-06T13:48:36.213141+0000","last_peered":"2024-04-06T13:48:36.213141+0000","last_clean":"2024-04-06T13:48:36.213141+0000","last_became_active":"2024-04-06T13:48:19.279009+0000","last_became_peered":"2024-04-06T13:48:19.279009+0000","last_unstale":"2024-04-06T13:48:36.213141+0000","last_undegraded":"2024-04-06T13:48:36.213141+0000","last_fullsized":"2024-04-06T13:48:36.213141+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:18.981232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.211568+0000","last_change":"2024-04-06T13:48:36.217779+0000","last_active":"2024-04-06T13:48:37.211568+0000","last_peered":"2024-04-06T13:48:37.211568+0000","last_clean":"2024-04-06T13:48:37.211568+0000","last_became_active":"2024-04-06T13:48:36.217253+0000","last_became_peered":"2024-04-06T13:48:36.217253+0000","last_unstale":"2024-04-06T13:48:37.211568+0000","last_undegraded":"2024-04-06T13:48:37.211568+0000","last_fullsized":"2024-04-06T13:48:37.211568+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:13:26.069286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.205490+0000","last_change":"2024-04-06T13:48:19.694094+0000","last_active":"2024-04-06T13:48:36.205490+0000","last_peered":"2024-04-06T13:48:36.205490+0000","last_clean":"2024-04-06T13:48:36.205490+0000","last_became_active":"2024-04-06T13:48:19.692958+0000","last_became_peered":"2024-04-06T13:48:19.692958+0000","last_unstale":"2024-04-06T13:48:36.205490+0000","last_undegraded":"2024-04-06T13:48:36.205490+0000","last_fullsized":"2024-04-06T13:48:36.205490+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:04:10.547739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.207489+0000","last_change":"2024-04-06T13:48:19.272078+0000","last_active":"2024-04-06T13:48:36.207489+0000","last_peered":"2024-04-06T13:48:36.207489+0000","last_clean":"2024-04-06T13:48:36.207489+0000","last_became_active":"2024-04-06T13:48:19.271672+0000","last_became_peered":"2024-04-06T13:48:19.271672+0000","last_unstale":"2024-04-06T13:48:36.207489+0000","last_undegraded":"2024-04-06T13:48:36.207489+0000","last_fullsized":"2024-04-06T13:48:36.207489+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:23.871607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203999+0000","last_change":"2024-04-06T13:48:36.221782+0000","last_active":"2024-04-06T13:48:38.203999+0000","last_peered":"2024-04-06T13:48:38.203999+0000","last_clean":"2024-04-06T13:48:38.203999+0000","last_became_active":"2024-04-06T13:48:36.221533+0000","last_became_peered":"2024-04-06T13:48:36.221533+0000","last_unstale":"2024-04-06T13:48:38.203999+0000","last_undegraded":"2024-04-06T13:48:38.203999+0000","last_fullsized":"2024-04-06T13:48:38.203999+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:38:01.256020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.214900+0000","last_change":"2024-04-06T13:48:19.702071+0000","last_active":"2024-04-06T13:48:36.214900+0000","last_peered":"2024-04-06T13:48:36.214900+0000","last_clean":"2024-04-06T13:48:36.214900+0000","last_became_active":"2024-04-06T13:48:19.698755+0000","last_became_peered":"2024-04-06T13:48:19.698755+0000","last_unstale":"2024-04-06T13:48:36.214900+0000","last_undegraded":"2024-04-06T13:48:36.214900+0000","last_fullsized":"2024-04-06T13:48:36.214900+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:06:58.060998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206596+0000","last_change":"2024-04-06T13:48:19.278939+0000","last_active":"2024-04-06T13:48:36.206596+0000","last_peered":"2024-04-06T13:48:36.206596+0000","last_clean":"2024-04-06T13:48:36.206596+0000","last_became_active":"2024-04-06T13:48:19.278677+0000","last_became_peered":"2024-04-06T13:48:19.278677+0000","last_unstale":"2024-04-06T13:48:36.206596+0000","last_undegraded":"2024-04-06T13:48:36.206596+0000","last_fullsized":"2024-04-06T13:48:36.206596+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:15.157954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204721+0000","last_change":"2024-04-06T13:48:36.226108+0000","last_active":"2024-04-06T13:48:38.204721+0000","last_peered":"2024-04-06T13:48:38.204721+0000","last_clean":"2024-04-06T13:48:38.204721+0000","last_became_active":"2024-04-06T13:48:36.225708+0000","last_became_peered":"2024-04-06T13:48:36.225708+0000","last_unstale":"2024-04-06T13:48:38.204721+0000","last_undegraded":"2024-04-06T13:48:38.204721+0000","last_fullsized":"2024-04-06T13:48:38.204721+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:50:11.010781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.205937+0000","last_change":"2024-04-06T13:48:19.693016+0000","last_active":"2024-04-06T13:48:36.205937+0000","last_peered":"2024-04-06T13:48:36.205937+0000","last_clean":"2024-04-06T13:48:36.205937+0000","last_became_active":"2024-04-06T13:48:19.692760+0000","last_became_peered":"2024-04-06T13:48:19.692760+0000","last_unstale":"2024-04-06T13:48:36.205937+0000","last_undegraded":"2024-04-06T13:48:36.205937+0000","last_fullsized":"2024-04-06T13:48:36.205937+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:41:37.714397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212291+0000","last_change":"2024-04-06T13:48:19.282189+0000","last_active":"2024-04-06T13:48:36.212291+0000","last_peered":"2024-04-06T13:48:36.212291+0000","last_clean":"2024-04-06T13:48:36.212291+0000","last_became_active":"2024-04-06T13:48:19.281907+0000","last_became_peered":"2024-04-06T13:48:19.281907+0000","last_unstale":"2024-04-06T13:48:36.212291+0000","last_undegraded":"2024-04-06T13:48:36.212291+0000","last_fullsized":"2024-04-06T13:48:36.212291+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:34.453253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3f","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205887+0000","last_change":"2024-04-06T13:48:36.229947+0000","last_active":"2024-04-06T13:48:38.205887+0000","last_peered":"2024-04-06T13:48:38.205887+0000","last_clean":"2024-04-06T13:48:38.205887+0000","last_became_active":"2024-04-06T13:48:36.229670+0000","last_became_peered":"2024-04-06T13:48:36.229670+0000","last_unstale":"2024-04-06T13:48:38.205887+0000","last_undegraded":"2024-04-06T13:48:38.205887+0000","last_fullsized":"2024-04-06T13:48:38.205887+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:30:28.148833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.202135+0000","last_change":"2024-04-06T13:48:19.689941+0000","last_active":"2024-04-06T13:48:36.202135+0000","last_peered":"2024-04-06T13:48:36.202135+0000","last_clean":"2024-04-06T13:48:36.202135+0000","last_became_active":"2024-04-06T13:48:19.689689+0000","last_became_peered":"2024-04-06T13:48:19.689689+0000","last_unstale":"2024-04-06T13:48:36.202135+0000","last_undegraded":"2024-04-06T13:48:36.202135+0000","last_fullsized":"2024-04-06T13:48:36.202135+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:55:19.563924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213316+0000","last_change":"2024-04-06T13:48:19.283271+0000","last_active":"2024-04-06T13:48:36.213316+0000","last_peered":"2024-04-06T13:48:36.213316+0000","last_clean":"2024-04-06T13:48:36.213316+0000","last_became_active":"2024-04-06T13:48:19.282980+0000","last_became_peered":"2024-04-06T13:48:19.282980+0000","last_unstale":"2024-04-06T13:48:36.213316+0000","last_undegraded":"2024-04-06T13:48:36.213316+0000","last_fullsized":"2024-04-06T13:48:36.213316+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:51:53.597002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.211904+0000","last_change":"2024-04-06T13:48:36.213917+0000","last_active":"2024-04-06T13:48:37.211904+0000","last_peered":"2024-04-06T13:48:37.211904+0000","last_clean":"2024-04-06T13:48:37.211904+0000","last_became_active":"2024-04-06T13:48:36.213679+0000","last_became_peered":"2024-04-06T13:48:36.213679+0000","last_unstale":"2024-04-06T13:48:37.211904+0000","last_undegraded":"2024-04-06T13:48:37.211904+0000","last_fullsized":"2024-04-06T13:48:37.211904+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:59:30.769144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204267+0000","last_change":"2024-04-06T13:48:19.704846+0000","last_active":"2024-04-06T13:48:36.204267+0000","last_peered":"2024-04-06T13:48:36.204267+0000","last_clean":"2024-04-06T13:48:36.204267+0000","last_became_active":"2024-04-06T13:48:19.704420+0000","last_became_peered":"2024-04-06T13:48:19.704420+0000","last_unstale":"2024-04-06T13:48:36.204267+0000","last_undegraded":"2024-04-06T13:48:36.204267+0000","last_fullsized":"2024-04-06T13:48:36.204267+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:36.178732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.205312+0000","last_change":"2024-04-06T13:48:19.277530+0000","last_active":"2024-04-06T13:48:36.205312+0000","last_peered":"2024-04-06T13:48:36.205312+0000","last_clean":"2024-04-06T13:48:36.205312+0000","last_became_active":"2024-04-06T13:48:19.276966+0000","last_became_peered":"2024-04-06T13:48:19.276966+0000","last_unstale":"2024-04-06T13:48:36.205312+0000","last_undegraded":"2024-04-06T13:48:36.205312+0000","last_fullsized":"2024-04-06T13:48:36.205312+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:42:14.962783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203678+0000","last_change":"2024-04-06T13:48:36.222153+0000","last_active":"2024-04-06T13:48:38.203678+0000","last_peered":"2024-04-06T13:48:38.203678+0000","last_clean":"2024-04-06T13:48:38.203678+0000","last_became_active":"2024-04-06T13:48:36.221901+0000","last_became_peered":"2024-04-06T13:48:36.221901+0000","last_unstale":"2024-04-06T13:48:38.203678+0000","last_undegraded":"2024-04-06T13:48:38.203678+0000","last_fullsized":"2024-04-06T13:48:38.203678+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:00:42.061268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213987+0000","last_change":"2024-04-06T13:48:19.688861+0000","last_active":"2024-04-06T13:48:36.213987+0000","last_peered":"2024-04-06T13:48:36.213987+0000","last_clean":"2024-04-06T13:48:36.213987+0000","last_became_active":"2024-04-06T13:48:19.688496+0000","last_became_peered":"2024-04-06T13:48:19.688496+0000","last_unstale":"2024-04-06T13:48:36.213987+0000","last_undegraded":"2024-04-06T13:48:36.213987+0000","last_fullsized":"2024-04-06T13:48:36.213987+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:36:47.917157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212664+0000","last_change":"2024-04-06T13:48:19.276467+0000","last_active":"2024-04-06T13:48:36.212664+0000","last_peered":"2024-04-06T13:48:36.212664+0000","last_clean":"2024-04-06T13:48:36.212664+0000","last_became_active":"2024-04-06T13:48:19.275988+0000","last_became_peered":"2024-04-06T13:48:19.275988+0000","last_unstale":"2024-04-06T13:48:36.212664+0000","last_undegraded":"2024-04-06T13:48:36.212664+0000","last_fullsized":"2024-04-06T13:48:36.212664+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:44.664354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212005+0000","last_change":"2024-04-06T13:48:36.211369+0000","last_active":"2024-04-06T13:48:37.212005+0000","last_peered":"2024-04-06T13:48:37.212005+0000","last_clean":"2024-04-06T13:48:37.212005+0000","last_became_active":"2024-04-06T13:48:36.210435+0000","last_became_peered":"2024-04-06T13:48:36.210435+0000","last_unstale":"2024-04-06T13:48:37.212005+0000","last_undegraded":"2024-04-06T13:48:37.212005+0000","last_fullsized":"2024-04-06T13:48:37.212005+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:49:52.157497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.201932+0000","last_change":"2024-04-06T13:48:19.700894+0000","last_active":"2024-04-06T13:48:36.201932+0000","last_peered":"2024-04-06T13:48:36.201932+0000","last_clean":"2024-04-06T13:48:36.201932+0000","last_became_active":"2024-04-06T13:48:19.700666+0000","last_became_peered":"2024-04-06T13:48:19.700666+0000","last_unstale":"2024-04-06T13:48:36.201932+0000","last_undegraded":"2024-04-06T13:48:36.201932+0000","last_fullsized":"2024-04-06T13:48:36.201932+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:41:21.910614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.202977+0000","last_change":"2024-04-06T13:48:19.277300+0000","last_active":"2024-04-06T13:48:36.202977+0000","last_peered":"2024-04-06T13:48:36.202977+0000","last_clean":"2024-04-06T13:48:36.202977+0000","last_became_active":"2024-04-06T13:48:19.276795+0000","last_became_peered":"2024-04-06T13:48:19.276795+0000","last_unstale":"2024-04-06T13:48:36.202977+0000","last_undegraded":"2024-04-06T13:48:36.202977+0000","last_fullsized":"2024-04-06T13:48:36.202977+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:54:25.950801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"233'370","reported_seq":1341,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212261+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:48:37.212261+0000","last_peered":"2024-04-06T13:48:37.212261+0000","last_clean":"2024-04-06T13:48:37.212261+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:48:37.212261+0000","last_undegraded":"2024-04-06T13:48:37.212261+0000","last_fullsized":"2024-04-06T13:48:37.212261+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":3611,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575996,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56956,"kb_used_data":3696,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314884,"statfs":{"total":96636764160,"available":96578441216,"internally_reserved":0,"allocated":3784704,"data_stored":2225830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57899999999999996,"15min":0.59499999999999997},"min":{"1min":0.35499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.72199999999999998,"5min":0.78400000000000003,"15min":0.82499999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56799999999999995,"15min":0.58999999999999997},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78500000000000003,"5min":0.80800000000000005,"15min":0.82899999999999996},"last":0.55700000000000005}]},{"osd":1,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.55000000000000004,"15min":0.57899999999999996},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77200000000000002,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.57699999999999996,"15min":0.59599999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.69899999999999995,"5min":0.83399999999999996,"15min":0.85299999999999998},"last":0.59199999999999997}]},{"osd":2,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.56299999999999994,"15min":0.58499999999999996},"min":{"1min":0.33200000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69399999999999995,"5min":0.81699999999999995,"15min":0.83899999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.58599999999999997,"15min":0.60099999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69999999999999996,"5min":0.755,"15min":0.79700000000000004},"last":0.502}]},{"osd":3,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.59399999999999997,"15min":0.60599999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.67200000000000004,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58799999999999997,"15min":0.60299999999999998},"min":{"1min":0.39000000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78900000000000003,"5min":0.82099999999999995,"15min":0.84099999999999997},"last":0.53200000000000003}]},{"osd":4,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45800000000000002,"15min":0.46600000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.81599999999999995},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.36799999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.82699999999999996},"last":0.41099999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.498,"15min":0.498},"min":{"1min":0.315,"5min":0.315,"15min":0.308},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56899999999999995,"15min":0.57799999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33600000000000002},"max":{"1min":0.74099999999999999,"5min":0.80700000000000005,"15min":0.85399999999999998},"last":0.44500000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.34899999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.879},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57099999999999995,"15min":0.58699999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81799999999999995,"5min":0.84899999999999998,"15min":0.89800000000000002},"last":0.49099999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575995,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56364,"kb_used_data":3096,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315476,"statfs":{"total":96636764160,"available":96579047424,"internally_reserved":0,"allocated":3170304,"data_stored":1626302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.53600000000000003,"15min":0.55700000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69399999999999995,"5min":0.79700000000000004,"15min":0.83799999999999997},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.54100000000000004,"15min":0.56499999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.62,"5min":0.76900000000000002,"15min":0.81999999999999995},"last":0.63}]},{"osd":1,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.60399999999999998,"15min":0.624},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76000000000000001,"5min":0.81699999999999995,"15min":0.85799999999999998},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.622,"15min":0.64400000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.378},"max":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.82699999999999996},"last":0.69799999999999995}]},{"osd":2,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56999999999999995,"15min":0.59399999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72099999999999997,"5min":0.79000000000000004,"15min":0.82899999999999996},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.56299999999999994,"15min":0.58599999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.80700000000000005},"last":0.65800000000000003}]},{"osd":3,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.58599999999999997,"15min":0.60199999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.73199999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.61899999999999999,"15min":0.64300000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.36099999999999999},"max":{"1min":0.80200000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.64400000000000002}]},{"osd":4,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70599999999999996,"5min":0.72299999999999998,"15min":0.81399999999999995},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.51100000000000001,"15min":0.53800000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.66400000000000003,"5min":0.82099999999999995,"15min":0.84099999999999997},"last":0.501}]},{"osd":5,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.48899999999999999,"15min":0.50600000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.33300000000000002},"max":{"1min":0.63,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52000000000000002,"15min":0.53100000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67400000000000004,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.51900000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53900000000000003,"15min":0.54800000000000004},"min":{"1min":0.34899999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.67700000000000005,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55300000000000005,"15min":0.56899999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.32800000000000001},"max":{"1min":0.71199999999999997,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.57599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575994,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60460,"kb_used_data":3104,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311380,"statfs":{"total":96636764160,"available":96574853120,"internally_reserved":0,"allocated":3178496,"data_stored":1633839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.38100000000000001,"15min":0.39100000000000001},"min":{"1min":0.248,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.378,"5min":0.621,"15min":0.621},"last":0.308},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.41799999999999998,"15min":0.42699999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.501,"5min":0.60799999999999998,"15min":0.69399999999999995},"last":0.22}]},{"osd":2,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.41299999999999998,"15min":0.42399999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.46899999999999997,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.378,"5min":0.44900000000000001,"15min":0.46400000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.55400000000000005,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.32700000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.46999999999999997,"15min":0.47999999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.52800000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.375},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.42699999999999999,"15min":0.439},"min":{"1min":0.28799999999999998,"5min":0.27600000000000002,"15min":0.26300000000000001},"max":{"1min":0.45500000000000002,"5min":0.63500000000000001,"15min":0.66100000000000003},"last":0.35099999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.628,"15min":0.64400000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.33800000000000002},"max":{"1min":0.70099999999999996,"5min":0.89900000000000002,"15min":0.97399999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.64400000000000002,"15min":0.65500000000000003},"min":{"1min":0.46100000000000002,"5min":0.44500000000000001,"15min":0.44400000000000001},"max":{"1min":0.73499999999999999,"5min":0.94599999999999995,"15min":0.995},"last":0.48899999999999999}]},{"osd":5,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.64100000000000001,"15min":0.65300000000000002},"min":{"1min":0.42999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73899999999999999,"5min":0.96899999999999997,"15min":1.073},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.65700000000000003,"15min":0.66800000000000004},"min":{"1min":0.45200000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.72799999999999998,"5min":0.96499999999999997,"15min":1.0209999999999999},"last":0.53800000000000003}]},{"osd":6,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.46300000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73899999999999999,"5min":0.89200000000000002,"15min":1},"last":0.502},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.67100000000000004,"15min":0.68400000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.72499999999999998,"5min":0.94599999999999995,"15min":1.036},"last":0.55000000000000004}]},{"osd":7,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.69399999999999995,"15min":0.69499999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85399999999999998,"5min":0.94899999999999995,"15min":1.0649999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65700000000000003,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.76300000000000001,"5min":0.92900000000000005,"15min":1.056},"last":0.47199999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575994,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61048,"kb_used_data":3692,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310792,"statfs":{"total":96636764160,"available":96574251008,"internally_reserved":0,"allocated":3780608,"data_stored":2226966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.38200000000000001,"15min":0.39300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.45100000000000001,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.39300000000000002,"15min":0.41699999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.48199999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.28100000000000003}]},{"osd":2,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.40100000000000002,"15min":0.42499999999999999},"min":{"1min":0.26100000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49399999999999999,"5min":0.66700000000000004,"15min":0.70099999999999996},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.375,"5min":0.42499999999999999,"15min":0.443},"min":{"1min":0.28399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.47999999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.34599999999999997}]},{"osd":3,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.40699999999999997,"15min":0.42499999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.50700000000000001,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.317},{"interface":"front","average":{"1min":0.373,"5min":0.45700000000000002,"15min":0.47799999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.51900000000000002,"5min":0.72999999999999998,"15min":0.75700000000000001},"last":0.41299999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.60499999999999998,"15min":0.64100000000000001},"min":{"1min":0.39100000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.80900000000000005,"5min":0.93100000000000005,"15min":1.0329999999999999},"last":0.499},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.61199999999999999,"15min":0.64300000000000002},"min":{"1min":0.372,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88500000000000001,"5min":0.999,"15min":0.999},"last":0.58899999999999997}]},{"osd":5,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.60199999999999998,"15min":0.63900000000000001},"min":{"1min":0.39300000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.0009999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.63800000000000001,"15min":0.66300000000000003},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.749,"5min":0.92800000000000005,"15min":1.052},"last":0.50900000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.64500000000000002,"15min":0.67400000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.874,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.64200000000000002,"15min":0.66900000000000004},"min":{"1min":0.40500000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77500000000000002,"5min":0.96399999999999997,"15min":1.085},"last":0.52000000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.66300000000000003,"15min":0.69099999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.80700000000000005,"5min":1.01,"15min":1.121},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.64700000000000002,"15min":0.67200000000000004},"min":{"1min":0.42199999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.93700000000000006,"5min":0.95799999999999996,"15min":1.0229999999999999},"last":0.65800000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575996,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60492,"kb_used_data":3124,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311348,"statfs":{"total":96636764160,"available":96574820352,"internally_reserved":0,"allocated":3198976,"data_stored":1636535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.376,"15min":0.38400000000000001},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.42299999999999999,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.40600000000000003,"15min":0.42199999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.249},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":1.044},"last":0.54600000000000004}]},{"osd":1,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.38500000000000001,"15min":0.40400000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51100000000000001,"5min":0.54000000000000004,"15min":0.753},"last":0.432},{"interface":"front","average":{"1min":0.37,"5min":0.42799999999999999,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.82199999999999995},"last":0.77200000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.439,"15min":0.45800000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.53200000000000003,"5min":0.69699999999999995,"15min":0.70299999999999996},"last":0.621},{"interface":"front","average":{"1min":0.377,"5min":0.437,"15min":0.44900000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.504,"5min":0.64500000000000002,"15min":0.68600000000000005},"last":0.65200000000000002}]},{"osd":4,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.621,"15min":0.63900000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.376},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.64400000000000002,"15min":0.66100000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.88900000000000001,"5min":1.022,"15min":1.022},"last":0.82499999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.64300000000000002,"15min":0.66300000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88100000000000001,"5min":0.98599999999999999,"15min":1.0149999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.65100000000000002,"15min":0.66700000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.80600000000000005}]},{"osd":6,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.65200000000000002,"15min":0.67200000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92000000000000004,"5min":0.97199999999999998,"15min":1.091},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.67600000000000005,"15min":0.69499999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.96099999999999997},"last":0.84099999999999997}]},{"osd":7,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.66600000000000004,"15min":0.68799999999999994},"min":{"1min":0.42699999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.95399999999999996},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.68300000000000005,"15min":0.70799999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.85299999999999998,"5min":1.01,"15min":1.01},"last":0.89100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575995,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48196,"kb_used_data":3128,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323644,"statfs":{"total":96636764160,"available":96587411456,"internally_reserved":0,"allocated":3203072,"data_stored":1638275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.37,"15min":0.38400000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.47399999999999998,"5min":0.55200000000000005,"15min":0.72799999999999998},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.39600000000000002,"15min":0.40799999999999997},"min":{"1min":0.25700000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.48799999999999999,"5min":0.58599999999999997,"15min":0.66700000000000004},"last":0.251}]},{"osd":1,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.39300000000000002,"15min":0.40100000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.46899999999999997,"5min":0.59499999999999997,"15min":0.65700000000000003},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.41699999999999998,"15min":0.437},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.502,"5min":0.63,"15min":0.75800000000000001},"last":0.30399999999999999}]},{"osd":2,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.46100000000000002,"15min":0.47999999999999998},"min":{"1min":0.32700000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.53900000000000003,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.47699999999999998,"15min":0.496},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.51900000000000002,"5min":0.67900000000000005,"15min":0.79300000000000004},"last":0.32700000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.622,"15min":0.63100000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76700000000000002,"5min":0.94099999999999995,"15min":1.018},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.628,"15min":0.64200000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.81100000000000005,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.47399999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.66100000000000003,"15min":0.67600000000000005},"min":{"1min":0.44700000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.872,"5min":0.95699999999999996,"15min":0.999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.67900000000000005,"15min":0.68700000000000006},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.86299999999999999,"5min":1.036,"15min":1.036},"last":0.55000000000000004}]},{"osd":6,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.64100000000000001,"15min":0.66100000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.76300000000000001,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.66000000000000003,"15min":0.67600000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41099999999999998},"max":{"1min":0.82799999999999996,"5min":0.91600000000000004,"15min":0.98799999999999999},"last":0.46500000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.65500000000000003,"15min":0.66800000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80200000000000005,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.67800000000000005,"15min":0.69499999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85199999999999998,"5min":1.0209999999999999,"15min":1.024},"last":0.58499999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575995,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48180,"kb_used_data":3104,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323660,"statfs":{"total":96636764160,"available":96587427840,"internally_reserved":0,"allocated":3178496,"data_stored":1628809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55900000000000005,"15min":0.58699999999999997},"min":{"1min":0.40699999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.753,"5min":0.876,"15min":0.88500000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.39300000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81100000000000005,"5min":1.202,"15min":1.202},"last":0.501}]},{"osd":1,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.61299999999999999,"15min":0.64000000000000001},"min":{"1min":0.40799999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71699999999999997,"5min":1.256,"15min":1.256},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.624,"15min":0.63},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.75900000000000001,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.61699999999999999}]},{"osd":2,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.57999999999999996,"15min":0.59999999999999998},"min":{"1min":0.41599999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77500000000000002,"5min":0.84599999999999997,"15min":0.877},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.60599999999999998,"15min":0.61599999999999999},"min":{"1min":0.39600000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.97899999999999998,"5min":1.294,"15min":1.294},"last":0.59399999999999997}]},{"osd":3,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84499999999999997,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.60899999999999999,"15min":0.626},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.78800000000000003,"5min":0.90200000000000002,"15min":0.91100000000000003},"last":0.53300000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55500000000000005,"15min":0.52000000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.92800000000000005,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":0.63},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.56399999999999995,"15min":0.54100000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.95199999999999996,"5min":4.931,"15min":4.931},"last":0.46400000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53500000000000003,"15min":0.53300000000000003},"min":{"1min":0.34999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34699999999999998},"max":{"1min":1.0609999999999999,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.437}]},{"osd":7,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59499999999999997,"15min":0.58199999999999996},"min":{"1min":0.36899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.91000000000000003,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59399999999999997,"15min":0.60199999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.373},"max":{"1min":0.86899999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.48599999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575995,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66212,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305628,"statfs":{"total":96636764160,"available":96568963072,"internally_reserved":0,"allocated":3162112,"data_stored":1626226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.57399999999999995,"15min":0.58499999999999996},"min":{"1min":0.36399999999999999,"5min":0.35899999999999999,"15min":0.33000000000000002},"max":{"1min":0.68000000000000005,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58099999999999996,"15min":0.59399999999999997},"min":{"1min":0.38100000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.76600000000000001,"5min":0.84699999999999998,"15min":0.89200000000000002},"last":0.61399999999999999}]},{"osd":1,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.58099999999999996,"15min":0.59399999999999997},"min":{"1min":0.38400000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79100000000000004,"5min":0.81299999999999994,"15min":0.81399999999999995},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.439,"5min":0.39400000000000002,"15min":0.37},"max":{"1min":0.70299999999999996,"5min":0.871,"15min":0.871},"last":0.56499999999999995}]},{"osd":2,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59299999999999997,"15min":0.59699999999999998},"min":{"1min":0.311,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69199999999999995,"5min":0.79700000000000004,"15min":0.84399999999999997},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58799999999999997,"15min":0.59599999999999997},"min":{"1min":0.44,"5min":0.378,"15min":0.378},"max":{"1min":0.73099999999999998,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.51400000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.79800000000000004,"5min":0.81100000000000005,"15min":0.94499999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61699999999999999,"15min":0.63200000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.74099999999999999,"5min":0.88200000000000001,"15min":0.91500000000000004},"last":0.54600000000000004}]},{"osd":4,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52900000000000003,"15min":0.50700000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.27400000000000002},"max":{"1min":0.70999999999999996,"5min":0.85699999999999998,"15min":1.0369999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.59999999999999998,"15min":0.57699999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.308},"max":{"1min":0.77800000000000002,"5min":0.88800000000000001,"15min":1.1639999999999999},"last":0.65800000000000003}]},{"osd":6,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54300000000000004,"15min":0.52600000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.27900000000000003},"max":{"1min":0.89100000000000001,"5min":0.94999999999999996,"15min":1.0169999999999999},"last":0.629},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54300000000000004,"15min":0.53100000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.32300000000000001},"max":{"1min":0.81899999999999995,"5min":0.878,"15min":1.0009999999999999},"last":0.57899999999999996}]},{"osd":7,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56499999999999995,"15min":0.54800000000000004},"min":{"1min":0.39200000000000002,"5min":0.35799999999999998,"15min":0.32900000000000001},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.66400000000000003,"15min":0.64800000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.35199999999999998},"max":{"1min":0.85299999999999998,"5min":0.92300000000000004,"15min":0.96599999999999997},"last":0.68200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:48:40.341 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:48:40.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.498+0000 7f0ea00f2700 1 Processor -- start 2024-04-06T13:48:40.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.499+0000 7f0ea00f2700 1 -- start start 2024-04-06T13:48:40.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.499+0000 7f0ea00f2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0e980cb870 0x7f0e9815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:40.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.499+0000 7f0ea00f2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e980cfa00 0x7f0e980cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:40.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.499+0000 7f0ea00f2700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e980cbd80 con 0x7f0e980d11b0 2024-04-06T13:48:40.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.499+0000 7f0ea00f2700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e980ccdc0 con 0x7f0e980cb870 2024-04-06T13:48:40.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.499+0000 7f0ea00f2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e980d0b50 con 0x7f0e980cfa00 2024-04-06T13:48:40.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.500+0000 7f0e9d68d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0e980cb870 0x7f0e9815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:40.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.500+0000 7f0e9d68d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0e980cb870 0x7f0e9815c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34568/0 (socket says 172.21.15.189:34568) 2024-04-06T13:48:40.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.500+0000 7f0e9d68d700 1 -- 172.21.15.189:0/2487599127 learned_addr learned my addr 172.21.15.189:0/2487599127 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:40.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.500+0000 7f0e9e68f700 1 --2- 172.21.15.189:0/2487599127 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e980cfa00 0x7f0e980cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:40.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.500+0000 7f0e9d68d700 1 -- 172.21.15.189:0/2487599127 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e980cfa00 msgr2=0x7f0e980cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:40.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.500+0000 7f0e9d68d700 1 --2- 172.21.15.189:0/2487599127 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e980cfa00 0x7f0e980cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.500+0000 7f0e9d68d700 1 -- 172.21.15.189:0/2487599127 >> v1:172.21.15.192:6789/0 conn(0x7f0e980d11b0 legacy=0x7f0e980d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:40.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.500+0000 7f0e9d68d700 1 -- 172.21.15.189:0/2487599127 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e980cef20 con 0x7f0e980cb870 2024-04-06T13:48:40.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.500+0000 7f0e9d68d700 1 --2- 172.21.15.189:0/2487599127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e980cb870 0x7f0e9815c280 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f0e940098f0 tx=0x7f0e94009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558fb719c63e99e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:40.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.500+0000 7f0e9ce8c700 1 -- 172.21.15.189:0/2487599127 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9400b8a0 con 0x7f0e980cb870 2024-04-06T13:48:40.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.500+0000 7f0e9ce8c700 1 -- 172.21.15.189:0/2487599127 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9400ba00 con 0x7f0e980cb870 2024-04-06T13:48:40.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.500+0000 7f0e9ce8c700 1 -- 172.21.15.189:0/2487599127 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9400bd90 con 0x7f0e980cb870 2024-04-06T13:48:40.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.501+0000 7f0ea00f2700 1 -- 172.21.15.189:0/2487599127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e980cb870 msgr2=0x7f0e9815c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:40.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.501+0000 7f0ea00f2700 1 --2- 172.21.15.189:0/2487599127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e980cb870 0x7f0e9815c280 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f0e940098f0 tx=0x7f0e94009c00 comp rx=0 tx=0).stop 2024-04-06T13:48:40.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.501+0000 7f0ea00f2700 1 -- 172.21.15.189:0/2487599127 shutdown_connections 2024-04-06T13:48:40.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.501+0000 7f0ea00f2700 1 --2- 172.21.15.189:0/2487599127 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e980cfa00 0x7f0e980cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.501+0000 7f0ea00f2700 1 --2- 172.21.15.189:0/2487599127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e980cb870 0x7f0e9815c280 unknown :-1 s=CLOSED pgs=2743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.501+0000 7f0ea00f2700 1 -- 172.21.15.189:0/2487599127 >> 172.21.15.189:0/2487599127 conn(0x7f0e980b2db0 msgr2=0x7f0e980b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:40.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.501+0000 7f0ea00f2700 1 -- 172.21.15.189:0/2487599127 shutdown_connections 2024-04-06T13:48:40.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.501+0000 7f0ea00f2700 1 -- 172.21.15.189:0/2487599127 wait complete. 2024-04-06T13:48:40.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.502+0000 7f0ea00f2700 1 Processor -- start 2024-04-06T13:48:40.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.502+0000 7f0ea00f2700 1 -- start start 2024-04-06T13:48:40.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.503+0000 7f0ea00f2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e980cb870 0x7f0e980b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:40.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.503+0000 7f0ea00f2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e980cfa00 0x7f0e98158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:40.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.503+0000 7f0ea00f2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e980d11b0 0x7f0e98151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:40.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.503+0000 7f0ea00f2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e98151c70 con 0x7f0e980cfa00 2024-04-06T13:48:40.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.503+0000 7f0ea00f2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e980b44f0 con 0x7f0e980d11b0 2024-04-06T13:48:40.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.503+0000 7f0ea00f2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e980b47d0 con 0x7f0e980cb870 2024-04-06T13:48:40.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.503+0000 7f0e9d68d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e980cfa00 0x7f0e98158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:40.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.504+0000 7f0e9d68d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e980cfa00 0x7f0e98158940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34574/0 (socket says 172.21.15.189:34574) 2024-04-06T13:48:40.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.504+0000 7f0e9d68d700 1 -- 172.21.15.189:0/628923152 learned_addr learned my addr 172.21.15.189:0/628923152 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:40.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.504+0000 7f0e9e68f700 1 --2- 172.21.15.189:0/628923152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e980d11b0 0x7f0e98151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:40.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.504+0000 7f0e9de8e700 1 --2- 172.21.15.189:0/628923152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e980cb870 0x7f0e980b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:40.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.504+0000 7f0e9d68d700 1 -- 172.21.15.189:0/628923152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e980cb870 msgr2=0x7f0e980b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:40.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.504+0000 7f0e9d68d700 1 --2- 172.21.15.189:0/628923152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e980cb870 0x7f0e980b3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.504+0000 7f0e9d68d700 1 -- 172.21.15.189:0/628923152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e980d11b0 msgr2=0x7f0e98151730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:40.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.504+0000 7f0e9d68d700 1 --2- 172.21.15.189:0/628923152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e980d11b0 0x7f0e98151730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.504+0000 7f0e9d68d700 1 -- 172.21.15.189:0/628923152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e94009550 con 0x7f0e980cfa00 2024-04-06T13:48:40.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.504+0000 7f0e9d68d700 1 --2- 172.21.15.189:0/628923152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e980cfa00 0x7f0e98158940 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f0e94000c00 tx=0x7f0e94004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:40.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.504+0000 7f0e8a7fc700 1 -- 172.21.15.189:0/628923152 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e94004400 con 0x7f0e980cfa00 2024-04-06T13:48:40.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.504+0000 7f0e8a7fc700 1 -- 172.21.15.189:0/628923152 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9401f070 con 0x7f0e980cfa00 2024-04-06T13:48:40.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.504+0000 7f0ea00f2700 1 -- 172.21.15.189:0/628923152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e980b4ab0 con 0x7f0e980cfa00 2024-04-06T13:48:40.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.504+0000 7f0e8a7fc700 1 -- 172.21.15.189:0/628923152 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e940038e0 con 0x7f0e980cfa00 2024-04-06T13:48:40.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.504+0000 7f0ea00f2700 1 -- 172.21.15.189:0/628923152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e980b5070 con 0x7f0e980cfa00 2024-04-06T13:48:40.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.505+0000 7f0ea00f2700 1 -- 172.21.15.189:0/628923152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e98074990 con 0x7f0e980cfa00 2024-04-06T13:48:40.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.506+0000 7f0e8a7fc700 1 -- 172.21.15.189:0/628923152 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f0e94021030 con 0x7f0e980cfa00 2024-04-06T13:48:40.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.506+0000 7f0e8a7fc700 1 --2- 172.21.15.189:0/628923152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0e7c06af00 0x7f0e7c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:40.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.506+0000 7f0e8a7fc700 1 -- 172.21.15.189:0/628923152 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f0e9408b150 con 0x7f0e980cfa00 2024-04-06T13:48:40.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.506+0000 7f0e9de8e700 1 --2- 172.21.15.189:0/628923152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0e7c06af00 0x7f0e7c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:40.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.506+0000 7f0e9de8e700 1 --2- 172.21.15.189:0/628923152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0e7c06af00 0x7f0e7c06d3c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c000f60 tx=0x7f0e8c009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:40.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.510+0000 7f0e8a7fc700 1 -- 172.21.15.189:0/628923152 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0e94055b70 con 0x7f0e980cfa00 2024-04-06T13:48:40.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.703+0000 7f0ea00f2700 1 -- 172.21.15.189:0/628923152 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0e980cc060 con 0x7f0e7c06af00 2024-04-06T13:48:40.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.716+0000 7f0e8a7fc700 1 -- 172.21.15.189:0/628923152 <== mgr.11733 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+529167 (secure 0 0 0) 0x7f0e8c0013f0 con 0x7f0e7c06af00 2024-04-06T13:48:40.724 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:40.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.721+0000 7f0ea00f2700 1 -- 172.21.15.189:0/628923152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0e7c06af00 msgr2=0x7f0e7c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:40.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.721+0000 7f0ea00f2700 1 --2- 172.21.15.189:0/628923152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0e7c06af00 0x7f0e7c06d3c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c000f60 tx=0x7f0e8c009040 comp rx=0 tx=0).stop 2024-04-06T13:48:40.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.721+0000 7f0ea00f2700 1 -- 172.21.15.189:0/628923152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e980cfa00 msgr2=0x7f0e98158940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:40.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.721+0000 7f0ea00f2700 1 --2- 172.21.15.189:0/628923152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e980cfa00 0x7f0e98158940 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f0e94000c00 tx=0x7f0e94004750 comp rx=0 tx=0).stop 2024-04-06T13:48:40.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.721+0000 7f0ea00f2700 1 -- 172.21.15.189:0/628923152 shutdown_connections 2024-04-06T13:48:40.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.721+0000 7f0ea00f2700 1 --2- 172.21.15.189:0/628923152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0e7c06af00 0x7f0e7c06d3c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.721+0000 7f0ea00f2700 1 --2- 172.21.15.189:0/628923152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e980d11b0 0x7f0e98151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.721+0000 7f0ea00f2700 1 --2- 172.21.15.189:0/628923152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e980cb870 0x7f0e980b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.721+0000 7f0ea00f2700 1 --2- 172.21.15.189:0/628923152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e980cfa00 0x7f0e98158940 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.721+0000 7f0ea00f2700 1 -- 172.21.15.189:0/628923152 >> 172.21.15.189:0/628923152 conn(0x7f0e980b2db0 msgr2=0x7f0e980d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:40.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.721+0000 7f0ea00f2700 1 -- 172.21.15.189:0/628923152 shutdown_connections 2024-04-06T13:48:40.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.721+0000 7f0ea00f2700 1 -- 172.21.15.189:0/628923152 wait complete. 2024-04-06T13:48:40.732 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:48:40.735 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-04-06T13:48:39.931078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596341,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":710,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":449,"ondisk_log_size":449,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":288,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":457908,"kb_used_data":26032,"kb_used_omap":12,"kb_used_meta":431731,"kb_avail":754516812,"statfs":{"total":773094113280,"available":772625215488,"internally_reserved":0,"allocated":26656768,"data_stored":14242782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":442093112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3349,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.150915"},"pg_stats":[{"pgid":"34.1d","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.208437+0000","last_change":"2024-04-06T13:48:36.223675+0000","last_active":"2024-04-06T13:48:37.208437+0000","last_peered":"2024-04-06T13:48:37.208437+0000","last_clean":"2024-04-06T13:48:37.208437+0000","last_became_active":"2024-04-06T13:48:36.223406+0000","last_became_peered":"2024-04-06T13:48:36.223406+0000","last_unstale":"2024-04-06T13:48:37.208437+0000","last_undegraded":"2024-04-06T13:48:37.208437+0000","last_fullsized":"2024-04-06T13:48:37.208437+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:01:12.661900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"251'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204182+0000","last_change":"2024-04-06T13:48:35.801830+0000","last_active":"2024-04-06T13:48:38.204182+0000","last_peered":"2024-04-06T13:48:38.204182+0000","last_clean":"2024-04-06T13:48:38.204182+0000","last_became_active":"2024-04-06T13:48:35.801558+0000","last_became_peered":"2024-04-06T13:48:35.801558+0000","last_unstale":"2024-04-06T13:48:38.204182+0000","last_undegraded":"2024-04-06T13:48:38.204182+0000","last_fullsized":"2024-04-06T13:48:38.204182+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:35.037760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204196+0000","last_change":"2024-04-06T13:48:19.691215+0000","last_active":"2024-04-06T13:48:36.204196+0000","last_peered":"2024-04-06T13:48:36.204196+0000","last_clean":"2024-04-06T13:48:36.204196+0000","last_became_active":"2024-04-06T13:48:19.689693+0000","last_became_peered":"2024-04-06T13:48:19.689693+0000","last_unstale":"2024-04-06T13:48:36.204196+0000","last_undegraded":"2024-04-06T13:48:36.204196+0000","last_fullsized":"2024-04-06T13:48:36.204196+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:49:37.511498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204036+0000","last_change":"2024-04-06T13:48:36.216544+0000","last_active":"2024-04-06T13:48:38.204036+0000","last_peered":"2024-04-06T13:48:38.204036+0000","last_clean":"2024-04-06T13:48:38.204036+0000","last_became_active":"2024-04-06T13:48:36.216226+0000","last_became_peered":"2024-04-06T13:48:36.216226+0000","last_unstale":"2024-04-06T13:48:38.204036+0000","last_undegraded":"2024-04-06T13:48:38.204036+0000","last_fullsized":"2024-04-06T13:48:38.204036+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:41.402605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"251'4","reported_seq":15,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.208661+0000","last_change":"2024-04-06T13:48:35.801039+0000","last_active":"2024-04-06T13:48:37.208661+0000","last_peered":"2024-04-06T13:48:37.208661+0000","last_clean":"2024-04-06T13:48:37.208661+0000","last_became_active":"2024-04-06T13:48:35.800800+0000","last_became_peered":"2024-04-06T13:48:35.800800+0000","last_unstale":"2024-04-06T13:48:37.208661+0000","last_undegraded":"2024-04-06T13:48:37.208661+0000","last_fullsized":"2024-04-06T13:48:37.208661+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:35.037760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204220+0000","last_change":"2024-04-06T13:48:19.704688+0000","last_active":"2024-04-06T13:48:36.204220+0000","last_peered":"2024-04-06T13:48:36.204220+0000","last_clean":"2024-04-06T13:48:36.204220+0000","last_became_active":"2024-04-06T13:48:19.704088+0000","last_became_peered":"2024-04-06T13:48:19.704088+0000","last_unstale":"2024-04-06T13:48:36.204220+0000","last_undegraded":"2024-04-06T13:48:36.204220+0000","last_fullsized":"2024-04-06T13:48:36.204220+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:49.108653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205108+0000","last_change":"2024-04-06T13:48:36.226178+0000","last_active":"2024-04-06T13:48:38.205108+0000","last_peered":"2024-04-06T13:48:38.205108+0000","last_clean":"2024-04-06T13:48:38.205108+0000","last_became_active":"2024-04-06T13:48:36.225943+0000","last_became_peered":"2024-04-06T13:48:36.225943+0000","last_unstale":"2024-04-06T13:48:38.205108+0000","last_undegraded":"2024-04-06T13:48:38.205108+0000","last_fullsized":"2024-04-06T13:48:38.205108+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:05:37.001407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204208+0000","last_change":"2024-04-06T13:48:35.801138+0000","last_active":"2024-04-06T13:48:38.204208+0000","last_peered":"2024-04-06T13:48:38.204208+0000","last_clean":"2024-04-06T13:48:38.204208+0000","last_became_active":"2024-04-06T13:48:35.800885+0000","last_became_peered":"2024-04-06T13:48:35.800885+0000","last_unstale":"2024-04-06T13:48:38.204208+0000","last_undegraded":"2024-04-06T13:48:38.204208+0000","last_fullsized":"2024-04-06T13:48:38.204208+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:55:27.291993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206106+0000","last_change":"2024-04-06T13:48:19.697461+0000","last_active":"2024-04-06T13:48:36.206106+0000","last_peered":"2024-04-06T13:48:36.206106+0000","last_clean":"2024-04-06T13:48:36.206106+0000","last_became_active":"2024-04-06T13:48:19.695459+0000","last_became_peered":"2024-04-06T13:48:19.695459+0000","last_unstale":"2024-04-06T13:48:36.206106+0000","last_undegraded":"2024-04-06T13:48:36.206106+0000","last_fullsized":"2024-04-06T13:48:36.206106+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:42:12.990594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.208330+0000","last_change":"2024-04-06T13:48:36.213647+0000","last_active":"2024-04-06T13:48:37.208330+0000","last_peered":"2024-04-06T13:48:37.208330+0000","last_clean":"2024-04-06T13:48:37.208330+0000","last_became_active":"2024-04-06T13:48:36.213389+0000","last_became_peered":"2024-04-06T13:48:36.213389+0000","last_unstale":"2024-04-06T13:48:37.208330+0000","last_undegraded":"2024-04-06T13:48:37.208330+0000","last_fullsized":"2024-04-06T13:48:37.208330+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:32.688952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.206177+0000","last_change":"2024-04-06T13:48:35.804173+0000","last_active":"2024-04-06T13:48:38.206177+0000","last_peered":"2024-04-06T13:48:38.206177+0000","last_clean":"2024-04-06T13:48:38.206177+0000","last_became_active":"2024-04-06T13:48:35.803800+0000","last_became_peered":"2024-04-06T13:48:35.803800+0000","last_unstale":"2024-04-06T13:48:38.206177+0000","last_undegraded":"2024-04-06T13:48:38.206177+0000","last_fullsized":"2024-04-06T13:48:38.206177+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:38:28.055682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206402+0000","last_change":"2024-04-06T13:48:19.690589+0000","last_active":"2024-04-06T13:48:36.206402+0000","last_peered":"2024-04-06T13:48:36.206402+0000","last_clean":"2024-04-06T13:48:36.206402+0000","last_became_active":"2024-04-06T13:48:19.690338+0000","last_became_peered":"2024-04-06T13:48:19.690338+0000","last_unstale":"2024-04-06T13:48:36.206402+0000","last_undegraded":"2024-04-06T13:48:36.206402+0000","last_fullsized":"2024-04-06T13:48:36.206402+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:51:42.234400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204276+0000","last_change":"2024-04-06T13:48:36.218773+0000","last_active":"2024-04-06T13:48:38.204276+0000","last_peered":"2024-04-06T13:48:38.204276+0000","last_clean":"2024-04-06T13:48:38.204276+0000","last_became_active":"2024-04-06T13:48:36.218530+0000","last_became_peered":"2024-04-06T13:48:36.218530+0000","last_unstale":"2024-04-06T13:48:38.204276+0000","last_undegraded":"2024-04-06T13:48:38.204276+0000","last_fullsized":"2024-04-06T13:48:38.204276+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:16:51.992527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"251'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204961+0000","last_change":"2024-04-06T13:48:35.808825+0000","last_active":"2024-04-06T13:48:38.204961+0000","last_peered":"2024-04-06T13:48:38.204961+0000","last_clean":"2024-04-06T13:48:38.204961+0000","last_became_active":"2024-04-06T13:48:35.808542+0000","last_became_peered":"2024-04-06T13:48:35.808542+0000","last_unstale":"2024-04-06T13:48:38.204961+0000","last_undegraded":"2024-04-06T13:48:38.204961+0000","last_fullsized":"2024-04-06T13:48:38.204961+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:15:23.628075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.214984+0000","last_change":"2024-04-06T13:48:19.698891+0000","last_active":"2024-04-06T13:48:36.214984+0000","last_peered":"2024-04-06T13:48:36.214984+0000","last_clean":"2024-04-06T13:48:36.214984+0000","last_became_active":"2024-04-06T13:48:19.698477+0000","last_became_peered":"2024-04-06T13:48:19.698477+0000","last_unstale":"2024-04-06T13:48:36.214984+0000","last_undegraded":"2024-04-06T13:48:36.214984+0000","last_fullsized":"2024-04-06T13:48:36.214984+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:02:19.171219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203426+0000","last_change":"2024-04-06T13:48:36.215117+0000","last_active":"2024-04-06T13:48:38.203426+0000","last_peered":"2024-04-06T13:48:38.203426+0000","last_clean":"2024-04-06T13:48:38.203426+0000","last_became_active":"2024-04-06T13:48:36.214844+0000","last_became_peered":"2024-04-06T13:48:36.214844+0000","last_unstale":"2024-04-06T13:48:38.203426+0000","last_undegraded":"2024-04-06T13:48:38.203426+0000","last_fullsized":"2024-04-06T13:48:38.203426+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:00:12.108708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"251'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203983+0000","last_change":"2024-04-06T13:48:35.803442+0000","last_active":"2024-04-06T13:48:38.203983+0000","last_peered":"2024-04-06T13:48:38.203983+0000","last_clean":"2024-04-06T13:48:38.203983+0000","last_became_active":"2024-04-06T13:48:35.803182+0000","last_became_peered":"2024-04-06T13:48:35.803182+0000","last_unstale":"2024-04-06T13:48:38.203983+0000","last_undegraded":"2024-04-06T13:48:38.203983+0000","last_fullsized":"2024-04-06T13:48:38.203983+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:55:12.893835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.203677+0000","last_change":"2024-04-06T13:48:19.684151+0000","last_active":"2024-04-06T13:48:36.203677+0000","last_peered":"2024-04-06T13:48:36.203677+0000","last_clean":"2024-04-06T13:48:36.203677+0000","last_became_active":"2024-04-06T13:48:19.683897+0000","last_became_peered":"2024-04-06T13:48:19.683897+0000","last_unstale":"2024-04-06T13:48:36.203677+0000","last_undegraded":"2024-04-06T13:48:36.203677+0000","last_fullsized":"2024-04-06T13:48:36.203677+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:47.270406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212114+0000","last_change":"2024-04-06T13:48:36.211844+0000","last_active":"2024-04-06T13:48:37.212114+0000","last_peered":"2024-04-06T13:48:37.212114+0000","last_clean":"2024-04-06T13:48:37.212114+0000","last_became_active":"2024-04-06T13:48:36.210158+0000","last_became_peered":"2024-04-06T13:48:36.210158+0000","last_unstale":"2024-04-06T13:48:37.212114+0000","last_undegraded":"2024-04-06T13:48:37.212114+0000","last_fullsized":"2024-04-06T13:48:37.212114+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:16:24.874180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203419+0000","last_change":"2024-04-06T13:48:35.798263+0000","last_active":"2024-04-06T13:48:38.203419+0000","last_peered":"2024-04-06T13:48:38.203419+0000","last_clean":"2024-04-06T13:48:38.203419+0000","last_became_active":"2024-04-06T13:48:35.798010+0000","last_became_peered":"2024-04-06T13:48:35.798010+0000","last_unstale":"2024-04-06T13:48:38.203419+0000","last_undegraded":"2024-04-06T13:48:38.203419+0000","last_fullsized":"2024-04-06T13:48:38.203419+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:05:41.494271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.201592+0000","last_change":"2024-04-06T13:48:19.695665+0000","last_active":"2024-04-06T13:48:36.201592+0000","last_peered":"2024-04-06T13:48:36.201592+0000","last_clean":"2024-04-06T13:48:36.201592+0000","last_became_active":"2024-04-06T13:48:19.695362+0000","last_became_peered":"2024-04-06T13:48:19.695362+0000","last_unstale":"2024-04-06T13:48:36.201592+0000","last_undegraded":"2024-04-06T13:48:36.201592+0000","last_fullsized":"2024-04-06T13:48:36.201592+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:26:21.699641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203763+0000","last_change":"2024-04-06T13:48:36.224661+0000","last_active":"2024-04-06T13:48:38.203763+0000","last_peered":"2024-04-06T13:48:38.203763+0000","last_clean":"2024-04-06T13:48:38.203763+0000","last_became_active":"2024-04-06T13:48:36.220854+0000","last_became_peered":"2024-04-06T13:48:36.220854+0000","last_unstale":"2024-04-06T13:48:38.203763+0000","last_undegraded":"2024-04-06T13:48:38.203763+0000","last_fullsized":"2024-04-06T13:48:38.203763+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:38:22.774628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204518+0000","last_change":"2024-04-06T13:48:35.799944+0000","last_active":"2024-04-06T13:48:38.204518+0000","last_peered":"2024-04-06T13:48:38.204518+0000","last_clean":"2024-04-06T13:48:38.204518+0000","last_became_active":"2024-04-06T13:48:35.799661+0000","last_became_peered":"2024-04-06T13:48:35.799661+0000","last_unstale":"2024-04-06T13:48:38.204518+0000","last_undegraded":"2024-04-06T13:48:38.204518+0000","last_fullsized":"2024-04-06T13:48:38.204518+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:40:56.233528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213567+0000","last_change":"2024-04-06T13:48:19.704200+0000","last_active":"2024-04-06T13:48:36.213567+0000","last_peered":"2024-04-06T13:48:36.213567+0000","last_clean":"2024-04-06T13:48:36.213567+0000","last_became_active":"2024-04-06T13:48:19.703951+0000","last_became_peered":"2024-04-06T13:48:19.703951+0000","last_unstale":"2024-04-06T13:48:36.213567+0000","last_undegraded":"2024-04-06T13:48:36.213567+0000","last_fullsized":"2024-04-06T13:48:36.213567+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:31:20.020434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.208218+0000","last_change":"2024-04-06T13:48:36.221195+0000","last_active":"2024-04-06T13:48:37.208218+0000","last_peered":"2024-04-06T13:48:37.208218+0000","last_clean":"2024-04-06T13:48:37.208218+0000","last_became_active":"2024-04-06T13:48:36.220949+0000","last_became_peered":"2024-04-06T13:48:36.220949+0000","last_unstale":"2024-04-06T13:48:37.208218+0000","last_undegraded":"2024-04-06T13:48:37.208218+0000","last_fullsized":"2024-04-06T13:48:37.208218+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:41.402605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204287+0000","last_change":"2024-04-06T13:48:35.800025+0000","last_active":"2024-04-06T13:48:38.204287+0000","last_peered":"2024-04-06T13:48:38.204287+0000","last_clean":"2024-04-06T13:48:38.204287+0000","last_became_active":"2024-04-06T13:48:35.799760+0000","last_became_peered":"2024-04-06T13:48:35.799760+0000","last_unstale":"2024-04-06T13:48:38.204287+0000","last_undegraded":"2024-04-06T13:48:38.204287+0000","last_fullsized":"2024-04-06T13:48:38.204287+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:17:41.311632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204323+0000","last_change":"2024-04-06T13:48:19.705385+0000","last_active":"2024-04-06T13:48:36.204323+0000","last_peered":"2024-04-06T13:48:36.204323+0000","last_clean":"2024-04-06T13:48:36.204323+0000","last_became_active":"2024-04-06T13:48:19.705091+0000","last_became_peered":"2024-04-06T13:48:19.705091+0000","last_unstale":"2024-04-06T13:48:36.204323+0000","last_undegraded":"2024-04-06T13:48:36.204323+0000","last_fullsized":"2024-04-06T13:48:36.204323+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:23:39.345908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212151+0000","last_change":"2024-04-06T13:48:36.224819+0000","last_active":"2024-04-06T13:48:37.212151+0000","last_peered":"2024-04-06T13:48:37.212151+0000","last_clean":"2024-04-06T13:48:37.212151+0000","last_became_active":"2024-04-06T13:48:36.224546+0000","last_became_peered":"2024-04-06T13:48:36.224546+0000","last_unstale":"2024-04-06T13:48:37.212151+0000","last_undegraded":"2024-04-06T13:48:37.212151+0000","last_fullsized":"2024-04-06T13:48:37.212151+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:12:07.931133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212331+0000","last_change":"2024-04-06T13:48:35.808073+0000","last_active":"2024-04-06T13:48:37.212331+0000","last_peered":"2024-04-06T13:48:37.212331+0000","last_clean":"2024-04-06T13:48:37.212331+0000","last_became_active":"2024-04-06T13:48:35.807847+0000","last_became_peered":"2024-04-06T13:48:35.807847+0000","last_unstale":"2024-04-06T13:48:37.212331+0000","last_undegraded":"2024-04-06T13:48:37.212331+0000","last_fullsized":"2024-04-06T13:48:37.212331+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:26:50.278260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206598+0000","last_change":"2024-04-06T13:48:19.700300+0000","last_active":"2024-04-06T13:48:36.206598+0000","last_peered":"2024-04-06T13:48:36.206598+0000","last_clean":"2024-04-06T13:48:36.206598+0000","last_became_active":"2024-04-06T13:48:19.700054+0000","last_became_peered":"2024-04-06T13:48:19.700054+0000","last_unstale":"2024-04-06T13:48:36.206598+0000","last_undegraded":"2024-04-06T13:48:36.206598+0000","last_fullsized":"2024-04-06T13:48:36.206598+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:30:14.717681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.206279+0000","last_change":"2024-04-06T13:48:36.219005+0000","last_active":"2024-04-06T13:48:38.206279+0000","last_peered":"2024-04-06T13:48:38.206279+0000","last_clean":"2024-04-06T13:48:38.206279+0000","last_became_active":"2024-04-06T13:48:36.218703+0000","last_became_peered":"2024-04-06T13:48:36.218703+0000","last_unstale":"2024-04-06T13:48:38.206279+0000","last_undegraded":"2024-04-06T13:48:38.206279+0000","last_fullsized":"2024-04-06T13:48:38.206279+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:41:54.237434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"251'6","reported_seq":17,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.208150+0000","last_change":"2024-04-06T13:48:35.798075+0000","last_active":"2024-04-06T13:48:37.208150+0000","last_peered":"2024-04-06T13:48:37.208150+0000","last_clean":"2024-04-06T13:48:37.208150+0000","last_became_active":"2024-04-06T13:48:35.797577+0000","last_became_peered":"2024-04-06T13:48:35.797577+0000","last_unstale":"2024-04-06T13:48:37.208150+0000","last_undegraded":"2024-04-06T13:48:37.208150+0000","last_fullsized":"2024-04-06T13:48:37.208150+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:17:41.311632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206244+0000","last_change":"2024-04-06T13:48:19.700151+0000","last_active":"2024-04-06T13:48:36.206244+0000","last_peered":"2024-04-06T13:48:36.206244+0000","last_clean":"2024-04-06T13:48:36.206244+0000","last_became_active":"2024-04-06T13:48:19.699882+0000","last_became_peered":"2024-04-06T13:48:19.699882+0000","last_unstale":"2024-04-06T13:48:36.206244+0000","last_undegraded":"2024-04-06T13:48:36.206244+0000","last_fullsized":"2024-04-06T13:48:36.206244+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:24.879568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.213000+0000","last_change":"2024-04-06T13:48:36.225507+0000","last_active":"2024-04-06T13:48:37.213000+0000","last_peered":"2024-04-06T13:48:37.213000+0000","last_clean":"2024-04-06T13:48:37.213000+0000","last_became_active":"2024-04-06T13:48:36.225261+0000","last_became_peered":"2024-04-06T13:48:36.225261+0000","last_unstale":"2024-04-06T13:48:37.213000+0000","last_undegraded":"2024-04-06T13:48:37.213000+0000","last_fullsized":"2024-04-06T13:48:37.213000+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:24:22.367478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"251'3","reported_seq":14,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212628+0000","last_change":"2024-04-06T13:48:35.808132+0000","last_active":"2024-04-06T13:48:37.212628+0000","last_peered":"2024-04-06T13:48:37.212628+0000","last_clean":"2024-04-06T13:48:37.212628+0000","last_became_active":"2024-04-06T13:48:35.807646+0000","last_became_peered":"2024-04-06T13:48:35.807646+0000","last_unstale":"2024-04-06T13:48:37.212628+0000","last_undegraded":"2024-04-06T13:48:37.212628+0000","last_fullsized":"2024-04-06T13:48:37.212628+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:35:18.103406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.203137+0000","last_change":"2024-04-06T13:48:19.683194+0000","last_active":"2024-04-06T13:48:36.203137+0000","last_peered":"2024-04-06T13:48:36.203137+0000","last_clean":"2024-04-06T13:48:36.203137+0000","last_became_active":"2024-04-06T13:48:19.682953+0000","last_became_peered":"2024-04-06T13:48:19.682953+0000","last_unstale":"2024-04-06T13:48:36.203137+0000","last_undegraded":"2024-04-06T13:48:36.203137+0000","last_fullsized":"2024-04-06T13:48:36.203137+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:33:52.351082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205355+0000","last_change":"2024-04-06T13:48:36.220815+0000","last_active":"2024-04-06T13:48:38.205355+0000","last_peered":"2024-04-06T13:48:38.205355+0000","last_clean":"2024-04-06T13:48:38.205355+0000","last_became_active":"2024-04-06T13:48:36.220545+0000","last_became_peered":"2024-04-06T13:48:36.220545+0000","last_unstale":"2024-04-06T13:48:38.205355+0000","last_undegraded":"2024-04-06T13:48:38.205355+0000","last_fullsized":"2024-04-06T13:48:38.205355+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:00:42.061268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.211883+0000","last_change":"2024-04-06T13:48:35.807074+0000","last_active":"2024-04-06T13:48:37.211883+0000","last_peered":"2024-04-06T13:48:37.211883+0000","last_clean":"2024-04-06T13:48:37.211883+0000","last_became_active":"2024-04-06T13:48:35.806811+0000","last_became_peered":"2024-04-06T13:48:35.806811+0000","last_unstale":"2024-04-06T13:48:37.211883+0000","last_undegraded":"2024-04-06T13:48:37.211883+0000","last_fullsized":"2024-04-06T13:48:37.211883+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:28:11.587938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212271+0000","last_change":"2024-04-06T13:48:19.690929+0000","last_active":"2024-04-06T13:48:36.212271+0000","last_peered":"2024-04-06T13:48:36.212271+0000","last_clean":"2024-04-06T13:48:36.212271+0000","last_became_active":"2024-04-06T13:48:19.690663+0000","last_became_peered":"2024-04-06T13:48:19.690663+0000","last_unstale":"2024-04-06T13:48:36.212271+0000","last_undegraded":"2024-04-06T13:48:36.212271+0000","last_fullsized":"2024-04-06T13:48:36.212271+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:52:01.901954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203338+0000","last_change":"2024-04-06T13:48:36.217030+0000","last_active":"2024-04-06T13:48:38.203338+0000","last_peered":"2024-04-06T13:48:38.203338+0000","last_clean":"2024-04-06T13:48:38.203338+0000","last_became_active":"2024-04-06T13:48:36.216760+0000","last_became_peered":"2024-04-06T13:48:36.216760+0000","last_unstale":"2024-04-06T13:48:38.203338+0000","last_undegraded":"2024-04-06T13:48:38.203338+0000","last_fullsized":"2024-04-06T13:48:38.203338+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:15:24.634356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204288+0000","last_change":"2024-04-06T13:48:35.800985+0000","last_active":"2024-04-06T13:48:38.204288+0000","last_peered":"2024-04-06T13:48:38.204288+0000","last_clean":"2024-04-06T13:48:38.204288+0000","last_became_active":"2024-04-06T13:48:35.800742+0000","last_became_peered":"2024-04-06T13:48:35.800742+0000","last_unstale":"2024-04-06T13:48:38.204288+0000","last_undegraded":"2024-04-06T13:48:38.204288+0000","last_fullsized":"2024-04-06T13:48:38.204288+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:56:50.591253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.207498+0000","last_change":"2024-04-06T13:48:19.693212+0000","last_active":"2024-04-06T13:48:36.207498+0000","last_peered":"2024-04-06T13:48:36.207498+0000","last_clean":"2024-04-06T13:48:36.207498+0000","last_became_active":"2024-04-06T13:48:19.692957+0000","last_became_peered":"2024-04-06T13:48:19.692957+0000","last_unstale":"2024-04-06T13:48:36.207498+0000","last_undegraded":"2024-04-06T13:48:36.207498+0000","last_fullsized":"2024-04-06T13:48:36.207498+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:00:25.538231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203624+0000","last_change":"2024-04-06T13:48:36.220425+0000","last_active":"2024-04-06T13:48:38.203624+0000","last_peered":"2024-04-06T13:48:38.203624+0000","last_clean":"2024-04-06T13:48:38.203624+0000","last_became_active":"2024-04-06T13:48:36.219730+0000","last_became_peered":"2024-04-06T13:48:36.219730+0000","last_unstale":"2024-04-06T13:48:38.203624+0000","last_undegraded":"2024-04-06T13:48:38.203624+0000","last_fullsized":"2024-04-06T13:48:38.203624+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:41:38.433651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"251'8","reported_seq":19,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.211712+0000","last_change":"2024-04-06T13:48:35.806208+0000","last_active":"2024-04-06T13:48:37.211712+0000","last_peered":"2024-04-06T13:48:37.211712+0000","last_clean":"2024-04-06T13:48:37.211712+0000","last_became_active":"2024-04-06T13:48:35.805798+0000","last_became_peered":"2024-04-06T13:48:35.805798+0000","last_unstale":"2024-04-06T13:48:37.211712+0000","last_undegraded":"2024-04-06T13:48:37.211712+0000","last_fullsized":"2024-04-06T13:48:37.211712+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:43:51.499864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206417+0000","last_change":"2024-04-06T13:48:19.693508+0000","last_active":"2024-04-06T13:48:36.206417+0000","last_peered":"2024-04-06T13:48:36.206417+0000","last_clean":"2024-04-06T13:48:36.206417+0000","last_became_active":"2024-04-06T13:48:19.693250+0000","last_became_peered":"2024-04-06T13:48:19.693250+0000","last_unstale":"2024-04-06T13:48:36.206417+0000","last_undegraded":"2024-04-06T13:48:36.206417+0000","last_fullsized":"2024-04-06T13:48:36.206417+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:00:56.138863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203462+0000","last_change":"2024-04-06T13:48:36.218751+0000","last_active":"2024-04-06T13:48:38.203462+0000","last_peered":"2024-04-06T13:48:38.203462+0000","last_clean":"2024-04-06T13:48:38.203462+0000","last_became_active":"2024-04-06T13:48:36.217434+0000","last_became_peered":"2024-04-06T13:48:36.217434+0000","last_unstale":"2024-04-06T13:48:38.203462+0000","last_undegraded":"2024-04-06T13:48:38.203462+0000","last_fullsized":"2024-04-06T13:48:38.203462+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:49:54.034535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.211807+0000","last_change":"2024-04-06T13:48:35.803827+0000","last_active":"2024-04-06T13:48:37.211807+0000","last_peered":"2024-04-06T13:48:37.211807+0000","last_clean":"2024-04-06T13:48:37.211807+0000","last_became_active":"2024-04-06T13:48:35.803457+0000","last_became_peered":"2024-04-06T13:48:35.803457+0000","last_unstale":"2024-04-06T13:48:37.211807+0000","last_undegraded":"2024-04-06T13:48:37.211807+0000","last_fullsized":"2024-04-06T13:48:37.211807+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:50:30.176310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.205584+0000","last_change":"2024-04-06T13:48:19.700696+0000","last_active":"2024-04-06T13:48:36.205584+0000","last_peered":"2024-04-06T13:48:36.205584+0000","last_clean":"2024-04-06T13:48:36.205584+0000","last_became_active":"2024-04-06T13:48:19.700444+0000","last_became_peered":"2024-04-06T13:48:19.700444+0000","last_unstale":"2024-04-06T13:48:36.205584+0000","last_undegraded":"2024-04-06T13:48:36.205584+0000","last_fullsized":"2024-04-06T13:48:36.205584+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:46.094428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.206066+0000","last_change":"2024-04-06T13:48:36.226075+0000","last_active":"2024-04-06T13:48:38.206066+0000","last_peered":"2024-04-06T13:48:38.206066+0000","last_clean":"2024-04-06T13:48:38.206066+0000","last_became_active":"2024-04-06T13:48:36.225818+0000","last_became_peered":"2024-04-06T13:48:36.225818+0000","last_unstale":"2024-04-06T13:48:38.206066+0000","last_undegraded":"2024-04-06T13:48:38.206066+0000","last_fullsized":"2024-04-06T13:48:38.206066+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:06:46.118211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.208572+0000","last_change":"2024-04-06T13:48:35.800512+0000","last_active":"2024-04-06T13:48:37.208572+0000","last_peered":"2024-04-06T13:48:37.208572+0000","last_clean":"2024-04-06T13:48:37.208572+0000","last_became_active":"2024-04-06T13:48:35.799815+0000","last_became_peered":"2024-04-06T13:48:35.799815+0000","last_unstale":"2024-04-06T13:48:37.208572+0000","last_undegraded":"2024-04-06T13:48:37.208572+0000","last_fullsized":"2024-04-06T13:48:37.208572+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:55:27.291993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213820+0000","last_change":"2024-04-06T13:48:19.700988+0000","last_active":"2024-04-06T13:48:36.213820+0000","last_peered":"2024-04-06T13:48:36.213820+0000","last_clean":"2024-04-06T13:48:36.213820+0000","last_became_active":"2024-04-06T13:48:19.700718+0000","last_became_peered":"2024-04-06T13:48:19.700718+0000","last_unstale":"2024-04-06T13:48:36.213820+0000","last_undegraded":"2024-04-06T13:48:36.213820+0000","last_fullsized":"2024-04-06T13:48:36.213820+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:37:50.946459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205901+0000","last_change":"2024-04-06T13:48:36.218918+0000","last_active":"2024-04-06T13:48:38.205901+0000","last_peered":"2024-04-06T13:48:38.205901+0000","last_clean":"2024-04-06T13:48:38.205901+0000","last_became_active":"2024-04-06T13:48:36.218625+0000","last_became_peered":"2024-04-06T13:48:36.218625+0000","last_unstale":"2024-04-06T13:48:38.205901+0000","last_undegraded":"2024-04-06T13:48:38.205901+0000","last_fullsized":"2024-04-06T13:48:38.205901+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:38:01.256020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"251'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.213136+0000","last_change":"2024-04-06T13:48:35.804317+0000","last_active":"2024-04-06T13:48:37.213136+0000","last_peered":"2024-04-06T13:48:37.213136+0000","last_clean":"2024-04-06T13:48:37.213136+0000","last_became_active":"2024-04-06T13:48:35.804011+0000","last_became_peered":"2024-04-06T13:48:35.804011+0000","last_unstale":"2024-04-06T13:48:37.213136+0000","last_undegraded":"2024-04-06T13:48:37.213136+0000","last_fullsized":"2024-04-06T13:48:37.213136+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:02:56.857425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206649+0000","last_change":"2024-04-06T13:48:19.700420+0000","last_active":"2024-04-06T13:48:36.206649+0000","last_peered":"2024-04-06T13:48:36.206649+0000","last_clean":"2024-04-06T13:48:36.206649+0000","last_became_active":"2024-04-06T13:48:19.698705+0000","last_became_peered":"2024-04-06T13:48:19.698705+0000","last_unstale":"2024-04-06T13:48:36.206649+0000","last_undegraded":"2024-04-06T13:48:36.206649+0000","last_fullsized":"2024-04-06T13:48:36.206649+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:29:58.555526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.206191+0000","last_change":"2024-04-06T13:48:36.220887+0000","last_active":"2024-04-06T13:48:38.206191+0000","last_peered":"2024-04-06T13:48:38.206191+0000","last_clean":"2024-04-06T13:48:38.206191+0000","last_became_active":"2024-04-06T13:48:36.220628+0000","last_became_peered":"2024-04-06T13:48:36.220628+0000","last_unstale":"2024-04-06T13:48:38.206191+0000","last_undegraded":"2024-04-06T13:48:38.206191+0000","last_fullsized":"2024-04-06T13:48:38.206191+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:50:20.772122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.207927+0000","last_change":"2024-04-06T13:48:35.800121+0000","last_active":"2024-04-06T13:48:37.207927+0000","last_peered":"2024-04-06T13:48:37.207927+0000","last_clean":"2024-04-06T13:48:37.207927+0000","last_became_active":"2024-04-06T13:48:35.799487+0000","last_became_peered":"2024-04-06T13:48:35.799487+0000","last_unstale":"2024-04-06T13:48:37.207927+0000","last_undegraded":"2024-04-06T13:48:37.207927+0000","last_fullsized":"2024-04-06T13:48:37.207927+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:31:41.226066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.207264+0000","last_change":"2024-04-06T13:48:19.695591+0000","last_active":"2024-04-06T13:48:36.207264+0000","last_peered":"2024-04-06T13:48:36.207264+0000","last_clean":"2024-04-06T13:48:36.207264+0000","last_became_active":"2024-04-06T13:48:19.695289+0000","last_became_peered":"2024-04-06T13:48:19.695289+0000","last_unstale":"2024-04-06T13:48:36.207264+0000","last_undegraded":"2024-04-06T13:48:36.207264+0000","last_fullsized":"2024-04-06T13:48:36.207264+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:16.165915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212010+0000","last_change":"2024-04-06T13:48:36.218655+0000","last_active":"2024-04-06T13:48:37.212010+0000","last_peered":"2024-04-06T13:48:37.212010+0000","last_clean":"2024-04-06T13:48:37.212010+0000","last_became_active":"2024-04-06T13:48:36.218402+0000","last_became_peered":"2024-04-06T13:48:36.218402+0000","last_unstale":"2024-04-06T13:48:37.212010+0000","last_undegraded":"2024-04-06T13:48:37.212010+0000","last_fullsized":"2024-04-06T13:48:37.212010+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:43:51.718388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"251'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204574+0000","last_change":"2024-04-06T13:48:35.805712+0000","last_active":"2024-04-06T13:48:38.204574+0000","last_peered":"2024-04-06T13:48:38.204574+0000","last_clean":"2024-04-06T13:48:38.204574+0000","last_became_active":"2024-04-06T13:48:35.804958+0000","last_became_peered":"2024-04-06T13:48:35.804958+0000","last_unstale":"2024-04-06T13:48:38.204574+0000","last_undegraded":"2024-04-06T13:48:38.204574+0000","last_fullsized":"2024-04-06T13:48:38.204574+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:31.682672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.203754+0000","last_change":"2024-04-06T13:48:19.702407+0000","last_active":"2024-04-06T13:48:36.203754+0000","last_peered":"2024-04-06T13:48:36.203754+0000","last_clean":"2024-04-06T13:48:36.203754+0000","last_became_active":"2024-04-06T13:48:19.702139+0000","last_became_peered":"2024-04-06T13:48:19.702139+0000","last_unstale":"2024-04-06T13:48:36.203754+0000","last_undegraded":"2024-04-06T13:48:36.203754+0000","last_fullsized":"2024-04-06T13:48:36.203754+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:57:33.923465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205476+0000","last_change":"2024-04-06T13:48:36.220266+0000","last_active":"2024-04-06T13:48:38.205476+0000","last_peered":"2024-04-06T13:48:38.205476+0000","last_clean":"2024-04-06T13:48:38.205476+0000","last_became_active":"2024-04-06T13:48:36.220004+0000","last_became_peered":"2024-04-06T13:48:36.220004+0000","last_unstale":"2024-04-06T13:48:38.205476+0000","last_undegraded":"2024-04-06T13:48:38.205476+0000","last_fullsized":"2024-04-06T13:48:38.205476+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:24:40.860500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204908+0000","last_change":"2024-04-06T13:48:35.802080+0000","last_active":"2024-04-06T13:48:38.204908+0000","last_peered":"2024-04-06T13:48:38.204908+0000","last_clean":"2024-04-06T13:48:38.204908+0000","last_became_active":"2024-04-06T13:48:35.801820+0000","last_became_peered":"2024-04-06T13:48:35.801820+0000","last_unstale":"2024-04-06T13:48:38.204908+0000","last_undegraded":"2024-04-06T13:48:38.204908+0000","last_fullsized":"2024-04-06T13:48:38.204908+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:36:48.849028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.200902+0000","last_change":"2024-04-06T13:48:19.692817+0000","last_active":"2024-04-06T13:48:36.200902+0000","last_peered":"2024-04-06T13:48:36.200902+0000","last_clean":"2024-04-06T13:48:36.200902+0000","last_became_active":"2024-04-06T13:48:19.692567+0000","last_became_peered":"2024-04-06T13:48:19.692567+0000","last_unstale":"2024-04-06T13:48:36.200902+0000","last_undegraded":"2024-04-06T13:48:36.200902+0000","last_fullsized":"2024-04-06T13:48:36.200902+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:22.432067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203845+0000","last_change":"2024-04-06T13:48:36.227364+0000","last_active":"2024-04-06T13:48:38.203845+0000","last_peered":"2024-04-06T13:48:38.203845+0000","last_clean":"2024-04-06T13:48:38.203845+0000","last_became_active":"2024-04-06T13:48:36.227165+0000","last_became_peered":"2024-04-06T13:48:36.227165+0000","last_unstale":"2024-04-06T13:48:38.203845+0000","last_undegraded":"2024-04-06T13:48:38.203845+0000","last_fullsized":"2024-04-06T13:48:38.203845+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:55:53.565152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204189+0000","last_change":"2024-04-06T13:48:35.809018+0000","last_active":"2024-04-06T13:48:38.204189+0000","last_peered":"2024-04-06T13:48:38.204189+0000","last_clean":"2024-04-06T13:48:38.204189+0000","last_became_active":"2024-04-06T13:48:35.808774+0000","last_became_peered":"2024-04-06T13:48:35.808774+0000","last_unstale":"2024-04-06T13:48:38.204189+0000","last_undegraded":"2024-04-06T13:48:38.204189+0000","last_fullsized":"2024-04-06T13:48:38.204189+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:58:35.871449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212065+0000","last_change":"2024-04-06T13:48:19.688698+0000","last_active":"2024-04-06T13:48:36.212065+0000","last_peered":"2024-04-06T13:48:36.212065+0000","last_clean":"2024-04-06T13:48:36.212065+0000","last_became_active":"2024-04-06T13:48:19.688377+0000","last_became_peered":"2024-04-06T13:48:19.688377+0000","last_unstale":"2024-04-06T13:48:36.212065+0000","last_undegraded":"2024-04-06T13:48:36.212065+0000","last_fullsized":"2024-04-06T13:48:36.212065+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:05:22.432067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204432+0000","last_change":"2024-04-06T13:48:36.225373+0000","last_active":"2024-04-06T13:48:38.204432+0000","last_peered":"2024-04-06T13:48:38.204432+0000","last_clean":"2024-04-06T13:48:38.204432+0000","last_became_active":"2024-04-06T13:48:36.220506+0000","last_became_peered":"2024-04-06T13:48:36.220506+0000","last_unstale":"2024-04-06T13:48:38.204432+0000","last_undegraded":"2024-04-06T13:48:38.204432+0000","last_fullsized":"2024-04-06T13:48:38.204432+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:34:44.870588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.211789+0000","last_change":"2024-04-06T13:48:35.806027+0000","last_active":"2024-04-06T13:48:37.211789+0000","last_peered":"2024-04-06T13:48:37.211789+0000","last_clean":"2024-04-06T13:48:37.211789+0000","last_became_active":"2024-04-06T13:48:35.804529+0000","last_became_peered":"2024-04-06T13:48:35.804529+0000","last_unstale":"2024-04-06T13:48:37.211789+0000","last_undegraded":"2024-04-06T13:48:37.211789+0000","last_fullsized":"2024-04-06T13:48:37.211789+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:28:01.756284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206334+0000","last_change":"2024-04-06T13:48:19.693370+0000","last_active":"2024-04-06T13:48:36.206334+0000","last_peered":"2024-04-06T13:48:36.206334+0000","last_clean":"2024-04-06T13:48:36.206334+0000","last_became_active":"2024-04-06T13:48:19.692893+0000","last_became_peered":"2024-04-06T13:48:19.692893+0000","last_unstale":"2024-04-06T13:48:36.206334+0000","last_undegraded":"2024-04-06T13:48:36.206334+0000","last_fullsized":"2024-04-06T13:48:36.206334+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:24.879568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.211799+0000","last_change":"2024-04-06T13:48:36.211404+0000","last_active":"2024-04-06T13:48:37.211799+0000","last_peered":"2024-04-06T13:48:37.211799+0000","last_clean":"2024-04-06T13:48:37.211799+0000","last_became_active":"2024-04-06T13:48:36.211170+0000","last_became_peered":"2024-04-06T13:48:36.211170+0000","last_unstale":"2024-04-06T13:48:37.211799+0000","last_undegraded":"2024-04-06T13:48:37.211799+0000","last_fullsized":"2024-04-06T13:48:37.211799+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:25:46.842981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204236+0000","last_change":"2024-04-06T13:48:35.802498+0000","last_active":"2024-04-06T13:48:38.204236+0000","last_peered":"2024-04-06T13:48:38.204236+0000","last_clean":"2024-04-06T13:48:38.204236+0000","last_became_active":"2024-04-06T13:48:35.801896+0000","last_became_peered":"2024-04-06T13:48:35.801896+0000","last_unstale":"2024-04-06T13:48:38.204236+0000","last_undegraded":"2024-04-06T13:48:38.204236+0000","last_fullsized":"2024-04-06T13:48:38.204236+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:58:02.787162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212571+0000","last_change":"2024-04-06T13:48:19.698243+0000","last_active":"2024-04-06T13:48:36.212571+0000","last_peered":"2024-04-06T13:48:36.212571+0000","last_clean":"2024-04-06T13:48:36.212571+0000","last_became_active":"2024-04-06T13:48:19.697956+0000","last_became_peered":"2024-04-06T13:48:19.697956+0000","last_unstale":"2024-04-06T13:48:36.212571+0000","last_undegraded":"2024-04-06T13:48:36.212571+0000","last_fullsized":"2024-04-06T13:48:36.212571+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:26:21.699641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212570+0000","last_change":"2024-04-06T13:48:36.213811+0000","last_active":"2024-04-06T13:48:37.212570+0000","last_peered":"2024-04-06T13:48:37.212570+0000","last_clean":"2024-04-06T13:48:37.212570+0000","last_became_active":"2024-04-06T13:48:36.213507+0000","last_became_peered":"2024-04-06T13:48:36.213507+0000","last_unstale":"2024-04-06T13:48:37.212570+0000","last_undegraded":"2024-04-06T13:48:37.212570+0000","last_fullsized":"2024-04-06T13:48:37.212570+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:34.972744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.208483+0000","last_change":"2024-04-06T13:48:35.800677+0000","last_active":"2024-04-06T13:48:37.208483+0000","last_peered":"2024-04-06T13:48:37.208483+0000","last_clean":"2024-04-06T13:48:37.208483+0000","last_became_active":"2024-04-06T13:48:35.800240+0000","last_became_peered":"2024-04-06T13:48:35.800240+0000","last_unstale":"2024-04-06T13:48:37.208483+0000","last_undegraded":"2024-04-06T13:48:37.208483+0000","last_fullsized":"2024-04-06T13:48:37.208483+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:40:56.233528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.205136+0000","last_change":"2024-04-06T13:48:19.694103+0000","last_active":"2024-04-06T13:48:36.205136+0000","last_peered":"2024-04-06T13:48:36.205136+0000","last_clean":"2024-04-06T13:48:36.205136+0000","last_became_active":"2024-04-06T13:48:19.693643+0000","last_became_peered":"2024-04-06T13:48:19.693643+0000","last_unstale":"2024-04-06T13:48:36.205136+0000","last_undegraded":"2024-04-06T13:48:36.205136+0000","last_fullsized":"2024-04-06T13:48:36.205136+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:00:35.709952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212674+0000","last_change":"2024-04-06T13:48:36.227862+0000","last_active":"2024-04-06T13:48:37.212674+0000","last_peered":"2024-04-06T13:48:37.212674+0000","last_clean":"2024-04-06T13:48:37.212674+0000","last_became_active":"2024-04-06T13:48:36.226969+0000","last_became_peered":"2024-04-06T13:48:36.226969+0000","last_unstale":"2024-04-06T13:48:37.212674+0000","last_undegraded":"2024-04-06T13:48:37.212674+0000","last_fullsized":"2024-04-06T13:48:37.212674+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:31:03.909867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"251'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212588+0000","last_change":"2024-04-06T13:48:35.807813+0000","last_active":"2024-04-06T13:48:37.212588+0000","last_peered":"2024-04-06T13:48:37.212588+0000","last_clean":"2024-04-06T13:48:37.212588+0000","last_became_active":"2024-04-06T13:48:35.807534+0000","last_became_peered":"2024-04-06T13:48:35.807534+0000","last_unstale":"2024-04-06T13:48:37.212588+0000","last_undegraded":"2024-04-06T13:48:37.212588+0000","last_fullsized":"2024-04-06T13:48:37.212588+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:05:16.392498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.202812+0000","last_change":"2024-04-06T13:48:19.697522+0000","last_active":"2024-04-06T13:48:36.202812+0000","last_peered":"2024-04-06T13:48:36.202812+0000","last_clean":"2024-04-06T13:48:36.202812+0000","last_became_active":"2024-04-06T13:48:19.695505+0000","last_became_peered":"2024-04-06T13:48:19.695505+0000","last_unstale":"2024-04-06T13:48:36.202812+0000","last_undegraded":"2024-04-06T13:48:36.202812+0000","last_fullsized":"2024-04-06T13:48:36.202812+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:37:50.946459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.213080+0000","last_change":"2024-04-06T13:48:36.223153+0000","last_active":"2024-04-06T13:48:37.213080+0000","last_peered":"2024-04-06T13:48:37.213080+0000","last_clean":"2024-04-06T13:48:37.213080+0000","last_became_active":"2024-04-06T13:48:36.222855+0000","last_became_peered":"2024-04-06T13:48:36.222855+0000","last_unstale":"2024-04-06T13:48:37.213080+0000","last_undegraded":"2024-04-06T13:48:37.213080+0000","last_fullsized":"2024-04-06T13:48:37.213080+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:50:08.995699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204589+0000","last_change":"2024-04-06T13:48:35.802506+0000","last_active":"2024-04-06T13:48:38.204589+0000","last_peered":"2024-04-06T13:48:38.204589+0000","last_clean":"2024-04-06T13:48:38.204589+0000","last_became_active":"2024-04-06T13:48:35.802227+0000","last_became_peered":"2024-04-06T13:48:35.802227+0000","last_unstale":"2024-04-06T13:48:38.204589+0000","last_undegraded":"2024-04-06T13:48:38.204589+0000","last_fullsized":"2024-04-06T13:48:38.204589+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:01:11.655619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213149+0000","last_change":"2024-04-06T13:48:19.699786+0000","last_active":"2024-04-06T13:48:36.213149+0000","last_peered":"2024-04-06T13:48:36.213149+0000","last_clean":"2024-04-06T13:48:36.213149+0000","last_became_active":"2024-04-06T13:48:19.698719+0000","last_became_peered":"2024-04-06T13:48:19.698719+0000","last_unstale":"2024-04-06T13:48:36.213149+0000","last_undegraded":"2024-04-06T13:48:36.213149+0000","last_fullsized":"2024-04-06T13:48:36.213149+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:54:26.958761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203992+0000","last_change":"2024-04-06T13:48:36.221117+0000","last_active":"2024-04-06T13:48:38.203992+0000","last_peered":"2024-04-06T13:48:38.203992+0000","last_clean":"2024-04-06T13:48:38.203992+0000","last_became_active":"2024-04-06T13:48:36.219036+0000","last_became_peered":"2024-04-06T13:48:36.219036+0000","last_unstale":"2024-04-06T13:48:38.203992+0000","last_undegraded":"2024-04-06T13:48:38.203992+0000","last_fullsized":"2024-04-06T13:48:38.203992+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:24:40.860500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203784+0000","last_change":"2024-04-06T13:48:35.804169+0000","last_active":"2024-04-06T13:48:38.203784+0000","last_peered":"2024-04-06T13:48:38.203784+0000","last_clean":"2024-04-06T13:48:38.203784+0000","last_became_active":"2024-04-06T13:48:35.803914+0000","last_became_peered":"2024-04-06T13:48:35.803914+0000","last_unstale":"2024-04-06T13:48:38.203784+0000","last_undegraded":"2024-04-06T13:48:38.203784+0000","last_fullsized":"2024-04-06T13:48:38.203784+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:42:28.507350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.203178+0000","last_change":"2024-04-06T13:48:19.704381+0000","last_active":"2024-04-06T13:48:36.203178+0000","last_peered":"2024-04-06T13:48:36.203178+0000","last_clean":"2024-04-06T13:48:36.203178+0000","last_became_active":"2024-04-06T13:48:19.704140+0000","last_became_peered":"2024-04-06T13:48:19.704140+0000","last_unstale":"2024-04-06T13:48:36.203178+0000","last_undegraded":"2024-04-06T13:48:36.203178+0000","last_fullsized":"2024-04-06T13:48:36.203178+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:30:47.386830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204158+0000","last_change":"2024-04-06T13:48:36.229581+0000","last_active":"2024-04-06T13:48:38.204158+0000","last_peered":"2024-04-06T13:48:38.204158+0000","last_clean":"2024-04-06T13:48:38.204158+0000","last_became_active":"2024-04-06T13:48:36.229260+0000","last_became_peered":"2024-04-06T13:48:36.229260+0000","last_unstale":"2024-04-06T13:48:38.204158+0000","last_undegraded":"2024-04-06T13:48:38.204158+0000","last_fullsized":"2024-04-06T13:48:38.204158+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:15:06.292012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.207690+0000","last_change":"2024-04-06T13:48:35.810188+0000","last_active":"2024-04-06T13:48:37.207690+0000","last_peered":"2024-04-06T13:48:37.207690+0000","last_clean":"2024-04-06T13:48:37.207690+0000","last_became_active":"2024-04-06T13:48:35.809890+0000","last_became_peered":"2024-04-06T13:48:35.809890+0000","last_unstale":"2024-04-06T13:48:37.207690+0000","last_undegraded":"2024-04-06T13:48:37.207690+0000","last_fullsized":"2024-04-06T13:48:37.207690+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:38:28.055682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.208700+0000","last_change":"2024-04-06T13:48:19.694038+0000","last_active":"2024-04-06T13:48:36.208700+0000","last_peered":"2024-04-06T13:48:36.208700+0000","last_clean":"2024-04-06T13:48:36.208700+0000","last_became_active":"2024-04-06T13:48:19.693775+0000","last_became_peered":"2024-04-06T13:48:19.693775+0000","last_unstale":"2024-04-06T13:48:36.208700+0000","last_undegraded":"2024-04-06T13:48:36.208700+0000","last_fullsized":"2024-04-06T13:48:36.208700+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:38:06.251591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.211631+0000","last_change":"2024-04-06T13:48:36.217070+0000","last_active":"2024-04-06T13:48:37.211631+0000","last_peered":"2024-04-06T13:48:37.211631+0000","last_clean":"2024-04-06T13:48:37.211631+0000","last_became_active":"2024-04-06T13:48:36.216807+0000","last_became_peered":"2024-04-06T13:48:36.216807+0000","last_unstale":"2024-04-06T13:48:37.211631+0000","last_undegraded":"2024-04-06T13:48:37.211631+0000","last_fullsized":"2024-04-06T13:48:37.211631+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:54:08.390619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.208024+0000","last_change":"2024-04-06T13:48:35.799922+0000","last_active":"2024-04-06T13:48:37.208024+0000","last_peered":"2024-04-06T13:48:37.208024+0000","last_clean":"2024-04-06T13:48:37.208024+0000","last_became_active":"2024-04-06T13:48:35.799654+0000","last_became_peered":"2024-04-06T13:48:35.799654+0000","last_unstale":"2024-04-06T13:48:37.208024+0000","last_undegraded":"2024-04-06T13:48:37.208024+0000","last_fullsized":"2024-04-06T13:48:37.208024+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:57:01.308231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.207573+0000","last_change":"2024-04-06T13:48:19.699860+0000","last_active":"2024-04-06T13:48:36.207573+0000","last_peered":"2024-04-06T13:48:36.207573+0000","last_clean":"2024-04-06T13:48:36.207573+0000","last_became_active":"2024-04-06T13:48:19.699567+0000","last_became_peered":"2024-04-06T13:48:19.699567+0000","last_unstale":"2024-04-06T13:48:36.207573+0000","last_undegraded":"2024-04-06T13:48:36.207573+0000","last_fullsized":"2024-04-06T13:48:36.207573+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:01:05.087706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204718+0000","last_change":"2024-04-06T13:48:36.215179+0000","last_active":"2024-04-06T13:48:38.204718+0000","last_peered":"2024-04-06T13:48:38.204718+0000","last_clean":"2024-04-06T13:48:38.204718+0000","last_became_active":"2024-04-06T13:48:36.213976+0000","last_became_peered":"2024-04-06T13:48:36.213976+0000","last_unstale":"2024-04-06T13:48:38.204718+0000","last_undegraded":"2024-04-06T13:48:38.204718+0000","last_fullsized":"2024-04-06T13:48:38.204718+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:42:29.513631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.207773+0000","last_change":"2024-04-06T13:48:35.797269+0000","last_active":"2024-04-06T13:48:37.207773+0000","last_peered":"2024-04-06T13:48:37.207773+0000","last_clean":"2024-04-06T13:48:37.207773+0000","last_became_active":"2024-04-06T13:48:35.796938+0000","last_became_peered":"2024-04-06T13:48:35.796938+0000","last_unstale":"2024-04-06T13:48:37.207773+0000","last_undegraded":"2024-04-06T13:48:37.207773+0000","last_fullsized":"2024-04-06T13:48:37.207773+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:47:01.611185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.203008+0000","last_change":"2024-04-06T13:48:19.697603+0000","last_active":"2024-04-06T13:48:36.203008+0000","last_peered":"2024-04-06T13:48:36.203008+0000","last_clean":"2024-04-06T13:48:36.203008+0000","last_became_active":"2024-04-06T13:48:19.697356+0000","last_became_peered":"2024-04-06T13:48:19.697356+0000","last_unstale":"2024-04-06T13:48:36.203008+0000","last_undegraded":"2024-04-06T13:48:36.203008+0000","last_fullsized":"2024-04-06T13:48:36.203008+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:44.773715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204757+0000","last_change":"2024-04-06T13:48:36.214926+0000","last_active":"2024-04-06T13:48:38.204757+0000","last_peered":"2024-04-06T13:48:38.204757+0000","last_clean":"2024-04-06T13:48:38.204757+0000","last_became_active":"2024-04-06T13:48:36.213877+0000","last_became_peered":"2024-04-06T13:48:36.213877+0000","last_unstale":"2024-04-06T13:48:38.204757+0000","last_undegraded":"2024-04-06T13:48:38.204757+0000","last_fullsized":"2024-04-06T13:48:38.204757+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:14:55.492477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204092+0000","last_change":"2024-04-06T13:48:35.805462+0000","last_active":"2024-04-06T13:48:38.204092+0000","last_peered":"2024-04-06T13:48:38.204092+0000","last_clean":"2024-04-06T13:48:38.204092+0000","last_became_active":"2024-04-06T13:48:35.805205+0000","last_became_peered":"2024-04-06T13:48:35.805205+0000","last_unstale":"2024-04-06T13:48:38.204092+0000","last_undegraded":"2024-04-06T13:48:38.204092+0000","last_fullsized":"2024-04-06T13:48:38.204092+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:34.773522+0000","last_clean_scrub_stamp":"2024-04-06T13:48:34.773522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:40.396325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.202358+0000","last_change":"2024-04-06T13:48:19.705129+0000","last_active":"2024-04-06T13:48:36.202358+0000","last_peered":"2024-04-06T13:48:36.202358+0000","last_clean":"2024-04-06T13:48:36.202358+0000","last_became_active":"2024-04-06T13:48:19.704722+0000","last_became_peered":"2024-04-06T13:48:19.704722+0000","last_unstale":"2024-04-06T13:48:36.202358+0000","last_undegraded":"2024-04-06T13:48:36.202358+0000","last_fullsized":"2024-04-06T13:48:36.202358+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:18.449707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212458+0000","last_change":"2024-04-06T13:48:36.225691+0000","last_active":"2024-04-06T13:48:37.212458+0000","last_peered":"2024-04-06T13:48:37.212458+0000","last_clean":"2024-04-06T13:48:37.212458+0000","last_became_active":"2024-04-06T13:48:36.225409+0000","last_became_peered":"2024-04-06T13:48:36.225409+0000","last_unstale":"2024-04-06T13:48:37.212458+0000","last_undegraded":"2024-04-06T13:48:37.212458+0000","last_fullsized":"2024-04-06T13:48:37.212458+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:36:41.699885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204692+0000","last_change":"2024-04-06T13:48:19.696446+0000","last_active":"2024-04-06T13:48:36.204692+0000","last_peered":"2024-04-06T13:48:36.204692+0000","last_clean":"2024-04-06T13:48:36.204692+0000","last_became_active":"2024-04-06T13:48:19.696090+0000","last_became_peered":"2024-04-06T13:48:19.696090+0000","last_unstale":"2024-04-06T13:48:36.204692+0000","last_undegraded":"2024-04-06T13:48:36.204692+0000","last_fullsized":"2024-04-06T13:48:36.204692+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:35.461214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"247'4","reported_seq":19,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204118+0000","last_change":"2024-04-06T13:48:19.281764+0000","last_active":"2024-04-06T13:48:36.204118+0000","last_peered":"2024-04-06T13:48:36.204118+0000","last_clean":"2024-04-06T13:48:36.204118+0000","last_became_active":"2024-04-06T13:48:19.281501+0000","last_became_peered":"2024-04-06T13:48:19.281501+0000","last_unstale":"2024-04-06T13:48:36.204118+0000","last_undegraded":"2024-04-06T13:48:36.204118+0000","last_fullsized":"2024-04-06T13:48:36.204118+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:54:56.369118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203369+0000","last_change":"2024-04-06T13:48:36.226891+0000","last_active":"2024-04-06T13:48:38.203369+0000","last_peered":"2024-04-06T13:48:38.203369+0000","last_clean":"2024-04-06T13:48:38.203369+0000","last_became_active":"2024-04-06T13:48:36.220372+0000","last_became_peered":"2024-04-06T13:48:36.220372+0000","last_unstale":"2024-04-06T13:48:38.203369+0000","last_undegraded":"2024-04-06T13:48:38.203369+0000","last_fullsized":"2024-04-06T13:48:38.203369+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:19:31.844777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206141+0000","last_change":"2024-04-06T13:48:19.699483+0000","last_active":"2024-04-06T13:48:36.206141+0000","last_peered":"2024-04-06T13:48:36.206141+0000","last_clean":"2024-04-06T13:48:36.206141+0000","last_became_active":"2024-04-06T13:48:19.699209+0000","last_became_peered":"2024-04-06T13:48:19.699209+0000","last_unstale":"2024-04-06T13:48:36.206141+0000","last_undegraded":"2024-04-06T13:48:36.206141+0000","last_fullsized":"2024-04-06T13:48:36.206141+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:42:12.990594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.214449+0000","last_change":"2024-04-06T13:48:19.280859+0000","last_active":"2024-04-06T13:48:36.214449+0000","last_peered":"2024-04-06T13:48:36.214449+0000","last_clean":"2024-04-06T13:48:36.214449+0000","last_became_active":"2024-04-06T13:48:19.280512+0000","last_became_peered":"2024-04-06T13:48:19.280512+0000","last_unstale":"2024-04-06T13:48:36.214449+0000","last_undegraded":"2024-04-06T13:48:36.214449+0000","last_fullsized":"2024-04-06T13:48:36.214449+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:43.765754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204376+0000","last_change":"2024-04-06T13:48:36.217497+0000","last_active":"2024-04-06T13:48:38.204376+0000","last_peered":"2024-04-06T13:48:38.204376+0000","last_clean":"2024-04-06T13:48:38.204376+0000","last_became_active":"2024-04-06T13:48:36.217261+0000","last_became_peered":"2024-04-06T13:48:36.217261+0000","last_unstale":"2024-04-06T13:48:38.204376+0000","last_undegraded":"2024-04-06T13:48:38.204376+0000","last_fullsized":"2024-04-06T13:48:38.204376+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:32:13.503864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.202674+0000","last_change":"2024-04-06T13:48:19.687508+0000","last_active":"2024-04-06T13:48:36.202674+0000","last_peered":"2024-04-06T13:48:36.202674+0000","last_clean":"2024-04-06T13:48:36.202674+0000","last_became_active":"2024-04-06T13:48:19.687246+0000","last_became_peered":"2024-04-06T13:48:19.687246+0000","last_unstale":"2024-04-06T13:48:36.202674+0000","last_undegraded":"2024-04-06T13:48:36.202674+0000","last_fullsized":"2024-04-06T13:48:36.202674+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:15:08.111319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206548+0000","last_change":"2024-04-06T13:48:19.280934+0000","last_active":"2024-04-06T13:48:36.206548+0000","last_peered":"2024-04-06T13:48:36.206548+0000","last_clean":"2024-04-06T13:48:36.206548+0000","last_became_active":"2024-04-06T13:48:19.280677+0000","last_became_peered":"2024-04-06T13:48:19.280677+0000","last_unstale":"2024-04-06T13:48:36.206548+0000","last_undegraded":"2024-04-06T13:48:36.206548+0000","last_fullsized":"2024-04-06T13:48:36.206548+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:30:23.776677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204905+0000","last_change":"2024-04-06T13:48:36.225949+0000","last_active":"2024-04-06T13:48:38.204905+0000","last_peered":"2024-04-06T13:48:38.204905+0000","last_clean":"2024-04-06T13:48:38.204905+0000","last_became_active":"2024-04-06T13:48:36.225375+0000","last_became_peered":"2024-04-06T13:48:36.225375+0000","last_unstale":"2024-04-06T13:48:38.204905+0000","last_undegraded":"2024-04-06T13:48:38.204905+0000","last_fullsized":"2024-04-06T13:48:38.204905+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:17:18.829437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206018+0000","last_change":"2024-04-06T13:48:19.695081+0000","last_active":"2024-04-06T13:48:36.206018+0000","last_peered":"2024-04-06T13:48:36.206018+0000","last_clean":"2024-04-06T13:48:36.206018+0000","last_became_active":"2024-04-06T13:48:19.694838+0000","last_became_peered":"2024-04-06T13:48:19.694838+0000","last_unstale":"2024-04-06T13:48:36.206018+0000","last_undegraded":"2024-04-06T13:48:36.206018+0000","last_fullsized":"2024-04-06T13:48:36.206018+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:37:44.732983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206452+0000","last_change":"2024-04-06T13:48:19.282906+0000","last_active":"2024-04-06T13:48:36.206452+0000","last_peered":"2024-04-06T13:48:36.206452+0000","last_clean":"2024-04-06T13:48:36.206452+0000","last_became_active":"2024-04-06T13:48:19.282628+0000","last_became_peered":"2024-04-06T13:48:19.282628+0000","last_unstale":"2024-04-06T13:48:36.206452+0000","last_undegraded":"2024-04-06T13:48:36.206452+0000","last_fullsized":"2024-04-06T13:48:36.206452+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:18.513042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204073+0000","last_change":"2024-04-06T13:48:36.221507+0000","last_active":"2024-04-06T13:48:38.204073+0000","last_peered":"2024-04-06T13:48:38.204073+0000","last_clean":"2024-04-06T13:48:38.204073+0000","last_became_active":"2024-04-06T13:48:36.221248+0000","last_became_peered":"2024-04-06T13:48:36.221248+0000","last_unstale":"2024-04-06T13:48:38.204073+0000","last_undegraded":"2024-04-06T13:48:38.204073+0000","last_fullsized":"2024-04-06T13:48:38.204073+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:06:46.118211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.202326+0000","last_change":"2024-04-06T13:48:19.694881+0000","last_active":"2024-04-06T13:48:36.202326+0000","last_peered":"2024-04-06T13:48:36.202326+0000","last_clean":"2024-04-06T13:48:36.202326+0000","last_became_active":"2024-04-06T13:48:19.693362+0000","last_became_peered":"2024-04-06T13:48:19.693362+0000","last_unstale":"2024-04-06T13:48:36.202326+0000","last_undegraded":"2024-04-06T13:48:36.202326+0000","last_fullsized":"2024-04-06T13:48:36.202326+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:36:47.917157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.207654+0000","last_change":"2024-04-06T13:48:19.274234+0000","last_active":"2024-04-06T13:48:36.207654+0000","last_peered":"2024-04-06T13:48:36.207654+0000","last_clean":"2024-04-06T13:48:36.207654+0000","last_became_active":"2024-04-06T13:48:19.273998+0000","last_became_peered":"2024-04-06T13:48:19.273998+0000","last_unstale":"2024-04-06T13:48:36.207654+0000","last_undegraded":"2024-04-06T13:48:36.207654+0000","last_fullsized":"2024-04-06T13:48:36.207654+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:49:23.791293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205696+0000","last_change":"2024-04-06T13:48:36.229400+0000","last_active":"2024-04-06T13:48:38.205696+0000","last_peered":"2024-04-06T13:48:38.205696+0000","last_clean":"2024-04-06T13:48:38.205696+0000","last_became_active":"2024-04-06T13:48:36.229163+0000","last_became_peered":"2024-04-06T13:48:36.229163+0000","last_unstale":"2024-04-06T13:48:38.205696+0000","last_undegraded":"2024-04-06T13:48:38.205696+0000","last_fullsized":"2024-04-06T13:48:38.205696+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:01:21.610743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206538+0000","last_change":"2024-04-06T13:48:19.694934+0000","last_active":"2024-04-06T13:48:36.206538+0000","last_peered":"2024-04-06T13:48:36.206538+0000","last_clean":"2024-04-06T13:48:36.206538+0000","last_became_active":"2024-04-06T13:48:19.692682+0000","last_became_peered":"2024-04-06T13:48:19.692682+0000","last_unstale":"2024-04-06T13:48:36.206538+0000","last_undegraded":"2024-04-06T13:48:36.206538+0000","last_fullsized":"2024-04-06T13:48:36.206538+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:40:43.027987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.205235+0000","last_change":"2024-04-06T13:48:19.279719+0000","last_active":"2024-04-06T13:48:36.205235+0000","last_peered":"2024-04-06T13:48:36.205235+0000","last_clean":"2024-04-06T13:48:36.205235+0000","last_became_active":"2024-04-06T13:48:19.279404+0000","last_became_peered":"2024-04-06T13:48:19.279404+0000","last_unstale":"2024-04-06T13:48:36.205235+0000","last_undegraded":"2024-04-06T13:48:36.205235+0000","last_fullsized":"2024-04-06T13:48:36.205235+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:42:35.927858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205956+0000","last_change":"2024-04-06T13:48:36.230010+0000","last_active":"2024-04-06T13:48:38.205956+0000","last_peered":"2024-04-06T13:48:38.205956+0000","last_clean":"2024-04-06T13:48:38.205956+0000","last_became_active":"2024-04-06T13:48:36.229745+0000","last_became_peered":"2024-04-06T13:48:36.229745+0000","last_unstale":"2024-04-06T13:48:38.205956+0000","last_undegraded":"2024-04-06T13:48:38.205956+0000","last_fullsized":"2024-04-06T13:48:38.205956+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:40:41.412534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212903+0000","last_change":"2024-04-06T13:48:19.702348+0000","last_active":"2024-04-06T13:48:36.212903+0000","last_peered":"2024-04-06T13:48:36.212903+0000","last_clean":"2024-04-06T13:48:36.212903+0000","last_became_active":"2024-04-06T13:48:19.701891+0000","last_became_peered":"2024-04-06T13:48:19.701891+0000","last_unstale":"2024-04-06T13:48:36.212903+0000","last_undegraded":"2024-04-06T13:48:36.212903+0000","last_fullsized":"2024-04-06T13:48:36.212903+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:52:52.810060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.202536+0000","last_change":"2024-04-06T13:48:19.281924+0000","last_active":"2024-04-06T13:48:36.202536+0000","last_peered":"2024-04-06T13:48:36.202536+0000","last_clean":"2024-04-06T13:48:36.202536+0000","last_became_active":"2024-04-06T13:48:19.281609+0000","last_became_peered":"2024-04-06T13:48:19.281609+0000","last_unstale":"2024-04-06T13:48:36.202536+0000","last_undegraded":"2024-04-06T13:48:36.202536+0000","last_fullsized":"2024-04-06T13:48:36.202536+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:02:13.092175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205733+0000","last_change":"2024-04-06T13:48:36.229390+0000","last_active":"2024-04-06T13:48:38.205733+0000","last_peered":"2024-04-06T13:48:38.205733+0000","last_clean":"2024-04-06T13:48:38.205733+0000","last_became_active":"2024-04-06T13:48:36.229111+0000","last_became_peered":"2024-04-06T13:48:36.229111+0000","last_unstale":"2024-04-06T13:48:38.205733+0000","last_undegraded":"2024-04-06T13:48:38.205733+0000","last_fullsized":"2024-04-06T13:48:38.205733+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:38:22.774628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.201909+0000","last_change":"2024-04-06T13:48:19.694595+0000","last_active":"2024-04-06T13:48:36.201909+0000","last_peered":"2024-04-06T13:48:36.201909+0000","last_clean":"2024-04-06T13:48:36.201909+0000","last_became_active":"2024-04-06T13:48:19.692548+0000","last_became_peered":"2024-04-06T13:48:19.692548+0000","last_unstale":"2024-04-06T13:48:36.201909+0000","last_undegraded":"2024-04-06T13:48:36.201909+0000","last_fullsized":"2024-04-06T13:48:36.201909+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:39:01.587642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.208178+0000","last_change":"2024-04-06T13:48:19.273745+0000","last_active":"2024-04-06T13:48:36.208178+0000","last_peered":"2024-04-06T13:48:36.208178+0000","last_clean":"2024-04-06T13:48:36.208178+0000","last_became_active":"2024-04-06T13:48:19.273478+0000","last_became_peered":"2024-04-06T13:48:19.273478+0000","last_unstale":"2024-04-06T13:48:36.208178+0000","last_undegraded":"2024-04-06T13:48:36.208178+0000","last_fullsized":"2024-04-06T13:48:36.208178+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:00:55.130902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203553+0000","last_change":"2024-04-06T13:48:36.220579+0000","last_active":"2024-04-06T13:48:38.203553+0000","last_peered":"2024-04-06T13:48:38.203553+0000","last_clean":"2024-04-06T13:48:38.203553+0000","last_became_active":"2024-04-06T13:48:36.220002+0000","last_became_peered":"2024-04-06T13:48:36.220002+0000","last_unstale":"2024-04-06T13:48:38.203553+0000","last_undegraded":"2024-04-06T13:48:38.203553+0000","last_fullsized":"2024-04-06T13:48:38.203553+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:34.972744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212322+0000","last_change":"2024-04-06T13:48:19.688682+0000","last_active":"2024-04-06T13:48:36.212322+0000","last_peered":"2024-04-06T13:48:36.212322+0000","last_clean":"2024-04-06T13:48:36.212322+0000","last_became_active":"2024-04-06T13:48:19.688397+0000","last_became_peered":"2024-04-06T13:48:19.688397+0000","last_unstale":"2024-04-06T13:48:36.212322+0000","last_undegraded":"2024-04-06T13:48:36.212322+0000","last_fullsized":"2024-04-06T13:48:36.212322+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:39:01.587642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206720+0000","last_change":"2024-04-06T13:48:19.278056+0000","last_active":"2024-04-06T13:48:36.206720+0000","last_peered":"2024-04-06T13:48:36.206720+0000","last_clean":"2024-04-06T13:48:36.206720+0000","last_became_active":"2024-04-06T13:48:19.277796+0000","last_became_peered":"2024-04-06T13:48:19.277796+0000","last_unstale":"2024-04-06T13:48:36.206720+0000","last_undegraded":"2024-04-06T13:48:36.206720+0000","last_fullsized":"2024-04-06T13:48:36.206720+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:31:43.320734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205161+0000","last_change":"2024-04-06T13:48:36.223212+0000","last_active":"2024-04-06T13:48:38.205161+0000","last_peered":"2024-04-06T13:48:38.205161+0000","last_clean":"2024-04-06T13:48:38.205161+0000","last_became_active":"2024-04-06T13:48:36.222954+0000","last_became_peered":"2024-04-06T13:48:36.222954+0000","last_unstale":"2024-04-06T13:48:38.205161+0000","last_undegraded":"2024-04-06T13:48:38.205161+0000","last_fullsized":"2024-04-06T13:48:38.205161+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:34:38.126512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.205572+0000","last_change":"2024-04-06T13:48:19.700325+0000","last_active":"2024-04-06T13:48:36.205572+0000","last_peered":"2024-04-06T13:48:36.205572+0000","last_clean":"2024-04-06T13:48:36.205572+0000","last_became_active":"2024-04-06T13:48:19.698751+0000","last_became_peered":"2024-04-06T13:48:19.698751+0000","last_unstale":"2024-04-06T13:48:36.205572+0000","last_undegraded":"2024-04-06T13:48:36.205572+0000","last_fullsized":"2024-04-06T13:48:36.205572+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:16.165915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"247'3","reported_seq":18,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213341+0000","last_change":"2024-04-06T13:48:19.283170+0000","last_active":"2024-04-06T13:48:36.213341+0000","last_peered":"2024-04-06T13:48:36.213341+0000","last_clean":"2024-04-06T13:48:36.213341+0000","last_became_active":"2024-04-06T13:48:19.282896+0000","last_became_peered":"2024-04-06T13:48:19.282896+0000","last_unstale":"2024-04-06T13:48:36.213341+0000","last_undegraded":"2024-04-06T13:48:36.213341+0000","last_fullsized":"2024-04-06T13:48:36.213341+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:06:05.488014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204391+0000","last_change":"2024-04-06T13:48:36.225295+0000","last_active":"2024-04-06T13:48:38.204391+0000","last_peered":"2024-04-06T13:48:38.204391+0000","last_clean":"2024-04-06T13:48:38.204391+0000","last_became_active":"2024-04-06T13:48:36.220340+0000","last_became_peered":"2024-04-06T13:48:36.220340+0000","last_unstale":"2024-04-06T13:48:38.204391+0000","last_undegraded":"2024-04-06T13:48:38.204391+0000","last_fullsized":"2024-04-06T13:48:38.204391+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:14:04.534874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.200896+0000","last_change":"2024-04-06T13:48:19.692164+0000","last_active":"2024-04-06T13:48:36.200896+0000","last_peered":"2024-04-06T13:48:36.200896+0000","last_clean":"2024-04-06T13:48:36.200896+0000","last_became_active":"2024-04-06T13:48:19.691910+0000","last_became_peered":"2024-04-06T13:48:19.691910+0000","last_unstale":"2024-04-06T13:48:36.200896+0000","last_undegraded":"2024-04-06T13:48:36.200896+0000","last_fullsized":"2024-04-06T13:48:36.200896+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:19:27.875509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"248'8","reported_seq":23,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.203561+0000","last_change":"2024-04-06T13:48:19.281276+0000","last_active":"2024-04-06T13:48:36.203561+0000","last_peered":"2024-04-06T13:48:36.203561+0000","last_clean":"2024-04-06T13:48:36.203561+0000","last_became_active":"2024-04-06T13:48:19.280081+0000","last_became_peered":"2024-04-06T13:48:19.280081+0000","last_unstale":"2024-04-06T13:48:36.203561+0000","last_undegraded":"2024-04-06T13:48:36.203561+0000","last_fullsized":"2024-04-06T13:48:36.203561+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:56:55.759595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2813,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.208268+0000","last_change":"2024-04-06T13:48:36.220050+0000","last_active":"2024-04-06T13:48:37.208268+0000","last_peered":"2024-04-06T13:48:37.208268+0000","last_clean":"2024-04-06T13:48:37.208268+0000","last_became_active":"2024-04-06T13:48:36.219803+0000","last_became_peered":"2024-04-06T13:48:36.219803+0000","last_unstale":"2024-04-06T13:48:37.208268+0000","last_undegraded":"2024-04-06T13:48:37.208268+0000","last_fullsized":"2024-04-06T13:48:37.208268+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:42:29.513631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.203704+0000","last_change":"2024-04-06T13:48:19.693858+0000","last_active":"2024-04-06T13:48:36.203704+0000","last_peered":"2024-04-06T13:48:36.203704+0000","last_clean":"2024-04-06T13:48:36.203704+0000","last_became_active":"2024-04-06T13:48:19.693515+0000","last_became_peered":"2024-04-06T13:48:19.693515+0000","last_unstale":"2024-04-06T13:48:36.203704+0000","last_undegraded":"2024-04-06T13:48:36.203704+0000","last_fullsized":"2024-04-06T13:48:36.203704+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:10:26.852793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206543+0000","last_change":"2024-04-06T13:48:19.283816+0000","last_active":"2024-04-06T13:48:36.206543+0000","last_peered":"2024-04-06T13:48:36.206543+0000","last_clean":"2024-04-06T13:48:36.206543+0000","last_became_active":"2024-04-06T13:48:19.283543+0000","last_became_peered":"2024-04-06T13:48:19.283543+0000","last_unstale":"2024-04-06T13:48:36.206543+0000","last_undegraded":"2024-04-06T13:48:36.206543+0000","last_fullsized":"2024-04-06T13:48:36.206543+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:31:24.701348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204733+0000","last_change":"2024-04-06T13:48:36.211366+0000","last_active":"2024-04-06T13:48:38.204733+0000","last_peered":"2024-04-06T13:48:38.204733+0000","last_clean":"2024-04-06T13:48:38.204733+0000","last_became_active":"2024-04-06T13:48:36.211180+0000","last_became_peered":"2024-04-06T13:48:36.211180+0000","last_unstale":"2024-04-06T13:48:38.204733+0000","last_undegraded":"2024-04-06T13:48:38.204733+0000","last_fullsized":"2024-04-06T13:48:38.204733+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:04:27.070776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.202245+0000","last_change":"2024-04-06T13:48:19.694858+0000","last_active":"2024-04-06T13:48:36.202245+0000","last_peered":"2024-04-06T13:48:36.202245+0000","last_clean":"2024-04-06T13:48:36.202245+0000","last_became_active":"2024-04-06T13:48:19.693049+0000","last_became_peered":"2024-04-06T13:48:19.693049+0000","last_unstale":"2024-04-06T13:48:36.202245+0000","last_undegraded":"2024-04-06T13:48:36.202245+0000","last_fullsized":"2024-04-06T13:48:36.202245+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:19:00.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213239+0000","last_change":"2024-04-06T13:48:19.282855+0000","last_active":"2024-04-06T13:48:36.213239+0000","last_peered":"2024-04-06T13:48:36.213239+0000","last_clean":"2024-04-06T13:48:36.213239+0000","last_became_active":"2024-04-06T13:48:19.282570+0000","last_became_peered":"2024-04-06T13:48:19.282570+0000","last_unstale":"2024-04-06T13:48:36.213239+0000","last_undegraded":"2024-04-06T13:48:36.213239+0000","last_fullsized":"2024-04-06T13:48:36.213239+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:35:17.934513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212985+0000","last_change":"2024-04-06T13:48:36.227966+0000","last_active":"2024-04-06T13:48:37.212985+0000","last_peered":"2024-04-06T13:48:37.212985+0000","last_clean":"2024-04-06T13:48:37.212985+0000","last_became_active":"2024-04-06T13:48:36.227294+0000","last_became_peered":"2024-04-06T13:48:36.227294+0000","last_unstale":"2024-04-06T13:48:37.212985+0000","last_undegraded":"2024-04-06T13:48:37.212985+0000","last_fullsized":"2024-04-06T13:48:37.212985+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:23:55.868945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.208874+0000","last_change":"2024-04-06T13:48:19.693208+0000","last_active":"2024-04-06T13:48:36.208874+0000","last_peered":"2024-04-06T13:48:36.208874+0000","last_clean":"2024-04-06T13:48:36.208874+0000","last_became_active":"2024-04-06T13:48:19.692939+0000","last_became_peered":"2024-04-06T13:48:19.692939+0000","last_unstale":"2024-04-06T13:48:36.208874+0000","last_undegraded":"2024-04-06T13:48:36.208874+0000","last_fullsized":"2024-04-06T13:48:36.208874+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:46:07.210444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212994+0000","last_change":"2024-04-06T13:48:19.283720+0000","last_active":"2024-04-06T13:48:36.212994+0000","last_peered":"2024-04-06T13:48:36.212994+0000","last_clean":"2024-04-06T13:48:36.212994+0000","last_became_active":"2024-04-06T13:48:19.283453+0000","last_became_peered":"2024-04-06T13:48:19.283453+0000","last_unstale":"2024-04-06T13:48:36.212994+0000","last_undegraded":"2024-04-06T13:48:36.212994+0000","last_fullsized":"2024-04-06T13:48:36.212994+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:02:23.692226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204247+0000","last_change":"2024-04-06T13:48:36.222844+0000","last_active":"2024-04-06T13:48:38.204247+0000","last_peered":"2024-04-06T13:48:38.204247+0000","last_clean":"2024-04-06T13:48:38.204247+0000","last_became_active":"2024-04-06T13:48:36.222573+0000","last_became_peered":"2024-04-06T13:48:36.222573+0000","last_unstale":"2024-04-06T13:48:38.204247+0000","last_undegraded":"2024-04-06T13:48:38.204247+0000","last_fullsized":"2024-04-06T13:48:38.204247+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:46:23.733481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212817+0000","last_change":"2024-04-06T13:48:19.700890+0000","last_active":"2024-04-06T13:48:36.212817+0000","last_peered":"2024-04-06T13:48:36.212817+0000","last_clean":"2024-04-06T13:48:36.212817+0000","last_became_active":"2024-04-06T13:48:19.700513+0000","last_became_peered":"2024-04-06T13:48:19.700513+0000","last_unstale":"2024-04-06T13:48:36.212817+0000","last_undegraded":"2024-04-06T13:48:36.212817+0000","last_fullsized":"2024-04-06T13:48:36.212817+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:35:45.830191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"247'8","reported_seq":23,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204608+0000","last_change":"2024-04-06T13:48:19.274645+0000","last_active":"2024-04-06T13:48:36.204608+0000","last_peered":"2024-04-06T13:48:36.204608+0000","last_clean":"2024-04-06T13:48:36.204608+0000","last_became_active":"2024-04-06T13:48:19.274308+0000","last_became_peered":"2024-04-06T13:48:19.274308+0000","last_unstale":"2024-04-06T13:48:36.204608+0000","last_undegraded":"2024-04-06T13:48:36.204608+0000","last_fullsized":"2024-04-06T13:48:36.204608+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:31:15.493970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205094+0000","last_change":"2024-04-06T13:48:36.214621+0000","last_active":"2024-04-06T13:48:38.205094+0000","last_peered":"2024-04-06T13:48:38.205094+0000","last_clean":"2024-04-06T13:48:38.205094+0000","last_became_active":"2024-04-06T13:48:36.213123+0000","last_became_peered":"2024-04-06T13:48:36.213123+0000","last_unstale":"2024-04-06T13:48:38.205094+0000","last_undegraded":"2024-04-06T13:48:38.205094+0000","last_fullsized":"2024-04-06T13:48:38.205094+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:39:13.811002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213492+0000","last_change":"2024-04-06T13:48:19.700124+0000","last_active":"2024-04-06T13:48:36.213492+0000","last_peered":"2024-04-06T13:48:36.213492+0000","last_clean":"2024-04-06T13:48:36.213492+0000","last_became_active":"2024-04-06T13:48:19.699857+0000","last_became_peered":"2024-04-06T13:48:19.699857+0000","last_unstale":"2024-04-06T13:48:36.213492+0000","last_undegraded":"2024-04-06T13:48:36.213492+0000","last_fullsized":"2024-04-06T13:48:36.213492+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:46:44.773715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204041+0000","last_change":"2024-04-06T13:48:19.284301+0000","last_active":"2024-04-06T13:48:36.204041+0000","last_peered":"2024-04-06T13:48:36.204041+0000","last_clean":"2024-04-06T13:48:36.204041+0000","last_became_active":"2024-04-06T13:48:19.284062+0000","last_became_peered":"2024-04-06T13:48:19.284062+0000","last_unstale":"2024-04-06T13:48:36.204041+0000","last_undegraded":"2024-04-06T13:48:36.204041+0000","last_fullsized":"2024-04-06T13:48:36.204041+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:49.041735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212045+0000","last_change":"2024-04-06T13:48:36.211538+0000","last_active":"2024-04-06T13:48:37.212045+0000","last_peered":"2024-04-06T13:48:37.212045+0000","last_clean":"2024-04-06T13:48:37.212045+0000","last_became_active":"2024-04-06T13:48:36.210725+0000","last_became_peered":"2024-04-06T13:48:36.210725+0000","last_unstale":"2024-04-06T13:48:37.212045+0000","last_undegraded":"2024-04-06T13:48:37.212045+0000","last_fullsized":"2024-04-06T13:48:37.212045+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:00:05.559228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212880+0000","last_change":"2024-04-06T13:48:19.702274+0000","last_active":"2024-04-06T13:48:36.212880+0000","last_peered":"2024-04-06T13:48:36.212880+0000","last_clean":"2024-04-06T13:48:36.212880+0000","last_became_active":"2024-04-06T13:48:19.701964+0000","last_became_peered":"2024-04-06T13:48:19.701964+0000","last_unstale":"2024-04-06T13:48:36.212880+0000","last_undegraded":"2024-04-06T13:48:36.212880+0000","last_fullsized":"2024-04-06T13:48:36.212880+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:46:45.240549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204097+0000","last_change":"2024-04-06T13:48:19.275494+0000","last_active":"2024-04-06T13:48:36.204097+0000","last_peered":"2024-04-06T13:48:36.204097+0000","last_clean":"2024-04-06T13:48:36.204097+0000","last_became_active":"2024-04-06T13:48:19.275194+0000","last_became_peered":"2024-04-06T13:48:19.275194+0000","last_unstale":"2024-04-06T13:48:36.204097+0000","last_undegraded":"2024-04-06T13:48:36.204097+0000","last_fullsized":"2024-04-06T13:48:36.204097+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:11:29.081483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204870+0000","last_change":"2024-04-06T13:48:36.216455+0000","last_active":"2024-04-06T13:48:38.204870+0000","last_peered":"2024-04-06T13:48:38.204870+0000","last_clean":"2024-04-06T13:48:38.204870+0000","last_became_active":"2024-04-06T13:48:36.215926+0000","last_became_peered":"2024-04-06T13:48:36.215926+0000","last_unstale":"2024-04-06T13:48:38.204870+0000","last_undegraded":"2024-04-06T13:48:38.204870+0000","last_fullsized":"2024-04-06T13:48:38.204870+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:32.688952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.203455+0000","last_change":"2024-04-06T13:48:19.702505+0000","last_active":"2024-04-06T13:48:36.203455+0000","last_peered":"2024-04-06T13:48:36.203455+0000","last_clean":"2024-04-06T13:48:36.203455+0000","last_became_active":"2024-04-06T13:48:19.702245+0000","last_became_peered":"2024-04-06T13:48:19.702245+0000","last_unstale":"2024-04-06T13:48:36.203455+0000","last_undegraded":"2024-04-06T13:48:36.203455+0000","last_fullsized":"2024-04-06T13:48:36.203455+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:45.672315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.214556+0000","last_change":"2024-04-06T13:48:19.281905+0000","last_active":"2024-04-06T13:48:36.214556+0000","last_peered":"2024-04-06T13:48:36.214556+0000","last_clean":"2024-04-06T13:48:36.214556+0000","last_became_active":"2024-04-06T13:48:19.281052+0000","last_became_peered":"2024-04-06T13:48:19.281052+0000","last_unstale":"2024-04-06T13:48:36.214556+0000","last_undegraded":"2024-04-06T13:48:36.214556+0000","last_fullsized":"2024-04-06T13:48:36.214556+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:48:48.100692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203933+0000","last_change":"2024-04-06T13:48:36.226131+0000","last_active":"2024-04-06T13:48:38.203933+0000","last_peered":"2024-04-06T13:48:38.203933+0000","last_clean":"2024-04-06T13:48:38.203933+0000","last_became_active":"2024-04-06T13:48:36.225885+0000","last_became_peered":"2024-04-06T13:48:36.225885+0000","last_unstale":"2024-04-06T13:48:38.203933+0000","last_undegraded":"2024-04-06T13:48:38.203933+0000","last_fullsized":"2024-04-06T13:48:38.203933+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:16:24.546765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.205615+0000","last_change":"2024-04-06T13:48:19.700428+0000","last_active":"2024-04-06T13:48:36.205615+0000","last_peered":"2024-04-06T13:48:36.205615+0000","last_clean":"2024-04-06T13:48:36.205615+0000","last_became_active":"2024-04-06T13:48:19.699896+0000","last_became_peered":"2024-04-06T13:48:19.699896+0000","last_unstale":"2024-04-06T13:48:36.205615+0000","last_undegraded":"2024-04-06T13:48:36.205615+0000","last_fullsized":"2024-04-06T13:48:36.205615+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:00:56.138863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206015+0000","last_change":"2024-04-06T13:48:19.286358+0000","last_active":"2024-04-06T13:48:36.206015+0000","last_peered":"2024-04-06T13:48:36.206015+0000","last_clean":"2024-04-06T13:48:36.206015+0000","last_became_active":"2024-04-06T13:48:19.286136+0000","last_became_peered":"2024-04-06T13:48:19.286136+0000","last_unstale":"2024-04-06T13:48:36.206015+0000","last_undegraded":"2024-04-06T13:48:36.206015+0000","last_fullsized":"2024-04-06T13:48:36.206015+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:55:10.767276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205017+0000","last_change":"2024-04-06T13:48:36.220625+0000","last_active":"2024-04-06T13:48:38.205017+0000","last_peered":"2024-04-06T13:48:38.205017+0000","last_clean":"2024-04-06T13:48:38.205017+0000","last_became_active":"2024-04-06T13:48:36.220345+0000","last_became_peered":"2024-04-06T13:48:36.220345+0000","last_unstale":"2024-04-06T13:48:38.205017+0000","last_undegraded":"2024-04-06T13:48:38.205017+0000","last_fullsized":"2024-04-06T13:48:38.205017+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:01:12.661900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206344+0000","last_change":"2024-04-06T13:48:19.697815+0000","last_active":"2024-04-06T13:48:36.206344+0000","last_peered":"2024-04-06T13:48:36.206344+0000","last_clean":"2024-04-06T13:48:36.206344+0000","last_became_active":"2024-04-06T13:48:19.695354+0000","last_became_peered":"2024-04-06T13:48:19.695354+0000","last_unstale":"2024-04-06T13:48:36.206344+0000","last_undegraded":"2024-04-06T13:48:36.206344+0000","last_fullsized":"2024-04-06T13:48:36.206344+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:38:12.538926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.205541+0000","last_change":"2024-04-06T13:48:19.278849+0000","last_active":"2024-04-06T13:48:36.205541+0000","last_peered":"2024-04-06T13:48:36.205541+0000","last_clean":"2024-04-06T13:48:36.205541+0000","last_became_active":"2024-04-06T13:48:19.278604+0000","last_became_peered":"2024-04-06T13:48:19.278604+0000","last_unstale":"2024-04-06T13:48:36.205541+0000","last_undegraded":"2024-04-06T13:48:36.205541+0000","last_fullsized":"2024-04-06T13:48:36.205541+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:19:26.867548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.213079+0000","last_change":"2024-04-06T13:48:36.223518+0000","last_active":"2024-04-06T13:48:37.213079+0000","last_peered":"2024-04-06T13:48:37.213079+0000","last_clean":"2024-04-06T13:48:37.213079+0000","last_became_active":"2024-04-06T13:48:36.223244+0000","last_became_peered":"2024-04-06T13:48:36.223244+0000","last_unstale":"2024-04-06T13:48:37.213079+0000","last_undegraded":"2024-04-06T13:48:37.213079+0000","last_fullsized":"2024-04-06T13:48:37.213079+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:41:38.433651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204402+0000","last_change":"2024-04-06T13:48:19.696316+0000","last_active":"2024-04-06T13:48:36.204402+0000","last_peered":"2024-04-06T13:48:36.204402+0000","last_clean":"2024-04-06T13:48:36.204402+0000","last_became_active":"2024-04-06T13:48:19.696025+0000","last_became_peered":"2024-04-06T13:48:19.696025+0000","last_unstale":"2024-04-06T13:48:36.204402+0000","last_undegraded":"2024-04-06T13:48:36.204402+0000","last_fullsized":"2024-04-06T13:48:36.204402+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:33:27.603807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212162+0000","last_change":"2024-04-06T13:48:19.282165+0000","last_active":"2024-04-06T13:48:36.212162+0000","last_peered":"2024-04-06T13:48:36.212162+0000","last_clean":"2024-04-06T13:48:36.212162+0000","last_became_active":"2024-04-06T13:48:19.281892+0000","last_became_peered":"2024-04-06T13:48:19.281892+0000","last_unstale":"2024-04-06T13:48:36.212162+0000","last_undegraded":"2024-04-06T13:48:36.212162+0000","last_fullsized":"2024-04-06T13:48:36.212162+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:10:25.844832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212938+0000","last_change":"2024-04-06T13:48:36.228092+0000","last_active":"2024-04-06T13:48:37.212938+0000","last_peered":"2024-04-06T13:48:37.212938+0000","last_clean":"2024-04-06T13:48:37.212938+0000","last_became_active":"2024-04-06T13:48:36.227645+0000","last_became_peered":"2024-04-06T13:48:36.227645+0000","last_unstale":"2024-04-06T13:48:37.212938+0000","last_undegraded":"2024-04-06T13:48:37.212938+0000","last_fullsized":"2024-04-06T13:48:37.212938+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:08.874119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.210661+0000","last_change":"2024-04-06T13:48:19.699890+0000","last_active":"2024-04-06T13:48:36.210661+0000","last_peered":"2024-04-06T13:48:36.210661+0000","last_clean":"2024-04-06T13:48:36.210661+0000","last_became_active":"2024-04-06T13:48:19.699566+0000","last_became_peered":"2024-04-06T13:48:19.699566+0000","last_unstale":"2024-04-06T13:48:36.210661+0000","last_undegraded":"2024-04-06T13:48:36.210661+0000","last_fullsized":"2024-04-06T13:48:36.210661+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:10:02.935661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212829+0000","last_change":"2024-04-06T13:48:19.276781+0000","last_active":"2024-04-06T13:48:36.212829+0000","last_peered":"2024-04-06T13:48:36.212829+0000","last_clean":"2024-04-06T13:48:36.212829+0000","last_became_active":"2024-04-06T13:48:19.275438+0000","last_became_peered":"2024-04-06T13:48:19.275438+0000","last_unstale":"2024-04-06T13:48:36.212829+0000","last_undegraded":"2024-04-06T13:48:36.212829+0000","last_fullsized":"2024-04-06T13:48:36.212829+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:35.170771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205845+0000","last_change":"2024-04-06T13:48:36.221932+0000","last_active":"2024-04-06T13:48:38.205845+0000","last_peered":"2024-04-06T13:48:38.205845+0000","last_clean":"2024-04-06T13:48:38.205845+0000","last_became_active":"2024-04-06T13:48:36.218607+0000","last_became_peered":"2024-04-06T13:48:36.218607+0000","last_unstale":"2024-04-06T13:48:38.205845+0000","last_undegraded":"2024-04-06T13:48:38.205845+0000","last_fullsized":"2024-04-06T13:48:38.205845+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:53:03.521273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213905+0000","last_change":"2024-04-06T13:48:19.685464+0000","last_active":"2024-04-06T13:48:36.213905+0000","last_peered":"2024-04-06T13:48:36.213905+0000","last_clean":"2024-04-06T13:48:36.213905+0000","last_became_active":"2024-04-06T13:48:19.685139+0000","last_became_peered":"2024-04-06T13:48:19.685139+0000","last_unstale":"2024-04-06T13:48:36.213905+0000","last_undegraded":"2024-04-06T13:48:36.213905+0000","last_fullsized":"2024-04-06T13:48:36.213905+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:19:00.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.207172+0000","last_change":"2024-04-06T13:48:19.279096+0000","last_active":"2024-04-06T13:48:36.207172+0000","last_peered":"2024-04-06T13:48:36.207172+0000","last_clean":"2024-04-06T13:48:36.207172+0000","last_became_active":"2024-04-06T13:48:19.278774+0000","last_became_peered":"2024-04-06T13:48:19.278774+0000","last_unstale":"2024-04-06T13:48:36.207172+0000","last_undegraded":"2024-04-06T13:48:36.207172+0000","last_fullsized":"2024-04-06T13:48:36.207172+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:46:06.202483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212409+0000","last_change":"2024-04-06T13:48:36.226888+0000","last_active":"2024-04-06T13:48:37.212409+0000","last_peered":"2024-04-06T13:48:37.212409+0000","last_clean":"2024-04-06T13:48:37.212409+0000","last_became_active":"2024-04-06T13:48:36.226621+0000","last_became_peered":"2024-04-06T13:48:36.226621+0000","last_unstale":"2024-04-06T13:48:37.212409+0000","last_undegraded":"2024-04-06T13:48:37.212409+0000","last_fullsized":"2024-04-06T13:48:37.212409+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:45:41.254229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.201824+0000","last_change":"2024-04-06T13:48:19.697387+0000","last_active":"2024-04-06T13:48:36.201824+0000","last_peered":"2024-04-06T13:48:36.201824+0000","last_clean":"2024-04-06T13:48:36.201824+0000","last_became_active":"2024-04-06T13:48:19.697096+0000","last_became_peered":"2024-04-06T13:48:19.697096+0000","last_unstale":"2024-04-06T13:48:36.201824+0000","last_undegraded":"2024-04-06T13:48:36.201824+0000","last_fullsized":"2024-04-06T13:48:36.201824+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:52:01.901954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"247'2","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206430+0000","last_change":"2024-04-06T13:48:19.271906+0000","last_active":"2024-04-06T13:48:36.206430+0000","last_peered":"2024-04-06T13:48:36.206430+0000","last_clean":"2024-04-06T13:48:36.206430+0000","last_became_active":"2024-04-06T13:48:19.271643+0000","last_became_peered":"2024-04-06T13:48:19.271643+0000","last_unstale":"2024-04-06T13:48:36.206430+0000","last_undegraded":"2024-04-06T13:48:36.206430+0000","last_fullsized":"2024-04-06T13:48:36.206430+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:54:44.219906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204091+0000","last_change":"2024-04-06T13:48:36.220263+0000","last_active":"2024-04-06T13:48:38.204091+0000","last_peered":"2024-04-06T13:48:38.204091+0000","last_clean":"2024-04-06T13:48:38.204091+0000","last_became_active":"2024-04-06T13:48:36.219217+0000","last_became_peered":"2024-04-06T13:48:36.219217+0000","last_unstale":"2024-04-06T13:48:38.204091+0000","last_undegraded":"2024-04-06T13:48:38.204091+0000","last_fullsized":"2024-04-06T13:48:38.204091+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:08.874119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206574+0000","last_change":"2024-04-06T13:48:19.697332+0000","last_active":"2024-04-06T13:48:36.206574+0000","last_peered":"2024-04-06T13:48:36.206574+0000","last_clean":"2024-04-06T13:48:36.206574+0000","last_became_active":"2024-04-06T13:48:19.697087+0000","last_became_peered":"2024-04-06T13:48:19.697087+0000","last_unstale":"2024-04-06T13:48:36.206574+0000","last_undegraded":"2024-04-06T13:48:36.206574+0000","last_fullsized":"2024-04-06T13:48:36.206574+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:49:24.799254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"248'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213141+0000","last_change":"2024-04-06T13:48:19.279294+0000","last_active":"2024-04-06T13:48:36.213141+0000","last_peered":"2024-04-06T13:48:36.213141+0000","last_clean":"2024-04-06T13:48:36.213141+0000","last_became_active":"2024-04-06T13:48:19.279009+0000","last_became_peered":"2024-04-06T13:48:19.279009+0000","last_unstale":"2024-04-06T13:48:36.213141+0000","last_undegraded":"2024-04-06T13:48:36.213141+0000","last_fullsized":"2024-04-06T13:48:36.213141+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:18.981232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.211568+0000","last_change":"2024-04-06T13:48:36.217779+0000","last_active":"2024-04-06T13:48:37.211568+0000","last_peered":"2024-04-06T13:48:37.211568+0000","last_clean":"2024-04-06T13:48:37.211568+0000","last_became_active":"2024-04-06T13:48:36.217253+0000","last_became_peered":"2024-04-06T13:48:36.217253+0000","last_unstale":"2024-04-06T13:48:37.211568+0000","last_undegraded":"2024-04-06T13:48:37.211568+0000","last_fullsized":"2024-04-06T13:48:37.211568+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:13:26.069286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.205490+0000","last_change":"2024-04-06T13:48:19.694094+0000","last_active":"2024-04-06T13:48:36.205490+0000","last_peered":"2024-04-06T13:48:36.205490+0000","last_clean":"2024-04-06T13:48:36.205490+0000","last_became_active":"2024-04-06T13:48:19.692958+0000","last_became_peered":"2024-04-06T13:48:19.692958+0000","last_unstale":"2024-04-06T13:48:36.205490+0000","last_undegraded":"2024-04-06T13:48:36.205490+0000","last_fullsized":"2024-04-06T13:48:36.205490+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:04:10.547739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.207489+0000","last_change":"2024-04-06T13:48:19.272078+0000","last_active":"2024-04-06T13:48:36.207489+0000","last_peered":"2024-04-06T13:48:36.207489+0000","last_clean":"2024-04-06T13:48:36.207489+0000","last_became_active":"2024-04-06T13:48:19.271672+0000","last_became_peered":"2024-04-06T13:48:19.271672+0000","last_unstale":"2024-04-06T13:48:36.207489+0000","last_undegraded":"2024-04-06T13:48:36.207489+0000","last_fullsized":"2024-04-06T13:48:36.207489+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:55:23.871607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203999+0000","last_change":"2024-04-06T13:48:36.221782+0000","last_active":"2024-04-06T13:48:38.203999+0000","last_peered":"2024-04-06T13:48:38.203999+0000","last_clean":"2024-04-06T13:48:38.203999+0000","last_became_active":"2024-04-06T13:48:36.221533+0000","last_became_peered":"2024-04-06T13:48:36.221533+0000","last_unstale":"2024-04-06T13:48:38.203999+0000","last_undegraded":"2024-04-06T13:48:38.203999+0000","last_fullsized":"2024-04-06T13:48:38.203999+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:38:01.256020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.214900+0000","last_change":"2024-04-06T13:48:19.702071+0000","last_active":"2024-04-06T13:48:36.214900+0000","last_peered":"2024-04-06T13:48:36.214900+0000","last_clean":"2024-04-06T13:48:36.214900+0000","last_became_active":"2024-04-06T13:48:19.698755+0000","last_became_peered":"2024-04-06T13:48:19.698755+0000","last_unstale":"2024-04-06T13:48:36.214900+0000","last_undegraded":"2024-04-06T13:48:36.214900+0000","last_fullsized":"2024-04-06T13:48:36.214900+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:06:58.060998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.206596+0000","last_change":"2024-04-06T13:48:19.278939+0000","last_active":"2024-04-06T13:48:36.206596+0000","last_peered":"2024-04-06T13:48:36.206596+0000","last_clean":"2024-04-06T13:48:36.206596+0000","last_became_active":"2024-04-06T13:48:19.278677+0000","last_became_peered":"2024-04-06T13:48:19.278677+0000","last_unstale":"2024-04-06T13:48:36.206596+0000","last_undegraded":"2024-04-06T13:48:36.206596+0000","last_fullsized":"2024-04-06T13:48:36.206596+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:59:15.157954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.204721+0000","last_change":"2024-04-06T13:48:36.226108+0000","last_active":"2024-04-06T13:48:38.204721+0000","last_peered":"2024-04-06T13:48:38.204721+0000","last_clean":"2024-04-06T13:48:38.204721+0000","last_became_active":"2024-04-06T13:48:36.225708+0000","last_became_peered":"2024-04-06T13:48:36.225708+0000","last_unstale":"2024-04-06T13:48:38.204721+0000","last_undegraded":"2024-04-06T13:48:38.204721+0000","last_fullsized":"2024-04-06T13:48:38.204721+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:50:11.010781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.205937+0000","last_change":"2024-04-06T13:48:19.693016+0000","last_active":"2024-04-06T13:48:36.205937+0000","last_peered":"2024-04-06T13:48:36.205937+0000","last_clean":"2024-04-06T13:48:36.205937+0000","last_became_active":"2024-04-06T13:48:19.692760+0000","last_became_peered":"2024-04-06T13:48:19.692760+0000","last_unstale":"2024-04-06T13:48:36.205937+0000","last_undegraded":"2024-04-06T13:48:36.205937+0000","last_fullsized":"2024-04-06T13:48:36.205937+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:41:37.714397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212291+0000","last_change":"2024-04-06T13:48:19.282189+0000","last_active":"2024-04-06T13:48:36.212291+0000","last_peered":"2024-04-06T13:48:36.212291+0000","last_clean":"2024-04-06T13:48:36.212291+0000","last_became_active":"2024-04-06T13:48:19.281907+0000","last_became_peered":"2024-04-06T13:48:19.281907+0000","last_unstale":"2024-04-06T13:48:36.212291+0000","last_undegraded":"2024-04-06T13:48:36.212291+0000","last_fullsized":"2024-04-06T13:48:36.212291+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:34.453253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3f","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.205887+0000","last_change":"2024-04-06T13:48:36.229947+0000","last_active":"2024-04-06T13:48:38.205887+0000","last_peered":"2024-04-06T13:48:38.205887+0000","last_clean":"2024-04-06T13:48:38.205887+0000","last_became_active":"2024-04-06T13:48:36.229670+0000","last_became_peered":"2024-04-06T13:48:36.229670+0000","last_unstale":"2024-04-06T13:48:38.205887+0000","last_undegraded":"2024-04-06T13:48:38.205887+0000","last_fullsized":"2024-04-06T13:48:38.205887+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:30:28.148833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.202135+0000","last_change":"2024-04-06T13:48:19.689941+0000","last_active":"2024-04-06T13:48:36.202135+0000","last_peered":"2024-04-06T13:48:36.202135+0000","last_clean":"2024-04-06T13:48:36.202135+0000","last_became_active":"2024-04-06T13:48:19.689689+0000","last_became_peered":"2024-04-06T13:48:19.689689+0000","last_unstale":"2024-04-06T13:48:36.202135+0000","last_undegraded":"2024-04-06T13:48:36.202135+0000","last_fullsized":"2024-04-06T13:48:36.202135+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:55:19.563924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213316+0000","last_change":"2024-04-06T13:48:19.283271+0000","last_active":"2024-04-06T13:48:36.213316+0000","last_peered":"2024-04-06T13:48:36.213316+0000","last_clean":"2024-04-06T13:48:36.213316+0000","last_became_active":"2024-04-06T13:48:19.282980+0000","last_became_peered":"2024-04-06T13:48:19.282980+0000","last_unstale":"2024-04-06T13:48:36.213316+0000","last_undegraded":"2024-04-06T13:48:36.213316+0000","last_fullsized":"2024-04-06T13:48:36.213316+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:51:53.597002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.211904+0000","last_change":"2024-04-06T13:48:36.213917+0000","last_active":"2024-04-06T13:48:37.211904+0000","last_peered":"2024-04-06T13:48:37.211904+0000","last_clean":"2024-04-06T13:48:37.211904+0000","last_became_active":"2024-04-06T13:48:36.213679+0000","last_became_peered":"2024-04-06T13:48:36.213679+0000","last_unstale":"2024-04-06T13:48:37.211904+0000","last_undegraded":"2024-04-06T13:48:37.211904+0000","last_fullsized":"2024-04-06T13:48:37.211904+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:59:30.769144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.204267+0000","last_change":"2024-04-06T13:48:19.704846+0000","last_active":"2024-04-06T13:48:36.204267+0000","last_peered":"2024-04-06T13:48:36.204267+0000","last_clean":"2024-04-06T13:48:36.204267+0000","last_became_active":"2024-04-06T13:48:19.704420+0000","last_became_peered":"2024-04-06T13:48:19.704420+0000","last_unstale":"2024-04-06T13:48:36.204267+0000","last_undegraded":"2024-04-06T13:48:36.204267+0000","last_fullsized":"2024-04-06T13:48:36.204267+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:57:36.178732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.205312+0000","last_change":"2024-04-06T13:48:19.277530+0000","last_active":"2024-04-06T13:48:36.205312+0000","last_peered":"2024-04-06T13:48:36.205312+0000","last_clean":"2024-04-06T13:48:36.205312+0000","last_became_active":"2024-04-06T13:48:19.276966+0000","last_became_peered":"2024-04-06T13:48:19.276966+0000","last_unstale":"2024-04-06T13:48:36.205312+0000","last_undegraded":"2024-04-06T13:48:36.205312+0000","last_fullsized":"2024-04-06T13:48:36.205312+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:42:14.962783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:38.203678+0000","last_change":"2024-04-06T13:48:36.222153+0000","last_active":"2024-04-06T13:48:38.203678+0000","last_peered":"2024-04-06T13:48:38.203678+0000","last_clean":"2024-04-06T13:48:38.203678+0000","last_became_active":"2024-04-06T13:48:36.221901+0000","last_became_peered":"2024-04-06T13:48:36.221901+0000","last_unstale":"2024-04-06T13:48:38.203678+0000","last_undegraded":"2024-04-06T13:48:38.203678+0000","last_fullsized":"2024-04-06T13:48:38.203678+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:00:42.061268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.213987+0000","last_change":"2024-04-06T13:48:19.688861+0000","last_active":"2024-04-06T13:48:36.213987+0000","last_peered":"2024-04-06T13:48:36.213987+0000","last_clean":"2024-04-06T13:48:36.213987+0000","last_became_active":"2024-04-06T13:48:19.688496+0000","last_became_peered":"2024-04-06T13:48:19.688496+0000","last_unstale":"2024-04-06T13:48:36.213987+0000","last_undegraded":"2024-04-06T13:48:36.213987+0000","last_fullsized":"2024-04-06T13:48:36.213987+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:36:47.917157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":15,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.212664+0000","last_change":"2024-04-06T13:48:19.276467+0000","last_active":"2024-04-06T13:48:36.212664+0000","last_peered":"2024-04-06T13:48:36.212664+0000","last_clean":"2024-04-06T13:48:36.212664+0000","last_became_active":"2024-04-06T13:48:19.275988+0000","last_became_peered":"2024-04-06T13:48:19.275988+0000","last_unstale":"2024-04-06T13:48:36.212664+0000","last_undegraded":"2024-04-06T13:48:36.212664+0000","last_fullsized":"2024-04-06T13:48:36.212664+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:57:44.664354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":10,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212005+0000","last_change":"2024-04-06T13:48:36.211369+0000","last_active":"2024-04-06T13:48:37.212005+0000","last_peered":"2024-04-06T13:48:37.212005+0000","last_clean":"2024-04-06T13:48:37.212005+0000","last_became_active":"2024-04-06T13:48:36.210435+0000","last_became_peered":"2024-04-06T13:48:36.210435+0000","last_unstale":"2024-04-06T13:48:37.212005+0000","last_undegraded":"2024-04-06T13:48:37.212005+0000","last_fullsized":"2024-04-06T13:48:37.212005+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:35.779803+0000","last_clean_scrub_stamp":"2024-04-06T13:48:35.779803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:49:52.157497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.201932+0000","last_change":"2024-04-06T13:48:19.700894+0000","last_active":"2024-04-06T13:48:36.201932+0000","last_peered":"2024-04-06T13:48:36.201932+0000","last_clean":"2024-04-06T13:48:36.201932+0000","last_became_active":"2024-04-06T13:48:19.700666+0000","last_became_peered":"2024-04-06T13:48:19.700666+0000","last_unstale":"2024-04-06T13:48:36.201932+0000","last_undegraded":"2024-04-06T13:48:36.201932+0000","last_fullsized":"2024-04-06T13:48:36.201932+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:19.256766+0000","last_clean_scrub_stamp":"2024-04-06T13:48:19.256766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:41:21.910614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"247'1","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-06T13:48:36.202977+0000","last_change":"2024-04-06T13:48:19.277300+0000","last_active":"2024-04-06T13:48:36.202977+0000","last_peered":"2024-04-06T13:48:36.202977+0000","last_clean":"2024-04-06T13:48:36.202977+0000","last_became_active":"2024-04-06T13:48:19.276795+0000","last_became_peered":"2024-04-06T13:48:19.276795+0000","last_unstale":"2024-04-06T13:48:36.202977+0000","last_undegraded":"2024-04-06T13:48:36.202977+0000","last_fullsized":"2024-04-06T13:48:36.202977+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:48:18.248805+0000","last_clean_scrub_stamp":"2024-04-06T13:48:18.248805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:54:25.950801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"233'370","reported_seq":1341,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-06T13:48:37.212261+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:48:37.212261+0000","last_peered":"2024-04-06T13:48:37.212261+0000","last_clean":"2024-04-06T13:48:37.212261+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:48:37.212261+0000","last_undegraded":"2024-04-06T13:48:37.212261+0000","last_fullsized":"2024-04-06T13:48:37.212261+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":3611,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575996,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56956,"kb_used_data":3696,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314884,"statfs":{"total":96636764160,"available":96578441216,"internally_reserved":0,"allocated":3784704,"data_stored":2225830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57899999999999996,"15min":0.59499999999999997},"min":{"1min":0.35499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.72199999999999998,"5min":0.78400000000000003,"15min":0.82499999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56799999999999995,"15min":0.58999999999999997},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78500000000000003,"5min":0.80800000000000005,"15min":0.82899999999999996},"last":0.55700000000000005}]},{"osd":1,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.55000000000000004,"15min":0.57899999999999996},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77200000000000002,"5min":0.77700000000000002,"15min":0.82399999999999995},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.57699999999999996,"15min":0.59599999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.69899999999999995,"5min":0.83399999999999996,"15min":0.85299999999999998},"last":0.59199999999999997}]},{"osd":2,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.56299999999999994,"15min":0.58499999999999996},"min":{"1min":0.33200000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69399999999999995,"5min":0.81699999999999995,"15min":0.83899999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.58599999999999997,"15min":0.60099999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69999999999999996,"5min":0.755,"15min":0.79700000000000004},"last":0.502}]},{"osd":3,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.59399999999999997,"15min":0.60599999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.67200000000000004,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58799999999999997,"15min":0.60299999999999998},"min":{"1min":0.39000000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78900000000000003,"5min":0.82099999999999995,"15min":0.84099999999999997},"last":0.53200000000000003}]},{"osd":4,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45800000000000002,"15min":0.46600000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.81599999999999995},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.36799999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.82699999999999996},"last":0.41099999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.498,"15min":0.498},"min":{"1min":0.315,"5min":0.315,"15min":0.308},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56899999999999995,"15min":0.57799999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33600000000000002},"max":{"1min":0.74099999999999999,"5min":0.80700000000000005,"15min":0.85399999999999998},"last":0.44500000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.34899999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.879},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57099999999999995,"15min":0.58699999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81799999999999995,"5min":0.84899999999999998,"15min":0.89800000000000002},"last":0.49099999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575995,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56364,"kb_used_data":3096,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315476,"statfs":{"total":96636764160,"available":96579047424,"internally_reserved":0,"allocated":3170304,"data_stored":1626302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.53600000000000003,"15min":0.55700000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69399999999999995,"5min":0.79700000000000004,"15min":0.83799999999999997},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.54100000000000004,"15min":0.56499999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.62,"5min":0.76900000000000002,"15min":0.81999999999999995},"last":0.63}]},{"osd":1,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.60399999999999998,"15min":0.624},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76000000000000001,"5min":0.81699999999999995,"15min":0.85799999999999998},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.622,"15min":0.64400000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.378},"max":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.82699999999999996},"last":0.69799999999999995}]},{"osd":2,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56999999999999995,"15min":0.59399999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72099999999999997,"5min":0.79000000000000004,"15min":0.82899999999999996},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.56299999999999994,"15min":0.58599999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.80700000000000005},"last":0.65800000000000003}]},{"osd":3,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.58599999999999997,"15min":0.60199999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.73199999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.61899999999999999,"15min":0.64300000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.36099999999999999},"max":{"1min":0.80200000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.64400000000000002}]},{"osd":4,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70599999999999996,"5min":0.72299999999999998,"15min":0.81399999999999995},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.51100000000000001,"15min":0.53800000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.66400000000000003,"5min":0.82099999999999995,"15min":0.84099999999999997},"last":0.501}]},{"osd":5,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.48899999999999999,"15min":0.50600000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.33300000000000002},"max":{"1min":0.63,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52000000000000002,"15min":0.53100000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67400000000000004,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.51900000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53900000000000003,"15min":0.54800000000000004},"min":{"1min":0.34899999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.67700000000000005,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55300000000000005,"15min":0.56899999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.32800000000000001},"max":{"1min":0.71199999999999997,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.57599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575994,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60460,"kb_used_data":3104,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311380,"statfs":{"total":96636764160,"available":96574853120,"internally_reserved":0,"allocated":3178496,"data_stored":1633839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.38100000000000001,"15min":0.39100000000000001},"min":{"1min":0.248,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.378,"5min":0.621,"15min":0.621},"last":0.308},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.41799999999999998,"15min":0.42699999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.501,"5min":0.60799999999999998,"15min":0.69399999999999995},"last":0.22}]},{"osd":2,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.41299999999999998,"15min":0.42399999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.46899999999999997,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.378,"5min":0.44900000000000001,"15min":0.46400000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.55400000000000005,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.32700000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.46999999999999997,"15min":0.47999999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.52800000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.375},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.42699999999999999,"15min":0.439},"min":{"1min":0.28799999999999998,"5min":0.27600000000000002,"15min":0.26300000000000001},"max":{"1min":0.45500000000000002,"5min":0.63500000000000001,"15min":0.66100000000000003},"last":0.35099999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.628,"15min":0.64400000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.33800000000000002},"max":{"1min":0.70099999999999996,"5min":0.89900000000000002,"15min":0.97399999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.64400000000000002,"15min":0.65500000000000003},"min":{"1min":0.46100000000000002,"5min":0.44500000000000001,"15min":0.44400000000000001},"max":{"1min":0.73499999999999999,"5min":0.94599999999999995,"15min":0.995},"last":0.48899999999999999}]},{"osd":5,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.64100000000000001,"15min":0.65300000000000002},"min":{"1min":0.42999999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73899999999999999,"5min":0.96899999999999997,"15min":1.073},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.65700000000000003,"15min":0.66800000000000004},"min":{"1min":0.45200000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.72799999999999998,"5min":0.96499999999999997,"15min":1.0209999999999999},"last":0.53800000000000003}]},{"osd":6,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.46300000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73899999999999999,"5min":0.89200000000000002,"15min":1},"last":0.502},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.67100000000000004,"15min":0.68400000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.72499999999999998,"5min":0.94599999999999995,"15min":1.036},"last":0.55000000000000004}]},{"osd":7,"last update":"Sat Apr 6 13:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.69399999999999995,"15min":0.69499999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85399999999999998,"5min":0.94899999999999995,"15min":1.0649999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65700000000000003,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.76300000000000001,"5min":0.92900000000000005,"15min":1.056},"last":0.47199999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575994,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61048,"kb_used_data":3692,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310792,"statfs":{"total":96636764160,"available":96574251008,"internally_reserved":0,"allocated":3780608,"data_stored":2226966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.38200000000000001,"15min":0.39300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.45100000000000001,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.39300000000000002,"15min":0.41699999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.48199999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.28100000000000003}]},{"osd":2,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.40100000000000002,"15min":0.42499999999999999},"min":{"1min":0.26100000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49399999999999999,"5min":0.66700000000000004,"15min":0.70099999999999996},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.375,"5min":0.42499999999999999,"15min":0.443},"min":{"1min":0.28399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.47999999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.34599999999999997}]},{"osd":3,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.40699999999999997,"15min":0.42499999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.50700000000000001,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.317},{"interface":"front","average":{"1min":0.373,"5min":0.45700000000000002,"15min":0.47799999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.51900000000000002,"5min":0.72999999999999998,"15min":0.75700000000000001},"last":0.41299999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.60499999999999998,"15min":0.64100000000000001},"min":{"1min":0.39100000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.80900000000000005,"5min":0.93100000000000005,"15min":1.0329999999999999},"last":0.499},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.61199999999999999,"15min":0.64300000000000002},"min":{"1min":0.372,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88500000000000001,"5min":0.999,"15min":0.999},"last":0.58899999999999997}]},{"osd":5,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.60199999999999998,"15min":0.63900000000000001},"min":{"1min":0.39300000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.0009999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.63800000000000001,"15min":0.66300000000000003},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.749,"5min":0.92800000000000005,"15min":1.052},"last":0.50900000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.64500000000000002,"15min":0.67400000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.874,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.64200000000000002,"15min":0.66900000000000004},"min":{"1min":0.40500000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77500000000000002,"5min":0.96399999999999997,"15min":1.085},"last":0.52000000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.66300000000000003,"15min":0.69099999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.80700000000000005,"5min":1.01,"15min":1.121},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.64700000000000002,"15min":0.67200000000000004},"min":{"1min":0.42199999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.93700000000000006,"5min":0.95799999999999996,"15min":1.0229999999999999},"last":0.65800000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575996,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60492,"kb_used_data":3124,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311348,"statfs":{"total":96636764160,"available":96574820352,"internally_reserved":0,"allocated":3198976,"data_stored":1636535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.376,"15min":0.38400000000000001},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.42299999999999999,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.40600000000000003,"15min":0.42199999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.249},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":1.044},"last":0.54600000000000004}]},{"osd":1,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.38500000000000001,"15min":0.40400000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51100000000000001,"5min":0.54000000000000004,"15min":0.753},"last":0.432},{"interface":"front","average":{"1min":0.37,"5min":0.42799999999999999,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.82199999999999995},"last":0.77200000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.439,"15min":0.45800000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.53200000000000003,"5min":0.69699999999999995,"15min":0.70299999999999996},"last":0.621},{"interface":"front","average":{"1min":0.377,"5min":0.437,"15min":0.44900000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.504,"5min":0.64500000000000002,"15min":0.68600000000000005},"last":0.65200000000000002}]},{"osd":4,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.621,"15min":0.63900000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.376},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.64400000000000002,"15min":0.66100000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.88900000000000001,"5min":1.022,"15min":1.022},"last":0.82499999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.64300000000000002,"15min":0.66300000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88100000000000001,"5min":0.98599999999999999,"15min":1.0149999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.65100000000000002,"15min":0.66700000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.80600000000000005}]},{"osd":6,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.65200000000000002,"15min":0.67200000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92000000000000004,"5min":0.97199999999999998,"15min":1.091},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.67600000000000005,"15min":0.69499999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.96099999999999997},"last":0.84099999999999997}]},{"osd":7,"last update":"Sat Apr 6 13:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.66600000000000004,"15min":0.68799999999999994},"min":{"1min":0.42699999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.95399999999999996},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.68300000000000005,"15min":0.70799999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.85299999999999998,"5min":1.01,"15min":1.01},"last":0.89100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575995,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48196,"kb_used_data":3128,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323644,"statfs":{"total":96636764160,"available":96587411456,"internally_reserved":0,"allocated":3203072,"data_stored":1638275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.37,"15min":0.38400000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.47399999999999998,"5min":0.55200000000000005,"15min":0.72799999999999998},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.39600000000000002,"15min":0.40799999999999997},"min":{"1min":0.25700000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.48799999999999999,"5min":0.58599999999999997,"15min":0.66700000000000004},"last":0.251}]},{"osd":1,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.39300000000000002,"15min":0.40100000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.46899999999999997,"5min":0.59499999999999997,"15min":0.65700000000000003},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.41699999999999998,"15min":0.437},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.502,"5min":0.63,"15min":0.75800000000000001},"last":0.30399999999999999}]},{"osd":2,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.46100000000000002,"15min":0.47999999999999998},"min":{"1min":0.32700000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.53900000000000003,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.47699999999999998,"15min":0.496},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.51900000000000002,"5min":0.67900000000000005,"15min":0.79300000000000004},"last":0.32700000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.622,"15min":0.63100000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76700000000000002,"5min":0.94099999999999995,"15min":1.018},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.628,"15min":0.64200000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.81100000000000005,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.47399999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.66100000000000003,"15min":0.67600000000000005},"min":{"1min":0.44700000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.872,"5min":0.95699999999999996,"15min":0.999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.67900000000000005,"15min":0.68700000000000006},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.86299999999999999,"5min":1.036,"15min":1.036},"last":0.55000000000000004}]},{"osd":6,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.64100000000000001,"15min":0.66100000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.76300000000000001,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.66000000000000003,"15min":0.67600000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41099999999999998},"max":{"1min":0.82799999999999996,"5min":0.91600000000000004,"15min":0.98799999999999999},"last":0.46500000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.65500000000000003,"15min":0.66800000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80200000000000005,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.67800000000000005,"15min":0.69499999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85199999999999998,"5min":1.0209999999999999,"15min":1.024},"last":0.58499999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575995,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48180,"kb_used_data":3104,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323660,"statfs":{"total":96636764160,"available":96587427840,"internally_reserved":0,"allocated":3178496,"data_stored":1628809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55900000000000005,"15min":0.58699999999999997},"min":{"1min":0.40699999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.753,"5min":0.876,"15min":0.88500000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.39300000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81100000000000005,"5min":1.202,"15min":1.202},"last":0.501}]},{"osd":1,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.61299999999999999,"15min":0.64000000000000001},"min":{"1min":0.40799999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71699999999999997,"5min":1.256,"15min":1.256},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.624,"15min":0.63},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.75900000000000001,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.61699999999999999}]},{"osd":2,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.57999999999999996,"15min":0.59999999999999998},"min":{"1min":0.41599999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77500000000000002,"5min":0.84599999999999997,"15min":0.877},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.60599999999999998,"15min":0.61599999999999999},"min":{"1min":0.39600000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.97899999999999998,"5min":1.294,"15min":1.294},"last":0.59399999999999997}]},{"osd":3,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84499999999999997,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.60899999999999999,"15min":0.626},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.78800000000000003,"5min":0.90200000000000002,"15min":0.91100000000000003},"last":0.53300000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55500000000000005,"15min":0.52000000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.92800000000000005,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":0.63},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.56399999999999995,"15min":0.54100000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.95199999999999996,"5min":4.931,"15min":4.931},"last":0.46400000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53500000000000003,"15min":0.53300000000000003},"min":{"1min":0.34999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34699999999999998},"max":{"1min":1.0609999999999999,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.437}]},{"osd":7,"last update":"Sat Apr 6 13:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59499999999999997,"15min":0.58199999999999996},"min":{"1min":0.36899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.91000000000000003,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59399999999999997,"15min":0.60199999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.373},"max":{"1min":0.86899999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.48599999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575995,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66212,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305628,"statfs":{"total":96636764160,"available":96568963072,"internally_reserved":0,"allocated":3162112,"data_stored":1626226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.57399999999999995,"15min":0.58499999999999996},"min":{"1min":0.36399999999999999,"5min":0.35899999999999999,"15min":0.33000000000000002},"max":{"1min":0.68000000000000005,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58099999999999996,"15min":0.59399999999999997},"min":{"1min":0.38100000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.76600000000000001,"5min":0.84699999999999998,"15min":0.89200000000000002},"last":0.61399999999999999}]},{"osd":1,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.58099999999999996,"15min":0.59399999999999997},"min":{"1min":0.38400000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79100000000000004,"5min":0.81299999999999994,"15min":0.81399999999999995},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.439,"5min":0.39400000000000002,"15min":0.37},"max":{"1min":0.70299999999999996,"5min":0.871,"15min":0.871},"last":0.56499999999999995}]},{"osd":2,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59299999999999997,"15min":0.59699999999999998},"min":{"1min":0.311,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69199999999999995,"5min":0.79700000000000004,"15min":0.84399999999999997},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58799999999999997,"15min":0.59599999999999997},"min":{"1min":0.44,"5min":0.378,"15min":0.378},"max":{"1min":0.73099999999999998,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.51400000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.79800000000000004,"5min":0.81100000000000005,"15min":0.94499999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61699999999999999,"15min":0.63200000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.74099999999999999,"5min":0.88200000000000001,"15min":0.91500000000000004},"last":0.54600000000000004}]},{"osd":4,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52900000000000003,"15min":0.50700000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.27400000000000002},"max":{"1min":0.70999999999999996,"5min":0.85699999999999998,"15min":1.0369999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.59999999999999998,"15min":0.57699999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.308},"max":{"1min":0.77800000000000002,"5min":0.88800000000000001,"15min":1.1639999999999999},"last":0.65800000000000003}]},{"osd":6,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54300000000000004,"15min":0.52600000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.27900000000000003},"max":{"1min":0.89100000000000001,"5min":0.94999999999999996,"15min":1.0169999999999999},"last":0.629},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54300000000000004,"15min":0.53100000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.32300000000000001},"max":{"1min":0.81899999999999995,"5min":0.878,"15min":1.0009999999999999},"last":0.57899999999999996}]},{"osd":7,"last update":"Sat Apr 6 13:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56499999999999995,"15min":0.54800000000000004},"min":{"1min":0.39200000000000002,"5min":0.35799999999999998,"15min":0.32900000000000001},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.66400000000000003,"15min":0.64800000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.35199999999999998},"max":{"1min":0.85299999999999998,"5min":0.92300000000000004,"15min":0.96599999999999997},"last":0.68200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:48:40.744 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-06T13:48:40.744 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:48:40.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.896+0000 7f81dc2d6700 1 Processor -- start 2024-04-06T13:48:40.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.896+0000 7f81dc2d6700 1 -- start start 2024-04-06T13:48:40.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.897+0000 7f81dc2d6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f81d415d7b0 0x7f81d40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:40.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.897+0000 7f81dc2d6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81d406a730 0x7f81d406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:40.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.897+0000 7f81dc2d6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d40b3990 con 0x7f81d415ddf0 2024-04-06T13:48:40.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.897+0000 7f81dc2d6700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d406b0f0 con 0x7f81d415d7b0 2024-04-06T13:48:40.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.897+0000 7f81dc2d6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d40b3060 con 0x7f81d406a730 2024-04-06T13:48:40.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.897+0000 7f81da072700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f81d415d7b0 0x7f81d40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:40.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.897+0000 7f81da873700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f81d415ddf0 0x7f81d415e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56526/0 (socket says 172.21.15.189:56526) 2024-04-06T13:48:40.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.897+0000 7f81da873700 1 -- 172.21.15.189:0/50105119 learned_addr learned my addr 172.21.15.189:0/50105119 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:40.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.897+0000 7f81d9871700 1 --2- 172.21.15.189:0/50105119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81d406a730 0x7f81d406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:40.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.897+0000 7f81da072700 1 -- 172.21.15.189:0/50105119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81d406a730 msgr2=0x7f81d406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:40.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.897+0000 7f81da072700 1 --2- 172.21.15.189:0/50105119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81d406a730 0x7f81d406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.897+0000 7f81da072700 1 -- 172.21.15.189:0/50105119 >> v1:172.21.15.189:6789/0 conn(0x7f81d415ddf0 legacy=0x7f81d415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:40.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.897+0000 7f81da072700 1 -- 172.21.15.189:0/50105119 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d40b3340 con 0x7f81d415d7b0 2024-04-06T13:48:40.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.897+0000 7f81d9070700 1 -- 172.21.15.189:0/50105119 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383546695 0 0) 0x7f81d40b3990 con 0x7f81d415ddf0 2024-04-06T13:48:40.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.897+0000 7f81d9871700 1 --2- 172.21.15.189:0/50105119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81d406a730 0x7f81d406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:40.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.897+0000 7f81da072700 1 --2- 172.21.15.189:0/50105119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81d415d7b0 0x7f81d40b3e20 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f81d0009840 tx=0x7f81d0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36948afdca11661 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:40.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.898+0000 7f81d9070700 1 -- 172.21.15.189:0/50105119 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d0011830 con 0x7f81d415d7b0 2024-04-06T13:48:40.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.898+0000 7f81d9070700 1 -- 172.21.15.189:0/50105119 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d0011990 con 0x7f81d415d7b0 2024-04-06T13:48:40.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.898+0000 7f81d9070700 1 -- 172.21.15.189:0/50105119 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d0011d20 con 0x7f81d415d7b0 2024-04-06T13:48:40.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.898+0000 7f81dc2d6700 1 -- 172.21.15.189:0/50105119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81d415d7b0 msgr2=0x7f81d40b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:40.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.898+0000 7f81dc2d6700 1 --2- 172.21.15.189:0/50105119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81d415d7b0 0x7f81d40b3e20 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f81d0009840 tx=0x7f81d0009b50 comp rx=0 tx=0).stop 2024-04-06T13:48:40.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.898+0000 7f81dc2d6700 1 -- 172.21.15.189:0/50105119 shutdown_connections 2024-04-06T13:48:40.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.898+0000 7f81dc2d6700 1 --2- 172.21.15.189:0/50105119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81d415d7b0 0x7f81d40b3e20 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.898+0000 7f81dc2d6700 1 --2- 172.21.15.189:0/50105119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81d406a730 0x7f81d406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.898+0000 7f81dc2d6700 1 -- 172.21.15.189:0/50105119 >> 172.21.15.189:0/50105119 conn(0x7f81d40d00b0 msgr2=0x7f81d40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:40.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.898+0000 7f81dc2d6700 1 -- 172.21.15.189:0/50105119 shutdown_connections 2024-04-06T13:48:40.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.898+0000 7f81dc2d6700 1 -- 172.21.15.189:0/50105119 wait complete. 2024-04-06T13:48:40.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.899+0000 7f81dc2d6700 1 Processor -- start 2024-04-06T13:48:40.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.900+0000 7f81dc2d6700 1 -- start start 2024-04-06T13:48:40.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.900+0000 7f81dc2d6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81d406a730 0x7f81d40b0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:40.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.900+0000 7f81dc2d6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81d415d7b0 0x7f81d40aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:40.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.900+0000 7f81dc2d6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81d415ddf0 0x7f81d40ce790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:40.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.900+0000 7f81dc2d6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d40cd6f0 con 0x7f81d406a730 2024-04-06T13:48:40.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.900+0000 7f81dc2d6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d40cd9d0 con 0x7f81d415ddf0 2024-04-06T13:48:40.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.900+0000 7f81dc2d6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d40cdcb0 con 0x7f81d415d7b0 2024-04-06T13:48:40.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.900+0000 7f81da072700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81d406a730 0x7f81d40b0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:40.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.900+0000 7f81da072700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81d406a730 0x7f81d40b0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34602/0 (socket says 172.21.15.189:34602) 2024-04-06T13:48:40.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.900+0000 7f81da072700 1 -- 172.21.15.189:0/159783177 learned_addr learned my addr 172.21.15.189:0/159783177 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:40.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.901+0000 7f81d9871700 1 --2- 172.21.15.189:0/159783177 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81d415d7b0 0x7f81d40aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:40.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.901+0000 7f81da873700 1 --2- 172.21.15.189:0/159783177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81d415ddf0 0x7f81d40ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:40.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.901+0000 7f81da072700 1 -- 172.21.15.189:0/159783177 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81d415d7b0 msgr2=0x7f81d40aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:40.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.901+0000 7f81da072700 1 --2- 172.21.15.189:0/159783177 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81d415d7b0 0x7f81d40aff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.901+0000 7f81da072700 1 -- 172.21.15.189:0/159783177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81d415ddf0 msgr2=0x7f81d40ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:40.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.901+0000 7f81da072700 1 --2- 172.21.15.189:0/159783177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81d415ddf0 0x7f81d40ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:40.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.901+0000 7f81da072700 1 -- 172.21.15.189:0/159783177 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d00094a0 con 0x7f81d406a730 2024-04-06T13:48:40.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.901+0000 7f81da873700 1 --2- 172.21.15.189:0/159783177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81d415ddf0 0x7f81d40ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:40.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.901+0000 7f81da072700 1 --2- 172.21.15.189:0/159783177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81d406a730 0x7f81d40b0c00 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7f81d00090b0 tx=0x7f81d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:40.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.901+0000 7f81caffd700 1 -- 172.21.15.189:0/159783177 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d00038a0 con 0x7f81d406a730 2024-04-06T13:48:40.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.901+0000 7f81caffd700 1 -- 172.21.15.189:0/159783177 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d001f070 con 0x7f81d406a730 2024-04-06T13:48:40.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.901+0000 7f81dc2d6700 1 -- 172.21.15.189:0/159783177 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d40cdf90 con 0x7f81d406a730 2024-04-06T13:48:40.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.901+0000 7f81caffd700 1 -- 172.21.15.189:0/159783177 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d0003c40 con 0x7f81d406a730 2024-04-06T13:48:40.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.901+0000 7f81dc2d6700 1 -- 172.21.15.189:0/159783177 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d415a370 con 0x7f81d406a730 2024-04-06T13:48:40.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.902+0000 7f81dc2d6700 1 -- 172.21.15.189:0/159783177 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81d415a500 con 0x7f81d406a730 2024-04-06T13:48:40.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.902+0000 7f81caffd700 1 -- 172.21.15.189:0/159783177 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f81d001a430 con 0x7f81d406a730 2024-04-06T13:48:40.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.903+0000 7f81caffd700 1 --2- 172.21.15.189:0/159783177 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f81b806aed0 0x7f81b806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:40.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.903+0000 7f81caffd700 1 -- 172.21.15.189:0/159783177 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f81d0021070 con 0x7f81d406a730 2024-04-06T13:48:40.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.903+0000 7f81d9871700 1 --2- 172.21.15.189:0/159783177 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f81b806aed0 0x7f81b806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:40.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.903+0000 7f81d9871700 1 --2- 172.21.15.189:0/159783177 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f81b806aed0 0x7f81b806d390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f81d4160d40 tx=0x7f81c4008040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:40.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:40.907+0000 7f81caffd700 1 -- 172.21.15.189:0/159783177 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f81d415a500 con 0x7f81d406a730 2024-04-06T13:48:41.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.148+0000 7f81dc2d6700 1 -- 172.21.15.189:0/159783177 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81d40ce4b0 con 0x7f81d406a730 2024-04-06T13:48:41.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.149+0000 7f81caffd700 1 -- 172.21.15.189:0/159783177 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5967 (secure 0 0 0) 0x7f81d40ce4b0 con 0x7f81d406a730 2024-04-06T13:48:41.150 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:41.150 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":122,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3323503154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3323503154},{"type":"v1","addr":"172.21.15.189:6836","nonce":3323503154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":9383,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3292345173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3292345173},{"type":"v1","addr":"172.21.15.189:6837","nonce":3292345173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:48:19.674205+0000","modified":"2024-04-06T13:48:22.682911+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12108},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12108":{"gid":12108,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6833/726816746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":726816746},{"type":"v1","addr":"172.21.15.192:6833","nonce":726816746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:48:36.197004+0000","modified":"2024-04-06T13:48:39.206755+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9395},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9395":{"gid":9395,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/455737532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":455737532},{"type":"v1","addr":"172.21.15.192:6835","nonce":455737532}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-04-06T13:48:41.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.151+0000 7f81dc2d6700 1 -- 172.21.15.189:0/159783177 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f81b806aed0 msgr2=0x7f81b806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:41.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.151+0000 7f81dc2d6700 1 --2- 172.21.15.189:0/159783177 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f81b806aed0 0x7f81b806d390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f81d4160d40 tx=0x7f81c4008040 comp rx=0 tx=0).stop 2024-04-06T13:48:41.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.151+0000 7f81dc2d6700 1 -- 172.21.15.189:0/159783177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81d406a730 msgr2=0x7f81d40b0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:41.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.151+0000 7f81dc2d6700 1 --2- 172.21.15.189:0/159783177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81d406a730 0x7f81d40b0c00 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7f81d00090b0 tx=0x7f81d0004750 comp rx=0 tx=0).stop 2024-04-06T13:48:41.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.152+0000 7f81dc2d6700 1 -- 172.21.15.189:0/159783177 shutdown_connections 2024-04-06T13:48:41.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.152+0000 7f81dc2d6700 1 --2- 172.21.15.189:0/159783177 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f81b806aed0 0x7f81b806d390 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.152+0000 7f81dc2d6700 1 --2- 172.21.15.189:0/159783177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81d415ddf0 0x7f81d40ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.152+0000 7f81dc2d6700 1 --2- 172.21.15.189:0/159783177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81d406a730 0x7f81d40b0c00 unknown :-1 s=CLOSED pgs=2747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.152+0000 7f81dc2d6700 1 --2- 172.21.15.189:0/159783177 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81d415d7b0 0x7f81d40aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.152+0000 7f81dc2d6700 1 -- 172.21.15.189:0/159783177 >> 172.21.15.189:0/159783177 conn(0x7f81d40d00b0 msgr2=0x7f81d4157110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:41.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.152+0000 7f81dc2d6700 1 -- 172.21.15.189:0/159783177 shutdown_connections 2024-04-06T13:48:41.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.152+0000 7f81dc2d6700 1 -- 172.21.15.189:0/159783177 wait complete. 2024-04-06T13:48:41.154 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 122 2024-04-06T13:48:41.568 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-06T13:48:36.197004+0000', 'modified': '2024-04-06T13:48:39.206755+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9395}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9395': {'gid': 9395, 'name': 'b', 'rank': 0, 'incarnation': 118, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.192:6835/455737532', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6834', 'nonce': 455737532}, {'type': 'v1', 'addr': '172.21.15.192:6835', 'nonce': 455737532}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-06T13:48:41.568 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-06T13:48:41.568 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-06T13:48:41.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.694+0000 7f558a2ee700 1 Processor -- start 2024-04-06T13:48:41.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.694+0000 7f558a2ee700 1 -- start start 2024-04-06T13:48:41.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.695+0000 7f558a2ee700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f55840cdae0 0x7f55840cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.695+0000 7f558a2ee700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558415a830 con 0x7f55840aec70 2024-04-06T13:48:41.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.695+0000 7f558a2ee700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55840cc050 con 0x7f55840cea80 2024-04-06T13:48:41.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.695+0000 7f558a2ee700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55840cd920 con 0x7f55840cdae0 2024-04-06T13:48:41.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.695+0000 7f558888b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f55840cdae0 0x7f55840cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.695+0000 7f5583fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f55840aec70 0x7f55840ce370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56542/0 (socket says 172.21.15.189:56542) 2024-04-06T13:48:41.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.695+0000 7f558888b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f55840cdae0 0x7f55840cd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34608/0 (socket says 172.21.15.189:34608) 2024-04-06T13:48:41.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.695+0000 7f5583fff700 1 -- 172.21.15.189:0/232475062 learned_addr learned my addr 172.21.15.189:0/232475062 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:41.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.695+0000 7f558888b700 1 -- 172.21.15.189:0/232475062 >> v1:172.21.15.192:6789/0 conn(0x7f55840cea80 legacy=0x7f558406ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:41.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.695+0000 7f558888b700 1 -- 172.21.15.189:0/232475062 >> v1:172.21.15.189:6789/0 conn(0x7f55840aec70 legacy=0x7f55840ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:41.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.695+0000 7f5582ffd700 1 -- 172.21.15.189:0/232475062 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542105408 0 0) 0x7f558415a830 con 0x7f55840aec70 2024-04-06T13:48:41.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.695+0000 7f558888b700 1 -- 172.21.15.189:0/232475062 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55840666f0 con 0x7f55840cdae0 2024-04-06T13:48:41.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.695+0000 7f558888b700 1 --2- 172.21.15.189:0/232475062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55840cdae0 0x7f55840cd3e0 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f557c009d90 tx=0x7f557c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc270b63fe20d8ce server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:41.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.696+0000 7f5582ffd700 1 -- 172.21.15.189:0/232475062 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c00e070 con 0x7f55840cdae0 2024-04-06T13:48:41.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.696+0000 7f5582ffd700 1 -- 172.21.15.189:0/232475062 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557c00bb90 con 0x7f55840cdae0 2024-04-06T13:48:41.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.696+0000 7f5582ffd700 1 -- 172.21.15.189:0/232475062 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c003a20 con 0x7f55840cdae0 2024-04-06T13:48:41.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.696+0000 7f558a2ee700 1 -- 172.21.15.189:0/232475062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55840cdae0 msgr2=0x7f55840cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:41.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.696+0000 7f558a2ee700 1 --2- 172.21.15.189:0/232475062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55840cdae0 0x7f55840cd3e0 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f557c009d90 tx=0x7f557c00b330 comp rx=0 tx=0).stop 2024-04-06T13:48:41.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.696+0000 7f558a2ee700 1 -- 172.21.15.189:0/232475062 shutdown_connections 2024-04-06T13:48:41.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.696+0000 7f558a2ee700 1 --2- 172.21.15.189:0/232475062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55840cdae0 0x7f55840cd3e0 unknown :-1 s=CLOSED pgs=2749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.696+0000 7f558a2ee700 1 -- 172.21.15.189:0/232475062 >> 172.21.15.189:0/232475062 conn(0x7f55840cc290 msgr2=0x7f55840cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:41.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.696+0000 7f558a2ee700 1 -- 172.21.15.189:0/232475062 shutdown_connections 2024-04-06T13:48:41.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.696+0000 7f558a2ee700 1 -- 172.21.15.189:0/232475062 wait complete. 2024-04-06T13:48:41.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.698+0000 7f558a2ee700 1 Processor -- start 2024-04-06T13:48:41.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.698+0000 7f558a2ee700 1 -- start start 2024-04-06T13:48:41.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.699+0000 7f558a2ee700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f55840aec70 0x7f558414a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.699+0000 7f558a2ee700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55840cdae0 0x7f55840c5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.699+0000 7f558a2ee700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55840cea80 0x7f55840c6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.699+0000 7f558a2ee700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55840c98b0 con 0x7f55840cea80 2024-04-06T13:48:41.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.699+0000 7f558a2ee700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55840c95d0 con 0x7f55840cdae0 2024-04-06T13:48:41.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.699+0000 7f558a2ee700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558414b0b0 con 0x7f55840aec70 2024-04-06T13:48:41.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.699+0000 7f558888b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55840cea80 0x7f55840c6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.699+0000 7f558888b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55840cea80 0x7f55840c6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34618/0 (socket says 172.21.15.189:34618) 2024-04-06T13:48:41.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.699+0000 7f558888b700 1 -- 172.21.15.189:0/2341232707 learned_addr learned my addr 172.21.15.189:0/2341232707 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:41.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.699+0000 7f55837fe700 1 --2- 172.21.15.189:0/2341232707 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55840cdae0 0x7f55840c5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.699+0000 7f5583fff700 1 --2- 172.21.15.189:0/2341232707 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f55840aec70 0x7f558414a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.699+0000 7f558888b700 1 -- 172.21.15.189:0/2341232707 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f55840aec70 msgr2=0x7f558414a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:41.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.699+0000 7f558888b700 1 --2- 172.21.15.189:0/2341232707 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f55840aec70 0x7f558414a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.706 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.699+0000 7f558888b700 1 -- 172.21.15.189:0/2341232707 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55840cdae0 msgr2=0x7f55840c5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:41.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.699+0000 7f558888b700 1 --2- 172.21.15.189:0/2341232707 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55840cdae0 0x7f55840c5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.699+0000 7f558888b700 1 -- 172.21.15.189:0/2341232707 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557c009a70 con 0x7f55840cea80 2024-04-06T13:48:41.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.699+0000 7f558888b700 1 --2- 172.21.15.189:0/2341232707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55840cea80 0x7f55840c6280 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f5584073430 tx=0x7f557c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:41.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.699+0000 7f5580ff9700 1 -- 172.21.15.189:0/2341232707 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c012630 con 0x7f55840cea80 2024-04-06T13:48:41.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.700+0000 7f5580ff9700 1 -- 172.21.15.189:0/2341232707 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557c0127b0 con 0x7f55840cea80 2024-04-06T13:48:41.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.700+0000 7f558a2ee700 1 -- 172.21.15.189:0/2341232707 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558414b390 con 0x7f55840cea80 2024-04-06T13:48:41.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.700+0000 7f5580ff9700 1 -- 172.21.15.189:0/2341232707 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c012b60 con 0x7f55840cea80 2024-04-06T13:48:41.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.700+0000 7f558a2ee700 1 -- 172.21.15.189:0/2341232707 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55840c88e0 con 0x7f55840cea80 2024-04-06T13:48:41.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.701+0000 7f5580ff9700 1 -- 172.21.15.189:0/2341232707 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f557c012cc0 con 0x7f55840cea80 2024-04-06T13:48:41.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.701+0000 7f5580ff9700 1 --2- 172.21.15.189:0/2341232707 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f556406aed0 0x7f556406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.702+0000 7f5580ff9700 1 -- 172.21.15.189:0/2341232707 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f557c08b8c0 con 0x7f55840cea80 2024-04-06T13:48:41.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.702+0000 7f5583fff700 1 --2- 172.21.15.189:0/2341232707 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f556406aed0 0x7f556406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.702+0000 7f5583fff700 1 --2- 172.21.15.189:0/2341232707 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f556406aed0 0x7f556406d390 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f556c000f60 tx=0x7f556c007be0 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:41.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.703+0000 7f55964a2b80 1 Processor -- start 2024-04-06T13:48:41.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.703+0000 7f55964a2b80 1 -- start start 2024-04-06T13:48:41.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.704+0000 7f55964a2b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 0x55bb73469b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.704+0000 7f55964a2b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb7346bc80 0x55bb734943e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.704+0000 7f55964a2b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b0860 0x55bb73494920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.704+0000 7f55964a2b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb733c1940 con 0x55bb734b0860 2024-04-06T13:48:41.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.704+0000 7f55964a2b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb733bf300 con 0x55bb7346bc80 2024-04-06T13:48:41.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.704+0000 7f55964a2b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb733babf0 con 0x55bb73469760 2024-04-06T13:48:41.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.704+0000 7f558888b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b0860 0x55bb73494920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.704+0000 7f558888b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b0860 0x55bb73494920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34620/0 (socket says 172.21.15.189:34620) 2024-04-06T13:48:41.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.704+0000 7f558888b700 1 -- 172.21.15.189:0/4221714970 learned_addr learned my addr 172.21.15.189:0/4221714970 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:41.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.705+0000 7f5583fff700 1 --2- 172.21.15.189:0/4221714970 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 0x55bb73469b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.705+0000 7f55837fe700 1 --2- 172.21.15.189:0/4221714970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb7346bc80 0x55bb734943e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.705+0000 7f558888b700 1 -- 172.21.15.189:0/4221714970 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 msgr2=0x55bb73469b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:41.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.705+0000 7f558888b700 1 --2- 172.21.15.189:0/4221714970 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 0x55bb73469b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.705+0000 7f558888b700 1 -- 172.21.15.189:0/4221714970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb7346bc80 msgr2=0x55bb734943e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:41.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.705+0000 7f558888b700 1 --2- 172.21.15.189:0/4221714970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb7346bc80 0x55bb734943e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.705+0000 7f558888b700 1 -- 172.21.15.189:0/4221714970 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb731e8b10 con 0x55bb734b0860 2024-04-06T13:48:41.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.705+0000 7f55837fe700 1 --2- 172.21.15.189:0/4221714970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb7346bc80 0x55bb734943e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:41.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.705+0000 7f558888b700 1 --2- 172.21.15.189:0/4221714970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b0860 0x55bb73494920 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7f557c056390 tx=0x7f557c059870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88461ac943faf06d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:41.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.705+0000 7f5579ffb700 1 -- 172.21.15.189:0/4221714970 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c05aea0 con 0x55bb734b0860 2024-04-06T13:48:41.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.705+0000 7f5579ffb700 1 -- 172.21.15.189:0/4221714970 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557c05dcc0 con 0x55bb734b0860 2024-04-06T13:48:41.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.705+0000 7f5579ffb700 1 -- 172.21.15.189:0/4221714970 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c062280 con 0x55bb734b0860 2024-04-06T13:48:41.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.705+0000 7f55964a2b80 1 -- 172.21.15.189:0/4221714970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b0860 msgr2=0x55bb73494920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:41.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.705+0000 7f55964a2b80 1 --2- 172.21.15.189:0/4221714970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b0860 0x55bb73494920 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7f557c056390 tx=0x7f557c059870 comp rx=0 tx=0).stop 2024-04-06T13:48:41.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.706+0000 7f55964a2b80 1 -- 172.21.15.189:0/4221714970 shutdown_connections 2024-04-06T13:48:41.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.706+0000 7f55964a2b80 1 --2- 172.21.15.189:0/4221714970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b0860 0x55bb73494920 unknown :-1 s=CLOSED pgs=2751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.706+0000 7f55964a2b80 1 --2- 172.21.15.189:0/4221714970 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 0x55bb73469b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.706+0000 7f55964a2b80 1 --2- 172.21.15.189:0/4221714970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb7346bc80 0x55bb734943e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.706+0000 7f55964a2b80 1 -- 172.21.15.189:0/4221714970 >> 172.21.15.189:0/4221714970 conn(0x55bb733c8650 msgr2=0x55bb734af3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:41.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.706+0000 7f55964a2b80 1 -- 172.21.15.189:0/4221714970 shutdown_connections 2024-04-06T13:48:41.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.706+0000 7f55964a2b80 1 -- 172.21.15.189:0/4221714970 wait complete. 2024-04-06T13:48:41.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.706+0000 7f55964a2b80 1 -- start start 2024-04-06T13:48:41.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.706+0000 7f55964a2b80 1 Processor -- start 2024-04-06T13:48:41.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.707+0000 7f55964a2b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 0x55bb73493d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.707+0000 7f55964a2b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b0860 0x55bb73497aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.707+0000 7f55964a2b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb734b3ab0 0x55bb73497fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.707+0000 7f55964a2b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb73356d30 con 0x55bb734b0860 2024-04-06T13:48:41.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.707+0000 7f55964a2b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb733569d0 con 0x55bb734b3ab0 2024-04-06T13:48:41.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.707+0000 7f55964a2b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb7323e650 con 0x55bb73469760 2024-04-06T13:48:41.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.707+0000 7f55964a2b80 10 client.0 _ll_register_callbacks cb 0x55bb73360420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T13:48:41.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.707+0000 7f55837fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b0860 0x55bb73497aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.707+0000 7f55837fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b0860 0x55bb73497aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34624/0 (socket says 172.21.15.189:34624) 2024-04-06T13:48:41.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.707+0000 7f55837fe700 1 -- 172.21.15.189:0/1789300091 learned_addr learned my addr 172.21.15.189:0/1789300091 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:41.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.707+0000 7f5583fff700 1 --2- 172.21.15.189:0/1789300091 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 0x55bb73493d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f558888b700 1 --2- 172.21.15.189:0/1789300091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb734b3ab0 0x55bb73497fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f55837fe700 1 -- 172.21.15.189:0/1789300091 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 msgr2=0x55bb73493d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:41.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f55837fe700 1 --2- 172.21.15.189:0/1789300091 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 0x55bb73493d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f55837fe700 1 -- 172.21.15.189:0/1789300091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb734b3ab0 msgr2=0x55bb73497fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:41.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f55837fe700 1 --2- 172.21.15.189:0/1789300091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb734b3ab0 0x55bb73497fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f55837fe700 1 -- 172.21.15.189:0/1789300091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55840c88e0 con 0x55bb734b0860 2024-04-06T13:48:41.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f5583fff700 1 --2- 172.21.15.189:0/1789300091 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 0x55bb73493d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:41.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f558888b700 1 --2- 172.21.15.189:0/1789300091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb734b3ab0 0x55bb73497fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:41.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f55837fe700 1 --2- 172.21.15.189:0/1789300091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b0860 0x55bb73497aa0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7f55841f32d0 tx=0x7f557400e500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd82d13938e4bd4c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:41.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f5562ffd700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T13:48:41.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f5562ffd700 1 -- 172.21.15.189:0/1789300091 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557400c070 con 0x55bb734b0860 2024-04-06T13:48:41.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f5562ffd700 1 -- 172.21.15.189:0/1789300091 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(252..252 src has 1..252) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f557400ebb0 con 0x55bb734b0860 2024-04-06T13:48:41.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f55964a2b80 1 -- 172.21.15.189:0/1789300091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bb7337b330 con 0x55bb734b0860 2024-04-06T13:48:41.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f5562ffd700 1 -- 172.21.15.189:0/1789300091 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7f557400ed70 con 0x55bb734b0860 2024-04-06T13:48:41.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f55964a2b80 10 client.12402 fetch_fsmap learned FSMap version 122 2024-04-06T13:48:41.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f55964a2b80 1 -- 172.21.15.189:0/1789300091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x55bb73380e50 con 0x55bb734b0860 2024-04-06T13:48:41.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f5562ffd700 1 -- 172.21.15.189:0/1789300091 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bb7337b330 con 0x55bb734b0860 2024-04-06T13:48:41.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f5562ffd700 1 -- 172.21.15.189:0/1789300091 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 122) v1 ==== 3681+0+0 (secure 0 0 0) 0x7f5574004020 con 0x55bb734b0860 2024-04-06T13:48:41.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f55964a2b80 10 client.12402 fetch_fsmap finished waiting for FSMap version 122 2024-04-06T13:48:41.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f55964a2b80 10 client.12402 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-06T13:48:41.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f55964a2b80 20 client.12402 populate_metadata read hostname 'smithi189' 2024-04-06T13:48:41.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f55964a2b80 1 --2- 172.21.15.189:0/1789300091 >> [v2:172.21.15.192:6834/455737532,v1:172.21.15.192:6835/455737532] conn(0x55bb734969e0 0x55bb734b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f55964a2b80 4 client.12402 mds_command: new command op to 9395 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-06T13:48:41.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.708+0000 7f55964a2b80 1 -- 172.21.15.189:0/1789300091 --> [v2:172.21.15.192:6834/455737532,v1:172.21.15.192:6835/455737532] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bb733b36b0 con 0x55bb734969e0 2024-04-06T13:48:41.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.709+0000 7f5583fff700 1 --2- 172.21.15.189:0/1789300091 >> [v2:172.21.15.192:6834/455737532,v1:172.21.15.192:6835/455737532] conn(0x55bb734969e0 0x55bb734b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.709+0000 7f5562ffd700 10 client.12402 ms_handle_connect on v2:172.21.15.192:6834/455737532 2024-04-06T13:48:41.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.709+0000 7f5583fff700 1 --2- 172.21.15.189:0/1789300091 >> [v2:172.21.15.192:6834/455737532,v1:172.21.15.192:6835/455737532] conn(0x55bb734969e0 0x55bb734b5b30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e8350f644644e2d server_cookie=547541106a346fb in_seq=0 out_seq=0 2024-04-06T13:48:41.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.710+0000 7f5562ffd700 1 -- 172.21.15.189:0/1789300091 <== mds.0 v2:172.21.15.192:6834/455737532 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55bb733b36b0 con 0x55bb734969e0 2024-04-06T13:48:41.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.710+0000 7f5562ffd700 10 client.12402 handle_command_reply: tid=0 2024-04-06T13:48:41.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.710+0000 7f55964a2b80 1 client.12402 shutdown 2024-04-06T13:48:41.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.710+0000 7f55964a2b80 1 -- 172.21.15.189:0/1789300091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b0860 msgr2=0x55bb73497aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:41.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.710+0000 7f55964a2b80 1 --2- 172.21.15.189:0/1789300091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b0860 0x55bb73497aa0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7f55841f32d0 tx=0x7f557400e500 comp rx=0 tx=0).stop 2024-04-06T13:48:41.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.710+0000 7f55964a2b80 1 -- 172.21.15.189:0/1789300091 shutdown_connections 2024-04-06T13:48:41.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.710+0000 7f55964a2b80 1 --2- 172.21.15.189:0/1789300091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb734b3ab0 0x55bb73497fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.710+0000 7f55964a2b80 1 --2- 172.21.15.189:0/1789300091 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 0x55bb73493d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.710+0000 7f55964a2b80 1 --2- 172.21.15.189:0/1789300091 >> [v2:172.21.15.192:6834/455737532,v1:172.21.15.192:6835/455737532] conn(0x55bb734969e0 0x55bb734b5b30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.710+0000 7f55964a2b80 1 --2- 172.21.15.189:0/1789300091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b0860 0x55bb73497aa0 unknown :-1 s=CLOSED pgs=2752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.710+0000 7f5562ffd700 0 client.12402 ms_handle_reset on v2:172.21.15.192:6834/455737532 2024-04-06T13:48:41.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.710+0000 7f55964a2b80 1 -- 172.21.15.189:0/1789300091 >> 172.21.15.189:0/1789300091 conn(0x55bb733c8650 msgr2=0x55bb734af2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:41.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.710+0000 7f55964a2b80 1 -- 172.21.15.189:0/1789300091 shutdown_connections 2024-04-06T13:48:41.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.710+0000 7f55964a2b80 1 -- 172.21.15.189:0/1789300091 wait complete. 2024-04-06T13:48:41.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.710+0000 7f55964a2b80 20 client.12402 trim_cache size 0 max 16384 2024-04-06T13:48:41.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.728+0000 7f55964a2b80 1 Processor -- start 2024-04-06T13:48:41.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.728+0000 7f55964a2b80 1 -- start start 2024-04-06T13:48:41.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.728+0000 7f55964a2b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 0x55bb734b8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.728+0000 7f55964a2b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb734b0860 0x55bb73491a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.728+0000 7f55964a2b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b3ab0 0x55bb734ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.728+0000 7f55964a2b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb7346b450 con 0x55bb734b3ab0 2024-04-06T13:48:41.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.728+0000 7f55964a2b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb734ace20 con 0x55bb734b0860 2024-04-06T13:48:41.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.728+0000 7f55964a2b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb73464760 con 0x55bb73469760 2024-04-06T13:48:41.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.729+0000 7f558888b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b3ab0 0x55bb734ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.729+0000 7f558888b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b3ab0 0x55bb734ac8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34638/0 (socket says 172.21.15.189:34638) 2024-04-06T13:48:41.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.729+0000 7f558888b700 1 -- 172.21.15.189:0/445826315 learned_addr learned my addr 172.21.15.189:0/445826315 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:41.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.729+0000 7f5583fff700 1 --2- 172.21.15.189:0/445826315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 0x55bb734b8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.729+0000 7f55837fe700 1 --2- 172.21.15.189:0/445826315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb734b0860 0x55bb73491a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.729+0000 7f558888b700 1 -- 172.21.15.189:0/445826315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 msgr2=0x55bb734b8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:41.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.729+0000 7f558888b700 1 --2- 172.21.15.189:0/445826315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 0x55bb734b8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.729+0000 7f558888b700 1 -- 172.21.15.189:0/445826315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb734b0860 msgr2=0x55bb73491a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:41.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.729+0000 7f558888b700 1 --2- 172.21.15.189:0/445826315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb734b0860 0x55bb73491a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.729+0000 7f558888b700 1 -- 172.21.15.189:0/445826315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb73240c80 con 0x55bb734b3ab0 2024-04-06T13:48:41.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.729+0000 7f558888b700 1 --2- 172.21.15.189:0/445826315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b3ab0 0x55bb734ac8e0 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7f557c0581e0 tx=0x7f557c071000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=638b6d651f4a3e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:41.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.729+0000 7f557a7fc700 1 -- 172.21.15.189:0/445826315 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c05aea0 con 0x55bb734b3ab0 2024-04-06T13:48:41.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.729+0000 7f557a7fc700 1 -- 172.21.15.189:0/445826315 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557c059c60 con 0x55bb734b3ab0 2024-04-06T13:48:41.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.729+0000 7f557a7fc700 1 -- 172.21.15.189:0/445826315 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c062280 con 0x55bb734b3ab0 2024-04-06T13:48:41.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.730+0000 7f55964a2b80 1 -- 172.21.15.189:0/445826315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b3ab0 msgr2=0x55bb734ac8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:41.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.730+0000 7f55964a2b80 1 --2- 172.21.15.189:0/445826315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b3ab0 0x55bb734ac8e0 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7f557c0581e0 tx=0x7f557c071000 comp rx=0 tx=0).stop 2024-04-06T13:48:41.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.730+0000 7f55964a2b80 1 -- 172.21.15.189:0/445826315 shutdown_connections 2024-04-06T13:48:41.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.730+0000 7f55964a2b80 1 --2- 172.21.15.189:0/445826315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b3ab0 0x55bb734ac8e0 unknown :-1 s=CLOSED pgs=2753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.730+0000 7f55964a2b80 1 --2- 172.21.15.189:0/445826315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 0x55bb734b8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.730+0000 7f55964a2b80 1 --2- 172.21.15.189:0/445826315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb734b0860 0x55bb73491a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.730+0000 7f55964a2b80 1 -- 172.21.15.189:0/445826315 >> 172.21.15.189:0/445826315 conn(0x55bb733c8650 msgr2=0x55bb734605c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:41.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.730+0000 7f55964a2b80 1 -- 172.21.15.189:0/445826315 shutdown_connections 2024-04-06T13:48:41.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.730+0000 7f55964a2b80 1 -- 172.21.15.189:0/445826315 wait complete. 2024-04-06T13:48:41.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.730+0000 7f55964a2b80 1 -- start start 2024-04-06T13:48:41.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.730+0000 7f55964a2b80 1 Processor -- start 2024-04-06T13:48:41.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.731+0000 7f55964a2b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 0x55bb73497ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.731+0000 7f55964a2b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb734b0860 0x55bb734b81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.731+0000 7f55964a2b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b3ab0 0x55bb734af1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.731+0000 7f55964a2b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb73497600 con 0x55bb734b3ab0 2024-04-06T13:48:41.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.731+0000 7f55964a2b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb7348e0f0 con 0x55bb734b0860 2024-04-06T13:48:41.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.731+0000 7f55964a2b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb7348d4e0 con 0x55bb73469760 2024-04-06T13:48:41.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.731+0000 7f55964a2b80 10 client.0 _ll_register_callbacks cb 0x55bb73360420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T13:48:41.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.731+0000 7f558888b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b3ab0 0x55bb734af1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.731+0000 7f558888b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b3ab0 0x55bb734af1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34654/0 (socket says 172.21.15.189:34654) 2024-04-06T13:48:41.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.731+0000 7f558888b700 1 -- 172.21.15.189:0/4038397057 learned_addr learned my addr 172.21.15.189:0/4038397057 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:41.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.731+0000 7f5583fff700 1 --2- 172.21.15.189:0/4038397057 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 0x55bb73497ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.731+0000 7f55837fe700 1 --2- 172.21.15.189:0/4038397057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb734b0860 0x55bb734b81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f558888b700 1 -- 172.21.15.189:0/4038397057 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 msgr2=0x55bb73497ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:41.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f558888b700 1 --2- 172.21.15.189:0/4038397057 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 0x55bb73497ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f558888b700 1 -- 172.21.15.189:0/4038397057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb734b0860 msgr2=0x55bb734b81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:41.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f558888b700 1 --2- 172.21.15.189:0/4038397057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb734b0860 0x55bb734b81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f558888b700 1 -- 172.21.15.189:0/4038397057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bb731e8b10 con 0x55bb734b3ab0 2024-04-06T13:48:41.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f558888b700 1 --2- 172.21.15.189:0/4038397057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b3ab0 0x55bb734af1c0 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7f557c00b730 tx=0x7f557c08b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bccb01c274a9071 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:41.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f5562ffd700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T13:48:41.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f5562ffd700 1 -- 172.21.15.189:0/4038397057 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c023510 con 0x55bb734b3ab0 2024-04-06T13:48:41.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f5562ffd700 1 -- 172.21.15.189:0/4038397057 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(252..252 src has 1..252) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f557c089200 con 0x55bb734b3ab0 2024-04-06T13:48:41.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f55964a2b80 1 -- 172.21.15.189:0/4038397057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f557c05dcc0 con 0x55bb734b3ab0 2024-04-06T13:48:41.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f5562ffd700 1 -- 172.21.15.189:0/4038397057 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7f557c078be0 con 0x55bb734b3ab0 2024-04-06T13:48:41.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f55964a2b80 10 client.12408 fetch_fsmap learned FSMap version 122 2024-04-06T13:48:41.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f55964a2b80 1 -- 172.21.15.189:0/4038397057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x55bb731e6fe0 con 0x55bb734b3ab0 2024-04-06T13:48:41.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f5562ffd700 1 -- 172.21.15.189:0/4038397057 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557c05dcc0 con 0x55bb734b3ab0 2024-04-06T13:48:41.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f5562ffd700 1 -- 172.21.15.189:0/4038397057 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 122) v1 ==== 3681+0+0 (secure 0 0 0) 0x7f557c019030 con 0x55bb734b3ab0 2024-04-06T13:48:41.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f55964a2b80 10 client.12408 fetch_fsmap finished waiting for FSMap version 122 2024-04-06T13:48:41.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f55964a2b80 10 client.12408 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-06T13:48:41.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f55964a2b80 20 client.12408 populate_metadata read hostname 'smithi189' 2024-04-06T13:48:41.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f55964a2b80 1 --2- 172.21.15.189:0/4038397057 >> [v2:172.21.15.192:6834/455737532,v1:172.21.15.192:6835/455737532] conn(0x55bb73517bf0 0x55bb73517fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f55964a2b80 4 client.12408 mds_command: new command op to 9395 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-06T13:48:41.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.732+0000 7f55964a2b80 1 -- 172.21.15.189:0/4038397057 --> [v2:172.21.15.192:6834/455737532,v1:172.21.15.192:6835/455737532] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bb73209ca0 con 0x55bb73517bf0 2024-04-06T13:48:41.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.733+0000 7f5583fff700 1 --2- 172.21.15.189:0/4038397057 >> [v2:172.21.15.192:6834/455737532,v1:172.21.15.192:6835/455737532] conn(0x55bb73517bf0 0x55bb73517fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.733+0000 7f5562ffd700 10 client.12408 ms_handle_connect on v2:172.21.15.192:6834/455737532 2024-04-06T13:48:41.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.733+0000 7f5583fff700 1 --2- 172.21.15.189:0/4038397057 >> [v2:172.21.15.192:6834/455737532,v1:172.21.15.192:6835/455737532] conn(0x55bb73517bf0 0x55bb73517fd0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=803f680d049509da server_cookie=127196fcf51970b4 in_seq=0 out_seq=0 2024-04-06T13:48:41.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.733+0000 7f5562ffd700 1 -- 172.21.15.189:0/4038397057 <== mds.0 v2:172.21.15.192:6834/455737532 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55bb73209ca0 con 0x55bb73517bf0 2024-04-06T13:48:41.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.733+0000 7f5562ffd700 10 client.12408 handle_command_reply: tid=0 2024-04-06T13:48:41.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.733+0000 7f55964a2b80 1 client.12408 shutdown 2024-04-06T13:48:41.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.733+0000 7f55964a2b80 1 -- 172.21.15.189:0/4038397057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b3ab0 msgr2=0x55bb734af1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:41.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.733+0000 7f55964a2b80 1 --2- 172.21.15.189:0/4038397057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b3ab0 0x55bb734af1c0 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7f557c00b730 tx=0x7f557c08b040 comp rx=0 tx=0).stop 2024-04-06T13:48:41.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.733+0000 7f55964a2b80 1 -- 172.21.15.189:0/4038397057 shutdown_connections 2024-04-06T13:48:41.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.733+0000 7f55964a2b80 1 --2- 172.21.15.189:0/4038397057 >> [v2:172.21.15.192:6834/455737532,v1:172.21.15.192:6835/455737532] conn(0x55bb73517bf0 0x55bb73517fd0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.733+0000 7f55964a2b80 1 --2- 172.21.15.189:0/4038397057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb734b3ab0 0x55bb734af1c0 unknown :-1 s=CLOSED pgs=2754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.733+0000 7f55964a2b80 1 --2- 172.21.15.189:0/4038397057 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55bb73469760 0x55bb73497ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.733+0000 7f55964a2b80 1 --2- 172.21.15.189:0/4038397057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb734b0860 0x55bb734b81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.733+0000 7f55964a2b80 1 -- 172.21.15.189:0/4038397057 >> 172.21.15.189:0/4038397057 conn(0x55bb733c8650 msgr2=0x55bb734919c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:41.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.733+0000 7f5562ffd700 0 client.12408 ms_handle_reset on v2:172.21.15.192:6834/455737532 2024-04-06T13:48:41.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.733+0000 7f55964a2b80 1 -- 172.21.15.189:0/4038397057 shutdown_connections 2024-04-06T13:48:41.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.733+0000 7f55964a2b80 1 -- 172.21.15.189:0/4038397057 wait complete. 2024-04-06T13:48:41.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.734+0000 7f55964a2b80 20 client.12408 trim_cache size 0 max 16384 2024-04-06T13:48:41.744 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:48:41.744 INFO:teuthology.orchestra.run.smithi189.stdout: "cluster_fsid": "e30041de-e9a3-4351-8ca7-29d7f3e3728c", 2024-04-06T13:48:41.744 INFO:teuthology.orchestra.run.smithi189.stdout: "whoami": 0, 2024-04-06T13:48:41.744 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 9395, 2024-04-06T13:48:41.744 INFO:teuthology.orchestra.run.smithi189.stdout: "want_state": "up:active", 2024-04-06T13:48:41.744 INFO:teuthology.orchestra.run.smithi189.stdout: "state": "up:active", 2024-04-06T13:48:41.745 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "backup_fs", 2024-04-06T13:48:41.745 INFO:teuthology.orchestra.run.smithi189.stdout: "rank_uptime": 5.5342646350000004, 2024-04-06T13:48:41.745 INFO:teuthology.orchestra.run.smithi189.stdout: "mdsmap_epoch": 122, 2024-04-06T13:48:41.745 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch": 252, 2024-04-06T13:48:41.745 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch_barrier": 252, 2024-04-06T13:48:41.745 INFO:teuthology.orchestra.run.smithi189.stdout: "uptime": 27.371432535 2024-04-06T13:48:41.745 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:48:41.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.734+0000 7f558a2ee700 1 -- 172.21.15.189:0/2341232707 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f556406aed0 msgr2=0x7f556406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:41.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.734+0000 7f558a2ee700 1 --2- 172.21.15.189:0/2341232707 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f556406aed0 0x7f556406d390 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f556c000f60 tx=0x7f556c007be0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.734+0000 7f558a2ee700 1 -- 172.21.15.189:0/2341232707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55840cea80 msgr2=0x7f55840c6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:41.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.734+0000 7f558a2ee700 1 --2- 172.21.15.189:0/2341232707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55840cea80 0x7f55840c6280 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f5584073430 tx=0x7f557c004ad0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.734+0000 7f558a2ee700 1 -- 172.21.15.189:0/2341232707 shutdown_connections 2024-04-06T13:48:41.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.734+0000 7f558a2ee700 1 --2- 172.21.15.189:0/2341232707 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f556406aed0 0x7f556406d390 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.734+0000 7f558a2ee700 1 --2- 172.21.15.189:0/2341232707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55840cea80 0x7f55840c6280 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.734+0000 7f558a2ee700 1 --2- 172.21.15.189:0/2341232707 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f55840aec70 0x7f558414a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.734+0000 7f558a2ee700 1 --2- 172.21.15.189:0/2341232707 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55840cdae0 0x7f55840c5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.734+0000 7f558a2ee700 1 -- 172.21.15.189:0/2341232707 >> 172.21.15.189:0/2341232707 conn(0x7f55840cc290 msgr2=0x7f558406ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:41.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.734+0000 7f558a2ee700 1 -- 172.21.15.189:0/2341232707 shutdown_connections 2024-04-06T13:48:41.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.734+0000 7f558a2ee700 1 -- 172.21.15.189:0/2341232707 wait complete. 2024-04-06T13:48:41.748 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-06T13:48:41.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.870+0000 7fef9ce4a700 1 Processor -- start 2024-04-06T13:48:41.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.870+0000 7fef9ce4a700 1 -- start start 2024-04-06T13:48:41.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.871+0000 7fef9ce4a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fef980cedf0 0x7fef980cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.871+0000 7fef9ce4a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef980cf730 con 0x7fef98146dd0 2024-04-06T13:48:41.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.871+0000 7fef9ce4a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9806d850 con 0x7fef980d17a0 2024-04-06T13:48:41.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.871+0000 7fef9ce4a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef9806beb0 con 0x7fef980cedf0 2024-04-06T13:48:41.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.871+0000 7fef95d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fef98146dd0 0x7fef9806d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56548/0 (socket says 172.21.15.189:56548) 2024-04-06T13:48:41.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.871+0000 7fef95d9b700 1 -- 172.21.15.189:0/3250167496 learned_addr learned my addr 172.21.15.189:0/3250167496 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:41.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.872+0000 7fef96d9d700 1 --2- 172.21.15.189:0/3250167496 >> v2:172.21.15.192:3300/0 conn(0x7fef980cedf0 0x7fef980cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.872+0000 7fef9559a700 1 -- 172.21.15.189:0/3250167496 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377070189 0 0) 0x7fef980cf730 con 0x7fef98146dd0 2024-04-06T13:48:41.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.872+0000 7fef9559a700 1 -- 172.21.15.189:0/3250167496 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef800035b0 con 0x7fef98146dd0 2024-04-06T13:48:41.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.872+0000 7fef96d9d700 1 -- 172.21.15.189:0/3250167496 >> v1:172.21.15.192:6789/0 conn(0x7fef980d17a0 legacy=0x7fef9806b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:41.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.872+0000 7fef96d9d700 1 -- 172.21.15.189:0/3250167496 >> v1:172.21.15.189:6789/0 conn(0x7fef98146dd0 legacy=0x7fef9806d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:41.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.872+0000 7fef96d9d700 1 -- 172.21.15.189:0/3250167496 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9806c0a0 con 0x7fef980cedf0 2024-04-06T13:48:41.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.872+0000 7fef96d9d700 1 --2- 172.21.15.189:0/3250167496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef980cedf0 0x7fef980cf1f0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fef90009d90 tx=0x7fef9000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4308a8934ec53ed9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:41.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.872+0000 7fef9559a700 1 -- 172.21.15.189:0/3250167496 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9000e070 con 0x7fef980cedf0 2024-04-06T13:48:41.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.872+0000 7fef9559a700 1 -- 172.21.15.189:0/3250167496 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9000bb20 con 0x7fef980cedf0 2024-04-06T13:48:41.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.872+0000 7fef9559a700 1 -- 172.21.15.189:0/3250167496 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9000be90 con 0x7fef980cedf0 2024-04-06T13:48:41.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.873+0000 7fef9ce4a700 1 -- 172.21.15.189:0/3250167496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef980cedf0 msgr2=0x7fef980cf1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:41.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.873+0000 7fef9ce4a700 1 --2- 172.21.15.189:0/3250167496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef980cedf0 0x7fef980cf1f0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fef90009d90 tx=0x7fef9000b330 comp rx=0 tx=0).stop 2024-04-06T13:48:41.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.873+0000 7fef9ce4a700 1 -- 172.21.15.189:0/3250167496 shutdown_connections 2024-04-06T13:48:41.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.873+0000 7fef9ce4a700 1 --2- 172.21.15.189:0/3250167496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef980cedf0 0x7fef980cf1f0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.873+0000 7fef9ce4a700 1 -- 172.21.15.189:0/3250167496 >> 172.21.15.189:0/3250167496 conn(0x7fef980b2db0 msgr2=0x7fef980b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:41.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.873+0000 7fef9ce4a700 1 -- 172.21.15.189:0/3250167496 shutdown_connections 2024-04-06T13:48:41.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.873+0000 7fef9ce4a700 1 -- 172.21.15.189:0/3250167496 wait complete. 2024-04-06T13:48:41.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.874+0000 7fef9ce4a700 1 Processor -- start 2024-04-06T13:48:41.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.875+0000 7fef9ce4a700 1 -- start start 2024-04-06T13:48:41.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.875+0000 7fef9ce4a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef980cedf0 0x7fef980ae380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.875+0000 7fef9ce4a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef980d17a0 0x7fef980ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.875+0000 7fef9ce4a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef98146dd0 0x7fef980aee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.875+0000 7fef9ce4a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef980acc00 con 0x7fef980d17a0 2024-04-06T13:48:41.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.875+0000 7fef9ce4a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef980ac8d0 con 0x7fef980cedf0 2024-04-06T13:48:41.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.875+0000 7fef9ce4a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef980b2300 con 0x7fef98146dd0 2024-04-06T13:48:41.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.876+0000 7fef95d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef980d17a0 0x7fef980ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.876+0000 7fef95d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef980d17a0 0x7fef980ae8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34664/0 (socket says 172.21.15.189:34664) 2024-04-06T13:48:41.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.876+0000 7fef95d9b700 1 -- 172.21.15.189:0/833284092 learned_addr learned my addr 172.21.15.189:0/833284092 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:41.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.876+0000 7fef9659c700 1 --2- 172.21.15.189:0/833284092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef980cedf0 0x7fef980ae380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.876+0000 7fef96d9d700 1 --2- 172.21.15.189:0/833284092 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef98146dd0 0x7fef980aee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.876+0000 7fef95d9b700 1 -- 172.21.15.189:0/833284092 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef98146dd0 msgr2=0x7fef980aee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:41.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.876+0000 7fef95d9b700 1 --2- 172.21.15.189:0/833284092 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef98146dd0 0x7fef980aee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.876+0000 7fef95d9b700 1 -- 172.21.15.189:0/833284092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef980cedf0 msgr2=0x7fef980ae380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:41.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.876+0000 7fef95d9b700 1 --2- 172.21.15.189:0/833284092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef980cedf0 0x7fef980ae380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:41.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.876+0000 7fef95d9b700 1 -- 172.21.15.189:0/833284092 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef90009a70 con 0x7fef980d17a0 2024-04-06T13:48:41.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.876+0000 7fef95d9b700 1 --2- 172.21.15.189:0/833284092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef980d17a0 0x7fef980ae8c0 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7fef88005910 tx=0x7fef8800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:41.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.876+0000 7fef86ffd700 1 -- 172.21.15.189:0/833284092 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88005d70 con 0x7fef980d17a0 2024-04-06T13:48:41.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.876+0000 7fef86ffd700 1 -- 172.21.15.189:0/833284092 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef880070d0 con 0x7fef980d17a0 2024-04-06T13:48:41.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.876+0000 7fef9ce4a700 1 -- 172.21.15.189:0/833284092 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef980c4880 con 0x7fef980d17a0 2024-04-06T13:48:41.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.876+0000 7fef86ffd700 1 -- 172.21.15.189:0/833284092 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88007460 con 0x7fef980d17a0 2024-04-06T13:48:41.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.876+0000 7fef9ce4a700 1 -- 172.21.15.189:0/833284092 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef980b2550 con 0x7fef980d17a0 2024-04-06T13:48:41.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.877+0000 7fef9ce4a700 1 -- 172.21.15.189:0/833284092 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef980acd40 con 0x7fef980d17a0 2024-04-06T13:48:41.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.878+0000 7fef86ffd700 1 -- 172.21.15.189:0/833284092 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fef880075c0 con 0x7fef980d17a0 2024-04-06T13:48:41.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.878+0000 7fef86ffd700 1 --2- 172.21.15.189:0/833284092 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fef7806aed0 0x7fef7806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:41.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.878+0000 7fef9659c700 1 --2- 172.21.15.189:0/833284092 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fef7806aed0 0x7fef7806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:41.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.878+0000 7fef86ffd700 1 -- 172.21.15.189:0/833284092 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6965+0+0 (secure 0 0 0) 0x7fef88089020 con 0x7fef980d17a0 2024-04-06T13:48:41.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.879+0000 7fef9659c700 1 --2- 172.21.15.189:0/833284092 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fef7806aed0 0x7fef7806d390 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fef8c007fd0 tx=0x7fef8c009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:41.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:41.883+0000 7fef86ffd700 1 -- 172.21.15.189:0/833284092 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fef980acd40 con 0x7fef980d17a0 2024-04-06T13:48:42.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.106+0000 7fef9ce4a700 1 -- 172.21.15.189:0/833284092 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fef9806c940 con 0x7fef7806aed0 2024-04-06T13:48:42.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.107+0000 7fef86ffd700 1 -- 172.21.15.189:0/833284092 <== mgr.11733 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7fef8c001d60 con 0x7fef7806aed0 2024-04-06T13:48:42.109 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T13:48:42.109 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:42.109 INFO:teuthology.orchestra.run.smithi189.stdout: "name": "cephfs" 2024-04-06T13:48:42.109 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:48:42.109 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:48:42.109 INFO:teuthology.orchestra.run.smithi189.stdout: "name": "backup_fs" 2024-04-06T13:48:42.109 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:48:42.109 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T13:48:42.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.109+0000 7fef9ce4a700 1 -- 172.21.15.189:0/833284092 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fef7806aed0 msgr2=0x7fef7806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:42.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.110+0000 7fef9ce4a700 1 --2- 172.21.15.189:0/833284092 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fef7806aed0 0x7fef7806d390 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fef8c007fd0 tx=0x7fef8c009040 comp rx=0 tx=0).stop 2024-04-06T13:48:42.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.110+0000 7fef9ce4a700 1 -- 172.21.15.189:0/833284092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef980d17a0 msgr2=0x7fef980ae8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:42.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.110+0000 7fef9ce4a700 1 --2- 172.21.15.189:0/833284092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef980d17a0 0x7fef980ae8c0 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7fef88005910 tx=0x7fef8800c9e0 comp rx=0 tx=0).stop 2024-04-06T13:48:42.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.110+0000 7fef9ce4a700 1 -- 172.21.15.189:0/833284092 shutdown_connections 2024-04-06T13:48:42.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.110+0000 7fef9ce4a700 1 --2- 172.21.15.189:0/833284092 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fef7806aed0 0x7fef7806d390 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:42.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.110+0000 7fef9ce4a700 1 --2- 172.21.15.189:0/833284092 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef98146dd0 0x7fef980aee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:42.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.110+0000 7fef9ce4a700 1 --2- 172.21.15.189:0/833284092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef980cedf0 0x7fef980ae380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:42.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.110+0000 7fef9ce4a700 1 --2- 172.21.15.189:0/833284092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef980d17a0 0x7fef980ae8c0 unknown :-1 s=CLOSED pgs=2756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:42.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.110+0000 7fef9ce4a700 1 -- 172.21.15.189:0/833284092 >> 172.21.15.189:0/833284092 conn(0x7fef980b2db0 msgr2=0x7fef9815cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:42.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.110+0000 7fef9ce4a700 1 -- 172.21.15.189:0/833284092 shutdown_connections 2024-04-06T13:48:42.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.110+0000 7fef9ce4a700 1 -- 172.21.15.189:0/833284092 wait complete. 2024-04-06T13:48:42.123 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-06T13:48:42.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.247+0000 7f960c0e9700 1 Processor -- start 2024-04-06T13:48:42.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.247+0000 7f960c0e9700 1 -- start start 2024-04-06T13:48:42.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.248+0000 7f960c0e9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f96040cfa00 0x7f96040cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:42.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.248+0000 7f960c0e9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96040cbd80 con 0x7f96040d11b0 2024-04-06T13:48:42.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.248+0000 7f960c0e9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96040cef20 con 0x7f96040cb870 2024-04-06T13:48:42.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.248+0000 7f960c0e9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96040ccdc0 con 0x7f96040cfa00 2024-04-06T13:48:42.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.248+0000 7f960a686700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f96040cfa00 0x7f96040cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.248+0000 7f9609e85700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f96040d11b0 0x7f96040d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56556/0 (socket says 172.21.15.189:56556) 2024-04-06T13:48:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.248+0000 7f9609e85700 1 -- 172.21.15.189:0/2366301707 learned_addr learned my addr 172.21.15.189:0/2366301707 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.249+0000 7f960a686700 1 -- 172.21.15.189:0/2366301707 >> v1:172.21.15.192:6789/0 conn(0x7f96040cb870 legacy=0x7f960415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.249+0000 7f960a686700 1 -- 172.21.15.189:0/2366301707 >> v1:172.21.15.189:6789/0 conn(0x7f96040d11b0 legacy=0x7f96040d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.249+0000 7f960a686700 1 -- 172.21.15.189:0/2366301707 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96040cf200 con 0x7f96040cfa00 2024-04-06T13:48:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.249+0000 7f9608e83700 1 -- 172.21.15.189:0/2366301707 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332304916 0 0) 0x7f96040cbd80 con 0x7f96040d11b0 2024-04-06T13:48:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.249+0000 7f960a686700 1 --2- 172.21.15.189:0/2366301707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96040cfa00 0x7f96040cd560 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f95fc009d90 tx=0x7f95fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64be67da0e304299 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.249+0000 7f9608e83700 1 -- 172.21.15.189:0/2366301707 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95fc00e070 con 0x7f96040cfa00 2024-04-06T13:48:42.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.249+0000 7f9608e83700 1 -- 172.21.15.189:0/2366301707 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95fc00bbd0 con 0x7f96040cfa00 2024-04-06T13:48:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.249+0000 7f9608e83700 1 -- 172.21.15.189:0/2366301707 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95fc012430 con 0x7f96040cfa00 2024-04-06T13:48:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.249+0000 7f960c0e9700 1 -- 172.21.15.189:0/2366301707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96040cfa00 msgr2=0x7f96040cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.249+0000 7f960c0e9700 1 --2- 172.21.15.189:0/2366301707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96040cfa00 0x7f96040cd560 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f95fc009d90 tx=0x7f95fc00b330 comp rx=0 tx=0).stop 2024-04-06T13:48:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.249+0000 7f960c0e9700 1 -- 172.21.15.189:0/2366301707 shutdown_connections 2024-04-06T13:48:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.249+0000 7f960c0e9700 1 --2- 172.21.15.189:0/2366301707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96040cfa00 0x7f96040cd560 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.249+0000 7f960c0e9700 1 -- 172.21.15.189:0/2366301707 >> 172.21.15.189:0/2366301707 conn(0x7f96040b2db0 msgr2=0x7f96040b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.250+0000 7f960c0e9700 1 -- 172.21.15.189:0/2366301707 shutdown_connections 2024-04-06T13:48:42.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.250+0000 7f960c0e9700 1 -- 172.21.15.189:0/2366301707 wait complete. 2024-04-06T13:48:42.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.251+0000 7f960c0e9700 1 Processor -- start 2024-04-06T13:48:42.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.251+0000 7f960c0e9700 1 -- start start 2024-04-06T13:48:42.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.251+0000 7f960c0e9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96040cb870 0x7f96040c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:42.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.251+0000 7f960c0e9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96040cfa00 0x7f9604155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:42.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.251+0000 7f960c0e9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96040d11b0 0x7f96041579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:42.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.251+0000 7f960c0e9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96040c5290 con 0x7f96040cb870 2024-04-06T13:48:42.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.251+0000 7f960c0e9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96040c8fb0 con 0x7f96040d11b0 2024-04-06T13:48:42.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.251+0000 7f960c0e9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96040c4ef0 con 0x7f96040cfa00 2024-04-06T13:48:42.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.252+0000 7f9609e85700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96040cb870 0x7f96040c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:42.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.252+0000 7f9609e85700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96040cb870 0x7f96040c82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34692/0 (socket says 172.21.15.189:34692) 2024-04-06T13:48:42.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.252+0000 7f9609e85700 1 -- 172.21.15.189:0/3519096409 learned_addr learned my addr 172.21.15.189:0/3519096409 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.252+0000 7f960a686700 1 --2- 172.21.15.189:0/3519096409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96040d11b0 0x7f96041579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.252+0000 7f9609684700 1 --2- 172.21.15.189:0/3519096409 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96040cfa00 0x7f9604155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.252+0000 7f9609e85700 1 -- 172.21.15.189:0/3519096409 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96040cfa00 msgr2=0x7f9604155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.252+0000 7f9609e85700 1 --2- 172.21.15.189:0/3519096409 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96040cfa00 0x7f9604155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.252+0000 7f9609e85700 1 -- 172.21.15.189:0/3519096409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96040d11b0 msgr2=0x7f96041579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.252+0000 7f9609e85700 1 --2- 172.21.15.189:0/3519096409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96040d11b0 0x7f96041579e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.252+0000 7f9609e85700 1 -- 172.21.15.189:0/3519096409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95fc009a70 con 0x7f96040cb870 2024-04-06T13:48:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.252+0000 7f960a686700 1 --2- 172.21.15.189:0/3519096409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96040d11b0 0x7f96041579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.252+0000 7f9609684700 1 --2- 172.21.15.189:0/3519096409 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96040cfa00 0x7f9604155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.252+0000 7f9609e85700 1 --2- 172.21.15.189:0/3519096409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96040cb870 0x7f96040c82f0 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f95f400c550 tx=0x7f95f400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.252+0000 7f95fa7fc700 1 -- 172.21.15.189:0/3519096409 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f4004c80 con 0x7f96040cb870 2024-04-06T13:48:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.252+0000 7f95fa7fc700 1 -- 172.21.15.189:0/3519096409 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95f4004e00 con 0x7f96040cb870 2024-04-06T13:48:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.252+0000 7f960c0e9700 1 -- 172.21.15.189:0/3519096409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96041e6600 con 0x7f96040cb870 2024-04-06T13:48:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.252+0000 7f95fa7fc700 1 -- 172.21.15.189:0/3519096409 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f4007310 con 0x7f96040cb870 2024-04-06T13:48:42.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.252+0000 7f960c0e9700 1 -- 172.21.15.189:0/3519096409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96040c8cd0 con 0x7f96040cb870 2024-04-06T13:48:42.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.253+0000 7f960c0e9700 1 -- 172.21.15.189:0/3519096409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96040c4d30 con 0x7f96040cb870 2024-04-06T13:48:42.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.254+0000 7f95fa7fc700 1 -- 172.21.15.189:0/3519096409 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f95f4007470 con 0x7f96040cb870 2024-04-06T13:48:42.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.254+0000 7f95fa7fc700 1 --2- 172.21.15.189:0/3519096409 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f95e806aed0 0x7f95e806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:42.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.254+0000 7f95fa7fc700 1 -- 172.21.15.189:0/3519096409 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f95f4089950 con 0x7f96040cb870 2024-04-06T13:48:42.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.254+0000 7f9609684700 1 --2- 172.21.15.189:0/3519096409 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f95e806aed0 0x7f95e806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:42.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.255+0000 7f9609684700 1 --2- 172.21.15.189:0/3519096409 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f95e806aed0 0x7f95e806d390 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9604158760 tx=0x7f9600009040 comp rx=0 tx=0).ready entity=mgr.11733 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:42.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.259+0000 7f95fa7fc700 1 -- 172.21.15.189:0/3519096409 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f95f40542e0 con 0x7f96040cb870 2024-04-06T13:48:42.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:42.510+0000 7f960c0e9700 1 -- 172.21.15.189:0/3519096409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f9604074990 con 0x7f96040cb870 2024-04-06T13:48:43.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.245+0000 7f95fa7fc700 1 -- 172.21.15.189:0/3519096409 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 151) v1 ==== 89366+0+0 (secure 0 0 0) 0x7f95f40546b0 con 0x7f96040cb870 2024-04-06T13:48:43.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.246+0000 7f95fa7fc700 1 -- 172.21.15.189:0/3519096409 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v151) v1 ==== 92+0+0 (secure 0 0 0) 0x7f95f4074600 con 0x7f96040cb870 2024-04-06T13:48:43.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.249+0000 7f960c0e9700 1 -- 172.21.15.189:0/3519096409 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f95e806aed0 msgr2=0x7f95e806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:43.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.249+0000 7f960c0e9700 1 --2- 172.21.15.189:0/3519096409 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f95e806aed0 0x7f95e806d390 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9604158760 tx=0x7f9600009040 comp rx=0 tx=0).stop 2024-04-06T13:48:43.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.249+0000 7f960c0e9700 1 -- 172.21.15.189:0/3519096409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96040cb870 msgr2=0x7f96040c82f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:43.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.249+0000 7f960c0e9700 1 --2- 172.21.15.189:0/3519096409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96040cb870 0x7f96040c82f0 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f95f400c550 tx=0x7f95f400c910 comp rx=0 tx=0).stop 2024-04-06T13:48:43.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.249+0000 7f960c0e9700 1 -- 172.21.15.189:0/3519096409 shutdown_connections 2024-04-06T13:48:43.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.249+0000 7f960c0e9700 1 --2- 172.21.15.189:0/3519096409 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f95e806aed0 0x7f95e806d390 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:43.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.249+0000 7f960c0e9700 1 --2- 172.21.15.189:0/3519096409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96040d11b0 0x7f96041579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:43.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.249+0000 7f960c0e9700 1 --2- 172.21.15.189:0/3519096409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96040cb870 0x7f96040c82f0 unknown :-1 s=CLOSED pgs=2759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:43.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.249+0000 7f960c0e9700 1 --2- 172.21.15.189:0/3519096409 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96040cfa00 0x7f9604155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:43.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.249+0000 7f960c0e9700 1 -- 172.21.15.189:0/3519096409 >> 172.21.15.189:0/3519096409 conn(0x7f96040b2db0 msgr2=0x7f96040b5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:43.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.249+0000 7f960c0e9700 1 -- 172.21.15.189:0/3519096409 shutdown_connections 2024-04-06T13:48:43.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.249+0000 7f960c0e9700 1 -- 172.21.15.189:0/3519096409 wait complete. 2024-04-06T13:48:43.265 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-06T13:48:43.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.397+0000 7eff69e9f700 1 Processor -- start 2024-04-06T13:48:43.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.397+0000 7eff69e9f700 1 -- start start 2024-04-06T13:48:43.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.398+0000 7eff69e9f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff640d11b0 0x7eff640d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:43.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.398+0000 7eff69e9f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7eff640cb870 0x7eff6415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:43.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.398+0000 7eff69e9f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff640d1ad0 con 0x7eff640cfa00 2024-04-06T13:48:43.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.398+0000 7eff69e9f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff640ccdc0 con 0x7eff640cb870 2024-04-06T13:48:43.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.398+0000 7eff69e9f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff640d0b50 con 0x7eff640d11b0 2024-04-06T13:48:43.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.398+0000 7eff5bfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7eff640cb870 0x7eff6415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:43.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.398+0000 7eff5bfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7eff640cb870 0x7eff6415c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34724/0 (socket says 172.21.15.189:34724) 2024-04-06T13:48:43.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.398+0000 7eff5bfff700 1 -- 172.21.15.189:0/4174838415 learned_addr learned my addr 172.21.15.189:0/4174838415 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:43.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.398+0000 7eff637fe700 1 --2- 172.21.15.189:0/4174838415 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff640d11b0 0x7eff640d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:43.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.399+0000 7eff5bfff700 1 -- 172.21.15.189:0/4174838415 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff640d11b0 msgr2=0x7eff640d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:43.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.399+0000 7eff5bfff700 1 --2- 172.21.15.189:0/4174838415 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff640d11b0 0x7eff640d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:43.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.399+0000 7eff5bfff700 1 -- 172.21.15.189:0/4174838415 >> v1:172.21.15.192:6789/0 conn(0x7eff640cfa00 legacy=0x7eff640cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:43.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.399+0000 7eff5bfff700 1 -- 172.21.15.189:0/4174838415 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff640cbd80 con 0x7eff640cb870 2024-04-06T13:48:43.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.399+0000 7eff62ffd700 1 -- 172.21.15.189:0/4174838415 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908003946 0 0) 0x7eff640d1ad0 con 0x7eff640cfa00 2024-04-06T13:48:43.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.399+0000 7eff5bfff700 1 --2- 172.21.15.189:0/4174838415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff640cb870 0x7eff6415c280 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7eff4c0098f0 tx=0x7eff4c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6068aa10ed494567 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:43.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.399+0000 7eff62ffd700 1 -- 172.21.15.189:0/4174838415 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c011830 con 0x7eff640cb870 2024-04-06T13:48:43.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.399+0000 7eff62ffd700 1 -- 172.21.15.189:0/4174838415 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff4c011990 con 0x7eff640cb870 2024-04-06T13:48:43.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.399+0000 7eff62ffd700 1 -- 172.21.15.189:0/4174838415 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c011d20 con 0x7eff640cb870 2024-04-06T13:48:43.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.400+0000 7eff69e9f700 1 -- 172.21.15.189:0/4174838415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff640cb870 msgr2=0x7eff6415c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:43.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.400+0000 7eff69e9f700 1 --2- 172.21.15.189:0/4174838415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff640cb870 0x7eff6415c280 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7eff4c0098f0 tx=0x7eff4c009c00 comp rx=0 tx=0).stop 2024-04-06T13:48:43.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.400+0000 7eff69e9f700 1 -- 172.21.15.189:0/4174838415 shutdown_connections 2024-04-06T13:48:43.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.400+0000 7eff69e9f700 1 --2- 172.21.15.189:0/4174838415 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff640d11b0 0x7eff640d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:43.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.400+0000 7eff69e9f700 1 --2- 172.21.15.189:0/4174838415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff640cb870 0x7eff6415c280 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:43.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.400+0000 7eff69e9f700 1 -- 172.21.15.189:0/4174838415 >> 172.21.15.189:0/4174838415 conn(0x7eff640b2db0 msgr2=0x7eff640b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:43.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.400+0000 7eff69e9f700 1 -- 172.21.15.189:0/4174838415 shutdown_connections 2024-04-06T13:48:43.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.400+0000 7eff69e9f700 1 -- 172.21.15.189:0/4174838415 wait complete. 2024-04-06T13:48:43.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.401+0000 7eff69e9f700 1 Processor -- start 2024-04-06T13:48:43.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.401+0000 7eff69e9f700 1 -- start start 2024-04-06T13:48:43.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.402+0000 7eff69e9f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff640cb870 0x7eff64158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:43.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.402+0000 7eff69e9f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff640cfa00 0x7eff640c7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:43.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.402+0000 7eff69e9f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff640d11b0 0x7eff6415d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:43.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.402+0000 7eff69e9f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff640c7e00 con 0x7eff640d11b0 2024-04-06T13:48:43.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.402+0000 7eff69e9f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff64157c70 con 0x7eff640cfa00 2024-04-06T13:48:43.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.402+0000 7eff69e9f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff64157340 con 0x7eff640cb870 2024-04-06T13:48:43.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.402+0000 7eff637fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff640cb870 0x7eff64158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:43.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.402+0000 7eff63fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff640d11b0 0x7eff6415d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:43.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.402+0000 7eff637fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff640cb870 0x7eff64158100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:32884/0 (socket says 172.21.15.189:32884) 2024-04-06T13:48:43.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.402+0000 7eff637fe700 1 -- 172.21.15.189:0/2934790456 learned_addr learned my addr 172.21.15.189:0/2934790456 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:43.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.402+0000 7eff5bfff700 1 --2- 172.21.15.189:0/2934790456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff640cfa00 0x7eff640c7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:43.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.402+0000 7eff637fe700 1 -- 172.21.15.189:0/2934790456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff640cfa00 msgr2=0x7eff640c7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:43.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.402+0000 7eff637fe700 1 --2- 172.21.15.189:0/2934790456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff640cfa00 0x7eff640c7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:43.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.402+0000 7eff637fe700 1 -- 172.21.15.189:0/2934790456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff640d11b0 msgr2=0x7eff6415d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:43.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.402+0000 7eff637fe700 1 --2- 172.21.15.189:0/2934790456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff640d11b0 0x7eff6415d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:43.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.402+0000 7eff637fe700 1 -- 172.21.15.189:0/2934790456 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff4c009550 con 0x7eff640cb870 2024-04-06T13:48:43.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.402+0000 7eff63fff700 1 --2- 172.21.15.189:0/2934790456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff640d11b0 0x7eff6415d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:43.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.402+0000 7eff637fe700 1 --2- 172.21.15.189:0/2934790456 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff640cb870 0x7eff64158100 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7eff5400e870 tx=0x7eff5400eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:43.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.403+0000 7eff60ff9700 1 -- 172.21.15.189:0/2934790456 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff54009590 con 0x7eff640cb870 2024-04-06T13:48:43.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.403+0000 7eff60ff9700 1 -- 172.21.15.189:0/2934790456 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff54013680 con 0x7eff640cb870 2024-04-06T13:48:43.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.403+0000 7eff60ff9700 1 -- 172.21.15.189:0/2934790456 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff540139f0 con 0x7eff640cb870 2024-04-06T13:48:43.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.403+0000 7eff69e9f700 1 -- 172.21.15.189:0/2934790456 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff64157620 con 0x7eff640cb870 2024-04-06T13:48:43.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.403+0000 7eff69e9f700 1 -- 172.21.15.189:0/2934790456 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff64155a40 con 0x7eff640cb870 2024-04-06T13:48:43.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.404+0000 7eff69e9f700 1 -- 172.21.15.189:0/2934790456 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff64157f50 con 0x7eff640cb870 2024-04-06T13:48:43.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.404+0000 7eff60ff9700 1 -- 172.21.15.189:0/2934790456 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 151) v1 ==== 89366+0+0 (secure 0 0 0) 0x7eff54013b50 con 0x7eff640cb870 2024-04-06T13:48:43.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.404+0000 7eff60ff9700 1 --2- 172.21.15.189:0/2934790456 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7eff4406af20 0x7eff4406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:43.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.404+0000 7eff5bfff700 1 -- 172.21.15.189:0/2934790456 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7eff4406af20 msgr2=0x7eff4406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:43.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.405+0000 7eff60ff9700 1 -- 172.21.15.189:0/2934790456 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6965+0+0 (secure 0 0 0) 0x7eff54089a70 con 0x7eff640cb870 2024-04-06T13:48:43.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.405+0000 7eff5bfff700 1 --2- 172.21.15.189:0/2934790456 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7eff4406af20 0x7eff4406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:48:43.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.409+0000 7eff60ff9700 1 -- 172.21.15.189:0/2934790456 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7eff54054490 con 0x7eff640cb870 2024-04-06T13:48:43.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.606+0000 7eff5bfff700 1 -- 172.21.15.189:0/2934790456 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7eff4406af20 msgr2=0x7eff4406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:43.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.606+0000 7eff5bfff700 1 --2- 172.21.15.189:0/2934790456 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7eff4406af20 0x7eff4406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:48:43.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.611+0000 7eff69e9f700 1 -- 172.21.15.189:0/2934790456 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7eff64074990 con 0x7eff640cb870 2024-04-06T13:48:43.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.617+0000 7eff60ff9700 1 -- 172.21.15.189:0/2934790456 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v83)=0 v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7eff54057ab0 con 0x7eff640cb870 2024-04-06T13:48:43.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.619+0000 7eff69e9f700 1 -- 172.21.15.189:0/2934790456 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7eff4406af20 msgr2=0x7eff4406d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:48:43.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.619+0000 7eff69e9f700 1 --2- 172.21.15.189:0/2934790456 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7eff4406af20 0x7eff4406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:43.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.619+0000 7eff69e9f700 1 -- 172.21.15.189:0/2934790456 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff640cb870 msgr2=0x7eff64158100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:43.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.619+0000 7eff69e9f700 1 --2- 172.21.15.189:0/2934790456 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff640cb870 0x7eff64158100 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7eff5400e870 tx=0x7eff5400eb80 comp rx=0 tx=0).stop 2024-04-06T13:48:43.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.619+0000 7eff69e9f700 1 -- 172.21.15.189:0/2934790456 shutdown_connections 2024-04-06T13:48:43.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.619+0000 7eff69e9f700 1 --2- 172.21.15.189:0/2934790456 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7eff4406af20 0x7eff4406d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:43.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.619+0000 7eff69e9f700 1 --2- 172.21.15.189:0/2934790456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff640d11b0 0x7eff6415d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:43.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.619+0000 7eff69e9f700 1 --2- 172.21.15.189:0/2934790456 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff640cb870 0x7eff64158100 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:43.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.619+0000 7eff69e9f700 1 --2- 172.21.15.189:0/2934790456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff640cfa00 0x7eff640c7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:43.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.619+0000 7eff69e9f700 1 -- 172.21.15.189:0/2934790456 >> 172.21.15.189:0/2934790456 conn(0x7eff640b2db0 msgr2=0x7eff6415abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:43.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.620+0000 7eff69e9f700 1 -- 172.21.15.189:0/2934790456 shutdown_connections 2024-04-06T13:48:43.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.620+0000 7eff69e9f700 1 -- 172.21.15.189:0/2934790456 wait complete. 2024-04-06T13:48:43.629 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:43.629+0000 7f9bdfa76200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:48:43.634 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-06T13:48:43.652 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:43.651+0000 7f934056e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:48:43.710 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:43.710+0000 7f9bdfa76200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:48:43.737 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:43.736+0000 7f934056e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:48:43.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.771+0000 7f8875610700 1 Processor -- start 2024-04-06T13:48:43.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.771+0000 7f8875610700 1 -- start start 2024-04-06T13:48:43.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.772+0000 7f8875610700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f88700b4ed0 0x7f88700b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:43.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.772+0000 7f8875610700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f88700d2880 0x7f88700d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:43.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.772+0000 7f8875610700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88700b5cc0 con 0x7f887015d590 2024-04-06T13:48:43.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.772+0000 7f8875610700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88700b58c0 con 0x7f88700d2880 2024-04-06T13:48:43.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.772+0000 7f8875610700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88700d1500 con 0x7f88700b4ed0 2024-04-06T13:48:43.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.772+0000 7f886f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f88700d2880 0x7f88700d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:43.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.772+0000 7f886f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f88700d2880 0x7f88700d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34736/0 (socket says 172.21.15.189:34736) 2024-04-06T13:48:43.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.772+0000 7f886f7fe700 1 -- 172.21.15.189:0/4287892749 learned_addr learned my addr 172.21.15.189:0/4287892749 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:43.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.772+0000 7f886e7fc700 1 --2- 172.21.15.189:0/4287892749 >> v2:172.21.15.192:3300/0 conn(0x7f88700b4ed0 0x7f88700b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:43.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.773+0000 7f886dffb700 1 -- 172.21.15.189:0/4287892749 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752917570 0 0) 0x7f88700b5cc0 con 0x7f887015d590 2024-04-06T13:48:43.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.773+0000 7f886dffb700 1 -- 172.21.15.189:0/4287892749 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88580035b0 con 0x7f887015d590 2024-04-06T13:48:43.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.773+0000 7f886f7fe700 1 -- 172.21.15.189:0/4287892749 >> v2:172.21.15.192:3300/0 conn(0x7f88700b4ed0 msgr2=0x7f88700b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:43.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.773+0000 7f886f7fe700 1 --2- 172.21.15.189:0/4287892749 >> v2:172.21.15.192:3300/0 conn(0x7f88700b4ed0 0x7f88700b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:43.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.773+0000 7f886f7fe700 1 -- 172.21.15.189:0/4287892749 >> v1:172.21.15.192:6789/0 conn(0x7f887015d590 legacy=0x7f88700b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:43.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.773+0000 7f886f7fe700 1 -- 172.21.15.189:0/4287892749 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88700d17e0 con 0x7f88700d2880 2024-04-06T13:48:43.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.773+0000 7f886f7fe700 1 --2- 172.21.15.189:0/4287892749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88700d2880 0x7f88700d2cd0 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f88700b6330 tx=0x7f886800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd98272daa0aacf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:43.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.773+0000 7f886dffb700 1 -- 172.21.15.189:0/4287892749 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886800e070 con 0x7f88700d2880 2024-04-06T13:48:43.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.773+0000 7f886dffb700 1 -- 172.21.15.189:0/4287892749 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886800b910 con 0x7f88700d2880 2024-04-06T13:48:43.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.773+0000 7f886dffb700 1 -- 172.21.15.189:0/4287892749 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886800bc80 con 0x7f88700d2880 2024-04-06T13:48:43.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.774+0000 7f8875610700 1 -- 172.21.15.189:0/4287892749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88700d2880 msgr2=0x7f88700d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:43.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.774+0000 7f8875610700 1 --2- 172.21.15.189:0/4287892749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88700d2880 0x7f88700d2cd0 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f88700b6330 tx=0x7f886800b330 comp rx=0 tx=0).stop 2024-04-06T13:48:43.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.774+0000 7f8875610700 1 -- 172.21.15.189:0/4287892749 shutdown_connections 2024-04-06T13:48:43.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.774+0000 7f8875610700 1 --2- 172.21.15.189:0/4287892749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88700d2880 0x7f88700d2cd0 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:43.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.774+0000 7f8875610700 1 --2- 172.21.15.189:0/4287892749 >> v2:172.21.15.192:3300/0 conn(0x7f88700b4ed0 0x7f88700b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:43.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.774+0000 7f8875610700 1 -- 172.21.15.189:0/4287892749 >> 172.21.15.189:0/4287892749 conn(0x7f887006bbc0 msgr2=0x7f887006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:43.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.774+0000 7f8875610700 1 -- 172.21.15.189:0/4287892749 shutdown_connections 2024-04-06T13:48:43.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.774+0000 7f8875610700 1 -- 172.21.15.189:0/4287892749 wait complete. 2024-04-06T13:48:43.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.775+0000 7f8875610700 1 Processor -- start 2024-04-06T13:48:43.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.775+0000 7f8875610700 1 -- start start 2024-04-06T13:48:43.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.776+0000 7f8875610700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88700b4ed0 0x7f88700cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:43.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.776+0000 7f8875610700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88700d2880 0x7f88701f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:43.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.776+0000 7f8875610700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f887015d590 0x7f88700b3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:43.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.776+0000 7f8875610700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88700ccfd0 con 0x7f88700b4ed0 2024-04-06T13:48:43.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.776+0000 7f8875610700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88700cc730 con 0x7f88700d2880 2024-04-06T13:48:43.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.776+0000 7f8875610700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88700caa20 con 0x7f887015d590 2024-04-06T13:48:43.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.776+0000 7f886effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88700b4ed0 0x7f88700cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:43.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.776+0000 7f886effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88700b4ed0 0x7f88700cca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34740/0 (socket says 172.21.15.189:34740) 2024-04-06T13:48:43.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.776+0000 7f886effd700 1 -- 172.21.15.189:0/4035284508 learned_addr learned my addr 172.21.15.189:0/4035284508 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:43.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.776+0000 7f886e7fc700 1 --2- 172.21.15.189:0/4035284508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88700d2880 0x7f88701f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:43.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.776+0000 7f886f7fe700 1 --2- 172.21.15.189:0/4035284508 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f887015d590 0x7f88700b3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:43.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.776+0000 7f886effd700 1 -- 172.21.15.189:0/4035284508 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f887015d590 msgr2=0x7f88700b3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:43.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.776+0000 7f886effd700 1 --2- 172.21.15.189:0/4035284508 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f887015d590 0x7f88700b3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:43.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.776+0000 7f886effd700 1 -- 172.21.15.189:0/4035284508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88700d2880 msgr2=0x7f88701f26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:43.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.776+0000 7f886effd700 1 --2- 172.21.15.189:0/4035284508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88700d2880 0x7f88701f26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:43.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.776+0000 7f886effd700 1 -- 172.21.15.189:0/4035284508 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8868009b30 con 0x7f88700b4ed0 2024-04-06T13:48:43.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.776+0000 7f886effd700 1 --2- 172.21.15.189:0/4035284508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88700b4ed0 0x7f88700cca90 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7f886000c450 tx=0x7f886000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:43.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.777+0000 7f88577fe700 1 -- 172.21.15.189:0/4035284508 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88600052d0 con 0x7f88700b4ed0 2024-04-06T13:48:43.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.777+0000 7f88577fe700 1 -- 172.21.15.189:0/4035284508 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8860005430 con 0x7f88700b4ed0 2024-04-06T13:48:43.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.777+0000 7f88577fe700 1 -- 172.21.15.189:0/4035284508 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88600038c0 con 0x7f88700b4ed0 2024-04-06T13:48:43.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.777+0000 7f8875610700 1 -- 172.21.15.189:0/4035284508 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88700cad00 con 0x7f88700b4ed0 2024-04-06T13:48:43.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.777+0000 7f8875610700 1 -- 172.21.15.189:0/4035284508 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88700cb2c0 con 0x7f88700b4ed0 2024-04-06T13:48:43.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.778+0000 7f8875610700 1 -- 172.21.15.189:0/4035284508 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8870074990 con 0x7f88700b4ed0 2024-04-06T13:48:43.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.778+0000 7f88577fe700 1 -- 172.21.15.189:0/4035284508 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 151) v1 ==== 89366+0+0 (secure 0 0 0) 0x7f8860003a20 con 0x7f88700b4ed0 2024-04-06T13:48:43.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.779+0000 7f88577fe700 1 --2- 172.21.15.189:0/4035284508 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f885c06af20 0x7f885c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:43.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.779+0000 7f886e7fc700 1 -- 172.21.15.189:0/4035284508 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f885c06af20 msgr2=0x7f885c06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:43.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.779+0000 7f88577fe700 1 -- 172.21.15.189:0/4035284508 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6965+0+0 (secure 0 0 0) 0x7f8860088e80 con 0x7f88700b4ed0 2024-04-06T13:48:43.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.779+0000 7f886e7fc700 1 --2- 172.21.15.189:0/4035284508 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f885c06af20 0x7f885c06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:48:43.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.783+0000 7f88577fe700 1 -- 172.21.15.189:0/4035284508 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8860053970 con 0x7f88700b4ed0 2024-04-06T13:48:43.798 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:43.784+0000 7f9bdfa76200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:48:43.829 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:43.828+0000 7f934056e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:48:43.853 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:43.853+0000 7f9bdfa76200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:48:43.904 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:43.899+0000 7f934056e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:48:43.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.980+0000 7f886e7fc700 1 -- 172.21.15.189:0/4035284508 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f885c06af20 msgr2=0x7f885c06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:43.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:43.980+0000 7f886e7fc700 1 --2- 172.21.15.189:0/4035284508 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f885c06af20 0x7f885c06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:48:44.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.005+0000 7f8875610700 1 -- 172.21.15.189:0/4035284508 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f887006c280 con 0x7f88700b4ed0 2024-04-06T13:48:44.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.009+0000 7f88577fe700 1 -- 172.21.15.189:0/4035284508 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v84)=0 v84) v1 ==== 133+0+0 (secure 0 0 0) 0x7f88600170c0 con 0x7f88700b4ed0 2024-04-06T13:48:44.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.011+0000 7f8875610700 1 -- 172.21.15.189:0/4035284508 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f885c06af20 msgr2=0x7f885c06d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:48:44.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.011+0000 7f8875610700 1 --2- 172.21.15.189:0/4035284508 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f885c06af20 0x7f885c06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:44.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.011+0000 7f8875610700 1 -- 172.21.15.189:0/4035284508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88700b4ed0 msgr2=0x7f88700cca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:44.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.011+0000 7f8875610700 1 --2- 172.21.15.189:0/4035284508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88700b4ed0 0x7f88700cca90 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7f886000c450 tx=0x7f886000c760 comp rx=0 tx=0).stop 2024-04-06T13:48:44.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.011+0000 7f8875610700 1 -- 172.21.15.189:0/4035284508 shutdown_connections 2024-04-06T13:48:44.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.011+0000 7f8875610700 1 --2- 172.21.15.189:0/4035284508 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f885c06af20 0x7f885c06d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:44.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.011+0000 7f8875610700 1 --2- 172.21.15.189:0/4035284508 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f887015d590 0x7f88700b3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:44.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.011+0000 7f8875610700 1 --2- 172.21.15.189:0/4035284508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88700b4ed0 0x7f88700cca90 unknown :-1 s=CLOSED pgs=2763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:44.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.011+0000 7f8875610700 1 --2- 172.21.15.189:0/4035284508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88700d2880 0x7f88701f26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:44.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.011+0000 7f8875610700 1 -- 172.21.15.189:0/4035284508 >> 172.21.15.189:0/4035284508 conn(0x7f887006bbc0 msgr2=0x7f887015be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:44.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.011+0000 7f8875610700 1 -- 172.21.15.189:0/4035284508 shutdown_connections 2024-04-06T13:48:44.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.011+0000 7f8875610700 1 -- 172.21.15.189:0/4035284508 wait complete. 2024-04-06T13:48:44.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:48:44.026 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2024-04-06T13:48:44.030 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:44.030+0000 7f9bdfa76200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:48:44.083 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:44.082+0000 7f934056e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:48:44.094 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir 1M 2024-04-06T13:48:44.186 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:44.185+0000 7f9bdfa76200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:48:44.251 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:44.250+0000 7f934056e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:48:44.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.269+0000 7fb08397a700 1 Processor -- start 2024-04-06T13:48:44.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.270+0000 7fb08397a700 1 -- start start 2024-04-06T13:48:44.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.271+0000 7fb08397a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb07c15d7b0 0x7fb07c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:44.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.271+0000 7fb08397a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb07c15e500 0x7fb07c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:44.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.271+0000 7fb08397a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb07c0b3990 con 0x7fb07c06a730 2024-04-06T13:48:44.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.271+0000 7fb08397a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb07c06b0f0 con 0x7fb07c15e500 2024-04-06T13:48:44.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.271+0000 7fb08397a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c0b3060 con 0x7fb07c15d7b0 2024-04-06T13:48:44.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.271+0000 7fb081f17700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb07c15e500 0x7fb07c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:44.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.271+0000 7fb080f15700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb07c06a730 0x7fb07c15ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56558/0 (socket says 172.21.15.189:56558) 2024-04-06T13:48:44.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.271+0000 7fb080f15700 1 -- 172.21.15.189:0/4006418886 learned_addr learned my addr 172.21.15.189:0/4006418886 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:44.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.271+0000 7fb081716700 1 --2- 172.21.15.189:0/4006418886 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb07c15d7b0 0x7fb07c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:44.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.271+0000 7fb081f17700 1 -- 172.21.15.189:0/4006418886 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb07c15d7b0 msgr2=0x7fb07c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:44.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.271+0000 7fb081f17700 1 --2- 172.21.15.189:0/4006418886 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb07c15d7b0 0x7fb07c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:44.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.271+0000 7fb081f17700 1 -- 172.21.15.189:0/4006418886 >> v1:172.21.15.189:6789/0 conn(0x7fb07c06a730 legacy=0x7fb07c15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:44.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.271+0000 7fb073fff700 1 -- 172.21.15.189:0/4006418886 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469397756 0 0) 0x7fb07c0b3990 con 0x7fb07c06a730 2024-04-06T13:48:44.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.271+0000 7fb081f17700 1 -- 172.21.15.189:0/4006418886 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07c0b3340 con 0x7fb07c15e500 2024-04-06T13:48:44.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.271+0000 7fb081f17700 1 --2- 172.21.15.189:0/4006418886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb07c15e500 0x7fb07c06abb0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7fb074009d90 tx=0x7fb07400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6adb65fda89de9a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:44.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.271+0000 7fb073fff700 1 -- 172.21.15.189:0/4006418886 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07400e070 con 0x7fb07c15e500 2024-04-06T13:48:44.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.271+0000 7fb073fff700 1 -- 172.21.15.189:0/4006418886 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb07400bb20 con 0x7fb07c15e500 2024-04-06T13:48:44.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.271+0000 7fb073fff700 1 -- 172.21.15.189:0/4006418886 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb074012430 con 0x7fb07c15e500 2024-04-06T13:48:44.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.272+0000 7fb08397a700 1 -- 172.21.15.189:0/4006418886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb07c15e500 msgr2=0x7fb07c06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:44.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.272+0000 7fb08397a700 1 --2- 172.21.15.189:0/4006418886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb07c15e500 0x7fb07c06abb0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7fb074009d90 tx=0x7fb07400b330 comp rx=0 tx=0).stop 2024-04-06T13:48:44.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.272+0000 7fb08397a700 1 -- 172.21.15.189:0/4006418886 shutdown_connections 2024-04-06T13:48:44.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.272+0000 7fb08397a700 1 --2- 172.21.15.189:0/4006418886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb07c15e500 0x7fb07c06abb0 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:44.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.272+0000 7fb08397a700 1 --2- 172.21.15.189:0/4006418886 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb07c15d7b0 0x7fb07c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:44.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.272+0000 7fb08397a700 1 -- 172.21.15.189:0/4006418886 >> 172.21.15.189:0/4006418886 conn(0x7fb07c0d00b0 msgr2=0x7fb07c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:44.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.272+0000 7fb08397a700 1 -- 172.21.15.189:0/4006418886 shutdown_connections 2024-04-06T13:48:44.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.272+0000 7fb08397a700 1 -- 172.21.15.189:0/4006418886 wait complete. 2024-04-06T13:48:44.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.274+0000 7fb08397a700 1 Processor -- start 2024-04-06T13:48:44.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.274+0000 7fb08397a700 1 -- start start 2024-04-06T13:48:44.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.275+0000 7fb08397a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb07c06a730 0x7fb07c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:44.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.275+0000 7fb08397a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb07c15d7b0 0x7fb07c154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:44.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.275+0000 7fb08397a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb07c15e500 0x7fb07c0aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:44.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.275+0000 7fb08397a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c15a640 con 0x7fb07c15d7b0 2024-04-06T13:48:44.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.275+0000 7fb08397a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c0cdd40 con 0x7fb07c06a730 2024-04-06T13:48:44.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.275+0000 7fb08397a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c15a310 con 0x7fb07c15e500 2024-04-06T13:48:44.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.275+0000 7fb080f15700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb07c15d7b0 0x7fb07c154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:44.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.275+0000 7fb080f15700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb07c15d7b0 0x7fb07c154740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34766/0 (socket says 172.21.15.189:34766) 2024-04-06T13:48:44.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.275+0000 7fb080f15700 1 -- 172.21.15.189:0/1965073909 learned_addr learned my addr 172.21.15.189:0/1965073909 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:44.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.275+0000 7fb081f17700 1 --2- 172.21.15.189:0/1965073909 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb07c15e500 0x7fb07c0aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:44.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.275+0000 7fb081716700 1 --2- 172.21.15.189:0/1965073909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb07c06a730 0x7fb07c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:44.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.275+0000 7fb080f15700 1 -- 172.21.15.189:0/1965073909 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb07c15e500 msgr2=0x7fb07c0aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:44.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.275+0000 7fb080f15700 1 --2- 172.21.15.189:0/1965073909 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb07c15e500 0x7fb07c0aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:44.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.275+0000 7fb080f15700 1 -- 172.21.15.189:0/1965073909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb07c06a730 msgr2=0x7fb07c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:44.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.275+0000 7fb080f15700 1 --2- 172.21.15.189:0/1965073909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb07c06a730 0x7fb07c0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:44.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.275+0000 7fb080f15700 1 -- 172.21.15.189:0/1965073909 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb074009a70 con 0x7fb07c15d7b0 2024-04-06T13:48:44.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.275+0000 7fb080f15700 1 --2- 172.21.15.189:0/1965073909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb07c15d7b0 0x7fb07c154740 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7fb06c0011e0 tx=0x7fb06c00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:44.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.275+0000 7fb071ffb700 1 -- 172.21.15.189:0/1965073909 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06c004c80 con 0x7fb07c15d7b0 2024-04-06T13:48:44.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.276+0000 7fb071ffb700 1 -- 172.21.15.189:0/1965073909 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06c004e00 con 0x7fb07c15d7b0 2024-04-06T13:48:44.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.276+0000 7fb08397a700 1 -- 172.21.15.189:0/1965073909 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb07c0b15b0 con 0x7fb07c15d7b0 2024-04-06T13:48:44.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.276+0000 7fb071ffb700 1 -- 172.21.15.189:0/1965073909 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06c007310 con 0x7fb07c15d7b0 2024-04-06T13:48:44.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.276+0000 7fb08397a700 1 -- 172.21.15.189:0/1965073909 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb07c0ce2c0 con 0x7fb07c15d7b0 2024-04-06T13:48:44.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.277+0000 7fb08397a700 1 -- 172.21.15.189:0/1965073909 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb07c067bb0 con 0x7fb07c15d7b0 2024-04-06T13:48:44.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.277+0000 7fb071ffb700 1 -- 172.21.15.189:0/1965073909 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 151) v1 ==== 89366+0+0 (secure 0 0 0) 0x7fb06c007470 con 0x7fb07c15d7b0 2024-04-06T13:48:44.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.278+0000 7fb071ffb700 1 --2- 172.21.15.189:0/1965073909 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb06006af20 0x7fb06006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:44.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.278+0000 7fb081716700 1 -- 172.21.15.189:0/1965073909 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb06006af20 msgr2=0x7fb06006d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:44.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.278+0000 7fb071ffb700 1 -- 172.21.15.189:0/1965073909 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6965+0+0 (secure 0 0 0) 0x7fb06c089f80 con 0x7fb07c15d7b0 2024-04-06T13:48:44.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.278+0000 7fb081716700 1 --2- 172.21.15.189:0/1965073909 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb06006af20 0x7fb06006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:48:44.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.282+0000 7fb071ffb700 1 -- 172.21.15.189:0/1965073909 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb06c054980 con 0x7fb07c15d7b0 2024-04-06T13:48:44.299 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:44.298+0000 7f9bdfa76200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:48:44.368 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:44.367+0000 7f934056e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:48:44.381 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:44.380+0000 7f9bdfa76200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:48:44.453 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:44.451+0000 7f934056e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:48:44.456 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:44.455+0000 7f9bdfa76200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:48:44.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.479+0000 7fb081716700 1 -- 172.21.15.189:0/1965073909 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb06006af20 msgr2=0x7fb06006d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:44.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.479+0000 7fb081716700 1 --2- 172.21.15.189:0/1965073909 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb06006af20 0x7fb06006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:48:44.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.513+0000 7fb08397a700 1 -- 172.21.15.189:0/1965073909 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fb07c15eb40 con 0x7fb06006af20 2024-04-06T13:48:44.530 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:44.529+0000 7f934056e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:48:44.659 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:44.658+0000 7f9bdfa76200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:48:44.734 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:44.734+0000 7f9bdfa76200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:48:44.735 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:44.734+0000 7f934056e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:48:44.811 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:44.810+0000 7f934056e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:48:44.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.880+0000 7fb081716700 1 -- 172.21.15.189:0/1965073909 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb06006af20 msgr2=0x7fb06006d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:44.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:44.880+0000 7fb081716700 1 --2- 172.21.15.189:0/1965073909 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb06006af20 0x7fb06006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-06T13:48:45.052 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:45.052+0000 7f9bdfa76200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:48:45.133 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:45.132+0000 7f934056e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:48:45.370 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:45.370+0000 7f9bdfa76200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:48:45.456 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:45.455+0000 7f934056e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:48:45.528 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:45.528+0000 7f9bdfa76200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:48:45.600 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:45.600+0000 7f9bdfa76200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:48:45.617 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:45.616+0000 7f934056e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:48:45.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:45.681+0000 7fb081716700 1 -- 172.21.15.189:0/1965073909 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb06006af20 msgr2=0x7fb06006d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:45.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:45.681+0000 7fb081716700 1 --2- 172.21.15.189:0/1965073909 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb06006af20 0x7fb06006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-06T13:48:45.690 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:45.689+0000 7f934056e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:48:46.213 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:46.213+0000 7f9bdfa76200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:48:46.289 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:46.289+0000 7f9bdfa76200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:48:46.314 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:46.314+0000 7f934056e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:48:46.394 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:46.393+0000 7f934056e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:48:46.415 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:46.415+0000 7f9bdfa76200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:48:46.521 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:46.521+0000 7f934056e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:48:46.922 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:46.922+0000 7f9bdfa76200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:48:47.037 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:47.036+0000 7f934056e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:48:47.076 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:47.076+0000 7f9bdfa76200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:48:47.194 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:47.193+0000 7f934056e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:48:47.232 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:47.232+0000 7f9bdfa76200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:48:47.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:47.282+0000 7fb081716700 1 -- 172.21.15.189:0/1965073909 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb06006af20 msgr2=0x7fb06006d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:47.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:47.282+0000 7fb081716700 1 --2- 172.21.15.189:0/1965073909 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb06006af20 0x7fb06006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-06T13:48:47.355 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:47.354+0000 7f934056e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:48:47.949 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:47.949+0000 7f9bdfa76200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:48:48.087 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:48.086+0000 7f934056e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:48:48.192 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:48.192+0000 7f9bdfa76200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:48:48.269 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:48.269+0000 7f9bdfa76200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:48:48.335 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:48.334+0000 7f934056e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:48:48.414 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:48.413+0000 7f934056e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:48:48.643 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:48.643+0000 7f9bdfa76200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:48:48.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:48.712+0000 7fb071ffb700 1 -- 172.21.15.189:0/1965073909 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 152) v1 ==== 89366+0+0 (secure 0 0 0) 0x7fb06c054d50 con 0x7fb07c15d7b0 2024-04-06T13:48:48.798 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:48.797+0000 7f934056e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:48:48.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:48.817+0000 7fb071ffb700 1 -- 172.21.15.189:0/1965073909 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mgrmap(e 153) v1 ==== 88873+0+0 (secure 0 0 0) 0x7fb06c0437f0 con 0x7fb07c15d7b0 2024-04-06T13:48:48.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:48.818+0000 7fb071ffb700 1 -- 172.21.15.189:0/1965073909 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb06006af20 msgr2=0x7fb06006d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:48:48.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:48.818+0000 7fb071ffb700 1 --2- 172.21.15.189:0/1965073909 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb06006af20 0x7fb06006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:49.032 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.031+0000 7f92b5c81700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.032 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.031+0000 7f92b5c81700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.032 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.031+0000 7f92b5c81700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.032 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.031+0000 7f92b5c81700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.032 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.031+0000 7f92b5c81700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.032 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.031+0000 7f92b2c7b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.032 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.031+0000 7f92b2c7b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.032 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.031+0000 7f92b2c7b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.033 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.031+0000 7f92b2c7b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.033 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.031+0000 7f92b2c7b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.041 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.041+0000 7f92b2c7b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.041 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.041+0000 7f92b2c7b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.042 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.041+0000 7f92b2c7b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.042 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.041+0000 7f92b2c7b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.042 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.041+0000 7f92b2c7b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.042 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.041+0000 7f92b5c81700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.042 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.041+0000 7f92b5c81700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.042 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.041+0000 7f92b5c81700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.042 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.041+0000 7f92b5c81700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.042 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.041+0000 7f92b5c81700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:49.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:49.821+0000 7fb071ffb700 1 -- 172.21.15.189:0/1965073909 <== mon.0 v2:172.21.15.189:3300/0 9 ==== mgrmap(e 154) v1 ==== 89000+0+0 (secure 0 0 0) 0x7fb06c055620 con 0x7fb07c15d7b0 2024-04-06T13:48:49.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:49.822+0000 7fb071ffb700 1 --2- 172.21.15.189:0/1965073909 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb06006af20 0x7fb06006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:49.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:49.822+0000 7fb071ffb700 1 -- 172.21.15.189:0/1965073909 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fb060027f50 con 0x7fb06006af20 2024-04-06T13:48:49.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:49.823+0000 7fb081716700 1 --2- 172.21.15.189:0/1965073909 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb06006af20 0x7fb06006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:49.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:49.825+0000 7fb081716700 1 --2- 172.21.15.189:0/1965073909 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb06006af20 0x7fb06006d3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb078009800 tx=0x7fb078009210 comp rx=0 tx=0).ready entity=mgr.9623 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:49.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:49.828+0000 7fb071ffb700 1 -- 172.21.15.189:0/1965073909 <== mgr.9623 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: -22 filesystem argument is required when there is more than one file system) v1 ==== 79+0+0 (secure 0 0 0) 0x7fb060027f50 con 0x7fb06006af20 2024-04-06T13:48:49.830 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:49.827+0000 7f92bcc8f700 -1 mgr.server reply reply (22) Invalid argument filesystem argument is required when there is more than one file system 2024-04-06T13:48:49.832 INFO:teuthology.orchestra.run.smithi189.stderr:Error EINVAL: filesystem argument is required when there is more than one file system 2024-04-06T13:48:49.837 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-06T13:48:49.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:48:49.838 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2024-04-06T13:48:49.860 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-06T13:48:50.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.028+0000 7f3122b6b700 1 Processor -- start 2024-04-06T13:48:50.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.028+0000 7f3122b6b700 1 -- start start 2024-04-06T13:48:50.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.028+0000 7f3122b6b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f311c0d11b0 0x7f311c0d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:50.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.028+0000 7f3122b6b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0cfa00 0x7f311c0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:50.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.028+0000 7f3122b6b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c0d1ad0 con 0x7f311c0cb870 2024-04-06T13:48:50.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.028+0000 7f3122b6b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f311c0ccdc0 con 0x7f311c0d11b0 2024-04-06T13:48:50.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.028+0000 7f3122b6b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0d0b50 con 0x7f311c0cfa00 2024-04-06T13:48:50.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.029+0000 7f3120907700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f311c0d11b0 0x7f311c0d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:50.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.029+0000 7f3120907700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f311c0d11b0 0x7f311c0d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39426/0 (socket says 172.21.15.189:39426) 2024-04-06T13:48:50.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.029+0000 7f3120907700 1 -- 172.21.15.189:0/3590190543 learned_addr learned my addr 172.21.15.189:0/3590190543 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:50.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.029+0000 7f3121108700 1 --2- 172.21.15.189:0/3590190543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0cfa00 0x7f311c0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:50.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.029+0000 7f3120907700 1 -- 172.21.15.189:0/3590190543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0cfa00 msgr2=0x7f311c0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:50.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.029+0000 7f3120907700 1 --2- 172.21.15.189:0/3590190543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0cfa00 0x7f311c0cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:50.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.029+0000 7f3120907700 1 -- 172.21.15.189:0/3590190543 >> v1:172.21.15.192:6789/0 conn(0x7f311c0cb870 legacy=0x7f311c15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:50.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.029+0000 7f3120907700 1 -- 172.21.15.189:0/3590190543 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c0cbd80 con 0x7f311c0d11b0 2024-04-06T13:48:50.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.029+0000 7f3121108700 1 --2- 172.21.15.189:0/3590190543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0cfa00 0x7f311c0cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:50.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.029+0000 7f3120907700 1 --2- 172.21.15.189:0/3590190543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f311c0d11b0 0x7f311c0d1590 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f31100098f0 tx=0x7f3110009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a59135eaaa934c4e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:50.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.029+0000 7f311b7fe700 1 -- 172.21.15.189:0/3590190543 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110004230 con 0x7f311c0d11b0 2024-04-06T13:48:50.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.029+0000 7f311b7fe700 1 -- 172.21.15.189:0/3590190543 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3110004390 con 0x7f311c0d11b0 2024-04-06T13:48:50.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.029+0000 7f311b7fe700 1 -- 172.21.15.189:0/3590190543 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31100036a0 con 0x7f311c0d11b0 2024-04-06T13:48:50.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.030+0000 7f3122b6b700 1 -- 172.21.15.189:0/3590190543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f311c0d11b0 msgr2=0x7f311c0d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:50.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.030+0000 7f3122b6b700 1 --2- 172.21.15.189:0/3590190543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f311c0d11b0 0x7f311c0d1590 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f31100098f0 tx=0x7f3110009c00 comp rx=0 tx=0).stop 2024-04-06T13:48:50.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.030+0000 7f3122b6b700 1 -- 172.21.15.189:0/3590190543 shutdown_connections 2024-04-06T13:48:50.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.030+0000 7f3122b6b700 1 --2- 172.21.15.189:0/3590190543 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0cfa00 0x7f311c0cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:50.034 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.030+0000 7f3122b6b700 1 --2- 172.21.15.189:0/3590190543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f311c0d11b0 0x7f311c0d1590 unknown :-1 s=CLOSED pgs=2780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:50.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.030+0000 7f3122b6b700 1 -- 172.21.15.189:0/3590190543 >> 172.21.15.189:0/3590190543 conn(0x7f311c0b2db0 msgr2=0x7f311c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:50.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.030+0000 7f3122b6b700 1 -- 172.21.15.189:0/3590190543 shutdown_connections 2024-04-06T13:48:50.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.030+0000 7f3122b6b700 1 -- 172.21.15.189:0/3590190543 wait complete. 2024-04-06T13:48:50.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.031+0000 7f3122b6b700 1 Processor -- start 2024-04-06T13:48:50.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.031+0000 7f3122b6b700 1 -- start start 2024-04-06T13:48:50.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f3122b6b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f311c0cb870 0x7f311c15f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:50.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f3122b6b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311c0cfa00 0x7f311c154ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:50.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f3122b6b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0d11b0 0x7f311c1e7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:50.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f3122b6b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c155410 con 0x7f311c0cb870 2024-04-06T13:48:50.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f3122b6b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c154bf0 con 0x7f311c0cfa00 2024-04-06T13:48:50.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f3122b6b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c1569c0 con 0x7f311c0d11b0 2024-04-06T13:48:50.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f3120907700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f311c0cb870 0x7f311c15f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:50.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f3120907700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f311c0cb870 0x7f311c15f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39434/0 (socket says 172.21.15.189:39434) 2024-04-06T13:48:50.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f3120907700 1 -- 172.21.15.189:0/1446153618 learned_addr learned my addr 172.21.15.189:0/1446153618 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:50.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f3121108700 1 --2- 172.21.15.189:0/1446153618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0d11b0 0x7f311c1e7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:50.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f311bfff700 1 --2- 172.21.15.189:0/1446153618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311c0cfa00 0x7f311c154ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:50.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f3120907700 1 -- 172.21.15.189:0/1446153618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0d11b0 msgr2=0x7f311c1e7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:50.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f3120907700 1 --2- 172.21.15.189:0/1446153618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0d11b0 0x7f311c1e7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:50.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f3120907700 1 -- 172.21.15.189:0/1446153618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311c0cfa00 msgr2=0x7f311c154ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:50.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f3120907700 1 --2- 172.21.15.189:0/1446153618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311c0cfa00 0x7f311c154ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:50.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f3120907700 1 -- 172.21.15.189:0/1446153618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3110009550 con 0x7f311c0cb870 2024-04-06T13:48:50.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f3121108700 1 --2- 172.21.15.189:0/1446153618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0d11b0 0x7f311c1e7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:50.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f311bfff700 1 --2- 172.21.15.189:0/1446153618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311c0cfa00 0x7f311c154ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:50.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f3120907700 1 --2- 172.21.15.189:0/1446153618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f311c0cb870 0x7f311c15f180 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f311c0677e0 tx=0x7f3110004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:50.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.032+0000 7f31197fa700 1 -- 172.21.15.189:0/1446153618 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110003e70 con 0x7f311c0cb870 2024-04-06T13:48:50.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.033+0000 7f31197fa700 1 -- 172.21.15.189:0/1446153618 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311001f070 con 0x7f311c0cb870 2024-04-06T13:48:50.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.033+0000 7f3122b6b700 1 -- 172.21.15.189:0/1446153618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311c156ca0 con 0x7f311c0cb870 2024-04-06T13:48:50.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.033+0000 7f31197fa700 1 -- 172.21.15.189:0/1446153618 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311000b8c0 con 0x7f311c0cb870 2024-04-06T13:48:50.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.033+0000 7f3122b6b700 1 -- 172.21.15.189:0/1446153618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311c1590e0 con 0x7f311c0cb870 2024-04-06T13:48:50.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.033+0000 7f3122b6b700 1 -- 172.21.15.189:0/1446153618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311c159270 con 0x7f311c0cb870 2024-04-06T13:48:50.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.034+0000 7f31197fa700 1 -- 172.21.15.189:0/1446153618 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 154) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f3110021030 con 0x7f311c0cb870 2024-04-06T13:48:50.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.034+0000 7f31197fa700 1 --2- 172.21.15.189:0/1446153618 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f310006ab30 0x7f310006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:50.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.035+0000 7f311bfff700 1 --2- 172.21.15.189:0/1446153618 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f310006ab30 0x7f310006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:50.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.035+0000 7f31197fa700 1 -- 172.21.15.189:0/1446153618 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7395+0+0 (secure 0 0 0) 0x7f311008ade0 con 0x7f311c0cb870 2024-04-06T13:48:50.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.035+0000 7f311bfff700 1 --2- 172.21.15.189:0/1446153618 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f310006ab30 0x7f310006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3108006fd0 tx=0x7f3108008040 comp rx=0 tx=0).ready entity=mgr.9623 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:50.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.039+0000 7f31197fa700 1 -- 172.21.15.189:0/1446153618 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3110055b90 con 0x7f311c0cb870 2024-04-06T13:48:50.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.288+0000 7f3122b6b700 1 -- 172.21.15.189:0/1446153618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f311c159270 con 0x7f311c0cb870 2024-04-06T13:48:50.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.832+0000 7f31197fa700 1 -- 172.21.15.189:0/1446153618 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 155) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f3110055f60 con 0x7f311c0cb870 2024-04-06T13:48:50.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.833+0000 7f31197fa700 1 -- 172.21.15.189:0/1446153618 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v155) v1 ==== 93+0+0 (secure 0 0 0) 0x7f3110070510 con 0x7f311c0cb870 2024-04-06T13:48:50.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.835+0000 7f3122b6b700 1 -- 172.21.15.189:0/1446153618 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f310006ab30 msgr2=0x7f310006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:50.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.835+0000 7f3122b6b700 1 --2- 172.21.15.189:0/1446153618 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f310006ab30 0x7f310006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3108006fd0 tx=0x7f3108008040 comp rx=0 tx=0).stop 2024-04-06T13:48:50.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.835+0000 7f3122b6b700 1 -- 172.21.15.189:0/1446153618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f311c0cb870 msgr2=0x7f311c15f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:50.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.835+0000 7f3122b6b700 1 --2- 172.21.15.189:0/1446153618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f311c0cb870 0x7f311c15f180 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f311c0677e0 tx=0x7f3110004750 comp rx=0 tx=0).stop 2024-04-06T13:48:50.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.836+0000 7f3122b6b700 1 -- 172.21.15.189:0/1446153618 shutdown_connections 2024-04-06T13:48:50.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.836+0000 7f3122b6b700 1 --2- 172.21.15.189:0/1446153618 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f310006ab30 0x7f310006cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:50.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.836+0000 7f3122b6b700 1 --2- 172.21.15.189:0/1446153618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0d11b0 0x7f311c1e7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:50.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.836+0000 7f3122b6b700 1 --2- 172.21.15.189:0/1446153618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f311c0cb870 0x7f311c15f180 unknown :-1 s=CLOSED pgs=2781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:50.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.836+0000 7f3122b6b700 1 --2- 172.21.15.189:0/1446153618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311c0cfa00 0x7f311c154ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:50.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.836+0000 7f3122b6b700 1 -- 172.21.15.189:0/1446153618 >> 172.21.15.189:0/1446153618 conn(0x7f311c0b2db0 msgr2=0x7f311c15a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:50.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.836+0000 7f3122b6b700 1 -- 172.21.15.189:0/1446153618 shutdown_connections 2024-04-06T13:48:50.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:50.836+0000 7f3122b6b700 1 -- 172.21.15.189:0/1446153618 wait complete. 2024-04-06T13:48:50.853 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T13:48:50.853 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:48:50.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:48:50.949 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T13:48:50.950 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:48:50.950 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:48:50.967 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:48:50.968 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:48:51.025 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:48:51.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.200+0000 7f6f3c687700 1 Processor -- start 2024-04-06T13:48:51.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.200+0000 7f6f3c687700 1 -- start start 2024-04-06T13:48:51.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f3c687700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6f3415d7b0 0x7f6f340b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f3c687700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f3406a730 0x7f6f3406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f3c687700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6f3415ddf0 0x7f6f3415e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f3c687700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f3406b0f0 con 0x7f6f3415d7b0 2024-04-06T13:48:51.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f3c687700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f340b3990 con 0x7f6f3415ddf0 2024-04-06T13:48:51.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f3c687700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f340b3060 con 0x7f6f3406a730 2024-04-06T13:48:51.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f3a423700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6f3415d7b0 0x7f6f340b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:51.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f3a423700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6f3415d7b0 0x7f6f340b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39450/0 (socket says 172.21.15.189:39450) 2024-04-06T13:48:51.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f3a423700 1 -- 172.21.15.189:0/2586053349 learned_addr learned my addr 172.21.15.189:0/2586053349 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:51.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f39c22700 1 --2- 172.21.15.189:0/2586053349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f3406a730 0x7f6f3406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:51.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f3ac24700 1 --2- 172.21.15.189:0/2586053349 >> v2:172.21.15.192:3300/0 conn(0x7f6f3415ddf0 0x7f6f3415e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:51.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f3a423700 1 -- 172.21.15.189:0/2586053349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f3406a730 msgr2=0x7f6f3406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:51.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f3a423700 1 --2- 172.21.15.189:0/2586053349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f3406a730 0x7f6f3406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f3a423700 1 -- 172.21.15.189:0/2586053349 >> v2:172.21.15.192:3300/0 conn(0x7f6f3415ddf0 msgr2=0x7f6f3415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:51.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f3a423700 1 --2- 172.21.15.189:0/2586053349 >> v2:172.21.15.192:3300/0 conn(0x7f6f3415ddf0 0x7f6f3415e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f3a423700 1 -- 172.21.15.189:0/2586053349 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f340b3340 con 0x7f6f3415d7b0 2024-04-06T13:48:51.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f39c22700 1 --2- 172.21.15.189:0/2586053349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f3406a730 0x7f6f3406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:51.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f3a423700 1 --2- 172.21.15.189:0/2586053349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f3415d7b0 0x7f6f340b3e20 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7f6f340b4360 tx=0x7f6f30009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6f157ba41aa4232 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:51.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f39421700 1 -- 172.21.15.189:0/2586053349 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3000b870 con 0x7f6f3415d7b0 2024-04-06T13:48:51.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f39421700 1 -- 172.21.15.189:0/2586053349 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3000b9d0 con 0x7f6f3415d7b0 2024-04-06T13:48:51.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.201+0000 7f6f39421700 1 -- 172.21.15.189:0/2586053349 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3000bd60 con 0x7f6f3415d7b0 2024-04-06T13:48:51.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.202+0000 7f6f3c687700 1 -- 172.21.15.189:0/2586053349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f3415d7b0 msgr2=0x7f6f340b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:51.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.202+0000 7f6f3c687700 1 --2- 172.21.15.189:0/2586053349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f3415d7b0 0x7f6f340b3e20 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7f6f340b4360 tx=0x7f6f30009bd0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.202+0000 7f6f3c687700 1 -- 172.21.15.189:0/2586053349 shutdown_connections 2024-04-06T13:48:51.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.202+0000 7f6f3c687700 1 --2- 172.21.15.189:0/2586053349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f3415d7b0 0x7f6f340b3e20 unknown :-1 s=CLOSED pgs=2785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.202+0000 7f6f3c687700 1 --2- 172.21.15.189:0/2586053349 >> v2:172.21.15.192:3300/0 conn(0x7f6f3415ddf0 0x7f6f3415e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.202+0000 7f6f3c687700 1 --2- 172.21.15.189:0/2586053349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f3406a730 0x7f6f3406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.202+0000 7f6f3c687700 1 -- 172.21.15.189:0/2586053349 >> 172.21.15.189:0/2586053349 conn(0x7f6f340d00b0 msgr2=0x7f6f340d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:51.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.202+0000 7f6f3c687700 1 -- 172.21.15.189:0/2586053349 shutdown_connections 2024-04-06T13:48:51.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.202+0000 7f6f3c687700 1 -- 172.21.15.189:0/2586053349 wait complete. 2024-04-06T13:48:51.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.203+0000 7f6f3c687700 1 Processor -- start 2024-04-06T13:48:51.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.203+0000 7f6f3c687700 1 -- start start 2024-04-06T13:48:51.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.204+0000 7f6f3c687700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f3406a730 0x7f6f340b1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.204+0000 7f6f3c687700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f3415d7b0 0x7f6f340aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.204+0000 7f6f3c687700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f3415ddf0 0x7f6f341502f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.204+0000 7f6f3c687700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f340b1b50 con 0x7f6f3406a730 2024-04-06T13:48:51.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.204+0000 7f6f3c687700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f34153420 con 0x7f6f3415d7b0 2024-04-06T13:48:51.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.204+0000 7f6f3c687700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f340af0c0 con 0x7f6f3415ddf0 2024-04-06T13:48:51.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.204+0000 7f6f3a423700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f3406a730 0x7f6f340b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:51.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.204+0000 7f6f3a423700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f3406a730 0x7f6f340b1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39460/0 (socket says 172.21.15.189:39460) 2024-04-06T13:48:51.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.204+0000 7f6f3a423700 1 -- 172.21.15.189:0/3592807074 learned_addr learned my addr 172.21.15.189:0/3592807074 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:51.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.204+0000 7f6f3ac24700 1 --2- 172.21.15.189:0/3592807074 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f3415ddf0 0x7f6f341502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:51.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.204+0000 7f6f39c22700 1 --2- 172.21.15.189:0/3592807074 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f3415d7b0 0x7f6f340aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:51.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.204+0000 7f6f3a423700 1 -- 172.21.15.189:0/3592807074 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f3415ddf0 msgr2=0x7f6f341502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:51.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.204+0000 7f6f3a423700 1 --2- 172.21.15.189:0/3592807074 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f3415ddf0 0x7f6f341502f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.205+0000 7f6f3a423700 1 -- 172.21.15.189:0/3592807074 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f3415d7b0 msgr2=0x7f6f340aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:51.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.205+0000 7f6f3a423700 1 --2- 172.21.15.189:0/3592807074 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f3415d7b0 0x7f6f340aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.205+0000 7f6f3a423700 1 -- 172.21.15.189:0/3592807074 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f30009550 con 0x7f6f3406a730 2024-04-06T13:48:51.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.205+0000 7f6f3ac24700 1 --2- 172.21.15.189:0/3592807074 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f3415ddf0 0x7f6f341502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:51.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.205+0000 7f6f3a423700 1 --2- 172.21.15.189:0/3592807074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f3406a730 0x7f6f340b1200 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7f6f30000c00 tx=0x7f6f30004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:51.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.205+0000 7f6f2affd700 1 -- 172.21.15.189:0/3592807074 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f30004250 con 0x7f6f3406a730 2024-04-06T13:48:51.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.205+0000 7f6f2affd700 1 -- 172.21.15.189:0/3592807074 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3001f070 con 0x7f6f3406a730 2024-04-06T13:48:51.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.205+0000 7f6f3c687700 1 -- 172.21.15.189:0/3592807074 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f34155f90 con 0x7f6f3406a730 2024-04-06T13:48:51.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.205+0000 7f6f2affd700 1 -- 172.21.15.189:0/3592807074 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f300036a0 con 0x7f6f3406a730 2024-04-06T13:48:51.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.205+0000 7f6f3c687700 1 -- 172.21.15.189:0/3592807074 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f341f61a0 con 0x7f6f3406a730 2024-04-06T13:48:51.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.206+0000 7f6f3c687700 1 -- 172.21.15.189:0/3592807074 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f341f6000 con 0x7f6f3406a730 2024-04-06T13:48:51.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.207+0000 7f6f2affd700 1 -- 172.21.15.189:0/3592807074 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 155) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f6f30021030 con 0x7f6f3406a730 2024-04-06T13:48:51.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.207+0000 7f6f2affd700 1 --2- 172.21.15.189:0/3592807074 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f1806aae0 0x7f6f1806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.207+0000 7f6f39c22700 1 -- 172.21.15.189:0/3592807074 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f1806aae0 msgr2=0x7f6f1806cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:51.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.207+0000 7f6f2affd700 1 -- 172.21.15.189:0/3592807074 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7395+0+0 (secure 0 0 0) 0x7f6f3008aed0 con 0x7f6f3406a730 2024-04-06T13:48:51.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.207+0000 7f6f39c22700 1 --2- 172.21.15.189:0/3592807074 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f1806aae0 0x7f6f1806cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:48:51.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.215+0000 7f6f2affd700 1 -- 172.21.15.189:0/3592807074 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6f30055da0 con 0x7f6f3406a730 2024-04-06T13:48:51.222 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:51.221+0000 7f3525abf200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:48:51.254 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:51.254+0000 7f0bf6538200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:48:51.300 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:51.300+0000 7f3525abf200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:48:51.339 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:51.338+0000 7f0bf6538200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:48:51.373 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:51.373+0000 7f3525abf200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:48:51.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.408+0000 7f6f39c22700 1 -- 172.21.15.189:0/3592807074 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f1806aae0 msgr2=0x7f6f1806cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:51.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.408+0000 7f6f39c22700 1 --2- 172.21.15.189:0/3592807074 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f1806aae0 0x7f6f1806cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:48:51.415 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:51.414+0000 7f0bf6538200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:48:51.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.422+0000 7f6f3c687700 1 -- 172.21.15.189:0/3592807074 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f341f6000 con 0x7f6f3406a730 2024-04-06T13:48:51.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.425+0000 7f6f2affd700 1 -- 172.21.15.189:0/3592807074 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v253) v1 ==== 74+0+20762 (secure 0 0 0) 0x7f6f341f6000 con 0x7f6f3406a730 2024-04-06T13:48:51.426 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:51.426 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":253,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:48:48.812449+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:48:17.447247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-04-06T13:48:18.646229+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:48:33.912100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:48:35.170642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/85704189":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2120970213":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/381758424":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2107771319":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2145696658":"2024-04-07T13:48:48.812402+0000","172.21.15.189:6833/111704":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/3440291825":"2024-04-07T13:48:48.812402+0000","172.21.15.189:6832/111704":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/485512536":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/643921686":"2024-04-07T13:48:48.812402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:48:51.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.427+0000 7f6f3c687700 1 -- 172.21.15.189:0/3592807074 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f1806aae0 msgr2=0x7f6f1806cfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:48:51.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.427+0000 7f6f3c687700 1 --2- 172.21.15.189:0/3592807074 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f1806aae0 0x7f6f1806cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.427+0000 7f6f3c687700 1 -- 172.21.15.189:0/3592807074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f3406a730 msgr2=0x7f6f340b1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:51.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.427+0000 7f6f3c687700 1 --2- 172.21.15.189:0/3592807074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f3406a730 0x7f6f340b1200 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7f6f30000c00 tx=0x7f6f30004750 comp rx=0 tx=0).stop 2024-04-06T13:48:51.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.427+0000 7f6f3c687700 1 -- 172.21.15.189:0/3592807074 shutdown_connections 2024-04-06T13:48:51.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.427+0000 7f6f3c687700 1 --2- 172.21.15.189:0/3592807074 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6f1806aae0 0x7f6f1806cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.427+0000 7f6f3c687700 1 --2- 172.21.15.189:0/3592807074 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f3415ddf0 0x7f6f341502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.427+0000 7f6f3c687700 1 --2- 172.21.15.189:0/3592807074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f3406a730 0x7f6f340b1200 unknown :-1 s=CLOSED pgs=2786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.427+0000 7f6f3c687700 1 --2- 172.21.15.189:0/3592807074 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f3415d7b0 0x7f6f340aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.427+0000 7f6f3c687700 1 -- 172.21.15.189:0/3592807074 >> 172.21.15.189:0/3592807074 conn(0x7f6f340d00b0 msgr2=0x7f6f3406eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:51.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.427+0000 7f6f3c687700 1 -- 172.21.15.189:0/3592807074 shutdown_connections 2024-04-06T13:48:51.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.427+0000 7f6f3c687700 1 -- 172.21.15.189:0/3592807074 wait complete. 2024-04-06T13:48:51.440 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:51.440+0000 7f3525abf200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:48:51.441 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T13:48:17.447247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '247', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T13:48:18.646229+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-04-06T13:48:33.912100+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '251', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2024-04-06T13:48:35.170642+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '252', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:48:51.442 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:48:51.489 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:51.488+0000 7f0bf6538200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:48:51.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.570+0000 7f39fcb0d700 1 Processor -- start 2024-04-06T13:48:51.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.571+0000 7f39fcb0d700 1 -- start start 2024-04-06T13:48:51.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.571+0000 7f39fcb0d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f39f815d7b0 0x7f39f80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.571+0000 7f39fcb0d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39f806a730 0x7f39f806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.572+0000 7f39fcb0d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f39f815ddf0 0x7f39f815e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.572+0000 7f39fcb0d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39f806b0f0 con 0x7f39f815ddf0 2024-04-06T13:48:51.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.572+0000 7f39fcb0d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39f80b3990 con 0x7f39f815d7b0 2024-04-06T13:48:51.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.572+0000 7f39fcb0d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f80b3060 con 0x7f39f806a730 2024-04-06T13:48:51.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.572+0000 7f39f6d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f39f815ddf0 0x7f39f815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:51.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.572+0000 7f39f6d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f39f815ddf0 0x7f39f815e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39462/0 (socket says 172.21.15.189:39462) 2024-04-06T13:48:51.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.572+0000 7f39f6d9d700 1 -- 172.21.15.189:0/899204532 learned_addr learned my addr 172.21.15.189:0/899204532 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:51.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.572+0000 7f39f5d9b700 1 --2- 172.21.15.189:0/899204532 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39f806a730 0x7f39f806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:51.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.572+0000 7f39f659c700 1 --2- 172.21.15.189:0/899204532 >> v2:172.21.15.192:3300/0 conn(0x7f39f815d7b0 0x7f39f80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:51.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.572+0000 7f39f6d9d700 1 -- 172.21.15.189:0/899204532 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39f806a730 msgr2=0x7f39f806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:51.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.572+0000 7f39f6d9d700 1 --2- 172.21.15.189:0/899204532 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39f806a730 0x7f39f806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.572+0000 7f39f6d9d700 1 -- 172.21.15.189:0/899204532 >> v2:172.21.15.192:3300/0 conn(0x7f39f815d7b0 msgr2=0x7f39f80b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:51.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.572+0000 7f39f6d9d700 1 --2- 172.21.15.189:0/899204532 >> v2:172.21.15.192:3300/0 conn(0x7f39f815d7b0 0x7f39f80b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.572+0000 7f39f6d9d700 1 -- 172.21.15.189:0/899204532 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f80b3340 con 0x7f39f815ddf0 2024-04-06T13:48:51.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.572+0000 7f39f6d9d700 1 --2- 172.21.15.189:0/899204532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f815ddf0 0x7f39f815e280 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f39f80b6200 tx=0x7f39f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8f2a6fec19bdfad server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:51.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.572+0000 7f39f559a700 1 -- 172.21.15.189:0/899204532 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f000e070 con 0x7f39f815ddf0 2024-04-06T13:48:51.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.572+0000 7f39f559a700 1 -- 172.21.15.189:0/899204532 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39f000b910 con 0x7f39f815ddf0 2024-04-06T13:48:51.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.572+0000 7f39f559a700 1 -- 172.21.15.189:0/899204532 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f000bcc0 con 0x7f39f815ddf0 2024-04-06T13:48:51.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.573+0000 7f39fcb0d700 1 -- 172.21.15.189:0/899204532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f815ddf0 msgr2=0x7f39f815e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:51.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.573+0000 7f39fcb0d700 1 --2- 172.21.15.189:0/899204532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f815ddf0 0x7f39f815e280 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f39f80b6200 tx=0x7f39f000b330 comp rx=0 tx=0).stop 2024-04-06T13:48:51.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.573+0000 7f39fcb0d700 1 -- 172.21.15.189:0/899204532 shutdown_connections 2024-04-06T13:48:51.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.573+0000 7f39fcb0d700 1 --2- 172.21.15.189:0/899204532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f815ddf0 0x7f39f815e280 unknown :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.573+0000 7f39fcb0d700 1 --2- 172.21.15.189:0/899204532 >> v2:172.21.15.192:3300/0 conn(0x7f39f815d7b0 0x7f39f80b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.573+0000 7f39fcb0d700 1 --2- 172.21.15.189:0/899204532 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39f806a730 0x7f39f806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.573+0000 7f39fcb0d700 1 -- 172.21.15.189:0/899204532 >> 172.21.15.189:0/899204532 conn(0x7f39f80d00b0 msgr2=0x7f39f80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:51.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.573+0000 7f39fcb0d700 1 -- 172.21.15.189:0/899204532 shutdown_connections 2024-04-06T13:48:51.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.573+0000 7f39fcb0d700 1 -- 172.21.15.189:0/899204532 wait complete. 2024-04-06T13:48:51.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.574+0000 7f39fcb0d700 1 Processor -- start 2024-04-06T13:48:51.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.575+0000 7f39fcb0d700 1 -- start start 2024-04-06T13:48:51.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.575+0000 7f39fcb0d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39f806a730 0x7f39f81551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.575+0000 7f39fcb0d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39f815d7b0 0x7f39f80ad960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.575+0000 7f39fcb0d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f815ddf0 0x7f39f80adea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.575+0000 7f39fcb0d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f80aff00 con 0x7f39f815ddf0 2024-04-06T13:48:51.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.575+0000 7f39fcb0d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f80cd120 con 0x7f39f815d7b0 2024-04-06T13:48:51.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.575+0000 7f39fcb0d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f80cd400 con 0x7f39f806a730 2024-04-06T13:48:51.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.575+0000 7f39f6d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f815ddf0 0x7f39f80adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:51.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.575+0000 7f39f6d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f815ddf0 0x7f39f80adea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39470/0 (socket says 172.21.15.189:39470) 2024-04-06T13:48:51.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.575+0000 7f39f6d9d700 1 -- 172.21.15.189:0/398442168 learned_addr learned my addr 172.21.15.189:0/398442168 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:51.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.576+0000 7f39f659c700 1 --2- 172.21.15.189:0/398442168 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39f806a730 0x7f39f81551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:51.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.576+0000 7f39f5d9b700 1 --2- 172.21.15.189:0/398442168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39f815d7b0 0x7f39f80ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:51.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.576+0000 7f39f6d9d700 1 -- 172.21.15.189:0/398442168 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39f806a730 msgr2=0x7f39f81551c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:51.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.576+0000 7f39f6d9d700 1 --2- 172.21.15.189:0/398442168 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39f806a730 0x7f39f81551c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.576+0000 7f39f6d9d700 1 -- 172.21.15.189:0/398442168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39f815d7b0 msgr2=0x7f39f80ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:51.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.576+0000 7f39f6d9d700 1 --2- 172.21.15.189:0/398442168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39f815d7b0 0x7f39f80ad960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.576+0000 7f39f6d9d700 1 -- 172.21.15.189:0/398442168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f0009b30 con 0x7f39f815ddf0 2024-04-06T13:48:51.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.576+0000 7f39f6d9d700 1 --2- 172.21.15.189:0/398442168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f815ddf0 0x7f39f80adea0 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7f39f806c820 tx=0x7f39f0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:51.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.576+0000 7f39e6ffd700 1 -- 172.21.15.189:0/398442168 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f00099b0 con 0x7f39f815ddf0 2024-04-06T13:48:51.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.576+0000 7f39e6ffd700 1 -- 172.21.15.189:0/398442168 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39f000ce80 con 0x7f39f815ddf0 2024-04-06T13:48:51.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.576+0000 7f39e6ffd700 1 -- 172.21.15.189:0/398442168 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f0004490 con 0x7f39f815ddf0 2024-04-06T13:48:51.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.576+0000 7f39fcb0d700 1 -- 172.21.15.189:0/398442168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f80cd6e0 con 0x7f39f815ddf0 2024-04-06T13:48:51.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.576+0000 7f39fcb0d700 1 -- 172.21.15.189:0/398442168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f80cc460 con 0x7f39f815ddf0 2024-04-06T13:48:51.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.577+0000 7f39fcb0d700 1 -- 172.21.15.189:0/398442168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f80b00b0 con 0x7f39f815ddf0 2024-04-06T13:48:51.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.578+0000 7f39e6ffd700 1 -- 172.21.15.189:0/398442168 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 155) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f39f00045f0 con 0x7f39f815ddf0 2024-04-06T13:48:51.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.578+0000 7f39e6ffd700 1 --2- 172.21.15.189:0/398442168 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f39d806aae0 0x7f39d806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.578+0000 7f39f659c700 1 -- 172.21.15.189:0/398442168 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f39d806aae0 msgr2=0x7f39d806cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:51.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.578+0000 7f39e6ffd700 1 -- 172.21.15.189:0/398442168 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7395+0+0 (secure 0 0 0) 0x7f39f008a7f0 con 0x7f39f815ddf0 2024-04-06T13:48:51.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.578+0000 7f39f659c700 1 --2- 172.21.15.189:0/398442168 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f39d806aae0 0x7f39d806cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:48:51.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.582+0000 7f39e6ffd700 1 -- 172.21.15.189:0/398442168 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f39f0055670 con 0x7f39f815ddf0 2024-04-06T13:48:51.616 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:51.616+0000 7f3525abf200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:48:51.677 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:51.676+0000 7f0bf6538200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:48:51.774 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:51.774+0000 7f3525abf200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:48:51.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.779+0000 7f39f659c700 1 -- 172.21.15.189:0/398442168 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f39d806aae0 msgr2=0x7f39d806cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:51.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.779+0000 7f39f659c700 1 --2- 172.21.15.189:0/398442168 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f39d806aae0 0x7f39d806cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:48:51.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.785+0000 7f39fcb0d700 1 -- 172.21.15.189:0/398442168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f39f8067bb0 con 0x7f39f815ddf0 2024-04-06T13:48:51.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.786+0000 7f39e6ffd700 1 -- 172.21.15.189:0/398442168 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v253) v1 ==== 93+0+10 (secure 0 0 0) 0x7f39f002a020 con 0x7f39f815ddf0 2024-04-06T13:48:51.787 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:48:51.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.788+0000 7f39fcb0d700 1 -- 172.21.15.189:0/398442168 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f39d806aae0 msgr2=0x7f39d806cfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:48:51.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.788+0000 7f39fcb0d700 1 --2- 172.21.15.189:0/398442168 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f39d806aae0 0x7f39d806cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.788+0000 7f39fcb0d700 1 -- 172.21.15.189:0/398442168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f815ddf0 msgr2=0x7f39f80adea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:51.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.788+0000 7f39fcb0d700 1 --2- 172.21.15.189:0/398442168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f815ddf0 0x7f39f80adea0 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7f39f806c820 tx=0x7f39f0004ad0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.788+0000 7f39fcb0d700 1 -- 172.21.15.189:0/398442168 shutdown_connections 2024-04-06T13:48:51.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.788+0000 7f39fcb0d700 1 --2- 172.21.15.189:0/398442168 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f39d806aae0 0x7f39d806cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.788+0000 7f39fcb0d700 1 --2- 172.21.15.189:0/398442168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f815ddf0 0x7f39f80adea0 unknown :-1 s=CLOSED pgs=2788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.788+0000 7f39fcb0d700 1 --2- 172.21.15.189:0/398442168 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39f806a730 0x7f39f81551c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.788+0000 7f39fcb0d700 1 --2- 172.21.15.189:0/398442168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39f815d7b0 0x7f39f80ad960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.788+0000 7f39fcb0d700 1 -- 172.21.15.189:0/398442168 >> 172.21.15.189:0/398442168 conn(0x7f39f80d00b0 msgr2=0x7f39f80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:51.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.788+0000 7f39fcb0d700 1 -- 172.21.15.189:0/398442168 shutdown_connections 2024-04-06T13:48:51.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.788+0000 7f39fcb0d700 1 -- 172.21.15.189:0/398442168 wait complete. 2024-04-06T13:48:51.803 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T13:48:51.849 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:51.848+0000 7f0bf6538200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:48:51.887 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:51.887+0000 7f3525abf200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:48:51.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.934+0000 7f1acaaa0700 1 Processor -- start 2024-04-06T13:48:51.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.934+0000 7f1acaaa0700 1 -- start start 2024-04-06T13:48:51.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.934+0000 7f1acaaa0700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1ac415d7e0 0x7f1ac40d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.935+0000 7f1acaaa0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac40d1cc0 0x7f1ac40b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.935+0000 7f1acaaa0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac40cdf80 con 0x7f1ac40d0890 2024-04-06T13:48:51.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.935+0000 7f1acaaa0700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac40d3230 con 0x7f1ac415d7e0 2024-04-06T13:48:51.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.935+0000 7f1acaaa0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac40d2da0 con 0x7f1ac40d1cc0 2024-04-06T13:48:51.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.935+0000 7f1ac903d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac40d1cc0 0x7f1ac40b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:51.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.935+0000 7f1ac883c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1ac415d7e0 0x7f1ac40d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:51.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.935+0000 7f1ac903d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac40d1cc0 0x7f1ac40b66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:58214/0 (socket says 172.21.15.189:58214) 2024-04-06T13:48:51.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.935+0000 7f1ac883c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1ac415d7e0 0x7f1ac40d0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:41514/0 (socket says 172.21.15.189:41514) 2024-04-06T13:48:51.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.935+0000 7f1ac903d700 1 -- 172.21.15.189:0/1219117882 learned_addr learned my addr 172.21.15.189:0/1219117882 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:51.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.935+0000 7f1ac903d700 1 -- 172.21.15.189:0/1219117882 >> v2:172.21.15.192:3300/0 conn(0x7f1ac415d7e0 msgr2=0x7f1ac40d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:51.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.935+0000 7f1ac903d700 1 --2- 172.21.15.189:0/1219117882 >> v2:172.21.15.192:3300/0 conn(0x7f1ac415d7e0 0x7f1ac40d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.935+0000 7f1ac903d700 1 -- 172.21.15.189:0/1219117882 >> v1:172.21.15.192:6789/0 conn(0x7f1ac40d0890 legacy=0x7f1ac40d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:51.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.935+0000 7f1ac903d700 1 -- 172.21.15.189:0/1219117882 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac40d0cf0 con 0x7f1ac40d1cc0 2024-04-06T13:48:51.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.935+0000 7f1ac883c700 1 --2- 172.21.15.189:0/1219117882 >> v2:172.21.15.192:3300/0 conn(0x7f1ac415d7e0 0x7f1ac40d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:51.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.935+0000 7f1ac903d700 1 --2- 172.21.15.189:0/1219117882 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac40d1cc0 0x7f1ac40b66b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1abc009e50 tx=0x7f1abc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b82219e1ad1ceb65 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:51.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.936+0000 7f1ac37fe700 1 -- 172.21.15.189:0/1219117882 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc00e070 con 0x7f1ac40d1cc0 2024-04-06T13:48:51.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.936+0000 7f1ac37fe700 1 -- 172.21.15.189:0/1219117882 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1abc00b910 con 0x7f1ac40d1cc0 2024-04-06T13:48:51.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.936+0000 7f1ac37fe700 1 -- 172.21.15.189:0/1219117882 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc00bcc0 con 0x7f1ac40d1cc0 2024-04-06T13:48:51.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.936+0000 7f1acaaa0700 1 -- 172.21.15.189:0/1219117882 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac40d1cc0 msgr2=0x7f1ac40b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:51.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.936+0000 7f1acaaa0700 1 --2- 172.21.15.189:0/1219117882 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac40d1cc0 0x7f1ac40b66b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1abc009e50 tx=0x7f1abc00b330 comp rx=0 tx=0).stop 2024-04-06T13:48:51.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.936+0000 7f1acaaa0700 1 -- 172.21.15.189:0/1219117882 shutdown_connections 2024-04-06T13:48:51.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.936+0000 7f1acaaa0700 1 --2- 172.21.15.189:0/1219117882 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac40d1cc0 0x7f1ac40b66b0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.936+0000 7f1acaaa0700 1 --2- 172.21.15.189:0/1219117882 >> v2:172.21.15.192:3300/0 conn(0x7f1ac415d7e0 0x7f1ac40d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.936+0000 7f1acaaa0700 1 -- 172.21.15.189:0/1219117882 >> 172.21.15.189:0/1219117882 conn(0x7f1ac4072250 msgr2=0x7f1ac4072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:51.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.936+0000 7f1acaaa0700 1 -- 172.21.15.189:0/1219117882 shutdown_connections 2024-04-06T13:48:51.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.936+0000 7f1acaaa0700 1 -- 172.21.15.189:0/1219117882 wait complete. 2024-04-06T13:48:51.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.938+0000 7f1acaaa0700 1 Processor -- start 2024-04-06T13:48:51.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.938+0000 7f1acaaa0700 1 -- start start 2024-04-06T13:48:51.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.939+0000 7f1acaaa0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac40d0890 0x7f1ac4154b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.939+0000 7f1acaaa0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac40d1cc0 0x7f1ac40b4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.939+0000 7f1acaaa0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ac415d7e0 0x7f1ac40c7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.939+0000 7f1acaaa0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac4155070 con 0x7f1ac40d0890 2024-04-06T13:48:51.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.939+0000 7f1acaaa0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac40b2cf0 con 0x7f1ac415d7e0 2024-04-06T13:48:51.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.939+0000 7f1acaaa0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac40b2f40 con 0x7f1ac40d1cc0 2024-04-06T13:48:51.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.939+0000 7f1ac883c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac40d0890 0x7f1ac4154b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:51.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.939+0000 7f1ac883c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac40d0890 0x7f1ac4154b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39472/0 (socket says 172.21.15.189:39472) 2024-04-06T13:48:51.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.939+0000 7f1ac883c700 1 -- 172.21.15.189:0/476502152 learned_addr learned my addr 172.21.15.189:0/476502152 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:51.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.939+0000 7f1ac903d700 1 --2- 172.21.15.189:0/476502152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ac415d7e0 0x7f1ac40c7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:51.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.939+0000 7f1ac3fff700 1 --2- 172.21.15.189:0/476502152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac40d1cc0 0x7f1ac40b4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:51.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.939+0000 7f1ac883c700 1 -- 172.21.15.189:0/476502152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac40d1cc0 msgr2=0x7f1ac40b4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:51.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.939+0000 7f1ac883c700 1 --2- 172.21.15.189:0/476502152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac40d1cc0 0x7f1ac40b4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.939+0000 7f1ac883c700 1 -- 172.21.15.189:0/476502152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ac415d7e0 msgr2=0x7f1ac40c7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:51.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.939+0000 7f1ac883c700 1 --2- 172.21.15.189:0/476502152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ac415d7e0 0x7f1ac40c7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:51.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.939+0000 7f1ac883c700 1 -- 172.21.15.189:0/476502152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab80094a0 con 0x7f1ac40d0890 2024-04-06T13:48:51.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.939+0000 7f1ac903d700 1 --2- 172.21.15.189:0/476502152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ac415d7e0 0x7f1ac40c7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:51.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.940+0000 7f1ac883c700 1 --2- 172.21.15.189:0/476502152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac40d0890 0x7f1ac4154b30 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7f1ab800ebe0 tx=0x7f1ab800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:51.944 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.940+0000 7f1ac17fa700 1 -- 172.21.15.189:0/476502152 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab800c850 con 0x7f1ac40d0890 2024-04-06T13:48:51.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.940+0000 7f1ac17fa700 1 -- 172.21.15.189:0/476502152 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab800c9b0 con 0x7f1ac40d0890 2024-04-06T13:48:51.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.940+0000 7f1acaaa0700 1 -- 172.21.15.189:0/476502152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1abc009b30 con 0x7f1ac40d0890 2024-04-06T13:48:51.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.940+0000 7f1ac17fa700 1 -- 172.21.15.189:0/476502152 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab800cd40 con 0x7f1ac40d0890 2024-04-06T13:48:51.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.940+0000 7f1acaaa0700 1 -- 172.21.15.189:0/476502152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac40b3500 con 0x7f1ac40d0890 2024-04-06T13:48:51.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.941+0000 7f1acaaa0700 1 -- 172.21.15.189:0/476502152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac40b3b50 con 0x7f1ac40d0890 2024-04-06T13:48:51.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.941+0000 7f1ac17fa700 1 -- 172.21.15.189:0/476502152 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 156) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f1ab8013470 con 0x7f1ac40d0890 2024-04-06T13:48:51.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.942+0000 7f1ac17fa700 1 --2- 172.21.15.189:0/476502152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1aa806aed0 0x7f1aa806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:51.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.942+0000 7f1ac3fff700 1 -- 172.21.15.189:0/476502152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1aa806aed0 msgr2=0x7f1aa806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:51.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.942+0000 7f1ac17fa700 1 -- 172.21.15.189:0/476502152 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7395+0+0 (secure 0 0 0) 0x7f1ab8089ef0 con 0x7f1ac40d0890 2024-04-06T13:48:51.945 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.942+0000 7f1ac3fff700 1 --2- 172.21.15.189:0/476502152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1aa806aed0 0x7f1aa806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:48:51.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:51.946+0000 7f1ac17fa700 1 -- 172.21.15.189:0/476502152 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1ab80546d0 con 0x7f1ac40d0890 2024-04-06T13:48:51.968 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:51.967+0000 7f0bf6538200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:48:51.969 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:51.970+0000 7f3525abf200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:48:52.044 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:52.044+0000 7f3525abf200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:48:52.054 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:52.053+0000 7f0bf6538200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:48:52.132 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:52.131+0000 7f0bf6538200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:48:52.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.143+0000 7f1ac3fff700 1 -- 172.21.15.189:0/476502152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1aa806aed0 msgr2=0x7f1aa806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:52.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.143+0000 7f1ac3fff700 1 --2- 172.21.15.189:0/476502152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1aa806aed0 0x7f1aa806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:48:52.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.149+0000 7f1acaaa0700 1 -- 172.21.15.189:0/476502152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1ac4074990 con 0x7f1ac40d0890 2024-04-06T13:48:52.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.149+0000 7f1ac17fa700 1 -- 172.21.15.189:0/476502152 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v253) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1ab8017020 con 0x7f1ac40d0890 2024-04-06T13:48:52.150 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:48:52.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.151+0000 7f1acaaa0700 1 -- 172.21.15.189:0/476502152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1aa806aed0 msgr2=0x7f1aa806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:48:52.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.151+0000 7f1acaaa0700 1 --2- 172.21.15.189:0/476502152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1aa806aed0 0x7f1aa806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.151+0000 7f1acaaa0700 1 -- 172.21.15.189:0/476502152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac40d0890 msgr2=0x7f1ac4154b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:52.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.151+0000 7f1acaaa0700 1 --2- 172.21.15.189:0/476502152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac40d0890 0x7f1ac4154b30 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7f1ab800ebe0 tx=0x7f1ab800eef0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.152+0000 7f1acaaa0700 1 -- 172.21.15.189:0/476502152 shutdown_connections 2024-04-06T13:48:52.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.152+0000 7f1acaaa0700 1 --2- 172.21.15.189:0/476502152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1aa806aed0 0x7f1aa806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.152+0000 7f1acaaa0700 1 --2- 172.21.15.189:0/476502152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ac415d7e0 0x7f1ac40c7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.152+0000 7f1acaaa0700 1 --2- 172.21.15.189:0/476502152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac40d0890 0x7f1ac4154b30 unknown :-1 s=CLOSED pgs=2789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.152+0000 7f1acaaa0700 1 --2- 172.21.15.189:0/476502152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ac40d1cc0 0x7f1ac40b4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.152+0000 7f1acaaa0700 1 -- 172.21.15.189:0/476502152 >> 172.21.15.189:0/476502152 conn(0x7f1ac4072250 msgr2=0x7f1ac415f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:52.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.152+0000 7f1acaaa0700 1 -- 172.21.15.189:0/476502152 shutdown_connections 2024-04-06T13:48:52.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.152+0000 7f1acaaa0700 1 -- 172.21.15.189:0/476502152 wait complete. 2024-04-06T13:48:52.165 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T13:48:52.247 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:52.247+0000 7f3525abf200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:48:52.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.302+0000 7efda5e43700 1 Processor -- start 2024-04-06T13:48:52.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.302+0000 7efda5e43700 1 -- start start 2024-04-06T13:48:52.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.303+0000 7efda5e43700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efda006a730 0x7efda00ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:52.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.303+0000 7efda5e43700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efda00b3ad0 0x7efda0072050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:52.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.303+0000 7efda5e43700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda0071bc0 con 0x7efda00cf2b0 2024-04-06T13:48:52.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.303+0000 7efda5e43700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efda00cf820 con 0x7efda006a730 2024-04-06T13:48:52.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.303+0000 7efda5e43700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efda00cfb00 con 0x7efda00b3ad0 2024-04-06T13:48:52.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.304+0000 7efd9f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efda006a730 0x7efda00ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:52.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.304+0000 7efd9f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efda006a730 0x7efda00ccb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39482/0 (socket says 172.21.15.189:39482) 2024-04-06T13:48:52.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.304+0000 7efd9f7fe700 1 -- 172.21.15.189:0/1344607246 learned_addr learned my addr 172.21.15.189:0/1344607246 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:52.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.304+0000 7efd9ffff700 1 --2- 172.21.15.189:0/1344607246 >> v2:172.21.15.192:3300/0 conn(0x7efda00b3ad0 0x7efda0072050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:52.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.304+0000 7efd9f7fe700 1 -- 172.21.15.189:0/1344607246 >> v2:172.21.15.192:3300/0 conn(0x7efda00b3ad0 msgr2=0x7efda0072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:52.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.304+0000 7efd9f7fe700 1 --2- 172.21.15.189:0/1344607246 >> v2:172.21.15.192:3300/0 conn(0x7efda00b3ad0 0x7efda0072050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.304+0000 7efd9f7fe700 1 -- 172.21.15.189:0/1344607246 >> v1:172.21.15.192:6789/0 conn(0x7efda00cf2b0 legacy=0x7efda006ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:52.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.304+0000 7efd9f7fe700 1 -- 172.21.15.189:0/1344607246 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda0071850 con 0x7efda006a730 2024-04-06T13:48:52.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.304+0000 7efd9f7fe700 1 --2- 172.21.15.189:0/1344607246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda006a730 0x7efda00ccb30 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7efda006b7f0 tx=0x7efd9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c19e9a38a0c6d4ed server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:52.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.305+0000 7efd9e7fc700 1 -- 172.21.15.189:0/1344607246 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9800e070 con 0x7efda006a730 2024-04-06T13:48:52.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.305+0000 7efd9e7fc700 1 -- 172.21.15.189:0/1344607246 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd98003a20 con 0x7efda006a730 2024-04-06T13:48:52.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.305+0000 7efd9e7fc700 1 -- 172.21.15.189:0/1344607246 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd98003dd0 con 0x7efda006a730 2024-04-06T13:48:52.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.306+0000 7efda5e43700 1 -- 172.21.15.189:0/1344607246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda006a730 msgr2=0x7efda00ccb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:52.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.306+0000 7efda5e43700 1 --2- 172.21.15.189:0/1344607246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda006a730 0x7efda00ccb30 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7efda006b7f0 tx=0x7efd9800b330 comp rx=0 tx=0).stop 2024-04-06T13:48:52.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.306+0000 7efda5e43700 1 -- 172.21.15.189:0/1344607246 shutdown_connections 2024-04-06T13:48:52.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.306+0000 7efda5e43700 1 --2- 172.21.15.189:0/1344607246 >> v2:172.21.15.192:3300/0 conn(0x7efda00b3ad0 0x7efda0072050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.306+0000 7efda5e43700 1 --2- 172.21.15.189:0/1344607246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda006a730 0x7efda00ccb30 unknown :-1 s=CLOSED pgs=2790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.306+0000 7efda5e43700 1 -- 172.21.15.189:0/1344607246 >> 172.21.15.189:0/1344607246 conn(0x7efda00d00b0 msgr2=0x7efda00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:52.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.306+0000 7efda5e43700 1 -- 172.21.15.189:0/1344607246 shutdown_connections 2024-04-06T13:48:52.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.306+0000 7efda5e43700 1 -- 172.21.15.189:0/1344607246 wait complete. 2024-04-06T13:48:52.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.308+0000 7efda5e43700 1 Processor -- start 2024-04-06T13:48:52.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.309+0000 7efda5e43700 1 -- start start 2024-04-06T13:48:52.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.310+0000 7efda5e43700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda006a730 0x7efda013d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:52.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.310+0000 7efda5e43700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efda00b3ad0 0x7efda013de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:52.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.310+0000 7efda5e43700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efda00cf2b0 0x7efda013a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:52.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.310+0000 7efda5e43700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda01397f0 con 0x7efda006a730 2024-04-06T13:48:52.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.310+0000 7efda5e43700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda0139ad0 con 0x7efda00cf2b0 2024-04-06T13:48:52.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.310+0000 7efda5e43700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda0139db0 con 0x7efda00b3ad0 2024-04-06T13:48:52.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.310+0000 7efd9ffff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efda00cf2b0 0x7efda013a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:52.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.310+0000 7efd9effd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efda00b3ad0 0x7efda013de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:52.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.310+0000 7efd9f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda006a730 0x7efda013d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:52.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.310+0000 7efd9ffff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efda00cf2b0 0x7efda013a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:41542/0 (socket says 172.21.15.189:41542) 2024-04-06T13:48:52.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.310+0000 7efd9ffff700 1 -- 172.21.15.189:0/2303799706 learned_addr learned my addr 172.21.15.189:0/2303799706 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:52.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.310+0000 7efd9ffff700 1 -- 172.21.15.189:0/2303799706 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efda00b3ad0 msgr2=0x7efda013de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:52.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.310+0000 7efd9ffff700 1 --2- 172.21.15.189:0/2303799706 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efda00b3ad0 0x7efda013de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.310+0000 7efd9ffff700 1 -- 172.21.15.189:0/2303799706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda006a730 msgr2=0x7efda013d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:52.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.310+0000 7efd9ffff700 1 --2- 172.21.15.189:0/2303799706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda006a730 0x7efda013d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.310+0000 7efd9ffff700 1 -- 172.21.15.189:0/2303799706 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd98009b30 con 0x7efda00cf2b0 2024-04-06T13:48:52.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.310+0000 7efd9f7fe700 1 --2- 172.21.15.189:0/2303799706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda006a730 0x7efda013d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:52.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.311+0000 7efd9ffff700 1 --2- 172.21.15.189:0/2303799706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efda00cf2b0 0x7efda013a970 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7efd9000eb60 tx=0x7efd9000ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:52.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.311+0000 7efd87fff700 1 -- 172.21.15.189:0/2303799706 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90009730 con 0x7efda00cf2b0 2024-04-06T13:48:52.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.311+0000 7efd87fff700 1 -- 172.21.15.189:0/2303799706 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd90009e70 con 0x7efda00cf2b0 2024-04-06T13:48:52.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.311+0000 7efd87fff700 1 -- 172.21.15.189:0/2303799706 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9000ca50 con 0x7efda00cf2b0 2024-04-06T13:48:52.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.311+0000 7efda5e43700 1 -- 172.21.15.189:0/2303799706 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda013a090 con 0x7efda00cf2b0 2024-04-06T13:48:52.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.311+0000 7efda5e43700 1 -- 172.21.15.189:0/2303799706 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda013d0f0 con 0x7efda00cf2b0 2024-04-06T13:48:52.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.312+0000 7efda5e43700 1 -- 172.21.15.189:0/2303799706 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda013a5b0 con 0x7efda00cf2b0 2024-04-06T13:48:52.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.313+0000 7efd87fff700 1 -- 172.21.15.189:0/2303799706 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 156) v1 ==== 89350+0+0 (secure 0 0 0) 0x7efd9000cbb0 con 0x7efda00cf2b0 2024-04-06T13:48:52.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.313+0000 7efd87fff700 1 --2- 172.21.15.189:0/2303799706 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efd8c06aed0 0x7efd8c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:52.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.313+0000 7efd9f7fe700 1 -- 172.21.15.189:0/2303799706 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efd8c06aed0 msgr2=0x7efd8c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:52.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.313+0000 7efd87fff700 1 -- 172.21.15.189:0/2303799706 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7395+0+0 (secure 0 0 0) 0x7efd900abd20 con 0x7efda00cf2b0 2024-04-06T13:48:52.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.313+0000 7efd9f7fe700 1 --2- 172.21.15.189:0/2303799706 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efd8c06aed0 0x7efd8c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:48:52.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.318+0000 7efd87fff700 1 -- 172.21.15.189:0/2303799706 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efda013a5b0 con 0x7efda00cf2b0 2024-04-06T13:48:52.325 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:52.325+0000 7f3525abf200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:48:52.357 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:52.352+0000 7f0bf6538200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:48:52.431 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:52.430+0000 7f0bf6538200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:48:52.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.514+0000 7efd9f7fe700 1 -- 172.21.15.189:0/2303799706 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efd8c06aed0 msgr2=0x7efd8c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:52.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.514+0000 7efd9f7fe700 1 --2- 172.21.15.189:0/2303799706 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efd8c06aed0 0x7efd8c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:48:52.544 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.543+0000 7efda5e43700 1 -- 172.21.15.189:0/2303799706 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efda013a5b0 con 0x7efda00cf2b0 2024-04-06T13:48:52.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.543+0000 7efd87fff700 1 -- 172.21.15.189:0/2303799706 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v253) v1 ==== 100+0+11 (secure 0 0 0) 0x7efda013a5b0 con 0x7efda00cf2b0 2024-04-06T13:48:52.545 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:48:52.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.546+0000 7efda5e43700 1 -- 172.21.15.189:0/2303799706 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efd8c06aed0 msgr2=0x7efd8c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:48:52.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.546+0000 7efda5e43700 1 --2- 172.21.15.189:0/2303799706 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efd8c06aed0 0x7efd8c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.546+0000 7efda5e43700 1 -- 172.21.15.189:0/2303799706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efda00cf2b0 msgr2=0x7efda013a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:52.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.546+0000 7efda5e43700 1 --2- 172.21.15.189:0/2303799706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efda00cf2b0 0x7efda013a970 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7efd9000eb60 tx=0x7efd9000ee70 comp rx=0 tx=0).stop 2024-04-06T13:48:52.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.546+0000 7efda5e43700 1 -- 172.21.15.189:0/2303799706 shutdown_connections 2024-04-06T13:48:52.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.546+0000 7efda5e43700 1 --2- 172.21.15.189:0/2303799706 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efd8c06aed0 0x7efd8c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.546+0000 7efda5e43700 1 --2- 172.21.15.189:0/2303799706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efda00cf2b0 0x7efda013a970 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.546+0000 7efda5e43700 1 --2- 172.21.15.189:0/2303799706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda006a730 0x7efda013d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.546+0000 7efda5e43700 1 --2- 172.21.15.189:0/2303799706 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efda00b3ad0 0x7efda013de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.546+0000 7efda5e43700 1 -- 172.21.15.189:0/2303799706 >> 172.21.15.189:0/2303799706 conn(0x7efda00d00b0 msgr2=0x7efda00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:52.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.546+0000 7efda5e43700 1 -- 172.21.15.189:0/2303799706 shutdown_connections 2024-04-06T13:48:52.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.546+0000 7efda5e43700 1 -- 172.21.15.189:0/2303799706 wait complete. 2024-04-06T13:48:52.560 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-04-06T13:48:52.644 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:52.643+0000 7f3525abf200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:48:52.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.689+0000 7fb53aa9a700 1 Processor -- start 2024-04-06T13:48:52.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.689+0000 7fb53aa9a700 1 -- start start 2024-04-06T13:48:52.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.690+0000 7fb53aa9a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb5340b5380 0x7fb5340d2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:52.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.690+0000 7fb53aa9a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5340b5cc0 con 0x7fb5340b4ed0 2024-04-06T13:48:52.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.690+0000 7fb53aa9a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5340b57f0 con 0x7fb53415d590 2024-04-06T13:48:52.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.690+0000 7fb53aa9a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5340d1500 con 0x7fb5340b5380 2024-04-06T13:48:52.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.690+0000 7fb533fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb5340b4ed0 0x7fb5340d2880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48586/0 (socket says 172.21.15.189:48586) 2024-04-06T13:48:52.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.690+0000 7fb533fff700 1 -- 172.21.15.189:0/898286412 learned_addr learned my addr 172.21.15.189:0/898286412 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:52.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.690+0000 7fb539037700 1 --2- 172.21.15.189:0/898286412 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb5340b5380 0x7fb5340d2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:52.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.690+0000 7fb5337fe700 1 -- 172.21.15.189:0/898286412 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125787349 0 0) 0x7fb5340b5cc0 con 0x7fb5340b4ed0 2024-04-06T13:48:52.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.690+0000 7fb5337fe700 1 -- 172.21.15.189:0/898286412 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5240035b0 con 0x7fb5340b4ed0 2024-04-06T13:48:52.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.690+0000 7fb5337fe700 1 -- 172.21.15.189:0/898286412 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1662657249 0 0) 0x7fb5340b57f0 con 0x7fb53415d590 2024-04-06T13:48:52.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.690+0000 7fb5337fe700 1 -- 172.21.15.189:0/898286412 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5340b5cc0 con 0x7fb53415d590 2024-04-06T13:48:52.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.690+0000 7fb539037700 1 -- 172.21.15.189:0/898286412 >> v1:172.21.15.192:6789/0 conn(0x7fb53415d590 legacy=0x7fb5340b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:52.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.690+0000 7fb539037700 1 -- 172.21.15.189:0/898286412 >> v1:172.21.15.189:6789/0 conn(0x7fb5340b4ed0 legacy=0x7fb5340d2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:52.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.690+0000 7fb5337fe700 1 -- 172.21.15.189:0/898286412 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2049996192 0 0) 0x7fb5240035b0 con 0x7fb5340b4ed0 2024-04-06T13:48:52.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.690+0000 7fb539037700 1 -- 172.21.15.189:0/898286412 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5340d17e0 con 0x7fb5340b5380 2024-04-06T13:48:52.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.691+0000 7fb539037700 1 --2- 172.21.15.189:0/898286412 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb5340b5380 0x7fb5340d2f90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb52c009dc0 tx=0x7fb52c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b296072b650c1973 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:52.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.691+0000 7fb5337fe700 1 -- 172.21.15.189:0/898286412 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52c00e070 con 0x7fb5340b5380 2024-04-06T13:48:52.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.691+0000 7fb5337fe700 1 -- 172.21.15.189:0/898286412 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52c003a20 con 0x7fb5340b5380 2024-04-06T13:48:52.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.691+0000 7fb5337fe700 1 -- 172.21.15.189:0/898286412 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520001670 con 0x7fb5340b5380 2024-04-06T13:48:52.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.691+0000 7fb53aa9a700 1 -- 172.21.15.189:0/898286412 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb5340b5380 msgr2=0x7fb5340d2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:52.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.691+0000 7fb53aa9a700 1 --2- 172.21.15.189:0/898286412 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb5340b5380 0x7fb5340d2f90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb52c009dc0 tx=0x7fb52c00b330 comp rx=0 tx=0).stop 2024-04-06T13:48:52.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.691+0000 7fb53aa9a700 1 -- 172.21.15.189:0/898286412 shutdown_connections 2024-04-06T13:48:52.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.691+0000 7fb53aa9a700 1 --2- 172.21.15.189:0/898286412 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb5340b5380 0x7fb5340d2f90 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.691+0000 7fb53aa9a700 1 -- 172.21.15.189:0/898286412 >> 172.21.15.189:0/898286412 conn(0x7fb53406bbc0 msgr2=0x7fb53406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:52.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.691+0000 7fb53aa9a700 1 -- 172.21.15.189:0/898286412 shutdown_connections 2024-04-06T13:48:52.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.692+0000 7fb53aa9a700 1 -- 172.21.15.189:0/898286412 wait complete. 2024-04-06T13:48:52.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.693+0000 7fb53aa9a700 1 Processor -- start 2024-04-06T13:48:52.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.693+0000 7fb53aa9a700 1 -- start start 2024-04-06T13:48:52.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.693+0000 7fb53aa9a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5340b4ed0 0x7fb53414dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:52.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb53aa9a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5340b5380 0x7fb5340cc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:52.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb53aa9a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb53415d590 0x7fb53414fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:52.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb53aa9a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53415e990 con 0x7fb5340b4ed0 2024-04-06T13:48:52.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb53aa9a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53415dd10 con 0x7fb5340b5380 2024-04-06T13:48:52.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb53aa9a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53415dff0 con 0x7fb53415d590 2024-04-06T13:48:52.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb538836700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5340b4ed0 0x7fb53414dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:52.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb538836700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5340b4ed0 0x7fb53414dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39500/0 (socket says 172.21.15.189:39500) 2024-04-06T13:48:52.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb538836700 1 -- 172.21.15.189:0/1016711339 learned_addr learned my addr 172.21.15.189:0/1016711339 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:52.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb533fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5340b5380 0x7fb5340cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:52.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb539037700 1 --2- 172.21.15.189:0/1016711339 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb53415d590 0x7fb53414fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:52.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb538836700 1 -- 172.21.15.189:0/1016711339 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb53415d590 msgr2=0x7fb53414fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:52.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb538836700 1 --2- 172.21.15.189:0/1016711339 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb53415d590 0x7fb53414fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb538836700 1 -- 172.21.15.189:0/1016711339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5340b5380 msgr2=0x7fb5340cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:52.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb538836700 1 --2- 172.21.15.189:0/1016711339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5340b5380 0x7fb5340cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb538836700 1 -- 172.21.15.189:0/1016711339 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52c009a70 con 0x7fb5340b4ed0 2024-04-06T13:48:52.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb533fff700 1 --2- 172.21.15.189:0/1016711339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5340b5380 0x7fb5340cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:52.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb538836700 1 --2- 172.21.15.189:0/1016711339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5340b4ed0 0x7fb53414dd50 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7fb52800c500 tx=0x7fb52800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:52.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb5317fa700 1 -- 172.21.15.189:0/1016711339 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5280052d0 con 0x7fb5340b4ed0 2024-04-06T13:48:52.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb5317fa700 1 -- 172.21.15.189:0/1016711339 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb528005450 con 0x7fb5340b4ed0 2024-04-06T13:48:52.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb53aa9a700 1 -- 172.21.15.189:0/1016711339 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb53415e2d0 con 0x7fb5340b4ed0 2024-04-06T13:48:52.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb5317fa700 1 -- 172.21.15.189:0/1016711339 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5280038c0 con 0x7fb5340b4ed0 2024-04-06T13:48:52.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.694+0000 7fb53aa9a700 1 -- 172.21.15.189:0/1016711339 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb53415d970 con 0x7fb5340b4ed0 2024-04-06T13:48:52.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.695+0000 7fb53aa9a700 1 -- 172.21.15.189:0/1016711339 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb534074990 con 0x7fb5340b4ed0 2024-04-06T13:48:52.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.696+0000 7fb5317fa700 1 -- 172.21.15.189:0/1016711339 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 156) v1 ==== 89350+0+0 (secure 0 0 0) 0x7fb528003a20 con 0x7fb5340b4ed0 2024-04-06T13:48:52.699 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.696+0000 7fb5317fa700 1 --2- 172.21.15.189:0/1016711339 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb51806aed0 0x7fb51806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:52.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.696+0000 7fb533fff700 1 -- 172.21.15.189:0/1016711339 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb51806aed0 msgr2=0x7fb51806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:52.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.696+0000 7fb5317fa700 1 -- 172.21.15.189:0/1016711339 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7395+0+0 (secure 0 0 0) 0x7fb528089000 con 0x7fb5340b4ed0 2024-04-06T13:48:52.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.696+0000 7fb533fff700 1 --2- 172.21.15.189:0/1016711339 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb51806aed0 0x7fb51806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:48:52.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.701+0000 7fb5317fa700 1 -- 172.21.15.189:0/1016711339 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb528053940 con 0x7fb5340b4ed0 2024-04-06T13:48:52.774 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:52.773+0000 7f0bf6538200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:48:52.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.897+0000 7fb533fff700 1 -- 172.21.15.189:0/1016711339 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb51806aed0 msgr2=0x7fb51806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:52.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.897+0000 7fb533fff700 1 --2- 172.21.15.189:0/1016711339 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb51806aed0 0x7fb51806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:48:52.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.918+0000 7fb53aa9a700 1 -- 172.21.15.189:0/1016711339 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb5340ce2a0 con 0x7fb5340b4ed0 2024-04-06T13:48:52.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.918+0000 7fb5317fa700 1 -- 172.21.15.189:0/1016711339 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v253) v1 ==== 107+0+11 (secure 0 0 0) 0x7fb528017020 con 0x7fb5340b4ed0 2024-04-06T13:48:52.919 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:48:52.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.920+0000 7fb53aa9a700 1 -- 172.21.15.189:0/1016711339 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb51806aed0 msgr2=0x7fb51806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:48:52.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.920+0000 7fb53aa9a700 1 --2- 172.21.15.189:0/1016711339 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb51806aed0 0x7fb51806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.920+0000 7fb53aa9a700 1 -- 172.21.15.189:0/1016711339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5340b4ed0 msgr2=0x7fb53414dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:52.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.920+0000 7fb53aa9a700 1 --2- 172.21.15.189:0/1016711339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5340b4ed0 0x7fb53414dd50 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7fb52800c500 tx=0x7fb52800c810 comp rx=0 tx=0).stop 2024-04-06T13:48:52.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.920+0000 7fb53aa9a700 1 -- 172.21.15.189:0/1016711339 shutdown_connections 2024-04-06T13:48:52.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.920+0000 7fb53aa9a700 1 --2- 172.21.15.189:0/1016711339 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb51806aed0 0x7fb51806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.920+0000 7fb53aa9a700 1 --2- 172.21.15.189:0/1016711339 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb53415d590 0x7fb53414fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.920+0000 7fb53aa9a700 1 --2- 172.21.15.189:0/1016711339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5340b4ed0 0x7fb53414dd50 unknown :-1 s=CLOSED pgs=2792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.920+0000 7fb53aa9a700 1 --2- 172.21.15.189:0/1016711339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5340b5380 0x7fb5340cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:52.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.920+0000 7fb53aa9a700 1 -- 172.21.15.189:0/1016711339 >> 172.21.15.189:0/1016711339 conn(0x7fb53406bbc0 msgr2=0x7fb53415bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:52.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.920+0000 7fb53aa9a700 1 -- 172.21.15.189:0/1016711339 shutdown_connections 2024-04-06T13:48:52.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:52.921+0000 7fb53aa9a700 1 -- 172.21.15.189:0/1016711339 wait complete. 2024-04-06T13:48:52.934 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-04-06T13:48:52.965 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:52.964+0000 7f3525abf200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:48:53.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.067+0000 7fe2ddcda700 1 Processor -- start 2024-04-06T13:48:53.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.067+0000 7fe2ddcda700 1 -- start start 2024-04-06T13:48:53.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.068+0000 7fe2ddcda700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2d80d0a60 0x7fe2d80d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:53.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.068+0000 7fe2ddcda700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d80cdf80 con 0x7fe2d815d7e0 2024-04-06T13:48:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.068+0000 7fe2ddcda700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d80d3230 con 0x7fe2d80d1af0 2024-04-06T13:48:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.068+0000 7fe2ddcda700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d80d1f80 con 0x7fe2d80d0a60 2024-04-06T13:48:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.068+0000 7fe2d77fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe2d815d7e0 0x7fe2d80d0350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48596/0 (socket says 172.21.15.189:48596) 2024-04-06T13:48:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.068+0000 7fe2d77fe700 1 -- 172.21.15.189:0/115262153 learned_addr learned my addr 172.21.15.189:0/115262153 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.068+0000 7fe2cffff700 1 --2- 172.21.15.189:0/115262153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2d80d0a60 0x7fe2d80d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.068+0000 7fe2d6ffd700 1 -- 172.21.15.189:0/115262153 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537840989 0 0) 0x7fe2d80cdf80 con 0x7fe2d815d7e0 2024-04-06T13:48:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.068+0000 7fe2d6ffd700 1 -- 172.21.15.189:0/115262153 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2c40035b0 con 0x7fe2d815d7e0 2024-04-06T13:48:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.068+0000 7fe2d6ffd700 1 -- 172.21.15.189:0/115262153 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253853888 0 0) 0x7fe2d0001490 con 0x7fe2d80d1af0 2024-04-06T13:48:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.068+0000 7fe2d6ffd700 1 -- 172.21.15.189:0/115262153 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d80cdf80 con 0x7fe2d80d1af0 2024-04-06T13:48:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.068+0000 7fe2d6ffd700 1 -- 172.21.15.189:0/115262153 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2584061713 0 0) 0x7fe2c40035b0 con 0x7fe2d815d7e0 2024-04-06T13:48:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.068+0000 7fe2cffff700 1 -- 172.21.15.189:0/115262153 >> v1:172.21.15.192:6789/0 conn(0x7fe2d80d1af0 legacy=0x7fe2d80b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.068+0000 7fe2cffff700 1 -- 172.21.15.189:0/115262153 >> v1:172.21.15.189:6789/0 conn(0x7fe2d815d7e0 legacy=0x7fe2d80d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:53.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.068+0000 7fe2cffff700 1 -- 172.21.15.189:0/115262153 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d80d2da0 con 0x7fe2d80d0a60 2024-04-06T13:48:53.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.069+0000 7fe2cffff700 1 --2- 172.21.15.189:0/115262153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2d80d0a60 0x7fe2d80d15b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0009870 tx=0x7fe2c0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5d9636a2eb9b88b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:53.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.069+0000 7fe2d6ffd700 1 -- 172.21.15.189:0/115262153 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c8001670 con 0x7fe2d80d0a60 2024-04-06T13:48:53.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.069+0000 7fe2d6ffd700 1 -- 172.21.15.189:0/115262153 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c0004030 con 0x7fe2d80d0a60 2024-04-06T13:48:53.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.069+0000 7fe2d6ffd700 1 -- 172.21.15.189:0/115262153 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c00043a0 con 0x7fe2d80d0a60 2024-04-06T13:48:53.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.069+0000 7fe2ddcda700 1 -- 172.21.15.189:0/115262153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2d80d0a60 msgr2=0x7fe2d80d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:53.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.069+0000 7fe2ddcda700 1 --2- 172.21.15.189:0/115262153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2d80d0a60 0x7fe2d80d15b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0009870 tx=0x7fe2c0009b80 comp rx=0 tx=0).stop 2024-04-06T13:48:53.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.069+0000 7fe2ddcda700 1 -- 172.21.15.189:0/115262153 shutdown_connections 2024-04-06T13:48:53.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.069+0000 7fe2ddcda700 1 --2- 172.21.15.189:0/115262153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2d80d0a60 0x7fe2d80d15b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.069+0000 7fe2ddcda700 1 -- 172.21.15.189:0/115262153 >> 172.21.15.189:0/115262153 conn(0x7fe2d8072250 msgr2=0x7fe2d8072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:53.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.069+0000 7fe2ddcda700 1 -- 172.21.15.189:0/115262153 shutdown_connections 2024-04-06T13:48:53.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.069+0000 7fe2ddcda700 1 -- 172.21.15.189:0/115262153 wait complete. 2024-04-06T13:48:53.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.071+0000 7fe2ddcda700 1 Processor -- start 2024-04-06T13:48:53.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.071+0000 7fe2ddcda700 1 -- start start 2024-04-06T13:48:53.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.071+0000 7fe2ddcda700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2d80d0a60 0x7fe2d80c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:53.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.071+0000 7fe2ddcda700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2d80d1af0 0x7fe2d81f2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:53.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.071+0000 7fe2ddcda700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2d815d7e0 0x7fe2d80b3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:53.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.071+0000 7fe2ddcda700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d80c9df0 con 0x7fe2d815d7e0 2024-04-06T13:48:53.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.071+0000 7fe2ddcda700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d8150660 con 0x7fe2d80d0a60 2024-04-06T13:48:53.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.071+0000 7fe2ddcda700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d814c890 con 0x7fe2d80d1af0 2024-04-06T13:48:53.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.071+0000 7fe2d7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2d815d7e0 0x7fe2d80b3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:53.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.072+0000 7fe2d7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2d815d7e0 0x7fe2d80b3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39512/0 (socket says 172.21.15.189:39512) 2024-04-06T13:48:53.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.072+0000 7fe2d7fff700 1 -- 172.21.15.189:0/3122463379 learned_addr learned my addr 172.21.15.189:0/3122463379 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:53.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.072+0000 7fe2d77fe700 1 --2- 172.21.15.189:0/3122463379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2d80d0a60 0x7fe2d80c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:53.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.072+0000 7fe2cffff700 1 --2- 172.21.15.189:0/3122463379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2d80d1af0 0x7fe2d81f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:53.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.072+0000 7fe2d7fff700 1 -- 172.21.15.189:0/3122463379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2d80d1af0 msgr2=0x7fe2d81f2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:53.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.072+0000 7fe2d7fff700 1 --2- 172.21.15.189:0/3122463379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2d80d1af0 0x7fe2d81f2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.072+0000 7fe2d7fff700 1 -- 172.21.15.189:0/3122463379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2d80d0a60 msgr2=0x7fe2d80c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:53.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.072+0000 7fe2d7fff700 1 --2- 172.21.15.189:0/3122463379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2d80d0a60 0x7fe2d80c98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.072+0000 7fe2d7fff700 1 -- 172.21.15.189:0/3122463379 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c00094a0 con 0x7fe2d815d7e0 2024-04-06T13:48:53.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.072+0000 7fe2d7fff700 1 --2- 172.21.15.189:0/3122463379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2d815d7e0 0x7fe2d80b3e40 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7fe2d000cc50 tx=0x7fe2d000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:53.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.072+0000 7fe2d4ff9700 1 -- 172.21.15.189:0/3122463379 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0003c00 con 0x7fe2d815d7e0 2024-04-06T13:48:53.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.072+0000 7fe2d4ff9700 1 -- 172.21.15.189:0/3122463379 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d0003d80 con 0x7fe2d815d7e0 2024-04-06T13:48:53.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.072+0000 7fe2ddcda700 1 -- 172.21.15.189:0/3122463379 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d814cb70 con 0x7fe2d815d7e0 2024-04-06T13:48:53.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.072+0000 7fe2d4ff9700 1 -- 172.21.15.189:0/3122463379 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d000aad0 con 0x7fe2d815d7e0 2024-04-06T13:48:53.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.072+0000 7fe2ddcda700 1 -- 172.21.15.189:0/3122463379 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d814d130 con 0x7fe2d815d7e0 2024-04-06T13:48:53.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.073+0000 7fe2ddcda700 1 -- 172.21.15.189:0/3122463379 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2d8074990 con 0x7fe2d815d7e0 2024-04-06T13:48:53.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.074+0000 7fe2d4ff9700 1 -- 172.21.15.189:0/3122463379 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 156) v1 ==== 89350+0+0 (secure 0 0 0) 0x7fe2d000ac30 con 0x7fe2d815d7e0 2024-04-06T13:48:53.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.074+0000 7fe2d4ff9700 1 --2- 172.21.15.189:0/3122463379 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe2b806aed0 0x7fe2b806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:53.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.074+0000 7fe2d77fe700 1 -- 172.21.15.189:0/3122463379 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe2b806aed0 msgr2=0x7fe2b806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:53.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.074+0000 7fe2d4ff9700 1 -- 172.21.15.189:0/3122463379 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7395+0+0 (secure 0 0 0) 0x7fe2d008a820 con 0x7fe2d815d7e0 2024-04-06T13:48:53.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.074+0000 7fe2d77fe700 1 --2- 172.21.15.189:0/3122463379 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe2b806aed0 0x7fe2b806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:48:53.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.078+0000 7fe2d4ff9700 1 -- 172.21.15.189:0/3122463379 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe2d0054f70 con 0x7fe2d815d7e0 2024-04-06T13:48:53.125 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:53.124+0000 7f3525abf200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:48:53.153 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:53.152+0000 7f0bf6538200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:48:53.197 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:53.196+0000 7f3525abf200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:48:53.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.275+0000 7fe2d77fe700 1 -- 172.21.15.189:0/3122463379 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe2b806aed0 msgr2=0x7fe2b806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:53.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.275+0000 7fe2d77fe700 1 --2- 172.21.15.189:0/3122463379 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe2b806aed0 0x7fe2b806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:48:53.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.315+0000 7fe2ddcda700 1 -- 172.21.15.189:0/3122463379 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7fe2d80d3210 con 0x7fe2d815d7e0 2024-04-06T13:48:53.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.315+0000 7fe2d4ff9700 1 -- 172.21.15.189:0/3122463379 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v253) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe2d0058590 con 0x7fe2d815d7e0 2024-04-06T13:48:53.316 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:48:53.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.317+0000 7fe2ddcda700 1 -- 172.21.15.189:0/3122463379 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe2b806aed0 msgr2=0x7fe2b806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:48:53.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.317+0000 7fe2ddcda700 1 --2- 172.21.15.189:0/3122463379 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe2b806aed0 0x7fe2b806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.317+0000 7fe2ddcda700 1 -- 172.21.15.189:0/3122463379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2d815d7e0 msgr2=0x7fe2d80b3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:53.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.317+0000 7fe2ddcda700 1 --2- 172.21.15.189:0/3122463379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2d815d7e0 0x7fe2d80b3e40 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7fe2d000cc50 tx=0x7fe2d000cf60 comp rx=0 tx=0).stop 2024-04-06T13:48:53.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.317+0000 7fe2ddcda700 1 -- 172.21.15.189:0/3122463379 shutdown_connections 2024-04-06T13:48:53.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.317+0000 7fe2ddcda700 1 --2- 172.21.15.189:0/3122463379 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe2b806aed0 0x7fe2b806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.317+0000 7fe2ddcda700 1 --2- 172.21.15.189:0/3122463379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2d815d7e0 0x7fe2d80b3e40 unknown :-1 s=CLOSED pgs=2794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.317+0000 7fe2ddcda700 1 --2- 172.21.15.189:0/3122463379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2d80d0a60 0x7fe2d80c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.317+0000 7fe2ddcda700 1 --2- 172.21.15.189:0/3122463379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe2d80d1af0 0x7fe2d81f2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.317+0000 7fe2ddcda700 1 -- 172.21.15.189:0/3122463379 >> 172.21.15.189:0/3122463379 conn(0x7fe2d8072250 msgr2=0x7fe2d80cd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:53.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.318+0000 7fe2ddcda700 1 -- 172.21.15.189:0/3122463379 shutdown_connections 2024-04-06T13:48:53.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.318+0000 7fe2ddcda700 1 -- 172.21.15.189:0/3122463379 wait complete. 2024-04-06T13:48:53.321 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:53.320+0000 7f0bf6538200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:48:53.332 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T13:48:53.401 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:53.401+0000 7f0bf6538200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:48:53.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.456+0000 7f4f011e9700 1 Processor -- start 2024-04-06T13:48:53.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.456+0000 7f4f011e9700 1 -- start start 2024-04-06T13:48:53.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.457+0000 7f4f011e9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4efc15d6f0 0x7f4efc15dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:53.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.457+0000 7f4f011e9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4efc0b8d70 0x7f4efc0ce020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:53.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.457+0000 7f4f011e9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4efc0ce560 0x7f4efc0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:53.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.457+0000 7f4f011e9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4efc15e010 con 0x7f4efc15d6f0 2024-04-06T13:48:53.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.457+0000 7f4f011e9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4efc0ce9b0 con 0x7f4efc0b8d70 2024-04-06T13:48:53.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.457+0000 7f4f011e9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4efc15eac0 con 0x7f4efc0ce560 2024-04-06T13:48:53.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.457+0000 7f4efad9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4efc15d6f0 0x7f4efc15dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:53.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.457+0000 7f4efad9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4efc15d6f0 0x7f4efc15dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39524/0 (socket says 172.21.15.189:39524) 2024-04-06T13:48:53.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.457+0000 7f4efad9d700 1 -- 172.21.15.189:0/1307436447 learned_addr learned my addr 172.21.15.189:0/1307436447 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:53.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.458+0000 7f4efb59e700 1 --2- 172.21.15.189:0/1307436447 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4efc0ce560 0x7f4efc0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:53.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.458+0000 7f4efa59c700 1 --2- 172.21.15.189:0/1307436447 >> v2:172.21.15.192:3300/0 conn(0x7f4efc0b8d70 0x7f4efc0ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:53.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.458+0000 7f4efad9d700 1 -- 172.21.15.189:0/1307436447 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4efc0ce560 msgr2=0x7f4efc0b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:53.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.458+0000 7f4efad9d700 1 --2- 172.21.15.189:0/1307436447 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4efc0ce560 0x7f4efc0b4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.458+0000 7f4efad9d700 1 -- 172.21.15.189:0/1307436447 >> v2:172.21.15.192:3300/0 conn(0x7f4efc0b8d70 msgr2=0x7f4efc0ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:53.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.458+0000 7f4efad9d700 1 --2- 172.21.15.189:0/1307436447 >> v2:172.21.15.192:3300/0 conn(0x7f4efc0b8d70 0x7f4efc0ce020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.458+0000 7f4efad9d700 1 -- 172.21.15.189:0/1307436447 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4efc15e220 con 0x7f4efc15d6f0 2024-04-06T13:48:53.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.458+0000 7f4efad9d700 1 --2- 172.21.15.189:0/1307436447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4efc15d6f0 0x7f4efc15dad0 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7f4efc0d0e40 tx=0x7f4eec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d24dcac299c51a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:53.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.458+0000 7f4ef9d9b700 1 -- 172.21.15.189:0/1307436447 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec011830 con 0x7f4efc15d6f0 2024-04-06T13:48:53.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.458+0000 7f4ef9d9b700 1 -- 172.21.15.189:0/1307436447 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eec011990 con 0x7f4efc15d6f0 2024-04-06T13:48:53.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.458+0000 7f4ef9d9b700 1 -- 172.21.15.189:0/1307436447 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec011d20 con 0x7f4efc15d6f0 2024-04-06T13:48:53.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.459+0000 7f4f011e9700 1 -- 172.21.15.189:0/1307436447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4efc15d6f0 msgr2=0x7f4efc15dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:53.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.459+0000 7f4f011e9700 1 --2- 172.21.15.189:0/1307436447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4efc15d6f0 0x7f4efc15dad0 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7f4efc0d0e40 tx=0x7f4eec009bd0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.459+0000 7f4f011e9700 1 -- 172.21.15.189:0/1307436447 shutdown_connections 2024-04-06T13:48:53.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.459+0000 7f4f011e9700 1 --2- 172.21.15.189:0/1307436447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4efc15d6f0 0x7f4efc15dad0 unknown :-1 s=CLOSED pgs=2795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.459+0000 7f4f011e9700 1 --2- 172.21.15.189:0/1307436447 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4efc0ce560 0x7f4efc0b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.459+0000 7f4f011e9700 1 --2- 172.21.15.189:0/1307436447 >> v2:172.21.15.192:3300/0 conn(0x7f4efc0b8d70 0x7f4efc0ce020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.459+0000 7f4f011e9700 1 -- 172.21.15.189:0/1307436447 >> 172.21.15.189:0/1307436447 conn(0x7f4efc072490 msgr2=0x7f4efc06b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:53.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.459+0000 7f4f011e9700 1 -- 172.21.15.189:0/1307436447 shutdown_connections 2024-04-06T13:48:53.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.459+0000 7f4f011e9700 1 -- 172.21.15.189:0/1307436447 wait complete. 2024-04-06T13:48:53.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.460+0000 7f4f011e9700 1 Processor -- start 2024-04-06T13:48:53.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.461+0000 7f4f011e9700 1 -- start start 2024-04-06T13:48:53.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.461+0000 7f4f011e9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4efc0b8d70 0x7f4efc0b5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:53.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.461+0000 7f4f011e9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4efc0ce560 0x7f4efc0c9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:53.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.461+0000 7f4f011e9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4efc15d6f0 0x7f4efc159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:53.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.461+0000 7f4f011e9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4efc0b42e0 con 0x7f4efc0b8d70 2024-04-06T13:48:53.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.461+0000 7f4f011e9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4efc0b45c0 con 0x7f4efc15d6f0 2024-04-06T13:48:53.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.461+0000 7f4f011e9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4efc0b48a0 con 0x7f4efc0ce560 2024-04-06T13:48:53.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.461+0000 7f4efad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4efc0b8d70 0x7f4efc0b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:53.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.461+0000 7f4efad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4efc0b8d70 0x7f4efc0b5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39530/0 (socket says 172.21.15.189:39530) 2024-04-06T13:48:53.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.461+0000 7f4efad9d700 1 -- 172.21.15.189:0/2735365547 learned_addr learned my addr 172.21.15.189:0/2735365547 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:53.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.462+0000 7f4efb59e700 1 --2- 172.21.15.189:0/2735365547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4efc15d6f0 0x7f4efc159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:53.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.462+0000 7f4efa59c700 1 --2- 172.21.15.189:0/2735365547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4efc0ce560 0x7f4efc0c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:53.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.462+0000 7f4efad9d700 1 -- 172.21.15.189:0/2735365547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4efc0ce560 msgr2=0x7f4efc0c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:53.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.462+0000 7f4efad9d700 1 --2- 172.21.15.189:0/2735365547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4efc0ce560 0x7f4efc0c9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.462+0000 7f4efad9d700 1 -- 172.21.15.189:0/2735365547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4efc15d6f0 msgr2=0x7f4efc159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:53.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.462+0000 7f4efad9d700 1 --2- 172.21.15.189:0/2735365547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4efc15d6f0 0x7f4efc159a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.462+0000 7f4efad9d700 1 -- 172.21.15.189:0/2735365547 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eec009550 con 0x7f4efc0b8d70 2024-04-06T13:48:53.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.462+0000 7f4efad9d700 1 --2- 172.21.15.189:0/2735365547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4efc0b8d70 0x7f4efc0b5380 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7f4eec000c00 tx=0x7f4eec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:53.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.462+0000 7f4ee37fe700 1 -- 172.21.15.189:0/2735365547 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec004240 con 0x7f4efc0b8d70 2024-04-06T13:48:53.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.462+0000 7f4ee37fe700 1 -- 172.21.15.189:0/2735365547 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eec01f070 con 0x7f4efc0b8d70 2024-04-06T13:48:53.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.462+0000 7f4f011e9700 1 -- 172.21.15.189:0/2735365547 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4efc0b4b80 con 0x7f4efc0b8d70 2024-04-06T13:48:53.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.462+0000 7f4ee37fe700 1 -- 172.21.15.189:0/2735365547 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec0036a0 con 0x7f4efc0b8d70 2024-04-06T13:48:53.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.462+0000 7f4f011e9700 1 -- 172.21.15.189:0/2735365547 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4efc0b58c0 con 0x7f4efc0b8d70 2024-04-06T13:48:53.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.463+0000 7f4f011e9700 1 -- 172.21.15.189:0/2735365547 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4efc0b50a0 con 0x7f4efc0b8d70 2024-04-06T13:48:53.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.464+0000 7f4ee37fe700 1 -- 172.21.15.189:0/2735365547 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 156) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f4eec021030 con 0x7f4efc0b8d70 2024-04-06T13:48:53.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.464+0000 7f4ee37fe700 1 --2- 172.21.15.189:0/2735365547 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4ee806aed0 0x7f4ee806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:53.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.464+0000 7f4efa59c700 1 -- 172.21.15.189:0/2735365547 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4ee806aed0 msgr2=0x7f4ee806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:53.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.464+0000 7f4ee37fe700 1 -- 172.21.15.189:0/2735365547 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7395+0+0 (secure 0 0 0) 0x7f4eec08aec0 con 0x7f4efc0b8d70 2024-04-06T13:48:53.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.464+0000 7f4efa59c700 1 --2- 172.21.15.189:0/2735365547 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4ee806aed0 0x7f4ee806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:48:53.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.468+0000 7f4ee37fe700 1 -- 172.21.15.189:0/2735365547 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4efc0b50a0 con 0x7f4efc0b8d70 2024-04-06T13:48:53.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.665+0000 7f4efa59c700 1 -- 172.21.15.189:0/2735365547 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4ee806aed0 msgr2=0x7f4ee806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:53.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.665+0000 7f4efa59c700 1 --2- 172.21.15.189:0/2735365547 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4ee806aed0 0x7f4ee806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:48:53.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.685+0000 7f4f011e9700 1 -- 172.21.15.189:0/2735365547 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4efc0b50a0 con 0x7f4efc0b8d70 2024-04-06T13:48:53.687 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.686+0000 7f4ee37fe700 1 -- 172.21.15.189:0/2735365547 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v253) v1 ==== 81+0+583 (secure 0 0 0) 0x7f4efc0b50a0 con 0x7f4efc0b8d70 2024-04-06T13:48:53.687 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/85704189 2024-04-07T13:48:48.812402+0000 2024-04-06T13:48:53.687 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2120970213 2024-04-07T13:48:48.812402+0000 2024-04-06T13:48:53.687 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/381758424 2024-04-07T13:48:48.812402+0000 2024-04-06T13:48:53.687 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2107771319 2024-04-07T13:48:48.812402+0000 2024-04-06T13:48:53.687 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2145696658 2024-04-07T13:48:48.812402+0000 2024-04-06T13:48:53.687 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6833/111704 2024-04-07T13:48:48.812402+0000 2024-04-06T13:48:53.687 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/3440291825 2024-04-07T13:48:48.812402+0000 2024-04-06T13:48:53.687 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6832/111704 2024-04-07T13:48:48.812402+0000 2024-04-06T13:48:53.687 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/485512536 2024-04-07T13:48:48.812402+0000 2024-04-06T13:48:53.687 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/643921686 2024-04-07T13:48:48.812402+0000 2024-04-06T13:48:53.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.689+0000 7f4f011e9700 1 -- 172.21.15.189:0/2735365547 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4ee806aed0 msgr2=0x7f4ee806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:48:53.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.689+0000 7f4f011e9700 1 --2- 172.21.15.189:0/2735365547 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4ee806aed0 0x7f4ee806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.689+0000 7f4f011e9700 1 -- 172.21.15.189:0/2735365547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4efc0b8d70 msgr2=0x7f4efc0b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:53.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.689+0000 7f4f011e9700 1 --2- 172.21.15.189:0/2735365547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4efc0b8d70 0x7f4efc0b5380 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7f4eec000c00 tx=0x7f4eec004750 comp rx=0 tx=0).stop 2024-04-06T13:48:53.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.689+0000 7f4f011e9700 1 -- 172.21.15.189:0/2735365547 shutdown_connections 2024-04-06T13:48:53.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.690+0000 7f4f011e9700 1 --2- 172.21.15.189:0/2735365547 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4ee806aed0 0x7f4ee806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.690+0000 7f4f011e9700 1 --2- 172.21.15.189:0/2735365547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4efc15d6f0 0x7f4efc159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.690+0000 7f4f011e9700 1 --2- 172.21.15.189:0/2735365547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4efc0b8d70 0x7f4efc0b5380 unknown :-1 s=CLOSED pgs=2796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.690+0000 7f4f011e9700 1 --2- 172.21.15.189:0/2735365547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4efc0ce560 0x7f4efc0c9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:53.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.690+0000 7f4f011e9700 1 -- 172.21.15.189:0/2735365547 >> 172.21.15.189:0/2735365547 conn(0x7f4efc072490 msgr2=0x7f4efc0b6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:53.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.690+0000 7f4f011e9700 1 -- 172.21.15.189:0/2735365547 shutdown_connections 2024-04-06T13:48:53.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:53.690+0000 7f4f011e9700 1 -- 172.21.15.189:0/2735365547 wait complete. 2024-04-06T13:48:53.693 INFO:teuthology.orchestra.run.smithi189.stderr:listed 10 entries 2024-04-06T13:48:53.707 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-06T13:48:53.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:48:53.708 DEBUG:teuthology.orchestra.run.smithi189:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-06T13:48:53.748 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T13:48:53.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:48:53.748 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-06T13:48:53.806 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:48:53.806 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:48:53.813 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:53.813+0000 7f3525abf200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:48:53.823 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:48:53.823 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:48:53.823 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:48:53.863 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:48:53.863 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:48:53.863 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:48:53.883 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:48:53.884 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:48:53.896 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:53.895+0000 7f3525abf200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:48:54.022 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:54.021+0000 7f3525abf200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:48:54.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.028+0000 7fee85a1e700 1 Processor -- start 2024-04-06T13:48:54.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.028+0000 7fee85a1e700 1 -- start start 2024-04-06T13:48:54.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.029+0000 7fee85a1e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee8015d6f0 0x7fee8015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:54.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.029+0000 7fee85a1e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fee800b8d70 0x7fee800ce020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:54.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.029+0000 7fee85a1e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee800ce9b0 con 0x7fee800ce560 2024-04-06T13:48:54.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.029+0000 7fee85a1e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee8015e010 con 0x7fee800b8d70 2024-04-06T13:48:54.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.029+0000 7fee85a1e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee8015eac0 con 0x7fee8015d6f0 2024-04-06T13:48:54.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.029+0000 7fee7effd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee8015d6f0 0x7fee8015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:54.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.029+0000 7fee7e7fc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fee800b8d70 0x7fee800ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:54.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.029+0000 7fee7effd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee8015d6f0 0x7fee8015dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:58284/0 (socket says 172.21.15.189:58284) 2024-04-06T13:48:54.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.029+0000 7fee7e7fc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fee800b8d70 0x7fee800ce020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:41600/0 (socket says 172.21.15.189:41600) 2024-04-06T13:48:54.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.029+0000 7fee7effd700 1 -- 172.21.15.189:0/3819995045 learned_addr learned my addr 172.21.15.189:0/3819995045 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:54.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.029+0000 7fee7effd700 1 -- 172.21.15.189:0/3819995045 >> v2:172.21.15.192:3300/0 conn(0x7fee800b8d70 msgr2=0x7fee800ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:54.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.029+0000 7fee7effd700 1 --2- 172.21.15.189:0/3819995045 >> v2:172.21.15.192:3300/0 conn(0x7fee800b8d70 0x7fee800ce020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.029+0000 7fee7effd700 1 -- 172.21.15.189:0/3819995045 >> v1:172.21.15.192:6789/0 conn(0x7fee800ce560 legacy=0x7fee800b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:54.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.029+0000 7fee7effd700 1 -- 172.21.15.189:0/3819995045 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8015e220 con 0x7fee8015d6f0 2024-04-06T13:48:54.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.029+0000 7fee7e7fc700 1 --2- 172.21.15.189:0/3819995045 >> v2:172.21.15.192:3300/0 conn(0x7fee800b8d70 0x7fee800ce020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:54.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.030+0000 7fee7effd700 1 --2- 172.21.15.189:0/3819995045 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee8015d6f0 0x7fee8015dad0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fee740098f0 tx=0x7fee74009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fcaec33e789422 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:54.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.030+0000 7fee7dffb700 1 -- 172.21.15.189:0/3819995045 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee74011830 con 0x7fee8015d6f0 2024-04-06T13:48:54.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.030+0000 7fee7dffb700 1 -- 172.21.15.189:0/3819995045 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee74011990 con 0x7fee8015d6f0 2024-04-06T13:48:54.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.030+0000 7fee7dffb700 1 -- 172.21.15.189:0/3819995045 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee74011d20 con 0x7fee8015d6f0 2024-04-06T13:48:54.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.030+0000 7fee85a1e700 1 -- 172.21.15.189:0/3819995045 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee8015d6f0 msgr2=0x7fee8015dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:54.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.030+0000 7fee85a1e700 1 --2- 172.21.15.189:0/3819995045 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee8015d6f0 0x7fee8015dad0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fee740098f0 tx=0x7fee74009c00 comp rx=0 tx=0).stop 2024-04-06T13:48:54.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.030+0000 7fee85a1e700 1 -- 172.21.15.189:0/3819995045 shutdown_connections 2024-04-06T13:48:54.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.030+0000 7fee85a1e700 1 --2- 172.21.15.189:0/3819995045 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee8015d6f0 0x7fee8015dad0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.030+0000 7fee85a1e700 1 --2- 172.21.15.189:0/3819995045 >> v2:172.21.15.192:3300/0 conn(0x7fee800b8d70 0x7fee800ce020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.030+0000 7fee85a1e700 1 -- 172.21.15.189:0/3819995045 >> 172.21.15.189:0/3819995045 conn(0x7fee80072490 msgr2=0x7fee8006b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:54.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.030+0000 7fee85a1e700 1 -- 172.21.15.189:0/3819995045 shutdown_connections 2024-04-06T13:48:54.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.030+0000 7fee85a1e700 1 -- 172.21.15.189:0/3819995045 wait complete. 2024-04-06T13:48:54.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.032+0000 7fee85a1e700 1 Processor -- start 2024-04-06T13:48:54.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.032+0000 7fee85a1e700 1 -- start start 2024-04-06T13:48:54.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.032+0000 7fee85a1e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee800b8d70 0x7fee800c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:54.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.032+0000 7fee85a1e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee800ce560 0x7fee801f24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:54.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.032+0000 7fee85a1e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee8015d6f0 0x7fee80153470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:54.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.032+0000 7fee85a1e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee800c9d70 con 0x7fee800ce560 2024-04-06T13:48:54.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.032+0000 7fee85a1e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee801482a0 con 0x7fee8015d6f0 2024-04-06T13:48:54.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.032+0000 7fee85a1e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee801567f0 con 0x7fee800b8d70 2024-04-06T13:48:54.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.033+0000 7fee7e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee800ce560 0x7fee801f24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:54.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.033+0000 7fee7effd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee800b8d70 0x7fee800c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:54.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.033+0000 7fee7e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee800ce560 0x7fee801f24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39532/0 (socket says 172.21.15.189:39532) 2024-04-06T13:48:54.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.033+0000 7fee7e7fc700 1 -- 172.21.15.189:0/391230597 learned_addr learned my addr 172.21.15.189:0/391230597 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:54.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.033+0000 7fee7effd700 1 --2- 172.21.15.189:0/391230597 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee800b8d70 0x7fee800c9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:58290/0 (socket says 172.21.15.189:58290) 2024-04-06T13:48:54.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.033+0000 7fee7f7fe700 1 --2- 172.21.15.189:0/391230597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee8015d6f0 0x7fee80153470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:54.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.033+0000 7fee7e7fc700 1 -- 172.21.15.189:0/391230597 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee800b8d70 msgr2=0x7fee800c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:54.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.033+0000 7fee7e7fc700 1 --2- 172.21.15.189:0/391230597 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee800b8d70 0x7fee800c9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.033+0000 7fee7e7fc700 1 -- 172.21.15.189:0/391230597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee8015d6f0 msgr2=0x7fee80153470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:54.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.033+0000 7fee7e7fc700 1 --2- 172.21.15.189:0/391230597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee8015d6f0 0x7fee80153470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.033+0000 7fee7e7fc700 1 -- 172.21.15.189:0/391230597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee700094a0 con 0x7fee800ce560 2024-04-06T13:48:54.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.033+0000 7fee7effd700 1 --2- 172.21.15.189:0/391230597 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee800b8d70 0x7fee800c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:54.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.033+0000 7fee7e7fc700 1 --2- 172.21.15.189:0/391230597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee800ce560 0x7fee801f24b0 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7fee7000ec10 tx=0x7fee7000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:54.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.033+0000 7fee67fff700 1 -- 172.21.15.189:0/391230597 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7000c850 con 0x7fee800ce560 2024-04-06T13:48:54.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.033+0000 7fee67fff700 1 -- 172.21.15.189:0/391230597 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee7000c9b0 con 0x7fee800ce560 2024-04-06T13:48:54.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.033+0000 7fee67fff700 1 -- 172.21.15.189:0/391230597 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7000cd20 con 0x7fee800ce560 2024-04-06T13:48:54.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.033+0000 7fee85a1e700 1 -- 172.21.15.189:0/391230597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee74009550 con 0x7fee800ce560 2024-04-06T13:48:54.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.033+0000 7fee85a1e700 1 -- 172.21.15.189:0/391230597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee80155a50 con 0x7fee800ce560 2024-04-06T13:48:54.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.034+0000 7fee85a1e700 1 -- 172.21.15.189:0/391230597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee800cf380 con 0x7fee800ce560 2024-04-06T13:48:54.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.035+0000 7fee67fff700 1 -- 172.21.15.189:0/391230597 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 156) v1 ==== 89350+0+0 (secure 0 0 0) 0x7fee70013470 con 0x7fee800ce560 2024-04-06T13:48:54.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.035+0000 7fee67fff700 1 --2- 172.21.15.189:0/391230597 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fee6c06aed0 0x7fee6c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:54.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.035+0000 7fee7effd700 1 -- 172.21.15.189:0/391230597 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fee6c06aed0 msgr2=0x7fee6c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:54.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.035+0000 7fee67fff700 1 -- 172.21.15.189:0/391230597 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7395+0+0 (secure 0 0 0) 0x7fee70089e80 con 0x7fee800ce560 2024-04-06T13:48:54.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.035+0000 7fee7effd700 1 --2- 172.21.15.189:0/391230597 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fee6c06aed0 0x7fee6c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:48:54.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.039+0000 7fee67fff700 1 -- 172.21.15.189:0/391230597 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fee70054650 con 0x7fee800ce560 2024-04-06T13:48:54.079 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:54.078+0000 7f0bf6538200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:48:54.161 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:54.160+0000 7f0bf6538200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:48:54.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.236+0000 7fee7effd700 1 -- 172.21.15.189:0/391230597 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fee6c06aed0 msgr2=0x7fee6c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:54.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.236+0000 7fee7effd700 1 --2- 172.21.15.189:0/391230597 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fee6c06aed0 0x7fee6c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:48:54.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.292+0000 7fee85a1e700 1 -- 172.21.15.189:0/391230597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee80074990 con 0x7fee800ce560 2024-04-06T13:48:54.293 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:54.292+0000 7f0bf6538200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:48:54.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.292+0000 7fee67fff700 1 -- 172.21.15.189:0/391230597 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5967 (secure 0 0 0) 0x7fee70057c70 con 0x7fee800ce560 2024-04-06T13:48:54.294 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:54.294 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":122,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3323503154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3323503154},{"type":"v1","addr":"172.21.15.189:6836","nonce":3323503154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":9383,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3292345173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3292345173},{"type":"v1","addr":"172.21.15.189:6837","nonce":3292345173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:48:19.674205+0000","modified":"2024-04-06T13:48:22.682911+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12108},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12108":{"gid":12108,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6833/726816746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":726816746},{"type":"v1","addr":"172.21.15.192:6833","nonce":726816746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:48:36.197004+0000","modified":"2024-04-06T13:48:39.206755+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9395},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9395":{"gid":9395,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/455737532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":455737532},{"type":"v1","addr":"172.21.15.192:6835","nonce":455737532}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-04-06T13:48:54.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.295+0000 7fee85a1e700 1 -- 172.21.15.189:0/391230597 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fee6c06aed0 msgr2=0x7fee6c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:48:54.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.295+0000 7fee85a1e700 1 --2- 172.21.15.189:0/391230597 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fee6c06aed0 0x7fee6c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.295+0000 7fee85a1e700 1 -- 172.21.15.189:0/391230597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee800ce560 msgr2=0x7fee801f24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:54.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.295+0000 7fee85a1e700 1 --2- 172.21.15.189:0/391230597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee800ce560 0x7fee801f24b0 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7fee7000ec10 tx=0x7fee7000ef20 comp rx=0 tx=0).stop 2024-04-06T13:48:54.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.295+0000 7fee85a1e700 1 -- 172.21.15.189:0/391230597 shutdown_connections 2024-04-06T13:48:54.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.295+0000 7fee85a1e700 1 --2- 172.21.15.189:0/391230597 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fee6c06aed0 0x7fee6c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.295+0000 7fee85a1e700 1 --2- 172.21.15.189:0/391230597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee8015d6f0 0x7fee80153470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.295+0000 7fee85a1e700 1 --2- 172.21.15.189:0/391230597 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee800b8d70 0x7fee800c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.295+0000 7fee85a1e700 1 --2- 172.21.15.189:0/391230597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee800ce560 0x7fee801f24b0 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.295+0000 7fee85a1e700 1 -- 172.21.15.189:0/391230597 >> 172.21.15.189:0/391230597 conn(0x7fee80072490 msgr2=0x7fee8015ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:54.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.295+0000 7fee85a1e700 1 -- 172.21.15.189:0/391230597 shutdown_connections 2024-04-06T13:48:54.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.295+0000 7fee85a1e700 1 -- 172.21.15.189:0/391230597 wait complete. 2024-04-06T13:48:54.298 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 122 2024-04-06T13:48:54.309 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:48:54.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.436+0000 7f2729401700 1 Processor -- start 2024-04-06T13:48:54.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.436+0000 7f2729401700 1 -- start start 2024-04-06T13:48:54.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.437+0000 7f2729401700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f272406ae40 0x7f27240b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:54.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.437+0000 7f2729401700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27240b3990 con 0x7f272415d7b0 2024-04-06T13:48:54.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.437+0000 7f2729401700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27240b3060 con 0x7f272415ddf0 2024-04-06T13:48:54.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.437+0000 7f2729401700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f272415e950 con 0x7f272406ae40 2024-04-06T13:48:54.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.437+0000 7f27227fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f272406ae40 0x7f27240b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:54.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.437+0000 7f2722ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f272415d7b0 0x7f272406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48602/0 (socket says 172.21.15.189:48602) 2024-04-06T13:48:54.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.437+0000 7f2722ffd700 1 -- 172.21.15.189:0/1266848349 learned_addr learned my addr 172.21.15.189:0/1266848349 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:54.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.438+0000 7f27227fc700 1 -- 172.21.15.189:0/1266848349 >> v1:172.21.15.192:6789/0 conn(0x7f272415ddf0 legacy=0x7f272415e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:54.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.438+0000 7f27227fc700 1 -- 172.21.15.189:0/1266848349 >> v1:172.21.15.189:6789/0 conn(0x7f272415d7b0 legacy=0x7f272406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:54.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.438+0000 7f27227fc700 1 -- 172.21.15.189:0/1266848349 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27240b3340 con 0x7f272406ae40 2024-04-06T13:48:54.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.438+0000 7f2721ffb700 1 -- 172.21.15.189:0/1266848349 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598305278 0 0) 0x7f27240b3990 con 0x7f272415d7b0 2024-04-06T13:48:54.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.438+0000 7f27227fc700 1 --2- 172.21.15.189:0/1266848349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f272406ae40 0x7f27240b3e20 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f2714009840 tx=0x7f2714009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f434025f345654c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:54.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.438+0000 7f2721ffb700 1 -- 172.21.15.189:0/1266848349 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2714011830 con 0x7f272406ae40 2024-04-06T13:48:54.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.438+0000 7f2721ffb700 1 -- 172.21.15.189:0/1266848349 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2714011990 con 0x7f272406ae40 2024-04-06T13:48:54.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.438+0000 7f2721ffb700 1 -- 172.21.15.189:0/1266848349 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2714011d20 con 0x7f272406ae40 2024-04-06T13:48:54.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.439+0000 7f2729401700 1 -- 172.21.15.189:0/1266848349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f272406ae40 msgr2=0x7f27240b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:54.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.439+0000 7f2729401700 1 --2- 172.21.15.189:0/1266848349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f272406ae40 0x7f27240b3e20 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f2714009840 tx=0x7f2714009b50 comp rx=0 tx=0).stop 2024-04-06T13:48:54.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.439+0000 7f2729401700 1 -- 172.21.15.189:0/1266848349 shutdown_connections 2024-04-06T13:48:54.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.439+0000 7f2729401700 1 --2- 172.21.15.189:0/1266848349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f272406ae40 0x7f27240b3e20 unknown :-1 s=CLOSED pgs=2799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.439+0000 7f2729401700 1 -- 172.21.15.189:0/1266848349 >> 172.21.15.189:0/1266848349 conn(0x7f27240d00b0 msgr2=0x7f27240d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:54.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.439+0000 7f2729401700 1 -- 172.21.15.189:0/1266848349 shutdown_connections 2024-04-06T13:48:54.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.439+0000 7f2729401700 1 -- 172.21.15.189:0/1266848349 wait complete. 2024-04-06T13:48:54.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.440+0000 7f2729401700 1 Processor -- start 2024-04-06T13:48:54.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.440+0000 7f2729401700 1 -- start start 2024-04-06T13:48:54.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.441+0000 7f2729401700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f272406ae40 0x7f272415c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:54.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.441+0000 7f2729401700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f272415d7b0 0x7f2724156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:54.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.441+0000 7f2729401700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f272415ddf0 0x7f27240aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:54.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.441+0000 7f2729401700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27240b1470 con 0x7f272415d7b0 2024-04-06T13:48:54.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.441+0000 7f2729401700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2724157530 con 0x7f272406ae40 2024-04-06T13:48:54.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.441+0000 7f2729401700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27240b0ad0 con 0x7f272415ddf0 2024-04-06T13:48:54.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.441+0000 7f27227fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f272415d7b0 0x7f2724156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:54.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.441+0000 7f27227fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f272415d7b0 0x7f2724156ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39548/0 (socket says 172.21.15.189:39548) 2024-04-06T13:48:54.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.441+0000 7f27227fc700 1 -- 172.21.15.189:0/3021896598 learned_addr learned my addr 172.21.15.189:0/3021896598 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:54.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.441+0000 7f27237fe700 1 --2- 172.21.15.189:0/3021896598 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f272415ddf0 0x7f27240aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:54.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.441+0000 7f2722ffd700 1 --2- 172.21.15.189:0/3021896598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f272406ae40 0x7f272415c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:54.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.441+0000 7f27227fc700 1 -- 172.21.15.189:0/3021896598 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f272415ddf0 msgr2=0x7f27240aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:54.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.441+0000 7f27227fc700 1 --2- 172.21.15.189:0/3021896598 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f272415ddf0 0x7f27240aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.442+0000 7f27227fc700 1 -- 172.21.15.189:0/3021896598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f272406ae40 msgr2=0x7f272415c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:54.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.442+0000 7f27227fc700 1 --2- 172.21.15.189:0/3021896598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f272406ae40 0x7f272415c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.442+0000 7f27227fc700 1 -- 172.21.15.189:0/3021896598 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27140094a0 con 0x7f272415d7b0 2024-04-06T13:48:54.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.442+0000 7f27227fc700 1 --2- 172.21.15.189:0/3021896598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f272415d7b0 0x7f2724156ff0 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f27140090b0 tx=0x7f2714004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:54.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.442+0000 7f270b7fe700 1 -- 172.21.15.189:0/3021896598 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27140045d0 con 0x7f272415d7b0 2024-04-06T13:48:54.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.442+0000 7f270b7fe700 1 -- 172.21.15.189:0/3021896598 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f271401f070 con 0x7f272415d7b0 2024-04-06T13:48:54.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.442+0000 7f270b7fe700 1 -- 172.21.15.189:0/3021896598 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27140038a0 con 0x7f272415d7b0 2024-04-06T13:48:54.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.442+0000 7f2729401700 1 -- 172.21.15.189:0/3021896598 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27240b0db0 con 0x7f272415d7b0 2024-04-06T13:48:54.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.442+0000 7f2729401700 1 -- 172.21.15.189:0/3021896598 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27240af0c0 con 0x7f272415d7b0 2024-04-06T13:48:54.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.443+0000 7f2729401700 1 -- 172.21.15.189:0/3021896598 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27240b15b0 con 0x7f272415d7b0 2024-04-06T13:48:54.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.444+0000 7f270b7fe700 1 -- 172.21.15.189:0/3021896598 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 156) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f2714003a00 con 0x7f272415d7b0 2024-04-06T13:48:54.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.444+0000 7f270b7fe700 1 --2- 172.21.15.189:0/3021896598 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f271006af00 0x7f271006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:54.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.444+0000 7f270b7fe700 1 -- 172.21.15.189:0/3021896598 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7395+0+0 (secure 0 0 0) 0x7f2714021070 con 0x7f272415d7b0 2024-04-06T13:48:54.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.444+0000 7f2722ffd700 1 -- 172.21.15.189:0/3021896598 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f271006af00 msgr2=0x7f271006d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:54.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.444+0000 7f2722ffd700 1 --2- 172.21.15.189:0/3021896598 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f271006af00 0x7f271006d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:48:54.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.449+0000 7f270b7fe700 1 -- 172.21.15.189:0/3021896598 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f27240b15b0 con 0x7f272415d7b0 2024-04-06T13:48:54.533 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:54.532+0000 7f3525abf200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:48:54.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.645+0000 7f2722ffd700 1 -- 172.21.15.189:0/3021896598 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f271006af00 msgr2=0x7f271006d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:54.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.645+0000 7f2722ffd700 1 --2- 172.21.15.189:0/3021896598 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f271006af00 0x7f271006d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:48:54.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.651+0000 7f2729401700 1 -- 172.21.15.189:0/3021896598 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27240b15b0 con 0x7f272415d7b0 2024-04-06T13:48:54.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.653+0000 7f270b7fe700 1 -- 172.21.15.189:0/3021896598 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v253) v1 ==== 74+0+20762 (secure 0 0 0) 0x7f27240b15b0 con 0x7f272415d7b0 2024-04-06T13:48:54.654 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:54.654 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":253,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:48:48.812449+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:48:17.447247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-04-06T13:48:18.646229+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:48:33.912100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:48:35.170642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/85704189":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2120970213":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/381758424":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2107771319":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2145696658":"2024-04-07T13:48:48.812402+0000","172.21.15.189:6833/111704":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/3440291825":"2024-04-07T13:48:48.812402+0000","172.21.15.189:6832/111704":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/485512536":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/643921686":"2024-04-07T13:48:48.812402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:48:54.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.655+0000 7f2729401700 1 -- 172.21.15.189:0/3021896598 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f271006af00 msgr2=0x7f271006d3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:48:54.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.655+0000 7f2729401700 1 --2- 172.21.15.189:0/3021896598 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f271006af00 0x7f271006d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.655+0000 7f2729401700 1 -- 172.21.15.189:0/3021896598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f272415d7b0 msgr2=0x7f2724156ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:54.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.655+0000 7f2729401700 1 --2- 172.21.15.189:0/3021896598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f272415d7b0 0x7f2724156ff0 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f27140090b0 tx=0x7f2714004750 comp rx=0 tx=0).stop 2024-04-06T13:48:54.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.655+0000 7f2729401700 1 -- 172.21.15.189:0/3021896598 shutdown_connections 2024-04-06T13:48:54.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.655+0000 7f2729401700 1 --2- 172.21.15.189:0/3021896598 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f271006af00 0x7f271006d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.655+0000 7f2729401700 1 --2- 172.21.15.189:0/3021896598 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f272415ddf0 0x7f27240aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.655+0000 7f2729401700 1 --2- 172.21.15.189:0/3021896598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f272406ae40 0x7f272415c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.655+0000 7f2729401700 1 --2- 172.21.15.189:0/3021896598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f272415d7b0 0x7f2724156ff0 unknown :-1 s=CLOSED pgs=2800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.655+0000 7f2729401700 1 -- 172.21.15.189:0/3021896598 >> 172.21.15.189:0/3021896598 conn(0x7f27240d00b0 msgr2=0x7f272415e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:54.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.655+0000 7f2729401700 1 -- 172.21.15.189:0/3021896598 shutdown_connections 2024-04-06T13:48:54.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.655+0000 7f2729401700 1 -- 172.21.15.189:0/3021896598 wait complete. 2024-04-06T13:48:54.670 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T13:48:17.447247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '247', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T13:48:18.646229+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-04-06T13:48:33.912100+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '251', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2024-04-06T13:48:35.170642+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '252', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:48:54.670 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:48:54.686 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:54.685+0000 7f3525abf200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:48:54.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.799+0000 7f11d2101700 1 Processor -- start 2024-04-06T13:48:54.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.799+0000 7f11d2101700 1 -- start start 2024-04-06T13:48:54.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.800+0000 7f11d2101700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f11cc0d11b0 0x7f11cc0d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:54.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.800+0000 7f11d2101700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11cc0d1ad0 con 0x7f11cc0cb870 2024-04-06T13:48:54.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.800+0000 7f11d2101700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11cc0cbd80 con 0x7f11cc0cfa00 2024-04-06T13:48:54.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.800+0000 7f11d2101700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11cc0ccdc0 con 0x7f11cc0d11b0 2024-04-06T13:48:54.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.800+0000 7f11cb7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f11cc0d11b0 0x7f11cc0d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:54.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.800+0000 7f11caffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f11cc0cb870 0x7f11cc15c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48606/0 (socket says 172.21.15.189:48606) 2024-04-06T13:48:54.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.800+0000 7f11caffd700 1 -- 172.21.15.189:0/1363578927 learned_addr learned my addr 172.21.15.189:0/1363578927 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:54.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.800+0000 7f11cb7fe700 1 -- 172.21.15.189:0/1363578927 >> v1:172.21.15.192:6789/0 conn(0x7f11cc0cfa00 legacy=0x7f11cc0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:54.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.800+0000 7f11cb7fe700 1 -- 172.21.15.189:0/1363578927 >> v1:172.21.15.189:6789/0 conn(0x7f11cc0cb870 legacy=0x7f11cc15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:54.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.801+0000 7f11cb7fe700 1 -- 172.21.15.189:0/1363578927 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11cc0cef20 con 0x7f11cc0d11b0 2024-04-06T13:48:54.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.801+0000 7f11ca7fc700 1 -- 172.21.15.189:0/1363578927 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088530150 0 0) 0x7f11cc0d1ad0 con 0x7f11cc0cb870 2024-04-06T13:48:54.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.801+0000 7f11cb7fe700 1 --2- 172.21.15.189:0/1363578927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11cc0d11b0 0x7f11cc0d1590 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009d90 tx=0x7f11bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36fe823922358b2f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:54.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.801+0000 7f11ca7fc700 1 -- 172.21.15.189:0/1363578927 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc00e070 con 0x7f11cc0d11b0 2024-04-06T13:48:54.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.801+0000 7f11ca7fc700 1 -- 172.21.15.189:0/1363578927 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11bc00bbd0 con 0x7f11cc0d11b0 2024-04-06T13:48:54.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.801+0000 7f11ca7fc700 1 -- 172.21.15.189:0/1363578927 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc012430 con 0x7f11cc0d11b0 2024-04-06T13:48:54.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.801+0000 7f11d2101700 1 -- 172.21.15.189:0/1363578927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11cc0d11b0 msgr2=0x7f11cc0d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:54.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.801+0000 7f11d2101700 1 --2- 172.21.15.189:0/1363578927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11cc0d11b0 0x7f11cc0d1590 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009d90 tx=0x7f11bc00b330 comp rx=0 tx=0).stop 2024-04-06T13:48:54.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.801+0000 7f11d2101700 1 -- 172.21.15.189:0/1363578927 shutdown_connections 2024-04-06T13:48:54.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.801+0000 7f11d2101700 1 --2- 172.21.15.189:0/1363578927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11cc0d11b0 0x7f11cc0d1590 unknown :-1 s=CLOSED pgs=2802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.801+0000 7f11d2101700 1 -- 172.21.15.189:0/1363578927 >> 172.21.15.189:0/1363578927 conn(0x7f11cc0b2db0 msgr2=0x7f11cc0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:54.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.802+0000 7f11d2101700 1 -- 172.21.15.189:0/1363578927 shutdown_connections 2024-04-06T13:48:54.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.802+0000 7f11d2101700 1 -- 172.21.15.189:0/1363578927 wait complete. 2024-04-06T13:48:54.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.803+0000 7f11d2101700 1 Processor -- start 2024-04-06T13:48:54.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.803+0000 7f11d2101700 1 -- start start 2024-04-06T13:48:54.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.804+0000 7f11d2101700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11cc0cb870 0x7f11cc06d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:54.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.804+0000 7f11d2101700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11cc0cfa00 0x7f11cc156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:54.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.804+0000 7f11d2101700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11cc0d11b0 0x7f11cc0c9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:54.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.804+0000 7f11d2101700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc06d8c0 con 0x7f11cc0d11b0 2024-04-06T13:48:54.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.804+0000 7f11d2101700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc1e5dd0 con 0x7f11cc0cfa00 2024-04-06T13:48:54.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.804+0000 7f11d2101700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc156bc0 con 0x7f11cc0cb870 2024-04-06T13:48:54.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.804+0000 7f11cbfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11cc0d11b0 0x7f11cc0c9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:54.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.804+0000 7f11cbfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11cc0d11b0 0x7f11cc0c9b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39564/0 (socket says 172.21.15.189:39564) 2024-04-06T13:48:54.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.804+0000 7f11cbfff700 1 -- 172.21.15.189:0/1543154633 learned_addr learned my addr 172.21.15.189:0/1543154633 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:54.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.804+0000 7f11caffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11cc0cfa00 0x7f11cc156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:54.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.804+0000 7f11cb7fe700 1 --2- 172.21.15.189:0/1543154633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11cc0cb870 0x7f11cc06d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:54.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.804+0000 7f11cbfff700 1 -- 172.21.15.189:0/1543154633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11cc0cb870 msgr2=0x7f11cc06d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:54.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.804+0000 7f11cbfff700 1 --2- 172.21.15.189:0/1543154633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11cc0cb870 0x7f11cc06d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.804+0000 7f11cbfff700 1 -- 172.21.15.189:0/1543154633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11cc0cfa00 msgr2=0x7f11cc156680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:54.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.804+0000 7f11cbfff700 1 --2- 172.21.15.189:0/1543154633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11cc0cfa00 0x7f11cc156680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:54.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.804+0000 7f11cbfff700 1 -- 172.21.15.189:0/1543154633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc009a70 con 0x7f11cc0d11b0 2024-04-06T13:48:54.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.804+0000 7f11cbfff700 1 --2- 172.21.15.189:0/1543154633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11cc0d11b0 0x7f11cc0c9b50 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7f11c400c500 tx=0x7f11c400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:54.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.804+0000 7f11c3fff700 1 -- 172.21.15.189:0/1543154633 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c40052d0 con 0x7f11cc0d11b0 2024-04-06T13:48:54.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.805+0000 7f11c3fff700 1 -- 172.21.15.189:0/1543154633 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c4005430 con 0x7f11cc0d11b0 2024-04-06T13:48:54.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.805+0000 7f11c3fff700 1 -- 172.21.15.189:0/1543154633 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c40038c0 con 0x7f11cc0d11b0 2024-04-06T13:48:54.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.805+0000 7f11d2101700 1 -- 172.21.15.189:0/1543154633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11cc1e57c0 con 0x7f11cc0d11b0 2024-04-06T13:48:54.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.805+0000 7f11d2101700 1 -- 172.21.15.189:0/1543154633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11cc1e5a60 con 0x7f11cc0d11b0 2024-04-06T13:48:54.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.805+0000 7f11d2101700 1 -- 172.21.15.189:0/1543154633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11cc1e5f60 con 0x7f11cc0d11b0 2024-04-06T13:48:54.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.807+0000 7f11c3fff700 1 -- 172.21.15.189:0/1543154633 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 156) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f11c4003a20 con 0x7f11cc0d11b0 2024-04-06T13:48:54.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.807+0000 7f11c3fff700 1 --2- 172.21.15.189:0/1543154633 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f11ac06aed0 0x7f11ac06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:54.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.807+0000 7f11cb7fe700 1 -- 172.21.15.189:0/1543154633 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f11ac06aed0 msgr2=0x7f11ac06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:54.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.807+0000 7f11c3fff700 1 -- 172.21.15.189:0/1543154633 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7395+0+0 (secure 0 0 0) 0x7f11c4089440 con 0x7f11cc0d11b0 2024-04-06T13:48:54.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.807+0000 7f11cb7fe700 1 --2- 172.21.15.189:0/1543154633 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f11ac06aed0 0x7f11ac06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:48:54.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:54.811+0000 7f11c3fff700 1 -- 172.21.15.189:0/1543154633 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f11c4053d60 con 0x7f11cc0d11b0 2024-04-06T13:48:54.845 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:54.844+0000 7f3525abf200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:48:54.853 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:54.852+0000 7f0bf6538200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:48:55.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.008+0000 7f11cb7fe700 1 -- 172.21.15.189:0/1543154633 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f11ac06aed0 msgr2=0x7f11ac06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:55.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.008+0000 7f11cb7fe700 1 --2- 172.21.15.189:0/1543154633 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f11ac06aed0 0x7f11ac06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:48:55.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.013+0000 7f11d2101700 1 -- 172.21.15.189:0/1543154633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f11cc074990 con 0x7f11cc0d11b0 2024-04-06T13:48:55.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.014+0000 7f11c3fff700 1 -- 172.21.15.189:0/1543154633 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v253) v1 ==== 93+0+10 (secure 0 0 0) 0x7f11c4017020 con 0x7f11cc0d11b0 2024-04-06T13:48:55.015 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:48:55.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.016+0000 7f11d2101700 1 -- 172.21.15.189:0/1543154633 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f11ac06aed0 msgr2=0x7f11ac06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:48:55.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.016+0000 7f11d2101700 1 --2- 172.21.15.189:0/1543154633 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f11ac06aed0 0x7f11ac06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.016+0000 7f11d2101700 1 -- 172.21.15.189:0/1543154633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11cc0d11b0 msgr2=0x7f11cc0c9b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:55.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.016+0000 7f11d2101700 1 --2- 172.21.15.189:0/1543154633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11cc0d11b0 0x7f11cc0c9b50 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7f11c400c500 tx=0x7f11c400c810 comp rx=0 tx=0).stop 2024-04-06T13:48:55.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.016+0000 7f11d2101700 1 -- 172.21.15.189:0/1543154633 shutdown_connections 2024-04-06T13:48:55.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.016+0000 7f11d2101700 1 --2- 172.21.15.189:0/1543154633 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f11ac06aed0 0x7f11ac06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.016+0000 7f11d2101700 1 --2- 172.21.15.189:0/1543154633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11cc0d11b0 0x7f11cc0c9b50 unknown :-1 s=CLOSED pgs=2803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.016+0000 7f11d2101700 1 --2- 172.21.15.189:0/1543154633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f11cc0cb870 0x7f11cc06d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.016+0000 7f11d2101700 1 --2- 172.21.15.189:0/1543154633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11cc0cfa00 0x7f11cc156680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.016+0000 7f11d2101700 1 -- 172.21.15.189:0/1543154633 >> 172.21.15.189:0/1543154633 conn(0x7f11cc0b2db0 msgr2=0x7f11cc15a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:55.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.016+0000 7f11d2101700 1 -- 172.21.15.189:0/1543154633 shutdown_connections 2024-04-06T13:48:55.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.016+0000 7f11d2101700 1 -- 172.21.15.189:0/1543154633 wait complete. 2024-04-06T13:48:55.019 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:55.018+0000 7f0bf6538200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:48:55.030 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T13:48:55.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.159+0000 7fc5b7ae7700 1 Processor -- start 2024-04-06T13:48:55.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.159+0000 7fc5b7ae7700 1 -- start start 2024-04-06T13:48:55.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.160+0000 7fc5b7ae7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b015d7e0 0x7fc5b00d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:55.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.160+0000 7fc5b7ae7700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc5b00d0890 0x7fc5b00d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:55.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.160+0000 7fc5b7ae7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc5b00d1af0 0x7fc5b00b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:55.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.160+0000 7fc5b7ae7700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b00d0d10 con 0x7fc5b00d1af0 2024-04-06T13:48:55.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.160+0000 7fc5b7ae7700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b00d1f80 con 0x7fc5b00d0890 2024-04-06T13:48:55.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.160+0000 7fc5b7ae7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b00cdf80 con 0x7fc5b015d7e0 2024-04-06T13:48:55.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.160+0000 7fc5b6084700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc5b00d1af0 0x7fc5b00b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:55.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.160+0000 7fc5b6084700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc5b00d1af0 0x7fc5b00b4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39572/0 (socket says 172.21.15.189:39572) 2024-04-06T13:48:55.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.160+0000 7fc5b6084700 1 -- 172.21.15.189:0/4004439975 learned_addr learned my addr 172.21.15.189:0/4004439975 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:55.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.160+0000 7fc5b5883700 1 --2- 172.21.15.189:0/4004439975 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b015d7e0 0x7fc5b00d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:55.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.160+0000 7fc5b5082700 1 --2- 172.21.15.189:0/4004439975 >> v2:172.21.15.192:3300/0 conn(0x7fc5b00d0890 0x7fc5b00d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:55.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.160+0000 7fc5b6084700 1 -- 172.21.15.189:0/4004439975 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b015d7e0 msgr2=0x7fc5b00d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:55.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.160+0000 7fc5b6084700 1 --2- 172.21.15.189:0/4004439975 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b015d7e0 0x7fc5b00d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.160+0000 7fc5b6084700 1 -- 172.21.15.189:0/4004439975 >> v2:172.21.15.192:3300/0 conn(0x7fc5b00d0890 msgr2=0x7fc5b00d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:55.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.161+0000 7fc5b6084700 1 --2- 172.21.15.189:0/4004439975 >> v2:172.21.15.192:3300/0 conn(0x7fc5b00d0890 0x7fc5b00d15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.161+0000 7fc5b6084700 1 -- 172.21.15.189:0/4004439975 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b00d3230 con 0x7fc5b00d1af0 2024-04-06T13:48:55.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.161+0000 7fc5b6084700 1 --2- 172.21.15.189:0/4004439975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5b00d1af0 0x7fc5b00b4e50 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7fc5b00b5f20 tx=0x7fc5a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e7c03b745d561b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:55.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.161+0000 7fc5b4881700 1 -- 172.21.15.189:0/4004439975 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a800e070 con 0x7fc5b00d1af0 2024-04-06T13:48:55.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.161+0000 7fc5b4881700 1 -- 172.21.15.189:0/4004439975 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a800b910 con 0x7fc5b00d1af0 2024-04-06T13:48:55.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.161+0000 7fc5b4881700 1 -- 172.21.15.189:0/4004439975 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a800bcc0 con 0x7fc5b00d1af0 2024-04-06T13:48:55.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.161+0000 7fc5b7ae7700 1 -- 172.21.15.189:0/4004439975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5b00d1af0 msgr2=0x7fc5b00b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:55.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.161+0000 7fc5b7ae7700 1 --2- 172.21.15.189:0/4004439975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5b00d1af0 0x7fc5b00b4e50 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7fc5b00b5f20 tx=0x7fc5a800b330 comp rx=0 tx=0).stop 2024-04-06T13:48:55.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.161+0000 7fc5b7ae7700 1 -- 172.21.15.189:0/4004439975 shutdown_connections 2024-04-06T13:48:55.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.161+0000 7fc5b7ae7700 1 --2- 172.21.15.189:0/4004439975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5b00d1af0 0x7fc5b00b4e50 unknown :-1 s=CLOSED pgs=2804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.161+0000 7fc5b7ae7700 1 --2- 172.21.15.189:0/4004439975 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b015d7e0 0x7fc5b00d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.161+0000 7fc5b7ae7700 1 --2- 172.21.15.189:0/4004439975 >> v2:172.21.15.192:3300/0 conn(0x7fc5b00d0890 0x7fc5b00d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.162+0000 7fc5b7ae7700 1 -- 172.21.15.189:0/4004439975 >> 172.21.15.189:0/4004439975 conn(0x7fc5b0072250 msgr2=0x7fc5b0072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:55.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.162+0000 7fc5b7ae7700 1 -- 172.21.15.189:0/4004439975 shutdown_connections 2024-04-06T13:48:55.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.162+0000 7fc5b7ae7700 1 -- 172.21.15.189:0/4004439975 wait complete. 2024-04-06T13:48:55.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.163+0000 7fc5b7ae7700 1 Processor -- start 2024-04-06T13:48:55.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.163+0000 7fc5b7ae7700 1 -- start start 2024-04-06T13:48:55.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.164+0000 7fc5b7ae7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5b00d0890 0x7fc5b01f2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:55.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.164+0000 7fc5b7ae7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b00d1af0 0x7fc5b01f5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:55.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.164+0000 7fc5b7ae7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5b015d7e0 0x7fc5b00abbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:55.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.164+0000 7fc5b7ae7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b00ac840 con 0x7fc5b00d0890 2024-04-06T13:48:55.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.164+0000 7fc5b7ae7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b01f6110 con 0x7fc5b015d7e0 2024-04-06T13:48:55.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.164+0000 7fc5b7ae7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b00a9da0 con 0x7fc5b00d1af0 2024-04-06T13:48:55.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.164+0000 7fc5b5883700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5b00d0890 0x7fc5b01f2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:55.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.164+0000 7fc5b5883700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5b00d0890 0x7fc5b01f2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39582/0 (socket says 172.21.15.189:39582) 2024-04-06T13:48:55.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.164+0000 7fc5b5883700 1 -- 172.21.15.189:0/2644214982 learned_addr learned my addr 172.21.15.189:0/2644214982 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:55.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.164+0000 7fc5b5082700 1 --2- 172.21.15.189:0/2644214982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b00d1af0 0x7fc5b01f5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:55.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.164+0000 7fc5b6084700 1 --2- 172.21.15.189:0/2644214982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5b015d7e0 0x7fc5b00abbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:55.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.165+0000 7fc5b5883700 1 -- 172.21.15.189:0/2644214982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b00d1af0 msgr2=0x7fc5b01f5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:55.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.165+0000 7fc5b5883700 1 --2- 172.21.15.189:0/2644214982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b00d1af0 0x7fc5b01f5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.165+0000 7fc5b5883700 1 -- 172.21.15.189:0/2644214982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5b015d7e0 msgr2=0x7fc5b00abbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:55.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.165+0000 7fc5b5883700 1 --2- 172.21.15.189:0/2644214982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5b015d7e0 0x7fc5b00abbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.165+0000 7fc5b5883700 1 -- 172.21.15.189:0/2644214982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a8009b30 con 0x7fc5b00d0890 2024-04-06T13:48:55.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.165+0000 7fc5b5883700 1 --2- 172.21.15.189:0/2644214982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5b00d0890 0x7fc5b01f2d60 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00e9b0 tx=0x7fc5ac00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:55.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.165+0000 7fc5a67fc700 1 -- 172.21.15.189:0/2644214982 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac009710 con 0x7fc5b00d0890 2024-04-06T13:48:55.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.165+0000 7fc5a67fc700 1 -- 172.21.15.189:0/2644214982 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ac00ee50 con 0x7fc5b00d0890 2024-04-06T13:48:55.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.165+0000 7fc5b7ae7700 1 -- 172.21.15.189:0/2644214982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b00ac560 con 0x7fc5b00d0890 2024-04-06T13:48:55.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.165+0000 7fc5a67fc700 1 -- 172.21.15.189:0/2644214982 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac00ca50 con 0x7fc5b00d0890 2024-04-06T13:48:55.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.165+0000 7fc5b7ae7700 1 -- 172.21.15.189:0/2644214982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b00aa2d0 con 0x7fc5b00d0890 2024-04-06T13:48:55.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.166+0000 7fc5b7ae7700 1 -- 172.21.15.189:0/2644214982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b00ca160 con 0x7fc5b00d0890 2024-04-06T13:48:55.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.167+0000 7fc5a67fc700 1 -- 172.21.15.189:0/2644214982 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 156) v1 ==== 89350+0+0 (secure 0 0 0) 0x7fc5ac00cbb0 con 0x7fc5b00d0890 2024-04-06T13:48:55.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.167+0000 7fc5a67fc700 1 --2- 172.21.15.189:0/2644214982 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc59406aed0 0x7fc59406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:55.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.167+0000 7fc5b5082700 1 -- 172.21.15.189:0/2644214982 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc59406aed0 msgr2=0x7fc59406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:55.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.167+0000 7fc5a67fc700 1 -- 172.21.15.189:0/2644214982 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7395+0+0 (secure 0 0 0) 0x7fc5ac089c90 con 0x7fc5b00d0890 2024-04-06T13:48:55.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.167+0000 7fc5b5082700 1 --2- 172.21.15.189:0/2644214982 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc59406aed0 0x7fc59406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:48:55.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.171+0000 7fc5a67fc700 1 -- 172.21.15.189:0/2644214982 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc5ac054500 con 0x7fc5b00d0890 2024-04-06T13:48:55.193 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:55.192+0000 7f0bf6538200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:48:55.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.368+0000 7fc5b5082700 1 -- 172.21.15.189:0/2644214982 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc59406aed0 msgr2=0x7fc59406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:55.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.368+0000 7fc5b5082700 1 --2- 172.21.15.189:0/2644214982 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc59406aed0 0x7fc59406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:48:55.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.377+0000 7fc5b7ae7700 1 -- 172.21.15.189:0/2644214982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc5b0074990 con 0x7fc5b00d0890 2024-04-06T13:48:55.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.377+0000 7fc5a67fc700 1 -- 172.21.15.189:0/2644214982 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v253) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc5ac00ce60 con 0x7fc5b00d0890 2024-04-06T13:48:55.378 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:48:55.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.379+0000 7fc5b7ae7700 1 -- 172.21.15.189:0/2644214982 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc59406aed0 msgr2=0x7fc59406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:48:55.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.379+0000 7fc5b7ae7700 1 --2- 172.21.15.189:0/2644214982 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc59406aed0 0x7fc59406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.379+0000 7fc5b7ae7700 1 -- 172.21.15.189:0/2644214982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5b00d0890 msgr2=0x7fc5b01f2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:55.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.379+0000 7fc5b7ae7700 1 --2- 172.21.15.189:0/2644214982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5b00d0890 0x7fc5b01f2d60 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00e9b0 tx=0x7fc5ac00ecc0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.379+0000 7fc5b7ae7700 1 -- 172.21.15.189:0/2644214982 shutdown_connections 2024-04-06T13:48:55.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.379+0000 7fc5b7ae7700 1 --2- 172.21.15.189:0/2644214982 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc59406aed0 0x7fc59406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.379+0000 7fc5b7ae7700 1 --2- 172.21.15.189:0/2644214982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5b015d7e0 0x7fc5b00abbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.379+0000 7fc5b7ae7700 1 --2- 172.21.15.189:0/2644214982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5b00d0890 0x7fc5b01f2d60 unknown :-1 s=CLOSED pgs=2805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.379+0000 7fc5b7ae7700 1 --2- 172.21.15.189:0/2644214982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b00d1af0 0x7fc5b01f5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.379+0000 7fc5b7ae7700 1 -- 172.21.15.189:0/2644214982 >> 172.21.15.189:0/2644214982 conn(0x7fc5b0072250 msgr2=0x7fc5b015dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:55.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.379+0000 7fc5b7ae7700 1 -- 172.21.15.189:0/2644214982 shutdown_connections 2024-04-06T13:48:55.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.379+0000 7fc5b7ae7700 1 -- 172.21.15.189:0/2644214982 wait complete. 2024-04-06T13:48:55.393 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T13:48:55.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.519+0000 7f6e95ef2700 1 Processor -- start 2024-04-06T13:48:55.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.519+0000 7f6e95ef2700 1 -- start start 2024-04-06T13:48:55.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.520+0000 7f6e95ef2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6e9015d7e0 0x7f6e900d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:55.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.520+0000 7f6e95ef2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e900d0890 0x7f6e900d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:55.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.520+0000 7f6e95ef2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e900cdf80 con 0x7f6e900d1af0 2024-04-06T13:48:55.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.520+0000 7f6e95ef2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e900d1f80 con 0x7f6e9015d7e0 2024-04-06T13:48:55.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.520+0000 7f6e95ef2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e900d3230 con 0x7f6e900d0890 2024-04-06T13:48:55.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.520+0000 7f6e8ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6e900d1af0 0x7f6e900b4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48620/0 (socket says 172.21.15.189:48620) 2024-04-06T13:48:55.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.520+0000 7f6e8ffff700 1 -- 172.21.15.189:0/527398940 learned_addr learned my addr 172.21.15.189:0/527398940 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:55.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.520+0000 7f6e8effd700 1 --2- 172.21.15.189:0/527398940 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e900d0890 0x7f6e900d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:55.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.520+0000 7f6e8f7fe700 1 --2- 172.21.15.189:0/527398940 >> v2:172.21.15.192:3300/0 conn(0x7f6e9015d7e0 0x7f6e900d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:55.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.521+0000 7f6e8e7fc700 1 -- 172.21.15.189:0/527398940 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280374518 0 0) 0x7f6e900cdf80 con 0x7f6e900d1af0 2024-04-06T13:48:55.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.521+0000 7f6e8e7fc700 1 -- 172.21.15.189:0/527398940 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e780035b0 con 0x7f6e900d1af0 2024-04-06T13:48:55.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.521+0000 7f6e8f7fe700 1 -- 172.21.15.189:0/527398940 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e900d0890 msgr2=0x7f6e900d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:55.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.521+0000 7f6e8f7fe700 1 --2- 172.21.15.189:0/527398940 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e900d0890 0x7f6e900d15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.521+0000 7f6e8f7fe700 1 -- 172.21.15.189:0/527398940 >> v1:172.21.15.189:6789/0 conn(0x7f6e900d1af0 legacy=0x7f6e900b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:55.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.521+0000 7f6e8f7fe700 1 -- 172.21.15.189:0/527398940 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e900d0cf0 con 0x7f6e9015d7e0 2024-04-06T13:48:55.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.521+0000 7f6e8effd700 1 --2- 172.21.15.189:0/527398940 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e900d0890 0x7f6e900d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:55.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.521+0000 7f6e8f7fe700 1 --2- 172.21.15.189:0/527398940 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e9015d7e0 0x7f6e900d0350 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f6e80009d90 tx=0x7f6e8000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbec07609a0392fe server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:55.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.521+0000 7f6e8e7fc700 1 -- 172.21.15.189:0/527398940 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8000e070 con 0x7f6e9015d7e0 2024-04-06T13:48:55.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.521+0000 7f6e8e7fc700 1 -- 172.21.15.189:0/527398940 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e8000bb20 con 0x7f6e9015d7e0 2024-04-06T13:48:55.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.521+0000 7f6e8e7fc700 1 -- 172.21.15.189:0/527398940 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e80012430 con 0x7f6e9015d7e0 2024-04-06T13:48:55.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.522+0000 7f6e95ef2700 1 -- 172.21.15.189:0/527398940 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e9015d7e0 msgr2=0x7f6e900d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:55.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.522+0000 7f6e95ef2700 1 --2- 172.21.15.189:0/527398940 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e9015d7e0 0x7f6e900d0350 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f6e80009d90 tx=0x7f6e8000b330 comp rx=0 tx=0).stop 2024-04-06T13:48:55.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.522+0000 7f6e95ef2700 1 -- 172.21.15.189:0/527398940 shutdown_connections 2024-04-06T13:48:55.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.522+0000 7f6e95ef2700 1 --2- 172.21.15.189:0/527398940 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e9015d7e0 0x7f6e900d0350 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.522+0000 7f6e95ef2700 1 --2- 172.21.15.189:0/527398940 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e900d0890 0x7f6e900d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.522+0000 7f6e95ef2700 1 -- 172.21.15.189:0/527398940 >> 172.21.15.189:0/527398940 conn(0x7f6e90072250 msgr2=0x7f6e90072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:55.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.522+0000 7f6e95ef2700 1 -- 172.21.15.189:0/527398940 shutdown_connections 2024-04-06T13:48:55.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.522+0000 7f6e95ef2700 1 -- 172.21.15.189:0/527398940 wait complete. 2024-04-06T13:48:55.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.523+0000 7f6e95ef2700 1 Processor -- start 2024-04-06T13:48:55.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.523+0000 7f6e95ef2700 1 -- start start 2024-04-06T13:48:55.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e95ef2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e900d0890 0x7f6e900ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:55.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e95ef2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e900d1af0 0x7f6e9014b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:55.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e95ef2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e9015d7e0 0x7f6e90156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:55.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e95ef2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e90154e90 con 0x7f6e900d0890 2024-04-06T13:48:55.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e95ef2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e900ca9e0 con 0x7f6e900d1af0 2024-04-06T13:48:55.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e95ef2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e900cb820 con 0x7f6e9015d7e0 2024-04-06T13:48:55.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e8f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e900d0890 0x7f6e900ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:55.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e8f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e900d0890 0x7f6e900ca4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39598/0 (socket says 172.21.15.189:39598) 2024-04-06T13:48:55.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e8f7fe700 1 -- 172.21.15.189:0/2493550232 learned_addr learned my addr 172.21.15.189:0/2493550232 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:55.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e8effd700 1 --2- 172.21.15.189:0/2493550232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e900d1af0 0x7f6e9014b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:55.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e8ffff700 1 --2- 172.21.15.189:0/2493550232 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e9015d7e0 0x7f6e90156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:55.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e8f7fe700 1 -- 172.21.15.189:0/2493550232 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e9015d7e0 msgr2=0x7f6e90156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:55.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e8f7fe700 1 --2- 172.21.15.189:0/2493550232 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e9015d7e0 0x7f6e90156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e8f7fe700 1 -- 172.21.15.189:0/2493550232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e900d1af0 msgr2=0x7f6e9014b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:55.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e8f7fe700 1 --2- 172.21.15.189:0/2493550232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e900d1af0 0x7f6e9014b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e8f7fe700 1 -- 172.21.15.189:0/2493550232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e840094a0 con 0x7f6e900d0890 2024-04-06T13:48:55.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e8f7fe700 1 --2- 172.21.15.189:0/2493550232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e900d0890 0x7f6e900ca4a0 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7f6e80014040 tx=0x7f6e80004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:55.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e77fff700 1 -- 172.21.15.189:0/2493550232 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e80012430 con 0x7f6e900d0890 2024-04-06T13:48:55.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e77fff700 1 -- 172.21.15.189:0/2493550232 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e80020070 con 0x7f6e900d0890 2024-04-06T13:48:55.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.524+0000 7f6e95ef2700 1 -- 172.21.15.189:0/2493550232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e80009a70 con 0x7f6e900d0890 2024-04-06T13:48:55.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.525+0000 7f6e77fff700 1 -- 172.21.15.189:0/2493550232 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e80004280 con 0x7f6e900d0890 2024-04-06T13:48:55.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.525+0000 7f6e95ef2700 1 -- 172.21.15.189:0/2493550232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e900caef0 con 0x7f6e900d0890 2024-04-06T13:48:55.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.525+0000 7f6e95ef2700 1 -- 172.21.15.189:0/2493550232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e90074990 con 0x7f6e900d0890 2024-04-06T13:48:55.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.526+0000 7f6e77fff700 1 -- 172.21.15.189:0/2493550232 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 156) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f6e800043e0 con 0x7f6e900d0890 2024-04-06T13:48:55.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.526+0000 7f6e77fff700 1 --2- 172.21.15.189:0/2493550232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6e7c06aed0 0x7f6e7c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:55.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.526+0000 7f6e8effd700 1 -- 172.21.15.189:0/2493550232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6e7c06aed0 msgr2=0x7f6e7c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:55.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.526+0000 7f6e77fff700 1 -- 172.21.15.189:0/2493550232 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7395+0+0 (secure 0 0 0) 0x7f6e80022070 con 0x7f6e900d0890 2024-04-06T13:48:55.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.526+0000 7f6e8effd700 1 --2- 172.21.15.189:0/2493550232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6e7c06aed0 0x7f6e7c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:48:55.535 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.531+0000 7f6e77fff700 1 -- 172.21.15.189:0/2493550232 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6e80056170 con 0x7f6e900d0890 2024-04-06T13:48:55.570 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:55.568+0000 7f3525abf200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:48:55.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.727+0000 7f6e8effd700 1 -- 172.21.15.189:0/2493550232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6e7c06aed0 msgr2=0x7f6e7c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:55.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.727+0000 7f6e8effd700 1 --2- 172.21.15.189:0/2493550232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6e7c06aed0 0x7f6e7c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:48:55.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.734+0000 7f6e95ef2700 1 -- 172.21.15.189:0/2493550232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6e900cdc40 con 0x7f6e900d0890 2024-04-06T13:48:55.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.734+0000 7f6e77fff700 1 -- 172.21.15.189:0/2493550232 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v253) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6e80059790 con 0x7f6e900d0890 2024-04-06T13:48:55.735 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:48:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.736+0000 7f6e95ef2700 1 -- 172.21.15.189:0/2493550232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6e7c06aed0 msgr2=0x7f6e7c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:48:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.736+0000 7f6e95ef2700 1 --2- 172.21.15.189:0/2493550232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6e7c06aed0 0x7f6e7c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.736+0000 7f6e95ef2700 1 -- 172.21.15.189:0/2493550232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e900d0890 msgr2=0x7f6e900ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.736+0000 7f6e95ef2700 1 --2- 172.21.15.189:0/2493550232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e900d0890 0x7f6e900ca4a0 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7f6e80014040 tx=0x7f6e80004ad0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.736+0000 7f6e95ef2700 1 -- 172.21.15.189:0/2493550232 shutdown_connections 2024-04-06T13:48:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.736+0000 7f6e95ef2700 1 --2- 172.21.15.189:0/2493550232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6e7c06aed0 0x7f6e7c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.736+0000 7f6e95ef2700 1 --2- 172.21.15.189:0/2493550232 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6e9015d7e0 0x7f6e90156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.736+0000 7f6e95ef2700 1 --2- 172.21.15.189:0/2493550232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e900d0890 0x7f6e900ca4a0 unknown :-1 s=CLOSED pgs=2807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.736+0000 7f6e95ef2700 1 --2- 172.21.15.189:0/2493550232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e900d1af0 0x7f6e9014b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.736+0000 7f6e95ef2700 1 -- 172.21.15.189:0/2493550232 >> 172.21.15.189:0/2493550232 conn(0x7f6e90072250 msgr2=0x7f6e900b8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.736+0000 7f6e95ef2700 1 -- 172.21.15.189:0/2493550232 shutdown_connections 2024-04-06T13:48:55.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.737+0000 7f6e95ef2700 1 -- 172.21.15.189:0/2493550232 wait complete. 2024-04-06T13:48:55.751 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-04-06T13:48:55.813 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:55.812+0000 7f3525abf200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:48:55.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.878+0000 7f26f13e7700 1 Processor -- start 2024-04-06T13:48:55.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.878+0000 7f26f13e7700 1 -- start start 2024-04-06T13:48:55.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.879+0000 7f26f13e7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ec0cb870 0x7f26ec15c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:55.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.879+0000 7f26f13e7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f26ec0cfa00 0x7f26ec0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:55.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.879+0000 7f26f13e7700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ec0cbd80 con 0x7f26ec0d11b0 2024-04-06T13:48:55.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.879+0000 7f26f13e7700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26ec0ccdc0 con 0x7f26ec0cfa00 2024-04-06T13:48:55.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.879+0000 7f26f13e7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec0d0b50 con 0x7f26ec0cb870 2024-04-06T13:48:55.882 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.879+0000 7f26eb7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f26ec0cfa00 0x7f26ec0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:55.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.879+0000 7f26eb7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f26ec0cfa00 0x7f26ec0cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39614/0 (socket says 172.21.15.189:39614) 2024-04-06T13:48:55.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.879+0000 7f26eb7fe700 1 -- 172.21.15.189:0/2882673396 learned_addr learned my addr 172.21.15.189:0/2882673396 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:55.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.879+0000 7f26ea7fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ec0cb870 0x7f26ec15c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:55.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.880+0000 7f26eb7fe700 1 -- 172.21.15.189:0/2882673396 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ec0cb870 msgr2=0x7f26ec15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:55.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.880+0000 7f26e9ffb700 1 -- 172.21.15.189:0/2882673396 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115770925 0 0) 0x7f26ec0cbd80 con 0x7f26ec0d11b0 2024-04-06T13:48:55.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.880+0000 7f26eb7fe700 1 --2- 172.21.15.189:0/2882673396 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ec0cb870 0x7f26ec15c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.880+0000 7f26eb7fe700 1 -- 172.21.15.189:0/2882673396 >> v1:172.21.15.192:6789/0 conn(0x7f26ec0d11b0 legacy=0x7f26ec0d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:55.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.880+0000 7f26eb7fe700 1 -- 172.21.15.189:0/2882673396 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec0cef20 con 0x7f26ec0cfa00 2024-04-06T13:48:55.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.880+0000 7f26eb7fe700 1 --2- 172.21.15.189:0/2882673396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26ec0cfa00 0x7f26ec0cd560 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009e50 tx=0x7f26e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecff3229987381bb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:55.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.880+0000 7f26e9ffb700 1 -- 172.21.15.189:0/2882673396 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e400e070 con 0x7f26ec0cfa00 2024-04-06T13:48:55.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.880+0000 7f26e9ffb700 1 -- 172.21.15.189:0/2882673396 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e400b910 con 0x7f26ec0cfa00 2024-04-06T13:48:55.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.880+0000 7f26e9ffb700 1 -- 172.21.15.189:0/2882673396 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e400bcc0 con 0x7f26ec0cfa00 2024-04-06T13:48:55.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.881+0000 7f26f13e7700 1 -- 172.21.15.189:0/2882673396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26ec0cfa00 msgr2=0x7f26ec0cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:55.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.881+0000 7f26f13e7700 1 --2- 172.21.15.189:0/2882673396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26ec0cfa00 0x7f26ec0cd560 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009e50 tx=0x7f26e400b330 comp rx=0 tx=0).stop 2024-04-06T13:48:55.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.881+0000 7f26f13e7700 1 -- 172.21.15.189:0/2882673396 shutdown_connections 2024-04-06T13:48:55.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.881+0000 7f26f13e7700 1 --2- 172.21.15.189:0/2882673396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26ec0cfa00 0x7f26ec0cd560 unknown :-1 s=CLOSED pgs=2808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.881+0000 7f26f13e7700 1 --2- 172.21.15.189:0/2882673396 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ec0cb870 0x7f26ec15c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.881+0000 7f26f13e7700 1 -- 172.21.15.189:0/2882673396 >> 172.21.15.189:0/2882673396 conn(0x7f26ec0b2db0 msgr2=0x7f26ec0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:55.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.881+0000 7f26f13e7700 1 -- 172.21.15.189:0/2882673396 shutdown_connections 2024-04-06T13:48:55.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.881+0000 7f26f13e7700 1 -- 172.21.15.189:0/2882673396 wait complete. 2024-04-06T13:48:55.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.882+0000 7f26f13e7700 1 Processor -- start 2024-04-06T13:48:55.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.882+0000 7f26f13e7700 1 -- start start 2024-04-06T13:48:55.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.883+0000 7f26f13e7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ec0cb870 0x7f26ec1552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:55.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.883+0000 7f26f13e7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ec0cfa00 0x7f26ec1f1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:55.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.883+0000 7f26f13e7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26ec0d11b0 0x7f26ec15efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:55.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.883+0000 7f26f13e7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec15d440 con 0x7f26ec0d11b0 2024-04-06T13:48:55.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.883+0000 7f26f13e7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec15d690 con 0x7f26ec0cb870 2024-04-06T13:48:55.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.883+0000 7f26f13e7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec15d970 con 0x7f26ec0cfa00 2024-04-06T13:48:55.887 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.883+0000 7f26eb7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26ec0d11b0 0x7f26ec15efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:55.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.883+0000 7f26eb7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26ec0d11b0 0x7f26ec15efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39628/0 (socket says 172.21.15.189:39628) 2024-04-06T13:48:55.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.883+0000 7f26eb7fe700 1 -- 172.21.15.189:0/263685296 learned_addr learned my addr 172.21.15.189:0/263685296 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:55.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.883+0000 7f26eaffd700 1 --2- 172.21.15.189:0/263685296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ec0cb870 0x7f26ec1552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:55.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.883+0000 7f26ea7fc700 1 --2- 172.21.15.189:0/263685296 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ec0cfa00 0x7f26ec1f1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:55.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.883+0000 7f26eb7fe700 1 -- 172.21.15.189:0/263685296 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ec0cfa00 msgr2=0x7f26ec1f1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:55.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.883+0000 7f26eb7fe700 1 --2- 172.21.15.189:0/263685296 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ec0cfa00 0x7f26ec1f1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.883+0000 7f26eb7fe700 1 -- 172.21.15.189:0/263685296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ec0cb870 msgr2=0x7f26ec1552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:55.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.883+0000 7f26eb7fe700 1 --2- 172.21.15.189:0/263685296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ec0cb870 0x7f26ec1552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:55.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.883+0000 7f26eb7fe700 1 -- 172.21.15.189:0/263685296 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e4009b30 con 0x7f26ec0d11b0 2024-04-06T13:48:55.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.883+0000 7f26eb7fe700 1 --2- 172.21.15.189:0/263685296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26ec0d11b0 0x7f26ec15efa0 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009e20 tx=0x7f26e4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:55.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.884+0000 7f26d37fe700 1 -- 172.21.15.189:0/263685296 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e4004880 con 0x7f26ec0d11b0 2024-04-06T13:48:55.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.884+0000 7f26d37fe700 1 -- 172.21.15.189:0/263685296 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e40099b0 con 0x7f26ec0d11b0 2024-04-06T13:48:55.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.884+0000 7f26d37fe700 1 -- 172.21.15.189:0/263685296 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e401c680 con 0x7f26ec0d11b0 2024-04-06T13:48:55.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.884+0000 7f26f13e7700 1 -- 172.21.15.189:0/263685296 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26ec0c9340 con 0x7f26ec0d11b0 2024-04-06T13:48:55.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.884+0000 7f26f13e7700 1 -- 172.21.15.189:0/263685296 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26ec0c9900 con 0x7f26ec0d11b0 2024-04-06T13:48:55.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.885+0000 7f26f13e7700 1 -- 172.21.15.189:0/263685296 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26ec074990 con 0x7f26ec0d11b0 2024-04-06T13:48:55.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.885+0000 7f26d37fe700 1 -- 172.21.15.189:0/263685296 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 156) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f26e401c7e0 con 0x7f26ec0d11b0 2024-04-06T13:48:55.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.886+0000 7f26d37fe700 1 --2- 172.21.15.189:0/263685296 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f26d806aed0 0x7f26d806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:55.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.886+0000 7f26eaffd700 1 -- 172.21.15.189:0/263685296 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f26d806aed0 msgr2=0x7f26d806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:55.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.886+0000 7f26d37fe700 1 -- 172.21.15.189:0/263685296 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7395+0+0 (secure 0 0 0) 0x7f26e408ba60 con 0x7f26ec0d11b0 2024-04-06T13:48:55.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.886+0000 7f26eaffd700 1 --2- 172.21.15.189:0/263685296 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f26d806aed0 0x7f26d806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:48:55.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:55.890+0000 7f26d37fe700 1 -- 172.21.15.189:0/263685296 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f26e40561c0 con 0x7f26ec0d11b0 2024-04-06T13:48:55.891 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:55.890+0000 7f3525abf200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:48:55.977 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:55.976+0000 7f0bf6538200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:48:56.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.087+0000 7f26eaffd700 1 -- 172.21.15.189:0/263685296 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f26d806aed0 msgr2=0x7f26d806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:56.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.087+0000 7f26eaffd700 1 --2- 172.21.15.189:0/263685296 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f26d806aed0 0x7f26d806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:48:56.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.092+0000 7f26f13e7700 1 -- 172.21.15.189:0/263685296 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f26ec0c80b0 con 0x7f26ec0d11b0 2024-04-06T13:48:56.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.092+0000 7f26d37fe700 1 -- 172.21.15.189:0/263685296 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v253) v1 ==== 107+0+11 (secure 0 0 0) 0x7f26e40597e0 con 0x7f26ec0d11b0 2024-04-06T13:48:56.093 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:48:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.095+0000 7f26f13e7700 1 -- 172.21.15.189:0/263685296 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f26d806aed0 msgr2=0x7f26d806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:48:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.095+0000 7f26f13e7700 1 --2- 172.21.15.189:0/263685296 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f26d806aed0 0x7f26d806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.095+0000 7f26f13e7700 1 -- 172.21.15.189:0/263685296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26ec0d11b0 msgr2=0x7f26ec15efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.095+0000 7f26f13e7700 1 --2- 172.21.15.189:0/263685296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26ec0d11b0 0x7f26ec15efa0 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009e20 tx=0x7f26e4004ad0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.095+0000 7f26f13e7700 1 -- 172.21.15.189:0/263685296 shutdown_connections 2024-04-06T13:48:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.095+0000 7f26f13e7700 1 --2- 172.21.15.189:0/263685296 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f26d806aed0 0x7f26d806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.095+0000 7f26f13e7700 1 --2- 172.21.15.189:0/263685296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26ec0d11b0 0x7f26ec15efa0 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.095+0000 7f26f13e7700 1 --2- 172.21.15.189:0/263685296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ec0cb870 0x7f26ec1552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.095+0000 7f26f13e7700 1 --2- 172.21.15.189:0/263685296 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26ec0cfa00 0x7f26ec1f1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.095+0000 7f26f13e7700 1 -- 172.21.15.189:0/263685296 >> 172.21.15.189:0/263685296 conn(0x7f26ec0b2db0 msgr2=0x7f26ec15a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:56.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.095+0000 7f26f13e7700 1 -- 172.21.15.189:0/263685296 shutdown_connections 2024-04-06T13:48:56.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.095+0000 7f26f13e7700 1 -- 172.21.15.189:0/263685296 wait complete. 2024-04-06T13:48:56.108 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-04-06T13:48:56.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.236+0000 7f623782c700 1 Processor -- start 2024-04-06T13:48:56.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.236+0000 7f623782c700 1 -- start start 2024-04-06T13:48:56.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.237+0000 7f623782c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f62300d17a0 0x7f623006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:56.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.237+0000 7f623782c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6230146dd0 0x7f623006d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:56.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.237+0000 7f623782c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62300cedf0 0x7f62300cf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:56.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.237+0000 7f623782c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62300cf780 con 0x7f6230146dd0 2024-04-06T13:48:56.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.237+0000 7f623782c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f623006c0a0 con 0x7f62300d17a0 2024-04-06T13:48:56.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.237+0000 7f623782c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f623006c380 con 0x7f62300cedf0 2024-04-06T13:48:56.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.237+0000 7f6234dc7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6230146dd0 0x7f623006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:56.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.237+0000 7f6234dc7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6230146dd0 0x7f623006d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39630/0 (socket says 172.21.15.189:39630) 2024-04-06T13:48:56.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.237+0000 7f6234dc7700 1 -- 172.21.15.189:0/1034334569 learned_addr learned my addr 172.21.15.189:0/1034334569 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:56.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.237+0000 7f6235dc9700 1 --2- 172.21.15.189:0/1034334569 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62300cedf0 0x7f62300cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:56.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.237+0000 7f62355c8700 1 --2- 172.21.15.189:0/1034334569 >> v2:172.21.15.192:3300/0 conn(0x7f62300d17a0 0x7f623006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:56.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.238+0000 7f6234dc7700 1 -- 172.21.15.189:0/1034334569 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62300cedf0 msgr2=0x7f62300cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:56.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.238+0000 7f6234dc7700 1 --2- 172.21.15.189:0/1034334569 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62300cedf0 0x7f62300cf240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.238+0000 7f6234dc7700 1 -- 172.21.15.189:0/1034334569 >> v2:172.21.15.192:3300/0 conn(0x7f62300d17a0 msgr2=0x7f623006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:56.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.238+0000 7f6234dc7700 1 --2- 172.21.15.189:0/1034334569 >> v2:172.21.15.192:3300/0 conn(0x7f62300d17a0 0x7f623006b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.238+0000 7f6234dc7700 1 -- 172.21.15.189:0/1034334569 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623006c660 con 0x7f6230146dd0 2024-04-06T13:48:56.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.238+0000 7f6234dc7700 1 --2- 172.21.15.189:0/1034334569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6230146dd0 0x7f623006d140 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7f623004fb40 tx=0x7f6220009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a2da0c4ea97bfc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:56.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.238+0000 7f6227fff700 1 -- 172.21.15.189:0/1034334569 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6220011830 con 0x7f6230146dd0 2024-04-06T13:48:56.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.238+0000 7f6227fff700 1 -- 172.21.15.189:0/1034334569 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6220011990 con 0x7f6230146dd0 2024-04-06T13:48:56.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.238+0000 7f6227fff700 1 -- 172.21.15.189:0/1034334569 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6220011d20 con 0x7f6230146dd0 2024-04-06T13:48:56.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.238+0000 7f623782c700 1 -- 172.21.15.189:0/1034334569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6230146dd0 msgr2=0x7f623006d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:56.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.238+0000 7f623782c700 1 --2- 172.21.15.189:0/1034334569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6230146dd0 0x7f623006d140 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7f623004fb40 tx=0x7f6220009bd0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.238+0000 7f623782c700 1 -- 172.21.15.189:0/1034334569 shutdown_connections 2024-04-06T13:48:56.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.238+0000 7f623782c700 1 --2- 172.21.15.189:0/1034334569 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62300cedf0 0x7f62300cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.238+0000 7f623782c700 1 --2- 172.21.15.189:0/1034334569 >> v2:172.21.15.192:3300/0 conn(0x7f62300d17a0 0x7f623006b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.238+0000 7f623782c700 1 --2- 172.21.15.189:0/1034334569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6230146dd0 0x7f623006d140 unknown :-1 s=CLOSED pgs=2810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.238+0000 7f623782c700 1 -- 172.21.15.189:0/1034334569 >> 172.21.15.189:0/1034334569 conn(0x7f62300b2db0 msgr2=0x7f62300b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:56.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.239+0000 7f623782c700 1 -- 172.21.15.189:0/1034334569 shutdown_connections 2024-04-06T13:48:56.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.239+0000 7f623782c700 1 -- 172.21.15.189:0/1034334569 wait complete. 2024-04-06T13:48:56.242 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.239+0000 7f0bf6538200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:48:56.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.240+0000 7f623782c700 1 Processor -- start 2024-04-06T13:48:56.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.240+0000 7f623782c700 1 -- start start 2024-04-06T13:48:56.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.241+0000 7f623782c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62300cedf0 0x7f62300add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:56.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.241+0000 7f623782c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62300d17a0 0x7f6230145a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:56.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.241+0000 7f623782c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6230146dd0 0x7f623015db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:56.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.241+0000 7f623782c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62300ae2d0 con 0x7f62300d17a0 2024-04-06T13:48:56.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.241+0000 7f623782c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6230145f60 con 0x7f6230146dd0 2024-04-06T13:48:56.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.241+0000 7f623782c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f623015e040 con 0x7f62300cedf0 2024-04-06T13:48:56.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.241+0000 7f6234dc7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62300d17a0 0x7f6230145a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:56.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.241+0000 7f6234dc7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62300d17a0 0x7f6230145a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39642/0 (socket says 172.21.15.189:39642) 2024-04-06T13:48:56.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.241+0000 7f6234dc7700 1 -- 172.21.15.189:0/1574264205 learned_addr learned my addr 172.21.15.189:0/1574264205 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:56.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.241+0000 7f62355c8700 1 --2- 172.21.15.189:0/1574264205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62300cedf0 0x7f62300add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:56.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.241+0000 7f6235dc9700 1 --2- 172.21.15.189:0/1574264205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6230146dd0 0x7f623015db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:56.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.242+0000 7f62355c8700 1 -- 172.21.15.189:0/1574264205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6230146dd0 msgr2=0x7f623015db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:56.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.242+0000 7f62355c8700 1 --2- 172.21.15.189:0/1574264205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6230146dd0 0x7f623015db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.242+0000 7f62355c8700 1 -- 172.21.15.189:0/1574264205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62300d17a0 msgr2=0x7f6230145a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:56.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.242+0000 7f62355c8700 1 --2- 172.21.15.189:0/1574264205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62300d17a0 0x7f6230145a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.242+0000 7f62355c8700 1 -- 172.21.15.189:0/1574264205 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6220009550 con 0x7f62300cedf0 2024-04-06T13:48:56.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.242+0000 7f6234dc7700 1 --2- 172.21.15.189:0/1574264205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62300d17a0 0x7f6230145a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:56.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.242+0000 7f62355c8700 1 --2- 172.21.15.189:0/1574264205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62300cedf0 0x7f62300add90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f622c00e8a0 tx=0x7f622c00ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:56.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.242+0000 7f6225ffb700 1 -- 172.21.15.189:0/1574264205 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622c013680 con 0x7f62300cedf0 2024-04-06T13:48:56.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.242+0000 7f6225ffb700 1 -- 172.21.15.189:0/1574264205 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622c0137e0 con 0x7f62300cedf0 2024-04-06T13:48:56.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.242+0000 7f6225ffb700 1 -- 172.21.15.189:0/1574264205 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622c013b70 con 0x7f62300cedf0 2024-04-06T13:48:56.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.242+0000 7f623782c700 1 -- 172.21.15.189:0/1574264205 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f623015e270 con 0x7f62300cedf0 2024-04-06T13:48:56.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.242+0000 7f623782c700 1 -- 172.21.15.189:0/1574264205 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f623015e830 con 0x7f62300cedf0 2024-04-06T13:48:56.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.243+0000 7f623782c700 1 -- 172.21.15.189:0/1574264205 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6230074990 con 0x7f62300cedf0 2024-04-06T13:48:56.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.244+0000 7f6225ffb700 1 -- 172.21.15.189:0/1574264205 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 156) v1 ==== 89350+0+0 (secure 0 0 0) 0x7f622c013cd0 con 0x7f62300cedf0 2024-04-06T13:48:56.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.244+0000 7f6225ffb700 1 --2- 172.21.15.189:0/1574264205 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f621406aed0 0x7f621406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:56.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.244+0000 7f6234dc7700 1 -- 172.21.15.189:0/1574264205 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f621406aed0 msgr2=0x7f621406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:56.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.244+0000 7f6225ffb700 1 -- 172.21.15.189:0/1574264205 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7395+0+0 (secure 0 0 0) 0x7f622c089a90 con 0x7f62300cedf0 2024-04-06T13:48:56.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.244+0000 7f6234dc7700 1 --2- 172.21.15.189:0/1574264205 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f621406aed0 0x7f621406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:48:56.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.249+0000 7f6225ffb700 1 -- 172.21.15.189:0/1574264205 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f622c054270 con 0x7f62300cedf0 2024-04-06T13:48:56.266 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:48:56.265+0000 7f3525abf200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:48:56.325 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.324+0000 7f0bf6538200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:48:56.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.445+0000 7f6234dc7700 1 -- 172.21.15.189:0/1574264205 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f621406aed0 msgr2=0x7f621406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:56.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.445+0000 7f6234dc7700 1 --2- 172.21.15.189:0/1574264205 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f621406aed0 0x7f621406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:48:56.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.450+0000 7f623782c700 1 -- 172.21.15.189:0/1574264205 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f62300d1e20 con 0x7f62300cedf0 2024-04-06T13:48:56.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.450+0000 7f6225ffb700 1 -- 172.21.15.189:0/1574264205 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v253) v1 ==== 103+0+11 (secure 0 0 0) 0x7f622c057890 con 0x7f62300cedf0 2024-04-06T13:48:56.451 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:48:56.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.452+0000 7f623782c700 1 -- 172.21.15.189:0/1574264205 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f621406aed0 msgr2=0x7f621406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:48:56.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.452+0000 7f623782c700 1 --2- 172.21.15.189:0/1574264205 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f621406aed0 0x7f621406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.453+0000 7f623782c700 1 -- 172.21.15.189:0/1574264205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62300cedf0 msgr2=0x7f62300add90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:56.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.453+0000 7f623782c700 1 --2- 172.21.15.189:0/1574264205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62300cedf0 0x7f62300add90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f622c00e8a0 tx=0x7f622c00ebb0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.453+0000 7f623782c700 1 -- 172.21.15.189:0/1574264205 shutdown_connections 2024-04-06T13:48:56.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.453+0000 7f623782c700 1 --2- 172.21.15.189:0/1574264205 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f621406aed0 0x7f621406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.453+0000 7f623782c700 1 --2- 172.21.15.189:0/1574264205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6230146dd0 0x7f623015db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.453+0000 7f623782c700 1 --2- 172.21.15.189:0/1574264205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62300cedf0 0x7f62300add90 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.454 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.453+0000 7f623782c700 1 --2- 172.21.15.189:0/1574264205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62300d17a0 0x7f6230145a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.453+0000 7f623782c700 1 -- 172.21.15.189:0/1574264205 >> 172.21.15.189:0/1574264205 conn(0x7f62300b2db0 msgr2=0x7f62300b3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:56.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.453+0000 7f623782c700 1 -- 172.21.15.189:0/1574264205 shutdown_connections 2024-04-06T13:48:56.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.453+0000 7f623782c700 1 -- 172.21.15.189:0/1574264205 wait complete. 2024-04-06T13:48:56.467 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:48:56.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.597+0000 7fb75c03c700 1 Processor -- start 2024-04-06T13:48:56.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.597+0000 7fb75c03c700 1 -- start start 2024-04-06T13:48:56.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.598+0000 7fb75c03c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb7541523d0 0x7fb7540cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:56.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.598+0000 7fb75c03c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb7540cbce0 0x7fb7540d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:56.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.598+0000 7fb75c03c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75415b9f0 con 0x7fb7540ced20 2024-04-06T13:48:56.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.598+0000 7fb75c03c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7541527b0 con 0x7fb7541523d0 2024-04-06T13:48:56.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.598+0000 7fb75c03c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb75415b7e0 con 0x7fb7540cbce0 2024-04-06T13:48:56.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.598+0000 7fb759dd8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb7541523d0 0x7fb7540cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:56.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.598+0000 7fb759dd8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb7541523d0 0x7fb7540cb7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39644/0 (socket says 172.21.15.189:39644) 2024-04-06T13:48:56.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.598+0000 7fb759dd8700 1 -- 172.21.15.189:0/2898984354 learned_addr learned my addr 172.21.15.189:0/2898984354 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:56.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.598+0000 7fb7595d7700 1 --2- 172.21.15.189:0/2898984354 >> v2:172.21.15.192:3300/0 conn(0x7fb7540cbce0 0x7fb7540d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:56.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.598+0000 7fb759dd8700 1 -- 172.21.15.189:0/2898984354 >> v2:172.21.15.192:3300/0 conn(0x7fb7540cbce0 msgr2=0x7fb7540d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:56.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.598+0000 7fb759dd8700 1 --2- 172.21.15.189:0/2898984354 >> v2:172.21.15.192:3300/0 conn(0x7fb7540cbce0 0x7fb7540d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.598+0000 7fb759dd8700 1 -- 172.21.15.189:0/2898984354 >> v1:172.21.15.192:6789/0 conn(0x7fb7540ced20 legacy=0x7fb7540d10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:56.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.598+0000 7fb759dd8700 1 -- 172.21.15.189:0/2898984354 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75415bcd0 con 0x7fb7541523d0 2024-04-06T13:48:56.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.598+0000 7fb7595d7700 1 --2- 172.21.15.189:0/2898984354 >> v2:172.21.15.192:3300/0 conn(0x7fb7540cbce0 0x7fb7540d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:56.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.598+0000 7fb759dd8700 1 --2- 172.21.15.189:0/2898984354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7541523d0 0x7fb7540cb7a0 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 crypto rx=0x7fb75415b920 tx=0x7fb750009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae99b2071eb3566 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:56.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.599+0000 7fb758dd6700 1 -- 172.21.15.189:0/2898984354 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb750011830 con 0x7fb7541523d0 2024-04-06T13:48:56.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.599+0000 7fb758dd6700 1 -- 172.21.15.189:0/2898984354 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb750011990 con 0x7fb7541523d0 2024-04-06T13:48:56.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.599+0000 7fb758dd6700 1 -- 172.21.15.189:0/2898984354 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb750011d20 con 0x7fb7541523d0 2024-04-06T13:48:56.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.599+0000 7fb75c03c700 1 -- 172.21.15.189:0/2898984354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7541523d0 msgr2=0x7fb7540cb7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:56.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.599+0000 7fb75c03c700 1 --2- 172.21.15.189:0/2898984354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7541523d0 0x7fb7540cb7a0 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 crypto rx=0x7fb75415b920 tx=0x7fb750009bd0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.599+0000 7fb75c03c700 1 -- 172.21.15.189:0/2898984354 shutdown_connections 2024-04-06T13:48:56.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.599+0000 7fb75c03c700 1 --2- 172.21.15.189:0/2898984354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7541523d0 0x7fb7540cb7a0 unknown :-1 s=CLOSED pgs=2811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.599+0000 7fb75c03c700 1 --2- 172.21.15.189:0/2898984354 >> v2:172.21.15.192:3300/0 conn(0x7fb7540cbce0 0x7fb7540d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.599+0000 7fb75c03c700 1 -- 172.21.15.189:0/2898984354 >> 172.21.15.189:0/2898984354 conn(0x7fb754066b80 msgr2=0x7fb75406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:56.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.599+0000 7fb75c03c700 1 -- 172.21.15.189:0/2898984354 shutdown_connections 2024-04-06T13:48:56.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.599+0000 7fb75c03c700 1 -- 172.21.15.189:0/2898984354 wait complete. 2024-04-06T13:48:56.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.601+0000 7fb75c03c700 1 Processor -- start 2024-04-06T13:48:56.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.601+0000 7fb75c03c700 1 -- start start 2024-04-06T13:48:56.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.601+0000 7fb75c03c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7540cbce0 0x7fb7540c8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:56.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.601+0000 7fb75c03c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7540ced20 0x7fb75413ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:56.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.601+0000 7fb75c03c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7541523d0 0x7fb7540c3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:56.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.601+0000 7fb75c03c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7540c8760 con 0x7fb7540cbce0 2024-04-06T13:48:56.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.601+0000 7fb75c03c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7540c8a00 con 0x7fb7541523d0 2024-04-06T13:48:56.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.601+0000 7fb75c03c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7540c8ce0 con 0x7fb7540ced20 2024-04-06T13:48:56.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.601+0000 7fb759dd8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7540cbce0 0x7fb7540c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:56.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.601+0000 7fb759dd8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7540cbce0 0x7fb7540c8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39652/0 (socket says 172.21.15.189:39652) 2024-04-06T13:48:56.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.601+0000 7fb759dd8700 1 -- 172.21.15.189:0/3287521986 learned_addr learned my addr 172.21.15.189:0/3287521986 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:56.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.602+0000 7fb75a5d9700 1 --2- 172.21.15.189:0/3287521986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7541523d0 0x7fb7540c3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:56.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.602+0000 7fb7595d7700 1 --2- 172.21.15.189:0/3287521986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7540ced20 0x7fb75413ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:56.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.602+0000 7fb759dd8700 1 -- 172.21.15.189:0/3287521986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7540ced20 msgr2=0x7fb75413ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:56.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.602+0000 7fb759dd8700 1 --2- 172.21.15.189:0/3287521986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7540ced20 0x7fb75413ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.602+0000 7fb759dd8700 1 -- 172.21.15.189:0/3287521986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7541523d0 msgr2=0x7fb7540c3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:56.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.602+0000 7fb759dd8700 1 --2- 172.21.15.189:0/3287521986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7541523d0 0x7fb7540c3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.602+0000 7fb759dd8700 1 -- 172.21.15.189:0/3287521986 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb750009550 con 0x7fb7540cbce0 2024-04-06T13:48:56.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.602+0000 7fb759dd8700 1 --2- 172.21.15.189:0/3287521986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7540cbce0 0x7fb7540c8220 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7fb750000c00 tx=0x7fb750004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:56.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.602+0000 7fb74a7fc700 1 -- 172.21.15.189:0/3287521986 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7500045e0 con 0x7fb7540cbce0 2024-04-06T13:48:56.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.602+0000 7fb74a7fc700 1 -- 172.21.15.189:0/3287521986 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75001f070 con 0x7fb7540cbce0 2024-04-06T13:48:56.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.602+0000 7fb75c03c700 1 -- 172.21.15.189:0/3287521986 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7540c8fc0 con 0x7fb7540cbce0 2024-04-06T13:48:56.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.602+0000 7fb74a7fc700 1 -- 172.21.15.189:0/3287521986 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7500038c0 con 0x7fb7540cbce0 2024-04-06T13:48:56.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.602+0000 7fb75c03c700 1 -- 172.21.15.189:0/3287521986 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7540c6900 con 0x7fb7540cbce0 2024-04-06T13:48:56.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.603+0000 7fb75c03c700 1 -- 172.21.15.189:0/3287521986 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7540748f0 con 0x7fb7540cbce0 2024-04-06T13:48:56.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.604+0000 7fb74a7fc700 1 -- 172.21.15.189:0/3287521986 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 156) v1 ==== 89350+0+0 (secure 0 0 0) 0x7fb750021030 con 0x7fb7540cbce0 2024-04-06T13:48:56.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.604+0000 7fb74a7fc700 1 --2- 172.21.15.189:0/3287521986 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb73806aed0 0x7fb73806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:56.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.604+0000 7fb7595d7700 1 -- 172.21.15.189:0/3287521986 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb73806aed0 msgr2=0x7fb73806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:48:56.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.604+0000 7fb74a7fc700 1 -- 172.21.15.189:0/3287521986 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7395+0+0 (secure 0 0 0) 0x7fb75008b3b0 con 0x7fb7540cbce0 2024-04-06T13:48:56.607 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.604+0000 7fb7595d7700 1 --2- 172.21.15.189:0/3287521986 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb73806aed0 0x7fb73806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:48:56.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.609+0000 7fb74a7fc700 1 -- 172.21.15.189:0/3287521986 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb750055c20 con 0x7fb7540cbce0 2024-04-06T13:48:56.724 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.724+0000 7f0bf6538200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:48:56.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.743+0000 7fb74a7fc700 1 -- 172.21.15.189:0/3287521986 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 157) v1 ==== 88856+0+0 (secure 0 0 0) 0x7fb750055ff0 con 0x7fb7540cbce0 2024-04-06T13:48:56.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.743+0000 7fb74a7fc700 1 -- 172.21.15.189:0/3287521986 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb73806aed0 msgr2=0x7fb73806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:48:56.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.743+0000 7fb74a7fc700 1 --2- 172.21.15.189:0/3287521986 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb73806aed0 0x7fb73806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.811 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.810+0000 7f0b6ec11700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.811 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.810+0000 7f0b6ec11700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.811 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.810+0000 7f0b6ec11700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.811 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.810+0000 7f0b6ec11700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.811 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.810+0000 7f0b6ec11700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.811 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.810+0000 7f0b71416700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.812 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.810+0000 7f0b71416700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.812 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.810+0000 7f0b71416700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.812 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.810+0000 7f0b71416700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.812 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.810+0000 7f0b71416700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.820 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.819+0000 7f0b71416700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.820 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.819+0000 7f0b71416700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.820 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.819+0000 7f0b71416700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.820 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.819+0000 7f0b71416700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.820 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.819+0000 7f0b71416700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.821 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.820+0000 7f0b6ec11700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.821 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.820+0000 7f0b6ec11700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.821 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.820+0000 7f0b6ec11700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.821 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.820+0000 7f0b6ec11700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.821 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:48:56.820+0000 7f0b6ec11700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:48:56.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.874+0000 7fb75c03c700 1 -- 172.21.15.189:0/3287521986 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7540cd620 con 0x7fb7540cbce0 2024-04-06T13:48:56.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.874+0000 7fb74a7fc700 1 -- 172.21.15.189:0/3287521986 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5967 (secure 0 0 0) 0x7fb75006edd0 con 0x7fb7540cbce0 2024-04-06T13:48:56.875 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:56.876 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":122,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3323503154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3323503154},{"type":"v1","addr":"172.21.15.189:6836","nonce":3323503154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":9383,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3292345173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3292345173},{"type":"v1","addr":"172.21.15.189:6837","nonce":3292345173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:48:19.674205+0000","modified":"2024-04-06T13:48:22.682911+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12108},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12108":{"gid":12108,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6833/726816746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":726816746},{"type":"v1","addr":"172.21.15.192:6833","nonce":726816746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:48:36.197004+0000","modified":"2024-04-06T13:48:39.206755+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9395},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9395":{"gid":9395,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/455737532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":455737532},{"type":"v1","addr":"172.21.15.192:6835","nonce":455737532}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-04-06T13:48:56.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.876+0000 7fb75c03c700 1 -- 172.21.15.189:0/3287521986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7540cbce0 msgr2=0x7fb7540c8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:56.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.876+0000 7fb75c03c700 1 --2- 172.21.15.189:0/3287521986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7540cbce0 0x7fb7540c8220 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7fb750000c00 tx=0x7fb750004750 comp rx=0 tx=0).stop 2024-04-06T13:48:56.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.876+0000 7fb75c03c700 1 -- 172.21.15.189:0/3287521986 shutdown_connections 2024-04-06T13:48:56.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.876+0000 7fb75c03c700 1 --2- 172.21.15.189:0/3287521986 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb73806aed0 0x7fb73806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.876+0000 7fb75c03c700 1 --2- 172.21.15.189:0/3287521986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7541523d0 0x7fb7540c3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.876+0000 7fb75c03c700 1 --2- 172.21.15.189:0/3287521986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7540cbce0 0x7fb7540c8220 unknown :-1 s=CLOSED pgs=2812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.876+0000 7fb75c03c700 1 --2- 172.21.15.189:0/3287521986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7540ced20 0x7fb75413ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:56.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.876+0000 7fb75c03c700 1 -- 172.21.15.189:0/3287521986 >> 172.21.15.189:0/3287521986 conn(0x7fb754066b80 msgr2=0x7fb75415c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:56.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.877+0000 7fb75c03c700 1 -- 172.21.15.189:0/3287521986 shutdown_connections 2024-04-06T13:48:56.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:56.877+0000 7fb75c03c700 1 -- 172.21.15.189:0/3287521986 wait complete. 2024-04-06T13:48:56.879 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 122 2024-04-06T13:48:56.890 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:48:57.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.010+0000 7efdac26a700 1 Processor -- start 2024-04-06T13:48:57.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.010+0000 7efdac26a700 1 -- start start 2024-04-06T13:48:57.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.011+0000 7efdac26a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efda406b8c0 0x7efda4168420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:57.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.011+0000 7efdac26a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda406bcd0 con 0x7efda415d7c0 2024-04-06T13:48:57.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.011+0000 7efdac26a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda406ceb0 con 0x7efda40cd3e0 2024-04-06T13:48:57.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.011+0000 7efdac26a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efda40cdfa0 con 0x7efda406b8c0 2024-04-06T13:48:57.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.011+0000 7efdaa807700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efda406b8c0 0x7efda4168420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:57.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.011+0000 7efda9805700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efda415d7c0 0x7efda406b1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48626/0 (socket says 172.21.15.189:48626) 2024-04-06T13:48:57.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.011+0000 7efda9805700 1 -- 172.21.15.189:0/1427596572 learned_addr learned my addr 172.21.15.189:0/1427596572 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:57.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.012+0000 7efdaa807700 1 -- 172.21.15.189:0/1427596572 >> v1:172.21.15.192:6789/0 conn(0x7efda40cd3e0 legacy=0x7efda40cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:57.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.012+0000 7efdaa807700 1 -- 172.21.15.189:0/1427596572 >> v1:172.21.15.189:6789/0 conn(0x7efda415d7c0 legacy=0x7efda406b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:57.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.012+0000 7efdaa807700 1 -- 172.21.15.189:0/1427596572 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda406d750 con 0x7efda406b8c0 2024-04-06T13:48:57.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.012+0000 7efdaa807700 1 --2- 172.21.15.189:0/1427596572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda406b8c0 0x7efda4168420 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7efd9c009d90 tx=0x7efd9c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6255ab5dc3760ea3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:57.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.012+0000 7efda9004700 1 -- 172.21.15.189:0/1427596572 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9c00e070 con 0x7efda406b8c0 2024-04-06T13:48:57.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.012+0000 7efda9004700 1 -- 172.21.15.189:0/1427596572 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd9c00bbd0 con 0x7efda406b8c0 2024-04-06T13:48:57.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.012+0000 7efda9004700 1 -- 172.21.15.189:0/1427596572 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9c012430 con 0x7efda406b8c0 2024-04-06T13:48:57.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.012+0000 7efdac26a700 1 -- 172.21.15.189:0/1427596572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda406b8c0 msgr2=0x7efda4168420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:57.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.012+0000 7efdac26a700 1 --2- 172.21.15.189:0/1427596572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda406b8c0 0x7efda4168420 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7efd9c009d90 tx=0x7efd9c00b330 comp rx=0 tx=0).stop 2024-04-06T13:48:57.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.012+0000 7efdac26a700 1 -- 172.21.15.189:0/1427596572 shutdown_connections 2024-04-06T13:48:57.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.012+0000 7efdac26a700 1 --2- 172.21.15.189:0/1427596572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda406b8c0 0x7efda4168420 unknown :-1 s=CLOSED pgs=2824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:57.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.012+0000 7efdac26a700 1 -- 172.21.15.189:0/1427596572 >> 172.21.15.189:0/1427596572 conn(0x7efda40cc290 msgr2=0x7efda40cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:57.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.013+0000 7efdac26a700 1 -- 172.21.15.189:0/1427596572 shutdown_connections 2024-04-06T13:48:57.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.013+0000 7efdac26a700 1 -- 172.21.15.189:0/1427596572 wait complete. 2024-04-06T13:48:57.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.014+0000 7efdac26a700 1 Processor -- start 2024-04-06T13:48:57.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.014+0000 7efdac26a700 1 -- start start 2024-04-06T13:48:57.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.014+0000 7efdac26a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda406b8c0 0x7efda4145fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:57.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.014+0000 7efdac26a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efda40cd3e0 0x7efda415f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:57.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.014+0000 7efdac26a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efda415d7c0 0x7efda415f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:57.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.014+0000 7efdac26a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda40c4d80 con 0x7efda406b8c0 2024-04-06T13:48:57.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.014+0000 7efdac26a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda41464f0 con 0x7efda415d7c0 2024-04-06T13:48:57.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.014+0000 7efdac26a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda40c49c0 con 0x7efda40cd3e0 2024-04-06T13:48:57.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.015+0000 7efdaa006700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda406b8c0 0x7efda4145fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:57.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.015+0000 7efdaa006700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda406b8c0 0x7efda4145fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39828/0 (socket says 172.21.15.189:39828) 2024-04-06T13:48:57.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.015+0000 7efdaa006700 1 -- 172.21.15.189:0/2728927073 learned_addr learned my addr 172.21.15.189:0/2728927073 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:57.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.015+0000 7efda9805700 1 --2- 172.21.15.189:0/2728927073 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efda40cd3e0 0x7efda415f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:57.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.015+0000 7efdaa807700 1 --2- 172.21.15.189:0/2728927073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efda415d7c0 0x7efda415f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:57.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.015+0000 7efdaa006700 1 -- 172.21.15.189:0/2728927073 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efda40cd3e0 msgr2=0x7efda415f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:57.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.015+0000 7efdaa006700 1 --2- 172.21.15.189:0/2728927073 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efda40cd3e0 0x7efda415f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:57.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.015+0000 7efdaa006700 1 -- 172.21.15.189:0/2728927073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efda415d7c0 msgr2=0x7efda415f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:57.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.015+0000 7efdaa006700 1 --2- 172.21.15.189:0/2728927073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efda415d7c0 0x7efda415f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:57.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.015+0000 7efdaa006700 1 -- 172.21.15.189:0/2728927073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9c009a70 con 0x7efda406b8c0 2024-04-06T13:48:57.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.015+0000 7efdaa006700 1 --2- 172.21.15.189:0/2728927073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda406b8c0 0x7efda4145fb0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7efd9400c450 tx=0x7efd9400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:57.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.015+0000 7efd9affd700 1 -- 172.21.15.189:0/2728927073 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd940052d0 con 0x7efda406b8c0 2024-04-06T13:48:57.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.015+0000 7efd9affd700 1 -- 172.21.15.189:0/2728927073 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd94005430 con 0x7efda406b8c0 2024-04-06T13:48:57.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.015+0000 7efdac26a700 1 -- 172.21.15.189:0/2728927073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda40b22a0 con 0x7efda406b8c0 2024-04-06T13:48:57.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.015+0000 7efd9affd700 1 -- 172.21.15.189:0/2728927073 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd940038c0 con 0x7efda406b8c0 2024-04-06T13:48:57.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.015+0000 7efdac26a700 1 -- 172.21.15.189:0/2728927073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda40b2860 con 0x7efda406b8c0 2024-04-06T13:48:57.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.016+0000 7efdac26a700 1 -- 172.21.15.189:0/2728927073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda4074990 con 0x7efda406b8c0 2024-04-06T13:48:57.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.017+0000 7efd9affd700 1 -- 172.21.15.189:0/2728927073 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 157) v1 ==== 88856+0+0 (secure 0 0 0) 0x7efd94003a20 con 0x7efda406b8c0 2024-04-06T13:48:57.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.017+0000 7efd9affd700 1 -- 172.21.15.189:0/2728927073 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7739+0+0 (secure 0 0 0) 0x7efd940888a0 con 0x7efda406b8c0 2024-04-06T13:48:57.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.022+0000 7efd9affd700 1 -- 172.21.15.189:0/2728927073 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efd940536e0 con 0x7efda406b8c0 2024-04-06T13:48:57.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.215+0000 7efdac26a700 1 -- 172.21.15.189:0/2728927073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efda40b5790 con 0x7efda406b8c0 2024-04-06T13:48:57.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.217+0000 7efd9affd700 1 -- 172.21.15.189:0/2728927073 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v254) v1 ==== 74+0+21265 (secure 0 0 0) 0x7efd9400ce60 con 0x7efda406b8c0 2024-04-06T13:48:57.218 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:57.218 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":254,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:48:56.738592+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:48:17.447247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-04-06T13:48:18.646229+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:48:33.912100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:48:35.170642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/1050000893":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/2601670651":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/3142793643":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/1266339172":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/3326966043":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/4043705133":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/85704189":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2120970213":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/381758424":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2635472538":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/2107771319":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2145696658":"2024-04-07T13:48:48.812402+0000","172.21.15.189:6833/111704":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/3440291825":"2024-04-07T13:48:48.812402+0000","172.21.15.189:6832/111704":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/598265929":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/485512536":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/643921686":"2024-04-07T13:48:48.812402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:48:57.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.219+0000 7efdac26a700 1 -- 172.21.15.189:0/2728927073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda406b8c0 msgr2=0x7efda4145fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:57.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.219+0000 7efdac26a700 1 --2- 172.21.15.189:0/2728927073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda406b8c0 0x7efda4145fb0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7efd9400c450 tx=0x7efd9400c760 comp rx=0 tx=0).stop 2024-04-06T13:48:57.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.219+0000 7efdac26a700 1 -- 172.21.15.189:0/2728927073 shutdown_connections 2024-04-06T13:48:57.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.219+0000 7efdac26a700 1 --2- 172.21.15.189:0/2728927073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efda415d7c0 0x7efda415f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:57.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.219+0000 7efdac26a700 1 --2- 172.21.15.189:0/2728927073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efda406b8c0 0x7efda4145fb0 unknown :-1 s=CLOSED pgs=2825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:57.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.219+0000 7efdac26a700 1 --2- 172.21.15.189:0/2728927073 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efda40cd3e0 0x7efda415f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:57.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.219+0000 7efdac26a700 1 -- 172.21.15.189:0/2728927073 >> 172.21.15.189:0/2728927073 conn(0x7efda40cc290 msgr2=0x7efda4070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:57.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.219+0000 7efdac26a700 1 -- 172.21.15.189:0/2728927073 shutdown_connections 2024-04-06T13:48:57.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.219+0000 7efdac26a700 1 -- 172.21.15.189:0/2728927073 wait complete. 2024-04-06T13:48:57.233 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-06T13:48:57.233 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:48:57.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.356+0000 7f8c27176700 1 Processor -- start 2024-04-06T13:48:57.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.356+0000 7f8c27176700 1 -- start start 2024-04-06T13:48:57.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.357+0000 7f8c27176700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c200d11b0 0x7f8c200d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:57.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.357+0000 7f8c27176700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c200d1ad0 con 0x7f8c200cb870 2024-04-06T13:48:57.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.357+0000 7f8c27176700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c200cbd80 con 0x7f8c200cfa00 2024-04-06T13:48:57.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.357+0000 7f8c27176700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c200ccdc0 con 0x7f8c200d11b0 2024-04-06T13:48:57.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.357+0000 7f8c1ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8c200cb870 0x7f8c2015c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48630/0 (socket says 172.21.15.189:48630) 2024-04-06T13:48:57.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.357+0000 7f8c1ffff700 1 -- 172.21.15.189:0/1391809278 learned_addr learned my addr 172.21.15.189:0/1391809278 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:57.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.357+0000 7f8c24f12700 1 --2- 172.21.15.189:0/1391809278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c200d11b0 0x7f8c200d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:57.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.357+0000 7f8c1f7fe700 1 -- 172.21.15.189:0/1391809278 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085129560 0 0) 0x7f8c200d1ad0 con 0x7f8c200cb870 2024-04-06T13:48:57.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.357+0000 7f8c1f7fe700 1 -- 172.21.15.189:0/1391809278 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c100035b0 con 0x7f8c200cb870 2024-04-06T13:48:57.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.358+0000 7f8c24f12700 1 -- 172.21.15.189:0/1391809278 >> v1:172.21.15.192:6789/0 conn(0x7f8c200cfa00 legacy=0x7f8c200cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:57.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.358+0000 7f8c24f12700 1 -- 172.21.15.189:0/1391809278 >> v1:172.21.15.189:6789/0 conn(0x7f8c200cb870 legacy=0x7f8c2015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:57.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.358+0000 7f8c1f7fe700 1 -- 172.21.15.189:0/1391809278 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253011237 0 0) 0x7f8c200cbd80 con 0x7f8c200cfa00 2024-04-06T13:48:57.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.358+0000 7f8c24f12700 1 -- 172.21.15.189:0/1391809278 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c200cef20 con 0x7f8c200d11b0 2024-04-06T13:48:57.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.358+0000 7f8c24f12700 1 --2- 172.21.15.189:0/1391809278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c200d11b0 0x7f8c200d1590 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8c14009870 tx=0x7f8c14009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3a70e584aa6316 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:57.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.358+0000 7f8c1f7fe700 1 -- 172.21.15.189:0/1391809278 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c14011830 con 0x7f8c200d11b0 2024-04-06T13:48:57.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.358+0000 7f8c1f7fe700 1 -- 172.21.15.189:0/1391809278 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c14011990 con 0x7f8c200d11b0 2024-04-06T13:48:57.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.358+0000 7f8c1f7fe700 1 -- 172.21.15.189:0/1391809278 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c14011d20 con 0x7f8c200d11b0 2024-04-06T13:48:57.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.358+0000 7f8c27176700 1 -- 172.21.15.189:0/1391809278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c200d11b0 msgr2=0x7f8c200d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:57.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.358+0000 7f8c27176700 1 --2- 172.21.15.189:0/1391809278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c200d11b0 0x7f8c200d1590 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8c14009870 tx=0x7f8c14009b80 comp rx=0 tx=0).stop 2024-04-06T13:48:57.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.359+0000 7f8c27176700 1 -- 172.21.15.189:0/1391809278 shutdown_connections 2024-04-06T13:48:57.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.359+0000 7f8c27176700 1 --2- 172.21.15.189:0/1391809278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c200d11b0 0x7f8c200d1590 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:57.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.359+0000 7f8c27176700 1 -- 172.21.15.189:0/1391809278 >> 172.21.15.189:0/1391809278 conn(0x7f8c200b2db0 msgr2=0x7f8c200b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:57.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.359+0000 7f8c27176700 1 -- 172.21.15.189:0/1391809278 shutdown_connections 2024-04-06T13:48:57.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.359+0000 7f8c27176700 1 -- 172.21.15.189:0/1391809278 wait complete. 2024-04-06T13:48:57.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.360+0000 7f8c27176700 1 Processor -- start 2024-04-06T13:48:57.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.360+0000 7f8c27176700 1 -- start start 2024-04-06T13:48:57.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c27176700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c200cb870 0x7f8c201527d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:57.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c27176700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c200cfa00 0x7f8c2015ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:57.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c27176700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c200d11b0 0x7f8c20157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:57.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c27176700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2015e0b0 con 0x7f8c200d11b0 2024-04-06T13:48:57.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c27176700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c201570e0 con 0x7f8c200cfa00 2024-04-06T13:48:57.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c27176700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c20152d10 con 0x7f8c200cb870 2024-04-06T13:48:57.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c25713700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c200d11b0 0x7f8c20157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:57.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c25713700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c200d11b0 0x7f8c20157570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39838/0 (socket says 172.21.15.189:39838) 2024-04-06T13:48:57.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c25713700 1 -- 172.21.15.189:0/2966047188 learned_addr learned my addr 172.21.15.189:0/2966047188 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:57.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c24f12700 1 --2- 172.21.15.189:0/2966047188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c200cb870 0x7f8c201527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:57.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c1ffff700 1 --2- 172.21.15.189:0/2966047188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c200cfa00 0x7f8c2015ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:57.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c25713700 1 -- 172.21.15.189:0/2966047188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c200cb870 msgr2=0x7f8c201527d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:57.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c25713700 1 --2- 172.21.15.189:0/2966047188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c200cb870 0x7f8c201527d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:57.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c25713700 1 -- 172.21.15.189:0/2966047188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c200cfa00 msgr2=0x7f8c2015ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:57.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c25713700 1 --2- 172.21.15.189:0/2966047188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c200cfa00 0x7f8c2015ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:57.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c25713700 1 -- 172.21.15.189:0/2966047188 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c140094a0 con 0x7f8c200d11b0 2024-04-06T13:48:57.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c1ffff700 1 --2- 172.21.15.189:0/2966047188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c200cfa00 0x7f8c2015ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:57.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c25713700 1 --2- 172.21.15.189:0/2966047188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c200d11b0 0x7f8c20157570 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7f8c18001560 tx=0x7f8c1800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:57.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c1d7fa700 1 -- 172.21.15.189:0/2966047188 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18003e30 con 0x7f8c200d11b0 2024-04-06T13:48:57.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c1d7fa700 1 -- 172.21.15.189:0/2966047188 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c18005070 con 0x7f8c200d11b0 2024-04-06T13:48:57.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c1d7fa700 1 -- 172.21.15.189:0/2966047188 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18005420 con 0x7f8c200d11b0 2024-04-06T13:48:57.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c27176700 1 -- 172.21.15.189:0/2966047188 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c2015f150 con 0x7f8c200d11b0 2024-04-06T13:48:57.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.361+0000 7f8c27176700 1 -- 172.21.15.189:0/2966047188 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c201567b0 con 0x7f8c200d11b0 2024-04-06T13:48:57.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.362+0000 7f8c27176700 1 -- 172.21.15.189:0/2966047188 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c20157290 con 0x7f8c200d11b0 2024-04-06T13:48:57.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.363+0000 7f8c1d7fa700 1 -- 172.21.15.189:0/2966047188 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 157) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f8c18005580 con 0x7f8c200d11b0 2024-04-06T13:48:57.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.363+0000 7f8c1d7fa700 1 -- 172.21.15.189:0/2966047188 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7739+0+0 (secure 0 0 0) 0x7f8c18088dd0 con 0x7f8c200d11b0 2024-04-06T13:48:57.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.368+0000 7f8c1d7fa700 1 -- 172.21.15.189:0/2966047188 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8c20157290 con 0x7f8c200d11b0 2024-04-06T13:48:57.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.604+0000 7f8c27176700 1 -- 172.21.15.189:0/2966047188 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c20157250 con 0x7f8c200d11b0 2024-04-06T13:48:57.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.605+0000 7f8c1d7fa700 1 -- 172.21.15.189:0/2966047188 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5967 (secure 0 0 0) 0x7f8c18017070 con 0x7f8c200d11b0 2024-04-06T13:48:57.606 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:57.606 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":122,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3323503154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3323503154},{"type":"v1","addr":"172.21.15.189:6836","nonce":3323503154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":9383,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3292345173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3292345173},{"type":"v1","addr":"172.21.15.189:6837","nonce":3292345173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:48:19.674205+0000","modified":"2024-04-06T13:48:22.682911+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12108},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12108":{"gid":12108,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6833/726816746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":726816746},{"type":"v1","addr":"172.21.15.192:6833","nonce":726816746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:48:36.197004+0000","modified":"2024-04-06T13:48:39.206755+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9395},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9395":{"gid":9395,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/455737532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":455737532},{"type":"v1","addr":"172.21.15.192:6835","nonce":455737532}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-04-06T13:48:57.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.607+0000 7f8c27176700 1 -- 172.21.15.189:0/2966047188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c200d11b0 msgr2=0x7f8c20157570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:57.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.607+0000 7f8c27176700 1 --2- 172.21.15.189:0/2966047188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c200d11b0 0x7f8c20157570 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7f8c18001560 tx=0x7f8c1800cc00 comp rx=0 tx=0).stop 2024-04-06T13:48:57.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.607+0000 7f8c27176700 1 -- 172.21.15.189:0/2966047188 shutdown_connections 2024-04-06T13:48:57.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.607+0000 7f8c27176700 1 --2- 172.21.15.189:0/2966047188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c200d11b0 0x7f8c20157570 unknown :-1 s=CLOSED pgs=2827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:57.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.607+0000 7f8c27176700 1 --2- 172.21.15.189:0/2966047188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c200cb870 0x7f8c201527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:57.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.607+0000 7f8c27176700 1 --2- 172.21.15.189:0/2966047188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c200cfa00 0x7f8c2015ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:57.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.607+0000 7f8c27176700 1 -- 172.21.15.189:0/2966047188 >> 172.21.15.189:0/2966047188 conn(0x7f8c200b2db0 msgr2=0x7f8c2015a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:57.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.607+0000 7f8c27176700 1 -- 172.21.15.189:0/2966047188 shutdown_connections 2024-04-06T13:48:57.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.607+0000 7f8c27176700 1 -- 172.21.15.189:0/2966047188 wait complete. 2024-04-06T13:48:57.609 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 122 2024-04-06T13:48:57.620 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:48:57.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.740+0000 7f3c88154700 1 Processor -- start 2024-04-06T13:48:57.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.740+0000 7f3c88154700 1 -- start start 2024-04-06T13:48:57.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.741+0000 7f3c88154700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3c800b4660 0x7f3c800b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:57.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.741+0000 7f3c88154700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c800b5030 con 0x7f3c800b2fc0 2024-04-06T13:48:57.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.741+0000 7f3c88154700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c800cdbe0 con 0x7f3c8015a550 2024-04-06T13:48:57.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.741+0000 7f3c88154700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c8015a940 con 0x7f3c800b4660 2024-04-06T13:48:57.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.741+0000 7f3c866f1700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3c800b2fc0 0x7f3c800b33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48634/0 (socket says 172.21.15.189:48634) 2024-04-06T13:48:57.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.742+0000 7f3c866f1700 1 -- 172.21.15.189:0/1776650359 learned_addr learned my addr 172.21.15.189:0/1776650359 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:57.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.742+0000 7f3c856ef700 1 --2- 172.21.15.189:0/1776650359 >> v2:172.21.15.192:3300/0 conn(0x7f3c800b4660 0x7f3c800b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:57.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.742+0000 7f3c84eee700 1 -- 172.21.15.189:0/1776650359 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581888810 0 0) 0x7f3c800b5030 con 0x7f3c800b2fc0 2024-04-06T13:48:57.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.742+0000 7f3c84eee700 1 -- 172.21.15.189:0/1776650359 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c6c0035b0 con 0x7f3c800b2fc0 2024-04-06T13:48:57.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.742+0000 7f3c856ef700 1 -- 172.21.15.189:0/1776650359 >> v1:172.21.15.192:6789/0 conn(0x7f3c8015a550 legacy=0x7f3c800cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:57.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.742+0000 7f3c84eee700 1 -- 172.21.15.189:0/1776650359 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4107893111 0 0) 0x7f3c6c0035b0 con 0x7f3c800b2fc0 2024-04-06T13:48:57.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.742+0000 7f3c856ef700 1 -- 172.21.15.189:0/1776650359 >> v1:172.21.15.189:6789/0 conn(0x7f3c800b2fc0 legacy=0x7f3c800b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:57.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.742+0000 7f3c856ef700 1 -- 172.21.15.189:0/1776650359 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8015ac00 con 0x7f3c800b4660 2024-04-06T13:48:57.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.742+0000 7f3c856ef700 1 --2- 172.21.15.189:0/1776650359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c800b4660 0x7f3c800b4af0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f3c70009840 tx=0x7f3c70009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13f05fcb4481296e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:57.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.742+0000 7f3c84eee700 1 -- 172.21.15.189:0/1776650359 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c780019f0 con 0x7f3c800b4660 2024-04-06T13:48:57.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.743+0000 7f3c84eee700 1 -- 172.21.15.189:0/1776650359 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c70004030 con 0x7f3c800b4660 2024-04-06T13:48:57.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.743+0000 7f3c84eee700 1 -- 172.21.15.189:0/1776650359 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c700043a0 con 0x7f3c800b4660 2024-04-06T13:48:57.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.743+0000 7f3c88154700 1 -- 172.21.15.189:0/1776650359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c800b4660 msgr2=0x7f3c800b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:57.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.743+0000 7f3c88154700 1 --2- 172.21.15.189:0/1776650359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c800b4660 0x7f3c800b4af0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f3c70009840 tx=0x7f3c70009b50 comp rx=0 tx=0).stop 2024-04-06T13:48:57.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.743+0000 7f3c88154700 1 -- 172.21.15.189:0/1776650359 shutdown_connections 2024-04-06T13:48:57.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.743+0000 7f3c88154700 1 --2- 172.21.15.189:0/1776650359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c800b4660 0x7f3c800b4af0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:57.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.743+0000 7f3c88154700 1 -- 172.21.15.189:0/1776650359 >> 172.21.15.189:0/1776650359 conn(0x7f3c800d00b0 msgr2=0x7f3c800d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:57.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.743+0000 7f3c88154700 1 -- 172.21.15.189:0/1776650359 shutdown_connections 2024-04-06T13:48:57.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.743+0000 7f3c88154700 1 -- 172.21.15.189:0/1776650359 wait complete. 2024-04-06T13:48:57.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.745+0000 7f3c88154700 1 Processor -- start 2024-04-06T13:48:57.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.745+0000 7f3c88154700 1 -- start start 2024-04-06T13:48:57.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.746+0000 7f3c88154700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c800b2fc0 0x7f3c800a8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:57.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.746+0000 7f3c88154700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c800b4660 0x7f3c800a9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:57.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.746+0000 7f3c88154700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c8015c3d0 0x7f3c801523c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:57.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.746+0000 7f3c88154700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8015c940 con 0x7f3c800b4660 2024-04-06T13:48:57.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.746+0000 7f3c88154700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8015cc20 con 0x7f3c800b2fc0 2024-04-06T13:48:57.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.746+0000 7f3c88154700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8015cf00 con 0x7f3c8015c3d0 2024-04-06T13:48:57.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.747+0000 7f3c866f1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c8015c3d0 0x7f3c801523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:57.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.747+0000 7f3c866f1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c8015c3d0 0x7f3c801523c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:58606/0 (socket says 172.21.15.189:58606) 2024-04-06T13:48:57.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.747+0000 7f3c866f1700 1 -- 172.21.15.189:0/824769513 learned_addr learned my addr 172.21.15.189:0/824769513 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:57.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.747+0000 7f3c85ef0700 1 --2- 172.21.15.189:0/824769513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c800b2fc0 0x7f3c800a8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:57.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.748+0000 7f3c85ef0700 1 -- 172.21.15.189:0/824769513 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c8015c3d0 msgr2=0x7f3c801523c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:57.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.748+0000 7f3c85ef0700 1 --2- 172.21.15.189:0/824769513 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c8015c3d0 0x7f3c801523c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:57.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.748+0000 7f3c85ef0700 1 -- 172.21.15.189:0/824769513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c800b4660 msgr2=0x7f3c800a9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:57.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.748+0000 7f3c85ef0700 1 --2- 172.21.15.189:0/824769513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c800b4660 0x7f3c800a9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:57.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.748+0000 7f3c85ef0700 1 -- 172.21.15.189:0/824769513 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c700094a0 con 0x7f3c800b2fc0 2024-04-06T13:48:57.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.751+0000 7f3c85ef0700 1 --2- 172.21.15.189:0/824769513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c800b2fc0 0x7f3c800a8520 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c01d3d0 tx=0x7f3c7c01d6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:57.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.751+0000 7f3c767fc700 1 -- 172.21.15.189:0/824769513 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c01de90 con 0x7f3c800b2fc0 2024-04-06T13:48:57.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.751+0000 7f3c88154700 1 -- 172.21.15.189:0/824769513 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8015d1e0 con 0x7f3c800b2fc0 2024-04-06T13:48:57.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.751+0000 7f3c767fc700 1 -- 172.21.15.189:0/824769513 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7c0050d0 con 0x7f3c800b2fc0 2024-04-06T13:48:57.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.751+0000 7f3c767fc700 1 -- 172.21.15.189:0/824769513 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c005480 con 0x7f3c800b2fc0 2024-04-06T13:48:57.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.751+0000 7f3c88154700 1 -- 172.21.15.189:0/824769513 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c800a9670 con 0x7f3c800b2fc0 2024-04-06T13:48:57.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.752+0000 7f3c88154700 1 -- 172.21.15.189:0/824769513 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c80067bb0 con 0x7f3c800b2fc0 2024-04-06T13:48:57.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.753+0000 7f3c767fc700 1 -- 172.21.15.189:0/824769513 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 158) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f3c7c003680 con 0x7f3c800b2fc0 2024-04-06T13:48:57.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.753+0000 7f3c767fc700 1 --2- 172.21.15.189:0/824769513 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3c6406aae0 0x7f3c6406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:57.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.753+0000 7f3c767fc700 1 -- 172.21.15.189:0/824769513 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7739+0+0 (secure 0 0 0) 0x7f3c7c099360 con 0x7f3c800b2fc0 2024-04-06T13:48:57.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.753+0000 7f3c856ef700 1 --2- 172.21.15.189:0/824769513 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3c6406aae0 0x7f3c6406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:57.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.754+0000 7f3c856ef700 1 --2- 172.21.15.189:0/824769513 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3c6406aae0 0x7f3c6406cfa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3c700090b0 tx=0x7f3c70010000 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:57.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:57.758+0000 7f3c767fc700 1 -- 172.21.15.189:0/824769513 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3c7c064010 con 0x7f3c800b2fc0 2024-04-06T13:48:58.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.001+0000 7f3c88154700 1 -- 172.21.15.189:0/824769513 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c80074990 con 0x7f3c800b2fc0 2024-04-06T13:48:58.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.002+0000 7f3c767fc700 1 -- 172.21.15.189:0/824769513 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5967 (secure 0 0 0) 0x7f3c7c067630 con 0x7f3c800b2fc0 2024-04-06T13:48:58.003 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:58.003 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":122,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3323503154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3323503154},{"type":"v1","addr":"172.21.15.189:6836","nonce":3323503154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":9383,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3292345173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3292345173},{"type":"v1","addr":"172.21.15.189:6837","nonce":3292345173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:48:19.674205+0000","modified":"2024-04-06T13:48:22.682911+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12108},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12108":{"gid":12108,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6833/726816746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":726816746},{"type":"v1","addr":"172.21.15.192:6833","nonce":726816746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:48:36.197004+0000","modified":"2024-04-06T13:48:39.206755+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9395},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9395":{"gid":9395,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/455737532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":455737532},{"type":"v1","addr":"172.21.15.192:6835","nonce":455737532}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-04-06T13:48:58.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.004+0000 7f3c88154700 1 -- 172.21.15.189:0/824769513 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3c6406aae0 msgr2=0x7f3c6406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:58.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.004+0000 7f3c88154700 1 --2- 172.21.15.189:0/824769513 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3c6406aae0 0x7f3c6406cfa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3c700090b0 tx=0x7f3c70010000 comp rx=0 tx=0).stop 2024-04-06T13:48:58.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.004+0000 7f3c88154700 1 -- 172.21.15.189:0/824769513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c800b2fc0 msgr2=0x7f3c800a8520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:58.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.004+0000 7f3c88154700 1 --2- 172.21.15.189:0/824769513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c800b2fc0 0x7f3c800a8520 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c01d3d0 tx=0x7f3c7c01d6e0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.004+0000 7f3c88154700 1 -- 172.21.15.189:0/824769513 shutdown_connections 2024-04-06T13:48:58.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.004+0000 7f3c88154700 1 --2- 172.21.15.189:0/824769513 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3c6406aae0 0x7f3c6406cfa0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.004+0000 7f3c88154700 1 --2- 172.21.15.189:0/824769513 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c8015c3d0 0x7f3c801523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.004+0000 7f3c88154700 1 --2- 172.21.15.189:0/824769513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c800b2fc0 0x7f3c800a8520 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.004+0000 7f3c88154700 1 --2- 172.21.15.189:0/824769513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c800b4660 0x7f3c800a9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.004+0000 7f3c88154700 1 -- 172.21.15.189:0/824769513 >> 172.21.15.189:0/824769513 conn(0x7f3c800d00b0 msgr2=0x7f3c80154dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:58.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.004+0000 7f3c88154700 1 -- 172.21.15.189:0/824769513 shutdown_connections 2024-04-06T13:48:58.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.004+0000 7f3c88154700 1 -- 172.21.15.189:0/824769513 wait complete. 2024-04-06T13:48:58.006 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 122 2024-04-06T13:48:58.017 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:48:58.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.137+0000 7f6259d5f700 1 Processor -- start 2024-04-06T13:48:58.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.137+0000 7f6259d5f700 1 -- start start 2024-04-06T13:48:58.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.138+0000 7f6259d5f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f62540b4660 0x7f62540b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:58.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.138+0000 7f6259d5f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f62540b2fc0 0x7f62540b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:58.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.138+0000 7f6259d5f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62540b5030 con 0x7f625415a550 2024-04-06T13:48:58.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.138+0000 7f6259d5f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62540cdbe0 con 0x7f62540b4660 2024-04-06T13:48:58.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.138+0000 7f6259d5f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f625415a940 con 0x7f62540b2fc0 2024-04-06T13:48:58.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.138+0000 7f6252ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f62540b4660 0x7f62540b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:58.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.138+0000 7f62537fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f625415a550 0x7f62540cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48636/0 (socket says 172.21.15.189:48636) 2024-04-06T13:48:58.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.138+0000 7f6252ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f62540b4660 0x7f62540b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39846/0 (socket says 172.21.15.189:39846) 2024-04-06T13:48:58.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.138+0000 7f62537fe700 1 -- 172.21.15.189:0/1032438187 learned_addr learned my addr 172.21.15.189:0/1032438187 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:58.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.139+0000 7f6253fff700 1 --2- 172.21.15.189:0/1032438187 >> v2:172.21.15.192:3300/0 conn(0x7f62540b2fc0 0x7f62540b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:58.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.139+0000 7f6252ffd700 1 -- 172.21.15.189:0/1032438187 >> v2:172.21.15.192:3300/0 conn(0x7f62540b2fc0 msgr2=0x7f62540b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:58.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.139+0000 7f6252ffd700 1 --2- 172.21.15.189:0/1032438187 >> v2:172.21.15.192:3300/0 conn(0x7f62540b2fc0 0x7f62540b33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.139+0000 7f6252ffd700 1 -- 172.21.15.189:0/1032438187 >> v1:172.21.15.189:6789/0 conn(0x7f625415a550 legacy=0x7f62540cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:58.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.139+0000 7f6252ffd700 1 -- 172.21.15.189:0/1032438187 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625415ac00 con 0x7f62540b4660 2024-04-06T13:48:58.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.139+0000 7f62527fc700 1 -- 172.21.15.189:0/1032438187 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570340992 0 0) 0x7f62540b5030 con 0x7f625415a550 2024-04-06T13:48:58.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.139+0000 7f6252ffd700 1 --2- 172.21.15.189:0/1032438187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62540b4660 0x7f62540b4af0 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 crypto rx=0x7f6254072830 tx=0x7f624400bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9c0a212a4a212c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:58.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.139+0000 7f62527fc700 1 -- 172.21.15.189:0/1032438187 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624400dc70 con 0x7f62540b4660 2024-04-06T13:48:58.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.139+0000 7f62527fc700 1 -- 172.21.15.189:0/1032438187 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624400ddd0 con 0x7f62540b4660 2024-04-06T13:48:58.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.139+0000 7f62527fc700 1 -- 172.21.15.189:0/1032438187 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6244010690 con 0x7f62540b4660 2024-04-06T13:48:58.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.140+0000 7f6259d5f700 1 -- 172.21.15.189:0/1032438187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62540b4660 msgr2=0x7f62540b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:58.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.140+0000 7f6259d5f700 1 --2- 172.21.15.189:0/1032438187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62540b4660 0x7f62540b4af0 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 crypto rx=0x7f6254072830 tx=0x7f624400bdc0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.140+0000 7f6259d5f700 1 -- 172.21.15.189:0/1032438187 shutdown_connections 2024-04-06T13:48:58.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.140+0000 7f6259d5f700 1 --2- 172.21.15.189:0/1032438187 >> v2:172.21.15.192:3300/0 conn(0x7f62540b2fc0 0x7f62540b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.140+0000 7f6259d5f700 1 --2- 172.21.15.189:0/1032438187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62540b4660 0x7f62540b4af0 unknown :-1 s=CLOSED pgs=2830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.140+0000 7f6259d5f700 1 -- 172.21.15.189:0/1032438187 >> 172.21.15.189:0/1032438187 conn(0x7f62540d00b0 msgr2=0x7f62540d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:58.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.140+0000 7f6259d5f700 1 -- 172.21.15.189:0/1032438187 shutdown_connections 2024-04-06T13:48:58.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.140+0000 7f6259d5f700 1 -- 172.21.15.189:0/1032438187 wait complete. 2024-04-06T13:48:58.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.141+0000 7f6259d5f700 1 Processor -- start 2024-04-06T13:48:58.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.141+0000 7f6259d5f700 1 -- start start 2024-04-06T13:48:58.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.142+0000 7f6259d5f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62540b2fc0 0x7f625414a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:58.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.142+0000 7f6259d5f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62540b4660 0x7f6254149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:58.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.142+0000 7f6259d5f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6254157f90 0x7f6254158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:58.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.142+0000 7f6259d5f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625415c410 con 0x7f6254157f90 2024-04-06T13:48:58.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.142+0000 7f6259d5f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62541565f0 con 0x7f62540b2fc0 2024-04-06T13:48:58.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.142+0000 7f6259d5f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62541568d0 con 0x7f62540b4660 2024-04-06T13:48:58.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.142+0000 7f6253fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6254157f90 0x7f6254158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:58.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.142+0000 7f6253fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6254157f90 0x7f6254158440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39848/0 (socket says 172.21.15.189:39848) 2024-04-06T13:48:58.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.142+0000 7f6253fff700 1 -- 172.21.15.189:0/861052971 learned_addr learned my addr 172.21.15.189:0/861052971 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:58.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.143+0000 7f62537fe700 1 --2- 172.21.15.189:0/861052971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62540b2fc0 0x7f625414a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:58.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.143+0000 7f6252ffd700 1 --2- 172.21.15.189:0/861052971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62540b4660 0x7f6254149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:58.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.143+0000 7f6253fff700 1 -- 172.21.15.189:0/861052971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62540b4660 msgr2=0x7f6254149940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:58.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.143+0000 7f6253fff700 1 --2- 172.21.15.189:0/861052971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62540b4660 0x7f6254149940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.143+0000 7f6253fff700 1 -- 172.21.15.189:0/861052971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62540b2fc0 msgr2=0x7f625414a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:58.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.143+0000 7f6253fff700 1 --2- 172.21.15.189:0/861052971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62540b2fc0 0x7f625414a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.143+0000 7f6253fff700 1 -- 172.21.15.189:0/861052971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f624400b740 con 0x7f6254157f90 2024-04-06T13:48:58.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.143+0000 7f6252ffd700 1 --2- 172.21.15.189:0/861052971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62540b4660 0x7f6254149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:58.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.143+0000 7f6253fff700 1 --2- 172.21.15.189:0/861052971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6254157f90 0x7f6254158440 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7f624c007e60 tx=0x7f624c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:58.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.143+0000 7f624bfff700 1 -- 172.21.15.189:0/861052971 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c00ede0 con 0x7f6254157f90 2024-04-06T13:48:58.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.143+0000 7f624bfff700 1 -- 172.21.15.189:0/861052971 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624c014420 con 0x7f6254157f90 2024-04-06T13:48:58.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.143+0000 7f6259d5f700 1 -- 172.21.15.189:0/861052971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f625415c6e0 con 0x7f6254157f90 2024-04-06T13:48:58.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.143+0000 7f624bfff700 1 -- 172.21.15.189:0/861052971 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c0147d0 con 0x7f6254157f90 2024-04-06T13:48:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.143+0000 7f6259d5f700 1 -- 172.21.15.189:0/861052971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6254156280 con 0x7f6254157f90 2024-04-06T13:48:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.144+0000 7f6259d5f700 1 -- 172.21.15.189:0/861052971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f625415c550 con 0x7f6254157f90 2024-04-06T13:48:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.144+0000 7f624bfff700 1 -- 172.21.15.189:0/861052971 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 158) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f624c014930 con 0x7f6254157f90 2024-04-06T13:48:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.145+0000 7f624bfff700 1 --2- 172.21.15.189:0/861052971 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f623406aae0 0x7f623406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.145+0000 7f624bfff700 1 -- 172.21.15.189:0/861052971 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7739+0+0 (secure 0 0 0) 0x7f624c089de0 con 0x7f6254157f90 2024-04-06T13:48:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.145+0000 7f62537fe700 1 --2- 172.21.15.189:0/861052971 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f623406aae0 0x7f623406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.145+0000 7f62537fe700 1 --2- 172.21.15.189:0/861052971 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f623406aae0 0x7f623406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f623c007fd0 tx=0x7f623c007de0 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:58.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.149+0000 7f624bfff700 1 -- 172.21.15.189:0/861052971 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f624c054b10 con 0x7f6254157f90 2024-04-06T13:48:58.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.344+0000 7f6259d5f700 1 -- 172.21.15.189:0/861052971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f625415cc00 con 0x7f6254157f90 2024-04-06T13:48:58.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.345+0000 7f624bfff700 1 -- 172.21.15.189:0/861052971 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v254) v1 ==== 74+0+21265 (secure 0 0 0) 0x7f625415cc00 con 0x7f6254157f90 2024-04-06T13:48:58.347 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:48:58.347 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":254,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:48:56.738592+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:48:17.447247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-04-06T13:48:18.646229+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:48:33.912100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:48:35.170642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/1050000893":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/2601670651":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/3142793643":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/1266339172":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/3326966043":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/4043705133":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/85704189":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2120970213":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/381758424":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2635472538":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/2107771319":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2145696658":"2024-04-07T13:48:48.812402+0000","172.21.15.189:6833/111704":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/3440291825":"2024-04-07T13:48:48.812402+0000","172.21.15.189:6832/111704":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/598265929":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/485512536":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/643921686":"2024-04-07T13:48:48.812402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:48:58.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.348+0000 7f6259d5f700 1 -- 172.21.15.189:0/861052971 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f623406aae0 msgr2=0x7f623406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:58.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.348+0000 7f6259d5f700 1 --2- 172.21.15.189:0/861052971 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f623406aae0 0x7f623406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f623c007fd0 tx=0x7f623c007de0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.348+0000 7f6259d5f700 1 -- 172.21.15.189:0/861052971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6254157f90 msgr2=0x7f6254158440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:58.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.348+0000 7f6259d5f700 1 --2- 172.21.15.189:0/861052971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6254157f90 0x7f6254158440 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7f624c007e60 tx=0x7f624c00e3c0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.348+0000 7f6259d5f700 1 -- 172.21.15.189:0/861052971 shutdown_connections 2024-04-06T13:48:58.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.348+0000 7f6259d5f700 1 --2- 172.21.15.189:0/861052971 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f623406aae0 0x7f623406cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.348+0000 7f6259d5f700 1 --2- 172.21.15.189:0/861052971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6254157f90 0x7f6254158440 unknown :-1 s=CLOSED pgs=2831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.348+0000 7f6259d5f700 1 --2- 172.21.15.189:0/861052971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62540b2fc0 0x7f625414a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.348+0000 7f6259d5f700 1 --2- 172.21.15.189:0/861052971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62540b4660 0x7f6254149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.348+0000 7f6259d5f700 1 -- 172.21.15.189:0/861052971 >> 172.21.15.189:0/861052971 conn(0x7f62540d00b0 msgr2=0x7f62540cc2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:58.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.348+0000 7f6259d5f700 1 -- 172.21.15.189:0/861052971 shutdown_connections 2024-04-06T13:48:58.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.348+0000 7f6259d5f700 1 -- 172.21.15.189:0/861052971 wait complete. 2024-04-06T13:48:58.361 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-06T13:48:58.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.483+0000 7f97dd6f4700 1 Processor -- start 2024-04-06T13:48:58.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.483+0000 7f97dd6f4700 1 -- start start 2024-04-06T13:48:58.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.484+0000 7f97dd6f4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f97d815d6f0 0x7f97d815dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:58.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.484+0000 7f97dd6f4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f97d80ce730 0x7f97d80d1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:58.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.484+0000 7f97dd6f4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d815eac0 con 0x7f97d80b8d70 2024-04-06T13:48:58.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.484+0000 7f97dd6f4700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d815e010 con 0x7f97d80ce730 2024-04-06T13:48:58.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.484+0000 7f97dd6f4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d815e220 con 0x7f97d815d6f0 2024-04-06T13:48:58.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.484+0000 7f97d67fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f97d80b8d70 0x7f97d80ce020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48648/0 (socket says 172.21.15.189:48648) 2024-04-06T13:48:58.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.484+0000 7f97d77fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f97d80ce730 0x7f97d80d1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:58.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.484+0000 7f97d67fc700 1 -- 172.21.15.189:0/477554590 learned_addr learned my addr 172.21.15.189:0/477554590 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:58.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.484+0000 7f97d77fe700 1 -- 172.21.15.189:0/477554590 >> v2:172.21.15.192:3300/0 conn(0x7f97d815d6f0 msgr2=0x7f97d815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:58.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.484+0000 7f97d6ffd700 1 --2- 172.21.15.189:0/477554590 >> v2:172.21.15.192:3300/0 conn(0x7f97d815d6f0 0x7f97d815dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:58.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.484+0000 7f97d77fe700 1 --2- 172.21.15.189:0/477554590 >> v2:172.21.15.192:3300/0 conn(0x7f97d815d6f0 0x7f97d815dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.484+0000 7f97d77fe700 1 -- 172.21.15.189:0/477554590 >> v1:172.21.15.189:6789/0 conn(0x7f97d80b8d70 legacy=0x7f97d80ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:58.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.484+0000 7f97d77fe700 1 -- 172.21.15.189:0/477554590 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d815e500 con 0x7f97d80ce730 2024-04-06T13:48:58.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.484+0000 7f97d5ffb700 1 -- 172.21.15.189:0/477554590 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380983560 0 0) 0x7f97d815eac0 con 0x7f97d80b8d70 2024-04-06T13:48:58.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.484+0000 7f97d77fe700 1 --2- 172.21.15.189:0/477554590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d80ce730 0x7f97d80d1e00 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7f97d80d4410 tx=0x7f97d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d895129fe3a2b0e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:58.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.484+0000 7f97d5ffb700 1 -- 172.21.15.189:0/477554590 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d000e070 con 0x7f97d80ce730 2024-04-06T13:48:58.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.484+0000 7f97d5ffb700 1 -- 172.21.15.189:0/477554590 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d000bbd0 con 0x7f97d80ce730 2024-04-06T13:48:58.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.484+0000 7f97d5ffb700 1 -- 172.21.15.189:0/477554590 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0012430 con 0x7f97d80ce730 2024-04-06T13:48:58.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.485+0000 7f97dd6f4700 1 -- 172.21.15.189:0/477554590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d80ce730 msgr2=0x7f97d80d1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:58.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.485+0000 7f97dd6f4700 1 --2- 172.21.15.189:0/477554590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d80ce730 0x7f97d80d1e00 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7f97d80d4410 tx=0x7f97d000b330 comp rx=0 tx=0).stop 2024-04-06T13:48:58.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.485+0000 7f97dd6f4700 1 -- 172.21.15.189:0/477554590 shutdown_connections 2024-04-06T13:48:58.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.485+0000 7f97dd6f4700 1 --2- 172.21.15.189:0/477554590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d80ce730 0x7f97d80d1e00 unknown :-1 s=CLOSED pgs=2833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.485+0000 7f97dd6f4700 1 --2- 172.21.15.189:0/477554590 >> v2:172.21.15.192:3300/0 conn(0x7f97d815d6f0 0x7f97d815dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.485+0000 7f97dd6f4700 1 -- 172.21.15.189:0/477554590 >> 172.21.15.189:0/477554590 conn(0x7f97d8072490 msgr2=0x7f97d806b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:58.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.485+0000 7f97dd6f4700 1 -- 172.21.15.189:0/477554590 shutdown_connections 2024-04-06T13:48:58.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.485+0000 7f97dd6f4700 1 -- 172.21.15.189:0/477554590 wait complete. 2024-04-06T13:48:58.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.486+0000 7f97dd6f4700 1 Processor -- start 2024-04-06T13:48:58.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.486+0000 7f97dd6f4700 1 -- start start 2024-04-06T13:48:58.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.487+0000 7f97dd6f4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97d80b8d70 0x7f97d80cd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:58.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.487+0000 7f97dd6f4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97d80ce730 0x7f97d81f2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:58.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.487+0000 7f97dd6f4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d815d6f0 0x7f97d81f2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:58.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.487+0000 7f97dd6f4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d81f2ad0 con 0x7f97d815d6f0 2024-04-06T13:48:58.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.487+0000 7f97dd6f4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d8152210 con 0x7f97d80ce730 2024-04-06T13:48:58.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.487+0000 7f97dd6f4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d80b5180 con 0x7f97d80b8d70 2024-04-06T13:48:58.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.487+0000 7f97d77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d815d6f0 0x7f97d81f2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:58.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.487+0000 7f97d77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d815d6f0 0x7f97d81f2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39862/0 (socket says 172.21.15.189:39862) 2024-04-06T13:48:58.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.487+0000 7f97d77fe700 1 -- 172.21.15.189:0/615498393 learned_addr learned my addr 172.21.15.189:0/615498393 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:58.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.487+0000 7f97d6ffd700 1 --2- 172.21.15.189:0/615498393 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97d80b8d70 0x7f97d80cd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:58.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.487+0000 7f97d67fc700 1 --2- 172.21.15.189:0/615498393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97d80ce730 0x7f97d81f2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:58.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.487+0000 7f97d77fe700 1 -- 172.21.15.189:0/615498393 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97d80b8d70 msgr2=0x7f97d80cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:58.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.488+0000 7f97d77fe700 1 --2- 172.21.15.189:0/615498393 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97d80b8d70 0x7f97d80cd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.488+0000 7f97d77fe700 1 -- 172.21.15.189:0/615498393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97d80ce730 msgr2=0x7f97d81f2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:58.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.488+0000 7f97d77fe700 1 --2- 172.21.15.189:0/615498393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97d80ce730 0x7f97d81f2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.488+0000 7f97d77fe700 1 -- 172.21.15.189:0/615498393 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d0009a70 con 0x7f97d815d6f0 2024-04-06T13:48:58.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.488+0000 7f97d6ffd700 1 --2- 172.21.15.189:0/615498393 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97d80b8d70 0x7f97d80cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:58.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.488+0000 7f97d67fc700 1 --2- 172.21.15.189:0/615498393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97d80ce730 0x7f97d81f2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:48:58.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.488+0000 7f97d77fe700 1 --2- 172.21.15.189:0/615498393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d815d6f0 0x7f97d81f2590 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7f97d80b5fd0 tx=0x7f97d0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:58.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.488+0000 7f97bf7fe700 1 -- 172.21.15.189:0/615498393 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0012430 con 0x7f97d815d6f0 2024-04-06T13:48:58.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.488+0000 7f97bf7fe700 1 -- 172.21.15.189:0/615498393 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d0020070 con 0x7f97d815d6f0 2024-04-06T13:48:58.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.488+0000 7f97dd6f4700 1 -- 172.21.15.189:0/615498393 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d80b5460 con 0x7f97d815d6f0 2024-04-06T13:48:58.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.488+0000 7f97bf7fe700 1 -- 172.21.15.189:0/615498393 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0004280 con 0x7f97d815d6f0 2024-04-06T13:48:58.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.488+0000 7f97dd6f4700 1 -- 172.21.15.189:0/615498393 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d814f690 con 0x7f97d815d6f0 2024-04-06T13:48:58.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.489+0000 7f97dd6f4700 1 -- 172.21.15.189:0/615498393 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d8074990 con 0x7f97d815d6f0 2024-04-06T13:48:58.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.489+0000 7f97bf7fe700 1 -- 172.21.15.189:0/615498393 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 158) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f97d00043e0 con 0x7f97d815d6f0 2024-04-06T13:48:58.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.490+0000 7f97bf7fe700 1 --2- 172.21.15.189:0/615498393 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f97c406aae0 0x7f97c406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:58.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.490+0000 7f97bf7fe700 1 -- 172.21.15.189:0/615498393 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7739+0+0 (secure 0 0 0) 0x7f97d0022070 con 0x7f97d815d6f0 2024-04-06T13:48:58.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.490+0000 7f97d6ffd700 1 --2- 172.21.15.189:0/615498393 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f97c406aae0 0x7f97c406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:58.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.490+0000 7f97d6ffd700 1 --2- 172.21.15.189:0/615498393 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f97c406aae0 0x7f97c406cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f97c8006fd0 tx=0x7f97c8008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:58.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.494+0000 7f97bf7fe700 1 -- 172.21.15.189:0/615498393 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f97d00563c0 con 0x7f97d815d6f0 2024-04-06T13:48:58.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.734+0000 7f97dd6f4700 1 -- 172.21.15.189:0/615498393 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f97d80d1e00 con 0x7f97d815d6f0 2024-04-06T13:48:58.736 INFO:tasks.ceph.mon.a.smithi189.stderr:2024-04-06T13:48:58.735+0000 7fa9fe1ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T13:48:58.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.744+0000 7f97bf7fe700 1 -- 172.21.15.189:0/615498393 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v123) v1 ==== 161+0+0 (secure 0 0 0) 0x7f97d00599e0 con 0x7f97d815d6f0 2024-04-06T13:48:58.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.747+0000 7f97dd6f4700 1 -- 172.21.15.189:0/615498393 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f97c406aae0 msgr2=0x7f97c406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:58.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.747+0000 7f97dd6f4700 1 --2- 172.21.15.189:0/615498393 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f97c406aae0 0x7f97c406cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f97c8006fd0 tx=0x7f97c8008040 comp rx=0 tx=0).stop 2024-04-06T13:48:58.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.747+0000 7f97dd6f4700 1 -- 172.21.15.189:0/615498393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d815d6f0 msgr2=0x7f97d81f2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:58.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.747+0000 7f97dd6f4700 1 --2- 172.21.15.189:0/615498393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d815d6f0 0x7f97d81f2590 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7f97d80b5fd0 tx=0x7f97d0004ad0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.747+0000 7f97dd6f4700 1 -- 172.21.15.189:0/615498393 shutdown_connections 2024-04-06T13:48:58.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.747+0000 7f97dd6f4700 1 --2- 172.21.15.189:0/615498393 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f97c406aae0 0x7f97c406cfa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.747+0000 7f97dd6f4700 1 --2- 172.21.15.189:0/615498393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d815d6f0 0x7f97d81f2590 unknown :-1 s=CLOSED pgs=2834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.747+0000 7f97dd6f4700 1 --2- 172.21.15.189:0/615498393 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97d80b8d70 0x7f97d80cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.747+0000 7f97dd6f4700 1 --2- 172.21.15.189:0/615498393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97d80ce730 0x7f97d81f2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.747+0000 7f97dd6f4700 1 -- 172.21.15.189:0/615498393 >> 172.21.15.189:0/615498393 conn(0x7f97d8072490 msgr2=0x7f97d80b6c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:58.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.747+0000 7f97dd6f4700 1 -- 172.21.15.189:0/615498393 shutdown_connections 2024-04-06T13:48:58.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.747+0000 7f97dd6f4700 1 -- 172.21.15.189:0/615498393 wait complete. 2024-04-06T13:48:58.749 INFO:teuthology.orchestra.run.smithi189.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-06T13:48:58.750 INFO:tasks.ceph.mds.d.smithi192.stderr: -5351> 2024-04-06T13:48:25.251+0000 7f36f3503700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T13:48:58.761 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-06T13:48:58.782 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2024-04-06T13:48:58.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.891+0000 7f73d2ed9700 1 Processor -- start 2024-04-06T13:48:58.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.891+0000 7f73d2ed9700 1 -- start start 2024-04-06T13:48:58.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.892+0000 7f73d2ed9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73cc15d6f0 0x7f73cc15dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:58.894 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.892+0000 7f73d2ed9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f73cc0ce730 0x7f73cc0d1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:58.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.892+0000 7f73d2ed9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73cc15eac0 con 0x7f73cc0b8d70 2024-04-06T13:48:58.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.892+0000 7f73d2ed9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73cc15e010 con 0x7f73cc0ce730 2024-04-06T13:48:58.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.892+0000 7f73d2ed9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc15e220 con 0x7f73cc15d6f0 2024-04-06T13:48:58.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.892+0000 7f73d1476700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f73cc0ce730 0x7f73cc0d1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:58.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.892+0000 7f73d1476700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f73cc0ce730 0x7f73cc0d1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39886/0 (socket says 172.21.15.189:39886) 2024-04-06T13:48:58.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.892+0000 7f73d1476700 1 -- 172.21.15.189:0/4118063207 learned_addr learned my addr 172.21.15.189:0/4118063207 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:58.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.892+0000 7f73d0c75700 1 --2- 172.21.15.189:0/4118063207 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73cc15d6f0 0x7f73cc15dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:58.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.892+0000 7f73d1476700 1 -- 172.21.15.189:0/4118063207 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73cc15d6f0 msgr2=0x7f73cc15dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:58.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.892+0000 7f73d1476700 1 --2- 172.21.15.189:0/4118063207 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73cc15d6f0 0x7f73cc15dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.892+0000 7f73d1476700 1 -- 172.21.15.189:0/4118063207 >> v1:172.21.15.192:6789/0 conn(0x7f73cc0b8d70 legacy=0x7f73cc0ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:58.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.892+0000 7f73d1476700 1 -- 172.21.15.189:0/4118063207 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73cc15e500 con 0x7f73cc0ce730 2024-04-06T13:48:58.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.892+0000 7f73d1476700 1 --2- 172.21.15.189:0/4118063207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73cc0ce730 0x7f73cc0d1e00 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f73c4009e50 tx=0x7f73c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d0a119b2c10002 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:58.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.892+0000 7f73cb7fe700 1 -- 172.21.15.189:0/4118063207 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c400e070 con 0x7f73cc0ce730 2024-04-06T13:48:58.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.892+0000 7f73cb7fe700 1 -- 172.21.15.189:0/4118063207 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c4003a20 con 0x7f73cc0ce730 2024-04-06T13:48:58.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.892+0000 7f73cb7fe700 1 -- 172.21.15.189:0/4118063207 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c4003dd0 con 0x7f73cc0ce730 2024-04-06T13:48:58.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.893+0000 7f73d2ed9700 1 -- 172.21.15.189:0/4118063207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73cc0ce730 msgr2=0x7f73cc0d1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:58.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.893+0000 7f73d2ed9700 1 --2- 172.21.15.189:0/4118063207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73cc0ce730 0x7f73cc0d1e00 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f73c4009e50 tx=0x7f73c400b330 comp rx=0 tx=0).stop 2024-04-06T13:48:58.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.893+0000 7f73d2ed9700 1 -- 172.21.15.189:0/4118063207 shutdown_connections 2024-04-06T13:48:58.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.893+0000 7f73d2ed9700 1 --2- 172.21.15.189:0/4118063207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73cc0ce730 0x7f73cc0d1e00 unknown :-1 s=CLOSED pgs=2836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.893+0000 7f73d2ed9700 1 --2- 172.21.15.189:0/4118063207 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73cc15d6f0 0x7f73cc15dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.893+0000 7f73d2ed9700 1 -- 172.21.15.189:0/4118063207 >> 172.21.15.189:0/4118063207 conn(0x7f73cc072490 msgr2=0x7f73cc06b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:58.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.893+0000 7f73d2ed9700 1 -- 172.21.15.189:0/4118063207 shutdown_connections 2024-04-06T13:48:58.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.893+0000 7f73d2ed9700 1 -- 172.21.15.189:0/4118063207 wait complete. 2024-04-06T13:48:58.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.894+0000 7f73d2ed9700 1 Processor -- start 2024-04-06T13:48:58.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.894+0000 7f73d2ed9700 1 -- start start 2024-04-06T13:48:58.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.895+0000 7f73d2ed9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73cc0b8d70 0x7f73cc158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:58.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.895+0000 7f73d2ed9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73cc0ce730 0x7f73cc14d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:58.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.895+0000 7f73d2ed9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73cc15d6f0 0x7f73cc0b7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:58.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.895+0000 7f73d2ed9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc1570f0 con 0x7f73cc15d6f0 2024-04-06T13:48:58.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.895+0000 7f73d2ed9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc14dc30 con 0x7f73cc0b8d70 2024-04-06T13:48:58.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.895+0000 7f73d2ed9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc157340 con 0x7f73cc0ce730 2024-04-06T13:48:58.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.895+0000 7f73d1476700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73cc15d6f0 0x7f73cc0b7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:58.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.895+0000 7f73d1476700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73cc15d6f0 0x7f73cc0b7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39892/0 (socket says 172.21.15.189:39892) 2024-04-06T13:48:58.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.895+0000 7f73d1476700 1 -- 172.21.15.189:0/22907171 learned_addr learned my addr 172.21.15.189:0/22907171 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:58.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.895+0000 7f73cbfff700 1 --2- 172.21.15.189:0/22907171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73cc0ce730 0x7f73cc14d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:58.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.895+0000 7f73d0c75700 1 --2- 172.21.15.189:0/22907171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73cc0b8d70 0x7f73cc158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:58.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.896+0000 7f73d1476700 1 -- 172.21.15.189:0/22907171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73cc0ce730 msgr2=0x7f73cc14d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:58.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.896+0000 7f73d1476700 1 --2- 172.21.15.189:0/22907171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73cc0ce730 0x7f73cc14d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.896+0000 7f73d1476700 1 -- 172.21.15.189:0/22907171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73cc0b8d70 msgr2=0x7f73cc158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:58.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.896+0000 7f73d1476700 1 --2- 172.21.15.189:0/22907171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73cc0b8d70 0x7f73cc158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:58.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.896+0000 7f73d1476700 1 -- 172.21.15.189:0/22907171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c4009b30 con 0x7f73cc15d6f0 2024-04-06T13:48:58.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.896+0000 7f73cbfff700 1 --2- 172.21.15.189:0/22907171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73cc0ce730 0x7f73cc14d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:48:58.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.896+0000 7f73d1476700 1 --2- 172.21.15.189:0/22907171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73cc15d6f0 0x7f73cc0b7de0 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f73c4004000 tx=0x7f73c4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:58.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.896+0000 7f73c97fa700 1 -- 172.21.15.189:0/22907171 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c4004440 con 0x7f73cc15d6f0 2024-04-06T13:48:58.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.896+0000 7f73c97fa700 1 -- 172.21.15.189:0/22907171 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c40045a0 con 0x7f73cc15d6f0 2024-04-06T13:48:58.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.896+0000 7f73d2ed9700 1 -- 172.21.15.189:0/22907171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73cc157620 con 0x7f73cc15d6f0 2024-04-06T13:48:58.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.896+0000 7f73c97fa700 1 -- 172.21.15.189:0/22907171 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c4004950 con 0x7f73cc15d6f0 2024-04-06T13:48:58.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.896+0000 7f73d2ed9700 1 -- 172.21.15.189:0/22907171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73cc157be0 con 0x7f73cc15d6f0 2024-04-06T13:48:58.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.897+0000 7f73d2ed9700 1 -- 172.21.15.189:0/22907171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73cc0670f0 con 0x7f73cc15d6f0 2024-04-06T13:48:58.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.897+0000 7f73c97fa700 1 -- 172.21.15.189:0/22907171 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 158) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f73c4012430 con 0x7f73cc15d6f0 2024-04-06T13:48:58.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.898+0000 7f73c97fa700 1 --2- 172.21.15.189:0/22907171 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f73b006ab10 0x7f73b006cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:58.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.898+0000 7f73c97fa700 1 -- 172.21.15.189:0/22907171 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 7825+0+0 (secure 0 0 0) 0x7f73c408adb0 con 0x7f73cc15d6f0 2024-04-06T13:48:58.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.898+0000 7f73d0c75700 1 --2- 172.21.15.189:0/22907171 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f73b006ab10 0x7f73b006cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:58.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.898+0000 7f73d0c75700 1 --2- 172.21.15.189:0/22907171 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f73b006ab10 0x7f73b006cfd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f73c000a950 tx=0x7f73c0008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:58.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:58.902+0000 7f73c97fa700 1 -- 172.21.15.189:0/22907171 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f73c4055a00 con 0x7f73cc15d6f0 2024-04-06T13:48:59.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.137+0000 7f73d2ed9700 1 -- 172.21.15.189:0/22907171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f73cc074990 con 0x7f73cc15d6f0 2024-04-06T13:48:59.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.763+0000 7f73c97fa700 1 -- 172.21.15.189:0/22907171 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 159) v1 ==== 89293+0+0 (secure 0 0 0) 0x7f73c4055dd0 con 0x7f73cc15d6f0 2024-04-06T13:48:59.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.764+0000 7f73c97fa700 1 -- 172.21.15.189:0/22907171 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v124) v1 ==== 104+0+0 (secure 0 0 0) 0x7f73c4059490 con 0x7f73cc15d6f0 2024-04-06T13:48:59.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.766+0000 7f73d2ed9700 1 -- 172.21.15.189:0/22907171 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f73b006ab10 msgr2=0x7f73b006cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:59.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.766+0000 7f73d2ed9700 1 --2- 172.21.15.189:0/22907171 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f73b006ab10 0x7f73b006cfd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f73c000a950 tx=0x7f73c0008040 comp rx=0 tx=0).stop 2024-04-06T13:48:59.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.766+0000 7f73d2ed9700 1 -- 172.21.15.189:0/22907171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73cc15d6f0 msgr2=0x7f73cc0b7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:59.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.766+0000 7f73d2ed9700 1 --2- 172.21.15.189:0/22907171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73cc15d6f0 0x7f73cc0b7de0 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f73c4004000 tx=0x7f73c4004ad0 comp rx=0 tx=0).stop 2024-04-06T13:48:59.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.767+0000 7f73d2ed9700 1 -- 172.21.15.189:0/22907171 shutdown_connections 2024-04-06T13:48:59.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.767+0000 7f73d2ed9700 1 --2- 172.21.15.189:0/22907171 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f73b006ab10 0x7f73b006cfd0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:59.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.767+0000 7f73d2ed9700 1 --2- 172.21.15.189:0/22907171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73cc15d6f0 0x7f73cc0b7de0 unknown :-1 s=CLOSED pgs=2837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:59.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.767+0000 7f73d2ed9700 1 --2- 172.21.15.189:0/22907171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73cc0b8d70 0x7f73cc158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:59.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.767+0000 7f73d2ed9700 1 --2- 172.21.15.189:0/22907171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73cc0ce730 0x7f73cc14d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:59.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.767+0000 7f73d2ed9700 1 -- 172.21.15.189:0/22907171 >> 172.21.15.189:0/22907171 conn(0x7f73cc072490 msgr2=0x7f73cc0b7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:59.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.767+0000 7f73d2ed9700 1 -- 172.21.15.189:0/22907171 shutdown_connections 2024-04-06T13:48:59.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.767+0000 7f73d2ed9700 1 -- 172.21.15.189:0/22907171 wait complete. 2024-04-06T13:48:59.781 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-06T13:48:59.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.911+0000 7fa8d1363700 1 Processor -- start 2024-04-06T13:48:59.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.911+0000 7fa8d1363700 1 -- start start 2024-04-06T13:48:59.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.912+0000 7fa8d1363700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8cc15d6f0 0x7fa8cc15dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.912+0000 7fa8d1363700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa8cc0ce730 0x7fa8cc0d1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.912+0000 7fa8d1363700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8cc15eac0 con 0x7fa8cc0b8d70 2024-04-06T13:48:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.912+0000 7fa8d1363700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8cc15e010 con 0x7fa8cc0ce730 2024-04-06T13:48:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.912+0000 7fa8d1363700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc15e220 con 0x7fa8cc15d6f0 2024-04-06T13:48:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.912+0000 7fa8ca7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa8cc0b8d70 0x7fa8cc0ce020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50426/0 (socket says 172.21.15.189:50426) 2024-04-06T13:48:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.912+0000 7fa8ca7fc700 1 -- 172.21.15.189:0/1894684981 learned_addr learned my addr 172.21.15.189:0/1894684981 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:48:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.912+0000 7fa8cb7fe700 1 --2- 172.21.15.189:0/1894684981 >> v2:172.21.15.192:3300/0 conn(0x7fa8cc0ce730 0x7fa8cc0d1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.912+0000 7fa8caffd700 1 --2- 172.21.15.189:0/1894684981 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8cc15d6f0 0x7fa8cc15dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.912+0000 7fa8c9ffb700 1 -- 172.21.15.189:0/1894684981 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376030593 0 0) 0x7fa8cc15eac0 con 0x7fa8cc0b8d70 2024-04-06T13:48:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.912+0000 7fa8c9ffb700 1 -- 172.21.15.189:0/1894684981 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b40035b0 con 0x7fa8cc0b8d70 2024-04-06T13:48:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.912+0000 7fa8cb7fe700 1 -- 172.21.15.189:0/1894684981 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8cc15d6f0 msgr2=0x7fa8cc15dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:48:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.912+0000 7fa8cb7fe700 1 --2- 172.21.15.189:0/1894684981 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8cc15d6f0 0x7fa8cc15dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.912+0000 7fa8cb7fe700 1 -- 172.21.15.189:0/1894684981 >> v1:172.21.15.189:6789/0 conn(0x7fa8cc0b8d70 legacy=0x7fa8cc0ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:59.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.912+0000 7fa8c9ffb700 1 -- 172.21.15.189:0/1894684981 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 215791187 0 0) 0x7fa8b40035b0 con 0x7fa8cc0b8d70 2024-04-06T13:48:59.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.912+0000 7fa8cb7fe700 1 -- 172.21.15.189:0/1894684981 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8cc15e500 con 0x7fa8cc0ce730 2024-04-06T13:48:59.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.912+0000 7fa8caffd700 1 --2- 172.21.15.189:0/1894684981 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8cc15d6f0 0x7fa8cc15dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:48:59.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.913+0000 7fa8cb7fe700 1 --2- 172.21.15.189:0/1894684981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8cc0ce730 0x7fa8cc0d1e00 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4009d90 tx=0x7fa8c400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93ce0b6529665d59 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:59.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.913+0000 7fa8c9ffb700 1 -- 172.21.15.189:0/1894684981 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c400e070 con 0x7fa8cc0ce730 2024-04-06T13:48:59.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.913+0000 7fa8c9ffb700 1 -- 172.21.15.189:0/1894684981 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c400bb20 con 0x7fa8cc0ce730 2024-04-06T13:48:59.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.913+0000 7fa8c9ffb700 1 -- 172.21.15.189:0/1894684981 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c4012430 con 0x7fa8cc0ce730 2024-04-06T13:48:59.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.913+0000 7fa8d1363700 1 -- 172.21.15.189:0/1894684981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8cc0ce730 msgr2=0x7fa8cc0d1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:59.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.913+0000 7fa8d1363700 1 --2- 172.21.15.189:0/1894684981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8cc0ce730 0x7fa8cc0d1e00 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4009d90 tx=0x7fa8c400b330 comp rx=0 tx=0).stop 2024-04-06T13:48:59.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.913+0000 7fa8d1363700 1 -- 172.21.15.189:0/1894684981 shutdown_connections 2024-04-06T13:48:59.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.913+0000 7fa8d1363700 1 --2- 172.21.15.189:0/1894684981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8cc0ce730 0x7fa8cc0d1e00 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:59.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.913+0000 7fa8d1363700 1 --2- 172.21.15.189:0/1894684981 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8cc15d6f0 0x7fa8cc15dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:59.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.913+0000 7fa8d1363700 1 -- 172.21.15.189:0/1894684981 >> 172.21.15.189:0/1894684981 conn(0x7fa8cc072490 msgr2=0x7fa8cc06b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:48:59.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.913+0000 7fa8d1363700 1 -- 172.21.15.189:0/1894684981 shutdown_connections 2024-04-06T13:48:59.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.913+0000 7fa8d1363700 1 -- 172.21.15.189:0/1894684981 wait complete. 2024-04-06T13:48:59.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.915+0000 7fa8d1363700 1 Processor -- start 2024-04-06T13:48:59.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.915+0000 7fa8d1363700 1 -- start start 2024-04-06T13:48:59.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.915+0000 7fa8d1363700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8cc0b8d70 0x7fa8cc1f3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:59.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.915+0000 7fa8d1363700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8cc0ce730 0x7fa8cc0cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:59.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.915+0000 7fa8d1363700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8cc15d6f0 0x7fa8cc0b7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:59.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.915+0000 7fa8d1363700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc1f31e0 con 0x7fa8cc0ce730 2024-04-06T13:48:59.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.915+0000 7fa8d1363700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc1f28d0 con 0x7fa8cc15d6f0 2024-04-06T13:48:59.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.915+0000 7fa8d1363700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc0c97d0 con 0x7fa8cc0b8d70 2024-04-06T13:48:59.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.916+0000 7fa8ca7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8cc0ce730 0x7fa8cc0cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:59.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.916+0000 7fa8ca7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8cc0ce730 0x7fa8cc0cb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37186/0 (socket says 172.21.15.189:37186) 2024-04-06T13:48:59.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.916+0000 7fa8ca7fc700 1 -- 172.21.15.189:0/2705355757 learned_addr learned my addr 172.21.15.189:0/2705355757 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:48:59.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.916+0000 7fa8caffd700 1 --2- 172.21.15.189:0/2705355757 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8cc0b8d70 0x7fa8cc1f3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:59.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.916+0000 7fa8cb7fe700 1 --2- 172.21.15.189:0/2705355757 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8cc15d6f0 0x7fa8cc0b7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:59.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.916+0000 7fa8ca7fc700 1 -- 172.21.15.189:0/2705355757 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8cc0b8d70 msgr2=0x7fa8cc1f3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:59.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.916+0000 7fa8ca7fc700 1 --2- 172.21.15.189:0/2705355757 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8cc0b8d70 0x7fa8cc1f3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:59.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.916+0000 7fa8ca7fc700 1 -- 172.21.15.189:0/2705355757 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8cc15d6f0 msgr2=0x7fa8cc0b7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:48:59.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.916+0000 7fa8ca7fc700 1 --2- 172.21.15.189:0/2705355757 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8cc15d6f0 0x7fa8cc0b7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:48:59.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.916+0000 7fa8ca7fc700 1 -- 172.21.15.189:0/2705355757 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c00094a0 con 0x7fa8cc0ce730 2024-04-06T13:48:59.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.916+0000 7fa8ca7fc700 1 --2- 172.21.15.189:0/2705355757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8cc0ce730 0x7fa8cc0cb2f0 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00c780 tx=0x7fa8bc00cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:59.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.916+0000 7fa8b37fe700 1 -- 172.21.15.189:0/2705355757 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc005d70 con 0x7fa8cc0ce730 2024-04-06T13:48:59.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.916+0000 7fa8b37fe700 1 -- 172.21.15.189:0/2705355757 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8bc0070d0 con 0x7fa8cc0ce730 2024-04-06T13:48:59.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.916+0000 7fa8d1363700 1 -- 172.21.15.189:0/2705355757 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c4009a70 con 0x7fa8cc0ce730 2024-04-06T13:48:59.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.916+0000 7fa8b37fe700 1 -- 172.21.15.189:0/2705355757 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc007460 con 0x7fa8cc0ce730 2024-04-06T13:48:59.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.916+0000 7fa8d1363700 1 -- 172.21.15.189:0/2705355757 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8cc1f2b20 con 0x7fa8cc0ce730 2024-04-06T13:48:59.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.917+0000 7fa8d1363700 1 -- 172.21.15.189:0/2705355757 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8cc074990 con 0x7fa8cc0ce730 2024-04-06T13:48:59.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.918+0000 7fa8b37fe700 1 -- 172.21.15.189:0/2705355757 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 159) v1 ==== 89293+0+0 (secure 0 0 0) 0x7fa8bc0075c0 con 0x7fa8cc0ce730 2024-04-06T13:48:59.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.918+0000 7fa8b37fe700 1 --2- 172.21.15.189:0/2705355757 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa8b806ae40 0x7fa8b806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:48:59.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.918+0000 7fa8b37fe700 1 -- 172.21.15.189:0/2705355757 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 7825+0+0 (secure 0 0 0) 0x7fa8bc089580 con 0x7fa8cc0ce730 2024-04-06T13:48:59.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.918+0000 7fa8caffd700 1 --2- 172.21.15.189:0/2705355757 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa8b806ae40 0x7fa8b806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:48:59.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.919+0000 7fa8caffd700 1 --2- 172.21.15.189:0/2705355757 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa8b806ae40 0x7fa8b806d300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009c60 tx=0x7fa8c0008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:48:59.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:48:59.922+0000 7fa8b37fe700 1 -- 172.21.15.189:0/2705355757 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa8bc053c00 con 0x7fa8cc0ce730 2024-04-06T13:49:00.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:00.116+0000 7fa8d1363700 1 -- 172.21.15.189:0/2705355757 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa8cc0b6d10 con 0x7fa8cc0ce730 2024-04-06T13:49:00.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:00.857+0000 7fa8b37fe700 1 -- 172.21.15.189:0/2705355757 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fa8bc053fd0 con 0x7fa8cc0ce730 2024-04-06T13:49:00.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:00.859+0000 7fa8b37fe700 1 -- 172.21.15.189:0/2705355757 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v256) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa8bc0723e0 con 0x7fa8cc0ce730 2024-04-06T13:49:00.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:00.863+0000 7fa8d1363700 1 -- 172.21.15.189:0/2705355757 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa8b806ae40 msgr2=0x7fa8b806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:00.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:00.863+0000 7fa8d1363700 1 --2- 172.21.15.189:0/2705355757 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa8b806ae40 0x7fa8b806d300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009c60 tx=0x7fa8c0008040 comp rx=0 tx=0).stop 2024-04-06T13:49:00.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:00.863+0000 7fa8d1363700 1 -- 172.21.15.189:0/2705355757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8cc0ce730 msgr2=0x7fa8cc0cb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:00.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:00.863+0000 7fa8d1363700 1 --2- 172.21.15.189:0/2705355757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8cc0ce730 0x7fa8cc0cb2f0 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00c780 tx=0x7fa8bc00cb40 comp rx=0 tx=0).stop 2024-04-06T13:49:00.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:00.863+0000 7fa8d1363700 1 -- 172.21.15.189:0/2705355757 shutdown_connections 2024-04-06T13:49:00.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:00.863+0000 7fa8d1363700 1 --2- 172.21.15.189:0/2705355757 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa8b806ae40 0x7fa8b806d300 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:00.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:00.863+0000 7fa8d1363700 1 --2- 172.21.15.189:0/2705355757 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8cc15d6f0 0x7fa8cc0b7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:00.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:00.863+0000 7fa8d1363700 1 --2- 172.21.15.189:0/2705355757 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8cc0b8d70 0x7fa8cc1f3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:00.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:00.863+0000 7fa8d1363700 1 --2- 172.21.15.189:0/2705355757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8cc0ce730 0x7fa8cc0cb2f0 unknown :-1 s=CLOSED pgs=2839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:00.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:00.863+0000 7fa8d1363700 1 -- 172.21.15.189:0/2705355757 >> 172.21.15.189:0/2705355757 conn(0x7fa8cc072490 msgr2=0x7fa8cc1602d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:00.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:00.863+0000 7fa8d1363700 1 -- 172.21.15.189:0/2705355757 shutdown_connections 2024-04-06T13:49:00.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:00.863+0000 7fa8d1363700 1 -- 172.21.15.189:0/2705355757 wait complete. 2024-04-06T13:49:00.865 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' removed 2024-04-06T13:49:00.879 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-06T13:49:01.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.007+0000 7f47d2a0c700 1 Processor -- start 2024-04-06T13:49:01.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.007+0000 7f47d2a0c700 1 -- start start 2024-04-06T13:49:01.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.008+0000 7f47d2a0c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f47cc0d11b0 0x7f47cc0d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:01.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.008+0000 7f47d2a0c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f47cc0cfa00 0x7f47cc0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:01.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.008+0000 7f47d2a0c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47cc0d1ad0 con 0x7f47cc0cb870 2024-04-06T13:49:01.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.008+0000 7f47d2a0c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47cc0ccdc0 con 0x7f47cc0cfa00 2024-04-06T13:49:01.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.008+0000 7f47d2a0c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47cc0d0b50 con 0x7f47cc0d11b0 2024-04-06T13:49:01.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.008+0000 7f47d0fa9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f47cc0cfa00 0x7f47cc0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:01.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.008+0000 7f47d0fa9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f47cc0cfa00 0x7f47cc0cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37188/0 (socket says 172.21.15.189:37188) 2024-04-06T13:49:01.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.008+0000 7f47d0fa9700 1 -- 172.21.15.189:0/157199946 learned_addr learned my addr 172.21.15.189:0/157199946 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:01.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.008+0000 7f47cbfff700 1 --2- 172.21.15.189:0/157199946 >> v2:172.21.15.192:3300/0 conn(0x7f47cc0d11b0 0x7f47cc0d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:01.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.008+0000 7f47d0fa9700 1 -- 172.21.15.189:0/157199946 >> v2:172.21.15.192:3300/0 conn(0x7f47cc0d11b0 msgr2=0x7f47cc0d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:01.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.008+0000 7f47d0fa9700 1 --2- 172.21.15.189:0/157199946 >> v2:172.21.15.192:3300/0 conn(0x7f47cc0d11b0 0x7f47cc0d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:01.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.008+0000 7f47d0fa9700 1 -- 172.21.15.189:0/157199946 >> v1:172.21.15.192:6789/0 conn(0x7f47cc0cb870 legacy=0x7f47cc15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:01.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.008+0000 7f47d0fa9700 1 -- 172.21.15.189:0/157199946 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47cc0cbd80 con 0x7f47cc0cfa00 2024-04-06T13:49:01.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.009+0000 7f47d0fa9700 1 --2- 172.21.15.189:0/157199946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47cc0cfa00 0x7f47cc0cd560 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0ced00 tx=0x7f47c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b6fb7cdf997649 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:01.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.009+0000 7f47caffd700 1 -- 172.21.15.189:0/157199946 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c400e070 con 0x7f47cc0cfa00 2024-04-06T13:49:01.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.009+0000 7f47caffd700 1 -- 172.21.15.189:0/157199946 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c400b9c0 con 0x7f47cc0cfa00 2024-04-06T13:49:01.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.009+0000 7f47caffd700 1 -- 172.21.15.189:0/157199946 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c400bd70 con 0x7f47cc0cfa00 2024-04-06T13:49:01.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.009+0000 7f47d2a0c700 1 -- 172.21.15.189:0/157199946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47cc0cfa00 msgr2=0x7f47cc0cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:01.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.009+0000 7f47d2a0c700 1 --2- 172.21.15.189:0/157199946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47cc0cfa00 0x7f47cc0cd560 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0ced00 tx=0x7f47c400b330 comp rx=0 tx=0).stop 2024-04-06T13:49:01.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.009+0000 7f47d2a0c700 1 -- 172.21.15.189:0/157199946 shutdown_connections 2024-04-06T13:49:01.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.009+0000 7f47d2a0c700 1 --2- 172.21.15.189:0/157199946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47cc0cfa00 0x7f47cc0cd560 unknown :-1 s=CLOSED pgs=2840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:01.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.009+0000 7f47d2a0c700 1 --2- 172.21.15.189:0/157199946 >> v2:172.21.15.192:3300/0 conn(0x7f47cc0d11b0 0x7f47cc0d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:01.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.009+0000 7f47d2a0c700 1 -- 172.21.15.189:0/157199946 >> 172.21.15.189:0/157199946 conn(0x7f47cc0b2db0 msgr2=0x7f47cc0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:01.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.009+0000 7f47d2a0c700 1 -- 172.21.15.189:0/157199946 shutdown_connections 2024-04-06T13:49:01.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.009+0000 7f47d2a0c700 1 -- 172.21.15.189:0/157199946 wait complete. 2024-04-06T13:49:01.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.011+0000 7f47d2a0c700 1 Processor -- start 2024-04-06T13:49:01.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.011+0000 7f47d2a0c700 1 -- start start 2024-04-06T13:49:01.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.011+0000 7f47d2a0c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47cc0cb870 0x7f47cc155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:01.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.011+0000 7f47d2a0c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47cc0cfa00 0x7f47cc15ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:01.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.011+0000 7f47d2a0c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47cc0d11b0 0x7f47cc0b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:01.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.011+0000 7f47d2a0c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc156480 con 0x7f47cc0cfa00 2024-04-06T13:49:01.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.011+0000 7f47d2a0c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc156760 con 0x7f47cc0cb870 2024-04-06T13:49:01.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.011+0000 7f47d2a0c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc156a40 con 0x7f47cc0d11b0 2024-04-06T13:49:01.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.012+0000 7f47cb7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47cc0cfa00 0x7f47cc15ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:01.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.012+0000 7f47cb7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47cc0cfa00 0x7f47cc15ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37198/0 (socket says 172.21.15.189:37198) 2024-04-06T13:49:01.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.012+0000 7f47cb7fe700 1 -- 172.21.15.189:0/575305108 learned_addr learned my addr 172.21.15.189:0/575305108 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:01.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.012+0000 7f47cbfff700 1 --2- 172.21.15.189:0/575305108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47cc0cb870 0x7f47cc155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:01.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.012+0000 7f47d0fa9700 1 --2- 172.21.15.189:0/575305108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47cc0d11b0 0x7f47cc0b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:01.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.012+0000 7f47cb7fe700 1 -- 172.21.15.189:0/575305108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47cc0d11b0 msgr2=0x7f47cc0b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:01.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.012+0000 7f47cb7fe700 1 --2- 172.21.15.189:0/575305108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47cc0d11b0 0x7f47cc0b43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:01.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.012+0000 7f47cb7fe700 1 -- 172.21.15.189:0/575305108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47cc0cb870 msgr2=0x7f47cc155080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:01.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.012+0000 7f47cb7fe700 1 --2- 172.21.15.189:0/575305108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47cc0cb870 0x7f47cc155080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:01.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.012+0000 7f47cb7fe700 1 -- 172.21.15.189:0/575305108 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c4009b30 con 0x7f47cc0cfa00 2024-04-06T13:49:01.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.012+0000 7f47cb7fe700 1 --2- 172.21.15.189:0/575305108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47cc0cfa00 0x7f47cc15ea70 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7f47bc00c420 tx=0x7f47bc00c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:01.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.012+0000 7f47c8ff9700 1 -- 172.21.15.189:0/575305108 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc0052d0 con 0x7f47cc0cfa00 2024-04-06T13:49:01.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.012+0000 7f47c8ff9700 1 -- 172.21.15.189:0/575305108 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47bc005430 con 0x7f47cc0cfa00 2024-04-06T13:49:01.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.012+0000 7f47d2a0c700 1 -- 172.21.15.189:0/575305108 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47cc156d20 con 0x7f47cc0cfa00 2024-04-06T13:49:01.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.012+0000 7f47c8ff9700 1 -- 172.21.15.189:0/575305108 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc0038c0 con 0x7f47cc0cfa00 2024-04-06T13:49:01.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.012+0000 7f47d2a0c700 1 -- 172.21.15.189:0/575305108 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47cc1555c0 con 0x7f47cc0cfa00 2024-04-06T13:49:01.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.013+0000 7f47d2a0c700 1 -- 172.21.15.189:0/575305108 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47cc1568a0 con 0x7f47cc0cfa00 2024-04-06T13:49:01.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.014+0000 7f47c8ff9700 1 -- 172.21.15.189:0/575305108 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f47bc003a20 con 0x7f47cc0cfa00 2024-04-06T13:49:01.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.014+0000 7f47c8ff9700 1 --2- 172.21.15.189:0/575305108 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f47b806aed0 0x7f47b806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:01.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.014+0000 7f47c8ff9700 1 -- 172.21.15.189:0/575305108 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 7387+0+0 (secure 0 0 0) 0x7f47bc089220 con 0x7f47cc0cfa00 2024-04-06T13:49:01.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.014+0000 7f47cbfff700 1 --2- 172.21.15.189:0/575305108 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f47b806aed0 0x7f47b806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:01.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.015+0000 7f47cbfff700 1 --2- 172.21.15.189:0/575305108 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f47b806aed0 0x7f47b806d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f47c0006b20 tx=0x7f47c0008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:01.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.018+0000 7f47c8ff9700 1 -- 172.21.15.189:0/575305108 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f47bc053bc0 con 0x7f47cc0cfa00 2024-04-06T13:49:01.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.211+0000 7f47d2a0c700 1 -- 172.21.15.189:0/575305108 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f47cc1568a0 con 0x7f47cc0cfa00 2024-04-06T13:49:01.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.868+0000 7f47c8ff9700 1 -- 172.21.15.189:0/575305108 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v257) v1 ==== 169+0+0 (secure 0 0 0) 0x7f47cc1568a0 con 0x7f47cc0cfa00 2024-04-06T13:49:01.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.872+0000 7f47d2a0c700 1 -- 172.21.15.189:0/575305108 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f47b806aed0 msgr2=0x7f47b806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:01.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.872+0000 7f47d2a0c700 1 --2- 172.21.15.189:0/575305108 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f47b806aed0 0x7f47b806d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f47c0006b20 tx=0x7f47c0008040 comp rx=0 tx=0).stop 2024-04-06T13:49:01.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.872+0000 7f47d2a0c700 1 -- 172.21.15.189:0/575305108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47cc0cfa00 msgr2=0x7f47cc15ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:01.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.872+0000 7f47d2a0c700 1 --2- 172.21.15.189:0/575305108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47cc0cfa00 0x7f47cc15ea70 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7f47bc00c420 tx=0x7f47bc00c730 comp rx=0 tx=0).stop 2024-04-06T13:49:01.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.872+0000 7f47d2a0c700 1 -- 172.21.15.189:0/575305108 shutdown_connections 2024-04-06T13:49:01.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.872+0000 7f47d2a0c700 1 --2- 172.21.15.189:0/575305108 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f47b806aed0 0x7f47b806d390 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:01.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.872+0000 7f47d2a0c700 1 --2- 172.21.15.189:0/575305108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47cc0d11b0 0x7f47cc0b43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:01.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.872+0000 7f47d2a0c700 1 --2- 172.21.15.189:0/575305108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47cc0cb870 0x7f47cc155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:01.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.872+0000 7f47d2a0c700 1 --2- 172.21.15.189:0/575305108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47cc0cfa00 0x7f47cc15ea70 unknown :-1 s=CLOSED pgs=2841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:01.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.872+0000 7f47d2a0c700 1 -- 172.21.15.189:0/575305108 >> 172.21.15.189:0/575305108 conn(0x7f47cc0b2db0 msgr2=0x7f47cc159c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:01.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.872+0000 7f47d2a0c700 1 -- 172.21.15.189:0/575305108 shutdown_connections 2024-04-06T13:49:01.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:01.872+0000 7f47d2a0c700 1 -- 172.21.15.189:0/575305108 wait complete. 2024-04-06T13:49:01.882 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' removed 2024-04-06T13:49:01.893 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:49:02.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.018+0000 7f0649b79700 1 Processor -- start 2024-04-06T13:49:02.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.018+0000 7f0649b79700 1 -- start start 2024-04-06T13:49:02.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.019+0000 7f0649b79700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f064415a550 0x7f06440cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.019+0000 7f0649b79700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06440cf900 0x7f06440b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.019+0000 7f0649b79700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06440b4ba0 0x7f064406eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.019+0000 7f0649b79700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06440cdbe0 con 0x7f06440b4ba0 2024-04-06T13:49:02.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.019+0000 7f0649b79700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06440b5010 con 0x7f064415a550 2024-04-06T13:49:02.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.019+0000 7f0649b79700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f064415a940 con 0x7f06440cf900 2024-04-06T13:49:02.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.019+0000 7f0643fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06440b4ba0 0x7f064406eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:02.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.019+0000 7f0643fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06440b4ba0 0x7f064406eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37200/0 (socket says 172.21.15.189:37200) 2024-04-06T13:49:02.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.019+0000 7f0643fff700 1 -- 172.21.15.189:0/3676247823 learned_addr learned my addr 172.21.15.189:0/3676247823 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:02.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.020+0000 7f06437fe700 1 --2- 172.21.15.189:0/3676247823 >> v2:172.21.15.192:3300/0 conn(0x7f064415a550 0x7f06440cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:02.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.020+0000 7f0643fff700 1 -- 172.21.15.189:0/3676247823 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06440cf900 msgr2=0x7f06440b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:02.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.020+0000 7f0642ffd700 1 --2- 172.21.15.189:0/3676247823 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06440cf900 0x7f06440b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:02.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.020+0000 7f0643fff700 1 --2- 172.21.15.189:0/3676247823 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06440cf900 0x7f06440b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.020+0000 7f0643fff700 1 -- 172.21.15.189:0/3676247823 >> v2:172.21.15.192:3300/0 conn(0x7f064415a550 msgr2=0x7f06440cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:02.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.020+0000 7f0643fff700 1 --2- 172.21.15.189:0/3676247823 >> v2:172.21.15.192:3300/0 conn(0x7f064415a550 0x7f06440cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.020+0000 7f0643fff700 1 -- 172.21.15.189:0/3676247823 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064415ac00 con 0x7f06440b4ba0 2024-04-06T13:49:02.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.020+0000 7f06437fe700 1 --2- 172.21.15.189:0/3676247823 >> v2:172.21.15.192:3300/0 conn(0x7f064415a550 0x7f06440cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:02.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.020+0000 7f0642ffd700 1 --2- 172.21.15.189:0/3676247823 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06440cf900 0x7f06440b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:02.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.020+0000 7f0643fff700 1 --2- 172.21.15.189:0/3676247823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06440b4ba0 0x7f064406eb50 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 crypto rx=0x7f0644073470 tx=0x7f063c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40336060592c4433 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:02.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.020+0000 7f06427fc700 1 -- 172.21.15.189:0/3676247823 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063c00b870 con 0x7f06440b4ba0 2024-04-06T13:49:02.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.020+0000 7f06427fc700 1 -- 172.21.15.189:0/3676247823 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f063c00b9d0 con 0x7f06440b4ba0 2024-04-06T13:49:02.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.020+0000 7f06427fc700 1 -- 172.21.15.189:0/3676247823 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063c00bd60 con 0x7f06440b4ba0 2024-04-06T13:49:02.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.020+0000 7f0649b79700 1 -- 172.21.15.189:0/3676247823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06440b4ba0 msgr2=0x7f064406eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:02.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.020+0000 7f0649b79700 1 --2- 172.21.15.189:0/3676247823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06440b4ba0 0x7f064406eb50 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 crypto rx=0x7f0644073470 tx=0x7f063c009bd0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.021+0000 7f0649b79700 1 -- 172.21.15.189:0/3676247823 shutdown_connections 2024-04-06T13:49:02.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.021+0000 7f0649b79700 1 --2- 172.21.15.189:0/3676247823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06440b4ba0 0x7f064406eb50 unknown :-1 s=CLOSED pgs=2842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.021+0000 7f0649b79700 1 --2- 172.21.15.189:0/3676247823 >> v2:172.21.15.192:3300/0 conn(0x7f064415a550 0x7f06440cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.021+0000 7f0649b79700 1 --2- 172.21.15.189:0/3676247823 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06440cf900 0x7f06440b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.021+0000 7f0649b79700 1 -- 172.21.15.189:0/3676247823 >> 172.21.15.189:0/3676247823 conn(0x7f06440d00b0 msgr2=0x7f06440d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:02.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.021+0000 7f0649b79700 1 -- 172.21.15.189:0/3676247823 shutdown_connections 2024-04-06T13:49:02.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.021+0000 7f0649b79700 1 -- 172.21.15.189:0/3676247823 wait complete. 2024-04-06T13:49:02.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.022+0000 7f0649b79700 1 Processor -- start 2024-04-06T13:49:02.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.022+0000 7f0649b79700 1 -- start start 2024-04-06T13:49:02.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.023+0000 7f0649b79700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06440b4ba0 0x7f06441f7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.023+0000 7f0649b79700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06440cf900 0x7f0644157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.023+0000 7f0649b79700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0644151270 0x7f06441439c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.023+0000 7f0649b79700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06440b0630 con 0x7f06440b4ba0 2024-04-06T13:49:02.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.023+0000 7f0649b79700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0644158440 con 0x7f06440cf900 2024-04-06T13:49:02.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.023+0000 7f0649b79700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06440afd40 con 0x7f0644151270 2024-04-06T13:49:02.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.023+0000 7f06437fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06440b4ba0 0x7f06441f7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:02.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.023+0000 7f06437fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06440b4ba0 0x7f06441f7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37210/0 (socket says 172.21.15.189:37210) 2024-04-06T13:49:02.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.023+0000 7f06437fe700 1 -- 172.21.15.189:0/977203715 learned_addr learned my addr 172.21.15.189:0/977203715 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:02.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.023+0000 7f0643fff700 1 --2- 172.21.15.189:0/977203715 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0644151270 0x7f06441439c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:02.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.023+0000 7f0642ffd700 1 --2- 172.21.15.189:0/977203715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06440cf900 0x7f0644157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:02.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.023+0000 7f06437fe700 1 -- 172.21.15.189:0/977203715 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0644151270 msgr2=0x7f06441439c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:02.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.023+0000 7f06437fe700 1 --2- 172.21.15.189:0/977203715 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0644151270 0x7f06441439c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.029 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.023+0000 7f06437fe700 1 -- 172.21.15.189:0/977203715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06440cf900 msgr2=0x7f0644157c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:02.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.023+0000 7f06437fe700 1 --2- 172.21.15.189:0/977203715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06440cf900 0x7f0644157c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.023+0000 7f06437fe700 1 -- 172.21.15.189:0/977203715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063c009550 con 0x7f06440b4ba0 2024-04-06T13:49:02.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.023+0000 7f06437fe700 1 --2- 172.21.15.189:0/977203715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06440b4ba0 0x7f06441f7010 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7f063400cf60 tx=0x7f063400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:02.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.023+0000 7f063bfff700 1 -- 172.21.15.189:0/977203715 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063400ada0 con 0x7f06440b4ba0 2024-04-06T13:49:02.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.023+0000 7f063bfff700 1 -- 172.21.15.189:0/977203715 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0634015440 con 0x7f06440b4ba0 2024-04-06T13:49:02.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.024+0000 7f0649b79700 1 -- 172.21.15.189:0/977203715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06441581a0 con 0x7f06440b4ba0 2024-04-06T13:49:02.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.024+0000 7f063bfff700 1 -- 172.21.15.189:0/977203715 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06340157d0 con 0x7f06440b4ba0 2024-04-06T13:49:02.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.024+0000 7f0649b79700 1 -- 172.21.15.189:0/977203715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06440b02c0 con 0x7f06440b4ba0 2024-04-06T13:49:02.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.024+0000 7f0649b79700 1 -- 172.21.15.189:0/977203715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0644067bb0 con 0x7f06440b4ba0 2024-04-06T13:49:02.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.025+0000 7f063bfff700 1 -- 172.21.15.189:0/977203715 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f0634015930 con 0x7f06440b4ba0 2024-04-06T13:49:02.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.025+0000 7f063bfff700 1 --2- 172.21.15.189:0/977203715 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f062406aed0 0x7f062406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.025+0000 7f063bfff700 1 -- 172.21.15.189:0/977203715 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6973+0+0 (secure 0 0 0) 0x7f0634089f40 con 0x7f06440b4ba0 2024-04-06T13:49:02.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.025+0000 7f0642ffd700 1 --2- 172.21.15.189:0/977203715 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f062406aed0 0x7f062406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:02.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.026+0000 7f0642ffd700 1 --2- 172.21.15.189:0/977203715 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f062406aed0 0x7f062406d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0644167150 tx=0x7f062c008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:02.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.030+0000 7f063bfff700 1 -- 172.21.15.189:0/977203715 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0634054a10 con 0x7f06440b4ba0 2024-04-06T13:49:02.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.226+0000 7f0649b79700 1 -- 172.21.15.189:0/977203715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06440d0e30 con 0x7f06440b4ba0 2024-04-06T13:49:02.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.227+0000 7f063bfff700 1 -- 172.21.15.189:0/977203715 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v257) v1 ==== 74+0+17683 (secure 0 0 0) 0x7f0634058030 con 0x7f06440b4ba0 2024-04-06T13:49:02.228 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:02.228 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":257,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:49:01.853081+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:48:33.912100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:48:35.170642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.192:6833/726816746":"2024-04-07T13:48:58.736634+0000","172.21.15.192:6832/726816746":"2024-04-07T13:48:58.736634+0000","172.21.15.189:0/1050000893":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/2601670651":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/3142793643":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/1266339172":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/3326966043":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/4043705133":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/85704189":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2120970213":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/381758424":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2635472538":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/2107771319":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2145696658":"2024-04-07T13:48:48.812402+0000","172.21.15.189:6833/111704":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/3440291825":"2024-04-07T13:48:48.812402+0000","172.21.15.189:6832/111704":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/598265929":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/485512536":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/643921686":"2024-04-07T13:48:48.812402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:49:02.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.229+0000 7f0649b79700 1 -- 172.21.15.189:0/977203715 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f062406aed0 msgr2=0x7f062406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:02.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.229+0000 7f0649b79700 1 --2- 172.21.15.189:0/977203715 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f062406aed0 0x7f062406d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0644167150 tx=0x7f062c008040 comp rx=0 tx=0).stop 2024-04-06T13:49:02.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.229+0000 7f0649b79700 1 -- 172.21.15.189:0/977203715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06440b4ba0 msgr2=0x7f06441f7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:02.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.229+0000 7f0649b79700 1 --2- 172.21.15.189:0/977203715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06440b4ba0 0x7f06441f7010 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7f063400cf60 tx=0x7f063400cf90 comp rx=0 tx=0).stop 2024-04-06T13:49:02.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.229+0000 7f0649b79700 1 -- 172.21.15.189:0/977203715 shutdown_connections 2024-04-06T13:49:02.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.229+0000 7f0649b79700 1 --2- 172.21.15.189:0/977203715 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f062406aed0 0x7f062406d390 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.229+0000 7f0649b79700 1 --2- 172.21.15.189:0/977203715 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0644151270 0x7f06441439c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.229+0000 7f0649b79700 1 --2- 172.21.15.189:0/977203715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06440b4ba0 0x7f06441f7010 unknown :-1 s=CLOSED pgs=2843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.229+0000 7f0649b79700 1 --2- 172.21.15.189:0/977203715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06440cf900 0x7f0644157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.229+0000 7f0649b79700 1 -- 172.21.15.189:0/977203715 >> 172.21.15.189:0/977203715 conn(0x7f06440d00b0 msgr2=0x7f0644071e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:02.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.230+0000 7f0649b79700 1 -- 172.21.15.189:0/977203715 shutdown_connections 2024-04-06T13:49:02.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.230+0000 7f0649b79700 1 -- 172.21.15.189:0/977203715 wait complete. 2024-04-06T13:49:02.243 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-04-06T13:48:33.912100+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '251', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2024-04-06T13:48:35.170642+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '252', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:49:02.243 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:49:02.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.367+0000 7f1681b18700 1 Processor -- start 2024-04-06T13:49:02.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.367+0000 7f1681b18700 1 -- start start 2024-04-06T13:49:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.368+0000 7f1681b18700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f167c15a550 0x7f167c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.368+0000 7f1681b18700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f167c0cdbe0 con 0x7f167c0b4d70 2024-04-06T13:49:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.368+0000 7f1681b18700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f167c15a940 con 0x7f167c0cf900 2024-04-06T13:49:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.368+0000 7f1681b18700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f167c15ac00 con 0x7f167c15a550 2024-04-06T13:49:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.368+0000 7f167b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f167c15a550 0x7f167c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.368+0000 7f167b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f167c15a550 0x7f167c0cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37212/0 (socket says 172.21.15.189:37212) 2024-04-06T13:49:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.368+0000 7f167b7fe700 1 -- 172.21.15.189:0/3133795190 learned_addr learned my addr 172.21.15.189:0/3133795190 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.368+0000 7f167bfff700 1 --1- 172.21.15.189:0/3133795190 >> v1:172.21.15.189:6789/0 conn(0x7f167c0b4d70 0x7f167c0b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50442/0 (socket says 172.21.15.189:50442) 2024-04-06T13:49:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.369+0000 7f167b7fe700 1 -- 172.21.15.189:0/3133795190 >> v1:172.21.15.192:6789/0 conn(0x7f167c0cf900 legacy=0x7f167c0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.369+0000 7f167b7fe700 1 -- 172.21.15.189:0/3133795190 >> v1:172.21.15.189:6789/0 conn(0x7f167c0b4d70 legacy=0x7f167c0b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.369+0000 7f167b7fe700 1 -- 172.21.15.189:0/3133795190 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c15aee0 con 0x7f167c15a550 2024-04-06T13:49:02.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.369+0000 7f167a7fc700 1 -- 172.21.15.189:0/3133795190 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1863047106 0 0) 0x7f167c0cdbe0 con 0x7f167c0b4d70 2024-04-06T13:49:02.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.369+0000 7f167b7fe700 1 --2- 172.21.15.189:0/3133795190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f167c15a550 0x7f167c0cf3c0 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7f16640098f0 tx=0x7f1664009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1f9e4f052793e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:02.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.369+0000 7f167a7fc700 1 -- 172.21.15.189:0/3133795190 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1664004230 con 0x7f167c15a550 2024-04-06T13:49:02.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.369+0000 7f167a7fc700 1 -- 172.21.15.189:0/3133795190 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1664004390 con 0x7f167c15a550 2024-04-06T13:49:02.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.369+0000 7f167a7fc700 1 -- 172.21.15.189:0/3133795190 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16640036a0 con 0x7f167c15a550 2024-04-06T13:49:02.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.370+0000 7f1681b18700 1 -- 172.21.15.189:0/3133795190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f167c15a550 msgr2=0x7f167c0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:02.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.370+0000 7f1681b18700 1 --2- 172.21.15.189:0/3133795190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f167c15a550 0x7f167c0cf3c0 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7f16640098f0 tx=0x7f1664009c00 comp rx=0 tx=0).stop 2024-04-06T13:49:02.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.370+0000 7f1681b18700 1 -- 172.21.15.189:0/3133795190 shutdown_connections 2024-04-06T13:49:02.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.370+0000 7f1681b18700 1 --2- 172.21.15.189:0/3133795190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f167c15a550 0x7f167c0cf3c0 unknown :-1 s=CLOSED pgs=2845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.370+0000 7f1681b18700 1 -- 172.21.15.189:0/3133795190 >> 172.21.15.189:0/3133795190 conn(0x7f167c0d00b0 msgr2=0x7f167c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:02.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.370+0000 7f1681b18700 1 -- 172.21.15.189:0/3133795190 shutdown_connections 2024-04-06T13:49:02.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.370+0000 7f1681b18700 1 -- 172.21.15.189:0/3133795190 wait complete. 2024-04-06T13:49:02.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.371+0000 7f1681b18700 1 Processor -- start 2024-04-06T13:49:02.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.371+0000 7f1681b18700 1 -- start start 2024-04-06T13:49:02.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f1681b18700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f167c0b4d70 0x7f167c144ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f1681b18700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f167c0cf900 0x7f167c06a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f1681b18700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f167c06ac70 0x7f167c0ae840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f1681b18700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167c06b120 con 0x7f167c0cf900 2024-04-06T13:49:02.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f1681b18700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167c145010 con 0x7f167c0b4d70 2024-04-06T13:49:02.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f1681b18700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f167c145250 con 0x7f167c06ac70 2024-04-06T13:49:02.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f167affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f167c0cf900 0x7f167c06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:02.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f167affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f167c0cf900 0x7f167c06a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37224/0 (socket says 172.21.15.189:37224) 2024-04-06T13:49:02.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f167affd700 1 -- 172.21.15.189:0/2600117541 learned_addr learned my addr 172.21.15.189:0/2600117541 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:02.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f167b7fe700 1 --2- 172.21.15.189:0/2600117541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f167c0b4d70 0x7f167c144ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:02.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f167bfff700 1 --2- 172.21.15.189:0/2600117541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f167c06ac70 0x7f167c0ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:02.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f167affd700 1 -- 172.21.15.189:0/2600117541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f167c06ac70 msgr2=0x7f167c0ae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:02.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f167affd700 1 --2- 172.21.15.189:0/2600117541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f167c06ac70 0x7f167c0ae840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f167affd700 1 -- 172.21.15.189:0/2600117541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f167c0b4d70 msgr2=0x7f167c144ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:02.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f167affd700 1 --2- 172.21.15.189:0/2600117541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f167c0b4d70 0x7f167c144ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f167affd700 1 -- 172.21.15.189:0/2600117541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1664009550 con 0x7f167c0cf900 2024-04-06T13:49:02.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f167affd700 1 --2- 172.21.15.189:0/2600117541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f167c0cf900 0x7f167c06a730 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7f166c00c4d0 tx=0x7f166c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:02.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f1673fff700 1 -- 172.21.15.189:0/2600117541 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c011790 con 0x7f167c0cf900 2024-04-06T13:49:02.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f1673fff700 1 -- 172.21.15.189:0/2600117541 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f166c0118f0 con 0x7f167c0cf900 2024-04-06T13:49:02.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f1681b18700 1 -- 172.21.15.189:0/2600117541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f167c145530 con 0x7f167c0cf900 2024-04-06T13:49:02.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.372+0000 7f1673fff700 1 -- 172.21.15.189:0/2600117541 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c011ca0 con 0x7f167c0cf900 2024-04-06T13:49:02.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.373+0000 7f1681b18700 1 -- 172.21.15.189:0/2600117541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f167c145af0 con 0x7f167c0cf900 2024-04-06T13:49:02.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.373+0000 7f1681b18700 1 -- 172.21.15.189:0/2600117541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f167c067bb0 con 0x7f167c0cf900 2024-04-06T13:49:02.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.374+0000 7f1673fff700 1 -- 172.21.15.189:0/2600117541 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f166c01c460 con 0x7f167c0cf900 2024-04-06T13:49:02.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.374+0000 7f1673fff700 1 --2- 172.21.15.189:0/2600117541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f165c06aed0 0x7f165c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.374+0000 7f1673fff700 1 -- 172.21.15.189:0/2600117541 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6973+0+0 (secure 0 0 0) 0x7f166c089810 con 0x7f167c0cf900 2024-04-06T13:49:02.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.374+0000 7f167b7fe700 1 --2- 172.21.15.189:0/2600117541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f165c06aed0 0x7f165c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:02.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.375+0000 7f167b7fe700 1 --2- 172.21.15.189:0/2600117541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f165c06aed0 0x7f165c06d390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1664009130 tx=0x7f1664008030 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:02.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.379+0000 7f1673fff700 1 -- 172.21.15.189:0/2600117541 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f166c054270 con 0x7f167c0cf900 2024-04-06T13:49:02.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.571+0000 7f1681b18700 1 -- 172.21.15.189:0/2600117541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f167c074990 con 0x7f167c0cf900 2024-04-06T13:49:02.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.572+0000 7f1673fff700 1 -- 172.21.15.189:0/2600117541 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v257) v1 ==== 93+0+10 (secure 0 0 0) 0x7f166c057890 con 0x7f167c0cf900 2024-04-06T13:49:02.573 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:49:02.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.574+0000 7f1681b18700 1 -- 172.21.15.189:0/2600117541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f165c06aed0 msgr2=0x7f165c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:02.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.574+0000 7f1681b18700 1 --2- 172.21.15.189:0/2600117541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f165c06aed0 0x7f165c06d390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1664009130 tx=0x7f1664008030 comp rx=0 tx=0).stop 2024-04-06T13:49:02.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.574+0000 7f1681b18700 1 -- 172.21.15.189:0/2600117541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f167c0cf900 msgr2=0x7f167c06a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:02.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.574+0000 7f1681b18700 1 --2- 172.21.15.189:0/2600117541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f167c0cf900 0x7f167c06a730 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7f166c00c4d0 tx=0x7f166c00c7e0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.574+0000 7f1681b18700 1 -- 172.21.15.189:0/2600117541 shutdown_connections 2024-04-06T13:49:02.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.574+0000 7f1681b18700 1 --2- 172.21.15.189:0/2600117541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f165c06aed0 0x7f165c06d390 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.574+0000 7f1681b18700 1 --2- 172.21.15.189:0/2600117541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f167c06ac70 0x7f167c0ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.574+0000 7f1681b18700 1 --2- 172.21.15.189:0/2600117541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f167c0b4d70 0x7f167c144ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.574+0000 7f1681b18700 1 --2- 172.21.15.189:0/2600117541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f167c0cf900 0x7f167c06a730 unknown :-1 s=CLOSED pgs=2846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.574+0000 7f1681b18700 1 -- 172.21.15.189:0/2600117541 >> 172.21.15.189:0/2600117541 conn(0x7f167c0d00b0 msgr2=0x7f167c0b2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:02.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.574+0000 7f1681b18700 1 -- 172.21.15.189:0/2600117541 shutdown_connections 2024-04-06T13:49:02.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.574+0000 7f1681b18700 1 -- 172.21.15.189:0/2600117541 wait complete. 2024-04-06T13:49:02.587 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-04-06T13:49:02.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.713+0000 7fef94ad5700 1 Processor -- start 2024-04-06T13:49:02.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.713+0000 7fef94ad5700 1 -- start start 2024-04-06T13:49:02.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.714+0000 7fef94ad5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef90146dd0 0x7fef9006d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.714+0000 7fef94ad5700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fef900cedf0 0x7fef900cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.714+0000 7fef94ad5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef900cf760 con 0x7fef900d17a0 2024-04-06T13:49:02.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.714+0000 7fef94ad5700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef9006beb0 con 0x7fef900cedf0 2024-04-06T13:49:02.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.714+0000 7fef94ad5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9006c0a0 con 0x7fef90146dd0 2024-04-06T13:49:02.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.714+0000 7fef8e59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fef900d17a0 0x7fef9006b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50448/0 (socket says 172.21.15.189:50448) 2024-04-06T13:49:02.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.714+0000 7fef8e59c700 1 -- 172.21.15.189:0/559372582 learned_addr learned my addr 172.21.15.189:0/559372582 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:02.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8ed9d700 1 --2- 172.21.15.189:0/559372582 >> v2:172.21.15.192:3300/0 conn(0x7fef900cedf0 0x7fef900cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:02.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8d59a700 1 -- 172.21.15.189:0/559372582 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77953878 0 0) 0x7fef900cf760 con 0x7fef900d17a0 2024-04-06T13:49:02.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8dd9b700 1 --2- 172.21.15.189:0/559372582 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef90146dd0 0x7fef9006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:02.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8d59a700 1 -- 172.21.15.189:0/559372582 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef780035b0 con 0x7fef900d17a0 2024-04-06T13:49:02.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8d59a700 1 -- 172.21.15.189:0/559372582 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1567074253 0 0) 0x7fef780035b0 con 0x7fef900d17a0 2024-04-06T13:49:02.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8d59a700 1 -- 172.21.15.189:0/559372582 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef90146dd0 msgr2=0x7fef9006d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:02.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8d59a700 1 --2- 172.21.15.189:0/559372582 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef90146dd0 0x7fef9006d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8d59a700 1 -- 172.21.15.189:0/559372582 >> v2:172.21.15.192:3300/0 conn(0x7fef900cedf0 msgr2=0x7fef900cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:02.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8d59a700 1 --2- 172.21.15.189:0/559372582 >> v2:172.21.15.192:3300/0 conn(0x7fef900cedf0 0x7fef900cf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8d59a700 1 -- 172.21.15.189:0/559372582 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9006c380 con 0x7fef900d17a0 2024-04-06T13:49:02.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8d59a700 1 -- 172.21.15.189:0/559372582 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fef84001670 con 0x7fef900d17a0 2024-04-06T13:49:02.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8d59a700 1 -- 172.21.15.189:0/559372582 >> v1:172.21.15.189:6789/0 conn(0x7fef900d17a0 legacy=0x7fef9006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:02.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8d59a700 1 --2- 172.21.15.189:0/559372582 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef90146dd0 0x7fef78003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8d59a700 1 --2- 172.21.15.189:0/559372582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef78003f80 0x7fef78004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8d59a700 1 --2- 172.21.15.189:0/559372582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef78004d90 0x7fef78005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8d59a700 1 -- 172.21.15.189:0/559372582 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9006c0a0 con 0x7fef78004d90 2024-04-06T13:49:02.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8d59a700 1 -- 172.21.15.189:0/559372582 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7800ba10 con 0x7fef78003f80 2024-04-06T13:49:02.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8d59a700 1 -- 172.21.15.189:0/559372582 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7800bb50 con 0x7fef90146dd0 2024-04-06T13:49:02.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8ed9d700 1 --2- 172.21.15.189:0/559372582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef78004d90 0x7fef78005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:02.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8dd9b700 1 --2- 172.21.15.189:0/559372582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef78003f80 0x7fef78004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:02.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8e59c700 1 --2- 172.21.15.189:0/559372582 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef90146dd0 0x7fef78003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:02.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8ed9d700 1 -- 172.21.15.189:0/559372582 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef90146dd0 msgr2=0x7fef78003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:02.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8ed9d700 1 --2- 172.21.15.189:0/559372582 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef90146dd0 0x7fef78003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8ed9d700 1 -- 172.21.15.189:0/559372582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef78003f80 msgr2=0x7fef78004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:02.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8ed9d700 1 --2- 172.21.15.189:0/559372582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef78003f80 0x7fef78004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.715+0000 7fef8ed9d700 1 -- 172.21.15.189:0/559372582 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef780035b0 con 0x7fef78004d90 2024-04-06T13:49:02.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.716+0000 7fef8ed9d700 1 --2- 172.21.15.189:0/559372582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef78004d90 0x7fef78005210 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 crypto rx=0x7fef88009fc0 tx=0x7fef78005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f429a16be5160f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:02.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.716+0000 7fef8d59a700 1 -- 172.21.15.189:0/559372582 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8800ebd0 con 0x7fef78004d90 2024-04-06T13:49:02.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.716+0000 7fef8d59a700 1 -- 172.21.15.189:0/559372582 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8800f040 con 0x7fef78004d90 2024-04-06T13:49:02.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.716+0000 7fef8d59a700 1 -- 172.21.15.189:0/559372582 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88014480 con 0x7fef78004d90 2024-04-06T13:49:02.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.716+0000 7fef94ad5700 1 -- 172.21.15.189:0/559372582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef78004d90 msgr2=0x7fef78005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:02.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.716+0000 7fef94ad5700 1 --2- 172.21.15.189:0/559372582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef78004d90 0x7fef78005210 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 crypto rx=0x7fef88009fc0 tx=0x7fef78005880 comp rx=0 tx=0).stop 2024-04-06T13:49:02.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.716+0000 7fef8ed9d700 1 -- 172.21.15.189:0/559372582 reap_dead start 2024-04-06T13:49:02.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.716+0000 7fef94ad5700 1 -- 172.21.15.189:0/559372582 shutdown_connections 2024-04-06T13:49:02.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.716+0000 7fef94ad5700 1 -- 172.21.15.189:0/559372582 >> 172.21.15.189:0/559372582 conn(0x7fef900b2db0 msgr2=0x7fef900b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:02.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.716+0000 7fef94ad5700 1 -- 172.21.15.189:0/559372582 shutdown_connections 2024-04-06T13:49:02.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.716+0000 7fef94ad5700 1 -- 172.21.15.189:0/559372582 wait complete. 2024-04-06T13:49:02.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.718+0000 7fef94ad5700 1 Processor -- start 2024-04-06T13:49:02.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.718+0000 7fef94ad5700 1 -- start start 2024-04-06T13:49:02.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.718+0000 7fef94ad5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef78004d90 0x7fef9015ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.718+0000 7fef94ad5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef90160480 0x7fef9014c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.718+0000 7fef94ad5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef9014cec0 0x7fef9014d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.718+0000 7fef94ad5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef90160900 con 0x7fef78004d90 2024-04-06T13:49:02.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.718+0000 7fef94ad5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9014ea50 con 0x7fef9014cec0 2024-04-06T13:49:02.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.718+0000 7fef94ad5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9014de80 con 0x7fef90160480 2024-04-06T13:49:02.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.719+0000 7fef8e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef78004d90 0x7fef9015ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:02.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.719+0000 7fef8e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef78004d90 0x7fef9015ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37236/0 (socket says 172.21.15.189:37236) 2024-04-06T13:49:02.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.719+0000 7fef8e59c700 1 -- 172.21.15.189:0/2727901265 learned_addr learned my addr 172.21.15.189:0/2727901265 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:02.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.719+0000 7fef8e59c700 1 -- 172.21.15.189:0/2727901265 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef90160480 msgr2=0x7fef9014c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:02.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.719+0000 7fef8e59c700 1 --2- 172.21.15.189:0/2727901265 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef90160480 0x7fef9014c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.719+0000 7fef8e59c700 1 -- 172.21.15.189:0/2727901265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef9014cec0 msgr2=0x7fef9014d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:02.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.719+0000 7fef8e59c700 1 --2- 172.21.15.189:0/2727901265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef9014cec0 0x7fef9014d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.719+0000 7fef8e59c700 1 -- 172.21.15.189:0/2727901265 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9014e120 con 0x7fef78004d90 2024-04-06T13:49:02.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.719+0000 7fef8e59c700 1 --2- 172.21.15.189:0/2727901265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef78004d90 0x7fef9015ff40 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7fef84001cc0 tx=0x7fef840076a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:02.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.719+0000 7fef7effd700 1 -- 172.21.15.189:0/2727901265 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8400f820 con 0x7fef78004d90 2024-04-06T13:49:02.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.719+0000 7fef7effd700 1 -- 172.21.15.189:0/2727901265 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8400f9c0 con 0x7fef78004d90 2024-04-06T13:49:02.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.719+0000 7fef94ad5700 1 -- 172.21.15.189:0/2727901265 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9014e400 con 0x7fef78004d90 2024-04-06T13:49:02.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.719+0000 7fef7effd700 1 -- 172.21.15.189:0/2727901265 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8400fd70 con 0x7fef78004d90 2024-04-06T13:49:02.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.719+0000 7fef94ad5700 1 -- 172.21.15.189:0/2727901265 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9014b660 con 0x7fef78004d90 2024-04-06T13:49:02.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.720+0000 7fef94ad5700 1 -- 172.21.15.189:0/2727901265 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef90074990 con 0x7fef78004d90 2024-04-06T13:49:02.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.720+0000 7fef7effd700 1 -- 172.21.15.189:0/2727901265 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fef8401d460 con 0x7fef78004d90 2024-04-06T13:49:02.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.721+0000 7fef7effd700 1 --2- 172.21.15.189:0/2727901265 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fef7006aed0 0x7fef7006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:02.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.721+0000 7fef7effd700 1 -- 172.21.15.189:0/2727901265 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6973+0+0 (secure 0 0 0) 0x7fef84089da0 con 0x7fef78004d90 2024-04-06T13:49:02.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.721+0000 7fef8dd9b700 1 --2- 172.21.15.189:0/2727901265 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fef7006aed0 0x7fef7006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:02.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.721+0000 7fef8dd9b700 1 --2- 172.21.15.189:0/2727901265 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fef7006aed0 0x7fef7006d390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fef9006e230 tx=0x7fef800093c0 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:02.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.725+0000 7fef7effd700 1 -- 172.21.15.189:0/2727901265 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fef84054730 con 0x7fef78004d90 2024-04-06T13:49:02.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.924+0000 7fef94ad5700 1 -- 172.21.15.189:0/2727901265 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fef900d1e20 con 0x7fef78004d90 2024-04-06T13:49:02.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.925+0000 7fef7effd700 1 -- 172.21.15.189:0/2727901265 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v257) v1 ==== 107+0+11 (secure 0 0 0) 0x7fef84057d50 con 0x7fef78004d90 2024-04-06T13:49:02.926 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:49:02.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.927+0000 7fef94ad5700 1 -- 172.21.15.189:0/2727901265 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fef7006aed0 msgr2=0x7fef7006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:02.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.927+0000 7fef94ad5700 1 --2- 172.21.15.189:0/2727901265 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fef7006aed0 0x7fef7006d390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fef9006e230 tx=0x7fef800093c0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.927+0000 7fef94ad5700 1 -- 172.21.15.189:0/2727901265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef78004d90 msgr2=0x7fef9015ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:02.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.927+0000 7fef94ad5700 1 --2- 172.21.15.189:0/2727901265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef78004d90 0x7fef9015ff40 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7fef84001cc0 tx=0x7fef840076a0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.927+0000 7fef94ad5700 1 -- 172.21.15.189:0/2727901265 shutdown_connections 2024-04-06T13:49:02.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.927+0000 7fef94ad5700 1 --2- 172.21.15.189:0/2727901265 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fef7006aed0 0x7fef7006d390 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.927+0000 7fef94ad5700 1 --2- 172.21.15.189:0/2727901265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef9014cec0 0x7fef9014d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.927+0000 7fef94ad5700 1 --2- 172.21.15.189:0/2727901265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef78004d90 0x7fef9015ff40 unknown :-1 s=CLOSED pgs=2849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.927+0000 7fef94ad5700 1 --2- 172.21.15.189:0/2727901265 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef90160480 0x7fef9014c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:02.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.928+0000 7fef94ad5700 1 -- 172.21.15.189:0/2727901265 >> 172.21.15.189:0/2727901265 conn(0x7fef900b2db0 msgr2=0x7fef900b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:02.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.928+0000 7fef94ad5700 1 -- 172.21.15.189:0/2727901265 shutdown_connections 2024-04-06T13:49:02.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:02.928+0000 7fef94ad5700 1 -- 172.21.15.189:0/2727901265 wait complete. 2024-04-06T13:49:02.941 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-04-06T13:49:03.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.064+0000 7f3fa1a8b700 1 Processor -- start 2024-04-06T13:49:03.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.065+0000 7f3fa1a8b700 1 -- start start 2024-04-06T13:49:03.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.065+0000 7f3fa1a8b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f9c15a550 0x7f3f9c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:03.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.065+0000 7f3fa1a8b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3f9c0b4d70 0x7f3f9c0b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:03.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.065+0000 7f3fa1a8b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f9c0cdbe0 con 0x7f3f9c0cf900 2024-04-06T13:49:03.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.065+0000 7f3fa1a8b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f9c15a940 con 0x7f3f9c0b4d70 2024-04-06T13:49:03.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.065+0000 7f3fa1a8b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c15ac00 con 0x7f3f9c15a550 2024-04-06T13:49:03.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.065+0000 7f3f9bfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3f9c0b4d70 0x7f3f9c0b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:03.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.065+0000 7f3f9bfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3f9c0b4d70 0x7f3f9c0b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37240/0 (socket says 172.21.15.189:37240) 2024-04-06T13:49:03.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.066+0000 7f3f9bfff700 1 -- 172.21.15.189:0/862192231 learned_addr learned my addr 172.21.15.189:0/862192231 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:03.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.066+0000 7f3f9b7fe700 1 --2- 172.21.15.189:0/862192231 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f9c15a550 0x7f3f9c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:03.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.066+0000 7f3f9bfff700 1 -- 172.21.15.189:0/862192231 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f9c15a550 msgr2=0x7f3f9c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:03.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.066+0000 7f3f9bfff700 1 --2- 172.21.15.189:0/862192231 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f9c15a550 0x7f3f9c0cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.066+0000 7f3f9bfff700 1 -- 172.21.15.189:0/862192231 >> v1:172.21.15.192:6789/0 conn(0x7f3f9c0cf900 legacy=0x7f3f9c0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:03.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.066+0000 7f3f9bfff700 1 -- 172.21.15.189:0/862192231 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9c15aee0 con 0x7f3f9c0b4d70 2024-04-06T13:49:03.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.066+0000 7f3f9b7fe700 1 --2- 172.21.15.189:0/862192231 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f9c15a550 0x7f3f9c0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:03.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.066+0000 7f3f9bfff700 1 --2- 172.21.15.189:0/862192231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c0b4d70 0x7f3f9c0b2fc0 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 crypto rx=0x7f3f94009e50 tx=0x7f3f9400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8283cf462d2d0da7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:03.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.066+0000 7f3f9a7fc700 1 -- 172.21.15.189:0/862192231 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9400e070 con 0x7f3f9c0b4d70 2024-04-06T13:49:03.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.066+0000 7f3f9a7fc700 1 -- 172.21.15.189:0/862192231 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f9400b9c0 con 0x7f3f9c0b4d70 2024-04-06T13:49:03.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.066+0000 7f3f9a7fc700 1 -- 172.21.15.189:0/862192231 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9400bd70 con 0x7f3f9c0b4d70 2024-04-06T13:49:03.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.067+0000 7f3fa1a8b700 1 -- 172.21.15.189:0/862192231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c0b4d70 msgr2=0x7f3f9c0b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:03.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.067+0000 7f3fa1a8b700 1 --2- 172.21.15.189:0/862192231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c0b4d70 0x7f3f9c0b2fc0 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 crypto rx=0x7f3f94009e50 tx=0x7f3f9400b330 comp rx=0 tx=0).stop 2024-04-06T13:49:03.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.067+0000 7f3fa1a8b700 1 -- 172.21.15.189:0/862192231 shutdown_connections 2024-04-06T13:49:03.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.067+0000 7f3fa1a8b700 1 --2- 172.21.15.189:0/862192231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c0b4d70 0x7f3f9c0b2fc0 unknown :-1 s=CLOSED pgs=2850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.067+0000 7f3fa1a8b700 1 --2- 172.21.15.189:0/862192231 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f9c15a550 0x7f3f9c0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.067+0000 7f3fa1a8b700 1 -- 172.21.15.189:0/862192231 >> 172.21.15.189:0/862192231 conn(0x7f3f9c0d00b0 msgr2=0x7f3f9c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:03.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.067+0000 7f3fa1a8b700 1 -- 172.21.15.189:0/862192231 shutdown_connections 2024-04-06T13:49:03.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.067+0000 7f3fa1a8b700 1 -- 172.21.15.189:0/862192231 wait complete. 2024-04-06T13:49:03.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.068+0000 7f3fa1a8b700 1 Processor -- start 2024-04-06T13:49:03.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.068+0000 7f3fa1a8b700 1 -- start start 2024-04-06T13:49:03.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.069+0000 7f3fa1a8b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f9c0b4d70 0x7f3f9c1579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:03.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.069+0000 7f3fa1a8b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f9c0cf900 0x7f3f9c0aed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:03.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.069+0000 7f3fa1a8b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c14eca0 0x7f3f9c0b0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:03.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.069+0000 7f3fa1a8b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c0af240 con 0x7f3f9c14eca0 2024-04-06T13:49:03.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.069+0000 7f3fa1a8b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c1f0840 con 0x7f3f9c0cf900 2024-04-06T13:49:03.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.069+0000 7f3fa1a8b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c1f0b10 con 0x7f3f9c0b4d70 2024-04-06T13:49:03.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.069+0000 7f3f9bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c14eca0 0x7f3f9c0b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:03.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.069+0000 7f3f9bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c14eca0 0x7f3f9c0b0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37256/0 (socket says 172.21.15.189:37256) 2024-04-06T13:49:03.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.069+0000 7f3f9bfff700 1 -- 172.21.15.189:0/3211213075 learned_addr learned my addr 172.21.15.189:0/3211213075 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:03.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.069+0000 7f3f9b7fe700 1 --2- 172.21.15.189:0/3211213075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f9c0b4d70 0x7f3f9c1579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:03.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.069+0000 7f3f9affd700 1 --2- 172.21.15.189:0/3211213075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f9c0cf900 0x7f3f9c0aed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:03.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.069+0000 7f3f9bfff700 1 -- 172.21.15.189:0/3211213075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f9c0b4d70 msgr2=0x7f3f9c1579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:03.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.069+0000 7f3f9bfff700 1 --2- 172.21.15.189:0/3211213075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f9c0b4d70 0x7f3f9c1579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.069+0000 7f3f9bfff700 1 -- 172.21.15.189:0/3211213075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f9c0cf900 msgr2=0x7f3f9c0aed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:03.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.069+0000 7f3f9bfff700 1 --2- 172.21.15.189:0/3211213075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f9c0cf900 0x7f3f9c0aed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.069+0000 7f3f9bfff700 1 -- 172.21.15.189:0/3211213075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f94009b30 con 0x7f3f9c14eca0 2024-04-06T13:49:03.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.069+0000 7f3f9bfff700 1 --2- 172.21.15.189:0/3211213075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c14eca0 0x7f3f9c0b0780 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f3f94012000 tx=0x7f3f94004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:03.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.070+0000 7f3f93fff700 1 -- 172.21.15.189:0/3211213075 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f94004800 con 0x7f3f9c14eca0 2024-04-06T13:49:03.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.070+0000 7f3f93fff700 1 -- 172.21.15.189:0/3211213075 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f94004960 con 0x7f3f9c14eca0 2024-04-06T13:49:03.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.070+0000 7f3fa1a8b700 1 -- 172.21.15.189:0/3211213075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f9c1f0df0 con 0x7f3f9c14eca0 2024-04-06T13:49:03.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.070+0000 7f3f93fff700 1 -- 172.21.15.189:0/3211213075 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9401c680 con 0x7f3f9c14eca0 2024-04-06T13:49:03.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.070+0000 7f3fa1a8b700 1 -- 172.21.15.189:0/3211213075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f9c1f13b0 con 0x7f3f9c14eca0 2024-04-06T13:49:03.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.071+0000 7f3fa1a8b700 1 -- 172.21.15.189:0/3211213075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f9c067bb0 con 0x7f3f9c14eca0 2024-04-06T13:49:03.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.071+0000 7f3f93fff700 1 -- 172.21.15.189:0/3211213075 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f3f9401c7e0 con 0x7f3f9c14eca0 2024-04-06T13:49:03.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.071+0000 7f3f93fff700 1 --2- 172.21.15.189:0/3211213075 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3f7c06aed0 0x7f3f7c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:03.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.071+0000 7f3f93fff700 1 -- 172.21.15.189:0/3211213075 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6973+0+0 (secure 0 0 0) 0x7f3f9408b570 con 0x7f3f9c14eca0 2024-04-06T13:49:03.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.071+0000 7f3f9b7fe700 1 --2- 172.21.15.189:0/3211213075 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3f7c06aed0 0x7f3f7c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:03.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.072+0000 7f3f9b7fe700 1 --2- 172.21.15.189:0/3211213075 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3f7c06aed0 0x7f3f7c06d390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400c940 tx=0x7f3f84008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:03.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.076+0000 7f3f93fff700 1 -- 172.21.15.189:0/3211213075 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3f94055e80 con 0x7f3f9c14eca0 2024-04-06T13:49:03.149 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:49:03.152 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:49:03.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.277+0000 7f3fa1a8b700 1 -- 172.21.15.189:0/3211213075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f3f9c074990 con 0x7f3f9c14eca0 2024-04-06T13:49:03.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.277+0000 7f3f93fff700 1 -- 172.21.15.189:0/3211213075 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v257) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3f940594a0 con 0x7f3f9c14eca0 2024-04-06T13:49:03.278 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:49:03.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.279+0000 7f3fa1a8b700 1 -- 172.21.15.189:0/3211213075 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3f7c06aed0 msgr2=0x7f3f7c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:03.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.279+0000 7f3fa1a8b700 1 --2- 172.21.15.189:0/3211213075 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3f7c06aed0 0x7f3f7c06d390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400c940 tx=0x7f3f84008040 comp rx=0 tx=0).stop 2024-04-06T13:49:03.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.279+0000 7f3fa1a8b700 1 -- 172.21.15.189:0/3211213075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c14eca0 msgr2=0x7f3f9c0b0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:03.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.279+0000 7f3fa1a8b700 1 --2- 172.21.15.189:0/3211213075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c14eca0 0x7f3f9c0b0780 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f3f94012000 tx=0x7f3f94004ad0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.280+0000 7f3fa1a8b700 1 -- 172.21.15.189:0/3211213075 shutdown_connections 2024-04-06T13:49:03.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.280+0000 7f3fa1a8b700 1 --2- 172.21.15.189:0/3211213075 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3f7c06aed0 0x7f3f7c06d390 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.280+0000 7f3fa1a8b700 1 --2- 172.21.15.189:0/3211213075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c14eca0 0x7f3f9c0b0780 unknown :-1 s=CLOSED pgs=2851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.280+0000 7f3fa1a8b700 1 --2- 172.21.15.189:0/3211213075 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f9c0b4d70 0x7f3f9c1579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.280+0000 7f3fa1a8b700 1 --2- 172.21.15.189:0/3211213075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f9c0cf900 0x7f3f9c0aed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.280+0000 7f3fa1a8b700 1 -- 172.21.15.189:0/3211213075 >> 172.21.15.189:0/3211213075 conn(0x7f3f9c0d00b0 msgr2=0x7f3f9c0b3860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:03.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.280+0000 7f3fa1a8b700 1 -- 172.21.15.189:0/3211213075 shutdown_connections 2024-04-06T13:49:03.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.280+0000 7f3fa1a8b700 1 -- 172.21.15.189:0/3211213075 wait complete. 2024-04-06T13:49:03.293 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:49:03.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.413+0000 7f87ec7b8700 1 Processor -- start 2024-04-06T13:49:03.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.413+0000 7f87ec7b8700 1 -- start start 2024-04-06T13:49:03.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.414+0000 7f87ec7b8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87e415d7b0 0x7f87e40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:03.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.414+0000 7f87ec7b8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f87e406a730 0x7f87e406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:03.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.414+0000 7f87ec7b8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e40b3990 con 0x7f87e415ddf0 2024-04-06T13:49:03.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.414+0000 7f87ec7b8700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e406b0f0 con 0x7f87e406a730 2024-04-06T13:49:03.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.414+0000 7f87ec7b8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e40b3060 con 0x7f87e415d7b0 2024-04-06T13:49:03.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.414+0000 7f87e9d53700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f87e406a730 0x7f87e406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:03.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.414+0000 7f87ead55700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f87e415ddf0 0x7f87e415e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50452/0 (socket says 172.21.15.189:50452) 2024-04-06T13:49:03.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.414+0000 7f87ead55700 1 -- 172.21.15.189:0/232349335 learned_addr learned my addr 172.21.15.189:0/232349335 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:03.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.415+0000 7f87e9d53700 1 -- 172.21.15.189:0/232349335 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87e415d7b0 msgr2=0x7f87e40b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:03.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.415+0000 7f87e9d53700 1 --2- 172.21.15.189:0/232349335 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87e415d7b0 0x7f87e40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.415+0000 7f87e9d53700 1 -- 172.21.15.189:0/232349335 >> v1:172.21.15.189:6789/0 conn(0x7f87e415ddf0 legacy=0x7f87e415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:03.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.415+0000 7f87e9d53700 1 -- 172.21.15.189:0/232349335 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e40b3340 con 0x7f87e406a730 2024-04-06T13:49:03.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.415+0000 7f87e9552700 1 -- 172.21.15.189:0/232349335 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010518373 0 0) 0x7f87e40b3990 con 0x7f87e415ddf0 2024-04-06T13:49:03.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.415+0000 7f87e9d53700 1 --2- 172.21.15.189:0/232349335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e406a730 0x7f87e406abb0 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009840 tx=0x7f87e0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea7ab46891ded28 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:03.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.415+0000 7f87e9552700 1 -- 172.21.15.189:0/232349335 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0011830 con 0x7f87e406a730 2024-04-06T13:49:03.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.415+0000 7f87e9552700 1 -- 172.21.15.189:0/232349335 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e0011990 con 0x7f87e406a730 2024-04-06T13:49:03.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.415+0000 7f87e9552700 1 -- 172.21.15.189:0/232349335 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0011d20 con 0x7f87e406a730 2024-04-06T13:49:03.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.416+0000 7f87ec7b8700 1 -- 172.21.15.189:0/232349335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e406a730 msgr2=0x7f87e406abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:03.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.416+0000 7f87ec7b8700 1 --2- 172.21.15.189:0/232349335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e406a730 0x7f87e406abb0 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009840 tx=0x7f87e0009b50 comp rx=0 tx=0).stop 2024-04-06T13:49:03.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.416+0000 7f87ec7b8700 1 -- 172.21.15.189:0/232349335 shutdown_connections 2024-04-06T13:49:03.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.416+0000 7f87ec7b8700 1 --2- 172.21.15.189:0/232349335 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87e415d7b0 0x7f87e40b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.416+0000 7f87ec7b8700 1 --2- 172.21.15.189:0/232349335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e406a730 0x7f87e406abb0 unknown :-1 s=CLOSED pgs=2853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.416+0000 7f87ec7b8700 1 -- 172.21.15.189:0/232349335 >> 172.21.15.189:0/232349335 conn(0x7f87e40d00b0 msgr2=0x7f87e40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:03.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.416+0000 7f87ec7b8700 1 -- 172.21.15.189:0/232349335 shutdown_connections 2024-04-06T13:49:03.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.416+0000 7f87ec7b8700 1 -- 172.21.15.189:0/232349335 wait complete. 2024-04-06T13:49:03.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.417+0000 7f87ec7b8700 1 Processor -- start 2024-04-06T13:49:03.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.417+0000 7f87ec7b8700 1 -- start start 2024-04-06T13:49:03.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87ec7b8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87e406a730 0x7f87e414fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:03.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87ec7b8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87e415d7b0 0x7f87e415b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:03.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87ec7b8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e415ddf0 0x7f87e4071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:03.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87ec7b8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e415bb00 con 0x7f87e415ddf0 2024-04-06T13:49:03.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87ec7b8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e415bde0 con 0x7f87e415d7b0 2024-04-06T13:49:03.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87ec7b8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e415c0c0 con 0x7f87e406a730 2024-04-06T13:49:03.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87ead55700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e415ddf0 0x7f87e4071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:03.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87ead55700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e415ddf0 0x7f87e4071c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37266/0 (socket says 172.21.15.189:37266) 2024-04-06T13:49:03.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87ead55700 1 -- 172.21.15.189:0/1133011311 learned_addr learned my addr 172.21.15.189:0/1133011311 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:03.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87ea554700 1 --2- 172.21.15.189:0/1133011311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87e406a730 0x7f87e414fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:03.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87e9d53700 1 --2- 172.21.15.189:0/1133011311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87e415d7b0 0x7f87e415b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:03.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87ead55700 1 -- 172.21.15.189:0/1133011311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87e406a730 msgr2=0x7f87e414fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:03.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87ead55700 1 --2- 172.21.15.189:0/1133011311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87e406a730 0x7f87e414fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87ead55700 1 -- 172.21.15.189:0/1133011311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87e415d7b0 msgr2=0x7f87e415b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:03.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87ead55700 1 --2- 172.21.15.189:0/1133011311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87e415d7b0 0x7f87e415b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87ead55700 1 -- 172.21.15.189:0/1133011311 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e00094a0 con 0x7f87e415ddf0 2024-04-06T13:49:03.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87ead55700 1 --2- 172.21.15.189:0/1133011311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e415ddf0 0x7f87e4071c60 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00cba0 tx=0x7f87dc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:03.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87daffd700 1 -- 172.21.15.189:0/1133011311 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc00ab80 con 0x7f87e415ddf0 2024-04-06T13:49:03.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87daffd700 1 -- 172.21.15.189:0/1133011311 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87dc00ad00 con 0x7f87e415ddf0 2024-04-06T13:49:03.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87ec7b8700 1 -- 172.21.15.189:0/1133011311 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e415c3a0 con 0x7f87e415ddf0 2024-04-06T13:49:03.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.418+0000 7f87daffd700 1 -- 172.21.15.189:0/1133011311 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc007310 con 0x7f87e415ddf0 2024-04-06T13:49:03.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.419+0000 7f87ec7b8700 1 -- 172.21.15.189:0/1133011311 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e4150410 con 0x7f87e415ddf0 2024-04-06T13:49:03.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.419+0000 7f87ec7b8700 1 -- 172.21.15.189:0/1133011311 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e415c8c0 con 0x7f87e415ddf0 2024-04-06T13:49:03.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.420+0000 7f87daffd700 1 -- 172.21.15.189:0/1133011311 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f87dc007470 con 0x7f87e415ddf0 2024-04-06T13:49:03.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.420+0000 7f87daffd700 1 --2- 172.21.15.189:0/1133011311 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f87c806aed0 0x7f87c806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:03.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.420+0000 7f87daffd700 1 -- 172.21.15.189:0/1133011311 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6973+0+0 (secure 0 0 0) 0x7f87dc089690 con 0x7f87e415ddf0 2024-04-06T13:49:03.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.420+0000 7f87ea554700 1 --2- 172.21.15.189:0/1133011311 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f87c806aed0 0x7f87c806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:03.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.421+0000 7f87ea554700 1 --2- 172.21.15.189:0/1133011311 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f87c806aed0 0x7f87c806d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f87d4006fd0 tx=0x7f87d4008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:03.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.425+0000 7f87daffd700 1 -- 172.21.15.189:0/1133011311 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f87e415c8c0 con 0x7f87e415ddf0 2024-04-06T13:49:03.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.664+0000 7f87ec7b8700 1 -- 172.21.15.189:0/1133011311 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87e415c8c0 con 0x7f87e415ddf0 2024-04-06T13:49:03.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.664+0000 7f87daffd700 1 -- 172.21.15.189:0/1133011311 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+4757 (secure 0 0 0) 0x7f87e415c8c0 con 0x7f87e415ddf0 2024-04-06T13:49:03.665 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:03.665 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":124,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3323503154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3323503154},{"type":"v1","addr":"172.21.15.189:6836","nonce":3323503154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":9383,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3292345173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3292345173},{"type":"v1","addr":"172.21.15.189:6837","nonce":3292345173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":11284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/1490067369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":1490067369},{"type":"v1","addr":"172.21.15.192:6833","nonce":1490067369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:48:36.197004+0000","modified":"2024-04-06T13:48:39.206755+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9395},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9395":{"gid":9395,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/455737532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":455737532},{"type":"v1","addr":"172.21.15.192:6835","nonce":455737532}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-04-06T13:49:03.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.666+0000 7f87ec7b8700 1 -- 172.21.15.189:0/1133011311 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f87c806aed0 msgr2=0x7f87c806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:03.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.666+0000 7f87ec7b8700 1 --2- 172.21.15.189:0/1133011311 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f87c806aed0 0x7f87c806d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f87d4006fd0 tx=0x7f87d4008040 comp rx=0 tx=0).stop 2024-04-06T13:49:03.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.666+0000 7f87ec7b8700 1 -- 172.21.15.189:0/1133011311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e415ddf0 msgr2=0x7f87e4071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:03.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.666+0000 7f87ec7b8700 1 --2- 172.21.15.189:0/1133011311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e415ddf0 0x7f87e4071c60 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00cba0 tx=0x7f87dc00cf60 comp rx=0 tx=0).stop 2024-04-06T13:49:03.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.666+0000 7f87ec7b8700 1 -- 172.21.15.189:0/1133011311 shutdown_connections 2024-04-06T13:49:03.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.666+0000 7f87ec7b8700 1 --2- 172.21.15.189:0/1133011311 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f87c806aed0 0x7f87c806d390 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.667+0000 7f87ec7b8700 1 --2- 172.21.15.189:0/1133011311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e415ddf0 0x7f87e4071c60 unknown :-1 s=CLOSED pgs=2854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.667+0000 7f87ec7b8700 1 --2- 172.21.15.189:0/1133011311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87e406a730 0x7f87e414fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.667+0000 7f87ec7b8700 1 --2- 172.21.15.189:0/1133011311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87e415d7b0 0x7f87e415b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.667+0000 7f87ec7b8700 1 -- 172.21.15.189:0/1133011311 >> 172.21.15.189:0/1133011311 conn(0x7f87e40d00b0 msgr2=0x7f87e406fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:03.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.667+0000 7f87ec7b8700 1 -- 172.21.15.189:0/1133011311 shutdown_connections 2024-04-06T13:49:03.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.667+0000 7f87ec7b8700 1 -- 172.21.15.189:0/1133011311 wait complete. 2024-04-06T13:49:03.669 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 124 2024-04-06T13:49:03.680 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:49:03.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.801+0000 7fa06aedc700 1 Processor -- start 2024-04-06T13:49:03.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.801+0000 7fa06aedc700 1 -- start start 2024-04-06T13:49:03.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.802+0000 7fa06aedc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa064144f40 0x7fa06406a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:03.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.802+0000 7fa06aedc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa06415fcd0 0x7fa06406ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:03.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.802+0000 7fa06aedc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa06415ba40 0x7fa06415be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:03.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.802+0000 7fa06aedc700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa06415ca20 con 0x7fa06415fcd0 2024-04-06T13:49:03.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.802+0000 7fa06aedc700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0640d0f40 con 0x7fa064144f40 2024-04-06T13:49:03.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.802+0000 7fa06aedc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa06415c740 con 0x7fa06415ba40 2024-04-06T13:49:03.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.802+0000 7fa063fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa06415fcd0 0x7fa06406ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:03.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.802+0000 7fa063fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa06415fcd0 0x7fa06406ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37278/0 (socket says 172.21.15.189:37278) 2024-04-06T13:49:03.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.802+0000 7fa063fff700 1 -- 172.21.15.189:0/2362738198 learned_addr learned my addr 172.21.15.189:0/2362738198 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:03.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.802+0000 7fa069479700 1 --2- 172.21.15.189:0/2362738198 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa06415ba40 0x7fa06415be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:03.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.802+0000 7fa068c78700 1 --2- 172.21.15.189:0/2362738198 >> v2:172.21.15.192:3300/0 conn(0x7fa064144f40 0x7fa06406a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:03.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.802+0000 7fa063fff700 1 -- 172.21.15.189:0/2362738198 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa06415ba40 msgr2=0x7fa06415be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:03.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.802+0000 7fa063fff700 1 --2- 172.21.15.189:0/2362738198 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa06415ba40 0x7fa06415be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.802+0000 7fa063fff700 1 -- 172.21.15.189:0/2362738198 >> v2:172.21.15.192:3300/0 conn(0x7fa064144f40 msgr2=0x7fa06406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:03.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.802+0000 7fa063fff700 1 --2- 172.21.15.189:0/2362738198 >> v2:172.21.15.192:3300/0 conn(0x7fa064144f40 0x7fa06406a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.802+0000 7fa063fff700 1 -- 172.21.15.189:0/2362738198 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06415f1f0 con 0x7fa06415fcd0 2024-04-06T13:49:03.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.802+0000 7fa068c78700 1 --2- 172.21.15.189:0/2362738198 >> v2:172.21.15.192:3300/0 conn(0x7fa064144f40 0x7fa06406a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:03.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.802+0000 7fa063fff700 1 --2- 172.21.15.189:0/2362738198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa06415fcd0 0x7fa06406ac70 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7fa064073470 tx=0x7fa050009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fca5a66ecc10be60 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:03.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.803+0000 7fa0637fe700 1 -- 172.21.15.189:0/2362738198 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05000b870 con 0x7fa06415fcd0 2024-04-06T13:49:03.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.803+0000 7fa0637fe700 1 -- 172.21.15.189:0/2362738198 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05000b9d0 con 0x7fa06415fcd0 2024-04-06T13:49:03.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.803+0000 7fa0637fe700 1 -- 172.21.15.189:0/2362738198 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05000bd60 con 0x7fa06415fcd0 2024-04-06T13:49:03.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.803+0000 7fa06aedc700 1 -- 172.21.15.189:0/2362738198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa06415fcd0 msgr2=0x7fa06406ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:03.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.803+0000 7fa06aedc700 1 --2- 172.21.15.189:0/2362738198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa06415fcd0 0x7fa06406ac70 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7fa064073470 tx=0x7fa050009bd0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.803+0000 7fa06aedc700 1 -- 172.21.15.189:0/2362738198 shutdown_connections 2024-04-06T13:49:03.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.803+0000 7fa06aedc700 1 --2- 172.21.15.189:0/2362738198 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa06415ba40 0x7fa06415be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.803+0000 7fa06aedc700 1 --2- 172.21.15.189:0/2362738198 >> v2:172.21.15.192:3300/0 conn(0x7fa064144f40 0x7fa06406a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.803+0000 7fa06aedc700 1 --2- 172.21.15.189:0/2362738198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa06415fcd0 0x7fa06406ac70 unknown :-1 s=CLOSED pgs=2855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.803+0000 7fa06aedc700 1 -- 172.21.15.189:0/2362738198 >> 172.21.15.189:0/2362738198 conn(0x7fa0640d00b0 msgr2=0x7fa0640d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:03.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.803+0000 7fa06aedc700 1 -- 172.21.15.189:0/2362738198 shutdown_connections 2024-04-06T13:49:03.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.803+0000 7fa06aedc700 1 -- 172.21.15.189:0/2362738198 wait complete. 2024-04-06T13:49:03.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.804+0000 7fa06aedc700 1 Processor -- start 2024-04-06T13:49:03.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.805+0000 7fa06aedc700 1 -- start start 2024-04-06T13:49:03.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.805+0000 7fa06aedc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa064144f40 0x7fa064159870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:03.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.805+0000 7fa06aedc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa06415ba40 0x7fa0641f70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:03.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.805+0000 7fa06aedc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa06415fcd0 0x7fa06413d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:03.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.805+0000 7fa06aedc700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa064159db0 con 0x7fa06415fcd0 2024-04-06T13:49:03.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.805+0000 7fa06aedc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0641f7610 con 0x7fa06415ba40 2024-04-06T13:49:03.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.805+0000 7fa06aedc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa064159f50 con 0x7fa064144f40 2024-04-06T13:49:03.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.806+0000 7fa069479700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa06415fcd0 0x7fa06413d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:03.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.806+0000 7fa069479700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa06415fcd0 0x7fa06413d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37282/0 (socket says 172.21.15.189:37282) 2024-04-06T13:49:03.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.806+0000 7fa069479700 1 -- 172.21.15.189:0/3878890171 learned_addr learned my addr 172.21.15.189:0/3878890171 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:03.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.806+0000 7fa068c78700 1 --2- 172.21.15.189:0/3878890171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa064144f40 0x7fa064159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:03.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.806+0000 7fa063fff700 1 --2- 172.21.15.189:0/3878890171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa06415ba40 0x7fa0641f70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:03.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.806+0000 7fa069479700 1 -- 172.21.15.189:0/3878890171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa064144f40 msgr2=0x7fa064159870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:03.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.806+0000 7fa069479700 1 --2- 172.21.15.189:0/3878890171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa064144f40 0x7fa064159870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.806+0000 7fa069479700 1 -- 172.21.15.189:0/3878890171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa06415ba40 msgr2=0x7fa0641f70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:03.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.806+0000 7fa069479700 1 --2- 172.21.15.189:0/3878890171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa06415ba40 0x7fa0641f70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:03.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.806+0000 7fa069479700 1 -- 172.21.15.189:0/3878890171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa050009550 con 0x7fa06415fcd0 2024-04-06T13:49:03.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.806+0000 7fa069479700 1 --2- 172.21.15.189:0/3878890171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa06415fcd0 0x7fa06413d9d0 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7fa05c007e40 tx=0x7fa05c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:03.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.806+0000 7fa0617fa700 1 -- 172.21.15.189:0/3878890171 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05c00eea0 con 0x7fa06415fcd0 2024-04-06T13:49:03.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.806+0000 7fa0617fa700 1 -- 172.21.15.189:0/3878890171 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05c014420 con 0x7fa06415fcd0 2024-04-06T13:49:03.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.806+0000 7fa06aedc700 1 -- 172.21.15.189:0/3878890171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa06415b290 con 0x7fa06415fcd0 2024-04-06T13:49:03.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.806+0000 7fa0617fa700 1 -- 172.21.15.189:0/3878890171 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05c0147b0 con 0x7fa06415fcd0 2024-04-06T13:49:03.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.806+0000 7fa06aedc700 1 -- 172.21.15.189:0/3878890171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa06415a4d0 con 0x7fa06415fcd0 2024-04-06T13:49:03.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.808+0000 7fa06aedc700 1 -- 172.21.15.189:0/3878890171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa064067bb0 con 0x7fa06415fcd0 2024-04-06T13:49:03.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.808+0000 7fa0617fa700 1 -- 172.21.15.189:0/3878890171 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fa05c014910 con 0x7fa06415fcd0 2024-04-06T13:49:03.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.808+0000 7fa0617fa700 1 --2- 172.21.15.189:0/3878890171 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa04806aed0 0x7fa04806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:03.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.808+0000 7fa0617fa700 1 -- 172.21.15.189:0/3878890171 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6973+0+0 (secure 0 0 0) 0x7fa05c08a2f0 con 0x7fa06415fcd0 2024-04-06T13:49:03.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.808+0000 7fa068c78700 1 --2- 172.21.15.189:0/3878890171 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa04806aed0 0x7fa04806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:03.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.809+0000 7fa068c78700 1 --2- 172.21.15.189:0/3878890171 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa04806aed0 0x7fa04806d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa058006ba0 tx=0x7fa058008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:03.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:03.813+0000 7fa0617fa700 1 -- 172.21.15.189:0/3878890171 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa05c054de0 con 0x7fa06415fcd0 2024-04-06T13:49:04.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.010+0000 7fa06aedc700 1 -- 172.21.15.189:0/3878890171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0640cd710 con 0x7fa06415fcd0 2024-04-06T13:49:04.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.012+0000 7fa0617fa700 1 -- 172.21.15.189:0/3878890171 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v257) v1 ==== 74+0+17683 (secure 0 0 0) 0x7fa05c058400 con 0x7fa06415fcd0 2024-04-06T13:49:04.013 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:04.013 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":257,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:49:01.853081+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:48:33.912100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:48:35.170642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.192:6833/726816746":"2024-04-07T13:48:58.736634+0000","172.21.15.192:6832/726816746":"2024-04-07T13:48:58.736634+0000","172.21.15.189:0/1050000893":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/2601670651":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/3142793643":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/1266339172":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/3326966043":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/4043705133":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/85704189":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2120970213":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/381758424":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2635472538":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/2107771319":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2145696658":"2024-04-07T13:48:48.812402+0000","172.21.15.189:6833/111704":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/3440291825":"2024-04-07T13:48:48.812402+0000","172.21.15.189:6832/111704":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/598265929":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/485512536":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/643921686":"2024-04-07T13:48:48.812402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:49:04.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.014+0000 7fa06aedc700 1 -- 172.21.15.189:0/3878890171 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa04806aed0 msgr2=0x7fa04806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:04.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.014+0000 7fa06aedc700 1 --2- 172.21.15.189:0/3878890171 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa04806aed0 0x7fa04806d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa058006ba0 tx=0x7fa058008040 comp rx=0 tx=0).stop 2024-04-06T13:49:04.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.014+0000 7fa06aedc700 1 -- 172.21.15.189:0/3878890171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa06415fcd0 msgr2=0x7fa06413d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:04.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.014+0000 7fa06aedc700 1 --2- 172.21.15.189:0/3878890171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa06415fcd0 0x7fa06413d9d0 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7fa05c007e40 tx=0x7fa05c00e3c0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.014+0000 7fa06aedc700 1 -- 172.21.15.189:0/3878890171 shutdown_connections 2024-04-06T13:49:04.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.014+0000 7fa06aedc700 1 --2- 172.21.15.189:0/3878890171 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa04806aed0 0x7fa04806d390 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.014+0000 7fa06aedc700 1 --2- 172.21.15.189:0/3878890171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa06415fcd0 0x7fa06413d9d0 unknown :-1 s=CLOSED pgs=2856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.014+0000 7fa06aedc700 1 --2- 172.21.15.189:0/3878890171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa064144f40 0x7fa064159870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.014+0000 7fa06aedc700 1 --2- 172.21.15.189:0/3878890171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa06415ba40 0x7fa0641f70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.014+0000 7fa06aedc700 1 -- 172.21.15.189:0/3878890171 >> 172.21.15.189:0/3878890171 conn(0x7fa0640d00b0 msgr2=0x7fa0640cf5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:04.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.014+0000 7fa06aedc700 1 -- 172.21.15.189:0/3878890171 shutdown_connections 2024-04-06T13:49:04.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.014+0000 7fa06aedc700 1 -- 172.21.15.189:0/3878890171 wait complete. 2024-04-06T13:49:04.028 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2024-04-06T13:49:04.028 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:49:04.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.148+0000 7f9d0f191700 1 Processor -- start 2024-04-06T13:49:04.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.149+0000 7f9d0f191700 1 -- start start 2024-04-06T13:49:04.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d0f191700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d0806ae40 0x7f9d0815bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d0f191700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9d0815c100 0x7f9d080ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d0f191700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0815f9e0 con 0x7f9d08158fe0 2024-04-06T13:49:04.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d0f191700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d0815f0d0 con 0x7f9d0815c100 2024-04-06T13:49:04.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d0f191700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0815c8f0 con 0x7f9d0806ae40 2024-04-06T13:49:04.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d0cf2d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9d08158fe0 0x7f9d0806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50466/0 (socket says 172.21.15.189:50466) 2024-04-06T13:49:04.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d0cf2d700 1 -- 172.21.15.189:0/2318022651 learned_addr learned my addr 172.21.15.189:0/2318022651 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:04.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d077fe700 1 -- 172.21.15.189:0/2318022651 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280141463 0 0) 0x7f9d0815f9e0 con 0x7f9d08158fe0 2024-04-06T13:49:04.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d077fe700 1 -- 172.21.15.189:0/2318022651 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cf80035b0 con 0x7f9d08158fe0 2024-04-06T13:49:04.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d0d72e700 1 --2- 172.21.15.189:0/2318022651 >> v2:172.21.15.192:3300/0 conn(0x7f9d0815c100 0x7f9d080ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:04.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d07fff700 1 --2- 172.21.15.189:0/2318022651 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d0806ae40 0x7f9d0815bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:04.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d077fe700 1 -- 172.21.15.189:0/2318022651 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3781234373 0 0) 0x7f9cf80035b0 con 0x7f9d08158fe0 2024-04-06T13:49:04.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d077fe700 1 -- 172.21.15.189:0/2318022651 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d0806ae40 msgr2=0x7f9d0815bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:04.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d077fe700 1 --2- 172.21.15.189:0/2318022651 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d0806ae40 0x7f9d0815bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d077fe700 1 -- 172.21.15.189:0/2318022651 >> v2:172.21.15.192:3300/0 conn(0x7f9d0815c100 msgr2=0x7f9d080ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:04.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d077fe700 1 --2- 172.21.15.189:0/2318022651 >> v2:172.21.15.192:3300/0 conn(0x7f9d0815c100 0x7f9d080ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d077fe700 1 -- 172.21.15.189:0/2318022651 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0815f390 con 0x7f9d08158fe0 2024-04-06T13:49:04.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d077fe700 1 -- 172.21.15.189:0/2318022651 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f9cfc001670 con 0x7f9d08158fe0 2024-04-06T13:49:04.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d077fe700 1 -- 172.21.15.189:0/2318022651 >> v1:172.21.15.189:6789/0 conn(0x7f9d08158fe0 legacy=0x7f9d0806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:04.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d077fe700 1 --2- 172.21.15.189:0/2318022651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9cf8003a00 0x7f9cf8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d077fe700 1 --2- 172.21.15.189:0/2318022651 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d0806ae40 0x7f9cf8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d077fe700 1 --2- 172.21.15.189:0/2318022651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cf8004d90 0x7f9cf8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d077fe700 1 -- 172.21.15.189:0/2318022651 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0815f0d0 con 0x7f9cf8004d90 2024-04-06T13:49:04.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d077fe700 1 -- 172.21.15.189:0/2318022651 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0815c8f0 con 0x7f9cf8003a00 2024-04-06T13:49:04.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.150+0000 7f9d077fe700 1 -- 172.21.15.189:0/2318022651 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf800ba10 con 0x7f9d0806ae40 2024-04-06T13:49:04.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.151+0000 7f9d0d72e700 1 --2- 172.21.15.189:0/2318022651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cf8004d90 0x7f9cf8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:04.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.151+0000 7f9d07fff700 1 --2- 172.21.15.189:0/2318022651 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d0806ae40 0x7f9cf8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:04.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.151+0000 7f9d0cf2d700 1 --2- 172.21.15.189:0/2318022651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9cf8003a00 0x7f9cf8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:04.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.151+0000 7f9d0d72e700 1 -- 172.21.15.189:0/2318022651 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d0806ae40 msgr2=0x7f9cf8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:04.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.151+0000 7f9d0d72e700 1 --2- 172.21.15.189:0/2318022651 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d0806ae40 0x7f9cf8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.151+0000 7f9d0d72e700 1 -- 172.21.15.189:0/2318022651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9cf8003a00 msgr2=0x7f9cf8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:04.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.151+0000 7f9d0d72e700 1 --2- 172.21.15.189:0/2318022651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9cf8003a00 0x7f9cf8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.151+0000 7f9d0d72e700 1 -- 172.21.15.189:0/2318022651 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf80035b0 con 0x7f9cf8004d90 2024-04-06T13:49:04.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.151+0000 7f9d0d72e700 1 --2- 172.21.15.189:0/2318022651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cf8004d90 0x7f9cf8005210 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7f9d00009fc0 tx=0x7f9cf8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc20640f74b8134 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:04.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.151+0000 7f9d077fe700 1 -- 172.21.15.189:0/2318022651 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0000ec30 con 0x7f9cf8004d90 2024-04-06T13:49:04.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.151+0000 7f9d077fe700 1 -- 172.21.15.189:0/2318022651 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d0000f040 con 0x7f9cf8004d90 2024-04-06T13:49:04.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.151+0000 7f9d077fe700 1 -- 172.21.15.189:0/2318022651 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00014480 con 0x7f9cf8004d90 2024-04-06T13:49:04.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.152+0000 7f9d0f191700 1 -- 172.21.15.189:0/2318022651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cf8004d90 msgr2=0x7f9cf8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:04.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.152+0000 7f9d0f191700 1 --2- 172.21.15.189:0/2318022651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cf8004d90 0x7f9cf8005210 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7f9d00009fc0 tx=0x7f9cf8005880 comp rx=0 tx=0).stop 2024-04-06T13:49:04.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.152+0000 7f9d0d72e700 1 -- 172.21.15.189:0/2318022651 reap_dead start 2024-04-06T13:49:04.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.152+0000 7f9d0f191700 1 -- 172.21.15.189:0/2318022651 shutdown_connections 2024-04-06T13:49:04.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.152+0000 7f9d0f191700 1 -- 172.21.15.189:0/2318022651 >> 172.21.15.189:0/2318022651 conn(0x7f9d080d00b0 msgr2=0x7f9d080d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:04.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.152+0000 7f9d0f191700 1 -- 172.21.15.189:0/2318022651 shutdown_connections 2024-04-06T13:49:04.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.152+0000 7f9d0f191700 1 -- 172.21.15.189:0/2318022651 wait complete. 2024-04-06T13:49:04.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.153+0000 7f9d0f191700 1 Processor -- start 2024-04-06T13:49:04.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.153+0000 7f9d0f191700 1 -- start start 2024-04-06T13:49:04.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.154+0000 7f9d0f191700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d08157d80 0x7f9d080c5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.154+0000 7f9d0f191700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d080717b0 0x7f9d08071c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.154+0000 7f9d0f191700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d0815e120 0x7f9d0815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.154+0000 7f9d0f191700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d08072170 con 0x7f9d08157d80 2024-04-06T13:49:04.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.154+0000 7f9d0f191700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d080c4f30 con 0x7f9d0815e120 2024-04-06T13:49:04.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.154+0000 7f9d0f191700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d080c5210 con 0x7f9d080717b0 2024-04-06T13:49:04.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.154+0000 7f9d0cf2d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d08157d80 0x7f9d080c5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:04.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.154+0000 7f9d0cf2d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d08157d80 0x7f9d080c5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37294/0 (socket says 172.21.15.189:37294) 2024-04-06T13:49:04.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.154+0000 7f9d0cf2d700 1 -- 172.21.15.189:0/3859938863 learned_addr learned my addr 172.21.15.189:0/3859938863 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:04.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.154+0000 7f9d0d72e700 1 --2- 172.21.15.189:0/3859938863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d0815e120 0x7f9d0815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:04.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.154+0000 7f9d07fff700 1 --2- 172.21.15.189:0/3859938863 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d080717b0 0x7f9d08071c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:04.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.154+0000 7f9d0cf2d700 1 -- 172.21.15.189:0/3859938863 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d080717b0 msgr2=0x7f9d08071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:04.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.154+0000 7f9d0cf2d700 1 --2- 172.21.15.189:0/3859938863 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d080717b0 0x7f9d08071c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.154+0000 7f9d0cf2d700 1 -- 172.21.15.189:0/3859938863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d0815e120 msgr2=0x7f9d0815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:04.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.154+0000 7f9d0cf2d700 1 --2- 172.21.15.189:0/3859938863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d0815e120 0x7f9d0815e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.154+0000 7f9d0cf2d700 1 -- 172.21.15.189:0/3859938863 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d080c54f0 con 0x7f9d08157d80 2024-04-06T13:49:04.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.154+0000 7f9d0cf2d700 1 --2- 172.21.15.189:0/3859938863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d08157d80 0x7f9d080c5fd0 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc001a30 tx=0x7f9cfc0076a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:04.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.154+0000 7f9d057fa700 1 -- 172.21.15.189:0/3859938863 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc00f8a0 con 0x7f9d08157d80 2024-04-06T13:49:04.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.155+0000 7f9d057fa700 1 -- 172.21.15.189:0/3859938863 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cfc00fa40 con 0x7f9d08157d80 2024-04-06T13:49:04.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.155+0000 7f9d0f191700 1 -- 172.21.15.189:0/3859938863 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d080c57d0 con 0x7f9d08157d80 2024-04-06T13:49:04.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.155+0000 7f9d057fa700 1 -- 172.21.15.189:0/3859938863 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc00fdf0 con 0x7f9d08157d80 2024-04-06T13:49:04.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.155+0000 7f9d0f191700 1 -- 172.21.15.189:0/3859938863 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d0815eae0 con 0x7f9d08157d80 2024-04-06T13:49:04.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.156+0000 7f9d0f191700 1 -- 172.21.15.189:0/3859938863 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d08074990 con 0x7f9d08157d80 2024-04-06T13:49:04.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.156+0000 7f9d057fa700 1 -- 172.21.15.189:0/3859938863 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f9cfc01d460 con 0x7f9d08157d80 2024-04-06T13:49:04.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.156+0000 7f9d057fa700 1 --2- 172.21.15.189:0/3859938863 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9cec06aed0 0x7f9cec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.157+0000 7f9d057fa700 1 -- 172.21.15.189:0/3859938863 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6973+0+0 (secure 0 0 0) 0x7f9cfc089d70 con 0x7f9d08157d80 2024-04-06T13:49:04.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.157+0000 7f9d07fff700 1 --2- 172.21.15.189:0/3859938863 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9cec06aed0 0x7f9cec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:04.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.157+0000 7f9d07fff700 1 --2- 172.21.15.189:0/3859938863 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9cec06aed0 0x7f9cec06d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9d0806aaa0 tx=0x7f9cf4008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:04.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.161+0000 7f9d057fa700 1 -- 172.21.15.189:0/3859938863 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9cfc054700 con 0x7f9d08157d80 2024-04-06T13:49:04.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.399+0000 7f9d0f191700 1 -- 172.21.15.189:0/3859938863 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d080ce040 con 0x7f9d08157d80 2024-04-06T13:49:04.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.400+0000 7f9d057fa700 1 -- 172.21.15.189:0/3859938863 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+4757 (secure 0 0 0) 0x7f9cfc007b60 con 0x7f9d08157d80 2024-04-06T13:49:04.401 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:04.401 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":124,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3323503154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3323503154},{"type":"v1","addr":"172.21.15.189:6836","nonce":3323503154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":9383,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3292345173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3292345173},{"type":"v1","addr":"172.21.15.189:6837","nonce":3292345173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":11284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/1490067369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":1490067369},{"type":"v1","addr":"172.21.15.192:6833","nonce":1490067369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:48:36.197004+0000","modified":"2024-04-06T13:48:39.206755+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9395},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9395":{"gid":9395,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/455737532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":455737532},{"type":"v1","addr":"172.21.15.192:6835","nonce":455737532}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-04-06T13:49:04.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.402+0000 7f9d0f191700 1 -- 172.21.15.189:0/3859938863 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9cec06aed0 msgr2=0x7f9cec06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:04.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.402+0000 7f9d0f191700 1 --2- 172.21.15.189:0/3859938863 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9cec06aed0 0x7f9cec06d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9d0806aaa0 tx=0x7f9cf4008040 comp rx=0 tx=0).stop 2024-04-06T13:49:04.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.402+0000 7f9d0f191700 1 -- 172.21.15.189:0/3859938863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d08157d80 msgr2=0x7f9d080c5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:04.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.402+0000 7f9d0f191700 1 --2- 172.21.15.189:0/3859938863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d08157d80 0x7f9d080c5fd0 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc001a30 tx=0x7f9cfc0076a0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.402+0000 7f9d0f191700 1 -- 172.21.15.189:0/3859938863 shutdown_connections 2024-04-06T13:49:04.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.402+0000 7f9d0f191700 1 --2- 172.21.15.189:0/3859938863 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9cec06aed0 0x7f9cec06d390 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.402+0000 7f9d0f191700 1 --2- 172.21.15.189:0/3859938863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d0815e120 0x7f9d0815e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.402+0000 7f9d0f191700 1 --2- 172.21.15.189:0/3859938863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d08157d80 0x7f9d080c5fd0 unknown :-1 s=CLOSED pgs=2859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.402+0000 7f9d0f191700 1 --2- 172.21.15.189:0/3859938863 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d080717b0 0x7f9d08071c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.402+0000 7f9d0f191700 1 -- 172.21.15.189:0/3859938863 >> 172.21.15.189:0/3859938863 conn(0x7f9d080d00b0 msgr2=0x7f9d0806f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:04.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.402+0000 7f9d0f191700 1 -- 172.21.15.189:0/3859938863 shutdown_connections 2024-04-06T13:49:04.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.402+0000 7f9d0f191700 1 -- 172.21.15.189:0/3859938863 wait complete. 2024-04-06T13:49:04.405 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 124 2024-04-06T13:49:04.416 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:49:04.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.536+0000 7fa54b51c700 1 Processor -- start 2024-04-06T13:49:04.538 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.537+0000 7fa54b51c700 1 -- start start 2024-04-06T13:49:04.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.537+0000 7fa54b51c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa5440cfa00 0x7fa5440cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.537+0000 7fa54b51c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5440cbd80 con 0x7fa5440d11b0 2024-04-06T13:49:04.540 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.537+0000 7fa54b51c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5440cef20 con 0x7fa5440cb870 2024-04-06T13:49:04.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.537+0000 7fa54b51c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5440ccdc0 con 0x7fa5440cfa00 2024-04-06T13:49:04.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.538+0000 7fa549ab9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa5440cfa00 0x7fa5440cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:04.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.538+0000 7fa5492b8700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa5440d11b0 0x7fa5440d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50474/0 (socket says 172.21.15.189:50474) 2024-04-06T13:49:04.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.538+0000 7fa549ab9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa5440cfa00 0x7fa5440cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37296/0 (socket says 172.21.15.189:37296) 2024-04-06T13:49:04.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.538+0000 7fa5492b8700 1 -- 172.21.15.189:0/3318463904 learned_addr learned my addr 172.21.15.189:0/3318463904 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:04.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.538+0000 7fa549ab9700 1 -- 172.21.15.189:0/3318463904 >> v1:172.21.15.192:6789/0 conn(0x7fa5440cb870 legacy=0x7fa54415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:04.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.538+0000 7fa549ab9700 1 -- 172.21.15.189:0/3318463904 >> v1:172.21.15.189:6789/0 conn(0x7fa5440d11b0 legacy=0x7fa5440d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:04.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.538+0000 7fa549ab9700 1 -- 172.21.15.189:0/3318463904 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5440cf200 con 0x7fa5440cfa00 2024-04-06T13:49:04.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.538+0000 7fa53bfff700 1 -- 172.21.15.189:0/3318463904 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733119362 0 0) 0x7fa5440cbd80 con 0x7fa5440d11b0 2024-04-06T13:49:04.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.538+0000 7fa549ab9700 1 --2- 172.21.15.189:0/3318463904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5440cfa00 0x7fa5440cd560 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7fa53c009d90 tx=0x7fa53c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae6b06a5af595c70 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:04.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.538+0000 7fa53bfff700 1 -- 172.21.15.189:0/3318463904 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53c00e070 con 0x7fa5440cfa00 2024-04-06T13:49:04.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.538+0000 7fa53bfff700 1 -- 172.21.15.189:0/3318463904 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa53c00bbd0 con 0x7fa5440cfa00 2024-04-06T13:49:04.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.538+0000 7fa53bfff700 1 -- 172.21.15.189:0/3318463904 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53c012430 con 0x7fa5440cfa00 2024-04-06T13:49:04.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.539+0000 7fa54b51c700 1 -- 172.21.15.189:0/3318463904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5440cfa00 msgr2=0x7fa5440cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:04.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.539+0000 7fa54b51c700 1 --2- 172.21.15.189:0/3318463904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5440cfa00 0x7fa5440cd560 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7fa53c009d90 tx=0x7fa53c00b330 comp rx=0 tx=0).stop 2024-04-06T13:49:04.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.539+0000 7fa54b51c700 1 -- 172.21.15.189:0/3318463904 shutdown_connections 2024-04-06T13:49:04.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.539+0000 7fa54b51c700 1 --2- 172.21.15.189:0/3318463904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5440cfa00 0x7fa5440cd560 unknown :-1 s=CLOSED pgs=2861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.539+0000 7fa54b51c700 1 -- 172.21.15.189:0/3318463904 >> 172.21.15.189:0/3318463904 conn(0x7fa5440b2db0 msgr2=0x7fa5440b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:04.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.539+0000 7fa54b51c700 1 -- 172.21.15.189:0/3318463904 shutdown_connections 2024-04-06T13:49:04.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.539+0000 7fa54b51c700 1 -- 172.21.15.189:0/3318463904 wait complete. 2024-04-06T13:49:04.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.540+0000 7fa54b51c700 1 Processor -- start 2024-04-06T13:49:04.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.540+0000 7fa54b51c700 1 -- start start 2024-04-06T13:49:04.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa54b51c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5440cb870 0x7fa544155560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa54b51c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5440cfa00 0x7fa5440b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa54b51c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5440d11b0 0x7fa5440b5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa54b51c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5441f0ef0 con 0x7fa5440cb870 2024-04-06T13:49:04.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa54b51c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5440ca640 con 0x7fa5440d11b0 2024-04-06T13:49:04.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa54b51c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5440ca920 con 0x7fa5440cfa00 2024-04-06T13:49:04.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa5492b8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5440cb870 0x7fa544155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:04.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa5492b8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5440cb870 0x7fa544155560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37300/0 (socket says 172.21.15.189:37300) 2024-04-06T13:49:04.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa5492b8700 1 -- 172.21.15.189:0/2807285012 learned_addr learned my addr 172.21.15.189:0/2807285012 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:04.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa549ab9700 1 --2- 172.21.15.189:0/2807285012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5440d11b0 0x7fa5440b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:04.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa548ab7700 1 --2- 172.21.15.189:0/2807285012 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5440cfa00 0x7fa5440b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:04.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa5492b8700 1 -- 172.21.15.189:0/2807285012 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5440cfa00 msgr2=0x7fa5440b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:04.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa5492b8700 1 --2- 172.21.15.189:0/2807285012 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5440cfa00 0x7fa5440b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa5492b8700 1 -- 172.21.15.189:0/2807285012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5440d11b0 msgr2=0x7fa5440b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:04.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa5492b8700 1 --2- 172.21.15.189:0/2807285012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5440d11b0 0x7fa5440b5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa5492b8700 1 -- 172.21.15.189:0/2807285012 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53c009a70 con 0x7fa5440cb870 2024-04-06T13:49:04.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa5492b8700 1 --2- 172.21.15.189:0/2807285012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5440cb870 0x7fa544155560 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7fa53400c520 tx=0x7fa53400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:04.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa539ffb700 1 -- 172.21.15.189:0/2807285012 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa534004c80 con 0x7fa5440cb870 2024-04-06T13:49:04.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa539ffb700 1 -- 172.21.15.189:0/2807285012 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa534004e00 con 0x7fa5440cb870 2024-04-06T13:49:04.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa54b51c700 1 -- 172.21.15.189:0/2807285012 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5440cac00 con 0x7fa5440cb870 2024-04-06T13:49:04.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.541+0000 7fa539ffb700 1 -- 172.21.15.189:0/2807285012 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa534007310 con 0x7fa5440cb870 2024-04-06T13:49:04.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.542+0000 7fa54b51c700 1 -- 172.21.15.189:0/2807285012 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa544155aa0 con 0x7fa5440cb870 2024-04-06T13:49:04.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.543+0000 7fa54b51c700 1 -- 172.21.15.189:0/2807285012 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5440cb120 con 0x7fa5440cb870 2024-04-06T13:49:04.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.543+0000 7fa539ffb700 1 -- 172.21.15.189:0/2807285012 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fa534007470 con 0x7fa5440cb870 2024-04-06T13:49:04.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.543+0000 7fa539ffb700 1 --2- 172.21.15.189:0/2807285012 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa52806aed0 0x7fa52806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.543+0000 7fa539ffb700 1 -- 172.21.15.189:0/2807285012 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6973+0+0 (secure 0 0 0) 0x7fa534089860 con 0x7fa5440cb870 2024-04-06T13:49:04.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.544+0000 7fa548ab7700 1 --2- 172.21.15.189:0/2807285012 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa52806aed0 0x7fa52806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:04.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.544+0000 7fa548ab7700 1 --2- 172.21.15.189:0/2807285012 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa52806aed0 0x7fa52806d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa5440b64c0 tx=0x7fa540009040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:04.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.548+0000 7fa539ffb700 1 -- 172.21.15.189:0/2807285012 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa5440cb120 con 0x7fa5440cb870 2024-04-06T13:49:04.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.786+0000 7fa54b51c700 1 -- 172.21.15.189:0/2807285012 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5440cb120 con 0x7fa5440cb870 2024-04-06T13:49:04.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.786+0000 7fa539ffb700 1 -- 172.21.15.189:0/2807285012 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+4757 (secure 0 0 0) 0x7fa5440cb120 con 0x7fa5440cb870 2024-04-06T13:49:04.787 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:04.787 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":124,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3323503154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3323503154},{"type":"v1","addr":"172.21.15.189:6836","nonce":3323503154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":9383,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3292345173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3292345173},{"type":"v1","addr":"172.21.15.189:6837","nonce":3292345173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":11284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/1490067369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":1490067369},{"type":"v1","addr":"172.21.15.192:6833","nonce":1490067369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:48:36.197004+0000","modified":"2024-04-06T13:48:39.206755+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9395},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9395":{"gid":9395,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/455737532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":455737532},{"type":"v1","addr":"172.21.15.192:6835","nonce":455737532}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-04-06T13:49:04.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.788+0000 7fa54b51c700 1 -- 172.21.15.189:0/2807285012 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa52806aed0 msgr2=0x7fa52806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:04.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.788+0000 7fa54b51c700 1 --2- 172.21.15.189:0/2807285012 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa52806aed0 0x7fa52806d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa5440b64c0 tx=0x7fa540009040 comp rx=0 tx=0).stop 2024-04-06T13:49:04.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.788+0000 7fa54b51c700 1 -- 172.21.15.189:0/2807285012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5440cb870 msgr2=0x7fa544155560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:04.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.788+0000 7fa54b51c700 1 --2- 172.21.15.189:0/2807285012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5440cb870 0x7fa544155560 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7fa53400c520 tx=0x7fa53400c830 comp rx=0 tx=0).stop 2024-04-06T13:49:04.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.788+0000 7fa54b51c700 1 -- 172.21.15.189:0/2807285012 shutdown_connections 2024-04-06T13:49:04.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.788+0000 7fa54b51c700 1 --2- 172.21.15.189:0/2807285012 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa52806aed0 0x7fa52806d390 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.788+0000 7fa54b51c700 1 --2- 172.21.15.189:0/2807285012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5440d11b0 0x7fa5440b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.788+0000 7fa54b51c700 1 --2- 172.21.15.189:0/2807285012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5440cb870 0x7fa544155560 unknown :-1 s=CLOSED pgs=2862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.788+0000 7fa54b51c700 1 --2- 172.21.15.189:0/2807285012 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5440cfa00 0x7fa5440b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.788+0000 7fa54b51c700 1 -- 172.21.15.189:0/2807285012 >> 172.21.15.189:0/2807285012 conn(0x7fa5440b2db0 msgr2=0x7fa5440d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:04.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.788+0000 7fa54b51c700 1 -- 172.21.15.189:0/2807285012 shutdown_connections 2024-04-06T13:49:04.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.788+0000 7fa54b51c700 1 -- 172.21.15.189:0/2807285012 wait complete. 2024-04-06T13:49:04.791 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 124 2024-04-06T13:49:04.802 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:49:04.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.924+0000 7f26a0222700 1 Processor -- start 2024-04-06T13:49:04.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.924+0000 7f26a0222700 1 -- start start 2024-04-06T13:49:04.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.925+0000 7f26a0222700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f269815e8a0 0x7f26980cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.925+0000 7f26a0222700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f269815fa70 0x7f269806aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.925+0000 7f26a0222700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26980669d0 con 0x7f269815eee0 2024-04-06T13:49:04.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.925+0000 7f26a0222700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f269806ede0 con 0x7f269815fa70 2024-04-06T13:49:04.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.925+0000 7f26a0222700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f269815ec80 con 0x7f269815e8a0 2024-04-06T13:49:04.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.925+0000 7f269e7bf700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f269815fa70 0x7f269806aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:04.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.925+0000 7f269d7bd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f269815eee0 0x7f269815f360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50480/0 (socket says 172.21.15.189:50480) 2024-04-06T13:49:04.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.925+0000 7f269d7bd700 1 -- 172.21.15.189:0/3574348461 learned_addr learned my addr 172.21.15.189:0/3574348461 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:04.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.925+0000 7f269e7bf700 1 -- 172.21.15.189:0/3574348461 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f269815e8a0 msgr2=0x7f26980cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:04.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.925+0000 7f269e7bf700 1 --2- 172.21.15.189:0/3574348461 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f269815e8a0 0x7f26980cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.925+0000 7f269e7bf700 1 -- 172.21.15.189:0/3574348461 >> v1:172.21.15.189:6789/0 conn(0x7f269815eee0 legacy=0x7f269815f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:04.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.925+0000 7f269e7bf700 1 -- 172.21.15.189:0/3574348461 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26980666f0 con 0x7f269815fa70 2024-04-06T13:49:04.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.925+0000 7f269cfbc700 1 -- 172.21.15.189:0/3574348461 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730490474 0 0) 0x7f26980669d0 con 0x7f269815eee0 2024-04-06T13:49:04.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.926+0000 7f269e7bf700 1 --2- 172.21.15.189:0/3574348461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f269815fa70 0x7f269806aa10 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7f2690009dc0 tx=0x7f269000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3d3d7fbe2c62f16 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:04.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.926+0000 7f269cfbc700 1 -- 172.21.15.189:0/3574348461 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269000e070 con 0x7f269815fa70 2024-04-06T13:49:04.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.926+0000 7f269cfbc700 1 -- 172.21.15.189:0/3574348461 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269000bb90 con 0x7f269815fa70 2024-04-06T13:49:04.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.926+0000 7f269cfbc700 1 -- 172.21.15.189:0/3574348461 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690012430 con 0x7f269815fa70 2024-04-06T13:49:04.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.926+0000 7f26a0222700 1 -- 172.21.15.189:0/3574348461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f269815fa70 msgr2=0x7f269806aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:04.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.926+0000 7f26a0222700 1 --2- 172.21.15.189:0/3574348461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f269815fa70 0x7f269806aa10 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7f2690009dc0 tx=0x7f269000b330 comp rx=0 tx=0).stop 2024-04-06T13:49:04.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.926+0000 7f26a0222700 1 -- 172.21.15.189:0/3574348461 shutdown_connections 2024-04-06T13:49:04.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.926+0000 7f26a0222700 1 --2- 172.21.15.189:0/3574348461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f269815fa70 0x7f269806aa10 unknown :-1 s=CLOSED pgs=2864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.926+0000 7f26a0222700 1 --2- 172.21.15.189:0/3574348461 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f269815e8a0 0x7f26980cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.926+0000 7f26a0222700 1 -- 172.21.15.189:0/3574348461 >> 172.21.15.189:0/3574348461 conn(0x7f26980cc290 msgr2=0x7f26980cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:04.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.926+0000 7f26a0222700 1 -- 172.21.15.189:0/3574348461 shutdown_connections 2024-04-06T13:49:04.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.926+0000 7f26a0222700 1 -- 172.21.15.189:0/3574348461 wait complete. 2024-04-06T13:49:04.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.927+0000 7f26a0222700 1 Processor -- start 2024-04-06T13:49:04.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.928+0000 7f26a0222700 1 -- start start 2024-04-06T13:49:04.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.928+0000 7f26a0222700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f269815e8a0 0x7f26981570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.928+0000 7f26a0222700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f269815eee0 0x7f26980b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.928+0000 7f26a0222700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f269815fa70 0x7f2698158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.928+0000 7f26a0222700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26980c3690 con 0x7f269815e8a0 2024-04-06T13:49:04.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.928+0000 7f26a0222700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26980b38c0 con 0x7f269815fa70 2024-04-06T13:49:04.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.928+0000 7f26a0222700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26980b3ba0 con 0x7f269815eee0 2024-04-06T13:49:04.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.928+0000 7f269dfbe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f269815e8a0 0x7f26981570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:04.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.928+0000 7f269dfbe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f269815e8a0 0x7f26981570b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37314/0 (socket says 172.21.15.189:37314) 2024-04-06T13:49:04.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.928+0000 7f269dfbe700 1 -- 172.21.15.189:0/1743478137 learned_addr learned my addr 172.21.15.189:0/1743478137 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:04.936 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.929+0000 7f269dfbe700 1 -- 172.21.15.189:0/1743478137 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f269815eee0 msgr2=0x7f26980b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.929+0000 7f269d7bd700 1 --2- 172.21.15.189:0/1743478137 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f269815eee0 0x7f26980b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.929+0000 7f269e7bf700 1 --2- 172.21.15.189:0/1743478137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f269815fa70 0x7f2698158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.929+0000 7f269dfbe700 1 --2- 172.21.15.189:0/1743478137 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f269815eee0 0x7f26980b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.929+0000 7f269dfbe700 1 -- 172.21.15.189:0/1743478137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f269815fa70 msgr2=0x7f2698158d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.929+0000 7f269dfbe700 1 --2- 172.21.15.189:0/1743478137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f269815fa70 0x7f2698158d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.929+0000 7f269dfbe700 1 -- 172.21.15.189:0/1743478137 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2690009a70 con 0x7f269815e8a0 2024-04-06T13:49:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.929+0000 7f269dfbe700 1 --2- 172.21.15.189:0/1743478137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f269815e8a0 0x7f26981570b0 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7f269400b530 tx=0x7f269400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.929+0000 7f268e7fc700 1 -- 172.21.15.189:0/1743478137 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2694011830 con 0x7f269815e8a0 2024-04-06T13:49:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.929+0000 7f268e7fc700 1 -- 172.21.15.189:0/1743478137 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26940119b0 con 0x7f269815e8a0 2024-04-06T13:49:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.929+0000 7f268e7fc700 1 -- 172.21.15.189:0/1743478137 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2694011d60 con 0x7f269815e8a0 2024-04-06T13:49:04.937 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.929+0000 7f26a0222700 1 -- 172.21.15.189:0/1743478137 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26980b3e80 con 0x7f269815e8a0 2024-04-06T13:49:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.929+0000 7f26a0222700 1 -- 172.21.15.189:0/1743478137 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26981575f0 con 0x7f269815e8a0 2024-04-06T13:49:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.930+0000 7f26a0222700 1 -- 172.21.15.189:0/1743478137 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2698072140 con 0x7f269815e8a0 2024-04-06T13:49:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.930+0000 7f268e7fc700 1 -- 172.21.15.189:0/1743478137 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f269401a400 con 0x7f269815e8a0 2024-04-06T13:49:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.931+0000 7f268e7fc700 1 --2- 172.21.15.189:0/1743478137 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f267c06aed0 0x7f267c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.931+0000 7f268e7fc700 1 -- 172.21.15.189:0/1743478137 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6973+0+0 (secure 0 0 0) 0x7f2694089dc0 con 0x7f269815e8a0 2024-04-06T13:49:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.931+0000 7f269d7bd700 1 --2- 172.21.15.189:0/1743478137 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f267c06aed0 0x7f267c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:04.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.931+0000 7f269d7bd700 1 --2- 172.21.15.189:0/1743478137 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f267c06aed0 0x7f267c06d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f26981f3f30 tx=0x7f2688009040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:04.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:04.936+0000 7f268e7fc700 1 -- 172.21.15.189:0/1743478137 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f26940546c0 con 0x7f269815e8a0 2024-04-06T13:49:05.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.132+0000 7f26a0222700 1 -- 172.21.15.189:0/1743478137 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26980b4680 con 0x7f269815e8a0 2024-04-06T13:49:05.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.134+0000 7f268e7fc700 1 -- 172.21.15.189:0/1743478137 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v257) v1 ==== 74+0+17683 (secure 0 0 0) 0x7f2694015070 con 0x7f269815e8a0 2024-04-06T13:49:05.135 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:05.135 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":257,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:49:01.853081+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-04-06T13:48:33.912100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-04-06T13:48:35.170642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.192:6833/726816746":"2024-04-07T13:48:58.736634+0000","172.21.15.192:6832/726816746":"2024-04-07T13:48:58.736634+0000","172.21.15.189:0/1050000893":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/2601670651":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/3142793643":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/1266339172":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/3326966043":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/4043705133":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/85704189":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2120970213":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/381758424":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2635472538":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/2107771319":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/2145696658":"2024-04-07T13:48:48.812402+0000","172.21.15.189:6833/111704":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/3440291825":"2024-04-07T13:48:48.812402+0000","172.21.15.189:6832/111704":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/598265929":"2024-04-07T13:48:56.738551+0000","172.21.15.189:0/485512536":"2024-04-07T13:48:48.812402+0000","172.21.15.189:0/643921686":"2024-04-07T13:48:48.812402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:49:05.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.136+0000 7f26a0222700 1 -- 172.21.15.189:0/1743478137 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f267c06aed0 msgr2=0x7f267c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:05.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.136+0000 7f26a0222700 1 --2- 172.21.15.189:0/1743478137 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f267c06aed0 0x7f267c06d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f26981f3f30 tx=0x7f2688009040 comp rx=0 tx=0).stop 2024-04-06T13:49:05.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.136+0000 7f26a0222700 1 -- 172.21.15.189:0/1743478137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f269815e8a0 msgr2=0x7f26981570b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:05.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.136+0000 7f26a0222700 1 --2- 172.21.15.189:0/1743478137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f269815e8a0 0x7f26981570b0 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7f269400b530 tx=0x7f269400b8f0 comp rx=0 tx=0).stop 2024-04-06T13:49:05.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.136+0000 7f26a0222700 1 -- 172.21.15.189:0/1743478137 shutdown_connections 2024-04-06T13:49:05.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.136+0000 7f26a0222700 1 --2- 172.21.15.189:0/1743478137 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f267c06aed0 0x7f267c06d390 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:05.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.136+0000 7f26a0222700 1 --2- 172.21.15.189:0/1743478137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f269815fa70 0x7f2698158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:05.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.136+0000 7f26a0222700 1 --2- 172.21.15.189:0/1743478137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f269815e8a0 0x7f26981570b0 unknown :-1 s=CLOSED pgs=2865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:05.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.136+0000 7f26a0222700 1 --2- 172.21.15.189:0/1743478137 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f269815eee0 0x7f26980b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:05.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.136+0000 7f26a0222700 1 -- 172.21.15.189:0/1743478137 >> 172.21.15.189:0/1743478137 conn(0x7f26980cc290 msgr2=0x7f269806b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:05.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.136+0000 7f26a0222700 1 -- 172.21.15.189:0/1743478137 shutdown_connections 2024-04-06T13:49:05.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.136+0000 7f26a0222700 1 -- 172.21.15.189:0/1743478137 wait complete. 2024-04-06T13:49:05.149 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2024-04-06T13:49:05.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.272+0000 7f9c900f4700 1 Processor -- start 2024-04-06T13:49:05.272 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.272+0000 7f9c900f4700 1 -- start start 2024-04-06T13:49:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.272+0000 7f9c900f4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9c8815a550 0x7f9c880cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.272+0000 7f9c900f4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c880cdbe0 con 0x7f9c880cf900 2024-04-06T13:49:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.272+0000 7f9c900f4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c8815a940 con 0x7f9c880b4d70 2024-04-06T13:49:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.272+0000 7f9c900f4700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c8815ac00 con 0x7f9c8815a550 2024-04-06T13:49:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.272+0000 7f9c8de90700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9c8815a550 0x7f9c880cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.272+0000 7f9c8d68f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9c880cf900 0x7f9c880b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50488/0 (socket says 172.21.15.189:50488) 2024-04-06T13:49:05.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.272+0000 7f9c8d68f700 1 -- 172.21.15.189:0/2884961080 learned_addr learned my addr 172.21.15.189:0/2884961080 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:05.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.272+0000 7f9c8de90700 1 -- 172.21.15.189:0/2884961080 >> v1:172.21.15.192:6789/0 conn(0x7f9c880b4d70 legacy=0x7f9c880b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:05.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.272+0000 7f9c8de90700 1 -- 172.21.15.189:0/2884961080 >> v1:172.21.15.189:6789/0 conn(0x7f9c880cf900 legacy=0x7f9c880b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:05.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.272+0000 7f9c8de90700 1 -- 172.21.15.189:0/2884961080 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c8815aee0 con 0x7f9c8815a550 2024-04-06T13:49:05.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.272+0000 7f9c8ce8e700 1 -- 172.21.15.189:0/2884961080 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470009469 0 0) 0x7f9c880cdbe0 con 0x7f9c880cf900 2024-04-06T13:49:05.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.272+0000 7f9c8de90700 1 --2- 172.21.15.189:0/2884961080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c8815a550 0x7f9c880cf3c0 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7f9c84009840 tx=0x7f9c84009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e0e95a98707c77 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:05.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.273+0000 7f9c8ce8e700 1 -- 172.21.15.189:0/2884961080 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c84004230 con 0x7f9c8815a550 2024-04-06T13:49:05.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.273+0000 7f9c8ce8e700 1 -- 172.21.15.189:0/2884961080 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c84004390 con 0x7f9c8815a550 2024-04-06T13:49:05.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.273+0000 7f9c8ce8e700 1 -- 172.21.15.189:0/2884961080 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c840036a0 con 0x7f9c8815a550 2024-04-06T13:49:05.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.273+0000 7f9c900f4700 1 -- 172.21.15.189:0/2884961080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c8815a550 msgr2=0x7f9c880cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:05.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.273+0000 7f9c900f4700 1 --2- 172.21.15.189:0/2884961080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c8815a550 0x7f9c880cf3c0 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7f9c84009840 tx=0x7f9c84009b50 comp rx=0 tx=0).stop 2024-04-06T13:49:05.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.273+0000 7f9c900f4700 1 -- 172.21.15.189:0/2884961080 shutdown_connections 2024-04-06T13:49:05.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.273+0000 7f9c900f4700 1 --2- 172.21.15.189:0/2884961080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c8815a550 0x7f9c880cf3c0 unknown :-1 s=CLOSED pgs=2867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:05.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.273+0000 7f9c900f4700 1 -- 172.21.15.189:0/2884961080 >> 172.21.15.189:0/2884961080 conn(0x7f9c880d00b0 msgr2=0x7f9c880d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:05.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.273+0000 7f9c900f4700 1 -- 172.21.15.189:0/2884961080 shutdown_connections 2024-04-06T13:49:05.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.273+0000 7f9c900f4700 1 -- 172.21.15.189:0/2884961080 wait complete. 2024-04-06T13:49:05.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.274+0000 7f9c900f4700 1 Processor -- start 2024-04-06T13:49:05.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.274+0000 7f9c900f4700 1 -- start start 2024-04-06T13:49:05.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.275+0000 7f9c900f4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c880b4d70 0x7f9c88155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:05.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.275+0000 7f9c900f4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c880cf900 0x7f9c881f6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:05.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.275+0000 7f9c900f4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c88154a40 0x7f9c880717b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:05.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.275+0000 7f9c900f4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c881f71f0 con 0x7f9c88154a40 2024-04-06T13:49:05.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.275+0000 7f9c900f4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c88155e70 con 0x7f9c880cf900 2024-04-06T13:49:05.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.275+0000 7f9c900f4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c88156150 con 0x7f9c880b4d70 2024-04-06T13:49:05.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.275+0000 7f9c8e691700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c88154a40 0x7f9c880717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:05.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.275+0000 7f9c8e691700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c88154a40 0x7f9c880717b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37324/0 (socket says 172.21.15.189:37324) 2024-04-06T13:49:05.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.275+0000 7f9c8e691700 1 -- 172.21.15.189:0/132870152 learned_addr learned my addr 172.21.15.189:0/132870152 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:05.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.275+0000 7f9c8de90700 1 --2- 172.21.15.189:0/132870152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c880b4d70 0x7f9c88155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:05.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.275+0000 7f9c8d68f700 1 --2- 172.21.15.189:0/132870152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c880cf900 0x7f9c881f6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:05.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.275+0000 7f9c8e691700 1 -- 172.21.15.189:0/132870152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c880b4d70 msgr2=0x7f9c88155890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:05.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.275+0000 7f9c8e691700 1 --2- 172.21.15.189:0/132870152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c880b4d70 0x7f9c88155890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:05.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.275+0000 7f9c8e691700 1 -- 172.21.15.189:0/132870152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c880cf900 msgr2=0x7f9c881f6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:05.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.275+0000 7f9c8e691700 1 --2- 172.21.15.189:0/132870152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c880cf900 0x7f9c881f6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:05.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.276+0000 7f9c8e691700 1 -- 172.21.15.189:0/132870152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c840094a0 con 0x7f9c88154a40 2024-04-06T13:49:05.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.276+0000 7f9c8d68f700 1 --2- 172.21.15.189:0/132870152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c880cf900 0x7f9c881f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:05.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.276+0000 7f9c8de90700 1 --2- 172.21.15.189:0/132870152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c880b4d70 0x7f9c88155890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:05.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.276+0000 7f9c8e691700 1 --2- 172.21.15.189:0/132870152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c88154a40 0x7f9c880717b0 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7f9c8000cc10 tx=0x7f9c8000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:05.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.276+0000 7f9c7e7fc700 1 -- 172.21.15.189:0/132870152 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8000abb0 con 0x7f9c88154a40 2024-04-06T13:49:05.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.276+0000 7f9c7e7fc700 1 -- 172.21.15.189:0/132870152 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8000ad10 con 0x7f9c88154a40 2024-04-06T13:49:05.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.276+0000 7f9c900f4700 1 -- 172.21.15.189:0/132870152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c88157ba0 con 0x7f9c88154a40 2024-04-06T13:49:05.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.276+0000 7f9c7e7fc700 1 -- 172.21.15.189:0/132870152 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c80012660 con 0x7f9c88154a40 2024-04-06T13:49:05.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.276+0000 7f9c900f4700 1 -- 172.21.15.189:0/132870152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c88154ed0 con 0x7f9c88154a40 2024-04-06T13:49:05.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.277+0000 7f9c900f4700 1 -- 172.21.15.189:0/132870152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c881580c0 con 0x7f9c88154a40 2024-04-06T13:49:05.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.277+0000 7f9c7e7fc700 1 -- 172.21.15.189:0/132870152 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f9c800127c0 con 0x7f9c88154a40 2024-04-06T13:49:05.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.278+0000 7f9c7e7fc700 1 --2- 172.21.15.189:0/132870152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9c6c06aed0 0x7f9c6c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:05.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.278+0000 7f9c7e7fc700 1 -- 172.21.15.189:0/132870152 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6973+0+0 (secure 0 0 0) 0x7f9c8008a360 con 0x7f9c88154a40 2024-04-06T13:49:05.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.278+0000 7f9c8de90700 1 --2- 172.21.15.189:0/132870152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9c6c06aed0 0x7f9c6c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:05.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.278+0000 7f9c8de90700 1 --2- 172.21.15.189:0/132870152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9c6c06aed0 0x7f9c6c06d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9c840090b0 tx=0x7f9c84008030 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:05.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.283+0000 7f9c7e7fc700 1 -- 172.21.15.189:0/132870152 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9c881580c0 con 0x7f9c88154a40 2024-04-06T13:49:05.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.519+0000 7f9c900f4700 1 -- 172.21.15.189:0/132870152 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7f9c881580c0 con 0x7f9c88154a40 2024-04-06T13:49:05.520 INFO:tasks.ceph.mon.a.smithi189.stderr:2024-04-06T13:49:05.520+0000 7fa9fe1ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T13:49:05.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.528+0000 7f9c7e7fc700 1 -- 172.21.15.189:0/132870152 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v125) v1 ==== 167+0+0 (secure 0 0 0) 0x7f9c881580c0 con 0x7f9c88154a40 2024-04-06T13:49:05.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.531+0000 7f9c900f4700 1 -- 172.21.15.189:0/132870152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9c6c06aed0 msgr2=0x7f9c6c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:05.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.531+0000 7f9c900f4700 1 --2- 172.21.15.189:0/132870152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9c6c06aed0 0x7f9c6c06d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9c840090b0 tx=0x7f9c84008030 comp rx=0 tx=0).stop 2024-04-06T13:49:05.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.531+0000 7f9c900f4700 1 -- 172.21.15.189:0/132870152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c88154a40 msgr2=0x7f9c880717b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:05.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.531+0000 7f9c900f4700 1 --2- 172.21.15.189:0/132870152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c88154a40 0x7f9c880717b0 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7f9c8000cc10 tx=0x7f9c8000a060 comp rx=0 tx=0).stop 2024-04-06T13:49:05.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.531+0000 7f9c900f4700 1 -- 172.21.15.189:0/132870152 shutdown_connections 2024-04-06T13:49:05.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.531+0000 7f9c900f4700 1 --2- 172.21.15.189:0/132870152 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9c6c06aed0 0x7f9c6c06d390 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:05.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.531+0000 7f9c900f4700 1 --2- 172.21.15.189:0/132870152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c88154a40 0x7f9c880717b0 unknown :-1 s=CLOSED pgs=2868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:05.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.531+0000 7f9c900f4700 1 --2- 172.21.15.189:0/132870152 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c880b4d70 0x7f9c88155890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:05.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.531+0000 7f9c900f4700 1 --2- 172.21.15.189:0/132870152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c880cf900 0x7f9c881f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:05.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.531+0000 7f9c900f4700 1 -- 172.21.15.189:0/132870152 >> 172.21.15.189:0/132870152 conn(0x7f9c880d00b0 msgr2=0x7f9c880b0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:05.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.531+0000 7f9c900f4700 1 -- 172.21.15.189:0/132870152 shutdown_connections 2024-04-06T13:49:05.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.531+0000 7f9c900f4700 1 -- 172.21.15.189:0/132870152 wait complete. 2024-04-06T13:49:05.533 INFO:teuthology.orchestra.run.smithi189.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-06T13:49:05.534 INFO:tasks.ceph.mds.b.smithi192.stderr: -1588> 2024-04-06T13:48:40.252+0000 7fe83f367700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T13:49:05.544 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2024-04-06T13:49:05.563 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T13:49:05.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.667+0000 7f61b3be7700 1 Processor -- start 2024-04-06T13:49:05.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.667+0000 7f61b3be7700 1 -- start start 2024-04-06T13:49:05.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.668+0000 7f61b3be7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61ac06ae40 0x7f61ac0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:05.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.668+0000 7f61b3be7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ac0b3990 con 0x7f61ac15d7b0 2024-04-06T13:49:05.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.668+0000 7f61b3be7700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ac0b3060 con 0x7f61ac15ddf0 2024-04-06T13:49:05.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.668+0000 7f61b3be7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac0b3340 con 0x7f61ac06ae40 2024-04-06T13:49:05.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.668+0000 7f61b1983700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f61ac15d7b0 0x7f61ac06a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50498/0 (socket says 172.21.15.189:50498) 2024-04-06T13:49:05.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.668+0000 7f61b1983700 1 -- 172.21.15.189:0/2074725224 learned_addr learned my addr 172.21.15.189:0/2074725224 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:05.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b0981700 1 -- 172.21.15.189:0/2074725224 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456348632 0 0) 0x7f61ac0b3990 con 0x7f61ac15d7b0 2024-04-06T13:49:05.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b1182700 1 --2- 172.21.15.189:0/2074725224 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61ac06ae40 0x7f61ac0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:05.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b0981700 1 -- 172.21.15.189:0/2074725224 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61980035b0 con 0x7f61ac15d7b0 2024-04-06T13:49:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b0981700 1 -- 172.21.15.189:0/2074725224 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4252635022 0 0) 0x7f61980035b0 con 0x7f61ac15d7b0 2024-04-06T13:49:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b0981700 1 -- 172.21.15.189:0/2074725224 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61ac06ae40 msgr2=0x7f61ac0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b0981700 1 --2- 172.21.15.189:0/2074725224 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61ac06ae40 0x7f61ac0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b0981700 1 -- 172.21.15.189:0/2074725224 >> v1:172.21.15.192:6789/0 conn(0x7f61ac15ddf0 legacy=0x7f61ac15e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b0981700 1 -- 172.21.15.189:0/2074725224 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac0b3620 con 0x7f61ac15d7b0 2024-04-06T13:49:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b0981700 1 -- 172.21.15.189:0/2074725224 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f619c001670 con 0x7f61ac15d7b0 2024-04-06T13:49:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b0981700 1 -- 172.21.15.189:0/2074725224 >> v1:172.21.15.189:6789/0 conn(0x7f61ac15d7b0 legacy=0x7f61ac06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b0981700 1 --2- 172.21.15.189:0/2074725224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6198003a30 0x7f6198003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b0981700 1 --2- 172.21.15.189:0/2074725224 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61ac06ae40 0x7f6198004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b0981700 1 --2- 172.21.15.189:0/2074725224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6198004d90 0x7f6198005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b0981700 1 -- 172.21.15.189:0/2074725224 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac0b3340 con 0x7f6198004d90 2024-04-06T13:49:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b0981700 1 -- 172.21.15.189:0/2074725224 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619800baf0 con 0x7f6198003a30 2024-04-06T13:49:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b0981700 1 -- 172.21.15.189:0/2074725224 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619800bcb0 con 0x7f61ac06ae40 2024-04-06T13:49:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b1983700 1 --2- 172.21.15.189:0/2074725224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6198003a30 0x7f6198003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:05.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b2184700 1 --2- 172.21.15.189:0/2074725224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6198004d90 0x7f6198005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b1182700 1 --2- 172.21.15.189:0/2074725224 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61ac06ae40 0x7f6198004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b2184700 1 -- 172.21.15.189:0/2074725224 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61ac06ae40 msgr2=0x7f6198004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b2184700 1 --2- 172.21.15.189:0/2074725224 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61ac06ae40 0x7f6198004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b2184700 1 -- 172.21.15.189:0/2074725224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6198003a30 msgr2=0x7f6198003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b2184700 1 --2- 172.21.15.189:0/2074725224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6198003a30 0x7f6198003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b2184700 1 -- 172.21.15.189:0/2074725224 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61980035b0 con 0x7f6198004d90 2024-04-06T13:49:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.669+0000 7f61b1983700 1 --2- 172.21.15.189:0/2074725224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6198003a30 0x7f6198003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.670+0000 7f61b2184700 1 --2- 172.21.15.189:0/2074725224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6198004d90 0x7f6198005210 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 crypto rx=0x7f61a400c290 tx=0x7f6198005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f45195765f388b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.670+0000 7f61b0981700 1 -- 172.21.15.189:0/2074725224 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a4001e70 con 0x7f6198004d90 2024-04-06T13:49:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.670+0000 7f61b0981700 1 -- 172.21.15.189:0/2074725224 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a400ce50 con 0x7f6198004d90 2024-04-06T13:49:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.670+0000 7f61b0981700 1 -- 172.21.15.189:0/2074725224 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a4012660 con 0x7f6198004d90 2024-04-06T13:49:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.670+0000 7f61b3be7700 1 -- 172.21.15.189:0/2074725224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6198004d90 msgr2=0x7f6198005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.670+0000 7f61b3be7700 1 --2- 172.21.15.189:0/2074725224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6198004d90 0x7f6198005210 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 crypto rx=0x7f61a400c290 tx=0x7f6198005880 comp rx=0 tx=0).stop 2024-04-06T13:49:05.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.670+0000 7f61b2184700 1 -- 172.21.15.189:0/2074725224 reap_dead start 2024-04-06T13:49:05.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.670+0000 7f61b3be7700 1 -- 172.21.15.189:0/2074725224 shutdown_connections 2024-04-06T13:49:05.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.670+0000 7f61b3be7700 1 -- 172.21.15.189:0/2074725224 >> 172.21.15.189:0/2074725224 conn(0x7f61ac0d00b0 msgr2=0x7f61ac0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:05.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.670+0000 7f61b3be7700 1 -- 172.21.15.189:0/2074725224 shutdown_connections 2024-04-06T13:49:05.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.670+0000 7f61b3be7700 1 -- 172.21.15.189:0/2074725224 wait complete. 2024-04-06T13:49:05.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.671+0000 7f61b3be7700 1 Processor -- start 2024-04-06T13:49:05.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.672+0000 7f61b3be7700 1 -- start start 2024-04-06T13:49:05.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.672+0000 7f61b3be7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61ac152b70 0x7f61ac071d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:05.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.672+0000 7f61b3be7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61ac152f50 0x7f61ac0aee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:05.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.672+0000 7f61b3be7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61ac157330 0x7f61ac0af390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:05.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.672+0000 7f61b3be7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac072290 con 0x7f61ac157330 2024-04-06T13:49:05.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.672+0000 7f61b3be7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac0cc540 con 0x7f61ac152b70 2024-04-06T13:49:05.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.672+0000 7f61b3be7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac0b0e80 con 0x7f61ac152f50 2024-04-06T13:49:05.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.672+0000 7f61b2184700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61ac157330 0x7f61ac0af390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:05.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.672+0000 7f61b2184700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61ac157330 0x7f61ac0af390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37342/0 (socket says 172.21.15.189:37342) 2024-04-06T13:49:05.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.672+0000 7f61b2184700 1 -- 172.21.15.189:0/1697863421 learned_addr learned my addr 172.21.15.189:0/1697863421 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:05.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.672+0000 7f61b1182700 1 --2- 172.21.15.189:0/1697863421 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61ac152f50 0x7f61ac0aee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:05.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.672+0000 7f61b1983700 1 --2- 172.21.15.189:0/1697863421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61ac152b70 0x7f61ac071d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:05.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.673+0000 7f61b2184700 1 -- 172.21.15.189:0/1697863421 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61ac152f50 msgr2=0x7f61ac0aee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:05.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.673+0000 7f61b2184700 1 --2- 172.21.15.189:0/1697863421 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61ac152f50 0x7f61ac0aee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:05.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.673+0000 7f61b2184700 1 -- 172.21.15.189:0/1697863421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61ac152b70 msgr2=0x7f61ac071d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:05.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.673+0000 7f61b2184700 1 --2- 172.21.15.189:0/1697863421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61ac152b70 0x7f61ac071d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:05.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.673+0000 7f61b2184700 1 -- 172.21.15.189:0/1697863421 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac0b1140 con 0x7f61ac157330 2024-04-06T13:49:05.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.673+0000 7f61b1182700 1 --2- 172.21.15.189:0/1697863421 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61ac152f50 0x7f61ac0aee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:05.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.673+0000 7f61b1983700 1 --2- 172.21.15.189:0/1697863421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61ac152b70 0x7f61ac071d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:05.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.673+0000 7f61b2184700 1 --2- 172.21.15.189:0/1697863421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61ac157330 0x7f61ac0af390 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f61a4001220 tx=0x7f61a40015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:05.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.673+0000 7f61a27fc700 1 -- 172.21.15.189:0/1697863421 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a4003d30 con 0x7f61ac157330 2024-04-06T13:49:05.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.673+0000 7f61a27fc700 1 -- 172.21.15.189:0/1697863421 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a4020070 con 0x7f61ac157330 2024-04-06T13:49:05.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.673+0000 7f61a27fc700 1 -- 172.21.15.189:0/1697863421 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a40052c0 con 0x7f61ac157330 2024-04-06T13:49:05.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.673+0000 7f61b3be7700 1 -- 172.21.15.189:0/1697863421 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61ac0b1420 con 0x7f61ac157330 2024-04-06T13:49:05.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.673+0000 7f61b3be7700 1 -- 172.21.15.189:0/1697863421 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61ac0b19e0 con 0x7f61ac157330 2024-04-06T13:49:05.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.674+0000 7f61b3be7700 1 -- 172.21.15.189:0/1697863421 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61ac073200 con 0x7f61ac157330 2024-04-06T13:49:05.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.674+0000 7f61a27fc700 1 -- 172.21.15.189:0/1697863421 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f61a4022030 con 0x7f61ac157330 2024-04-06T13:49:05.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.675+0000 7f61a27fc700 1 --2- 172.21.15.189:0/1697863421 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f619006aed0 0x7f619006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:05.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.675+0000 7f61a27fc700 1 -- 172.21.15.189:0/1697863421 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7059+0+0 (secure 0 0 0) 0x7f61a408a880 con 0x7f61ac157330 2024-04-06T13:49:05.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.675+0000 7f61b1983700 1 --2- 172.21.15.189:0/1697863421 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f619006aed0 0x7f619006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:05.681 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.675+0000 7f61b1983700 1 --2- 172.21.15.189:0/1697863421 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f619006aed0 0x7f619006d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f619c0010e0 tx=0x7f619c009040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:05.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.679+0000 7f61a27fc700 1 -- 172.21.15.189:0/1697863421 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f61ac073200 con 0x7f61ac157330 2024-04-06T13:49:05.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:05.919+0000 7f61b3be7700 1 -- 172.21.15.189:0/1697863421 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f61ac073200 con 0x7f61ac157330 2024-04-06T13:49:06.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.532+0000 7f61a27fc700 1 -- 172.21.15.189:0/1697863421 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v126) v1 ==== 107+0+0 (secure 0 0 0) 0x7f61ac073200 con 0x7f61ac157330 2024-04-06T13:49:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.535+0000 7f61b3be7700 1 -- 172.21.15.189:0/1697863421 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f619006aed0 msgr2=0x7f619006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.535+0000 7f61b3be7700 1 --2- 172.21.15.189:0/1697863421 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f619006aed0 0x7f619006d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f619c0010e0 tx=0x7f619c009040 comp rx=0 tx=0).stop 2024-04-06T13:49:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.535+0000 7f61b3be7700 1 -- 172.21.15.189:0/1697863421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61ac157330 msgr2=0x7f61ac0af390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.535+0000 7f61b3be7700 1 --2- 172.21.15.189:0/1697863421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61ac157330 0x7f61ac0af390 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f61a4001220 tx=0x7f61a40015b0 comp rx=0 tx=0).stop 2024-04-06T13:49:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.536+0000 7f61b3be7700 1 -- 172.21.15.189:0/1697863421 shutdown_connections 2024-04-06T13:49:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.536+0000 7f61b3be7700 1 --2- 172.21.15.189:0/1697863421 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f619006aed0 0x7f619006d390 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.536+0000 7f61b3be7700 1 --2- 172.21.15.189:0/1697863421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61ac157330 0x7f61ac0af390 unknown :-1 s=CLOSED pgs=2872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.536+0000 7f61b3be7700 1 --2- 172.21.15.189:0/1697863421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61ac152b70 0x7f61ac071d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:06.536 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.536+0000 7f61b3be7700 1 --2- 172.21.15.189:0/1697863421 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61ac152f50 0x7f61ac0aee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:06.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.536+0000 7f61b3be7700 1 -- 172.21.15.189:0/1697863421 >> 172.21.15.189:0/1697863421 conn(0x7f61ac0d00b0 msgr2=0x7f61ac06f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:06.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.536+0000 7f61b3be7700 1 -- 172.21.15.189:0/1697863421 shutdown_connections 2024-04-06T13:49:06.537 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.536+0000 7f61b3be7700 1 -- 172.21.15.189:0/1697863421 wait complete. 2024-04-06T13:49:06.551 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2024-04-06T13:49:06.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.682+0000 7f640beb0700 1 Processor -- start 2024-04-06T13:49:06.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.682+0000 7f640beb0700 1 -- start start 2024-04-06T13:49:06.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.683+0000 7f640beb0700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f64040cd3e0 0x7f64040cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:06.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.683+0000 7f640beb0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f640406b8c0 0x7f640406dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:06.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.683+0000 7f640beb0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f640406ceb0 con 0x7f640415d7c0 2024-04-06T13:49:06.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.683+0000 7f640beb0700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64040cdfa0 con 0x7f64040cd3e0 2024-04-06T13:49:06.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.683+0000 7f640beb0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f640406bcf0 con 0x7f640406b8c0 2024-04-06T13:49:06.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.683+0000 7f640944b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f640415d7c0 0x7f640406b1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50514/0 (socket says 172.21.15.189:50514) 2024-04-06T13:49:06.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.683+0000 7f640944b700 1 -- 172.21.15.189:0/3991751039 learned_addr learned my addr 172.21.15.189:0/3991751039 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:06.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.683+0000 7f640a44d700 1 --2- 172.21.15.189:0/3991751039 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f640406b8c0 0x7f640406dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:06.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.683+0000 7f6409c4c700 1 --2- 172.21.15.189:0/3991751039 >> v2:172.21.15.192:3300/0 conn(0x7f64040cd3e0 0x7f64040cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:06.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.683+0000 7f6408c4a700 1 -- 172.21.15.189:0/3991751039 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3631760661 0 0) 0x7f640406ceb0 con 0x7f640415d7c0 2024-04-06T13:49:06.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.683+0000 7f6408c4a700 1 -- 172.21.15.189:0/3991751039 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f00035b0 con 0x7f640415d7c0 2024-04-06T13:49:06.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f6408c4a700 1 -- 172.21.15.189:0/3991751039 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3217191549 0 0) 0x7f63f00035b0 con 0x7f640415d7c0 2024-04-06T13:49:06.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f6408c4a700 1 -- 172.21.15.189:0/3991751039 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f640406b8c0 msgr2=0x7f640406dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:06.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f6408c4a700 1 --2- 172.21.15.189:0/3991751039 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f640406b8c0 0x7f640406dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:06.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f6408c4a700 1 -- 172.21.15.189:0/3991751039 >> v2:172.21.15.192:3300/0 conn(0x7f64040cd3e0 msgr2=0x7f64040cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:06.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f6408c4a700 1 --2- 172.21.15.189:0/3991751039 >> v2:172.21.15.192:3300/0 conn(0x7f64040cd3e0 0x7f64040cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:06.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f6408c4a700 1 -- 172.21.15.189:0/3991751039 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640406d750 con 0x7f640415d7c0 2024-04-06T13:49:06.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f6408c4a700 1 -- 172.21.15.189:0/3991751039 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f6400001670 con 0x7f640415d7c0 2024-04-06T13:49:06.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f6408c4a700 1 -- 172.21.15.189:0/3991751039 >> v1:172.21.15.189:6789/0 conn(0x7f640415d7c0 legacy=0x7f640406b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:06.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f6408c4a700 1 --2- 172.21.15.189:0/3991751039 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f640406b8c0 0x7f63f0003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:06.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f6408c4a700 1 --2- 172.21.15.189:0/3991751039 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63f0003f80 0x7f63f0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:06.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f6408c4a700 1 --2- 172.21.15.189:0/3991751039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f0004d90 0x7f63f0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:06.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f6408c4a700 1 -- 172.21.15.189:0/3991751039 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64040cdfa0 con 0x7f63f0004d90 2024-04-06T13:49:06.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f6408c4a700 1 -- 172.21.15.189:0/3991751039 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640406bcf0 con 0x7f63f0003f80 2024-04-06T13:49:06.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f6408c4a700 1 -- 172.21.15.189:0/3991751039 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f000ba60 con 0x7f640406b8c0 2024-04-06T13:49:06.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f640a44d700 1 --2- 172.21.15.189:0/3991751039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f0004d90 0x7f63f0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:06.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f640944b700 1 --2- 172.21.15.189:0/3991751039 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63f0003f80 0x7f63f0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:06.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f6409c4c700 1 --2- 172.21.15.189:0/3991751039 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f640406b8c0 0x7f63f0003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:06.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f640a44d700 1 -- 172.21.15.189:0/3991751039 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f640406b8c0 msgr2=0x7f63f0003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:06.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f640a44d700 1 --2- 172.21.15.189:0/3991751039 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f640406b8c0 0x7f63f0003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:06.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f640a44d700 1 -- 172.21.15.189:0/3991751039 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63f0003f80 msgr2=0x7f63f0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:06.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f640a44d700 1 --2- 172.21.15.189:0/3991751039 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63f0003f80 0x7f63f0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:06.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f640a44d700 1 -- 172.21.15.189:0/3991751039 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f00035b0 con 0x7f63f0004d90 2024-04-06T13:49:06.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f640a44d700 1 --2- 172.21.15.189:0/3991751039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f0004d90 0x7f63f0005210 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 crypto rx=0x7f63fc00e170 tx=0x7f63f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a00936c8f9be1a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:06.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.684+0000 7f6408c4a700 1 -- 172.21.15.189:0/3991751039 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00ec90 con 0x7f63f0004d90 2024-04-06T13:49:06.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.685+0000 7f6408c4a700 1 -- 172.21.15.189:0/3991751039 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00f040 con 0x7f63f0004d90 2024-04-06T13:49:06.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.685+0000 7f6408c4a700 1 -- 172.21.15.189:0/3991751039 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc014480 con 0x7f63f0004d90 2024-04-06T13:49:06.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.685+0000 7f640beb0700 1 -- 172.21.15.189:0/3991751039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f0004d90 msgr2=0x7f63f0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:06.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.685+0000 7f640beb0700 1 --2- 172.21.15.189:0/3991751039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f0004d90 0x7f63f0005210 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 crypto rx=0x7f63fc00e170 tx=0x7f63f0005880 comp rx=0 tx=0).stop 2024-04-06T13:49:06.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.685+0000 7f640a44d700 1 -- 172.21.15.189:0/3991751039 reap_dead start 2024-04-06T13:49:06.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.685+0000 7f640beb0700 1 -- 172.21.15.189:0/3991751039 shutdown_connections 2024-04-06T13:49:06.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.685+0000 7f640beb0700 1 -- 172.21.15.189:0/3991751039 >> 172.21.15.189:0/3991751039 conn(0x7f64040cc290 msgr2=0x7f64040cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:06.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.685+0000 7f640beb0700 1 -- 172.21.15.189:0/3991751039 shutdown_connections 2024-04-06T13:49:06.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.685+0000 7f640beb0700 1 -- 172.21.15.189:0/3991751039 wait complete. 2024-04-06T13:49:06.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.686+0000 7f640beb0700 1 Processor -- start 2024-04-06T13:49:06.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.686+0000 7f640beb0700 1 -- start start 2024-04-06T13:49:06.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.687+0000 7f640beb0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63f0004d90 0x7f6404159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:06.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.687+0000 7f640beb0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6404160330 0x7f640415dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:06.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.687+0000 7f640beb0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64040c5a20 0x7f640415e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:06.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.687+0000 7f640beb0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6404160b80 con 0x7f64040c5a20 2024-04-06T13:49:06.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.687+0000 7f640beb0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64041f6170 con 0x7f6404160330 2024-04-06T13:49:06.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.687+0000 7f640beb0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f640415fa50 con 0x7f63f0004d90 2024-04-06T13:49:06.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.687+0000 7f640a44d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64040c5a20 0x7f640415e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:06.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.687+0000 7f640a44d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64040c5a20 0x7f640415e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37362/0 (socket says 172.21.15.189:37362) 2024-04-06T13:49:06.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.687+0000 7f640a44d700 1 -- 172.21.15.189:0/1983270634 learned_addr learned my addr 172.21.15.189:0/1983270634 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:06.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.688+0000 7f6409c4c700 1 --2- 172.21.15.189:0/1983270634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63f0004d90 0x7f6404159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:06.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.688+0000 7f640a44d700 1 -- 172.21.15.189:0/1983270634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63f0004d90 msgr2=0x7f6404159b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:06.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.688+0000 7f640a44d700 1 --2- 172.21.15.189:0/1983270634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63f0004d90 0x7f6404159b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:06.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.688+0000 7f640a44d700 1 -- 172.21.15.189:0/1983270634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6404160330 msgr2=0x7f640415dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:06.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.688+0000 7f640a44d700 1 --2- 172.21.15.189:0/1983270634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6404160330 0x7f640415dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:06.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.688+0000 7f640a44d700 1 -- 172.21.15.189:0/1983270634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64041608a0 con 0x7f64040c5a20 2024-04-06T13:49:06.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.688+0000 7f640a44d700 1 --2- 172.21.15.189:0/1983270634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64040c5a20 0x7f640415e400 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7f63f0004790 tx=0x7f63fc004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:06.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.688+0000 7f63fa7fc700 1 -- 172.21.15.189:0/1983270634 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc007000 con 0x7f64040c5a20 2024-04-06T13:49:06.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.688+0000 7f63fa7fc700 1 -- 172.21.15.189:0/1983270634 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00ee80 con 0x7f64040c5a20 2024-04-06T13:49:06.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.688+0000 7f640beb0700 1 -- 172.21.15.189:0/1983270634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64040c4f40 con 0x7f64040c5a20 2024-04-06T13:49:06.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.688+0000 7f63fa7fc700 1 -- 172.21.15.189:0/1983270634 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc003c60 con 0x7f64040c5a20 2024-04-06T13:49:06.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.688+0000 7f640beb0700 1 -- 172.21.15.189:0/1983270634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64040c5500 con 0x7f64040c5a20 2024-04-06T13:49:06.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.689+0000 7f640beb0700 1 -- 172.21.15.189:0/1983270634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64040c4d70 con 0x7f64040c5a20 2024-04-06T13:49:06.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.690+0000 7f63fa7fc700 1 -- 172.21.15.189:0/1983270634 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f63fc024070 con 0x7f64040c5a20 2024-04-06T13:49:06.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.690+0000 7f63fa7fc700 1 --2- 172.21.15.189:0/1983270634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f63e806aed0 0x7f63e806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:06.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.690+0000 7f6409c4c700 1 --2- 172.21.15.189:0/1983270634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f63e806aed0 0x7f63e806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:06.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.690+0000 7f63fa7fc700 1 -- 172.21.15.189:0/1983270634 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7059+0+0 (secure 0 0 0) 0x7f63fc08bb20 con 0x7f64040c5a20 2024-04-06T13:49:06.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.691+0000 7f6409c4c700 1 --2- 172.21.15.189:0/1983270634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f63e806aed0 0x7f63e806d390 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f63f4009b20 tx=0x7f63f4008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:06.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.694+0000 7f63fa7fc700 1 -- 172.21.15.189:0/1983270634 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f63fc056540 con 0x7f64040c5a20 2024-04-06T13:49:06.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:06.893+0000 7f640beb0700 1 -- 172.21.15.189:0/1983270634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6404074990 con 0x7f64040c5a20 2024-04-06T13:49:07.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:07.881+0000 7f63fa7fc700 1 -- 172.21.15.189:0/1983270634 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v259) v1 ==== 190+0+0 (secure 0 0 0) 0x7f63fc018020 con 0x7f64040c5a20 2024-04-06T13:49:07.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:07.885+0000 7f640beb0700 1 -- 172.21.15.189:0/1983270634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f63e806aed0 msgr2=0x7f63e806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:07.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:07.885+0000 7f640beb0700 1 --2- 172.21.15.189:0/1983270634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f63e806aed0 0x7f63e806d390 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f63f4009b20 tx=0x7f63f4008040 comp rx=0 tx=0).stop 2024-04-06T13:49:07.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:07.885+0000 7f640beb0700 1 -- 172.21.15.189:0/1983270634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64040c5a20 msgr2=0x7f640415e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:07.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:07.885+0000 7f640beb0700 1 --2- 172.21.15.189:0/1983270634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64040c5a20 0x7f640415e400 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7f63f0004790 tx=0x7f63fc004b40 comp rx=0 tx=0).stop 2024-04-06T13:49:07.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:07.885+0000 7f640beb0700 1 -- 172.21.15.189:0/1983270634 shutdown_connections 2024-04-06T13:49:07.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:07.885+0000 7f640beb0700 1 --2- 172.21.15.189:0/1983270634 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f63e806aed0 0x7f63e806d390 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:07.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:07.885+0000 7f640beb0700 1 --2- 172.21.15.189:0/1983270634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64040c5a20 0x7f640415e400 unknown :-1 s=CLOSED pgs=2875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:07.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:07.885+0000 7f640beb0700 1 --2- 172.21.15.189:0/1983270634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63f0004d90 0x7f6404159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:07.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:07.885+0000 7f640beb0700 1 --2- 172.21.15.189:0/1983270634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6404160330 0x7f640415dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:07.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:07.885+0000 7f640beb0700 1 -- 172.21.15.189:0/1983270634 >> 172.21.15.189:0/1983270634 conn(0x7f64040cc290 msgr2=0x7f6404070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:07.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:07.885+0000 7f640beb0700 1 -- 172.21.15.189:0/1983270634 shutdown_connections 2024-04-06T13:49:07.886 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:07.885+0000 7f640beb0700 1 -- 172.21.15.189:0/1983270634 wait complete. 2024-04-06T13:49:07.886 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'backup_fs_metadata' removed 2024-04-06T13:49:07.902 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2024-04-06T13:49:08.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.036+0000 7fe77111e700 1 Processor -- start 2024-04-06T13:49:08.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.036+0000 7fe77111e700 1 -- start start 2024-04-06T13:49:08.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.037+0000 7fe77111e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe76c15d7b0 0x7fe76c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:08.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.037+0000 7fe77111e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe76c15e500 0x7fe76c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:08.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.037+0000 7fe77111e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76c0b3990 con 0x7fe76c06a730 2024-04-06T13:49:08.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.037+0000 7fe77111e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe76c06b0f0 con 0x7fe76c15e500 2024-04-06T13:49:08.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.037+0000 7fe77111e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c0b3060 con 0x7fe76c15d7b0 2024-04-06T13:49:08.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.037+0000 7fe76a59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe76c06a730 0x7fe76c15ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50518/0 (socket says 172.21.15.189:50518) 2024-04-06T13:49:08.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.037+0000 7fe76a59c700 1 -- 172.21.15.189:0/742245229 learned_addr learned my addr 172.21.15.189:0/742245229 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:08.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.037+0000 7fe76b59e700 1 --2- 172.21.15.189:0/742245229 >> v2:172.21.15.192:3300/0 conn(0x7fe76c15e500 0x7fe76c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:08.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.037+0000 7fe76ad9d700 1 --2- 172.21.15.189:0/742245229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe76c15d7b0 0x7fe76c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:08.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.037+0000 7fe769d9b700 1 -- 172.21.15.189:0/742245229 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 763095890 0 0) 0x7fe76c0b3990 con 0x7fe76c06a730 2024-04-06T13:49:08.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.037+0000 7fe769d9b700 1 -- 172.21.15.189:0/742245229 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7540035b0 con 0x7fe76c06a730 2024-04-06T13:49:08.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.037+0000 7fe769d9b700 1 -- 172.21.15.189:0/742245229 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1405686937 0 0) 0x7fe7540035b0 con 0x7fe76c06a730 2024-04-06T13:49:08.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.038+0000 7fe76b59e700 1 -- 172.21.15.189:0/742245229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe76c15d7b0 msgr2=0x7fe76c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:08.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.038+0000 7fe76b59e700 1 --2- 172.21.15.189:0/742245229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe76c15d7b0 0x7fe76c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:08.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.038+0000 7fe76b59e700 1 -- 172.21.15.189:0/742245229 >> v1:172.21.15.189:6789/0 conn(0x7fe76c06a730 legacy=0x7fe76c15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:08.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.038+0000 7fe76b59e700 1 -- 172.21.15.189:0/742245229 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76c0b3340 con 0x7fe76c15e500 2024-04-06T13:49:08.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.038+0000 7fe76b59e700 1 --2- 172.21.15.189:0/742245229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe76c15e500 0x7fe76c06abb0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe764009d90 tx=0x7fe76400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=127b52c9f303a100 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:08.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.038+0000 7fe769d9b700 1 -- 172.21.15.189:0/742245229 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76400e070 con 0x7fe76c15e500 2024-04-06T13:49:08.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.038+0000 7fe769d9b700 1 -- 172.21.15.189:0/742245229 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe764003a20 con 0x7fe76c15e500 2024-04-06T13:49:08.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.038+0000 7fe769d9b700 1 -- 172.21.15.189:0/742245229 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c001670 con 0x7fe76c15e500 2024-04-06T13:49:08.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.038+0000 7fe77111e700 1 -- 172.21.15.189:0/742245229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe76c15e500 msgr2=0x7fe76c06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:08.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.038+0000 7fe77111e700 1 --2- 172.21.15.189:0/742245229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe76c15e500 0x7fe76c06abb0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe764009d90 tx=0x7fe76400b330 comp rx=0 tx=0).stop 2024-04-06T13:49:08.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.038+0000 7fe77111e700 1 -- 172.21.15.189:0/742245229 shutdown_connections 2024-04-06T13:49:08.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.038+0000 7fe77111e700 1 --2- 172.21.15.189:0/742245229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe76c15e500 0x7fe76c06abb0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:08.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.038+0000 7fe77111e700 1 --2- 172.21.15.189:0/742245229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe76c15d7b0 0x7fe76c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:08.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.038+0000 7fe77111e700 1 -- 172.21.15.189:0/742245229 >> 172.21.15.189:0/742245229 conn(0x7fe76c0d00b0 msgr2=0x7fe76c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:08.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.038+0000 7fe77111e700 1 -- 172.21.15.189:0/742245229 shutdown_connections 2024-04-06T13:49:08.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.038+0000 7fe77111e700 1 -- 172.21.15.189:0/742245229 wait complete. 2024-04-06T13:49:08.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.040+0000 7fe77111e700 1 Processor -- start 2024-04-06T13:49:08.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.040+0000 7fe77111e700 1 -- start start 2024-04-06T13:49:08.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.041+0000 7fe77111e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe76c06a730 0x7fe76c15afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:08.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.041+0000 7fe77111e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe76c15d7b0 0x7fe76c147b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:08.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.041+0000 7fe77111e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe76c15e500 0x7fe76c15a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:08.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.041+0000 7fe77111e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c15bbf0 con 0x7fe76c15e500 2024-04-06T13:49:08.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.041+0000 7fe77111e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c148090 con 0x7fe76c06a730 2024-04-06T13:49:08.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.041+0000 7fe77111e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c15aa50 con 0x7fe76c15d7b0 2024-04-06T13:49:08.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.041+0000 7fe76b59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe76c15e500 0x7fe76c15a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:08.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.041+0000 7fe76b59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe76c15e500 0x7fe76c15a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37378/0 (socket says 172.21.15.189:37378) 2024-04-06T13:49:08.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.041+0000 7fe76b59e700 1 -- 172.21.15.189:0/3544778035 learned_addr learned my addr 172.21.15.189:0/3544778035 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:08.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.041+0000 7fe76ad9d700 1 --2- 172.21.15.189:0/3544778035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe76c06a730 0x7fe76c15afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:08.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.041+0000 7fe76a59c700 1 --2- 172.21.15.189:0/3544778035 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe76c15d7b0 0x7fe76c147b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:08.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.041+0000 7fe76b59e700 1 -- 172.21.15.189:0/3544778035 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe76c15d7b0 msgr2=0x7fe76c147b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:08.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.041+0000 7fe76b59e700 1 --2- 172.21.15.189:0/3544778035 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe76c15d7b0 0x7fe76c147b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:08.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.041+0000 7fe76b59e700 1 -- 172.21.15.189:0/3544778035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe76c06a730 msgr2=0x7fe76c15afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:08.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.041+0000 7fe76b59e700 1 --2- 172.21.15.189:0/3544778035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe76c06a730 0x7fe76c15afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:08.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.041+0000 7fe76b59e700 1 -- 172.21.15.189:0/3544778035 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe764009a70 con 0x7fe76c15e500 2024-04-06T13:49:08.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.041+0000 7fe76ad9d700 1 --2- 172.21.15.189:0/3544778035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe76c06a730 0x7fe76c15afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:08.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.041+0000 7fe76b59e700 1 --2- 172.21.15.189:0/3544778035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe76c15e500 0x7fe76c15a510 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7fe76c06c820 tx=0x7fe764004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:08.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.041+0000 7fe7537fe700 1 -- 172.21.15.189:0/3544778035 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe764004540 con 0x7fe76c15e500 2024-04-06T13:49:08.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.042+0000 7fe7537fe700 1 -- 172.21.15.189:0/3544778035 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7640046c0 con 0x7fe76c15e500 2024-04-06T13:49:08.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.042+0000 7fe77111e700 1 -- 172.21.15.189:0/3544778035 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76c15ee40 con 0x7fe76c15e500 2024-04-06T13:49:08.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.042+0000 7fe7537fe700 1 -- 172.21.15.189:0/3544778035 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76401c440 con 0x7fe76c15e500 2024-04-06T13:49:08.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.042+0000 7fe77111e700 1 -- 172.21.15.189:0/3544778035 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76c15b910 con 0x7fe76c15e500 2024-04-06T13:49:08.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.042+0000 7fe77111e700 1 -- 172.21.15.189:0/3544778035 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76c067bb0 con 0x7fe76c15e500 2024-04-06T13:49:08.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.043+0000 7fe7537fe700 1 -- 172.21.15.189:0/3544778035 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fe76401c5a0 con 0x7fe76c15e500 2024-04-06T13:49:08.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.043+0000 7fe7537fe700 1 --2- 172.21.15.189:0/3544778035 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe75806aed0 0x7fe75806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:08.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.044+0000 7fe7537fe700 1 -- 172.21.15.189:0/3544778035 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 6615+0+0 (secure 0 0 0) 0x7fe76408b7c0 con 0x7fe76c15e500 2024-04-06T13:49:08.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.044+0000 7fe76ad9d700 1 --2- 172.21.15.189:0/3544778035 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe75806aed0 0x7fe75806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:08.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.044+0000 7fe76ad9d700 1 --2- 172.21.15.189:0/3544778035 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe75806aed0 0x7fe75806d390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe760009fd0 tx=0x7fe760009210 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:08.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.048+0000 7fe7537fe700 1 -- 172.21.15.189:0/3544778035 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe764056230 con 0x7fe76c15e500 2024-04-06T13:49:08.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.245+0000 7fe77111e700 1 -- 172.21.15.189:0/3544778035 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe76c074990 con 0x7fe76c15e500 2024-04-06T13:49:08.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.896+0000 7fe7537fe700 1 -- 172.21.15.189:0/3544778035 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v260) v1 ==== 178+0+0 (secure 0 0 0) 0x7fe764059850 con 0x7fe76c15e500 2024-04-06T13:49:08.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.900+0000 7fe77111e700 1 -- 172.21.15.189:0/3544778035 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe75806aed0 msgr2=0x7fe75806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:08.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.900+0000 7fe77111e700 1 --2- 172.21.15.189:0/3544778035 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe75806aed0 0x7fe75806d390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe760009fd0 tx=0x7fe760009210 comp rx=0 tx=0).stop 2024-04-06T13:49:08.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.900+0000 7fe77111e700 1 -- 172.21.15.189:0/3544778035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe76c15e500 msgr2=0x7fe76c15a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:08.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.900+0000 7fe77111e700 1 --2- 172.21.15.189:0/3544778035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe76c15e500 0x7fe76c15a510 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7fe76c06c820 tx=0x7fe764004ad0 comp rx=0 tx=0).stop 2024-04-06T13:49:08.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.900+0000 7fe77111e700 1 -- 172.21.15.189:0/3544778035 shutdown_connections 2024-04-06T13:49:08.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.900+0000 7fe77111e700 1 --2- 172.21.15.189:0/3544778035 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe75806aed0 0x7fe75806d390 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:08.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.900+0000 7fe77111e700 1 --2- 172.21.15.189:0/3544778035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe76c15e500 0x7fe76c15a510 unknown :-1 s=CLOSED pgs=2877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:08.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.900+0000 7fe77111e700 1 --2- 172.21.15.189:0/3544778035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe76c06a730 0x7fe76c15afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:08.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.900+0000 7fe77111e700 1 --2- 172.21.15.189:0/3544778035 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe76c15d7b0 0x7fe76c147b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:08.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.900+0000 7fe77111e700 1 -- 172.21.15.189:0/3544778035 >> 172.21.15.189:0/3544778035 conn(0x7fe76c0d00b0 msgr2=0x7fe76c06ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:08.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.900+0000 7fe77111e700 1 -- 172.21.15.189:0/3544778035 shutdown_connections 2024-04-06T13:49:08.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:08.900+0000 7fe77111e700 1 -- 172.21.15.189:0/3544778035 wait complete. 2024-04-06T13:49:08.901 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'backup_fs_data' removed 2024-04-06T13:49:08.915 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-06T13:49:09.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.040+0000 7f31fde36700 1 Processor -- start 2024-04-06T13:49:09.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.040+0000 7f31fde36700 1 -- start start 2024-04-06T13:49:09.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.041+0000 7f31fde36700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f31f80d11b0 0x7f31f80d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:09.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.041+0000 7f31fde36700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f31f80cb870 0x7f31f815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:09.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.041+0000 7f31fde36700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f80d1ad0 con 0x7f31f80cfa00 2024-04-06T13:49:09.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.041+0000 7f31fde36700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f80ccdc0 con 0x7f31f80cb870 2024-04-06T13:49:09.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.041+0000 7f31fde36700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f80d0b50 con 0x7f31f80d11b0 2024-04-06T13:49:09.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.041+0000 7f31f6ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f31f80cb870 0x7f31f815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.041+0000 7f31f6ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f31f80cb870 0x7f31f815c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37386/0 (socket says 172.21.15.189:37386) 2024-04-06T13:49:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.041+0000 7f31f6ffd700 1 -- 172.21.15.189:0/595272001 learned_addr learned my addr 172.21.15.189:0/595272001 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.041+0000 7f31f77fe700 1 --2- 172.21.15.189:0/595272001 >> v2:172.21.15.192:3300/0 conn(0x7f31f80d11b0 0x7f31f80d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.041+0000 7f31f6ffd700 1 -- 172.21.15.189:0/595272001 >> v2:172.21.15.192:3300/0 conn(0x7f31f80d11b0 msgr2=0x7f31f80d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.041+0000 7f31f6ffd700 1 --2- 172.21.15.189:0/595272001 >> v2:172.21.15.192:3300/0 conn(0x7f31f80d11b0 0x7f31f80d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.041+0000 7f31f6ffd700 1 -- 172.21.15.189:0/595272001 >> v1:172.21.15.192:6789/0 conn(0x7f31f80cfa00 legacy=0x7f31f80cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.041+0000 7f31f6ffd700 1 -- 172.21.15.189:0/595272001 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f80cbd80 con 0x7f31f80cb870 2024-04-06T13:49:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.041+0000 7f31f77fe700 1 --2- 172.21.15.189:0/595272001 >> v2:172.21.15.192:3300/0 conn(0x7f31f80d11b0 0x7f31f80d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.041+0000 7f31f6ffd700 1 --2- 172.21.15.189:0/595272001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f80cb870 0x7f31f815c280 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7f31f80ced00 tx=0x7f31e8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad4276a997f1c868 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.041+0000 7f31f67fc700 1 -- 172.21.15.189:0/595272001 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e8004230 con 0x7f31f80cb870 2024-04-06T13:49:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.041+0000 7f31f67fc700 1 -- 172.21.15.189:0/595272001 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e8004390 con 0x7f31f80cb870 2024-04-06T13:49:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.042+0000 7f31f67fc700 1 -- 172.21.15.189:0/595272001 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e80036a0 con 0x7f31f80cb870 2024-04-06T13:49:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.042+0000 7f31fde36700 1 -- 172.21.15.189:0/595272001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f80cb870 msgr2=0x7f31f815c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.042+0000 7f31fde36700 1 --2- 172.21.15.189:0/595272001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f80cb870 0x7f31f815c280 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7f31f80ced00 tx=0x7f31e8009bd0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.042+0000 7f31fde36700 1 -- 172.21.15.189:0/595272001 shutdown_connections 2024-04-06T13:49:09.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.042+0000 7f31fde36700 1 --2- 172.21.15.189:0/595272001 >> v2:172.21.15.192:3300/0 conn(0x7f31f80d11b0 0x7f31f80d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.042+0000 7f31fde36700 1 --2- 172.21.15.189:0/595272001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f80cb870 0x7f31f815c280 unknown :-1 s=CLOSED pgs=2878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.042+0000 7f31fde36700 1 -- 172.21.15.189:0/595272001 >> 172.21.15.189:0/595272001 conn(0x7f31f80b2db0 msgr2=0x7f31f80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:09.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.042+0000 7f31fde36700 1 -- 172.21.15.189:0/595272001 shutdown_connections 2024-04-06T13:49:09.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.042+0000 7f31fde36700 1 -- 172.21.15.189:0/595272001 wait complete. 2024-04-06T13:49:09.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.043+0000 7f31fde36700 1 Processor -- start 2024-04-06T13:49:09.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.043+0000 7f31fde36700 1 -- start start 2024-04-06T13:49:09.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.044+0000 7f31fde36700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f80cb870 0x7f31f80c5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:09.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.044+0000 7f31fde36700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31f80cfa00 0x7f31f81542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:09.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.044+0000 7f31fde36700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31f80d11b0 0x7f31f8154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:09.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.044+0000 7f31fde36700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f8154d40 con 0x7f31f80cb870 2024-04-06T13:49:09.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.044+0000 7f31fde36700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f8158960 con 0x7f31f80cfa00 2024-04-06T13:49:09.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.044+0000 7f31fde36700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f80b4c00 con 0x7f31f80d11b0 2024-04-06T13:49:09.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.044+0000 7f31f77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f80cb870 0x7f31f80c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:09.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.044+0000 7f31f77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f80cb870 0x7f31f80c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:37398/0 (socket says 172.21.15.189:37398) 2024-04-06T13:49:09.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.044+0000 7f31f77fe700 1 -- 172.21.15.189:0/3383693797 learned_addr learned my addr 172.21.15.189:0/3383693797 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:09.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.044+0000 7f31f6ffd700 1 --2- 172.21.15.189:0/3383693797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31f80cfa00 0x7f31f81542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:09.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.044+0000 7f31f7fff700 1 --2- 172.21.15.189:0/3383693797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31f80d11b0 0x7f31f8154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:09.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.045+0000 7f31f77fe700 1 -- 172.21.15.189:0/3383693797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31f80d11b0 msgr2=0x7f31f8154800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:09.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.045+0000 7f31f77fe700 1 --2- 172.21.15.189:0/3383693797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31f80d11b0 0x7f31f8154800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.045+0000 7f31f77fe700 1 -- 172.21.15.189:0/3383693797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31f80cfa00 msgr2=0x7f31f81542c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:09.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.045+0000 7f31f77fe700 1 --2- 172.21.15.189:0/3383693797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31f80cfa00 0x7f31f81542c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.045+0000 7f31f77fe700 1 -- 172.21.15.189:0/3383693797 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e8009550 con 0x7f31f80cb870 2024-04-06T13:49:09.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.045+0000 7f31f6ffd700 1 --2- 172.21.15.189:0/3383693797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31f80cfa00 0x7f31f81542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:09.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.045+0000 7f31f7fff700 1 --2- 172.21.15.189:0/3383693797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31f80d11b0 0x7f31f8154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:09.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.045+0000 7f31f77fe700 1 --2- 172.21.15.189:0/3383693797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f80cb870 0x7f31f80c5340 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 crypto rx=0x7f31ec00c750 tx=0x7f31ec00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:09.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.045+0000 7f31dffff700 1 -- 172.21.15.189:0/3383693797 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec011800 con 0x7f31f80cb870 2024-04-06T13:49:09.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.045+0000 7f31dffff700 1 -- 172.21.15.189:0/3383693797 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ec011960 con 0x7f31f80cb870 2024-04-06T13:49:09.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.045+0000 7f31fde36700 1 -- 172.21.15.189:0/3383693797 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f80b4ee0 con 0x7f31f80cb870 2024-04-06T13:49:09.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.045+0000 7f31dffff700 1 -- 172.21.15.189:0/3383693797 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec011cd0 con 0x7f31f80cb870 2024-04-06T13:49:09.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.045+0000 7f31fde36700 1 -- 172.21.15.189:0/3383693797 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f80b54a0 con 0x7f31f80cb870 2024-04-06T13:49:09.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.046+0000 7f31fde36700 1 -- 172.21.15.189:0/3383693797 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f8074990 con 0x7f31f80cb870 2024-04-06T13:49:09.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.047+0000 7f31dffff700 1 -- 172.21.15.189:0/3383693797 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f31ec021350 con 0x7f31f80cb870 2024-04-06T13:49:09.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.047+0000 7f31dffff700 1 --2- 172.21.15.189:0/3383693797 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f31e406aed0 0x7f31e406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:09.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.047+0000 7f31dffff700 1 -- 172.21.15.189:0/3383693797 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 6195+0+0 (secure 0 0 0) 0x7f31ec089980 con 0x7f31f80cb870 2024-04-06T13:49:09.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.047+0000 7f31f6ffd700 1 --2- 172.21.15.189:0/3383693797 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f31e406aed0 0x7f31e406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:09.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.047+0000 7f31f6ffd700 1 --2- 172.21.15.189:0/3383693797 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f31e406aed0 0x7f31e406d390 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f31f80cbcb0 tx=0x7f31e8008030 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:09.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.051+0000 7f31dffff700 1 -- 172.21.15.189:0/3383693797 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f31ec054590 con 0x7f31f80cb870 2024-04-06T13:49:09.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.245+0000 7f31fde36700 1 -- 172.21.15.189:0/3383693797 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f31f80cb630 con 0x7f31f80cb870 2024-04-06T13:49:09.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.249+0000 7f31dffff700 1 -- 172.21.15.189:0/3383693797 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v85) v1 ==== 104+0+0 (secure 0 0 0) 0x7f31ec057bb0 con 0x7f31f80cb870 2024-04-06T13:49:09.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.251+0000 7f31fde36700 1 -- 172.21.15.189:0/3383693797 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f31e406aed0 msgr2=0x7f31e406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:09.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.251+0000 7f31fde36700 1 --2- 172.21.15.189:0/3383693797 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f31e406aed0 0x7f31e406d390 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f31f80cbcb0 tx=0x7f31e8008030 comp rx=0 tx=0).stop 2024-04-06T13:49:09.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.251+0000 7f31fde36700 1 -- 172.21.15.189:0/3383693797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f80cb870 msgr2=0x7f31f80c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:09.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.251+0000 7f31fde36700 1 --2- 172.21.15.189:0/3383693797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f80cb870 0x7f31f80c5340 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 crypto rx=0x7f31ec00c750 tx=0x7f31ec00cb10 comp rx=0 tx=0).stop 2024-04-06T13:49:09.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.251+0000 7f31fde36700 1 -- 172.21.15.189:0/3383693797 shutdown_connections 2024-04-06T13:49:09.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.251+0000 7f31fde36700 1 --2- 172.21.15.189:0/3383693797 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f31e406aed0 0x7f31e406d390 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.251+0000 7f31fde36700 1 --2- 172.21.15.189:0/3383693797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31f80d11b0 0x7f31f8154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.251+0000 7f31fde36700 1 --2- 172.21.15.189:0/3383693797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f80cb870 0x7f31f80c5340 unknown :-1 s=CLOSED pgs=2879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.251+0000 7f31fde36700 1 --2- 172.21.15.189:0/3383693797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31f80cfa00 0x7f31f81542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.251+0000 7f31fde36700 1 -- 172.21.15.189:0/3383693797 >> 172.21.15.189:0/3383693797 conn(0x7f31f80b2db0 msgr2=0x7f31f815a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:09.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.251+0000 7f31fde36700 1 -- 172.21.15.189:0/3383693797 shutdown_connections 2024-04-06T13:49:09.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.251+0000 7f31fde36700 1 -- 172.21.15.189:0/3383693797 wait complete. 2024-04-06T13:49:09.263 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-06T13:49:09.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.387+0000 7efd91262700 1 Processor -- start 2024-04-06T13:49:09.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.387+0000 7efd91262700 1 -- start start 2024-04-06T13:49:09.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.388+0000 7efd91262700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd8c15d610 0x7efd8c0b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:09.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.388+0000 7efd91262700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efd8c06abb0 0x7efd8c161810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:09.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.388+0000 7efd91262700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd8c15d9f0 con 0x7efd8c06a730 2024-04-06T13:49:09.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.388+0000 7efd91262700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd8c0d1040 con 0x7efd8c06abb0 2024-04-06T13:49:09.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.388+0000 7efd91262700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c072760 con 0x7efd8c15d610 2024-04-06T13:49:09.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.388+0000 7efd8a7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efd8c06a730 0x7efd8c161100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48824/0 (socket says 172.21.15.189:48824) 2024-04-06T13:49:09.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.388+0000 7efd8a7fc700 1 -- 172.21.15.189:0/2735849482 learned_addr learned my addr 172.21.15.189:0/2735849482 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:09.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.388+0000 7efd8affd700 1 --2- 172.21.15.189:0/2735849482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd8c15d610 0x7efd8c0b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:09.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.388+0000 7efd8b7fe700 1 --2- 172.21.15.189:0/2735849482 >> v2:172.21.15.192:3300/0 conn(0x7efd8c06abb0 0x7efd8c161810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:09.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.388+0000 7efd89ffb700 1 -- 172.21.15.189:0/2735849482 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224421775 0 0) 0x7efd8c15d9f0 con 0x7efd8c06a730 2024-04-06T13:49:09.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.388+0000 7efd89ffb700 1 -- 172.21.15.189:0/2735849482 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd740035b0 con 0x7efd8c06a730 2024-04-06T13:49:09.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.388+0000 7efd89ffb700 1 -- 172.21.15.189:0/2735849482 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3377407364 0 0) 0x7efd740035b0 con 0x7efd8c06a730 2024-04-06T13:49:09.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.388+0000 7efd8affd700 1 -- 172.21.15.189:0/2735849482 >> v2:172.21.15.192:3300/0 conn(0x7efd8c06abb0 msgr2=0x7efd8c161810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:09.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.388+0000 7efd8affd700 1 --2- 172.21.15.189:0/2735849482 >> v2:172.21.15.192:3300/0 conn(0x7efd8c06abb0 0x7efd8c161810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.388+0000 7efd8affd700 1 -- 172.21.15.189:0/2735849482 >> v1:172.21.15.189:6789/0 conn(0x7efd8c06a730 legacy=0x7efd8c161100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:09.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.388+0000 7efd8affd700 1 -- 172.21.15.189:0/2735849482 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8c06b040 con 0x7efd8c15d610 2024-04-06T13:49:09.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.388+0000 7efd8b7fe700 1 --2- 172.21.15.189:0/2735849482 >> v2:172.21.15.192:3300/0 conn(0x7efd8c06abb0 0x7efd8c161810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:09.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.388+0000 7efd8affd700 1 --2- 172.21.15.189:0/2735849482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd8c15d610 0x7efd8c0b3de0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7efd7c009840 tx=0x7efd7c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ce5c657eeebc353 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:09.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.389+0000 7efd89ffb700 1 -- 172.21.15.189:0/2735849482 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd80001670 con 0x7efd8c15d610 2024-04-06T13:49:09.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.389+0000 7efd89ffb700 1 -- 172.21.15.189:0/2735849482 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7c004030 con 0x7efd8c15d610 2024-04-06T13:49:09.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.389+0000 7efd89ffb700 1 -- 172.21.15.189:0/2735849482 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7c0043c0 con 0x7efd8c15d610 2024-04-06T13:49:09.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.389+0000 7efd91262700 1 -- 172.21.15.189:0/2735849482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd8c15d610 msgr2=0x7efd8c0b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:09.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.389+0000 7efd91262700 1 --2- 172.21.15.189:0/2735849482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd8c15d610 0x7efd8c0b3de0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7efd7c009840 tx=0x7efd7c009b50 comp rx=0 tx=0).stop 2024-04-06T13:49:09.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.389+0000 7efd91262700 1 -- 172.21.15.189:0/2735849482 shutdown_connections 2024-04-06T13:49:09.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.389+0000 7efd91262700 1 --2- 172.21.15.189:0/2735849482 >> v2:172.21.15.192:3300/0 conn(0x7efd8c06abb0 0x7efd8c161810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.389+0000 7efd91262700 1 --2- 172.21.15.189:0/2735849482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd8c15d610 0x7efd8c0b3de0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.389+0000 7efd91262700 1 -- 172.21.15.189:0/2735849482 >> 172.21.15.189:0/2735849482 conn(0x7efd8c0d00b0 msgr2=0x7efd8c0d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:09.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.389+0000 7efd91262700 1 -- 172.21.15.189:0/2735849482 shutdown_connections 2024-04-06T13:49:09.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.389+0000 7efd91262700 1 -- 172.21.15.189:0/2735849482 wait complete. 2024-04-06T13:49:09.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.391+0000 7efd91262700 1 Processor -- start 2024-04-06T13:49:09.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.391+0000 7efd91262700 1 -- start start 2024-04-06T13:49:09.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.391+0000 7efd91262700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd8c06a730 0x7efd8c14ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:09.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.391+0000 7efd91262700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd8c06abb0 0x7efd8c0ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:09.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.391+0000 7efd91262700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd8c15d610 0x7efd8c0cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:09.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.391+0000 7efd91262700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c14bfa0 con 0x7efd8c15d610 2024-04-06T13:49:09.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.391+0000 7efd91262700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c0cc860 con 0x7efd8c06abb0 2024-04-06T13:49:09.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.391+0000 7efd91262700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c0c6af0 con 0x7efd8c06a730 2024-04-06T13:49:09.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.391+0000 7efd8b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd8c15d610 0x7efd8c0cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:09.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.392+0000 7efd8b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd8c15d610 0x7efd8c0cad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60712/0 (socket says 172.21.15.189:60712) 2024-04-06T13:49:09.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.392+0000 7efd8b7fe700 1 -- 172.21.15.189:0/1374850336 learned_addr learned my addr 172.21.15.189:0/1374850336 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:09.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.392+0000 7efd8affd700 1 --2- 172.21.15.189:0/1374850336 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd8c06a730 0x7efd8c14ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:09.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.392+0000 7efd8a7fc700 1 --2- 172.21.15.189:0/1374850336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd8c06abb0 0x7efd8c0ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:09.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.392+0000 7efd8b7fe700 1 -- 172.21.15.189:0/1374850336 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd8c06a730 msgr2=0x7efd8c14ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:09.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.392+0000 7efd8b7fe700 1 --2- 172.21.15.189:0/1374850336 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd8c06a730 0x7efd8c14ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.392+0000 7efd8b7fe700 1 -- 172.21.15.189:0/1374850336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd8c06abb0 msgr2=0x7efd8c0ca800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:09.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.392+0000 7efd8b7fe700 1 --2- 172.21.15.189:0/1374850336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd8c06abb0 0x7efd8c0ca800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.392+0000 7efd8b7fe700 1 -- 172.21.15.189:0/1374850336 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd84009a70 con 0x7efd8c15d610 2024-04-06T13:49:09.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.392+0000 7efd8b7fe700 1 --2- 172.21.15.189:0/1374850336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd8c15d610 0x7efd8c0cad40 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7efd8400efc0 tx=0x7efd8400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:09.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.392+0000 7efd737fe700 1 -- 172.21.15.189:0/1374850336 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd84009d70 con 0x7efd8c15d610 2024-04-06T13:49:09.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.392+0000 7efd737fe700 1 -- 172.21.15.189:0/1374850336 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8400f040 con 0x7efd8c15d610 2024-04-06T13:49:09.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.392+0000 7efd91262700 1 -- 172.21.15.189:0/1374850336 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7c0094a0 con 0x7efd8c15d610 2024-04-06T13:49:09.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.392+0000 7efd737fe700 1 -- 172.21.15.189:0/1374850336 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8400ce00 con 0x7efd8c15d610 2024-04-06T13:49:09.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.392+0000 7efd91262700 1 -- 172.21.15.189:0/1374850336 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd8c14c680 con 0x7efd8c15d610 2024-04-06T13:49:09.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.393+0000 7efd91262700 1 -- 172.21.15.189:0/1374850336 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd8c074990 con 0x7efd8c15d610 2024-04-06T13:49:09.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.394+0000 7efd737fe700 1 -- 172.21.15.189:0/1374850336 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7efd84014480 con 0x7efd8c15d610 2024-04-06T13:49:09.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.394+0000 7efd737fe700 1 --2- 172.21.15.189:0/1374850336 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efd7806aed0 0x7efd7806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:09.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.394+0000 7efd737fe700 1 -- 172.21.15.189:0/1374850336 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 6195+0+0 (secure 0 0 0) 0x7efd8408a340 con 0x7efd8c15d610 2024-04-06T13:49:09.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.394+0000 7efd8affd700 1 --2- 172.21.15.189:0/1374850336 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efd7806aed0 0x7efd7806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:09.396 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.394+0000 7efd8affd700 1 --2- 172.21.15.189:0/1374850336 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efd7806aed0 0x7efd7806d390 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efd7c000c00 tx=0x7efd7c013040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:09.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.399+0000 7efd737fe700 1 -- 172.21.15.189:0/1374850336 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efd84054db0 con 0x7efd8c15d610 2024-04-06T13:49:09.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.597+0000 7efd91262700 1 -- 172.21.15.189:0/1374850336 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7efd8c0cfaa0 con 0x7efd8c15d610 2024-04-06T13:49:09.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.601+0000 7efd737fe700 1 -- 172.21.15.189:0/1374850336 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v86) v1 ==== 115+0+0 (secure 0 0 0) 0x7efd840583d0 con 0x7efd8c15d610 2024-04-06T13:49:09.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.604+0000 7efd91262700 1 -- 172.21.15.189:0/1374850336 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efd7806aed0 msgr2=0x7efd7806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:09.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.604+0000 7efd91262700 1 --2- 172.21.15.189:0/1374850336 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efd7806aed0 0x7efd7806d390 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efd7c000c00 tx=0x7efd7c013040 comp rx=0 tx=0).stop 2024-04-06T13:49:09.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.604+0000 7efd91262700 1 -- 172.21.15.189:0/1374850336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd8c15d610 msgr2=0x7efd8c0cad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:09.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.604+0000 7efd91262700 1 --2- 172.21.15.189:0/1374850336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd8c15d610 0x7efd8c0cad40 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7efd8400efc0 tx=0x7efd8400c3a0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.604+0000 7efd91262700 1 -- 172.21.15.189:0/1374850336 shutdown_connections 2024-04-06T13:49:09.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.604+0000 7efd91262700 1 --2- 172.21.15.189:0/1374850336 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efd7806aed0 0x7efd7806d390 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.604+0000 7efd91262700 1 --2- 172.21.15.189:0/1374850336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd8c15d610 0x7efd8c0cad40 unknown :-1 s=CLOSED pgs=2881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.604+0000 7efd91262700 1 --2- 172.21.15.189:0/1374850336 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd8c06a730 0x7efd8c14ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.604+0000 7efd91262700 1 --2- 172.21.15.189:0/1374850336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd8c06abb0 0x7efd8c0ca800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.604+0000 7efd91262700 1 -- 172.21.15.189:0/1374850336 >> 172.21.15.189:0/1374850336 conn(0x7efd8c0d00b0 msgr2=0x7efd8c0ac8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:09.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.604+0000 7efd91262700 1 -- 172.21.15.189:0/1374850336 shutdown_connections 2024-04-06T13:49:09.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.604+0000 7efd91262700 1 -- 172.21.15.189:0/1374850336 wait complete. 2024-04-06T13:49:09.617 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-06T13:49:09.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.743+0000 7fd5e2cb3700 1 Processor -- start 2024-04-06T13:49:09.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.743+0000 7fd5e2cb3700 1 -- start start 2024-04-06T13:49:09.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.744+0000 7fd5e2cb3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd5dc15a550 0x7fd5dc0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:09.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.745+0000 7fd5e2cb3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5dc0cdbe0 con 0x7fd5dc0b4d70 2024-04-06T13:49:09.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.745+0000 7fd5e2cb3700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5dc15a940 con 0x7fd5dc0cf900 2024-04-06T13:49:09.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.745+0000 7fd5e2cb3700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5dc15ac00 con 0x7fd5dc15a550 2024-04-06T13:49:09.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.745+0000 7fd5e0a4f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd5dc15a550 0x7fd5dc0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:09.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.745+0000 7fd5e1250700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd5dc0b4d70 0x7fd5dc0b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48836/0 (socket says 172.21.15.189:48836) 2024-04-06T13:49:09.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.745+0000 7fd5e1250700 1 -- 172.21.15.189:0/3976195129 learned_addr learned my addr 172.21.15.189:0/3976195129 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:09.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.745+0000 7fd5e0a4f700 1 -- 172.21.15.189:0/3976195129 >> v1:172.21.15.192:6789/0 conn(0x7fd5dc0cf900 legacy=0x7fd5dc0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:09.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.745+0000 7fd5e0a4f700 1 -- 172.21.15.189:0/3976195129 >> v1:172.21.15.189:6789/0 conn(0x7fd5dc0b4d70 legacy=0x7fd5dc0b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:09.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.745+0000 7fd5e0a4f700 1 -- 172.21.15.189:0/3976195129 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5dc15aee0 con 0x7fd5dc15a550 2024-04-06T13:49:09.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.745+0000 7fd5e0a4f700 1 --2- 172.21.15.189:0/3976195129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5dc15a550 0x7fd5dc0cf3c0 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009840 tx=0x7fd5d0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708e8c63fb0d7c9b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:09.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.745+0000 7fd5db7fe700 1 -- 172.21.15.189:0/3976195129 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0004230 con 0x7fd5dc15a550 2024-04-06T13:49:09.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.745+0000 7fd5db7fe700 1 -- 172.21.15.189:0/3976195129 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d0004390 con 0x7fd5dc15a550 2024-04-06T13:49:09.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.745+0000 7fd5db7fe700 1 -- 172.21.15.189:0/3976195129 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d00036a0 con 0x7fd5dc15a550 2024-04-06T13:49:09.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.746+0000 7fd5e2cb3700 1 -- 172.21.15.189:0/3976195129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5dc15a550 msgr2=0x7fd5dc0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:09.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.746+0000 7fd5e2cb3700 1 --2- 172.21.15.189:0/3976195129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5dc15a550 0x7fd5dc0cf3c0 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009840 tx=0x7fd5d0009b50 comp rx=0 tx=0).stop 2024-04-06T13:49:09.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.746+0000 7fd5e2cb3700 1 -- 172.21.15.189:0/3976195129 shutdown_connections 2024-04-06T13:49:09.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.746+0000 7fd5e2cb3700 1 --2- 172.21.15.189:0/3976195129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5dc15a550 0x7fd5dc0cf3c0 unknown :-1 s=CLOSED pgs=2883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.746+0000 7fd5e2cb3700 1 -- 172.21.15.189:0/3976195129 >> 172.21.15.189:0/3976195129 conn(0x7fd5dc0d00b0 msgr2=0x7fd5dc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:09.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.746+0000 7fd5e2cb3700 1 -- 172.21.15.189:0/3976195129 shutdown_connections 2024-04-06T13:49:09.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.746+0000 7fd5e2cb3700 1 -- 172.21.15.189:0/3976195129 wait complete. 2024-04-06T13:49:09.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.747+0000 7fd5e2cb3700 1 Processor -- start 2024-04-06T13:49:09.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.747+0000 7fd5e2cb3700 1 -- start start 2024-04-06T13:49:09.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.748+0000 7fd5e2cb3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5dc0b4d70 0x7fd5dc155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:09.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.748+0000 7fd5e2cb3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5dc0cf900 0x7fd5dc0adf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:09.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.748+0000 7fd5e2cb3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5dc0abe50 0x7fd5dc06ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:09.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.748+0000 7fd5e2cb3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc156300 con 0x7fd5dc0abe50 2024-04-06T13:49:09.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.748+0000 7fd5e2cb3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc1f0ae0 con 0x7fd5dc0b4d70 2024-04-06T13:49:09.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.748+0000 7fd5e2cb3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc0ac2e0 con 0x7fd5dc0cf900 2024-04-06T13:49:09.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.748+0000 7fd5e1250700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5dc0abe50 0x7fd5dc06ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:09.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.748+0000 7fd5e1250700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5dc0abe50 0x7fd5dc06ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60734/0 (socket says 172.21.15.189:60734) 2024-04-06T13:49:09.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.748+0000 7fd5e1250700 1 -- 172.21.15.189:0/2677285404 learned_addr learned my addr 172.21.15.189:0/2677285404 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:09.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.748+0000 7fd5dbfff700 1 --2- 172.21.15.189:0/2677285404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5dc0cf900 0x7fd5dc0adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:09.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.748+0000 7fd5e0a4f700 1 --2- 172.21.15.189:0/2677285404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5dc0b4d70 0x7fd5dc155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:09.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.748+0000 7fd5e1250700 1 -- 172.21.15.189:0/2677285404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5dc0cf900 msgr2=0x7fd5dc0adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:09.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.748+0000 7fd5e1250700 1 --2- 172.21.15.189:0/2677285404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5dc0cf900 0x7fd5dc0adf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.748+0000 7fd5e1250700 1 -- 172.21.15.189:0/2677285404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5dc0b4d70 msgr2=0x7fd5dc155dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:09.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.748+0000 7fd5e1250700 1 --2- 172.21.15.189:0/2677285404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5dc0b4d70 0x7fd5dc155dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.748+0000 7fd5e1250700 1 -- 172.21.15.189:0/2677285404 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d00094a0 con 0x7fd5dc0abe50 2024-04-06T13:49:09.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.748+0000 7fd5dbfff700 1 --2- 172.21.15.189:0/2677285404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5dc0cf900 0x7fd5dc0adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:09.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.748+0000 7fd5e0a4f700 1 --2- 172.21.15.189:0/2677285404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5dc0b4d70 0x7fd5dc155dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:09.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.749+0000 7fd5e1250700 1 --2- 172.21.15.189:0/2677285404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5dc0abe50 0x7fd5dc06ac10 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7fd5d400cbc0 tx=0x7fd5d400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:09.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.749+0000 7fd5d97fa700 1 -- 172.21.15.189:0/2677285404 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d400ab50 con 0x7fd5dc0abe50 2024-04-06T13:49:09.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.749+0000 7fd5d97fa700 1 -- 172.21.15.189:0/2677285404 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d400acb0 con 0x7fd5dc0abe50 2024-04-06T13:49:09.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.749+0000 7fd5e2cb3700 1 -- 172.21.15.189:0/2677285404 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5dc0ac5c0 con 0x7fd5dc0abe50 2024-04-06T13:49:09.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.749+0000 7fd5d97fa700 1 -- 172.21.15.189:0/2677285404 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4012600 con 0x7fd5dc0abe50 2024-04-06T13:49:09.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.749+0000 7fd5e2cb3700 1 -- 172.21.15.189:0/2677285404 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5dc0acb80 con 0x7fd5dc0abe50 2024-04-06T13:49:09.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.750+0000 7fd5e2cb3700 1 -- 172.21.15.189:0/2677285404 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5dc1f0d30 con 0x7fd5dc0abe50 2024-04-06T13:49:09.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.750+0000 7fd5d97fa700 1 -- 172.21.15.189:0/2677285404 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fd5d401b3b0 con 0x7fd5dc0abe50 2024-04-06T13:49:09.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.751+0000 7fd5d97fa700 1 --2- 172.21.15.189:0/2677285404 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd5c006aed0 0x7fd5c006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:09.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.751+0000 7fd5d97fa700 1 -- 172.21.15.189:0/2677285404 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 6195+0+0 (secure 0 0 0) 0x7fd5d4089600 con 0x7fd5dc0abe50 2024-04-06T13:49:09.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.751+0000 7fd5e0a4f700 1 --2- 172.21.15.189:0/2677285404 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd5c006aed0 0x7fd5c006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:09.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.751+0000 7fd5e0a4f700 1 --2- 172.21.15.189:0/2677285404 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd5c006aed0 0x7fd5c006d390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd5d00090b0 tx=0x7fd5d0008030 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:09.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.755+0000 7fd5d97fa700 1 -- 172.21.15.189:0/2677285404 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd5dc1f0d30 con 0x7fd5dc0abe50 2024-04-06T13:49:09.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.955+0000 7fd5e2cb3700 1 -- 172.21.15.189:0/2677285404 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fd5dc1f0d30 con 0x7fd5dc0abe50 2024-04-06T13:49:09.961 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.961+0000 7fd5d97fa700 1 -- 172.21.15.189:0/2677285404 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v87) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd5dc1f0d30 con 0x7fd5dc0abe50 2024-04-06T13:49:09.964 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.964+0000 7fd5e2cb3700 1 -- 172.21.15.189:0/2677285404 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd5c006aed0 msgr2=0x7fd5c006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:09.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.964+0000 7fd5e2cb3700 1 --2- 172.21.15.189:0/2677285404 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd5c006aed0 0x7fd5c006d390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd5d00090b0 tx=0x7fd5d0008030 comp rx=0 tx=0).stop 2024-04-06T13:49:09.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.964+0000 7fd5e2cb3700 1 -- 172.21.15.189:0/2677285404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5dc0abe50 msgr2=0x7fd5dc06ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:09.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.964+0000 7fd5e2cb3700 1 --2- 172.21.15.189:0/2677285404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5dc0abe50 0x7fd5dc06ac10 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7fd5d400cbc0 tx=0x7fd5d400cf80 comp rx=0 tx=0).stop 2024-04-06T13:49:09.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.964+0000 7fd5e2cb3700 1 -- 172.21.15.189:0/2677285404 shutdown_connections 2024-04-06T13:49:09.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.964+0000 7fd5e2cb3700 1 --2- 172.21.15.189:0/2677285404 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd5c006aed0 0x7fd5c006d390 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.964+0000 7fd5e2cb3700 1 --2- 172.21.15.189:0/2677285404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5dc0abe50 0x7fd5dc06ac10 unknown :-1 s=CLOSED pgs=2884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.964+0000 7fd5e2cb3700 1 --2- 172.21.15.189:0/2677285404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5dc0b4d70 0x7fd5dc155dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.964+0000 7fd5e2cb3700 1 --2- 172.21.15.189:0/2677285404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd5dc0cf900 0x7fd5dc0adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:09.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.964+0000 7fd5e2cb3700 1 -- 172.21.15.189:0/2677285404 >> 172.21.15.189:0/2677285404 conn(0x7fd5dc0d00b0 msgr2=0x7fd5dc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:09.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.964+0000 7fd5e2cb3700 1 -- 172.21.15.189:0/2677285404 shutdown_connections 2024-04-06T13:49:09.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:09.964+0000 7fd5e2cb3700 1 -- 172.21.15.189:0/2677285404 wait complete. 2024-04-06T13:49:09.977 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2024-04-06T13:49:10.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.099+0000 7fbb35bae700 1 Processor -- start 2024-04-06T13:49:10.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.099+0000 7fbb35bae700 1 -- start start 2024-04-06T13:49:10.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.100+0000 7fbb35bae700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb3006ae40 0x7fbb300b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:10.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.100+0000 7fbb35bae700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbb3015ddf0 0x7fbb3015e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:10.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.100+0000 7fbb35bae700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb300b3990 con 0x7fbb3015d7b0 2024-04-06T13:49:10.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.100+0000 7fbb35bae700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb300b3060 con 0x7fbb3015ddf0 2024-04-06T13:49:10.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.100+0000 7fbb35bae700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb300b3340 con 0x7fbb3006ae40 2024-04-06T13:49:10.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.100+0000 7fbb2effd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb3006ae40 0x7fbb300b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:10.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.100+0000 7fbb2ffff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbb3015ddf0 0x7fbb3015e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:10.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.100+0000 7fbb2f7fe700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fbb3015d7b0 0x7fbb3006a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:33100/0 (socket says 172.21.15.189:33100) 2024-04-06T13:49:10.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.100+0000 7fbb2effd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb3006ae40 0x7fbb300b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:48062/0 (socket says 172.21.15.189:48062) 2024-04-06T13:49:10.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.100+0000 7fbb2f7fe700 1 -- 172.21.15.189:0/2696027840 learned_addr learned my addr 172.21.15.189:0/2696027840 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:10.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.100+0000 7fbb2ffff700 1 -- 172.21.15.189:0/2696027840 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb3006ae40 msgr2=0x7fbb300b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:10.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.100+0000 7fbb2ffff700 1 --2- 172.21.15.189:0/2696027840 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb3006ae40 0x7fbb300b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:10.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.100+0000 7fbb2ffff700 1 -- 172.21.15.189:0/2696027840 >> v1:172.21.15.192:6789/0 conn(0x7fbb3015d7b0 legacy=0x7fbb3006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:10.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.100+0000 7fbb2ffff700 1 -- 172.21.15.189:0/2696027840 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb300b3620 con 0x7fbb3015ddf0 2024-04-06T13:49:10.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.100+0000 7fbb2e7fc700 1 -- 172.21.15.189:0/2696027840 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116444691 0 0) 0x7fbb300b3990 con 0x7fbb3015d7b0 2024-04-06T13:49:10.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.100+0000 7fbb2effd700 1 --2- 172.21.15.189:0/2696027840 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb3006ae40 0x7fbb300b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:10.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.101+0000 7fbb2ffff700 1 --2- 172.21.15.189:0/2696027840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb3015ddf0 0x7fbb3015e2a0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009d90 tx=0x7fbb2800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6118ae83327126b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:10.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.101+0000 7fbb2e7fc700 1 -- 172.21.15.189:0/2696027840 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2800e070 con 0x7fbb3015ddf0 2024-04-06T13:49:10.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.101+0000 7fbb2e7fc700 1 -- 172.21.15.189:0/2696027840 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2800bb20 con 0x7fbb3015ddf0 2024-04-06T13:49:10.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.101+0000 7fbb2e7fc700 1 -- 172.21.15.189:0/2696027840 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb28012430 con 0x7fbb3015ddf0 2024-04-06T13:49:10.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.101+0000 7fbb35bae700 1 -- 172.21.15.189:0/2696027840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb3015ddf0 msgr2=0x7fbb3015e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:10.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.101+0000 7fbb35bae700 1 --2- 172.21.15.189:0/2696027840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb3015ddf0 0x7fbb3015e2a0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009d90 tx=0x7fbb2800b330 comp rx=0 tx=0).stop 2024-04-06T13:49:10.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.101+0000 7fbb35bae700 1 -- 172.21.15.189:0/2696027840 shutdown_connections 2024-04-06T13:49:10.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.101+0000 7fbb35bae700 1 --2- 172.21.15.189:0/2696027840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb3015ddf0 0x7fbb3015e2a0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:10.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.101+0000 7fbb35bae700 1 --2- 172.21.15.189:0/2696027840 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb3006ae40 0x7fbb300b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:10.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.101+0000 7fbb35bae700 1 -- 172.21.15.189:0/2696027840 >> 172.21.15.189:0/2696027840 conn(0x7fbb300d00b0 msgr2=0x7fbb300d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:10.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.101+0000 7fbb35bae700 1 -- 172.21.15.189:0/2696027840 shutdown_connections 2024-04-06T13:49:10.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.101+0000 7fbb35bae700 1 -- 172.21.15.189:0/2696027840 wait complete. 2024-04-06T13:49:10.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.103+0000 7fbb35bae700 1 Processor -- start 2024-04-06T13:49:10.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.103+0000 7fbb35bae700 1 -- start start 2024-04-06T13:49:10.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.103+0000 7fbb35bae700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb3006ae40 0x7fbb3014ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:10.107 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.103+0000 7fbb35bae700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb3015d7b0 0x7fbb3015bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:10.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.103+0000 7fbb35bae700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb3015ddf0 0x7fbb300adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:10.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.103+0000 7fbb35bae700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb300af800 con 0x7fbb3006ae40 2024-04-06T13:49:10.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.103+0000 7fbb35bae700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3014b350 con 0x7fbb3015d7b0 2024-04-06T13:49:10.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.103+0000 7fbb35bae700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb300aeb80 con 0x7fbb3015ddf0 2024-04-06T13:49:10.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.104+0000 7fbb2f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb3006ae40 0x7fbb3014ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:10.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.104+0000 7fbb2f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb3006ae40 0x7fbb3014ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60748/0 (socket says 172.21.15.189:60748) 2024-04-06T13:49:10.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.104+0000 7fbb2f7fe700 1 -- 172.21.15.189:0/2386938774 learned_addr learned my addr 172.21.15.189:0/2386938774 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:10.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.104+0000 7fbb2effd700 1 --2- 172.21.15.189:0/2386938774 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb3015d7b0 0x7fbb3015bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:10.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.104+0000 7fbb2ffff700 1 --2- 172.21.15.189:0/2386938774 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb3015ddf0 0x7fbb300adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:10.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.104+0000 7fbb2f7fe700 1 -- 172.21.15.189:0/2386938774 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb3015ddf0 msgr2=0x7fbb300adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:10.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.104+0000 7fbb2f7fe700 1 --2- 172.21.15.189:0/2386938774 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb3015ddf0 0x7fbb300adb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:10.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.104+0000 7fbb2f7fe700 1 -- 172.21.15.189:0/2386938774 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb3015d7b0 msgr2=0x7fbb3015bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:10.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.104+0000 7fbb2f7fe700 1 --2- 172.21.15.189:0/2386938774 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb3015d7b0 0x7fbb3015bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:10.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.104+0000 7fbb2f7fe700 1 -- 172.21.15.189:0/2386938774 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb180094a0 con 0x7fbb3006ae40 2024-04-06T13:49:10.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.104+0000 7fbb2f7fe700 1 --2- 172.21.15.189:0/2386938774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb3006ae40 0x7fbb3014ae10 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7fbb200011e0 tx=0x7fbb2000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:10.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.104+0000 7fbb27fff700 1 -- 172.21.15.189:0/2386938774 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb20004c80 con 0x7fbb3006ae40 2024-04-06T13:49:10.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.104+0000 7fbb27fff700 1 -- 172.21.15.189:0/2386938774 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb20004e00 con 0x7fbb3006ae40 2024-04-06T13:49:10.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.104+0000 7fbb35bae700 1 -- 172.21.15.189:0/2386938774 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb28009a70 con 0x7fbb3006ae40 2024-04-06T13:49:10.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.104+0000 7fbb27fff700 1 -- 172.21.15.189:0/2386938774 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb20007310 con 0x7fbb3006ae40 2024-04-06T13:49:10.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.104+0000 7fbb35bae700 1 -- 172.21.15.189:0/2386938774 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb300af140 con 0x7fbb3006ae40 2024-04-06T13:49:10.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.105+0000 7fbb35bae700 1 -- 172.21.15.189:0/2386938774 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb30067bb0 con 0x7fbb3006ae40 2024-04-06T13:49:10.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.106+0000 7fbb27fff700 1 -- 172.21.15.189:0/2386938774 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fbb20007470 con 0x7fbb3006ae40 2024-04-06T13:49:10.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.106+0000 7fbb27fff700 1 --2- 172.21.15.189:0/2386938774 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbb1006aed0 0x7fbb1006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:10.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.106+0000 7fbb27fff700 1 -- 172.21.15.189:0/2386938774 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 6195+0+0 (secure 0 0 0) 0x7fbb200895f0 con 0x7fbb3006ae40 2024-04-06T13:49:10.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.106+0000 7fbb2effd700 1 --2- 172.21.15.189:0/2386938774 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbb1006aed0 0x7fbb1006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:10.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.107+0000 7fbb2effd700 1 --2- 172.21.15.189:0/2386938774 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbb1006aed0 0x7fbb1006d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbb301f6ae0 tx=0x7fbb18009210 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:10.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.111+0000 7fbb27fff700 1 -- 172.21.15.189:0/2386938774 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbb20054280 con 0x7fbb3006ae40 2024-04-06T13:49:10.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.390+0000 7fbb35bae700 1 -- 172.21.15.189:0/2386938774 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7fbb30074990 con 0x7fbb3006ae40 2024-04-06T13:49:10.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.897+0000 7fbb27fff700 1 -- 172.21.15.189:0/2386938774 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3203) v1 ==== 187+0+0 (secure 0 0 0) 0x7fbb200578a0 con 0x7fbb3006ae40 2024-04-06T13:49:10.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.899+0000 7fbb35bae700 1 -- 172.21.15.189:0/2386938774 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbb1006aed0 msgr2=0x7fbb1006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:10.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.899+0000 7fbb35bae700 1 --2- 172.21.15.189:0/2386938774 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbb1006aed0 0x7fbb1006d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbb301f6ae0 tx=0x7fbb18009210 comp rx=0 tx=0).stop 2024-04-06T13:49:10.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.899+0000 7fbb35bae700 1 -- 172.21.15.189:0/2386938774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb3006ae40 msgr2=0x7fbb3014ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:10.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.899+0000 7fbb35bae700 1 --2- 172.21.15.189:0/2386938774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb3006ae40 0x7fbb3014ae10 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7fbb200011e0 tx=0x7fbb2000ca30 comp rx=0 tx=0).stop 2024-04-06T13:49:10.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.899+0000 7fbb35bae700 1 -- 172.21.15.189:0/2386938774 shutdown_connections 2024-04-06T13:49:10.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.899+0000 7fbb35bae700 1 --2- 172.21.15.189:0/2386938774 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbb1006aed0 0x7fbb1006d390 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:10.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.899+0000 7fbb35bae700 1 --2- 172.21.15.189:0/2386938774 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb3015ddf0 0x7fbb300adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:10.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.899+0000 7fbb35bae700 1 --2- 172.21.15.189:0/2386938774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb3006ae40 0x7fbb3014ae10 unknown :-1 s=CLOSED pgs=2885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:10.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.899+0000 7fbb35bae700 1 --2- 172.21.15.189:0/2386938774 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb3015d7b0 0x7fbb3015bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:10.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.899+0000 7fbb35bae700 1 -- 172.21.15.189:0/2386938774 >> 172.21.15.189:0/2386938774 conn(0x7fbb300d00b0 msgr2=0x7fbb30157110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:10.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.900+0000 7fbb35bae700 1 -- 172.21.15.189:0/2386938774 shutdown_connections 2024-04-06T13:49:10.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:10.900+0000 7fbb35bae700 1 -- 172.21.15.189:0/2386938774 wait complete. 2024-04-06T13:49:10.915 INFO:tasks.cephfs_test_runner:Test command fails without --fs argument in presence of multiple fs ... ok 2024-04-06T13:49:10.916 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2024-04-06T13:49:10.916 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2024-04-06T13:49:10.916 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2024-04-06T13:49:11.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.050+0000 7f6cee9c5700 1 Processor -- start 2024-04-06T13:49:11.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.050+0000 7f6cee9c5700 1 -- start start 2024-04-06T13:49:11.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.051+0000 7f6cee9c5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6ce815e8a0 0x7f6ce80cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:11.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.051+0000 7f6cee9c5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce80669d0 con 0x7f6ce815eee0 2024-04-06T13:49:11.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.051+0000 7f6cee9c5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce806ede0 con 0x7f6ce815fa70 2024-04-06T13:49:11.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.051+0000 7f6cee9c5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce815ec80 con 0x7f6ce815e8a0 2024-04-06T13:49:11.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.051+0000 7f6ce7fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6ce815e8a0 0x7f6ce80cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:11.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.051+0000 7f6ce7fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6ce815e8a0 0x7f6ce80cbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60754/0 (socket says 172.21.15.189:60754) 2024-04-06T13:49:11.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.051+0000 7f6ce7fff700 1 -- 172.21.15.189:0/257180668 learned_addr learned my addr 172.21.15.189:0/257180668 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:11.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.051+0000 7f6ce77fe700 1 --1- 172.21.15.189:0/257180668 >> v1:172.21.15.189:6789/0 conn(0x7f6ce815eee0 0x7f6ce815f360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48838/0 (socket says 172.21.15.189:48838) 2024-04-06T13:49:11.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.051+0000 7f6ce7fff700 1 -- 172.21.15.189:0/257180668 >> v1:172.21.15.192:6789/0 conn(0x7f6ce815fa70 legacy=0x7f6ce806aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:11.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.051+0000 7f6ce7fff700 1 -- 172.21.15.189:0/257180668 >> v1:172.21.15.189:6789/0 conn(0x7f6ce815eee0 legacy=0x7f6ce815f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:11.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.051+0000 7f6ce7fff700 1 -- 172.21.15.189:0/257180668 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce80666f0 con 0x7f6ce815e8a0 2024-04-06T13:49:11.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.051+0000 7f6ce6ffd700 1 -- 172.21.15.189:0/257180668 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037418908 0 0) 0x7f6ce80669d0 con 0x7f6ce815eee0 2024-04-06T13:49:11.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.052+0000 7f6ce7fff700 1 --2- 172.21.15.189:0/257180668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ce815e8a0 0x7f6ce80cbc20 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc0098f0 tx=0x7f6cdc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76eebd6c0bf2c775 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:11.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.052+0000 7f6ce6ffd700 1 -- 172.21.15.189:0/257180668 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc004230 con 0x7f6ce815e8a0 2024-04-06T13:49:11.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.052+0000 7f6ce6ffd700 1 -- 172.21.15.189:0/257180668 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cdc004390 con 0x7f6ce815e8a0 2024-04-06T13:49:11.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.052+0000 7f6ce6ffd700 1 -- 172.21.15.189:0/257180668 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc0036a0 con 0x7f6ce815e8a0 2024-04-06T13:49:11.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.052+0000 7f6cee9c5700 1 -- 172.21.15.189:0/257180668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ce815e8a0 msgr2=0x7f6ce80cbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:11.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.052+0000 7f6cee9c5700 1 --2- 172.21.15.189:0/257180668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ce815e8a0 0x7f6ce80cbc20 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc0098f0 tx=0x7f6cdc009c00 comp rx=0 tx=0).stop 2024-04-06T13:49:11.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.052+0000 7f6cee9c5700 1 -- 172.21.15.189:0/257180668 shutdown_connections 2024-04-06T13:49:11.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.052+0000 7f6cee9c5700 1 --2- 172.21.15.189:0/257180668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ce815e8a0 0x7f6ce80cbc20 unknown :-1 s=CLOSED pgs=2887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:11.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.052+0000 7f6cee9c5700 1 -- 172.21.15.189:0/257180668 >> 172.21.15.189:0/257180668 conn(0x7f6ce80cc290 msgr2=0x7f6ce80cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:11.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.052+0000 7f6cee9c5700 1 -- 172.21.15.189:0/257180668 shutdown_connections 2024-04-06T13:49:11.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.052+0000 7f6cee9c5700 1 -- 172.21.15.189:0/257180668 wait complete. 2024-04-06T13:49:11.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.054+0000 7f6cee9c5700 1 Processor -- start 2024-04-06T13:49:11.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.054+0000 7f6cee9c5700 1 -- start start 2024-04-06T13:49:11.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.055+0000 7f6cee9c5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ce815e8a0 0x7f6ce80c55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:11.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.055+0000 7f6cee9c5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ce815eee0 0x7f6ce8159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:11.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.055+0000 7f6cee9c5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ce815fa70 0x7f6ce81599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:11.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.055+0000 7f6cee9c5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce80aca70 con 0x7f6ce815e8a0 2024-04-06T13:49:11.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.055+0000 7f6cee9c5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce80acd50 con 0x7f6ce815eee0 2024-04-06T13:49:11.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.055+0000 7f6cee9c5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce80ad030 con 0x7f6ce815fa70 2024-04-06T13:49:11.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.055+0000 7f6ce7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ce815e8a0 0x7f6ce80c55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:11.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.055+0000 7f6ce7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ce815e8a0 0x7f6ce80c55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60762/0 (socket says 172.21.15.189:60762) 2024-04-06T13:49:11.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.055+0000 7f6ce7fff700 1 -- 172.21.15.189:0/1564328591 learned_addr learned my addr 172.21.15.189:0/1564328591 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:11.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.055+0000 7f6ce7fff700 1 -- 172.21.15.189:0/1564328591 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ce815fa70 msgr2=0x7f6ce81599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:11.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.055+0000 7f6ce7fff700 1 --2- 172.21.15.189:0/1564328591 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ce815fa70 0x7f6ce81599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.055+0000 7f6ce7fff700 1 -- 172.21.15.189:0/1564328591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ce815eee0 msgr2=0x7f6ce8159460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.055+0000 7f6ce7fff700 1 --2- 172.21.15.189:0/1564328591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ce815eee0 0x7f6ce8159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.055+0000 7f6ce7fff700 1 -- 172.21.15.189:0/1564328591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc009550 con 0x7f6ce815e8a0 2024-04-06T13:49:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.055+0000 7f6ce7fff700 1 --2- 172.21.15.189:0/1564328591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ce815e8a0 0x7f6ce80c55b0 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8073430 tx=0x7f6cdc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.056+0000 7f6ce4ff9700 1 -- 172.21.15.189:0/1564328591 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc003d80 con 0x7f6ce815e8a0 2024-04-06T13:49:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.056+0000 7f6ce4ff9700 1 -- 172.21.15.189:0/1564328591 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cdc01f070 con 0x7f6ce815e8a0 2024-04-06T13:49:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.056+0000 7f6cee9c5700 1 -- 172.21.15.189:0/1564328591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce80ad310 con 0x7f6ce815e8a0 2024-04-06T13:49:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.056+0000 7f6ce4ff9700 1 -- 172.21.15.189:0/1564328591 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc00b8e0 con 0x7f6ce815e8a0 2024-04-06T13:49:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.056+0000 7f6cee9c5700 1 -- 172.21.15.189:0/1564328591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce80c5af0 con 0x7f6ce815e8a0 2024-04-06T13:49:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.057+0000 7f6cee9c5700 1 -- 172.21.15.189:0/1564328591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce8072140 con 0x7f6ce815e8a0 2024-04-06T13:49:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.057+0000 7f6ce4ff9700 1 -- 172.21.15.189:0/1564328591 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f6cdc021030 con 0x7f6ce815e8a0 2024-04-06T13:49:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.058+0000 7f6ce4ff9700 1 --2- 172.21.15.189:0/1564328591 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6cd406aed0 0x7f6cd406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.058+0000 7f6ce77fe700 1 --2- 172.21.15.189:0/1564328591 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6cd406aed0 0x7f6cd406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.058+0000 7f6ce4ff9700 1 -- 172.21.15.189:0/1564328591 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 6195+0+0 (secure 0 0 0) 0x7f6cdc08b000 con 0x7f6ce815e8a0 2024-04-06T13:49:11.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.058+0000 7f6ce77fe700 1 --2- 172.21.15.189:0/1564328591 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6cd406aed0 0x7f6cd406d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6cd80011e0 tx=0x7f6cd80079b0 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:11.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.062+0000 7f6ce4ff9700 1 -- 172.21.15.189:0/1564328591 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6cdc055df0 con 0x7f6ce815e8a0 2024-04-06T13:49:11.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.338+0000 7f6cee9c5700 1 -- 172.21.15.189:0/1564328591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f6ce8072140 con 0x7f6ce815e8a0 2024-04-06T13:49:11.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.900+0000 7f6ce4ff9700 1 -- 172.21.15.189:0/1564328591 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3204) v1 ==== 158+0+0 (secure 0 0 0) 0x7f6cdc059410 con 0x7f6ce815e8a0 2024-04-06T13:49:11.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.902+0000 7f6cee9c5700 1 -- 172.21.15.189:0/1564328591 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6cd406aed0 msgr2=0x7f6cd406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:11.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.902+0000 7f6cee9c5700 1 --2- 172.21.15.189:0/1564328591 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6cd406aed0 0x7f6cd406d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6cd80011e0 tx=0x7f6cd80079b0 comp rx=0 tx=0).stop 2024-04-06T13:49:11.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.902+0000 7f6cee9c5700 1 -- 172.21.15.189:0/1564328591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ce815e8a0 msgr2=0x7f6ce80c55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:11.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.902+0000 7f6cee9c5700 1 --2- 172.21.15.189:0/1564328591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ce815e8a0 0x7f6ce80c55b0 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8073430 tx=0x7f6cdc004750 comp rx=0 tx=0).stop 2024-04-06T13:49:11.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.902+0000 7f6cee9c5700 1 -- 172.21.15.189:0/1564328591 shutdown_connections 2024-04-06T13:49:11.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.902+0000 7f6cee9c5700 1 --2- 172.21.15.189:0/1564328591 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f6cd406aed0 0x7f6cd406d390 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:11.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.902+0000 7f6cee9c5700 1 --2- 172.21.15.189:0/1564328591 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ce815fa70 0x7f6ce81599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:11.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.902+0000 7f6cee9c5700 1 --2- 172.21.15.189:0/1564328591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ce815e8a0 0x7f6ce80c55b0 unknown :-1 s=CLOSED pgs=2888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:11.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.902+0000 7f6cee9c5700 1 --2- 172.21.15.189:0/1564328591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ce815eee0 0x7f6ce8159460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:11.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.902+0000 7f6cee9c5700 1 -- 172.21.15.189:0/1564328591 >> 172.21.15.189:0/1564328591 conn(0x7f6ce80cc290 msgr2=0x7f6ce815d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:11.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.902+0000 7f6cee9c5700 1 -- 172.21.15.189:0/1564328591 shutdown_connections 2024-04-06T13:49:11.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:11.903+0000 7f6cee9c5700 1 -- 172.21.15.189:0/1564328591 wait complete. 2024-04-06T13:49:11.917 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-06T13:49:12.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.049+0000 7f9a4f92d700 1 Processor -- start 2024-04-06T13:49:12.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.049+0000 7f9a4f92d700 1 -- start start 2024-04-06T13:49:12.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a4f92d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a4806ae40 0x7f9a480b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:12.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a4f92d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9a4815ddf0 0x7f9a4815e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:12.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a4f92d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a480b3990 con 0x7f9a4815d7b0 2024-04-06T13:49:12.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a4f92d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a480b3060 con 0x7f9a4815ddf0 2024-04-06T13:49:12.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a4f92d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a480b3340 con 0x7f9a4806ae40 2024-04-06T13:49:12.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a4d6c9700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9a4815d7b0 0x7f9a4806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48846/0 (socket says 172.21.15.189:48846) 2024-04-06T13:49:12.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a4d6c9700 1 -- 172.21.15.189:0/2023678864 learned_addr learned my addr 172.21.15.189:0/2023678864 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:12.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a4cec8700 1 --2- 172.21.15.189:0/2023678864 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a4806ae40 0x7f9a480b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:12.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a4deca700 1 --2- 172.21.15.189:0/2023678864 >> v2:172.21.15.192:3300/0 conn(0x7f9a4815ddf0 0x7f9a4815e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:12.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a3ffff700 1 -- 172.21.15.189:0/2023678864 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493122458 0 0) 0x7f9a480b3990 con 0x7f9a4815d7b0 2024-04-06T13:49:12.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a3ffff700 1 -- 172.21.15.189:0/2023678864 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a340035b0 con 0x7f9a4815d7b0 2024-04-06T13:49:12.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a3ffff700 1 -- 172.21.15.189:0/2023678864 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 204607980 0 0) 0x7f9a340035b0 con 0x7f9a4815d7b0 2024-04-06T13:49:12.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a3ffff700 1 -- 172.21.15.189:0/2023678864 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a4806ae40 msgr2=0x7f9a480b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:12.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a3ffff700 1 --2- 172.21.15.189:0/2023678864 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a4806ae40 0x7f9a480b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a3ffff700 1 -- 172.21.15.189:0/2023678864 >> v2:172.21.15.192:3300/0 conn(0x7f9a4815ddf0 msgr2=0x7f9a4815e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:12.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a3ffff700 1 --2- 172.21.15.189:0/2023678864 >> v2:172.21.15.192:3300/0 conn(0x7f9a4815ddf0 0x7f9a4815e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a3ffff700 1 -- 172.21.15.189:0/2023678864 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a480b3620 con 0x7f9a4815d7b0 2024-04-06T13:49:12.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a3ffff700 1 -- 172.21.15.189:0/2023678864 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f9a44001670 con 0x7f9a4815d7b0 2024-04-06T13:49:12.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a3ffff700 1 -- 172.21.15.189:0/2023678864 >> v1:172.21.15.189:6789/0 conn(0x7f9a4815d7b0 legacy=0x7f9a4806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:12.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.050+0000 7f9a3ffff700 1 --2- 172.21.15.189:0/2023678864 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a4806ae40 0x7f9a34003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:12.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.051+0000 7f9a3ffff700 1 --2- 172.21.15.189:0/2023678864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a34003f80 0x7f9a34004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:12.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.051+0000 7f9a3ffff700 1 --2- 172.21.15.189:0/2023678864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a34004d90 0x7f9a34005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:12.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.051+0000 7f9a3ffff700 1 -- 172.21.15.189:0/2023678864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a480b3060 con 0x7f9a34004d90 2024-04-06T13:49:12.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.051+0000 7f9a3ffff700 1 -- 172.21.15.189:0/2023678864 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a480b3340 con 0x7f9a34003f80 2024-04-06T13:49:12.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.051+0000 7f9a3ffff700 1 -- 172.21.15.189:0/2023678864 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3400ba60 con 0x7f9a4806ae40 2024-04-06T13:49:12.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.051+0000 7f9a4deca700 1 --2- 172.21.15.189:0/2023678864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a34004d90 0x7f9a34005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:12.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.051+0000 7f9a4d6c9700 1 --2- 172.21.15.189:0/2023678864 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a4806ae40 0x7f9a34003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:12.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.051+0000 7f9a4cec8700 1 --2- 172.21.15.189:0/2023678864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a34003f80 0x7f9a34004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:12.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.051+0000 7f9a4deca700 1 -- 172.21.15.189:0/2023678864 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a4806ae40 msgr2=0x7f9a34003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:12.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.051+0000 7f9a4deca700 1 --2- 172.21.15.189:0/2023678864 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a4806ae40 0x7f9a34003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.051+0000 7f9a4deca700 1 -- 172.21.15.189:0/2023678864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a34003f80 msgr2=0x7f9a34004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:12.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.051+0000 7f9a4deca700 1 --2- 172.21.15.189:0/2023678864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a34003f80 0x7f9a34004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.051+0000 7f9a4deca700 1 -- 172.21.15.189:0/2023678864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a340035b0 con 0x7f9a34004d90 2024-04-06T13:49:12.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.051+0000 7f9a4deca700 1 --2- 172.21.15.189:0/2023678864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a34004d90 0x7f9a34005210 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7f9a4000e170 tx=0x7f9a34005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fb9d991c6110996 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:12.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.051+0000 7f9a3ffff700 1 -- 172.21.15.189:0/2023678864 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4000ec90 con 0x7f9a34004d90 2024-04-06T13:49:12.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.051+0000 7f9a3ffff700 1 -- 172.21.15.189:0/2023678864 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4000f040 con 0x7f9a34004d90 2024-04-06T13:49:12.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.051+0000 7f9a3ffff700 1 -- 172.21.15.189:0/2023678864 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a40014480 con 0x7f9a34004d90 2024-04-06T13:49:12.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.052+0000 7f9a4f92d700 1 -- 172.21.15.189:0/2023678864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a34004d90 msgr2=0x7f9a34005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:12.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.052+0000 7f9a4f92d700 1 --2- 172.21.15.189:0/2023678864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a34004d90 0x7f9a34005210 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7f9a4000e170 tx=0x7f9a34005880 comp rx=0 tx=0).stop 2024-04-06T13:49:12.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.052+0000 7f9a4deca700 1 -- 172.21.15.189:0/2023678864 reap_dead start 2024-04-06T13:49:12.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.052+0000 7f9a4f92d700 1 -- 172.21.15.189:0/2023678864 shutdown_connections 2024-04-06T13:49:12.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.052+0000 7f9a4f92d700 1 -- 172.21.15.189:0/2023678864 >> 172.21.15.189:0/2023678864 conn(0x7f9a480d00b0 msgr2=0x7f9a480d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:12.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.052+0000 7f9a4f92d700 1 -- 172.21.15.189:0/2023678864 shutdown_connections 2024-04-06T13:49:12.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.052+0000 7f9a4f92d700 1 -- 172.21.15.189:0/2023678864 wait complete. 2024-04-06T13:49:12.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.053+0000 7f9a4f92d700 1 Processor -- start 2024-04-06T13:49:12.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.053+0000 7f9a4f92d700 1 -- start start 2024-04-06T13:49:12.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.054+0000 7f9a4f92d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a481512a0 0x7f9a4815ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:12.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.054+0000 7f9a4f92d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a481f61e0 0x7f9a48155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:12.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.054+0000 7f9a4f92d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a481f6660 0x7f9a481561d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:12.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.054+0000 7f9a4f92d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a481f6bc0 con 0x7f9a481512a0 2024-04-06T13:49:12.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.054+0000 7f9a4f92d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a481f6ea0 con 0x7f9a481f6660 2024-04-06T13:49:12.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.054+0000 7f9a4f92d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4815a630 con 0x7f9a481f61e0 2024-04-06T13:49:12.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.054+0000 7f9a4d6c9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a481512a0 0x7f9a4815ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:12.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.054+0000 7f9a4d6c9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a481512a0 0x7f9a4815ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60776/0 (socket says 172.21.15.189:60776) 2024-04-06T13:49:12.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.054+0000 7f9a4d6c9700 1 -- 172.21.15.189:0/1638018016 learned_addr learned my addr 172.21.15.189:0/1638018016 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:12.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.055+0000 7f9a4deca700 1 --2- 172.21.15.189:0/1638018016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a481f6660 0x7f9a481561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:12.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.055+0000 7f9a4cec8700 1 --2- 172.21.15.189:0/1638018016 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a481f61e0 0x7f9a48155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:12.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.055+0000 7f9a4d6c9700 1 -- 172.21.15.189:0/1638018016 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a481f61e0 msgr2=0x7f9a48155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:12.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.055+0000 7f9a4d6c9700 1 --2- 172.21.15.189:0/1638018016 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a481f61e0 0x7f9a48155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.055+0000 7f9a4d6c9700 1 -- 172.21.15.189:0/1638018016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a481f6660 msgr2=0x7f9a481561d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:12.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.055+0000 7f9a4d6c9700 1 --2- 172.21.15.189:0/1638018016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a481f6660 0x7f9a481561d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.055+0000 7f9a4d6c9700 1 -- 172.21.15.189:0/1638018016 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4815a910 con 0x7f9a481512a0 2024-04-06T13:49:12.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.055+0000 7f9a4deca700 1 --2- 172.21.15.189:0/1638018016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a481f6660 0x7f9a481561d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:12.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.055+0000 7f9a4cec8700 1 --2- 172.21.15.189:0/1638018016 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a481f61e0 0x7f9a48155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:12.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.055+0000 7f9a4d6c9700 1 --2- 172.21.15.189:0/1638018016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a481512a0 0x7f9a4815ae30 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7f9a44001cc0 tx=0x7f9a4400f4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:12.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.055+0000 7f9a3dffb700 1 -- 172.21.15.189:0/1638018016 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a44007d30 con 0x7f9a481512a0 2024-04-06T13:49:12.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.055+0000 7f9a3dffb700 1 -- 172.21.15.189:0/1638018016 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4400fa90 con 0x7f9a481512a0 2024-04-06T13:49:12.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.055+0000 7f9a4f92d700 1 -- 172.21.15.189:0/1638018016 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a4815b370 con 0x7f9a481512a0 2024-04-06T13:49:12.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.055+0000 7f9a3dffb700 1 -- 172.21.15.189:0/1638018016 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4400fe40 con 0x7f9a481512a0 2024-04-06T13:49:12.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.055+0000 7f9a4f92d700 1 -- 172.21.15.189:0/1638018016 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a480cf800 con 0x7f9a481512a0 2024-04-06T13:49:12.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.056+0000 7f9a4f92d700 1 -- 172.21.15.189:0/1638018016 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a4815a490 con 0x7f9a481512a0 2024-04-06T13:49:12.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.056+0000 7f9a3dffb700 1 -- 172.21.15.189:0/1638018016 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f9a4401d460 con 0x7f9a481512a0 2024-04-06T13:49:12.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.057+0000 7f9a3dffb700 1 --2- 172.21.15.189:0/1638018016 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9a2c06aed0 0x7f9a2c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:12.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.057+0000 7f9a3dffb700 1 -- 172.21.15.189:0/1638018016 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 6195+0+0 (secure 0 0 0) 0x7f9a44089ac0 con 0x7f9a481512a0 2024-04-06T13:49:12.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.057+0000 7f9a4cec8700 1 --2- 172.21.15.189:0/1638018016 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9a2c06aed0 0x7f9a2c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:12.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.057+0000 7f9a4cec8700 1 --2- 172.21.15.189:0/1638018016 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9a2c06aed0 0x7f9a2c06d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9a38006ba0 tx=0x7f9a38008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:12.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.061+0000 7f9a3dffb700 1 -- 172.21.15.189:0/1638018016 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9a4815a490 con 0x7f9a481512a0 2024-04-06T13:49:12.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.255+0000 7f9a4f92d700 1 -- 172.21.15.189:0/1638018016 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9a4815a490 con 0x7f9a481512a0 2024-04-06T13:49:12.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.261+0000 7f9a3dffb700 1 -- 172.21.15.189:0/1638018016 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v88)=0 v88) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9a4815a490 con 0x7f9a481512a0 2024-04-06T13:49:12.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.263+0000 7f9a4f92d700 1 -- 172.21.15.189:0/1638018016 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9a2c06aed0 msgr2=0x7f9a2c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:12.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.263+0000 7f9a4f92d700 1 --2- 172.21.15.189:0/1638018016 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9a2c06aed0 0x7f9a2c06d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9a38006ba0 tx=0x7f9a38008040 comp rx=0 tx=0).stop 2024-04-06T13:49:12.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.263+0000 7f9a4f92d700 1 -- 172.21.15.189:0/1638018016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a481512a0 msgr2=0x7f9a4815ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:12.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.263+0000 7f9a4f92d700 1 --2- 172.21.15.189:0/1638018016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a481512a0 0x7f9a4815ae30 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7f9a44001cc0 tx=0x7f9a4400f4b0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.263+0000 7f9a4f92d700 1 -- 172.21.15.189:0/1638018016 shutdown_connections 2024-04-06T13:49:12.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.263+0000 7f9a4f92d700 1 --2- 172.21.15.189:0/1638018016 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9a2c06aed0 0x7f9a2c06d390 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.263+0000 7f9a4f92d700 1 --2- 172.21.15.189:0/1638018016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a481f6660 0x7f9a481561d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.264 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.263+0000 7f9a4f92d700 1 --2- 172.21.15.189:0/1638018016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a481512a0 0x7f9a4815ae30 unknown :-1 s=CLOSED pgs=2891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.264 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.263+0000 7f9a4f92d700 1 --2- 172.21.15.189:0/1638018016 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a481f61e0 0x7f9a48155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.264 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.263+0000 7f9a4f92d700 1 -- 172.21.15.189:0/1638018016 >> 172.21.15.189:0/1638018016 conn(0x7f9a480d00b0 msgr2=0x7f9a480ae1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:12.264 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.263+0000 7f9a4f92d700 1 -- 172.21.15.189:0/1638018016 shutdown_connections 2024-04-06T13:49:12.264 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.263+0000 7f9a4f92d700 1 -- 172.21.15.189:0/1638018016 wait complete. 2024-04-06T13:49:12.275 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T13:49:12.275 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:49:12.325 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:49:12.374 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T13:49:12.375 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:49:12.375 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:49:12.395 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:49:12.395 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:49:12.395 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:49:12.417 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:49:12.417 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:49:12.417 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:49:12.454 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:49:12.454 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:49:12.455 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:49:12.476 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:49:12.477 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-06T13:49:12.477 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:49:12.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.618+0000 7f4c6766b700 1 Processor -- start 2024-04-06T13:49:12.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.619+0000 7f4c6766b700 1 -- start start 2024-04-06T13:49:12.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.619+0000 7f4c6766b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c6015a550 0x7f4c600cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:12.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.619+0000 7f4c6766b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4c600cf900 0x7f4c600b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:12.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.619+0000 7f4c6766b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c600cdbe0 con 0x7f4c600b4ba0 2024-04-06T13:49:12.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.619+0000 7f4c6766b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c600b5010 con 0x7f4c600cf900 2024-04-06T13:49:12.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.619+0000 7f4c6766b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6015a940 con 0x7f4c6015a550 2024-04-06T13:49:12.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.620+0000 7f4c64c06700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4c600cf900 0x7f4c600b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:12.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.620+0000 7f4c65c08700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4c600b4ba0 0x7f4c6006eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48852/0 (socket says 172.21.15.189:48852) 2024-04-06T13:49:12.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.620+0000 7f4c64c06700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4c600cf900 0x7f4c600b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60786/0 (socket says 172.21.15.189:60786) 2024-04-06T13:49:12.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.620+0000 7f4c65c08700 1 -- 172.21.15.189:0/997088417 learned_addr learned my addr 172.21.15.189:0/997088417 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:12.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.620+0000 7f4c65407700 1 --2- 172.21.15.189:0/997088417 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c6015a550 0x7f4c600cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:12.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.620+0000 7f4c64c06700 1 -- 172.21.15.189:0/997088417 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c6015a550 msgr2=0x7f4c600cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:12.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.620+0000 7f4c64c06700 1 --2- 172.21.15.189:0/997088417 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c6015a550 0x7f4c600cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.620+0000 7f4c64c06700 1 -- 172.21.15.189:0/997088417 >> v1:172.21.15.189:6789/0 conn(0x7f4c600b4ba0 legacy=0x7f4c6006eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:12.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.620+0000 7f4c64c06700 1 -- 172.21.15.189:0/997088417 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c6015ac00 con 0x7f4c600cf900 2024-04-06T13:49:12.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.620+0000 7f4c64c06700 1 --2- 172.21.15.189:0/997088417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c600cf900 0x7f4c600b4660 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7f4c54009840 tx=0x7f4c54009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8d6fef3329fbbcd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:12.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.620+0000 7f4c53fff700 1 -- 172.21.15.189:0/997088417 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54011830 con 0x7f4c600cf900 2024-04-06T13:49:12.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.620+0000 7f4c53fff700 1 -- 172.21.15.189:0/997088417 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c54011990 con 0x7f4c600cf900 2024-04-06T13:49:12.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.620+0000 7f4c53fff700 1 -- 172.21.15.189:0/997088417 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54011d20 con 0x7f4c600cf900 2024-04-06T13:49:12.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.621+0000 7f4c6766b700 1 -- 172.21.15.189:0/997088417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c600cf900 msgr2=0x7f4c600b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:12.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.621+0000 7f4c6766b700 1 --2- 172.21.15.189:0/997088417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c600cf900 0x7f4c600b4660 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7f4c54009840 tx=0x7f4c54009b50 comp rx=0 tx=0).stop 2024-04-06T13:49:12.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.621+0000 7f4c6766b700 1 -- 172.21.15.189:0/997088417 shutdown_connections 2024-04-06T13:49:12.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.621+0000 7f4c6766b700 1 --2- 172.21.15.189:0/997088417 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c6015a550 0x7f4c600cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.621+0000 7f4c6766b700 1 --2- 172.21.15.189:0/997088417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c600cf900 0x7f4c600b4660 unknown :-1 s=CLOSED pgs=2893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.621+0000 7f4c6766b700 1 -- 172.21.15.189:0/997088417 >> 172.21.15.189:0/997088417 conn(0x7f4c600d00b0 msgr2=0x7f4c600d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:12.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.621+0000 7f4c6766b700 1 -- 172.21.15.189:0/997088417 shutdown_connections 2024-04-06T13:49:12.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.621+0000 7f4c6766b700 1 -- 172.21.15.189:0/997088417 wait complete. 2024-04-06T13:49:12.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.622+0000 7f4c6766b700 1 Processor -- start 2024-04-06T13:49:12.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.622+0000 7f4c6766b700 1 -- start start 2024-04-06T13:49:12.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.623+0000 7f4c6766b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c600b4ba0 0x7f4c600aef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:12.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.623+0000 7f4c6766b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c600cf900 0x7f4c600ab090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:12.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.623+0000 7f4c6766b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c60153da0 0x7f4c60158a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:12.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.623+0000 7f4c6766b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c600ab5d0 con 0x7f4c600cf900 2024-04-06T13:49:12.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.623+0000 7f4c6766b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c600ab870 con 0x7f4c600b4ba0 2024-04-06T13:49:12.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.623+0000 7f4c6766b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c600a8550 con 0x7f4c60153da0 2024-04-06T13:49:12.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.623+0000 7f4c64c06700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c600cf900 0x7f4c600ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:12.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.623+0000 7f4c64c06700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c600cf900 0x7f4c600ab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60792/0 (socket says 172.21.15.189:60792) 2024-04-06T13:49:12.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.623+0000 7f4c64c06700 1 -- 172.21.15.189:0/1779437868 learned_addr learned my addr 172.21.15.189:0/1779437868 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:12.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.623+0000 7f4c65407700 1 --2- 172.21.15.189:0/1779437868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c600b4ba0 0x7f4c600aef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:12.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.623+0000 7f4c65c08700 1 --2- 172.21.15.189:0/1779437868 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c60153da0 0x7f4c60158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:12.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.623+0000 7f4c64c06700 1 -- 172.21.15.189:0/1779437868 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c60153da0 msgr2=0x7f4c60158a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:12.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.623+0000 7f4c64c06700 1 --2- 172.21.15.189:0/1779437868 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c60153da0 0x7f4c60158a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.623+0000 7f4c64c06700 1 -- 172.21.15.189:0/1779437868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c600b4ba0 msgr2=0x7f4c600aef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:12.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.623+0000 7f4c64c06700 1 --2- 172.21.15.189:0/1779437868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c600b4ba0 0x7f4c600aef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.623+0000 7f4c64c06700 1 -- 172.21.15.189:0/1779437868 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c540094a0 con 0x7f4c600cf900 2024-04-06T13:49:12.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.623+0000 7f4c64c06700 1 --2- 172.21.15.189:0/1779437868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c600cf900 0x7f4c600ab090 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7f4c54009b90 tx=0x7f4c54004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:12.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.624+0000 7f4c51ffb700 1 -- 172.21.15.189:0/1779437868 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c540036a0 con 0x7f4c600cf900 2024-04-06T13:49:12.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.624+0000 7f4c51ffb700 1 -- 172.21.15.189:0/1779437868 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5401f070 con 0x7f4c600cf900 2024-04-06T13:49:12.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.624+0000 7f4c6766b700 1 -- 172.21.15.189:0/1779437868 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c600a8830 con 0x7f4c600cf900 2024-04-06T13:49:12.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.624+0000 7f4c51ffb700 1 -- 172.21.15.189:0/1779437868 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54003a40 con 0x7f4c600cf900 2024-04-06T13:49:12.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.624+0000 7f4c6766b700 1 -- 172.21.15.189:0/1779437868 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c60154230 con 0x7f4c600cf900 2024-04-06T13:49:12.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.625+0000 7f4c6766b700 1 -- 172.21.15.189:0/1779437868 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c600a8d50 con 0x7f4c600cf900 2024-04-06T13:49:12.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.625+0000 7f4c51ffb700 1 -- 172.21.15.189:0/1779437868 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f4c54003ba0 con 0x7f4c600cf900 2024-04-06T13:49:12.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.626+0000 7f4c51ffb700 1 --2- 172.21.15.189:0/1779437868 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4c4406aed0 0x7f4c4406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:12.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.626+0000 7f4c51ffb700 1 -- 172.21.15.189:0/1779437868 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 6195+0+0 (secure 0 0 0) 0x7f4c54021070 con 0x7f4c600cf900 2024-04-06T13:49:12.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.626+0000 7f4c65407700 1 --2- 172.21.15.189:0/1779437868 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4c4406aed0 0x7f4c4406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:12.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.626+0000 7f4c65407700 1 --2- 172.21.15.189:0/1779437868 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4c4406aed0 0x7f4c4406d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00b9b0 tx=0x7f4c5c008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:12.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.630+0000 7f4c51ffb700 1 -- 172.21.15.189:0/1779437868 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4c600a8d50 con 0x7f4c600cf900 2024-04-06T13:49:12.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.872+0000 7f4c6766b700 1 -- 172.21.15.189:0/1779437868 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c600a8d50 con 0x7f4c600cf900 2024-04-06T13:49:12.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.872+0000 7f4c51ffb700 1 -- 172.21.15.189:0/1779437868 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+3548 (secure 0 0 0) 0x7f4c54015df0 con 0x7f4c600cf900 2024-04-06T13:49:12.872 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:12.872 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":126,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3323503154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3323503154},{"type":"v1","addr":"172.21.15.189:6836","nonce":3323503154}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":9383,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3292345173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3292345173},{"type":"v1","addr":"172.21.15.189:6837","nonce":3292345173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":11284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/1490067369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":1490067369},{"type":"v1","addr":"172.21.15.192:6833","nonce":1490067369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":124},{"gid":11347,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1725391886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1725391886},{"type":"v1","addr":"172.21.15.192:6835","nonce":1725391886}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":126}],"filesystems":[]} 2024-04-06T13:49:12.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.874+0000 7f4c6766b700 1 -- 172.21.15.189:0/1779437868 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4c4406aed0 msgr2=0x7f4c4406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:12.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.874+0000 7f4c6766b700 1 --2- 172.21.15.189:0/1779437868 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4c4406aed0 0x7f4c4406d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00b9b0 tx=0x7f4c5c008040 comp rx=0 tx=0).stop 2024-04-06T13:49:12.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.874+0000 7f4c6766b700 1 -- 172.21.15.189:0/1779437868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c600cf900 msgr2=0x7f4c600ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:12.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.874+0000 7f4c6766b700 1 --2- 172.21.15.189:0/1779437868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c600cf900 0x7f4c600ab090 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7f4c54009b90 tx=0x7f4c54004750 comp rx=0 tx=0).stop 2024-04-06T13:49:12.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.874+0000 7f4c6766b700 1 -- 172.21.15.189:0/1779437868 shutdown_connections 2024-04-06T13:49:12.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.874+0000 7f4c6766b700 1 --2- 172.21.15.189:0/1779437868 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4c4406aed0 0x7f4c4406d390 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.874+0000 7f4c6766b700 1 --2- 172.21.15.189:0/1779437868 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c60153da0 0x7f4c60158a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.874+0000 7f4c6766b700 1 --2- 172.21.15.189:0/1779437868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c600b4ba0 0x7f4c600aef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.874+0000 7f4c6766b700 1 --2- 172.21.15.189:0/1779437868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c600cf900 0x7f4c600ab090 unknown :-1 s=CLOSED pgs=2894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:12.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.874+0000 7f4c6766b700 1 -- 172.21.15.189:0/1779437868 >> 172.21.15.189:0/1779437868 conn(0x7f4c600d00b0 msgr2=0x7f4c6006eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:12.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.874+0000 7f4c6766b700 1 -- 172.21.15.189:0/1779437868 shutdown_connections 2024-04-06T13:49:12.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:12.874+0000 7f4c6766b700 1 -- 172.21.15.189:0/1779437868 wait complete. 2024-04-06T13:49:12.875 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 126 2024-04-06T13:49:12.887 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-06T13:49:12.887 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-06T13:49:12.887 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-06T13:49:12.887 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:49:12.887 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-06T13:49:12.887 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-06T13:49:12.888 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-06T13:49:12.888 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:49:12.888 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-06T13:49:12.888 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-06T13:49:12.888 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-06T13:49:12.888 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:49:12.888 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-06T13:49:12.889 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-06T13:49:12.889 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-06T13:49:12.889 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:49:12.897 INFO:tasks.ceph.mds.a:Stopped 2024-04-06T13:49:12.897 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-06T13:49:12.898 INFO:tasks.ceph.mds.c:Stopped 2024-04-06T13:49:12.898 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-06T13:49:12.900 INFO:tasks.ceph.mds.d:Stopped 2024-04-06T13:49:12.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-06T13:49:12.901 INFO:tasks.ceph.mds.b:Stopped 2024-04-06T13:49:12.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-06T13:49:12.941 INFO:tasks.ceph.mds.a:Started 2024-04-06T13:49:12.941 DEBUG:teuthology.parallel:result is None 2024-04-06T13:49:12.942 INFO:tasks.ceph.mds.c:Started 2024-04-06T13:49:12.942 DEBUG:teuthology.parallel:result is None 2024-04-06T13:49:12.944 INFO:tasks.ceph.mds.d:Started 2024-04-06T13:49:12.944 DEBUG:teuthology.parallel:result is None 2024-04-06T13:49:12.944 INFO:tasks.ceph.mds.b:Started 2024-04-06T13:49:12.945 DEBUG:teuthology.parallel:result is None 2024-04-06T13:49:12.945 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-06T13:49:13.021 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T13:49:13.021 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2024-04-06T13:49:13.031 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T13:49:13.032 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2024-04-06T13:49:13.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.076+0000 7fd18f266700 1 Processor -- start 2024-04-06T13:49:13.076 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.076+0000 7fd18f266700 1 -- start start 2024-04-06T13:49:13.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.077+0000 7fd18f266700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd18815d590 0x7fd1880b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:13.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.077+0000 7fd18f266700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd1880d3890 0x7fd1880b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:13.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.077+0000 7fd18f266700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1880b5cc0 con 0x7fd1880b4ed0 2024-04-06T13:49:13.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.077+0000 7fd18f266700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1880b58c0 con 0x7fd1880d3890 2024-04-06T13:49:13.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.077+0000 7fd18f266700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1880d1500 con 0x7fd18815d590 2024-04-06T13:49:13.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.078+0000 7fd18c801700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd1880b4ed0 0x7fd1880b6580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48856/0 (socket says 172.21.15.189:48856) 2024-04-06T13:49:13.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.078+0000 7fd18c801700 1 -- 172.21.15.189:0/2329383362 learned_addr learned my addr 172.21.15.189:0/2329383362 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:13.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.078+0000 7fd18d803700 1 --2- 172.21.15.189:0/2329383362 >> v2:172.21.15.189:3300/0 conn(0x7fd1880d3890 0x7fd1880b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:13.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.078+0000 7fd18d002700 1 --2- 172.21.15.189:0/2329383362 >> v2:172.21.15.192:3300/0 conn(0x7fd18815d590 0x7fd1880b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:13.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.078+0000 7fd17ffff700 1 -- 172.21.15.189:0/2329383362 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236506434 0 0) 0x7fd1880b5cc0 con 0x7fd1880b4ed0 2024-04-06T13:49:13.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.078+0000 7fd17ffff700 1 -- 172.21.15.189:0/2329383362 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1740035b0 con 0x7fd1880b4ed0 2024-04-06T13:49:13.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.078+0000 7fd18d803700 1 -- 172.21.15.189:0/2329383362 >> v2:172.21.15.192:3300/0 conn(0x7fd18815d590 msgr2=0x7fd1880b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:13.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.078+0000 7fd18d803700 1 --2- 172.21.15.189:0/2329383362 >> v2:172.21.15.192:3300/0 conn(0x7fd18815d590 0x7fd1880b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:13.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.078+0000 7fd18d803700 1 -- 172.21.15.189:0/2329383362 >> v1:172.21.15.189:6789/0 conn(0x7fd1880b4ed0 legacy=0x7fd1880b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:13.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.078+0000 7fd18d803700 1 -- 172.21.15.189:0/2329383362 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1880d17e0 con 0x7fd1880d3890 2024-04-06T13:49:13.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.078+0000 7fd18d803700 1 --2- 172.21.15.189:0/2329383362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1880d3890 0x7fd1880b5380 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7fd1880b5fe0 tx=0x7fd18000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92920e049b66bcb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:13.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.078+0000 7fd17ffff700 1 -- 172.21.15.189:0/2329383362 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18000e070 con 0x7fd1880d3890 2024-04-06T13:49:13.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.078+0000 7fd17ffff700 1 -- 172.21.15.189:0/2329383362 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd18000bb20 con 0x7fd1880d3890 2024-04-06T13:49:13.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.078+0000 7fd17ffff700 1 -- 172.21.15.189:0/2329383362 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd180012430 con 0x7fd1880d3890 2024-04-06T13:49:13.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.079+0000 7fd18f266700 1 -- 172.21.15.189:0/2329383362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1880d3890 msgr2=0x7fd1880b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:13.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.079+0000 7fd18f266700 1 --2- 172.21.15.189:0/2329383362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1880d3890 0x7fd1880b5380 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7fd1880b5fe0 tx=0x7fd18000b330 comp rx=0 tx=0).stop 2024-04-06T13:49:13.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.079+0000 7fd18f266700 1 -- 172.21.15.189:0/2329383362 shutdown_connections 2024-04-06T13:49:13.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.079+0000 7fd18f266700 1 --2- 172.21.15.189:0/2329383362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1880d3890 0x7fd1880b5380 unknown :-1 s=CLOSED pgs=2900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:13.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.079+0000 7fd18f266700 1 --2- 172.21.15.189:0/2329383362 >> v2:172.21.15.192:3300/0 conn(0x7fd18815d590 0x7fd1880b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:13.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.079+0000 7fd18f266700 1 -- 172.21.15.189:0/2329383362 >> 172.21.15.189:0/2329383362 conn(0x7fd18806bbc0 msgr2=0x7fd18806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:13.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.079+0000 7fd18f266700 1 -- 172.21.15.189:0/2329383362 shutdown_connections 2024-04-06T13:49:13.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.079+0000 7fd18f266700 1 -- 172.21.15.189:0/2329383362 wait complete. 2024-04-06T13:49:13.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.080+0000 7fd18f266700 1 Processor -- start 2024-04-06T13:49:13.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.080+0000 7fd18f266700 1 -- start start 2024-04-06T13:49:13.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.081+0000 7fd18f266700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1880b4ed0 0x7fd18814d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:13.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.081+0000 7fd18f266700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1880d3890 0x7fd1880cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:13.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.081+0000 7fd18f266700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd18815d590 0x7fd18815d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:13.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.081+0000 7fd18f266700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1880cb3a0 con 0x7fd1880d3890 2024-04-06T13:49:13.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.081+0000 7fd18f266700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1880cb680 con 0x7fd1880b4ed0 2024-04-06T13:49:13.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.081+0000 7fd18f266700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1880cb960 con 0x7fd18815d590 2024-04-06T13:49:13.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.081+0000 7fd18c801700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1880d3890 0x7fd1880cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:13.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.081+0000 7fd18c801700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1880d3890 0x7fd1880cc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60846/0 (socket says 172.21.15.189:60846) 2024-04-06T13:49:13.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.081+0000 7fd18c801700 1 -- 172.21.15.189:0/485605726 learned_addr learned my addr 172.21.15.189:0/485605726 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:13.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.081+0000 7fd18d002700 1 --2- 172.21.15.189:0/485605726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1880b4ed0 0x7fd18814d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:13.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.081+0000 7fd18c801700 1 -- 172.21.15.189:0/485605726 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd18815d590 msgr2=0x7fd18815d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:13.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.081+0000 7fd18c801700 1 --2- 172.21.15.189:0/485605726 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd18815d590 0x7fd18815d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:13.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.081+0000 7fd18c801700 1 -- 172.21.15.189:0/485605726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1880b4ed0 msgr2=0x7fd18814d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:13.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.081+0000 7fd18c801700 1 --2- 172.21.15.189:0/485605726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1880b4ed0 0x7fd18814d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:13.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.082+0000 7fd18c801700 1 -- 172.21.15.189:0/485605726 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd180009a70 con 0x7fd1880d3890 2024-04-06T13:49:13.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.082+0000 7fd18c801700 1 --2- 172.21.15.189:0/485605726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1880d3890 0x7fd1880cc5f0 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7fd17800c700 tx=0x7fd17800cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:13.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.082+0000 7fd17dffb700 1 -- 172.21.15.189:0/485605726 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd178005d70 con 0x7fd1880d3890 2024-04-06T13:49:13.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.082+0000 7fd17dffb700 1 -- 172.21.15.189:0/485605726 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1780070d0 con 0x7fd1880d3890 2024-04-06T13:49:13.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.082+0000 7fd18f266700 1 -- 172.21.15.189:0/485605726 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1880cbc40 con 0x7fd1880d3890 2024-04-06T13:49:13.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.082+0000 7fd17dffb700 1 -- 172.21.15.189:0/485605726 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd178007460 con 0x7fd1880d3890 2024-04-06T13:49:13.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.082+0000 7fd18f266700 1 -- 172.21.15.189:0/485605726 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1880ccb30 con 0x7fd1880d3890 2024-04-06T13:49:13.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.083+0000 7fd18f266700 1 -- 172.21.15.189:0/485605726 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1880d1d80 con 0x7fd1880d3890 2024-04-06T13:49:13.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.083+0000 7fd17dffb700 1 -- 172.21.15.189:0/485605726 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fd1780075c0 con 0x7fd1880d3890 2024-04-06T13:49:13.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.084+0000 7fd17dffb700 1 --2- 172.21.15.189:0/485605726 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd16c06aed0 0x7fd16c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:13.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.084+0000 7fd17dffb700 1 -- 172.21.15.189:0/485605726 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6195+0+0 (secure 0 0 0) 0x7fd178089000 con 0x7fd1880d3890 2024-04-06T13:49:13.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.084+0000 7fd18d002700 1 --2- 172.21.15.189:0/485605726 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd16c06aed0 0x7fd16c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:13.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.084+0000 7fd18d002700 1 --2- 172.21.15.189:0/485605726 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd16c06aed0 0x7fd16c06d390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd184009a10 tx=0x7fd184008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:13.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.088+0000 7fd17dffb700 1 -- 172.21.15.189:0/485605726 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd1880d1d80 con 0x7fd1880d3890 2024-04-06T13:49:13.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:13.298+0000 7fd18f266700 1 -- 172.21.15.189:0/485605726 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd1880cc160 con 0x7fd1880d3890 2024-04-06T13:49:14.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.078+0000 7fd17dffb700 1 -- 172.21.15.189:0/485605726 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v265) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd1880cc160 con 0x7fd1880d3890 2024-04-06T13:49:14.082 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.082+0000 7fd18f266700 1 -- 172.21.15.189:0/485605726 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd16c06aed0 msgr2=0x7fd16c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.082+0000 7fd18f266700 1 --2- 172.21.15.189:0/485605726 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd16c06aed0 0x7fd16c06d390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd184009a10 tx=0x7fd184008040 comp rx=0 tx=0).stop 2024-04-06T13:49:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.082+0000 7fd18f266700 1 -- 172.21.15.189:0/485605726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1880d3890 msgr2=0x7fd1880cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.082+0000 7fd18f266700 1 --2- 172.21.15.189:0/485605726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1880d3890 0x7fd1880cc5f0 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7fd17800c700 tx=0x7fd17800cac0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.082+0000 7fd18f266700 1 -- 172.21.15.189:0/485605726 shutdown_connections 2024-04-06T13:49:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.082+0000 7fd18f266700 1 --2- 172.21.15.189:0/485605726 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd16c06aed0 0x7fd16c06d390 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.082+0000 7fd18f266700 1 --2- 172.21.15.189:0/485605726 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd18815d590 0x7fd18815d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.082+0000 7fd18f266700 1 --2- 172.21.15.189:0/485605726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1880b4ed0 0x7fd18814d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.082+0000 7fd18f266700 1 --2- 172.21.15.189:0/485605726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1880d3890 0x7fd1880cc5f0 unknown :-1 s=CLOSED pgs=2901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.082+0000 7fd18f266700 1 -- 172.21.15.189:0/485605726 >> 172.21.15.189:0/485605726 conn(0x7fd18806bbc0 msgr2=0x7fd1880b31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.082+0000 7fd18f266700 1 -- 172.21.15.189:0/485605726 shutdown_connections 2024-04-06T13:49:14.083 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.082+0000 7fd18f266700 1 -- 172.21.15.189:0/485605726 wait complete. 2024-04-06T13:49:14.083 INFO:teuthology.orchestra.run.smithi189.stderr: removed all blocklist entries 2024-04-06T13:49:14.097 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-06T13:49:14.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.225+0000 7fe76b91a700 1 Processor -- start 2024-04-06T13:49:14.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.225+0000 7fe76b91a700 1 -- start start 2024-04-06T13:49:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe76b91a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe7640cdc60 0x7fe7640ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe76b91a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe76406e470 0x7fe76406e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe76b91a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76406c080 con 0x7fe764159b20 2024-04-06T13:49:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe76b91a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7640ce650 con 0x7fe7640cdc60 2024-04-06T13:49:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe76b91a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe764159f00 con 0x7fe76406e470 2024-04-06T13:49:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe7696b6700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe764159b20 0x7fe76406b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48870/0 (socket says 172.21.15.189:48870) 2024-04-06T13:49:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe7696b6700 1 -- 172.21.15.189:0/1017372076 learned_addr learned my addr 172.21.15.189:0/1017372076 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe75bfff700 1 -- 172.21.15.189:0/1017372076 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903420470 0 0) 0x7fe76406c080 con 0x7fe764159b20 2024-04-06T13:49:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe768eb5700 1 --2- 172.21.15.189:0/1017372076 >> v2:172.21.15.192:3300/0 conn(0x7fe7640cdc60 0x7fe7640ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:14.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe769eb7700 1 --2- 172.21.15.189:0/1017372076 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe76406e470 0x7fe76406e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:14.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe75bfff700 1 -- 172.21.15.189:0/1017372076 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7500035b0 con 0x7fe764159b20 2024-04-06T13:49:14.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe75bfff700 1 -- 172.21.15.189:0/1017372076 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2538118903 0 0) 0x7fe7500035b0 con 0x7fe764159b20 2024-04-06T13:49:14.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe75bfff700 1 -- 172.21.15.189:0/1017372076 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe76406e470 msgr2=0x7fe76406e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:14.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe75bfff700 1 --2- 172.21.15.189:0/1017372076 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe76406e470 0x7fe76406e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe75bfff700 1 -- 172.21.15.189:0/1017372076 >> v2:172.21.15.192:3300/0 conn(0x7fe7640cdc60 msgr2=0x7fe7640ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:14.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe75bfff700 1 --2- 172.21.15.189:0/1017372076 >> v2:172.21.15.192:3300/0 conn(0x7fe7640cdc60 0x7fe7640ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe75bfff700 1 -- 172.21.15.189:0/1017372076 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76406edc0 con 0x7fe764159b20 2024-04-06T13:49:14.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe75bfff700 1 -- 172.21.15.189:0/1017372076 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fe760001670 con 0x7fe764159b20 2024-04-06T13:49:14.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe75bfff700 1 -- 172.21.15.189:0/1017372076 >> v1:172.21.15.189:6789/0 conn(0x7fe764159b20 legacy=0x7fe76406b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:14.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe75bfff700 1 --2- 172.21.15.189:0/1017372076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe750003a30 0x7fe750003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:14.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe75bfff700 1 --2- 172.21.15.189:0/1017372076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe750004850 0x7fe750004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:14.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe75bfff700 1 --2- 172.21.15.189:0/1017372076 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe76406e470 0x7fe750005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:14.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe75bfff700 1 -- 172.21.15.189:0/1017372076 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7640ce650 con 0x7fe750003a30 2024-04-06T13:49:14.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe75bfff700 1 -- 172.21.15.189:0/1017372076 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe764159f00 con 0x7fe750004850 2024-04-06T13:49:14.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.226+0000 7fe75bfff700 1 -- 172.21.15.189:0/1017372076 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75000ba40 con 0x7fe76406e470 2024-04-06T13:49:14.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.227+0000 7fe7696b6700 1 --2- 172.21.15.189:0/1017372076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe750003a30 0x7fe750003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:14.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.227+0000 7fe768eb5700 1 --2- 172.21.15.189:0/1017372076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe750004850 0x7fe750004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:14.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.227+0000 7fe769eb7700 1 --2- 172.21.15.189:0/1017372076 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe76406e470 0x7fe750005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:14.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.227+0000 7fe7696b6700 1 -- 172.21.15.189:0/1017372076 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe76406e470 msgr2=0x7fe750005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:14.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.227+0000 7fe7696b6700 1 --2- 172.21.15.189:0/1017372076 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe76406e470 0x7fe750005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.227+0000 7fe7696b6700 1 -- 172.21.15.189:0/1017372076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe750004850 msgr2=0x7fe750004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:14.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.227+0000 7fe7696b6700 1 --2- 172.21.15.189:0/1017372076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe750004850 0x7fe750004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.227+0000 7fe7696b6700 1 -- 172.21.15.189:0/1017372076 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7500035b0 con 0x7fe750003a30 2024-04-06T13:49:14.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.227+0000 7fe769eb7700 1 --2- 172.21.15.189:0/1017372076 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe76406e470 0x7fe750005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:14.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.227+0000 7fe7696b6700 1 --2- 172.21.15.189:0/1017372076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe750003a30 0x7fe750003e50 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 crypto rx=0x7fe760007d30 tx=0x7fe750005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2275eb244d9cd18 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:14.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.227+0000 7fe75bfff700 1 -- 172.21.15.189:0/1017372076 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76000cd50 con 0x7fe750003a30 2024-04-06T13:49:14.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.227+0000 7fe75bfff700 1 -- 172.21.15.189:0/1017372076 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe760005960 con 0x7fe750003a30 2024-04-06T13:49:14.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.227+0000 7fe75bfff700 1 -- 172.21.15.189:0/1017372076 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe760005d10 con 0x7fe750003a30 2024-04-06T13:49:14.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.228+0000 7fe76b91a700 1 -- 172.21.15.189:0/1017372076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe750003a30 msgr2=0x7fe750003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:14.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.228+0000 7fe76b91a700 1 --2- 172.21.15.189:0/1017372076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe750003a30 0x7fe750003e50 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 crypto rx=0x7fe760007d30 tx=0x7fe750005880 comp rx=0 tx=0).stop 2024-04-06T13:49:14.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.228+0000 7fe769eb7700 1 -- 172.21.15.189:0/1017372076 reap_dead start 2024-04-06T13:49:14.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.228+0000 7fe76b91a700 1 -- 172.21.15.189:0/1017372076 shutdown_connections 2024-04-06T13:49:14.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.228+0000 7fe76b91a700 1 -- 172.21.15.189:0/1017372076 >> 172.21.15.189:0/1017372076 conn(0x7fe7640cc290 msgr2=0x7fe7640cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:14.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.228+0000 7fe76b91a700 1 -- 172.21.15.189:0/1017372076 shutdown_connections 2024-04-06T13:49:14.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.228+0000 7fe76b91a700 1 -- 172.21.15.189:0/1017372076 wait complete. 2024-04-06T13:49:14.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.229+0000 7fe76b91a700 1 Processor -- start 2024-04-06T13:49:14.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.229+0000 7fe76b91a700 1 -- start start 2024-04-06T13:49:14.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe76b91a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe750003a30 0x7fe76415a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:14.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe76b91a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7641580d0 0x7fe764148c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:14.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe76b91a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe764149460 0x7fe7641e33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:14.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe76b91a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7640b4590 con 0x7fe750003a30 2024-04-06T13:49:14.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe76b91a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe764149180 con 0x7fe7641580d0 2024-04-06T13:49:14.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe76b91a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7640b3910 con 0x7fe764149460 2024-04-06T13:49:14.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe7696b6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe750003a30 0x7fe76415a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:14.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe7696b6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe750003a30 0x7fe76415a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60864/0 (socket says 172.21.15.189:60864) 2024-04-06T13:49:14.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe7696b6700 1 -- 172.21.15.189:0/2075653098 learned_addr learned my addr 172.21.15.189:0/2075653098 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:14.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe769eb7700 1 --2- 172.21.15.189:0/2075653098 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe764149460 0x7fe7641e33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:14.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe768eb5700 1 --2- 172.21.15.189:0/2075653098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7641580d0 0x7fe764148c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:14.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe7696b6700 1 -- 172.21.15.189:0/2075653098 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe764149460 msgr2=0x7fe7641e33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:14.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe7696b6700 1 --2- 172.21.15.189:0/2075653098 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe764149460 0x7fe7641e33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe7696b6700 1 -- 172.21.15.189:0/2075653098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7641580d0 msgr2=0x7fe764148c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:14.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe7696b6700 1 --2- 172.21.15.189:0/2075653098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7641580d0 0x7fe764148c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe7696b6700 1 -- 172.21.15.189:0/2075653098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7640b3bf0 con 0x7fe750003a30 2024-04-06T13:49:14.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe769eb7700 1 --2- 172.21.15.189:0/2075653098 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe764149460 0x7fe7641e33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:14.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe768eb5700 1 --2- 172.21.15.189:0/2075653098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7641580d0 0x7fe764148c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:14.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe7696b6700 1 --2- 172.21.15.189:0/2075653098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe750003a30 0x7fe76415a040 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 crypto rx=0x7fe760005d00 tx=0x7fe760001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:14.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe759ffb700 1 -- 172.21.15.189:0/2075653098 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe760021070 con 0x7fe750003a30 2024-04-06T13:49:14.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe759ffb700 1 -- 172.21.15.189:0/2075653098 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe760000ed0 con 0x7fe750003a30 2024-04-06T13:49:14.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe759ffb700 1 -- 172.21.15.189:0/2075653098 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7600072e0 con 0x7fe750003a30 2024-04-06T13:49:14.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.230+0000 7fe76b91a700 1 -- 172.21.15.189:0/2075653098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7640b3ed0 con 0x7fe750003a30 2024-04-06T13:49:14.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.231+0000 7fe76b91a700 1 -- 172.21.15.189:0/2075653098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7640b35a0 con 0x7fe750003a30 2024-04-06T13:49:14.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.231+0000 7fe76b91a700 1 -- 172.21.15.189:0/2075653098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe764074990 con 0x7fe750003a30 2024-04-06T13:49:14.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.232+0000 7fe759ffb700 1 -- 172.21.15.189:0/2075653098 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fe760007440 con 0x7fe750003a30 2024-04-06T13:49:14.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.232+0000 7fe759ffb700 1 --2- 172.21.15.189:0/2075653098 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe74806aed0 0x7fe74806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:14.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.232+0000 7fe759ffb700 1 -- 172.21.15.189:0/2075653098 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fe76008bda0 con 0x7fe750003a30 2024-04-06T13:49:14.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.232+0000 7fe768eb5700 1 --2- 172.21.15.189:0/2075653098 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe74806aed0 0x7fe74806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:14.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.233+0000 7fe768eb5700 1 --2- 172.21.15.189:0/2075653098 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe74806aed0 0x7fe74806d390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe7641f1bd0 tx=0x7fe754008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:14.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.237+0000 7fe759ffb700 1 -- 172.21.15.189:0/2075653098 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe760056f90 con 0x7fe750003a30 2024-04-06T13:49:14.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.508+0000 7fe76b91a700 1 -- 172.21.15.189:0/2075653098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe7640b5790 con 0x7fe750003a30 2024-04-06T13:49:14.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.509+0000 7fe759ffb700 1 -- 172.21.15.189:0/2075653098 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v53) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fe76005a5b0 con 0x7fe750003a30 2024-04-06T13:49:14.509 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:14.509 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:49:14.509 INFO:teuthology.orchestra.run.smithi189.stdout: "auth_dump": [ 2024-04-06T13:49:14.509 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:14.509 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.a", 2024-04-06T13:49:14.509 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmTjYMNBAA5voWGV4Pa7pE27VogrdipQ==", 2024-04-06T13:49:14.509 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:14.509 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:49:14.510 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:49:14.510 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:49:14.510 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:14.510 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.510 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:14.510 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:14.510 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.b", 2024-04-06T13:49:14.510 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm9A1OBRAAnHzubCvRUX8UdW8hmDzKbQ==", 2024-04-06T13:49:14.510 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:14.510 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:49:14.510 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:49:14.510 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:49:14.511 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:14.511 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.511 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:14.511 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:14.511 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.c", 2024-04-06T13:49:14.511 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmz0hhOxAAD3/GINNmGoFk+oPyiHHJXg==", 2024-04-06T13:49:14.511 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:14.511 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:49:14.511 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:49:14.511 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:49:14.511 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:14.511 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.512 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:14.512 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:14.512 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.d", 2024-04-06T13:49:14.512 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmhoAeDRAAoQC0vicWumBRTwpi3PtYaw==", 2024-04-06T13:49:14.512 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:14.512 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:49:14.512 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:49:14.512 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:49:14.512 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:14.512 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.512 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:14.512 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:14.512 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.0", 2024-04-06T13:49:14.513 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDkOxFmkxLWJhAAFXewCNOFX7dhAZVgUmewpQ==", 2024-04-06T13:49:14.513 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:14.513 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:49:14.513 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:49:14.513 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:14.513 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.513 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:14.513 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:14.513 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.1", 2024-04-06T13:49:14.513 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDnOxFmHYqWJRAABlsVCiv3e7YJMbfodJTWag==", 2024-04-06T13:49:14.513 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:14.513 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:49:14.513 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:49:14.513 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:14.514 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.514 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:14.514 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:14.514 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.2", 2024-04-06T13:49:14.514 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDqOxFmkhcvJBAA8NMEpHVGbGALjV9RISKP8g==", 2024-04-06T13:49:14.514 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:14.514 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:49:14.514 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:49:14.514 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:14.514 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.514 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:14.515 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:14.515 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.3", 2024-04-06T13:49:14.515 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDtOxFmDGzhIxAAamBoAbE8+lAel4os67meyQ==", 2024-04-06T13:49:14.515 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:14.515 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:49:14.515 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:49:14.515 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:14.515 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.515 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:14.515 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:14.515 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.4", 2024-04-06T13:49:14.515 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDzOxFmobZDJxAA9l8wdS81Eyw6+17I5ac6bw==", 2024-04-06T13:49:14.515 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:14.515 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:49:14.516 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:49:14.516 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:14.516 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.516 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:14.516 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:14.516 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.5", 2024-04-06T13:49:14.516 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD2OxFmi+ruJRAASlQ0o9jX/wA7U5AosnL+Wg==", 2024-04-06T13:49:14.516 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:14.516 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:49:14.516 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:49:14.516 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:14.516 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.516 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:14.516 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:14.517 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.6", 2024-04-06T13:49:14.517 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD5OxFmbwuuJBAA0wqJRmnJyskQPZJ7/LnpvA==", 2024-04-06T13:49:14.517 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:14.517 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:49:14.517 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:49:14.517 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:14.517 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.517 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:14.517 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:14.517 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.7", 2024-04-06T13:49:14.518 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD8OxFmDmRgIxAAzyW9AGAbmb67ZvB5qunwLQ==", 2024-04-06T13:49:14.518 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:14.518 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:49:14.518 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:49:14.518 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:14.518 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.518 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:14.518 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:14.519 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.0", 2024-04-06T13:49:14.519 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm0ByJERAAoEV8Yfc09+Go6u95XafElQ==", 2024-04-06T13:49:14.519 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:14.519 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:49:14.519 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow r", 2024-04-06T13:49:14.519 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-06T13:49:14.519 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.519 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:14.519 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:14.519 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.1", 2024-04-06T13:49:14.519 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmSV5lGBAAavDEo2LGy0IHpWSAuitHXg==", 2024-04-06T13:49:14.519 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:14.519 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:49:14.519 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:49:14.520 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:49:14.520 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:49:14.520 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.520 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:14.520 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:14.520 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.2", 2024-04-06T13:49:14.520 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm/Gz5FBAA3xX/rsAyWZ1JoUVlQlBIvQ==", 2024-04-06T13:49:14.520 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:14.520 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:49:14.520 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:49:14.520 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:49:14.520 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:49:14.520 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.520 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:14.521 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:14.521 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.3", 2024-04-06T13:49:14.521 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmzra7GxAADbnzIvWnRTo65LvPwtxWrg==", 2024-04-06T13:49:14.521 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:14.521 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:49:14.521 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:49:14.521 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:49:14.521 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:49:14.521 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.521 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:14.521 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:14.521 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.admin", 2024-04-06T13:49:14.521 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmxG+wCBAANg5kt6TOUQdj3/X0Mqgx1g==", 2024-04-06T13:49:14.521 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:14.522 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:49:14.522 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:49:14.522 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:49:14.522 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:14.522 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.522 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:14.522 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:14.522 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.x", 2024-04-06T13:49:14.522 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmPLyBMBAAYfhkI9Z1CPlf1Tijm+L7xA==", 2024-04-06T13:49:14.522 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:14.522 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:49:14.522 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T13:49:14.522 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:14.522 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.523 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:14.523 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:14.523 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.y", 2024-04-06T13:49:14.523 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFm1nHJHhAAQWRe8cU8duWuUJiYaDeTaQ==", 2024-04-06T13:49:14.523 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:14.523 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:49:14.523 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T13:49:14.523 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:14.523 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.523 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:14.523 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-04-06T13:49:14.523 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:49:14.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.511+0000 7fe76b91a700 1 -- 172.21.15.189:0/2075653098 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe74806aed0 msgr2=0x7fe74806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:14.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.511+0000 7fe76b91a700 1 --2- 172.21.15.189:0/2075653098 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe74806aed0 0x7fe74806d390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe7641f1bd0 tx=0x7fe754008040 comp rx=0 tx=0).stop 2024-04-06T13:49:14.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.511+0000 7fe76b91a700 1 -- 172.21.15.189:0/2075653098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe750003a30 msgr2=0x7fe76415a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:14.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.511+0000 7fe76b91a700 1 --2- 172.21.15.189:0/2075653098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe750003a30 0x7fe76415a040 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 crypto rx=0x7fe760005d00 tx=0x7fe760001a00 comp rx=0 tx=0).stop 2024-04-06T13:49:14.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.511+0000 7fe76b91a700 1 -- 172.21.15.189:0/2075653098 shutdown_connections 2024-04-06T13:49:14.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.511+0000 7fe76b91a700 1 --2- 172.21.15.189:0/2075653098 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe74806aed0 0x7fe74806d390 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.511+0000 7fe76b91a700 1 --2- 172.21.15.189:0/2075653098 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe764149460 0x7fe7641e33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.511+0000 7fe76b91a700 1 --2- 172.21.15.189:0/2075653098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe750003a30 0x7fe76415a040 unknown :-1 s=CLOSED pgs=2904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.511+0000 7fe76b91a700 1 --2- 172.21.15.189:0/2075653098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7641580d0 0x7fe764148c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.511+0000 7fe76b91a700 1 -- 172.21.15.189:0/2075653098 >> 172.21.15.189:0/2075653098 conn(0x7fe7640cc290 msgr2=0x7fe7640af5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:14.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.511+0000 7fe76b91a700 1 -- 172.21.15.189:0/2075653098 shutdown_connections 2024-04-06T13:49:14.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.511+0000 7fe76b91a700 1 -- 172.21.15.189:0/2075653098 wait complete. 2024-04-06T13:49:14.527 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:49:14.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.647+0000 7f318ffa8700 1 Processor -- start 2024-04-06T13:49:14.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.648+0000 7f318ffa8700 1 -- start start 2024-04-06T13:49:14.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.648+0000 7f318ffa8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f318815d7b0 0x7f31880b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:14.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.648+0000 7f318ffa8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f318806a730 0x7f318806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:14.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.648+0000 7f318ffa8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f318815ddf0 0x7f318815e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:14.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.648+0000 7f318ffa8700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f318806b0f0 con 0x7f318815d7b0 2024-04-06T13:49:14.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.648+0000 7f318ffa8700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31880b3990 con 0x7f318806a730 2024-04-06T13:49:14.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.648+0000 7f318ffa8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31880b3060 con 0x7f318815ddf0 2024-04-06T13:49:14.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.649+0000 7f318dd44700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f318815d7b0 0x7f31880b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:14.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.649+0000 7f318dd44700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f318815d7b0 0x7f31880b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60872/0 (socket says 172.21.15.189:60872) 2024-04-06T13:49:14.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.649+0000 7f318dd44700 1 -- 172.21.15.189:0/2541720718 learned_addr learned my addr 172.21.15.189:0/2541720718 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:14.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.649+0000 7f318e545700 1 --2- 172.21.15.189:0/2541720718 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f318815ddf0 0x7f318815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:14.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.649+0000 7f318d543700 1 --2- 172.21.15.189:0/2541720718 >> v2:172.21.15.192:3300/0 conn(0x7f318806a730 0x7f318806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:14.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.649+0000 7f318dd44700 1 -- 172.21.15.189:0/2541720718 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f318815ddf0 msgr2=0x7f318815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:14.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.649+0000 7f318dd44700 1 --2- 172.21.15.189:0/2541720718 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f318815ddf0 0x7f318815e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.649+0000 7f318dd44700 1 -- 172.21.15.189:0/2541720718 >> v2:172.21.15.192:3300/0 conn(0x7f318806a730 msgr2=0x7f318806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:14.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.649+0000 7f318dd44700 1 --2- 172.21.15.189:0/2541720718 >> v2:172.21.15.192:3300/0 conn(0x7f318806a730 0x7f318806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.649+0000 7f318dd44700 1 -- 172.21.15.189:0/2541720718 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31880b3340 con 0x7f318815d7b0 2024-04-06T13:49:14.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.649+0000 7f318dd44700 1 --2- 172.21.15.189:0/2541720718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f318815d7b0 0x7f31880b3e20 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7f31880afc00 tx=0x7f3178009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf34709d1acbf759 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:14.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.649+0000 7f318cd42700 1 -- 172.21.15.189:0/2541720718 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317800b870 con 0x7f318815d7b0 2024-04-06T13:49:14.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.649+0000 7f318cd42700 1 -- 172.21.15.189:0/2541720718 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317800b9d0 con 0x7f318815d7b0 2024-04-06T13:49:14.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.649+0000 7f318cd42700 1 -- 172.21.15.189:0/2541720718 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317800bd60 con 0x7f318815d7b0 2024-04-06T13:49:14.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.650+0000 7f318ffa8700 1 -- 172.21.15.189:0/2541720718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f318815d7b0 msgr2=0x7f31880b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:14.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.650+0000 7f318ffa8700 1 --2- 172.21.15.189:0/2541720718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f318815d7b0 0x7f31880b3e20 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7f31880afc00 tx=0x7f3178009bd0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.650+0000 7f318ffa8700 1 -- 172.21.15.189:0/2541720718 shutdown_connections 2024-04-06T13:49:14.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.650+0000 7f318ffa8700 1 --2- 172.21.15.189:0/2541720718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f318815d7b0 0x7f31880b3e20 unknown :-1 s=CLOSED pgs=2905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.650+0000 7f318ffa8700 1 --2- 172.21.15.189:0/2541720718 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f318815ddf0 0x7f318815e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.650+0000 7f318ffa8700 1 --2- 172.21.15.189:0/2541720718 >> v2:172.21.15.192:3300/0 conn(0x7f318806a730 0x7f318806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.650+0000 7f318ffa8700 1 -- 172.21.15.189:0/2541720718 >> 172.21.15.189:0/2541720718 conn(0x7f31880d00b0 msgr2=0x7f31880d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:14.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.650+0000 7f318ffa8700 1 -- 172.21.15.189:0/2541720718 shutdown_connections 2024-04-06T13:49:14.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.650+0000 7f318ffa8700 1 -- 172.21.15.189:0/2541720718 wait complete. 2024-04-06T13:49:14.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.651+0000 7f318ffa8700 1 Processor -- start 2024-04-06T13:49:14.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.651+0000 7f318ffa8700 1 -- start start 2024-04-06T13:49:14.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f318ffa8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f318806a730 0x7f31881484e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:14.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f318ffa8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f318815d7b0 0x7f318815b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:14.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f318ffa8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f318815ddf0 0x7f318815fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:14.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f318ffa8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31881492f0 con 0x7f318815d7b0 2024-04-06T13:49:14.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f318ffa8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318815c420 con 0x7f318806a730 2024-04-06T13:49:14.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f318ffa8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f318815be60 con 0x7f318815ddf0 2024-04-06T13:49:14.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f318d543700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f318815d7b0 0x7f318815b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:14.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f318d543700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f318815d7b0 0x7f318815b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60878/0 (socket says 172.21.15.189:60878) 2024-04-06T13:49:14.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f318d543700 1 -- 172.21.15.189:0/1642704601 learned_addr learned my addr 172.21.15.189:0/1642704601 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:14.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f318dd44700 1 --2- 172.21.15.189:0/1642704601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f318806a730 0x7f31881484e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:14.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f318e545700 1 --2- 172.21.15.189:0/1642704601 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f318815ddf0 0x7f318815fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:14.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f318d543700 1 -- 172.21.15.189:0/1642704601 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f318815ddf0 msgr2=0x7f318815fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:14.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f318d543700 1 --2- 172.21.15.189:0/1642704601 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f318815ddf0 0x7f318815fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f318d543700 1 -- 172.21.15.189:0/1642704601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f318806a730 msgr2=0x7f31881484e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:14.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f318d543700 1 --2- 172.21.15.189:0/1642704601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f318806a730 0x7f31881484e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f318d543700 1 -- 172.21.15.189:0/1642704601 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3178009550 con 0x7f318815d7b0 2024-04-06T13:49:14.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f318d543700 1 --2- 172.21.15.189:0/1642704601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f318815d7b0 0x7f318815b3a0 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 crypto rx=0x7f318400c7f0 tx=0x7f318400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:14.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f317e7fc700 1 -- 172.21.15.189:0/1642704601 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3184004220 con 0x7f318815d7b0 2024-04-06T13:49:14.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f317e7fc700 1 -- 172.21.15.189:0/1642704601 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3184004380 con 0x7f318815d7b0 2024-04-06T13:49:14.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f318ffa8700 1 -- 172.21.15.189:0/1642704601 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f318815c140 con 0x7f318815d7b0 2024-04-06T13:49:14.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.652+0000 7f317e7fc700 1 -- 172.21.15.189:0/1642704601 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31840036a0 con 0x7f318815d7b0 2024-04-06T13:49:14.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.653+0000 7f318ffa8700 1 -- 172.21.15.189:0/1642704601 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f318814b080 con 0x7f318815d7b0 2024-04-06T13:49:14.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.653+0000 7f318ffa8700 1 -- 172.21.15.189:0/1642704601 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31881494a0 con 0x7f318815d7b0 2024-04-06T13:49:14.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.654+0000 7f317e7fc700 1 -- 172.21.15.189:0/1642704601 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f3184003800 con 0x7f318815d7b0 2024-04-06T13:49:14.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.654+0000 7f317e7fc700 1 --2- 172.21.15.189:0/1642704601 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f316c06aed0 0x7f316c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:14.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.654+0000 7f317e7fc700 1 -- 172.21.15.189:0/1642704601 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f31840891a0 con 0x7f318815d7b0 2024-04-06T13:49:14.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.654+0000 7f318dd44700 1 --2- 172.21.15.189:0/1642704601 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f316c06aed0 0x7f316c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:14.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.655+0000 7f318dd44700 1 --2- 172.21.15.189:0/1642704601 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f316c06aed0 0x7f316c06d390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3178000c00 tx=0x7f3178008030 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:14.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.659+0000 7f317e7fc700 1 -- 172.21.15.189:0/1642704601 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f31881494a0 con 0x7f318815d7b0 2024-04-06T13:49:14.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.854+0000 7f318ffa8700 1 -- 172.21.15.189:0/1642704601 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3188067bb0 con 0x7f318815d7b0 2024-04-06T13:49:14.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.855+0000 7f317e7fc700 1 -- 172.21.15.189:0/1642704601 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v265) v1 ==== 74+0+12768 (secure 0 0 0) 0x7f31840540c0 con 0x7f318815d7b0 2024-04-06T13:49:14.855 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:14.855 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":265,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:49:14.070451+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:49:14.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.857+0000 7f318ffa8700 1 -- 172.21.15.189:0/1642704601 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f316c06aed0 msgr2=0x7f316c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:14.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.857+0000 7f318ffa8700 1 --2- 172.21.15.189:0/1642704601 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f316c06aed0 0x7f316c06d390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3178000c00 tx=0x7f3178008030 comp rx=0 tx=0).stop 2024-04-06T13:49:14.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.857+0000 7f318ffa8700 1 -- 172.21.15.189:0/1642704601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f318815d7b0 msgr2=0x7f318815b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:14.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.857+0000 7f318ffa8700 1 --2- 172.21.15.189:0/1642704601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f318815d7b0 0x7f318815b3a0 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 crypto rx=0x7f318400c7f0 tx=0x7f318400cb00 comp rx=0 tx=0).stop 2024-04-06T13:49:14.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.857+0000 7f318ffa8700 1 -- 172.21.15.189:0/1642704601 shutdown_connections 2024-04-06T13:49:14.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.857+0000 7f318ffa8700 1 --2- 172.21.15.189:0/1642704601 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f316c06aed0 0x7f316c06d390 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.857+0000 7f318ffa8700 1 --2- 172.21.15.189:0/1642704601 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f318815ddf0 0x7f318815fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.857+0000 7f318ffa8700 1 --2- 172.21.15.189:0/1642704601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f318806a730 0x7f31881484e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.857+0000 7f318ffa8700 1 --2- 172.21.15.189:0/1642704601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f318815d7b0 0x7f318815b3a0 unknown :-1 s=CLOSED pgs=2906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.857+0000 7f318ffa8700 1 -- 172.21.15.189:0/1642704601 >> 172.21.15.189:0/1642704601 conn(0x7f31880d00b0 msgr2=0x7f31880723d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:14.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.857+0000 7f318ffa8700 1 -- 172.21.15.189:0/1642704601 shutdown_connections 2024-04-06T13:49:14.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.857+0000 7f318ffa8700 1 -- 172.21.15.189:0/1642704601 wait complete. 2024-04-06T13:49:14.870 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:49:14.870 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:49:14.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.995+0000 7f09c8709700 1 Processor -- start 2024-04-06T13:49:14.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.995+0000 7f09c8709700 1 -- start start 2024-04-06T13:49:14.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.996+0000 7f09c8709700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f09c00d11b0 0x7f09c00d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:14.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.996+0000 7f09c8709700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f09c00cfa00 0x7f09c00cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:14.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.996+0000 7f09c8709700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c00d1ad0 con 0x7f09c00cb870 2024-04-06T13:49:14.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.996+0000 7f09c8709700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c00ccdc0 con 0x7f09c00cfa00 2024-04-06T13:49:14.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.996+0000 7f09c8709700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c00d0b50 con 0x7f09c00d11b0 2024-04-06T13:49:14.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.996+0000 7f09c6ca6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f09c00cfa00 0x7f09c00cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:14.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.996+0000 7f09c5ca4700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f09c00cb870 0x7f09c015c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48878/0 (socket says 172.21.15.189:48878) 2024-04-06T13:49:14.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.996+0000 7f09c5ca4700 1 -- 172.21.15.189:0/403636979 learned_addr learned my addr 172.21.15.189:0/403636979 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:14.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.997+0000 7f09c6ca6700 1 -- 172.21.15.189:0/403636979 >> v2:172.21.15.192:3300/0 conn(0x7f09c00d11b0 msgr2=0x7f09c00d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:14.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.997+0000 7f09c64a5700 1 --2- 172.21.15.189:0/403636979 >> v2:172.21.15.192:3300/0 conn(0x7f09c00d11b0 0x7f09c00d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:14.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.997+0000 7f09c6ca6700 1 --2- 172.21.15.189:0/403636979 >> v2:172.21.15.192:3300/0 conn(0x7f09c00d11b0 0x7f09c00d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:14.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.997+0000 7f09c54a3700 1 -- 172.21.15.189:0/403636979 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569018645 0 0) 0x7f09c00d1ad0 con 0x7f09c00cb870 2024-04-06T13:49:14.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.997+0000 7f09c6ca6700 1 -- 172.21.15.189:0/403636979 >> v1:172.21.15.189:6789/0 conn(0x7f09c00cb870 legacy=0x7f09c015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:14.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.997+0000 7f09c6ca6700 1 -- 172.21.15.189:0/403636979 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c00cbd80 con 0x7f09c00cfa00 2024-04-06T13:49:14.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.997+0000 7f09c6ca6700 1 --2- 172.21.15.189:0/403636979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c00cfa00 0x7f09c00cd560 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7f09c00ced00 tx=0x7f09b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6453c4f65e028043 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:14.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.997+0000 7f09c54a3700 1 -- 172.21.15.189:0/403636979 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b800e070 con 0x7f09c00cfa00 2024-04-06T13:49:14.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.997+0000 7f09c54a3700 1 -- 172.21.15.189:0/403636979 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b800bbd0 con 0x7f09c00cfa00 2024-04-06T13:49:14.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.997+0000 7f09c54a3700 1 -- 172.21.15.189:0/403636979 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b8012430 con 0x7f09c00cfa00 2024-04-06T13:49:14.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.997+0000 7f09c8709700 1 -- 172.21.15.189:0/403636979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c00cfa00 msgr2=0x7f09c00cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:15.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.997+0000 7f09c8709700 1 --2- 172.21.15.189:0/403636979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c00cfa00 0x7f09c00cd560 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7f09c00ced00 tx=0x7f09b800b330 comp rx=0 tx=0).stop 2024-04-06T13:49:15.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.998+0000 7f09c8709700 1 -- 172.21.15.189:0/403636979 shutdown_connections 2024-04-06T13:49:15.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.998+0000 7f09c8709700 1 --2- 172.21.15.189:0/403636979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c00cfa00 0x7f09c00cd560 unknown :-1 s=CLOSED pgs=2908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.998+0000 7f09c8709700 1 --2- 172.21.15.189:0/403636979 >> v2:172.21.15.192:3300/0 conn(0x7f09c00d11b0 0x7f09c00d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.998+0000 7f09c8709700 1 -- 172.21.15.189:0/403636979 >> 172.21.15.189:0/403636979 conn(0x7f09c00b2db0 msgr2=0x7f09c00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:15.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.998+0000 7f09c8709700 1 -- 172.21.15.189:0/403636979 shutdown_connections 2024-04-06T13:49:15.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.998+0000 7f09c8709700 1 -- 172.21.15.189:0/403636979 wait complete. 2024-04-06T13:49:15.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.999+0000 7f09c8709700 1 Processor -- start 2024-04-06T13:49:15.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:14.999+0000 7f09c8709700 1 -- start start 2024-04-06T13:49:15.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.000+0000 7f09c8709700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09c00cb870 0x7f09c0158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:15.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.000+0000 7f09c8709700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c00cfa00 0x7f09c00c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:15.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.000+0000 7f09c8709700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09c00d11b0 0x7f09c00ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:15.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.000+0000 7f09c8709700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c01576b0 con 0x7f09c00cfa00 2024-04-06T13:49:15.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.000+0000 7f09c8709700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c0157990 con 0x7f09c00cb870 2024-04-06T13:49:15.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.000+0000 7f09c8709700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c0157c70 con 0x7f09c00d11b0 2024-04-06T13:49:15.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.000+0000 7f09c5ca4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c00cfa00 0x7f09c00c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:15.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.000+0000 7f09c5ca4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c00cfa00 0x7f09c00c5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60900/0 (socket says 172.21.15.189:60900) 2024-04-06T13:49:15.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.000+0000 7f09c5ca4700 1 -- 172.21.15.189:0/2267558514 learned_addr learned my addr 172.21.15.189:0/2267558514 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:15.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.000+0000 7f09c64a5700 1 --2- 172.21.15.189:0/2267558514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09c00cb870 0x7f09c0158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:15.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.000+0000 7f09c6ca6700 1 --2- 172.21.15.189:0/2267558514 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09c00d11b0 0x7f09c00ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:15.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.001+0000 7f09c5ca4700 1 -- 172.21.15.189:0/2267558514 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09c00d11b0 msgr2=0x7f09c00ca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:15.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.001+0000 7f09c5ca4700 1 --2- 172.21.15.189:0/2267558514 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09c00d11b0 0x7f09c00ca4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.001+0000 7f09c5ca4700 1 -- 172.21.15.189:0/2267558514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09c00cb870 msgr2=0x7f09c0158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:15.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.001+0000 7f09c5ca4700 1 --2- 172.21.15.189:0/2267558514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09c00cb870 0x7f09c0158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.001+0000 7f09c5ca4700 1 -- 172.21.15.189:0/2267558514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b8009a70 con 0x7f09c00cfa00 2024-04-06T13:49:15.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.001+0000 7f09c6ca6700 1 --2- 172.21.15.189:0/2267558514 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09c00d11b0 0x7f09c00ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:15.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.001+0000 7f09c64a5700 1 --2- 172.21.15.189:0/2267558514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09c00cb870 0x7f09c0158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:15.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.001+0000 7f09c5ca4700 1 --2- 172.21.15.189:0/2267558514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c00cfa00 0x7f09c00c5b00 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00c520 tx=0x7f09bc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:15.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.001+0000 7f09b6ffd700 1 -- 172.21.15.189:0/2267558514 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc004c80 con 0x7f09c00cfa00 2024-04-06T13:49:15.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.001+0000 7f09b6ffd700 1 -- 172.21.15.189:0/2267558514 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09bc004de0 con 0x7f09c00cfa00 2024-04-06T13:49:15.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.001+0000 7f09c8709700 1 -- 172.21.15.189:0/2267558514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c0157f50 con 0x7f09c00cfa00 2024-04-06T13:49:15.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.001+0000 7f09b6ffd700 1 -- 172.21.15.189:0/2267558514 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc0072d0 con 0x7f09c00cfa00 2024-04-06T13:49:15.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.001+0000 7f09c8709700 1 -- 172.21.15.189:0/2267558514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c0158c90 con 0x7f09c00cfa00 2024-04-06T13:49:15.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.002+0000 7f09c8709700 1 -- 172.21.15.189:0/2267558514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c0074990 con 0x7f09c00cfa00 2024-04-06T13:49:15.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.002+0000 7f09b6ffd700 1 -- 172.21.15.189:0/2267558514 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f09bc007430 con 0x7f09c00cfa00 2024-04-06T13:49:15.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.003+0000 7f09b6ffd700 1 --2- 172.21.15.189:0/2267558514 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f09a406aed0 0x7f09a406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:15.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.003+0000 7f09c64a5700 1 --2- 172.21.15.189:0/2267558514 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f09a406aed0 0x7f09a406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:15.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.003+0000 7f09b6ffd700 1 -- 172.21.15.189:0/2267558514 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f09bc0892f0 con 0x7f09c00cfa00 2024-04-06T13:49:15.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.003+0000 7f09c64a5700 1 --2- 172.21.15.189:0/2267558514 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f09a406aed0 0x7f09a406d390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f09b000aa50 tx=0x7f09b0008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:15.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.007+0000 7f09b6ffd700 1 -- 172.21.15.189:0/2267558514 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f09bc054330 con 0x7f09c00cfa00 2024-04-06T13:49:15.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.201+0000 7f09c8709700 1 -- 172.21.15.189:0/2267558514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f09c0158f90 con 0x7f09c00cfa00 2024-04-06T13:49:15.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.202+0000 7f09b6ffd700 1 -- 172.21.15.189:0/2267558514 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v265) v1 ==== 93+0+10 (secure 0 0 0) 0x7f09bc057950 con 0x7f09c00cfa00 2024-04-06T13:49:15.202 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:49:15.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.204+0000 7f09c8709700 1 -- 172.21.15.189:0/2267558514 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f09a406aed0 msgr2=0x7f09a406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:15.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.204+0000 7f09c8709700 1 --2- 172.21.15.189:0/2267558514 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f09a406aed0 0x7f09a406d390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f09b000aa50 tx=0x7f09b0008040 comp rx=0 tx=0).stop 2024-04-06T13:49:15.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.204+0000 7f09c8709700 1 -- 172.21.15.189:0/2267558514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c00cfa00 msgr2=0x7f09c00c5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:15.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.204+0000 7f09c8709700 1 --2- 172.21.15.189:0/2267558514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c00cfa00 0x7f09c00c5b00 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00c520 tx=0x7f09bc00c8e0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.204+0000 7f09c8709700 1 -- 172.21.15.189:0/2267558514 shutdown_connections 2024-04-06T13:49:15.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.204+0000 7f09c8709700 1 --2- 172.21.15.189:0/2267558514 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f09a406aed0 0x7f09a406d390 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.204+0000 7f09c8709700 1 --2- 172.21.15.189:0/2267558514 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09c00d11b0 0x7f09c00ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.204+0000 7f09c8709700 1 --2- 172.21.15.189:0/2267558514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09c00cb870 0x7f09c0158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.204+0000 7f09c8709700 1 --2- 172.21.15.189:0/2267558514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c00cfa00 0x7f09c00c5b00 unknown :-1 s=CLOSED pgs=2909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.204+0000 7f09c8709700 1 -- 172.21.15.189:0/2267558514 >> 172.21.15.189:0/2267558514 conn(0x7f09c00b2db0 msgr2=0x7f09c0159e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:15.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.204+0000 7f09c8709700 1 -- 172.21.15.189:0/2267558514 shutdown_connections 2024-04-06T13:49:15.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.204+0000 7f09c8709700 1 -- 172.21.15.189:0/2267558514 wait complete. 2024-04-06T13:49:15.217 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-06T13:49:15.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.339+0000 7fb8369dd700 1 Processor -- start 2024-04-06T13:49:15.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.339+0000 7fb8369dd700 1 -- start start 2024-04-06T13:49:15.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.339+0000 7fb8369dd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb830144f40 0x7fb83006a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:15.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.339+0000 7fb8369dd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb83015c150 0x7fb83006ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:15.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.339+0000 7fb8369dd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb83015c740 con 0x7fb83015fcd0 2024-04-06T13:49:15.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.339+0000 7fb8369dd700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8300cfcb0 con 0x7fb83015c150 2024-04-06T13:49:15.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.339+0000 7fb8369dd700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb83015ca20 con 0x7fb830144f40 2024-04-06T13:49:15.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.340+0000 7fb834f7a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb83015c150 0x7fb83006ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:15.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.340+0000 7fb82f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb83015fcd0 0x7fb83015ba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48892/0 (socket says 172.21.15.189:48892) 2024-04-06T13:49:15.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.340+0000 7fb82f7fe700 1 -- 172.21.15.189:0/2978717022 learned_addr learned my addr 172.21.15.189:0/2978717022 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:15.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.340+0000 7fb82ffff700 1 --2- 172.21.15.189:0/2978717022 >> v2:172.21.15.192:3300/0 conn(0x7fb830144f40 0x7fb83006a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:15.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.340+0000 7fb834f7a700 1 -- 172.21.15.189:0/2978717022 >> v2:172.21.15.192:3300/0 conn(0x7fb830144f40 msgr2=0x7fb83006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:15.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.340+0000 7fb834f7a700 1 --2- 172.21.15.189:0/2978717022 >> v2:172.21.15.192:3300/0 conn(0x7fb830144f40 0x7fb83006a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.340+0000 7fb834f7a700 1 -- 172.21.15.189:0/2978717022 >> v1:172.21.15.189:6789/0 conn(0x7fb83015fcd0 legacy=0x7fb83015ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:15.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.340+0000 7fb834f7a700 1 -- 172.21.15.189:0/2978717022 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83015f1f0 con 0x7fb83015c150 2024-04-06T13:49:15.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.340+0000 7fb82ffff700 1 --2- 172.21.15.189:0/2978717022 >> v2:172.21.15.192:3300/0 conn(0x7fb830144f40 0x7fb83006a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:15.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.340+0000 7fb82effd700 1 -- 172.21.15.189:0/2978717022 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330649549 0 0) 0x7fb83015c740 con 0x7fb83015fcd0 2024-04-06T13:49:15.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.340+0000 7fb834f7a700 1 --2- 172.21.15.189:0/2978717022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb83015c150 0x7fb83006ac70 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 crypto rx=0x7fb830073470 tx=0x7fb82800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=991944cfaf86bcb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:15.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.340+0000 7fb82effd700 1 -- 172.21.15.189:0/2978717022 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82800e070 con 0x7fb83015c150 2024-04-06T13:49:15.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.340+0000 7fb82effd700 1 -- 172.21.15.189:0/2978717022 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82800bbd0 con 0x7fb83015c150 2024-04-06T13:49:15.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.340+0000 7fb82effd700 1 -- 172.21.15.189:0/2978717022 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb828012430 con 0x7fb83015c150 2024-04-06T13:49:15.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.341+0000 7fb8369dd700 1 -- 172.21.15.189:0/2978717022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb83015c150 msgr2=0x7fb83006ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:15.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.341+0000 7fb8369dd700 1 --2- 172.21.15.189:0/2978717022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb83015c150 0x7fb83006ac70 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 crypto rx=0x7fb830073470 tx=0x7fb82800b330 comp rx=0 tx=0).stop 2024-04-06T13:49:15.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.341+0000 7fb8369dd700 1 -- 172.21.15.189:0/2978717022 shutdown_connections 2024-04-06T13:49:15.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.341+0000 7fb8369dd700 1 --2- 172.21.15.189:0/2978717022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb83015c150 0x7fb83006ac70 unknown :-1 s=CLOSED pgs=2911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.341+0000 7fb8369dd700 1 --2- 172.21.15.189:0/2978717022 >> v2:172.21.15.192:3300/0 conn(0x7fb830144f40 0x7fb83006a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.341+0000 7fb8369dd700 1 -- 172.21.15.189:0/2978717022 >> 172.21.15.189:0/2978717022 conn(0x7fb8300d00b0 msgr2=0x7fb8300d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:15.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.341+0000 7fb8369dd700 1 -- 172.21.15.189:0/2978717022 shutdown_connections 2024-04-06T13:49:15.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.341+0000 7fb8369dd700 1 -- 172.21.15.189:0/2978717022 wait complete. 2024-04-06T13:49:15.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.342+0000 7fb8369dd700 1 Processor -- start 2024-04-06T13:49:15.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.343+0000 7fb8369dd700 1 -- start start 2024-04-06T13:49:15.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.343+0000 7fb8369dd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb830144f40 0x7fb8300ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:15.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.343+0000 7fb8369dd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb83015c150 0x7fb83012f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:15.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.343+0000 7fb8369dd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb83015fcd0 0x7fb830130cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:15.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.343+0000 7fb8369dd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83012fc30 con 0x7fb830144f40 2024-04-06T13:49:15.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.343+0000 7fb8369dd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83012ff10 con 0x7fb83015c150 2024-04-06T13:49:15.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.343+0000 7fb8369dd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8301301f0 con 0x7fb83015fcd0 2024-04-06T13:49:15.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.343+0000 7fb82ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb830144f40 0x7fb8300ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:15.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.343+0000 7fb82ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb830144f40 0x7fb8300ca560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60910/0 (socket says 172.21.15.189:60910) 2024-04-06T13:49:15.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.343+0000 7fb82ffff700 1 -- 172.21.15.189:0/2973840437 learned_addr learned my addr 172.21.15.189:0/2973840437 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:15.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.344+0000 7fb82ffff700 1 -- 172.21.15.189:0/2973840437 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb83015fcd0 msgr2=0x7fb830130cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:15.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.344+0000 7fb82ffff700 1 --2- 172.21.15.189:0/2973840437 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb83015fcd0 0x7fb830130cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.344+0000 7fb82ffff700 1 -- 172.21.15.189:0/2973840437 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb83015c150 msgr2=0x7fb83012f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:15.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.344+0000 7fb82ffff700 1 --2- 172.21.15.189:0/2973840437 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb83015c150 0x7fb83012f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.344+0000 7fb82ffff700 1 -- 172.21.15.189:0/2973840437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb828009a70 con 0x7fb830144f40 2024-04-06T13:49:15.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.344+0000 7fb82ffff700 1 --2- 172.21.15.189:0/2973840437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb830144f40 0x7fb8300ca560 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 crypto rx=0x7fb82400e810 tx=0x7fb82400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:15.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.344+0000 7fb82cff9700 1 -- 172.21.15.189:0/2973840437 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824004230 con 0x7fb830144f40 2024-04-06T13:49:15.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.344+0000 7fb82cff9700 1 -- 172.21.15.189:0/2973840437 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb824004390 con 0x7fb830144f40 2024-04-06T13:49:15.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.344+0000 7fb82cff9700 1 -- 172.21.15.189:0/2973840437 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8240036a0 con 0x7fb830144f40 2024-04-06T13:49:15.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.344+0000 7fb8369dd700 1 -- 172.21.15.189:0/2973840437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8301304d0 con 0x7fb830144f40 2024-04-06T13:49:15.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.344+0000 7fb8369dd700 1 -- 172.21.15.189:0/2973840437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8301353d0 con 0x7fb830144f40 2024-04-06T13:49:15.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.345+0000 7fb8369dd700 1 -- 172.21.15.189:0/2973840437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb830067bb0 con 0x7fb830144f40 2024-04-06T13:49:15.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.345+0000 7fb82cff9700 1 -- 172.21.15.189:0/2973840437 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fb824003800 con 0x7fb830144f40 2024-04-06T13:49:15.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.346+0000 7fb82cff9700 1 --2- 172.21.15.189:0/2973840437 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb81c06aed0 0x7fb81c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:15.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.346+0000 7fb82cff9700 1 -- 172.21.15.189:0/2973840437 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb8240892c0 con 0x7fb830144f40 2024-04-06T13:49:15.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.346+0000 7fb82f7fe700 1 --2- 172.21.15.189:0/2973840437 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb81c06aed0 0x7fb81c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:15.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.346+0000 7fb82f7fe700 1 --2- 172.21.15.189:0/2973840437 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb81c06aed0 0x7fb81c06d390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb83006ff90 tx=0x7fb8200079f0 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:15.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.350+0000 7fb82cff9700 1 -- 172.21.15.189:0/2973840437 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb8240540c0 con 0x7fb830144f40 2024-04-06T13:49:15.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.545+0000 7fb8369dd700 1 -- 172.21.15.189:0/2973840437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb830074990 con 0x7fb830144f40 2024-04-06T13:49:15.546 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.546+0000 7fb82cff9700 1 -- 172.21.15.189:0/2973840437 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v265) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb824020060 con 0x7fb830144f40 2024-04-06T13:49:15.546 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:15.546 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T13:49:15.546 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:15.546 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 1, 2024-04-06T13:49:15.546 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": ".mgr" 2024-04-06T13:49:15.546 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:15.547 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T13:49:15.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.548+0000 7fb8369dd700 1 -- 172.21.15.189:0/2973840437 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb81c06aed0 msgr2=0x7fb81c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:15.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.548+0000 7fb8369dd700 1 --2- 172.21.15.189:0/2973840437 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb81c06aed0 0x7fb81c06d390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb83006ff90 tx=0x7fb8200079f0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.548+0000 7fb8369dd700 1 -- 172.21.15.189:0/2973840437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb830144f40 msgr2=0x7fb8300ca560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:15.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.548+0000 7fb8369dd700 1 --2- 172.21.15.189:0/2973840437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb830144f40 0x7fb8300ca560 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 crypto rx=0x7fb82400e810 tx=0x7fb82400ebd0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.548+0000 7fb8369dd700 1 -- 172.21.15.189:0/2973840437 shutdown_connections 2024-04-06T13:49:15.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.548+0000 7fb8369dd700 1 --2- 172.21.15.189:0/2973840437 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb81c06aed0 0x7fb81c06d390 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.548+0000 7fb8369dd700 1 --2- 172.21.15.189:0/2973840437 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb83015fcd0 0x7fb830130cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.548+0000 7fb8369dd700 1 --2- 172.21.15.189:0/2973840437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb830144f40 0x7fb8300ca560 unknown :-1 s=CLOSED pgs=2912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.548+0000 7fb8369dd700 1 --2- 172.21.15.189:0/2973840437 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb83015c150 0x7fb83012f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.548+0000 7fb8369dd700 1 -- 172.21.15.189:0/2973840437 >> 172.21.15.189:0/2973840437 conn(0x7fb8300d00b0 msgr2=0x7fb8300cf3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:15.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.548+0000 7fb8369dd700 1 -- 172.21.15.189:0/2973840437 shutdown_connections 2024-04-06T13:49:15.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.548+0000 7fb8369dd700 1 -- 172.21.15.189:0/2973840437 wait complete. 2024-04-06T13:49:15.560 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-06T13:49:15.560 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-06T13:49:15.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.686+0000 7f0e075b5700 1 Processor -- start 2024-04-06T13:49:15.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.686+0000 7f0e075b5700 1 -- start start 2024-04-06T13:49:15.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.687+0000 7f0e075b5700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0e0015d7b0 0x7f0e000b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:15.688 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.687+0000 7f0e075b5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e0015e500 0x7f0e0006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:15.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.687+0000 7f0e075b5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e000b3990 con 0x7f0e0006a730 2024-04-06T13:49:15.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.687+0000 7f0e075b5700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e0006b0f0 con 0x7f0e0015d7b0 2024-04-06T13:49:15.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.687+0000 7f0e075b5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e000b3060 con 0x7f0e0015e500 2024-04-06T13:49:15.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.687+0000 7f0e05351700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0e0015d7b0 0x7f0e000b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:15.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.687+0000 7f0e05b52700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e0015e500 0x7f0e0006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:15.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.687+0000 7f0e05351700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0e0015d7b0 0x7f0e000b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:45408/0 (socket says 172.21.15.189:45408) 2024-04-06T13:49:15.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.687+0000 7f0e05351700 1 -- 172.21.15.189:0/2987533787 learned_addr learned my addr 172.21.15.189:0/2987533787 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:15.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.688+0000 7f0e05b52700 1 -- 172.21.15.189:0/2987533787 >> v2:172.21.15.192:3300/0 conn(0x7f0e0015d7b0 msgr2=0x7f0e000b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:15.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.688+0000 7f0e05b52700 1 --2- 172.21.15.189:0/2987533787 >> v2:172.21.15.192:3300/0 conn(0x7f0e0015d7b0 0x7f0e000b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.688+0000 7f0e05b52700 1 -- 172.21.15.189:0/2987533787 >> v1:172.21.15.192:6789/0 conn(0x7f0e0006a730 legacy=0x7f0e0015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:15.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.688+0000 7f0e05b52700 1 -- 172.21.15.189:0/2987533787 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e000b3340 con 0x7f0e0015e500 2024-04-06T13:49:15.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.688+0000 7f0e05351700 1 --2- 172.21.15.189:0/2987533787 >> v2:172.21.15.192:3300/0 conn(0x7f0e0015d7b0 0x7f0e000b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:15.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.688+0000 7f0e05b52700 1 --2- 172.21.15.189:0/2987533787 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e0015e500 0x7f0e0006abb0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009d90 tx=0x7f0df800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c0beec3f6a31824 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:15.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.688+0000 7f0df7fff700 1 -- 172.21.15.189:0/2987533787 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df800e070 con 0x7f0e0015e500 2024-04-06T13:49:15.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.688+0000 7f0df7fff700 1 -- 172.21.15.189:0/2987533787 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df800bb20 con 0x7f0e0015e500 2024-04-06T13:49:15.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.688+0000 7f0df7fff700 1 -- 172.21.15.189:0/2987533787 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df8012430 con 0x7f0e0015e500 2024-04-06T13:49:15.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.689+0000 7f0e075b5700 1 -- 172.21.15.189:0/2987533787 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e0015e500 msgr2=0x7f0e0006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:15.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.689+0000 7f0e075b5700 1 --2- 172.21.15.189:0/2987533787 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e0015e500 0x7f0e0006abb0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009d90 tx=0x7f0df800b330 comp rx=0 tx=0).stop 2024-04-06T13:49:15.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.689+0000 7f0e075b5700 1 -- 172.21.15.189:0/2987533787 shutdown_connections 2024-04-06T13:49:15.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.689+0000 7f0e075b5700 1 --2- 172.21.15.189:0/2987533787 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e0015e500 0x7f0e0006abb0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.689+0000 7f0e075b5700 1 --2- 172.21.15.189:0/2987533787 >> v2:172.21.15.192:3300/0 conn(0x7f0e0015d7b0 0x7f0e000b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.689+0000 7f0e075b5700 1 -- 172.21.15.189:0/2987533787 >> 172.21.15.189:0/2987533787 conn(0x7f0e000d00b0 msgr2=0x7f0e000d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:15.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.689+0000 7f0e075b5700 1 -- 172.21.15.189:0/2987533787 shutdown_connections 2024-04-06T13:49:15.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.689+0000 7f0e075b5700 1 -- 172.21.15.189:0/2987533787 wait complete. 2024-04-06T13:49:15.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.690+0000 7f0e075b5700 1 Processor -- start 2024-04-06T13:49:15.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.690+0000 7f0e075b5700 1 -- start start 2024-04-06T13:49:15.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.691+0000 7f0e075b5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e0006a730 0x7f0e0014fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:15.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.691+0000 7f0e075b5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e0015d7b0 0x7f0e00151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:15.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.691+0000 7f0e075b5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e0015e500 0x7f0e0015fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:15.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.691+0000 7f0e075b5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0015c140 con 0x7f0e0006a730 2024-04-06T13:49:15.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.691+0000 7f0e075b5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0015b4c0 con 0x7f0e0015e500 2024-04-06T13:49:15.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.691+0000 7f0e075b5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0015b7a0 con 0x7f0e0015d7b0 2024-04-06T13:49:15.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.691+0000 7f0e05351700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e0006a730 0x7f0e0014fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:15.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.691+0000 7f0e05351700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e0006a730 0x7f0e0014fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60916/0 (socket says 172.21.15.189:60916) 2024-04-06T13:49:15.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.691+0000 7f0e05351700 1 -- 172.21.15.189:0/2921872572 learned_addr learned my addr 172.21.15.189:0/2921872572 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:15.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.691+0000 7f0e05b52700 1 --2- 172.21.15.189:0/2921872572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e0015e500 0x7f0e0015fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:15.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.691+0000 7f0e04b50700 1 --2- 172.21.15.189:0/2921872572 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e0015d7b0 0x7f0e00151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:15.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.692+0000 7f0e05351700 1 -- 172.21.15.189:0/2921872572 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e0015d7b0 msgr2=0x7f0e00151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:15.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.692+0000 7f0e05351700 1 --2- 172.21.15.189:0/2921872572 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e0015d7b0 0x7f0e00151df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.692+0000 7f0e05351700 1 -- 172.21.15.189:0/2921872572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e0015e500 msgr2=0x7f0e0015fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:15.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.692+0000 7f0e05351700 1 --2- 172.21.15.189:0/2921872572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e0015e500 0x7f0e0015fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:15.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.692+0000 7f0e05351700 1 -- 172.21.15.189:0/2921872572 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dfc009550 con 0x7f0e0006a730 2024-04-06T13:49:15.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.692+0000 7f0e05351700 1 --2- 172.21.15.189:0/2921872572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e0006a730 0x7f0e0014fe60 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00ebe0 tx=0x7f0dfc00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:15.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.692+0000 7f0df5ffb700 1 -- 172.21.15.189:0/2921872572 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc00c850 con 0x7f0e0006a730 2024-04-06T13:49:15.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.692+0000 7f0df5ffb700 1 -- 172.21.15.189:0/2921872572 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dfc00c9b0 con 0x7f0e0006a730 2024-04-06T13:49:15.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.692+0000 7f0e075b5700 1 -- 172.21.15.189:0/2921872572 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df8009a70 con 0x7f0e0006a730 2024-04-06T13:49:15.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.692+0000 7f0df5ffb700 1 -- 172.21.15.189:0/2921872572 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc00cd40 con 0x7f0e0006a730 2024-04-06T13:49:15.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.692+0000 7f0e075b5700 1 -- 172.21.15.189:0/2921872572 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0015bb30 con 0x7f0e0006a730 2024-04-06T13:49:15.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.693+0000 7f0e075b5700 1 -- 172.21.15.189:0/2921872572 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e00067bb0 con 0x7f0e0006a730 2024-04-06T13:49:15.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.694+0000 7f0df5ffb700 1 -- 172.21.15.189:0/2921872572 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f0dfc013470 con 0x7f0e0006a730 2024-04-06T13:49:15.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.694+0000 7f0df5ffb700 1 --2- 172.21.15.189:0/2921872572 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0de406aed0 0x7f0de406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:15.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.694+0000 7f0df5ffb700 1 -- 172.21.15.189:0/2921872572 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f0dfc089590 con 0x7f0e0006a730 2024-04-06T13:49:15.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.694+0000 7f0e04b50700 1 --2- 172.21.15.189:0/2921872572 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0de406aed0 0x7f0de406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:15.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.694+0000 7f0e04b50700 1 --2- 172.21.15.189:0/2921872572 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0de406aed0 0x7f0de406d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0e0006b540 tx=0x7f0df0009040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:15.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.698+0000 7f0df5ffb700 1 -- 172.21.15.189:0/2921872572 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0dfc054730 con 0x7f0e0006a730 2024-04-06T13:49:15.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:15.897+0000 7f0e075b5700 1 -- 172.21.15.189:0/2921872572 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0e00074990 con 0x7f0e0006a730 2024-04-06T13:49:16.081 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.081+0000 7f0df5ffb700 1 -- 172.21.15.189:0/2921872572 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v266) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0dfc057d50 con 0x7f0e0006a730 2024-04-06T13:49:16.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.083+0000 7f0e075b5700 1 -- 172.21.15.189:0/2921872572 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0de406aed0 msgr2=0x7f0de406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:16.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.083+0000 7f0e075b5700 1 --2- 172.21.15.189:0/2921872572 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0de406aed0 0x7f0de406d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0e0006b540 tx=0x7f0df0009040 comp rx=0 tx=0).stop 2024-04-06T13:49:16.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.084+0000 7f0e075b5700 1 -- 172.21.15.189:0/2921872572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e0006a730 msgr2=0x7f0e0014fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:16.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.084+0000 7f0e075b5700 1 --2- 172.21.15.189:0/2921872572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e0006a730 0x7f0e0014fe60 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00ebe0 tx=0x7f0dfc00eef0 comp rx=0 tx=0).stop 2024-04-06T13:49:16.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.084+0000 7f0e075b5700 1 -- 172.21.15.189:0/2921872572 shutdown_connections 2024-04-06T13:49:16.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.084+0000 7f0e075b5700 1 --2- 172.21.15.189:0/2921872572 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0de406aed0 0x7f0de406d390 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:16.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.084+0000 7f0e075b5700 1 --2- 172.21.15.189:0/2921872572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e0015e500 0x7f0e0015fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:16.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.084+0000 7f0e075b5700 1 --2- 172.21.15.189:0/2921872572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e0006a730 0x7f0e0014fe60 unknown :-1 s=CLOSED pgs=2913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:16.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.084+0000 7f0e075b5700 1 --2- 172.21.15.189:0/2921872572 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e0015d7b0 0x7f0e00151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:16.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.084+0000 7f0e075b5700 1 -- 172.21.15.189:0/2921872572 >> 172.21.15.189:0/2921872572 conn(0x7f0e000d00b0 msgr2=0x7f0e001507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:16.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.084+0000 7f0e075b5700 1 -- 172.21.15.189:0/2921872572 shutdown_connections 2024-04-06T13:49:16.085 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.085+0000 7f0e075b5700 1 -- 172.21.15.189:0/2921872572 wait complete. 2024-04-06T13:49:16.087 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' created 2024-04-06T13:49:16.100 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-06T13:49:16.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.231+0000 7f5753ced700 1 Processor -- start 2024-04-06T13:49:16.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.231+0000 7f5753ced700 1 -- start start 2024-04-06T13:49:16.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.232+0000 7f5753ced700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f574c15d7b0 0x7f574c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:16.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.232+0000 7f5753ced700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f574c0b3990 con 0x7f574c06a730 2024-04-06T13:49:16.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.232+0000 7f5753ced700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f574c0b3060 con 0x7f574c15e500 2024-04-06T13:49:16.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.232+0000 7f5753ced700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574c15e950 con 0x7f574c15d7b0 2024-04-06T13:49:16.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.232+0000 7f5751288700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f574c06a730 0x7f574c15ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48896/0 (socket says 172.21.15.189:48896) 2024-04-06T13:49:16.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.232+0000 7f5751288700 1 -- 172.21.15.189:0/3071124959 learned_addr learned my addr 172.21.15.189:0/3071124959 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:16.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5751a89700 1 --2- 172.21.15.189:0/3071124959 >> v2:172.21.15.192:3300/0 conn(0x7f574c15d7b0 0x7f574c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:16.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5750a87700 1 -- 172.21.15.189:0/3071124959 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 736734721 0 0) 0x7f574c0b3990 con 0x7f574c06a730 2024-04-06T13:49:16.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5750a87700 1 -- 172.21.15.189:0/3071124959 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57380035b0 con 0x7f574c06a730 2024-04-06T13:49:16.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5750a87700 1 -- 172.21.15.189:0/3071124959 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1018097123 0 0) 0x7f57380035b0 con 0x7f574c06a730 2024-04-06T13:49:16.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5750a87700 1 -- 172.21.15.189:0/3071124959 >> v2:172.21.15.192:3300/0 conn(0x7f574c15d7b0 msgr2=0x7f574c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:16.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5750a87700 1 --2- 172.21.15.189:0/3071124959 >> v2:172.21.15.192:3300/0 conn(0x7f574c15d7b0 0x7f574c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:16.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5750a87700 1 -- 172.21.15.189:0/3071124959 >> v1:172.21.15.192:6789/0 conn(0x7f574c15e500 legacy=0x7f574c0717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:16.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5750a87700 1 -- 172.21.15.189:0/3071124959 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574c0b3340 con 0x7f574c06a730 2024-04-06T13:49:16.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5750a87700 1 -- 172.21.15.189:0/3071124959 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f5748001670 con 0x7f574c06a730 2024-04-06T13:49:16.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5750a87700 1 -- 172.21.15.189:0/3071124959 >> v1:172.21.15.189:6789/0 conn(0x7f574c06a730 legacy=0x7f574c15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:16.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5750a87700 1 --2- 172.21.15.189:0/3071124959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5738003a00 0x7f5738003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:16.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5750a87700 1 --2- 172.21.15.189:0/3071124959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5738004850 0x7f5738004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:16.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5750a87700 1 --2- 172.21.15.189:0/3071124959 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5738005210 0x7f573800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:16.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5750a87700 1 -- 172.21.15.189:0/3071124959 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573800c180 con 0x7f5738003a00 2024-04-06T13:49:16.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5750a87700 1 -- 172.21.15.189:0/3071124959 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573800c2f0 con 0x7f5738004850 2024-04-06T13:49:16.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5750a87700 1 -- 172.21.15.189:0/3071124959 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573800c4b0 con 0x7f5738005210 2024-04-06T13:49:16.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5751a89700 1 --2- 172.21.15.189:0/3071124959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5738003a00 0x7f5738003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:16.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5751288700 1 --2- 172.21.15.189:0/3071124959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5738004850 0x7f5738004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:16.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f575228a700 1 --2- 172.21.15.189:0/3071124959 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5738005210 0x7f573800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:16.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5751a89700 1 -- 172.21.15.189:0/3071124959 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5738005210 msgr2=0x7f573800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:16.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5751a89700 1 --2- 172.21.15.189:0/3071124959 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5738005210 0x7f573800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:16.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5751a89700 1 -- 172.21.15.189:0/3071124959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5738004850 msgr2=0x7f5738004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:16.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.233+0000 7f5751a89700 1 --2- 172.21.15.189:0/3071124959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5738004850 0x7f5738004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:16.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.234+0000 7f5751a89700 1 -- 172.21.15.189:0/3071124959 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57380035b0 con 0x7f5738003a00 2024-04-06T13:49:16.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.234+0000 7f575228a700 1 -- 172.21.15.189:0/3071124959 reap_dead start 2024-04-06T13:49:16.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.234+0000 7f5751a89700 1 --2- 172.21.15.189:0/3071124959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5738003a00 0x7f5738003e20 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7f573c009b60 tx=0x7f5738005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9617db6f985c73e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:16.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.234+0000 7f5750a87700 1 -- 172.21.15.189:0/3071124959 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c00e6c0 con 0x7f5738003a00 2024-04-06T13:49:16.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.234+0000 7f5750a87700 1 -- 172.21.15.189:0/3071124959 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573c00e820 con 0x7f5738003a00 2024-04-06T13:49:16.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.234+0000 7f5750a87700 1 -- 172.21.15.189:0/3071124959 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c00eb90 con 0x7f5738003a00 2024-04-06T13:49:16.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.234+0000 7f5753ced700 1 -- 172.21.15.189:0/3071124959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5738003a00 msgr2=0x7f5738003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:16.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.234+0000 7f5753ced700 1 --2- 172.21.15.189:0/3071124959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5738003a00 0x7f5738003e20 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7f573c009b60 tx=0x7f5738005880 comp rx=0 tx=0).stop 2024-04-06T13:49:16.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.234+0000 7f5753ced700 1 -- 172.21.15.189:0/3071124959 shutdown_connections 2024-04-06T13:49:16.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.234+0000 7f5753ced700 1 --2- 172.21.15.189:0/3071124959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5738003a00 0x7f5738003e20 unknown :-1 s=CLOSED pgs=2915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:16.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.234+0000 7f5753ced700 1 -- 172.21.15.189:0/3071124959 >> 172.21.15.189:0/3071124959 conn(0x7f574c0d00b0 msgr2=0x7f574c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:16.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.234+0000 7f5753ced700 1 -- 172.21.15.189:0/3071124959 shutdown_connections 2024-04-06T13:49:16.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.234+0000 7f5753ced700 1 -- 172.21.15.189:0/3071124959 wait complete. 2024-04-06T13:49:16.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.236+0000 7f5753ced700 1 Processor -- start 2024-04-06T13:49:16.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.236+0000 7f5753ced700 1 -- start start 2024-04-06T13:49:16.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.236+0000 7f5753ced700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5738003a00 0x7f574c146b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:16.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.236+0000 7f5753ced700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f574c0cdb80 0x7f574c0aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:16.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.236+0000 7f5753ced700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574c0af400 0x7f574c158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:16.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.236+0000 7f5753ced700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c0b0730 con 0x7f574c0af400 2024-04-06T13:49:16.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.236+0000 7f5753ced700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c0cec20 con 0x7f5738003a00 2024-04-06T13:49:16.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.236+0000 7f5753ced700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f574c147080 con 0x7f574c0cdb80 2024-04-06T13:49:16.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.236+0000 7f575228a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574c0af400 0x7f574c158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:16.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.236+0000 7f575228a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574c0af400 0x7f574c158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60930/0 (socket says 172.21.15.189:60930) 2024-04-06T13:49:16.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.236+0000 7f575228a700 1 -- 172.21.15.189:0/3940084173 learned_addr learned my addr 172.21.15.189:0/3940084173 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:16.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.237+0000 7f5751a89700 1 --2- 172.21.15.189:0/3940084173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5738003a00 0x7f574c146b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:16.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.237+0000 7f5751288700 1 --2- 172.21.15.189:0/3940084173 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f574c0cdb80 0x7f574c0aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:16.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.237+0000 7f575228a700 1 -- 172.21.15.189:0/3940084173 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f574c0cdb80 msgr2=0x7f574c0aeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:16.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.237+0000 7f575228a700 1 --2- 172.21.15.189:0/3940084173 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f574c0cdb80 0x7f574c0aeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:16.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.237+0000 7f575228a700 1 -- 172.21.15.189:0/3940084173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5738003a00 msgr2=0x7f574c146b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:16.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.237+0000 7f575228a700 1 --2- 172.21.15.189:0/3940084173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5738003a00 0x7f574c146b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:16.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.237+0000 7f575228a700 1 -- 172.21.15.189:0/3940084173 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574c0ce010 con 0x7f574c0af400 2024-04-06T13:49:16.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.237+0000 7f575228a700 1 --2- 172.21.15.189:0/3940084173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574c0af400 0x7f574c158770 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7f574c0b4800 tx=0x7f574400d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:16.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.237+0000 7f57427fc700 1 -- 172.21.15.189:0/3940084173 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5744003a00 con 0x7f574c0af400 2024-04-06T13:49:16.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.237+0000 7f57427fc700 1 -- 172.21.15.189:0/3940084173 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5744003b80 con 0x7f574c0af400 2024-04-06T13:49:16.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.237+0000 7f5753ced700 1 -- 172.21.15.189:0/3940084173 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f574c0ce2f0 con 0x7f574c0af400 2024-04-06T13:49:16.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.237+0000 7f57427fc700 1 -- 172.21.15.189:0/3940084173 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574400d930 con 0x7f574c0af400 2024-04-06T13:49:16.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.237+0000 7f5753ced700 1 -- 172.21.15.189:0/3940084173 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574c0ce8b0 con 0x7f574c0af400 2024-04-06T13:49:16.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.238+0000 7f5753ced700 1 -- 172.21.15.189:0/3940084173 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574c0cedb0 con 0x7f574c0af400 2024-04-06T13:49:16.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.239+0000 7f57427fc700 1 -- 172.21.15.189:0/3940084173 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f574400da90 con 0x7f574c0af400 2024-04-06T13:49:16.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.239+0000 7f57427fc700 1 --2- 172.21.15.189:0/3940084173 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f573006aed0 0x7f573006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:16.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.239+0000 7f57427fc700 1 -- 172.21.15.189:0/3940084173 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f574408a280 con 0x7f574c0af400 2024-04-06T13:49:16.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.239+0000 7f5751a89700 1 --2- 172.21.15.189:0/3940084173 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f573006aed0 0x7f573006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:16.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.240+0000 7f5751a89700 1 --2- 172.21.15.189:0/3940084173 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f573006aed0 0x7f573006d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f573c0090b0 tx=0x7f573c01e040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:16.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.243+0000 7f57427fc700 1 -- 172.21.15.189:0/3940084173 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5744055230 con 0x7f574c0af400 2024-04-06T13:49:16.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:16.438+0000 7f5753ced700 1 -- 172.21.15.189:0/3940084173 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f574c067bb0 con 0x7f574c0af400 2024-04-06T13:49:17.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.084+0000 7f57427fc700 1 -- 172.21.15.189:0/3940084173 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v267) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5744058850 con 0x7f574c0af400 2024-04-06T13:49:17.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.089+0000 7f5753ced700 1 -- 172.21.15.189:0/3940084173 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f573006aed0 msgr2=0x7f573006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:17.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.089+0000 7f5753ced700 1 --2- 172.21.15.189:0/3940084173 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f573006aed0 0x7f573006d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f573c0090b0 tx=0x7f573c01e040 comp rx=0 tx=0).stop 2024-04-06T13:49:17.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.089+0000 7f5753ced700 1 -- 172.21.15.189:0/3940084173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574c0af400 msgr2=0x7f574c158770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:17.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.089+0000 7f5753ced700 1 --2- 172.21.15.189:0/3940084173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574c0af400 0x7f574c158770 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7f574c0b4800 tx=0x7f574400d430 comp rx=0 tx=0).stop 2024-04-06T13:49:17.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.091+0000 7f5753ced700 1 -- 172.21.15.189:0/3940084173 shutdown_connections 2024-04-06T13:49:17.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.091+0000 7f5753ced700 1 --2- 172.21.15.189:0/3940084173 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f573006aed0 0x7f573006d390 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.091+0000 7f5753ced700 1 --2- 172.21.15.189:0/3940084173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574c0af400 0x7f574c158770 unknown :-1 s=CLOSED pgs=2916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.091+0000 7f5753ced700 1 --2- 172.21.15.189:0/3940084173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5738003a00 0x7f574c146b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.091+0000 7f5753ced700 1 --2- 172.21.15.189:0/3940084173 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f574c0cdb80 0x7f574c0aeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.091+0000 7f5753ced700 1 -- 172.21.15.189:0/3940084173 >> 172.21.15.189:0/3940084173 conn(0x7f574c0d00b0 msgr2=0x7f574c0ade30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:17.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.091+0000 7f5753ced700 1 -- 172.21.15.189:0/3940084173 shutdown_connections 2024-04-06T13:49:17.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.092+0000 7f5753ced700 1 -- 172.21.15.189:0/3940084173 wait complete. 2024-04-06T13:49:17.102 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' created 2024-04-06T13:49:17.112 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-06T13:49:17.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.238+0000 7f9403b95700 1 Processor -- start 2024-04-06T13:49:17.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.238+0000 7f9403b95700 1 -- start start 2024-04-06T13:49:17.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.239+0000 7f9403b95700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f93fc06ba20 0x7f93fc166270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:17.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.239+0000 7f9403b95700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93fc0669d0 con 0x7f93fc15e8a0 2024-04-06T13:49:17.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.239+0000 7f9403b95700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93fc06ede0 con 0x7f93fc0cbc20 2024-04-06T13:49:17.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.239+0000 7f9403b95700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93fc15fd00 con 0x7f93fc06ba20 2024-04-06T13:49:17.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.239+0000 7f9401931700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f93fc15e8a0 0x7f93fc15eee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48912/0 (socket says 172.21.15.189:48912) 2024-04-06T13:49:17.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.239+0000 7f9401931700 1 -- 172.21.15.189:0/1554393466 learned_addr learned my addr 172.21.15.189:0/1554393466 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:17.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.239+0000 7f9402132700 1 --2- 172.21.15.189:0/1554393466 >> v2:172.21.15.192:3300/0 conn(0x7f93fc06ba20 0x7f93fc166270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:17.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.239+0000 7f940092f700 1 -- 172.21.15.189:0/1554393466 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772350281 0 0) 0x7f93fc0669d0 con 0x7f93fc15e8a0 2024-04-06T13:49:17.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.239+0000 7f940092f700 1 -- 172.21.15.189:0/1554393466 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93e80035b0 con 0x7f93fc15e8a0 2024-04-06T13:49:17.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f940092f700 1 -- 172.21.15.189:0/1554393466 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 799473418 0 0) 0x7f93e80035b0 con 0x7f93fc15e8a0 2024-04-06T13:49:17.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f940092f700 1 -- 172.21.15.189:0/1554393466 >> v2:172.21.15.192:3300/0 conn(0x7f93fc06ba20 msgr2=0x7f93fc166270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:17.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f940092f700 1 --2- 172.21.15.189:0/1554393466 >> v2:172.21.15.192:3300/0 conn(0x7f93fc06ba20 0x7f93fc166270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f940092f700 1 -- 172.21.15.189:0/1554393466 >> v1:172.21.15.192:6789/0 conn(0x7f93fc0cbc20 legacy=0x7f93fc15f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:17.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f940092f700 1 -- 172.21.15.189:0/1554393466 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93fc15ec80 con 0x7f93fc15e8a0 2024-04-06T13:49:17.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f940092f700 1 -- 172.21.15.189:0/1554393466 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f93ec001670 con 0x7f93fc15e8a0 2024-04-06T13:49:17.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f940092f700 1 -- 172.21.15.189:0/1554393466 >> v1:172.21.15.189:6789/0 conn(0x7f93fc15e8a0 legacy=0x7f93fc15eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:17.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f940092f700 1 --2- 172.21.15.189:0/1554393466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93e8003a00 0x7f93e8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:17.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f940092f700 1 --2- 172.21.15.189:0/1554393466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93e8004850 0x7f93e8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:17.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f940092f700 1 --2- 172.21.15.189:0/1554393466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93e8005330 0x7f93e800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:17.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f940092f700 1 -- 172.21.15.189:0/1554393466 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc15fd00 con 0x7f93e8004850 2024-04-06T13:49:17.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f940092f700 1 -- 172.21.15.189:0/1554393466 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e800c280 con 0x7f93e8003a00 2024-04-06T13:49:17.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f940092f700 1 -- 172.21.15.189:0/1554393466 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e800c440 con 0x7f93e8005330 2024-04-06T13:49:17.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f9402132700 1 --2- 172.21.15.189:0/1554393466 >> v2:172.21.15.192:3300/0 conn(0x7f93fc06ba20 0x7f93fc166270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:17.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f9401130700 1 --2- 172.21.15.189:0/1554393466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93e8004850 0x7f93e8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:17.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f9401931700 1 --2- 172.21.15.189:0/1554393466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93e8003a00 0x7f93e8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:17.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f9402132700 1 --2- 172.21.15.189:0/1554393466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93e8005330 0x7f93e800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:17.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f9401130700 1 -- 172.21.15.189:0/1554393466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93e8005330 msgr2=0x7f93e800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:17.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f9401130700 1 --2- 172.21.15.189:0/1554393466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93e8005330 0x7f93e800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f9401130700 1 -- 172.21.15.189:0/1554393466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93e8003a00 msgr2=0x7f93e8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:17.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f9401130700 1 --2- 172.21.15.189:0/1554393466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93e8003a00 0x7f93e8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f9402132700 1 -- 172.21.15.189:0/1554393466 reap_dead start 2024-04-06T13:49:17.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f9401130700 1 -- 172.21.15.189:0/1554393466 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e80035b0 con 0x7f93e8004850 2024-04-06T13:49:17.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f9401130700 1 --2- 172.21.15.189:0/1554393466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93e8004850 0x7f93e8004cd0 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f93f8007fd0 tx=0x7f93e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b24139f2e795ca2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:17.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f940092f700 1 -- 172.21.15.189:0/1554393466 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f8016070 con 0x7f93e8004850 2024-04-06T13:49:17.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f940092f700 1 -- 172.21.15.189:0/1554393466 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f8006860 con 0x7f93e8004850 2024-04-06T13:49:17.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.240+0000 7f940092f700 1 -- 172.21.15.189:0/1554393466 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f8003950 con 0x7f93e8004850 2024-04-06T13:49:17.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.241+0000 7f9403b95700 1 -- 172.21.15.189:0/1554393466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93e8004850 msgr2=0x7f93e8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:17.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.241+0000 7f9403b95700 1 --2- 172.21.15.189:0/1554393466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93e8004850 0x7f93e8004cd0 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f93f8007fd0 tx=0x7f93e8005880 comp rx=0 tx=0).stop 2024-04-06T13:49:17.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.241+0000 7f9403b95700 1 -- 172.21.15.189:0/1554393466 shutdown_connections 2024-04-06T13:49:17.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.241+0000 7f9403b95700 1 --2- 172.21.15.189:0/1554393466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93e8004850 0x7f93e8004cd0 unknown :-1 s=CLOSED pgs=2918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.241+0000 7f9403b95700 1 -- 172.21.15.189:0/1554393466 >> 172.21.15.189:0/1554393466 conn(0x7f93fc0cc290 msgr2=0x7f93fc0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:17.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.241+0000 7f9403b95700 1 -- 172.21.15.189:0/1554393466 shutdown_connections 2024-04-06T13:49:17.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.241+0000 7f9403b95700 1 -- 172.21.15.189:0/1554393466 wait complete. 2024-04-06T13:49:17.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.242+0000 7f9403b95700 1 Processor -- start 2024-04-06T13:49:17.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.242+0000 7f9403b95700 1 -- start start 2024-04-06T13:49:17.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.243+0000 7f9403b95700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93e8004850 0x7f93fc0ce960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:17.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.243+0000 7f9403b95700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93fc0cd3e0 0x7f93fc0b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:17.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.243+0000 7f9403b95700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93fc0c3ce0 0x7f93fc152940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:17.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.243+0000 7f9403b95700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc0cd8c0 con 0x7f93fc0cd3e0 2024-04-06T13:49:17.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.243+0000 7f9403b95700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc0cdba0 con 0x7f93e8004850 2024-04-06T13:49:17.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.243+0000 7f9403b95700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc0cde80 con 0x7f93fc0c3ce0 2024-04-06T13:49:17.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.243+0000 7f9401130700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93fc0cd3e0 0x7f93fc0b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:17.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.243+0000 7f9401130700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93fc0cd3e0 0x7f93fc0b4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60944/0 (socket says 172.21.15.189:60944) 2024-04-06T13:49:17.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.243+0000 7f9401130700 1 -- 172.21.15.189:0/2922090878 learned_addr learned my addr 172.21.15.189:0/2922090878 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:17.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.243+0000 7f9401931700 1 --2- 172.21.15.189:0/2922090878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93e8004850 0x7f93fc0ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:17.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.243+0000 7f9402132700 1 --2- 172.21.15.189:0/2922090878 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93fc0c3ce0 0x7f93fc152940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:17.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.243+0000 7f9401130700 1 -- 172.21.15.189:0/2922090878 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93fc0c3ce0 msgr2=0x7f93fc152940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:17.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.243+0000 7f9401130700 1 --2- 172.21.15.189:0/2922090878 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93fc0c3ce0 0x7f93fc152940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.243+0000 7f9401130700 1 -- 172.21.15.189:0/2922090878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93e8004850 msgr2=0x7f93fc0ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:17.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.243+0000 7f9401130700 1 --2- 172.21.15.189:0/2922090878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93e8004850 0x7f93fc0ce960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.243+0000 7f9401130700 1 -- 172.21.15.189:0/2922090878 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93fc0ce160 con 0x7f93fc0cd3e0 2024-04-06T13:49:17.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.243+0000 7f9402132700 1 --2- 172.21.15.189:0/2922090878 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93fc0c3ce0 0x7f93fc152940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:17.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.243+0000 7f9401130700 1 --2- 172.21.15.189:0/2922090878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93fc0cd3e0 0x7f93fc0b4830 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f93e800c250 tx=0x7f93f8003780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:17.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.244+0000 7f93f27fc700 1 -- 172.21.15.189:0/2922090878 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f8016040 con 0x7f93fc0cd3e0 2024-04-06T13:49:17.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.244+0000 7f93f27fc700 1 -- 172.21.15.189:0/2922090878 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f8005540 con 0x7f93fc0cd3e0 2024-04-06T13:49:17.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.244+0000 7f9403b95700 1 -- 172.21.15.189:0/2922090878 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93fc0ce440 con 0x7f93fc0cd3e0 2024-04-06T13:49:17.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.244+0000 7f93f27fc700 1 -- 172.21.15.189:0/2922090878 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f8007360 con 0x7f93fc0cd3e0 2024-04-06T13:49:17.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.244+0000 7f9403b95700 1 -- 172.21.15.189:0/2922090878 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93fc0b4f50 con 0x7f93fc0cd3e0 2024-04-06T13:49:17.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.245+0000 7f9403b95700 1 -- 172.21.15.189:0/2922090878 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93fc072140 con 0x7f93fc0cd3e0 2024-04-06T13:49:17.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.245+0000 7f93f27fc700 1 -- 172.21.15.189:0/2922090878 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f93f80074c0 con 0x7f93fc0cd3e0 2024-04-06T13:49:17.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.245+0000 7f93f27fc700 1 --2- 172.21.15.189:0/2922090878 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f93e006aed0 0x7f93e006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:17.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.246+0000 7f93f27fc700 1 -- 172.21.15.189:0/2922090878 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f93f8089c70 con 0x7f93fc0cd3e0 2024-04-06T13:49:17.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.246+0000 7f9401931700 1 --2- 172.21.15.189:0/2922090878 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f93e006aed0 0x7f93e006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:17.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.246+0000 7f9401931700 1 --2- 172.21.15.189:0/2922090878 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f93e006aed0 0x7f93e006d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f93fc068450 tx=0x7f93ec000fa0 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:17.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.250+0000 7f93f27fc700 1 -- 172.21.15.189:0/2922090878 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f93f8054bd0 con 0x7f93fc0cd3e0 2024-04-06T13:49:17.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.491+0000 7f9403b95700 1 -- 172.21.15.189:0/2922090878 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f93fc0b4f50 con 0x7f93fc0cd3e0 2024-04-06T13:49:17.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.498+0000 7f93f27fc700 1 -- 172.21.15.189:0/2922090878 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 35 and data pool 36 v128) v1 ==== 174+0+0 (secure 0 0 0) 0x7f93fc0b4f50 con 0x7f93fc0cd3e0 2024-04-06T13:49:17.505 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.505+0000 7f9403b95700 1 -- 172.21.15.189:0/2922090878 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f93e006aed0 msgr2=0x7f93e006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:17.505 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.505+0000 7f9403b95700 1 --2- 172.21.15.189:0/2922090878 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f93e006aed0 0x7f93e006d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f93fc068450 tx=0x7f93ec000fa0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.505 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.505+0000 7f9403b95700 1 -- 172.21.15.189:0/2922090878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93fc0cd3e0 msgr2=0x7f93fc0b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:17.505 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.505+0000 7f9403b95700 1 --2- 172.21.15.189:0/2922090878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93fc0cd3e0 0x7f93fc0b4830 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f93e800c250 tx=0x7f93f8003780 comp rx=0 tx=0).stop 2024-04-06T13:49:17.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.518+0000 7f9403b95700 1 -- 172.21.15.189:0/2922090878 shutdown_connections 2024-04-06T13:49:17.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.518+0000 7f9403b95700 1 --2- 172.21.15.189:0/2922090878 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f93e006aed0 0x7f93e006d390 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.518+0000 7f9403b95700 1 --2- 172.21.15.189:0/2922090878 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93fc0c3ce0 0x7f93fc152940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.518+0000 7f9403b95700 1 --2- 172.21.15.189:0/2922090878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93e8004850 0x7f93fc0ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.518+0000 7f9403b95700 1 --2- 172.21.15.189:0/2922090878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93fc0cd3e0 0x7f93fc0b4830 unknown :-1 s=CLOSED pgs=2919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.518+0000 7f9403b95700 1 -- 172.21.15.189:0/2922090878 >> 172.21.15.189:0/2922090878 conn(0x7f93fc0cc290 msgr2=0x7f93fc0712b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:17.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.519+0000 7f9403b95700 1 -- 172.21.15.189:0/2922090878 shutdown_connections 2024-04-06T13:49:17.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.519+0000 7f9403b95700 1 -- 172.21.15.189:0/2922090878 wait complete. 2024-04-06T13:49:17.522 INFO:teuthology.orchestra.run.smithi189.stderr:new fs with metadata pool 35 and data pool 36 2024-04-06T13:49:17.534 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:49:17.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.656+0000 7f8ebd0ca700 1 Processor -- start 2024-04-06T13:49:17.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.656+0000 7f8ebd0ca700 1 -- start start 2024-04-06T13:49:17.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.657+0000 7f8ebd0ca700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8eb815d7b0 0x7f8eb80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:17.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.657+0000 7f8ebd0ca700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb80b3990 con 0x7f8eb806a730 2024-04-06T13:49:17.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.657+0000 7f8ebd0ca700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb80b3060 con 0x7f8eb815e500 2024-04-06T13:49:17.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.657+0000 7f8ebd0ca700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb80b3340 con 0x7f8eb815d7b0 2024-04-06T13:49:17.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.657+0000 7f8eb659c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8eb806a730 0x7f8eb815ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48922/0 (socket says 172.21.15.189:48922) 2024-04-06T13:49:17.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.657+0000 7f8eb659c700 1 -- 172.21.15.189:0/1967946910 learned_addr learned my addr 172.21.15.189:0/1967946910 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:17.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.657+0000 7f8eb6d9d700 1 --2- 172.21.15.189:0/1967946910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8eb815d7b0 0x7f8eb80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:17.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.657+0000 7f8eb5d9b700 1 -- 172.21.15.189:0/1967946910 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727211601 0 0) 0x7f8eb80b3990 con 0x7f8eb806a730 2024-04-06T13:49:17.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.657+0000 7f8eb5d9b700 1 -- 172.21.15.189:0/1967946910 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ea40035b0 con 0x7f8eb806a730 2024-04-06T13:49:17.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb5d9b700 1 -- 172.21.15.189:0/1967946910 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1606327256 0 0) 0x7f8ea40035b0 con 0x7f8eb806a730 2024-04-06T13:49:17.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb5d9b700 1 -- 172.21.15.189:0/1967946910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8eb815d7b0 msgr2=0x7f8eb80b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:17.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb5d9b700 1 --2- 172.21.15.189:0/1967946910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8eb815d7b0 0x7f8eb80b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb5d9b700 1 -- 172.21.15.189:0/1967946910 >> v1:172.21.15.192:6789/0 conn(0x7f8eb815e500 legacy=0x7f8eb80717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:17.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb5d9b700 1 -- 172.21.15.189:0/1967946910 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb80b3620 con 0x7f8eb806a730 2024-04-06T13:49:17.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb5d9b700 1 -- 172.21.15.189:0/1967946910 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f8ea0001670 con 0x7f8eb806a730 2024-04-06T13:49:17.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb5d9b700 1 -- 172.21.15.189:0/1967946910 >> v1:172.21.15.189:6789/0 conn(0x7f8eb806a730 legacy=0x7f8eb815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:17.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb5d9b700 1 --2- 172.21.15.189:0/1967946910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ea4003a30 0x7f8ea4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:17.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb5d9b700 1 --2- 172.21.15.189:0/1967946910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ea40043e0 0x7f8ea40048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:17.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb5d9b700 1 --2- 172.21.15.189:0/1967946910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8eb815d7b0 0x7f8ea40051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:17.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb5d9b700 1 -- 172.21.15.189:0/1967946910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb80b3340 con 0x7f8ea4003a30 2024-04-06T13:49:17.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb5d9b700 1 -- 172.21.15.189:0/1967946910 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea400bac0 con 0x7f8ea40043e0 2024-04-06T13:49:17.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb5d9b700 1 -- 172.21.15.189:0/1967946910 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea400bc80 con 0x7f8eb815d7b0 2024-04-06T13:49:17.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb6d9d700 1 --2- 172.21.15.189:0/1967946910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ea4003a30 0x7f8ea4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:17.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb659c700 1 --2- 172.21.15.189:0/1967946910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ea40043e0 0x7f8ea40048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:17.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb759e700 1 --2- 172.21.15.189:0/1967946910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8eb815d7b0 0x7f8ea40051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:17.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb6d9d700 1 -- 172.21.15.189:0/1967946910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8eb815d7b0 msgr2=0x7f8ea40051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:17.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb6d9d700 1 --2- 172.21.15.189:0/1967946910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8eb815d7b0 0x7f8ea40051b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb6d9d700 1 -- 172.21.15.189:0/1967946910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ea40043e0 msgr2=0x7f8ea40048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:17.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb6d9d700 1 --2- 172.21.15.189:0/1967946910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ea40043e0 0x7f8ea40048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb6d9d700 1 -- 172.21.15.189:0/1967946910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea40035b0 con 0x7f8ea4003a30 2024-04-06T13:49:17.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb6d9d700 1 --2- 172.21.15.189:0/1967946910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ea4003a30 0x7f8ea4003e10 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8009b60 tx=0x7f8ea4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e878292eafec186 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:17.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb5d9b700 1 -- 172.21.15.189:0/1967946910 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea800e6c0 con 0x7f8ea4003a30 2024-04-06T13:49:17.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb5d9b700 1 -- 172.21.15.189:0/1967946910 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea800e820 con 0x7f8ea4003a30 2024-04-06T13:49:17.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.658+0000 7f8eb5d9b700 1 -- 172.21.15.189:0/1967946910 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea800eb90 con 0x7f8ea4003a30 2024-04-06T13:49:17.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.659+0000 7f8ebd0ca700 1 -- 172.21.15.189:0/1967946910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ea4003a30 msgr2=0x7f8ea4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:17.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.659+0000 7f8ebd0ca700 1 --2- 172.21.15.189:0/1967946910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ea4003a30 0x7f8ea4003e10 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8009b60 tx=0x7f8ea4005880 comp rx=0 tx=0).stop 2024-04-06T13:49:17.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.659+0000 7f8eb759e700 1 -- 172.21.15.189:0/1967946910 reap_dead start 2024-04-06T13:49:17.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.659+0000 7f8ebd0ca700 1 -- 172.21.15.189:0/1967946910 shutdown_connections 2024-04-06T13:49:17.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.659+0000 7f8ebd0ca700 1 -- 172.21.15.189:0/1967946910 >> 172.21.15.189:0/1967946910 conn(0x7f8eb80d00b0 msgr2=0x7f8eb80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:17.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.659+0000 7f8ebd0ca700 1 -- 172.21.15.189:0/1967946910 shutdown_connections 2024-04-06T13:49:17.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.659+0000 7f8ebd0ca700 1 -- 172.21.15.189:0/1967946910 wait complete. 2024-04-06T13:49:17.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.660+0000 7f8ebd0ca700 1 Processor -- start 2024-04-06T13:49:17.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.661+0000 7f8ebd0ca700 1 -- start start 2024-04-06T13:49:17.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.661+0000 7f8ebd0ca700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ea4003a30 0x7f8eb8147260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:17.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.661+0000 7f8ebd0ca700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8eb815bbc0 0x7f8eb8143da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:17.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.661+0000 7f8ebd0ca700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eb8159d50 0x7f8eb815a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:17.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.661+0000 7f8ebd0ca700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb815a740 con 0x7f8eb8159d50 2024-04-06T13:49:17.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.661+0000 7f8ebd0ca700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb815cf00 con 0x7f8eb815bbc0 2024-04-06T13:49:17.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.661+0000 7f8ebd0ca700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb815c040 con 0x7f8ea4003a30 2024-04-06T13:49:17.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.661+0000 7f8eb759e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eb8159d50 0x7f8eb815a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:17.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.661+0000 7f8eb759e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eb8159d50 0x7f8eb815a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60958/0 (socket says 172.21.15.189:60958) 2024-04-06T13:49:17.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.661+0000 7f8eb759e700 1 -- 172.21.15.189:0/890766665 learned_addr learned my addr 172.21.15.189:0/890766665 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:17.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.661+0000 7f8eb6d9d700 1 --2- 172.21.15.189:0/890766665 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ea4003a30 0x7f8eb8147260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:17.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.662+0000 7f8eb659c700 1 --2- 172.21.15.189:0/890766665 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8eb815bbc0 0x7f8eb8143da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:17.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.662+0000 7f8eb759e700 1 -- 172.21.15.189:0/890766665 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ea4003a30 msgr2=0x7f8eb8147260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:17.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.662+0000 7f8eb759e700 1 --2- 172.21.15.189:0/890766665 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ea4003a30 0x7f8eb8147260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.662+0000 7f8eb759e700 1 -- 172.21.15.189:0/890766665 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8eb815bbc0 msgr2=0x7f8eb8143da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:17.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.662+0000 7f8eb759e700 1 --2- 172.21.15.189:0/890766665 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8eb815bbc0 0x7f8eb8143da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.662+0000 7f8eb759e700 1 -- 172.21.15.189:0/890766665 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb815c2f0 con 0x7f8eb8159d50 2024-04-06T13:49:17.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.662+0000 7f8eb759e700 1 --2- 172.21.15.189:0/890766665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eb8159d50 0x7f8eb815a200 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7f8eb806d2c0 tx=0x7f8eb000d660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:17.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.662+0000 7f8eaf7fe700 1 -- 172.21.15.189:0/890766665 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb000db10 con 0x7f8eb8159d50 2024-04-06T13:49:17.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.662+0000 7f8eaf7fe700 1 -- 172.21.15.189:0/890766665 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb000dc70 con 0x7f8eb8159d50 2024-04-06T13:49:17.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.662+0000 7f8ebd0ca700 1 -- 172.21.15.189:0/890766665 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb815c5d0 con 0x7f8eb8159d50 2024-04-06T13:49:17.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.662+0000 7f8eaf7fe700 1 -- 172.21.15.189:0/890766665 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0003a00 con 0x7f8eb8159d50 2024-04-06T13:49:17.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.662+0000 7f8ebd0ca700 1 -- 172.21.15.189:0/890766665 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb815cb90 con 0x7f8eb8159d50 2024-04-06T13:49:17.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.663+0000 7f8ebd0ca700 1 -- 172.21.15.189:0/890766665 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb815cd50 con 0x7f8eb8159d50 2024-04-06T13:49:17.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.663+0000 7f8eaf7fe700 1 -- 172.21.15.189:0/890766665 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f8eb0003b60 con 0x7f8eb8159d50 2024-04-06T13:49:17.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.664+0000 7f8eaf7fe700 1 --2- 172.21.15.189:0/890766665 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8e9806aed0 0x7f8e9806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:17.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.664+0000 7f8eaf7fe700 1 -- 172.21.15.189:0/890766665 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8eb0089b80 con 0x7f8eb8159d50 2024-04-06T13:49:17.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.664+0000 7f8eb6d9d700 1 --2- 172.21.15.189:0/890766665 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8e9806aed0 0x7f8e9806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:17.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.664+0000 7f8eb6d9d700 1 --2- 172.21.15.189:0/890766665 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8e9806aed0 0x7f8e9806d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8ea80090b0 tx=0x7f8ea8008030 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:17.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.668+0000 7f8eaf7fe700 1 -- 172.21.15.189:0/890766665 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8eb0054870 con 0x7f8eb8159d50 2024-04-06T13:49:17.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.868+0000 7f8ebd0ca700 1 -- 172.21.15.189:0/890766665 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8eb8067bb0 con 0x7f8eb8159d50 2024-04-06T13:49:17.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.869+0000 7f8eaf7fe700 1 -- 172.21.15.189:0/890766665 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v268) v1 ==== 74+0+16429 (secure 0 0 0) 0x7f8eb0057e90 con 0x7f8eb8159d50 2024-04-06T13:49:17.869 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:17.870 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":268,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:49:17.492793+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:49:15.919092+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"268","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-04-06T13:49:16.459885+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"268","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:49:17.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.871+0000 7f8ebd0ca700 1 -- 172.21.15.189:0/890766665 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8e9806aed0 msgr2=0x7f8e9806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:17.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.871+0000 7f8ebd0ca700 1 --2- 172.21.15.189:0/890766665 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8e9806aed0 0x7f8e9806d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8ea80090b0 tx=0x7f8ea8008030 comp rx=0 tx=0).stop 2024-04-06T13:49:17.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.871+0000 7f8ebd0ca700 1 -- 172.21.15.189:0/890766665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eb8159d50 msgr2=0x7f8eb815a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:17.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.871+0000 7f8ebd0ca700 1 --2- 172.21.15.189:0/890766665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eb8159d50 0x7f8eb815a200 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7f8eb806d2c0 tx=0x7f8eb000d660 comp rx=0 tx=0).stop 2024-04-06T13:49:17.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.871+0000 7f8ebd0ca700 1 -- 172.21.15.189:0/890766665 shutdown_connections 2024-04-06T13:49:17.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.871+0000 7f8ebd0ca700 1 --2- 172.21.15.189:0/890766665 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8e9806aed0 0x7f8e9806d390 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.871+0000 7f8ebd0ca700 1 --2- 172.21.15.189:0/890766665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eb8159d50 0x7f8eb815a200 unknown :-1 s=CLOSED pgs=2922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.871+0000 7f8ebd0ca700 1 --2- 172.21.15.189:0/890766665 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ea4003a30 0x7f8eb8147260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.871+0000 7f8ebd0ca700 1 --2- 172.21.15.189:0/890766665 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8eb815bbc0 0x7f8eb8143da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:17.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.871+0000 7f8ebd0ca700 1 -- 172.21.15.189:0/890766665 >> 172.21.15.189:0/890766665 conn(0x7f8eb80d00b0 msgr2=0x7f8eb80ad220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:17.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.872+0000 7f8ebd0ca700 1 -- 172.21.15.189:0/890766665 shutdown_connections 2024-04-06T13:49:17.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:17.872+0000 7f8ebd0ca700 1 -- 172.21.15.189:0/890766665 wait complete. 2024-04-06T13:49:17.884 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:49:18.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.005+0000 7f31bb338700 1 Processor -- start 2024-04-06T13:49:18.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.006+0000 7f31bb338700 1 -- start start 2024-04-06T13:49:18.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.006+0000 7f31bb338700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f31b406ae40 0x7f31b40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:18.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.006+0000 7f31bb338700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31b415ddf0 0x7f31b415e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:18.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.006+0000 7f31bb338700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b40b3990 con 0x7f31b415d7b0 2024-04-06T13:49:18.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.006+0000 7f31bb338700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31b40b3060 con 0x7f31b406ae40 2024-04-06T13:49:18.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.006+0000 7f31bb338700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b40b3340 con 0x7f31b415ddf0 2024-04-06T13:49:18.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.007+0000 7f31b90d4700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f31b415d7b0 0x7f31b406a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48926/0 (socket says 172.21.15.189:48926) 2024-04-06T13:49:18.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.007+0000 7f31b90d4700 1 -- 172.21.15.189:0/1861495917 learned_addr learned my addr 172.21.15.189:0/1861495917 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:18.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.007+0000 7f31b88d3700 1 --2- 172.21.15.189:0/1861495917 >> v2:172.21.15.192:3300/0 conn(0x7f31b406ae40 0x7f31b40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:18.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.007+0000 7f31b98d5700 1 --2- 172.21.15.189:0/1861495917 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31b415ddf0 0x7f31b415e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:18.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.007+0000 7f31a7fff700 1 -- 172.21.15.189:0/1861495917 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272403106 0 0) 0x7f31b40b3990 con 0x7f31b415d7b0 2024-04-06T13:49:18.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.007+0000 7f31a7fff700 1 -- 172.21.15.189:0/1861495917 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31a00035b0 con 0x7f31b415d7b0 2024-04-06T13:49:18.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.007+0000 7f31a7fff700 1 -- 172.21.15.189:0/1861495917 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2310321158 0 0) 0x7f31a00035b0 con 0x7f31b415d7b0 2024-04-06T13:49:18.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.007+0000 7f31b98d5700 1 -- 172.21.15.189:0/1861495917 >> v2:172.21.15.192:3300/0 conn(0x7f31b406ae40 msgr2=0x7f31b40b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:18.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.007+0000 7f31b98d5700 1 --2- 172.21.15.189:0/1861495917 >> v2:172.21.15.192:3300/0 conn(0x7f31b406ae40 0x7f31b40b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:18.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.007+0000 7f31b98d5700 1 -- 172.21.15.189:0/1861495917 >> v1:172.21.15.189:6789/0 conn(0x7f31b415d7b0 legacy=0x7f31b406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:18.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.007+0000 7f31b98d5700 1 -- 172.21.15.189:0/1861495917 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b40b3620 con 0x7f31b415ddf0 2024-04-06T13:49:18.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.007+0000 7f31b98d5700 1 --2- 172.21.15.189:0/1861495917 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31b415ddf0 0x7f31b415e280 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f31ac009d90 tx=0x7f31ac00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6019ace4eb0e104 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:18.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.007+0000 7f31a7fff700 1 -- 172.21.15.189:0/1861495917 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ac00e070 con 0x7f31b415ddf0 2024-04-06T13:49:18.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.007+0000 7f31a7fff700 1 -- 172.21.15.189:0/1861495917 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ac003a20 con 0x7f31b415ddf0 2024-04-06T13:49:18.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.007+0000 7f31a7fff700 1 -- 172.21.15.189:0/1861495917 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8001670 con 0x7f31b415ddf0 2024-04-06T13:49:18.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.008+0000 7f31bb338700 1 -- 172.21.15.189:0/1861495917 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31b415ddf0 msgr2=0x7f31b415e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:18.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.008+0000 7f31bb338700 1 --2- 172.21.15.189:0/1861495917 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31b415ddf0 0x7f31b415e280 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f31ac009d90 tx=0x7f31ac00b330 comp rx=0 tx=0).stop 2024-04-06T13:49:18.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.008+0000 7f31bb338700 1 -- 172.21.15.189:0/1861495917 shutdown_connections 2024-04-06T13:49:18.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.008+0000 7f31bb338700 1 --2- 172.21.15.189:0/1861495917 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31b415ddf0 0x7f31b415e280 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:18.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.008+0000 7f31bb338700 1 --2- 172.21.15.189:0/1861495917 >> v2:172.21.15.192:3300/0 conn(0x7f31b406ae40 0x7f31b40b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:18.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.008+0000 7f31bb338700 1 -- 172.21.15.189:0/1861495917 >> 172.21.15.189:0/1861495917 conn(0x7f31b40d00b0 msgr2=0x7f31b40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:18.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.008+0000 7f31bb338700 1 -- 172.21.15.189:0/1861495917 shutdown_connections 2024-04-06T13:49:18.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.008+0000 7f31bb338700 1 -- 172.21.15.189:0/1861495917 wait complete. 2024-04-06T13:49:18.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.009+0000 7f31bb338700 1 Processor -- start 2024-04-06T13:49:18.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.009+0000 7f31bb338700 1 -- start start 2024-04-06T13:49:18.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.010+0000 7f31bb338700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31b406ae40 0x7f31b40af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:18.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.010+0000 7f31bb338700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31b415d7b0 0x7f31b40c6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:18.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.010+0000 7f31bb338700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31b415ddf0 0x7f31b415ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:18.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.010+0000 7f31bb338700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b41f3a30 con 0x7f31b415ddf0 2024-04-06T13:49:18.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.010+0000 7f31bb338700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b40aed00 con 0x7f31b406ae40 2024-04-06T13:49:18.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.010+0000 7f31bb338700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b40aefe0 con 0x7f31b415d7b0 2024-04-06T13:49:18.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.010+0000 7f31b98d5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31b415ddf0 0x7f31b415ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:18.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.010+0000 7f31b98d5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31b415ddf0 0x7f31b415ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60972/0 (socket says 172.21.15.189:60972) 2024-04-06T13:49:18.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.010+0000 7f31b98d5700 1 -- 172.21.15.189:0/3475301113 learned_addr learned my addr 172.21.15.189:0/3475301113 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:18.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.010+0000 7f31b88d3700 1 --2- 172.21.15.189:0/3475301113 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31b415d7b0 0x7f31b40c6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:18.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.010+0000 7f31b90d4700 1 --2- 172.21.15.189:0/3475301113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31b406ae40 0x7f31b40af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:18.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.010+0000 7f31b98d5700 1 -- 172.21.15.189:0/3475301113 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31b415d7b0 msgr2=0x7f31b40c6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:18.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.010+0000 7f31b98d5700 1 --2- 172.21.15.189:0/3475301113 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31b415d7b0 0x7f31b40c6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:18.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.010+0000 7f31b98d5700 1 -- 172.21.15.189:0/3475301113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31b406ae40 msgr2=0x7f31b40af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:18.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.010+0000 7f31b98d5700 1 --2- 172.21.15.189:0/3475301113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31b406ae40 0x7f31b40af7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:18.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.010+0000 7f31b98d5700 1 -- 172.21.15.189:0/3475301113 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ac009a70 con 0x7f31b415ddf0 2024-04-06T13:49:18.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.010+0000 7f31b88d3700 1 --2- 172.21.15.189:0/3475301113 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31b415d7b0 0x7f31b40c6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:18.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.010+0000 7f31b98d5700 1 --2- 172.21.15.189:0/3475301113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31b415ddf0 0x7f31b415ff30 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f31b40b3480 tx=0x7f31ac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:18.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.011+0000 7f31a5ffb700 1 -- 172.21.15.189:0/3475301113 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ac0045c0 con 0x7f31b415ddf0 2024-04-06T13:49:18.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.011+0000 7f31a5ffb700 1 -- 172.21.15.189:0/3475301113 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ac004740 con 0x7f31b415ddf0 2024-04-06T13:49:18.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.011+0000 7f31bb338700 1 -- 172.21.15.189:0/3475301113 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b40af2c0 con 0x7f31b415ddf0 2024-04-06T13:49:18.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.011+0000 7f31a5ffb700 1 -- 172.21.15.189:0/3475301113 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ac012430 con 0x7f31b415ddf0 2024-04-06T13:49:18.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.011+0000 7f31bb338700 1 -- 172.21.15.189:0/3475301113 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b4160470 con 0x7f31b415ddf0 2024-04-06T13:49:18.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.012+0000 7f31bb338700 1 -- 172.21.15.189:0/3475301113 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b4067bb0 con 0x7f31b415ddf0 2024-04-06T13:49:18.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.012+0000 7f31a5ffb700 1 -- 172.21.15.189:0/3475301113 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f31ac0125b0 con 0x7f31b415ddf0 2024-04-06T13:49:18.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.013+0000 7f31a5ffb700 1 --2- 172.21.15.189:0/3475301113 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f319806aed0 0x7f319806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:18.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.013+0000 7f31a5ffb700 1 -- 172.21.15.189:0/3475301113 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f31ac08b560 con 0x7f31b415ddf0 2024-04-06T13:49:18.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.013+0000 7f31b90d4700 1 --2- 172.21.15.189:0/3475301113 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f319806aed0 0x7f319806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:18.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.013+0000 7f31b90d4700 1 --2- 172.21.15.189:0/3475301113 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f319806aed0 0x7f319806d390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f31a80011e0 tx=0x7f31a8001550 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:18.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.017+0000 7f31a5ffb700 1 -- 172.21.15.189:0/3475301113 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f31ac059c60 con 0x7f31b415ddf0 2024-04-06T13:49:18.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.214+0000 7f31bb338700 1 -- 172.21.15.189:0/3475301113 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31b4074990 con 0x7f31b415ddf0 2024-04-06T13:49:18.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.215+0000 7f31a5ffb700 1 -- 172.21.15.189:0/3475301113 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v268) v1 ==== 74+0+16429 (secure 0 0 0) 0x7f31ac0597f0 con 0x7f31b415ddf0 2024-04-06T13:49:18.215 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:18.215 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":268,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:49:17.492793+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:49:15.919092+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"268","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-04-06T13:49:16.459885+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"268","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:49:18.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.217+0000 7f31bb338700 1 -- 172.21.15.189:0/3475301113 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f319806aed0 msgr2=0x7f319806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:18.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.217+0000 7f31bb338700 1 --2- 172.21.15.189:0/3475301113 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f319806aed0 0x7f319806d390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f31a80011e0 tx=0x7f31a8001550 comp rx=0 tx=0).stop 2024-04-06T13:49:18.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.217+0000 7f31bb338700 1 -- 172.21.15.189:0/3475301113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31b415ddf0 msgr2=0x7f31b415ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:18.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.217+0000 7f31bb338700 1 --2- 172.21.15.189:0/3475301113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31b415ddf0 0x7f31b415ff30 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f31b40b3480 tx=0x7f31ac004ad0 comp rx=0 tx=0).stop 2024-04-06T13:49:18.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.217+0000 7f31bb338700 1 -- 172.21.15.189:0/3475301113 shutdown_connections 2024-04-06T13:49:18.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.217+0000 7f31bb338700 1 --2- 172.21.15.189:0/3475301113 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f319806aed0 0x7f319806d390 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:18.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.217+0000 7f31bb338700 1 --2- 172.21.15.189:0/3475301113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31b415ddf0 0x7f31b415ff30 unknown :-1 s=CLOSED pgs=2924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:18.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.217+0000 7f31bb338700 1 --2- 172.21.15.189:0/3475301113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31b406ae40 0x7f31b40af7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:18.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.217+0000 7f31bb338700 1 --2- 172.21.15.189:0/3475301113 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31b415d7b0 0x7f31b40c6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:18.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.217+0000 7f31bb338700 1 -- 172.21.15.189:0/3475301113 >> 172.21.15.189:0/3475301113 conn(0x7f31b40d00b0 msgr2=0x7f31b415f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:18.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.217+0000 7f31bb338700 1 -- 172.21.15.189:0/3475301113 shutdown_connections 2024-04-06T13:49:18.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.217+0000 7f31bb338700 1 -- 172.21.15.189:0/3475301113 wait complete. 2024-04-06T13:49:18.230 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-06T13:49:18.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.357+0000 7fb4f4fa2700 1 Processor -- start 2024-04-06T13:49:18.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.357+0000 7fb4f4fa2700 1 -- start start 2024-04-06T13:49:18.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.358+0000 7fb4f4fa2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb4f0161a70 0x7fb4f00b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:18.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.358+0000 7fb4f4fa2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f00b23d0 0x7fb4f00b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.358+0000 7fb4f4fa2700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f00b2830 con 0x7fb4f00b3490 2024-04-06T13:49:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.358+0000 7fb4f4fa2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f015ea50 con 0x7fb4f0161a70 2024-04-06T13:49:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.358+0000 7fb4f4fa2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f0072520 con 0x7fb4f00b23d0 2024-04-06T13:49:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.358+0000 7fb4eeffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f00b23d0 0x7fb4f00b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.358+0000 7fb4ef7fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb4f0161a70 0x7fb4f00b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.358+0000 7fb4eeffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f00b23d0 0x7fb4f00b2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:48334/0 (socket says 172.21.15.189:48334) 2024-04-06T13:49:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.358+0000 7fb4eeffd700 1 -- 172.21.15.189:0/3516858250 learned_addr learned my addr 172.21.15.189:0/3516858250 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.358+0000 7fb4ef7fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb4f0161a70 0x7fb4f00b1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:45516/0 (socket says 172.21.15.189:45516) 2024-04-06T13:49:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.359+0000 7fb4eeffd700 1 -- 172.21.15.189:0/3516858250 >> v2:172.21.15.192:3300/0 conn(0x7fb4f0161a70 msgr2=0x7fb4f00b1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.359+0000 7fb4eeffd700 1 --2- 172.21.15.189:0/3516858250 >> v2:172.21.15.192:3300/0 conn(0x7fb4f0161a70 0x7fb4f00b1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.359+0000 7fb4eeffd700 1 -- 172.21.15.189:0/3516858250 >> v1:172.21.15.192:6789/0 conn(0x7fb4f00b3490 legacy=0x7fb4f01571f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.359+0000 7fb4eeffd700 1 -- 172.21.15.189:0/3516858250 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f0072090 con 0x7fb4f00b23d0 2024-04-06T13:49:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.359+0000 7fb4ef7fe700 1 --2- 172.21.15.189:0/3516858250 >> v2:172.21.15.192:3300/0 conn(0x7fb4f0161a70 0x7fb4f00b1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.359+0000 7fb4eeffd700 1 --2- 172.21.15.189:0/3516858250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f00b23d0 0x7fb4f00b2f50 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00098f0 tx=0x7fb4e0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc3b77e9d7913c81 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.359+0000 7fb4ee7fc700 1 -- 172.21.15.189:0/3516858250 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e0011830 con 0x7fb4f00b23d0 2024-04-06T13:49:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.359+0000 7fb4ee7fc700 1 -- 172.21.15.189:0/3516858250 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e0011990 con 0x7fb4f00b23d0 2024-04-06T13:49:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.359+0000 7fb4ee7fc700 1 -- 172.21.15.189:0/3516858250 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e0011d20 con 0x7fb4f00b23d0 2024-04-06T13:49:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.359+0000 7fb4f4fa2700 1 -- 172.21.15.189:0/3516858250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f00b23d0 msgr2=0x7fb4f00b2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.359+0000 7fb4f4fa2700 1 --2- 172.21.15.189:0/3516858250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f00b23d0 0x7fb4f00b2f50 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00098f0 tx=0x7fb4e0009c00 comp rx=0 tx=0).stop 2024-04-06T13:49:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.360+0000 7fb4f4fa2700 1 -- 172.21.15.189:0/3516858250 shutdown_connections 2024-04-06T13:49:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.360+0000 7fb4f4fa2700 1 --2- 172.21.15.189:0/3516858250 >> v2:172.21.15.192:3300/0 conn(0x7fb4f0161a70 0x7fb4f00b1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.360+0000 7fb4f4fa2700 1 --2- 172.21.15.189:0/3516858250 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f00b23d0 0x7fb4f00b2f50 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.360+0000 7fb4f4fa2700 1 -- 172.21.15.189:0/3516858250 >> 172.21.15.189:0/3516858250 conn(0x7fb4f00b5470 msgr2=0x7fb4f00b5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.360+0000 7fb4f4fa2700 1 -- 172.21.15.189:0/3516858250 shutdown_connections 2024-04-06T13:49:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.360+0000 7fb4f4fa2700 1 -- 172.21.15.189:0/3516858250 wait complete. 2024-04-06T13:49:18.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.361+0000 7fb4f4fa2700 1 Processor -- start 2024-04-06T13:49:18.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.361+0000 7fb4f4fa2700 1 -- start start 2024-04-06T13:49:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4f4fa2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4f00b23d0 0x7fb4f015c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4f4fa2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f00b3490 0x7fb4f015fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4f4fa2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4f0161a70 0x7fb4f01471f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4f4fa2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f014d650 con 0x7fb4f0161a70 2024-04-06T13:49:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4f4fa2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f00cdf60 con 0x7fb4f00b23d0 2024-04-06T13:49:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4f4fa2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f015c670 con 0x7fb4f00b3490 2024-04-06T13:49:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4effff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4f0161a70 0x7fb4f01471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4effff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4f0161a70 0x7fb4f01471f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60978/0 (socket says 172.21.15.189:60978) 2024-04-06T13:49:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4effff700 1 -- 172.21.15.189:0/3948566914 learned_addr learned my addr 172.21.15.189:0/3948566914 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4ef7fe700 1 --2- 172.21.15.189:0/3948566914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4f00b23d0 0x7fb4f015c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4eeffd700 1 --2- 172.21.15.189:0/3948566914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f00b3490 0x7fb4f015fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4effff700 1 -- 172.21.15.189:0/3948566914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f00b3490 msgr2=0x7fb4f015fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4effff700 1 --2- 172.21.15.189:0/3948566914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f00b3490 0x7fb4f015fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4effff700 1 -- 172.21.15.189:0/3948566914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4f00b23d0 msgr2=0x7fb4f015c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4effff700 1 --2- 172.21.15.189:0/3948566914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4f00b23d0 0x7fb4f015c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4effff700 1 -- 172.21.15.189:0/3948566914 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d80094a0 con 0x7fb4f0161a70 2024-04-06T13:49:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4effff700 1 --2- 172.21.15.189:0/3948566914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4f0161a70 0x7fb4f01471f0 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7fb4e800cc50 tx=0x7fb4e800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4e7fff700 1 -- 172.21.15.189:0/3948566914 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e800ad80 con 0x7fb4f0161a70 2024-04-06T13:49:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4e7fff700 1 -- 172.21.15.189:0/3948566914 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e800ce50 con 0x7fb4f0161a70 2024-04-06T13:49:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4f4fa2700 1 -- 172.21.15.189:0/3948566914 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4e0009550 con 0x7fb4f0161a70 2024-04-06T13:49:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.362+0000 7fb4e7fff700 1 -- 172.21.15.189:0/3948566914 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8003c40 con 0x7fb4f0161a70 2024-04-06T13:49:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.363+0000 7fb4f4fa2700 1 -- 172.21.15.189:0/3948566914 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f00cd1a0 con 0x7fb4f0161a70 2024-04-06T13:49:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.363+0000 7fb4f4fa2700 1 -- 172.21.15.189:0/3948566914 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f00676d0 con 0x7fb4f0161a70 2024-04-06T13:49:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.364+0000 7fb4e7fff700 1 -- 172.21.15.189:0/3948566914 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fb4e80050d0 con 0x7fb4f0161a70 2024-04-06T13:49:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.364+0000 7fb4e7fff700 1 --2- 172.21.15.189:0/3948566914 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb4d006aed0 0x7fb4d006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.364+0000 7fb4e7fff700 1 -- 172.21.15.189:0/3948566914 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb4e80899c0 con 0x7fb4f0161a70 2024-04-06T13:49:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.364+0000 7fb4ef7fe700 1 --2- 172.21.15.189:0/3948566914 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb4d006aed0 0x7fb4d006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.365+0000 7fb4ef7fe700 1 --2- 172.21.15.189:0/3948566914 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb4d006aed0 0x7fb4d006d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8009d00 tx=0x7fb4d8008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:18.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.369+0000 7fb4e7fff700 1 -- 172.21.15.189:0/3948566914 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb4e80546b0 con 0x7fb4f0161a70 2024-04-06T13:49:18.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:18.609+0000 7fb4f4fa2700 1 -- 172.21.15.189:0/3948566914 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb4f00cfdb0 con 0x7fb4f0161a70 2024-04-06T13:49:18.913 INFO:tasks.ceph.mds.b.smithi192.stderr:2024-04-06T13:49:18.913+0000 7f940c17e700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T13:49:19.505 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.505+0000 7fb4e7fff700 1 -- 172.21.15.189:0/3948566914 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v130) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb4e8057cd0 con 0x7fb4f0161a70 2024-04-06T13:49:19.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.508+0000 7fb4f4fa2700 1 -- 172.21.15.189:0/3948566914 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb4d006aed0 msgr2=0x7fb4d006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:19.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.508+0000 7fb4f4fa2700 1 --2- 172.21.15.189:0/3948566914 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb4d006aed0 0x7fb4d006d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8009d00 tx=0x7fb4d8008040 comp rx=0 tx=0).stop 2024-04-06T13:49:19.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.508+0000 7fb4f4fa2700 1 -- 172.21.15.189:0/3948566914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4f0161a70 msgr2=0x7fb4f01471f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:19.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.508+0000 7fb4f4fa2700 1 --2- 172.21.15.189:0/3948566914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4f0161a70 0x7fb4f01471f0 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7fb4e800cc50 tx=0x7fb4e800a340 comp rx=0 tx=0).stop 2024-04-06T13:49:19.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.508+0000 7fb4f4fa2700 1 -- 172.21.15.189:0/3948566914 shutdown_connections 2024-04-06T13:49:19.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.508+0000 7fb4f4fa2700 1 --2- 172.21.15.189:0/3948566914 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb4d006aed0 0x7fb4d006d390 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:19.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.508+0000 7fb4f4fa2700 1 --2- 172.21.15.189:0/3948566914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4f0161a70 0x7fb4f01471f0 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:19.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.508+0000 7fb4f4fa2700 1 --2- 172.21.15.189:0/3948566914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4f00b23d0 0x7fb4f015c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:19.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.508+0000 7fb4f4fa2700 1 --2- 172.21.15.189:0/3948566914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f00b3490 0x7fb4f015fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:19.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.508+0000 7fb4f4fa2700 1 -- 172.21.15.189:0/3948566914 >> 172.21.15.189:0/3948566914 conn(0x7fb4f00b5470 msgr2=0x7fb4f00cc140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:19.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.508+0000 7fb4f4fa2700 1 -- 172.21.15.189:0/3948566914 shutdown_connections 2024-04-06T13:49:19.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.508+0000 7fb4f4fa2700 1 -- 172.21.15.189:0/3948566914 wait complete. 2024-04-06T13:49:19.523 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-06T13:49:19.523 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-06T13:49:19.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.653+0000 7fcfff8e6700 1 Processor -- start 2024-04-06T13:49:19.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.654+0000 7fcfff8e6700 1 -- start start 2024-04-06T13:49:19.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.654+0000 7fcfff8e6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcff815d7b0 0x7fcff80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:19.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfff8e6700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcff806a730 0x7fcff806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:19.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfff8e6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff80b3990 con 0x7fcff815ddf0 2024-04-06T13:49:19.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfff8e6700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcff806b0f0 con 0x7fcff806a730 2024-04-06T13:49:19.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfff8e6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff80b3060 con 0x7fcff815d7b0 2024-04-06T13:49:19.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcffde83700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcff815ddf0 0x7fcff815e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40020/0 (socket says 172.21.15.189:40020) 2024-04-06T13:49:19.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcffde83700 1 -- 172.21.15.189:0/977406341 learned_addr learned my addr 172.21.15.189:0/977406341 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:19.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfeffff700 1 -- 172.21.15.189:0/977406341 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371437377 0 0) 0x7fcff80b3990 con 0x7fcff815ddf0 2024-04-06T13:49:19.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfeffff700 1 -- 172.21.15.189:0/977406341 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfe40035b0 con 0x7fcff815ddf0 2024-04-06T13:49:19.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcffce81700 1 --2- 172.21.15.189:0/977406341 >> v2:172.21.15.192:3300/0 conn(0x7fcff806a730 0x7fcff806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:19.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcffd682700 1 --2- 172.21.15.189:0/977406341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcff815d7b0 0x7fcff80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:19.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfeffff700 1 -- 172.21.15.189:0/977406341 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 51998771 0 0) 0x7fcfe40035b0 con 0x7fcff815ddf0 2024-04-06T13:49:19.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfeffff700 1 -- 172.21.15.189:0/977406341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcff815d7b0 msgr2=0x7fcff80b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:19.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfeffff700 1 --2- 172.21.15.189:0/977406341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcff815d7b0 0x7fcff80b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:19.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfeffff700 1 -- 172.21.15.189:0/977406341 >> v2:172.21.15.192:3300/0 conn(0x7fcff806a730 msgr2=0x7fcff806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:19.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfeffff700 1 --2- 172.21.15.189:0/977406341 >> v2:172.21.15.192:3300/0 conn(0x7fcff806a730 0x7fcff806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:19.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfeffff700 1 -- 172.21.15.189:0/977406341 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff80b3340 con 0x7fcff815ddf0 2024-04-06T13:49:19.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfeffff700 1 -- 172.21.15.189:0/977406341 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fcff00019f0 con 0x7fcff815ddf0 2024-04-06T13:49:19.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcffce81700 1 --2- 172.21.15.189:0/977406341 >> v2:172.21.15.192:3300/0 conn(0x7fcff806a730 0x7fcff806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:19.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfeffff700 1 -- 172.21.15.189:0/977406341 >> v1:172.21.15.189:6789/0 conn(0x7fcff815ddf0 legacy=0x7fcff815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:19.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfeffff700 1 --2- 172.21.15.189:0/977406341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcff815d7b0 0x7fcfe4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:19.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfeffff700 1 --2- 172.21.15.189:0/977406341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfe4003f80 0x7fcfe4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:19.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfeffff700 1 --2- 172.21.15.189:0/977406341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfe4004d90 0x7fcfe4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:19.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfeffff700 1 -- 172.21.15.189:0/977406341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff806b0f0 con 0x7fcfe4004d90 2024-04-06T13:49:19.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfeffff700 1 -- 172.21.15.189:0/977406341 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff80b3060 con 0x7fcfe4003f80 2024-04-06T13:49:19.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcfeffff700 1 -- 172.21.15.189:0/977406341 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe400bb10 con 0x7fcff815d7b0 2024-04-06T13:49:19.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcffde83700 1 --2- 172.21.15.189:0/977406341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfe4004d90 0x7fcfe4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:19.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcffd682700 1 --2- 172.21.15.189:0/977406341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcff815d7b0 0x7fcfe4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:19.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.655+0000 7fcffce81700 1 --2- 172.21.15.189:0/977406341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfe4003f80 0x7fcfe4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:19.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.656+0000 7fcffde83700 1 -- 172.21.15.189:0/977406341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcff815d7b0 msgr2=0x7fcfe4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:19.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.656+0000 7fcffde83700 1 --2- 172.21.15.189:0/977406341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcff815d7b0 0x7fcfe4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:19.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.656+0000 7fcffde83700 1 -- 172.21.15.189:0/977406341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfe4003f80 msgr2=0x7fcfe4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:19.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.656+0000 7fcffde83700 1 --2- 172.21.15.189:0/977406341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfe4003f80 0x7fcfe4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:19.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.656+0000 7fcffde83700 1 -- 172.21.15.189:0/977406341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe40035b0 con 0x7fcfe4004d90 2024-04-06T13:49:19.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.656+0000 7fcffde83700 1 --2- 172.21.15.189:0/977406341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfe4004d90 0x7fcfe4005210 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7fcff000a900 tx=0x7fcfe4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d1265d7317ad1b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:19.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.656+0000 7fcfeffff700 1 -- 172.21.15.189:0/977406341 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff000a340 con 0x7fcfe4004d90 2024-04-06T13:49:19.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.656+0000 7fcfeffff700 1 -- 172.21.15.189:0/977406341 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff0012620 con 0x7fcfe4004d90 2024-04-06T13:49:19.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.656+0000 7fcfeffff700 1 -- 172.21.15.189:0/977406341 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff00129d0 con 0x7fcfe4004d90 2024-04-06T13:49:19.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.656+0000 7fcfff8e6700 1 -- 172.21.15.189:0/977406341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfe4004d90 msgr2=0x7fcfe4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:19.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.656+0000 7fcfff8e6700 1 --2- 172.21.15.189:0/977406341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfe4004d90 0x7fcfe4005210 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7fcff000a900 tx=0x7fcfe4005880 comp rx=0 tx=0).stop 2024-04-06T13:49:19.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.656+0000 7fcffde83700 1 -- 172.21.15.189:0/977406341 reap_dead start 2024-04-06T13:49:19.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.657+0000 7fcfff8e6700 1 -- 172.21.15.189:0/977406341 shutdown_connections 2024-04-06T13:49:19.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.657+0000 7fcfff8e6700 1 -- 172.21.15.189:0/977406341 >> 172.21.15.189:0/977406341 conn(0x7fcff80d00b0 msgr2=0x7fcff80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:19.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.657+0000 7fcfff8e6700 1 -- 172.21.15.189:0/977406341 shutdown_connections 2024-04-06T13:49:19.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.657+0000 7fcfff8e6700 1 -- 172.21.15.189:0/977406341 wait complete. 2024-04-06T13:49:19.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.658+0000 7fcfff8e6700 1 Processor -- start 2024-04-06T13:49:19.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.658+0000 7fcfff8e6700 1 -- start start 2024-04-06T13:49:19.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.659+0000 7fcfff8e6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcff8149e90 0x7fcff815adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:19.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.659+0000 7fcfff8e6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcff814a270 0x7fcff815f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:19.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.659+0000 7fcfff8e6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcff815fdb0 0x7fcff81f62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:19.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.659+0000 7fcfff8e6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff8160240 con 0x7fcff814a270 2024-04-06T13:49:19.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.659+0000 7fcfff8e6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff815ccb0 con 0x7fcff815fdb0 2024-04-06T13:49:19.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.659+0000 7fcfff8e6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff815c0a0 con 0x7fcff8149e90 2024-04-06T13:49:19.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.659+0000 7fcffce81700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcff814a270 0x7fcff815f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:19.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.659+0000 7fcffce81700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcff814a270 0x7fcff815f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46176/0 (socket says 172.21.15.189:46176) 2024-04-06T13:49:19.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.659+0000 7fcffce81700 1 -- 172.21.15.189:0/3641193541 learned_addr learned my addr 172.21.15.189:0/3641193541 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:19.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.659+0000 7fcffd682700 1 --2- 172.21.15.189:0/3641193541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcff8149e90 0x7fcff815adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:19.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.659+0000 7fcffde83700 1 --2- 172.21.15.189:0/3641193541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcff815fdb0 0x7fcff81f62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:19.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.659+0000 7fcffce81700 1 -- 172.21.15.189:0/3641193541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcff8149e90 msgr2=0x7fcff815adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:19.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.660+0000 7fcffce81700 1 --2- 172.21.15.189:0/3641193541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcff8149e90 0x7fcff815adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:19.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.660+0000 7fcffce81700 1 -- 172.21.15.189:0/3641193541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcff815fdb0 msgr2=0x7fcff81f62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:19.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.660+0000 7fcffce81700 1 --2- 172.21.15.189:0/3641193541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcff815fdb0 0x7fcff81f62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:19.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.660+0000 7fcffce81700 1 -- 172.21.15.189:0/3641193541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff815c380 con 0x7fcff814a270 2024-04-06T13:49:19.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.660+0000 7fcffd682700 1 --2- 172.21.15.189:0/3641193541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcff8149e90 0x7fcff815adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:19.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.660+0000 7fcffde83700 1 --2- 172.21.15.189:0/3641193541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcff815fdb0 0x7fcff81f62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:19.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.660+0000 7fcffce81700 1 --2- 172.21.15.189:0/3641193541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcff814a270 0x7fcff815f870 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80116e0 tx=0x7fcfe8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:19.669 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.660+0000 7fcfedffb700 1 -- 172.21.15.189:0/3641193541 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe8004440 con 0x7fcff814a270 2024-04-06T13:49:19.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.660+0000 7fcfedffb700 1 -- 172.21.15.189:0/3641193541 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe80045a0 con 0x7fcff814a270 2024-04-06T13:49:19.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.660+0000 7fcfff8e6700 1 -- 172.21.15.189:0/3641193541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff815c660 con 0x7fcff814a270 2024-04-06T13:49:19.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.660+0000 7fcfedffb700 1 -- 172.21.15.189:0/3641193541 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe80038a0 con 0x7fcff814a270 2024-04-06T13:49:19.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.660+0000 7fcfff8e6700 1 -- 172.21.15.189:0/3641193541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff814aeb0 con 0x7fcff814a270 2024-04-06T13:49:19.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.661+0000 7fcfff8e6700 1 -- 172.21.15.189:0/3641193541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff8074990 con 0x7fcff814a270 2024-04-06T13:49:19.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.661+0000 7fcfedffb700 1 -- 172.21.15.189:0/3641193541 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fcfe8003a00 con 0x7fcff814a270 2024-04-06T13:49:19.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.662+0000 7fcfedffb700 1 --2- 172.21.15.189:0/3641193541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcfdc06aed0 0x7fcfdc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:19.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.662+0000 7fcffd682700 1 --2- 172.21.15.189:0/3641193541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcfdc06aed0 0x7fcfdc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:19.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.662+0000 7fcfedffb700 1 -- 172.21.15.189:0/3641193541 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcfe8089e00 con 0x7fcff814a270 2024-04-06T13:49:19.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.662+0000 7fcffd682700 1 --2- 172.21.15.189:0/3641193541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcfdc06aed0 0x7fcfdc06d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fcff814abb0 tx=0x7fcff4008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:19.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.666+0000 7fcfedffb700 1 -- 172.21.15.189:0/3641193541 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fcfe80549e0 con 0x7fcff814a270 2024-04-06T13:49:19.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:19.906+0000 7fcfff8e6700 1 -- 172.21.15.189:0/3641193541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fcff80af4b0 con 0x7fcff814a270 2024-04-06T13:49:20.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.509+0000 7fcfedffb700 1 -- 172.21.15.189:0/3641193541 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v131) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcfe800c0c0 con 0x7fcff814a270 2024-04-06T13:49:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.511+0000 7fcfff8e6700 1 -- 172.21.15.189:0/3641193541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcfdc06aed0 msgr2=0x7fcfdc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.511+0000 7fcfff8e6700 1 --2- 172.21.15.189:0/3641193541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcfdc06aed0 0x7fcfdc06d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fcff814abb0 tx=0x7fcff4008040 comp rx=0 tx=0).stop 2024-04-06T13:49:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.511+0000 7fcfff8e6700 1 -- 172.21.15.189:0/3641193541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcff814a270 msgr2=0x7fcff815f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.511+0000 7fcfff8e6700 1 --2- 172.21.15.189:0/3641193541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcff814a270 0x7fcff815f870 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80116e0 tx=0x7fcfe8011aa0 comp rx=0 tx=0).stop 2024-04-06T13:49:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.511+0000 7fcfff8e6700 1 -- 172.21.15.189:0/3641193541 shutdown_connections 2024-04-06T13:49:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.511+0000 7fcfff8e6700 1 --2- 172.21.15.189:0/3641193541 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcfdc06aed0 0x7fcfdc06d390 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.511+0000 7fcfff8e6700 1 --2- 172.21.15.189:0/3641193541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcff815fdb0 0x7fcff81f62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.511+0000 7fcfff8e6700 1 --2- 172.21.15.189:0/3641193541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcff8149e90 0x7fcff815adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.511+0000 7fcfff8e6700 1 --2- 172.21.15.189:0/3641193541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcff814a270 0x7fcff815f870 unknown :-1 s=CLOSED pgs=2928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.512+0000 7fcfff8e6700 1 -- 172.21.15.189:0/3641193541 >> 172.21.15.189:0/3641193541 conn(0x7fcff80d00b0 msgr2=0x7fcff806fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.512+0000 7fcfff8e6700 1 -- 172.21.15.189:0/3641193541 shutdown_connections 2024-04-06T13:49:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.512+0000 7fcfff8e6700 1 -- 172.21.15.189:0/3641193541 wait complete. 2024-04-06T13:49:20.526 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-06T13:49:20.526 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:49:20.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.654+0000 7f48f56c0700 1 Processor -- start 2024-04-06T13:49:20.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.654+0000 7f48f56c0700 1 -- start start 2024-04-06T13:49:20.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.654+0000 7f48f56c0700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f48f015d6f0 0x7f48f015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:20.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.654+0000 7f48f56c0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f48f00ce730 0x7f48f00d1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:20.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.655+0000 7f48f56c0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f015eac0 con 0x7f48f00b8d70 2024-04-06T13:49:20.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.655+0000 7f48f56c0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48f015e010 con 0x7f48f00ce730 2024-04-06T13:49:20.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.655+0000 7f48f56c0700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48f015e220 con 0x7f48f015d6f0 2024-04-06T13:49:20.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.655+0000 7f48ef7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f48f00ce730 0x7f48f00d1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:20.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.655+0000 7f48ee7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f48f00b8d70 0x7f48f00ce020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40036/0 (socket says 172.21.15.189:40036) 2024-04-06T13:49:20.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.655+0000 7f48ee7fc700 1 -- 172.21.15.189:0/2999652365 learned_addr learned my addr 172.21.15.189:0/2999652365 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:20.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.655+0000 7f48eeffd700 1 --2- 172.21.15.189:0/2999652365 >> v2:172.21.15.192:3300/0 conn(0x7f48f015d6f0 0x7f48f015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:20.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.655+0000 7f48ef7fe700 1 -- 172.21.15.189:0/2999652365 >> v2:172.21.15.192:3300/0 conn(0x7f48f015d6f0 msgr2=0x7f48f015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:20.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.655+0000 7f48ef7fe700 1 --2- 172.21.15.189:0/2999652365 >> v2:172.21.15.192:3300/0 conn(0x7f48f015d6f0 0x7f48f015dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:20.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.655+0000 7f48ef7fe700 1 -- 172.21.15.189:0/2999652365 >> v1:172.21.15.189:6789/0 conn(0x7f48f00b8d70 legacy=0x7f48f00ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:20.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.655+0000 7f48edffb700 1 -- 172.21.15.189:0/2999652365 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111858223 0 0) 0x7f48f015eac0 con 0x7f48f00b8d70 2024-04-06T13:49:20.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.655+0000 7f48ef7fe700 1 -- 172.21.15.189:0/2999652365 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f015e500 con 0x7f48f00ce730 2024-04-06T13:49:20.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.655+0000 7f48eeffd700 1 --2- 172.21.15.189:0/2999652365 >> v2:172.21.15.192:3300/0 conn(0x7f48f015d6f0 0x7f48f015dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:20.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.655+0000 7f48ef7fe700 1 --2- 172.21.15.189:0/2999652365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f00ce730 0x7f48f00d1e00 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7f48f00d4410 tx=0x7f48e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c29825281b19a8a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:20.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.655+0000 7f48edffb700 1 -- 172.21.15.189:0/2999652365 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e800e070 con 0x7f48f00ce730 2024-04-06T13:49:20.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.655+0000 7f48edffb700 1 -- 172.21.15.189:0/2999652365 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e800bbd0 con 0x7f48f00ce730 2024-04-06T13:49:20.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.655+0000 7f48edffb700 1 -- 172.21.15.189:0/2999652365 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e8012430 con 0x7f48f00ce730 2024-04-06T13:49:20.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.656+0000 7f48f56c0700 1 -- 172.21.15.189:0/2999652365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f00ce730 msgr2=0x7f48f00d1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:20.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.656+0000 7f48f56c0700 1 --2- 172.21.15.189:0/2999652365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f00ce730 0x7f48f00d1e00 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7f48f00d4410 tx=0x7f48e800b330 comp rx=0 tx=0).stop 2024-04-06T13:49:20.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.656+0000 7f48f56c0700 1 -- 172.21.15.189:0/2999652365 shutdown_connections 2024-04-06T13:49:20.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.656+0000 7f48f56c0700 1 --2- 172.21.15.189:0/2999652365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f00ce730 0x7f48f00d1e00 unknown :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:20.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.656+0000 7f48f56c0700 1 --2- 172.21.15.189:0/2999652365 >> v2:172.21.15.192:3300/0 conn(0x7f48f015d6f0 0x7f48f015dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:20.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.656+0000 7f48f56c0700 1 -- 172.21.15.189:0/2999652365 >> 172.21.15.189:0/2999652365 conn(0x7f48f0072490 msgr2=0x7f48f006b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:20.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.656+0000 7f48f56c0700 1 -- 172.21.15.189:0/2999652365 shutdown_connections 2024-04-06T13:49:20.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.656+0000 7f48f56c0700 1 -- 172.21.15.189:0/2999652365 wait complete. 2024-04-06T13:49:20.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.657+0000 7f48f56c0700 1 Processor -- start 2024-04-06T13:49:20.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.658+0000 7f48f56c0700 1 -- start start 2024-04-06T13:49:20.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.658+0000 7f48f56c0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f00b8d70 0x7f48f0157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:20.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.658+0000 7f48f56c0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48f00ce730 0x7f48f0159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:20.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.658+0000 7f48f56c0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48f015d6f0 0x7f48f00b4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:20.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.658+0000 7f48f56c0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f00cdc30 con 0x7f48f00b8d70 2024-04-06T13:49:20.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.658+0000 7f48f56c0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f0159c90 con 0x7f48f015d6f0 2024-04-06T13:49:20.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.658+0000 7f48f56c0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f00cde60 con 0x7f48f00ce730 2024-04-06T13:49:20.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.659+0000 7f48eeffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f00b8d70 0x7f48f0157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:20.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.659+0000 7f48eeffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f00b8d70 0x7f48f0157940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46194/0 (socket says 172.21.15.189:46194) 2024-04-06T13:49:20.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.659+0000 7f48eeffd700 1 -- 172.21.15.189:0/896058747 learned_addr learned my addr 172.21.15.189:0/896058747 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:20.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.659+0000 7f48ef7fe700 1 --2- 172.21.15.189:0/896058747 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48f015d6f0 0x7f48f00b4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:20.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.659+0000 7f48ee7fc700 1 --2- 172.21.15.189:0/896058747 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48f00ce730 0x7f48f0159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:20.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.659+0000 7f48eeffd700 1 -- 172.21.15.189:0/896058747 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48f00ce730 msgr2=0x7f48f0159750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:20.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.659+0000 7f48eeffd700 1 --2- 172.21.15.189:0/896058747 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48f00ce730 0x7f48f0159750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:20.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.659+0000 7f48eeffd700 1 -- 172.21.15.189:0/896058747 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48f015d6f0 msgr2=0x7f48f00b4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:20.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.659+0000 7f48eeffd700 1 --2- 172.21.15.189:0/896058747 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48f015d6f0 0x7f48f00b4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:20.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.659+0000 7f48eeffd700 1 -- 172.21.15.189:0/896058747 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e8009a70 con 0x7f48f00b8d70 2024-04-06T13:49:20.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.659+0000 7f48ee7fc700 1 --2- 172.21.15.189:0/896058747 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48f00ce730 0x7f48f0159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:20.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.659+0000 7f48ef7fe700 1 --2- 172.21.15.189:0/896058747 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48f015d6f0 0x7f48f00b4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:20.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.659+0000 7f48eeffd700 1 --2- 172.21.15.189:0/896058747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f00b8d70 0x7f48f0157940 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7f48e000c7f0 tx=0x7f48e000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:20.662 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.659+0000 7f48d77fe700 1 -- 172.21.15.189:0/896058747 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e0011800 con 0x7f48f00b8d70 2024-04-06T13:49:20.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.659+0000 7f48d77fe700 1 -- 172.21.15.189:0/896058747 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e0011960 con 0x7f48f00b8d70 2024-04-06T13:49:20.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.659+0000 7f48f56c0700 1 -- 172.21.15.189:0/896058747 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f00ce140 con 0x7f48f00b8d70 2024-04-06T13:49:20.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.659+0000 7f48d77fe700 1 -- 172.21.15.189:0/896058747 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e0011d10 con 0x7f48f00b8d70 2024-04-06T13:49:20.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.659+0000 7f48f56c0700 1 -- 172.21.15.189:0/896058747 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f00ca4b0 con 0x7f48f00b8d70 2024-04-06T13:49:20.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.660+0000 7f48f56c0700 1 -- 172.21.15.189:0/896058747 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48f0074990 con 0x7f48f00b8d70 2024-04-06T13:49:20.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.661+0000 7f48d77fe700 1 -- 172.21.15.189:0/896058747 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f48e0018360 con 0x7f48f00b8d70 2024-04-06T13:49:20.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.661+0000 7f48d77fe700 1 --2- 172.21.15.189:0/896058747 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f48dc06aed0 0x7f48dc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:20.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.661+0000 7f48d77fe700 1 -- 172.21.15.189:0/896058747 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f48e0089a20 con 0x7f48f00b8d70 2024-04-06T13:49:20.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.661+0000 7f48ee7fc700 1 --2- 172.21.15.189:0/896058747 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f48dc06aed0 0x7f48dc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:20.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.661+0000 7f48ee7fc700 1 --2- 172.21.15.189:0/896058747 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f48dc06aed0 0x7f48dc06d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f48f00b4990 tx=0x7f48e4009040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:20.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.665+0000 7f48d77fe700 1 -- 172.21.15.189:0/896058747 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f48e0054580 con 0x7f48f00b8d70 2024-04-06T13:49:20.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.906+0000 7f48f56c0700 1 -- 172.21.15.189:0/896058747 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48f00d1e00 con 0x7f48f00b8d70 2024-04-06T13:49:20.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.906+0000 7f48d77fe700 1 -- 172.21.15.189:0/896058747 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+4757 (secure 0 0 0) 0x7f48e0057ba0 con 0x7f48f00b8d70 2024-04-06T13:49:20.906 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:20.906 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":131,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3073866011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3073866011},{"type":"v1","addr":"172.21.15.189:6836","nonce":3073866011}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":9917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/4213774859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4213774859},{"type":"v1","addr":"172.21.15.192:6834","nonce":4213774859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":11392,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/963373121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":963373121},{"type":"v1","addr":"172.21.15.189:6837","nonce":963373121}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:49:17.493194+0000","modified":"2024-04-06T13:49:20.503775+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12885},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12885":{"gid":12885,"name":"b","rank":0,"incarnation":128,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/1094572246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":1094572246},{"type":"v1","addr":"172.21.15.192:6835","nonce":1094572246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-04-06T13:49:20.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.908+0000 7f48f56c0700 1 -- 172.21.15.189:0/896058747 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f48dc06aed0 msgr2=0x7f48dc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:20.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.908+0000 7f48f56c0700 1 --2- 172.21.15.189:0/896058747 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f48dc06aed0 0x7f48dc06d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f48f00b4990 tx=0x7f48e4009040 comp rx=0 tx=0).stop 2024-04-06T13:49:20.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.908+0000 7f48f56c0700 1 -- 172.21.15.189:0/896058747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f00b8d70 msgr2=0x7f48f0157940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:20.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.908+0000 7f48f56c0700 1 --2- 172.21.15.189:0/896058747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f00b8d70 0x7f48f0157940 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7f48e000c7f0 tx=0x7f48e000cbb0 comp rx=0 tx=0).stop 2024-04-06T13:49:20.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.908+0000 7f48f56c0700 1 -- 172.21.15.189:0/896058747 shutdown_connections 2024-04-06T13:49:20.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.908+0000 7f48f56c0700 1 --2- 172.21.15.189:0/896058747 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f48dc06aed0 0x7f48dc06d390 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:20.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.908+0000 7f48f56c0700 1 --2- 172.21.15.189:0/896058747 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48f015d6f0 0x7f48f00b4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:20.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.908+0000 7f48f56c0700 1 --2- 172.21.15.189:0/896058747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f00b8d70 0x7f48f0157940 unknown :-1 s=CLOSED pgs=2931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:20.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.908+0000 7f48f56c0700 1 --2- 172.21.15.189:0/896058747 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48f00ce730 0x7f48f0159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:20.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.909+0000 7f48f56c0700 1 -- 172.21.15.189:0/896058747 >> 172.21.15.189:0/896058747 conn(0x7f48f0072490 msgr2=0x7f48f00b7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:20.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.909+0000 7f48f56c0700 1 -- 172.21.15.189:0/896058747 shutdown_connections 2024-04-06T13:49:20.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:20.909+0000 7f48f56c0700 1 -- 172.21.15.189:0/896058747 wait complete. 2024-04-06T13:49:20.910 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 131 2024-04-06T13:49:20.921 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:49:21.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.042+0000 7fa3831c0700 1 Processor -- start 2024-04-06T13:49:21.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.042+0000 7fa3831c0700 1 -- start start 2024-04-06T13:49:21.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.043+0000 7fa3831c0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa37c0cedf0 0x7fa37c0cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:21.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.043+0000 7fa3831c0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37c0cf730 con 0x7fa37c146dd0 2024-04-06T13:49:21.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.043+0000 7fa3831c0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37c06d850 con 0x7fa37c0d17a0 2024-04-06T13:49:21.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.043+0000 7fa3831c0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa37c06beb0 con 0x7fa37c0cedf0 2024-04-06T13:49:21.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.043+0000 7fa38175d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa37c0cedf0 0x7fa37c0cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:21.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.043+0000 7fa37bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa37c146dd0 0x7fa37c06d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40044/0 (socket says 172.21.15.189:40044) 2024-04-06T13:49:21.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.043+0000 7fa37bfff700 1 -- 172.21.15.189:0/1508400778 learned_addr learned my addr 172.21.15.189:0/1508400778 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:21.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.043+0000 7fa38175d700 1 -- 172.21.15.189:0/1508400778 >> v1:172.21.15.192:6789/0 conn(0x7fa37c0d17a0 legacy=0x7fa37c06b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:21.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.043+0000 7fa38175d700 1 -- 172.21.15.189:0/1508400778 >> v1:172.21.15.189:6789/0 conn(0x7fa37c146dd0 legacy=0x7fa37c06d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:21.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.043+0000 7fa37b7fe700 1 -- 172.21.15.189:0/1508400778 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525238820 0 0) 0x7fa37c0cf730 con 0x7fa37c146dd0 2024-04-06T13:49:21.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.043+0000 7fa38175d700 1 -- 172.21.15.189:0/1508400778 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c06c0a0 con 0x7fa37c0cedf0 2024-04-06T13:49:21.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.044+0000 7fa38175d700 1 --2- 172.21.15.189:0/1508400778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa37c0cedf0 0x7fa37c0cf1f0 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7fa374009d90 tx=0x7fa37400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4372a51529cb61b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:21.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.044+0000 7fa37b7fe700 1 -- 172.21.15.189:0/1508400778 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37400e070 con 0x7fa37c0cedf0 2024-04-06T13:49:21.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.044+0000 7fa37b7fe700 1 -- 172.21.15.189:0/1508400778 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37400bb90 con 0x7fa37c0cedf0 2024-04-06T13:49:21.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.044+0000 7fa37b7fe700 1 -- 172.21.15.189:0/1508400778 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa374003a20 con 0x7fa37c0cedf0 2024-04-06T13:49:21.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.044+0000 7fa3831c0700 1 -- 172.21.15.189:0/1508400778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa37c0cedf0 msgr2=0x7fa37c0cf1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:21.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.044+0000 7fa3831c0700 1 --2- 172.21.15.189:0/1508400778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa37c0cedf0 0x7fa37c0cf1f0 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7fa374009d90 tx=0x7fa37400b330 comp rx=0 tx=0).stop 2024-04-06T13:49:21.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.044+0000 7fa3831c0700 1 -- 172.21.15.189:0/1508400778 shutdown_connections 2024-04-06T13:49:21.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.044+0000 7fa3831c0700 1 --2- 172.21.15.189:0/1508400778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa37c0cedf0 0x7fa37c0cf1f0 unknown :-1 s=CLOSED pgs=2933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.044+0000 7fa3831c0700 1 -- 172.21.15.189:0/1508400778 >> 172.21.15.189:0/1508400778 conn(0x7fa37c0b2db0 msgr2=0x7fa37c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:21.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.044+0000 7fa3831c0700 1 -- 172.21.15.189:0/1508400778 shutdown_connections 2024-04-06T13:49:21.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.044+0000 7fa3831c0700 1 -- 172.21.15.189:0/1508400778 wait complete. 2024-04-06T13:49:21.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.045+0000 7fa3831c0700 1 Processor -- start 2024-04-06T13:49:21.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.045+0000 7fa3831c0700 1 -- start start 2024-04-06T13:49:21.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.046+0000 7fa3831c0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa37c0cedf0 0x7fa37c15fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:21.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.046+0000 7fa3831c0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa37c0d17a0 0x7fa37c0b4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:21.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.046+0000 7fa3831c0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa37c146dd0 0x7fa37c15ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:21.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.046+0000 7fa3831c0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c15ff50 con 0x7fa37c146dd0 2024-04-06T13:49:21.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.046+0000 7fa3831c0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c0b54e0 con 0x7fa37c0cedf0 2024-04-06T13:49:21.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.046+0000 7fa3831c0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c160180 con 0x7fa37c0d17a0 2024-04-06T13:49:21.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.046+0000 7fa38175d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa37c146dd0 0x7fa37c15ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:21.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.046+0000 7fa38175d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa37c146dd0 0x7fa37c15ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46208/0 (socket says 172.21.15.189:46208) 2024-04-06T13:49:21.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.046+0000 7fa38175d700 1 -- 172.21.15.189:0/888882572 learned_addr learned my addr 172.21.15.189:0/888882572 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:21.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.047+0000 7fa38175d700 1 -- 172.21.15.189:0/888882572 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa37c0d17a0 msgr2=0x7fa37c0b4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:21.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.047+0000 7fa38175d700 1 --2- 172.21.15.189:0/888882572 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa37c0d17a0 0x7fa37c0b4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.047+0000 7fa38175d700 1 -- 172.21.15.189:0/888882572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa37c0cedf0 msgr2=0x7fa37c15fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:21.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.047+0000 7fa38175d700 1 --2- 172.21.15.189:0/888882572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa37c0cedf0 0x7fa37c15fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.047+0000 7fa38175d700 1 -- 172.21.15.189:0/888882572 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa374009a70 con 0x7fa37c146dd0 2024-04-06T13:49:21.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.047+0000 7fa38175d700 1 --2- 172.21.15.189:0/888882572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa37c146dd0 0x7fa37c15ade0 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7fa37c0677e0 tx=0x7fa374004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:21.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.047+0000 7fa3797fa700 1 -- 172.21.15.189:0/888882572 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa374012840 con 0x7fa37c146dd0 2024-04-06T13:49:21.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.047+0000 7fa3797fa700 1 -- 172.21.15.189:0/888882572 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3740129c0 con 0x7fa37c146dd0 2024-04-06T13:49:21.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.047+0000 7fa3831c0700 1 -- 172.21.15.189:0/888882572 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa37c160460 con 0x7fa37c146dd0 2024-04-06T13:49:21.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.047+0000 7fa3797fa700 1 -- 172.21.15.189:0/888882572 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa374012d70 con 0x7fa37c146dd0 2024-04-06T13:49:21.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.047+0000 7fa3831c0700 1 -- 172.21.15.189:0/888882572 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa37c160a20 con 0x7fa37c146dd0 2024-04-06T13:49:21.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.048+0000 7fa3831c0700 1 -- 172.21.15.189:0/888882572 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa37c146790 con 0x7fa37c146dd0 2024-04-06T13:49:21.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.048+0000 7fa3797fa700 1 -- 172.21.15.189:0/888882572 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fa37401d420 con 0x7fa37c146dd0 2024-04-06T13:49:21.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.049+0000 7fa3797fa700 1 --2- 172.21.15.189:0/888882572 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa36006aed0 0x7fa36006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:21.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.049+0000 7fa3797fa700 1 -- 172.21.15.189:0/888882572 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa37408b5b0 con 0x7fa37c146dd0 2024-04-06T13:49:21.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.049+0000 7fa380f5c700 1 --2- 172.21.15.189:0/888882572 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa36006aed0 0x7fa36006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:21.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.049+0000 7fa380f5c700 1 --2- 172.21.15.189:0/888882572 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa36006aed0 0x7fa36006d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa3700096f0 tx=0x7fa370006c50 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:21.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.053+0000 7fa3797fa700 1 -- 172.21.15.189:0/888882572 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa374056220 con 0x7fa37c146dd0 2024-04-06T13:49:21.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.249+0000 7fa3831c0700 1 -- 172.21.15.189:0/888882572 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa37c074990 con 0x7fa37c146dd0 2024-04-06T13:49:21.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.250+0000 7fa3797fa700 1 -- 172.21.15.189:0/888882572 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v269) v1 ==== 74+0+16416 (secure 0 0 0) 0x7fa374059840 con 0x7fa37c146dd0 2024-04-06T13:49:21.250 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:21.250 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":269,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:49:18.494935+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:49:15.919092+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"268","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-04-06T13:49:16.459885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"269","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:49:21.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.252+0000 7fa3831c0700 1 -- 172.21.15.189:0/888882572 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa36006aed0 msgr2=0x7fa36006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:21.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.252+0000 7fa3831c0700 1 --2- 172.21.15.189:0/888882572 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa36006aed0 0x7fa36006d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa3700096f0 tx=0x7fa370006c50 comp rx=0 tx=0).stop 2024-04-06T13:49:21.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.252+0000 7fa3831c0700 1 -- 172.21.15.189:0/888882572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa37c146dd0 msgr2=0x7fa37c15ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:21.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.252+0000 7fa3831c0700 1 --2- 172.21.15.189:0/888882572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa37c146dd0 0x7fa37c15ade0 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7fa37c0677e0 tx=0x7fa374004ad0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.252+0000 7fa3831c0700 1 -- 172.21.15.189:0/888882572 shutdown_connections 2024-04-06T13:49:21.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.252+0000 7fa3831c0700 1 --2- 172.21.15.189:0/888882572 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa36006aed0 0x7fa36006d390 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.252+0000 7fa3831c0700 1 --2- 172.21.15.189:0/888882572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa37c146dd0 0x7fa37c15ade0 unknown :-1 s=CLOSED pgs=2934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.252+0000 7fa3831c0700 1 --2- 172.21.15.189:0/888882572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa37c0cedf0 0x7fa37c15fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.252+0000 7fa3831c0700 1 --2- 172.21.15.189:0/888882572 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa37c0d17a0 0x7fa37c0b4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.252+0000 7fa3831c0700 1 -- 172.21.15.189:0/888882572 >> 172.21.15.189:0/888882572 conn(0x7fa37c0b2db0 msgr2=0x7fa37c06d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:21.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.252+0000 7fa3831c0700 1 -- 172.21.15.189:0/888882572 shutdown_connections 2024-04-06T13:49:21.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.252+0000 7fa3831c0700 1 -- 172.21.15.189:0/888882572 wait complete. 2024-04-06T13:49:21.264 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-06T13:49:21.265 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:49:21.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.387+0000 7fac38060700 1 Processor -- start 2024-04-06T13:49:21.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.387+0000 7fac38060700 1 -- start start 2024-04-06T13:49:21.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.388+0000 7fac38060700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fac3015d7e0 0x7fac300d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:21.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.388+0000 7fac38060700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac300cdf80 con 0x7fac300d1cc0 2024-04-06T13:49:21.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.388+0000 7fac38060700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac300d0cf0 con 0x7fac300d0890 2024-04-06T13:49:21.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.388+0000 7fac38060700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac300d3230 con 0x7fac3015d7e0 2024-04-06T13:49:21.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.388+0000 7fac35dfc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fac3015d7e0 0x7fac300d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:21.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.388+0000 7fac365fd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fac300d1cc0 0x7fac300b66b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40046/0 (socket says 172.21.15.189:40046) 2024-04-06T13:49:21.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.388+0000 7fac365fd700 1 -- 172.21.15.189:0/12374455 learned_addr learned my addr 172.21.15.189:0/12374455 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:21.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.388+0000 7fac35dfc700 1 -- 172.21.15.189:0/12374455 >> v1:172.21.15.192:6789/0 conn(0x7fac300d0890 legacy=0x7fac300d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:21.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.388+0000 7fac35dfc700 1 -- 172.21.15.189:0/12374455 >> v1:172.21.15.189:6789/0 conn(0x7fac300d1cc0 legacy=0x7fac300b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:21.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.388+0000 7fac35dfc700 1 -- 172.21.15.189:0/12374455 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac300d2da0 con 0x7fac3015d7e0 2024-04-06T13:49:21.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.388+0000 7fac35dfc700 1 --2- 172.21.15.189:0/12374455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac3015d7e0 0x7fac300d0350 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7fac2c009840 tx=0x7fac2c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31a9b34c32075c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:21.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.388+0000 7fac34dfa700 1 -- 172.21.15.189:0/12374455 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c004230 con 0x7fac3015d7e0 2024-04-06T13:49:21.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.388+0000 7fac34dfa700 1 -- 172.21.15.189:0/12374455 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2c004390 con 0x7fac3015d7e0 2024-04-06T13:49:21.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.388+0000 7fac34dfa700 1 -- 172.21.15.189:0/12374455 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0036a0 con 0x7fac3015d7e0 2024-04-06T13:49:21.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.389+0000 7fac38060700 1 -- 172.21.15.189:0/12374455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac3015d7e0 msgr2=0x7fac300d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:21.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.389+0000 7fac38060700 1 --2- 172.21.15.189:0/12374455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac3015d7e0 0x7fac300d0350 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7fac2c009840 tx=0x7fac2c009b50 comp rx=0 tx=0).stop 2024-04-06T13:49:21.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.389+0000 7fac38060700 1 -- 172.21.15.189:0/12374455 shutdown_connections 2024-04-06T13:49:21.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.389+0000 7fac38060700 1 --2- 172.21.15.189:0/12374455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac3015d7e0 0x7fac300d0350 unknown :-1 s=CLOSED pgs=2936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.389+0000 7fac38060700 1 -- 172.21.15.189:0/12374455 >> 172.21.15.189:0/12374455 conn(0x7fac30072250 msgr2=0x7fac30072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:21.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.389+0000 7fac38060700 1 -- 172.21.15.189:0/12374455 shutdown_connections 2024-04-06T13:49:21.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.389+0000 7fac38060700 1 -- 172.21.15.189:0/12374455 wait complete. 2024-04-06T13:49:21.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.390+0000 7fac38060700 1 Processor -- start 2024-04-06T13:49:21.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.390+0000 7fac38060700 1 -- start start 2024-04-06T13:49:21.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.391+0000 7fac38060700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac300d0890 0x7fac3014bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:21.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.391+0000 7fac38060700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac300d1cc0 0x7fac300c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:21.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.391+0000 7fac38060700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac3015d7e0 0x7fac300c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:21.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.391+0000 7fac38060700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3014cdf0 con 0x7fac300d1cc0 2024-04-06T13:49:21.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.391+0000 7fac38060700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3014c450 con 0x7fac3015d7e0 2024-04-06T13:49:21.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.391+0000 7fac38060700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac300b5d00 con 0x7fac300d0890 2024-04-06T13:49:21.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.391+0000 7fac355fb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac300d1cc0 0x7fac300c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:21.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.391+0000 7fac355fb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac300d1cc0 0x7fac300c6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46226/0 (socket says 172.21.15.189:46226) 2024-04-06T13:49:21.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.391+0000 7fac355fb700 1 -- 172.21.15.189:0/1172638386 learned_addr learned my addr 172.21.15.189:0/1172638386 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:21.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.391+0000 7fac355fb700 1 -- 172.21.15.189:0/1172638386 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac300d0890 msgr2=0x7fac3014bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:21.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.391+0000 7fac355fb700 1 --2- 172.21.15.189:0/1172638386 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac300d0890 0x7fac3014bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.391+0000 7fac355fb700 1 -- 172.21.15.189:0/1172638386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac3015d7e0 msgr2=0x7fac300c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:21.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.391+0000 7fac355fb700 1 --2- 172.21.15.189:0/1172638386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac3015d7e0 0x7fac300c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.391+0000 7fac355fb700 1 -- 172.21.15.189:0/1172638386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2c0094a0 con 0x7fac300d1cc0 2024-04-06T13:49:21.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.392+0000 7fac355fb700 1 --2- 172.21.15.189:0/1172638386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac300d1cc0 0x7fac300c6fd0 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7fac2400b4a0 tx=0x7fac2400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:21.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.392+0000 7fac227fc700 1 -- 172.21.15.189:0/1172638386 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac24011830 con 0x7fac300d1cc0 2024-04-06T13:49:21.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.392+0000 7fac227fc700 1 -- 172.21.15.189:0/1172638386 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac240119b0 con 0x7fac300d1cc0 2024-04-06T13:49:21.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.392+0000 7fac227fc700 1 -- 172.21.15.189:0/1172638386 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac24011d60 con 0x7fac300d1cc0 2024-04-06T13:49:21.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.392+0000 7fac38060700 1 -- 172.21.15.189:0/1172638386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac300b5a20 con 0x7fac300d1cc0 2024-04-06T13:49:21.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.392+0000 7fac38060700 1 -- 172.21.15.189:0/1172638386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac3014ca80 con 0x7fac300d1cc0 2024-04-06T13:49:21.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.393+0000 7fac38060700 1 -- 172.21.15.189:0/1172638386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac3014cc40 con 0x7fac300d1cc0 2024-04-06T13:49:21.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.393+0000 7fac227fc700 1 -- 172.21.15.189:0/1172638386 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fac2401a400 con 0x7fac300d1cc0 2024-04-06T13:49:21.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.394+0000 7fac227fc700 1 --2- 172.21.15.189:0/1172638386 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fac1406aed0 0x7fac1406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:21.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.394+0000 7fac227fc700 1 -- 172.21.15.189:0/1172638386 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fac240898f0 con 0x7fac300d1cc0 2024-04-06T13:49:21.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.394+0000 7fac35dfc700 1 --2- 172.21.15.189:0/1172638386 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fac1406aed0 0x7fac1406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:21.395 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.394+0000 7fac35dfc700 1 --2- 172.21.15.189:0/1172638386 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fac1406aed0 0x7fac1406d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fac2c000c00 tx=0x7fac2c010000 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:21.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.398+0000 7fac227fc700 1 -- 172.21.15.189:0/1172638386 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fac240545e0 con 0x7fac300d1cc0 2024-04-06T13:49:21.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.589+0000 7fac38060700 1 -- 172.21.15.189:0/1172638386 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fac300d2470 con 0x7fac1406aed0 2024-04-06T13:49:21.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.598+0000 7fac227fc700 1 -- 172.21.15.189:0/1172638386 <== mgr.12522 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289886 (secure 0 0 0) 0x7fac2c01e070 con 0x7fac1406aed0 2024-04-06T13:49:21.601 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:21.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.601+0000 7fac38060700 1 -- 172.21.15.189:0/1172638386 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fac1406aed0 msgr2=0x7fac1406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:21.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.601+0000 7fac38060700 1 --2- 172.21.15.189:0/1172638386 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fac1406aed0 0x7fac1406d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fac2c000c00 tx=0x7fac2c010000 comp rx=0 tx=0).stop 2024-04-06T13:49:21.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.601+0000 7fac38060700 1 -- 172.21.15.189:0/1172638386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac300d1cc0 msgr2=0x7fac300c6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:21.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.601+0000 7fac38060700 1 --2- 172.21.15.189:0/1172638386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac300d1cc0 0x7fac300c6fd0 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7fac2400b4a0 tx=0x7fac2400b860 comp rx=0 tx=0).stop 2024-04-06T13:49:21.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.601+0000 7fac38060700 1 -- 172.21.15.189:0/1172638386 shutdown_connections 2024-04-06T13:49:21.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.601+0000 7fac38060700 1 --2- 172.21.15.189:0/1172638386 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fac1406aed0 0x7fac1406d390 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.601+0000 7fac38060700 1 --2- 172.21.15.189:0/1172638386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac3015d7e0 0x7fac300c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.601+0000 7fac38060700 1 --2- 172.21.15.189:0/1172638386 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac300d0890 0x7fac3014bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.601+0000 7fac38060700 1 --2- 172.21.15.189:0/1172638386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac300d1cc0 0x7fac300c6fd0 unknown :-1 s=CLOSED pgs=2937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.601+0000 7fac38060700 1 -- 172.21.15.189:0/1172638386 >> 172.21.15.189:0/1172638386 conn(0x7fac30072250 msgr2=0x7fac3015f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:21.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.601+0000 7fac38060700 1 -- 172.21.15.189:0/1172638386 shutdown_connections 2024-04-06T13:49:21.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.601+0000 7fac38060700 1 -- 172.21.15.189:0/1172638386 wait complete. 2024-04-06T13:49:21.605 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:49:21.613 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-04-06T13:49:20.756983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":683,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":419,"ondisk_log_size":419,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":463420,"kb_used_data":27464,"kb_used_omap":12,"kb_used_meta":435827,"kb_avail":754511300,"statfs":{"total":773094113280,"available":772619571200,"internally_reserved":0,"allocated":28123136,"data_stored":15077379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":446287416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.677624"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.520614+0000","last_change":"2024-04-06T13:49:17.516210+0000","last_active":"2024-04-06T13:49:17.520614+0000","last_peered":"2024-04-06T13:49:17.520614+0000","last_clean":"2024-04-06T13:49:17.520614+0000","last_became_active":"2024-04-06T13:49:17.515918+0000","last_became_peered":"2024-04-06T13:49:17.515918+0000","last_unstale":"2024-04-06T13:49:17.520614+0000","last_undegraded":"2024-04-06T13:49:17.520614+0000","last_fullsized":"2024-04-06T13:49:17.520614+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:36:00.408855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522243+0000","last_change":"2024-04-06T13:49:17.519084+0000","last_active":"2024-04-06T13:49:17.522243+0000","last_peered":"2024-04-06T13:49:17.522243+0000","last_clean":"2024-04-06T13:49:17.522243+0000","last_became_active":"2024-04-06T13:49:17.518820+0000","last_became_peered":"2024-04-06T13:49:17.518820+0000","last_unstale":"2024-04-06T13:49:17.522243+0000","last_undegraded":"2024-04-06T13:49:17.522243+0000","last_fullsized":"2024-04-06T13:49:17.522243+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:05:58.697947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.513364+0000","last_change":"2024-04-06T13:49:17.510525+0000","last_active":"2024-04-06T13:49:17.513364+0000","last_peered":"2024-04-06T13:49:17.513364+0000","last_clean":"2024-04-06T13:49:17.513364+0000","last_became_active":"2024-04-06T13:49:17.510094+0000","last_became_peered":"2024-04-06T13:49:17.510094+0000","last_unstale":"2024-04-06T13:49:17.513364+0000","last_undegraded":"2024-04-06T13:49:17.513364+0000","last_fullsized":"2024-04-06T13:49:17.513364+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:52:44.549832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.524977+0000","last_change":"2024-04-06T13:49:17.524610+0000","last_active":"2024-04-06T13:49:17.524977+0000","last_peered":"2024-04-06T13:49:17.524977+0000","last_clean":"2024-04-06T13:49:17.524977+0000","last_became_active":"2024-04-06T13:49:17.524396+0000","last_became_peered":"2024-04-06T13:49:17.524396+0000","last_unstale":"2024-04-06T13:49:17.524977+0000","last_undegraded":"2024-04-06T13:49:17.524977+0000","last_fullsized":"2024-04-06T13:49:17.524977+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:32:23.301371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514489+0000","last_change":"2024-04-06T13:49:17.508190+0000","last_active":"2024-04-06T13:49:17.514489+0000","last_peered":"2024-04-06T13:49:17.514489+0000","last_clean":"2024-04-06T13:49:17.514489+0000","last_became_active":"2024-04-06T13:49:17.507959+0000","last_became_peered":"2024-04-06T13:49:17.507959+0000","last_unstale":"2024-04-06T13:49:17.514489+0000","last_undegraded":"2024-04-06T13:49:17.514489+0000","last_fullsized":"2024-04-06T13:49:17.514489+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:40:45.826207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514078+0000","last_change":"2024-04-06T13:49:17.507071+0000","last_active":"2024-04-06T13:49:17.514078+0000","last_peered":"2024-04-06T13:49:17.514078+0000","last_clean":"2024-04-06T13:49:17.514078+0000","last_became_active":"2024-04-06T13:49:17.506737+0000","last_became_peered":"2024-04-06T13:49:17.506737+0000","last_unstale":"2024-04-06T13:49:17.514078+0000","last_undegraded":"2024-04-06T13:49:17.514078+0000","last_fullsized":"2024-04-06T13:49:17.514078+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:53:28.014667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519186+0000","last_change":"2024-04-06T13:49:17.515069+0000","last_active":"2024-04-06T13:49:17.519186+0000","last_peered":"2024-04-06T13:49:17.519186+0000","last_clean":"2024-04-06T13:49:17.519186+0000","last_became_active":"2024-04-06T13:49:17.506210+0000","last_became_peered":"2024-04-06T13:49:17.506210+0000","last_unstale":"2024-04-06T13:49:17.519186+0000","last_undegraded":"2024-04-06T13:49:17.519186+0000","last_fullsized":"2024-04-06T13:49:17.519186+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:14:21.286281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.521054+0000","last_change":"2024-04-06T13:49:17.520637+0000","last_active":"2024-04-06T13:49:17.521054+0000","last_peered":"2024-04-06T13:49:17.521054+0000","last_clean":"2024-04-06T13:49:17.521054+0000","last_became_active":"2024-04-06T13:49:17.520388+0000","last_became_peered":"2024-04-06T13:49:17.520388+0000","last_unstale":"2024-04-06T13:49:17.521054+0000","last_undegraded":"2024-04-06T13:49:17.521054+0000","last_fullsized":"2024-04-06T13:49:17.521054+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:12.068312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.516486+0000","last_change":"2024-04-06T13:49:17.506537+0000","last_active":"2024-04-06T13:49:17.516486+0000","last_peered":"2024-04-06T13:49:17.516486+0000","last_clean":"2024-04-06T13:49:17.516486+0000","last_became_active":"2024-04-06T13:49:17.506232+0000","last_became_peered":"2024-04-06T13:49:17.506232+0000","last_unstale":"2024-04-06T13:49:17.516486+0000","last_undegraded":"2024-04-06T13:49:17.516486+0000","last_fullsized":"2024-04-06T13:49:17.516486+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:02:05.602108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.520190+0000","last_change":"2024-04-06T13:49:17.517199+0000","last_active":"2024-04-06T13:49:17.520190+0000","last_peered":"2024-04-06T13:49:17.520190+0000","last_clean":"2024-04-06T13:49:17.520190+0000","last_became_active":"2024-04-06T13:49:17.516867+0000","last_became_peered":"2024-04-06T13:49:17.516867+0000","last_unstale":"2024-04-06T13:49:17.520190+0000","last_undegraded":"2024-04-06T13:49:17.520190+0000","last_fullsized":"2024-04-06T13:49:17.520190+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:10:03.724857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.517970+0000","last_change":"2024-04-06T13:49:17.515732+0000","last_active":"2024-04-06T13:49:17.517970+0000","last_peered":"2024-04-06T13:49:17.517970+0000","last_clean":"2024-04-06T13:49:17.517970+0000","last_became_active":"2024-04-06T13:49:17.515484+0000","last_became_peered":"2024-04-06T13:49:17.515484+0000","last_unstale":"2024-04-06T13:49:17.517970+0000","last_undegraded":"2024-04-06T13:49:17.517970+0000","last_fullsized":"2024-04-06T13:49:17.517970+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:49:29.329764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522789+0000","last_change":"2024-04-06T13:49:17.521398+0000","last_active":"2024-04-06T13:49:17.522789+0000","last_peered":"2024-04-06T13:49:17.522789+0000","last_clean":"2024-04-06T13:49:17.522789+0000","last_became_active":"2024-04-06T13:49:17.521122+0000","last_became_peered":"2024-04-06T13:49:17.521122+0000","last_unstale":"2024-04-06T13:49:17.522789+0000","last_undegraded":"2024-04-06T13:49:17.522789+0000","last_fullsized":"2024-04-06T13:49:17.522789+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:12.481758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523016+0000","last_change":"2024-04-06T13:49:17.521408+0000","last_active":"2024-04-06T13:49:17.523016+0000","last_peered":"2024-04-06T13:49:17.523016+0000","last_clean":"2024-04-06T13:49:17.523016+0000","last_became_active":"2024-04-06T13:49:17.521217+0000","last_became_peered":"2024-04-06T13:49:17.521217+0000","last_unstale":"2024-04-06T13:49:17.523016+0000","last_undegraded":"2024-04-06T13:49:17.523016+0000","last_fullsized":"2024-04-06T13:49:17.523016+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:18:13.304903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.524375+0000","last_change":"2024-04-06T13:49:17.518243+0000","last_active":"2024-04-06T13:49:17.524375+0000","last_peered":"2024-04-06T13:49:17.524375+0000","last_clean":"2024-04-06T13:49:17.524375+0000","last_became_active":"2024-04-06T13:49:17.516954+0000","last_became_peered":"2024-04-06T13:49:17.516954+0000","last_unstale":"2024-04-06T13:49:17.524375+0000","last_undegraded":"2024-04-06T13:49:17.524375+0000","last_fullsized":"2024-04-06T13:49:17.524375+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:36:20.518005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.517577+0000","last_change":"2024-04-06T13:49:17.512021+0000","last_active":"2024-04-06T13:49:17.517577+0000","last_peered":"2024-04-06T13:49:17.517577+0000","last_clean":"2024-04-06T13:49:17.517577+0000","last_became_active":"2024-04-06T13:49:17.511763+0000","last_became_peered":"2024-04-06T13:49:17.511763+0000","last_unstale":"2024-04-06T13:49:17.517577+0000","last_undegraded":"2024-04-06T13:49:17.517577+0000","last_fullsized":"2024-04-06T13:49:17.517577+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:12:49.230301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.512003+0000","last_change":"2024-04-06T13:49:17.508744+0000","last_active":"2024-04-06T13:49:17.512003+0000","last_peered":"2024-04-06T13:49:17.512003+0000","last_clean":"2024-04-06T13:49:17.512003+0000","last_became_active":"2024-04-06T13:49:17.508450+0000","last_became_peered":"2024-04-06T13:49:17.508450+0000","last_unstale":"2024-04-06T13:49:17.512003+0000","last_undegraded":"2024-04-06T13:49:17.512003+0000","last_fullsized":"2024-04-06T13:49:17.512003+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:12.068312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519816+0000","last_change":"2024-04-06T13:49:17.517174+0000","last_active":"2024-04-06T13:49:17.519816+0000","last_peered":"2024-04-06T13:49:17.519816+0000","last_clean":"2024-04-06T13:49:17.519816+0000","last_became_active":"2024-04-06T13:49:17.516215+0000","last_became_peered":"2024-04-06T13:49:17.516215+0000","last_unstale":"2024-04-06T13:49:17.519816+0000","last_undegraded":"2024-04-06T13:49:17.519816+0000","last_fullsized":"2024-04-06T13:49:17.519816+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:07.368454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.517476+0000","last_change":"2024-04-06T13:49:17.511953+0000","last_active":"2024-04-06T13:49:17.517476+0000","last_peered":"2024-04-06T13:49:17.517476+0000","last_clean":"2024-04-06T13:49:17.517476+0000","last_became_active":"2024-04-06T13:49:17.511661+0000","last_became_peered":"2024-04-06T13:49:17.511661+0000","last_unstale":"2024-04-06T13:49:17.517476+0000","last_undegraded":"2024-04-06T13:49:17.517476+0000","last_fullsized":"2024-04-06T13:49:17.517476+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:26:28.142149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522145+0000","last_change":"2024-04-06T13:49:17.521073+0000","last_active":"2024-04-06T13:49:17.522145+0000","last_peered":"2024-04-06T13:49:17.522145+0000","last_clean":"2024-04-06T13:49:17.522145+0000","last_became_active":"2024-04-06T13:49:17.520811+0000","last_became_peered":"2024-04-06T13:49:17.520811+0000","last_unstale":"2024-04-06T13:49:17.522145+0000","last_undegraded":"2024-04-06T13:49:17.522145+0000","last_fullsized":"2024-04-06T13:49:17.522145+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:17:06.173348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.510215+0000","last_change":"2024-04-06T13:49:17.507429+0000","last_active":"2024-04-06T13:49:17.510215+0000","last_peered":"2024-04-06T13:49:17.510215+0000","last_clean":"2024-04-06T13:49:17.510215+0000","last_became_active":"2024-04-06T13:49:17.507181+0000","last_became_peered":"2024-04-06T13:49:17.507181+0000","last_unstale":"2024-04-06T13:49:17.510215+0000","last_undegraded":"2024-04-06T13:49:17.510215+0000","last_fullsized":"2024-04-06T13:49:17.510215+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:32:23.301371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514185+0000","last_change":"2024-04-06T13:49:17.513611+0000","last_active":"2024-04-06T13:49:17.514185+0000","last_peered":"2024-04-06T13:49:17.514185+0000","last_clean":"2024-04-06T13:49:17.514185+0000","last_became_active":"2024-04-06T13:49:17.509723+0000","last_became_peered":"2024-04-06T13:49:17.509723+0000","last_unstale":"2024-04-06T13:49:17.514185+0000","last_undegraded":"2024-04-06T13:49:17.514185+0000","last_fullsized":"2024-04-06T13:49:17.514185+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:55:51.420328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.515396+0000","last_change":"2024-04-06T13:49:17.512230+0000","last_active":"2024-04-06T13:49:17.515396+0000","last_peered":"2024-04-06T13:49:17.515396+0000","last_clean":"2024-04-06T13:49:17.515396+0000","last_became_active":"2024-04-06T13:49:17.511986+0000","last_became_peered":"2024-04-06T13:49:17.511986+0000","last_unstale":"2024-04-06T13:49:17.515396+0000","last_undegraded":"2024-04-06T13:49:17.515396+0000","last_fullsized":"2024-04-06T13:49:17.515396+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:27:37.281369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.511808+0000","last_change":"2024-04-06T13:49:17.501401+0000","last_active":"2024-04-06T13:49:17.511808+0000","last_peered":"2024-04-06T13:49:17.511808+0000","last_clean":"2024-04-06T13:49:17.511808+0000","last_became_active":"2024-04-06T13:49:17.501139+0000","last_became_peered":"2024-04-06T13:49:17.501139+0000","last_unstale":"2024-04-06T13:49:17.511808+0000","last_undegraded":"2024-04-06T13:49:17.511808+0000","last_fullsized":"2024-04-06T13:49:17.511808+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:59:09.454807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523385+0000","last_change":"2024-04-06T13:49:17.522743+0000","last_active":"2024-04-06T13:49:17.523385+0000","last_peered":"2024-04-06T13:49:17.523385+0000","last_clean":"2024-04-06T13:49:17.523385+0000","last_became_active":"2024-04-06T13:49:17.522434+0000","last_became_peered":"2024-04-06T13:49:17.522434+0000","last_unstale":"2024-04-06T13:49:17.523385+0000","last_undegraded":"2024-04-06T13:49:17.523385+0000","last_fullsized":"2024-04-06T13:49:17.523385+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:36:30.017284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.520593+0000","last_change":"2024-04-06T13:49:17.512281+0000","last_active":"2024-04-06T13:49:17.520593+0000","last_peered":"2024-04-06T13:49:17.520593+0000","last_clean":"2024-04-06T13:49:17.520593+0000","last_became_active":"2024-04-06T13:49:17.511719+0000","last_became_peered":"2024-04-06T13:49:17.511719+0000","last_unstale":"2024-04-06T13:49:17.520593+0000","last_undegraded":"2024-04-06T13:49:17.520593+0000","last_fullsized":"2024-04-06T13:49:17.520593+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:27:32.583709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514836+0000","last_change":"2024-04-06T13:49:17.513918+0000","last_active":"2024-04-06T13:49:17.514836+0000","last_peered":"2024-04-06T13:49:17.514836+0000","last_clean":"2024-04-06T13:49:17.514836+0000","last_became_active":"2024-04-06T13:49:17.513687+0000","last_became_peered":"2024-04-06T13:49:17.513687+0000","last_unstale":"2024-04-06T13:49:17.514836+0000","last_undegraded":"2024-04-06T13:49:17.514836+0000","last_fullsized":"2024-04-06T13:49:17.514836+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:14:21.286281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514395+0000","last_change":"2024-04-06T13:49:17.507416+0000","last_active":"2024-04-06T13:49:17.514395+0000","last_peered":"2024-04-06T13:49:17.514395+0000","last_clean":"2024-04-06T13:49:17.514395+0000","last_became_active":"2024-04-06T13:49:17.507112+0000","last_became_peered":"2024-04-06T13:49:17.507112+0000","last_unstale":"2024-04-06T13:49:17.514395+0000","last_undegraded":"2024-04-06T13:49:17.514395+0000","last_fullsized":"2024-04-06T13:49:17.514395+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:05:36.115690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.510969+0000","last_change":"2024-04-06T13:49:17.506454+0000","last_active":"2024-04-06T13:49:17.510969+0000","last_peered":"2024-04-06T13:49:17.510969+0000","last_clean":"2024-04-06T13:49:17.510969+0000","last_became_active":"2024-04-06T13:49:17.505920+0000","last_became_peered":"2024-04-06T13:49:17.505920+0000","last_unstale":"2024-04-06T13:49:17.510969+0000","last_undegraded":"2024-04-06T13:49:17.510969+0000","last_fullsized":"2024-04-06T13:49:17.510969+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:51:06.786292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.521699+0000","last_change":"2024-04-06T13:49:17.514816+0000","last_active":"2024-04-06T13:49:17.521699+0000","last_peered":"2024-04-06T13:49:17.521699+0000","last_clean":"2024-04-06T13:49:17.521699+0000","last_became_active":"2024-04-06T13:49:17.514548+0000","last_became_peered":"2024-04-06T13:49:17.514548+0000","last_unstale":"2024-04-06T13:49:17.521699+0000","last_undegraded":"2024-04-06T13:49:17.521699+0000","last_fullsized":"2024-04-06T13:49:17.521699+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:44:33.017556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.510940+0000","last_change":"2024-04-06T13:49:17.506195+0000","last_active":"2024-04-06T13:49:17.510940+0000","last_peered":"2024-04-06T13:49:17.510940+0000","last_clean":"2024-04-06T13:49:17.510940+0000","last_became_active":"2024-04-06T13:49:17.505887+0000","last_became_peered":"2024-04-06T13:49:17.505887+0000","last_unstale":"2024-04-06T13:49:17.510940+0000","last_undegraded":"2024-04-06T13:49:17.510940+0000","last_fullsized":"2024-04-06T13:49:17.510940+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:16:38.686659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519769+0000","last_change":"2024-04-06T13:49:17.515234+0000","last_active":"2024-04-06T13:49:17.519769+0000","last_peered":"2024-04-06T13:49:17.519769+0000","last_clean":"2024-04-06T13:49:17.519769+0000","last_became_active":"2024-04-06T13:49:17.514959+0000","last_became_peered":"2024-04-06T13:49:17.514959+0000","last_unstale":"2024-04-06T13:49:17.519769+0000","last_undegraded":"2024-04-06T13:49:17.519769+0000","last_fullsized":"2024-04-06T13:49:17.519769+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:02:31.039549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523372+0000","last_change":"2024-04-06T13:49:17.522490+0000","last_active":"2024-04-06T13:49:17.523372+0000","last_peered":"2024-04-06T13:49:17.523372+0000","last_clean":"2024-04-06T13:49:17.523372+0000","last_became_active":"2024-04-06T13:49:17.522102+0000","last_became_peered":"2024-04-06T13:49:17.522102+0000","last_unstale":"2024-04-06T13:49:17.523372+0000","last_undegraded":"2024-04-06T13:49:17.523372+0000","last_fullsized":"2024-04-06T13:49:17.523372+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:12:09.850584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"268'4","reported_seq":14,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523550+0000","last_change":"2024-04-06T13:49:17.096966+0000","last_active":"2024-04-06T13:49:17.523550+0000","last_peered":"2024-04-06T13:49:17.523550+0000","last_clean":"2024-04-06T13:49:17.523550+0000","last_became_active":"2024-04-06T13:49:17.096695+0000","last_became_peered":"2024-04-06T13:49:17.096695+0000","last_unstale":"2024-04-06T13:49:17.523550+0000","last_undegraded":"2024-04-06T13:49:17.523550+0000","last_fullsized":"2024-04-06T13:49:17.523550+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:28:50.074433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.525127+0000","last_change":"2024-04-06T13:49:17.523240+0000","last_active":"2024-04-06T13:49:17.525127+0000","last_peered":"2024-04-06T13:49:17.525127+0000","last_clean":"2024-04-06T13:49:17.525127+0000","last_became_active":"2024-04-06T13:49:17.522980+0000","last_became_peered":"2024-04-06T13:49:17.522980+0000","last_unstale":"2024-04-06T13:49:17.525127+0000","last_undegraded":"2024-04-06T13:49:17.525127+0000","last_fullsized":"2024-04-06T13:49:17.525127+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:38:33.450798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"268'1","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522994+0000","last_change":"2024-04-06T13:49:17.098551+0000","last_active":"2024-04-06T13:49:17.522994+0000","last_peered":"2024-04-06T13:49:17.522994+0000","last_clean":"2024-04-06T13:49:17.522994+0000","last_became_active":"2024-04-06T13:49:17.098354+0000","last_became_peered":"2024-04-06T13:49:17.098354+0000","last_unstale":"2024-04-06T13:49:17.522994+0000","last_undegraded":"2024-04-06T13:49:17.522994+0000","last_fullsized":"2024-04-06T13:49:17.522994+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:43:16.510351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522212+0000","last_change":"2024-04-06T13:49:17.518725+0000","last_active":"2024-04-06T13:49:17.522212+0000","last_peered":"2024-04-06T13:49:17.522212+0000","last_clean":"2024-04-06T13:49:17.522212+0000","last_became_active":"2024-04-06T13:49:17.518434+0000","last_became_peered":"2024-04-06T13:49:17.518434+0000","last_unstale":"2024-04-06T13:49:17.522212+0000","last_undegraded":"2024-04-06T13:49:17.522212+0000","last_fullsized":"2024-04-06T13:49:17.522212+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:25:59.367813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1d","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522514+0000","last_change":"2024-04-06T13:49:17.104889+0000","last_active":"2024-04-06T13:49:17.522514+0000","last_peered":"2024-04-06T13:49:17.522514+0000","last_clean":"2024-04-06T13:49:17.522514+0000","last_became_active":"2024-04-06T13:49:17.104619+0000","last_became_peered":"2024-04-06T13:49:17.104619+0000","last_unstale":"2024-04-06T13:49:17.522514+0000","last_undegraded":"2024-04-06T13:49:17.522514+0000","last_fullsized":"2024-04-06T13:49:17.522514+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:09:00.021457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.515331+0000","last_change":"2024-04-06T13:49:17.503291+0000","last_active":"2024-04-06T13:49:17.515331+0000","last_peered":"2024-04-06T13:49:17.515331+0000","last_clean":"2024-04-06T13:49:17.515331+0000","last_became_active":"2024-04-06T13:49:17.503011+0000","last_became_peered":"2024-04-06T13:49:17.503011+0000","last_unstale":"2024-04-06T13:49:17.515331+0000","last_undegraded":"2024-04-06T13:49:17.515331+0000","last_fullsized":"2024-04-06T13:49:17.515331+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:09.947236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522373+0000","last_change":"2024-04-06T13:49:17.105027+0000","last_active":"2024-04-06T13:49:17.522373+0000","last_peered":"2024-04-06T13:49:17.522373+0000","last_clean":"2024-04-06T13:49:17.522373+0000","last_became_active":"2024-04-06T13:49:17.104656+0000","last_became_peered":"2024-04-06T13:49:17.104656+0000","last_unstale":"2024-04-06T13:49:17.522373+0000","last_undegraded":"2024-04-06T13:49:17.522373+0000","last_fullsized":"2024-04-06T13:49:17.522373+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:15:00.333710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.510882+0000","last_change":"2024-04-06T13:49:17.510173+0000","last_active":"2024-04-06T13:49:17.510882+0000","last_peered":"2024-04-06T13:49:17.510882+0000","last_clean":"2024-04-06T13:49:17.510882+0000","last_became_active":"2024-04-06T13:49:17.509947+0000","last_became_peered":"2024-04-06T13:49:17.509947+0000","last_unstale":"2024-04-06T13:49:17.510882+0000","last_undegraded":"2024-04-06T13:49:17.510882+0000","last_fullsized":"2024-04-06T13:49:17.510882+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:59:33.036233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1b","version":"268'1","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.520763+0000","last_change":"2024-04-06T13:49:17.097606+0000","last_active":"2024-04-06T13:49:17.520763+0000","last_peered":"2024-04-06T13:49:17.520763+0000","last_clean":"2024-04-06T13:49:17.520763+0000","last_became_active":"2024-04-06T13:49:17.097312+0000","last_became_peered":"2024-04-06T13:49:17.097312+0000","last_unstale":"2024-04-06T13:49:17.520763+0000","last_undegraded":"2024-04-06T13:49:17.520763+0000","last_fullsized":"2024-04-06T13:49:17.520763+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:35.789803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.518610+0000","last_change":"2024-04-06T13:49:17.517731+0000","last_active":"2024-04-06T13:49:17.518610+0000","last_peered":"2024-04-06T13:49:17.518610+0000","last_clean":"2024-04-06T13:49:17.518610+0000","last_became_active":"2024-04-06T13:49:17.517330+0000","last_became_peered":"2024-04-06T13:49:17.517330+0000","last_unstale":"2024-04-06T13:49:17.518610+0000","last_undegraded":"2024-04-06T13:49:17.518610+0000","last_fullsized":"2024-04-06T13:49:17.518610+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:27:55.678136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1a","version":"268'1","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.524674+0000","last_change":"2024-04-06T13:49:17.093102+0000","last_active":"2024-04-06T13:49:17.524674+0000","last_peered":"2024-04-06T13:49:17.524674+0000","last_clean":"2024-04-06T13:49:17.524674+0000","last_became_active":"2024-04-06T13:49:17.092879+0000","last_became_peered":"2024-04-06T13:49:17.092879+0000","last_unstale":"2024-04-06T13:49:17.524674+0000","last_undegraded":"2024-04-06T13:49:17.524674+0000","last_fullsized":"2024-04-06T13:49:17.524674+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:54:48.687945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.515261+0000","last_change":"2024-04-06T13:49:17.512942+0000","last_active":"2024-04-06T13:49:17.515261+0000","last_peered":"2024-04-06T13:49:17.515261+0000","last_clean":"2024-04-06T13:49:17.515261+0000","last_became_active":"2024-04-06T13:49:17.512730+0000","last_became_peered":"2024-04-06T13:49:17.512730+0000","last_unstale":"2024-04-06T13:49:17.515261+0000","last_undegraded":"2024-04-06T13:49:17.515261+0000","last_fullsized":"2024-04-06T13:49:17.515261+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:16:45.905450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.501191+0000","last_change":"2024-04-06T13:49:17.086842+0000","last_active":"2024-04-06T13:49:17.501191+0000","last_peered":"2024-04-06T13:49:17.501191+0000","last_clean":"2024-04-06T13:49:17.501191+0000","last_became_active":"2024-04-06T13:49:17.086555+0000","last_became_peered":"2024-04-06T13:49:17.086555+0000","last_unstale":"2024-04-06T13:49:17.501191+0000","last_undegraded":"2024-04-06T13:49:17.501191+0000","last_fullsized":"2024-04-06T13:49:17.501191+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:05:57.696105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519835+0000","last_change":"2024-04-06T13:49:17.517335+0000","last_active":"2024-04-06T13:49:17.519835+0000","last_peered":"2024-04-06T13:49:17.519835+0000","last_clean":"2024-04-06T13:49:17.519835+0000","last_became_active":"2024-04-06T13:49:17.516574+0000","last_became_peered":"2024-04-06T13:49:17.516574+0000","last_unstale":"2024-04-06T13:49:17.519835+0000","last_undegraded":"2024-04-06T13:49:17.519835+0000","last_fullsized":"2024-04-06T13:49:17.519835+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:44:33.017556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.510632+0000","last_change":"2024-04-06T13:49:17.101047+0000","last_active":"2024-04-06T13:49:17.510632+0000","last_peered":"2024-04-06T13:49:17.510632+0000","last_clean":"2024-04-06T13:49:17.510632+0000","last_became_active":"2024-04-06T13:49:17.100809+0000","last_became_peered":"2024-04-06T13:49:17.100809+0000","last_unstale":"2024-04-06T13:49:17.510632+0000","last_undegraded":"2024-04-06T13:49:17.510632+0000","last_fullsized":"2024-04-06T13:49:17.510632+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:17:59.126763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523051+0000","last_change":"2024-04-06T13:49:17.521760+0000","last_active":"2024-04-06T13:49:17.523051+0000","last_peered":"2024-04-06T13:49:17.523051+0000","last_clean":"2024-04-06T13:49:17.523051+0000","last_became_active":"2024-04-06T13:49:17.521593+0000","last_became_peered":"2024-04-06T13:49:17.521593+0000","last_unstale":"2024-04-06T13:49:17.523051+0000","last_undegraded":"2024-04-06T13:49:17.523051+0000","last_fullsized":"2024-04-06T13:49:17.523051+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:50.418088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.504021+0000","last_change":"2024-04-06T13:49:17.102458+0000","last_active":"2024-04-06T13:49:17.504021+0000","last_peered":"2024-04-06T13:49:17.504021+0000","last_clean":"2024-04-06T13:49:17.504021+0000","last_became_active":"2024-04-06T13:49:17.101510+0000","last_became_peered":"2024-04-06T13:49:17.101510+0000","last_unstale":"2024-04-06T13:49:17.504021+0000","last_undegraded":"2024-04-06T13:49:17.504021+0000","last_fullsized":"2024-04-06T13:49:17.504021+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:28:43.059891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.520130+0000","last_change":"2024-04-06T13:49:17.516948+0000","last_active":"2024-04-06T13:49:17.520130+0000","last_peered":"2024-04-06T13:49:17.520130+0000","last_clean":"2024-04-06T13:49:17.520130+0000","last_became_active":"2024-04-06T13:49:17.516687+0000","last_became_peered":"2024-04-06T13:49:17.516687+0000","last_unstale":"2024-04-06T13:49:17.520130+0000","last_undegraded":"2024-04-06T13:49:17.520130+0000","last_fullsized":"2024-04-06T13:49:17.520130+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:44:33.805313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.501644+0000","last_change":"2024-04-06T13:49:17.086910+0000","last_active":"2024-04-06T13:49:17.501644+0000","last_peered":"2024-04-06T13:49:17.501644+0000","last_clean":"2024-04-06T13:49:17.501644+0000","last_became_active":"2024-04-06T13:49:17.086616+0000","last_became_peered":"2024-04-06T13:49:17.086616+0000","last_unstale":"2024-04-06T13:49:17.501644+0000","last_undegraded":"2024-04-06T13:49:17.501644+0000","last_fullsized":"2024-04-06T13:49:17.501644+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:44:32.803471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519799+0000","last_change":"2024-04-06T13:49:17.515970+0000","last_active":"2024-04-06T13:49:17.519799+0000","last_peered":"2024-04-06T13:49:17.519799+0000","last_clean":"2024-04-06T13:49:17.519799+0000","last_became_active":"2024-04-06T13:49:17.515734+0000","last_became_peered":"2024-04-06T13:49:17.515734+0000","last_unstale":"2024-04-06T13:49:17.519799+0000","last_undegraded":"2024-04-06T13:49:17.519799+0000","last_fullsized":"2024-04-06T13:49:17.519799+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:05:52.789141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"268'3","reported_seq":13,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.515626+0000","last_change":"2024-04-06T13:49:17.100609+0000","last_active":"2024-04-06T13:49:17.515626+0000","last_peered":"2024-04-06T13:49:17.515626+0000","last_clean":"2024-04-06T13:49:17.515626+0000","last_became_active":"2024-04-06T13:49:17.100323+0000","last_became_peered":"2024-04-06T13:49:17.100323+0000","last_unstale":"2024-04-06T13:49:17.515626+0000","last_undegraded":"2024-04-06T13:49:17.515626+0000","last_fullsized":"2024-04-06T13:49:17.515626+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:35:59.407013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514727+0000","last_change":"2024-04-06T13:49:17.512077+0000","last_active":"2024-04-06T13:49:17.514727+0000","last_peered":"2024-04-06T13:49:17.514727+0000","last_clean":"2024-04-06T13:49:17.514727+0000","last_became_active":"2024-04-06T13:49:17.511254+0000","last_became_peered":"2024-04-06T13:49:17.511254+0000","last_unstale":"2024-04-06T13:49:17.514727+0000","last_undegraded":"2024-04-06T13:49:17.514727+0000","last_fullsized":"2024-04-06T13:49:17.514727+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:44:10.089826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"268'6","reported_seq":16,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.526636+0000","last_change":"2024-04-06T13:49:17.105013+0000","last_active":"2024-04-06T13:49:17.526636+0000","last_peered":"2024-04-06T13:49:17.526636+0000","last_clean":"2024-04-06T13:49:17.526636+0000","last_became_active":"2024-04-06T13:49:17.104721+0000","last_became_peered":"2024-04-06T13:49:17.104721+0000","last_unstale":"2024-04-06T13:49:17.526636+0000","last_undegraded":"2024-04-06T13:49:17.526636+0000","last_fullsized":"2024-04-06T13:49:17.526636+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:11:07.791203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522267+0000","last_change":"2024-04-06T13:49:17.519398+0000","last_active":"2024-04-06T13:49:17.522267+0000","last_peered":"2024-04-06T13:49:17.522267+0000","last_clean":"2024-04-06T13:49:17.522267+0000","last_became_active":"2024-04-06T13:49:17.516546+0000","last_became_peered":"2024-04-06T13:49:17.516546+0000","last_unstale":"2024-04-06T13:49:17.522267+0000","last_undegraded":"2024-04-06T13:49:17.522267+0000","last_fullsized":"2024-04-06T13:49:17.522267+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:28:44.061733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523061+0000","last_change":"2024-04-06T13:49:17.105019+0000","last_active":"2024-04-06T13:49:17.523061+0000","last_peered":"2024-04-06T13:49:17.523061+0000","last_clean":"2024-04-06T13:49:17.523061+0000","last_became_active":"2024-04-06T13:49:17.104653+0000","last_became_peered":"2024-04-06T13:49:17.104653+0000","last_unstale":"2024-04-06T13:49:17.523061+0000","last_undegraded":"2024-04-06T13:49:17.523061+0000","last_fullsized":"2024-04-06T13:49:17.523061+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:02:30.037708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519988+0000","last_change":"2024-04-06T13:49:17.513786+0000","last_active":"2024-04-06T13:49:17.519988+0000","last_peered":"2024-04-06T13:49:17.519988+0000","last_clean":"2024-04-06T13:49:17.519988+0000","last_became_active":"2024-04-06T13:49:17.513533+0000","last_became_peered":"2024-04-06T13:49:17.513533+0000","last_unstale":"2024-04-06T13:49:17.519988+0000","last_undegraded":"2024-04-06T13:49:17.519988+0000","last_fullsized":"2024-04-06T13:49:17.519988+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:07.368454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514604+0000","last_change":"2024-04-06T13:49:17.098866+0000","last_active":"2024-04-06T13:49:17.514604+0000","last_peered":"2024-04-06T13:49:17.514604+0000","last_clean":"2024-04-06T13:49:17.514604+0000","last_became_active":"2024-04-06T13:49:17.098587+0000","last_became_peered":"2024-04-06T13:49:17.098587+0000","last_unstale":"2024-04-06T13:49:17.514604+0000","last_undegraded":"2024-04-06T13:49:17.514604+0000","last_fullsized":"2024-04-06T13:49:17.514604+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:17:32.289853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519633+0000","last_change":"2024-04-06T13:49:17.517876+0000","last_active":"2024-04-06T13:49:17.519633+0000","last_peered":"2024-04-06T13:49:17.519633+0000","last_clean":"2024-04-06T13:49:17.519633+0000","last_became_active":"2024-04-06T13:49:17.517610+0000","last_became_peered":"2024-04-06T13:49:17.517610+0000","last_unstale":"2024-04-06T13:49:17.519633+0000","last_undegraded":"2024-04-06T13:49:17.519633+0000","last_fullsized":"2024-04-06T13:49:17.519633+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:50.418088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.512956+0000","last_change":"2024-04-06T13:49:17.094349+0000","last_active":"2024-04-06T13:49:17.512956+0000","last_peered":"2024-04-06T13:49:17.512956+0000","last_clean":"2024-04-06T13:49:17.512956+0000","last_became_active":"2024-04-06T13:49:17.094054+0000","last_became_peered":"2024-04-06T13:49:17.094054+0000","last_unstale":"2024-04-06T13:49:17.512956+0000","last_undegraded":"2024-04-06T13:49:17.512956+0000","last_fullsized":"2024-04-06T13:49:17.512956+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:17:59.126763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519850+0000","last_change":"2024-04-06T13:49:17.517953+0000","last_active":"2024-04-06T13:49:17.519850+0000","last_peered":"2024-04-06T13:49:17.519850+0000","last_clean":"2024-04-06T13:49:17.519850+0000","last_became_active":"2024-04-06T13:49:17.516926+0000","last_became_peered":"2024-04-06T13:49:17.516926+0000","last_unstale":"2024-04-06T13:49:17.519850+0000","last_undegraded":"2024-04-06T13:49:17.519850+0000","last_fullsized":"2024-04-06T13:49:17.519850+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:26:28.142149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"268'8","reported_seq":18,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.525370+0000","last_change":"2024-04-06T13:49:17.093692+0000","last_active":"2024-04-06T13:49:17.525370+0000","last_peered":"2024-04-06T13:49:17.525370+0000","last_clean":"2024-04-06T13:49:17.525370+0000","last_became_active":"2024-04-06T13:49:17.093394+0000","last_became_peered":"2024-04-06T13:49:17.093394+0000","last_unstale":"2024-04-06T13:49:17.525370+0000","last_undegraded":"2024-04-06T13:49:17.525370+0000","last_fullsized":"2024-04-06T13:49:17.525370+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:10.515169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514564+0000","last_change":"2024-04-06T13:49:17.507798+0000","last_active":"2024-04-06T13:49:17.514564+0000","last_peered":"2024-04-06T13:49:17.514564+0000","last_clean":"2024-04-06T13:49:17.514564+0000","last_became_active":"2024-04-06T13:49:17.507537+0000","last_became_peered":"2024-04-06T13:49:17.507537+0000","last_unstale":"2024-04-06T13:49:17.514564+0000","last_undegraded":"2024-04-06T13:49:17.514564+0000","last_fullsized":"2024-04-06T13:49:17.514564+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:08:18.059036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"268'1","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522825+0000","last_change":"2024-04-06T13:49:17.088149+0000","last_active":"2024-04-06T13:49:17.522825+0000","last_peered":"2024-04-06T13:49:17.522825+0000","last_clean":"2024-04-06T13:49:17.522825+0000","last_became_active":"2024-04-06T13:49:17.086151+0000","last_became_peered":"2024-04-06T13:49:17.086151+0000","last_unstale":"2024-04-06T13:49:17.522825+0000","last_undegraded":"2024-04-06T13:49:17.522825+0000","last_fullsized":"2024-04-06T13:49:17.522825+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:35:20.802113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523020+0000","last_change":"2024-04-06T13:49:17.522315+0000","last_active":"2024-04-06T13:49:17.523020+0000","last_peered":"2024-04-06T13:49:17.523020+0000","last_clean":"2024-04-06T13:49:17.523020+0000","last_became_active":"2024-04-06T13:49:17.522060+0000","last_became_peered":"2024-04-06T13:49:17.522060+0000","last_unstale":"2024-04-06T13:49:17.523020+0000","last_undegraded":"2024-04-06T13:49:17.523020+0000","last_fullsized":"2024-04-06T13:49:17.523020+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:32:51.983606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.e","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519127+0000","last_change":"2024-04-06T13:49:17.100106+0000","last_active":"2024-04-06T13:49:17.519127+0000","last_peered":"2024-04-06T13:49:17.519127+0000","last_clean":"2024-04-06T13:49:17.519127+0000","last_became_active":"2024-04-06T13:49:17.099832+0000","last_became_peered":"2024-04-06T13:49:17.099832+0000","last_unstale":"2024-04-06T13:49:17.519127+0000","last_undegraded":"2024-04-06T13:49:17.519127+0000","last_fullsized":"2024-04-06T13:49:17.519127+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:00:45.856554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514621+0000","last_change":"2024-04-06T13:49:17.507897+0000","last_active":"2024-04-06T13:49:17.514621+0000","last_peered":"2024-04-06T13:49:17.514621+0000","last_clean":"2024-04-06T13:49:17.514621+0000","last_became_active":"2024-04-06T13:49:17.507580+0000","last_became_peered":"2024-04-06T13:49:17.507580+0000","last_unstale":"2024-04-06T13:49:17.514621+0000","last_undegraded":"2024-04-06T13:49:17.514621+0000","last_fullsized":"2024-04-06T13:49:17.514621+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:54.132584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"268'1","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.525068+0000","last_change":"2024-04-06T13:49:17.087553+0000","last_active":"2024-04-06T13:49:17.525068+0000","last_peered":"2024-04-06T13:49:17.525068+0000","last_clean":"2024-04-06T13:49:17.525068+0000","last_became_active":"2024-04-06T13:49:17.086381+0000","last_became_peered":"2024-04-06T13:49:17.086381+0000","last_unstale":"2024-04-06T13:49:17.525068+0000","last_undegraded":"2024-04-06T13:49:17.525068+0000","last_fullsized":"2024-04-06T13:49:17.525068+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:54:55.120647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.518675+0000","last_change":"2024-04-06T13:49:17.513256+0000","last_active":"2024-04-06T13:49:17.518675+0000","last_peered":"2024-04-06T13:49:17.518675+0000","last_clean":"2024-04-06T13:49:17.518675+0000","last_became_active":"2024-04-06T13:49:17.511402+0000","last_became_peered":"2024-04-06T13:49:17.511402+0000","last_unstale":"2024-04-06T13:49:17.518675+0000","last_undegraded":"2024-04-06T13:49:17.518675+0000","last_fullsized":"2024-04-06T13:49:17.518675+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:58:35.003812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.501189+0000","last_change":"2024-04-06T13:49:17.105686+0000","last_active":"2024-04-06T13:49:17.501189+0000","last_peered":"2024-04-06T13:49:17.501189+0000","last_clean":"2024-04-06T13:49:17.501189+0000","last_became_active":"2024-04-06T13:49:17.105408+0000","last_became_peered":"2024-04-06T13:49:17.105408+0000","last_unstale":"2024-04-06T13:49:17.501189+0000","last_undegraded":"2024-04-06T13:49:17.501189+0000","last_fullsized":"2024-04-06T13:49:17.501189+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:52:43.547990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.515517+0000","last_change":"2024-04-06T13:49:17.514716+0000","last_active":"2024-04-06T13:49:17.515517+0000","last_peered":"2024-04-06T13:49:17.515517+0000","last_clean":"2024-04-06T13:49:17.515517+0000","last_became_active":"2024-04-06T13:49:17.514456+0000","last_became_peered":"2024-04-06T13:49:17.514456+0000","last_unstale":"2024-04-06T13:49:17.515517+0000","last_undegraded":"2024-04-06T13:49:17.515517+0000","last_fullsized":"2024-04-06T13:49:17.515517+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:01:01.224352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523098+0000","last_change":"2024-04-06T13:49:17.091792+0000","last_active":"2024-04-06T13:49:17.523098+0000","last_peered":"2024-04-06T13:49:17.523098+0000","last_clean":"2024-04-06T13:49:17.523098+0000","last_became_active":"2024-04-06T13:49:17.091444+0000","last_became_peered":"2024-04-06T13:49:17.091444+0000","last_unstale":"2024-04-06T13:49:17.523098+0000","last_undegraded":"2024-04-06T13:49:17.523098+0000","last_fullsized":"2024-04-06T13:49:17.523098+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:59:32.034392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514941+0000","last_change":"2024-04-06T13:49:17.509208+0000","last_active":"2024-04-06T13:49:17.514941+0000","last_peered":"2024-04-06T13:49:17.514941+0000","last_clean":"2024-04-06T13:49:17.514941+0000","last_became_active":"2024-04-06T13:49:17.508894+0000","last_became_peered":"2024-04-06T13:49:17.508894+0000","last_unstale":"2024-04-06T13:49:17.514941+0000","last_undegraded":"2024-04-06T13:49:17.514941+0000","last_fullsized":"2024-04-06T13:49:17.514941+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:51:52.963672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.506787+0000","last_change":"2024-04-06T13:49:17.108306+0000","last_active":"2024-04-06T13:49:17.506787+0000","last_peered":"2024-04-06T13:49:17.506787+0000","last_clean":"2024-04-06T13:49:17.506787+0000","last_became_active":"2024-04-06T13:49:17.108064+0000","last_became_peered":"2024-04-06T13:49:17.108064+0000","last_unstale":"2024-04-06T13:49:17.506787+0000","last_undegraded":"2024-04-06T13:49:17.506787+0000","last_fullsized":"2024-04-06T13:49:17.506787+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:32:53.801190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519613+0000","last_change":"2024-04-06T13:49:17.517755+0000","last_active":"2024-04-06T13:49:17.519613+0000","last_peered":"2024-04-06T13:49:17.519613+0000","last_clean":"2024-04-06T13:49:17.519613+0000","last_became_active":"2024-04-06T13:49:17.517262+0000","last_became_peered":"2024-04-06T13:49:17.517262+0000","last_unstale":"2024-04-06T13:49:17.519613+0000","last_undegraded":"2024-04-06T13:49:17.519613+0000","last_fullsized":"2024-04-06T13:49:17.519613+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:18:13.304903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.497595+0000","last_change":"2024-04-06T13:49:17.103918+0000","last_active":"2024-04-06T13:49:17.497595+0000","last_peered":"2024-04-06T13:49:17.497595+0000","last_clean":"2024-04-06T13:49:17.497595+0000","last_became_active":"2024-04-06T13:49:17.103733+0000","last_became_peered":"2024-04-06T13:49:17.103733+0000","last_unstale":"2024-04-06T13:49:17.497595+0000","last_undegraded":"2024-04-06T13:49:17.497595+0000","last_fullsized":"2024-04-06T13:49:17.497595+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:10:02.723015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.513040+0000","last_change":"2024-04-06T13:49:17.506938+0000","last_active":"2024-04-06T13:49:17.513040+0000","last_peered":"2024-04-06T13:49:17.513040+0000","last_clean":"2024-04-06T13:49:17.513040+0000","last_became_active":"2024-04-06T13:49:17.506692+0000","last_became_peered":"2024-04-06T13:49:17.506692+0000","last_unstale":"2024-04-06T13:49:17.513040+0000","last_undegraded":"2024-04-06T13:49:17.513040+0000","last_fullsized":"2024-04-06T13:49:17.513040+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:10:53.815420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.501228+0000","last_change":"2024-04-06T13:49:17.105401+0000","last_active":"2024-04-06T13:49:17.501228+0000","last_peered":"2024-04-06T13:49:17.501228+0000","last_clean":"2024-04-06T13:49:17.501228+0000","last_became_active":"2024-04-06T13:49:17.104938+0000","last_became_peered":"2024-04-06T13:49:17.104938+0000","last_unstale":"2024-04-06T13:49:17.501228+0000","last_undegraded":"2024-04-06T13:49:17.501228+0000","last_fullsized":"2024-04-06T13:49:17.501228+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:15:28.868445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523071+0000","last_change":"2024-04-06T13:49:17.521772+0000","last_active":"2024-04-06T13:49:17.523071+0000","last_peered":"2024-04-06T13:49:17.523071+0000","last_clean":"2024-04-06T13:49:17.523071+0000","last_became_active":"2024-04-06T13:49:17.521481+0000","last_became_peered":"2024-04-06T13:49:17.521481+0000","last_unstale":"2024-04-06T13:49:17.523071+0000","last_undegraded":"2024-04-06T13:49:17.523071+0000","last_fullsized":"2024-04-06T13:49:17.523071+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:28:44.061733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.7","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.521896+0000","last_change":"2024-04-06T13:49:17.097613+0000","last_active":"2024-04-06T13:49:17.521896+0000","last_peered":"2024-04-06T13:49:17.521896+0000","last_clean":"2024-04-06T13:49:17.521896+0000","last_became_active":"2024-04-06T13:49:17.097322+0000","last_became_peered":"2024-04-06T13:49:17.097322+0000","last_unstale":"2024-04-06T13:49:17.521896+0000","last_undegraded":"2024-04-06T13:49:17.521896+0000","last_fullsized":"2024-04-06T13:49:17.521896+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:58:34.001970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.511979+0000","last_change":"2024-04-06T13:49:17.509489+0000","last_active":"2024-04-06T13:49:17.511979+0000","last_peered":"2024-04-06T13:49:17.511979+0000","last_clean":"2024-04-06T13:49:17.511979+0000","last_became_active":"2024-04-06T13:49:17.509263+0000","last_became_peered":"2024-04-06T13:49:17.509263+0000","last_unstale":"2024-04-06T13:49:17.511979+0000","last_undegraded":"2024-04-06T13:49:17.511979+0000","last_fullsized":"2024-04-06T13:49:17.511979+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:12:49.230301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.510779+0000","last_change":"2024-04-06T13:49:17.098847+0000","last_active":"2024-04-06T13:49:17.510779+0000","last_peered":"2024-04-06T13:49:17.510779+0000","last_clean":"2024-04-06T13:49:17.510779+0000","last_became_active":"2024-04-06T13:49:17.098539+0000","last_became_peered":"2024-04-06T13:49:17.098539+0000","last_unstale":"2024-04-06T13:49:17.510779+0000","last_undegraded":"2024-04-06T13:49:17.510779+0000","last_fullsized":"2024-04-06T13:49:17.510779+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:17:04.844091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523083+0000","last_change":"2024-04-06T13:49:17.521904+0000","last_active":"2024-04-06T13:49:17.523083+0000","last_peered":"2024-04-06T13:49:17.523083+0000","last_clean":"2024-04-06T13:49:17.523083+0000","last_became_active":"2024-04-06T13:49:17.521642+0000","last_became_peered":"2024-04-06T13:49:17.521642+0000","last_unstale":"2024-04-06T13:49:17.523083+0000","last_undegraded":"2024-04-06T13:49:17.523083+0000","last_fullsized":"2024-04-06T13:49:17.523083+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:47:54.488793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.499034+0000","last_change":"2024-04-06T13:49:17.101607+0000","last_active":"2024-04-06T13:49:17.499034+0000","last_peered":"2024-04-06T13:49:17.499034+0000","last_clean":"2024-04-06T13:49:17.499034+0000","last_became_active":"2024-04-06T13:49:17.101311+0000","last_became_peered":"2024-04-06T13:49:17.101311+0000","last_unstale":"2024-04-06T13:49:17.499034+0000","last_undegraded":"2024-04-06T13:49:17.499034+0000","last_fullsized":"2024-04-06T13:49:17.499034+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:25:27.198338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519622+0000","last_change":"2024-04-06T13:49:17.506594+0000","last_active":"2024-04-06T13:49:17.519622+0000","last_peered":"2024-04-06T13:49:17.519622+0000","last_clean":"2024-04-06T13:49:17.519622+0000","last_became_active":"2024-04-06T13:49:17.503484+0000","last_became_peered":"2024-04-06T13:49:17.503484+0000","last_unstale":"2024-04-06T13:49:17.519622+0000","last_undegraded":"2024-04-06T13:49:17.519622+0000","last_fullsized":"2024-04-06T13:49:17.519622+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:05:36.115690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.511185+0000","last_change":"2024-04-06T13:49:17.094864+0000","last_active":"2024-04-06T13:49:17.511185+0000","last_peered":"2024-04-06T13:49:17.511185+0000","last_clean":"2024-04-06T13:49:17.511185+0000","last_became_active":"2024-04-06T13:49:17.094507+0000","last_became_peered":"2024-04-06T13:49:17.094507+0000","last_unstale":"2024-04-06T13:49:17.511185+0000","last_undegraded":"2024-04-06T13:49:17.511185+0000","last_fullsized":"2024-04-06T13:49:17.511185+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:32:53.801190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522817+0000","last_change":"2024-04-06T13:49:17.500686+0000","last_active":"2024-04-06T13:49:17.522817+0000","last_peered":"2024-04-06T13:49:17.522817+0000","last_clean":"2024-04-06T13:49:17.522817+0000","last_became_active":"2024-04-06T13:49:17.500455+0000","last_became_peered":"2024-04-06T13:49:17.500455+0000","last_unstale":"2024-04-06T13:49:17.522817+0000","last_undegraded":"2024-04-06T13:49:17.522817+0000","last_fullsized":"2024-04-06T13:49:17.522817+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:43:17.512192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.501433+0000","last_change":"2024-04-06T13:49:17.097515+0000","last_active":"2024-04-06T13:49:17.501433+0000","last_peered":"2024-04-06T13:49:17.501433+0000","last_clean":"2024-04-06T13:49:17.501433+0000","last_became_active":"2024-04-06T13:49:17.097211+0000","last_became_peered":"2024-04-06T13:49:17.097211+0000","last_unstale":"2024-04-06T13:49:17.501433+0000","last_undegraded":"2024-04-06T13:49:17.501433+0000","last_fullsized":"2024-04-06T13:49:17.501433+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:00:54.806750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.520547+0000","last_change":"2024-04-06T13:49:17.518763+0000","last_active":"2024-04-06T13:49:17.520547+0000","last_peered":"2024-04-06T13:49:17.520547+0000","last_clean":"2024-04-06T13:49:17.520547+0000","last_became_active":"2024-04-06T13:49:17.516583+0000","last_became_peered":"2024-04-06T13:49:17.516583+0000","last_unstale":"2024-04-06T13:49:17.520547+0000","last_undegraded":"2024-04-06T13:49:17.520547+0000","last_fullsized":"2024-04-06T13:49:17.520547+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:44:33.017556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.506905+0000","last_change":"2024-04-06T13:49:17.108195+0000","last_active":"2024-04-06T13:49:17.506905+0000","last_peered":"2024-04-06T13:49:17.506905+0000","last_clean":"2024-04-06T13:49:17.506905+0000","last_became_active":"2024-04-06T13:49:17.107932+0000","last_became_peered":"2024-04-06T13:49:17.107932+0000","last_unstale":"2024-04-06T13:49:17.506905+0000","last_undegraded":"2024-04-06T13:49:17.506905+0000","last_fullsized":"2024-04-06T13:49:17.506905+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:15:46.589339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.520178+0000","last_change":"2024-04-06T13:49:17.518240+0000","last_active":"2024-04-06T13:49:17.520178+0000","last_peered":"2024-04-06T13:49:17.520178+0000","last_clean":"2024-04-06T13:49:17.520178+0000","last_became_active":"2024-04-06T13:49:17.517985+0000","last_became_peered":"2024-04-06T13:49:17.517985+0000","last_unstale":"2024-04-06T13:49:17.520178+0000","last_undegraded":"2024-04-06T13:49:17.520178+0000","last_fullsized":"2024-04-06T13:49:17.520178+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:27:31.865608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.503806+0000","last_change":"2024-04-06T13:49:17.102416+0000","last_active":"2024-04-06T13:49:17.503806+0000","last_peered":"2024-04-06T13:49:17.503806+0000","last_clean":"2024-04-06T13:49:17.503806+0000","last_became_active":"2024-04-06T13:49:17.102080+0000","last_became_peered":"2024-04-06T13:49:17.102080+0000","last_unstale":"2024-04-06T13:49:17.503806+0000","last_undegraded":"2024-04-06T13:49:17.503806+0000","last_fullsized":"2024-04-06T13:49:17.503806+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:27:31.581867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.517664+0000","last_change":"2024-04-06T13:49:17.511968+0000","last_active":"2024-04-06T13:49:17.517664+0000","last_peered":"2024-04-06T13:49:17.517664+0000","last_clean":"2024-04-06T13:49:17.517664+0000","last_became_active":"2024-04-06T13:49:17.511704+0000","last_became_peered":"2024-04-06T13:49:17.511704+0000","last_unstale":"2024-04-06T13:49:17.517664+0000","last_undegraded":"2024-04-06T13:49:17.517664+0000","last_fullsized":"2024-04-06T13:49:17.517664+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:36:00.408855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.501319+0000","last_change":"2024-04-06T13:49:17.105215+0000","last_active":"2024-04-06T13:49:17.501319+0000","last_peered":"2024-04-06T13:49:17.501319+0000","last_clean":"2024-04-06T13:49:17.501319+0000","last_became_active":"2024-04-06T13:49:17.104577+0000","last_became_peered":"2024-04-06T13:49:17.104577+0000","last_unstale":"2024-04-06T13:49:17.501319+0000","last_undegraded":"2024-04-06T13:49:17.501319+0000","last_fullsized":"2024-04-06T13:49:17.501319+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:14:52.150676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.509604+0000","last_change":"2024-04-06T13:49:17.507776+0000","last_active":"2024-04-06T13:49:17.509604+0000","last_peered":"2024-04-06T13:49:17.509604+0000","last_clean":"2024-04-06T13:49:17.509604+0000","last_became_active":"2024-04-06T13:49:17.507456+0000","last_became_peered":"2024-04-06T13:49:17.507456+0000","last_unstale":"2024-04-06T13:49:17.509604+0000","last_undegraded":"2024-04-06T13:49:17.509604+0000","last_fullsized":"2024-04-06T13:49:17.509604+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:50.418088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'381","reported_seq":1396,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.085080+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:49:17.085080+0000","last_peered":"2024-04-06T13:49:17.085080+0000","last_clean":"2024-04-06T13:49:17.085080+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:49:17.085080+0000","last_undegraded":"2024-04-06T13:49:17.085080+0000","last_fullsized":"2024-04-06T13:49:17.085080+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576002,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57124,"kb_used_data":3864,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314716,"statfs":{"total":96636764160,"available":96578269184,"internally_reserved":0,"allocated":3956736,"data_stored":2324678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55700000000000005,"15min":0.59199999999999997},"min":{"1min":0.34499999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.747,"5min":0.751,"15min":0.82499999999999996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.54700000000000004,"15min":0.58299999999999996},"min":{"1min":0.35099999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.753,"5min":0.78500000000000003,"15min":0.82899999999999996},"last":0.46400000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53800000000000003,"15min":0.57499999999999996},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.69199999999999995,"5min":0.77700000000000002,"15min":0.81299999999999994},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":0.58799999999999997},"min":{"1min":0.39200000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.85299999999999998},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.53700000000000003,"15min":0.57599999999999996},"min":{"1min":0.379,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.63800000000000001,"5min":0.78600000000000003,"15min":0.83899999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57099999999999995,"15min":0.59599999999999997},"min":{"1min":0.373,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.79700000000000004},"last":0.44}]},{"osd":3,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.56399999999999995,"15min":0.59699999999999998},"min":{"1min":0.36799999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.72699999999999998,"5min":0.76700000000000002,"15min":0.81100000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.57399999999999995,"15min":0.59799999999999998},"min":{"1min":0.436,"5min":0.39000000000000001,"15min":0.34999999999999998},"max":{"1min":0.66500000000000004,"5min":0.78900000000000003,"15min":0.84099999999999997},"last":0.58999999999999997}]},{"osd":4,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.45700000000000002,"15min":0.46300000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.67600000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54200000000000004,"15min":0.54000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64700000000000002,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.70299999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.498,"15min":0.496},"min":{"1min":0.316,"5min":0.315,"15min":0.308},"max":{"1min":0.68999999999999995,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.44500000000000001,"5min":0.35599999999999998,"15min":0.33600000000000002},"max":{"1min":0.77300000000000002,"5min":0.79700000000000004,"15min":0.82099999999999995},"last":0.68600000000000005}]},{"osd":6,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55400000000000005,"15min":0.55500000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.69099999999999995,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56100000000000005,"15min":0.58099999999999996},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.65700000000000003,"5min":0.81799999999999995,"15min":0.88200000000000001},"last":0.63900000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56556,"kb_used_data":3292,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315284,"statfs":{"total":96636764160,"available":96578850816,"internally_reserved":0,"allocated":3371008,"data_stored":1739958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.51900000000000002,"15min":0.55000000000000004},"min":{"1min":0.35199999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.59199999999999997,"5min":0.79700000000000004,"15min":0.83799999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.497,"5min":0.53300000000000003,"15min":0.56200000000000006},"min":{"1min":0.314,"5min":0.31,"15min":0.31},"max":{"1min":0.63,"5min":0.76900000000000002,"15min":0.81999999999999995},"last":0.44}]},{"osd":1,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.59099999999999997,"15min":0.61699999999999999},"min":{"1min":0.39600000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70699999999999996,"5min":0.81699999999999995,"15min":0.85799999999999998},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.60299999999999998,"15min":0.63900000000000001},"min":{"1min":0.45100000000000001,"5min":0.41099999999999998,"15min":0.378},"max":{"1min":0.74199999999999999,"5min":0.78900000000000003,"15min":0.82699999999999996},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56100000000000005,"15min":0.58899999999999997},"min":{"1min":0.41799999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72899999999999998,"5min":0.79000000000000004,"15min":0.82899999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.54200000000000004,"15min":0.57899999999999996},"min":{"1min":0.35899999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67400000000000004,"5min":0.77700000000000002,"15min":0.80700000000000005},"last":0.49199999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.56699999999999995,"15min":0.59799999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69999999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61599999999999999,"15min":0.63900000000000001},"min":{"1min":0.441,"5min":0.39900000000000002,"15min":0.36099999999999999},"max":{"1min":0.77200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.59199999999999997}]},{"osd":4,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53300000000000003,"15min":0.53100000000000003},"min":{"1min":0.36099999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.81399999999999995},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51500000000000001,"15min":0.53400000000000003},"min":{"1min":0.32900000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.73699999999999999,"5min":0.82099999999999995,"15min":0.84099999999999997},"last":0.61799999999999999}]},{"osd":5,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.501,"15min":0.50600000000000001},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.33300000000000002},"max":{"1min":0.754,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.628},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52200000000000002,"15min":0.53100000000000003},"min":{"1min":0.40400000000000003,"5min":0.316,"15min":0.316},"max":{"1min":0.72599999999999998,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.56399999999999995}]},{"osd":7,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79300000000000004,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55800000000000005,"15min":0.56699999999999995},"min":{"1min":0.39900000000000002,"5min":0.35399999999999998,"15min":0.32800000000000001},"max":{"1min":0.77200000000000002,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.60199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834576000,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64764,"kb_used_data":3308,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307076,"statfs":{"total":96636764160,"available":96570445824,"internally_reserved":0,"allocated":3387392,"data_stored":1747215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.36299999999999999,"15min":0.38500000000000001},"min":{"1min":0.26100000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.443,"5min":0.621,"15min":0.621},"last":1.2230000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.39200000000000002,"15min":0.41999999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.504,"5min":0.56100000000000005,"15min":0.69399999999999995},"last":2.0539999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.39400000000000002,"15min":0.41699999999999998},"min":{"1min":0.25600000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.41999999999999998,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.42699999999999999,"15min":0.45600000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.51900000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":2.0169999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.44600000000000001,"15min":0.47199999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.51100000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":4.2699999999999996},{"interface":"front","average":{"1min":0.374,"5min":0.40799999999999997,"15min":0.434},"min":{"1min":0.29599999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.51700000000000002,"5min":0.63500000000000001,"15min":0.66100000000000003},"last":2.153}]},{"osd":4,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.59399999999999997,"15min":0.63400000000000001},"min":{"1min":0.377,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.80300000000000005,"5min":0.89900000000000002,"15min":0.97399999999999998},"last":1.1140000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.61199999999999999,"15min":0.64300000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.79300000000000004,"5min":0.877,"15min":0.995},"last":0.82499999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.60499999999999998,"15min":0.64200000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.72699999999999998,"5min":0.91800000000000004,"15min":1.073},"last":1.1779999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.626,"15min":0.65900000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.73999999999999999,"5min":0.96499999999999997,"15min":1.0209999999999999},"last":4.2030000000000003}]},{"osd":6,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.61799999999999999,"15min":0.64900000000000002},"min":{"1min":0.435,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.753,"5min":0.89200000000000002,"15min":1},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.63600000000000001,"15min":0.67200000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76600000000000001,"5min":0.94599999999999995,"15min":1.036},"last":4.2889999999999997}]},{"osd":7,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.66400000000000003,"15min":0.68799999999999994},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.94899999999999995,"15min":1.0649999999999999},"last":4.0730000000000004},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.626,"15min":0.66000000000000003},"min":{"1min":0.44700000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.78100000000000003,"5min":0.92900000000000005,"15min":1.056},"last":2.194}]}]},{"osd":0,"up_from":13,"seq":55834576000,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61220,"kb_used_data":3860,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310620,"statfs":{"total":96636764160,"available":96574074880,"internally_reserved":0,"allocated":3952640,"data_stored":2323659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.36899999999999999,"15min":0.38900000000000001},"min":{"1min":0.24299999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.47199999999999998,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.41299999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.50900000000000001,"5min":0.58599999999999997,"15min":0.70799999999999996},"last":0.377}]},{"osd":2,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39000000000000001,"15min":0.42199999999999999},"min":{"1min":0.28299999999999997,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.53900000000000003,"5min":0.65400000000000003,"15min":0.70099999999999996},"last":0.315},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.41699999999999998,"15min":0.441},"min":{"1min":0.29899999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.58199999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.33200000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.39600000000000002,"15min":0.42099999999999999},"min":{"1min":0.27300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52400000000000002,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.44800000000000001,"15min":0.47399999999999998},"min":{"1min":0.313,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.60899999999999999,"5min":0.71899999999999997,"15min":0.75700000000000001},"last":0.35299999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.59099999999999997,"15min":0.63600000000000001},"min":{"1min":0.38500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.85199999999999998,"5min":0.93100000000000005,"15min":1.0329999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.60299999999999998,"15min":0.64100000000000001},"min":{"1min":0.39800000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86099999999999999,"5min":0.999,"15min":0.999},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59499999999999997,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.83899999999999997,"5min":0.91800000000000004,"15min":1.0009999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.627,"15min":0.66200000000000003},"min":{"1min":0.42699999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.86499999999999999,"5min":0.92800000000000005,"15min":1.052},"last":0.56699999999999995}]},{"osd":6,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63100000000000001,"15min":0.67000000000000004},"min":{"1min":0.438,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.81699999999999995,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.629,"15min":0.66800000000000004},"min":{"1min":0.44900000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":1.085},"last":0.50700000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.64900000000000002,"15min":0.68799999999999994},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.89400000000000002,"5min":1.01,"15min":1.121},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.626,"5min":0.64000000000000001,"15min":0.67000000000000004},"min":{"1min":0.47299999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.82999999999999996,"5min":0.95799999999999996,"15min":1.0229999999999999},"last":0.48499999999999999}]}]},{"osd":2,"up_from":13,"seq":55834576002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60640,"kb_used_data":3280,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311200,"statfs":{"total":96636764160,"available":96574668800,"internally_reserved":0,"allocated":3358720,"data_stored":1733291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.36799999999999999,"15min":0.38100000000000001},"min":{"1min":0.22,"5min":0.184,"15min":0.184},"max":{"1min":0.59399999999999997,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.39700000000000002,"15min":0.41899999999999998},"min":{"1min":0.26200000000000001,"5min":0.251,"15min":0.249},"max":{"1min":0.54600000000000004,"5min":0.71499999999999997,"15min":1.044},"last":0.53700000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.371,"15min":0.39900000000000002},"min":{"1min":0.23599999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.46899999999999997,"5min":0.52900000000000003,"15min":0.753},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.41799999999999998,"15min":0.441},"min":{"1min":0.27000000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.82199999999999995},"last":0.32400000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.42199999999999999,"15min":0.45200000000000001},"min":{"1min":0.27000000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.621,"5min":0.63900000000000001,"15min":0.70299999999999996},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.42499999999999999,"15min":0.443},"min":{"1min":0.27200000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.433}]},{"osd":4,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61099999999999999,"15min":0.63300000000000001},"min":{"1min":0.436,"5min":0.40200000000000002,"15min":0.376},"max":{"1min":0.77600000000000002,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.63,"15min":0.65400000000000003},"min":{"1min":0.40999999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82799999999999996,"5min":1.022,"15min":1.022},"last":0.76300000000000001}]},{"osd":5,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.629,"15min":0.65800000000000003},"min":{"1min":0.44900000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79000000000000004,"5min":0.98599999999999999,"15min":1.0149999999999999},"last":0.752},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.63800000000000001,"15min":0.66000000000000003},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85199999999999998,"5min":1.028,"15min":1.028},"last":0.67200000000000004}]},{"osd":6,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64500000000000002,"15min":0.66600000000000004},"min":{"1min":0.45300000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86899999999999999,"5min":0.97199999999999998,"15min":1.091},"last":0.63},{"interface":"front","average":{"1min":0.625,"5min":0.65900000000000003,"15min":0.68799999999999994},"min":{"1min":0.47699999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84099999999999997,"5min":0.95699999999999996,"15min":0.96099999999999997},"last":0.70199999999999996}]},{"osd":7,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.66100000000000003,"15min":0.68300000000000005},"min":{"1min":0.47299999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91300000000000003,"5min":0.94499999999999995,"15min":0.95399999999999996},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.66900000000000004,"15min":0.69999999999999996},"min":{"1min":0.46100000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.89100000000000001,"5min":1.01,"15min":1.01},"last":0.79500000000000004}]}]},{"osd":3,"up_from":13,"seq":55834576001,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48352,"kb_used_data":3284,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323488,"statfs":{"total":96636764160,"available":96587251712,"internally_reserved":0,"allocated":3362816,"data_stored":1734310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.37,"15min":0.38400000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.47399999999999998,"5min":0.55200000000000005,"15min":0.72799999999999998},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.39600000000000002,"15min":0.40799999999999997},"min":{"1min":0.25700000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.48799999999999999,"5min":0.58599999999999997,"15min":0.66700000000000004},"last":0.307}]},{"osd":1,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.39300000000000002,"15min":0.40100000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.46899999999999997,"5min":0.59499999999999997,"15min":0.65700000000000003},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.41699999999999998,"15min":0.437},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.502,"5min":0.63,"15min":0.75800000000000001},"last":0.32300000000000001}]},{"osd":2,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.46100000000000002,"15min":0.47999999999999998},"min":{"1min":0.32700000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.53900000000000003,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.47699999999999998,"15min":0.496},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.51900000000000002,"5min":0.67900000000000005,"15min":0.79300000000000004},"last":0.42799999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.622,"15min":0.63100000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76700000000000002,"5min":0.94099999999999995,"15min":1.018},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.628,"15min":0.64200000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.81100000000000005,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.63900000000000001}]},{"osd":5,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.66100000000000003,"15min":0.67600000000000005},"min":{"1min":0.44700000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.872,"5min":0.95699999999999996,"15min":0.999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.67900000000000005,"15min":0.68700000000000006},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.86299999999999999,"5min":1.036,"15min":1.036},"last":0.627}]},{"osd":6,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.64100000000000001,"15min":0.66100000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.76300000000000001,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.66000000000000003,"15min":0.67600000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41099999999999998},"max":{"1min":0.82799999999999996,"5min":0.91600000000000004,"15min":0.98799999999999999},"last":0.55000000000000004}]},{"osd":7,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.65500000000000003,"15min":0.66800000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80200000000000005,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.67800000000000005,"15min":0.69499999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85199999999999998,"5min":1.0209999999999999,"15min":1.024},"last":0.67400000000000004}]}]},{"osd":4,"up_from":13,"seq":55834576001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48364,"kb_used_data":3296,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323476,"statfs":{"total":96636764160,"available":96587239424,"internally_reserved":0,"allocated":3375104,"data_stored":1740977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55300000000000005,"15min":0.57999999999999996},"min":{"1min":0.317,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73299999999999998,"5min":0.80900000000000005,"15min":0.88500000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.57599999999999996,"15min":0.59099999999999997},"min":{"1min":0.41699999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79200000000000004,"5min":1.202,"15min":1.202},"last":0.60999999999999999}]},{"osd":1,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60099999999999998,"15min":0.63400000000000001},"min":{"1min":0.379,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.74299999999999999,"5min":1.256,"15min":1.256},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61499999999999999,"15min":0.624},"min":{"1min":0.39300000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.71999999999999997,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.57999999999999996}]},{"osd":2,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.56899999999999995,"15min":0.59099999999999997},"min":{"1min":0.377,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.68200000000000005,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.59299999999999997,"15min":0.61199999999999999},"min":{"1min":0.373,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89500000000000002,"5min":1.294,"15min":1.294},"last":0.67900000000000005}]},{"osd":3,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.57499999999999996,"15min":0.58999999999999997},"min":{"1min":0.35999999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80800000000000005,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60399999999999998,"15min":0.62},"min":{"1min":0.439,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76700000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.69299999999999995}]},{"osd":5,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56100000000000005,"15min":0.52000000000000002},"min":{"1min":0.33800000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.75,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56999999999999995,"15min":0.54100000000000004},"min":{"1min":0.372,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.75,"5min":4.931,"15min":4.931},"last":0.73799999999999999}]},{"osd":6,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53900000000000003,"15min":0.53400000000000003},"min":{"1min":0.40699999999999997,"5min":0.34100000000000003,"15min":0.33300000000000002},"max":{"1min":0.78300000000000003,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56599999999999995,"15min":0.56999999999999995},"min":{"1min":0.41099999999999998,"5min":0.35799999999999998,"15min":0.34699999999999998},"max":{"1min":0.97099999999999997,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.81000000000000005}]},{"osd":7,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60199999999999998,"15min":0.58199999999999996},"min":{"1min":0.35899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.84999999999999998,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.59399999999999997,"15min":0.60099999999999998},"min":{"1min":0.48599999999999999,"5min":0.38,"15min":0.373},"max":{"1min":0.82799999999999996,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83499999999999996}]}]},{"osd":5,"up_from":13,"seq":55834576001,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66400,"kb_used_data":3280,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305440,"statfs":{"total":96636764160,"available":96568770560,"internally_reserved":0,"allocated":3358720,"data_stored":1733291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.56100000000000005,"15min":0.58299999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.72799999999999998,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.56599999999999995,"15min":0.58899999999999997},"min":{"1min":0.29899999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.74399999999999999,"5min":0.84699999999999998,"15min":0.89200000000000002},"last":0.53700000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.56200000000000006,"15min":0.58799999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.71299999999999997,"5min":0.81299999999999994,"15min":0.81399999999999995},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.56100000000000005,"15min":0.58299999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.752,"5min":0.871,"15min":0.871},"last":0.47099999999999997}]},{"osd":2,"last update":"Sat Apr 6 13:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.32300000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.57099999999999995,"15min":0.59399999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.41399999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.57599999999999996,"15min":0.59599999999999997},"min":{"1min":0.375,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.72199999999999998,"5min":0.81100000000000005,"15min":0.94499999999999995},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.60699999999999998,"15min":0.63100000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.80200000000000005,"5min":0.88200000000000001,"15min":0.91500000000000004},"last":0.50900000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53000000000000003,"15min":0.50900000000000001},"min":{"1min":0.35499999999999998,"5min":0.32700000000000001,"15min":0.27400000000000002},"max":{"1min":0.72399999999999998,"5min":0.85699999999999998,"15min":1.0369999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59599999999999997,"15min":0.58099999999999996},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.76600000000000001,"5min":0.88800000000000001,"15min":1.1639999999999999},"last":0.59799999999999998}]},{"osd":6,"last update":"Sat Apr 6 13:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54700000000000004,"15min":0.53100000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.28899999999999998},"max":{"1min":0.80000000000000004,"5min":0.94999999999999996,"15min":1.0169999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54400000000000004,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72499999999999998,"5min":0.878,"15min":1.0009999999999999},"last":0.56599999999999995}]},{"osd":7,"last update":"Sat Apr 6 13:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.56100000000000005,"15min":0.55000000000000004},"min":{"1min":0.38300000000000001,"5min":0.35799999999999998,"15min":0.32900000000000001},"max":{"1min":0.77100000000000002,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.66200000000000003,"15min":0.65100000000000002},"min":{"1min":0.45800000000000002,"5min":0.41699999999999998,"15min":0.38300000000000001},"max":{"1min":0.82999999999999996,"5min":0.92300000000000004,"15min":0.96599999999999997},"last":0.622}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:49:21.618 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:49:21.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.775+0000 7f736ca48700 1 Processor -- start 2024-04-06T13:49:21.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.775+0000 7f736ca48700 1 -- start start 2024-04-06T13:49:21.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736ca48700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73680d11b0 0x7f73680d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:21.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736ca48700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f73680cb870 0x7f736815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:21.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736ca48700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73680d1ad0 con 0x7f73680cfa00 2024-04-06T13:49:21.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736ca48700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73680ccdc0 con 0x7f73680cb870 2024-04-06T13:49:21.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736ca48700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73680d0b50 con 0x7f73680d11b0 2024-04-06T13:49:21.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f7366d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f73680cfa00 0x7f73680cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40052/0 (socket says 172.21.15.189:40052) 2024-04-06T13:49:21.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f7366d9d700 1 -- 172.21.15.189:0/2052262093 learned_addr learned my addr 172.21.15.189:0/2052262093 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:21.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f7365d9b700 1 --2- 172.21.15.189:0/2052262093 >> v2:172.21.15.192:3300/0 conn(0x7f73680cb870 0x7f736815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:21.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736659c700 1 --2- 172.21.15.189:0/2052262093 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73680d11b0 0x7f73680d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:21.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736559a700 1 -- 172.21.15.189:0/2052262093 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608188700 0 0) 0x7f73680d1ad0 con 0x7f73680cfa00 2024-04-06T13:49:21.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736559a700 1 -- 172.21.15.189:0/2052262093 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73500035b0 con 0x7f73680cfa00 2024-04-06T13:49:21.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736559a700 1 -- 172.21.15.189:0/2052262093 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3374821851 0 0) 0x7f73500035b0 con 0x7f73680cfa00 2024-04-06T13:49:21.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736559a700 1 -- 172.21.15.189:0/2052262093 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73680d11b0 msgr2=0x7f73680d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:21.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736559a700 1 --2- 172.21.15.189:0/2052262093 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73680d11b0 0x7f73680d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736559a700 1 -- 172.21.15.189:0/2052262093 >> v2:172.21.15.192:3300/0 conn(0x7f73680cb870 msgr2=0x7f736815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:21.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736559a700 1 --2- 172.21.15.189:0/2052262093 >> v2:172.21.15.192:3300/0 conn(0x7f73680cb870 0x7f736815c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736559a700 1 -- 172.21.15.189:0/2052262093 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73680cbd80 con 0x7f73680cfa00 2024-04-06T13:49:21.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736559a700 1 -- 172.21.15.189:0/2052262093 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f73600019f0 con 0x7f73680cfa00 2024-04-06T13:49:21.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736559a700 1 -- 172.21.15.189:0/2052262093 >> v1:172.21.15.189:6789/0 conn(0x7f73680cfa00 legacy=0x7f73680cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:21.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736559a700 1 --2- 172.21.15.189:0/2052262093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7350003a00 0x7f7350003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:21.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736559a700 1 --2- 172.21.15.189:0/2052262093 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73680d11b0 0x7f7350004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:21.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736559a700 1 --2- 172.21.15.189:0/2052262093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7350004d90 0x7f7350005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:21.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736559a700 1 -- 172.21.15.189:0/2052262093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73680d0b50 con 0x7f7350004d90 2024-04-06T13:49:21.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736559a700 1 -- 172.21.15.189:0/2052262093 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735000ba10 con 0x7f7350003a00 2024-04-06T13:49:21.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.776+0000 7f736559a700 1 -- 172.21.15.189:0/2052262093 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735000bb50 con 0x7f73680d11b0 2024-04-06T13:49:21.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.777+0000 7f7365d9b700 1 --2- 172.21.15.189:0/2052262093 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73680d11b0 0x7f7350004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:21.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.777+0000 7f736659c700 1 --2- 172.21.15.189:0/2052262093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7350003a00 0x7f7350003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:21.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.777+0000 7f7366d9d700 1 --2- 172.21.15.189:0/2052262093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7350004d90 0x7f7350005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:21.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.777+0000 7f7366d9d700 1 -- 172.21.15.189:0/2052262093 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73680d11b0 msgr2=0x7f7350004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:21.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.777+0000 7f7366d9d700 1 --2- 172.21.15.189:0/2052262093 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73680d11b0 0x7f7350004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.777+0000 7f7366d9d700 1 -- 172.21.15.189:0/2052262093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7350003a00 msgr2=0x7f7350003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:21.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.777+0000 7f7366d9d700 1 --2- 172.21.15.189:0/2052262093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7350003a00 0x7f7350003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.777+0000 7f7366d9d700 1 -- 172.21.15.189:0/2052262093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73500035b0 con 0x7f7350004d90 2024-04-06T13:49:21.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.777+0000 7f7366d9d700 1 --2- 172.21.15.189:0/2052262093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7350004d90 0x7f7350005210 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 crypto rx=0x7f736000a860 tx=0x7f7350005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc0cdbb0f026e8d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:21.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.777+0000 7f736559a700 1 -- 172.21.15.189:0/2052262093 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360012600 con 0x7f7350004d90 2024-04-06T13:49:21.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.777+0000 7f736559a700 1 -- 172.21.15.189:0/2052262093 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7360012780 con 0x7f7350004d90 2024-04-06T13:49:21.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.777+0000 7f736559a700 1 -- 172.21.15.189:0/2052262093 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360012b30 con 0x7f7350004d90 2024-04-06T13:49:21.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.778+0000 7f736ca48700 1 -- 172.21.15.189:0/2052262093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7350004d90 msgr2=0x7f7350005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:21.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.778+0000 7f736ca48700 1 --2- 172.21.15.189:0/2052262093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7350004d90 0x7f7350005210 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 crypto rx=0x7f736000a860 tx=0x7f7350005880 comp rx=0 tx=0).stop 2024-04-06T13:49:21.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.778+0000 7f7366d9d700 1 -- 172.21.15.189:0/2052262093 reap_dead start 2024-04-06T13:49:21.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.778+0000 7f736ca48700 1 -- 172.21.15.189:0/2052262093 shutdown_connections 2024-04-06T13:49:21.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.778+0000 7f736ca48700 1 -- 172.21.15.189:0/2052262093 >> 172.21.15.189:0/2052262093 conn(0x7f73680b2db0 msgr2=0x7f73680b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:21.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.778+0000 7f736ca48700 1 -- 172.21.15.189:0/2052262093 shutdown_connections 2024-04-06T13:49:21.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.778+0000 7f736ca48700 1 -- 172.21.15.189:0/2052262093 wait complete. 2024-04-06T13:49:21.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.779+0000 7f736ca48700 1 Processor -- start 2024-04-06T13:49:21.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.779+0000 7f736ca48700 1 -- start start 2024-04-06T13:49:21.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.780+0000 7f736ca48700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7350004d90 0x7f7368154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:21.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.780+0000 7f736ca48700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7368155090 0x7f73681f12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:21.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.780+0000 7f736ca48700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73681f17e0 0x7f73680c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:21.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.780+0000 7f736ca48700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7368156360 con 0x7f7350004d90 2024-04-06T13:49:21.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.780+0000 7f736ca48700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7368156640 con 0x7f73681f17e0 2024-04-06T13:49:21.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.780+0000 7f736ca48700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7368156920 con 0x7f7368155090 2024-04-06T13:49:21.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.780+0000 7f736659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7350004d90 0x7f7368154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:21.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.780+0000 7f736659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7350004d90 0x7f7368154a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46240/0 (socket says 172.21.15.189:46240) 2024-04-06T13:49:21.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.780+0000 7f736659c700 1 -- 172.21.15.189:0/926518374 learned_addr learned my addr 172.21.15.189:0/926518374 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:21.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.780+0000 7f7365d9b700 1 --2- 172.21.15.189:0/926518374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7368155090 0x7f73681f12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:21.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.780+0000 7f7366d9d700 1 --2- 172.21.15.189:0/926518374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73681f17e0 0x7f73680c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:21.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.780+0000 7f736659c700 1 -- 172.21.15.189:0/926518374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7368155090 msgr2=0x7f73681f12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:21.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.780+0000 7f736659c700 1 --2- 172.21.15.189:0/926518374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7368155090 0x7f73681f12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.780+0000 7f736659c700 1 -- 172.21.15.189:0/926518374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73681f17e0 msgr2=0x7f73680c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:21.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.780+0000 7f736659c700 1 --2- 172.21.15.189:0/926518374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73681f17e0 0x7f73680c6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.780+0000 7f736659c700 1 -- 172.21.15.189:0/926518374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7368156c00 con 0x7f7350004d90 2024-04-06T13:49:21.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.780+0000 7f736659c700 1 --2- 172.21.15.189:0/926518374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7350004d90 0x7f7368154a30 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f73580119a0 tx=0x7f7358011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:21.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.781+0000 7f7356ffd700 1 -- 172.21.15.189:0/926518374 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735800f940 con 0x7f7350004d90 2024-04-06T13:49:21.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.781+0000 7f7356ffd700 1 -- 172.21.15.189:0/926518374 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735800faa0 con 0x7f7350004d90 2024-04-06T13:49:21.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.781+0000 7f736ca48700 1 -- 172.21.15.189:0/926518374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7368156ee0 con 0x7f7350004d90 2024-04-06T13:49:21.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.781+0000 7f7356ffd700 1 -- 172.21.15.189:0/926518374 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735800fe30 con 0x7f7350004d90 2024-04-06T13:49:21.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.781+0000 7f736ca48700 1 -- 172.21.15.189:0/926518374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7368155510 con 0x7f7350004d90 2024-04-06T13:49:21.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.782+0000 7f736ca48700 1 -- 172.21.15.189:0/926518374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7368073e40 con 0x7f7350004d90 2024-04-06T13:49:21.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.782+0000 7f7356ffd700 1 -- 172.21.15.189:0/926518374 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f7358014440 con 0x7f7350004d90 2024-04-06T13:49:21.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.783+0000 7f7356ffd700 1 --2- 172.21.15.189:0/926518374 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f734806aed0 0x7f734806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:21.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.783+0000 7f7365d9b700 1 --2- 172.21.15.189:0/926518374 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f734806aed0 0x7f734806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:21.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.783+0000 7f7356ffd700 1 -- 172.21.15.189:0/926518374 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f735808a080 con 0x7f7350004d90 2024-04-06T13:49:21.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.783+0000 7f7365d9b700 1 --2- 172.21.15.189:0/926518374 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f734806aed0 0x7f734806d390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f73680cbcb0 tx=0x7f735c008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:21.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.787+0000 7f7356ffd700 1 -- 172.21.15.189:0/926518374 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7358054d70 con 0x7f7350004d90 2024-04-06T13:49:21.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.980+0000 7f736ca48700 1 -- 172.21.15.189:0/926518374 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f73681567b0 con 0x7f734806aed0 2024-04-06T13:49:21.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.988+0000 7f7356ffd700 1 -- 172.21.15.189:0/926518374 <== mgr.12522 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289886 (secure 0 0 0) 0x7f73681567b0 con 0x7f734806aed0 2024-04-06T13:49:21.993 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:21.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.992+0000 7f736ca48700 1 -- 172.21.15.189:0/926518374 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f734806aed0 msgr2=0x7f734806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:21.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.992+0000 7f736ca48700 1 --2- 172.21.15.189:0/926518374 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f734806aed0 0x7f734806d390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f73680cbcb0 tx=0x7f735c008040 comp rx=0 tx=0).stop 2024-04-06T13:49:21.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.992+0000 7f736ca48700 1 -- 172.21.15.189:0/926518374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7350004d90 msgr2=0x7f7368154a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:21.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.992+0000 7f736ca48700 1 --2- 172.21.15.189:0/926518374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7350004d90 0x7f7368154a30 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f73580119a0 tx=0x7f7358011d60 comp rx=0 tx=0).stop 2024-04-06T13:49:21.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.992+0000 7f736ca48700 1 -- 172.21.15.189:0/926518374 shutdown_connections 2024-04-06T13:49:21.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.992+0000 7f736ca48700 1 --2- 172.21.15.189:0/926518374 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f734806aed0 0x7f734806d390 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.992+0000 7f736ca48700 1 --2- 172.21.15.189:0/926518374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73681f17e0 0x7f73680c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.992+0000 7f736ca48700 1 --2- 172.21.15.189:0/926518374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7350004d90 0x7f7368154a30 unknown :-1 s=CLOSED pgs=2940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.992+0000 7f736ca48700 1 --2- 172.21.15.189:0/926518374 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7368155090 0x7f73681f12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:21.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.992+0000 7f736ca48700 1 -- 172.21.15.189:0/926518374 >> 172.21.15.189:0/926518374 conn(0x7f73680b2db0 msgr2=0x7f73680cd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:21.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.992+0000 7f736ca48700 1 -- 172.21.15.189:0/926518374 shutdown_connections 2024-04-06T13:49:21.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:21.992+0000 7f736ca48700 1 -- 172.21.15.189:0/926518374 wait complete. 2024-04-06T13:49:21.997 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:49:22.004 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-04-06T13:49:20.756983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":683,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":419,"ondisk_log_size":419,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":463420,"kb_used_data":27464,"kb_used_omap":12,"kb_used_meta":435827,"kb_avail":754511300,"statfs":{"total":773094113280,"available":772619571200,"internally_reserved":0,"allocated":28123136,"data_stored":15077379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":446287416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.677624"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.520614+0000","last_change":"2024-04-06T13:49:17.516210+0000","last_active":"2024-04-06T13:49:17.520614+0000","last_peered":"2024-04-06T13:49:17.520614+0000","last_clean":"2024-04-06T13:49:17.520614+0000","last_became_active":"2024-04-06T13:49:17.515918+0000","last_became_peered":"2024-04-06T13:49:17.515918+0000","last_unstale":"2024-04-06T13:49:17.520614+0000","last_undegraded":"2024-04-06T13:49:17.520614+0000","last_fullsized":"2024-04-06T13:49:17.520614+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:36:00.408855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522243+0000","last_change":"2024-04-06T13:49:17.519084+0000","last_active":"2024-04-06T13:49:17.522243+0000","last_peered":"2024-04-06T13:49:17.522243+0000","last_clean":"2024-04-06T13:49:17.522243+0000","last_became_active":"2024-04-06T13:49:17.518820+0000","last_became_peered":"2024-04-06T13:49:17.518820+0000","last_unstale":"2024-04-06T13:49:17.522243+0000","last_undegraded":"2024-04-06T13:49:17.522243+0000","last_fullsized":"2024-04-06T13:49:17.522243+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:05:58.697947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.513364+0000","last_change":"2024-04-06T13:49:17.510525+0000","last_active":"2024-04-06T13:49:17.513364+0000","last_peered":"2024-04-06T13:49:17.513364+0000","last_clean":"2024-04-06T13:49:17.513364+0000","last_became_active":"2024-04-06T13:49:17.510094+0000","last_became_peered":"2024-04-06T13:49:17.510094+0000","last_unstale":"2024-04-06T13:49:17.513364+0000","last_undegraded":"2024-04-06T13:49:17.513364+0000","last_fullsized":"2024-04-06T13:49:17.513364+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:52:44.549832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.524977+0000","last_change":"2024-04-06T13:49:17.524610+0000","last_active":"2024-04-06T13:49:17.524977+0000","last_peered":"2024-04-06T13:49:17.524977+0000","last_clean":"2024-04-06T13:49:17.524977+0000","last_became_active":"2024-04-06T13:49:17.524396+0000","last_became_peered":"2024-04-06T13:49:17.524396+0000","last_unstale":"2024-04-06T13:49:17.524977+0000","last_undegraded":"2024-04-06T13:49:17.524977+0000","last_fullsized":"2024-04-06T13:49:17.524977+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:32:23.301371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514489+0000","last_change":"2024-04-06T13:49:17.508190+0000","last_active":"2024-04-06T13:49:17.514489+0000","last_peered":"2024-04-06T13:49:17.514489+0000","last_clean":"2024-04-06T13:49:17.514489+0000","last_became_active":"2024-04-06T13:49:17.507959+0000","last_became_peered":"2024-04-06T13:49:17.507959+0000","last_unstale":"2024-04-06T13:49:17.514489+0000","last_undegraded":"2024-04-06T13:49:17.514489+0000","last_fullsized":"2024-04-06T13:49:17.514489+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:40:45.826207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514078+0000","last_change":"2024-04-06T13:49:17.507071+0000","last_active":"2024-04-06T13:49:17.514078+0000","last_peered":"2024-04-06T13:49:17.514078+0000","last_clean":"2024-04-06T13:49:17.514078+0000","last_became_active":"2024-04-06T13:49:17.506737+0000","last_became_peered":"2024-04-06T13:49:17.506737+0000","last_unstale":"2024-04-06T13:49:17.514078+0000","last_undegraded":"2024-04-06T13:49:17.514078+0000","last_fullsized":"2024-04-06T13:49:17.514078+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:53:28.014667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519186+0000","last_change":"2024-04-06T13:49:17.515069+0000","last_active":"2024-04-06T13:49:17.519186+0000","last_peered":"2024-04-06T13:49:17.519186+0000","last_clean":"2024-04-06T13:49:17.519186+0000","last_became_active":"2024-04-06T13:49:17.506210+0000","last_became_peered":"2024-04-06T13:49:17.506210+0000","last_unstale":"2024-04-06T13:49:17.519186+0000","last_undegraded":"2024-04-06T13:49:17.519186+0000","last_fullsized":"2024-04-06T13:49:17.519186+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:14:21.286281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.521054+0000","last_change":"2024-04-06T13:49:17.520637+0000","last_active":"2024-04-06T13:49:17.521054+0000","last_peered":"2024-04-06T13:49:17.521054+0000","last_clean":"2024-04-06T13:49:17.521054+0000","last_became_active":"2024-04-06T13:49:17.520388+0000","last_became_peered":"2024-04-06T13:49:17.520388+0000","last_unstale":"2024-04-06T13:49:17.521054+0000","last_undegraded":"2024-04-06T13:49:17.521054+0000","last_fullsized":"2024-04-06T13:49:17.521054+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:12.068312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.516486+0000","last_change":"2024-04-06T13:49:17.506537+0000","last_active":"2024-04-06T13:49:17.516486+0000","last_peered":"2024-04-06T13:49:17.516486+0000","last_clean":"2024-04-06T13:49:17.516486+0000","last_became_active":"2024-04-06T13:49:17.506232+0000","last_became_peered":"2024-04-06T13:49:17.506232+0000","last_unstale":"2024-04-06T13:49:17.516486+0000","last_undegraded":"2024-04-06T13:49:17.516486+0000","last_fullsized":"2024-04-06T13:49:17.516486+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:02:05.602108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.520190+0000","last_change":"2024-04-06T13:49:17.517199+0000","last_active":"2024-04-06T13:49:17.520190+0000","last_peered":"2024-04-06T13:49:17.520190+0000","last_clean":"2024-04-06T13:49:17.520190+0000","last_became_active":"2024-04-06T13:49:17.516867+0000","last_became_peered":"2024-04-06T13:49:17.516867+0000","last_unstale":"2024-04-06T13:49:17.520190+0000","last_undegraded":"2024-04-06T13:49:17.520190+0000","last_fullsized":"2024-04-06T13:49:17.520190+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:10:03.724857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.517970+0000","last_change":"2024-04-06T13:49:17.515732+0000","last_active":"2024-04-06T13:49:17.517970+0000","last_peered":"2024-04-06T13:49:17.517970+0000","last_clean":"2024-04-06T13:49:17.517970+0000","last_became_active":"2024-04-06T13:49:17.515484+0000","last_became_peered":"2024-04-06T13:49:17.515484+0000","last_unstale":"2024-04-06T13:49:17.517970+0000","last_undegraded":"2024-04-06T13:49:17.517970+0000","last_fullsized":"2024-04-06T13:49:17.517970+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:49:29.329764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522789+0000","last_change":"2024-04-06T13:49:17.521398+0000","last_active":"2024-04-06T13:49:17.522789+0000","last_peered":"2024-04-06T13:49:17.522789+0000","last_clean":"2024-04-06T13:49:17.522789+0000","last_became_active":"2024-04-06T13:49:17.521122+0000","last_became_peered":"2024-04-06T13:49:17.521122+0000","last_unstale":"2024-04-06T13:49:17.522789+0000","last_undegraded":"2024-04-06T13:49:17.522789+0000","last_fullsized":"2024-04-06T13:49:17.522789+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:51:12.481758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523016+0000","last_change":"2024-04-06T13:49:17.521408+0000","last_active":"2024-04-06T13:49:17.523016+0000","last_peered":"2024-04-06T13:49:17.523016+0000","last_clean":"2024-04-06T13:49:17.523016+0000","last_became_active":"2024-04-06T13:49:17.521217+0000","last_became_peered":"2024-04-06T13:49:17.521217+0000","last_unstale":"2024-04-06T13:49:17.523016+0000","last_undegraded":"2024-04-06T13:49:17.523016+0000","last_fullsized":"2024-04-06T13:49:17.523016+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:18:13.304903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.524375+0000","last_change":"2024-04-06T13:49:17.518243+0000","last_active":"2024-04-06T13:49:17.524375+0000","last_peered":"2024-04-06T13:49:17.524375+0000","last_clean":"2024-04-06T13:49:17.524375+0000","last_became_active":"2024-04-06T13:49:17.516954+0000","last_became_peered":"2024-04-06T13:49:17.516954+0000","last_unstale":"2024-04-06T13:49:17.524375+0000","last_undegraded":"2024-04-06T13:49:17.524375+0000","last_fullsized":"2024-04-06T13:49:17.524375+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:36:20.518005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.517577+0000","last_change":"2024-04-06T13:49:17.512021+0000","last_active":"2024-04-06T13:49:17.517577+0000","last_peered":"2024-04-06T13:49:17.517577+0000","last_clean":"2024-04-06T13:49:17.517577+0000","last_became_active":"2024-04-06T13:49:17.511763+0000","last_became_peered":"2024-04-06T13:49:17.511763+0000","last_unstale":"2024-04-06T13:49:17.517577+0000","last_undegraded":"2024-04-06T13:49:17.517577+0000","last_fullsized":"2024-04-06T13:49:17.517577+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:12:49.230301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.512003+0000","last_change":"2024-04-06T13:49:17.508744+0000","last_active":"2024-04-06T13:49:17.512003+0000","last_peered":"2024-04-06T13:49:17.512003+0000","last_clean":"2024-04-06T13:49:17.512003+0000","last_became_active":"2024-04-06T13:49:17.508450+0000","last_became_peered":"2024-04-06T13:49:17.508450+0000","last_unstale":"2024-04-06T13:49:17.512003+0000","last_undegraded":"2024-04-06T13:49:17.512003+0000","last_fullsized":"2024-04-06T13:49:17.512003+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:12.068312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519816+0000","last_change":"2024-04-06T13:49:17.517174+0000","last_active":"2024-04-06T13:49:17.519816+0000","last_peered":"2024-04-06T13:49:17.519816+0000","last_clean":"2024-04-06T13:49:17.519816+0000","last_became_active":"2024-04-06T13:49:17.516215+0000","last_became_peered":"2024-04-06T13:49:17.516215+0000","last_unstale":"2024-04-06T13:49:17.519816+0000","last_undegraded":"2024-04-06T13:49:17.519816+0000","last_fullsized":"2024-04-06T13:49:17.519816+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:07.368454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.517476+0000","last_change":"2024-04-06T13:49:17.511953+0000","last_active":"2024-04-06T13:49:17.517476+0000","last_peered":"2024-04-06T13:49:17.517476+0000","last_clean":"2024-04-06T13:49:17.517476+0000","last_became_active":"2024-04-06T13:49:17.511661+0000","last_became_peered":"2024-04-06T13:49:17.511661+0000","last_unstale":"2024-04-06T13:49:17.517476+0000","last_undegraded":"2024-04-06T13:49:17.517476+0000","last_fullsized":"2024-04-06T13:49:17.517476+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:26:28.142149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522145+0000","last_change":"2024-04-06T13:49:17.521073+0000","last_active":"2024-04-06T13:49:17.522145+0000","last_peered":"2024-04-06T13:49:17.522145+0000","last_clean":"2024-04-06T13:49:17.522145+0000","last_became_active":"2024-04-06T13:49:17.520811+0000","last_became_peered":"2024-04-06T13:49:17.520811+0000","last_unstale":"2024-04-06T13:49:17.522145+0000","last_undegraded":"2024-04-06T13:49:17.522145+0000","last_fullsized":"2024-04-06T13:49:17.522145+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:17:06.173348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.510215+0000","last_change":"2024-04-06T13:49:17.507429+0000","last_active":"2024-04-06T13:49:17.510215+0000","last_peered":"2024-04-06T13:49:17.510215+0000","last_clean":"2024-04-06T13:49:17.510215+0000","last_became_active":"2024-04-06T13:49:17.507181+0000","last_became_peered":"2024-04-06T13:49:17.507181+0000","last_unstale":"2024-04-06T13:49:17.510215+0000","last_undegraded":"2024-04-06T13:49:17.510215+0000","last_fullsized":"2024-04-06T13:49:17.510215+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:32:23.301371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514185+0000","last_change":"2024-04-06T13:49:17.513611+0000","last_active":"2024-04-06T13:49:17.514185+0000","last_peered":"2024-04-06T13:49:17.514185+0000","last_clean":"2024-04-06T13:49:17.514185+0000","last_became_active":"2024-04-06T13:49:17.509723+0000","last_became_peered":"2024-04-06T13:49:17.509723+0000","last_unstale":"2024-04-06T13:49:17.514185+0000","last_undegraded":"2024-04-06T13:49:17.514185+0000","last_fullsized":"2024-04-06T13:49:17.514185+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:55:51.420328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.515396+0000","last_change":"2024-04-06T13:49:17.512230+0000","last_active":"2024-04-06T13:49:17.515396+0000","last_peered":"2024-04-06T13:49:17.515396+0000","last_clean":"2024-04-06T13:49:17.515396+0000","last_became_active":"2024-04-06T13:49:17.511986+0000","last_became_peered":"2024-04-06T13:49:17.511986+0000","last_unstale":"2024-04-06T13:49:17.515396+0000","last_undegraded":"2024-04-06T13:49:17.515396+0000","last_fullsized":"2024-04-06T13:49:17.515396+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:27:37.281369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.511808+0000","last_change":"2024-04-06T13:49:17.501401+0000","last_active":"2024-04-06T13:49:17.511808+0000","last_peered":"2024-04-06T13:49:17.511808+0000","last_clean":"2024-04-06T13:49:17.511808+0000","last_became_active":"2024-04-06T13:49:17.501139+0000","last_became_peered":"2024-04-06T13:49:17.501139+0000","last_unstale":"2024-04-06T13:49:17.511808+0000","last_undegraded":"2024-04-06T13:49:17.511808+0000","last_fullsized":"2024-04-06T13:49:17.511808+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:59:09.454807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523385+0000","last_change":"2024-04-06T13:49:17.522743+0000","last_active":"2024-04-06T13:49:17.523385+0000","last_peered":"2024-04-06T13:49:17.523385+0000","last_clean":"2024-04-06T13:49:17.523385+0000","last_became_active":"2024-04-06T13:49:17.522434+0000","last_became_peered":"2024-04-06T13:49:17.522434+0000","last_unstale":"2024-04-06T13:49:17.523385+0000","last_undegraded":"2024-04-06T13:49:17.523385+0000","last_fullsized":"2024-04-06T13:49:17.523385+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:36:30.017284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.520593+0000","last_change":"2024-04-06T13:49:17.512281+0000","last_active":"2024-04-06T13:49:17.520593+0000","last_peered":"2024-04-06T13:49:17.520593+0000","last_clean":"2024-04-06T13:49:17.520593+0000","last_became_active":"2024-04-06T13:49:17.511719+0000","last_became_peered":"2024-04-06T13:49:17.511719+0000","last_unstale":"2024-04-06T13:49:17.520593+0000","last_undegraded":"2024-04-06T13:49:17.520593+0000","last_fullsized":"2024-04-06T13:49:17.520593+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:27:32.583709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514836+0000","last_change":"2024-04-06T13:49:17.513918+0000","last_active":"2024-04-06T13:49:17.514836+0000","last_peered":"2024-04-06T13:49:17.514836+0000","last_clean":"2024-04-06T13:49:17.514836+0000","last_became_active":"2024-04-06T13:49:17.513687+0000","last_became_peered":"2024-04-06T13:49:17.513687+0000","last_unstale":"2024-04-06T13:49:17.514836+0000","last_undegraded":"2024-04-06T13:49:17.514836+0000","last_fullsized":"2024-04-06T13:49:17.514836+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:14:21.286281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514395+0000","last_change":"2024-04-06T13:49:17.507416+0000","last_active":"2024-04-06T13:49:17.514395+0000","last_peered":"2024-04-06T13:49:17.514395+0000","last_clean":"2024-04-06T13:49:17.514395+0000","last_became_active":"2024-04-06T13:49:17.507112+0000","last_became_peered":"2024-04-06T13:49:17.507112+0000","last_unstale":"2024-04-06T13:49:17.514395+0000","last_undegraded":"2024-04-06T13:49:17.514395+0000","last_fullsized":"2024-04-06T13:49:17.514395+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:05:36.115690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.510969+0000","last_change":"2024-04-06T13:49:17.506454+0000","last_active":"2024-04-06T13:49:17.510969+0000","last_peered":"2024-04-06T13:49:17.510969+0000","last_clean":"2024-04-06T13:49:17.510969+0000","last_became_active":"2024-04-06T13:49:17.505920+0000","last_became_peered":"2024-04-06T13:49:17.505920+0000","last_unstale":"2024-04-06T13:49:17.510969+0000","last_undegraded":"2024-04-06T13:49:17.510969+0000","last_fullsized":"2024-04-06T13:49:17.510969+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:51:06.786292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.521699+0000","last_change":"2024-04-06T13:49:17.514816+0000","last_active":"2024-04-06T13:49:17.521699+0000","last_peered":"2024-04-06T13:49:17.521699+0000","last_clean":"2024-04-06T13:49:17.521699+0000","last_became_active":"2024-04-06T13:49:17.514548+0000","last_became_peered":"2024-04-06T13:49:17.514548+0000","last_unstale":"2024-04-06T13:49:17.521699+0000","last_undegraded":"2024-04-06T13:49:17.521699+0000","last_fullsized":"2024-04-06T13:49:17.521699+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:44:33.017556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.510940+0000","last_change":"2024-04-06T13:49:17.506195+0000","last_active":"2024-04-06T13:49:17.510940+0000","last_peered":"2024-04-06T13:49:17.510940+0000","last_clean":"2024-04-06T13:49:17.510940+0000","last_became_active":"2024-04-06T13:49:17.505887+0000","last_became_peered":"2024-04-06T13:49:17.505887+0000","last_unstale":"2024-04-06T13:49:17.510940+0000","last_undegraded":"2024-04-06T13:49:17.510940+0000","last_fullsized":"2024-04-06T13:49:17.510940+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:16:38.686659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519769+0000","last_change":"2024-04-06T13:49:17.515234+0000","last_active":"2024-04-06T13:49:17.519769+0000","last_peered":"2024-04-06T13:49:17.519769+0000","last_clean":"2024-04-06T13:49:17.519769+0000","last_became_active":"2024-04-06T13:49:17.514959+0000","last_became_peered":"2024-04-06T13:49:17.514959+0000","last_unstale":"2024-04-06T13:49:17.519769+0000","last_undegraded":"2024-04-06T13:49:17.519769+0000","last_fullsized":"2024-04-06T13:49:17.519769+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:02:31.039549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523372+0000","last_change":"2024-04-06T13:49:17.522490+0000","last_active":"2024-04-06T13:49:17.523372+0000","last_peered":"2024-04-06T13:49:17.523372+0000","last_clean":"2024-04-06T13:49:17.523372+0000","last_became_active":"2024-04-06T13:49:17.522102+0000","last_became_peered":"2024-04-06T13:49:17.522102+0000","last_unstale":"2024-04-06T13:49:17.523372+0000","last_undegraded":"2024-04-06T13:49:17.523372+0000","last_fullsized":"2024-04-06T13:49:17.523372+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:12:09.850584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"268'4","reported_seq":14,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523550+0000","last_change":"2024-04-06T13:49:17.096966+0000","last_active":"2024-04-06T13:49:17.523550+0000","last_peered":"2024-04-06T13:49:17.523550+0000","last_clean":"2024-04-06T13:49:17.523550+0000","last_became_active":"2024-04-06T13:49:17.096695+0000","last_became_peered":"2024-04-06T13:49:17.096695+0000","last_unstale":"2024-04-06T13:49:17.523550+0000","last_undegraded":"2024-04-06T13:49:17.523550+0000","last_fullsized":"2024-04-06T13:49:17.523550+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:28:50.074433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.525127+0000","last_change":"2024-04-06T13:49:17.523240+0000","last_active":"2024-04-06T13:49:17.525127+0000","last_peered":"2024-04-06T13:49:17.525127+0000","last_clean":"2024-04-06T13:49:17.525127+0000","last_became_active":"2024-04-06T13:49:17.522980+0000","last_became_peered":"2024-04-06T13:49:17.522980+0000","last_unstale":"2024-04-06T13:49:17.525127+0000","last_undegraded":"2024-04-06T13:49:17.525127+0000","last_fullsized":"2024-04-06T13:49:17.525127+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:38:33.450798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"268'1","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522994+0000","last_change":"2024-04-06T13:49:17.098551+0000","last_active":"2024-04-06T13:49:17.522994+0000","last_peered":"2024-04-06T13:49:17.522994+0000","last_clean":"2024-04-06T13:49:17.522994+0000","last_became_active":"2024-04-06T13:49:17.098354+0000","last_became_peered":"2024-04-06T13:49:17.098354+0000","last_unstale":"2024-04-06T13:49:17.522994+0000","last_undegraded":"2024-04-06T13:49:17.522994+0000","last_fullsized":"2024-04-06T13:49:17.522994+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:43:16.510351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522212+0000","last_change":"2024-04-06T13:49:17.518725+0000","last_active":"2024-04-06T13:49:17.522212+0000","last_peered":"2024-04-06T13:49:17.522212+0000","last_clean":"2024-04-06T13:49:17.522212+0000","last_became_active":"2024-04-06T13:49:17.518434+0000","last_became_peered":"2024-04-06T13:49:17.518434+0000","last_unstale":"2024-04-06T13:49:17.522212+0000","last_undegraded":"2024-04-06T13:49:17.522212+0000","last_fullsized":"2024-04-06T13:49:17.522212+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:25:59.367813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1d","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522514+0000","last_change":"2024-04-06T13:49:17.104889+0000","last_active":"2024-04-06T13:49:17.522514+0000","last_peered":"2024-04-06T13:49:17.522514+0000","last_clean":"2024-04-06T13:49:17.522514+0000","last_became_active":"2024-04-06T13:49:17.104619+0000","last_became_peered":"2024-04-06T13:49:17.104619+0000","last_unstale":"2024-04-06T13:49:17.522514+0000","last_undegraded":"2024-04-06T13:49:17.522514+0000","last_fullsized":"2024-04-06T13:49:17.522514+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:09:00.021457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.515331+0000","last_change":"2024-04-06T13:49:17.503291+0000","last_active":"2024-04-06T13:49:17.515331+0000","last_peered":"2024-04-06T13:49:17.515331+0000","last_clean":"2024-04-06T13:49:17.515331+0000","last_became_active":"2024-04-06T13:49:17.503011+0000","last_became_peered":"2024-04-06T13:49:17.503011+0000","last_unstale":"2024-04-06T13:49:17.515331+0000","last_undegraded":"2024-04-06T13:49:17.515331+0000","last_fullsized":"2024-04-06T13:49:17.515331+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:09.947236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522373+0000","last_change":"2024-04-06T13:49:17.105027+0000","last_active":"2024-04-06T13:49:17.522373+0000","last_peered":"2024-04-06T13:49:17.522373+0000","last_clean":"2024-04-06T13:49:17.522373+0000","last_became_active":"2024-04-06T13:49:17.104656+0000","last_became_peered":"2024-04-06T13:49:17.104656+0000","last_unstale":"2024-04-06T13:49:17.522373+0000","last_undegraded":"2024-04-06T13:49:17.522373+0000","last_fullsized":"2024-04-06T13:49:17.522373+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:15:00.333710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.510882+0000","last_change":"2024-04-06T13:49:17.510173+0000","last_active":"2024-04-06T13:49:17.510882+0000","last_peered":"2024-04-06T13:49:17.510882+0000","last_clean":"2024-04-06T13:49:17.510882+0000","last_became_active":"2024-04-06T13:49:17.509947+0000","last_became_peered":"2024-04-06T13:49:17.509947+0000","last_unstale":"2024-04-06T13:49:17.510882+0000","last_undegraded":"2024-04-06T13:49:17.510882+0000","last_fullsized":"2024-04-06T13:49:17.510882+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:59:33.036233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1b","version":"268'1","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.520763+0000","last_change":"2024-04-06T13:49:17.097606+0000","last_active":"2024-04-06T13:49:17.520763+0000","last_peered":"2024-04-06T13:49:17.520763+0000","last_clean":"2024-04-06T13:49:17.520763+0000","last_became_active":"2024-04-06T13:49:17.097312+0000","last_became_peered":"2024-04-06T13:49:17.097312+0000","last_unstale":"2024-04-06T13:49:17.520763+0000","last_undegraded":"2024-04-06T13:49:17.520763+0000","last_fullsized":"2024-04-06T13:49:17.520763+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:35.789803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.518610+0000","last_change":"2024-04-06T13:49:17.517731+0000","last_active":"2024-04-06T13:49:17.518610+0000","last_peered":"2024-04-06T13:49:17.518610+0000","last_clean":"2024-04-06T13:49:17.518610+0000","last_became_active":"2024-04-06T13:49:17.517330+0000","last_became_peered":"2024-04-06T13:49:17.517330+0000","last_unstale":"2024-04-06T13:49:17.518610+0000","last_undegraded":"2024-04-06T13:49:17.518610+0000","last_fullsized":"2024-04-06T13:49:17.518610+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:27:55.678136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1a","version":"268'1","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.524674+0000","last_change":"2024-04-06T13:49:17.093102+0000","last_active":"2024-04-06T13:49:17.524674+0000","last_peered":"2024-04-06T13:49:17.524674+0000","last_clean":"2024-04-06T13:49:17.524674+0000","last_became_active":"2024-04-06T13:49:17.092879+0000","last_became_peered":"2024-04-06T13:49:17.092879+0000","last_unstale":"2024-04-06T13:49:17.524674+0000","last_undegraded":"2024-04-06T13:49:17.524674+0000","last_fullsized":"2024-04-06T13:49:17.524674+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:54:48.687945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.515261+0000","last_change":"2024-04-06T13:49:17.512942+0000","last_active":"2024-04-06T13:49:17.515261+0000","last_peered":"2024-04-06T13:49:17.515261+0000","last_clean":"2024-04-06T13:49:17.515261+0000","last_became_active":"2024-04-06T13:49:17.512730+0000","last_became_peered":"2024-04-06T13:49:17.512730+0000","last_unstale":"2024-04-06T13:49:17.515261+0000","last_undegraded":"2024-04-06T13:49:17.515261+0000","last_fullsized":"2024-04-06T13:49:17.515261+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:16:45.905450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.501191+0000","last_change":"2024-04-06T13:49:17.086842+0000","last_active":"2024-04-06T13:49:17.501191+0000","last_peered":"2024-04-06T13:49:17.501191+0000","last_clean":"2024-04-06T13:49:17.501191+0000","last_became_active":"2024-04-06T13:49:17.086555+0000","last_became_peered":"2024-04-06T13:49:17.086555+0000","last_unstale":"2024-04-06T13:49:17.501191+0000","last_undegraded":"2024-04-06T13:49:17.501191+0000","last_fullsized":"2024-04-06T13:49:17.501191+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:05:57.696105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519835+0000","last_change":"2024-04-06T13:49:17.517335+0000","last_active":"2024-04-06T13:49:17.519835+0000","last_peered":"2024-04-06T13:49:17.519835+0000","last_clean":"2024-04-06T13:49:17.519835+0000","last_became_active":"2024-04-06T13:49:17.516574+0000","last_became_peered":"2024-04-06T13:49:17.516574+0000","last_unstale":"2024-04-06T13:49:17.519835+0000","last_undegraded":"2024-04-06T13:49:17.519835+0000","last_fullsized":"2024-04-06T13:49:17.519835+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:44:33.017556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.510632+0000","last_change":"2024-04-06T13:49:17.101047+0000","last_active":"2024-04-06T13:49:17.510632+0000","last_peered":"2024-04-06T13:49:17.510632+0000","last_clean":"2024-04-06T13:49:17.510632+0000","last_became_active":"2024-04-06T13:49:17.100809+0000","last_became_peered":"2024-04-06T13:49:17.100809+0000","last_unstale":"2024-04-06T13:49:17.510632+0000","last_undegraded":"2024-04-06T13:49:17.510632+0000","last_fullsized":"2024-04-06T13:49:17.510632+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:17:59.126763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523051+0000","last_change":"2024-04-06T13:49:17.521760+0000","last_active":"2024-04-06T13:49:17.523051+0000","last_peered":"2024-04-06T13:49:17.523051+0000","last_clean":"2024-04-06T13:49:17.523051+0000","last_became_active":"2024-04-06T13:49:17.521593+0000","last_became_peered":"2024-04-06T13:49:17.521593+0000","last_unstale":"2024-04-06T13:49:17.523051+0000","last_undegraded":"2024-04-06T13:49:17.523051+0000","last_fullsized":"2024-04-06T13:49:17.523051+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:50.418088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.504021+0000","last_change":"2024-04-06T13:49:17.102458+0000","last_active":"2024-04-06T13:49:17.504021+0000","last_peered":"2024-04-06T13:49:17.504021+0000","last_clean":"2024-04-06T13:49:17.504021+0000","last_became_active":"2024-04-06T13:49:17.101510+0000","last_became_peered":"2024-04-06T13:49:17.101510+0000","last_unstale":"2024-04-06T13:49:17.504021+0000","last_undegraded":"2024-04-06T13:49:17.504021+0000","last_fullsized":"2024-04-06T13:49:17.504021+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:28:43.059891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.520130+0000","last_change":"2024-04-06T13:49:17.516948+0000","last_active":"2024-04-06T13:49:17.520130+0000","last_peered":"2024-04-06T13:49:17.520130+0000","last_clean":"2024-04-06T13:49:17.520130+0000","last_became_active":"2024-04-06T13:49:17.516687+0000","last_became_peered":"2024-04-06T13:49:17.516687+0000","last_unstale":"2024-04-06T13:49:17.520130+0000","last_undegraded":"2024-04-06T13:49:17.520130+0000","last_fullsized":"2024-04-06T13:49:17.520130+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:44:33.805313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.501644+0000","last_change":"2024-04-06T13:49:17.086910+0000","last_active":"2024-04-06T13:49:17.501644+0000","last_peered":"2024-04-06T13:49:17.501644+0000","last_clean":"2024-04-06T13:49:17.501644+0000","last_became_active":"2024-04-06T13:49:17.086616+0000","last_became_peered":"2024-04-06T13:49:17.086616+0000","last_unstale":"2024-04-06T13:49:17.501644+0000","last_undegraded":"2024-04-06T13:49:17.501644+0000","last_fullsized":"2024-04-06T13:49:17.501644+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:44:32.803471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519799+0000","last_change":"2024-04-06T13:49:17.515970+0000","last_active":"2024-04-06T13:49:17.519799+0000","last_peered":"2024-04-06T13:49:17.519799+0000","last_clean":"2024-04-06T13:49:17.519799+0000","last_became_active":"2024-04-06T13:49:17.515734+0000","last_became_peered":"2024-04-06T13:49:17.515734+0000","last_unstale":"2024-04-06T13:49:17.519799+0000","last_undegraded":"2024-04-06T13:49:17.519799+0000","last_fullsized":"2024-04-06T13:49:17.519799+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:05:52.789141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"268'3","reported_seq":13,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.515626+0000","last_change":"2024-04-06T13:49:17.100609+0000","last_active":"2024-04-06T13:49:17.515626+0000","last_peered":"2024-04-06T13:49:17.515626+0000","last_clean":"2024-04-06T13:49:17.515626+0000","last_became_active":"2024-04-06T13:49:17.100323+0000","last_became_peered":"2024-04-06T13:49:17.100323+0000","last_unstale":"2024-04-06T13:49:17.515626+0000","last_undegraded":"2024-04-06T13:49:17.515626+0000","last_fullsized":"2024-04-06T13:49:17.515626+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:35:59.407013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514727+0000","last_change":"2024-04-06T13:49:17.512077+0000","last_active":"2024-04-06T13:49:17.514727+0000","last_peered":"2024-04-06T13:49:17.514727+0000","last_clean":"2024-04-06T13:49:17.514727+0000","last_became_active":"2024-04-06T13:49:17.511254+0000","last_became_peered":"2024-04-06T13:49:17.511254+0000","last_unstale":"2024-04-06T13:49:17.514727+0000","last_undegraded":"2024-04-06T13:49:17.514727+0000","last_fullsized":"2024-04-06T13:49:17.514727+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:44:10.089826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"268'6","reported_seq":16,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.526636+0000","last_change":"2024-04-06T13:49:17.105013+0000","last_active":"2024-04-06T13:49:17.526636+0000","last_peered":"2024-04-06T13:49:17.526636+0000","last_clean":"2024-04-06T13:49:17.526636+0000","last_became_active":"2024-04-06T13:49:17.104721+0000","last_became_peered":"2024-04-06T13:49:17.104721+0000","last_unstale":"2024-04-06T13:49:17.526636+0000","last_undegraded":"2024-04-06T13:49:17.526636+0000","last_fullsized":"2024-04-06T13:49:17.526636+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:11:07.791203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522267+0000","last_change":"2024-04-06T13:49:17.519398+0000","last_active":"2024-04-06T13:49:17.522267+0000","last_peered":"2024-04-06T13:49:17.522267+0000","last_clean":"2024-04-06T13:49:17.522267+0000","last_became_active":"2024-04-06T13:49:17.516546+0000","last_became_peered":"2024-04-06T13:49:17.516546+0000","last_unstale":"2024-04-06T13:49:17.522267+0000","last_undegraded":"2024-04-06T13:49:17.522267+0000","last_fullsized":"2024-04-06T13:49:17.522267+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:28:44.061733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523061+0000","last_change":"2024-04-06T13:49:17.105019+0000","last_active":"2024-04-06T13:49:17.523061+0000","last_peered":"2024-04-06T13:49:17.523061+0000","last_clean":"2024-04-06T13:49:17.523061+0000","last_became_active":"2024-04-06T13:49:17.104653+0000","last_became_peered":"2024-04-06T13:49:17.104653+0000","last_unstale":"2024-04-06T13:49:17.523061+0000","last_undegraded":"2024-04-06T13:49:17.523061+0000","last_fullsized":"2024-04-06T13:49:17.523061+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:02:30.037708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519988+0000","last_change":"2024-04-06T13:49:17.513786+0000","last_active":"2024-04-06T13:49:17.519988+0000","last_peered":"2024-04-06T13:49:17.519988+0000","last_clean":"2024-04-06T13:49:17.519988+0000","last_became_active":"2024-04-06T13:49:17.513533+0000","last_became_peered":"2024-04-06T13:49:17.513533+0000","last_unstale":"2024-04-06T13:49:17.519988+0000","last_undegraded":"2024-04-06T13:49:17.519988+0000","last_fullsized":"2024-04-06T13:49:17.519988+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:14:07.368454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514604+0000","last_change":"2024-04-06T13:49:17.098866+0000","last_active":"2024-04-06T13:49:17.514604+0000","last_peered":"2024-04-06T13:49:17.514604+0000","last_clean":"2024-04-06T13:49:17.514604+0000","last_became_active":"2024-04-06T13:49:17.098587+0000","last_became_peered":"2024-04-06T13:49:17.098587+0000","last_unstale":"2024-04-06T13:49:17.514604+0000","last_undegraded":"2024-04-06T13:49:17.514604+0000","last_fullsized":"2024-04-06T13:49:17.514604+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:17:32.289853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519633+0000","last_change":"2024-04-06T13:49:17.517876+0000","last_active":"2024-04-06T13:49:17.519633+0000","last_peered":"2024-04-06T13:49:17.519633+0000","last_clean":"2024-04-06T13:49:17.519633+0000","last_became_active":"2024-04-06T13:49:17.517610+0000","last_became_peered":"2024-04-06T13:49:17.517610+0000","last_unstale":"2024-04-06T13:49:17.519633+0000","last_undegraded":"2024-04-06T13:49:17.519633+0000","last_fullsized":"2024-04-06T13:49:17.519633+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:50.418088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.512956+0000","last_change":"2024-04-06T13:49:17.094349+0000","last_active":"2024-04-06T13:49:17.512956+0000","last_peered":"2024-04-06T13:49:17.512956+0000","last_clean":"2024-04-06T13:49:17.512956+0000","last_became_active":"2024-04-06T13:49:17.094054+0000","last_became_peered":"2024-04-06T13:49:17.094054+0000","last_unstale":"2024-04-06T13:49:17.512956+0000","last_undegraded":"2024-04-06T13:49:17.512956+0000","last_fullsized":"2024-04-06T13:49:17.512956+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:17:59.126763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519850+0000","last_change":"2024-04-06T13:49:17.517953+0000","last_active":"2024-04-06T13:49:17.519850+0000","last_peered":"2024-04-06T13:49:17.519850+0000","last_clean":"2024-04-06T13:49:17.519850+0000","last_became_active":"2024-04-06T13:49:17.516926+0000","last_became_peered":"2024-04-06T13:49:17.516926+0000","last_unstale":"2024-04-06T13:49:17.519850+0000","last_undegraded":"2024-04-06T13:49:17.519850+0000","last_fullsized":"2024-04-06T13:49:17.519850+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:26:28.142149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"268'8","reported_seq":18,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.525370+0000","last_change":"2024-04-06T13:49:17.093692+0000","last_active":"2024-04-06T13:49:17.525370+0000","last_peered":"2024-04-06T13:49:17.525370+0000","last_clean":"2024-04-06T13:49:17.525370+0000","last_became_active":"2024-04-06T13:49:17.093394+0000","last_became_peered":"2024-04-06T13:49:17.093394+0000","last_unstale":"2024-04-06T13:49:17.525370+0000","last_undegraded":"2024-04-06T13:49:17.525370+0000","last_fullsized":"2024-04-06T13:49:17.525370+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:10.515169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514564+0000","last_change":"2024-04-06T13:49:17.507798+0000","last_active":"2024-04-06T13:49:17.514564+0000","last_peered":"2024-04-06T13:49:17.514564+0000","last_clean":"2024-04-06T13:49:17.514564+0000","last_became_active":"2024-04-06T13:49:17.507537+0000","last_became_peered":"2024-04-06T13:49:17.507537+0000","last_unstale":"2024-04-06T13:49:17.514564+0000","last_undegraded":"2024-04-06T13:49:17.514564+0000","last_fullsized":"2024-04-06T13:49:17.514564+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:08:18.059036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"268'1","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522825+0000","last_change":"2024-04-06T13:49:17.088149+0000","last_active":"2024-04-06T13:49:17.522825+0000","last_peered":"2024-04-06T13:49:17.522825+0000","last_clean":"2024-04-06T13:49:17.522825+0000","last_became_active":"2024-04-06T13:49:17.086151+0000","last_became_peered":"2024-04-06T13:49:17.086151+0000","last_unstale":"2024-04-06T13:49:17.522825+0000","last_undegraded":"2024-04-06T13:49:17.522825+0000","last_fullsized":"2024-04-06T13:49:17.522825+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:35:20.802113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523020+0000","last_change":"2024-04-06T13:49:17.522315+0000","last_active":"2024-04-06T13:49:17.523020+0000","last_peered":"2024-04-06T13:49:17.523020+0000","last_clean":"2024-04-06T13:49:17.523020+0000","last_became_active":"2024-04-06T13:49:17.522060+0000","last_became_peered":"2024-04-06T13:49:17.522060+0000","last_unstale":"2024-04-06T13:49:17.523020+0000","last_undegraded":"2024-04-06T13:49:17.523020+0000","last_fullsized":"2024-04-06T13:49:17.523020+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:32:51.983606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.e","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519127+0000","last_change":"2024-04-06T13:49:17.100106+0000","last_active":"2024-04-06T13:49:17.519127+0000","last_peered":"2024-04-06T13:49:17.519127+0000","last_clean":"2024-04-06T13:49:17.519127+0000","last_became_active":"2024-04-06T13:49:17.099832+0000","last_became_peered":"2024-04-06T13:49:17.099832+0000","last_unstale":"2024-04-06T13:49:17.519127+0000","last_undegraded":"2024-04-06T13:49:17.519127+0000","last_fullsized":"2024-04-06T13:49:17.519127+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:00:45.856554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514621+0000","last_change":"2024-04-06T13:49:17.507897+0000","last_active":"2024-04-06T13:49:17.514621+0000","last_peered":"2024-04-06T13:49:17.514621+0000","last_clean":"2024-04-06T13:49:17.514621+0000","last_became_active":"2024-04-06T13:49:17.507580+0000","last_became_peered":"2024-04-06T13:49:17.507580+0000","last_unstale":"2024-04-06T13:49:17.514621+0000","last_undegraded":"2024-04-06T13:49:17.514621+0000","last_fullsized":"2024-04-06T13:49:17.514621+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:33:54.132584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"268'1","reported_seq":11,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.525068+0000","last_change":"2024-04-06T13:49:17.087553+0000","last_active":"2024-04-06T13:49:17.525068+0000","last_peered":"2024-04-06T13:49:17.525068+0000","last_clean":"2024-04-06T13:49:17.525068+0000","last_became_active":"2024-04-06T13:49:17.086381+0000","last_became_peered":"2024-04-06T13:49:17.086381+0000","last_unstale":"2024-04-06T13:49:17.525068+0000","last_undegraded":"2024-04-06T13:49:17.525068+0000","last_fullsized":"2024-04-06T13:49:17.525068+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:54:55.120647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.518675+0000","last_change":"2024-04-06T13:49:17.513256+0000","last_active":"2024-04-06T13:49:17.518675+0000","last_peered":"2024-04-06T13:49:17.518675+0000","last_clean":"2024-04-06T13:49:17.518675+0000","last_became_active":"2024-04-06T13:49:17.511402+0000","last_became_peered":"2024-04-06T13:49:17.511402+0000","last_unstale":"2024-04-06T13:49:17.518675+0000","last_undegraded":"2024-04-06T13:49:17.518675+0000","last_fullsized":"2024-04-06T13:49:17.518675+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:58:35.003812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.501189+0000","last_change":"2024-04-06T13:49:17.105686+0000","last_active":"2024-04-06T13:49:17.501189+0000","last_peered":"2024-04-06T13:49:17.501189+0000","last_clean":"2024-04-06T13:49:17.501189+0000","last_became_active":"2024-04-06T13:49:17.105408+0000","last_became_peered":"2024-04-06T13:49:17.105408+0000","last_unstale":"2024-04-06T13:49:17.501189+0000","last_undegraded":"2024-04-06T13:49:17.501189+0000","last_fullsized":"2024-04-06T13:49:17.501189+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:52:43.547990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.515517+0000","last_change":"2024-04-06T13:49:17.514716+0000","last_active":"2024-04-06T13:49:17.515517+0000","last_peered":"2024-04-06T13:49:17.515517+0000","last_clean":"2024-04-06T13:49:17.515517+0000","last_became_active":"2024-04-06T13:49:17.514456+0000","last_became_peered":"2024-04-06T13:49:17.514456+0000","last_unstale":"2024-04-06T13:49:17.515517+0000","last_undegraded":"2024-04-06T13:49:17.515517+0000","last_fullsized":"2024-04-06T13:49:17.515517+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:01:01.224352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523098+0000","last_change":"2024-04-06T13:49:17.091792+0000","last_active":"2024-04-06T13:49:17.523098+0000","last_peered":"2024-04-06T13:49:17.523098+0000","last_clean":"2024-04-06T13:49:17.523098+0000","last_became_active":"2024-04-06T13:49:17.091444+0000","last_became_peered":"2024-04-06T13:49:17.091444+0000","last_unstale":"2024-04-06T13:49:17.523098+0000","last_undegraded":"2024-04-06T13:49:17.523098+0000","last_fullsized":"2024-04-06T13:49:17.523098+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:59:32.034392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.514941+0000","last_change":"2024-04-06T13:49:17.509208+0000","last_active":"2024-04-06T13:49:17.514941+0000","last_peered":"2024-04-06T13:49:17.514941+0000","last_clean":"2024-04-06T13:49:17.514941+0000","last_became_active":"2024-04-06T13:49:17.508894+0000","last_became_peered":"2024-04-06T13:49:17.508894+0000","last_unstale":"2024-04-06T13:49:17.514941+0000","last_undegraded":"2024-04-06T13:49:17.514941+0000","last_fullsized":"2024-04-06T13:49:17.514941+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:51:52.963672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.506787+0000","last_change":"2024-04-06T13:49:17.108306+0000","last_active":"2024-04-06T13:49:17.506787+0000","last_peered":"2024-04-06T13:49:17.506787+0000","last_clean":"2024-04-06T13:49:17.506787+0000","last_became_active":"2024-04-06T13:49:17.108064+0000","last_became_peered":"2024-04-06T13:49:17.108064+0000","last_unstale":"2024-04-06T13:49:17.506787+0000","last_undegraded":"2024-04-06T13:49:17.506787+0000","last_fullsized":"2024-04-06T13:49:17.506787+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:32:53.801190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519613+0000","last_change":"2024-04-06T13:49:17.517755+0000","last_active":"2024-04-06T13:49:17.519613+0000","last_peered":"2024-04-06T13:49:17.519613+0000","last_clean":"2024-04-06T13:49:17.519613+0000","last_became_active":"2024-04-06T13:49:17.517262+0000","last_became_peered":"2024-04-06T13:49:17.517262+0000","last_unstale":"2024-04-06T13:49:17.519613+0000","last_undegraded":"2024-04-06T13:49:17.519613+0000","last_fullsized":"2024-04-06T13:49:17.519613+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:18:13.304903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.497595+0000","last_change":"2024-04-06T13:49:17.103918+0000","last_active":"2024-04-06T13:49:17.497595+0000","last_peered":"2024-04-06T13:49:17.497595+0000","last_clean":"2024-04-06T13:49:17.497595+0000","last_became_active":"2024-04-06T13:49:17.103733+0000","last_became_peered":"2024-04-06T13:49:17.103733+0000","last_unstale":"2024-04-06T13:49:17.497595+0000","last_undegraded":"2024-04-06T13:49:17.497595+0000","last_fullsized":"2024-04-06T13:49:17.497595+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:10:02.723015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.513040+0000","last_change":"2024-04-06T13:49:17.506938+0000","last_active":"2024-04-06T13:49:17.513040+0000","last_peered":"2024-04-06T13:49:17.513040+0000","last_clean":"2024-04-06T13:49:17.513040+0000","last_became_active":"2024-04-06T13:49:17.506692+0000","last_became_peered":"2024-04-06T13:49:17.506692+0000","last_unstale":"2024-04-06T13:49:17.513040+0000","last_undegraded":"2024-04-06T13:49:17.513040+0000","last_fullsized":"2024-04-06T13:49:17.513040+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:10:53.815420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.501228+0000","last_change":"2024-04-06T13:49:17.105401+0000","last_active":"2024-04-06T13:49:17.501228+0000","last_peered":"2024-04-06T13:49:17.501228+0000","last_clean":"2024-04-06T13:49:17.501228+0000","last_became_active":"2024-04-06T13:49:17.104938+0000","last_became_peered":"2024-04-06T13:49:17.104938+0000","last_unstale":"2024-04-06T13:49:17.501228+0000","last_undegraded":"2024-04-06T13:49:17.501228+0000","last_fullsized":"2024-04-06T13:49:17.501228+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:15:28.868445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523071+0000","last_change":"2024-04-06T13:49:17.521772+0000","last_active":"2024-04-06T13:49:17.523071+0000","last_peered":"2024-04-06T13:49:17.523071+0000","last_clean":"2024-04-06T13:49:17.523071+0000","last_became_active":"2024-04-06T13:49:17.521481+0000","last_became_peered":"2024-04-06T13:49:17.521481+0000","last_unstale":"2024-04-06T13:49:17.523071+0000","last_undegraded":"2024-04-06T13:49:17.523071+0000","last_fullsized":"2024-04-06T13:49:17.523071+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:28:44.061733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.7","version":"268'2","reported_seq":12,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.521896+0000","last_change":"2024-04-06T13:49:17.097613+0000","last_active":"2024-04-06T13:49:17.521896+0000","last_peered":"2024-04-06T13:49:17.521896+0000","last_clean":"2024-04-06T13:49:17.521896+0000","last_became_active":"2024-04-06T13:49:17.097322+0000","last_became_peered":"2024-04-06T13:49:17.097322+0000","last_unstale":"2024-04-06T13:49:17.521896+0000","last_undegraded":"2024-04-06T13:49:17.521896+0000","last_fullsized":"2024-04-06T13:49:17.521896+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:58:34.001970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.511979+0000","last_change":"2024-04-06T13:49:17.509489+0000","last_active":"2024-04-06T13:49:17.511979+0000","last_peered":"2024-04-06T13:49:17.511979+0000","last_clean":"2024-04-06T13:49:17.511979+0000","last_became_active":"2024-04-06T13:49:17.509263+0000","last_became_peered":"2024-04-06T13:49:17.509263+0000","last_unstale":"2024-04-06T13:49:17.511979+0000","last_undegraded":"2024-04-06T13:49:17.511979+0000","last_fullsized":"2024-04-06T13:49:17.511979+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:12:49.230301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.510779+0000","last_change":"2024-04-06T13:49:17.098847+0000","last_active":"2024-04-06T13:49:17.510779+0000","last_peered":"2024-04-06T13:49:17.510779+0000","last_clean":"2024-04-06T13:49:17.510779+0000","last_became_active":"2024-04-06T13:49:17.098539+0000","last_became_peered":"2024-04-06T13:49:17.098539+0000","last_unstale":"2024-04-06T13:49:17.510779+0000","last_undegraded":"2024-04-06T13:49:17.510779+0000","last_fullsized":"2024-04-06T13:49:17.510779+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:17:04.844091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.523083+0000","last_change":"2024-04-06T13:49:17.521904+0000","last_active":"2024-04-06T13:49:17.523083+0000","last_peered":"2024-04-06T13:49:17.523083+0000","last_clean":"2024-04-06T13:49:17.523083+0000","last_became_active":"2024-04-06T13:49:17.521642+0000","last_became_peered":"2024-04-06T13:49:17.521642+0000","last_unstale":"2024-04-06T13:49:17.523083+0000","last_undegraded":"2024-04-06T13:49:17.523083+0000","last_fullsized":"2024-04-06T13:49:17.523083+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:47:54.488793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.499034+0000","last_change":"2024-04-06T13:49:17.101607+0000","last_active":"2024-04-06T13:49:17.499034+0000","last_peered":"2024-04-06T13:49:17.499034+0000","last_clean":"2024-04-06T13:49:17.499034+0000","last_became_active":"2024-04-06T13:49:17.101311+0000","last_became_peered":"2024-04-06T13:49:17.101311+0000","last_unstale":"2024-04-06T13:49:17.499034+0000","last_undegraded":"2024-04-06T13:49:17.499034+0000","last_fullsized":"2024-04-06T13:49:17.499034+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:25:27.198338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.519622+0000","last_change":"2024-04-06T13:49:17.506594+0000","last_active":"2024-04-06T13:49:17.519622+0000","last_peered":"2024-04-06T13:49:17.519622+0000","last_clean":"2024-04-06T13:49:17.519622+0000","last_became_active":"2024-04-06T13:49:17.503484+0000","last_became_peered":"2024-04-06T13:49:17.503484+0000","last_unstale":"2024-04-06T13:49:17.519622+0000","last_undegraded":"2024-04-06T13:49:17.519622+0000","last_fullsized":"2024-04-06T13:49:17.519622+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:05:36.115690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.511185+0000","last_change":"2024-04-06T13:49:17.094864+0000","last_active":"2024-04-06T13:49:17.511185+0000","last_peered":"2024-04-06T13:49:17.511185+0000","last_clean":"2024-04-06T13:49:17.511185+0000","last_became_active":"2024-04-06T13:49:17.094507+0000","last_became_peered":"2024-04-06T13:49:17.094507+0000","last_unstale":"2024-04-06T13:49:17.511185+0000","last_undegraded":"2024-04-06T13:49:17.511185+0000","last_fullsized":"2024-04-06T13:49:17.511185+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:32:53.801190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.522817+0000","last_change":"2024-04-06T13:49:17.500686+0000","last_active":"2024-04-06T13:49:17.522817+0000","last_peered":"2024-04-06T13:49:17.522817+0000","last_clean":"2024-04-06T13:49:17.522817+0000","last_became_active":"2024-04-06T13:49:17.500455+0000","last_became_peered":"2024-04-06T13:49:17.500455+0000","last_unstale":"2024-04-06T13:49:17.522817+0000","last_undegraded":"2024-04-06T13:49:17.522817+0000","last_fullsized":"2024-04-06T13:49:17.522817+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:43:17.512192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.501433+0000","last_change":"2024-04-06T13:49:17.097515+0000","last_active":"2024-04-06T13:49:17.501433+0000","last_peered":"2024-04-06T13:49:17.501433+0000","last_clean":"2024-04-06T13:49:17.501433+0000","last_became_active":"2024-04-06T13:49:17.097211+0000","last_became_peered":"2024-04-06T13:49:17.097211+0000","last_unstale":"2024-04-06T13:49:17.501433+0000","last_undegraded":"2024-04-06T13:49:17.501433+0000","last_fullsized":"2024-04-06T13:49:17.501433+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:00:54.806750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.520547+0000","last_change":"2024-04-06T13:49:17.518763+0000","last_active":"2024-04-06T13:49:17.520547+0000","last_peered":"2024-04-06T13:49:17.520547+0000","last_clean":"2024-04-06T13:49:17.520547+0000","last_became_active":"2024-04-06T13:49:17.516583+0000","last_became_peered":"2024-04-06T13:49:17.516583+0000","last_unstale":"2024-04-06T13:49:17.520547+0000","last_undegraded":"2024-04-06T13:49:17.520547+0000","last_fullsized":"2024-04-06T13:49:17.520547+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:44:33.017556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.506905+0000","last_change":"2024-04-06T13:49:17.108195+0000","last_active":"2024-04-06T13:49:17.506905+0000","last_peered":"2024-04-06T13:49:17.506905+0000","last_clean":"2024-04-06T13:49:17.506905+0000","last_became_active":"2024-04-06T13:49:17.107932+0000","last_became_peered":"2024-04-06T13:49:17.107932+0000","last_unstale":"2024-04-06T13:49:17.506905+0000","last_undegraded":"2024-04-06T13:49:17.506905+0000","last_fullsized":"2024-04-06T13:49:17.506905+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:15:46.589339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.520178+0000","last_change":"2024-04-06T13:49:17.518240+0000","last_active":"2024-04-06T13:49:17.520178+0000","last_peered":"2024-04-06T13:49:17.520178+0000","last_clean":"2024-04-06T13:49:17.520178+0000","last_became_active":"2024-04-06T13:49:17.517985+0000","last_became_peered":"2024-04-06T13:49:17.517985+0000","last_unstale":"2024-04-06T13:49:17.520178+0000","last_undegraded":"2024-04-06T13:49:17.520178+0000","last_fullsized":"2024-04-06T13:49:17.520178+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:27:31.865608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.503806+0000","last_change":"2024-04-06T13:49:17.102416+0000","last_active":"2024-04-06T13:49:17.503806+0000","last_peered":"2024-04-06T13:49:17.503806+0000","last_clean":"2024-04-06T13:49:17.503806+0000","last_became_active":"2024-04-06T13:49:17.102080+0000","last_became_peered":"2024-04-06T13:49:17.102080+0000","last_unstale":"2024-04-06T13:49:17.503806+0000","last_undegraded":"2024-04-06T13:49:17.503806+0000","last_fullsized":"2024-04-06T13:49:17.503806+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:27:31.581867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.517664+0000","last_change":"2024-04-06T13:49:17.511968+0000","last_active":"2024-04-06T13:49:17.517664+0000","last_peered":"2024-04-06T13:49:17.517664+0000","last_clean":"2024-04-06T13:49:17.517664+0000","last_became_active":"2024-04-06T13:49:17.511704+0000","last_became_peered":"2024-04-06T13:49:17.511704+0000","last_unstale":"2024-04-06T13:49:17.517664+0000","last_undegraded":"2024-04-06T13:49:17.517664+0000","last_fullsized":"2024-04-06T13:49:17.517664+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:36:00.408855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":10,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.501319+0000","last_change":"2024-04-06T13:49:17.105215+0000","last_active":"2024-04-06T13:49:17.501319+0000","last_peered":"2024-04-06T13:49:17.501319+0000","last_clean":"2024-04-06T13:49:17.501319+0000","last_became_active":"2024-04-06T13:49:17.104577+0000","last_became_peered":"2024-04-06T13:49:17.104577+0000","last_unstale":"2024-04-06T13:49:17.501319+0000","last_undegraded":"2024-04-06T13:49:17.501319+0000","last_fullsized":"2024-04-06T13:49:17.501319+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":266,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:16.077129+0000","last_clean_scrub_stamp":"2024-04-06T13:49:16.077129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:14:52.150676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":9,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.509604+0000","last_change":"2024-04-06T13:49:17.507776+0000","last_active":"2024-04-06T13:49:17.509604+0000","last_peered":"2024-04-06T13:49:17.509604+0000","last_clean":"2024-04-06T13:49:17.509604+0000","last_became_active":"2024-04-06T13:49:17.507456+0000","last_became_peered":"2024-04-06T13:49:17.507456+0000","last_unstale":"2024-04-06T13:49:17.509604+0000","last_undegraded":"2024-04-06T13:49:17.509604+0000","last_fullsized":"2024-04-06T13:49:17.509604+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:17.078971+0000","last_clean_scrub_stamp":"2024-04-06T13:49:17.078971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:42:50.418088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'381","reported_seq":1396,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-04-06T13:49:17.085080+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:49:17.085080+0000","last_peered":"2024-04-06T13:49:17.085080+0000","last_clean":"2024-04-06T13:49:17.085080+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:49:17.085080+0000","last_undegraded":"2024-04-06T13:49:17.085080+0000","last_fullsized":"2024-04-06T13:49:17.085080+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576002,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57124,"kb_used_data":3864,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314716,"statfs":{"total":96636764160,"available":96578269184,"internally_reserved":0,"allocated":3956736,"data_stored":2324678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55700000000000005,"15min":0.59199999999999997},"min":{"1min":0.34499999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.747,"5min":0.751,"15min":0.82499999999999996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.54700000000000004,"15min":0.58299999999999996},"min":{"1min":0.35099999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.753,"5min":0.78500000000000003,"15min":0.82899999999999996},"last":0.46400000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53800000000000003,"15min":0.57499999999999996},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.69199999999999995,"5min":0.77700000000000002,"15min":0.81299999999999994},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":0.58799999999999997},"min":{"1min":0.39200000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.85299999999999998},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.53700000000000003,"15min":0.57599999999999996},"min":{"1min":0.379,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.63800000000000001,"5min":0.78600000000000003,"15min":0.83899999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57099999999999995,"15min":0.59599999999999997},"min":{"1min":0.373,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.79700000000000004},"last":0.44}]},{"osd":3,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.56399999999999995,"15min":0.59699999999999998},"min":{"1min":0.36799999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.72699999999999998,"5min":0.76700000000000002,"15min":0.81100000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.57399999999999995,"15min":0.59799999999999998},"min":{"1min":0.436,"5min":0.39000000000000001,"15min":0.34999999999999998},"max":{"1min":0.66500000000000004,"5min":0.78900000000000003,"15min":0.84099999999999997},"last":0.58999999999999997}]},{"osd":4,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.45700000000000002,"15min":0.46300000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.67600000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54200000000000004,"15min":0.54000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64700000000000002,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.70299999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.498,"15min":0.496},"min":{"1min":0.316,"5min":0.315,"15min":0.308},"max":{"1min":0.68999999999999995,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.44500000000000001,"5min":0.35599999999999998,"15min":0.33600000000000002},"max":{"1min":0.77300000000000002,"5min":0.79700000000000004,"15min":0.82099999999999995},"last":0.68600000000000005}]},{"osd":6,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55400000000000005,"15min":0.55500000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.69099999999999995,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56100000000000005,"15min":0.58099999999999996},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.65700000000000003,"5min":0.81799999999999995,"15min":0.88200000000000001},"last":0.63900000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56556,"kb_used_data":3292,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315284,"statfs":{"total":96636764160,"available":96578850816,"internally_reserved":0,"allocated":3371008,"data_stored":1739958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.51900000000000002,"15min":0.55000000000000004},"min":{"1min":0.35199999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.59199999999999997,"5min":0.79700000000000004,"15min":0.83799999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.497,"5min":0.53300000000000003,"15min":0.56200000000000006},"min":{"1min":0.314,"5min":0.31,"15min":0.31},"max":{"1min":0.63,"5min":0.76900000000000002,"15min":0.81999999999999995},"last":0.44}]},{"osd":1,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.59099999999999997,"15min":0.61699999999999999},"min":{"1min":0.39600000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70699999999999996,"5min":0.81699999999999995,"15min":0.85799999999999998},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.60299999999999998,"15min":0.63900000000000001},"min":{"1min":0.45100000000000001,"5min":0.41099999999999998,"15min":0.378},"max":{"1min":0.74199999999999999,"5min":0.78900000000000003,"15min":0.82699999999999996},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56100000000000005,"15min":0.58899999999999997},"min":{"1min":0.41799999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72899999999999998,"5min":0.79000000000000004,"15min":0.82899999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.54200000000000004,"15min":0.57899999999999996},"min":{"1min":0.35899999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67400000000000004,"5min":0.77700000000000002,"15min":0.80700000000000005},"last":0.49199999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.56699999999999995,"15min":0.59799999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69999999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61599999999999999,"15min":0.63900000000000001},"min":{"1min":0.441,"5min":0.39900000000000002,"15min":0.36099999999999999},"max":{"1min":0.77200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.59199999999999997}]},{"osd":4,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53300000000000003,"15min":0.53100000000000003},"min":{"1min":0.36099999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.81399999999999995},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51500000000000001,"15min":0.53400000000000003},"min":{"1min":0.32900000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.73699999999999999,"5min":0.82099999999999995,"15min":0.84099999999999997},"last":0.61799999999999999}]},{"osd":5,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.501,"15min":0.50600000000000001},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.33300000000000002},"max":{"1min":0.754,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.628},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52200000000000002,"15min":0.53100000000000003},"min":{"1min":0.40400000000000003,"5min":0.316,"15min":0.316},"max":{"1min":0.72599999999999998,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.56399999999999995}]},{"osd":7,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79300000000000004,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55800000000000005,"15min":0.56699999999999995},"min":{"1min":0.39900000000000002,"5min":0.35399999999999998,"15min":0.32800000000000001},"max":{"1min":0.77200000000000002,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.60199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834576000,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64764,"kb_used_data":3308,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307076,"statfs":{"total":96636764160,"available":96570445824,"internally_reserved":0,"allocated":3387392,"data_stored":1747215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.36299999999999999,"15min":0.38500000000000001},"min":{"1min":0.26100000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.443,"5min":0.621,"15min":0.621},"last":1.2230000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.39200000000000002,"15min":0.41999999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.504,"5min":0.56100000000000005,"15min":0.69399999999999995},"last":2.0539999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.39400000000000002,"15min":0.41699999999999998},"min":{"1min":0.25600000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.41999999999999998,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.42699999999999999,"15min":0.45600000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.51900000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":2.0169999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.44600000000000001,"15min":0.47199999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.51100000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":4.2699999999999996},{"interface":"front","average":{"1min":0.374,"5min":0.40799999999999997,"15min":0.434},"min":{"1min":0.29599999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.51700000000000002,"5min":0.63500000000000001,"15min":0.66100000000000003},"last":2.153}]},{"osd":4,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.59399999999999997,"15min":0.63400000000000001},"min":{"1min":0.377,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.80300000000000005,"5min":0.89900000000000002,"15min":0.97399999999999998},"last":1.1140000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.61199999999999999,"15min":0.64300000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.79300000000000004,"5min":0.877,"15min":0.995},"last":0.82499999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.60499999999999998,"15min":0.64200000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.72699999999999998,"5min":0.91800000000000004,"15min":1.073},"last":1.1779999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.626,"15min":0.65900000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.73999999999999999,"5min":0.96499999999999997,"15min":1.0209999999999999},"last":4.2030000000000003}]},{"osd":6,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.61799999999999999,"15min":0.64900000000000002},"min":{"1min":0.435,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.753,"5min":0.89200000000000002,"15min":1},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.63600000000000001,"15min":0.67200000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76600000000000001,"5min":0.94599999999999995,"15min":1.036},"last":4.2889999999999997}]},{"osd":7,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.66400000000000003,"15min":0.68799999999999994},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.94899999999999995,"15min":1.0649999999999999},"last":4.0730000000000004},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.626,"15min":0.66000000000000003},"min":{"1min":0.44700000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.78100000000000003,"5min":0.92900000000000005,"15min":1.056},"last":2.194}]}]},{"osd":0,"up_from":13,"seq":55834576000,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61220,"kb_used_data":3860,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310620,"statfs":{"total":96636764160,"available":96574074880,"internally_reserved":0,"allocated":3952640,"data_stored":2323659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.36899999999999999,"15min":0.38900000000000001},"min":{"1min":0.24299999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.47199999999999998,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.41299999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.50900000000000001,"5min":0.58599999999999997,"15min":0.70799999999999996},"last":0.377}]},{"osd":2,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39000000000000001,"15min":0.42199999999999999},"min":{"1min":0.28299999999999997,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.53900000000000003,"5min":0.65400000000000003,"15min":0.70099999999999996},"last":0.315},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.41699999999999998,"15min":0.441},"min":{"1min":0.29899999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.58199999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.33200000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.39600000000000002,"15min":0.42099999999999999},"min":{"1min":0.27300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52400000000000002,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.44800000000000001,"15min":0.47399999999999998},"min":{"1min":0.313,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.60899999999999999,"5min":0.71899999999999997,"15min":0.75700000000000001},"last":0.35299999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.59099999999999997,"15min":0.63600000000000001},"min":{"1min":0.38500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.85199999999999998,"5min":0.93100000000000005,"15min":1.0329999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.60299999999999998,"15min":0.64100000000000001},"min":{"1min":0.39800000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86099999999999999,"5min":0.999,"15min":0.999},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59499999999999997,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.83899999999999997,"5min":0.91800000000000004,"15min":1.0009999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.627,"15min":0.66200000000000003},"min":{"1min":0.42699999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.86499999999999999,"5min":0.92800000000000005,"15min":1.052},"last":0.56699999999999995}]},{"osd":6,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63100000000000001,"15min":0.67000000000000004},"min":{"1min":0.438,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.81699999999999995,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.629,"15min":0.66800000000000004},"min":{"1min":0.44900000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":1.085},"last":0.50700000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.64900000000000002,"15min":0.68799999999999994},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.89400000000000002,"5min":1.01,"15min":1.121},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.626,"5min":0.64000000000000001,"15min":0.67000000000000004},"min":{"1min":0.47299999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.82999999999999996,"5min":0.95799999999999996,"15min":1.0229999999999999},"last":0.48499999999999999}]}]},{"osd":2,"up_from":13,"seq":55834576002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60640,"kb_used_data":3280,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311200,"statfs":{"total":96636764160,"available":96574668800,"internally_reserved":0,"allocated":3358720,"data_stored":1733291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.36799999999999999,"15min":0.38100000000000001},"min":{"1min":0.22,"5min":0.184,"15min":0.184},"max":{"1min":0.59399999999999997,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.39700000000000002,"15min":0.41899999999999998},"min":{"1min":0.26200000000000001,"5min":0.251,"15min":0.249},"max":{"1min":0.54600000000000004,"5min":0.71499999999999997,"15min":1.044},"last":0.53700000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.371,"15min":0.39900000000000002},"min":{"1min":0.23599999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.46899999999999997,"5min":0.52900000000000003,"15min":0.753},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.41799999999999998,"15min":0.441},"min":{"1min":0.27000000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.82199999999999995},"last":0.32400000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.42199999999999999,"15min":0.45200000000000001},"min":{"1min":0.27000000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.621,"5min":0.63900000000000001,"15min":0.70299999999999996},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.42499999999999999,"15min":0.443},"min":{"1min":0.27200000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.433}]},{"osd":4,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61099999999999999,"15min":0.63300000000000001},"min":{"1min":0.436,"5min":0.40200000000000002,"15min":0.376},"max":{"1min":0.77600000000000002,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.63,"15min":0.65400000000000003},"min":{"1min":0.40999999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82799999999999996,"5min":1.022,"15min":1.022},"last":0.76300000000000001}]},{"osd":5,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.629,"15min":0.65800000000000003},"min":{"1min":0.44900000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79000000000000004,"5min":0.98599999999999999,"15min":1.0149999999999999},"last":0.752},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.63800000000000001,"15min":0.66000000000000003},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85199999999999998,"5min":1.028,"15min":1.028},"last":0.67200000000000004}]},{"osd":6,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64500000000000002,"15min":0.66600000000000004},"min":{"1min":0.45300000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86899999999999999,"5min":0.97199999999999998,"15min":1.091},"last":0.63},{"interface":"front","average":{"1min":0.625,"5min":0.65900000000000003,"15min":0.68799999999999994},"min":{"1min":0.47699999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84099999999999997,"5min":0.95699999999999996,"15min":0.96099999999999997},"last":0.70199999999999996}]},{"osd":7,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.66100000000000003,"15min":0.68300000000000005},"min":{"1min":0.47299999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91300000000000003,"5min":0.94499999999999995,"15min":0.95399999999999996},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.66900000000000004,"15min":0.69999999999999996},"min":{"1min":0.46100000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.89100000000000001,"5min":1.01,"15min":1.01},"last":0.79500000000000004}]}]},{"osd":3,"up_from":13,"seq":55834576001,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48352,"kb_used_data":3284,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323488,"statfs":{"total":96636764160,"available":96587251712,"internally_reserved":0,"allocated":3362816,"data_stored":1734310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.37,"15min":0.38400000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.47399999999999998,"5min":0.55200000000000005,"15min":0.72799999999999998},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.39600000000000002,"15min":0.40799999999999997},"min":{"1min":0.25700000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.48799999999999999,"5min":0.58599999999999997,"15min":0.66700000000000004},"last":0.307}]},{"osd":1,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.39300000000000002,"15min":0.40100000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.46899999999999997,"5min":0.59499999999999997,"15min":0.65700000000000003},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.41699999999999998,"15min":0.437},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.502,"5min":0.63,"15min":0.75800000000000001},"last":0.32300000000000001}]},{"osd":2,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.46100000000000002,"15min":0.47999999999999998},"min":{"1min":0.32700000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.53900000000000003,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.47699999999999998,"15min":0.496},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.51900000000000002,"5min":0.67900000000000005,"15min":0.79300000000000004},"last":0.42799999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.622,"15min":0.63100000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76700000000000002,"5min":0.94099999999999995,"15min":1.018},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.628,"15min":0.64200000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.81100000000000005,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.63900000000000001}]},{"osd":5,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.66100000000000003,"15min":0.67600000000000005},"min":{"1min":0.44700000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.872,"5min":0.95699999999999996,"15min":0.999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.67900000000000005,"15min":0.68700000000000006},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.86299999999999999,"5min":1.036,"15min":1.036},"last":0.627}]},{"osd":6,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.64100000000000001,"15min":0.66100000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.76300000000000001,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.66000000000000003,"15min":0.67600000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41099999999999998},"max":{"1min":0.82799999999999996,"5min":0.91600000000000004,"15min":0.98799999999999999},"last":0.55000000000000004}]},{"osd":7,"last update":"Sat Apr 6 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.65500000000000003,"15min":0.66800000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80200000000000005,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.67800000000000005,"15min":0.69499999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85199999999999998,"5min":1.0209999999999999,"15min":1.024},"last":0.67400000000000004}]}]},{"osd":4,"up_from":13,"seq":55834576001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48364,"kb_used_data":3296,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323476,"statfs":{"total":96636764160,"available":96587239424,"internally_reserved":0,"allocated":3375104,"data_stored":1740977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55300000000000005,"15min":0.57999999999999996},"min":{"1min":0.317,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73299999999999998,"5min":0.80900000000000005,"15min":0.88500000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.57599999999999996,"15min":0.59099999999999997},"min":{"1min":0.41699999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79200000000000004,"5min":1.202,"15min":1.202},"last":0.60999999999999999}]},{"osd":1,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60099999999999998,"15min":0.63400000000000001},"min":{"1min":0.379,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.74299999999999999,"5min":1.256,"15min":1.256},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61499999999999999,"15min":0.624},"min":{"1min":0.39300000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.71999999999999997,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.57999999999999996}]},{"osd":2,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.56899999999999995,"15min":0.59099999999999997},"min":{"1min":0.377,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.68200000000000005,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.59299999999999997,"15min":0.61199999999999999},"min":{"1min":0.373,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89500000000000002,"5min":1.294,"15min":1.294},"last":0.67900000000000005}]},{"osd":3,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.57499999999999996,"15min":0.58999999999999997},"min":{"1min":0.35999999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80800000000000005,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60399999999999998,"15min":0.62},"min":{"1min":0.439,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76700000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.69299999999999995}]},{"osd":5,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56100000000000005,"15min":0.52000000000000002},"min":{"1min":0.33800000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.75,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56999999999999995,"15min":0.54100000000000004},"min":{"1min":0.372,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.75,"5min":4.931,"15min":4.931},"last":0.73799999999999999}]},{"osd":6,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53900000000000003,"15min":0.53400000000000003},"min":{"1min":0.40699999999999997,"5min":0.34100000000000003,"15min":0.33300000000000002},"max":{"1min":0.78300000000000003,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56599999999999995,"15min":0.56999999999999995},"min":{"1min":0.41099999999999998,"5min":0.35799999999999998,"15min":0.34699999999999998},"max":{"1min":0.97099999999999997,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.81000000000000005}]},{"osd":7,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60199999999999998,"15min":0.58199999999999996},"min":{"1min":0.35899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.84999999999999998,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.59399999999999997,"15min":0.60099999999999998},"min":{"1min":0.48599999999999999,"5min":0.38,"15min":0.373},"max":{"1min":0.82799999999999996,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83499999999999996}]}]},{"osd":5,"up_from":13,"seq":55834576001,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66400,"kb_used_data":3280,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305440,"statfs":{"total":96636764160,"available":96568770560,"internally_reserved":0,"allocated":3358720,"data_stored":1733291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.56100000000000005,"15min":0.58299999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.72799999999999998,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.56599999999999995,"15min":0.58899999999999997},"min":{"1min":0.29899999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.74399999999999999,"5min":0.84699999999999998,"15min":0.89200000000000002},"last":0.53700000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.56200000000000006,"15min":0.58799999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.71299999999999997,"5min":0.81299999999999994,"15min":0.81399999999999995},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.56100000000000005,"15min":0.58299999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.752,"5min":0.871,"15min":0.871},"last":0.47099999999999997}]},{"osd":2,"last update":"Sat Apr 6 13:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.32300000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.57099999999999995,"15min":0.59399999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.41399999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.57599999999999996,"15min":0.59599999999999997},"min":{"1min":0.375,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.72199999999999998,"5min":0.81100000000000005,"15min":0.94499999999999995},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.60699999999999998,"15min":0.63100000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.80200000000000005,"5min":0.88200000000000001,"15min":0.91500000000000004},"last":0.50900000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53000000000000003,"15min":0.50900000000000001},"min":{"1min":0.35499999999999998,"5min":0.32700000000000001,"15min":0.27400000000000002},"max":{"1min":0.72399999999999998,"5min":0.85699999999999998,"15min":1.0369999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59599999999999997,"15min":0.58099999999999996},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.76600000000000001,"5min":0.88800000000000001,"15min":1.1639999999999999},"last":0.59799999999999998}]},{"osd":6,"last update":"Sat Apr 6 13:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54700000000000004,"15min":0.53100000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.28899999999999998},"max":{"1min":0.80000000000000004,"5min":0.94999999999999996,"15min":1.0169999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54400000000000004,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72499999999999998,"5min":0.878,"15min":1.0009999999999999},"last":0.56599999999999995}]},{"osd":7,"last update":"Sat Apr 6 13:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.56100000000000005,"15min":0.55000000000000004},"min":{"1min":0.38300000000000001,"5min":0.35799999999999998,"15min":0.32900000000000001},"max":{"1min":0.77100000000000002,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.66200000000000003,"15min":0.65100000000000002},"min":{"1min":0.45800000000000002,"5min":0.41699999999999998,"15min":0.38300000000000001},"max":{"1min":0.82999999999999996,"5min":0.92300000000000004,"15min":0.96599999999999997},"last":0.622}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:49:22.008 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-06T13:49:22.009 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-06T13:49:22.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.172+0000 7ff87f9bd700 1 Processor -- start 2024-04-06T13:49:22.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.172+0000 7ff87f9bd700 1 -- start start 2024-04-06T13:49:22.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.173+0000 7ff87f9bd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff8780d11b0 0x7ff8780d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:22.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.173+0000 7ff87f9bd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8780cb870 0x7ff87815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:22.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.173+0000 7ff87f9bd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff8780cfa00 0x7ff8780cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:22.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.173+0000 7ff87f9bd700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8780ccdc0 con 0x7ff8780d11b0 2024-04-06T13:49:22.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.173+0000 7ff87f9bd700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8780d0b50 con 0x7ff8780cfa00 2024-04-06T13:49:22.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.173+0000 7ff87f9bd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8780d1ad0 con 0x7ff8780cb870 2024-04-06T13:49:22.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.173+0000 7ff87d759700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff8780d11b0 0x7ff8780d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:22.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.173+0000 7ff87d759700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff8780d11b0 0x7ff8780d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46246/0 (socket says 172.21.15.189:46246) 2024-04-06T13:49:22.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.173+0000 7ff87d759700 1 -- 172.21.15.189:0/4198019620 learned_addr learned my addr 172.21.15.189:0/4198019620 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:22.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.174+0000 7ff87df5a700 1 --2- 172.21.15.189:0/4198019620 >> v2:172.21.15.192:3300/0 conn(0x7ff8780cfa00 0x7ff8780cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:22.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.174+0000 7ff87cf58700 1 --2- 172.21.15.189:0/4198019620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8780cb870 0x7ff87815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:22.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.174+0000 7ff87d759700 1 -- 172.21.15.189:0/4198019620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8780cb870 msgr2=0x7ff87815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:22.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.174+0000 7ff87d759700 1 --2- 172.21.15.189:0/4198019620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8780cb870 0x7ff87815c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:22.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.174+0000 7ff87d759700 1 -- 172.21.15.189:0/4198019620 >> v2:172.21.15.192:3300/0 conn(0x7ff8780cfa00 msgr2=0x7ff8780cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:22.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.174+0000 7ff87d759700 1 --2- 172.21.15.189:0/4198019620 >> v2:172.21.15.192:3300/0 conn(0x7ff8780cfa00 0x7ff8780cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:22.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.174+0000 7ff87d759700 1 -- 172.21.15.189:0/4198019620 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8780cbd80 con 0x7ff8780d11b0 2024-04-06T13:49:22.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.174+0000 7ff87df5a700 1 --2- 172.21.15.189:0/4198019620 >> v2:172.21.15.192:3300/0 conn(0x7ff8780cfa00 0x7ff8780cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:22.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.174+0000 7ff87cf58700 1 --2- 172.21.15.189:0/4198019620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8780cb870 0x7ff87815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:22.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.174+0000 7ff87d759700 1 --2- 172.21.15.189:0/4198019620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8780d11b0 0x7ff8780d1590 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 crypto rx=0x7ff8780ccd10 tx=0x7ff868009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb69271a1b6afca7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:22.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.174+0000 7ff86ffff700 1 -- 172.21.15.189:0/4198019620 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86800b870 con 0x7ff8780d11b0 2024-04-06T13:49:22.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.174+0000 7ff86ffff700 1 -- 172.21.15.189:0/4198019620 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86800b9d0 con 0x7ff8780d11b0 2024-04-06T13:49:22.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.174+0000 7ff86ffff700 1 -- 172.21.15.189:0/4198019620 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86800bd60 con 0x7ff8780d11b0 2024-04-06T13:49:22.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.175+0000 7ff87f9bd700 1 -- 172.21.15.189:0/4198019620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8780d11b0 msgr2=0x7ff8780d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:22.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.175+0000 7ff87f9bd700 1 --2- 172.21.15.189:0/4198019620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8780d11b0 0x7ff8780d1590 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 crypto rx=0x7ff8780ccd10 tx=0x7ff868009bd0 comp rx=0 tx=0).stop 2024-04-06T13:49:22.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.175+0000 7ff87f9bd700 1 -- 172.21.15.189:0/4198019620 shutdown_connections 2024-04-06T13:49:22.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.175+0000 7ff87f9bd700 1 --2- 172.21.15.189:0/4198019620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8780d11b0 0x7ff8780d1590 unknown :-1 s=CLOSED pgs=2941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:22.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.175+0000 7ff87f9bd700 1 --2- 172.21.15.189:0/4198019620 >> v2:172.21.15.192:3300/0 conn(0x7ff8780cfa00 0x7ff8780cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:22.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.175+0000 7ff87f9bd700 1 --2- 172.21.15.189:0/4198019620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8780cb870 0x7ff87815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:22.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.175+0000 7ff87f9bd700 1 -- 172.21.15.189:0/4198019620 >> 172.21.15.189:0/4198019620 conn(0x7ff8780b2db0 msgr2=0x7ff8780b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:22.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.175+0000 7ff87f9bd700 1 -- 172.21.15.189:0/4198019620 shutdown_connections 2024-04-06T13:49:22.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.175+0000 7ff87f9bd700 1 -- 172.21.15.189:0/4198019620 wait complete. 2024-04-06T13:49:22.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.176+0000 7ff87f9bd700 1 Processor -- start 2024-04-06T13:49:22.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.176+0000 7ff87f9bd700 1 -- start start 2024-04-06T13:49:22.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff87f9bd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8780cb870 0x7ff8780c9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:22.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff87f9bd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8780cfa00 0x7ff8781547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:22.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff87f9bd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8780d11b0 0x7ff87806c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:22.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff87f9bd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8780c87f0 con 0x7ff8780cb870 2024-04-06T13:49:22.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff87f9bd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8780c8ad0 con 0x7ff8780cfa00 2024-04-06T13:49:22.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff87f9bd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8780c8db0 con 0x7ff8780d11b0 2024-04-06T13:49:22.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff87d759700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8780cb870 0x7ff8780c9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:22.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff87d759700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8780cb870 0x7ff8780c9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46252/0 (socket says 172.21.15.189:46252) 2024-04-06T13:49:22.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff87d759700 1 -- 172.21.15.189:0/3338200136 learned_addr learned my addr 172.21.15.189:0/3338200136 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:22.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff87df5a700 1 --2- 172.21.15.189:0/3338200136 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8780d11b0 0x7ff87806c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:22.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff87cf58700 1 --2- 172.21.15.189:0/3338200136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8780cfa00 0x7ff8781547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:22.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff87d759700 1 -- 172.21.15.189:0/3338200136 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8780d11b0 msgr2=0x7ff87806c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:22.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff87d759700 1 --2- 172.21.15.189:0/3338200136 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8780d11b0 0x7ff87806c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:22.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff87d759700 1 -- 172.21.15.189:0/3338200136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8780cfa00 msgr2=0x7ff8781547d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:22.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff87d759700 1 --2- 172.21.15.189:0/3338200136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8780cfa00 0x7ff8781547d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:22.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff87d759700 1 -- 172.21.15.189:0/3338200136 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff868009550 con 0x7ff8780cb870 2024-04-06T13:49:22.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff87cf58700 1 --2- 172.21.15.189:0/3338200136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8780cfa00 0x7ff8781547d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:22.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff87d759700 1 --2- 172.21.15.189:0/3338200136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8780cb870 0x7ff8780c9890 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 crypto rx=0x7ff8780677e0 tx=0x7ff868004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:22.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff86dffb700 1 -- 172.21.15.189:0/3338200136 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff868004310 con 0x7ff8780cb870 2024-04-06T13:49:22.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff86dffb700 1 -- 172.21.15.189:0/3338200136 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86801f070 con 0x7ff8780cb870 2024-04-06T13:49:22.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff87f9bd700 1 -- 172.21.15.189:0/3338200136 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8780c9090 con 0x7ff8780cb870 2024-04-06T13:49:22.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.177+0000 7ff86dffb700 1 -- 172.21.15.189:0/3338200136 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8680036a0 con 0x7ff8780cb870 2024-04-06T13:49:22.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.178+0000 7ff87f9bd700 1 -- 172.21.15.189:0/3338200136 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8780c9dd0 con 0x7ff8780cb870 2024-04-06T13:49:22.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.179+0000 7ff87f9bd700 1 -- 172.21.15.189:0/3338200136 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff878074990 con 0x7ff8780cb870 2024-04-06T13:49:22.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.179+0000 7ff86dffb700 1 -- 172.21.15.189:0/3338200136 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7ff868021030 con 0x7ff8780cb870 2024-04-06T13:49:22.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.179+0000 7ff86dffb700 1 --2- 172.21.15.189:0/3338200136 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff85c06aed0 0x7ff85c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:22.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.179+0000 7ff86dffb700 1 -- 172.21.15.189:0/3338200136 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff86808ac80 con 0x7ff8780cb870 2024-04-06T13:49:22.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.179+0000 7ff87cf58700 1 --2- 172.21.15.189:0/3338200136 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff85c06aed0 0x7ff85c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:22.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.180+0000 7ff87cf58700 1 --2- 172.21.15.189:0/3338200136 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff85c06aed0 0x7ff85c06d390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff8780b1d10 tx=0x7ff874008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:22.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.184+0000 7ff86dffb700 1 -- 172.21.15.189:0/3338200136 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff868055a00 con 0x7ff8780cb870 2024-04-06T13:49:22.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.459+0000 7ff87f9bd700 1 -- 172.21.15.189:0/3338200136 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff8780b0a00 con 0x7ff8780cb870 2024-04-06T13:49:22.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.463+0000 7ff86dffb700 1 -- 172.21.15.189:0/3338200136 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v54) v1 ==== 190+0+0 (secure 0 0 0) 0x7ff8680150e0 con 0x7ff8780cb870 2024-04-06T13:49:22.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.465+0000 7ff87f9bd700 1 -- 172.21.15.189:0/3338200136 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff85c06aed0 msgr2=0x7ff85c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:22.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.465+0000 7ff87f9bd700 1 --2- 172.21.15.189:0/3338200136 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff85c06aed0 0x7ff85c06d390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff8780b1d10 tx=0x7ff874008040 comp rx=0 tx=0).stop 2024-04-06T13:49:22.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.465+0000 7ff87f9bd700 1 -- 172.21.15.189:0/3338200136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8780cb870 msgr2=0x7ff8780c9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:22.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.465+0000 7ff87f9bd700 1 --2- 172.21.15.189:0/3338200136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8780cb870 0x7ff8780c9890 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 crypto rx=0x7ff8780677e0 tx=0x7ff868004750 comp rx=0 tx=0).stop 2024-04-06T13:49:22.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.465+0000 7ff87f9bd700 1 -- 172.21.15.189:0/3338200136 shutdown_connections 2024-04-06T13:49:22.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.465+0000 7ff87f9bd700 1 --2- 172.21.15.189:0/3338200136 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff85c06aed0 0x7ff85c06d390 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:22.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.465+0000 7ff87f9bd700 1 --2- 172.21.15.189:0/3338200136 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8780d11b0 0x7ff87806c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:22.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.465+0000 7ff87f9bd700 1 --2- 172.21.15.189:0/3338200136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8780cb870 0x7ff8780c9890 unknown :-1 s=CLOSED pgs=2942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:22.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.465+0000 7ff87f9bd700 1 --2- 172.21.15.189:0/3338200136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8780cfa00 0x7ff8781547d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:22.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.465+0000 7ff87f9bd700 1 -- 172.21.15.189:0/3338200136 >> 172.21.15.189:0/3338200136 conn(0x7ff8780b2db0 msgr2=0x7ff8780b6190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:22.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.465+0000 7ff87f9bd700 1 -- 172.21.15.189:0/3338200136 shutdown_connections 2024-04-06T13:49:22.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.465+0000 7ff87f9bd700 1 -- 172.21.15.189:0/3338200136 wait complete. 2024-04-06T13:49:22.467 INFO:teuthology.orchestra.run.smithi189.stderr:updated caps for client.0 2024-04-06T13:49:22.478 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:49:22.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.601+0000 7fbede56c700 1 Processor -- start 2024-04-06T13:49:22.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.601+0000 7fbede56c700 1 -- start start 2024-04-06T13:49:22.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.602+0000 7fbede56c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbed815d7b0 0x7fbed80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:22.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.602+0000 7fbede56c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed80b3990 con 0x7fbed815e500 2024-04-06T13:49:22.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.602+0000 7fbede56c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed80b3060 con 0x7fbed806a730 2024-04-06T13:49:22.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.602+0000 7fbede56c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbed815e950 con 0x7fbed815d7b0 2024-04-06T13:49:22.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.602+0000 7fbedcb09700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbed815e500 0x7fbed80717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40068/0 (socket says 172.21.15.189:40068) 2024-04-06T13:49:22.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.602+0000 7fbedcb09700 1 -- 172.21.15.189:0/3682329080 learned_addr learned my addr 172.21.15.189:0/3682329080 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:22.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.602+0000 7fbed7fff700 1 --2- 172.21.15.189:0/3682329080 >> v2:172.21.15.192:3300/0 conn(0x7fbed815d7b0 0x7fbed80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:22.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.602+0000 7fbed6ffd700 1 -- 172.21.15.189:0/3682329080 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334546644 0 0) 0x7fbed80b3990 con 0x7fbed815e500 2024-04-06T13:49:22.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.602+0000 7fbed6ffd700 1 -- 172.21.15.189:0/3682329080 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbec00035b0 con 0x7fbed815e500 2024-04-06T13:49:22.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.602+0000 7fbed6ffd700 1 -- 172.21.15.189:0/3682329080 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2274056179 0 0) 0x7fbec00035b0 con 0x7fbed815e500 2024-04-06T13:49:22.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.602+0000 7fbed7fff700 1 -- 172.21.15.189:0/3682329080 >> v1:172.21.15.192:6789/0 conn(0x7fbed806a730 legacy=0x7fbed815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:22.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.602+0000 7fbed7fff700 1 -- 172.21.15.189:0/3682329080 >> v1:172.21.15.189:6789/0 conn(0x7fbed815e500 legacy=0x7fbed80717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:22.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.602+0000 7fbed7fff700 1 -- 172.21.15.189:0/3682329080 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed80b3340 con 0x7fbed815d7b0 2024-04-06T13:49:22.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.603+0000 7fbed7fff700 1 --2- 172.21.15.189:0/3682329080 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbed815d7b0 0x7fbed80b3e20 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009840 tx=0x7fbecc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa7293e606d14584 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:22.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.603+0000 7fbed6ffd700 1 -- 172.21.15.189:0/3682329080 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed00019f0 con 0x7fbed815d7b0 2024-04-06T13:49:22.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.603+0000 7fbed6ffd700 1 -- 172.21.15.189:0/3682329080 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbecc004030 con 0x7fbed815d7b0 2024-04-06T13:49:22.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.603+0000 7fbed6ffd700 1 -- 172.21.15.189:0/3682329080 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc0043c0 con 0x7fbed815d7b0 2024-04-06T13:49:22.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.603+0000 7fbede56c700 1 -- 172.21.15.189:0/3682329080 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbed815d7b0 msgr2=0x7fbed80b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:22.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.603+0000 7fbede56c700 1 --2- 172.21.15.189:0/3682329080 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbed815d7b0 0x7fbed80b3e20 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009840 tx=0x7fbecc009b50 comp rx=0 tx=0).stop 2024-04-06T13:49:22.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.603+0000 7fbede56c700 1 -- 172.21.15.189:0/3682329080 shutdown_connections 2024-04-06T13:49:22.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.603+0000 7fbede56c700 1 --2- 172.21.15.189:0/3682329080 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbed815d7b0 0x7fbed80b3e20 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:22.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.603+0000 7fbede56c700 1 -- 172.21.15.189:0/3682329080 >> 172.21.15.189:0/3682329080 conn(0x7fbed80d00b0 msgr2=0x7fbed80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:22.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.603+0000 7fbede56c700 1 -- 172.21.15.189:0/3682329080 shutdown_connections 2024-04-06T13:49:22.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.603+0000 7fbede56c700 1 -- 172.21.15.189:0/3682329080 wait complete. 2024-04-06T13:49:22.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.605+0000 7fbede56c700 1 Processor -- start 2024-04-06T13:49:22.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.605+0000 7fbede56c700 1 -- start start 2024-04-06T13:49:22.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbede56c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbed806a730 0x7fbed8151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:22.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbede56c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed815d7b0 0x7fbed80b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:22.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbede56c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbed815e500 0x7fbed80aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:22.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbede56c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed80b1810 con 0x7fbed815d7b0 2024-04-06T13:49:22.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbede56c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed80af0c0 con 0x7fbed806a730 2024-04-06T13:49:22.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbede56c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed80af2f0 con 0x7fbed815e500 2024-04-06T13:49:22.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbed77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed815d7b0 0x7fbed80b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:22.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbed77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed815d7b0 0x7fbed80b12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46262/0 (socket says 172.21.15.189:46262) 2024-04-06T13:49:22.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbed77fe700 1 -- 172.21.15.189:0/3409045375 learned_addr learned my addr 172.21.15.189:0/3409045375 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:22.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbed7fff700 1 --2- 172.21.15.189:0/3409045375 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbed806a730 0x7fbed8151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:22.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbedcb09700 1 --2- 172.21.15.189:0/3409045375 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbed815e500 0x7fbed80aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:22.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbed77fe700 1 -- 172.21.15.189:0/3409045375 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbed815e500 msgr2=0x7fbed80aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:22.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbed77fe700 1 --2- 172.21.15.189:0/3409045375 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbed815e500 0x7fbed80aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:22.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbed77fe700 1 -- 172.21.15.189:0/3409045375 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbed806a730 msgr2=0x7fbed8151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:22.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbed77fe700 1 --2- 172.21.15.189:0/3409045375 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbed806a730 0x7fbed8151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:22.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbed77fe700 1 -- 172.21.15.189:0/3409045375 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbecc0094a0 con 0x7fbed815d7b0 2024-04-06T13:49:22.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbed77fe700 1 --2- 172.21.15.189:0/3409045375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed815d7b0 0x7fbed80b12d0 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 crypto rx=0x7fbec800c730 tx=0x7fbec800ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:22.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbed4ff9700 1 -- 172.21.15.189:0/3409045375 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec80052d0 con 0x7fbed815d7b0 2024-04-06T13:49:22.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbed4ff9700 1 -- 172.21.15.189:0/3409045375 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec8005430 con 0x7fbed815d7b0 2024-04-06T13:49:22.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbede56c700 1 -- 172.21.15.189:0/3409045375 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbed80af5d0 con 0x7fbed815d7b0 2024-04-06T13:49:22.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.606+0000 7fbed4ff9700 1 -- 172.21.15.189:0/3409045375 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec80038a0 con 0x7fbed815d7b0 2024-04-06T13:49:22.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.607+0000 7fbede56c700 1 -- 172.21.15.189:0/3409045375 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed80afb90 con 0x7fbed815d7b0 2024-04-06T13:49:22.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.608+0000 7fbede56c700 1 -- 172.21.15.189:0/3409045375 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed80aff00 con 0x7fbed815d7b0 2024-04-06T13:49:22.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.608+0000 7fbed4ff9700 1 -- 172.21.15.189:0/3409045375 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fbec8003a00 con 0x7fbed815d7b0 2024-04-06T13:49:22.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.608+0000 7fbed4ff9700 1 --2- 172.21.15.189:0/3409045375 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbec406aed0 0x7fbec406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:22.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.608+0000 7fbed7fff700 1 --2- 172.21.15.189:0/3409045375 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbec406aed0 0x7fbec406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:22.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.609+0000 7fbed7fff700 1 --2- 172.21.15.189:0/3409045375 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbec406aed0 0x7fbec406d390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0090b0 tx=0x7fbecc008030 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:22.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.609+0000 7fbed4ff9700 1 -- 172.21.15.189:0/3409045375 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbec8057d80 con 0x7fbed815d7b0 2024-04-06T13:49:22.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.614+0000 7fbed4ff9700 1 -- 172.21.15.189:0/3409045375 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbec80ba650 con 0x7fbed815d7b0 2024-04-06T13:49:22.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.853+0000 7fbede56c700 1 -- 172.21.15.189:0/3409045375 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbed8067bb0 con 0x7fbed815d7b0 2024-04-06T13:49:22.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.853+0000 7fbed4ff9700 1 -- 172.21.15.189:0/3409045375 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+4757 (secure 0 0 0) 0x7fbec8017020 con 0x7fbed815d7b0 2024-04-06T13:49:22.854 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:22.854 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":131,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3073866011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3073866011},{"type":"v1","addr":"172.21.15.189:6836","nonce":3073866011}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":9917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/4213774859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4213774859},{"type":"v1","addr":"172.21.15.192:6834","nonce":4213774859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":11392,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/963373121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":963373121},{"type":"v1","addr":"172.21.15.189:6837","nonce":963373121}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:49:17.493194+0000","modified":"2024-04-06T13:49:20.503775+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12885},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12885":{"gid":12885,"name":"b","rank":0,"incarnation":128,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/1094572246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":1094572246},{"type":"v1","addr":"172.21.15.192:6835","nonce":1094572246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-04-06T13:49:22.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.856+0000 7fbede56c700 1 -- 172.21.15.189:0/3409045375 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbec406aed0 msgr2=0x7fbec406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:22.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.856+0000 7fbede56c700 1 --2- 172.21.15.189:0/3409045375 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbec406aed0 0x7fbec406d390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0090b0 tx=0x7fbecc008030 comp rx=0 tx=0).stop 2024-04-06T13:49:22.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.856+0000 7fbede56c700 1 -- 172.21.15.189:0/3409045375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed815d7b0 msgr2=0x7fbed80b12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:22.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.856+0000 7fbede56c700 1 --2- 172.21.15.189:0/3409045375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed815d7b0 0x7fbed80b12d0 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 crypto rx=0x7fbec800c730 tx=0x7fbec800ca40 comp rx=0 tx=0).stop 2024-04-06T13:49:22.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.856+0000 7fbede56c700 1 -- 172.21.15.189:0/3409045375 shutdown_connections 2024-04-06T13:49:22.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.856+0000 7fbede56c700 1 --2- 172.21.15.189:0/3409045375 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbec406aed0 0x7fbec406d390 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:22.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.856+0000 7fbede56c700 1 --2- 172.21.15.189:0/3409045375 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbed815e500 0x7fbed80aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:22.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.856+0000 7fbede56c700 1 --2- 172.21.15.189:0/3409045375 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbed806a730 0x7fbed8151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:22.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.856+0000 7fbede56c700 1 --2- 172.21.15.189:0/3409045375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed815d7b0 0x7fbed80b12d0 unknown :-1 s=CLOSED pgs=2944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:22.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.856+0000 7fbede56c700 1 -- 172.21.15.189:0/3409045375 >> 172.21.15.189:0/3409045375 conn(0x7fbed80d00b0 msgr2=0x7fbed80ae210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:22.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.856+0000 7fbede56c700 1 -- 172.21.15.189:0/3409045375 shutdown_connections 2024-04-06T13:49:22.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:22.856+0000 7fbede56c700 1 -- 172.21.15.189:0/3409045375 wait complete. 2024-04-06T13:49:22.857 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 131 2024-04-06T13:49:23.272 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 131, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-06T13:49:17.493194+0000', 'modified': '2024-04-06T13:49:20.503775+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12885}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12885': {'gid': 12885, 'name': 'b', 'rank': 0, 'incarnation': 128, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.192:6835/1094572246', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6833', 'nonce': 1094572246}, {'type': 'v1', 'addr': '172.21.15.192:6835', 'nonce': 1094572246}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [36], 'metadata_pool': 35, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-06T13:49:23.272 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-06T13:49:23.272 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-06T13:49:23.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.399+0000 7f02398ac700 1 Processor -- start 2024-04-06T13:49:23.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.399+0000 7f02398ac700 1 -- start start 2024-04-06T13:49:23.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.400+0000 7f02398ac700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f02340b6560 0x7f02340ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.400+0000 7f02398ac700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f02340d1500 0x7f02340ce640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.400+0000 7f02398ac700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02340b90a0 con 0x7f023415d470 2024-04-06T13:49:23.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.400+0000 7f02398ac700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02340b6a40 con 0x7f02340b6560 2024-04-06T13:49:23.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.400+0000 7f02398ac700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02340b6d20 con 0x7f02340d1500 2024-04-06T13:49:23.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.400+0000 7f02327fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f02340b6560 0x7f02340ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.400+0000 7f02327fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f02340b6560 0x7f02340ce100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46266/0 (socket says 172.21.15.189:46266) 2024-04-06T13:49:23.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.400+0000 7f02327fc700 1 -- 172.21.15.189:0/1821805018 learned_addr learned my addr 172.21.15.189:0/1821805018 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:23.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.400+0000 7f02337fe700 1 --2- 172.21.15.189:0/1821805018 >> v2:172.21.15.192:3300/0 conn(0x7f02340d1500 0x7f02340ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.400+0000 7f02327fc700 1 -- 172.21.15.189:0/1821805018 >> v2:172.21.15.192:3300/0 conn(0x7f02340d1500 msgr2=0x7f02340ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:23.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.400+0000 7f02327fc700 1 --2- 172.21.15.189:0/1821805018 >> v2:172.21.15.192:3300/0 conn(0x7f02340d1500 0x7f02340ce640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.400+0000 7f02327fc700 1 -- 172.21.15.189:0/1821805018 >> v1:172.21.15.192:6789/0 conn(0x7f023415d470 legacy=0x7f02340d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:23.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.400+0000 7f02327fc700 1 -- 172.21.15.189:0/1821805018 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02340b7000 con 0x7f02340b6560 2024-04-06T13:49:23.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.400+0000 7f02327fc700 1 --2- 172.21.15.189:0/1821805018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02340b6560 0x7f02340ce100 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 crypto rx=0x7f02340c6d80 tx=0x7f0228009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e4178a13fab7b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:23.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.400+0000 7f0231ffb700 1 -- 172.21.15.189:0/1821805018 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0228011830 con 0x7f02340b6560 2024-04-06T13:49:23.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.400+0000 7f0231ffb700 1 -- 172.21.15.189:0/1821805018 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0228011990 con 0x7f02340b6560 2024-04-06T13:49:23.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.400+0000 7f0231ffb700 1 -- 172.21.15.189:0/1821805018 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0228011d20 con 0x7f02340b6560 2024-04-06T13:49:23.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.401+0000 7f02398ac700 1 -- 172.21.15.189:0/1821805018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02340b6560 msgr2=0x7f02340ce100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:23.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.401+0000 7f02398ac700 1 --2- 172.21.15.189:0/1821805018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02340b6560 0x7f02340ce100 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 crypto rx=0x7f02340c6d80 tx=0x7f0228009bd0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.401+0000 7f02398ac700 1 -- 172.21.15.189:0/1821805018 shutdown_connections 2024-04-06T13:49:23.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.401+0000 7f02398ac700 1 --2- 172.21.15.189:0/1821805018 >> v2:172.21.15.192:3300/0 conn(0x7f02340d1500 0x7f02340ce640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.401+0000 7f02398ac700 1 --2- 172.21.15.189:0/1821805018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02340b6560 0x7f02340ce100 unknown :-1 s=CLOSED pgs=2945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.401+0000 7f02398ac700 1 -- 172.21.15.189:0/1821805018 >> 172.21.15.189:0/1821805018 conn(0x7f023406bbc0 msgr2=0x7f023406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:23.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.401+0000 7f02398ac700 1 -- 172.21.15.189:0/1821805018 shutdown_connections 2024-04-06T13:49:23.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.401+0000 7f02398ac700 1 -- 172.21.15.189:0/1821805018 wait complete. 2024-04-06T13:49:23.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.402+0000 7f02398ac700 1 Processor -- start 2024-04-06T13:49:23.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.402+0000 7f02398ac700 1 -- start start 2024-04-06T13:49:23.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.403+0000 7f02398ac700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02340b6560 0x7f02341486d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.403+0000 7f02398ac700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02340d1500 0x7f02340c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.403+0000 7f02398ac700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f023415d470 0x7f02340c7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.403+0000 7f02398ac700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02340c9c60 con 0x7f02340d1500 2024-04-06T13:49:23.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.403+0000 7f02398ac700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02340d2f00 con 0x7f023415d470 2024-04-06T13:49:23.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.403+0000 7f02398ac700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02340d31e0 con 0x7f02340b6560 2024-04-06T13:49:23.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.403+0000 7f02327fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02340d1500 0x7f02340c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.403+0000 7f02327fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02340d1500 0x7f02340c7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46268/0 (socket says 172.21.15.189:46268) 2024-04-06T13:49:23.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.403+0000 7f02327fc700 1 -- 172.21.15.189:0/1466833571 learned_addr learned my addr 172.21.15.189:0/1466833571 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:23.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.403+0000 7f02337fe700 1 --2- 172.21.15.189:0/1466833571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f023415d470 0x7f02340c7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.403+0000 7f02327fc700 1 -- 172.21.15.189:0/1466833571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02340b6560 msgr2=0x7f02341486d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:23.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.403+0000 7f0232ffd700 1 --2- 172.21.15.189:0/1466833571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02340b6560 0x7f02341486d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.403+0000 7f02327fc700 1 --2- 172.21.15.189:0/1466833571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02340b6560 0x7f02341486d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.403+0000 7f02327fc700 1 -- 172.21.15.189:0/1466833571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f023415d470 msgr2=0x7f02340c7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:23.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.403+0000 7f02327fc700 1 --2- 172.21.15.189:0/1466833571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f023415d470 0x7f02340c7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.404+0000 7f02327fc700 1 -- 172.21.15.189:0/1466833571 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0228009550 con 0x7f02340d1500 2024-04-06T13:49:23.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.404+0000 7f02337fe700 1 --2- 172.21.15.189:0/1466833571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f023415d470 0x7f02340c7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:23.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.404+0000 7f02327fc700 1 --2- 172.21.15.189:0/1466833571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02340d1500 0x7f02340c7750 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 crypto rx=0x7f0228000c00 tx=0x7f0228004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:23.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.404+0000 7f021bfff700 1 -- 172.21.15.189:0/1466833571 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0228004570 con 0x7f02340d1500 2024-04-06T13:49:23.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.404+0000 7f021bfff700 1 -- 172.21.15.189:0/1466833571 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022801f070 con 0x7f02340d1500 2024-04-06T13:49:23.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.404+0000 7f02398ac700 1 -- 172.21.15.189:0/1466833571 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02341543f0 con 0x7f02340d1500 2024-04-06T13:49:23.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.404+0000 7f021bfff700 1 -- 172.21.15.189:0/1466833571 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02280038e0 con 0x7f02340d1500 2024-04-06T13:49:23.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.404+0000 7f02398ac700 1 -- 172.21.15.189:0/1466833571 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02341549b0 con 0x7f02340d1500 2024-04-06T13:49:23.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.405+0000 7f021bfff700 1 -- 172.21.15.189:0/1466833571 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f0228021030 con 0x7f02340d1500 2024-04-06T13:49:23.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.406+0000 7f021bfff700 1 --2- 172.21.15.189:0/1466833571 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f022006aed0 0x7f022006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.406+0000 7f0232ffd700 1 --2- 172.21.15.189:0/1466833571 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f022006aed0 0x7f022006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.406+0000 7f021bfff700 1 -- 172.21.15.189:0/1466833571 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f022808ada0 con 0x7f02340d1500 2024-04-06T13:49:23.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.406+0000 7f0232ffd700 1 --2- 172.21.15.189:0/1466833571 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f022006aed0 0x7f022006d390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0224000f60 tx=0x7f0224009040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:23.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.408+0000 7f0245a60b80 1 Processor -- start 2024-04-06T13:49:23.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.408+0000 7f0245a60b80 1 -- start start 2024-04-06T13:49:23.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f0245a60b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe80c20 0x55daafe84a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f0245a60b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe85fc0 0x55daafe5c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f0245a60b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe5b980 0x55daafe5cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f0245a60b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55daafdb5940 con 0x55daafe80c20 2024-04-06T13:49:23.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f0245a60b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55daafdb3300 con 0x55daafe5b980 2024-04-06T13:49:23.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f0245a60b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55daafdaebf0 con 0x55daafe85fc0 2024-04-06T13:49:23.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f0232ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe80c20 0x55daafe84a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f0232ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe80c20 0x55daafe84a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46280/0 (socket says 172.21.15.189:46280) 2024-04-06T13:49:23.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f0232ffd700 1 -- 172.21.15.189:0/3888948618 learned_addr learned my addr 172.21.15.189:0/3888948618 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:23.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f02337fe700 1 --2- 172.21.15.189:0/3888948618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe5b980 0x55daafe5cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f02327fc700 1 --2- 172.21.15.189:0/3888948618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe85fc0 0x55daafe5c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f0232ffd700 1 -- 172.21.15.189:0/3888948618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe85fc0 msgr2=0x55daafe5c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:23.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f0232ffd700 1 --2- 172.21.15.189:0/3888948618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe85fc0 0x55daafe5c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f0232ffd700 1 -- 172.21.15.189:0/3888948618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe5b980 msgr2=0x55daafe5cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:23.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f0232ffd700 1 --2- 172.21.15.189:0/3888948618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe5b980 0x55daafe5cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f0232ffd700 1 -- 172.21.15.189:0/3888948618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55daafbdcb10 con 0x55daafe80c20 2024-04-06T13:49:23.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f0232ffd700 1 --2- 172.21.15.189:0/3888948618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe80c20 0x55daafe84a60 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 crypto rx=0x7f0224005360 tx=0x7f0224010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deec46eb589a362 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:23.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f02197fa700 1 -- 172.21.15.189:0/3888948618 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022401c070 con 0x55daafe80c20 2024-04-06T13:49:23.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f02197fa700 1 -- 172.21.15.189:0/3888948618 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0224003d40 con 0x55daafe80c20 2024-04-06T13:49:23.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.409+0000 7f02197fa700 1 -- 172.21.15.189:0/3888948618 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0224021610 con 0x55daafe80c20 2024-04-06T13:49:23.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.410+0000 7f0245a60b80 1 -- 172.21.15.189:0/3888948618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe80c20 msgr2=0x55daafe84a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:23.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.410+0000 7f0245a60b80 1 --2- 172.21.15.189:0/3888948618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe80c20 0x55daafe84a60 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 crypto rx=0x7f0224005360 tx=0x7f0224010000 comp rx=0 tx=0).stop 2024-04-06T13:49:23.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.410+0000 7f0245a60b80 1 -- 172.21.15.189:0/3888948618 shutdown_connections 2024-04-06T13:49:23.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.410+0000 7f0245a60b80 1 --2- 172.21.15.189:0/3888948618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe5b980 0x55daafe5cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.410+0000 7f0245a60b80 1 --2- 172.21.15.189:0/3888948618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe80c20 0x55daafe84a60 unknown :-1 s=CLOSED pgs=2947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.410+0000 7f0245a60b80 1 --2- 172.21.15.189:0/3888948618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe85fc0 0x55daafe5c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.410+0000 7f0245a60b80 1 -- 172.21.15.189:0/3888948618 >> 172.21.15.189:0/3888948618 conn(0x55daafdbc650 msgr2=0x55daafea33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:23.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.410+0000 7f0245a60b80 1 -- 172.21.15.189:0/3888948618 shutdown_connections 2024-04-06T13:49:23.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.410+0000 7f0245a60b80 1 -- 172.21.15.189:0/3888948618 wait complete. 2024-04-06T13:49:23.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.411+0000 7f0245a60b80 1 -- start start 2024-04-06T13:49:23.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.411+0000 7f0245a60b80 1 Processor -- start 2024-04-06T13:49:23.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f0245a60b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe5b980 0x55daafe5e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f0245a60b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe80c20 0x55daafeb2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f0245a60b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe85fc0 0x55daafe9b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f0245a60b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55daafd4ad30 con 0x55daafe85fc0 2024-04-06T13:49:23.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f0245a60b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55daafd4a9d0 con 0x55daafe5b980 2024-04-06T13:49:23.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f0245a60b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55daafc32650 con 0x55daafe80c20 2024-04-06T13:49:23.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f0245a60b80 10 client.0 _ll_register_callbacks cb 0x55daafd54420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T13:49:23.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f02337fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe85fc0 0x55daafe9b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f02337fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe85fc0 0x55daafe9b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46292/0 (socket says 172.21.15.189:46292) 2024-04-06T13:49:23.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f02337fe700 1 -- 172.21.15.189:0/3036829742 learned_addr learned my addr 172.21.15.189:0/3036829742 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:23.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f02327fc700 1 --2- 172.21.15.189:0/3036829742 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe80c20 0x55daafeb2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f0232ffd700 1 --2- 172.21.15.189:0/3036829742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe5b980 0x55daafe5e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f02337fe700 1 -- 172.21.15.189:0/3036829742 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe80c20 msgr2=0x55daafeb2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:23.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f02337fe700 1 --2- 172.21.15.189:0/3036829742 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe80c20 0x55daafeb2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f02337fe700 1 -- 172.21.15.189:0/3036829742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe5b980 msgr2=0x55daafe5e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:23.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f02337fe700 1 --2- 172.21.15.189:0/3036829742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe5b980 0x55daafe5e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f02337fe700 1 -- 172.21.15.189:0/3036829742 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02340b90a0 con 0x55daafe85fc0 2024-04-06T13:49:23.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f02337fe700 1 --2- 172.21.15.189:0/3036829742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe85fc0 0x55daafe9b620 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 crypto rx=0x7f022c007da0 tx=0x7f022c012520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c934bb5ea522e3b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:23.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f02127fc700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T13:49:23.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f02127fc700 1 -- 172.21.15.189:0/3036829742 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c012c00 con 0x55daafe85fc0 2024-04-06T13:49:23.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f02127fc700 1 -- 172.21.15.189:0/3036829742 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(269..269 src has 1..269) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f022c012d60 con 0x55daafe85fc0 2024-04-06T13:49:23.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.412+0000 7f0245a60b80 1 -- 172.21.15.189:0/3036829742 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55daafd6f330 con 0x55daafe85fc0 2024-04-06T13:49:23.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.413+0000 7f02127fc700 1 -- 172.21.15.189:0/3036829742 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=131) v2 ==== 24+0+0 (secure 0 0 0) 0x7f022c018470 con 0x55daafe85fc0 2024-04-06T13:49:23.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.413+0000 7f0245a60b80 10 client.13017 fetch_fsmap learned FSMap version 131 2024-04-06T13:49:23.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.413+0000 7f0245a60b80 1 -- 172.21.15.189:0/3036829742 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=131,monmap=2+}) v3 -- 0x55daafd74e50 con 0x55daafe85fc0 2024-04-06T13:49:23.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.413+0000 7f02127fc700 1 -- 172.21.15.189:0/3036829742 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55daafd6f330 con 0x55daafe85fc0 2024-04-06T13:49:23.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.413+0000 7f02127fc700 1 -- 172.21.15.189:0/3036829742 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 131) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f022c014040 con 0x55daafe85fc0 2024-04-06T13:49:23.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.413+0000 7f0245a60b80 10 client.13017 fetch_fsmap finished waiting for FSMap version 131 2024-04-06T13:49:23.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.413+0000 7f0245a60b80 10 client.13017 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-06T13:49:23.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.413+0000 7f0245a60b80 20 client.13017 populate_metadata read hostname 'smithi189' 2024-04-06T13:49:23.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.413+0000 7f0245a60b80 1 --2- 172.21.15.189:0/3036829742 >> [v2:172.21.15.192:6833/1094572246,v1:172.21.15.192:6835/1094572246] conn(0x55daafe7f1b0 0x55daafe89580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.413+0000 7f0245a60b80 4 client.13017 mds_command: new command op to 12885 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-06T13:49:23.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.413+0000 7f0245a60b80 1 -- 172.21.15.189:0/3036829742 --> [v2:172.21.15.192:6833/1094572246,v1:172.21.15.192:6835/1094572246] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55daafda76b0 con 0x55daafe7f1b0 2024-04-06T13:49:23.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.413+0000 7f0232ffd700 1 --2- 172.21.15.189:0/3036829742 >> [v2:172.21.15.192:6833/1094572246,v1:172.21.15.192:6835/1094572246] conn(0x55daafe7f1b0 0x55daafe89580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.413+0000 7f02127fc700 10 client.13017 ms_handle_connect on v2:172.21.15.192:6833/1094572246 2024-04-06T13:49:23.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.413+0000 7f0232ffd700 1 --2- 172.21.15.189:0/3036829742 >> [v2:172.21.15.192:6833/1094572246,v1:172.21.15.192:6835/1094572246] conn(0x55daafe7f1b0 0x55daafe89580 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b776f6ac68a05cbe server_cookie=86d3073d75b6e61e in_seq=0 out_seq=0 2024-04-06T13:49:23.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.414+0000 7f02127fc700 1 -- 172.21.15.189:0/3036829742 <== mds.0 v2:172.21.15.192:6833/1094572246 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55daafda76b0 con 0x55daafe7f1b0 2024-04-06T13:49:23.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.414+0000 7f02127fc700 10 client.13017 handle_command_reply: tid=0 2024-04-06T13:49:23.420 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.414+0000 7f0245a60b80 1 client.13017 shutdown 2024-04-06T13:49:23.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.414+0000 7f0245a60b80 1 -- 172.21.15.189:0/3036829742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe85fc0 msgr2=0x55daafe9b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:23.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.414+0000 7f0245a60b80 1 --2- 172.21.15.189:0/3036829742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe85fc0 0x55daafe9b620 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 crypto rx=0x7f022c007da0 tx=0x7f022c012520 comp rx=0 tx=0).stop 2024-04-06T13:49:23.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.414+0000 7f0245a60b80 1 -- 172.21.15.189:0/3036829742 shutdown_connections 2024-04-06T13:49:23.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.414+0000 7f0245a60b80 1 --2- 172.21.15.189:0/3036829742 >> [v2:172.21.15.192:6833/1094572246,v1:172.21.15.192:6835/1094572246] conn(0x55daafe7f1b0 0x55daafe89580 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.414+0000 7f0245a60b80 1 --2- 172.21.15.189:0/3036829742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe85fc0 0x55daafe9b620 unknown :-1 s=CLOSED pgs=2948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.414+0000 7f02127fc700 0 client.13017 ms_handle_reset on v2:172.21.15.192:6833/1094572246 2024-04-06T13:49:23.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.414+0000 7f0245a60b80 1 --2- 172.21.15.189:0/3036829742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe5b980 0x55daafe5e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.414+0000 7f0245a60b80 1 --2- 172.21.15.189:0/3036829742 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe80c20 0x55daafeb2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.414+0000 7f0245a60b80 1 -- 172.21.15.189:0/3036829742 >> 172.21.15.189:0/3036829742 conn(0x55daafdbc650 msgr2=0x55daafe5c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:23.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.414+0000 7f0245a60b80 1 -- 172.21.15.189:0/3036829742 shutdown_connections 2024-04-06T13:49:23.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.414+0000 7f0245a60b80 1 -- 172.21.15.189:0/3036829742 wait complete. 2024-04-06T13:49:23.421 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.415+0000 7f0245a60b80 20 client.13017 trim_cache size 0 max 16384 2024-04-06T13:49:23.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.432+0000 7f0245a60b80 1 Processor -- start 2024-04-06T13:49:23.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.432+0000 7f0245a60b80 1 -- start start 2024-04-06T13:49:23.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.433+0000 7f0245a60b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe5b980 0x55daaff01320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.433+0000 7f0245a60b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe7f1b0 0x55daafe5c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.433+0000 7f0245a60b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe80c20 0x55daafe5c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.433+0000 7f0245a60b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55daafe9ce50 con 0x55daafe5b980 2024-04-06T13:49:23.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.433+0000 7f0245a60b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55daafe58cf0 con 0x55daafe80c20 2024-04-06T13:49:23.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.433+0000 7f0245a60b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55daafe58600 con 0x55daafe7f1b0 2024-04-06T13:49:23.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.433+0000 7f0232ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe5b980 0x55daaff01320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.433+0000 7f0232ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe5b980 0x55daaff01320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46306/0 (socket says 172.21.15.189:46306) 2024-04-06T13:49:23.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.433+0000 7f0232ffd700 1 -- 172.21.15.189:0/1214798164 learned_addr learned my addr 172.21.15.189:0/1214798164 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:23.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.433+0000 7f02337fe700 1 --2- 172.21.15.189:0/1214798164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe80c20 0x55daafe5c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.433+0000 7f02327fc700 1 --2- 172.21.15.189:0/1214798164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe7f1b0 0x55daafe5c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.434+0000 7f0232ffd700 1 -- 172.21.15.189:0/1214798164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe7f1b0 msgr2=0x55daafe5c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:23.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.434+0000 7f0232ffd700 1 --2- 172.21.15.189:0/1214798164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe7f1b0 0x55daafe5c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.434+0000 7f0232ffd700 1 -- 172.21.15.189:0/1214798164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe80c20 msgr2=0x55daafe5c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:23.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.434+0000 7f0232ffd700 1 --2- 172.21.15.189:0/1214798164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe80c20 0x55daafe5c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.434+0000 7f0232ffd700 1 -- 172.21.15.189:0/1214798164 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55daafe7f860 con 0x55daafe5b980 2024-04-06T13:49:23.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.434+0000 7f02337fe700 1 --2- 172.21.15.189:0/1214798164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe80c20 0x55daafe5c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:23.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.434+0000 7f02327fc700 1 --2- 172.21.15.189:0/1214798164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe7f1b0 0x55daafe5c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:23.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.434+0000 7f0232ffd700 1 --2- 172.21.15.189:0/1214798164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe5b980 0x55daaff01320 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7f0224003fd0 tx=0x7f0224005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e151dffc874cadbf server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:23.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.434+0000 7f0219ffb700 1 -- 172.21.15.189:0/1214798164 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022402c5b0 con 0x55daafe5b980 2024-04-06T13:49:23.436 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.434+0000 7f0219ffb700 1 -- 172.21.15.189:0/1214798164 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0224027040 con 0x55daafe5b980 2024-04-06T13:49:23.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.434+0000 7f0219ffb700 1 -- 172.21.15.189:0/1214798164 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022402cb30 con 0x55daafe5b980 2024-04-06T13:49:23.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.434+0000 7f0245a60b80 1 -- 172.21.15.189:0/1214798164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe5b980 msgr2=0x55daaff01320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:23.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.434+0000 7f0245a60b80 1 --2- 172.21.15.189:0/1214798164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe5b980 0x55daaff01320 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7f0224003fd0 tx=0x7f0224005000 comp rx=0 tx=0).stop 2024-04-06T13:49:23.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.435+0000 7f0245a60b80 1 -- 172.21.15.189:0/1214798164 shutdown_connections 2024-04-06T13:49:23.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.435+0000 7f0245a60b80 1 --2- 172.21.15.189:0/1214798164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe80c20 0x55daafe5c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.435+0000 7f0245a60b80 1 --2- 172.21.15.189:0/1214798164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe5b980 0x55daaff01320 unknown :-1 s=CLOSED pgs=2949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.435+0000 7f0245a60b80 1 --2- 172.21.15.189:0/1214798164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe7f1b0 0x55daafe5c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.435+0000 7f0245a60b80 1 -- 172.21.15.189:0/1214798164 >> 172.21.15.189:0/1214798164 conn(0x55daafdbc650 msgr2=0x55daafe9b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:23.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.435+0000 7f0245a60b80 1 -- 172.21.15.189:0/1214798164 shutdown_connections 2024-04-06T13:49:23.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.435+0000 7f0245a60b80 1 -- 172.21.15.189:0/1214798164 wait complete. 2024-04-06T13:49:23.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.435+0000 7f0245a60b80 1 -- start start 2024-04-06T13:49:23.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.435+0000 7f0245a60b80 1 Processor -- start 2024-04-06T13:49:23.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f0245a60b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe5b980 0x55daafe7b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f0245a60b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe7f1b0 0x55daafea20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f0245a60b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe80c20 0x55daaff01320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.441 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f0245a60b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55daafe7bae0 con 0x55daafe80c20 2024-04-06T13:49:23.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f0245a60b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55daaff020b0 con 0x55daafe5b980 2024-04-06T13:49:23.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f0245a60b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55daafefde40 con 0x55daafe7f1b0 2024-04-06T13:49:23.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f0245a60b80 10 client.0 _ll_register_callbacks cb 0x55daafd54420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T13:49:23.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f02337fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe80c20 0x55daaff01320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f02337fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe80c20 0x55daaff01320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46316/0 (socket says 172.21.15.189:46316) 2024-04-06T13:49:23.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f02337fe700 1 -- 172.21.15.189:0/1765054520 learned_addr learned my addr 172.21.15.189:0/1765054520 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:23.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f0232ffd700 1 --2- 172.21.15.189:0/1765054520 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe5b980 0x55daafe7b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f02327fc700 1 --2- 172.21.15.189:0/1765054520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe7f1b0 0x55daafea20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f02337fe700 1 -- 172.21.15.189:0/1765054520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe7f1b0 msgr2=0x55daafea20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:23.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f02337fe700 1 --2- 172.21.15.189:0/1765054520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe7f1b0 0x55daafea20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f02337fe700 1 -- 172.21.15.189:0/1765054520 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe5b980 msgr2=0x55daafe7b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:23.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f02337fe700 1 --2- 172.21.15.189:0/1765054520 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe5b980 0x55daafe7b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f02337fe700 1 -- 172.21.15.189:0/1765054520 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55daafbdcb10 con 0x55daafe80c20 2024-04-06T13:49:23.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f02327fc700 1 --2- 172.21.15.189:0/1765054520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe7f1b0 0x55daafea20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:23.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f02337fe700 1 --2- 172.21.15.189:0/1765054520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe80c20 0x55daaff01320 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 crypto rx=0x7f022c009b90 tx=0x7f022c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b2870186b0e4446 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:23.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.436+0000 7f02127fc700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T13:49:23.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.437+0000 7f02127fc700 1 -- 172.21.15.189:0/1765054520 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c018960 con 0x55daafe80c20 2024-04-06T13:49:23.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.437+0000 7f02127fc700 1 -- 172.21.15.189:0/1765054520 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(269..269 src has 1..269) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f022c034cb0 con 0x55daafe80c20 2024-04-06T13:49:23.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.437+0000 7f0245a60b80 1 -- 172.21.15.189:0/1765054520 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0224003d40 con 0x55daafe80c20 2024-04-06T13:49:23.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.437+0000 7f02127fc700 1 -- 172.21.15.189:0/1765054520 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=131) v2 ==== 24+0+0 (secure 0 0 0) 0x7f022c034e70 con 0x55daafe80c20 2024-04-06T13:49:23.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.437+0000 7f0245a60b80 10 client.13023 fetch_fsmap learned FSMap version 131 2024-04-06T13:49:23.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.437+0000 7f0245a60b80 1 -- 172.21.15.189:0/1765054520 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=131,monmap=2+}) v3 -- 0x55daafc34c80 con 0x55daafe80c20 2024-04-06T13:49:23.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.437+0000 7f02127fc700 1 -- 172.21.15.189:0/1765054520 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0224003d40 con 0x55daafe80c20 2024-04-06T13:49:23.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.437+0000 7f02127fc700 1 -- 172.21.15.189:0/1765054520 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 131) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f022c012240 con 0x55daafe80c20 2024-04-06T13:49:23.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.437+0000 7f0245a60b80 10 client.13023 fetch_fsmap finished waiting for FSMap version 131 2024-04-06T13:49:23.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.437+0000 7f0245a60b80 10 client.13023 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-06T13:49:23.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.437+0000 7f0245a60b80 20 client.13023 populate_metadata read hostname 'smithi189' 2024-04-06T13:49:23.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.437+0000 7f0245a60b80 1 --2- 172.21.15.189:0/1765054520 >> [v2:172.21.15.192:6833/1094572246,v1:172.21.15.192:6835/1094572246] conn(0x55daafe85fc0 0x55daafea11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.437+0000 7f0245a60b80 4 client.13023 mds_command: new command op to 12885 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-06T13:49:23.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.437+0000 7f0245a60b80 1 -- 172.21.15.189:0/1765054520 --> [v2:172.21.15.192:6833/1094572246,v1:172.21.15.192:6835/1094572246] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55daafbfdca0 con 0x55daafe85fc0 2024-04-06T13:49:23.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.437+0000 7f0232ffd700 1 --2- 172.21.15.189:0/1765054520 >> [v2:172.21.15.192:6833/1094572246,v1:172.21.15.192:6835/1094572246] conn(0x55daafe85fc0 0x55daafea11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.438+0000 7f02127fc700 10 client.13023 ms_handle_connect on v2:172.21.15.192:6833/1094572246 2024-04-06T13:49:23.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.438+0000 7f0232ffd700 1 --2- 172.21.15.189:0/1765054520 >> [v2:172.21.15.192:6833/1094572246,v1:172.21.15.192:6835/1094572246] conn(0x55daafe85fc0 0x55daafea11d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc96f500630575fe server_cookie=c7178f489b24e12a in_seq=0 out_seq=0 2024-04-06T13:49:23.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.438+0000 7f02127fc700 1 -- 172.21.15.189:0/1765054520 <== mds.0 v2:172.21.15.192:6833/1094572246 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55daafbfdca0 con 0x55daafe85fc0 2024-04-06T13:49:23.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.438+0000 7f02127fc700 10 client.13023 handle_command_reply: tid=0 2024-04-06T13:49:23.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.438+0000 7f0245a60b80 1 client.13023 shutdown 2024-04-06T13:49:23.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.438+0000 7f0245a60b80 1 -- 172.21.15.189:0/1765054520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe80c20 msgr2=0x55daaff01320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:23.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.438+0000 7f0245a60b80 1 --2- 172.21.15.189:0/1765054520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe80c20 0x55daaff01320 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 crypto rx=0x7f022c009b90 tx=0x7f022c004ad0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.438+0000 7f0245a60b80 1 -- 172.21.15.189:0/1765054520 shutdown_connections 2024-04-06T13:49:23.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.438+0000 7f0245a60b80 1 --2- 172.21.15.189:0/1765054520 >> [v2:172.21.15.192:6833/1094572246,v1:172.21.15.192:6835/1094572246] conn(0x55daafe85fc0 0x55daafea11d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.438+0000 7f0245a60b80 1 --2- 172.21.15.189:0/1765054520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55daafe80c20 0x55daaff01320 unknown :-1 s=CLOSED pgs=2950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.438+0000 7f0245a60b80 1 --2- 172.21.15.189:0/1765054520 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55daafe5b980 0x55daafe7b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.438+0000 7f02127fc700 0 client.13023 ms_handle_reset on v2:172.21.15.192:6833/1094572246 2024-04-06T13:49:23.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.438+0000 7f0245a60b80 1 --2- 172.21.15.189:0/1765054520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55daafe7f1b0 0x55daafea20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.438+0000 7f0245a60b80 1 -- 172.21.15.189:0/1765054520 >> 172.21.15.189:0/1765054520 conn(0x55daafdbc650 msgr2=0x55daafea3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:23.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.438+0000 7f0245a60b80 1 -- 172.21.15.189:0/1765054520 shutdown_connections 2024-04-06T13:49:23.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.438+0000 7f0245a60b80 1 -- 172.21.15.189:0/1765054520 wait complete. 2024-04-06T13:49:23.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.438+0000 7f0245a60b80 20 client.13023 trim_cache size 0 max 16384 2024-04-06T13:49:23.445 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:49:23.445 INFO:teuthology.orchestra.run.smithi189.stdout: "cluster_fsid": "e30041de-e9a3-4351-8ca7-29d7f3e3728c", 2024-04-06T13:49:23.445 INFO:teuthology.orchestra.run.smithi189.stdout: "whoami": 0, 2024-04-06T13:49:23.445 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 12885, 2024-04-06T13:49:23.445 INFO:teuthology.orchestra.run.smithi189.stdout: "want_state": "up:active", 2024-04-06T13:49:23.445 INFO:teuthology.orchestra.run.smithi189.stdout: "state": "up:active", 2024-04-06T13:49:23.445 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs", 2024-04-06T13:49:23.446 INFO:teuthology.orchestra.run.smithi189.stdout: "rank_uptime": 5.9430737320000002, 2024-04-06T13:49:23.446 INFO:teuthology.orchestra.run.smithi189.stdout: "mdsmap_epoch": 131, 2024-04-06T13:49:23.446 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch": 269, 2024-04-06T13:49:23.446 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch_barrier": 269, 2024-04-06T13:49:23.446 INFO:teuthology.orchestra.run.smithi189.stdout: "uptime": 10.405732217000001 2024-04-06T13:49:23.446 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:49:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.439+0000 7f02398ac700 1 -- 172.21.15.189:0/1466833571 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f022006aed0 msgr2=0x7f022006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.439+0000 7f02398ac700 1 --2- 172.21.15.189:0/1466833571 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f022006aed0 0x7f022006d390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0224000f60 tx=0x7f0224009040 comp rx=0 tx=0).stop 2024-04-06T13:49:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.439+0000 7f02398ac700 1 -- 172.21.15.189:0/1466833571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02340d1500 msgr2=0x7f02340c7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.439+0000 7f02398ac700 1 --2- 172.21.15.189:0/1466833571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02340d1500 0x7f02340c7750 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 crypto rx=0x7f0228000c00 tx=0x7f0228004750 comp rx=0 tx=0).stop 2024-04-06T13:49:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.439+0000 7f02398ac700 1 -- 172.21.15.189:0/1466833571 shutdown_connections 2024-04-06T13:49:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.439+0000 7f02398ac700 1 --2- 172.21.15.189:0/1466833571 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f022006aed0 0x7f022006d390 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.439+0000 7f02398ac700 1 --2- 172.21.15.189:0/1466833571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f023415d470 0x7f02340c7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.439+0000 7f02398ac700 1 --2- 172.21.15.189:0/1466833571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02340b6560 0x7f02341486d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.439+0000 7f02398ac700 1 --2- 172.21.15.189:0/1466833571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02340d1500 0x7f02340c7750 unknown :-1 s=CLOSED pgs=2946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.439+0000 7f02398ac700 1 -- 172.21.15.189:0/1466833571 >> 172.21.15.189:0/1466833571 conn(0x7f023406bbc0 msgr2=0x7f0234072790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:23.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.439+0000 7f02398ac700 1 -- 172.21.15.189:0/1466833571 shutdown_connections 2024-04-06T13:49:23.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.439+0000 7f02398ac700 1 -- 172.21.15.189:0/1466833571 wait complete. 2024-04-06T13:49:23.451 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T13:49:23.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.572+0000 7f13d6aa1700 1 Processor -- start 2024-04-06T13:49:23.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.572+0000 7f13d6aa1700 1 -- start start 2024-04-06T13:49:23.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.573+0000 7f13d6aa1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13d015a550 0x7f13d00cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.573+0000 7f13d6aa1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f13d00b4d70 0x7f13d00b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.573+0000 7f13d6aa1700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d00cdbe0 con 0x7f13d00cf900 2024-04-06T13:49:23.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.573+0000 7f13d6aa1700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13d015a940 con 0x7f13d00b4d70 2024-04-06T13:49:23.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.573+0000 7f13d6aa1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d015ac00 con 0x7f13d015a550 2024-04-06T13:49:23.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.573+0000 7f13d503e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f13d00b4d70 0x7f13d00b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.573+0000 7f13d503e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f13d00b4d70 0x7f13d00b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46332/0 (socket says 172.21.15.189:46332) 2024-04-06T13:49:23.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.573+0000 7f13d503e700 1 -- 172.21.15.189:0/1235391144 learned_addr learned my addr 172.21.15.189:0/1235391144 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:23.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.573+0000 7f13d483d700 1 --2- 172.21.15.189:0/1235391144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13d015a550 0x7f13d00cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.574+0000 7f13d503e700 1 -- 172.21.15.189:0/1235391144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13d015a550 msgr2=0x7f13d00cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:23.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.574+0000 7f13d503e700 1 --2- 172.21.15.189:0/1235391144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13d015a550 0x7f13d00cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.574+0000 7f13d503e700 1 -- 172.21.15.189:0/1235391144 >> v1:172.21.15.192:6789/0 conn(0x7f13d00cf900 legacy=0x7f13d00b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:23.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.574+0000 7f13d503e700 1 -- 172.21.15.189:0/1235391144 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d015aee0 con 0x7f13d00b4d70 2024-04-06T13:49:23.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.574+0000 7f13d503e700 1 --2- 172.21.15.189:0/1235391144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13d00b4d70 0x7f13d00b2fc0 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 crypto rx=0x7f13c800d150 tx=0x7f13c800d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d516489202e1c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:23.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.574+0000 7f13cf7fe700 1 -- 172.21.15.189:0/1235391144 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c8010070 con 0x7f13d00b4d70 2024-04-06T13:49:23.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.574+0000 7f13cf7fe700 1 -- 172.21.15.189:0/1235391144 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c800dd10 con 0x7f13d00b4d70 2024-04-06T13:49:23.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.574+0000 7f13cf7fe700 1 -- 172.21.15.189:0/1235391144 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c800b310 con 0x7f13d00b4d70 2024-04-06T13:49:23.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.574+0000 7f13d6aa1700 1 -- 172.21.15.189:0/1235391144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13d00b4d70 msgr2=0x7f13d00b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:23.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.574+0000 7f13d6aa1700 1 --2- 172.21.15.189:0/1235391144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13d00b4d70 0x7f13d00b2fc0 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 crypto rx=0x7f13c800d150 tx=0x7f13c800d460 comp rx=0 tx=0).stop 2024-04-06T13:49:23.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.575+0000 7f13d6aa1700 1 -- 172.21.15.189:0/1235391144 shutdown_connections 2024-04-06T13:49:23.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.575+0000 7f13d6aa1700 1 --2- 172.21.15.189:0/1235391144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13d00b4d70 0x7f13d00b2fc0 unknown :-1 s=CLOSED pgs=2951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.575+0000 7f13d6aa1700 1 --2- 172.21.15.189:0/1235391144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13d015a550 0x7f13d00cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.575+0000 7f13d6aa1700 1 -- 172.21.15.189:0/1235391144 >> 172.21.15.189:0/1235391144 conn(0x7f13d00d00b0 msgr2=0x7f13d00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:23.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.575+0000 7f13d6aa1700 1 -- 172.21.15.189:0/1235391144 shutdown_connections 2024-04-06T13:49:23.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.575+0000 7f13d6aa1700 1 -- 172.21.15.189:0/1235391144 wait complete. 2024-04-06T13:49:23.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.576+0000 7f13d6aa1700 1 Processor -- start 2024-04-06T13:49:23.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.576+0000 7f13d6aa1700 1 -- start start 2024-04-06T13:49:23.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.577+0000 7f13d6aa1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13d00b4d70 0x7f13d014c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.577+0000 7f13d6aa1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13d00cf900 0x7f13d014d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.577+0000 7f13d6aa1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13d013d2e0 0x7f13d006eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.577+0000 7f13d6aa1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d014df00 con 0x7f13d00cf900 2024-04-06T13:49:23.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.577+0000 7f13d6aa1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d013a5e0 con 0x7f13d013d2e0 2024-04-06T13:49:23.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.577+0000 7f13d6aa1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d014ed20 con 0x7f13d00b4d70 2024-04-06T13:49:23.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.577+0000 7f13cffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13d00cf900 0x7f13d014d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.577+0000 7f13cffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13d00cf900 0x7f13d014d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46334/0 (socket says 172.21.15.189:46334) 2024-04-06T13:49:23.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.577+0000 7f13cffff700 1 -- 172.21.15.189:0/2286921603 learned_addr learned my addr 172.21.15.189:0/2286921603 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:23.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.577+0000 7f13d483d700 1 --2- 172.21.15.189:0/2286921603 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13d00b4d70 0x7f13d014c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.577+0000 7f13d503e700 1 --2- 172.21.15.189:0/2286921603 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13d013d2e0 0x7f13d006eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.578+0000 7f13cffff700 1 -- 172.21.15.189:0/2286921603 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13d00b4d70 msgr2=0x7f13d014c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:23.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.578+0000 7f13cffff700 1 --2- 172.21.15.189:0/2286921603 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13d00b4d70 0x7f13d014c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.578+0000 7f13cffff700 1 -- 172.21.15.189:0/2286921603 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13d013d2e0 msgr2=0x7f13d006eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:23.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.578+0000 7f13cffff700 1 --2- 172.21.15.189:0/2286921603 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13d013d2e0 0x7f13d006eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.578+0000 7f13cffff700 1 -- 172.21.15.189:0/2286921603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c800bd40 con 0x7f13d00cf900 2024-04-06T13:49:23.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.578+0000 7f13cffff700 1 --2- 172.21.15.189:0/2286921603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13d00cf900 0x7f13d014d9c0 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7f13bc00c510 tx=0x7f13bc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:23.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.578+0000 7f13cd7fa700 1 -- 172.21.15.189:0/2286921603 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13bc0052d0 con 0x7f13d00cf900 2024-04-06T13:49:23.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.578+0000 7f13cd7fa700 1 -- 172.21.15.189:0/2286921603 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13bc005430 con 0x7f13d00cf900 2024-04-06T13:49:23.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.578+0000 7f13cd7fa700 1 -- 172.21.15.189:0/2286921603 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13bc0038c0 con 0x7f13d00cf900 2024-04-06T13:49:23.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.578+0000 7f13d6aa1700 1 -- 172.21.15.189:0/2286921603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d014e110 con 0x7f13d00cf900 2024-04-06T13:49:23.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.578+0000 7f13d6aa1700 1 -- 172.21.15.189:0/2286921603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d014e6d0 con 0x7f13d00cf900 2024-04-06T13:49:23.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.579+0000 7f13d6aa1700 1 -- 172.21.15.189:0/2286921603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d0067bb0 con 0x7f13d00cf900 2024-04-06T13:49:23.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.580+0000 7f13cd7fa700 1 -- 172.21.15.189:0/2286921603 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f13bc003a20 con 0x7f13d00cf900 2024-04-06T13:49:23.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.580+0000 7f13cd7fa700 1 --2- 172.21.15.189:0/2286921603 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f13b406aed0 0x7f13b406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:23.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.580+0000 7f13cd7fa700 1 -- 172.21.15.189:0/2286921603 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f13bc088c50 con 0x7f13d00cf900 2024-04-06T13:49:23.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.580+0000 7f13d483d700 1 --2- 172.21.15.189:0/2286921603 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f13b406aed0 0x7f13b406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:23.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.580+0000 7f13d483d700 1 --2- 172.21.15.189:0/2286921603 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f13b406aed0 0x7f13b406d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f13d01548e0 tx=0x7f13c4008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:23.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.584+0000 7f13cd7fa700 1 -- 172.21.15.189:0/2286921603 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f13bc08e020 con 0x7f13d00cf900 2024-04-06T13:49:23.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.828+0000 7f13d6aa1700 1 -- 172.21.15.189:0/2286921603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f13d0074990 con 0x7f13d00cf900 2024-04-06T13:49:23.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.828+0000 7f13cd7fa700 1 -- 172.21.15.189:0/2286921603 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v131) v1 ==== 53+0+73 (secure 0 0 0) 0x7f13bc017070 con 0x7f13d00cf900 2024-04-06T13:49:23.828 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-06T13:49:23.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.830+0000 7f13d6aa1700 1 -- 172.21.15.189:0/2286921603 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f13b406aed0 msgr2=0x7f13b406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:23.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.830+0000 7f13d6aa1700 1 --2- 172.21.15.189:0/2286921603 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f13b406aed0 0x7f13b406d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f13d01548e0 tx=0x7f13c4008040 comp rx=0 tx=0).stop 2024-04-06T13:49:23.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.830+0000 7f13d6aa1700 1 -- 172.21.15.189:0/2286921603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13d00cf900 msgr2=0x7f13d014d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:23.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.830+0000 7f13d6aa1700 1 --2- 172.21.15.189:0/2286921603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13d00cf900 0x7f13d014d9c0 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7f13bc00c510 tx=0x7f13bc00c8d0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.830+0000 7f13d6aa1700 1 -- 172.21.15.189:0/2286921603 shutdown_connections 2024-04-06T13:49:23.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.830+0000 7f13d6aa1700 1 --2- 172.21.15.189:0/2286921603 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f13b406aed0 0x7f13b406d390 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.830+0000 7f13d6aa1700 1 --2- 172.21.15.189:0/2286921603 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13d013d2e0 0x7f13d006eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.830+0000 7f13d6aa1700 1 --2- 172.21.15.189:0/2286921603 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13d00b4d70 0x7f13d014c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.830+0000 7f13d6aa1700 1 --2- 172.21.15.189:0/2286921603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13d00cf900 0x7f13d014d9c0 unknown :-1 s=CLOSED pgs=2952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:23.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.830+0000 7f13d6aa1700 1 -- 172.21.15.189:0/2286921603 >> 172.21.15.189:0/2286921603 conn(0x7f13d00d00b0 msgr2=0x7f13d00b44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:23.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.831+0000 7f13d6aa1700 1 -- 172.21.15.189:0/2286921603 shutdown_connections 2024-04-06T13:49:23.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:23.831+0000 7f13d6aa1700 1 -- 172.21.15.189:0/2286921603 wait complete. 2024-04-06T13:49:23.842 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-06T13:49:23.842 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T13:49:23.842 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-06T13:49:23.843 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-06T13:49:23.843 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T13:49:23.843 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-06T13:49:23.843 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-06T13:49:23.843 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T13:49:23.843 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:49:23.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:49:23.843 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-06T13:49:23.861 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T13:49:23.861 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T13:49:23.861 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T13:49:23.861 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:49:23.861 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T13:49:23.861 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:49:23.861 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:49:23.861 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:49:23.861 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-06T13:49:23.861 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:49:23.862 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:49:23.862 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-06T13:49:23.862 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:49:23.862 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:49:23.862 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T13:49:23.862 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:49:23.862 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T13:49:23.862 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 36525sec preferred_lft 36525sec 2024-04-06T13:49:23.862 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::8592:df2c:92e2:dace/64 scope link noprefixroute 2024-04-06T13:49:23.862 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:49:23.862 INFO:teuthology.orchestra.run.smithi189.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T13:49:23.862 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff 2024-04-06T13:49:23.862 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-06T13:49:23.862 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:49:23.863 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::88e:b6ff:fe6f:1cb0/64 scope link 2024-04-06T13:49:23.863 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:49:23.863 INFO:teuthology.orchestra.run.smithi189.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T13:49:23.863 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 62:a5:8d:c5:1a:9c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T13:49:23.863 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::60a5:8dff:fec5:1a9c/64 scope link 2024-04-06T13:49:23.863 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:49:23.863 INFO:teuthology.orchestra.run.smithi189.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T13:49:23.863 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-06T13:49:23.863 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::1c7e:c6ff:febb:1dab/64 scope link 2024-04-06T13:49:23.863 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:49:23.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:49:23.864 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T13:49:23.928 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-06T13:49:23.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:49:23.929 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-06T13:49:23.989 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-06T13:49:23.989 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T13:49:23.989 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T13:49:23.989 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T13:49:23.989 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T13:49:23.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:49:23.990 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T13:49:23.990 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T13:49:23.990 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T13:49:23.990 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T13:49:23.990 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T13:49:23.990 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T13:49:24.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:49:24.096 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T13:49:24.113 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-06T13:49:24.113 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-06T13:49:24.113 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:49:24.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:49:24.113 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-04-06T13:49:24.180 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T13:49:24.180 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-06T13:49:24.237 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:49:24.238 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T13:49:24.238 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-06T13:49:24.294 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T13:49:24.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:49:24.294 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:49:24.381 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:49:24.381 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:49:24.381 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:49:24.389 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:49:24.389 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:49:24.390 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:49:24.390 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T13:49:24.390 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T13:49:24.409 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:49:24.409 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:49:24.409 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:49:24.409 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T13:49:24.409 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T13:49:24.409 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790" 2024-04-06T13:49:24.409 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2024-04-06T13:49:24.410 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e30041de-e9a3-4351-8ca7-29d7f3e3728c" will pass to kernel 2024-04-06T13:49:24.411 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T13:49:24.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:49:24.411 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-06T13:49:24.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:49:24.440 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T13:49:24.529 INFO:teuthology.orchestra.run.smithi189.stdout:client11509 2024-04-06T13:49:24.530 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:49:24.530 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:49:24.547 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-06T13:49:24.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.710+0000 7ff7e25f8700 1 Processor -- start 2024-04-06T13:49:24.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.710+0000 7ff7e25f8700 1 -- start start 2024-04-06T13:49:24.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.711+0000 7ff7e25f8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff7dc146dd0 0x7ff7dc06d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:24.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.711+0000 7ff7e25f8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc0cedf0 0x7ff7dc0cf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:24.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.711+0000 7ff7e25f8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7dc0cf780 con 0x7ff7dc0d17a0 2024-04-06T13:49:24.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.711+0000 7ff7e25f8700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7dc06beb0 con 0x7ff7dc146dd0 2024-04-06T13:49:24.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.711+0000 7ff7e25f8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc06c0a0 con 0x7ff7dc0cedf0 2024-04-06T13:49:24.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.711+0000 7ff7db7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff7dc146dd0 0x7ff7dc06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:24.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.711+0000 7ff7db7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff7dc146dd0 0x7ff7dc06d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46342/0 (socket says 172.21.15.189:46342) 2024-04-06T13:49:24.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.711+0000 7ff7db7fe700 1 -- 172.21.15.189:0/3231405721 learned_addr learned my addr 172.21.15.189:0/3231405721 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:24.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.711+0000 7ff7e0b95700 1 --2- 172.21.15.189:0/3231405721 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc0cedf0 0x7ff7dc0cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:24.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.711+0000 7ff7db7fe700 1 -- 172.21.15.189:0/3231405721 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc0cedf0 msgr2=0x7ff7dc0cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:24.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.711+0000 7ff7db7fe700 1 --2- 172.21.15.189:0/3231405721 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc0cedf0 0x7ff7dc0cf240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:24.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.711+0000 7ff7db7fe700 1 -- 172.21.15.189:0/3231405721 >> v1:172.21.15.192:6789/0 conn(0x7ff7dc0d17a0 legacy=0x7ff7dc06b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:24.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.711+0000 7ff7db7fe700 1 -- 172.21.15.189:0/3231405721 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7dc06c380 con 0x7ff7dc146dd0 2024-04-06T13:49:24.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.711+0000 7ff7db7fe700 1 --2- 172.21.15.189:0/3231405721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7dc146dd0 0x7ff7dc06d140 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc0098f0 tx=0x7ff7cc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea7aa27288974f40 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:24.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.711+0000 7ff7daffd700 1 -- 172.21.15.189:0/3231405721 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc00b8a0 con 0x7ff7dc146dd0 2024-04-06T13:49:24.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.712+0000 7ff7daffd700 1 -- 172.21.15.189:0/3231405721 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7cc00ba00 con 0x7ff7dc146dd0 2024-04-06T13:49:24.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.712+0000 7ff7daffd700 1 -- 172.21.15.189:0/3231405721 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc00bd90 con 0x7ff7dc146dd0 2024-04-06T13:49:24.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.712+0000 7ff7e25f8700 1 -- 172.21.15.189:0/3231405721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7dc146dd0 msgr2=0x7ff7dc06d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:24.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.712+0000 7ff7e25f8700 1 --2- 172.21.15.189:0/3231405721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7dc146dd0 0x7ff7dc06d140 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc0098f0 tx=0x7ff7cc009c00 comp rx=0 tx=0).stop 2024-04-06T13:49:24.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.712+0000 7ff7e25f8700 1 -- 172.21.15.189:0/3231405721 shutdown_connections 2024-04-06T13:49:24.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.712+0000 7ff7e25f8700 1 --2- 172.21.15.189:0/3231405721 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc0cedf0 0x7ff7dc0cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:24.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.712+0000 7ff7e25f8700 1 --2- 172.21.15.189:0/3231405721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7dc146dd0 0x7ff7dc06d140 unknown :-1 s=CLOSED pgs=2953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:24.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.712+0000 7ff7e25f8700 1 -- 172.21.15.189:0/3231405721 >> 172.21.15.189:0/3231405721 conn(0x7ff7dc0b2db0 msgr2=0x7ff7dc0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:24.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.712+0000 7ff7e25f8700 1 -- 172.21.15.189:0/3231405721 shutdown_connections 2024-04-06T13:49:24.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.712+0000 7ff7e25f8700 1 -- 172.21.15.189:0/3231405721 wait complete. 2024-04-06T13:49:24.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.713+0000 7ff7e25f8700 1 Processor -- start 2024-04-06T13:49:24.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.713+0000 7ff7e25f8700 1 -- start start 2024-04-06T13:49:24.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.714+0000 7ff7e25f8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7dc0cedf0 0x7ff7dc15c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:24.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.714+0000 7ff7e25f8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc0d17a0 0x7ff7dc0b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:24.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.714+0000 7ff7e25f8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7dc146dd0 0x7ff7dc0aec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:24.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.714+0000 7ff7e25f8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc0af180 con 0x7ff7dc146dd0 2024-04-06T13:49:24.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.714+0000 7ff7e25f8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc0b4f70 con 0x7ff7dc0cedf0 2024-04-06T13:49:24.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.714+0000 7ff7e25f8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc0b5250 con 0x7ff7dc0d17a0 2024-04-06T13:49:24.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.714+0000 7ff7e0b95700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7dc146dd0 0x7ff7dc0aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:24.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.714+0000 7ff7e0b95700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7dc146dd0 0x7ff7dc0aec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46354/0 (socket says 172.21.15.189:46354) 2024-04-06T13:49:24.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.714+0000 7ff7e0b95700 1 -- 172.21.15.189:0/2011456368 learned_addr learned my addr 172.21.15.189:0/2011456368 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:24.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.714+0000 7ff7dbfff700 1 --2- 172.21.15.189:0/2011456368 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7dc0cedf0 0x7ff7dc15c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:24.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.715+0000 7ff7db7fe700 1 --2- 172.21.15.189:0/2011456368 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc0d17a0 0x7ff7dc0b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:24.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.715+0000 7ff7e0b95700 1 -- 172.21.15.189:0/2011456368 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc0d17a0 msgr2=0x7ff7dc0b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:24.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.715+0000 7ff7e0b95700 1 --2- 172.21.15.189:0/2011456368 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc0d17a0 0x7ff7dc0b48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:24.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.715+0000 7ff7e0b95700 1 -- 172.21.15.189:0/2011456368 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7dc0cedf0 msgr2=0x7ff7dc15c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:24.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.715+0000 7ff7e0b95700 1 --2- 172.21.15.189:0/2011456368 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7dc0cedf0 0x7ff7dc15c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:24.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.715+0000 7ff7e0b95700 1 -- 172.21.15.189:0/2011456368 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7cc009550 con 0x7ff7dc146dd0 2024-04-06T13:49:24.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.715+0000 7ff7e0b95700 1 --2- 172.21.15.189:0/2011456368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7dc146dd0 0x7ff7dc0aec40 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4007e60 tx=0x7ff7d400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:24.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.715+0000 7ff7d8ff9700 1 -- 172.21.15.189:0/2011456368 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d400ede0 con 0x7ff7dc146dd0 2024-04-06T13:49:24.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.715+0000 7ff7d8ff9700 1 -- 172.21.15.189:0/2011456368 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7d4014420 con 0x7ff7dc146dd0 2024-04-06T13:49:24.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.715+0000 7ff7e25f8700 1 -- 172.21.15.189:0/2011456368 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7dc0b5530 con 0x7ff7dc146dd0 2024-04-06T13:49:24.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.715+0000 7ff7d8ff9700 1 -- 172.21.15.189:0/2011456368 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d40147d0 con 0x7ff7dc146dd0 2024-04-06T13:49:24.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.715+0000 7ff7e25f8700 1 -- 172.21.15.189:0/2011456368 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7dc1f2b40 con 0x7ff7dc146dd0 2024-04-06T13:49:24.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.716+0000 7ff7e25f8700 1 -- 172.21.15.189:0/2011456368 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7dc1f2dc0 con 0x7ff7dc146dd0 2024-04-06T13:49:24.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.716+0000 7ff7d8ff9700 1 -- 172.21.15.189:0/2011456368 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7ff7d4014930 con 0x7ff7dc146dd0 2024-04-06T13:49:24.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.717+0000 7ff7d8ff9700 1 --2- 172.21.15.189:0/2011456368 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff7c806aed0 0x7ff7c806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:24.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.717+0000 7ff7d8ff9700 1 -- 172.21.15.189:0/2011456368 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff7d4089f70 con 0x7ff7dc146dd0 2024-04-06T13:49:24.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.717+0000 7ff7dbfff700 1 --2- 172.21.15.189:0/2011456368 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff7c806aed0 0x7ff7c806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:24.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.717+0000 7ff7dbfff700 1 --2- 172.21.15.189:0/2011456368 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff7c806aed0 0x7ff7c806d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0b08e0 tx=0x7ff7d0009040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:24.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.721+0000 7ff7d8ff9700 1 -- 172.21.15.189:0/2011456368 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff7d40bd650 con 0x7ff7dc146dd0 2024-04-06T13:49:24.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.948+0000 7ff7e25f8700 1 -- 172.21.15.189:0/2011456368 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7dc06cc20 con 0x7ff7c806aed0 2024-04-06T13:49:24.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.949+0000 7ff7d8ff9700 1 -- 172.21.15.189:0/2011456368 <== mgr.12522 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7ff7d00050d0 con 0x7ff7c806aed0 2024-04-06T13:49:24.949 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T13:49:24.949 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:24.949 INFO:teuthology.orchestra.run.smithi189.stdout: "name": "cephfs" 2024-04-06T13:49:24.949 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:24.949 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T13:49:24.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.951+0000 7ff7e25f8700 1 -- 172.21.15.189:0/2011456368 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff7c806aed0 msgr2=0x7ff7c806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:24.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.951+0000 7ff7e25f8700 1 --2- 172.21.15.189:0/2011456368 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff7c806aed0 0x7ff7c806d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0b08e0 tx=0x7ff7d0009040 comp rx=0 tx=0).stop 2024-04-06T13:49:24.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.951+0000 7ff7e25f8700 1 -- 172.21.15.189:0/2011456368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7dc146dd0 msgr2=0x7ff7dc0aec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:24.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.951+0000 7ff7e25f8700 1 --2- 172.21.15.189:0/2011456368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7dc146dd0 0x7ff7dc0aec40 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4007e60 tx=0x7ff7d400e3c0 comp rx=0 tx=0).stop 2024-04-06T13:49:24.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.951+0000 7ff7e25f8700 1 -- 172.21.15.189:0/2011456368 shutdown_connections 2024-04-06T13:49:24.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.951+0000 7ff7e25f8700 1 --2- 172.21.15.189:0/2011456368 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff7c806aed0 0x7ff7c806d390 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:24.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.951+0000 7ff7e25f8700 1 --2- 172.21.15.189:0/2011456368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7dc146dd0 0x7ff7dc0aec40 unknown :-1 s=CLOSED pgs=2954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:24.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.951+0000 7ff7e25f8700 1 --2- 172.21.15.189:0/2011456368 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7dc0cedf0 0x7ff7dc15c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:24.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.951+0000 7ff7e25f8700 1 --2- 172.21.15.189:0/2011456368 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc0d17a0 0x7ff7dc0b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:24.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.951+0000 7ff7e25f8700 1 -- 172.21.15.189:0/2011456368 >> 172.21.15.189:0/2011456368 conn(0x7ff7dc0b2db0 msgr2=0x7ff7dc06b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:24.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.951+0000 7ff7e25f8700 1 -- 172.21.15.189:0/2011456368 shutdown_connections 2024-04-06T13:49:24.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:24.951+0000 7ff7e25f8700 1 -- 172.21.15.189:0/2011456368 wait complete. 2024-04-06T13:49:24.963 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-06T13:49:25.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.088+0000 7fe10e8fa700 1 Processor -- start 2024-04-06T13:49:25.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.088+0000 7fe10e8fa700 1 -- start start 2024-04-06T13:49:25.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.089+0000 7fe10e8fa700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe10815d7b0 0x7fe1080b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:25.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.089+0000 7fe10e8fa700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe10806a730 0x7fe10806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:25.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.089+0000 7fe10e8fa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1080b3990 con 0x7fe10815ddf0 2024-04-06T13:49:25.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.089+0000 7fe10e8fa700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10815e950 con 0x7fe10806a730 2024-04-06T13:49:25.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.089+0000 7fe10e8fa700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10806b0f0 con 0x7fe10815d7b0 2024-04-06T13:49:25.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.089+0000 7fe1077fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe10806a730 0x7fe10806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:25.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.089+0000 7fe10ce97700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe10815ddf0 0x7fe10815e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40072/0 (socket says 172.21.15.189:40072) 2024-04-06T13:49:25.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.089+0000 7fe10ce97700 1 -- 172.21.15.189:0/173659697 learned_addr learned my addr 172.21.15.189:0/173659697 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:25.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.089+0000 7fe107fff700 1 --2- 172.21.15.189:0/173659697 >> v2:172.21.15.192:3300/0 conn(0x7fe10815d7b0 0x7fe1080b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:25.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.089+0000 7fe1077fe700 1 -- 172.21.15.189:0/173659697 >> v2:172.21.15.192:3300/0 conn(0x7fe10815d7b0 msgr2=0x7fe1080b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:25.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.089+0000 7fe1077fe700 1 --2- 172.21.15.189:0/173659697 >> v2:172.21.15.192:3300/0 conn(0x7fe10815d7b0 0x7fe1080b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:25.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.089+0000 7fe1077fe700 1 -- 172.21.15.189:0/173659697 >> v1:172.21.15.189:6789/0 conn(0x7fe10815ddf0 legacy=0x7fe10815e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:25.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.089+0000 7fe1077fe700 1 -- 172.21.15.189:0/173659697 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1080b3060 con 0x7fe10806a730 2024-04-06T13:49:25.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.089+0000 7fe106ffd700 1 -- 172.21.15.189:0/173659697 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584604764 0 0) 0x7fe1080b3990 con 0x7fe10815ddf0 2024-04-06T13:49:25.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.089+0000 7fe107fff700 1 --2- 172.21.15.189:0/173659697 >> v2:172.21.15.192:3300/0 conn(0x7fe10815d7b0 0x7fe1080b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:25.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.089+0000 7fe1077fe700 1 --2- 172.21.15.189:0/173659697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe10806a730 0x7fe10806abb0 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 crypto rx=0x7fe1080b6200 tx=0x7fe0fc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d986f1886e9e1a93 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:25.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.089+0000 7fe106ffd700 1 -- 172.21.15.189:0/173659697 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc011830 con 0x7fe10806a730 2024-04-06T13:49:25.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.089+0000 7fe106ffd700 1 -- 172.21.15.189:0/173659697 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0fc011990 con 0x7fe10806a730 2024-04-06T13:49:25.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.089+0000 7fe106ffd700 1 -- 172.21.15.189:0/173659697 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc011d20 con 0x7fe10806a730 2024-04-06T13:49:25.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.090+0000 7fe10e8fa700 1 -- 172.21.15.189:0/173659697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe10806a730 msgr2=0x7fe10806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:25.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.090+0000 7fe10e8fa700 1 --2- 172.21.15.189:0/173659697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe10806a730 0x7fe10806abb0 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 crypto rx=0x7fe1080b6200 tx=0x7fe0fc009b20 comp rx=0 tx=0).stop 2024-04-06T13:49:25.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.090+0000 7fe10e8fa700 1 -- 172.21.15.189:0/173659697 shutdown_connections 2024-04-06T13:49:25.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.090+0000 7fe10e8fa700 1 --2- 172.21.15.189:0/173659697 >> v2:172.21.15.192:3300/0 conn(0x7fe10815d7b0 0x7fe1080b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:25.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.090+0000 7fe10e8fa700 1 --2- 172.21.15.189:0/173659697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe10806a730 0x7fe10806abb0 unknown :-1 s=CLOSED pgs=2956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:25.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.090+0000 7fe10e8fa700 1 -- 172.21.15.189:0/173659697 >> 172.21.15.189:0/173659697 conn(0x7fe1080d00b0 msgr2=0x7fe1080d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:25.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.090+0000 7fe10e8fa700 1 -- 172.21.15.189:0/173659697 shutdown_connections 2024-04-06T13:49:25.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.090+0000 7fe10e8fa700 1 -- 172.21.15.189:0/173659697 wait complete. 2024-04-06T13:49:25.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.091+0000 7fe10e8fa700 1 Processor -- start 2024-04-06T13:49:25.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.091+0000 7fe10e8fa700 1 -- start start 2024-04-06T13:49:25.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.092+0000 7fe10e8fa700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe10806a730 0x7fe108149b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:25.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.092+0000 7fe10e8fa700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe10815d7b0 0x7fe1080cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:25.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.092+0000 7fe10e8fa700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe10815ddf0 0x7fe1080adc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:25.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.092+0000 7fe10e8fa700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1080cd910 con 0x7fe10815d7b0 2024-04-06T13:49:25.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.092+0000 7fe10e8fa700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10814a390 con 0x7fe10806a730 2024-04-06T13:49:25.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.092+0000 7fe10e8fa700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10814a090 con 0x7fe10815ddf0 2024-04-06T13:49:25.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.092+0000 7fe1077fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe10815d7b0 0x7fe1080cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:25.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.092+0000 7fe1077fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe10815d7b0 0x7fe1080cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46378/0 (socket says 172.21.15.189:46378) 2024-04-06T13:49:25.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.092+0000 7fe1077fe700 1 -- 172.21.15.189:0/3849936102 learned_addr learned my addr 172.21.15.189:0/3849936102 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:25.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.092+0000 7fe107fff700 1 --2- 172.21.15.189:0/3849936102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe10806a730 0x7fe108149b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:25.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.092+0000 7fe10ce97700 1 --2- 172.21.15.189:0/3849936102 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe10815ddf0 0x7fe1080adc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:25.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.093+0000 7fe1077fe700 1 -- 172.21.15.189:0/3849936102 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe10815ddf0 msgr2=0x7fe1080adc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:25.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.093+0000 7fe1077fe700 1 --2- 172.21.15.189:0/3849936102 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe10815ddf0 0x7fe1080adc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:25.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.093+0000 7fe1077fe700 1 -- 172.21.15.189:0/3849936102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe10806a730 msgr2=0x7fe108149b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:25.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.093+0000 7fe1077fe700 1 --2- 172.21.15.189:0/3849936102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe10806a730 0x7fe108149b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:25.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.093+0000 7fe1077fe700 1 -- 172.21.15.189:0/3849936102 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0fc0094a0 con 0x7fe10815d7b0 2024-04-06T13:49:25.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.093+0000 7fe10ce97700 1 --2- 172.21.15.189:0/3849936102 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe10815ddf0 0x7fe1080adc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:25.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.093+0000 7fe1077fe700 1 --2- 172.21.15.189:0/3849936102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe10815d7b0 0x7fe1080cce00 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc000c00 tx=0x7fe0fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:25.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.093+0000 7fe104ff9700 1 -- 172.21.15.189:0/3849936102 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc0038a0 con 0x7fe10815d7b0 2024-04-06T13:49:25.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.093+0000 7fe104ff9700 1 -- 172.21.15.189:0/3849936102 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0fc01f070 con 0x7fe10815d7b0 2024-04-06T13:49:25.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.093+0000 7fe10e8fa700 1 -- 172.21.15.189:0/3849936102 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1080cd630 con 0x7fe10815d7b0 2024-04-06T13:49:25.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.093+0000 7fe104ff9700 1 -- 172.21.15.189:0/3849936102 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc003c40 con 0x7fe10815d7b0 2024-04-06T13:49:25.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.093+0000 7fe10e8fa700 1 -- 172.21.15.189:0/3849936102 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe10814a880 con 0x7fe10815d7b0 2024-04-06T13:49:25.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.094+0000 7fe10e8fa700 1 -- 172.21.15.189:0/3849936102 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe108067bb0 con 0x7fe10815d7b0 2024-04-06T13:49:25.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.094+0000 7fe104ff9700 1 -- 172.21.15.189:0/3849936102 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 160) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fe0fc026480 con 0x7fe10815d7b0 2024-04-06T13:49:25.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.095+0000 7fe104ff9700 1 --2- 172.21.15.189:0/3849936102 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe0f406aed0 0x7fe0f406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:25.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.095+0000 7fe104ff9700 1 -- 172.21.15.189:0/3849936102 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe0fc021070 con 0x7fe10815d7b0 2024-04-06T13:49:25.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.095+0000 7fe107fff700 1 --2- 172.21.15.189:0/3849936102 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe0f406aed0 0x7fe0f406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:25.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.095+0000 7fe107fff700 1 --2- 172.21.15.189:0/3849936102 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe0f406aed0 0x7fe0f406d390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8006fd0 tx=0x7fe0f8008040 comp rx=0 tx=0).ready entity=mgr.12522 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:25.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.099+0000 7fe104ff9700 1 -- 172.21.15.189:0/3849936102 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe0fc056080 con 0x7fe10815d7b0 2024-04-06T13:49:25.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.351+0000 7fe10e8fa700 1 -- 172.21.15.189:0/3849936102 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fe108074990 con 0x7fe10815d7b0 2024-04-06T13:49:25.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.626+0000 7fe104ff9700 1 -- 172.21.15.189:0/3849936102 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 161) v1 ==== 89366+0+0 (secure 0 0 0) 0x7fe0fc056450 con 0x7fe10815d7b0 2024-04-06T13:49:25.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.627+0000 7fe104ff9700 1 -- 172.21.15.189:0/3849936102 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v161) v1 ==== 92+0+0 (secure 0 0 0) 0x7fe0fc083220 con 0x7fe10815d7b0 2024-04-06T13:49:25.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.630+0000 7fe10e8fa700 1 -- 172.21.15.189:0/3849936102 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe0f406aed0 msgr2=0x7fe0f406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:25.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.630+0000 7fe10e8fa700 1 --2- 172.21.15.189:0/3849936102 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe0f406aed0 0x7fe0f406d390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8006fd0 tx=0x7fe0f8008040 comp rx=0 tx=0).stop 2024-04-06T13:49:25.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.630+0000 7fe10e8fa700 1 -- 172.21.15.189:0/3849936102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe10815d7b0 msgr2=0x7fe1080cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:25.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.630+0000 7fe10e8fa700 1 --2- 172.21.15.189:0/3849936102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe10815d7b0 0x7fe1080cce00 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc000c00 tx=0x7fe0fc004750 comp rx=0 tx=0).stop 2024-04-06T13:49:25.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.630+0000 7fe10e8fa700 1 -- 172.21.15.189:0/3849936102 shutdown_connections 2024-04-06T13:49:25.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.630+0000 7fe10e8fa700 1 --2- 172.21.15.189:0/3849936102 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe0f406aed0 0x7fe0f406d390 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:25.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.630+0000 7fe10e8fa700 1 --2- 172.21.15.189:0/3849936102 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe10815ddf0 0x7fe1080adc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:25.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.630+0000 7fe10e8fa700 1 --2- 172.21.15.189:0/3849936102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe10806a730 0x7fe108149b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:25.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.630+0000 7fe10e8fa700 1 --2- 172.21.15.189:0/3849936102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe10815d7b0 0x7fe1080cce00 unknown :-1 s=CLOSED pgs=2957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:25.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.630+0000 7fe10e8fa700 1 -- 172.21.15.189:0/3849936102 >> 172.21.15.189:0/3849936102 conn(0x7fe1080d00b0 msgr2=0x7fe10815f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:25.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.630+0000 7fe10e8fa700 1 -- 172.21.15.189:0/3849936102 shutdown_connections 2024-04-06T13:49:25.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.630+0000 7fe10e8fa700 1 -- 172.21.15.189:0/3849936102 wait complete. 2024-04-06T13:49:25.644 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-06T13:49:25.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.773+0000 7f7481229700 1 Processor -- start 2024-04-06T13:49:25.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.773+0000 7f7481229700 1 -- start start 2024-04-06T13:49:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.774+0000 7f7481229700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f747c15a550 0x7f747c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.774+0000 7f7481229700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f747c0b4d70 0x7f747c0b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.774+0000 7f7481229700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747c0cdbe0 con 0x7f747c0cf900 2024-04-06T13:49:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.774+0000 7f7481229700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f747c15a940 con 0x7f747c0b4d70 2024-04-06T13:49:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.774+0000 7f7481229700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747c15ac00 con 0x7f747c15a550 2024-04-06T13:49:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.774+0000 7f747b59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f747c0b4d70 0x7f747c0b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.774+0000 7f747b59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f747c0b4d70 0x7f747c0b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46392/0 (socket says 172.21.15.189:46392) 2024-04-06T13:49:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.774+0000 7f747b59e700 1 -- 172.21.15.189:0/2753908420 learned_addr learned my addr 172.21.15.189:0/2753908420 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:25.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.774+0000 7f747ad9d700 1 --2- 172.21.15.189:0/2753908420 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f747c15a550 0x7f747c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.775+0000 7f747b59e700 1 -- 172.21.15.189:0/2753908420 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f747c15a550 msgr2=0x7f747c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.775+0000 7f747b59e700 1 --2- 172.21.15.189:0/2753908420 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f747c15a550 0x7f747c0cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.775+0000 7f747b59e700 1 -- 172.21.15.189:0/2753908420 >> v1:172.21.15.192:6789/0 conn(0x7f747c0cf900 legacy=0x7f747c0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.775+0000 7f7479d9b700 1 -- 172.21.15.189:0/2753908420 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976483580 0 0) 0x7f747c0cdbe0 con 0x7f747c0cf900 2024-04-06T13:49:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.775+0000 7f747b59e700 1 -- 172.21.15.189:0/2753908420 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c15aee0 con 0x7f747c0b4d70 2024-04-06T13:49:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.775+0000 7f747b59e700 1 --2- 172.21.15.189:0/2753908420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f747c0b4d70 0x7f747c0b2fc0 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 crypto rx=0x7f7474009e50 tx=0x7f747400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c6855a239fc5e45 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.775+0000 7f7479d9b700 1 -- 172.21.15.189:0/2753908420 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747400e070 con 0x7f747c0b4d70 2024-04-06T13:49:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.775+0000 7f7479d9b700 1 -- 172.21.15.189:0/2753908420 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f747400b910 con 0x7f747c0b4d70 2024-04-06T13:49:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.775+0000 7f7479d9b700 1 -- 172.21.15.189:0/2753908420 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747400bcc0 con 0x7f747c0b4d70 2024-04-06T13:49:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.776+0000 7f7481229700 1 -- 172.21.15.189:0/2753908420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f747c0b4d70 msgr2=0x7f747c0b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.776+0000 7f7481229700 1 --2- 172.21.15.189:0/2753908420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f747c0b4d70 0x7f747c0b2fc0 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 crypto rx=0x7f7474009e50 tx=0x7f747400b330 comp rx=0 tx=0).stop 2024-04-06T13:49:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.776+0000 7f7481229700 1 -- 172.21.15.189:0/2753908420 shutdown_connections 2024-04-06T13:49:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.776+0000 7f7481229700 1 --2- 172.21.15.189:0/2753908420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f747c0b4d70 0x7f747c0b2fc0 unknown :-1 s=CLOSED pgs=2961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.776+0000 7f7481229700 1 --2- 172.21.15.189:0/2753908420 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f747c15a550 0x7f747c0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:25.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.776+0000 7f7481229700 1 -- 172.21.15.189:0/2753908420 >> 172.21.15.189:0/2753908420 conn(0x7f747c0d00b0 msgr2=0x7f747c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:25.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.776+0000 7f7481229700 1 -- 172.21.15.189:0/2753908420 shutdown_connections 2024-04-06T13:49:25.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.776+0000 7f7481229700 1 -- 172.21.15.189:0/2753908420 wait complete. 2024-04-06T13:49:25.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.777+0000 7f7481229700 1 Processor -- start 2024-04-06T13:49:25.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.777+0000 7f7481229700 1 -- start start 2024-04-06T13:49:25.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.778+0000 7f7481229700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f747c0b4d70 0x7f747c0a9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:25.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.778+0000 7f7481229700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f747c0cf900 0x7f747c152e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:25.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.778+0000 7f7481229700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f747c0a8ec0 0x7f747c06d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:25.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.778+0000 7f7481229700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c0aaea0 con 0x7f747c0a8ec0 2024-04-06T13:49:25.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.778+0000 7f7481229700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c0aa320 con 0x7f747c0b4d70 2024-04-06T13:49:25.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.778+0000 7f7481229700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747c0aa600 con 0x7f747c0cf900 2024-04-06T13:49:25.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.778+0000 7f747b59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f747c0a8ec0 0x7f747c06d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:25.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.778+0000 7f747b59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f747c0a8ec0 0x7f747c06d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46400/0 (socket says 172.21.15.189:46400) 2024-04-06T13:49:25.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.778+0000 7f747b59e700 1 -- 172.21.15.189:0/979430697 learned_addr learned my addr 172.21.15.189:0/979430697 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:25.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.778+0000 7f747a59c700 1 --2- 172.21.15.189:0/979430697 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f747c0cf900 0x7f747c152e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:25.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.778+0000 7f747ad9d700 1 --2- 172.21.15.189:0/979430697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f747c0b4d70 0x7f747c0a9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:25.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.778+0000 7f747a59c700 1 -- 172.21.15.189:0/979430697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f747c0b4d70 msgr2=0x7f747c0a9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:25.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.778+0000 7f747a59c700 1 --2- 172.21.15.189:0/979430697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f747c0b4d70 0x7f747c0a9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:25.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.778+0000 7f747a59c700 1 -- 172.21.15.189:0/979430697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f747c0a8ec0 msgr2=0x7f747c06d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:25.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.778+0000 7f747a59c700 1 --2- 172.21.15.189:0/979430697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f747c0a8ec0 0x7f747c06d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:25.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.778+0000 7f747a59c700 1 -- 172.21.15.189:0/979430697 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7474009b30 con 0x7f747c0cf900 2024-04-06T13:49:25.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.778+0000 7f747b59e700 1 --2- 172.21.15.189:0/979430697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f747c0a8ec0 0x7f747c06d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:25.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.779+0000 7f747a59c700 1 --2- 172.21.15.189:0/979430697 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f747c0cf900 0x7f747c152e20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f747000c520 tx=0x7f747000c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:25.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.779+0000 7f74637fe700 1 -- 172.21.15.189:0/979430697 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74700054e0 con 0x7f747c0cf900 2024-04-06T13:49:25.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.779+0000 7f74637fe700 1 -- 172.21.15.189:0/979430697 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7470003680 con 0x7f747c0cf900 2024-04-06T13:49:25.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.779+0000 7f7481229700 1 -- 172.21.15.189:0/979430697 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747c0aa8e0 con 0x7f747c0cf900 2024-04-06T13:49:25.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.779+0000 7f74637fe700 1 -- 172.21.15.189:0/979430697 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7470003a30 con 0x7f747c0cf900 2024-04-06T13:49:25.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.779+0000 7f7481229700 1 -- 172.21.15.189:0/979430697 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f747c0a9370 con 0x7f747c0cf900 2024-04-06T13:49:25.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.780+0000 7f7481229700 1 -- 172.21.15.189:0/979430697 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f747c0ac4d0 con 0x7f747c0cf900 2024-04-06T13:49:25.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.781+0000 7f74637fe700 1 -- 172.21.15.189:0/979430697 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 161) v1 ==== 89366+0+0 (secure 0 0 0) 0x7f7470003b90 con 0x7f747c0cf900 2024-04-06T13:49:25.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.781+0000 7f74637fe700 1 --2- 172.21.15.189:0/979430697 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f746806af20 0x7f746806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:25.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.781+0000 7f747ad9d700 1 -- 172.21.15.189:0/979430697 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f746806af20 msgr2=0x7f746806d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:25.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.781+0000 7f74637fe700 1 -- 172.21.15.189:0/979430697 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7470088b40 con 0x7f747c0cf900 2024-04-06T13:49:25.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.781+0000 7f747ad9d700 1 --2- 172.21.15.189:0/979430697 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f746806af20 0x7f746806d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:49:25.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.786+0000 7f74637fe700 1 -- 172.21.15.189:0/979430697 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f747c0ac4d0 con 0x7f747c0cf900 2024-04-06T13:49:25.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.982+0000 7f747ad9d700 1 -- 172.21.15.189:0/979430697 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f746806af20 msgr2=0x7f746806d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:25.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.982+0000 7f747ad9d700 1 --2- 172.21.15.189:0/979430697 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f746806af20 0x7f746806d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:49:25.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.987+0000 7f7481229700 1 -- 172.21.15.189:0/979430697 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f747c0ac4d0 con 0x7f747c0cf900 2024-04-06T13:49:25.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.992+0000 7f74637fe700 1 -- 172.21.15.189:0/979430697 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v89)=0 v89) v1 ==== 138+0+0 (secure 0 0 0) 0x7f747c0ac4d0 con 0x7f747c0cf900 2024-04-06T13:49:25.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.995+0000 7f7481229700 1 -- 172.21.15.189:0/979430697 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f746806af20 msgr2=0x7f746806d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:49:25.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.995+0000 7f7481229700 1 --2- 172.21.15.189:0/979430697 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f746806af20 0x7f746806d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:25.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.995+0000 7f7481229700 1 -- 172.21.15.189:0/979430697 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f747c0cf900 msgr2=0x7f747c152e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:25.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.995+0000 7f7481229700 1 --2- 172.21.15.189:0/979430697 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f747c0cf900 0x7f747c152e20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f747000c520 tx=0x7f747000c830 comp rx=0 tx=0).stop 2024-04-06T13:49:25.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.995+0000 7f7481229700 1 -- 172.21.15.189:0/979430697 shutdown_connections 2024-04-06T13:49:25.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.995+0000 7f7481229700 1 --2- 172.21.15.189:0/979430697 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f746806af20 0x7f746806d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:25.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.995+0000 7f7481229700 1 --2- 172.21.15.189:0/979430697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f747c0a8ec0 0x7f747c06d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:25.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.995+0000 7f7481229700 1 --2- 172.21.15.189:0/979430697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f747c0b4d70 0x7f747c0a9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:25.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.995+0000 7f7481229700 1 --2- 172.21.15.189:0/979430697 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f747c0cf900 0x7f747c152e20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:25.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.995+0000 7f7481229700 1 -- 172.21.15.189:0/979430697 >> 172.21.15.189:0/979430697 conn(0x7f747c0d00b0 msgr2=0x7f747c154e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:25.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.995+0000 7f7481229700 1 -- 172.21.15.189:0/979430697 shutdown_connections 2024-04-06T13:49:25.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:25.995+0000 7f7481229700 1 -- 172.21.15.189:0/979430697 wait complete. 2024-04-06T13:49:26.006 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:26.006+0000 7fea8bf15200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:49:26.007 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-06T13:49:26.033 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:26.032+0000 7fb1408eb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:49:26.085 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:26.085+0000 7fea8bf15200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:49:26.116 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:26.116+0000 7fb1408eb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:49:26.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.135+0000 7fcbea2dd700 1 Processor -- start 2024-04-06T13:49:26.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.135+0000 7fcbea2dd700 1 -- start start 2024-04-06T13:49:26.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.136+0000 7fcbea2dd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcbe415d590 0x7fcbe40b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:26.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.136+0000 7fcbea2dd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe40b5cc0 con 0x7fcbe40b4ed0 2024-04-06T13:49:26.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.136+0000 7fcbea2dd700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe40d1500 con 0x7fcbe40d3890 2024-04-06T13:49:26.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.136+0000 7fcbea2dd700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe40d17e0 con 0x7fcbe415d590 2024-04-06T13:49:26.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.136+0000 7fcbe37fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcbe40b4ed0 0x7fcbe40b6580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40100/0 (socket says 172.21.15.189:40100) 2024-04-06T13:49:26.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.136+0000 7fcbe37fe700 1 -- 172.21.15.189:0/4123731060 learned_addr learned my addr 172.21.15.189:0/4123731060 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:26.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.136+0000 7fcbe3fff700 1 --2- 172.21.15.189:0/4123731060 >> v2:172.21.15.192:3300/0 conn(0x7fcbe415d590 0x7fcbe40b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:26.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.136+0000 7fcbe2ffd700 1 -- 172.21.15.189:0/4123731060 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550378525 0 0) 0x7fcbe40b5cc0 con 0x7fcbe40b4ed0 2024-04-06T13:49:26.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.136+0000 7fcbe2ffd700 1 -- 172.21.15.189:0/4123731060 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbcc0035b0 con 0x7fcbe40b4ed0 2024-04-06T13:49:26.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.136+0000 7fcbe3fff700 1 -- 172.21.15.189:0/4123731060 >> v1:172.21.15.192:6789/0 conn(0x7fcbe40d3890 legacy=0x7fcbe40b6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:26.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.136+0000 7fcbe3fff700 1 -- 172.21.15.189:0/4123731060 >> v1:172.21.15.189:6789/0 conn(0x7fcbe40b4ed0 legacy=0x7fcbe40b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:26.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.136+0000 7fcbe3fff700 1 -- 172.21.15.189:0/4123731060 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe40d1ac0 con 0x7fcbe415d590 2024-04-06T13:49:26.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.136+0000 7fcbe3fff700 1 --2- 172.21.15.189:0/4123731060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbe415d590 0x7fcbe40b8d70 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009840 tx=0x7fcbd4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98d024fb2f19cb7e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:26.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.137+0000 7fcbe2ffd700 1 -- 172.21.15.189:0/4123731060 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4011830 con 0x7fcbe415d590 2024-04-06T13:49:26.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.137+0000 7fcbe2ffd700 1 -- 172.21.15.189:0/4123731060 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd4011990 con 0x7fcbe415d590 2024-04-06T13:49:26.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.137+0000 7fcbe2ffd700 1 -- 172.21.15.189:0/4123731060 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4011d20 con 0x7fcbe415d590 2024-04-06T13:49:26.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.137+0000 7fcbea2dd700 1 -- 172.21.15.189:0/4123731060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbe415d590 msgr2=0x7fcbe40b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:26.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.137+0000 7fcbea2dd700 1 --2- 172.21.15.189:0/4123731060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbe415d590 0x7fcbe40b8d70 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009840 tx=0x7fcbd4009b50 comp rx=0 tx=0).stop 2024-04-06T13:49:26.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.137+0000 7fcbea2dd700 1 -- 172.21.15.189:0/4123731060 shutdown_connections 2024-04-06T13:49:26.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.137+0000 7fcbea2dd700 1 --2- 172.21.15.189:0/4123731060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbe415d590 0x7fcbe40b8d70 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:26.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.137+0000 7fcbea2dd700 1 -- 172.21.15.189:0/4123731060 >> 172.21.15.189:0/4123731060 conn(0x7fcbe406bbc0 msgr2=0x7fcbe406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:26.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.137+0000 7fcbea2dd700 1 -- 172.21.15.189:0/4123731060 shutdown_connections 2024-04-06T13:49:26.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.137+0000 7fcbea2dd700 1 -- 172.21.15.189:0/4123731060 wait complete. 2024-04-06T13:49:26.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.138+0000 7fcbea2dd700 1 Processor -- start 2024-04-06T13:49:26.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.139+0000 7fcbea2dd700 1 -- start start 2024-04-06T13:49:26.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.139+0000 7fcbea2dd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbe40b4ed0 0x7fcbe414fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:26.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.139+0000 7fcbea2dd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbe40d3890 0x7fcbe415c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:26.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.139+0000 7fcbea2dd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbe415d590 0x7fcbe415c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:26.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.139+0000 7fcbea2dd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe4150080 con 0x7fcbe415d590 2024-04-06T13:49:26.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.139+0000 7fcbea2dd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe40ccf90 con 0x7fcbe40d3890 2024-04-06T13:49:26.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.139+0000 7fcbea2dd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe40cc380 con 0x7fcbe40b4ed0 2024-04-06T13:49:26.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.140+0000 7fcbe37fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbe40d3890 0x7fcbe415c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:26.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.140+0000 7fcbe3fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbe40b4ed0 0x7fcbe414fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:26.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.140+0000 7fcbe887a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbe415d590 0x7fcbe415c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:26.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.140+0000 7fcbe3fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbe40b4ed0 0x7fcbe414fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:48246/0 (socket says 172.21.15.189:48246) 2024-04-06T13:49:26.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.140+0000 7fcbe3fff700 1 -- 172.21.15.189:0/1987082423 learned_addr learned my addr 172.21.15.189:0/1987082423 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:26.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.140+0000 7fcbe37fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbe40d3890 0x7fcbe415c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:43262/0 (socket says 172.21.15.189:43262) 2024-04-06T13:49:26.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.140+0000 7fcbe3fff700 1 -- 172.21.15.189:0/1987082423 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbe40d3890 msgr2=0x7fcbe415c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:26.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.140+0000 7fcbe3fff700 1 --2- 172.21.15.189:0/1987082423 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbe40d3890 0x7fcbe415c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:26.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.140+0000 7fcbe3fff700 1 -- 172.21.15.189:0/1987082423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbe415d590 msgr2=0x7fcbe415c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:26.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.140+0000 7fcbe3fff700 1 --2- 172.21.15.189:0/1987082423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbe415d590 0x7fcbe415c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:26.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.140+0000 7fcbe3fff700 1 -- 172.21.15.189:0/1987082423 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd40094a0 con 0x7fcbe40b4ed0 2024-04-06T13:49:26.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.140+0000 7fcbe3fff700 1 --2- 172.21.15.189:0/1987082423 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbe40b4ed0 0x7fcbe414fb40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009b90 tx=0x7fcbd4004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:26.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.140+0000 7fcbe0ff9700 1 -- 172.21.15.189:0/1987082423 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd40036a0 con 0x7fcbe40b4ed0 2024-04-06T13:49:26.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.140+0000 7fcbe0ff9700 1 -- 172.21.15.189:0/1987082423 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd401e070 con 0x7fcbe40b4ed0 2024-04-06T13:49:26.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.140+0000 7fcbe0ff9700 1 -- 172.21.15.189:0/1987082423 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4003a60 con 0x7fcbe40b4ed0 2024-04-06T13:49:26.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.140+0000 7fcbea2dd700 1 -- 172.21.15.189:0/1987082423 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe40cc660 con 0x7fcbe40b4ed0 2024-04-06T13:49:26.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.140+0000 7fcbea2dd700 1 -- 172.21.15.189:0/1987082423 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe40ccc20 con 0x7fcbe40b4ed0 2024-04-06T13:49:26.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.141+0000 7fcbea2dd700 1 -- 172.21.15.189:0/1987082423 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe40cd270 con 0x7fcbe40b4ed0 2024-04-06T13:49:26.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.142+0000 7fcbe0ff9700 1 -- 172.21.15.189:0/1987082423 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 161) v1 ==== 89366+0+0 (secure 0 0 0) 0x7fcbd4003bc0 con 0x7fcbe40b4ed0 2024-04-06T13:49:26.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.142+0000 7fcbe0ff9700 1 --2- 172.21.15.189:0/1987082423 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcbd006af20 0x7fcbd006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:26.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.142+0000 7fcbe37fe700 1 -- 172.21.15.189:0/1987082423 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcbd006af20 msgr2=0x7fcbd006d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:26.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.142+0000 7fcbe0ff9700 1 -- 172.21.15.189:0/1987082423 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcbd408a7a0 con 0x7fcbe40b4ed0 2024-04-06T13:49:26.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.142+0000 7fcbe37fe700 1 --2- 172.21.15.189:0/1987082423 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcbd006af20 0x7fcbd006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:49:26.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.147+0000 7fcbe0ff9700 1 -- 172.21.15.189:0/1987082423 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fcbd4055520 con 0x7fcbe40b4ed0 2024-04-06T13:49:26.158 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:26.158+0000 7fea8bf15200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:49:26.195 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:26.195+0000 7fb1408eb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:49:26.225 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:26.225+0000 7fea8bf15200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:49:26.266 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:26.266+0000 7fb1408eb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:49:26.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.343+0000 7fcbe37fe700 1 -- 172.21.15.189:0/1987082423 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcbd006af20 msgr2=0x7fcbd006d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:26.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.343+0000 7fcbe37fe700 1 --2- 172.21.15.189:0/1987082423 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcbd006af20 0x7fcbd006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:49:26.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.353+0000 7fcbea2dd700 1 -- 172.21.15.189:0/1987082423 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fcbe4074990 con 0x7fcbe40b4ed0 2024-04-06T13:49:26.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.358+0000 7fcbe0ff9700 1 -- 172.21.15.189:0/1987082423 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v90)=0 v90) v1 ==== 133+0+0 (secure 0 0 0) 0x7fcbd4058b40 con 0x7fcbe40b4ed0 2024-04-06T13:49:26.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.360+0000 7fcbea2dd700 1 -- 172.21.15.189:0/1987082423 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcbd006af20 msgr2=0x7fcbd006d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:49:26.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.360+0000 7fcbea2dd700 1 --2- 172.21.15.189:0/1987082423 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcbd006af20 0x7fcbd006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:26.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.360+0000 7fcbea2dd700 1 -- 172.21.15.189:0/1987082423 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbe40b4ed0 msgr2=0x7fcbe414fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:26.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.360+0000 7fcbea2dd700 1 --2- 172.21.15.189:0/1987082423 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbe40b4ed0 0x7fcbe414fb40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009b90 tx=0x7fcbd4004750 comp rx=0 tx=0).stop 2024-04-06T13:49:26.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.361+0000 7fcbea2dd700 1 -- 172.21.15.189:0/1987082423 shutdown_connections 2024-04-06T13:49:26.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.361+0000 7fcbea2dd700 1 --2- 172.21.15.189:0/1987082423 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcbd006af20 0x7fcbd006d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:26.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.361+0000 7fcbea2dd700 1 --2- 172.21.15.189:0/1987082423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbe415d590 0x7fcbe415c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:26.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.361+0000 7fcbea2dd700 1 --2- 172.21.15.189:0/1987082423 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbe40b4ed0 0x7fcbe414fb40 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:26.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.361+0000 7fcbea2dd700 1 --2- 172.21.15.189:0/1987082423 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbe40d3890 0x7fcbe415c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:26.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.361+0000 7fcbea2dd700 1 -- 172.21.15.189:0/1987082423 >> 172.21.15.189:0/1987082423 conn(0x7fcbe406bbc0 msgr2=0x7fcbe40cdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:26.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.361+0000 7fcbea2dd700 1 -- 172.21.15.189:0/1987082423 shutdown_connections 2024-04-06T13:49:26.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.361+0000 7fcbea2dd700 1 -- 172.21.15.189:0/1987082423 wait complete. 2024-04-06T13:49:26.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:49:26.374 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-06T13:49:26.403 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:26.403+0000 7fea8bf15200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:49:26.435 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-04-06T13:49:26.451 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:26.452+0000 7fb1408eb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:49:26.560 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:26.560+0000 7fea8bf15200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:49:26.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.578+0000 7f5408606700 1 Processor -- start 2024-04-06T13:49:26.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.578+0000 7f5408606700 1 -- start start 2024-04-06T13:49:26.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.579+0000 7f5408606700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f54000b4660 0x7f54000b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:26.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.579+0000 7f5408606700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54000b2fc0 0x7f54000b3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:26.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.579+0000 7f5408606700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54000b5030 con 0x7f540015a550 2024-04-06T13:49:26.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.579+0000 7f5408606700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54000cdbe0 con 0x7f54000b4660 2024-04-06T13:49:26.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.579+0000 7f5408606700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540015a940 con 0x7f54000b2fc0 2024-04-06T13:49:26.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.580+0000 7f5405ba1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f54000b4660 0x7f54000b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:26.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.580+0000 7f5405ba1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f54000b4660 0x7f54000b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46424/0 (socket says 172.21.15.189:46424) 2024-04-06T13:49:26.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.580+0000 7f5405ba1700 1 -- 172.21.15.189:0/833321644 learned_addr learned my addr 172.21.15.189:0/833321644 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:26.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.580+0000 7f5406ba3700 1 --2- 172.21.15.189:0/833321644 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54000b2fc0 0x7f54000b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:26.581 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.580+0000 7f5405ba1700 1 -- 172.21.15.189:0/833321644 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54000b2fc0 msgr2=0x7f54000b3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:26.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.580+0000 7f5405ba1700 1 --2- 172.21.15.189:0/833321644 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54000b2fc0 0x7f54000b3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:26.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.580+0000 7f5405ba1700 1 -- 172.21.15.189:0/833321644 >> v1:172.21.15.192:6789/0 conn(0x7f540015a550 legacy=0x7f54000cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:26.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.580+0000 7f5405ba1700 1 -- 172.21.15.189:0/833321644 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540015ac00 con 0x7f54000b4660 2024-04-06T13:49:26.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.580+0000 7f5405ba1700 1 --2- 172.21.15.189:0/833321644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54000b4660 0x7f54000b4af0 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7f53f00098f0 tx=0x7f53f0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=601219420829bdde server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:26.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.580+0000 7f54053a0700 1 -- 172.21.15.189:0/833321644 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f0011830 con 0x7f54000b4660 2024-04-06T13:49:26.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.580+0000 7f54053a0700 1 -- 172.21.15.189:0/833321644 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f0011990 con 0x7f54000b4660 2024-04-06T13:49:26.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.580+0000 7f54053a0700 1 -- 172.21.15.189:0/833321644 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f0011d20 con 0x7f54000b4660 2024-04-06T13:49:26.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.581+0000 7f5408606700 1 -- 172.21.15.189:0/833321644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54000b4660 msgr2=0x7f54000b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:26.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.581+0000 7f5408606700 1 --2- 172.21.15.189:0/833321644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54000b4660 0x7f54000b4af0 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7f53f00098f0 tx=0x7f53f0009c00 comp rx=0 tx=0).stop 2024-04-06T13:49:26.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.581+0000 7f5408606700 1 -- 172.21.15.189:0/833321644 shutdown_connections 2024-04-06T13:49:26.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.581+0000 7f5408606700 1 --2- 172.21.15.189:0/833321644 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54000b2fc0 0x7f54000b3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:26.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.581+0000 7f5408606700 1 --2- 172.21.15.189:0/833321644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54000b4660 0x7f54000b4af0 unknown :-1 s=CLOSED pgs=2963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:26.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.581+0000 7f5408606700 1 -- 172.21.15.189:0/833321644 >> 172.21.15.189:0/833321644 conn(0x7f54000d00b0 msgr2=0x7f54000d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:26.582 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.581+0000 7f5408606700 1 -- 172.21.15.189:0/833321644 shutdown_connections 2024-04-06T13:49:26.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.581+0000 7f5408606700 1 -- 172.21.15.189:0/833321644 wait complete. 2024-04-06T13:49:26.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.582+0000 7f5408606700 1 Processor -- start 2024-04-06T13:49:26.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.582+0000 7f5408606700 1 -- start start 2024-04-06T13:49:26.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.583+0000 7f5408606700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54000b2fc0 0x7f540014eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:26.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.583+0000 7f5408606700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54000b4660 0x7f5400154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:26.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.583+0000 7f5408606700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54000ce590 0x7f54000cea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:26.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.583+0000 7f5408606700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540014fee0 con 0x7f54000b2fc0 2024-04-06T13:49:26.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.583+0000 7f5408606700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54000cef60 con 0x7f54000b4660 2024-04-06T13:49:26.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.583+0000 7f5408606700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5400155480 con 0x7f54000ce590 2024-04-06T13:49:26.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.583+0000 7f54063a2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54000b2fc0 0x7f540014eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:26.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.583+0000 7f54063a2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54000b2fc0 0x7f540014eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46428/0 (socket says 172.21.15.189:46428) 2024-04-06T13:49:26.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.583+0000 7f54063a2700 1 -- 172.21.15.189:0/1708332120 learned_addr learned my addr 172.21.15.189:0/1708332120 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:26.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.583+0000 7f5406ba3700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54000ce590 0x7f54000cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:26.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.583+0000 7f5405ba1700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54000b4660 0x7f5400154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:26.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.583+0000 7f54063a2700 1 -- 172.21.15.189:0/1708332120 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54000ce590 msgr2=0x7f54000cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:26.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.583+0000 7f54063a2700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54000ce590 0x7f54000cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:26.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.583+0000 7f54063a2700 1 -- 172.21.15.189:0/1708332120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54000b4660 msgr2=0x7f5400154f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:26.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.583+0000 7f54063a2700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54000b4660 0x7f5400154f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:26.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.583+0000 7f54063a2700 1 -- 172.21.15.189:0/1708332120 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f0009550 con 0x7f54000b2fc0 2024-04-06T13:49:26.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.583+0000 7f54063a2700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54000b2fc0 0x7f540014eb60 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7f53fc00c4e0 tx=0x7f53fc00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:26.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.584+0000 7f53f6ffd700 1 -- 172.21.15.189:0/1708332120 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc0052d0 con 0x7f54000b2fc0 2024-04-06T13:49:26.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.584+0000 7f53f6ffd700 1 -- 172.21.15.189:0/1708332120 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53fc005430 con 0x7f54000b2fc0 2024-04-06T13:49:26.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.584+0000 7f5408606700 1 -- 172.21.15.189:0/1708332120 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f540014f260 con 0x7f54000b2fc0 2024-04-06T13:49:26.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.584+0000 7f53f6ffd700 1 -- 172.21.15.189:0/1708332120 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc0038c0 con 0x7f54000b2fc0 2024-04-06T13:49:26.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.584+0000 7f5408606700 1 -- 172.21.15.189:0/1708332120 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f540014f820 con 0x7f54000b2fc0 2024-04-06T13:49:26.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.585+0000 7f5408606700 1 -- 172.21.15.189:0/1708332120 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5400067bb0 con 0x7f54000b2fc0 2024-04-06T13:49:26.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.585+0000 7f53f6ffd700 1 -- 172.21.15.189:0/1708332120 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 161) v1 ==== 89366+0+0 (secure 0 0 0) 0x7f53fc003a20 con 0x7f54000b2fc0 2024-04-06T13:49:26.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.586+0000 7f53f6ffd700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f53e406af20 0x7f53e406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:26.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.586+0000 7f5405ba1700 1 -- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f53e406af20 msgr2=0x7f53e406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:26.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.586+0000 7f53f6ffd700 1 -- 172.21.15.189:0/1708332120 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f53fc088cb0 con 0x7f54000b2fc0 2024-04-06T13:49:26.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.586+0000 7f5405ba1700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f53e406af20 0x7f53e406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:49:26.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.590+0000 7f53f6ffd700 1 -- 172.21.15.189:0/1708332120 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f53fc0539d0 con 0x7f54000b2fc0 2024-04-06T13:49:26.620 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:26.621+0000 7fb1408eb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:49:26.673 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:26.673+0000 7fea8bf15200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:49:26.736 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:26.736+0000 7fb1408eb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:49:26.754 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:26.754+0000 7fea8bf15200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:49:26.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.787+0000 7f5405ba1700 1 -- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f53e406af20 msgr2=0x7f53e406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:26.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.787+0000 7f5405ba1700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f53e406af20 0x7f53e406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:49:26.822 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:26.822+0000 7fb1408eb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:49:26.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:26.823+0000 7f5408606700 1 -- 172.21.15.189:0/1708332120 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f540015b4a0 con 0x7f53e406af20 2024-04-06T13:49:26.827 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:26.828+0000 7fea8bf15200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:49:26.898 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:26.898+0000 7fb1408eb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:49:27.030 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:27.030+0000 7fea8bf15200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:49:27.104 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:27.104+0000 7fb1408eb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:49:27.107 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:27.107+0000 7fea8bf15200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:49:27.181 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:27.181+0000 7fb1408eb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:49:27.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:27.188+0000 7f5405ba1700 1 -- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f53e406af20 msgr2=0x7f53e406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:27.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:27.188+0000 7f5405ba1700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f53e406af20 0x7f53e406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-06T13:49:27.427 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:27.427+0000 7fea8bf15200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:49:27.504 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:27.504+0000 7fb1408eb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:49:27.748 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:27.748+0000 7fea8bf15200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:49:27.830 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:27.830+0000 7fb1408eb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:49:27.907 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:27.907+0000 7fea8bf15200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:49:27.979 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:27.979+0000 7fea8bf15200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:49:27.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:27.989+0000 7f5405ba1700 1 -- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f53e406af20 msgr2=0x7f53e406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:27.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:27.989+0000 7f5405ba1700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f53e406af20 0x7f53e406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-06T13:49:27.992 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:27.992+0000 7fb1408eb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:49:28.066 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:28.067+0000 7fb1408eb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:49:28.594 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:28.594+0000 7fea8bf15200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:49:28.672 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:28.672+0000 7fea8bf15200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:49:28.695 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:28.696+0000 7fb1408eb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:49:28.775 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:28.775+0000 7fb1408eb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:49:28.797 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:28.797+0000 7fea8bf15200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:49:28.903 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:28.903+0000 7fb1408eb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:49:29.305 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:29.304+0000 7fea8bf15200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:49:29.422 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:29.422+0000 7fb1408eb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:49:29.458 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:29.459+0000 7fea8bf15200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:49:29.580 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:29.580+0000 7fb1408eb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:49:29.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:29.590+0000 7f5405ba1700 1 -- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f53e406af20 msgr2=0x7f53e406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:29.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:29.590+0000 7f5405ba1700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f53e406af20 0x7f53e406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-06T13:49:29.614 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:29.614+0000 7fea8bf15200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:49:29.742 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:29.742+0000 7fb1408eb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:49:30.335 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:30.335+0000 7fea8bf15200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:49:30.481 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:30.480+0000 7fb1408eb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:49:30.577 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:30.577+0000 7fea8bf15200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:49:30.653 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:30.653+0000 7fea8bf15200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:49:30.730 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:30.729+0000 7fb1408eb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:49:30.809 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:30.808+0000 7fb1408eb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:49:31.028 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:31.028+0000 7fea8bf15200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:49:31.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:31.099+0000 7f53f6ffd700 1 -- 172.21.15.189:0/1708332120 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 162) v1 ==== 89366+0+0 (secure 0 0 0) 0x7f53fc053da0 con 0x7f54000b2fc0 2024-04-06T13:49:31.194 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:31.193+0000 7fb1408eb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:49:31.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:31.214+0000 7f53f6ffd700 1 -- 172.21.15.189:0/1708332120 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mgrmap(e 163) v1 ==== 88873+0+0 (secure 0 0 0) 0x7f53fc05f5f0 con 0x7f54000b2fc0 2024-04-06T13:49:31.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:31.214+0000 7f53f6ffd700 1 -- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f53e406af20 msgr2=0x7f53e406d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:49:31.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:31.214+0000 7f53f6ffd700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f53e406af20 0x7f53e406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:31.357 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:31.355+0000 7fb0b67bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:31.357 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:31.355+0000 7fb0b67bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:31.357 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:31.355+0000 7fb0b67bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:31.357 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:31.355+0000 7fb0b67bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:31.357 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:31.355+0000 7fb0b67bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:31.357 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:31.355+0000 7fb0b37b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:31.357 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:31.355+0000 7fb0b37b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:31.357 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:31.355+0000 7fb0b37b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:31.357 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:31.355+0000 7fb0b37b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:31.358 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:31.355+0000 7fb0b37b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:32.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.216+0000 7f53f6ffd700 1 -- 172.21.15.189:0/1708332120 <== mon.0 v2:172.21.15.189:3300/0 9 ==== mgrmap(e 164) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f53fc07b580 con 0x7f54000b2fc0 2024-04-06T13:49:32.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.217+0000 7f53f6ffd700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f53e406af20 0x7f53e406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:32.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.217+0000 7f53f6ffd700 1 -- 172.21.15.189:0/1708332120 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f53e4028500 con 0x7f53e406af20 2024-04-06T13:49:32.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.220+0000 7f5405ba1700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f53e406af20 0x7f53e406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:32.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.221+0000 7f5405ba1700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f53e406af20 0x7f53e406d360 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f53f0009c40 tx=0x7f53f0008030 comp rx=0 tx=0).ready entity=mgr.13056 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:32.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.282+0000 7f53f6ffd700 1 -- 172.21.15.189:0/1708332120 <== mgr.13056 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f53e4028500 con 0x7f53e406af20 2024-04-06T13:49:32.283 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path snap_test_dir1 2024-04-06T13:49:32.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.284+0000 7f5408606700 1 -- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f53e406af20 msgr2=0x7f53e406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:32.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.284+0000 7f5408606700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f53e406af20 0x7f53e406d360 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f53f0009c40 tx=0x7f53f0008030 comp rx=0 tx=0).stop 2024-04-06T13:49:32.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.284+0000 7f5408606700 1 -- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54000b2fc0 msgr2=0x7f540014eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:32.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.284+0000 7f5408606700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54000b2fc0 0x7f540014eb60 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7f53fc00c4e0 tx=0x7f53fc00c7f0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.284+0000 7f5408606700 1 -- 172.21.15.189:0/1708332120 shutdown_connections 2024-04-06T13:49:32.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.285+0000 7f5408606700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f53e406af20 0x7f53e406d360 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.285+0000 7f5408606700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f54000ce590 0x7f54000cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.285+0000 7f5408606700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54000b2fc0 0x7f540014eb60 unknown :-1 s=CLOSED pgs=2964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.285+0000 7f5408606700 1 --2- 172.21.15.189:0/1708332120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54000b4660 0x7f5400154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.285+0000 7f5408606700 1 -- 172.21.15.189:0/1708332120 >> 172.21.15.189:0/1708332120 conn(0x7f54000d00b0 msgr2=0x7f5400159120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:32.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.285+0000 7f5408606700 1 -- 172.21.15.189:0/1708332120 shutdown_connections 2024-04-06T13:49:32.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.285+0000 7f5408606700 1 -- 172.21.15.189:0/1708332120 wait complete. 2024-04-06T13:49:32.298 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-04-06T13:49:32.298 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2024-04-06T13:49:32.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.423+0000 7faa2a6c5700 1 Processor -- start 2024-04-06T13:49:32.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.423+0000 7faa2a6c5700 1 -- start start 2024-04-06T13:49:32.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.424+0000 7faa2a6c5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faa240d17a0 0x7faa2406b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:32.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.424+0000 7faa2a6c5700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7faa240cedf0 0x7faa240cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:32.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.424+0000 7faa2a6c5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa240cf730 con 0x7faa24146dd0 2024-04-06T13:49:32.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.424+0000 7faa2a6c5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa2406d850 con 0x7faa240d17a0 2024-04-06T13:49:32.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.424+0000 7faa2a6c5700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa2406c0a0 con 0x7faa240cedf0 2024-04-06T13:49:32.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.424+0000 7faa23fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faa240d17a0 0x7faa2406b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:32.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.424+0000 7faa23fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faa240d17a0 0x7faa2406b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51476/0 (socket says 172.21.15.189:51476) 2024-04-06T13:49:32.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.424+0000 7faa23fff700 1 -- 172.21.15.189:0/2947321588 learned_addr learned my addr 172.21.15.189:0/2947321588 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:32.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.424+0000 7faa28c62700 1 --2- 172.21.15.189:0/2947321588 >> v2:172.21.15.192:3300/0 conn(0x7faa240cedf0 0x7faa240cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:32.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.424+0000 7faa23fff700 1 -- 172.21.15.189:0/2947321588 >> v2:172.21.15.192:3300/0 conn(0x7faa240cedf0 msgr2=0x7faa240cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:32.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.424+0000 7faa23fff700 1 --2- 172.21.15.189:0/2947321588 >> v2:172.21.15.192:3300/0 conn(0x7faa240cedf0 0x7faa240cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.424+0000 7faa23fff700 1 -- 172.21.15.189:0/2947321588 >> v1:172.21.15.192:6789/0 conn(0x7faa24146dd0 legacy=0x7faa2406d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:32.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.424+0000 7faa23fff700 1 -- 172.21.15.189:0/2947321588 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2406c380 con 0x7faa240d17a0 2024-04-06T13:49:32.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.424+0000 7faa28c62700 1 --2- 172.21.15.189:0/2947321588 >> v2:172.21.15.192:3300/0 conn(0x7faa240cedf0 0x7faa240cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:32.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.424+0000 7faa23fff700 1 --2- 172.21.15.189:0/2947321588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa240d17a0 0x7faa2406b7a0 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7faa24160fb0 tx=0x7faa18009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea287d0267c6bda server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:32.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.425+0000 7faa22ffd700 1 -- 172.21.15.189:0/2947321588 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18004230 con 0x7faa240d17a0 2024-04-06T13:49:32.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.425+0000 7faa22ffd700 1 -- 172.21.15.189:0/2947321588 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa18004390 con 0x7faa240d17a0 2024-04-06T13:49:32.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.425+0000 7faa22ffd700 1 -- 172.21.15.189:0/2947321588 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa180036a0 con 0x7faa240d17a0 2024-04-06T13:49:32.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.425+0000 7faa2a6c5700 1 -- 172.21.15.189:0/2947321588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa240d17a0 msgr2=0x7faa2406b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:32.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.425+0000 7faa2a6c5700 1 --2- 172.21.15.189:0/2947321588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa240d17a0 0x7faa2406b7a0 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7faa24160fb0 tx=0x7faa18009bd0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.425+0000 7faa2a6c5700 1 -- 172.21.15.189:0/2947321588 shutdown_connections 2024-04-06T13:49:32.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.425+0000 7faa2a6c5700 1 --2- 172.21.15.189:0/2947321588 >> v2:172.21.15.192:3300/0 conn(0x7faa240cedf0 0x7faa240cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.425+0000 7faa2a6c5700 1 --2- 172.21.15.189:0/2947321588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa240d17a0 0x7faa2406b7a0 unknown :-1 s=CLOSED pgs=2976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.425+0000 7faa2a6c5700 1 -- 172.21.15.189:0/2947321588 >> 172.21.15.189:0/2947321588 conn(0x7faa240b2db0 msgr2=0x7faa240b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:32.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.425+0000 7faa2a6c5700 1 -- 172.21.15.189:0/2947321588 shutdown_connections 2024-04-06T13:49:32.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.425+0000 7faa2a6c5700 1 -- 172.21.15.189:0/2947321588 wait complete. 2024-04-06T13:49:32.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.426+0000 7faa2a6c5700 1 Processor -- start 2024-04-06T13:49:32.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.426+0000 7faa2a6c5700 1 -- start start 2024-04-06T13:49:32.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.427+0000 7faa2a6c5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa240cedf0 0x7faa240cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:32.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.427+0000 7faa2a6c5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa240d17a0 0x7faa240c4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:32.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.427+0000 7faa2a6c5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa24146dd0 0x7faa240b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:32.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.427+0000 7faa2a6c5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa240afe50 con 0x7faa240d17a0 2024-04-06T13:49:32.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.427+0000 7faa2a6c5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa240cbff0 con 0x7faa24146dd0 2024-04-06T13:49:32.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.427+0000 7faa2a6c5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa240afb70 con 0x7faa240cedf0 2024-04-06T13:49:32.431 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.427+0000 7faa237fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa240d17a0 0x7faa240c4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:32.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.427+0000 7faa237fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa240d17a0 0x7faa240c4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51488/0 (socket says 172.21.15.189:51488) 2024-04-06T13:49:32.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.427+0000 7faa237fe700 1 -- 172.21.15.189:0/1672969530 learned_addr learned my addr 172.21.15.189:0/1672969530 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:32.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.427+0000 7faa28c62700 1 --2- 172.21.15.189:0/1672969530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa24146dd0 0x7faa240b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:32.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.427+0000 7faa23fff700 1 --2- 172.21.15.189:0/1672969530 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa240cedf0 0x7faa240cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:32.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.428+0000 7faa237fe700 1 -- 172.21.15.189:0/1672969530 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa240cedf0 msgr2=0x7faa240cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:32.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.428+0000 7faa237fe700 1 --2- 172.21.15.189:0/1672969530 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa240cedf0 0x7faa240cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.428+0000 7faa237fe700 1 -- 172.21.15.189:0/1672969530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa24146dd0 msgr2=0x7faa240b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:32.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.428+0000 7faa237fe700 1 --2- 172.21.15.189:0/1672969530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa24146dd0 0x7faa240b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.428+0000 7faa237fe700 1 -- 172.21.15.189:0/1672969530 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa18009550 con 0x7faa240d17a0 2024-04-06T13:49:32.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.428+0000 7faa237fe700 1 --2- 172.21.15.189:0/1672969530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa240d17a0 0x7faa240c4360 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7faa1400c4a0 tx=0x7faa1400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:32.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.428+0000 7faa20ff9700 1 -- 172.21.15.189:0/1672969530 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa14011790 con 0x7faa240d17a0 2024-04-06T13:49:32.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.428+0000 7faa20ff9700 1 -- 172.21.15.189:0/1672969530 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa140118f0 con 0x7faa240d17a0 2024-04-06T13:49:32.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.428+0000 7faa2a6c5700 1 -- 172.21.15.189:0/1672969530 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa240c8e30 con 0x7faa240d17a0 2024-04-06T13:49:32.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.428+0000 7faa20ff9700 1 -- 172.21.15.189:0/1672969530 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa14011ca0 con 0x7faa240d17a0 2024-04-06T13:49:32.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.428+0000 7faa2a6c5700 1 -- 172.21.15.189:0/1672969530 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa240c93f0 con 0x7faa240d17a0 2024-04-06T13:49:32.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.429+0000 7faa2a6c5700 1 -- 172.21.15.189:0/1672969530 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa24074990 con 0x7faa240d17a0 2024-04-06T13:49:32.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.429+0000 7faa20ff9700 1 -- 172.21.15.189:0/1672969530 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 164) v1 ==== 89000+0+0 (secure 0 0 0) 0x7faa1401c460 con 0x7faa240d17a0 2024-04-06T13:49:32.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.430+0000 7faa20ff9700 1 --2- 172.21.15.189:0/1672969530 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faa1006ab30 0x7faa1006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:32.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.430+0000 7faa20ff9700 1 -- 172.21.15.189:0/1672969530 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6531+0+0 (secure 0 0 0) 0x7faa14088fa0 con 0x7faa240d17a0 2024-04-06T13:49:32.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.430+0000 7faa23fff700 1 --2- 172.21.15.189:0/1672969530 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faa1006ab30 0x7faa1006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:32.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.430+0000 7faa23fff700 1 --2- 172.21.15.189:0/1672969530 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faa1006ab30 0x7faa1006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faa18009130 tx=0x7faa18008030 comp rx=0 tx=0).ready entity=mgr.13056 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:32.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.434+0000 7faa20ff9700 1 -- 172.21.15.189:0/1672969530 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7faa140bc650 con 0x7faa240d17a0 2024-04-06T13:49:32.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.654+0000 7faa2a6c5700 1 -- 172.21.15.189:0/1672969530 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7faa2406cc20 con 0x7faa1006ab30 2024-04-06T13:49:32.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.709+0000 7faa20ff9700 1 -- 172.21.15.189:0/1672969530 <== mgr.13056 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7faa18006c60 con 0x7faa1006ab30 2024-04-06T13:49:32.710 INFO:teuthology.orchestra.run.smithi189.stdout:Retention added to path snap_test_dir1 2024-04-06T13:49:32.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.712+0000 7faa2a6c5700 1 -- 172.21.15.189:0/1672969530 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faa1006ab30 msgr2=0x7faa1006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:32.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.712+0000 7faa2a6c5700 1 --2- 172.21.15.189:0/1672969530 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faa1006ab30 0x7faa1006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faa18009130 tx=0x7faa18008030 comp rx=0 tx=0).stop 2024-04-06T13:49:32.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.712+0000 7faa2a6c5700 1 -- 172.21.15.189:0/1672969530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa240d17a0 msgr2=0x7faa240c4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:32.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.712+0000 7faa2a6c5700 1 --2- 172.21.15.189:0/1672969530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa240d17a0 0x7faa240c4360 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7faa1400c4a0 tx=0x7faa1400c7b0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.712+0000 7faa2a6c5700 1 -- 172.21.15.189:0/1672969530 shutdown_connections 2024-04-06T13:49:32.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.712+0000 7faa2a6c5700 1 --2- 172.21.15.189:0/1672969530 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7faa1006ab30 0x7faa1006cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.712+0000 7faa2a6c5700 1 --2- 172.21.15.189:0/1672969530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa24146dd0 0x7faa240b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.712+0000 7faa2a6c5700 1 --2- 172.21.15.189:0/1672969530 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa240cedf0 0x7faa240cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.712+0000 7faa2a6c5700 1 --2- 172.21.15.189:0/1672969530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa240d17a0 0x7faa240c4360 unknown :-1 s=CLOSED pgs=2977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.712+0000 7faa2a6c5700 1 -- 172.21.15.189:0/1672969530 >> 172.21.15.189:0/1672969530 conn(0x7faa240b2db0 msgr2=0x7faa2406d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:32.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.712+0000 7faa2a6c5700 1 -- 172.21.15.189:0/1672969530 shutdown_connections 2024-04-06T13:49:32.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.712+0000 7faa2a6c5700 1 -- 172.21.15.189:0/1672969530 wait complete. 2024-04-06T13:49:32.725 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2024-04-06T13:49:32.725 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-04-06-13_50_00 in ~28s... 2024-04-06T13:49:32.726 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2024-04-06T13:49:32.726 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-04-06T13:49:32.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.849+0000 7fa8ef0cf700 1 Processor -- start 2024-04-06T13:49:32.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.849+0000 7fa8ef0cf700 1 -- start start 2024-04-06T13:49:32.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.850+0000 7fa8ef0cf700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa8e806ae40 0x7fa8e80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:32.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.850+0000 7fa8ef0cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e815ddf0 0x7fa8e815e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:32.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.850+0000 7fa8ef0cf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e80b3990 con 0x7fa8e815d7b0 2024-04-06T13:49:32.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.850+0000 7fa8ef0cf700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8e80b3060 con 0x7fa8e806ae40 2024-04-06T13:49:32.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.850+0000 7fa8ef0cf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e80b3340 con 0x7fa8e815ddf0 2024-04-06T13:49:32.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.850+0000 7fa8ece6b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa8e815d7b0 0x7fa8e806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46226/0 (socket says 172.21.15.189:46226) 2024-04-06T13:49:32.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.850+0000 7fa8ece6b700 1 -- 172.21.15.189:0/1098290698 learned_addr learned my addr 172.21.15.189:0/1098290698 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:32.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.850+0000 7fa8ed66c700 1 --2- 172.21.15.189:0/1098290698 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e815ddf0 0x7fa8e815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:32.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.850+0000 7fa8e7fff700 1 --2- 172.21.15.189:0/1098290698 >> v2:172.21.15.192:3300/0 conn(0x7fa8e806ae40 0x7fa8e80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:32.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.850+0000 7fa8e77fe700 1 -- 172.21.15.189:0/1098290698 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2760691407 0 0) 0x7fa8e80b3990 con 0x7fa8e815d7b0 2024-04-06T13:49:32.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.850+0000 7fa8e77fe700 1 -- 172.21.15.189:0/1098290698 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8d80035b0 con 0x7fa8e815d7b0 2024-04-06T13:49:32.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.850+0000 7fa8ed66c700 1 -- 172.21.15.189:0/1098290698 >> v2:172.21.15.192:3300/0 conn(0x7fa8e806ae40 msgr2=0x7fa8e80b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:32.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.850+0000 7fa8ed66c700 1 --2- 172.21.15.189:0/1098290698 >> v2:172.21.15.192:3300/0 conn(0x7fa8e806ae40 0x7fa8e80b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.850+0000 7fa8ed66c700 1 -- 172.21.15.189:0/1098290698 >> v1:172.21.15.189:6789/0 conn(0x7fa8e815d7b0 legacy=0x7fa8e806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:32.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.850+0000 7fa8ed66c700 1 -- 172.21.15.189:0/1098290698 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e80b3620 con 0x7fa8e815ddf0 2024-04-06T13:49:32.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.850+0000 7fa8e7fff700 1 --2- 172.21.15.189:0/1098290698 >> v2:172.21.15.192:3300/0 conn(0x7fa8e806ae40 0x7fa8e80b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:32.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.850+0000 7fa8ed66c700 1 --2- 172.21.15.189:0/1098290698 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e815ddf0 0x7fa8e815e280 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0009d90 tx=0x7fa8e000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8130e82adef32bd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:32.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.850+0000 7fa8e77fe700 1 -- 172.21.15.189:0/1098290698 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e000e070 con 0x7fa8e815ddf0 2024-04-06T13:49:32.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.851+0000 7fa8e77fe700 1 -- 172.21.15.189:0/1098290698 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e000bb20 con 0x7fa8e815ddf0 2024-04-06T13:49:32.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.851+0000 7fa8e77fe700 1 -- 172.21.15.189:0/1098290698 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0012430 con 0x7fa8e815ddf0 2024-04-06T13:49:32.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.851+0000 7fa8ef0cf700 1 -- 172.21.15.189:0/1098290698 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e815ddf0 msgr2=0x7fa8e815e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:32.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.851+0000 7fa8ef0cf700 1 --2- 172.21.15.189:0/1098290698 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e815ddf0 0x7fa8e815e280 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0009d90 tx=0x7fa8e000b330 comp rx=0 tx=0).stop 2024-04-06T13:49:32.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.851+0000 7fa8ef0cf700 1 -- 172.21.15.189:0/1098290698 shutdown_connections 2024-04-06T13:49:32.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.851+0000 7fa8ef0cf700 1 --2- 172.21.15.189:0/1098290698 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e815ddf0 0x7fa8e815e280 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.851+0000 7fa8ef0cf700 1 --2- 172.21.15.189:0/1098290698 >> v2:172.21.15.192:3300/0 conn(0x7fa8e806ae40 0x7fa8e80b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.851+0000 7fa8ef0cf700 1 -- 172.21.15.189:0/1098290698 >> 172.21.15.189:0/1098290698 conn(0x7fa8e80d00b0 msgr2=0x7fa8e80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:32.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.851+0000 7fa8ef0cf700 1 -- 172.21.15.189:0/1098290698 shutdown_connections 2024-04-06T13:49:32.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.851+0000 7fa8ef0cf700 1 -- 172.21.15.189:0/1098290698 wait complete. 2024-04-06T13:49:32.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.852+0000 7fa8ef0cf700 1 Processor -- start 2024-04-06T13:49:32.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.852+0000 7fa8ef0cf700 1 -- start start 2024-04-06T13:49:32.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.853+0000 7fa8ef0cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e806ae40 0x7fa8e81497d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:32.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.853+0000 7fa8ef0cf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8e815d7b0 0x7fa8e80b0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:32.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.853+0000 7fa8ef0cf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e815ddf0 0x7fa8e81559c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:32.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.853+0000 7fa8ef0cf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e81555a0 con 0x7fa8e815ddf0 2024-04-06T13:49:32.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.853+0000 7fa8ef0cf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e8149d10 con 0x7fa8e815d7b0 2024-04-06T13:49:32.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.853+0000 7fa8ef0cf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e80b1540 con 0x7fa8e806ae40 2024-04-06T13:49:32.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.853+0000 7fa8ed66c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e815ddf0 0x7fa8e81559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:32.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.853+0000 7fa8ed66c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e815ddf0 0x7fa8e81559c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51498/0 (socket says 172.21.15.189:51498) 2024-04-06T13:49:32.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.853+0000 7fa8ed66c700 1 -- 172.21.15.189:0/3085634816 learned_addr learned my addr 172.21.15.189:0/3085634816 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:32.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.854+0000 7fa8e7fff700 1 --2- 172.21.15.189:0/3085634816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8e815d7b0 0x7fa8e80b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:32.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.854+0000 7fa8ece6b700 1 --2- 172.21.15.189:0/3085634816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e806ae40 0x7fa8e81497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:32.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.854+0000 7fa8ed66c700 1 -- 172.21.15.189:0/3085634816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e806ae40 msgr2=0x7fa8e81497d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:32.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.854+0000 7fa8ed66c700 1 --2- 172.21.15.189:0/3085634816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e806ae40 0x7fa8e81497d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.854+0000 7fa8ed66c700 1 -- 172.21.15.189:0/3085634816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8e815d7b0 msgr2=0x7fa8e80b0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:32.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.854+0000 7fa8ed66c700 1 --2- 172.21.15.189:0/3085634816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8e815d7b0 0x7fa8e80b0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:32.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.854+0000 7fa8ed66c700 1 -- 172.21.15.189:0/3085634816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d40094a0 con 0x7fa8e815ddf0 2024-04-06T13:49:32.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.854+0000 7fa8ed66c700 1 --2- 172.21.15.189:0/3085634816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e815ddf0 0x7fa8e81559c0 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0014040 tx=0x7fa8e0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:32.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.854+0000 7fa8e57fa700 1 -- 172.21.15.189:0/3085634816 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e00129f0 con 0x7fa8e815ddf0 2024-04-06T13:49:32.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.854+0000 7fa8e57fa700 1 -- 172.21.15.189:0/3085634816 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e0020070 con 0x7fa8e815ddf0 2024-04-06T13:49:32.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.854+0000 7fa8e57fa700 1 -- 172.21.15.189:0/3085634816 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0012db0 con 0x7fa8e815ddf0 2024-04-06T13:49:32.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.854+0000 7fa8ef0cf700 1 -- 172.21.15.189:0/3085634816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e0009a70 con 0x7fa8e815ddf0 2024-04-06T13:49:32.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.854+0000 7fa8ef0cf700 1 -- 172.21.15.189:0/3085634816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e80b1260 con 0x7fa8e815ddf0 2024-04-06T13:49:32.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.855+0000 7fa8ef0cf700 1 -- 172.21.15.189:0/3085634816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e8067bb0 con 0x7fa8e815ddf0 2024-04-06T13:49:32.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.855+0000 7fa8e57fa700 1 -- 172.21.15.189:0/3085634816 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 164) v1 ==== 89000+0+0 (secure 0 0 0) 0x7fa8e0004040 con 0x7fa8e815ddf0 2024-04-06T13:49:32.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.856+0000 7fa8e57fa700 1 --2- 172.21.15.189:0/3085634816 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa8cc06ab30 0x7fa8cc06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:32.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.856+0000 7fa8e57fa700 1 -- 172.21.15.189:0/3085634816 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6531+0+0 (secure 0 0 0) 0x7fa8e0022070 con 0x7fa8e815ddf0 2024-04-06T13:49:32.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.856+0000 7fa8ece6b700 1 --2- 172.21.15.189:0/3085634816 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa8cc06ab30 0x7fa8cc06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:32.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.856+0000 7fa8ece6b700 1 --2- 172.21.15.189:0/3085634816 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa8cc06ab30 0x7fa8cc06cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc007fd0 tx=0x7fa8dc007dc0 comp rx=0 tx=0).ready entity=mgr.13056 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:32.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:32.861+0000 7fa8e57fa700 1 -- 172.21.15.189:0/3085634816 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa8e0056060 con 0x7fa8e815ddf0 2024-04-06T13:49:33.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.079+0000 7fa8ef0cf700 1 -- 172.21.15.189:0/3085634816 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa8e815e7c0 con 0x7fa8cc06ab30 2024-04-06T13:49:33.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.089+0000 7fa8e57fa700 1 -- 172.21.15.189:0/3085634816 <== mgr.13056 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7fa8dc006bd0 con 0x7fa8cc06ab30 2024-04-06T13:49:33.090 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:33.090 INFO:teuthology.orchestra.run.smithi189.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-04-06T13:49:33.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.091+0000 7fa8ef0cf700 1 -- 172.21.15.189:0/3085634816 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa8cc06ab30 msgr2=0x7fa8cc06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:33.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.091+0000 7fa8ef0cf700 1 --2- 172.21.15.189:0/3085634816 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa8cc06ab30 0x7fa8cc06cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc007fd0 tx=0x7fa8dc007dc0 comp rx=0 tx=0).stop 2024-04-06T13:49:33.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.091+0000 7fa8ef0cf700 1 -- 172.21.15.189:0/3085634816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e815ddf0 msgr2=0x7fa8e81559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:33.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.091+0000 7fa8ef0cf700 1 --2- 172.21.15.189:0/3085634816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e815ddf0 0x7fa8e81559c0 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0014040 tx=0x7fa8e0004ad0 comp rx=0 tx=0).stop 2024-04-06T13:49:33.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.091+0000 7fa8ef0cf700 1 -- 172.21.15.189:0/3085634816 shutdown_connections 2024-04-06T13:49:33.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.091+0000 7fa8ef0cf700 1 --2- 172.21.15.189:0/3085634816 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa8cc06ab30 0x7fa8cc06cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:33.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.091+0000 7fa8ef0cf700 1 --2- 172.21.15.189:0/3085634816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e815ddf0 0x7fa8e81559c0 unknown :-1 s=CLOSED pgs=2979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:33.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.091+0000 7fa8ef0cf700 1 --2- 172.21.15.189:0/3085634816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e806ae40 0x7fa8e81497d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:33.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.091+0000 7fa8ef0cf700 1 --2- 172.21.15.189:0/3085634816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8e815d7b0 0x7fa8e80b0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:33.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.091+0000 7fa8ef0cf700 1 -- 172.21.15.189:0/3085634816 >> 172.21.15.189:0/3085634816 conn(0x7fa8e80d00b0 msgr2=0x7fa8e815f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:33.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.091+0000 7fa8ef0cf700 1 -- 172.21.15.189:0/3085634816 shutdown_connections 2024-04-06T13:49:33.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.091+0000 7fa8ef0cf700 1 -- 172.21.15.189:0/3085634816 wait complete. 2024-04-06T13:49:33.105 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-04-06T13:49:33.105 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2024-04-06T13:49:33.105 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-04-06T13:49:33.212 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:49:33.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:49:33.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.230+0000 7fd8edc7a700 1 Processor -- start 2024-04-06T13:49:33.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.230+0000 7fd8edc7a700 1 -- start start 2024-04-06T13:49:33.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.230+0000 7fd8edc7a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8e8158fe0 0x7fd8e806a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:33.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.230+0000 7fd8edc7a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e815f9e0 con 0x7fd8e806ac70 2024-04-06T13:49:33.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.231+0000 7fd8edc7a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e815f0d0 con 0x7fd8e815c2d0 2024-04-06T13:49:33.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.231+0000 7fd8edc7a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e806b0f0 con 0x7fd8e8158fe0 2024-04-06T13:49:33.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.231+0000 7fd8e6ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd8e806ac70 0x7fd8e815bbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46240/0 (socket says 172.21.15.189:46240) 2024-04-06T13:49:33.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.231+0000 7fd8e6ffd700 1 -- 172.21.15.189:0/2251168115 learned_addr learned my addr 172.21.15.189:0/2251168115 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:33.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.231+0000 7fd8e77fe700 1 --2- 172.21.15.189:0/2251168115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8e8158fe0 0x7fd8e806a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:33.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.231+0000 7fd8e67fc700 1 -- 172.21.15.189:0/2251168115 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574657846 0 0) 0x7fd8e815f9e0 con 0x7fd8e806ac70 2024-04-06T13:49:33.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.231+0000 7fd8e67fc700 1 -- 172.21.15.189:0/2251168115 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8d00035b0 con 0x7fd8e806ac70 2024-04-06T13:49:33.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.231+0000 7fd8e67fc700 1 -- 172.21.15.189:0/2251168115 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914957295 0 0) 0x7fd8e815f0d0 con 0x7fd8e815c2d0 2024-04-06T13:49:33.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.231+0000 7fd8e67fc700 1 -- 172.21.15.189:0/2251168115 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8e815f9e0 con 0x7fd8e815c2d0 2024-04-06T13:49:33.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.231+0000 7fd8e77fe700 1 -- 172.21.15.189:0/2251168115 >> v1:172.21.15.192:6789/0 conn(0x7fd8e815c2d0 legacy=0x7fd8e80ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:33.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.231+0000 7fd8e77fe700 1 -- 172.21.15.189:0/2251168115 >> v1:172.21.15.189:6789/0 conn(0x7fd8e806ac70 legacy=0x7fd8e815bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:33.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.231+0000 7fd8e77fe700 1 -- 172.21.15.189:0/2251168115 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e815c8f0 con 0x7fd8e8158fe0 2024-04-06T13:49:33.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.231+0000 7fd8e77fe700 1 --2- 172.21.15.189:0/2251168115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8e8158fe0 0x7fd8e806a730 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8009870 tx=0x7fd8d8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee25cb342dec1f72 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:33.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.231+0000 7fd8e67fc700 1 -- 172.21.15.189:0/2251168115 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8011830 con 0x7fd8e8158fe0 2024-04-06T13:49:33.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.231+0000 7fd8e67fc700 1 -- 172.21.15.189:0/2251168115 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d8011990 con 0x7fd8e8158fe0 2024-04-06T13:49:33.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.232+0000 7fd8e67fc700 1 -- 172.21.15.189:0/2251168115 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8011d00 con 0x7fd8e8158fe0 2024-04-06T13:49:33.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.232+0000 7fd8edc7a700 1 -- 172.21.15.189:0/2251168115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8e8158fe0 msgr2=0x7fd8e806a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:33.234 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.232+0000 7fd8edc7a700 1 --2- 172.21.15.189:0/2251168115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8e8158fe0 0x7fd8e806a730 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8009870 tx=0x7fd8d8009b80 comp rx=0 tx=0).stop 2024-04-06T13:49:33.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.232+0000 7fd8edc7a700 1 -- 172.21.15.189:0/2251168115 shutdown_connections 2024-04-06T13:49:33.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.232+0000 7fd8edc7a700 1 --2- 172.21.15.189:0/2251168115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8e8158fe0 0x7fd8e806a730 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:33.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.232+0000 7fd8edc7a700 1 -- 172.21.15.189:0/2251168115 >> 172.21.15.189:0/2251168115 conn(0x7fd8e80d00b0 msgr2=0x7fd8e80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:33.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.232+0000 7fd8edc7a700 1 -- 172.21.15.189:0/2251168115 shutdown_connections 2024-04-06T13:49:33.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.232+0000 7fd8edc7a700 1 -- 172.21.15.189:0/2251168115 wait complete. 2024-04-06T13:49:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.233+0000 7fd8edc7a700 1 Processor -- start 2024-04-06T13:49:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.234+0000 7fd8edc7a700 1 -- start start 2024-04-06T13:49:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.234+0000 7fd8edc7a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8e806ac70 0x7fd8e815e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.234+0000 7fd8edc7a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8e8158fe0 0x7fd8e813cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.234+0000 7fd8edc7a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8e815c2d0 0x7fd8e80d0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.234+0000 7fd8edc7a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e815d700 con 0x7fd8e815c2d0 2024-04-06T13:49:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.234+0000 7fd8edc7a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e815d9e0 con 0x7fd8e8158fe0 2024-04-06T13:49:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.234+0000 7fd8edc7a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e815dcc0 con 0x7fd8e806ac70 2024-04-06T13:49:33.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.234+0000 7fd8e7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8e815c2d0 0x7fd8e80d0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.234+0000 7fd8e77fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8e806ac70 0x7fd8e815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.234+0000 7fd8e7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8e815c2d0 0x7fd8e80d0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51502/0 (socket says 172.21.15.189:51502) 2024-04-06T13:49:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.235+0000 7fd8e6ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8e8158fe0 0x7fd8e813cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.235+0000 7fd8e7fff700 1 -- 172.21.15.189:0/630731824 learned_addr learned my addr 172.21.15.189:0/630731824 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.235+0000 7fd8e7fff700 1 -- 172.21.15.189:0/630731824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8e806ac70 msgr2=0x7fd8e815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.235+0000 7fd8e7fff700 1 --2- 172.21.15.189:0/630731824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8e806ac70 0x7fd8e815e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.235+0000 7fd8e7fff700 1 -- 172.21.15.189:0/630731824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8e8158fe0 msgr2=0x7fd8e813cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.235+0000 7fd8e7fff700 1 --2- 172.21.15.189:0/630731824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8e8158fe0 0x7fd8e813cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.235+0000 7fd8e7fff700 1 -- 172.21.15.189:0/630731824 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d80094a0 con 0x7fd8e815c2d0 2024-04-06T13:49:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.235+0000 7fd8e7fff700 1 --2- 172.21.15.189:0/630731824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8e815c2d0 0x7fd8e80d0490 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000cbc0 tx=0x7fd8e000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.235+0000 7fd8cffff700 1 -- 172.21.15.189:0/630731824 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0003c00 con 0x7fd8e815c2d0 2024-04-06T13:49:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.235+0000 7fd8cffff700 1 -- 172.21.15.189:0/630731824 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e0003d60 con 0x7fd8e815c2d0 2024-04-06T13:49:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.235+0000 7fd8edc7a700 1 -- 172.21.15.189:0/630731824 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e815dfa0 con 0x7fd8e815c2d0 2024-04-06T13:49:33.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.235+0000 7fd8cffff700 1 -- 172.21.15.189:0/630731824 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e000aaa0 con 0x7fd8e815c2d0 2024-04-06T13:49:33.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.235+0000 7fd8edc7a700 1 -- 172.21.15.189:0/630731824 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e815cda0 con 0x7fd8e815c2d0 2024-04-06T13:49:33.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.236+0000 7fd8edc7a700 1 -- 172.21.15.189:0/630731824 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e815e4c0 con 0x7fd8e815c2d0 2024-04-06T13:49:33.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.237+0000 7fd8cffff700 1 -- 172.21.15.189:0/630731824 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 164) v1 ==== 89000+0+0 (secure 0 0 0) 0x7fd8e000ac00 con 0x7fd8e815c2d0 2024-04-06T13:49:33.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.237+0000 7fd8cffff700 1 --2- 172.21.15.189:0/630731824 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd8d406ab30 0x7fd8d406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:33.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.237+0000 7fd8cffff700 1 -- 172.21.15.189:0/630731824 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6531+0+0 (secure 0 0 0) 0x7fd8e0089280 con 0x7fd8e815c2d0 2024-04-06T13:49:33.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.237+0000 7fd8e77fe700 1 --2- 172.21.15.189:0/630731824 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd8d406ab30 0x7fd8d406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:33.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.238+0000 7fd8e77fe700 1 --2- 172.21.15.189:0/630731824 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd8d406ab30 0x7fd8d406cff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd8d80090b0 tx=0x7fd8d8008030 comp rx=0 tx=0).ready entity=mgr.13056 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:33.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.242+0000 7fd8cffff700 1 -- 172.21.15.189:0/630731824 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd8e815e4c0 con 0x7fd8e815c2d0 2024-04-06T13:49:33.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.472+0000 7fd8edc7a700 1 -- 172.21.15.189:0/630731824 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8e80acba0 con 0x7fd8d406ab30 2024-04-06T13:49:33.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.529+0000 7fd8cffff700 1 -- 172.21.15.189:0/630731824 <== mgr.13056 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fd8e80acba0 con 0x7fd8d406ab30 2024-04-06T13:49:33.530 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule removed for path snap_test_dir1 2024-04-06T13:49:33.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.531+0000 7fd8edc7a700 1 -- 172.21.15.189:0/630731824 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd8d406ab30 msgr2=0x7fd8d406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.531+0000 7fd8edc7a700 1 --2- 172.21.15.189:0/630731824 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd8d406ab30 0x7fd8d406cff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd8d80090b0 tx=0x7fd8d8008030 comp rx=0 tx=0).stop 2024-04-06T13:49:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.531+0000 7fd8edc7a700 1 -- 172.21.15.189:0/630731824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8e815c2d0 msgr2=0x7fd8e80d0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.531+0000 7fd8edc7a700 1 --2- 172.21.15.189:0/630731824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8e815c2d0 0x7fd8e80d0490 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000cbc0 tx=0x7fd8e000ced0 comp rx=0 tx=0).stop 2024-04-06T13:49:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.531+0000 7fd8edc7a700 1 -- 172.21.15.189:0/630731824 shutdown_connections 2024-04-06T13:49:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.531+0000 7fd8edc7a700 1 --2- 172.21.15.189:0/630731824 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd8d406ab30 0x7fd8d406cff0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.531+0000 7fd8edc7a700 1 --2- 172.21.15.189:0/630731824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8e815c2d0 0x7fd8e80d0490 unknown :-1 s=CLOSED pgs=2981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.531+0000 7fd8edc7a700 1 --2- 172.21.15.189:0/630731824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8e806ac70 0x7fd8e815e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.531+0000 7fd8edc7a700 1 --2- 172.21.15.189:0/630731824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8e8158fe0 0x7fd8e813cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.531+0000 7fd8edc7a700 1 -- 172.21.15.189:0/630731824 >> 172.21.15.189:0/630731824 conn(0x7fd8e80d00b0 msgr2=0x7fd8e806f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.531+0000 7fd8edc7a700 1 -- 172.21.15.189:0/630731824 shutdown_connections 2024-04-06T13:49:33.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.531+0000 7fd8edc7a700 1 -- 172.21.15.189:0/630731824 wait complete. 2024-04-06T13:49:33.545 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-04-06T13:49:33.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:49:33.545 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-06T13:49:33.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:49:33.563 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-06T13:49:33.622 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-06T13:49:33.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.786+0000 7f6393d45700 1 Processor -- start 2024-04-06T13:49:33.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.786+0000 7f6393d45700 1 -- start start 2024-04-06T13:49:33.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.787+0000 7f6393d45700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f638c15a550 0x7f638c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:33.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.787+0000 7f6393d45700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f638c0b4d70 0x7f638c0b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.787+0000 7f6393d45700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c0cdbe0 con 0x7f638c0cf900 2024-04-06T13:49:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.787+0000 7f6393d45700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f638c15a940 con 0x7f638c0b4d70 2024-04-06T13:49:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.787+0000 7f6393d45700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f638c15ac00 con 0x7f638c15a550 2024-04-06T13:49:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.787+0000 7f63922e2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f638c0b4d70 0x7f638c0b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.787+0000 7f63922e2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f638c0b4d70 0x7f638c0b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51514/0 (socket says 172.21.15.189:51514) 2024-04-06T13:49:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.787+0000 7f63922e2700 1 -- 172.21.15.189:0/3278910792 learned_addr learned my addr 172.21.15.189:0/3278910792 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.787+0000 7f6391ae1700 1 --2- 172.21.15.189:0/3278910792 >> v2:172.21.15.192:3300/0 conn(0x7f638c15a550 0x7f638c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.787+0000 7f63922e2700 1 -- 172.21.15.189:0/3278910792 >> v2:172.21.15.192:3300/0 conn(0x7f638c15a550 msgr2=0x7f638c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.787+0000 7f63922e2700 1 --2- 172.21.15.189:0/3278910792 >> v2:172.21.15.192:3300/0 conn(0x7f638c15a550 0x7f638c0cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.787+0000 7f63922e2700 1 -- 172.21.15.189:0/3278910792 >> v1:172.21.15.189:6789/0 conn(0x7f638c0cf900 legacy=0x7f638c0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.787+0000 7f63922e2700 1 -- 172.21.15.189:0/3278910792 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638c15aee0 con 0x7f638c0b4d70 2024-04-06T13:49:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.787+0000 7f63922e2700 1 --2- 172.21.15.189:0/3278910792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f638c0b4d70 0x7f638c0b2fc0 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7f638c073470 tx=0x7f638400d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b37d64b7138a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.787+0000 7f6390adf700 1 -- 172.21.15.189:0/3278910792 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6384010070 con 0x7f638c0b4d70 2024-04-06T13:49:33.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.787+0000 7f6390adf700 1 -- 172.21.15.189:0/3278910792 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638400dd90 con 0x7f638c0b4d70 2024-04-06T13:49:33.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.787+0000 7f6390adf700 1 -- 172.21.15.189:0/3278910792 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638400b310 con 0x7f638c0b4d70 2024-04-06T13:49:33.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.788+0000 7f6393d45700 1 -- 172.21.15.189:0/3278910792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f638c0b4d70 msgr2=0x7f638c0b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:33.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.788+0000 7f6393d45700 1 --2- 172.21.15.189:0/3278910792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f638c0b4d70 0x7f638c0b2fc0 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7f638c073470 tx=0x7f638400d430 comp rx=0 tx=0).stop 2024-04-06T13:49:33.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.788+0000 7f6393d45700 1 -- 172.21.15.189:0/3278910792 shutdown_connections 2024-04-06T13:49:33.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.788+0000 7f6393d45700 1 --2- 172.21.15.189:0/3278910792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f638c0b4d70 0x7f638c0b2fc0 unknown :-1 s=CLOSED pgs=2985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:33.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.788+0000 7f6393d45700 1 --2- 172.21.15.189:0/3278910792 >> v2:172.21.15.192:3300/0 conn(0x7f638c15a550 0x7f638c0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:33.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.788+0000 7f6393d45700 1 -- 172.21.15.189:0/3278910792 >> 172.21.15.189:0/3278910792 conn(0x7f638c0d00b0 msgr2=0x7f638c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:33.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.788+0000 7f6393d45700 1 -- 172.21.15.189:0/3278910792 shutdown_connections 2024-04-06T13:49:33.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.788+0000 7f6393d45700 1 -- 172.21.15.189:0/3278910792 wait complete. 2024-04-06T13:49:33.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.789+0000 7f6393d45700 1 Processor -- start 2024-04-06T13:49:33.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.789+0000 7f6393d45700 1 -- start start 2024-04-06T13:49:33.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.790+0000 7f6393d45700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f638c0b4d70 0x7f638c157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:33.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.790+0000 7f6393d45700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f638c0cf900 0x7f638c1f2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:33.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.790+0000 7f6393d45700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f638c151600 0x7f638c1436e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:33.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.790+0000 7f6393d45700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638c1f31c0 con 0x7f638c0b4d70 2024-04-06T13:49:33.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.790+0000 7f6393d45700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638c1f2c00 con 0x7f638c151600 2024-04-06T13:49:33.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.790+0000 7f6393d45700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638c1f2ee0 con 0x7f638c0cf900 2024-04-06T13:49:33.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.790+0000 7f6391ae1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f638c0b4d70 0x7f638c157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:33.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.790+0000 7f6391ae1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f638c0b4d70 0x7f638c157bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51524/0 (socket says 172.21.15.189:51524) 2024-04-06T13:49:33.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.790+0000 7f6391ae1700 1 -- 172.21.15.189:0/1893819350 learned_addr learned my addr 172.21.15.189:0/1893819350 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:33.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.790+0000 7f6391ae1700 1 -- 172.21.15.189:0/1893819350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f638c0cf900 msgr2=0x7f638c1f2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:33.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.790+0000 7f6391ae1700 1 --2- 172.21.15.189:0/1893819350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f638c0cf900 0x7f638c1f2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:33.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.790+0000 7f6391ae1700 1 -- 172.21.15.189:0/1893819350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f638c151600 msgr2=0x7f638c1436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:33.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.790+0000 7f63922e2700 1 --2- 172.21.15.189:0/1893819350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f638c151600 0x7f638c1436e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:33.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.790+0000 7f6391ae1700 1 --2- 172.21.15.189:0/1893819350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f638c151600 0x7f638c1436e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:33.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.790+0000 7f6391ae1700 1 -- 172.21.15.189:0/1893819350 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638400bd40 con 0x7f638c0b4d70 2024-04-06T13:49:33.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.791+0000 7f6391ae1700 1 --2- 172.21.15.189:0/1893819350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f638c0b4d70 0x7f638c157bf0 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7f638800d800 tx=0x7f638800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:33.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.791+0000 7f63827fc700 1 -- 172.21.15.189:0/1893819350 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63880125f0 con 0x7f638c0b4d70 2024-04-06T13:49:33.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.791+0000 7f63827fc700 1 -- 172.21.15.189:0/1893819350 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6388012750 con 0x7f638c0b4d70 2024-04-06T13:49:33.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.791+0000 7f63827fc700 1 -- 172.21.15.189:0/1893819350 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6388012b00 con 0x7f638c0b4d70 2024-04-06T13:49:33.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.791+0000 7f6393d45700 1 -- 172.21.15.189:0/1893819350 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f638c15ba20 con 0x7f638c0b4d70 2024-04-06T13:49:33.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.791+0000 7f6393d45700 1 -- 172.21.15.189:0/1893819350 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f638c158130 con 0x7f638c0b4d70 2024-04-06T13:49:33.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.792+0000 7f6393d45700 1 -- 172.21.15.189:0/1893819350 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f638c15b880 con 0x7f638c0b4d70 2024-04-06T13:49:33.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.792+0000 7f63827fc700 1 -- 172.21.15.189:0/1893819350 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 164) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f6388012c60 con 0x7f638c0b4d70 2024-04-06T13:49:33.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.793+0000 7f63827fc700 1 --2- 172.21.15.189:0/1893819350 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f637006ab30 0x7f637006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:33.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.793+0000 7f63912e0700 1 --2- 172.21.15.189:0/1893819350 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f637006ab30 0x7f637006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:33.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.793+0000 7f63827fc700 1 -- 172.21.15.189:0/1893819350 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6531+0+0 (secure 0 0 0) 0x7f6388089690 con 0x7f638c0b4d70 2024-04-06T13:49:33.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.793+0000 7f63912e0700 1 --2- 172.21.15.189:0/1893819350 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f637006ab30 0x7f637006cff0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f637c000f60 tx=0x7f637c007a40 comp rx=0 tx=0).ready entity=mgr.13056 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:33.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:33.797+0000 7f63827fc700 1 -- 172.21.15.189:0/1893819350 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f638c15b880 con 0x7f638c0b4d70 2024-04-06T13:49:34.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.049+0000 7f6393d45700 1 -- 172.21.15.189:0/1893819350 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f638c15bf40 con 0x7f638c0b4d70 2024-04-06T13:49:34.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.230+0000 7f63827fc700 1 -- 172.21.15.189:0/1893819350 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 165) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f638805e030 con 0x7f638c0b4d70 2024-04-06T13:49:34.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.231+0000 7f63827fc700 1 -- 172.21.15.189:0/1893819350 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v165) v1 ==== 93+0+0 (secure 0 0 0) 0x7f6388086f10 con 0x7f638c0b4d70 2024-04-06T13:49:34.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.234+0000 7f6393d45700 1 -- 172.21.15.189:0/1893819350 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f637006ab30 msgr2=0x7f637006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:34.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.234+0000 7f6393d45700 1 --2- 172.21.15.189:0/1893819350 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f637006ab30 0x7f637006cff0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f637c000f60 tx=0x7f637c007a40 comp rx=0 tx=0).stop 2024-04-06T13:49:34.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.234+0000 7f6393d45700 1 -- 172.21.15.189:0/1893819350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f638c0b4d70 msgr2=0x7f638c157bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:34.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.234+0000 7f6393d45700 1 --2- 172.21.15.189:0/1893819350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f638c0b4d70 0x7f638c157bf0 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7f638800d800 tx=0x7f638800dbc0 comp rx=0 tx=0).stop 2024-04-06T13:49:34.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.234+0000 7f6393d45700 1 -- 172.21.15.189:0/1893819350 shutdown_connections 2024-04-06T13:49:34.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.234+0000 7f6393d45700 1 --2- 172.21.15.189:0/1893819350 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f637006ab30 0x7f637006cff0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:34.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.234+0000 7f6393d45700 1 --2- 172.21.15.189:0/1893819350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f638c151600 0x7f638c1436e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:34.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.234+0000 7f6393d45700 1 --2- 172.21.15.189:0/1893819350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f638c0b4d70 0x7f638c157bf0 unknown :-1 s=CLOSED pgs=2986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:34.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.234+0000 7f6393d45700 1 --2- 172.21.15.189:0/1893819350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f638c0cf900 0x7f638c1f2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:34.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.234+0000 7f6393d45700 1 -- 172.21.15.189:0/1893819350 >> 172.21.15.189:0/1893819350 conn(0x7f638c0d00b0 msgr2=0x7f638c06dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:34.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.234+0000 7f6393d45700 1 -- 172.21.15.189:0/1893819350 shutdown_connections 2024-04-06T13:49:34.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.234+0000 7f6393d45700 1 -- 172.21.15.189:0/1893819350 wait complete. 2024-04-06T13:49:34.248 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T13:49:34.249 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:49:34.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:49:34.338 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T13:49:34.338 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:49:34.339 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:49:34.356 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:49:34.356 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:49:34.414 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:49:34.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.579+0000 7f1015624700 1 Processor -- start 2024-04-06T13:49:34.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.579+0000 7f1015624700 1 -- start start 2024-04-06T13:49:34.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.580+0000 7f1015624700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10100cfa00 0x7f10100cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:34.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.580+0000 7f1015624700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10100cbd80 con 0x7f10100d11b0 2024-04-06T13:49:34.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.580+0000 7f1015624700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10100cef20 con 0x7f10100cb870 2024-04-06T13:49:34.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.580+0000 7f1015624700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10100ccdc0 con 0x7f10100cfa00 2024-04-06T13:49:34.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.580+0000 7f100effd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f10100d11b0 0x7f10100d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46244/0 (socket says 172.21.15.189:46244) 2024-04-06T13:49:34.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.580+0000 7f100effd700 1 -- 172.21.15.189:0/685789033 learned_addr learned my addr 172.21.15.189:0/685789033 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:34.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.580+0000 7f100f7fe700 1 --2- 172.21.15.189:0/685789033 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10100cfa00 0x7f10100cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:34.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.580+0000 7f100dffb700 1 -- 172.21.15.189:0/685789033 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401729263 0 0) 0x7f10100cbd80 con 0x7f10100d11b0 2024-04-06T13:49:34.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.580+0000 7f100dffb700 1 -- 172.21.15.189:0/685789033 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff80035b0 con 0x7f10100d11b0 2024-04-06T13:49:34.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.580+0000 7f100dffb700 1 -- 172.21.15.189:0/685789033 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348252654 0 0) 0x7f10100cef20 con 0x7f10100cb870 2024-04-06T13:49:34.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.581+0000 7f100dffb700 1 -- 172.21.15.189:0/685789033 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10100cbd80 con 0x7f10100cb870 2024-04-06T13:49:34.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.581+0000 7f100f7fe700 1 -- 172.21.15.189:0/685789033 >> v1:172.21.15.192:6789/0 conn(0x7f10100cb870 legacy=0x7f101015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:34.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.581+0000 7f100f7fe700 1 -- 172.21.15.189:0/685789033 >> v1:172.21.15.189:6789/0 conn(0x7f10100d11b0 legacy=0x7f10100d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:34.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.581+0000 7f100f7fe700 1 -- 172.21.15.189:0/685789033 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10100cf200 con 0x7f10100cfa00 2024-04-06T13:49:34.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.581+0000 7f100f7fe700 1 --2- 172.21.15.189:0/685789033 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10100cfa00 0x7f10100cd560 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1008009dc0 tx=0x7f100800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20fcff497fc50ce2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:34.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.581+0000 7f100dffb700 1 -- 172.21.15.189:0/685789033 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100800e070 con 0x7f10100cfa00 2024-04-06T13:49:34.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.581+0000 7f100dffb700 1 -- 172.21.15.189:0/685789033 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f100800bb20 con 0x7f10100cfa00 2024-04-06T13:49:34.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.581+0000 7f100dffb700 1 -- 172.21.15.189:0/685789033 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100800be90 con 0x7f10100cfa00 2024-04-06T13:49:34.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.582+0000 7f1015624700 1 -- 172.21.15.189:0/685789033 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10100cfa00 msgr2=0x7f10100cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:34.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.582+0000 7f1015624700 1 --2- 172.21.15.189:0/685789033 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10100cfa00 0x7f10100cd560 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1008009dc0 tx=0x7f100800b330 comp rx=0 tx=0).stop 2024-04-06T13:49:34.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.582+0000 7f1015624700 1 -- 172.21.15.189:0/685789033 shutdown_connections 2024-04-06T13:49:34.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.582+0000 7f1015624700 1 --2- 172.21.15.189:0/685789033 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10100cfa00 0x7f10100cd560 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:34.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.582+0000 7f1015624700 1 -- 172.21.15.189:0/685789033 >> 172.21.15.189:0/685789033 conn(0x7f10100b2db0 msgr2=0x7f10100b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:34.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.582+0000 7f1015624700 1 -- 172.21.15.189:0/685789033 shutdown_connections 2024-04-06T13:49:34.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.582+0000 7f1015624700 1 -- 172.21.15.189:0/685789033 wait complete. 2024-04-06T13:49:34.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.583+0000 7f1015624700 1 Processor -- start 2024-04-06T13:49:34.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.583+0000 7f1015624700 1 -- start start 2024-04-06T13:49:34.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.584+0000 7f1015624700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10100cb870 0x7f1010155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:34.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.584+0000 7f1015624700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10100cfa00 0x7f10100b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:34.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.584+0000 7f1015624700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10100d11b0 0x7f101015c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:34.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.584+0000 7f1015624700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1010155b40 con 0x7f10100cfa00 2024-04-06T13:49:34.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.584+0000 7f1015624700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10101555c0 con 0x7f10100cb870 2024-04-06T13:49:34.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.584+0000 7f1015624700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10100c5b00 con 0x7f10100d11b0 2024-04-06T13:49:34.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.584+0000 7f100effd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10100cb870 0x7f1010155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:34.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.584+0000 7f100effd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10100cb870 0x7f1010155080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:51362/0 (socket says 172.21.15.189:51362) 2024-04-06T13:49:34.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.584+0000 7f100e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10100cfa00 0x7f10100b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:34.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.584+0000 7f100effd700 1 -- 172.21.15.189:0/3563342746 learned_addr learned my addr 172.21.15.189:0/3563342746 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:34.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.584+0000 7f100f7fe700 1 --2- 172.21.15.189:0/3563342746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10100d11b0 0x7f101015c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:34.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.584+0000 7f100effd700 1 -- 172.21.15.189:0/3563342746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10100d11b0 msgr2=0x7f101015c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:34.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.584+0000 7f100effd700 1 --2- 172.21.15.189:0/3563342746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10100d11b0 0x7f101015c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:34.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.584+0000 7f100effd700 1 -- 172.21.15.189:0/3563342746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10100cfa00 msgr2=0x7f10100b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:34.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.584+0000 7f100effd700 1 --2- 172.21.15.189:0/3563342746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10100cfa00 0x7f10100b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:34.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.584+0000 7f100effd700 1 -- 172.21.15.189:0/3563342746 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1008009a70 con 0x7f10100cb870 2024-04-06T13:49:34.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.584+0000 7f100f7fe700 1 --2- 172.21.15.189:0/3563342746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10100d11b0 0x7f101015c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:34.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.585+0000 7f100effd700 1 --2- 172.21.15.189:0/3563342746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10100cb870 0x7f1010155080 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f100000c700 tx=0x7f100000ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:34.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.585+0000 7f0ff77fe700 1 -- 172.21.15.189:0/3563342746 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1000005d70 con 0x7f10100cb870 2024-04-06T13:49:34.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.585+0000 7f0ff77fe700 1 -- 172.21.15.189:0/3563342746 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10000070d0 con 0x7f10100cb870 2024-04-06T13:49:34.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.585+0000 7f1015624700 1 -- 172.21.15.189:0/3563342746 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10100c5de0 con 0x7f10100cb870 2024-04-06T13:49:34.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.585+0000 7f0ff77fe700 1 -- 172.21.15.189:0/3563342746 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1000007440 con 0x7f10100cb870 2024-04-06T13:49:34.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.585+0000 7f1015624700 1 -- 172.21.15.189:0/3563342746 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10100c63a0 con 0x7f10100cb870 2024-04-06T13:49:34.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.586+0000 7f1015624700 1 -- 172.21.15.189:0/3563342746 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1010074990 con 0x7f10100cb870 2024-04-06T13:49:34.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.587+0000 7f0ff77fe700 1 -- 172.21.15.189:0/3563342746 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 165) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f10000075a0 con 0x7f10100cb870 2024-04-06T13:49:34.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.587+0000 7f0ff77fe700 1 --2- 172.21.15.189:0/3563342746 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0ffc06adb0 0x7f0ffc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:34.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.587+0000 7f100e7fc700 1 -- 172.21.15.189:0/3563342746 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0ffc06adb0 msgr2=0x7f0ffc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:34.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.587+0000 7f0ff77fe700 1 -- 172.21.15.189:0/3563342746 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6531+0+0 (secure 0 0 0) 0x7f1000088a40 con 0x7f10100cb870 2024-04-06T13:49:34.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.587+0000 7f100e7fc700 1 --2- 172.21.15.189:0/3563342746 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0ffc06adb0 0x7f0ffc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:49:34.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.592+0000 7f0ff77fe700 1 -- 172.21.15.189:0/3563342746 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f100008ae60 con 0x7f10100cb870 2024-04-06T13:49:34.620 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:34.620+0000 7f1a8caf9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:49:34.633 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:34.632+0000 7fb3b94b6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:49:34.699 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:34.698+0000 7f1a8caf9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:49:34.713 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:34.713+0000 7fb3b94b6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:49:34.771 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:34.771+0000 7f1a8caf9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:49:34.788 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:34.787+0000 7fb3b94b6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:49:34.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.788+0000 7f100e7fc700 1 -- 172.21.15.189:0/3563342746 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0ffc06adb0 msgr2=0x7f0ffc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:34.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.788+0000 7f100e7fc700 1 --2- 172.21.15.189:0/3563342746 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0ffc06adb0 0x7f0ffc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:49:34.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.798+0000 7f1015624700 1 -- 172.21.15.189:0/3563342746 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10100b6ec0 con 0x7f10100cb870 2024-04-06T13:49:34.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.800+0000 7f0ff77fe700 1 -- 172.21.15.189:0/3563342746 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v270) v1 ==== 74+0+17040 (secure 0 0 0) 0x7f1000053890 con 0x7f10100cb870 2024-04-06T13:49:34.801 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:34.801 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":270,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:49:31.207810+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:49:15.919092+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"268","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-04-06T13:49:16.459885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"269","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/2571565303":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/106800846":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/792254642":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/1837391793":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/1485104551":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/3877255335":"2024-04-07T13:49:31.207769+0000","172.21.15.189:6832/111704":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/811188054":"2024-04-07T13:49:31.207769+0000","172.21.15.189:6833/111704":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/2544306464":"2024-04-07T13:49:31.207769+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:49:34.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.802+0000 7f1015624700 1 -- 172.21.15.189:0/3563342746 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0ffc06adb0 msgr2=0x7f0ffc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:49:34.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.802+0000 7f1015624700 1 --2- 172.21.15.189:0/3563342746 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0ffc06adb0 0x7f0ffc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:34.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.802+0000 7f1015624700 1 -- 172.21.15.189:0/3563342746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10100cb870 msgr2=0x7f1010155080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:34.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.802+0000 7f1015624700 1 --2- 172.21.15.189:0/3563342746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10100cb870 0x7f1010155080 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f100000c700 tx=0x7f100000ca10 comp rx=0 tx=0).stop 2024-04-06T13:49:34.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.802+0000 7f1015624700 1 -- 172.21.15.189:0/3563342746 shutdown_connections 2024-04-06T13:49:34.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.802+0000 7f1015624700 1 --2- 172.21.15.189:0/3563342746 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0ffc06adb0 0x7f0ffc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:34.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.802+0000 7f1015624700 1 --2- 172.21.15.189:0/3563342746 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10100d11b0 0x7f101015c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:34.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.802+0000 7f1015624700 1 --2- 172.21.15.189:0/3563342746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10100cb870 0x7f1010155080 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:34.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.802+0000 7f1015624700 1 --2- 172.21.15.189:0/3563342746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10100cfa00 0x7f10100b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:34.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.802+0000 7f1015624700 1 -- 172.21.15.189:0/3563342746 >> 172.21.15.189:0/3563342746 conn(0x7f10100b2db0 msgr2=0x7f10100d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:34.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.802+0000 7f1015624700 1 -- 172.21.15.189:0/3563342746 shutdown_connections 2024-04-06T13:49:34.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.802+0000 7f1015624700 1 -- 172.21.15.189:0/3563342746 wait complete. 2024-04-06T13:49:34.816 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T13:49:15.919092+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '268', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T13:49:16.459885+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '269', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:49:34.816 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:49:34.841 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:34.839+0000 7f1a8caf9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:49:34.861 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:34.860+0000 7fb3b94b6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:49:34.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.945+0000 7f4ffec39700 1 Processor -- start 2024-04-06T13:49:34.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.945+0000 7f4ffec39700 1 -- start start 2024-04-06T13:49:34.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.946+0000 7f4ffec39700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4ff815d7b0 0x7f4ff80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:34.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.946+0000 7f4ffec39700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ff806a730 0x7f4ff806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:34.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.946+0000 7f4ffec39700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff80b3990 con 0x7f4ff815ddf0 2024-04-06T13:49:34.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.946+0000 7f4ffec39700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ff806b0f0 con 0x7f4ff815d7b0 2024-04-06T13:49:34.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.946+0000 7f4ffec39700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff80b3060 con 0x7f4ff806a730 2024-04-06T13:49:34.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.946+0000 7f4ffc9d5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4ff815d7b0 0x7f4ff80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:34.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.946+0000 7f4ffc9d5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4ff815d7b0 0x7f4ff80b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51548/0 (socket says 172.21.15.189:51548) 2024-04-06T13:49:34.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.946+0000 7f4ffc9d5700 1 -- 172.21.15.189:0/17374228 learned_addr learned my addr 172.21.15.189:0/17374228 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:34.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.946+0000 7f4ff7fff700 1 --2- 172.21.15.189:0/17374228 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ff806a730 0x7f4ff806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:34.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.946+0000 7f4ffc9d5700 1 -- 172.21.15.189:0/17374228 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ff806a730 msgr2=0x7f4ff806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:34.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.946+0000 7f4ffc9d5700 1 --2- 172.21.15.189:0/17374228 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ff806a730 0x7f4ff806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:34.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.946+0000 7f4ffc9d5700 1 -- 172.21.15.189:0/17374228 >> v1:172.21.15.192:6789/0 conn(0x7f4ff815ddf0 legacy=0x7f4ff815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:34.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.946+0000 7f4ffc9d5700 1 -- 172.21.15.189:0/17374228 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff80b3340 con 0x7f4ff815d7b0 2024-04-06T13:49:34.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.947+0000 7f4ffc9d5700 1 --2- 172.21.15.189:0/17374228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ff815d7b0 0x7f4ff80b3e20 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0098f0 tx=0x7f4fec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38938d25dbe1fc51 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:34.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.947+0000 7f4ff77fe700 1 -- 172.21.15.189:0/17374228 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec011830 con 0x7f4ff815d7b0 2024-04-06T13:49:34.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.947+0000 7f4ff77fe700 1 -- 172.21.15.189:0/17374228 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec011990 con 0x7f4ff815d7b0 2024-04-06T13:49:34.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.947+0000 7f4ff77fe700 1 -- 172.21.15.189:0/17374228 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec011d20 con 0x7f4ff815d7b0 2024-04-06T13:49:34.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.947+0000 7f4ffec39700 1 -- 172.21.15.189:0/17374228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ff815d7b0 msgr2=0x7f4ff80b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:34.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.947+0000 7f4ffec39700 1 --2- 172.21.15.189:0/17374228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ff815d7b0 0x7f4ff80b3e20 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0098f0 tx=0x7f4fec009c00 comp rx=0 tx=0).stop 2024-04-06T13:49:34.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.947+0000 7f4ffec39700 1 -- 172.21.15.189:0/17374228 shutdown_connections 2024-04-06T13:49:34.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.947+0000 7f4ffec39700 1 --2- 172.21.15.189:0/17374228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ff815d7b0 0x7f4ff80b3e20 unknown :-1 s=CLOSED pgs=2990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:34.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.947+0000 7f4ffec39700 1 --2- 172.21.15.189:0/17374228 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ff806a730 0x7f4ff806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:34.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.947+0000 7f4ffec39700 1 -- 172.21.15.189:0/17374228 >> 172.21.15.189:0/17374228 conn(0x7f4ff80d00b0 msgr2=0x7f4ff80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:34.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.948+0000 7f4ffec39700 1 -- 172.21.15.189:0/17374228 shutdown_connections 2024-04-06T13:49:34.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.948+0000 7f4ffec39700 1 -- 172.21.15.189:0/17374228 wait complete. 2024-04-06T13:49:34.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.949+0000 7f4ffec39700 1 Processor -- start 2024-04-06T13:49:34.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.949+0000 7f4ffec39700 1 -- start start 2024-04-06T13:49:34.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ffec39700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ff806a730 0x7f4ff81f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ffec39700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ff815d7b0 0x7f4ff81f6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ffec39700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ff815ddf0 0x7f4ff81f6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ffec39700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff8148c90 con 0x7f4ff815ddf0 2024-04-06T13:49:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ffec39700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff80b0d70 con 0x7f4ff806a730 2024-04-06T13:49:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ffec39700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff80b1050 con 0x7f4ff815d7b0 2024-04-06T13:49:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ffd1d6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ff815ddf0 0x7f4ff81f6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ffd1d6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ff815ddf0 0x7f4ff81f6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51564/0 (socket says 172.21.15.189:51564) 2024-04-06T13:49:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ffd1d6700 1 -- 172.21.15.189:0/3365870219 learned_addr learned my addr 172.21.15.189:0/3365870219 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ff7fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ff815d7b0 0x7f4ff81f6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ffc9d5700 1 --2- 172.21.15.189:0/3365870219 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ff806a730 0x7f4ff81f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ff7fff700 1 -- 172.21.15.189:0/3365870219 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ff806a730 msgr2=0x7f4ff81f28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ff7fff700 1 --2- 172.21.15.189:0/3365870219 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ff806a730 0x7f4ff81f28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ff7fff700 1 -- 172.21.15.189:0/3365870219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ff815ddf0 msgr2=0x7f4ff81f6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:34.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ff7fff700 1 --2- 172.21.15.189:0/3365870219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ff815ddf0 0x7f4ff81f6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:34.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ff7fff700 1 -- 172.21.15.189:0/3365870219 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec009550 con 0x7f4ff815d7b0 2024-04-06T13:49:34.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ffd1d6700 1 --2- 172.21.15.189:0/3365870219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ff815ddf0 0x7f4ff81f6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:34.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ff7fff700 1 --2- 172.21.15.189:0/3365870219 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ff815d7b0 0x7f4ff81f6450 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4fe400e900 tx=0x7f4fe400ec10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:34.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ff57fa700 1 -- 172.21.15.189:0/3365870219 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4013680 con 0x7f4ff815d7b0 2024-04-06T13:49:34.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ff57fa700 1 -- 172.21.15.189:0/3365870219 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe40137e0 con 0x7f4ff815d7b0 2024-04-06T13:49:34.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ffec39700 1 -- 172.21.15.189:0/3365870219 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff80b1330 con 0x7f4ff815d7b0 2024-04-06T13:49:34.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ff57fa700 1 -- 172.21.15.189:0/3365870219 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4013b50 con 0x7f4ff815d7b0 2024-04-06T13:49:34.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.950+0000 7f4ffec39700 1 -- 172.21.15.189:0/3365870219 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff8148460 con 0x7f4ff815d7b0 2024-04-06T13:49:34.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.951+0000 7f4ffec39700 1 -- 172.21.15.189:0/3365870219 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff80b1980 con 0x7f4ff815d7b0 2024-04-06T13:49:34.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.952+0000 7f4ff57fa700 1 -- 172.21.15.189:0/3365870219 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 165) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f4fe4013cb0 con 0x7f4ff815d7b0 2024-04-06T13:49:34.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.952+0000 7f4ff57fa700 1 --2- 172.21.15.189:0/3365870219 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4fdc06adb0 0x7f4fdc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:34.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.952+0000 7f4ffc9d5700 1 -- 172.21.15.189:0/3365870219 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4fdc06adb0 msgr2=0x7f4fdc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:34.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.952+0000 7f4ff57fa700 1 -- 172.21.15.189:0/3365870219 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6531+0+0 (secure 0 0 0) 0x7f4fe40899c0 con 0x7f4ff815d7b0 2024-04-06T13:49:34.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.952+0000 7f4ffc9d5700 1 --2- 172.21.15.189:0/3365870219 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4fdc06adb0 0x7f4fdc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:49:34.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:34.957+0000 7f4ff57fa700 1 -- 172.21.15.189:0/3365870219 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4fe40547e0 con 0x7f4ff815d7b0 2024-04-06T13:49:35.017 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:35.017+0000 7f1a8caf9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:49:35.045 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:35.044+0000 7fb3b94b6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:49:35.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.153+0000 7f4ffc9d5700 1 -- 172.21.15.189:0/3365870219 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4fdc06adb0 msgr2=0x7f4fdc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:35.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.153+0000 7f4ffc9d5700 1 --2- 172.21.15.189:0/3365870219 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4fdc06adb0 0x7f4fdc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:49:35.159 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.158+0000 7f4ffec39700 1 -- 172.21.15.189:0/3365870219 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4ff8067bb0 con 0x7f4ff815d7b0 2024-04-06T13:49:35.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.159+0000 7f4ff57fa700 1 -- 172.21.15.189:0/3365870219 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v270) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4fe40170a0 con 0x7f4ff815d7b0 2024-04-06T13:49:35.160 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:49:35.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.161+0000 7f4ffec39700 1 -- 172.21.15.189:0/3365870219 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4fdc06adb0 msgr2=0x7f4fdc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:49:35.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.161+0000 7f4ffec39700 1 --2- 172.21.15.189:0/3365870219 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4fdc06adb0 0x7f4fdc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.161+0000 7f4ffec39700 1 -- 172.21.15.189:0/3365870219 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ff815d7b0 msgr2=0x7f4ff81f6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:35.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.161+0000 7f4ffec39700 1 --2- 172.21.15.189:0/3365870219 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ff815d7b0 0x7f4ff81f6450 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4fe400e900 tx=0x7f4fe400ec10 comp rx=0 tx=0).stop 2024-04-06T13:49:35.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.161+0000 7f4ffec39700 1 -- 172.21.15.189:0/3365870219 shutdown_connections 2024-04-06T13:49:35.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.161+0000 7f4ffec39700 1 --2- 172.21.15.189:0/3365870219 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4fdc06adb0 0x7f4fdc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.161+0000 7f4ffec39700 1 --2- 172.21.15.189:0/3365870219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ff815ddf0 0x7f4ff81f6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.161+0000 7f4ffec39700 1 --2- 172.21.15.189:0/3365870219 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ff806a730 0x7f4ff81f28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.161+0000 7f4ffec39700 1 --2- 172.21.15.189:0/3365870219 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ff815d7b0 0x7f4ff81f6450 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.161+0000 7f4ffec39700 1 -- 172.21.15.189:0/3365870219 >> 172.21.15.189:0/3365870219 conn(0x7f4ff80d00b0 msgr2=0x7f4ff80adbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:35.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.162+0000 7f4ffec39700 1 -- 172.21.15.189:0/3365870219 shutdown_connections 2024-04-06T13:49:35.165 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.162+0000 7f4ffec39700 1 -- 172.21.15.189:0/3365870219 wait complete. 2024-04-06T13:49:35.173 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:35.173+0000 7f1a8caf9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:49:35.175 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T13:49:35.208 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:35.207+0000 7fb3b94b6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:49:35.283 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:35.283+0000 7f1a8caf9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:49:35.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.306+0000 7fd74e3e7700 1 Processor -- start 2024-04-06T13:49:35.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.306+0000 7fd74e3e7700 1 -- start start 2024-04-06T13:49:35.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.307+0000 7fd74e3e7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7480cd3e0 0x7fd7480cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:35.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.307+0000 7fd74e3e7700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd74815d7c0 0x7fd74806b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:35.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.307+0000 7fd74e3e7700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd74806ceb0 con 0x7fd74806b6f0 2024-04-06T13:49:35.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.307+0000 7fd74e3e7700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7480cdfa0 con 0x7fd74815d7c0 2024-04-06T13:49:35.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.307+0000 7fd74e3e7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74806d750 con 0x7fd7480cd3e0 2024-04-06T13:49:35.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.307+0000 7fd747fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7480cd3e0 0x7fd7480cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:35.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.307+0000 7fd7477fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd74815d7c0 0x7fd74806b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:35.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.307+0000 7fd747fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7480cd3e0 0x7fd7480cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:34506/0 (socket says 172.21.15.189:34506) 2024-04-06T13:49:35.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.307+0000 7fd74c984700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fd74806b6f0 0x7fd7481610f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:60906/0 (socket says 172.21.15.189:60906) 2024-04-06T13:49:35.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.307+0000 7fd747fff700 1 -- 172.21.15.189:0/277079019 learned_addr learned my addr 172.21.15.189:0/277079019 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:35.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.307+0000 7fd747fff700 1 -- 172.21.15.189:0/277079019 >> v2:172.21.15.192:3300/0 conn(0x7fd74815d7c0 msgr2=0x7fd74806b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:35.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.307+0000 7fd747fff700 1 --2- 172.21.15.189:0/277079019 >> v2:172.21.15.192:3300/0 conn(0x7fd74815d7c0 0x7fd74806b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.307+0000 7fd747fff700 1 -- 172.21.15.189:0/277079019 >> v1:172.21.15.192:6789/0 conn(0x7fd74806b6f0 legacy=0x7fd7481610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:35.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.307+0000 7fd747fff700 1 -- 172.21.15.189:0/277079019 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7480cdd00 con 0x7fd7480cd3e0 2024-04-06T13:49:35.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.307+0000 7fd7477fe700 1 --2- 172.21.15.189:0/277079019 >> v2:172.21.15.192:3300/0 conn(0x7fd74815d7c0 0x7fd74806b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:35.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.307+0000 7fd747fff700 1 --2- 172.21.15.189:0/277079019 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7480cd3e0 0x7fd7480cd7c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd7380098f0 tx=0x7fd738009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9eeb85c6f58bef14 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:35.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.308+0000 7fd746ffd700 1 -- 172.21.15.189:0/277079019 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738011830 con 0x7fd7480cd3e0 2024-04-06T13:49:35.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.308+0000 7fd746ffd700 1 -- 172.21.15.189:0/277079019 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd738011990 con 0x7fd7480cd3e0 2024-04-06T13:49:35.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.308+0000 7fd746ffd700 1 -- 172.21.15.189:0/277079019 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738011d20 con 0x7fd7480cd3e0 2024-04-06T13:49:35.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.308+0000 7fd74e3e7700 1 -- 172.21.15.189:0/277079019 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7480cd3e0 msgr2=0x7fd7480cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:35.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.308+0000 7fd74e3e7700 1 --2- 172.21.15.189:0/277079019 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7480cd3e0 0x7fd7480cd7c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd7380098f0 tx=0x7fd738009c00 comp rx=0 tx=0).stop 2024-04-06T13:49:35.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.308+0000 7fd74e3e7700 1 -- 172.21.15.189:0/277079019 shutdown_connections 2024-04-06T13:49:35.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.308+0000 7fd74e3e7700 1 --2- 172.21.15.189:0/277079019 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7480cd3e0 0x7fd7480cd7c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.308+0000 7fd74e3e7700 1 --2- 172.21.15.189:0/277079019 >> v2:172.21.15.192:3300/0 conn(0x7fd74815d7c0 0x7fd74806b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.308+0000 7fd74e3e7700 1 -- 172.21.15.189:0/277079019 >> 172.21.15.189:0/277079019 conn(0x7fd7480cc290 msgr2=0x7fd7480cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:35.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.308+0000 7fd74e3e7700 1 -- 172.21.15.189:0/277079019 shutdown_connections 2024-04-06T13:49:35.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.308+0000 7fd74e3e7700 1 -- 172.21.15.189:0/277079019 wait complete. 2024-04-06T13:49:35.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.309+0000 7fd74e3e7700 1 Processor -- start 2024-04-06T13:49:35.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.310+0000 7fd74e3e7700 1 -- start start 2024-04-06T13:49:35.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.310+0000 7fd74e3e7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd74806b6f0 0x7fd74815f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:35.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.310+0000 7fd74e3e7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7480cd3e0 0x7fd7481f0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:35.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.310+0000 7fd74e3e7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd74815d7c0 0x7fd7481f05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:35.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.310+0000 7fd74e3e7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7481f6e80 con 0x7fd74815d7c0 2024-04-06T13:49:35.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.310+0000 7fd74e3e7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7481f6570 con 0x7fd7480cd3e0 2024-04-06T13:49:35.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.310+0000 7fd74e3e7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd748155250 con 0x7fd74806b6f0 2024-04-06T13:49:35.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.310+0000 7fd74c984700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd74815d7c0 0x7fd7481f05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:35.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.310+0000 7fd74c984700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd74815d7c0 0x7fd7481f05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51578/0 (socket says 172.21.15.189:51578) 2024-04-06T13:49:35.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.310+0000 7fd747fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd74806b6f0 0x7fd74815f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:35.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.310+0000 7fd74c984700 1 -- 172.21.15.189:0/1817853980 learned_addr learned my addr 172.21.15.189:0/1817853980 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:35.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.310+0000 7fd7477fe700 1 --2- 172.21.15.189:0/1817853980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7480cd3e0 0x7fd7481f0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:35.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.311+0000 7fd747fff700 1 -- 172.21.15.189:0/1817853980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7480cd3e0 msgr2=0x7fd7481f0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:35.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.311+0000 7fd747fff700 1 --2- 172.21.15.189:0/1817853980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7480cd3e0 0x7fd7481f0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.311+0000 7fd747fff700 1 -- 172.21.15.189:0/1817853980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd74815d7c0 msgr2=0x7fd7481f05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:35.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.311+0000 7fd747fff700 1 --2- 172.21.15.189:0/1817853980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd74815d7c0 0x7fd7481f05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.311+0000 7fd747fff700 1 -- 172.21.15.189:0/1817853980 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73c0094a0 con 0x7fd74806b6f0 2024-04-06T13:49:35.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.311+0000 7fd74c984700 1 --2- 172.21.15.189:0/1817853980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd74815d7c0 0x7fd7481f05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:35.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.311+0000 7fd747fff700 1 --2- 172.21.15.189:0/1817853980 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd74806b6f0 0x7fd74815f240 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd738016000 tx=0x7fd738004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:35.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.311+0000 7fd744ff9700 1 -- 172.21.15.189:0/1817853980 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738004590 con 0x7fd74806b6f0 2024-04-06T13:49:35.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.311+0000 7fd744ff9700 1 -- 172.21.15.189:0/1817853980 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73801e070 con 0x7fd74806b6f0 2024-04-06T13:49:35.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.311+0000 7fd744ff9700 1 -- 172.21.15.189:0/1817853980 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7380038a0 con 0x7fd74806b6f0 2024-04-06T13:49:35.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.311+0000 7fd74e3e7700 1 -- 172.21.15.189:0/1817853980 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd738009550 con 0x7fd74806b6f0 2024-04-06T13:49:35.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.311+0000 7fd74e3e7700 1 -- 172.21.15.189:0/1817853980 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7481f6b10 con 0x7fd74806b6f0 2024-04-06T13:49:35.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.312+0000 7fd74e3e7700 1 -- 172.21.15.189:0/1817853980 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd748074990 con 0x7fd74806b6f0 2024-04-06T13:49:35.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.313+0000 7fd744ff9700 1 -- 172.21.15.189:0/1817853980 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 165) v1 ==== 89248+0+0 (secure 0 0 0) 0x7fd738003a00 con 0x7fd74806b6f0 2024-04-06T13:49:35.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.313+0000 7fd744ff9700 1 --2- 172.21.15.189:0/1817853980 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd73406adb0 0x7fd73406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:35.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.313+0000 7fd7477fe700 1 -- 172.21.15.189:0/1817853980 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd73406adb0 msgr2=0x7fd73406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:35.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.313+0000 7fd744ff9700 1 -- 172.21.15.189:0/1817853980 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6531+0+0 (secure 0 0 0) 0x7fd73808a640 con 0x7fd74806b6f0 2024-04-06T13:49:35.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.313+0000 7fd7477fe700 1 --2- 172.21.15.189:0/1817853980 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd73406adb0 0x7fd73406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:49:35.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.318+0000 7fd744ff9700 1 -- 172.21.15.189:0/1817853980 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd738055460 con 0x7fd74806b6f0 2024-04-06T13:49:35.324 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:35.323+0000 7fb3b94b6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:49:35.366 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:35.366+0000 7f1a8caf9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:49:35.409 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:35.408+0000 7fb3b94b6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:49:35.440 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:35.440+0000 7f1a8caf9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:49:35.486 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:35.485+0000 7fb3b94b6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:49:35.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.514+0000 7fd7477fe700 1 -- 172.21.15.189:0/1817853980 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd73406adb0 msgr2=0x7fd73406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:35.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.514+0000 7fd7477fe700 1 --2- 172.21.15.189:0/1817853980 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd73406adb0 0x7fd73406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:49:35.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.521+0000 7fd74e3e7700 1 -- 172.21.15.189:0/1817853980 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd7480b5790 con 0x7fd74806b6f0 2024-04-06T13:49:35.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.521+0000 7fd744ff9700 1 -- 172.21.15.189:0/1817853980 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v270) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd738003cb0 con 0x7fd74806b6f0 2024-04-06T13:49:35.522 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:49:35.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.524+0000 7fd74e3e7700 1 -- 172.21.15.189:0/1817853980 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd73406adb0 msgr2=0x7fd73406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:49:35.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.524+0000 7fd74e3e7700 1 --2- 172.21.15.189:0/1817853980 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd73406adb0 0x7fd73406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.524+0000 7fd74e3e7700 1 -- 172.21.15.189:0/1817853980 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd74806b6f0 msgr2=0x7fd74815f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:35.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.524+0000 7fd74e3e7700 1 --2- 172.21.15.189:0/1817853980 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd74806b6f0 0x7fd74815f240 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd738016000 tx=0x7fd738004750 comp rx=0 tx=0).stop 2024-04-06T13:49:35.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.524+0000 7fd74e3e7700 1 -- 172.21.15.189:0/1817853980 shutdown_connections 2024-04-06T13:49:35.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.524+0000 7fd74e3e7700 1 --2- 172.21.15.189:0/1817853980 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd73406adb0 0x7fd73406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.524+0000 7fd74e3e7700 1 --2- 172.21.15.189:0/1817853980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd74815d7c0 0x7fd7481f05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.524+0000 7fd74e3e7700 1 --2- 172.21.15.189:0/1817853980 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd74806b6f0 0x7fd74815f240 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.524+0000 7fd74e3e7700 1 --2- 172.21.15.189:0/1817853980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7480cd3e0 0x7fd7481f0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.524+0000 7fd74e3e7700 1 -- 172.21.15.189:0/1817853980 >> 172.21.15.189:0/1817853980 conn(0x7fd7480cc290 msgr2=0x7fd7480af7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:35.525 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.524+0000 7fd74e3e7700 1 -- 172.21.15.189:0/1817853980 shutdown_connections 2024-04-06T13:49:35.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.524+0000 7fd74e3e7700 1 -- 172.21.15.189:0/1817853980 wait complete. 2024-04-06T13:49:35.538 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T13:49:35.643 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:35.643+0000 7f1a8caf9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:49:35.700 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:35.699+0000 7fb3b94b6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:49:35.720 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:35.720+0000 7f1a8caf9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:49:35.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.730+0000 7fd353899700 1 Processor -- start 2024-04-06T13:49:35.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.730+0000 7fd353899700 1 -- start start 2024-04-06T13:49:35.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd353899700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd34c15d590 0x7fd34c0b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:35.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd353899700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd34c0b4ed0 0x7fd34c0b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:35.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd353899700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd34c0b6580 0x7fd34c0b69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:35.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd353899700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd34c0b58c0 con 0x7fd34c0b4ed0 2024-04-06T13:49:35.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd353899700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd34c0b5cc0 con 0x7fd34c15d590 2024-04-06T13:49:35.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd353899700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c0d1500 con 0x7fd34c0b6580 2024-04-06T13:49:35.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd350e34700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd34c0b4ed0 0x7fd34c0b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:35.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd350e34700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd34c0b4ed0 0x7fd34c0b5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51584/0 (socket says 172.21.15.189:51584) 2024-04-06T13:49:35.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd350e34700 1 -- 172.21.15.189:0/1025851597 learned_addr learned my addr 172.21.15.189:0/1025851597 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:35.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd351635700 1 --2- 172.21.15.189:0/1025851597 >> v2:172.21.15.192:3300/0 conn(0x7fd34c15d590 0x7fd34c0b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:35.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd351e36700 1 --2- 172.21.15.189:0/1025851597 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd34c0b6580 0x7fd34c0b69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:35.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd350e34700 1 -- 172.21.15.189:0/1025851597 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd34c0b6580 msgr2=0x7fd34c0b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:35.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd350e34700 1 --2- 172.21.15.189:0/1025851597 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd34c0b6580 0x7fd34c0b69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd350e34700 1 -- 172.21.15.189:0/1025851597 >> v2:172.21.15.192:3300/0 conn(0x7fd34c15d590 msgr2=0x7fd34c0b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:35.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd350e34700 1 --2- 172.21.15.189:0/1025851597 >> v2:172.21.15.192:3300/0 conn(0x7fd34c15d590 0x7fd34c0b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd350e34700 1 -- 172.21.15.189:0/1025851597 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c0d17e0 con 0x7fd34c0b4ed0 2024-04-06T13:49:35.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd350e34700 1 --2- 172.21.15.189:0/1025851597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd34c0b4ed0 0x7fd34c0b5380 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 crypto rx=0x7fd34c0b5fe0 tx=0x7fd33c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41d701a08dfce407 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:35.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd343fff700 1 -- 172.21.15.189:0/1025851597 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c011830 con 0x7fd34c0b4ed0 2024-04-06T13:49:35.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd343fff700 1 -- 172.21.15.189:0/1025851597 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33c011990 con 0x7fd34c0b4ed0 2024-04-06T13:49:35.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.731+0000 7fd343fff700 1 -- 172.21.15.189:0/1025851597 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c011d20 con 0x7fd34c0b4ed0 2024-04-06T13:49:35.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.732+0000 7fd353899700 1 -- 172.21.15.189:0/1025851597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd34c0b4ed0 msgr2=0x7fd34c0b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:35.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.732+0000 7fd353899700 1 --2- 172.21.15.189:0/1025851597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd34c0b4ed0 0x7fd34c0b5380 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 crypto rx=0x7fd34c0b5fe0 tx=0x7fd33c009bd0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.732+0000 7fd353899700 1 -- 172.21.15.189:0/1025851597 shutdown_connections 2024-04-06T13:49:35.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.732+0000 7fd353899700 1 --2- 172.21.15.189:0/1025851597 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd34c0b6580 0x7fd34c0b69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.732+0000 7fd353899700 1 --2- 172.21.15.189:0/1025851597 >> v2:172.21.15.192:3300/0 conn(0x7fd34c15d590 0x7fd34c0b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.732+0000 7fd353899700 1 --2- 172.21.15.189:0/1025851597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd34c0b4ed0 0x7fd34c0b5380 unknown :-1 s=CLOSED pgs=2991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.732+0000 7fd353899700 1 -- 172.21.15.189:0/1025851597 >> 172.21.15.189:0/1025851597 conn(0x7fd34c06bbc0 msgr2=0x7fd34c06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:35.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.732+0000 7fd353899700 1 -- 172.21.15.189:0/1025851597 shutdown_connections 2024-04-06T13:49:35.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.732+0000 7fd353899700 1 -- 172.21.15.189:0/1025851597 wait complete. 2024-04-06T13:49:35.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.733+0000 7fd353899700 1 Processor -- start 2024-04-06T13:49:35.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.733+0000 7fd353899700 1 -- start start 2024-04-06T13:49:35.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.734+0000 7fd353899700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd34c0b4ed0 0x7fd34c0c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:35.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.734+0000 7fd353899700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd34c0b6580 0x7fd34c147150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:35.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.734+0000 7fd353899700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd34c15d590 0x7fd34c156430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:35.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.734+0000 7fd353899700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c15a0e0 con 0x7fd34c0b4ed0 2024-04-06T13:49:35.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.734+0000 7fd353899700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c0cb9b0 con 0x7fd34c15d590 2024-04-06T13:49:35.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.734+0000 7fd353899700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c0cbbc0 con 0x7fd34c0b6580 2024-04-06T13:49:35.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.734+0000 7fd351635700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd34c0b4ed0 0x7fd34c0c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:35.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.734+0000 7fd351635700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd34c0b4ed0 0x7fd34c0c8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51594/0 (socket says 172.21.15.189:51594) 2024-04-06T13:49:35.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.734+0000 7fd351635700 1 -- 172.21.15.189:0/1497757692 learned_addr learned my addr 172.21.15.189:0/1497757692 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:35.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.734+0000 7fd350e34700 1 --2- 172.21.15.189:0/1497757692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd34c0b6580 0x7fd34c147150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:35.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.734+0000 7fd351e36700 1 --2- 172.21.15.189:0/1497757692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd34c15d590 0x7fd34c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:35.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.735+0000 7fd350e34700 1 -- 172.21.15.189:0/1497757692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd34c15d590 msgr2=0x7fd34c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:35.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.735+0000 7fd350e34700 1 --2- 172.21.15.189:0/1497757692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd34c15d590 0x7fd34c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.735+0000 7fd350e34700 1 -- 172.21.15.189:0/1497757692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd34c0b4ed0 msgr2=0x7fd34c0c8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:35.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.735+0000 7fd350e34700 1 --2- 172.21.15.189:0/1497757692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd34c0b4ed0 0x7fd34c0c8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.735+0000 7fd350e34700 1 -- 172.21.15.189:0/1497757692 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c009550 con 0x7fd34c0b6580 2024-04-06T13:49:35.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.735+0000 7fd351635700 1 --2- 172.21.15.189:0/1497757692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd34c0b4ed0 0x7fd34c0c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:35.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.735+0000 7fd350e34700 1 --2- 172.21.15.189:0/1497757692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd34c0b6580 0x7fd34c147150 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd33c016000 tx=0x7fd33c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:35.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.735+0000 7fd341ffb700 1 -- 172.21.15.189:0/1497757692 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c004450 con 0x7fd34c0b6580 2024-04-06T13:49:35.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.735+0000 7fd341ffb700 1 -- 172.21.15.189:0/1497757692 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33c01f070 con 0x7fd34c0b6580 2024-04-06T13:49:35.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.735+0000 7fd353899700 1 -- 172.21.15.189:0/1497757692 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34c0cbea0 con 0x7fd34c0b6580 2024-04-06T13:49:35.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.735+0000 7fd341ffb700 1 -- 172.21.15.189:0/1497757692 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c0038e0 con 0x7fd34c0b6580 2024-04-06T13:49:35.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.735+0000 7fd353899700 1 -- 172.21.15.189:0/1497757692 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34c0cc460 con 0x7fd34c0b6580 2024-04-06T13:49:35.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.736+0000 7fd353899700 1 -- 172.21.15.189:0/1497757692 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34c074990 con 0x7fd34c0b6580 2024-04-06T13:49:35.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.737+0000 7fd341ffb700 1 -- 172.21.15.189:0/1497757692 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 165) v1 ==== 89248+0+0 (secure 0 0 0) 0x7fd33c003a40 con 0x7fd34c0b6580 2024-04-06T13:49:35.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.737+0000 7fd341ffb700 1 --2- 172.21.15.189:0/1497757692 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd33006adb0 0x7fd33006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:35.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.737+0000 7fd351635700 1 -- 172.21.15.189:0/1497757692 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd33006adb0 msgr2=0x7fd33006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:35.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.737+0000 7fd341ffb700 1 -- 172.21.15.189:0/1497757692 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6531+0+0 (secure 0 0 0) 0x7fd33c021070 con 0x7fd34c0b6580 2024-04-06T13:49:35.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.737+0000 7fd351635700 1 --2- 172.21.15.189:0/1497757692 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd33006adb0 0x7fd33006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:49:35.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.742+0000 7fd341ffb700 1 -- 172.21.15.189:0/1497757692 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd33c091080 con 0x7fd34c0b6580 2024-04-06T13:49:35.779 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:35.778+0000 7fb3b94b6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:49:35.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.938+0000 7fd351635700 1 -- 172.21.15.189:0/1497757692 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd33006adb0 msgr2=0x7fd33006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:35.939 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.938+0000 7fd351635700 1 --2- 172.21.15.189:0/1497757692 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd33006adb0 0x7fd33006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:49:35.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.942+0000 7fd353899700 1 -- 172.21.15.189:0/1497757692 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd34c158180 con 0x7fd34c0b6580 2024-04-06T13:49:35.943 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.943+0000 7fd341ffb700 1 -- 172.21.15.189:0/1497757692 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v270) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd33c015020 con 0x7fd34c0b6580 2024-04-06T13:49:35.944 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:49:35.946 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.945+0000 7fd353899700 1 -- 172.21.15.189:0/1497757692 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd33006adb0 msgr2=0x7fd33006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:49:35.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.945+0000 7fd353899700 1 --2- 172.21.15.189:0/1497757692 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd33006adb0 0x7fd33006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.945+0000 7fd353899700 1 -- 172.21.15.189:0/1497757692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd34c0b6580 msgr2=0x7fd34c147150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:35.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.945+0000 7fd353899700 1 --2- 172.21.15.189:0/1497757692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd34c0b6580 0x7fd34c147150 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd33c016000 tx=0x7fd33c004750 comp rx=0 tx=0).stop 2024-04-06T13:49:35.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.945+0000 7fd353899700 1 -- 172.21.15.189:0/1497757692 shutdown_connections 2024-04-06T13:49:35.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.945+0000 7fd353899700 1 --2- 172.21.15.189:0/1497757692 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd33006adb0 0x7fd33006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.945+0000 7fd353899700 1 --2- 172.21.15.189:0/1497757692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd34c15d590 0x7fd34c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.945+0000 7fd353899700 1 --2- 172.21.15.189:0/1497757692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd34c0b4ed0 0x7fd34c0c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.945+0000 7fd353899700 1 --2- 172.21.15.189:0/1497757692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd34c0b6580 0x7fd34c147150 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:35.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.945+0000 7fd353899700 1 -- 172.21.15.189:0/1497757692 >> 172.21.15.189:0/1497757692 conn(0x7fd34c06bbc0 msgr2=0x7fd34c0d2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:35.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.945+0000 7fd353899700 1 -- 172.21.15.189:0/1497757692 shutdown_connections 2024-04-06T13:49:35.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:35.945+0000 7fd353899700 1 -- 172.21.15.189:0/1497757692 wait complete. 2024-04-06T13:49:35.961 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T13:49:36.043 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:36.043+0000 7f1a8caf9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:49:36.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.086+0000 7f2b058f4700 1 Processor -- start 2024-04-06T13:49:36.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.086+0000 7f2b058f4700 1 -- start start 2024-04-06T13:49:36.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.086+0000 7f2b058f4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2b0015d7e0 0x7f2b000d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:36.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.086+0000 7f2b058f4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2b000d0890 0x7f2b000d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:36.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.086+0000 7f2b058f4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b000cdf80 con 0x7f2b000d1af0 2024-04-06T13:49:36.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.086+0000 7f2b058f4700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b000d1f80 con 0x7f2b000d0890 2024-04-06T13:49:36.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.086+0000 7f2b058f4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b000d3230 con 0x7f2b0015d7e0 2024-04-06T13:49:36.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.087+0000 7f2afe7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2b000d0890 0x7f2b000d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:36.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.087+0000 7f2afe7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2b000d0890 0x7f2b000d15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51602/0 (socket says 172.21.15.189:51602) 2024-04-06T13:49:36.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.087+0000 7f2afe7fc700 1 -- 172.21.15.189:0/4062531264 learned_addr learned my addr 172.21.15.189:0/4062531264 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:36.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.087+0000 7f2afeffd700 1 --2- 172.21.15.189:0/4062531264 >> v2:172.21.15.192:3300/0 conn(0x7f2b0015d7e0 0x7f2b000d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:36.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.087+0000 7f2afe7fc700 1 -- 172.21.15.189:0/4062531264 >> v2:172.21.15.192:3300/0 conn(0x7f2b0015d7e0 msgr2=0x7f2b000d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:36.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.087+0000 7f2afe7fc700 1 --2- 172.21.15.189:0/4062531264 >> v2:172.21.15.192:3300/0 conn(0x7f2b0015d7e0 0x7f2b000d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.087+0000 7f2afe7fc700 1 -- 172.21.15.189:0/4062531264 >> v1:172.21.15.192:6789/0 conn(0x7f2b000d1af0 legacy=0x7f2b000b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:36.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.087+0000 7f2afe7fc700 1 -- 172.21.15.189:0/4062531264 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b000d0cf0 con 0x7f2b000d0890 2024-04-06T13:49:36.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.087+0000 7f2afeffd700 1 --2- 172.21.15.189:0/4062531264 >> v2:172.21.15.192:3300/0 conn(0x7f2b0015d7e0 0x7f2b000d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:36.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.087+0000 7f2afe7fc700 1 --2- 172.21.15.189:0/4062531264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b000d0890 0x7f2b000d15b0 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 crypto rx=0x7f2b0008c190 tx=0x7f2af4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=875758fcd1cb44db server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:36.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.087+0000 7f2afdffb700 1 -- 172.21.15.189:0/4062531264 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af400b870 con 0x7f2b000d0890 2024-04-06T13:49:36.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.087+0000 7f2afdffb700 1 -- 172.21.15.189:0/4062531264 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af400b9d0 con 0x7f2b000d0890 2024-04-06T13:49:36.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.087+0000 7f2afdffb700 1 -- 172.21.15.189:0/4062531264 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af400bd60 con 0x7f2b000d0890 2024-04-06T13:49:36.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.088+0000 7f2b058f4700 1 -- 172.21.15.189:0/4062531264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b000d0890 msgr2=0x7f2b000d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:36.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.088+0000 7f2b058f4700 1 --2- 172.21.15.189:0/4062531264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b000d0890 0x7f2b000d15b0 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 crypto rx=0x7f2b0008c190 tx=0x7f2af4009bd0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.088+0000 7f2b058f4700 1 -- 172.21.15.189:0/4062531264 shutdown_connections 2024-04-06T13:49:36.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.088+0000 7f2b058f4700 1 --2- 172.21.15.189:0/4062531264 >> v2:172.21.15.192:3300/0 conn(0x7f2b0015d7e0 0x7f2b000d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.088+0000 7f2b058f4700 1 --2- 172.21.15.189:0/4062531264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b000d0890 0x7f2b000d15b0 unknown :-1 s=CLOSED pgs=2992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.088+0000 7f2b058f4700 1 -- 172.21.15.189:0/4062531264 >> 172.21.15.189:0/4062531264 conn(0x7f2b00072250 msgr2=0x7f2b00072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:36.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.088+0000 7f2b058f4700 1 -- 172.21.15.189:0/4062531264 shutdown_connections 2024-04-06T13:49:36.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.088+0000 7f2b058f4700 1 -- 172.21.15.189:0/4062531264 wait complete. 2024-04-06T13:49:36.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.089+0000 7f2b058f4700 1 Processor -- start 2024-04-06T13:49:36.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.089+0000 7f2b058f4700 1 -- start start 2024-04-06T13:49:36.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.090+0000 7f2b058f4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b000d0890 0x7f2b001f37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:36.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.090+0000 7f2b058f4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b000d1af0 0x7f2b000b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:36.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.090+0000 7f2b058f4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2b0015d7e0 0x7f2b001f6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:36.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.090+0000 7f2b058f4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0014c3b0 con 0x7f2b000d0890 2024-04-06T13:49:36.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.090+0000 7f2b058f4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b000b1c60 con 0x7f2b000d1af0 2024-04-06T13:49:36.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.090+0000 7f2b058f4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b000b3e30 con 0x7f2b0015d7e0 2024-04-06T13:49:36.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.090+0000 7f2afeffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b000d0890 0x7f2b001f37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:36.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.090+0000 7f2afeffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b000d0890 0x7f2b001f37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51618/0 (socket says 172.21.15.189:51618) 2024-04-06T13:49:36.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.090+0000 7f2afeffd700 1 -- 172.21.15.189:0/800750139 learned_addr learned my addr 172.21.15.189:0/800750139 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:36.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.090+0000 7f2afe7fc700 1 --2- 172.21.15.189:0/800750139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b000d1af0 0x7f2b000b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:36.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.090+0000 7f2aff7fe700 1 --2- 172.21.15.189:0/800750139 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2b0015d7e0 0x7f2b001f6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:36.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.090+0000 7f2afeffd700 1 -- 172.21.15.189:0/800750139 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2b0015d7e0 msgr2=0x7f2b001f6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:36.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.090+0000 7f2afeffd700 1 --2- 172.21.15.189:0/800750139 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2b0015d7e0 0x7f2b001f6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.090+0000 7f2afeffd700 1 -- 172.21.15.189:0/800750139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b000d1af0 msgr2=0x7f2b000b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.090+0000 7f2afeffd700 1 --2- 172.21.15.189:0/800750139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b000d1af0 0x7f2b000b38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.090+0000 7f2afeffd700 1 -- 172.21.15.189:0/800750139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af4009550 con 0x7f2b000d0890 2024-04-06T13:49:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.091+0000 7f2afeffd700 1 --2- 172.21.15.189:0/800750139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b000d0890 0x7f2b001f37c0 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 crypto rx=0x7f2af000e810 tx=0x7f2af000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.091+0000 7f2ae7fff700 1 -- 172.21.15.189:0/800750139 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af0013610 con 0x7f2b000d0890 2024-04-06T13:49:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.091+0000 7f2ae7fff700 1 -- 172.21.15.189:0/800750139 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af0013770 con 0x7f2b000d0890 2024-04-06T13:49:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.091+0000 7f2b058f4700 1 -- 172.21.15.189:0/800750139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b000b4090 con 0x7f2b000d0890 2024-04-06T13:49:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.091+0000 7f2ae7fff700 1 -- 172.21.15.189:0/800750139 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af0013b20 con 0x7f2b000d0890 2024-04-06T13:49:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.091+0000 7f2b058f4700 1 -- 172.21.15.189:0/800750139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b000b4650 con 0x7f2b000d0890 2024-04-06T13:49:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.092+0000 7f2ae7fff700 1 -- 172.21.15.189:0/800750139 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 165) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f2af0013c80 con 0x7f2b000d0890 2024-04-06T13:49:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.092+0000 7f2b058f4700 1 -- 172.21.15.189:0/800750139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b00074990 con 0x7f2b000d0890 2024-04-06T13:49:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.093+0000 7f2ae7fff700 1 --2- 172.21.15.189:0/800750139 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2aec06adb0 0x7f2aec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.093+0000 7f2afe7fc700 1 -- 172.21.15.189:0/800750139 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2aec06adb0 msgr2=0x7f2aec06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.093+0000 7f2ae7fff700 1 -- 172.21.15.189:0/800750139 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6531+0+0 (secure 0 0 0) 0x7f2af0089480 con 0x7f2b000d0890 2024-04-06T13:49:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.093+0000 7f2afe7fc700 1 --2- 172.21.15.189:0/800750139 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2aec06adb0 0x7f2aec06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:49:36.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.097+0000 7f2ae7fff700 1 -- 172.21.15.189:0/800750139 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2af0054100 con 0x7f2b000d0890 2024-04-06T13:49:36.117 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:36.116+0000 7fb3b94b6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:49:36.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.294+0000 7f2afe7fc700 1 -- 172.21.15.189:0/800750139 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2aec06adb0 msgr2=0x7f2aec06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:36.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.294+0000 7f2afe7fc700 1 --2- 172.21.15.189:0/800750139 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2aec06adb0 0x7f2aec06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:49:36.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.316+0000 7f2b058f4700 1 -- 172.21.15.189:0/800750139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2b000b7310 con 0x7f2b000d0890 2024-04-06T13:49:36.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.317+0000 7f2ae7fff700 1 -- 172.21.15.189:0/800750139 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v270) v1 ==== 81+0+585 (secure 0 0 0) 0x7f2af0057720 con 0x7f2b000d0890 2024-04-06T13:49:36.318 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2571565303 2024-04-07T13:49:31.207769+0000 2024-04-06T13:49:36.318 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/106800846 2024-04-07T13:49:31.207769+0000 2024-04-06T13:49:36.318 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/792254642 2024-04-07T13:49:31.207769+0000 2024-04-06T13:49:36.318 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1837391793 2024-04-07T13:49:31.207769+0000 2024-04-06T13:49:36.318 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/1485104551 2024-04-07T13:49:31.207769+0000 2024-04-06T13:49:36.318 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/3877255335 2024-04-07T13:49:31.207769+0000 2024-04-06T13:49:36.318 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6832/111704 2024-04-07T13:49:31.207769+0000 2024-04-06T13:49:36.318 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/811188054 2024-04-07T13:49:31.207769+0000 2024-04-06T13:49:36.318 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6833/111704 2024-04-07T13:49:31.207769+0000 2024-04-06T13:49:36.319 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2544306464 2024-04-07T13:49:31.207769+0000 2024-04-06T13:49:36.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.319+0000 7f2b058f4700 1 -- 172.21.15.189:0/800750139 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2aec06adb0 msgr2=0x7f2aec06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:49:36.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.319+0000 7f2b058f4700 1 --2- 172.21.15.189:0/800750139 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2aec06adb0 0x7f2aec06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.319+0000 7f2b058f4700 1 -- 172.21.15.189:0/800750139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b000d0890 msgr2=0x7f2b001f37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:36.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.319+0000 7f2b058f4700 1 --2- 172.21.15.189:0/800750139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b000d0890 0x7f2b001f37c0 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 crypto rx=0x7f2af000e810 tx=0x7f2af000eb20 comp rx=0 tx=0).stop 2024-04-06T13:49:36.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.319+0000 7f2b058f4700 1 -- 172.21.15.189:0/800750139 shutdown_connections 2024-04-06T13:49:36.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.319+0000 7f2b058f4700 1 --2- 172.21.15.189:0/800750139 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2aec06adb0 0x7f2aec06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.319+0000 7f2b058f4700 1 --2- 172.21.15.189:0/800750139 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2b0015d7e0 0x7f2b001f6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.319+0000 7f2b058f4700 1 --2- 172.21.15.189:0/800750139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b000d0890 0x7f2b001f37c0 unknown :-1 s=CLOSED pgs=2993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.319+0000 7f2b058f4700 1 --2- 172.21.15.189:0/800750139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b000d1af0 0x7f2b000b38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.319+0000 7f2b058f4700 1 -- 172.21.15.189:0/800750139 >> 172.21.15.189:0/800750139 conn(0x7f2b00072250 msgr2=0x7f2b0015dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:36.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.319+0000 7f2b058f4700 1 -- 172.21.15.189:0/800750139 shutdown_connections 2024-04-06T13:49:36.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.319+0000 7f2b058f4700 1 -- 172.21.15.189:0/800750139 wait complete. 2024-04-06T13:49:36.321 INFO:teuthology.orchestra.run.smithi189.stderr:listed 10 entries 2024-04-06T13:49:36.333 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-06T13:49:36.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:49:36.333 DEBUG:teuthology.orchestra.run.smithi189:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-06T13:49:36.365 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:36.365+0000 7f1a8caf9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:49:36.378 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T13:49:36.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:49:36.378 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-06T13:49:36.395 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:49:36.395 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:49:36.412 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:49:36.412 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:49:36.412 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:49:36.452 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:49:36.460 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:49:36.461 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:49:36.461 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:36.453+0000 7fb3b94b6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:49:36.479 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:49:36.479 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:49:36.532 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:36.532+0000 7f1a8caf9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:49:36.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.604+0000 7f1fd2b1f700 1 Processor -- start 2024-04-06T13:49:36.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.604+0000 7f1fd2b1f700 1 -- start start 2024-04-06T13:49:36.606 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:36.605+0000 7f1a8caf9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:49:36.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.605+0000 7f1fd2b1f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1fcc0d0a60 0x7f1fcc0d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:36.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.605+0000 7f1fd2b1f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1fcc0d1af0 0x7f1fcc0b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:36.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.605+0000 7f1fd2b1f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc0cdf80 con 0x7f1fcc15d7e0 2024-04-06T13:49:36.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.605+0000 7f1fd2b1f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fcc0d3230 con 0x7f1fcc0d0a60 2024-04-06T13:49:36.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.605+0000 7f1fd2b1f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fcc0d2da0 con 0x7f1fcc0d1af0 2024-04-06T13:49:36.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.605+0000 7f1fcbfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1fcc0d0a60 0x7f1fcc0d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:36.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.605+0000 7f1fd08bb700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1fcc15d7e0 0x7f1fcc0d0350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46246/0 (socket says 172.21.15.189:46246) 2024-04-06T13:49:36.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.605+0000 7f1fd08bb700 1 -- 172.21.15.189:0/2528160183 learned_addr learned my addr 172.21.15.189:0/2528160183 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:36.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.605+0000 7f1fd10bc700 1 --2- 172.21.15.189:0/2528160183 >> v2:172.21.15.192:3300/0 conn(0x7f1fcc0d1af0 0x7f1fcc0b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:36.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.605+0000 7f1fcbfff700 1 -- 172.21.15.189:0/2528160183 >> v2:172.21.15.192:3300/0 conn(0x7f1fcc0d1af0 msgr2=0x7f1fcc0b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:36.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.605+0000 7f1fcbfff700 1 --2- 172.21.15.189:0/2528160183 >> v2:172.21.15.192:3300/0 conn(0x7f1fcc0d1af0 0x7f1fcc0b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.605+0000 7f1fcbfff700 1 -- 172.21.15.189:0/2528160183 >> v1:172.21.15.189:6789/0 conn(0x7f1fcc15d7e0 legacy=0x7f1fcc0d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:36.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.605+0000 7f1fcbfff700 1 -- 172.21.15.189:0/2528160183 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc0d1f60 con 0x7f1fcc0d0a60 2024-04-06T13:49:36.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.605+0000 7f1fcb7fe700 1 -- 172.21.15.189:0/2528160183 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1970819666 0 0) 0x7f1fcc0cdf80 con 0x7f1fcc15d7e0 2024-04-06T13:49:36.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.605+0000 7f1fcbfff700 1 --2- 172.21.15.189:0/2528160183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fcc0d0a60 0x7f1fcc0d15b0 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc0b5ed0 tx=0x7f1fb8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf34731b3c137bd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:36.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.605+0000 7f1fcb7fe700 1 -- 172.21.15.189:0/2528160183 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8011830 con 0x7f1fcc0d0a60 2024-04-06T13:49:36.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.605+0000 7f1fcb7fe700 1 -- 172.21.15.189:0/2528160183 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb8011990 con 0x7f1fcc0d0a60 2024-04-06T13:49:36.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.605+0000 7f1fcb7fe700 1 -- 172.21.15.189:0/2528160183 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8011d20 con 0x7f1fcc0d0a60 2024-04-06T13:49:36.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.606+0000 7f1fd2b1f700 1 -- 172.21.15.189:0/2528160183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fcc0d0a60 msgr2=0x7f1fcc0d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:36.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.606+0000 7f1fd2b1f700 1 --2- 172.21.15.189:0/2528160183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fcc0d0a60 0x7f1fcc0d15b0 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc0b5ed0 tx=0x7f1fb8009b20 comp rx=0 tx=0).stop 2024-04-06T13:49:36.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.606+0000 7f1fd2b1f700 1 -- 172.21.15.189:0/2528160183 shutdown_connections 2024-04-06T13:49:36.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.606+0000 7f1fd2b1f700 1 --2- 172.21.15.189:0/2528160183 >> v2:172.21.15.192:3300/0 conn(0x7f1fcc0d1af0 0x7f1fcc0b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.606+0000 7f1fd2b1f700 1 --2- 172.21.15.189:0/2528160183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fcc0d0a60 0x7f1fcc0d15b0 unknown :-1 s=CLOSED pgs=2995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.606+0000 7f1fd2b1f700 1 -- 172.21.15.189:0/2528160183 >> 172.21.15.189:0/2528160183 conn(0x7f1fcc072250 msgr2=0x7f1fcc072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:36.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.606+0000 7f1fd2b1f700 1 -- 172.21.15.189:0/2528160183 shutdown_connections 2024-04-06T13:49:36.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.606+0000 7f1fd2b1f700 1 -- 172.21.15.189:0/2528160183 wait complete. 2024-04-06T13:49:36.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.607+0000 7f1fd2b1f700 1 Processor -- start 2024-04-06T13:49:36.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.607+0000 7f1fd2b1f700 1 -- start start 2024-04-06T13:49:36.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.608+0000 7f1fd2b1f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fcc0d0a60 0x7f1fcc157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:36.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.608+0000 7f1fd2b1f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fcc0d1af0 0x7f1fcc0c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:36.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.608+0000 7f1fd2b1f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fcc15d7e0 0x7f1fcc0ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:36.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.608+0000 7f1fd2b1f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc158680 con 0x7f1fcc0d1af0 2024-04-06T13:49:36.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.608+0000 7f1fd2b1f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc0c6680 con 0x7f1fcc15d7e0 2024-04-06T13:49:36.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.608+0000 7f1fd2b1f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc0cd290 con 0x7f1fcc0d0a60 2024-04-06T13:49:36.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.608+0000 7f1fcbfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fcc0d1af0 0x7f1fcc0c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:36.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.608+0000 7f1fcbfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fcc0d1af0 0x7f1fcc0c6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51636/0 (socket says 172.21.15.189:51636) 2024-04-06T13:49:36.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.608+0000 7f1fcbfff700 1 -- 172.21.15.189:0/3321478625 learned_addr learned my addr 172.21.15.189:0/3321478625 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:36.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.608+0000 7f1fd08bb700 1 --2- 172.21.15.189:0/3321478625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fcc0d0a60 0x7f1fcc157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:36.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.608+0000 7f1fd10bc700 1 --2- 172.21.15.189:0/3321478625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fcc15d7e0 0x7f1fcc0ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:36.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.609+0000 7f1fcbfff700 1 -- 172.21.15.189:0/3321478625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fcc0d0a60 msgr2=0x7f1fcc157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:36.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.609+0000 7f1fcbfff700 1 --2- 172.21.15.189:0/3321478625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fcc0d0a60 0x7f1fcc157490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.609+0000 7f1fcbfff700 1 -- 172.21.15.189:0/3321478625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fcc15d7e0 msgr2=0x7f1fcc0ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:36.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.609+0000 7f1fcbfff700 1 --2- 172.21.15.189:0/3321478625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fcc15d7e0 0x7f1fcc0ccd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.609+0000 7f1fcbfff700 1 -- 172.21.15.189:0/3321478625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb80094a0 con 0x7f1fcc0d1af0 2024-04-06T13:49:36.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.609+0000 7f1fcbfff700 1 --2- 172.21.15.189:0/3321478625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fcc0d1af0 0x7f1fcc0c6140 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8014040 tx=0x7f1fb8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:36.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.609+0000 7f1fc97fa700 1 -- 172.21.15.189:0/3321478625 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb80036a0 con 0x7f1fcc0d1af0 2024-04-06T13:49:36.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.609+0000 7f1fc97fa700 1 -- 172.21.15.189:0/3321478625 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb801e070 con 0x7f1fcc0d1af0 2024-04-06T13:49:36.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.609+0000 7f1fd2b1f700 1 -- 172.21.15.189:0/3321478625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fcc157a70 con 0x7f1fcc0d1af0 2024-04-06T13:49:36.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.609+0000 7f1fc97fa700 1 -- 172.21.15.189:0/3321478625 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8003a20 con 0x7f1fcc0d1af0 2024-04-06T13:49:36.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.609+0000 7f1fd2b1f700 1 -- 172.21.15.189:0/3321478625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fcc158030 con 0x7f1fcc0d1af0 2024-04-06T13:49:36.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.610+0000 7f1fd2b1f700 1 -- 172.21.15.189:0/3321478625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fcc0c6d30 con 0x7f1fcc0d1af0 2024-04-06T13:49:36.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.610+0000 7f1fc97fa700 1 -- 172.21.15.189:0/3321478625 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 165) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f1fb8003b80 con 0x7f1fcc0d1af0 2024-04-06T13:49:36.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.611+0000 7f1fc97fa700 1 --2- 172.21.15.189:0/3321478625 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1fb006adb0 0x7f1fb006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:36.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.611+0000 7f1fd08bb700 1 -- 172.21.15.189:0/3321478625 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1fb006adb0 msgr2=0x7f1fb006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:36.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.611+0000 7f1fc97fa700 1 -- 172.21.15.189:0/3321478625 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6531+0+0 (secure 0 0 0) 0x7f1fb808a580 con 0x7f1fcc0d1af0 2024-04-06T13:49:36.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.611+0000 7f1fd08bb700 1 --2- 172.21.15.189:0/3321478625 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1fb006adb0 0x7f1fb006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:49:36.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.619+0000 7f1fc97fa700 1 -- 172.21.15.189:0/3321478625 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1fcc0c6d30 con 0x7f1fcc0d1af0 2024-04-06T13:49:36.628 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:36.627+0000 7fb3b94b6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:49:36.704 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:36.703+0000 7fb3b94b6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:49:36.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.812+0000 7f1fd08bb700 1 -- 172.21.15.189:0/3321478625 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1fb006adb0 msgr2=0x7f1fb006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:36.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.812+0000 7f1fd08bb700 1 --2- 172.21.15.189:0/3321478625 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1fb006adb0 0x7f1fb006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:49:36.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.870+0000 7f1fd2b1f700 1 -- 172.21.15.189:0/3321478625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fcc0c6d30 con 0x7f1fcc0d1af0 2024-04-06T13:49:36.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.871+0000 7f1fc97fa700 1 -- 172.21.15.189:0/3321478625 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+4757 (secure 0 0 0) 0x7f1fcc0c6d30 con 0x7f1fcc0d1af0 2024-04-06T13:49:36.872 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:36.872 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":131,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3073866011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3073866011},{"type":"v1","addr":"172.21.15.189:6836","nonce":3073866011}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":9917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/4213774859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4213774859},{"type":"v1","addr":"172.21.15.192:6834","nonce":4213774859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":11392,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/963373121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":963373121},{"type":"v1","addr":"172.21.15.189:6837","nonce":963373121}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:49:17.493194+0000","modified":"2024-04-06T13:49:20.503775+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12885},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12885":{"gid":12885,"name":"b","rank":0,"incarnation":128,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/1094572246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":1094572246},{"type":"v1","addr":"172.21.15.192:6835","nonce":1094572246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-04-06T13:49:36.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.873+0000 7f1fd2b1f700 1 -- 172.21.15.189:0/3321478625 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1fb006adb0 msgr2=0x7f1fb006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:49:36.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.873+0000 7f1fd2b1f700 1 --2- 172.21.15.189:0/3321478625 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1fb006adb0 0x7f1fb006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.873+0000 7f1fd2b1f700 1 -- 172.21.15.189:0/3321478625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fcc0d1af0 msgr2=0x7f1fcc0c6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:36.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.873+0000 7f1fd2b1f700 1 --2- 172.21.15.189:0/3321478625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fcc0d1af0 0x7f1fcc0c6140 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8014040 tx=0x7f1fb8004750 comp rx=0 tx=0).stop 2024-04-06T13:49:36.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.873+0000 7f1fd2b1f700 1 -- 172.21.15.189:0/3321478625 shutdown_connections 2024-04-06T13:49:36.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.873+0000 7f1fd2b1f700 1 --2- 172.21.15.189:0/3321478625 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1fb006adb0 0x7f1fb006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.873+0000 7f1fd2b1f700 1 --2- 172.21.15.189:0/3321478625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fcc15d7e0 0x7f1fcc0ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.873+0000 7f1fd2b1f700 1 --2- 172.21.15.189:0/3321478625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1fcc0d0a60 0x7f1fcc157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.873+0000 7f1fd2b1f700 1 --2- 172.21.15.189:0/3321478625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fcc0d1af0 0x7f1fcc0c6140 unknown :-1 s=CLOSED pgs=2996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:36.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.873+0000 7f1fd2b1f700 1 -- 172.21.15.189:0/3321478625 >> 172.21.15.189:0/3321478625 conn(0x7f1fcc072250 msgr2=0x7f1fcc15f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:36.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.873+0000 7f1fd2b1f700 1 -- 172.21.15.189:0/3321478625 shutdown_connections 2024-04-06T13:49:36.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:36.873+0000 7f1fd2b1f700 1 -- 172.21.15.189:0/3321478625 wait complete. 2024-04-06T13:49:36.876 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 131 2024-04-06T13:49:36.887 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:49:37.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.017+0000 7fe598ebc700 1 Processor -- start 2024-04-06T13:49:37.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.017+0000 7fe598ebc700 1 -- start start 2024-04-06T13:49:37.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.018+0000 7fe598ebc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe5940b4660 0x7fe5940b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:37.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.018+0000 7fe598ebc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe5940b2fc0 0x7fe5940b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:37.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.018+0000 7fe598ebc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5940b5030 con 0x7fe59415a550 2024-04-06T13:49:37.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.018+0000 7fe598ebc700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5940cdbe0 con 0x7fe5940b4660 2024-04-06T13:49:37.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.018+0000 7fe598ebc700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe59415a940 con 0x7fe5940b2fc0 2024-04-06T13:49:37.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.018+0000 7fe591d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe5940b4660 0x7fe5940b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:37.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.018+0000 7fe59259c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe59415a550 0x7fe5940cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46252/0 (socket says 172.21.15.189:46252) 2024-04-06T13:49:37.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.018+0000 7fe59259c700 1 -- 172.21.15.189:0/1286665122 learned_addr learned my addr 172.21.15.189:0/1286665122 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:37.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.018+0000 7fe592d9d700 1 --2- 172.21.15.189:0/1286665122 >> v2:172.21.15.192:3300/0 conn(0x7fe5940b2fc0 0x7fe5940b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:37.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.018+0000 7fe591d9b700 1 -- 172.21.15.189:0/1286665122 >> v2:172.21.15.192:3300/0 conn(0x7fe5940b2fc0 msgr2=0x7fe5940b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:37.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.018+0000 7fe591d9b700 1 --2- 172.21.15.189:0/1286665122 >> v2:172.21.15.192:3300/0 conn(0x7fe5940b2fc0 0x7fe5940b33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.018+0000 7fe591d9b700 1 -- 172.21.15.189:0/1286665122 >> v1:172.21.15.189:6789/0 conn(0x7fe59415a550 legacy=0x7fe5940cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:37.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.018+0000 7fe591d9b700 1 -- 172.21.15.189:0/1286665122 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe59415ac00 con 0x7fe5940b4660 2024-04-06T13:49:37.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.018+0000 7fe591d9b700 1 --2- 172.21.15.189:0/1286665122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5940b4660 0x7fe5940b4af0 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7fe594072830 tx=0x7fe58800bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e25f4db635e39a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:37.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.019+0000 7fe59159a700 1 -- 172.21.15.189:0/1286665122 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58800dc70 con 0x7fe5940b4660 2024-04-06T13:49:37.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.019+0000 7fe59159a700 1 -- 172.21.15.189:0/1286665122 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe58800ddd0 con 0x7fe5940b4660 2024-04-06T13:49:37.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.019+0000 7fe59159a700 1 -- 172.21.15.189:0/1286665122 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe588010690 con 0x7fe5940b4660 2024-04-06T13:49:37.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.019+0000 7fe598ebc700 1 -- 172.21.15.189:0/1286665122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5940b4660 msgr2=0x7fe5940b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:37.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.019+0000 7fe598ebc700 1 --2- 172.21.15.189:0/1286665122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5940b4660 0x7fe5940b4af0 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7fe594072830 tx=0x7fe58800bdc0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.019+0000 7fe598ebc700 1 -- 172.21.15.189:0/1286665122 shutdown_connections 2024-04-06T13:49:37.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.019+0000 7fe598ebc700 1 --2- 172.21.15.189:0/1286665122 >> v2:172.21.15.192:3300/0 conn(0x7fe5940b2fc0 0x7fe5940b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.019+0000 7fe598ebc700 1 --2- 172.21.15.189:0/1286665122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5940b4660 0x7fe5940b4af0 unknown :-1 s=CLOSED pgs=2998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.019+0000 7fe598ebc700 1 -- 172.21.15.189:0/1286665122 >> 172.21.15.189:0/1286665122 conn(0x7fe5940d00b0 msgr2=0x7fe5940d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:37.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.020+0000 7fe598ebc700 1 -- 172.21.15.189:0/1286665122 shutdown_connections 2024-04-06T13:49:37.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.020+0000 7fe598ebc700 1 -- 172.21.15.189:0/1286665122 wait complete. 2024-04-06T13:49:37.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.022+0000 7fe598ebc700 1 Processor -- start 2024-04-06T13:49:37.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.022+0000 7fe598ebc700 1 -- start start 2024-04-06T13:49:37.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.023+0000 7fe598ebc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5940b2fc0 0x7fe594157ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:37.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.023+0000 7fe598ebc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5940b4660 0x7fe594149d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:37.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.023+0000 7fe598ebc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe59415c2d0 0x7fe594155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:37.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.023+0000 7fe598ebc700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59414a290 con 0x7fe5940b2fc0 2024-04-06T13:49:37.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.023+0000 7fe598ebc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59415c760 con 0x7fe5940b4660 2024-04-06T13:49:37.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.023+0000 7fe598ebc700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59415ca40 con 0x7fe59415c2d0 2024-04-06T13:49:37.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.023+0000 7fe59259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5940b2fc0 0x7fe594157ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:37.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.023+0000 7fe59259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5940b2fc0 0x7fe594157ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51650/0 (socket says 172.21.15.189:51650) 2024-04-06T13:49:37.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.023+0000 7fe59259c700 1 -- 172.21.15.189:0/1143083655 learned_addr learned my addr 172.21.15.189:0/1143083655 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:37.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.023+0000 7fe592d9d700 1 --2- 172.21.15.189:0/1143083655 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe59415c2d0 0x7fe594155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:37.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.023+0000 7fe591d9b700 1 --2- 172.21.15.189:0/1143083655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5940b4660 0x7fe594149d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:37.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.023+0000 7fe59259c700 1 -- 172.21.15.189:0/1143083655 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe59415c2d0 msgr2=0x7fe594155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:37.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.023+0000 7fe59259c700 1 --2- 172.21.15.189:0/1143083655 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe59415c2d0 0x7fe594155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.023+0000 7fe59259c700 1 -- 172.21.15.189:0/1143083655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5940b4660 msgr2=0x7fe594149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:37.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.023+0000 7fe59259c700 1 --2- 172.21.15.189:0/1143083655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5940b4660 0x7fe594149d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.023+0000 7fe59259c700 1 -- 172.21.15.189:0/1143083655 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58800b740 con 0x7fe5940b2fc0 2024-04-06T13:49:37.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.024+0000 7fe59259c700 1 --2- 172.21.15.189:0/1143083655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5940b2fc0 0x7fe594157ea0 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7fe58400c520 tx=0x7fe58400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:37.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.024+0000 7fe582ffd700 1 -- 172.21.15.189:0/1143083655 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe584005d70 con 0x7fe5940b2fc0 2024-04-06T13:49:37.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.024+0000 7fe582ffd700 1 -- 172.21.15.189:0/1143083655 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5840070d0 con 0x7fe5940b2fc0 2024-04-06T13:49:37.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.024+0000 7fe598ebc700 1 -- 172.21.15.189:0/1143083655 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe59415cd20 con 0x7fe5940b2fc0 2024-04-06T13:49:37.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.024+0000 7fe582ffd700 1 -- 172.21.15.189:0/1143083655 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe584007440 con 0x7fe5940b2fc0 2024-04-06T13:49:37.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.024+0000 7fe598ebc700 1 -- 172.21.15.189:0/1143083655 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe59415d2e0 con 0x7fe5940b2fc0 2024-04-06T13:49:37.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.025+0000 7fe598ebc700 1 -- 172.21.15.189:0/1143083655 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe594067bb0 con 0x7fe5940b2fc0 2024-04-06T13:49:37.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.025+0000 7fe582ffd700 1 -- 172.21.15.189:0/1143083655 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 165) v1 ==== 89248+0+0 (secure 0 0 0) 0x7fe5840075a0 con 0x7fe5940b2fc0 2024-04-06T13:49:37.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.026+0000 7fe582ffd700 1 --2- 172.21.15.189:0/1143083655 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe57406adb0 0x7fe57406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:37.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.026+0000 7fe591d9b700 1 -- 172.21.15.189:0/1143083655 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe57406adb0 msgr2=0x7fe57406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:37.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.026+0000 7fe582ffd700 1 -- 172.21.15.189:0/1143083655 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6531+0+0 (secure 0 0 0) 0x7fe5840889e0 con 0x7fe5940b2fc0 2024-04-06T13:49:37.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.026+0000 7fe591d9b700 1 --2- 172.21.15.189:0/1143083655 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe57406adb0 0x7fe57406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:49:37.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.030+0000 7fe582ffd700 1 -- 172.21.15.189:0/1143083655 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe5840538b0 con 0x7fe5940b2fc0 2024-04-06T13:49:37.225 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:37.225+0000 7f1a8caf9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:49:37.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.227+0000 7fe591d9b700 1 -- 172.21.15.189:0/1143083655 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe57406adb0 msgr2=0x7fe57406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:37.228 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.227+0000 7fe591d9b700 1 --2- 172.21.15.189:0/1143083655 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe57406adb0 0x7fe57406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:49:37.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.235+0000 7fe598ebc700 1 -- 172.21.15.189:0/1143083655 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe594074990 con 0x7fe5940b2fc0 2024-04-06T13:49:37.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.236+0000 7fe582ffd700 1 -- 172.21.15.189:0/1143083655 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v270) v1 ==== 74+0+17040 (secure 0 0 0) 0x7fe584017090 con 0x7fe5940b2fc0 2024-04-06T13:49:37.237 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:37.238 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":270,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:49:31.207810+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:49:15.919092+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"268","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-04-06T13:49:16.459885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"269","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/2571565303":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/106800846":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/792254642":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/1837391793":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/1485104551":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/3877255335":"2024-04-07T13:49:31.207769+0000","172.21.15.189:6832/111704":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/811188054":"2024-04-07T13:49:31.207769+0000","172.21.15.189:6833/111704":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/2544306464":"2024-04-07T13:49:31.207769+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:49:37.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.239+0000 7fe598ebc700 1 -- 172.21.15.189:0/1143083655 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe57406adb0 msgr2=0x7fe57406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:49:37.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.239+0000 7fe598ebc700 1 --2- 172.21.15.189:0/1143083655 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe57406adb0 0x7fe57406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.239+0000 7fe598ebc700 1 -- 172.21.15.189:0/1143083655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5940b2fc0 msgr2=0x7fe594157ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:37.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.239+0000 7fe598ebc700 1 --2- 172.21.15.189:0/1143083655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5940b2fc0 0x7fe594157ea0 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7fe58400c520 tx=0x7fe58400c830 comp rx=0 tx=0).stop 2024-04-06T13:49:37.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.239+0000 7fe598ebc700 1 -- 172.21.15.189:0/1143083655 shutdown_connections 2024-04-06T13:49:37.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.239+0000 7fe598ebc700 1 --2- 172.21.15.189:0/1143083655 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe57406adb0 0x7fe57406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.239+0000 7fe598ebc700 1 --2- 172.21.15.189:0/1143083655 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe59415c2d0 0x7fe594155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.239+0000 7fe598ebc700 1 --2- 172.21.15.189:0/1143083655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5940b2fc0 0x7fe594157ea0 unknown :-1 s=CLOSED pgs=2999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.239+0000 7fe598ebc700 1 --2- 172.21.15.189:0/1143083655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5940b4660 0x7fe594149d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.239+0000 7fe598ebc700 1 -- 172.21.15.189:0/1143083655 >> 172.21.15.189:0/1143083655 conn(0x7fe5940d00b0 msgr2=0x7fe59406d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:37.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.239+0000 7fe598ebc700 1 -- 172.21.15.189:0/1143083655 shutdown_connections 2024-04-06T13:49:37.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.239+0000 7fe598ebc700 1 -- 172.21.15.189:0/1143083655 wait complete. 2024-04-06T13:49:37.252 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T13:49:15.919092+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '268', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T13:49:16.459885+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '269', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:49:37.253 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:49:37.306 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:37.306+0000 7f1a8caf9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:49:37.359 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:37.359+0000 7fb3b94b6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:49:37.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.379+0000 7f6200030700 1 Processor -- start 2024-04-06T13:49:37.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.379+0000 7f6200030700 1 -- start start 2024-04-06T13:49:37.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.380+0000 7f6200030700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f61f815d7b0 0x7f61f80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:37.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.380+0000 7f6200030700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f61f806a730 0x7f61f806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:37.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.380+0000 7f6200030700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f80b3990 con 0x7f61f815ddf0 2024-04-06T13:49:37.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.380+0000 7f6200030700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61f815e950 con 0x7f61f815d7b0 2024-04-06T13:49:37.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.380+0000 7f6200030700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61f806b0f0 con 0x7f61f806a730 2024-04-06T13:49:37.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.380+0000 7f61fe5cd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f61f815ddf0 0x7f61f815e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46262/0 (socket says 172.21.15.189:46262) 2024-04-06T13:49:37.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.380+0000 7f61fddcc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f61f815d7b0 0x7f61f80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:37.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.380+0000 7f61fe5cd700 1 -- 172.21.15.189:0/4103123948 learned_addr learned my addr 172.21.15.189:0/4103123948 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:37.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.380+0000 7f61fd5cb700 1 --2- 172.21.15.189:0/4103123948 >> v2:172.21.15.192:3300/0 conn(0x7f61f806a730 0x7f61f806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:37.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.380+0000 7f61fcdca700 1 -- 172.21.15.189:0/4103123948 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358597679 0 0) 0x7f61f80b3990 con 0x7f61f815ddf0 2024-04-06T13:49:37.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.380+0000 7f61fcdca700 1 -- 172.21.15.189:0/4103123948 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e40035b0 con 0x7f61f815ddf0 2024-04-06T13:49:37.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.380+0000 7f61fddcc700 1 -- 172.21.15.189:0/4103123948 >> v2:172.21.15.192:3300/0 conn(0x7f61f806a730 msgr2=0x7f61f806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:37.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.380+0000 7f61fddcc700 1 --2- 172.21.15.189:0/4103123948 >> v2:172.21.15.192:3300/0 conn(0x7f61f806a730 0x7f61f806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.380+0000 7f61fddcc700 1 -- 172.21.15.189:0/4103123948 >> v1:172.21.15.189:6789/0 conn(0x7f61f815ddf0 legacy=0x7f61f815e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:37.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.380+0000 7f61fddcc700 1 -- 172.21.15.189:0/4103123948 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f80b3060 con 0x7f61f815d7b0 2024-04-06T13:49:37.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.381+0000 7f61fddcc700 1 --2- 172.21.15.189:0/4103123948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f815d7b0 0x7f61f80b3e20 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 crypto rx=0x7f61f80afc00 tx=0x7f61f4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a08e751d4f1e6343 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:37.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.381+0000 7f61fcdca700 1 -- 172.21.15.189:0/4103123948 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f4011830 con 0x7f61f815d7b0 2024-04-06T13:49:37.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.381+0000 7f61fcdca700 1 -- 172.21.15.189:0/4103123948 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f4011990 con 0x7f61f815d7b0 2024-04-06T13:49:37.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.381+0000 7f61fcdca700 1 -- 172.21.15.189:0/4103123948 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f4011d20 con 0x7f61f815d7b0 2024-04-06T13:49:37.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.381+0000 7f6200030700 1 -- 172.21.15.189:0/4103123948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f815d7b0 msgr2=0x7f61f80b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:37.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.381+0000 7f6200030700 1 --2- 172.21.15.189:0/4103123948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f815d7b0 0x7f61f80b3e20 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 crypto rx=0x7f61f80afc00 tx=0x7f61f4009b20 comp rx=0 tx=0).stop 2024-04-06T13:49:37.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.381+0000 7f6200030700 1 -- 172.21.15.189:0/4103123948 shutdown_connections 2024-04-06T13:49:37.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.381+0000 7f6200030700 1 --2- 172.21.15.189:0/4103123948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f815d7b0 0x7f61f80b3e20 unknown :-1 s=CLOSED pgs=3001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.381+0000 7f6200030700 1 --2- 172.21.15.189:0/4103123948 >> v2:172.21.15.192:3300/0 conn(0x7f61f806a730 0x7f61f806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.381+0000 7f6200030700 1 -- 172.21.15.189:0/4103123948 >> 172.21.15.189:0/4103123948 conn(0x7f61f80d00b0 msgr2=0x7f61f80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:37.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.381+0000 7f6200030700 1 -- 172.21.15.189:0/4103123948 shutdown_connections 2024-04-06T13:49:37.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.381+0000 7f6200030700 1 -- 172.21.15.189:0/4103123948 wait complete. 2024-04-06T13:49:37.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.383+0000 7f6200030700 1 Processor -- start 2024-04-06T13:49:37.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.383+0000 7f6200030700 1 -- start start 2024-04-06T13:49:37.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.383+0000 7f6200030700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61f806a730 0x7f61f81550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:37.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.383+0000 7f6200030700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61f815d7b0 0x7f61f80b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:37.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.383+0000 7f6200030700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f815ddf0 0x7f61f80b11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:37.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.383+0000 7f6200030700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f80cd7f0 con 0x7f61f815ddf0 2024-04-06T13:49:37.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.383+0000 7f6200030700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f80cdad0 con 0x7f61f815d7b0 2024-04-06T13:49:37.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.383+0000 7f6200030700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f80cddb0 con 0x7f61f806a730 2024-04-06T13:49:37.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.383+0000 7f61fe5cd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f815ddf0 0x7f61f80b11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:37.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.383+0000 7f61fe5cd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f815ddf0 0x7f61f80b11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51678/0 (socket says 172.21.15.189:51678) 2024-04-06T13:49:37.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.384+0000 7f61fe5cd700 1 -- 172.21.15.189:0/3007801965 learned_addr learned my addr 172.21.15.189:0/3007801965 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:37.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.384+0000 7f61fd5cb700 1 --2- 172.21.15.189:0/3007801965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61f815d7b0 0x7f61f80b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:37.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.384+0000 7f61fddcc700 1 --2- 172.21.15.189:0/3007801965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61f806a730 0x7f61f81550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:37.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.384+0000 7f61fe5cd700 1 -- 172.21.15.189:0/3007801965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61f806a730 msgr2=0x7f61f81550e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:37.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.384+0000 7f61fe5cd700 1 --2- 172.21.15.189:0/3007801965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61f806a730 0x7f61f81550e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.384+0000 7f61fe5cd700 1 -- 172.21.15.189:0/3007801965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61f815d7b0 msgr2=0x7f61f80b0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:37.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.384+0000 7f61fe5cd700 1 --2- 172.21.15.189:0/3007801965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61f815d7b0 0x7f61f80b0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.384+0000 7f61fe5cd700 1 -- 172.21.15.189:0/3007801965 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f40094a0 con 0x7f61f815ddf0 2024-04-06T13:49:37.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.384+0000 7f61fe5cd700 1 --2- 172.21.15.189:0/3007801965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f815ddf0 0x7f61f80b11c0 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 crypto rx=0x7f61f000caa0 tx=0x7f61f000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:37.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.384+0000 7f61ee7fc700 1 -- 172.21.15.189:0/3007801965 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f000a9e0 con 0x7f61f815ddf0 2024-04-06T13:49:37.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.384+0000 7f61ee7fc700 1 -- 172.21.15.189:0/3007801965 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f000ab40 con 0x7f61f815ddf0 2024-04-06T13:49:37.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.384+0000 7f6200030700 1 -- 172.21.15.189:0/3007801965 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f80ce090 con 0x7f61f815ddf0 2024-04-06T13:49:37.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.384+0000 7f61ee7fc700 1 -- 172.21.15.189:0/3007801965 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0012400 con 0x7f61f815ddf0 2024-04-06T13:49:37.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.384+0000 7f6200030700 1 -- 172.21.15.189:0/3007801965 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f80ce5b0 con 0x7f61f815ddf0 2024-04-06T13:49:37.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.385+0000 7f6200030700 1 -- 172.21.15.189:0/3007801965 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f80cd930 con 0x7f61f815ddf0 2024-04-06T13:49:37.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.386+0000 7f61ee7fc700 1 -- 172.21.15.189:0/3007801965 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 165) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f61f001b3b0 con 0x7f61f815ddf0 2024-04-06T13:49:37.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.386+0000 7f61ee7fc700 1 --2- 172.21.15.189:0/3007801965 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f61dc06adb0 0x7f61dc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:37.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.386+0000 7f61fddcc700 1 -- 172.21.15.189:0/3007801965 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f61dc06adb0 msgr2=0x7f61dc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:37.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.386+0000 7f61ee7fc700 1 -- 172.21.15.189:0/3007801965 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6531+0+0 (secure 0 0 0) 0x7f61f0089350 con 0x7f61f815ddf0 2024-04-06T13:49:37.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.386+0000 7f61fddcc700 1 --2- 172.21.15.189:0/3007801965 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f61dc06adb0 0x7f61dc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:49:37.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.390+0000 7f61ee7fc700 1 -- 172.21.15.189:0/3007801965 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f61f80cd930 con 0x7f61f815ddf0 2024-04-06T13:49:37.431 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:37.431+0000 7f1a8caf9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:49:37.442 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:37.442+0000 7fb3b94b6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:49:37.573 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:37.572+0000 7fb3b94b6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:49:37.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.587+0000 7f61fddcc700 1 -- 172.21.15.189:0/3007801965 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f61dc06adb0 msgr2=0x7f61dc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:37.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.587+0000 7f61fddcc700 1 --2- 172.21.15.189:0/3007801965 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f61dc06adb0 0x7f61dc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:49:37.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.596+0000 7f6200030700 1 -- 172.21.15.189:0/3007801965 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f61f80cd930 con 0x7f61f815ddf0 2024-04-06T13:49:37.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.597+0000 7f61ee7fc700 1 -- 172.21.15.189:0/3007801965 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v270) v1 ==== 93+0+10 (secure 0 0 0) 0x7f61f80cd930 con 0x7f61f815ddf0 2024-04-06T13:49:37.598 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:49:37.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.599+0000 7f6200030700 1 -- 172.21.15.189:0/3007801965 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f61dc06adb0 msgr2=0x7f61dc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:49:37.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.599+0000 7f6200030700 1 --2- 172.21.15.189:0/3007801965 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f61dc06adb0 0x7f61dc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.599+0000 7f6200030700 1 -- 172.21.15.189:0/3007801965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f815ddf0 msgr2=0x7f61f80b11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:37.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.599+0000 7f6200030700 1 --2- 172.21.15.189:0/3007801965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f815ddf0 0x7f61f80b11c0 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 crypto rx=0x7f61f000caa0 tx=0x7f61f000cdb0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.599+0000 7f6200030700 1 -- 172.21.15.189:0/3007801965 shutdown_connections 2024-04-06T13:49:37.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.599+0000 7f6200030700 1 --2- 172.21.15.189:0/3007801965 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f61dc06adb0 0x7f61dc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.599+0000 7f6200030700 1 --2- 172.21.15.189:0/3007801965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f815ddf0 0x7f61f80b11c0 unknown :-1 s=CLOSED pgs=3002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.599+0000 7f6200030700 1 --2- 172.21.15.189:0/3007801965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61f806a730 0x7f61f81550e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.599+0000 7f6200030700 1 --2- 172.21.15.189:0/3007801965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61f815d7b0 0x7f61f80b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.599+0000 7f6200030700 1 -- 172.21.15.189:0/3007801965 >> 172.21.15.189:0/3007801965 conn(0x7f61f80d00b0 msgr2=0x7f61f80ae010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:37.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.599+0000 7f6200030700 1 -- 172.21.15.189:0/3007801965 shutdown_connections 2024-04-06T13:49:37.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.599+0000 7f6200030700 1 -- 172.21.15.189:0/3007801965 wait complete. 2024-04-06T13:49:37.613 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T13:49:37.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.746+0000 7f9d0dd13700 1 Processor -- start 2024-04-06T13:49:37.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.746+0000 7f9d0dd13700 1 -- start start 2024-04-06T13:49:37.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.747+0000 7f9d0dd13700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9d080cedf0 0x7f9d080cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:37.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.747+0000 7f9d0dd13700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d080cf730 con 0x7f9d080d17a0 2024-04-06T13:49:37.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.747+0000 7f9d0dd13700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0806d850 con 0x7f9d08146dd0 2024-04-06T13:49:37.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.747+0000 7f9d0dd13700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d0806beb0 con 0x7f9d080cedf0 2024-04-06T13:49:37.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.747+0000 7f9d077fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9d080d17a0 0x7f9d0806b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46278/0 (socket says 172.21.15.189:46278) 2024-04-06T13:49:37.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.747+0000 7f9d077fe700 1 -- 172.21.15.189:0/3701298626 learned_addr learned my addr 172.21.15.189:0/3701298626 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.747+0000 7f9d07fff700 1 --2- 172.21.15.189:0/3701298626 >> v2:172.21.15.192:3300/0 conn(0x7f9d080cedf0 0x7f9d080cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.747+0000 7f9d06ffd700 1 -- 172.21.15.189:0/3701298626 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514308400 0 0) 0x7f9d080cf730 con 0x7f9d080d17a0 2024-04-06T13:49:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.748+0000 7f9d06ffd700 1 -- 172.21.15.189:0/3701298626 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cf40035b0 con 0x7f9d080d17a0 2024-04-06T13:49:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.748+0000 7f9d06ffd700 1 -- 172.21.15.189:0/3701298626 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968962096 0 0) 0x7f9d0806d850 con 0x7f9d08146dd0 2024-04-06T13:49:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.748+0000 7f9d06ffd700 1 -- 172.21.15.189:0/3701298626 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d080cf730 con 0x7f9d08146dd0 2024-04-06T13:49:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.748+0000 7f9d07fff700 1 -- 172.21.15.189:0/3701298626 >> v1:172.21.15.192:6789/0 conn(0x7f9d08146dd0 legacy=0x7f9d0806d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.748+0000 7f9d07fff700 1 -- 172.21.15.189:0/3701298626 >> v1:172.21.15.189:6789/0 conn(0x7f9d080d17a0 legacy=0x7f9d0806b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.748+0000 7f9d07fff700 1 -- 172.21.15.189:0/3701298626 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0806c0a0 con 0x7f9d080cedf0 2024-04-06T13:49:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.748+0000 7f9d07fff700 1 --2- 172.21.15.189:0/3701298626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d080cedf0 0x7f9d080cf1f0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f9d00009d90 tx=0x7f9d0000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df46715a9661cad2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.748+0000 7f9d06ffd700 1 -- 172.21.15.189:0/3701298626 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0000e070 con 0x7f9d080cedf0 2024-04-06T13:49:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.748+0000 7f9d06ffd700 1 -- 172.21.15.189:0/3701298626 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d0000bb20 con 0x7f9d080cedf0 2024-04-06T13:49:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.748+0000 7f9d06ffd700 1 -- 172.21.15.189:0/3701298626 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0000be90 con 0x7f9d080cedf0 2024-04-06T13:49:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.749+0000 7f9d0dd13700 1 -- 172.21.15.189:0/3701298626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d080cedf0 msgr2=0x7f9d080cf1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.749+0000 7f9d0dd13700 1 --2- 172.21.15.189:0/3701298626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d080cedf0 0x7f9d080cf1f0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f9d00009d90 tx=0x7f9d0000b330 comp rx=0 tx=0).stop 2024-04-06T13:49:37.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.749+0000 7f9d0dd13700 1 -- 172.21.15.189:0/3701298626 shutdown_connections 2024-04-06T13:49:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.749+0000 7f9d0dd13700 1 --2- 172.21.15.189:0/3701298626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d080cedf0 0x7f9d080cf1f0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.749+0000 7f9d0dd13700 1 -- 172.21.15.189:0/3701298626 >> 172.21.15.189:0/3701298626 conn(0x7f9d080b2db0 msgr2=0x7f9d080b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.749+0000 7f9d0dd13700 1 -- 172.21.15.189:0/3701298626 shutdown_connections 2024-04-06T13:49:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.749+0000 7f9d0dd13700 1 -- 172.21.15.189:0/3701298626 wait complete. 2024-04-06T13:49:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.750+0000 7f9d0dd13700 1 Processor -- start 2024-04-06T13:49:37.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.750+0000 7f9d0dd13700 1 -- start start 2024-04-06T13:49:37.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.751+0000 7f9d0dd13700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d080cedf0 0x7f9d08155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.751+0000 7f9d0dd13700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d080d17a0 0x7f9d081561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.751+0000 7f9d0dd13700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d08146dd0 0x7f9d080afba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.751+0000 7f9d0dd13700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d08156730 con 0x7f9d08146dd0 2024-04-06T13:49:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.751+0000 7f9d0dd13700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d08158570 con 0x7f9d080d17a0 2024-04-06T13:49:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.751+0000 7f9d0dd13700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d08158850 con 0x7f9d080cedf0 2024-04-06T13:49:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.751+0000 7f9d07fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d08146dd0 0x7f9d080afba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.751+0000 7f9d07fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d08146dd0 0x7f9d080afba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51692/0 (socket says 172.21.15.189:51692) 2024-04-06T13:49:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.751+0000 7f9d07fff700 1 -- 172.21.15.189:0/3231284203 learned_addr learned my addr 172.21.15.189:0/3231284203 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.751+0000 7f9d077fe700 1 --2- 172.21.15.189:0/3231284203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d080cedf0 0x7f9d08155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.751+0000 7f9cfeffd700 1 --2- 172.21.15.189:0/3231284203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d080d17a0 0x7f9d081561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.751+0000 7f9d07fff700 1 -- 172.21.15.189:0/3231284203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d080cedf0 msgr2=0x7f9d08155cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.751+0000 7f9d07fff700 1 --2- 172.21.15.189:0/3231284203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d080cedf0 0x7f9d08155cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.751+0000 7f9d07fff700 1 -- 172.21.15.189:0/3231284203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d080d17a0 msgr2=0x7f9d081561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.751+0000 7f9d07fff700 1 --2- 172.21.15.189:0/3231284203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d080d17a0 0x7f9d081561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.752+0000 7f9d07fff700 1 -- 172.21.15.189:0/3231284203 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d00009a70 con 0x7f9d08146dd0 2024-04-06T13:49:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.752+0000 7f9d077fe700 1 --2- 172.21.15.189:0/3231284203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d080cedf0 0x7f9d08155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.752+0000 7f9d07fff700 1 --2- 172.21.15.189:0/3231284203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d08146dd0 0x7f9d080afba0 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 crypto rx=0x7f9d080677e0 tx=0x7f9d00004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.752+0000 7f9d04ff9700 1 -- 172.21.15.189:0/3231284203 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d000048c0 con 0x7f9d08146dd0 2024-04-06T13:49:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.752+0000 7f9d04ff9700 1 -- 172.21.15.189:0/3231284203 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d00020070 con 0x7f9d08146dd0 2024-04-06T13:49:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.752+0000 7f9d0dd13700 1 -- 172.21.15.189:0/3231284203 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d080aca70 con 0x7f9d08146dd0 2024-04-06T13:49:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.752+0000 7f9d04ff9700 1 -- 172.21.15.189:0/3231284203 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d000276c0 con 0x7f9d08146dd0 2024-04-06T13:49:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.752+0000 7f9d0dd13700 1 -- 172.21.15.189:0/3231284203 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d081f3190 con 0x7f9d08146dd0 2024-04-06T13:49:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.753+0000 7f9d0dd13700 1 -- 172.21.15.189:0/3231284203 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d080ac8d0 con 0x7f9d08146dd0 2024-04-06T13:49:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.754+0000 7f9d04ff9700 1 -- 172.21.15.189:0/3231284203 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 165) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f9d00012430 con 0x7f9d08146dd0 2024-04-06T13:49:37.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.754+0000 7f9d04ff9700 1 --2- 172.21.15.189:0/3231284203 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9ce806adb0 0x7f9ce806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:37.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.754+0000 7f9d077fe700 1 -- 172.21.15.189:0/3231284203 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9ce806adb0 msgr2=0x7f9ce806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:37.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.754+0000 7f9d04ff9700 1 -- 172.21.15.189:0/3231284203 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6531+0+0 (secure 0 0 0) 0x7f9d00022070 con 0x7f9d08146dd0 2024-04-06T13:49:37.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.754+0000 7f9d077fe700 1 --2- 172.21.15.189:0/3231284203 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9ce806adb0 0x7f9ce806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:49:37.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.765+0000 7f9d04ff9700 1 -- 172.21.15.189:0/3231284203 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9d080ac8d0 con 0x7f9d08146dd0 2024-04-06T13:49:37.943 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:37.943+0000 7f1a8caf9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:49:37.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.955+0000 7f9d077fe700 1 -- 172.21.15.189:0/3231284203 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9ce806adb0 msgr2=0x7f9ce806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:37.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.955+0000 7f9d077fe700 1 --2- 172.21.15.189:0/3231284203 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9ce806adb0 0x7f9ce806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:49:37.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.967+0000 7f9d0dd13700 1 -- 172.21.15.189:0/3231284203 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9d080acf90 con 0x7f9d08146dd0 2024-04-06T13:49:37.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.968+0000 7f9d04ff9700 1 -- 172.21.15.189:0/3231284203 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v270) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9d080acf90 con 0x7f9d08146dd0 2024-04-06T13:49:37.969 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:49:37.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.970+0000 7f9d0dd13700 1 -- 172.21.15.189:0/3231284203 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9ce806adb0 msgr2=0x7f9ce806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:49:37.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.970+0000 7f9d0dd13700 1 --2- 172.21.15.189:0/3231284203 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9ce806adb0 0x7f9ce806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.970+0000 7f9d0dd13700 1 -- 172.21.15.189:0/3231284203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d08146dd0 msgr2=0x7f9d080afba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:37.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.970+0000 7f9d0dd13700 1 --2- 172.21.15.189:0/3231284203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d08146dd0 0x7f9d080afba0 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 crypto rx=0x7f9d080677e0 tx=0x7f9d00004ad0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.970+0000 7f9d0dd13700 1 -- 172.21.15.189:0/3231284203 shutdown_connections 2024-04-06T13:49:37.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.970+0000 7f9d0dd13700 1 --2- 172.21.15.189:0/3231284203 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9ce806adb0 0x7f9ce806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.970+0000 7f9d0dd13700 1 --2- 172.21.15.189:0/3231284203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d08146dd0 0x7f9d080afba0 unknown :-1 s=CLOSED pgs=3004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.970+0000 7f9d0dd13700 1 --2- 172.21.15.189:0/3231284203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d080cedf0 0x7f9d08155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.970+0000 7f9d0dd13700 1 --2- 172.21.15.189:0/3231284203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d080d17a0 0x7f9d081561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:37.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.970+0000 7f9d0dd13700 1 -- 172.21.15.189:0/3231284203 >> 172.21.15.189:0/3231284203 conn(0x7f9d080b2db0 msgr2=0x7f9d080b4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:37.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.970+0000 7f9d0dd13700 1 -- 172.21.15.189:0/3231284203 shutdown_connections 2024-04-06T13:49:37.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:37.970+0000 7f9d0dd13700 1 -- 172.21.15.189:0/3231284203 wait complete. 2024-04-06T13:49:37.984 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T13:49:38.098 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:38.098+0000 7f1a8caf9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:49:38.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.111+0000 7f36bc78d700 1 Processor -- start 2024-04-06T13:49:38.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.111+0000 7f36bc78d700 1 -- start start 2024-04-06T13:49:38.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.112+0000 7f36bc78d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f36b406ae40 0x7f36b415ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:38.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.112+0000 7f36bc78d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f36b415fcd0 0x7f36b415bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:38.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.112+0000 7f36bc78d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b415c4c0 con 0x7f36b4144f40 2024-04-06T13:49:38.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.112+0000 7f36bc78d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36b40cfcb0 con 0x7f36b415fcd0 2024-04-06T13:49:38.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.112+0000 7f36bc78d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b415ca20 con 0x7f36b406ae40 2024-04-06T13:49:38.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.112+0000 7f36bad2a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f36b415fcd0 0x7f36b415bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:38.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.112+0000 7f36bad2a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f36b415fcd0 0x7f36b415bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51700/0 (socket says 172.21.15.189:51700) 2024-04-06T13:49:38.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.113+0000 7f36bad2a700 1 -- 172.21.15.189:0/4229300680 learned_addr learned my addr 172.21.15.189:0/4229300680 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:38.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.113+0000 7f36b9d28700 1 --2- 172.21.15.189:0/4229300680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f36b406ae40 0x7f36b415ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:38.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.113+0000 7f36bad2a700 1 -- 172.21.15.189:0/4229300680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f36b406ae40 msgr2=0x7f36b415ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:38.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.113+0000 7f36bad2a700 1 --2- 172.21.15.189:0/4229300680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f36b406ae40 0x7f36b415ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.113+0000 7f36bad2a700 1 -- 172.21.15.189:0/4229300680 >> v1:172.21.15.192:6789/0 conn(0x7f36b4144f40 legacy=0x7f36b406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:38.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.113+0000 7f36bad2a700 1 -- 172.21.15.189:0/4229300680 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b415c740 con 0x7f36b415fcd0 2024-04-06T13:49:38.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.113+0000 7f36bad2a700 1 --2- 172.21.15.189:0/4229300680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36b415fcd0 0x7f36b415bf80 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7f36ac009e50 tx=0x7f36ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=316839a81da2588a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:38.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.113+0000 7f36b9527700 1 -- 172.21.15.189:0/4229300680 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ac00e070 con 0x7f36b415fcd0 2024-04-06T13:49:38.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.113+0000 7f36b9527700 1 -- 172.21.15.189:0/4229300680 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36ac00b910 con 0x7f36b415fcd0 2024-04-06T13:49:38.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.113+0000 7f36b9527700 1 -- 172.21.15.189:0/4229300680 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ac00bcc0 con 0x7f36b415fcd0 2024-04-06T13:49:38.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.114+0000 7f36bc78d700 1 -- 172.21.15.189:0/4229300680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36b415fcd0 msgr2=0x7f36b415bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:38.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.114+0000 7f36bc78d700 1 --2- 172.21.15.189:0/4229300680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36b415fcd0 0x7f36b415bf80 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7f36ac009e50 tx=0x7f36ac00b330 comp rx=0 tx=0).stop 2024-04-06T13:49:38.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.114+0000 7f36bc78d700 1 -- 172.21.15.189:0/4229300680 shutdown_connections 2024-04-06T13:49:38.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.114+0000 7f36bc78d700 1 --2- 172.21.15.189:0/4229300680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36b415fcd0 0x7f36b415bf80 unknown :-1 s=CLOSED pgs=3005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.114+0000 7f36bc78d700 1 --2- 172.21.15.189:0/4229300680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f36b406ae40 0x7f36b415ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.114+0000 7f36bc78d700 1 -- 172.21.15.189:0/4229300680 >> 172.21.15.189:0/4229300680 conn(0x7f36b40d00b0 msgr2=0x7f36b40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:38.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.114+0000 7f36bc78d700 1 -- 172.21.15.189:0/4229300680 shutdown_connections 2024-04-06T13:49:38.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.114+0000 7f36bc78d700 1 -- 172.21.15.189:0/4229300680 wait complete. 2024-04-06T13:49:38.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.115+0000 7f36bc78d700 1 Processor -- start 2024-04-06T13:49:38.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.115+0000 7f36bc78d700 1 -- start start 2024-04-06T13:49:38.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.116+0000 7f36bc78d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36b406ae40 0x7f36b40c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:38.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.116+0000 7f36bc78d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f36b4144f40 0x7f36b40c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:38.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.116+0000 7f36bc78d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36b415fcd0 0x7f36b40c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:38.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.116+0000 7f36bc78d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b40c54b0 con 0x7f36b415fcd0 2024-04-06T13:49:38.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.116+0000 7f36bc78d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b40c5790 con 0x7f36b406ae40 2024-04-06T13:49:38.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.116+0000 7f36bc78d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b40c5a70 con 0x7f36b4144f40 2024-04-06T13:49:38.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.116+0000 7f36bad2a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36b415fcd0 0x7f36b40c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:38.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.116+0000 7f36bad2a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36b415fcd0 0x7f36b40c9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51706/0 (socket says 172.21.15.189:51706) 2024-04-06T13:49:38.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.116+0000 7f36bad2a700 1 -- 172.21.15.189:0/1306892232 learned_addr learned my addr 172.21.15.189:0/1306892232 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:38.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.116+0000 7f36ba529700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36b406ae40 0x7f36b40c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:38.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.116+0000 7f36b9d28700 1 --2- 172.21.15.189:0/1306892232 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f36b4144f40 0x7f36b40c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:38.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.116+0000 7f36bad2a700 1 -- 172.21.15.189:0/1306892232 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f36b4144f40 msgr2=0x7f36b40c8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:38.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.116+0000 7f36bad2a700 1 --2- 172.21.15.189:0/1306892232 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f36b4144f40 0x7f36b40c8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.116+0000 7f36bad2a700 1 -- 172.21.15.189:0/1306892232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36b406ae40 msgr2=0x7f36b40c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:38.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.116+0000 7f36bad2a700 1 --2- 172.21.15.189:0/1306892232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36b406ae40 0x7f36b40c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.116+0000 7f36bad2a700 1 -- 172.21.15.189:0/1306892232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ac009b30 con 0x7f36b415fcd0 2024-04-06T13:49:38.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.116+0000 7f36bad2a700 1 --2- 172.21.15.189:0/1306892232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36b415fcd0 0x7f36b40c9bd0 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7f36ac012000 tx=0x7f36ac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:38.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.117+0000 7f36aaffd700 1 -- 172.21.15.189:0/1306892232 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ac004800 con 0x7f36b415fcd0 2024-04-06T13:49:38.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.117+0000 7f36aaffd700 1 -- 172.21.15.189:0/1306892232 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36ac004960 con 0x7f36b415fcd0 2024-04-06T13:49:38.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.117+0000 7f36aaffd700 1 -- 172.21.15.189:0/1306892232 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ac01b670 con 0x7f36b415fcd0 2024-04-06T13:49:38.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.117+0000 7f36bc78d700 1 -- 172.21.15.189:0/1306892232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b40c5d50 con 0x7f36b415fcd0 2024-04-06T13:49:38.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.117+0000 7f36bc78d700 1 -- 172.21.15.189:0/1306892232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b40c87a0 con 0x7f36b415fcd0 2024-04-06T13:49:38.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.118+0000 7f36bc78d700 1 -- 172.21.15.189:0/1306892232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b4067bb0 con 0x7f36b415fcd0 2024-04-06T13:49:38.122 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:38.118+0000 7fb3b94b6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:49:38.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.118+0000 7f36aaffd700 1 -- 172.21.15.189:0/1306892232 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 165) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f36ac01b7d0 con 0x7f36b415fcd0 2024-04-06T13:49:38.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.119+0000 7f36aaffd700 1 --2- 172.21.15.189:0/1306892232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f369806adb0 0x7f369806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:38.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.119+0000 7f36ba529700 1 -- 172.21.15.189:0/1306892232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f369806adb0 msgr2=0x7f369806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:38.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.119+0000 7f36aaffd700 1 -- 172.21.15.189:0/1306892232 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6531+0+0 (secure 0 0 0) 0x7f36ac08b0c0 con 0x7f36b415fcd0 2024-04-06T13:49:38.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.119+0000 7f36ba529700 1 --2- 172.21.15.189:0/1306892232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f369806adb0 0x7f369806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:49:38.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.123+0000 7f36aaffd700 1 -- 172.21.15.189:0/1306892232 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f36ac055d50 con 0x7f36b415fcd0 2024-04-06T13:49:38.257 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:38.257+0000 7f1a8caf9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:49:38.281 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:38.280+0000 7fb3b94b6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:49:38.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.320+0000 7f36ba529700 1 -- 172.21.15.189:0/1306892232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f369806adb0 msgr2=0x7f369806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:38.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.320+0000 7f36ba529700 1 --2- 172.21.15.189:0/1306892232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f369806adb0 0x7f369806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:49:38.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.327+0000 7f36bc78d700 1 -- 172.21.15.189:0/1306892232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f36b40c6210 con 0x7f36b415fcd0 2024-04-06T13:49:38.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.328+0000 7f36aaffd700 1 -- 172.21.15.189:0/1306892232 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v270) v1 ==== 100+0+11 (secure 0 0 0) 0x7f36ac059370 con 0x7f36b415fcd0 2024-04-06T13:49:38.328 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:49:38.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.330+0000 7f36bc78d700 1 -- 172.21.15.189:0/1306892232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f369806adb0 msgr2=0x7f369806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:49:38.331 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.330+0000 7f36bc78d700 1 --2- 172.21.15.189:0/1306892232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f369806adb0 0x7f369806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.330+0000 7f36bc78d700 1 -- 172.21.15.189:0/1306892232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36b415fcd0 msgr2=0x7f36b40c9bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:38.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.330+0000 7f36bc78d700 1 --2- 172.21.15.189:0/1306892232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36b415fcd0 0x7f36b40c9bd0 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7f36ac012000 tx=0x7f36ac004ad0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.330+0000 7f36bc78d700 1 -- 172.21.15.189:0/1306892232 shutdown_connections 2024-04-06T13:49:38.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.330+0000 7f36bc78d700 1 --2- 172.21.15.189:0/1306892232 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f369806adb0 0x7f369806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.330+0000 7f36bc78d700 1 --2- 172.21.15.189:0/1306892232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36b415fcd0 0x7f36b40c9bd0 unknown :-1 s=CLOSED pgs=3006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.330+0000 7f36bc78d700 1 --2- 172.21.15.189:0/1306892232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36b406ae40 0x7f36b40c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.330+0000 7f36bc78d700 1 --2- 172.21.15.189:0/1306892232 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f36b4144f40 0x7f36b40c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.330+0000 7f36bc78d700 1 -- 172.21.15.189:0/1306892232 >> 172.21.15.189:0/1306892232 conn(0x7f36b40d00b0 msgr2=0x7f36b415bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:38.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.330+0000 7f36bc78d700 1 -- 172.21.15.189:0/1306892232 shutdown_connections 2024-04-06T13:49:38.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.330+0000 7f36bc78d700 1 -- 172.21.15.189:0/1306892232 wait complete. 2024-04-06T13:49:38.344 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:49:38.451 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:38.450+0000 7fb3b94b6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:49:38.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.471+0000 7fe54ffa5700 1 Processor -- start 2024-04-06T13:49:38.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.471+0000 7fe54ffa5700 1 -- start start 2024-04-06T13:49:38.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.472+0000 7fe54ffa5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe54815d590 0x7fe5480b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:38.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.472+0000 7fe54ffa5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5480d3890 0x7fe5480b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:38.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.472+0000 7fe54ffa5700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5480b5cc0 con 0x7fe5480b4ed0 2024-04-06T13:49:38.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.472+0000 7fe54ffa5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5480b58c0 con 0x7fe54815d590 2024-04-06T13:49:38.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.472+0000 7fe54ffa5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5480d1500 con 0x7fe5480d3890 2024-04-06T13:49:38.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.472+0000 7fe54dd41700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe54815d590 0x7fe5480b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:38.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.472+0000 7fe54dd41700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe54815d590 0x7fe5480b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51718/0 (socket says 172.21.15.189:51718) 2024-04-06T13:49:38.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.472+0000 7fe54dd41700 1 -- 172.21.15.189:0/261866128 learned_addr learned my addr 172.21.15.189:0/261866128 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:38.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.472+0000 7fe54e542700 1 --2- 172.21.15.189:0/261866128 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5480d3890 0x7fe5480b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:38.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.472+0000 7fe54dd41700 1 -- 172.21.15.189:0/261866128 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5480d3890 msgr2=0x7fe5480b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:38.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.472+0000 7fe54dd41700 1 --2- 172.21.15.189:0/261866128 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5480d3890 0x7fe5480b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.472+0000 7fe54dd41700 1 -- 172.21.15.189:0/261866128 >> v1:172.21.15.192:6789/0 conn(0x7fe5480b4ed0 legacy=0x7fe5480b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:38.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.472+0000 7fe54dd41700 1 -- 172.21.15.189:0/261866128 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5480d17e0 con 0x7fe54815d590 2024-04-06T13:49:38.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.472+0000 7fe54dd41700 1 --2- 172.21.15.189:0/261866128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe54815d590 0x7fe5480b8d70 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 crypto rx=0x7fe5440098f0 tx=0x7fe544009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2082cc1f5443829b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:38.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.472+0000 7fe54cd3f700 1 -- 172.21.15.189:0/261866128 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe544011830 con 0x7fe54815d590 2024-04-06T13:49:38.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.473+0000 7fe54cd3f700 1 -- 172.21.15.189:0/261866128 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe544011990 con 0x7fe54815d590 2024-04-06T13:49:38.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.473+0000 7fe54cd3f700 1 -- 172.21.15.189:0/261866128 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe544011d20 con 0x7fe54815d590 2024-04-06T13:49:38.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.473+0000 7fe54ffa5700 1 -- 172.21.15.189:0/261866128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe54815d590 msgr2=0x7fe5480b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:38.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.473+0000 7fe54ffa5700 1 --2- 172.21.15.189:0/261866128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe54815d590 0x7fe5480b8d70 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 crypto rx=0x7fe5440098f0 tx=0x7fe544009c00 comp rx=0 tx=0).stop 2024-04-06T13:49:38.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.473+0000 7fe54ffa5700 1 -- 172.21.15.189:0/261866128 shutdown_connections 2024-04-06T13:49:38.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.473+0000 7fe54ffa5700 1 --2- 172.21.15.189:0/261866128 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5480d3890 0x7fe5480b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.473+0000 7fe54ffa5700 1 --2- 172.21.15.189:0/261866128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe54815d590 0x7fe5480b8d70 unknown :-1 s=CLOSED pgs=3007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.473+0000 7fe54ffa5700 1 -- 172.21.15.189:0/261866128 >> 172.21.15.189:0/261866128 conn(0x7fe54806bbc0 msgr2=0x7fe54806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:38.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.473+0000 7fe54ffa5700 1 -- 172.21.15.189:0/261866128 shutdown_connections 2024-04-06T13:49:38.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.473+0000 7fe54ffa5700 1 -- 172.21.15.189:0/261866128 wait complete. 2024-04-06T13:49:38.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.474+0000 7fe54ffa5700 1 Processor -- start 2024-04-06T13:49:38.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.475+0000 7fe54ffa5700 1 -- start start 2024-04-06T13:49:38.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.475+0000 7fe54ffa5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5480b4ed0 0x7fe5480c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:38.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.475+0000 7fe54ffa5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5480d3890 0x7fe5481588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:38.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.475+0000 7fe54ffa5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe54815d590 0x7fe548158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:38.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.475+0000 7fe54ffa5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5480cbff0 con 0x7fe5480b4ed0 2024-04-06T13:49:38.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.475+0000 7fe54ffa5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54815f620 con 0x7fe5480d3890 2024-04-06T13:49:38.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.475+0000 7fe54ffa5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe54815e580 con 0x7fe54815d590 2024-04-06T13:49:38.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.475+0000 7fe54dd41700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5480b4ed0 0x7fe5480c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:38.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.475+0000 7fe54dd41700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5480b4ed0 0x7fe5480c8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51726/0 (socket says 172.21.15.189:51726) 2024-04-06T13:49:38.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.475+0000 7fe54dd41700 1 -- 172.21.15.189:0/995042119 learned_addr learned my addr 172.21.15.189:0/995042119 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:38.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.475+0000 7fe54e542700 1 --2- 172.21.15.189:0/995042119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe54815d590 0x7fe548158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:38.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.475+0000 7fe54d540700 1 --2- 172.21.15.189:0/995042119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5480d3890 0x7fe5481588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:38.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.476+0000 7fe54dd41700 1 -- 172.21.15.189:0/995042119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe54815d590 msgr2=0x7fe548158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:38.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.476+0000 7fe54dd41700 1 --2- 172.21.15.189:0/995042119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe54815d590 0x7fe548158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.476+0000 7fe54dd41700 1 -- 172.21.15.189:0/995042119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5480d3890 msgr2=0x7fe5481588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:38.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.476+0000 7fe54dd41700 1 --2- 172.21.15.189:0/995042119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5480d3890 0x7fe5481588e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.476+0000 7fe54dd41700 1 -- 172.21.15.189:0/995042119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe544009550 con 0x7fe5480b4ed0 2024-04-06T13:49:38.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.476+0000 7fe54dd41700 1 --2- 172.21.15.189:0/995042119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5480b4ed0 0x7fe5480c8f40 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7fe544009c40 tx=0x7fe544004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:38.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.476+0000 7fe53e7fc700 1 -- 172.21.15.189:0/995042119 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe544004510 con 0x7fe5480b4ed0 2024-04-06T13:49:38.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.476+0000 7fe53e7fc700 1 -- 172.21.15.189:0/995042119 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54401e070 con 0x7fe5480b4ed0 2024-04-06T13:49:38.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.476+0000 7fe54ffa5700 1 -- 172.21.15.189:0/995042119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54815e860 con 0x7fe5480b4ed0 2024-04-06T13:49:38.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.476+0000 7fe53e7fc700 1 -- 172.21.15.189:0/995042119 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5440038e0 con 0x7fe5480b4ed0 2024-04-06T13:49:38.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.476+0000 7fe54ffa5700 1 -- 172.21.15.189:0/995042119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54815ee20 con 0x7fe5480b4ed0 2024-04-06T13:49:38.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.477+0000 7fe54ffa5700 1 -- 172.21.15.189:0/995042119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe548074990 con 0x7fe5480b4ed0 2024-04-06T13:49:38.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.478+0000 7fe53e7fc700 1 -- 172.21.15.189:0/995042119 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 165) v1 ==== 89248+0+0 (secure 0 0 0) 0x7fe544003a40 con 0x7fe5480b4ed0 2024-04-06T13:49:38.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.478+0000 7fe53e7fc700 1 --2- 172.21.15.189:0/995042119 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe52c06ade0 0x7fe52c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:38.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.478+0000 7fe54d540700 1 -- 172.21.15.189:0/995042119 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe52c06ade0 msgr2=0x7fe52c06d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:38.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.478+0000 7fe53e7fc700 1 -- 172.21.15.189:0/995042119 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6531+0+0 (secure 0 0 0) 0x7fe54408a5f0 con 0x7fe5480b4ed0 2024-04-06T13:49:38.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.478+0000 7fe54d540700 1 --2- 172.21.15.189:0/995042119 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe52c06ade0 0x7fe52c06d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:49:38.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.486+0000 7fe53e7fc700 1 -- 172.21.15.189:0/995042119 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe544055390 con 0x7fe5480b4ed0 2024-04-06T13:49:38.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.679+0000 7fe54d540700 1 -- 172.21.15.189:0/995042119 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe52c06ade0 msgr2=0x7fe52c06d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:38.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.679+0000 7fe54d540700 1 --2- 172.21.15.189:0/995042119 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe52c06ade0 0x7fe52c06d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:49:38.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.738+0000 7fe54ffa5700 1 -- 172.21.15.189:0/995042119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5480b56a0 con 0x7fe5480b4ed0 2024-04-06T13:49:38.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.738+0000 7fe53e7fc700 1 -- 172.21.15.189:0/995042119 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+4757 (secure 0 0 0) 0x7fe5440589b0 con 0x7fe5480b4ed0 2024-04-06T13:49:38.739 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:38.739 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":131,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3073866011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3073866011},{"type":"v1","addr":"172.21.15.189:6836","nonce":3073866011}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":9917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/4213774859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4213774859},{"type":"v1","addr":"172.21.15.192:6834","nonce":4213774859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":11392,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/963373121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":963373121},{"type":"v1","addr":"172.21.15.189:6837","nonce":963373121}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:49:17.493194+0000","modified":"2024-04-06T13:49:20.503775+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12885},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12885":{"gid":12885,"name":"b","rank":0,"incarnation":128,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/1094572246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":1094572246},{"type":"v1","addr":"172.21.15.192:6835","nonce":1094572246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-04-06T13:49:38.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.741+0000 7fe54ffa5700 1 -- 172.21.15.189:0/995042119 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe52c06ade0 msgr2=0x7fe52c06d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:49:38.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.741+0000 7fe54ffa5700 1 --2- 172.21.15.189:0/995042119 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe52c06ade0 0x7fe52c06d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.741+0000 7fe54ffa5700 1 -- 172.21.15.189:0/995042119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5480b4ed0 msgr2=0x7fe5480c8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:38.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.741+0000 7fe54ffa5700 1 --2- 172.21.15.189:0/995042119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5480b4ed0 0x7fe5480c8f40 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7fe544009c40 tx=0x7fe544004750 comp rx=0 tx=0).stop 2024-04-06T13:49:38.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.741+0000 7fe54ffa5700 1 -- 172.21.15.189:0/995042119 shutdown_connections 2024-04-06T13:49:38.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.741+0000 7fe54ffa5700 1 --2- 172.21.15.189:0/995042119 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe52c06ade0 0x7fe52c06d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.741+0000 7fe54ffa5700 1 --2- 172.21.15.189:0/995042119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe54815d590 0x7fe548158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.741+0000 7fe54ffa5700 1 --2- 172.21.15.189:0/995042119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5480b4ed0 0x7fe5480c8f40 unknown :-1 s=CLOSED pgs=3008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.741+0000 7fe54ffa5700 1 --2- 172.21.15.189:0/995042119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5480d3890 0x7fe5481588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.741+0000 7fe54ffa5700 1 -- 172.21.15.189:0/995042119 >> 172.21.15.189:0/995042119 conn(0x7fe54806bbc0 msgr2=0x7fe5480cd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:38.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.741+0000 7fe54ffa5700 1 -- 172.21.15.189:0/995042119 shutdown_connections 2024-04-06T13:49:38.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.741+0000 7fe54ffa5700 1 -- 172.21.15.189:0/995042119 wait complete. 2024-04-06T13:49:38.743 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 131 2024-04-06T13:49:38.756 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:49:38.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.880+0000 7f620bc5e700 1 Processor -- start 2024-04-06T13:49:38.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.880+0000 7f620bc5e700 1 -- start start 2024-04-06T13:49:38.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.881+0000 7f620bc5e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62040d0a60 0x7f62040d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:38.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.881+0000 7f620bc5e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62040cdf80 con 0x7f620415d7e0 2024-04-06T13:49:38.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.881+0000 7f620bc5e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62040d3230 con 0x7f62040d1af0 2024-04-06T13:49:38.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.881+0000 7f620bc5e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62040d1f80 con 0x7f62040d0a60 2024-04-06T13:49:38.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.881+0000 7f62099fa700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f620415d7e0 0x7f62040d0350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46282/0 (socket says 172.21.15.189:46282) 2024-04-06T13:49:38.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.881+0000 7f62099fa700 1 -- 172.21.15.189:0/2380145116 learned_addr learned my addr 172.21.15.189:0/2380145116 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:38.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.881+0000 7f62091f9700 1 --2- 172.21.15.189:0/2380145116 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62040d0a60 0x7f62040d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:38.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.881+0000 7f62089f8700 1 -- 172.21.15.189:0/2380145116 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582208922 0 0) 0x7f62040cdf80 con 0x7f620415d7e0 2024-04-06T13:49:38.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.881+0000 7f62089f8700 1 -- 172.21.15.189:0/2380145116 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f00035b0 con 0x7f620415d7e0 2024-04-06T13:49:38.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.881+0000 7f62089f8700 1 -- 172.21.15.189:0/2380145116 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927196272 0 0) 0x7f61fc001490 con 0x7f62040d1af0 2024-04-06T13:49:38.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.881+0000 7f62089f8700 1 -- 172.21.15.189:0/2380145116 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62040cdf80 con 0x7f62040d1af0 2024-04-06T13:49:38.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.881+0000 7f62091f9700 1 -- 172.21.15.189:0/2380145116 >> v1:172.21.15.192:6789/0 conn(0x7f62040d1af0 legacy=0x7f62040b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:38.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.881+0000 7f62091f9700 1 -- 172.21.15.189:0/2380145116 >> v1:172.21.15.189:6789/0 conn(0x7f620415d7e0 legacy=0x7f62040d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:38.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.881+0000 7f62091f9700 1 -- 172.21.15.189:0/2380145116 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62040d2da0 con 0x7f62040d0a60 2024-04-06T13:49:38.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.882+0000 7f62091f9700 1 --2- 172.21.15.189:0/2380145116 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62040d0a60 0x7f62040d15b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f61f4009870 tx=0x7f61f4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2015392c2297756 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:38.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.882+0000 7f62089f8700 1 -- 172.21.15.189:0/2380145116 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f4011830 con 0x7f62040d0a60 2024-04-06T13:49:38.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.882+0000 7f62089f8700 1 -- 172.21.15.189:0/2380145116 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f4011990 con 0x7f62040d0a60 2024-04-06T13:49:38.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.882+0000 7f62089f8700 1 -- 172.21.15.189:0/2380145116 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f4011d20 con 0x7f62040d0a60 2024-04-06T13:49:38.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.882+0000 7f620bc5e700 1 -- 172.21.15.189:0/2380145116 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62040d0a60 msgr2=0x7f62040d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:38.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.882+0000 7f620bc5e700 1 --2- 172.21.15.189:0/2380145116 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62040d0a60 0x7f62040d15b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f61f4009870 tx=0x7f61f4009b80 comp rx=0 tx=0).stop 2024-04-06T13:49:38.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.882+0000 7f620bc5e700 1 -- 172.21.15.189:0/2380145116 shutdown_connections 2024-04-06T13:49:38.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.882+0000 7f620bc5e700 1 --2- 172.21.15.189:0/2380145116 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62040d0a60 0x7f62040d15b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.882+0000 7f620bc5e700 1 -- 172.21.15.189:0/2380145116 >> 172.21.15.189:0/2380145116 conn(0x7f6204072250 msgr2=0x7f6204072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:38.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.883+0000 7f620bc5e700 1 -- 172.21.15.189:0/2380145116 shutdown_connections 2024-04-06T13:49:38.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.883+0000 7f620bc5e700 1 -- 172.21.15.189:0/2380145116 wait complete. 2024-04-06T13:49:38.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.884+0000 7f620bc5e700 1 Processor -- start 2024-04-06T13:49:38.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.884+0000 7f620bc5e700 1 -- start start 2024-04-06T13:49:38.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.885+0000 7f620bc5e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62040d0a60 0x7f62040b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:38.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.885+0000 7f620bc5e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62040d1af0 0x7f62041541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:38.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.885+0000 7f620bc5e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f620415d7e0 0x7f62040cc920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:38.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.885+0000 7f620bc5e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62040b3c50 con 0x7f62040d1af0 2024-04-06T13:49:38.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.885+0000 7f620bc5e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62040b3e80 con 0x7f62040d0a60 2024-04-06T13:49:38.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.885+0000 7f620bc5e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62040b4160 con 0x7f620415d7e0 2024-04-06T13:49:38.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.885+0000 7f62091f9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62040d1af0 0x7f62041541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:38.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.885+0000 7f62091f9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62040d1af0 0x7f62041541e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51734/0 (socket says 172.21.15.189:51734) 2024-04-06T13:49:38.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.885+0000 7f62091f9700 1 -- 172.21.15.189:0/1958580048 learned_addr learned my addr 172.21.15.189:0/1958580048 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:38.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.885+0000 7f62099fa700 1 --2- 172.21.15.189:0/1958580048 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62040d0a60 0x7f62040b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:38.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.885+0000 7f620a1fb700 1 --2- 172.21.15.189:0/1958580048 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f620415d7e0 0x7f62040cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:38.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.885+0000 7f62091f9700 1 -- 172.21.15.189:0/1958580048 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f620415d7e0 msgr2=0x7f62040cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:38.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.885+0000 7f62091f9700 1 --2- 172.21.15.189:0/1958580048 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f620415d7e0 0x7f62040cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.885+0000 7f62091f9700 1 -- 172.21.15.189:0/1958580048 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62040d0a60 msgr2=0x7f62040b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:38.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.885+0000 7f62091f9700 1 --2- 172.21.15.189:0/1958580048 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62040d0a60 0x7f62040b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:38.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.885+0000 7f62091f9700 1 -- 172.21.15.189:0/1958580048 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f40094a0 con 0x7f62040d1af0 2024-04-06T13:49:38.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.885+0000 7f620a1fb700 1 --2- 172.21.15.189:0/1958580048 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f620415d7e0 0x7f62040cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:38.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.886+0000 7f62091f9700 1 --2- 172.21.15.189:0/1958580048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62040d1af0 0x7f62041541e0 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7f61f40090b0 tx=0x7f61f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:38.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.886+0000 7f61fa7fc700 1 -- 172.21.15.189:0/1958580048 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f401a760 con 0x7f62040d1af0 2024-04-06T13:49:38.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.886+0000 7f61fa7fc700 1 -- 172.21.15.189:0/1958580048 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f401a8c0 con 0x7f62040d1af0 2024-04-06T13:49:38.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.886+0000 7f620bc5e700 1 -- 172.21.15.189:0/1958580048 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62040b4440 con 0x7f62040d1af0 2024-04-06T13:49:38.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.886+0000 7f61fa7fc700 1 -- 172.21.15.189:0/1958580048 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f401ac30 con 0x7f62040d1af0 2024-04-06T13:49:38.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.886+0000 7f620bc5e700 1 -- 172.21.15.189:0/1958580048 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62040b4a00 con 0x7f62040d1af0 2024-04-06T13:49:38.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.887+0000 7f620bc5e700 1 -- 172.21.15.189:0/1958580048 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6204074990 con 0x7f62040d1af0 2024-04-06T13:49:38.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.887+0000 7f61fa7fc700 1 -- 172.21.15.189:0/1958580048 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 165) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f61f4022070 con 0x7f62040d1af0 2024-04-06T13:49:38.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.887+0000 7f61fa7fc700 1 --2- 172.21.15.189:0/1958580048 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f61e806adb0 0x7f61e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:38.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.888+0000 7f62099fa700 1 -- 172.21.15.189:0/1958580048 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f61e806adb0 msgr2=0x7f61e806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:38.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.888+0000 7f61fa7fc700 1 -- 172.21.15.189:0/1958580048 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6531+0+0 (secure 0 0 0) 0x7f61f408a820 con 0x7f62040d1af0 2024-04-06T13:49:38.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.888+0000 7f62099fa700 1 --2- 172.21.15.189:0/1958580048 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f61e806adb0 0x7f61e806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:49:38.892 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:38.892+0000 7f61fa7fc700 1 -- 172.21.15.189:0/1958580048 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f61f4055640 con 0x7f62040d1af0 2024-04-06T13:49:38.981 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:38.980+0000 7f1a8caf9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:49:39.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.089+0000 7f62099fa700 1 -- 172.21.15.189:0/1958580048 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f61e806adb0 msgr2=0x7f61e806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:39.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.089+0000 7f62099fa700 1 --2- 172.21.15.189:0/1958580048 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f61e806adb0 0x7f61e806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:49:39.095 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.094+0000 7f620bc5e700 1 -- 172.21.15.189:0/1958580048 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62040ce9a0 con 0x7f62040d1af0 2024-04-06T13:49:39.096 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.095+0000 7f61fa7fc700 1 -- 172.21.15.189:0/1958580048 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v270) v1 ==== 74+0+17040 (secure 0 0 0) 0x7f61f4029090 con 0x7f62040d1af0 2024-04-06T13:49:39.097 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:39.097 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":270,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:49:31.207810+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:49:15.919092+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"268","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-04-06T13:49:16.459885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"269","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/2571565303":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/106800846":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/792254642":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/1837391793":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/1485104551":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/3877255335":"2024-04-07T13:49:31.207769+0000","172.21.15.189:6832/111704":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/811188054":"2024-04-07T13:49:31.207769+0000","172.21.15.189:6833/111704":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/2544306464":"2024-04-07T13:49:31.207769+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:49:39.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.098+0000 7f620bc5e700 1 -- 172.21.15.189:0/1958580048 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f61e806adb0 msgr2=0x7f61e806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:49:39.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.098+0000 7f620bc5e700 1 --2- 172.21.15.189:0/1958580048 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f61e806adb0 0x7f61e806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.098+0000 7f620bc5e700 1 -- 172.21.15.189:0/1958580048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62040d1af0 msgr2=0x7f62041541e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:39.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.098+0000 7f620bc5e700 1 --2- 172.21.15.189:0/1958580048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62040d1af0 0x7f62041541e0 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7f61f40090b0 tx=0x7f61f4004750 comp rx=0 tx=0).stop 2024-04-06T13:49:39.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.098+0000 7f620bc5e700 1 -- 172.21.15.189:0/1958580048 shutdown_connections 2024-04-06T13:49:39.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.098+0000 7f620bc5e700 1 --2- 172.21.15.189:0/1958580048 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f61e806adb0 0x7f61e806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.098+0000 7f620bc5e700 1 --2- 172.21.15.189:0/1958580048 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f620415d7e0 0x7f62040cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.098+0000 7f620bc5e700 1 --2- 172.21.15.189:0/1958580048 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62040d0a60 0x7f62040b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.099 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.098+0000 7f620bc5e700 1 --2- 172.21.15.189:0/1958580048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62040d1af0 0x7f62041541e0 unknown :-1 s=CLOSED pgs=3010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.098+0000 7f620bc5e700 1 -- 172.21.15.189:0/1958580048 >> 172.21.15.189:0/1958580048 conn(0x7f6204072250 msgr2=0x7f620415ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:39.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.098+0000 7f620bc5e700 1 -- 172.21.15.189:0/1958580048 shutdown_connections 2024-04-06T13:49:39.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.098+0000 7f620bc5e700 1 -- 172.21.15.189:0/1958580048 wait complete. 2024-04-06T13:49:39.112 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-06T13:49:39.112 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:49:39.223 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:39.223+0000 7f1a8caf9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:49:39.225 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:39.224+0000 7fb3b94b6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:49:39.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.236+0000 7f527a52c700 1 Processor -- start 2024-04-06T13:49:39.237 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.236+0000 7f527a52c700 1 -- start start 2024-04-06T13:49:39.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.237+0000 7f527a52c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f52740cd3e0 0x7f52740cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:39.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.237+0000 7f527a52c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f527415d7c0 0x7f527406b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:39.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.237+0000 7f527a52c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527406ceb0 con 0x7f527406b6f0 2024-04-06T13:49:39.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.237+0000 7f527a52c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52740cdfa0 con 0x7f52740cd3e0 2024-04-06T13:49:39.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.237+0000 7f527a52c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f527406d750 con 0x7f527415d7c0 2024-04-06T13:49:39.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.237+0000 7f5273fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f52740cd3e0 0x7f52740cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:39.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.237+0000 7f5273fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f52740cd3e0 0x7f52740cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51744/0 (socket says 172.21.15.189:51744) 2024-04-06T13:49:39.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.237+0000 7f5273fff700 1 -- 172.21.15.189:0/2484988711 learned_addr learned my addr 172.21.15.189:0/2484988711 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:39.240 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.237+0000 7f52737fe700 1 --2- 172.21.15.189:0/2484988711 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f527415d7c0 0x7f527406b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:39.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.238+0000 7f5273fff700 1 -- 172.21.15.189:0/2484988711 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f527415d7c0 msgr2=0x7f527406b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:39.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.238+0000 7f5273fff700 1 --2- 172.21.15.189:0/2484988711 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f527415d7c0 0x7f527406b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.238+0000 7f5273fff700 1 -- 172.21.15.189:0/2484988711 >> v1:172.21.15.192:6789/0 conn(0x7f527406b6f0 legacy=0x7f52741610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:39.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.238+0000 7f5273fff700 1 -- 172.21.15.189:0/2484988711 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52740cdd00 con 0x7f52740cd3e0 2024-04-06T13:49:39.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.238+0000 7f5272ffd700 1 -- 172.21.15.189:0/2484988711 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909777259 0 0) 0x7f527406ceb0 con 0x7f527406b6f0 2024-04-06T13:49:39.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.238+0000 7f5273fff700 1 --2- 172.21.15.189:0/2484988711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52740cd3e0 0x7f52740cd7c0 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 crypto rx=0x7f52680098f0 tx=0x7f5268009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185d31d0bb6eb40f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:39.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.238+0000 7f5272ffd700 1 -- 172.21.15.189:0/2484988711 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5268011830 con 0x7f52740cd3e0 2024-04-06T13:49:39.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.238+0000 7f5272ffd700 1 -- 172.21.15.189:0/2484988711 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5268011990 con 0x7f52740cd3e0 2024-04-06T13:49:39.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.238+0000 7f5272ffd700 1 -- 172.21.15.189:0/2484988711 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5268011d20 con 0x7f52740cd3e0 2024-04-06T13:49:39.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.239+0000 7f527a52c700 1 -- 172.21.15.189:0/2484988711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52740cd3e0 msgr2=0x7f52740cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:39.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.239+0000 7f527a52c700 1 --2- 172.21.15.189:0/2484988711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52740cd3e0 0x7f52740cd7c0 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 crypto rx=0x7f52680098f0 tx=0x7f5268009c00 comp rx=0 tx=0).stop 2024-04-06T13:49:39.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.239+0000 7f527a52c700 1 -- 172.21.15.189:0/2484988711 shutdown_connections 2024-04-06T13:49:39.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.239+0000 7f527a52c700 1 --2- 172.21.15.189:0/2484988711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52740cd3e0 0x7f52740cd7c0 unknown :-1 s=CLOSED pgs=3011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.239+0000 7f527a52c700 1 --2- 172.21.15.189:0/2484988711 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f527415d7c0 0x7f527406b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.239+0000 7f527a52c700 1 -- 172.21.15.189:0/2484988711 >> 172.21.15.189:0/2484988711 conn(0x7f52740cc290 msgr2=0x7f52740cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:39.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.239+0000 7f527a52c700 1 -- 172.21.15.189:0/2484988711 shutdown_connections 2024-04-06T13:49:39.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.239+0000 7f527a52c700 1 -- 172.21.15.189:0/2484988711 wait complete. 2024-04-06T13:49:39.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.240+0000 7f527a52c700 1 Processor -- start 2024-04-06T13:49:39.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.240+0000 7f527a52c700 1 -- start start 2024-04-06T13:49:39.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f527a52c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f527406b6f0 0x7f527415cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:39.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f527a52c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52740cd3e0 0x7f52740c5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:39.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f527a52c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f527415d7c0 0x7f527415a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:39.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f527a52c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52741f5100 con 0x7f527415d7c0 2024-04-06T13:49:39.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f527a52c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52741f53e0 con 0x7f52740cd3e0 2024-04-06T13:49:39.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f527a52c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52741f56c0 con 0x7f527406b6f0 2024-04-06T13:49:39.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f5278ac9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f527415d7c0 0x7f527415a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:39.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f5273fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f527406b6f0 0x7f527415cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:39.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f52737fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52740cd3e0 0x7f52740c5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:39.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f5273fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f527406b6f0 0x7f527415cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:34630/0 (socket says 172.21.15.189:34630) 2024-04-06T13:49:39.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f52737fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52740cd3e0 0x7f52740c5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:51520/0 (socket says 172.21.15.189:51520) 2024-04-06T13:49:39.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f5278ac9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f527415d7c0 0x7f527415a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51760/0 (socket says 172.21.15.189:51760) 2024-04-06T13:49:39.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f5273fff700 1 -- 172.21.15.189:0/2774753914 learned_addr learned my addr 172.21.15.189:0/2774753914 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:39.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f52737fe700 1 -- 172.21.15.189:0/2774753914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f527406b6f0 msgr2=0x7f527415cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:39.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f52737fe700 1 --2- 172.21.15.189:0/2774753914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f527406b6f0 0x7f527415cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f52737fe700 1 -- 172.21.15.189:0/2774753914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f527415d7c0 msgr2=0x7f527415a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:39.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f52737fe700 1 --2- 172.21.15.189:0/2774753914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f527415d7c0 0x7f527415a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f52737fe700 1 -- 172.21.15.189:0/2774753914 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5268009550 con 0x7f52740cd3e0 2024-04-06T13:49:39.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f5273fff700 1 --2- 172.21.15.189:0/2774753914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f527406b6f0 0x7f527415cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:39.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.241+0000 7f52737fe700 1 --2- 172.21.15.189:0/2774753914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52740cd3e0 0x7f52740c5390 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f526400e8d0 tx=0x7f526400ebe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:39.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.242+0000 7f5270ff9700 1 -- 172.21.15.189:0/2774753914 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5264013680 con 0x7f52740cd3e0 2024-04-06T13:49:39.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.242+0000 7f5270ff9700 1 -- 172.21.15.189:0/2774753914 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52640137e0 con 0x7f52740cd3e0 2024-04-06T13:49:39.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.242+0000 7f527a52c700 1 -- 172.21.15.189:0/2774753914 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52741f59a0 con 0x7f52740cd3e0 2024-04-06T13:49:39.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.242+0000 7f5270ff9700 1 -- 172.21.15.189:0/2774753914 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5264013b50 con 0x7f52740cd3e0 2024-04-06T13:49:39.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.242+0000 7f527a52c700 1 -- 172.21.15.189:0/2774753914 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52740c58d0 con 0x7f52740cd3e0 2024-04-06T13:49:39.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.243+0000 7f527a52c700 1 -- 172.21.15.189:0/2774753914 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5274072d60 con 0x7f52740cd3e0 2024-04-06T13:49:39.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.243+0000 7f5270ff9700 1 -- 172.21.15.189:0/2774753914 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 165) v1 ==== 89248+0+0 (secure 0 0 0) 0x7f5264013cb0 con 0x7f52740cd3e0 2024-04-06T13:49:39.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.243+0000 7f5270ff9700 1 --2- 172.21.15.189:0/2774753914 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f526006adb0 0x7f526006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:39.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.243+0000 7f5273fff700 1 -- 172.21.15.189:0/2774753914 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f526006adb0 msgr2=0x7f526006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:39.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.243+0000 7f5270ff9700 1 -- 172.21.15.189:0/2774753914 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6531+0+0 (secure 0 0 0) 0x7f52640896c0 con 0x7f52740cd3e0 2024-04-06T13:49:39.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.243+0000 7f5273fff700 1 --2- 172.21.15.189:0/2774753914 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f526006adb0 0x7f526006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:49:39.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.248+0000 7f5270ff9700 1 -- 172.21.15.189:0/2774753914 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5264054450 con 0x7f52740cd3e0 2024-04-06T13:49:39.301 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:39.301+0000 7f1a8caf9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:49:39.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.445+0000 7f5273fff700 1 -- 172.21.15.189:0/2774753914 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f526006adb0 msgr2=0x7f526006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:39.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.445+0000 7f5273fff700 1 --2- 172.21.15.189:0/2774753914 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f526006adb0 0x7f526006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:49:39.481 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:39.481+0000 7fb3b94b6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:49:39.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.497+0000 7f527a52c700 1 -- 172.21.15.189:0/2774753914 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5274074990 con 0x7f52740cd3e0 2024-04-06T13:49:39.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.498+0000 7f5270ff9700 1 -- 172.21.15.189:0/2774753914 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+4757 (secure 0 0 0) 0x7f526405ce60 con 0x7f52740cd3e0 2024-04-06T13:49:39.499 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:39.499 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":131,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3073866011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3073866011},{"type":"v1","addr":"172.21.15.189:6836","nonce":3073866011}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":9917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/4213774859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4213774859},{"type":"v1","addr":"172.21.15.192:6834","nonce":4213774859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":11392,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/963373121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":963373121},{"type":"v1","addr":"172.21.15.189:6837","nonce":963373121}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:49:17.493194+0000","modified":"2024-04-06T13:49:20.503775+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12885},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12885":{"gid":12885,"name":"b","rank":0,"incarnation":128,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/1094572246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":1094572246},{"type":"v1","addr":"172.21.15.192:6835","nonce":1094572246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-04-06T13:49:39.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.500+0000 7f527a52c700 1 -- 172.21.15.189:0/2774753914 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f526006adb0 msgr2=0x7f526006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:49:39.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.500+0000 7f527a52c700 1 --2- 172.21.15.189:0/2774753914 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f526006adb0 0x7f526006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.500+0000 7f527a52c700 1 -- 172.21.15.189:0/2774753914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52740cd3e0 msgr2=0x7f52740c5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:39.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.500+0000 7f527a52c700 1 --2- 172.21.15.189:0/2774753914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52740cd3e0 0x7f52740c5390 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f526400e8d0 tx=0x7f526400ebe0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.500+0000 7f527a52c700 1 -- 172.21.15.189:0/2774753914 shutdown_connections 2024-04-06T13:49:39.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.500+0000 7f527a52c700 1 --2- 172.21.15.189:0/2774753914 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f526006adb0 0x7f526006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.500+0000 7f527a52c700 1 --2- 172.21.15.189:0/2774753914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f527415d7c0 0x7f527415a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.500+0000 7f527a52c700 1 --2- 172.21.15.189:0/2774753914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f527406b6f0 0x7f527415cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.500+0000 7f527a52c700 1 --2- 172.21.15.189:0/2774753914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52740cd3e0 0x7f52740c5390 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.500+0000 7f527a52c700 1 -- 172.21.15.189:0/2774753914 >> 172.21.15.189:0/2774753914 conn(0x7f52740cc290 msgr2=0x7f52740b0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:39.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.500+0000 7f527a52c700 1 -- 172.21.15.189:0/2774753914 shutdown_connections 2024-04-06T13:49:39.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.500+0000 7f527a52c700 1 -- 172.21.15.189:0/2774753914 wait complete. 2024-04-06T13:49:39.502 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 131 2024-04-06T13:49:39.514 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:49:39.567 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:39.566+0000 7fb3b94b6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:49:39.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.639+0000 7fe6701ca700 1 Processor -- start 2024-04-06T13:49:39.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.639+0000 7fe6701ca700 1 -- start start 2024-04-06T13:49:39.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.640+0000 7fe6701ca700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe668158fe0 0x7fe66806a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:39.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.640+0000 7fe6701ca700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe66806ac70 0x7fe66815bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:39.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.640+0000 7fe6701ca700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66815f9e0 con 0x7fe66815c100 2024-04-06T13:49:39.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.640+0000 7fe6701ca700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe66806b0f0 con 0x7fe66806ac70 2024-04-06T13:49:39.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.640+0000 7fe6701ca700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66815f0d0 con 0x7fe668158fe0 2024-04-06T13:49:39.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.640+0000 7fe66df66700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe668158fe0 0x7fe66806a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:39.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.640+0000 7fe66e767700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fe66815c100 0x7fe6680717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:33640/0 (socket says 172.21.15.189:33640) 2024-04-06T13:49:39.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.640+0000 7fe66e767700 1 -- 172.21.15.189:0/430300955 learned_addr learned my addr 172.21.15.189:0/430300955 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:39.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.640+0000 7fe66d765700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe66806ac70 0x7fe66815bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:39.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.640+0000 7fe66cf64700 1 -- 172.21.15.189:0/430300955 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703382018 0 0) 0x7fe66815f9e0 con 0x7fe66815c100 2024-04-06T13:49:39.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.640+0000 7fe66cf64700 1 -- 172.21.15.189:0/430300955 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6540035b0 con 0x7fe66815c100 2024-04-06T13:49:39.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.640+0000 7fe66df66700 1 -- 172.21.15.189:0/430300955 >> v2:172.21.15.192:3300/0 conn(0x7fe66806ac70 msgr2=0x7fe66815bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:39.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.640+0000 7fe66df66700 1 --2- 172.21.15.189:0/430300955 >> v2:172.21.15.192:3300/0 conn(0x7fe66806ac70 0x7fe66815bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.640+0000 7fe66df66700 1 -- 172.21.15.189:0/430300955 >> v1:172.21.15.192:6789/0 conn(0x7fe66815c100 legacy=0x7fe6680717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:39.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.640+0000 7fe66df66700 1 -- 172.21.15.189:0/430300955 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66815c8f0 con 0x7fe668158fe0 2024-04-06T13:49:39.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.640+0000 7fe66d765700 1 --2- 172.21.15.189:0/430300955 >> v2:172.21.15.192:3300/0 conn(0x7fe66806ac70 0x7fe66815bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:39.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.641+0000 7fe66df66700 1 --2- 172.21.15.189:0/430300955 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe668158fe0 0x7fe66806a730 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe664009840 tx=0x7fe664009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16da2e3440b691c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:39.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.641+0000 7fe66cf64700 1 -- 172.21.15.189:0/430300955 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe664011830 con 0x7fe668158fe0 2024-04-06T13:49:39.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.641+0000 7fe66cf64700 1 -- 172.21.15.189:0/430300955 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe664011990 con 0x7fe668158fe0 2024-04-06T13:49:39.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.641+0000 7fe66cf64700 1 -- 172.21.15.189:0/430300955 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe664011d20 con 0x7fe668158fe0 2024-04-06T13:49:39.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.641+0000 7fe6701ca700 1 -- 172.21.15.189:0/430300955 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe668158fe0 msgr2=0x7fe66806a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:39.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.641+0000 7fe6701ca700 1 --2- 172.21.15.189:0/430300955 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe668158fe0 0x7fe66806a730 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe664009840 tx=0x7fe664009b50 comp rx=0 tx=0).stop 2024-04-06T13:49:39.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.641+0000 7fe6701ca700 1 -- 172.21.15.189:0/430300955 shutdown_connections 2024-04-06T13:49:39.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.641+0000 7fe6701ca700 1 --2- 172.21.15.189:0/430300955 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe668158fe0 0x7fe66806a730 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.641+0000 7fe6701ca700 1 --2- 172.21.15.189:0/430300955 >> v2:172.21.15.192:3300/0 conn(0x7fe66806ac70 0x7fe66815bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.641+0000 7fe6701ca700 1 -- 172.21.15.189:0/430300955 >> 172.21.15.189:0/430300955 conn(0x7fe6680d00b0 msgr2=0x7fe6680d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:39.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.641+0000 7fe6701ca700 1 -- 172.21.15.189:0/430300955 shutdown_connections 2024-04-06T13:49:39.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.641+0000 7fe6701ca700 1 -- 172.21.15.189:0/430300955 wait complete. 2024-04-06T13:49:39.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.643+0000 7fe6701ca700 1 Processor -- start 2024-04-06T13:49:39.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.643+0000 7fe6701ca700 1 -- start start 2024-04-06T13:49:39.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.643+0000 7fe6701ca700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe66806ac70 0x7fe66806f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:39.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.643+0000 7fe6701ca700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe668158fe0 0x7fe6680aeed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:39.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.643+0000 7fe6701ca700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe66815c100 0x7fe6680af410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:39.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.643+0000 7fe6701ca700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6680adae0 con 0x7fe668158fe0 2024-04-06T13:49:39.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.643+0000 7fe6701ca700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66815df80 con 0x7fe66806ac70 2024-04-06T13:49:39.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.643+0000 7fe6701ca700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66815d090 con 0x7fe66815c100 2024-04-06T13:49:39.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.643+0000 7fe66d765700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe668158fe0 0x7fe6680aeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:39.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.643+0000 7fe66d765700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe668158fe0 0x7fe6680aeed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44828/0 (socket says 172.21.15.189:44828) 2024-04-06T13:49:39.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.643+0000 7fe66d765700 1 -- 172.21.15.189:0/2550249090 learned_addr learned my addr 172.21.15.189:0/2550249090 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:39.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.643+0000 7fe66e767700 1 --2- 172.21.15.189:0/2550249090 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe66815c100 0x7fe6680af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:39.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.644+0000 7fe66df66700 1 --2- 172.21.15.189:0/2550249090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe66806ac70 0x7fe66806f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:39.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.644+0000 7fe66df66700 1 -- 172.21.15.189:0/2550249090 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe66815c100 msgr2=0x7fe6680af410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:39.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.644+0000 7fe66df66700 1 --2- 172.21.15.189:0/2550249090 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe66815c100 0x7fe6680af410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.644+0000 7fe66df66700 1 -- 172.21.15.189:0/2550249090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe668158fe0 msgr2=0x7fe6680aeed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:39.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.644+0000 7fe66df66700 1 --2- 172.21.15.189:0/2550249090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe668158fe0 0x7fe6680aeed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.644+0000 7fe66df66700 1 -- 172.21.15.189:0/2550249090 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6580094a0 con 0x7fe66806ac70 2024-04-06T13:49:39.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.644+0000 7fe66d765700 1 --2- 172.21.15.189:0/2550249090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe668158fe0 0x7fe6680aeed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:39.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.644+0000 7fe66df66700 1 --2- 172.21.15.189:0/2550249090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe66806ac70 0x7fe66806f650 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe664016000 tx=0x7fe664004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:39.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.644+0000 7fe65e7fc700 1 -- 172.21.15.189:0/2550249090 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6640036a0 con 0x7fe66806ac70 2024-04-06T13:49:39.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.644+0000 7fe65e7fc700 1 -- 172.21.15.189:0/2550249090 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66401f070 con 0x7fe66806ac70 2024-04-06T13:49:39.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.644+0000 7fe6701ca700 1 -- 172.21.15.189:0/2550249090 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6640094a0 con 0x7fe66806ac70 2024-04-06T13:49:39.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.644+0000 7fe65e7fc700 1 -- 172.21.15.189:0/2550249090 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe664003a40 con 0x7fe66806ac70 2024-04-06T13:49:39.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.644+0000 7fe6701ca700 1 -- 172.21.15.189:0/2550249090 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe66815d930 con 0x7fe66806ac70 2024-04-06T13:49:39.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.645+0000 7fe6701ca700 1 -- 172.21.15.189:0/2550249090 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe668067bb0 con 0x7fe66806ac70 2024-04-06T13:49:39.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.646+0000 7fe65e7fc700 1 -- 172.21.15.189:0/2550249090 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 165) v1 ==== 89248+0+0 (secure 0 0 0) 0x7fe664003ba0 con 0x7fe66806ac70 2024-04-06T13:49:39.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.646+0000 7fe65e7fc700 1 --2- 172.21.15.189:0/2550249090 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe64c06adb0 0x7fe64c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:39.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.646+0000 7fe66d765700 1 -- 172.21.15.189:0/2550249090 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe64c06adb0 msgr2=0x7fe64c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:39.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.646+0000 7fe65e7fc700 1 -- 172.21.15.189:0/2550249090 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6531+0+0 (secure 0 0 0) 0x7fe664021070 con 0x7fe66806ac70 2024-04-06T13:49:39.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.646+0000 7fe66d765700 1 --2- 172.21.15.189:0/2550249090 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe64c06adb0 0x7fe64c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:49:39.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.651+0000 7fe65e7fc700 1 -- 172.21.15.189:0/2550249090 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe664055c60 con 0x7fe66806ac70 2024-04-06T13:49:39.675 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:49:39.674+0000 7f1a8caf9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:49:39.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.847+0000 7fe66d765700 1 -- 172.21.15.189:0/2550249090 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe64c06adb0 msgr2=0x7fe64c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:49:39.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.847+0000 7fe66d765700 1 --2- 172.21.15.189:0/2550249090 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe64c06adb0 0x7fe64c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:49:39.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.897+0000 7fe6701ca700 1 -- 172.21.15.189:0/2550249090 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe668074990 con 0x7fe66806ac70 2024-04-06T13:49:39.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.898+0000 7fe65e7fc700 1 -- 172.21.15.189:0/2550249090 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+4757 (secure 0 0 0) 0x7fe66402b090 con 0x7fe66806ac70 2024-04-06T13:49:39.900 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:39.900 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":131,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3073866011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3073866011},{"type":"v1","addr":"172.21.15.189:6836","nonce":3073866011}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":9917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/4213774859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4213774859},{"type":"v1","addr":"172.21.15.192:6834","nonce":4213774859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":11392,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/963373121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":963373121},{"type":"v1","addr":"172.21.15.189:6837","nonce":963373121}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:49:17.493194+0000","modified":"2024-04-06T13:49:20.503775+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12885},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12885":{"gid":12885,"name":"b","rank":0,"incarnation":128,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6835/1094572246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":1094572246},{"type":"v1","addr":"172.21.15.192:6835","nonce":1094572246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-04-06T13:49:39.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.900+0000 7fe6701ca700 1 -- 172.21.15.189:0/2550249090 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe64c06adb0 msgr2=0x7fe64c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:49:39.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.900+0000 7fe6701ca700 1 --2- 172.21.15.189:0/2550249090 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe64c06adb0 0x7fe64c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.900+0000 7fe6701ca700 1 -- 172.21.15.189:0/2550249090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe66806ac70 msgr2=0x7fe66806f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:39.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.901+0000 7fe6701ca700 1 --2- 172.21.15.189:0/2550249090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe66806ac70 0x7fe66806f650 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe664016000 tx=0x7fe664004750 comp rx=0 tx=0).stop 2024-04-06T13:49:39.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.901+0000 7fe6701ca700 1 -- 172.21.15.189:0/2550249090 shutdown_connections 2024-04-06T13:49:39.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.901+0000 7fe6701ca700 1 --2- 172.21.15.189:0/2550249090 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe64c06adb0 0x7fe64c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.901+0000 7fe6701ca700 1 --2- 172.21.15.189:0/2550249090 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe66815c100 0x7fe6680af410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.901+0000 7fe6701ca700 1 --2- 172.21.15.189:0/2550249090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe66806ac70 0x7fe66806f650 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.901+0000 7fe6701ca700 1 --2- 172.21.15.189:0/2550249090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe668158fe0 0x7fe6680aeed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:39.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.901+0000 7fe6701ca700 1 -- 172.21.15.189:0/2550249090 >> 172.21.15.189:0/2550249090 conn(0x7fe6680d00b0 msgr2=0x7fe6680ccac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:39.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.901+0000 7fe6701ca700 1 -- 172.21.15.189:0/2550249090 shutdown_connections 2024-04-06T13:49:39.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:39.901+0000 7fe6701ca700 1 -- 172.21.15.189:0/2550249090 wait complete. 2024-04-06T13:49:39.903 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 131 2024-04-06T13:49:39.914 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:49:39.966 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:39.965+0000 7fb3b94b6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:49:40.047 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:40.044+0000 7fb334bd5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:40.047 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:40.044+0000 7fb334bd5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:40.047 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:40.044+0000 7fb334bd5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:40.047 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:40.044+0000 7fb334bd5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:40.047 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:40.044+0000 7fb334bd5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:40.047 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:40.046+0000 7fb331bcf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:40.047 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:40.046+0000 7fb331bcf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:40.047 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:40.046+0000 7fb331bcf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:40.047 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:40.046+0000 7fb331bcf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:40.048 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:49:40.046+0000 7fb331bcf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:49:40.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.047+0000 7f9aea498700 1 Processor -- start 2024-04-06T13:49:40.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.047+0000 7f9aea498700 1 -- start start 2024-04-06T13:49:40.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.048+0000 7f9aea498700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9ae40b4af0 0x7f9ae40b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.048+0000 7f9aea498700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae40cdbe0 con 0x7f9ae40b4660 2024-04-06T13:49:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.048+0000 7f9aea498700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae415a940 con 0x7f9ae415a550 2024-04-06T13:49:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.048+0000 7f9aea498700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae415ac00 con 0x7f9ae40b4af0 2024-04-06T13:49:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.048+0000 7f9ae37fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9ae40b4660 0x7f9ae40b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46370/0 (socket says 172.21.15.189:46370) 2024-04-06T13:49:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.048+0000 7f9ae37fe700 1 -- 172.21.15.189:0/1948605615 learned_addr learned my addr 172.21.15.189:0/1948605615 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.048+0000 7f9ae8a35700 1 --2- 172.21.15.189:0/1948605615 >> v2:172.21.15.192:3300/0 conn(0x7f9ae40b4af0 0x7f9ae40b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.048+0000 7f9ae2ffd700 1 -- 172.21.15.189:0/1948605615 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379527621 0 0) 0x7f9ae40cdbe0 con 0x7f9ae40b4660 2024-04-06T13:49:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.048+0000 7f9ae2ffd700 1 -- 172.21.15.189:0/1948605615 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9acc0057c0 con 0x7f9ae40b4660 2024-04-06T13:49:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.048+0000 7f9ae2ffd700 1 -- 172.21.15.189:0/1948605615 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671997402 0 0) 0x7f9ae415a940 con 0x7f9ae415a550 2024-04-06T13:49:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.048+0000 7f9ae2ffd700 1 -- 172.21.15.189:0/1948605615 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ae40cdbe0 con 0x7f9ae415a550 2024-04-06T13:49:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.048+0000 7f9ae8a35700 1 -- 172.21.15.189:0/1948605615 >> v1:172.21.15.192:6789/0 conn(0x7f9ae415a550 legacy=0x7f9ae40cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.048+0000 7f9ae8a35700 1 -- 172.21.15.189:0/1948605615 >> v1:172.21.15.189:6789/0 conn(0x7f9ae40b4660 legacy=0x7f9ae40b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.049+0000 7f9ae2ffd700 1 -- 172.21.15.189:0/1948605615 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3265293239 0 0) 0x7f9acc0057c0 con 0x7f9ae40b4660 2024-04-06T13:49:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.049+0000 7f9ae8a35700 1 -- 172.21.15.189:0/1948605615 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae415aee0 con 0x7f9ae40b4af0 2024-04-06T13:49:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.049+0000 7f9ae8a35700 1 --2- 172.21.15.189:0/1948605615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae40b4af0 0x7f9ae40b36d0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009d90 tx=0x7f9adc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fdbad01df08cd61 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.049+0000 7f9ae2ffd700 1 -- 172.21.15.189:0/1948605615 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc00e070 con 0x7f9ae40b4af0 2024-04-06T13:49:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.049+0000 7f9ae2ffd700 1 -- 172.21.15.189:0/1948605615 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9adc003a20 con 0x7f9ae40b4af0 2024-04-06T13:49:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.049+0000 7f9aea498700 1 -- 172.21.15.189:0/1948605615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae40b4af0 msgr2=0x7f9ae40b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.049+0000 7f9aea498700 1 --2- 172.21.15.189:0/1948605615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae40b4af0 0x7f9ae40b36d0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009d90 tx=0x7f9adc00b330 comp rx=0 tx=0).stop 2024-04-06T13:49:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.050+0000 7f9aea498700 1 -- 172.21.15.189:0/1948605615 shutdown_connections 2024-04-06T13:49:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.050+0000 7f9aea498700 1 --2- 172.21.15.189:0/1948605615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae40b4af0 0x7f9ae40b36d0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.050+0000 7f9aea498700 1 -- 172.21.15.189:0/1948605615 >> 172.21.15.189:0/1948605615 conn(0x7f9ae40d00b0 msgr2=0x7f9ae40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.050+0000 7f9aea498700 1 -- 172.21.15.189:0/1948605615 shutdown_connections 2024-04-06T13:49:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.050+0000 7f9aea498700 1 -- 172.21.15.189:0/1948605615 wait complete. 2024-04-06T13:49:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.051+0000 7f9aea498700 1 Processor -- start 2024-04-06T13:49:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.051+0000 7f9aea498700 1 -- start start 2024-04-06T13:49:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.052+0000 7f9aea498700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae40b4660 0x7f9ae41f7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.052+0000 7f9aea498700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ae40b4af0 0x7f9ae4156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.052+0000 7f9aea498700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae40b2d70 0x7f9ae40b3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.052+0000 7f9aea498700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae40b3740 con 0x7f9ae40b4af0 2024-04-06T13:49:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.052+0000 7f9aea498700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae41f74e0 con 0x7f9ae40b4660 2024-04-06T13:49:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.052+0000 7f9aea498700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae4157090 con 0x7f9ae40b2d70 2024-04-06T13:49:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.052+0000 7f9ae37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ae40b4af0 0x7f9ae4156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.052+0000 7f9ae37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ae40b4af0 0x7f9ae4156280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44940/0 (socket says 172.21.15.189:44940) 2024-04-06T13:49:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.052+0000 7f9ae37fe700 1 -- 172.21.15.189:0/2666030791 learned_addr learned my addr 172.21.15.189:0/2666030791 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.052+0000 7f9ae3fff700 1 --2- 172.21.15.189:0/2666030791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae40b4660 0x7f9ae41f7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.052+0000 7f9ae8a35700 1 --2- 172.21.15.189:0/2666030791 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae40b2d70 0x7f9ae40b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.052+0000 7f9ae3fff700 1 -- 172.21.15.189:0/2666030791 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae40b2d70 msgr2=0x7f9ae40b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.052+0000 7f9ae3fff700 1 --2- 172.21.15.189:0/2666030791 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae40b2d70 0x7f9ae40b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.052+0000 7f9ae3fff700 1 -- 172.21.15.189:0/2666030791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ae40b4af0 msgr2=0x7f9ae4156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.052+0000 7f9ae3fff700 1 --2- 172.21.15.189:0/2666030791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ae40b4af0 0x7f9ae4156280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.052+0000 7f9ae3fff700 1 -- 172.21.15.189:0/2666030791 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9adc009a70 con 0x7f9ae40b4660 2024-04-06T13:49:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.052+0000 7f9ae8a35700 1 --2- 172.21.15.189:0/2666030791 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae40b2d70 0x7f9ae40b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.052+0000 7f9ae37fe700 1 --2- 172.21.15.189:0/2666030791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ae40b4af0 0x7f9ae4156280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.052+0000 7f9ae3fff700 1 --2- 172.21.15.189:0/2666030791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae40b4660 0x7f9ae41f7ef0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400c3c0 tx=0x7f9ad400c6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.053+0000 7f9ae0ff9700 1 -- 172.21.15.189:0/2666030791 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad40052d0 con 0x7f9ae40b4660 2024-04-06T13:49:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.053+0000 7f9ae0ff9700 1 -- 172.21.15.189:0/2666030791 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad4005430 con 0x7f9ae40b4660 2024-04-06T13:49:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.053+0000 7f9ae0ff9700 1 -- 172.21.15.189:0/2666030791 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad40038c0 con 0x7f9ae40b4660 2024-04-06T13:49:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.053+0000 7f9aea498700 1 -- 172.21.15.189:0/2666030791 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae4156ad0 con 0x7f9ae40b4660 2024-04-06T13:49:40.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.053+0000 7f9aea498700 1 -- 172.21.15.189:0/2666030791 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae41f76f0 con 0x7f9ae40b4660 2024-04-06T13:49:40.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.054+0000 7f9aea498700 1 -- 172.21.15.189:0/2666030791 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae4067bb0 con 0x7f9ae40b4660 2024-04-06T13:49:40.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.054+0000 7f9ae0ff9700 1 -- 172.21.15.189:0/2666030791 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 166) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f9ad4003a20 con 0x7f9ae40b4660 2024-04-06T13:49:40.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.054+0000 7f9ae0ff9700 1 -- 172.21.15.189:0/2666030791 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6789+0+0 (secure 0 0 0) 0x7f9ad40884d0 con 0x7f9ae40b4660 2024-04-06T13:49:40.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.059+0000 7f9ae0ff9700 1 -- 172.21.15.189:0/2666030791 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9ad4053780 con 0x7f9ae40b4660 2024-04-06T13:49:40.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.254+0000 7f9aea498700 1 -- 172.21.15.189:0/2666030791 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ae4074990 con 0x7f9ae40b4660 2024-04-06T13:49:40.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.255+0000 7f9ae0ff9700 1 -- 172.21.15.189:0/2666030791 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v271) v1 ==== 74+0+17418 (secure 0 0 0) 0x7f9ad4056da0 con 0x7f9ae40b4660 2024-04-06T13:49:40.256 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:40.256 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":271,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:49:39.980576+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:49:15.919092+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"268","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-04-06T13:49:16.459885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"269","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/2558044264":"2024-04-07T13:49:39.980542+0000","172.21.15.189:0/3563229223":"2024-04-07T13:49:39.980542+0000","172.21.15.189:0/1866217683":"2024-04-07T13:49:39.980542+0000","172.21.15.189:0/4273820182":"2024-04-07T13:49:39.980542+0000","172.21.15.189:0/1222359801":"2024-04-07T13:49:39.980542+0000","172.21.15.189:0/2061589774":"2024-04-07T13:49:39.980542+0000","172.21.15.189:0/2571565303":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/106800846":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/792254642":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/1837391793":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/1485104551":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/3877255335":"2024-04-07T13:49:31.207769+0000","172.21.15.189:6832/111704":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/811188054":"2024-04-07T13:49:31.207769+0000","172.21.15.189:6833/111704":"2024-04-07T13:49:31.207769+0000","172.21.15.189:0/2544306464":"2024-04-07T13:49:31.207769+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:49:40.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.257+0000 7f9aea498700 1 -- 172.21.15.189:0/2666030791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae40b4660 msgr2=0x7f9ae41f7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:40.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.257+0000 7f9aea498700 1 --2- 172.21.15.189:0/2666030791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae40b4660 0x7f9ae41f7ef0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400c3c0 tx=0x7f9ad400c6d0 comp rx=0 tx=0).stop 2024-04-06T13:49:40.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.257+0000 7f9aea498700 1 -- 172.21.15.189:0/2666030791 shutdown_connections 2024-04-06T13:49:40.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.257+0000 7f9aea498700 1 --2- 172.21.15.189:0/2666030791 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae40b2d70 0x7f9ae40b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:40.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.257+0000 7f9aea498700 1 --2- 172.21.15.189:0/2666030791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae40b4660 0x7f9ae41f7ef0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:40.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.257+0000 7f9aea498700 1 --2- 172.21.15.189:0/2666030791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ae40b4af0 0x7f9ae4156280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:40.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.257+0000 7f9aea498700 1 -- 172.21.15.189:0/2666030791 >> 172.21.15.189:0/2666030791 conn(0x7f9ae40d00b0 msgr2=0x7f9ae40cc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:40.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.257+0000 7f9aea498700 1 -- 172.21.15.189:0/2666030791 shutdown_connections 2024-04-06T13:49:40.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.257+0000 7f9aea498700 1 -- 172.21.15.189:0/2666030791 wait complete. 2024-04-06T13:49:40.272 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-06T13:49:40.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.393+0000 7fb548a51700 1 Processor -- start 2024-04-06T13:49:40.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.393+0000 7fb548a51700 1 -- start start 2024-04-06T13:49:40.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb548a51700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb5440b4660 0x7fb5440b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:40.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb548a51700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5440b5030 con 0x7fb54415a550 2024-04-06T13:49:40.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb548a51700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5440cdbe0 con 0x7fb5440b2fc0 2024-04-06T13:49:40.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb548a51700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb54415a940 con 0x7fb5440b4660 2024-04-06T13:49:40.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb54259c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb54415a550 0x7fb5440cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46376/0 (socket says 172.21.15.189:46376) 2024-04-06T13:49:40.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb54259c700 1 -- 172.21.15.189:0/842591565 learned_addr learned my addr 172.21.15.189:0/842591565 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:40.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb541d9b700 1 --2- 172.21.15.189:0/842591565 >> v2:172.21.15.192:3300/0 conn(0x7fb5440b4660 0x7fb5440b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:40.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb54159a700 1 -- 172.21.15.189:0/842591565 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070897590 0 0) 0x7fb5440b5030 con 0x7fb54415a550 2024-04-06T13:49:40.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb54159a700 1 -- 172.21.15.189:0/842591565 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5300035b0 con 0x7fb54415a550 2024-04-06T13:49:40.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb54159a700 1 -- 172.21.15.189:0/842591565 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3668047164 0 0) 0x7fb5300035b0 con 0x7fb54415a550 2024-04-06T13:49:40.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb54159a700 1 -- 172.21.15.189:0/842591565 >> v2:172.21.15.192:3300/0 conn(0x7fb5440b4660 msgr2=0x7fb5440b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:40.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb54159a700 1 --2- 172.21.15.189:0/842591565 >> v2:172.21.15.192:3300/0 conn(0x7fb5440b4660 0x7fb5440b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:40.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb54159a700 1 -- 172.21.15.189:0/842591565 >> v1:172.21.15.192:6789/0 conn(0x7fb5440b2fc0 legacy=0x7fb5440b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:40.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb54159a700 1 -- 172.21.15.189:0/842591565 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54415ac00 con 0x7fb54415a550 2024-04-06T13:49:40.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb54159a700 1 -- 172.21.15.189:0/842591565 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fb52c001670 con 0x7fb54415a550 2024-04-06T13:49:40.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb54159a700 1 -- 172.21.15.189:0/842591565 >> v1:172.21.15.189:6789/0 conn(0x7fb54415a550 legacy=0x7fb5440cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:40.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb54159a700 1 --2- 172.21.15.189:0/842591565 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb530003a00 0x7fb530003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:40.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb54159a700 1 --2- 172.21.15.189:0/842591565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb530004400 0x7fb5300048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:40.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb54159a700 1 --2- 172.21.15.189:0/842591565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5300050d0 0x7fb53000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:40.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb54159a700 1 -- 172.21.15.189:0/842591565 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54415a940 con 0x7fb530004400 2024-04-06T13:49:40.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb54159a700 1 -- 172.21.15.189:0/842591565 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53000c030 con 0x7fb5300050d0 2024-04-06T13:49:40.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.395+0000 7fb54159a700 1 -- 172.21.15.189:0/842591565 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53000c1f0 con 0x7fb530003a00 2024-04-06T13:49:40.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.396+0000 7fb541d9b700 1 --2- 172.21.15.189:0/842591565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb530004400 0x7fb5300048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:40.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.396+0000 7fb54259c700 1 --2- 172.21.15.189:0/842591565 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb530003a00 0x7fb530003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:40.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.396+0000 7fb542d9d700 1 --2- 172.21.15.189:0/842591565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5300050d0 0x7fb53000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:40.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.396+0000 7fb541d9b700 1 -- 172.21.15.189:0/842591565 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb530003a00 msgr2=0x7fb530003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:40.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.396+0000 7fb541d9b700 1 --2- 172.21.15.189:0/842591565 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb530003a00 0x7fb530003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:40.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.396+0000 7fb541d9b700 1 -- 172.21.15.189:0/842591565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5300050d0 msgr2=0x7fb53000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:40.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.396+0000 7fb541d9b700 1 --2- 172.21.15.189:0/842591565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5300050d0 0x7fb53000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:40.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.396+0000 7fb541d9b700 1 -- 172.21.15.189:0/842591565 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5300035b0 con 0x7fb530004400 2024-04-06T13:49:40.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.396+0000 7fb542d9d700 1 -- 172.21.15.189:0/842591565 reap_dead start 2024-04-06T13:49:40.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.396+0000 7fb54259c700 1 --2- 172.21.15.189:0/842591565 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb530003a00 0x7fb530003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:40.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.396+0000 7fb541d9b700 1 --2- 172.21.15.189:0/842591565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb530004400 0x7fb5300048d0 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 crypto rx=0x7fb534009b60 tx=0x7fb530005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38554029e87ac739 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:40.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.396+0000 7fb54159a700 1 -- 172.21.15.189:0/842591565 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53400e6c0 con 0x7fb530004400 2024-04-06T13:49:40.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.396+0000 7fb54159a700 1 -- 172.21.15.189:0/842591565 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53400e820 con 0x7fb530004400 2024-04-06T13:49:40.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.396+0000 7fb54159a700 1 -- 172.21.15.189:0/842591565 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53400eb90 con 0x7fb530004400 2024-04-06T13:49:40.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.397+0000 7fb548a51700 1 -- 172.21.15.189:0/842591565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb530004400 msgr2=0x7fb5300048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:40.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.397+0000 7fb548a51700 1 --2- 172.21.15.189:0/842591565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb530004400 0x7fb5300048d0 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 crypto rx=0x7fb534009b60 tx=0x7fb530005880 comp rx=0 tx=0).stop 2024-04-06T13:49:40.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.397+0000 7fb548a51700 1 -- 172.21.15.189:0/842591565 shutdown_connections 2024-04-06T13:49:40.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.397+0000 7fb548a51700 1 --2- 172.21.15.189:0/842591565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb530004400 0x7fb5300048d0 unknown :-1 s=CLOSED pgs=3022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:40.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.397+0000 7fb548a51700 1 -- 172.21.15.189:0/842591565 >> 172.21.15.189:0/842591565 conn(0x7fb5440d00b0 msgr2=0x7fb5440d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:40.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.397+0000 7fb548a51700 1 -- 172.21.15.189:0/842591565 shutdown_connections 2024-04-06T13:49:40.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.397+0000 7fb548a51700 1 -- 172.21.15.189:0/842591565 wait complete. 2024-04-06T13:49:40.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.398+0000 7fb548a51700 1 Processor -- start 2024-04-06T13:49:40.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.398+0000 7fb548a51700 1 -- start start 2024-04-06T13:49:40.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.399+0000 7fb548a51700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb530004400 0x7fb5440afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:40.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.399+0000 7fb548a51700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb544158010 0x7fb544155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:40.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.399+0000 7fb548a51700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5440b1450 0x7fb544148a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:40.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.399+0000 7fb548a51700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5441495a0 con 0x7fb5440b1450 2024-04-06T13:49:40.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.399+0000 7fb548a51700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5440b0210 con 0x7fb530004400 2024-04-06T13:49:40.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.399+0000 7fb548a51700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5440b04f0 con 0x7fb544158010 2024-04-06T13:49:40.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.399+0000 7fb542d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5440b1450 0x7fb544148a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:40.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.399+0000 7fb541d9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb544158010 0x7fb544155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:40.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.399+0000 7fb542d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5440b1450 0x7fb544148a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44958/0 (socket says 172.21.15.189:44958) 2024-04-06T13:49:40.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.399+0000 7fb542d9d700 1 -- 172.21.15.189:0/418838697 learned_addr learned my addr 172.21.15.189:0/418838697 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:40.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.399+0000 7fb54259c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb530004400 0x7fb5440afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:40.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.399+0000 7fb542d9d700 1 -- 172.21.15.189:0/418838697 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb544158010 msgr2=0x7fb544155df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:40.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.399+0000 7fb542d9d700 1 --2- 172.21.15.189:0/418838697 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb544158010 0x7fb544155df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:40.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.399+0000 7fb542d9d700 1 -- 172.21.15.189:0/418838697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb530004400 msgr2=0x7fb5440afc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:40.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.399+0000 7fb542d9d700 1 --2- 172.21.15.189:0/418838697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb530004400 0x7fb5440afc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:40.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.399+0000 7fb542d9d700 1 -- 172.21.15.189:0/418838697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5440b07d0 con 0x7fb5440b1450 2024-04-06T13:49:40.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.399+0000 7fb541d9b700 1 --2- 172.21.15.189:0/418838697 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb544158010 0x7fb544155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:40.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.399+0000 7fb542d9d700 1 --2- 172.21.15.189:0/418838697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5440b1450 0x7fb544148a20 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 crypto rx=0x7fb5440d1110 tx=0x7fb53c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:40.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.399+0000 7fb53affd700 1 -- 172.21.15.189:0/418838697 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c003c10 con 0x7fb5440b1450 2024-04-06T13:49:40.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.400+0000 7fb53affd700 1 -- 172.21.15.189:0/418838697 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53c003db0 con 0x7fb5440b1450 2024-04-06T13:49:40.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.400+0000 7fb548a51700 1 -- 172.21.15.189:0/418838697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5440b0ab0 con 0x7fb5440b1450 2024-04-06T13:49:40.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.400+0000 7fb53affd700 1 -- 172.21.15.189:0/418838697 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c005310 con 0x7fb5440b1450 2024-04-06T13:49:40.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.400+0000 7fb548a51700 1 -- 172.21.15.189:0/418838697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb544156330 con 0x7fb5440b1450 2024-04-06T13:49:40.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.401+0000 7fb548a51700 1 -- 172.21.15.189:0/418838697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5440b0cd0 con 0x7fb5440b1450 2024-04-06T13:49:40.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.401+0000 7fb53affd700 1 -- 172.21.15.189:0/418838697 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 166) v1 ==== 88856+0+0 (secure 0 0 0) 0x7fb53c005470 con 0x7fb5440b1450 2024-04-06T13:49:40.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.401+0000 7fb53affd700 1 -- 172.21.15.189:0/418838697 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6789+0+0 (secure 0 0 0) 0x7fb53c089270 con 0x7fb5440b1450 2024-04-06T13:49:40.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.406+0000 7fb53affd700 1 -- 172.21.15.189:0/418838697 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb53c054460 con 0x7fb5440b1450 2024-04-06T13:49:40.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.643+0000 7fb548a51700 1 -- 172.21.15.189:0/418838697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb544067b80 con 0x7fb5440b1450 2024-04-06T13:49:40.645 INFO:tasks.ceph.mon.a.smithi189.stderr:2024-04-06T13:49:40.644+0000 7fa9fe1ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T13:49:40.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.656+0000 7fb53affd700 1 -- 172.21.15.189:0/418838697 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v132) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb53c057a80 con 0x7fb5440b1450 2024-04-06T13:49:40.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.659+0000 7fb548a51700 1 -- 172.21.15.189:0/418838697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5440b1450 msgr2=0x7fb544148a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.659+0000 7fb548a51700 1 --2- 172.21.15.189:0/418838697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5440b1450 0x7fb544148a20 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 crypto rx=0x7fb5440d1110 tx=0x7fb53c00ee70 comp rx=0 tx=0).stop 2024-04-06T13:49:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.659+0000 7fb548a51700 1 -- 172.21.15.189:0/418838697 shutdown_connections 2024-04-06T13:49:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.659+0000 7fb548a51700 1 --2- 172.21.15.189:0/418838697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5440b1450 0x7fb544148a20 unknown :-1 s=CLOSED pgs=3023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.659+0000 7fb548a51700 1 --2- 172.21.15.189:0/418838697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb530004400 0x7fb5440afc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.659+0000 7fb548a51700 1 --2- 172.21.15.189:0/418838697 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb544158010 0x7fb544155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.659+0000 7fb548a51700 1 -- 172.21.15.189:0/418838697 >> 172.21.15.189:0/418838697 conn(0x7fb5440d00b0 msgr2=0x7fb5440cdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.659+0000 7fb548a51700 1 -- 172.21.15.189:0/418838697 shutdown_connections 2024-04-06T13:49:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.659+0000 7fb548a51700 1 -- 172.21.15.189:0/418838697 wait complete. 2024-04-06T13:49:40.661 INFO:teuthology.orchestra.run.smithi189.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-06T13:49:40.662 INFO:tasks.ceph.mds.b.smithi192.stderr: -5217> 2024-04-06T13:49:18.913+0000 7f940c17e700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T13:49:40.672 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-06T13:49:40.693 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T13:49:40.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.795+0000 7f64f59b2700 1 Processor -- start 2024-04-06T13:49:40.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.795+0000 7f64f59b2700 1 -- start start 2024-04-06T13:49:40.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.796+0000 7f64f59b2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64f015d040 0x7f64f015dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:40.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.796+0000 7f64f59b2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f015d8d0 con 0x7f64f006cbf0 2024-04-06T13:49:40.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.796+0000 7f64f59b2700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f015d580 con 0x7f64f015e2a0 2024-04-06T13:49:40.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.796+0000 7f64f59b2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f00cba40 con 0x7f64f015d040 2024-04-06T13:49:40.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.796+0000 7f64ef7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f64f006cbf0 0x7f64f00cc1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46378/0 (socket says 172.21.15.189:46378) 2024-04-06T13:49:40.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.796+0000 7f64ef7fe700 1 -- 172.21.15.189:0/883725634 learned_addr learned my addr 172.21.15.189:0/883725634 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:40.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.796+0000 7f64eeffd700 1 --2- 172.21.15.189:0/883725634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64f015d040 0x7f64f015dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:40.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.796+0000 7f64edffb700 1 -- 172.21.15.189:0/883725634 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625888009 0 0) 0x7f64f015d8d0 con 0x7f64f006cbf0 2024-04-06T13:49:40.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.796+0000 7f64edffb700 1 -- 172.21.15.189:0/883725634 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64d80035b0 con 0x7f64f006cbf0 2024-04-06T13:49:40.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.796+0000 7f64edffb700 1 -- 172.21.15.189:0/883725634 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1382406698 0 0) 0x7f64f015d580 con 0x7f64f015e2a0 2024-04-06T13:49:40.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.796+0000 7f64edffb700 1 -- 172.21.15.189:0/883725634 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64f015d8d0 con 0x7f64f015e2a0 2024-04-06T13:49:40.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.796+0000 7f64eeffd700 1 -- 172.21.15.189:0/883725634 >> v1:172.21.15.192:6789/0 conn(0x7f64f015e2a0 legacy=0x7f64f006c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:40.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.796+0000 7f64eeffd700 1 -- 172.21.15.189:0/883725634 >> v1:172.21.15.189:6789/0 conn(0x7f64f006cbf0 legacy=0x7f64f00cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:40.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.796+0000 7f64eeffd700 1 -- 172.21.15.189:0/883725634 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f00b3240 con 0x7f64f015d040 2024-04-06T13:49:40.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.797+0000 7f64eeffd700 1 --2- 172.21.15.189:0/883725634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64f015d040 0x7f64f015dd60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f64e400aa10 tx=0x7f64e400ad20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cede380f82ef87aa server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:40.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.797+0000 7f64edffb700 1 -- 172.21.15.189:0/883725634 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e400cb70 con 0x7f64f015d040 2024-04-06T13:49:40.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.797+0000 7f64edffb700 1 -- 172.21.15.189:0/883725634 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e400ccd0 con 0x7f64f015d040 2024-04-06T13:49:40.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.797+0000 7f64edffb700 1 -- 172.21.15.189:0/883725634 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e4018620 con 0x7f64f015d040 2024-04-06T13:49:40.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.797+0000 7f64f59b2700 1 -- 172.21.15.189:0/883725634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64f015d040 msgr2=0x7f64f015dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:40.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.797+0000 7f64f59b2700 1 --2- 172.21.15.189:0/883725634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64f015d040 0x7f64f015dd60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f64e400aa10 tx=0x7f64e400ad20 comp rx=0 tx=0).stop 2024-04-06T13:49:40.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.797+0000 7f64f59b2700 1 -- 172.21.15.189:0/883725634 shutdown_connections 2024-04-06T13:49:40.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.797+0000 7f64f59b2700 1 --2- 172.21.15.189:0/883725634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64f015d040 0x7f64f015dd60 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:40.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.797+0000 7f64f59b2700 1 -- 172.21.15.189:0/883725634 >> 172.21.15.189:0/883725634 conn(0x7f64f006a730 msgr2=0x7f64f006ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:40.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.797+0000 7f64f59b2700 1 -- 172.21.15.189:0/883725634 shutdown_connections 2024-04-06T13:49:40.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.797+0000 7f64f59b2700 1 -- 172.21.15.189:0/883725634 wait complete. 2024-04-06T13:49:40.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.799+0000 7f64f59b2700 1 Processor -- start 2024-04-06T13:49:40.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.799+0000 7f64f59b2700 1 -- start start 2024-04-06T13:49:40.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.799+0000 7f64f59b2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64f006cbf0 0x7f64f0158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:40.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.799+0000 7f64f59b2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64f015d040 0x7f64f01515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:40.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.799+0000 7f64f59b2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64f015c0d0 0x7f64f015b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:40.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.799+0000 7f64f59b2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f015c580 con 0x7f64f015c0d0 2024-04-06T13:49:40.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.799+0000 7f64f59b2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f0152580 con 0x7f64f015d040 2024-04-06T13:49:40.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.799+0000 7f64f59b2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f0151ce0 con 0x7f64f006cbf0 2024-04-06T13:49:40.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.800+0000 7f64ef7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64f015c0d0 0x7f64f015b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:40.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.800+0000 7f64ef7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64f015c0d0 0x7f64f015b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44964/0 (socket says 172.21.15.189:44964) 2024-04-06T13:49:40.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.800+0000 7f64ef7fe700 1 -- 172.21.15.189:0/308730943 learned_addr learned my addr 172.21.15.189:0/308730943 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:40.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.800+0000 7f64eeffd700 1 --2- 172.21.15.189:0/308730943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64f006cbf0 0x7f64f0158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:40.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.800+0000 7f64ee7fc700 1 --2- 172.21.15.189:0/308730943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64f015d040 0x7f64f01515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:40.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.800+0000 7f64ef7fe700 1 -- 172.21.15.189:0/308730943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64f006cbf0 msgr2=0x7f64f0158470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:40.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.800+0000 7f64ef7fe700 1 --2- 172.21.15.189:0/308730943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64f006cbf0 0x7f64f0158470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:40.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.800+0000 7f64ef7fe700 1 -- 172.21.15.189:0/308730943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64f015d040 msgr2=0x7f64f01515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:40.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.800+0000 7f64ef7fe700 1 --2- 172.21.15.189:0/308730943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64f015d040 0x7f64f01515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:40.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.800+0000 7f64ef7fe700 1 -- 172.21.15.189:0/308730943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e400a640 con 0x7f64f015c0d0 2024-04-06T13:49:40.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.800+0000 7f64ef7fe700 1 --2- 172.21.15.189:0/308730943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64f015c0d0 0x7f64f015b060 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7f64e800cb70 tx=0x7f64e800ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:40.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.800+0000 7f64d7fff700 1 -- 172.21.15.189:0/308730943 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e800aa40 con 0x7f64f015c0d0 2024-04-06T13:49:40.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.800+0000 7f64d7fff700 1 -- 172.21.15.189:0/308730943 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e800aba0 con 0x7f64f015c0d0 2024-04-06T13:49:40.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.800+0000 7f64f59b2700 1 -- 172.21.15.189:0/308730943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f0151fc0 con 0x7f64f015c0d0 2024-04-06T13:49:40.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.800+0000 7f64d7fff700 1 -- 172.21.15.189:0/308730943 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e8012400 con 0x7f64f015c0d0 2024-04-06T13:49:40.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.800+0000 7f64f59b2700 1 -- 172.21.15.189:0/308730943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f01f0f20 con 0x7f64f015c0d0 2024-04-06T13:49:40.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.801+0000 7f64f59b2700 1 -- 172.21.15.189:0/308730943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f0067bf0 con 0x7f64f015c0d0 2024-04-06T13:49:40.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.802+0000 7f64d7fff700 1 -- 172.21.15.189:0/308730943 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 166) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f64e801b3b0 con 0x7f64f015c0d0 2024-04-06T13:49:40.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.802+0000 7f64d7fff700 1 -- 172.21.15.189:0/308730943 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 6875+0+0 (secure 0 0 0) 0x7f64e8088ee0 con 0x7f64f015c0d0 2024-04-06T13:49:40.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.806+0000 7f64d7fff700 1 -- 172.21.15.189:0/308730943 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f64e8054080 con 0x7f64f015c0d0 2024-04-06T13:49:40.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.989+0000 7f64d7fff700 1 -- 172.21.15.189:0/308730943 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 167) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f64e8053b30 con 0x7f64f015c0d0 2024-04-06T13:49:40.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.990+0000 7f64d7fff700 1 --2- 172.21.15.189:0/308730943 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f64dc036150 0x7f64dc072b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:40.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.991+0000 7f64eeffd700 1 --2- 172.21.15.189:0/308730943 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f64dc036150 0x7f64dc072b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:40.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:40.991+0000 7f64eeffd700 1 --2- 172.21.15.189:0/308730943 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f64dc036150 0x7f64dc072b20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f64e400a1c0 tx=0x7f64e401b040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:41.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.056+0000 7f64f59b2700 1 -- 172.21.15.189:0/308730943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f64f01f1140 con 0x7f64f015c0d0 2024-04-06T13:49:41.661 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.660+0000 7f64d7fff700 1 -- 172.21.15.189:0/308730943 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v133) v1 ==== 104+0+0 (secure 0 0 0) 0x7f64e8073340 con 0x7f64f015c0d0 2024-04-06T13:49:41.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.662+0000 7f64f59b2700 1 -- 172.21.15.189:0/308730943 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f64dc036150 msgr2=0x7f64dc072b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:41.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.662+0000 7f64f59b2700 1 --2- 172.21.15.189:0/308730943 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f64dc036150 0x7f64dc072b20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f64e400a1c0 tx=0x7f64e401b040 comp rx=0 tx=0).stop 2024-04-06T13:49:41.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.662+0000 7f64f59b2700 1 -- 172.21.15.189:0/308730943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64f015c0d0 msgr2=0x7f64f015b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:41.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.662+0000 7f64f59b2700 1 --2- 172.21.15.189:0/308730943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64f015c0d0 0x7f64f015b060 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7f64e800cb70 tx=0x7f64e800ce80 comp rx=0 tx=0).stop 2024-04-06T13:49:41.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.662+0000 7f64f59b2700 1 -- 172.21.15.189:0/308730943 shutdown_connections 2024-04-06T13:49:41.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.662+0000 7f64f59b2700 1 --2- 172.21.15.189:0/308730943 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f64dc036150 0x7f64dc072b20 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:41.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.662+0000 7f64f59b2700 1 --2- 172.21.15.189:0/308730943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64f015c0d0 0x7f64f015b060 unknown :-1 s=CLOSED pgs=3025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:41.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.662+0000 7f64f59b2700 1 --2- 172.21.15.189:0/308730943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64f006cbf0 0x7f64f0158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:41.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.662+0000 7f64f59b2700 1 --2- 172.21.15.189:0/308730943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64f015d040 0x7f64f01515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:41.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.662+0000 7f64f59b2700 1 -- 172.21.15.189:0/308730943 >> 172.21.15.189:0/308730943 conn(0x7f64f006a730 msgr2=0x7f64f006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:41.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.662+0000 7f64f59b2700 1 -- 172.21.15.189:0/308730943 shutdown_connections 2024-04-06T13:49:41.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.662+0000 7f64f59b2700 1 -- 172.21.15.189:0/308730943 wait complete. 2024-04-06T13:49:41.676 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-06T13:49:41.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.801+0000 7f16e79b3700 1 Processor -- start 2024-04-06T13:49:41.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.801+0000 7f16e79b3700 1 -- start start 2024-04-06T13:49:41.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.802+0000 7f16e79b3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f16e00d17a0 0x7f16e006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:41.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.802+0000 7f16e79b3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f16e0146dd0 0x7f16e006d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:41.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.802+0000 7f16e79b3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f16e00cedf0 0x7f16e00cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:41.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.802+0000 7f16e79b3700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16e00cf760 con 0x7f16e00cedf0 2024-04-06T13:49:41.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.802+0000 7f16e79b3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16e006c0a0 con 0x7f16e00d17a0 2024-04-06T13:49:41.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.802+0000 7f16e79b3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16e006c380 con 0x7f16e0146dd0 2024-04-06T13:49:41.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.802+0000 7f16e5f50700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f16e00cedf0 0x7f16e00cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:41.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.802+0000 7f16e5f50700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f16e00cedf0 0x7f16e00cf220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44972/0 (socket says 172.21.15.189:44972) 2024-04-06T13:49:41.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.802+0000 7f16e5f50700 1 -- 172.21.15.189:0/3978192430 learned_addr learned my addr 172.21.15.189:0/3978192430 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:41.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.803+0000 7f16e4f4e700 1 --2- 172.21.15.189:0/3978192430 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f16e0146dd0 0x7f16e006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:41.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.803+0000 7f16e574f700 1 --2- 172.21.15.189:0/3978192430 >> v2:172.21.15.192:3300/0 conn(0x7f16e00d17a0 0x7f16e006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.803+0000 7f16e5f50700 1 -- 172.21.15.189:0/3978192430 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f16e0146dd0 msgr2=0x7f16e006d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.803+0000 7f16e5f50700 1 --2- 172.21.15.189:0/3978192430 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f16e0146dd0 0x7f16e006d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.803+0000 7f16e5f50700 1 -- 172.21.15.189:0/3978192430 >> v2:172.21.15.192:3300/0 conn(0x7f16e00d17a0 msgr2=0x7f16e006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.803+0000 7f16e5f50700 1 --2- 172.21.15.189:0/3978192430 >> v2:172.21.15.192:3300/0 conn(0x7f16e00d17a0 0x7f16e006b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.803+0000 7f16e5f50700 1 -- 172.21.15.189:0/3978192430 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e006c660 con 0x7f16e00cedf0 2024-04-06T13:49:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.803+0000 7f16e574f700 1 --2- 172.21.15.189:0/3978192430 >> v2:172.21.15.192:3300/0 conn(0x7f16e00d17a0 0x7f16e006b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.803+0000 7f16e5f50700 1 --2- 172.21.15.189:0/3978192430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e00cedf0 0x7f16e00cf220 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7f16e004fb40 tx=0x7f16d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b346c1bb07997623 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.803+0000 7f16d7fff700 1 -- 172.21.15.189:0/3978192430 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d800e070 con 0x7f16e00cedf0 2024-04-06T13:49:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.803+0000 7f16d7fff700 1 -- 172.21.15.189:0/3978192430 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d800b9c0 con 0x7f16e00cedf0 2024-04-06T13:49:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.803+0000 7f16d7fff700 1 -- 172.21.15.189:0/3978192430 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d800bd70 con 0x7f16e00cedf0 2024-04-06T13:49:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.803+0000 7f16e79b3700 1 -- 172.21.15.189:0/3978192430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e00cedf0 msgr2=0x7f16e00cf220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.803+0000 7f16e79b3700 1 --2- 172.21.15.189:0/3978192430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e00cedf0 0x7f16e00cf220 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7f16e004fb40 tx=0x7f16d800b330 comp rx=0 tx=0).stop 2024-04-06T13:49:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.803+0000 7f16e79b3700 1 -- 172.21.15.189:0/3978192430 shutdown_connections 2024-04-06T13:49:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.804+0000 7f16e79b3700 1 --2- 172.21.15.189:0/3978192430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e00cedf0 0x7f16e00cf220 unknown :-1 s=CLOSED pgs=3026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:41.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.804+0000 7f16e79b3700 1 --2- 172.21.15.189:0/3978192430 >> v2:172.21.15.192:3300/0 conn(0x7f16e00d17a0 0x7f16e006b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.804+0000 7f16e79b3700 1 --2- 172.21.15.189:0/3978192430 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f16e0146dd0 0x7f16e006d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.804+0000 7f16e79b3700 1 -- 172.21.15.189:0/3978192430 >> 172.21.15.189:0/3978192430 conn(0x7f16e00b2db0 msgr2=0x7f16e00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.804+0000 7f16e79b3700 1 -- 172.21.15.189:0/3978192430 shutdown_connections 2024-04-06T13:49:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.804+0000 7f16e79b3700 1 -- 172.21.15.189:0/3978192430 wait complete. 2024-04-06T13:49:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.805+0000 7f16e79b3700 1 Processor -- start 2024-04-06T13:49:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.805+0000 7f16e79b3700 1 -- start start 2024-04-06T13:49:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.805+0000 7f16e79b3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e00cedf0 0x7f16e01f5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.805+0000 7f16e79b3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f16e00d17a0 0x7f16e015fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.805+0000 7f16e79b3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16e0146dd0 0x7f16e00ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.805+0000 7f16e79b3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e01f5c50 con 0x7f16e00cedf0 2024-04-06T13:49:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.805+0000 7f16e79b3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e00c6a90 con 0x7f16e0146dd0 2024-04-06T13:49:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.805+0000 7f16e79b3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16e00c6d70 con 0x7f16e00d17a0 2024-04-06T13:49:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.806+0000 7f16e574f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e00cedf0 0x7f16e01f5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:41.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.806+0000 7f16e574f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e00cedf0 0x7f16e01f5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44976/0 (socket says 172.21.15.189:44976) 2024-04-06T13:49:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.806+0000 7f16e574f700 1 -- 172.21.15.189:0/3074976542 learned_addr learned my addr 172.21.15.189:0/3074976542 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.806+0000 7f16e4f4e700 1 --2- 172.21.15.189:0/3074976542 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f16e00d17a0 0x7f16e015fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.806+0000 7f16e5f50700 1 --2- 172.21.15.189:0/3074976542 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16e0146dd0 0x7f16e00ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.806+0000 7f16e574f700 1 -- 172.21.15.189:0/3074976542 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f16e00d17a0 msgr2=0x7f16e015fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.806+0000 7f16e574f700 1 --2- 172.21.15.189:0/3074976542 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f16e00d17a0 0x7f16e015fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.806+0000 7f16e574f700 1 -- 172.21.15.189:0/3074976542 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16e0146dd0 msgr2=0x7f16e00ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.806+0000 7f16e574f700 1 --2- 172.21.15.189:0/3074976542 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16e0146dd0 0x7f16e00ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.806+0000 7f16e574f700 1 -- 172.21.15.189:0/3074976542 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d8009b30 con 0x7f16e00cedf0 2024-04-06T13:49:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.806+0000 7f16e5f50700 1 --2- 172.21.15.189:0/3074976542 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16e0146dd0 0x7f16e00ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.806+0000 7f16e574f700 1 --2- 172.21.15.189:0/3074976542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e00cedf0 0x7f16e01f5710 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 crypto rx=0x7f16d000c7f0 tx=0x7f16d000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.806+0000 7f16d5ffb700 1 -- 172.21.15.189:0/3074976542 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d0011800 con 0x7f16e00cedf0 2024-04-06T13:49:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.806+0000 7f16d5ffb700 1 -- 172.21.15.189:0/3074976542 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d0011960 con 0x7f16e00cedf0 2024-04-06T13:49:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.806+0000 7f16d5ffb700 1 -- 172.21.15.189:0/3074976542 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d0011d10 con 0x7f16e00cedf0 2024-04-06T13:49:41.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.806+0000 7f16e79b3700 1 -- 172.21.15.189:0/3074976542 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16e00c7050 con 0x7f16e00cedf0 2024-04-06T13:49:41.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.806+0000 7f16e79b3700 1 -- 172.21.15.189:0/3074976542 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e00c7610 con 0x7f16e00cedf0 2024-04-06T13:49:41.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.807+0000 7f16e79b3700 1 -- 172.21.15.189:0/3074976542 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16e0074990 con 0x7f16e00cedf0 2024-04-06T13:49:41.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.808+0000 7f16d5ffb700 1 -- 172.21.15.189:0/3074976542 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 167) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f16d0018360 con 0x7f16e00cedf0 2024-04-06T13:49:41.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.808+0000 7f16d5ffb700 1 --2- 172.21.15.189:0/3074976542 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f16c406aae0 0x7f16c406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:41.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.808+0000 7f16d5ffb700 1 -- 172.21.15.189:0/3074976542 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 6875+0+0 (secure 0 0 0) 0x7f16d0089350 con 0x7f16e00cedf0 2024-04-06T13:49:41.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.808+0000 7f16e4f4e700 1 --2- 172.21.15.189:0/3074976542 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f16c406aae0 0x7f16c406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:41.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.809+0000 7f16e4f4e700 1 --2- 172.21.15.189:0/3074976542 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f16c406aae0 0x7f16c406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f16e0072ea0 tx=0x7f16dc008040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:41.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:41.813+0000 7f16d5ffb700 1 -- 172.21.15.189:0/3074976542 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f16d00543e0 con 0x7f16e00cedf0 2024-04-06T13:49:42.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:42.011+0000 7f16e79b3700 1 -- 172.21.15.189:0/3074976542 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f16e00d1e20 con 0x7f16e00cedf0 2024-04-06T13:49:42.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:42.997+0000 7f16d5ffb700 1 -- 172.21.15.189:0/3074976542 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 168) v1 ==== 89191+0+0 (secure 0 0 0) 0x7f16d00547b0 con 0x7f16e00cedf0 2024-04-06T13:49:42.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:42.999+0000 7f16d5ffb700 1 -- 172.21.15.189:0/3074976542 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v273) v1 ==== 181+0+0 (secure 0 0 0) 0x7f16d0070e00 con 0x7f16e00cedf0 2024-04-06T13:49:43.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.002+0000 7f16e79b3700 1 -- 172.21.15.189:0/3074976542 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f16c406aae0 msgr2=0x7f16c406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:43.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.002+0000 7f16e79b3700 1 --2- 172.21.15.189:0/3074976542 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f16c406aae0 0x7f16c406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f16e0072ea0 tx=0x7f16dc008040 comp rx=0 tx=0).stop 2024-04-06T13:49:43.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.002+0000 7f16e79b3700 1 -- 172.21.15.189:0/3074976542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e00cedf0 msgr2=0x7f16e01f5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:43.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.002+0000 7f16e79b3700 1 --2- 172.21.15.189:0/3074976542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e00cedf0 0x7f16e01f5710 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 crypto rx=0x7f16d000c7f0 tx=0x7f16d000cbb0 comp rx=0 tx=0).stop 2024-04-06T13:49:43.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.003+0000 7f16e79b3700 1 -- 172.21.15.189:0/3074976542 shutdown_connections 2024-04-06T13:49:43.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.003+0000 7f16e79b3700 1 --2- 172.21.15.189:0/3074976542 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f16c406aae0 0x7f16c406cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:43.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.003+0000 7f16e79b3700 1 --2- 172.21.15.189:0/3074976542 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16e0146dd0 0x7f16e00ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:43.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.003+0000 7f16e79b3700 1 --2- 172.21.15.189:0/3074976542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e00cedf0 0x7f16e01f5710 unknown :-1 s=CLOSED pgs=3027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:43.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.003+0000 7f16e79b3700 1 --2- 172.21.15.189:0/3074976542 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f16e00d17a0 0x7f16e015fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:43.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.003+0000 7f16e79b3700 1 -- 172.21.15.189:0/3074976542 >> 172.21.15.189:0/3074976542 conn(0x7f16e00b2db0 msgr2=0x7f16e0160fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:43.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.003+0000 7f16e79b3700 1 -- 172.21.15.189:0/3074976542 shutdown_connections 2024-04-06T13:49:43.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.003+0000 7f16e79b3700 1 -- 172.21.15.189:0/3074976542 wait complete. 2024-04-06T13:49:43.005 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' removed 2024-04-06T13:49:43.019 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-06T13:49:43.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.155+0000 7f7759f01700 1 Processor -- start 2024-04-06T13:49:43.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.155+0000 7f7759f01700 1 -- start start 2024-04-06T13:49:43.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.157+0000 7f7759f01700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f775406ed70 0x7f77540b2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:43.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.157+0000 7f7759f01700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f775406da20 0x7f775406deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:43.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.157+0000 7f7759f01700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7754070a20 con 0x7f7754157c50 2024-04-06T13:49:43.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.157+0000 7f7759f01700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f775406e3f0 con 0x7f775406da20 2024-04-06T13:49:43.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.157+0000 7f7759f01700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77540706f0 con 0x7f775406ed70 2024-04-06T13:49:43.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.157+0000 7f77537fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7754157c50 0x7f775415dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46392/0 (socket says 172.21.15.189:46392) 2024-04-06T13:49:43.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.157+0000 7f77537fe700 1 -- 172.21.15.189:0/2437329986 learned_addr learned my addr 172.21.15.189:0/2437329986 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:43.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.157+0000 7f7753fff700 1 --2- 172.21.15.189:0/2437329986 >> v2:172.21.15.192:3300/0 conn(0x7f775406da20 0x7f775406deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:43.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.158+0000 7f7753fff700 1 -- 172.21.15.189:0/2437329986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f775406ed70 msgr2=0x7f77540b2370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-06T13:49:43.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.158+0000 7f7753fff700 1 --2- 172.21.15.189:0/2437329986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f775406ed70 0x7f77540b2370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:43.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.158+0000 7f7753fff700 1 -- 172.21.15.189:0/2437329986 >> v1:172.21.15.189:6789/0 conn(0x7f7754157c50 legacy=0x7f775415dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:43.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.158+0000 7f7753fff700 1 -- 172.21.15.189:0/2437329986 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775415e480 con 0x7f775406da20 2024-04-06T13:49:43.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.158+0000 7f7753fff700 1 --2- 172.21.15.189:0/2437329986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f775406da20 0x7f775406deb0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f774c009d90 tx=0x7f774c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8995f38f00fced02 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:43.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.159+0000 7f77527fc700 1 -- 172.21.15.189:0/2437329986 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774c00e070 con 0x7f775406da20 2024-04-06T13:49:43.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.159+0000 7f77527fc700 1 -- 172.21.15.189:0/2437329986 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f774c00bbd0 con 0x7f775406da20 2024-04-06T13:49:43.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.159+0000 7f77527fc700 1 -- 172.21.15.189:0/2437329986 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774c012430 con 0x7f775406da20 2024-04-06T13:49:43.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.159+0000 7f7759f01700 1 -- 172.21.15.189:0/2437329986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f775406da20 msgr2=0x7f775406deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:43.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.159+0000 7f7759f01700 1 --2- 172.21.15.189:0/2437329986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f775406da20 0x7f775406deb0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f774c009d90 tx=0x7f774c00b330 comp rx=0 tx=0).stop 2024-04-06T13:49:43.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.160+0000 7f7759f01700 1 -- 172.21.15.189:0/2437329986 shutdown_connections 2024-04-06T13:49:43.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.160+0000 7f7759f01700 1 --2- 172.21.15.189:0/2437329986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f775406da20 0x7f775406deb0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:43.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.160+0000 7f7759f01700 1 --2- 172.21.15.189:0/2437329986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f775406ed70 0x7f77540b2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:43.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.160+0000 7f7759f01700 1 -- 172.21.15.189:0/2437329986 >> 172.21.15.189:0/2437329986 conn(0x7f77540cefa0 msgr2=0x7f77540cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:43.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.160+0000 7f7759f01700 1 -- 172.21.15.189:0/2437329986 shutdown_connections 2024-04-06T13:49:43.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.160+0000 7f7759f01700 1 -- 172.21.15.189:0/2437329986 wait complete. 2024-04-06T13:49:43.163 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.162+0000 7f7759f01700 1 Processor -- start 2024-04-06T13:49:43.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.163+0000 7f7759f01700 1 -- start start 2024-04-06T13:49:43.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.163+0000 7f7759f01700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f775406da20 0x7f7754152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:43.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.163+0000 7f7759f01700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f775406ed70 0x7f77541f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:43.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.163+0000 7f7759f01700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7754157c50 0x7f77541f25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:43.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.163+0000 7f7759f01700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775414f0f0 con 0x7f775406ed70 2024-04-06T13:49:43.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.163+0000 7f7759f01700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775414c5e0 con 0x7f775406da20 2024-04-06T13:49:43.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.163+0000 7f7759f01700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f775414ea30 con 0x7f7754157c50 2024-04-06T13:49:43.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.164+0000 7f7752ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f775406ed70 0x7f77541f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:43.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.164+0000 7f7752ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f775406ed70 0x7f77541f20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45008/0 (socket says 172.21.15.189:45008) 2024-04-06T13:49:43.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.164+0000 7f7752ffd700 1 -- 172.21.15.189:0/1838106817 learned_addr learned my addr 172.21.15.189:0/1838106817 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:43.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.164+0000 7f77537fe700 1 --2- 172.21.15.189:0/1838106817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f775406da20 0x7f7754152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:43.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.164+0000 7f7753fff700 1 --2- 172.21.15.189:0/1838106817 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7754157c50 0x7f77541f25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:43.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.164+0000 7f7752ffd700 1 -- 172.21.15.189:0/1838106817 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7754157c50 msgr2=0x7f77541f25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:43.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.164+0000 7f7752ffd700 1 --2- 172.21.15.189:0/1838106817 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7754157c50 0x7f77541f25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:43.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.164+0000 7f7752ffd700 1 -- 172.21.15.189:0/1838106817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f775406da20 msgr2=0x7f7754152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:43.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.164+0000 7f7752ffd700 1 --2- 172.21.15.189:0/1838106817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f775406da20 0x7f7754152950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:43.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.164+0000 7f7752ffd700 1 -- 172.21.15.189:0/1838106817 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f774c009a70 con 0x7f775406ed70 2024-04-06T13:49:43.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.164+0000 7f7753fff700 1 --2- 172.21.15.189:0/1838106817 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7754157c50 0x7f77541f25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:43.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.164+0000 7f7752ffd700 1 --2- 172.21.15.189:0/1838106817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f775406ed70 0x7f77541f20a0 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7f774800b550 tx=0x7f774800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:43.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.164+0000 7f773bfff700 1 -- 172.21.15.189:0/1838106817 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7748011830 con 0x7f775406ed70 2024-04-06T13:49:43.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.164+0000 7f773bfff700 1 -- 172.21.15.189:0/1838106817 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77480119b0 con 0x7f775406ed70 2024-04-06T13:49:43.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.164+0000 7f773bfff700 1 -- 172.21.15.189:0/1838106817 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7748011d60 con 0x7f775406ed70 2024-04-06T13:49:43.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.164+0000 7f7759f01700 1 -- 172.21.15.189:0/1838106817 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f775414ed10 con 0x7f775406ed70 2024-04-06T13:49:43.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.164+0000 7f7759f01700 1 -- 172.21.15.189:0/1838106817 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77541f2b20 con 0x7f775406ed70 2024-04-06T13:49:43.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.165+0000 7f7759f01700 1 -- 172.21.15.189:0/1838106817 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f775414f230 con 0x7f775406ed70 2024-04-06T13:49:43.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.166+0000 7f773bfff700 1 -- 172.21.15.189:0/1838106817 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 168) v1 ==== 89191+0+0 (secure 0 0 0) 0x7f77480193b0 con 0x7f775406ed70 2024-04-06T13:49:43.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.166+0000 7f773bfff700 1 --2- 172.21.15.189:0/1838106817 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f774006ad20 0x7f774006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:43.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.166+0000 7f77537fe700 1 --2- 172.21.15.189:0/1838106817 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f774006ad20 0x7f774006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:43.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.167+0000 7f773bfff700 1 -- 172.21.15.189:0/1838106817 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 6437+0+0 (secure 0 0 0) 0x7f77480892b0 con 0x7f775406ed70 2024-04-06T13:49:43.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.167+0000 7f77537fe700 1 --2- 172.21.15.189:0/1838106817 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f774006ad20 0x7f774006d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7744005910 tx=0x7f7744009040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:43.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.171+0000 7f773bfff700 1 -- 172.21.15.189:0/1838106817 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f775414f230 con 0x7f775406ed70 2024-04-06T13:49:43.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:43.366+0000 7f7759f01700 1 -- 172.21.15.189:0/1838106817 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f775414f230 con 0x7f775406ed70 2024-04-06T13:49:44.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.255+0000 7f773bfff700 1 -- 172.21.15.189:0/1838106817 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7748054620 con 0x7f775406ed70 2024-04-06T13:49:44.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.257+0000 7f773bfff700 1 -- 172.21.15.189:0/1838106817 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v274) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7748054250 con 0x7f775406ed70 2024-04-06T13:49:44.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.262+0000 7f7759f01700 1 -- 172.21.15.189:0/1838106817 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f774006ad20 msgr2=0x7f774006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:44.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.262+0000 7f7759f01700 1 --2- 172.21.15.189:0/1838106817 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f774006ad20 0x7f774006d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7744005910 tx=0x7f7744009040 comp rx=0 tx=0).stop 2024-04-06T13:49:44.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.262+0000 7f7759f01700 1 -- 172.21.15.189:0/1838106817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f775406ed70 msgr2=0x7f77541f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:44.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.262+0000 7f7759f01700 1 --2- 172.21.15.189:0/1838106817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f775406ed70 0x7f77541f20a0 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7f774800b550 tx=0x7f774800b860 comp rx=0 tx=0).stop 2024-04-06T13:49:44.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.262+0000 7f7759f01700 1 -- 172.21.15.189:0/1838106817 shutdown_connections 2024-04-06T13:49:44.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.262+0000 7f7759f01700 1 --2- 172.21.15.189:0/1838106817 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f774006ad20 0x7f774006d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.262+0000 7f7759f01700 1 --2- 172.21.15.189:0/1838106817 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7754157c50 0x7f77541f25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.262+0000 7f7759f01700 1 --2- 172.21.15.189:0/1838106817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f775406da20 0x7f7754152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.262+0000 7f7759f01700 1 --2- 172.21.15.189:0/1838106817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f775406ed70 0x7f77541f20a0 unknown :-1 s=CLOSED pgs=3031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.262+0000 7f7759f01700 1 -- 172.21.15.189:0/1838106817 >> 172.21.15.189:0/1838106817 conn(0x7f77540cefa0 msgr2=0x7f77540b2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:44.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.262+0000 7f7759f01700 1 -- 172.21.15.189:0/1838106817 shutdown_connections 2024-04-06T13:49:44.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.262+0000 7f7759f01700 1 -- 172.21.15.189:0/1838106817 wait complete. 2024-04-06T13:49:44.267 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' removed 2024-04-06T13:49:44.278 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-06T13:49:44.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.405+0000 7fac86651700 1 Processor -- start 2024-04-06T13:49:44.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.406+0000 7fac86651700 1 -- start start 2024-04-06T13:49:44.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.406+0000 7fac86651700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac800cd3e0 0x7fac800cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:44.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.406+0000 7fac86651700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fac8015d7c0 0x7fac8006b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:44.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.406+0000 7fac86651700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac8006ceb0 con 0x7fac8006b6f0 2024-04-06T13:49:44.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.406+0000 7fac86651700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac800cdfa0 con 0x7fac8015d7c0 2024-04-06T13:49:44.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.406+0000 7fac86651700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac8006d750 con 0x7fac800cd3e0 2024-04-06T13:49:44.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.407+0000 7fac84bee700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fac8006b6f0 0x7fac801610f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46396/0 (socket says 172.21.15.189:46396) 2024-04-06T13:49:44.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.407+0000 7fac84bee700 1 -- 172.21.15.189:0/1866269387 learned_addr learned my addr 172.21.15.189:0/1866269387 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:44.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.407+0000 7fac7ffff700 1 --2- 172.21.15.189:0/1866269387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac800cd3e0 0x7fac800cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:44.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.407+0000 7fac7f7fe700 1 --2- 172.21.15.189:0/1866269387 >> v2:172.21.15.192:3300/0 conn(0x7fac8015d7c0 0x7fac8006b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:44.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.407+0000 7fac7effd700 1 -- 172.21.15.189:0/1866269387 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081090798 0 0) 0x7fac8006ceb0 con 0x7fac8006b6f0 2024-04-06T13:49:44.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.407+0000 7fac7effd700 1 -- 172.21.15.189:0/1866269387 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac680035b0 con 0x7fac8006b6f0 2024-04-06T13:49:44.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.407+0000 7fac7ffff700 1 -- 172.21.15.189:0/1866269387 >> v2:172.21.15.192:3300/0 conn(0x7fac8015d7c0 msgr2=0x7fac8006b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:44.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.407+0000 7fac7ffff700 1 --2- 172.21.15.189:0/1866269387 >> v2:172.21.15.192:3300/0 conn(0x7fac8015d7c0 0x7fac8006b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.407+0000 7fac7ffff700 1 -- 172.21.15.189:0/1866269387 >> v1:172.21.15.189:6789/0 conn(0x7fac8006b6f0 legacy=0x7fac801610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:44.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.407+0000 7fac7ffff700 1 -- 172.21.15.189:0/1866269387 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac800cdd00 con 0x7fac800cd3e0 2024-04-06T13:49:44.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.407+0000 7fac7ffff700 1 --2- 172.21.15.189:0/1866269387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac800cd3e0 0x7fac800cd7c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fac74009840 tx=0x7fac74009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c0cdd09b0d46151 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:44.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.407+0000 7fac7effd700 1 -- 172.21.15.189:0/1866269387 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac74011830 con 0x7fac800cd3e0 2024-04-06T13:49:44.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.407+0000 7fac7effd700 1 -- 172.21.15.189:0/1866269387 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac74011990 con 0x7fac800cd3e0 2024-04-06T13:49:44.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.407+0000 7fac7effd700 1 -- 172.21.15.189:0/1866269387 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac74011d20 con 0x7fac800cd3e0 2024-04-06T13:49:44.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.408+0000 7fac86651700 1 -- 172.21.15.189:0/1866269387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac800cd3e0 msgr2=0x7fac800cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:44.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.408+0000 7fac86651700 1 --2- 172.21.15.189:0/1866269387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac800cd3e0 0x7fac800cd7c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fac74009840 tx=0x7fac74009b50 comp rx=0 tx=0).stop 2024-04-06T13:49:44.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.408+0000 7fac86651700 1 -- 172.21.15.189:0/1866269387 shutdown_connections 2024-04-06T13:49:44.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.408+0000 7fac86651700 1 --2- 172.21.15.189:0/1866269387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac800cd3e0 0x7fac800cd7c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.408+0000 7fac86651700 1 --2- 172.21.15.189:0/1866269387 >> v2:172.21.15.192:3300/0 conn(0x7fac8015d7c0 0x7fac8006b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.408+0000 7fac86651700 1 -- 172.21.15.189:0/1866269387 >> 172.21.15.189:0/1866269387 conn(0x7fac800cc290 msgr2=0x7fac800cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:44.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.408+0000 7fac86651700 1 -- 172.21.15.189:0/1866269387 shutdown_connections 2024-04-06T13:49:44.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.408+0000 7fac86651700 1 -- 172.21.15.189:0/1866269387 wait complete. 2024-04-06T13:49:44.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.409+0000 7fac86651700 1 Processor -- start 2024-04-06T13:49:44.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.409+0000 7fac86651700 1 -- start start 2024-04-06T13:49:44.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.410+0000 7fac86651700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac8006b6f0 0x7fac800c5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:44.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.410+0000 7fac86651700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac800cd3e0 0x7fac800c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:44.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.410+0000 7fac86651700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac8015d7c0 0x7fac800b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:44.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.410+0000 7fac86651700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac800c55a0 con 0x7fac8015d7c0 2024-04-06T13:49:44.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.410+0000 7fac86651700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac8015a540 con 0x7fac8006b6f0 2024-04-06T13:49:44.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.410+0000 7fac86651700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac80154650 con 0x7fac800cd3e0 2024-04-06T13:49:44.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.410+0000 7fac84bee700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac8015d7c0 0x7fac800b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:44.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.410+0000 7fac84bee700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac8015d7c0 0x7fac800b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45024/0 (socket says 172.21.15.189:45024) 2024-04-06T13:49:44.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.410+0000 7fac84bee700 1 -- 172.21.15.189:0/1261397451 learned_addr learned my addr 172.21.15.189:0/1261397451 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:44.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.410+0000 7fac7f7fe700 1 --2- 172.21.15.189:0/1261397451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac800cd3e0 0x7fac800c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:44.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.410+0000 7fac7ffff700 1 --2- 172.21.15.189:0/1261397451 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac8006b6f0 0x7fac800c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:44.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.410+0000 7fac84bee700 1 -- 172.21.15.189:0/1261397451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac800cd3e0 msgr2=0x7fac800c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:44.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.410+0000 7fac84bee700 1 --2- 172.21.15.189:0/1261397451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac800cd3e0 0x7fac800c5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.414 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.410+0000 7fac84bee700 1 -- 172.21.15.189:0/1261397451 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac8006b6f0 msgr2=0x7fac800c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:44.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.410+0000 7fac84bee700 1 --2- 172.21.15.189:0/1261397451 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac8006b6f0 0x7fac800c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.410+0000 7fac84bee700 1 -- 172.21.15.189:0/1261397451 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac740094a0 con 0x7fac8015d7c0 2024-04-06T13:49:44.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.411+0000 7fac84bee700 1 --2- 172.21.15.189:0/1261397451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac8015d7c0 0x7fac800b04c0 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 crypto rx=0x7fac7800caa0 tx=0x7fac7800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:44.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.411+0000 7fac7cff9700 1 -- 172.21.15.189:0/1261397451 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7800a9e0 con 0x7fac8015d7c0 2024-04-06T13:49:44.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.411+0000 7fac7cff9700 1 -- 172.21.15.189:0/1261397451 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac7800ab40 con 0x7fac8015d7c0 2024-04-06T13:49:44.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.411+0000 7fac86651700 1 -- 172.21.15.189:0/1261397451 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac80154930 con 0x7fac8015d7c0 2024-04-06T13:49:44.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.411+0000 7fac7cff9700 1 -- 172.21.15.189:0/1261397451 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac78012400 con 0x7fac8015d7c0 2024-04-06T13:49:44.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.411+0000 7fac86651700 1 -- 172.21.15.189:0/1261397451 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac800c5ca0 con 0x7fac8015d7c0 2024-04-06T13:49:44.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.412+0000 7fac86651700 1 -- 172.21.15.189:0/1261397451 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac80154e50 con 0x7fac8015d7c0 2024-04-06T13:49:44.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.412+0000 7fac7cff9700 1 -- 172.21.15.189:0/1261397451 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fac7801b3b0 con 0x7fac8015d7c0 2024-04-06T13:49:44.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.413+0000 7fac7cff9700 1 --2- 172.21.15.189:0/1261397451 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fac6c06adb0 0x7fac6c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:44.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.413+0000 7fac7cff9700 1 -- 172.21.15.189:0/1261397451 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6023+0+0 (secure 0 0 0) 0x7fac780890e0 con 0x7fac8015d7c0 2024-04-06T13:49:44.415 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.413+0000 7fac7ffff700 1 --2- 172.21.15.189:0/1261397451 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fac6c06adb0 0x7fac6c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:44.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.413+0000 7fac7ffff700 1 --2- 172.21.15.189:0/1261397451 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fac6c06adb0 0x7fac6c06d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fac74000c00 tx=0x7fac7401c040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:44.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.417+0000 7fac7cff9700 1 -- 172.21.15.189:0/1261397451 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fac80154e50 con 0x7fac8015d7c0 2024-04-06T13:49:44.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.613+0000 7fac86651700 1 -- 172.21.15.189:0/1261397451 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fac80154e50 con 0x7fac8015d7c0 2024-04-06T13:49:44.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.617+0000 7fac7cff9700 1 -- 172.21.15.189:0/1261397451 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v91) v1 ==== 104+0+0 (secure 0 0 0) 0x7fac80154e50 con 0x7fac8015d7c0 2024-04-06T13:49:44.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.619+0000 7fac86651700 1 -- 172.21.15.189:0/1261397451 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fac6c06adb0 msgr2=0x7fac6c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:44.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.619+0000 7fac86651700 1 --2- 172.21.15.189:0/1261397451 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fac6c06adb0 0x7fac6c06d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fac74000c00 tx=0x7fac7401c040 comp rx=0 tx=0).stop 2024-04-06T13:49:44.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.619+0000 7fac86651700 1 -- 172.21.15.189:0/1261397451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac8015d7c0 msgr2=0x7fac800b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:44.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.619+0000 7fac86651700 1 --2- 172.21.15.189:0/1261397451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac8015d7c0 0x7fac800b04c0 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 crypto rx=0x7fac7800caa0 tx=0x7fac7800cdb0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.619+0000 7fac86651700 1 -- 172.21.15.189:0/1261397451 shutdown_connections 2024-04-06T13:49:44.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.619+0000 7fac86651700 1 --2- 172.21.15.189:0/1261397451 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fac6c06adb0 0x7fac6c06d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.619+0000 7fac86651700 1 --2- 172.21.15.189:0/1261397451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac8015d7c0 0x7fac800b04c0 unknown :-1 s=CLOSED pgs=3033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.619+0000 7fac86651700 1 --2- 172.21.15.189:0/1261397451 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac8006b6f0 0x7fac800c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.619+0000 7fac86651700 1 --2- 172.21.15.189:0/1261397451 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac800cd3e0 0x7fac800c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.619+0000 7fac86651700 1 -- 172.21.15.189:0/1261397451 >> 172.21.15.189:0/1261397451 conn(0x7fac800cc290 msgr2=0x7fac800aeae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:44.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.619+0000 7fac86651700 1 -- 172.21.15.189:0/1261397451 shutdown_connections 2024-04-06T13:49:44.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.619+0000 7fac86651700 1 -- 172.21.15.189:0/1261397451 wait complete. 2024-04-06T13:49:44.632 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-06T13:49:44.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.755+0000 7f3079de3700 1 Processor -- start 2024-04-06T13:49:44.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.755+0000 7f3079de3700 1 -- start start 2024-04-06T13:49:44.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.756+0000 7f3079de3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f30740d11b0 0x7f30740d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:44.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.756+0000 7f3079de3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f30740cb870 0x7f307415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:44.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.756+0000 7f3079de3700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30740d1ad0 con 0x7f30740cfa00 2024-04-06T13:49:44.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.756+0000 7f3079de3700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30740ccdc0 con 0x7f30740d11b0 2024-04-06T13:49:44.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.756+0000 7f3079de3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30740d0b50 con 0x7f30740cb870 2024-04-06T13:49:44.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.756+0000 7f30737fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f30740d11b0 0x7f30740d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:44.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.756+0000 7f30737fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f30740d11b0 0x7f30740d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45038/0 (socket says 172.21.15.189:45038) 2024-04-06T13:49:44.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.756+0000 7f30737fe700 1 -- 172.21.15.189:0/540536616 learned_addr learned my addr 172.21.15.189:0/540536616 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:44.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.756+0000 7f30737fe700 1 -- 172.21.15.189:0/540536616 >> v2:172.21.15.192:3300/0 conn(0x7f30740cb870 msgr2=0x7f307415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:44.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.756+0000 7f3072ffd700 1 --2- 172.21.15.189:0/540536616 >> v2:172.21.15.192:3300/0 conn(0x7f30740cb870 0x7f307415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:44.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.756+0000 7f30737fe700 1 --2- 172.21.15.189:0/540536616 >> v2:172.21.15.192:3300/0 conn(0x7f30740cb870 0x7f307415c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.756+0000 7f30737fe700 1 -- 172.21.15.189:0/540536616 >> v1:172.21.15.192:6789/0 conn(0x7f30740cfa00 legacy=0x7f30740cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:44.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.756+0000 7f30737fe700 1 -- 172.21.15.189:0/540536616 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30740cbd80 con 0x7f30740d11b0 2024-04-06T13:49:44.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.756+0000 7f30737fe700 1 --2- 172.21.15.189:0/540536616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30740d11b0 0x7f30740d1590 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 crypto rx=0x7f307404fb40 tx=0x7f3064009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa38c4a84115c12 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:44.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.756+0000 7f30727fc700 1 -- 172.21.15.189:0/540536616 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064004230 con 0x7f30740d11b0 2024-04-06T13:49:44.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.756+0000 7f30727fc700 1 -- 172.21.15.189:0/540536616 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3064004390 con 0x7f30740d11b0 2024-04-06T13:49:44.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.756+0000 7f30727fc700 1 -- 172.21.15.189:0/540536616 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30640036a0 con 0x7f30740d11b0 2024-04-06T13:49:44.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.757+0000 7f3079de3700 1 -- 172.21.15.189:0/540536616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30740d11b0 msgr2=0x7f30740d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:44.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.757+0000 7f3079de3700 1 --2- 172.21.15.189:0/540536616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30740d11b0 0x7f30740d1590 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 crypto rx=0x7f307404fb40 tx=0x7f3064009bd0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.757+0000 7f3079de3700 1 -- 172.21.15.189:0/540536616 shutdown_connections 2024-04-06T13:49:44.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.757+0000 7f3079de3700 1 --2- 172.21.15.189:0/540536616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30740d11b0 0x7f30740d1590 unknown :-1 s=CLOSED pgs=3034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.757+0000 7f3079de3700 1 --2- 172.21.15.189:0/540536616 >> v2:172.21.15.192:3300/0 conn(0x7f30740cb870 0x7f307415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.757+0000 7f3079de3700 1 -- 172.21.15.189:0/540536616 >> 172.21.15.189:0/540536616 conn(0x7f30740b2db0 msgr2=0x7f30740b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:44.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.757+0000 7f3079de3700 1 -- 172.21.15.189:0/540536616 shutdown_connections 2024-04-06T13:49:44.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.757+0000 7f3079de3700 1 -- 172.21.15.189:0/540536616 wait complete. 2024-04-06T13:49:44.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.758+0000 7f3079de3700 1 Processor -- start 2024-04-06T13:49:44.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.758+0000 7f3079de3700 1 -- start start 2024-04-06T13:49:44.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.759+0000 7f3079de3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f30740cb870 0x7f30741f2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:44.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.759+0000 7f3079de3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30740cfa00 0x7f30740b6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:44.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.759+0000 7f3079de3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30740d11b0 0x7f30740b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:44.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.759+0000 7f3079de3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30740b5230 con 0x7f30740cfa00 2024-04-06T13:49:44.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.759+0000 7f3079de3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30740b68c0 con 0x7f30740d11b0 2024-04-06T13:49:44.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.759+0000 7f3079de3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30740b4450 con 0x7f30740cb870 2024-04-06T13:49:44.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.759+0000 7f3072ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30740cfa00 0x7f30740b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:44.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.759+0000 7f3072ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30740cfa00 0x7f30740b6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45046/0 (socket says 172.21.15.189:45046) 2024-04-06T13:49:44.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.759+0000 7f3072ffd700 1 -- 172.21.15.189:0/2438011623 learned_addr learned my addr 172.21.15.189:0/2438011623 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:44.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.759+0000 7f3073fff700 1 --2- 172.21.15.189:0/2438011623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30740d11b0 0x7f30740b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:44.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.760+0000 7f3072ffd700 1 -- 172.21.15.189:0/2438011623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f30740cb870 msgr2=0x7f30741f2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:44.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.760+0000 7f3072ffd700 1 --2- 172.21.15.189:0/2438011623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f30740cb870 0x7f30741f2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.760+0000 7f3072ffd700 1 -- 172.21.15.189:0/2438011623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30740d11b0 msgr2=0x7f30740b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:44.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.760+0000 7f3072ffd700 1 --2- 172.21.15.189:0/2438011623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30740d11b0 0x7f30740b3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.760+0000 7f3072ffd700 1 -- 172.21.15.189:0/2438011623 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3064009550 con 0x7f30740cfa00 2024-04-06T13:49:44.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.760+0000 7f3072ffd700 1 --2- 172.21.15.189:0/2438011623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30740cfa00 0x7f30740b6380 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7f306800c7f0 tx=0x7f306800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:44.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.760+0000 7f305bfff700 1 -- 172.21.15.189:0/2438011623 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3068011800 con 0x7f30740cfa00 2024-04-06T13:49:44.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.760+0000 7f305bfff700 1 -- 172.21.15.189:0/2438011623 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3068011960 con 0x7f30740cfa00 2024-04-06T13:49:44.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.760+0000 7f3079de3700 1 -- 172.21.15.189:0/2438011623 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30740c58b0 con 0x7f30740cfa00 2024-04-06T13:49:44.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.760+0000 7f305bfff700 1 -- 172.21.15.189:0/2438011623 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3068011d10 con 0x7f30740cfa00 2024-04-06T13:49:44.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.760+0000 7f3079de3700 1 -- 172.21.15.189:0/2438011623 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30740b4990 con 0x7f30740cfa00 2024-04-06T13:49:44.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.761+0000 7f3079de3700 1 -- 172.21.15.189:0/2438011623 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3074074990 con 0x7f30740cfa00 2024-04-06T13:49:44.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.761+0000 7f305bfff700 1 -- 172.21.15.189:0/2438011623 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f3068018360 con 0x7f30740cfa00 2024-04-06T13:49:44.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.762+0000 7f305bfff700 1 --2- 172.21.15.189:0/2438011623 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f306006adb0 0x7f306006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:44.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.762+0000 7f305bfff700 1 -- 172.21.15.189:0/2438011623 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6023+0+0 (secure 0 0 0) 0x7f3068089700 con 0x7f30740cfa00 2024-04-06T13:49:44.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.762+0000 7f30737fe700 1 --2- 172.21.15.189:0/2438011623 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f306006adb0 0x7f306006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:44.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.762+0000 7f30737fe700 1 --2- 172.21.15.189:0/2438011623 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f306006adb0 0x7f306006d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3064000c00 tx=0x7f3064010000 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:44.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.766+0000 7f305bfff700 1 -- 172.21.15.189:0/2438011623 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3068054460 con 0x7f30740cfa00 2024-04-06T13:49:44.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.967+0000 7f3079de3700 1 -- 172.21.15.189:0/2438011623 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f30740c5a40 con 0x7f30740cfa00 2024-04-06T13:49:44.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.971+0000 7f305bfff700 1 -- 172.21.15.189:0/2438011623 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v92) v1 ==== 115+0+0 (secure 0 0 0) 0x7f3068057a80 con 0x7f30740cfa00 2024-04-06T13:49:44.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.973+0000 7f3079de3700 1 -- 172.21.15.189:0/2438011623 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f306006adb0 msgr2=0x7f306006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:44.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.973+0000 7f3079de3700 1 --2- 172.21.15.189:0/2438011623 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f306006adb0 0x7f306006d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3064000c00 tx=0x7f3064010000 comp rx=0 tx=0).stop 2024-04-06T13:49:44.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.973+0000 7f3079de3700 1 -- 172.21.15.189:0/2438011623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30740cfa00 msgr2=0x7f30740b6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:44.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.973+0000 7f3079de3700 1 --2- 172.21.15.189:0/2438011623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30740cfa00 0x7f30740b6380 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7f306800c7f0 tx=0x7f306800cbb0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.973+0000 7f3079de3700 1 -- 172.21.15.189:0/2438011623 shutdown_connections 2024-04-06T13:49:44.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.973+0000 7f3079de3700 1 --2- 172.21.15.189:0/2438011623 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f306006adb0 0x7f306006d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.973+0000 7f3079de3700 1 --2- 172.21.15.189:0/2438011623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30740d11b0 0x7f30740b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.973+0000 7f3079de3700 1 --2- 172.21.15.189:0/2438011623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f30740cb870 0x7f30741f2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.973+0000 7f3079de3700 1 --2- 172.21.15.189:0/2438011623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30740cfa00 0x7f30740b6380 unknown :-1 s=CLOSED pgs=3035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:44.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.973+0000 7f3079de3700 1 -- 172.21.15.189:0/2438011623 >> 172.21.15.189:0/2438011623 conn(0x7f30740b2db0 msgr2=0x7f30740b1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:44.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.973+0000 7f3079de3700 1 -- 172.21.15.189:0/2438011623 shutdown_connections 2024-04-06T13:49:44.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:44.973+0000 7f3079de3700 1 -- 172.21.15.189:0/2438011623 wait complete. 2024-04-06T13:49:44.987 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-06T13:49:45.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.111+0000 7f1439f0b700 1 Processor -- start 2024-04-06T13:49:45.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.111+0000 7f1439f0b700 1 -- start start 2024-04-06T13:49:45.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.112+0000 7f1439f0b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1434159b20 0x7f143406b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:45.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.112+0000 7f1439f0b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f143406eb80 0x7f14340ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:45.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.112+0000 7f1439f0b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f143406a8d0 con 0x7f14340cdc60 2024-04-06T13:49:45.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.112+0000 7f1439f0b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14340ce650 con 0x7f1434159b20 2024-04-06T13:49:45.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.112+0000 7f1439f0b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1434159f00 con 0x7f143406eb80 2024-04-06T13:49:45.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.112+0000 7f1432ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f14340cdc60 0x7f143406e470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46404/0 (socket says 172.21.15.189:46404) 2024-04-06T13:49:45.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.112+0000 7f1432ffd700 1 -- 172.21.15.189:0/4244379294 learned_addr learned my addr 172.21.15.189:0/4244379294 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:45.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f14337fe700 1 --2- 172.21.15.189:0/4244379294 >> v2:172.21.15.192:3300/0 conn(0x7f1434159b20 0x7f143406b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:45.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f1433fff700 1 --2- 172.21.15.189:0/4244379294 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f143406eb80 0x7f14340ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:45.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f14327fc700 1 -- 172.21.15.189:0/4244379294 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615795781 0 0) 0x7f143406a8d0 con 0x7f14340cdc60 2024-04-06T13:49:45.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f14327fc700 1 -- 172.21.15.189:0/4244379294 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14200035b0 con 0x7f14340cdc60 2024-04-06T13:49:45.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f14327fc700 1 -- 172.21.15.189:0/4244379294 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2082587494 0 0) 0x7f14200035b0 con 0x7f14340cdc60 2024-04-06T13:49:45.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f14327fc700 1 -- 172.21.15.189:0/4244379294 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f143406eb80 msgr2=0x7f14340ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:45.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f14327fc700 1 --2- 172.21.15.189:0/4244379294 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f143406eb80 0x7f14340ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:45.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f14327fc700 1 -- 172.21.15.189:0/4244379294 >> v2:172.21.15.192:3300/0 conn(0x7f1434159b20 msgr2=0x7f143406b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:45.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f14327fc700 1 --2- 172.21.15.189:0/4244379294 >> v2:172.21.15.192:3300/0 conn(0x7f1434159b20 0x7f143406b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:45.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f14327fc700 1 -- 172.21.15.189:0/4244379294 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143406abb0 con 0x7f14340cdc60 2024-04-06T13:49:45.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f14327fc700 1 -- 172.21.15.189:0/4244379294 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f141c001670 con 0x7f14340cdc60 2024-04-06T13:49:45.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f14327fc700 1 -- 172.21.15.189:0/4244379294 >> v1:172.21.15.189:6789/0 conn(0x7f14340cdc60 legacy=0x7f143406e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:45.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f14327fc700 1 --2- 172.21.15.189:0/4244379294 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f143406eb80 0x7f1420003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:45.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f14327fc700 1 --2- 172.21.15.189:0/4244379294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1420003fb0 0x7f1420004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:45.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f14327fc700 1 --2- 172.21.15.189:0/4244379294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1420004d90 0x7f1420005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:45.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f14327fc700 1 -- 172.21.15.189:0/4244379294 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14340ce650 con 0x7f1420003fb0 2024-04-06T13:49:45.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f14327fc700 1 -- 172.21.15.189:0/4244379294 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1434159f00 con 0x7f1420004d90 2024-04-06T13:49:45.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f14327fc700 1 -- 172.21.15.189:0/4244379294 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f142000cba0 con 0x7f143406eb80 2024-04-06T13:49:45.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f1432ffd700 1 --2- 172.21.15.189:0/4244379294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1420003fb0 0x7f1420004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:45.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f14337fe700 1 --2- 172.21.15.189:0/4244379294 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f143406eb80 0x7f1420003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:45.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f1433fff700 1 --2- 172.21.15.189:0/4244379294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1420004d90 0x7f1420005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:45.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f1432ffd700 1 -- 172.21.15.189:0/4244379294 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f143406eb80 msgr2=0x7f1420003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:45.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f1432ffd700 1 --2- 172.21.15.189:0/4244379294 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f143406eb80 0x7f1420003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:45.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f1432ffd700 1 -- 172.21.15.189:0/4244379294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1420004d90 msgr2=0x7f1420005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:45.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f1432ffd700 1 --2- 172.21.15.189:0/4244379294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1420004d90 0x7f1420005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:45.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.113+0000 7f1432ffd700 1 -- 172.21.15.189:0/4244379294 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14200035b0 con 0x7f1420003fb0 2024-04-06T13:49:45.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.114+0000 7f1432ffd700 1 --2- 172.21.15.189:0/4244379294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1420003fb0 0x7f1420004850 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 crypto rx=0x7f141c007d30 tx=0x7f1420005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b9d1db481b6beb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:45.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.114+0000 7f14327fc700 1 -- 172.21.15.189:0/4244379294 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141c00cdb0 con 0x7f1420003fb0 2024-04-06T13:49:45.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.114+0000 7f14327fc700 1 -- 172.21.15.189:0/4244379294 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141c005960 con 0x7f1420003fb0 2024-04-06T13:49:45.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.114+0000 7f14327fc700 1 -- 172.21.15.189:0/4244379294 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141c005d10 con 0x7f1420003fb0 2024-04-06T13:49:45.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.114+0000 7f1439f0b700 1 -- 172.21.15.189:0/4244379294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1420003fb0 msgr2=0x7f1420004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:45.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.114+0000 7f1439f0b700 1 --2- 172.21.15.189:0/4244379294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1420003fb0 0x7f1420004850 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 crypto rx=0x7f141c007d30 tx=0x7f1420005880 comp rx=0 tx=0).stop 2024-04-06T13:49:45.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.114+0000 7f1433fff700 1 -- 172.21.15.189:0/4244379294 reap_dead start 2024-04-06T13:49:45.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.114+0000 7f1439f0b700 1 -- 172.21.15.189:0/4244379294 shutdown_connections 2024-04-06T13:49:45.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.114+0000 7f1439f0b700 1 -- 172.21.15.189:0/4244379294 >> 172.21.15.189:0/4244379294 conn(0x7f14340cc290 msgr2=0x7f14340cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:45.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.114+0000 7f1439f0b700 1 -- 172.21.15.189:0/4244379294 shutdown_connections 2024-04-06T13:49:45.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.114+0000 7f1439f0b700 1 -- 172.21.15.189:0/4244379294 wait complete. 2024-04-06T13:49:45.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.115+0000 7f1439f0b700 1 Processor -- start 2024-04-06T13:49:45.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.116+0000 7f1439f0b700 1 -- start start 2024-04-06T13:49:45.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.116+0000 7f1439f0b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1420003fb0 0x7f14340ad170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:45.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.116+0000 7f1439f0b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14341e3dd0 0x7f14341f2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:45.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.116+0000 7f1439f0b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14340b46e0 0x7f14340ae090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:45.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.116+0000 7f1439f0b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14340acd50 con 0x7f14340b46e0 2024-04-06T13:49:45.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.116+0000 7f1439f0b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14341f37d0 con 0x7f14341e3dd0 2024-04-06T13:49:45.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.116+0000 7f1439f0b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14341f3ab0 con 0x7f1420003fb0 2024-04-06T13:49:45.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.117+0000 7f1433fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14340b46e0 0x7f14340ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:45.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.117+0000 7f1433fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14340b46e0 0x7f14340ae090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45068/0 (socket says 172.21.15.189:45068) 2024-04-06T13:49:45.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.117+0000 7f1433fff700 1 -- 172.21.15.189:0/1571217768 learned_addr learned my addr 172.21.15.189:0/1571217768 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:45.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.117+0000 7f14337fe700 1 --2- 172.21.15.189:0/1571217768 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1420003fb0 0x7f14340ad170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:45.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.117+0000 7f1432ffd700 1 --2- 172.21.15.189:0/1571217768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14341e3dd0 0x7f14341f2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:45.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.117+0000 7f1433fff700 1 -- 172.21.15.189:0/1571217768 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1420003fb0 msgr2=0x7f14340ad170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:45.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.117+0000 7f1433fff700 1 --2- 172.21.15.189:0/1571217768 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1420003fb0 0x7f14340ad170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:45.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.117+0000 7f1433fff700 1 -- 172.21.15.189:0/1571217768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14341e3dd0 msgr2=0x7f14341f2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:45.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.117+0000 7f1433fff700 1 --2- 172.21.15.189:0/1571217768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14341e3dd0 0x7f14341f2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:45.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.117+0000 7f1433fff700 1 -- 172.21.15.189:0/1571217768 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14341f3d90 con 0x7f14340b46e0 2024-04-06T13:49:45.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.117+0000 7f1432ffd700 1 --2- 172.21.15.189:0/1571217768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14341e3dd0 0x7f14341f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:45.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.117+0000 7f1433fff700 1 --2- 172.21.15.189:0/1571217768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14340b46e0 0x7f14340ae090 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7f14340b5740 tx=0x7f142c011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:45.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.117+0000 7f142bfff700 1 -- 172.21.15.189:0/1571217768 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142c00e040 con 0x7f14340b46e0 2024-04-06T13:49:45.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.117+0000 7f142bfff700 1 -- 172.21.15.189:0/1571217768 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142c00f040 con 0x7f14340b46e0 2024-04-06T13:49:45.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.117+0000 7f1439f0b700 1 -- 172.21.15.189:0/1571217768 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14340ac970 con 0x7f14340b46e0 2024-04-06T13:49:45.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.117+0000 7f142bfff700 1 -- 172.21.15.189:0/1571217768 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142c00fa00 con 0x7f14340b46e0 2024-04-06T13:49:45.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.117+0000 7f1439f0b700 1 -- 172.21.15.189:0/1571217768 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14341f33f0 con 0x7f14340b46e0 2024-04-06T13:49:45.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.118+0000 7f1439f0b700 1 -- 172.21.15.189:0/1571217768 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14340ace90 con 0x7f14340b46e0 2024-04-06T13:49:45.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.119+0000 7f142bfff700 1 -- 172.21.15.189:0/1571217768 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f142c00fb60 con 0x7f14340b46e0 2024-04-06T13:49:45.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.119+0000 7f142bfff700 1 --2- 172.21.15.189:0/1571217768 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f141406adb0 0x7f141406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:45.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.119+0000 7f142bfff700 1 -- 172.21.15.189:0/1571217768 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6023+0+0 (secure 0 0 0) 0x7f142c089ce0 con 0x7f14340b46e0 2024-04-06T13:49:45.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.119+0000 7f14337fe700 1 --2- 172.21.15.189:0/1571217768 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f141406adb0 0x7f141406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:45.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.119+0000 7f14337fe700 1 --2- 172.21.15.189:0/1571217768 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f141406adb0 0x7f141406d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1424009ba0 tx=0x7f1424008040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:45.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.123+0000 7f142bfff700 1 -- 172.21.15.189:0/1571217768 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f14340ace90 con 0x7f14340b46e0 2024-04-06T13:49:45.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.320+0000 7f1439f0b700 1 -- 172.21.15.189:0/1571217768 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f14340ace90 con 0x7f14340b46e0 2024-04-06T13:49:45.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.324+0000 7f142bfff700 1 -- 172.21.15.189:0/1571217768 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v93) v1 ==== 120+0+0 (secure 0 0 0) 0x7f14340ace90 con 0x7f14340b46e0 2024-04-06T13:49:45.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.326+0000 7f1439f0b700 1 -- 172.21.15.189:0/1571217768 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f141406adb0 msgr2=0x7f141406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:45.327 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.326+0000 7f1439f0b700 1 --2- 172.21.15.189:0/1571217768 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f141406adb0 0x7f141406d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1424009ba0 tx=0x7f1424008040 comp rx=0 tx=0).stop 2024-04-06T13:49:45.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.326+0000 7f1439f0b700 1 -- 172.21.15.189:0/1571217768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14340b46e0 msgr2=0x7f14340ae090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:45.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.326+0000 7f1439f0b700 1 --2- 172.21.15.189:0/1571217768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14340b46e0 0x7f14340ae090 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7f14340b5740 tx=0x7f142c011f80 comp rx=0 tx=0).stop 2024-04-06T13:49:45.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.326+0000 7f1439f0b700 1 -- 172.21.15.189:0/1571217768 shutdown_connections 2024-04-06T13:49:45.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.326+0000 7f1439f0b700 1 --2- 172.21.15.189:0/1571217768 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f141406adb0 0x7f141406d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:45.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.326+0000 7f1439f0b700 1 --2- 172.21.15.189:0/1571217768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14340b46e0 0x7f14340ae090 unknown :-1 s=CLOSED pgs=3038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:45.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.326+0000 7f1439f0b700 1 --2- 172.21.15.189:0/1571217768 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1420003fb0 0x7f14340ad170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:45.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.326+0000 7f1439f0b700 1 --2- 172.21.15.189:0/1571217768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14341e3dd0 0x7f14341f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:45.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.326+0000 7f1439f0b700 1 -- 172.21.15.189:0/1571217768 >> 172.21.15.189:0/1571217768 conn(0x7f14340cc290 msgr2=0x7f14340cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:45.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.326+0000 7f1439f0b700 1 -- 172.21.15.189:0/1571217768 shutdown_connections 2024-04-06T13:49:45.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.326+0000 7f1439f0b700 1 -- 172.21.15.189:0/1571217768 wait complete. 2024-04-06T13:49:45.340 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2024-04-06T13:49:45.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.460+0000 7fdfaf74d700 1 Processor -- start 2024-04-06T13:49:45.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.461+0000 7fdfaf74d700 1 -- start start 2024-04-06T13:49:45.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.461+0000 7fdfaf74d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa80aec70 0x7fdfa80cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:45.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.461+0000 7fdfaf74d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdfa80cdae0 0x7fdfa80ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:45.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.461+0000 7fdfaf74d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdfa80ce8b0 0x7fdfa815df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:45.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.461+0000 7fdfaf74d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa80cd920 con 0x7fdfa80cdae0 2024-04-06T13:49:45.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.461+0000 7fdfaf74d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa80ced20 con 0x7fdfa80ce8b0 2024-04-06T13:49:45.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.461+0000 7fdfaf74d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa815a830 con 0x7fdfa80aec70 2024-04-06T13:49:45.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.462+0000 7fdfacce8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdfa80cdae0 0x7fdfa80ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:45.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.462+0000 7fdfacce8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdfa80cdae0 0x7fdfa80ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45074/0 (socket says 172.21.15.189:45074) 2024-04-06T13:49:45.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.462+0000 7fdfacce8700 1 -- 172.21.15.189:0/3582288784 learned_addr learned my addr 172.21.15.189:0/3582288784 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:45.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.462+0000 7fdfad4e9700 1 --2- 172.21.15.189:0/3582288784 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa80aec70 0x7fdfa80cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:45.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.462+0000 7fdfadcea700 1 --2- 172.21.15.189:0/3582288784 >> v2:172.21.15.192:3300/0 conn(0x7fdfa80ce8b0 0x7fdfa815df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:45.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.462+0000 7fdfacce8700 1 -- 172.21.15.189:0/3582288784 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa80aec70 msgr2=0x7fdfa80cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:45.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.462+0000 7fdfacce8700 1 --2- 172.21.15.189:0/3582288784 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa80aec70 0x7fdfa80cd3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:45.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.462+0000 7fdfacce8700 1 -- 172.21.15.189:0/3582288784 >> v2:172.21.15.192:3300/0 conn(0x7fdfa80ce8b0 msgr2=0x7fdfa815df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:45.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.462+0000 7fdfacce8700 1 --2- 172.21.15.189:0/3582288784 >> v2:172.21.15.192:3300/0 conn(0x7fdfa80ce8b0 0x7fdfa815df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:45.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.462+0000 7fdfacce8700 1 -- 172.21.15.189:0/3582288784 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa80cc050 con 0x7fdfa80cdae0 2024-04-06T13:49:45.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.462+0000 7fdfad4e9700 1 --2- 172.21.15.189:0/3582288784 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa80aec70 0x7fdfa80cd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:45.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.462+0000 7fdfacce8700 1 --2- 172.21.15.189:0/3582288784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa80cdae0 0x7fdfa80ce370 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7fdfa80727f0 tx=0x7fdfa4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f3438a9be17c0cd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:45.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.462+0000 7fdf9ffff700 1 -- 172.21.15.189:0/3582288784 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa400b870 con 0x7fdfa80cdae0 2024-04-06T13:49:45.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.462+0000 7fdf9ffff700 1 -- 172.21.15.189:0/3582288784 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa400b9d0 con 0x7fdfa80cdae0 2024-04-06T13:49:45.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.462+0000 7fdf9ffff700 1 -- 172.21.15.189:0/3582288784 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa400bd60 con 0x7fdfa80cdae0 2024-04-06T13:49:45.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.463+0000 7fdfaf74d700 1 -- 172.21.15.189:0/3582288784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa80cdae0 msgr2=0x7fdfa80ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:45.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.463+0000 7fdfaf74d700 1 --2- 172.21.15.189:0/3582288784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa80cdae0 0x7fdfa80ce370 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7fdfa80727f0 tx=0x7fdfa4009bd0 comp rx=0 tx=0).stop 2024-04-06T13:49:45.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.463+0000 7fdfaf74d700 1 -- 172.21.15.189:0/3582288784 shutdown_connections 2024-04-06T13:49:45.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.463+0000 7fdfaf74d700 1 --2- 172.21.15.189:0/3582288784 >> v2:172.21.15.192:3300/0 conn(0x7fdfa80ce8b0 0x7fdfa815df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:45.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.463+0000 7fdfaf74d700 1 --2- 172.21.15.189:0/3582288784 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa80aec70 0x7fdfa80cd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:45.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.463+0000 7fdfaf74d700 1 --2- 172.21.15.189:0/3582288784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa80cdae0 0x7fdfa80ce370 unknown :-1 s=CLOSED pgs=3039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:45.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.463+0000 7fdfaf74d700 1 -- 172.21.15.189:0/3582288784 >> 172.21.15.189:0/3582288784 conn(0x7fdfa80cc290 msgr2=0x7fdfa80cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:45.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.463+0000 7fdfaf74d700 1 -- 172.21.15.189:0/3582288784 shutdown_connections 2024-04-06T13:49:45.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.463+0000 7fdfaf74d700 1 -- 172.21.15.189:0/3582288784 wait complete. 2024-04-06T13:49:45.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.464+0000 7fdfaf74d700 1 Processor -- start 2024-04-06T13:49:45.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.464+0000 7fdfaf74d700 1 -- start start 2024-04-06T13:49:45.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdfaf74d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa80aec70 0x7fdfa80a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:45.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdfaf74d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa80cdae0 0x7fdfa80c4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:45.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdfaf74d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfa80ce8b0 0x7fdfa80c5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:45.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdfaf74d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa80c7f30 con 0x7fdfa80aec70 2024-04-06T13:49:45.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdfaf74d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa80c6980 con 0x7fdfa80ce8b0 2024-04-06T13:49:45.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdfaf74d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa80c6c60 con 0x7fdfa80cdae0 2024-04-06T13:49:45.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdfad4e9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa80aec70 0x7fdfa80a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:45.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdfad4e9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa80aec70 0x7fdfa80a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45090/0 (socket says 172.21.15.189:45090) 2024-04-06T13:49:45.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdfad4e9700 1 -- 172.21.15.189:0/3356855876 learned_addr learned my addr 172.21.15.189:0/3356855876 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:45.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdfadcea700 1 --2- 172.21.15.189:0/3356855876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfa80ce8b0 0x7fdfa80c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:45.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdfacce8700 1 --2- 172.21.15.189:0/3356855876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa80cdae0 0x7fdfa80c4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:45.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdfad4e9700 1 -- 172.21.15.189:0/3356855876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa80cdae0 msgr2=0x7fdfa80c4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:45.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdfad4e9700 1 --2- 172.21.15.189:0/3356855876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa80cdae0 0x7fdfa80c4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:45.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdfad4e9700 1 -- 172.21.15.189:0/3356855876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfa80ce8b0 msgr2=0x7fdfa80c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:45.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdfad4e9700 1 --2- 172.21.15.189:0/3356855876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfa80ce8b0 0x7fdfa80c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:45.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdfad4e9700 1 -- 172.21.15.189:0/3356855876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa4009550 con 0x7fdfa80aec70 2024-04-06T13:49:45.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdfadcea700 1 --2- 172.21.15.189:0/3356855876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfa80ce8b0 0x7fdfa80c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:45.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdfad4e9700 1 --2- 172.21.15.189:0/3356855876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa80aec70 0x7fdfa80a84c0 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 crypto rx=0x7fdf9800d800 tx=0x7fdf9800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:45.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdf9dffb700 1 -- 172.21.15.189:0/3356855876 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf98004030 con 0x7fdfa80aec70 2024-04-06T13:49:45.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdf9dffb700 1 -- 172.21.15.189:0/3356855876 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf98004190 con 0x7fdfa80aec70 2024-04-06T13:49:45.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdfaf74d700 1 -- 172.21.15.189:0/3356855876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa80c6f40 con 0x7fdfa80aec70 2024-04-06T13:49:45.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.465+0000 7fdf9dffb700 1 -- 172.21.15.189:0/3356855876 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf98004520 con 0x7fdfa80aec70 2024-04-06T13:49:45.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.466+0000 7fdfaf74d700 1 -- 172.21.15.189:0/3356855876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa80c7500 con 0x7fdfa80aec70 2024-04-06T13:49:45.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.466+0000 7fdfaf74d700 1 -- 172.21.15.189:0/3356855876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa80c7870 con 0x7fdfa80aec70 2024-04-06T13:49:45.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.467+0000 7fdf9dffb700 1 -- 172.21.15.189:0/3356855876 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fdf980036a0 con 0x7fdfa80aec70 2024-04-06T13:49:45.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.467+0000 7fdf9dffb700 1 --2- 172.21.15.189:0/3356855876 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdf8c06adb0 0x7fdf8c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:45.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.467+0000 7fdf9dffb700 1 -- 172.21.15.189:0/3356855876 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6023+0+0 (secure 0 0 0) 0x7fdf98088b70 con 0x7fdfa80aec70 2024-04-06T13:49:45.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.467+0000 7fdfacce8700 1 --2- 172.21.15.189:0/3356855876 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdf8c06adb0 0x7fdf8c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:45.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.468+0000 7fdfacce8700 1 --2- 172.21.15.189:0/3356855876 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdf8c06adb0 0x7fdf8c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4000c00 tx=0x7fdfa4008030 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:45.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.472+0000 7fdf9dffb700 1 -- 172.21.15.189:0/3356855876 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdf98053c40 con 0x7fdfa80aec70 2024-04-06T13:49:45.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:45.748+0000 7fdfaf74d700 1 -- 172.21.15.189:0/3356855876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7fdfa8074990 con 0x7fdfa80aec70 2024-04-06T13:49:46.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.252+0000 7fdf9dffb700 1 -- 172.21.15.189:0/3356855876 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3234) v1 ==== 155+0+0 (secure 0 0 0) 0x7fdf98018020 con 0x7fdfa80aec70 2024-04-06T13:49:46.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.255+0000 7fdfaf74d700 1 -- 172.21.15.189:0/3356855876 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdf8c06adb0 msgr2=0x7fdf8c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:46.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.255+0000 7fdfaf74d700 1 --2- 172.21.15.189:0/3356855876 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdf8c06adb0 0x7fdf8c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4000c00 tx=0x7fdfa4008030 comp rx=0 tx=0).stop 2024-04-06T13:49:46.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.255+0000 7fdfaf74d700 1 -- 172.21.15.189:0/3356855876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa80aec70 msgr2=0x7fdfa80a84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:46.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.255+0000 7fdfaf74d700 1 --2- 172.21.15.189:0/3356855876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa80aec70 0x7fdfa80a84c0 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 crypto rx=0x7fdf9800d800 tx=0x7fdf9800db10 comp rx=0 tx=0).stop 2024-04-06T13:49:46.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.255+0000 7fdfaf74d700 1 -- 172.21.15.189:0/3356855876 shutdown_connections 2024-04-06T13:49:46.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.255+0000 7fdfaf74d700 1 --2- 172.21.15.189:0/3356855876 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdf8c06adb0 0x7fdf8c06d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:46.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.255+0000 7fdfaf74d700 1 --2- 172.21.15.189:0/3356855876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfa80ce8b0 0x7fdfa80c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:46.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.255+0000 7fdfaf74d700 1 --2- 172.21.15.189:0/3356855876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa80aec70 0x7fdfa80a84c0 unknown :-1 s=CLOSED pgs=3040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:46.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.255+0000 7fdfaf74d700 1 --2- 172.21.15.189:0/3356855876 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfa80cdae0 0x7fdfa80c4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:46.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.255+0000 7fdfaf74d700 1 -- 172.21.15.189:0/3356855876 >> 172.21.15.189:0/3356855876 conn(0x7fdfa80cc290 msgr2=0x7fdfa80b4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:46.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.255+0000 7fdfaf74d700 1 -- 172.21.15.189:0/3356855876 shutdown_connections 2024-04-06T13:49:46.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.255+0000 7fdfaf74d700 1 -- 172.21.15.189:0/3356855876 wait complete. 2024-04-06T13:49:46.269 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2024-04-06T13:49:46.270 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) Test subvol schedule creation succeeds for default subvolgroup. 2024-04-06T13:49:46.270 INFO:tasks.cephfs_test_runner:test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-04-06T13:49:46.270 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01' 2024-04-06T13:49:46.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.396+0000 7feac8beb700 1 Processor -- start 2024-04-06T13:49:46.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.396+0000 7feac8beb700 1 -- start start 2024-04-06T13:49:46.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.397+0000 7feac8beb700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7feac415db10 0x7feac40b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:46.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.397+0000 7feac8beb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac415eb40 con 0x7feac4143470 2024-04-06T13:49:46.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.397+0000 7feac8beb700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac40d0bf0 con 0x7feac4160b60 2024-04-06T13:49:46.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.397+0000 7feac8beb700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feac4160f40 con 0x7feac415db10 2024-04-06T13:49:46.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.397+0000 7feac1d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7feac415db10 0x7feac40b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:46.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.397+0000 7feac2d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7feac4143470 0x7feac40cd890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46414/0 (socket says 172.21.15.189:46414) 2024-04-06T13:49:46.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.397+0000 7feac1d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7feac415db10 0x7feac40b37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45100/0 (socket says 172.21.15.189:45100) 2024-04-06T13:49:46.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.397+0000 7feac2d9d700 1 -- 172.21.15.189:0/3846124521 learned_addr learned my addr 172.21.15.189:0/3846124521 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:46.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.397+0000 7feac1d9b700 1 -- 172.21.15.189:0/3846124521 >> v1:172.21.15.192:6789/0 conn(0x7feac4160b60 legacy=0x7feac4142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:46.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.397+0000 7feac1d9b700 1 -- 172.21.15.189:0/3846124521 >> v1:172.21.15.189:6789/0 conn(0x7feac4143470 legacy=0x7feac40cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:46.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.397+0000 7feac1d9b700 1 -- 172.21.15.189:0/3846124521 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac415dfc0 con 0x7feac415db10 2024-04-06T13:49:46.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.397+0000 7feac159a700 1 -- 172.21.15.189:0/3846124521 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809164517 0 0) 0x7feabc001490 con 0x7feac4143470 2024-04-06T13:49:46.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.397+0000 7feac1d9b700 1 --2- 172.21.15.189:0/3846124521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feac415db10 0x7feac40b37f0 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 crypto rx=0x7feab8009820 tx=0x7feab8009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c3ac91bfde93dc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:46.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.398+0000 7feac159a700 1 -- 172.21.15.189:0/3846124521 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab8005e30 con 0x7feac415db10 2024-04-06T13:49:46.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.398+0000 7feac159a700 1 -- 172.21.15.189:0/3846124521 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab8004030 con 0x7feac415db10 2024-04-06T13:49:46.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.398+0000 7feac159a700 1 -- 172.21.15.189:0/3846124521 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab80043c0 con 0x7feac415db10 2024-04-06T13:49:46.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.398+0000 7feac8beb700 1 -- 172.21.15.189:0/3846124521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feac415db10 msgr2=0x7feac40b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:46.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.398+0000 7feac8beb700 1 --2- 172.21.15.189:0/3846124521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feac415db10 0x7feac40b37f0 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 crypto rx=0x7feab8009820 tx=0x7feab8009b30 comp rx=0 tx=0).stop 2024-04-06T13:49:46.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.398+0000 7feac8beb700 1 -- 172.21.15.189:0/3846124521 shutdown_connections 2024-04-06T13:49:46.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.398+0000 7feac8beb700 1 --2- 172.21.15.189:0/3846124521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feac415db10 0x7feac40b37f0 unknown :-1 s=CLOSED pgs=3042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:46.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.398+0000 7feac8beb700 1 -- 172.21.15.189:0/3846124521 >> 172.21.15.189:0/3846124521 conn(0x7feac406a730 msgr2=0x7feac406ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:46.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.398+0000 7feac8beb700 1 -- 172.21.15.189:0/3846124521 shutdown_connections 2024-04-06T13:49:46.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.398+0000 7feac8beb700 1 -- 172.21.15.189:0/3846124521 wait complete. 2024-04-06T13:49:46.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.399+0000 7feac8beb700 1 Processor -- start 2024-04-06T13:49:46.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.400+0000 7feac8beb700 1 -- start start 2024-04-06T13:49:46.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.400+0000 7feac8beb700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feac4143470 0x7feac40ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:46.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.400+0000 7feac8beb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feac415db10 0x7feac40b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:46.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.400+0000 7feac8beb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feac4160b60 0x7feac40b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:46.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.400+0000 7feac8beb700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac40b4270 con 0x7feac415db10 2024-04-06T13:49:46.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.400+0000 7feac8beb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac40c9810 con 0x7feac4160b60 2024-04-06T13:49:46.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.400+0000 7feac8beb700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac40c9af0 con 0x7feac4143470 2024-04-06T13:49:46.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.400+0000 7feac1d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feac415db10 0x7feac40b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:46.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.400+0000 7feac1d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feac415db10 0x7feac40b37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45102/0 (socket says 172.21.15.189:45102) 2024-04-06T13:49:46.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.400+0000 7feac1d9b700 1 -- 172.21.15.189:0/2572837695 learned_addr learned my addr 172.21.15.189:0/2572837695 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:46.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.401+0000 7feac2d9d700 1 --2- 172.21.15.189:0/2572837695 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feac4160b60 0x7feac40b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:46.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.401+0000 7feac259c700 1 --2- 172.21.15.189:0/2572837695 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feac4143470 0x7feac40ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:46.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.401+0000 7feac1d9b700 1 -- 172.21.15.189:0/2572837695 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feac4143470 msgr2=0x7feac40ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:46.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.401+0000 7feac1d9b700 1 --2- 172.21.15.189:0/2572837695 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feac4143470 0x7feac40ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:46.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.401+0000 7feac1d9b700 1 -- 172.21.15.189:0/2572837695 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feac4160b60 msgr2=0x7feac40b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:46.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.401+0000 7feac1d9b700 1 --2- 172.21.15.189:0/2572837695 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feac4160b60 0x7feac40b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:46.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.401+0000 7feac1d9b700 1 -- 172.21.15.189:0/2572837695 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab8009430 con 0x7feac415db10 2024-04-06T13:49:46.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.401+0000 7feac259c700 1 --2- 172.21.15.189:0/2572837695 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feac4143470 0x7feac40ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:46.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.401+0000 7feac1d9b700 1 --2- 172.21.15.189:0/2572837695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feac415db10 0x7feac40b37f0 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 crypto rx=0x7feab8000c00 tx=0x7feab8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:46.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.401+0000 7feab2ffd700 1 -- 172.21.15.189:0/2572837695 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab801d630 con 0x7feac415db10 2024-04-06T13:49:46.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.401+0000 7feab2ffd700 1 -- 172.21.15.189:0/2572837695 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab801d7b0 con 0x7feac415db10 2024-04-06T13:49:46.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.401+0000 7feab2ffd700 1 -- 172.21.15.189:0/2572837695 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab801db60 con 0x7feac415db10 2024-04-06T13:49:46.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.401+0000 7feac8beb700 1 -- 172.21.15.189:0/2572837695 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac40c9dd0 con 0x7feac415db10 2024-04-06T13:49:46.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.401+0000 7feac8beb700 1 -- 172.21.15.189:0/2572837695 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac40cab10 con 0x7feac415db10 2024-04-06T13:49:46.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.402+0000 7feac8beb700 1 -- 172.21.15.189:0/2572837695 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac4067bf0 con 0x7feac415db10 2024-04-06T13:49:46.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.402+0000 7feab2ffd700 1 -- 172.21.15.189:0/2572837695 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7feab801dcc0 con 0x7feac415db10 2024-04-06T13:49:46.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.403+0000 7feab2ffd700 1 --2- 172.21.15.189:0/2572837695 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feaa406adb0 0x7feaa406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:46.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.403+0000 7feab2ffd700 1 -- 172.21.15.189:0/2572837695 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6023+0+0 (secure 0 0 0) 0x7feab808ad40 con 0x7feac415db10 2024-04-06T13:49:46.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.403+0000 7feac259c700 1 --2- 172.21.15.189:0/2572837695 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feaa406adb0 0x7feaa406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:46.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.403+0000 7feac259c700 1 --2- 172.21.15.189:0/2572837695 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feaa406adb0 0x7feaa406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7feac41f6e10 tx=0x7feab4008040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:46.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.407+0000 7feab2ffd700 1 -- 172.21.15.189:0/2572837695 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7feab8055d60 con 0x7feac415db10 2024-04-06T13:49:46.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:46.682+0000 7feac8beb700 1 -- 172.21.15.189:0/2572837695 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]} v 0) v1 -- 0x7feac40ca2f0 con 0x7feac415db10 2024-04-06T13:49:47.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.256+0000 7feab2ffd700 1 -- 172.21.15.189:0/2572837695 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]}]=0 v3235) v1 ==== 204+0+0 (secure 0 0 0) 0x7feac40ca2f0 con 0x7feac415db10 2024-04-06T13:49:47.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.259+0000 7feac8beb700 1 -- 172.21.15.189:0/2572837695 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feaa406adb0 msgr2=0x7feaa406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:47.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.259+0000 7feac8beb700 1 --2- 172.21.15.189:0/2572837695 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feaa406adb0 0x7feaa406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7feac41f6e10 tx=0x7feab4008040 comp rx=0 tx=0).stop 2024-04-06T13:49:47.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.259+0000 7feac8beb700 1 -- 172.21.15.189:0/2572837695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feac415db10 msgr2=0x7feac40b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:47.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.259+0000 7feac8beb700 1 --2- 172.21.15.189:0/2572837695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feac415db10 0x7feac40b37f0 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 crypto rx=0x7feab8000c00 tx=0x7feab8004750 comp rx=0 tx=0).stop 2024-04-06T13:49:47.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.259+0000 7feac8beb700 1 -- 172.21.15.189:0/2572837695 shutdown_connections 2024-04-06T13:49:47.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.259+0000 7feac8beb700 1 --2- 172.21.15.189:0/2572837695 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feaa406adb0 0x7feaa406d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:47.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.259+0000 7feac8beb700 1 --2- 172.21.15.189:0/2572837695 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feac4160b60 0x7feac40b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:47.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.259+0000 7feac8beb700 1 --2- 172.21.15.189:0/2572837695 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feac4143470 0x7feac40ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:47.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.259+0000 7feac8beb700 1 --2- 172.21.15.189:0/2572837695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feac415db10 0x7feac40b37f0 unknown :-1 s=CLOSED pgs=3043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:47.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.259+0000 7feac8beb700 1 -- 172.21.15.189:0/2572837695 >> 172.21.15.189:0/2572837695 conn(0x7feac406a730 msgr2=0x7feac4143ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:47.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.259+0000 7feac8beb700 1 -- 172.21.15.189:0/2572837695 shutdown_connections 2024-04-06T13:49:47.261 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.259+0000 7feac8beb700 1 -- 172.21.15.189:0/2572837695 wait complete. 2024-04-06T13:49:47.275 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-06T13:49:47.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.405+0000 7f79370cd700 1 Processor -- start 2024-04-06T13:49:47.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.405+0000 7f79370cd700 1 -- start start 2024-04-06T13:49:47.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.405+0000 7f79370cd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f793006eb80 0x7f79300ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:47.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f79370cd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f793006c080 con 0x7f79300cdc60 2024-04-06T13:49:47.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f79370cd700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f793006a8d0 con 0x7f7930159b20 2024-04-06T13:49:47.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f79370cd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79300ce650 con 0x7f793006eb80 2024-04-06T13:49:47.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f79300cdc60 0x7f793006e470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46418/0 (socket says 172.21.15.189:46418) 2024-04-06T13:49:47.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792ffff700 1 -- 172.21.15.189:0/3146042245 learned_addr learned my addr 172.21.15.189:0/3146042245 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:47.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f793566a700 1 --2- 172.21.15.189:0/3146042245 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f793006eb80 0x7f79300ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:47.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792f7fe700 1 -- 172.21.15.189:0/3146042245 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826034125 0 0) 0x7f793006c080 con 0x7f79300cdc60 2024-04-06T13:49:47.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792f7fe700 1 -- 172.21.15.189:0/3146042245 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79200035b0 con 0x7f79300cdc60 2024-04-06T13:49:47.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792f7fe700 1 -- 172.21.15.189:0/3146042245 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064089872 0 0) 0x7f793006a8d0 con 0x7f7930159b20 2024-04-06T13:49:47.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792f7fe700 1 -- 172.21.15.189:0/3146042245 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f793006c080 con 0x7f7930159b20 2024-04-06T13:49:47.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792f7fe700 1 -- 172.21.15.189:0/3146042245 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 787906301 0 0) 0x7f79200035b0 con 0x7f79300cdc60 2024-04-06T13:49:47.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792f7fe700 1 -- 172.21.15.189:0/3146042245 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f793006eb80 msgr2=0x7f79300ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:47.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792f7fe700 1 --2- 172.21.15.189:0/3146042245 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f793006eb80 0x7f79300ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:47.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792f7fe700 1 -- 172.21.15.189:0/3146042245 >> v1:172.21.15.192:6789/0 conn(0x7f7930159b20 legacy=0x7f793006b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:47.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792f7fe700 1 -- 172.21.15.189:0/3146042245 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f793006abb0 con 0x7f79300cdc60 2024-04-06T13:49:47.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792f7fe700 1 -- 172.21.15.189:0/3146042245 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f791c001670 con 0x7f79300cdc60 2024-04-06T13:49:47.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792f7fe700 1 -- 172.21.15.189:0/3146042245 >> v1:172.21.15.189:6789/0 conn(0x7f79300cdc60 legacy=0x7f793006e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:47.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792f7fe700 1 --2- 172.21.15.189:0/3146042245 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f793006eb80 0x7f7920003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:47.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792f7fe700 1 --2- 172.21.15.189:0/3146042245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7920004170 0x7f7920004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:47.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792f7fe700 1 --2- 172.21.15.189:0/3146042245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7920004d90 0x7f7920005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:47.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792f7fe700 1 -- 172.21.15.189:0/3146042245 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79300ce650 con 0x7f7920004170 2024-04-06T13:49:47.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792f7fe700 1 -- 172.21.15.189:0/3146042245 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792000dde0 con 0x7f7920004d90 2024-04-06T13:49:47.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792f7fe700 1 -- 172.21.15.189:0/3146042245 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f792000dfa0 con 0x7f793006eb80 2024-04-06T13:49:47.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.406+0000 7f792ffff700 1 --2- 172.21.15.189:0/3146042245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7920004170 0x7f7920004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:47.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.407+0000 7f793566a700 1 --2- 172.21.15.189:0/3146042245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7920004d90 0x7f7920005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:47.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.407+0000 7f7934e69700 1 --2- 172.21.15.189:0/3146042245 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f793006eb80 0x7f7920003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:47.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.407+0000 7f792ffff700 1 -- 172.21.15.189:0/3146042245 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f793006eb80 msgr2=0x7f7920003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:47.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.407+0000 7f792ffff700 1 --2- 172.21.15.189:0/3146042245 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f793006eb80 0x7f7920003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:47.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.407+0000 7f792ffff700 1 -- 172.21.15.189:0/3146042245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7920004d90 msgr2=0x7f7920005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:47.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.407+0000 7f792ffff700 1 --2- 172.21.15.189:0/3146042245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7920004d90 0x7f7920005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:47.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.407+0000 7f792ffff700 1 -- 172.21.15.189:0/3146042245 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79200035b0 con 0x7f7920004170 2024-04-06T13:49:47.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.407+0000 7f792ffff700 1 --2- 172.21.15.189:0/3146042245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7920004170 0x7f7920004850 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7f791c00c110 tx=0x7f7920005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35379cc3574c14e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:47.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.407+0000 7f792f7fe700 1 -- 172.21.15.189:0/3146042245 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c00ccb0 con 0x7f7920004170 2024-04-06T13:49:47.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.407+0000 7f792f7fe700 1 -- 172.21.15.189:0/3146042245 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f791c00ce10 con 0x7f7920004170 2024-04-06T13:49:47.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.407+0000 7f792f7fe700 1 -- 172.21.15.189:0/3146042245 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c005ba0 con 0x7f7920004170 2024-04-06T13:49:47.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.407+0000 7f79370cd700 1 -- 172.21.15.189:0/3146042245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7920004170 msgr2=0x7f7920004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:47.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.407+0000 7f79370cd700 1 --2- 172.21.15.189:0/3146042245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7920004170 0x7f7920004850 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7f791c00c110 tx=0x7f7920005880 comp rx=0 tx=0).stop 2024-04-06T13:49:47.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.407+0000 7f793566a700 1 -- 172.21.15.189:0/3146042245 reap_dead start 2024-04-06T13:49:47.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.408+0000 7f79370cd700 1 -- 172.21.15.189:0/3146042245 shutdown_connections 2024-04-06T13:49:47.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.408+0000 7f79370cd700 1 -- 172.21.15.189:0/3146042245 >> 172.21.15.189:0/3146042245 conn(0x7f79300cc290 msgr2=0x7f79300cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:47.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.408+0000 7f79370cd700 1 -- 172.21.15.189:0/3146042245 shutdown_connections 2024-04-06T13:49:47.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.408+0000 7f79370cd700 1 -- 172.21.15.189:0/3146042245 wait complete. 2024-04-06T13:49:47.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.409+0000 7f79370cd700 1 Processor -- start 2024-04-06T13:49:47.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.409+0000 7f79370cd700 1 -- start start 2024-04-06T13:49:47.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.410+0000 7f79370cd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7920004170 0x7f793014c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:47.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.410+0000 7f79370cd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f79300ad380 0x7f79300c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:47.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.410+0000 7f79370cd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f793014bf00 0x7f79300c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:47.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.410+0000 7f79370cd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79300ad8f0 con 0x7f7920004170 2024-04-06T13:49:47.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.410+0000 7f79370cd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79301e2220 con 0x7f793014bf00 2024-04-06T13:49:47.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.410+0000 7f79370cd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79301e2500 con 0x7f79300ad380 2024-04-06T13:49:47.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.410+0000 7f7934e69700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7920004170 0x7f793014c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:47.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.410+0000 7f7934e69700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7920004170 0x7f793014c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45124/0 (socket says 172.21.15.189:45124) 2024-04-06T13:49:47.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.410+0000 7f7934e69700 1 -- 172.21.15.189:0/1800301748 learned_addr learned my addr 172.21.15.189:0/1800301748 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:47.416 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.410+0000 7f7934e69700 1 -- 172.21.15.189:0/1800301748 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f79300ad380 msgr2=0x7f79300c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:47.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.410+0000 7f7934e69700 1 --2- 172.21.15.189:0/1800301748 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f79300ad380 0x7f79300c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:47.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.410+0000 7f7934e69700 1 -- 172.21.15.189:0/1800301748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f793014bf00 msgr2=0x7f79300c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:47.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.410+0000 7f7934e69700 1 --2- 172.21.15.189:0/1800301748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f793014bf00 0x7f79300c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:47.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.410+0000 7f7934e69700 1 -- 172.21.15.189:0/1800301748 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79301e27e0 con 0x7f7920004170 2024-04-06T13:49:47.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.411+0000 7f7934e69700 1 --2- 172.21.15.189:0/1800301748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7920004170 0x7f793014c750 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 crypto rx=0x7f792400f7f0 tx=0x7f792400fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:47.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.411+0000 7f792d7fa700 1 -- 172.21.15.189:0/1800301748 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7924005320 con 0x7f7920004170 2024-04-06T13:49:47.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.411+0000 7f792d7fa700 1 -- 172.21.15.189:0/1800301748 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7924005480 con 0x7f7920004170 2024-04-06T13:49:47.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.411+0000 7f79370cd700 1 -- 172.21.15.189:0/1800301748 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79301e2ac0 con 0x7f7920004170 2024-04-06T13:49:47.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.411+0000 7f792d7fa700 1 -- 172.21.15.189:0/1800301748 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7924006aa0 con 0x7f7920004170 2024-04-06T13:49:47.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.411+0000 7f79370cd700 1 -- 172.21.15.189:0/1800301748 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79301584b0 con 0x7f7920004170 2024-04-06T13:49:47.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.412+0000 7f79370cd700 1 -- 172.21.15.189:0/1800301748 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79301e2cb0 con 0x7f7920004170 2024-04-06T13:49:47.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.412+0000 7f792d7fa700 1 -- 172.21.15.189:0/1800301748 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7924006c00 con 0x7f7920004170 2024-04-06T13:49:47.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.412+0000 7f792d7fa700 1 --2- 172.21.15.189:0/1800301748 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f791406adb0 0x7f791406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:47.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.413+0000 7f792d7fa700 1 -- 172.21.15.189:0/1800301748 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6023+0+0 (secure 0 0 0) 0x7f7924088f20 con 0x7f7920004170 2024-04-06T13:49:47.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.413+0000 7f792ffff700 1 --2- 172.21.15.189:0/1800301748 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f791406adb0 0x7f791406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:47.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.413+0000 7f792ffff700 1 --2- 172.21.15.189:0/1800301748 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f791406adb0 0x7f791406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f791c00c110 tx=0x7f791c00a000 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:47.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.417+0000 7f792d7fa700 1 -- 172.21.15.189:0/1800301748 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7924053eb0 con 0x7f7920004170 2024-04-06T13:49:47.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.612+0000 7f79370cd700 1 -- 172.21.15.189:0/1800301748 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f7930074990 con 0x7f7920004170 2024-04-06T13:49:47.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.620+0000 7f792d7fa700 1 -- 172.21.15.189:0/1800301748 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v94)=0 v94) v1 ==== 122+0+0 (secure 0 0 0) 0x7f792401a0c0 con 0x7f7920004170 2024-04-06T13:49:47.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.622+0000 7f79370cd700 1 -- 172.21.15.189:0/1800301748 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f791406adb0 msgr2=0x7f791406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:47.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.622+0000 7f79370cd700 1 --2- 172.21.15.189:0/1800301748 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f791406adb0 0x7f791406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f791c00c110 tx=0x7f791c00a000 comp rx=0 tx=0).stop 2024-04-06T13:49:47.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.622+0000 7f79370cd700 1 -- 172.21.15.189:0/1800301748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7920004170 msgr2=0x7f793014c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:47.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.622+0000 7f79370cd700 1 --2- 172.21.15.189:0/1800301748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7920004170 0x7f793014c750 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 crypto rx=0x7f792400f7f0 tx=0x7f792400fbb0 comp rx=0 tx=0).stop 2024-04-06T13:49:47.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.622+0000 7f79370cd700 1 -- 172.21.15.189:0/1800301748 shutdown_connections 2024-04-06T13:49:47.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.622+0000 7f79370cd700 1 --2- 172.21.15.189:0/1800301748 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f791406adb0 0x7f791406d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:47.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.622+0000 7f79370cd700 1 --2- 172.21.15.189:0/1800301748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f793014bf00 0x7f79300c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:47.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.622+0000 7f79370cd700 1 --2- 172.21.15.189:0/1800301748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7920004170 0x7f793014c750 unknown :-1 s=CLOSED pgs=3046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:47.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.622+0000 7f79370cd700 1 --2- 172.21.15.189:0/1800301748 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f79300ad380 0x7f79300c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:47.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.622+0000 7f79370cd700 1 -- 172.21.15.189:0/1800301748 >> 172.21.15.189:0/1800301748 conn(0x7f79300cc290 msgr2=0x7f79300666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:47.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.622+0000 7f79370cd700 1 -- 172.21.15.189:0/1800301748 shutdown_connections 2024-04-06T13:49:47.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.622+0000 7f79370cd700 1 -- 172.21.15.189:0/1800301748 wait complete. 2024-04-06T13:49:47.635 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T13:49:47.636 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:49:47.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:49:47.724 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T13:49:47.724 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:49:47.724 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:49:47.744 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:49:47.744 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:49:47.744 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:49:47.764 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:49:47.764 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:49:47.764 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:49:47.801 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:49:47.801 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:49:47.801 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:49:47.823 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:49:47.823 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-06T13:49:47.823 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:49:47.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.962+0000 7f19b7901700 1 Processor -- start 2024-04-06T13:49:47.963 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.962+0000 7f19b7901700 1 -- start start 2024-04-06T13:49:47.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.963+0000 7f19b7901700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f19b0146dd0 0x7f19b006d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:47.965 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.963+0000 7f19b7901700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f19b00cedf0 0x7f19b00cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:47.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.963+0000 7f19b7901700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b00cf730 con 0x7f19b00d17a0 2024-04-06T13:49:47.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.963+0000 7f19b7901700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19b006beb0 con 0x7f19b0146dd0 2024-04-06T13:49:47.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.963+0000 7f19b7901700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19b006c0a0 con 0x7f19b00cedf0 2024-04-06T13:49:47.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.963+0000 7f19b4e9c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f19b0146dd0 0x7f19b006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:47.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.963+0000 7f19b569d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f19b00d17a0 0x7f19b006b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46432/0 (socket says 172.21.15.189:46432) 2024-04-06T13:49:47.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.963+0000 7f19b569d700 1 -- 172.21.15.189:0/1807572025 learned_addr learned my addr 172.21.15.189:0/1807572025 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:47.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.964+0000 7f19b4e9c700 1 -- 172.21.15.189:0/1807572025 >> v2:172.21.15.192:3300/0 conn(0x7f19b00cedf0 msgr2=0x7f19b00cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:47.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.964+0000 7f19b4e9c700 1 --2- 172.21.15.189:0/1807572025 >> v2:172.21.15.192:3300/0 conn(0x7f19b00cedf0 0x7f19b00cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:47.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.964+0000 7f19b4e9c700 1 -- 172.21.15.189:0/1807572025 >> v1:172.21.15.189:6789/0 conn(0x7f19b00d17a0 legacy=0x7f19b006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:47.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.964+0000 7f19b4e9c700 1 -- 172.21.15.189:0/1807572025 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b006c380 con 0x7f19b0146dd0 2024-04-06T13:49:47.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.964+0000 7f19a7fff700 1 -- 172.21.15.189:0/1807572025 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94275910 0 0) 0x7f19b00cf730 con 0x7f19b00d17a0 2024-04-06T13:49:47.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.964+0000 7f19b4e9c700 1 --2- 172.21.15.189:0/1807572025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19b0146dd0 0x7f19b006d140 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7f19b0160fb0 tx=0x7f19ac009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=194134a52aca43e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:47.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.964+0000 7f19a7fff700 1 -- 172.21.15.189:0/1807572025 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac011830 con 0x7f19b0146dd0 2024-04-06T13:49:47.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.964+0000 7f19a7fff700 1 -- 172.21.15.189:0/1807572025 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ac011990 con 0x7f19b0146dd0 2024-04-06T13:49:47.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.964+0000 7f19a7fff700 1 -- 172.21.15.189:0/1807572025 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac011d20 con 0x7f19b0146dd0 2024-04-06T13:49:47.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.964+0000 7f19b7901700 1 -- 172.21.15.189:0/1807572025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19b0146dd0 msgr2=0x7f19b006d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:47.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.964+0000 7f19b7901700 1 --2- 172.21.15.189:0/1807572025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19b0146dd0 0x7f19b006d140 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7f19b0160fb0 tx=0x7f19ac009b20 comp rx=0 tx=0).stop 2024-04-06T13:49:47.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.964+0000 7f19b7901700 1 -- 172.21.15.189:0/1807572025 shutdown_connections 2024-04-06T13:49:47.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.964+0000 7f19b7901700 1 --2- 172.21.15.189:0/1807572025 >> v2:172.21.15.192:3300/0 conn(0x7f19b00cedf0 0x7f19b00cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:47.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.964+0000 7f19b7901700 1 --2- 172.21.15.189:0/1807572025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19b0146dd0 0x7f19b006d140 unknown :-1 s=CLOSED pgs=3048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:47.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.964+0000 7f19b7901700 1 -- 172.21.15.189:0/1807572025 >> 172.21.15.189:0/1807572025 conn(0x7f19b00b2db0 msgr2=0x7f19b00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:47.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.965+0000 7f19b7901700 1 -- 172.21.15.189:0/1807572025 shutdown_connections 2024-04-06T13:49:47.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.965+0000 7f19b7901700 1 -- 172.21.15.189:0/1807572025 wait complete. 2024-04-06T13:49:47.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.966+0000 7f19b7901700 1 Processor -- start 2024-04-06T13:49:47.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.966+0000 7f19b7901700 1 -- start start 2024-04-06T13:49:47.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.966+0000 7f19b7901700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19b00cedf0 0x7f19b015c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:47.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.966+0000 7f19b7901700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19b00d17a0 0x7f19b00ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:47.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.966+0000 7f19b7901700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19b0146dd0 0x7f19b00c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:47.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.966+0000 7f19b7901700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b00af250 con 0x7f19b0146dd0 2024-04-06T13:49:47.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.966+0000 7f19b7901700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b00ae3e0 con 0x7f19b00cedf0 2024-04-06T13:49:47.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.966+0000 7f19b7901700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19b00ae6c0 con 0x7f19b00d17a0 2024-04-06T13:49:47.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.967+0000 7f19b5e9e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19b0146dd0 0x7f19b00c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:47.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.967+0000 7f19b5e9e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19b0146dd0 0x7f19b00c6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45146/0 (socket says 172.21.15.189:45146) 2024-04-06T13:49:47.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.967+0000 7f19b5e9e700 1 -- 172.21.15.189:0/3929128450 learned_addr learned my addr 172.21.15.189:0/3929128450 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:47.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.967+0000 7f19b569d700 1 --2- 172.21.15.189:0/3929128450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19b00cedf0 0x7f19b015c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:47.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.967+0000 7f19b4e9c700 1 --2- 172.21.15.189:0/3929128450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19b00d17a0 0x7f19b00ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:47.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.967+0000 7f19b5e9e700 1 -- 172.21.15.189:0/3929128450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19b00d17a0 msgr2=0x7f19b00ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:47.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.967+0000 7f19b5e9e700 1 --2- 172.21.15.189:0/3929128450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19b00d17a0 0x7f19b00ad920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:47.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.967+0000 7f19b5e9e700 1 -- 172.21.15.189:0/3929128450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19b00cedf0 msgr2=0x7f19b015c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:47.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.967+0000 7f19b5e9e700 1 --2- 172.21.15.189:0/3929128450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19b00cedf0 0x7f19b015c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:47.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.967+0000 7f19b5e9e700 1 -- 172.21.15.189:0/3929128450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ac0094a0 con 0x7f19b0146dd0 2024-04-06T13:49:47.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.967+0000 7f19b569d700 1 --2- 172.21.15.189:0/3929128450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19b00cedf0 0x7f19b015c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:47.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.967+0000 7f19b5e9e700 1 --2- 172.21.15.189:0/3929128450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19b0146dd0 0x7f19b00c6d60 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7f19a800b970 tx=0x7f19a800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:47.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.967+0000 7f19a5ffb700 1 -- 172.21.15.189:0/3929128450 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a800c9b0 con 0x7f19b0146dd0 2024-04-06T13:49:47.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.967+0000 7f19a5ffb700 1 -- 172.21.15.189:0/3929128450 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a800cb30 con 0x7f19b0146dd0 2024-04-06T13:49:47.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.967+0000 7f19a5ffb700 1 -- 172.21.15.189:0/3929128450 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a8014440 con 0x7f19b0146dd0 2024-04-06T13:49:47.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.967+0000 7f19b7901700 1 -- 172.21.15.189:0/3929128450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b00ae9a0 con 0x7f19b0146dd0 2024-04-06T13:49:47.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.967+0000 7f19b7901700 1 -- 172.21.15.189:0/3929128450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b00c8850 con 0x7f19b0146dd0 2024-04-06T13:49:47.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.968+0000 7f19b7901700 1 -- 172.21.15.189:0/3929128450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b0074990 con 0x7f19b0146dd0 2024-04-06T13:49:47.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.969+0000 7f19a5ffb700 1 -- 172.21.15.189:0/3929128450 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f19a801d3b0 con 0x7f19b0146dd0 2024-04-06T13:49:47.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.969+0000 7f19a5ffb700 1 --2- 172.21.15.189:0/3929128450 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f199406adb0 0x7f199406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:47.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.969+0000 7f19a5ffb700 1 -- 172.21.15.189:0/3929128450 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6023+0+0 (secure 0 0 0) 0x7f19a80892f0 con 0x7f19b0146dd0 2024-04-06T13:49:47.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.969+0000 7f19b569d700 1 --2- 172.21.15.189:0/3929128450 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f199406adb0 0x7f199406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:47.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.970+0000 7f19b569d700 1 --2- 172.21.15.189:0/3929128450 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f199406adb0 0x7f199406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f19a0000f60 tx=0x7f19a0009040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:47.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:47.974+0000 7f19a5ffb700 1 -- 172.21.15.189:0/3929128450 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f19a8054310 con 0x7f19b0146dd0 2024-04-06T13:49:48.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.212+0000 7f19b7901700 1 -- 172.21.15.189:0/3929128450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19b00d1e20 con 0x7f19b0146dd0 2024-04-06T13:49:48.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.212+0000 7f19a5ffb700 1 -- 172.21.15.189:0/3929128450 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+3545 (secure 0 0 0) 0x7f19a8019df0 con 0x7f19b0146dd0 2024-04-06T13:49:48.213 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:48.214 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":133,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3073866011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3073866011},{"type":"v1","addr":"172.21.15.189:6836","nonce":3073866011}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":9917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/4213774859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4213774859},{"type":"v1","addr":"172.21.15.192:6834","nonce":4213774859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":10187,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3804930185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":3804930185},{"type":"v1","addr":"172.21.15.192:6835","nonce":3804930185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":11392,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/963373121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":963373121},{"type":"v1","addr":"172.21.15.189:6837","nonce":963373121}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127}],"filesystems":[]} 2024-04-06T13:49:48.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.215+0000 7f19b7901700 1 -- 172.21.15.189:0/3929128450 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f199406adb0 msgr2=0x7f199406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:48.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.215+0000 7f19b7901700 1 --2- 172.21.15.189:0/3929128450 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f199406adb0 0x7f199406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f19a0000f60 tx=0x7f19a0009040 comp rx=0 tx=0).stop 2024-04-06T13:49:48.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.215+0000 7f19b7901700 1 -- 172.21.15.189:0/3929128450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19b0146dd0 msgr2=0x7f19b00c6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:48.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.215+0000 7f19b7901700 1 --2- 172.21.15.189:0/3929128450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19b0146dd0 0x7f19b00c6d60 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7f19a800b970 tx=0x7f19a800bd30 comp rx=0 tx=0).stop 2024-04-06T13:49:48.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.215+0000 7f19b7901700 1 -- 172.21.15.189:0/3929128450 shutdown_connections 2024-04-06T13:49:48.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.215+0000 7f19b7901700 1 --2- 172.21.15.189:0/3929128450 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f199406adb0 0x7f199406d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:48.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.215+0000 7f19b7901700 1 --2- 172.21.15.189:0/3929128450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19b0146dd0 0x7f19b00c6d60 unknown :-1 s=CLOSED pgs=3049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:48.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.215+0000 7f19b7901700 1 --2- 172.21.15.189:0/3929128450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19b00cedf0 0x7f19b015c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:48.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.215+0000 7f19b7901700 1 --2- 172.21.15.189:0/3929128450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19b00d17a0 0x7f19b00ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:48.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.215+0000 7f19b7901700 1 -- 172.21.15.189:0/3929128450 >> 172.21.15.189:0/3929128450 conn(0x7f19b00b2db0 msgr2=0x7f19b00b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:48.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.215+0000 7f19b7901700 1 -- 172.21.15.189:0/3929128450 shutdown_connections 2024-04-06T13:49:48.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.215+0000 7f19b7901700 1 -- 172.21.15.189:0/3929128450 wait complete. 2024-04-06T13:49:48.217 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 133 2024-04-06T13:49:48.228 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-06T13:49:48.228 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-06T13:49:48.228 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-06T13:49:48.228 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:49:48.229 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-06T13:49:48.229 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-06T13:49:48.229 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-06T13:49:48.229 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:49:48.229 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-06T13:49:48.229 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-06T13:49:48.230 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-06T13:49:48.230 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:49:48.230 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-06T13:49:48.230 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-06T13:49:48.230 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-06T13:49:48.230 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:49:48.237 INFO:tasks.ceph.mds.a:Stopped 2024-04-06T13:49:48.237 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-06T13:49:48.238 INFO:tasks.ceph.mds.c:Stopped 2024-04-06T13:49:48.238 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-06T13:49:48.241 INFO:tasks.ceph.mds.b:Stopped 2024-04-06T13:49:48.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-06T13:49:48.243 INFO:tasks.ceph.mds.d:Stopped 2024-04-06T13:49:48.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-06T13:49:48.281 INFO:tasks.ceph.mds.a:Started 2024-04-06T13:49:48.281 DEBUG:teuthology.parallel:result is None 2024-04-06T13:49:48.282 INFO:tasks.ceph.mds.c:Started 2024-04-06T13:49:48.282 DEBUG:teuthology.parallel:result is None 2024-04-06T13:49:48.285 INFO:tasks.ceph.mds.b:Started 2024-04-06T13:49:48.285 DEBUG:teuthology.parallel:result is None 2024-04-06T13:49:48.286 INFO:tasks.ceph.mds.d:Started 2024-04-06T13:49:48.287 DEBUG:teuthology.parallel:result is None 2024-04-06T13:49:48.287 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-06T13:49:48.362 INFO:tasks.ceph.mds.c.smithi189.stdout:starting mds.c at 2024-04-06T13:49:48.362 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2024-04-06T13:49:48.374 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2024-04-06T13:49:48.374 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2024-04-06T13:49:48.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.418+0000 7f447de79700 1 Processor -- start 2024-04-06T13:49:48.419 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.418+0000 7f447de79700 1 -- start start 2024-04-06T13:49:48.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.419+0000 7f447de79700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f44780b4ed0 0x7f44780b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:48.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.419+0000 7f447de79700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44780d2880 0x7f44780d2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:48.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.419+0000 7f447de79700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44780b5cc0 con 0x7f447815d590 2024-04-06T13:49:48.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.419+0000 7f447de79700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44780b58c0 con 0x7f44780b4ed0 2024-04-06T13:49:48.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.419+0000 7f447de79700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44780d1500 con 0x7f44780d2880 2024-04-06T13:49:48.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.419+0000 7f4476ffd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f44780b4ed0 0x7f44780b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:48.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.419+0000 7f4477fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44780d2880 0x7f44780d2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:48.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.419+0000 7f44777fe700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f447815d590 0x7f44780b8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:33724/0 (socket says 172.21.15.189:33724) 2024-04-06T13:49:48.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.419+0000 7f44777fe700 1 -- 172.21.15.189:0/4248957282 learned_addr learned my addr 172.21.15.189:0/4248957282 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:48.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.419+0000 7f4476ffd700 1 -- 172.21.15.189:0/4248957282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44780d2880 msgr2=0x7f44780d2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:48.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.419+0000 7f4476ffd700 1 --2- 172.21.15.189:0/4248957282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44780d2880 0x7f44780d2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:48.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.419+0000 7f4476ffd700 1 -- 172.21.15.189:0/4248957282 >> v1:172.21.15.192:6789/0 conn(0x7f447815d590 legacy=0x7f44780b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:48.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.419+0000 7f4476ffd700 1 -- 172.21.15.189:0/4248957282 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44780d17e0 con 0x7f44780b4ed0 2024-04-06T13:49:48.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.419+0000 7f4477fff700 1 --2- 172.21.15.189:0/4248957282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44780d2880 0x7f44780d2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:48.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.419+0000 7f44767fc700 1 -- 172.21.15.189:0/4248957282 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841303087 0 0) 0x7f44780b5cc0 con 0x7f447815d590 2024-04-06T13:49:48.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.419+0000 7f4476ffd700 1 --2- 172.21.15.189:0/4248957282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44780b4ed0 0x7f44780b5380 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f446c009840 tx=0x7f446c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3f3983feb562c66 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:48.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.420+0000 7f44767fc700 1 -- 172.21.15.189:0/4248957282 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446c011830 con 0x7f44780b4ed0 2024-04-06T13:49:48.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.420+0000 7f44767fc700 1 -- 172.21.15.189:0/4248957282 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446c011990 con 0x7f44780b4ed0 2024-04-06T13:49:48.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.420+0000 7f44767fc700 1 -- 172.21.15.189:0/4248957282 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446c011d20 con 0x7f44780b4ed0 2024-04-06T13:49:48.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.420+0000 7f447de79700 1 -- 172.21.15.189:0/4248957282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44780b4ed0 msgr2=0x7f44780b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:48.423 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.420+0000 7f447de79700 1 --2- 172.21.15.189:0/4248957282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44780b4ed0 0x7f44780b5380 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f446c009840 tx=0x7f446c009b50 comp rx=0 tx=0).stop 2024-04-06T13:49:48.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.420+0000 7f447de79700 1 -- 172.21.15.189:0/4248957282 shutdown_connections 2024-04-06T13:49:48.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.420+0000 7f447de79700 1 --2- 172.21.15.189:0/4248957282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44780d2880 0x7f44780d2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:48.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.420+0000 7f447de79700 1 --2- 172.21.15.189:0/4248957282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44780b4ed0 0x7f44780b5380 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:48.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.420+0000 7f447de79700 1 -- 172.21.15.189:0/4248957282 >> 172.21.15.189:0/4248957282 conn(0x7f447806bbc0 msgr2=0x7f447806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:48.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.420+0000 7f447de79700 1 -- 172.21.15.189:0/4248957282 shutdown_connections 2024-04-06T13:49:48.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.420+0000 7f447de79700 1 -- 172.21.15.189:0/4248957282 wait complete. 2024-04-06T13:49:48.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.421+0000 7f447de79700 1 Processor -- start 2024-04-06T13:49:48.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.421+0000 7f447de79700 1 -- start start 2024-04-06T13:49:48.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.422+0000 7f447de79700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44780b4ed0 0x7f44780cce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:48.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.422+0000 7f447de79700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44780d2880 0x7f44780caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:48.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.422+0000 7f447de79700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f447815d590 0x7f44781566f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:48.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.422+0000 7f447de79700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44780cd370 con 0x7f447815d590 2024-04-06T13:49:48.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.422+0000 7f447de79700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4478155650 con 0x7f44780b4ed0 2024-04-06T13:49:48.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.422+0000 7f447de79700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4478155930 con 0x7f44780d2880 2024-04-06T13:49:48.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.422+0000 7f4477fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f447815d590 0x7f44781566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:48.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.422+0000 7f4477fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f447815d590 0x7f44781566f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45190/0 (socket says 172.21.15.189:45190) 2024-04-06T13:49:48.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.422+0000 7f4477fff700 1 -- 172.21.15.189:0/2080605651 learned_addr learned my addr 172.21.15.189:0/2080605651 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:48.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.422+0000 7f44777fe700 1 --2- 172.21.15.189:0/2080605651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44780b4ed0 0x7f44780cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:48.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.422+0000 7f4476ffd700 1 --2- 172.21.15.189:0/2080605651 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44780d2880 0x7f44780caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:48.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.423+0000 7f4477fff700 1 -- 172.21.15.189:0/2080605651 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44780d2880 msgr2=0x7f44780caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:48.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.423+0000 7f4477fff700 1 --2- 172.21.15.189:0/2080605651 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44780d2880 0x7f44780caf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:48.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.423+0000 7f4477fff700 1 -- 172.21.15.189:0/2080605651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44780b4ed0 msgr2=0x7f44780cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:48.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.423+0000 7f4477fff700 1 --2- 172.21.15.189:0/2080605651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44780b4ed0 0x7f44780cce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:48.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.423+0000 7f4477fff700 1 -- 172.21.15.189:0/2080605651 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4470009a70 con 0x7f447815d590 2024-04-06T13:49:48.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.423+0000 7f44777fe700 1 --2- 172.21.15.189:0/2080605651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44780b4ed0 0x7f44780cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:48.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.423+0000 7f4477fff700 1 --2- 172.21.15.189:0/2080605651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f447815d590 0x7f44781566f0 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f447000efc0 tx=0x7f447000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:48.428 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.423+0000 7f445ffff700 1 -- 172.21.15.189:0/2080605651 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447000ca90 con 0x7f447815d590 2024-04-06T13:49:48.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.423+0000 7f445ffff700 1 -- 172.21.15.189:0/2080605651 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f447000f040 con 0x7f447815d590 2024-04-06T13:49:48.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.423+0000 7f447de79700 1 -- 172.21.15.189:0/2080605651 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446c0094a0 con 0x7f447815d590 2024-04-06T13:49:48.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.423+0000 7f445ffff700 1 -- 172.21.15.189:0/2080605651 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447000ce10 con 0x7f447815d590 2024-04-06T13:49:48.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.423+0000 7f447de79700 1 -- 172.21.15.189:0/2080605651 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4478155ef0 con 0x7f447815d590 2024-04-06T13:49:48.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.424+0000 7f447de79700 1 -- 172.21.15.189:0/2080605651 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4478074990 con 0x7f447815d590 2024-04-06T13:49:48.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.424+0000 7f445ffff700 1 -- 172.21.15.189:0/2080605651 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f4470014480 con 0x7f447815d590 2024-04-06T13:49:48.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.425+0000 7f445ffff700 1 --2- 172.21.15.189:0/2080605651 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f446406adb0 0x7f446406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:48.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.425+0000 7f445ffff700 1 -- 172.21.15.189:0/2080605651 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6023+0+0 (secure 0 0 0) 0x7f4470089da0 con 0x7f447815d590 2024-04-06T13:49:48.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.425+0000 7f44777fe700 1 --2- 172.21.15.189:0/2080605651 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f446406adb0 0x7f446406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:48.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.425+0000 7f44777fe700 1 --2- 172.21.15.189:0/2080605651 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f446406adb0 0x7f446406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4468000f60 tx=0x7f4468007e70 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:48.430 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.429+0000 7f445ffff700 1 -- 172.21.15.189:0/2080605651 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4470054c20 con 0x7f447815d590 2024-04-06T13:49:48.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:48.640+0000 7f447de79700 1 -- 172.21.15.189:0/2080605651 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f44780d2050 con 0x7f447815d590 2024-04-06T13:49:49.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.624+0000 7f445ffff700 1 -- 172.21.15.189:0/2080605651 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v279) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4470009d70 con 0x7f447815d590 2024-04-06T13:49:49.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.627+0000 7f447de79700 1 -- 172.21.15.189:0/2080605651 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f446406adb0 msgr2=0x7f446406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:49.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.627+0000 7f447de79700 1 --2- 172.21.15.189:0/2080605651 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f446406adb0 0x7f446406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4468000f60 tx=0x7f4468007e70 comp rx=0 tx=0).stop 2024-04-06T13:49:49.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.627+0000 7f447de79700 1 -- 172.21.15.189:0/2080605651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f447815d590 msgr2=0x7f44781566f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:49.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.627+0000 7f447de79700 1 --2- 172.21.15.189:0/2080605651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f447815d590 0x7f44781566f0 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f447000efc0 tx=0x7f447000c3a0 comp rx=0 tx=0).stop 2024-04-06T13:49:49.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.627+0000 7f447de79700 1 -- 172.21.15.189:0/2080605651 shutdown_connections 2024-04-06T13:49:49.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.627+0000 7f447de79700 1 --2- 172.21.15.189:0/2080605651 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f446406adb0 0x7f446406d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:49.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.627+0000 7f447de79700 1 --2- 172.21.15.189:0/2080605651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f447815d590 0x7f44781566f0 unknown :-1 s=CLOSED pgs=3058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:49.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.627+0000 7f447de79700 1 --2- 172.21.15.189:0/2080605651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44780b4ed0 0x7f44780cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:49.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.628+0000 7f447de79700 1 --2- 172.21.15.189:0/2080605651 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44780d2880 0x7f44780caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:49.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.628+0000 7f447de79700 1 -- 172.21.15.189:0/2080605651 >> 172.21.15.189:0/2080605651 conn(0x7f447806bbc0 msgr2=0x7f447815c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:49.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.628+0000 7f447de79700 1 -- 172.21.15.189:0/2080605651 shutdown_connections 2024-04-06T13:49:49.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.628+0000 7f447de79700 1 -- 172.21.15.189:0/2080605651 wait complete. 2024-04-06T13:49:49.630 INFO:teuthology.orchestra.run.smithi189.stderr: removed all blocklist entries 2024-04-06T13:49:49.644 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-06T13:49:49.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.771+0000 7f0dd8444700 1 Processor -- start 2024-04-06T13:49:49.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.771+0000 7f0dd8444700 1 -- start start 2024-04-06T13:49:49.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.772+0000 7f0dd8444700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0dd006ae40 0x7f0dd00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:49.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.772+0000 7f0dd8444700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd015ddf0 0x7f0dd015e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:49.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.772+0000 7f0dd8444700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd00b3990 con 0x7f0dd015d7b0 2024-04-06T13:49:49.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.772+0000 7f0dd8444700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd00b3060 con 0x7f0dd006ae40 2024-04-06T13:49:49.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.772+0000 7f0dd8444700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd00b3340 con 0x7f0dd015ddf0 2024-04-06T13:49:49.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.772+0000 7f0dd61e0700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0dd015d7b0 0x7f0dd006a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49842/0 (socket says 172.21.15.189:49842) 2024-04-06T13:49:49.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.772+0000 7f0dd61e0700 1 -- 172.21.15.189:0/3208629124 learned_addr learned my addr 172.21.15.189:0/3208629124 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:49.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.772+0000 7f0dd69e1700 1 --2- 172.21.15.189:0/3208629124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd015ddf0 0x7f0dd015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:49.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.772+0000 7f0dd59df700 1 --2- 172.21.15.189:0/3208629124 >> v2:172.21.15.192:3300/0 conn(0x7f0dd006ae40 0x7f0dd00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:49.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.772+0000 7f0dd51de700 1 -- 172.21.15.189:0/3208629124 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321020546 0 0) 0x7f0dd00b3990 con 0x7f0dd015d7b0 2024-04-06T13:49:49.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.772+0000 7f0dd51de700 1 -- 172.21.15.189:0/3208629124 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dbc0035b0 con 0x7f0dd015d7b0 2024-04-06T13:49:49.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd51de700 1 -- 172.21.15.189:0/3208629124 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 721614945 0 0) 0x7f0dbc0035b0 con 0x7f0dd015d7b0 2024-04-06T13:49:49.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd51de700 1 -- 172.21.15.189:0/3208629124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd015ddf0 msgr2=0x7f0dd015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:49.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd51de700 1 --2- 172.21.15.189:0/3208629124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd015ddf0 0x7f0dd015e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:49.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd51de700 1 -- 172.21.15.189:0/3208629124 >> v2:172.21.15.192:3300/0 conn(0x7f0dd006ae40 msgr2=0x7f0dd00b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:49.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd51de700 1 --2- 172.21.15.189:0/3208629124 >> v2:172.21.15.192:3300/0 conn(0x7f0dd006ae40 0x7f0dd00b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:49.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd51de700 1 -- 172.21.15.189:0/3208629124 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd00b3620 con 0x7f0dd015d7b0 2024-04-06T13:49:49.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd51de700 1 -- 172.21.15.189:0/3208629124 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f0dc0001670 con 0x7f0dd015d7b0 2024-04-06T13:49:49.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd51de700 1 -- 172.21.15.189:0/3208629124 >> v1:172.21.15.189:6789/0 conn(0x7f0dd015d7b0 legacy=0x7f0dd006a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:49.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd51de700 1 --2- 172.21.15.189:0/3208629124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dbc003a00 0x7f0dbc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:49.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd51de700 1 --2- 172.21.15.189:0/3208629124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd015ddf0 0x7f0dbc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:49.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd51de700 1 --2- 172.21.15.189:0/3208629124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dbc004d90 0x7f0dbc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:49.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd51de700 1 -- 172.21.15.189:0/3208629124 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd00b3060 con 0x7f0dbc003a00 2024-04-06T13:49:49.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd51de700 1 -- 172.21.15.189:0/3208629124 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd00b3340 con 0x7f0dbc004d90 2024-04-06T13:49:49.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd51de700 1 -- 172.21.15.189:0/3208629124 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc00ba40 con 0x7f0dd015ddf0 2024-04-06T13:49:49.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd61e0700 1 --2- 172.21.15.189:0/3208629124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dbc003a00 0x7f0dbc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:49.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd59df700 1 --2- 172.21.15.189:0/3208629124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd015ddf0 0x7f0dbc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:49.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd69e1700 1 --2- 172.21.15.189:0/3208629124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dbc004d90 0x7f0dbc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:49.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd61e0700 1 -- 172.21.15.189:0/3208629124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd015ddf0 msgr2=0x7f0dbc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:49.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd61e0700 1 --2- 172.21.15.189:0/3208629124 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd015ddf0 0x7f0dbc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:49.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd61e0700 1 -- 172.21.15.189:0/3208629124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dbc004d90 msgr2=0x7f0dbc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:49.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd61e0700 1 --2- 172.21.15.189:0/3208629124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dbc004d90 0x7f0dbc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:49.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd61e0700 1 -- 172.21.15.189:0/3208629124 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc0035b0 con 0x7f0dbc003a00 2024-04-06T13:49:49.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd61e0700 1 --2- 172.21.15.189:0/3208629124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dbc003a00 0x7f0dbc003e20 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000c210 tx=0x7f0dbc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdae912b48c25ae5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:49.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.773+0000 7f0dd51de700 1 -- 172.21.15.189:0/3208629124 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc000cd50 con 0x7f0dbc003a00 2024-04-06T13:49:49.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.774+0000 7f0dd51de700 1 -- 172.21.15.189:0/3208629124 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc0005960 con 0x7f0dbc003a00 2024-04-06T13:49:49.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.774+0000 7f0dd51de700 1 -- 172.21.15.189:0/3208629124 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0005d10 con 0x7f0dbc003a00 2024-04-06T13:49:49.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.774+0000 7f0dd8444700 1 -- 172.21.15.189:0/3208629124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dbc003a00 msgr2=0x7f0dbc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:49.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.774+0000 7f0dd8444700 1 --2- 172.21.15.189:0/3208629124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dbc003a00 0x7f0dbc003e20 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000c210 tx=0x7f0dbc005880 comp rx=0 tx=0).stop 2024-04-06T13:49:49.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.774+0000 7f0dd69e1700 1 -- 172.21.15.189:0/3208629124 reap_dead start 2024-04-06T13:49:49.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.774+0000 7f0dd8444700 1 -- 172.21.15.189:0/3208629124 shutdown_connections 2024-04-06T13:49:49.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.774+0000 7f0dd8444700 1 -- 172.21.15.189:0/3208629124 >> 172.21.15.189:0/3208629124 conn(0x7f0dd00d00b0 msgr2=0x7f0dd00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:49.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.774+0000 7f0dd8444700 1 -- 172.21.15.189:0/3208629124 shutdown_connections 2024-04-06T13:49:49.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.774+0000 7f0dd8444700 1 -- 172.21.15.189:0/3208629124 wait complete. 2024-04-06T13:49:49.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.776+0000 7f0dd8444700 1 Processor -- start 2024-04-06T13:49:49.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.776+0000 7f0dd8444700 1 -- start start 2024-04-06T13:49:49.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dd8444700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dbc003a00 0x7f0dd00cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:49.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dd8444700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dd014b130 0x7f0dd0148110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:49.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dd8444700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd014b5b0 0x7f0dd0148650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:49.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dd8444700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd0148b90 con 0x7f0dd014b5b0 2024-04-06T13:49:49.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dd8444700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd0148ed0 con 0x7f0dd014b130 2024-04-06T13:49:49.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dd8444700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd01491b0 con 0x7f0dbc003a00 2024-04-06T13:49:49.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dd69e1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd014b5b0 0x7f0dd0148650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:49.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dd69e1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd014b5b0 0x7f0dd0148650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59220/0 (socket says 172.21.15.189:59220) 2024-04-06T13:49:49.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dd69e1700 1 -- 172.21.15.189:0/1525373514 learned_addr learned my addr 172.21.15.189:0/1525373514 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:49.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dd61e0700 1 --2- 172.21.15.189:0/1525373514 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dbc003a00 0x7f0dd00cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:49.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dd59df700 1 --2- 172.21.15.189:0/1525373514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dd014b130 0x7f0dd0148110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:49.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dd69e1700 1 -- 172.21.15.189:0/1525373514 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dbc003a00 msgr2=0x7f0dd00cc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:49.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dd69e1700 1 --2- 172.21.15.189:0/1525373514 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dbc003a00 0x7f0dd00cc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:49.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dd69e1700 1 -- 172.21.15.189:0/1525373514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dd014b130 msgr2=0x7f0dd0148110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:49.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dd69e1700 1 --2- 172.21.15.189:0/1525373514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dd014b130 0x7f0dd0148110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:49.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dd69e1700 1 -- 172.21.15.189:0/1525373514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd0149490 con 0x7f0dd014b5b0 2024-04-06T13:49:49.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dd69e1700 1 --2- 172.21.15.189:0/1525373514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd014b5b0 0x7f0dd0148650 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7f0dd00b4800 tx=0x7f0dc8011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:49.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dc6ffd700 1 -- 172.21.15.189:0/1525373514 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc800e040 con 0x7f0dd014b5b0 2024-04-06T13:49:49.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dc6ffd700 1 -- 172.21.15.189:0/1525373514 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc800f040 con 0x7f0dd014b5b0 2024-04-06T13:49:49.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.777+0000 7f0dd8444700 1 -- 172.21.15.189:0/1525373514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd0149770 con 0x7f0dd014b5b0 2024-04-06T13:49:49.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.778+0000 7f0dc6ffd700 1 -- 172.21.15.189:0/1525373514 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc800fc40 con 0x7f0dd014b5b0 2024-04-06T13:49:49.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.778+0000 7f0dd8444700 1 -- 172.21.15.189:0/1525373514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd0149f70 con 0x7f0dd014b5b0 2024-04-06T13:49:49.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.778+0000 7f0dd8444700 1 -- 172.21.15.189:0/1525373514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd0149c90 con 0x7f0dd014b5b0 2024-04-06T13:49:49.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.779+0000 7f0dc6ffd700 1 -- 172.21.15.189:0/1525373514 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0dc8022460 con 0x7f0dd014b5b0 2024-04-06T13:49:49.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.779+0000 7f0dc6ffd700 1 --2- 172.21.15.189:0/1525373514 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0db406adb0 0x7f0db406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:49.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.780+0000 7f0dd61e0700 1 --2- 172.21.15.189:0/1525373514 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0db406adb0 0x7f0db406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:49.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.780+0000 7f0dc6ffd700 1 -- 172.21.15.189:0/1525373514 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f0dc80898f0 con 0x7f0dd014b5b0 2024-04-06T13:49:49.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.780+0000 7f0dd61e0700 1 --2- 172.21.15.189:0/1525373514 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0db406adb0 0x7f0db406d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000c210 tx=0x7f0dc000a000 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:49.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:49.784+0000 7f0dc6ffd700 1 -- 172.21.15.189:0/1525373514 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0dd0149c90 con 0x7f0dd014b5b0 2024-04-06T13:49:50.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.059+0000 7f0dd8444700 1 -- 172.21.15.189:0/1525373514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0dd0149c90 con 0x7f0dd014b5b0 2024-04-06T13:49:50.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.060+0000 7f0dc6ffd700 1 -- 172.21.15.189:0/1525373514 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v54) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0dd0149c90 con 0x7f0dd014b5b0 2024-04-06T13:49:50.061 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:50.061 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:49:50.061 INFO:teuthology.orchestra.run.smithi189.stdout: "auth_dump": [ 2024-04-06T13:49:50.061 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:50.061 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.a", 2024-04-06T13:49:50.061 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmTjYMNBAA5voWGV4Pa7pE27VogrdipQ==", 2024-04-06T13:49:50.061 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:50.061 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:49:50.061 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:49:50.061 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:49:50.061 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:50.061 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.061 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:50.062 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:50.062 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.b", 2024-04-06T13:49:50.062 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm9A1OBRAAnHzubCvRUX8UdW8hmDzKbQ==", 2024-04-06T13:49:50.062 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:50.062 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:49:50.062 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:49:50.062 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:49:50.062 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:50.062 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.062 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:50.062 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:50.062 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.c", 2024-04-06T13:49:50.062 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmz0hhOxAAD3/GINNmGoFk+oPyiHHJXg==", 2024-04-06T13:49:50.062 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:50.062 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:49:50.063 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:49:50.063 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:49:50.063 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:50.063 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.063 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:50.063 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:50.063 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mds.d", 2024-04-06T13:49:50.063 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmhoAeDRAAoQC0vicWumBRTwpi3PtYaw==", 2024-04-06T13:49:50.063 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:50.063 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:49:50.063 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:49:50.063 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:49:50.063 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:50.063 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.063 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:50.064 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:50.064 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.0", 2024-04-06T13:49:50.064 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDkOxFmkxLWJhAAFXewCNOFX7dhAZVgUmewpQ==", 2024-04-06T13:49:50.064 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:50.064 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:49:50.064 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:49:50.064 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:50.064 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.064 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:50.064 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:50.064 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.1", 2024-04-06T13:49:50.064 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDnOxFmHYqWJRAABlsVCiv3e7YJMbfodJTWag==", 2024-04-06T13:49:50.064 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:50.064 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:49:50.064 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:49:50.064 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:50.065 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.065 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:50.065 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:50.065 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.2", 2024-04-06T13:49:50.065 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDqOxFmkhcvJBAA8NMEpHVGbGALjV9RISKP8g==", 2024-04-06T13:49:50.065 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:50.065 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:49:50.065 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:49:50.065 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:50.065 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.065 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:50.065 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:50.065 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.3", 2024-04-06T13:49:50.065 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDtOxFmDGzhIxAAamBoAbE8+lAel4os67meyQ==", 2024-04-06T13:49:50.066 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:50.066 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:49:50.066 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:49:50.066 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:50.066 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.066 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:50.066 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:50.066 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.4", 2024-04-06T13:49:50.066 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDzOxFmobZDJxAA9l8wdS81Eyw6+17I5ac6bw==", 2024-04-06T13:49:50.066 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:50.066 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:49:50.066 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:49:50.066 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:50.066 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.066 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:50.067 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:50.067 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.5", 2024-04-06T13:49:50.067 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD2OxFmi+ruJRAASlQ0o9jX/wA7U5AosnL+Wg==", 2024-04-06T13:49:50.067 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:50.067 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:49:50.067 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:49:50.067 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:50.067 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.067 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:50.067 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:50.067 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.6", 2024-04-06T13:49:50.067 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD5OxFmbwuuJBAA0wqJRmnJyskQPZJ7/LnpvA==", 2024-04-06T13:49:50.067 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:50.067 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:49:50.068 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:49:50.068 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:50.068 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.068 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:50.068 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:50.068 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "osd.7", 2024-04-06T13:49:50.068 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQD8OxFmDmRgIxAAzyW9AGAbmb67ZvB5qunwLQ==", 2024-04-06T13:49:50.068 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:50.068 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow profile osd", 2024-04-06T13:49:50.068 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile osd", 2024-04-06T13:49:50.068 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:50.068 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.068 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:50.068 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:50.068 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.0", 2024-04-06T13:49:50.069 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm0ByJERAAoEV8Yfc09+Go6u95XafElQ==", 2024-04-06T13:49:50.069 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:50.069 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:49:50.069 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow r", 2024-04-06T13:49:50.069 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-06T13:49:50.069 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.069 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:50.069 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:50.069 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.1", 2024-04-06T13:49:50.069 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmSV5lGBAAavDEo2LGy0IHpWSAuitHXg==", 2024-04-06T13:49:50.069 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:50.069 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:49:50.069 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:49:50.069 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:49:50.069 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:49:50.070 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.070 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:50.070 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:50.070 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.2", 2024-04-06T13:49:50.070 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFm/Gz5FBAA3xX/rsAyWZ1JoUVlQlBIvQ==", 2024-04-06T13:49:50.070 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:50.070 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:49:50.070 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:49:50.070 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:49:50.070 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:49:50.070 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.070 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:50.070 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:50.070 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.3", 2024-04-06T13:49:50.070 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDhOxFmzra7GxAADbnzIvWnRTo65LvPwtxWrg==", 2024-04-06T13:49:50.071 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:50.071 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow", 2024-04-06T13:49:50.071 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow r", 2024-04-06T13:49:50.071 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow rw", 2024-04-06T13:49:50.071 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow rwx" 2024-04-06T13:49:50.071 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.071 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:50.071 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:50.071 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "client.admin", 2024-04-06T13:49:50.071 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmxG+wCBAANg5kt6TOUQdj3/X0Mqgx1g==", 2024-04-06T13:49:50.071 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:50.071 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:49:50.071 INFO:teuthology.orchestra.run.smithi189.stdout: "mgr": "allow *", 2024-04-06T13:49:50.071 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow *", 2024-04-06T13:49:50.071 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:50.072 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.072 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:50.072 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:50.072 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.x", 2024-04-06T13:49:50.072 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFmPLyBMBAAYfhkI9Z1CPlf1Tijm+L7xA==", 2024-04-06T13:49:50.072 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:50.072 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:49:50.072 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T13:49:50.072 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:50.072 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.072 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-06T13:49:50.072 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:50.072 INFO:teuthology.orchestra.run.smithi189.stdout: "entity": "mgr.y", 2024-04-06T13:49:50.072 INFO:teuthology.orchestra.run.smithi189.stdout: "key": "AQDgOxFm1nHJHhAAQWRe8cU8duWuUJiYaDeTaQ==", 2024-04-06T13:49:50.072 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": { 2024-04-06T13:49:50.072 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": "allow *", 2024-04-06T13:49:50.073 INFO:teuthology.orchestra.run.smithi189.stdout: "mon": "allow profile mgr", 2024-04-06T13:49:50.073 INFO:teuthology.orchestra.run.smithi189.stdout: "osd": "allow *" 2024-04-06T13:49:50.073 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.073 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:50.073 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-04-06T13:49:50.073 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:49:50.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.062+0000 7f0dd8444700 1 -- 172.21.15.189:0/1525373514 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0db406adb0 msgr2=0x7f0db406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:50.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.062+0000 7f0dd8444700 1 --2- 172.21.15.189:0/1525373514 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0db406adb0 0x7f0db406d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000c210 tx=0x7f0dc000a000 comp rx=0 tx=0).stop 2024-04-06T13:49:50.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.062+0000 7f0dd8444700 1 -- 172.21.15.189:0/1525373514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd014b5b0 msgr2=0x7f0dd0148650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:50.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.062+0000 7f0dd8444700 1 --2- 172.21.15.189:0/1525373514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd014b5b0 0x7f0dd0148650 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7f0dd00b4800 tx=0x7f0dc8011ed0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.062+0000 7f0dd8444700 1 -- 172.21.15.189:0/1525373514 shutdown_connections 2024-04-06T13:49:50.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.062+0000 7f0dd8444700 1 --2- 172.21.15.189:0/1525373514 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0db406adb0 0x7f0db406d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.062+0000 7f0dd8444700 1 --2- 172.21.15.189:0/1525373514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd014b5b0 0x7f0dd0148650 unknown :-1 s=CLOSED pgs=3061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.062+0000 7f0dd8444700 1 --2- 172.21.15.189:0/1525373514 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dbc003a00 0x7f0dd00cc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.062+0000 7f0dd8444700 1 --2- 172.21.15.189:0/1525373514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dd014b130 0x7f0dd0148110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.062+0000 7f0dd8444700 1 -- 172.21.15.189:0/1525373514 >> 172.21.15.189:0/1525373514 conn(0x7f0dd00d00b0 msgr2=0x7f0dd015c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:50.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.062+0000 7f0dd8444700 1 -- 172.21.15.189:0/1525373514 shutdown_connections 2024-04-06T13:49:50.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.062+0000 7f0dd8444700 1 -- 172.21.15.189:0/1525373514 wait complete. 2024-04-06T13:49:50.076 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:49:50.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.196+0000 7f764ed7c700 1 Processor -- start 2024-04-06T13:49:50.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.196+0000 7f764ed7c700 1 -- start start 2024-04-06T13:49:50.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.197+0000 7f764ed7c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f764806ae40 0x7f76480b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.197+0000 7f764ed7c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f764815ddf0 0x7f764815e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.197+0000 7f764ed7c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76480b3990 con 0x7f764815d7b0 2024-04-06T13:49:50.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.197+0000 7f764ed7c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76480b3060 con 0x7f764815ddf0 2024-04-06T13:49:50.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.197+0000 7f764ed7c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76480b3340 con 0x7f764806ae40 2024-04-06T13:49:50.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.197+0000 7f764d319700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f764815ddf0 0x7f764815e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.197+0000 7f764cb18700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f764815d7b0 0x7f764806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49850/0 (socket says 172.21.15.189:49850) 2024-04-06T13:49:50.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.197+0000 7f764cb18700 1 -- 172.21.15.189:0/2784647726 learned_addr learned my addr 172.21.15.189:0/2784647726 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:50.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.198+0000 7f7647fff700 1 --2- 172.21.15.189:0/2784647726 >> v2:172.21.15.192:3300/0 conn(0x7f764806ae40 0x7f76480b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.198+0000 7f764d319700 1 -- 172.21.15.189:0/2784647726 >> v2:172.21.15.192:3300/0 conn(0x7f764806ae40 msgr2=0x7f76480b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:50.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.198+0000 7f764d319700 1 --2- 172.21.15.189:0/2784647726 >> v2:172.21.15.192:3300/0 conn(0x7f764806ae40 0x7f76480b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.198+0000 7f764d319700 1 -- 172.21.15.189:0/2784647726 >> v1:172.21.15.189:6789/0 conn(0x7f764815d7b0 legacy=0x7f764806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:50.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.198+0000 7f764d319700 1 -- 172.21.15.189:0/2784647726 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76480b3620 con 0x7f764815ddf0 2024-04-06T13:49:50.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.198+0000 7f76477fe700 1 -- 172.21.15.189:0/2784647726 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 426288193 0 0) 0x7f76480b3990 con 0x7f764815d7b0 2024-04-06T13:49:50.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.198+0000 7f764d319700 1 --2- 172.21.15.189:0/2784647726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f764815ddf0 0x7f764815e240 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7f76480afc00 tx=0x7f764000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c027b949df5b8ed server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:50.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.198+0000 7f76477fe700 1 -- 172.21.15.189:0/2784647726 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764000e070 con 0x7f764815ddf0 2024-04-06T13:49:50.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.198+0000 7f76477fe700 1 -- 172.21.15.189:0/2784647726 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764000bbd0 con 0x7f764815ddf0 2024-04-06T13:49:50.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.198+0000 7f76477fe700 1 -- 172.21.15.189:0/2784647726 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7640012430 con 0x7f764815ddf0 2024-04-06T13:49:50.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.199+0000 7f764ed7c700 1 -- 172.21.15.189:0/2784647726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f764815ddf0 msgr2=0x7f764815e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:50.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.199+0000 7f764ed7c700 1 --2- 172.21.15.189:0/2784647726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f764815ddf0 0x7f764815e240 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7f76480afc00 tx=0x7f764000b330 comp rx=0 tx=0).stop 2024-04-06T13:49:50.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.199+0000 7f764ed7c700 1 -- 172.21.15.189:0/2784647726 shutdown_connections 2024-04-06T13:49:50.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.199+0000 7f764ed7c700 1 --2- 172.21.15.189:0/2784647726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f764815ddf0 0x7f764815e240 unknown :-1 s=CLOSED pgs=3063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.199+0000 7f764ed7c700 1 --2- 172.21.15.189:0/2784647726 >> v2:172.21.15.192:3300/0 conn(0x7f764806ae40 0x7f76480b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.199+0000 7f764ed7c700 1 -- 172.21.15.189:0/2784647726 >> 172.21.15.189:0/2784647726 conn(0x7f76480d00b0 msgr2=0x7f76480d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:50.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.199+0000 7f764ed7c700 1 -- 172.21.15.189:0/2784647726 shutdown_connections 2024-04-06T13:49:50.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.199+0000 7f764ed7c700 1 -- 172.21.15.189:0/2784647726 wait complete. 2024-04-06T13:49:50.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.200+0000 7f764ed7c700 1 Processor -- start 2024-04-06T13:49:50.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.200+0000 7f764ed7c700 1 -- start start 2024-04-06T13:49:50.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f764ed7c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f764806ae40 0x7f76480cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f764ed7c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f764815d7b0 0x7f764815fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f764ed7c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f764815ddf0 0x7f7648160320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f764ed7c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76481f33b0 con 0x7f764815d7b0 2024-04-06T13:49:50.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f764ed7c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76481f3840 con 0x7f764806ae40 2024-04-06T13:49:50.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f764ed7c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76481f2a80 con 0x7f764815ddf0 2024-04-06T13:49:50.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f7647fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f764815d7b0 0x7f764815fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f7647fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f764815d7b0 0x7f764815fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59238/0 (socket says 172.21.15.189:59238) 2024-04-06T13:49:50.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f7647fff700 1 -- 172.21.15.189:0/3586866080 learned_addr learned my addr 172.21.15.189:0/3586866080 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:50.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f764d319700 1 --2- 172.21.15.189:0/3586866080 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f764815ddf0 0x7f7648160320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f764cb18700 1 --2- 172.21.15.189:0/3586866080 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f764806ae40 0x7f76480cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f7647fff700 1 -- 172.21.15.189:0/3586866080 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f764815ddf0 msgr2=0x7f7648160320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:50.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f7647fff700 1 --2- 172.21.15.189:0/3586866080 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f764815ddf0 0x7f7648160320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f7647fff700 1 -- 172.21.15.189:0/3586866080 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f764806ae40 msgr2=0x7f76480cdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:50.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f7647fff700 1 --2- 172.21.15.189:0/3586866080 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f764806ae40 0x7f76480cdc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f7647fff700 1 -- 172.21.15.189:0/3586866080 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7640009a70 con 0x7f764815d7b0 2024-04-06T13:49:50.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f7647fff700 1 --2- 172.21.15.189:0/3586866080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f764815d7b0 0x7f764815fde0 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7f763400e7e0 tx=0x7f763400eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:50.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f76457fa700 1 -- 172.21.15.189:0/3586866080 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7634013610 con 0x7f764815d7b0 2024-04-06T13:49:50.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f76457fa700 1 -- 172.21.15.189:0/3586866080 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7634013770 con 0x7f764815d7b0 2024-04-06T13:49:50.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f76457fa700 1 -- 172.21.15.189:0/3586866080 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7634013ae0 con 0x7f764815d7b0 2024-04-06T13:49:50.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.201+0000 7f764ed7c700 1 -- 172.21.15.189:0/3586866080 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76481f2d60 con 0x7f764815d7b0 2024-04-06T13:49:50.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.202+0000 7f764ed7c700 1 -- 172.21.15.189:0/3586866080 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76481f2750 con 0x7f764815d7b0 2024-04-06T13:49:50.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.202+0000 7f764ed7c700 1 -- 172.21.15.189:0/3586866080 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7648067bb0 con 0x7f764815d7b0 2024-04-06T13:49:50.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.203+0000 7f76457fa700 1 -- 172.21.15.189:0/3586866080 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7634013c40 con 0x7f764815d7b0 2024-04-06T13:49:50.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.203+0000 7f76457fa700 1 --2- 172.21.15.189:0/3586866080 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f762c06adb0 0x7f762c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.203+0000 7f76457fa700 1 -- 172.21.15.189:0/3586866080 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f7634089110 con 0x7f764815d7b0 2024-04-06T13:49:50.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.203+0000 7f764cb18700 1 --2- 172.21.15.189:0/3586866080 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f762c06adb0 0x7f762c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.204+0000 7f764cb18700 1 --2- 172.21.15.189:0/3586866080 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f762c06adb0 0x7f762c06d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f763c000f60 tx=0x7f763c009040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:50.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.208+0000 7f76457fa700 1 -- 172.21.15.189:0/3586866080 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f76340541f0 con 0x7f764815d7b0 2024-04-06T13:49:50.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.404+0000 7f764ed7c700 1 -- 172.21.15.189:0/3586866080 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7648074990 con 0x7f764815d7b0 2024-04-06T13:49:50.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.404+0000 7f76457fa700 1 -- 172.21.15.189:0/3586866080 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v279) v1 ==== 74+0+12768 (secure 0 0 0) 0x7f7634057810 con 0x7f764815d7b0 2024-04-06T13:49:50.406 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:50.406 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":279,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:49:49.621227+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:49:50.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.407+0000 7f764ed7c700 1 -- 172.21.15.189:0/3586866080 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f762c06adb0 msgr2=0x7f762c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:50.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.407+0000 7f764ed7c700 1 --2- 172.21.15.189:0/3586866080 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f762c06adb0 0x7f762c06d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f763c000f60 tx=0x7f763c009040 comp rx=0 tx=0).stop 2024-04-06T13:49:50.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.407+0000 7f764ed7c700 1 -- 172.21.15.189:0/3586866080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f764815d7b0 msgr2=0x7f764815fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:50.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.407+0000 7f764ed7c700 1 --2- 172.21.15.189:0/3586866080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f764815d7b0 0x7f764815fde0 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7f763400e7e0 tx=0x7f763400eba0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.407+0000 7f764ed7c700 1 -- 172.21.15.189:0/3586866080 shutdown_connections 2024-04-06T13:49:50.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.407+0000 7f764ed7c700 1 --2- 172.21.15.189:0/3586866080 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f762c06adb0 0x7f762c06d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.407+0000 7f764ed7c700 1 --2- 172.21.15.189:0/3586866080 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f764815ddf0 0x7f7648160320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.407+0000 7f764ed7c700 1 --2- 172.21.15.189:0/3586866080 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f764806ae40 0x7f76480cdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.407+0000 7f764ed7c700 1 --2- 172.21.15.189:0/3586866080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f764815d7b0 0x7f764815fde0 unknown :-1 s=CLOSED pgs=3064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.407+0000 7f764ed7c700 1 -- 172.21.15.189:0/3586866080 >> 172.21.15.189:0/3586866080 conn(0x7f76480d00b0 msgr2=0x7f764806de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:50.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.407+0000 7f764ed7c700 1 -- 172.21.15.189:0/3586866080 shutdown_connections 2024-04-06T13:49:50.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.407+0000 7f764ed7c700 1 -- 172.21.15.189:0/3586866080 wait complete. 2024-04-06T13:49:50.420 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:49:50.421 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:49:50.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.544+0000 7f8d2e55f700 1 Processor -- start 2024-04-06T13:49:50.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.544+0000 7f8d2e55f700 1 -- start start 2024-04-06T13:49:50.547 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.545+0000 7f8d2e55f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8d2815a550 0x7f8d280cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.545+0000 7f8d2e55f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8d280b4d70 0x7f8d280b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.545+0000 7f8d2e55f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d280cdbe0 con 0x7f8d280cf900 2024-04-06T13:49:50.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.545+0000 7f8d2e55f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d2815a940 con 0x7f8d2815a550 2024-04-06T13:49:50.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.545+0000 7f8d2e55f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d2815ac00 con 0x7f8d280b4d70 2024-04-06T13:49:50.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.545+0000 7f8d27fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8d2815a550 0x7f8d280cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.545+0000 7f8d277fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8d280cf900 0x7f8d280b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49864/0 (socket says 172.21.15.189:49864) 2024-04-06T13:49:50.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.545+0000 7f8d277fe700 1 -- 172.21.15.189:0/3361311267 learned_addr learned my addr 172.21.15.189:0/3361311267 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:50.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.545+0000 7f8d2cafc700 1 --2- 172.21.15.189:0/3361311267 >> v2:172.21.15.192:3300/0 conn(0x7f8d280b4d70 0x7f8d280b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.545+0000 7f8d27fff700 1 -- 172.21.15.189:0/3361311267 >> v2:172.21.15.192:3300/0 conn(0x7f8d280b4d70 msgr2=0x7f8d280b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:50.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.545+0000 7f8d27fff700 1 --2- 172.21.15.189:0/3361311267 >> v2:172.21.15.192:3300/0 conn(0x7f8d280b4d70 0x7f8d280b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.545+0000 7f8d27fff700 1 -- 172.21.15.189:0/3361311267 >> v1:172.21.15.189:6789/0 conn(0x7f8d280cf900 legacy=0x7f8d280b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:50.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.545+0000 7f8d26ffd700 1 -- 172.21.15.189:0/3361311267 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827084908 0 0) 0x7f8d280cdbe0 con 0x7f8d280cf900 2024-04-06T13:49:50.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.545+0000 7f8d27fff700 1 -- 172.21.15.189:0/3361311267 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2815aee0 con 0x7f8d2815a550 2024-04-06T13:49:50.548 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.545+0000 7f8d27fff700 1 --2- 172.21.15.189:0/3361311267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d2815a550 0x7f8d280cf3c0 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7f8d28072830 tx=0x7f8d10009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f458f86f28dc69dc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:50.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.546+0000 7f8d26ffd700 1 -- 172.21.15.189:0/3361311267 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10011830 con 0x7f8d2815a550 2024-04-06T13:49:50.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.546+0000 7f8d26ffd700 1 -- 172.21.15.189:0/3361311267 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d10011990 con 0x7f8d2815a550 2024-04-06T13:49:50.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.546+0000 7f8d26ffd700 1 -- 172.21.15.189:0/3361311267 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10011d20 con 0x7f8d2815a550 2024-04-06T13:49:50.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.546+0000 7f8d2e55f700 1 -- 172.21.15.189:0/3361311267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d2815a550 msgr2=0x7f8d280cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:50.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.546+0000 7f8d2e55f700 1 --2- 172.21.15.189:0/3361311267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d2815a550 0x7f8d280cf3c0 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7f8d28072830 tx=0x7f8d10009b20 comp rx=0 tx=0).stop 2024-04-06T13:49:50.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.546+0000 7f8d2e55f700 1 -- 172.21.15.189:0/3361311267 shutdown_connections 2024-04-06T13:49:50.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.546+0000 7f8d2e55f700 1 --2- 172.21.15.189:0/3361311267 >> v2:172.21.15.192:3300/0 conn(0x7f8d280b4d70 0x7f8d280b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.546+0000 7f8d2e55f700 1 --2- 172.21.15.189:0/3361311267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d2815a550 0x7f8d280cf3c0 unknown :-1 s=CLOSED pgs=3066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.546+0000 7f8d2e55f700 1 -- 172.21.15.189:0/3361311267 >> 172.21.15.189:0/3361311267 conn(0x7f8d280d00b0 msgr2=0x7f8d280d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:50.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.546+0000 7f8d2e55f700 1 -- 172.21.15.189:0/3361311267 shutdown_connections 2024-04-06T13:49:50.549 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.546+0000 7f8d2e55f700 1 -- 172.21.15.189:0/3361311267 wait complete. 2024-04-06T13:49:50.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.547+0000 7f8d2e55f700 1 Processor -- start 2024-04-06T13:49:50.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.548+0000 7f8d2e55f700 1 -- start start 2024-04-06T13:49:50.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.548+0000 7f8d2e55f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d280b4d70 0x7f8d280a9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.548+0000 7f8d2e55f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d280cf900 0x7f8d280a8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.548+0000 7f8d2e55f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d280a94e0 0x7f8d281f1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.548+0000 7f8d2e55f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d280aa300 con 0x7f8d280cf900 2024-04-06T13:49:50.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.548+0000 7f8d2e55f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d280aa5e0 con 0x7f8d280b4d70 2024-04-06T13:49:50.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.548+0000 7f8d2e55f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d280aa8c0 con 0x7f8d280a94e0 2024-04-06T13:49:50.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.548+0000 7f8d277fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d280cf900 0x7f8d280a8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.548+0000 7f8d277fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d280cf900 0x7f8d280a8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59260/0 (socket says 172.21.15.189:59260) 2024-04-06T13:49:50.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.548+0000 7f8d277fe700 1 -- 172.21.15.189:0/539395089 learned_addr learned my addr 172.21.15.189:0/539395089 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:50.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.549+0000 7f8d2cafc700 1 --2- 172.21.15.189:0/539395089 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d280a94e0 0x7f8d281f1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.549+0000 7f8d27fff700 1 --2- 172.21.15.189:0/539395089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d280b4d70 0x7f8d280a9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.549+0000 7f8d277fe700 1 -- 172.21.15.189:0/539395089 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d280a94e0 msgr2=0x7f8d281f1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:50.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.549+0000 7f8d277fe700 1 --2- 172.21.15.189:0/539395089 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d280a94e0 0x7f8d281f1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.549+0000 7f8d277fe700 1 -- 172.21.15.189:0/539395089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d280b4d70 msgr2=0x7f8d280a9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:50.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.549+0000 7f8d277fe700 1 --2- 172.21.15.189:0/539395089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d280b4d70 0x7f8d280a9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.549+0000 7f8d277fe700 1 -- 172.21.15.189:0/539395089 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d100094a0 con 0x7f8d280cf900 2024-04-06T13:49:50.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.549+0000 7f8d277fe700 1 --2- 172.21.15.189:0/539395089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d280cf900 0x7f8d280a8fa0 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f8d180011e0 tx=0x7f8d1800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:50.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.549+0000 7f8d24ff9700 1 -- 172.21.15.189:0/539395089 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d18004c80 con 0x7f8d280cf900 2024-04-06T13:49:50.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.549+0000 7f8d24ff9700 1 -- 172.21.15.189:0/539395089 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d18004e00 con 0x7f8d280cf900 2024-04-06T13:49:50.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.549+0000 7f8d2e55f700 1 -- 172.21.15.189:0/539395089 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d280aaba0 con 0x7f8d280cf900 2024-04-06T13:49:50.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.549+0000 7f8d24ff9700 1 -- 172.21.15.189:0/539395089 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d18007310 con 0x7f8d280cf900 2024-04-06T13:49:50.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.549+0000 7f8d2e55f700 1 -- 172.21.15.189:0/539395089 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d281f2170 con 0x7f8d280cf900 2024-04-06T13:49:50.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.550+0000 7f8d2e55f700 1 -- 172.21.15.189:0/539395089 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d280ab0c0 con 0x7f8d280cf900 2024-04-06T13:49:50.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.551+0000 7f8d24ff9700 1 -- 172.21.15.189:0/539395089 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f8d18007470 con 0x7f8d280cf900 2024-04-06T13:49:50.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.551+0000 7f8d24ff9700 1 --2- 172.21.15.189:0/539395089 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8d0806adb0 0x7f8d0806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.551+0000 7f8d24ff9700 1 -- 172.21.15.189:0/539395089 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8d180894d0 con 0x7f8d280cf900 2024-04-06T13:49:50.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.551+0000 7f8d27fff700 1 --2- 172.21.15.189:0/539395089 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8d0806adb0 0x7f8d0806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.551+0000 7f8d27fff700 1 --2- 172.21.15.189:0/539395089 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8d0806adb0 0x7f8d0806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8d100090b0 tx=0x7f8d1001c040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:50.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.555+0000 7f8d24ff9700 1 -- 172.21.15.189:0/539395089 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8d18054840 con 0x7f8d280cf900 2024-04-06T13:49:50.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.748+0000 7f8d2e55f700 1 -- 172.21.15.189:0/539395089 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8d280ab0c0 con 0x7f8d280cf900 2024-04-06T13:49:50.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.748+0000 7f8d24ff9700 1 -- 172.21.15.189:0/539395089 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v279) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8d280ab0c0 con 0x7f8d280cf900 2024-04-06T13:49:50.750 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:49:50.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.751+0000 7f8d2e55f700 1 -- 172.21.15.189:0/539395089 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8d0806adb0 msgr2=0x7f8d0806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:50.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.751+0000 7f8d2e55f700 1 --2- 172.21.15.189:0/539395089 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8d0806adb0 0x7f8d0806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8d100090b0 tx=0x7f8d1001c040 comp rx=0 tx=0).stop 2024-04-06T13:49:50.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.751+0000 7f8d2e55f700 1 -- 172.21.15.189:0/539395089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d280cf900 msgr2=0x7f8d280a8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:50.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.751+0000 7f8d2e55f700 1 --2- 172.21.15.189:0/539395089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d280cf900 0x7f8d280a8fa0 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f8d180011e0 tx=0x7f8d1800cae0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.751+0000 7f8d2e55f700 1 -- 172.21.15.189:0/539395089 shutdown_connections 2024-04-06T13:49:50.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.751+0000 7f8d2e55f700 1 --2- 172.21.15.189:0/539395089 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8d0806adb0 0x7f8d0806d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.751+0000 7f8d2e55f700 1 --2- 172.21.15.189:0/539395089 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8d280a94e0 0x7f8d281f1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.751+0000 7f8d2e55f700 1 --2- 172.21.15.189:0/539395089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d280b4d70 0x7f8d280a9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.751+0000 7f8d2e55f700 1 --2- 172.21.15.189:0/539395089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d280cf900 0x7f8d280a8fa0 unknown :-1 s=CLOSED pgs=3067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.751+0000 7f8d2e55f700 1 -- 172.21.15.189:0/539395089 >> 172.21.15.189:0/539395089 conn(0x7f8d280d00b0 msgr2=0x7f8d28159780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:50.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.751+0000 7f8d2e55f700 1 -- 172.21.15.189:0/539395089 shutdown_connections 2024-04-06T13:49:50.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.751+0000 7f8d2e55f700 1 -- 172.21.15.189:0/539395089 wait complete. 2024-04-06T13:49:50.765 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-06T13:49:50.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.887+0000 7fe7b27dd700 1 Processor -- start 2024-04-06T13:49:50.888 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.887+0000 7fe7b27dd700 1 -- start start 2024-04-06T13:49:50.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.887+0000 7fe7b27dd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac06ae40 0x7fe7ac0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.887+0000 7fe7b27dd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7ac0b3990 con 0x7fe7ac15d7b0 2024-04-06T13:49:50.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.887+0000 7fe7b27dd700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7ac0b3060 con 0x7fe7ac15ddf0 2024-04-06T13:49:50.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.887+0000 7fe7b27dd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac0b3340 con 0x7fe7ac06ae40 2024-04-06T13:49:50.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7abfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe7ac15d7b0 0x7fe7ac06a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49876/0 (socket says 172.21.15.189:49876) 2024-04-06T13:49:50.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7abfff700 1 -- 172.21.15.189:0/3378512168 learned_addr learned my addr 172.21.15.189:0/3378512168 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:50.895 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7aaffd700 1 -- 172.21.15.189:0/3378512168 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481334444 0 0) 0x7fe7ac0b3990 con 0x7fe7ac15d7b0 2024-04-06T13:49:50.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7aaffd700 1 -- 172.21.15.189:0/3378512168 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7940035b0 con 0x7fe7ac15d7b0 2024-04-06T13:49:50.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7ab7fe700 1 --2- 172.21.15.189:0/3378512168 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac06ae40 0x7fe7ac0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7aaffd700 1 -- 172.21.15.189:0/3378512168 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3000243808 0 0) 0x7fe7940035b0 con 0x7fe7ac15d7b0 2024-04-06T13:49:50.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7aaffd700 1 -- 172.21.15.189:0/3378512168 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac06ae40 msgr2=0x7fe7ac0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:50.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7aaffd700 1 --2- 172.21.15.189:0/3378512168 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac06ae40 0x7fe7ac0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7aaffd700 1 -- 172.21.15.189:0/3378512168 >> v1:172.21.15.192:6789/0 conn(0x7fe7ac15ddf0 legacy=0x7fe7ac15e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:50.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7aaffd700 1 -- 172.21.15.189:0/3378512168 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ac0b3620 con 0x7fe7ac15d7b0 2024-04-06T13:49:50.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7aaffd700 1 -- 172.21.15.189:0/3378512168 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fe79c001670 con 0x7fe7ac15d7b0 2024-04-06T13:49:50.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7aaffd700 1 -- 172.21.15.189:0/3378512168 >> v1:172.21.15.189:6789/0 conn(0x7fe7ac15d7b0 legacy=0x7fe7ac06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:50.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7aaffd700 1 --2- 172.21.15.189:0/3378512168 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac06ae40 0x7fe794003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7aaffd700 1 --2- 172.21.15.189:0/3378512168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe794003fb0 0x7fe794004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7aaffd700 1 --2- 172.21.15.189:0/3378512168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe794004d90 0x7fe794005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7aaffd700 1 -- 172.21.15.189:0/3378512168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac0b3340 con 0x7fe794004d90 2024-04-06T13:49:50.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7aaffd700 1 -- 172.21.15.189:0/3378512168 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79400bba0 con 0x7fe794003fb0 2024-04-06T13:49:50.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7aaffd700 1 -- 172.21.15.189:0/3378512168 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe79400bd60 con 0x7fe7ac06ae40 2024-04-06T13:49:50.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7b0d7a700 1 --2- 172.21.15.189:0/3378512168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe794004d90 0x7fe794005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7ab7fe700 1 --2- 172.21.15.189:0/3378512168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe794003fb0 0x7fe794004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.888+0000 7fe7abfff700 1 --2- 172.21.15.189:0/3378512168 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac06ae40 0x7fe794003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.889+0000 7fe7b0d7a700 1 -- 172.21.15.189:0/3378512168 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac06ae40 msgr2=0x7fe794003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:50.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.889+0000 7fe7b0d7a700 1 --2- 172.21.15.189:0/3378512168 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac06ae40 0x7fe794003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.889+0000 7fe7b0d7a700 1 -- 172.21.15.189:0/3378512168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe794003fb0 msgr2=0x7fe794004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:50.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.889+0000 7fe7b0d7a700 1 --2- 172.21.15.189:0/3378512168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe794003fb0 0x7fe794004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.889+0000 7fe7b0d7a700 1 -- 172.21.15.189:0/3378512168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7940035b0 con 0x7fe794004d90 2024-04-06T13:49:50.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.889+0000 7fe7b0d7a700 1 --2- 172.21.15.189:0/3378512168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe794004d90 0x7fe794005210 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7fe7a400c190 tx=0x7fe794005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c14829d6cc483240 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:50.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.889+0000 7fe7aaffd700 1 -- 172.21.15.189:0/3378512168 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4001db0 con 0x7fe794004d90 2024-04-06T13:49:50.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.889+0000 7fe7aaffd700 1 -- 172.21.15.189:0/3378512168 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a400ccd0 con 0x7fe794004d90 2024-04-06T13:49:50.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.889+0000 7fe7aaffd700 1 -- 172.21.15.189:0/3378512168 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4003c40 con 0x7fe794004d90 2024-04-06T13:49:50.897 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.889+0000 7fe7b27dd700 1 -- 172.21.15.189:0/3378512168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe794004d90 msgr2=0x7fe794005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:50.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.889+0000 7fe7b27dd700 1 --2- 172.21.15.189:0/3378512168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe794004d90 0x7fe794005210 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7fe7a400c190 tx=0x7fe794005880 comp rx=0 tx=0).stop 2024-04-06T13:49:50.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.889+0000 7fe7b0d7a700 1 -- 172.21.15.189:0/3378512168 reap_dead start 2024-04-06T13:49:50.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.889+0000 7fe7b27dd700 1 -- 172.21.15.189:0/3378512168 shutdown_connections 2024-04-06T13:49:50.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.889+0000 7fe7b27dd700 1 -- 172.21.15.189:0/3378512168 >> 172.21.15.189:0/3378512168 conn(0x7fe7ac0d00b0 msgr2=0x7fe7ac0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:50.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.889+0000 7fe7b27dd700 1 -- 172.21.15.189:0/3378512168 shutdown_connections 2024-04-06T13:49:50.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.889+0000 7fe7b27dd700 1 -- 172.21.15.189:0/3378512168 wait complete. 2024-04-06T13:49:50.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.891+0000 7fe7b27dd700 1 Processor -- start 2024-04-06T13:49:50.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.891+0000 7fe7b27dd700 1 -- start start 2024-04-06T13:49:50.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7b27dd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe794004d90 0x7fe7ac14d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7b27dd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7ac1533a0 0x7fe7ac0b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7b27dd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac153820 0x7fe7ac0b0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7b27dd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac14d800 con 0x7fe7ac1533a0 2024-04-06T13:49:50.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7b27dd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac1f6560 con 0x7fe794004d90 2024-04-06T13:49:50.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7b27dd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac1f6840 con 0x7fe7ac153820 2024-04-06T13:49:50.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7ab7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7ac1533a0 0x7fe7ac0b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7ab7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7ac1533a0 0x7fe7ac0b0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59276/0 (socket says 172.21.15.189:59276) 2024-04-06T13:49:50.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7ab7fe700 1 -- 172.21.15.189:0/3619426939 learned_addr learned my addr 172.21.15.189:0/3619426939 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:50.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7abfff700 1 --2- 172.21.15.189:0/3619426939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe794004d90 0x7fe7ac14d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7b0d7a700 1 --2- 172.21.15.189:0/3619426939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac153820 0x7fe7ac0b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7ab7fe700 1 -- 172.21.15.189:0/3619426939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac153820 msgr2=0x7fe7ac0b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:50.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7ab7fe700 1 --2- 172.21.15.189:0/3619426939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac153820 0x7fe7ac0b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7ab7fe700 1 -- 172.21.15.189:0/3619426939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe794004d90 msgr2=0x7fe7ac14d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:50.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7ab7fe700 1 --2- 172.21.15.189:0/3619426939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe794004d90 0x7fe7ac14d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:50.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7ab7fe700 1 -- 172.21.15.189:0/3619426939 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ac1f6b20 con 0x7fe7ac1533a0 2024-04-06T13:49:50.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7ab7fe700 1 --2- 172.21.15.189:0/3619426939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7ac1533a0 0x7fe7ac0b0330 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7fe7a00119c0 tx=0x7fe7a0011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:50.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7a8ff9700 1 -- 172.21.15.189:0/3619426939 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a000f940 con 0x7fe7ac1533a0 2024-04-06T13:49:50.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7a8ff9700 1 -- 172.21.15.189:0/3619426939 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a000faa0 con 0x7fe7ac1533a0 2024-04-06T13:49:50.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7b27dd700 1 -- 172.21.15.189:0/3619426939 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7ac1f6e00 con 0x7fe7ac1533a0 2024-04-06T13:49:50.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7a8ff9700 1 -- 172.21.15.189:0/3619426939 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a000fe50 con 0x7fe7ac1533a0 2024-04-06T13:49:50.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.892+0000 7fe7b27dd700 1 -- 172.21.15.189:0/3619426939 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7ac1f62d0 con 0x7fe7ac1533a0 2024-04-06T13:49:50.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.893+0000 7fe7b27dd700 1 -- 172.21.15.189:0/3619426939 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7ac067bb0 con 0x7fe7ac1533a0 2024-04-06T13:49:50.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.894+0000 7fe7a8ff9700 1 -- 172.21.15.189:0/3619426939 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fe7a0014440 con 0x7fe7ac1533a0 2024-04-06T13:49:50.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.894+0000 7fe7a8ff9700 1 --2- 172.21.15.189:0/3619426939 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe79806adb0 0x7fe79806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:50.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.894+0000 7fe7a8ff9700 1 -- 172.21.15.189:0/3619426939 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fe7a0089a70 con 0x7fe7ac1533a0 2024-04-06T13:49:50.900 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.894+0000 7fe7abfff700 1 --2- 172.21.15.189:0/3619426939 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe79806adb0 0x7fe79806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:50.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.894+0000 7fe7abfff700 1 --2- 172.21.15.189:0/3619426939 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe79806adb0 0x7fe79806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe79c007e10 tx=0x7fe79c009040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:50.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:50.898+0000 7fe7a8ff9700 1 -- 172.21.15.189:0/3619426939 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe7a0054d00 con 0x7fe7ac1533a0 2024-04-06T13:49:51.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.096+0000 7fe7b27dd700 1 -- 172.21.15.189:0/3619426939 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe7ac1f6fc0 con 0x7fe7ac1533a0 2024-04-06T13:49:51.098 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.097+0000 7fe7a8ff9700 1 -- 172.21.15.189:0/3619426939 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v279) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe7a0058320 con 0x7fe7ac1533a0 2024-04-06T13:49:51.098 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:51.098 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T13:49:51.098 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:49:51.098 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 1, 2024-04-06T13:49:51.098 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": ".mgr" 2024-04-06T13:49:51.098 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:49:51.098 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T13:49:51.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.099+0000 7fe7b27dd700 1 -- 172.21.15.189:0/3619426939 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe79806adb0 msgr2=0x7fe79806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:51.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.099+0000 7fe7b27dd700 1 --2- 172.21.15.189:0/3619426939 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe79806adb0 0x7fe79806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe79c007e10 tx=0x7fe79c009040 comp rx=0 tx=0).stop 2024-04-06T13:49:51.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.099+0000 7fe7b27dd700 1 -- 172.21.15.189:0/3619426939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7ac1533a0 msgr2=0x7fe7ac0b0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:51.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.099+0000 7fe7b27dd700 1 --2- 172.21.15.189:0/3619426939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7ac1533a0 0x7fe7ac0b0330 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7fe7a00119c0 tx=0x7fe7a0011d80 comp rx=0 tx=0).stop 2024-04-06T13:49:51.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.099+0000 7fe7b27dd700 1 -- 172.21.15.189:0/3619426939 shutdown_connections 2024-04-06T13:49:51.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.099+0000 7fe7b27dd700 1 --2- 172.21.15.189:0/3619426939 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe79806adb0 0x7fe79806d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:51.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.099+0000 7fe7b27dd700 1 --2- 172.21.15.189:0/3619426939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7ac153820 0x7fe7ac0b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:51.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.099+0000 7fe7b27dd700 1 --2- 172.21.15.189:0/3619426939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe794004d90 0x7fe7ac14d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:51.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.099+0000 7fe7b27dd700 1 --2- 172.21.15.189:0/3619426939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7ac1533a0 0x7fe7ac0b0330 unknown :-1 s=CLOSED pgs=3070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:51.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.099+0000 7fe7b27dd700 1 -- 172.21.15.189:0/3619426939 >> 172.21.15.189:0/3619426939 conn(0x7fe7ac0d00b0 msgr2=0x7fe7ac0ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:51.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.099+0000 7fe7b27dd700 1 -- 172.21.15.189:0/3619426939 shutdown_connections 2024-04-06T13:49:51.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.099+0000 7fe7b27dd700 1 -- 172.21.15.189:0/3619426939 wait complete. 2024-04-06T13:49:51.113 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-06T13:49:51.113 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-06T13:49:51.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.238+0000 7f2bd57c7700 1 Processor -- start 2024-04-06T13:49:51.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.238+0000 7f2bd57c7700 1 -- start start 2024-04-06T13:49:51.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bd57c7700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2bd015db10 0x7f2bd00b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:51.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bd57c7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bd015eb40 con 0x7f2bd0160b60 2024-04-06T13:49:51.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bd57c7700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bd00d0bf0 con 0x7f2bd0143470 2024-04-06T13:49:51.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bd57c7700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd0160f40 con 0x7f2bd015db10 2024-04-06T13:49:51.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bceffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2bd0160b60 0x7f2bd0142d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49892/0 (socket says 172.21.15.189:49892) 2024-04-06T13:49:51.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bceffd700 1 -- 172.21.15.189:0/3561425286 learned_addr learned my addr 172.21.15.189:0/3561425286 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:51.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bce7fc700 1 --2- 172.21.15.189:0/3561425286 >> v2:172.21.15.192:3300/0 conn(0x7f2bd015db10 0x7f2bd00b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:51.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bcdffb700 1 -- 172.21.15.189:0/3561425286 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037751565 0 0) 0x7f2bc4001110 con 0x7f2bd0160b60 2024-04-06T13:49:51.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bcdffb700 1 -- 172.21.15.189:0/3561425286 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bb80035b0 con 0x7f2bd0160b60 2024-04-06T13:49:51.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bcdffb700 1 -- 172.21.15.189:0/3561425286 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3190291460 0 0) 0x7f2bb80035b0 con 0x7f2bd0160b60 2024-04-06T13:49:51.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bcdffb700 1 -- 172.21.15.189:0/3561425286 >> v2:172.21.15.192:3300/0 conn(0x7f2bd015db10 msgr2=0x7f2bd00b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:51.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bcdffb700 1 --2- 172.21.15.189:0/3561425286 >> v2:172.21.15.192:3300/0 conn(0x7f2bd015db10 0x7f2bd00b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:51.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bcdffb700 1 -- 172.21.15.189:0/3561425286 >> v1:172.21.15.192:6789/0 conn(0x7f2bd0143470 legacy=0x7f2bd00cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:51.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bcdffb700 1 -- 172.21.15.189:0/3561425286 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd015dfc0 con 0x7f2bd0160b60 2024-04-06T13:49:51.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bcdffb700 1 -- 172.21.15.189:0/3561425286 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f2bc4001800 con 0x7f2bd0160b60 2024-04-06T13:49:51.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bcdffb700 1 -- 172.21.15.189:0/3561425286 >> v1:172.21.15.189:6789/0 conn(0x7f2bd0160b60 legacy=0x7f2bd0142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:51.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bcdffb700 1 --2- 172.21.15.189:0/3561425286 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bb8003a00 0x7f2bb8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:51.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bcdffb700 1 --2- 172.21.15.189:0/3561425286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bb8004850 0x7f2bb8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:51.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bcdffb700 1 --2- 172.21.15.189:0/3561425286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bb8005210 0x7f2bb800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:51.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bcdffb700 1 -- 172.21.15.189:0/3561425286 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd0160f40 con 0x7f2bb8004850 2024-04-06T13:49:51.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bcdffb700 1 -- 172.21.15.189:0/3561425286 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb800c0b0 con 0x7f2bb8005210 2024-04-06T13:49:51.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.239+0000 7f2bcdffb700 1 -- 172.21.15.189:0/3561425286 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb800c270 con 0x7f2bb8003a00 2024-04-06T13:49:51.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.240+0000 7f2bce7fc700 1 --2- 172.21.15.189:0/3561425286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bb8004850 0x7f2bb8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:51.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.240+0000 7f2bceffd700 1 --2- 172.21.15.189:0/3561425286 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bb8003a00 0x7f2bb8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:51.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.240+0000 7f2bcf7fe700 1 --2- 172.21.15.189:0/3561425286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bb8005210 0x7f2bb800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:51.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.240+0000 7f2bce7fc700 1 -- 172.21.15.189:0/3561425286 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bb8003a00 msgr2=0x7f2bb8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:51.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.240+0000 7f2bce7fc700 1 --2- 172.21.15.189:0/3561425286 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bb8003a00 0x7f2bb8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:51.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.240+0000 7f2bce7fc700 1 -- 172.21.15.189:0/3561425286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bb8005210 msgr2=0x7f2bb800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:51.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.240+0000 7f2bce7fc700 1 --2- 172.21.15.189:0/3561425286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bb8005210 0x7f2bb800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:51.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.240+0000 7f2bce7fc700 1 -- 172.21.15.189:0/3561425286 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb80035b0 con 0x7f2bb8004850 2024-04-06T13:49:51.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.240+0000 7f2bcf7fe700 1 --2- 172.21.15.189:0/3561425286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bb8005210 0x7f2bb800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:51.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.240+0000 7f2bcf7fe700 1 -- 172.21.15.189:0/3561425286 reap_dead start 2024-04-06T13:49:51.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.240+0000 7f2bce7fc700 1 --2- 172.21.15.189:0/3561425286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bb8004850 0x7f2bb8004cd0 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0009b40 tx=0x7f2bb8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921207d2f3399d5e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:51.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.240+0000 7f2bcdffb700 1 -- 172.21.15.189:0/3561425286 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0005a20 con 0x7f2bb8004850 2024-04-06T13:49:51.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.240+0000 7f2bcdffb700 1 -- 172.21.15.189:0/3561425286 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc0005b80 con 0x7f2bb8004850 2024-04-06T13:49:51.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.240+0000 7f2bcdffb700 1 -- 172.21.15.189:0/3561425286 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc000e6c0 con 0x7f2bb8004850 2024-04-06T13:49:51.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.240+0000 7f2bd57c7700 1 -- 172.21.15.189:0/3561425286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bb8004850 msgr2=0x7f2bb8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:51.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.240+0000 7f2bd57c7700 1 --2- 172.21.15.189:0/3561425286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bb8004850 0x7f2bb8004cd0 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0009b40 tx=0x7f2bb8005880 comp rx=0 tx=0).stop 2024-04-06T13:49:51.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.241+0000 7f2bd57c7700 1 -- 172.21.15.189:0/3561425286 shutdown_connections 2024-04-06T13:49:51.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.241+0000 7f2bd57c7700 1 --2- 172.21.15.189:0/3561425286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bb8004850 0x7f2bb8004cd0 unknown :-1 s=CLOSED pgs=3072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:51.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.241+0000 7f2bd57c7700 1 -- 172.21.15.189:0/3561425286 >> 172.21.15.189:0/3561425286 conn(0x7f2bd006a730 msgr2=0x7f2bd006ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:51.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.241+0000 7f2bd57c7700 1 -- 172.21.15.189:0/3561425286 shutdown_connections 2024-04-06T13:49:51.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.241+0000 7f2bd57c7700 1 -- 172.21.15.189:0/3561425286 wait complete. 2024-04-06T13:49:51.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.242+0000 7f2bd57c7700 1 Processor -- start 2024-04-06T13:49:51.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.242+0000 7f2bd57c7700 1 -- start start 2024-04-06T13:49:51.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bd57c7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bb8004850 0x7f2bd00cac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:51.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bd57c7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bd00c8c50 0x7f2bd00c4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:51.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bd57c7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bd00c6750 0x7f2bd00c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:51.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bd57c7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd00c7140 con 0x7f2bd00c8c50 2024-04-06T13:49:51.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bd57c7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd00cb1c0 con 0x7f2bb8004850 2024-04-06T13:49:51.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bd57c7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd00c9410 con 0x7f2bd00c6750 2024-04-06T13:49:51.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bce7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bd00c8c50 0x7f2bd00c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:51.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bce7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bd00c8c50 0x7f2bd00c4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59294/0 (socket says 172.21.15.189:59294) 2024-04-06T13:49:51.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bce7fc700 1 -- 172.21.15.189:0/264342037 learned_addr learned my addr 172.21.15.189:0/264342037 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:51.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bceffd700 1 --2- 172.21.15.189:0/264342037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bb8004850 0x7f2bd00cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:51.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bcf7fe700 1 --2- 172.21.15.189:0/264342037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bd00c6750 0x7f2bd00c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:51.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bce7fc700 1 -- 172.21.15.189:0/264342037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bd00c6750 msgr2=0x7f2bd00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:51.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bce7fc700 1 --2- 172.21.15.189:0/264342037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bd00c6750 0x7f2bd00c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:51.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bce7fc700 1 -- 172.21.15.189:0/264342037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bb8004850 msgr2=0x7f2bd00cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:51.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bce7fc700 1 --2- 172.21.15.189:0/264342037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bb8004850 0x7f2bd00cac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:51.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bce7fc700 1 -- 172.21.15.189:0/264342037 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd00c96f0 con 0x7f2bd00c8c50 2024-04-06T13:49:51.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bceffd700 1 --2- 172.21.15.189:0/264342037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bb8004850 0x7f2bd00cac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:51.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bce7fc700 1 --2- 172.21.15.189:0/264342037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bd00c8c50 0x7f2bd00c4490 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800c080 tx=0x7f2bc00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:51.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bb77fe700 1 -- 172.21.15.189:0/264342037 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0020070 con 0x7f2bd00c8c50 2024-04-06T13:49:51.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bb77fe700 1 -- 172.21.15.189:0/264342037 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc0005e30 con 0x7f2bd00c8c50 2024-04-06T13:49:51.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bd57c7700 1 -- 172.21.15.189:0/264342037 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd00c99d0 con 0x7f2bd00c8c50 2024-04-06T13:49:51.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.243+0000 7f2bb77fe700 1 -- 172.21.15.189:0/264342037 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0004460 con 0x7f2bd00c8c50 2024-04-06T13:49:51.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.244+0000 7f2bd57c7700 1 -- 172.21.15.189:0/264342037 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd01547c0 con 0x7f2bd00c8c50 2024-04-06T13:49:51.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.245+0000 7f2bd57c7700 1 -- 172.21.15.189:0/264342037 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd0154980 con 0x7f2bd00c8c50 2024-04-06T13:49:51.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.245+0000 7f2bb77fe700 1 -- 172.21.15.189:0/264342037 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f2bc00036a0 con 0x7f2bd00c8c50 2024-04-06T13:49:51.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.245+0000 7f2bb77fe700 1 --2- 172.21.15.189:0/264342037 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2bbc06adb0 0x7f2bbc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:51.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.245+0000 7f2bb77fe700 1 -- 172.21.15.189:0/264342037 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f2bc008a6b0 con 0x7f2bd00c8c50 2024-04-06T13:49:51.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.245+0000 7f2bceffd700 1 --2- 172.21.15.189:0/264342037 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2bbc06adb0 0x7f2bbc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:51.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.246+0000 7f2bceffd700 1 --2- 172.21.15.189:0/264342037 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2bbc06adb0 0x7f2bbc06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4001050 tx=0x7f2bc4009040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:51.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.250+0000 7f2bb77fe700 1 -- 172.21.15.189:0/264342037 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2bc00559d0 con 0x7f2bd00c8c50 2024-04-06T13:49:51.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:51.444+0000 7f2bd57c7700 1 -- 172.21.15.189:0/264342037 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2bd0074990 con 0x7f2bd00c8c50 2024-04-06T13:49:52.273 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.272+0000 7f2bb77fe700 1 -- 172.21.15.189:0/264342037 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v280) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2bc0058ff0 con 0x7f2bd00c8c50 2024-04-06T13:49:52.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.277+0000 7f2bd57c7700 1 -- 172.21.15.189:0/264342037 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2bbc06adb0 msgr2=0x7f2bbc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:52.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.277+0000 7f2bd57c7700 1 --2- 172.21.15.189:0/264342037 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2bbc06adb0 0x7f2bbc06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4001050 tx=0x7f2bc4009040 comp rx=0 tx=0).stop 2024-04-06T13:49:52.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.277+0000 7f2bd57c7700 1 -- 172.21.15.189:0/264342037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bd00c8c50 msgr2=0x7f2bd00c4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:52.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.277+0000 7f2bd57c7700 1 --2- 172.21.15.189:0/264342037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bd00c8c50 0x7f2bd00c4490 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800c080 tx=0x7f2bc00047c0 comp rx=0 tx=0).stop 2024-04-06T13:49:52.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.278+0000 7f2bd57c7700 1 -- 172.21.15.189:0/264342037 shutdown_connections 2024-04-06T13:49:52.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.278+0000 7f2bd57c7700 1 --2- 172.21.15.189:0/264342037 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2bbc06adb0 0x7f2bbc06d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:52.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.278+0000 7f2bd57c7700 1 --2- 172.21.15.189:0/264342037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bd00c6750 0x7f2bd00c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:52.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.278+0000 7f2bd57c7700 1 --2- 172.21.15.189:0/264342037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bb8004850 0x7f2bd00cac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:52.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.278+0000 7f2bd57c7700 1 --2- 172.21.15.189:0/264342037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bd00c8c50 0x7f2bd00c4490 unknown :-1 s=CLOSED pgs=3073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:52.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.278+0000 7f2bd57c7700 1 -- 172.21.15.189:0/264342037 >> 172.21.15.189:0/264342037 conn(0x7f2bd006a730 msgr2=0x7f2bd006ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:52.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.278+0000 7f2bd57c7700 1 -- 172.21.15.189:0/264342037 shutdown_connections 2024-04-06T13:49:52.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.278+0000 7f2bd57c7700 1 -- 172.21.15.189:0/264342037 wait complete. 2024-04-06T13:49:52.280 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' created 2024-04-06T13:49:52.294 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-06T13:49:52.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.428+0000 7efdc5056700 1 Processor -- start 2024-04-06T13:49:52.429 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.428+0000 7efdc5056700 1 -- start start 2024-04-06T13:49:52.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdc5056700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efdc015d7b0 0x7efdc00af040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:52.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdc5056700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efdc00b39d0 0x7efdc00ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:52.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdc5056700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdc00ace00 0x7efdc00723f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:52.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdc5056700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc0071950 con 0x7efdc015d7b0 2024-04-06T13:49:52.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdc5056700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc00b4170 con 0x7efdc00b39d0 2024-04-06T13:49:52.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdc5056700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc00b3e90 con 0x7efdc00ace00 2024-04-06T13:49:52.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdbed9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efdc015d7b0 0x7efdc00af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:52.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdbed9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efdc015d7b0 0x7efdc00af040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59300/0 (socket says 172.21.15.189:59300) 2024-04-06T13:49:52.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdbed9d700 1 -- 172.21.15.189:0/3725193225 learned_addr learned my addr 172.21.15.189:0/3725193225 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:52.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdbe59c700 1 --2- 172.21.15.189:0/3725193225 >> v2:172.21.15.192:3300/0 conn(0x7efdc00b39d0 0x7efdc00ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:52.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdbf59e700 1 --2- 172.21.15.189:0/3725193225 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdc00ace00 0x7efdc00723f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:52.432 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdbed9d700 1 -- 172.21.15.189:0/3725193225 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdc00ace00 msgr2=0x7efdc00723f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:52.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdbed9d700 1 --2- 172.21.15.189:0/3725193225 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdc00ace00 0x7efdc00723f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:52.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdbed9d700 1 -- 172.21.15.189:0/3725193225 >> v2:172.21.15.192:3300/0 conn(0x7efdc00b39d0 msgr2=0x7efdc00ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:52.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdbed9d700 1 --2- 172.21.15.189:0/3725193225 >> v2:172.21.15.192:3300/0 conn(0x7efdc00b39d0 0x7efdc00ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:52.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdbed9d700 1 -- 172.21.15.189:0/3725193225 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc0071bf0 con 0x7efdc015d7b0 2024-04-06T13:49:52.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdbe59c700 1 --2- 172.21.15.189:0/3725193225 >> v2:172.21.15.192:3300/0 conn(0x7efdc00b39d0 0x7efdc00ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:52.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdbed9d700 1 --2- 172.21.15.189:0/3725193225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdc015d7b0 0x7efdc00af040 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7efdc0067b80 tx=0x7efdb4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7409f2dfa22a5260 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:52.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdbdd9b700 1 -- 172.21.15.189:0/3725193225 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb400b870 con 0x7efdc015d7b0 2024-04-06T13:49:52.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.429+0000 7efdbdd9b700 1 -- 172.21.15.189:0/3725193225 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb400b9d0 con 0x7efdc015d7b0 2024-04-06T13:49:52.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.430+0000 7efdbdd9b700 1 -- 172.21.15.189:0/3725193225 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb400bd60 con 0x7efdc015d7b0 2024-04-06T13:49:52.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.430+0000 7efdc5056700 1 -- 172.21.15.189:0/3725193225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdc015d7b0 msgr2=0x7efdc00af040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:52.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.430+0000 7efdc5056700 1 --2- 172.21.15.189:0/3725193225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdc015d7b0 0x7efdc00af040 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7efdc0067b80 tx=0x7efdb4009bd0 comp rx=0 tx=0).stop 2024-04-06T13:49:52.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.430+0000 7efdc5056700 1 -- 172.21.15.189:0/3725193225 shutdown_connections 2024-04-06T13:49:52.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.430+0000 7efdc5056700 1 --2- 172.21.15.189:0/3725193225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdc015d7b0 0x7efdc00af040 unknown :-1 s=CLOSED pgs=3074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:52.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.430+0000 7efdc5056700 1 --2- 172.21.15.189:0/3725193225 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdc00ace00 0x7efdc00723f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:52.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.430+0000 7efdc5056700 1 --2- 172.21.15.189:0/3725193225 >> v2:172.21.15.192:3300/0 conn(0x7efdc00b39d0 0x7efdc00ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:52.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.430+0000 7efdc5056700 1 -- 172.21.15.189:0/3725193225 >> 172.21.15.189:0/3725193225 conn(0x7efdc00d00b0 msgr2=0x7efdc00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:52.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.430+0000 7efdc5056700 1 -- 172.21.15.189:0/3725193225 shutdown_connections 2024-04-06T13:49:52.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.430+0000 7efdc5056700 1 -- 172.21.15.189:0/3725193225 wait complete. 2024-04-06T13:49:52.437 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.431+0000 7efdc5056700 1 Processor -- start 2024-04-06T13:49:52.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.432+0000 7efdc5056700 1 -- start start 2024-04-06T13:49:52.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.432+0000 7efdc5056700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdc00ace00 0x7efdc0150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:52.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.432+0000 7efdc5056700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdc00b39d0 0x7efdc015fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:52.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.432+0000 7efdc5056700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efdc015ff60 0x7efdc00cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:52.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.432+0000 7efdc5056700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc00ce020 con 0x7efdc00ace00 2024-04-06T13:49:52.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.432+0000 7efdc5056700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc0151330 con 0x7efdc015ff60 2024-04-06T13:49:52.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.432+0000 7efdc5056700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc01603f0 con 0x7efdc00b39d0 2024-04-06T13:49:52.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.432+0000 7efdbed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdc00ace00 0x7efdc0150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:52.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.432+0000 7efdbed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdc00ace00 0x7efdc0150df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59312/0 (socket says 172.21.15.189:59312) 2024-04-06T13:49:52.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.432+0000 7efdbed9d700 1 -- 172.21.15.189:0/4134324261 learned_addr learned my addr 172.21.15.189:0/4134324261 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:52.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.433+0000 7efdbf59e700 1 --2- 172.21.15.189:0/4134324261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efdc015ff60 0x7efdc00cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:52.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.433+0000 7efdbe59c700 1 --2- 172.21.15.189:0/4134324261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdc00b39d0 0x7efdc015fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:52.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.433+0000 7efdbed9d700 1 -- 172.21.15.189:0/4134324261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdc00b39d0 msgr2=0x7efdc015fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:52.438 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.433+0000 7efdbed9d700 1 --2- 172.21.15.189:0/4134324261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdc00b39d0 0x7efdc015fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:52.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.433+0000 7efdbed9d700 1 -- 172.21.15.189:0/4134324261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efdc015ff60 msgr2=0x7efdc00cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:52.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.433+0000 7efdbed9d700 1 --2- 172.21.15.189:0/4134324261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efdc015ff60 0x7efdc00cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:52.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.433+0000 7efdbed9d700 1 -- 172.21.15.189:0/4134324261 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb4009550 con 0x7efdc00ace00 2024-04-06T13:49:52.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.433+0000 7efdbf59e700 1 --2- 172.21.15.189:0/4134324261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efdc015ff60 0x7efdc00cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:52.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.433+0000 7efdbed9d700 1 --2- 172.21.15.189:0/4134324261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdc00ace00 0x7efdc0150df0 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7efdb4014040 tx=0x7efdb4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:52.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.433+0000 7efda77fe700 1 -- 172.21.15.189:0/4134324261 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb4004520 con 0x7efdc00ace00 2024-04-06T13:49:52.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.433+0000 7efda77fe700 1 -- 172.21.15.189:0/4134324261 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb401e070 con 0x7efdc00ace00 2024-04-06T13:49:52.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.433+0000 7efdc5056700 1 -- 172.21.15.189:0/4134324261 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc0155750 con 0x7efdc00ace00 2024-04-06T13:49:52.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.433+0000 7efda77fe700 1 -- 172.21.15.189:0/4134324261 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb40038e0 con 0x7efdc00ace00 2024-04-06T13:49:52.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.433+0000 7efdc5056700 1 -- 172.21.15.189:0/4134324261 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc00cd780 con 0x7efdc00ace00 2024-04-06T13:49:52.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.434+0000 7efdc5056700 1 -- 172.21.15.189:0/4134324261 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc0067bb0 con 0x7efdc00ace00 2024-04-06T13:49:52.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.435+0000 7efda77fe700 1 -- 172.21.15.189:0/4134324261 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7efdb4003a40 con 0x7efdc00ace00 2024-04-06T13:49:52.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.435+0000 7efda77fe700 1 --2- 172.21.15.189:0/4134324261 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efdac06adb0 0x7efdac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:52.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.435+0000 7efda77fe700 1 -- 172.21.15.189:0/4134324261 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 5619+0+0 (secure 0 0 0) 0x7efdb4020070 con 0x7efdc00ace00 2024-04-06T13:49:52.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.435+0000 7efdbe59c700 1 --2- 172.21.15.189:0/4134324261 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efdac06adb0 0x7efdac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:52.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.435+0000 7efdbe59c700 1 --2- 172.21.15.189:0/4134324261 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efdac06adb0 0x7efdac06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efdc00b16c0 tx=0x7efdb0008040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:52.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.440+0000 7efda77fe700 1 -- 172.21.15.189:0/4134324261 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efdb4055c30 con 0x7efdc00ace00 2024-04-06T13:49:52.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:52.634+0000 7efdc5056700 1 -- 172.21.15.189:0/4134324261 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7efdc00cfe90 con 0x7efdc00ace00 2024-04-06T13:49:53.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.276+0000 7efda77fe700 1 -- 172.21.15.189:0/4134324261 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v281) v1 ==== 170+0+0 (secure 0 0 0) 0x7efdb4015020 con 0x7efdc00ace00 2024-04-06T13:49:53.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.281+0000 7efdc5056700 1 -- 172.21.15.189:0/4134324261 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efdac06adb0 msgr2=0x7efdac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:53.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.281+0000 7efdc5056700 1 --2- 172.21.15.189:0/4134324261 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efdac06adb0 0x7efdac06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efdc00b16c0 tx=0x7efdb0008040 comp rx=0 tx=0).stop 2024-04-06T13:49:53.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.281+0000 7efdc5056700 1 -- 172.21.15.189:0/4134324261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdc00ace00 msgr2=0x7efdc0150df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:53.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.281+0000 7efdc5056700 1 --2- 172.21.15.189:0/4134324261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdc00ace00 0x7efdc0150df0 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7efdb4014040 tx=0x7efdb4004750 comp rx=0 tx=0).stop 2024-04-06T13:49:53.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.282+0000 7efdc5056700 1 -- 172.21.15.189:0/4134324261 shutdown_connections 2024-04-06T13:49:53.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.282+0000 7efdc5056700 1 --2- 172.21.15.189:0/4134324261 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efdac06adb0 0x7efdac06d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:53.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.282+0000 7efdc5056700 1 --2- 172.21.15.189:0/4134324261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efdc015ff60 0x7efdc00cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:53.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.282+0000 7efdc5056700 1 --2- 172.21.15.189:0/4134324261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdc00ace00 0x7efdc0150df0 secure :-1 s=CLOSED pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7efdb4014040 tx=0x7efdb4004750 comp rx=0 tx=0).stop 2024-04-06T13:49:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.282+0000 7efdc5056700 1 --2- 172.21.15.189:0/4134324261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdc00b39d0 0x7efdc015fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:53.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.282+0000 7efdc5056700 1 -- 172.21.15.189:0/4134324261 >> 172.21.15.189:0/4134324261 conn(0x7efdc00d00b0 msgr2=0x7efdc00b2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:53.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.285+0000 7efdc5056700 1 -- 172.21.15.189:0/4134324261 shutdown_connections 2024-04-06T13:49:53.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.288+0000 7efdc5056700 1 -- 172.21.15.189:0/4134324261 wait complete. 2024-04-06T13:49:53.295 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' created 2024-04-06T13:49:53.307 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-06T13:49:53.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.432+0000 7f31d6084700 1 Processor -- start 2024-04-06T13:49:53.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.432+0000 7f31d6084700 1 -- start start 2024-04-06T13:49:53.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.433+0000 7f31d6084700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31d00b4660 0x7f31d00b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:53.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.433+0000 7f31d6084700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d00cdbe0 con 0x7f31d00b2fc0 2024-04-06T13:49:53.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.433+0000 7f31d6084700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d00b5000 con 0x7f31d015a550 2024-04-06T13:49:53.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.433+0000 7f31d6084700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d015a940 con 0x7f31d00b4660 2024-04-06T13:49:53.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.433+0000 7f31cffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f31d00b2fc0 0x7f31d00b3430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49894/0 (socket says 172.21.15.189:49894) 2024-04-06T13:49:53.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.433+0000 7f31cffff700 1 -- 172.21.15.189:0/1425023010 learned_addr learned my addr 172.21.15.189:0/1425023010 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:53.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31ceffd700 1 --2- 172.21.15.189:0/1425023010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31d00b4660 0x7f31d00b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:53.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31ce7fc700 1 -- 172.21.15.189:0/1425023010 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430267540 0 0) 0x7f31d00cdbe0 con 0x7f31d00b2fc0 2024-04-06T13:49:53.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31ce7fc700 1 -- 172.21.15.189:0/1425023010 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31bc0035b0 con 0x7f31d00b2fc0 2024-04-06T13:49:53.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31ce7fc700 1 -- 172.21.15.189:0/1425023010 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1514472401 0 0) 0x7f31bc0035b0 con 0x7f31d00b2fc0 2024-04-06T13:49:53.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31ce7fc700 1 -- 172.21.15.189:0/1425023010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31d00b4660 msgr2=0x7f31d00b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:53.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31ce7fc700 1 --2- 172.21.15.189:0/1425023010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31d00b4660 0x7f31d00b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:53.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31ce7fc700 1 -- 172.21.15.189:0/1425023010 >> v1:172.21.15.192:6789/0 conn(0x7f31d015a550 legacy=0x7f31d00cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:53.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31ce7fc700 1 -- 172.21.15.189:0/1425023010 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d015ac00 con 0x7f31d00b2fc0 2024-04-06T13:49:53.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31ce7fc700 1 -- 172.21.15.189:0/1425023010 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f31c80019f0 con 0x7f31d00b2fc0 2024-04-06T13:49:53.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31ce7fc700 1 -- 172.21.15.189:0/1425023010 >> v1:172.21.15.189:6789/0 conn(0x7f31d00b2fc0 legacy=0x7f31d00b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:53.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31ce7fc700 1 --2- 172.21.15.189:0/1425023010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31d00b4660 0x7f31bc003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:53.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31ce7fc700 1 --2- 172.21.15.189:0/1425023010 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31bc003f70 0x7f31bc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:53.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31ce7fc700 1 --2- 172.21.15.189:0/1425023010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31bc004d90 0x7f31bc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:53.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31ce7fc700 1 -- 172.21.15.189:0/1425023010 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d015a940 con 0x7f31bc004d90 2024-04-06T13:49:53.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31ce7fc700 1 -- 172.21.15.189:0/1425023010 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc00ba70 con 0x7f31bc003f70 2024-04-06T13:49:53.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31ce7fc700 1 -- 172.21.15.189:0/1425023010 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc00bc30 con 0x7f31d00b4660 2024-04-06T13:49:53.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31cffff700 1 --2- 172.21.15.189:0/1425023010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31bc004d90 0x7f31bc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:53.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31ceffd700 1 --2- 172.21.15.189:0/1425023010 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31bc003f70 0x7f31bc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:53.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31cf7fe700 1 --2- 172.21.15.189:0/1425023010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31d00b4660 0x7f31bc003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:53.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31cffff700 1 -- 172.21.15.189:0/1425023010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31d00b4660 msgr2=0x7f31bc003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:53.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31cffff700 1 --2- 172.21.15.189:0/1425023010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31d00b4660 0x7f31bc003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:53.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31cffff700 1 -- 172.21.15.189:0/1425023010 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31bc003f70 msgr2=0x7f31bc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:53.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31cffff700 1 --2- 172.21.15.189:0/1425023010 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31bc003f70 0x7f31bc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:53.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.434+0000 7f31cffff700 1 -- 172.21.15.189:0/1425023010 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31bc0035b0 con 0x7f31bc004d90 2024-04-06T13:49:53.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.435+0000 7f31cffff700 1 --2- 172.21.15.189:0/1425023010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31bc004d90 0x7f31bc005210 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7f31c800a830 tx=0x7f31bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=695d1e6e1b7eca1e server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:53.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.435+0000 7f31ce7fc700 1 -- 172.21.15.189:0/1425023010 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c8012600 con 0x7f31bc004d90 2024-04-06T13:49:53.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.435+0000 7f31ce7fc700 1 -- 172.21.15.189:0/1425023010 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c8012780 con 0x7f31bc004d90 2024-04-06T13:49:53.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.435+0000 7f31ce7fc700 1 -- 172.21.15.189:0/1425023010 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c8012b30 con 0x7f31bc004d90 2024-04-06T13:49:53.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.435+0000 7f31d6084700 1 -- 172.21.15.189:0/1425023010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31bc004d90 msgr2=0x7f31bc005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:53.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.435+0000 7f31d6084700 1 --2- 172.21.15.189:0/1425023010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31bc004d90 0x7f31bc005210 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7f31c800a830 tx=0x7f31bc005880 comp rx=0 tx=0).stop 2024-04-06T13:49:53.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.435+0000 7f31cffff700 1 -- 172.21.15.189:0/1425023010 reap_dead start 2024-04-06T13:49:53.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.435+0000 7f31d6084700 1 -- 172.21.15.189:0/1425023010 shutdown_connections 2024-04-06T13:49:53.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.435+0000 7f31d6084700 1 -- 172.21.15.189:0/1425023010 >> 172.21.15.189:0/1425023010 conn(0x7f31d00d00b0 msgr2=0x7f31d00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:53.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.435+0000 7f31d6084700 1 -- 172.21.15.189:0/1425023010 shutdown_connections 2024-04-06T13:49:53.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.435+0000 7f31d6084700 1 -- 172.21.15.189:0/1425023010 wait complete. 2024-04-06T13:49:53.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.436+0000 7f31d6084700 1 Processor -- start 2024-04-06T13:49:53.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.437+0000 7f31d6084700 1 -- start start 2024-04-06T13:49:53.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.437+0000 7f31d6084700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31bc004d90 0x7f31d015b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:53.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.437+0000 7f31d6084700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31d006b2f0 0x7f31d0153ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:53.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.437+0000 7f31d6084700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31d0157240 0x7f31d01576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:53.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.437+0000 7f31d6084700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d0157c10 con 0x7f31d006b2f0 2024-04-06T13:49:53.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.437+0000 7f31d6084700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d01540e0 con 0x7f31d0157240 2024-04-06T13:49:53.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.437+0000 7f31d6084700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d01f6080 con 0x7f31bc004d90 2024-04-06T13:49:53.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.438+0000 7f31ceffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31d006b2f0 0x7f31d0153ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:53.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.438+0000 7f31ceffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31d006b2f0 0x7f31d0153ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59330/0 (socket says 172.21.15.189:59330) 2024-04-06T13:49:53.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.438+0000 7f31ceffd700 1 -- 172.21.15.189:0/2780406175 learned_addr learned my addr 172.21.15.189:0/2780406175 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:53.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.438+0000 7f31cffff700 1 --2- 172.21.15.189:0/2780406175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31d0157240 0x7f31d01576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:53.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.438+0000 7f31cf7fe700 1 --2- 172.21.15.189:0/2780406175 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31bc004d90 0x7f31d015b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:53.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.438+0000 7f31ceffd700 1 -- 172.21.15.189:0/2780406175 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31bc004d90 msgr2=0x7f31d015b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:53.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.438+0000 7f31ceffd700 1 --2- 172.21.15.189:0/2780406175 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31bc004d90 0x7f31d015b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:53.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.438+0000 7f31ceffd700 1 -- 172.21.15.189:0/2780406175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31d0157240 msgr2=0x7f31d01576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:53.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.438+0000 7f31ceffd700 1 --2- 172.21.15.189:0/2780406175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31d0157240 0x7f31d01576d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:53.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.438+0000 7f31ceffd700 1 -- 172.21.15.189:0/2780406175 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d01f6e00 con 0x7f31d006b2f0 2024-04-06T13:49:53.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.438+0000 7f31ceffd700 1 --2- 172.21.15.189:0/2780406175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31d006b2f0 0x7f31d0153ba0 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 crypto rx=0x7f31b8011aa0 tx=0x7f31b8011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:53.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.438+0000 7f31c7fff700 1 -- 172.21.15.189:0/2780406175 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b800f920 con 0x7f31d006b2f0 2024-04-06T13:49:53.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.438+0000 7f31c7fff700 1 -- 172.21.15.189:0/2780406175 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b800fa80 con 0x7f31d006b2f0 2024-04-06T13:49:53.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.438+0000 7f31d6084700 1 -- 172.21.15.189:0/2780406175 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d01f6320 con 0x7f31d006b2f0 2024-04-06T13:49:53.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.438+0000 7f31c7fff700 1 -- 172.21.15.189:0/2780406175 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b800fe30 con 0x7f31d006b2f0 2024-04-06T13:49:53.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.438+0000 7f31d6084700 1 -- 172.21.15.189:0/2780406175 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d01f68e0 con 0x7f31d006b2f0 2024-04-06T13:49:53.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.439+0000 7f31d6084700 1 -- 172.21.15.189:0/2780406175 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d0067bb0 con 0x7f31d006b2f0 2024-04-06T13:49:53.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.440+0000 7f31c7fff700 1 -- 172.21.15.189:0/2780406175 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f31b8014440 con 0x7f31d006b2f0 2024-04-06T13:49:53.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.440+0000 7f31c7fff700 1 --2- 172.21.15.189:0/2780406175 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f31b006adb0 0x7f31b006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:53.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.440+0000 7f31c7fff700 1 -- 172.21.15.189:0/2780406175 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(281..281 src has 1..281) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f31b808a400 con 0x7f31d006b2f0 2024-04-06T13:49:53.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.440+0000 7f31cf7fe700 1 --2- 172.21.15.189:0/2780406175 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f31b006adb0 0x7f31b006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:53.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.441+0000 7f31cf7fe700 1 --2- 172.21.15.189:0/2780406175 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f31b006adb0 0x7f31b006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f31d0155740 tx=0x7f31c0009040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:53.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.444+0000 7f31c7fff700 1 -- 172.21.15.189:0/2780406175 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f31b8055440 con 0x7f31d006b2f0 2024-04-06T13:49:53.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.685+0000 7f31d6084700 1 -- 172.21.15.189:0/2780406175 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f31d0074990 con 0x7f31d006b2f0 2024-04-06T13:49:53.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.691+0000 7f31c7fff700 1 -- 172.21.15.189:0/2780406175 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 37 and data pool 38 v135) v1 ==== 174+0+0 (secure 0 0 0) 0x7f31b8058a60 con 0x7f31d006b2f0 2024-04-06T13:49:53.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.696+0000 7f31d6084700 1 -- 172.21.15.189:0/2780406175 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f31b006adb0 msgr2=0x7f31b006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:53.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.696+0000 7f31d6084700 1 --2- 172.21.15.189:0/2780406175 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f31b006adb0 0x7f31b006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f31d0155740 tx=0x7f31c0009040 comp rx=0 tx=0).stop 2024-04-06T13:49:53.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.696+0000 7f31d6084700 1 -- 172.21.15.189:0/2780406175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31d006b2f0 msgr2=0x7f31d0153ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:53.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.696+0000 7f31d6084700 1 --2- 172.21.15.189:0/2780406175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31d006b2f0 0x7f31d0153ba0 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 crypto rx=0x7f31b8011aa0 tx=0x7f31b8011e60 comp rx=0 tx=0).stop 2024-04-06T13:49:53.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.696+0000 7f31d6084700 1 -- 172.21.15.189:0/2780406175 shutdown_connections 2024-04-06T13:49:53.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.696+0000 7f31d6084700 1 --2- 172.21.15.189:0/2780406175 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f31b006adb0 0x7f31b006d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:53.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.696+0000 7f31d6084700 1 --2- 172.21.15.189:0/2780406175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31d0157240 0x7f31d01576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:53.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.696+0000 7f31d6084700 1 --2- 172.21.15.189:0/2780406175 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31bc004d90 0x7f31d015b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:53.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.696+0000 7f31d6084700 1 --2- 172.21.15.189:0/2780406175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31d006b2f0 0x7f31d0153ba0 secure :-1 s=CLOSED pgs=3078 cs=0 l=1 rev1=1 crypto rx=0x7f31b8011aa0 tx=0x7f31b8011e60 comp rx=0 tx=0).stop 2024-04-06T13:49:53.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.696+0000 7f31d6084700 1 -- 172.21.15.189:0/2780406175 >> 172.21.15.189:0/2780406175 conn(0x7f31d00d00b0 msgr2=0x7f31d01582c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:53.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.697+0000 7f31d6084700 1 -- 172.21.15.189:0/2780406175 shutdown_connections 2024-04-06T13:49:53.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.697+0000 7f31d6084700 1 -- 172.21.15.189:0/2780406175 wait complete. 2024-04-06T13:49:53.708 INFO:teuthology.orchestra.run.smithi189.stderr:new fs with metadata pool 37 and data pool 38 2024-04-06T13:49:53.725 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:49:53.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.847+0000 7f43205b2700 1 Processor -- start 2024-04-06T13:49:53.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.847+0000 7f43205b2700 1 -- start start 2024-04-06T13:49:53.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.848+0000 7f43205b2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f431815a550 0x7f43180cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:53.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.848+0000 7f43205b2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43180cdbe0 con 0x7f43180b4d70 2024-04-06T13:49:53.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.848+0000 7f43205b2700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431815a940 con 0x7f43180cf900 2024-04-06T13:49:53.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.848+0000 7f43205b2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f431815ac00 con 0x7f431815a550 2024-04-06T13:49:53.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.848+0000 7f431eb4f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f43180b4d70 0x7f43180b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49896/0 (socket says 172.21.15.189:49896) 2024-04-06T13:49:53.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.848+0000 7f431eb4f700 1 -- 172.21.15.189:0/1152909558 learned_addr learned my addr 172.21.15.189:0/1152909558 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:53.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.848+0000 7f431e34e700 1 --2- 172.21.15.189:0/1152909558 >> v2:172.21.15.192:3300/0 conn(0x7f431815a550 0x7f43180cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:53.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.848+0000 7f431d34c700 1 -- 172.21.15.189:0/1152909558 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135377234 0 0) 0x7f43180cdbe0 con 0x7f43180b4d70 2024-04-06T13:49:53.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.848+0000 7f431d34c700 1 -- 172.21.15.189:0/1152909558 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43040035b0 con 0x7f43180b4d70 2024-04-06T13:49:53.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431d34c700 1 -- 172.21.15.189:0/1152909558 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880703924 0 0) 0x7f431815a940 con 0x7f43180cf900 2024-04-06T13:49:53.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431d34c700 1 -- 172.21.15.189:0/1152909558 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43180cdbe0 con 0x7f43180cf900 2024-04-06T13:49:53.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431d34c700 1 -- 172.21.15.189:0/1152909558 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2427833768 0 0) 0x7f43040035b0 con 0x7f43180b4d70 2024-04-06T13:49:53.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431d34c700 1 -- 172.21.15.189:0/1152909558 >> v2:172.21.15.192:3300/0 conn(0x7f431815a550 msgr2=0x7f43180cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:53.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431d34c700 1 --2- 172.21.15.189:0/1152909558 >> v2:172.21.15.192:3300/0 conn(0x7f431815a550 0x7f43180cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:53.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431d34c700 1 -- 172.21.15.189:0/1152909558 >> v1:172.21.15.192:6789/0 conn(0x7f43180cf900 legacy=0x7f43180b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:53.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431d34c700 1 -- 172.21.15.189:0/1152909558 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431815aee0 con 0x7f43180b4d70 2024-04-06T13:49:53.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431d34c700 1 -- 172.21.15.189:0/1152909558 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f43100019f0 con 0x7f43180b4d70 2024-04-06T13:49:53.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431d34c700 1 -- 172.21.15.189:0/1152909558 >> v1:172.21.15.189:6789/0 conn(0x7f43180b4d70 legacy=0x7f43180b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:53.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431d34c700 1 --2- 172.21.15.189:0/1152909558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4304003b90 0x7f4304003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:53.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431d34c700 1 --2- 172.21.15.189:0/1152909558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4304004850 0x7f4304004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:53.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431d34c700 1 --2- 172.21.15.189:0/1152909558 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4304005210 0x7f430400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:53.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431d34c700 1 -- 172.21.15.189:0/1152909558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431815ac00 con 0x7f4304003b90 2024-04-06T13:49:53.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431d34c700 1 -- 172.21.15.189:0/1152909558 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430400c090 con 0x7f4304004850 2024-04-06T13:49:53.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431d34c700 1 -- 172.21.15.189:0/1152909558 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f430400c250 con 0x7f4304005210 2024-04-06T13:49:53.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431e34e700 1 --2- 172.21.15.189:0/1152909558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4304003b90 0x7f4304003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:53.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431db4d700 1 --2- 172.21.15.189:0/1152909558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4304004850 0x7f4304004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:53.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431eb4f700 1 --2- 172.21.15.189:0/1152909558 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4304005210 0x7f430400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:53.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431e34e700 1 -- 172.21.15.189:0/1152909558 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4304005210 msgr2=0x7f430400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:53.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431e34e700 1 --2- 172.21.15.189:0/1152909558 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4304005210 0x7f430400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:53.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431e34e700 1 -- 172.21.15.189:0/1152909558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4304004850 msgr2=0x7f4304004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:53.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431e34e700 1 --2- 172.21.15.189:0/1152909558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4304004850 0x7f4304004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:53.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431e34e700 1 -- 172.21.15.189:0/1152909558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43040035b0 con 0x7f4304003b90 2024-04-06T13:49:53.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431eb4f700 1 --2- 172.21.15.189:0/1152909558 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4304005210 0x7f430400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:53.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431eb4f700 1 -- 172.21.15.189:0/1152909558 reap_dead start 2024-04-06T13:49:53.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431e34e700 1 --2- 172.21.15.189:0/1152909558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4304003b90 0x7f4304003f90 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 crypto rx=0x7f43080096f0 tx=0x7f4304005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83aa40e6ae6f08ba server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:53.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431d34c700 1 -- 172.21.15.189:0/1152909558 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430800e6c0 con 0x7f4304003b90 2024-04-06T13:49:53.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431d34c700 1 -- 172.21.15.189:0/1152909558 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430800e820 con 0x7f4304003b90 2024-04-06T13:49:53.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.849+0000 7f431d34c700 1 -- 172.21.15.189:0/1152909558 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430800ebb0 con 0x7f4304003b90 2024-04-06T13:49:53.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.850+0000 7f43205b2700 1 -- 172.21.15.189:0/1152909558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4304003b90 msgr2=0x7f4304003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:53.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.850+0000 7f43205b2700 1 --2- 172.21.15.189:0/1152909558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4304003b90 0x7f4304003f90 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 crypto rx=0x7f43080096f0 tx=0x7f4304005880 comp rx=0 tx=0).stop 2024-04-06T13:49:53.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.850+0000 7f43205b2700 1 -- 172.21.15.189:0/1152909558 shutdown_connections 2024-04-06T13:49:53.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.850+0000 7f43205b2700 1 --2- 172.21.15.189:0/1152909558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4304003b90 0x7f4304003f90 unknown :-1 s=CLOSED pgs=3080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:53.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.850+0000 7f43205b2700 1 -- 172.21.15.189:0/1152909558 >> 172.21.15.189:0/1152909558 conn(0x7f43180d00b0 msgr2=0x7f43180d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:53.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.850+0000 7f43205b2700 1 -- 172.21.15.189:0/1152909558 shutdown_connections 2024-04-06T13:49:53.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.850+0000 7f43205b2700 1 -- 172.21.15.189:0/1152909558 wait complete. 2024-04-06T13:49:53.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.851+0000 7f43205b2700 1 Processor -- start 2024-04-06T13:49:53.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.851+0000 7f43205b2700 1 -- start start 2024-04-06T13:49:53.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.852+0000 7f43205b2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4304003b90 0x7f4318157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:53.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.852+0000 7f43205b2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43181595e0 0x7f43181f6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:53.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.852+0000 7f43205b2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4318159a60 0x7f43181f6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:53.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.852+0000 7f43205b2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43181457e0 con 0x7f4304003b90 2024-04-06T13:49:53.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.852+0000 7f43205b2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4318144910 con 0x7f43181595e0 2024-04-06T13:49:53.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.852+0000 7f43205b2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43181578a0 con 0x7f4318159a60 2024-04-06T13:49:53.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.852+0000 7f431e34e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4304003b90 0x7f4318157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:53.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.852+0000 7f431e34e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4304003b90 0x7f4318157360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59346/0 (socket says 172.21.15.189:59346) 2024-04-06T13:49:53.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.852+0000 7f431e34e700 1 -- 172.21.15.189:0/3747543849 learned_addr learned my addr 172.21.15.189:0/3747543849 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:53.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.852+0000 7f431e34e700 1 -- 172.21.15.189:0/3747543849 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4318159a60 msgr2=0x7f43181f6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:53.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.852+0000 7f431eb4f700 1 --2- 172.21.15.189:0/3747543849 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4318159a60 0x7f43181f6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:53.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.852+0000 7f431e34e700 1 --2- 172.21.15.189:0/3747543849 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4318159a60 0x7f43181f6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:53.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.852+0000 7f431e34e700 1 -- 172.21.15.189:0/3747543849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43181595e0 msgr2=0x7f43181f6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:53.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.852+0000 7f431db4d700 1 --2- 172.21.15.189:0/3747543849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43181595e0 0x7f43181f6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:53.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.852+0000 7f431e34e700 1 --2- 172.21.15.189:0/3747543849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43181595e0 0x7f43181f6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:53.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.852+0000 7f431e34e700 1 -- 172.21.15.189:0/3747543849 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4318144b60 con 0x7f4304003b90 2024-04-06T13:49:53.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.853+0000 7f431e34e700 1 --2- 172.21.15.189:0/3747543849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4304003b90 0x7f4318157360 secure :-1 s=READY pgs=3081 cs=0 l=1 rev1=1 crypto rx=0x7f430800e4b0 tx=0x7f43080047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:53.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.853+0000 7f430effd700 1 -- 172.21.15.189:0/3747543849 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430800e6c0 con 0x7f4304003b90 2024-04-06T13:49:53.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.853+0000 7f430effd700 1 -- 172.21.15.189:0/3747543849 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4308003ad0 con 0x7f4304003b90 2024-04-06T13:49:53.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.853+0000 7f43205b2700 1 -- 172.21.15.189:0/3747543849 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4318144e40 con 0x7f4304003b90 2024-04-06T13:49:53.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.853+0000 7f430effd700 1 -- 172.21.15.189:0/3747543849 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308006e60 con 0x7f4304003b90 2024-04-06T13:49:53.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.853+0000 7f43205b2700 1 -- 172.21.15.189:0/3747543849 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4318145400 con 0x7f4304003b90 2024-04-06T13:49:53.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.854+0000 7f43205b2700 1 -- 172.21.15.189:0/3747543849 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4318145590 con 0x7f4304003b90 2024-04-06T13:49:53.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.854+0000 7f430effd700 1 -- 172.21.15.189:0/3747543849 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f4308019440 con 0x7f4304003b90 2024-04-06T13:49:53.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.854+0000 7f430effd700 1 --2- 172.21.15.189:0/3747543849 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42fc06adb0 0x7f42fc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:53.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.854+0000 7f430effd700 1 -- 172.21.15.189:0/3747543849 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(282..282 src has 1..282) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f430808a3f0 con 0x7f4304003b90 2024-04-06T13:49:53.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.855+0000 7f431db4d700 1 --2- 172.21.15.189:0/3747543849 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42fc06adb0 0x7f42fc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:53.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.855+0000 7f431db4d700 1 --2- 172.21.15.189:0/3747543849 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42fc06adb0 0x7f42fc06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4318072b60 tx=0x7f4314009040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:53.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:53.859+0000 7f430effd700 1 -- 172.21.15.189:0/3747543849 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f43080554e0 con 0x7f4304003b90 2024-04-06T13:49:54.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.054+0000 7f43205b2700 1 -- 172.21.15.189:0/3747543849 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43180cf390 con 0x7f4304003b90 2024-04-06T13:49:54.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.055+0000 7f430effd700 1 -- 172.21.15.189:0/3747543849 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v282) v1 ==== 74+0+16437 (secure 0 0 0) 0x7f4308058b00 con 0x7f4304003b90 2024-04-06T13:49:54.057 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:54.057 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":282,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:49:53.687685+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:49:51.467569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"282","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-04-06T13:49:52.656784+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"282","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:49:54.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.058+0000 7f43205b2700 1 -- 172.21.15.189:0/3747543849 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42fc06adb0 msgr2=0x7f42fc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:54.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.058+0000 7f43205b2700 1 --2- 172.21.15.189:0/3747543849 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42fc06adb0 0x7f42fc06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4318072b60 tx=0x7f4314009040 comp rx=0 tx=0).stop 2024-04-06T13:49:54.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.058+0000 7f43205b2700 1 -- 172.21.15.189:0/3747543849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4304003b90 msgr2=0x7f4318157360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:54.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.058+0000 7f43205b2700 1 --2- 172.21.15.189:0/3747543849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4304003b90 0x7f4318157360 secure :-1 s=READY pgs=3081 cs=0 l=1 rev1=1 crypto rx=0x7f430800e4b0 tx=0x7f43080047c0 comp rx=0 tx=0).stop 2024-04-06T13:49:54.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.058+0000 7f43205b2700 1 -- 172.21.15.189:0/3747543849 shutdown_connections 2024-04-06T13:49:54.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.058+0000 7f43205b2700 1 --2- 172.21.15.189:0/3747543849 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42fc06adb0 0x7f42fc06d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:54.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.058+0000 7f43205b2700 1 --2- 172.21.15.189:0/3747543849 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4318159a60 0x7f43181f6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:54.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.058+0000 7f43205b2700 1 --2- 172.21.15.189:0/3747543849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4304003b90 0x7f4318157360 unknown :-1 s=CLOSED pgs=3081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:54.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.058+0000 7f43205b2700 1 --2- 172.21.15.189:0/3747543849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43181595e0 0x7f43181f6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:54.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.058+0000 7f43205b2700 1 -- 172.21.15.189:0/3747543849 >> 172.21.15.189:0/3747543849 conn(0x7f43180d00b0 msgr2=0x7f431814f830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:54.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.058+0000 7f43205b2700 1 -- 172.21.15.189:0/3747543849 shutdown_connections 2024-04-06T13:49:54.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.058+0000 7f43205b2700 1 -- 172.21.15.189:0/3747543849 wait complete. 2024-04-06T13:49:54.072 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:49:54.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.192+0000 7f7bf1669700 1 Processor -- start 2024-04-06T13:49:54.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.192+0000 7f7bf1669700 1 -- start start 2024-04-06T13:49:54.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.193+0000 7f7bf1669700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7bec158fe0 0x7f7bec06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:54.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.193+0000 7f7bf1669700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bec15f9e0 con 0x7f7bec06ac70 2024-04-06T13:49:54.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.193+0000 7f7bf1669700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bec15f0d0 con 0x7f7bec15c2d0 2024-04-06T13:49:54.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.193+0000 7f7bf1669700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bec06b0f0 con 0x7f7bec158fe0 2024-04-06T13:49:54.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.193+0000 7f7beaffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7bec158fe0 0x7f7bec06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:54.196 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.193+0000 7f7bea7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7bec06ac70 0x7f7bec15bbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49900/0 (socket says 172.21.15.189:49900) 2024-04-06T13:49:54.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.193+0000 7f7bea7fc700 1 -- 172.21.15.189:0/3976695613 learned_addr learned my addr 172.21.15.189:0/3976695613 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:54.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.194+0000 7f7beaffd700 1 -- 172.21.15.189:0/3976695613 >> v1:172.21.15.192:6789/0 conn(0x7f7bec15c2d0 legacy=0x7f7bec0ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:54.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.194+0000 7f7beaffd700 1 -- 172.21.15.189:0/3976695613 >> v1:172.21.15.189:6789/0 conn(0x7f7bec06ac70 legacy=0x7f7bec15bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:54.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.194+0000 7f7be9ffb700 1 -- 172.21.15.189:0/3976695613 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835178290 0 0) 0x7f7bec15f9e0 con 0x7f7bec06ac70 2024-04-06T13:49:54.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.194+0000 7f7beaffd700 1 -- 172.21.15.189:0/3976695613 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bec15c8f0 con 0x7f7bec158fe0 2024-04-06T13:49:54.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.194+0000 7f7beaffd700 1 --2- 172.21.15.189:0/3976695613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bec158fe0 0x7f7bec06a730 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009840 tx=0x7f7be0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a93b4c3a4339e1b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:54.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.194+0000 7f7be9ffb700 1 -- 172.21.15.189:0/3976695613 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be0011830 con 0x7f7bec158fe0 2024-04-06T13:49:54.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.194+0000 7f7be9ffb700 1 -- 172.21.15.189:0/3976695613 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be0011990 con 0x7f7bec158fe0 2024-04-06T13:49:54.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.194+0000 7f7be9ffb700 1 -- 172.21.15.189:0/3976695613 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be0011d20 con 0x7f7bec158fe0 2024-04-06T13:49:54.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.194+0000 7f7bf1669700 1 -- 172.21.15.189:0/3976695613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bec158fe0 msgr2=0x7f7bec06a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:54.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.194+0000 7f7bf1669700 1 --2- 172.21.15.189:0/3976695613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bec158fe0 0x7f7bec06a730 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009840 tx=0x7f7be0009b50 comp rx=0 tx=0).stop 2024-04-06T13:49:54.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.194+0000 7f7bf1669700 1 -- 172.21.15.189:0/3976695613 shutdown_connections 2024-04-06T13:49:54.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.194+0000 7f7bf1669700 1 --2- 172.21.15.189:0/3976695613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bec158fe0 0x7f7bec06a730 unknown :-1 s=CLOSED pgs=3083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:54.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.194+0000 7f7bf1669700 1 -- 172.21.15.189:0/3976695613 >> 172.21.15.189:0/3976695613 conn(0x7f7bec0d00b0 msgr2=0x7f7bec0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:54.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.194+0000 7f7bf1669700 1 -- 172.21.15.189:0/3976695613 shutdown_connections 2024-04-06T13:49:54.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.195+0000 7f7bf1669700 1 -- 172.21.15.189:0/3976695613 wait complete. 2024-04-06T13:49:54.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.196+0000 7f7bf1669700 1 Processor -- start 2024-04-06T13:49:54.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.196+0000 7f7bf1669700 1 -- start start 2024-04-06T13:49:54.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.197+0000 7f7bf1669700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7bec06ac70 0x7f7bec0c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:54.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.197+0000 7f7bf1669700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bec158fe0 0x7f7bec157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:54.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.197+0000 7f7bf1669700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7bec15c2d0 0x7f7bec0c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:54.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.197+0000 7f7bf1669700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec15d8a0 con 0x7f7bec158fe0 2024-04-06T13:49:54.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.197+0000 7f7bf1669700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec1579d0 con 0x7f7bec15c2d0 2024-04-06T13:49:54.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.197+0000 7f7bf1669700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec140b00 con 0x7f7bec06ac70 2024-04-06T13:49:54.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.197+0000 7f7bea7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bec158fe0 0x7f7bec157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:54.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.197+0000 7f7bea7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bec158fe0 0x7f7bec157490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59360/0 (socket says 172.21.15.189:59360) 2024-04-06T13:49:54.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.197+0000 7f7bea7fc700 1 -- 172.21.15.189:0/1429417092 learned_addr learned my addr 172.21.15.189:0/1429417092 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:54.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.197+0000 7f7beb7fe700 1 --2- 172.21.15.189:0/1429417092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7bec15c2d0 0x7f7bec0c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:54.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.197+0000 7f7beaffd700 1 --2- 172.21.15.189:0/1429417092 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7bec06ac70 0x7f7bec0c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:54.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.197+0000 7f7bea7fc700 1 -- 172.21.15.189:0/1429417092 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7bec06ac70 msgr2=0x7f7bec0c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:54.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.197+0000 7f7bea7fc700 1 --2- 172.21.15.189:0/1429417092 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7bec06ac70 0x7f7bec0c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:54.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.197+0000 7f7bea7fc700 1 -- 172.21.15.189:0/1429417092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7bec15c2d0 msgr2=0x7f7bec0c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:54.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.197+0000 7f7bea7fc700 1 --2- 172.21.15.189:0/1429417092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7bec15c2d0 0x7f7bec0c9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:54.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.197+0000 7f7bea7fc700 1 -- 172.21.15.189:0/1429417092 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be00094a0 con 0x7f7bec158fe0 2024-04-06T13:49:54.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.197+0000 7f7beaffd700 1 --2- 172.21.15.189:0/1429417092 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7bec06ac70 0x7f7bec0c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:54.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.198+0000 7f7bea7fc700 1 --2- 172.21.15.189:0/1429417092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bec158fe0 0x7f7bec157490 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc00c520 tx=0x7f7bdc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:54.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.198+0000 7f7bd37fe700 1 -- 172.21.15.189:0/1429417092 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc004c80 con 0x7f7bec158fe0 2024-04-06T13:49:54.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.198+0000 7f7bd37fe700 1 -- 172.21.15.189:0/1429417092 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bdc004e00 con 0x7f7bec158fe0 2024-04-06T13:49:54.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.198+0000 7f7bf1669700 1 -- 172.21.15.189:0/1429417092 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bec15daf0 con 0x7f7bec158fe0 2024-04-06T13:49:54.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.198+0000 7f7bd37fe700 1 -- 172.21.15.189:0/1429417092 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc007310 con 0x7f7bec158fe0 2024-04-06T13:49:54.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.198+0000 7f7bf1669700 1 -- 172.21.15.189:0/1429417092 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bec15e0b0 con 0x7f7bec158fe0 2024-04-06T13:49:54.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.199+0000 7f7bf1669700 1 -- 172.21.15.189:0/1429417092 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bec15fe50 con 0x7f7bec158fe0 2024-04-06T13:49:54.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.199+0000 7f7bd37fe700 1 -- 172.21.15.189:0/1429417092 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7bdc007470 con 0x7f7bec158fe0 2024-04-06T13:49:54.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.200+0000 7f7bd37fe700 1 --2- 172.21.15.189:0/1429417092 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7bd806adb0 0x7f7bd806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:54.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.200+0000 7f7bd37fe700 1 -- 172.21.15.189:0/1429417092 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(282..282 src has 1..282) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7bdc0893d0 con 0x7f7bec158fe0 2024-04-06T13:49:54.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.200+0000 7f7beaffd700 1 --2- 172.21.15.189:0/1429417092 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7bd806adb0 0x7f7bd806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:54.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.200+0000 7f7beaffd700 1 --2- 172.21.15.189:0/1429417092 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7bd806adb0 0x7f7bd806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7be00090b0 tx=0x7f7be0008030 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:54.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.204+0000 7f7bd37fe700 1 -- 172.21.15.189:0/1429417092 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7bec15fe50 con 0x7f7bec158fe0 2024-04-06T13:49:54.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.400+0000 7f7bf1669700 1 -- 172.21.15.189:0/1429417092 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7bec15fe50 con 0x7f7bec158fe0 2024-04-06T13:49:54.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.401+0000 7f7bd37fe700 1 -- 172.21.15.189:0/1429417092 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v282) v1 ==== 74+0+16437 (secure 0 0 0) 0x7f7bec15fe50 con 0x7f7bec158fe0 2024-04-06T13:49:54.402 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:54.402 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":282,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:49:53.687685+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:49:51.467569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"282","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-04-06T13:49:52.656784+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"282","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:49:54.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.403+0000 7f7bf1669700 1 -- 172.21.15.189:0/1429417092 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7bd806adb0 msgr2=0x7f7bd806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:54.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.403+0000 7f7bf1669700 1 --2- 172.21.15.189:0/1429417092 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7bd806adb0 0x7f7bd806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7be00090b0 tx=0x7f7be0008030 comp rx=0 tx=0).stop 2024-04-06T13:49:54.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.403+0000 7f7bf1669700 1 -- 172.21.15.189:0/1429417092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bec158fe0 msgr2=0x7f7bec157490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:54.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.403+0000 7f7bf1669700 1 --2- 172.21.15.189:0/1429417092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bec158fe0 0x7f7bec157490 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc00c520 tx=0x7f7bdc00c8e0 comp rx=0 tx=0).stop 2024-04-06T13:49:54.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.403+0000 7f7bf1669700 1 -- 172.21.15.189:0/1429417092 shutdown_connections 2024-04-06T13:49:54.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.403+0000 7f7bf1669700 1 --2- 172.21.15.189:0/1429417092 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7bd806adb0 0x7f7bd806d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:54.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.403+0000 7f7bf1669700 1 --2- 172.21.15.189:0/1429417092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7bec15c2d0 0x7f7bec0c9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:54.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.403+0000 7f7bf1669700 1 --2- 172.21.15.189:0/1429417092 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7bec06ac70 0x7f7bec0c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:54.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.403+0000 7f7bf1669700 1 --2- 172.21.15.189:0/1429417092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bec158fe0 0x7f7bec157490 unknown :-1 s=CLOSED pgs=3084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:54.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.403+0000 7f7bf1669700 1 -- 172.21.15.189:0/1429417092 >> 172.21.15.189:0/1429417092 conn(0x7f7bec0d00b0 msgr2=0x7f7bec158250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:54.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.403+0000 7f7bf1669700 1 -- 172.21.15.189:0/1429417092 shutdown_connections 2024-04-06T13:49:54.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.403+0000 7f7bf1669700 1 -- 172.21.15.189:0/1429417092 wait complete. 2024-04-06T13:49:54.417 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-06T13:49:54.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.541+0000 7f96d86a3700 1 Processor -- start 2024-04-06T13:49:54.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.541+0000 7f96d86a3700 1 -- start start 2024-04-06T13:49:54.550 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d86a3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96d0159b20 0x7f96d006b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:54.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d86a3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f96d006eb80 0x7f96d00ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:54.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d86a3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96d006a8d0 con 0x7f96d00cdc60 2024-04-06T13:49:54.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d86a3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96d00ce620 con 0x7f96d006eb80 2024-04-06T13:49:54.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d86a3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96d0159f00 con 0x7f96d0159b20 2024-04-06T13:49:54.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d5c3e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f96d00cdc60 0x7f96d006e470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49906/0 (socket says 172.21.15.189:49906) 2024-04-06T13:49:54.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d5c3e700 1 -- 172.21.15.189:0/1520888356 learned_addr learned my addr 172.21.15.189:0/1520888356 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:54.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d543d700 1 -- 172.21.15.189:0/1520888356 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772132523 0 0) 0x7f96d006a8d0 con 0x7f96d00cdc60 2024-04-06T13:49:54.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d543d700 1 -- 172.21.15.189:0/1520888356 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96bc0035b0 con 0x7f96d00cdc60 2024-04-06T13:49:54.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d6c40700 1 --2- 172.21.15.189:0/1520888356 >> v2:172.21.15.192:3300/0 conn(0x7f96d006eb80 0x7f96d00ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:54.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d643f700 1 --2- 172.21.15.189:0/1520888356 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96d0159b20 0x7f96d006b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:54.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d543d700 1 -- 172.21.15.189:0/1520888356 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1642124232 0 0) 0x7f96bc0035b0 con 0x7f96d00cdc60 2024-04-06T13:49:54.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d543d700 1 -- 172.21.15.189:0/1520888356 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96d0159b20 msgr2=0x7f96d006b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:54.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d543d700 1 --2- 172.21.15.189:0/1520888356 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96d0159b20 0x7f96d006b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:54.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d543d700 1 -- 172.21.15.189:0/1520888356 >> v2:172.21.15.192:3300/0 conn(0x7f96d006eb80 msgr2=0x7f96d00ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:54.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d543d700 1 --2- 172.21.15.189:0/1520888356 >> v2:172.21.15.192:3300/0 conn(0x7f96d006eb80 0x7f96d00ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:54.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d543d700 1 -- 172.21.15.189:0/1520888356 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d006abb0 con 0x7f96d00cdc60 2024-04-06T13:49:54.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d543d700 1 -- 172.21.15.189:0/1520888356 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f96c0001670 con 0x7f96d00cdc60 2024-04-06T13:49:54.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d543d700 1 -- 172.21.15.189:0/1520888356 >> v1:172.21.15.189:6789/0 conn(0x7f96d00cdc60 legacy=0x7f96d006e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:54.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d543d700 1 --2- 172.21.15.189:0/1520888356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96bc003a00 0x7f96bc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:54.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.542+0000 7f96d543d700 1 --2- 172.21.15.189:0/1520888356 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96bc004850 0x7f96bc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:54.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.543+0000 7f96d543d700 1 --2- 172.21.15.189:0/1520888356 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96d0159b20 0x7f96bc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:54.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.543+0000 7f96d543d700 1 -- 172.21.15.189:0/1520888356 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d00ce620 con 0x7f96bc003a00 2024-04-06T13:49:54.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.543+0000 7f96d543d700 1 -- 172.21.15.189:0/1520888356 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d0159f00 con 0x7f96bc004850 2024-04-06T13:49:54.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.543+0000 7f96d543d700 1 -- 172.21.15.189:0/1520888356 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc00ba10 con 0x7f96d0159b20 2024-04-06T13:49:54.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.543+0000 7f96d643f700 1 --2- 172.21.15.189:0/1520888356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96bc003a00 0x7f96bc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:54.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.543+0000 7f96d5c3e700 1 --2- 172.21.15.189:0/1520888356 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96bc004850 0x7f96bc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:54.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.543+0000 7f96d6c40700 1 --2- 172.21.15.189:0/1520888356 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96d0159b20 0x7f96bc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:54.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.543+0000 7f96d643f700 1 -- 172.21.15.189:0/1520888356 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96d0159b20 msgr2=0x7f96bc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:54.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.543+0000 7f96d643f700 1 --2- 172.21.15.189:0/1520888356 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96d0159b20 0x7f96bc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:54.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.543+0000 7f96d643f700 1 -- 172.21.15.189:0/1520888356 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96bc004850 msgr2=0x7f96bc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:54.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.543+0000 7f96d643f700 1 --2- 172.21.15.189:0/1520888356 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96bc004850 0x7f96bc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:54.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.543+0000 7f96d643f700 1 -- 172.21.15.189:0/1520888356 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96bc0035b0 con 0x7f96bc003a00 2024-04-06T13:49:54.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.543+0000 7f96d5c3e700 1 --2- 172.21.15.189:0/1520888356 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96bc004850 0x7f96bc004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:54.553 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.543+0000 7f96d643f700 1 --2- 172.21.15.189:0/1520888356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96bc003a00 0x7f96bc003e20 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009b90 tx=0x7f96bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f82291436a5bf66 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:54.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.543+0000 7f96d543d700 1 -- 172.21.15.189:0/1520888356 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc00e6c0 con 0x7f96bc003a00 2024-04-06T13:49:54.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.543+0000 7f96d543d700 1 -- 172.21.15.189:0/1520888356 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96cc00e820 con 0x7f96bc003a00 2024-04-06T13:49:54.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.543+0000 7f96d543d700 1 -- 172.21.15.189:0/1520888356 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc00eb90 con 0x7f96bc003a00 2024-04-06T13:49:54.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.544+0000 7f96d86a3700 1 -- 172.21.15.189:0/1520888356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96bc003a00 msgr2=0x7f96bc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:54.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.544+0000 7f96d86a3700 1 --2- 172.21.15.189:0/1520888356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96bc003a00 0x7f96bc003e20 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009b90 tx=0x7f96bc005880 comp rx=0 tx=0).stop 2024-04-06T13:49:54.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.544+0000 7f96d6c40700 1 -- 172.21.15.189:0/1520888356 reap_dead start 2024-04-06T13:49:54.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.544+0000 7f96d86a3700 1 -- 172.21.15.189:0/1520888356 shutdown_connections 2024-04-06T13:49:54.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.544+0000 7f96d86a3700 1 -- 172.21.15.189:0/1520888356 >> 172.21.15.189:0/1520888356 conn(0x7f96d00cc290 msgr2=0x7f96d00cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:54.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.544+0000 7f96d86a3700 1 -- 172.21.15.189:0/1520888356 shutdown_connections 2024-04-06T13:49:54.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.544+0000 7f96d86a3700 1 -- 172.21.15.189:0/1520888356 wait complete. 2024-04-06T13:49:54.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.545+0000 7f96d86a3700 1 Processor -- start 2024-04-06T13:49:54.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.545+0000 7f96d86a3700 1 -- start start 2024-04-06T13:49:54.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96d86a3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96bc003a00 0x7f96d0152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:54.554 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96d86a3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96d00c92b0 0x7f96d00b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:54.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96d86a3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d00b42d0 0x7f96d014dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:54.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96d86a3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d00c6f90 con 0x7f96d00b42d0 2024-04-06T13:49:54.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96d86a3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d00c6980 con 0x7f96bc003a00 2024-04-06T13:49:54.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96d86a3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96d00c9790 con 0x7f96d00c92b0 2024-04-06T13:49:54.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96d6c40700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d00b42d0 0x7f96d014dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:54.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96d6c40700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d00b42d0 0x7f96d014dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59376/0 (socket says 172.21.15.189:59376) 2024-04-06T13:49:54.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96d6c40700 1 -- 172.21.15.189:0/4228002625 learned_addr learned my addr 172.21.15.189:0/4228002625 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:54.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96d643f700 1 --2- 172.21.15.189:0/4228002625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96bc003a00 0x7f96d0152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:54.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96d5c3e700 1 --2- 172.21.15.189:0/4228002625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96d00c92b0 0x7f96d00b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:54.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96d6c40700 1 -- 172.21.15.189:0/4228002625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96d00c92b0 msgr2=0x7f96d00b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:54.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96d6c40700 1 --2- 172.21.15.189:0/4228002625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96d00c92b0 0x7f96d00b2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:54.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96d6c40700 1 -- 172.21.15.189:0/4228002625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96bc003a00 msgr2=0x7f96d0152d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:54.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96d6c40700 1 --2- 172.21.15.189:0/4228002625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96bc003a00 0x7f96d0152d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:54.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96d6c40700 1 -- 172.21.15.189:0/4228002625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d00c6c20 con 0x7f96d00b42d0 2024-04-06T13:49:54.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96d6c40700 1 --2- 172.21.15.189:0/4228002625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d00b42d0 0x7f96d014dcb0 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 crypto rx=0x7f96d0072380 tx=0x7f96c80114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:54.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96c6ffd700 1 -- 172.21.15.189:0/4228002625 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c800e070 con 0x7f96d00b42d0 2024-04-06T13:49:54.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96c6ffd700 1 -- 172.21.15.189:0/4228002625 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c8003cf0 con 0x7f96d00b42d0 2024-04-06T13:49:54.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96d86a3700 1 -- 172.21.15.189:0/4228002625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d00c9a70 con 0x7f96d00b42d0 2024-04-06T13:49:54.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96c6ffd700 1 -- 172.21.15.189:0/4228002625 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c8003ea0 con 0x7f96d00b42d0 2024-04-06T13:49:54.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.546+0000 7f96d86a3700 1 -- 172.21.15.189:0/4228002625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96d00b34d0 con 0x7f96d00b42d0 2024-04-06T13:49:54.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.547+0000 7f96d86a3700 1 -- 172.21.15.189:0/4228002625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96d00c7140 con 0x7f96d00b42d0 2024-04-06T13:49:54.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.548+0000 7f96c6ffd700 1 -- 172.21.15.189:0/4228002625 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f96c8011960 con 0x7f96d00b42d0 2024-04-06T13:49:54.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.548+0000 7f96c6ffd700 1 --2- 172.21.15.189:0/4228002625 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f96b406adb0 0x7f96b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:54.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.549+0000 7f96d643f700 1 --2- 172.21.15.189:0/4228002625 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f96b406adb0 0x7f96b406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:54.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.549+0000 7f96c6ffd700 1 -- 172.21.15.189:0/4228002625 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(282..282 src has 1..282) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f96c80897e0 con 0x7f96d00b42d0 2024-04-06T13:49:54.556 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.549+0000 7f96d643f700 1 --2- 172.21.15.189:0/4228002625 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f96b406adb0 0x7f96b406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f96cc0090b0 tx=0x7f96cc01f040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:54.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.553+0000 7f96c6ffd700 1 -- 172.21.15.189:0/4228002625 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f96c8054600 con 0x7f96d00b42d0 2024-04-06T13:49:54.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:54.796+0000 7f96d86a3700 1 -- 172.21.15.189:0/4228002625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f96d00b34d0 con 0x7f96d00b42d0 2024-04-06T13:49:55.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.699+0000 7f96c6ffd700 1 -- 172.21.15.189:0/4228002625 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v138) v1 ==== 118+0+0 (secure 0 0 0) 0x7f96d00b34d0 con 0x7f96d00b42d0 2024-04-06T13:49:55.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.703+0000 7f96d86a3700 1 -- 172.21.15.189:0/4228002625 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f96b406adb0 msgr2=0x7f96b406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:55.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.703+0000 7f96d86a3700 1 --2- 172.21.15.189:0/4228002625 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f96b406adb0 0x7f96b406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f96cc0090b0 tx=0x7f96cc01f040 comp rx=0 tx=0).stop 2024-04-06T13:49:55.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.703+0000 7f96d86a3700 1 -- 172.21.15.189:0/4228002625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d00b42d0 msgr2=0x7f96d014dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:55.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.703+0000 7f96d86a3700 1 --2- 172.21.15.189:0/4228002625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d00b42d0 0x7f96d014dcb0 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 crypto rx=0x7f96d0072380 tx=0x7f96c80114b0 comp rx=0 tx=0).stop 2024-04-06T13:49:55.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.703+0000 7f96d86a3700 1 -- 172.21.15.189:0/4228002625 shutdown_connections 2024-04-06T13:49:55.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.703+0000 7f96d86a3700 1 --2- 172.21.15.189:0/4228002625 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f96b406adb0 0x7f96b406d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:55.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.703+0000 7f96d86a3700 1 --2- 172.21.15.189:0/4228002625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d00b42d0 0x7f96d014dcb0 unknown :-1 s=CLOSED pgs=3087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:55.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.703+0000 7f96d86a3700 1 --2- 172.21.15.189:0/4228002625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96bc003a00 0x7f96d0152d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:55.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.703+0000 7f96d86a3700 1 --2- 172.21.15.189:0/4228002625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96d00c92b0 0x7f96d00b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:55.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.703+0000 7f96d86a3700 1 -- 172.21.15.189:0/4228002625 >> 172.21.15.189:0/4228002625 conn(0x7f96d00cc290 msgr2=0x7f96d00af2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:55.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.703+0000 7f96d86a3700 1 -- 172.21.15.189:0/4228002625 shutdown_connections 2024-04-06T13:49:55.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.703+0000 7f96d86a3700 1 -- 172.21.15.189:0/4228002625 wait complete. 2024-04-06T13:49:55.719 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-06T13:49:55.719 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-06T13:49:55.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.849+0000 7fdb586d0700 1 Processor -- start 2024-04-06T13:49:55.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.849+0000 7fdb586d0700 1 -- start start 2024-04-06T13:49:55.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.850+0000 7fdb586d0700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdb5015d7b0 0x7fdb500b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:55.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.850+0000 7fdb586d0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb5006a730 0x7fdb5006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:55.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.850+0000 7fdb586d0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb500b3990 con 0x7fdb5015ddf0 2024-04-06T13:49:55.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.850+0000 7fdb586d0700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb5006b0f0 con 0x7fdb5015d7b0 2024-04-06T13:49:55.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.850+0000 7fdb586d0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb500b3060 con 0x7fdb5006a730 2024-04-06T13:49:55.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.850+0000 7fdb5646c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdb5015d7b0 0x7fdb500b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:55.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.850+0000 7fdb55c6b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb5006a730 0x7fdb5006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:55.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.850+0000 7fdb5646c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdb5015d7b0 0x7fdb500b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:39258/0 (socket says 172.21.15.189:39258) 2024-04-06T13:49:55.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.850+0000 7fdb5646c700 1 -- 172.21.15.189:0/2204509995 learned_addr learned my addr 172.21.15.189:0/2204509995 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:55.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.850+0000 7fdb55c6b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb5006a730 0x7fdb5006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:34208/0 (socket says 172.21.15.189:34208) 2024-04-06T13:49:55.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.851+0000 7fdb55c6b700 1 -- 172.21.15.189:0/2204509995 >> v2:172.21.15.192:3300/0 conn(0x7fdb5015d7b0 msgr2=0x7fdb500b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:55.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.851+0000 7fdb55c6b700 1 --2- 172.21.15.189:0/2204509995 >> v2:172.21.15.192:3300/0 conn(0x7fdb5015d7b0 0x7fdb500b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:55.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.851+0000 7fdb55c6b700 1 -- 172.21.15.189:0/2204509995 >> v1:172.21.15.192:6789/0 conn(0x7fdb5015ddf0 legacy=0x7fdb5015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:55.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.851+0000 7fdb55c6b700 1 -- 172.21.15.189:0/2204509995 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb500b3340 con 0x7fdb5006a730 2024-04-06T13:49:55.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.851+0000 7fdb5546a700 1 -- 172.21.15.189:0/2204509995 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772926109 0 0) 0x7fdb500b3990 con 0x7fdb5015ddf0 2024-04-06T13:49:55.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.851+0000 7fdb5646c700 1 --2- 172.21.15.189:0/2204509995 >> v2:172.21.15.192:3300/0 conn(0x7fdb5015d7b0 0x7fdb500b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:55.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.851+0000 7fdb55c6b700 1 --2- 172.21.15.189:0/2204509995 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb5006a730 0x7fdb5006abb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009840 tx=0x7fdb40009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61e4659aac051ad1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:55.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.851+0000 7fdb5546a700 1 -- 172.21.15.189:0/2204509995 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb40011830 con 0x7fdb5006a730 2024-04-06T13:49:55.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.851+0000 7fdb5546a700 1 -- 172.21.15.189:0/2204509995 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb40011990 con 0x7fdb5006a730 2024-04-06T13:49:55.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.851+0000 7fdb5546a700 1 -- 172.21.15.189:0/2204509995 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb40011d20 con 0x7fdb5006a730 2024-04-06T13:49:55.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.851+0000 7fdb586d0700 1 -- 172.21.15.189:0/2204509995 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb5006a730 msgr2=0x7fdb5006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:55.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.851+0000 7fdb586d0700 1 --2- 172.21.15.189:0/2204509995 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb5006a730 0x7fdb5006abb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009840 tx=0x7fdb40009b50 comp rx=0 tx=0).stop 2024-04-06T13:49:55.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.851+0000 7fdb586d0700 1 -- 172.21.15.189:0/2204509995 shutdown_connections 2024-04-06T13:49:55.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.851+0000 7fdb586d0700 1 --2- 172.21.15.189:0/2204509995 >> v2:172.21.15.192:3300/0 conn(0x7fdb5015d7b0 0x7fdb500b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:55.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.851+0000 7fdb586d0700 1 --2- 172.21.15.189:0/2204509995 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb5006a730 0x7fdb5006abb0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:55.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.851+0000 7fdb586d0700 1 -- 172.21.15.189:0/2204509995 >> 172.21.15.189:0/2204509995 conn(0x7fdb500d00b0 msgr2=0x7fdb500d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:55.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.852+0000 7fdb586d0700 1 -- 172.21.15.189:0/2204509995 shutdown_connections 2024-04-06T13:49:55.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.852+0000 7fdb586d0700 1 -- 172.21.15.189:0/2204509995 wait complete. 2024-04-06T13:49:55.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.853+0000 7fdb586d0700 1 Processor -- start 2024-04-06T13:49:55.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.853+0000 7fdb586d0700 1 -- start start 2024-04-06T13:49:55.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.853+0000 7fdb586d0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb5006a730 0x7fdb5015c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:55.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.853+0000 7fdb586d0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb5015d7b0 0x7fdb5015c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:55.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.853+0000 7fdb586d0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb5015ddf0 0x7fdb500b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:55.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.853+0000 7fdb586d0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb50154ed0 con 0x7fdb5015d7b0 2024-04-06T13:49:55.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.853+0000 7fdb586d0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb500b18d0 con 0x7fdb5015ddf0 2024-04-06T13:49:55.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.853+0000 7fdb586d0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb50155170 con 0x7fdb5006a730 2024-04-06T13:49:55.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.854+0000 7fdb55c6b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb5015d7b0 0x7fdb5015c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:55.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.854+0000 7fdb55c6b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb5015d7b0 0x7fdb5015c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59388/0 (socket says 172.21.15.189:59388) 2024-04-06T13:49:55.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.854+0000 7fdb55c6b700 1 -- 172.21.15.189:0/1546330520 learned_addr learned my addr 172.21.15.189:0/1546330520 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:55.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.854+0000 7fdb5646c700 1 --2- 172.21.15.189:0/1546330520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb5006a730 0x7fdb5015c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:55.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.854+0000 7fdb56c6d700 1 --2- 172.21.15.189:0/1546330520 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb5015ddf0 0x7fdb500b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:55.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.854+0000 7fdb55c6b700 1 -- 172.21.15.189:0/1546330520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb5006a730 msgr2=0x7fdb5015c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:55.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.854+0000 7fdb55c6b700 1 --2- 172.21.15.189:0/1546330520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb5006a730 0x7fdb5015c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:55.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.854+0000 7fdb55c6b700 1 -- 172.21.15.189:0/1546330520 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb5015ddf0 msgr2=0x7fdb500b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:55.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.854+0000 7fdb55c6b700 1 --2- 172.21.15.189:0/1546330520 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb5015ddf0 0x7fdb500b0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:55.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.854+0000 7fdb55c6b700 1 -- 172.21.15.189:0/1546330520 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4c009550 con 0x7fdb5015d7b0 2024-04-06T13:49:55.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.854+0000 7fdb55c6b700 1 --2- 172.21.15.189:0/1546330520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb5015d7b0 0x7fdb5015c8c0 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009fd0 tx=0x7fdb40004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:55.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.854+0000 7fdb46ffd700 1 -- 172.21.15.189:0/1546330520 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb40004550 con 0x7fdb5015d7b0 2024-04-06T13:49:55.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.854+0000 7fdb46ffd700 1 -- 172.21.15.189:0/1546330520 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4001f070 con 0x7fdb5015d7b0 2024-04-06T13:49:55.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.854+0000 7fdb586d0700 1 -- 172.21.15.189:0/1546330520 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb400094a0 con 0x7fdb5015d7b0 2024-04-06T13:49:55.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.854+0000 7fdb46ffd700 1 -- 172.21.15.189:0/1546330520 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb400038e0 con 0x7fdb5015d7b0 2024-04-06T13:49:55.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.854+0000 7fdb586d0700 1 -- 172.21.15.189:0/1546330520 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb50155730 con 0x7fdb5015d7b0 2024-04-06T13:49:55.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.855+0000 7fdb586d0700 1 -- 172.21.15.189:0/1546330520 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb50067bb0 con 0x7fdb5015d7b0 2024-04-06T13:49:55.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.856+0000 7fdb46ffd700 1 -- 172.21.15.189:0/1546330520 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fdb40003a40 con 0x7fdb5015d7b0 2024-04-06T13:49:55.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.856+0000 7fdb46ffd700 1 --2- 172.21.15.189:0/1546330520 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdb3406adb0 0x7fdb3406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:55.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.856+0000 7fdb46ffd700 1 -- 172.21.15.189:0/1546330520 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdb40021070 con 0x7fdb5015d7b0 2024-04-06T13:49:55.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.856+0000 7fdb5646c700 1 --2- 172.21.15.189:0/1546330520 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdb3406adb0 0x7fdb3406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:55.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.856+0000 7fdb5646c700 1 --2- 172.21.15.189:0/1546330520 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdb3406adb0 0x7fdb3406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c006d30 tx=0x7fdb4c0092c0 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:55.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:55.861+0000 7fdb46ffd700 1 -- 172.21.15.189:0/1546330520 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fdb40055e50 con 0x7fdb5015d7b0 2024-04-06T13:49:56.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.100+0000 7fdb586d0700 1 -- 172.21.15.189:0/1546330520 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fdb50074990 con 0x7fdb5015d7b0 2024-04-06T13:49:56.709 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.708+0000 7fdb46ffd700 1 -- 172.21.15.189:0/1546330520 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v139) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdb4002b020 con 0x7fdb5015d7b0 2024-04-06T13:49:56.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.710+0000 7fdb586d0700 1 -- 172.21.15.189:0/1546330520 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdb3406adb0 msgr2=0x7fdb3406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:56.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.710+0000 7fdb586d0700 1 --2- 172.21.15.189:0/1546330520 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdb3406adb0 0x7fdb3406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c006d30 tx=0x7fdb4c0092c0 comp rx=0 tx=0).stop 2024-04-06T13:49:56.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.710+0000 7fdb586d0700 1 -- 172.21.15.189:0/1546330520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb5015d7b0 msgr2=0x7fdb5015c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:56.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.710+0000 7fdb586d0700 1 --2- 172.21.15.189:0/1546330520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb5015d7b0 0x7fdb5015c8c0 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009fd0 tx=0x7fdb40004750 comp rx=0 tx=0).stop 2024-04-06T13:49:56.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.711+0000 7fdb586d0700 1 -- 172.21.15.189:0/1546330520 shutdown_connections 2024-04-06T13:49:56.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.711+0000 7fdb586d0700 1 --2- 172.21.15.189:0/1546330520 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdb3406adb0 0x7fdb3406d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:56.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.711+0000 7fdb586d0700 1 --2- 172.21.15.189:0/1546330520 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb5015ddf0 0x7fdb500b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:56.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.711+0000 7fdb586d0700 1 --2- 172.21.15.189:0/1546330520 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdb5006a730 0x7fdb5015c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:56.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.711+0000 7fdb586d0700 1 --2- 172.21.15.189:0/1546330520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb5015d7b0 0x7fdb5015c8c0 unknown :-1 s=CLOSED pgs=3088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:56.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.711+0000 7fdb586d0700 1 -- 172.21.15.189:0/1546330520 >> 172.21.15.189:0/1546330520 conn(0x7fdb500d00b0 msgr2=0x7fdb5015f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:56.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.711+0000 7fdb586d0700 1 -- 172.21.15.189:0/1546330520 shutdown_connections 2024-04-06T13:49:56.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.711+0000 7fdb586d0700 1 -- 172.21.15.189:0/1546330520 wait complete. 2024-04-06T13:49:56.727 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-06T13:49:56.727 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:49:56.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.854+0000 7fc1f3c45700 1 Processor -- start 2024-04-06T13:49:56.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.854+0000 7fc1f3c45700 1 -- start start 2024-04-06T13:49:56.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.855+0000 7fc1f3c45700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc1ec15a550 0x7fc1ec0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:56.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.855+0000 7fc1f3c45700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc1ec0b4d70 0x7fc1ec0b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:56.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.855+0000 7fc1f3c45700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1ec0cdbe0 con 0x7fc1ec0cf900 2024-04-06T13:49:56.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.855+0000 7fc1f3c45700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1ec15a940 con 0x7fc1ec15a550 2024-04-06T13:49:56.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.855+0000 7fc1f3c45700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1ec15ac00 con 0x7fc1ec0b4d70 2024-04-06T13:49:56.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.855+0000 7fc1f19e1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc1ec15a550 0x7fc1ec0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:56.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.855+0000 7fc1f19e1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc1ec15a550 0x7fc1ec0cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59394/0 (socket says 172.21.15.189:59394) 2024-04-06T13:49:56.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.855+0000 7fc1f19e1700 1 -- 172.21.15.189:0/2445076056 learned_addr learned my addr 172.21.15.189:0/2445076056 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:56.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.856+0000 7fc1f21e2700 1 --2- 172.21.15.189:0/2445076056 >> v2:172.21.15.192:3300/0 conn(0x7fc1ec0b4d70 0x7fc1ec0b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:56.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.856+0000 7fc1f19e1700 1 -- 172.21.15.189:0/2445076056 >> v2:172.21.15.192:3300/0 conn(0x7fc1ec0b4d70 msgr2=0x7fc1ec0b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:56.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.856+0000 7fc1f19e1700 1 --2- 172.21.15.189:0/2445076056 >> v2:172.21.15.192:3300/0 conn(0x7fc1ec0b4d70 0x7fc1ec0b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:56.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.856+0000 7fc1f19e1700 1 -- 172.21.15.189:0/2445076056 >> v1:172.21.15.192:6789/0 conn(0x7fc1ec0cf900 legacy=0x7fc1ec0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:56.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.856+0000 7fc1f19e1700 1 -- 172.21.15.189:0/2445076056 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ec15aee0 con 0x7fc1ec15a550 2024-04-06T13:49:56.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.856+0000 7fc1f21e2700 1 --2- 172.21.15.189:0/2445076056 >> v2:172.21.15.192:3300/0 conn(0x7fc1ec0b4d70 0x7fc1ec0b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:56.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.856+0000 7fc1f19e1700 1 --2- 172.21.15.189:0/2445076056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1ec15a550 0x7fc1ec0cf3c0 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec072830 tx=0x7fc1dc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e1bfd01a690a01f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:56.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.856+0000 7fc1f09df700 1 -- 172.21.15.189:0/2445076056 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc012b80 con 0x7fc1ec15a550 2024-04-06T13:49:56.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.856+0000 7fc1f09df700 1 -- 172.21.15.189:0/2445076056 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1dc012ce0 con 0x7fc1ec15a550 2024-04-06T13:49:56.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.856+0000 7fc1f09df700 1 -- 172.21.15.189:0/2445076056 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc004250 con 0x7fc1ec15a550 2024-04-06T13:49:56.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.856+0000 7fc1f3c45700 1 -- 172.21.15.189:0/2445076056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1ec15a550 msgr2=0x7fc1ec0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:56.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.856+0000 7fc1f3c45700 1 --2- 172.21.15.189:0/2445076056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1ec15a550 0x7fc1ec0cf3c0 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec072830 tx=0x7fc1dc009bd0 comp rx=0 tx=0).stop 2024-04-06T13:49:56.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.856+0000 7fc1f3c45700 1 -- 172.21.15.189:0/2445076056 shutdown_connections 2024-04-06T13:49:56.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.856+0000 7fc1f3c45700 1 --2- 172.21.15.189:0/2445076056 >> v2:172.21.15.192:3300/0 conn(0x7fc1ec0b4d70 0x7fc1ec0b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:56.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.856+0000 7fc1f3c45700 1 --2- 172.21.15.189:0/2445076056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1ec15a550 0x7fc1ec0cf3c0 unknown :-1 s=CLOSED pgs=3089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:56.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.856+0000 7fc1f3c45700 1 -- 172.21.15.189:0/2445076056 >> 172.21.15.189:0/2445076056 conn(0x7fc1ec0d00b0 msgr2=0x7fc1ec0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:56.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.857+0000 7fc1f3c45700 1 -- 172.21.15.189:0/2445076056 shutdown_connections 2024-04-06T13:49:56.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.857+0000 7fc1f3c45700 1 -- 172.21.15.189:0/2445076056 wait complete. 2024-04-06T13:49:56.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.858+0000 7fc1f3c45700 1 Processor -- start 2024-04-06T13:49:56.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.858+0000 7fc1f3c45700 1 -- start start 2024-04-06T13:49:56.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.858+0000 7fc1f3c45700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1ec0b4d70 0x7fc1ec1535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:56.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.858+0000 7fc1f3c45700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1ec0cf900 0x7fc1ec0b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:56.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.858+0000 7fc1f3c45700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1ec1527b0 0x7fc1ec0b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:56.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.858+0000 7fc1f3c45700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec0af390 con 0x7fc1ec0b4d70 2024-04-06T13:49:56.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.858+0000 7fc1f3c45700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec1ee730 con 0x7fc1ec1527b0 2024-04-06T13:49:56.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.858+0000 7fc1f3c45700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec153af0 con 0x7fc1ec0cf900 2024-04-06T13:49:56.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.859+0000 7fc1f19e1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1ec0b4d70 0x7fc1ec1535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:56.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.859+0000 7fc1f19e1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1ec0b4d70 0x7fc1ec1535b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59406/0 (socket says 172.21.15.189:59406) 2024-04-06T13:49:56.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.859+0000 7fc1f19e1700 1 -- 172.21.15.189:0/3051915590 learned_addr learned my addr 172.21.15.189:0/3051915590 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:56.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.859+0000 7fc1f21e2700 1 --2- 172.21.15.189:0/3051915590 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1ec1527b0 0x7fc1ec0b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:56.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.859+0000 7fc1f11e0700 1 --2- 172.21.15.189:0/3051915590 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1ec0cf900 0x7fc1ec0b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:56.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.859+0000 7fc1f19e1700 1 -- 172.21.15.189:0/3051915590 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1ec0cf900 msgr2=0x7fc1ec0b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:56.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.859+0000 7fc1f19e1700 1 --2- 172.21.15.189:0/3051915590 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1ec0cf900 0x7fc1ec0b0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:56.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.859+0000 7fc1f19e1700 1 -- 172.21.15.189:0/3051915590 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1ec1527b0 msgr2=0x7fc1ec0b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:56.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.859+0000 7fc1f19e1700 1 --2- 172.21.15.189:0/3051915590 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1ec1527b0 0x7fc1ec0b12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:56.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.859+0000 7fc1f19e1700 1 -- 172.21.15.189:0/3051915590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1dc009550 con 0x7fc1ec0b4d70 2024-04-06T13:49:56.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.859+0000 7fc1f19e1700 1 --2- 172.21.15.189:0/3051915590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1ec0b4d70 0x7fc1ec1535b0 secure :-1 s=READY pgs=3090 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec06c820 tx=0x7fc1dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:56.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.859+0000 7fc1e27fc700 1 -- 172.21.15.189:0/3051915590 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc021070 con 0x7fc1ec0b4d70 2024-04-06T13:49:56.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.859+0000 7fc1e27fc700 1 -- 172.21.15.189:0/3051915590 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1dc003e40 con 0x7fc1ec0b4d70 2024-04-06T13:49:56.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.859+0000 7fc1f3c45700 1 -- 172.21.15.189:0/3051915590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ec153dc0 con 0x7fc1ec0b4d70 2024-04-06T13:49:56.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.859+0000 7fc1e27fc700 1 -- 172.21.15.189:0/3051915590 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc004230 con 0x7fc1ec0b4d70 2024-04-06T13:49:56.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.859+0000 7fc1f3c45700 1 -- 172.21.15.189:0/3051915590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ec154380 con 0x7fc1ec0b4d70 2024-04-06T13:49:56.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.860+0000 7fc1f3c45700 1 -- 172.21.15.189:0/3051915590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ec154570 con 0x7fc1ec0b4d70 2024-04-06T13:49:56.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.861+0000 7fc1e27fc700 1 -- 172.21.15.189:0/3051915590 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fc1dc025410 con 0x7fc1ec0b4d70 2024-04-06T13:49:56.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.861+0000 7fc1e27fc700 1 --2- 172.21.15.189:0/3051915590 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc1d006adb0 0x7fc1d006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:56.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.861+0000 7fc1e27fc700 1 -- 172.21.15.189:0/3051915590 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc1dc08e3f0 con 0x7fc1ec0b4d70 2024-04-06T13:49:56.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.861+0000 7fc1f11e0700 1 --2- 172.21.15.189:0/3051915590 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc1d006adb0 0x7fc1d006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:56.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.862+0000 7fc1f11e0700 1 --2- 172.21.15.189:0/3051915590 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc1d006adb0 0x7fc1d006d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec0cc580 tx=0x7fc1e8007ec0 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:56.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:56.866+0000 7fc1e27fc700 1 -- 172.21.15.189:0/3051915590 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc1ec154570 con 0x7fc1ec0b4d70 2024-04-06T13:49:57.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.102+0000 7fc1f3c45700 1 -- 172.21.15.189:0/3051915590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1ec154570 con 0x7fc1ec0b4d70 2024-04-06T13:49:57.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.102+0000 7fc1e27fc700 1 -- 172.21.15.189:0/3051915590 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4759 (secure 0 0 0) 0x7fc1ec154570 con 0x7fc1ec0b4d70 2024-04-06T13:49:57.102 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:57.103 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":139,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11827,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/819921213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":819921213},{"type":"v1","addr":"172.21.15.192:6835","nonce":819921213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134},{"gid":13368,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3616349784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3616349784},{"type":"v1","addr":"172.21.15.189:6836","nonce":3616349784}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134},{"gid":13371,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3640528525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3640528525},{"type":"v1","addr":"172.21.15.189:6837","nonce":3640528525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:49:53.688018+0000","modified":"2024-04-06T13:49:56.698920+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13380},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13380":{"gid":13380,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6834/3894184871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3894184871},{"type":"v1","addr":"172.21.15.192:6834","nonce":3894184871}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-04-06T13:49:57.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.104+0000 7fc1f3c45700 1 -- 172.21.15.189:0/3051915590 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc1d006adb0 msgr2=0x7fc1d006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:57.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.104+0000 7fc1f3c45700 1 --2- 172.21.15.189:0/3051915590 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc1d006adb0 0x7fc1d006d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec0cc580 tx=0x7fc1e8007ec0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.104+0000 7fc1f3c45700 1 -- 172.21.15.189:0/3051915590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1ec0b4d70 msgr2=0x7fc1ec1535b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:57.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.104+0000 7fc1f3c45700 1 --2- 172.21.15.189:0/3051915590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1ec0b4d70 0x7fc1ec1535b0 secure :-1 s=READY pgs=3090 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec06c820 tx=0x7fc1dc004750 comp rx=0 tx=0).stop 2024-04-06T13:49:57.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.104+0000 7fc1f3c45700 1 -- 172.21.15.189:0/3051915590 shutdown_connections 2024-04-06T13:49:57.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.104+0000 7fc1f3c45700 1 --2- 172.21.15.189:0/3051915590 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc1d006adb0 0x7fc1d006d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.104+0000 7fc1f3c45700 1 --2- 172.21.15.189:0/3051915590 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1ec1527b0 0x7fc1ec0b12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.104+0000 7fc1f3c45700 1 --2- 172.21.15.189:0/3051915590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1ec0b4d70 0x7fc1ec1535b0 unknown :-1 s=CLOSED pgs=3090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.104+0000 7fc1f3c45700 1 --2- 172.21.15.189:0/3051915590 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1ec0cf900 0x7fc1ec0b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.104+0000 7fc1f3c45700 1 -- 172.21.15.189:0/3051915590 >> 172.21.15.189:0/3051915590 conn(0x7fc1ec0d00b0 msgr2=0x7fc1ec06eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:57.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.104+0000 7fc1f3c45700 1 -- 172.21.15.189:0/3051915590 shutdown_connections 2024-04-06T13:49:57.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.104+0000 7fc1f3c45700 1 -- 172.21.15.189:0/3051915590 wait complete. 2024-04-06T13:49:57.106 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 139 2024-04-06T13:49:57.117 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:49:57.238 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.238+0000 7f2fc70d8700 1 Processor -- start 2024-04-06T13:49:57.239 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.238+0000 7f2fc70d8700 1 -- start start 2024-04-06T13:49:57.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.239+0000 7f2fc70d8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2fc015d7b0 0x7f2fc00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.239+0000 7f2fc70d8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fc006a730 0x7f2fc006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.239+0000 7f2fc70d8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2fc015ddf0 0x7f2fc015e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.239+0000 7f2fc70d8700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc006b0f0 con 0x7f2fc015ddf0 2024-04-06T13:49:57.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.239+0000 7f2fc70d8700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc00b3990 con 0x7f2fc015d7b0 2024-04-06T13:49:57.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.239+0000 7f2fc70d8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc00b3060 con 0x7f2fc006a730 2024-04-06T13:49:57.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.239+0000 7f2fc5675700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2fc015ddf0 0x7f2fc015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.239+0000 7f2fc5675700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2fc015ddf0 0x7f2fc015e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59410/0 (socket says 172.21.15.189:59410) 2024-04-06T13:49:57.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.239+0000 7f2fc5675700 1 -- 172.21.15.189:0/3538466729 learned_addr learned my addr 172.21.15.189:0/3538466729 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:57.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.240+0000 7f2fbffff700 1 --2- 172.21.15.189:0/3538466729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fc006a730 0x7f2fc006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.240+0000 7f2fc5675700 1 -- 172.21.15.189:0/3538466729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fc006a730 msgr2=0x7f2fc006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:57.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.240+0000 7f2fc5675700 1 --2- 172.21.15.189:0/3538466729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fc006a730 0x7f2fc006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.240+0000 7f2fc4e74700 1 --2- 172.21.15.189:0/3538466729 >> v2:172.21.15.192:3300/0 conn(0x7f2fc015d7b0 0x7f2fc00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.240+0000 7f2fc5675700 1 -- 172.21.15.189:0/3538466729 >> v2:172.21.15.192:3300/0 conn(0x7f2fc015d7b0 msgr2=0x7f2fc00b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:57.242 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.240+0000 7f2fc5675700 1 --2- 172.21.15.189:0/3538466729 >> v2:172.21.15.192:3300/0 conn(0x7f2fc015d7b0 0x7f2fc00b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.240+0000 7f2fc5675700 1 -- 172.21.15.189:0/3538466729 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc00b3340 con 0x7f2fc015ddf0 2024-04-06T13:49:57.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.240+0000 7f2fc5675700 1 --2- 172.21.15.189:0/3538466729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fc015ddf0 0x7f2fc015e280 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7f2fc00b6200 tx=0x7f2fb800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f71a0585be6570c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:57.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.240+0000 7f2fbf7fe700 1 -- 172.21.15.189:0/3538466729 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb800e070 con 0x7f2fc015ddf0 2024-04-06T13:49:57.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.240+0000 7f2fbf7fe700 1 -- 172.21.15.189:0/3538466729 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb800b9c0 con 0x7f2fc015ddf0 2024-04-06T13:49:57.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.240+0000 7f2fbf7fe700 1 -- 172.21.15.189:0/3538466729 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb800bd70 con 0x7f2fc015ddf0 2024-04-06T13:49:57.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.241+0000 7f2fc70d8700 1 -- 172.21.15.189:0/3538466729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fc015ddf0 msgr2=0x7f2fc015e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:57.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.241+0000 7f2fc70d8700 1 --2- 172.21.15.189:0/3538466729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fc015ddf0 0x7f2fc015e280 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7f2fc00b6200 tx=0x7f2fb800b330 comp rx=0 tx=0).stop 2024-04-06T13:49:57.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.241+0000 7f2fc70d8700 1 -- 172.21.15.189:0/3538466729 shutdown_connections 2024-04-06T13:49:57.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.241+0000 7f2fc70d8700 1 --2- 172.21.15.189:0/3538466729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fc015ddf0 0x7f2fc015e280 unknown :-1 s=CLOSED pgs=3091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.241+0000 7f2fc70d8700 1 --2- 172.21.15.189:0/3538466729 >> v2:172.21.15.192:3300/0 conn(0x7f2fc015d7b0 0x7f2fc00b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.241+0000 7f2fc70d8700 1 --2- 172.21.15.189:0/3538466729 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fc006a730 0x7f2fc006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.241+0000 7f2fc70d8700 1 -- 172.21.15.189:0/3538466729 >> 172.21.15.189:0/3538466729 conn(0x7f2fc00d00b0 msgr2=0x7f2fc00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:57.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.241+0000 7f2fc70d8700 1 -- 172.21.15.189:0/3538466729 shutdown_connections 2024-04-06T13:49:57.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.241+0000 7f2fc70d8700 1 -- 172.21.15.189:0/3538466729 wait complete. 2024-04-06T13:49:57.243 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.242+0000 7f2fc70d8700 1 Processor -- start 2024-04-06T13:49:57.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.242+0000 7f2fc70d8700 1 -- start start 2024-04-06T13:49:57.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fc70d8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fc006a730 0x7f2fc00b1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fc70d8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fc015d7b0 0x7f2fc0146bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fc70d8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fc015ddf0 0x7f2fc00cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fc70d8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc00b1200 con 0x7f2fc015ddf0 2024-04-06T13:49:57.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fc70d8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc01470f0 con 0x7f2fc015d7b0 2024-04-06T13:49:57.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fc70d8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc00b0180 con 0x7f2fc006a730 2024-04-06T13:49:57.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fc5675700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fc015ddf0 0x7f2fc00cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fc5675700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fc015ddf0 0x7f2fc00cc540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59422/0 (socket says 172.21.15.189:59422) 2024-04-06T13:49:57.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fc5675700 1 -- 172.21.15.189:0/4064534338 learned_addr learned my addr 172.21.15.189:0/4064534338 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:57.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fc4e74700 1 --2- 172.21.15.189:0/4064534338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fc006a730 0x7f2fc00b1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fbffff700 1 --2- 172.21.15.189:0/4064534338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fc015d7b0 0x7f2fc0146bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fc5675700 1 -- 172.21.15.189:0/4064534338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fc006a730 msgr2=0x7f2fc00b1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:57.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fc5675700 1 --2- 172.21.15.189:0/4064534338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fc006a730 0x7f2fc00b1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fc5675700 1 -- 172.21.15.189:0/4064534338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fc015d7b0 msgr2=0x7f2fc0146bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:57.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fc5675700 1 --2- 172.21.15.189:0/4064534338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fc015d7b0 0x7f2fc0146bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fc5675700 1 -- 172.21.15.189:0/4064534338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb8009b30 con 0x7f2fc015ddf0 2024-04-06T13:49:57.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fc5675700 1 --2- 172.21.15.189:0/4064534338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fc015ddf0 0x7f2fc00cc540 secure :-1 s=READY pgs=3092 cs=0 l=1 rev1=1 crypto rx=0x7f2fb800b370 tx=0x7f2fb8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:57.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fbd7fa700 1 -- 172.21.15.189:0/4064534338 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8004310 con 0x7f2fc015ddf0 2024-04-06T13:49:57.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fbd7fa700 1 -- 172.21.15.189:0/4064534338 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb8004470 con 0x7f2fc015ddf0 2024-04-06T13:49:57.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fc70d8700 1 -- 172.21.15.189:0/4064534338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc00b0460 con 0x7f2fc015ddf0 2024-04-06T13:49:57.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fbd7fa700 1 -- 172.21.15.189:0/4064534338 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8004820 con 0x7f2fc015ddf0 2024-04-06T13:49:57.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.243+0000 7f2fc70d8700 1 -- 172.21.15.189:0/4064534338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc00b0e90 con 0x7f2fc015ddf0 2024-04-06T13:49:57.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.244+0000 7f2fc70d8700 1 -- 172.21.15.189:0/4064534338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc0067bb0 con 0x7f2fc015ddf0 2024-04-06T13:49:57.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.245+0000 7f2fbd7fa700 1 -- 172.21.15.189:0/4064534338 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f2fb801c440 con 0x7f2fc015ddf0 2024-04-06T13:49:57.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.245+0000 7f2fbd7fa700 1 --2- 172.21.15.189:0/4064534338 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2fa406adb0 0x7f2fa406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.245+0000 7f2fbd7fa700 1 -- 172.21.15.189:0/4064534338 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2fb808ae60 con 0x7f2fc015ddf0 2024-04-06T13:49:57.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.245+0000 7f2fc4e74700 1 --2- 172.21.15.189:0/4064534338 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2fa406adb0 0x7f2fa406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.246+0000 7f2fc4e74700 1 --2- 172.21.15.189:0/4064534338 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2fa406adb0 0x7f2fa406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2fb400aae0 tx=0x7f2fb4008040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:57.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.250+0000 7f2fbd7fa700 1 -- 172.21.15.189:0/4064534338 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2fb8055d20 con 0x7f2fc015ddf0 2024-04-06T13:49:57.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.446+0000 7f2fc70d8700 1 -- 172.21.15.189:0/4064534338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fc0074990 con 0x7f2fc015ddf0 2024-04-06T13:49:57.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.447+0000 7f2fbd7fa700 1 -- 172.21.15.189:0/4064534338 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v283) v1 ==== 74+0+16424 (secure 0 0 0) 0x7f2fb8059340 con 0x7f2fc015ddf0 2024-04-06T13:49:57.447 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:57.447 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":283,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:49:54.690019+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:49:51.467569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"282","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-04-06T13:49:52.656784+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"283","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:49:57.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.450+0000 7f2fc70d8700 1 -- 172.21.15.189:0/4064534338 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2fa406adb0 msgr2=0x7f2fa406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:57.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.450+0000 7f2fc70d8700 1 --2- 172.21.15.189:0/4064534338 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2fa406adb0 0x7f2fa406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2fb400aae0 tx=0x7f2fb4008040 comp rx=0 tx=0).stop 2024-04-06T13:49:57.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.450+0000 7f2fc70d8700 1 -- 172.21.15.189:0/4064534338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fc015ddf0 msgr2=0x7f2fc00cc540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:57.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.450+0000 7f2fc70d8700 1 --2- 172.21.15.189:0/4064534338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fc015ddf0 0x7f2fc00cc540 secure :-1 s=READY pgs=3092 cs=0 l=1 rev1=1 crypto rx=0x7f2fb800b370 tx=0x7f2fb8004ad0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.450+0000 7f2fc70d8700 1 -- 172.21.15.189:0/4064534338 shutdown_connections 2024-04-06T13:49:57.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.450+0000 7f2fc70d8700 1 --2- 172.21.15.189:0/4064534338 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2fa406adb0 0x7f2fa406d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.450+0000 7f2fc70d8700 1 --2- 172.21.15.189:0/4064534338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fc015ddf0 0x7f2fc00cc540 unknown :-1 s=CLOSED pgs=3092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.450+0000 7f2fc70d8700 1 --2- 172.21.15.189:0/4064534338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fc006a730 0x7f2fc00b1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.450+0000 7f2fc70d8700 1 --2- 172.21.15.189:0/4064534338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fc015d7b0 0x7f2fc0146bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.450+0000 7f2fc70d8700 1 -- 172.21.15.189:0/4064534338 >> 172.21.15.189:0/4064534338 conn(0x7f2fc00d00b0 msgr2=0x7f2fc00adfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:57.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.450+0000 7f2fc70d8700 1 -- 172.21.15.189:0/4064534338 shutdown_connections 2024-04-06T13:49:57.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.450+0000 7f2fc70d8700 1 -- 172.21.15.189:0/4064534338 wait complete. 2024-04-06T13:49:57.462 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-06T13:49:57.463 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:49:57.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.584+0000 7fc2a1933700 1 Processor -- start 2024-04-06T13:49:57.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.584+0000 7fc2a1933700 1 -- start start 2024-04-06T13:49:57.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.585+0000 7fc2a1933700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc29c0b4ed0 0x7fc29c0b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.585+0000 7fc2a1933700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29c0b5cc0 con 0x7fc29c0d2880 2024-04-06T13:49:57.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.585+0000 7fc2a1933700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29c0d1500 con 0x7fc29c15d590 2024-04-06T13:49:57.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.585+0000 7fc2a1933700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc29c0b58c0 con 0x7fc29c0b4ed0 2024-04-06T13:49:57.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.585+0000 7fc29b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc29c0d2880 0x7fc29c0d2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49916/0 (socket says 172.21.15.189:49916) 2024-04-06T13:49:57.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.585+0000 7fc29b7fe700 1 -- 172.21.15.189:0/1036306907 learned_addr learned my addr 172.21.15.189:0/1036306907 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:57.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.585+0000 7fc29a7fc700 1 --2- 172.21.15.189:0/1036306907 >> v2:172.21.15.192:3300/0 conn(0x7fc29c0b4ed0 0x7fc29c0b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.585+0000 7fc299ffb700 1 -- 172.21.15.189:0/1036306907 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869326347 0 0) 0x7fc29c0b5cc0 con 0x7fc29c0d2880 2024-04-06T13:49:57.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.585+0000 7fc299ffb700 1 -- 172.21.15.189:0/1036306907 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2840035b0 con 0x7fc29c0d2880 2024-04-06T13:49:57.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc299ffb700 1 -- 172.21.15.189:0/1036306907 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 669958588 0 0) 0x7fc2840035b0 con 0x7fc29c0d2880 2024-04-06T13:49:57.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc299ffb700 1 -- 172.21.15.189:0/1036306907 >> v2:172.21.15.192:3300/0 conn(0x7fc29c0b4ed0 msgr2=0x7fc29c0b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:57.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc299ffb700 1 --2- 172.21.15.189:0/1036306907 >> v2:172.21.15.192:3300/0 conn(0x7fc29c0b4ed0 0x7fc29c0b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc299ffb700 1 -- 172.21.15.189:0/1036306907 >> v1:172.21.15.192:6789/0 conn(0x7fc29c15d590 legacy=0x7fc29c0b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:57.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc299ffb700 1 -- 172.21.15.189:0/1036306907 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29c0d17e0 con 0x7fc29c0d2880 2024-04-06T13:49:57.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc299ffb700 1 -- 172.21.15.189:0/1036306907 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fc2940019f0 con 0x7fc29c0d2880 2024-04-06T13:49:57.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc299ffb700 1 -- 172.21.15.189:0/1036306907 >> v1:172.21.15.189:6789/0 conn(0x7fc29c0d2880 legacy=0x7fc29c0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:57.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc299ffb700 1 --2- 172.21.15.189:0/1036306907 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc284003a00 0x7fc284003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc299ffb700 1 --2- 172.21.15.189:0/1036306907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc284004850 0x7fc284004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc299ffb700 1 --2- 172.21.15.189:0/1036306907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc284005210 0x7fc28400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc299ffb700 1 -- 172.21.15.189:0/1036306907 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c0b58c0 con 0x7fc284005210 2024-04-06T13:49:57.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc299ffb700 1 -- 172.21.15.189:0/1036306907 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28400c0b0 con 0x7fc284004850 2024-04-06T13:49:57.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc299ffb700 1 -- 172.21.15.189:0/1036306907 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28400c270 con 0x7fc284003a00 2024-04-06T13:49:57.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc29b7fe700 1 --2- 172.21.15.189:0/1036306907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc284005210 0x7fc28400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc29affd700 1 --2- 172.21.15.189:0/1036306907 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc284003a00 0x7fc284003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc29a7fc700 1 --2- 172.21.15.189:0/1036306907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc284004850 0x7fc284004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc29b7fe700 1 -- 172.21.15.189:0/1036306907 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc284003a00 msgr2=0x7fc284003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:57.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc29b7fe700 1 --2- 172.21.15.189:0/1036306907 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc284003a00 0x7fc284003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc29b7fe700 1 -- 172.21.15.189:0/1036306907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc284004850 msgr2=0x7fc284004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:57.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc29b7fe700 1 --2- 172.21.15.189:0/1036306907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc284004850 0x7fc284004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc29b7fe700 1 -- 172.21.15.189:0/1036306907 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2840035b0 con 0x7fc284005210 2024-04-06T13:49:57.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc29b7fe700 1 -- 172.21.15.189:0/1036306907 reap_dead start 2024-04-06T13:49:57.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc29b7fe700 1 --2- 172.21.15.189:0/1036306907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc284005210 0x7fc28400ba10 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7fc29400a830 tx=0x7fc284005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f792042f487fb14 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:57.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc299ffb700 1 -- 172.21.15.189:0/1036306907 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29400a2f0 con 0x7fc284005210 2024-04-06T13:49:57.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc299ffb700 1 -- 172.21.15.189:0/1036306907 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc294012600 con 0x7fc284005210 2024-04-06T13:49:57.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.586+0000 7fc299ffb700 1 -- 172.21.15.189:0/1036306907 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2940129b0 con 0x7fc284005210 2024-04-06T13:49:57.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.587+0000 7fc2a1933700 1 -- 172.21.15.189:0/1036306907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc284005210 msgr2=0x7fc28400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:57.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.587+0000 7fc2a1933700 1 --2- 172.21.15.189:0/1036306907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc284005210 0x7fc28400ba10 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7fc29400a830 tx=0x7fc284005880 comp rx=0 tx=0).stop 2024-04-06T13:49:57.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.587+0000 7fc2a1933700 1 -- 172.21.15.189:0/1036306907 shutdown_connections 2024-04-06T13:49:57.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.587+0000 7fc2a1933700 1 --2- 172.21.15.189:0/1036306907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc284005210 0x7fc28400ba10 unknown :-1 s=CLOSED pgs=3094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.587+0000 7fc2a1933700 1 -- 172.21.15.189:0/1036306907 >> 172.21.15.189:0/1036306907 conn(0x7fc29c06bbc0 msgr2=0x7fc29c06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:57.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.587+0000 7fc2a1933700 1 -- 172.21.15.189:0/1036306907 shutdown_connections 2024-04-06T13:49:57.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.587+0000 7fc2a1933700 1 -- 172.21.15.189:0/1036306907 wait complete. 2024-04-06T13:49:57.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.588+0000 7fc2a1933700 1 Processor -- start 2024-04-06T13:49:57.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.588+0000 7fc2a1933700 1 -- start start 2024-04-06T13:49:57.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.589+0000 7fc2a1933700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc284005210 0x7fc29c1539f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.589+0000 7fc2a1933700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc29c0cd0c0 0x7fc29c1f3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.589+0000 7fc2a1933700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc29c158800 0x7fc29c158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.589+0000 7fc2a1933700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c1591f0 con 0x7fc29c158800 2024-04-06T13:49:57.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.589+0000 7fc2a1933700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c153f30 con 0x7fc284005210 2024-04-06T13:49:57.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.589+0000 7fc2a1933700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c15da10 con 0x7fc29c0cd0c0 2024-04-06T13:49:57.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.589+0000 7fc29b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc29c158800 0x7fc29c158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.589+0000 7fc29b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc29c158800 0x7fc29c158cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59432/0 (socket says 172.21.15.189:59432) 2024-04-06T13:49:57.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.589+0000 7fc29b7fe700 1 -- 172.21.15.189:0/3818652901 learned_addr learned my addr 172.21.15.189:0/3818652901 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:57.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.589+0000 7fc29affd700 1 --2- 172.21.15.189:0/3818652901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc284005210 0x7fc29c1539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.589+0000 7fc29a7fc700 1 --2- 172.21.15.189:0/3818652901 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc29c0cd0c0 0x7fc29c1f3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.590+0000 7fc29b7fe700 1 -- 172.21.15.189:0/3818652901 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc29c0cd0c0 msgr2=0x7fc29c1f3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:57.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.590+0000 7fc29b7fe700 1 --2- 172.21.15.189:0/3818652901 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc29c0cd0c0 0x7fc29c1f3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.590+0000 7fc29b7fe700 1 -- 172.21.15.189:0/3818652901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc284005210 msgr2=0x7fc29c1539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:57.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.590+0000 7fc29b7fe700 1 --2- 172.21.15.189:0/3818652901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc284005210 0x7fc29c1539f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.590+0000 7fc29b7fe700 1 -- 172.21.15.189:0/3818652901 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29c15dcf0 con 0x7fc29c158800 2024-04-06T13:49:57.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.590+0000 7fc29b7fe700 1 --2- 172.21.15.189:0/3818652901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc29c158800 0x7fc29c158cb0 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 crypto rx=0x7fc29c0cf570 tx=0x7fc294001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:57.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.590+0000 7fc283fff700 1 -- 172.21.15.189:0/3818652901 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc294001250 con 0x7fc29c158800 2024-04-06T13:49:57.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.590+0000 7fc283fff700 1 -- 172.21.15.189:0/3818652901 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2940017b0 con 0x7fc29c158800 2024-04-06T13:49:57.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.590+0000 7fc2a1933700 1 -- 172.21.15.189:0/3818652901 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc29c15dfd0 con 0x7fc29c158800 2024-04-06T13:49:57.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.590+0000 7fc283fff700 1 -- 172.21.15.189:0/3818652901 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29401d420 con 0x7fc29c158800 2024-04-06T13:49:57.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.590+0000 7fc2a1933700 1 -- 172.21.15.189:0/3818652901 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29c15e590 con 0x7fc29c158800 2024-04-06T13:49:57.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.591+0000 7fc2a1933700 1 -- 172.21.15.189:0/3818652901 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc29c0d2060 con 0x7fc29c158800 2024-04-06T13:49:57.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.591+0000 7fc283fff700 1 -- 172.21.15.189:0/3818652901 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fc29401d580 con 0x7fc29c158800 2024-04-06T13:49:57.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.592+0000 7fc283fff700 1 --2- 172.21.15.189:0/3818652901 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc28806adb0 0x7fc28806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.592+0000 7fc283fff700 1 -- 172.21.15.189:0/3818652901 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc29408acb0 con 0x7fc29c158800 2024-04-06T13:49:57.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.592+0000 7fc29affd700 1 --2- 172.21.15.189:0/3818652901 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc28806adb0 0x7fc28806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.592+0000 7fc29affd700 1 --2- 172.21.15.189:0/3818652901 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc28806adb0 0x7fc28806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc290000c00 tx=0x7fc290009040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:57.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.596+0000 7fc283fff700 1 -- 172.21.15.189:0/3818652901 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc29c0d2060 con 0x7fc29c158800 2024-04-06T13:49:57.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.789+0000 7fc2a1933700 1 -- 172.21.15.189:0/3818652901 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc29c0d1da0 con 0x7fc28806adb0 2024-04-06T13:49:57.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.797+0000 7fc283fff700 1 -- 172.21.15.189:0/3818652901 <== mgr.13152 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289904 (secure 0 0 0) 0x7fc290001ca0 con 0x7fc28806adb0 2024-04-06T13:49:57.801 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:57.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.800+0000 7fc2a1933700 1 -- 172.21.15.189:0/3818652901 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc28806adb0 msgr2=0x7fc28806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:57.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.800+0000 7fc2a1933700 1 --2- 172.21.15.189:0/3818652901 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc28806adb0 0x7fc28806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc290000c00 tx=0x7fc290009040 comp rx=0 tx=0).stop 2024-04-06T13:49:57.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.800+0000 7fc2a1933700 1 -- 172.21.15.189:0/3818652901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc29c158800 msgr2=0x7fc29c158cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:57.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.800+0000 7fc2a1933700 1 --2- 172.21.15.189:0/3818652901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc29c158800 0x7fc29c158cb0 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 crypto rx=0x7fc29c0cf570 tx=0x7fc294001d80 comp rx=0 tx=0).stop 2024-04-06T13:49:57.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.800+0000 7fc2a1933700 1 -- 172.21.15.189:0/3818652901 shutdown_connections 2024-04-06T13:49:57.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.800+0000 7fc2a1933700 1 --2- 172.21.15.189:0/3818652901 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc28806adb0 0x7fc28806d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.801+0000 7fc2a1933700 1 --2- 172.21.15.189:0/3818652901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc29c158800 0x7fc29c158cb0 unknown :-1 s=CLOSED pgs=3095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.801+0000 7fc2a1933700 1 --2- 172.21.15.189:0/3818652901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc284005210 0x7fc29c1539f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.801+0000 7fc2a1933700 1 --2- 172.21.15.189:0/3818652901 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc29c0cd0c0 0x7fc29c1f3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.801+0000 7fc2a1933700 1 -- 172.21.15.189:0/3818652901 >> 172.21.15.189:0/3818652901 conn(0x7fc29c06bbc0 msgr2=0x7fc29c0b6ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:57.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.801+0000 7fc2a1933700 1 -- 172.21.15.189:0/3818652901 shutdown_connections 2024-04-06T13:49:57.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.801+0000 7fc2a1933700 1 -- 172.21.15.189:0/3818652901 wait complete. 2024-04-06T13:49:57.804 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:49:57.813 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-06T13:49:55.998474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":723,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":441,"ondisk_log_size":441,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":473348,"kb_used_data":29172,"kb_used_omap":12,"kb_used_meta":444019,"kb_avail":754501372,"statfs":{"total":773094113280,"available":772609404928,"internally_reserved":0,"allocated":29872128,"data_stored":15963765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":454676024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000673"},"pg_stats":[{"pgid":"38.3f","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710317+0000","last_change":"2024-04-06T13:49:53.698145+0000","last_active":"2024-04-06T13:49:54.710317+0000","last_peered":"2024-04-06T13:49:54.710317+0000","last_clean":"2024-04-06T13:49:54.710317+0000","last_became_active":"2024-04-06T13:49:53.697937+0000","last_became_peered":"2024-04-06T13:49:53.697937+0000","last_unstale":"2024-04-06T13:49:54.710317+0000","last_undegraded":"2024-04-06T13:49:54.710317+0000","last_fullsized":"2024-04-06T13:49:54.710317+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:42:43.790123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.3e","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697798+0000","last_change":"2024-04-06T13:49:53.716385+0000","last_active":"2024-04-06T13:49:54.697798+0000","last_peered":"2024-04-06T13:49:54.697798+0000","last_clean":"2024-04-06T13:49:54.697798+0000","last_became_active":"2024-04-06T13:49:53.716147+0000","last_became_peered":"2024-04-06T13:49:53.716147+0000","last_unstale":"2024-04-06T13:49:54.697798+0000","last_undegraded":"2024-04-06T13:49:54.697798+0000","last_fullsized":"2024-04-06T13:49:54.697798+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:43:30.859611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.3d","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.709965+0000","last_change":"2024-04-06T13:49:53.713242+0000","last_active":"2024-04-06T13:49:54.709965+0000","last_peered":"2024-04-06T13:49:54.709965+0000","last_clean":"2024-04-06T13:49:54.709965+0000","last_became_active":"2024-04-06T13:49:53.712846+0000","last_became_peered":"2024-04-06T13:49:53.712846+0000","last_unstale":"2024-04-06T13:49:54.709965+0000","last_undegraded":"2024-04-06T13:49:54.709965+0000","last_fullsized":"2024-04-06T13:49:54.709965+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:03:26.515333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.3c","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699161+0000","last_change":"2024-04-06T13:49:53.713265+0000","last_active":"2024-04-06T13:49:54.699161+0000","last_peered":"2024-04-06T13:49:54.699161+0000","last_clean":"2024-04-06T13:49:54.699161+0000","last_became_active":"2024-04-06T13:49:53.712513+0000","last_became_peered":"2024-04-06T13:49:53.712513+0000","last_unstale":"2024-04-06T13:49:54.699161+0000","last_undegraded":"2024-04-06T13:49:54.699161+0000","last_fullsized":"2024-04-06T13:49:54.699161+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:32:56.445394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.3b","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710783+0000","last_change":"2024-04-06T13:49:53.709755+0000","last_active":"2024-04-06T13:49:54.710783+0000","last_peered":"2024-04-06T13:49:54.710783+0000","last_clean":"2024-04-06T13:49:54.710783+0000","last_became_active":"2024-04-06T13:49:53.709247+0000","last_became_peered":"2024-04-06T13:49:53.709247+0000","last_unstale":"2024-04-06T13:49:54.710783+0000","last_undegraded":"2024-04-06T13:49:54.710783+0000","last_fullsized":"2024-04-06T13:49:54.710783+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:35:57.994706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.3a","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710742+0000","last_change":"2024-04-06T13:49:53.707246+0000","last_active":"2024-04-06T13:49:54.710742+0000","last_peered":"2024-04-06T13:49:54.710742+0000","last_clean":"2024-04-06T13:49:54.710742+0000","last_became_active":"2024-04-06T13:49:53.706917+0000","last_became_peered":"2024-04-06T13:49:53.706917+0000","last_unstale":"2024-04-06T13:49:54.710742+0000","last_undegraded":"2024-04-06T13:49:54.710742+0000","last_fullsized":"2024-04-06T13:49:54.710742+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:26:04.390932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.39","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710799+0000","last_change":"2024-04-06T13:49:53.714607+0000","last_active":"2024-04-06T13:49:54.710799+0000","last_peered":"2024-04-06T13:49:54.710799+0000","last_clean":"2024-04-06T13:49:54.710799+0000","last_became_active":"2024-04-06T13:49:53.714305+0000","last_became_peered":"2024-04-06T13:49:53.714305+0000","last_unstale":"2024-04-06T13:49:54.710799+0000","last_undegraded":"2024-04-06T13:49:54.710799+0000","last_fullsized":"2024-04-06T13:49:54.710799+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:28:13.472120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.38","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.709889+0000","last_change":"2024-04-06T13:49:53.703689+0000","last_active":"2024-04-06T13:49:54.709889+0000","last_peered":"2024-04-06T13:49:54.709889+0000","last_clean":"2024-04-06T13:49:54.709889+0000","last_became_active":"2024-04-06T13:49:53.703436+0000","last_became_peered":"2024-04-06T13:49:53.703436+0000","last_unstale":"2024-04-06T13:49:54.709889+0000","last_undegraded":"2024-04-06T13:49:54.709889+0000","last_fullsized":"2024-04-06T13:49:54.709889+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:11:11.857561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.37","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698482+0000","last_change":"2024-04-06T13:49:53.712491+0000","last_active":"2024-04-06T13:49:54.698482+0000","last_peered":"2024-04-06T13:49:54.698482+0000","last_clean":"2024-04-06T13:49:54.698482+0000","last_became_active":"2024-04-06T13:49:53.712235+0000","last_became_peered":"2024-04-06T13:49:53.712235+0000","last_unstale":"2024-04-06T13:49:54.698482+0000","last_undegraded":"2024-04-06T13:49:54.698482+0000","last_fullsized":"2024-04-06T13:49:54.698482+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:15.876796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.36","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.711166+0000","last_change":"2024-04-06T13:49:53.716746+0000","last_active":"2024-04-06T13:49:54.711166+0000","last_peered":"2024-04-06T13:49:54.711166+0000","last_clean":"2024-04-06T13:49:54.711166+0000","last_became_active":"2024-04-06T13:49:53.716519+0000","last_became_peered":"2024-04-06T13:49:53.716519+0000","last_unstale":"2024-04-06T13:49:54.711166+0000","last_undegraded":"2024-04-06T13:49:54.711166+0000","last_fullsized":"2024-04-06T13:49:54.711166+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:55:32.313240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.35","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698460+0000","last_change":"2024-04-06T13:49:53.719282+0000","last_active":"2024-04-06T13:49:54.698460+0000","last_peered":"2024-04-06T13:49:54.698460+0000","last_clean":"2024-04-06T13:49:54.698460+0000","last_became_active":"2024-04-06T13:49:53.719046+0000","last_became_peered":"2024-04-06T13:49:53.719046+0000","last_unstale":"2024-04-06T13:49:54.698460+0000","last_undegraded":"2024-04-06T13:49:54.698460+0000","last_fullsized":"2024-04-06T13:49:54.698460+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:31:26.745492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.34","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.711307+0000","last_change":"2024-04-06T13:49:53.704657+0000","last_active":"2024-04-06T13:49:54.711307+0000","last_peered":"2024-04-06T13:49:54.711307+0000","last_clean":"2024-04-06T13:49:54.711307+0000","last_became_active":"2024-04-06T13:49:53.704287+0000","last_became_peered":"2024-04-06T13:49:53.704287+0000","last_unstale":"2024-04-06T13:49:54.711307+0000","last_undegraded":"2024-04-06T13:49:54.711307+0000","last_fullsized":"2024-04-06T13:49:54.711307+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:11:11.857561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.33","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710606+0000","last_change":"2024-04-06T13:49:53.709949+0000","last_active":"2024-04-06T13:49:54.710606+0000","last_peered":"2024-04-06T13:49:54.710606+0000","last_clean":"2024-04-06T13:49:54.710606+0000","last_became_active":"2024-04-06T13:49:53.709511+0000","last_became_peered":"2024-04-06T13:49:53.709511+0000","last_unstale":"2024-04-06T13:49:54.710606+0000","last_undegraded":"2024-04-06T13:49:54.710606+0000","last_fullsized":"2024-04-06T13:49:54.710606+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:48:15.813195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.32","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697114+0000","last_change":"2024-04-06T13:49:53.708002+0000","last_active":"2024-04-06T13:49:54.697114+0000","last_peered":"2024-04-06T13:49:54.697114+0000","last_clean":"2024-04-06T13:49:54.697114+0000","last_became_active":"2024-04-06T13:49:53.707771+0000","last_became_peered":"2024-04-06T13:49:53.707771+0000","last_unstale":"2024-04-06T13:49:54.697114+0000","last_undegraded":"2024-04-06T13:49:54.697114+0000","last_fullsized":"2024-04-06T13:49:54.697114+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:06.099346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.31","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697213+0000","last_change":"2024-04-06T13:49:53.710656+0000","last_active":"2024-04-06T13:49:54.697213+0000","last_peered":"2024-04-06T13:49:54.697213+0000","last_clean":"2024-04-06T13:49:54.697213+0000","last_became_active":"2024-04-06T13:49:53.710361+0000","last_became_peered":"2024-04-06T13:49:53.710361+0000","last_unstale":"2024-04-06T13:49:54.697213+0000","last_undegraded":"2024-04-06T13:49:54.697213+0000","last_fullsized":"2024-04-06T13:49:54.697213+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:45:06.633037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.30","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.711022+0000","last_change":"2024-04-06T13:49:53.720546+0000","last_active":"2024-04-06T13:49:54.711022+0000","last_peered":"2024-04-06T13:49:54.711022+0000","last_clean":"2024-04-06T13:49:54.711022+0000","last_became_active":"2024-04-06T13:49:53.720299+0000","last_became_peered":"2024-04-06T13:49:53.720299+0000","last_unstale":"2024-04-06T13:49:54.711022+0000","last_undegraded":"2024-04-06T13:49:54.711022+0000","last_fullsized":"2024-04-06T13:49:54.711022+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:09:37.214050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.2f","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698529+0000","last_change":"2024-04-06T13:49:53.712193+0000","last_active":"2024-04-06T13:49:54.698529+0000","last_peered":"2024-04-06T13:49:54.698529+0000","last_clean":"2024-04-06T13:49:54.698529+0000","last_became_active":"2024-04-06T13:49:53.711893+0000","last_became_peered":"2024-04-06T13:49:53.711893+0000","last_unstale":"2024-04-06T13:49:54.698529+0000","last_undegraded":"2024-04-06T13:49:54.698529+0000","last_fullsized":"2024-04-06T13:49:54.698529+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:31:34.206103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.2e","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699040+0000","last_change":"2024-04-06T13:49:53.711062+0000","last_active":"2024-04-06T13:49:54.699040+0000","last_peered":"2024-04-06T13:49:54.699040+0000","last_clean":"2024-04-06T13:49:54.699040+0000","last_became_active":"2024-04-06T13:49:53.710770+0000","last_became_peered":"2024-04-06T13:49:53.710770+0000","last_unstale":"2024-04-06T13:49:54.699040+0000","last_undegraded":"2024-04-06T13:49:54.699040+0000","last_fullsized":"2024-04-06T13:49:54.699040+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:13.245715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2d","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710061+0000","last_change":"2024-04-06T13:49:53.713054+0000","last_active":"2024-04-06T13:49:54.710061+0000","last_peered":"2024-04-06T13:49:54.710061+0000","last_clean":"2024-04-06T13:49:54.710061+0000","last_became_active":"2024-04-06T13:49:53.712502+0000","last_became_peered":"2024-04-06T13:49:53.712502+0000","last_unstale":"2024-04-06T13:49:54.710061+0000","last_undegraded":"2024-04-06T13:49:54.710061+0000","last_fullsized":"2024-04-06T13:49:54.710061+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:29:11.366508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.2c","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699287+0000","last_change":"2024-04-06T13:49:53.717998+0000","last_active":"2024-04-06T13:49:54.699287+0000","last_peered":"2024-04-06T13:49:54.699287+0000","last_clean":"2024-04-06T13:49:54.699287+0000","last_became_active":"2024-04-06T13:49:53.717305+0000","last_became_peered":"2024-04-06T13:49:53.717305+0000","last_unstale":"2024-04-06T13:49:54.699287+0000","last_undegraded":"2024-04-06T13:49:54.699287+0000","last_fullsized":"2024-04-06T13:49:54.699287+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:55:00.463548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2b","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697400+0000","last_change":"2024-04-06T13:49:53.711352+0000","last_active":"2024-04-06T13:49:54.697400+0000","last_peered":"2024-04-06T13:49:54.697400+0000","last_clean":"2024-04-06T13:49:54.697400+0000","last_became_active":"2024-04-06T13:49:53.711110+0000","last_became_peered":"2024-04-06T13:49:53.711110+0000","last_unstale":"2024-04-06T13:49:54.697400+0000","last_undegraded":"2024-04-06T13:49:54.697400+0000","last_fullsized":"2024-04-06T13:49:54.697400+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:31:25.186211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.2a","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699145+0000","last_change":"2024-04-06T13:49:53.718419+0000","last_active":"2024-04-06T13:49:54.699145+0000","last_peered":"2024-04-06T13:49:54.699145+0000","last_clean":"2024-04-06T13:49:54.699145+0000","last_became_active":"2024-04-06T13:49:53.713389+0000","last_became_peered":"2024-04-06T13:49:53.713389+0000","last_unstale":"2024-04-06T13:49:54.699145+0000","last_undegraded":"2024-04-06T13:49:54.699145+0000","last_fullsized":"2024-04-06T13:49:54.699145+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:13:36.768561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.29","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698931+0000","last_change":"2024-04-06T13:49:53.722123+0000","last_active":"2024-04-06T13:49:54.698931+0000","last_peered":"2024-04-06T13:49:54.698931+0000","last_clean":"2024-04-06T13:49:54.698931+0000","last_became_active":"2024-04-06T13:49:53.719359+0000","last_became_peered":"2024-04-06T13:49:53.719359+0000","last_unstale":"2024-04-06T13:49:54.698931+0000","last_undegraded":"2024-04-06T13:49:54.698931+0000","last_fullsized":"2024-04-06T13:49:54.698931+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:15:49.763249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.28","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.709578+0000","last_change":"2024-04-06T13:49:53.707950+0000","last_active":"2024-04-06T13:49:54.709578+0000","last_peered":"2024-04-06T13:49:54.709578+0000","last_clean":"2024-04-06T13:49:54.709578+0000","last_became_active":"2024-04-06T13:49:53.707514+0000","last_became_peered":"2024-04-06T13:49:53.707514+0000","last_unstale":"2024-04-06T13:49:54.709578+0000","last_undegraded":"2024-04-06T13:49:54.709578+0000","last_fullsized":"2024-04-06T13:49:54.709578+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:05:17.703470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.26","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698490+0000","last_change":"2024-04-06T13:49:53.712113+0000","last_active":"2024-04-06T13:49:54.698490+0000","last_peered":"2024-04-06T13:49:54.698490+0000","last_clean":"2024-04-06T13:49:54.698490+0000","last_became_active":"2024-04-06T13:49:53.711852+0000","last_became_peered":"2024-04-06T13:49:53.711852+0000","last_unstale":"2024-04-06T13:49:54.698490+0000","last_undegraded":"2024-04-06T13:49:54.698490+0000","last_fullsized":"2024-04-06T13:49:54.698490+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:12:58.661980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.25","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699199+0000","last_change":"2024-04-06T13:49:53.720979+0000","last_active":"2024-04-06T13:49:54.699199+0000","last_peered":"2024-04-06T13:49:54.699199+0000","last_clean":"2024-04-06T13:49:54.699199+0000","last_became_active":"2024-04-06T13:49:53.720430+0000","last_became_peered":"2024-04-06T13:49:53.720430+0000","last_unstale":"2024-04-06T13:49:54.699199+0000","last_undegraded":"2024-04-06T13:49:54.699199+0000","last_fullsized":"2024-04-06T13:49:54.699199+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:16.805997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.24","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698367+0000","last_change":"2024-04-06T13:49:53.715901+0000","last_active":"2024-04-06T13:49:54.698367+0000","last_peered":"2024-04-06T13:49:54.698367+0000","last_clean":"2024-04-06T13:49:54.698367+0000","last_became_active":"2024-04-06T13:49:53.715646+0000","last_became_peered":"2024-04-06T13:49:53.715646+0000","last_unstale":"2024-04-06T13:49:54.698367+0000","last_undegraded":"2024-04-06T13:49:54.698367+0000","last_fullsized":"2024-04-06T13:49:54.698367+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:03:35.006421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.23","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710475+0000","last_change":"2024-04-06T13:49:53.715109+0000","last_active":"2024-04-06T13:49:54.710475+0000","last_peered":"2024-04-06T13:49:54.710475+0000","last_clean":"2024-04-06T13:49:54.710475+0000","last_became_active":"2024-04-06T13:49:53.714657+0000","last_became_peered":"2024-04-06T13:49:53.714657+0000","last_unstale":"2024-04-06T13:49:54.710475+0000","last_undegraded":"2024-04-06T13:49:54.710475+0000","last_fullsized":"2024-04-06T13:49:54.710475+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:37:06.208036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.22","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697219+0000","last_change":"2024-04-06T13:49:53.697123+0000","last_active":"2024-04-06T13:49:54.697219+0000","last_peered":"2024-04-06T13:49:54.697219+0000","last_clean":"2024-04-06T13:49:54.697219+0000","last_became_active":"2024-04-06T13:49:53.696854+0000","last_became_peered":"2024-04-06T13:49:53.696854+0000","last_unstale":"2024-04-06T13:49:54.697219+0000","last_undegraded":"2024-04-06T13:49:54.697219+0000","last_fullsized":"2024-04-06T13:49:54.697219+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:29:59.193762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.21","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710895+0000","last_change":"2024-04-06T13:49:53.714011+0000","last_active":"2024-04-06T13:49:54.710895+0000","last_peered":"2024-04-06T13:49:54.710895+0000","last_clean":"2024-04-06T13:49:54.710895+0000","last_became_active":"2024-04-06T13:49:53.713717+0000","last_became_peered":"2024-04-06T13:49:53.713717+0000","last_unstale":"2024-04-06T13:49:54.710895+0000","last_undegraded":"2024-04-06T13:49:54.710895+0000","last_fullsized":"2024-04-06T13:49:54.710895+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:29.154423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.20","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698745+0000","last_change":"2024-04-06T13:49:53.722141+0000","last_active":"2024-04-06T13:49:54.698745+0000","last_peered":"2024-04-06T13:49:54.698745+0000","last_clean":"2024-04-06T13:49:54.698745+0000","last_became_active":"2024-04-06T13:49:53.720421+0000","last_became_peered":"2024-04-06T13:49:53.720421+0000","last_unstale":"2024-04-06T13:49:54.698745+0000","last_undegraded":"2024-04-06T13:49:54.698745+0000","last_fullsized":"2024-04-06T13:49:54.698745+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:52:58.376388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1c","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699292+0000","last_change":"2024-04-06T13:49:53.718638+0000","last_active":"2024-04-06T13:49:54.699292+0000","last_peered":"2024-04-06T13:49:54.699292+0000","last_clean":"2024-04-06T13:49:54.699292+0000","last_became_active":"2024-04-06T13:49:53.713132+0000","last_became_peered":"2024-04-06T13:49:53.713132+0000","last_unstale":"2024-04-06T13:49:54.699292+0000","last_undegraded":"2024-04-06T13:49:54.699292+0000","last_fullsized":"2024-04-06T13:49:54.699292+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:28:04.138418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1f","version":"282'4","reported_seq":15,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.711169+0000","last_change":"2024-04-06T13:49:53.291574+0000","last_active":"2024-04-06T13:49:54.711169+0000","last_peered":"2024-04-06T13:49:54.711169+0000","last_clean":"2024-04-06T13:49:54.711169+0000","last_became_active":"2024-04-06T13:49:53.291311+0000","last_became_peered":"2024-04-06T13:49:53.291311+0000","last_unstale":"2024-04-06T13:49:54.711169+0000","last_undegraded":"2024-04-06T13:49:54.711169+0000","last_fullsized":"2024-04-06T13:49:54.711169+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:31:58.213975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.1d","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698387+0000","last_change":"2024-04-06T13:49:53.719249+0000","last_active":"2024-04-06T13:49:54.698387+0000","last_peered":"2024-04-06T13:49:54.698387+0000","last_clean":"2024-04-06T13:49:54.698387+0000","last_became_active":"2024-04-06T13:49:53.719020+0000","last_became_peered":"2024-04-06T13:49:53.719020+0000","last_unstale":"2024-04-06T13:49:54.698387+0000","last_undegraded":"2024-04-06T13:49:54.698387+0000","last_fullsized":"2024-04-06T13:49:54.698387+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:30:18.780722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1e","version":"282'1","reported_seq":12,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699138+0000","last_change":"2024-04-06T13:49:53.294979+0000","last_active":"2024-04-06T13:49:54.699138+0000","last_peered":"2024-04-06T13:49:54.699138+0000","last_clean":"2024-04-06T13:49:54.699138+0000","last_became_active":"2024-04-06T13:49:53.294713+0000","last_became_peered":"2024-04-06T13:49:53.294713+0000","last_unstale":"2024-04-06T13:49:54.699138+0000","last_undegraded":"2024-04-06T13:49:54.699138+0000","last_fullsized":"2024-04-06T13:49:54.699138+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:04:47.588644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1e","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697694+0000","last_change":"2024-04-06T13:49:53.714295+0000","last_active":"2024-04-06T13:49:54.697694+0000","last_peered":"2024-04-06T13:49:54.697694+0000","last_clean":"2024-04-06T13:49:54.697694+0000","last_became_active":"2024-04-06T13:49:53.714037+0000","last_became_peered":"2024-04-06T13:49:53.714037+0000","last_unstale":"2024-04-06T13:49:54.697694+0000","last_undegraded":"2024-04-06T13:49:54.697694+0000","last_fullsized":"2024-04-06T13:49:54.697694+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:21.536648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"282'2","reported_seq":13,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710143+0000","last_change":"2024-04-06T13:49:53.292744+0000","last_active":"2024-04-06T13:49:54.710143+0000","last_peered":"2024-04-06T13:49:54.710143+0000","last_clean":"2024-04-06T13:49:54.710143+0000","last_became_active":"2024-04-06T13:49:53.292484+0000","last_became_peered":"2024-04-06T13:49:53.292484+0000","last_unstale":"2024-04-06T13:49:54.710143+0000","last_undegraded":"2024-04-06T13:49:54.710143+0000","last_fullsized":"2024-04-06T13:49:54.710143+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:29:50.333793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1f","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697806+0000","last_change":"2024-04-06T13:49:53.698278+0000","last_active":"2024-04-06T13:49:54.697806+0000","last_peered":"2024-04-06T13:49:54.697806+0000","last_clean":"2024-04-06T13:49:54.697806+0000","last_became_active":"2024-04-06T13:49:53.698038+0000","last_became_peered":"2024-04-06T13:49:53.698038+0000","last_unstale":"2024-04-06T13:49:54.697806+0000","last_undegraded":"2024-04-06T13:49:54.697806+0000","last_fullsized":"2024-04-06T13:49:54.697806+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:16:22.805098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"282'2","reported_seq":13,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697436+0000","last_change":"2024-04-06T13:49:53.296089+0000","last_active":"2024-04-06T13:49:54.697436+0000","last_peered":"2024-04-06T13:49:54.697436+0000","last_clean":"2024-04-06T13:49:54.697436+0000","last_became_active":"2024-04-06T13:49:53.295881+0000","last_became_peered":"2024-04-06T13:49:53.295881+0000","last_unstale":"2024-04-06T13:49:54.697436+0000","last_undegraded":"2024-04-06T13:49:54.697436+0000","last_fullsized":"2024-04-06T13:49:54.697436+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:12.240799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.18","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.709978+0000","last_change":"2024-04-06T13:49:53.719108+0000","last_active":"2024-04-06T13:49:54.709978+0000","last_peered":"2024-04-06T13:49:54.709978+0000","last_clean":"2024-04-06T13:49:54.709978+0000","last_became_active":"2024-04-06T13:49:53.718880+0000","last_became_peered":"2024-04-06T13:49:53.718880+0000","last_unstale":"2024-04-06T13:49:54.709978+0000","last_undegraded":"2024-04-06T13:49:54.709978+0000","last_fullsized":"2024-04-06T13:49:54.709978+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:00:49.668717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1b","version":"282'1","reported_seq":12,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698177+0000","last_change":"2024-04-06T13:49:53.291374+0000","last_active":"2024-04-06T13:49:54.698177+0000","last_peered":"2024-04-06T13:49:54.698177+0000","last_clean":"2024-04-06T13:49:54.698177+0000","last_became_active":"2024-04-06T13:49:53.291086+0000","last_became_peered":"2024-04-06T13:49:53.291086+0000","last_unstale":"2024-04-06T13:49:54.698177+0000","last_undegraded":"2024-04-06T13:49:54.698177+0000","last_fullsized":"2024-04-06T13:49:54.698177+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:30:27.012134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.19","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698121+0000","last_change":"2024-04-06T13:49:53.709992+0000","last_active":"2024-04-06T13:49:54.698121+0000","last_peered":"2024-04-06T13:49:54.698121+0000","last_clean":"2024-04-06T13:49:54.698121+0000","last_became_active":"2024-04-06T13:49:53.709726+0000","last_became_peered":"2024-04-06T13:49:53.709726+0000","last_unstale":"2024-04-06T13:49:54.698121+0000","last_undegraded":"2024-04-06T13:49:54.698121+0000","last_fullsized":"2024-04-06T13:49:54.698121+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:18:02.185633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1a","version":"282'1","reported_seq":12,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.709866+0000","last_change":"2024-04-06T13:49:53.286094+0000","last_active":"2024-04-06T13:49:54.709866+0000","last_peered":"2024-04-06T13:49:54.709866+0000","last_clean":"2024-04-06T13:49:54.709866+0000","last_became_active":"2024-04-06T13:49:53.285843+0000","last_became_peered":"2024-04-06T13:49:53.285843+0000","last_unstale":"2024-04-06T13:49:54.709866+0000","last_undegraded":"2024-04-06T13:49:54.709866+0000","last_fullsized":"2024-04-06T13:49:54.709866+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:56:43.852373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1a","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698851+0000","last_change":"2024-04-06T13:49:53.721943+0000","last_active":"2024-04-06T13:49:54.698851+0000","last_peered":"2024-04-06T13:49:54.698851+0000","last_clean":"2024-04-06T13:49:54.698851+0000","last_became_active":"2024-04-06T13:49:53.720565+0000","last_became_peered":"2024-04-06T13:49:53.720565+0000","last_unstale":"2024-04-06T13:49:54.698851+0000","last_undegraded":"2024-04-06T13:49:54.698851+0000","last_fullsized":"2024-04-06T13:49:54.698851+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:38:33.928853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697391+0000","last_change":"2024-04-06T13:49:53.295030+0000","last_active":"2024-04-06T13:49:54.697391+0000","last_peered":"2024-04-06T13:49:54.697391+0000","last_clean":"2024-04-06T13:49:54.697391+0000","last_became_active":"2024-04-06T13:49:53.294793+0000","last_became_peered":"2024-04-06T13:49:53.294793+0000","last_unstale":"2024-04-06T13:49:54.697391+0000","last_undegraded":"2024-04-06T13:49:54.697391+0000","last_fullsized":"2024-04-06T13:49:54.697391+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:25:32.159801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.1b","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697231+0000","last_change":"2024-04-06T13:49:53.716089+0000","last_active":"2024-04-06T13:49:54.697231+0000","last_peered":"2024-04-06T13:49:54.697231+0000","last_clean":"2024-04-06T13:49:54.697231+0000","last_became_active":"2024-04-06T13:49:53.715842+0000","last_became_peered":"2024-04-06T13:49:53.715842+0000","last_unstale":"2024-04-06T13:49:54.697231+0000","last_undegraded":"2024-04-06T13:49:54.697231+0000","last_fullsized":"2024-04-06T13:49:54.697231+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:30:28.017050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.709884+0000","last_change":"2024-04-06T13:49:53.293116+0000","last_active":"2024-04-06T13:49:54.709884+0000","last_peered":"2024-04-06T13:49:54.709884+0000","last_clean":"2024-04-06T13:49:54.709884+0000","last_became_active":"2024-04-06T13:49:53.292825+0000","last_became_peered":"2024-04-06T13:49:53.292825+0000","last_unstale":"2024-04-06T13:49:54.709884+0000","last_undegraded":"2024-04-06T13:49:54.709884+0000","last_fullsized":"2024-04-06T13:49:54.709884+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:16:14.577622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.14","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699402+0000","last_change":"2024-04-06T13:49:53.717594+0000","last_active":"2024-04-06T13:49:54.699402+0000","last_peered":"2024-04-06T13:49:54.699402+0000","last_clean":"2024-04-06T13:49:54.699402+0000","last_became_active":"2024-04-06T13:49:53.713401+0000","last_became_peered":"2024-04-06T13:49:53.713401+0000","last_unstale":"2024-04-06T13:49:54.699402+0000","last_undegraded":"2024-04-06T13:49:54.699402+0000","last_fullsized":"2024-04-06T13:49:54.699402+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:22:41.350836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.711230+0000","last_change":"2024-04-06T13:49:53.295170+0000","last_active":"2024-04-06T13:49:54.711230+0000","last_peered":"2024-04-06T13:49:54.711230+0000","last_clean":"2024-04-06T13:49:54.711230+0000","last_became_active":"2024-04-06T13:49:53.294914+0000","last_became_peered":"2024-04-06T13:49:53.294914+0000","last_unstale":"2024-04-06T13:49:54.711230+0000","last_undegraded":"2024-04-06T13:49:54.711230+0000","last_fullsized":"2024-04-06T13:49:54.711230+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:14:15.037776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.15","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710920+0000","last_change":"2024-04-06T13:49:53.704942+0000","last_active":"2024-04-06T13:49:54.710920+0000","last_peered":"2024-04-06T13:49:54.710920+0000","last_clean":"2024-04-06T13:49:54.710920+0000","last_became_active":"2024-04-06T13:49:53.704679+0000","last_became_peered":"2024-04-06T13:49:53.704679+0000","last_unstale":"2024-04-06T13:49:54.710920+0000","last_undegraded":"2024-04-06T13:49:54.710920+0000","last_fullsized":"2024-04-06T13:49:54.710920+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:48:15.813195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697779+0000","last_change":"2024-04-06T13:49:53.295042+0000","last_active":"2024-04-06T13:49:54.697779+0000","last_peered":"2024-04-06T13:49:54.697779+0000","last_clean":"2024-04-06T13:49:54.697779+0000","last_became_active":"2024-04-06T13:49:53.294638+0000","last_became_peered":"2024-04-06T13:49:53.294638+0000","last_unstale":"2024-04-06T13:49:54.697779+0000","last_undegraded":"2024-04-06T13:49:54.697779+0000","last_fullsized":"2024-04-06T13:49:54.697779+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:10:02.994827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.16","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697601+0000","last_change":"2024-04-06T13:49:53.713788+0000","last_active":"2024-04-06T13:49:54.697601+0000","last_peered":"2024-04-06T13:49:54.697601+0000","last_clean":"2024-04-06T13:49:54.697601+0000","last_became_active":"2024-04-06T13:49:53.713437+0000","last_became_peered":"2024-04-06T13:49:53.713437+0000","last_unstale":"2024-04-06T13:49:54.697601+0000","last_undegraded":"2024-04-06T13:49:54.697601+0000","last_fullsized":"2024-04-06T13:49:54.697601+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:08:39.564204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.15","version":"282'3","reported_seq":14,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697401+0000","last_change":"2024-04-06T13:49:53.297806+0000","last_active":"2024-04-06T13:49:54.697401+0000","last_peered":"2024-04-06T13:49:54.697401+0000","last_clean":"2024-04-06T13:49:54.697401+0000","last_became_active":"2024-04-06T13:49:53.297518+0000","last_became_peered":"2024-04-06T13:49:53.297518+0000","last_unstale":"2024-04-06T13:49:54.697401+0000","last_undegraded":"2024-04-06T13:49:54.697401+0000","last_fullsized":"2024-04-06T13:49:54.697401+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:13:35.763646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.17","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699300+0000","last_change":"2024-04-06T13:49:53.712762+0000","last_active":"2024-04-06T13:49:54.699300+0000","last_peered":"2024-04-06T13:49:54.699300+0000","last_clean":"2024-04-06T13:49:54.699300+0000","last_became_active":"2024-04-06T13:49:53.711489+0000","last_became_peered":"2024-04-06T13:49:53.711489+0000","last_unstale":"2024-04-06T13:49:54.699300+0000","last_undegraded":"2024-04-06T13:49:54.699300+0000","last_fullsized":"2024-04-06T13:49:54.699300+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:25:33.164716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.14","version":"282'6","reported_seq":17,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710191+0000","last_change":"2024-04-06T13:49:53.293553+0000","last_active":"2024-04-06T13:49:54.710191+0000","last_peered":"2024-04-06T13:49:54.710191+0000","last_clean":"2024-04-06T13:49:54.710191+0000","last_became_active":"2024-04-06T13:49:53.293297+0000","last_became_peered":"2024-04-06T13:49:53.293297+0000","last_unstale":"2024-04-06T13:49:54.710191+0000","last_undegraded":"2024-04-06T13:49:54.710191+0000","last_fullsized":"2024-04-06T13:49:54.710191+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:25.980159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.10","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698177+0000","last_change":"2024-04-06T13:49:53.709777+0000","last_active":"2024-04-06T13:49:54.698177+0000","last_peered":"2024-04-06T13:49:54.698177+0000","last_clean":"2024-04-06T13:49:54.698177+0000","last_became_active":"2024-04-06T13:49:53.709511+0000","last_became_peered":"2024-04-06T13:49:53.709511+0000","last_unstale":"2024-04-06T13:49:54.698177+0000","last_undegraded":"2024-04-06T13:49:54.698177+0000","last_fullsized":"2024-04-06T13:49:54.698177+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:46:38.182093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.13","version":"282'2","reported_seq":13,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710924+0000","last_change":"2024-04-06T13:49:53.286397+0000","last_active":"2024-04-06T13:49:54.710924+0000","last_peered":"2024-04-06T13:49:54.710924+0000","last_clean":"2024-04-06T13:49:54.710924+0000","last_became_active":"2024-04-06T13:49:53.286064+0000","last_became_peered":"2024-04-06T13:49:53.286064+0000","last_unstale":"2024-04-06T13:49:54.710924+0000","last_undegraded":"2024-04-06T13:49:54.710924+0000","last_fullsized":"2024-04-06T13:49:54.710924+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:28:12.467204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.11","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710343+0000","last_change":"2024-04-06T13:49:53.715124+0000","last_active":"2024-04-06T13:49:54.710343+0000","last_peered":"2024-04-06T13:49:54.710343+0000","last_clean":"2024-04-06T13:49:54.710343+0000","last_became_active":"2024-04-06T13:49:53.714882+0000","last_became_peered":"2024-04-06T13:49:53.714882+0000","last_unstale":"2024-04-06T13:49:54.710343+0000","last_undegraded":"2024-04-06T13:49:54.710343+0000","last_fullsized":"2024-04-06T13:49:54.710343+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:09:37.214050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710413+0000","last_change":"2024-04-06T13:49:53.299524+0000","last_active":"2024-04-06T13:49:54.710413+0000","last_peered":"2024-04-06T13:49:54.710413+0000","last_clean":"2024-04-06T13:49:54.710413+0000","last_became_active":"2024-04-06T13:49:53.299276+0000","last_became_peered":"2024-04-06T13:49:53.299276+0000","last_unstale":"2024-04-06T13:49:54.710413+0000","last_undegraded":"2024-04-06T13:49:54.710413+0000","last_fullsized":"2024-04-06T13:49:54.710413+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:57:56.001057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.12","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698978+0000","last_change":"2024-04-06T13:49:53.718373+0000","last_active":"2024-04-06T13:49:54.698978+0000","last_peered":"2024-04-06T13:49:54.698978+0000","last_clean":"2024-04-06T13:49:54.698978+0000","last_became_active":"2024-04-06T13:49:53.713088+0000","last_became_peered":"2024-04-06T13:49:53.713088+0000","last_unstale":"2024-04-06T13:49:54.698978+0000","last_undegraded":"2024-04-06T13:49:54.698978+0000","last_fullsized":"2024-04-06T13:49:54.698978+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:24:58.981619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710543+0000","last_change":"2024-04-06T13:49:53.300253+0000","last_active":"2024-04-06T13:49:54.710543+0000","last_peered":"2024-04-06T13:49:54.710543+0000","last_clean":"2024-04-06T13:49:54.710543+0000","last_became_active":"2024-04-06T13:49:53.300051+0000","last_became_peered":"2024-04-06T13:49:53.300051+0000","last_unstale":"2024-04-06T13:49:54.710543+0000","last_undegraded":"2024-04-06T13:49:54.710543+0000","last_fullsized":"2024-04-06T13:49:54.710543+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:06:27.253841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.13","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710776+0000","last_change":"2024-04-06T13:49:53.715115+0000","last_active":"2024-04-06T13:49:54.710776+0000","last_peered":"2024-04-06T13:49:54.710776+0000","last_clean":"2024-04-06T13:49:54.710776+0000","last_became_active":"2024-04-06T13:49:53.714823+0000","last_became_peered":"2024-04-06T13:49:53.714823+0000","last_unstale":"2024-04-06T13:49:54.710776+0000","last_undegraded":"2024-04-06T13:49:54.710776+0000","last_fullsized":"2024-04-06T13:49:54.710776+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:35:57.994706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.10","version":"282'8","reported_seq":19,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698855+0000","last_change":"2024-04-06T13:49:53.294367+0000","last_active":"2024-04-06T13:49:54.698855+0000","last_peered":"2024-04-06T13:49:54.698855+0000","last_clean":"2024-04-06T13:49:54.698855+0000","last_became_active":"2024-04-06T13:49:53.294109+0000","last_became_peered":"2024-04-06T13:49:53.294109+0000","last_unstale":"2024-04-06T13:49:54.698855+0000","last_undegraded":"2024-04-06T13:49:54.698855+0000","last_fullsized":"2024-04-06T13:49:54.698855+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:43:39.597248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.c","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710276+0000","last_change":"2024-04-06T13:49:53.699165+0000","last_active":"2024-04-06T13:49:54.710276+0000","last_peered":"2024-04-06T13:49:54.710276+0000","last_clean":"2024-04-06T13:49:54.710276+0000","last_became_active":"2024-04-06T13:49:53.698914+0000","last_became_peered":"2024-04-06T13:49:53.698914+0000","last_unstale":"2024-04-06T13:49:54.710276+0000","last_undegraded":"2024-04-06T13:49:54.710276+0000","last_fullsized":"2024-04-06T13:49:54.710276+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:55:32.313240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.f","version":"282'1","reported_seq":12,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699372+0000","last_change":"2024-04-06T13:49:53.295113+0000","last_active":"2024-04-06T13:49:54.699372+0000","last_peered":"2024-04-06T13:49:54.699372+0000","last_clean":"2024-04-06T13:49:54.699372+0000","last_became_active":"2024-04-06T13:49:53.294809+0000","last_became_peered":"2024-04-06T13:49:53.294809+0000","last_unstale":"2024-04-06T13:49:54.699372+0000","last_undegraded":"2024-04-06T13:49:54.699372+0000","last_fullsized":"2024-04-06T13:49:54.699372+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:53:13.312600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.d","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710362+0000","last_change":"2024-04-06T13:49:53.710731+0000","last_active":"2024-04-06T13:49:54.710362+0000","last_peered":"2024-04-06T13:49:54.710362+0000","last_clean":"2024-04-06T13:49:54.710362+0000","last_became_active":"2024-04-06T13:49:53.710476+0000","last_became_peered":"2024-04-06T13:49:53.710476+0000","last_unstale":"2024-04-06T13:49:54.710362+0000","last_undegraded":"2024-04-06T13:49:54.710362+0000","last_fullsized":"2024-04-06T13:49:54.710362+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:41:27.888300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.e","version":"282'2","reported_seq":13,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710580+0000","last_change":"2024-04-06T13:49:53.290628+0000","last_active":"2024-04-06T13:49:54.710580+0000","last_peered":"2024-04-06T13:49:54.710580+0000","last_clean":"2024-04-06T13:49:54.710580+0000","last_became_active":"2024-04-06T13:49:53.290367+0000","last_became_peered":"2024-04-06T13:49:53.290367+0000","last_unstale":"2024-04-06T13:49:54.710580+0000","last_undegraded":"2024-04-06T13:49:54.710580+0000","last_fullsized":"2024-04-06T13:49:54.710580+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:31:58.213975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.e","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.711271+0000","last_change":"2024-04-06T13:49:53.706895+0000","last_active":"2024-04-06T13:49:54.711271+0000","last_peered":"2024-04-06T13:49:54.711271+0000","last_clean":"2024-04-06T13:49:54.711271+0000","last_became_active":"2024-04-06T13:49:53.706561+0000","last_became_peered":"2024-04-06T13:49:53.706561+0000","last_unstale":"2024-04-06T13:49:54.711271+0000","last_undegraded":"2024-04-06T13:49:54.711271+0000","last_fullsized":"2024-04-06T13:49:54.711271+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:41:27.888300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.d","version":"282'1","reported_seq":12,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710459+0000","last_change":"2024-04-06T13:49:53.293292+0000","last_active":"2024-04-06T13:49:54.710459+0000","last_peered":"2024-04-06T13:49:54.710459+0000","last_clean":"2024-04-06T13:49:54.710459+0000","last_became_active":"2024-04-06T13:49:53.292868+0000","last_became_peered":"2024-04-06T13:49:53.292868+0000","last_unstale":"2024-04-06T13:49:54.710459+0000","last_undegraded":"2024-04-06T13:49:54.710459+0000","last_fullsized":"2024-04-06T13:49:54.710459+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:36:09.149670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.f","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710279+0000","last_change":"2024-04-06T13:49:53.707093+0000","last_active":"2024-04-06T13:49:54.710279+0000","last_peered":"2024-04-06T13:49:54.710279+0000","last_clean":"2024-04-06T13:49:54.710279+0000","last_became_active":"2024-04-06T13:49:53.706930+0000","last_became_peered":"2024-04-06T13:49:53.706930+0000","last_unstale":"2024-04-06T13:49:54.710279+0000","last_undegraded":"2024-04-06T13:49:54.710279+0000","last_fullsized":"2024-04-06T13:49:54.710279+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:56:48.466109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697473+0000","last_change":"2024-04-06T13:49:53.297510+0000","last_active":"2024-04-06T13:49:54.697473+0000","last_peered":"2024-04-06T13:49:54.697473+0000","last_clean":"2024-04-06T13:49:54.697473+0000","last_became_active":"2024-04-06T13:49:53.296479+0000","last_became_peered":"2024-04-06T13:49:53.296479+0000","last_unstale":"2024-04-06T13:49:54.697473+0000","last_undegraded":"2024-04-06T13:49:54.697473+0000","last_fullsized":"2024-04-06T13:49:54.697473+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:38:32.923938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.8","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699346+0000","last_change":"2024-04-06T13:49:53.722370+0000","last_active":"2024-04-06T13:49:54.699346+0000","last_peered":"2024-04-06T13:49:54.699346+0000","last_clean":"2024-04-06T13:49:54.699346+0000","last_became_active":"2024-04-06T13:49:53.720753+0000","last_became_peered":"2024-04-06T13:49:53.720753+0000","last_unstale":"2024-04-06T13:49:54.699346+0000","last_undegraded":"2024-04-06T13:49:54.699346+0000","last_fullsized":"2024-04-06T13:49:54.699346+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:13:24.517768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.b","version":"282'2","reported_seq":13,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698258+0000","last_change":"2024-04-06T13:49:53.291683+0000","last_active":"2024-04-06T13:49:54.698258+0000","last_peered":"2024-04-06T13:49:54.698258+0000","last_clean":"2024-04-06T13:49:54.698258+0000","last_became_active":"2024-04-06T13:49:53.291409+0000","last_became_peered":"2024-04-06T13:49:53.291409+0000","last_unstale":"2024-04-06T13:49:54.698258+0000","last_undegraded":"2024-04-06T13:49:54.698258+0000","last_fullsized":"2024-04-06T13:49:54.698258+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:45:05.628121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.9","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698221+0000","last_change":"2024-04-06T13:49:53.709932+0000","last_active":"2024-04-06T13:49:54.698221+0000","last_peered":"2024-04-06T13:49:54.698221+0000","last_clean":"2024-04-06T13:49:54.698221+0000","last_became_active":"2024-04-06T13:49:53.709544+0000","last_became_peered":"2024-04-06T13:49:53.709544+0000","last_unstale":"2024-04-06T13:49:54.698221+0000","last_undegraded":"2024-04-06T13:49:54.698221+0000","last_fullsized":"2024-04-06T13:49:54.698221+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:51:45.375899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699134+0000","last_change":"2024-04-06T13:49:53.294508+0000","last_active":"2024-04-06T13:49:54.699134+0000","last_peered":"2024-04-06T13:49:54.699134+0000","last_clean":"2024-04-06T13:49:54.699134+0000","last_became_active":"2024-04-06T13:49:53.294243+0000","last_became_peered":"2024-04-06T13:49:53.294243+0000","last_unstale":"2024-04-06T13:49:54.699134+0000","last_undegraded":"2024-04-06T13:49:54.699134+0000","last_fullsized":"2024-04-06T13:49:54.699134+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:11:56.282961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.a","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710393+0000","last_change":"2024-04-06T13:49:53.700719+0000","last_active":"2024-04-06T13:49:54.710393+0000","last_peered":"2024-04-06T13:49:54.710393+0000","last_clean":"2024-04-06T13:49:54.710393+0000","last_became_active":"2024-04-06T13:49:53.700418+0000","last_became_peered":"2024-04-06T13:49:53.700418+0000","last_unstale":"2024-04-06T13:49:54.710393+0000","last_undegraded":"2024-04-06T13:49:54.710393+0000","last_fullsized":"2024-04-06T13:49:54.710393+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:52:26.703747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710467+0000","last_change":"2024-04-06T13:49:53.292197+0000","last_active":"2024-04-06T13:49:54.710467+0000","last_peered":"2024-04-06T13:49:54.710467+0000","last_clean":"2024-04-06T13:49:54.710467+0000","last_became_active":"2024-04-06T13:49:53.291949+0000","last_became_peered":"2024-04-06T13:49:53.291949+0000","last_unstale":"2024-04-06T13:49:54.710467+0000","last_undegraded":"2024-04-06T13:49:54.710467+0000","last_fullsized":"2024-04-06T13:49:54.710467+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:40:08.863359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.b","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710903+0000","last_change":"2024-04-06T13:49:53.708630+0000","last_active":"2024-04-06T13:49:54.710903+0000","last_peered":"2024-04-06T13:49:54.710903+0000","last_clean":"2024-04-06T13:49:54.710903+0000","last_became_active":"2024-04-06T13:49:53.708141+0000","last_became_peered":"2024-04-06T13:49:53.708141+0000","last_unstale":"2024-04-06T13:49:54.710903+0000","last_undegraded":"2024-04-06T13:49:54.710903+0000","last_fullsized":"2024-04-06T13:49:54.710903+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:52:26.703747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.709561+0000","last_change":"2024-04-06T13:49:53.286973+0000","last_active":"2024-04-06T13:49:54.709561+0000","last_peered":"2024-04-06T13:49:54.709561+0000","last_clean":"2024-04-06T13:49:54.709561+0000","last_became_active":"2024-04-06T13:49:53.286685+0000","last_became_peered":"2024-04-06T13:49:53.286685+0000","last_unstale":"2024-04-06T13:49:54.709561+0000","last_undegraded":"2024-04-06T13:49:54.709561+0000","last_fullsized":"2024-04-06T13:49:54.709561+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:10.089185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.4","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699415+0000","last_change":"2024-04-06T13:49:53.717776+0000","last_active":"2024-04-06T13:49:54.699415+0000","last_peered":"2024-04-06T13:49:54.699415+0000","last_clean":"2024-04-06T13:49:54.699415+0000","last_became_active":"2024-04-06T13:49:53.713428+0000","last_became_peered":"2024-04-06T13:49:53.713428+0000","last_unstale":"2024-04-06T13:49:54.699415+0000","last_undegraded":"2024-04-06T13:49:54.699415+0000","last_fullsized":"2024-04-06T13:49:54.699415+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:10:03.999743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.7","version":"282'2","reported_seq":13,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710260+0000","last_change":"2024-04-06T13:49:53.291926+0000","last_active":"2024-04-06T13:49:54.710260+0000","last_peered":"2024-04-06T13:49:54.710260+0000","last_clean":"2024-04-06T13:49:54.710260+0000","last_became_active":"2024-04-06T13:49:53.291655+0000","last_became_peered":"2024-04-06T13:49:53.291655+0000","last_unstale":"2024-04-06T13:49:54.710260+0000","last_undegraded":"2024-04-06T13:49:54.710260+0000","last_fullsized":"2024-04-06T13:49:54.710260+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:28:30.863971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.5","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710630+0000","last_change":"2024-04-06T13:49:53.715836+0000","last_active":"2024-04-06T13:49:54.710630+0000","last_peered":"2024-04-06T13:49:54.710630+0000","last_clean":"2024-04-06T13:49:54.710630+0000","last_became_active":"2024-04-06T13:49:53.715586+0000","last_became_peered":"2024-04-06T13:49:53.715586+0000","last_unstale":"2024-04-06T13:49:54.710630+0000","last_undegraded":"2024-04-06T13:49:54.710630+0000","last_fullsized":"2024-04-06T13:49:54.710630+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:12:46.041335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698488+0000","last_change":"2024-04-06T13:49:53.285149+0000","last_active":"2024-04-06T13:49:54.698488+0000","last_peered":"2024-04-06T13:49:54.698488+0000","last_clean":"2024-04-06T13:49:54.698488+0000","last_became_active":"2024-04-06T13:49:53.284852+0000","last_became_peered":"2024-04-06T13:49:53.284852+0000","last_unstale":"2024-04-06T13:49:54.698488+0000","last_undegraded":"2024-04-06T13:49:54.698488+0000","last_fullsized":"2024-04-06T13:49:54.698488+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:05.094430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.6","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697588+0000","last_change":"2024-04-06T13:49:53.716258+0000","last_active":"2024-04-06T13:49:54.697588+0000","last_peered":"2024-04-06T13:49:54.697588+0000","last_clean":"2024-04-06T13:49:54.697588+0000","last_became_active":"2024-04-06T13:49:53.715803+0000","last_became_peered":"2024-04-06T13:49:53.715803+0000","last_unstale":"2024-04-06T13:49:54.697588+0000","last_undegraded":"2024-04-06T13:49:54.697588+0000","last_fullsized":"2024-04-06T13:49:54.697588+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:05:19.850459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.711013+0000","last_change":"2024-04-06T13:49:53.291155+0000","last_active":"2024-04-06T13:49:54.711013+0000","last_peered":"2024-04-06T13:49:54.711013+0000","last_clean":"2024-04-06T13:49:54.711013+0000","last_became_active":"2024-04-06T13:49:53.290659+0000","last_became_peered":"2024-04-06T13:49:53.290659+0000","last_unstale":"2024-04-06T13:49:54.711013+0000","last_undegraded":"2024-04-06T13:49:54.711013+0000","last_fullsized":"2024-04-06T13:49:54.711013+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:34:21.772869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.7","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698584+0000","last_change":"2024-04-06T13:49:53.718207+0000","last_active":"2024-04-06T13:49:54.698584+0000","last_peered":"2024-04-06T13:49:54.698584+0000","last_clean":"2024-04-06T13:49:54.698584+0000","last_became_active":"2024-04-06T13:49:53.717978+0000","last_became_peered":"2024-04-06T13:49:53.717978+0000","last_unstale":"2024-04-06T13:49:54.698584+0000","last_undegraded":"2024-04-06T13:49:54.698584+0000","last_fullsized":"2024-04-06T13:49:54.698584+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:57:30.699609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710076+0000","last_change":"2024-04-06T13:49:53.293115+0000","last_active":"2024-04-06T13:49:54.710076+0000","last_peered":"2024-04-06T13:49:54.710076+0000","last_clean":"2024-04-06T13:49:54.710076+0000","last_became_active":"2024-04-06T13:49:53.292836+0000","last_became_peered":"2024-04-06T13:49:53.292836+0000","last_unstale":"2024-04-06T13:49:54.710076+0000","last_undegraded":"2024-04-06T13:49:54.710076+0000","last_fullsized":"2024-04-06T13:49:54.710076+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:25.980159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.0","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710160+0000","last_change":"2024-04-06T13:49:53.719482+0000","last_active":"2024-04-06T13:49:54.710160+0000","last_peered":"2024-04-06T13:49:54.710160+0000","last_clean":"2024-04-06T13:49:54.710160+0000","last_became_active":"2024-04-06T13:49:53.718852+0000","last_became_peered":"2024-04-06T13:49:53.718852+0000","last_unstale":"2024-04-06T13:49:54.710160+0000","last_undegraded":"2024-04-06T13:49:54.710160+0000","last_fullsized":"2024-04-06T13:49:54.710160+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:26:04.390932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.709964+0000","last_change":"2024-04-06T13:49:53.294361+0000","last_active":"2024-04-06T13:49:54.709964+0000","last_peered":"2024-04-06T13:49:54.709964+0000","last_clean":"2024-04-06T13:49:54.709964+0000","last_became_active":"2024-04-06T13:49:53.294107+0000","last_became_peered":"2024-04-06T13:49:53.294107+0000","last_unstale":"2024-04-06T13:49:54.709964+0000","last_undegraded":"2024-04-06T13:49:54.709964+0000","last_fullsized":"2024-04-06T13:49:54.709964+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:57:56.001057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.1","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697319+0000","last_change":"2024-04-06T13:49:53.714973+0000","last_active":"2024-04-06T13:49:54.697319+0000","last_peered":"2024-04-06T13:49:54.697319+0000","last_clean":"2024-04-06T13:49:54.697319+0000","last_became_active":"2024-04-06T13:49:53.714730+0000","last_became_peered":"2024-04-06T13:49:53.714730+0000","last_unstale":"2024-04-06T13:49:54.697319+0000","last_undegraded":"2024-04-06T13:49:54.697319+0000","last_fullsized":"2024-04-06T13:49:54.697319+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:37:11.722361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697217+0000","last_change":"2024-04-06T13:49:53.295583+0000","last_active":"2024-04-06T13:49:54.697217+0000","last_peered":"2024-04-06T13:49:54.697217+0000","last_clean":"2024-04-06T13:49:54.697217+0000","last_became_active":"2024-04-06T13:49:53.295256+0000","last_became_peered":"2024-04-06T13:49:53.295256+0000","last_unstale":"2024-04-06T13:49:54.697217+0000","last_undegraded":"2024-04-06T13:49:54.697217+0000","last_fullsized":"2024-04-06T13:49:54.697217+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:24:57.976703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.2","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698129+0000","last_change":"2024-04-06T13:49:53.718300+0000","last_active":"2024-04-06T13:49:54.698129+0000","last_peered":"2024-04-06T13:49:54.698129+0000","last_clean":"2024-04-06T13:49:54.698129+0000","last_became_active":"2024-04-06T13:49:53.718063+0000","last_became_peered":"2024-04-06T13:49:53.718063+0000","last_unstale":"2024-04-06T13:49:54.698129+0000","last_undegraded":"2024-04-06T13:49:54.698129+0000","last_fullsized":"2024-04-06T13:49:54.698129+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:08:52.340541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710650+0000","last_change":"2024-04-06T13:49:53.300260+0000","last_active":"2024-04-06T13:49:54.710650+0000","last_peered":"2024-04-06T13:49:54.710650+0000","last_clean":"2024-04-06T13:49:54.710650+0000","last_became_active":"2024-04-06T13:49:53.300036+0000","last_became_peered":"2024-04-06T13:49:53.300036+0000","last_unstale":"2024-04-06T13:49:54.710650+0000","last_undegraded":"2024-04-06T13:49:54.710650+0000","last_fullsized":"2024-04-06T13:49:54.710650+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:37:05.203120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.3","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710579+0000","last_change":"2024-04-06T13:49:53.714764+0000","last_active":"2024-04-06T13:49:54.710579+0000","last_peered":"2024-04-06T13:49:54.710579+0000","last_clean":"2024-04-06T13:49:54.710579+0000","last_became_active":"2024-04-06T13:49:53.714003+0000","last_became_peered":"2024-04-06T13:49:53.714003+0000","last_unstale":"2024-04-06T13:49:54.710579+0000","last_undegraded":"2024-04-06T13:49:54.710579+0000","last_fullsized":"2024-04-06T13:49:54.710579+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:17:22.096202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.709643+0000","last_change":"2024-04-06T13:49:53.292671+0000","last_active":"2024-04-06T13:49:54.709643+0000","last_peered":"2024-04-06T13:49:54.709643+0000","last_clean":"2024-04-06T13:49:54.709643+0000","last_became_active":"2024-04-06T13:49:53.292399+0000","last_became_peered":"2024-04-06T13:49:53.292399+0000","last_unstale":"2024-04-06T13:49:54.709643+0000","last_undegraded":"2024-04-06T13:49:54.709643+0000","last_fullsized":"2024-04-06T13:49:54.709643+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:42:37.395499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.27","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.711313+0000","last_change":"2024-04-06T13:49:53.708856+0000","last_active":"2024-04-06T13:49:54.711313+0000","last_peered":"2024-04-06T13:49:54.711313+0000","last_clean":"2024-04-06T13:49:54.711313+0000","last_became_active":"2024-04-06T13:49:53.708527+0000","last_became_peered":"2024-04-06T13:49:53.708527+0000","last_unstale":"2024-04-06T13:49:54.711313+0000","last_undegraded":"2024-04-06T13:49:54.711313+0000","last_fullsized":"2024-04-06T13:49:54.711313+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:15:37.526304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"272'403","reported_seq":1489,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-04-06T13:49:53.703707+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:49:53.703707+0000","last_peered":"2024-04-06T13:49:53.703707+0000","last_clean":"2024-04-06T13:49:53.703707+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:49:53.703707+0000","last_undegraded":"2024-04-06T13:49:53.703707+0000","last_fullsized":"2024-04-06T13:49:53.703707+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":38,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":37,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576008,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57340,"kb_used_data":4080,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314500,"statfs":{"total":96636764160,"available":96578048000,"internally_reserved":0,"allocated":4177920,"data_stored":2438315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55700000000000005,"15min":0.59199999999999997},"min":{"1min":0.34499999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.747,"5min":0.751,"15min":0.82499999999999996},"last":0.628},{"interface":"front","average":{"1min":0.505,"5min":0.54700000000000004,"15min":0.58299999999999996},"min":{"1min":0.35099999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.753,"5min":0.78500000000000003,"15min":0.82899999999999996},"last":0.53700000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53800000000000003,"15min":0.57499999999999996},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.69199999999999995,"5min":0.77700000000000002,"15min":0.81299999999999994},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":0.58799999999999997},"min":{"1min":0.39200000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.85299999999999998},"last":0.60999999999999999}]},{"osd":2,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.53700000000000003,"15min":0.57599999999999996},"min":{"1min":0.379,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.63800000000000001,"5min":0.78600000000000003,"15min":0.83899999999999997},"last":0.434},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57099999999999995,"15min":0.59599999999999997},"min":{"1min":0.373,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.79700000000000004},"last":0.57499999999999996}]},{"osd":3,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.56399999999999995,"15min":0.59699999999999998},"min":{"1min":0.36799999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.72699999999999998,"5min":0.76700000000000002,"15min":0.81100000000000005},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.57399999999999995,"15min":0.59799999999999998},"min":{"1min":0.436,"5min":0.39000000000000001,"15min":0.34999999999999998},"max":{"1min":0.66500000000000004,"5min":0.78900000000000003,"15min":0.84099999999999997},"last":0.65600000000000003}]},{"osd":4,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.45700000000000002,"15min":0.46300000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.67600000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54200000000000004,"15min":0.54000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64700000000000002,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.56100000000000005}]},{"osd":5,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.498,"15min":0.496},"min":{"1min":0.316,"5min":0.315,"15min":0.308},"max":{"1min":0.68999999999999995,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.44500000000000001,"5min":0.35599999999999998,"15min":0.33600000000000002},"max":{"1min":0.77300000000000002,"5min":0.79700000000000004,"15min":0.82099999999999995},"last":0.67000000000000004}]},{"osd":6,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55400000000000005,"15min":0.55500000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.69099999999999995,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56100000000000005,"15min":0.58099999999999996},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.65700000000000003,"5min":0.81799999999999995,"15min":0.88200000000000001},"last":0.59999999999999998}]}]},{"osd":6,"up_from":13,"seq":55834576007,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60856,"kb_used_data":3500,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310984,"statfs":{"total":96636764160,"available":96574447616,"internally_reserved":0,"allocated":3584000,"data_stored":1847510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.51900000000000002,"15min":0.55000000000000004},"min":{"1min":0.35199999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.59199999999999997,"5min":0.79700000000000004,"15min":0.83799999999999997},"last":0.439},{"interface":"front","average":{"1min":0.497,"5min":0.53300000000000003,"15min":0.56200000000000006},"min":{"1min":0.314,"5min":0.31,"15min":0.31},"max":{"1min":0.63,"5min":0.76900000000000002,"15min":0.81999999999999995},"last":0.49099999999999999}]},{"osd":1,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.59099999999999997,"15min":0.61699999999999999},"min":{"1min":0.39600000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70699999999999996,"5min":0.81699999999999995,"15min":0.85799999999999998},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.60299999999999998,"15min":0.63900000000000001},"min":{"1min":0.45100000000000001,"5min":0.41099999999999998,"15min":0.378},"max":{"1min":0.74199999999999999,"5min":0.78900000000000003,"15min":0.82699999999999996},"last":0.57199999999999995}]},{"osd":2,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56100000000000005,"15min":0.58899999999999997},"min":{"1min":0.41799999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72899999999999998,"5min":0.79000000000000004,"15min":0.82899999999999996},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.54200000000000004,"15min":0.57899999999999996},"min":{"1min":0.35899999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67400000000000004,"5min":0.77700000000000002,"15min":0.80700000000000005},"last":0.59799999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.56699999999999995,"15min":0.59799999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69999999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61599999999999999,"15min":0.63900000000000001},"min":{"1min":0.441,"5min":0.39900000000000002,"15min":0.36099999999999999},"max":{"1min":0.77200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.55500000000000005}]},{"osd":4,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53300000000000003,"15min":0.53100000000000003},"min":{"1min":0.36099999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.81399999999999995},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51500000000000001,"15min":0.53400000000000003},"min":{"1min":0.32900000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.73699999999999999,"5min":0.82099999999999995,"15min":0.84099999999999997},"last":0.58299999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.501,"15min":0.50600000000000001},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.33300000000000002},"max":{"1min":0.754,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52200000000000002,"15min":0.53100000000000003},"min":{"1min":0.40400000000000003,"5min":0.316,"15min":0.316},"max":{"1min":0.72599999999999998,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.52200000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79300000000000004,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55800000000000005,"15min":0.56699999999999995},"min":{"1min":0.39900000000000002,"5min":0.35399999999999998,"15min":0.32800000000000001},"max":{"1min":0.77200000000000002,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.61899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834576006,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64952,"kb_used_data":3492,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306888,"statfs":{"total":96636764160,"available":96570253312,"internally_reserved":0,"allocated":3575808,"data_stored":1846906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.36299999999999999,"15min":0.38500000000000001},"min":{"1min":0.26100000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.443,"5min":0.621,"15min":0.621},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.39200000000000002,"15min":0.41999999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.504,"5min":0.56100000000000005,"15min":0.69399999999999995},"last":0.39500000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.39400000000000002,"15min":0.41699999999999998},"min":{"1min":0.25600000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.41999999999999998,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.42699999999999999,"15min":0.45600000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.51900000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.33200000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.44600000000000001,"15min":0.47199999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.51100000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.374,"5min":0.40799999999999997,"15min":0.434},"min":{"1min":0.29599999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.51700000000000002,"5min":0.63500000000000001,"15min":0.66100000000000003},"last":0.26700000000000002}]},{"osd":4,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.59399999999999997,"15min":0.63400000000000001},"min":{"1min":0.377,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.80300000000000005,"5min":0.89900000000000002,"15min":0.97399999999999998},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.61199999999999999,"15min":0.64300000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.79300000000000004,"5min":0.877,"15min":0.995},"last":0.56299999999999994}]},{"osd":5,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.60499999999999998,"15min":0.64200000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.72699999999999998,"5min":0.91800000000000004,"15min":1.073},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.626,"15min":0.65900000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.73999999999999999,"5min":0.96499999999999997,"15min":1.0209999999999999},"last":0.69199999999999995}]},{"osd":6,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.61799999999999999,"15min":0.64900000000000002},"min":{"1min":0.435,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.753,"5min":0.89200000000000002,"15min":1},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.63600000000000001,"15min":0.67200000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76600000000000001,"5min":0.94599999999999995,"15min":1.036},"last":0.68200000000000005}]},{"osd":7,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.66400000000000003,"15min":0.68799999999999994},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.94899999999999995,"15min":1.0649999999999999},"last":0.746},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.626,"15min":0.66000000000000003},"min":{"1min":0.44700000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.78100000000000003,"5min":0.92900000000000005,"15min":1.056},"last":0.66800000000000004}]}]},{"osd":0,"up_from":13,"seq":55834576006,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61440,"kb_used_data":4076,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310400,"statfs":{"total":96636764160,"available":96573849600,"internally_reserved":0,"allocated":4173824,"data_stored":2437364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.36899999999999999,"15min":0.38900000000000001},"min":{"1min":0.24299999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.47199999999999998,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.245},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.41299999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.50900000000000001,"5min":0.58599999999999997,"15min":0.70799999999999996},"last":0.314}]},{"osd":2,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39000000000000001,"15min":0.42199999999999999},"min":{"1min":0.28299999999999997,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.53900000000000003,"5min":0.65400000000000003,"15min":0.70099999999999996},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.41699999999999998,"15min":0.441},"min":{"1min":0.29899999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.58199999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.28699999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.39600000000000002,"15min":0.42099999999999999},"min":{"1min":0.27300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52400000000000002,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.44800000000000001,"15min":0.47399999999999998},"min":{"1min":0.313,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.60899999999999999,"5min":0.71899999999999997,"15min":0.75700000000000001},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.59099999999999997,"15min":0.63600000000000001},"min":{"1min":0.38500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.85199999999999998,"5min":0.93100000000000005,"15min":1.0329999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.60299999999999998,"15min":0.64100000000000001},"min":{"1min":0.39800000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86099999999999999,"5min":0.999,"15min":0.999},"last":0.57499999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59499999999999997,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.83899999999999997,"5min":0.91800000000000004,"15min":1.0009999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.627,"15min":0.66200000000000003},"min":{"1min":0.42699999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.86499999999999999,"5min":0.92800000000000005,"15min":1.052},"last":0.67600000000000005}]},{"osd":6,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63100000000000001,"15min":0.67000000000000004},"min":{"1min":0.438,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.81699999999999995,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.629,"15min":0.66800000000000004},"min":{"1min":0.44900000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":1.085},"last":0.56000000000000005}]},{"osd":7,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.64900000000000002,"15min":0.68799999999999994},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.89400000000000002,"5min":1.01,"15min":1.121},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.64000000000000001,"15min":0.67000000000000004},"min":{"1min":0.47299999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.82999999999999996,"5min":0.95799999999999996,"15min":1.0229999999999999},"last":0.59199999999999997}]}]},{"osd":2,"up_from":13,"seq":55834576008,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60872,"kb_used_data":3508,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310968,"statfs":{"total":96636764160,"available":96574431232,"internally_reserved":0,"allocated":3592192,"data_stored":1848540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.36799999999999999,"15min":0.38100000000000001},"min":{"1min":0.22,"5min":0.184,"15min":0.184},"max":{"1min":0.59399999999999997,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.39700000000000002,"15min":0.41899999999999998},"min":{"1min":0.26200000000000001,"5min":0.251,"15min":0.249},"max":{"1min":0.54600000000000004,"5min":0.71499999999999997,"15min":1.044},"last":0.41799999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.371,"15min":0.39900000000000002},"min":{"1min":0.23599999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.46899999999999997,"5min":0.52900000000000003,"15min":0.753},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.41799999999999998,"15min":0.441},"min":{"1min":0.27000000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.82199999999999995},"last":0.44900000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.42199999999999999,"15min":0.45200000000000001},"min":{"1min":0.27000000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.621,"5min":0.63900000000000001,"15min":0.70299999999999996},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.42499999999999999,"15min":0.443},"min":{"1min":0.27200000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.39600000000000002}]},{"osd":4,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61099999999999999,"15min":0.63300000000000001},"min":{"1min":0.436,"5min":0.40200000000000002,"15min":0.376},"max":{"1min":0.77600000000000002,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.623},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.63,"15min":0.65400000000000003},"min":{"1min":0.40999999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82799999999999996,"5min":1.022,"15min":1.022},"last":0.78900000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.629,"15min":0.65800000000000003},"min":{"1min":0.44900000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79000000000000004,"5min":0.98599999999999999,"15min":1.0149999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.63800000000000001,"15min":0.66000000000000003},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85199999999999998,"5min":1.028,"15min":1.028},"last":0.66100000000000003}]},{"osd":6,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64500000000000002,"15min":0.66600000000000004},"min":{"1min":0.45300000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86899999999999999,"5min":0.97199999999999998,"15min":1.091},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.625,"5min":0.65900000000000003,"15min":0.68799999999999994},"min":{"1min":0.47699999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84099999999999997,"5min":0.95699999999999996,"15min":0.96099999999999997},"last":0.65200000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.66100000000000003,"15min":0.68300000000000005},"min":{"1min":0.47299999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91300000000000003,"5min":0.94499999999999995,"15min":0.95399999999999996},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.66900000000000004,"15min":0.69999999999999996},"min":{"1min":0.46100000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.89100000000000001,"5min":1.01,"15min":1.01},"last":0.78200000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576007,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52672,"kb_used_data":3500,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319168,"statfs":{"total":96636764160,"available":96582828032,"internally_reserved":0,"allocated":3584000,"data_stored":1847959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.36099999999999999,"15min":0.379},"min":{"1min":0.23599999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.51300000000000001,"5min":0.55200000000000005,"15min":0.72799999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.317,"5min":0.375,"15min":0.40300000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.48299999999999998,"5min":0.58599999999999997,"15min":0.66700000000000004},"last":0.311}]},{"osd":1,"last update":"Sat Apr 6 13:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.378,"15min":0.39800000000000002},"min":{"1min":0.24399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.44600000000000001,"5min":0.59499999999999997,"15min":0.65700000000000003},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.40400000000000003,"15min":0.42999999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.59199999999999997,"5min":0.63,"15min":0.75800000000000001},"last":0.40799999999999997}]},{"osd":2,"last update":"Sat Apr 6 13:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.439,"15min":0.47299999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.56899999999999995,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.45800000000000002,"15min":0.48899999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.63400000000000001,"5min":0.67800000000000005,"15min":0.79300000000000004},"last":0.435}]},{"osd":4,"last update":"Sat Apr 6 13:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.60199999999999998,"15min":0.626},"min":{"1min":0.435,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.77100000000000002,"5min":0.88200000000000001,"15min":1.018},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.61299999999999999,"15min":0.63700000000000001},"min":{"1min":0.371,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.85399999999999998,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.51600000000000001}]},{"osd":5,"last update":"Sat Apr 6 13:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64700000000000002,"15min":0.67200000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.76700000000000002,"5min":0.95699999999999996,"15min":0.999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66000000000000003,"15min":0.68200000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84999999999999998,"5min":1.036,"15min":1.036},"last":0.71799999999999997}]},{"osd":6,"last update":"Sat Apr 6 13:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.63100000000000001,"15min":0.65500000000000003},"min":{"1min":0.40600000000000003,"5min":0.31,"15min":0.31},"max":{"1min":0.77900000000000003,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.64500000000000002,"15min":0.67200000000000004},"min":{"1min":0.45700000000000002,"5min":0.41999999999999998,"15min":0.41099999999999998},"max":{"1min":0.80500000000000005,"5min":0.91600000000000004,"15min":0.98799999999999999},"last":0.69999999999999996}]},{"osd":7,"last update":"Sat Apr 6 13:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.64100000000000001,"15min":0.66400000000000003},"min":{"1min":0.434,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85299999999999998,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.754},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.65600000000000003,"15min":0.68799999999999994},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81999999999999995,"5min":1.0209999999999999,"15min":1.024},"last":0.66000000000000003}]}]},{"osd":4,"up_from":13,"seq":55834576007,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48596,"kb_used_data":3520,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323244,"statfs":{"total":96636764160,"available":96587001856,"internally_reserved":0,"allocated":3604480,"data_stored":1850231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55300000000000005,"15min":0.57999999999999996},"min":{"1min":0.317,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73299999999999998,"5min":0.80900000000000005,"15min":0.88500000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.57599999999999996,"15min":0.59099999999999997},"min":{"1min":0.41699999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79200000000000004,"5min":1.202,"15min":1.202},"last":0.55300000000000005}]},{"osd":1,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60099999999999998,"15min":0.63400000000000001},"min":{"1min":0.379,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.74299999999999999,"5min":1.256,"15min":1.256},"last":0.625},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61499999999999999,"15min":0.624},"min":{"1min":0.39300000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.71999999999999997,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.58699999999999997}]},{"osd":2,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.56899999999999995,"15min":0.59099999999999997},"min":{"1min":0.377,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.68200000000000005,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.59299999999999997,"15min":0.61199999999999999},"min":{"1min":0.373,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89500000000000002,"5min":1.294,"15min":1.294},"last":0.65400000000000003}]},{"osd":3,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.57499999999999996,"15min":0.58999999999999997},"min":{"1min":0.35999999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80800000000000005,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60399999999999998,"15min":0.62},"min":{"1min":0.439,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76700000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.60199999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56100000000000005,"15min":0.52000000000000002},"min":{"1min":0.33800000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.75,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56999999999999995,"15min":0.54100000000000004},"min":{"1min":0.372,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.75,"5min":4.931,"15min":4.931},"last":0.42599999999999999}]},{"osd":6,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53900000000000003,"15min":0.53400000000000003},"min":{"1min":0.40699999999999997,"5min":0.34100000000000003,"15min":0.33300000000000002},"max":{"1min":0.78300000000000003,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56599999999999995,"15min":0.56999999999999995},"min":{"1min":0.41099999999999998,"5min":0.35799999999999998,"15min":0.34699999999999998},"max":{"1min":0.97099999999999997,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.52200000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60199999999999998,"15min":0.58199999999999996},"min":{"1min":0.35899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.84999999999999998,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.59399999999999997,"15min":0.60099999999999998},"min":{"1min":0.48599999999999999,"5min":0.38,"15min":0.373},"max":{"1min":0.82799999999999996,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.40300000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576007,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66620,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305220,"statfs":{"total":96636764160,"available":96568545280,"internally_reserved":0,"allocated":3579904,"data_stored":1846940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.54000000000000004,"15min":0.57699999999999996},"min":{"1min":0.33100000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69099999999999995,"5min":0.74099999999999999,"15min":0.82199999999999995},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.33000000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.62,"5min":0.81999999999999995,"15min":0.89200000000000002},"last":0.45200000000000001}]},{"osd":1,"last update":"Sat Apr 6 13:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.53900000000000003,"15min":0.58099999999999996},"min":{"1min":0.34699999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70899999999999996,"5min":0.80300000000000005,"15min":0.81399999999999995},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.505,"5min":0.53900000000000003,"15min":0.57699999999999996},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.64900000000000002,"5min":0.871,"15min":0.871},"last":0.63}]},{"osd":2,"last update":"Sat Apr 6 13:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.55300000000000005,"15min":0.58999999999999997},"min":{"1min":0.374,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72899999999999998,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.55400000000000005,"15min":0.58799999999999997},"min":{"1min":0.376,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70999999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.745}]},{"osd":3,"last update":"Sat Apr 6 13:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55200000000000005,"15min":0.58999999999999997},"min":{"1min":0.37,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.75900000000000001,"5min":0.81100000000000005,"15min":0.94499999999999995},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.58699999999999997,"15min":0.624},"min":{"1min":0.38,"5min":0.371,"15min":0.371},"max":{"1min":0.77800000000000002,"5min":0.88200000000000001,"15min":0.91500000000000004},"last":0.58999999999999997}]},{"osd":4,"last update":"Sat Apr 6 13:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.52200000000000002,"15min":0.50700000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.27400000000000002},"max":{"1min":0.75900000000000001,"5min":0.85699999999999998,"15min":1.0369999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.34699999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.82999999999999996,"5min":0.88800000000000001,"15min":1.1639999999999999},"last":0.69699999999999995}]},{"osd":6,"last update":"Sat Apr 6 13:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54600000000000004,"15min":0.53200000000000003},"min":{"1min":0.32600000000000001,"5min":0.313,"15min":0.28899999999999998},"max":{"1min":0.89900000000000002,"5min":0.94999999999999996,"15min":1.0169999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53800000000000003,"15min":0.53200000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.84299999999999997,"5min":0.878,"15min":1.0009999999999999},"last":0.61399999999999999}]},{"osd":7,"last update":"Sat Apr 6 13:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.55400000000000005,"15min":0.54800000000000004},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.85599999999999998,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.64800000000000002,"15min":0.64900000000000002},"min":{"1min":0.45400000000000001,"5min":0.41699999999999998,"15min":0.38300000000000001},"max":{"1min":0.87,"5min":0.92300000000000004,"15min":0.96599999999999997},"last":0.72399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:49:57.819 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:49:57.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.976+0000 7f12ad6cf700 1 Processor -- start 2024-04-06T13:49:57.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.976+0000 7f12ad6cf700 1 -- start start 2024-04-06T13:49:57.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.977+0000 7f12ad6cf700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f12a815d590 0x7f12a80b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.977+0000 7f12ad6cf700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f12a80b4ed0 0x7f12a80b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.977+0000 7f12ad6cf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a80b5cc0 con 0x7f12a80b6580 2024-04-06T13:49:57.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.977+0000 7f12ad6cf700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a80b58c0 con 0x7f12a80b4ed0 2024-04-06T13:49:57.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.977+0000 7f12ad6cf700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a80d1500 con 0x7f12a815d590 2024-04-06T13:49:57.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.977+0000 7f12a67fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f12a80b4ed0 0x7f12a80b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.977+0000 7f12a77fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f12a80b6580 0x7f12a80b69f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49924/0 (socket says 172.21.15.189:49924) 2024-04-06T13:49:57.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.977+0000 7f12a77fe700 1 -- 172.21.15.189:0/667228893 learned_addr learned my addr 172.21.15.189:0/667228893 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:57.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.977+0000 7f12a6ffd700 1 --2- 172.21.15.189:0/667228893 >> v2:172.21.15.192:3300/0 conn(0x7f12a815d590 0x7f12a80b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.978+0000 7f12a67fc700 1 -- 172.21.15.189:0/667228893 >> v2:172.21.15.192:3300/0 conn(0x7f12a815d590 msgr2=0x7f12a80b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:57.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.978+0000 7f12a67fc700 1 --2- 172.21.15.189:0/667228893 >> v2:172.21.15.192:3300/0 conn(0x7f12a815d590 0x7f12a80b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.978+0000 7f12a67fc700 1 -- 172.21.15.189:0/667228893 >> v1:172.21.15.189:6789/0 conn(0x7f12a80b6580 legacy=0x7f12a80b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:57.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.978+0000 7f12a67fc700 1 -- 172.21.15.189:0/667228893 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a80d17e0 con 0x7f12a80b4ed0 2024-04-06T13:49:57.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.978+0000 7f12a5ffb700 1 -- 172.21.15.189:0/667228893 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210012395 0 0) 0x7f12a80b5cc0 con 0x7f12a80b6580 2024-04-06T13:49:57.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.978+0000 7f12a67fc700 1 --2- 172.21.15.189:0/667228893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a80b4ed0 0x7f12a80b5380 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7f12a80b5fe0 tx=0x7f1298009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7e94242266f5f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:57.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.978+0000 7f12a5ffb700 1 -- 172.21.15.189:0/667228893 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1298011830 con 0x7f12a80b4ed0 2024-04-06T13:49:57.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.978+0000 7f12a5ffb700 1 -- 172.21.15.189:0/667228893 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1298011990 con 0x7f12a80b4ed0 2024-04-06T13:49:57.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.978+0000 7f12a5ffb700 1 -- 172.21.15.189:0/667228893 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1298011d20 con 0x7f12a80b4ed0 2024-04-06T13:49:57.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.978+0000 7f12ad6cf700 1 -- 172.21.15.189:0/667228893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a80b4ed0 msgr2=0x7f12a80b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:57.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.978+0000 7f12ad6cf700 1 --2- 172.21.15.189:0/667228893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a80b4ed0 0x7f12a80b5380 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7f12a80b5fe0 tx=0x7f1298009b20 comp rx=0 tx=0).stop 2024-04-06T13:49:57.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.978+0000 7f12ad6cf700 1 -- 172.21.15.189:0/667228893 shutdown_connections 2024-04-06T13:49:57.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.978+0000 7f12ad6cf700 1 --2- 172.21.15.189:0/667228893 >> v2:172.21.15.192:3300/0 conn(0x7f12a815d590 0x7f12a80b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.978+0000 7f12ad6cf700 1 --2- 172.21.15.189:0/667228893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a80b4ed0 0x7f12a80b5380 unknown :-1 s=CLOSED pgs=3097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.978+0000 7f12ad6cf700 1 -- 172.21.15.189:0/667228893 >> 172.21.15.189:0/667228893 conn(0x7f12a806bbc0 msgr2=0x7f12a806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:57.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.978+0000 7f12ad6cf700 1 -- 172.21.15.189:0/667228893 shutdown_connections 2024-04-06T13:49:57.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.978+0000 7f12ad6cf700 1 -- 172.21.15.189:0/667228893 wait complete. 2024-04-06T13:49:57.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.980+0000 7f12ad6cf700 1 Processor -- start 2024-04-06T13:49:57.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.980+0000 7f12ad6cf700 1 -- start start 2024-04-06T13:49:57.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.980+0000 7f12ad6cf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a80b4ed0 0x7f12a8152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.980+0000 7f12ad6cf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12a80b6580 0x7f12a80ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.980+0000 7f12ad6cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12a815d590 0x7f12a80c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.980+0000 7f12ad6cf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a8155e80 con 0x7f12a80b4ed0 2024-04-06T13:49:57.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.980+0000 7f12ad6cf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a80cd410 con 0x7f12a80b6580 2024-04-06T13:49:57.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.980+0000 7f12ad6cf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a80cd150 con 0x7f12a815d590 2024-04-06T13:49:57.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.981+0000 7f12a6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a80b4ed0 0x7f12a8152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.981+0000 7f12a6ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a80b4ed0 0x7f12a8152e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59442/0 (socket says 172.21.15.189:59442) 2024-04-06T13:49:57.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.981+0000 7f12a6ffd700 1 -- 172.21.15.189:0/1822325494 learned_addr learned my addr 172.21.15.189:0/1822325494 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:57.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.981+0000 7f12a67fc700 1 --2- 172.21.15.189:0/1822325494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12a80b6580 0x7f12a80ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.981+0000 7f12a77fe700 1 --2- 172.21.15.189:0/1822325494 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12a815d590 0x7f12a80c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.981+0000 7f12a6ffd700 1 -- 172.21.15.189:0/1822325494 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12a815d590 msgr2=0x7f12a80c8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:57.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.981+0000 7f12a6ffd700 1 --2- 172.21.15.189:0/1822325494 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12a815d590 0x7f12a80c8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.981+0000 7f12a6ffd700 1 -- 172.21.15.189:0/1822325494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12a80b6580 msgr2=0x7f12a80ccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:57.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.981+0000 7f12a6ffd700 1 --2- 172.21.15.189:0/1822325494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12a80b6580 0x7f12a80ccc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:57.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.981+0000 7f12a6ffd700 1 -- 172.21.15.189:0/1822325494 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12980094a0 con 0x7f12a80b4ed0 2024-04-06T13:49:57.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.981+0000 7f12a67fc700 1 --2- 172.21.15.189:0/1822325494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12a80b6580 0x7f12a80ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:57.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.981+0000 7f12a77fe700 1 --2- 172.21.15.189:0/1822325494 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12a815d590 0x7f12a80c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:57.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.981+0000 7f12a6ffd700 1 --2- 172.21.15.189:0/1822325494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a80b4ed0 0x7f12a8152e60 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 crypto rx=0x7f129c00d800 tx=0x7f129c00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:57.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.981+0000 7f128f7fe700 1 -- 172.21.15.189:0/1822325494 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c0125f0 con 0x7f12a80b4ed0 2024-04-06T13:49:57.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.981+0000 7f128f7fe700 1 -- 172.21.15.189:0/1822325494 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129c012750 con 0x7f12a80b4ed0 2024-04-06T13:49:57.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.981+0000 7f12ad6cf700 1 -- 172.21.15.189:0/1822325494 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a81554e0 con 0x7f12a80b4ed0 2024-04-06T13:49:57.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.981+0000 7f128f7fe700 1 -- 172.21.15.189:0/1822325494 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c012b00 con 0x7f12a80b4ed0 2024-04-06T13:49:57.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.981+0000 7f12ad6cf700 1 -- 172.21.15.189:0/1822325494 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a8155aa0 con 0x7f12a80b4ed0 2024-04-06T13:49:57.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.982+0000 7f12ad6cf700 1 -- 172.21.15.189:0/1822325494 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a8074990 con 0x7f12a80b4ed0 2024-04-06T13:49:57.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.983+0000 7f128f7fe700 1 -- 172.21.15.189:0/1822325494 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f129c012c60 con 0x7f12a80b4ed0 2024-04-06T13:49:57.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.983+0000 7f128f7fe700 1 --2- 172.21.15.189:0/1822325494 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f129406ade0 0x7f129406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:57.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.983+0000 7f128f7fe700 1 -- 172.21.15.189:0/1822325494 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f129c089a80 con 0x7f12a80b4ed0 2024-04-06T13:49:57.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.983+0000 7f12a67fc700 1 --2- 172.21.15.189:0/1822325494 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f129406ade0 0x7f129406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:57.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.984+0000 7f12a67fc700 1 --2- 172.21.15.189:0/1822325494 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f129406ade0 0x7f129406d2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f12a81528c0 tx=0x7f1298008030 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:57.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:57.988+0000 7f128f7fe700 1 -- 172.21.15.189:0/1822325494 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f129c054930 con 0x7f12a80b4ed0 2024-04-06T13:49:58.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.177+0000 7f12ad6cf700 1 -- 172.21.15.189:0/1822325494 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f12a80d2080 con 0x7f129406ade0 2024-04-06T13:49:58.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.185+0000 7f128f7fe700 1 -- 172.21.15.189:0/1822325494 <== mgr.13152 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289904 (secure 0 0 0) 0x7f1298011b00 con 0x7f129406ade0 2024-04-06T13:49:58.189 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:58.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.188+0000 7f12ad6cf700 1 -- 172.21.15.189:0/1822325494 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f129406ade0 msgr2=0x7f129406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:58.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.188+0000 7f12ad6cf700 1 --2- 172.21.15.189:0/1822325494 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f129406ade0 0x7f129406d2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f12a81528c0 tx=0x7f1298008030 comp rx=0 tx=0).stop 2024-04-06T13:49:58.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.188+0000 7f12ad6cf700 1 -- 172.21.15.189:0/1822325494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a80b4ed0 msgr2=0x7f12a8152e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:58.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.188+0000 7f12ad6cf700 1 --2- 172.21.15.189:0/1822325494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a80b4ed0 0x7f12a8152e60 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 crypto rx=0x7f129c00d800 tx=0x7f129c00dbc0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.188+0000 7f12ad6cf700 1 -- 172.21.15.189:0/1822325494 shutdown_connections 2024-04-06T13:49:58.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.188+0000 7f12ad6cf700 1 --2- 172.21.15.189:0/1822325494 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f129406ade0 0x7f129406d2a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.188+0000 7f12ad6cf700 1 --2- 172.21.15.189:0/1822325494 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12a815d590 0x7f12a80c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.188+0000 7f12ad6cf700 1 --2- 172.21.15.189:0/1822325494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a80b4ed0 0x7f12a8152e60 unknown :-1 s=CLOSED pgs=3098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.188+0000 7f12ad6cf700 1 --2- 172.21.15.189:0/1822325494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12a80b6580 0x7f12a80ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.188+0000 7f12ad6cf700 1 -- 172.21.15.189:0/1822325494 >> 172.21.15.189:0/1822325494 conn(0x7f12a806bbc0 msgr2=0x7f12a806bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:58.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.188+0000 7f12ad6cf700 1 -- 172.21.15.189:0/1822325494 shutdown_connections 2024-04-06T13:49:58.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.188+0000 7f12ad6cf700 1 -- 172.21.15.189:0/1822325494 wait complete. 2024-04-06T13:49:58.193 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:49:58.201 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-06T13:49:57.999015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":723,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":441,"ondisk_log_size":441,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":473348,"kb_used_data":29172,"kb_used_omap":12,"kb_used_meta":444019,"kb_avail":754501372,"statfs":{"total":773094113280,"available":772609404928,"internally_reserved":0,"allocated":29872128,"data_stored":15963765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":454676024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.727819"},"pg_stats":[{"pgid":"38.3f","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710317+0000","last_change":"2024-04-06T13:49:53.698145+0000","last_active":"2024-04-06T13:49:54.710317+0000","last_peered":"2024-04-06T13:49:54.710317+0000","last_clean":"2024-04-06T13:49:54.710317+0000","last_became_active":"2024-04-06T13:49:53.697937+0000","last_became_peered":"2024-04-06T13:49:53.697937+0000","last_unstale":"2024-04-06T13:49:54.710317+0000","last_undegraded":"2024-04-06T13:49:54.710317+0000","last_fullsized":"2024-04-06T13:49:54.710317+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:42:43.790123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.3e","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697798+0000","last_change":"2024-04-06T13:49:53.716385+0000","last_active":"2024-04-06T13:49:54.697798+0000","last_peered":"2024-04-06T13:49:54.697798+0000","last_clean":"2024-04-06T13:49:54.697798+0000","last_became_active":"2024-04-06T13:49:53.716147+0000","last_became_peered":"2024-04-06T13:49:53.716147+0000","last_unstale":"2024-04-06T13:49:54.697798+0000","last_undegraded":"2024-04-06T13:49:54.697798+0000","last_fullsized":"2024-04-06T13:49:54.697798+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:43:30.859611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.3d","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.709965+0000","last_change":"2024-04-06T13:49:53.713242+0000","last_active":"2024-04-06T13:49:54.709965+0000","last_peered":"2024-04-06T13:49:54.709965+0000","last_clean":"2024-04-06T13:49:54.709965+0000","last_became_active":"2024-04-06T13:49:53.712846+0000","last_became_peered":"2024-04-06T13:49:53.712846+0000","last_unstale":"2024-04-06T13:49:54.709965+0000","last_undegraded":"2024-04-06T13:49:54.709965+0000","last_fullsized":"2024-04-06T13:49:54.709965+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:03:26.515333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.3c","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699161+0000","last_change":"2024-04-06T13:49:53.713265+0000","last_active":"2024-04-06T13:49:54.699161+0000","last_peered":"2024-04-06T13:49:54.699161+0000","last_clean":"2024-04-06T13:49:54.699161+0000","last_became_active":"2024-04-06T13:49:53.712513+0000","last_became_peered":"2024-04-06T13:49:53.712513+0000","last_unstale":"2024-04-06T13:49:54.699161+0000","last_undegraded":"2024-04-06T13:49:54.699161+0000","last_fullsized":"2024-04-06T13:49:54.699161+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:32:56.445394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.3b","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710783+0000","last_change":"2024-04-06T13:49:53.709755+0000","last_active":"2024-04-06T13:49:54.710783+0000","last_peered":"2024-04-06T13:49:54.710783+0000","last_clean":"2024-04-06T13:49:54.710783+0000","last_became_active":"2024-04-06T13:49:53.709247+0000","last_became_peered":"2024-04-06T13:49:53.709247+0000","last_unstale":"2024-04-06T13:49:54.710783+0000","last_undegraded":"2024-04-06T13:49:54.710783+0000","last_fullsized":"2024-04-06T13:49:54.710783+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:35:57.994706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.3a","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710742+0000","last_change":"2024-04-06T13:49:53.707246+0000","last_active":"2024-04-06T13:49:54.710742+0000","last_peered":"2024-04-06T13:49:54.710742+0000","last_clean":"2024-04-06T13:49:54.710742+0000","last_became_active":"2024-04-06T13:49:53.706917+0000","last_became_peered":"2024-04-06T13:49:53.706917+0000","last_unstale":"2024-04-06T13:49:54.710742+0000","last_undegraded":"2024-04-06T13:49:54.710742+0000","last_fullsized":"2024-04-06T13:49:54.710742+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:26:04.390932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.39","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710799+0000","last_change":"2024-04-06T13:49:53.714607+0000","last_active":"2024-04-06T13:49:54.710799+0000","last_peered":"2024-04-06T13:49:54.710799+0000","last_clean":"2024-04-06T13:49:54.710799+0000","last_became_active":"2024-04-06T13:49:53.714305+0000","last_became_peered":"2024-04-06T13:49:53.714305+0000","last_unstale":"2024-04-06T13:49:54.710799+0000","last_undegraded":"2024-04-06T13:49:54.710799+0000","last_fullsized":"2024-04-06T13:49:54.710799+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:28:13.472120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.38","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.709889+0000","last_change":"2024-04-06T13:49:53.703689+0000","last_active":"2024-04-06T13:49:54.709889+0000","last_peered":"2024-04-06T13:49:54.709889+0000","last_clean":"2024-04-06T13:49:54.709889+0000","last_became_active":"2024-04-06T13:49:53.703436+0000","last_became_peered":"2024-04-06T13:49:53.703436+0000","last_unstale":"2024-04-06T13:49:54.709889+0000","last_undegraded":"2024-04-06T13:49:54.709889+0000","last_fullsized":"2024-04-06T13:49:54.709889+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:11:11.857561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.37","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698482+0000","last_change":"2024-04-06T13:49:53.712491+0000","last_active":"2024-04-06T13:49:54.698482+0000","last_peered":"2024-04-06T13:49:54.698482+0000","last_clean":"2024-04-06T13:49:54.698482+0000","last_became_active":"2024-04-06T13:49:53.712235+0000","last_became_peered":"2024-04-06T13:49:53.712235+0000","last_unstale":"2024-04-06T13:49:54.698482+0000","last_undegraded":"2024-04-06T13:49:54.698482+0000","last_fullsized":"2024-04-06T13:49:54.698482+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:53:15.876796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.36","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.711166+0000","last_change":"2024-04-06T13:49:53.716746+0000","last_active":"2024-04-06T13:49:54.711166+0000","last_peered":"2024-04-06T13:49:54.711166+0000","last_clean":"2024-04-06T13:49:54.711166+0000","last_became_active":"2024-04-06T13:49:53.716519+0000","last_became_peered":"2024-04-06T13:49:53.716519+0000","last_unstale":"2024-04-06T13:49:54.711166+0000","last_undegraded":"2024-04-06T13:49:54.711166+0000","last_fullsized":"2024-04-06T13:49:54.711166+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:55:32.313240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.35","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698460+0000","last_change":"2024-04-06T13:49:53.719282+0000","last_active":"2024-04-06T13:49:54.698460+0000","last_peered":"2024-04-06T13:49:54.698460+0000","last_clean":"2024-04-06T13:49:54.698460+0000","last_became_active":"2024-04-06T13:49:53.719046+0000","last_became_peered":"2024-04-06T13:49:53.719046+0000","last_unstale":"2024-04-06T13:49:54.698460+0000","last_undegraded":"2024-04-06T13:49:54.698460+0000","last_fullsized":"2024-04-06T13:49:54.698460+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:31:26.745492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.34","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.711307+0000","last_change":"2024-04-06T13:49:53.704657+0000","last_active":"2024-04-06T13:49:54.711307+0000","last_peered":"2024-04-06T13:49:54.711307+0000","last_clean":"2024-04-06T13:49:54.711307+0000","last_became_active":"2024-04-06T13:49:53.704287+0000","last_became_peered":"2024-04-06T13:49:53.704287+0000","last_unstale":"2024-04-06T13:49:54.711307+0000","last_undegraded":"2024-04-06T13:49:54.711307+0000","last_fullsized":"2024-04-06T13:49:54.711307+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:11:11.857561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.33","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710606+0000","last_change":"2024-04-06T13:49:53.709949+0000","last_active":"2024-04-06T13:49:54.710606+0000","last_peered":"2024-04-06T13:49:54.710606+0000","last_clean":"2024-04-06T13:49:54.710606+0000","last_became_active":"2024-04-06T13:49:53.709511+0000","last_became_peered":"2024-04-06T13:49:53.709511+0000","last_unstale":"2024-04-06T13:49:54.710606+0000","last_undegraded":"2024-04-06T13:49:54.710606+0000","last_fullsized":"2024-04-06T13:49:54.710606+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:48:15.813195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.32","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697114+0000","last_change":"2024-04-06T13:49:53.708002+0000","last_active":"2024-04-06T13:49:54.697114+0000","last_peered":"2024-04-06T13:49:54.697114+0000","last_clean":"2024-04-06T13:49:54.697114+0000","last_became_active":"2024-04-06T13:49:53.707771+0000","last_became_peered":"2024-04-06T13:49:53.707771+0000","last_unstale":"2024-04-06T13:49:54.697114+0000","last_undegraded":"2024-04-06T13:49:54.697114+0000","last_fullsized":"2024-04-06T13:49:54.697114+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:06.099346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.31","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697213+0000","last_change":"2024-04-06T13:49:53.710656+0000","last_active":"2024-04-06T13:49:54.697213+0000","last_peered":"2024-04-06T13:49:54.697213+0000","last_clean":"2024-04-06T13:49:54.697213+0000","last_became_active":"2024-04-06T13:49:53.710361+0000","last_became_peered":"2024-04-06T13:49:53.710361+0000","last_unstale":"2024-04-06T13:49:54.697213+0000","last_undegraded":"2024-04-06T13:49:54.697213+0000","last_fullsized":"2024-04-06T13:49:54.697213+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:45:06.633037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.30","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.711022+0000","last_change":"2024-04-06T13:49:53.720546+0000","last_active":"2024-04-06T13:49:54.711022+0000","last_peered":"2024-04-06T13:49:54.711022+0000","last_clean":"2024-04-06T13:49:54.711022+0000","last_became_active":"2024-04-06T13:49:53.720299+0000","last_became_peered":"2024-04-06T13:49:53.720299+0000","last_unstale":"2024-04-06T13:49:54.711022+0000","last_undegraded":"2024-04-06T13:49:54.711022+0000","last_fullsized":"2024-04-06T13:49:54.711022+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:09:37.214050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.2f","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698529+0000","last_change":"2024-04-06T13:49:53.712193+0000","last_active":"2024-04-06T13:49:54.698529+0000","last_peered":"2024-04-06T13:49:54.698529+0000","last_clean":"2024-04-06T13:49:54.698529+0000","last_became_active":"2024-04-06T13:49:53.711893+0000","last_became_peered":"2024-04-06T13:49:53.711893+0000","last_unstale":"2024-04-06T13:49:54.698529+0000","last_undegraded":"2024-04-06T13:49:54.698529+0000","last_fullsized":"2024-04-06T13:49:54.698529+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:31:34.206103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.2e","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699040+0000","last_change":"2024-04-06T13:49:53.711062+0000","last_active":"2024-04-06T13:49:54.699040+0000","last_peered":"2024-04-06T13:49:54.699040+0000","last_clean":"2024-04-06T13:49:54.699040+0000","last_became_active":"2024-04-06T13:49:53.710770+0000","last_became_peered":"2024-04-06T13:49:53.710770+0000","last_unstale":"2024-04-06T13:49:54.699040+0000","last_undegraded":"2024-04-06T13:49:54.699040+0000","last_fullsized":"2024-04-06T13:49:54.699040+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:13.245715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2d","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710061+0000","last_change":"2024-04-06T13:49:53.713054+0000","last_active":"2024-04-06T13:49:54.710061+0000","last_peered":"2024-04-06T13:49:54.710061+0000","last_clean":"2024-04-06T13:49:54.710061+0000","last_became_active":"2024-04-06T13:49:53.712502+0000","last_became_peered":"2024-04-06T13:49:53.712502+0000","last_unstale":"2024-04-06T13:49:54.710061+0000","last_undegraded":"2024-04-06T13:49:54.710061+0000","last_fullsized":"2024-04-06T13:49:54.710061+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:29:11.366508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.2c","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699287+0000","last_change":"2024-04-06T13:49:53.717998+0000","last_active":"2024-04-06T13:49:54.699287+0000","last_peered":"2024-04-06T13:49:54.699287+0000","last_clean":"2024-04-06T13:49:54.699287+0000","last_became_active":"2024-04-06T13:49:53.717305+0000","last_became_peered":"2024-04-06T13:49:53.717305+0000","last_unstale":"2024-04-06T13:49:54.699287+0000","last_undegraded":"2024-04-06T13:49:54.699287+0000","last_fullsized":"2024-04-06T13:49:54.699287+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:55:00.463548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2b","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697400+0000","last_change":"2024-04-06T13:49:53.711352+0000","last_active":"2024-04-06T13:49:54.697400+0000","last_peered":"2024-04-06T13:49:54.697400+0000","last_clean":"2024-04-06T13:49:54.697400+0000","last_became_active":"2024-04-06T13:49:53.711110+0000","last_became_peered":"2024-04-06T13:49:53.711110+0000","last_unstale":"2024-04-06T13:49:54.697400+0000","last_undegraded":"2024-04-06T13:49:54.697400+0000","last_fullsized":"2024-04-06T13:49:54.697400+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:31:25.186211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.2a","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699145+0000","last_change":"2024-04-06T13:49:53.718419+0000","last_active":"2024-04-06T13:49:54.699145+0000","last_peered":"2024-04-06T13:49:54.699145+0000","last_clean":"2024-04-06T13:49:54.699145+0000","last_became_active":"2024-04-06T13:49:53.713389+0000","last_became_peered":"2024-04-06T13:49:53.713389+0000","last_unstale":"2024-04-06T13:49:54.699145+0000","last_undegraded":"2024-04-06T13:49:54.699145+0000","last_fullsized":"2024-04-06T13:49:54.699145+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:13:36.768561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.29","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698931+0000","last_change":"2024-04-06T13:49:53.722123+0000","last_active":"2024-04-06T13:49:54.698931+0000","last_peered":"2024-04-06T13:49:54.698931+0000","last_clean":"2024-04-06T13:49:54.698931+0000","last_became_active":"2024-04-06T13:49:53.719359+0000","last_became_peered":"2024-04-06T13:49:53.719359+0000","last_unstale":"2024-04-06T13:49:54.698931+0000","last_undegraded":"2024-04-06T13:49:54.698931+0000","last_fullsized":"2024-04-06T13:49:54.698931+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:15:49.763249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.28","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.709578+0000","last_change":"2024-04-06T13:49:53.707950+0000","last_active":"2024-04-06T13:49:54.709578+0000","last_peered":"2024-04-06T13:49:54.709578+0000","last_clean":"2024-04-06T13:49:54.709578+0000","last_became_active":"2024-04-06T13:49:53.707514+0000","last_became_peered":"2024-04-06T13:49:53.707514+0000","last_unstale":"2024-04-06T13:49:54.709578+0000","last_undegraded":"2024-04-06T13:49:54.709578+0000","last_fullsized":"2024-04-06T13:49:54.709578+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:05:17.703470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.26","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698490+0000","last_change":"2024-04-06T13:49:53.712113+0000","last_active":"2024-04-06T13:49:54.698490+0000","last_peered":"2024-04-06T13:49:54.698490+0000","last_clean":"2024-04-06T13:49:54.698490+0000","last_became_active":"2024-04-06T13:49:53.711852+0000","last_became_peered":"2024-04-06T13:49:53.711852+0000","last_unstale":"2024-04-06T13:49:54.698490+0000","last_undegraded":"2024-04-06T13:49:54.698490+0000","last_fullsized":"2024-04-06T13:49:54.698490+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:12:58.661980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.25","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699199+0000","last_change":"2024-04-06T13:49:53.720979+0000","last_active":"2024-04-06T13:49:54.699199+0000","last_peered":"2024-04-06T13:49:54.699199+0000","last_clean":"2024-04-06T13:49:54.699199+0000","last_became_active":"2024-04-06T13:49:53.720430+0000","last_became_peered":"2024-04-06T13:49:53.720430+0000","last_unstale":"2024-04-06T13:49:54.699199+0000","last_undegraded":"2024-04-06T13:49:54.699199+0000","last_fullsized":"2024-04-06T13:49:54.699199+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:34:16.805997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.24","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698367+0000","last_change":"2024-04-06T13:49:53.715901+0000","last_active":"2024-04-06T13:49:54.698367+0000","last_peered":"2024-04-06T13:49:54.698367+0000","last_clean":"2024-04-06T13:49:54.698367+0000","last_became_active":"2024-04-06T13:49:53.715646+0000","last_became_peered":"2024-04-06T13:49:53.715646+0000","last_unstale":"2024-04-06T13:49:54.698367+0000","last_undegraded":"2024-04-06T13:49:54.698367+0000","last_fullsized":"2024-04-06T13:49:54.698367+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:03:35.006421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.23","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710475+0000","last_change":"2024-04-06T13:49:53.715109+0000","last_active":"2024-04-06T13:49:54.710475+0000","last_peered":"2024-04-06T13:49:54.710475+0000","last_clean":"2024-04-06T13:49:54.710475+0000","last_became_active":"2024-04-06T13:49:53.714657+0000","last_became_peered":"2024-04-06T13:49:53.714657+0000","last_unstale":"2024-04-06T13:49:54.710475+0000","last_undegraded":"2024-04-06T13:49:54.710475+0000","last_fullsized":"2024-04-06T13:49:54.710475+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:37:06.208036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.22","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697219+0000","last_change":"2024-04-06T13:49:53.697123+0000","last_active":"2024-04-06T13:49:54.697219+0000","last_peered":"2024-04-06T13:49:54.697219+0000","last_clean":"2024-04-06T13:49:54.697219+0000","last_became_active":"2024-04-06T13:49:53.696854+0000","last_became_peered":"2024-04-06T13:49:53.696854+0000","last_unstale":"2024-04-06T13:49:54.697219+0000","last_undegraded":"2024-04-06T13:49:54.697219+0000","last_fullsized":"2024-04-06T13:49:54.697219+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:29:59.193762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.21","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710895+0000","last_change":"2024-04-06T13:49:53.714011+0000","last_active":"2024-04-06T13:49:54.710895+0000","last_peered":"2024-04-06T13:49:54.710895+0000","last_clean":"2024-04-06T13:49:54.710895+0000","last_became_active":"2024-04-06T13:49:53.713717+0000","last_became_peered":"2024-04-06T13:49:53.713717+0000","last_unstale":"2024-04-06T13:49:54.710895+0000","last_undegraded":"2024-04-06T13:49:54.710895+0000","last_fullsized":"2024-04-06T13:49:54.710895+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:52:29.154423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.20","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698745+0000","last_change":"2024-04-06T13:49:53.722141+0000","last_active":"2024-04-06T13:49:54.698745+0000","last_peered":"2024-04-06T13:49:54.698745+0000","last_clean":"2024-04-06T13:49:54.698745+0000","last_became_active":"2024-04-06T13:49:53.720421+0000","last_became_peered":"2024-04-06T13:49:53.720421+0000","last_unstale":"2024-04-06T13:49:54.698745+0000","last_undegraded":"2024-04-06T13:49:54.698745+0000","last_fullsized":"2024-04-06T13:49:54.698745+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:52:58.376388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1c","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699292+0000","last_change":"2024-04-06T13:49:53.718638+0000","last_active":"2024-04-06T13:49:54.699292+0000","last_peered":"2024-04-06T13:49:54.699292+0000","last_clean":"2024-04-06T13:49:54.699292+0000","last_became_active":"2024-04-06T13:49:53.713132+0000","last_became_peered":"2024-04-06T13:49:53.713132+0000","last_unstale":"2024-04-06T13:49:54.699292+0000","last_undegraded":"2024-04-06T13:49:54.699292+0000","last_fullsized":"2024-04-06T13:49:54.699292+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:28:04.138418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1f","version":"282'4","reported_seq":15,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.711169+0000","last_change":"2024-04-06T13:49:53.291574+0000","last_active":"2024-04-06T13:49:54.711169+0000","last_peered":"2024-04-06T13:49:54.711169+0000","last_clean":"2024-04-06T13:49:54.711169+0000","last_became_active":"2024-04-06T13:49:53.291311+0000","last_became_peered":"2024-04-06T13:49:53.291311+0000","last_unstale":"2024-04-06T13:49:54.711169+0000","last_undegraded":"2024-04-06T13:49:54.711169+0000","last_fullsized":"2024-04-06T13:49:54.711169+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:31:58.213975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.1d","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698387+0000","last_change":"2024-04-06T13:49:53.719249+0000","last_active":"2024-04-06T13:49:54.698387+0000","last_peered":"2024-04-06T13:49:54.698387+0000","last_clean":"2024-04-06T13:49:54.698387+0000","last_became_active":"2024-04-06T13:49:53.719020+0000","last_became_peered":"2024-04-06T13:49:53.719020+0000","last_unstale":"2024-04-06T13:49:54.698387+0000","last_undegraded":"2024-04-06T13:49:54.698387+0000","last_fullsized":"2024-04-06T13:49:54.698387+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:30:18.780722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1e","version":"282'1","reported_seq":12,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699138+0000","last_change":"2024-04-06T13:49:53.294979+0000","last_active":"2024-04-06T13:49:54.699138+0000","last_peered":"2024-04-06T13:49:54.699138+0000","last_clean":"2024-04-06T13:49:54.699138+0000","last_became_active":"2024-04-06T13:49:53.294713+0000","last_became_peered":"2024-04-06T13:49:53.294713+0000","last_unstale":"2024-04-06T13:49:54.699138+0000","last_undegraded":"2024-04-06T13:49:54.699138+0000","last_fullsized":"2024-04-06T13:49:54.699138+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:04:47.588644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1e","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697694+0000","last_change":"2024-04-06T13:49:53.714295+0000","last_active":"2024-04-06T13:49:54.697694+0000","last_peered":"2024-04-06T13:49:54.697694+0000","last_clean":"2024-04-06T13:49:54.697694+0000","last_became_active":"2024-04-06T13:49:53.714037+0000","last_became_peered":"2024-04-06T13:49:53.714037+0000","last_unstale":"2024-04-06T13:49:54.697694+0000","last_undegraded":"2024-04-06T13:49:54.697694+0000","last_fullsized":"2024-04-06T13:49:54.697694+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:15:21.536648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"282'2","reported_seq":13,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710143+0000","last_change":"2024-04-06T13:49:53.292744+0000","last_active":"2024-04-06T13:49:54.710143+0000","last_peered":"2024-04-06T13:49:54.710143+0000","last_clean":"2024-04-06T13:49:54.710143+0000","last_became_active":"2024-04-06T13:49:53.292484+0000","last_became_peered":"2024-04-06T13:49:53.292484+0000","last_unstale":"2024-04-06T13:49:54.710143+0000","last_undegraded":"2024-04-06T13:49:54.710143+0000","last_fullsized":"2024-04-06T13:49:54.710143+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:29:50.333793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1f","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697806+0000","last_change":"2024-04-06T13:49:53.698278+0000","last_active":"2024-04-06T13:49:54.697806+0000","last_peered":"2024-04-06T13:49:54.697806+0000","last_clean":"2024-04-06T13:49:54.697806+0000","last_became_active":"2024-04-06T13:49:53.698038+0000","last_became_peered":"2024-04-06T13:49:53.698038+0000","last_unstale":"2024-04-06T13:49:54.697806+0000","last_undegraded":"2024-04-06T13:49:54.697806+0000","last_fullsized":"2024-04-06T13:49:54.697806+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:16:22.805098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"282'2","reported_seq":13,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697436+0000","last_change":"2024-04-06T13:49:53.296089+0000","last_active":"2024-04-06T13:49:54.697436+0000","last_peered":"2024-04-06T13:49:54.697436+0000","last_clean":"2024-04-06T13:49:54.697436+0000","last_became_active":"2024-04-06T13:49:53.295881+0000","last_became_peered":"2024-04-06T13:49:53.295881+0000","last_unstale":"2024-04-06T13:49:54.697436+0000","last_undegraded":"2024-04-06T13:49:54.697436+0000","last_fullsized":"2024-04-06T13:49:54.697436+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:00:12.240799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.18","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.709978+0000","last_change":"2024-04-06T13:49:53.719108+0000","last_active":"2024-04-06T13:49:54.709978+0000","last_peered":"2024-04-06T13:49:54.709978+0000","last_clean":"2024-04-06T13:49:54.709978+0000","last_became_active":"2024-04-06T13:49:53.718880+0000","last_became_peered":"2024-04-06T13:49:53.718880+0000","last_unstale":"2024-04-06T13:49:54.709978+0000","last_undegraded":"2024-04-06T13:49:54.709978+0000","last_fullsized":"2024-04-06T13:49:54.709978+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:00:49.668717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1b","version":"282'1","reported_seq":12,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698177+0000","last_change":"2024-04-06T13:49:53.291374+0000","last_active":"2024-04-06T13:49:54.698177+0000","last_peered":"2024-04-06T13:49:54.698177+0000","last_clean":"2024-04-06T13:49:54.698177+0000","last_became_active":"2024-04-06T13:49:53.291086+0000","last_became_peered":"2024-04-06T13:49:53.291086+0000","last_unstale":"2024-04-06T13:49:54.698177+0000","last_undegraded":"2024-04-06T13:49:54.698177+0000","last_fullsized":"2024-04-06T13:49:54.698177+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:30:27.012134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.19","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698121+0000","last_change":"2024-04-06T13:49:53.709992+0000","last_active":"2024-04-06T13:49:54.698121+0000","last_peered":"2024-04-06T13:49:54.698121+0000","last_clean":"2024-04-06T13:49:54.698121+0000","last_became_active":"2024-04-06T13:49:53.709726+0000","last_became_peered":"2024-04-06T13:49:53.709726+0000","last_unstale":"2024-04-06T13:49:54.698121+0000","last_undegraded":"2024-04-06T13:49:54.698121+0000","last_fullsized":"2024-04-06T13:49:54.698121+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:18:02.185633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1a","version":"282'1","reported_seq":12,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.709866+0000","last_change":"2024-04-06T13:49:53.286094+0000","last_active":"2024-04-06T13:49:54.709866+0000","last_peered":"2024-04-06T13:49:54.709866+0000","last_clean":"2024-04-06T13:49:54.709866+0000","last_became_active":"2024-04-06T13:49:53.285843+0000","last_became_peered":"2024-04-06T13:49:53.285843+0000","last_unstale":"2024-04-06T13:49:54.709866+0000","last_undegraded":"2024-04-06T13:49:54.709866+0000","last_fullsized":"2024-04-06T13:49:54.709866+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:56:43.852373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1a","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698851+0000","last_change":"2024-04-06T13:49:53.721943+0000","last_active":"2024-04-06T13:49:54.698851+0000","last_peered":"2024-04-06T13:49:54.698851+0000","last_clean":"2024-04-06T13:49:54.698851+0000","last_became_active":"2024-04-06T13:49:53.720565+0000","last_became_peered":"2024-04-06T13:49:53.720565+0000","last_unstale":"2024-04-06T13:49:54.698851+0000","last_undegraded":"2024-04-06T13:49:54.698851+0000","last_fullsized":"2024-04-06T13:49:54.698851+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:38:33.928853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697391+0000","last_change":"2024-04-06T13:49:53.295030+0000","last_active":"2024-04-06T13:49:54.697391+0000","last_peered":"2024-04-06T13:49:54.697391+0000","last_clean":"2024-04-06T13:49:54.697391+0000","last_became_active":"2024-04-06T13:49:53.294793+0000","last_became_peered":"2024-04-06T13:49:53.294793+0000","last_unstale":"2024-04-06T13:49:54.697391+0000","last_undegraded":"2024-04-06T13:49:54.697391+0000","last_fullsized":"2024-04-06T13:49:54.697391+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:25:32.159801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.1b","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697231+0000","last_change":"2024-04-06T13:49:53.716089+0000","last_active":"2024-04-06T13:49:54.697231+0000","last_peered":"2024-04-06T13:49:54.697231+0000","last_clean":"2024-04-06T13:49:54.697231+0000","last_became_active":"2024-04-06T13:49:53.715842+0000","last_became_peered":"2024-04-06T13:49:53.715842+0000","last_unstale":"2024-04-06T13:49:54.697231+0000","last_undegraded":"2024-04-06T13:49:54.697231+0000","last_fullsized":"2024-04-06T13:49:54.697231+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:30:28.017050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.709884+0000","last_change":"2024-04-06T13:49:53.293116+0000","last_active":"2024-04-06T13:49:54.709884+0000","last_peered":"2024-04-06T13:49:54.709884+0000","last_clean":"2024-04-06T13:49:54.709884+0000","last_became_active":"2024-04-06T13:49:53.292825+0000","last_became_peered":"2024-04-06T13:49:53.292825+0000","last_unstale":"2024-04-06T13:49:54.709884+0000","last_undegraded":"2024-04-06T13:49:54.709884+0000","last_fullsized":"2024-04-06T13:49:54.709884+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:16:14.577622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.14","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699402+0000","last_change":"2024-04-06T13:49:53.717594+0000","last_active":"2024-04-06T13:49:54.699402+0000","last_peered":"2024-04-06T13:49:54.699402+0000","last_clean":"2024-04-06T13:49:54.699402+0000","last_became_active":"2024-04-06T13:49:53.713401+0000","last_became_peered":"2024-04-06T13:49:53.713401+0000","last_unstale":"2024-04-06T13:49:54.699402+0000","last_undegraded":"2024-04-06T13:49:54.699402+0000","last_fullsized":"2024-04-06T13:49:54.699402+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:22:41.350836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.711230+0000","last_change":"2024-04-06T13:49:53.295170+0000","last_active":"2024-04-06T13:49:54.711230+0000","last_peered":"2024-04-06T13:49:54.711230+0000","last_clean":"2024-04-06T13:49:54.711230+0000","last_became_active":"2024-04-06T13:49:53.294914+0000","last_became_peered":"2024-04-06T13:49:53.294914+0000","last_unstale":"2024-04-06T13:49:54.711230+0000","last_undegraded":"2024-04-06T13:49:54.711230+0000","last_fullsized":"2024-04-06T13:49:54.711230+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:14:15.037776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.15","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710920+0000","last_change":"2024-04-06T13:49:53.704942+0000","last_active":"2024-04-06T13:49:54.710920+0000","last_peered":"2024-04-06T13:49:54.710920+0000","last_clean":"2024-04-06T13:49:54.710920+0000","last_became_active":"2024-04-06T13:49:53.704679+0000","last_became_peered":"2024-04-06T13:49:53.704679+0000","last_unstale":"2024-04-06T13:49:54.710920+0000","last_undegraded":"2024-04-06T13:49:54.710920+0000","last_fullsized":"2024-04-06T13:49:54.710920+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:48:15.813195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697779+0000","last_change":"2024-04-06T13:49:53.295042+0000","last_active":"2024-04-06T13:49:54.697779+0000","last_peered":"2024-04-06T13:49:54.697779+0000","last_clean":"2024-04-06T13:49:54.697779+0000","last_became_active":"2024-04-06T13:49:53.294638+0000","last_became_peered":"2024-04-06T13:49:53.294638+0000","last_unstale":"2024-04-06T13:49:54.697779+0000","last_undegraded":"2024-04-06T13:49:54.697779+0000","last_fullsized":"2024-04-06T13:49:54.697779+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:10:02.994827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.16","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697601+0000","last_change":"2024-04-06T13:49:53.713788+0000","last_active":"2024-04-06T13:49:54.697601+0000","last_peered":"2024-04-06T13:49:54.697601+0000","last_clean":"2024-04-06T13:49:54.697601+0000","last_became_active":"2024-04-06T13:49:53.713437+0000","last_became_peered":"2024-04-06T13:49:53.713437+0000","last_unstale":"2024-04-06T13:49:54.697601+0000","last_undegraded":"2024-04-06T13:49:54.697601+0000","last_fullsized":"2024-04-06T13:49:54.697601+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:08:39.564204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.15","version":"282'3","reported_seq":14,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697401+0000","last_change":"2024-04-06T13:49:53.297806+0000","last_active":"2024-04-06T13:49:54.697401+0000","last_peered":"2024-04-06T13:49:54.697401+0000","last_clean":"2024-04-06T13:49:54.697401+0000","last_became_active":"2024-04-06T13:49:53.297518+0000","last_became_peered":"2024-04-06T13:49:53.297518+0000","last_unstale":"2024-04-06T13:49:54.697401+0000","last_undegraded":"2024-04-06T13:49:54.697401+0000","last_fullsized":"2024-04-06T13:49:54.697401+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:13:35.763646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.17","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699300+0000","last_change":"2024-04-06T13:49:53.712762+0000","last_active":"2024-04-06T13:49:54.699300+0000","last_peered":"2024-04-06T13:49:54.699300+0000","last_clean":"2024-04-06T13:49:54.699300+0000","last_became_active":"2024-04-06T13:49:53.711489+0000","last_became_peered":"2024-04-06T13:49:53.711489+0000","last_unstale":"2024-04-06T13:49:54.699300+0000","last_undegraded":"2024-04-06T13:49:54.699300+0000","last_fullsized":"2024-04-06T13:49:54.699300+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:25:33.164716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.14","version":"282'6","reported_seq":17,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710191+0000","last_change":"2024-04-06T13:49:53.293553+0000","last_active":"2024-04-06T13:49:54.710191+0000","last_peered":"2024-04-06T13:49:54.710191+0000","last_clean":"2024-04-06T13:49:54.710191+0000","last_became_active":"2024-04-06T13:49:53.293297+0000","last_became_peered":"2024-04-06T13:49:53.293297+0000","last_unstale":"2024-04-06T13:49:54.710191+0000","last_undegraded":"2024-04-06T13:49:54.710191+0000","last_fullsized":"2024-04-06T13:49:54.710191+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:25.980159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.10","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698177+0000","last_change":"2024-04-06T13:49:53.709777+0000","last_active":"2024-04-06T13:49:54.698177+0000","last_peered":"2024-04-06T13:49:54.698177+0000","last_clean":"2024-04-06T13:49:54.698177+0000","last_became_active":"2024-04-06T13:49:53.709511+0000","last_became_peered":"2024-04-06T13:49:53.709511+0000","last_unstale":"2024-04-06T13:49:54.698177+0000","last_undegraded":"2024-04-06T13:49:54.698177+0000","last_fullsized":"2024-04-06T13:49:54.698177+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:46:38.182093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.13","version":"282'2","reported_seq":13,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710924+0000","last_change":"2024-04-06T13:49:53.286397+0000","last_active":"2024-04-06T13:49:54.710924+0000","last_peered":"2024-04-06T13:49:54.710924+0000","last_clean":"2024-04-06T13:49:54.710924+0000","last_became_active":"2024-04-06T13:49:53.286064+0000","last_became_peered":"2024-04-06T13:49:53.286064+0000","last_unstale":"2024-04-06T13:49:54.710924+0000","last_undegraded":"2024-04-06T13:49:54.710924+0000","last_fullsized":"2024-04-06T13:49:54.710924+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:28:12.467204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.11","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710343+0000","last_change":"2024-04-06T13:49:53.715124+0000","last_active":"2024-04-06T13:49:54.710343+0000","last_peered":"2024-04-06T13:49:54.710343+0000","last_clean":"2024-04-06T13:49:54.710343+0000","last_became_active":"2024-04-06T13:49:53.714882+0000","last_became_peered":"2024-04-06T13:49:53.714882+0000","last_unstale":"2024-04-06T13:49:54.710343+0000","last_undegraded":"2024-04-06T13:49:54.710343+0000","last_fullsized":"2024-04-06T13:49:54.710343+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:09:37.214050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710413+0000","last_change":"2024-04-06T13:49:53.299524+0000","last_active":"2024-04-06T13:49:54.710413+0000","last_peered":"2024-04-06T13:49:54.710413+0000","last_clean":"2024-04-06T13:49:54.710413+0000","last_became_active":"2024-04-06T13:49:53.299276+0000","last_became_peered":"2024-04-06T13:49:53.299276+0000","last_unstale":"2024-04-06T13:49:54.710413+0000","last_undegraded":"2024-04-06T13:49:54.710413+0000","last_fullsized":"2024-04-06T13:49:54.710413+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:57:56.001057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.12","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698978+0000","last_change":"2024-04-06T13:49:53.718373+0000","last_active":"2024-04-06T13:49:54.698978+0000","last_peered":"2024-04-06T13:49:54.698978+0000","last_clean":"2024-04-06T13:49:54.698978+0000","last_became_active":"2024-04-06T13:49:53.713088+0000","last_became_peered":"2024-04-06T13:49:53.713088+0000","last_unstale":"2024-04-06T13:49:54.698978+0000","last_undegraded":"2024-04-06T13:49:54.698978+0000","last_fullsized":"2024-04-06T13:49:54.698978+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:24:58.981619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710543+0000","last_change":"2024-04-06T13:49:53.300253+0000","last_active":"2024-04-06T13:49:54.710543+0000","last_peered":"2024-04-06T13:49:54.710543+0000","last_clean":"2024-04-06T13:49:54.710543+0000","last_became_active":"2024-04-06T13:49:53.300051+0000","last_became_peered":"2024-04-06T13:49:53.300051+0000","last_unstale":"2024-04-06T13:49:54.710543+0000","last_undegraded":"2024-04-06T13:49:54.710543+0000","last_fullsized":"2024-04-06T13:49:54.710543+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:06:27.253841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.13","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710776+0000","last_change":"2024-04-06T13:49:53.715115+0000","last_active":"2024-04-06T13:49:54.710776+0000","last_peered":"2024-04-06T13:49:54.710776+0000","last_clean":"2024-04-06T13:49:54.710776+0000","last_became_active":"2024-04-06T13:49:53.714823+0000","last_became_peered":"2024-04-06T13:49:53.714823+0000","last_unstale":"2024-04-06T13:49:54.710776+0000","last_undegraded":"2024-04-06T13:49:54.710776+0000","last_fullsized":"2024-04-06T13:49:54.710776+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:35:57.994706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.10","version":"282'8","reported_seq":19,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698855+0000","last_change":"2024-04-06T13:49:53.294367+0000","last_active":"2024-04-06T13:49:54.698855+0000","last_peered":"2024-04-06T13:49:54.698855+0000","last_clean":"2024-04-06T13:49:54.698855+0000","last_became_active":"2024-04-06T13:49:53.294109+0000","last_became_peered":"2024-04-06T13:49:53.294109+0000","last_unstale":"2024-04-06T13:49:54.698855+0000","last_undegraded":"2024-04-06T13:49:54.698855+0000","last_fullsized":"2024-04-06T13:49:54.698855+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:43:39.597248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.c","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710276+0000","last_change":"2024-04-06T13:49:53.699165+0000","last_active":"2024-04-06T13:49:54.710276+0000","last_peered":"2024-04-06T13:49:54.710276+0000","last_clean":"2024-04-06T13:49:54.710276+0000","last_became_active":"2024-04-06T13:49:53.698914+0000","last_became_peered":"2024-04-06T13:49:53.698914+0000","last_unstale":"2024-04-06T13:49:54.710276+0000","last_undegraded":"2024-04-06T13:49:54.710276+0000","last_fullsized":"2024-04-06T13:49:54.710276+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:55:32.313240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.f","version":"282'1","reported_seq":12,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699372+0000","last_change":"2024-04-06T13:49:53.295113+0000","last_active":"2024-04-06T13:49:54.699372+0000","last_peered":"2024-04-06T13:49:54.699372+0000","last_clean":"2024-04-06T13:49:54.699372+0000","last_became_active":"2024-04-06T13:49:53.294809+0000","last_became_peered":"2024-04-06T13:49:53.294809+0000","last_unstale":"2024-04-06T13:49:54.699372+0000","last_undegraded":"2024-04-06T13:49:54.699372+0000","last_fullsized":"2024-04-06T13:49:54.699372+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:53:13.312600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.d","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710362+0000","last_change":"2024-04-06T13:49:53.710731+0000","last_active":"2024-04-06T13:49:54.710362+0000","last_peered":"2024-04-06T13:49:54.710362+0000","last_clean":"2024-04-06T13:49:54.710362+0000","last_became_active":"2024-04-06T13:49:53.710476+0000","last_became_peered":"2024-04-06T13:49:53.710476+0000","last_unstale":"2024-04-06T13:49:54.710362+0000","last_undegraded":"2024-04-06T13:49:54.710362+0000","last_fullsized":"2024-04-06T13:49:54.710362+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:41:27.888300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.e","version":"282'2","reported_seq":13,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710580+0000","last_change":"2024-04-06T13:49:53.290628+0000","last_active":"2024-04-06T13:49:54.710580+0000","last_peered":"2024-04-06T13:49:54.710580+0000","last_clean":"2024-04-06T13:49:54.710580+0000","last_became_active":"2024-04-06T13:49:53.290367+0000","last_became_peered":"2024-04-06T13:49:53.290367+0000","last_unstale":"2024-04-06T13:49:54.710580+0000","last_undegraded":"2024-04-06T13:49:54.710580+0000","last_fullsized":"2024-04-06T13:49:54.710580+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T00:31:58.213975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.e","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.711271+0000","last_change":"2024-04-06T13:49:53.706895+0000","last_active":"2024-04-06T13:49:54.711271+0000","last_peered":"2024-04-06T13:49:54.711271+0000","last_clean":"2024-04-06T13:49:54.711271+0000","last_became_active":"2024-04-06T13:49:53.706561+0000","last_became_peered":"2024-04-06T13:49:53.706561+0000","last_unstale":"2024-04-06T13:49:54.711271+0000","last_undegraded":"2024-04-06T13:49:54.711271+0000","last_fullsized":"2024-04-06T13:49:54.711271+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:41:27.888300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.d","version":"282'1","reported_seq":12,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710459+0000","last_change":"2024-04-06T13:49:53.293292+0000","last_active":"2024-04-06T13:49:54.710459+0000","last_peered":"2024-04-06T13:49:54.710459+0000","last_clean":"2024-04-06T13:49:54.710459+0000","last_became_active":"2024-04-06T13:49:53.292868+0000","last_became_peered":"2024-04-06T13:49:53.292868+0000","last_unstale":"2024-04-06T13:49:54.710459+0000","last_undegraded":"2024-04-06T13:49:54.710459+0000","last_fullsized":"2024-04-06T13:49:54.710459+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:36:09.149670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.f","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710279+0000","last_change":"2024-04-06T13:49:53.707093+0000","last_active":"2024-04-06T13:49:54.710279+0000","last_peered":"2024-04-06T13:49:54.710279+0000","last_clean":"2024-04-06T13:49:54.710279+0000","last_became_active":"2024-04-06T13:49:53.706930+0000","last_became_peered":"2024-04-06T13:49:53.706930+0000","last_unstale":"2024-04-06T13:49:54.710279+0000","last_undegraded":"2024-04-06T13:49:54.710279+0000","last_fullsized":"2024-04-06T13:49:54.710279+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:56:48.466109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697473+0000","last_change":"2024-04-06T13:49:53.297510+0000","last_active":"2024-04-06T13:49:54.697473+0000","last_peered":"2024-04-06T13:49:54.697473+0000","last_clean":"2024-04-06T13:49:54.697473+0000","last_became_active":"2024-04-06T13:49:53.296479+0000","last_became_peered":"2024-04-06T13:49:53.296479+0000","last_unstale":"2024-04-06T13:49:54.697473+0000","last_undegraded":"2024-04-06T13:49:54.697473+0000","last_fullsized":"2024-04-06T13:49:54.697473+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:38:32.923938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.8","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699346+0000","last_change":"2024-04-06T13:49:53.722370+0000","last_active":"2024-04-06T13:49:54.699346+0000","last_peered":"2024-04-06T13:49:54.699346+0000","last_clean":"2024-04-06T13:49:54.699346+0000","last_became_active":"2024-04-06T13:49:53.720753+0000","last_became_peered":"2024-04-06T13:49:53.720753+0000","last_unstale":"2024-04-06T13:49:54.699346+0000","last_undegraded":"2024-04-06T13:49:54.699346+0000","last_fullsized":"2024-04-06T13:49:54.699346+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:13:24.517768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.b","version":"282'2","reported_seq":13,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698258+0000","last_change":"2024-04-06T13:49:53.291683+0000","last_active":"2024-04-06T13:49:54.698258+0000","last_peered":"2024-04-06T13:49:54.698258+0000","last_clean":"2024-04-06T13:49:54.698258+0000","last_became_active":"2024-04-06T13:49:53.291409+0000","last_became_peered":"2024-04-06T13:49:53.291409+0000","last_unstale":"2024-04-06T13:49:54.698258+0000","last_undegraded":"2024-04-06T13:49:54.698258+0000","last_fullsized":"2024-04-06T13:49:54.698258+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:45:05.628121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.9","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698221+0000","last_change":"2024-04-06T13:49:53.709932+0000","last_active":"2024-04-06T13:49:54.698221+0000","last_peered":"2024-04-06T13:49:54.698221+0000","last_clean":"2024-04-06T13:49:54.698221+0000","last_became_active":"2024-04-06T13:49:53.709544+0000","last_became_peered":"2024-04-06T13:49:53.709544+0000","last_unstale":"2024-04-06T13:49:54.698221+0000","last_undegraded":"2024-04-06T13:49:54.698221+0000","last_fullsized":"2024-04-06T13:49:54.698221+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:51:45.375899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699134+0000","last_change":"2024-04-06T13:49:53.294508+0000","last_active":"2024-04-06T13:49:54.699134+0000","last_peered":"2024-04-06T13:49:54.699134+0000","last_clean":"2024-04-06T13:49:54.699134+0000","last_became_active":"2024-04-06T13:49:53.294243+0000","last_became_peered":"2024-04-06T13:49:53.294243+0000","last_unstale":"2024-04-06T13:49:54.699134+0000","last_undegraded":"2024-04-06T13:49:54.699134+0000","last_fullsized":"2024-04-06T13:49:54.699134+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:11:56.282961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.a","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710393+0000","last_change":"2024-04-06T13:49:53.700719+0000","last_active":"2024-04-06T13:49:54.710393+0000","last_peered":"2024-04-06T13:49:54.710393+0000","last_clean":"2024-04-06T13:49:54.710393+0000","last_became_active":"2024-04-06T13:49:53.700418+0000","last_became_peered":"2024-04-06T13:49:53.700418+0000","last_unstale":"2024-04-06T13:49:54.710393+0000","last_undegraded":"2024-04-06T13:49:54.710393+0000","last_fullsized":"2024-04-06T13:49:54.710393+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:52:26.703747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710467+0000","last_change":"2024-04-06T13:49:53.292197+0000","last_active":"2024-04-06T13:49:54.710467+0000","last_peered":"2024-04-06T13:49:54.710467+0000","last_clean":"2024-04-06T13:49:54.710467+0000","last_became_active":"2024-04-06T13:49:53.291949+0000","last_became_peered":"2024-04-06T13:49:53.291949+0000","last_unstale":"2024-04-06T13:49:54.710467+0000","last_undegraded":"2024-04-06T13:49:54.710467+0000","last_fullsized":"2024-04-06T13:49:54.710467+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:40:08.863359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.b","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710903+0000","last_change":"2024-04-06T13:49:53.708630+0000","last_active":"2024-04-06T13:49:54.710903+0000","last_peered":"2024-04-06T13:49:54.710903+0000","last_clean":"2024-04-06T13:49:54.710903+0000","last_became_active":"2024-04-06T13:49:53.708141+0000","last_became_peered":"2024-04-06T13:49:53.708141+0000","last_unstale":"2024-04-06T13:49:54.710903+0000","last_undegraded":"2024-04-06T13:49:54.710903+0000","last_fullsized":"2024-04-06T13:49:54.710903+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:52:26.703747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.709561+0000","last_change":"2024-04-06T13:49:53.286973+0000","last_active":"2024-04-06T13:49:54.709561+0000","last_peered":"2024-04-06T13:49:54.709561+0000","last_clean":"2024-04-06T13:49:54.709561+0000","last_became_active":"2024-04-06T13:49:53.286685+0000","last_became_peered":"2024-04-06T13:49:53.286685+0000","last_unstale":"2024-04-06T13:49:54.709561+0000","last_undegraded":"2024-04-06T13:49:54.709561+0000","last_fullsized":"2024-04-06T13:49:54.709561+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:59:10.089185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.4","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.699415+0000","last_change":"2024-04-06T13:49:53.717776+0000","last_active":"2024-04-06T13:49:54.699415+0000","last_peered":"2024-04-06T13:49:54.699415+0000","last_clean":"2024-04-06T13:49:54.699415+0000","last_became_active":"2024-04-06T13:49:53.713428+0000","last_became_peered":"2024-04-06T13:49:53.713428+0000","last_unstale":"2024-04-06T13:49:54.699415+0000","last_undegraded":"2024-04-06T13:49:54.699415+0000","last_fullsized":"2024-04-06T13:49:54.699415+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T23:10:03.999743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.7","version":"282'2","reported_seq":13,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710260+0000","last_change":"2024-04-06T13:49:53.291926+0000","last_active":"2024-04-06T13:49:54.710260+0000","last_peered":"2024-04-06T13:49:54.710260+0000","last_clean":"2024-04-06T13:49:54.710260+0000","last_became_active":"2024-04-06T13:49:53.291655+0000","last_became_peered":"2024-04-06T13:49:53.291655+0000","last_unstale":"2024-04-06T13:49:54.710260+0000","last_undegraded":"2024-04-06T13:49:54.710260+0000","last_fullsized":"2024-04-06T13:49:54.710260+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T01:28:30.863971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.5","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710630+0000","last_change":"2024-04-06T13:49:53.715836+0000","last_active":"2024-04-06T13:49:54.710630+0000","last_peered":"2024-04-06T13:49:54.710630+0000","last_clean":"2024-04-06T13:49:54.710630+0000","last_became_active":"2024-04-06T13:49:53.715586+0000","last_became_peered":"2024-04-06T13:49:53.715586+0000","last_unstale":"2024-04-06T13:49:54.710630+0000","last_undegraded":"2024-04-06T13:49:54.710630+0000","last_fullsized":"2024-04-06T13:49:54.710630+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:12:46.041335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698488+0000","last_change":"2024-04-06T13:49:53.285149+0000","last_active":"2024-04-06T13:49:54.698488+0000","last_peered":"2024-04-06T13:49:54.698488+0000","last_clean":"2024-04-06T13:49:54.698488+0000","last_became_active":"2024-04-06T13:49:53.284852+0000","last_became_peered":"2024-04-06T13:49:53.284852+0000","last_unstale":"2024-04-06T13:49:54.698488+0000","last_undegraded":"2024-04-06T13:49:54.698488+0000","last_fullsized":"2024-04-06T13:49:54.698488+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:38:05.094430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.6","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697588+0000","last_change":"2024-04-06T13:49:53.716258+0000","last_active":"2024-04-06T13:49:54.697588+0000","last_peered":"2024-04-06T13:49:54.697588+0000","last_clean":"2024-04-06T13:49:54.697588+0000","last_became_active":"2024-04-06T13:49:53.715803+0000","last_became_peered":"2024-04-06T13:49:53.715803+0000","last_unstale":"2024-04-06T13:49:54.697588+0000","last_undegraded":"2024-04-06T13:49:54.697588+0000","last_fullsized":"2024-04-06T13:49:54.697588+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:05:19.850459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.711013+0000","last_change":"2024-04-06T13:49:53.291155+0000","last_active":"2024-04-06T13:49:54.711013+0000","last_peered":"2024-04-06T13:49:54.711013+0000","last_clean":"2024-04-06T13:49:54.711013+0000","last_became_active":"2024-04-06T13:49:53.290659+0000","last_became_peered":"2024-04-06T13:49:53.290659+0000","last_unstale":"2024-04-06T13:49:54.711013+0000","last_undegraded":"2024-04-06T13:49:54.711013+0000","last_fullsized":"2024-04-06T13:49:54.711013+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:34:21.772869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.7","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698584+0000","last_change":"2024-04-06T13:49:53.718207+0000","last_active":"2024-04-06T13:49:54.698584+0000","last_peered":"2024-04-06T13:49:54.698584+0000","last_clean":"2024-04-06T13:49:54.698584+0000","last_became_active":"2024-04-06T13:49:53.717978+0000","last_became_peered":"2024-04-06T13:49:53.717978+0000","last_unstale":"2024-04-06T13:49:54.698584+0000","last_undegraded":"2024-04-06T13:49:54.698584+0000","last_fullsized":"2024-04-06T13:49:54.698584+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:57:30.699609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710076+0000","last_change":"2024-04-06T13:49:53.293115+0000","last_active":"2024-04-06T13:49:54.710076+0000","last_peered":"2024-04-06T13:49:54.710076+0000","last_clean":"2024-04-06T13:49:54.710076+0000","last_became_active":"2024-04-06T13:49:53.292836+0000","last_became_peered":"2024-04-06T13:49:53.292836+0000","last_unstale":"2024-04-06T13:49:54.710076+0000","last_undegraded":"2024-04-06T13:49:54.710076+0000","last_fullsized":"2024-04-06T13:49:54.710076+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:59:25.980159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.0","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710160+0000","last_change":"2024-04-06T13:49:53.719482+0000","last_active":"2024-04-06T13:49:54.710160+0000","last_peered":"2024-04-06T13:49:54.710160+0000","last_clean":"2024-04-06T13:49:54.710160+0000","last_became_active":"2024-04-06T13:49:53.718852+0000","last_became_peered":"2024-04-06T13:49:53.718852+0000","last_unstale":"2024-04-06T13:49:54.710160+0000","last_undegraded":"2024-04-06T13:49:54.710160+0000","last_fullsized":"2024-04-06T13:49:54.710160+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:26:04.390932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.709964+0000","last_change":"2024-04-06T13:49:53.294361+0000","last_active":"2024-04-06T13:49:54.709964+0000","last_peered":"2024-04-06T13:49:54.709964+0000","last_clean":"2024-04-06T13:49:54.709964+0000","last_became_active":"2024-04-06T13:49:53.294107+0000","last_became_peered":"2024-04-06T13:49:53.294107+0000","last_unstale":"2024-04-06T13:49:54.709964+0000","last_undegraded":"2024-04-06T13:49:54.709964+0000","last_fullsized":"2024-04-06T13:49:54.709964+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:57:56.001057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.1","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697319+0000","last_change":"2024-04-06T13:49:53.714973+0000","last_active":"2024-04-06T13:49:54.697319+0000","last_peered":"2024-04-06T13:49:54.697319+0000","last_clean":"2024-04-06T13:49:54.697319+0000","last_became_active":"2024-04-06T13:49:53.714730+0000","last_became_peered":"2024-04-06T13:49:53.714730+0000","last_unstale":"2024-04-06T13:49:54.697319+0000","last_undegraded":"2024-04-06T13:49:54.697319+0000","last_fullsized":"2024-04-06T13:49:54.697319+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:37:11.722361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.697217+0000","last_change":"2024-04-06T13:49:53.295583+0000","last_active":"2024-04-06T13:49:54.697217+0000","last_peered":"2024-04-06T13:49:54.697217+0000","last_clean":"2024-04-06T13:49:54.697217+0000","last_became_active":"2024-04-06T13:49:53.295256+0000","last_became_peered":"2024-04-06T13:49:53.295256+0000","last_unstale":"2024-04-06T13:49:54.697217+0000","last_undegraded":"2024-04-06T13:49:54.697217+0000","last_fullsized":"2024-04-06T13:49:54.697217+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:24:57.976703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.2","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.698129+0000","last_change":"2024-04-06T13:49:53.718300+0000","last_active":"2024-04-06T13:49:54.698129+0000","last_peered":"2024-04-06T13:49:54.698129+0000","last_clean":"2024-04-06T13:49:54.698129+0000","last_became_active":"2024-04-06T13:49:53.718063+0000","last_became_peered":"2024-04-06T13:49:53.718063+0000","last_unstale":"2024-04-06T13:49:54.698129+0000","last_undegraded":"2024-04-06T13:49:54.698129+0000","last_fullsized":"2024-04-06T13:49:54.698129+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:08:52.340541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710650+0000","last_change":"2024-04-06T13:49:53.300260+0000","last_active":"2024-04-06T13:49:54.710650+0000","last_peered":"2024-04-06T13:49:54.710650+0000","last_clean":"2024-04-06T13:49:54.710650+0000","last_became_active":"2024-04-06T13:49:53.300036+0000","last_became_peered":"2024-04-06T13:49:53.300036+0000","last_unstale":"2024-04-06T13:49:54.710650+0000","last_undegraded":"2024-04-06T13:49:54.710650+0000","last_fullsized":"2024-04-06T13:49:54.710650+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T22:37:05.203120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.3","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.710579+0000","last_change":"2024-04-06T13:49:53.714764+0000","last_active":"2024-04-06T13:49:54.710579+0000","last_peered":"2024-04-06T13:49:54.710579+0000","last_clean":"2024-04-06T13:49:54.710579+0000","last_became_active":"2024-04-06T13:49:53.714003+0000","last_became_peered":"2024-04-06T13:49:53.714003+0000","last_unstale":"2024-04-06T13:49:54.710579+0000","last_undegraded":"2024-04-06T13:49:54.710579+0000","last_fullsized":"2024-04-06T13:49:54.710579+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:17:22.096202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":11,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.709643+0000","last_change":"2024-04-06T13:49:53.292671+0000","last_active":"2024-04-06T13:49:54.709643+0000","last_peered":"2024-04-06T13:49:54.709643+0000","last_clean":"2024-04-06T13:49:54.709643+0000","last_became_active":"2024-04-06T13:49:53.292399+0000","last_became_peered":"2024-04-06T13:49:53.292399+0000","last_unstale":"2024-04-06T13:49:54.709643+0000","last_undegraded":"2024-04-06T13:49:54.709643+0000","last_fullsized":"2024-04-06T13:49:54.709643+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:52.264807+0000","last_clean_scrub_stamp":"2024-04-06T13:49:52.264807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:42:37.395499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.27","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-06T13:49:54.711313+0000","last_change":"2024-04-06T13:49:53.708856+0000","last_active":"2024-04-06T13:49:54.711313+0000","last_peered":"2024-04-06T13:49:54.711313+0000","last_clean":"2024-04-06T13:49:54.711313+0000","last_became_active":"2024-04-06T13:49:53.708527+0000","last_became_peered":"2024-04-06T13:49:53.708527+0000","last_unstale":"2024-04-06T13:49:54.711313+0000","last_undegraded":"2024-04-06T13:49:54.711313+0000","last_fullsized":"2024-04-06T13:49:54.711313+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":281,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T13:49:53.269722+0000","last_clean_scrub_stamp":"2024-04-06T13:49:53.269722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:15:37.526304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"272'403","reported_seq":1489,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-04-06T13:49:53.703707+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:49:53.703707+0000","last_peered":"2024-04-06T13:49:53.703707+0000","last_clean":"2024-04-06T13:49:53.703707+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:49:53.703707+0000","last_undegraded":"2024-04-06T13:49:53.703707+0000","last_fullsized":"2024-04-06T13:49:53.703707+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":38,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":37,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576008,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57340,"kb_used_data":4080,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314500,"statfs":{"total":96636764160,"available":96578048000,"internally_reserved":0,"allocated":4177920,"data_stored":2438315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55700000000000005,"15min":0.59199999999999997},"min":{"1min":0.34499999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.747,"5min":0.751,"15min":0.82499999999999996},"last":0.628},{"interface":"front","average":{"1min":0.505,"5min":0.54700000000000004,"15min":0.58299999999999996},"min":{"1min":0.35099999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.753,"5min":0.78500000000000003,"15min":0.82899999999999996},"last":0.53700000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53800000000000003,"15min":0.57499999999999996},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.69199999999999995,"5min":0.77700000000000002,"15min":0.81299999999999994},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":0.58799999999999997},"min":{"1min":0.39200000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.85299999999999998},"last":0.60999999999999999}]},{"osd":2,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.53700000000000003,"15min":0.57599999999999996},"min":{"1min":0.379,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.63800000000000001,"5min":0.78600000000000003,"15min":0.83899999999999997},"last":0.434},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57099999999999995,"15min":0.59599999999999997},"min":{"1min":0.373,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.79700000000000004},"last":0.57499999999999996}]},{"osd":3,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.56399999999999995,"15min":0.59699999999999998},"min":{"1min":0.36799999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.72699999999999998,"5min":0.76700000000000002,"15min":0.81100000000000005},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.57399999999999995,"15min":0.59799999999999998},"min":{"1min":0.436,"5min":0.39000000000000001,"15min":0.34999999999999998},"max":{"1min":0.66500000000000004,"5min":0.78900000000000003,"15min":0.84099999999999997},"last":0.65600000000000003}]},{"osd":4,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.45700000000000002,"15min":0.46300000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.67600000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54200000000000004,"15min":0.54000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64700000000000002,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.56100000000000005}]},{"osd":5,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.498,"15min":0.496},"min":{"1min":0.316,"5min":0.315,"15min":0.308},"max":{"1min":0.68999999999999995,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.44500000000000001,"5min":0.35599999999999998,"15min":0.33600000000000002},"max":{"1min":0.77300000000000002,"5min":0.79700000000000004,"15min":0.82099999999999995},"last":0.67000000000000004}]},{"osd":6,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55400000000000005,"15min":0.55500000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34899999999999998},"max":{"1min":0.69099999999999995,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56100000000000005,"15min":0.58099999999999996},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.65700000000000003,"5min":0.81799999999999995,"15min":0.88200000000000001},"last":0.59999999999999998}]}]},{"osd":6,"up_from":13,"seq":55834576007,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60856,"kb_used_data":3500,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310984,"statfs":{"total":96636764160,"available":96574447616,"internally_reserved":0,"allocated":3584000,"data_stored":1847510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.51900000000000002,"15min":0.55000000000000004},"min":{"1min":0.35199999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.59199999999999997,"5min":0.79700000000000004,"15min":0.83799999999999997},"last":0.439},{"interface":"front","average":{"1min":0.497,"5min":0.53300000000000003,"15min":0.56200000000000006},"min":{"1min":0.314,"5min":0.31,"15min":0.31},"max":{"1min":0.63,"5min":0.76900000000000002,"15min":0.81999999999999995},"last":0.49099999999999999}]},{"osd":1,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.59099999999999997,"15min":0.61699999999999999},"min":{"1min":0.39600000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70699999999999996,"5min":0.81699999999999995,"15min":0.85799999999999998},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.60299999999999998,"15min":0.63900000000000001},"min":{"1min":0.45100000000000001,"5min":0.41099999999999998,"15min":0.378},"max":{"1min":0.74199999999999999,"5min":0.78900000000000003,"15min":0.82699999999999996},"last":0.57199999999999995}]},{"osd":2,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56100000000000005,"15min":0.58899999999999997},"min":{"1min":0.41799999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72899999999999998,"5min":0.79000000000000004,"15min":0.82899999999999996},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.54200000000000004,"15min":0.57899999999999996},"min":{"1min":0.35899999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67400000000000004,"5min":0.77700000000000002,"15min":0.80700000000000005},"last":0.59799999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.56699999999999995,"15min":0.59799999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69999999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61599999999999999,"15min":0.63900000000000001},"min":{"1min":0.441,"5min":0.39900000000000002,"15min":0.36099999999999999},"max":{"1min":0.77200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.55500000000000005}]},{"osd":4,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53300000000000003,"15min":0.53100000000000003},"min":{"1min":0.36099999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.81399999999999995},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51500000000000001,"15min":0.53400000000000003},"min":{"1min":0.32900000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.73699999999999999,"5min":0.82099999999999995,"15min":0.84099999999999997},"last":0.58299999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.501,"15min":0.50600000000000001},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.33300000000000002},"max":{"1min":0.754,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52200000000000002,"15min":0.53100000000000003},"min":{"1min":0.40400000000000003,"5min":0.316,"15min":0.316},"max":{"1min":0.72599999999999998,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.52200000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79300000000000004,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55800000000000005,"15min":0.56699999999999995},"min":{"1min":0.39900000000000002,"5min":0.35399999999999998,"15min":0.32800000000000001},"max":{"1min":0.77200000000000002,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.61899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834576006,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64952,"kb_used_data":3492,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306888,"statfs":{"total":96636764160,"available":96570253312,"internally_reserved":0,"allocated":3575808,"data_stored":1846906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.36299999999999999,"15min":0.38500000000000001},"min":{"1min":0.26100000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.443,"5min":0.621,"15min":0.621},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.39200000000000002,"15min":0.41999999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.504,"5min":0.56100000000000005,"15min":0.69399999999999995},"last":0.39500000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.39400000000000002,"15min":0.41699999999999998},"min":{"1min":0.25600000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.41999999999999998,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.42699999999999999,"15min":0.45600000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.51900000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.33200000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.44600000000000001,"15min":0.47199999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.51100000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.374,"5min":0.40799999999999997,"15min":0.434},"min":{"1min":0.29599999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.51700000000000002,"5min":0.63500000000000001,"15min":0.66100000000000003},"last":0.26700000000000002}]},{"osd":4,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.59399999999999997,"15min":0.63400000000000001},"min":{"1min":0.377,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.80300000000000005,"5min":0.89900000000000002,"15min":0.97399999999999998},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.61199999999999999,"15min":0.64300000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.79300000000000004,"5min":0.877,"15min":0.995},"last":0.56299999999999994}]},{"osd":5,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.60499999999999998,"15min":0.64200000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.72699999999999998,"5min":0.91800000000000004,"15min":1.073},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.626,"15min":0.65900000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.73999999999999999,"5min":0.96499999999999997,"15min":1.0209999999999999},"last":0.69199999999999995}]},{"osd":6,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.61799999999999999,"15min":0.64900000000000002},"min":{"1min":0.435,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.753,"5min":0.89200000000000002,"15min":1},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.63600000000000001,"15min":0.67200000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76600000000000001,"5min":0.94599999999999995,"15min":1.036},"last":0.68200000000000005}]},{"osd":7,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.66400000000000003,"15min":0.68799999999999994},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.94899999999999995,"15min":1.0649999999999999},"last":0.746},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.626,"15min":0.66000000000000003},"min":{"1min":0.44700000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.78100000000000003,"5min":0.92900000000000005,"15min":1.056},"last":0.66800000000000004}]}]},{"osd":0,"up_from":13,"seq":55834576006,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61440,"kb_used_data":4076,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310400,"statfs":{"total":96636764160,"available":96573849600,"internally_reserved":0,"allocated":4173824,"data_stored":2437364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.36899999999999999,"15min":0.38900000000000001},"min":{"1min":0.24299999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.47199999999999998,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.245},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.41299999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.50900000000000001,"5min":0.58599999999999997,"15min":0.70799999999999996},"last":0.314}]},{"osd":2,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39000000000000001,"15min":0.42199999999999999},"min":{"1min":0.28299999999999997,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.53900000000000003,"5min":0.65400000000000003,"15min":0.70099999999999996},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.41699999999999998,"15min":0.441},"min":{"1min":0.29899999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.58199999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.28699999999999998}]},{"osd":3,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.39600000000000002,"15min":0.42099999999999999},"min":{"1min":0.27300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52400000000000002,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.44800000000000001,"15min":0.47399999999999998},"min":{"1min":0.313,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.60899999999999999,"5min":0.71899999999999997,"15min":0.75700000000000001},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.59099999999999997,"15min":0.63600000000000001},"min":{"1min":0.38500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.85199999999999998,"5min":0.93100000000000005,"15min":1.0329999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.60299999999999998,"15min":0.64100000000000001},"min":{"1min":0.39800000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86099999999999999,"5min":0.999,"15min":0.999},"last":0.57499999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59499999999999997,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.83899999999999997,"5min":0.91800000000000004,"15min":1.0009999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.627,"15min":0.66200000000000003},"min":{"1min":0.42699999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.86499999999999999,"5min":0.92800000000000005,"15min":1.052},"last":0.67600000000000005}]},{"osd":6,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63100000000000001,"15min":0.67000000000000004},"min":{"1min":0.438,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.81699999999999995,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.629,"15min":0.66800000000000004},"min":{"1min":0.44900000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":1.085},"last":0.56000000000000005}]},{"osd":7,"last update":"Sat Apr 6 13:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.64900000000000002,"15min":0.68799999999999994},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.89400000000000002,"5min":1.01,"15min":1.121},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.64000000000000001,"15min":0.67000000000000004},"min":{"1min":0.47299999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.82999999999999996,"5min":0.95799999999999996,"15min":1.0229999999999999},"last":0.59199999999999997}]}]},{"osd":2,"up_from":13,"seq":55834576008,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60872,"kb_used_data":3508,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310968,"statfs":{"total":96636764160,"available":96574431232,"internally_reserved":0,"allocated":3592192,"data_stored":1848540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.36799999999999999,"15min":0.38100000000000001},"min":{"1min":0.22,"5min":0.184,"15min":0.184},"max":{"1min":0.59399999999999997,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.39700000000000002,"15min":0.41899999999999998},"min":{"1min":0.26200000000000001,"5min":0.251,"15min":0.249},"max":{"1min":0.54600000000000004,"5min":0.71499999999999997,"15min":1.044},"last":0.41799999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.371,"15min":0.39900000000000002},"min":{"1min":0.23599999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.46899999999999997,"5min":0.52900000000000003,"15min":0.753},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.41799999999999998,"15min":0.441},"min":{"1min":0.27000000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.82199999999999995},"last":0.44900000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.42199999999999999,"15min":0.45200000000000001},"min":{"1min":0.27000000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.621,"5min":0.63900000000000001,"15min":0.70299999999999996},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.42499999999999999,"15min":0.443},"min":{"1min":0.27200000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.39600000000000002}]},{"osd":4,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61099999999999999,"15min":0.63300000000000001},"min":{"1min":0.436,"5min":0.40200000000000002,"15min":0.376},"max":{"1min":0.77600000000000002,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.623},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.63,"15min":0.65400000000000003},"min":{"1min":0.40999999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82799999999999996,"5min":1.022,"15min":1.022},"last":0.78900000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.629,"15min":0.65800000000000003},"min":{"1min":0.44900000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79000000000000004,"5min":0.98599999999999999,"15min":1.0149999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.63800000000000001,"15min":0.66000000000000003},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85199999999999998,"5min":1.028,"15min":1.028},"last":0.66100000000000003}]},{"osd":6,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64500000000000002,"15min":0.66600000000000004},"min":{"1min":0.45300000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86899999999999999,"5min":0.97199999999999998,"15min":1.091},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.625,"5min":0.65900000000000003,"15min":0.68799999999999994},"min":{"1min":0.47699999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84099999999999997,"5min":0.95699999999999996,"15min":0.96099999999999997},"last":0.65200000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.66100000000000003,"15min":0.68300000000000005},"min":{"1min":0.47299999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91300000000000003,"5min":0.94499999999999995,"15min":0.95399999999999996},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.66900000000000004,"15min":0.69999999999999996},"min":{"1min":0.46100000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.89100000000000001,"5min":1.01,"15min":1.01},"last":0.78200000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576007,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52672,"kb_used_data":3500,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319168,"statfs":{"total":96636764160,"available":96582828032,"internally_reserved":0,"allocated":3584000,"data_stored":1847959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.36099999999999999,"15min":0.379},"min":{"1min":0.23599999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.51300000000000001,"5min":0.55200000000000005,"15min":0.72799999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.317,"5min":0.375,"15min":0.40300000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.48299999999999998,"5min":0.58599999999999997,"15min":0.66700000000000004},"last":0.311}]},{"osd":1,"last update":"Sat Apr 6 13:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.378,"15min":0.39800000000000002},"min":{"1min":0.24399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.44600000000000001,"5min":0.59499999999999997,"15min":0.65700000000000003},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.40400000000000003,"15min":0.42999999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.59199999999999997,"5min":0.63,"15min":0.75800000000000001},"last":0.40799999999999997}]},{"osd":2,"last update":"Sat Apr 6 13:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.439,"15min":0.47299999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.56899999999999995,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.45800000000000002,"15min":0.48899999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.63400000000000001,"5min":0.67800000000000005,"15min":0.79300000000000004},"last":0.435}]},{"osd":4,"last update":"Sat Apr 6 13:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.60199999999999998,"15min":0.626},"min":{"1min":0.435,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.77100000000000002,"5min":0.88200000000000001,"15min":1.018},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.61299999999999999,"15min":0.63700000000000001},"min":{"1min":0.371,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.85399999999999998,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.51600000000000001}]},{"osd":5,"last update":"Sat Apr 6 13:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64700000000000002,"15min":0.67200000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.76700000000000002,"5min":0.95699999999999996,"15min":0.999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66000000000000003,"15min":0.68200000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84999999999999998,"5min":1.036,"15min":1.036},"last":0.71799999999999997}]},{"osd":6,"last update":"Sat Apr 6 13:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.63100000000000001,"15min":0.65500000000000003},"min":{"1min":0.40600000000000003,"5min":0.31,"15min":0.31},"max":{"1min":0.77900000000000003,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.64500000000000002,"15min":0.67200000000000004},"min":{"1min":0.45700000000000002,"5min":0.41999999999999998,"15min":0.41099999999999998},"max":{"1min":0.80500000000000005,"5min":0.91600000000000004,"15min":0.98799999999999999},"last":0.69999999999999996}]},{"osd":7,"last update":"Sat Apr 6 13:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.64100000000000001,"15min":0.66400000000000003},"min":{"1min":0.434,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85299999999999998,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.754},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.65600000000000003,"15min":0.68799999999999994},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81999999999999995,"5min":1.0209999999999999,"15min":1.024},"last":0.66000000000000003}]}]},{"osd":4,"up_from":13,"seq":55834576007,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48596,"kb_used_data":3520,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323244,"statfs":{"total":96636764160,"available":96587001856,"internally_reserved":0,"allocated":3604480,"data_stored":1850231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55300000000000005,"15min":0.57999999999999996},"min":{"1min":0.317,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73299999999999998,"5min":0.80900000000000005,"15min":0.88500000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.57599999999999996,"15min":0.59099999999999997},"min":{"1min":0.41699999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79200000000000004,"5min":1.202,"15min":1.202},"last":0.55300000000000005}]},{"osd":1,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60099999999999998,"15min":0.63400000000000001},"min":{"1min":0.379,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.74299999999999999,"5min":1.256,"15min":1.256},"last":0.625},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61499999999999999,"15min":0.624},"min":{"1min":0.39300000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.71999999999999997,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.58699999999999997}]},{"osd":2,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.56899999999999995,"15min":0.59099999999999997},"min":{"1min":0.377,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.68200000000000005,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.59299999999999997,"15min":0.61199999999999999},"min":{"1min":0.373,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89500000000000002,"5min":1.294,"15min":1.294},"last":0.65400000000000003}]},{"osd":3,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.57499999999999996,"15min":0.58999999999999997},"min":{"1min":0.35999999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80800000000000005,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60399999999999998,"15min":0.62},"min":{"1min":0.439,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76700000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.60199999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56100000000000005,"15min":0.52000000000000002},"min":{"1min":0.33800000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.75,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56999999999999995,"15min":0.54100000000000004},"min":{"1min":0.372,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.75,"5min":4.931,"15min":4.931},"last":0.42599999999999999}]},{"osd":6,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53900000000000003,"15min":0.53400000000000003},"min":{"1min":0.40699999999999997,"5min":0.34100000000000003,"15min":0.33300000000000002},"max":{"1min":0.78300000000000003,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56599999999999995,"15min":0.56999999999999995},"min":{"1min":0.41099999999999998,"5min":0.35799999999999998,"15min":0.34699999999999998},"max":{"1min":0.97099999999999997,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.52200000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60199999999999998,"15min":0.58199999999999996},"min":{"1min":0.35899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.84999999999999998,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.59399999999999997,"15min":0.60099999999999998},"min":{"1min":0.48599999999999999,"5min":0.38,"15min":0.373},"max":{"1min":0.82799999999999996,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.40300000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576007,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66620,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305220,"statfs":{"total":96636764160,"available":96568545280,"internally_reserved":0,"allocated":3579904,"data_stored":1846940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.54000000000000004,"15min":0.57699999999999996},"min":{"1min":0.33100000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69099999999999995,"5min":0.74099999999999999,"15min":0.82199999999999995},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.33000000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.62,"5min":0.81999999999999995,"15min":0.89200000000000002},"last":0.45200000000000001}]},{"osd":1,"last update":"Sat Apr 6 13:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.53900000000000003,"15min":0.58099999999999996},"min":{"1min":0.34699999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70899999999999996,"5min":0.80300000000000005,"15min":0.81399999999999995},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.505,"5min":0.53900000000000003,"15min":0.57699999999999996},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.64900000000000002,"5min":0.871,"15min":0.871},"last":0.63}]},{"osd":2,"last update":"Sat Apr 6 13:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.55300000000000005,"15min":0.58999999999999997},"min":{"1min":0.374,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72899999999999998,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.55400000000000005,"15min":0.58799999999999997},"min":{"1min":0.376,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70999999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.745}]},{"osd":3,"last update":"Sat Apr 6 13:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55200000000000005,"15min":0.58999999999999997},"min":{"1min":0.37,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.75900000000000001,"5min":0.81100000000000005,"15min":0.94499999999999995},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.58699999999999997,"15min":0.624},"min":{"1min":0.38,"5min":0.371,"15min":0.371},"max":{"1min":0.77800000000000002,"5min":0.88200000000000001,"15min":0.91500000000000004},"last":0.58999999999999997}]},{"osd":4,"last update":"Sat Apr 6 13:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.52200000000000002,"15min":0.50700000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.27400000000000002},"max":{"1min":0.75900000000000001,"5min":0.85699999999999998,"15min":1.0369999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.34699999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.82999999999999996,"5min":0.88800000000000001,"15min":1.1639999999999999},"last":0.69699999999999995}]},{"osd":6,"last update":"Sat Apr 6 13:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54600000000000004,"15min":0.53200000000000003},"min":{"1min":0.32600000000000001,"5min":0.313,"15min":0.28899999999999998},"max":{"1min":0.89900000000000002,"5min":0.94999999999999996,"15min":1.0169999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53800000000000003,"15min":0.53200000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.84299999999999997,"5min":0.878,"15min":1.0009999999999999},"last":0.61399999999999999}]},{"osd":7,"last update":"Sat Apr 6 13:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.55400000000000005,"15min":0.54800000000000004},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.85599999999999998,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.64800000000000002,"15min":0.64900000000000002},"min":{"1min":0.45400000000000001,"5min":0.41699999999999998,"15min":0.38300000000000001},"max":{"1min":0.87,"5min":0.92300000000000004,"15min":0.96599999999999997},"last":0.72399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:49:58.206 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-06T13:49:58.206 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-06T13:49:58.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.369+0000 7f44a29a2700 1 Processor -- start 2024-04-06T13:49:58.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.369+0000 7f44a29a2700 1 -- start start 2024-04-06T13:49:58.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.370+0000 7f44a29a2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f449c13d030 0x7f449c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:58.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.370+0000 7f44a29a2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449c0cea20 con 0x7f449c06ebe0 2024-04-06T13:49:58.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.370+0000 7f44a29a2700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449c15ec40 con 0x7f449c15e1e0 2024-04-06T13:49:58.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.370+0000 7f44a29a2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f449c13d410 con 0x7f449c13d030 2024-04-06T13:49:58.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.370+0000 7f44a21a1700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f449c06ebe0 0x7f449c0cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49928/0 (socket says 172.21.15.189:49928) 2024-04-06T13:49:58.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.370+0000 7f44a21a1700 1 -- 172.21.15.189:0/2683427337 learned_addr learned my addr 172.21.15.189:0/2683427337 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:58.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.370+0000 7f44a19a0700 1 --2- 172.21.15.189:0/2683427337 >> v2:172.21.15.192:3300/0 conn(0x7f449c13d030 0x7f449c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:58.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a099e700 1 -- 172.21.15.189:0/2683427337 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019216617 0 0) 0x7f449c0cea20 con 0x7f449c06ebe0 2024-04-06T13:49:58.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a099e700 1 -- 172.21.15.189:0/2683427337 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44880035b0 con 0x7f449c06ebe0 2024-04-06T13:49:58.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a099e700 1 -- 172.21.15.189:0/2683427337 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 374760655 0 0) 0x7f44880035b0 con 0x7f449c06ebe0 2024-04-06T13:49:58.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a099e700 1 -- 172.21.15.189:0/2683427337 >> v2:172.21.15.192:3300/0 conn(0x7f449c13d030 msgr2=0x7f449c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:58.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a099e700 1 --2- 172.21.15.189:0/2683427337 >> v2:172.21.15.192:3300/0 conn(0x7f449c13d030 0x7f449c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a099e700 1 -- 172.21.15.189:0/2683427337 >> v1:172.21.15.192:6789/0 conn(0x7f449c15e1e0 legacy=0x7f449c13d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:58.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a099e700 1 -- 172.21.15.189:0/2683427337 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c15e680 con 0x7f449c06ebe0 2024-04-06T13:49:58.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a099e700 1 -- 172.21.15.189:0/2683427337 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f44940019f0 con 0x7f449c06ebe0 2024-04-06T13:49:58.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a099e700 1 -- 172.21.15.189:0/2683427337 >> v1:172.21.15.189:6789/0 conn(0x7f449c06ebe0 legacy=0x7f449c0cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:58.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a099e700 1 --2- 172.21.15.189:0/2683427337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4488003a00 0x7f4488003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:58.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a099e700 1 --2- 172.21.15.189:0/2683427337 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4488004850 0x7f4488004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:58.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a099e700 1 --2- 172.21.15.189:0/2683427337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4488005210 0x7f448800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:58.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a099e700 1 -- 172.21.15.189:0/2683427337 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449c13d410 con 0x7f4488005210 2024-04-06T13:49:58.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a099e700 1 -- 172.21.15.189:0/2683427337 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448800c0b0 con 0x7f4488003a00 2024-04-06T13:49:58.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a099e700 1 -- 172.21.15.189:0/2683427337 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f448800e480 con 0x7f4488004850 2024-04-06T13:49:58.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a21a1700 1 --2- 172.21.15.189:0/2683427337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4488005210 0x7f448800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:58.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a19a0700 1 --2- 172.21.15.189:0/2683427337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4488003a00 0x7f4488003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:58.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a119f700 1 --2- 172.21.15.189:0/2683427337 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4488004850 0x7f4488004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:58.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a21a1700 1 -- 172.21.15.189:0/2683427337 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4488004850 msgr2=0x7f4488004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:58.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a21a1700 1 --2- 172.21.15.189:0/2683427337 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4488004850 0x7f4488004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a21a1700 1 -- 172.21.15.189:0/2683427337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4488003a00 msgr2=0x7f4488003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:58.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a21a1700 1 --2- 172.21.15.189:0/2683427337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4488003a00 0x7f4488003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a21a1700 1 -- 172.21.15.189:0/2683427337 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44880035b0 con 0x7f4488005210 2024-04-06T13:49:58.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a21a1700 1 -- 172.21.15.189:0/2683427337 reap_dead start 2024-04-06T13:49:58.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.371+0000 7f44a21a1700 1 --2- 172.21.15.189:0/2683427337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4488005210 0x7f448800ba10 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 crypto rx=0x7f449400a830 tx=0x7f4488005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bbadb4ba23ba611 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:58.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.372+0000 7f44a099e700 1 -- 172.21.15.189:0/2683427337 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449400a2f0 con 0x7f4488005210 2024-04-06T13:49:58.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.372+0000 7f44a099e700 1 -- 172.21.15.189:0/2683427337 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4494012600 con 0x7f4488005210 2024-04-06T13:49:58.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.372+0000 7f44a099e700 1 -- 172.21.15.189:0/2683427337 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44940129b0 con 0x7f4488005210 2024-04-06T13:49:58.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.372+0000 7f44a29a2700 1 -- 172.21.15.189:0/2683427337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4488005210 msgr2=0x7f448800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:58.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.372+0000 7f44a29a2700 1 --2- 172.21.15.189:0/2683427337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4488005210 0x7f448800ba10 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 crypto rx=0x7f449400a830 tx=0x7f4488005880 comp rx=0 tx=0).stop 2024-04-06T13:49:58.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.372+0000 7f44a29a2700 1 -- 172.21.15.189:0/2683427337 shutdown_connections 2024-04-06T13:49:58.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.372+0000 7f44a29a2700 1 --2- 172.21.15.189:0/2683427337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4488005210 0x7f448800ba10 unknown :-1 s=CLOSED pgs=3100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.372+0000 7f44a29a2700 1 -- 172.21.15.189:0/2683427337 >> 172.21.15.189:0/2683427337 conn(0x7f449c06d730 msgr2=0x7f449c06db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:58.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.372+0000 7f44a29a2700 1 -- 172.21.15.189:0/2683427337 shutdown_connections 2024-04-06T13:49:58.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.372+0000 7f44a29a2700 1 -- 172.21.15.189:0/2683427337 wait complete. 2024-04-06T13:49:58.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.373+0000 7f44a29a2700 1 Processor -- start 2024-04-06T13:49:58.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.374+0000 7f44a29a2700 1 -- start start 2024-04-06T13:49:58.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.374+0000 7f44a29a2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4488005210 0x7f449c1593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:58.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.374+0000 7f44a29a2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f449c15a720 0x7f449c0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:58.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.374+0000 7f44a29a2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f449c1f5b20 0x7f449c1f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:58.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.374+0000 7f44a29a2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449c15abb0 con 0x7f449c15a720 2024-04-06T13:49:58.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.374+0000 7f44a29a2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449c15ae90 con 0x7f449c1f5b20 2024-04-06T13:49:58.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.374+0000 7f44a29a2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f449c15b170 con 0x7f4488005210 2024-04-06T13:49:58.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.374+0000 7f44a119f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f449c15a720 0x7f449c0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:58.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.374+0000 7f44a119f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f449c15a720 0x7f449c0b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59452/0 (socket says 172.21.15.189:59452) 2024-04-06T13:49:58.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.374+0000 7f44a119f700 1 -- 172.21.15.189:0/2254072640 learned_addr learned my addr 172.21.15.189:0/2254072640 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:58.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.375+0000 7f44a19a0700 1 --2- 172.21.15.189:0/2254072640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4488005210 0x7f449c1593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:58.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.375+0000 7f44a21a1700 1 --2- 172.21.15.189:0/2254072640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f449c1f5b20 0x7f449c1f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:58.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.375+0000 7f44a119f700 1 -- 172.21.15.189:0/2254072640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4488005210 msgr2=0x7f449c1593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:58.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.375+0000 7f44a119f700 1 --2- 172.21.15.189:0/2254072640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4488005210 0x7f449c1593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.375+0000 7f44a119f700 1 -- 172.21.15.189:0/2254072640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f449c1f5b20 msgr2=0x7f449c1f5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:58.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.375+0000 7f44a119f700 1 --2- 172.21.15.189:0/2254072640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f449c1f5b20 0x7f449c1f5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.375+0000 7f44a119f700 1 -- 172.21.15.189:0/2254072640 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c15b450 con 0x7f449c15a720 2024-04-06T13:49:58.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.375+0000 7f44a21a1700 1 --2- 172.21.15.189:0/2254072640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f449c1f5b20 0x7f449c1f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:58.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.375+0000 7f44a119f700 1 --2- 172.21.15.189:0/2254072640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f449c15a720 0x7f449c0b4c00 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 crypto rx=0x7f448c00f5e0 tx=0x7f448c00f9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:58.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.375+0000 7f44927fc700 1 -- 172.21.15.189:0/2254072640 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c0052e0 con 0x7f449c15a720 2024-04-06T13:49:58.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.375+0000 7f44927fc700 1 -- 172.21.15.189:0/2254072640 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f448c005440 con 0x7f449c15a720 2024-04-06T13:49:58.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.375+0000 7f44a29a2700 1 -- 172.21.15.189:0/2254072640 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f449c15b730 con 0x7f449c15a720 2024-04-06T13:49:58.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.375+0000 7f44927fc700 1 -- 172.21.15.189:0/2254072640 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c0038c0 con 0x7f449c15a720 2024-04-06T13:49:58.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.375+0000 7f44a29a2700 1 -- 172.21.15.189:0/2254072640 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f449c159930 con 0x7f449c15a720 2024-04-06T13:49:58.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.376+0000 7f44a29a2700 1 -- 172.21.15.189:0/2254072640 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f449c0676d0 con 0x7f449c15a720 2024-04-06T13:49:58.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.376+0000 7f44927fc700 1 -- 172.21.15.189:0/2254072640 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f448c003a20 con 0x7f449c15a720 2024-04-06T13:49:58.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.377+0000 7f44927fc700 1 --2- 172.21.15.189:0/2254072640 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f448006adb0 0x7f448006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:58.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.377+0000 7f44927fc700 1 -- 172.21.15.189:0/2254072640 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f448c089380 con 0x7f449c15a720 2024-04-06T13:49:58.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.377+0000 7f44a19a0700 1 --2- 172.21.15.189:0/2254072640 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f448006adb0 0x7f448006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:58.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.377+0000 7f44a19a0700 1 --2- 172.21.15.189:0/2254072640 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f448006adb0 0x7f448006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f449c0af4b0 tx=0x7f4498008040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:58.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.381+0000 7f44927fc700 1 -- 172.21.15.189:0/2254072640 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f448c0542c0 con 0x7f449c15a720 2024-04-06T13:49:58.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.652+0000 7f44a29a2700 1 -- 172.21.15.189:0/2254072640 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f449c15b8c0 con 0x7f449c15a720 2024-04-06T13:49:58.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.656+0000 7f44927fc700 1 -- 172.21.15.189:0/2254072640 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v55) v1 ==== 190+0+0 (secure 0 0 0) 0x7f448c0578e0 con 0x7f449c15a720 2024-04-06T13:49:58.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.658+0000 7f44a29a2700 1 -- 172.21.15.189:0/2254072640 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f448006adb0 msgr2=0x7f448006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:58.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.658+0000 7f44a29a2700 1 --2- 172.21.15.189:0/2254072640 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f448006adb0 0x7f448006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f449c0af4b0 tx=0x7f4498008040 comp rx=0 tx=0).stop 2024-04-06T13:49:58.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.658+0000 7f44a29a2700 1 -- 172.21.15.189:0/2254072640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f449c15a720 msgr2=0x7f449c0b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:58.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.658+0000 7f44a29a2700 1 --2- 172.21.15.189:0/2254072640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f449c15a720 0x7f449c0b4c00 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 crypto rx=0x7f448c00f5e0 tx=0x7f448c00f9a0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.658+0000 7f44a29a2700 1 -- 172.21.15.189:0/2254072640 shutdown_connections 2024-04-06T13:49:58.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.658+0000 7f44a29a2700 1 --2- 172.21.15.189:0/2254072640 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f448006adb0 0x7f448006d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.658+0000 7f44a29a2700 1 --2- 172.21.15.189:0/2254072640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f449c1f5b20 0x7f449c1f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.658+0000 7f44a29a2700 1 --2- 172.21.15.189:0/2254072640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4488005210 0x7f449c1593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.658+0000 7f44a29a2700 1 --2- 172.21.15.189:0/2254072640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f449c15a720 0x7f449c0b4c00 unknown :-1 s=CLOSED pgs=3101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.658+0000 7f44a29a2700 1 -- 172.21.15.189:0/2254072640 >> 172.21.15.189:0/2254072640 conn(0x7f449c06d730 msgr2=0x7f449c152160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:58.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.658+0000 7f44a29a2700 1 -- 172.21.15.189:0/2254072640 shutdown_connections 2024-04-06T13:49:58.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.658+0000 7f44a29a2700 1 -- 172.21.15.189:0/2254072640 wait complete. 2024-04-06T13:49:58.661 INFO:teuthology.orchestra.run.smithi189.stderr:updated caps for client.0 2024-04-06T13:49:58.671 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:49:58.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.794+0000 7fc490164700 1 Processor -- start 2024-04-06T13:49:58.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.795+0000 7fc490164700 1 -- start start 2024-04-06T13:49:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.795+0000 7fc490164700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc48815d7b0 0x7fc4880b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.795+0000 7fc490164700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc48806a730 0x7fc48806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.795+0000 7fc490164700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc48815ddf0 0x7fc48815e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.795+0000 7fc490164700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc48806b0f0 con 0x7fc48815d7b0 2024-04-06T13:49:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.795+0000 7fc490164700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4880b3990 con 0x7fc48815ddf0 2024-04-06T13:49:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.795+0000 7fc490164700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4880b3060 con 0x7fc48806a730 2024-04-06T13:49:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.795+0000 7fc48df00700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc48815d7b0 0x7fc4880b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.795+0000 7fc48df00700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc48815d7b0 0x7fc4880b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59468/0 (socket says 172.21.15.189:59468) 2024-04-06T13:49:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.796+0000 7fc48df00700 1 -- 172.21.15.189:0/2341614739 learned_addr learned my addr 172.21.15.189:0/2341614739 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.796+0000 7fc48d6ff700 1 --2- 172.21.15.189:0/2341614739 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc48806a730 0x7fc48806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.796+0000 7fc48e701700 1 --2- 172.21.15.189:0/2341614739 >> v2:172.21.15.192:3300/0 conn(0x7fc48815ddf0 0x7fc48815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.796+0000 7fc48df00700 1 -- 172.21.15.189:0/2341614739 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc48806a730 msgr2=0x7fc48806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.796+0000 7fc48df00700 1 --2- 172.21.15.189:0/2341614739 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc48806a730 0x7fc48806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.796+0000 7fc48df00700 1 -- 172.21.15.189:0/2341614739 >> v2:172.21.15.192:3300/0 conn(0x7fc48815ddf0 msgr2=0x7fc48815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.796+0000 7fc48df00700 1 --2- 172.21.15.189:0/2341614739 >> v2:172.21.15.192:3300/0 conn(0x7fc48815ddf0 0x7fc48815e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.796+0000 7fc48df00700 1 -- 172.21.15.189:0/2341614739 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4880b3340 con 0x7fc48815d7b0 2024-04-06T13:49:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.796+0000 7fc48df00700 1 --2- 172.21.15.189:0/2341614739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc48815d7b0 0x7fc4880b3e20 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 crypto rx=0x7fc4880b4360 tx=0x7fc478009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c34b1d78e3ce8724 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.796+0000 7fc48cefe700 1 -- 172.21.15.189:0/2341614739 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47800b870 con 0x7fc48815d7b0 2024-04-06T13:49:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.796+0000 7fc48cefe700 1 -- 172.21.15.189:0/2341614739 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc47800b9d0 con 0x7fc48815d7b0 2024-04-06T13:49:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.796+0000 7fc48cefe700 1 -- 172.21.15.189:0/2341614739 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47800bd60 con 0x7fc48815d7b0 2024-04-06T13:49:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.797+0000 7fc490164700 1 -- 172.21.15.189:0/2341614739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc48815d7b0 msgr2=0x7fc4880b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.797+0000 7fc490164700 1 --2- 172.21.15.189:0/2341614739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc48815d7b0 0x7fc4880b3e20 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 crypto rx=0x7fc4880b4360 tx=0x7fc478009bd0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.797+0000 7fc490164700 1 -- 172.21.15.189:0/2341614739 shutdown_connections 2024-04-06T13:49:58.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.797+0000 7fc490164700 1 --2- 172.21.15.189:0/2341614739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc48815d7b0 0x7fc4880b3e20 unknown :-1 s=CLOSED pgs=3102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.797+0000 7fc490164700 1 --2- 172.21.15.189:0/2341614739 >> v2:172.21.15.192:3300/0 conn(0x7fc48815ddf0 0x7fc48815e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.797+0000 7fc490164700 1 --2- 172.21.15.189:0/2341614739 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc48806a730 0x7fc48806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.797+0000 7fc490164700 1 -- 172.21.15.189:0/2341614739 >> 172.21.15.189:0/2341614739 conn(0x7fc4880d00b0 msgr2=0x7fc4880d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:58.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.797+0000 7fc490164700 1 -- 172.21.15.189:0/2341614739 shutdown_connections 2024-04-06T13:49:58.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.797+0000 7fc490164700 1 -- 172.21.15.189:0/2341614739 wait complete. 2024-04-06T13:49:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.798+0000 7fc490164700 1 Processor -- start 2024-04-06T13:49:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.798+0000 7fc490164700 1 -- start start 2024-04-06T13:49:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc490164700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc48806a730 0x7fc4881492d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc490164700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc48815d7b0 0x7fc4880cf570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc490164700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc48815ddf0 0x7fc488146a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc490164700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc488147420 con 0x7fc48815ddf0 2024-04-06T13:49:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc490164700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4880cfab0 con 0x7fc48806a730 2024-04-06T13:49:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc490164700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc48814a220 con 0x7fc48815d7b0 2024-04-06T13:49:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc48e701700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc48815ddf0 0x7fc488146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc48e701700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc48815ddf0 0x7fc488146a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59470/0 (socket says 172.21.15.189:59470) 2024-04-06T13:49:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc48e701700 1 -- 172.21.15.189:0/15193657 learned_addr learned my addr 172.21.15.189:0/15193657 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc48df00700 1 --2- 172.21.15.189:0/15193657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc48806a730 0x7fc4881492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc48d6ff700 1 --2- 172.21.15.189:0/15193657 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc48815d7b0 0x7fc4880cf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc48e701700 1 -- 172.21.15.189:0/15193657 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc48815d7b0 msgr2=0x7fc4880cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc48e701700 1 --2- 172.21.15.189:0/15193657 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc48815d7b0 0x7fc4880cf570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc48e701700 1 -- 172.21.15.189:0/15193657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc48806a730 msgr2=0x7fc4881492d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc48e701700 1 --2- 172.21.15.189:0/15193657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc48806a730 0x7fc4881492d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:58.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc48e701700 1 -- 172.21.15.189:0/15193657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc478009550 con 0x7fc48815ddf0 2024-04-06T13:49:58.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc48e701700 1 --2- 172.21.15.189:0/15193657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc48815ddf0 0x7fc488146a10 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 crypto rx=0x7fc480007e90 tx=0x7fc48000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:58.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc47e7fc700 1 -- 172.21.15.189:0/15193657 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48000ed30 con 0x7fc48815ddf0 2024-04-06T13:49:58.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc47e7fc700 1 -- 172.21.15.189:0/15193657 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48000ee90 con 0x7fc48815ddf0 2024-04-06T13:49:58.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.799+0000 7fc490164700 1 -- 172.21.15.189:0/15193657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc488146f50 con 0x7fc48815ddf0 2024-04-06T13:49:58.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.800+0000 7fc47e7fc700 1 -- 172.21.15.189:0/15193657 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480014660 con 0x7fc48815ddf0 2024-04-06T13:49:58.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.800+0000 7fc490164700 1 -- 172.21.15.189:0/15193657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc488149c60 con 0x7fc48815ddf0 2024-04-06T13:49:58.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.800+0000 7fc490164700 1 -- 172.21.15.189:0/15193657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc488147560 con 0x7fc48815ddf0 2024-04-06T13:49:58.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.801+0000 7fc47e7fc700 1 -- 172.21.15.189:0/15193657 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fc4800147c0 con 0x7fc48815ddf0 2024-04-06T13:49:58.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.801+0000 7fc47e7fc700 1 --2- 172.21.15.189:0/15193657 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc46c06adb0 0x7fc46c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:58.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.801+0000 7fc47e7fc700 1 -- 172.21.15.189:0/15193657 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc480089bf0 con 0x7fc48815ddf0 2024-04-06T13:49:58.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.801+0000 7fc48df00700 1 --2- 172.21.15.189:0/15193657 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc46c06adb0 0x7fc46c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:58.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.802+0000 7fc48df00700 1 --2- 172.21.15.189:0/15193657 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc46c06adb0 0x7fc46c06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc478000c00 tx=0x7fc478008030 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:58.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:58.806+0000 7fc47e7fc700 1 -- 172.21.15.189:0/15193657 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc488147560 con 0x7fc48815ddf0 2024-04-06T13:49:59.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.047+0000 7fc490164700 1 -- 172.21.15.189:0/15193657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc488147560 con 0x7fc48815ddf0 2024-04-06T13:49:59.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.047+0000 7fc47e7fc700 1 -- 172.21.15.189:0/15193657 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4759 (secure 0 0 0) 0x7fc488147560 con 0x7fc48815ddf0 2024-04-06T13:49:59.048 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:49:59.048 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":139,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11827,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/819921213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":819921213},{"type":"v1","addr":"172.21.15.192:6835","nonce":819921213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134},{"gid":13368,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3616349784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3616349784},{"type":"v1","addr":"172.21.15.189:6836","nonce":3616349784}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134},{"gid":13371,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3640528525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3640528525},{"type":"v1","addr":"172.21.15.189:6837","nonce":3640528525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:49:53.688018+0000","modified":"2024-04-06T13:49:56.698920+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13380},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13380":{"gid":13380,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6834/3894184871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3894184871},{"type":"v1","addr":"172.21.15.192:6834","nonce":3894184871}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-04-06T13:49:59.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.050+0000 7fc490164700 1 -- 172.21.15.189:0/15193657 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc46c06adb0 msgr2=0x7fc46c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:59.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.050+0000 7fc490164700 1 --2- 172.21.15.189:0/15193657 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc46c06adb0 0x7fc46c06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc478000c00 tx=0x7fc478008030 comp rx=0 tx=0).stop 2024-04-06T13:49:59.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.050+0000 7fc490164700 1 -- 172.21.15.189:0/15193657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc48815ddf0 msgr2=0x7fc488146a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:59.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.050+0000 7fc490164700 1 --2- 172.21.15.189:0/15193657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc48815ddf0 0x7fc488146a10 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 crypto rx=0x7fc480007e90 tx=0x7fc48000e3c0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.050+0000 7fc490164700 1 -- 172.21.15.189:0/15193657 shutdown_connections 2024-04-06T13:49:59.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.050+0000 7fc490164700 1 --2- 172.21.15.189:0/15193657 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc46c06adb0 0x7fc46c06d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.050+0000 7fc490164700 1 --2- 172.21.15.189:0/15193657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc48815ddf0 0x7fc488146a10 unknown :-1 s=CLOSED pgs=3103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.050+0000 7fc490164700 1 --2- 172.21.15.189:0/15193657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc48806a730 0x7fc4881492d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.050+0000 7fc490164700 1 --2- 172.21.15.189:0/15193657 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc48815d7b0 0x7fc4880cf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.050+0000 7fc490164700 1 -- 172.21.15.189:0/15193657 >> 172.21.15.189:0/15193657 conn(0x7fc4880d00b0 msgr2=0x7fc48815f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:59.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.050+0000 7fc490164700 1 -- 172.21.15.189:0/15193657 shutdown_connections 2024-04-06T13:49:59.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.050+0000 7fc490164700 1 -- 172.21.15.189:0/15193657 wait complete. 2024-04-06T13:49:59.051 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 139 2024-04-06T13:49:59.268 INFO:tasks.ceph.mds.d.smithi192.stderr:2024-04-06T13:49:59.268+0000 7f4e5dd5d700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-06T13:49:59.464 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 139, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-06T13:49:53.688018+0000', 'modified': '2024-04-06T13:49:56.698920+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 13380}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_13380': {'gid': 13380, 'name': 'd', 'rank': 0, 'incarnation': 135, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.192:6834/3894184871', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6832', 'nonce': 3894184871}, {'type': 'v1', 'addr': '172.21.15.192:6834', 'nonce': 3894184871}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [38], 'metadata_pool': 37, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-06T13:49:59.464 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-06T13:49:59.465 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-06T13:49:59.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.592+0000 7fd64023e700 1 Processor -- start 2024-04-06T13:49:59.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.592+0000 7fd64023e700 1 -- start start 2024-04-06T13:49:59.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.593+0000 7fd64023e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6380d17a0 0x7fd63806b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.593+0000 7fd64023e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd6380cedf0 0x7fd6380cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.593+0000 7fd64023e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6380cf760 con 0x7fd638146dd0 2024-04-06T13:49:59.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.593+0000 7fd64023e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd63806d850 con 0x7fd6380cedf0 2024-04-06T13:49:59.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.593+0000 7fd64023e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63806c0a0 con 0x7fd6380d17a0 2024-04-06T13:49:59.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.593+0000 7fd63e7db700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd6380cedf0 0x7fd6380cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.593+0000 7fd63dfda700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6380d17a0 0x7fd63806b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.593+0000 7fd63d7d9700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fd638146dd0 0x7fd63806d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:46942/0 (socket says 172.21.15.189:46942) 2024-04-06T13:49:59.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.593+0000 7fd63d7d9700 1 -- 172.21.15.189:0/2033871076 learned_addr learned my addr 172.21.15.189:0/2033871076 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:49:59.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.593+0000 7fd63e7db700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd6380cedf0 0x7fd6380cf220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:42624/0 (socket says 172.21.15.189:42624) 2024-04-06T13:49:59.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.593+0000 7fd63e7db700 1 -- 172.21.15.189:0/2033871076 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6380d17a0 msgr2=0x7fd63806b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:59.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.593+0000 7fd63e7db700 1 --2- 172.21.15.189:0/2033871076 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6380d17a0 0x7fd63806b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.593+0000 7fd63e7db700 1 -- 172.21.15.189:0/2033871076 >> v1:172.21.15.192:6789/0 conn(0x7fd638146dd0 legacy=0x7fd63806d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:59.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.593+0000 7fd63e7db700 1 -- 172.21.15.189:0/2033871076 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63806c380 con 0x7fd6380cedf0 2024-04-06T13:49:59.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.593+0000 7fd63dfda700 1 --2- 172.21.15.189:0/2033871076 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6380d17a0 0x7fd63806b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:49:59.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.594+0000 7fd63e7db700 1 --2- 172.21.15.189:0/2033871076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6380cedf0 0x7fd6380cf220 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd630009d90 tx=0x7fd63000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c8b7a687ff630d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:59.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.594+0000 7fd63cfd8700 1 -- 172.21.15.189:0/2033871076 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63000e070 con 0x7fd6380cedf0 2024-04-06T13:49:59.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.594+0000 7fd63cfd8700 1 -- 172.21.15.189:0/2033871076 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd630003a20 con 0x7fd6380cedf0 2024-04-06T13:49:59.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.594+0000 7fd63cfd8700 1 -- 172.21.15.189:0/2033871076 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630003dd0 con 0x7fd6380cedf0 2024-04-06T13:49:59.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.594+0000 7fd64023e700 1 -- 172.21.15.189:0/2033871076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6380cedf0 msgr2=0x7fd6380cf220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:59.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.594+0000 7fd64023e700 1 --2- 172.21.15.189:0/2033871076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6380cedf0 0x7fd6380cf220 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd630009d90 tx=0x7fd63000b330 comp rx=0 tx=0).stop 2024-04-06T13:49:59.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.595+0000 7fd64023e700 1 -- 172.21.15.189:0/2033871076 shutdown_connections 2024-04-06T13:49:59.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.595+0000 7fd64023e700 1 --2- 172.21.15.189:0/2033871076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6380cedf0 0x7fd6380cf220 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.595+0000 7fd64023e700 1 --2- 172.21.15.189:0/2033871076 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6380d17a0 0x7fd63806b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.595+0000 7fd64023e700 1 -- 172.21.15.189:0/2033871076 >> 172.21.15.189:0/2033871076 conn(0x7fd6380b2db0 msgr2=0x7fd6380b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:59.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.595+0000 7fd64023e700 1 -- 172.21.15.189:0/2033871076 shutdown_connections 2024-04-06T13:49:59.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.595+0000 7fd64023e700 1 -- 172.21.15.189:0/2033871076 wait complete. 2024-04-06T13:49:59.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.596+0000 7fd64023e700 1 Processor -- start 2024-04-06T13:49:59.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.596+0000 7fd64023e700 1 -- start start 2024-04-06T13:49:59.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd64023e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6380cedf0 0x7fd638156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd64023e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6380d17a0 0x7fd6380b6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd64023e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd638146dd0 0x7fd6380b65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd64023e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6381f2c40 con 0x7fd638146dd0 2024-04-06T13:49:59.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd64023e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6380b06e0 con 0x7fd6380d17a0 2024-04-06T13:49:59.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd64023e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd638156c80 con 0x7fd6380cedf0 2024-04-06T13:49:59.599 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd63e7db700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd638146dd0 0x7fd6380b65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd63e7db700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd638146dd0 0x7fd6380b65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35136/0 (socket says 172.21.15.189:35136) 2024-04-06T13:49:59.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd63e7db700 1 -- 172.21.15.189:0/2995528507 learned_addr learned my addr 172.21.15.189:0/2995528507 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:59.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd63d7d9700 1 --2- 172.21.15.189:0/2995528507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6380d17a0 0x7fd6380b6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd63dfda700 1 --2- 172.21.15.189:0/2995528507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6380cedf0 0x7fd638156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd63e7db700 1 -- 172.21.15.189:0/2995528507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6380cedf0 msgr2=0x7fd638156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:59.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd63e7db700 1 --2- 172.21.15.189:0/2995528507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6380cedf0 0x7fd638156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd63e7db700 1 -- 172.21.15.189:0/2995528507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6380d17a0 msgr2=0x7fd6380b6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:59.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd63e7db700 1 --2- 172.21.15.189:0/2995528507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6380d17a0 0x7fd6380b6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd63e7db700 1 -- 172.21.15.189:0/2995528507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6340094a0 con 0x7fd638146dd0 2024-04-06T13:49:59.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd63d7d9700 1 --2- 172.21.15.189:0/2995528507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6380d17a0 0x7fd6380b6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:49:59.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd63e7db700 1 --2- 172.21.15.189:0/2995528507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd638146dd0 0x7fd6380b65b0 secure :-1 s=READY pgs=3104 cs=0 l=1 rev1=1 crypto rx=0x7fd6380677e0 tx=0x7fd630004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:59.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd62e7fc700 1 -- 172.21.15.189:0/2995528507 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630009410 con 0x7fd638146dd0 2024-04-06T13:49:59.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd62e7fc700 1 -- 172.21.15.189:0/2995528507 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd630004740 con 0x7fd638146dd0 2024-04-06T13:49:59.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd64023e700 1 -- 172.21.15.189:0/2995528507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd630009a70 con 0x7fd638146dd0 2024-04-06T13:49:59.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd62e7fc700 1 -- 172.21.15.189:0/2995528507 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630012430 con 0x7fd638146dd0 2024-04-06T13:49:59.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.597+0000 7fd64023e700 1 -- 172.21.15.189:0/2995528507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6380afdb0 con 0x7fd638146dd0 2024-04-06T13:49:59.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.599+0000 7fd62e7fc700 1 -- 172.21.15.189:0/2995528507 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fd630012590 con 0x7fd638146dd0 2024-04-06T13:49:59.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.599+0000 7fd62e7fc700 1 --2- 172.21.15.189:0/2995528507 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd61c06adb0 0x7fd61c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.599+0000 7fd62e7fc700 1 -- 172.21.15.189:0/2995528507 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd63008b440 con 0x7fd638146dd0 2024-04-06T13:49:59.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.599+0000 7fd63dfda700 1 --2- 172.21.15.189:0/2995528507 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd61c06adb0 0x7fd61c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.600+0000 7fd63dfda700 1 --2- 172.21.15.189:0/2995528507 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd61c06adb0 0x7fd61c06d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd634009c60 tx=0x7fd634008040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:59.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.601+0000 7fd64c3f2b80 1 Processor -- start 2024-04-06T13:49:59.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.601+0000 7fd64c3f2b80 1 -- start start 2024-04-06T13:49:59.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.602+0000 7fd64c3f2b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4c8c20 0x55c74e4cca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.602+0000 7fd64c3f2b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4cdfc0 0x55c74e4a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.602+0000 7fd64c3f2b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4a3980 0x55c74e4a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.602+0000 7fd64c3f2b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c74e3fd940 con 0x55c74e4cdfc0 2024-04-06T13:49:59.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.602+0000 7fd64c3f2b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c74e3fb300 con 0x55c74e4c8c20 2024-04-06T13:49:59.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.602+0000 7fd64c3f2b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c74e3f6bf0 con 0x55c74e4a3980 2024-04-06T13:49:59.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.602+0000 7fd63d7d9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4cdfc0 0x55c74e4a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.602+0000 7fd63d7d9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4cdfc0 0x55c74e4a45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35152/0 (socket says 172.21.15.189:35152) 2024-04-06T13:49:59.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.602+0000 7fd63d7d9700 1 -- 172.21.15.189:0/2243435625 learned_addr learned my addr 172.21.15.189:0/2243435625 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:59.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.602+0000 7fd63dfda700 1 --2- 172.21.15.189:0/2243435625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4c8c20 0x55c74e4cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.602+0000 7fd63e7db700 1 --2- 172.21.15.189:0/2243435625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4a3980 0x55c74e4a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.602+0000 7fd63d7d9700 1 -- 172.21.15.189:0/2243435625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4a3980 msgr2=0x55c74e4a4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:59.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.602+0000 7fd63d7d9700 1 --2- 172.21.15.189:0/2243435625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4a3980 0x55c74e4a4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.602+0000 7fd63d7d9700 1 -- 172.21.15.189:0/2243435625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4c8c20 msgr2=0x55c74e4cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:59.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.602+0000 7fd63d7d9700 1 --2- 172.21.15.189:0/2243435625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4c8c20 0x55c74e4cca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.602+0000 7fd63d7d9700 1 -- 172.21.15.189:0/2243435625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c74e224b10 con 0x55c74e4cdfc0 2024-04-06T13:49:59.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.602+0000 7fd63dfda700 1 --2- 172.21.15.189:0/2243435625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4c8c20 0x55c74e4cca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:59.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.602+0000 7fd63e7db700 1 --2- 172.21.15.189:0/2243435625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4a3980 0x55c74e4a4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:59.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.603+0000 7fd63d7d9700 1 --2- 172.21.15.189:0/2243435625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4cdfc0 0x55c74e4a45f0 secure :-1 s=READY pgs=3105 cs=0 l=1 rev1=1 crypto rx=0x7fd628005ed0 tx=0x7fd628007330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baebf770c7d72697 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:59.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.603+0000 7fd61b7fe700 1 -- 172.21.15.189:0/2243435625 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6280015e0 con 0x55c74e4cdfc0 2024-04-06T13:49:59.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.603+0000 7fd61b7fe700 1 -- 172.21.15.189:0/2243435625 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd628001740 con 0x55c74e4cdfc0 2024-04-06T13:49:59.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.603+0000 7fd61b7fe700 1 -- 172.21.15.189:0/2243435625 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628001ab0 con 0x55c74e4cdfc0 2024-04-06T13:49:59.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.603+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/2243435625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4cdfc0 msgr2=0x55c74e4a45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:59.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.603+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/2243435625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4cdfc0 0x55c74e4a45f0 secure :-1 s=READY pgs=3105 cs=0 l=1 rev1=1 crypto rx=0x7fd628005ed0 tx=0x7fd628007330 comp rx=0 tx=0).stop 2024-04-06T13:49:59.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.603+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/2243435625 shutdown_connections 2024-04-06T13:49:59.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.603+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/2243435625 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4a3980 0x55c74e4a4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.603+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/2243435625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4c8c20 0x55c74e4cca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.603+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/2243435625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4cdfc0 0x55c74e4a45f0 unknown :-1 s=CLOSED pgs=3105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.603+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/2243435625 >> 172.21.15.189:0/2243435625 conn(0x55c74e404650 msgr2=0x55c74e4eb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:59.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.603+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/2243435625 shutdown_connections 2024-04-06T13:49:59.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.604+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/2243435625 wait complete. 2024-04-06T13:49:59.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.604+0000 7fd64c3f2b80 1 -- start start 2024-04-06T13:49:59.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.604+0000 7fd64c3f2b80 1 Processor -- start 2024-04-06T13:49:59.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.605+0000 7fd64c3f2b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4a3980 0x55c74e4c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.605+0000 7fd64c3f2b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4c8c20 0x55c74e4d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.605+0000 7fd64c3f2b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4cdfc0 0x55c74e4d2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.605+0000 7fd64c3f2b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c74e392d30 con 0x55c74e4c8c20 2024-04-06T13:49:59.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.605+0000 7fd64c3f2b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c74e3929d0 con 0x55c74e4cdfc0 2024-04-06T13:49:59.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.605+0000 7fd64c3f2b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c74e27a650 con 0x55c74e4a3980 2024-04-06T13:49:59.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.605+0000 7fd64c3f2b80 10 client.0 _ll_register_callbacks cb 0x55c74e39c420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T13:49:59.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.605+0000 7fd63d7d9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4c8c20 0x55c74e4d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.605+0000 7fd63d7d9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4c8c20 0x55c74e4d2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35154/0 (socket says 172.21.15.189:35154) 2024-04-06T13:49:59.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.605+0000 7fd63d7d9700 1 -- 172.21.15.189:0/2115539351 learned_addr learned my addr 172.21.15.189:0/2115539351 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:59.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.605+0000 7fd63e7db700 1 --2- 172.21.15.189:0/2115539351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4cdfc0 0x55c74e4d2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.605+0000 7fd63dfda700 1 --2- 172.21.15.189:0/2115539351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4a3980 0x55c74e4c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.605+0000 7fd63d7d9700 1 -- 172.21.15.189:0/2115539351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4a3980 msgr2=0x55c74e4c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:59.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.605+0000 7fd63d7d9700 1 --2- 172.21.15.189:0/2115539351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4a3980 0x55c74e4c71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.605+0000 7fd63d7d9700 1 -- 172.21.15.189:0/2115539351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4cdfc0 msgr2=0x55c74e4d2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:59.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.605+0000 7fd63d7d9700 1 --2- 172.21.15.189:0/2115539351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4cdfc0 0x55c74e4d2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.605+0000 7fd63d7d9700 1 -- 172.21.15.189:0/2115539351 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd62800f040 con 0x55c74e4c8c20 2024-04-06T13:49:59.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.606+0000 7fd63d7d9700 1 --2- 172.21.15.189:0/2115539351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4c8c20 0x55c74e4d2850 secure :-1 s=READY pgs=3106 cs=0 l=1 rev1=1 crypto rx=0x7fd628007fd0 tx=0x7fd628007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fabe961001e563f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:59.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.606+0000 7fd618ff9700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T13:49:59.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.606+0000 7fd618ff9700 1 -- 172.21.15.189:0/2115539351 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6280015e0 con 0x55c74e4c8c20 2024-04-06T13:49:59.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.606+0000 7fd618ff9700 1 -- 172.21.15.189:0/2115539351 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(283..283 src has 1..283) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd628014690 con 0x55c74e4c8c20 2024-04-06T13:49:59.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.606+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/2115539351 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c74e3b7330 con 0x55c74e4c8c20 2024-04-06T13:49:59.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.606+0000 7fd618ff9700 1 -- 172.21.15.189:0/2115539351 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd628014850 con 0x55c74e4c8c20 2024-04-06T13:49:59.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.606+0000 7fd64c3f2b80 10 client.13506 fetch_fsmap learned FSMap version 139 2024-04-06T13:49:59.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.606+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/2115539351 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=2+}) v3 -- 0x55c74e3bce50 con 0x55c74e4c8c20 2024-04-06T13:49:59.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.606+0000 7fd618ff9700 1 -- 172.21.15.189:0/2115539351 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c74e3b7330 con 0x55c74e4c8c20 2024-04-06T13:49:59.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.606+0000 7fd618ff9700 1 -- 172.21.15.189:0/2115539351 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 139) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd6280149b0 con 0x55c74e4c8c20 2024-04-06T13:49:59.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.606+0000 7fd64c3f2b80 10 client.13506 fetch_fsmap finished waiting for FSMap version 139 2024-04-06T13:49:59.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.606+0000 7fd64c3f2b80 10 client.13506 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-06T13:49:59.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.606+0000 7fd64c3f2b80 20 client.13506 populate_metadata read hostname 'smithi189' 2024-04-06T13:49:59.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.606+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/2115539351 >> [v2:172.21.15.192:6832/3894184871,v1:172.21.15.192:6834/3894184871] conn(0x55c74e4d03a0 0x55c74e4d0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.606+0000 7fd64c3f2b80 4 client.13506 mds_command: new command op to 13380 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-06T13:49:59.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.606+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/2115539351 --> [v2:172.21.15.192:6832/3894184871,v1:172.21.15.192:6834/3894184871] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c74e3ef6b0 con 0x55c74e4d03a0 2024-04-06T13:49:59.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.606+0000 7fd63dfda700 1 --2- 172.21.15.189:0/2115539351 >> [v2:172.21.15.192:6832/3894184871,v1:172.21.15.192:6834/3894184871] conn(0x55c74e4d03a0 0x55c74e4d0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.607+0000 7fd618ff9700 10 client.13506 ms_handle_connect on v2:172.21.15.192:6832/3894184871 2024-04-06T13:49:59.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.607+0000 7fd63dfda700 1 --2- 172.21.15.189:0/2115539351 >> [v2:172.21.15.192:6832/3894184871,v1:172.21.15.192:6834/3894184871] conn(0x55c74e4d03a0 0x55c74e4d0780 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adb5d9ba44fddd23 server_cookie=253b7d753465cc4f in_seq=0 out_seq=0 2024-04-06T13:49:59.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.607+0000 7fd618ff9700 1 -- 172.21.15.189:0/2115539351 <== mds.0 v2:172.21.15.192:6832/3894184871 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c74e3ef6b0 con 0x55c74e4d03a0 2024-04-06T13:49:59.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.607+0000 7fd618ff9700 10 client.13506 handle_command_reply: tid=0 2024-04-06T13:49:59.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.607+0000 7fd64c3f2b80 1 client.13506 shutdown 2024-04-06T13:49:59.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.608+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/2115539351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4c8c20 msgr2=0x55c74e4d2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:59.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.608+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/2115539351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4c8c20 0x55c74e4d2850 secure :-1 s=READY pgs=3106 cs=0 l=1 rev1=1 crypto rx=0x7fd628007fd0 tx=0x7fd628007bb0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.608+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/2115539351 shutdown_connections 2024-04-06T13:49:59.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.608+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/2115539351 >> [v2:172.21.15.192:6832/3894184871,v1:172.21.15.192:6834/3894184871] conn(0x55c74e4d03a0 0x55c74e4d0780 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.608+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/2115539351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4cdfc0 0x55c74e4d2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.608+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/2115539351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4a3980 0x55c74e4c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.608+0000 7fd618ff9700 0 client.13506 ms_handle_reset on v2:172.21.15.192:6832/3894184871 2024-04-06T13:49:59.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.608+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/2115539351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4c8c20 0x55c74e4d2850 unknown :-1 s=CLOSED pgs=3106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.608+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/2115539351 >> 172.21.15.189:0/2115539351 conn(0x55c74e404650 msgr2=0x55c74e4a5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:59.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.608+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/2115539351 shutdown_connections 2024-04-06T13:49:59.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.608+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/2115539351 wait complete. 2024-04-06T13:49:59.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.608+0000 7fd64c3f2b80 20 client.13506 trim_cache size 0 max 16384 2024-04-06T13:49:59.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.626+0000 7fd64c3f2b80 1 Processor -- start 2024-04-06T13:49:59.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.626+0000 7fd64c3f2b80 1 -- start start 2024-04-06T13:49:59.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd64c3f2b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4a3980 0x55c74e4e1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd64c3f2b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4c8c20 0x55c74e4e24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd64c3f2b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4cdfc0 0x55c74e4c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd64c3f2b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c74e4ec3c0 con 0x55c74e4a3980 2024-04-06T13:49:59.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd64c3f2b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c74e4c54f0 con 0x55c74e4c8c20 2024-04-06T13:49:59.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd64c3f2b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c74e4a44f0 con 0x55c74e4cdfc0 2024-04-06T13:49:59.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd63dfda700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4a3980 0x55c74e4e1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd63dfda700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4a3980 0x55c74e4e1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35170/0 (socket says 172.21.15.189:35170) 2024-04-06T13:49:59.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd63dfda700 1 -- 172.21.15.189:0/2947578656 learned_addr learned my addr 172.21.15.189:0/2947578656 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:59.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd63d7d9700 1 --2- 172.21.15.189:0/2947578656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4c8c20 0x55c74e4e24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd63e7db700 1 --2- 172.21.15.189:0/2947578656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4cdfc0 0x55c74e4c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd63dfda700 1 -- 172.21.15.189:0/2947578656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4cdfc0 msgr2=0x55c74e4c73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:59.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd63dfda700 1 --2- 172.21.15.189:0/2947578656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4cdfc0 0x55c74e4c73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd63dfda700 1 -- 172.21.15.189:0/2947578656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4c8c20 msgr2=0x55c74e4e24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:59.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd63dfda700 1 --2- 172.21.15.189:0/2947578656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4c8c20 0x55c74e4e24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd63dfda700 1 -- 172.21.15.189:0/2947578656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c74e4faba0 con 0x55c74e4a3980 2024-04-06T13:49:59.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd63d7d9700 1 --2- 172.21.15.189:0/2947578656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4c8c20 0x55c74e4e24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:59.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd63e7db700 1 --2- 172.21.15.189:0/2947578656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4cdfc0 0x55c74e4c73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:59.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd63dfda700 1 --2- 172.21.15.189:0/2947578656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4a3980 0x55c74e4e1fb0 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7fd634009a40 tx=0x7fd634017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff8bbb0e5f539fb server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:59.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd61bfff700 1 -- 172.21.15.189:0/2947578656 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd634027040 con 0x55c74e4a3980 2024-04-06T13:49:59.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd61bfff700 1 -- 172.21.15.189:0/2947578656 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd634022bc0 con 0x55c74e4a3980 2024-04-06T13:49:59.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.627+0000 7fd61bfff700 1 -- 172.21.15.189:0/2947578656 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd634028240 con 0x55c74e4a3980 2024-04-06T13:49:59.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.628+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/2947578656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4a3980 msgr2=0x55c74e4e1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:59.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.628+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/2947578656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4a3980 0x55c74e4e1fb0 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7fd634009a40 tx=0x7fd634017000 comp rx=0 tx=0).stop 2024-04-06T13:49:59.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.628+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/2947578656 shutdown_connections 2024-04-06T13:49:59.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.628+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/2947578656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4cdfc0 0x55c74e4c73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.628+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/2947578656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4a3980 0x55c74e4e1fb0 unknown :-1 s=CLOSED pgs=3107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.628+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/2947578656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4c8c20 0x55c74e4e24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.628+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/2947578656 >> 172.21.15.189:0/2947578656 conn(0x55c74e404650 msgr2=0x55c74e4a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:59.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.628+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/2947578656 shutdown_connections 2024-04-06T13:49:59.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.628+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/2947578656 wait complete. 2024-04-06T13:49:59.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.628+0000 7fd64c3f2b80 1 -- start start 2024-04-06T13:49:59.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.629+0000 7fd64c3f2b80 1 Processor -- start 2024-04-06T13:49:59.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.629+0000 7fd64c3f2b80 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4a3980 0x55c74e5529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.629+0000 7fd64c3f2b80 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4c8c20 0x55c74e4cb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.629+0000 7fd64c3f2b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4cdfc0 0x55c74e4c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.629+0000 7fd64c3f2b80 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c74e4fdbe0 con 0x55c74e4a3980 2024-04-06T13:49:59.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.629+0000 7fd64c3f2b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c74e4cb550 con 0x55c74e4cdfc0 2024-04-06T13:49:59.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.629+0000 7fd64c3f2b80 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c74e4e1240 con 0x55c74e4c8c20 2024-04-06T13:49:59.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd64c3f2b80 10 client.0 _ll_register_callbacks cb 0x55c74e39c420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-06T13:49:59.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd63dfda700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4a3980 0x55c74e5529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd63dfda700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4a3980 0x55c74e5529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35186/0 (socket says 172.21.15.189:35186) 2024-04-06T13:49:59.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd63dfda700 1 -- 172.21.15.189:0/3759007930 learned_addr learned my addr 172.21.15.189:0/3759007930 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:59.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd63d7d9700 1 --2- 172.21.15.189:0/3759007930 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4c8c20 0x55c74e4cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd63e7db700 1 --2- 172.21.15.189:0/3759007930 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4cdfc0 0x55c74e4c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd63dfda700 1 -- 172.21.15.189:0/3759007930 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4c8c20 msgr2=0x55c74e4cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:59.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd63dfda700 1 --2- 172.21.15.189:0/3759007930 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4c8c20 0x55c74e4cb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd63dfda700 1 -- 172.21.15.189:0/3759007930 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4cdfc0 msgr2=0x55c74e4c73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:59.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd63dfda700 1 --2- 172.21.15.189:0/3759007930 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4cdfc0 0x55c74e4c73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd63dfda700 1 -- 172.21.15.189:0/3759007930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c74e27cc80 con 0x55c74e4a3980 2024-04-06T13:49:59.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd63d7d9700 1 --2- 172.21.15.189:0/3759007930 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4c8c20 0x55c74e4cb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:59.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd63dfda700 1 --2- 172.21.15.189:0/3759007930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4a3980 0x55c74e5529d0 secure :-1 s=READY pgs=3108 cs=0 l=1 rev1=1 crypto rx=0x7fd634029000 tx=0x7fd634004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85f92e5d3cc352d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:59.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd618ff9700 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-06T13:49:59.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd618ff9700 1 -- 172.21.15.189:0/3759007930 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd634027040 con 0x55c74e4a3980 2024-04-06T13:49:59.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd618ff9700 1 -- 172.21.15.189:0/3759007930 <== mon.0 v2:172.21.15.189:3300/0 2 ==== osd_map(283..283 src has 1..283) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd634028480 con 0x55c74e4a3980 2024-04-06T13:49:59.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/3759007930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd628001740 con 0x55c74e4a3980 2024-04-06T13:49:59.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd618ff9700 1 -- 172.21.15.189:0/3759007930 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6340287a0 con 0x55c74e4a3980 2024-04-06T13:49:59.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd64c3f2b80 10 client.13512 fetch_fsmap learned FSMap version 139 2024-04-06T13:49:59.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/3759007930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=2+}) v3 -- 0x55c74e222fe0 con 0x55c74e4a3980 2024-04-06T13:49:59.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.630+0000 7fd618ff9700 1 -- 172.21.15.189:0/3759007930 <== mon.0 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd628001740 con 0x55c74e4a3980 2024-04-06T13:49:59.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.631+0000 7fd618ff9700 1 -- 172.21.15.189:0/3759007930 <== mon.0 v2:172.21.15.189:3300/0 5 ==== fsmap(e 139) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd634028900 con 0x55c74e4a3980 2024-04-06T13:49:59.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.631+0000 7fd64c3f2b80 10 client.13512 fetch_fsmap finished waiting for FSMap version 139 2024-04-06T13:49:59.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.631+0000 7fd64c3f2b80 10 client.13512 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-06T13:49:59.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.631+0000 7fd64c3f2b80 20 client.13512 populate_metadata read hostname 'smithi189' 2024-04-06T13:49:59.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.631+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/3759007930 >> [v2:172.21.15.192:6832/3894184871,v1:172.21.15.192:6834/3894184871] conn(0x55c74e4cce90 0x55c74e4cd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.631+0000 7fd64c3f2b80 4 client.13512 mds_command: new command op to 13380 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-06T13:49:59.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.631+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/3759007930 --> [v2:172.21.15.192:6832/3894184871,v1:172.21.15.192:6834/3894184871] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c74e245ca0 con 0x55c74e4cce90 2024-04-06T13:49:59.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.631+0000 7fd63d7d9700 1 --2- 172.21.15.189:0/3759007930 >> [v2:172.21.15.192:6832/3894184871,v1:172.21.15.192:6834/3894184871] conn(0x55c74e4cce90 0x55c74e4cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.631+0000 7fd618ff9700 10 client.13512 ms_handle_connect on v2:172.21.15.192:6832/3894184871 2024-04-06T13:49:59.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.631+0000 7fd63d7d9700 1 --2- 172.21.15.189:0/3759007930 >> [v2:172.21.15.192:6832/3894184871,v1:172.21.15.192:6834/3894184871] conn(0x55c74e4cce90 0x55c74e4cd270 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef38e9d74620a650 server_cookie=5334ac708178c5f7 in_seq=0 out_seq=0 2024-04-06T13:49:59.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.631+0000 7fd618ff9700 1 -- 172.21.15.189:0/3759007930 <== mds.0 v2:172.21.15.192:6832/3894184871 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55c74e245ca0 con 0x55c74e4cce90 2024-04-06T13:49:59.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.631+0000 7fd618ff9700 10 client.13512 handle_command_reply: tid=0 2024-04-06T13:49:59.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.631+0000 7fd64c3f2b80 1 client.13512 shutdown 2024-04-06T13:49:59.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.631+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/3759007930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4a3980 msgr2=0x55c74e5529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:59.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.631+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/3759007930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4a3980 0x55c74e5529d0 secure :-1 s=READY pgs=3108 cs=0 l=1 rev1=1 crypto rx=0x7fd634029000 tx=0x7fd634004750 comp rx=0 tx=0).stop 2024-04-06T13:49:59.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/3759007930 shutdown_connections 2024-04-06T13:49:59.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/3759007930 >> [v2:172.21.15.192:6832/3894184871,v1:172.21.15.192:6834/3894184871] conn(0x55c74e4cce90 0x55c74e4cd270 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/3759007930 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74e4cdfc0 0x55c74e4c73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/3759007930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c74e4a3980 0x55c74e5529d0 unknown :-1 s=CLOSED pgs=3108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64c3f2b80 1 --2- 172.21.15.189:0/3759007930 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55c74e4c8c20 0x55c74e4cb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd618ff9700 0 client.13512 ms_handle_reset on v2:172.21.15.192:6832/3894184871 2024-04-06T13:49:59.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/3759007930 >> 172.21.15.189:0/3759007930 conn(0x55c74e404650 msgr2=0x55c74e4e20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:59.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/3759007930 shutdown_connections 2024-04-06T13:49:59.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64c3f2b80 1 -- 172.21.15.189:0/3759007930 wait complete. 2024-04-06T13:49:59.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64c3f2b80 20 client.13512 trim_cache size 0 max 16384 2024-04-06T13:49:59.639 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:49:59.639 INFO:teuthology.orchestra.run.smithi189.stdout: "cluster_fsid": "e30041de-e9a3-4351-8ca7-29d7f3e3728c", 2024-04-06T13:49:59.639 INFO:teuthology.orchestra.run.smithi189.stdout: "whoami": 0, 2024-04-06T13:49:59.639 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 13380, 2024-04-06T13:49:59.639 INFO:teuthology.orchestra.run.smithi189.stdout: "want_state": "up:active", 2024-04-06T13:49:59.639 INFO:teuthology.orchestra.run.smithi189.stdout: "state": "up:active", 2024-04-06T13:49:59.639 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs", 2024-04-06T13:49:59.639 INFO:teuthology.orchestra.run.smithi189.stdout: "rank_uptime": 5.941986183, 2024-04-06T13:49:59.639 INFO:teuthology.orchestra.run.smithi189.stdout: "mdsmap_epoch": 139, 2024-04-06T13:49:59.639 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch": 283, 2024-04-06T13:49:59.639 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_epoch_barrier": 283, 2024-04-06T13:49:59.639 INFO:teuthology.orchestra.run.smithi189.stdout: "uptime": 11.255914704 2024-04-06T13:49:59.639 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:49:59.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64023e700 1 -- 172.21.15.189:0/2995528507 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd61c06adb0 msgr2=0x7fd61c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:59.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64023e700 1 --2- 172.21.15.189:0/2995528507 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd61c06adb0 0x7fd61c06d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd634009c60 tx=0x7fd634008040 comp rx=0 tx=0).stop 2024-04-06T13:49:59.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64023e700 1 -- 172.21.15.189:0/2995528507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd638146dd0 msgr2=0x7fd6380b65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:59.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64023e700 1 --2- 172.21.15.189:0/2995528507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd638146dd0 0x7fd6380b65b0 secure :-1 s=READY pgs=3104 cs=0 l=1 rev1=1 crypto rx=0x7fd6380677e0 tx=0x7fd630004ad0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64023e700 1 -- 172.21.15.189:0/2995528507 shutdown_connections 2024-04-06T13:49:59.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64023e700 1 --2- 172.21.15.189:0/2995528507 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd61c06adb0 0x7fd61c06d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64023e700 1 --2- 172.21.15.189:0/2995528507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd638146dd0 0x7fd6380b65b0 unknown :-1 s=CLOSED pgs=3104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64023e700 1 --2- 172.21.15.189:0/2995528507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6380cedf0 0x7fd638156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64023e700 1 --2- 172.21.15.189:0/2995528507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6380d17a0 0x7fd6380b6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64023e700 1 -- 172.21.15.189:0/2995528507 >> 172.21.15.189:0/2995528507 conn(0x7fd6380b2db0 msgr2=0x7fd63806d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:59.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64023e700 1 -- 172.21.15.189:0/2995528507 shutdown_connections 2024-04-06T13:49:59.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.632+0000 7fd64023e700 1 -- 172.21.15.189:0/2995528507 wait complete. 2024-04-06T13:49:59.645 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-06T13:49:59.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.766+0000 7fd9b59fa700 1 Processor -- start 2024-04-06T13:49:59.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.767+0000 7fd9b59fa700 1 -- start start 2024-04-06T13:49:59.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.767+0000 7fd9b59fa700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd9b015b130 0x7fd9b00b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.767+0000 7fd9b59fa700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9b015b510 0x7fd9b00b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.767+0000 7fd9b59fa700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b00b1fb0 con 0x7fd9b0071de0 2024-04-06T13:49:59.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.767+0000 7fd9b59fa700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9b00b2200 con 0x7fd9b015b130 2024-04-06T13:49:59.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.767+0000 7fd9b59fa700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b00b24e0 con 0x7fd9b015b510 2024-04-06T13:49:59.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.767+0000 7fd9aeffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd9b015b130 0x7fd9b00b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.768+0000 7fd9aeffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd9b015b130 0x7fd9b00b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35200/0 (socket says 172.21.15.189:35200) 2024-04-06T13:49:59.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.768+0000 7fd9aeffd700 1 -- 172.21.15.189:0/3233431898 learned_addr learned my addr 172.21.15.189:0/3233431898 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:59.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.768+0000 7fd9ae7fc700 1 --2- 172.21.15.189:0/3233431898 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9b015b510 0x7fd9b00b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.768+0000 7fd9aeffd700 1 -- 172.21.15.189:0/3233431898 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9b015b510 msgr2=0x7fd9b00b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:59.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.768+0000 7fd9aeffd700 1 --2- 172.21.15.189:0/3233431898 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9b015b510 0x7fd9b00b39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.768+0000 7fd9aeffd700 1 -- 172.21.15.189:0/3233431898 >> v1:172.21.15.192:6789/0 conn(0x7fd9b0071de0 legacy=0x7fd9b0072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:49:59.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.768+0000 7fd9aeffd700 1 -- 172.21.15.189:0/3233431898 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b00b27c0 con 0x7fd9b015b130 2024-04-06T13:49:59.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.768+0000 7fd9ae7fc700 1 --2- 172.21.15.189:0/3233431898 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9b015b510 0x7fd9b00b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:49:59.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.768+0000 7fd9aeffd700 1 --2- 172.21.15.189:0/3233431898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9b015b130 0x7fd9b00b3470 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00098f0 tx=0x7fd9a0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a53d756bb6dcaa42 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:59.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.768+0000 7fd9adffb700 1 -- 172.21.15.189:0/3233431898 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0004230 con 0x7fd9b015b130 2024-04-06T13:49:59.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.768+0000 7fd9adffb700 1 -- 172.21.15.189:0/3233431898 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a0004390 con 0x7fd9b015b130 2024-04-06T13:49:59.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.768+0000 7fd9adffb700 1 -- 172.21.15.189:0/3233431898 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a00036a0 con 0x7fd9b015b130 2024-04-06T13:49:59.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.769+0000 7fd9b59fa700 1 -- 172.21.15.189:0/3233431898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9b015b130 msgr2=0x7fd9b00b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:59.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.769+0000 7fd9b59fa700 1 --2- 172.21.15.189:0/3233431898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9b015b130 0x7fd9b00b3470 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00098f0 tx=0x7fd9a0009c00 comp rx=0 tx=0).stop 2024-04-06T13:49:59.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.769+0000 7fd9b59fa700 1 -- 172.21.15.189:0/3233431898 shutdown_connections 2024-04-06T13:49:59.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.769+0000 7fd9b59fa700 1 --2- 172.21.15.189:0/3233431898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9b015b130 0x7fd9b00b3470 unknown :-1 s=CLOSED pgs=3109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.769+0000 7fd9b59fa700 1 --2- 172.21.15.189:0/3233431898 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9b015b510 0x7fd9b00b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.769+0000 7fd9b59fa700 1 -- 172.21.15.189:0/3233431898 >> 172.21.15.189:0/3233431898 conn(0x7fd9b006a730 msgr2=0x7fd9b006ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:49:59.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.769+0000 7fd9b59fa700 1 -- 172.21.15.189:0/3233431898 shutdown_connections 2024-04-06T13:49:59.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.769+0000 7fd9b59fa700 1 -- 172.21.15.189:0/3233431898 wait complete. 2024-04-06T13:49:59.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.770+0000 7fd9b59fa700 1 Processor -- start 2024-04-06T13:49:59.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.770+0000 7fd9b59fa700 1 -- start start 2024-04-06T13:49:59.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.771+0000 7fd9b59fa700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9b0071de0 0x7fd9b00ad2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.771+0000 7fd9b59fa700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9b015b130 0x7fd9b00aebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.771+0000 7fd9b59fa700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9b015b510 0x7fd9b006c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.771+0000 7fd9b59fa700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b013e840 con 0x7fd9b015b510 2024-04-06T13:49:59.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.771+0000 7fd9b59fa700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b013e030 con 0x7fd9b0071de0 2024-04-06T13:49:59.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.771+0000 7fd9b59fa700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b013ab40 con 0x7fd9b015b130 2024-04-06T13:49:59.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.771+0000 7fd9af7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9b015b510 0x7fd9b006c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.771+0000 7fd9af7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9b015b510 0x7fd9b006c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35202/0 (socket says 172.21.15.189:35202) 2024-04-06T13:49:59.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.771+0000 7fd9af7fe700 1 -- 172.21.15.189:0/2275356038 learned_addr learned my addr 172.21.15.189:0/2275356038 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:49:59.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.771+0000 7fd9aeffd700 1 --2- 172.21.15.189:0/2275356038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9b0071de0 0x7fd9b00ad2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.771+0000 7fd9ae7fc700 1 --2- 172.21.15.189:0/2275356038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9b015b130 0x7fd9b00aebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.771+0000 7fd9af7fe700 1 -- 172.21.15.189:0/2275356038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9b015b130 msgr2=0x7fd9b00aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.771+0000 7fd9af7fe700 1 --2- 172.21.15.189:0/2275356038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9b015b130 0x7fd9b00aebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.771+0000 7fd9af7fe700 1 -- 172.21.15.189:0/2275356038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9b0071de0 msgr2=0x7fd9b00ad2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:49:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.771+0000 7fd9af7fe700 1 --2- 172.21.15.189:0/2275356038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9b0071de0 0x7fd9b00ad2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:49:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.771+0000 7fd9af7fe700 1 -- 172.21.15.189:0/2275356038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a0009550 con 0x7fd9b015b510 2024-04-06T13:49:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.771+0000 7fd9af7fe700 1 --2- 172.21.15.189:0/2275356038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9b015b510 0x7fd9b006c750 secure :-1 s=READY pgs=3110 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800cc40 tx=0x7fd9a800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.771+0000 7fd9a7fff700 1 -- 172.21.15.189:0/2275356038 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a800aa80 con 0x7fd9b015b510 2024-04-06T13:49:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.772+0000 7fd9a7fff700 1 -- 172.21.15.189:0/2275356038 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a800abe0 con 0x7fd9b015b510 2024-04-06T13:49:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.772+0000 7fd9a7fff700 1 -- 172.21.15.189:0/2275356038 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8012420 con 0x7fd9b015b510 2024-04-06T13:49:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.772+0000 7fd9b59fa700 1 -- 172.21.15.189:0/2275356038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b013a230 con 0x7fd9b015b510 2024-04-06T13:49:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.772+0000 7fd9b59fa700 1 -- 172.21.15.189:0/2275356038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b013e560 con 0x7fd9b015b510 2024-04-06T13:49:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.773+0000 7fd9b59fa700 1 -- 172.21.15.189:0/2275356038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b013a910 con 0x7fd9b015b510 2024-04-06T13:49:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.773+0000 7fd9a7fff700 1 -- 172.21.15.189:0/2275356038 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fd9a8012580 con 0x7fd9b015b510 2024-04-06T13:49:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.773+0000 7fd9a7fff700 1 --2- 172.21.15.189:0/2275356038 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd99006ade0 0x7fd99006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:49:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.773+0000 7fd9a7fff700 1 -- 172.21.15.189:0/2275356038 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd9a8089b60 con 0x7fd9b015b510 2024-04-06T13:49:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.773+0000 7fd9aeffd700 1 --2- 172.21.15.189:0/2275356038 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd99006ade0 0x7fd99006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:49:59.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.774+0000 7fd9aeffd700 1 --2- 172.21.15.189:0/2275356038 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd99006ade0 0x7fd99006d2a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0009130 tx=0x7fd9a0008030 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:49:59.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:49:59.778+0000 7fd9a7fff700 1 -- 172.21.15.189:0/2275356038 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd9b013a910 con 0x7fd9b015b510 2024-04-06T13:50:00.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.017+0000 7fd9b59fa700 1 -- 172.21.15.189:0/2275356038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd9b013a910 con 0x7fd9b015b510 2024-04-06T13:50:00.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.018+0000 7fd9a7fff700 1 -- 172.21.15.189:0/2275356038 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v139) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd9b013a910 con 0x7fd9b015b510 2024-04-06T13:50:00.020 INFO:teuthology.orchestra.run.smithi189.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-06T13:50:00.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.020+0000 7fd9b59fa700 1 -- 172.21.15.189:0/2275356038 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd99006ade0 msgr2=0x7fd99006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:00.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.020+0000 7fd9b59fa700 1 --2- 172.21.15.189:0/2275356038 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd99006ade0 0x7fd99006d2a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0009130 tx=0x7fd9a0008030 comp rx=0 tx=0).stop 2024-04-06T13:50:00.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.020+0000 7fd9b59fa700 1 -- 172.21.15.189:0/2275356038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9b015b510 msgr2=0x7fd9b006c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:00.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.020+0000 7fd9b59fa700 1 --2- 172.21.15.189:0/2275356038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9b015b510 0x7fd9b006c750 secure :-1 s=READY pgs=3110 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800cc40 tx=0x7fd9a800cf50 comp rx=0 tx=0).stop 2024-04-06T13:50:00.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.020+0000 7fd9b59fa700 1 -- 172.21.15.189:0/2275356038 shutdown_connections 2024-04-06T13:50:00.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.020+0000 7fd9b59fa700 1 --2- 172.21.15.189:0/2275356038 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fd99006ade0 0x7fd99006d2a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:00.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.020+0000 7fd9b59fa700 1 --2- 172.21.15.189:0/2275356038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9b015b510 0x7fd9b006c750 unknown :-1 s=CLOSED pgs=3110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:00.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.020+0000 7fd9b59fa700 1 --2- 172.21.15.189:0/2275356038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9b0071de0 0x7fd9b00ad2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:00.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.020+0000 7fd9b59fa700 1 --2- 172.21.15.189:0/2275356038 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd9b015b130 0x7fd9b00aebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:00.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.020+0000 7fd9b59fa700 1 -- 172.21.15.189:0/2275356038 >> 172.21.15.189:0/2275356038 conn(0x7fd9b006a730 msgr2=0x7fd9b00b1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:50:00.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.020+0000 7fd9b59fa700 1 -- 172.21.15.189:0/2275356038 shutdown_connections 2024-04-06T13:50:00.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.020+0000 7fd9b59fa700 1 -- 172.21.15.189:0/2275356038 wait complete. 2024-04-06T13:50:00.032 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-06T13:50:00.032 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T13:50:00.032 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-06T13:50:00.032 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-06T13:50:00.032 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T13:50:00.032 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-06T13:50:00.032 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-06T13:50:00.033 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T13:50:00.033 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:50:00.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:50:00.033 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-06T13:50:00.051 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T13:50:00.051 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T13:50:00.051 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T13:50:00.051 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:50:00.051 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-06T13:50:00.051 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:50:00.051 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:50:00.051 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:50:00.051 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-06T13:50:00.051 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:50:00.051 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:50:00.051 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-06T13:50:00.052 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T13:50:00.052 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:50:00.052 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T13:50:00.052 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-06T13:50:00.052 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T13:50:00.052 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 36490sec preferred_lft 36490sec 2024-04-06T13:50:00.052 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::8592:df2c:92e2:dace/64 scope link noprefixroute 2024-04-06T13:50:00.052 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:50:00.052 INFO:teuthology.orchestra.run.smithi189.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-06T13:50:00.052 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff 2024-04-06T13:50:00.052 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-06T13:50:00.052 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:50:00.052 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::88e:b6ff:fe6f:1cb0/64 scope link 2024-04-06T13:50:00.052 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:50:00.052 INFO:teuthology.orchestra.run.smithi189.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T13:50:00.053 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 62:a5:8d:c5:1a:9c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T13:50:00.053 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::60a5:8dff:fec5:1a9c/64 scope link 2024-04-06T13:50:00.053 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:50:00.053 INFO:teuthology.orchestra.run.smithi189.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-06T13:50:00.053 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 1e:7e:c6:bb:1d:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-06T13:50:00.053 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::1c7e:c6ff:febb:1dab/64 scope link 2024-04-06T13:50:00.053 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-06T13:50:00.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:50:00.053 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T13:50:00.117 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-06T13:50:00.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:50:00.119 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-06T13:50:00.178 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-06T13:50:00.178 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T13:50:00.178 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T13:50:00.178 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T13:50:00.178 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T13:50:00.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:50:00.179 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-04-06T13:50:00.179 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-06T13:50:00.179 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T13:50:00.179 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T13:50:00.179 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T13:50:00.179 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-06T13:50:00.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:50:00.288 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T13:50:00.305 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-06T13:50:00.305 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-06T13:50:00.305 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:50:00.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:50:00.305 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-04-06T13:50:00.373 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T13:50:00.374 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-06T13:50:00.431 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-06T13:50:00.431 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T13:50:00.431 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-06T13:50:00.489 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T13:50:00.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:50:00.489 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:50:00.575 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:50:00.575 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:50:00.575 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:50:00.584 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:50:00.584 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:50:00.584 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:50:00.585 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T13:50:00.585 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T13:50:00.604 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T13:50:00.605 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T13:50:00.605 INFO:teuthology.orchestra.run.smithi189.stdout:invalid new device string format 2024-04-06T13:50:00.605 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:3300,172.21.15.192:3300,172.21.15.192:3301" 2024-04-06T13:50:00.605 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T13:50:00.605 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790" 2024-04-06T13:50:00.605 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.189:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2024-04-06T13:50:00.605 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e30041de-e9a3-4351-8ca7-29d7f3e3728c" will pass to kernel 2024-04-06T13:50:00.607 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T13:50:00.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:50:00.607 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-06T13:50:00.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:50:00.636 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T13:50:00.724 INFO:teuthology.orchestra.run.smithi189.stdout:client11935 2024-04-06T13:50:00.725 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:50:00.725 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:50:00.742 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-06T13:50:00.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.907+0000 7fb400d83700 1 Processor -- start 2024-04-06T13:50:00.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.907+0000 7fb400d83700 1 -- start start 2024-04-06T13:50:00.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.908+0000 7fb400d83700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb3fc15d040 0x7fb3fc15dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:00.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.908+0000 7fb400d83700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3fc06cbf0 0x7fb3fc0cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:00.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.908+0000 7fb400d83700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc15d8d0 con 0x7fb3fc15e2a0 2024-04-06T13:50:00.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.908+0000 7fb400d83700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3fc0cba40 con 0x7fb3fc15d040 2024-04-06T13:50:00.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.908+0000 7fb400d83700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc15d580 con 0x7fb3fc06cbf0 2024-04-06T13:50:00.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.909+0000 7fb3fad9d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3fc06cbf0 0x7fb3fc0cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:00.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.909+0000 7fb3fa59c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb3fc15d040 0x7fb3fc15dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:00.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.909+0000 7fb3fad9d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3fc06cbf0 0x7fb3fc0cc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:35002/0 (socket says 172.21.15.189:35002) 2024-04-06T13:50:00.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.909+0000 7fb3fad9d700 1 -- 172.21.15.189:0/4291696164 learned_addr learned my addr 172.21.15.189:0/4291696164 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:50:00.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.909+0000 7fb3fad9d700 1 -- 172.21.15.189:0/4291696164 >> v2:172.21.15.192:3300/0 conn(0x7fb3fc15d040 msgr2=0x7fb3fc15dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:50:00.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.909+0000 7fb3fad9d700 1 --2- 172.21.15.189:0/4291696164 >> v2:172.21.15.192:3300/0 conn(0x7fb3fc15d040 0x7fb3fc15dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:00.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.909+0000 7fb3fad9d700 1 -- 172.21.15.189:0/4291696164 >> v1:172.21.15.192:6789/0 conn(0x7fb3fc15e2a0 legacy=0x7fb3fc06c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:00.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.909+0000 7fb3fad9d700 1 -- 172.21.15.189:0/4291696164 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc0b3240 con 0x7fb3fc06cbf0 2024-04-06T13:50:00.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.909+0000 7fb3fa59c700 1 --2- 172.21.15.189:0/4291696164 >> v2:172.21.15.192:3300/0 conn(0x7fb3fc15d040 0x7fb3fc15dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:50:00.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.909+0000 7fb3fad9d700 1 --2- 172.21.15.189:0/4291696164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3fc06cbf0 0x7fb3fc0cc1e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4009e50 tx=0x7fb3f400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65d5278a02dabc97 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:50:00.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.909+0000 7fb3f959a700 1 -- 172.21.15.189:0/4291696164 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f400e070 con 0x7fb3fc06cbf0 2024-04-06T13:50:00.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.909+0000 7fb3f959a700 1 -- 172.21.15.189:0/4291696164 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f400b910 con 0x7fb3fc06cbf0 2024-04-06T13:50:00.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.909+0000 7fb3f959a700 1 -- 172.21.15.189:0/4291696164 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f400bcc0 con 0x7fb3fc06cbf0 2024-04-06T13:50:00.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.910+0000 7fb400d83700 1 -- 172.21.15.189:0/4291696164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3fc06cbf0 msgr2=0x7fb3fc0cc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:00.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.910+0000 7fb400d83700 1 --2- 172.21.15.189:0/4291696164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3fc06cbf0 0x7fb3fc0cc1e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4009e50 tx=0x7fb3f400b330 comp rx=0 tx=0).stop 2024-04-06T13:50:00.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.910+0000 7fb400d83700 1 -- 172.21.15.189:0/4291696164 shutdown_connections 2024-04-06T13:50:00.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.910+0000 7fb400d83700 1 --2- 172.21.15.189:0/4291696164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3fc06cbf0 0x7fb3fc0cc1e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:00.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.910+0000 7fb400d83700 1 --2- 172.21.15.189:0/4291696164 >> v2:172.21.15.192:3300/0 conn(0x7fb3fc15d040 0x7fb3fc15dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:00.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.910+0000 7fb400d83700 1 -- 172.21.15.189:0/4291696164 >> 172.21.15.189:0/4291696164 conn(0x7fb3fc06a730 msgr2=0x7fb3fc06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:50:00.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.910+0000 7fb400d83700 1 -- 172.21.15.189:0/4291696164 shutdown_connections 2024-04-06T13:50:00.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.910+0000 7fb400d83700 1 -- 172.21.15.189:0/4291696164 wait complete. 2024-04-06T13:50:00.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.911+0000 7fb400d83700 1 Processor -- start 2024-04-06T13:50:00.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.911+0000 7fb400d83700 1 -- start start 2024-04-06T13:50:00.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.912+0000 7fb400d83700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3fc06cbf0 0x7fb3fc146680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:00.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.912+0000 7fb400d83700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3fc15d040 0x7fb3fc15b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:00.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.912+0000 7fb400d83700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3fc14fce0 0x7fb3fc159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:00.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.912+0000 7fb400d83700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc14a890 con 0x7fb3fc06cbf0 2024-04-06T13:50:00.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.912+0000 7fb400d83700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc15be50 con 0x7fb3fc14fce0 2024-04-06T13:50:00.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.912+0000 7fb400d83700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc15c130 con 0x7fb3fc15d040 2024-04-06T13:50:00.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.912+0000 7fb3fa59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3fc06cbf0 0x7fb3fc146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:00.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.912+0000 7fb3fa59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3fc06cbf0 0x7fb3fc146680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35206/0 (socket says 172.21.15.189:35206) 2024-04-06T13:50:00.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.912+0000 7fb3fa59c700 1 -- 172.21.15.189:0/919828083 learned_addr learned my addr 172.21.15.189:0/919828083 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:50:00.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.912+0000 7fb3f9d9b700 1 --2- 172.21.15.189:0/919828083 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3fc15d040 0x7fb3fc15b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:00.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.912+0000 7fb3fad9d700 1 --2- 172.21.15.189:0/919828083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3fc14fce0 0x7fb3fc159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:00.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.912+0000 7fb3fa59c700 1 -- 172.21.15.189:0/919828083 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3fc15d040 msgr2=0x7fb3fc15b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:00.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.912+0000 7fb3fa59c700 1 --2- 172.21.15.189:0/919828083 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3fc15d040 0x7fb3fc15b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:00.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.912+0000 7fb3fa59c700 1 -- 172.21.15.189:0/919828083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3fc14fce0 msgr2=0x7fb3fc159380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:00.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.912+0000 7fb3fa59c700 1 --2- 172.21.15.189:0/919828083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3fc14fce0 0x7fb3fc159380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:00.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.912+0000 7fb3fa59c700 1 -- 172.21.15.189:0/919828083 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f000a640 con 0x7fb3fc06cbf0 2024-04-06T13:50:00.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.913+0000 7fb3f9d9b700 1 --2- 172.21.15.189:0/919828083 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3fc15d040 0x7fb3fc15b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:50:00.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.913+0000 7fb3fad9d700 1 --2- 172.21.15.189:0/919828083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3fc14fce0 0x7fb3fc159380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:50:00.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.913+0000 7fb3fa59c700 1 --2- 172.21.15.189:0/919828083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3fc06cbf0 0x7fb3fc146680 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000e9a0 tx=0x7fb3f000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:50:00.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.913+0000 7fb3eaffd700 1 -- 172.21.15.189:0/919828083 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f000cb50 con 0x7fb3fc06cbf0 2024-04-06T13:50:00.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.913+0000 7fb3eaffd700 1 -- 172.21.15.189:0/919828083 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f000ccb0 con 0x7fb3fc06cbf0 2024-04-06T13:50:00.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.913+0000 7fb400d83700 1 -- 172.21.15.189:0/919828083 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f4009b30 con 0x7fb3fc06cbf0 2024-04-06T13:50:00.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.913+0000 7fb3eaffd700 1 -- 172.21.15.189:0/919828083 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0011610 con 0x7fb3fc06cbf0 2024-04-06T13:50:00.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.913+0000 7fb400d83700 1 -- 172.21.15.189:0/919828083 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3fc14adc0 con 0x7fb3fc06cbf0 2024-04-06T13:50:00.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.914+0000 7fb400d83700 1 -- 172.21.15.189:0/919828083 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3fc14b130 con 0x7fb3fc06cbf0 2024-04-06T13:50:00.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.914+0000 7fb3eaffd700 1 -- 172.21.15.189:0/919828083 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fb3f0011770 con 0x7fb3fc06cbf0 2024-04-06T13:50:00.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.915+0000 7fb3eaffd700 1 --2- 172.21.15.189:0/919828083 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb3dc06adb0 0x7fb3dc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:00.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.915+0000 7fb3eaffd700 1 -- 172.21.15.189:0/919828083 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb3f00894a0 con 0x7fb3fc06cbf0 2024-04-06T13:50:00.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.915+0000 7fb3f9d9b700 1 --2- 172.21.15.189:0/919828083 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb3dc06adb0 0x7fb3dc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:00.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.915+0000 7fb3f9d9b700 1 --2- 172.21.15.189:0/919828083 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb3dc06adb0 0x7fb3dc06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc150230 tx=0x7fb3ec009040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:50:00.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:00.919+0000 7fb3eaffd700 1 -- 172.21.15.189:0/919828083 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb3f0054360 con 0x7fb3fc06cbf0 2024-04-06T13:50:01.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.145+0000 7fb400d83700 1 -- 172.21.15.189:0/919828083 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3fc0b3800 con 0x7fb3dc06adb0 2024-04-06T13:50:01.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.146+0000 7fb3eaffd700 1 -- 172.21.15.189:0/919828083 <== mgr.13152 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fb3ec006860 con 0x7fb3dc06adb0 2024-04-06T13:50:01.147 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2024-04-06T13:50:01.147 INFO:teuthology.orchestra.run.smithi189.stdout: { 2024-04-06T13:50:01.147 INFO:teuthology.orchestra.run.smithi189.stdout: "name": "cephfs" 2024-04-06T13:50:01.147 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-06T13:50:01.147 INFO:teuthology.orchestra.run.smithi189.stdout:] 2024-04-06T13:50:01.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.149+0000 7fb400d83700 1 -- 172.21.15.189:0/919828083 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb3dc06adb0 msgr2=0x7fb3dc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:01.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.149+0000 7fb400d83700 1 --2- 172.21.15.189:0/919828083 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb3dc06adb0 0x7fb3dc06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc150230 tx=0x7fb3ec009040 comp rx=0 tx=0).stop 2024-04-06T13:50:01.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.149+0000 7fb400d83700 1 -- 172.21.15.189:0/919828083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3fc06cbf0 msgr2=0x7fb3fc146680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:01.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.149+0000 7fb400d83700 1 --2- 172.21.15.189:0/919828083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3fc06cbf0 0x7fb3fc146680 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000e9a0 tx=0x7fb3f000ed60 comp rx=0 tx=0).stop 2024-04-06T13:50:01.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.149+0000 7fb400d83700 1 -- 172.21.15.189:0/919828083 shutdown_connections 2024-04-06T13:50:01.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.149+0000 7fb400d83700 1 --2- 172.21.15.189:0/919828083 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb3dc06adb0 0x7fb3dc06d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:01.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.149+0000 7fb400d83700 1 --2- 172.21.15.189:0/919828083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3fc14fce0 0x7fb3fc159380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:01.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.149+0000 7fb400d83700 1 --2- 172.21.15.189:0/919828083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3fc06cbf0 0x7fb3fc146680 unknown :-1 s=CLOSED pgs=3111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:01.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.149+0000 7fb400d83700 1 --2- 172.21.15.189:0/919828083 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3fc15d040 0x7fb3fc15b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:01.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.149+0000 7fb400d83700 1 -- 172.21.15.189:0/919828083 >> 172.21.15.189:0/919828083 conn(0x7fb3fc06a730 msgr2=0x7fb3fc071e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:50:01.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.149+0000 7fb400d83700 1 -- 172.21.15.189:0/919828083 shutdown_connections 2024-04-06T13:50:01.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.149+0000 7fb400d83700 1 -- 172.21.15.189:0/919828083 wait complete. 2024-04-06T13:50:01.162 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-06T13:50:01.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.286+0000 7f5ad97fe700 1 Processor -- start 2024-04-06T13:50:01.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.286+0000 7f5ad97fe700 1 -- start start 2024-04-06T13:50:01.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.287+0000 7f5ad97fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5ad40cfa00 0x7f5ad40cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:01.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.287+0000 7f5ad97fe700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad40cbd80 con 0x7f5ad40d11b0 2024-04-06T13:50:01.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.287+0000 7f5ad97fe700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad40cef20 con 0x7f5ad40cb870 2024-04-06T13:50:01.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.287+0000 7f5ad97fe700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad40ccdc0 con 0x7f5ad40cfa00 2024-04-06T13:50:01.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.287+0000 7f5ad2ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5ad40d11b0 0x7f5ad40d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39414/0 (socket says 172.21.15.189:39414) 2024-04-06T13:50:01.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.287+0000 7f5ad2ffd700 1 -- 172.21.15.189:0/4019664753 learned_addr learned my addr 172.21.15.189:0/4019664753 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:50:01.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.287+0000 7f5ad37fe700 1 --2- 172.21.15.189:0/4019664753 >> v2:172.21.15.189:3300/0 conn(0x7f5ad40cfa00 0x7f5ad40cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:01.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.287+0000 7f5ad1ffb700 1 -- 172.21.15.189:0/4019664753 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223065260 0 0) 0x7f5ad40cbd80 con 0x7f5ad40d11b0 2024-04-06T13:50:01.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.288+0000 7f5ad1ffb700 1 -- 172.21.15.189:0/4019664753 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5abc0035b0 con 0x7f5ad40d11b0 2024-04-06T13:50:01.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.288+0000 7f5ad37fe700 1 -- 172.21.15.189:0/4019664753 >> v1:172.21.15.192:6789/0 conn(0x7f5ad40cb870 legacy=0x7f5ad415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:01.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.288+0000 7f5ad1ffb700 1 -- 172.21.15.189:0/4019664753 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98653705 0 0) 0x7f5ad40cef20 con 0x7f5ad40cb870 2024-04-06T13:50:01.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.288+0000 7f5ad37fe700 1 -- 172.21.15.189:0/4019664753 >> v1:172.21.15.189:6789/0 conn(0x7f5ad40d11b0 legacy=0x7f5ad40d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:01.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.288+0000 7f5ad37fe700 1 -- 172.21.15.189:0/4019664753 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad40cf200 con 0x7f5ad40cfa00 2024-04-06T13:50:01.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.288+0000 7f5ad37fe700 1 --2- 172.21.15.189:0/4019664753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad40cfa00 0x7f5ad40cd560 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 crypto rx=0x7f5acc009d90 tx=0x7f5acc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc5856a7a5e4d98b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:50:01.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.288+0000 7f5ad1ffb700 1 -- 172.21.15.189:0/4019664753 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5acc00e070 con 0x7f5ad40cfa00 2024-04-06T13:50:01.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.288+0000 7f5ad1ffb700 1 -- 172.21.15.189:0/4019664753 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5acc00bb20 con 0x7f5ad40cfa00 2024-04-06T13:50:01.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.288+0000 7f5ad1ffb700 1 -- 172.21.15.189:0/4019664753 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5acc00be90 con 0x7f5ad40cfa00 2024-04-06T13:50:01.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.288+0000 7f5ad97fe700 1 -- 172.21.15.189:0/4019664753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad40cfa00 msgr2=0x7f5ad40cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:01.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.288+0000 7f5ad97fe700 1 --2- 172.21.15.189:0/4019664753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad40cfa00 0x7f5ad40cd560 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 crypto rx=0x7f5acc009d90 tx=0x7f5acc00b330 comp rx=0 tx=0).stop 2024-04-06T13:50:01.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.288+0000 7f5ad97fe700 1 -- 172.21.15.189:0/4019664753 shutdown_connections 2024-04-06T13:50:01.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.288+0000 7f5ad97fe700 1 --2- 172.21.15.189:0/4019664753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad40cfa00 0x7f5ad40cd560 unknown :-1 s=CLOSED pgs=3113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:01.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.288+0000 7f5ad97fe700 1 -- 172.21.15.189:0/4019664753 >> 172.21.15.189:0/4019664753 conn(0x7f5ad40b2db0 msgr2=0x7f5ad40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:50:01.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.288+0000 7f5ad97fe700 1 -- 172.21.15.189:0/4019664753 shutdown_connections 2024-04-06T13:50:01.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.289+0000 7f5ad97fe700 1 -- 172.21.15.189:0/4019664753 wait complete. 2024-04-06T13:50:01.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.290+0000 7f5ad97fe700 1 Processor -- start 2024-04-06T13:50:01.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.290+0000 7f5ad97fe700 1 -- start start 2024-04-06T13:50:01.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad97fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ad40cb870 0x7f5ad41553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:01.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad97fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad40cfa00 0x7f5ad406cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:01.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad97fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5ad40d11b0 0x7f5ad4159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:01.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad97fe700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad40c8f50 con 0x7f5ad40cfa00 2024-04-06T13:50:01.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad97fe700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad40c82f0 con 0x7f5ad40cb870 2024-04-06T13:50:01.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad97fe700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad40c85b0 con 0x7f5ad40d11b0 2024-04-06T13:50:01.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad27fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad40cfa00 0x7f5ad406cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:01.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad27fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad40cfa00 0x7f5ad406cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35230/0 (socket says 172.21.15.189:35230) 2024-04-06T13:50:01.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad27fc700 1 -- 172.21.15.189:0/1587677158 learned_addr learned my addr 172.21.15.189:0/1587677158 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:50:01.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad37fe700 1 --2- 172.21.15.189:0/1587677158 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5ad40d11b0 0x7f5ad4159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:01.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad2ffd700 1 --2- 172.21.15.189:0/1587677158 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ad40cb870 0x7f5ad41553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:01.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad27fc700 1 -- 172.21.15.189:0/1587677158 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5ad40d11b0 msgr2=0x7f5ad4159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:01.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad27fc700 1 --2- 172.21.15.189:0/1587677158 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5ad40d11b0 0x7f5ad4159f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:01.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad27fc700 1 -- 172.21.15.189:0/1587677158 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ad40cb870 msgr2=0x7f5ad41553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:01.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad27fc700 1 --2- 172.21.15.189:0/1587677158 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ad40cb870 0x7f5ad41553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:01.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad27fc700 1 -- 172.21.15.189:0/1587677158 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5acc009a70 con 0x7f5ad40cfa00 2024-04-06T13:50:01.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad37fe700 1 --2- 172.21.15.189:0/1587677158 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5ad40d11b0 0x7f5ad4159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:50:01.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad2ffd700 1 --2- 172.21.15.189:0/1587677158 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ad40cb870 0x7f5ad41553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:50:01.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad27fc700 1 --2- 172.21.15.189:0/1587677158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad40cfa00 0x7f5ad406cad0 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8001eb0 tx=0x7f5ac800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:50:01.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5abb7fe700 1 -- 172.21.15.189:0/1587677158 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac80054e0 con 0x7f5ad40cfa00 2024-04-06T13:50:01.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5abb7fe700 1 -- 172.21.15.189:0/1587677158 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac8003680 con 0x7f5ad40cfa00 2024-04-06T13:50:01.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad97fe700 1 -- 172.21.15.189:0/1587677158 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad40c8890 con 0x7f5ad40cfa00 2024-04-06T13:50:01.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5abb7fe700 1 -- 172.21.15.189:0/1587677158 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8003a30 con 0x7f5ad40cfa00 2024-04-06T13:50:01.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.291+0000 7f5ad97fe700 1 -- 172.21.15.189:0/1587677158 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad40c9750 con 0x7f5ad40cfa00 2024-04-06T13:50:01.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.293+0000 7f5ad97fe700 1 -- 172.21.15.189:0/1587677158 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad40c9970 con 0x7f5ad40cfa00 2024-04-06T13:50:01.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.293+0000 7f5abb7fe700 1 -- 172.21.15.189:0/1587677158 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 169) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f5ac8003b90 con 0x7f5ad40cfa00 2024-04-06T13:50:01.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.293+0000 7f5abb7fe700 1 --2- 172.21.15.189:0/1587677158 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5ac006adb0 0x7f5ac006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:01.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.294+0000 7f5abb7fe700 1 -- 172.21.15.189:0/1587677158 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5ac8088aa0 con 0x7f5ad40cfa00 2024-04-06T13:50:01.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.294+0000 7f5ad2ffd700 1 --2- 172.21.15.189:0/1587677158 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5ac006adb0 0x7f5ac006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:01.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.294+0000 7f5ad2ffd700 1 --2- 172.21.15.189:0/1587677158 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5ac006adb0 0x7f5ac006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4005910 tx=0x7f5ac4009040 comp rx=0 tx=0).ready entity=mgr.13152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:50:01.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.298+0000 7f5abb7fe700 1 -- 172.21.15.189:0/1587677158 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5ad40c9970 con 0x7f5ad40cfa00 2024-04-06T13:50:01.551 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.551+0000 7f5ad97fe700 1 -- 172.21.15.189:0/1587677158 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f5ad40c9090 con 0x7f5ad40cfa00 2024-04-06T13:50:01.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.733+0000 7f5abb7fe700 1 -- 172.21.15.189:0/1587677158 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 170) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f5ac8053dc0 con 0x7f5ad40cfa00 2024-04-06T13:50:01.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.734+0000 7f5abb7fe700 1 -- 172.21.15.189:0/1587677158 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v170) v1 ==== 92+0+0 (secure 0 0 0) 0x7f5ac8083dd0 con 0x7f5ad40cfa00 2024-04-06T13:50:01.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.736+0000 7f5ad97fe700 1 -- 172.21.15.189:0/1587677158 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5ac006adb0 msgr2=0x7f5ac006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:01.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.736+0000 7f5ad97fe700 1 --2- 172.21.15.189:0/1587677158 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5ac006adb0 0x7f5ac006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4005910 tx=0x7f5ac4009040 comp rx=0 tx=0).stop 2024-04-06T13:50:01.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.736+0000 7f5ad97fe700 1 -- 172.21.15.189:0/1587677158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad40cfa00 msgr2=0x7f5ad406cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:01.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.736+0000 7f5ad97fe700 1 --2- 172.21.15.189:0/1587677158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad40cfa00 0x7f5ad406cad0 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8001eb0 tx=0x7f5ac800c7e0 comp rx=0 tx=0).stop 2024-04-06T13:50:01.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.736+0000 7f5ad97fe700 1 -- 172.21.15.189:0/1587677158 shutdown_connections 2024-04-06T13:50:01.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.736+0000 7f5ad97fe700 1 --2- 172.21.15.189:0/1587677158 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5ac006adb0 0x7f5ac006d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:01.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.736+0000 7f5ad97fe700 1 --2- 172.21.15.189:0/1587677158 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5ad40d11b0 0x7f5ad4159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:01.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.736+0000 7f5ad97fe700 1 --2- 172.21.15.189:0/1587677158 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ad40cb870 0x7f5ad41553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:01.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.736+0000 7f5ad97fe700 1 --2- 172.21.15.189:0/1587677158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ad40cfa00 0x7f5ad406cad0 unknown :-1 s=CLOSED pgs=3114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:01.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.736+0000 7f5ad97fe700 1 -- 172.21.15.189:0/1587677158 >> 172.21.15.189:0/1587677158 conn(0x7f5ad40b2db0 msgr2=0x7f5ad40d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:50:01.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.736+0000 7f5ad97fe700 1 -- 172.21.15.189:0/1587677158 shutdown_connections 2024-04-06T13:50:01.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.737+0000 7f5ad97fe700 1 -- 172.21.15.189:0/1587677158 wait complete. 2024-04-06T13:50:01.750 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-06T13:50:01.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.881+0000 7fbd398ad700 1 Processor -- start 2024-04-06T13:50:01.883 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.881+0000 7fbd398ad700 1 -- start start 2024-04-06T13:50:01.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.882+0000 7fbd398ad700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd3415d7b0 0x7fbd340af040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:01.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.882+0000 7fbd398ad700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbd340b39d0 0x7fbd340ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:01.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.882+0000 7fbd398ad700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd34071950 con 0x7fbd340ace00 2024-04-06T13:50:01.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.882+0000 7fbd398ad700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd340b4170 con 0x7fbd340b39d0 2024-04-06T13:50:01.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.882+0000 7fbd398ad700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd340b3e90 con 0x7fbd3415d7b0 2024-04-06T13:50:01.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.882+0000 7fbd337fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbd340ace00 0x7fbd340723f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39428/0 (socket says 172.21.15.189:39428) 2024-04-06T13:50:01.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.882+0000 7fbd337fe700 1 -- 172.21.15.189:0/3628992349 learned_addr learned my addr 172.21.15.189:0/3628992349 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:50:01.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.882+0000 7fbd327fc700 1 --2- 172.21.15.189:0/3628992349 >> v2:172.21.15.192:3300/0 conn(0x7fbd340b39d0 0x7fbd340ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:01.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.882+0000 7fbd32ffd700 1 --2- 172.21.15.189:0/3628992349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd3415d7b0 0x7fbd340af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:01.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.882+0000 7fbd31ffb700 1 -- 172.21.15.189:0/3628992349 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475598931 0 0) 0x7fbd34071950 con 0x7fbd340ace00 2024-04-06T13:50:01.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.882+0000 7fbd31ffb700 1 -- 172.21.15.189:0/3628992349 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd1c0035b0 con 0x7fbd340ace00 2024-04-06T13:50:01.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.882+0000 7fbd32ffd700 1 -- 172.21.15.189:0/3628992349 >> v2:172.21.15.192:3300/0 conn(0x7fbd340b39d0 msgr2=0x7fbd340ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:50:01.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.882+0000 7fbd32ffd700 1 --2- 172.21.15.189:0/3628992349 >> v2:172.21.15.192:3300/0 conn(0x7fbd340b39d0 0x7fbd340ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:01.884 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.882+0000 7fbd32ffd700 1 -- 172.21.15.189:0/3628992349 >> v1:172.21.15.189:6789/0 conn(0x7fbd340ace00 legacy=0x7fbd340723f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:01.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.882+0000 7fbd32ffd700 1 -- 172.21.15.189:0/3628992349 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd34071bf0 con 0x7fbd3415d7b0 2024-04-06T13:50:01.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.883+0000 7fbd32ffd700 1 --2- 172.21.15.189:0/3628992349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd3415d7b0 0x7fbd340af040 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbd24009840 tx=0x7fbd24009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a483f6df158f297 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:50:01.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.883+0000 7fbd31ffb700 1 -- 172.21.15.189:0/3628992349 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd24011830 con 0x7fbd3415d7b0 2024-04-06T13:50:01.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.883+0000 7fbd31ffb700 1 -- 172.21.15.189:0/3628992349 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd24011990 con 0x7fbd3415d7b0 2024-04-06T13:50:01.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.883+0000 7fbd31ffb700 1 -- 172.21.15.189:0/3628992349 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd24011d20 con 0x7fbd3415d7b0 2024-04-06T13:50:01.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.883+0000 7fbd398ad700 1 -- 172.21.15.189:0/3628992349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd3415d7b0 msgr2=0x7fbd340af040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:01.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.883+0000 7fbd398ad700 1 --2- 172.21.15.189:0/3628992349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd3415d7b0 0x7fbd340af040 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbd24009840 tx=0x7fbd24009b50 comp rx=0 tx=0).stop 2024-04-06T13:50:01.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.883+0000 7fbd398ad700 1 -- 172.21.15.189:0/3628992349 shutdown_connections 2024-04-06T13:50:01.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.883+0000 7fbd398ad700 1 --2- 172.21.15.189:0/3628992349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd3415d7b0 0x7fbd340af040 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:01.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.883+0000 7fbd398ad700 1 --2- 172.21.15.189:0/3628992349 >> v2:172.21.15.192:3300/0 conn(0x7fbd340b39d0 0x7fbd340ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:01.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.883+0000 7fbd398ad700 1 -- 172.21.15.189:0/3628992349 >> 172.21.15.189:0/3628992349 conn(0x7fbd340d00b0 msgr2=0x7fbd340d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:50:01.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.883+0000 7fbd398ad700 1 -- 172.21.15.189:0/3628992349 shutdown_connections 2024-04-06T13:50:01.885 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.883+0000 7fbd398ad700 1 -- 172.21.15.189:0/3628992349 wait complete. 2024-04-06T13:50:01.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.885+0000 7fbd398ad700 1 Processor -- start 2024-04-06T13:50:01.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.885+0000 7fbd398ad700 1 -- start start 2024-04-06T13:50:01.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.885+0000 7fbd398ad700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd340ace00 0x7fbd341f5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:01.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.885+0000 7fbd398ad700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd340b39d0 0x7fbd3415baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:01.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.885+0000 7fbd398ad700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd3415bfe0 0x7fbd340cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:01.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.885+0000 7fbd398ad700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd341580e0 con 0x7fbd340ace00 2024-04-06T13:50:01.889 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.885+0000 7fbd398ad700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3415c490 con 0x7fbd340b39d0 2024-04-06T13:50:01.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.885+0000 7fbd398ad700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd341f6130 con 0x7fbd3415bfe0 2024-04-06T13:50:01.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.886+0000 7fbd32ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd340ace00 0x7fbd341f5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:01.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.886+0000 7fbd32ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd340ace00 0x7fbd341f5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35246/0 (socket says 172.21.15.189:35246) 2024-04-06T13:50:01.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.886+0000 7fbd32ffd700 1 -- 172.21.15.189:0/1188422287 learned_addr learned my addr 172.21.15.189:0/1188422287 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:50:01.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.886+0000 7fbd327fc700 1 --2- 172.21.15.189:0/1188422287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd340b39d0 0x7fbd3415baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:01.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.886+0000 7fbd337fe700 1 --2- 172.21.15.189:0/1188422287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd3415bfe0 0x7fbd340cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:01.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.886+0000 7fbd32ffd700 1 -- 172.21.15.189:0/1188422287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd3415bfe0 msgr2=0x7fbd340cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:01.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.886+0000 7fbd32ffd700 1 --2- 172.21.15.189:0/1188422287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd3415bfe0 0x7fbd340cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:01.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.886+0000 7fbd32ffd700 1 -- 172.21.15.189:0/1188422287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd340b39d0 msgr2=0x7fbd3415baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:01.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.886+0000 7fbd32ffd700 1 --2- 172.21.15.189:0/1188422287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd340b39d0 0x7fbd3415baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:01.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.886+0000 7fbd32ffd700 1 -- 172.21.15.189:0/1188422287 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd240094a0 con 0x7fbd340ace00 2024-04-06T13:50:01.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.886+0000 7fbd32ffd700 1 --2- 172.21.15.189:0/1188422287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd340ace00 0x7fbd341f5bf0 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 crypto rx=0x7fbd240090b0 tx=0x7fbd24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:50:01.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.886+0000 7fbd1bfff700 1 -- 172.21.15.189:0/1188422287 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd24004590 con 0x7fbd340ace00 2024-04-06T13:50:01.890 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.886+0000 7fbd1bfff700 1 -- 172.21.15.189:0/1188422287 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2401f070 con 0x7fbd340ace00 2024-04-06T13:50:01.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.886+0000 7fbd398ad700 1 -- 172.21.15.189:0/1188422287 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd341f4200 con 0x7fbd340ace00 2024-04-06T13:50:01.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.886+0000 7fbd1bfff700 1 -- 172.21.15.189:0/1188422287 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd240038e0 con 0x7fbd340ace00 2024-04-06T13:50:01.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.886+0000 7fbd398ad700 1 -- 172.21.15.189:0/1188422287 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd340cd370 con 0x7fbd340ace00 2024-04-06T13:50:01.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.887+0000 7fbd398ad700 1 -- 172.21.15.189:0/1188422287 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd34067bb0 con 0x7fbd340ace00 2024-04-06T13:50:01.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.888+0000 7fbd1bfff700 1 -- 172.21.15.189:0/1188422287 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 170) v1 ==== 89264+0+0 (secure 0 0 0) 0x7fbd24003a40 con 0x7fbd340ace00 2024-04-06T13:50:01.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.888+0000 7fbd1bfff700 1 --2- 172.21.15.189:0/1188422287 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbd2006ae00 0x7fbd2006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:01.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.888+0000 7fbd327fc700 1 -- 172.21.15.189:0/1188422287 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbd2006ae00 msgr2=0x7fbd2006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:50:01.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.888+0000 7fbd1bfff700 1 -- 172.21.15.189:0/1188422287 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbd24021070 con 0x7fbd340ace00 2024-04-06T13:50:01.891 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.888+0000 7fbd327fc700 1 --2- 172.21.15.189:0/1188422287 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbd2006ae00 0x7fbd2006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:50:01.893 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:01.893+0000 7fbd1bfff700 1 -- 172.21.15.189:0/1188422287 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbd24090020 con 0x7fbd340ace00 2024-04-06T13:50:02.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.089+0000 7fbd327fc700 1 -- 172.21.15.189:0/1188422287 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbd2006ae00 msgr2=0x7fbd2006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:50:02.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.089+0000 7fbd327fc700 1 --2- 172.21.15.189:0/1188422287 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbd2006ae00 0x7fbd2006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:50:02.097 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.097+0000 7fbd398ad700 1 -- 172.21.15.189:0/1188422287 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fbd340d0e50 con 0x7fbd340ace00 2024-04-06T13:50:02.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.100+0000 7fbd1bfff700 1 -- 172.21.15.189:0/1188422287 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v95)=0 v95) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbd240150d0 con 0x7fbd340ace00 2024-04-06T13:50:02.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.103+0000 7fbd398ad700 1 -- 172.21.15.189:0/1188422287 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbd2006ae00 msgr2=0x7fbd2006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:50:02.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.103+0000 7fbd398ad700 1 --2- 172.21.15.189:0/1188422287 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbd2006ae00 0x7fbd2006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.103+0000 7fbd398ad700 1 -- 172.21.15.189:0/1188422287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd340ace00 msgr2=0x7fbd341f5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:02.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.103+0000 7fbd398ad700 1 --2- 172.21.15.189:0/1188422287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd340ace00 0x7fbd341f5bf0 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 crypto rx=0x7fbd240090b0 tx=0x7fbd24004750 comp rx=0 tx=0).stop 2024-04-06T13:50:02.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.103+0000 7fbd398ad700 1 -- 172.21.15.189:0/1188422287 shutdown_connections 2024-04-06T13:50:02.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.103+0000 7fbd398ad700 1 --2- 172.21.15.189:0/1188422287 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbd2006ae00 0x7fbd2006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.103+0000 7fbd398ad700 1 --2- 172.21.15.189:0/1188422287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd3415bfe0 0x7fbd340cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.103+0000 7fbd398ad700 1 --2- 172.21.15.189:0/1188422287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd340ace00 0x7fbd341f5bf0 unknown :-1 s=CLOSED pgs=3119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.103+0000 7fbd398ad700 1 --2- 172.21.15.189:0/1188422287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd340b39d0 0x7fbd3415baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.103+0000 7fbd398ad700 1 -- 172.21.15.189:0/1188422287 >> 172.21.15.189:0/1188422287 conn(0x7fbd340d00b0 msgr2=0x7fbd340d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:50:02.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.103+0000 7fbd398ad700 1 -- 172.21.15.189:0/1188422287 shutdown_connections 2024-04-06T13:50:02.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.103+0000 7fbd398ad700 1 -- 172.21.15.189:0/1188422287 wait complete. 2024-04-06T13:50:02.114 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:02.114+0000 7f9d07faa200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:50:02.116 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-06T13:50:02.137 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:02.137+0000 7f9c831ed200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:50:02.192 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:02.192+0000 7f9d07faa200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:50:02.220 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:02.220+0000 7f9c831ed200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:50:02.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.243+0000 7f9f0f9c0700 1 Processor -- start 2024-04-06T13:50:02.244 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.244+0000 7f9f0f9c0700 1 -- start start 2024-04-06T13:50:02.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.245+0000 7f9f0f9c0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9f080b3e20 0x7f9f0815e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:02.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.245+0000 7f9f0f9c0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f080b3990 con 0x7f9f0806ae40 2024-04-06T13:50:02.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.245+0000 7f9f0f9c0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f080b3060 con 0x7f9f0815d7b0 2024-04-06T13:50:02.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.245+0000 7f9f0f9c0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f080b4230 con 0x7f9f080b3e20 2024-04-06T13:50:02.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.245+0000 7f9f0df5d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9f080b3e20 0x7f9f0815e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:02.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.245+0000 7f9f0cf5b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9f0806ae40 0x7f9f0815ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39442/0 (socket says 172.21.15.189:39442) 2024-04-06T13:50:02.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.245+0000 7f9f0cf5b700 1 -- 172.21.15.189:0/2361140241 learned_addr learned my addr 172.21.15.189:0/2361140241 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:50:02.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.245+0000 7f9f0df5d700 1 -- 172.21.15.189:0/2361140241 >> v1:172.21.15.192:6789/0 conn(0x7f9f0815d7b0 legacy=0x7f9f0806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:02.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.245+0000 7f9f0df5d700 1 -- 172.21.15.189:0/2361140241 >> v1:172.21.15.189:6789/0 conn(0x7f9f0806ae40 legacy=0x7f9f0815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:02.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.245+0000 7f9f0df5d700 1 -- 172.21.15.189:0/2361140241 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f080b3340 con 0x7f9f080b3e20 2024-04-06T13:50:02.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.245+0000 7f9efffff700 1 -- 172.21.15.189:0/2361140241 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891461875 0 0) 0x7f9f080b3990 con 0x7f9f0806ae40 2024-04-06T13:50:02.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.245+0000 7f9f0df5d700 1 --2- 172.21.15.189:0/2361140241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f080b3e20 0x7f9f0815e500 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 crypto rx=0x7f9f00009d90 tx=0x7f9f0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e1e37724b1c5134 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:50:02.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.245+0000 7f9efffff700 1 -- 172.21.15.189:0/2361140241 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0000e070 con 0x7f9f080b3e20 2024-04-06T13:50:02.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.245+0000 7f9efffff700 1 -- 172.21.15.189:0/2361140241 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0000bbd0 con 0x7f9f080b3e20 2024-04-06T13:50:02.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.245+0000 7f9efffff700 1 -- 172.21.15.189:0/2361140241 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f00012430 con 0x7f9f080b3e20 2024-04-06T13:50:02.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.246+0000 7f9f0f9c0700 1 -- 172.21.15.189:0/2361140241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f080b3e20 msgr2=0x7f9f0815e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:02.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.246+0000 7f9f0f9c0700 1 --2- 172.21.15.189:0/2361140241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f080b3e20 0x7f9f0815e500 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 crypto rx=0x7f9f00009d90 tx=0x7f9f0000b330 comp rx=0 tx=0).stop 2024-04-06T13:50:02.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.246+0000 7f9f0f9c0700 1 -- 172.21.15.189:0/2361140241 shutdown_connections 2024-04-06T13:50:02.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.246+0000 7f9f0f9c0700 1 --2- 172.21.15.189:0/2361140241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f080b3e20 0x7f9f0815e500 unknown :-1 s=CLOSED pgs=3121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.246+0000 7f9f0f9c0700 1 -- 172.21.15.189:0/2361140241 >> 172.21.15.189:0/2361140241 conn(0x7f9f080d00b0 msgr2=0x7f9f080d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:50:02.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.246+0000 7f9f0f9c0700 1 -- 172.21.15.189:0/2361140241 shutdown_connections 2024-04-06T13:50:02.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.246+0000 7f9f0f9c0700 1 -- 172.21.15.189:0/2361140241 wait complete. 2024-04-06T13:50:02.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.248+0000 7f9f0f9c0700 1 Processor -- start 2024-04-06T13:50:02.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.248+0000 7f9f0f9c0700 1 -- start start 2024-04-06T13:50:02.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.248+0000 7f9f0f9c0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f0806ae40 0x7f9f081f62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:02.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9f0f9c0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f080b3e20 0x7f9f081f2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:02.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9f0f9c0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f0815d7b0 0x7f9f0815b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:02.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9f0f9c0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f081f6be0 con 0x7f9f080b3e20 2024-04-06T13:50:02.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9f0f9c0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f081f6ec0 con 0x7f9f0815d7b0 2024-04-06T13:50:02.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9f0f9c0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f081f71a0 con 0x7f9f0806ae40 2024-04-06T13:50:02.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9f0cf5b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f080b3e20 0x7f9f081f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:02.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9f0cf5b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f080b3e20 0x7f9f081f2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35266/0 (socket says 172.21.15.189:35266) 2024-04-06T13:50:02.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9f0cf5b700 1 -- 172.21.15.189:0/1593372176 learned_addr learned my addr 172.21.15.189:0/1593372176 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:50:02.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9f0df5d700 1 --2- 172.21.15.189:0/1593372176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f0815d7b0 0x7f9f0815b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:02.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9f0d75c700 1 --2- 172.21.15.189:0/1593372176 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f0806ae40 0x7f9f081f62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:02.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9f0cf5b700 1 -- 172.21.15.189:0/1593372176 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f0806ae40 msgr2=0x7f9f081f62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:02.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9f0cf5b700 1 --2- 172.21.15.189:0/1593372176 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f0806ae40 0x7f9f081f62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9f0cf5b700 1 -- 172.21.15.189:0/1593372176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f0815d7b0 msgr2=0x7f9f0815b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:02.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9f0cf5b700 1 --2- 172.21.15.189:0/1593372176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f0815d7b0 0x7f9f0815b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9f0cf5b700 1 -- 172.21.15.189:0/1593372176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f00009a70 con 0x7f9f080b3e20 2024-04-06T13:50:02.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9f0cf5b700 1 --2- 172.21.15.189:0/1593372176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f080b3e20 0x7f9f081f2510 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800c520 tx=0x7f9ef800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:50:02.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9efdffb700 1 -- 172.21.15.189:0/1593372176 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8004c80 con 0x7f9f080b3e20 2024-04-06T13:50:02.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9efdffb700 1 -- 172.21.15.189:0/1593372176 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef8004de0 con 0x7f9f080b3e20 2024-04-06T13:50:02.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9f0f9c0700 1 -- 172.21.15.189:0/1593372176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f080cc5e0 con 0x7f9f080b3e20 2024-04-06T13:50:02.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9efdffb700 1 -- 172.21.15.189:0/1593372176 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8007310 con 0x7f9f080b3e20 2024-04-06T13:50:02.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.249+0000 7f9f0f9c0700 1 -- 172.21.15.189:0/1593372176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f081f6820 con 0x7f9f080b3e20 2024-04-06T13:50:02.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.251+0000 7f9f0f9c0700 1 -- 172.21.15.189:0/1593372176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f080ccb00 con 0x7f9f080b3e20 2024-04-06T13:50:02.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.251+0000 7f9efdffb700 1 -- 172.21.15.189:0/1593372176 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 170) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f9ef8007470 con 0x7f9f080b3e20 2024-04-06T13:50:02.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.251+0000 7f9efdffb700 1 --2- 172.21.15.189:0/1593372176 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9eec06ae00 0x7f9eec06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:02.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.251+0000 7f9f0d75c700 1 -- 172.21.15.189:0/1593372176 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9eec06ae00 msgr2=0x7f9eec06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:50:02.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.251+0000 7f9efdffb700 1 -- 172.21.15.189:0/1593372176 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9ef8089260 con 0x7f9f080b3e20 2024-04-06T13:50:02.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.251+0000 7f9f0d75c700 1 --2- 172.21.15.189:0/1593372176 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9eec06ae00 0x7f9eec06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:50:02.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.256+0000 7f9efdffb700 1 -- 172.21.15.189:0/1593372176 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9f080ccb00 con 0x7f9f080b3e20 2024-04-06T13:50:02.265 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:02.265+0000 7f9d07faa200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:50:02.298 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:02.298+0000 7f9c831ed200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:50:02.333 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:02.333+0000 7f9d07faa200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:50:02.369 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:02.369+0000 7f9c831ed200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:50:02.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.452+0000 7f9f0d75c700 1 -- 172.21.15.189:0/1593372176 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9eec06ae00 msgr2=0x7f9eec06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:50:02.452 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.452+0000 7f9f0d75c700 1 --2- 172.21.15.189:0/1593372176 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9eec06ae00 0x7f9eec06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:50:02.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.460+0000 7f9f0f9c0700 1 -- 172.21.15.189:0/1593372176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f9f080ccb00 con 0x7f9f080b3e20 2024-04-06T13:50:02.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.463+0000 7f9efdffb700 1 -- 172.21.15.189:0/1593372176 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v96)=0 v96) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9f080ccb00 con 0x7f9f080b3e20 2024-04-06T13:50:02.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.466+0000 7f9f0f9c0700 1 -- 172.21.15.189:0/1593372176 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9eec06ae00 msgr2=0x7f9eec06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:50:02.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.466+0000 7f9f0f9c0700 1 --2- 172.21.15.189:0/1593372176 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9eec06ae00 0x7f9eec06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.466+0000 7f9f0f9c0700 1 -- 172.21.15.189:0/1593372176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f080b3e20 msgr2=0x7f9f081f2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:02.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.466+0000 7f9f0f9c0700 1 --2- 172.21.15.189:0/1593372176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f080b3e20 0x7f9f081f2510 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800c520 tx=0x7f9ef800c830 comp rx=0 tx=0).stop 2024-04-06T13:50:02.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.466+0000 7f9f0f9c0700 1 -- 172.21.15.189:0/1593372176 shutdown_connections 2024-04-06T13:50:02.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.466+0000 7f9f0f9c0700 1 --2- 172.21.15.189:0/1593372176 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f9eec06ae00 0x7f9eec06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.466+0000 7f9f0f9c0700 1 --2- 172.21.15.189:0/1593372176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f0815d7b0 0x7f9f0815b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.466+0000 7f9f0f9c0700 1 --2- 172.21.15.189:0/1593372176 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f0806ae40 0x7f9f081f62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.466 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.466+0000 7f9f0f9c0700 1 --2- 172.21.15.189:0/1593372176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f080b3e20 0x7f9f081f2510 unknown :-1 s=CLOSED pgs=3122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.466+0000 7f9f0f9c0700 1 -- 172.21.15.189:0/1593372176 >> 172.21.15.189:0/1593372176 conn(0x7f9f080d00b0 msgr2=0x7f9f08071cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:50:02.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.466+0000 7f9f0f9c0700 1 -- 172.21.15.189:0/1593372176 shutdown_connections 2024-04-06T13:50:02.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.466+0000 7f9f0f9c0700 1 -- 172.21.15.189:0/1593372176 wait complete. 2024-04-06T13:50:02.479 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv01 2024-04-06T13:50:02.509 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:02.509+0000 7f9d07faa200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:50:02.551 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:02.551+0000 7f9c831ed200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:50:02.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.648+0000 7f42c8178700 1 Processor -- start 2024-04-06T13:50:02.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.648+0000 7f42c8178700 1 -- start start 2024-04-06T13:50:02.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.649+0000 7f42c8178700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42c00d17a0 0x7f42c006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:02.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.649+0000 7f42c8178700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f42c0146dd0 0x7f42c006d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:02.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.649+0000 7f42c8178700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f42c00cedf0 0x7f42c00cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:02.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.649+0000 7f42c8178700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42c00cf760 con 0x7f42c0146dd0 2024-04-06T13:50:02.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.649+0000 7f42c8178700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42c006c0a0 con 0x7f42c00cedf0 2024-04-06T13:50:02.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.649+0000 7f42c8178700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c006c380 con 0x7f42c00d17a0 2024-04-06T13:50:02.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.649+0000 7f42c5713700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f42c0146dd0 0x7f42c006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:02.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.649+0000 7f42c5713700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f42c0146dd0 0x7f42c006d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35274/0 (socket says 172.21.15.189:35274) 2024-04-06T13:50:02.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.649+0000 7f42c5713700 1 -- 172.21.15.189:0/3591397611 learned_addr learned my addr 172.21.15.189:0/3591397611 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:50:02.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.649+0000 7f42c5f14700 1 --2- 172.21.15.189:0/3591397611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42c00d17a0 0x7f42c006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:02.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.649+0000 7f42c6715700 1 --2- 172.21.15.189:0/3591397611 >> v2:172.21.15.192:3300/0 conn(0x7f42c00cedf0 0x7f42c00cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:02.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.650+0000 7f42c5713700 1 -- 172.21.15.189:0/3591397611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42c00d17a0 msgr2=0x7f42c006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:50:02.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.650+0000 7f42c5713700 1 --2- 172.21.15.189:0/3591397611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42c00d17a0 0x7f42c006b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.650+0000 7f42c5713700 1 -- 172.21.15.189:0/3591397611 >> v2:172.21.15.192:3300/0 conn(0x7f42c00cedf0 msgr2=0x7f42c00cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:50:02.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.650+0000 7f42c5713700 1 --2- 172.21.15.189:0/3591397611 >> v2:172.21.15.192:3300/0 conn(0x7f42c00cedf0 0x7f42c00cf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.650+0000 7f42c5713700 1 -- 172.21.15.189:0/3591397611 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c006c660 con 0x7f42c0146dd0 2024-04-06T13:50:02.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.650+0000 7f42c6715700 1 --2- 172.21.15.189:0/3591397611 >> v2:172.21.15.192:3300/0 conn(0x7f42c00cedf0 0x7f42c00cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:50:02.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.650+0000 7f42c5713700 1 --2- 172.21.15.189:0/3591397611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42c0146dd0 0x7f42c006d140 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 crypto rx=0x7f42c0162120 tx=0x7f42b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba7739cee90e6f4b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:50:02.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.650+0000 7f42c4f12700 1 -- 172.21.15.189:0/3591397611 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b0011830 con 0x7f42c0146dd0 2024-04-06T13:50:02.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.650+0000 7f42c4f12700 1 -- 172.21.15.189:0/3591397611 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b0011990 con 0x7f42c0146dd0 2024-04-06T13:50:02.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.650+0000 7f42c4f12700 1 -- 172.21.15.189:0/3591397611 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b0011d20 con 0x7f42c0146dd0 2024-04-06T13:50:02.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.650+0000 7f42c8178700 1 -- 172.21.15.189:0/3591397611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42c0146dd0 msgr2=0x7f42c006d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:02.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.650+0000 7f42c8178700 1 --2- 172.21.15.189:0/3591397611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42c0146dd0 0x7f42c006d140 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 crypto rx=0x7f42c0162120 tx=0x7f42b0009bd0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.651+0000 7f42c8178700 1 -- 172.21.15.189:0/3591397611 shutdown_connections 2024-04-06T13:50:02.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.651+0000 7f42c8178700 1 --2- 172.21.15.189:0/3591397611 >> v2:172.21.15.192:3300/0 conn(0x7f42c00cedf0 0x7f42c00cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.651+0000 7f42c8178700 1 --2- 172.21.15.189:0/3591397611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42c00d17a0 0x7f42c006b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.651+0000 7f42c8178700 1 --2- 172.21.15.189:0/3591397611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42c0146dd0 0x7f42c006d140 unknown :-1 s=CLOSED pgs=3123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.651+0000 7f42c8178700 1 -- 172.21.15.189:0/3591397611 >> 172.21.15.189:0/3591397611 conn(0x7f42c00b2db0 msgr2=0x7f42c00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:50:02.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.651+0000 7f42c8178700 1 -- 172.21.15.189:0/3591397611 shutdown_connections 2024-04-06T13:50:02.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.651+0000 7f42c8178700 1 -- 172.21.15.189:0/3591397611 wait complete. 2024-04-06T13:50:02.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.652+0000 7f42c8178700 1 Processor -- start 2024-04-06T13:50:02.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.652+0000 7f42c8178700 1 -- start start 2024-04-06T13:50:02.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.653+0000 7f42c8178700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42c00cedf0 0x7f42c01f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:02.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.653+0000 7f42c8178700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42c00d17a0 0x7f42c00b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:02.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.653+0000 7f42c8178700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42c0146dd0 0x7f42c00acc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:02.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.653+0000 7f42c8178700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c0157930 con 0x7f42c00d17a0 2024-04-06T13:50:02.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.653+0000 7f42c8178700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c00b0610 con 0x7f42c0146dd0 2024-04-06T13:50:02.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.653+0000 7f42c8178700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c01f3f00 con 0x7f42c00cedf0 2024-04-06T13:50:02.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.653+0000 7f42c5713700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42c00d17a0 0x7f42c00b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:02.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.653+0000 7f42c5713700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42c00d17a0 0x7f42c00b16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35284/0 (socket says 172.21.15.189:35284) 2024-04-06T13:50:02.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.653+0000 7f42c5713700 1 -- 172.21.15.189:0/2078133511 learned_addr learned my addr 172.21.15.189:0/2078133511 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:50:02.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.653+0000 7f42c5f14700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42c00cedf0 0x7f42c01f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:02.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.653+0000 7f42c6715700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42c0146dd0 0x7f42c00acc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:02.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.654+0000 7f42c5f14700 1 -- 172.21.15.189:0/2078133511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42c0146dd0 msgr2=0x7f42c00acc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:02.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.654+0000 7f42c5f14700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42c0146dd0 0x7f42c00acc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.654+0000 7f42c5f14700 1 -- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42c00d17a0 msgr2=0x7f42c00b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:02.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.654+0000 7f42c5f14700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42c00d17a0 0x7f42c00b16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:02.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.654+0000 7f42c5f14700 1 -- 172.21.15.189:0/2078133511 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b8009a70 con 0x7f42c00cedf0 2024-04-06T13:50:02.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.654+0000 7f42c5713700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42c00d17a0 0x7f42c00b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:50:02.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.654+0000 7f42c5f14700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42c00cedf0 0x7f42c01f2830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00e870 tx=0x7f42bc00eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:50:02.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.654+0000 7f42b67fc700 1 -- 172.21.15.189:0/2078133511 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc013680 con 0x7f42c00cedf0 2024-04-06T13:50:02.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.654+0000 7f42b67fc700 1 -- 172.21.15.189:0/2078133511 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42bc0137e0 con 0x7f42c00cedf0 2024-04-06T13:50:02.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.654+0000 7f42b67fc700 1 -- 172.21.15.189:0/2078133511 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc013b70 con 0x7f42c00cedf0 2024-04-06T13:50:02.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.654+0000 7f42c8178700 1 -- 172.21.15.189:0/2078133511 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b0009550 con 0x7f42c00cedf0 2024-04-06T13:50:02.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.654+0000 7f42c8178700 1 -- 172.21.15.189:0/2078133511 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c01f38b0 con 0x7f42c00cedf0 2024-04-06T13:50:02.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.655+0000 7f42c8178700 1 -- 172.21.15.189:0/2078133511 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c0074990 con 0x7f42c00cedf0 2024-04-06T13:50:02.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.656+0000 7f42b67fc700 1 -- 172.21.15.189:0/2078133511 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 170) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f42bc013cd0 con 0x7f42c00cedf0 2024-04-06T13:50:02.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.656+0000 7f42b67fc700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42a406ae00 0x7f42a406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:02.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.656+0000 7f42c5713700 1 -- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42a406ae00 msgr2=0x7f42a406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:50:02.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.656+0000 7f42b67fc700 1 -- 172.21.15.189:0/2078133511 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f42bc089370 con 0x7f42c00cedf0 2024-04-06T13:50:02.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.656+0000 7f42c5713700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42a406ae00 0x7f42a406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:50:02.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.660+0000 7f42b67fc700 1 -- 172.21.15.189:0/2078133511 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f42bc054320 con 0x7f42c00cedf0 2024-04-06T13:50:02.668 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:02.668+0000 7f9d07faa200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:50:02.717 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:02.717+0000 7f9c831ed200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:50:02.779 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:02.779+0000 7f9d07faa200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:50:02.831 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:02.831+0000 7f9c831ed200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:50:02.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.857+0000 7f42c5713700 1 -- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42a406ae00 msgr2=0x7f42a406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:50:02.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.857+0000 7f42c5713700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42a406ae00 0x7f42a406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:50:02.860 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:02.860+0000 7f9d07faa200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:50:02.896 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:02.896+0000 7f42c8178700 1 -- 172.21.15.189:0/2078133511 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) v1 -- 0x7f42c00d1b80 con 0x7f42a406ae00 2024-04-06T13:50:02.916 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:02.916+0000 7f9c831ed200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:50:02.934 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:02.934+0000 7f9d07faa200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:50:02.991 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:02.991+0000 7f9c831ed200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:50:03.136 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:03.136+0000 7f9d07faa200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:50:03.195 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:03.195+0000 7f9c831ed200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:50:03.213 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:03.213+0000 7f9d07faa200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:50:03.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:03.258+0000 7f42c5713700 1 -- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42a406ae00 msgr2=0x7f42a406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:50:03.259 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:03.258+0000 7f42c5713700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42a406ae00 0x7f42a406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-06T13:50:03.271 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:03.271+0000 7f9c831ed200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:50:03.274 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:50:03.276 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:50:03.540 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:03.540+0000 7f9d07faa200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:50:03.599 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:03.600+0000 7f9c831ed200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:50:03.858 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:03.858+0000 7f9d07faa200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:50:03.921 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:03.921+0000 7f9c831ed200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:50:04.017 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:04.017+0000 7f9d07faa200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:50:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:04.059+0000 7f42c5713700 1 -- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42a406ae00 msgr2=0x7f42a406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:50:04.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:04.059+0000 7f42c5713700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42a406ae00 0x7f42a406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-06T13:50:04.080 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:04.080+0000 7f9c831ed200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:50:04.089 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:04.089+0000 7f9d07faa200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:50:04.153 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:04.153+0000 7f9c831ed200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:50:04.702 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:04.702+0000 7f9d07faa200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:50:04.771 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:04.771+0000 7f9c831ed200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:50:04.780 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:04.780+0000 7f9d07faa200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:50:04.850 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:04.850+0000 7f9c831ed200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:50:04.905 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:04.905+0000 7f9d07faa200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:50:04.977 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:04.977+0000 7f9c831ed200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:50:05.412 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:05.412+0000 7f9d07faa200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:50:05.492 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:05.492+0000 7f9c831ed200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:50:05.566 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:05.566+0000 7f9d07faa200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:50:05.648 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:05.648+0000 7f9c831ed200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:50:05.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:05.660+0000 7f42c5713700 1 -- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42a406ae00 msgr2=0x7f42a406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:50:05.660 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:05.660+0000 7f42c5713700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42a406ae00 0x7f42a406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-06T13:50:05.722 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:05.722+0000 7f9d07faa200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:50:05.808 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:05.808+0000 7f9c831ed200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:50:06.444 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:06.444+0000 7f9d07faa200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:50:06.538 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:06.538+0000 7f9c831ed200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:50:06.686 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:06.685+0000 7f9d07faa200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:50:06.762 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:06.762+0000 7f9d07faa200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:50:06.784 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:06.784+0000 7f9c831ed200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:50:06.862 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:06.862+0000 7f9c831ed200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:50:07.134 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:50:07.134+0000 7f9d07faa200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:50:07.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:07.205+0000 7f42b67fc700 1 -- 172.21.15.189:0/2078133511 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mgrmap(e 171) v1 ==== 89264+0+0 (secure 0 0 0) 0x7f42bc0546f0 con 0x7f42c00cedf0 2024-04-06T13:50:07.242 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:07.243+0000 7f9c831ed200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:50:07.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:07.268+0000 7f42b67fc700 1 -- 172.21.15.189:0/2078133511 <== mon.2 v2:172.21.15.192:3301/0 8 ==== mgrmap(e 172) v1 ==== 88873+0+0 (secure 0 0 0) 0x7f42bc08a340 con 0x7f42c00cedf0 2024-04-06T13:50:07.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:07.268+0000 7f42b67fc700 1 -- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42a406ae00 msgr2=0x7f42a406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:50:07.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:07.268+0000 7f42b67fc700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42a406ae00 0x7f42a406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:07.399 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:07.399+0000 7f9bf60fb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:50:07.399 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:07.399+0000 7f9bf60fb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:50:07.399 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:07.399+0000 7f9bf60fb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:50:07.399 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:07.399+0000 7f9bf60fb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:50:07.399 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:07.399+0000 7f9bf60fb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:50:07.399 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:07.399+0000 7f9bf9101700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:50:07.399 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:07.399+0000 7f9bf9101700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:50:07.400 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:07.399+0000 7f9bf9101700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:50:07.400 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:07.399+0000 7f9bf9101700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:50:07.400 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:07.399+0000 7f9bf9101700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:50:08.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.269+0000 7f42b67fc700 1 -- 172.21.15.189:0/2078133511 <== mon.2 v2:172.21.15.192:3301/0 9 ==== mgrmap(e 173) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f42bc054fc0 con 0x7f42c00cedf0 2024-04-06T13:50:08.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.269+0000 7f42b67fc700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42a406ae00 0x7f42a406d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:08.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.269+0000 7f42b67fc700 1 -- 172.21.15.189:0/2078133511 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) v1 -- 0x7f42a4003570 con 0x7f42a406ae00 2024-04-06T13:50:08.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.271+0000 7f42c5713700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42a406ae00 0x7f42a406d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:08.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.274+0000 7f42c5713700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42a406ae00 0x7f42a406d240 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f42c0160f40 tx=0x7f42b0008030 comp rx=0 tx=0).ready entity=mgr.13542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:50:08.280 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:08.280+0000 7f9bf30f5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:50:08.280 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:08.280+0000 7f9bf30f5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:50:08.280 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:08.280+0000 7f9bf30f5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:50:08.280 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:08.280+0000 7f9bf30f5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:50:08.280 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:50:08.280+0000 7f9bf30f5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:50:08.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.318+0000 7f42b67fc700 1 -- 172.21.15.189:0/2078133511 <== mgr.13542 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f42a4003570 con 0x7f42a406ae00 2024-04-06T13:50:08.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.321+0000 7f42c8178700 1 -- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42a406ae00 msgr2=0x7f42a406d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:08.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.321+0000 7f42c8178700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42a406ae00 0x7f42a406d240 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f42c0160f40 tx=0x7f42b0008030 comp rx=0 tx=0).stop 2024-04-06T13:50:08.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.321+0000 7f42c8178700 1 -- 172.21.15.189:0/2078133511 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42c00cedf0 msgr2=0x7f42c01f2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:08.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.321+0000 7f42c8178700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42c00cedf0 0x7f42c01f2830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00e870 tx=0x7f42bc00eb80 comp rx=0 tx=0).stop 2024-04-06T13:50:08.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.321+0000 7f42c8178700 1 -- 172.21.15.189:0/2078133511 shutdown_connections 2024-04-06T13:50:08.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.321+0000 7f42c8178700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f42a406ae00 0x7f42a406d240 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:08.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.321+0000 7f42c8178700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42c0146dd0 0x7f42c00acc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:08.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.321+0000 7f42c8178700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42c00cedf0 0x7f42c01f2830 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:08.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.321+0000 7f42c8178700 1 --2- 172.21.15.189:0/2078133511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42c00d17a0 0x7f42c00b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:08.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.321+0000 7f42c8178700 1 -- 172.21.15.189:0/2078133511 >> 172.21.15.189:0/2078133511 conn(0x7f42c00b2db0 msgr2=0x7f42c015cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:50:08.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.321+0000 7f42c8178700 1 -- 172.21.15.189:0/2078133511 shutdown_connections 2024-04-06T13:50:08.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.321+0000 7f42c8178700 1 -- 172.21.15.189:0/2078133511 wait complete. 2024-04-06T13:50:08.335 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --subvol sv01 . 1m 2024-04-06T13:50:08.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.465+0000 7f27e9e9e700 1 Processor -- start 2024-04-06T13:50:08.465 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.465+0000 7f27e9e9e700 1 -- start start 2024-04-06T13:50:08.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.466+0000 7f27e9e9e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f27e40b4af0 0x7f27e40b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:08.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.466+0000 7f27e9e9e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e40cdbe0 con 0x7f27e40b4660 2024-04-06T13:50:08.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.466+0000 7f27e9e9e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e415a940 con 0x7f27e415a550 2024-04-06T13:50:08.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.466+0000 7f27e9e9e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e415ac00 con 0x7f27e40b4af0 2024-04-06T13:50:08.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.466+0000 7f27e2ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f27e40b4660 0x7f27e40b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39458/0 (socket says 172.21.15.189:39458) 2024-04-06T13:50:08.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.466+0000 7f27e2ffd700 1 -- 172.21.15.189:0/54253201 learned_addr learned my addr 172.21.15.189:0/54253201 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:50:08.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.466+0000 7f27e3fff700 1 --2- 172.21.15.189:0/54253201 >> v2:172.21.15.192:3300/0 conn(0x7f27e40b4af0 0x7f27e40b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:08.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.466+0000 7f27e27fc700 1 -- 172.21.15.189:0/54253201 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969716883 0 0) 0x7f27e40cdbe0 con 0x7f27e40b4660 2024-04-06T13:50:08.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.466+0000 7f27e27fc700 1 -- 172.21.15.189:0/54253201 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27cc0035b0 con 0x7f27e40b4660 2024-04-06T13:50:08.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.466+0000 7f27e27fc700 1 -- 172.21.15.189:0/54253201 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4063478700 0 0) 0x7f27cc0035b0 con 0x7f27e40b4660 2024-04-06T13:50:08.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.466+0000 7f27e27fc700 1 -- 172.21.15.189:0/54253201 >> v2:172.21.15.192:3300/0 conn(0x7f27e40b4af0 msgr2=0x7f27e40b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:50:08.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.466+0000 7f27e27fc700 1 --2- 172.21.15.189:0/54253201 >> v2:172.21.15.192:3300/0 conn(0x7f27e40b4af0 0x7f27e40b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:08.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.466+0000 7f27e27fc700 1 -- 172.21.15.189:0/54253201 >> v1:172.21.15.192:6789/0 conn(0x7f27e415a550 legacy=0x7f27e40cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:08.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.466+0000 7f27e27fc700 1 -- 172.21.15.189:0/54253201 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e415aee0 con 0x7f27e40b4660 2024-04-06T13:50:08.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.466+0000 7f27e27fc700 1 -- 172.21.15.189:0/54253201 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f27d4001670 con 0x7f27e40b4660 2024-04-06T13:50:08.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e27fc700 1 -- 172.21.15.189:0/54253201 >> v1:172.21.15.189:6789/0 conn(0x7f27e40b4660 legacy=0x7f27e40b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:08.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e27fc700 1 --2- 172.21.15.189:0/54253201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27cc003a00 0x7f27cc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:08.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e27fc700 1 --2- 172.21.15.189:0/54253201 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27cc004850 0x7f27cc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:08.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e27fc700 1 --2- 172.21.15.189:0/54253201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27cc005210 0x7f27cc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:08.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e27fc700 1 -- 172.21.15.189:0/54253201 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e415ac00 con 0x7f27cc003a00 2024-04-06T13:50:08.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e27fc700 1 -- 172.21.15.189:0/54253201 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc00c0b0 con 0x7f27cc005210 2024-04-06T13:50:08.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e27fc700 1 -- 172.21.15.189:0/54253201 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc00c270 con 0x7f27cc004850 2024-04-06T13:50:08.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e37fe700 1 --2- 172.21.15.189:0/54253201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27cc003a00 0x7f27cc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:08.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e3fff700 1 --2- 172.21.15.189:0/54253201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27cc005210 0x7f27cc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:08.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e2ffd700 1 --2- 172.21.15.189:0/54253201 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27cc004850 0x7f27cc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:08.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e37fe700 1 -- 172.21.15.189:0/54253201 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27cc004850 msgr2=0x7f27cc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:50:08.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e37fe700 1 --2- 172.21.15.189:0/54253201 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27cc004850 0x7f27cc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:08.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e37fe700 1 -- 172.21.15.189:0/54253201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27cc005210 msgr2=0x7f27cc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:50:08.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e37fe700 1 --2- 172.21.15.189:0/54253201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27cc005210 0x7f27cc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:08.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e37fe700 1 -- 172.21.15.189:0/54253201 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc0035b0 con 0x7f27cc003a00 2024-04-06T13:50:08.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e3fff700 1 -- 172.21.15.189:0/54253201 reap_dead start 2024-04-06T13:50:08.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e37fe700 1 --2- 172.21.15.189:0/54253201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27cc003a00 0x7f27cc003e20 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 crypto rx=0x7f27d80017c0 tx=0x7f27cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba9d82e48b859029 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:50:08.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e27fc700 1 -- 172.21.15.189:0/54253201 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d80052d0 con 0x7f27cc003a00 2024-04-06T13:50:08.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e27fc700 1 -- 172.21.15.189:0/54253201 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d8005450 con 0x7f27cc003a00 2024-04-06T13:50:08.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.467+0000 7f27e27fc700 1 -- 172.21.15.189:0/54253201 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d80038c0 con 0x7f27cc003a00 2024-04-06T13:50:08.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.468+0000 7f27e9e9e700 1 -- 172.21.15.189:0/54253201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27cc003a00 msgr2=0x7f27cc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:08.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.468+0000 7f27e9e9e700 1 --2- 172.21.15.189:0/54253201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27cc003a00 0x7f27cc003e20 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 crypto rx=0x7f27d80017c0 tx=0x7f27cc005880 comp rx=0 tx=0).stop 2024-04-06T13:50:08.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.468+0000 7f27e9e9e700 1 -- 172.21.15.189:0/54253201 shutdown_connections 2024-04-06T13:50:08.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.468+0000 7f27e9e9e700 1 --2- 172.21.15.189:0/54253201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27cc003a00 0x7f27cc003e20 unknown :-1 s=CLOSED pgs=3136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:08.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.468+0000 7f27e9e9e700 1 -- 172.21.15.189:0/54253201 >> 172.21.15.189:0/54253201 conn(0x7f27e40d00b0 msgr2=0x7f27e40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:50:08.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.468+0000 7f27e9e9e700 1 -- 172.21.15.189:0/54253201 shutdown_connections 2024-04-06T13:50:08.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.468+0000 7f27e9e9e700 1 -- 172.21.15.189:0/54253201 wait complete. 2024-04-06T13:50:08.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.469+0000 7f27e9e9e700 1 Processor -- start 2024-04-06T13:50:08.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.469+0000 7f27e9e9e700 1 -- start start 2024-04-06T13:50:08.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.470+0000 7f27e9e9e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27cc003a00 0x7f27e415ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:08.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.470+0000 7f27e9e9e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27e40b17d0 0x7f27e414aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:08.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.470+0000 7f27e9e9e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27e414b900 0x7f27e414b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:08.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.470+0000 7f27e9e9e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e4159250 con 0x7f27cc003a00 2024-04-06T13:50:08.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.470+0000 7f27e9e9e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e40aebb0 con 0x7f27e40b17d0 2024-04-06T13:50:08.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.470+0000 7f27e9e9e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e414b720 con 0x7f27e414b900 2024-04-06T13:50:08.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.470+0000 7f27e37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27cc003a00 0x7f27e415ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:08.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.470+0000 7f27e37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27cc003a00 0x7f27e415ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35428/0 (socket says 172.21.15.189:35428) 2024-04-06T13:50:08.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.470+0000 7f27e37fe700 1 -- 172.21.15.189:0/196630163 learned_addr learned my addr 172.21.15.189:0/196630163 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:50:08.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.470+0000 7f27e2ffd700 1 --2- 172.21.15.189:0/196630163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27e40b17d0 0x7f27e414aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:08.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.470+0000 7f27e3fff700 1 --2- 172.21.15.189:0/196630163 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27e414b900 0x7f27e414b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:08.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.470+0000 7f27e37fe700 1 -- 172.21.15.189:0/196630163 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27e414b900 msgr2=0x7f27e414b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:08.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.470+0000 7f27e37fe700 1 --2- 172.21.15.189:0/196630163 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27e414b900 0x7f27e414b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:08.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.470+0000 7f27e37fe700 1 -- 172.21.15.189:0/196630163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27e40b17d0 msgr2=0x7f27e414aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:08.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.470+0000 7f27e37fe700 1 --2- 172.21.15.189:0/196630163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27e40b17d0 0x7f27e414aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:08.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.470+0000 7f27e37fe700 1 -- 172.21.15.189:0/196630163 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e40aee00 con 0x7f27cc003a00 2024-04-06T13:50:08.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.470+0000 7f27e37fe700 1 --2- 172.21.15.189:0/196630163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27cc003a00 0x7f27e415ce10 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7f27d8000c00 tx=0x7f27d8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:50:08.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.471+0000 7f27cbfff700 1 -- 172.21.15.189:0/196630163 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d8001290 con 0x7f27cc003a00 2024-04-06T13:50:08.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.471+0000 7f27cbfff700 1 -- 172.21.15.189:0/196630163 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d8007040 con 0x7f27cc003a00 2024-04-06T13:50:08.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.471+0000 7f27e9e9e700 1 -- 172.21.15.189:0/196630163 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e40af0e0 con 0x7f27cc003a00 2024-04-06T13:50:08.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.471+0000 7f27cbfff700 1 -- 172.21.15.189:0/196630163 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d80073f0 con 0x7f27cc003a00 2024-04-06T13:50:08.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.471+0000 7f27e9e9e700 1 -- 172.21.15.189:0/196630163 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e4159710 con 0x7f27cc003a00 2024-04-06T13:50:08.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.472+0000 7f27e9e9e700 1 -- 172.21.15.189:0/196630163 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e41598a0 con 0x7f27cc003a00 2024-04-06T13:50:08.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.472+0000 7f27cbfff700 1 -- 172.21.15.189:0/196630163 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 173) v1 ==== 89000+0+0 (secure 0 0 0) 0x7f27d8007550 con 0x7f27cc003a00 2024-04-06T13:50:08.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.472+0000 7f27cbfff700 1 --2- 172.21.15.189:0/196630163 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f27d006ab30 0x7f27d006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:50:08.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.472+0000 7f27cbfff700 1 -- 172.21.15.189:0/196630163 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f27d8089e20 con 0x7f27cc003a00 2024-04-06T13:50:08.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.472+0000 7f27e2ffd700 1 --2- 172.21.15.189:0/196630163 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f27d006ab30 0x7f27d006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:50:08.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.473+0000 7f27e2ffd700 1 --2- 172.21.15.189:0/196630163 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f27d006ab30 0x7f27d006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f27d4000c00 tx=0x7f27d4009040 comp rx=0 tx=0).ready entity=mgr.13542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:50:08.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.477+0000 7f27cbfff700 1 -- 172.21.15.189:0/196630163 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f27d8055180 con 0x7f27cc003a00 2024-04-06T13:50:08.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.696+0000 7f27e9e9e700 1 -- 172.21.15.189:0/196630163 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "subvol": "sv01", "path": ".", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f27e406fdd0 con 0x7f27d006ab30 2024-04-06T13:50:08.765 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.765+0000 7f27cbfff700 1 -- 172.21.15.189:0/196630163 <== mgr.13542 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+85 (secure 0 0 0) 0x7f27d4004d30 con 0x7f27d006ab30 2024-04-06T13:50:08.765 INFO:teuthology.orchestra.run.smithi189.stdout:Schedule set for path /volumes/_nogroup/sv01/b99e82de-bfe3-4cd6-ae23-2f0d4f4372ff/.. 2024-04-06T13:50:08.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.767+0000 7f27e9e9e700 1 -- 172.21.15.189:0/196630163 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f27d006ab30 msgr2=0x7f27d006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:08.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.767+0000 7f27e9e9e700 1 --2- 172.21.15.189:0/196630163 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f27d006ab30 0x7f27d006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f27d4000c00 tx=0x7f27d4009040 comp rx=0 tx=0).stop 2024-04-06T13:50:08.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.767+0000 7f27e9e9e700 1 -- 172.21.15.189:0/196630163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27cc003a00 msgr2=0x7f27e415ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:50:08.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.767+0000 7f27e9e9e700 1 --2- 172.21.15.189:0/196630163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27cc003a00 0x7f27e415ce10 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7f27d8000c00 tx=0x7f27d8001120 comp rx=0 tx=0).stop 2024-04-06T13:50:08.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.767+0000 7f27e9e9e700 1 -- 172.21.15.189:0/196630163 shutdown_connections 2024-04-06T13:50:08.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.767+0000 7f27e9e9e700 1 --2- 172.21.15.189:0/196630163 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f27d006ab30 0x7f27d006cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:08.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.767+0000 7f27e9e9e700 1 --2- 172.21.15.189:0/196630163 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f27e414b900 0x7f27e414b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:08.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.767+0000 7f27e9e9e700 1 --2- 172.21.15.189:0/196630163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27cc003a00 0x7f27e415ce10 unknown :-1 s=CLOSED pgs=3137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:08.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.767+0000 7f27e9e9e700 1 --2- 172.21.15.189:0/196630163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27e40b17d0 0x7f27e414aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:50:08.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.767+0000 7f27e9e9e700 1 -- 172.21.15.189:0/196630163 >> 172.21.15.189:0/196630163 conn(0x7f27e40d00b0 msgr2=0x7f27e4155a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:50:08.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.767+0000 7f27e9e9e700 1 -- 172.21.15.189:0/196630163 shutdown_connections 2024-04-06T13:50:08.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:50:08.767+0000 7f27e9e9e700 1 -- 172.21.15.189:0/196630163 wait complete. 2024-04-06T13:50:08.780 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /volumes/_nogroup/sv01/b99e82de-bfe3-4cd6-ae23-2f0d4f4372ff/.. 2024-04-06T13:50:33.336 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:50:33.340 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:51:00.776 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:00.776+0000 7f9be4097700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:51:00.777 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:00.776+0000 7f9be4097700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:51:00.777 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:00.776+0000 7f9be4097700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:51:00.777 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:00.776+0000 7f9be4097700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:51:00.777 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:00.776+0000 7f9be4097700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:51:03.491 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:51:03.495 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:51:23.780 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-06T13:51:23.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.917+0000 7f88ff6c3700 1 Processor -- start 2024-04-06T13:51:23.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.917+0000 7f88ff6c3700 1 -- start start 2024-04-06T13:51:23.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.918+0000 7f88ff6c3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f88f8158fe0 0x7f88f806a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:23.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.918+0000 7f88ff6c3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f806ac70 0x7f88f815bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:23.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.918+0000 7f88ff6c3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f815f9e0 con 0x7f88f815c100 2024-04-06T13:51:23.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.918+0000 7f88ff6c3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88f806b0f0 con 0x7f88f8158fe0 2024-04-06T13:51:23.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.918+0000 7f88ff6c3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f815f0d0 con 0x7f88f806ac70 2024-04-06T13:51:23.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.918+0000 7f88fdc60700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f88f815c100 0x7f88f80717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34474/0 (socket says 172.21.15.189:34474) 2024-04-06T13:51:23.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.918+0000 7f88fdc60700 1 -- 172.21.15.189:0/294620122 learned_addr learned my addr 172.21.15.189:0/294620122 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:23.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.918+0000 7f88fd45f700 1 --2- 172.21.15.189:0/294620122 >> v2:172.21.15.192:3300/0 conn(0x7f88f8158fe0 0x7f88f806a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:23.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.918+0000 7f88fcc5e700 1 --2- 172.21.15.189:0/294620122 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f806ac70 0x7f88f815bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:23.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.918+0000 7f88effff700 1 -- 172.21.15.189:0/294620122 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130179264 0 0) 0x7f88f815f9e0 con 0x7f88f815c100 2024-04-06T13:51:23.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88effff700 1 -- 172.21.15.189:0/294620122 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88e40035b0 con 0x7f88f815c100 2024-04-06T13:51:23.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88effff700 1 -- 172.21.15.189:0/294620122 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3683464025 0 0) 0x7f88e40035b0 con 0x7f88f815c100 2024-04-06T13:51:23.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88effff700 1 -- 172.21.15.189:0/294620122 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f806ac70 msgr2=0x7f88f815bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:23.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88effff700 1 --2- 172.21.15.189:0/294620122 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f806ac70 0x7f88f815bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:23.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88effff700 1 -- 172.21.15.189:0/294620122 >> v2:172.21.15.192:3300/0 conn(0x7f88f8158fe0 msgr2=0x7f88f806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:23.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88effff700 1 --2- 172.21.15.189:0/294620122 >> v2:172.21.15.192:3300/0 conn(0x7f88f8158fe0 0x7f88f806a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:23.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88effff700 1 -- 172.21.15.189:0/294620122 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f815c8f0 con 0x7f88f815c100 2024-04-06T13:51:23.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88effff700 1 -- 172.21.15.189:0/294620122 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f88f00019f0 con 0x7f88f815c100 2024-04-06T13:51:23.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88effff700 1 -- 172.21.15.189:0/294620122 >> v1:172.21.15.189:6789/0 conn(0x7f88f815c100 legacy=0x7f88f80717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:23.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88effff700 1 --2- 172.21.15.189:0/294620122 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f806ac70 0x7f88e4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:23.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88effff700 1 --2- 172.21.15.189:0/294620122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88e4003f80 0x7f88e4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:23.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88effff700 1 --2- 172.21.15.189:0/294620122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88e4004d90 0x7f88e4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:23.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88effff700 1 -- 172.21.15.189:0/294620122 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f806b0f0 con 0x7f88e4003f80 2024-04-06T13:51:23.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88effff700 1 -- 172.21.15.189:0/294620122 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f815f0d0 con 0x7f88e4004d90 2024-04-06T13:51:23.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88effff700 1 -- 172.21.15.189:0/294620122 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e400ba60 con 0x7f88f806ac70 2024-04-06T13:51:23.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88fcc5e700 1 --2- 172.21.15.189:0/294620122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88e4003f80 0x7f88e4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:23.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88fd45f700 1 --2- 172.21.15.189:0/294620122 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f806ac70 0x7f88e4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:23.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88fdc60700 1 --2- 172.21.15.189:0/294620122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88e4004d90 0x7f88e4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:23.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88fcc5e700 1 -- 172.21.15.189:0/294620122 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f806ac70 msgr2=0x7f88e4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:23.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88fcc5e700 1 --2- 172.21.15.189:0/294620122 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f806ac70 0x7f88e4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:23.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88fcc5e700 1 -- 172.21.15.189:0/294620122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88e4004d90 msgr2=0x7f88e4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:23.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88fcc5e700 1 --2- 172.21.15.189:0/294620122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88e4004d90 0x7f88e4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:23.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88fcc5e700 1 -- 172.21.15.189:0/294620122 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e40035b0 con 0x7f88e4003f80 2024-04-06T13:51:23.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.919+0000 7f88fcc5e700 1 --2- 172.21.15.189:0/294620122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88e4003f80 0x7f88e4004850 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 crypto rx=0x7f88f4009b90 tx=0x7f88e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dcb146e0829989a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:23.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.920+0000 7f88effff700 1 -- 172.21.15.189:0/294620122 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f400e6c0 con 0x7f88e4003f80 2024-04-06T13:51:23.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.920+0000 7f88effff700 1 -- 172.21.15.189:0/294620122 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f400e820 con 0x7f88e4003f80 2024-04-06T13:51:23.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.920+0000 7f88effff700 1 -- 172.21.15.189:0/294620122 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f400eb90 con 0x7f88e4003f80 2024-04-06T13:51:23.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.920+0000 7f88ff6c3700 1 -- 172.21.15.189:0/294620122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88e4003f80 msgr2=0x7f88e4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:23.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.920+0000 7f88ff6c3700 1 --2- 172.21.15.189:0/294620122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88e4003f80 0x7f88e4004850 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 crypto rx=0x7f88f4009b90 tx=0x7f88e4005880 comp rx=0 tx=0).stop 2024-04-06T13:51:23.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.920+0000 7f88fdc60700 1 -- 172.21.15.189:0/294620122 reap_dead start 2024-04-06T13:51:23.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.920+0000 7f88ff6c3700 1 -- 172.21.15.189:0/294620122 shutdown_connections 2024-04-06T13:51:23.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.920+0000 7f88ff6c3700 1 -- 172.21.15.189:0/294620122 >> 172.21.15.189:0/294620122 conn(0x7f88f80d00b0 msgr2=0x7f88f80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:23.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.920+0000 7f88ff6c3700 1 -- 172.21.15.189:0/294620122 shutdown_connections 2024-04-06T13:51:23.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.920+0000 7f88ff6c3700 1 -- 172.21.15.189:0/294620122 wait complete. 2024-04-06T13:51:23.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.921+0000 7f88ff6c3700 1 Processor -- start 2024-04-06T13:51:23.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.922+0000 7f88ff6c3700 1 -- start start 2024-04-06T13:51:23.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.922+0000 7f88ff6c3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88e4003f80 0x7f88f80b4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:23.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.922+0000 7f88ff6c3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f81421a0 0x7f88f80b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:23.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.922+0000 7f88ff6c3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88f815e1c0 0x7f88f815e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:23.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.922+0000 7f88ff6c3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f81f5e70 con 0x7f88e4003f80 2024-04-06T13:51:23.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.922+0000 7f88ff6c3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f815eb90 con 0x7f88f815e1c0 2024-04-06T13:51:23.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.922+0000 7f88ff6c3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f80b4fb0 con 0x7f88f81421a0 2024-04-06T13:51:23.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.922+0000 7f88fd45f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88e4003f80 0x7f88f80b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:23.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.922+0000 7f88fd45f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88e4003f80 0x7f88f80b4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54472/0 (socket says 172.21.15.189:54472) 2024-04-06T13:51:23.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.922+0000 7f88fd45f700 1 -- 172.21.15.189:0/2042705824 learned_addr learned my addr 172.21.15.189:0/2042705824 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:23.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.923+0000 7f88fcc5e700 1 --2- 172.21.15.189:0/2042705824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f81421a0 0x7f88f80b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:23.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.923+0000 7f88fd45f700 1 -- 172.21.15.189:0/2042705824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f81421a0 msgr2=0x7f88f80b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:23.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.923+0000 7f88fdc60700 1 --2- 172.21.15.189:0/2042705824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88f815e1c0 0x7f88f815e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:23.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.923+0000 7f88fd45f700 1 --2- 172.21.15.189:0/2042705824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f81421a0 0x7f88f80b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:23.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.923+0000 7f88fd45f700 1 -- 172.21.15.189:0/2042705824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88f815e1c0 msgr2=0x7f88f815e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:23.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.923+0000 7f88fd45f700 1 --2- 172.21.15.189:0/2042705824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88f815e1c0 0x7f88f815e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:23.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.923+0000 7f88fd45f700 1 -- 172.21.15.189:0/2042705824 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f81410a0 con 0x7f88e4003f80 2024-04-06T13:51:23.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.923+0000 7f88fdc60700 1 --2- 172.21.15.189:0/2042705824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88f815e1c0 0x7f88f815e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:51:23.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.923+0000 7f88fd45f700 1 --2- 172.21.15.189:0/2042705824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88e4003f80 0x7f88f80b4110 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7f88e8011ad0 tx=0x7f88e8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:23.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.923+0000 7f88edffb700 1 -- 172.21.15.189:0/2042705824 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e800f9a0 con 0x7f88e4003f80 2024-04-06T13:51:23.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.923+0000 7f88edffb700 1 -- 172.21.15.189:0/2042705824 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e800fb00 con 0x7f88e4003f80 2024-04-06T13:51:23.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.923+0000 7f88ff6c3700 1 -- 172.21.15.189:0/2042705824 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f80b4cd0 con 0x7f88e4003f80 2024-04-06T13:51:23.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.923+0000 7f88edffb700 1 -- 172.21.15.189:0/2042705824 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e800feb0 con 0x7f88e4003f80 2024-04-06T13:51:23.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.923+0000 7f88ff6c3700 1 -- 172.21.15.189:0/2042705824 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f81419a0 con 0x7f88e4003f80 2024-04-06T13:51:23.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.924+0000 7f88ff6c3700 1 -- 172.21.15.189:0/2042705824 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f8141520 con 0x7f88e4003f80 2024-04-06T13:51:23.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.925+0000 7f88edffb700 1 -- 172.21.15.189:0/2042705824 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 176) v1 ==== 89429+0+0 (secure 0 0 0) 0x7f88e8014440 con 0x7f88e4003f80 2024-04-06T13:51:23.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.925+0000 7f88edffb700 1 --2- 172.21.15.189:0/2042705824 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f88dc06afb0 0x7f88dc06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:23.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.925+0000 7f88edffb700 1 -- 172.21.15.189:0/2042705824 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f88e808a380 con 0x7f88e4003f80 2024-04-06T13:51:23.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.925+0000 7f88fcc5e700 1 --2- 172.21.15.189:0/2042705824 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f88dc06afb0 0x7f88dc06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:23.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.926+0000 7f88fcc5e700 1 --2- 172.21.15.189:0/2042705824 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f88dc06afb0 0x7f88dc06d470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f88f4009b30 tx=0x7f88f4008030 comp rx=0 tx=0).ready entity=mgr.13542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:23.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:23.930+0000 7f88edffb700 1 -- 172.21.15.189:0/2042705824 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f88f8141520 con 0x7f88e4003f80 2024-04-06T13:51:24.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:24.180+0000 7f88ff6c3700 1 -- 172.21.15.189:0/2042705824 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f88f8141520 con 0x7f88e4003f80 2024-04-06T13:51:24.658 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:24.657+0000 7f88edffb700 1 -- 172.21.15.189:0/2042705824 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 177) v1 ==== 89412+0+0 (secure 0 0 0) 0x7f88e80551c0 con 0x7f88e4003f80 2024-04-06T13:51:24.659 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:24.659+0000 7f88edffb700 1 -- 172.21.15.189:0/2042705824 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v177) v1 ==== 93+0+0 (secure 0 0 0) 0x7f88e8085a30 con 0x7f88e4003f80 2024-04-06T13:51:24.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:24.661+0000 7f88ff6c3700 1 -- 172.21.15.189:0/2042705824 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f88dc06afb0 msgr2=0x7f88dc06d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:24.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:24.661+0000 7f88ff6c3700 1 --2- 172.21.15.189:0/2042705824 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f88dc06afb0 0x7f88dc06d470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f88f4009b30 tx=0x7f88f4008030 comp rx=0 tx=0).stop 2024-04-06T13:51:24.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:24.661+0000 7f88ff6c3700 1 -- 172.21.15.189:0/2042705824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88e4003f80 msgr2=0x7f88f80b4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:24.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:24.661+0000 7f88ff6c3700 1 --2- 172.21.15.189:0/2042705824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88e4003f80 0x7f88f80b4110 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7f88e8011ad0 tx=0x7f88e8011e90 comp rx=0 tx=0).stop 2024-04-06T13:51:24.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:24.661+0000 7f88ff6c3700 1 -- 172.21.15.189:0/2042705824 shutdown_connections 2024-04-06T13:51:24.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:24.661+0000 7f88ff6c3700 1 --2- 172.21.15.189:0/2042705824 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f88dc06afb0 0x7f88dc06d470 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:24.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:24.661+0000 7f88ff6c3700 1 --2- 172.21.15.189:0/2042705824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88f815e1c0 0x7f88f815e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:24.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:24.661+0000 7f88ff6c3700 1 --2- 172.21.15.189:0/2042705824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88e4003f80 0x7f88f80b4110 unknown :-1 s=CLOSED pgs=3144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:24.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:24.661+0000 7f88ff6c3700 1 --2- 172.21.15.189:0/2042705824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88f81421a0 0x7f88f80b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:24.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:24.661+0000 7f88ff6c3700 1 -- 172.21.15.189:0/2042705824 >> 172.21.15.189:0/2042705824 conn(0x7f88f80d00b0 msgr2=0x7f88f8144780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:24.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:24.661+0000 7f88ff6c3700 1 -- 172.21.15.189:0/2042705824 shutdown_connections 2024-04-06T13:51:24.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:24.662+0000 7f88ff6c3700 1 -- 172.21.15.189:0/2042705824 wait complete. 2024-04-06T13:51:24.676 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-06T13:51:24.677 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:51:24.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-06T13:51:24.761 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-06T13:51:24.761 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:51:24.761 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:51:24.779 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:51:24.779 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:51:24.837 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:51:25.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.000+0000 7f8a91a10700 1 Processor -- start 2024-04-06T13:51:25.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.000+0000 7f8a91a10700 1 -- start start 2024-04-06T13:51:25.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.001+0000 7f8a91a10700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8a8c0d11b0 0x7f8a8c0d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:25.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.001+0000 7f8a91a10700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8a8c0cb870 0x7f8a8c15c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:25.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.001+0000 7f8a91a10700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a8c0d1ad0 con 0x7f8a8c0cfa00 2024-04-06T13:51:25.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.001+0000 7f8a91a10700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a8c0ccdc0 con 0x7f8a8c0d11b0 2024-04-06T13:51:25.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.001+0000 7f8a91a10700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a8c0d0b50 con 0x7f8a8c0cb870 2024-04-06T13:51:25.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.001+0000 7f8a8affd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8a8c0d11b0 0x7f8a8c0d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:25.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.001+0000 7f8a8b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8a8c0cfa00 0x7f8a8c0cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34486/0 (socket says 172.21.15.189:34486) 2024-04-06T13:51:25.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.001+0000 7f8a8b7fe700 1 -- 172.21.15.189:0/3645307263 learned_addr learned my addr 172.21.15.189:0/3645307263 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:25.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.001+0000 7f8a8a7fc700 1 --2- 172.21.15.189:0/3645307263 >> v2:172.21.15.192:3300/0 conn(0x7f8a8c0cb870 0x7f8a8c15c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:25.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.001+0000 7f8a89ffb700 1 -- 172.21.15.189:0/3645307263 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289831547 0 0) 0x7f8a8c0d1ad0 con 0x7f8a8c0cfa00 2024-04-06T13:51:25.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.002+0000 7f8a89ffb700 1 -- 172.21.15.189:0/3645307263 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a740035b0 con 0x7f8a8c0cfa00 2024-04-06T13:51:25.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.002+0000 7f8a8affd700 1 -- 172.21.15.189:0/3645307263 >> v2:172.21.15.192:3300/0 conn(0x7f8a8c0cb870 msgr2=0x7f8a8c15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:25.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.002+0000 7f8a8affd700 1 --2- 172.21.15.189:0/3645307263 >> v2:172.21.15.192:3300/0 conn(0x7f8a8c0cb870 0x7f8a8c15c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.002+0000 7f8a8affd700 1 -- 172.21.15.189:0/3645307263 >> v1:172.21.15.189:6789/0 conn(0x7f8a8c0cfa00 legacy=0x7f8a8c0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:25.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.002+0000 7f8a8affd700 1 -- 172.21.15.189:0/3645307263 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a8c0cbd80 con 0x7f8a8c0d11b0 2024-04-06T13:51:25.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.002+0000 7f8a8affd700 1 --2- 172.21.15.189:0/3645307263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a8c0d11b0 0x7f8a8c0d1590 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c04fb40 tx=0x7f8a7c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e049094a34e7aed server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:25.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.002+0000 7f8a89ffb700 1 -- 172.21.15.189:0/3645307263 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c011830 con 0x7f8a8c0d11b0 2024-04-06T13:51:25.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.002+0000 7f8a89ffb700 1 -- 172.21.15.189:0/3645307263 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7c011990 con 0x7f8a8c0d11b0 2024-04-06T13:51:25.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.002+0000 7f8a89ffb700 1 -- 172.21.15.189:0/3645307263 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c011d20 con 0x7f8a8c0d11b0 2024-04-06T13:51:25.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.003+0000 7f8a91a10700 1 -- 172.21.15.189:0/3645307263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a8c0d11b0 msgr2=0x7f8a8c0d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:25.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.003+0000 7f8a91a10700 1 --2- 172.21.15.189:0/3645307263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a8c0d11b0 0x7f8a8c0d1590 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c04fb40 tx=0x7f8a7c009b20 comp rx=0 tx=0).stop 2024-04-06T13:51:25.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.003+0000 7f8a91a10700 1 -- 172.21.15.189:0/3645307263 shutdown_connections 2024-04-06T13:51:25.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.003+0000 7f8a91a10700 1 --2- 172.21.15.189:0/3645307263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a8c0d11b0 0x7f8a8c0d1590 unknown :-1 s=CLOSED pgs=3147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.003+0000 7f8a91a10700 1 --2- 172.21.15.189:0/3645307263 >> v2:172.21.15.192:3300/0 conn(0x7f8a8c0cb870 0x7f8a8c15c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.003+0000 7f8a91a10700 1 -- 172.21.15.189:0/3645307263 >> 172.21.15.189:0/3645307263 conn(0x7f8a8c0b2db0 msgr2=0x7f8a8c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:25.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.003+0000 7f8a91a10700 1 -- 172.21.15.189:0/3645307263 shutdown_connections 2024-04-06T13:51:25.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.003+0000 7f8a91a10700 1 -- 172.21.15.189:0/3645307263 wait complete. 2024-04-06T13:51:25.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.004+0000 7f8a91a10700 1 Processor -- start 2024-04-06T13:51:25.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.004+0000 7f8a91a10700 1 -- start start 2024-04-06T13:51:25.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.005+0000 7f8a91a10700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a8c0cb870 0x7f8a8c153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:25.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.005+0000 7f8a91a10700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a8c0cfa00 0x7f8a8c158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:25.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.005+0000 7f8a91a10700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a8c0d11b0 0x7f8a8c158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:25.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.005+0000 7f8a91a10700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c153ed0 con 0x7f8a8c0d11b0 2024-04-06T13:51:25.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.005+0000 7f8a91a10700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c06d830 con 0x7f8a8c0cfa00 2024-04-06T13:51:25.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.005+0000 7f8a91a10700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c1543b0 con 0x7f8a8c0cb870 2024-04-06T13:51:25.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.005+0000 7f8a8b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a8c0d11b0 0x7f8a8c158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:25.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.005+0000 7f8a8a7fc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a8c0cfa00 0x7f8a8c158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:25.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.005+0000 7f8a8b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a8c0d11b0 0x7f8a8c158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54494/0 (socket says 172.21.15.189:54494) 2024-04-06T13:51:25.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.005+0000 7f8a8b7fe700 1 -- 172.21.15.189:0/2011308363 learned_addr learned my addr 172.21.15.189:0/2011308363 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:25.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.005+0000 7f8a8affd700 1 --2- 172.21.15.189:0/2011308363 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a8c0cb870 0x7f8a8c153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:25.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.005+0000 7f8a8b7fe700 1 -- 172.21.15.189:0/2011308363 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a8c0cb870 msgr2=0x7f8a8c153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:25.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.005+0000 7f8a8b7fe700 1 --2- 172.21.15.189:0/2011308363 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a8c0cb870 0x7f8a8c153990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.005+0000 7f8a8b7fe700 1 -- 172.21.15.189:0/2011308363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a8c0cfa00 msgr2=0x7f8a8c158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:25.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.005+0000 7f8a8b7fe700 1 --2- 172.21.15.189:0/2011308363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a8c0cfa00 0x7f8a8c158150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.005+0000 7f8a8b7fe700 1 -- 172.21.15.189:0/2011308363 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a7c0094a0 con 0x7f8a8c0d11b0 2024-04-06T13:51:25.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.005+0000 7f8a8a7fc700 1 --2- 172.21.15.189:0/2011308363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a8c0cfa00 0x7f8a8c158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:51:25.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.005+0000 7f8a8b7fe700 1 --2- 172.21.15.189:0/2011308363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a8c0d11b0 0x7f8a8c158690 secure :-1 s=READY pgs=3148 cs=0 l=1 rev1=1 crypto rx=0x7f8a8400caa0 tx=0x7f8a8400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:25.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.006+0000 7f8a73fff700 1 -- 172.21.15.189:0/2011308363 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8400a9e0 con 0x7f8a8c0d11b0 2024-04-06T13:51:25.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.006+0000 7f8a73fff700 1 -- 172.21.15.189:0/2011308363 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8400ab40 con 0x7f8a8c0d11b0 2024-04-06T13:51:25.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.006+0000 7f8a73fff700 1 -- 172.21.15.189:0/2011308363 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84012400 con 0x7f8a8c0d11b0 2024-04-06T13:51:25.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.006+0000 7f8a91a10700 1 -- 172.21.15.189:0/2011308363 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a8c154690 con 0x7f8a8c0d11b0 2024-04-06T13:51:25.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.006+0000 7f8a91a10700 1 -- 172.21.15.189:0/2011308363 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a8c154c50 con 0x7f8a8c0d11b0 2024-04-06T13:51:25.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.007+0000 7f8a91a10700 1 -- 172.21.15.189:0/2011308363 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a8c154e70 con 0x7f8a8c0d11b0 2024-04-06T13:51:25.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.007+0000 7f8a73fff700 1 -- 172.21.15.189:0/2011308363 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 177) v1 ==== 89412+0+0 (secure 0 0 0) 0x7f8a8401b3b0 con 0x7f8a8c0d11b0 2024-04-06T13:51:25.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.008+0000 7f8a73fff700 1 --2- 172.21.15.189:0/2011308363 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8a7806af60 0x7f8a7806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:25.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.008+0000 7f8a8affd700 1 -- 172.21.15.189:0/2011308363 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8a7806af60 msgr2=0x7f8a7806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:25.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.008+0000 7f8a73fff700 1 -- 172.21.15.189:0/2011308363 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f8a840896b0 con 0x7f8a8c0d11b0 2024-04-06T13:51:25.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.008+0000 7f8a8affd700 1 --2- 172.21.15.189:0/2011308363 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8a7806af60 0x7f8a7806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:51:25.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.012+0000 7f8a73fff700 1 -- 172.21.15.189:0/2011308363 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f8a840542c0 con 0x7f8a8c0d11b0 2024-04-06T13:51:25.046 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:25.045+0000 7f72d02d9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:51:25.075 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:25.074+0000 7f3fecaf2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T13:51:25.124 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:25.122+0000 7f72d02d9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:51:25.157 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:25.156+0000 7f3fecaf2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T13:51:25.196 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:25.195+0000 7f72d02d9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:51:25.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.209+0000 7f8a8affd700 1 -- 172.21.15.189:0/2011308363 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8a7806af60 msgr2=0x7f8a7806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:25.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.209+0000 7f8a8affd700 1 --2- 172.21.15.189:0/2011308363 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8a7806af60 0x7f8a7806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:51:25.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.229+0000 7f8a91a10700 1 -- 172.21.15.189:0/2011308363 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a8c074990 con 0x7f8a8c0d11b0 2024-04-06T13:51:25.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.230+0000 7f8a73fff700 1 -- 172.21.15.189:0/2011308363 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v286) v1 ==== 74+0+17125 (secure 0 0 0) 0x7f8a840578e0 con 0x7f8a8c0d11b0 2024-04-06T13:51:25.232 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:51:25.232 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":286,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:51:08.736525+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:49:51.467569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"282","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-04-06T13:49:52.656784+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"283","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"38.5","mappings":[{"from":7,"to":2}]},{"pgid":"38.2b","mappings":[{"from":7,"to":4}]},{"pgid":"38.2e","mappings":[{"from":7,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/2987126770":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/2905400170":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/893866137":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/2221278556":"2024-04-07T13:50:07.256457+0000","172.21.15.189:6833/111704":"2024-04-07T13:50:07.256457+0000","172.21.15.189:6832/111704":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/4009223680":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/3345146368":"2024-04-07T13:50:07.256457+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:51:25.233 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:25.232+0000 7f3fecaf2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T13:51:25.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.232+0000 7f8a91a10700 1 -- 172.21.15.189:0/2011308363 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8a7806af60 msgr2=0x7f8a7806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:51:25.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.232+0000 7f8a91a10700 1 --2- 172.21.15.189:0/2011308363 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8a7806af60 0x7f8a7806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.232+0000 7f8a91a10700 1 -- 172.21.15.189:0/2011308363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a8c0d11b0 msgr2=0x7f8a8c158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:25.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.232+0000 7f8a91a10700 1 --2- 172.21.15.189:0/2011308363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a8c0d11b0 0x7f8a8c158690 secure :-1 s=READY pgs=3148 cs=0 l=1 rev1=1 crypto rx=0x7f8a8400caa0 tx=0x7f8a8400cdb0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.235 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.232+0000 7f8a91a10700 1 -- 172.21.15.189:0/2011308363 shutdown_connections 2024-04-06T13:51:25.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.232+0000 7f8a91a10700 1 --2- 172.21.15.189:0/2011308363 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f8a7806af60 0x7f8a7806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.232+0000 7f8a91a10700 1 --2- 172.21.15.189:0/2011308363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a8c0d11b0 0x7f8a8c158690 unknown :-1 s=CLOSED pgs=3148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.232+0000 7f8a91a10700 1 --2- 172.21.15.189:0/2011308363 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a8c0cb870 0x7f8a8c153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.232+0000 7f8a91a10700 1 --2- 172.21.15.189:0/2011308363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a8c0cfa00 0x7f8a8c158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.232+0000 7f8a91a10700 1 -- 172.21.15.189:0/2011308363 >> 172.21.15.189:0/2011308363 conn(0x7f8a8c0b2db0 msgr2=0x7f8a8c159d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:25.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.232+0000 7f8a91a10700 1 -- 172.21.15.189:0/2011308363 shutdown_connections 2024-04-06T13:51:25.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.232+0000 7f8a91a10700 1 -- 172.21.15.189:0/2011308363 wait complete. 2024-04-06T13:51:25.247 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T13:49:51.467569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '282', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 38, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T13:49:52.656784+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '283', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:51:25.247 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:51:25.263 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:25.262+0000 7f72d02d9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:51:25.307 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:25.306+0000 7f3fecaf2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T13:51:25.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.376+0000 7f1510032700 1 Processor -- start 2024-04-06T13:51:25.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.376+0000 7f1510032700 1 -- start start 2024-04-06T13:51:25.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.377+0000 7f1510032700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f150806ae40 0x7f15080b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:25.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.377+0000 7f1510032700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150815ddf0 0x7f150815e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:25.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.377+0000 7f1510032700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15080b3990 con 0x7f150815d7b0 2024-04-06T13:51:25.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.377+0000 7f1510032700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15080b3060 con 0x7f150806ae40 2024-04-06T13:51:25.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.377+0000 7f1510032700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15080b3340 con 0x7f150815ddf0 2024-04-06T13:51:25.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.377+0000 7f150ddce700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f150815d7b0 0x7f150806a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34494/0 (socket says 172.21.15.189:34494) 2024-04-06T13:51:25.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.377+0000 7f150ddce700 1 -- 172.21.15.189:0/2845173585 learned_addr learned my addr 172.21.15.189:0/2845173585 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:25.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.377+0000 7f150e5cf700 1 --2- 172.21.15.189:0/2845173585 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150815ddf0 0x7f150815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:25.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.377+0000 7f150d5cd700 1 --2- 172.21.15.189:0/2845173585 >> v2:172.21.15.192:3300/0 conn(0x7f150806ae40 0x7f15080b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:25.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.377+0000 7f150cdcc700 1 -- 172.21.15.189:0/2845173585 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322884566 0 0) 0x7f15080b3990 con 0x7f150815d7b0 2024-04-06T13:51:25.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.377+0000 7f150cdcc700 1 -- 172.21.15.189:0/2845173585 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14f40035b0 con 0x7f150815d7b0 2024-04-06T13:51:25.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.377+0000 7f150cdcc700 1 -- 172.21.15.189:0/2845173585 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1082850511 0 0) 0x7f14f40035b0 con 0x7f150815d7b0 2024-04-06T13:51:25.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.377+0000 7f150d5cd700 1 -- 172.21.15.189:0/2845173585 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150815ddf0 msgr2=0x7f150815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:25.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.377+0000 7f150d5cd700 1 --2- 172.21.15.189:0/2845173585 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150815ddf0 0x7f150815e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.377+0000 7f150d5cd700 1 -- 172.21.15.189:0/2845173585 >> v1:172.21.15.189:6789/0 conn(0x7f150815d7b0 legacy=0x7f150806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:25.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.377+0000 7f150d5cd700 1 -- 172.21.15.189:0/2845173585 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15080b3620 con 0x7f150806ae40 2024-04-06T13:51:25.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.377+0000 7f150d5cd700 1 --2- 172.21.15.189:0/2845173585 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f150806ae40 0x7f15080b3e20 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f14f8009840 tx=0x7f14f8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f77adf8e69ee5aae server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:25.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.378+0000 7f150cdcc700 1 -- 172.21.15.189:0/2845173585 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1504001670 con 0x7f150806ae40 2024-04-06T13:51:25.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.378+0000 7f150cdcc700 1 -- 172.21.15.189:0/2845173585 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f8004030 con 0x7f150806ae40 2024-04-06T13:51:25.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.378+0000 7f150cdcc700 1 -- 172.21.15.189:0/2845173585 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f80043c0 con 0x7f150806ae40 2024-04-06T13:51:25.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.378+0000 7f1510032700 1 -- 172.21.15.189:0/2845173585 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f150806ae40 msgr2=0x7f15080b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:25.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.378+0000 7f1510032700 1 --2- 172.21.15.189:0/2845173585 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f150806ae40 0x7f15080b3e20 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f14f8009840 tx=0x7f14f8009b50 comp rx=0 tx=0).stop 2024-04-06T13:51:25.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.378+0000 7f1510032700 1 -- 172.21.15.189:0/2845173585 shutdown_connections 2024-04-06T13:51:25.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.378+0000 7f1510032700 1 --2- 172.21.15.189:0/2845173585 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150815ddf0 0x7f150815e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.378+0000 7f1510032700 1 --2- 172.21.15.189:0/2845173585 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f150806ae40 0x7f15080b3e20 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.378+0000 7f1510032700 1 -- 172.21.15.189:0/2845173585 >> 172.21.15.189:0/2845173585 conn(0x7f15080d00b0 msgr2=0x7f15080d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:25.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.378+0000 7f1510032700 1 -- 172.21.15.189:0/2845173585 shutdown_connections 2024-04-06T13:51:25.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.378+0000 7f1510032700 1 -- 172.21.15.189:0/2845173585 wait complete. 2024-04-06T13:51:25.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.380+0000 7f1510032700 1 Processor -- start 2024-04-06T13:51:25.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.380+0000 7f1510032700 1 -- start start 2024-04-06T13:51:25.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.380+0000 7f1510032700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f150806ae40 0x7f15081f2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.380+0000 7f1510032700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150815d7b0 0x7f1508153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.380+0000 7f1510032700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150815ddf0 0x7f15080aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.380+0000 7f1510032700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15080b03f0 con 0x7f150815d7b0 2024-04-06T13:51:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.380+0000 7f1510032700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1508157200 con 0x7f150806ae40 2024-04-06T13:51:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.380+0000 7f1510032700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15080af870 con 0x7f150815ddf0 2024-04-06T13:51:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.380+0000 7f150d5cd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150815d7b0 0x7f1508153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.380+0000 7f150d5cd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150815d7b0 0x7f1508153ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54502/0 (socket says 172.21.15.189:54502) 2024-04-06T13:51:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.380+0000 7f150d5cd700 1 -- 172.21.15.189:0/2325862695 learned_addr learned my addr 172.21.15.189:0/2325862695 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.380+0000 7f150ddce700 1 --2- 172.21.15.189:0/2325862695 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f150806ae40 0x7f15081f2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.380+0000 7f150e5cf700 1 --2- 172.21.15.189:0/2325862695 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150815ddf0 0x7f15080aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.381+0000 7f150d5cd700 1 -- 172.21.15.189:0/2325862695 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150815ddf0 msgr2=0x7f15080aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.381+0000 7f150d5cd700 1 --2- 172.21.15.189:0/2325862695 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150815ddf0 0x7f15080aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.381+0000 7f150d5cd700 1 -- 172.21.15.189:0/2325862695 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f150806ae40 msgr2=0x7f15081f2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:25.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.381+0000 7f150d5cd700 1 --2- 172.21.15.189:0/2325862695 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f150806ae40 0x7f15081f2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.381+0000 7f150d5cd700 1 -- 172.21.15.189:0/2325862695 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f80094a0 con 0x7f150815d7b0 2024-04-06T13:51:25.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.381+0000 7f150d5cd700 1 --2- 172.21.15.189:0/2325862695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150815d7b0 0x7f1508153ca0 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 crypto rx=0x7f14f800a010 tx=0x7f14f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:25.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.381+0000 7f14fe7fc700 1 -- 172.21.15.189:0/2325862695 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f8003cf0 con 0x7f150815d7b0 2024-04-06T13:51:25.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.381+0000 7f14fe7fc700 1 -- 172.21.15.189:0/2325862695 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f801f070 con 0x7f150815d7b0 2024-04-06T13:51:25.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.381+0000 7f1510032700 1 -- 172.21.15.189:0/2325862695 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15080afb50 con 0x7f150815d7b0 2024-04-06T13:51:25.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.381+0000 7f14fe7fc700 1 -- 172.21.15.189:0/2325862695 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f8011870 con 0x7f150815d7b0 2024-04-06T13:51:25.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.381+0000 7f1510032700 1 -- 172.21.15.189:0/2325862695 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15080b0110 con 0x7f150815d7b0 2024-04-06T13:51:25.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.382+0000 7f1510032700 1 -- 172.21.15.189:0/2325862695 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1508067bb0 con 0x7f150815d7b0 2024-04-06T13:51:25.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.383+0000 7f14fe7fc700 1 -- 172.21.15.189:0/2325862695 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 177) v1 ==== 89412+0+0 (secure 0 0 0) 0x7f14f80119d0 con 0x7f150815d7b0 2024-04-06T13:51:25.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.383+0000 7f14fe7fc700 1 --2- 172.21.15.189:0/2325862695 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f14ec06af60 0x7f14ec06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:25.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.383+0000 7f150ddce700 1 -- 172.21.15.189:0/2325862695 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f14ec06af60 msgr2=0x7f14ec06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:25.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.383+0000 7f14fe7fc700 1 -- 172.21.15.189:0/2325862695 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f14f8021070 con 0x7f150815d7b0 2024-04-06T13:51:25.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.383+0000 7f150ddce700 1 --2- 172.21.15.189:0/2325862695 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f14ec06af60 0x7f14ec06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:51:25.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.387+0000 7f14fe7fc700 1 -- 172.21.15.189:0/2325862695 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f14f80be650 con 0x7f150815d7b0 2024-04-06T13:51:25.439 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:25.438+0000 7f72d02d9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:51:25.494 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:25.493+0000 7f3fecaf2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:51:25.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.584+0000 7f150ddce700 1 -- 172.21.15.189:0/2325862695 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f14ec06af60 msgr2=0x7f14ec06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:25.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.584+0000 7f150ddce700 1 --2- 172.21.15.189:0/2325862695 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f14ec06af60 0x7f14ec06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:51:25.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.592+0000 7f1510032700 1 -- 172.21.15.189:0/2325862695 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1508074990 con 0x7f150815d7b0 2024-04-06T13:51:25.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.592+0000 7f14fe7fc700 1 -- 172.21.15.189:0/2325862695 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v286) v1 ==== 93+0+10 (secure 0 0 0) 0x7f14f8055da0 con 0x7f150815d7b0 2024-04-06T13:51:25.593 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:51:25.595 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:25.594+0000 7f72d02d9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:51:25.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.595+0000 7f1510032700 1 -- 172.21.15.189:0/2325862695 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f14ec06af60 msgr2=0x7f14ec06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:51:25.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.595+0000 7f1510032700 1 --2- 172.21.15.189:0/2325862695 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f14ec06af60 0x7f14ec06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.595+0000 7f1510032700 1 -- 172.21.15.189:0/2325862695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150815d7b0 msgr2=0x7f1508153ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:25.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.595+0000 7f1510032700 1 --2- 172.21.15.189:0/2325862695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150815d7b0 0x7f1508153ca0 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 crypto rx=0x7f14f800a010 tx=0x7f14f8004750 comp rx=0 tx=0).stop 2024-04-06T13:51:25.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.595+0000 7f1510032700 1 -- 172.21.15.189:0/2325862695 shutdown_connections 2024-04-06T13:51:25.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.595+0000 7f1510032700 1 --2- 172.21.15.189:0/2325862695 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f14ec06af60 0x7f14ec06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.595+0000 7f1510032700 1 --2- 172.21.15.189:0/2325862695 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f150815ddf0 0x7f15080aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.595+0000 7f1510032700 1 --2- 172.21.15.189:0/2325862695 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f150806ae40 0x7f15081f2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.595+0000 7f1510032700 1 --2- 172.21.15.189:0/2325862695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150815d7b0 0x7f1508153ca0 unknown :-1 s=CLOSED pgs=3150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.595+0000 7f1510032700 1 -- 172.21.15.189:0/2325862695 >> 172.21.15.189:0/2325862695 conn(0x7f15080d00b0 msgr2=0x7f150815f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:25.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.595+0000 7f1510032700 1 -- 172.21.15.189:0/2325862695 shutdown_connections 2024-04-06T13:51:25.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.595+0000 7f1510032700 1 -- 172.21.15.189:0/2325862695 wait complete. 2024-04-06T13:51:25.609 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T13:51:25.662 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:25.661+0000 7f3fecaf2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T13:51:25.705 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:25.703+0000 7f72d02d9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:51:25.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.738+0000 7f9076e11700 1 Processor -- start 2024-04-06T13:51:25.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.738+0000 7f9076e11700 1 -- start start 2024-04-06T13:51:25.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.739+0000 7f9076e11700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f90700b4660 0x7f90700b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:25.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.739+0000 7f9076e11700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90700b5030 con 0x7f907015a550 2024-04-06T13:51:25.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.739+0000 7f9076e11700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90700cdbe0 con 0x7f90700b2fc0 2024-04-06T13:51:25.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.739+0000 7f9076e11700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f907015a940 con 0x7f90700b4660 2024-04-06T13:51:25.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.739+0000 7f9074bad700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f907015a550 0x7f90700cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34498/0 (socket says 172.21.15.189:34498) 2024-04-06T13:51:25.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.739+0000 7f906ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f90700b4660 0x7f90700b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:25.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.739+0000 7f9074bad700 1 -- 172.21.15.189:0/2772182025 learned_addr learned my addr 172.21.15.189:0/2772182025 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:25.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.739+0000 7f906ffff700 1 -- 172.21.15.189:0/2772182025 >> v1:172.21.15.192:6789/0 conn(0x7f90700b2fc0 legacy=0x7f90700b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:25.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.739+0000 7f906ffff700 1 -- 172.21.15.189:0/2772182025 >> v1:172.21.15.189:6789/0 conn(0x7f907015a550 legacy=0x7f90700cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:25.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.739+0000 7f906ffff700 1 -- 172.21.15.189:0/2772182025 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907015ac00 con 0x7f90700b4660 2024-04-06T13:51:25.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.740+0000 7f906ffff700 1 --2- 172.21.15.189:0/2772182025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90700b4660 0x7f90700b4af0 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 crypto rx=0x7f905c009840 tx=0x7f905c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbd1b248d8dc5612 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:25.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.740+0000 7f906f7fe700 1 -- 172.21.15.189:0/2772182025 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c011830 con 0x7f90700b4660 2024-04-06T13:51:25.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.740+0000 7f906f7fe700 1 -- 172.21.15.189:0/2772182025 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905c011990 con 0x7f90700b4660 2024-04-06T13:51:25.741 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.740+0000 7f906f7fe700 1 -- 172.21.15.189:0/2772182025 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c011d20 con 0x7f90700b4660 2024-04-06T13:51:25.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.740+0000 7f9076e11700 1 -- 172.21.15.189:0/2772182025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90700b4660 msgr2=0x7f90700b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:25.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.740+0000 7f9076e11700 1 --2- 172.21.15.189:0/2772182025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90700b4660 0x7f90700b4af0 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 crypto rx=0x7f905c009840 tx=0x7f905c009b50 comp rx=0 tx=0).stop 2024-04-06T13:51:25.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.740+0000 7f9076e11700 1 -- 172.21.15.189:0/2772182025 shutdown_connections 2024-04-06T13:51:25.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.740+0000 7f9076e11700 1 --2- 172.21.15.189:0/2772182025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90700b4660 0x7f90700b4af0 unknown :-1 s=CLOSED pgs=3152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.740+0000 7f9076e11700 1 -- 172.21.15.189:0/2772182025 >> 172.21.15.189:0/2772182025 conn(0x7f90700d00b0 msgr2=0x7f90700d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:25.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.740+0000 7f9076e11700 1 -- 172.21.15.189:0/2772182025 shutdown_connections 2024-04-06T13:51:25.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.740+0000 7f9076e11700 1 -- 172.21.15.189:0/2772182025 wait complete. 2024-04-06T13:51:25.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.742+0000 7f9076e11700 1 Processor -- start 2024-04-06T13:51:25.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.742+0000 7f9076e11700 1 -- start start 2024-04-06T13:51:25.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.743+0000 7f9076e11700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90700b2fc0 0x7f907013f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:25.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.743+0000 7f9076e11700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90700b4660 0x7f9070142740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:25.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.743+0000 7f9076e11700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f907006d610 0x7f907006daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:25.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.743+0000 7f9076e11700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907006dfe0 con 0x7f90700b2fc0 2024-04-06T13:51:25.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.743+0000 7f9076e11700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9070142c80 con 0x7f90700b4660 2024-04-06T13:51:25.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.743+0000 7f9076e11700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f907006f760 con 0x7f907006d610 2024-04-06T13:51:25.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.743+0000 7f9074bad700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90700b2fc0 0x7f907013f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:25.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.743+0000 7f9074bad700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90700b2fc0 0x7f907013f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54522/0 (socket says 172.21.15.189:54522) 2024-04-06T13:51:25.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.743+0000 7f9074bad700 1 -- 172.21.15.189:0/3971394832 learned_addr learned my addr 172.21.15.189:0/3971394832 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:25.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.743+0000 7f906ffff700 1 --2- 172.21.15.189:0/3971394832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90700b4660 0x7f9070142740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:25.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.743+0000 7f90753ae700 1 --2- 172.21.15.189:0/3971394832 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f907006d610 0x7f907006daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:25.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.743+0000 7f9074bad700 1 -- 172.21.15.189:0/3971394832 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f907006d610 msgr2=0x7f907006daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:25.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.743+0000 7f9074bad700 1 --2- 172.21.15.189:0/3971394832 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f907006d610 0x7f907006daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.743+0000 7f9074bad700 1 -- 172.21.15.189:0/3971394832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90700b4660 msgr2=0x7f9070142740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:25.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.743+0000 7f9074bad700 1 --2- 172.21.15.189:0/3971394832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90700b4660 0x7f9070142740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.743+0000 7f9074bad700 1 -- 172.21.15.189:0/3971394832 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905c0094a0 con 0x7f90700b2fc0 2024-04-06T13:51:25.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.744+0000 7f90753ae700 1 --2- 172.21.15.189:0/3971394832 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f907006d610 0x7f907006daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:51:25.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.744+0000 7f9074bad700 1 --2- 172.21.15.189:0/3971394832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90700b2fc0 0x7f907013f560 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7f906400c480 tx=0x7f906400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:25.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.744+0000 7f906d7fa700 1 -- 172.21.15.189:0/3971394832 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064005d70 con 0x7f90700b2fc0 2024-04-06T13:51:25.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.744+0000 7f9076e11700 1 -- 172.21.15.189:0/3971394832 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f907006fa40 con 0x7f90700b2fc0 2024-04-06T13:51:25.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.744+0000 7f906d7fa700 1 -- 172.21.15.189:0/3971394832 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90640070d0 con 0x7f90700b2fc0 2024-04-06T13:51:25.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.744+0000 7f906d7fa700 1 -- 172.21.15.189:0/3971394832 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064007440 con 0x7f90700b2fc0 2024-04-06T13:51:25.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.744+0000 7f9076e11700 1 -- 172.21.15.189:0/3971394832 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90701558b0 con 0x7f90700b2fc0 2024-04-06T13:51:25.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.745+0000 7f9076e11700 1 -- 172.21.15.189:0/3971394832 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9070067bb0 con 0x7f90700b2fc0 2024-04-06T13:51:25.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.746+0000 7f906d7fa700 1 -- 172.21.15.189:0/3971394832 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 177) v1 ==== 89412+0+0 (secure 0 0 0) 0x7f90640075a0 con 0x7f90700b2fc0 2024-04-06T13:51:25.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.747+0000 7f906d7fa700 1 --2- 172.21.15.189:0/3971394832 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f905406af60 0x7f905406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:25.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.747+0000 7f906ffff700 1 -- 172.21.15.189:0/3971394832 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f905406af60 msgr2=0x7f905406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:25.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.747+0000 7f906d7fa700 1 -- 172.21.15.189:0/3971394832 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f90640892d0 con 0x7f90700b2fc0 2024-04-06T13:51:25.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.747+0000 7f906ffff700 1 --2- 172.21.15.189:0/3971394832 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f905406af60 0x7f905406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:51:25.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.750+0000 7f906d7fa700 1 -- 172.21.15.189:0/3971394832 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9064053f90 con 0x7f90700b2fc0 2024-04-06T13:51:25.781 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:25.780+0000 7f3fecaf2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T13:51:25.787 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:25.786+0000 7f72d02d9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:51:25.863 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:25.862+0000 7f72d02d9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:51:25.866 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:25.865+0000 7f3fecaf2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T13:51:25.944 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:25.943+0000 7f3fecaf2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T13:51:25.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.948+0000 7f906ffff700 1 -- 172.21.15.189:0/3971394832 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f905406af60 msgr2=0x7f905406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:25.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.948+0000 7f906ffff700 1 --2- 172.21.15.189:0/3971394832 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f905406af60 0x7f905406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:51:25.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.951+0000 7f9076e11700 1 -- 172.21.15.189:0/3971394832 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9070074990 con 0x7f90700b2fc0 2024-04-06T13:51:25.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.952+0000 7f906d7fa700 1 -- 172.21.15.189:0/3971394832 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v286) v1 ==== 104+0+11 (secure 0 0 0) 0x7f90640575b0 con 0x7f90700b2fc0 2024-04-06T13:51:25.953 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:51:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.954+0000 7f9076e11700 1 -- 172.21.15.189:0/3971394832 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f905406af60 msgr2=0x7f905406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:51:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.954+0000 7f9076e11700 1 --2- 172.21.15.189:0/3971394832 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f905406af60 0x7f905406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.954+0000 7f9076e11700 1 -- 172.21.15.189:0/3971394832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90700b2fc0 msgr2=0x7f907013f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.954+0000 7f9076e11700 1 --2- 172.21.15.189:0/3971394832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90700b2fc0 0x7f907013f560 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7f906400c480 tx=0x7f906400c790 comp rx=0 tx=0).stop 2024-04-06T13:51:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.954+0000 7f9076e11700 1 -- 172.21.15.189:0/3971394832 shutdown_connections 2024-04-06T13:51:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.954+0000 7f9076e11700 1 --2- 172.21.15.189:0/3971394832 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f905406af60 0x7f905406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.954+0000 7f9076e11700 1 --2- 172.21.15.189:0/3971394832 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f907006d610 0x7f907006daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.954+0000 7f9076e11700 1 --2- 172.21.15.189:0/3971394832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90700b2fc0 0x7f907013f560 unknown :-1 s=CLOSED pgs=3153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.954+0000 7f9076e11700 1 --2- 172.21.15.189:0/3971394832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90700b4660 0x7f9070142740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:25.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.954+0000 7f9076e11700 1 -- 172.21.15.189:0/3971394832 >> 172.21.15.189:0/3971394832 conn(0x7f90700d00b0 msgr2=0x7f9070157c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:25.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.954+0000 7f9076e11700 1 -- 172.21.15.189:0/3971394832 shutdown_connections 2024-04-06T13:51:25.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:25.954+0000 7f9076e11700 1 -- 172.21.15.189:0/3971394832 wait complete. 2024-04-06T13:51:25.968 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T13:51:26.068 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:26.066+0000 7f72d02d9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:51:26.100 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.099+0000 7f22db785700 1 Processor -- start 2024-04-06T13:51:26.101 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.099+0000 7f22db785700 1 -- start start 2024-04-06T13:51:26.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.100+0000 7f22db785700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f22d415d590 0x7f22d40b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:26.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.100+0000 7f22db785700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22d40b4ed0 0x7f22d40b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:26.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.100+0000 7f22db785700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d40b5cc0 con 0x7f22d40b6580 2024-04-06T13:51:26.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.100+0000 7f22db785700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22d40b5890 con 0x7f22d415d590 2024-04-06T13:51:26.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.100+0000 7f22db785700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22d40d1500 con 0x7f22d40b4ed0 2024-04-06T13:51:26.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.100+0000 7f22d9521700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f22d415d590 0x7f22d40b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:26.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.100+0000 7f22d9521700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f22d415d590 0x7f22d40b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54524/0 (socket says 172.21.15.189:54524) 2024-04-06T13:51:26.102 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.100+0000 7f22d9521700 1 -- 172.21.15.189:0/1260327962 learned_addr learned my addr 172.21.15.189:0/1260327962 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:26.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.100+0000 7f22d8d20700 1 --2- 172.21.15.189:0/1260327962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22d40b4ed0 0x7f22d40b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:26.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.101+0000 7f22d9521700 1 -- 172.21.15.189:0/1260327962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22d40b4ed0 msgr2=0x7f22d40b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:26.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.101+0000 7f22d9521700 1 --2- 172.21.15.189:0/1260327962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22d40b4ed0 0x7f22d40b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.101+0000 7f22d9521700 1 -- 172.21.15.189:0/1260327962 >> v1:172.21.15.192:6789/0 conn(0x7f22d40b6580 legacy=0x7f22d40b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:26.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.101+0000 7f22d9521700 1 -- 172.21.15.189:0/1260327962 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d40d17e0 con 0x7f22d415d590 2024-04-06T13:51:26.103 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.101+0000 7f22d9521700 1 --2- 172.21.15.189:0/1260327962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22d415d590 0x7f22d40b8d70 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 crypto rx=0x7f22d0009920 tx=0x7f22d0009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11056b5eb1aa35d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:26.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.101+0000 7f22cbfff700 1 -- 172.21.15.189:0/1260327962 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d000ba20 con 0x7f22d415d590 2024-04-06T13:51:26.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.102+0000 7f22cbfff700 1 -- 172.21.15.189:0/1260327962 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d000bb80 con 0x7f22d415d590 2024-04-06T13:51:26.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.102+0000 7f22cbfff700 1 -- 172.21.15.189:0/1260327962 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d0009e70 con 0x7f22d415d590 2024-04-06T13:51:26.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.102+0000 7f22db785700 1 -- 172.21.15.189:0/1260327962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22d415d590 msgr2=0x7f22d40b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:26.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.102+0000 7f22db785700 1 --2- 172.21.15.189:0/1260327962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22d415d590 0x7f22d40b8d70 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 crypto rx=0x7f22d0009920 tx=0x7f22d0009c30 comp rx=0 tx=0).stop 2024-04-06T13:51:26.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.102+0000 7f22db785700 1 -- 172.21.15.189:0/1260327962 shutdown_connections 2024-04-06T13:51:26.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.102+0000 7f22db785700 1 --2- 172.21.15.189:0/1260327962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22d415d590 0x7f22d40b8d70 unknown :-1 s=CLOSED pgs=3154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.102+0000 7f22db785700 1 --2- 172.21.15.189:0/1260327962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22d40b4ed0 0x7f22d40b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.102+0000 7f22db785700 1 -- 172.21.15.189:0/1260327962 >> 172.21.15.189:0/1260327962 conn(0x7f22d406bbc0 msgr2=0x7f22d406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:26.106 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.106+0000 7f22db785700 1 -- 172.21.15.189:0/1260327962 shutdown_connections 2024-04-06T13:51:26.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.106+0000 7f22db785700 1 -- 172.21.15.189:0/1260327962 wait complete. 2024-04-06T13:51:26.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.108+0000 7f22db785700 1 Processor -- start 2024-04-06T13:51:26.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.108+0000 7f22db785700 1 -- start start 2024-04-06T13:51:26.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.109+0000 7f22db785700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22d40b6580 0x7f22d415d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:26.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.109+0000 7f22db785700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22d415d590 0x7f22d415fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:26.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.109+0000 7f22db785700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22d4204420 0x7f22d4160080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:26.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.109+0000 7f22db785700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d415edd0 con 0x7f22d4204420 2024-04-06T13:51:26.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.109+0000 7f22db785700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d415e1c0 con 0x7f22d40b6580 2024-04-06T13:51:26.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.109+0000 7f22db785700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22d415e4a0 con 0x7f22d415d590 2024-04-06T13:51:26.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.109+0000 7f22d9521700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22d40b6580 0x7f22d415d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:26.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.109+0000 7f22d9521700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22d40b6580 0x7f22d415d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.189:47140/0 (socket says 172.21.15.189:47140) 2024-04-06T13:51:26.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.109+0000 7f22d9521700 1 -- 172.21.15.189:0/1036192895 learned_addr learned my addr 172.21.15.189:0/1036192895 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:26.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.109+0000 7f22d8d20700 1 --2- 172.21.15.189:0/1036192895 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22d415d590 0x7f22d415fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:26.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.109+0000 7f22d9d22700 1 --2- 172.21.15.189:0/1036192895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22d4204420 0x7f22d4160080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:26.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.110+0000 7f22d9521700 1 -- 172.21.15.189:0/1036192895 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22d415d590 msgr2=0x7f22d415fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:26.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.110+0000 7f22d9521700 1 --2- 172.21.15.189:0/1036192895 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22d415d590 0x7f22d415fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.110+0000 7f22d9521700 1 -- 172.21.15.189:0/1036192895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22d4204420 msgr2=0x7f22d4160080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:26.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.110+0000 7f22d9521700 1 --2- 172.21.15.189:0/1036192895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22d4204420 0x7f22d4160080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.110+0000 7f22d9521700 1 -- 172.21.15.189:0/1036192895 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d0009550 con 0x7f22d40b6580 2024-04-06T13:51:26.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.110+0000 7f22d9521700 1 --2- 172.21.15.189:0/1036192895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22d40b6580 0x7f22d415d970 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f22d0014040 tx=0x7f22d0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:26.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.110+0000 7f22c9ffb700 1 -- 172.21.15.189:0/1036192895 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d00044d0 con 0x7f22d40b6580 2024-04-06T13:51:26.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.110+0000 7f22db785700 1 -- 172.21.15.189:0/1036192895 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22d415e780 con 0x7f22d40b6580 2024-04-06T13:51:26.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.111+0000 7f22db785700 1 -- 172.21.15.189:0/1036192895 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22d415deb0 con 0x7f22d40b6580 2024-04-06T13:51:26.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.111+0000 7f22c9ffb700 1 -- 172.21.15.189:0/1036192895 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d001e070 con 0x7f22d40b6580 2024-04-06T13:51:26.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.111+0000 7f22c9ffb700 1 -- 172.21.15.189:0/1036192895 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d00038e0 con 0x7f22d40b6580 2024-04-06T13:51:26.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.112+0000 7f22db785700 1 -- 172.21.15.189:0/1036192895 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22d4067080 con 0x7f22d40b6580 2024-04-06T13:51:26.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.113+0000 7f22c9ffb700 1 -- 172.21.15.189:0/1036192895 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 177) v1 ==== 89412+0+0 (secure 0 0 0) 0x7f22d0003a40 con 0x7f22d40b6580 2024-04-06T13:51:26.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.113+0000 7f22c9ffb700 1 --2- 172.21.15.189:0/1036192895 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f22b806af60 0x7f22b806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:26.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.113+0000 7f22c9ffb700 1 -- 172.21.15.189:0/1036192895 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f22d0020070 con 0x7f22d40b6580 2024-04-06T13:51:26.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.113+0000 7f22d8d20700 1 -- 172.21.15.189:0/1036192895 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f22b806af60 msgr2=0x7f22b806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:26.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.114+0000 7f22d8d20700 1 --2- 172.21.15.189:0/1036192895 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f22b806af60 0x7f22b806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:51:26.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.122+0000 7f22c9ffb700 1 -- 172.21.15.189:0/1036192895 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f22d0055a10 con 0x7f22d40b6580 2024-04-06T13:51:26.149 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:26.147+0000 7f72d02d9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:51:26.155 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:26.154+0000 7f3fecaf2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T13:51:26.235 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:26.234+0000 7f3fecaf2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T13:51:26.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.315+0000 7f22d8d20700 1 -- 172.21.15.189:0/1036192895 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f22b806af60 msgr2=0x7f22b806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:26.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.315+0000 7f22d8d20700 1 --2- 172.21.15.189:0/1036192895 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f22b806af60 0x7f22b806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:51:26.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.348+0000 7f22db785700 1 -- 172.21.15.189:0/1036192895 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f22d4074990 con 0x7f22d40b6580 2024-04-06T13:51:26.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.349+0000 7f22c9ffb700 1 -- 172.21.15.189:0/1036192895 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v286) v1 ==== 100+0+11 (secure 0 0 0) 0x7f22d00150a0 con 0x7f22d40b6580 2024-04-06T13:51:26.350 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:51:26.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.351+0000 7f22db785700 1 -- 172.21.15.189:0/1036192895 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f22b806af60 msgr2=0x7f22b806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:51:26.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.351+0000 7f22db785700 1 --2- 172.21.15.189:0/1036192895 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f22b806af60 0x7f22b806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.351+0000 7f22db785700 1 -- 172.21.15.189:0/1036192895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22d40b6580 msgr2=0x7f22d415d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:26.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.351+0000 7f22db785700 1 --2- 172.21.15.189:0/1036192895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22d40b6580 0x7f22d415d970 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f22d0014040 tx=0x7f22d0004750 comp rx=0 tx=0).stop 2024-04-06T13:51:26.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.351+0000 7f22db785700 1 -- 172.21.15.189:0/1036192895 shutdown_connections 2024-04-06T13:51:26.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.351+0000 7f22db785700 1 --2- 172.21.15.189:0/1036192895 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f22b806af60 0x7f22b806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.351+0000 7f22db785700 1 --2- 172.21.15.189:0/1036192895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22d4204420 0x7f22d4160080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.351+0000 7f22db785700 1 --2- 172.21.15.189:0/1036192895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22d40b6580 0x7f22d415d970 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.351+0000 7f22db785700 1 --2- 172.21.15.189:0/1036192895 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22d415d590 0x7f22d415fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.351+0000 7f22db785700 1 -- 172.21.15.189:0/1036192895 >> 172.21.15.189:0/1036192895 conn(0x7f22d406bbc0 msgr2=0x7f22d41596d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:26.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.351+0000 7f22db785700 1 -- 172.21.15.189:0/1036192895 shutdown_connections 2024-04-06T13:51:26.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.351+0000 7f22db785700 1 -- 172.21.15.189:0/1036192895 wait complete. 2024-04-06T13:51:26.365 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-06T13:51:26.469 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:26.468+0000 7f72d02d9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:51:26.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.493+0000 7efdf5ffd700 1 Processor -- start 2024-04-06T13:51:26.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.493+0000 7efdf5ffd700 1 -- start start 2024-04-06T13:51:26.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.493+0000 7efdf5ffd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efdf00cb870 0x7efdf015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:26.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.493+0000 7efdf5ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efdf00cfa00 0x7efdf00cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:26.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.494+0000 7efdf5ffd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf00cbd80 con 0x7efdf00d11b0 2024-04-06T13:51:26.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.494+0000 7efdf5ffd700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf00ccdc0 con 0x7efdf00cfa00 2024-04-06T13:51:26.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.494+0000 7efdf5ffd700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf00d0b50 con 0x7efdf00cb870 2024-04-06T13:51:26.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.494+0000 7efdeffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efdf00cfa00 0x7efdf00cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:26.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.494+0000 7efdef7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efdf00d11b0 0x7efdf00d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34506/0 (socket says 172.21.15.189:34506) 2024-04-06T13:51:26.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.494+0000 7efdef7fe700 1 -- 172.21.15.189:0/1588865414 learned_addr learned my addr 172.21.15.189:0/1588865414 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:26.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.494+0000 7efdeeffd700 1 --2- 172.21.15.189:0/1588865414 >> v2:172.21.15.192:3300/0 conn(0x7efdf00cb870 0x7efdf015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:26.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.494+0000 7efdeffff700 1 -- 172.21.15.189:0/1588865414 >> v2:172.21.15.192:3300/0 conn(0x7efdf00cb870 msgr2=0x7efdf015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:26.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.494+0000 7efdeffff700 1 --2- 172.21.15.189:0/1588865414 >> v2:172.21.15.192:3300/0 conn(0x7efdf00cb870 0x7efdf015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.494+0000 7efdeffff700 1 -- 172.21.15.189:0/1588865414 >> v1:172.21.15.189:6789/0 conn(0x7efdf00d11b0 legacy=0x7efdf00d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:26.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.494+0000 7efdeffff700 1 -- 172.21.15.189:0/1588865414 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf00cef20 con 0x7efdf00cfa00 2024-04-06T13:51:26.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.494+0000 7efdee7fc700 1 -- 172.21.15.189:0/1588865414 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198790867 0 0) 0x7efdf00cbd80 con 0x7efdf00d11b0 2024-04-06T13:51:26.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.494+0000 7efdeffff700 1 --2- 172.21.15.189:0/1588865414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdf00cfa00 0x7efdf00cd560 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 crypto rx=0x7efdf004fb40 tx=0x7efde8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32259d2f3e4f74f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:26.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.494+0000 7efdee7fc700 1 -- 172.21.15.189:0/1588865414 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde8011830 con 0x7efdf00cfa00 2024-04-06T13:51:26.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.494+0000 7efdee7fc700 1 -- 172.21.15.189:0/1588865414 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde8011990 con 0x7efdf00cfa00 2024-04-06T13:51:26.497 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.494+0000 7efdee7fc700 1 -- 172.21.15.189:0/1588865414 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde8011d20 con 0x7efdf00cfa00 2024-04-06T13:51:26.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.495+0000 7efdf5ffd700 1 -- 172.21.15.189:0/1588865414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdf00cfa00 msgr2=0x7efdf00cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:26.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.495+0000 7efdf5ffd700 1 --2- 172.21.15.189:0/1588865414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdf00cfa00 0x7efdf00cd560 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 crypto rx=0x7efdf004fb40 tx=0x7efde8009b20 comp rx=0 tx=0).stop 2024-04-06T13:51:26.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.495+0000 7efdf5ffd700 1 -- 172.21.15.189:0/1588865414 shutdown_connections 2024-04-06T13:51:26.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.495+0000 7efdf5ffd700 1 --2- 172.21.15.189:0/1588865414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdf00cfa00 0x7efdf00cd560 unknown :-1 s=CLOSED pgs=3156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.495+0000 7efdf5ffd700 1 --2- 172.21.15.189:0/1588865414 >> v2:172.21.15.192:3300/0 conn(0x7efdf00cb870 0x7efdf015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.495+0000 7efdf5ffd700 1 -- 172.21.15.189:0/1588865414 >> 172.21.15.189:0/1588865414 conn(0x7efdf00b2db0 msgr2=0x7efdf00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:26.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.495+0000 7efdf5ffd700 1 -- 172.21.15.189:0/1588865414 shutdown_connections 2024-04-06T13:51:26.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.495+0000 7efdf5ffd700 1 -- 172.21.15.189:0/1588865414 wait complete. 2024-04-06T13:51:26.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.497+0000 7efdf5ffd700 1 Processor -- start 2024-04-06T13:51:26.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.497+0000 7efdf5ffd700 1 -- start start 2024-04-06T13:51:26.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdf5ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efdf00cb870 0x7efdf00b6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:26.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdf5ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdf00cfa00 0x7efdf0157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:26.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdf5ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdf00d11b0 0x7efdf0158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:26.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdf5ffd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf015aa90 con 0x7efdf00cfa00 2024-04-06T13:51:26.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdf5ffd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf0156390 con 0x7efdf00cb870 2024-04-06T13:51:26.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdf5ffd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf00b6ec0 con 0x7efdf00d11b0 2024-04-06T13:51:26.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdeeffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdf00cfa00 0x7efdf0157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:26.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdeeffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdf00cfa00 0x7efdf0157f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54552/0 (socket says 172.21.15.189:54552) 2024-04-06T13:51:26.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdeeffd700 1 -- 172.21.15.189:0/3817756087 learned_addr learned my addr 172.21.15.189:0/3817756087 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:26.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdef7fe700 1 --2- 172.21.15.189:0/3817756087 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efdf00cb870 0x7efdf00b6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:26.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdeffff700 1 --2- 172.21.15.189:0/3817756087 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdf00d11b0 0x7efdf0158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:26.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdeeffd700 1 -- 172.21.15.189:0/3817756087 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdf00d11b0 msgr2=0x7efdf0158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:26.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdeeffd700 1 --2- 172.21.15.189:0/3817756087 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdf00d11b0 0x7efdf0158450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdeeffd700 1 -- 172.21.15.189:0/3817756087 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efdf00cb870 msgr2=0x7efdf00b6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:26.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdeeffd700 1 --2- 172.21.15.189:0/3817756087 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efdf00cb870 0x7efdf00b6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdeeffd700 1 -- 172.21.15.189:0/3817756087 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde80094a0 con 0x7efdf00cfa00 2024-04-06T13:51:26.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdef7fe700 1 --2- 172.21.15.189:0/3817756087 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efdf00cb870 0x7efdf00b6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:51:26.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdeeffd700 1 --2- 172.21.15.189:0/3817756087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdf00cfa00 0x7efdf0157f10 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7efde400e840 tx=0x7efde400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:26.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdd7fff700 1 -- 172.21.15.189:0/3817756087 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde4009590 con 0x7efdf00cfa00 2024-04-06T13:51:26.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.499+0000 7efdd7fff700 1 -- 172.21.15.189:0/3817756087 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde4013680 con 0x7efdf00cfa00 2024-04-06T13:51:26.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.500+0000 7efdf5ffd700 1 -- 172.21.15.189:0/3817756087 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf014c440 con 0x7efdf00cfa00 2024-04-06T13:51:26.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.500+0000 7efdd7fff700 1 -- 172.21.15.189:0/3817756087 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde4013a30 con 0x7efdf00cfa00 2024-04-06T13:51:26.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.500+0000 7efdf5ffd700 1 -- 172.21.15.189:0/3817756087 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf00b5070 con 0x7efdf00cfa00 2024-04-06T13:51:26.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.501+0000 7efdf5ffd700 1 -- 172.21.15.189:0/3817756087 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf0156520 con 0x7efdf00cfa00 2024-04-06T13:51:26.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.501+0000 7efdd7fff700 1 -- 172.21.15.189:0/3817756087 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 177) v1 ==== 89412+0+0 (secure 0 0 0) 0x7efde4013b90 con 0x7efdf00cfa00 2024-04-06T13:51:26.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.501+0000 7efdd7fff700 1 --2- 172.21.15.189:0/3817756087 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efddc06af60 0x7efddc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:26.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.501+0000 7efdef7fe700 1 -- 172.21.15.189:0/3817756087 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efddc06af60 msgr2=0x7efddc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:26.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.502+0000 7efdd7fff700 1 -- 172.21.15.189:0/3817756087 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6532+0+0 (secure 0 0 0) 0x7efde4089930 con 0x7efdf00cfa00 2024-04-06T13:51:26.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.502+0000 7efdef7fe700 1 --2- 172.21.15.189:0/3817756087 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efddc06af60 0x7efddc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:51:26.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.506+0000 7efdd7fff700 1 -- 172.21.15.189:0/3817756087 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efde4054440 con 0x7efdf00cfa00 2024-04-06T13:51:26.577 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:26.576+0000 7f3fecaf2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T13:51:26.703 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.703+0000 7efdef7fe700 1 -- 172.21.15.189:0/3817756087 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efddc06af60 msgr2=0x7efddc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:26.704 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.703+0000 7efdef7fe700 1 --2- 172.21.15.189:0/3817756087 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efddc06af60 0x7efddc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:51:26.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.725+0000 7efdf5ffd700 1 -- 172.21.15.189:0/3817756087 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7efdf0074990 con 0x7efdf00cfa00 2024-04-06T13:51:26.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.726+0000 7efdd7fff700 1 -- 172.21.15.189:0/3817756087 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v286) v1 ==== 80+0+469 (secure 0 0 0) 0x7efde4057a60 con 0x7efdf00cfa00 2024-04-06T13:51:26.727 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2987126770 2024-04-07T13:50:07.256457+0000 2024-04-06T13:51:26.727 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2905400170 2024-04-07T13:50:07.256457+0000 2024-04-06T13:51:26.727 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/893866137 2024-04-07T13:50:07.256457+0000 2024-04-06T13:51:26.727 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/2221278556 2024-04-07T13:50:07.256457+0000 2024-04-06T13:51:26.727 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6833/111704 2024-04-07T13:50:07.256457+0000 2024-04-06T13:51:26.727 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:6832/111704 2024-04-07T13:50:07.256457+0000 2024-04-06T13:51:26.727 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/4009223680 2024-04-07T13:50:07.256457+0000 2024-04-06T13:51:26.727 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189:0/3345146368 2024-04-07T13:50:07.256457+0000 2024-04-06T13:51:26.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.728+0000 7efdf5ffd700 1 -- 172.21.15.189:0/3817756087 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efddc06af60 msgr2=0x7efddc06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:51:26.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.728+0000 7efdf5ffd700 1 --2- 172.21.15.189:0/3817756087 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efddc06af60 0x7efddc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.728+0000 7efdf5ffd700 1 -- 172.21.15.189:0/3817756087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdf00cfa00 msgr2=0x7efdf0157f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:26.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.728+0000 7efdf5ffd700 1 --2- 172.21.15.189:0/3817756087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdf00cfa00 0x7efdf0157f10 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7efde400e840 tx=0x7efde400eb50 comp rx=0 tx=0).stop 2024-04-06T13:51:26.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.728+0000 7efdf5ffd700 1 -- 172.21.15.189:0/3817756087 shutdown_connections 2024-04-06T13:51:26.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.728+0000 7efdf5ffd700 1 --2- 172.21.15.189:0/3817756087 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efddc06af60 0x7efddc06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.728+0000 7efdf5ffd700 1 --2- 172.21.15.189:0/3817756087 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efdf00d11b0 0x7efdf0158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.728+0000 7efdf5ffd700 1 --2- 172.21.15.189:0/3817756087 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efdf00cb870 0x7efdf00b6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.728+0000 7efdf5ffd700 1 --2- 172.21.15.189:0/3817756087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdf00cfa00 0x7efdf0157f10 unknown :-1 s=CLOSED pgs=3157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:26.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.728+0000 7efdf5ffd700 1 -- 172.21.15.189:0/3817756087 >> 172.21.15.189:0/3817756087 conn(0x7efdf00b2db0 msgr2=0x7efdf0159b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:26.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.728+0000 7efdf5ffd700 1 -- 172.21.15.189:0/3817756087 shutdown_connections 2024-04-06T13:51:26.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:26.728+0000 7efdf5ffd700 1 -- 172.21.15.189:0/3817756087 wait complete. 2024-04-06T13:51:26.731 INFO:teuthology.orchestra.run.smithi189.stderr:listed 8 entries 2024-04-06T13:51:26.742 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-06T13:51:26.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:51:26.742 DEBUG:teuthology.orchestra.run.smithi189:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-06T13:51:26.791 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T13:51:26.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:51:26.792 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-06T13:51:26.793 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:26.792+0000 7f72d02d9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:51:26.809 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:51:26.809 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:51:26.828 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:51:26.828 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:51:26.828 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:51:26.868 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:51:26.868 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:51:26.868 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:51:26.888 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:51:26.889 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:51:26.914 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:26.913+0000 7f3fecaf2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T13:51:26.963 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:26.962+0000 7f72d02d9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:51:27.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.031+0000 7fcf07552700 1 Processor -- start 2024-04-06T13:51:27.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.032+0000 7fcf07552700 1 -- start start 2024-04-06T13:51:27.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.032+0000 7fcf07552700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcf0015d590 0x7fcf000b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:27.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.032+0000 7fcf07552700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf000b4ed0 0x7fcf000b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:27.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.032+0000 7fcf07552700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcf000b6580 0x7fcf000b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:27.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.032+0000 7fcf07552700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf000b5890 con 0x7fcf000b6580 2024-04-06T13:51:27.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.032+0000 7fcf07552700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf000b5cc0 con 0x7fcf0015d590 2024-04-06T13:51:27.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.032+0000 7fcf07552700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf000d1500 con 0x7fcf000b4ed0 2024-04-06T13:51:27.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.033+0000 7fcf05aef700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcf000b6580 0x7fcf000b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:27.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.033+0000 7fcf05aef700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcf000b6580 0x7fcf000b6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54562/0 (socket says 172.21.15.189:54562) 2024-04-06T13:51:27.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.033+0000 7fcf05aef700 1 -- 172.21.15.189:0/1839876777 learned_addr learned my addr 172.21.15.189:0/1839876777 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:27.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.033+0000 7fcf04aed700 1 --2- 172.21.15.189:0/1839876777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf000b4ed0 0x7fcf000b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:27.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.033+0000 7fcf052ee700 1 --2- 172.21.15.189:0/1839876777 >> v2:172.21.15.192:3300/0 conn(0x7fcf0015d590 0x7fcf000b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:27.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.033+0000 7fcf05aef700 1 -- 172.21.15.189:0/1839876777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf000b4ed0 msgr2=0x7fcf000b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:27.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.033+0000 7fcf05aef700 1 --2- 172.21.15.189:0/1839876777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf000b4ed0 0x7fcf000b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.033+0000 7fcf05aef700 1 -- 172.21.15.189:0/1839876777 >> v2:172.21.15.192:3300/0 conn(0x7fcf0015d590 msgr2=0x7fcf000b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:27.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.033+0000 7fcf05aef700 1 --2- 172.21.15.189:0/1839876777 >> v2:172.21.15.192:3300/0 conn(0x7fcf0015d590 0x7fcf000b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.033+0000 7fcf05aef700 1 -- 172.21.15.189:0/1839876777 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf000d17e0 con 0x7fcf000b6580 2024-04-06T13:51:27.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.033+0000 7fcf05aef700 1 --2- 172.21.15.189:0/1839876777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf000b6580 0x7fcf000b6a10 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 crypto rx=0x7fcef8009e50 tx=0x7fcef800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e34365952f2563 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:27.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.033+0000 7fcef7fff700 1 -- 172.21.15.189:0/1839876777 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef800e070 con 0x7fcf000b6580 2024-04-06T13:51:27.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.033+0000 7fcef7fff700 1 -- 172.21.15.189:0/1839876777 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef800b8d0 con 0x7fcf000b6580 2024-04-06T13:51:27.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.033+0000 7fcef7fff700 1 -- 172.21.15.189:0/1839876777 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef800bc80 con 0x7fcf000b6580 2024-04-06T13:51:27.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.034+0000 7fcf07552700 1 -- 172.21.15.189:0/1839876777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf000b6580 msgr2=0x7fcf000b6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:27.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.034+0000 7fcf07552700 1 --2- 172.21.15.189:0/1839876777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf000b6580 0x7fcf000b6a10 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 crypto rx=0x7fcef8009e50 tx=0x7fcef800b330 comp rx=0 tx=0).stop 2024-04-06T13:51:27.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.034+0000 7fcf07552700 1 -- 172.21.15.189:0/1839876777 shutdown_connections 2024-04-06T13:51:27.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.034+0000 7fcf07552700 1 --2- 172.21.15.189:0/1839876777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf000b6580 0x7fcf000b6a10 unknown :-1 s=CLOSED pgs=3158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.034+0000 7fcf07552700 1 --2- 172.21.15.189:0/1839876777 >> v2:172.21.15.192:3300/0 conn(0x7fcf0015d590 0x7fcf000b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.034+0000 7fcf07552700 1 --2- 172.21.15.189:0/1839876777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf000b4ed0 0x7fcf000b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.034+0000 7fcf07552700 1 -- 172.21.15.189:0/1839876777 >> 172.21.15.189:0/1839876777 conn(0x7fcf0006bbc0 msgr2=0x7fcf0006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:27.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.034+0000 7fcf07552700 1 -- 172.21.15.189:0/1839876777 shutdown_connections 2024-04-06T13:51:27.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.034+0000 7fcf07552700 1 -- 172.21.15.189:0/1839876777 wait complete. 2024-04-06T13:51:27.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.036+0000 7fcf07552700 1 Processor -- start 2024-04-06T13:51:27.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.036+0000 7fcf07552700 1 -- start start 2024-04-06T13:51:27.041 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:27.035+0000 7f72d02d9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:51:27.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.036+0000 7fcf07552700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf000b4ed0 0x7fcf001f25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:27.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcf07552700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf000b6580 0x7fcf001564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:27.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcf07552700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf0015d590 0x7fcf001569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:27.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcf07552700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf001f2ea0 con 0x7fcf000b6580 2024-04-06T13:51:27.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcf07552700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf001f3180 con 0x7fcf000b4ed0 2024-04-06T13:51:27.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcf07552700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf001f3460 con 0x7fcf0015d590 2024-04-06T13:51:27.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcf04aed700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf000b6580 0x7fcf001564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:27.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcf04aed700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf000b6580 0x7fcf001564a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54564/0 (socket says 172.21.15.189:54564) 2024-04-06T13:51:27.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcf04aed700 1 -- 172.21.15.189:0/2804119279 learned_addr learned my addr 172.21.15.189:0/2804119279 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:27.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcf052ee700 1 --2- 172.21.15.189:0/2804119279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf000b4ed0 0x7fcf001f25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:27.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcf05aef700 1 --2- 172.21.15.189:0/2804119279 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf0015d590 0x7fcf001569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:27.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcf04aed700 1 -- 172.21.15.189:0/2804119279 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf0015d590 msgr2=0x7fcf001569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:27.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcf04aed700 1 --2- 172.21.15.189:0/2804119279 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf0015d590 0x7fcf001569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcf04aed700 1 -- 172.21.15.189:0/2804119279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf000b4ed0 msgr2=0x7fcf001f25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:27.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcf04aed700 1 --2- 172.21.15.189:0/2804119279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf000b4ed0 0x7fcf001f25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcf04aed700 1 -- 172.21.15.189:0/2804119279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef8009b30 con 0x7fcf000b6580 2024-04-06T13:51:27.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcf04aed700 1 --2- 172.21.15.189:0/2804119279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf000b6580 0x7fcf001564a0 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 crypto rx=0x7fcef000e870 tx=0x7fcef000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:27.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcef5ffb700 1 -- 172.21.15.189:0/2804119279 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef0009530 con 0x7fcf000b6580 2024-04-06T13:51:27.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcef5ffb700 1 -- 172.21.15.189:0/2804119279 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef0013680 con 0x7fcf000b6580 2024-04-06T13:51:27.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcef5ffb700 1 -- 172.21.15.189:0/2804119279 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef0013a30 con 0x7fcf000b6580 2024-04-06T13:51:27.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.037+0000 7fcf07552700 1 -- 172.21.15.189:0/2804119279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf001f3740 con 0x7fcf000b6580 2024-04-06T13:51:27.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.038+0000 7fcf07552700 1 -- 172.21.15.189:0/2804119279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf000cc0b0 con 0x7fcf000b6580 2024-04-06T13:51:27.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.039+0000 7fcf07552700 1 -- 172.21.15.189:0/2804119279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf001f3930 con 0x7fcf000b6580 2024-04-06T13:51:27.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.039+0000 7fcef5ffb700 1 -- 172.21.15.189:0/2804119279 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 177) v1 ==== 89412+0+0 (secure 0 0 0) 0x7fcef0013b90 con 0x7fcf000b6580 2024-04-06T13:51:27.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.039+0000 7fcef5ffb700 1 --2- 172.21.15.189:0/2804119279 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcee406af60 0x7fcee406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:27.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.039+0000 7fcf052ee700 1 -- 172.21.15.189:0/2804119279 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcee406af60 msgr2=0x7fcee406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:27.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.039+0000 7fcef5ffb700 1 -- 172.21.15.189:0/2804119279 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcef0089880 con 0x7fcf000b6580 2024-04-06T13:51:27.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.039+0000 7fcf052ee700 1 --2- 172.21.15.189:0/2804119279 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcee406af60 0x7fcee406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:51:27.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.044+0000 7fcef5ffb700 1 -- 172.21.15.189:0/2804119279 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fcef00543d0 con 0x7fcf000b6580 2024-04-06T13:51:27.087 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:27.085+0000 7f3fecaf2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T13:51:27.163 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:27.162+0000 7f3fecaf2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T13:51:27.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.240+0000 7fcf052ee700 1 -- 172.21.15.189:0/2804119279 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcee406af60 msgr2=0x7fcee406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:27.241 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.240+0000 7fcf052ee700 1 --2- 172.21.15.189:0/2804119279 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcee406af60 0x7fcee406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:51:27.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.293+0000 7fcf07552700 1 -- 172.21.15.189:0/2804119279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf001f2fe0 con 0x7fcf000b6580 2024-04-06T13:51:27.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.294+0000 7fcef5ffb700 1 -- 172.21.15.189:0/2804119279 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4759 (secure 0 0 0) 0x7fcf001f2fe0 con 0x7fcf000b6580 2024-04-06T13:51:27.295 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:51:27.295 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":139,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11827,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/819921213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":819921213},{"type":"v1","addr":"172.21.15.192:6835","nonce":819921213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134},{"gid":13368,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3616349784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3616349784},{"type":"v1","addr":"172.21.15.189:6836","nonce":3616349784}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134},{"gid":13371,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3640528525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3640528525},{"type":"v1","addr":"172.21.15.189:6837","nonce":3640528525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:49:53.688018+0000","modified":"2024-04-06T13:49:56.698920+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13380},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13380":{"gid":13380,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6834/3894184871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3894184871},{"type":"v1","addr":"172.21.15.192:6834","nonce":3894184871}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-04-06T13:51:27.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.296+0000 7fcf07552700 1 -- 172.21.15.189:0/2804119279 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcee406af60 msgr2=0x7fcee406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:51:27.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.296+0000 7fcf07552700 1 --2- 172.21.15.189:0/2804119279 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcee406af60 0x7fcee406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.296+0000 7fcf07552700 1 -- 172.21.15.189:0/2804119279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf000b6580 msgr2=0x7fcf001564a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:27.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.296+0000 7fcf07552700 1 --2- 172.21.15.189:0/2804119279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf000b6580 0x7fcf001564a0 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 crypto rx=0x7fcef000e870 tx=0x7fcef000eb80 comp rx=0 tx=0).stop 2024-04-06T13:51:27.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.296+0000 7fcf07552700 1 -- 172.21.15.189:0/2804119279 shutdown_connections 2024-04-06T13:51:27.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.296+0000 7fcf07552700 1 --2- 172.21.15.189:0/2804119279 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fcee406af60 0x7fcee406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.296+0000 7fcf07552700 1 --2- 172.21.15.189:0/2804119279 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf0015d590 0x7fcf001569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.296+0000 7fcf07552700 1 --2- 172.21.15.189:0/2804119279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf000b4ed0 0x7fcf001f25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.296+0000 7fcf07552700 1 --2- 172.21.15.189:0/2804119279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf000b6580 0x7fcf001564a0 unknown :-1 s=CLOSED pgs=3159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.296+0000 7fcf07552700 1 -- 172.21.15.189:0/2804119279 >> 172.21.15.189:0/2804119279 conn(0x7fcf0006bbc0 msgr2=0x7fcf000cd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:27.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.296+0000 7fcf07552700 1 -- 172.21.15.189:0/2804119279 shutdown_connections 2024-04-06T13:51:27.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.296+0000 7fcf07552700 1 -- 172.21.15.189:0/2804119279 wait complete. 2024-04-06T13:51:27.299 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 139 2024-04-06T13:51:27.310 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:51:27.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.484+0000 7f26bf58c700 1 Processor -- start 2024-04-06T13:51:27.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.485+0000 7f26bf58c700 1 -- start start 2024-04-06T13:51:27.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.485+0000 7f26bf58c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f26b815d7b0 0x7f26b80b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:27.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.485+0000 7f26bf58c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26b815e500 0x7f26b806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:27.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.485+0000 7f26bf58c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b80b3990 con 0x7f26b806a730 2024-04-06T13:51:27.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.485+0000 7f26bf58c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b806b0f0 con 0x7f26b815d7b0 2024-04-06T13:51:27.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.485+0000 7f26bf58c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b80b3060 con 0x7f26b815e500 2024-04-06T13:51:27.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.486+0000 7f26bd328700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f26b815d7b0 0x7f26b80b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:27.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.486+0000 7f26bd328700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f26b815d7b0 0x7f26b80b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54568/0 (socket says 172.21.15.189:54568) 2024-04-06T13:51:27.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.486+0000 7f26bd328700 1 -- 172.21.15.189:0/186653890 learned_addr learned my addr 172.21.15.189:0/186653890 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:27.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.486+0000 7f26bdb29700 1 --2- 172.21.15.189:0/186653890 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26b815e500 0x7f26b806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:27.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.486+0000 7f26bd328700 1 -- 172.21.15.189:0/186653890 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26b815e500 msgr2=0x7f26b806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:27.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.486+0000 7f26bd328700 1 --2- 172.21.15.189:0/186653890 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26b815e500 0x7f26b806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.486+0000 7f26bd328700 1 -- 172.21.15.189:0/186653890 >> v1:172.21.15.192:6789/0 conn(0x7f26b806a730 legacy=0x7f26b815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:27.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.486+0000 7f26bd328700 1 -- 172.21.15.189:0/186653890 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b80b3340 con 0x7f26b815d7b0 2024-04-06T13:51:27.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.486+0000 7f26bd328700 1 --2- 172.21.15.189:0/186653890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b815d7b0 0x7f26b80b3e20 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 crypto rx=0x7f26b40098f0 tx=0x7f26b4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b39e545f70f4078a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:27.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.486+0000 7f26affff700 1 -- 172.21.15.189:0/186653890 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b400b8a0 con 0x7f26b815d7b0 2024-04-06T13:51:27.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.486+0000 7f26affff700 1 -- 172.21.15.189:0/186653890 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b400ba00 con 0x7f26b815d7b0 2024-04-06T13:51:27.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.486+0000 7f26affff700 1 -- 172.21.15.189:0/186653890 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b400bd90 con 0x7f26b815d7b0 2024-04-06T13:51:27.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.487+0000 7f26bf58c700 1 -- 172.21.15.189:0/186653890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b815d7b0 msgr2=0x7f26b80b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:27.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.487+0000 7f26bf58c700 1 --2- 172.21.15.189:0/186653890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b815d7b0 0x7f26b80b3e20 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 crypto rx=0x7f26b40098f0 tx=0x7f26b4009c00 comp rx=0 tx=0).stop 2024-04-06T13:51:27.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.487+0000 7f26bf58c700 1 -- 172.21.15.189:0/186653890 shutdown_connections 2024-04-06T13:51:27.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.487+0000 7f26bf58c700 1 --2- 172.21.15.189:0/186653890 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26b815e500 0x7f26b806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.487+0000 7f26bf58c700 1 --2- 172.21.15.189:0/186653890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b815d7b0 0x7f26b80b3e20 unknown :-1 s=CLOSED pgs=3160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.487+0000 7f26bf58c700 1 -- 172.21.15.189:0/186653890 >> 172.21.15.189:0/186653890 conn(0x7f26b80d00b0 msgr2=0x7f26b80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:27.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.487+0000 7f26bf58c700 1 -- 172.21.15.189:0/186653890 shutdown_connections 2024-04-06T13:51:27.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.487+0000 7f26bf58c700 1 -- 172.21.15.189:0/186653890 wait complete. 2024-04-06T13:51:27.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.488+0000 7f26bf58c700 1 Processor -- start 2024-04-06T13:51:27.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.488+0000 7f26bf58c700 1 -- start start 2024-04-06T13:51:27.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.489+0000 7f26bf58c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26b806a730 0x7f26b81f4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:27.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.489+0000 7f26bf58c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26b815d7b0 0x7f26b8146a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:27.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.489+0000 7f26bf58c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b815e500 0x7f26b80ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:27.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.489+0000 7f26bf58c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b815ad90 con 0x7f26b815e500 2024-04-06T13:51:27.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.489+0000 7f26bf58c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b8160490 con 0x7f26b815d7b0 2024-04-06T13:51:27.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.489+0000 7f26bf58c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b815afe0 con 0x7f26b806a730 2024-04-06T13:51:27.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.489+0000 7f26bdb29700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b815e500 0x7f26b80ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:27.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.489+0000 7f26bdb29700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b815e500 0x7f26b80ccd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54580/0 (socket says 172.21.15.189:54580) 2024-04-06T13:51:27.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.489+0000 7f26bdb29700 1 -- 172.21.15.189:0/244666647 learned_addr learned my addr 172.21.15.189:0/244666647 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:27.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.489+0000 7f26bcb27700 1 --2- 172.21.15.189:0/244666647 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26b815d7b0 0x7f26b8146a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:27.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.489+0000 7f26bd328700 1 --2- 172.21.15.189:0/244666647 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26b806a730 0x7f26b81f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:27.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.490+0000 7f26bdb29700 1 -- 172.21.15.189:0/244666647 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26b806a730 msgr2=0x7f26b81f4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:27.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.490+0000 7f26bdb29700 1 --2- 172.21.15.189:0/244666647 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26b806a730 0x7f26b81f4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.490+0000 7f26bdb29700 1 -- 172.21.15.189:0/244666647 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26b815d7b0 msgr2=0x7f26b8146a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:27.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.490+0000 7f26bdb29700 1 --2- 172.21.15.189:0/244666647 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26b815d7b0 0x7f26b8146a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.490+0000 7f26bdb29700 1 -- 172.21.15.189:0/244666647 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b4009550 con 0x7f26b815e500 2024-04-06T13:51:27.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.490+0000 7f26bcb27700 1 --2- 172.21.15.189:0/244666647 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26b815d7b0 0x7f26b8146a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:51:27.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.490+0000 7f26bdb29700 1 --2- 172.21.15.189:0/244666647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b815e500 0x7f26b80ccd90 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7f26b0007e60 tx=0x7f26b000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:27.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.490+0000 7f26adffb700 1 -- 172.21.15.189:0/244666647 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b000ed30 con 0x7f26b815e500 2024-04-06T13:51:27.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.490+0000 7f26adffb700 1 -- 172.21.15.189:0/244666647 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b0014420 con 0x7f26b815e500 2024-04-06T13:51:27.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.490+0000 7f26bf58c700 1 -- 172.21.15.189:0/244666647 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b815b2c0 con 0x7f26b815e500 2024-04-06T13:51:27.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.490+0000 7f26adffb700 1 -- 172.21.15.189:0/244666647 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b00147d0 con 0x7f26b815e500 2024-04-06T13:51:27.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.490+0000 7f26bf58c700 1 -- 172.21.15.189:0/244666647 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b8146f80 con 0x7f26b815e500 2024-04-06T13:51:27.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.491+0000 7f26bf58c700 1 -- 172.21.15.189:0/244666647 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b815b7e0 con 0x7f26b815e500 2024-04-06T13:51:27.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.492+0000 7f26adffb700 1 -- 172.21.15.189:0/244666647 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 177) v1 ==== 89412+0+0 (secure 0 0 0) 0x7f26b0014930 con 0x7f26b815e500 2024-04-06T13:51:27.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.492+0000 7f26adffb700 1 --2- 172.21.15.189:0/244666647 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f269c06af60 0x7f269c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:27.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.492+0000 7f26adffb700 1 -- 172.21.15.189:0/244666647 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f26b008a110 con 0x7f26b815e500 2024-04-06T13:51:27.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.492+0000 7f26bd328700 1 -- 172.21.15.189:0/244666647 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f269c06af60 msgr2=0x7f269c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:27.496 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.492+0000 7f26bd328700 1 --2- 172.21.15.189:0/244666647 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f269c06af60 0x7f269c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:51:27.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.502+0000 7f26adffb700 1 -- 172.21.15.189:0/244666647 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f26b815b7e0 con 0x7f26b815e500 2024-04-06T13:51:27.655 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:27.654+0000 7f72d02d9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:51:27.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.693+0000 7f26bd328700 1 -- 172.21.15.189:0/244666647 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f269c06af60 msgr2=0x7f269c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:27.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.693+0000 7f26bd328700 1 --2- 172.21.15.189:0/244666647 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f269c06af60 0x7f269c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:51:27.708 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.707+0000 7f26bf58c700 1 -- 172.21.15.189:0/244666647 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26b815b7e0 con 0x7f26b815e500 2024-04-06T13:51:27.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.709+0000 7f26adffb700 1 -- 172.21.15.189:0/244666647 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v286) v1 ==== 74+0+17125 (secure 0 0 0) 0x7f26b815b7e0 con 0x7f26b815e500 2024-04-06T13:51:27.710 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:51:27.710 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":286,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:51:08.736525+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:49:51.467569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"282","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-04-06T13:49:52.656784+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"283","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"38.5","mappings":[{"from":7,"to":2}]},{"pgid":"38.2b","mappings":[{"from":7,"to":4}]},{"pgid":"38.2e","mappings":[{"from":7,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/2987126770":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/2905400170":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/893866137":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/2221278556":"2024-04-07T13:50:07.256457+0000","172.21.15.189:6833/111704":"2024-04-07T13:50:07.256457+0000","172.21.15.189:6832/111704":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/4009223680":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/3345146368":"2024-04-07T13:50:07.256457+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:51:27.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.711+0000 7f26bf58c700 1 -- 172.21.15.189:0/244666647 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f269c06af60 msgr2=0x7f269c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:51:27.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.711+0000 7f26bf58c700 1 --2- 172.21.15.189:0/244666647 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f269c06af60 0x7f269c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.711+0000 7f26bf58c700 1 -- 172.21.15.189:0/244666647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b815e500 msgr2=0x7f26b80ccd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:27.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.711+0000 7f26bf58c700 1 --2- 172.21.15.189:0/244666647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b815e500 0x7f26b80ccd90 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7f26b0007e60 tx=0x7f26b000e3c0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.711+0000 7f26bf58c700 1 -- 172.21.15.189:0/244666647 shutdown_connections 2024-04-06T13:51:27.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.711+0000 7f26bf58c700 1 --2- 172.21.15.189:0/244666647 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f269c06af60 0x7f269c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.711+0000 7f26bf58c700 1 --2- 172.21.15.189:0/244666647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26b815e500 0x7f26b80ccd90 unknown :-1 s=CLOSED pgs=3161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.711+0000 7f26bf58c700 1 --2- 172.21.15.189:0/244666647 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26b806a730 0x7f26b81f4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.711+0000 7f26bf58c700 1 --2- 172.21.15.189:0/244666647 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26b815d7b0 0x7f26b8146a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.711+0000 7f26bf58c700 1 -- 172.21.15.189:0/244666647 >> 172.21.15.189:0/244666647 conn(0x7f26b80d00b0 msgr2=0x7f26b806eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:27.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.711+0000 7f26bf58c700 1 -- 172.21.15.189:0/244666647 shutdown_connections 2024-04-06T13:51:27.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.712+0000 7f26bf58c700 1 -- 172.21.15.189:0/244666647 wait complete. 2024-04-06T13:51:27.726 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T12:11:58.609282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-06T13:49:51.467569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '282', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 38, 'pool_name': 'cephfs_data', 'create_time': '2024-04-06T13:49:52.656784+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '283', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T13:51:27.727 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T13:51:27.735 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:27.734+0000 7f72d02d9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:51:27.831 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:27.830+0000 7f3fecaf2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T13:51:27.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.854+0000 7ff59ca0a700 1 Processor -- start 2024-04-06T13:51:27.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.854+0000 7ff59ca0a700 1 -- start start 2024-04-06T13:51:27.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.855+0000 7ff59ca0a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5980d0a60 0x7ff5980d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:27.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.855+0000 7ff59ca0a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff5980d1af0 0x7ff5980b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:27.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.855+0000 7ff59ca0a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5980cdf80 con 0x7ff59815d7e0 2024-04-06T13:51:27.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.855+0000 7ff59ca0a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5980d1f80 con 0x7ff5980d1af0 2024-04-06T13:51:27.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.855+0000 7ff59ca0a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5980d3230 con 0x7ff5980d0a60 2024-04-06T13:51:27.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.855+0000 7ff596d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff5980d1af0 0x7ff5980b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:27.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.855+0000 7ff596d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff5980d1af0 0x7ff5980b66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54592/0 (socket says 172.21.15.189:54592) 2024-04-06T13:51:27.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.855+0000 7ff596d9d700 1 -- 172.21.15.189:0/3533999893 learned_addr learned my addr 172.21.15.189:0/3533999893 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:27.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.855+0000 7ff595d9b700 1 --2- 172.21.15.189:0/3533999893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5980d0a60 0x7ff5980d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:27.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.856+0000 7ff596d9d700 1 -- 172.21.15.189:0/3533999893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5980d0a60 msgr2=0x7ff5980d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:27.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.856+0000 7ff596d9d700 1 --2- 172.21.15.189:0/3533999893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5980d0a60 0x7ff5980d15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.856+0000 7ff596d9d700 1 -- 172.21.15.189:0/3533999893 >> v1:172.21.15.192:6789/0 conn(0x7ff59815d7e0 legacy=0x7ff5980d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:27.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.856+0000 7ff596d9d700 1 -- 172.21.15.189:0/3533999893 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5980d2da0 con 0x7ff5980d1af0 2024-04-06T13:51:27.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.856+0000 7ff595d9b700 1 --2- 172.21.15.189:0/3533999893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5980d0a60 0x7ff5980d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:51:27.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.856+0000 7ff596d9d700 1 --2- 172.21.15.189:0/3533999893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5980d1af0 0x7ff5980b66b0 secure :-1 s=READY pgs=3162 cs=0 l=1 rev1=1 crypto rx=0x7ff590009e50 tx=0x7ff59000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da5af3eda934bde server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:27.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.856+0000 7ff59559a700 1 -- 172.21.15.189:0/3533999893 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59000e070 con 0x7ff5980d1af0 2024-04-06T13:51:27.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.856+0000 7ff59559a700 1 -- 172.21.15.189:0/3533999893 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59000b9c0 con 0x7ff5980d1af0 2024-04-06T13:51:27.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.856+0000 7ff59559a700 1 -- 172.21.15.189:0/3533999893 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59000bd70 con 0x7ff5980d1af0 2024-04-06T13:51:27.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.856+0000 7ff59ca0a700 1 -- 172.21.15.189:0/3533999893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5980d1af0 msgr2=0x7ff5980b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:27.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.856+0000 7ff59ca0a700 1 --2- 172.21.15.189:0/3533999893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5980d1af0 0x7ff5980b66b0 secure :-1 s=READY pgs=3162 cs=0 l=1 rev1=1 crypto rx=0x7ff590009e50 tx=0x7ff59000b330 comp rx=0 tx=0).stop 2024-04-06T13:51:27.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.857+0000 7ff59ca0a700 1 -- 172.21.15.189:0/3533999893 shutdown_connections 2024-04-06T13:51:27.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.857+0000 7ff59ca0a700 1 --2- 172.21.15.189:0/3533999893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5980d1af0 0x7ff5980b66b0 unknown :-1 s=CLOSED pgs=3162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.857+0000 7ff59ca0a700 1 --2- 172.21.15.189:0/3533999893 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5980d0a60 0x7ff5980d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.857+0000 7ff59ca0a700 1 -- 172.21.15.189:0/3533999893 >> 172.21.15.189:0/3533999893 conn(0x7ff598072250 msgr2=0x7ff598072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:27.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.857+0000 7ff59ca0a700 1 -- 172.21.15.189:0/3533999893 shutdown_connections 2024-04-06T13:51:27.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.857+0000 7ff59ca0a700 1 -- 172.21.15.189:0/3533999893 wait complete. 2024-04-06T13:51:27.864 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:27.859+0000 7f72d02d9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:51:27.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.858+0000 7ff59ca0a700 1 Processor -- start 2024-04-06T13:51:27.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.858+0000 7ff59ca0a700 1 -- start start 2024-04-06T13:51:27.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.859+0000 7ff59ca0a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5980d0a60 0x7ff5980c9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:27.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.859+0000 7ff59ca0a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5980d1af0 0x7ff5980c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:27.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.859+0000 7ff59ca0a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff59815d7e0 0x7ff5981f6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:27.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.859+0000 7ff59ca0a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5980cab90 con 0x7ff59815d7e0 2024-04-06T13:51:27.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.859+0000 7ff59ca0a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5980c9670 con 0x7ff5980d1af0 2024-04-06T13:51:27.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.859+0000 7ff59ca0a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5980c9950 con 0x7ff5980d0a60 2024-04-06T13:51:27.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.859+0000 7ff596d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff59815d7e0 0x7ff5981f6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:27.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.859+0000 7ff59659c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5980d0a60 0x7ff5980c9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:27.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.859+0000 7ff595d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5980d1af0 0x7ff5980c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:27.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.859+0000 7ff596d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff59815d7e0 0x7ff5981f6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54604/0 (socket says 172.21.15.189:54604) 2024-04-06T13:51:27.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.859+0000 7ff59659c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5980d0a60 0x7ff5980c9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:35174/0 (socket says 172.21.15.189:35174) 2024-04-06T13:51:27.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.859+0000 7ff596d9d700 1 -- 172.21.15.189:0/1039888519 learned_addr learned my addr 172.21.15.189:0/1039888519 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:27.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.859+0000 7ff596d9d700 1 -- 172.21.15.189:0/1039888519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5980d0a60 msgr2=0x7ff5980c9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:27.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.859+0000 7ff596d9d700 1 --2- 172.21.15.189:0/1039888519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5980d0a60 0x7ff5980c9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.859+0000 7ff596d9d700 1 -- 172.21.15.189:0/1039888519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5980d1af0 msgr2=0x7ff5980c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:27.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.859+0000 7ff596d9d700 1 --2- 172.21.15.189:0/1039888519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5980d1af0 0x7ff5980c6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:27.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.859+0000 7ff596d9d700 1 -- 172.21.15.189:0/1039888519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff590009b30 con 0x7ff59815d7e0 2024-04-06T13:51:27.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.859+0000 7ff596d9d700 1 --2- 172.21.15.189:0/1039888519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff59815d7e0 0x7ff5981f6400 secure :-1 s=READY pgs=3163 cs=0 l=1 rev1=1 crypto rx=0x7ff59806b8c0 tx=0x7ff590004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:27.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.859+0000 7ff586ffd700 1 -- 172.21.15.189:0/1039888519 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff590004840 con 0x7ff59815d7e0 2024-04-06T13:51:27.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.860+0000 7ff586ffd700 1 -- 172.21.15.189:0/1039888519 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5900099b0 con 0x7ff59815d7e0 2024-04-06T13:51:27.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.860+0000 7ff59ca0a700 1 -- 172.21.15.189:0/1039888519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5980cade0 con 0x7ff59815d7e0 2024-04-06T13:51:27.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.860+0000 7ff586ffd700 1 -- 172.21.15.189:0/1039888519 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59001c680 con 0x7ff59815d7e0 2024-04-06T13:51:27.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.860+0000 7ff59ca0a700 1 -- 172.21.15.189:0/1039888519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5980c7010 con 0x7ff59815d7e0 2024-04-06T13:51:27.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.861+0000 7ff59ca0a700 1 -- 172.21.15.189:0/1039888519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5980cb2a0 con 0x7ff59815d7e0 2024-04-06T13:51:27.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.861+0000 7ff586ffd700 1 -- 172.21.15.189:0/1039888519 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 177) v1 ==== 89412+0+0 (secure 0 0 0) 0x7ff59001c7e0 con 0x7ff59815d7e0 2024-04-06T13:51:27.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.862+0000 7ff586ffd700 1 --2- 172.21.15.189:0/1039888519 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff57806af60 0x7ff57806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:27.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.862+0000 7ff59659c700 1 -- 172.21.15.189:0/1039888519 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff57806af60 msgr2=0x7ff57806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:27.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.862+0000 7ff586ffd700 1 -- 172.21.15.189:0/1039888519 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff59008b790 con 0x7ff59815d7e0 2024-04-06T13:51:27.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.862+0000 7ff59659c700 1 --2- 172.21.15.189:0/1039888519 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff57806af60 0x7ff57806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:51:27.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:27.870+0000 7ff586ffd700 1 -- 172.21.15.189:0/1039888519 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7ff5900561e0 con 0x7ff59815d7e0 2024-04-06T13:51:27.919 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:27.918+0000 7f3fecaf2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T13:51:28.051 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:28.050+0000 7f3fecaf2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T13:51:28.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.063+0000 7ff59659c700 1 -- 172.21.15.189:0/1039888519 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff57806af60 msgr2=0x7ff57806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:28.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.063+0000 7ff59659c700 1 --2- 172.21.15.189:0/1039888519 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff57806af60 0x7ff57806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:51:28.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.073+0000 7ff59ca0a700 1 -- 172.21.15.189:0/1039888519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff598074990 con 0x7ff59815d7e0 2024-04-06T13:51:28.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.073+0000 7ff586ffd700 1 -- 172.21.15.189:0/1039888519 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v286) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff590059800 con 0x7ff59815d7e0 2024-04-06T13:51:28.074 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2024-04-06T13:51:28.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.076+0000 7ff59ca0a700 1 -- 172.21.15.189:0/1039888519 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff57806af60 msgr2=0x7ff57806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:51:28.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.076+0000 7ff59ca0a700 1 --2- 172.21.15.189:0/1039888519 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff57806af60 0x7ff57806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.076+0000 7ff59ca0a700 1 -- 172.21.15.189:0/1039888519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff59815d7e0 msgr2=0x7ff5981f6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:28.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.076+0000 7ff59ca0a700 1 --2- 172.21.15.189:0/1039888519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff59815d7e0 0x7ff5981f6400 secure :-1 s=READY pgs=3163 cs=0 l=1 rev1=1 crypto rx=0x7ff59806b8c0 tx=0x7ff590004ad0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.076+0000 7ff59ca0a700 1 -- 172.21.15.189:0/1039888519 shutdown_connections 2024-04-06T13:51:28.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.076+0000 7ff59ca0a700 1 --2- 172.21.15.189:0/1039888519 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff57806af60 0x7ff57806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.076+0000 7ff59ca0a700 1 --2- 172.21.15.189:0/1039888519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff59815d7e0 0x7ff5981f6400 unknown :-1 s=CLOSED pgs=3163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.076+0000 7ff59ca0a700 1 --2- 172.21.15.189:0/1039888519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5980d0a60 0x7ff5980c9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.076+0000 7ff59ca0a700 1 --2- 172.21.15.189:0/1039888519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5980d1af0 0x7ff5980c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.076+0000 7ff59ca0a700 1 -- 172.21.15.189:0/1039888519 >> 172.21.15.189:0/1039888519 conn(0x7ff598072250 msgr2=0x7ff59815f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:28.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.076+0000 7ff59ca0a700 1 -- 172.21.15.189:0/1039888519 shutdown_connections 2024-04-06T13:51:28.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.076+0000 7ff59ca0a700 1 -- 172.21.15.189:0/1039888519 wait complete. 2024-04-06T13:51:28.089 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-06T13:51:28.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.221+0000 7fb22c45d700 1 Processor -- start 2024-04-06T13:51:28.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.221+0000 7fb22c45d700 1 -- start start 2024-04-06T13:51:28.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.222+0000 7fb22c45d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb22415d7e0 0x7fb2240d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:28.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.222+0000 7fb22c45d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb2240d1cc0 0x7fb2240b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:28.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.222+0000 7fb22c45d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2240cdf80 con 0x7fb2240d0890 2024-04-06T13:51:28.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.222+0000 7fb22c45d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2240d3230 con 0x7fb22415d7e0 2024-04-06T13:51:28.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.222+0000 7fb22c45d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2240d2da0 con 0x7fb2240d1cc0 2024-04-06T13:51:28.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.222+0000 7fb22a1f9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb22415d7e0 0x7fb2240d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:28.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.222+0000 7fb2299f8700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb2240d0890 0x7fb2240d15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34510/0 (socket says 172.21.15.189:34510) 2024-04-06T13:51:28.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.222+0000 7fb2299f8700 1 -- 172.21.15.189:0/233098802 learned_addr learned my addr 172.21.15.189:0/233098802 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:28.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.222+0000 7fb22a9fa700 1 --2- 172.21.15.189:0/233098802 >> v2:172.21.15.192:3300/0 conn(0x7fb2240d1cc0 0x7fb2240b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:28.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.223+0000 7fb22a1f9700 1 -- 172.21.15.189:0/233098802 >> v2:172.21.15.192:3300/0 conn(0x7fb2240d1cc0 msgr2=0x7fb2240b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:28.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.223+0000 7fb22a1f9700 1 --2- 172.21.15.189:0/233098802 >> v2:172.21.15.192:3300/0 conn(0x7fb2240d1cc0 0x7fb2240b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.223+0000 7fb22a1f9700 1 -- 172.21.15.189:0/233098802 >> v1:172.21.15.189:6789/0 conn(0x7fb2240d0890 legacy=0x7fb2240d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:28.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.223+0000 7fb22a1f9700 1 -- 172.21.15.189:0/233098802 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2240d0cf0 con 0x7fb22415d7e0 2024-04-06T13:51:28.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.223+0000 7fb2291f7700 1 -- 172.21.15.189:0/233098802 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477709745 0 0) 0x7fb2240cdf80 con 0x7fb2240d0890 2024-04-06T13:51:28.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.223+0000 7fb22a1f9700 1 --2- 172.21.15.189:0/233098802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb22415d7e0 0x7fb2240d0350 secure :-1 s=READY pgs=3165 cs=0 l=1 rev1=1 crypto rx=0x7fb2240ca3a0 tx=0x7fb214009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed98a547cab764aa server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:28.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.223+0000 7fb2291f7700 1 -- 172.21.15.189:0/233098802 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb214011830 con 0x7fb22415d7e0 2024-04-06T13:51:28.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.223+0000 7fb2291f7700 1 -- 172.21.15.189:0/233098802 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb214011990 con 0x7fb22415d7e0 2024-04-06T13:51:28.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.223+0000 7fb2291f7700 1 -- 172.21.15.189:0/233098802 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb214011d20 con 0x7fb22415d7e0 2024-04-06T13:51:28.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.224+0000 7fb22c45d700 1 -- 172.21.15.189:0/233098802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb22415d7e0 msgr2=0x7fb2240d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:28.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.224+0000 7fb22c45d700 1 --2- 172.21.15.189:0/233098802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb22415d7e0 0x7fb2240d0350 secure :-1 s=READY pgs=3165 cs=0 l=1 rev1=1 crypto rx=0x7fb2240ca3a0 tx=0x7fb214009b20 comp rx=0 tx=0).stop 2024-04-06T13:51:28.226 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.224+0000 7fb22c45d700 1 -- 172.21.15.189:0/233098802 shutdown_connections 2024-04-06T13:51:28.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.224+0000 7fb22c45d700 1 --2- 172.21.15.189:0/233098802 >> v2:172.21.15.192:3300/0 conn(0x7fb2240d1cc0 0x7fb2240b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.224+0000 7fb22c45d700 1 --2- 172.21.15.189:0/233098802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb22415d7e0 0x7fb2240d0350 unknown :-1 s=CLOSED pgs=3165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.224+0000 7fb22c45d700 1 -- 172.21.15.189:0/233098802 >> 172.21.15.189:0/233098802 conn(0x7fb224072250 msgr2=0x7fb224072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:28.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.224+0000 7fb22c45d700 1 -- 172.21.15.189:0/233098802 shutdown_connections 2024-04-06T13:51:28.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.224+0000 7fb22c45d700 1 -- 172.21.15.189:0/233098802 wait complete. 2024-04-06T13:51:28.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.226+0000 7fb22c45d700 1 Processor -- start 2024-04-06T13:51:28.227 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.226+0000 7fb22c45d700 1 -- start start 2024-04-06T13:51:28.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.228+0000 7fb22c45d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2240d0890 0x7fb2240a9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:28.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.228+0000 7fb22c45d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2240d1cc0 0x7fb2240aa900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:28.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.228+0000 7fb22c45d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb22415d7e0 0x7fb2241f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:28.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.228+0000 7fb22c45d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2240aa2e0 con 0x7fb22415d7e0 2024-04-06T13:51:28.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.228+0000 7fb22c45d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2240c9420 con 0x7fb2240d1cc0 2024-04-06T13:51:28.230 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.228+0000 7fb22c45d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2240aae40 con 0x7fb2240d0890 2024-04-06T13:51:28.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.228+0000 7fb22a9fa700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb22415d7e0 0x7fb2241f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:28.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.228+0000 7fb22a9fa700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb22415d7e0 0x7fb2241f28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54622/0 (socket says 172.21.15.189:54622) 2024-04-06T13:51:28.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.228+0000 7fb22a9fa700 1 -- 172.21.15.189:0/3983269337 learned_addr learned my addr 172.21.15.189:0/3983269337 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:28.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.228+0000 7fb22a1f9700 1 --2- 172.21.15.189:0/3983269337 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2240d0890 0x7fb2240a9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:28.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.228+0000 7fb2299f8700 1 --2- 172.21.15.189:0/3983269337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2240d1cc0 0x7fb2240aa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:28.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.228+0000 7fb22a9fa700 1 -- 172.21.15.189:0/3983269337 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2240d0890 msgr2=0x7fb2240a9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:28.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.228+0000 7fb22a9fa700 1 --2- 172.21.15.189:0/3983269337 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2240d0890 0x7fb2240a9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.228+0000 7fb22a9fa700 1 -- 172.21.15.189:0/3983269337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2240d1cc0 msgr2=0x7fb2240aa900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:28.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.228+0000 7fb22a9fa700 1 --2- 172.21.15.189:0/3983269337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2240d1cc0 0x7fb2240aa900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.228+0000 7fb22a9fa700 1 -- 172.21.15.189:0/3983269337 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2140094a0 con 0x7fb22415d7e0 2024-04-06T13:51:28.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.228+0000 7fb22a9fa700 1 --2- 172.21.15.189:0/3983269337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb22415d7e0 0x7fb2241f28d0 secure :-1 s=READY pgs=3166 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00ef90 tx=0x7fb21c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:28.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.229+0000 7fb21affd700 1 -- 172.21.15.189:0/3983269337 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21c00cc90 con 0x7fb22415d7e0 2024-04-06T13:51:28.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.229+0000 7fb21affd700 1 -- 172.21.15.189:0/3983269337 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21c00f040 con 0x7fb22415d7e0 2024-04-06T13:51:28.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.229+0000 7fb21affd700 1 -- 172.21.15.189:0/3983269337 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21c014480 con 0x7fb22415d7e0 2024-04-06T13:51:28.231 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.229+0000 7fb22c45d700 1 -- 172.21.15.189:0/3983269337 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2240c8660 con 0x7fb22415d7e0 2024-04-06T13:51:28.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.229+0000 7fb22c45d700 1 -- 172.21.15.189:0/3983269337 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2240c8c20 con 0x7fb22415d7e0 2024-04-06T13:51:28.232 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.230+0000 7fb22c45d700 1 -- 172.21.15.189:0/3983269337 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2241f4ba0 con 0x7fb22415d7e0 2024-04-06T13:51:28.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.230+0000 7fb21affd700 1 -- 172.21.15.189:0/3983269337 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 177) v1 ==== 89412+0+0 (secure 0 0 0) 0x7fb21c0145e0 con 0x7fb22415d7e0 2024-04-06T13:51:28.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.231+0000 7fb21affd700 1 --2- 172.21.15.189:0/3983269337 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb20806af60 0x7fb20806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:28.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.231+0000 7fb22a1f9700 1 -- 172.21.15.189:0/3983269337 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb20806af60 msgr2=0x7fb20806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:28.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.231+0000 7fb21affd700 1 -- 172.21.15.189:0/3983269337 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb21c08a780 con 0x7fb22415d7e0 2024-04-06T13:51:28.233 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.231+0000 7fb22a1f9700 1 --2- 172.21.15.189:0/3983269337 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb20806af60 0x7fb20806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:51:28.236 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.235+0000 7fb21affd700 1 -- 172.21.15.189:0/3983269337 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb21c055140 con 0x7fb22415d7e0 2024-04-06T13:51:28.373 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:28.372+0000 7f72d02d9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:51:28.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.432+0000 7fb22a1f9700 1 -- 172.21.15.189:0/3983269337 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb20806af60 msgr2=0x7fb20806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:28.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.432+0000 7fb22a1f9700 1 --2- 172.21.15.189:0/3983269337 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb20806af60 0x7fb20806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:51:28.439 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.438+0000 7fb22c45d700 1 -- 172.21.15.189:0/3983269337 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb224074990 con 0x7fb22415d7e0 2024-04-06T13:51:28.440 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.439+0000 7fb21affd700 1 -- 172.21.15.189:0/3983269337 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v286) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb21c009ca0 con 0x7fb22415d7e0 2024-04-06T13:51:28.440 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 32 2024-04-06T13:51:28.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.441+0000 7fb22c45d700 1 -- 172.21.15.189:0/3983269337 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb20806af60 msgr2=0x7fb20806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:51:28.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.441+0000 7fb22c45d700 1 --2- 172.21.15.189:0/3983269337 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb20806af60 0x7fb20806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.441+0000 7fb22c45d700 1 -- 172.21.15.189:0/3983269337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb22415d7e0 msgr2=0x7fb2241f28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:28.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.441+0000 7fb22c45d700 1 --2- 172.21.15.189:0/3983269337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb22415d7e0 0x7fb2241f28d0 secure :-1 s=READY pgs=3166 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00ef90 tx=0x7fb21c00c3a0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.441+0000 7fb22c45d700 1 -- 172.21.15.189:0/3983269337 shutdown_connections 2024-04-06T13:51:28.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.441+0000 7fb22c45d700 1 --2- 172.21.15.189:0/3983269337 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb20806af60 0x7fb20806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.441+0000 7fb22c45d700 1 --2- 172.21.15.189:0/3983269337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb22415d7e0 0x7fb2241f28d0 unknown :-1 s=CLOSED pgs=3166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.441+0000 7fb22c45d700 1 --2- 172.21.15.189:0/3983269337 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2240d0890 0x7fb2240a9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.441+0000 7fb22c45d700 1 --2- 172.21.15.189:0/3983269337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2240d1cc0 0x7fb2240aa900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.441+0000 7fb22c45d700 1 -- 172.21.15.189:0/3983269337 >> 172.21.15.189:0/3983269337 conn(0x7fb224072250 msgr2=0x7fb2240cda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:28.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.441+0000 7fb22c45d700 1 -- 172.21.15.189:0/3983269337 shutdown_connections 2024-04-06T13:51:28.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.441+0000 7fb22c45d700 1 -- 172.21.15.189:0/3983269337 wait complete. 2024-04-06T13:51:28.455 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-06T13:51:28.527 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:28.526+0000 7f72d02d9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:51:28.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.584+0000 7f2bd179d700 1 Processor -- start 2024-04-06T13:51:28.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.584+0000 7f2bd179d700 1 -- start start 2024-04-06T13:51:28.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.584+0000 7f2bd179d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bcc0d17a0 0x7f2bcc06b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:28.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.584+0000 7f2bd179d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2bcc146dd0 0x7f2bcc06d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:28.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.584+0000 7f2bd179d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2bcc0cedf0 0x7f2bcc0cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:28.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.584+0000 7f2bd179d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bcc0cf760 con 0x7f2bcc146dd0 2024-04-06T13:51:28.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.584+0000 7f2bd179d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bcc06c0a0 con 0x7f2bcc0cedf0 2024-04-06T13:51:28.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.584+0000 7f2bd179d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc06c380 con 0x7f2bcc0d17a0 2024-04-06T13:51:28.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.585+0000 7f2bca7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2bcc146dd0 0x7f2bcc06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:28.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.585+0000 7f2bca7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2bcc146dd0 0x7f2bcc06d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54632/0 (socket says 172.21.15.189:54632) 2024-04-06T13:51:28.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.585+0000 7f2bca7fc700 1 -- 172.21.15.189:0/2654564576 learned_addr learned my addr 172.21.15.189:0/2654564576 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:28.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.585+0000 7f2bcaffd700 1 --2- 172.21.15.189:0/2654564576 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bcc0d17a0 0x7f2bcc06b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:28.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.585+0000 7f2bcb7fe700 1 --2- 172.21.15.189:0/2654564576 >> v2:172.21.15.192:3300/0 conn(0x7f2bcc0cedf0 0x7f2bcc0cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:28.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.585+0000 7f2bca7fc700 1 -- 172.21.15.189:0/2654564576 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bcc0d17a0 msgr2=0x7f2bcc06b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:28.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.585+0000 7f2bca7fc700 1 --2- 172.21.15.189:0/2654564576 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bcc0d17a0 0x7f2bcc06b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.585+0000 7f2bca7fc700 1 -- 172.21.15.189:0/2654564576 >> v2:172.21.15.192:3300/0 conn(0x7f2bcc0cedf0 msgr2=0x7f2bcc0cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:28.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.585+0000 7f2bca7fc700 1 --2- 172.21.15.189:0/2654564576 >> v2:172.21.15.192:3300/0 conn(0x7f2bcc0cedf0 0x7f2bcc0cf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.585+0000 7f2bca7fc700 1 -- 172.21.15.189:0/2654564576 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bcc06c660 con 0x7f2bcc146dd0 2024-04-06T13:51:28.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.585+0000 7f2bca7fc700 1 --2- 172.21.15.189:0/2654564576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bcc146dd0 0x7f2bcc06d140 secure :-1 s=READY pgs=3167 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc162120 tx=0x7f2bc0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f97b761a4d660d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:28.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.585+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/2654564576 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0011830 con 0x7f2bcc146dd0 2024-04-06T13:51:28.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.585+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/2654564576 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc0011990 con 0x7f2bcc146dd0 2024-04-06T13:51:28.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.585+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/2654564576 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0011d20 con 0x7f2bcc146dd0 2024-04-06T13:51:28.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.586+0000 7f2bd179d700 1 -- 172.21.15.189:0/2654564576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bcc146dd0 msgr2=0x7f2bcc06d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:28.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.586+0000 7f2bd179d700 1 --2- 172.21.15.189:0/2654564576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bcc146dd0 0x7f2bcc06d140 secure :-1 s=READY pgs=3167 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc162120 tx=0x7f2bc0009bd0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.586+0000 7f2bd179d700 1 -- 172.21.15.189:0/2654564576 shutdown_connections 2024-04-06T13:51:28.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.586+0000 7f2bd179d700 1 --2- 172.21.15.189:0/2654564576 >> v2:172.21.15.192:3300/0 conn(0x7f2bcc0cedf0 0x7f2bcc0cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.586+0000 7f2bd179d700 1 --2- 172.21.15.189:0/2654564576 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bcc0d17a0 0x7f2bcc06b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.586+0000 7f2bd179d700 1 --2- 172.21.15.189:0/2654564576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bcc146dd0 0x7f2bcc06d140 unknown :-1 s=CLOSED pgs=3167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.586+0000 7f2bd179d700 1 -- 172.21.15.189:0/2654564576 >> 172.21.15.189:0/2654564576 conn(0x7f2bcc0b2db0 msgr2=0x7f2bcc0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:28.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.586+0000 7f2bd179d700 1 -- 172.21.15.189:0/2654564576 shutdown_connections 2024-04-06T13:51:28.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.586+0000 7f2bd179d700 1 -- 172.21.15.189:0/2654564576 wait complete. 2024-04-06T13:51:28.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.587+0000 7f2bd179d700 1 Processor -- start 2024-04-06T13:51:28.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.587+0000 7f2bd179d700 1 -- start start 2024-04-06T13:51:28.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.588+0000 7f2bd179d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bcc0cedf0 0x7f2bcc0b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:28.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.588+0000 7f2bd179d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bcc0d17a0 0x7f2bcc0b5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:28.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.588+0000 7f2bd179d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bcc146dd0 0x7f2bcc1f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:28.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.588+0000 7f2bd179d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc0c5280 con 0x7f2bcc0d17a0 2024-04-06T13:51:28.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.588+0000 7f2bd179d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc0c5560 con 0x7f2bcc0cedf0 2024-04-06T13:51:28.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.588+0000 7f2bd179d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc0c5840 con 0x7f2bcc146dd0 2024-04-06T13:51:28.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.588+0000 7f2bca7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bcc0d17a0 0x7f2bcc0b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:28.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.588+0000 7f2bca7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bcc0d17a0 0x7f2bcc0b5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54636/0 (socket says 172.21.15.189:54636) 2024-04-06T13:51:28.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.588+0000 7f2bca7fc700 1 -- 172.21.15.189:0/3583587371 learned_addr learned my addr 172.21.15.189:0/3583587371 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:28.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.588+0000 7f2bcaffd700 1 --2- 172.21.15.189:0/3583587371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bcc0cedf0 0x7f2bcc0b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:28.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.588+0000 7f2bcb7fe700 1 --2- 172.21.15.189:0/3583587371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bcc146dd0 0x7f2bcc1f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:28.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.588+0000 7f2bca7fc700 1 -- 172.21.15.189:0/3583587371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bcc146dd0 msgr2=0x7f2bcc1f2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:28.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.588+0000 7f2bca7fc700 1 --2- 172.21.15.189:0/3583587371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bcc146dd0 0x7f2bcc1f2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.588+0000 7f2bca7fc700 1 -- 172.21.15.189:0/3583587371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bcc0cedf0 msgr2=0x7f2bcc0b0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:28.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.588+0000 7f2bca7fc700 1 --2- 172.21.15.189:0/3583587371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bcc0cedf0 0x7f2bcc0b0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.588+0000 7f2bca7fc700 1 -- 172.21.15.189:0/3583587371 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc0009550 con 0x7f2bcc0d17a0 2024-04-06T13:51:28.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.589+0000 7f2bca7fc700 1 --2- 172.21.15.189:0/3583587371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bcc0d17a0 0x7f2bcc0b5860 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0000c00 tx=0x7f2bc0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:28.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.589+0000 7f2bb37fe700 1 -- 172.21.15.189:0/3583587371 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0004240 con 0x7f2bcc0d17a0 2024-04-06T13:51:28.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.589+0000 7f2bb37fe700 1 -- 172.21.15.189:0/3583587371 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc001f070 con 0x7f2bcc0d17a0 2024-04-06T13:51:28.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.589+0000 7f2bb37fe700 1 -- 172.21.15.189:0/3583587371 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc00036a0 con 0x7f2bcc0d17a0 2024-04-06T13:51:28.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.589+0000 7f2bd179d700 1 -- 172.21.15.189:0/3583587371 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bcc0c5b20 con 0x7f2bcc0d17a0 2024-04-06T13:51:28.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.589+0000 7f2bd179d700 1 -- 172.21.15.189:0/3583587371 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bcc1f3320 con 0x7f2bcc0d17a0 2024-04-06T13:51:28.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.590+0000 7f2bd179d700 1 -- 172.21.15.189:0/3583587371 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bcc1f34b0 con 0x7f2bcc0d17a0 2024-04-06T13:51:28.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.590+0000 7f2bb37fe700 1 -- 172.21.15.189:0/3583587371 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 177) v1 ==== 89412+0+0 (secure 0 0 0) 0x7f2bc0021030 con 0x7f2bcc0d17a0 2024-04-06T13:51:28.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.591+0000 7f2bb37fe700 1 --2- 172.21.15.189:0/3583587371 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2bb806af60 0x7f2bb806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:28.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.591+0000 7f2bcaffd700 1 -- 172.21.15.189:0/3583587371 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2bb806af60 msgr2=0x7f2bb806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:28.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.591+0000 7f2bb37fe700 1 -- 172.21.15.189:0/3583587371 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2bc008ab90 con 0x7f2bcc0d17a0 2024-04-06T13:51:28.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.591+0000 7f2bcaffd700 1 --2- 172.21.15.189:0/3583587371 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2bb806af60 0x7f2bb806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:51:28.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.595+0000 7f2bb37fe700 1 -- 172.21.15.189:0/3583587371 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2bc0055770 con 0x7f2bcc0d17a0 2024-04-06T13:51:28.604 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:28.603+0000 7f3fecaf2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T13:51:28.686 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:28.685+0000 7f72d02d9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:51:28.769 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:28.768+0000 7f3fecaf2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T13:51:28.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.792+0000 7f2bcaffd700 1 -- 172.21.15.189:0/3583587371 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2bb806af60 msgr2=0x7f2bb806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:28.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.792+0000 7f2bcaffd700 1 --2- 172.21.15.189:0/3583587371 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2bb806af60 0x7f2bb806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:51:28.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.799+0000 7f2bd179d700 1 -- 172.21.15.189:0/3583587371 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2bcc0c6040 con 0x7f2bcc0d17a0 2024-04-06T13:51:28.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.799+0000 7f2bb37fe700 1 -- 172.21.15.189:0/3583587371 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v286) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2bcc0c6040 con 0x7f2bcc0d17a0 2024-04-06T13:51:28.800 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 64 2024-04-06T13:51:28.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.801+0000 7f2bd179d700 1 -- 172.21.15.189:0/3583587371 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2bb806af60 msgr2=0x7f2bb806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:51:28.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.801+0000 7f2bd179d700 1 --2- 172.21.15.189:0/3583587371 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2bb806af60 0x7f2bb806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.801+0000 7f2bd179d700 1 -- 172.21.15.189:0/3583587371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bcc0d17a0 msgr2=0x7f2bcc0b5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:28.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.801+0000 7f2bd179d700 1 --2- 172.21.15.189:0/3583587371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bcc0d17a0 0x7f2bcc0b5860 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0000c00 tx=0x7f2bc0004750 comp rx=0 tx=0).stop 2024-04-06T13:51:28.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.802+0000 7f2bd179d700 1 -- 172.21.15.189:0/3583587371 shutdown_connections 2024-04-06T13:51:28.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.802+0000 7f2bd179d700 1 --2- 172.21.15.189:0/3583587371 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2bb806af60 0x7f2bb806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.802+0000 7f2bd179d700 1 --2- 172.21.15.189:0/3583587371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bcc146dd0 0x7f2bcc1f2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.802+0000 7f2bd179d700 1 --2- 172.21.15.189:0/3583587371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bcc0cedf0 0x7f2bcc0b0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.802+0000 7f2bd179d700 1 --2- 172.21.15.189:0/3583587371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bcc0d17a0 0x7f2bcc0b5860 unknown :-1 s=CLOSED pgs=3168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.802+0000 7f2bd179d700 1 -- 172.21.15.189:0/3583587371 >> 172.21.15.189:0/3583587371 conn(0x7f2bcc0b2db0 msgr2=0x7f2bcc15d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:28.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.802+0000 7f2bd179d700 1 -- 172.21.15.189:0/3583587371 shutdown_connections 2024-04-06T13:51:28.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.802+0000 7f2bd179d700 1 -- 172.21.15.189:0/3583587371 wait complete. 2024-04-06T13:51:28.816 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:51:28.941 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:28.938+0000 7f3fecaf2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T13:51:28.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.940+0000 7fe5df0d4700 1 Processor -- start 2024-04-06T13:51:28.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.940+0000 7fe5df0d4700 1 -- start start 2024-04-06T13:51:28.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.941+0000 7fe5df0d4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d80d17a0 0x7fe5d806b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:28.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.941+0000 7fe5df0d4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe5d8146dd0 0x7fe5d806d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:28.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.941+0000 7fe5df0d4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d806c0a0 con 0x7fe5d80cedf0 2024-04-06T13:51:28.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.941+0000 7fe5df0d4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5d806c380 con 0x7fe5d8146dd0 2024-04-06T13:51:28.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.941+0000 7fe5df0d4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d806c660 con 0x7fe5d80d17a0 2024-04-06T13:51:28.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.941+0000 7fe5dd671700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe5d80cedf0 0x7fe5d81441f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34526/0 (socket says 172.21.15.189:34526) 2024-04-06T13:51:28.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.941+0000 7fe5dd671700 1 -- 172.21.15.189:0/3490119126 learned_addr learned my addr 172.21.15.189:0/3490119126 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:28.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.941+0000 7fe5d7fff700 1 --2- 172.21.15.189:0/3490119126 >> v2:172.21.15.192:3300/0 conn(0x7fe5d8146dd0 0x7fe5d806d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:28.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.941+0000 7fe5dce70700 1 --2- 172.21.15.189:0/3490119126 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d80d17a0 0x7fe5d806b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:28.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.941+0000 7fe5d77fe700 1 -- 172.21.15.189:0/3490119126 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2357540659 0 0) 0x7fe5d806c0a0 con 0x7fe5d80cedf0 2024-04-06T13:51:28.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.941+0000 7fe5d77fe700 1 -- 172.21.15.189:0/3490119126 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5c80035b0 con 0x7fe5d80cedf0 2024-04-06T13:51:28.948 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.941+0000 7fe5dce70700 1 -- 172.21.15.189:0/3490119126 >> v2:172.21.15.192:3300/0 conn(0x7fe5d8146dd0 msgr2=0x7fe5d806d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:28.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.941+0000 7fe5dce70700 1 --2- 172.21.15.189:0/3490119126 >> v2:172.21.15.192:3300/0 conn(0x7fe5d8146dd0 0x7fe5d806d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.941+0000 7fe5dce70700 1 -- 172.21.15.189:0/3490119126 >> v1:172.21.15.189:6789/0 conn(0x7fe5d80cedf0 legacy=0x7fe5d81441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:28.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.941+0000 7fe5dce70700 1 -- 172.21.15.189:0/3490119126 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d806c940 con 0x7fe5d80d17a0 2024-04-06T13:51:28.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.941+0000 7fe5d7fff700 1 --2- 172.21.15.189:0/3490119126 >> v2:172.21.15.192:3300/0 conn(0x7fe5d8146dd0 0x7fe5d806d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:51:28.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.941+0000 7fe5dce70700 1 --2- 172.21.15.189:0/3490119126 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d80d17a0 0x7fe5d806b7a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc009840 tx=0x7fe5cc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b3b483fd51d58bd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:28.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.942+0000 7fe5d77fe700 1 -- 172.21.15.189:0/3490119126 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc011830 con 0x7fe5d80d17a0 2024-04-06T13:51:28.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.942+0000 7fe5d77fe700 1 -- 172.21.15.189:0/3490119126 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5cc011990 con 0x7fe5d80d17a0 2024-04-06T13:51:28.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.942+0000 7fe5d77fe700 1 -- 172.21.15.189:0/3490119126 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc011d20 con 0x7fe5d80d17a0 2024-04-06T13:51:28.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.942+0000 7fe5df0d4700 1 -- 172.21.15.189:0/3490119126 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d80d17a0 msgr2=0x7fe5d806b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:28.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.942+0000 7fe5df0d4700 1 --2- 172.21.15.189:0/3490119126 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d80d17a0 0x7fe5d806b7a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc009840 tx=0x7fe5cc009b50 comp rx=0 tx=0).stop 2024-04-06T13:51:28.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.942+0000 7fe5df0d4700 1 -- 172.21.15.189:0/3490119126 shutdown_connections 2024-04-06T13:51:28.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.942+0000 7fe5df0d4700 1 --2- 172.21.15.189:0/3490119126 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d80d17a0 0x7fe5d806b7a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.942+0000 7fe5df0d4700 1 --2- 172.21.15.189:0/3490119126 >> v2:172.21.15.192:3300/0 conn(0x7fe5d8146dd0 0x7fe5d806d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.942+0000 7fe5df0d4700 1 -- 172.21.15.189:0/3490119126 >> 172.21.15.189:0/3490119126 conn(0x7fe5d80b2db0 msgr2=0x7fe5d80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:28.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.942+0000 7fe5df0d4700 1 -- 172.21.15.189:0/3490119126 shutdown_connections 2024-04-06T13:51:28.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.942+0000 7fe5df0d4700 1 -- 172.21.15.189:0/3490119126 wait complete. 2024-04-06T13:51:28.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.943+0000 7fe5df0d4700 1 Processor -- start 2024-04-06T13:51:28.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.944+0000 7fe5df0d4700 1 -- start start 2024-04-06T13:51:28.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.944+0000 7fe5df0d4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5d80cedf0 0x7fe5d815a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:28.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.944+0000 7fe5df0d4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5d80d17a0 0x7fe5d80c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:28.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.944+0000 7fe5df0d4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d8146dd0 0x7fe5d80b6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:28.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.944+0000 7fe5df0d4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80b5c00 con 0x7fe5d80d17a0 2024-04-06T13:51:28.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.944+0000 7fe5df0d4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80b4ff0 con 0x7fe5d80cedf0 2024-04-06T13:51:28.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.944+0000 7fe5df0d4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80b52d0 con 0x7fe5d8146dd0 2024-04-06T13:51:28.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.945+0000 7fe5d7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5d80d17a0 0x7fe5d80c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:28.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.945+0000 7fe5d7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5d80d17a0 0x7fe5d80c6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54638/0 (socket says 172.21.15.189:54638) 2024-04-06T13:51:28.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.945+0000 7fe5d7fff700 1 -- 172.21.15.189:0/882702789 learned_addr learned my addr 172.21.15.189:0/882702789 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:28.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.945+0000 7fe5dd671700 1 --2- 172.21.15.189:0/882702789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d8146dd0 0x7fe5d80b6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:28.951 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.945+0000 7fe5dce70700 1 --2- 172.21.15.189:0/882702789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5d80cedf0 0x7fe5d815a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:28.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.945+0000 7fe5d7fff700 1 -- 172.21.15.189:0/882702789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d8146dd0 msgr2=0x7fe5d80b6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:28.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.945+0000 7fe5d7fff700 1 --2- 172.21.15.189:0/882702789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d8146dd0 0x7fe5d80b6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.945+0000 7fe5d7fff700 1 -- 172.21.15.189:0/882702789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5d80cedf0 msgr2=0x7fe5d815a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:28.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.945+0000 7fe5d7fff700 1 --2- 172.21.15.189:0/882702789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5d80cedf0 0x7fe5d815a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:28.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.945+0000 7fe5d7fff700 1 -- 172.21.15.189:0/882702789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c40094a0 con 0x7fe5d80d17a0 2024-04-06T13:51:28.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.945+0000 7fe5d7fff700 1 --2- 172.21.15.189:0/882702789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5d80d17a0 0x7fe5d80c6f40 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400ebe0 tx=0x7fe5c400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:28.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.945+0000 7fe5d57fa700 1 -- 172.21.15.189:0/882702789 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c400c850 con 0x7fe5d80d17a0 2024-04-06T13:51:28.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.945+0000 7fe5d57fa700 1 -- 172.21.15.189:0/882702789 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c400c9b0 con 0x7fe5d80d17a0 2024-04-06T13:51:28.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.945+0000 7fe5df0d4700 1 -- 172.21.15.189:0/882702789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5cc0094a0 con 0x7fe5d80d17a0 2024-04-06T13:51:28.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.945+0000 7fe5d57fa700 1 -- 172.21.15.189:0/882702789 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c400cd40 con 0x7fe5d80d17a0 2024-04-06T13:51:28.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.945+0000 7fe5df0d4700 1 -- 172.21.15.189:0/882702789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d80b5890 con 0x7fe5d80d17a0 2024-04-06T13:51:28.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.946+0000 7fe5df0d4700 1 -- 172.21.15.189:0/882702789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d8074990 con 0x7fe5d80d17a0 2024-04-06T13:51:28.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.947+0000 7fe5d57fa700 1 -- 172.21.15.189:0/882702789 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 177) v1 ==== 89412+0+0 (secure 0 0 0) 0x7fe5c4013470 con 0x7fe5d80d17a0 2024-04-06T13:51:28.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.947+0000 7fe5d57fa700 1 --2- 172.21.15.189:0/882702789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe5bc06af60 0x7fe5bc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:28.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.947+0000 7fe5dce70700 1 -- 172.21.15.189:0/882702789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe5bc06af60 msgr2=0x7fe5bc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:28.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.947+0000 7fe5d57fa700 1 -- 172.21.15.189:0/882702789 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe5c408a250 con 0x7fe5d80d17a0 2024-04-06T13:51:28.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.947+0000 7fe5dce70700 1 --2- 172.21.15.189:0/882702789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe5bc06af60 0x7fe5bc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:51:28.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:28.952+0000 7fe5d57fa700 1 -- 172.21.15.189:0/882702789 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe5c4054ea0 con 0x7fe5d80d17a0 2024-04-06T13:51:29.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.148+0000 7fe5dce70700 1 -- 172.21.15.189:0/882702789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe5bc06af60 msgr2=0x7fe5bc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:29.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.148+0000 7fe5dce70700 1 --2- 172.21.15.189:0/882702789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe5bc06af60 0x7fe5bc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:51:29.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.201+0000 7fe5df0d4700 1 -- 172.21.15.189:0/882702789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5d80d1e20 con 0x7fe5d80d17a0 2024-04-06T13:51:29.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.201+0000 7fe5d57fa700 1 -- 172.21.15.189:0/882702789 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4759 (secure 0 0 0) 0x7fe5c4017020 con 0x7fe5d80d17a0 2024-04-06T13:51:29.203 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:51:29.203 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":139,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11827,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/819921213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":819921213},{"type":"v1","addr":"172.21.15.192:6835","nonce":819921213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134},{"gid":13368,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3616349784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3616349784},{"type":"v1","addr":"172.21.15.189:6836","nonce":3616349784}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134},{"gid":13371,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3640528525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3640528525},{"type":"v1","addr":"172.21.15.189:6837","nonce":3640528525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:49:53.688018+0000","modified":"2024-04-06T13:49:56.698920+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13380},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13380":{"gid":13380,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6834/3894184871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3894184871},{"type":"v1","addr":"172.21.15.192:6834","nonce":3894184871}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-04-06T13:51:29.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.203+0000 7fe5df0d4700 1 -- 172.21.15.189:0/882702789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe5bc06af60 msgr2=0x7fe5bc06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:51:29.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.203+0000 7fe5df0d4700 1 --2- 172.21.15.189:0/882702789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe5bc06af60 0x7fe5bc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.203+0000 7fe5df0d4700 1 -- 172.21.15.189:0/882702789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5d80d17a0 msgr2=0x7fe5d80c6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:29.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.203+0000 7fe5df0d4700 1 --2- 172.21.15.189:0/882702789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5d80d17a0 0x7fe5d80c6f40 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400ebe0 tx=0x7fe5c400eef0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.204+0000 7fe5df0d4700 1 -- 172.21.15.189:0/882702789 shutdown_connections 2024-04-06T13:51:29.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.204+0000 7fe5df0d4700 1 --2- 172.21.15.189:0/882702789 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fe5bc06af60 0x7fe5bc06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.204+0000 7fe5df0d4700 1 --2- 172.21.15.189:0/882702789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d8146dd0 0x7fe5d80b6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.204+0000 7fe5df0d4700 1 --2- 172.21.15.189:0/882702789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5d80cedf0 0x7fe5d815a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.204+0000 7fe5df0d4700 1 --2- 172.21.15.189:0/882702789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5d80d17a0 0x7fe5d80c6f40 unknown :-1 s=CLOSED pgs=3170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.204+0000 7fe5df0d4700 1 -- 172.21.15.189:0/882702789 >> 172.21.15.189:0/882702789 conn(0x7fe5d80b2db0 msgr2=0x7fe5d80b25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:29.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.204+0000 7fe5df0d4700 1 -- 172.21.15.189:0/882702789 shutdown_connections 2024-04-06T13:51:29.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.204+0000 7fe5df0d4700 1 -- 172.21.15.189:0/882702789 wait complete. 2024-04-06T13:51:29.206 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 139 2024-04-06T13:51:29.217 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:51:29.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.343+0000 7fae22dde700 1 Processor -- start 2024-04-06T13:51:29.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.343+0000 7fae22dde700 1 -- start start 2024-04-06T13:51:29.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.344+0000 7fae22dde700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fae1c15b310 0x7fae1c0b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:29.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.344+0000 7fae22dde700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fae1c0d1a50 0x7fae1c15ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:29.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.344+0000 7fae22dde700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae1c0cb910 con 0x7fae1c0cdfd0 2024-04-06T13:51:29.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.344+0000 7fae22dde700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae1c0cbb80 con 0x7fae1c0d1a50 2024-04-06T13:51:29.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.344+0000 7fae22dde700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae1c0cbe60 con 0x7fae1c15b310 2024-04-06T13:51:29.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.344+0000 7fae1bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fae1c0cdfd0 0x7fae1c0d1340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54006/0 (socket says 172.21.15.189:54006) 2024-04-06T13:51:29.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.344+0000 7fae1bfff700 1 -- 172.21.15.189:0/1742569241 learned_addr learned my addr 172.21.15.189:0/1742569241 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:29.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.344+0000 7fae2137b700 1 --2- 172.21.15.189:0/1742569241 >> v2:172.21.15.189:3300/0 conn(0x7fae1c0d1a50 0x7fae1c15ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:29.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.344+0000 7fae20b7a700 1 --2- 172.21.15.189:0/1742569241 >> v2:172.21.15.192:3300/0 conn(0x7fae1c15b310 0x7fae1c0b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:29.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.344+0000 7fae1b7fe700 1 -- 172.21.15.189:0/1742569241 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190873475 0 0) 0x7fae1c0cb910 con 0x7fae1c0cdfd0 2024-04-06T13:51:29.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.344+0000 7fae1b7fe700 1 -- 172.21.15.189:0/1742569241 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae0c0035b0 con 0x7fae1c0cdfd0 2024-04-06T13:51:29.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.344+0000 7fae2137b700 1 -- 172.21.15.189:0/1742569241 >> v2:172.21.15.192:3300/0 conn(0x7fae1c15b310 msgr2=0x7fae1c0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:29.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.344+0000 7fae2137b700 1 --2- 172.21.15.189:0/1742569241 >> v2:172.21.15.192:3300/0 conn(0x7fae1c15b310 0x7fae1c0b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.344+0000 7fae2137b700 1 -- 172.21.15.189:0/1742569241 >> v1:172.21.15.189:6789/0 conn(0x7fae1c0cdfd0 legacy=0x7fae1c0d1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:29.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.344+0000 7fae2137b700 1 -- 172.21.15.189:0/1742569241 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1c0d0b40 con 0x7fae1c0d1a50 2024-04-06T13:51:29.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.345+0000 7fae2137b700 1 --2- 172.21.15.189:0/1742569241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae1c0d1a50 0x7fae1c15ef10 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7fae1c0d2290 tx=0x7fae1400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a06892c2e156a573 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:29.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.345+0000 7fae1b7fe700 1 -- 172.21.15.189:0/1742569241 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1400e070 con 0x7fae1c0d1a50 2024-04-06T13:51:29.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.345+0000 7fae1b7fe700 1 -- 172.21.15.189:0/1742569241 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae1400bb20 con 0x7fae1c0d1a50 2024-04-06T13:51:29.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.345+0000 7fae1b7fe700 1 -- 172.21.15.189:0/1742569241 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1400be90 con 0x7fae1c0d1a50 2024-04-06T13:51:29.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.345+0000 7fae22dde700 1 -- 172.21.15.189:0/1742569241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae1c0d1a50 msgr2=0x7fae1c15ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:29.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.345+0000 7fae22dde700 1 --2- 172.21.15.189:0/1742569241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae1c0d1a50 0x7fae1c15ef10 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7fae1c0d2290 tx=0x7fae1400b330 comp rx=0 tx=0).stop 2024-04-06T13:51:29.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.345+0000 7fae22dde700 1 -- 172.21.15.189:0/1742569241 shutdown_connections 2024-04-06T13:51:29.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.345+0000 7fae22dde700 1 --2- 172.21.15.189:0/1742569241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae1c0d1a50 0x7fae1c15ef10 unknown :-1 s=CLOSED pgs=3172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.345+0000 7fae22dde700 1 --2- 172.21.15.189:0/1742569241 >> v2:172.21.15.192:3300/0 conn(0x7fae1c15b310 0x7fae1c0b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.345+0000 7fae22dde700 1 -- 172.21.15.189:0/1742569241 >> 172.21.15.189:0/1742569241 conn(0x7fae1c06a730 msgr2=0x7fae1c06ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:29.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.346+0000 7fae22dde700 1 -- 172.21.15.189:0/1742569241 shutdown_connections 2024-04-06T13:51:29.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.346+0000 7fae22dde700 1 -- 172.21.15.189:0/1742569241 wait complete. 2024-04-06T13:51:29.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.347+0000 7fae22dde700 1 Processor -- start 2024-04-06T13:51:29.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.347+0000 7fae22dde700 1 -- start start 2024-04-06T13:51:29.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae22dde700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae1c0cdfd0 0x7fae1c06b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:29.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae22dde700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae1c0d1a50 0x7fae1c15df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:29.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae22dde700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae1c15b310 0x7fae1c14f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:29.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae22dde700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c06be40 con 0x7fae1c15b310 2024-04-06T13:51:29.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae22dde700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c06c120 con 0x7fae1c0cdfd0 2024-04-06T13:51:29.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae22dde700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c06c400 con 0x7fae1c0d1a50 2024-04-06T13:51:29.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae2137b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae1c15b310 0x7fae1c14f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:29.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae2137b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae1c15b310 0x7fae1c14f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59970/0 (socket says 172.21.15.189:59970) 2024-04-06T13:51:29.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae2137b700 1 -- 172.21.15.189:0/2752534732 learned_addr learned my addr 172.21.15.189:0/2752534732 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:29.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae20b7a700 1 --2- 172.21.15.189:0/2752534732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae1c0cdfd0 0x7fae1c06b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:29.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae1bfff700 1 --2- 172.21.15.189:0/2752534732 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae1c0d1a50 0x7fae1c15df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:29.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae2137b700 1 -- 172.21.15.189:0/2752534732 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae1c0d1a50 msgr2=0x7fae1c15df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:29.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae2137b700 1 --2- 172.21.15.189:0/2752534732 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae1c0d1a50 0x7fae1c15df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae2137b700 1 -- 172.21.15.189:0/2752534732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae1c0cdfd0 msgr2=0x7fae1c06b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:29.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae2137b700 1 --2- 172.21.15.189:0/2752534732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae1c0cdfd0 0x7fae1c06b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae2137b700 1 -- 172.21.15.189:0/2752534732 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae14009a70 con 0x7fae1c15b310 2024-04-06T13:51:29.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae1bfff700 1 --2- 172.21.15.189:0/2752534732 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae1c0d1a50 0x7fae1c15df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:51:29.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae2137b700 1 --2- 172.21.15.189:0/2752534732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae1c15b310 0x7fae1c14f830 secure :-1 s=READY pgs=3173 cs=0 l=1 rev1=1 crypto rx=0x7fae1400b370 tx=0x7fae14004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:29.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae197fa700 1 -- 172.21.15.189:0/2752534732 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae14004930 con 0x7fae1c15b310 2024-04-06T13:51:29.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae197fa700 1 -- 172.21.15.189:0/2752534732 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae14020070 con 0x7fae1c15b310 2024-04-06T13:51:29.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae197fa700 1 -- 172.21.15.189:0/2752534732 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae140276a0 con 0x7fae1c15b310 2024-04-06T13:51:29.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae22dde700 1 -- 172.21.15.189:0/2752534732 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae1c06c6e0 con 0x7fae1c15b310 2024-04-06T13:51:29.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.348+0000 7fae22dde700 1 -- 172.21.15.189:0/2752534732 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae1c158060 con 0x7fae1c15b310 2024-04-06T13:51:29.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.350+0000 7fae22dde700 1 -- 172.21.15.189:0/2752534732 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae1c0c4340 con 0x7fae1c15b310 2024-04-06T13:51:29.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.350+0000 7fae197fa700 1 -- 172.21.15.189:0/2752534732 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 177) v1 ==== 89412+0+0 (secure 0 0 0) 0x7fae14012430 con 0x7fae1c15b310 2024-04-06T13:51:29.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.350+0000 7fae197fa700 1 --2- 172.21.15.189:0/2752534732 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fae0006af60 0x7fae0006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:29.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.350+0000 7fae20b7a700 1 -- 172.21.15.189:0/2752534732 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fae0006af60 msgr2=0x7fae0006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:29.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.350+0000 7fae197fa700 1 -- 172.21.15.189:0/2752534732 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fae14022070 con 0x7fae1c15b310 2024-04-06T13:51:29.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.350+0000 7fae20b7a700 1 --2- 172.21.15.189:0/2752534732 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fae0006af60 0x7fae0006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:51:29.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.358+0000 7fae197fa700 1 -- 172.21.15.189:0/2752534732 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fae14056230 con 0x7fae1c15b310 2024-04-06T13:51:29.413 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:29.412+0000 7f72d02d9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:51:29.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.551+0000 7fae20b7a700 1 -- 172.21.15.189:0/2752534732 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fae0006af60 msgr2=0x7fae0006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:29.552 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.551+0000 7fae20b7a700 1 --2- 172.21.15.189:0/2752534732 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fae0006af60 0x7fae0006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:51:29.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.570+0000 7fae22dde700 1 -- 172.21.15.189:0/2752534732 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae1c06cba0 con 0x7fae1c15b310 2024-04-06T13:51:29.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.572+0000 7fae197fa700 1 -- 172.21.15.189:0/2752534732 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v286) v1 ==== 74+0+17125 (secure 0 0 0) 0x7fae14059850 con 0x7fae1c15b310 2024-04-06T13:51:29.573 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:51:29.574 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":286,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:51:08.736525+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:49:51.467569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"282","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-04-06T13:49:52.656784+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"283","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"38.5","mappings":[{"from":7,"to":2}]},{"pgid":"38.2b","mappings":[{"from":7,"to":4}]},{"pgid":"38.2e","mappings":[{"from":7,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/2987126770":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/2905400170":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/893866137":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/2221278556":"2024-04-07T13:50:07.256457+0000","172.21.15.189:6833/111704":"2024-04-07T13:50:07.256457+0000","172.21.15.189:6832/111704":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/4009223680":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/3345146368":"2024-04-07T13:50:07.256457+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:51:29.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.574+0000 7fae22dde700 1 -- 172.21.15.189:0/2752534732 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fae0006af60 msgr2=0x7fae0006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:51:29.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.574+0000 7fae22dde700 1 --2- 172.21.15.189:0/2752534732 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fae0006af60 0x7fae0006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.574+0000 7fae22dde700 1 -- 172.21.15.189:0/2752534732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae1c15b310 msgr2=0x7fae1c14f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:29.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.574+0000 7fae22dde700 1 --2- 172.21.15.189:0/2752534732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae1c15b310 0x7fae1c14f830 secure :-1 s=READY pgs=3173 cs=0 l=1 rev1=1 crypto rx=0x7fae1400b370 tx=0x7fae14004ad0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.575+0000 7fae22dde700 1 -- 172.21.15.189:0/2752534732 shutdown_connections 2024-04-06T13:51:29.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.575+0000 7fae22dde700 1 --2- 172.21.15.189:0/2752534732 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fae0006af60 0x7fae0006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.575+0000 7fae22dde700 1 --2- 172.21.15.189:0/2752534732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae1c15b310 0x7fae1c14f830 unknown :-1 s=CLOSED pgs=3173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.575+0000 7fae22dde700 1 --2- 172.21.15.189:0/2752534732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae1c0cdfd0 0x7fae1c06b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.575+0000 7fae22dde700 1 --2- 172.21.15.189:0/2752534732 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae1c0d1a50 0x7fae1c15df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.575+0000 7fae22dde700 1 -- 172.21.15.189:0/2752534732 >> 172.21.15.189:0/2752534732 conn(0x7fae1c06a730 msgr2=0x7fae1c0b6210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:29.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.575+0000 7fae22dde700 1 -- 172.21.15.189:0/2752534732 shutdown_connections 2024-04-06T13:51:29.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.575+0000 7fae22dde700 1 -- 172.21.15.189:0/2752534732 wait complete. 2024-04-06T13:51:29.589 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-06T13:51:29.589 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:51:29.655 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:29.654+0000 7f72d02d9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:51:29.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.715+0000 7f7d9c907700 1 Processor -- start 2024-04-06T13:51:29.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.715+0000 7f7d9c907700 1 -- start start 2024-04-06T13:51:29.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.716+0000 7f7d9c907700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d9815d7b0 0x7f7d980b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:29.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.716+0000 7f7d9c907700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d9806a730 0x7f7d9806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:29.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.716+0000 7f7d9c907700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7d9815ddf0 0x7f7d9815e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:29.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.716+0000 7f7d9c907700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d9806b0f0 con 0x7f7d9806a730 2024-04-06T13:51:29.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.716+0000 7f7d9c907700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d980b3990 con 0x7f7d9815ddf0 2024-04-06T13:51:29.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.716+0000 7f7d9c907700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d980b3060 con 0x7f7d9815d7b0 2024-04-06T13:51:29.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.716+0000 7f7d95d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d9806a730 0x7f7d9806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.716+0000 7f7d95d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d9806a730 0x7f7d9806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59972/0 (socket says 172.21.15.189:59972) 2024-04-06T13:51:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.716+0000 7f7d95d9b700 1 -- 172.21.15.189:0/3416724488 learned_addr learned my addr 172.21.15.189:0/3416724488 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.716+0000 7f7d9659c700 1 --2- 172.21.15.189:0/3416724488 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d9815d7b0 0x7f7d980b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.716+0000 7f7d96d9d700 1 --2- 172.21.15.189:0/3416724488 >> v2:172.21.15.192:3300/0 conn(0x7f7d9815ddf0 0x7f7d9815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.717+0000 7f7d95d9b700 1 -- 172.21.15.189:0/3416724488 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d9815d7b0 msgr2=0x7f7d980b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.717+0000 7f7d95d9b700 1 --2- 172.21.15.189:0/3416724488 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d9815d7b0 0x7f7d980b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.717+0000 7f7d95d9b700 1 -- 172.21.15.189:0/3416724488 >> v2:172.21.15.192:3300/0 conn(0x7f7d9815ddf0 msgr2=0x7f7d9815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.717+0000 7f7d95d9b700 1 --2- 172.21.15.189:0/3416724488 >> v2:172.21.15.192:3300/0 conn(0x7f7d9815ddf0 0x7f7d9815e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.717+0000 7f7d95d9b700 1 -- 172.21.15.189:0/3416724488 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d980b3340 con 0x7f7d9806a730 2024-04-06T13:51:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.717+0000 7f7d95d9b700 1 --2- 172.21.15.189:0/3416724488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d9806a730 0x7f7d9806abb0 secure :-1 s=READY pgs=3174 cs=0 l=1 rev1=1 crypto rx=0x7f7d980b4800 tx=0x7f7d80009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=166845c7d5d9dc73 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.717+0000 7f7d9559a700 1 -- 172.21.15.189:0/3416724488 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d80011830 con 0x7f7d9806a730 2024-04-06T13:51:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.717+0000 7f7d9559a700 1 -- 172.21.15.189:0/3416724488 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d80011990 con 0x7f7d9806a730 2024-04-06T13:51:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.717+0000 7f7d9559a700 1 -- 172.21.15.189:0/3416724488 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d80011d20 con 0x7f7d9806a730 2024-04-06T13:51:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.718+0000 7f7d9c907700 1 -- 172.21.15.189:0/3416724488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d9806a730 msgr2=0x7f7d9806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.718+0000 7f7d9c907700 1 --2- 172.21.15.189:0/3416724488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d9806a730 0x7f7d9806abb0 secure :-1 s=READY pgs=3174 cs=0 l=1 rev1=1 crypto rx=0x7f7d980b4800 tx=0x7f7d80009bd0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.718+0000 7f7d9c907700 1 -- 172.21.15.189:0/3416724488 shutdown_connections 2024-04-06T13:51:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.718+0000 7f7d9c907700 1 --2- 172.21.15.189:0/3416724488 >> v2:172.21.15.192:3300/0 conn(0x7f7d9815ddf0 0x7f7d9815e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.718+0000 7f7d9c907700 1 --2- 172.21.15.189:0/3416724488 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d9815d7b0 0x7f7d980b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.718+0000 7f7d9c907700 1 --2- 172.21.15.189:0/3416724488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d9806a730 0x7f7d9806abb0 unknown :-1 s=CLOSED pgs=3174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.718+0000 7f7d9c907700 1 -- 172.21.15.189:0/3416724488 >> 172.21.15.189:0/3416724488 conn(0x7f7d980d00b0 msgr2=0x7f7d980d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.718+0000 7f7d9c907700 1 -- 172.21.15.189:0/3416724488 shutdown_connections 2024-04-06T13:51:29.721 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.718+0000 7f7d9c907700 1 -- 172.21.15.189:0/3416724488 wait complete. 2024-04-06T13:51:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.719+0000 7f7d9c907700 1 Processor -- start 2024-04-06T13:51:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.720+0000 7f7d9c907700 1 -- start start 2024-04-06T13:51:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.721+0000 7f7d9c907700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d9806a730 0x7f7d98149140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.721+0000 7f7d9c907700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d9815d7b0 0x7f7d9814c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.721+0000 7f7d9c907700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d9815ddf0 0x7f7d981498f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.721+0000 7f7d9c907700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9814fd50 con 0x7f7d9806a730 2024-04-06T13:51:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.721+0000 7f7d9c907700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d98149680 con 0x7f7d9815ddf0 2024-04-06T13:51:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.721+0000 7f7d9c907700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9814c8d0 con 0x7f7d9815d7b0 2024-04-06T13:51:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.721+0000 7f7d9659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d9806a730 0x7f7d98149140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.721+0000 7f7d9659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d9806a730 0x7f7d98149140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59974/0 (socket says 172.21.15.189:59974) 2024-04-06T13:51:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.721+0000 7f7d9659c700 1 -- 172.21.15.189:0/585876728 learned_addr learned my addr 172.21.15.189:0/585876728 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:29.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.721+0000 7f7d96d9d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d9815ddf0 0x7f7d981498f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.721+0000 7f7d95d9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d9815d7b0 0x7f7d9814c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.721+0000 7f7d96d9d700 1 -- 172.21.15.189:0/585876728 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d9815d7b0 msgr2=0x7f7d9814c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.721+0000 7f7d96d9d700 1 --2- 172.21.15.189:0/585876728 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d9815d7b0 0x7f7d9814c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.721+0000 7f7d96d9d700 1 -- 172.21.15.189:0/585876728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d9806a730 msgr2=0x7f7d98149140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.721+0000 7f7d96d9d700 1 --2- 172.21.15.189:0/585876728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d9806a730 0x7f7d98149140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.721+0000 7f7d96d9d700 1 -- 172.21.15.189:0/585876728 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d80009550 con 0x7f7d9815ddf0 2024-04-06T13:51:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.721+0000 7f7d95d9b700 1 --2- 172.21.15.189:0/585876728 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d9815d7b0 0x7f7d9814c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:51:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.721+0000 7f7d9659c700 1 --2- 172.21.15.189:0/585876728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d9806a730 0x7f7d98149140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:51:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.721+0000 7f7d96d9d700 1 --2- 172.21.15.189:0/585876728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d9815ddf0 0x7f7d981498f0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000e820 tx=0x7f7d9000eb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.722+0000 7f7d8effd700 1 -- 172.21.15.189:0/585876728 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d90013680 con 0x7f7d9815ddf0 2024-04-06T13:51:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.722+0000 7f7d8effd700 1 -- 172.21.15.189:0/585876728 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d90013800 con 0x7f7d9815ddf0 2024-04-06T13:51:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.722+0000 7f7d9c907700 1 -- 172.21.15.189:0/585876728 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d9815b460 con 0x7f7d9815ddf0 2024-04-06T13:51:29.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.722+0000 7f7d8effd700 1 -- 172.21.15.189:0/585876728 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d90013bb0 con 0x7f7d9815ddf0 2024-04-06T13:51:29.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.722+0000 7f7d9c907700 1 -- 172.21.15.189:0/585876728 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d9815ba20 con 0x7f7d9815ddf0 2024-04-06T13:51:29.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.723+0000 7f7d9c907700 1 -- 172.21.15.189:0/585876728 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d98067bb0 con 0x7f7d9815ddf0 2024-04-06T13:51:29.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.723+0000 7f7d8effd700 1 -- 172.21.15.189:0/585876728 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 177) v1 ==== 89412+0+0 (secure 0 0 0) 0x7f7d90013d10 con 0x7f7d9815ddf0 2024-04-06T13:51:29.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.724+0000 7f7d8effd700 1 --2- 172.21.15.189:0/585876728 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7d7806af60 0x7f7d7806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:29.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.724+0000 7f7d9659c700 1 -- 172.21.15.189:0/585876728 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7d7806af60 msgr2=0x7f7d7806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:29.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.724+0000 7f7d8effd700 1 -- 172.21.15.189:0/585876728 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7d90089480 con 0x7f7d9815ddf0 2024-04-06T13:51:29.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.724+0000 7f7d9659c700 1 --2- 172.21.15.189:0/585876728 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7d7806af60 0x7f7d7806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:51:29.728 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:29.727+0000 7f3fecaf2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T13:51:29.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.732+0000 7f7d8effd700 1 -- 172.21.15.189:0/585876728 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7d90053f90 con 0x7f7d9815ddf0 2024-04-06T13:51:29.738 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:29.737+0000 7f72d02d9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:51:29.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.925+0000 7f7d9659c700 1 -- 172.21.15.189:0/585876728 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7d7806af60 msgr2=0x7f7d7806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:29.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.925+0000 7f7d9659c700 1 --2- 172.21.15.189:0/585876728 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7d7806af60 0x7f7d7806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:51:29.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.982+0000 7f7d9c907700 1 -- 172.21.15.189:0/585876728 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d98074990 con 0x7f7d9815ddf0 2024-04-06T13:51:29.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.983+0000 7f7d8effd700 1 -- 172.21.15.189:0/585876728 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4759 (secure 0 0 0) 0x7f7d900575b0 con 0x7f7d9815ddf0 2024-04-06T13:51:29.984 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:51:29.984 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":139,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11827,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/819921213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":819921213},{"type":"v1","addr":"172.21.15.192:6835","nonce":819921213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134},{"gid":13368,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3616349784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3616349784},{"type":"v1","addr":"172.21.15.189:6836","nonce":3616349784}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134},{"gid":13371,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3640528525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3640528525},{"type":"v1","addr":"172.21.15.189:6837","nonce":3640528525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:49:53.688018+0000","modified":"2024-04-06T13:49:56.698920+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13380},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13380":{"gid":13380,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6834/3894184871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3894184871},{"type":"v1","addr":"172.21.15.192:6834","nonce":3894184871}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-04-06T13:51:29.985 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:29.984+0000 7f3fecaf2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T13:51:29.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.985+0000 7f7d9c907700 1 -- 172.21.15.189:0/585876728 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7d7806af60 msgr2=0x7f7d7806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:51:29.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.985+0000 7f7d9c907700 1 --2- 172.21.15.189:0/585876728 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7d7806af60 0x7f7d7806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.985+0000 7f7d9c907700 1 -- 172.21.15.189:0/585876728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d9815ddf0 msgr2=0x7f7d981498f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:29.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.985+0000 7f7d9c907700 1 --2- 172.21.15.189:0/585876728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d9815ddf0 0x7f7d981498f0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000e820 tx=0x7f7d9000eb30 comp rx=0 tx=0).stop 2024-04-06T13:51:29.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.985+0000 7f7d9c907700 1 -- 172.21.15.189:0/585876728 shutdown_connections 2024-04-06T13:51:29.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.985+0000 7f7d9c907700 1 --2- 172.21.15.189:0/585876728 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7d7806af60 0x7f7d7806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.985+0000 7f7d9c907700 1 --2- 172.21.15.189:0/585876728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d9815ddf0 0x7f7d981498f0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.985+0000 7f7d9c907700 1 --2- 172.21.15.189:0/585876728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d9806a730 0x7f7d98149140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.985+0000 7f7d9c907700 1 --2- 172.21.15.189:0/585876728 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d9815d7b0 0x7f7d9814c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:29.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.985+0000 7f7d9c907700 1 -- 172.21.15.189:0/585876728 >> 172.21.15.189:0/585876728 conn(0x7f7d980d00b0 msgr2=0x7f7d980ad470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:29.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.986+0000 7f7d9c907700 1 -- 172.21.15.189:0/585876728 shutdown_connections 2024-04-06T13:51:29.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:29.986+0000 7f7d9c907700 1 -- 172.21.15.189:0/585876728 wait complete. 2024-04-06T13:51:29.988 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 139 2024-04-06T13:51:29.999 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:51:30.071 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:30.068+0000 7f3fecaf2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T13:51:30.116 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-06T13:51:30.115+0000 7f72d02d9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:51:30.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.123+0000 7f3dc1680700 1 Processor -- start 2024-04-06T13:51:30.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.123+0000 7f3dc1680700 1 -- start start 2024-04-06T13:51:30.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.124+0000 7f3dc1680700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3dbc15a550 0x7f3dbc0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:30.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.124+0000 7f3dc1680700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3dbc0cf900 0x7f3dbc0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:30.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.124+0000 7f3dc1680700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3dbc0b4ba0 0x7f3dbc06eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:30.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.124+0000 7f3dc1680700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dbc0cdbe0 con 0x7f3dbc15a550 2024-04-06T13:51:30.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.124+0000 7f3dc1680700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dbc0b4ff0 con 0x7f3dbc0cf900 2024-04-06T13:51:30.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.124+0000 7f3dc1680700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dbc15a940 con 0x7f3dbc0b4ba0 2024-04-06T13:51:30.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.124+0000 7f3dbaffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3dbc15a550 0x7f3dbc0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:30.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.124+0000 7f3dbaffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3dbc15a550 0x7f3dbc0cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59980/0 (socket says 172.21.15.189:59980) 2024-04-06T13:51:30.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.124+0000 7f3dbaffd700 1 -- 172.21.15.189:0/3788196640 learned_addr learned my addr 172.21.15.189:0/3788196640 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:30.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.124+0000 7f3dba7fc700 1 --2- 172.21.15.189:0/3788196640 >> v2:172.21.15.192:3300/0 conn(0x7f3dbc0cf900 0x7f3dbc0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:30.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.124+0000 7f3dbaffd700 1 -- 172.21.15.189:0/3788196640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3dbc0b4ba0 msgr2=0x7f3dbc06eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:30.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.124+0000 7f3dbb7fe700 1 --2- 172.21.15.189:0/3788196640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3dbc0b4ba0 0x7f3dbc06eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:30.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.124+0000 7f3dbaffd700 1 --2- 172.21.15.189:0/3788196640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3dbc0b4ba0 0x7f3dbc06eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.124+0000 7f3dbaffd700 1 -- 172.21.15.189:0/3788196640 >> v2:172.21.15.192:3300/0 conn(0x7f3dbc0cf900 msgr2=0x7f3dbc0b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:30.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.124+0000 7f3dbaffd700 1 --2- 172.21.15.189:0/3788196640 >> v2:172.21.15.192:3300/0 conn(0x7f3dbc0cf900 0x7f3dbc0b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.125+0000 7f3dbaffd700 1 -- 172.21.15.189:0/3788196640 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dbc15ac00 con 0x7f3dbc15a550 2024-04-06T13:51:30.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.125+0000 7f3dbaffd700 1 --2- 172.21.15.189:0/3788196640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3dbc15a550 0x7f3dbc0cf3c0 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc0b2d40 tx=0x7f3dac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6f02c58e821b947 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:30.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.125+0000 7f3db9ffb700 1 -- 172.21.15.189:0/3788196640 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac00b870 con 0x7f3dbc15a550 2024-04-06T13:51:30.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.125+0000 7f3db9ffb700 1 -- 172.21.15.189:0/3788196640 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dac00b9d0 con 0x7f3dbc15a550 2024-04-06T13:51:30.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.125+0000 7f3db9ffb700 1 -- 172.21.15.189:0/3788196640 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac00bd60 con 0x7f3dbc15a550 2024-04-06T13:51:30.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.126+0000 7f3dc1680700 1 -- 172.21.15.189:0/3788196640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3dbc15a550 msgr2=0x7f3dbc0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:30.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.126+0000 7f3dc1680700 1 --2- 172.21.15.189:0/3788196640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3dbc15a550 0x7f3dbc0cf3c0 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc0b2d40 tx=0x7f3dac009bd0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.126+0000 7f3dc1680700 1 -- 172.21.15.189:0/3788196640 shutdown_connections 2024-04-06T13:51:30.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.126+0000 7f3dc1680700 1 --2- 172.21.15.189:0/3788196640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3dbc15a550 0x7f3dbc0cf3c0 unknown :-1 s=CLOSED pgs=3175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.126+0000 7f3dc1680700 1 --2- 172.21.15.189:0/3788196640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3dbc0b4ba0 0x7f3dbc06eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.126+0000 7f3dc1680700 1 --2- 172.21.15.189:0/3788196640 >> v2:172.21.15.192:3300/0 conn(0x7f3dbc0cf900 0x7f3dbc0b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.126+0000 7f3dc1680700 1 -- 172.21.15.189:0/3788196640 >> 172.21.15.189:0/3788196640 conn(0x7f3dbc0d00b0 msgr2=0x7f3dbc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:30.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.126+0000 7f3dc1680700 1 -- 172.21.15.189:0/3788196640 shutdown_connections 2024-04-06T13:51:30.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.126+0000 7f3dc1680700 1 -- 172.21.15.189:0/3788196640 wait complete. 2024-04-06T13:51:30.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.128+0000 7f3dc1680700 1 Processor -- start 2024-04-06T13:51:30.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.128+0000 7f3dc1680700 1 -- start start 2024-04-06T13:51:30.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.129+0000 7f3dc1680700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3dbc0b4ba0 0x7f3dbc14c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:30.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.129+0000 7f3dc1680700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3dbc0cf900 0x7f3dbc158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:30.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.129+0000 7f3dc1680700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3dbc1570b0 0x7f3dbc15ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:30.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.129+0000 7f3dc1680700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dbc1510e0 con 0x7f3dbc0b4ba0 2024-04-06T13:51:30.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.129+0000 7f3dc1680700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dbc1ef7c0 con 0x7f3dbc0cf900 2024-04-06T13:51:30.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.129+0000 7f3dc1680700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dbc158130 con 0x7f3dbc1570b0 2024-04-06T13:51:30.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.129+0000 7f3dbaffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3dbc0b4ba0 0x7f3dbc14c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:30.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.129+0000 7f3dbaffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3dbc0b4ba0 0x7f3dbc14c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59990/0 (socket says 172.21.15.189:59990) 2024-04-06T13:51:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.129+0000 7f3dba7fc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3dbc0cf900 0x7f3dbc158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.129+0000 7f3dbaffd700 1 -- 172.21.15.189:0/1370263000 learned_addr learned my addr 172.21.15.189:0/1370263000 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.129+0000 7f3dbb7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3dbc1570b0 0x7f3dbc15ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.129+0000 7f3dbaffd700 1 -- 172.21.15.189:0/1370263000 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3dbc1570b0 msgr2=0x7f3dbc15ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.129+0000 7f3dbaffd700 1 --2- 172.21.15.189:0/1370263000 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3dbc1570b0 0x7f3dbc15ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.129+0000 7f3dbaffd700 1 -- 172.21.15.189:0/1370263000 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3dbc0cf900 msgr2=0x7f3dbc158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.129+0000 7f3dbaffd700 1 --2- 172.21.15.189:0/1370263000 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3dbc0cf900 0x7f3dbc158f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.129+0000 7f3dbaffd700 1 -- 172.21.15.189:0/1370263000 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dac009550 con 0x7f3dbc0b4ba0 2024-04-06T13:51:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.129+0000 7f3dbaffd700 1 --2- 172.21.15.189:0/1370263000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3dbc0b4ba0 0x7f3dbc14c750 secure :-1 s=READY pgs=3176 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc06c820 tx=0x7f3dac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.130+0000 7f3da37fe700 1 -- 172.21.15.189:0/1370263000 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac004520 con 0x7f3dbc0b4ba0 2024-04-06T13:51:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.130+0000 7f3da37fe700 1 -- 172.21.15.189:0/1370263000 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dac0036a0 con 0x7f3dbc0b4ba0 2024-04-06T13:51:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.130+0000 7f3dc1680700 1 -- 172.21.15.189:0/1370263000 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dbc1575b0 con 0x7f3dbc0b4ba0 2024-04-06T13:51:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.130+0000 7f3da37fe700 1 -- 172.21.15.189:0/1370263000 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac003a30 con 0x7f3dbc0b4ba0 2024-04-06T13:51:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.130+0000 7f3dc1680700 1 -- 172.21.15.189:0/1370263000 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dbc157b70 con 0x7f3dbc0b4ba0 2024-04-06T13:51:30.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.131+0000 7f3dc1680700 1 -- 172.21.15.189:0/1370263000 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dbc14db80 con 0x7f3dbc0b4ba0 2024-04-06T13:51:30.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.131+0000 7f3da37fe700 1 -- 172.21.15.189:0/1370263000 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 177) v1 ==== 89412+0+0 (secure 0 0 0) 0x7f3dac003b90 con 0x7f3dbc0b4ba0 2024-04-06T13:51:30.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.132+0000 7f3da37fe700 1 --2- 172.21.15.189:0/1370263000 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3da806af60 0x7f3da806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:30.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.132+0000 7f3dba7fc700 1 -- 172.21.15.189:0/1370263000 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3da806af60 msgr2=0x7f3da806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:30.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.132+0000 7f3da37fe700 1 -- 172.21.15.189:0/1370263000 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3dac08ca60 con 0x7f3dbc0b4ba0 2024-04-06T13:51:30.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.132+0000 7f3dba7fc700 1 --2- 172.21.15.189:0/1370263000 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3da806af60 0x7f3da806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T13:51:30.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.136+0000 7f3da37fe700 1 -- 172.21.15.189:0/1370263000 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3dbc14db80 con 0x7f3dbc0b4ba0 2024-04-06T13:51:30.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.333+0000 7f3dba7fc700 1 -- 172.21.15.189:0/1370263000 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3da806af60 msgr2=0x7f3da806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:6832/111704 2024-04-06T13:51:30.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.333+0000 7f3dba7fc700 1 --2- 172.21.15.189:0/1370263000 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3da806af60 0x7f3da806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T13:51:30.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.386+0000 7f3dc1680700 1 -- 172.21.15.189:0/1370263000 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3dbc14db80 con 0x7f3dbc0b4ba0 2024-04-06T13:51:30.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.386+0000 7f3da37fe700 1 -- 172.21.15.189:0/1370263000 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4759 (secure 0 0 0) 0x7f3dbc14db80 con 0x7f3dbc0b4ba0 2024-04-06T13:51:30.387 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:51:30.387 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":139,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11827,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/819921213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":819921213},{"type":"v1","addr":"172.21.15.192:6835","nonce":819921213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134},{"gid":13368,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3616349784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3616349784},{"type":"v1","addr":"172.21.15.189:6836","nonce":3616349784}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134},{"gid":13371,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3640528525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3640528525},{"type":"v1","addr":"172.21.15.189:6837","nonce":3640528525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T13:49:53.688018+0000","modified":"2024-04-06T13:49:56.698920+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13380},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13380":{"gid":13380,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6834/3894184871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3894184871},{"type":"v1","addr":"172.21.15.192:6834","nonce":3894184871}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-04-06T13:51:30.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.389+0000 7f3dc1680700 1 -- 172.21.15.189:0/1370263000 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3da806af60 msgr2=0x7f3da806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T13:51:30.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.389+0000 7f3dc1680700 1 --2- 172.21.15.189:0/1370263000 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3da806af60 0x7f3da806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.389+0000 7f3dc1680700 1 -- 172.21.15.189:0/1370263000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3dbc0b4ba0 msgr2=0x7f3dbc14c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:30.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.389+0000 7f3dc1680700 1 --2- 172.21.15.189:0/1370263000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3dbc0b4ba0 0x7f3dbc14c750 secure :-1 s=READY pgs=3176 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc06c820 tx=0x7f3dac004750 comp rx=0 tx=0).stop 2024-04-06T13:51:30.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.389+0000 7f3dc1680700 1 -- 172.21.15.189:0/1370263000 shutdown_connections 2024-04-06T13:51:30.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.389+0000 7f3dc1680700 1 --2- 172.21.15.189:0/1370263000 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3da806af60 0x7f3da806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.389+0000 7f3dc1680700 1 --2- 172.21.15.189:0/1370263000 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3dbc1570b0 0x7f3dbc15ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.389+0000 7f3dc1680700 1 --2- 172.21.15.189:0/1370263000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3dbc0b4ba0 0x7f3dbc14c750 unknown :-1 s=CLOSED pgs=3176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.389+0000 7f3dc1680700 1 --2- 172.21.15.189:0/1370263000 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3dbc0cf900 0x7f3dbc158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.389+0000 7f3dc1680700 1 -- 172.21.15.189:0/1370263000 >> 172.21.15.189:0/1370263000 conn(0x7f3dbc0d00b0 msgr2=0x7f3dbc06f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:30.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.389+0000 7f3dc1680700 1 -- 172.21.15.189:0/1370263000 shutdown_connections 2024-04-06T13:51:30.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.389+0000 7f3dc1680700 1 -- 172.21.15.189:0/1370263000 wait complete. 2024-04-06T13:51:30.391 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 139 2024-04-06T13:51:30.403 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:51:30.470 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:30.469+0000 7f3fecaf2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T13:51:30.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.554+0000 7f71a6a8d700 1 Processor -- start 2024-04-06T13:51:30.555 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.554+0000 7f71a6a8d700 1 -- start start 2024-04-06T13:51:30.559 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.555+0000 7f71a6a8d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71a014b4c0 0x7f71a00b2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:30.559 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.555+0000 7f71a6a8d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71a01620e0 con 0x7f71a014b0e0 2024-04-06T13:51:30.559 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.555+0000 7f71a6a8d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71a01623a0 con 0x7f71a00cf640 2024-04-06T13:51:30.559 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.555+0000 7f71a6a8d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a0162680 con 0x7f71a014b4c0 2024-04-06T13:51:30.559 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.555+0000 7f71a4829700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f71a014b0e0 0x7f71a00cef30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54008/0 (socket says 172.21.15.189:54008) 2024-04-06T13:51:30.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.555+0000 7f71a4829700 1 -- 172.21.15.189:0/3002269253 learned_addr learned my addr 172.21.15.189:0/3002269253 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:30.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.555+0000 7f71a502a700 1 --2- 172.21.15.189:0/3002269253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71a014b4c0 0x7f71a00b2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:30.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.555+0000 7f719f7fe700 1 -- 172.21.15.189:0/3002269253 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854898169 0 0) 0x7f71a01620e0 con 0x7f71a014b0e0 2024-04-06T13:51:30.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.555+0000 7f719f7fe700 1 -- 172.21.15.189:0/3002269253 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71900035b0 con 0x7f71a014b0e0 2024-04-06T13:51:30.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.555+0000 7f719f7fe700 1 -- 172.21.15.189:0/3002269253 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2641631133 0 0) 0x7f71900035b0 con 0x7f71a014b0e0 2024-04-06T13:51:30.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.555+0000 7f719f7fe700 1 -- 172.21.15.189:0/3002269253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71a014b4c0 msgr2=0x7f71a00b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:30.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.555+0000 7f719f7fe700 1 --2- 172.21.15.189:0/3002269253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71a014b4c0 0x7f71a00b2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.555+0000 7f719f7fe700 1 -- 172.21.15.189:0/3002269253 >> v1:172.21.15.192:6789/0 conn(0x7f71a00cf640 legacy=0x7f71a00b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:30.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.555+0000 7f719f7fe700 1 -- 172.21.15.189:0/3002269253 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a0162960 con 0x7f71a014b0e0 2024-04-06T13:51:30.560 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.555+0000 7f719f7fe700 1 -- 172.21.15.189:0/3002269253 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f7194001670 con 0x7f71a014b0e0 2024-04-06T13:51:30.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.556+0000 7f719f7fe700 1 -- 172.21.15.189:0/3002269253 >> v1:172.21.15.189:6789/0 conn(0x7f71a014b0e0 legacy=0x7f71a00cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:30.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.556+0000 7f719f7fe700 1 --2- 172.21.15.189:0/3002269253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71a014b4c0 0x7f7190003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:30.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.556+0000 7f719f7fe700 1 --2- 172.21.15.189:0/3002269253 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7190003f70 0x7f7190004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:30.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.556+0000 7f719f7fe700 1 --2- 172.21.15.189:0/3002269253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7190004d90 0x7f7190005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:30.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.556+0000 7f719f7fe700 1 -- 172.21.15.189:0/3002269253 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a0162680 con 0x7f7190004d90 2024-04-06T13:51:30.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.556+0000 7f719f7fe700 1 -- 172.21.15.189:0/3002269253 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719000ba70 con 0x7f7190003f70 2024-04-06T13:51:30.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.556+0000 7f719f7fe700 1 -- 172.21.15.189:0/3002269253 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f719000bc30 con 0x7f71a014b4c0 2024-04-06T13:51:30.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.556+0000 7f71a502a700 1 --2- 172.21.15.189:0/3002269253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7190004d90 0x7f7190005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:30.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.556+0000 7f71a4829700 1 --2- 172.21.15.189:0/3002269253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71a014b4c0 0x7f7190003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:30.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.556+0000 7f719ffff700 1 --2- 172.21.15.189:0/3002269253 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7190003f70 0x7f7190004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:30.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.556+0000 7f71a502a700 1 -- 172.21.15.189:0/3002269253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71a014b4c0 msgr2=0x7f7190003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:30.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.556+0000 7f71a502a700 1 --2- 172.21.15.189:0/3002269253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71a014b4c0 0x7f7190003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.556+0000 7f71a502a700 1 -- 172.21.15.189:0/3002269253 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7190003f70 msgr2=0x7f7190004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:30.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.556+0000 7f71a502a700 1 --2- 172.21.15.189:0/3002269253 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7190003f70 0x7f7190004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.556+0000 7f71a502a700 1 -- 172.21.15.189:0/3002269253 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71900035b0 con 0x7f7190004d90 2024-04-06T13:51:30.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.556+0000 7f71a502a700 1 --2- 172.21.15.189:0/3002269253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7190004d90 0x7f7190005210 secure :-1 s=READY pgs=3181 cs=0 l=1 rev1=1 crypto rx=0x7f7198009fc0 tx=0x7f7190005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=819994728d5dd114 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:30.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.556+0000 7f719f7fe700 1 -- 172.21.15.189:0/3002269253 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719800ec30 con 0x7f7190004d90 2024-04-06T13:51:30.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.556+0000 7f719f7fe700 1 -- 172.21.15.189:0/3002269253 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719800f040 con 0x7f7190004d90 2024-04-06T13:51:30.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.556+0000 7f719f7fe700 1 -- 172.21.15.189:0/3002269253 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7198014480 con 0x7f7190004d90 2024-04-06T13:51:30.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.557+0000 7f71a6a8d700 1 -- 172.21.15.189:0/3002269253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7190004d90 msgr2=0x7f7190005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:30.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.557+0000 7f71a6a8d700 1 --2- 172.21.15.189:0/3002269253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7190004d90 0x7f7190005210 secure :-1 s=READY pgs=3181 cs=0 l=1 rev1=1 crypto rx=0x7f7198009fc0 tx=0x7f7190005880 comp rx=0 tx=0).stop 2024-04-06T13:51:30.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.557+0000 7f71a502a700 1 -- 172.21.15.189:0/3002269253 reap_dead start 2024-04-06T13:51:30.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.557+0000 7f71a6a8d700 1 -- 172.21.15.189:0/3002269253 shutdown_connections 2024-04-06T13:51:30.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.557+0000 7f71a6a8d700 1 -- 172.21.15.189:0/3002269253 >> 172.21.15.189:0/3002269253 conn(0x7f71a006a730 msgr2=0x7f71a006ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:30.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.557+0000 7f71a6a8d700 1 -- 172.21.15.189:0/3002269253 shutdown_connections 2024-04-06T13:51:30.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.557+0000 7f71a6a8d700 1 -- 172.21.15.189:0/3002269253 wait complete. 2024-04-06T13:51:30.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.558+0000 7f71a6a8d700 1 Processor -- start 2024-04-06T13:51:30.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.558+0000 7f71a6a8d700 1 -- start start 2024-04-06T13:51:30.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.559+0000 7f71a6a8d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7190004d90 0x7f71a01fd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:30.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.559+0000 7f71a6a8d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71a0200090 0x7f71a0200510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:30.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.559+0000 7f71a6a8d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71a00c6590 0x7f71a00c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:30.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.559+0000 7f71a6a8d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a00c6f50 con 0x7f71a00c6590 2024-04-06T13:51:30.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.559+0000 7f71a6a8d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a02010d0 con 0x7f7190004d90 2024-04-06T13:51:30.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.559+0000 7f71a6a8d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a0200b10 con 0x7f71a0200090 2024-04-06T13:51:30.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.560+0000 7f71a502a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71a00c6590 0x7f71a00c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:30.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.560+0000 7f71a502a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71a00c6590 0x7f71a00c6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60032/0 (socket says 172.21.15.189:60032) 2024-04-06T13:51:30.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.560+0000 7f71a502a700 1 -- 172.21.15.189:0/4118118130 learned_addr learned my addr 172.21.15.189:0/4118118130 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:30.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.560+0000 7f719ffff700 1 --2- 172.21.15.189:0/4118118130 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71a0200090 0x7f71a0200510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:30.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.560+0000 7f71a4829700 1 --2- 172.21.15.189:0/4118118130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7190004d90 0x7f71a01fd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:30.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.560+0000 7f71a502a700 1 -- 172.21.15.189:0/4118118130 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71a0200090 msgr2=0x7f71a0200510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:30.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.560+0000 7f71a502a700 1 --2- 172.21.15.189:0/4118118130 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71a0200090 0x7f71a0200510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.560+0000 7f71a502a700 1 -- 172.21.15.189:0/4118118130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7190004d90 msgr2=0x7f71a01fd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:30.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.560+0000 7f71a502a700 1 --2- 172.21.15.189:0/4118118130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7190004d90 0x7f71a01fd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.560+0000 7f71a502a700 1 -- 172.21.15.189:0/4118118130 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a0200df0 con 0x7f71a00c6590 2024-04-06T13:51:30.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.560+0000 7f71a502a700 1 --2- 172.21.15.189:0/4118118130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71a00c6590 0x7f71a00c6a10 secure :-1 s=READY pgs=3182 cs=0 l=1 rev1=1 crypto rx=0x7f7198017040 tx=0x7f7198009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:30.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.560+0000 7f719d7fa700 1 -- 172.21.15.189:0/4118118130 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7198007000 con 0x7f71a00c6590 2024-04-06T13:51:30.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.560+0000 7f719d7fa700 1 -- 172.21.15.189:0/4118118130 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7198003e50 con 0x7f71a00c6590 2024-04-06T13:51:30.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.560+0000 7f71a6a8d700 1 -- 172.21.15.189:0/4118118130 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a015f930 con 0x7f71a00c6590 2024-04-06T13:51:30.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.560+0000 7f719d7fa700 1 -- 172.21.15.189:0/4118118130 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71980149f0 con 0x7f71a00c6590 2024-04-06T13:51:30.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.560+0000 7f71a6a8d700 1 -- 172.21.15.189:0/4118118130 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a015b8d0 con 0x7f71a00c6590 2024-04-06T13:51:30.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.561+0000 7f71a6a8d700 1 -- 172.21.15.189:0/4118118130 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a0073270 con 0x7f71a00c6590 2024-04-06T13:51:30.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.563+0000 7f719d7fa700 1 -- 172.21.15.189:0/4118118130 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 178) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f7198024020 con 0x7f71a00c6590 2024-04-06T13:51:30.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.563+0000 7f719d7fa700 1 -- 172.21.15.189:0/4118118130 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 6919+0+0 (secure 0 0 0) 0x7f719808b240 con 0x7f71a00c6590 2024-04-06T13:51:30.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.568+0000 7f719d7fa700 1 -- 172.21.15.189:0/4118118130 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7198056340 con 0x7f71a00c6590 2024-04-06T13:51:30.598 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:30.597+0000 7f3f651cb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:51:30.598 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:30.597+0000 7f3f651cb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:51:30.598 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:30.597+0000 7f3f651cb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:51:30.598 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:30.597+0000 7f3f651cb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:51:30.598 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:30.597+0000 7f3f651cb700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:51:30.598 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:30.598+0000 7f3f681d1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:51:30.599 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:30.598+0000 7f3f681d1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:51:30.599 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:30.598+0000 7f3f681d1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:51:30.599 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:30.598+0000 7f3f681d1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:51:30.599 INFO:tasks.ceph.mgr.y.smithi189.stderr:2024-04-06T13:51:30.598+0000 7f3f681d1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-06T13:51:30.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.776+0000 7f71a6a8d700 1 -- 172.21.15.189:0/4118118130 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71a0074990 con 0x7f71a00c6590 2024-04-06T13:51:30.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.777+0000 7f719d7fa700 1 -- 172.21.15.189:0/4118118130 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v287) v1 ==== 74+0+17689 (secure 0 0 0) 0x7f719802c090 con 0x7f71a00c6590 2024-04-06T13:51:30.779 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:51:30.779 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":287,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:51:30.485333+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-04-06T13:49:51.467569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"282","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-04-06T13:49:52.656784+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"283","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"38.5","mappings":[{"from":7,"to":2}]},{"pgid":"38.2b","mappings":[{"from":7,"to":4}]},{"pgid":"38.2e","mappings":[{"from":7,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:0/1303197157":"2024-04-07T13:51:30.485281+0000","172.21.15.189:0/3577027484":"2024-04-07T13:51:30.485281+0000","172.21.15.189:0/4161423243":"2024-04-07T13:51:30.485281+0000","172.21.15.189:0/2780155014":"2024-04-07T13:51:30.485281+0000","172.21.15.189:0/769203804":"2024-04-07T13:51:30.485281+0000","172.21.15.189:0/489741497":"2024-04-07T13:51:30.485281+0000","172.21.15.189:0/1941733448":"2024-04-07T13:51:30.485281+0000","172.21.15.189:0/4106081342":"2024-04-07T13:51:30.485281+0000","172.21.15.189:0/867158792":"2024-04-07T13:51:30.485281+0000","172.21.15.189:0/2987126770":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/2905400170":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/893866137":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/2221278556":"2024-04-07T13:50:07.256457+0000","172.21.15.189:6833/111704":"2024-04-07T13:50:07.256457+0000","172.21.15.189:6832/111704":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/4009223680":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/3345146368":"2024-04-07T13:50:07.256457+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:51:30.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.779+0000 7f71a6a8d700 1 -- 172.21.15.189:0/4118118130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71a00c6590 msgr2=0x7f71a00c6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:30.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.779+0000 7f71a6a8d700 1 --2- 172.21.15.189:0/4118118130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71a00c6590 0x7f71a00c6a10 secure :-1 s=READY pgs=3182 cs=0 l=1 rev1=1 crypto rx=0x7f7198017040 tx=0x7f7198009750 comp rx=0 tx=0).stop 2024-04-06T13:51:30.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.779+0000 7f71a6a8d700 1 -- 172.21.15.189:0/4118118130 shutdown_connections 2024-04-06T13:51:30.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.779+0000 7f71a6a8d700 1 --2- 172.21.15.189:0/4118118130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71a00c6590 0x7f71a00c6a10 unknown :-1 s=CLOSED pgs=3182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.779+0000 7f71a6a8d700 1 --2- 172.21.15.189:0/4118118130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7190004d90 0x7f71a01fd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.779+0000 7f71a6a8d700 1 --2- 172.21.15.189:0/4118118130 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71a0200090 0x7f71a0200510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.779+0000 7f71a6a8d700 1 -- 172.21.15.189:0/4118118130 >> 172.21.15.189:0/4118118130 conn(0x7f71a006a730 msgr2=0x7f71a00ce6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:30.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.780+0000 7f71a6a8d700 1 -- 172.21.15.189:0/4118118130 shutdown_connections 2024-04-06T13:51:30.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.780+0000 7f71a6a8d700 1 -- 172.21.15.189:0/4118118130 wait complete. 2024-04-06T13:51:30.793 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-06T13:51:30.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.914+0000 7f49d783c700 1 Processor -- start 2024-04-06T13:51:30.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.914+0000 7f49d783c700 1 -- start start 2024-04-06T13:51:30.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.915+0000 7f49d783c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f49d0144f40 0x7f49d006a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:30.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.915+0000 7f49d783c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d015c740 con 0x7f49d015c150 2024-04-06T13:51:30.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.915+0000 7f49d783c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d015f1f0 con 0x7f49d015fcd0 2024-04-06T13:51:30.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.915+0000 7f49d783c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d015ca20 con 0x7f49d0144f40 2024-04-06T13:51:30.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.915+0000 7f49d55d8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f49d0144f40 0x7f49d006a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:30.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.915+0000 7f49d5dd9700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f49d015c150 0x7f49d00ceca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54012/0 (socket says 172.21.15.189:54012) 2024-04-06T13:51:30.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.915+0000 7f49d55d8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f49d0144f40 0x7f49d006a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60130/0 (socket says 172.21.15.189:60130) 2024-04-06T13:51:30.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.915+0000 7f49d5dd9700 1 -- 172.21.15.189:0/86815927 learned_addr learned my addr 172.21.15.189:0/86815927 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:30.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.915+0000 7f49d55d8700 1 -- 172.21.15.189:0/86815927 >> v1:172.21.15.192:6789/0 conn(0x7f49d015fcd0 legacy=0x7f49d015ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:30.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.915+0000 7f49d55d8700 1 -- 172.21.15.189:0/86815927 >> v1:172.21.15.189:6789/0 conn(0x7f49d015c150 legacy=0x7f49d00ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:30.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.915+0000 7f49c7fff700 1 -- 172.21.15.189:0/86815927 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240937419 0 0) 0x7f49d015f1f0 con 0x7f49d015fcd0 2024-04-06T13:51:30.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.915+0000 7f49d55d8700 1 -- 172.21.15.189:0/86815927 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d015f4d0 con 0x7f49d0144f40 2024-04-06T13:51:30.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.916+0000 7f49d55d8700 1 --2- 172.21.15.189:0/86815927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d0144f40 0x7f49d006a730 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009840 tx=0x7f49c0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bdcf2d7fe251572 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:30.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.916+0000 7f49c7fff700 1 -- 172.21.15.189:0/86815927 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0004230 con 0x7f49d0144f40 2024-04-06T13:51:30.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.916+0000 7f49c7fff700 1 -- 172.21.15.189:0/86815927 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c0004390 con 0x7f49d0144f40 2024-04-06T13:51:30.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.916+0000 7f49c7fff700 1 -- 172.21.15.189:0/86815927 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c00036a0 con 0x7f49d0144f40 2024-04-06T13:51:30.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.916+0000 7f49d783c700 1 -- 172.21.15.189:0/86815927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d0144f40 msgr2=0x7f49d006a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:30.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.916+0000 7f49d783c700 1 --2- 172.21.15.189:0/86815927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d0144f40 0x7f49d006a730 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009840 tx=0x7f49c0009b50 comp rx=0 tx=0).stop 2024-04-06T13:51:30.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.916+0000 7f49d783c700 1 -- 172.21.15.189:0/86815927 shutdown_connections 2024-04-06T13:51:30.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.916+0000 7f49d783c700 1 --2- 172.21.15.189:0/86815927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d0144f40 0x7f49d006a730 unknown :-1 s=CLOSED pgs=3187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.916+0000 7f49d783c700 1 -- 172.21.15.189:0/86815927 >> 172.21.15.189:0/86815927 conn(0x7f49d00d00b0 msgr2=0x7f49d00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:30.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.916+0000 7f49d783c700 1 -- 172.21.15.189:0/86815927 shutdown_connections 2024-04-06T13:51:30.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.916+0000 7f49d783c700 1 -- 172.21.15.189:0/86815927 wait complete. 2024-04-06T13:51:30.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.918+0000 7f49d783c700 1 Processor -- start 2024-04-06T13:51:30.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.918+0000 7f49d783c700 1 -- start start 2024-04-06T13:51:30.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.919+0000 7f49d783c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d0144f40 0x7f49d013a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:30.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.919+0000 7f49d783c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49d015c150 0x7f49d01577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:30.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.919+0000 7f49d783c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49d015fcd0 0x7f49d0143160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:30.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.919+0000 7f49d783c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d0143db0 con 0x7f49d0144f40 2024-04-06T13:51:30.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.919+0000 7f49d783c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d0141340 con 0x7f49d015fcd0 2024-04-06T13:51:30.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.919+0000 7f49d783c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d0143ad0 con 0x7f49d015c150 2024-04-06T13:51:30.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.919+0000 7f49d55d8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d0144f40 0x7f49d013a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:30.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.919+0000 7f49d55d8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d0144f40 0x7f49d013a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60134/0 (socket says 172.21.15.189:60134) 2024-04-06T13:51:30.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.919+0000 7f49d55d8700 1 -- 172.21.15.189:0/3729943230 learned_addr learned my addr 172.21.15.189:0/3729943230 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:30.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.919+0000 7f49d4dd7700 1 --2- 172.21.15.189:0/3729943230 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49d015c150 0x7f49d01577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:30.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.919+0000 7f49d5dd9700 1 --2- 172.21.15.189:0/3729943230 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49d015fcd0 0x7f49d0143160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:30.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.919+0000 7f49d55d8700 1 -- 172.21.15.189:0/3729943230 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49d015c150 msgr2=0x7f49d01577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:30.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.919+0000 7f49d55d8700 1 --2- 172.21.15.189:0/3729943230 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49d015c150 0x7f49d01577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.919+0000 7f49d55d8700 1 -- 172.21.15.189:0/3729943230 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49d015fcd0 msgr2=0x7f49d0143160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:30.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.919+0000 7f49d55d8700 1 --2- 172.21.15.189:0/3729943230 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49d015fcd0 0x7f49d0143160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:30.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.919+0000 7f49d55d8700 1 -- 172.21.15.189:0/3729943230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c00094a0 con 0x7f49d0144f40 2024-04-06T13:51:30.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.919+0000 7f49d5dd9700 1 --2- 172.21.15.189:0/3729943230 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49d015fcd0 0x7f49d0143160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:51:30.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.919+0000 7f49d55d8700 1 --2- 172.21.15.189:0/3729943230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d0144f40 0x7f49d013a2a0 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 crypto rx=0x7f49c00090b0 tx=0x7f49c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:30.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.920+0000 7f49c5ffb700 1 -- 172.21.15.189:0/3729943230 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0011830 con 0x7f49d0144f40 2024-04-06T13:51:30.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.920+0000 7f49c5ffb700 1 -- 172.21.15.189:0/3729943230 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c001f070 con 0x7f49d0144f40 2024-04-06T13:51:30.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.920+0000 7f49d783c700 1 -- 172.21.15.189:0/3729943230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d0140cf0 con 0x7f49d0144f40 2024-04-06T13:51:30.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.920+0000 7f49c5ffb700 1 -- 172.21.15.189:0/3729943230 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0011b90 con 0x7f49d0144f40 2024-04-06T13:51:30.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.920+0000 7f49d783c700 1 -- 172.21.15.189:0/3729943230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d01436a0 con 0x7f49d0144f40 2024-04-06T13:51:30.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.921+0000 7f49d783c700 1 -- 172.21.15.189:0/3729943230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d0140b50 con 0x7f49d0144f40 2024-04-06T13:51:30.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.921+0000 7f49c5ffb700 1 -- 172.21.15.189:0/3729943230 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 178) v1 ==== 88856+0+0 (secure 0 0 0) 0x7f49c0011cf0 con 0x7f49d0144f40 2024-04-06T13:51:30.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.922+0000 7f49c5ffb700 1 -- 172.21.15.189:0/3729943230 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 6919+0+0 (secure 0 0 0) 0x7f49c0021070 con 0x7f49d0144f40 2024-04-06T13:51:30.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:30.926+0000 7f49c5ffb700 1 -- 172.21.15.189:0/3729943230 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f49d0140b50 con 0x7f49d0144f40 2024-04-06T13:51:31.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.160+0000 7f49d783c700 1 -- 172.21.15.189:0/3729943230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f49d0140b50 con 0x7f49d0144f40 2024-04-06T13:51:31.162 INFO:tasks.ceph.mon.a.smithi189.stderr:2024-04-06T13:51:31.161+0000 7fa9fe1ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T13:51:31.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.171+0000 7f49c5ffb700 1 -- 172.21.15.189:0/3729943230 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v140) v1 ==== 161+0+0 (secure 0 0 0) 0x7f49d0140b50 con 0x7f49d0144f40 2024-04-06T13:51:31.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.173+0000 7f49d783c700 1 -- 172.21.15.189:0/3729943230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d0144f40 msgr2=0x7f49d013a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:31.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.173+0000 7f49d783c700 1 --2- 172.21.15.189:0/3729943230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d0144f40 0x7f49d013a2a0 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 crypto rx=0x7f49c00090b0 tx=0x7f49c0004750 comp rx=0 tx=0).stop 2024-04-06T13:51:31.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.173+0000 7f49d783c700 1 -- 172.21.15.189:0/3729943230 shutdown_connections 2024-04-06T13:51:31.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.173+0000 7f49d783c700 1 --2- 172.21.15.189:0/3729943230 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49d015fcd0 0x7f49d0143160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:31.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.173+0000 7f49d783c700 1 --2- 172.21.15.189:0/3729943230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d0144f40 0x7f49d013a2a0 unknown :-1 s=CLOSED pgs=3188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:31.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.173+0000 7f49d783c700 1 --2- 172.21.15.189:0/3729943230 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49d015c150 0x7f49d01577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:31.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.173+0000 7f49d783c700 1 -- 172.21.15.189:0/3729943230 >> 172.21.15.189:0/3729943230 conn(0x7f49d00d00b0 msgr2=0x7f49d00cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:31.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.173+0000 7f49d783c700 1 -- 172.21.15.189:0/3729943230 shutdown_connections 2024-04-06T13:51:31.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.173+0000 7f49d783c700 1 -- 172.21.15.189:0/3729943230 wait complete. 2024-04-06T13:51:31.177 INFO:teuthology.orchestra.run.smithi189.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-06T13:51:31.187 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-06T13:51:31.208 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2024-04-06T13:51:31.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.309+0000 7fea5a113700 1 Processor -- start 2024-04-06T13:51:31.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.309+0000 7fea5a113700 1 -- start start 2024-04-06T13:51:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.310+0000 7fea5a113700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fea5415e8a0 0x7fea540cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.310+0000 7fea5a113700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea5415eee0 0x7fea5415f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.310+0000 7fea5a113700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fea5415f8a0 0x7fea5406ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.310+0000 7fea5a113700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea540669d0 con 0x7fea5415e8a0 2024-04-06T13:51:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.310+0000 7fea5a113700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea5406ede0 con 0x7fea5415f8a0 2024-04-06T13:51:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.310+0000 7fea5a113700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea5415ec80 con 0x7fea5415eee0 2024-04-06T13:51:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.310+0000 7fea537fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fea5415e8a0 0x7fea540cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:31.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.310+0000 7fea537fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fea5415e8a0 0x7fea540cbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60142/0 (socket says 172.21.15.189:60142) 2024-04-06T13:51:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.310+0000 7fea537fe700 1 -- 172.21.15.189:0/2431533968 learned_addr learned my addr 172.21.15.189:0/2431533968 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.310+0000 7fea53fff700 1 --2- 172.21.15.189:0/2431533968 >> v2:172.21.15.192:3300/0 conn(0x7fea5415f8a0 0x7fea5406ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.310+0000 7fea52ffd700 1 --2- 172.21.15.189:0/2431533968 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea5415eee0 0x7fea5415f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.311+0000 7fea537fe700 1 -- 172.21.15.189:0/2431533968 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea5415eee0 msgr2=0x7fea5415f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.311+0000 7fea537fe700 1 --2- 172.21.15.189:0/2431533968 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea5415eee0 0x7fea5415f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.311+0000 7fea537fe700 1 -- 172.21.15.189:0/2431533968 >> v2:172.21.15.192:3300/0 conn(0x7fea5415f8a0 msgr2=0x7fea5406ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.311+0000 7fea537fe700 1 --2- 172.21.15.189:0/2431533968 >> v2:172.21.15.192:3300/0 conn(0x7fea5415f8a0 0x7fea5406ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.311+0000 7fea537fe700 1 -- 172.21.15.189:0/2431533968 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea540666f0 con 0x7fea5415e8a0 2024-04-06T13:51:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.311+0000 7fea52ffd700 1 --2- 172.21.15.189:0/2431533968 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea5415eee0 0x7fea5415f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:51:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.311+0000 7fea537fe700 1 --2- 172.21.15.189:0/2431533968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea5415e8a0 0x7fea540cbc20 secure :-1 s=READY pgs=3189 cs=0 l=1 rev1=1 crypto rx=0x7fea540c9df0 tx=0x7fea48009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a93e67d4db2ebd5d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.311+0000 7fea527fc700 1 -- 172.21.15.189:0/2431533968 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea48011830 con 0x7fea5415e8a0 2024-04-06T13:51:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.311+0000 7fea527fc700 1 -- 172.21.15.189:0/2431533968 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea48011990 con 0x7fea5415e8a0 2024-04-06T13:51:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.311+0000 7fea527fc700 1 -- 172.21.15.189:0/2431533968 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea48011d20 con 0x7fea5415e8a0 2024-04-06T13:51:31.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.311+0000 7fea5a113700 1 -- 172.21.15.189:0/2431533968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea5415e8a0 msgr2=0x7fea540cbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:31.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.311+0000 7fea5a113700 1 --2- 172.21.15.189:0/2431533968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea5415e8a0 0x7fea540cbc20 secure :-1 s=READY pgs=3189 cs=0 l=1 rev1=1 crypto rx=0x7fea540c9df0 tx=0x7fea48009bd0 comp rx=0 tx=0).stop 2024-04-06T13:51:31.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.311+0000 7fea5a113700 1 -- 172.21.15.189:0/2431533968 shutdown_connections 2024-04-06T13:51:31.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.311+0000 7fea5a113700 1 --2- 172.21.15.189:0/2431533968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea5415e8a0 0x7fea540cbc20 unknown :-1 s=CLOSED pgs=3189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:31.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.311+0000 7fea5a113700 1 --2- 172.21.15.189:0/2431533968 >> v2:172.21.15.192:3300/0 conn(0x7fea5415f8a0 0x7fea5406ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:31.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.311+0000 7fea5a113700 1 --2- 172.21.15.189:0/2431533968 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea5415eee0 0x7fea5415f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:31.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.311+0000 7fea5a113700 1 -- 172.21.15.189:0/2431533968 >> 172.21.15.189:0/2431533968 conn(0x7fea540cc290 msgr2=0x7fea540cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:31.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.312+0000 7fea5a113700 1 -- 172.21.15.189:0/2431533968 shutdown_connections 2024-04-06T13:51:31.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.312+0000 7fea5a113700 1 -- 172.21.15.189:0/2431533968 wait complete. 2024-04-06T13:51:31.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.313+0000 7fea5a113700 1 Processor -- start 2024-04-06T13:51:31.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.313+0000 7fea5a113700 1 -- start start 2024-04-06T13:51:31.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.314+0000 7fea5a113700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea5415e8a0 0x7fea540c9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:31.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.314+0000 7fea5a113700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea5415eee0 0x7fea540c59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:31.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.314+0000 7fea5a113700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea5415f8a0 0x7fea540b4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:31.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.314+0000 7fea5a113700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea540c4120 con 0x7fea5415f8a0 2024-04-06T13:51:31.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.314+0000 7fea5a113700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea540c4400 con 0x7fea5415e8a0 2024-04-06T13:51:31.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.314+0000 7fea5a113700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea540c46e0 con 0x7fea5415eee0 2024-04-06T13:51:31.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.314+0000 7fea53fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea5415f8a0 0x7fea540b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:31.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.314+0000 7fea53fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea5415f8a0 0x7fea540b4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60144/0 (socket says 172.21.15.189:60144) 2024-04-06T13:51:31.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.314+0000 7fea53fff700 1 -- 172.21.15.189:0/3737336892 learned_addr learned my addr 172.21.15.189:0/3737336892 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:31.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.314+0000 7fea537fe700 1 --2- 172.21.15.189:0/3737336892 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea5415e8a0 0x7fea540c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:31.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.314+0000 7fea52ffd700 1 --2- 172.21.15.189:0/3737336892 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea5415eee0 0x7fea540c59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:31.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.315+0000 7fea53fff700 1 -- 172.21.15.189:0/3737336892 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea5415eee0 msgr2=0x7fea540c59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:31.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.315+0000 7fea53fff700 1 --2- 172.21.15.189:0/3737336892 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea5415eee0 0x7fea540c59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:31.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.315+0000 7fea53fff700 1 -- 172.21.15.189:0/3737336892 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea5415e8a0 msgr2=0x7fea540c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:31.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.315+0000 7fea53fff700 1 --2- 172.21.15.189:0/3737336892 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea5415e8a0 0x7fea540c9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:31.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.315+0000 7fea53fff700 1 -- 172.21.15.189:0/3737336892 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea48009550 con 0x7fea5415f8a0 2024-04-06T13:51:31.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.315+0000 7fea53fff700 1 --2- 172.21.15.189:0/3737336892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea5415f8a0 0x7fea540b4240 secure :-1 s=READY pgs=3190 cs=0 l=1 rev1=1 crypto rx=0x7fea4c00efc0 tx=0x7fea4c00c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:31.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.315+0000 7fea3bfff700 1 -- 172.21.15.189:0/3737336892 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c00cad0 con 0x7fea5415f8a0 2024-04-06T13:51:31.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.315+0000 7fea3bfff700 1 -- 172.21.15.189:0/3737336892 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea4c00f040 con 0x7fea5415f8a0 2024-04-06T13:51:31.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.315+0000 7fea5a113700 1 -- 172.21.15.189:0/3737336892 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea540c49c0 con 0x7fea5415f8a0 2024-04-06T13:51:31.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.315+0000 7fea3bfff700 1 -- 172.21.15.189:0/3737336892 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c00ce70 con 0x7fea5415f8a0 2024-04-06T13:51:31.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.315+0000 7fea5a113700 1 -- 172.21.15.189:0/3737336892 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea540c9b80 con 0x7fea5415f8a0 2024-04-06T13:51:31.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.316+0000 7fea5a113700 1 -- 172.21.15.189:0/3737336892 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea54072140 con 0x7fea5415f8a0 2024-04-06T13:51:31.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.317+0000 7fea3bfff700 1 -- 172.21.15.189:0/3737336892 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 178) v1 ==== 88856+0+0 (secure 0 0 0) 0x7fea4c014480 con 0x7fea5415f8a0 2024-04-06T13:51:31.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.318+0000 7fea3bfff700 1 -- 172.21.15.189:0/3737336892 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 7005+0+0 (secure 0 0 0) 0x7fea4c08a5f0 con 0x7fea5415f8a0 2024-04-06T13:51:31.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.321+0000 7fea3bfff700 1 -- 172.21.15.189:0/3737336892 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fea4c055620 con 0x7fea5415f8a0 2024-04-06T13:51:31.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.498+0000 7fea3bfff700 1 -- 172.21.15.189:0/3737336892 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 179) v1 ==== 88983+0+0 (secure 0 0 0) 0x7fea4c0550d0 con 0x7fea5415f8a0 2024-04-06T13:51:31.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.498+0000 7fea3bfff700 1 --2- 172.21.15.189:0/3737336892 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fea40036150 0x7fea40072c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:31.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.499+0000 7fea537fe700 1 --2- 172.21.15.189:0/3737336892 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fea40036150 0x7fea40072c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:31.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.499+0000 7fea537fe700 1 --2- 172.21.15.189:0/3737336892 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fea40036150 0x7fea40072c70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fea48009c10 tx=0x7fea48008030 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:31.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:31.569+0000 7fea5a113700 1 -- 172.21.15.189:0/3737336892 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fea540c4ee0 con 0x7fea5415f8a0 2024-04-06T13:51:32.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.174+0000 7fea3bfff700 1 -- 172.21.15.189:0/3737336892 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v141) v1 ==== 104+0+0 (secure 0 0 0) 0x7fea540c4ee0 con 0x7fea5415f8a0 2024-04-06T13:51:32.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.177+0000 7fea5a113700 1 -- 172.21.15.189:0/3737336892 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fea40036150 msgr2=0x7fea40072c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:32.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.177+0000 7fea5a113700 1 --2- 172.21.15.189:0/3737336892 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fea40036150 0x7fea40072c70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fea48009c10 tx=0x7fea48008030 comp rx=0 tx=0).stop 2024-04-06T13:51:32.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.177+0000 7fea5a113700 1 -- 172.21.15.189:0/3737336892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea5415f8a0 msgr2=0x7fea540b4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:32.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.177+0000 7fea5a113700 1 --2- 172.21.15.189:0/3737336892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea5415f8a0 0x7fea540b4240 secure :-1 s=READY pgs=3190 cs=0 l=1 rev1=1 crypto rx=0x7fea4c00efc0 tx=0x7fea4c00c0c0 comp rx=0 tx=0).stop 2024-04-06T13:51:32.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.178+0000 7fea5a113700 1 -- 172.21.15.189:0/3737336892 shutdown_connections 2024-04-06T13:51:32.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.178+0000 7fea5a113700 1 --2- 172.21.15.189:0/3737336892 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fea40036150 0x7fea40072c70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:32.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.178+0000 7fea5a113700 1 --2- 172.21.15.189:0/3737336892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea5415f8a0 0x7fea540b4240 unknown :-1 s=CLOSED pgs=3190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:32.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.178+0000 7fea5a113700 1 --2- 172.21.15.189:0/3737336892 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea5415e8a0 0x7fea540c9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:32.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.178+0000 7fea5a113700 1 --2- 172.21.15.189:0/3737336892 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea5415eee0 0x7fea540c59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:32.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.178+0000 7fea5a113700 1 -- 172.21.15.189:0/3737336892 >> 172.21.15.189:0/3737336892 conn(0x7fea540cc290 msgr2=0x7fea5415d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:32.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.178+0000 7fea5a113700 1 -- 172.21.15.189:0/3737336892 shutdown_connections 2024-04-06T13:51:32.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.178+0000 7fea5a113700 1 -- 172.21.15.189:0/3737336892 wait complete. 2024-04-06T13:51:32.194 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-06T13:51:32.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.324+0000 7f6120512700 1 Processor -- start 2024-04-06T13:51:32.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.324+0000 7f6120512700 1 -- start start 2024-04-06T13:51:32.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.325+0000 7f6120512700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6118144f40 0x7f611806a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:32.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.325+0000 7f6120512700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f611815c150 0x7f611806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:32.328 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.325+0000 7f6120512700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f611815c740 con 0x7f611815fcd0 2024-04-06T13:51:32.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.325+0000 7f6120512700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61180cfcb0 con 0x7f6118144f40 2024-04-06T13:51:32.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.325+0000 7f6120512700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f611815ca20 con 0x7f611815c150 2024-04-06T13:51:32.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.326+0000 7f611e2ae700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6118144f40 0x7f611806a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:32.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.326+0000 7f611e2ae700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6118144f40 0x7f611806a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60154/0 (socket says 172.21.15.189:60154) 2024-04-06T13:51:32.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.326+0000 7f611e2ae700 1 -- 172.21.15.189:0/3297374901 learned_addr learned my addr 172.21.15.189:0/3297374901 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:32.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.326+0000 7f611e2ae700 1 -- 172.21.15.189:0/3297374901 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f611815c150 msgr2=0x7f611806ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:32.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.326+0000 7f611eaaf700 1 --2- 172.21.15.189:0/3297374901 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f611815c150 0x7f611806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:32.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.326+0000 7f611e2ae700 1 --2- 172.21.15.189:0/3297374901 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f611815c150 0x7f611806ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:32.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.326+0000 7f611e2ae700 1 -- 172.21.15.189:0/3297374901 >> v1:172.21.15.192:6789/0 conn(0x7f611815fcd0 legacy=0x7f611815ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:32.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.326+0000 7f611e2ae700 1 -- 172.21.15.189:0/3297374901 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611815f1f0 con 0x7f6118144f40 2024-04-06T13:51:32.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.326+0000 7f611eaaf700 1 --2- 172.21.15.189:0/3297374901 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f611815c150 0x7f611806ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:51:32.329 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.326+0000 7f611e2ae700 1 --2- 172.21.15.189:0/3297374901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6118144f40 0x7f611806a730 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 crypto rx=0x7f61100098f0 tx=0x7f6110009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d953a6fca47090 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:32.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.326+0000 7f611d2ac700 1 -- 172.21.15.189:0/3297374901 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6110004230 con 0x7f6118144f40 2024-04-06T13:51:32.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.326+0000 7f611d2ac700 1 -- 172.21.15.189:0/3297374901 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6110004390 con 0x7f6118144f40 2024-04-06T13:51:32.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.326+0000 7f611d2ac700 1 -- 172.21.15.189:0/3297374901 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61100036a0 con 0x7f6118144f40 2024-04-06T13:51:32.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.327+0000 7f6120512700 1 -- 172.21.15.189:0/3297374901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6118144f40 msgr2=0x7f611806a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:32.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.327+0000 7f6120512700 1 --2- 172.21.15.189:0/3297374901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6118144f40 0x7f611806a730 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 crypto rx=0x7f61100098f0 tx=0x7f6110009c00 comp rx=0 tx=0).stop 2024-04-06T13:51:32.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.327+0000 7f6120512700 1 -- 172.21.15.189:0/3297374901 shutdown_connections 2024-04-06T13:51:32.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.327+0000 7f6120512700 1 --2- 172.21.15.189:0/3297374901 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f611815c150 0x7f611806ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:32.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.327+0000 7f6120512700 1 --2- 172.21.15.189:0/3297374901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6118144f40 0x7f611806a730 unknown :-1 s=CLOSED pgs=3191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:32.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.327+0000 7f6120512700 1 -- 172.21.15.189:0/3297374901 >> 172.21.15.189:0/3297374901 conn(0x7f61180d00b0 msgr2=0x7f61180d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:32.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.327+0000 7f6120512700 1 -- 172.21.15.189:0/3297374901 shutdown_connections 2024-04-06T13:51:32.330 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.327+0000 7f6120512700 1 -- 172.21.15.189:0/3297374901 wait complete. 2024-04-06T13:51:32.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.328+0000 7f6120512700 1 Processor -- start 2024-04-06T13:51:32.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.328+0000 7f6120512700 1 -- start start 2024-04-06T13:51:32.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f6120512700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6118144f40 0x7f61180cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:32.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f6120512700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f611815c150 0x7f6118143e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:32.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f6120512700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f611815fcd0 0x7f6118133a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:32.333 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f6120512700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61180c7520 con 0x7f611815fcd0 2024-04-06T13:51:32.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f6120512700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61180c6910 con 0x7f611815c150 2024-04-06T13:51:32.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f6120512700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61180c6bf0 con 0x7f6118144f40 2024-04-06T13:51:32.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f611eaaf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f611815fcd0 0x7f6118133a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:32.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f611eaaf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f611815fcd0 0x7f6118133a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60168/0 (socket says 172.21.15.189:60168) 2024-04-06T13:51:32.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f611eaaf700 1 -- 172.21.15.189:0/4057554995 learned_addr learned my addr 172.21.15.189:0/4057554995 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:32.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f611e2ae700 1 --2- 172.21.15.189:0/4057554995 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6118144f40 0x7f61180cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:32.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f611daad700 1 --2- 172.21.15.189:0/4057554995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f611815c150 0x7f6118143e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:32.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f611eaaf700 1 -- 172.21.15.189:0/4057554995 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6118144f40 msgr2=0x7f61180cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:32.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f611eaaf700 1 --2- 172.21.15.189:0/4057554995 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6118144f40 0x7f61180cb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:32.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f611eaaf700 1 -- 172.21.15.189:0/4057554995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f611815c150 msgr2=0x7f6118143e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:32.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f611eaaf700 1 --2- 172.21.15.189:0/4057554995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f611815c150 0x7f6118143e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:32.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f611eaaf700 1 -- 172.21.15.189:0/4057554995 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6110009550 con 0x7f611815fcd0 2024-04-06T13:51:32.334 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f611eaaf700 1 --2- 172.21.15.189:0/4057554995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f611815fcd0 0x7f6118133a20 secure :-1 s=READY pgs=3192 cs=0 l=1 rev1=1 crypto rx=0x7f611400cef0 tx=0x7f611400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:32.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f610effd700 1 -- 172.21.15.189:0/4057554995 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611400acf0 con 0x7f611815fcd0 2024-04-06T13:51:32.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f610effd700 1 -- 172.21.15.189:0/4057554995 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f611400ae50 con 0x7f611815fcd0 2024-04-06T13:51:32.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f610effd700 1 -- 172.21.15.189:0/4057554995 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6114015680 con 0x7f611815fcd0 2024-04-06T13:51:32.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.329+0000 7f6120512700 1 -- 172.21.15.189:0/4057554995 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61180c6ed0 con 0x7f611815fcd0 2024-04-06T13:51:32.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.330+0000 7f6120512700 1 -- 172.21.15.189:0/4057554995 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61180cbe00 con 0x7f611815fcd0 2024-04-06T13:51:32.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.330+0000 7f6120512700 1 -- 172.21.15.189:0/4057554995 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61180c76d0 con 0x7f611815fcd0 2024-04-06T13:51:32.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.331+0000 7f610effd700 1 -- 172.21.15.189:0/4057554995 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 179) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f611400d430 con 0x7f611815fcd0 2024-04-06T13:51:32.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.331+0000 7f610effd700 1 --2- 172.21.15.189:0/4057554995 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f60fc06aae0 0x7f60fc06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:32.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.332+0000 7f610effd700 1 -- 172.21.15.189:0/4057554995 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 7005+0+0 (secure 0 0 0) 0x7f6114089b20 con 0x7f611815fcd0 2024-04-06T13:51:32.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.332+0000 7f611e2ae700 1 --2- 172.21.15.189:0/4057554995 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f60fc06aae0 0x7f60fc06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:32.335 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.332+0000 7f611e2ae700 1 --2- 172.21.15.189:0/4057554995 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f60fc06aae0 0x7f60fc06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6110009130 tx=0x7f6110008030 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:32.336 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.336+0000 7f610effd700 1 -- 172.21.15.189:0/4057554995 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f6114054bf0 con 0x7f611815fcd0 2024-04-06T13:51:32.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:32.533+0000 7f6120512700 1 -- 172.21.15.189:0/4057554995 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6118067bb0 con 0x7f611815fcd0 2024-04-06T13:51:33.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.505+0000 7f610effd700 1 -- 172.21.15.189:0/4057554995 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 180) v1 ==== 89191+0+0 (secure 0 0 0) 0x7f6114054fc0 con 0x7f611815fcd0 2024-04-06T13:51:33.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.508+0000 7f610effd700 1 -- 172.21.15.189:0/4057554995 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v289) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6114058680 con 0x7f611815fcd0 2024-04-06T13:51:33.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.511+0000 7f6120512700 1 -- 172.21.15.189:0/4057554995 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f60fc06aae0 msgr2=0x7f60fc06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:33.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.511+0000 7f6120512700 1 --2- 172.21.15.189:0/4057554995 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f60fc06aae0 0x7f60fc06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6110009130 tx=0x7f6110008030 comp rx=0 tx=0).stop 2024-04-06T13:51:33.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.511+0000 7f6120512700 1 -- 172.21.15.189:0/4057554995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f611815fcd0 msgr2=0x7f6118133a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:33.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.511+0000 7f6120512700 1 --2- 172.21.15.189:0/4057554995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f611815fcd0 0x7f6118133a20 secure :-1 s=READY pgs=3192 cs=0 l=1 rev1=1 crypto rx=0x7f611400cef0 tx=0x7f611400cf20 comp rx=0 tx=0).stop 2024-04-06T13:51:33.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.511+0000 7f6120512700 1 -- 172.21.15.189:0/4057554995 shutdown_connections 2024-04-06T13:51:33.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.511+0000 7f6120512700 1 --2- 172.21.15.189:0/4057554995 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f60fc06aae0 0x7f60fc06cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:33.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.511+0000 7f6120512700 1 --2- 172.21.15.189:0/4057554995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f611815fcd0 0x7f6118133a20 unknown :-1 s=CLOSED pgs=3192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:33.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.511+0000 7f6120512700 1 --2- 172.21.15.189:0/4057554995 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6118144f40 0x7f61180cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:33.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.511+0000 7f6120512700 1 --2- 172.21.15.189:0/4057554995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f611815c150 0x7f6118143e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:33.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.511+0000 7f6120512700 1 -- 172.21.15.189:0/4057554995 >> 172.21.15.189:0/4057554995 conn(0x7f61180d00b0 msgr2=0x7f611806f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:33.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.511+0000 7f6120512700 1 -- 172.21.15.189:0/4057554995 shutdown_connections 2024-04-06T13:51:33.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.511+0000 7f6120512700 1 -- 172.21.15.189:0/4057554995 wait complete. 2024-04-06T13:51:33.513 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' removed 2024-04-06T13:51:33.527 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-06T13:51:33.560 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:51:33.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:51:33.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.662+0000 7f752626b700 1 Processor -- start 2024-04-06T13:51:33.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.662+0000 7f752626b700 1 -- start start 2024-04-06T13:51:33.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.663+0000 7f752626b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f75200cb870 0x7f752015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:33.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.663+0000 7f752626b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75200cfa00 0x7f75200cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:33.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.663+0000 7f752626b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75200cbd80 con 0x7f75200d11b0 2024-04-06T13:51:33.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.663+0000 7f752626b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75200ccdc0 con 0x7f75200cb870 2024-04-06T13:51:33.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.663+0000 7f752626b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75200d0b50 con 0x7f75200cfa00 2024-04-06T13:51:33.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.663+0000 7f7517fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f75200cb870 0x7f752015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:33.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.663+0000 7f751ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f75200d11b0 0x7f75200d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54026/0 (socket says 172.21.15.189:54026) 2024-04-06T13:51:33.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.663+0000 7f751ffff700 1 -- 172.21.15.189:0/3505238147 learned_addr learned my addr 172.21.15.189:0/3505238147 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:33.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.663+0000 7f7524808700 1 --2- 172.21.15.189:0/3505238147 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75200cfa00 0x7f75200cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:33.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.664+0000 7f7517fff700 1 -- 172.21.15.189:0/3505238147 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75200cfa00 msgr2=0x7f75200cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:33.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.664+0000 7f7517fff700 1 --2- 172.21.15.189:0/3505238147 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75200cfa00 0x7f75200cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:33.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.664+0000 7f7517fff700 1 -- 172.21.15.189:0/3505238147 >> v1:172.21.15.189:6789/0 conn(0x7f75200d11b0 legacy=0x7f75200d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:33.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.664+0000 7f7517fff700 1 -- 172.21.15.189:0/3505238147 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75200cef20 con 0x7f75200cb870 2024-04-06T13:51:33.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.664+0000 7f7517fff700 1 --2- 172.21.15.189:0/3505238147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75200cb870 0x7f752015c280 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 crypto rx=0x7f7508009840 tx=0x7f7508009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd48a61ee442c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:33.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.664+0000 7f751f7fe700 1 -- 172.21.15.189:0/3505238147 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7508011830 con 0x7f75200cb870 2024-04-06T13:51:33.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.664+0000 7f751f7fe700 1 -- 172.21.15.189:0/3505238147 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7508011990 con 0x7f75200cb870 2024-04-06T13:51:33.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.664+0000 7f751f7fe700 1 -- 172.21.15.189:0/3505238147 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7508011d20 con 0x7f75200cb870 2024-04-06T13:51:33.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.665+0000 7f752626b700 1 -- 172.21.15.189:0/3505238147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75200cb870 msgr2=0x7f752015c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:33.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.665+0000 7f752626b700 1 --2- 172.21.15.189:0/3505238147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75200cb870 0x7f752015c280 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 crypto rx=0x7f7508009840 tx=0x7f7508009b50 comp rx=0 tx=0).stop 2024-04-06T13:51:33.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.665+0000 7f752626b700 1 -- 172.21.15.189:0/3505238147 shutdown_connections 2024-04-06T13:51:33.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.665+0000 7f752626b700 1 --2- 172.21.15.189:0/3505238147 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75200cfa00 0x7f75200cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:33.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.665+0000 7f752626b700 1 --2- 172.21.15.189:0/3505238147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75200cb870 0x7f752015c280 unknown :-1 s=CLOSED pgs=3196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:33.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.665+0000 7f752626b700 1 -- 172.21.15.189:0/3505238147 >> 172.21.15.189:0/3505238147 conn(0x7f75200b2db0 msgr2=0x7f75200b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:33.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.665+0000 7f752626b700 1 -- 172.21.15.189:0/3505238147 shutdown_connections 2024-04-06T13:51:33.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.665+0000 7f752626b700 1 -- 172.21.15.189:0/3505238147 wait complete. 2024-04-06T13:51:33.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.666+0000 7f752626b700 1 Processor -- start 2024-04-06T13:51:33.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.666+0000 7f752626b700 1 -- start start 2024-04-06T13:51:33.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.667+0000 7f752626b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75200cb870 0x7f752015f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:33.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.667+0000 7f752626b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75200cfa00 0x7f752006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:33.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.667+0000 7f752626b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75200d11b0 0x7f75201e6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:33.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.667+0000 7f752626b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752015f9f0 con 0x7f75200cb870 2024-04-06T13:51:33.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.667+0000 7f752626b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75200c9cd0 con 0x7f75200d11b0 2024-04-06T13:51:33.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.667+0000 7f752626b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75200c9fb0 con 0x7f75200cfa00 2024-04-06T13:51:33.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.667+0000 7f751ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75200cb870 0x7f752015f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:33.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.667+0000 7f751ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75200cb870 0x7f752015f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60202/0 (socket says 172.21.15.189:60202) 2024-04-06T13:51:33.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.667+0000 7f751ffff700 1 -- 172.21.15.189:0/739960784 learned_addr learned my addr 172.21.15.189:0/739960784 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:33.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.667+0000 7f7517fff700 1 --2- 172.21.15.189:0/739960784 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75200cfa00 0x7f752006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:33.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.667+0000 7f7524808700 1 --2- 172.21.15.189:0/739960784 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75200d11b0 0x7f75201e6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:33.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.667+0000 7f751ffff700 1 -- 172.21.15.189:0/739960784 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75200cfa00 msgr2=0x7f752006c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:33.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.667+0000 7f751ffff700 1 --2- 172.21.15.189:0/739960784 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75200cfa00 0x7f752006c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:33.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.667+0000 7f751ffff700 1 -- 172.21.15.189:0/739960784 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75200d11b0 msgr2=0x7f75201e6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:33.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.667+0000 7f751ffff700 1 --2- 172.21.15.189:0/739960784 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75200d11b0 0x7f75201e6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:33.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.667+0000 7f751ffff700 1 -- 172.21.15.189:0/739960784 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75080094a0 con 0x7f75200cb870 2024-04-06T13:51:33.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.667+0000 7f751ffff700 1 --2- 172.21.15.189:0/739960784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75200cb870 0x7f752015f4b0 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 crypto rx=0x7f751000c480 tx=0x7f751000c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:33.671 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.668+0000 7f751d7fa700 1 -- 172.21.15.189:0/739960784 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7510005d70 con 0x7f75200cb870 2024-04-06T13:51:33.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.668+0000 7f751d7fa700 1 -- 172.21.15.189:0/739960784 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f751000ce90 con 0x7f75200cb870 2024-04-06T13:51:33.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.668+0000 7f752626b700 1 -- 172.21.15.189:0/739960784 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75200ca290 con 0x7f75200cb870 2024-04-06T13:51:33.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.668+0000 7f751d7fa700 1 -- 172.21.15.189:0/739960784 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75100072d0 con 0x7f75200cb870 2024-04-06T13:51:33.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.668+0000 7f752626b700 1 -- 172.21.15.189:0/739960784 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f752015d370 con 0x7f75200cb870 2024-04-06T13:51:33.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.669+0000 7f752626b700 1 -- 172.21.15.189:0/739960784 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f752015d1b0 con 0x7f75200cb870 2024-04-06T13:51:33.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.669+0000 7f751d7fa700 1 -- 172.21.15.189:0/739960784 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 180) v1 ==== 89191+0+0 (secure 0 0 0) 0x7f7510007430 con 0x7f75200cb870 2024-04-06T13:51:33.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.669+0000 7f751d7fa700 1 --2- 172.21.15.189:0/739960784 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f750006ad20 0x7f750006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:33.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.669+0000 7f751d7fa700 1 -- 172.21.15.189:0/739960784 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(289..289 src has 1..289) v4 ==== 6567+0+0 (secure 0 0 0) 0x7f7510088830 con 0x7f75200cb870 2024-04-06T13:51:33.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.670+0000 7f7517fff700 1 --2- 172.21.15.189:0/739960784 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f750006ad20 0x7f750006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:33.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.670+0000 7f7517fff700 1 --2- 172.21.15.189:0/739960784 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f750006ad20 0x7f750006d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f75200cbcb0 tx=0x7f7508008030 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:33.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.674+0000 7f751d7fa700 1 -- 172.21.15.189:0/739960784 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7510053800 con 0x7f75200cb870 2024-04-06T13:51:33.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:33.870+0000 7f752626b700 1 -- 172.21.15.189:0/739960784 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f75200670c0 con 0x7f75200cb870 2024-04-06T13:51:34.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.677+0000 7f751d7fa700 1 -- 172.21.15.189:0/739960784 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7510053bd0 con 0x7f75200cb870 2024-04-06T13:51:34.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.677+0000 7f751d7fa700 1 -- 172.21.15.189:0/739960784 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v290) v1 ==== 169+0+0 (secure 0 0 0) 0x7f751004fe40 con 0x7f75200cb870 2024-04-06T13:51:34.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.681+0000 7f752626b700 1 -- 172.21.15.189:0/739960784 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f750006ad20 msgr2=0x7f750006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:34.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.681+0000 7f752626b700 1 --2- 172.21.15.189:0/739960784 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f750006ad20 0x7f750006d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f75200cbcb0 tx=0x7f7508008030 comp rx=0 tx=0).stop 2024-04-06T13:51:34.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.681+0000 7f752626b700 1 -- 172.21.15.189:0/739960784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75200cb870 msgr2=0x7f752015f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:34.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.681+0000 7f752626b700 1 --2- 172.21.15.189:0/739960784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75200cb870 0x7f752015f4b0 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 crypto rx=0x7f751000c480 tx=0x7f751000c790 comp rx=0 tx=0).stop 2024-04-06T13:51:34.682 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.681+0000 7f752626b700 1 -- 172.21.15.189:0/739960784 shutdown_connections 2024-04-06T13:51:34.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.681+0000 7f752626b700 1 --2- 172.21.15.189:0/739960784 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f750006ad20 0x7f750006d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:34.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.681+0000 7f752626b700 1 --2- 172.21.15.189:0/739960784 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75200d11b0 0x7f75201e6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:34.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.681+0000 7f752626b700 1 --2- 172.21.15.189:0/739960784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75200cb870 0x7f752015f4b0 unknown :-1 s=CLOSED pgs=3197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:34.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.681+0000 7f752626b700 1 --2- 172.21.15.189:0/739960784 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f75200cfa00 0x7f752006c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:34.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.681+0000 7f752626b700 1 -- 172.21.15.189:0/739960784 >> 172.21.15.189:0/739960784 conn(0x7f75200b2db0 msgr2=0x7f75200b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:34.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.681+0000 7f752626b700 1 -- 172.21.15.189:0/739960784 shutdown_connections 2024-04-06T13:51:34.683 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.681+0000 7f752626b700 1 -- 172.21.15.189:0/739960784 wait complete. 2024-04-06T13:51:34.684 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' removed 2024-04-06T13:51:34.696 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-06T13:51:34.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.861+0000 7f71b6911700 1 Processor -- start 2024-04-06T13:51:34.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.861+0000 7f71b6911700 1 -- start start 2024-04-06T13:51:34.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.862+0000 7f71b6911700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71b00b4660 0x7f71b00b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:34.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.862+0000 7f71b6911700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f71b00b2fc0 0x7f71b00b3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:34.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.862+0000 7f71b6911700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b00cdbe0 con 0x7f71b015a550 2024-04-06T13:51:34.864 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.862+0000 7f71b6911700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71b00b5000 con 0x7f71b00b2fc0 2024-04-06T13:51:34.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.862+0000 7f71b6911700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b015a940 con 0x7f71b00b4660 2024-04-06T13:51:34.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.862+0000 7f71b4eae700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f71b00b2fc0 0x7f71b00b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:34.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.862+0000 7f71b4eae700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f71b00b2fc0 0x7f71b00b3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60208/0 (socket says 172.21.15.189:60208) 2024-04-06T13:51:34.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.862+0000 7f71b4eae700 1 -- 172.21.15.189:0/1224781610 learned_addr learned my addr 172.21.15.189:0/1224781610 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:34.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.862+0000 7f71af7fe700 1 --2- 172.21.15.189:0/1224781610 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71b00b4660 0x7f71b00b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:34.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.862+0000 7f71b4eae700 1 -- 172.21.15.189:0/1224781610 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71b00b4660 msgr2=0x7f71b00b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:34.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.862+0000 7f71b4eae700 1 --2- 172.21.15.189:0/1224781610 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71b00b4660 0x7f71b00b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:34.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.862+0000 7f71b4eae700 1 -- 172.21.15.189:0/1224781610 >> v1:172.21.15.192:6789/0 conn(0x7f71b015a550 legacy=0x7f71b00cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:34.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.862+0000 7f71b4eae700 1 -- 172.21.15.189:0/1224781610 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b015ac00 con 0x7f71b00b2fc0 2024-04-06T13:51:34.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.862+0000 7f71b4eae700 1 --2- 172.21.15.189:0/1224781610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71b00b2fc0 0x7f71b00b3430 secure :-1 s=READY pgs=3198 cs=0 l=1 rev1=1 crypto rx=0x7f71a8009e50 tx=0x7f71a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc550af9f55de9d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:34.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.862+0000 7f71aeffd700 1 -- 172.21.15.189:0/1224781610 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a800e070 con 0x7f71b00b2fc0 2024-04-06T13:51:34.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.863+0000 7f71aeffd700 1 -- 172.21.15.189:0/1224781610 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a800b9c0 con 0x7f71b00b2fc0 2024-04-06T13:51:34.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.863+0000 7f71aeffd700 1 -- 172.21.15.189:0/1224781610 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a800bd70 con 0x7f71b00b2fc0 2024-04-06T13:51:34.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.863+0000 7f71b6911700 1 -- 172.21.15.189:0/1224781610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71b00b2fc0 msgr2=0x7f71b00b3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:34.865 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.863+0000 7f71b6911700 1 --2- 172.21.15.189:0/1224781610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71b00b2fc0 0x7f71b00b3430 secure :-1 s=READY pgs=3198 cs=0 l=1 rev1=1 crypto rx=0x7f71a8009e50 tx=0x7f71a800b330 comp rx=0 tx=0).stop 2024-04-06T13:51:34.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.863+0000 7f71b6911700 1 -- 172.21.15.189:0/1224781610 shutdown_connections 2024-04-06T13:51:34.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.863+0000 7f71b6911700 1 --2- 172.21.15.189:0/1224781610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71b00b2fc0 0x7f71b00b3430 unknown :-1 s=CLOSED pgs=3198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:34.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.863+0000 7f71b6911700 1 --2- 172.21.15.189:0/1224781610 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71b00b4660 0x7f71b00b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:34.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.863+0000 7f71b6911700 1 -- 172.21.15.189:0/1224781610 >> 172.21.15.189:0/1224781610 conn(0x7f71b00d00b0 msgr2=0x7f71b00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:34.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.863+0000 7f71b6911700 1 -- 172.21.15.189:0/1224781610 shutdown_connections 2024-04-06T13:51:34.866 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.863+0000 7f71b6911700 1 -- 172.21.15.189:0/1224781610 wait complete. 2024-04-06T13:51:34.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.864+0000 7f71b6911700 1 Processor -- start 2024-04-06T13:51:34.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.864+0000 7f71b6911700 1 -- start start 2024-04-06T13:51:34.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.865+0000 7f71b6911700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71b00b2fc0 0x7f71b00afc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:34.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.865+0000 7f71b6911700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71b00b4660 0x7f71b015ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:34.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.865+0000 7f71b6911700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71b00b0150 0x7f71b015d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:34.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.865+0000 7f71b6911700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b015d880 con 0x7f71b00b2fc0 2024-04-06T13:51:34.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.865+0000 7f71b6911700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b01544f0 con 0x7f71b00b4660 2024-04-06T13:51:34.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.865+0000 7f71b6911700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b015c970 con 0x7f71b00b0150 2024-04-06T13:51:34.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.865+0000 7f71affff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71b00b2fc0 0x7f71b00afc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:34.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.865+0000 7f71affff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71b00b2fc0 0x7f71b00afc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60218/0 (socket says 172.21.15.189:60218) 2024-04-06T13:51:34.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.865+0000 7f71affff700 1 -- 172.21.15.189:0/1093003900 learned_addr learned my addr 172.21.15.189:0/1093003900 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:34.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.866+0000 7f71af7fe700 1 --2- 172.21.15.189:0/1093003900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71b00b4660 0x7f71b015ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:34.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.866+0000 7f71b4eae700 1 --2- 172.21.15.189:0/1093003900 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71b00b0150 0x7f71b015d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:34.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.866+0000 7f71affff700 1 -- 172.21.15.189:0/1093003900 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71b00b0150 msgr2=0x7f71b015d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:34.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.866+0000 7f71affff700 1 --2- 172.21.15.189:0/1093003900 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71b00b0150 0x7f71b015d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:34.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.866+0000 7f71affff700 1 -- 172.21.15.189:0/1093003900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71b00b4660 msgr2=0x7f71b015ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:34.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.866+0000 7f71affff700 1 --2- 172.21.15.189:0/1093003900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71b00b4660 0x7f71b015ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:34.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.866+0000 7f71affff700 1 -- 172.21.15.189:0/1093003900 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a8009b30 con 0x7f71b00b2fc0 2024-04-06T13:51:34.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.866+0000 7f71affff700 1 --2- 172.21.15.189:0/1093003900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71b00b2fc0 0x7f71b00afc10 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 crypto rx=0x7f71a400c4e0 tx=0x7f71a400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:34.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.866+0000 7f71acff9700 1 -- 172.21.15.189:0/1093003900 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a40052d0 con 0x7f71b00b2fc0 2024-04-06T13:51:34.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.866+0000 7f71acff9700 1 -- 172.21.15.189:0/1093003900 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a4005430 con 0x7f71b00b2fc0 2024-04-06T13:51:34.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.866+0000 7f71b6911700 1 -- 172.21.15.189:0/1093003900 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b00ae0b0 con 0x7f71b00b2fc0 2024-04-06T13:51:34.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.866+0000 7f71acff9700 1 -- 172.21.15.189:0/1093003900 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a40038c0 con 0x7f71b00b2fc0 2024-04-06T13:51:34.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.866+0000 7f71b6911700 1 -- 172.21.15.189:0/1093003900 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b015c320 con 0x7f71b00b2fc0 2024-04-06T13:51:34.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.867+0000 7f71b6911700 1 -- 172.21.15.189:0/1093003900 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b0067bb0 con 0x7f71b00b2fc0 2024-04-06T13:51:34.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.867+0000 7f71acff9700 1 -- 172.21.15.189:0/1093003900 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f71a4003a20 con 0x7f71b00b2fc0 2024-04-06T13:51:34.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.868+0000 7f71acff9700 1 --2- 172.21.15.189:0/1093003900 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f719c06adb0 0x7f719c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:34.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.868+0000 7f71acff9700 1 -- 172.21.15.189:0/1093003900 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f71a4088ab0 con 0x7f71b00b2fc0 2024-04-06T13:51:34.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.868+0000 7f71af7fe700 1 --2- 172.21.15.189:0/1093003900 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f719c06adb0 0x7f719c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:34.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.868+0000 7f71af7fe700 1 --2- 172.21.15.189:0/1093003900 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f719c06adb0 0x7f719c06d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f71b015bbe0 tx=0x7f71a0008040 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:34.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:34.872+0000 7f71acff9700 1 -- 172.21.15.189:0/1093003900 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f71a4053a20 con 0x7f71b00b2fc0 2024-04-06T13:51:35.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.068+0000 7f71b6911700 1 -- 172.21.15.189:0/1093003900 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f71b0074990 con 0x7f71b00b2fc0 2024-04-06T13:51:35.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.074+0000 7f71acff9700 1 -- 172.21.15.189:0/1093003900 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v97) v1 ==== 104+0+0 (secure 0 0 0) 0x7f71a4057040 con 0x7f71b00b2fc0 2024-04-06T13:51:35.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.076+0000 7f71b6911700 1 -- 172.21.15.189:0/1093003900 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f719c06adb0 msgr2=0x7f719c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:35.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.076+0000 7f71b6911700 1 --2- 172.21.15.189:0/1093003900 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f719c06adb0 0x7f719c06d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f71b015bbe0 tx=0x7f71a0008040 comp rx=0 tx=0).stop 2024-04-06T13:51:35.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.076+0000 7f71b6911700 1 -- 172.21.15.189:0/1093003900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71b00b2fc0 msgr2=0x7f71b00afc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:35.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.076+0000 7f71b6911700 1 --2- 172.21.15.189:0/1093003900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71b00b2fc0 0x7f71b00afc10 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 crypto rx=0x7f71a400c4e0 tx=0x7f71a400c8a0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.076+0000 7f71b6911700 1 -- 172.21.15.189:0/1093003900 shutdown_connections 2024-04-06T13:51:35.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.076+0000 7f71b6911700 1 --2- 172.21.15.189:0/1093003900 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f719c06adb0 0x7f719c06d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.076+0000 7f71b6911700 1 --2- 172.21.15.189:0/1093003900 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71b00b0150 0x7f71b015d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.076+0000 7f71b6911700 1 --2- 172.21.15.189:0/1093003900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71b00b2fc0 0x7f71b00afc10 unknown :-1 s=CLOSED pgs=3199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.076+0000 7f71b6911700 1 --2- 172.21.15.189:0/1093003900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71b00b4660 0x7f71b015ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.076+0000 7f71b6911700 1 -- 172.21.15.189:0/1093003900 >> 172.21.15.189:0/1093003900 conn(0x7f71b00d00b0 msgr2=0x7f71b006b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:35.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.076+0000 7f71b6911700 1 -- 172.21.15.189:0/1093003900 shutdown_connections 2024-04-06T13:51:35.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.076+0000 7f71b6911700 1 -- 172.21.15.189:0/1093003900 wait complete. 2024-04-06T13:51:35.090 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-06T13:51:35.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.213+0000 7f2e9aac1700 1 Processor -- start 2024-04-06T13:51:35.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.213+0000 7f2e9aac1700 1 -- start start 2024-04-06T13:51:35.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.213+0000 7f2e9aac1700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2e9415d7b0 0x7f2e940b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:35.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.213+0000 7f2e9aac1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e9415e500 0x7f2e9406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:35.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.213+0000 7f2e9aac1700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e940b3990 con 0x7f2e9406a730 2024-04-06T13:51:35.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.213+0000 7f2e9aac1700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e9406b0f0 con 0x7f2e9415d7b0 2024-04-06T13:51:35.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.213+0000 7f2e9aac1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e940b3060 con 0x7f2e9415e500 2024-04-06T13:51:35.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.214+0000 7f2e9885d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2e9415d7b0 0x7f2e940b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:35.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.214+0000 7f2e9905e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e9415e500 0x7f2e9406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:35.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.214+0000 7f2e93fff700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f2e9406a730 0x7f2e9415ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:48676/0 (socket says 172.21.15.189:48676) 2024-04-06T13:51:35.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.214+0000 7f2e93fff700 1 -- 172.21.15.189:0/4006046252 learned_addr learned my addr 172.21.15.189:0/4006046252 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:35.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.214+0000 7f2e9905e700 1 -- 172.21.15.189:0/4006046252 >> v2:172.21.15.192:3300/0 conn(0x7f2e9415d7b0 msgr2=0x7f2e940b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:35.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.214+0000 7f2e9905e700 1 --2- 172.21.15.189:0/4006046252 >> v2:172.21.15.192:3300/0 conn(0x7f2e9415d7b0 0x7f2e940b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.214+0000 7f2e9905e700 1 -- 172.21.15.189:0/4006046252 >> v1:172.21.15.192:6789/0 conn(0x7f2e9406a730 legacy=0x7f2e9415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:35.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.214+0000 7f2e9905e700 1 -- 172.21.15.189:0/4006046252 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e940b3340 con 0x7f2e9415e500 2024-04-06T13:51:35.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.214+0000 7f2e9885d700 1 --2- 172.21.15.189:0/4006046252 >> v2:172.21.15.192:3300/0 conn(0x7f2e9415d7b0 0x7f2e940b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:51:35.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.214+0000 7f2e9905e700 1 --2- 172.21.15.189:0/4006046252 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e9415e500 0x7f2e9406abb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c009d90 tx=0x7f2e8c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2998f14fc9b21d72 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:35.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.214+0000 7f2e937fe700 1 -- 172.21.15.189:0/4006046252 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c00e070 con 0x7f2e9415e500 2024-04-06T13:51:35.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.214+0000 7f2e937fe700 1 -- 172.21.15.189:0/4006046252 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8c00bb20 con 0x7f2e9415e500 2024-04-06T13:51:35.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.215+0000 7f2e937fe700 1 -- 172.21.15.189:0/4006046252 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c012430 con 0x7f2e9415e500 2024-04-06T13:51:35.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.215+0000 7f2e9aac1700 1 -- 172.21.15.189:0/4006046252 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e9415e500 msgr2=0x7f2e9406abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:35.217 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.215+0000 7f2e9aac1700 1 --2- 172.21.15.189:0/4006046252 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e9415e500 0x7f2e9406abb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c009d90 tx=0x7f2e8c00b330 comp rx=0 tx=0).stop 2024-04-06T13:51:35.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.215+0000 7f2e9aac1700 1 -- 172.21.15.189:0/4006046252 shutdown_connections 2024-04-06T13:51:35.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.215+0000 7f2e9aac1700 1 --2- 172.21.15.189:0/4006046252 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e9415e500 0x7f2e9406abb0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.215+0000 7f2e9aac1700 1 --2- 172.21.15.189:0/4006046252 >> v2:172.21.15.192:3300/0 conn(0x7f2e9415d7b0 0x7f2e940b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.215+0000 7f2e9aac1700 1 -- 172.21.15.189:0/4006046252 >> 172.21.15.189:0/4006046252 conn(0x7f2e940d00b0 msgr2=0x7f2e940d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:35.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.215+0000 7f2e9aac1700 1 -- 172.21.15.189:0/4006046252 shutdown_connections 2024-04-06T13:51:35.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.215+0000 7f2e9aac1700 1 -- 172.21.15.189:0/4006046252 wait complete. 2024-04-06T13:51:35.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.216+0000 7f2e9aac1700 1 Processor -- start 2024-04-06T13:51:35.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.216+0000 7f2e9aac1700 1 -- start start 2024-04-06T13:51:35.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.217+0000 7f2e9aac1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e9406a730 0x7f2e9415b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:35.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.217+0000 7f2e9aac1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e9415d7b0 0x7f2e9414fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:35.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.217+0000 7f2e9aac1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e9415e500 0x7f2e940cd310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:35.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.217+0000 7f2e9aac1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e94150110 con 0x7f2e9415d7b0 2024-04-06T13:51:35.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.217+0000 7f2e9aac1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9415b670 con 0x7f2e9406a730 2024-04-06T13:51:35.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.217+0000 7f2e9aac1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9415b950 con 0x7f2e9415e500 2024-04-06T13:51:35.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.217+0000 7f2e93fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e9415d7b0 0x7f2e9414fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:35.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.217+0000 7f2e93fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e9415d7b0 0x7f2e9414fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60228/0 (socket says 172.21.15.189:60228) 2024-04-06T13:51:35.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.217+0000 7f2e93fff700 1 -- 172.21.15.189:0/4183822103 learned_addr learned my addr 172.21.15.189:0/4183822103 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:35.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.218+0000 7f2e9905e700 1 --2- 172.21.15.189:0/4183822103 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e9415e500 0x7f2e940cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:35.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.218+0000 7f2e9885d700 1 --2- 172.21.15.189:0/4183822103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e9406a730 0x7f2e9415b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:35.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.218+0000 7f2e93fff700 1 -- 172.21.15.189:0/4183822103 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e9415e500 msgr2=0x7f2e940cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:35.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.218+0000 7f2e93fff700 1 --2- 172.21.15.189:0/4183822103 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e9415e500 0x7f2e940cd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.218+0000 7f2e93fff700 1 -- 172.21.15.189:0/4183822103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e9406a730 msgr2=0x7f2e9415b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:35.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.218+0000 7f2e93fff700 1 --2- 172.21.15.189:0/4183822103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e9406a730 0x7f2e9415b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.218+0000 7f2e93fff700 1 -- 172.21.15.189:0/4183822103 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e880094a0 con 0x7f2e9415d7b0 2024-04-06T13:51:35.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.218+0000 7f2e9885d700 1 --2- 172.21.15.189:0/4183822103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e9406a730 0x7f2e9415b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:51:35.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.218+0000 7f2e93fff700 1 --2- 172.21.15.189:0/4183822103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e9415d7b0 0x7f2e9414fbd0 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000c490 tx=0x7f2e8000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:35.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.218+0000 7f2e917fa700 1 -- 172.21.15.189:0/4183822103 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e80005d70 con 0x7f2e9415d7b0 2024-04-06T13:51:35.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.218+0000 7f2e917fa700 1 -- 172.21.15.189:0/4183822103 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e800070d0 con 0x7f2e9415d7b0 2024-04-06T13:51:35.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.218+0000 7f2e9aac1700 1 -- 172.21.15.189:0/4183822103 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e8c009a70 con 0x7f2e9415d7b0 2024-04-06T13:51:35.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.218+0000 7f2e917fa700 1 -- 172.21.15.189:0/4183822103 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e80007460 con 0x7f2e9415d7b0 2024-04-06T13:51:35.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.218+0000 7f2e9aac1700 1 -- 172.21.15.189:0/4183822103 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e9415bf10 con 0x7f2e9415d7b0 2024-04-06T13:51:35.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.219+0000 7f2e9aac1700 1 -- 172.21.15.189:0/4183822103 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e94067bb0 con 0x7f2e9415d7b0 2024-04-06T13:51:35.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.219+0000 7f2e917fa700 1 -- 172.21.15.189:0/4183822103 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f2e800075c0 con 0x7f2e9415d7b0 2024-04-06T13:51:35.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.220+0000 7f2e917fa700 1 --2- 172.21.15.189:0/4183822103 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2e7806adb0 0x7f2e7806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:35.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.220+0000 7f2e917fa700 1 -- 172.21.15.189:0/4183822103 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f2e80088ba0 con 0x7f2e9415d7b0 2024-04-06T13:51:35.223 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.220+0000 7f2e9885d700 1 --2- 172.21.15.189:0/4183822103 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2e7806adb0 0x7f2e7806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:35.224 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.220+0000 7f2e9885d700 1 --2- 172.21.15.189:0/4183822103 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2e7806adb0 0x7f2e7806d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2e88009cf0 tx=0x7f2e88008040 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:35.225 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.224+0000 7f2e917fa700 1 -- 172.21.15.189:0/4183822103 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2e80053a60 con 0x7f2e9415d7b0 2024-04-06T13:51:35.417 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.416+0000 7f2e9aac1700 1 -- 172.21.15.189:0/4183822103 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f2e94074990 con 0x7f2e9415d7b0 2024-04-06T13:51:35.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.423+0000 7f2e917fa700 1 -- 172.21.15.189:0/4183822103 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v98) v1 ==== 115+0+0 (secure 0 0 0) 0x7f2e80017020 con 0x7f2e9415d7b0 2024-04-06T13:51:35.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.425+0000 7f2e9aac1700 1 -- 172.21.15.189:0/4183822103 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2e7806adb0 msgr2=0x7f2e7806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:35.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.425+0000 7f2e9aac1700 1 --2- 172.21.15.189:0/4183822103 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2e7806adb0 0x7f2e7806d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2e88009cf0 tx=0x7f2e88008040 comp rx=0 tx=0).stop 2024-04-06T13:51:35.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.425+0000 7f2e9aac1700 1 -- 172.21.15.189:0/4183822103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e9415d7b0 msgr2=0x7f2e9414fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:35.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.425+0000 7f2e9aac1700 1 --2- 172.21.15.189:0/4183822103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e9415d7b0 0x7f2e9414fbd0 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000c490 tx=0x7f2e8000c850 comp rx=0 tx=0).stop 2024-04-06T13:51:35.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.425+0000 7f2e9aac1700 1 -- 172.21.15.189:0/4183822103 shutdown_connections 2024-04-06T13:51:35.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.425+0000 7f2e9aac1700 1 --2- 172.21.15.189:0/4183822103 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f2e7806adb0 0x7f2e7806d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.425+0000 7f2e9aac1700 1 --2- 172.21.15.189:0/4183822103 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e9415e500 0x7f2e940cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.425+0000 7f2e9aac1700 1 --2- 172.21.15.189:0/4183822103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e9406a730 0x7f2e9415b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.425+0000 7f2e9aac1700 1 --2- 172.21.15.189:0/4183822103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e9415d7b0 0x7f2e9414fbd0 unknown :-1 s=CLOSED pgs=3200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.425+0000 7f2e9aac1700 1 -- 172.21.15.189:0/4183822103 >> 172.21.15.189:0/4183822103 conn(0x7f2e940d00b0 msgr2=0x7f2e94150740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:35.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.425+0000 7f2e9aac1700 1 -- 172.21.15.189:0/4183822103 shutdown_connections 2024-04-06T13:51:35.427 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.425+0000 7f2e9aac1700 1 -- 172.21.15.189:0/4183822103 wait complete. 2024-04-06T13:51:35.439 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-06T13:51:35.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.561+0000 7f7caa834700 1 Processor -- start 2024-04-06T13:51:35.562 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.561+0000 7f7caa834700 1 -- start start 2024-04-06T13:51:35.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.562+0000 7f7caa834700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7ca415a550 0x7f7ca40cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:35.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.562+0000 7f7caa834700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ca40b4d70 0x7f7ca40b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:35.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.562+0000 7f7caa834700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ca40cdbe0 con 0x7f7ca40cf900 2024-04-06T13:51:35.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.562+0000 7f7caa834700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca415a940 con 0x7f7ca415a550 2024-04-06T13:51:35.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.562+0000 7f7caa834700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca415ac00 con 0x7f7ca40b4d70 2024-04-06T13:51:35.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.562+0000 7f7ca3fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7ca415a550 0x7f7ca40cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:35.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.562+0000 7f7ca3fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7ca415a550 0x7f7ca40cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60238/0 (socket says 172.21.15.189:60238) 2024-04-06T13:51:35.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.562+0000 7f7ca3fff700 1 -- 172.21.15.189:0/1728740111 learned_addr learned my addr 172.21.15.189:0/1728740111 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:35.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.562+0000 7f7ca8dd1700 1 --2- 172.21.15.189:0/1728740111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ca40b4d70 0x7f7ca40b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:35.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.562+0000 7f7ca3fff700 1 -- 172.21.15.189:0/1728740111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ca40b4d70 msgr2=0x7f7ca40b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:35.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.562+0000 7f7ca3fff700 1 --2- 172.21.15.189:0/1728740111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ca40b4d70 0x7f7ca40b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.562+0000 7f7ca3fff700 1 -- 172.21.15.189:0/1728740111 >> v1:172.21.15.192:6789/0 conn(0x7f7ca40cf900 legacy=0x7f7ca40b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:35.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.562+0000 7f7ca3fff700 1 -- 172.21.15.189:0/1728740111 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca415aee0 con 0x7f7ca415a550 2024-04-06T13:51:35.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.562+0000 7f7ca3fff700 1 --2- 172.21.15.189:0/1728740111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca415a550 0x7f7ca40cf3c0 secure :-1 s=READY pgs=3201 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800bb90 tx=0x7f7c9800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da34400bcfe9bc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:35.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.563+0000 7f7ca2ffd700 1 -- 172.21.15.189:0/1728740111 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9800dd30 con 0x7f7ca415a550 2024-04-06T13:51:35.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.563+0000 7f7ca2ffd700 1 -- 172.21.15.189:0/1728740111 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9800de90 con 0x7f7ca415a550 2024-04-06T13:51:35.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.563+0000 7f7ca2ffd700 1 -- 172.21.15.189:0/1728740111 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98010690 con 0x7f7ca415a550 2024-04-06T13:51:35.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.563+0000 7f7caa834700 1 -- 172.21.15.189:0/1728740111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca415a550 msgr2=0x7f7ca40cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:35.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.563+0000 7f7caa834700 1 --2- 172.21.15.189:0/1728740111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca415a550 0x7f7ca40cf3c0 secure :-1 s=READY pgs=3201 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800bb90 tx=0x7f7c9800bea0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.563+0000 7f7caa834700 1 -- 172.21.15.189:0/1728740111 shutdown_connections 2024-04-06T13:51:35.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.563+0000 7f7caa834700 1 --2- 172.21.15.189:0/1728740111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ca40b4d70 0x7f7ca40b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.563+0000 7f7caa834700 1 --2- 172.21.15.189:0/1728740111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca415a550 0x7f7ca40cf3c0 unknown :-1 s=CLOSED pgs=3201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.563+0000 7f7caa834700 1 -- 172.21.15.189:0/1728740111 >> 172.21.15.189:0/1728740111 conn(0x7f7ca40d00b0 msgr2=0x7f7ca40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:35.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.563+0000 7f7caa834700 1 -- 172.21.15.189:0/1728740111 shutdown_connections 2024-04-06T13:51:35.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.563+0000 7f7caa834700 1 -- 172.21.15.189:0/1728740111 wait complete. 2024-04-06T13:51:35.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.564+0000 7f7caa834700 1 Processor -- start 2024-04-06T13:51:35.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.565+0000 7f7caa834700 1 -- start start 2024-04-06T13:51:35.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.565+0000 7f7caa834700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca40b4d70 0x7f7ca4152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:35.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.565+0000 7f7caa834700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ca40cf900 0x7f7ca40a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:35.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.565+0000 7f7caa834700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ca4154860 0x7f7ca4154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:35.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.565+0000 7f7caa834700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca4155250 con 0x7f7ca40b4d70 2024-04-06T13:51:35.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.565+0000 7f7caa834700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca41f2d30 con 0x7f7ca40cf900 2024-04-06T13:51:35.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.565+0000 7f7caa834700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca41f2120 con 0x7f7ca4154860 2024-04-06T13:51:35.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.565+0000 7f7ca3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca40b4d70 0x7f7ca4152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:35.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.565+0000 7f7ca3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca40b4d70 0x7f7ca4152d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60254/0 (socket says 172.21.15.189:60254) 2024-04-06T13:51:35.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.565+0000 7f7ca3fff700 1 -- 172.21.15.189:0/2260058738 learned_addr learned my addr 172.21.15.189:0/2260058738 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:35.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.566+0000 7f7ca8dd1700 1 --2- 172.21.15.189:0/2260058738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ca4154860 0x7f7ca4154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:35.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.566+0000 7f7ca37fe700 1 --2- 172.21.15.189:0/2260058738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ca40cf900 0x7f7ca40a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:35.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.566+0000 7f7ca3fff700 1 -- 172.21.15.189:0/2260058738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ca4154860 msgr2=0x7f7ca4154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:35.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.566+0000 7f7ca3fff700 1 --2- 172.21.15.189:0/2260058738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ca4154860 0x7f7ca4154d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.566+0000 7f7ca3fff700 1 -- 172.21.15.189:0/2260058738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ca40cf900 msgr2=0x7f7ca40a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:35.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.566+0000 7f7ca3fff700 1 --2- 172.21.15.189:0/2260058738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ca40cf900 0x7f7ca40a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.566+0000 7f7ca3fff700 1 -- 172.21.15.189:0/2260058738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c9800b7f0 con 0x7f7ca40b4d70 2024-04-06T13:51:35.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.566+0000 7f7ca3fff700 1 --2- 172.21.15.189:0/2260058738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca40b4d70 0x7f7ca4152d00 secure :-1 s=READY pgs=3202 cs=0 l=1 rev1=1 crypto rx=0x7f7c98000c00 tx=0x7f7c98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:35.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.566+0000 7f7ca0ff9700 1 -- 172.21.15.189:0/2260058738 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9801f070 con 0x7f7ca40b4d70 2024-04-06T13:51:35.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.566+0000 7f7ca0ff9700 1 -- 172.21.15.189:0/2260058738 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c98003a70 con 0x7f7ca40b4d70 2024-04-06T13:51:35.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.566+0000 7f7ca0ff9700 1 -- 172.21.15.189:0/2260058738 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98003e00 con 0x7f7ca40b4d70 2024-04-06T13:51:35.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.566+0000 7f7caa834700 1 -- 172.21.15.189:0/2260058738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca41f2400 con 0x7f7ca40b4d70 2024-04-06T13:51:35.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.566+0000 7f7caa834700 1 -- 172.21.15.189:0/2260058738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca41f29c0 con 0x7f7ca40b4d70 2024-04-06T13:51:35.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.567+0000 7f7caa834700 1 -- 172.21.15.189:0/2260058738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca4067bb0 con 0x7f7ca40b4d70 2024-04-06T13:51:35.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.568+0000 7f7ca0ff9700 1 -- 172.21.15.189:0/2260058738 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f7c98019400 con 0x7f7ca40b4d70 2024-04-06T13:51:35.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.568+0000 7f7ca0ff9700 1 --2- 172.21.15.189:0/2260058738 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7c9006adb0 0x7f7c9006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:35.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.568+0000 7f7ca0ff9700 1 -- 172.21.15.189:0/2260058738 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f7c9808cc60 con 0x7f7ca40b4d70 2024-04-06T13:51:35.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.568+0000 7f7ca37fe700 1 --2- 172.21.15.189:0/2260058738 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7c9006adb0 0x7f7c9006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:35.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.569+0000 7f7ca37fe700 1 --2- 172.21.15.189:0/2260058738 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7c9006adb0 0x7f7c9006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7ca406a950 tx=0x7f7c94009040 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:35.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.572+0000 7f7ca0ff9700 1 -- 172.21.15.189:0/2260058738 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7c98057b20 con 0x7f7ca40b4d70 2024-04-06T13:51:35.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.780+0000 7f7caa834700 1 -- 172.21.15.189:0/2260058738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f7ca4074990 con 0x7f7ca40b4d70 2024-04-06T13:51:35.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.784+0000 7f7ca0ff9700 1 -- 172.21.15.189:0/2260058738 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v99) v1 ==== 120+0+0 (secure 0 0 0) 0x7f7c9805b140 con 0x7f7ca40b4d70 2024-04-06T13:51:35.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.786+0000 7f7caa834700 1 -- 172.21.15.189:0/2260058738 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7c9006adb0 msgr2=0x7f7c9006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:35.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.786+0000 7f7caa834700 1 --2- 172.21.15.189:0/2260058738 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7c9006adb0 0x7f7c9006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7ca406a950 tx=0x7f7c94009040 comp rx=0 tx=0).stop 2024-04-06T13:51:35.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.786+0000 7f7caa834700 1 -- 172.21.15.189:0/2260058738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca40b4d70 msgr2=0x7f7ca4152d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:35.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.786+0000 7f7caa834700 1 --2- 172.21.15.189:0/2260058738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca40b4d70 0x7f7ca4152d00 secure :-1 s=READY pgs=3202 cs=0 l=1 rev1=1 crypto rx=0x7f7c98000c00 tx=0x7f7c98004750 comp rx=0 tx=0).stop 2024-04-06T13:51:35.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.786+0000 7f7caa834700 1 -- 172.21.15.189:0/2260058738 shutdown_connections 2024-04-06T13:51:35.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.786+0000 7f7caa834700 1 --2- 172.21.15.189:0/2260058738 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f7c9006adb0 0x7f7c9006d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.786+0000 7f7caa834700 1 --2- 172.21.15.189:0/2260058738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ca4154860 0x7f7ca4154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.786+0000 7f7caa834700 1 --2- 172.21.15.189:0/2260058738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca40b4d70 0x7f7ca4152d00 unknown :-1 s=CLOSED pgs=3202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.786+0000 7f7caa834700 1 --2- 172.21.15.189:0/2260058738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ca40cf900 0x7f7ca40a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.786+0000 7f7caa834700 1 -- 172.21.15.189:0/2260058738 >> 172.21.15.189:0/2260058738 conn(0x7f7ca40d00b0 msgr2=0x7f7ca406f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:35.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.787+0000 7f7caa834700 1 -- 172.21.15.189:0/2260058738 shutdown_connections 2024-04-06T13:51:35.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.787+0000 7f7caa834700 1 -- 172.21.15.189:0/2260058738 wait complete. 2024-04-06T13:51:35.800 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01' 2024-04-06T13:51:35.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.923+0000 7fc0b9417700 1 Processor -- start 2024-04-06T13:51:35.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.923+0000 7fc0b9417700 1 -- start start 2024-04-06T13:51:35.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.923+0000 7fc0b9417700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0b415db10 0x7fc0b40b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:35.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.923+0000 7fc0b9417700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc0b4143470 0x7fc0b40cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:35.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.923+0000 7fc0b9417700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b415eb40 con 0x7fc0b4160b60 2024-04-06T13:51:35.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.923+0000 7fc0b9417700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b4160f40 con 0x7fc0b4143470 2024-04-06T13:51:35.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.923+0000 7fc0b9417700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b40d0bf0 con 0x7fc0b415db10 2024-04-06T13:51:35.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.924+0000 7fc0b37fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc0b4143470 0x7fc0b40cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:35.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.924+0000 7fc0b37fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc0b4143470 0x7fc0b40cd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60264/0 (socket says 172.21.15.189:60264) 2024-04-06T13:51:35.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.924+0000 7fc0b37fe700 1 -- 172.21.15.189:0/3411469027 learned_addr learned my addr 172.21.15.189:0/3411469027 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:35.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.924+0000 7fc0b27fc700 1 --2- 172.21.15.189:0/3411469027 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0b415db10 0x7fc0b40b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:35.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.924+0000 7fc0b37fe700 1 -- 172.21.15.189:0/3411469027 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0b415db10 msgr2=0x7fc0b40b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:35.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.924+0000 7fc0b37fe700 1 --2- 172.21.15.189:0/3411469027 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0b415db10 0x7fc0b40b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.924+0000 7fc0b37fe700 1 -- 172.21.15.189:0/3411469027 >> v1:172.21.15.192:6789/0 conn(0x7fc0b4160b60 legacy=0x7fc0b4142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:35.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.924+0000 7fc0b37fe700 1 -- 172.21.15.189:0/3411469027 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b415dfc0 con 0x7fc0b4143470 2024-04-06T13:51:35.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.924+0000 7fc0b27fc700 1 --2- 172.21.15.189:0/3411469027 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0b415db10 0x7fc0b40b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:51:35.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.924+0000 7fc0b37fe700 1 --2- 172.21.15.189:0/3411469027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0b4143470 0x7fc0b40cd890 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac009e50 tx=0x7fc0ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410afedaed98c3f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:35.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.924+0000 7fc0b1ffb700 1 -- 172.21.15.189:0/3411469027 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac00e070 con 0x7fc0b4143470 2024-04-06T13:51:35.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.924+0000 7fc0b1ffb700 1 -- 172.21.15.189:0/3411469027 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ac003a20 con 0x7fc0b4143470 2024-04-06T13:51:35.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.924+0000 7fc0b1ffb700 1 -- 172.21.15.189:0/3411469027 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac003dd0 con 0x7fc0b4143470 2024-04-06T13:51:35.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.925+0000 7fc0b9417700 1 -- 172.21.15.189:0/3411469027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0b4143470 msgr2=0x7fc0b40cd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:35.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.925+0000 7fc0b9417700 1 --2- 172.21.15.189:0/3411469027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0b4143470 0x7fc0b40cd890 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac009e50 tx=0x7fc0ac00b330 comp rx=0 tx=0).stop 2024-04-06T13:51:35.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.925+0000 7fc0b9417700 1 -- 172.21.15.189:0/3411469027 shutdown_connections 2024-04-06T13:51:35.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.925+0000 7fc0b9417700 1 --2- 172.21.15.189:0/3411469027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0b4143470 0x7fc0b40cd890 unknown :-1 s=CLOSED pgs=3203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.925+0000 7fc0b9417700 1 --2- 172.21.15.189:0/3411469027 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0b415db10 0x7fc0b40b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.925+0000 7fc0b9417700 1 -- 172.21.15.189:0/3411469027 >> 172.21.15.189:0/3411469027 conn(0x7fc0b406a730 msgr2=0x7fc0b406ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:35.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.925+0000 7fc0b9417700 1 -- 172.21.15.189:0/3411469027 shutdown_connections 2024-04-06T13:51:35.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.925+0000 7fc0b9417700 1 -- 172.21.15.189:0/3411469027 wait complete. 2024-04-06T13:51:35.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.926+0000 7fc0b9417700 1 Processor -- start 2024-04-06T13:51:35.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.926+0000 7fc0b9417700 1 -- start start 2024-04-06T13:51:35.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.927+0000 7fc0b9417700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0b4143470 0x7fc0b415ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:35.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.927+0000 7fc0b9417700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0b415db10 0x7fc0b40c8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:35.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.927+0000 7fc0b9417700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0b4160b60 0x7fc0b40c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:35.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.927+0000 7fc0b9417700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b40c7330 con 0x7fc0b4143470 2024-04-06T13:51:35.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.927+0000 7fc0b9417700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b40c92e0 con 0x7fc0b415db10 2024-04-06T13:51:35.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.927+0000 7fc0b9417700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b40c8d20 con 0x7fc0b4160b60 2024-04-06T13:51:35.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.927+0000 7fc0b2ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0b4143470 0x7fc0b415ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:35.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.927+0000 7fc0b2ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0b4143470 0x7fc0b415ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60272/0 (socket says 172.21.15.189:60272) 2024-04-06T13:51:35.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.927+0000 7fc0b2ffd700 1 -- 172.21.15.189:0/2753697287 learned_addr learned my addr 172.21.15.189:0/2753697287 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:35.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.927+0000 7fc0b37fe700 1 --2- 172.21.15.189:0/2753697287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0b4160b60 0x7fc0b40c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:35.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.927+0000 7fc0b27fc700 1 --2- 172.21.15.189:0/2753697287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0b415db10 0x7fc0b40c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:35.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.927+0000 7fc0b2ffd700 1 -- 172.21.15.189:0/2753697287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0b4160b60 msgr2=0x7fc0b40c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:35.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.927+0000 7fc0b2ffd700 1 --2- 172.21.15.189:0/2753697287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0b4160b60 0x7fc0b40c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.927+0000 7fc0b2ffd700 1 -- 172.21.15.189:0/2753697287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0b415db10 msgr2=0x7fc0b40c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:35.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.927+0000 7fc0b2ffd700 1 --2- 172.21.15.189:0/2753697287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0b415db10 0x7fc0b40c8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:35.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.927+0000 7fc0b2ffd700 1 -- 172.21.15.189:0/2753697287 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ac009b30 con 0x7fc0b4143470 2024-04-06T13:51:35.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.927+0000 7fc0b2ffd700 1 --2- 172.21.15.189:0/2753697287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0b4143470 0x7fc0b415ce10 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 crypto rx=0x7fc0a400b4a0 tx=0x7fc0a400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:35.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.928+0000 7fc09b7fe700 1 -- 172.21.15.189:0/2753697287 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a4011830 con 0x7fc0b4143470 2024-04-06T13:51:35.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.928+0000 7fc09b7fe700 1 -- 172.21.15.189:0/2753697287 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a4011990 con 0x7fc0b4143470 2024-04-06T13:51:35.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.928+0000 7fc0b9417700 1 -- 172.21.15.189:0/2753697287 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b40c9000 con 0x7fc0b4143470 2024-04-06T13:51:35.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.928+0000 7fc09b7fe700 1 -- 172.21.15.189:0/2753697287 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a4011d40 con 0x7fc0b4143470 2024-04-06T13:51:35.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.928+0000 7fc0b9417700 1 -- 172.21.15.189:0/2753697287 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b40b6ef0 con 0x7fc0b4143470 2024-04-06T13:51:35.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.929+0000 7fc0b9417700 1 -- 172.21.15.189:0/2753697287 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b40c75a0 con 0x7fc0b4143470 2024-04-06T13:51:35.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.929+0000 7fc09b7fe700 1 -- 172.21.15.189:0/2753697287 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fc0a401a400 con 0x7fc0b4143470 2024-04-06T13:51:35.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.929+0000 7fc09b7fe700 1 --2- 172.21.15.189:0/2753697287 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc0a006adb0 0x7fc0a006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:35.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.930+0000 7fc0b27fc700 1 --2- 172.21.15.189:0/2753697287 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc0a006adb0 0x7fc0a006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:35.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.930+0000 7fc09b7fe700 1 -- 172.21.15.189:0/2753697287 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fc0a4089360 con 0x7fc0b4143470 2024-04-06T13:51:35.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.930+0000 7fc0b27fc700 1 --2- 172.21.15.189:0/2753697287 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc0a006adb0 0x7fc0a006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc0b415def0 tx=0x7fc0a8008040 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:35.935 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:35.934+0000 7fc09b7fe700 1 -- 172.21.15.189:0/2753697287 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc0a40542c0 con 0x7fc0b4143470 2024-04-06T13:51:36.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:36.209+0000 7fc0b9417700 1 -- 172.21.15.189:0/2753697287 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]} v 0) v1 -- 0x7fc0b4074990 con 0x7fc0b4143470 2024-04-06T13:51:36.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:36.674+0000 7fc09b7fe700 1 -- 172.21.15.189:0/2753697287 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]}]=0 v3306) v1 ==== 201+0+0 (secure 0 0 0) 0x7fc0a40578e0 con 0x7fc0b4143470 2024-04-06T13:51:36.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:36.676+0000 7fc0b9417700 1 -- 172.21.15.189:0/2753697287 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc0a006adb0 msgr2=0x7fc0a006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:36.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:36.676+0000 7fc0b9417700 1 --2- 172.21.15.189:0/2753697287 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc0a006adb0 0x7fc0a006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc0b415def0 tx=0x7fc0a8008040 comp rx=0 tx=0).stop 2024-04-06T13:51:36.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:36.676+0000 7fc0b9417700 1 -- 172.21.15.189:0/2753697287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0b4143470 msgr2=0x7fc0b415ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:36.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:36.676+0000 7fc0b9417700 1 --2- 172.21.15.189:0/2753697287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0b4143470 0x7fc0b415ce10 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 crypto rx=0x7fc0a400b4a0 tx=0x7fc0a400b860 comp rx=0 tx=0).stop 2024-04-06T13:51:36.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:36.676+0000 7fc0b9417700 1 -- 172.21.15.189:0/2753697287 shutdown_connections 2024-04-06T13:51:36.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:36.676+0000 7fc0b9417700 1 --2- 172.21.15.189:0/2753697287 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc0a006adb0 0x7fc0a006d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:36.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:36.676+0000 7fc0b9417700 1 --2- 172.21.15.189:0/2753697287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0b4160b60 0x7fc0b40c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:36.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:36.676+0000 7fc0b9417700 1 --2- 172.21.15.189:0/2753697287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0b4143470 0x7fc0b415ce10 unknown :-1 s=CLOSED pgs=3204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:36.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:36.676+0000 7fc0b9417700 1 --2- 172.21.15.189:0/2753697287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0b415db10 0x7fc0b40c8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:36.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:36.676+0000 7fc0b9417700 1 -- 172.21.15.189:0/2753697287 >> 172.21.15.189:0/2753697287 conn(0x7fc0b406a730 msgr2=0x7fc0b40b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:36.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:36.676+0000 7fc0b9417700 1 -- 172.21.15.189:0/2753697287 shutdown_connections 2024-04-06T13:51:36.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:36.676+0000 7fc0b9417700 1 -- 172.21.15.189:0/2753697287 wait complete. 2024-04-06T13:51:36.695 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. ... ERROR 2024-04-06T13:51:36.695 INFO:tasks.cephfs_test_runner: 2024-04-06T13:51:36.695 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-06T13:51:36.695 INFO:tasks.cephfs_test_runner:ERROR: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-04-06T13:51:36.696 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. 2024-04-06T13:51:36.696 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-06T13:51:36.696 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-06T13:51:36.696 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa/tasks/cephfs/test_snap_schedules.py", line 678, in test_snap_schedule_subvol_and_group_arguments_01 2024-04-06T13:51:36.696 INFO:tasks.cephfs_test_runner: self._verify_snap_schedule(self.CREATE_VERSION, 'sv01', None) 2024-04-06T13:51:36.696 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa/tasks/cephfs/test_snap_schedules.py", line 660, in _verify_snap_schedule 2024-04-06T13:51:36.696 INFO:tasks.cephfs_test_runner: path = self._get_subvol_snapdir_path(version, subvol, group) 2024-04-06T13:51:36.696 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa/tasks/cephfs/test_snap_schedules.py", line 653, in _get_subvol_snapdir_path 2024-04-06T13:51:36.696 INFO:tasks.cephfs_test_runner: path = self.get_ceph_cmd_stdout("fs", *args).rstrip() 2024-04-06T13:51:36.696 INFO:tasks.cephfs_test_runner:AttributeError: 'TestSnapSchedulesSubvolAndGroupArguments' object has no attribute 'get_ceph_cmd_stdout' 2024-04-06T13:51:36.696 INFO:tasks.cephfs_test_runner: 2024-04-06T13:51:36.696 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-06T13:51:36.696 INFO:tasks.cephfs_test_runner:Ran 15 tests in 5922.540s 2024-04-06T13:51:36.696 INFO:tasks.cephfs_test_runner: 2024-04-06T13:51:36.696 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-04-06T13:51:36.696 INFO:tasks.cephfs_test_runner: 2024-04-06T13:51:36.696 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-06T13:51:36.697 INFO:tasks.cephfs_test_runner:ERROR: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-04-06T13:51:36.697 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. 2024-04-06T13:51:36.697 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-06T13:51:36.697 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-06T13:51:36.697 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa/tasks/cephfs/test_snap_schedules.py", line 678, in test_snap_schedule_subvol_and_group_arguments_01 2024-04-06T13:51:36.697 INFO:tasks.cephfs_test_runner: self._verify_snap_schedule(self.CREATE_VERSION, 'sv01', None) 2024-04-06T13:51:36.697 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa/tasks/cephfs/test_snap_schedules.py", line 660, in _verify_snap_schedule 2024-04-06T13:51:36.697 INFO:tasks.cephfs_test_runner: path = self._get_subvol_snapdir_path(version, subvol, group) 2024-04-06T13:51:36.697 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa/tasks/cephfs/test_snap_schedules.py", line 653, in _get_subvol_snapdir_path 2024-04-06T13:51:36.697 INFO:tasks.cephfs_test_runner: path = self.get_ceph_cmd_stdout("fs", *args).rstrip() 2024-04-06T13:51:36.697 INFO:tasks.cephfs_test_runner:AttributeError: 'TestSnapSchedulesSubvolAndGroupArguments' object has no attribute 'get_ceph_cmd_stdout' 2024-04-06T13:51:36.697 INFO:tasks.cephfs_test_runner: 2024-04-06T13:51:36.697 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-04-06T13:51:37.626 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=01aec8aa09b140a395b304664c8e49e2 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d7d6e60867cadf19fbfd7655a551ae7d58060757/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_800ec6d64a7fa0648ad53edfc6e50b41fc4cd674/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-04-06T13:51:37.628 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-06T13:51:37.645 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-06T13:51:37.686 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-06T13:51:37.686 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:51:37.686 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:51:37.713 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:51:37.713 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-06T13:51:37.713 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:51:37.774 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244184k,nr_inodes=4061046,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1474 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252520k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-06T13:51:37.774 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:51:37.774 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:51:37.796 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:51:37.796 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T13:51:37.796 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T13:51:37.856 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16244132k,nr_inodes=4061033,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=644 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3252512k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-06T13:51:37.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:51:37.857 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T13:51:37.878 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-06T13:51:37.878 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-06T13:51:37.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:51:37.879 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-06T13:51:37.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:51:37.962 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T13:51:38.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:51:38.057 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link delete ceph-brx 2024-04-06T13:51:38.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:51:38.143 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-06T13:51:38.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:51:38.161 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link delete ceph-brx 2024-04-06T13:51:38.233 INFO:teuthology.orchestra.run.smithi189.stderr:Cannot find device "ceph-brx" 2024-04-06T13:51:38.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T13:51:38.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:51:38.236 DEBUG:teuthology.orchestra.run.smithi192:> ip netns list 2024-04-06T13:51:38.257 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-06T13:51:38.257 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-06T13:51:38.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:51:38.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-06T13:51:38.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:51:38.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-06T13:51:38.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:51:38.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link delete ceph-brx 2024-04-06T13:51:38.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:51:38.484 DEBUG:teuthology.orchestra.run.smithi192:> ip netns list 2024-04-06T13:51:38.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T13:51:38.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link delete ceph-brx 2024-04-06T13:51:38.575 INFO:teuthology.orchestra.run.smithi192.stderr:Cannot find device "ceph-brx" 2024-04-06T13:51:38.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T13:51:38.580 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-06T13:51:38.589 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-06T13:51:38.590 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:51:38.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.713+0000 7f5f3ce9a700 1 Processor -- start 2024-04-06T13:51:38.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.713+0000 7f5f3ce9a700 1 -- start start 2024-04-06T13:51:38.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.714+0000 7f5f3ce9a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f380d11b0 0x7f5f380d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:38.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.714+0000 7f5f3ce9a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5f380cfa00 0x7f5f380cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:38.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.714+0000 7f5f3ce9a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f380d1ad0 con 0x7f5f380cb870 2024-04-06T13:51:38.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.714+0000 7f5f3ce9a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f380ccdc0 con 0x7f5f380cfa00 2024-04-06T13:51:38.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.714+0000 7f5f3ce9a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f380d0b50 con 0x7f5f380d11b0 2024-04-06T13:51:38.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.714+0000 7f5f36d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5f380cfa00 0x7f5f380cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:38.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.714+0000 7f5f36d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5f380cfa00 0x7f5f380cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60286/0 (socket says 172.21.15.189:60286) 2024-04-06T13:51:38.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.714+0000 7f5f36d9d700 1 -- 172.21.15.189:0/416732062 learned_addr learned my addr 172.21.15.189:0/416732062 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:38.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.714+0000 7f5f3659c700 1 --2- 172.21.15.189:0/416732062 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f380d11b0 0x7f5f380d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:38.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.714+0000 7f5f36d9d700 1 -- 172.21.15.189:0/416732062 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f380d11b0 msgr2=0x7f5f380d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:38.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.714+0000 7f5f36d9d700 1 --2- 172.21.15.189:0/416732062 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f380d11b0 0x7f5f380d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:38.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.714+0000 7f5f36d9d700 1 -- 172.21.15.189:0/416732062 >> v1:172.21.15.192:6789/0 conn(0x7f5f380cb870 legacy=0x7f5f3815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:38.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.714+0000 7f5f36d9d700 1 -- 172.21.15.189:0/416732062 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f380cbd80 con 0x7f5f380cfa00 2024-04-06T13:51:38.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.714+0000 7f5f3559a700 1 -- 172.21.15.189:0/416732062 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2589840418 0 0) 0x7f5f380d1ad0 con 0x7f5f380cb870 2024-04-06T13:51:38.717 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.714+0000 7f5f36d9d700 1 --2- 172.21.15.189:0/416732062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f380cfa00 0x7f5f380cd560 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 crypto rx=0x7f5f30009e50 tx=0x7f5f3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fff42069e936ff8 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:38.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.714+0000 7f5f3559a700 1 -- 172.21.15.189:0/416732062 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3000e070 con 0x7f5f380cfa00 2024-04-06T13:51:38.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.715+0000 7f5f3559a700 1 -- 172.21.15.189:0/416732062 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3000b910 con 0x7f5f380cfa00 2024-04-06T13:51:38.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.715+0000 7f5f3559a700 1 -- 172.21.15.189:0/416732062 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3000bcc0 con 0x7f5f380cfa00 2024-04-06T13:51:38.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.715+0000 7f5f3ce9a700 1 -- 172.21.15.189:0/416732062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f380cfa00 msgr2=0x7f5f380cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:38.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.715+0000 7f5f3ce9a700 1 --2- 172.21.15.189:0/416732062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f380cfa00 0x7f5f380cd560 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 crypto rx=0x7f5f30009e50 tx=0x7f5f3000b330 comp rx=0 tx=0).stop 2024-04-06T13:51:38.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.715+0000 7f5f3ce9a700 1 -- 172.21.15.189:0/416732062 shutdown_connections 2024-04-06T13:51:38.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.715+0000 7f5f3ce9a700 1 --2- 172.21.15.189:0/416732062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f380cfa00 0x7f5f380cd560 unknown :-1 s=CLOSED pgs=3205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:38.718 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.715+0000 7f5f3ce9a700 1 --2- 172.21.15.189:0/416732062 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f380d11b0 0x7f5f380d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:38.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.715+0000 7f5f3ce9a700 1 -- 172.21.15.189:0/416732062 >> 172.21.15.189:0/416732062 conn(0x7f5f380b2db0 msgr2=0x7f5f380b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:38.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.715+0000 7f5f3ce9a700 1 -- 172.21.15.189:0/416732062 shutdown_connections 2024-04-06T13:51:38.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.715+0000 7f5f3ce9a700 1 -- 172.21.15.189:0/416732062 wait complete. 2024-04-06T13:51:38.722 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.716+0000 7f5f3ce9a700 1 Processor -- start 2024-04-06T13:51:38.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.716+0000 7f5f3ce9a700 1 -- start start 2024-04-06T13:51:38.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.717+0000 7f5f3ce9a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f380cb870 0x7f5f38154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:38.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.717+0000 7f5f3ce9a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f380cfa00 0x7f5f380b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:38.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.717+0000 7f5f3ce9a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f380d11b0 0x7f5f38155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:38.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.717+0000 7f5f3ce9a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3815ca40 con 0x7f5f380d11b0 2024-04-06T13:51:38.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.717+0000 7f5f3ce9a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f380b5d10 con 0x7f5f380cb870 2024-04-06T13:51:38.723 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.717+0000 7f5f3ce9a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f380b18e0 con 0x7f5f380cfa00 2024-04-06T13:51:38.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.717+0000 7f5f36d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f380d11b0 0x7f5f38155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:38.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.717+0000 7f5f36d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f380d11b0 0x7f5f38155010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60294/0 (socket says 172.21.15.189:60294) 2024-04-06T13:51:38.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.717+0000 7f5f36d9d700 1 -- 172.21.15.189:0/3146707616 learned_addr learned my addr 172.21.15.189:0/3146707616 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:38.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.717+0000 7f5f35d9b700 1 --2- 172.21.15.189:0/3146707616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f380cfa00 0x7f5f380b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:38.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.718+0000 7f5f3659c700 1 --2- 172.21.15.189:0/3146707616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f380cb870 0x7f5f38154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:38.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.718+0000 7f5f36d9d700 1 -- 172.21.15.189:0/3146707616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f380cfa00 msgr2=0x7f5f380b61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:38.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.718+0000 7f5f36d9d700 1 --2- 172.21.15.189:0/3146707616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f380cfa00 0x7f5f380b61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:38.724 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.718+0000 7f5f36d9d700 1 -- 172.21.15.189:0/3146707616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f380cb870 msgr2=0x7f5f38154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:38.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.718+0000 7f5f36d9d700 1 --2- 172.21.15.189:0/3146707616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f380cb870 0x7f5f38154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:38.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.718+0000 7f5f36d9d700 1 -- 172.21.15.189:0/3146707616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f30009b30 con 0x7f5f380d11b0 2024-04-06T13:51:38.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.718+0000 7f5f36d9d700 1 --2- 172.21.15.189:0/3146707616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f380d11b0 0x7f5f38155010 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 crypto rx=0x7f5f380677e0 tx=0x7f5f30004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:38.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.718+0000 7f5f26ffd700 1 -- 172.21.15.189:0/3146707616 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f30003ea0 con 0x7f5f380d11b0 2024-04-06T13:51:38.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.718+0000 7f5f26ffd700 1 -- 172.21.15.189:0/3146707616 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f300048e0 con 0x7f5f380d11b0 2024-04-06T13:51:38.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.718+0000 7f5f3ce9a700 1 -- 172.21.15.189:0/3146707616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f380b1bc0 con 0x7f5f380d11b0 2024-04-06T13:51:38.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.718+0000 7f5f26ffd700 1 -- 172.21.15.189:0/3146707616 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3001b670 con 0x7f5f380d11b0 2024-04-06T13:51:38.725 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.718+0000 7f5f3ce9a700 1 -- 172.21.15.189:0/3146707616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f380b59a0 con 0x7f5f380d11b0 2024-04-06T13:51:38.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.719+0000 7f5f3ce9a700 1 -- 172.21.15.189:0/3146707616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f380b5b30 con 0x7f5f380d11b0 2024-04-06T13:51:38.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.719+0000 7f5f26ffd700 1 -- 172.21.15.189:0/3146707616 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f5f3001b7d0 con 0x7f5f380d11b0 2024-04-06T13:51:38.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.720+0000 7f5f26ffd700 1 --2- 172.21.15.189:0/3146707616 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5f1806adb0 0x7f5f1806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:38.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.720+0000 7f5f26ffd700 1 -- 172.21.15.189:0/3146707616 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f5f3008aea0 con 0x7f5f380d11b0 2024-04-06T13:51:38.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.720+0000 7f5f3659c700 1 --2- 172.21.15.189:0/3146707616 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5f1806adb0 0x7f5f1806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:38.726 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.720+0000 7f5f3659c700 1 --2- 172.21.15.189:0/3146707616 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5f1806adb0 0x7f5f1806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5f380cae20 tx=0x7f5f28009210 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:38.727 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.724+0000 7f5f26ffd700 1 -- 172.21.15.189:0/3146707616 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5f30055d80 con 0x7f5f380d11b0 2024-04-06T13:51:38.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.915+0000 7f5f3ce9a700 1 -- 172.21.15.189:0/3146707616 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5f380cf4e0 con 0x7f5f1806adb0 2024-04-06T13:51:38.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.917+0000 7f5f26ffd700 1 -- 172.21.15.189:0/3146707616 <== mgr.10415 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51096 (secure 0 0 0) 0x7f5f28011840 con 0x7f5f1806adb0 2024-04-06T13:51:38.918 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:51:38.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.920+0000 7f5f3ce9a700 1 -- 172.21.15.189:0/3146707616 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5f1806adb0 msgr2=0x7f5f1806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:38.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.920+0000 7f5f3ce9a700 1 --2- 172.21.15.189:0/3146707616 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5f1806adb0 0x7f5f1806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5f380cae20 tx=0x7f5f28009210 comp rx=0 tx=0).stop 2024-04-06T13:51:38.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.920+0000 7f5f3ce9a700 1 -- 172.21.15.189:0/3146707616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f380d11b0 msgr2=0x7f5f38155010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:38.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.920+0000 7f5f3ce9a700 1 --2- 172.21.15.189:0/3146707616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f380d11b0 0x7f5f38155010 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 crypto rx=0x7f5f380677e0 tx=0x7f5f30004ad0 comp rx=0 tx=0).stop 2024-04-06T13:51:38.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.920+0000 7f5f3ce9a700 1 -- 172.21.15.189:0/3146707616 shutdown_connections 2024-04-06T13:51:38.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.920+0000 7f5f3ce9a700 1 --2- 172.21.15.189:0/3146707616 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f5f1806adb0 0x7f5f1806d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:38.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.920+0000 7f5f3ce9a700 1 --2- 172.21.15.189:0/3146707616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f380d11b0 0x7f5f38155010 unknown :-1 s=CLOSED pgs=3206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:38.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.920+0000 7f5f3ce9a700 1 --2- 172.21.15.189:0/3146707616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f380cb870 0x7f5f38154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:38.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.920+0000 7f5f3ce9a700 1 --2- 172.21.15.189:0/3146707616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f380cfa00 0x7f5f380b61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:38.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.920+0000 7f5f3ce9a700 1 -- 172.21.15.189:0/3146707616 >> 172.21.15.189:0/3146707616 conn(0x7f5f380b2db0 msgr2=0x7f5f380b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:38.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.920+0000 7f5f3ce9a700 1 -- 172.21.15.189:0/3146707616 shutdown_connections 2024-04-06T13:51:38.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:38.920+0000 7f5f3ce9a700 1 -- 172.21.15.189:0/3146707616 wait complete. 2024-04-06T13:51:38.922 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:51:38.932 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-06T13:51:38.511306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":133,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":482168,"kb_used_data":29832,"kb_used_omap":12,"kb_used_meta":452211,"kb_avail":754492552,"statfs":{"total":773094113280,"available":772600373248,"internally_reserved":0,"allocated":30547968,"data_stored":16347617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":463064632},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-766700,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-200,"num_read_kb":-56,"num_write":-292,"num_write_kb":-2300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-123,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.008272"},"pg_stats":[{"pgid":"1.0","version":"288'425","reported_seq":1575,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-06T13:51:33.507521+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:51:33.507521+0000","last_peered":"2024-04-06T13:51:33.507521+0000","last_clean":"2024-04-06T13:51:33.507521+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:51:33.507521+0000","last_undegraded":"2024-04-06T13:51:33.507521+0000","last_fullsized":"2024-04-06T13:51:33.507521+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576027,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57428,"kb_used_data":4168,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314412,"statfs":{"total":96636764160,"available":96577957888,"internally_reserved":0,"allocated":4268032,"data_stored":2487692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55500000000000005,"15min":0.58799999999999997},"min":{"1min":0.34499999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.82499999999999996},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53200000000000003,"15min":0.57399999999999995},"min":{"1min":0.47499999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.76200000000000001,"5min":0.78500000000000003,"15min":0.82899999999999996},"last":0.60099999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.53900000000000003,"15min":0.57099999999999995},"min":{"1min":0.39700000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.748},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.54400000000000004,"15min":0.58399999999999996},"min":{"1min":0.41999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.85299999999999998},"last":0.51900000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.52100000000000002,"15min":0.56999999999999995},"min":{"1min":0.36399999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.83899999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56100000000000005,"15min":0.59199999999999997},"min":{"1min":0.45000000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76800000000000002,"5min":0.76900000000000002,"15min":0.79700000000000004},"last":0.67900000000000005}]},{"osd":3,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54600000000000004,"15min":0.58999999999999997},"min":{"1min":0.39700000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.75,"5min":0.75600000000000001,"15min":0.81100000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.622,"5min":0.57299999999999995,"15min":0.59599999999999997},"min":{"1min":0.497,"5min":0.39000000000000001,"15min":0.34999999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.84099999999999997},"last":0.64800000000000002}]},{"osd":4,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45400000000000001,"15min":0.46300000000000002},"min":{"1min":0.32400000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.60999999999999999,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.54100000000000004,"15min":0.54300000000000004},"min":{"1min":0.432,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70999999999999996,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.66400000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.5,"15min":0.498},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64200000000000002,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55300000000000005,"15min":0.57399999999999995},"min":{"1min":0.375,"5min":0.35599999999999998,"15min":0.34799999999999998},"max":{"1min":0.66800000000000004,"5min":0.77800000000000002,"15min":0.82099999999999995},"last":0.68899999999999995}]},{"osd":6,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.53800000000000003,"15min":0.55400000000000005},"min":{"1min":0.39200000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67400000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.57999999999999996},"min":{"1min":0.433,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67500000000000004,"5min":0.86299999999999999,"15min":0.88200000000000001},"last":0.62}]}]},{"osd":6,"up_from":13,"seq":55834576026,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60944,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310896,"statfs":{"total":96636764160,"available":96574357504,"internally_reserved":0,"allocated":3670016,"data_stored":1896305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.50900000000000001,"15min":0.54800000000000004},"min":{"1min":0.373,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.74199999999999999,"5min":1.244,"15min":1.244},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.53200000000000003,"15min":0.56100000000000005},"min":{"1min":0.42999999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.68799999999999994,"5min":1.304,"15min":1.304},"last":0.52000000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56100000000000005,"15min":0.60899999999999999},"min":{"1min":0.41799999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.85799999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.60799999999999998,"15min":0.63400000000000001},"min":{"1min":0.49399999999999999,"5min":0.40000000000000002,"15min":0.378},"max":{"1min":0.84299999999999997,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.65500000000000003}]},{"osd":2,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54400000000000004,"15min":0.58099999999999996},"min":{"1min":0.34999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69199999999999995,"5min":0.76700000000000002,"15min":0.82899999999999996},"last":0.5},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.55200000000000005,"15min":0.57899999999999996},"min":{"1min":0.47399999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80800000000000005,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.59399999999999997}]},{"osd":3,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.58999999999999997},"min":{"1min":0.41699999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.78400000000000003,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.60599999999999998,"15min":0.63500000000000001},"min":{"1min":0.501,"5min":0.39900000000000002,"15min":0.36099999999999999},"max":{"1min":0.79600000000000004,"5min":0.80200000000000005,"15min":0.90200000000000002},"last":0.71699999999999997}]},{"osd":4,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55300000000000005,"15min":0.53900000000000003},"min":{"1min":0.39900000000000002,"5min":0.308,"15min":0.308},"max":{"1min":0.72699999999999998,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.34200000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.73899999999999999,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.46300000000000002}]},{"osd":5,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.501,"15min":0.50700000000000001},"min":{"1min":0.34499999999999997,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.62,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54200000000000004,"15min":0.53600000000000003},"min":{"1min":0.32900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.77900000000000003,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":0.77700000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56299999999999994,"15min":0.55200000000000005},"min":{"1min":0.35499999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76700000000000002,"5min":1.603,"15min":1.603},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58399999999999996,"15min":0.57399999999999995},"min":{"1min":0.40500000000000003,"5min":0.33200000000000002,"15min":0.32800000000000001},"max":{"1min":0.80200000000000005,"5min":2.75,"15min":2.75},"last":0.75600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576025,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65036,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306804,"statfs":{"total":96636764160,"available":96570167296,"internally_reserved":0,"allocated":3670016,"data_stored":1896305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.371,"15min":0.38400000000000001},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65600000000000003,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.22},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40300000000000002,"15min":0.41999999999999998},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.67200000000000004,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":0.28899999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.38300000000000001,"15min":0.41099999999999998},"min":{"1min":0.33000000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.251},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.42899999999999999,"15min":0.45700000000000002},"min":{"1min":0.34100000000000003,"5min":0.25,"15min":0.25},"max":{"1min":0.69999999999999996,"5min":2.0169999999999999,"15min":2.0169999999999999},"last":0.26000000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47999999999999998},"min":{"1min":0.31,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61699999999999999,"5min":4.2699999999999996,"15min":4.2699999999999996},"last":0.317},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.41899999999999998,"15min":0.434},"min":{"1min":0.29299999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66600000000000004,"5min":2.153,"15min":2.153},"last":0.34699999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.58399999999999996,"15min":0.624},"min":{"1min":0.495,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.89900000000000002,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.59699999999999998,"15min":0.63},"min":{"1min":0.52700000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80500000000000005,"5min":0.877,"15min":0.995},"last":0.59899999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.60699999999999998,"15min":0.63200000000000001},"min":{"1min":0.50900000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84499999999999997,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.64700000000000002,"15min":0.66100000000000003},"min":{"1min":0.51100000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.88600000000000001,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.68799999999999994}]},{"osd":6,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.61299999999999999,"15min":0.63800000000000001},"min":{"1min":0.52000000000000002,"5min":0.42599999999999999,"15min":0.40799999999999997},"max":{"1min":0.872,"5min":0.89200000000000002,"15min":0.94399999999999995},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.65500000000000003,"15min":0.67000000000000004},"min":{"1min":0.47199999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.91700000000000004,"5min":4.2889999999999997,"15min":4.2889999999999997},"last":0.65600000000000003}]},{"osd":7,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67700000000000005,"15min":0.68899999999999995},"min":{"1min":0.52200000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.86699999999999999,"5min":4.0730000000000004,"15min":4.0730000000000004},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.627,"15min":0.65200000000000002},"min":{"1min":0.46400000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.94199999999999995,"5min":2.194,"15min":2.194},"last":0.625}]}]},{"osd":0,"up_from":13,"seq":55834576025,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61520,"kb_used_data":4164,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310320,"statfs":{"total":96636764160,"available":96573767680,"internally_reserved":0,"allocated":4263936,"data_stored":2486673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.36099999999999999,"15min":0.38100000000000001},"min":{"1min":0.27200000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55800000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.375,"15min":0.40699999999999997},"min":{"1min":0.28499999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.70799999999999996},"last":0.439}]},{"osd":2,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.38700000000000001,"15min":0.41599999999999998},"min":{"1min":0.316,"5min":0.26100000000000001,"15min":0.23899999999999999},"max":{"1min":0.50600000000000001,"5min":0.55900000000000005,"15min":0.70099999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.41499999999999998,"15min":0.435},"min":{"1min":0.33500000000000002,"5min":0.28399999999999997,"15min":0.27100000000000002},"max":{"1min":0.627,"5min":0.627,"15min":0.70499999999999996},"last":0.498}]},{"osd":3,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.38200000000000001,"15min":0.41299999999999998},"min":{"1min":0.28599999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54300000000000004,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.435,"15min":0.46600000000000003},"min":{"1min":0.38500000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.59699999999999998,"5min":0.64100000000000001,"15min":0.75700000000000001},"last":0.56100000000000005}]},{"osd":4,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58999999999999997,"15min":0.626},"min":{"1min":0.45400000000000001,"5min":0.38500000000000001,"15min":0.37},"max":{"1min":0.82399999999999995,"5min":0.93100000000000005,"15min":0.93400000000000005},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59399999999999997,"15min":0.63200000000000001},"min":{"1min":0.42999999999999999,"5min":0.372,"15min":0.36399999999999999},"max":{"1min":0.83599999999999997,"5min":0.999,"15min":0.999},"last":0.69399999999999995}]},{"osd":5,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.57899999999999996,"15min":0.622},"min":{"1min":0.41699999999999998,"5min":0.373,"15min":0.33000000000000002},"max":{"1min":0.80600000000000005,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.623,"15min":0.65400000000000003},"min":{"1min":0.45800000000000002,"5min":0.41399999999999998,"15min":0.378},"max":{"1min":0.872,"5min":0.89500000000000002,"15min":0.96799999999999997},"last":0.76000000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.623,"15min":0.65900000000000003},"min":{"1min":0.42699999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85999999999999999,"5min":0.94399999999999995,"15min":1.0269999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.61599999999999999,"15min":0.65500000000000003},"min":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.39000000000000001},"max":{"1min":0.85699999999999998,"5min":0.93600000000000005,"15min":0.96599999999999997},"last":0.60999999999999999}]},{"osd":7,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.63500000000000001,"15min":0.67600000000000005},"min":{"1min":0.47699999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81000000000000005,"5min":0.91200000000000003,"15min":1.01},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.63200000000000001,"15min":0.66000000000000003},"min":{"1min":0.44600000000000001,"5min":0.42199999999999999,"15min":0.36099999999999999},"max":{"1min":0.88700000000000001,"5min":0.95799999999999996,"15min":0.97699999999999998},"last":0.748}]}]},{"osd":2,"up_from":13,"seq":55834576027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65024,"kb_used_data":3572,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306816,"statfs":{"total":96636764160,"available":96570179584,"internally_reserved":0,"allocated":3657728,"data_stored":1889689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.374},"min":{"1min":0.24199999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.54600000000000004,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.39600000000000002,"15min":0.41399999999999998},"min":{"1min":0.31900000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.66400000000000003,"5min":0.71499999999999997,"15min":1.044},"last":0.377}]},{"osd":1,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.36399999999999999,"15min":0.39300000000000002},"min":{"1min":0.25600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.753},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.441,"15min":0.44600000000000001},"min":{"1min":0.34999999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.60299999999999998,"5min":3.3999999999999999,"15min":3.3999999999999999},"last":0.26100000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.40899999999999997,"15min":0.44600000000000001},"min":{"1min":0.30199999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.70299999999999996},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.41099999999999998,"15min":0.436},"min":{"1min":0.29299999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.69599999999999995,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.34200000000000003}]},{"osd":4,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59199999999999997,"15min":0.626},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.88800000000000001,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.60899999999999999,"15min":0.64600000000000002},"min":{"1min":0.42299999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86199999999999999,"5min":1.022,"15min":1.022},"last":0.60199999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60299999999999998,"15min":0.64800000000000002},"min":{"1min":0.40400000000000003,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80100000000000005,"5min":0.98599999999999999,"15min":1.0149999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.627,"5min":0.61799999999999999,"15min":0.65200000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84699999999999998,"5min":1.028,"15min":1.028},"last":0.54600000000000004}]},{"osd":6,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.63400000000000001,"15min":0.65800000000000003},"min":{"1min":0.51100000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92400000000000004,"5min":0.97199999999999998,"15min":1.091},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.64000000000000001,"15min":0.67900000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91200000000000003,"5min":0.95699999999999996,"15min":0.96099999999999997},"last":0.626}]},{"osd":7,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.64900000000000002,"15min":0.67500000000000004},"min":{"1min":0.435,"5min":0.42299999999999999,"15min":0.41599999999999998},"max":{"1min":0.93899999999999995,"5min":0.94499999999999995,"15min":0.95399999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.65600000000000003,"15min":0.68999999999999995},"min":{"1min":0.44700000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.94899999999999995,"5min":1.01,"15min":1.01},"last":0.67000000000000004}]}]},{"osd":3,"up_from":13,"seq":55834576026,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52756,"kb_used_data":3588,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319084,"statfs":{"total":96636764160,"available":96582742016,"internally_reserved":0,"allocated":3674112,"data_stored":1897324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.36299999999999999,"15min":0.376},"min":{"1min":0.255,"5min":0.223,"15min":0.223},"max":{"1min":0.53400000000000003,"5min":0.55200000000000005,"15min":0.72799999999999998},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.36799999999999999,"15min":0.39700000000000002},"min":{"1min":0.32000000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.63800000000000001},"last":0.34300000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.36699999999999999,"15min":0.39500000000000002},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.309},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.40600000000000003,"15min":0.42799999999999999},"min":{"1min":0.29299999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.60299999999999998,"5min":0.63,"15min":0.75800000000000001},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.434,"15min":0.46899999999999997},"min":{"1min":0.39200000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.67000000000000004,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.44},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.45700000000000002,"15min":0.48399999999999999},"min":{"1min":0.38700000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65200000000000002,"5min":0.67800000000000005,"15min":0.79300000000000004},"last":0.37}]},{"osd":4,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.59099999999999997,"15min":0.61899999999999999},"min":{"1min":0.42299999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":1.018},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.59999999999999998,"15min":0.63300000000000001},"min":{"1min":0.44900000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.0169999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.495}]},{"osd":5,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.64200000000000002,"15min":0.66700000000000004},"min":{"1min":0.47099999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.65200000000000002,"15min":0.67900000000000005},"min":{"1min":0.501,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.94399999999999995,"5min":1.036,"15min":1.036},"last":0.63300000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.626,"15min":0.64900000000000002},"min":{"1min":0.47499999999999998,"5min":0.31,"15min":0.31},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.63,"15min":0.66300000000000003},"min":{"1min":0.433,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.60199999999999998}]},{"osd":7,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.628,"15min":0.65900000000000003},"min":{"1min":0.46000000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90500000000000003,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.64500000000000002,"15min":0.68100000000000005},"min":{"1min":0.53000000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.61899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576026,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52756,"kb_used_data":3588,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319084,"statfs":{"total":96636764160,"available":96582742016,"internally_reserved":0,"allocated":3674112,"data_stored":1897324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54800000000000004,"15min":0.56999999999999995},"min":{"1min":0.38400000000000001,"5min":0.30499999999999999,"15min":0.26300000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.876},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.47899999999999998,"5min":0.36599999999999999,"15min":0.34499999999999997},"max":{"1min":0.77400000000000002,"5min":0.81100000000000005,"15min":1.202},"last":0.72899999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.60899999999999999,"15min":0.628},"min":{"1min":0.46200000000000002,"5min":0.379,"15min":0.35099999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":1.256},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.60299999999999998,"15min":0.61799999999999999},"min":{"1min":0.49199999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.89700000000000002},"last":0.70299999999999996}]},{"osd":2,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56200000000000006,"15min":0.58399999999999996},"min":{"1min":0.438,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80100000000000005,"5min":0.81799999999999995,"15min":0.84599999999999997},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59199999999999997,"15min":0.60999999999999999},"min":{"1min":0.46700000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70799999999999996,"5min":0.97899999999999998,"15min":1.294},"last":0.75800000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56599999999999995,"15min":0.58399999999999996},"min":{"1min":0.498,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.94299999999999995},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.59399999999999997,"15min":0.61599999999999999},"min":{"1min":0.47099999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.90200000000000002},"last":0.71499999999999997}]},{"osd":5,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.50900000000000001,"15min":0.52000000000000002},"min":{"1min":0.35599999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.64100000000000001,"5min":0.92800000000000005,"15min":5.0119999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.54200000000000004},"min":{"1min":0.35199999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68600000000000005,"5min":0.95199999999999996,"15min":4.931},"last":0.52100000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52900000000000003,"15min":0.53200000000000003},"min":{"1min":0.35299999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.77800000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.497},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.40300000000000002,"5min":0.35799999999999998,"15min":0.34699999999999998},"max":{"1min":0.71199999999999997,"5min":1.0609999999999999,"15min":1.3580000000000001},"last":0.63700000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.54900000000000004,"15min":0.57999999999999996},"min":{"1min":0.378,"5min":0.35899999999999999,"15min":0.34000000000000002},"max":{"1min":0.66600000000000004,"5min":0.91000000000000003,"15min":4.9889999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58599999999999997,"15min":0.59899999999999998},"min":{"1min":0.40899999999999997,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72699999999999998,"5min":0.86899999999999999,"15min":0.88100000000000001},"last":0.60199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834576026,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":66704,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305136,"statfs":{"total":96636764160,"available":96568459264,"internally_reserved":0,"allocated":3670016,"data_stored":1896305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54300000000000004,"15min":0.57499999999999996},"min":{"1min":0.438,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.82199999999999995},"last":0.627},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54200000000000004,"15min":0.57699999999999996},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.28100000000000003},"max":{"1min":0.73999999999999999,"5min":0.76600000000000001,"15min":0.84699999999999998},"last":0.58099999999999996}]},{"osd":1,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54400000000000004,"15min":0.57699999999999996},"min":{"1min":0.437,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.81299999999999994},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54300000000000004,"15min":0.57499999999999996},"min":{"1min":0.42799999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69199999999999995,"5min":0.752,"15min":0.871},"last":0.48699999999999999}]},{"osd":2,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.55600000000000005,"15min":0.58699999999999997},"min":{"1min":0.46800000000000003,"5min":0.311,"15min":0.30599999999999999},"max":{"1min":0.72099999999999997,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.56200000000000006,"15min":0.58799999999999997},"min":{"1min":0.44600000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.745,"5min":0.745,"15min":0.83399999999999996},"last":0.54700000000000004}]},{"osd":3,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.55600000000000005,"15min":0.58699999999999997},"min":{"1min":0.40300000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77800000000000002,"5min":0.79800000000000004,"15min":0.81100000000000005},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58399999999999996,"15min":0.621},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.371},"max":{"1min":0.76100000000000001,"5min":0.81399999999999995,"15min":0.88200000000000001},"last":0.76400000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51000000000000001,"15min":0.50700000000000001},"min":{"1min":0.32300000000000001,"5min":0.29899999999999999,"15min":0.27400000000000002},"max":{"1min":0.70999999999999996,"5min":0.75900000000000001,"15min":0.85699999999999998},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57199999999999995,"15min":0.57999999999999996},"min":{"1min":0.34799999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.73999999999999999,"5min":0.82999999999999996,"15min":0.88800000000000001},"last":0.83399999999999996}]},{"osd":6,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35799999999999998,"5min":0.313,"15min":0.28899999999999998},"max":{"1min":0.75700000000000001,"5min":0.89900000000000002,"15min":0.94999999999999996},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52900000000000003,"15min":0.53200000000000003},"min":{"1min":0.372,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.76100000000000001,"5min":0.84299999999999997,"15min":0.878},"last":0.81799999999999995}]},{"osd":7,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.39100000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.76800000000000002,"5min":0.85599999999999998,"15min":0.90800000000000003},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.63800000000000001,"15min":0.64800000000000002},"min":{"1min":0.47399999999999998,"5min":0.41699999999999998,"15min":0.38300000000000001},"max":{"1min":0.79000000000000004,"5min":0.87,"15min":0.92300000000000004},"last":0.90400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:51:38.934 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:51:39.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.055+0000 7f41c4b70700 1 Processor -- start 2024-04-06T13:51:39.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.055+0000 7f41c4b70700 1 -- start start 2024-04-06T13:51:39.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.056+0000 7f41c4b70700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f41c015d7b0 0x7f41c00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.056+0000 7f41c4b70700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f41c006a730 0x7f41c006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.056+0000 7f41c4b70700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c00b3990 con 0x7f41c015ddf0 2024-04-06T13:51:39.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.056+0000 7f41c4b70700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41c015e950 con 0x7f41c015d7b0 2024-04-06T13:51:39.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.056+0000 7f41c4b70700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41c006b0f0 con 0x7f41c006a730 2024-04-06T13:51:39.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.056+0000 7f41be59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f41c015d7b0 0x7f41c00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.056+0000 7f41bed9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f41c015ddf0 0x7f41c015e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54042/0 (socket says 172.21.15.189:54042) 2024-04-06T13:51:39.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.056+0000 7f41bed9d700 1 -- 172.21.15.189:0/572930959 learned_addr learned my addr 172.21.15.189:0/572930959 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:39.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.057+0000 7f41bdd9b700 1 --2- 172.21.15.189:0/572930959 >> v2:172.21.15.192:3300/0 conn(0x7f41c006a730 0x7f41c006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.057+0000 7f41be59c700 1 -- 172.21.15.189:0/572930959 >> v2:172.21.15.192:3300/0 conn(0x7f41c006a730 msgr2=0x7f41c006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:39.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.057+0000 7f41be59c700 1 --2- 172.21.15.189:0/572930959 >> v2:172.21.15.192:3300/0 conn(0x7f41c006a730 0x7f41c006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.057+0000 7f41be59c700 1 -- 172.21.15.189:0/572930959 >> v1:172.21.15.189:6789/0 conn(0x7f41c015ddf0 legacy=0x7f41c015e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:39.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.057+0000 7f41be59c700 1 -- 172.21.15.189:0/572930959 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c00b3060 con 0x7f41c015d7b0 2024-04-06T13:51:39.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.057+0000 7f41bd59a700 1 -- 172.21.15.189:0/572930959 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826591194 0 0) 0x7f41c00b3990 con 0x7f41c015ddf0 2024-04-06T13:51:39.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.057+0000 7f41bdd9b700 1 --2- 172.21.15.189:0/572930959 >> v2:172.21.15.192:3300/0 conn(0x7f41c006a730 0x7f41c006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:51:39.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.057+0000 7f41be59c700 1 --2- 172.21.15.189:0/572930959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41c015d7b0 0x7f41c00b3e20 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 crypto rx=0x7f41c00afc00 tx=0x7f41b4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d5c62e14351aef server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:39.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.057+0000 7f41bd59a700 1 -- 172.21.15.189:0/572930959 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b4011830 con 0x7f41c015d7b0 2024-04-06T13:51:39.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.057+0000 7f41bd59a700 1 -- 172.21.15.189:0/572930959 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b4011990 con 0x7f41c015d7b0 2024-04-06T13:51:39.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.057+0000 7f41bd59a700 1 -- 172.21.15.189:0/572930959 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b4011d20 con 0x7f41c015d7b0 2024-04-06T13:51:39.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.058+0000 7f41c4b70700 1 -- 172.21.15.189:0/572930959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41c015d7b0 msgr2=0x7f41c00b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:39.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.058+0000 7f41c4b70700 1 --2- 172.21.15.189:0/572930959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41c015d7b0 0x7f41c00b3e20 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 crypto rx=0x7f41c00afc00 tx=0x7f41b4009b20 comp rx=0 tx=0).stop 2024-04-06T13:51:39.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.058+0000 7f41c4b70700 1 -- 172.21.15.189:0/572930959 shutdown_connections 2024-04-06T13:51:39.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.058+0000 7f41c4b70700 1 --2- 172.21.15.189:0/572930959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41c015d7b0 0x7f41c00b3e20 unknown :-1 s=CLOSED pgs=3208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.058+0000 7f41c4b70700 1 --2- 172.21.15.189:0/572930959 >> v2:172.21.15.192:3300/0 conn(0x7f41c006a730 0x7f41c006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.058+0000 7f41c4b70700 1 -- 172.21.15.189:0/572930959 >> 172.21.15.189:0/572930959 conn(0x7f41c00d00b0 msgr2=0x7f41c00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:39.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.058+0000 7f41c4b70700 1 -- 172.21.15.189:0/572930959 shutdown_connections 2024-04-06T13:51:39.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.058+0000 7f41c4b70700 1 -- 172.21.15.189:0/572930959 wait complete. 2024-04-06T13:51:39.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.059+0000 7f41c4b70700 1 Processor -- start 2024-04-06T13:51:39.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.059+0000 7f41c4b70700 1 -- start start 2024-04-06T13:51:39.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41c4b70700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41c006a730 0x7f41c0148cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41c4b70700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41c015d7b0 0x7f41c006eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41c4b70700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41c015ddf0 0x7f41c006f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41c4b70700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c0149750 con 0x7f41c015ddf0 2024-04-06T13:51:39.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41c4b70700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c0149a30 con 0x7f41c006a730 2024-04-06T13:51:39.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41c4b70700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c0149d10 con 0x7f41c015d7b0 2024-04-06T13:51:39.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41bed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41c015ddf0 0x7f41c006f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41bed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41c015ddf0 0x7f41c006f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:60318/0 (socket says 172.21.15.189:60318) 2024-04-06T13:51:39.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41bed9d700 1 -- 172.21.15.189:0/2871341204 learned_addr learned my addr 172.21.15.189:0/2871341204 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:39.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41be59c700 1 --2- 172.21.15.189:0/2871341204 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41c006a730 0x7f41c0148cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41bdd9b700 1 --2- 172.21.15.189:0/2871341204 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41c015d7b0 0x7f41c006eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41bed9d700 1 -- 172.21.15.189:0/2871341204 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41c015d7b0 msgr2=0x7f41c006eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:39.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41bed9d700 1 --2- 172.21.15.189:0/2871341204 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41c015d7b0 0x7f41c006eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41bed9d700 1 -- 172.21.15.189:0/2871341204 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41c006a730 msgr2=0x7f41c0148cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:39.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41bed9d700 1 --2- 172.21.15.189:0/2871341204 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41c006a730 0x7f41c0148cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41bed9d700 1 -- 172.21.15.189:0/2871341204 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b40094a0 con 0x7f41c015ddf0 2024-04-06T13:51:39.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41bed9d700 1 --2- 172.21.15.189:0/2871341204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41c015ddf0 0x7f41c006f090 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7f41b800cba0 tx=0x7f41b800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:39.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41aeffd700 1 -- 172.21.15.189:0/2871341204 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b800ab80 con 0x7f41c015ddf0 2024-04-06T13:51:39.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41aeffd700 1 -- 172.21.15.189:0/2871341204 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b800ad00 con 0x7f41c015ddf0 2024-04-06T13:51:39.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41c4b70700 1 -- 172.21.15.189:0/2871341204 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c0149ff0 con 0x7f41c015ddf0 2024-04-06T13:51:39.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41aeffd700 1 -- 172.21.15.189:0/2871341204 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b8007310 con 0x7f41c015ddf0 2024-04-06T13:51:39.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.060+0000 7f41c4b70700 1 -- 172.21.15.189:0/2871341204 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c00cc9d0 con 0x7f41c015ddf0 2024-04-06T13:51:39.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.061+0000 7f41c4b70700 1 -- 172.21.15.189:0/2871341204 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c00ccc50 con 0x7f41c015ddf0 2024-04-06T13:51:39.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.062+0000 7f41aeffd700 1 -- 172.21.15.189:0/2871341204 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f41b8007470 con 0x7f41c015ddf0 2024-04-06T13:51:39.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.062+0000 7f41aeffd700 1 --2- 172.21.15.189:0/2871341204 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f41a006adb0 0x7f41a006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.062+0000 7f41aeffd700 1 -- 172.21.15.189:0/2871341204 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f41b8089430 con 0x7f41c015ddf0 2024-04-06T13:51:39.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.062+0000 7f41be59c700 1 --2- 172.21.15.189:0/2871341204 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f41a006adb0 0x7f41a006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.063+0000 7f41be59c700 1 --2- 172.21.15.189:0/2871341204 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f41a006adb0 0x7f41a006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f41b40090b0 tx=0x7f41b4008030 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:39.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.067+0000 7f41aeffd700 1 -- 172.21.15.189:0/2871341204 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f41b8054390 con 0x7f41c015ddf0 2024-04-06T13:51:39.258 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.258+0000 7f41c4b70700 1 -- 172.21.15.189:0/2871341204 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f41c015eb40 con 0x7f41a006adb0 2024-04-06T13:51:39.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.260+0000 7f41aeffd700 1 -- 172.21.15.189:0/2871341204 <== mgr.10415 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51096 (secure 0 0 0) 0x7f41b4011c00 con 0x7f41a006adb0 2024-04-06T13:51:39.261 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:51:39.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.262+0000 7f41c4b70700 1 -- 172.21.15.189:0/2871341204 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f41a006adb0 msgr2=0x7f41a006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:39.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.262+0000 7f41c4b70700 1 --2- 172.21.15.189:0/2871341204 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f41a006adb0 0x7f41a006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f41b40090b0 tx=0x7f41b4008030 comp rx=0 tx=0).stop 2024-04-06T13:51:39.262 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.262+0000 7f41c4b70700 1 -- 172.21.15.189:0/2871341204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41c015ddf0 msgr2=0x7f41c006f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:39.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.262+0000 7f41c4b70700 1 --2- 172.21.15.189:0/2871341204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41c015ddf0 0x7f41c006f090 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7f41b800cba0 tx=0x7f41b800cf60 comp rx=0 tx=0).stop 2024-04-06T13:51:39.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.262+0000 7f41c4b70700 1 -- 172.21.15.189:0/2871341204 shutdown_connections 2024-04-06T13:51:39.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.262+0000 7f41c4b70700 1 --2- 172.21.15.189:0/2871341204 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f41a006adb0 0x7f41a006d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.262+0000 7f41c4b70700 1 --2- 172.21.15.189:0/2871341204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41c015ddf0 0x7f41c006f090 unknown :-1 s=CLOSED pgs=3209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.262+0000 7f41c4b70700 1 --2- 172.21.15.189:0/2871341204 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41c006a730 0x7f41c0148cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.262+0000 7f41c4b70700 1 --2- 172.21.15.189:0/2871341204 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41c015d7b0 0x7f41c006eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.262+0000 7f41c4b70700 1 -- 172.21.15.189:0/2871341204 >> 172.21.15.189:0/2871341204 conn(0x7f41c00d00b0 msgr2=0x7f41c006fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:39.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.262+0000 7f41c4b70700 1 -- 172.21.15.189:0/2871341204 shutdown_connections 2024-04-06T13:51:39.264 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.262+0000 7f41c4b70700 1 -- 172.21.15.189:0/2871341204 wait complete. 2024-04-06T13:51:39.264 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:51:39.274 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-06T13:51:38.511306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":133,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":482168,"kb_used_data":29832,"kb_used_omap":12,"kb_used_meta":452211,"kb_avail":754492552,"statfs":{"total":773094113280,"available":772600373248,"internally_reserved":0,"allocated":30547968,"data_stored":16347617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":463064632},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-766700,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-200,"num_read_kb":-56,"num_write":-292,"num_write_kb":-2300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-123,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.008272"},"pg_stats":[{"pgid":"1.0","version":"288'425","reported_seq":1575,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-06T13:51:33.507521+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:51:33.507521+0000","last_peered":"2024-04-06T13:51:33.507521+0000","last_clean":"2024-04-06T13:51:33.507521+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:51:33.507521+0000","last_undegraded":"2024-04-06T13:51:33.507521+0000","last_fullsized":"2024-04-06T13:51:33.507521+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576027,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57428,"kb_used_data":4168,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314412,"statfs":{"total":96636764160,"available":96577957888,"internally_reserved":0,"allocated":4268032,"data_stored":2487692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55500000000000005,"15min":0.58799999999999997},"min":{"1min":0.34499999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.82499999999999996},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53200000000000003,"15min":0.57399999999999995},"min":{"1min":0.47499999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.76200000000000001,"5min":0.78500000000000003,"15min":0.82899999999999996},"last":0.60099999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.53900000000000003,"15min":0.57099999999999995},"min":{"1min":0.39700000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.748},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.54400000000000004,"15min":0.58399999999999996},"min":{"1min":0.41999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.85299999999999998},"last":0.51900000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.52100000000000002,"15min":0.56999999999999995},"min":{"1min":0.36399999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.83899999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56100000000000005,"15min":0.59199999999999997},"min":{"1min":0.45000000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76800000000000002,"5min":0.76900000000000002,"15min":0.79700000000000004},"last":0.67900000000000005}]},{"osd":3,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54600000000000004,"15min":0.58999999999999997},"min":{"1min":0.39700000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.75,"5min":0.75600000000000001,"15min":0.81100000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.622,"5min":0.57299999999999995,"15min":0.59599999999999997},"min":{"1min":0.497,"5min":0.39000000000000001,"15min":0.34999999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.84099999999999997},"last":0.64800000000000002}]},{"osd":4,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45400000000000001,"15min":0.46300000000000002},"min":{"1min":0.32400000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.60999999999999999,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.54100000000000004,"15min":0.54300000000000004},"min":{"1min":0.432,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70999999999999996,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.66400000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.5,"15min":0.498},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64200000000000002,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55300000000000005,"15min":0.57399999999999995},"min":{"1min":0.375,"5min":0.35599999999999998,"15min":0.34799999999999998},"max":{"1min":0.66800000000000004,"5min":0.77800000000000002,"15min":0.82099999999999995},"last":0.68899999999999995}]},{"osd":6,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.53800000000000003,"15min":0.55400000000000005},"min":{"1min":0.39200000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67400000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.57999999999999996},"min":{"1min":0.433,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67500000000000004,"5min":0.86299999999999999,"15min":0.88200000000000001},"last":0.62}]}]},{"osd":6,"up_from":13,"seq":55834576026,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60944,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310896,"statfs":{"total":96636764160,"available":96574357504,"internally_reserved":0,"allocated":3670016,"data_stored":1896305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.50900000000000001,"15min":0.54800000000000004},"min":{"1min":0.373,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.74199999999999999,"5min":1.244,"15min":1.244},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.53200000000000003,"15min":0.56100000000000005},"min":{"1min":0.42999999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.68799999999999994,"5min":1.304,"15min":1.304},"last":0.52000000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56100000000000005,"15min":0.60899999999999999},"min":{"1min":0.41799999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.85799999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.60799999999999998,"15min":0.63400000000000001},"min":{"1min":0.49399999999999999,"5min":0.40000000000000002,"15min":0.378},"max":{"1min":0.84299999999999997,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.65500000000000003}]},{"osd":2,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54400000000000004,"15min":0.58099999999999996},"min":{"1min":0.34999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69199999999999995,"5min":0.76700000000000002,"15min":0.82899999999999996},"last":0.5},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.55200000000000005,"15min":0.57899999999999996},"min":{"1min":0.47399999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80800000000000005,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.59399999999999997}]},{"osd":3,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.58999999999999997},"min":{"1min":0.41699999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.78400000000000003,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.60599999999999998,"15min":0.63500000000000001},"min":{"1min":0.501,"5min":0.39900000000000002,"15min":0.36099999999999999},"max":{"1min":0.79600000000000004,"5min":0.80200000000000005,"15min":0.90200000000000002},"last":0.71699999999999997}]},{"osd":4,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55300000000000005,"15min":0.53900000000000003},"min":{"1min":0.39900000000000002,"5min":0.308,"15min":0.308},"max":{"1min":0.72699999999999998,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.34200000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.73899999999999999,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.46300000000000002}]},{"osd":5,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.501,"15min":0.50700000000000001},"min":{"1min":0.34499999999999997,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.62,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54200000000000004,"15min":0.53600000000000003},"min":{"1min":0.32900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.77900000000000003,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":0.77700000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56299999999999994,"15min":0.55200000000000005},"min":{"1min":0.35499999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76700000000000002,"5min":1.603,"15min":1.603},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58399999999999996,"15min":0.57399999999999995},"min":{"1min":0.40500000000000003,"5min":0.33200000000000002,"15min":0.32800000000000001},"max":{"1min":0.80200000000000005,"5min":2.75,"15min":2.75},"last":0.75600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576025,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65036,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306804,"statfs":{"total":96636764160,"available":96570167296,"internally_reserved":0,"allocated":3670016,"data_stored":1896305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.371,"15min":0.38400000000000001},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65600000000000003,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.22},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40300000000000002,"15min":0.41999999999999998},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.67200000000000004,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":0.28899999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.38300000000000001,"15min":0.41099999999999998},"min":{"1min":0.33000000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.251},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.42899999999999999,"15min":0.45700000000000002},"min":{"1min":0.34100000000000003,"5min":0.25,"15min":0.25},"max":{"1min":0.69999999999999996,"5min":2.0169999999999999,"15min":2.0169999999999999},"last":0.26000000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47999999999999998},"min":{"1min":0.31,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61699999999999999,"5min":4.2699999999999996,"15min":4.2699999999999996},"last":0.317},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.41899999999999998,"15min":0.434},"min":{"1min":0.29299999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66600000000000004,"5min":2.153,"15min":2.153},"last":0.34699999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.58399999999999996,"15min":0.624},"min":{"1min":0.495,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.89900000000000002,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.59699999999999998,"15min":0.63},"min":{"1min":0.52700000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80500000000000005,"5min":0.877,"15min":0.995},"last":0.59899999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.60699999999999998,"15min":0.63200000000000001},"min":{"1min":0.50900000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84499999999999997,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.64700000000000002,"15min":0.66100000000000003},"min":{"1min":0.51100000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.88600000000000001,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.68799999999999994}]},{"osd":6,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.61299999999999999,"15min":0.63800000000000001},"min":{"1min":0.52000000000000002,"5min":0.42599999999999999,"15min":0.40799999999999997},"max":{"1min":0.872,"5min":0.89200000000000002,"15min":0.94399999999999995},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.65500000000000003,"15min":0.67000000000000004},"min":{"1min":0.47199999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.91700000000000004,"5min":4.2889999999999997,"15min":4.2889999999999997},"last":0.65600000000000003}]},{"osd":7,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67700000000000005,"15min":0.68899999999999995},"min":{"1min":0.52200000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.86699999999999999,"5min":4.0730000000000004,"15min":4.0730000000000004},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.627,"15min":0.65200000000000002},"min":{"1min":0.46400000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.94199999999999995,"5min":2.194,"15min":2.194},"last":0.625}]}]},{"osd":0,"up_from":13,"seq":55834576025,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61520,"kb_used_data":4164,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310320,"statfs":{"total":96636764160,"available":96573767680,"internally_reserved":0,"allocated":4263936,"data_stored":2486673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.36099999999999999,"15min":0.38100000000000001},"min":{"1min":0.27200000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55800000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.375,"15min":0.40699999999999997},"min":{"1min":0.28499999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.70799999999999996},"last":0.439}]},{"osd":2,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.38700000000000001,"15min":0.41599999999999998},"min":{"1min":0.316,"5min":0.26100000000000001,"15min":0.23899999999999999},"max":{"1min":0.50600000000000001,"5min":0.55900000000000005,"15min":0.70099999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.41499999999999998,"15min":0.435},"min":{"1min":0.33500000000000002,"5min":0.28399999999999997,"15min":0.27100000000000002},"max":{"1min":0.627,"5min":0.627,"15min":0.70499999999999996},"last":0.498}]},{"osd":3,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.38200000000000001,"15min":0.41299999999999998},"min":{"1min":0.28599999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54300000000000004,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.435,"15min":0.46600000000000003},"min":{"1min":0.38500000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.59699999999999998,"5min":0.64100000000000001,"15min":0.75700000000000001},"last":0.56100000000000005}]},{"osd":4,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58999999999999997,"15min":0.626},"min":{"1min":0.45400000000000001,"5min":0.38500000000000001,"15min":0.37},"max":{"1min":0.82399999999999995,"5min":0.93100000000000005,"15min":0.93400000000000005},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59399999999999997,"15min":0.63200000000000001},"min":{"1min":0.42999999999999999,"5min":0.372,"15min":0.36399999999999999},"max":{"1min":0.83599999999999997,"5min":0.999,"15min":0.999},"last":0.69399999999999995}]},{"osd":5,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.57899999999999996,"15min":0.622},"min":{"1min":0.41699999999999998,"5min":0.373,"15min":0.33000000000000002},"max":{"1min":0.80600000000000005,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.623,"15min":0.65400000000000003},"min":{"1min":0.45800000000000002,"5min":0.41399999999999998,"15min":0.378},"max":{"1min":0.872,"5min":0.89500000000000002,"15min":0.96799999999999997},"last":0.76000000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.623,"15min":0.65900000000000003},"min":{"1min":0.42699999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85999999999999999,"5min":0.94399999999999995,"15min":1.0269999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.61599999999999999,"15min":0.65500000000000003},"min":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.39000000000000001},"max":{"1min":0.85699999999999998,"5min":0.93600000000000005,"15min":0.96599999999999997},"last":0.60999999999999999}]},{"osd":7,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.63500000000000001,"15min":0.67600000000000005},"min":{"1min":0.47699999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81000000000000005,"5min":0.91200000000000003,"15min":1.01},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.63200000000000001,"15min":0.66000000000000003},"min":{"1min":0.44600000000000001,"5min":0.42199999999999999,"15min":0.36099999999999999},"max":{"1min":0.88700000000000001,"5min":0.95799999999999996,"15min":0.97699999999999998},"last":0.748}]}]},{"osd":2,"up_from":13,"seq":55834576027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65024,"kb_used_data":3572,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306816,"statfs":{"total":96636764160,"available":96570179584,"internally_reserved":0,"allocated":3657728,"data_stored":1889689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.374},"min":{"1min":0.24199999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.54600000000000004,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.39600000000000002,"15min":0.41399999999999998},"min":{"1min":0.31900000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.66400000000000003,"5min":0.71499999999999997,"15min":1.044},"last":0.377}]},{"osd":1,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.36399999999999999,"15min":0.39300000000000002},"min":{"1min":0.25600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.753},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.441,"15min":0.44600000000000001},"min":{"1min":0.34999999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.60299999999999998,"5min":3.3999999999999999,"15min":3.3999999999999999},"last":0.26100000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.40899999999999997,"15min":0.44600000000000001},"min":{"1min":0.30199999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.70299999999999996},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.41099999999999998,"15min":0.436},"min":{"1min":0.29299999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.69599999999999995,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.34200000000000003}]},{"osd":4,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59199999999999997,"15min":0.626},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.88800000000000001,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.60899999999999999,"15min":0.64600000000000002},"min":{"1min":0.42299999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86199999999999999,"5min":1.022,"15min":1.022},"last":0.60199999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60299999999999998,"15min":0.64800000000000002},"min":{"1min":0.40400000000000003,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80100000000000005,"5min":0.98599999999999999,"15min":1.0149999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.627,"5min":0.61799999999999999,"15min":0.65200000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84699999999999998,"5min":1.028,"15min":1.028},"last":0.54600000000000004}]},{"osd":6,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.63400000000000001,"15min":0.65800000000000003},"min":{"1min":0.51100000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92400000000000004,"5min":0.97199999999999998,"15min":1.091},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.64000000000000001,"15min":0.67900000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91200000000000003,"5min":0.95699999999999996,"15min":0.96099999999999997},"last":0.626}]},{"osd":7,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.64900000000000002,"15min":0.67500000000000004},"min":{"1min":0.435,"5min":0.42299999999999999,"15min":0.41599999999999998},"max":{"1min":0.93899999999999995,"5min":0.94499999999999995,"15min":0.95399999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.65600000000000003,"15min":0.68999999999999995},"min":{"1min":0.44700000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.94899999999999995,"5min":1.01,"15min":1.01},"last":0.67000000000000004}]}]},{"osd":3,"up_from":13,"seq":55834576026,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52756,"kb_used_data":3588,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319084,"statfs":{"total":96636764160,"available":96582742016,"internally_reserved":0,"allocated":3674112,"data_stored":1897324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.36299999999999999,"15min":0.376},"min":{"1min":0.255,"5min":0.223,"15min":0.223},"max":{"1min":0.53400000000000003,"5min":0.55200000000000005,"15min":0.72799999999999998},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.36799999999999999,"15min":0.39700000000000002},"min":{"1min":0.32000000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.63800000000000001},"last":0.34300000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.36699999999999999,"15min":0.39500000000000002},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.309},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.40600000000000003,"15min":0.42799999999999999},"min":{"1min":0.29299999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.60299999999999998,"5min":0.63,"15min":0.75800000000000001},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.434,"15min":0.46899999999999997},"min":{"1min":0.39200000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.67000000000000004,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.44},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.45700000000000002,"15min":0.48399999999999999},"min":{"1min":0.38700000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65200000000000002,"5min":0.67800000000000005,"15min":0.79300000000000004},"last":0.37}]},{"osd":4,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.59099999999999997,"15min":0.61899999999999999},"min":{"1min":0.42299999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":1.018},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.59999999999999998,"15min":0.63300000000000001},"min":{"1min":0.44900000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.0169999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.495}]},{"osd":5,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.64200000000000002,"15min":0.66700000000000004},"min":{"1min":0.47099999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.65200000000000002,"15min":0.67900000000000005},"min":{"1min":0.501,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.94399999999999995,"5min":1.036,"15min":1.036},"last":0.63300000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.626,"15min":0.64900000000000002},"min":{"1min":0.47499999999999998,"5min":0.31,"15min":0.31},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.63,"15min":0.66300000000000003},"min":{"1min":0.433,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.60199999999999998}]},{"osd":7,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.628,"15min":0.65900000000000003},"min":{"1min":0.46000000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90500000000000003,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.64500000000000002,"15min":0.68100000000000005},"min":{"1min":0.53000000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.61899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576026,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52756,"kb_used_data":3588,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319084,"statfs":{"total":96636764160,"available":96582742016,"internally_reserved":0,"allocated":3674112,"data_stored":1897324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54800000000000004,"15min":0.56999999999999995},"min":{"1min":0.38400000000000001,"5min":0.30499999999999999,"15min":0.26300000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.876},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.47899999999999998,"5min":0.36599999999999999,"15min":0.34499999999999997},"max":{"1min":0.77400000000000002,"5min":0.81100000000000005,"15min":1.202},"last":0.72899999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.60899999999999999,"15min":0.628},"min":{"1min":0.46200000000000002,"5min":0.379,"15min":0.35099999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":1.256},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.60299999999999998,"15min":0.61799999999999999},"min":{"1min":0.49199999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.89700000000000002},"last":0.70299999999999996}]},{"osd":2,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56200000000000006,"15min":0.58399999999999996},"min":{"1min":0.438,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80100000000000005,"5min":0.81799999999999995,"15min":0.84599999999999997},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59199999999999997,"15min":0.60999999999999999},"min":{"1min":0.46700000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70799999999999996,"5min":0.97899999999999998,"15min":1.294},"last":0.75800000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56599999999999995,"15min":0.58399999999999996},"min":{"1min":0.498,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.94299999999999995},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.59399999999999997,"15min":0.61599999999999999},"min":{"1min":0.47099999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.90200000000000002},"last":0.71499999999999997}]},{"osd":5,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.50900000000000001,"15min":0.52000000000000002},"min":{"1min":0.35599999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.64100000000000001,"5min":0.92800000000000005,"15min":5.0119999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.54200000000000004},"min":{"1min":0.35199999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68600000000000005,"5min":0.95199999999999996,"15min":4.931},"last":0.52100000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52900000000000003,"15min":0.53200000000000003},"min":{"1min":0.35299999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.77800000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.497},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.40300000000000002,"5min":0.35799999999999998,"15min":0.34699999999999998},"max":{"1min":0.71199999999999997,"5min":1.0609999999999999,"15min":1.3580000000000001},"last":0.63700000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.54900000000000004,"15min":0.57999999999999996},"min":{"1min":0.378,"5min":0.35899999999999999,"15min":0.34000000000000002},"max":{"1min":0.66600000000000004,"5min":0.91000000000000003,"15min":4.9889999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58599999999999997,"15min":0.59899999999999998},"min":{"1min":0.40899999999999997,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72699999999999998,"5min":0.86899999999999999,"15min":0.88100000000000001},"last":0.60199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834576026,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":66704,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305136,"statfs":{"total":96636764160,"available":96568459264,"internally_reserved":0,"allocated":3670016,"data_stored":1896305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54300000000000004,"15min":0.57499999999999996},"min":{"1min":0.438,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.82199999999999995},"last":0.627},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54200000000000004,"15min":0.57699999999999996},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.28100000000000003},"max":{"1min":0.73999999999999999,"5min":0.76600000000000001,"15min":0.84699999999999998},"last":0.58099999999999996}]},{"osd":1,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54400000000000004,"15min":0.57699999999999996},"min":{"1min":0.437,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.81299999999999994},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54300000000000004,"15min":0.57499999999999996},"min":{"1min":0.42799999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69199999999999995,"5min":0.752,"15min":0.871},"last":0.48699999999999999}]},{"osd":2,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.55600000000000005,"15min":0.58699999999999997},"min":{"1min":0.46800000000000003,"5min":0.311,"15min":0.30599999999999999},"max":{"1min":0.72099999999999997,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.56200000000000006,"15min":0.58799999999999997},"min":{"1min":0.44600000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.745,"5min":0.745,"15min":0.83399999999999996},"last":0.54700000000000004}]},{"osd":3,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.55600000000000005,"15min":0.58699999999999997},"min":{"1min":0.40300000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77800000000000002,"5min":0.79800000000000004,"15min":0.81100000000000005},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58399999999999996,"15min":0.621},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.371},"max":{"1min":0.76100000000000001,"5min":0.81399999999999995,"15min":0.88200000000000001},"last":0.76400000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51000000000000001,"15min":0.50700000000000001},"min":{"1min":0.32300000000000001,"5min":0.29899999999999999,"15min":0.27400000000000002},"max":{"1min":0.70999999999999996,"5min":0.75900000000000001,"15min":0.85699999999999998},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57199999999999995,"15min":0.57999999999999996},"min":{"1min":0.34799999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.73999999999999999,"5min":0.82999999999999996,"15min":0.88800000000000001},"last":0.83399999999999996}]},{"osd":6,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35799999999999998,"5min":0.313,"15min":0.28899999999999998},"max":{"1min":0.75700000000000001,"5min":0.89900000000000002,"15min":0.94999999999999996},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52900000000000003,"15min":0.53200000000000003},"min":{"1min":0.372,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.76100000000000001,"5min":0.84299999999999997,"15min":0.878},"last":0.81799999999999995}]},{"osd":7,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.39100000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.76800000000000002,"5min":0.85599999999999998,"15min":0.90800000000000003},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.63800000000000001,"15min":0.64800000000000002},"min":{"1min":0.47399999999999998,"5min":0.41699999999999998,"15min":0.38300000000000001},"max":{"1min":0.79000000000000004,"5min":0.87,"15min":0.92300000000000004},"last":0.90400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:51:39.276 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-06T13:51:39.276 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:51:39.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.397+0000 7efcfb7cd700 1 Processor -- start 2024-04-06T13:51:39.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.397+0000 7efcfb7cd700 1 -- start start 2024-04-06T13:51:39.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcfb7cd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efcf40cfa00 0x7efcf40cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcfb7cd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf40cbd80 con 0x7efcf40cb870 2024-04-06T13:51:39.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcfb7cd700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf40cef20 con 0x7efcf40d11b0 2024-04-06T13:51:39.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcfb7cd700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcf40ccdc0 con 0x7efcf40cfa00 2024-04-06T13:51:39.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcf8d68700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efcf40cb870 0x7efcf415c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:32846/0 (socket says 172.21.15.189:32846) 2024-04-06T13:51:39.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcf8d68700 1 -- 172.21.15.189:0/4268400740 learned_addr learned my addr 172.21.15.189:0/4268400740 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:39.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcf9d6a700 1 --2- 172.21.15.189:0/4268400740 >> v2:172.21.15.192:3300/0 conn(0x7efcf40cfa00 0x7efcf40cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcebfff700 1 -- 172.21.15.189:0/4268400740 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137574020 0 0) 0x7efcf40cbd80 con 0x7efcf40cb870 2024-04-06T13:51:39.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcebfff700 1 -- 172.21.15.189:0/4268400740 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efce00035b0 con 0x7efcf40cb870 2024-04-06T13:51:39.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcebfff700 1 -- 172.21.15.189:0/4268400740 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214824416 0 0) 0x7efcf40cef20 con 0x7efcf40d11b0 2024-04-06T13:51:39.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcebfff700 1 -- 172.21.15.189:0/4268400740 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcf40cbd80 con 0x7efcf40d11b0 2024-04-06T13:51:39.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcebfff700 1 -- 172.21.15.189:0/4268400740 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3408940184 0 0) 0x7efce00035b0 con 0x7efcf40cb870 2024-04-06T13:51:39.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcebfff700 1 -- 172.21.15.189:0/4268400740 >> v2:172.21.15.192:3300/0 conn(0x7efcf40cfa00 msgr2=0x7efcf40cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcebfff700 1 --2- 172.21.15.189:0/4268400740 >> v2:172.21.15.192:3300/0 conn(0x7efcf40cfa00 0x7efcf40cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcebfff700 1 -- 172.21.15.189:0/4268400740 >> v1:172.21.15.192:6789/0 conn(0x7efcf40d11b0 legacy=0x7efcf40d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcebfff700 1 -- 172.21.15.189:0/4268400740 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf40cf200 con 0x7efcf40cb870 2024-04-06T13:51:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcebfff700 1 -- 172.21.15.189:0/4268400740 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7efce4001670 con 0x7efcf40cb870 2024-04-06T13:51:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcebfff700 1 -- 172.21.15.189:0/4268400740 >> v1:172.21.15.189:6789/0 conn(0x7efcf40cb870 legacy=0x7efcf415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcebfff700 1 --2- 172.21.15.189:0/4268400740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efce0003be0 0x7efce0003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcebfff700 1 --2- 172.21.15.189:0/4268400740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efce0004850 0x7efce0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcebfff700 1 --2- 172.21.15.189:0/4268400740 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efce0005260 0x7efce000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcebfff700 1 -- 172.21.15.189:0/4268400740 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce000c160 con 0x7efce0003be0 2024-04-06T13:51:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcebfff700 1 -- 172.21.15.189:0/4268400740 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce000c2d0 con 0x7efce0004850 2024-04-06T13:51:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.398+0000 7efcebfff700 1 -- 172.21.15.189:0/4268400740 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce000c490 con 0x7efce0005260 2024-04-06T13:51:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.399+0000 7efcf9569700 1 --2- 172.21.15.189:0/4268400740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efce0003be0 0x7efce0003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.399+0000 7efcf8d68700 1 --2- 172.21.15.189:0/4268400740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efce0004850 0x7efce0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.399+0000 7efcf9d6a700 1 --2- 172.21.15.189:0/4268400740 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efce0005260 0x7efce000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.399+0000 7efcf9569700 1 -- 172.21.15.189:0/4268400740 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efce0005260 msgr2=0x7efce000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.399+0000 7efcf9569700 1 --2- 172.21.15.189:0/4268400740 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efce0005260 0x7efce000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.399+0000 7efcf9569700 1 -- 172.21.15.189:0/4268400740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efce0004850 msgr2=0x7efce0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:39.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.399+0000 7efcf9569700 1 --2- 172.21.15.189:0/4268400740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efce0004850 0x7efce0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.399+0000 7efcf9569700 1 -- 172.21.15.189:0/4268400740 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce00035b0 con 0x7efce0003be0 2024-04-06T13:51:39.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.399+0000 7efcf9d6a700 1 -- 172.21.15.189:0/4268400740 reap_dead start 2024-04-06T13:51:39.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.399+0000 7efcf9569700 1 --2- 172.21.15.189:0/4268400740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efce0003be0 0x7efce0003fe0 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7efcf0001940 tx=0x7efce0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3956c7147af5c78a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:39.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.399+0000 7efcebfff700 1 -- 172.21.15.189:0/4268400740 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf0001430 con 0x7efce0003be0 2024-04-06T13:51:39.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.399+0000 7efcebfff700 1 -- 172.21.15.189:0/4268400740 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf0001d70 con 0x7efce0003be0 2024-04-06T13:51:39.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.399+0000 7efcebfff700 1 -- 172.21.15.189:0/4268400740 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf0005510 con 0x7efce0003be0 2024-04-06T13:51:39.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.400+0000 7efcfb7cd700 1 -- 172.21.15.189:0/4268400740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efce0003be0 msgr2=0x7efce0003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:39.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.400+0000 7efcfb7cd700 1 --2- 172.21.15.189:0/4268400740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efce0003be0 0x7efce0003fe0 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7efcf0001940 tx=0x7efce0005880 comp rx=0 tx=0).stop 2024-04-06T13:51:39.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.400+0000 7efcfb7cd700 1 -- 172.21.15.189:0/4268400740 shutdown_connections 2024-04-06T13:51:39.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.400+0000 7efcfb7cd700 1 --2- 172.21.15.189:0/4268400740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efce0003be0 0x7efce0003fe0 unknown :-1 s=CLOSED pgs=3211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.400+0000 7efcfb7cd700 1 -- 172.21.15.189:0/4268400740 >> 172.21.15.189:0/4268400740 conn(0x7efcf40b2db0 msgr2=0x7efcf40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:39.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.400+0000 7efcfb7cd700 1 -- 172.21.15.189:0/4268400740 shutdown_connections 2024-04-06T13:51:39.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.400+0000 7efcfb7cd700 1 -- 172.21.15.189:0/4268400740 wait complete. 2024-04-06T13:51:39.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.401+0000 7efcfb7cd700 1 Processor -- start 2024-04-06T13:51:39.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.401+0000 7efcfb7cd700 1 -- start start 2024-04-06T13:51:39.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efcfb7cd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efce0003be0 0x7efcf406c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efcfb7cd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcf415eef0 0x7efcf41547f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efcfb7cd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcf4154fa0 0x7efcf415a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efcfb7cd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf406b840 con 0x7efce0003be0 2024-04-06T13:51:39.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efcfb7cd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf406d470 con 0x7efcf4154fa0 2024-04-06T13:51:39.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efcfb7cd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf406d750 con 0x7efcf415eef0 2024-04-06T13:51:39.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efcf9569700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efce0003be0 0x7efcf406c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efcf9569700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efce0003be0 0x7efcf406c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44198/0 (socket says 172.21.15.189:44198) 2024-04-06T13:51:39.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efcf9569700 1 -- 172.21.15.189:0/3344532215 learned_addr learned my addr 172.21.15.189:0/3344532215 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:39.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efcf8d68700 1 --2- 172.21.15.189:0/3344532215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcf415eef0 0x7efcf41547f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efcf9d6a700 1 --2- 172.21.15.189:0/3344532215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcf4154fa0 0x7efcf415a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efcf9569700 1 -- 172.21.15.189:0/3344532215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcf415eef0 msgr2=0x7efcf41547f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:39.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efcf9569700 1 --2- 172.21.15.189:0/3344532215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcf415eef0 0x7efcf41547f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efcf9569700 1 -- 172.21.15.189:0/3344532215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcf4154fa0 msgr2=0x7efcf415a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:39.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efcf9569700 1 --2- 172.21.15.189:0/3344532215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcf4154fa0 0x7efcf415a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efcf9569700 1 -- 172.21.15.189:0/3344532215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf406bb00 con 0x7efce0003be0 2024-04-06T13:51:39.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efcf9569700 1 --2- 172.21.15.189:0/3344532215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efce0003be0 0x7efcf406c300 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7efce000c2a0 tx=0x7efcf00012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:39.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efce9ffb700 1 -- 172.21.15.189:0/3344532215 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf00013d0 con 0x7efce0003be0 2024-04-06T13:51:39.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efce9ffb700 1 -- 172.21.15.189:0/3344532215 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf0006a70 con 0x7efce0003be0 2024-04-06T13:51:39.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efcfb7cd700 1 -- 172.21.15.189:0/3344532215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf406bde0 con 0x7efce0003be0 2024-04-06T13:51:39.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.402+0000 7efce9ffb700 1 -- 172.21.15.189:0/3344532215 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf0006e20 con 0x7efce0003be0 2024-04-06T13:51:39.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.403+0000 7efcfb7cd700 1 -- 172.21.15.189:0/3344532215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf406d090 con 0x7efce0003be0 2024-04-06T13:51:39.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.403+0000 7efcfb7cd700 1 -- 172.21.15.189:0/3344532215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf4074990 con 0x7efce0003be0 2024-04-06T13:51:39.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.404+0000 7efce9ffb700 1 -- 172.21.15.189:0/3344532215 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7efcf0007040 con 0x7efce0003be0 2024-04-06T13:51:39.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.404+0000 7efce9ffb700 1 --2- 172.21.15.189:0/3344532215 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efcd806ade0 0x7efcd806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.404+0000 7efce9ffb700 1 -- 172.21.15.189:0/3344532215 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7efcf0089f40 con 0x7efce0003be0 2024-04-06T13:51:39.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.404+0000 7efcf8d68700 1 --2- 172.21.15.189:0/3344532215 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efcd806ade0 0x7efcd806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.405+0000 7efcf8d68700 1 --2- 172.21.15.189:0/3344532215 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efcd806ade0 0x7efcd806d2a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efcf415f330 tx=0x7efce4001020 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:39.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.409+0000 7efce9ffb700 1 -- 172.21.15.189:0/3344532215 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efcf0054eb0 con 0x7efce0003be0 2024-04-06T13:51:39.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.598+0000 7efcfb7cd700 1 -- 172.21.15.189:0/3344532215 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efcf40cc060 con 0x7efcd806ade0 2024-04-06T13:51:39.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.600+0000 7efce9ffb700 1 -- 172.21.15.189:0/3344532215 <== mgr.10415 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51096 (secure 0 0 0) 0x7efce4007e50 con 0x7efcd806ade0 2024-04-06T13:51:39.601 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:51:39.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.603+0000 7efcfb7cd700 1 -- 172.21.15.189:0/3344532215 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efcd806ade0 msgr2=0x7efcd806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:39.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.603+0000 7efcfb7cd700 1 --2- 172.21.15.189:0/3344532215 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efcd806ade0 0x7efcd806d2a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efcf415f330 tx=0x7efce4001020 comp rx=0 tx=0).stop 2024-04-06T13:51:39.603 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.603+0000 7efcfb7cd700 1 -- 172.21.15.189:0/3344532215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efce0003be0 msgr2=0x7efcf406c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:39.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.603+0000 7efcfb7cd700 1 --2- 172.21.15.189:0/3344532215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efce0003be0 0x7efcf406c300 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7efce000c2a0 tx=0x7efcf00012d0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.603+0000 7efcfb7cd700 1 -- 172.21.15.189:0/3344532215 shutdown_connections 2024-04-06T13:51:39.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.603+0000 7efcfb7cd700 1 --2- 172.21.15.189:0/3344532215 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7efcd806ade0 0x7efcd806d2a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.603+0000 7efcfb7cd700 1 --2- 172.21.15.189:0/3344532215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcf4154fa0 0x7efcf415a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.603+0000 7efcfb7cd700 1 --2- 172.21.15.189:0/3344532215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efce0003be0 0x7efcf406c300 unknown :-1 s=CLOSED pgs=3212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.603+0000 7efcfb7cd700 1 --2- 172.21.15.189:0/3344532215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efcf415eef0 0x7efcf41547f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.603+0000 7efcfb7cd700 1 -- 172.21.15.189:0/3344532215 >> 172.21.15.189:0/3344532215 conn(0x7efcf40b2db0 msgr2=0x7efcf40ce010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:39.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.603+0000 7efcfb7cd700 1 -- 172.21.15.189:0/3344532215 shutdown_connections 2024-04-06T13:51:39.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.603+0000 7efcfb7cd700 1 -- 172.21.15.189:0/3344532215 wait complete. 2024-04-06T13:51:39.605 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:51:39.615 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-06T13:51:38.511306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":133,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":482168,"kb_used_data":29832,"kb_used_omap":12,"kb_used_meta":452211,"kb_avail":754492552,"statfs":{"total":773094113280,"available":772600373248,"internally_reserved":0,"allocated":30547968,"data_stored":16347617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":463064632},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-766700,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-200,"num_read_kb":-56,"num_write":-292,"num_write_kb":-2300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-123,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.008272"},"pg_stats":[{"pgid":"1.0","version":"288'425","reported_seq":1575,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-06T13:51:33.507521+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:51:33.507521+0000","last_peered":"2024-04-06T13:51:33.507521+0000","last_clean":"2024-04-06T13:51:33.507521+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:51:33.507521+0000","last_undegraded":"2024-04-06T13:51:33.507521+0000","last_fullsized":"2024-04-06T13:51:33.507521+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576027,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57428,"kb_used_data":4168,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314412,"statfs":{"total":96636764160,"available":96577957888,"internally_reserved":0,"allocated":4268032,"data_stored":2487692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55500000000000005,"15min":0.58799999999999997},"min":{"1min":0.34499999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.82499999999999996},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53200000000000003,"15min":0.57399999999999995},"min":{"1min":0.47499999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.76200000000000001,"5min":0.78500000000000003,"15min":0.82899999999999996},"last":0.60099999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.53900000000000003,"15min":0.57099999999999995},"min":{"1min":0.39700000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.748},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.54400000000000004,"15min":0.58399999999999996},"min":{"1min":0.41999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.85299999999999998},"last":0.51900000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.52100000000000002,"15min":0.56999999999999995},"min":{"1min":0.36399999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.83899999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56100000000000005,"15min":0.59199999999999997},"min":{"1min":0.45000000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76800000000000002,"5min":0.76900000000000002,"15min":0.79700000000000004},"last":0.67900000000000005}]},{"osd":3,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54600000000000004,"15min":0.58999999999999997},"min":{"1min":0.39700000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.75,"5min":0.75600000000000001,"15min":0.81100000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.622,"5min":0.57299999999999995,"15min":0.59599999999999997},"min":{"1min":0.497,"5min":0.39000000000000001,"15min":0.34999999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.84099999999999997},"last":0.64800000000000002}]},{"osd":4,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45400000000000001,"15min":0.46300000000000002},"min":{"1min":0.32400000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.60999999999999999,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.54100000000000004,"15min":0.54300000000000004},"min":{"1min":0.432,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70999999999999996,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.66400000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.5,"15min":0.498},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64200000000000002,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55300000000000005,"15min":0.57399999999999995},"min":{"1min":0.375,"5min":0.35599999999999998,"15min":0.34799999999999998},"max":{"1min":0.66800000000000004,"5min":0.77800000000000002,"15min":0.82099999999999995},"last":0.68899999999999995}]},{"osd":6,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.53800000000000003,"15min":0.55400000000000005},"min":{"1min":0.39200000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67400000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.57999999999999996},"min":{"1min":0.433,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67500000000000004,"5min":0.86299999999999999,"15min":0.88200000000000001},"last":0.62}]}]},{"osd":6,"up_from":13,"seq":55834576026,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60944,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310896,"statfs":{"total":96636764160,"available":96574357504,"internally_reserved":0,"allocated":3670016,"data_stored":1896305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.50900000000000001,"15min":0.54800000000000004},"min":{"1min":0.373,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.74199999999999999,"5min":1.244,"15min":1.244},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.53200000000000003,"15min":0.56100000000000005},"min":{"1min":0.42999999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.68799999999999994,"5min":1.304,"15min":1.304},"last":0.52000000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56100000000000005,"15min":0.60899999999999999},"min":{"1min":0.41799999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.85799999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.60799999999999998,"15min":0.63400000000000001},"min":{"1min":0.49399999999999999,"5min":0.40000000000000002,"15min":0.378},"max":{"1min":0.84299999999999997,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.65500000000000003}]},{"osd":2,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54400000000000004,"15min":0.58099999999999996},"min":{"1min":0.34999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69199999999999995,"5min":0.76700000000000002,"15min":0.82899999999999996},"last":0.5},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.55200000000000005,"15min":0.57899999999999996},"min":{"1min":0.47399999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80800000000000005,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.59399999999999997}]},{"osd":3,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.58999999999999997},"min":{"1min":0.41699999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.78400000000000003,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.60599999999999998,"15min":0.63500000000000001},"min":{"1min":0.501,"5min":0.39900000000000002,"15min":0.36099999999999999},"max":{"1min":0.79600000000000004,"5min":0.80200000000000005,"15min":0.90200000000000002},"last":0.71699999999999997}]},{"osd":4,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55300000000000005,"15min":0.53900000000000003},"min":{"1min":0.39900000000000002,"5min":0.308,"15min":0.308},"max":{"1min":0.72699999999999998,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.34200000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.73899999999999999,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.46300000000000002}]},{"osd":5,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.501,"15min":0.50700000000000001},"min":{"1min":0.34499999999999997,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.62,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54200000000000004,"15min":0.53600000000000003},"min":{"1min":0.32900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.77900000000000003,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":0.77700000000000002}]},{"osd":7,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56299999999999994,"15min":0.55200000000000005},"min":{"1min":0.35499999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76700000000000002,"5min":1.603,"15min":1.603},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58399999999999996,"15min":0.57399999999999995},"min":{"1min":0.40500000000000003,"5min":0.33200000000000002,"15min":0.32800000000000001},"max":{"1min":0.80200000000000005,"5min":2.75,"15min":2.75},"last":0.75600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576025,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65036,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306804,"statfs":{"total":96636764160,"available":96570167296,"internally_reserved":0,"allocated":3670016,"data_stored":1896305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.371,"15min":0.38400000000000001},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65600000000000003,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.22},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40300000000000002,"15min":0.41999999999999998},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.67200000000000004,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":0.28899999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.38300000000000001,"15min":0.41099999999999998},"min":{"1min":0.33000000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.251},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.42899999999999999,"15min":0.45700000000000002},"min":{"1min":0.34100000000000003,"5min":0.25,"15min":0.25},"max":{"1min":0.69999999999999996,"5min":2.0169999999999999,"15min":2.0169999999999999},"last":0.26000000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47999999999999998},"min":{"1min":0.31,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61699999999999999,"5min":4.2699999999999996,"15min":4.2699999999999996},"last":0.317},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.41899999999999998,"15min":0.434},"min":{"1min":0.29299999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66600000000000004,"5min":2.153,"15min":2.153},"last":0.34699999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.58399999999999996,"15min":0.624},"min":{"1min":0.495,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.89900000000000002,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.59699999999999998,"15min":0.63},"min":{"1min":0.52700000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80500000000000005,"5min":0.877,"15min":0.995},"last":0.59899999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.60699999999999998,"15min":0.63200000000000001},"min":{"1min":0.50900000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84499999999999997,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.64700000000000002,"15min":0.66100000000000003},"min":{"1min":0.51100000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.88600000000000001,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.68799999999999994}]},{"osd":6,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.61299999999999999,"15min":0.63800000000000001},"min":{"1min":0.52000000000000002,"5min":0.42599999999999999,"15min":0.40799999999999997},"max":{"1min":0.872,"5min":0.89200000000000002,"15min":0.94399999999999995},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.65500000000000003,"15min":0.67000000000000004},"min":{"1min":0.47199999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.91700000000000004,"5min":4.2889999999999997,"15min":4.2889999999999997},"last":0.65600000000000003}]},{"osd":7,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67700000000000005,"15min":0.68899999999999995},"min":{"1min":0.52200000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.86699999999999999,"5min":4.0730000000000004,"15min":4.0730000000000004},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.627,"15min":0.65200000000000002},"min":{"1min":0.46400000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.94199999999999995,"5min":2.194,"15min":2.194},"last":0.625}]}]},{"osd":0,"up_from":13,"seq":55834576025,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61520,"kb_used_data":4164,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310320,"statfs":{"total":96636764160,"available":96573767680,"internally_reserved":0,"allocated":4263936,"data_stored":2486673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.36099999999999999,"15min":0.38100000000000001},"min":{"1min":0.27200000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55800000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.375,"15min":0.40699999999999997},"min":{"1min":0.28499999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.70799999999999996},"last":0.439}]},{"osd":2,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.38700000000000001,"15min":0.41599999999999998},"min":{"1min":0.316,"5min":0.26100000000000001,"15min":0.23899999999999999},"max":{"1min":0.50600000000000001,"5min":0.55900000000000005,"15min":0.70099999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.41499999999999998,"15min":0.435},"min":{"1min":0.33500000000000002,"5min":0.28399999999999997,"15min":0.27100000000000002},"max":{"1min":0.627,"5min":0.627,"15min":0.70499999999999996},"last":0.498}]},{"osd":3,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.38200000000000001,"15min":0.41299999999999998},"min":{"1min":0.28599999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54300000000000004,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.435,"15min":0.46600000000000003},"min":{"1min":0.38500000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.59699999999999998,"5min":0.64100000000000001,"15min":0.75700000000000001},"last":0.56100000000000005}]},{"osd":4,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58999999999999997,"15min":0.626},"min":{"1min":0.45400000000000001,"5min":0.38500000000000001,"15min":0.37},"max":{"1min":0.82399999999999995,"5min":0.93100000000000005,"15min":0.93400000000000005},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59399999999999997,"15min":0.63200000000000001},"min":{"1min":0.42999999999999999,"5min":0.372,"15min":0.36399999999999999},"max":{"1min":0.83599999999999997,"5min":0.999,"15min":0.999},"last":0.69399999999999995}]},{"osd":5,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.57899999999999996,"15min":0.622},"min":{"1min":0.41699999999999998,"5min":0.373,"15min":0.33000000000000002},"max":{"1min":0.80600000000000005,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.623,"15min":0.65400000000000003},"min":{"1min":0.45800000000000002,"5min":0.41399999999999998,"15min":0.378},"max":{"1min":0.872,"5min":0.89500000000000002,"15min":0.96799999999999997},"last":0.76000000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.623,"15min":0.65900000000000003},"min":{"1min":0.42699999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85999999999999999,"5min":0.94399999999999995,"15min":1.0269999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.61599999999999999,"15min":0.65500000000000003},"min":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.39000000000000001},"max":{"1min":0.85699999999999998,"5min":0.93600000000000005,"15min":0.96599999999999997},"last":0.60999999999999999}]},{"osd":7,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.63500000000000001,"15min":0.67600000000000005},"min":{"1min":0.47699999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81000000000000005,"5min":0.91200000000000003,"15min":1.01},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.63200000000000001,"15min":0.66000000000000003},"min":{"1min":0.44600000000000001,"5min":0.42199999999999999,"15min":0.36099999999999999},"max":{"1min":0.88700000000000001,"5min":0.95799999999999996,"15min":0.97699999999999998},"last":0.748}]}]},{"osd":2,"up_from":13,"seq":55834576027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65024,"kb_used_data":3572,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306816,"statfs":{"total":96636764160,"available":96570179584,"internally_reserved":0,"allocated":3657728,"data_stored":1889689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.374},"min":{"1min":0.24199999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.54600000000000004,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.39600000000000002,"15min":0.41399999999999998},"min":{"1min":0.31900000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.66400000000000003,"5min":0.71499999999999997,"15min":1.044},"last":0.377}]},{"osd":1,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.36399999999999999,"15min":0.39300000000000002},"min":{"1min":0.25600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.753},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.441,"15min":0.44600000000000001},"min":{"1min":0.34999999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.60299999999999998,"5min":3.3999999999999999,"15min":3.3999999999999999},"last":0.26100000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.40899999999999997,"15min":0.44600000000000001},"min":{"1min":0.30199999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.70299999999999996},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.41099999999999998,"15min":0.436},"min":{"1min":0.29299999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.69599999999999995,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.34200000000000003}]},{"osd":4,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59199999999999997,"15min":0.626},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.88800000000000001,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.60899999999999999,"15min":0.64600000000000002},"min":{"1min":0.42299999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86199999999999999,"5min":1.022,"15min":1.022},"last":0.60199999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60299999999999998,"15min":0.64800000000000002},"min":{"1min":0.40400000000000003,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80100000000000005,"5min":0.98599999999999999,"15min":1.0149999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.627,"5min":0.61799999999999999,"15min":0.65200000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84699999999999998,"5min":1.028,"15min":1.028},"last":0.54600000000000004}]},{"osd":6,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.63400000000000001,"15min":0.65800000000000003},"min":{"1min":0.51100000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92400000000000004,"5min":0.97199999999999998,"15min":1.091},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.64000000000000001,"15min":0.67900000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91200000000000003,"5min":0.95699999999999996,"15min":0.96099999999999997},"last":0.626}]},{"osd":7,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.64900000000000002,"15min":0.67500000000000004},"min":{"1min":0.435,"5min":0.42299999999999999,"15min":0.41599999999999998},"max":{"1min":0.93899999999999995,"5min":0.94499999999999995,"15min":0.95399999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.65600000000000003,"15min":0.68999999999999995},"min":{"1min":0.44700000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.94899999999999995,"5min":1.01,"15min":1.01},"last":0.67000000000000004}]}]},{"osd":3,"up_from":13,"seq":55834576026,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52756,"kb_used_data":3588,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319084,"statfs":{"total":96636764160,"available":96582742016,"internally_reserved":0,"allocated":3674112,"data_stored":1897324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.36299999999999999,"15min":0.376},"min":{"1min":0.255,"5min":0.223,"15min":0.223},"max":{"1min":0.53400000000000003,"5min":0.55200000000000005,"15min":0.72799999999999998},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.36799999999999999,"15min":0.39700000000000002},"min":{"1min":0.32000000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.63800000000000001},"last":0.34300000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.36699999999999999,"15min":0.39500000000000002},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.309},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.40600000000000003,"15min":0.42799999999999999},"min":{"1min":0.29299999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.60299999999999998,"5min":0.63,"15min":0.75800000000000001},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.434,"15min":0.46899999999999997},"min":{"1min":0.39200000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.67000000000000004,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.44},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.45700000000000002,"15min":0.48399999999999999},"min":{"1min":0.38700000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65200000000000002,"5min":0.67800000000000005,"15min":0.79300000000000004},"last":0.37}]},{"osd":4,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.59099999999999997,"15min":0.61899999999999999},"min":{"1min":0.42299999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":1.018},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.59999999999999998,"15min":0.63300000000000001},"min":{"1min":0.44900000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.0169999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.495}]},{"osd":5,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.64200000000000002,"15min":0.66700000000000004},"min":{"1min":0.47099999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.65200000000000002,"15min":0.67900000000000005},"min":{"1min":0.501,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.94399999999999995,"5min":1.036,"15min":1.036},"last":0.63300000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.626,"15min":0.64900000000000002},"min":{"1min":0.47499999999999998,"5min":0.31,"15min":0.31},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.63,"15min":0.66300000000000003},"min":{"1min":0.433,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.60199999999999998}]},{"osd":7,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.628,"15min":0.65900000000000003},"min":{"1min":0.46000000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90500000000000003,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.64500000000000002,"15min":0.68100000000000005},"min":{"1min":0.53000000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.61899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576026,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52756,"kb_used_data":3588,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319084,"statfs":{"total":96636764160,"available":96582742016,"internally_reserved":0,"allocated":3674112,"data_stored":1897324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54800000000000004,"15min":0.56999999999999995},"min":{"1min":0.38400000000000001,"5min":0.30499999999999999,"15min":0.26300000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.876},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.47899999999999998,"5min":0.36599999999999999,"15min":0.34499999999999997},"max":{"1min":0.77400000000000002,"5min":0.81100000000000005,"15min":1.202},"last":0.72899999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.60899999999999999,"15min":0.628},"min":{"1min":0.46200000000000002,"5min":0.379,"15min":0.35099999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":1.256},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.60299999999999998,"15min":0.61799999999999999},"min":{"1min":0.49199999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.89700000000000002},"last":0.70299999999999996}]},{"osd":2,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56200000000000006,"15min":0.58399999999999996},"min":{"1min":0.438,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80100000000000005,"5min":0.81799999999999995,"15min":0.84599999999999997},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59199999999999997,"15min":0.60999999999999999},"min":{"1min":0.46700000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70799999999999996,"5min":0.97899999999999998,"15min":1.294},"last":0.75800000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56599999999999995,"15min":0.58399999999999996},"min":{"1min":0.498,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.94299999999999995},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.59399999999999997,"15min":0.61599999999999999},"min":{"1min":0.47099999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.90200000000000002},"last":0.71499999999999997}]},{"osd":5,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.50900000000000001,"15min":0.52000000000000002},"min":{"1min":0.35599999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.64100000000000001,"5min":0.92800000000000005,"15min":5.0119999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.54200000000000004},"min":{"1min":0.35199999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68600000000000005,"5min":0.95199999999999996,"15min":4.931},"last":0.52100000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52900000000000003,"15min":0.53200000000000003},"min":{"1min":0.35299999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.77800000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.497},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.40300000000000002,"5min":0.35799999999999998,"15min":0.34699999999999998},"max":{"1min":0.71199999999999997,"5min":1.0609999999999999,"15min":1.3580000000000001},"last":0.63700000000000001}]},{"osd":7,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.54900000000000004,"15min":0.57999999999999996},"min":{"1min":0.378,"5min":0.35899999999999999,"15min":0.34000000000000002},"max":{"1min":0.66600000000000004,"5min":0.91000000000000003,"15min":4.9889999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58599999999999997,"15min":0.59899999999999998},"min":{"1min":0.40899999999999997,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72699999999999998,"5min":0.86899999999999999,"15min":0.88100000000000001},"last":0.60199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834576026,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":66704,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305136,"statfs":{"total":96636764160,"available":96568459264,"internally_reserved":0,"allocated":3670016,"data_stored":1896305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54300000000000004,"15min":0.57499999999999996},"min":{"1min":0.438,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.82199999999999995},"last":0.627},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54200000000000004,"15min":0.57699999999999996},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.28100000000000003},"max":{"1min":0.73999999999999999,"5min":0.76600000000000001,"15min":0.84699999999999998},"last":0.58099999999999996}]},{"osd":1,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54400000000000004,"15min":0.57699999999999996},"min":{"1min":0.437,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.81299999999999994},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54300000000000004,"15min":0.57499999999999996},"min":{"1min":0.42799999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69199999999999995,"5min":0.752,"15min":0.871},"last":0.48699999999999999}]},{"osd":2,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.55600000000000005,"15min":0.58699999999999997},"min":{"1min":0.46800000000000003,"5min":0.311,"15min":0.30599999999999999},"max":{"1min":0.72099999999999997,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.56200000000000006,"15min":0.58799999999999997},"min":{"1min":0.44600000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.745,"5min":0.745,"15min":0.83399999999999996},"last":0.54700000000000004}]},{"osd":3,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.55600000000000005,"15min":0.58699999999999997},"min":{"1min":0.40300000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77800000000000002,"5min":0.79800000000000004,"15min":0.81100000000000005},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58399999999999996,"15min":0.621},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.371},"max":{"1min":0.76100000000000001,"5min":0.81399999999999995,"15min":0.88200000000000001},"last":0.76400000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51000000000000001,"15min":0.50700000000000001},"min":{"1min":0.32300000000000001,"5min":0.29899999999999999,"15min":0.27400000000000002},"max":{"1min":0.70999999999999996,"5min":0.75900000000000001,"15min":0.85699999999999998},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57199999999999995,"15min":0.57999999999999996},"min":{"1min":0.34799999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.73999999999999999,"5min":0.82999999999999996,"15min":0.88800000000000001},"last":0.83399999999999996}]},{"osd":6,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35799999999999998,"5min":0.313,"15min":0.28899999999999998},"max":{"1min":0.75700000000000001,"5min":0.89900000000000002,"15min":0.94999999999999996},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52900000000000003,"15min":0.53200000000000003},"min":{"1min":0.372,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.76100000000000001,"5min":0.84299999999999997,"15min":0.878},"last":0.81799999999999995}]},{"osd":7,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.39100000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.76800000000000002,"5min":0.85599999999999998,"15min":0.90800000000000003},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.63800000000000001,"15min":0.64800000000000002},"min":{"1min":0.47399999999999998,"5min":0.41699999999999998,"15min":0.38300000000000001},"max":{"1min":0.79000000000000004,"5min":0.87,"15min":0.92300000000000004},"last":0.90400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:51:39.616 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T13:51:39.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.740+0000 7f5513ef7700 1 Processor -- start 2024-04-06T13:51:39.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.740+0000 7f5513ef7700 1 -- start start 2024-04-06T13:51:39.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.741+0000 7f5513ef7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f550c15a550 0x7f550c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.741+0000 7f5513ef7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f550c0cf900 0x7f550c0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.741+0000 7f5513ef7700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f550c0cdbe0 con 0x7f550c0b4ba0 2024-04-06T13:51:39.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.741+0000 7f5513ef7700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f550c0b5010 con 0x7f550c0cf900 2024-04-06T13:51:39.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.741+0000 7f5513ef7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f550c15a940 con 0x7f550c15a550 2024-04-06T13:51:39.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.741+0000 7f5511492700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f550c0cf900 0x7f550c0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.741+0000 7f5511492700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f550c0cf900 0x7f550c0b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44212/0 (socket says 172.21.15.189:44212) 2024-04-06T13:51:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.741+0000 7f5511492700 1 -- 172.21.15.189:0/3272755488 learned_addr learned my addr 172.21.15.189:0/3272755488 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.742+0000 7f5511c93700 1 --2- 172.21.15.189:0/3272755488 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f550c15a550 0x7f550c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.742+0000 7f5511492700 1 -- 172.21.15.189:0/3272755488 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f550c15a550 msgr2=0x7f550c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.742+0000 7f5511492700 1 --2- 172.21.15.189:0/3272755488 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f550c15a550 0x7f550c0cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.742+0000 7f5511492700 1 -- 172.21.15.189:0/3272755488 >> v1:172.21.15.192:6789/0 conn(0x7f550c0b4ba0 legacy=0x7f550c06eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.742+0000 7f5511492700 1 -- 172.21.15.189:0/3272755488 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f550c15ac00 con 0x7f550c0cf900 2024-04-06T13:51:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.742+0000 7f5511c93700 1 --2- 172.21.15.189:0/3272755488 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f550c15a550 0x7f550c0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:51:39.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.742+0000 7f5511492700 1 --2- 172.21.15.189:0/3272755488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f550c0cf900 0x7f550c0b4660 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7f54fc0098f0 tx=0x7f54fc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45ee1c510f52f0be server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:39.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.742+0000 7f5510c91700 1 -- 172.21.15.189:0/3272755488 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54fc004230 con 0x7f550c0cf900 2024-04-06T13:51:39.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.742+0000 7f5510c91700 1 -- 172.21.15.189:0/3272755488 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54fc004390 con 0x7f550c0cf900 2024-04-06T13:51:39.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.742+0000 7f5510c91700 1 -- 172.21.15.189:0/3272755488 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54fc0036a0 con 0x7f550c0cf900 2024-04-06T13:51:39.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.742+0000 7f5513ef7700 1 -- 172.21.15.189:0/3272755488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f550c0cf900 msgr2=0x7f550c0b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:39.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.742+0000 7f5513ef7700 1 --2- 172.21.15.189:0/3272755488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f550c0cf900 0x7f550c0b4660 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7f54fc0098f0 tx=0x7f54fc009c00 comp rx=0 tx=0).stop 2024-04-06T13:51:39.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.742+0000 7f5513ef7700 1 -- 172.21.15.189:0/3272755488 shutdown_connections 2024-04-06T13:51:39.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.742+0000 7f5513ef7700 1 --2- 172.21.15.189:0/3272755488 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f550c15a550 0x7f550c0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.742+0000 7f5513ef7700 1 --2- 172.21.15.189:0/3272755488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f550c0cf900 0x7f550c0b4660 unknown :-1 s=CLOSED pgs=3213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.742+0000 7f5513ef7700 1 -- 172.21.15.189:0/3272755488 >> 172.21.15.189:0/3272755488 conn(0x7f550c0d00b0 msgr2=0x7f550c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:39.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.743+0000 7f5513ef7700 1 -- 172.21.15.189:0/3272755488 shutdown_connections 2024-04-06T13:51:39.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.743+0000 7f5513ef7700 1 -- 172.21.15.189:0/3272755488 wait complete. 2024-04-06T13:51:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.744+0000 7f5513ef7700 1 Processor -- start 2024-04-06T13:51:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.744+0000 7f5513ef7700 1 -- start start 2024-04-06T13:51:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.744+0000 7f5513ef7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f550c0b4ba0 0x7f550c1558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.744+0000 7f5513ef7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f550c0cf900 0x7f550c144dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.744+0000 7f5513ef7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f550c158300 0x7f550c1418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.744+0000 7f5513ef7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550c142230 con 0x7f550c158300 2024-04-06T13:51:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.744+0000 7f5513ef7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550c142510 con 0x7f550c0b4ba0 2024-04-06T13:51:39.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.744+0000 7f5513ef7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f550c1427f0 con 0x7f550c0cf900 2024-04-06T13:51:39.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.745+0000 7f5512494700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f550c158300 0x7f550c1418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.745+0000 7f5512494700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f550c158300 0x7f550c1418c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44220/0 (socket says 172.21.15.189:44220) 2024-04-06T13:51:39.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.745+0000 7f5512494700 1 -- 172.21.15.189:0/3048889866 learned_addr learned my addr 172.21.15.189:0/3048889866 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:39.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.745+0000 7f5511492700 1 --2- 172.21.15.189:0/3048889866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f550c0cf900 0x7f550c144dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.745+0000 7f5511c93700 1 --2- 172.21.15.189:0/3048889866 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f550c0b4ba0 0x7f550c1558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.745+0000 7f5512494700 1 -- 172.21.15.189:0/3048889866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f550c0cf900 msgr2=0x7f550c144dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:39.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.745+0000 7f5512494700 1 --2- 172.21.15.189:0/3048889866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f550c0cf900 0x7f550c144dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.745+0000 7f5512494700 1 -- 172.21.15.189:0/3048889866 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f550c0b4ba0 msgr2=0x7f550c1558e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:39.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.745+0000 7f5512494700 1 --2- 172.21.15.189:0/3048889866 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f550c0b4ba0 0x7f550c1558e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.745+0000 7f5512494700 1 -- 172.21.15.189:0/3048889866 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc009550 con 0x7f550c158300 2024-04-06T13:51:39.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.745+0000 7f5512494700 1 --2- 172.21.15.189:0/3048889866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f550c158300 0x7f550c1418c0 secure :-1 s=READY pgs=3214 cs=0 l=1 rev1=1 crypto rx=0x7f550400cc10 tx=0x7f550400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:39.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.745+0000 7f55027fc700 1 -- 172.21.15.189:0/3048889866 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550400aa60 con 0x7f550c158300 2024-04-06T13:51:39.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.745+0000 7f55027fc700 1 -- 172.21.15.189:0/3048889866 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f550400abc0 con 0x7f550c158300 2024-04-06T13:51:39.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.745+0000 7f5513ef7700 1 -- 172.21.15.189:0/3048889866 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f550c142ad0 con 0x7f550c158300 2024-04-06T13:51:39.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.745+0000 7f55027fc700 1 -- 172.21.15.189:0/3048889866 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5504012420 con 0x7f550c158300 2024-04-06T13:51:39.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.745+0000 7f5513ef7700 1 -- 172.21.15.189:0/3048889866 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f550c158790 con 0x7f550c158300 2024-04-06T13:51:39.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.746+0000 7f5513ef7700 1 -- 172.21.15.189:0/3048889866 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f550c067bb0 con 0x7f550c158300 2024-04-06T13:51:39.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.747+0000 7f55027fc700 1 -- 172.21.15.189:0/3048889866 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f5504012580 con 0x7f550c158300 2024-04-06T13:51:39.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.747+0000 7f55027fc700 1 --2- 172.21.15.189:0/3048889866 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f54f006adb0 0x7f54f006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:39.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.747+0000 7f5511c93700 1 --2- 172.21.15.189:0/3048889866 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f54f006adb0 0x7f54f006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:39.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.747+0000 7f55027fc700 1 -- 172.21.15.189:0/3048889866 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f5504089980 con 0x7f550c158300 2024-04-06T13:51:39.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.748+0000 7f5511c93700 1 --2- 172.21.15.189:0/3048889866 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f54f006adb0 0x7f54f006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5508006fd0 tx=0x7f5508008040 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:39.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.752+0000 7f55027fc700 1 -- 172.21.15.189:0/3048889866 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5504054850 con 0x7f550c158300 2024-04-06T13:51:39.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.949+0000 7f5513ef7700 1 -- 172.21.15.189:0/3048889866 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f550c074990 con 0x7f550c158300 2024-04-06T13:51:39.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.950+0000 7f55027fc700 1 -- 172.21.15.189:0/3048889866 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v290) v1 ==== 74+0+13964 (secure 0 0 0) 0x7f5504057e70 con 0x7f550c158300 2024-04-06T13:51:39.950 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:51:39.950 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":290,"fsid":"e30041de-e9a3-4351-8ca7-29d7f3e3728c","created":"2024-04-06T12:11:46.556453+0000","modified":"2024-04-06T13:51:34.663885+0000","last_up_change":"2024-04-06T12:11:57.614001+0000","last_in_change":"2024-04-06T12:11:49.184725+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T12:11:58.609282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5572dd3f-16ee-46b3-bbb1-af1f04ab1a9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":47415},{"type":"v1","addr":"172.21.15.189:6809","nonce":47415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":47415},{"type":"v1","addr":"172.21.15.189:6811","nonce":47415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":47415},{"type":"v1","addr":"172.21.15.189:6815","nonce":47415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":47415},{"type":"v1","addr":"172.21.15.189:6813","nonce":47415}]},"public_addr":"172.21.15.189:6809/47415","cluster_addr":"172.21.15.189:6811/47415","heartbeat_back_addr":"172.21.15.189:6815/47415","heartbeat_front_addr":"172.21.15.189:6813/47415","state":["exists","up"]},{"osd":1,"uuid":"2d7c167b-3876-470d-9cff-e889bca278c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":47414},{"type":"v1","addr":"172.21.15.189:6801","nonce":47414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":47414},{"type":"v1","addr":"172.21.15.189:6803","nonce":47414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":47414},{"type":"v1","addr":"172.21.15.189:6807","nonce":47414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":47414},{"type":"v1","addr":"172.21.15.189:6805","nonce":47414}]},"public_addr":"172.21.15.189:6801/47414","cluster_addr":"172.21.15.189:6803/47414","heartbeat_back_addr":"172.21.15.189:6807/47414","heartbeat_front_addr":"172.21.15.189:6805/47414","state":["exists","up"]},{"osd":2,"uuid":"fc510b86-46ed-4c2c-8a50-211c375a1d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":47416},{"type":"v1","addr":"172.21.15.189:6817","nonce":47416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":47416},{"type":"v1","addr":"172.21.15.189:6819","nonce":47416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":47416},{"type":"v1","addr":"172.21.15.189:6823","nonce":47416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":47416},{"type":"v1","addr":"172.21.15.189:6821","nonce":47416}]},"public_addr":"172.21.15.189:6817/47416","cluster_addr":"172.21.15.189:6819/47416","heartbeat_back_addr":"172.21.15.189:6823/47416","heartbeat_front_addr":"172.21.15.189:6821/47416","state":["exists","up"]},{"osd":3,"uuid":"56f5a991-198d-4087-90d4-72338925912c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":47425},{"type":"v1","addr":"172.21.15.189:6825","nonce":47425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":47425},{"type":"v1","addr":"172.21.15.189:6827","nonce":47425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":47425},{"type":"v1","addr":"172.21.15.189:6831","nonce":47425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":47425},{"type":"v1","addr":"172.21.15.189:6829","nonce":47425}]},"public_addr":"172.21.15.189:6825/47425","cluster_addr":"172.21.15.189:6827/47425","heartbeat_back_addr":"172.21.15.189:6831/47425","heartbeat_front_addr":"172.21.15.189:6829/47425","state":["exists","up"]},{"osd":4,"uuid":"c27c81cb-146f-4950-8f79-fd82d2ba6c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":47716},{"type":"v1","addr":"172.21.15.192:6817","nonce":47716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":47716},{"type":"v1","addr":"172.21.15.192:6820","nonce":47716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":47716},{"type":"v1","addr":"172.21.15.192:6828","nonce":47716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":47716},{"type":"v1","addr":"172.21.15.192:6824","nonce":47716}]},"public_addr":"172.21.15.192:6817/47716","cluster_addr":"172.21.15.192:6820/47716","heartbeat_back_addr":"172.21.15.192:6828/47716","heartbeat_front_addr":"172.21.15.192:6824/47716","state":["exists","up"]},{"osd":5,"uuid":"8dd707b1-5813-478c-9579-7072ee3070c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":47714},{"type":"v1","addr":"172.21.15.192:6801","nonce":47714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":47714},{"type":"v1","addr":"172.21.15.192:6805","nonce":47714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":47714},{"type":"v1","addr":"172.21.15.192:6813","nonce":47714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":47714},{"type":"v1","addr":"172.21.15.192:6809","nonce":47714}]},"public_addr":"172.21.15.192:6801/47714","cluster_addr":"172.21.15.192:6805/47714","heartbeat_back_addr":"172.21.15.192:6813/47714","heartbeat_front_addr":"172.21.15.192:6809/47714","state":["exists","up"]},{"osd":6,"uuid":"8af1f576-d0e4-4f37-b782-6b2b084c89d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6819","nonce":47717},{"type":"v1","addr":"172.21.15.192:6821","nonce":47717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":47717},{"type":"v1","addr":"172.21.15.192:6825","nonce":47717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":47717},{"type":"v1","addr":"172.21.15.192:6831","nonce":47717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6827","nonce":47717},{"type":"v1","addr":"172.21.15.192:6829","nonce":47717}]},"public_addr":"172.21.15.192:6821/47717","cluster_addr":"172.21.15.192:6825/47717","heartbeat_back_addr":"172.21.15.192:6831/47717","heartbeat_front_addr":"172.21.15.192:6829/47717","state":["exists","up"]},{"osd":7,"uuid":"b24dafce-ea55-4073-b8e6-23c2c08cbb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":47715},{"type":"v1","addr":"172.21.15.192:6804","nonce":47715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":47715},{"type":"v1","addr":"172.21.15.192:6808","nonce":47715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":47715},{"type":"v1","addr":"172.21.15.192:6815","nonce":47715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":47715},{"type":"v1","addr":"172.21.15.192:6812","nonce":47715}]},"public_addr":"172.21.15.192:6804/47715","cluster_addr":"172.21.15.192:6808/47715","heartbeat_back_addr":"172.21.15.192:6815/47715","heartbeat_front_addr":"172.21.15.192:6812/47715","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.416685+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.291108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.392541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.420219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.364358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.327270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.397044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T12:11:54.328154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.192:6834/3894184871":"2024-04-07T13:51:31.162267+0000","172.21.15.192:6832/3894184871":"2024-04-07T13:51:31.162267+0000","172.21.15.189:0/1303197157":"2024-04-07T13:51:30.485281+0000","172.21.15.189:0/3577027484":"2024-04-07T13:51:30.485281+0000","172.21.15.189:0/4161423243":"2024-04-07T13:51:30.485281+0000","172.21.15.189:0/2780155014":"2024-04-07T13:51:30.485281+0000","172.21.15.189:0/769203804":"2024-04-07T13:51:30.485281+0000","172.21.15.189:0/489741497":"2024-04-07T13:51:30.485281+0000","172.21.15.189:0/1941733448":"2024-04-07T13:51:30.485281+0000","172.21.15.189:0/4106081342":"2024-04-07T13:51:30.485281+0000","172.21.15.189:0/867158792":"2024-04-07T13:51:30.485281+0000","172.21.15.189:0/2987126770":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/2905400170":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/893866137":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/2221278556":"2024-04-07T13:50:07.256457+0000","172.21.15.189:6833/111704":"2024-04-07T13:50:07.256457+0000","172.21.15.189:6832/111704":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/4009223680":"2024-04-07T13:50:07.256457+0000","172.21.15.189:0/3345146368":"2024-04-07T13:50:07.256457+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T13:51:39.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.952+0000 7f5513ef7700 1 -- 172.21.15.189:0/3048889866 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f54f006adb0 msgr2=0x7f54f006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:39.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.952+0000 7f5513ef7700 1 --2- 172.21.15.189:0/3048889866 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f54f006adb0 0x7f54f006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5508006fd0 tx=0x7f5508008040 comp rx=0 tx=0).stop 2024-04-06T13:51:39.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.952+0000 7f5513ef7700 1 -- 172.21.15.189:0/3048889866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f550c158300 msgr2=0x7f550c1418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:39.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.952+0000 7f5513ef7700 1 --2- 172.21.15.189:0/3048889866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f550c158300 0x7f550c1418c0 secure :-1 s=READY pgs=3214 cs=0 l=1 rev1=1 crypto rx=0x7f550400cc10 tx=0x7f550400cf20 comp rx=0 tx=0).stop 2024-04-06T13:51:39.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.952+0000 7f5513ef7700 1 -- 172.21.15.189:0/3048889866 shutdown_connections 2024-04-06T13:51:39.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.952+0000 7f5513ef7700 1 --2- 172.21.15.189:0/3048889866 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f54f006adb0 0x7f54f006d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.952+0000 7f5513ef7700 1 --2- 172.21.15.189:0/3048889866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f550c158300 0x7f550c1418c0 unknown :-1 s=CLOSED pgs=3214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.952+0000 7f5513ef7700 1 --2- 172.21.15.189:0/3048889866 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f550c0b4ba0 0x7f550c1558e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.952+0000 7f5513ef7700 1 --2- 172.21.15.189:0/3048889866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f550c0cf900 0x7f550c144dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:39.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.952+0000 7f5513ef7700 1 -- 172.21.15.189:0/3048889866 >> 172.21.15.189:0/3048889866 conn(0x7f550c0d00b0 msgr2=0x7f550c0b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:39.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.952+0000 7f5513ef7700 1 -- 172.21.15.189:0/3048889866 shutdown_connections 2024-04-06T13:51:39.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:39.952+0000 7f5513ef7700 1 -- 172.21.15.189:0/3048889866 wait complete. 2024-04-06T13:51:40.965 INFO:tasks.ceph:Scrubbing osd.0 2024-04-06T13:51:40.965 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-06T13:51:41.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.105+0000 7fbaf96ca700 1 Processor -- start 2024-04-06T13:51:41.105 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.105+0000 7fbaf96ca700 1 -- start start 2024-04-06T13:51:41.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.106+0000 7fbaf96ca700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbaf415d6f0 0x7fbaf415dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.106+0000 7fbaf96ca700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbaf40ce730 0x7fbaf40d1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.106+0000 7fbaf96ca700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf415eac0 con 0x7fbaf40b8d70 2024-04-06T13:51:41.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.106+0000 7fbaf96ca700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf415e010 con 0x7fbaf40ce730 2024-04-06T13:51:41.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.106+0000 7fbaf96ca700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf415e220 con 0x7fbaf415d6f0 2024-04-06T13:51:41.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.106+0000 7fbaf37fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbaf40ce730 0x7fbaf40d1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.116 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.106+0000 7fbaf37fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbaf40ce730 0x7fbaf40d1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44224/0 (socket says 172.21.15.189:44224) 2024-04-06T13:51:41.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.106+0000 7fbaf37fe700 1 -- 172.21.15.189:0/137793047 learned_addr learned my addr 172.21.15.189:0/137793047 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:41.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.106+0000 7fbaf2ffd700 1 --2- 172.21.15.189:0/137793047 >> v2:172.21.15.192:3300/0 conn(0x7fbaf415d6f0 0x7fbaf415dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.106+0000 7fbaf37fe700 1 -- 172.21.15.189:0/137793047 >> v2:172.21.15.192:3300/0 conn(0x7fbaf415d6f0 msgr2=0x7fbaf415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:41.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.106+0000 7fbaf37fe700 1 --2- 172.21.15.189:0/137793047 >> v2:172.21.15.192:3300/0 conn(0x7fbaf415d6f0 0x7fbaf415dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.106+0000 7fbaf37fe700 1 -- 172.21.15.189:0/137793047 >> v1:172.21.15.192:6789/0 conn(0x7fbaf40b8d70 legacy=0x7fbaf40ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:41.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.106+0000 7fbaf37fe700 1 -- 172.21.15.189:0/137793047 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf415e500 con 0x7fbaf40ce730 2024-04-06T13:51:41.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.106+0000 7fbaf2ffd700 1 --2- 172.21.15.189:0/137793047 >> v2:172.21.15.192:3300/0 conn(0x7fbaf415d6f0 0x7fbaf415dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:51:41.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.106+0000 7fbaf37fe700 1 --2- 172.21.15.189:0/137793047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbaf40ce730 0x7fbaf40d1e00 secure :-1 s=READY pgs=3215 cs=0 l=1 rev1=1 crypto rx=0x7fbaf40d4410 tx=0x7fbaec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c73c1477d5da6de5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:41.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.106+0000 7fbaf1ffb700 1 -- 172.21.15.189:0/137793047 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec00e070 con 0x7fbaf40ce730 2024-04-06T13:51:41.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.106+0000 7fbaf1ffb700 1 -- 172.21.15.189:0/137793047 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaec003a20 con 0x7fbaf40ce730 2024-04-06T13:51:41.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.107+0000 7fbaf1ffb700 1 -- 172.21.15.189:0/137793047 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec003dd0 con 0x7fbaf40ce730 2024-04-06T13:51:41.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.107+0000 7fbaf96ca700 1 -- 172.21.15.189:0/137793047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbaf40ce730 msgr2=0x7fbaf40d1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.107+0000 7fbaf96ca700 1 --2- 172.21.15.189:0/137793047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbaf40ce730 0x7fbaf40d1e00 secure :-1 s=READY pgs=3215 cs=0 l=1 rev1=1 crypto rx=0x7fbaf40d4410 tx=0x7fbaec00b330 comp rx=0 tx=0).stop 2024-04-06T13:51:41.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.107+0000 7fbaf96ca700 1 -- 172.21.15.189:0/137793047 shutdown_connections 2024-04-06T13:51:41.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.107+0000 7fbaf96ca700 1 --2- 172.21.15.189:0/137793047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbaf40ce730 0x7fbaf40d1e00 unknown :-1 s=CLOSED pgs=3215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.107+0000 7fbaf96ca700 1 --2- 172.21.15.189:0/137793047 >> v2:172.21.15.192:3300/0 conn(0x7fbaf415d6f0 0x7fbaf415dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.121 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.107+0000 7fbaf96ca700 1 -- 172.21.15.189:0/137793047 >> 172.21.15.189:0/137793047 conn(0x7fbaf4072490 msgr2=0x7fbaf406b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:41.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.107+0000 7fbaf96ca700 1 -- 172.21.15.189:0/137793047 shutdown_connections 2024-04-06T13:51:41.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.107+0000 7fbaf96ca700 1 -- 172.21.15.189:0/137793047 wait complete. 2024-04-06T13:51:41.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.108+0000 7fbaf96ca700 1 Processor -- start 2024-04-06T13:51:41.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.108+0000 7fbaf96ca700 1 -- start start 2024-04-06T13:51:41.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.109+0000 7fbaf96ca700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbaf40b8d70 0x7fbaf4157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.109+0000 7fbaf96ca700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbaf40ce730 0x7fbaf4157ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.109+0000 7fbaf96ca700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbaf415d6f0 0x7fbaf41f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.109+0000 7fbaf96ca700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf40b2e60 con 0x7fbaf40ce730 2024-04-06T13:51:41.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.109+0000 7fbaf96ca700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf40b2b10 con 0x7fbaf40b8d70 2024-04-06T13:51:41.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.109+0000 7fbaf96ca700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf4158330 con 0x7fbaf415d6f0 2024-04-06T13:51:41.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.109+0000 7fbaf27fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbaf40ce730 0x7fbaf4157ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.109+0000 7fbaf27fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbaf40ce730 0x7fbaf4157ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44230/0 (socket says 172.21.15.189:44230) 2024-04-06T13:51:41.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.109+0000 7fbaf27fc700 1 -- 172.21.15.189:0/206579019 learned_addr learned my addr 172.21.15.189:0/206579019 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:41.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.109+0000 7fbaf2ffd700 1 --2- 172.21.15.189:0/206579019 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbaf40b8d70 0x7fbaf4157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.109+0000 7fbaf37fe700 1 --2- 172.21.15.189:0/206579019 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbaf415d6f0 0x7fbaf41f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.110+0000 7fbaf27fc700 1 -- 172.21.15.189:0/206579019 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbaf415d6f0 msgr2=0x7fbaf41f62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.110+0000 7fbaf27fc700 1 --2- 172.21.15.189:0/206579019 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbaf415d6f0 0x7fbaf41f62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.110+0000 7fbaf27fc700 1 -- 172.21.15.189:0/206579019 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbaf40b8d70 msgr2=0x7fbaf4157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.110+0000 7fbaf27fc700 1 --2- 172.21.15.189:0/206579019 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbaf40b8d70 0x7fbaf4157590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.110+0000 7fbaf27fc700 1 -- 172.21.15.189:0/206579019 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaec009b30 con 0x7fbaf40ce730 2024-04-06T13:51:41.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.110+0000 7fbaf27fc700 1 --2- 172.21.15.189:0/206579019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbaf40ce730 0x7fbaf4157ad0 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 crypto rx=0x7fbae800c4a0 tx=0x7fbae800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:41.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.110+0000 7fbadb7fe700 1 -- 172.21.15.189:0/206579019 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae8011790 con 0x7fbaf40ce730 2024-04-06T13:51:41.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.110+0000 7fbadb7fe700 1 -- 172.21.15.189:0/206579019 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae80118f0 con 0x7fbaf40ce730 2024-04-06T13:51:41.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.110+0000 7fbadb7fe700 1 -- 172.21.15.189:0/206579019 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae8011ca0 con 0x7fbaf40ce730 2024-04-06T13:51:41.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.110+0000 7fbaf96ca700 1 -- 172.21.15.189:0/206579019 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf4158610 con 0x7fbaf40ce730 2024-04-06T13:51:41.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.110+0000 7fbaf96ca700 1 -- 172.21.15.189:0/206579019 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf4158010 con 0x7fbaf40ce730 2024-04-06T13:51:41.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.111+0000 7fbaf96ca700 1 -- 172.21.15.189:0/206579019 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbaf40d0750 con 0x7fbaf40ce730 2024-04-06T13:51:41.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.111+0000 7fbadb7fe700 1 -- 172.21.15.189:0/206579019 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fbae801c460 con 0x7fbaf40ce730 2024-04-06T13:51:41.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.112+0000 7fbadb7fe700 1 --2- 172.21.15.189:0/206579019 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbae006adb0 0x7fbae006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.112+0000 7fbadb7fe700 1 -- 172.21.15.189:0/206579019 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fbae8089330 con 0x7fbaf40ce730 2024-04-06T13:51:41.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.112+0000 7fbaf2ffd700 1 --2- 172.21.15.189:0/206579019 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbae006adb0 0x7fbae006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.112+0000 7fbadb7fe700 1 --2- 172.21.15.189:0/206579019 >> [v2:172.21.15.189:6808/47415,v1:172.21.15.189:6809/47415] conn(0x7fbae00716f0 0x7fbae0073b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.112+0000 7fbadb7fe700 1 -- 172.21.15.189:0/206579019 --> [v2:172.21.15.189:6808/47415,v1:172.21.15.189:6809/47415] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbae0074220 con 0x7fbae00716f0 2024-04-06T13:51:41.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.112+0000 7fbadb7fe700 1 -- 172.21.15.189:0/206579019 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=290) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbae808b9f0 con 0x7fbaf40ce730 2024-04-06T13:51:41.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.112+0000 7fbaf37fe700 1 --2- 172.21.15.189:0/206579019 >> [v2:172.21.15.189:6808/47415,v1:172.21.15.189:6809/47415] conn(0x7fbae00716f0 0x7fbae0073b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.112+0000 7fbaf2ffd700 1 --2- 172.21.15.189:0/206579019 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbae006adb0 0x7fbae006d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbae4006fd0 tx=0x7fbae4008040 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:41.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.112+0000 7fbaf37fe700 1 --2- 172.21.15.189:0/206579019 >> [v2:172.21.15.189:6808/47415,v1:172.21.15.189:6809/47415] conn(0x7fbae00716f0 0x7fbae0073b50 crc :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:41.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.113+0000 7fbadb7fe700 1 -- 172.21.15.189:0/206579019 <== osd.0 v2:172.21.15.189:6808/47415 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fbae0074220 con 0x7fbae00716f0 2024-04-06T13:51:41.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.127+0000 7fbaf96ca700 1 -- 172.21.15.189:0/206579019 --> [v2:172.21.15.189:6808/47415,v1:172.21.15.189:6809/47415] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fbaf40d18e0 con 0x7fbae00716f0 2024-04-06T13:51:41.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.127+0000 7fbadb7fe700 1 -- 172.21.15.189:0/206579019 <== osd.0 v2:172.21.15.189:6808/47415 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fbaec003df0 con 0x7fbae00716f0 2024-04-06T13:51:41.135 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:51:41.135 INFO:teuthology.orchestra.run.smithi189.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-06T13:51:41.135 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:51:41.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.128+0000 7fbaf96ca700 1 -- 172.21.15.189:0/206579019 >> [v2:172.21.15.189:6808/47415,v1:172.21.15.189:6809/47415] conn(0x7fbae00716f0 msgr2=0x7fbae0073b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.128+0000 7fbaf96ca700 1 --2- 172.21.15.189:0/206579019 >> [v2:172.21.15.189:6808/47415,v1:172.21.15.189:6809/47415] conn(0x7fbae00716f0 0x7fbae0073b50 crc :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.128+0000 7fbaf96ca700 1 -- 172.21.15.189:0/206579019 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbae006adb0 msgr2=0x7fbae006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.128+0000 7fbaf96ca700 1 --2- 172.21.15.189:0/206579019 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbae006adb0 0x7fbae006d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbae4006fd0 tx=0x7fbae4008040 comp rx=0 tx=0).stop 2024-04-06T13:51:41.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.128+0000 7fbaf96ca700 1 -- 172.21.15.189:0/206579019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbaf40ce730 msgr2=0x7fbaf4157ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.128+0000 7fbaf96ca700 1 --2- 172.21.15.189:0/206579019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbaf40ce730 0x7fbaf4157ad0 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 crypto rx=0x7fbae800c4a0 tx=0x7fbae800c7b0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.128+0000 7fbaf37fe700 1 -- 172.21.15.189:0/206579019 reap_dead start 2024-04-06T13:51:41.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.128+0000 7fbaf96ca700 1 -- 172.21.15.189:0/206579019 shutdown_connections 2024-04-06T13:51:41.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.128+0000 7fbaf96ca700 1 -- 172.21.15.189:0/206579019 >> 172.21.15.189:0/206579019 conn(0x7fbaf4072490 msgr2=0x7fbaf40b8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:41.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.128+0000 7fbaf96ca700 1 -- 172.21.15.189:0/206579019 shutdown_connections 2024-04-06T13:51:41.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.128+0000 7fbaf96ca700 1 -- 172.21.15.189:0/206579019 wait complete. 2024-04-06T13:51:41.142 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-06T13:51:41.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.266+0000 7f0fa8a29700 1 Processor -- start 2024-04-06T13:51:41.266 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.266+0000 7f0fa8a29700 1 -- start start 2024-04-06T13:51:41.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.267+0000 7f0fa8a29700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0fa40b4af0 0x7f0fa40b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.267+0000 7f0fa8a29700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa40cdbe0 con 0x7f0fa40b4660 2024-04-06T13:51:41.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.267+0000 7f0fa8a29700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa415a940 con 0x7f0fa415a550 2024-04-06T13:51:41.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.267+0000 7f0fa8a29700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fa415ac00 con 0x7f0fa40b4af0 2024-04-06T13:51:41.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.267+0000 7f0fa1d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0fa40b4660 0x7f0fa40b2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:32854/0 (socket says 172.21.15.189:32854) 2024-04-06T13:51:41.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.267+0000 7f0fa1d9b700 1 -- 172.21.15.189:0/3502234866 learned_addr learned my addr 172.21.15.189:0/3502234866 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:41.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa159a700 1 -- 172.21.15.189:0/3502234866 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399298774 0 0) 0x7f0fa40cdbe0 con 0x7f0fa40b4660 2024-04-06T13:51:41.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa159a700 1 -- 172.21.15.189:0/3502234866 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f8c0035b0 con 0x7f0fa40b4660 2024-04-06T13:51:41.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa2d9d700 1 --2- 172.21.15.189:0/3502234866 >> v2:172.21.15.192:3300/0 conn(0x7f0fa40b4af0 0x7f0fa40b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa159a700 1 -- 172.21.15.189:0/3502234866 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3845608235 0 0) 0x7f0f8c0035b0 con 0x7f0fa40b4660 2024-04-06T13:51:41.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa159a700 1 -- 172.21.15.189:0/3502234866 >> v2:172.21.15.192:3300/0 conn(0x7f0fa40b4af0 msgr2=0x7f0fa40b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:41.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa159a700 1 --2- 172.21.15.189:0/3502234866 >> v2:172.21.15.192:3300/0 conn(0x7f0fa40b4af0 0x7f0fa40b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa159a700 1 -- 172.21.15.189:0/3502234866 >> v1:172.21.15.192:6789/0 conn(0x7f0fa415a550 legacy=0x7f0fa40cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa159a700 1 -- 172.21.15.189:0/3502234866 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa415aee0 con 0x7f0fa40b4660 2024-04-06T13:51:41.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa159a700 1 -- 172.21.15.189:0/3502234866 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f0f98001670 con 0x7f0fa40b4660 2024-04-06T13:51:41.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa2d9d700 1 --2- 172.21.15.189:0/3502234866 >> v2:172.21.15.192:3300/0 conn(0x7f0fa40b4af0 0x7f0fa40b36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:51:41.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa159a700 1 -- 172.21.15.189:0/3502234866 >> v1:172.21.15.189:6789/0 conn(0x7f0fa40b4660 legacy=0x7f0fa40b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa159a700 1 --2- 172.21.15.189:0/3502234866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f8c003a00 0x7f0f8c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa159a700 1 --2- 172.21.15.189:0/3502234866 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c004850 0x7f0f8c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa159a700 1 --2- 172.21.15.189:0/3502234866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f8c005210 0x7f0f8c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa159a700 1 -- 172.21.15.189:0/3502234866 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa415ac00 con 0x7f0f8c003a00 2024-04-06T13:51:41.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa159a700 1 -- 172.21.15.189:0/3502234866 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c00c160 con 0x7f0f8c004850 2024-04-06T13:51:41.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa159a700 1 -- 172.21.15.189:0/3502234866 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c00c320 con 0x7f0f8c005210 2024-04-06T13:51:41.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa259c700 1 --2- 172.21.15.189:0/3502234866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f8c003a00 0x7f0f8c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa1d9b700 1 --2- 172.21.15.189:0/3502234866 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c004850 0x7f0f8c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa2d9d700 1 --2- 172.21.15.189:0/3502234866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f8c005210 0x7f0f8c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa259c700 1 -- 172.21.15.189:0/3502234866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f8c005210 msgr2=0x7f0f8c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:41.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa259c700 1 --2- 172.21.15.189:0/3502234866 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f8c005210 0x7f0f8c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa259c700 1 -- 172.21.15.189:0/3502234866 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c004850 msgr2=0x7f0f8c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:41.284 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa259c700 1 --2- 172.21.15.189:0/3502234866 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c004850 0x7f0f8c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa259c700 1 -- 172.21.15.189:0/3502234866 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8c0035b0 con 0x7f0f8c003a00 2024-04-06T13:51:41.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.268+0000 7f0fa2d9d700 1 -- 172.21.15.189:0/3502234866 reap_dead start 2024-04-06T13:51:41.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.269+0000 7f0fa259c700 1 --2- 172.21.15.189:0/3502234866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f8c003a00 0x7f0f8c003e20 secure :-1 s=READY pgs=3218 cs=0 l=1 rev1=1 crypto rx=0x7f0f940017c0 tx=0x7f0f8c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e1c1bd5f65fe494 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:41.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.269+0000 7f0fa159a700 1 -- 172.21.15.189:0/3502234866 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f940052d0 con 0x7f0f8c003a00 2024-04-06T13:51:41.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.269+0000 7f0fa159a700 1 -- 172.21.15.189:0/3502234866 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f94005430 con 0x7f0f8c003a00 2024-04-06T13:51:41.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.269+0000 7f0fa159a700 1 -- 172.21.15.189:0/3502234866 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f940038c0 con 0x7f0f8c003a00 2024-04-06T13:51:41.285 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.269+0000 7f0fa8a29700 1 -- 172.21.15.189:0/3502234866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f8c003a00 msgr2=0x7f0f8c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.269+0000 7f0fa8a29700 1 --2- 172.21.15.189:0/3502234866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f8c003a00 0x7f0f8c003e20 secure :-1 s=READY pgs=3218 cs=0 l=1 rev1=1 crypto rx=0x7f0f940017c0 tx=0x7f0f8c005880 comp rx=0 tx=0).stop 2024-04-06T13:51:41.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.269+0000 7f0fa8a29700 1 -- 172.21.15.189:0/3502234866 shutdown_connections 2024-04-06T13:51:41.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.269+0000 7f0fa8a29700 1 --2- 172.21.15.189:0/3502234866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f8c003a00 0x7f0f8c003e20 unknown :-1 s=CLOSED pgs=3218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.269+0000 7f0fa8a29700 1 -- 172.21.15.189:0/3502234866 >> 172.21.15.189:0/3502234866 conn(0x7f0fa40d00b0 msgr2=0x7f0fa40d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:41.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.269+0000 7f0fa8a29700 1 -- 172.21.15.189:0/3502234866 shutdown_connections 2024-04-06T13:51:41.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.269+0000 7f0fa8a29700 1 -- 172.21.15.189:0/3502234866 wait complete. 2024-04-06T13:51:41.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.271+0000 7f0fa8a29700 1 Processor -- start 2024-04-06T13:51:41.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.271+0000 7f0fa8a29700 1 -- start start 2024-04-06T13:51:41.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.271+0000 7f0fa8a29700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f8c003a00 0x7f0fa415b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.271+0000 7f0fa8a29700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fa40aefb0 0x7f0fa414ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.271+0000 7f0fa8a29700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0fa4071cc0 0x7f0fa4072140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.271+0000 7f0fa8a29700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa4072680 con 0x7f0f8c003a00 2024-04-06T13:51:41.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.271+0000 7f0fa8a29700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa415c180 con 0x7f0fa40aefb0 2024-04-06T13:51:41.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.271+0000 7f0fa8a29700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa415c460 con 0x7f0fa4071cc0 2024-04-06T13:51:41.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.272+0000 7f0fa259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f8c003a00 0x7f0fa415b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.272+0000 7f0fa259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f8c003a00 0x7f0fa415b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44244/0 (socket says 172.21.15.189:44244) 2024-04-06T13:51:41.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.272+0000 7f0fa259c700 1 -- 172.21.15.189:0/1751054684 learned_addr learned my addr 172.21.15.189:0/1751054684 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:41.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.272+0000 7f0fa1d9b700 1 --2- 172.21.15.189:0/1751054684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fa40aefb0 0x7f0fa414ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.272+0000 7f0fa259c700 1 -- 172.21.15.189:0/1751054684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0fa4071cc0 msgr2=0x7f0fa4072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.272+0000 7f0fa2d9d700 1 --2- 172.21.15.189:0/1751054684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0fa4071cc0 0x7f0fa4072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.272+0000 7f0fa259c700 1 --2- 172.21.15.189:0/1751054684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0fa4071cc0 0x7f0fa4072140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.272+0000 7f0fa259c700 1 -- 172.21.15.189:0/1751054684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fa40aefb0 msgr2=0x7f0fa414ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.272+0000 7f0fa259c700 1 --2- 172.21.15.189:0/1751054684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fa40aefb0 0x7f0fa414ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.272+0000 7f0fa259c700 1 -- 172.21.15.189:0/1751054684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa415c740 con 0x7f0f8c003a00 2024-04-06T13:51:41.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.272+0000 7f0fa1d9b700 1 --2- 172.21.15.189:0/1751054684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fa40aefb0 0x7f0fa414ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:51:41.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.272+0000 7f0fa259c700 1 --2- 172.21.15.189:0/1751054684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f8c003a00 0x7f0fa415b930 secure :-1 s=READY pgs=3219 cs=0 l=1 rev1=1 crypto rx=0x7f0f94005040 tx=0x7f0f94001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:41.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.272+0000 7f0f92ffd700 1 -- 172.21.15.189:0/1751054684 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f94001290 con 0x7f0f8c003a00 2024-04-06T13:51:41.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.272+0000 7f0f92ffd700 1 -- 172.21.15.189:0/1751054684 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f94007040 con 0x7f0f8c003a00 2024-04-06T13:51:41.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.272+0000 7f0fa8a29700 1 -- 172.21.15.189:0/1751054684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa415ca20 con 0x7f0f8c003a00 2024-04-06T13:51:41.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.272+0000 7f0f92ffd700 1 -- 172.21.15.189:0/1751054684 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f940073d0 con 0x7f0f8c003a00 2024-04-06T13:51:41.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.272+0000 7f0fa8a29700 1 -- 172.21.15.189:0/1751054684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa414f200 con 0x7f0f8c003a00 2024-04-06T13:51:41.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.273+0000 7f0fa8a29700 1 -- 172.21.15.189:0/1751054684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa41542c0 con 0x7f0f8c003a00 2024-04-06T13:51:41.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.273+0000 7f0f92ffd700 1 -- 172.21.15.189:0/1751054684 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f0f94007530 con 0x7f0f8c003a00 2024-04-06T13:51:41.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.274+0000 7f0f92ffd700 1 --2- 172.21.15.189:0/1751054684 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0f8406adb0 0x7f0f8406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.274+0000 7f0f92ffd700 1 -- 172.21.15.189:0/1751054684 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f0f94089fb0 con 0x7f0f8c003a00 2024-04-06T13:51:41.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.274+0000 7f0fa1d9b700 1 --2- 172.21.15.189:0/1751054684 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0f8406adb0 0x7f0f8406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.274+0000 7f0fa1d9b700 1 --2- 172.21.15.189:0/1751054684 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0f8406adb0 0x7f0f8406d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0fa40b53c0 tx=0x7f0f98009040 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:41.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.278+0000 7f0f92ffd700 1 -- 172.21.15.189:0/1751054684 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0f940bd650 con 0x7f0f8c003a00 2024-04-06T13:51:41.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.472+0000 7f0fa8a29700 1 -- 172.21.15.189:0/1751054684 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fa406fdd0 con 0x7f0f8406adb0 2024-04-06T13:51:41.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.472+0000 7f0f92ffd700 1 -- 172.21.15.189:0/1751054684 <== mgr.10415 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0f98001b20 con 0x7f0f8406adb0 2024-04-06T13:51:41.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.474+0000 7f0fa8a29700 1 -- 172.21.15.189:0/1751054684 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0f8406adb0 msgr2=0x7f0f8406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.474+0000 7f0fa8a29700 1 --2- 172.21.15.189:0/1751054684 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0f8406adb0 0x7f0f8406d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0fa40b53c0 tx=0x7f0f98009040 comp rx=0 tx=0).stop 2024-04-06T13:51:41.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.474+0000 7f0fa8a29700 1 -- 172.21.15.189:0/1751054684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f8c003a00 msgr2=0x7f0fa415b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.474+0000 7f0fa8a29700 1 --2- 172.21.15.189:0/1751054684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f8c003a00 0x7f0fa415b930 secure :-1 s=READY pgs=3219 cs=0 l=1 rev1=1 crypto rx=0x7f0f94005040 tx=0x7f0f94001120 comp rx=0 tx=0).stop 2024-04-06T13:51:41.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.474+0000 7f0fa8a29700 1 -- 172.21.15.189:0/1751054684 shutdown_connections 2024-04-06T13:51:41.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.474+0000 7f0fa8a29700 1 --2- 172.21.15.189:0/1751054684 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f0f8406adb0 0x7f0f8406d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.474+0000 7f0fa8a29700 1 --2- 172.21.15.189:0/1751054684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0fa4071cc0 0x7f0fa4072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.474+0000 7f0fa8a29700 1 --2- 172.21.15.189:0/1751054684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f8c003a00 0x7f0fa415b930 unknown :-1 s=CLOSED pgs=3219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.474+0000 7f0fa8a29700 1 --2- 172.21.15.189:0/1751054684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fa40aefb0 0x7f0fa414ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.474+0000 7f0fa8a29700 1 -- 172.21.15.189:0/1751054684 >> 172.21.15.189:0/1751054684 conn(0x7f0fa40d00b0 msgr2=0x7f0fa4159550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:41.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.475+0000 7f0fa8a29700 1 -- 172.21.15.189:0/1751054684 shutdown_connections 2024-04-06T13:51:41.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.475+0000 7f0fa8a29700 1 -- 172.21.15.189:0/1751054684 wait complete. 2024-04-06T13:51:41.477 INFO:teuthology.orchestra.run.smithi189.stderr:instructed osd(s) 0 to deep-scrub 2024-04-06T13:51:41.487 INFO:tasks.ceph:Scrubbing osd.1 2024-04-06T13:51:41.487 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-06T13:51:41.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.613+0000 7f22b4dc4700 1 Processor -- start 2024-04-06T13:51:41.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.613+0000 7f22b4dc4700 1 -- start start 2024-04-06T13:51:41.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.614+0000 7f22b4dc4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f22b015a550 0x7f22b00cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.614+0000 7f22b4dc4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22b00b4d70 0x7f22b00b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.614+0000 7f22b4dc4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b00cdbe0 con 0x7f22b00cf900 2024-04-06T13:51:41.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.614+0000 7f22b4dc4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22b015a940 con 0x7f22b015a550 2024-04-06T13:51:41.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.614+0000 7f22b4dc4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22b015ac00 con 0x7f22b00b4d70 2024-04-06T13:51:41.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.614+0000 7f22add9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f22b00cf900 0x7f22b00b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:32866/0 (socket says 172.21.15.189:32866) 2024-04-06T13:51:41.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.614+0000 7f22add9b700 1 -- 172.21.15.189:0/3644820509 learned_addr learned my addr 172.21.15.189:0/3644820509 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:41.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.614+0000 7f22ae59c700 1 --2- 172.21.15.189:0/3644820509 >> v2:172.21.15.192:3300/0 conn(0x7f22b015a550 0x7f22b00cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.614+0000 7f22aed9d700 1 --2- 172.21.15.189:0/3644820509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22b00b4d70 0x7f22b00b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.614+0000 7f22ad59a700 1 -- 172.21.15.189:0/3644820509 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603170647 0 0) 0x7f22b00cdbe0 con 0x7f22b00cf900 2024-04-06T13:51:41.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.614+0000 7f22ad59a700 1 -- 172.21.15.189:0/3644820509 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22980035b0 con 0x7f22b00cf900 2024-04-06T13:51:41.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.614+0000 7f22ae59c700 1 -- 172.21.15.189:0/3644820509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22b00b4d70 msgr2=0x7f22b00b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:41.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.614+0000 7f22ae59c700 1 --2- 172.21.15.189:0/3644820509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22b00b4d70 0x7f22b00b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.614+0000 7f22ae59c700 1 -- 172.21.15.189:0/3644820509 >> v1:172.21.15.189:6789/0 conn(0x7f22b00cf900 legacy=0x7f22b00b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.614+0000 7f22ad59a700 1 -- 172.21.15.189:0/3644820509 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2126609220 0 0) 0x7f22980035b0 con 0x7f22b00cf900 2024-04-06T13:51:41.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.614+0000 7f22ae59c700 1 -- 172.21.15.189:0/3644820509 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b015aee0 con 0x7f22b015a550 2024-04-06T13:51:41.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.614+0000 7f22aed9d700 1 --2- 172.21.15.189:0/3644820509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22b00b4d70 0x7f22b00b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:51:41.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.615+0000 7f22ae59c700 1 --2- 172.21.15.189:0/3644820509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22b015a550 0x7f22b00cf3c0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f22a4009840 tx=0x7f22a4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e7dd64975085125 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:41.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.615+0000 7f22ad59a700 1 -- 172.21.15.189:0/3644820509 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a0001670 con 0x7f22b015a550 2024-04-06T13:51:41.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.615+0000 7f22ad59a700 1 -- 172.21.15.189:0/3644820509 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a4004030 con 0x7f22b015a550 2024-04-06T13:51:41.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.615+0000 7f22ad59a700 1 -- 172.21.15.189:0/3644820509 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a40043c0 con 0x7f22b015a550 2024-04-06T13:51:41.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.615+0000 7f22b4dc4700 1 -- 172.21.15.189:0/3644820509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22b015a550 msgr2=0x7f22b00cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.615+0000 7f22b4dc4700 1 --2- 172.21.15.189:0/3644820509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22b015a550 0x7f22b00cf3c0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f22a4009840 tx=0x7f22a4009b50 comp rx=0 tx=0).stop 2024-04-06T13:51:41.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.615+0000 7f22b4dc4700 1 -- 172.21.15.189:0/3644820509 shutdown_connections 2024-04-06T13:51:41.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.615+0000 7f22b4dc4700 1 --2- 172.21.15.189:0/3644820509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22b00b4d70 0x7f22b00b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.615+0000 7f22b4dc4700 1 --2- 172.21.15.189:0/3644820509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22b015a550 0x7f22b00cf3c0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.615+0000 7f22b4dc4700 1 -- 172.21.15.189:0/3644820509 >> 172.21.15.189:0/3644820509 conn(0x7f22b00d00b0 msgr2=0x7f22b00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:41.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.615+0000 7f22b4dc4700 1 -- 172.21.15.189:0/3644820509 shutdown_connections 2024-04-06T13:51:41.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.615+0000 7f22b4dc4700 1 -- 172.21.15.189:0/3644820509 wait complete. 2024-04-06T13:51:41.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.617+0000 7f22b4dc4700 1 Processor -- start 2024-04-06T13:51:41.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.617+0000 7f22b4dc4700 1 -- start start 2024-04-06T13:51:41.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.617+0000 7f22b4dc4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22b00b4d70 0x7f22b0150bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.617+0000 7f22b4dc4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22b00cf900 0x7f22b015c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.617+0000 7f22b4dc4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22b015cd50 0x7f22b006ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.617+0000 7f22b4dc4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b015d6a0 con 0x7f22b00b4d70 2024-04-06T13:51:41.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.617+0000 7f22b4dc4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b015d1e0 con 0x7f22b00cf900 2024-04-06T13:51:41.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.617+0000 7f22b4dc4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22b00ad120 con 0x7f22b015cd50 2024-04-06T13:51:41.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.618+0000 7f22ae59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22b00b4d70 0x7f22b0150bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.618+0000 7f22ae59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22b00b4d70 0x7f22b0150bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44256/0 (socket says 172.21.15.189:44256) 2024-04-06T13:51:41.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.618+0000 7f22ae59c700 1 -- 172.21.15.189:0/3724491403 learned_addr learned my addr 172.21.15.189:0/3724491403 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:41.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.618+0000 7f22aed9d700 1 --2- 172.21.15.189:0/3724491403 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22b015cd50 0x7f22b006ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.618+0000 7f22add9b700 1 --2- 172.21.15.189:0/3724491403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22b00cf900 0x7f22b015c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.618+0000 7f22ae59c700 1 -- 172.21.15.189:0/3724491403 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22b015cd50 msgr2=0x7f22b006ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.618+0000 7f22ae59c700 1 --2- 172.21.15.189:0/3724491403 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f22b015cd50 0x7f22b006ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.618+0000 7f22ae59c700 1 -- 172.21.15.189:0/3724491403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22b00cf900 msgr2=0x7f22b015c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.618+0000 7f22ae59c700 1 --2- 172.21.15.189:0/3724491403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22b00cf900 0x7f22b015c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.618+0000 7f22ae59c700 1 -- 172.21.15.189:0/3724491403 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a8009a70 con 0x7f22b00b4d70 2024-04-06T13:51:41.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.618+0000 7f22ae59c700 1 --2- 172.21.15.189:0/3724491403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22b00b4d70 0x7f22b0150bc0 secure :-1 s=READY pgs=3221 cs=0 l=1 rev1=1 crypto rx=0x7f22b006c820 tx=0x7f22a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:41.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.618+0000 7f229effd700 1 -- 172.21.15.189:0/3724491403 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a4003bc0 con 0x7f22b00b4d70 2024-04-06T13:51:41.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.618+0000 7f229effd700 1 -- 172.21.15.189:0/3724491403 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a401f070 con 0x7f22b00b4d70 2024-04-06T13:51:41.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.618+0000 7f22b4dc4700 1 -- 172.21.15.189:0/3724491403 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a40094a0 con 0x7f22b00b4d70 2024-04-06T13:51:41.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.618+0000 7f229effd700 1 -- 172.21.15.189:0/3724491403 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a4011630 con 0x7f22b00b4d70 2024-04-06T13:51:41.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.618+0000 7f22b4dc4700 1 -- 172.21.15.189:0/3724491403 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b015d3c0 con 0x7f22b00b4d70 2024-04-06T13:51:41.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.619+0000 7f22b4dc4700 1 -- 172.21.15.189:0/3724491403 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f22b006fdd0 con 0x7f22b00b4d70 2024-04-06T13:51:41.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.620+0000 7f229effd700 1 -- 172.21.15.189:0/3724491403 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f22a4021030 con 0x7f22b00b4d70 2024-04-06T13:51:41.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.620+0000 7f229effd700 1 --2- 172.21.15.189:0/3724491403 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f229006adb0 0x7f229006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.620+0000 7f229effd700 1 -- 172.21.15.189:0/3724491403 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f22a408a9e0 con 0x7f22b00b4d70 2024-04-06T13:51:41.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.620+0000 7f22add9b700 1 --2- 172.21.15.189:0/3724491403 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f229006adb0 0x7f229006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.620+0000 7f229effd700 1 --2- 172.21.15.189:0/3724491403 >> [v2:172.21.15.189:6800/47414,v1:172.21.15.189:6801/47414] conn(0x7f22900716f0 0x7f2290073b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.620+0000 7f229effd700 1 -- 172.21.15.189:0/3724491403 --> [v2:172.21.15.189:6800/47414,v1:172.21.15.189:6801/47414] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2290074220 con 0x7f22900716f0 2024-04-06T13:51:41.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.620+0000 7f229effd700 1 -- 172.21.15.189:0/3724491403 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=290) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22a408f2c0 con 0x7f22b00b4d70 2024-04-06T13:51:41.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.620+0000 7f22aed9d700 1 --2- 172.21.15.189:0/3724491403 >> [v2:172.21.15.189:6800/47414,v1:172.21.15.189:6801/47414] conn(0x7f22900716f0 0x7f2290073b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.620+0000 7f22add9b700 1 --2- 172.21.15.189:0/3724491403 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f229006adb0 0x7f229006d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f22b01548c0 tx=0x7f22a0001550 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:41.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.621+0000 7f22aed9d700 1 --2- 172.21.15.189:0/3724491403 >> [v2:172.21.15.189:6800/47414,v1:172.21.15.189:6801/47414] conn(0x7f22900716f0 0x7f2290073b50 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:41.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.621+0000 7f229effd700 1 -- 172.21.15.189:0/3724491403 <== osd.1 v2:172.21.15.189:6800/47414 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2290074220 con 0x7f22900716f0 2024-04-06T13:51:41.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.635+0000 7f22b4dc4700 1 -- 172.21.15.189:0/3724491403 --> [v2:172.21.15.189:6800/47414,v1:172.21.15.189:6801/47414] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f22b015b4a0 con 0x7f22900716f0 2024-04-06T13:51:41.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.635+0000 7f229effd700 1 -- 172.21.15.189:0/3724491403 <== osd.1 v2:172.21.15.189:6800/47414 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f22a800f040 con 0x7f22900716f0 2024-04-06T13:51:41.636 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:51:41.636 INFO:teuthology.orchestra.run.smithi189.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-06T13:51:41.636 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:51:41.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.636+0000 7f22b4dc4700 1 -- 172.21.15.189:0/3724491403 >> [v2:172.21.15.189:6800/47414,v1:172.21.15.189:6801/47414] conn(0x7f22900716f0 msgr2=0x7f2290073b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.636+0000 7f22b4dc4700 1 --2- 172.21.15.189:0/3724491403 >> [v2:172.21.15.189:6800/47414,v1:172.21.15.189:6801/47414] conn(0x7f22900716f0 0x7f2290073b50 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.636+0000 7f22b4dc4700 1 -- 172.21.15.189:0/3724491403 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f229006adb0 msgr2=0x7f229006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.636+0000 7f22b4dc4700 1 --2- 172.21.15.189:0/3724491403 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f229006adb0 0x7f229006d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f22b01548c0 tx=0x7f22a0001550 comp rx=0 tx=0).stop 2024-04-06T13:51:41.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.636+0000 7f22b4dc4700 1 -- 172.21.15.189:0/3724491403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22b00b4d70 msgr2=0x7f22b0150bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.636+0000 7f22b4dc4700 1 --2- 172.21.15.189:0/3724491403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22b00b4d70 0x7f22b0150bc0 secure :-1 s=READY pgs=3221 cs=0 l=1 rev1=1 crypto rx=0x7f22b006c820 tx=0x7f22a4004750 comp rx=0 tx=0).stop 2024-04-06T13:51:41.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.636+0000 7f22aed9d700 1 -- 172.21.15.189:0/3724491403 reap_dead start 2024-04-06T13:51:41.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.636+0000 7f22b4dc4700 1 -- 172.21.15.189:0/3724491403 shutdown_connections 2024-04-06T13:51:41.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.636+0000 7f22b4dc4700 1 -- 172.21.15.189:0/3724491403 >> 172.21.15.189:0/3724491403 conn(0x7f22b00d00b0 msgr2=0x7f22b006eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:41.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.636+0000 7f22b4dc4700 1 -- 172.21.15.189:0/3724491403 shutdown_connections 2024-04-06T13:51:41.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.636+0000 7f22b4dc4700 1 -- 172.21.15.189:0/3724491403 wait complete. 2024-04-06T13:51:41.648 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-06T13:51:41.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.771+0000 7fc7656a3700 1 Processor -- start 2024-04-06T13:51:41.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.771+0000 7fc7656a3700 1 -- start start 2024-04-06T13:51:41.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc7656a3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc760160b60 0x7fc7600b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc7656a3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc76015db10 0x7fc760142d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc7656a3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc7601432a0 0x7fc7600cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc7656a3700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc760160f40 con 0x7fc7601432a0 2024-04-06T13:51:41.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc7656a3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc76015eb40 con 0x7fc76015db10 2024-04-06T13:51:41.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc7656a3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7601436e0 con 0x7fc760160b60 2024-04-06T13:51:41.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc75f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc7601432a0 0x7fc7600cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc75f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc7601432a0 0x7fc7600cd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44258/0 (socket says 172.21.15.189:44258) 2024-04-06T13:51:41.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc75f7fe700 1 -- 172.21.15.189:0/1572099649 learned_addr learned my addr 172.21.15.189:0/1572099649 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:41.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc75effd700 1 --2- 172.21.15.189:0/1572099649 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc760160b60 0x7fc7600b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc75e7fc700 1 --2- 172.21.15.189:0/1572099649 >> v2:172.21.15.192:3300/0 conn(0x7fc76015db10 0x7fc760142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc75f7fe700 1 -- 172.21.15.189:0/1572099649 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc760160b60 msgr2=0x7fc7600b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:41.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc75f7fe700 1 --2- 172.21.15.189:0/1572099649 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc760160b60 0x7fc7600b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc75f7fe700 1 -- 172.21.15.189:0/1572099649 >> v2:172.21.15.192:3300/0 conn(0x7fc76015db10 msgr2=0x7fc760142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:41.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc75f7fe700 1 --2- 172.21.15.189:0/1572099649 >> v2:172.21.15.192:3300/0 conn(0x7fc76015db10 0x7fc760142d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc75f7fe700 1 -- 172.21.15.189:0/1572099649 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7600d0bf0 con 0x7fc7601432a0 2024-04-06T13:51:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc75effd700 1 --2- 172.21.15.189:0/1572099649 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc760160b60 0x7fc7600b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:51:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc75f7fe700 1 --2- 172.21.15.189:0/1572099649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7601432a0 0x7fc7600cd890 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 crypto rx=0x7fc7600ceda0 tx=0x7fc75800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bae18ef3f15cfb6a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc75dffb700 1 -- 172.21.15.189:0/1572099649 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75800e070 con 0x7fc7601432a0 2024-04-06T13:51:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc75dffb700 1 -- 172.21.15.189:0/1572099649 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75800b910 con 0x7fc7601432a0 2024-04-06T13:51:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.772+0000 7fc75dffb700 1 -- 172.21.15.189:0/1572099649 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75800bcc0 con 0x7fc7601432a0 2024-04-06T13:51:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.773+0000 7fc7656a3700 1 -- 172.21.15.189:0/1572099649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7601432a0 msgr2=0x7fc7600cd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.773+0000 7fc7656a3700 1 --2- 172.21.15.189:0/1572099649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7601432a0 0x7fc7600cd890 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 crypto rx=0x7fc7600ceda0 tx=0x7fc75800b330 comp rx=0 tx=0).stop 2024-04-06T13:51:41.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.773+0000 7fc7656a3700 1 -- 172.21.15.189:0/1572099649 shutdown_connections 2024-04-06T13:51:41.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.773+0000 7fc7656a3700 1 --2- 172.21.15.189:0/1572099649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7601432a0 0x7fc7600cd890 unknown :-1 s=CLOSED pgs=3222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.773+0000 7fc7656a3700 1 --2- 172.21.15.189:0/1572099649 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc760160b60 0x7fc7600b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.773+0000 7fc7656a3700 1 --2- 172.21.15.189:0/1572099649 >> v2:172.21.15.192:3300/0 conn(0x7fc76015db10 0x7fc760142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.773+0000 7fc7656a3700 1 -- 172.21.15.189:0/1572099649 >> 172.21.15.189:0/1572099649 conn(0x7fc76006a730 msgr2=0x7fc76006ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:41.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.773+0000 7fc7656a3700 1 -- 172.21.15.189:0/1572099649 shutdown_connections 2024-04-06T13:51:41.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.773+0000 7fc7656a3700 1 -- 172.21.15.189:0/1572099649 wait complete. 2024-04-06T13:51:41.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.774+0000 7fc7656a3700 1 Processor -- start 2024-04-06T13:51:41.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.774+0000 7fc7656a3700 1 -- start start 2024-04-06T13:51:41.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.775+0000 7fc7656a3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7601432a0 0x7fc7600c4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.775+0000 7fc7656a3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc76015db10 0x7fc7601f7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.775+0000 7fc7656a3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc760160b60 0x7fc7600c6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.775+0000 7fc7656a3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7601f43a0 con 0x7fc76015db10 2024-04-06T13:51:41.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.775+0000 7fc7656a3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7601f7680 con 0x7fc7601432a0 2024-04-06T13:51:41.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.775+0000 7fc7656a3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc760152de0 con 0x7fc760160b60 2024-04-06T13:51:41.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.775+0000 7fc75e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc76015db10 0x7fc7601f7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.775+0000 7fc75e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc76015db10 0x7fc7601f7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44262/0 (socket says 172.21.15.189:44262) 2024-04-06T13:51:41.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.775+0000 7fc75e7fc700 1 -- 172.21.15.189:0/1108330037 learned_addr learned my addr 172.21.15.189:0/1108330037 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:41.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.775+0000 7fc75effd700 1 --2- 172.21.15.189:0/1108330037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7601432a0 0x7fc7600c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.775+0000 7fc75f7fe700 1 --2- 172.21.15.189:0/1108330037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc760160b60 0x7fc7600c6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.775+0000 7fc75e7fc700 1 -- 172.21.15.189:0/1108330037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc760160b60 msgr2=0x7fc7600c6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.775+0000 7fc75e7fc700 1 --2- 172.21.15.189:0/1108330037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc760160b60 0x7fc7600c6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.775+0000 7fc75e7fc700 1 -- 172.21.15.189:0/1108330037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7601432a0 msgr2=0x7fc7600c4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.775+0000 7fc75e7fc700 1 --2- 172.21.15.189:0/1108330037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7601432a0 0x7fc7600c4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.775+0000 7fc75e7fc700 1 -- 172.21.15.189:0/1108330037 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc758009b30 con 0x7fc76015db10 2024-04-06T13:51:41.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.776+0000 7fc75e7fc700 1 --2- 172.21.15.189:0/1108330037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc76015db10 0x7fc7601f7140 secure :-1 s=READY pgs=3223 cs=0 l=1 rev1=1 crypto rx=0x7fc75400e960 tx=0x7fc75400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:41.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.776+0000 7fc7477fe700 1 -- 172.21.15.189:0/1108330037 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7540096f0 con 0x7fc76015db10 2024-04-06T13:51:41.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.776+0000 7fc7477fe700 1 -- 172.21.15.189:0/1108330037 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc754013680 con 0x7fc76015db10 2024-04-06T13:51:41.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.776+0000 7fc7656a3700 1 -- 172.21.15.189:0/1108330037 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7600c7810 con 0x7fc76015db10 2024-04-06T13:51:41.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.776+0000 7fc7477fe700 1 -- 172.21.15.189:0/1108330037 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc754013a30 con 0x7fc76015db10 2024-04-06T13:51:41.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.776+0000 7fc7656a3700 1 -- 172.21.15.189:0/1108330037 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7601f48f0 con 0x7fc76015db10 2024-04-06T13:51:41.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.777+0000 7fc7656a3700 1 -- 172.21.15.189:0/1108330037 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7601f4ab0 con 0x7fc76015db10 2024-04-06T13:51:41.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.777+0000 7fc7477fe700 1 -- 172.21.15.189:0/1108330037 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fc754013b90 con 0x7fc76015db10 2024-04-06T13:51:41.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.778+0000 7fc7477fe700 1 --2- 172.21.15.189:0/1108330037 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc74c06adb0 0x7fc74c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:41.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.778+0000 7fc7477fe700 1 -- 172.21.15.189:0/1108330037 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fc754089900 con 0x7fc76015db10 2024-04-06T13:51:41.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.778+0000 7fc75effd700 1 --2- 172.21.15.189:0/1108330037 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc74c06adb0 0x7fc74c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:41.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.778+0000 7fc75effd700 1 --2- 172.21.15.189:0/1108330037 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc74c06adb0 0x7fc74c06d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc76015d310 tx=0x7fc750008040 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:41.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.782+0000 7fc7477fe700 1 -- 172.21.15.189:0/1108330037 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc754054860 con 0x7fc76015db10 2024-04-06T13:51:41.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.978+0000 7fc7656a3700 1 -- 172.21.15.189:0/1108330037 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc76015e580 con 0x7fc74c06adb0 2024-04-06T13:51:41.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.978+0000 7fc7477fe700 1 -- 172.21.15.189:0/1108330037 <== mgr.10415 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc750009d80 con 0x7fc74c06adb0 2024-04-06T13:51:41.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.981+0000 7fc7656a3700 1 -- 172.21.15.189:0/1108330037 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc74c06adb0 msgr2=0x7fc74c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.981+0000 7fc7656a3700 1 --2- 172.21.15.189:0/1108330037 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc74c06adb0 0x7fc74c06d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc76015d310 tx=0x7fc750008040 comp rx=0 tx=0).stop 2024-04-06T13:51:41.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.981+0000 7fc7656a3700 1 -- 172.21.15.189:0/1108330037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc76015db10 msgr2=0x7fc7601f7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:41.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.981+0000 7fc7656a3700 1 --2- 172.21.15.189:0/1108330037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc76015db10 0x7fc7601f7140 secure :-1 s=READY pgs=3223 cs=0 l=1 rev1=1 crypto rx=0x7fc75400e960 tx=0x7fc75400ec70 comp rx=0 tx=0).stop 2024-04-06T13:51:41.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.981+0000 7fc7656a3700 1 -- 172.21.15.189:0/1108330037 shutdown_connections 2024-04-06T13:51:41.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.981+0000 7fc7656a3700 1 --2- 172.21.15.189:0/1108330037 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc74c06adb0 0x7fc74c06d270 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.981+0000 7fc7656a3700 1 --2- 172.21.15.189:0/1108330037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc760160b60 0x7fc7600c6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.981+0000 7fc7656a3700 1 --2- 172.21.15.189:0/1108330037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7601432a0 0x7fc7600c4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.981+0000 7fc7656a3700 1 --2- 172.21.15.189:0/1108330037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc76015db10 0x7fc7601f7140 unknown :-1 s=CLOSED pgs=3223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:41.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.981+0000 7fc7656a3700 1 -- 172.21.15.189:0/1108330037 >> 172.21.15.189:0/1108330037 conn(0x7fc76006a730 msgr2=0x7fc7600b1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:41.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.981+0000 7fc7656a3700 1 -- 172.21.15.189:0/1108330037 shutdown_connections 2024-04-06T13:51:41.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:41.981+0000 7fc7656a3700 1 -- 172.21.15.189:0/1108330037 wait complete. 2024-04-06T13:51:41.987 INFO:teuthology.orchestra.run.smithi189.stderr:instructed osd(s) 1 to deep-scrub 2024-04-06T13:51:41.993 INFO:tasks.ceph:Scrubbing osd.2 2024-04-06T13:51:41.993 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-06T13:51:42.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.120+0000 7f0730dd4700 1 Processor -- start 2024-04-06T13:51:42.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.120+0000 7f0730dd4700 1 -- start start 2024-04-06T13:51:42.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.121+0000 7f0730dd4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f072c15fcd0 0x7f072c15c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.121+0000 7f0730dd4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f072c15c740 con 0x7f072c06ae40 2024-04-06T13:51:42.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.121+0000 7f0730dd4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f072c15f1f0 con 0x7f072c144f40 2024-04-06T13:51:42.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.121+0000 7f0730dd4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f072c0cfcb0 con 0x7f072c15fcd0 2024-04-06T13:51:42.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.121+0000 7f0729d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f072c06ae40 0x7f072c15ba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:32872/0 (socket says 172.21.15.189:32872) 2024-04-06T13:51:42.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.121+0000 7f0729d9b700 1 -- 172.21.15.189:0/890260953 learned_addr learned my addr 172.21.15.189:0/890260953 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:42.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.121+0000 7f072ad9d700 1 --2- 172.21.15.189:0/890260953 >> v2:172.21.15.192:3300/0 conn(0x7f072c15fcd0 0x7f072c15c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.121+0000 7f072959a700 1 -- 172.21.15.189:0/890260953 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531753186 0 0) 0x7f072c15c740 con 0x7f072c06ae40 2024-04-06T13:51:42.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.121+0000 7f072959a700 1 -- 172.21.15.189:0/890260953 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07140035b0 con 0x7f072c06ae40 2024-04-06T13:51:42.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.121+0000 7f072959a700 1 -- 172.21.15.189:0/890260953 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489141677 0 0) 0x7f072c15f1f0 con 0x7f072c144f40 2024-04-06T13:51:42.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.121+0000 7f072959a700 1 -- 172.21.15.189:0/890260953 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f072c15c740 con 0x7f072c144f40 2024-04-06T13:51:42.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.121+0000 7f072ad9d700 1 -- 172.21.15.189:0/890260953 >> v1:172.21.15.192:6789/0 conn(0x7f072c144f40 legacy=0x7f072c06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.121+0000 7f072ad9d700 1 -- 172.21.15.189:0/890260953 >> v1:172.21.15.189:6789/0 conn(0x7f072c06ae40 legacy=0x7f072c15ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.121+0000 7f072959a700 1 -- 172.21.15.189:0/890260953 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1735136091 0 0) 0x7f07140035b0 con 0x7f072c06ae40 2024-04-06T13:51:42.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.121+0000 7f072ad9d700 1 -- 172.21.15.189:0/890260953 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072c15f4d0 con 0x7f072c15fcd0 2024-04-06T13:51:42.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.122+0000 7f072ad9d700 1 --2- 172.21.15.189:0/890260953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f072c15fcd0 0x7f072c15c150 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f0724009d90 tx=0x7f072400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b94c0b8479e2ed0f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:42.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.122+0000 7f072959a700 1 -- 172.21.15.189:0/890260953 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072400e070 con 0x7f072c15fcd0 2024-04-06T13:51:42.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.122+0000 7f072959a700 1 -- 172.21.15.189:0/890260953 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0724003a20 con 0x7f072c15fcd0 2024-04-06T13:51:42.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.122+0000 7f072959a700 1 -- 172.21.15.189:0/890260953 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0720001670 con 0x7f072c15fcd0 2024-04-06T13:51:42.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.122+0000 7f0730dd4700 1 -- 172.21.15.189:0/890260953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f072c15fcd0 msgr2=0x7f072c15c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.122+0000 7f0730dd4700 1 --2- 172.21.15.189:0/890260953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f072c15fcd0 0x7f072c15c150 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f0724009d90 tx=0x7f072400b330 comp rx=0 tx=0).stop 2024-04-06T13:51:42.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.122+0000 7f0730dd4700 1 -- 172.21.15.189:0/890260953 shutdown_connections 2024-04-06T13:51:42.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.122+0000 7f0730dd4700 1 --2- 172.21.15.189:0/890260953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f072c15fcd0 0x7f072c15c150 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.122+0000 7f0730dd4700 1 -- 172.21.15.189:0/890260953 >> 172.21.15.189:0/890260953 conn(0x7f072c0d00b0 msgr2=0x7f072c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:42.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.122+0000 7f0730dd4700 1 -- 172.21.15.189:0/890260953 shutdown_connections 2024-04-06T13:51:42.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.122+0000 7f0730dd4700 1 -- 172.21.15.189:0/890260953 wait complete. 2024-04-06T13:51:42.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.124+0000 7f0730dd4700 1 Processor -- start 2024-04-06T13:51:42.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.124+0000 7f0730dd4700 1 -- start start 2024-04-06T13:51:42.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.124+0000 7f0730dd4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f072c06ae40 0x7f072c1647e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.124+0000 7f0730dd4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f072c144f40 0x7f072c140570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.124+0000 7f0730dd4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f072c15fcd0 0x7f072c140ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.124+0000 7f0730dd4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c141d40 con 0x7f072c06ae40 2024-04-06T13:51:42.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.124+0000 7f0730dd4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c141130 con 0x7f072c144f40 2024-04-06T13:51:42.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.124+0000 7f0730dd4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072c141410 con 0x7f072c15fcd0 2024-04-06T13:51:42.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.125+0000 7f072a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f072c06ae40 0x7f072c1647e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.125+0000 7f072a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f072c06ae40 0x7f072c1647e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44276/0 (socket says 172.21.15.189:44276) 2024-04-06T13:51:42.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.125+0000 7f072a59c700 1 -- 172.21.15.189:0/3673979222 learned_addr learned my addr 172.21.15.189:0/3673979222 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:42.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.125+0000 7f0729d9b700 1 --2- 172.21.15.189:0/3673979222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f072c144f40 0x7f072c140570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.125+0000 7f072ad9d700 1 --2- 172.21.15.189:0/3673979222 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f072c15fcd0 0x7f072c140ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.125+0000 7f072a59c700 1 -- 172.21.15.189:0/3673979222 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f072c15fcd0 msgr2=0x7f072c140ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.125+0000 7f072a59c700 1 --2- 172.21.15.189:0/3673979222 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f072c15fcd0 0x7f072c140ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.125+0000 7f072a59c700 1 -- 172.21.15.189:0/3673979222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f072c144f40 msgr2=0x7f072c140570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.125+0000 7f072a59c700 1 --2- 172.21.15.189:0/3673979222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f072c144f40 0x7f072c140570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.125+0000 7f072a59c700 1 -- 172.21.15.189:0/3673979222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0724009a70 con 0x7f072c06ae40 2024-04-06T13:51:42.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.125+0000 7f072a59c700 1 --2- 172.21.15.189:0/3673979222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f072c06ae40 0x7f072c1647e0 secure :-1 s=READY pgs=3225 cs=0 l=1 rev1=1 crypto rx=0x7f071c00c520 tx=0x7f071c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:42.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.125+0000 7f071affd700 1 -- 172.21.15.189:0/3673979222 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071c0052d0 con 0x7f072c06ae40 2024-04-06T13:51:42.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.125+0000 7f071affd700 1 -- 172.21.15.189:0/3673979222 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071c005430 con 0x7f072c06ae40 2024-04-06T13:51:42.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.125+0000 7f0730dd4700 1 -- 172.21.15.189:0/3673979222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f072c1416f0 con 0x7f072c06ae40 2024-04-06T13:51:42.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.125+0000 7f071affd700 1 -- 172.21.15.189:0/3673979222 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071c0038c0 con 0x7f072c06ae40 2024-04-06T13:51:42.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.125+0000 7f0730dd4700 1 -- 172.21.15.189:0/3673979222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f072c13ce60 con 0x7f072c06ae40 2024-04-06T13:51:42.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.126+0000 7f0730dd4700 1 -- 172.21.15.189:0/3673979222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f072c067bb0 con 0x7f072c06ae40 2024-04-06T13:51:42.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.127+0000 7f071affd700 1 -- 172.21.15.189:0/3673979222 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f071c003a20 con 0x7f072c06ae40 2024-04-06T13:51:42.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.127+0000 7f071affd700 1 --2- 172.21.15.189:0/3673979222 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f070c06adb0 0x7f070c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.127+0000 7f071affd700 1 -- 172.21.15.189:0/3673979222 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f071c088a60 con 0x7f072c06ae40 2024-04-06T13:51:42.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.127+0000 7f0729d9b700 1 --2- 172.21.15.189:0/3673979222 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f070c06adb0 0x7f070c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.127+0000 7f071affd700 1 --2- 172.21.15.189:0/3673979222 >> [v2:172.21.15.189:6816/47416,v1:172.21.15.189:6817/47416] conn(0x7f070c0716f0 0x7f070c073b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.127+0000 7f071affd700 1 -- 172.21.15.189:0/3673979222 --> [v2:172.21.15.189:6816/47416,v1:172.21.15.189:6817/47416] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f070c074220 con 0x7f070c0716f0 2024-04-06T13:51:42.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.127+0000 7f071affd700 1 -- 172.21.15.189:0/3673979222 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=290) v2 ==== 24+0+0 (secure 0 0 0) 0x7f071c08d2c0 con 0x7f072c06ae40 2024-04-06T13:51:42.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.127+0000 7f072ad9d700 1 --2- 172.21.15.189:0/3673979222 >> [v2:172.21.15.189:6816/47416,v1:172.21.15.189:6817/47416] conn(0x7f070c0716f0 0x7f070c073b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.127+0000 7f0729d9b700 1 --2- 172.21.15.189:0/3673979222 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f070c06adb0 0x7f070c06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f072c0cd510 tx=0x7f0720001e40 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:42.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.128+0000 7f072ad9d700 1 --2- 172.21.15.189:0/3673979222 >> [v2:172.21.15.189:6816/47416,v1:172.21.15.189:6817/47416] conn(0x7f070c0716f0 0x7f070c073b50 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:42.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.128+0000 7f071affd700 1 -- 172.21.15.189:0/3673979222 <== osd.2 v2:172.21.15.189:6816/47416 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f070c074220 con 0x7f070c0716f0 2024-04-06T13:51:42.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.142+0000 7f0730dd4700 1 -- 172.21.15.189:0/3673979222 --> [v2:172.21.15.189:6816/47416,v1:172.21.15.189:6817/47416] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f072c141e80 con 0x7f070c0716f0 2024-04-06T13:51:42.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.143+0000 7f071affd700 1 -- 172.21.15.189:0/3673979222 <== osd.2 v2:172.21.15.189:6816/47416 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f0724009470 con 0x7f070c0716f0 2024-04-06T13:51:42.143 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:51:42.143 INFO:teuthology.orchestra.run.smithi189.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-06T13:51:42.143 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:51:42.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.143+0000 7f0730dd4700 1 -- 172.21.15.189:0/3673979222 >> [v2:172.21.15.189:6816/47416,v1:172.21.15.189:6817/47416] conn(0x7f070c0716f0 msgr2=0x7f070c073b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.143+0000 7f0730dd4700 1 --2- 172.21.15.189:0/3673979222 >> [v2:172.21.15.189:6816/47416,v1:172.21.15.189:6817/47416] conn(0x7f070c0716f0 0x7f070c073b50 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.143+0000 7f0730dd4700 1 -- 172.21.15.189:0/3673979222 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f070c06adb0 msgr2=0x7f070c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.143+0000 7f0730dd4700 1 --2- 172.21.15.189:0/3673979222 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f070c06adb0 0x7f070c06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f072c0cd510 tx=0x7f0720001e40 comp rx=0 tx=0).stop 2024-04-06T13:51:42.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.143+0000 7f0730dd4700 1 -- 172.21.15.189:0/3673979222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f072c06ae40 msgr2=0x7f072c1647e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.143+0000 7f0730dd4700 1 --2- 172.21.15.189:0/3673979222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f072c06ae40 0x7f072c1647e0 secure :-1 s=READY pgs=3225 cs=0 l=1 rev1=1 crypto rx=0x7f071c00c520 tx=0x7f071c00c830 comp rx=0 tx=0).stop 2024-04-06T13:51:42.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.143+0000 7f072ad9d700 1 -- 172.21.15.189:0/3673979222 reap_dead start 2024-04-06T13:51:42.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.143+0000 7f0730dd4700 1 -- 172.21.15.189:0/3673979222 shutdown_connections 2024-04-06T13:51:42.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.143+0000 7f0730dd4700 1 -- 172.21.15.189:0/3673979222 >> 172.21.15.189:0/3673979222 conn(0x7f072c0d00b0 msgr2=0x7f072c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:42.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.143+0000 7f0730dd4700 1 -- 172.21.15.189:0/3673979222 shutdown_connections 2024-04-06T13:51:42.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.143+0000 7f0730dd4700 1 -- 172.21.15.189:0/3673979222 wait complete. 2024-04-06T13:51:42.155 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-06T13:51:42.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.278+0000 7f1db1dc0700 1 Processor -- start 2024-04-06T13:51:42.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.278+0000 7f1db1dc0700 1 -- start start 2024-04-06T13:51:42.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.279+0000 7f1db1dc0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1dac0b4660 0x7f1dac0b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.279+0000 7f1db1dc0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac0cdbe0 con 0x7f1dac0b2fc0 2024-04-06T13:51:42.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.279+0000 7f1db1dc0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac0b5000 con 0x7f1dac15a550 2024-04-06T13:51:42.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.279+0000 7f1db1dc0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac15a940 con 0x7f1dac0b4660 2024-04-06T13:51:42.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.279+0000 7f1dabfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1dac0b2fc0 0x7f1dac0b3430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:32884/0 (socket says 172.21.15.189:32884) 2024-04-06T13:51:42.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.279+0000 7f1dabfff700 1 -- 172.21.15.189:0/2007798215 learned_addr learned my addr 172.21.15.189:0/2007798215 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:42.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.279+0000 7f1daaffd700 1 --2- 172.21.15.189:0/2007798215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1dac0b4660 0x7f1dac0b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daa7fc700 1 -- 172.21.15.189:0/2007798215 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969126548 0 0) 0x7f1dac0cdbe0 con 0x7f1dac0b2fc0 2024-04-06T13:51:42.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daa7fc700 1 -- 172.21.15.189:0/2007798215 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d940035b0 con 0x7f1dac0b2fc0 2024-04-06T13:51:42.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daa7fc700 1 -- 172.21.15.189:0/2007798215 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086604044 0 0) 0x7f1dac0b5000 con 0x7f1dac15a550 2024-04-06T13:51:42.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daa7fc700 1 -- 172.21.15.189:0/2007798215 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dac0cdbe0 con 0x7f1dac15a550 2024-04-06T13:51:42.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daa7fc700 1 -- 172.21.15.189:0/2007798215 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1478172383 0 0) 0x7f1d940035b0 con 0x7f1dac0b2fc0 2024-04-06T13:51:42.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daa7fc700 1 -- 172.21.15.189:0/2007798215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1dac0b4660 msgr2=0x7f1dac0b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:42.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daa7fc700 1 --2- 172.21.15.189:0/2007798215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1dac0b4660 0x7f1dac0b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daa7fc700 1 -- 172.21.15.189:0/2007798215 >> v1:172.21.15.192:6789/0 conn(0x7f1dac15a550 legacy=0x7f1dac0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daa7fc700 1 -- 172.21.15.189:0/2007798215 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dac15ac00 con 0x7f1dac0b2fc0 2024-04-06T13:51:42.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daa7fc700 1 -- 172.21.15.189:0/2007798215 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f1da40019f0 con 0x7f1dac0b2fc0 2024-04-06T13:51:42.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daa7fc700 1 -- 172.21.15.189:0/2007798215 >> v1:172.21.15.189:6789/0 conn(0x7f1dac0b2fc0 legacy=0x7f1dac0b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daa7fc700 1 --2- 172.21.15.189:0/2007798215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1dac0b4660 0x7f1d94003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daa7fc700 1 --2- 172.21.15.189:0/2007798215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d94004120 0x7f1d94004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daa7fc700 1 --2- 172.21.15.189:0/2007798215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d94004d90 0x7f1d94005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.295 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daa7fc700 1 -- 172.21.15.189:0/2007798215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac15a940 con 0x7f1d94004120 2024-04-06T13:51:42.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daa7fc700 1 -- 172.21.15.189:0/2007798215 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9400baf0 con 0x7f1d94004d90 2024-04-06T13:51:42.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daa7fc700 1 -- 172.21.15.189:0/2007798215 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9400bcb0 con 0x7f1dac0b4660 2024-04-06T13:51:42.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daaffd700 1 --2- 172.21.15.189:0/2007798215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d94004120 0x7f1d94004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1dab7fe700 1 --2- 172.21.15.189:0/2007798215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1dac0b4660 0x7f1d94003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1dabfff700 1 --2- 172.21.15.189:0/2007798215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d94004d90 0x7f1d94005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daaffd700 1 -- 172.21.15.189:0/2007798215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1dac0b4660 msgr2=0x7f1d94003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:42.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daaffd700 1 --2- 172.21.15.189:0/2007798215 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1dac0b4660 0x7f1d94003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daaffd700 1 -- 172.21.15.189:0/2007798215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d94004d90 msgr2=0x7f1d94005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:42.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daaffd700 1 --2- 172.21.15.189:0/2007798215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d94004d90 0x7f1d94005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daaffd700 1 -- 172.21.15.189:0/2007798215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d940035b0 con 0x7f1d94004120 2024-04-06T13:51:42.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.280+0000 7f1daaffd700 1 --2- 172.21.15.189:0/2007798215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d94004120 0x7f1d94004850 secure :-1 s=READY pgs=3227 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c009b60 tx=0x7f1d94005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac311e6ce8e27f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:42.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.281+0000 7f1daa7fc700 1 -- 172.21.15.189:0/2007798215 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c00e6c0 con 0x7f1d94004120 2024-04-06T13:51:42.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.281+0000 7f1daa7fc700 1 -- 172.21.15.189:0/2007798215 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d9c00e820 con 0x7f1d94004120 2024-04-06T13:51:42.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.281+0000 7f1daa7fc700 1 -- 172.21.15.189:0/2007798215 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c00eb90 con 0x7f1d94004120 2024-04-06T13:51:42.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.281+0000 7f1db1dc0700 1 -- 172.21.15.189:0/2007798215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d94004120 msgr2=0x7f1d94004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.281+0000 7f1db1dc0700 1 --2- 172.21.15.189:0/2007798215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d94004120 0x7f1d94004850 secure :-1 s=READY pgs=3227 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c009b60 tx=0x7f1d94005880 comp rx=0 tx=0).stop 2024-04-06T13:51:42.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.281+0000 7f1dabfff700 1 -- 172.21.15.189:0/2007798215 reap_dead start 2024-04-06T13:51:42.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.281+0000 7f1db1dc0700 1 -- 172.21.15.189:0/2007798215 shutdown_connections 2024-04-06T13:51:42.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.281+0000 7f1db1dc0700 1 -- 172.21.15.189:0/2007798215 >> 172.21.15.189:0/2007798215 conn(0x7f1dac0d00b0 msgr2=0x7f1dac0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:42.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.281+0000 7f1db1dc0700 1 -- 172.21.15.189:0/2007798215 shutdown_connections 2024-04-06T13:51:42.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.281+0000 7f1db1dc0700 1 -- 172.21.15.189:0/2007798215 wait complete. 2024-04-06T13:51:42.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.282+0000 7f1db1dc0700 1 Processor -- start 2024-04-06T13:51:42.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.282+0000 7f1db1dc0700 1 -- start start 2024-04-06T13:51:42.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.283+0000 7f1db1dc0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d94004120 0x7f1dac141300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.283+0000 7f1db1dc0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1dac13e8b0 0x7f1dac1506f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.283+0000 7f1db1dc0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dac13fe10 0x7f1dac1402a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.283+0000 7f1db1dc0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac06aa80 con 0x7f1dac13fe10 2024-04-06T13:51:42.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.283+0000 7f1db1dc0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac141840 con 0x7f1dac13e8b0 2024-04-06T13:51:42.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.283+0000 7f1db1dc0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac13ed70 con 0x7f1d94004120 2024-04-06T13:51:42.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.283+0000 7f1dabfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dac13fe10 0x7f1dac1402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.283+0000 7f1dabfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dac13fe10 0x7f1dac1402a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44302/0 (socket says 172.21.15.189:44302) 2024-04-06T13:51:42.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.283+0000 7f1dabfff700 1 -- 172.21.15.189:0/3625667229 learned_addr learned my addr 172.21.15.189:0/3625667229 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:42.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.283+0000 7f1dab7fe700 1 --2- 172.21.15.189:0/3625667229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d94004120 0x7f1dac141300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.283+0000 7f1daaffd700 1 --2- 172.21.15.189:0/3625667229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1dac13e8b0 0x7f1dac1506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.284+0000 7f1dabfff700 1 -- 172.21.15.189:0/3625667229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d94004120 msgr2=0x7f1dac141300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.284+0000 7f1dabfff700 1 --2- 172.21.15.189:0/3625667229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d94004120 0x7f1dac141300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.284+0000 7f1dabfff700 1 -- 172.21.15.189:0/3625667229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1dac13e8b0 msgr2=0x7f1dac1506f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.284+0000 7f1dabfff700 1 --2- 172.21.15.189:0/3625667229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1dac13e8b0 0x7f1dac1506f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.284+0000 7f1dabfff700 1 -- 172.21.15.189:0/3625667229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dac13f050 con 0x7f1dac13fe10 2024-04-06T13:51:42.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.284+0000 7f1dabfff700 1 --2- 172.21.15.189:0/3625667229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dac13fe10 0x7f1dac1402a0 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 crypto rx=0x7f1da4000f80 tx=0x7f1da400a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:42.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.284+0000 7f1d93fff700 1 -- 172.21.15.189:0/3625667229 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da400ad20 con 0x7f1dac13fe10 2024-04-06T13:51:42.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.284+0000 7f1d93fff700 1 -- 172.21.15.189:0/3625667229 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da40133f0 con 0x7f1dac13fe10 2024-04-06T13:51:42.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.284+0000 7f1db1dc0700 1 -- 172.21.15.189:0/3625667229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dac13f330 con 0x7f1dac13fe10 2024-04-06T13:51:42.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.284+0000 7f1d93fff700 1 -- 172.21.15.189:0/3625667229 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da40137a0 con 0x7f1dac13fe10 2024-04-06T13:51:42.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.284+0000 7f1db1dc0700 1 -- 172.21.15.189:0/3625667229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac13f8f0 con 0x7f1dac13fe10 2024-04-06T13:51:42.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.285+0000 7f1db1dc0700 1 -- 172.21.15.189:0/3625667229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dac067bb0 con 0x7f1dac13fe10 2024-04-06T13:51:42.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.285+0000 7f1d93fff700 1 -- 172.21.15.189:0/3625667229 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f1da4013900 con 0x7f1dac13fe10 2024-04-06T13:51:42.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.286+0000 7f1d93fff700 1 --2- 172.21.15.189:0/3625667229 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1d9806adb0 0x7f1d9806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.286+0000 7f1d93fff700 1 -- 172.21.15.189:0/3625667229 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f1da4089dc0 con 0x7f1dac13fe10 2024-04-06T13:51:42.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.286+0000 7f1dab7fe700 1 --2- 172.21.15.189:0/3625667229 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1d9806adb0 0x7f1d9806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.286+0000 7f1dab7fe700 1 --2- 172.21.15.189:0/3625667229 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1d9806adb0 0x7f1d9806d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1da0007fd0 tx=0x7f1da0009040 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:42.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.290+0000 7f1d93fff700 1 -- 172.21.15.189:0/3625667229 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1da4054c90 con 0x7f1dac13fe10 2024-04-06T13:51:42.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.480+0000 7f1db1dc0700 1 -- 172.21.15.189:0/3625667229 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1dac15b4a0 con 0x7f1d9806adb0 2024-04-06T13:51:42.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.481+0000 7f1d93fff700 1 -- 172.21.15.189:0/3625667229 <== mgr.10415 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1da0001710 con 0x7f1d9806adb0 2024-04-06T13:51:42.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.483+0000 7f1db1dc0700 1 -- 172.21.15.189:0/3625667229 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1d9806adb0 msgr2=0x7f1d9806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.483+0000 7f1db1dc0700 1 --2- 172.21.15.189:0/3625667229 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1d9806adb0 0x7f1d9806d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1da0007fd0 tx=0x7f1da0009040 comp rx=0 tx=0).stop 2024-04-06T13:51:42.483 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.483+0000 7f1db1dc0700 1 -- 172.21.15.189:0/3625667229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dac13fe10 msgr2=0x7f1dac1402a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.483+0000 7f1db1dc0700 1 --2- 172.21.15.189:0/3625667229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dac13fe10 0x7f1dac1402a0 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 crypto rx=0x7f1da4000f80 tx=0x7f1da400a5b0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.483+0000 7f1db1dc0700 1 -- 172.21.15.189:0/3625667229 shutdown_connections 2024-04-06T13:51:42.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.483+0000 7f1db1dc0700 1 --2- 172.21.15.189:0/3625667229 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f1d9806adb0 0x7f1d9806d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.483+0000 7f1db1dc0700 1 --2- 172.21.15.189:0/3625667229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dac13fe10 0x7f1dac1402a0 unknown :-1 s=CLOSED pgs=3228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.483+0000 7f1db1dc0700 1 --2- 172.21.15.189:0/3625667229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d94004120 0x7f1dac141300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.483+0000 7f1db1dc0700 1 --2- 172.21.15.189:0/3625667229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1dac13e8b0 0x7f1dac1506f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.483+0000 7f1db1dc0700 1 -- 172.21.15.189:0/3625667229 >> 172.21.15.189:0/3625667229 conn(0x7f1dac0d00b0 msgr2=0x7f1dac0cf1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:42.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.483+0000 7f1db1dc0700 1 -- 172.21.15.189:0/3625667229 shutdown_connections 2024-04-06T13:51:42.484 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.483+0000 7f1db1dc0700 1 -- 172.21.15.189:0/3625667229 wait complete. 2024-04-06T13:51:42.485 INFO:teuthology.orchestra.run.smithi189.stderr:instructed osd(s) 2 to deep-scrub 2024-04-06T13:51:42.495 INFO:tasks.ceph:Scrubbing osd.3 2024-04-06T13:51:42.495 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-06T13:51:42.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.621+0000 7fb22c6b2700 1 Processor -- start 2024-04-06T13:51:42.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.622+0000 7fb22c6b2700 1 -- start start 2024-04-06T13:51:42.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.623+0000 7fb22c6b2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb22415a550 0x7fb2240cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.623+0000 7fb22c6b2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb2240cf900 0x7fb2240b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.623+0000 7fb22c6b2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2240cdbe0 con 0x7fb2240b4ba0 2024-04-06T13:51:42.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.623+0000 7fb22c6b2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2240b5010 con 0x7fb2240cf900 2024-04-06T13:51:42.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.623+0000 7fb22c6b2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb22415a940 con 0x7fb22415a550 2024-04-06T13:51:42.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.623+0000 7fb22ac4f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb2240b4ba0 0x7fb22406eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:32886/0 (socket says 172.21.15.189:32886) 2024-04-06T13:51:42.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.623+0000 7fb22ac4f700 1 -- 172.21.15.189:0/3172652804 learned_addr learned my addr 172.21.15.189:0/3172652804 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:42.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.623+0000 7fb229c4d700 1 --2- 172.21.15.189:0/3172652804 >> v2:172.21.15.192:3300/0 conn(0x7fb2240cf900 0x7fb2240b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.623+0000 7fb22a44e700 1 --2- 172.21.15.189:0/3172652804 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb22415a550 0x7fb2240cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.623+0000 7fb22944c700 1 -- 172.21.15.189:0/3172652804 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256294467 0 0) 0x7fb2240cdbe0 con 0x7fb2240b4ba0 2024-04-06T13:51:42.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.623+0000 7fb22944c700 1 -- 172.21.15.189:0/3172652804 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2100035b0 con 0x7fb2240b4ba0 2024-04-06T13:51:42.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.623+0000 7fb22a44e700 1 -- 172.21.15.189:0/3172652804 >> v2:172.21.15.192:3300/0 conn(0x7fb2240cf900 msgr2=0x7fb2240b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:42.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.623+0000 7fb22a44e700 1 --2- 172.21.15.189:0/3172652804 >> v2:172.21.15.192:3300/0 conn(0x7fb2240cf900 0x7fb2240b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.623+0000 7fb22a44e700 1 -- 172.21.15.189:0/3172652804 >> v1:172.21.15.189:6789/0 conn(0x7fb2240b4ba0 legacy=0x7fb22406eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.623+0000 7fb22a44e700 1 -- 172.21.15.189:0/3172652804 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22415ac00 con 0x7fb22415a550 2024-04-06T13:51:42.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.623+0000 7fb229c4d700 1 --2- 172.21.15.189:0/3172652804 >> v2:172.21.15.192:3300/0 conn(0x7fb2240cf900 0x7fb2240b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:51:42.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.623+0000 7fb22a44e700 1 --2- 172.21.15.189:0/3172652804 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb22415a550 0x7fb2240cf3c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb214009840 tx=0x7fb214009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3600bca2fcbeaacf server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:42.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.624+0000 7fb22944c700 1 -- 172.21.15.189:0/3172652804 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb214011830 con 0x7fb22415a550 2024-04-06T13:51:42.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.624+0000 7fb22944c700 1 -- 172.21.15.189:0/3172652804 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb214011990 con 0x7fb22415a550 2024-04-06T13:51:42.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.624+0000 7fb22944c700 1 -- 172.21.15.189:0/3172652804 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb214011d20 con 0x7fb22415a550 2024-04-06T13:51:42.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.624+0000 7fb22c6b2700 1 -- 172.21.15.189:0/3172652804 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb22415a550 msgr2=0x7fb2240cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.624+0000 7fb22c6b2700 1 --2- 172.21.15.189:0/3172652804 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb22415a550 0x7fb2240cf3c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb214009840 tx=0x7fb214009b50 comp rx=0 tx=0).stop 2024-04-06T13:51:42.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.624+0000 7fb22c6b2700 1 -- 172.21.15.189:0/3172652804 shutdown_connections 2024-04-06T13:51:42.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.624+0000 7fb22c6b2700 1 --2- 172.21.15.189:0/3172652804 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb22415a550 0x7fb2240cf3c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.624+0000 7fb22c6b2700 1 --2- 172.21.15.189:0/3172652804 >> v2:172.21.15.192:3300/0 conn(0x7fb2240cf900 0x7fb2240b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.624+0000 7fb22c6b2700 1 -- 172.21.15.189:0/3172652804 >> 172.21.15.189:0/3172652804 conn(0x7fb2240d00b0 msgr2=0x7fb2240d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:42.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.624+0000 7fb22c6b2700 1 -- 172.21.15.189:0/3172652804 shutdown_connections 2024-04-06T13:51:42.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.624+0000 7fb22c6b2700 1 -- 172.21.15.189:0/3172652804 wait complete. 2024-04-06T13:51:42.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.625+0000 7fb22c6b2700 1 Processor -- start 2024-04-06T13:51:42.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.625+0000 7fb22c6b2700 1 -- start start 2024-04-06T13:51:42.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.626+0000 7fb22c6b2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2240b4ba0 0x7fb224159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.626+0000 7fb22c6b2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2240cf900 0x7fb2240c5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.626+0000 7fb22c6b2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2241f60a0 0x7fb2240b0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.626+0000 7fb22c6b2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2240c5e80 con 0x7fb2240b4ba0 2024-04-06T13:51:42.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.626+0000 7fb22c6b2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2240c60e0 con 0x7fb2240cf900 2024-04-06T13:51:42.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.626+0000 7fb22c6b2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2240c63c0 con 0x7fb2241f60a0 2024-04-06T13:51:42.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.626+0000 7fb22a44e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2240b4ba0 0x7fb224159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.626+0000 7fb22a44e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2240b4ba0 0x7fb224159130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44318/0 (socket says 172.21.15.189:44318) 2024-04-06T13:51:42.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.626+0000 7fb22a44e700 1 -- 172.21.15.189:0/4010934806 learned_addr learned my addr 172.21.15.189:0/4010934806 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:42.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.626+0000 7fb229c4d700 1 --2- 172.21.15.189:0/4010934806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2240cf900 0x7fb2240c5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.626+0000 7fb22ac4f700 1 --2- 172.21.15.189:0/4010934806 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2241f60a0 0x7fb2240b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.626+0000 7fb22a44e700 1 -- 172.21.15.189:0/4010934806 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2241f60a0 msgr2=0x7fb2240b0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.626+0000 7fb22a44e700 1 --2- 172.21.15.189:0/4010934806 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2241f60a0 0x7fb2240b0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.626+0000 7fb22a44e700 1 -- 172.21.15.189:0/4010934806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2240cf900 msgr2=0x7fb2240c5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.626+0000 7fb22a44e700 1 --2- 172.21.15.189:0/4010934806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2240cf900 0x7fb2240c5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.627+0000 7fb22a44e700 1 -- 172.21.15.189:0/4010934806 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2200094a0 con 0x7fb2240b4ba0 2024-04-06T13:51:42.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.627+0000 7fb22ac4f700 1 --2- 172.21.15.189:0/4010934806 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb2241f60a0 0x7fb2240b0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:51:42.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.627+0000 7fb22a44e700 1 --2- 172.21.15.189:0/4010934806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2240b4ba0 0x7fb224159130 secure :-1 s=READY pgs=3230 cs=0 l=1 rev1=1 crypto rx=0x7fb214014040 tx=0x7fb214004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:42.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.627+0000 7fb21affd700 1 -- 172.21.15.189:0/4010934806 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2140036a0 con 0x7fb2240b4ba0 2024-04-06T13:51:42.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.627+0000 7fb21affd700 1 -- 172.21.15.189:0/4010934806 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21401f070 con 0x7fb2240b4ba0 2024-04-06T13:51:42.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.627+0000 7fb22c6b2700 1 -- 172.21.15.189:0/4010934806 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2140094a0 con 0x7fb2240b4ba0 2024-04-06T13:51:42.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.627+0000 7fb21affd700 1 -- 172.21.15.189:0/4010934806 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb214003a20 con 0x7fb2240b4ba0 2024-04-06T13:51:42.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.627+0000 7fb22c6b2700 1 -- 172.21.15.189:0/4010934806 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2240c6980 con 0x7fb2240b4ba0 2024-04-06T13:51:42.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.627+0000 7fb22c6b2700 1 -- 172.21.15.189:0/4010934806 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb22415b4a0 con 0x7fb2240b4ba0 2024-04-06T13:51:42.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.628+0000 7fb21affd700 1 -- 172.21.15.189:0/4010934806 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fb214003b80 con 0x7fb2240b4ba0 2024-04-06T13:51:42.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.629+0000 7fb21affd700 1 --2- 172.21.15.189:0/4010934806 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb20806adb0 0x7fb20806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.629+0000 7fb21affd700 1 -- 172.21.15.189:0/4010934806 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fb214021070 con 0x7fb2240b4ba0 2024-04-06T13:51:42.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.629+0000 7fb229c4d700 1 --2- 172.21.15.189:0/4010934806 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb20806adb0 0x7fb20806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.629+0000 7fb21affd700 1 --2- 172.21.15.189:0/4010934806 >> [v2:172.21.15.189:6824/47425,v1:172.21.15.189:6825/47425] conn(0x7fb2080716f0 0x7fb208073b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.629+0000 7fb21affd700 1 -- 172.21.15.189:0/4010934806 --> [v2:172.21.15.189:6824/47425,v1:172.21.15.189:6825/47425] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb208074220 con 0x7fb2080716f0 2024-04-06T13:51:42.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.629+0000 7fb21affd700 1 -- 172.21.15.189:0/4010934806 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=290) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb21408b850 con 0x7fb2240b4ba0 2024-04-06T13:51:42.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.629+0000 7fb22ac4f700 1 --2- 172.21.15.189:0/4010934806 >> [v2:172.21.15.189:6824/47425,v1:172.21.15.189:6825/47425] conn(0x7fb2080716f0 0x7fb208073b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.629+0000 7fb229c4d700 1 --2- 172.21.15.189:0/4010934806 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb20806adb0 0x7fb20806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb2240adde0 tx=0x7fb220009210 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:42.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.629+0000 7fb22ac4f700 1 --2- 172.21.15.189:0/4010934806 >> [v2:172.21.15.189:6824/47425,v1:172.21.15.189:6825/47425] conn(0x7fb2080716f0 0x7fb208073b50 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:42.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.630+0000 7fb21affd700 1 -- 172.21.15.189:0/4010934806 <== osd.3 v2:172.21.15.189:6824/47425 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb208074220 con 0x7fb2080716f0 2024-04-06T13:51:42.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.645+0000 7fb22c6b2700 1 -- 172.21.15.189:0/4010934806 --> [v2:172.21.15.189:6824/47425,v1:172.21.15.189:6825/47425] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb22415b1c0 con 0x7fb2080716f0 2024-04-06T13:51:42.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.646+0000 7fb21affd700 1 -- 172.21.15.189:0/4010934806 <== osd.3 v2:172.21.15.189:6824/47425 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb21c0070e0 con 0x7fb2080716f0 2024-04-06T13:51:42.646 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:51:42.646 INFO:teuthology.orchestra.run.smithi189.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-06T13:51:42.647 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:51:42.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.646+0000 7fb22c6b2700 1 -- 172.21.15.189:0/4010934806 >> [v2:172.21.15.189:6824/47425,v1:172.21.15.189:6825/47425] conn(0x7fb2080716f0 msgr2=0x7fb208073b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.646+0000 7fb22c6b2700 1 --2- 172.21.15.189:0/4010934806 >> [v2:172.21.15.189:6824/47425,v1:172.21.15.189:6825/47425] conn(0x7fb2080716f0 0x7fb208073b50 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.646+0000 7fb22c6b2700 1 -- 172.21.15.189:0/4010934806 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb20806adb0 msgr2=0x7fb20806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.646+0000 7fb22c6b2700 1 --2- 172.21.15.189:0/4010934806 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb20806adb0 0x7fb20806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb2240adde0 tx=0x7fb220009210 comp rx=0 tx=0).stop 2024-04-06T13:51:42.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.646+0000 7fb22c6b2700 1 -- 172.21.15.189:0/4010934806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2240b4ba0 msgr2=0x7fb224159130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.646+0000 7fb22c6b2700 1 --2- 172.21.15.189:0/4010934806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2240b4ba0 0x7fb224159130 secure :-1 s=READY pgs=3230 cs=0 l=1 rev1=1 crypto rx=0x7fb214014040 tx=0x7fb214004750 comp rx=0 tx=0).stop 2024-04-06T13:51:42.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.646+0000 7fb22ac4f700 1 -- 172.21.15.189:0/4010934806 reap_dead start 2024-04-06T13:51:42.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.646+0000 7fb22c6b2700 1 -- 172.21.15.189:0/4010934806 shutdown_connections 2024-04-06T13:51:42.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.646+0000 7fb22c6b2700 1 -- 172.21.15.189:0/4010934806 >> 172.21.15.189:0/4010934806 conn(0x7fb2240d00b0 msgr2=0x7fb2240b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:42.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.646+0000 7fb22c6b2700 1 -- 172.21.15.189:0/4010934806 shutdown_connections 2024-04-06T13:51:42.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.646+0000 7fb22c6b2700 1 -- 172.21.15.189:0/4010934806 wait complete. 2024-04-06T13:51:42.658 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-06T13:51:42.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.781+0000 7fa8e860b700 1 Processor -- start 2024-04-06T13:51:42.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.782+0000 7fa8e860b700 1 -- start start 2024-04-06T13:51:42.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.783+0000 7fa8e860b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e015a550 0x7fa8e00cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.783+0000 7fa8e860b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa8e00cf900 0x7fa8e00b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.783+0000 7fa8e860b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e00cdbe0 con 0x7fa8e00b4ba0 2024-04-06T13:51:42.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.783+0000 7fa8e860b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8e00b5010 con 0x7fa8e00cf900 2024-04-06T13:51:42.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.783+0000 7fa8e860b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e015a940 con 0x7fa8e015a550 2024-04-06T13:51:42.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.783+0000 7fa8e5ba6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa8e00cf900 0x7fa8e00b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.783+0000 7fa8e5ba6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa8e00cf900 0x7fa8e00b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44326/0 (socket says 172.21.15.189:44326) 2024-04-06T13:51:42.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.783+0000 7fa8e5ba6700 1 -- 172.21.15.189:0/241472142 learned_addr learned my addr 172.21.15.189:0/241472142 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:42.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.783+0000 7fa8e63a7700 1 --2- 172.21.15.189:0/241472142 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e015a550 0x7fa8e00cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.783+0000 7fa8e5ba6700 1 -- 172.21.15.189:0/241472142 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e015a550 msgr2=0x7fa8e00cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:42.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.783+0000 7fa8e5ba6700 1 --2- 172.21.15.189:0/241472142 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e015a550 0x7fa8e00cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.783+0000 7fa8e5ba6700 1 -- 172.21.15.189:0/241472142 >> v1:172.21.15.192:6789/0 conn(0x7fa8e00b4ba0 legacy=0x7fa8e006eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.783+0000 7fa8e5ba6700 1 -- 172.21.15.189:0/241472142 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e015ac00 con 0x7fa8e00cf900 2024-04-06T13:51:42.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.783+0000 7fa8e5ba6700 1 --2- 172.21.15.189:0/241472142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e00cf900 0x7fa8e00b4660 secure :-1 s=READY pgs=3231 cs=0 l=1 rev1=1 crypto rx=0x7fa8d40098f0 tx=0x7fa8d4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=166477d2e25bf49d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:42.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.784+0000 7fa8e53a5700 1 -- 172.21.15.189:0/241472142 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4011830 con 0x7fa8e00cf900 2024-04-06T13:51:42.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.784+0000 7fa8e53a5700 1 -- 172.21.15.189:0/241472142 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d4011990 con 0x7fa8e00cf900 2024-04-06T13:51:42.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.784+0000 7fa8e53a5700 1 -- 172.21.15.189:0/241472142 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4011d20 con 0x7fa8e00cf900 2024-04-06T13:51:42.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.784+0000 7fa8e860b700 1 -- 172.21.15.189:0/241472142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e00cf900 msgr2=0x7fa8e00b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.784+0000 7fa8e860b700 1 --2- 172.21.15.189:0/241472142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e00cf900 0x7fa8e00b4660 secure :-1 s=READY pgs=3231 cs=0 l=1 rev1=1 crypto rx=0x7fa8d40098f0 tx=0x7fa8d4009c00 comp rx=0 tx=0).stop 2024-04-06T13:51:42.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.784+0000 7fa8e860b700 1 -- 172.21.15.189:0/241472142 shutdown_connections 2024-04-06T13:51:42.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.784+0000 7fa8e860b700 1 --2- 172.21.15.189:0/241472142 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e015a550 0x7fa8e00cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.784+0000 7fa8e860b700 1 --2- 172.21.15.189:0/241472142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e00cf900 0x7fa8e00b4660 unknown :-1 s=CLOSED pgs=3231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.784+0000 7fa8e860b700 1 -- 172.21.15.189:0/241472142 >> 172.21.15.189:0/241472142 conn(0x7fa8e00d00b0 msgr2=0x7fa8e00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:42.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.784+0000 7fa8e860b700 1 -- 172.21.15.189:0/241472142 shutdown_connections 2024-04-06T13:51:42.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.784+0000 7fa8e860b700 1 -- 172.21.15.189:0/241472142 wait complete. 2024-04-06T13:51:42.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.786+0000 7fa8e860b700 1 Processor -- start 2024-04-06T13:51:42.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.786+0000 7fa8e860b700 1 -- start start 2024-04-06T13:51:42.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8e860b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e00b4ba0 0x7fa8e0145c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8e860b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8e00cf900 0x7fa8e0149960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8e860b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e01343d0 0x7fa8e0157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8e860b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e014a220 con 0x7fa8e00b4ba0 2024-04-06T13:51:42.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8e860b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e014a500 con 0x7fa8e00cf900 2024-04-06T13:51:42.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8e860b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e014a7e0 con 0x7fa8e01343d0 2024-04-06T13:51:42.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8e63a7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e00b4ba0 0x7fa8e0145c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8e63a7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e00b4ba0 0x7fa8e0145c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44342/0 (socket says 172.21.15.189:44342) 2024-04-06T13:51:42.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8e63a7700 1 -- 172.21.15.189:0/3986990046 learned_addr learned my addr 172.21.15.189:0/3986990046 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:42.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8e5ba6700 1 --2- 172.21.15.189:0/3986990046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8e00cf900 0x7fa8e0149960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8e6ba8700 1 --2- 172.21.15.189:0/3986990046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e01343d0 0x7fa8e0157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8e63a7700 1 -- 172.21.15.189:0/3986990046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e01343d0 msgr2=0x7fa8e0157980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8e63a7700 1 --2- 172.21.15.189:0/3986990046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e01343d0 0x7fa8e0157980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8e63a7700 1 -- 172.21.15.189:0/3986990046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8e00cf900 msgr2=0x7fa8e0149960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8e63a7700 1 --2- 172.21.15.189:0/3986990046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8e00cf900 0x7fa8e0149960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8e63a7700 1 -- 172.21.15.189:0/3986990046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d4009550 con 0x7fa8e00b4ba0 2024-04-06T13:51:42.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8e63a7700 1 --2- 172.21.15.189:0/3986990046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e00b4ba0 0x7fa8e0145c20 secure :-1 s=READY pgs=3232 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc00e9b0 tx=0x7fa8dc00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:42.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8d2ffd700 1 -- 172.21.15.189:0/3986990046 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc009710 con 0x7fa8e00b4ba0 2024-04-06T13:51:42.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8d2ffd700 1 -- 172.21.15.189:0/3986990046 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8dc00ee50 con 0x7fa8e00b4ba0 2024-04-06T13:51:42.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8e860b700 1 -- 172.21.15.189:0/3986990046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e014aac0 con 0x7fa8e00b4ba0 2024-04-06T13:51:42.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8d2ffd700 1 -- 172.21.15.189:0/3986990046 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc00ca50 con 0x7fa8e00b4ba0 2024-04-06T13:51:42.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.787+0000 7fa8e860b700 1 -- 172.21.15.189:0/3986990046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e0149ea0 con 0x7fa8e00b4ba0 2024-04-06T13:51:42.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.788+0000 7fa8e860b700 1 -- 172.21.15.189:0/3986990046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e00af960 con 0x7fa8e00b4ba0 2024-04-06T13:51:42.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.789+0000 7fa8d2ffd700 1 -- 172.21.15.189:0/3986990046 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fa8dc00cbb0 con 0x7fa8e00b4ba0 2024-04-06T13:51:42.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.789+0000 7fa8d2ffd700 1 --2- 172.21.15.189:0/3986990046 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa8c406adb0 0x7fa8c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:42.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.789+0000 7fa8e5ba6700 1 --2- 172.21.15.189:0/3986990046 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa8c406adb0 0x7fa8c406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:42.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.789+0000 7fa8d2ffd700 1 -- 172.21.15.189:0/3986990046 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fa8dc0895b0 con 0x7fa8e00b4ba0 2024-04-06T13:51:42.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.790+0000 7fa8e5ba6700 1 --2- 172.21.15.189:0/3986990046 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa8c406adb0 0x7fa8c406d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4009c40 tx=0x7fa8d4008030 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:42.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.794+0000 7fa8d2ffd700 1 -- 172.21.15.189:0/3986990046 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa8dc054510 con 0x7fa8e00b4ba0 2024-04-06T13:51:42.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.987+0000 7fa8e860b700 1 -- 172.21.15.189:0/3986990046 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8e015b4a0 con 0x7fa8c406adb0 2024-04-06T13:51:42.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.987+0000 7fa8d2ffd700 1 -- 172.21.15.189:0/3986990046 <== mgr.10415 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa8d4011c00 con 0x7fa8c406adb0 2024-04-06T13:51:42.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.989+0000 7fa8e860b700 1 -- 172.21.15.189:0/3986990046 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa8c406adb0 msgr2=0x7fa8c406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.989+0000 7fa8e860b700 1 --2- 172.21.15.189:0/3986990046 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa8c406adb0 0x7fa8c406d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4009c40 tx=0x7fa8d4008030 comp rx=0 tx=0).stop 2024-04-06T13:51:42.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.989+0000 7fa8e860b700 1 -- 172.21.15.189:0/3986990046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e00b4ba0 msgr2=0x7fa8e0145c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:42.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.989+0000 7fa8e860b700 1 --2- 172.21.15.189:0/3986990046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e00b4ba0 0x7fa8e0145c20 secure :-1 s=READY pgs=3232 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc00e9b0 tx=0x7fa8dc00ecc0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.989+0000 7fa8e860b700 1 -- 172.21.15.189:0/3986990046 shutdown_connections 2024-04-06T13:51:42.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.989+0000 7fa8e860b700 1 --2- 172.21.15.189:0/3986990046 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fa8c406adb0 0x7fa8c406d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.989+0000 7fa8e860b700 1 --2- 172.21.15.189:0/3986990046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa8e01343d0 0x7fa8e0157980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.989+0000 7fa8e860b700 1 --2- 172.21.15.189:0/3986990046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e00b4ba0 0x7fa8e0145c20 unknown :-1 s=CLOSED pgs=3232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.989+0000 7fa8e860b700 1 --2- 172.21.15.189:0/3986990046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8e00cf900 0x7fa8e0149960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:42.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.989+0000 7fa8e860b700 1 -- 172.21.15.189:0/3986990046 >> 172.21.15.189:0/3986990046 conn(0x7fa8e00d00b0 msgr2=0x7fa8e006eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:42.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.989+0000 7fa8e860b700 1 -- 172.21.15.189:0/3986990046 shutdown_connections 2024-04-06T13:51:42.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:42.989+0000 7fa8e860b700 1 -- 172.21.15.189:0/3986990046 wait complete. 2024-04-06T13:51:42.991 INFO:teuthology.orchestra.run.smithi189.stderr:instructed osd(s) 3 to deep-scrub 2024-04-06T13:51:43.002 INFO:tasks.ceph:Scrubbing osd.4 2024-04-06T13:51:43.002 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-06T13:51:43.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.128+0000 7fb532971700 1 Processor -- start 2024-04-06T13:51:43.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.128+0000 7fb532971700 1 -- start start 2024-04-06T13:51:43.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.128+0000 7fb532971700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb52c15a550 0x7fb52c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.128+0000 7fb532971700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb52c0cf900 0x7fb52c0b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.129+0000 7fb532971700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb52c0cdbe0 con 0x7fb52c0b4ba0 2024-04-06T13:51:43.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.129+0000 7fb532971700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb52c15a940 con 0x7fb52c0cf900 2024-04-06T13:51:43.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.129+0000 7fb532971700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb52c15ac00 con 0x7fb52c15a550 2024-04-06T13:51:43.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.129+0000 7fb52b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb52c0cf900 0x7fb52c0b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.129+0000 7fb530f0e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb52c0b4ba0 0x7fb52c06eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:32898/0 (socket says 172.21.15.189:32898) 2024-04-06T13:51:43.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.129+0000 7fb530f0e700 1 -- 172.21.15.189:0/3869073772 learned_addr learned my addr 172.21.15.189:0/3869073772 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:43.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.129+0000 7fb52bfff700 1 --2- 172.21.15.189:0/3869073772 >> v2:172.21.15.192:3300/0 conn(0x7fb52c15a550 0x7fb52c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.129+0000 7fb52b7fe700 1 -- 172.21.15.189:0/3869073772 >> v2:172.21.15.192:3300/0 conn(0x7fb52c15a550 msgr2=0x7fb52c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:43.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.129+0000 7fb52b7fe700 1 --2- 172.21.15.189:0/3869073772 >> v2:172.21.15.192:3300/0 conn(0x7fb52c15a550 0x7fb52c0cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.129+0000 7fb52b7fe700 1 -- 172.21.15.189:0/3869073772 >> v1:172.21.15.189:6789/0 conn(0x7fb52c0b4ba0 legacy=0x7fb52c06eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.129+0000 7fb52b7fe700 1 -- 172.21.15.189:0/3869073772 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52c15aee0 con 0x7fb52c0cf900 2024-04-06T13:51:43.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.129+0000 7fb52b7fe700 1 --2- 172.21.15.189:0/3869073772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb52c0cf900 0x7fb52c0b4660 secure :-1 s=READY pgs=3234 cs=0 l=1 rev1=1 crypto rx=0x7fb52c073470 tx=0x7fb52000bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=437f7932bddf2bee server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:43.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.129+0000 7fb52affd700 1 -- 172.21.15.189:0/3869073772 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52000dc70 con 0x7fb52c0cf900 2024-04-06T13:51:43.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.129+0000 7fb52affd700 1 -- 172.21.15.189:0/3869073772 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52000ddd0 con 0x7fb52c0cf900 2024-04-06T13:51:43.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.129+0000 7fb52affd700 1 -- 172.21.15.189:0/3869073772 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520010690 con 0x7fb52c0cf900 2024-04-06T13:51:43.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.130+0000 7fb532971700 1 -- 172.21.15.189:0/3869073772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb52c0cf900 msgr2=0x7fb52c0b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.130+0000 7fb532971700 1 --2- 172.21.15.189:0/3869073772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb52c0cf900 0x7fb52c0b4660 secure :-1 s=READY pgs=3234 cs=0 l=1 rev1=1 crypto rx=0x7fb52c073470 tx=0x7fb52000bdc0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.130+0000 7fb532971700 1 -- 172.21.15.189:0/3869073772 shutdown_connections 2024-04-06T13:51:43.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.130+0000 7fb532971700 1 --2- 172.21.15.189:0/3869073772 >> v2:172.21.15.192:3300/0 conn(0x7fb52c15a550 0x7fb52c0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.130+0000 7fb532971700 1 --2- 172.21.15.189:0/3869073772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb52c0cf900 0x7fb52c0b4660 unknown :-1 s=CLOSED pgs=3234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.130+0000 7fb532971700 1 -- 172.21.15.189:0/3869073772 >> 172.21.15.189:0/3869073772 conn(0x7fb52c0d00b0 msgr2=0x7fb52c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:43.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.130+0000 7fb532971700 1 -- 172.21.15.189:0/3869073772 shutdown_connections 2024-04-06T13:51:43.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.130+0000 7fb532971700 1 -- 172.21.15.189:0/3869073772 wait complete. 2024-04-06T13:51:43.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.131+0000 7fb532971700 1 Processor -- start 2024-04-06T13:51:43.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.132+0000 7fb532971700 1 -- start start 2024-04-06T13:51:43.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.132+0000 7fb532971700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb52c0b4ba0 0x7fb52c153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.132+0000 7fb532971700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb52c0cf900 0x7fb52c0b1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.132+0000 7fb532971700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb52c158ad0 0x7fb52c06d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.132+0000 7fb532971700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c1494a0 con 0x7fb52c0cf900 2024-04-06T13:51:43.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.132+0000 7fb532971700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c0b1ac0 con 0x7fb52c158ad0 2024-04-06T13:51:43.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.132+0000 7fb532971700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c159040 con 0x7fb52c0b4ba0 2024-04-06T13:51:43.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.132+0000 7fb52b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb52c0cf900 0x7fb52c0b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.133+0000 7fb52b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb52c0cf900 0x7fb52c0b1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44352/0 (socket says 172.21.15.189:44352) 2024-04-06T13:51:43.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.133+0000 7fb52b7fe700 1 -- 172.21.15.189:0/545502454 learned_addr learned my addr 172.21.15.189:0/545502454 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:43.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.133+0000 7fb530f0e700 1 --2- 172.21.15.189:0/545502454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb52c158ad0 0x7fb52c06d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.133+0000 7fb52bfff700 1 --2- 172.21.15.189:0/545502454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb52c0b4ba0 0x7fb52c153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.133+0000 7fb52b7fe700 1 -- 172.21.15.189:0/545502454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb52c0b4ba0 msgr2=0x7fb52c153c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.133+0000 7fb52b7fe700 1 --2- 172.21.15.189:0/545502454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb52c0b4ba0 0x7fb52c153c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.133+0000 7fb52b7fe700 1 -- 172.21.15.189:0/545502454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb52c158ad0 msgr2=0x7fb52c06d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.133+0000 7fb52b7fe700 1 --2- 172.21.15.189:0/545502454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb52c158ad0 0x7fb52c06d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.133+0000 7fb52b7fe700 1 -- 172.21.15.189:0/545502454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52000b740 con 0x7fb52c0cf900 2024-04-06T13:51:43.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.133+0000 7fb52b7fe700 1 --2- 172.21.15.189:0/545502454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb52c0cf900 0x7fb52c0b1580 secure :-1 s=READY pgs=3235 cs=0 l=1 rev1=1 crypto rx=0x7fb52000b2c0 tx=0x7fb520004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:43.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.133+0000 7fb528ff9700 1 -- 172.21.15.189:0/545502454 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52001f070 con 0x7fb52c0cf900 2024-04-06T13:51:43.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.133+0000 7fb528ff9700 1 -- 172.21.15.189:0/545502454 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb520003970 con 0x7fb52c0cf900 2024-04-06T13:51:43.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.133+0000 7fb532971700 1 -- 172.21.15.189:0/545502454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb52c159320 con 0x7fb52c0cf900 2024-04-06T13:51:43.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.133+0000 7fb528ff9700 1 -- 172.21.15.189:0/545502454 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520003d20 con 0x7fb52c0cf900 2024-04-06T13:51:43.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.133+0000 7fb532971700 1 -- 172.21.15.189:0/545502454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb52c1598e0 con 0x7fb52c0cf900 2024-04-06T13:51:43.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.134+0000 7fb532971700 1 -- 172.21.15.189:0/545502454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb52c0b4fb0 con 0x7fb52c0cf900 2024-04-06T13:51:43.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.135+0000 7fb528ff9700 1 -- 172.21.15.189:0/545502454 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fb520024470 con 0x7fb52c0cf900 2024-04-06T13:51:43.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.135+0000 7fb528ff9700 1 --2- 172.21.15.189:0/545502454 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb51806adb0 0x7fb51806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.135+0000 7fb528ff9700 1 -- 172.21.15.189:0/545502454 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fb52008cbe0 con 0x7fb52c0cf900 2024-04-06T13:51:43.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.135+0000 7fb52bfff700 1 --2- 172.21.15.189:0/545502454 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb51806adb0 0x7fb51806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.135+0000 7fb528ff9700 1 --2- 172.21.15.189:0/545502454 >> [v2:172.21.15.192:6816/47716,v1:172.21.15.192:6817/47716] conn(0x7fb5180716f0 0x7fb518073b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.135+0000 7fb528ff9700 1 -- 172.21.15.189:0/545502454 --> [v2:172.21.15.192:6816/47716,v1:172.21.15.192:6817/47716] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb518074220 con 0x7fb5180716f0 2024-04-06T13:51:43.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.135+0000 7fb528ff9700 1 -- 172.21.15.189:0/545502454 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=290) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5200912c0 con 0x7fb52c0cf900 2024-04-06T13:51:43.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.135+0000 7fb530f0e700 1 --2- 172.21.15.189:0/545502454 >> [v2:172.21.15.192:6816/47716,v1:172.21.15.192:6817/47716] conn(0x7fb5180716f0 0x7fb518073b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.135+0000 7fb52bfff700 1 --2- 172.21.15.189:0/545502454 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb51806adb0 0x7fb51806d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb51c006b20 tx=0x7fb51c008040 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:43.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.136+0000 7fb530f0e700 1 --2- 172.21.15.189:0/545502454 >> [v2:172.21.15.192:6816/47716,v1:172.21.15.192:6817/47716] conn(0x7fb5180716f0 0x7fb518073b50 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:43.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.137+0000 7fb528ff9700 1 -- 172.21.15.189:0/545502454 <== osd.4 v2:172.21.15.192:6816/47716 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb518074220 con 0x7fb5180716f0 2024-04-06T13:51:43.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.151+0000 7fb532971700 1 -- 172.21.15.189:0/545502454 --> [v2:172.21.15.192:6816/47716,v1:172.21.15.192:6817/47716] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb52c15b4a0 con 0x7fb5180716f0 2024-04-06T13:51:43.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.151+0000 7fb528ff9700 1 -- 172.21.15.189:0/545502454 <== osd.4 v2:172.21.15.192:6816/47716 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb524001f00 con 0x7fb5180716f0 2024-04-06T13:51:43.152 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:51:43.152 INFO:teuthology.orchestra.run.smithi189.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-06T13:51:43.152 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:51:43.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.151+0000 7fb532971700 1 -- 172.21.15.189:0/545502454 >> [v2:172.21.15.192:6816/47716,v1:172.21.15.192:6817/47716] conn(0x7fb5180716f0 msgr2=0x7fb518073b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.151+0000 7fb532971700 1 --2- 172.21.15.189:0/545502454 >> [v2:172.21.15.192:6816/47716,v1:172.21.15.192:6817/47716] conn(0x7fb5180716f0 0x7fb518073b50 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.151+0000 7fb532971700 1 -- 172.21.15.189:0/545502454 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb51806adb0 msgr2=0x7fb51806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.151+0000 7fb532971700 1 --2- 172.21.15.189:0/545502454 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fb51806adb0 0x7fb51806d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb51c006b20 tx=0x7fb51c008040 comp rx=0 tx=0).stop 2024-04-06T13:51:43.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.151+0000 7fb532971700 1 -- 172.21.15.189:0/545502454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb52c0cf900 msgr2=0x7fb52c0b1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.151+0000 7fb532971700 1 --2- 172.21.15.189:0/545502454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb52c0cf900 0x7fb52c0b1580 secure :-1 s=READY pgs=3235 cs=0 l=1 rev1=1 crypto rx=0x7fb52000b2c0 tx=0x7fb520004750 comp rx=0 tx=0).stop 2024-04-06T13:51:43.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.151+0000 7fb530f0e700 1 -- 172.21.15.189:0/545502454 reap_dead start 2024-04-06T13:51:43.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.151+0000 7fb532971700 1 -- 172.21.15.189:0/545502454 shutdown_connections 2024-04-06T13:51:43.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.151+0000 7fb532971700 1 -- 172.21.15.189:0/545502454 >> 172.21.15.189:0/545502454 conn(0x7fb52c0d00b0 msgr2=0x7fb52c0b30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:43.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.151+0000 7fb532971700 1 -- 172.21.15.189:0/545502454 shutdown_connections 2024-04-06T13:51:43.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.152+0000 7fb532971700 1 -- 172.21.15.189:0/545502454 wait complete. 2024-04-06T13:51:43.163 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-06T13:51:43.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.286+0000 7f46bb0e4700 1 Processor -- start 2024-04-06T13:51:43.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.286+0000 7f46bb0e4700 1 -- start start 2024-04-06T13:51:43.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.287+0000 7f46bb0e4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f46b40aec70 0x7f46b40cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.287+0000 7f46bb0e4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46b40cdae0 0x7f46b40ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.287+0000 7f46bb0e4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b415a830 con 0x7f46b40ce8b0 2024-04-06T13:51:43.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.287+0000 7f46bb0e4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46b40cd920 con 0x7f46b40aec70 2024-04-06T13:51:43.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.287+0000 7f46bb0e4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b40ced20 con 0x7f46b40cdae0 2024-04-06T13:51:43.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.287+0000 7f46b8e80700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f46b40aec70 0x7f46b40cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.287+0000 7f46b9681700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f46b40ce8b0 0x7f46b415df20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.189:52524/0 (socket says 172.21.15.189:52524) 2024-04-06T13:51:43.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.287+0000 7f46b9681700 1 -- 172.21.15.189:0/2024641895 learned_addr learned my addr 172.21.15.189:0/2024641895 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:43.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.287+0000 7f46b3fff700 1 --2- 172.21.15.189:0/2024641895 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46b40cdae0 0x7f46b40ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.287+0000 7f46b8e80700 1 -- 172.21.15.189:0/2024641895 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46b40cdae0 msgr2=0x7f46b40ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:43.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.287+0000 7f46b8e80700 1 --2- 172.21.15.189:0/2024641895 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46b40cdae0 0x7f46b40ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.287+0000 7f46b8e80700 1 -- 172.21.15.189:0/2024641895 >> v1:172.21.15.192:6789/0 conn(0x7f46b40ce8b0 legacy=0x7f46b415df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.287+0000 7f46b8e80700 1 -- 172.21.15.189:0/2024641895 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b40cc050 con 0x7f46b40aec70 2024-04-06T13:51:43.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.287+0000 7f46b37fe700 1 -- 172.21.15.189:0/2024641895 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100159343 0 0) 0x7f46b415a830 con 0x7f46b40ce8b0 2024-04-06T13:51:43.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.287+0000 7f46b8e80700 1 --2- 172.21.15.189:0/2024641895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46b40aec70 0x7f46b40cd3e0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f46a8009840 tx=0x7f46a8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc6089e15a7a14b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:43.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.288+0000 7f46b37fe700 1 -- 172.21.15.189:0/2024641895 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a8011830 con 0x7f46b40aec70 2024-04-06T13:51:43.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.288+0000 7f46b37fe700 1 -- 172.21.15.189:0/2024641895 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a8011990 con 0x7f46b40aec70 2024-04-06T13:51:43.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.288+0000 7f46b37fe700 1 -- 172.21.15.189:0/2024641895 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a8011d20 con 0x7f46b40aec70 2024-04-06T13:51:43.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.288+0000 7f46bb0e4700 1 -- 172.21.15.189:0/2024641895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46b40aec70 msgr2=0x7f46b40cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.288+0000 7f46bb0e4700 1 --2- 172.21.15.189:0/2024641895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46b40aec70 0x7f46b40cd3e0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f46a8009840 tx=0x7f46a8009b50 comp rx=0 tx=0).stop 2024-04-06T13:51:43.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.288+0000 7f46bb0e4700 1 -- 172.21.15.189:0/2024641895 shutdown_connections 2024-04-06T13:51:43.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.288+0000 7f46bb0e4700 1 --2- 172.21.15.189:0/2024641895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46b40aec70 0x7f46b40cd3e0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.288+0000 7f46bb0e4700 1 --2- 172.21.15.189:0/2024641895 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46b40cdae0 0x7f46b40ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.288+0000 7f46bb0e4700 1 -- 172.21.15.189:0/2024641895 >> 172.21.15.189:0/2024641895 conn(0x7f46b40cc290 msgr2=0x7f46b40cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:43.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.288+0000 7f46bb0e4700 1 -- 172.21.15.189:0/2024641895 shutdown_connections 2024-04-06T13:51:43.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.288+0000 7f46bb0e4700 1 -- 172.21.15.189:0/2024641895 wait complete. 2024-04-06T13:51:43.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.289+0000 7f46bb0e4700 1 Processor -- start 2024-04-06T13:51:43.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.289+0000 7f46bb0e4700 1 -- start start 2024-04-06T13:51:43.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.290+0000 7f46bb0e4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46b40aec70 0x7f46b41f3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.290+0000 7f46bb0e4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46b40cdae0 0x7f46b40c5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.290+0000 7f46bb0e4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46b40ce8b0 0x7f46b41459e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.290+0000 7f46bb0e4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b40c4e50 con 0x7f46b40cdae0 2024-04-06T13:51:43.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.290+0000 7f46bb0e4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b40c3f80 con 0x7f46b40ce8b0 2024-04-06T13:51:43.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.290+0000 7f46bb0e4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b41f3d70 con 0x7f46b40aec70 2024-04-06T13:51:43.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.290+0000 7f46b3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46b40cdae0 0x7f46b40c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.290+0000 7f46b8e80700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46b40aec70 0x7f46b41f3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.290+0000 7f46b8e80700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46b40aec70 0x7f46b41f3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.189:58086/0 (socket says 172.21.15.189:58086) 2024-04-06T13:51:43.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.290+0000 7f46b3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46b40cdae0 0x7f46b40c5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44368/0 (socket says 172.21.15.189:44368) 2024-04-06T13:51:43.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.290+0000 7f46b8e80700 1 -- 172.21.15.189:0/4010223209 learned_addr learned my addr 172.21.15.189:0/4010223209 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:43.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.290+0000 7f46b9681700 1 --2- 172.21.15.189:0/4010223209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46b40ce8b0 0x7f46b41459e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.291+0000 7f46b8e80700 1 -- 172.21.15.189:0/4010223209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46b40ce8b0 msgr2=0x7f46b41459e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.291+0000 7f46b8e80700 1 --2- 172.21.15.189:0/4010223209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46b40ce8b0 0x7f46b41459e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.291+0000 7f46b8e80700 1 -- 172.21.15.189:0/4010223209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46b40cdae0 msgr2=0x7f46b40c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.291+0000 7f46b8e80700 1 --2- 172.21.15.189:0/4010223209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46b40cdae0 0x7f46b40c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.291+0000 7f46b8e80700 1 -- 172.21.15.189:0/4010223209 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a80094a0 con 0x7f46b40aec70 2024-04-06T13:51:43.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.291+0000 7f46b9681700 1 --2- 172.21.15.189:0/4010223209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46b40ce8b0 0x7f46b41459e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:51:43.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.291+0000 7f46b3fff700 1 --2- 172.21.15.189:0/4010223209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46b40cdae0 0x7f46b40c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:51:43.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.291+0000 7f46b8e80700 1 --2- 172.21.15.189:0/4010223209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46b40aec70 0x7f46b41f3830 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f46a8009b90 tx=0x7f46a8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:43.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.291+0000 7f46b17fa700 1 -- 172.21.15.189:0/4010223209 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a80036a0 con 0x7f46b40aec70 2024-04-06T13:51:43.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.291+0000 7f46b17fa700 1 -- 172.21.15.189:0/4010223209 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a801e070 con 0x7f46b40aec70 2024-04-06T13:51:43.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.291+0000 7f46bb0e4700 1 -- 172.21.15.189:0/4010223209 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b40c41d0 con 0x7f46b40aec70 2024-04-06T13:51:43.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.291+0000 7f46b17fa700 1 -- 172.21.15.189:0/4010223209 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a8003a60 con 0x7f46b40aec70 2024-04-06T13:51:43.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.291+0000 7f46bb0e4700 1 -- 172.21.15.189:0/4010223209 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b40c4790 con 0x7f46b40aec70 2024-04-06T13:51:43.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.292+0000 7f46bb0e4700 1 -- 172.21.15.189:0/4010223209 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b40c4f90 con 0x7f46b40aec70 2024-04-06T13:51:43.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.293+0000 7f46b17fa700 1 -- 172.21.15.189:0/4010223209 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f46a8003bc0 con 0x7f46b40aec70 2024-04-06T13:51:43.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.293+0000 7f46b17fa700 1 --2- 172.21.15.189:0/4010223209 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f469806adb0 0x7f469806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.293+0000 7f46b17fa700 1 -- 172.21.15.189:0/4010223209 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f46a808a700 con 0x7f46b40aec70 2024-04-06T13:51:43.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.293+0000 7f46b3fff700 1 --2- 172.21.15.189:0/4010223209 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f469806adb0 0x7f469806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.294+0000 7f46b3fff700 1 --2- 172.21.15.189:0/4010223209 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f469806adb0 0x7f469806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f46a0009fd0 tx=0x7f46a0009210 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:43.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.297+0000 7f46b17fa700 1 -- 172.21.15.189:0/4010223209 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f46a80557d0 con 0x7f46b40aec70 2024-04-06T13:51:43.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.489+0000 7f46bb0e4700 1 -- 172.21.15.189:0/4010223209 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f46b40cb850 con 0x7f469806adb0 2024-04-06T13:51:43.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.490+0000 7f46b17fa700 1 -- 172.21.15.189:0/4010223209 <== mgr.10415 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f46a0011c50 con 0x7f469806adb0 2024-04-06T13:51:43.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.492+0000 7f46bb0e4700 1 -- 172.21.15.189:0/4010223209 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f469806adb0 msgr2=0x7f469806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.492+0000 7f46bb0e4700 1 --2- 172.21.15.189:0/4010223209 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f469806adb0 0x7f469806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f46a0009fd0 tx=0x7f46a0009210 comp rx=0 tx=0).stop 2024-04-06T13:51:43.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.492+0000 7f46bb0e4700 1 -- 172.21.15.189:0/4010223209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46b40aec70 msgr2=0x7f46b41f3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.492+0000 7f46bb0e4700 1 --2- 172.21.15.189:0/4010223209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46b40aec70 0x7f46b41f3830 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f46a8009b90 tx=0x7f46a8004750 comp rx=0 tx=0).stop 2024-04-06T13:51:43.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.492+0000 7f46bb0e4700 1 -- 172.21.15.189:0/4010223209 shutdown_connections 2024-04-06T13:51:43.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.492+0000 7f46bb0e4700 1 --2- 172.21.15.189:0/4010223209 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f469806adb0 0x7f469806d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.492+0000 7f46bb0e4700 1 --2- 172.21.15.189:0/4010223209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46b40ce8b0 0x7f46b41459e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.492+0000 7f46bb0e4700 1 --2- 172.21.15.189:0/4010223209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46b40aec70 0x7f46b41f3830 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.492+0000 7f46bb0e4700 1 --2- 172.21.15.189:0/4010223209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46b40cdae0 0x7f46b40c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.492+0000 7f46bb0e4700 1 -- 172.21.15.189:0/4010223209 >> 172.21.15.189:0/4010223209 conn(0x7f46b40cc290 msgr2=0x7f46b40b00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:43.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.492+0000 7f46bb0e4700 1 -- 172.21.15.189:0/4010223209 shutdown_connections 2024-04-06T13:51:43.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.492+0000 7f46bb0e4700 1 -- 172.21.15.189:0/4010223209 wait complete. 2024-04-06T13:51:43.494 INFO:teuthology.orchestra.run.smithi189.stderr:instructed osd(s) 4 to deep-scrub 2024-04-06T13:51:43.504 INFO:tasks.ceph:Scrubbing osd.5 2024-04-06T13:51:43.504 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-06T13:51:43.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.630+0000 7ff6770b1700 1 Processor -- start 2024-04-06T13:51:43.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.630+0000 7ff6770b1700 1 -- start start 2024-04-06T13:51:43.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.631+0000 7ff6770b1700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff6700cb870 0x7ff67015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.631+0000 7ff6770b1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff6700cfa00 0x7ff6700cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.631+0000 7ff6770b1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6700cbd80 con 0x7ff6700d11b0 2024-04-06T13:51:43.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.631+0000 7ff6770b1700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6700ccdc0 con 0x7ff6700cb870 2024-04-06T13:51:43.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.631+0000 7ff6770b1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6700d0b50 con 0x7ff6700cfa00 2024-04-06T13:51:43.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.631+0000 7ff674e4d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff6700d11b0 0x7ff6700d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:32902/0 (socket says 172.21.15.189:32902) 2024-04-06T13:51:43.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.631+0000 7ff674e4d700 1 -- 172.21.15.189:0/3026103492 learned_addr learned my addr 172.21.15.189:0/3026103492 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:43.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.631+0000 7ff66ffff700 1 --2- 172.21.15.189:0/3026103492 >> v2:172.21.15.192:3300/0 conn(0x7ff6700cb870 0x7ff67015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.631+0000 7ff67564e700 1 --2- 172.21.15.189:0/3026103492 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff6700cfa00 0x7ff6700cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.632+0000 7ff66f7fe700 1 -- 172.21.15.189:0/3026103492 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630947672 0 0) 0x7ff6700cbd80 con 0x7ff6700d11b0 2024-04-06T13:51:43.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.632+0000 7ff66f7fe700 1 -- 172.21.15.189:0/3026103492 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6600035b0 con 0x7ff6700d11b0 2024-04-06T13:51:43.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.632+0000 7ff67564e700 1 -- 172.21.15.189:0/3026103492 >> v2:172.21.15.192:3300/0 conn(0x7ff6700cb870 msgr2=0x7ff67015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:43.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.632+0000 7ff67564e700 1 --2- 172.21.15.189:0/3026103492 >> v2:172.21.15.192:3300/0 conn(0x7ff6700cb870 0x7ff67015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.632+0000 7ff67564e700 1 -- 172.21.15.189:0/3026103492 >> v1:172.21.15.189:6789/0 conn(0x7ff6700d11b0 legacy=0x7ff6700d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.632+0000 7ff67564e700 1 -- 172.21.15.189:0/3026103492 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6700cef20 con 0x7ff6700cfa00 2024-04-06T13:51:43.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.632+0000 7ff66ffff700 1 --2- 172.21.15.189:0/3026103492 >> v2:172.21.15.192:3300/0 conn(0x7ff6700cb870 0x7ff67015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:51:43.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.632+0000 7ff67564e700 1 --2- 172.21.15.189:0/3026103492 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff6700cfa00 0x7ff6700cd560 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff668009d90 tx=0x7ff66800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44a13f0bd9e068d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:43.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.632+0000 7ff66f7fe700 1 -- 172.21.15.189:0/3026103492 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66800e070 con 0x7ff6700cfa00 2024-04-06T13:51:43.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.632+0000 7ff66f7fe700 1 -- 172.21.15.189:0/3026103492 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff66800bb20 con 0x7ff6700cfa00 2024-04-06T13:51:43.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.632+0000 7ff66f7fe700 1 -- 172.21.15.189:0/3026103492 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff668012430 con 0x7ff6700cfa00 2024-04-06T13:51:43.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.633+0000 7ff6770b1700 1 -- 172.21.15.189:0/3026103492 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff6700cfa00 msgr2=0x7ff6700cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.633+0000 7ff6770b1700 1 --2- 172.21.15.189:0/3026103492 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff6700cfa00 0x7ff6700cd560 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff668009d90 tx=0x7ff66800b330 comp rx=0 tx=0).stop 2024-04-06T13:51:43.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.633+0000 7ff6770b1700 1 -- 172.21.15.189:0/3026103492 shutdown_connections 2024-04-06T13:51:43.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.633+0000 7ff6770b1700 1 --2- 172.21.15.189:0/3026103492 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff6700cfa00 0x7ff6700cd560 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.633+0000 7ff6770b1700 1 --2- 172.21.15.189:0/3026103492 >> v2:172.21.15.192:3300/0 conn(0x7ff6700cb870 0x7ff67015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.636 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.633+0000 7ff6770b1700 1 -- 172.21.15.189:0/3026103492 >> 172.21.15.189:0/3026103492 conn(0x7ff6700b2db0 msgr2=0x7ff6700b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:43.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.633+0000 7ff6770b1700 1 -- 172.21.15.189:0/3026103492 shutdown_connections 2024-04-06T13:51:43.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.633+0000 7ff6770b1700 1 -- 172.21.15.189:0/3026103492 wait complete. 2024-04-06T13:51:43.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.634+0000 7ff6770b1700 1 Processor -- start 2024-04-06T13:51:43.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.634+0000 7ff6770b1700 1 -- start start 2024-04-06T13:51:43.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.635+0000 7ff6770b1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff6700cb870 0x7ff6700ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.635+0000 7ff6770b1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6700cfa00 0x7ff67006efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.635+0000 7ff6770b1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6700d11b0 0x7ff67006ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.635+0000 7ff6770b1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6700c9230 con 0x7ff6700cfa00 2024-04-06T13:51:43.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.635+0000 7ff6770b1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6700c9510 con 0x7ff6700d11b0 2024-04-06T13:51:43.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.635+0000 7ff6770b1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6700c97f0 con 0x7ff6700cb870 2024-04-06T13:51:43.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.635+0000 7ff66ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6700cfa00 0x7ff67006efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.635+0000 7ff66ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6700cfa00 0x7ff67006efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44378/0 (socket says 172.21.15.189:44378) 2024-04-06T13:51:43.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.635+0000 7ff66ffff700 1 -- 172.21.15.189:0/3820240054 learned_addr learned my addr 172.21.15.189:0/3820240054 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:43.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.635+0000 7ff67564e700 1 --2- 172.21.15.189:0/3820240054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6700d11b0 0x7ff67006ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.635+0000 7ff674e4d700 1 --2- 172.21.15.189:0/3820240054 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff6700cb870 0x7ff6700ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.635+0000 7ff66ffff700 1 -- 172.21.15.189:0/3820240054 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff6700cb870 msgr2=0x7ff6700ca2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.635+0000 7ff66ffff700 1 --2- 172.21.15.189:0/3820240054 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff6700cb870 0x7ff6700ca2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.635+0000 7ff66ffff700 1 -- 172.21.15.189:0/3820240054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6700d11b0 msgr2=0x7ff67006ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.635+0000 7ff66ffff700 1 --2- 172.21.15.189:0/3820240054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6700d11b0 0x7ff67006ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.635+0000 7ff66ffff700 1 -- 172.21.15.189:0/3820240054 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65c0094a0 con 0x7ff6700cfa00 2024-04-06T13:51:43.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.636+0000 7ff66ffff700 1 --2- 172.21.15.189:0/3820240054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6700cfa00 0x7ff67006efd0 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 crypto rx=0x7ff65c00ebe0 tx=0x7ff65c009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:43.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.636+0000 7ff66d7fa700 1 -- 172.21.15.189:0/3820240054 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c00c890 con 0x7ff6700cfa00 2024-04-06T13:51:43.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.636+0000 7ff66d7fa700 1 -- 172.21.15.189:0/3820240054 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65c00c9f0 con 0x7ff6700cfa00 2024-04-06T13:51:43.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.636+0000 7ff6770b1700 1 -- 172.21.15.189:0/3820240054 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff668009a70 con 0x7ff6700cfa00 2024-04-06T13:51:43.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.636+0000 7ff66d7fa700 1 -- 172.21.15.189:0/3820240054 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c00cda0 con 0x7ff6700cfa00 2024-04-06T13:51:43.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.636+0000 7ff6770b1700 1 -- 172.21.15.189:0/3820240054 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6700c9db0 con 0x7ff6700cfa00 2024-04-06T13:51:43.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.636+0000 7ff6770b1700 1 -- 172.21.15.189:0/3820240054 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff6700cc060 con 0x7ff6700cfa00 2024-04-06T13:51:43.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.637+0000 7ff66d7fa700 1 -- 172.21.15.189:0/3820240054 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7ff65c013470 con 0x7ff6700cfa00 2024-04-06T13:51:43.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.637+0000 7ff66d7fa700 1 --2- 172.21.15.189:0/3820240054 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff65406adb0 0x7ff65406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.638+0000 7ff66d7fa700 1 -- 172.21.15.189:0/3820240054 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff65c089ad0 con 0x7ff6700cfa00 2024-04-06T13:51:43.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.638+0000 7ff674e4d700 1 --2- 172.21.15.189:0/3820240054 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff65406adb0 0x7ff65406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.638+0000 7ff66d7fa700 1 --2- 172.21.15.189:0/3820240054 >> [v2:172.21.15.192:6800/47714,v1:172.21.15.192:6801/47714] conn(0x7ff6540716f0 0x7ff654073b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.638+0000 7ff66d7fa700 1 -- 172.21.15.189:0/3820240054 --> [v2:172.21.15.192:6800/47714,v1:172.21.15.192:6801/47714] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff654074220 con 0x7ff6540716f0 2024-04-06T13:51:43.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.638+0000 7ff66d7fa700 1 -- 172.21.15.189:0/3820240054 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=290) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff65c089f20 con 0x7ff6700cfa00 2024-04-06T13:51:43.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.638+0000 7ff67564e700 1 --2- 172.21.15.189:0/3820240054 >> [v2:172.21.15.192:6800/47714,v1:172.21.15.192:6801/47714] conn(0x7ff6540716f0 0x7ff654073b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.638+0000 7ff674e4d700 1 --2- 172.21.15.189:0/3820240054 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff65406adb0 0x7ff65406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff6640011e0 tx=0x7ff664009040 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.638+0000 7ff67564e700 1 --2- 172.21.15.189:0/3820240054 >> [v2:172.21.15.192:6800/47714,v1:172.21.15.192:6801/47714] conn(0x7ff6540716f0 0x7ff654073b50 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.639+0000 7ff66d7fa700 1 -- 172.21.15.189:0/3820240054 <== osd.5 v2:172.21.15.192:6800/47714 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7ff654074220 con 0x7ff6540716f0 2024-04-06T13:51:43.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.653+0000 7ff6770b1700 1 -- 172.21.15.189:0/3820240054 --> [v2:172.21.15.192:6800/47714,v1:172.21.15.192:6801/47714] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff6700cf4e0 con 0x7ff6540716f0 2024-04-06T13:51:43.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.654+0000 7ff66d7fa700 1 -- 172.21.15.189:0/3820240054 <== osd.5 v2:172.21.15.192:6800/47714 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff6680098f0 con 0x7ff6540716f0 2024-04-06T13:51:43.654 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:51:43.654 INFO:teuthology.orchestra.run.smithi189.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-06T13:51:43.654 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:51:43.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.654+0000 7ff6770b1700 1 -- 172.21.15.189:0/3820240054 >> [v2:172.21.15.192:6800/47714,v1:172.21.15.192:6801/47714] conn(0x7ff6540716f0 msgr2=0x7ff654073b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.654+0000 7ff6770b1700 1 --2- 172.21.15.189:0/3820240054 >> [v2:172.21.15.192:6800/47714,v1:172.21.15.192:6801/47714] conn(0x7ff6540716f0 0x7ff654073b50 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.654+0000 7ff6770b1700 1 -- 172.21.15.189:0/3820240054 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff65406adb0 msgr2=0x7ff65406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.654+0000 7ff6770b1700 1 --2- 172.21.15.189:0/3820240054 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7ff65406adb0 0x7ff65406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff6640011e0 tx=0x7ff664009040 comp rx=0 tx=0).stop 2024-04-06T13:51:43.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.654+0000 7ff6770b1700 1 -- 172.21.15.189:0/3820240054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6700cfa00 msgr2=0x7ff67006efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.654+0000 7ff6770b1700 1 --2- 172.21.15.189:0/3820240054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6700cfa00 0x7ff67006efd0 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 crypto rx=0x7ff65c00ebe0 tx=0x7ff65c009e60 comp rx=0 tx=0).stop 2024-04-06T13:51:43.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.654+0000 7ff67564e700 1 -- 172.21.15.189:0/3820240054 reap_dead start 2024-04-06T13:51:43.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.654+0000 7ff6770b1700 1 -- 172.21.15.189:0/3820240054 shutdown_connections 2024-04-06T13:51:43.656 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.654+0000 7ff6770b1700 1 -- 172.21.15.189:0/3820240054 >> 172.21.15.189:0/3820240054 conn(0x7ff6700b2db0 msgr2=0x7ff6700b2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:43.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.654+0000 7ff6770b1700 1 -- 172.21.15.189:0/3820240054 shutdown_connections 2024-04-06T13:51:43.657 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.654+0000 7ff6770b1700 1 -- 172.21.15.189:0/3820240054 wait complete. 2024-04-06T13:51:43.665 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-06T13:51:43.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.789+0000 7f4c11cf3700 1 Processor -- start 2024-04-06T13:51:43.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.789+0000 7f4c11cf3700 1 -- start start 2024-04-06T13:51:43.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.789+0000 7f4c11cf3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c0c0d11b0 0x7f4c0c0d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.789+0000 7f4c11cf3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4c0c0cfa00 0x7f4c0c0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.789+0000 7f4c11cf3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0c0d1ad0 con 0x7f4c0c0cb870 2024-04-06T13:51:43.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.789+0000 7f4c11cf3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c0c0ccdc0 con 0x7f4c0c0cfa00 2024-04-06T13:51:43.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.789+0000 7f4c11cf3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c0d0b50 con 0x7f4c0c0d11b0 2024-04-06T13:51:43.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.790+0000 7f4c0affd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4c0c0cb870 0x7f4c0c15c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:32914/0 (socket says 172.21.15.189:32914) 2024-04-06T13:51:43.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.790+0000 7f4c0affd700 1 -- 172.21.15.189:0/1669984273 learned_addr learned my addr 172.21.15.189:0/1669984273 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:43.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.790+0000 7f4c0bfff700 1 --2- 172.21.15.189:0/1669984273 >> v2:172.21.15.192:3300/0 conn(0x7f4c0c0cfa00 0x7f4c0c0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.790+0000 7f4c0b7fe700 1 --2- 172.21.15.189:0/1669984273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c0c0d11b0 0x7f4c0c0d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.790+0000 7f4c0a7fc700 1 -- 172.21.15.189:0/1669984273 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742384835 0 0) 0x7f4c0c0d1ad0 con 0x7f4c0c0cb870 2024-04-06T13:51:43.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.790+0000 7f4c0a7fc700 1 -- 172.21.15.189:0/1669984273 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bf80035b0 con 0x7f4c0c0cb870 2024-04-06T13:51:43.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.790+0000 7f4c0a7fc700 1 -- 172.21.15.189:0/1669984273 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2646873112 0 0) 0x7f4bf80035b0 con 0x7f4c0c0cb870 2024-04-06T13:51:43.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.790+0000 7f4c0bfff700 1 -- 172.21.15.189:0/1669984273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c0c0d11b0 msgr2=0x7f4c0c0d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:43.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.790+0000 7f4c0bfff700 1 --2- 172.21.15.189:0/1669984273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c0c0d11b0 0x7f4c0c0d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.790+0000 7f4c0bfff700 1 -- 172.21.15.189:0/1669984273 >> v1:172.21.15.189:6789/0 conn(0x7f4c0c0cb870 legacy=0x7f4c0c15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.790+0000 7f4c0bfff700 1 -- 172.21.15.189:0/1669984273 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0c0cbd80 con 0x7f4c0c0cfa00 2024-04-06T13:51:43.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.790+0000 7f4c0b7fe700 1 --2- 172.21.15.189:0/1669984273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c0c0d11b0 0x7f4c0c0d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T13:51:43.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.790+0000 7f4c0bfff700 1 --2- 172.21.15.189:0/1669984273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c0c0cfa00 0x7f4c0c0cd560 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f4c04009d90 tx=0x7f4c0400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18271048f8a06a87 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:43.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.790+0000 7f4c0a7fc700 1 -- 172.21.15.189:0/1669984273 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0400e070 con 0x7f4c0c0cfa00 2024-04-06T13:51:43.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.790+0000 7f4c0a7fc700 1 -- 172.21.15.189:0/1669984273 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c04003a20 con 0x7f4c0c0cfa00 2024-04-06T13:51:43.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.790+0000 7f4c0a7fc700 1 -- 172.21.15.189:0/1669984273 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4001670 con 0x7f4c0c0cfa00 2024-04-06T13:51:43.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.791+0000 7f4c11cf3700 1 -- 172.21.15.189:0/1669984273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c0c0cfa00 msgr2=0x7f4c0c0cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.791+0000 7f4c11cf3700 1 --2- 172.21.15.189:0/1669984273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c0c0cfa00 0x7f4c0c0cd560 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f4c04009d90 tx=0x7f4c0400b330 comp rx=0 tx=0).stop 2024-04-06T13:51:43.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.791+0000 7f4c11cf3700 1 -- 172.21.15.189:0/1669984273 shutdown_connections 2024-04-06T13:51:43.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.791+0000 7f4c11cf3700 1 --2- 172.21.15.189:0/1669984273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c0c0cfa00 0x7f4c0c0cd560 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.791+0000 7f4c11cf3700 1 --2- 172.21.15.189:0/1669984273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c0c0d11b0 0x7f4c0c0d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.791+0000 7f4c11cf3700 1 -- 172.21.15.189:0/1669984273 >> 172.21.15.189:0/1669984273 conn(0x7f4c0c0b2db0 msgr2=0x7f4c0c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:43.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.791+0000 7f4c11cf3700 1 -- 172.21.15.189:0/1669984273 shutdown_connections 2024-04-06T13:51:43.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.791+0000 7f4c11cf3700 1 -- 172.21.15.189:0/1669984273 wait complete. 2024-04-06T13:51:43.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.792+0000 7f4c11cf3700 1 Processor -- start 2024-04-06T13:51:43.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.792+0000 7f4c11cf3700 1 -- start start 2024-04-06T13:51:43.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.793+0000 7f4c11cf3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c0c0cb870 0x7f4c0c1485a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.793+0000 7f4c11cf3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c0c0cfa00 0x7f4c0c0c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.793+0000 7f4c11cf3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c0c0d11b0 0x7f4c0c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.793+0000 7f4c11cf3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c0c5730 con 0x7f4c0c0d11b0 2024-04-06T13:51:43.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.793+0000 7f4c11cf3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c0c5a10 con 0x7f4c0c0cb870 2024-04-06T13:51:43.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.793+0000 7f4c11cf3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c0c5cf0 con 0x7f4c0c0cfa00 2024-04-06T13:51:43.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.793+0000 7f4c0bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c0c0d11b0 0x7f4c0c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.793+0000 7f4c0bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c0c0d11b0 0x7f4c0c0ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44386/0 (socket says 172.21.15.189:44386) 2024-04-06T13:51:43.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.793+0000 7f4c0bfff700 1 -- 172.21.15.189:0/687855001 learned_addr learned my addr 172.21.15.189:0/687855001 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:43.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.793+0000 7f4c0b7fe700 1 --2- 172.21.15.189:0/687855001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c0c0cb870 0x7f4c0c1485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.793+0000 7f4c0affd700 1 --2- 172.21.15.189:0/687855001 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c0c0cfa00 0x7f4c0c0c4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.793+0000 7f4c0bfff700 1 -- 172.21.15.189:0/687855001 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c0c0cfa00 msgr2=0x7f4c0c0c4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.793+0000 7f4c0bfff700 1 --2- 172.21.15.189:0/687855001 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c0c0cfa00 0x7f4c0c0c4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.793+0000 7f4c0bfff700 1 -- 172.21.15.189:0/687855001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c0c0cb870 msgr2=0x7f4c0c1485a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.793+0000 7f4c0bfff700 1 --2- 172.21.15.189:0/687855001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c0c0cb870 0x7f4c0c1485a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.793+0000 7f4c0bfff700 1 -- 172.21.15.189:0/687855001 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bfc0094a0 con 0x7f4c0c0d11b0 2024-04-06T13:51:43.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.793+0000 7f4c0b7fe700 1 --2- 172.21.15.189:0/687855001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c0c0cb870 0x7f4c0c1485a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:51:43.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.794+0000 7f4c0bfff700 1 --2- 172.21.15.189:0/687855001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c0c0d11b0 0x7f4c0c0ca3b0 secure :-1 s=READY pgs=3239 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c0677e0 tx=0x7f4c04004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:43.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.794+0000 7f4c03fff700 1 -- 172.21.15.189:0/687855001 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04004600 con 0x7f4c0c0d11b0 2024-04-06T13:51:43.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.794+0000 7f4c03fff700 1 -- 172.21.15.189:0/687855001 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c04004760 con 0x7f4c0c0d11b0 2024-04-06T13:51:43.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.794+0000 7f4c11cf3700 1 -- 172.21.15.189:0/687855001 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c04009a70 con 0x7f4c0c0d11b0 2024-04-06T13:51:43.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.794+0000 7f4c03fff700 1 -- 172.21.15.189:0/687855001 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04012430 con 0x7f4c0c0d11b0 2024-04-06T13:51:43.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.794+0000 7f4c11cf3700 1 -- 172.21.15.189:0/687855001 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c0c0c62b0 con 0x7f4c0c0d11b0 2024-04-06T13:51:43.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.795+0000 7f4c11cf3700 1 -- 172.21.15.189:0/687855001 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c0c074990 con 0x7f4c0c0d11b0 2024-04-06T13:51:43.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.795+0000 7f4c03fff700 1 -- 172.21.15.189:0/687855001 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f4c040125b0 con 0x7f4c0c0d11b0 2024-04-06T13:51:43.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.796+0000 7f4c03fff700 1 --2- 172.21.15.189:0/687855001 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4bec06adb0 0x7f4bec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:43.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.796+0000 7f4c03fff700 1 -- 172.21.15.189:0/687855001 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f4c0408b4c0 con 0x7f4c0c0d11b0 2024-04-06T13:51:43.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.796+0000 7f4c0b7fe700 1 --2- 172.21.15.189:0/687855001 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4bec06adb0 0x7f4bec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:43.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.796+0000 7f4c0b7fe700 1 --2- 172.21.15.189:0/687855001 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4bec06adb0 0x7f4bec06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc009c50 tx=0x7f4bfc008040 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:43.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.800+0000 7f4c03fff700 1 -- 172.21.15.189:0/687855001 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4c040563a0 con 0x7f4c0c0d11b0 2024-04-06T13:51:43.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.993+0000 7f4c11cf3700 1 -- 172.21.15.189:0/687855001 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c0c0c5b80 con 0x7f4bec06adb0 2024-04-06T13:51:43.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.994+0000 7f4c03fff700 1 -- 172.21.15.189:0/687855001 <== mgr.10415 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4c0c0c5b80 con 0x7f4bec06adb0 2024-04-06T13:51:43.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.996+0000 7f4c11cf3700 1 -- 172.21.15.189:0/687855001 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4bec06adb0 msgr2=0x7f4bec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.996+0000 7f4c11cf3700 1 --2- 172.21.15.189:0/687855001 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4bec06adb0 0x7f4bec06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc009c50 tx=0x7f4bfc008040 comp rx=0 tx=0).stop 2024-04-06T13:51:43.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.996+0000 7f4c11cf3700 1 -- 172.21.15.189:0/687855001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c0c0d11b0 msgr2=0x7f4c0c0ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:43.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.996+0000 7f4c11cf3700 1 --2- 172.21.15.189:0/687855001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c0c0d11b0 0x7f4c0c0ca3b0 secure :-1 s=READY pgs=3239 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c0677e0 tx=0x7f4c04004ad0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.996+0000 7f4c11cf3700 1 -- 172.21.15.189:0/687855001 shutdown_connections 2024-04-06T13:51:43.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.996+0000 7f4c11cf3700 1 --2- 172.21.15.189:0/687855001 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f4bec06adb0 0x7f4bec06d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.996+0000 7f4c11cf3700 1 --2- 172.21.15.189:0/687855001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c0c0d11b0 0x7f4c0c0ca3b0 unknown :-1 s=CLOSED pgs=3239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.996+0000 7f4c11cf3700 1 --2- 172.21.15.189:0/687855001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c0c0cb870 0x7f4c0c1485a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.996+0000 7f4c11cf3700 1 --2- 172.21.15.189:0/687855001 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4c0c0cfa00 0x7f4c0c0c4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:43.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.996+0000 7f4c11cf3700 1 -- 172.21.15.189:0/687855001 >> 172.21.15.189:0/687855001 conn(0x7f4c0c0b2db0 msgr2=0x7f4c0c06c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:43.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.996+0000 7f4c11cf3700 1 -- 172.21.15.189:0/687855001 shutdown_connections 2024-04-06T13:51:43.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:43.996+0000 7f4c11cf3700 1 -- 172.21.15.189:0/687855001 wait complete. 2024-04-06T13:51:43.998 INFO:teuthology.orchestra.run.smithi189.stderr:instructed osd(s) 5 to deep-scrub 2024-04-06T13:51:44.008 INFO:tasks.ceph:Scrubbing osd.6 2024-04-06T13:51:44.008 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-06T13:51:44.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.134+0000 7fbd9fccf700 1 Processor -- start 2024-04-06T13:51:44.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.134+0000 7fbd9fccf700 1 -- start start 2024-04-06T13:51:44.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.135+0000 7fbd9fccf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd980b4af0 0x7fbd980b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.135+0000 7fbd9fccf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd980cdbe0 con 0x7fbd9815a550 2024-04-06T13:51:44.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.135+0000 7fbd9fccf700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd9815a940 con 0x7fbd980b4660 2024-04-06T13:51:44.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.135+0000 7fbd9fccf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9815ac00 con 0x7fbd980b4af0 2024-04-06T13:51:44.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.135+0000 7fbd9da6b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbd9815a550 0x7fbd980cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:32918/0 (socket says 172.21.15.189:32918) 2024-04-06T13:51:44.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.135+0000 7fbd9da6b700 1 -- 172.21.15.189:0/3465516382 learned_addr learned my addr 172.21.15.189:0/3465516382 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:44.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.135+0000 7fbd9ca69700 1 -- 172.21.15.189:0/3465516382 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1657617351 0 0) 0x7fbd980cdbe0 con 0x7fbd9815a550 2024-04-06T13:51:44.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.135+0000 7fbd9ca69700 1 -- 172.21.15.189:0/3465516382 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd840035b0 con 0x7fbd9815a550 2024-04-06T13:51:44.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9e26c700 1 --2- 172.21.15.189:0/3465516382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd980b4af0 0x7fbd980b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9ca69700 1 -- 172.21.15.189:0/3465516382 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1806701574 0 0) 0x7fbd840035b0 con 0x7fbd9815a550 2024-04-06T13:51:44.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9ca69700 1 -- 172.21.15.189:0/3465516382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd980b4af0 msgr2=0x7fbd980b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:44.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9ca69700 1 --2- 172.21.15.189:0/3465516382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd980b4af0 0x7fbd980b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9ca69700 1 -- 172.21.15.189:0/3465516382 >> v1:172.21.15.192:6789/0 conn(0x7fbd980b4660 legacy=0x7fbd980b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9ca69700 1 -- 172.21.15.189:0/3465516382 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9815aee0 con 0x7fbd9815a550 2024-04-06T13:51:44.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9ca69700 1 -- 172.21.15.189:0/3465516382 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7fbd94001670 con 0x7fbd9815a550 2024-04-06T13:51:44.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9ca69700 1 -- 172.21.15.189:0/3465516382 >> v1:172.21.15.189:6789/0 conn(0x7fbd9815a550 legacy=0x7fbd980cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9ca69700 1 --2- 172.21.15.189:0/3465516382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd84003a30 0x7fbd84003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9ca69700 1 --2- 172.21.15.189:0/3465516382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd980b4af0 0x7fbd84004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9ca69700 1 --2- 172.21.15.189:0/3465516382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd84004d90 0x7fbd84005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9ca69700 1 -- 172.21.15.189:0/3465516382 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9815ac00 con 0x7fbd84003a30 2024-04-06T13:51:44.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9ca69700 1 -- 172.21.15.189:0/3465516382 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8400bba0 con 0x7fbd84004d90 2024-04-06T13:51:44.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9ca69700 1 -- 172.21.15.189:0/3465516382 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8400bd30 con 0x7fbd980b4af0 2024-04-06T13:51:44.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9da6b700 1 --2- 172.21.15.189:0/3465516382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd84003a30 0x7fbd84003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9d26a700 1 --2- 172.21.15.189:0/3465516382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd980b4af0 0x7fbd84004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9e26c700 1 --2- 172.21.15.189:0/3465516382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd84004d90 0x7fbd84005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9da6b700 1 -- 172.21.15.189:0/3465516382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd980b4af0 msgr2=0x7fbd84004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:44.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9da6b700 1 --2- 172.21.15.189:0/3465516382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd980b4af0 0x7fbd84004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9da6b700 1 -- 172.21.15.189:0/3465516382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd84004d90 msgr2=0x7fbd84005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:44.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9da6b700 1 --2- 172.21.15.189:0/3465516382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd84004d90 0x7fbd84005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9da6b700 1 -- 172.21.15.189:0/3465516382 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd840035b0 con 0x7fbd84003a30 2024-04-06T13:51:44.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9d26a700 1 --2- 172.21.15.189:0/3465516382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd980b4af0 0x7fbd84004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:51:44.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9da6b700 1 --2- 172.21.15.189:0/3465516382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd84003a30 0x7fbd84003e50 secure :-1 s=READY pgs=3241 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400c270 tx=0x7fbd84005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=487b376500f80324 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:44.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9ca69700 1 -- 172.21.15.189:0/3465516382 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd94007c10 con 0x7fbd84003a30 2024-04-06T13:51:44.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9ca69700 1 -- 172.21.15.189:0/3465516382 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9400cdf0 con 0x7fbd84003a30 2024-04-06T13:51:44.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.136+0000 7fbd9ca69700 1 -- 172.21.15.189:0/3465516382 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd94005ba0 con 0x7fbd84003a30 2024-04-06T13:51:44.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.137+0000 7fbd9fccf700 1 -- 172.21.15.189:0/3465516382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd84003a30 msgr2=0x7fbd84003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.137+0000 7fbd9fccf700 1 --2- 172.21.15.189:0/3465516382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd84003a30 0x7fbd84003e50 secure :-1 s=READY pgs=3241 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400c270 tx=0x7fbd84005880 comp rx=0 tx=0).stop 2024-04-06T13:51:44.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.137+0000 7fbd9e26c700 1 -- 172.21.15.189:0/3465516382 reap_dead start 2024-04-06T13:51:44.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.137+0000 7fbd9fccf700 1 -- 172.21.15.189:0/3465516382 shutdown_connections 2024-04-06T13:51:44.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.137+0000 7fbd9fccf700 1 -- 172.21.15.189:0/3465516382 >> 172.21.15.189:0/3465516382 conn(0x7fbd980d00b0 msgr2=0x7fbd980d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:44.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.137+0000 7fbd9fccf700 1 -- 172.21.15.189:0/3465516382 shutdown_connections 2024-04-06T13:51:44.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.137+0000 7fbd9fccf700 1 -- 172.21.15.189:0/3465516382 wait complete. 2024-04-06T13:51:44.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.138+0000 7fbd9fccf700 1 Processor -- start 2024-04-06T13:51:44.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.138+0000 7fbd9fccf700 1 -- start start 2024-04-06T13:51:44.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.139+0000 7fbd9fccf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd84003a30 0x7fbd980adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.139+0000 7fbd9fccf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd98156e40 0x7fbd980ad3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.139+0000 7fbd9fccf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd980b2d20 0x7fbd980b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.139+0000 7fbd9fccf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd980b36e0 con 0x7fbd84003a30 2024-04-06T13:51:44.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.139+0000 7fbd9fccf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd980ad930 con 0x7fbd98156e40 2024-04-06T13:51:44.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.139+0000 7fbd9fccf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd981572c0 con 0x7fbd980b2d20 2024-04-06T13:51:44.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.139+0000 7fbd9da6b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd84003a30 0x7fbd980adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.139+0000 7fbd9da6b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd84003a30 0x7fbd980adcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44406/0 (socket says 172.21.15.189:44406) 2024-04-06T13:51:44.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.139+0000 7fbd9da6b700 1 -- 172.21.15.189:0/2322755479 learned_addr learned my addr 172.21.15.189:0/2322755479 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:44.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.139+0000 7fbd9da6b700 1 -- 172.21.15.189:0/2322755479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd980b2d20 msgr2=0x7fbd980b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.139+0000 7fbd9d26a700 1 --2- 172.21.15.189:0/2322755479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd98156e40 0x7fbd980ad3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.139+0000 7fbd9da6b700 1 --2- 172.21.15.189:0/2322755479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd980b2d20 0x7fbd980b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.139+0000 7fbd9da6b700 1 -- 172.21.15.189:0/2322755479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd98156e40 msgr2=0x7fbd980ad3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.139+0000 7fbd9da6b700 1 --2- 172.21.15.189:0/2322755479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd98156e40 0x7fbd980ad3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.139+0000 7fbd9da6b700 1 -- 172.21.15.189:0/2322755479 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd980ae3c0 con 0x7fbd84003a30 2024-04-06T13:51:44.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.140+0000 7fbd9da6b700 1 --2- 172.21.15.189:0/2322755479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd84003a30 0x7fbd980adcc0 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 crypto rx=0x7fbd840057f0 tx=0x7fbd94001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:44.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.140+0000 7fbd8e7fc700 1 -- 172.21.15.189:0/2322755479 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd94004c80 con 0x7fbd84003a30 2024-04-06T13:51:44.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.140+0000 7fbd8e7fc700 1 -- 172.21.15.189:0/2322755479 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd94004de0 con 0x7fbd84003a30 2024-04-06T13:51:44.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.140+0000 7fbd9fccf700 1 -- 172.21.15.189:0/2322755479 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd980c4fd0 con 0x7fbd84003a30 2024-04-06T13:51:44.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.140+0000 7fbd8e7fc700 1 -- 172.21.15.189:0/2322755479 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd94007310 con 0x7fbd84003a30 2024-04-06T13:51:44.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.140+0000 7fbd9fccf700 1 -- 172.21.15.189:0/2322755479 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd980c5590 con 0x7fbd84003a30 2024-04-06T13:51:44.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.140+0000 7fbd9fccf700 1 -- 172.21.15.189:0/2322755479 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbd9806fdd0 con 0x7fbd84003a30 2024-04-06T13:51:44.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.141+0000 7fbd8e7fc700 1 -- 172.21.15.189:0/2322755479 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fbd94019070 con 0x7fbd84003a30 2024-04-06T13:51:44.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.141+0000 7fbd8e7fc700 1 --2- 172.21.15.189:0/2322755479 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbd7c06adb0 0x7fbd7c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.142+0000 7fbd8e7fc700 1 -- 172.21.15.189:0/2322755479 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fbd9408ad70 con 0x7fbd84003a30 2024-04-06T13:51:44.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.142+0000 7fbd9d26a700 1 --2- 172.21.15.189:0/2322755479 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbd7c06adb0 0x7fbd7c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.142+0000 7fbd8e7fc700 1 --2- 172.21.15.189:0/2322755479 >> [v2:172.21.15.192:6819/47717,v1:172.21.15.192:6821/47717] conn(0x7fbd7c0716f0 0x7fbd7c073b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.142+0000 7fbd8e7fc700 1 -- 172.21.15.189:0/2322755479 --> [v2:172.21.15.192:6819/47717,v1:172.21.15.192:6821/47717] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd7c074220 con 0x7fbd7c0716f0 2024-04-06T13:51:44.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.142+0000 7fbd8e7fc700 1 -- 172.21.15.189:0/2322755479 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=290) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd9408b1c0 con 0x7fbd84003a30 2024-04-06T13:51:44.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.142+0000 7fbd9d26a700 1 --2- 172.21.15.189:0/2322755479 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbd7c06adb0 0x7fbd7c06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbd9806b810 tx=0x7fbd88009040 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:44.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.142+0000 7fbd9e26c700 1 --2- 172.21.15.189:0/2322755479 >> [v2:172.21.15.192:6819/47717,v1:172.21.15.192:6821/47717] conn(0x7fbd7c0716f0 0x7fbd7c073b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.142+0000 7fbd9e26c700 1 --2- 172.21.15.189:0/2322755479 >> [v2:172.21.15.192:6819/47717,v1:172.21.15.192:6821/47717] conn(0x7fbd7c0716f0 0x7fbd7c073b50 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:44.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.143+0000 7fbd8e7fc700 1 -- 172.21.15.189:0/2322755479 <== osd.6 v2:172.21.15.192:6819/47717 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fbd7c074220 con 0x7fbd7c0716f0 2024-04-06T13:51:44.157 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.157+0000 7fbd9fccf700 1 -- 172.21.15.189:0/2322755479 --> [v2:172.21.15.192:6819/47717,v1:172.21.15.192:6821/47717] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fbd9815b4a0 con 0x7fbd7c0716f0 2024-04-06T13:51:44.158 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.158+0000 7fbd8e7fc700 1 -- 172.21.15.189:0/2322755479 <== osd.6 v2:172.21.15.192:6819/47717 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fbd90009ca0 con 0x7fbd7c0716f0 2024-04-06T13:51:44.158 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:51:44.158 INFO:teuthology.orchestra.run.smithi189.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-06T13:51:44.159 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:51:44.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.158+0000 7fbd9fccf700 1 -- 172.21.15.189:0/2322755479 >> [v2:172.21.15.192:6819/47717,v1:172.21.15.192:6821/47717] conn(0x7fbd7c0716f0 msgr2=0x7fbd7c073b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.158+0000 7fbd9fccf700 1 --2- 172.21.15.189:0/2322755479 >> [v2:172.21.15.192:6819/47717,v1:172.21.15.192:6821/47717] conn(0x7fbd7c0716f0 0x7fbd7c073b50 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.158+0000 7fbd9fccf700 1 -- 172.21.15.189:0/2322755479 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbd7c06adb0 msgr2=0x7fbd7c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.158+0000 7fbd9fccf700 1 --2- 172.21.15.189:0/2322755479 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fbd7c06adb0 0x7fbd7c06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbd9806b810 tx=0x7fbd88009040 comp rx=0 tx=0).stop 2024-04-06T13:51:44.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.158+0000 7fbd9fccf700 1 -- 172.21.15.189:0/2322755479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd84003a30 msgr2=0x7fbd980adcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.158+0000 7fbd9fccf700 1 --2- 172.21.15.189:0/2322755479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd84003a30 0x7fbd980adcc0 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 crypto rx=0x7fbd840057f0 tx=0x7fbd94001a00 comp rx=0 tx=0).stop 2024-04-06T13:51:44.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.158+0000 7fbd9e26c700 1 -- 172.21.15.189:0/2322755479 reap_dead start 2024-04-06T13:51:44.160 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.158+0000 7fbd9fccf700 1 -- 172.21.15.189:0/2322755479 shutdown_connections 2024-04-06T13:51:44.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.158+0000 7fbd9fccf700 1 -- 172.21.15.189:0/2322755479 >> 172.21.15.189:0/2322755479 conn(0x7fbd980d00b0 msgr2=0x7fbd980717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:44.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.158+0000 7fbd9fccf700 1 -- 172.21.15.189:0/2322755479 shutdown_connections 2024-04-06T13:51:44.161 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.158+0000 7fbd9fccf700 1 -- 172.21.15.189:0/2322755479 wait complete. 2024-04-06T13:51:44.171 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-06T13:51:44.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.294+0000 7feb21421700 1 Processor -- start 2024-04-06T13:51:44.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.294+0000 7feb21421700 1 -- start start 2024-04-06T13:51:44.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.294+0000 7feb21421700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb1c15d6f0 0x7feb1c15dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.296 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.294+0000 7feb21421700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7feb1c0b8d70 0x7feb1c0ce020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.294+0000 7feb21421700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1c0ce9b0 con 0x7feb1c0ce560 2024-04-06T13:51:44.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.294+0000 7feb21421700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb1c15e010 con 0x7feb1c0b8d70 2024-04-06T13:51:44.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.294+0000 7feb21421700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb1c15eac0 con 0x7feb1c15d6f0 2024-04-06T13:51:44.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.295+0000 7feb1b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7feb1c0ce560 0x7feb1c0b4e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:32926/0 (socket says 172.21.15.189:32926) 2024-04-06T13:51:44.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.295+0000 7feb1b7fe700 1 -- 172.21.15.189:0/161012034 learned_addr learned my addr 172.21.15.189:0/161012034 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:44.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.295+0000 7feb1a7fc700 1 --2- 172.21.15.189:0/161012034 >> v2:172.21.15.192:3300/0 conn(0x7feb1c0b8d70 0x7feb1c0ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.295+0000 7feb1affd700 1 --2- 172.21.15.189:0/161012034 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb1c15d6f0 0x7feb1c15dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.295+0000 7feb19ffb700 1 -- 172.21.15.189:0/161012034 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970844772 0 0) 0x7feb14001490 con 0x7feb1c0ce560 2024-04-06T13:51:44.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.295+0000 7feb19ffb700 1 -- 172.21.15.189:0/161012034 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb040035b0 con 0x7feb1c0ce560 2024-04-06T13:51:44.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.295+0000 7feb1affd700 1 -- 172.21.15.189:0/161012034 >> v2:172.21.15.192:3300/0 conn(0x7feb1c0b8d70 msgr2=0x7feb1c0ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:44.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.295+0000 7feb1affd700 1 --2- 172.21.15.189:0/161012034 >> v2:172.21.15.192:3300/0 conn(0x7feb1c0b8d70 0x7feb1c0ce020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.295+0000 7feb1affd700 1 -- 172.21.15.189:0/161012034 >> v1:172.21.15.189:6789/0 conn(0x7feb1c0ce560 legacy=0x7feb1c0b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.295+0000 7feb1affd700 1 -- 172.21.15.189:0/161012034 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb1c15e220 con 0x7feb1c15d6f0 2024-04-06T13:51:44.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.295+0000 7feb1affd700 1 --2- 172.21.15.189:0/161012034 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb1c15d6f0 0x7feb1c15dad0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7feb0c009840 tx=0x7feb0c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d3f47fed622ac5d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:44.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.295+0000 7feb19ffb700 1 -- 172.21.15.189:0/161012034 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c011830 con 0x7feb1c15d6f0 2024-04-06T13:51:44.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.295+0000 7feb19ffb700 1 -- 172.21.15.189:0/161012034 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0c011990 con 0x7feb1c15d6f0 2024-04-06T13:51:44.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.295+0000 7feb19ffb700 1 -- 172.21.15.189:0/161012034 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c011d00 con 0x7feb1c15d6f0 2024-04-06T13:51:44.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.296+0000 7feb21421700 1 -- 172.21.15.189:0/161012034 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb1c15d6f0 msgr2=0x7feb1c15dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.296+0000 7feb21421700 1 --2- 172.21.15.189:0/161012034 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb1c15d6f0 0x7feb1c15dad0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7feb0c009840 tx=0x7feb0c009b50 comp rx=0 tx=0).stop 2024-04-06T13:51:44.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.296+0000 7feb21421700 1 -- 172.21.15.189:0/161012034 shutdown_connections 2024-04-06T13:51:44.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.296+0000 7feb21421700 1 --2- 172.21.15.189:0/161012034 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb1c15d6f0 0x7feb1c15dad0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.296+0000 7feb21421700 1 --2- 172.21.15.189:0/161012034 >> v2:172.21.15.192:3300/0 conn(0x7feb1c0b8d70 0x7feb1c0ce020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.296+0000 7feb21421700 1 -- 172.21.15.189:0/161012034 >> 172.21.15.189:0/161012034 conn(0x7feb1c072490 msgr2=0x7feb1c06b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:44.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.296+0000 7feb21421700 1 -- 172.21.15.189:0/161012034 shutdown_connections 2024-04-06T13:51:44.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.296+0000 7feb21421700 1 -- 172.21.15.189:0/161012034 wait complete. 2024-04-06T13:51:44.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.297+0000 7feb21421700 1 Processor -- start 2024-04-06T13:51:44.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.297+0000 7feb21421700 1 -- start start 2024-04-06T13:51:44.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.298+0000 7feb21421700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb1c0b8d70 0x7feb1c14af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.298+0000 7feb21421700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb1c0ce560 0x7feb1c0d22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.298+0000 7feb21421700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb1c15d6f0 0x7feb1c0d27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.298+0000 7feb21421700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1c14e310 con 0x7feb1c0ce560 2024-04-06T13:51:44.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.298+0000 7feb21421700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1c1f6df0 con 0x7feb1c15d6f0 2024-04-06T13:51:44.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.298+0000 7feb21421700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb1c14d970 con 0x7feb1c0b8d70 2024-04-06T13:51:44.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.298+0000 7feb1a7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb1c0ce560 0x7feb1c0d22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.298+0000 7feb1a7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb1c0ce560 0x7feb1c0d22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44422/0 (socket says 172.21.15.189:44422) 2024-04-06T13:51:44.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.298+0000 7feb1a7fc700 1 -- 172.21.15.189:0/2683008388 learned_addr learned my addr 172.21.15.189:0/2683008388 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:44.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.298+0000 7feb1b7fe700 1 --2- 172.21.15.189:0/2683008388 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb1c15d6f0 0x7feb1c0d27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.298+0000 7feb1affd700 1 --2- 172.21.15.189:0/2683008388 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb1c0b8d70 0x7feb1c14af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.298+0000 7feb1a7fc700 1 -- 172.21.15.189:0/2683008388 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb1c0b8d70 msgr2=0x7feb1c14af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.298+0000 7feb1a7fc700 1 --2- 172.21.15.189:0/2683008388 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb1c0b8d70 0x7feb1c14af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.298+0000 7feb1a7fc700 1 -- 172.21.15.189:0/2683008388 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb1c15d6f0 msgr2=0x7feb1c0d27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.298+0000 7feb1a7fc700 1 --2- 172.21.15.189:0/2683008388 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb1c15d6f0 0x7feb1c0d27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.298+0000 7feb1a7fc700 1 -- 172.21.15.189:0/2683008388 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0c0094a0 con 0x7feb1c0ce560 2024-04-06T13:51:44.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.299+0000 7feb1a7fc700 1 --2- 172.21.15.189:0/2683008388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb1c0ce560 0x7feb1c0d22a0 secure :-1 s=READY pgs=3244 cs=0 l=1 rev1=1 crypto rx=0x7feb1000e870 tx=0x7feb1000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:44.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.299+0000 7feb037fe700 1 -- 172.21.15.189:0/2683008388 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb10009590 con 0x7feb1c0ce560 2024-04-06T13:51:44.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.299+0000 7feb037fe700 1 -- 172.21.15.189:0/2683008388 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1000ee70 con 0x7feb1c0ce560 2024-04-06T13:51:44.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.299+0000 7feb21421700 1 -- 172.21.15.189:0/2683008388 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb1c14dc50 con 0x7feb1c0ce560 2024-04-06T13:51:44.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.299+0000 7feb037fe700 1 -- 172.21.15.189:0/2683008388 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1000ca50 con 0x7feb1c0ce560 2024-04-06T13:51:44.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.299+0000 7feb21421700 1 -- 172.21.15.189:0/2683008388 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb1c14e730 con 0x7feb1c0ce560 2024-04-06T13:51:44.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.300+0000 7feb21421700 1 -- 172.21.15.189:0/2683008388 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb1c14e450 con 0x7feb1c0ce560 2024-04-06T13:51:44.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.300+0000 7feb037fe700 1 -- 172.21.15.189:0/2683008388 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7feb1000cbb0 con 0x7feb1c0ce560 2024-04-06T13:51:44.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.301+0000 7feb037fe700 1 --2- 172.21.15.189:0/2683008388 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feb0806adb0 0x7feb0806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.301+0000 7feb037fe700 1 -- 172.21.15.189:0/2683008388 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7feb100898d0 con 0x7feb1c0ce560 2024-04-06T13:51:44.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.301+0000 7feb1affd700 1 --2- 172.21.15.189:0/2683008388 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feb0806adb0 0x7feb0806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.301+0000 7feb1affd700 1 --2- 172.21.15.189:0/2683008388 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feb0806adb0 0x7feb0806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feb0c000c00 tx=0x7feb0c008030 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:44.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.305+0000 7feb037fe700 1 -- 172.21.15.189:0/2683008388 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7feb1c14e450 con 0x7feb1c0ce560 2024-04-06T13:51:44.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.500+0000 7feb21421700 1 -- 172.21.15.189:0/2683008388 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7feb1c0d18e0 con 0x7feb0806adb0 2024-04-06T13:51:44.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.500+0000 7feb037fe700 1 -- 172.21.15.189:0/2683008388 <== mgr.10415 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7feb0c011630 con 0x7feb0806adb0 2024-04-06T13:51:44.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.503+0000 7feb21421700 1 -- 172.21.15.189:0/2683008388 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feb0806adb0 msgr2=0x7feb0806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.503+0000 7feb21421700 1 --2- 172.21.15.189:0/2683008388 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feb0806adb0 0x7feb0806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feb0c000c00 tx=0x7feb0c008030 comp rx=0 tx=0).stop 2024-04-06T13:51:44.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.503+0000 7feb21421700 1 -- 172.21.15.189:0/2683008388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb1c0ce560 msgr2=0x7feb1c0d22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.503+0000 7feb21421700 1 --2- 172.21.15.189:0/2683008388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb1c0ce560 0x7feb1c0d22a0 secure :-1 s=READY pgs=3244 cs=0 l=1 rev1=1 crypto rx=0x7feb1000e870 tx=0x7feb1000ec30 comp rx=0 tx=0).stop 2024-04-06T13:51:44.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.503+0000 7feb21421700 1 -- 172.21.15.189:0/2683008388 shutdown_connections 2024-04-06T13:51:44.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.503+0000 7feb21421700 1 --2- 172.21.15.189:0/2683008388 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7feb0806adb0 0x7feb0806d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.503+0000 7feb21421700 1 --2- 172.21.15.189:0/2683008388 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb1c15d6f0 0x7feb1c0d27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.503+0000 7feb21421700 1 --2- 172.21.15.189:0/2683008388 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7feb1c0b8d70 0x7feb1c14af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.503+0000 7feb21421700 1 --2- 172.21.15.189:0/2683008388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb1c0ce560 0x7feb1c0d22a0 unknown :-1 s=CLOSED pgs=3244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.503+0000 7feb21421700 1 -- 172.21.15.189:0/2683008388 >> 172.21.15.189:0/2683008388 conn(0x7feb1c072490 msgr2=0x7feb1c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:44.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.503+0000 7feb21421700 1 -- 172.21.15.189:0/2683008388 shutdown_connections 2024-04-06T13:51:44.505 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.503+0000 7feb21421700 1 -- 172.21.15.189:0/2683008388 wait complete. 2024-04-06T13:51:44.505 INFO:teuthology.orchestra.run.smithi189.stderr:instructed osd(s) 6 to deep-scrub 2024-04-06T13:51:44.515 INFO:tasks.ceph:Scrubbing osd.7 2024-04-06T13:51:44.515 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-06T13:51:44.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.642+0000 7fdba5317700 1 Processor -- start 2024-04-06T13:51:44.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.642+0000 7fdba5317700 1 -- start start 2024-04-06T13:51:44.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.643+0000 7fdba5317700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdba0143470 0x7fdba00b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.643+0000 7fdba5317700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba015eb40 con 0x7fdba0160b60 2024-04-06T13:51:44.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.643+0000 7fdba5317700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba00d0bf0 con 0x7fdba015db10 2024-04-06T13:51:44.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.643+0000 7fdba5317700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba0160f40 con 0x7fdba0143470 2024-04-06T13:51:44.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.643+0000 7fdb9effd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdba0160b60 0x7fdba0142d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:32930/0 (socket says 172.21.15.189:32930) 2024-04-06T13:51:44.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.643+0000 7fdb9effd700 1 -- 172.21.15.189:0/2585656083 learned_addr learned my addr 172.21.15.189:0/2585656083 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:44.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.643+0000 7fdb9f7fe700 1 --2- 172.21.15.189:0/2585656083 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdba0143470 0x7fdba00b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.643+0000 7fdb9dffb700 1 -- 172.21.15.189:0/2585656083 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100636274 0 0) 0x7fdb94001110 con 0x7fdba0160b60 2024-04-06T13:51:44.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.643+0000 7fdb9dffb700 1 -- 172.21.15.189:0/2585656083 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb880035b0 con 0x7fdba0160b60 2024-04-06T13:51:44.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.643+0000 7fdb9dffb700 1 -- 172.21.15.189:0/2585656083 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979996539 0 0) 0x7fdba00d0bf0 con 0x7fdba015db10 2024-04-06T13:51:44.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.643+0000 7fdb9dffb700 1 -- 172.21.15.189:0/2585656083 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb94001110 con 0x7fdba015db10 2024-04-06T13:51:44.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.643+0000 7fdb9f7fe700 1 -- 172.21.15.189:0/2585656083 >> v1:172.21.15.192:6789/0 conn(0x7fdba015db10 legacy=0x7fdba00cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.643+0000 7fdb9f7fe700 1 -- 172.21.15.189:0/2585656083 >> v1:172.21.15.189:6789/0 conn(0x7fdba0160b60 legacy=0x7fdba0142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.643+0000 7fdb9f7fe700 1 -- 172.21.15.189:0/2585656083 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba015dfc0 con 0x7fdba0143470 2024-04-06T13:51:44.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.643+0000 7fdb9f7fe700 1 --2- 172.21.15.189:0/2585656083 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdba0143470 0x7fdba00b37f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fdb98009dc0 tx=0x7fdb9800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c9dcca965593aaf server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:44.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.644+0000 7fdb9dffb700 1 -- 172.21.15.189:0/2585656083 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9800e070 con 0x7fdba0143470 2024-04-06T13:51:44.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.644+0000 7fdb9dffb700 1 -- 172.21.15.189:0/2585656083 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9800bb20 con 0x7fdba0143470 2024-04-06T13:51:44.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.644+0000 7fdb9dffb700 1 -- 172.21.15.189:0/2585656083 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9800be90 con 0x7fdba0143470 2024-04-06T13:51:44.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.644+0000 7fdba5317700 1 -- 172.21.15.189:0/2585656083 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdba0143470 msgr2=0x7fdba00b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.644+0000 7fdba5317700 1 --2- 172.21.15.189:0/2585656083 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdba0143470 0x7fdba00b37f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fdb98009dc0 tx=0x7fdb9800b330 comp rx=0 tx=0).stop 2024-04-06T13:51:44.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.644+0000 7fdba5317700 1 -- 172.21.15.189:0/2585656083 shutdown_connections 2024-04-06T13:51:44.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.644+0000 7fdba5317700 1 --2- 172.21.15.189:0/2585656083 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdba0143470 0x7fdba00b37f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.644+0000 7fdba5317700 1 -- 172.21.15.189:0/2585656083 >> 172.21.15.189:0/2585656083 conn(0x7fdba006a730 msgr2=0x7fdba006ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:44.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.644+0000 7fdba5317700 1 -- 172.21.15.189:0/2585656083 shutdown_connections 2024-04-06T13:51:44.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.644+0000 7fdba5317700 1 -- 172.21.15.189:0/2585656083 wait complete. 2024-04-06T13:51:44.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.645+0000 7fdba5317700 1 Processor -- start 2024-04-06T13:51:44.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.646+0000 7fdba5317700 1 -- start start 2024-04-06T13:51:44.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.646+0000 7fdba5317700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdba0143470 0x7fdba015caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.646+0000 7fdba5317700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdba015db10 0x7fdba00cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.646+0000 7fdba5317700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdba0160b60 0x7fdba00ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.646+0000 7fdba5317700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba00c8390 con 0x7fdba0143470 2024-04-06T13:51:44.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.646+0000 7fdba5317700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba00cdad0 con 0x7fdba015db10 2024-04-06T13:51:44.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.646+0000 7fdba5317700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba00cb000 con 0x7fdba0160b60 2024-04-06T13:51:44.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.646+0000 7fdb9effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdba0143470 0x7fdba015caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.646+0000 7fdb9effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdba0143470 0x7fdba015caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44424/0 (socket says 172.21.15.189:44424) 2024-04-06T13:51:44.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.646+0000 7fdb9effd700 1 -- 172.21.15.189:0/267811312 learned_addr learned my addr 172.21.15.189:0/267811312 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:44.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.647+0000 7fdb9f7fe700 1 --2- 172.21.15.189:0/267811312 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdba0160b60 0x7fdba00ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.647+0000 7fdb9e7fc700 1 --2- 172.21.15.189:0/267811312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdba015db10 0x7fdba00cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.647+0000 7fdb9effd700 1 -- 172.21.15.189:0/267811312 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdba0160b60 msgr2=0x7fdba00ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.647+0000 7fdb9effd700 1 --2- 172.21.15.189:0/267811312 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdba0160b60 0x7fdba00ccde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.647+0000 7fdb9effd700 1 -- 172.21.15.189:0/267811312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdba015db10 msgr2=0x7fdba00cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.647+0000 7fdb9effd700 1 --2- 172.21.15.189:0/267811312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdba015db10 0x7fdba00cc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.647+0000 7fdb9effd700 1 -- 172.21.15.189:0/267811312 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb98009a70 con 0x7fdba0143470 2024-04-06T13:51:44.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.647+0000 7fdb9effd700 1 --2- 172.21.15.189:0/267811312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdba0143470 0x7fdba015caf0 secure :-1 s=READY pgs=3246 cs=0 l=1 rev1=1 crypto rx=0x7fdb9400c650 tx=0x7fdb9400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:44.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.647+0000 7fdb877fe700 1 -- 172.21.15.189:0/267811312 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb94005d70 con 0x7fdba0143470 2024-04-06T13:51:44.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.647+0000 7fdb877fe700 1 -- 172.21.15.189:0/267811312 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb94007070 con 0x7fdba0143470 2024-04-06T13:51:44.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.647+0000 7fdba5317700 1 -- 172.21.15.189:0/267811312 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba00c8600 con 0x7fdba0143470 2024-04-06T13:51:44.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.647+0000 7fdb877fe700 1 -- 172.21.15.189:0/267811312 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb94007420 con 0x7fdba0143470 2024-04-06T13:51:44.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.647+0000 7fdba5317700 1 -- 172.21.15.189:0/267811312 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba015d030 con 0x7fdba0143470 2024-04-06T13:51:44.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.648+0000 7fdba5317700 1 -- 172.21.15.189:0/267811312 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdba015e860 con 0x7fdba0143470 2024-04-06T13:51:44.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.648+0000 7fdb877fe700 1 -- 172.21.15.189:0/267811312 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fdb94007580 con 0x7fdba0143470 2024-04-06T13:51:44.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.649+0000 7fdb877fe700 1 --2- 172.21.15.189:0/267811312 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdb8c06adb0 0x7fdb8c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.649+0000 7fdb877fe700 1 -- 172.21.15.189:0/267811312 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fdb94088af0 con 0x7fdba0143470 2024-04-06T13:51:44.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.649+0000 7fdb9e7fc700 1 --2- 172.21.15.189:0/267811312 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdb8c06adb0 0x7fdb8c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.649+0000 7fdb877fe700 1 --2- 172.21.15.189:0/267811312 >> [v2:172.21.15.192:6802/47715,v1:172.21.15.192:6804/47715] conn(0x7fdb8c0716f0 0x7fdb8c073b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.649+0000 7fdb877fe700 1 -- 172.21.15.189:0/267811312 --> [v2:172.21.15.192:6802/47715,v1:172.21.15.192:6804/47715] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb8c074220 con 0x7fdb8c0716f0 2024-04-06T13:51:44.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.649+0000 7fdb877fe700 1 -- 172.21.15.189:0/267811312 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=290) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb9408d2a0 con 0x7fdba0143470 2024-04-06T13:51:44.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.649+0000 7fdb9e7fc700 1 --2- 172.21.15.189:0/267811312 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdb8c06adb0 0x7fdb8c06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdb90007fd0 tx=0x7fdb90009040 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:44.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.649+0000 7fdb9f7fe700 1 --2- 172.21.15.189:0/267811312 >> [v2:172.21.15.192:6802/47715,v1:172.21.15.192:6804/47715] conn(0x7fdb8c0716f0 0x7fdb8c073b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.654 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.650+0000 7fdb9f7fe700 1 --2- 172.21.15.189:0/267811312 >> [v2:172.21.15.192:6802/47715,v1:172.21.15.192:6804/47715] conn(0x7fdb8c0716f0 0x7fdb8c073b50 crc :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:44.655 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.651+0000 7fdb877fe700 1 -- 172.21.15.189:0/267811312 <== osd.7 v2:172.21.15.192:6802/47715 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fdb8c074220 con 0x7fdb8c0716f0 2024-04-06T13:51:44.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.665+0000 7fdba5317700 1 -- 172.21.15.189:0/267811312 --> [v2:172.21.15.192:6802/47715,v1:172.21.15.192:6804/47715] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fdba015e580 con 0x7fdb8c0716f0 2024-04-06T13:51:44.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.665+0000 7fdb877fe700 1 -- 172.21.15.189:0/267811312 <== osd.7 v2:172.21.15.192:6802/47715 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fdb98003d00 con 0x7fdb8c0716f0 2024-04-06T13:51:44.665 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-06T13:51:44.666 INFO:teuthology.orchestra.run.smithi189.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-06T13:51:44.666 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-06T13:51:44.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.666+0000 7fdba5317700 1 -- 172.21.15.189:0/267811312 >> [v2:172.21.15.192:6802/47715,v1:172.21.15.192:6804/47715] conn(0x7fdb8c0716f0 msgr2=0x7fdb8c073b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.666+0000 7fdba5317700 1 --2- 172.21.15.189:0/267811312 >> [v2:172.21.15.192:6802/47715,v1:172.21.15.192:6804/47715] conn(0x7fdb8c0716f0 0x7fdb8c073b50 crc :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.666+0000 7fdba5317700 1 -- 172.21.15.189:0/267811312 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdb8c06adb0 msgr2=0x7fdb8c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.666+0000 7fdba5317700 1 --2- 172.21.15.189:0/267811312 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fdb8c06adb0 0x7fdb8c06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdb90007fd0 tx=0x7fdb90009040 comp rx=0 tx=0).stop 2024-04-06T13:51:44.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.666+0000 7fdba5317700 1 -- 172.21.15.189:0/267811312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdba0143470 msgr2=0x7fdba015caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.666+0000 7fdba5317700 1 --2- 172.21.15.189:0/267811312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdba0143470 0x7fdba015caf0 secure :-1 s=READY pgs=3246 cs=0 l=1 rev1=1 crypto rx=0x7fdb9400c650 tx=0x7fdb9400ca10 comp rx=0 tx=0).stop 2024-04-06T13:51:44.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.666+0000 7fdb9f7fe700 1 -- 172.21.15.189:0/267811312 reap_dead start 2024-04-06T13:51:44.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.666+0000 7fdba5317700 1 -- 172.21.15.189:0/267811312 shutdown_connections 2024-04-06T13:51:44.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.666+0000 7fdba5317700 1 -- 172.21.15.189:0/267811312 >> 172.21.15.189:0/267811312 conn(0x7fdba006a730 msgr2=0x7fdba00b0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:44.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.666+0000 7fdba5317700 1 -- 172.21.15.189:0/267811312 shutdown_connections 2024-04-06T13:51:44.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.666+0000 7fdba5317700 1 -- 172.21.15.189:0/267811312 wait complete. 2024-04-06T13:51:44.677 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-06T13:51:44.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.801+0000 7fc486d76700 1 Processor -- start 2024-04-06T13:51:44.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.801+0000 7fc486d76700 1 -- start start 2024-04-06T13:51:44.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.801+0000 7fc486d76700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc480146dd0 0x7fc48006d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.801+0000 7fc486d76700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc48006c0a0 con 0x7fc4800cedf0 2024-04-06T13:51:44.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.801+0000 7fc486d76700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc48006c380 con 0x7fc4800d17a0 2024-04-06T13:51:44.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.801+0000 7fc486d76700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc48006beb0 con 0x7fc480146dd0 2024-04-06T13:51:44.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.802+0000 7fc485313700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc4800cedf0 0x7fc4801441f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:32946/0 (socket says 172.21.15.189:32946) 2024-04-06T13:51:44.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.802+0000 7fc485313700 1 -- 172.21.15.189:0/3914004328 learned_addr learned my addr 172.21.15.189:0/3914004328 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:51:44.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.802+0000 7fc47ffff700 1 --2- 172.21.15.189:0/3914004328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc480146dd0 0x7fc48006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.802+0000 7fc47f7fe700 1 -- 172.21.15.189:0/3914004328 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062703770 0 0) 0x7fc48006c0a0 con 0x7fc4800cedf0 2024-04-06T13:51:44.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.802+0000 7fc47f7fe700 1 -- 172.21.15.189:0/3914004328 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4700035b0 con 0x7fc4800cedf0 2024-04-06T13:51:44.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.802+0000 7fc47f7fe700 1 -- 172.21.15.189:0/3914004328 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085008458 0 0) 0x7fc48006c380 con 0x7fc4800d17a0 2024-04-06T13:51:44.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.802+0000 7fc47f7fe700 1 -- 172.21.15.189:0/3914004328 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc48006c0a0 con 0x7fc4800d17a0 2024-04-06T13:51:44.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.802+0000 7fc47f7fe700 1 -- 172.21.15.189:0/3914004328 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 608746992 0 0) 0x7fc4700035b0 con 0x7fc4800cedf0 2024-04-06T13:51:44.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.802+0000 7fc47ffff700 1 -- 172.21.15.189:0/3914004328 >> v1:172.21.15.192:6789/0 conn(0x7fc4800d17a0 legacy=0x7fc48006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.802+0000 7fc47ffff700 1 -- 172.21.15.189:0/3914004328 >> v1:172.21.15.189:6789/0 conn(0x7fc4800cedf0 legacy=0x7fc4801441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.802+0000 7fc47ffff700 1 -- 172.21.15.189:0/3914004328 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48006c660 con 0x7fc480146dd0 2024-04-06T13:51:44.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.802+0000 7fc47ffff700 1 --2- 172.21.15.189:0/3914004328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc480146dd0 0x7fc48006d140 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc46c009870 tx=0x7fc46c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ce0de67ead0e7ee server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:44.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.802+0000 7fc47f7fe700 1 -- 172.21.15.189:0/3914004328 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4780019f0 con 0x7fc480146dd0 2024-04-06T13:51:44.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.802+0000 7fc47f7fe700 1 -- 172.21.15.189:0/3914004328 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46c004030 con 0x7fc480146dd0 2024-04-06T13:51:44.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.802+0000 7fc47f7fe700 1 -- 172.21.15.189:0/3914004328 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c0043a0 con 0x7fc480146dd0 2024-04-06T13:51:44.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.803+0000 7fc486d76700 1 -- 172.21.15.189:0/3914004328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc480146dd0 msgr2=0x7fc48006d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.803+0000 7fc486d76700 1 --2- 172.21.15.189:0/3914004328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc480146dd0 0x7fc48006d140 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc46c009870 tx=0x7fc46c009b80 comp rx=0 tx=0).stop 2024-04-06T13:51:44.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.803+0000 7fc486d76700 1 -- 172.21.15.189:0/3914004328 shutdown_connections 2024-04-06T13:51:44.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.803+0000 7fc486d76700 1 --2- 172.21.15.189:0/3914004328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc480146dd0 0x7fc48006d140 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.803+0000 7fc486d76700 1 -- 172.21.15.189:0/3914004328 >> 172.21.15.189:0/3914004328 conn(0x7fc4800b2db0 msgr2=0x7fc4800b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:44.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.803+0000 7fc486d76700 1 -- 172.21.15.189:0/3914004328 shutdown_connections 2024-04-06T13:51:44.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.803+0000 7fc486d76700 1 -- 172.21.15.189:0/3914004328 wait complete. 2024-04-06T13:51:44.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.804+0000 7fc486d76700 1 Processor -- start 2024-04-06T13:51:44.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.804+0000 7fc486d76700 1 -- start start 2024-04-06T13:51:44.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.805+0000 7fc486d76700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4800cedf0 0x7fc4800c4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.805+0000 7fc486d76700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4800d17a0 0x7fc4800affe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.805+0000 7fc486d76700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc480146dd0 0x7fc480161010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.805+0000 7fc486d76700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4800ae9d0 con 0x7fc4800cedf0 2024-04-06T13:51:44.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.805+0000 7fc486d76700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4800addc0 con 0x7fc480146dd0 2024-04-06T13:51:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.805+0000 7fc486d76700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4800ae0a0 con 0x7fc4800d17a0 2024-04-06T13:51:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.805+0000 7fc484b12700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4800cedf0 0x7fc4800c4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.805+0000 7fc484b12700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4800cedf0 0x7fc4800c4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44438/0 (socket says 172.21.15.189:44438) 2024-04-06T13:51:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.805+0000 7fc484b12700 1 -- 172.21.15.189:0/1760901951 learned_addr learned my addr 172.21.15.189:0/1760901951 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.805+0000 7fc47ffff700 1 --2- 172.21.15.189:0/1760901951 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4800d17a0 0x7fc4800affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.805+0000 7fc485313700 1 --2- 172.21.15.189:0/1760901951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc480146dd0 0x7fc480161010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.806+0000 7fc484b12700 1 -- 172.21.15.189:0/1760901951 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4800d17a0 msgr2=0x7fc4800affe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.806+0000 7fc484b12700 1 --2- 172.21.15.189:0/1760901951 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4800d17a0 0x7fc4800affe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.806+0000 7fc484b12700 1 -- 172.21.15.189:0/1760901951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc480146dd0 msgr2=0x7fc480161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:44.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.806+0000 7fc484b12700 1 --2- 172.21.15.189:0/1760901951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc480146dd0 0x7fc480161010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:44.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.806+0000 7fc484b12700 1 -- 172.21.15.189:0/1760901951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46c0094a0 con 0x7fc4800cedf0 2024-04-06T13:51:44.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.806+0000 7fc484b12700 1 --2- 172.21.15.189:0/1760901951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4800cedf0 0x7fc4800c4210 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7fc47400c500 tx=0x7fc47400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:44.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.806+0000 7fc47d7fa700 1 -- 172.21.15.189:0/1760901951 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4740052d0 con 0x7fc4800cedf0 2024-04-06T13:51:44.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.806+0000 7fc47d7fa700 1 -- 172.21.15.189:0/1760901951 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc474005450 con 0x7fc4800cedf0 2024-04-06T13:51:44.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.806+0000 7fc47d7fa700 1 -- 172.21.15.189:0/1760901951 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4740038c0 con 0x7fc4800cedf0 2024-04-06T13:51:44.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.806+0000 7fc486d76700 1 -- 172.21.15.189:0/1760901951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4800ae380 con 0x7fc4800cedf0 2024-04-06T13:51:44.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.806+0000 7fc486d76700 1 -- 172.21.15.189:0/1760901951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4800ada50 con 0x7fc4800cedf0 2024-04-06T13:51:44.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.807+0000 7fc486d76700 1 -- 172.21.15.189:0/1760901951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc48015b2d0 con 0x7fc4800cedf0 2024-04-06T13:51:44.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.807+0000 7fc47d7fa700 1 -- 172.21.15.189:0/1760901951 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fc474003a20 con 0x7fc4800cedf0 2024-04-06T13:51:44.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.808+0000 7fc47d7fa700 1 --2- 172.21.15.189:0/1760901951 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc46406adb0 0x7fc46406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:44.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.808+0000 7fc47ffff700 1 --2- 172.21.15.189:0/1760901951 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc46406adb0 0x7fc46406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:44.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.808+0000 7fc47d7fa700 1 -- 172.21.15.189:0/1760901951 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fc474088b60 con 0x7fc4800cedf0 2024-04-06T13:51:44.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.808+0000 7fc47ffff700 1 --2- 172.21.15.189:0/1760901951 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc46406adb0 0x7fc46406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc46c0097c0 tx=0x7fc46c008030 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:44.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:44.812+0000 7fc47d7fa700 1 -- 172.21.15.189:0/1760901951 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc474053a30 con 0x7fc4800cedf0 2024-04-06T13:51:45.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.007+0000 7fc486d76700 1 -- 172.21.15.189:0/1760901951 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4800d1b80 con 0x7fc46406adb0 2024-04-06T13:51:45.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.007+0000 7fc47d7fa700 1 -- 172.21.15.189:0/1760901951 <== mgr.10415 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc46c0042e0 con 0x7fc46406adb0 2024-04-06T13:51:45.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.009+0000 7fc486d76700 1 -- 172.21.15.189:0/1760901951 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc46406adb0 msgr2=0x7fc46406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:45.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.009+0000 7fc486d76700 1 --2- 172.21.15.189:0/1760901951 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc46406adb0 0x7fc46406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc46c0097c0 tx=0x7fc46c008030 comp rx=0 tx=0).stop 2024-04-06T13:51:45.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.009+0000 7fc486d76700 1 -- 172.21.15.189:0/1760901951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4800cedf0 msgr2=0x7fc4800c4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:45.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.009+0000 7fc486d76700 1 --2- 172.21.15.189:0/1760901951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4800cedf0 0x7fc4800c4210 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7fc47400c500 tx=0x7fc47400c8c0 comp rx=0 tx=0).stop 2024-04-06T13:51:45.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.010+0000 7fc486d76700 1 -- 172.21.15.189:0/1760901951 shutdown_connections 2024-04-06T13:51:45.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.010+0000 7fc486d76700 1 --2- 172.21.15.189:0/1760901951 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fc46406adb0 0x7fc46406d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:45.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.010+0000 7fc486d76700 1 --2- 172.21.15.189:0/1760901951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc480146dd0 0x7fc480161010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:45.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.010+0000 7fc486d76700 1 --2- 172.21.15.189:0/1760901951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4800cedf0 0x7fc4800c4210 unknown :-1 s=CLOSED pgs=3248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:45.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.010+0000 7fc486d76700 1 --2- 172.21.15.189:0/1760901951 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc4800d17a0 0x7fc4800affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:45.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.010+0000 7fc486d76700 1 -- 172.21.15.189:0/1760901951 >> 172.21.15.189:0/1760901951 conn(0x7fc4800b2db0 msgr2=0x7fc4800b3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:45.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.010+0000 7fc486d76700 1 -- 172.21.15.189:0/1760901951 shutdown_connections 2024-04-06T13:51:45.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.010+0000 7fc486d76700 1 -- 172.21.15.189:0/1760901951 wait complete. 2024-04-06T13:51:45.012 INFO:teuthology.orchestra.run.smithi189.stderr:instructed osd(s) 7 to deep-scrub 2024-04-06T13:51:45.022 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:51:45.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.142+0000 7f3af83a2700 1 Processor -- start 2024-04-06T13:51:45.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.142+0000 7f3af83a2700 1 -- start start 2024-04-06T13:51:45.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.143+0000 7f3af83a2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3af00d11b0 0x7f3af00d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:45.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.143+0000 7f3af83a2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3af00cb870 0x7f3af015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:45.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.143+0000 7f3af83a2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3af00cfa00 0x7f3af00cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:45.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.143+0000 7f3af83a2700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af00ccdc0 con 0x7f3af00d11b0 2024-04-06T13:51:45.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.143+0000 7f3af83a2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af00d0b50 con 0x7f3af00cb870 2024-04-06T13:51:45.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.143+0000 7f3af83a2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af00d1ad0 con 0x7f3af00cfa00 2024-04-06T13:51:45.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.143+0000 7f3af613e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3af00d11b0 0x7f3af00d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:45.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.143+0000 7f3af613e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3af00d11b0 0x7f3af00d1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44440/0 (socket says 172.21.15.189:44440) 2024-04-06T13:51:45.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.143+0000 7f3af613e700 1 -- 172.21.15.189:0/3522586776 learned_addr learned my addr 172.21.15.189:0/3522586776 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:45.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.144+0000 7f3af593d700 1 --2- 172.21.15.189:0/3522586776 >> v2:172.21.15.192:3300/0 conn(0x7f3af00cb870 0x7f3af015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:45.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.144+0000 7f3af613e700 1 -- 172.21.15.189:0/3522586776 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3af00cfa00 msgr2=0x7f3af00cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:45.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.144+0000 7f3af693f700 1 --2- 172.21.15.189:0/3522586776 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3af00cfa00 0x7f3af00cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:45.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.144+0000 7f3af613e700 1 --2- 172.21.15.189:0/3522586776 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3af00cfa00 0x7f3af00cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:45.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.144+0000 7f3af613e700 1 -- 172.21.15.189:0/3522586776 >> v2:172.21.15.192:3300/0 conn(0x7f3af00cb870 msgr2=0x7f3af015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:51:45.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.144+0000 7f3af613e700 1 --2- 172.21.15.189:0/3522586776 >> v2:172.21.15.192:3300/0 conn(0x7f3af00cb870 0x7f3af015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:45.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.144+0000 7f3af613e700 1 -- 172.21.15.189:0/3522586776 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af00cbd80 con 0x7f3af00d11b0 2024-04-06T13:51:45.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.144+0000 7f3af593d700 1 --2- 172.21.15.189:0/3522586776 >> v2:172.21.15.192:3300/0 conn(0x7f3af00cb870 0x7f3af015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:51:45.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.144+0000 7f3af613e700 1 --2- 172.21.15.189:0/3522586776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3af00d11b0 0x7f3af00d1590 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 crypto rx=0x7f3af004fb40 tx=0x7f3aec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f26f4db9090d435 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:45.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.144+0000 7f3af513c700 1 -- 172.21.15.189:0/3522586776 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec00b870 con 0x7f3af00d11b0 2024-04-06T13:51:45.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.144+0000 7f3af513c700 1 -- 172.21.15.189:0/3522586776 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aec00b9d0 con 0x7f3af00d11b0 2024-04-06T13:51:45.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.144+0000 7f3af513c700 1 -- 172.21.15.189:0/3522586776 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec00bd60 con 0x7f3af00d11b0 2024-04-06T13:51:45.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.145+0000 7f3af83a2700 1 -- 172.21.15.189:0/3522586776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3af00d11b0 msgr2=0x7f3af00d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:45.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.145+0000 7f3af83a2700 1 --2- 172.21.15.189:0/3522586776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3af00d11b0 0x7f3af00d1590 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 crypto rx=0x7f3af004fb40 tx=0x7f3aec009bd0 comp rx=0 tx=0).stop 2024-04-06T13:51:45.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.145+0000 7f3af83a2700 1 -- 172.21.15.189:0/3522586776 shutdown_connections 2024-04-06T13:51:45.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.145+0000 7f3af83a2700 1 --2- 172.21.15.189:0/3522586776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3af00d11b0 0x7f3af00d1590 unknown :-1 s=CLOSED pgs=3249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:45.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.145+0000 7f3af83a2700 1 --2- 172.21.15.189:0/3522586776 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3af00cfa00 0x7f3af00cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:45.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.145+0000 7f3af83a2700 1 --2- 172.21.15.189:0/3522586776 >> v2:172.21.15.192:3300/0 conn(0x7f3af00cb870 0x7f3af015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:45.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.145+0000 7f3af83a2700 1 -- 172.21.15.189:0/3522586776 >> 172.21.15.189:0/3522586776 conn(0x7f3af00b2db0 msgr2=0x7f3af00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:45.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.145+0000 7f3af83a2700 1 -- 172.21.15.189:0/3522586776 shutdown_connections 2024-04-06T13:51:45.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.145+0000 7f3af83a2700 1 -- 172.21.15.189:0/3522586776 wait complete. 2024-04-06T13:51:45.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.146+0000 7f3af83a2700 1 Processor -- start 2024-04-06T13:51:45.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.146+0000 7f3af83a2700 1 -- start start 2024-04-06T13:51:45.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3af83a2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3af00cb870 0x7f3af015d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:45.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3af83a2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3af00cfa00 0x7f3af01572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:45.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3af83a2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3af00d11b0 0x7f3af00c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:45.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3af83a2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af015ddc0 con 0x7f3af00d11b0 2024-04-06T13:51:45.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3af83a2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af0156280 con 0x7f3af00cfa00 2024-04-06T13:51:45.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3af83a2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af015e410 con 0x7f3af00cb870 2024-04-06T13:51:45.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3af693f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3af00d11b0 0x7f3af00c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:45.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3af693f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3af00d11b0 0x7f3af00c9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:44448/0 (socket says 172.21.15.189:44448) 2024-04-06T13:51:45.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3af693f700 1 -- 172.21.15.189:0/3232743527 learned_addr learned my addr 172.21.15.189:0/3232743527 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:51:45.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3af613e700 1 --2- 172.21.15.189:0/3232743527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3af00cb870 0x7f3af015d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:45.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3af593d700 1 --2- 172.21.15.189:0/3232743527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3af00cfa00 0x7f3af01572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:45.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3af693f700 1 -- 172.21.15.189:0/3232743527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3af00cb870 msgr2=0x7f3af015d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:45.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3af693f700 1 --2- 172.21.15.189:0/3232743527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3af00cb870 0x7f3af015d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:45.154 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3af693f700 1 -- 172.21.15.189:0/3232743527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3af00cfa00 msgr2=0x7f3af01572c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:45.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3af693f700 1 --2- 172.21.15.189:0/3232743527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3af00cfa00 0x7f3af01572c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:45.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3af693f700 1 -- 172.21.15.189:0/3232743527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aec009550 con 0x7f3af00d11b0 2024-04-06T13:51:45.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3af693f700 1 --2- 172.21.15.189:0/3232743527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3af00d11b0 0x7f3af00c9b80 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 crypto rx=0x7f3ae800cc30 tx=0x7f3ae800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:45.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3ae6ffd700 1 -- 172.21.15.189:0/3232743527 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae800aa50 con 0x7f3af00d11b0 2024-04-06T13:51:45.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3ae6ffd700 1 -- 172.21.15.189:0/3232743527 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae800abb0 con 0x7f3af00d11b0 2024-04-06T13:51:45.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3ae6ffd700 1 -- 172.21.15.189:0/3232743527 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae8003a20 con 0x7f3af00d11b0 2024-04-06T13:51:45.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.147+0000 7f3af83a2700 1 -- 172.21.15.189:0/3232743527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af015e6f0 con 0x7f3af00d11b0 2024-04-06T13:51:45.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.148+0000 7f3af83a2700 1 -- 172.21.15.189:0/3232743527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af015ecb0 con 0x7f3af00d11b0 2024-04-06T13:51:45.155 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.148+0000 7f3af83a2700 1 -- 172.21.15.189:0/3232743527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af00cfeb0 con 0x7f3af00d11b0 2024-04-06T13:51:45.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.149+0000 7f3ae6ffd700 1 -- 172.21.15.189:0/3232743527 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f3ae800f3f0 con 0x7f3af00d11b0 2024-04-06T13:51:45.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.149+0000 7f3ae6ffd700 1 --2- 172.21.15.189:0/3232743527 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3ad406adb0 0x7f3ad406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:51:45.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.149+0000 7f3ae6ffd700 1 -- 172.21.15.189:0/3232743527 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f3ae8089750 con 0x7f3af00d11b0 2024-04-06T13:51:45.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.149+0000 7f3af613e700 1 --2- 172.21.15.189:0/3232743527 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3ad406adb0 0x7f3ad406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:51:45.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.150+0000 7f3af613e700 1 --2- 172.21.15.189:0/3232743527 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3ad406adb0 0x7f3ad406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3aec009840 tx=0x7f3aec008030 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:51:45.156 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.154+0000 7f3ae6ffd700 1 -- 172.21.15.189:0/3232743527 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3ae8054630 con 0x7f3af00d11b0 2024-04-06T13:51:45.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.344+0000 7f3af83a2700 1 -- 172.21.15.189:0/3232743527 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3af00cf4e0 con 0x7f3ad406adb0 2024-04-06T13:51:45.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.346+0000 7f3ae6ffd700 1 -- 172.21.15.189:0/3232743527 <== mgr.10415 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50979 (secure 0 0 0) 0x7f3aec00bc40 con 0x7f3ad406adb0 2024-04-06T13:51:45.347 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:51:45.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.349+0000 7f3af83a2700 1 -- 172.21.15.189:0/3232743527 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3ad406adb0 msgr2=0x7f3ad406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:45.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.349+0000 7f3af83a2700 1 --2- 172.21.15.189:0/3232743527 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3ad406adb0 0x7f3ad406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3aec009840 tx=0x7f3aec008030 comp rx=0 tx=0).stop 2024-04-06T13:51:45.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.349+0000 7f3af83a2700 1 -- 172.21.15.189:0/3232743527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3af00d11b0 msgr2=0x7f3af00c9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:51:45.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.349+0000 7f3af83a2700 1 --2- 172.21.15.189:0/3232743527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3af00d11b0 0x7f3af00c9b80 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 crypto rx=0x7f3ae800cc30 tx=0x7f3ae800cf40 comp rx=0 tx=0).stop 2024-04-06T13:51:45.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.349+0000 7f3af83a2700 1 -- 172.21.15.189:0/3232743527 shutdown_connections 2024-04-06T13:51:45.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.349+0000 7f3af83a2700 1 --2- 172.21.15.189:0/3232743527 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f3ad406adb0 0x7f3ad406d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:45.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.349+0000 7f3af83a2700 1 --2- 172.21.15.189:0/3232743527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3af00d11b0 0x7f3af00c9b80 unknown :-1 s=CLOSED pgs=3250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:45.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.349+0000 7f3af83a2700 1 --2- 172.21.15.189:0/3232743527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3af00cb870 0x7f3af015d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:45.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.349+0000 7f3af83a2700 1 --2- 172.21.15.189:0/3232743527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3af00cfa00 0x7f3af01572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:51:45.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.349+0000 7f3af83a2700 1 -- 172.21.15.189:0/3232743527 >> 172.21.15.189:0/3232743527 conn(0x7f3af00b2db0 msgr2=0x7f3af006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:51:45.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.349+0000 7f3af83a2700 1 -- 172.21.15.189:0/3232743527 shutdown_connections 2024-04-06T13:51:45.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:51:45.349+0000 7f3af83a2700 1 -- 172.21.15.189:0/3232743527 wait complete. 2024-04-06T13:51:45.351 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:51:45.361 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-06T13:51:44.512558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":482184,"kb_used_data":29844,"kb_used_omap":12,"kb_used_meta":452211,"kb_avail":754492536,"statfs":{"total":773094113280,"available":772600356864,"internally_reserved":0,"allocated":30560256,"data_stored":16354233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":463064632},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-123,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":19,"num_write_kb":-1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002119"},"pg_stats":[{"pgid":"1.0","version":"288'425","reported_seq":1575,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-06T13:51:33.507521+0000","last_change":"2024-04-06T12:12:00.340167+0000","last_active":"2024-04-06T13:51:33.507521+0000","last_peered":"2024-04-06T13:51:33.507521+0000","last_clean":"2024-04-06T13:51:33.507521+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:51:33.507521+0000","last_undegraded":"2024-04-06T13:51:33.507521+0000","last_fullsized":"2024-04-06T13:51:33.507521+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T12:11:58.746427+0000","last_clean_scrub_stamp":"2024-04-06T12:11:58.746427+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:27:21.956906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576028,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57428,"kb_used_data":4168,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314412,"statfs":{"total":96636764160,"available":96577957888,"internally_reserved":0,"allocated":4268032,"data_stored":2487692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55500000000000005,"15min":0.58799999999999997},"min":{"1min":0.34499999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.82499999999999996},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53200000000000003,"15min":0.57399999999999995},"min":{"1min":0.47499999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.76200000000000001,"5min":0.78500000000000003,"15min":0.82899999999999996},"last":0.60099999999999998}]},{"osd":1,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.53900000000000003,"15min":0.57099999999999995},"min":{"1min":0.39700000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.748},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.54400000000000004,"15min":0.58399999999999996},"min":{"1min":0.41999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.85299999999999998},"last":0.51900000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.52100000000000002,"15min":0.56999999999999995},"min":{"1min":0.36399999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.83899999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56100000000000005,"15min":0.59199999999999997},"min":{"1min":0.45000000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76800000000000002,"5min":0.76900000000000002,"15min":0.79700000000000004},"last":0.67900000000000005}]},{"osd":3,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54600000000000004,"15min":0.58999999999999997},"min":{"1min":0.39700000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.75,"5min":0.75600000000000001,"15min":0.81100000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.622,"5min":0.57299999999999995,"15min":0.59599999999999997},"min":{"1min":0.497,"5min":0.39000000000000001,"15min":0.34999999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.84099999999999997},"last":0.64800000000000002}]},{"osd":4,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45400000000000001,"15min":0.46300000000000002},"min":{"1min":0.32400000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.60999999999999999,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.54100000000000004,"15min":0.54300000000000004},"min":{"1min":0.432,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70999999999999996,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.66400000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.5,"15min":0.498},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64200000000000002,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55300000000000005,"15min":0.57399999999999995},"min":{"1min":0.375,"5min":0.35599999999999998,"15min":0.34799999999999998},"max":{"1min":0.66800000000000004,"5min":0.77800000000000002,"15min":0.82099999999999995},"last":0.68899999999999995}]},{"osd":6,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.53800000000000003,"15min":0.55400000000000005},"min":{"1min":0.39200000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67400000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.57999999999999996},"min":{"1min":0.433,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67500000000000004,"5min":0.86299999999999999,"15min":0.88200000000000001},"last":0.62}]}]},{"osd":6,"up_from":13,"seq":55834576027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60944,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310896,"statfs":{"total":96636764160,"available":96574357504,"internally_reserved":0,"allocated":3670016,"data_stored":1896305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.50900000000000001,"15min":0.54800000000000004},"min":{"1min":0.373,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.74199999999999999,"5min":1.244,"15min":1.244},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.53200000000000003,"15min":0.56100000000000005},"min":{"1min":0.42999999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.68799999999999994,"5min":1.304,"15min":1.304},"last":0.499}]},{"osd":1,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56100000000000005,"15min":0.60899999999999999},"min":{"1min":0.41799999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.85799999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.60799999999999998,"15min":0.63400000000000001},"min":{"1min":0.49399999999999999,"5min":0.40000000000000002,"15min":0.378},"max":{"1min":0.84299999999999997,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.67300000000000004}]},{"osd":2,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54400000000000004,"15min":0.58099999999999996},"min":{"1min":0.34999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69199999999999995,"5min":0.76700000000000002,"15min":0.82899999999999996},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.55200000000000005,"15min":0.57899999999999996},"min":{"1min":0.47399999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80800000000000005,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.55100000000000005}]},{"osd":3,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.58999999999999997},"min":{"1min":0.41699999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.78400000000000003,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.60599999999999998,"15min":0.63500000000000001},"min":{"1min":0.501,"5min":0.39900000000000002,"15min":0.36099999999999999},"max":{"1min":0.79600000000000004,"5min":0.80200000000000005,"15min":0.90200000000000002},"last":0.68799999999999994}]},{"osd":4,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55300000000000005,"15min":0.53900000000000003},"min":{"1min":0.39900000000000002,"5min":0.308,"15min":0.308},"max":{"1min":0.72699999999999998,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":0.63},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.34200000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.73899999999999999,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.70399999999999996}]},{"osd":5,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.501,"15min":0.50700000000000001},"min":{"1min":0.34499999999999997,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.62,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54200000000000004,"15min":0.53600000000000003},"min":{"1min":0.32900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.77900000000000003,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":0.73299999999999998}]},{"osd":7,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56299999999999994,"15min":0.55200000000000005},"min":{"1min":0.35499999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76700000000000002,"5min":1.603,"15min":1.603},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58399999999999996,"15min":0.57399999999999995},"min":{"1min":0.40500000000000003,"5min":0.33200000000000002,"15min":0.32800000000000001},"max":{"1min":0.80200000000000005,"5min":2.75,"15min":2.75},"last":0.76900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834576026,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65036,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306804,"statfs":{"total":96636764160,"available":96570167296,"internally_reserved":0,"allocated":3670016,"data_stored":1896305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.371,"15min":0.38400000000000001},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65600000000000003,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40300000000000002,"15min":0.41999999999999998},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.67200000000000004,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":0.245}]},{"osd":2,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.38300000000000001,"15min":0.41099999999999998},"min":{"1min":0.33000000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.42899999999999999,"15min":0.45700000000000002},"min":{"1min":0.34100000000000003,"5min":0.25,"15min":0.25},"max":{"1min":0.69999999999999996,"5min":2.0169999999999999,"15min":2.0169999999999999},"last":0.28299999999999997}]},{"osd":3,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47999999999999998},"min":{"1min":0.31,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61699999999999999,"5min":4.2699999999999996,"15min":4.2699999999999996},"last":0.373},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.41899999999999998,"15min":0.434},"min":{"1min":0.29299999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66600000000000004,"5min":2.153,"15min":2.153},"last":0.34499999999999997}]},{"osd":4,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.58399999999999996,"15min":0.624},"min":{"1min":0.495,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.89900000000000002,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.59699999999999998,"15min":0.63},"min":{"1min":0.52700000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80500000000000005,"5min":0.877,"15min":0.995},"last":0.51600000000000001}]},{"osd":5,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.60699999999999998,"15min":0.63200000000000001},"min":{"1min":0.50900000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84499999999999997,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.64700000000000002,"15min":0.66100000000000003},"min":{"1min":0.51100000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.88600000000000001,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.53100000000000003}]},{"osd":6,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.61299999999999999,"15min":0.63800000000000001},"min":{"1min":0.52000000000000002,"5min":0.42599999999999999,"15min":0.40799999999999997},"max":{"1min":0.872,"5min":0.89200000000000002,"15min":0.94399999999999995},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.65500000000000003,"15min":0.67000000000000004},"min":{"1min":0.47199999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.91700000000000004,"5min":4.2889999999999997,"15min":4.2889999999999997},"last":0.54500000000000004}]},{"osd":7,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67700000000000005,"15min":0.68899999999999995},"min":{"1min":0.52200000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.86699999999999999,"5min":4.0730000000000004,"15min":4.0730000000000004},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.627,"15min":0.65200000000000002},"min":{"1min":0.46400000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.94199999999999995,"5min":2.194,"15min":2.194},"last":0.55600000000000005}]}]},{"osd":0,"up_from":13,"seq":55834576026,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61520,"kb_used_data":4164,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310320,"statfs":{"total":96636764160,"available":96573767680,"internally_reserved":0,"allocated":4263936,"data_stored":2486673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.36099999999999999,"15min":0.38100000000000001},"min":{"1min":0.27200000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55800000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.375,"15min":0.40699999999999997},"min":{"1min":0.28499999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.70799999999999996},"last":0.499}]},{"osd":2,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.38700000000000001,"15min":0.41599999999999998},"min":{"1min":0.316,"5min":0.26100000000000001,"15min":0.23899999999999999},"max":{"1min":0.50600000000000001,"5min":0.55900000000000005,"15min":0.70099999999999996},"last":0.433},{"interface":"front","average":{"1min":0.442,"5min":0.41499999999999998,"15min":0.435},"min":{"1min":0.33500000000000002,"5min":0.28399999999999997,"15min":0.27100000000000002},"max":{"1min":0.627,"5min":0.627,"15min":0.70499999999999996},"last":0.56499999999999995}]},{"osd":3,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.38200000000000001,"15min":0.41299999999999998},"min":{"1min":0.28599999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54300000000000004,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.435,"15min":0.46600000000000003},"min":{"1min":0.38500000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.59699999999999998,"5min":0.64100000000000001,"15min":0.75700000000000001},"last":0.52800000000000002}]},{"osd":4,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58999999999999997,"15min":0.626},"min":{"1min":0.45400000000000001,"5min":0.38500000000000001,"15min":0.37},"max":{"1min":0.82399999999999995,"5min":0.93100000000000005,"15min":0.93400000000000005},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59399999999999997,"15min":0.63200000000000001},"min":{"1min":0.42999999999999999,"5min":0.372,"15min":0.36399999999999999},"max":{"1min":0.83599999999999997,"5min":0.999,"15min":0.999},"last":0.54700000000000004}]},{"osd":5,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.57899999999999996,"15min":0.622},"min":{"1min":0.41699999999999998,"5min":0.373,"15min":0.33000000000000002},"max":{"1min":0.80600000000000005,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.623,"15min":0.65400000000000003},"min":{"1min":0.45800000000000002,"5min":0.41399999999999998,"15min":0.378},"max":{"1min":0.872,"5min":0.89500000000000002,"15min":0.96799999999999997},"last":0.79600000000000004}]},{"osd":6,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.623,"15min":0.65900000000000003},"min":{"1min":0.42699999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85999999999999999,"5min":0.94399999999999995,"15min":1.0269999999999999},"last":0.745},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.61599999999999999,"15min":0.65500000000000003},"min":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.39000000000000001},"max":{"1min":0.85699999999999998,"5min":0.93600000000000005,"15min":0.96599999999999997},"last":0.78400000000000003}]},{"osd":7,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.63500000000000001,"15min":0.67600000000000005},"min":{"1min":0.47699999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81000000000000005,"5min":0.91200000000000003,"15min":1.01},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.63200000000000001,"15min":0.66000000000000003},"min":{"1min":0.44600000000000001,"5min":0.42199999999999999,"15min":0.36099999999999999},"max":{"1min":0.88700000000000001,"5min":0.95799999999999996,"15min":0.97699999999999998},"last":0.626}]}]},{"osd":2,"up_from":13,"seq":55834576028,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65040,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306800,"statfs":{"total":96636764160,"available":96570163200,"internally_reserved":0,"allocated":3670016,"data_stored":1896305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.374},"min":{"1min":0.24199999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.54600000000000004,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.39600000000000002,"15min":0.41399999999999998},"min":{"1min":0.31900000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.66400000000000003,"5min":0.71499999999999997,"15min":1.044},"last":0.377}]},{"osd":1,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.36399999999999999,"15min":0.39300000000000002},"min":{"1min":0.25600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.753},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.441,"15min":0.44600000000000001},"min":{"1min":0.34999999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.60299999999999998,"5min":3.3999999999999999,"15min":3.3999999999999999},"last":0.26100000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.40899999999999997,"15min":0.44600000000000001},"min":{"1min":0.30199999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.70299999999999996},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.41099999999999998,"15min":0.436},"min":{"1min":0.29299999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.69599999999999995,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.34200000000000003}]},{"osd":4,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59199999999999997,"15min":0.626},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.88800000000000001,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.60899999999999999,"15min":0.64600000000000002},"min":{"1min":0.42299999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86199999999999999,"5min":1.022,"15min":1.022},"last":0.60199999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60299999999999998,"15min":0.64800000000000002},"min":{"1min":0.40400000000000003,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80100000000000005,"5min":0.98599999999999999,"15min":1.0149999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.627,"5min":0.61799999999999999,"15min":0.65200000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84699999999999998,"5min":1.028,"15min":1.028},"last":0.54600000000000004}]},{"osd":6,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.63400000000000001,"15min":0.65800000000000003},"min":{"1min":0.51100000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92400000000000004,"5min":0.97199999999999998,"15min":1.091},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.64000000000000001,"15min":0.67900000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91200000000000003,"5min":0.95699999999999996,"15min":0.96099999999999997},"last":0.626}]},{"osd":7,"last update":"Sat Apr 6 13:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.64900000000000002,"15min":0.67500000000000004},"min":{"1min":0.435,"5min":0.42299999999999999,"15min":0.41599999999999998},"max":{"1min":0.93899999999999995,"5min":0.94499999999999995,"15min":0.95399999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.65600000000000003,"15min":0.68999999999999995},"min":{"1min":0.44700000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.94899999999999995,"5min":1.01,"15min":1.01},"last":0.67000000000000004}]}]},{"osd":3,"up_from":13,"seq":55834576027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52756,"kb_used_data":3588,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319084,"statfs":{"total":96636764160,"available":96582742016,"internally_reserved":0,"allocated":3674112,"data_stored":1897324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.36299999999999999,"15min":0.376},"min":{"1min":0.255,"5min":0.223,"15min":0.223},"max":{"1min":0.53400000000000003,"5min":0.55200000000000005,"15min":0.72799999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.36799999999999999,"15min":0.39700000000000002},"min":{"1min":0.32000000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.63800000000000001},"last":0.38800000000000001}]},{"osd":1,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.36699999999999999,"15min":0.39500000000000002},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.40600000000000003,"15min":0.42799999999999999},"min":{"1min":0.29299999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.60299999999999998,"5min":0.63,"15min":0.75800000000000001},"last":0.48099999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.434,"15min":0.46899999999999997},"min":{"1min":0.39200000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.67000000000000004,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.495},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.45700000000000002,"15min":0.48399999999999999},"min":{"1min":0.38700000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65200000000000002,"5min":0.67800000000000005,"15min":0.79300000000000004},"last":0.55300000000000005}]},{"osd":4,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.59099999999999997,"15min":0.61899999999999999},"min":{"1min":0.42299999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":1.018},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.59999999999999998,"15min":0.63300000000000001},"min":{"1min":0.44900000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.0169999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.78200000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.64200000000000002,"15min":0.66700000000000004},"min":{"1min":0.47099999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.75},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.65200000000000002,"15min":0.67900000000000005},"min":{"1min":0.501,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.94399999999999995,"5min":1.036,"15min":1.036},"last":0.67600000000000005}]},{"osd":6,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.626,"15min":0.64900000000000002},"min":{"1min":0.47499999999999998,"5min":0.31,"15min":0.31},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.63,"15min":0.66300000000000003},"min":{"1min":0.433,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.79600000000000004}]},{"osd":7,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.628,"15min":0.65900000000000003},"min":{"1min":0.46000000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90500000000000003,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.64500000000000002,"15min":0.68100000000000005},"min":{"1min":0.53000000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.73899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52756,"kb_used_data":3588,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319084,"statfs":{"total":96636764160,"available":96582742016,"internally_reserved":0,"allocated":3674112,"data_stored":1897324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54800000000000004,"15min":0.56999999999999995},"min":{"1min":0.38400000000000001,"5min":0.30499999999999999,"15min":0.26300000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.876},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.47899999999999998,"5min":0.36599999999999999,"15min":0.34499999999999997},"max":{"1min":0.77400000000000002,"5min":0.81100000000000005,"15min":1.202},"last":0.46899999999999997}]},{"osd":1,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.60899999999999999,"15min":0.628},"min":{"1min":0.46200000000000002,"5min":0.379,"15min":0.35099999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":1.256},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.60299999999999998,"15min":0.61799999999999999},"min":{"1min":0.49199999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.89700000000000002},"last":0.67600000000000005}]},{"osd":2,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56200000000000006,"15min":0.58399999999999996},"min":{"1min":0.438,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80100000000000005,"5min":0.81799999999999995,"15min":0.84599999999999997},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59199999999999997,"15min":0.60999999999999999},"min":{"1min":0.46700000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70799999999999996,"5min":0.97899999999999998,"15min":1.294},"last":0.54800000000000004}]},{"osd":3,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56599999999999995,"15min":0.58399999999999996},"min":{"1min":0.498,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.94299999999999995},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.59399999999999997,"15min":0.61599999999999999},"min":{"1min":0.47099999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.90200000000000002},"last":0.624}]},{"osd":5,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.50900000000000001,"15min":0.52000000000000002},"min":{"1min":0.35599999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.64100000000000001,"5min":0.92800000000000005,"15min":5.0119999999999996},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.54200000000000004},"min":{"1min":0.35199999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68600000000000005,"5min":0.95199999999999996,"15min":4.931},"last":0.48899999999999999}]},{"osd":6,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52900000000000003,"15min":0.53200000000000003},"min":{"1min":0.35299999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.77800000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.40300000000000002,"5min":0.35799999999999998,"15min":0.34699999999999998},"max":{"1min":0.71199999999999997,"5min":1.0609999999999999,"15min":1.3580000000000001},"last":0.505}]},{"osd":7,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.54900000000000004,"15min":0.57999999999999996},"min":{"1min":0.378,"5min":0.35899999999999999,"15min":0.34000000000000002},"max":{"1min":0.66600000000000004,"5min":0.91000000000000003,"15min":4.9889999999999999},"last":0.438},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58599999999999997,"15min":0.59899999999999998},"min":{"1min":0.40899999999999997,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72699999999999998,"5min":0.86899999999999999,"15min":0.88100000000000001},"last":0.56799999999999995}]}]},{"osd":5,"up_from":13,"seq":55834576027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":66704,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305136,"statfs":{"total":96636764160,"available":96568459264,"internally_reserved":0,"allocated":3670016,"data_stored":1896305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54300000000000004,"15min":0.57499999999999996},"min":{"1min":0.438,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.82199999999999995},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54200000000000004,"15min":0.57699999999999996},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.28100000000000003},"max":{"1min":0.73999999999999999,"5min":0.76600000000000001,"15min":0.84699999999999998},"last":0.52000000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54400000000000004,"15min":0.57699999999999996},"min":{"1min":0.437,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.81299999999999994},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54300000000000004,"15min":0.57499999999999996},"min":{"1min":0.42799999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69199999999999995,"5min":0.752,"15min":0.871},"last":0.65100000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.55600000000000005,"15min":0.58699999999999997},"min":{"1min":0.46800000000000003,"5min":0.311,"15min":0.30599999999999999},"max":{"1min":0.72099999999999997,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.56200000000000006,"15min":0.58799999999999997},"min":{"1min":0.44600000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.745,"5min":0.745,"15min":0.83399999999999996},"last":0.622}]},{"osd":3,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.55600000000000005,"15min":0.58699999999999997},"min":{"1min":0.40300000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77800000000000002,"5min":0.79800000000000004,"15min":0.81100000000000005},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58399999999999996,"15min":0.621},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.371},"max":{"1min":0.76100000000000001,"5min":0.81399999999999995,"15min":0.88200000000000001},"last":0.57799999999999996}]},{"osd":4,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51000000000000001,"15min":0.50700000000000001},"min":{"1min":0.32300000000000001,"5min":0.29899999999999999,"15min":0.27400000000000002},"max":{"1min":0.70999999999999996,"5min":0.75900000000000001,"15min":0.85699999999999998},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57199999999999995,"15min":0.57999999999999996},"min":{"1min":0.34799999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.73999999999999999,"5min":0.82999999999999996,"15min":0.88800000000000001},"last":0.499}]},{"osd":6,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35799999999999998,"5min":0.313,"15min":0.28899999999999998},"max":{"1min":0.75700000000000001,"5min":0.89900000000000002,"15min":0.94999999999999996},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52900000000000003,"15min":0.53200000000000003},"min":{"1min":0.372,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.76100000000000001,"5min":0.84299999999999997,"15min":0.878},"last":0.48099999999999998}]},{"osd":7,"last update":"Sat Apr 6 13:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.39100000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.76800000000000002,"5min":0.85599999999999998,"15min":0.90800000000000003},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.63800000000000001,"15min":0.64800000000000002},"min":{"1min":0.47399999999999998,"5min":0.41699999999999998,"15min":0.38300000000000001},"max":{"1min":0.79000000000000004,"5min":0.87,"15min":0.92300000000000004},"last":0.63700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:51:45.367 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-06T12:11:58.746427+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=12, tm_min=11, tm_sec=58, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=13, tm_min=51, tm_sec=39, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T13:51:45.367 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-06T13:52:03.717 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:52:03.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:52:05.368 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-06T13:52:05.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.501+0000 7f84964d0700 1 Processor -- start 2024-04-06T13:52:05.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.501+0000 7f84964d0700 1 -- start start 2024-04-06T13:52:05.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.502+0000 7f84964d0700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f849015d7b0 0x7f84900b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:05.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.502+0000 7f84964d0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f849015e500 0x7f849006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:05.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.502+0000 7f84964d0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84900b3990 con 0x7f849006a730 2024-04-06T13:52:05.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.502+0000 7f84964d0700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f849006b0f0 con 0x7f849015d7b0 2024-04-06T13:52:05.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.502+0000 7f84964d0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84900b3060 con 0x7f849015e500 2024-04-06T13:52:05.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.502+0000 7f848f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f849006a730 0x7f849015ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57614/0 (socket says 172.21.15.189:57614) 2024-04-06T13:52:05.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.502+0000 7f848f7fe700 1 -- 172.21.15.189:0/2884759548 learned_addr learned my addr 172.21.15.189:0/2884759548 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:52:05.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.502+0000 7f8494a6d700 1 --2- 172.21.15.189:0/2884759548 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f849015e500 0x7f849006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:05.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.502+0000 7f848ffff700 1 --2- 172.21.15.189:0/2884759548 >> v2:172.21.15.192:3300/0 conn(0x7f849015d7b0 0x7f84900b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:05.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.502+0000 7f848effd700 1 -- 172.21.15.189:0/2884759548 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369745016 0 0) 0x7f84900b3990 con 0x7f849006a730 2024-04-06T13:52:05.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.502+0000 7f848effd700 1 -- 172.21.15.189:0/2884759548 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84780035b0 con 0x7f849006a730 2024-04-06T13:52:05.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.502+0000 7f848effd700 1 -- 172.21.15.189:0/2884759548 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 6539154 0 0) 0x7f84780035b0 con 0x7f849006a730 2024-04-06T13:52:05.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.502+0000 7f848effd700 1 -- 172.21.15.189:0/2884759548 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f849015e500 msgr2=0x7f849006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:52:05.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.502+0000 7f848effd700 1 --2- 172.21.15.189:0/2884759548 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f849015e500 0x7f849006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:05.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.502+0000 7f848effd700 1 -- 172.21.15.189:0/2884759548 >> v2:172.21.15.192:3300/0 conn(0x7f849015d7b0 msgr2=0x7f84900b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:52:05.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.502+0000 7f848effd700 1 --2- 172.21.15.189:0/2884759548 >> v2:172.21.15.192:3300/0 conn(0x7f849015d7b0 0x7f84900b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:05.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.502+0000 7f848effd700 1 -- 172.21.15.189:0/2884759548 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84900b3340 con 0x7f849006a730 2024-04-06T13:52:05.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.502+0000 7f848effd700 1 -- 172.21.15.189:0/2884759548 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f8480001670 con 0x7f849006a730 2024-04-06T13:52:05.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.503+0000 7f848effd700 1 -- 172.21.15.189:0/2884759548 >> v1:172.21.15.189:6789/0 conn(0x7f849006a730 legacy=0x7f849015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:52:05.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.503+0000 7f848effd700 1 --2- 172.21.15.189:0/2884759548 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f849015e500 0x7f8478003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:05.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.503+0000 7f848effd700 1 --2- 172.21.15.189:0/2884759548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8478003f80 0x7f8478004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:05.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.503+0000 7f848effd700 1 --2- 172.21.15.189:0/2884759548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8478004d90 0x7f8478005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:05.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.503+0000 7f848effd700 1 -- 172.21.15.189:0/2884759548 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849006b0f0 con 0x7f8478004d90 2024-04-06T13:52:05.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.503+0000 7f848effd700 1 -- 172.21.15.189:0/2884759548 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84900b3060 con 0x7f8478003f80 2024-04-06T13:52:05.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.503+0000 7f848effd700 1 -- 172.21.15.189:0/2884759548 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847800ba60 con 0x7f849015e500 2024-04-06T13:52:05.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.503+0000 7f8494a6d700 1 --2- 172.21.15.189:0/2884759548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8478004d90 0x7f8478005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:05.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.503+0000 7f848ffff700 1 --2- 172.21.15.189:0/2884759548 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f849015e500 0x7f8478003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:05.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.503+0000 7f848f7fe700 1 --2- 172.21.15.189:0/2884759548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8478003f80 0x7f8478004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:05.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.503+0000 7f8494a6d700 1 -- 172.21.15.189:0/2884759548 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f849015e500 msgr2=0x7f8478003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:52:05.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.503+0000 7f8494a6d700 1 --2- 172.21.15.189:0/2884759548 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f849015e500 0x7f8478003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:05.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.503+0000 7f8494a6d700 1 -- 172.21.15.189:0/2884759548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8478003f80 msgr2=0x7f8478004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:52:05.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.503+0000 7f8494a6d700 1 --2- 172.21.15.189:0/2884759548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8478003f80 0x7f8478004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:05.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.503+0000 7f8494a6d700 1 -- 172.21.15.189:0/2884759548 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84780035b0 con 0x7f8478004d90 2024-04-06T13:52:05.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.503+0000 7f8494a6d700 1 --2- 172.21.15.189:0/2884759548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8478004d90 0x7f8478005210 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 crypto rx=0x7f848800e170 tx=0x7f8478005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40e43d2e0d498275 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:52:05.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.503+0000 7f848effd700 1 -- 172.21.15.189:0/2884759548 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848800ec90 con 0x7f8478004d90 2024-04-06T13:52:05.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.503+0000 7f848effd700 1 -- 172.21.15.189:0/2884759548 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848800f040 con 0x7f8478004d90 2024-04-06T13:52:05.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.503+0000 7f848effd700 1 -- 172.21.15.189:0/2884759548 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8488014480 con 0x7f8478004d90 2024-04-06T13:52:05.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.504+0000 7f84964d0700 1 -- 172.21.15.189:0/2884759548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8478004d90 msgr2=0x7f8478005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:52:05.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.504+0000 7f84964d0700 1 --2- 172.21.15.189:0/2884759548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8478004d90 0x7f8478005210 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 crypto rx=0x7f848800e170 tx=0x7f8478005880 comp rx=0 tx=0).stop 2024-04-06T13:52:05.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.504+0000 7f8494a6d700 1 -- 172.21.15.189:0/2884759548 reap_dead start 2024-04-06T13:52:05.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.504+0000 7f84964d0700 1 -- 172.21.15.189:0/2884759548 shutdown_connections 2024-04-06T13:52:05.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.504+0000 7f84964d0700 1 -- 172.21.15.189:0/2884759548 >> 172.21.15.189:0/2884759548 conn(0x7f84900d00b0 msgr2=0x7f84900d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:52:05.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.504+0000 7f84964d0700 1 -- 172.21.15.189:0/2884759548 shutdown_connections 2024-04-06T13:52:05.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.504+0000 7f84964d0700 1 -- 172.21.15.189:0/2884759548 wait complete. 2024-04-06T13:52:05.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.505+0000 7f84964d0700 1 Processor -- start 2024-04-06T13:52:05.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.505+0000 7f84964d0700 1 -- start start 2024-04-06T13:52:05.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.506+0000 7f84964d0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8478004d90 0x7f84900af800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:05.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.506+0000 7f84964d0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84900b04d0 0x7f8490153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:05.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.506+0000 7f84964d0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f84900b0d50 0x7f84900ac950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:05.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.506+0000 7f84964d0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84900b1910 con 0x7f8478004d90 2024-04-06T13:52:05.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.506+0000 7f84964d0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84900b1630 con 0x7f84900b04d0 2024-04-06T13:52:05.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.506+0000 7f84964d0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84900aed20 con 0x7f84900b0d50 2024-04-06T13:52:05.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.506+0000 7f848ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8478004d90 0x7f84900af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:05.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.506+0000 7f848ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8478004d90 0x7f84900af800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54876/0 (socket says 172.21.15.189:54876) 2024-04-06T13:52:05.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.506+0000 7f848ffff700 1 -- 172.21.15.189:0/355806255 learned_addr learned my addr 172.21.15.189:0/355806255 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:52:05.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.506+0000 7f8494a6d700 1 --2- 172.21.15.189:0/355806255 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f84900b0d50 0x7f84900ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:05.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.506+0000 7f848ffff700 1 -- 172.21.15.189:0/355806255 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f84900b0d50 msgr2=0x7f84900ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:52:05.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.506+0000 7f848ffff700 1 --2- 172.21.15.189:0/355806255 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f84900b0d50 0x7f84900ac950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:05.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.506+0000 7f848ffff700 1 -- 172.21.15.189:0/355806255 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84900b04d0 msgr2=0x7f8490153c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:52:05.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.506+0000 7f848ffff700 1 --2- 172.21.15.189:0/355806255 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84900b04d0 0x7f8490153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:05.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.506+0000 7f848ffff700 1 -- 172.21.15.189:0/355806255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84900af000 con 0x7f8478004d90 2024-04-06T13:52:05.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.507+0000 7f848ffff700 1 --2- 172.21.15.189:0/355806255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8478004d90 0x7f84900af800 secure :-1 s=READY pgs=3253 cs=0 l=1 rev1=1 crypto rx=0x7f8484011aa0 tx=0x7f8484011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:52:05.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.507+0000 7f848cff9700 1 -- 172.21.15.189:0/355806255 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848400f9e0 con 0x7f8478004d90 2024-04-06T13:52:05.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.507+0000 7f848cff9700 1 -- 172.21.15.189:0/355806255 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848400fb40 con 0x7f8478004d90 2024-04-06T13:52:05.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.507+0000 7f84964d0700 1 -- 172.21.15.189:0/355806255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84900af2e0 con 0x7f8478004d90 2024-04-06T13:52:05.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.507+0000 7f848cff9700 1 -- 172.21.15.189:0/355806255 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8484014440 con 0x7f8478004d90 2024-04-06T13:52:05.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.507+0000 7f84964d0700 1 -- 172.21.15.189:0/355806255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84900b1200 con 0x7f8478004d90 2024-04-06T13:52:05.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.508+0000 7f84964d0700 1 -- 172.21.15.189:0/355806255 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84900aeb80 con 0x7f8478004d90 2024-04-06T13:52:05.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.508+0000 7f848cff9700 1 -- 172.21.15.189:0/355806255 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f84840145a0 con 0x7f8478004d90 2024-04-06T13:52:05.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.509+0000 7f848cff9700 1 --2- 172.21.15.189:0/355806255 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f847c06adb0 0x7f847c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:05.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.509+0000 7f848cff9700 1 -- 172.21.15.189:0/355806255 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f8484089dd0 con 0x7f8478004d90 2024-04-06T13:52:05.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.509+0000 7f848f7fe700 1 --2- 172.21.15.189:0/355806255 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f847c06adb0 0x7f847c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:05.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.509+0000 7f848f7fe700 1 --2- 172.21.15.189:0/355806255 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f847c06adb0 0x7f847c06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8480001cc0 tx=0x7f8480001b70 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:52:05.523 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.513+0000 7f848cff9700 1 -- 172.21.15.189:0/355806255 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f84900aeb80 con 0x7f8478004d90 2024-04-06T13:52:05.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.704+0000 7f84964d0700 1 -- 172.21.15.189:0/355806255 --> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f849015eb40 con 0x7f847c06adb0 2024-04-06T13:52:05.707 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.706+0000 7f848cff9700 1 -- 172.21.15.189:0/355806255 <== mgr.10415 v2:172.21.15.189:6832/111704 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51030 (secure 0 0 0) 0x7f8480007da0 con 0x7f847c06adb0 2024-04-06T13:52:05.708 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:52:05.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.709+0000 7f84964d0700 1 -- 172.21.15.189:0/355806255 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f847c06adb0 msgr2=0x7f847c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:52:05.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.709+0000 7f84964d0700 1 --2- 172.21.15.189:0/355806255 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f847c06adb0 0x7f847c06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8480001cc0 tx=0x7f8480001b70 comp rx=0 tx=0).stop 2024-04-06T13:52:05.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.709+0000 7f84964d0700 1 -- 172.21.15.189:0/355806255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8478004d90 msgr2=0x7f84900af800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:52:05.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.709+0000 7f84964d0700 1 --2- 172.21.15.189:0/355806255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8478004d90 0x7f84900af800 secure :-1 s=READY pgs=3253 cs=0 l=1 rev1=1 crypto rx=0x7f8484011aa0 tx=0x7f8484011e60 comp rx=0 tx=0).stop 2024-04-06T13:52:05.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.709+0000 7f84964d0700 1 -- 172.21.15.189:0/355806255 shutdown_connections 2024-04-06T13:52:05.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.709+0000 7f84964d0700 1 --2- 172.21.15.189:0/355806255 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f847c06adb0 0x7f847c06d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:05.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.709+0000 7f84964d0700 1 --2- 172.21.15.189:0/355806255 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f84900b0d50 0x7f84900ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:05.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.709+0000 7f84964d0700 1 --2- 172.21.15.189:0/355806255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8478004d90 0x7f84900af800 unknown :-1 s=CLOSED pgs=3253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:05.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.709+0000 7f84964d0700 1 --2- 172.21.15.189:0/355806255 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84900b04d0 0x7f8490153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:05.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.709+0000 7f84964d0700 1 -- 172.21.15.189:0/355806255 >> 172.21.15.189:0/355806255 conn(0x7f84900d00b0 msgr2=0x7f8490159460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:52:05.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.709+0000 7f84964d0700 1 -- 172.21.15.189:0/355806255 shutdown_connections 2024-04-06T13:52:05.711 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.709+0000 7f84964d0700 1 -- 172.21.15.189:0/355806255 wait complete. 2024-04-06T13:52:05.712 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2024-04-06T13:52:05.722 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-06T13:52:04.516864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":426,"ondisk_log_size":426,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":482176,"kb_used_data":29844,"kb_used_omap":12,"kb_used_meta":452211,"kb_avail":754492544,"statfs":{"total":773094113280,"available":772600365056,"internally_reserved":0,"allocated":30560256,"data_stored":16354233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":463064632},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002554"},"pg_stats":[{"pgid":"1.0","version":"290'426","reported_seq":1588,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-04-06T13:51:48.947855+0000","last_change":"2024-04-06T13:51:48.947963+0000","last_active":"2024-04-06T13:51:48.947855+0000","last_peered":"2024-04-06T13:51:48.947855+0000","last_clean":"2024-04-06T13:51:48.947855+0000","last_became_active":"2024-04-06T12:12:00.339260+0000","last_became_peered":"2024-04-06T12:12:00.339260+0000","last_unstale":"2024-04-06T13:51:48.947855+0000","last_undegraded":"2024-04-06T13:51:48.947855+0000","last_fullsized":"2024-04-06T13:51:48.947855+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'426","last_scrub_stamp":"2024-04-06T13:51:48.947829+0000","last_deep_scrub":"290'426","last_deep_scrub_stamp":"2024-04-06T13:51:48.947829+0000","last_clean_scrub_stamp":"2024-04-06T13:51:48.947829+0000","objects_scrubbed":2,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:00:49.682254+0000","scrub_duration":0.0059255769999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":426,"ondisk_log_size":426,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576032,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57424,"kb_used_data":4168,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314416,"statfs":{"total":96636764160,"available":96577961984,"internally_reserved":0,"allocated":4268032,"data_stored":2487692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55500000000000005,"15min":0.58799999999999997},"min":{"1min":0.34499999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.82499999999999996},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53200000000000003,"15min":0.57399999999999995},"min":{"1min":0.47499999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.76200000000000001,"5min":0.78500000000000003,"15min":0.82899999999999996},"last":0.57399999999999995}]},{"osd":1,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.53900000000000003,"15min":0.57099999999999995},"min":{"1min":0.39700000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.54400000000000004,"15min":0.58399999999999996},"min":{"1min":0.41999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.85299999999999998},"last":0.52000000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.52100000000000002,"15min":0.56999999999999995},"min":{"1min":0.36399999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.83899999999999997},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56100000000000005,"15min":0.59199999999999997},"min":{"1min":0.45000000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76800000000000002,"5min":0.76900000000000002,"15min":0.79700000000000004},"last":0.61799999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54600000000000004,"15min":0.58999999999999997},"min":{"1min":0.39700000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.75,"5min":0.75600000000000001,"15min":0.81100000000000005},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.57299999999999995,"15min":0.59599999999999997},"min":{"1min":0.497,"5min":0.39000000000000001,"15min":0.34999999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.84099999999999997},"last":0.65500000000000003}]},{"osd":4,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45400000000000001,"15min":0.46300000000000002},"min":{"1min":0.32400000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.60999999999999999,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.54100000000000004,"15min":0.54300000000000004},"min":{"1min":0.432,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70999999999999996,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.59299999999999997}]},{"osd":5,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.5,"15min":0.498},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64200000000000002,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55300000000000005,"15min":0.57399999999999995},"min":{"1min":0.375,"5min":0.35599999999999998,"15min":0.34799999999999998},"max":{"1min":0.66800000000000004,"5min":0.77800000000000002,"15min":0.82099999999999995},"last":0.46999999999999997}]},{"osd":6,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.53800000000000003,"15min":0.55400000000000005},"min":{"1min":0.39200000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67400000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.57999999999999996},"min":{"1min":0.433,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67500000000000004,"5min":0.86299999999999999,"15min":0.88200000000000001},"last":0.45300000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576031,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60944,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310896,"statfs":{"total":96636764160,"available":96574357504,"internally_reserved":0,"allocated":3670016,"data_stored":1896305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.50900000000000001,"15min":0.54800000000000004},"min":{"1min":0.373,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.74199999999999999,"5min":1.244,"15min":1.244},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.53200000000000003,"15min":0.56100000000000005},"min":{"1min":0.42999999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.68799999999999994,"5min":1.304,"15min":1.304},"last":0.52100000000000002}]},{"osd":1,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56100000000000005,"15min":0.60899999999999999},"min":{"1min":0.41799999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.85799999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.60799999999999998,"15min":0.63400000000000001},"min":{"1min":0.49399999999999999,"5min":0.40000000000000002,"15min":0.378},"max":{"1min":0.84299999999999997,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.66000000000000003}]},{"osd":2,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54400000000000004,"15min":0.58099999999999996},"min":{"1min":0.34999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69199999999999995,"5min":0.76700000000000002,"15min":0.82899999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.55200000000000005,"15min":0.57899999999999996},"min":{"1min":0.47399999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80800000000000005,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.627}]},{"osd":3,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.58999999999999997},"min":{"1min":0.41699999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.78400000000000003,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.60599999999999998,"15min":0.63500000000000001},"min":{"1min":0.501,"5min":0.39900000000000002,"15min":0.36099999999999999},"max":{"1min":0.79600000000000004,"5min":0.80200000000000005,"15min":0.90200000000000002},"last":0.53200000000000003}]},{"osd":4,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55300000000000005,"15min":0.53900000000000003},"min":{"1min":0.39900000000000002,"5min":0.308,"15min":0.308},"max":{"1min":0.72699999999999998,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":0.504},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.34200000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.73899999999999999,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.48499999999999999}]},{"osd":5,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.501,"15min":0.50700000000000001},"min":{"1min":0.34499999999999997,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.62,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54200000000000004,"15min":0.53600000000000003},"min":{"1min":0.32900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.77900000000000003,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":0.438}]},{"osd":7,"last update":"Sat Apr 6 13:51:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56299999999999994,"15min":0.55200000000000005},"min":{"1min":0.35499999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76700000000000002,"5min":1.603,"15min":1.603},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58399999999999996,"15min":0.57399999999999995},"min":{"1min":0.40500000000000003,"5min":0.33200000000000002,"15min":0.32800000000000001},"max":{"1min":0.80200000000000005,"5min":2.75,"15min":2.75},"last":0.58199999999999996}]}]},{"osd":1,"up_from":13,"seq":55834576030,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65036,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306804,"statfs":{"total":96636764160,"available":96570167296,"internally_reserved":0,"allocated":3670016,"data_stored":1896305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.371,"15min":0.38400000000000001},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65600000000000003,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40300000000000002,"15min":0.41999999999999998},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.67200000000000004,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":0.59499999999999997}]},{"osd":2,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.38300000000000001,"15min":0.41099999999999998},"min":{"1min":0.33000000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.42899999999999999,"15min":0.45700000000000002},"min":{"1min":0.34100000000000003,"5min":0.25,"15min":0.25},"max":{"1min":0.69999999999999996,"5min":2.0169999999999999,"15min":2.0169999999999999},"last":0.63800000000000001}]},{"osd":3,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47999999999999998},"min":{"1min":0.31,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61699999999999999,"5min":4.2699999999999996,"15min":4.2699999999999996},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.41899999999999998,"15min":0.434},"min":{"1min":0.29299999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66600000000000004,"5min":2.153,"15min":2.153},"last":0.65500000000000003}]},{"osd":4,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.58399999999999996,"15min":0.624},"min":{"1min":0.495,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.89900000000000002,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.59699999999999998,"15min":0.63},"min":{"1min":0.52700000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80500000000000005,"5min":0.877,"15min":0.995},"last":0.84999999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.60699999999999998,"15min":0.63200000000000001},"min":{"1min":0.50900000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84499999999999997,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.64700000000000002,"15min":0.66100000000000003},"min":{"1min":0.51100000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.88600000000000001,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.75}]},{"osd":6,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.61299999999999999,"15min":0.63800000000000001},"min":{"1min":0.52000000000000002,"5min":0.42599999999999999,"15min":0.40799999999999997},"max":{"1min":0.872,"5min":0.89200000000000002,"15min":0.94399999999999995},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.65500000000000003,"15min":0.67000000000000004},"min":{"1min":0.47199999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.91700000000000004,"5min":4.2889999999999997,"15min":4.2889999999999997},"last":0.874}]},{"osd":7,"last update":"Sat Apr 6 13:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67700000000000005,"15min":0.68899999999999995},"min":{"1min":0.52200000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.86699999999999999,"5min":4.0730000000000004,"15min":4.0730000000000004},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.627,"15min":0.65200000000000002},"min":{"1min":0.46400000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.94199999999999995,"5min":2.194,"15min":2.194},"last":0.76800000000000002}]}]},{"osd":0,"up_from":13,"seq":55834576030,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61516,"kb_used_data":4164,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310324,"statfs":{"total":96636764160,"available":96573771776,"internally_reserved":0,"allocated":4263936,"data_stored":2486673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.36099999999999999,"15min":0.38100000000000001},"min":{"1min":0.27200000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55800000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.375,"15min":0.40699999999999997},"min":{"1min":0.28499999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.70799999999999996},"last":0.46500000000000002}]},{"osd":2,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.38700000000000001,"15min":0.41599999999999998},"min":{"1min":0.316,"5min":0.26100000000000001,"15min":0.23899999999999999},"max":{"1min":0.50600000000000001,"5min":0.55900000000000005,"15min":0.70099999999999996},"last":0.377},{"interface":"front","average":{"1min":0.442,"5min":0.41499999999999998,"15min":0.435},"min":{"1min":0.33500000000000002,"5min":0.28399999999999997,"15min":0.27100000000000002},"max":{"1min":0.627,"5min":0.627,"15min":0.70499999999999996},"last":0.42899999999999999}]},{"osd":3,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.38200000000000001,"15min":0.41299999999999998},"min":{"1min":0.28599999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54300000000000004,"5min":0.63800000000000001,"15min":0.67400000000000004},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.435,"15min":0.46600000000000003},"min":{"1min":0.38500000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.59699999999999998,"5min":0.64100000000000001,"15min":0.75700000000000001},"last":0.40999999999999998}]},{"osd":4,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58999999999999997,"15min":0.626},"min":{"1min":0.45400000000000001,"5min":0.38500000000000001,"15min":0.37},"max":{"1min":0.82399999999999995,"5min":0.93100000000000005,"15min":0.93400000000000005},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59399999999999997,"15min":0.63200000000000001},"min":{"1min":0.42999999999999999,"5min":0.372,"15min":0.36399999999999999},"max":{"1min":0.83599999999999997,"5min":0.999,"15min":0.999},"last":0.59899999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.57899999999999996,"15min":0.622},"min":{"1min":0.41699999999999998,"5min":0.373,"15min":0.33000000000000002},"max":{"1min":0.80600000000000005,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.623,"15min":0.65400000000000003},"min":{"1min":0.45800000000000002,"5min":0.41399999999999998,"15min":0.378},"max":{"1min":0.872,"5min":0.89500000000000002,"15min":0.96799999999999997},"last":0.58399999999999996}]},{"osd":6,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.623,"15min":0.65900000000000003},"min":{"1min":0.42699999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85999999999999999,"5min":0.94399999999999995,"15min":1.0269999999999999},"last":0.63},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.61599999999999999,"15min":0.65500000000000003},"min":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.39000000000000001},"max":{"1min":0.85699999999999998,"5min":0.93600000000000005,"15min":0.96599999999999997},"last":0.71099999999999997}]},{"osd":7,"last update":"Sat Apr 6 13:51:07 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.63500000000000001,"15min":0.67600000000000005},"min":{"1min":0.47699999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81000000000000005,"5min":0.91200000000000003,"15min":1.01},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.63200000000000001,"15min":0.66000000000000003},"min":{"1min":0.44600000000000001,"5min":0.42199999999999999,"15min":0.36099999999999999},"max":{"1min":0.88700000000000001,"5min":0.95799999999999996,"15min":0.97699999999999998},"last":0.69499999999999995}]}]},{"osd":2,"up_from":13,"seq":55834576032,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65040,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306800,"statfs":{"total":96636764160,"available":96570163200,"internally_reserved":0,"allocated":3670016,"data_stored":1896305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.34200000000000003,"15min":0.372},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":0.56999999999999995,"5min":0.60899999999999999,"15min":0.65700000000000003},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38700000000000001,"15min":0.41099999999999998},"min":{"1min":0.24399999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.58899999999999997,"5min":0.71499999999999997,"15min":1.044},"last":0.34100000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.35999999999999999,"15min":0.39200000000000002},"min":{"1min":0.26000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.753},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.42599999999999999,"15min":0.441},"min":{"1min":0.26100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.53300000000000003,"5min":3.3999999999999999,"15min":3.3999999999999999},"last":0.39600000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.39500000000000002,"15min":0.442},"min":{"1min":0.27300000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.55200000000000005,"5min":0.68400000000000005,"15min":0.70299999999999996},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.39900000000000002,"15min":0.434},"min":{"1min":0.32000000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.61299999999999999,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.32000000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56899999999999995,"15min":0.61899999999999999},"min":{"1min":0.40400000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69599999999999995,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59499999999999997,"15min":0.64200000000000002},"min":{"1min":0.42199999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.022},"last":0.60699999999999998}]},{"osd":5,"last update":"Sat Apr 6 13:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58699999999999997,"15min":0.64400000000000002},"min":{"1min":0.38400000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":1.0149999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.64700000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.75700000000000001,"5min":1.028,"15min":1.028},"last":0.59099999999999997}]},{"osd":6,"last update":"Sat Apr 6 13:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.61199999999999999,"15min":0.65500000000000003},"min":{"1min":0.47299999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83999999999999997,"5min":0.92400000000000004,"15min":1.091},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.627,"15min":0.67600000000000005},"min":{"1min":0.39200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.55300000000000005}]},{"osd":7,"last update":"Sat Apr 6 13:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.627,"15min":0.66900000000000004},"min":{"1min":0.42799999999999999,"5min":0.42299999999999999,"15min":0.41599999999999998},"max":{"1min":0.87,"5min":0.94499999999999995,"15min":0.95399999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.63500000000000001,"15min":0.68500000000000005},"min":{"1min":0.42999999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.004,"5min":1.004,"15min":1.01},"last":0.61899999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576031,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52756,"kb_used_data":3588,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319084,"statfs":{"total":96636764160,"available":96582742016,"internally_reserved":0,"allocated":3674112,"data_stored":1897324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.36299999999999999,"15min":0.376},"min":{"1min":0.255,"5min":0.223,"15min":0.223},"max":{"1min":0.53400000000000003,"5min":0.55200000000000005,"15min":0.72799999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.36799999999999999,"15min":0.39700000000000002},"min":{"1min":0.32000000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.63800000000000001},"last":0.44400000000000001}]},{"osd":1,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.36699999999999999,"15min":0.39500000000000002},"min":{"1min":0.30199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.40600000000000003,"15min":0.42799999999999999},"min":{"1min":0.29299999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.60299999999999998,"5min":0.63,"15min":0.75800000000000001},"last":0.48099999999999998}]},{"osd":2,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.434,"15min":0.46899999999999997},"min":{"1min":0.39200000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.67000000000000004,"5min":0.69899999999999995,"15min":0.70299999999999996},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.45700000000000002,"15min":0.48399999999999999},"min":{"1min":0.38700000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65200000000000002,"5min":0.67800000000000005,"15min":0.79300000000000004},"last":0.42399999999999999}]},{"osd":4,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.59099999999999997,"15min":0.61899999999999999},"min":{"1min":0.42299999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":1.018},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.59999999999999998,"15min":0.63300000000000001},"min":{"1min":0.44900000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.0169999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.78700000000000003}]},{"osd":5,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.64200000000000002,"15min":0.66700000000000004},"min":{"1min":0.47099999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.65200000000000002,"15min":0.67900000000000005},"min":{"1min":0.501,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.94399999999999995,"5min":1.036,"15min":1.036},"last":0.76900000000000002}]},{"osd":6,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.626,"15min":0.64900000000000002},"min":{"1min":0.47499999999999998,"5min":0.31,"15min":0.31},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.63,"15min":0.66300000000000003},"min":{"1min":0.433,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.80600000000000005}]},{"osd":7,"last update":"Sat Apr 6 13:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.628,"15min":0.65900000000000003},"min":{"1min":0.46000000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90500000000000003,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.64500000000000002,"15min":0.68100000000000005},"min":{"1min":0.53000000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.73699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576031,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52756,"kb_used_data":3588,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319084,"statfs":{"total":96636764160,"available":96582742016,"internally_reserved":0,"allocated":3674112,"data_stored":1897324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54800000000000004,"15min":0.56999999999999995},"min":{"1min":0.38400000000000001,"5min":0.30499999999999999,"15min":0.26300000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.876},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.47899999999999998,"5min":0.36599999999999999,"15min":0.34499999999999997},"max":{"1min":0.77400000000000002,"5min":0.81100000000000005,"15min":1.202},"last":0.63700000000000001}]},{"osd":1,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.60899999999999999,"15min":0.628},"min":{"1min":0.46200000000000002,"5min":0.379,"15min":0.35099999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":1.256},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.60299999999999998,"15min":0.61799999999999999},"min":{"1min":0.49199999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.89700000000000002},"last":0.68899999999999995}]},{"osd":2,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56200000000000006,"15min":0.58399999999999996},"min":{"1min":0.438,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80100000000000005,"5min":0.81799999999999995,"15min":0.84599999999999997},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59199999999999997,"15min":0.60999999999999999},"min":{"1min":0.46700000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70799999999999996,"5min":0.97899999999999998,"15min":1.294},"last":0.70499999999999996}]},{"osd":3,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56599999999999995,"15min":0.58399999999999996},"min":{"1min":0.498,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.94299999999999995},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.59399999999999997,"15min":0.61599999999999999},"min":{"1min":0.47099999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.90200000000000002},"last":0.626}]},{"osd":5,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.50900000000000001,"15min":0.52000000000000002},"min":{"1min":0.35599999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.64100000000000001,"5min":0.92800000000000005,"15min":5.0119999999999996},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.54200000000000004},"min":{"1min":0.35199999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68600000000000005,"5min":0.95199999999999996,"15min":4.931},"last":0.502}]},{"osd":6,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52900000000000003,"15min":0.53200000000000003},"min":{"1min":0.35299999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.77800000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.40300000000000002,"5min":0.35799999999999998,"15min":0.34699999999999998},"max":{"1min":0.71199999999999997,"5min":1.0609999999999999,"15min":1.3580000000000001},"last":0.53500000000000003}]},{"osd":7,"last update":"Sat Apr 6 13:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.54900000000000004,"15min":0.57999999999999996},"min":{"1min":0.378,"5min":0.35899999999999999,"15min":0.34000000000000002},"max":{"1min":0.66600000000000004,"5min":0.91000000000000003,"15min":4.9889999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58599999999999997,"15min":0.59899999999999998},"min":{"1min":0.40899999999999997,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72699999999999998,"5min":0.86899999999999999,"15min":0.88100000000000001},"last":0.51800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576031,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":66704,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305136,"statfs":{"total":96636764160,"available":96568459264,"internally_reserved":0,"allocated":3670016,"data_stored":1896305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 13:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.54200000000000004,"15min":0.57499999999999996},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.82199999999999995},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54100000000000004,"15min":0.57499999999999996},"min":{"1min":0.33900000000000002,"5min":0.29899999999999999,"15min":0.28100000000000003},"max":{"1min":0.71899999999999997,"5min":0.76600000000000001,"15min":0.84699999999999998},"last":0.53600000000000003}]},{"osd":1,"last update":"Sat Apr 6 13:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53300000000000003,"15min":0.57499999999999996},"min":{"1min":0.41999999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67000000000000004,"5min":0.80800000000000005,"15min":0.81299999999999994},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.54400000000000004,"15min":0.57399999999999995},"min":{"1min":0.40000000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.70799999999999996,"5min":0.752,"15min":0.871},"last":0.55400000000000005}]},{"osd":2,"last update":"Sat Apr 6 13:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.38,"5min":0.311,"15min":0.30599999999999999},"max":{"1min":0.69399999999999995,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56499999999999995,"15min":0.58799999999999997},"min":{"1min":0.42899999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.75,"5min":0.75,"15min":0.83399999999999996},"last":0.52100000000000002}]},{"osd":3,"last update":"Sat Apr 6 13:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.55400000000000005,"15min":0.58699999999999997},"min":{"1min":0.35099999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.74399999999999999,"5min":0.79800000000000004,"15min":0.81100000000000005},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57999999999999996,"15min":0.61799999999999999},"min":{"1min":0.375,"5min":0.371,"15min":0.371},"max":{"1min":0.76400000000000001,"5min":0.80200000000000005,"15min":0.88200000000000001},"last":0.45200000000000001}]},{"osd":4,"last update":"Sat Apr 6 13:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.501,"15min":0.50700000000000001},"min":{"1min":0.35099999999999998,"5min":0.29899999999999999,"15min":0.27400000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.85699999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56299999999999994,"15min":0.58099999999999996},"min":{"1min":0.308,"5min":0.307,"15min":0.307},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.88800000000000001},"last":0.38900000000000001}]},{"osd":6,"last update":"Sat Apr 6 13:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.52400000000000002,"15min":0.52900000000000003},"min":{"1min":0.34699999999999998,"5min":0.313,"15min":0.28899999999999998},"max":{"1min":0.79100000000000004,"5min":0.89900000000000002,"15min":0.94999999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52500000000000002,"15min":0.53400000000000003},"min":{"1min":0.32800000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.81799999999999995,"5min":0.84299999999999997,"15min":0.878},"last":0.40600000000000003}]},{"osd":7,"last update":"Sat Apr 6 13:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.53800000000000003,"15min":0.55000000000000004},"min":{"1min":0.40899999999999997,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.72199999999999998,"5min":0.85599999999999998,"15min":0.90800000000000003},"last":0.434},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.63100000000000001,"15min":0.64800000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.92300000000000004},"last":0.64200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T13:52:05.724 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-06T13:52:05.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.845+0000 7fae9f72a700 1 Processor -- start 2024-04-06T13:52:05.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.846+0000 7fae9f72a700 1 -- start start 2024-04-06T13:52:05.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.846+0000 7fae9f72a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fae9815d7c0 0x7fae9806b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:05.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.847+0000 7fae9f72a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fae9806b6f0 0x7fae9806dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:05.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.847+0000 7fae9f72a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9806ceb0 con 0x7fae980cd3e0 2024-04-06T13:52:05.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.847+0000 7fae9f72a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae980cdfa0 con 0x7fae9806b6f0 2024-04-06T13:52:05.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.847+0000 7fae9f72a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9806d750 con 0x7fae9815d7c0 2024-04-06T13:52:05.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.847+0000 7fae9dcc7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fae9806b6f0 0x7fae9806dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:05.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.847+0000 7fae9d4c6700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fae980cd3e0 0x7fae980cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57630/0 (socket says 172.21.15.189:57630) 2024-04-06T13:52:05.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.847+0000 7fae9d4c6700 1 -- 172.21.15.189:0/830459682 learned_addr learned my addr 172.21.15.189:0/830459682 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:52:05.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.847+0000 7fae9ccc5700 1 --2- 172.21.15.189:0/830459682 >> v2:172.21.15.192:3300/0 conn(0x7fae9815d7c0 0x7fae9806b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:05.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.847+0000 7fae9dcc7700 1 -- 172.21.15.189:0/830459682 >> v2:172.21.15.192:3300/0 conn(0x7fae9815d7c0 msgr2=0x7fae9806b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:52:05.854 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.847+0000 7fae9dcc7700 1 --2- 172.21.15.189:0/830459682 >> v2:172.21.15.192:3300/0 conn(0x7fae9815d7c0 0x7fae9806b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:05.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.847+0000 7fae9dcc7700 1 -- 172.21.15.189:0/830459682 >> v1:172.21.15.189:6789/0 conn(0x7fae980cd3e0 legacy=0x7fae980cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:52:05.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.847+0000 7fae9dcc7700 1 -- 172.21.15.189:0/830459682 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9806bf00 con 0x7fae9806b6f0 2024-04-06T13:52:05.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.847+0000 7fae9ccc5700 1 --2- 172.21.15.189:0/830459682 >> v2:172.21.15.192:3300/0 conn(0x7fae9815d7c0 0x7fae9806b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T13:52:05.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.847+0000 7fae9dcc7700 1 --2- 172.21.15.189:0/830459682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae9806b6f0 0x7fae9806dbe0 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 crypto rx=0x7fae98073390 tx=0x7fae9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1946b13cfb21ce36 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:52:05.855 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.847+0000 7fae8ffff700 1 -- 172.21.15.189:0/830459682 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9000e070 con 0x7fae9806b6f0 2024-04-06T13:52:05.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.847+0000 7fae8ffff700 1 -- 172.21.15.189:0/830459682 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9000bbd0 con 0x7fae9806b6f0 2024-04-06T13:52:05.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.847+0000 7fae8ffff700 1 -- 172.21.15.189:0/830459682 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae90012430 con 0x7fae9806b6f0 2024-04-06T13:52:05.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.848+0000 7fae9f72a700 1 -- 172.21.15.189:0/830459682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae9806b6f0 msgr2=0x7fae9806dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:52:05.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.848+0000 7fae9f72a700 1 --2- 172.21.15.189:0/830459682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae9806b6f0 0x7fae9806dbe0 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 crypto rx=0x7fae98073390 tx=0x7fae9000b330 comp rx=0 tx=0).stop 2024-04-06T13:52:05.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.848+0000 7fae9f72a700 1 -- 172.21.15.189:0/830459682 shutdown_connections 2024-04-06T13:52:05.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.848+0000 7fae9f72a700 1 --2- 172.21.15.189:0/830459682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae9806b6f0 0x7fae9806dbe0 unknown :-1 s=CLOSED pgs=3255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:05.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.848+0000 7fae9f72a700 1 --2- 172.21.15.189:0/830459682 >> v2:172.21.15.192:3300/0 conn(0x7fae9815d7c0 0x7fae9806b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:05.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.848+0000 7fae9f72a700 1 -- 172.21.15.189:0/830459682 >> 172.21.15.189:0/830459682 conn(0x7fae980cc290 msgr2=0x7fae980cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:52:05.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.848+0000 7fae9f72a700 1 -- 172.21.15.189:0/830459682 shutdown_connections 2024-04-06T13:52:05.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.848+0000 7fae9f72a700 1 -- 172.21.15.189:0/830459682 wait complete. 2024-04-06T13:52:05.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.849+0000 7fae9f72a700 1 Processor -- start 2024-04-06T13:52:05.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.849+0000 7fae9f72a700 1 -- start start 2024-04-06T13:52:05.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.850+0000 7fae9f72a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae9806b6f0 0x7fae981606e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:05.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.850+0000 7fae9f72a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae980cd3e0 0x7fae980b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:05.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.850+0000 7fae9f72a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae9815d7c0 0x7fae980c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:05.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.850+0000 7fae9f72a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae980b2eb0 con 0x7fae980cd3e0 2024-04-06T13:52:05.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.850+0000 7fae9f72a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae980b22a0 con 0x7fae9815d7c0 2024-04-06T13:52:05.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.850+0000 7fae9f72a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae980b2580 con 0x7fae9806b6f0 2024-04-06T13:52:05.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.850+0000 7fae9ccc5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae980cd3e0 0x7fae980b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:05.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.850+0000 7fae9ccc5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae980cd3e0 0x7fae980b4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54892/0 (socket says 172.21.15.189:54892) 2024-04-06T13:52:05.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.850+0000 7fae9ccc5700 1 -- 172.21.15.189:0/3886018179 learned_addr learned my addr 172.21.15.189:0/3886018179 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:52:05.859 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.850+0000 7fae9d4c6700 1 --2- 172.21.15.189:0/3886018179 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae9806b6f0 0x7fae981606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.850+0000 7fae9dcc7700 1 --2- 172.21.15.189:0/3886018179 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae9815d7c0 0x7fae980c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.850+0000 7fae9ccc5700 1 -- 172.21.15.189:0/3886018179 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae9806b6f0 msgr2=0x7fae981606e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:52:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.850+0000 7fae9ccc5700 1 --2- 172.21.15.189:0/3886018179 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae9806b6f0 0x7fae981606e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.850+0000 7fae9ccc5700 1 -- 172.21.15.189:0/3886018179 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae9815d7c0 msgr2=0x7fae980c55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:52:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.850+0000 7fae9ccc5700 1 --2- 172.21.15.189:0/3886018179 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae9815d7c0 0x7fae980c55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.850+0000 7fae9ccc5700 1 -- 172.21.15.189:0/3886018179 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae90009a70 con 0x7fae980cd3e0 2024-04-06T13:52:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.850+0000 7fae9ccc5700 1 --2- 172.21.15.189:0/3886018179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae980cd3e0 0x7fae980b4bc0 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7fae9400c750 tx=0x7fae9400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:52:05.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.850+0000 7fae8dffb700 1 -- 172.21.15.189:0/3886018179 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae94011800 con 0x7fae980cd3e0 2024-04-06T13:52:05.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.851+0000 7fae8dffb700 1 -- 172.21.15.189:0/3886018179 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae94011960 con 0x7fae980cd3e0 2024-04-06T13:52:05.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.851+0000 7fae9f72a700 1 -- 172.21.15.189:0/3886018179 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae980b2860 con 0x7fae980cd3e0 2024-04-06T13:52:05.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.851+0000 7fae8dffb700 1 -- 172.21.15.189:0/3886018179 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae94011cd0 con 0x7fae980cd3e0 2024-04-06T13:52:05.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.851+0000 7fae9f72a700 1 -- 172.21.15.189:0/3886018179 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae98153fc0 con 0x7fae980cd3e0 2024-04-06T13:52:05.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.852+0000 7fae9f72a700 1 -- 172.21.15.189:0/3886018179 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae980b3020 con 0x7fae980cd3e0 2024-04-06T13:52:05.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.852+0000 7fae8dffb700 1 -- 172.21.15.189:0/3886018179 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7fae94021350 con 0x7fae980cd3e0 2024-04-06T13:52:05.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.853+0000 7fae8dffb700 1 --2- 172.21.15.189:0/3886018179 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fae7c06adb0 0x7fae7c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:05.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.853+0000 7fae8dffb700 1 -- 172.21.15.189:0/3886018179 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fae94089770 con 0x7fae980cd3e0 2024-04-06T13:52:05.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.853+0000 7fae9d4c6700 1 --2- 172.21.15.189:0/3886018179 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fae7c06adb0 0x7fae7c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:05.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.853+0000 7fae9d4c6700 1 --2- 172.21.15.189:0/3886018179 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fae7c06adb0 0x7fae7c06d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fae88005910 tx=0x7fae88009040 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:52:05.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:05.857+0000 7fae8dffb700 1 -- 172.21.15.189:0/3886018179 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fae94054520 con 0x7fae980cd3e0 2024-04-06T13:52:06.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.051+0000 7fae9f72a700 1 -- 172.21.15.189:0/3886018179 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fae98074990 con 0x7fae980cd3e0 2024-04-06T13:52:06.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.055+0000 7fae8dffb700 1 -- 172.21.15.189:0/3886018179 <== mon.0 v2:172.21.15.189:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fae94054520 con 0x7fae980cd3e0 2024-04-06T13:52:06.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.056+0000 7fae8dffb700 1 -- 172.21.15.189:0/3886018179 <== mon.0 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v100)=0 v100) v1 ==== 123+0+0 (secure 0 0 0) 0x7fae94057b40 con 0x7fae980cd3e0 2024-04-06T13:52:06.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.059+0000 7fae9f72a700 1 -- 172.21.15.189:0/3886018179 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fae7c06adb0 msgr2=0x7fae7c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:52:06.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.059+0000 7fae9f72a700 1 --2- 172.21.15.189:0/3886018179 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fae7c06adb0 0x7fae7c06d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fae88005910 tx=0x7fae88009040 comp rx=0 tx=0).stop 2024-04-06T13:52:06.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.059+0000 7fae9f72a700 1 -- 172.21.15.189:0/3886018179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae980cd3e0 msgr2=0x7fae980b4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:52:06.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.059+0000 7fae9f72a700 1 --2- 172.21.15.189:0/3886018179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae980cd3e0 0x7fae980b4bc0 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7fae9400c750 tx=0x7fae9400cb10 comp rx=0 tx=0).stop 2024-04-06T13:52:06.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.059+0000 7fae9f72a700 1 -- 172.21.15.189:0/3886018179 shutdown_connections 2024-04-06T13:52:06.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.059+0000 7fae9f72a700 1 --2- 172.21.15.189:0/3886018179 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7fae7c06adb0 0x7fae7c06d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:06.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.059+0000 7fae9f72a700 1 --2- 172.21.15.189:0/3886018179 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae9815d7c0 0x7fae980c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:06.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.059+0000 7fae9f72a700 1 --2- 172.21.15.189:0/3886018179 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae9806b6f0 0x7fae981606e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:06.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.059+0000 7fae9f72a700 1 --2- 172.21.15.189:0/3886018179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae980cd3e0 0x7fae980b4bc0 unknown :-1 s=CLOSED pgs=3256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:06.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.059+0000 7fae9f72a700 1 -- 172.21.15.189:0/3886018179 >> 172.21.15.189:0/3886018179 conn(0x7fae980cc290 msgr2=0x7fae9806dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:52:06.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.059+0000 7fae9f72a700 1 -- 172.21.15.189:0/3886018179 shutdown_connections 2024-04-06T13:52:06.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.059+0000 7fae9f72a700 1 -- 172.21.15.189:0/3886018179 wait complete. 2024-04-06T13:52:06.072 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-06T13:52:06.072 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T13:52:06.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.193+0000 7f60f0ad2700 1 Processor -- start 2024-04-06T13:52:06.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.193+0000 7f60f0ad2700 1 -- start start 2024-04-06T13:52:06.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.194+0000 7f60f0ad2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f60ec06b8c0 0x7f60ec168420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:06.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.194+0000 7f60f0ad2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60ec06bcd0 con 0x7f60ec15d7c0 2024-04-06T13:52:06.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.194+0000 7f60f0ad2700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60ec06ceb0 con 0x7f60ec0cd3e0 2024-04-06T13:52:06.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.194+0000 7f60f0ad2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60ec0cdfa0 con 0x7f60ec06b8c0 2024-04-06T13:52:06.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60e9d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f60ec15d7c0 0x7f60ec06b1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57640/0 (socket says 172.21.15.189:57640) 2024-04-06T13:52:06.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60e9d9b700 1 -- 172.21.15.189:0/140586437 learned_addr learned my addr 172.21.15.189:0/140586437 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-06T13:52:06.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60ead9d700 1 --2- 172.21.15.189:0/140586437 >> v2:172.21.15.192:3300/0 conn(0x7f60ec06b8c0 0x7f60ec168420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:06.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60e959a700 1 -- 172.21.15.189:0/140586437 <== mon.0 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615018545 0 0) 0x7f60ec06bcd0 con 0x7f60ec15d7c0 2024-04-06T13:52:06.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60e959a700 1 -- 172.21.15.189:0/140586437 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60d40035b0 con 0x7f60ec15d7c0 2024-04-06T13:52:06.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60e959a700 1 -- 172.21.15.189:0/140586437 <== mon.0 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 165203625 0 0) 0x7f60d40035b0 con 0x7f60ec15d7c0 2024-04-06T13:52:06.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60e959a700 1 -- 172.21.15.189:0/140586437 >> v2:172.21.15.192:3300/0 conn(0x7f60ec06b8c0 msgr2=0x7f60ec168420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:52:06.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60e959a700 1 --2- 172.21.15.189:0/140586437 >> v2:172.21.15.192:3300/0 conn(0x7f60ec06b8c0 0x7f60ec168420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:06.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60e959a700 1 -- 172.21.15.189:0/140586437 >> v1:172.21.15.192:6789/0 conn(0x7f60ec0cd3e0 legacy=0x7f60ec0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:52:06.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60e959a700 1 -- 172.21.15.189:0/140586437 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ec06d750 con 0x7f60ec15d7c0 2024-04-06T13:52:06.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60e959a700 1 -- 172.21.15.189:0/140586437 <== mon.0 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 567490119 0 0) 0x7f60dc001670 con 0x7f60ec15d7c0 2024-04-06T13:52:06.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60e959a700 1 -- 172.21.15.189:0/140586437 >> v1:172.21.15.189:6789/0 conn(0x7f60ec15d7c0 legacy=0x7f60ec06b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:52:06.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60e959a700 1 --2- 172.21.15.189:0/140586437 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60d4003a00 0x7f60d4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:06.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60e959a700 1 --2- 172.21.15.189:0/140586437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60d4004850 0x7f60d4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:06.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60e959a700 1 --2- 172.21.15.189:0/140586437 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60d4005210 0x7f60d400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:06.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60e959a700 1 -- 172.21.15.189:0/140586437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec0cdfa0 con 0x7f60d4004850 2024-04-06T13:52:06.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60e959a700 1 -- 172.21.15.189:0/140586437 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d400c1b0 con 0x7f60d4005210 2024-04-06T13:52:06.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60e959a700 1 -- 172.21.15.189:0/140586437 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d400c370 con 0x7f60d4003a00 2024-04-06T13:52:06.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60e9d9b700 1 --2- 172.21.15.189:0/140586437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60d4004850 0x7f60d4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:06.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60ea59c700 1 --2- 172.21.15.189:0/140586437 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60d4003a00 0x7f60d4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:06.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.195+0000 7f60ead9d700 1 --2- 172.21.15.189:0/140586437 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60d4005210 0x7f60d400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:06.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.196+0000 7f60e9d9b700 1 -- 172.21.15.189:0/140586437 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60d4003a00 msgr2=0x7f60d4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:52:06.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.196+0000 7f60e9d9b700 1 --2- 172.21.15.189:0/140586437 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60d4003a00 0x7f60d4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:06.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.196+0000 7f60e9d9b700 1 -- 172.21.15.189:0/140586437 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60d4005210 msgr2=0x7f60d400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-06T13:52:06.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.196+0000 7f60e9d9b700 1 --2- 172.21.15.189:0/140586437 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60d4005210 0x7f60d400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:06.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.196+0000 7f60e9d9b700 1 -- 172.21.15.189:0/140586437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d40035b0 con 0x7f60d4004850 2024-04-06T13:52:06.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.196+0000 7f60ead9d700 1 -- 172.21.15.189:0/140586437 reap_dead start 2024-04-06T13:52:06.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.196+0000 7f60e9d9b700 1 --2- 172.21.15.189:0/140586437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60d4004850 0x7f60d4004cd0 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00c240 tx=0x7f60d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4270a2e457eb0ca7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:52:06.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.196+0000 7f60e959a700 1 -- 172.21.15.189:0/140586437 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc007c10 con 0x7f60d4004850 2024-04-06T13:52:06.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.196+0000 7f60e959a700 1 -- 172.21.15.189:0/140586437 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f60dc00cde0 con 0x7f60d4004850 2024-04-06T13:52:06.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.196+0000 7f60e959a700 1 -- 172.21.15.189:0/140586437 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc005bc0 con 0x7f60d4004850 2024-04-06T13:52:06.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.196+0000 7f60f0ad2700 1 -- 172.21.15.189:0/140586437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60d4004850 msgr2=0x7f60d4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:52:06.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.196+0000 7f60f0ad2700 1 --2- 172.21.15.189:0/140586437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60d4004850 0x7f60d4004cd0 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00c240 tx=0x7f60d4005880 comp rx=0 tx=0).stop 2024-04-06T13:52:06.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.197+0000 7f60f0ad2700 1 -- 172.21.15.189:0/140586437 shutdown_connections 2024-04-06T13:52:06.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.197+0000 7f60f0ad2700 1 --2- 172.21.15.189:0/140586437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60d4004850 0x7f60d4004cd0 unknown :-1 s=CLOSED pgs=3258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:06.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.197+0000 7f60f0ad2700 1 -- 172.21.15.189:0/140586437 >> 172.21.15.189:0/140586437 conn(0x7f60ec0cc290 msgr2=0x7f60ec0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:52:06.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.197+0000 7f60f0ad2700 1 -- 172.21.15.189:0/140586437 shutdown_connections 2024-04-06T13:52:06.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.197+0000 7f60f0ad2700 1 -- 172.21.15.189:0/140586437 wait complete. 2024-04-06T13:52:06.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.198+0000 7f60f0ad2700 1 Processor -- start 2024-04-06T13:52:06.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.198+0000 7f60f0ad2700 1 -- start start 2024-04-06T13:52:06.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60f0ad2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60d4004850 0x7f60ec1465a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:06.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60f0ad2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60ec160130 0x7f60ec0ad9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:06.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60f0ad2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60ec0adf10 0x7f60ec0c4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:06.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60f0ad2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec0ae3a0 con 0x7f60ec160130 2024-04-06T13:52:06.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60f0ad2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec15f160 con 0x7f60ec0adf10 2024-04-06T13:52:06.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60f0ad2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec15f370 con 0x7f60d4004850 2024-04-06T13:52:06.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60e9d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60ec160130 0x7f60ec0ad9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:06.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60e9d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60ec160130 0x7f60ec0ad9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:54910/0 (socket says 172.21.15.189:54910) 2024-04-06T13:52:06.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60e9d9b700 1 -- 172.21.15.189:0/2727371716 learned_addr learned my addr 172.21.15.189:0/2727371716 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-06T13:52:06.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60ea59c700 1 --2- 172.21.15.189:0/2727371716 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60d4004850 0x7f60ec1465a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:06.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60ead9d700 1 --2- 172.21.15.189:0/2727371716 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60ec0adf10 0x7f60ec0c4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:06.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60e9d9b700 1 -- 172.21.15.189:0/2727371716 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60d4004850 msgr2=0x7f60ec1465a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:52:06.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60e9d9b700 1 --2- 172.21.15.189:0/2727371716 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60d4004850 0x7f60ec1465a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:06.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60e9d9b700 1 -- 172.21.15.189:0/2727371716 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60ec0adf10 msgr2=0x7f60ec0c4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:52:06.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60e9d9b700 1 --2- 172.21.15.189:0/2727371716 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60ec0adf10 0x7f60ec0c4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:06.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60e9d9b700 1 -- 172.21.15.189:0/2727371716 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ec15f650 con 0x7f60ec160130 2024-04-06T13:52:06.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60ead9d700 1 --2- 172.21.15.189:0/2727371716 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60ec0adf10 0x7f60ec0c4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T13:52:06.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60e9d9b700 1 --2- 172.21.15.189:0/2727371716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60ec160130 0x7f60ec0ad9d0 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 crypto rx=0x7f60d400c180 tx=0x7f60dc001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:52:06.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60daffd700 1 -- 172.21.15.189:0/2727371716 <== mon.0 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc001640 con 0x7f60ec160130 2024-04-06T13:52:06.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60daffd700 1 -- 172.21.15.189:0/2727371716 <== mon.0 v2:172.21.15.189:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f60dc0070d0 con 0x7f60ec160130 2024-04-06T13:52:06.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60f0ad2700 1 -- 172.21.15.189:0/2727371716 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60ec15f930 con 0x7f60ec160130 2024-04-06T13:52:06.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60daffd700 1 -- 172.21.15.189:0/2727371716 <== mon.0 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc007460 con 0x7f60ec160130 2024-04-06T13:52:06.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.199+0000 7f60f0ad2700 1 -- 172.21.15.189:0/2727371716 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60ec146ae0 con 0x7f60ec160130 2024-04-06T13:52:06.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.200+0000 7f60f0ad2700 1 -- 172.21.15.189:0/2727371716 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60ec0b5790 con 0x7f60ec160130 2024-04-06T13:52:06.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.201+0000 7f60daffd700 1 -- 172.21.15.189:0/2727371716 <== mon.0 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 181) v1 ==== 89247+0+0 (secure 0 0 0) 0x7f60dc0075c0 con 0x7f60ec160130 2024-04-06T13:52:06.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.201+0000 7f60daffd700 1 --2- 172.21.15.189:0/2727371716 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f60cc06adb0 0x7f60cc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T13:52:06.210 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.201+0000 7f60daffd700 1 -- 172.21.15.189:0/2727371716 <== mon.0 v2:172.21.15.189:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f60dc08a670 con 0x7f60ec160130 2024-04-06T13:52:06.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.201+0000 7f60ea59c700 1 --2- 172.21.15.189:0/2727371716 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f60cc06adb0 0x7f60cc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T13:52:06.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.202+0000 7f60ea59c700 1 --2- 172.21.15.189:0/2727371716 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f60cc06adb0 0x7f60cc06d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f60e0000c00 tx=0x7f60e0009040 comp rx=0 tx=0).ready entity=mgr.10415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T13:52:06.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.206+0000 7f60daffd700 1 -- 172.21.15.189:0/2727371716 <== mon.0 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f60dc055530 con 0x7f60ec160130 2024-04-06T13:52:06.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.445+0000 7f60f0ad2700 1 -- 172.21.15.189:0/2727371716 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60ec074970 con 0x7f60ec160130 2024-04-06T13:52:06.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.445+0000 7f60daffd700 1 -- 172.21.15.189:0/2727371716 <== mon.0 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+3547 (secure 0 0 0) 0x7f60dc017df0 con 0x7f60ec160130 2024-04-06T13:52:06.447 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-06T13:52:06.448 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":141,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11827,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/819921213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":819921213},{"type":"v1","addr":"172.21.15.192:6835","nonce":819921213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134},{"gid":12178,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/3570723137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3570723137},{"type":"v1","addr":"172.21.15.192:6834","nonce":3570723137}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":141},{"gid":13368,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3616349784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3616349784},{"type":"v1","addr":"172.21.15.189:6836","nonce":3616349784}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134},{"gid":13371,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3640528525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3640528525},{"type":"v1","addr":"172.21.15.189:6837","nonce":3640528525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134}],"filesystems":[]} 2024-04-06T13:52:06.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.447+0000 7f60f0ad2700 1 -- 172.21.15.189:0/2727371716 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f60cc06adb0 msgr2=0x7f60cc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:52:06.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.447+0000 7f60f0ad2700 1 --2- 172.21.15.189:0/2727371716 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f60cc06adb0 0x7f60cc06d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f60e0000c00 tx=0x7f60e0009040 comp rx=0 tx=0).stop 2024-04-06T13:52:06.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.447+0000 7f60f0ad2700 1 -- 172.21.15.189:0/2727371716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60ec160130 msgr2=0x7f60ec0ad9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T13:52:06.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.447+0000 7f60f0ad2700 1 --2- 172.21.15.189:0/2727371716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60ec160130 0x7f60ec0ad9d0 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 crypto rx=0x7f60d400c180 tx=0x7f60dc001a00 comp rx=0 tx=0).stop 2024-04-06T13:52:06.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.447+0000 7f60f0ad2700 1 -- 172.21.15.189:0/2727371716 shutdown_connections 2024-04-06T13:52:06.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.447+0000 7f60f0ad2700 1 --2- 172.21.15.189:0/2727371716 >> [v2:172.21.15.189:6832/111704,v1:172.21.15.189:6833/111704] conn(0x7f60cc06adb0 0x7f60cc06d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:06.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.447+0000 7f60f0ad2700 1 --2- 172.21.15.189:0/2727371716 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60ec0adf10 0x7f60ec0c4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:06.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.447+0000 7f60f0ad2700 1 --2- 172.21.15.189:0/2727371716 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f60d4004850 0x7f60ec1465a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:06.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.447+0000 7f60f0ad2700 1 --2- 172.21.15.189:0/2727371716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60ec160130 0x7f60ec0ad9d0 unknown :-1 s=CLOSED pgs=3259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T13:52:06.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.447+0000 7f60f0ad2700 1 -- 172.21.15.189:0/2727371716 >> 172.21.15.189:0/2727371716 conn(0x7f60ec0cc290 msgr2=0x7f60ec0afce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T13:52:06.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.447+0000 7f60f0ad2700 1 -- 172.21.15.189:0/2727371716 shutdown_connections 2024-04-06T13:52:06.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-06T13:52:06.447+0000 7f60f0ad2700 1 -- 172.21.15.189:0/2727371716 wait complete. 2024-04-06T13:52:06.451 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 141 2024-04-06T13:52:06.463 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-06T13:52:06.463 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-06T13:52:06.464 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-06T13:52:06.464 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:52:06.472 INFO:tasks.ceph.mds.a:Stopped 2024-04-06T13:52:06.473 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-06T13:52:06.473 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:52:06.521 INFO:tasks.ceph.mds.c:Stopped 2024-04-06T13:52:06.522 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-06T13:52:06.522 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:52:06.533 INFO:tasks.ceph.mds.b:Stopped 2024-04-06T13:52:06.533 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-06T13:52:06.533 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:52:06.584 INFO:tasks.ceph.mds.d:Stopped 2024-04-06T13:52:06.584 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-06T13:52:06.585 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-06T13:52:06.585 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:52:06.620 INFO:tasks.ceph.osd.0:Stopped 2024-04-06T13:52:06.621 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-06T13:52:06.621 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:52:06.700 INFO:tasks.ceph.osd.1:Stopped 2024-04-06T13:52:06.700 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-06T13:52:06.700 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:52:06.768 INFO:tasks.ceph.osd.2:Stopped 2024-04-06T13:52:06.769 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-06T13:52:06.769 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:52:06.838 INFO:tasks.ceph.osd.3:Stopped 2024-04-06T13:52:06.839 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-06T13:52:06.839 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:52:06.878 INFO:tasks.ceph.osd.4:Stopped 2024-04-06T13:52:06.879 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-06T13:52:06.879 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:52:06.948 INFO:tasks.ceph.osd.5:Stopped 2024-04-06T13:52:06.948 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-06T13:52:06.948 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:52:07.029 INFO:tasks.ceph.osd.6:Stopped 2024-04-06T13:52:07.029 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-06T13:52:07.030 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:52:07.098 INFO:tasks.ceph.osd.7:Stopped 2024-04-06T13:52:07.098 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-06T13:52:07.099 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-06T13:52:07.099 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:52:07.128 INFO:tasks.ceph.mgr.y:Stopped 2024-04-06T13:52:07.129 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-06T13:52:07.129 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:52:07.168 INFO:tasks.ceph.mgr.x:Stopped 2024-04-06T13:52:07.168 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-06T13:52:07.168 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-06T13:52:07.168 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:52:07.185 INFO:tasks.ceph.mon.a:Stopped 2024-04-06T13:52:07.185 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-06T13:52:07.185 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:52:07.223 INFO:tasks.ceph.mon.b:Stopped 2024-04-06T13:52:07.223 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-06T13:52:07.223 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T13:52:07.277 INFO:tasks.ceph.mon.c:Stopped 2024-04-06T13:52:07.278 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-06T13:52:07.278 DEBUG:teuthology.orchestra.run.smithi189:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2024-04-06T13:52:07.315 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T13:52:07.315 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-06T13:52:07.564 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T13:52:07.564 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-06T13:52:07.666 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T13:52:07.666 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-06T13:52:07.748 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T13:52:07.748 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-06T13:52:07.819 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T13:52:07.820 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-06T13:52:08.032 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T13:52:08.032 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-06T13:52:08.102 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T13:52:08.102 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-06T13:52:08.166 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T13:52:08.166 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-06T13:52:08.246 INFO:tasks.ceph:Archiving mon data... 2024-04-06T13:52:08.248 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7638946/data/mon.a.tgz 2024-04-06T13:52:08.248 DEBUG:teuthology.orchestra.run.smithi189:> mktemp 2024-04-06T13:52:08.267 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.0rlNLr9RvT 2024-04-06T13:52:08.268 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.0rlNLr9RvT 2024-04-06T13:52:08.961 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.0rlNLr9RvT 2024-04-06T13:52:09.048 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.0rlNLr9RvT is 4MB 2024-04-06T13:52:09.220 DEBUG:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.0rlNLr9RvT 2024-04-06T13:52:09.238 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7638946/data/mon.b.tgz 2024-04-06T13:52:09.238 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2024-04-06T13:52:09.258 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.OVqpMUSIqk 2024-04-06T13:52:09.259 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.OVqpMUSIqk 2024-04-06T13:52:10.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.OVqpMUSIqk 2024-04-06T13:52:10.095 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.OVqpMUSIqk is 4MB 2024-04-06T13:52:10.253 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.OVqpMUSIqk 2024-04-06T13:52:10.271 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7638946/data/mon.c.tgz 2024-04-06T13:52:10.271 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2024-04-06T13:52:10.327 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.SPGS6oi7H5 2024-04-06T13:52:10.328 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.SPGS6oi7H5 2024-04-06T13:52:11.026 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.SPGS6oi7H5 2024-04-06T13:52:11.112 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.SPGS6oi7H5 is 4MB 2024-04-06T13:52:11.273 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.SPGS6oi7H5 2024-04-06T13:52:11.291 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-06T13:52:11.291 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-06T13:52:11.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-06T13:52:11.358 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-06T13:52:11.358 DEBUG:teuthology.orchestra.run.smithi189:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-06T13:52:11.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-06T13:52:11.397 INFO:teuthology.orchestra.run.smithi189.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-06T13:52:11.425 INFO:teuthology.orchestra.run.smithi192.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-06T13:52:11.431 INFO:tasks.ceph:Archiving crash dumps... 2024-04-06T13:52:11.433 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7638946/remote/smithi189/crash 2024-04-06T13:52:11.434 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-06T13:52:11.474 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7638946/remote/smithi192/crash 2024-04-06T13:52:11.475 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-06T13:52:11.501 INFO:tasks.ceph:Shutting down logrotate 2024-04-06T13:52:11.501 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:52:11.518 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-06T13:52:11.607 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-06T13:52:11.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-06T13:52:11.662 INFO:tasks.ceph:Compressing logs... 2024-04-06T13:52:11.663 DEBUG:teuthology.orchestra.run.smithi189:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T13:52:11.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T13:52:39.652 INFO:tasks.ceph:Archiving logs... 2024-04-06T13:52:39.653 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7638946/remote/smithi189/log 2024-04-06T13:52:39.654 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-06T13:52:44.125 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7638946/remote/smithi192/log 2024-04-06T13:52:44.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-06T13:52:47.259 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-06T13:52:47.271 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-06T13:52:47.272 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-06T13:52:47.275 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-06T13:52:47.364 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-06T13:52:47.374 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-06T13:52:47.374 DEBUG:teuthology.orchestra.run.smithi189:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T13:52:47.377 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T13:52:47.392 INFO:teuthology.orchestra.run.smithi189.stderr:bash: ntpq: command not found 2024-04-06T13:52:47.395 INFO:teuthology.orchestra.run.smithi192.stderr:bash: ntpq: command not found 2024-04-06T13:52:47.397 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-06T13:52:47.397 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-04-06T13:52:47.397 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 -825us[ -867us] +/- 93ms 2024-04-06T13:52:47.397 INFO:teuthology.orchestra.run.smithi189.stdout:^* hv02.front.sepia.ceph.com 3 7 377 54 +292us[ +249us] +/- 65ms 2024-04-06T13:52:47.397 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 54 +250us[ +250us] +/- 68ms 2024-04-06T13:52:47.398 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T13:52:47.400 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-06T13:52:47.401 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-04-06T13:52:47.401 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 115 -791us[ -801us] +/- 92ms 2024-04-06T13:52:47.401 INFO:teuthology.orchestra.run.smithi192.stdout:^* hv02.front.sepia.ceph.com 3 8 377 109 +292us[ +281us] +/- 64ms 2024-04-06T13:52:47.401 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 +271us[ +271us] +/- 68ms 2024-04-06T13:52:47.401 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T13:52:47.401 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-06T13:52:47.411 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-06T13:52:47.412 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-06T13:52:47.445 DEBUG:teuthology.orchestra.run.smithi189:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-06T13:52:47.552 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-06T13:52:47.631 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-06T13:52:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T13:52:47.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-06T13:52:47.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T13:52:47.698 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-06T13:52:47.708 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-06T13:52:47.736 INFO:teuthology.task.internal:Duration was 6836.086023 seconds 2024-04-06T13:52:47.737 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-06T13:52:47.778 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-06T13:52:47.779 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-06T13:52:47.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-06T13:52:47.827 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-06T13:52:47.830 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-06T13:52:48.144 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-06T13:52:48.144 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi189.front.sepia.ceph.com 2024-04-06T13:52:48.144 DEBUG:teuthology.orchestra.run.smithi189:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-06T13:52:48.166 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T13:52:48.167 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-06T13:52:48.189 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-06T13:52:48.189 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T13:52:48.210 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T13:52:48.275 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-06T13:52:48.276 DEBUG:teuthology.orchestra.run.smithi189:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-06T13:52:48.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-06T13:52:48.391 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-06T13:52:48.401 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-06T13:52:48.402 DEBUG:teuthology.orchestra.run.smithi189:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-06T13:52:48.429 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-06T13:52:48.457 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-06T13:52:48.468 DEBUG:teuthology.orchestra.run.smithi189:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-06T13:52:48.472 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-06T13:52:48.495 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = core 2024-04-06T13:52:48.523 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2024-04-06T13:52:48.538 DEBUG:teuthology.orchestra.run.smithi189:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-06T13:52:48.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T13:52:48.566 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-06T13:52:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T13:52:48.594 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-06T13:52:48.605 INFO:teuthology.task.internal:Transferring archived files... 2024-04-06T13:52:48.606 DEBUG:teuthology.misc:Transferring archived files from smithi189:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7638946/remote/smithi189 2024-04-06T13:52:48.606 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-06T13:52:48.652 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7638946/remote/smithi192 2024-04-06T13:52:48.653 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-06T13:52:48.692 INFO:teuthology.task.internal:Removing archive directory... 2024-04-06T13:52:48.692 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-06T13:52:48.695 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-06T13:52:48.742 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-06T13:52:48.754 INFO:teuthology.task.internal:Not uploading archives. 2024-04-06T13:52:48.754 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-06T13:52:48.779 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-06T13:52:48.779 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-06T13:52:48.782 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-06T13:52:48.796 INFO:teuthology.orchestra.run.smithi189.stdout: 262175 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 6 13:52 /home/ubuntu/cephtest 2024-04-06T13:52:48.798 INFO:teuthology.orchestra.run.smithi192.stdout: 262175 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 6 13:52 /home/ubuntu/cephtest 2024-04-06T13:52:48.800 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-06T13:52:48.821 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-06T13:52:48.906 INFO:teuthology.nuke:Checking targets against current locks 2024-04-06T13:52:48.933 DEBUG:teuthology.nuke:shortname: smithi189 2024-04-06T13:52:48.934 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T13:52:48.959 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7638946', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 11:53:19.488950', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOXPuN01E+uZRPA3yUcIGKyxrMNnqr19RUH/qeLjk0BP0SOkrpy/5Y/DlQM9mxutjjN3CWjMmARBPcUTj2zBRV0='} 2024-04-06T13:52:48.963 DEBUG:teuthology.nuke:shortname: smithi192 2024-04-06T13:52:48.963 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T13:52:48.983 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-03_22:24:02-fs-reef-distro-default-smithi/7638946', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 11:53:19.490898', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNuSqPRtJ6bQFn5hEQRh6snW6a/7ZpRBA3dX92nc3WrdVqhFxOKLgJ9IREoVgDeyxeuIF4P2zNIcRk9MH9o9b2I='} 2024-04-06T13:52:49.018 INFO:teuthology.orchestra.console.smithi189:Power off 2024-04-06T13:52:49.019 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-06T13:52:49.043 DEBUG:teuthology.orchestra.console.smithi189:power off output: Chassis Power Control: Down/Off 2024-04-06T13:52:49.044 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T13:52:49.062 INFO:teuthology.orchestra.console.smithi192:Power off 2024-04-06T13:52:49.062 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-06T13:52:49.074 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-04-06T13:52:49.086 DEBUG:teuthology.orchestra.console.smithi192:power off output: Chassis Power Control: Down/Off 2024-04-06T13:52:49.087 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T13:52:49.114 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-04-06T13:52:53.076 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T13:52:53.115 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T13:52:53.187 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-04-06T13:52:53.226 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-04-06T13:52:57.190 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T13:52:57.227 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T13:52:57.301 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-04-06T13:52:57.340 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-04-06T13:53:01.303 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T13:53:01.341 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T13:53:01.414 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is off 2024-04-06T13:53:01.452 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is off 2024-04-06T13:53:01.515 INFO:teuthology.orchestra.console.smithi189:Power off completed 2024-04-06T13:53:01.553 INFO:teuthology.orchestra.console.smithi192:Power off completed 2024-04-06T13:53:01.658 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} duration: 6836.086022853851 failure_reason: 'Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments)' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=01aec8aa09b140a395b304664c8e49e2 status: fail success: false 2024-04-06T13:53:01.659 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T13:53:01.749 INFO:teuthology.run:FAIL